ujddlZ ddlZddlZddlZddlZddlmZmZmZddlZddlZejjdejjejje ddlmZej ej"j$dZdZej*ej,edz j/dZd Zd d d d dddd dddd dddd dddd dZdZdZdZdZd,dZe d k(rddl!Z!e!jDZ#e#jId!d"#e#jId$e%%e#jMZ'e'jPxs)e)ejTd&kDxrejTd&d'k(Z(ejZ+ee(e'jX(e-d)eje+z d*d+yy)-N)datetimetimezone timedelta)push_to_searchdbu 大西北网zhttp://www.dxbei.comiG)daysz%Y-%m-%dz[Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/125.0.0.0 Safari/537.36z /a/xinjiang/i1u大西北-新疆)pathtidnamez /a/ningxia/i2u大西北-宁夏z /a/qinghai/i4u大西北-青海z /a/shanxi/i3u大西北-陕西z/news/gansuxinwen/,u大西北-甘肃)xinjiangningxiaqinghaishanxigansuctj}|jjtdd |j t d|S#Y|SxYw)Nzzh-CN,zh;q=0.9)z User-AgentzAccept-Languagetimeout)requestsSessionheadersupdateUAgetBASE_URL)ss /root/gov_crawler/crawl_dxbei.py make_sessionrsSAIIB;KLM  h# H  Hs AActdD]k} |j|d}d|_|jj rd|jddvr|jcSt j dmy#YxYw)Nrzutf-8z 403 Forbiddend)rangerencodingtextstriptimesleep)sessionurl_rs r fetch_htmlr-&sv 1X  C ,A AJvv||~/"Evv  1    s AA;;A?cg}tjd|D]x}|jd}tjdd|jdj }|r|j dr\t |z}|j||dgztjd|}ttt|t|D] }||||d<|S)u提取 (title, url, date_str)z7]*href="(/[^"]+)"[^>]*class="title"[^>]*>(.*?)<[^>]+>z/w/u1日期[：:]\s*(\d{4}-\d{2}-\d{2})) refinditergroupsubr& startswithrappendfindallr#minlen) htmlr itemsmhreftitler*datesis r parse_listrC2s E [[SUY Z'wwqzz2qwwqz288:. o eS"%& ' JJKT RE 3s5z3u:. /Aha  LcHt||}|syi}tjd|}|r"|jdj |d<tjd|tj }|r|jdj }tj dd|tj tjz}tj d d|tj tjz}tj d d t|}tj d d t|}||d<tjd|}|r|jddd|d<|jd|jd|jdfS)N)NNNu)([^<]+?)(?:_大西北网)?r/r@z1
]*>(.*?)z]*>.*?r1)flagsz]*>.*?zsrc="(?!https?://)zsrc="zhref="(?!https?://)zhref="contentz'(\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2}) pub_date) r-r3searchr5r&DOTALLr6Irr)r)r*r<resultr>rGs r fetch_detailrNCsS gs #D  F >EA''!***,w FbiiXA''!*""$&&5r7"))VXVZVZJZ[&&3R TVTXTXHXY&&.%z0BGL&&/6(1DgN#y td| dt|dt|dg}|D]\}}}|r |tkr|r| |k\rntd| dzd|dddddt!||\}}}|xs|}|xs|xsd }t j"d!d|xsd j%dd"}t j"d#d|}|j'|d|||xsd |||dd$| dz } td%t)j*d&|r t-|d'|r| |k\rnt)j*d(td)|dd*| d+|| z }tddd,|d-dycc} wcc}}w).Nr z2==================================================u 🏠 r ru❌ 首页获取失败，跳过r/list_r u_(\d+)\.html[^>]*>末页z _(\d+)\.htmlu 📊 共 u 页u🔄 增量模式: 取前 r+z.htmlu ⏹ 第u 页最早 u ，停止u 📄 第u页 (~)z [z] 2z... T)endflushr1r0r z\s+) site_namer@r*rGrIsummarytagsu✅g?dxbeig?u ✅ z: u 条u 🎉 全部完成! 共 u 条 )rSECTIONSr=printr-rr3rJintr5r9maxr:r#rCTHREE_YEARS_AGOrNr6r&r8r'r(r) incrementallimitr) total_allkeysecr<pagesr>xnumscountpg page_htmlr*r=r+d page_datesoldestbatchr@ list_datedt_titlerGrI final_title final_daterYs rmainrtZsnGINN$HS 6('#f+b9:'hZF }#=> 3 4  IIs5zl*BCT J  OE$&JJ%E |rs  22277??277??8#<=>(++BBC !8<< - u0EEOOPZ[b(=OP&=OP&=OP%=OP-cCUV    "N.LK\ z $X $ $ &F  =  ,    D""Ps388}q'8'OSXXa[C=OK B[ 3 LR,A ./rD