qjO.dZddlZddlZddlZddlZddlZddlZddlZddlZ ddl Z ddl Z e jZ de _e je _ddlmZmZmZdZdZdZdZd d d d d ddZdddddddZdaej6j9ddZej6j9ddZej6j9ddZd\Z Z!d\Z"Z#d Z$d!Z%hd"Z&d#Z'd$Z(d%Z)d1d&Z*d1d'Z+d(Z,d)Z-d*Z.d+Z/d,Z0d-Z1d.Z2d/Z3e4d0k(re3yy)2u 江苏环评网 js-eia.cn 首页公示专栏抓取 (getIndexData 全类型) https://www.js-eia.cn/ → 公示专栏 (AJAX: /getIndexData) 策略 (用户确认 2026-08-18): - 数据源 = 首页公示专栏 getIndexData (聚合所有类型, 带 create_time) - 只抓当日与前一日发布的公示 (2天窗口) - 7 个分组: first=信息公开(1) second=公参公示(2) full=全本公示(3) completion=竣工公示(4) adjust=调试公示(5) check=验收公示(6) pubother=其他环境信息/土壤调查/水土保持等 (pubtype 1-6) - 详情 URL: project/detail?type=N&proid=xxx 或 pubother/detail?pubid=xxx - 限速防封: 站点明确警告"异常高频访问会封号+封IP" - 列表请求间隔 3-5s, 详情请求间隔 6-10s (随机) - 登录: 复用扫码登录 cookie (jseia_cookie.txt), 失效时提示重新扫码 用法: python3 crawl_jseia.py [--pages=N] [--jsonl] [--dryrun] [--days=1] 默认: 直接写 /root/search.db NF)datetime timedeltatimezonezhttps://www.js-eia.cnu江苏环评信息公示平台u江苏zcrawl_jseia.py)u 信息公开)u 公参公示)u 全本公示)u 竣工公示)u 调试公示)u 验收公示)firstsecondfull completionadjustchecku企业环境信息u清洁生产审核u应急预案公开其他环境信息u土壤状况调查u水土保持信息)rrrr r r rDB_PATHz/root/search.db JSONL_PATH JSEIA_COOKIEz"/root/gov_crawler/jseia_cookie.txt)rr )r rzoMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36>)pdddldth1h2h3h4h5h6hrlioltdthtruldivnavpreformmainasidetabletbodytfoottheadcenterdialogfigurefooterheaderhgroupaddressarticlecaptiondetailssectionfieldset blockquote figcaptioncv|s|S|Dcgc]}t|ts|}}t|t|k7r|Stg}}|D])}||vs|j ||j |+|Dcgc]&tdk\rt fd|Ds(}}|r|S|Scc}wcc}w)u6剔除「容器段」：find_all(['p','div']) 会同时收下容器
与它内部的

， 导致同一内容重复（容器那份常还被 get_text(strip=True) 拍平）。 判据：先按值去重，再剔除被其它段完全包含的段。保护：剔除后为空则返回去重结果。 (c3:K|]}|uxr|xr|vywN).0bas /root/gov_crawler/crawl_jseia.py z(_drop_container_parts..Vs&(S!)D)Da1f)D(Ss) isinstancestrlensetaddappendany)partsrpsseenuniqrHkeeps ` rI_drop_container_partsrWGs   1jC0! 1B 1 2w#e* $D ( D= HHQKQ( V!FbLS(Sd(S%S  VD V4!T! 2 VsB1B1>+B6c|yddl}ddlmgfd|dj}|j dd|}|j dd |}|j S) ul块级边界出换行、行内节点直接拼接、
出换行（≈ 浏览器看到的换行结构）。Nrr)NavigableStringcF|jD]}t|rjt|*t |dddk(rjdLt |ddt vr+jd|jd|y)Nnamebr )childrenrKrPrLgetattr _BLOCK_TAGS)nodechrYoutwalks rIrdzbody_text..walkbs-- B"o. 3r7#VT*d2 4 VT*k9 4 R 4 R z[ \t\r\f\v]*\n[ \t\r\f\v]*r]z\n{3,} )rebs4rYjoinsubstrip)el_retrYrcrds @@@rI body_textroZsb z# C  H  A -tQ7A  61%A 779reci}tjjts7t dtt j t jdttdD]p}|j}|r|jdr'|jd}t|dk\r |d ||d <Rd |vsW|jd d\}}|||<r|S) Nu [FATAL] cookie 文件不存在: filerutf-8encoding# r r =) ospathexists COOKIE_FILEprintsysstderrexitopenrk startswithsplitrM)cookieslinerRkvs rI load_cookiesrusG 77>>+ & 0 >SZZP  [73 zz|ts+  4  u:? %aGE!H  D[::c1%DAqGAJ  Nrectdddjd|jDd}|r||d<tjj ||} tjj |dt }|jjd d S#t$r.}td |ddd|tjYd}~yd}~wwxYw)Nz?text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8zzh-CN,zh;q=0.9; c30K|]\}}|d|ywryNrErFrrs rIrJzhttp_get..C41aqc1#JC) User-AgentAcceptzAccept-LanguageCookieRefererheaderstimeoutcontextrsreplaceerrorsz [GET fail] F rq)UAriitemsurllibrequestRequesturlopen_SSL_CTXreaddecode Exceptionr~rrurlrrefererrreqrespes rIhttp_getrsS+))C7==?CC G $  .. g 6C~~%%c2x%Hyy{!!')!<<  c#2hZq,3::>sAB C)$CCctdjd|jD|xs tdzddd}tj j ||} tj j|dt }tj|jjd d S#t$r.}td |ddd|tj Yd}~yd}~wwxYw)u?JSON 接口请求 (getIndexData 需要 XHR 头, 否则 500/空)rc30K|]\}}|d|ywrrErs rIrJz http_get_json..rr/z.application/json, text/javascript, */*; q=0.01XMLHttpRequest)rrrrzX-Requested-Withrrrrsrrz [GETJSON fail] Nrrrq)rrirBASE_URLrrrrrjsonloadsrrrr~rrrs rI http_get_jsonrs))C7==?CC,hnB, G .. g 6C~~%%c2x%Hzz$))+,,WY,GHH  !#cr(1QC0szzBsAB88 C/$C**C/c tjd|xsd}|rK|jddt|jdddt|jddSy)u#'2026年08月17日' -> '2026-08-17'u"(\d{4})年(\d{1,2})月(\d{1,2})日rr-r02dr)rgsearchgroupint)sms rI parse_date_cnrsZ 7bAA''!*Qs1771:s31S_S4IJJ recttdz|tjt j t tttdz|}|sy|S)u,GET 首页建立会话 + getIndexData 全量rz /getIndexDataN) rrtimesleeprandomuniformLIST_DELAY_MINLIST_DELAY_MAXr)rds rI fetch_indexrsC X^W%JJv~~nn=>h0':A  Hrec g}tjD]\}\}}|j|gD]}t|jdd}t j |t tdjd} || cxkr|kscnf|jd|||jd|jd d | d |jd gD]}t|jdd}t j |t tdjd} || cxkr|kscnf|jd d} t| } |jd tj| d| |jd|jd d | d|S#ttf$rd} YbwxYw)u&从 getIndexData 收集窗口内条目 create_time0hours)tz%Y-%m-%dprojectproidtitler)kindtypenametypeidrrdatepubotherpubtyper rid)rrrpubidrr) GROUPSrgetrr fromtimestamprrstrftimerP TypeError ValueError PUBTYPE_MAP) rcutofftodayrrtnrittsrptptis rIcollect_window_itemsrs E!<<>>..AB /X/?wtG}oVWh/Z/X/?@7H- 6>>..AB 1$w-AWh/ >T) Krec d}tjd|tj}|r4tjdd|j dj }|s[tjd|tj}|r4tjdd|j dj }|smtjd|tj}|rFtjdd|j dj j dd}tjd d |j }|s|d }d}tjd |}|rt|j d}|s2tjd |}|rt|j d}|s|d}d}dD]}tj||}|s|j}d}|} | t|krJ|dkDrE|| | dzdk(r |dz }| dz } n|| | dzdk(r |dz}| dz } n| dz } | t|kr|dkDrE||| }ng} |rtjd|tjD]}|j dtjdd|j dj } } | jdr t| z} n| jds tdz| z} | j| xs| j dd| dt} g}| D]/}|d| vs | j!|d|j|1|} |rb ddlm}||d}|j'dd }|s5|j'dtj(d!tj }|s5|j'dtj(d"tj#}g}|r|j+gd$d%&D]}|j,d'vr!|j/d(r|j,d)k(r3|j,d(k(r|j/d(rT|j,d(k(rMt1|}tjd*d+|}tjd,d-|}|j||j,dk(r=|j'gd.d/&r|j3dd%0}|r|j||j3dd%0}|j5d1dj5d2dj5d3d}|j,d4k(r t7|}|s_tj8d5|rw|j|| Dcgc]}|d6 }}g|D]}|r r d|k(rrt|dkr~|jd7smtdz dk\r5t;fd8|Dr dzdk\rt;fd8|Dr dk\rxx|z cc<j|j|}t=|}g}|D]<}|jd7r|j|&|jd9|zd:z>d;j?|}nd}nd}| D]}d<|dd=|d6d>}||vs|d?|zz } t|j d@kr4|dAdBk(rdnd }|||| dIScc}w#t@$rd}YwxYw)Ju=从详情页提取标题/日期/正文/附件 (bs4 分段版)rzF]*style=["\'](?:[^"\']*24px[^"\']*)["\'][^>]*>(.*?)z<[^>]+>rz']*>\s*]*>(.*?)z(.*?)_rz\s+rru3发布日期[：:]\s*(\d{4}年\d{1,2}月\d{1,2}日)u(\d{4}年\d{1,2}月\d{1,2}日)rN)z+]*id=["\']article_content["\'][^>]*>zM]*class=["\'](?:content|article|detail-content|text)[^"\']*["\'][^>]*>zB]*id=["\'](?:content|article|zoom|detail)[^"\']*["\'][^>]*>r zzV]*href=["\']([^"\']+\.(?:docx?|pdf|xlsx?|rar|zip|wps|et|dps))["\'][^>]*>(.*?)rrhttp)r[rr) BeautifulSoupz html.parserr)article_content)rz#article_content|content|zoom|detailzcontent|article|notice)class_)rr+r/r)T) recursive)rr+r/rzhref=["\'](/[^"\']+)["\']c8dt|jddS)Nzhref="r"rrrs rIzparse_detail..Gs^fhihohopqhrMsrezsrc=["\'](/[^"\']+)["\']c8dt|jddS)Nzsrc="rrrrs rIrzparse_detail..Hs\dfgfmfmnofpLqre)rr+r/F)rku​u‌u‍r+uM^(责任编辑|初审|复审|终审|\[纠错\]|浏览次数|字号|分享到)r[.`s2,hYZVAY!^-Tvay?Q?QRS?T-T,hs(+z

z

r]z

z

rfrrrrrrrr)rrcontent attachments)!rgrSrjrrkrrendrMfinditerIrrrPrNrOrhrfindcompilefind_allr[ find_parentrLget_textrromatchrQrWri ImportError)rrrr date_text content_divpatstartdepthir hreftextrTattsrH_BS_soup_cdrRrltbltxt att_namesr html_partsr linkrrs @@rI parse_detailr(s^ E []acecgcghAz2qwwqz288:  II@$ M FF:r1771:6<<>E  II-tRTT : FF:r1771:6<<>DDSI!LE FF63 & , , .E W I H$OA!!''!*-  II7 > %aggaj1I L KW IIc4  EEGEEAc$i-EAI!A#;&(QJEFA!AaC[H,QJEFAFAc$i-EAIuQ-K %*KvyDFHFJFJK SARVVJAGGAJ%G%M%M%O$Ds#$__V,#~,   (C 30CDQ R  Su Axt#5" A  ?  0m,E**U'8*9Cjj2::6\^`^b^b+cjdjjrzz:SUWUYUY/Zj[E,,'Ct,T*Bww,.2>>'3JrwwZ]~ ww')bnnW.E ww')!"g ff%ACsuxy ff%@Bqsvw S) ww%'77#8E7J$ kk"Dk9!LL- ++b+5C++h3;;HbIQQRZ\^_Cww%''m288,|B$C S)/*21<<1QvY< < )AF2J!O #a&A+all36GK!O1f,h^g,h)hFA 1f,h^g,h)h6"1INI"MM!, a( )-e4 >A||C("))!,"))%!)f*<= > ))J/%ahZ'9!F)HM w  v} $G%  7==?b vz|BwCGQwQL *A$w-Pbchbiiqr  *?X?OwW[\cWdVeewx}w~GH I'R] ^^W=6 G s97H]]]0 ]];B]] ]'&]'cV|ddk(rtd|dStd|dd|dS) Nrrrrrrrr)r)rs rIitem_urlr*sE F|z!24=/BBZ,T(^,.s;Aqx;r z\d{4}-\d{2}-\d{2}ru 新增: u , 已存在: )rrwriterdumpsr~rMsqlite3connectrexecutecursorfetchone SITE_NAME GROUP_NAMErirgrrr SCRIPT_NAMErowcountcommitclose)rfrconnc new_count dup_counts rIpush_to_searchdbrEs *cG 4 C C 2E:TAB C C  U -. ??7B /DLL-. AII  ;bi\J ::< NI   w [)ZEBvJ9 WY YY;M):; ;13:NPRSYPZ1[SF##C, -ab    :: NI  KKMJJL Jykyk :;7 C Cs 1F&&F0c\ ttjD]6\}}|jds t |j ddda8dtjv}t}ttd}tj|jd}tj|tt z jd}td t d |d |tj t!|}|s0td tj tj"dt%|||}|j'dd|sWt(sPt+j,t.d} | j1d| j3} |D cgc] } t5| } } t7} t9dt;| dD]g}dj=dt;| ||dzz}| j1d|d| ||dz| j?d| jADi| jCt;|}|D cgc]} t5| | vs| }} td|dt;|d|t;|z dtj tdt;|dtj dd l"m#}|d!|DjID]&\}}td"|d#|tj (g}t|D]\}}tKjLtOjPtRtTtW||}|s7td$|dzd%t;|d&|d'dd(tj |tY||}|j[t5||d'|d)|d*|d+d,td$|dzd%t;|d-|d.d/|d'dd0d1|d) tj td2t;|d3tj |r(|D]"}tt]j^|d45$yta|y#t$rYwxYwcc} wcc} w)6Nz--days=ryrz--dryrunrrr)daysu[窗口] 当日+前u日: z ~ rqu![FATAL] getIndexData 获取失败c |dS)NrrE)xs rIrzmain..s 6reT)keyreverser.r/r0r,?z0SELECT page_url FROM gov_raw WHERE page_url IN ()c3&K|] }|d yw)rNrE)rFrs rIrJzmain..s4!14r2u [去重] u → u (已存在 u[列表] 窗口内 u 条待抓)Counterc3&K|] }|d yw)rNrE)rFrIs rIrJzmain..s:1::r2z z: z [ru] 详情失败: rrrr r )rrrr r z] [rz] &z | u 总计: u 条Fr,)1 enumeraterargvrrr DAYS_BACKrrrrrnowrr~rrrrsortrr5r6rr7r8r*rNrangerMriupdatefetchallr? collectionsrR most_commonrrrrDETAIL_DELAY_MINDETAIL_DELAY_MAXrr(rPrr4rE)rrHdryrunrrrrr col_itemsrArBrIurlsknownphbeforerRrr all_itemsidxrrrrs rIr-r-s1#((#1 << " Q 23 388 #FnG )!$ %B LL  % %j 1Ell2 !:: D DZ PF  {%xs5' BTGA  1 C  $Q6I NN*DN9 *w3 12 KKM%./ //q#d)S) 5A#D1S5M 223B IIHANPTUVWXY\W\P] ^ LL4qzz|4 4 5 Y )F1Xa[-EQF F &s9~&6l6CPYNCZB[[\]dgdndno I/z :L#: ::FFH/A 2$bn3::./Iy) w T 6>>"24DEFGT* CAwaI//?W cr@R?ST[^[e[e f  $ %D>["V*Im,  CE7!C N+3tJ/?.@2g;sPRCSBTTWXZ[aXbWcdknkukuv w Js9~&d +#**=  6B $**Re4 5 6 #w  .0Gs#R-R$*R)=R) R! R!__main__rD)5__doc__rrzrgrrrr5urllib.requestr urllib.parsesslcreate_default_contextrcheck_hostname CERT_NONE verify_moderrrrr:r;r<rrrWenvironrrrr}rrr_r` MAX_RETRYrr`rWrorrrrrrrr(r*rEr-__name__rErerIrus]&0///# %3 % % '}}22 " ,   ('''''  '+?DX&+?DXZ   **..$5 6 ZZ^^L" - jjnn^-QR "&%*"" v* "&6$$$ <(U_pT <@A$H zFre