$jDdZddlZddlZddlZddlZddlZddlZddlZ ddl m Z m Z  ddl m Z ejj!ddZdZdZd Zd Zd Zd Zed zZedzZe j4e dz j7dZddddZdZeej@D]k\Z!Z"e"jGdr e$e"jKdddZ4e"dk(s:e!dze'ej@ksS e$ej@e!dzZmedk\rendZ(dZ)d"dZ*dZ+dZ,dZ-d Z.e/d!k(re.yy#e$rdZ YwxYw#e&$rYwxYw#e&$rYwxYw)#u crawl_xaepb.py — 西安市生态环境局 通知公告 https://xaepb.xa.gov.cn/xwzx/tzgg/1.html CMS: 静态 HTML 站群（/xwzx/tzgg/N.html 分页） 列表:
11
2026-09

标题

摘要…
分页: 第 1-10 页 = /xwzx/tzgg/{N}.html；第 11 页起 = /xwzx/tzgg/list/{N}.html 详情: /xwzx/tzgg/.html meta ArticleTitle(完整标题) / PubDate(带时分) 正文: div.m-txt-article#article（保留

/） N)datetime timedelta) BeautifulSoup JSONL_PATHutf-8u%西安市生态环境局-通知公告u陕西zcrawl_xaepb.pyu 通知公告zhttps://xaepb.xa.gov.cnz/xwzx/tzgg/%d.htmlz/xwzx/tzgg/list/%d.htmliG)daysz%Y-%m-%dzkMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0 Safari/537.36z?text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8zzh-CN,zh;q=0.9)z User-AgentAcceptzAccept-Languagez--pages==z--pagesc|sytj|}|jddjddjddjdd}tjdd|}tjdd|}tjdd|}tjd d |}|j S) Nru​u‌u‍u<[^>]+>u3^\s*(?:·|·|\u00b7)?\s*(?: |\u00a0)?\s*u ^[•·]\s*\s+ )html_libunescapereplaceresubstrip)ts crawl_xaepb.py clean_titler>s !A (B''"5==hKSST\^`aA z2q!A Er1MA ~r1%A vsAA 779ctt}d}tdD]} ddl}ddl}|j |j j|j|||d}|jdk(rt|_ |jcSd|jz}tj dt#d |dd d |d y#t$r}t|dd}Yd}~Jd}~wwxYw)NrF)headerstimeoutverifyzHTTP %sPz [fetch fail] Zz -> Tflush)dictHEADERSrangerequestsurllib3disable_warnings exceptionsInsecureRequestWarningget status_codeENCencodingtext Exceptionstrtimesleepprint) urlrhlast_errattemptr*r+res rfetchr?Js W AH8  # $  $ $W%7%7%N%N O S!WU KA}}#  vv  1==0H 1   s3Bx :$G   #1vcr{H #sA'C C C)C$$C)c .&'(|sddgfStjdd|tj}tjdd|tj}tjdd|tjtjz}tjdd|tjtjz}g&i'tjdd|}tr t |d }|j d d D]}|d }|j ds;|j ds*tjd|tjsd|vsU|xsdj}|j dxs|j dxsdj}|r(tjd|tjs|}n|xs|}|jd xs[|j dxsH|j dxs5tjj|jddxsd}tj j#|} &j%| |fdt''z} d| d|d'| <|j)| |j dD]`} | j dxs| j dxsd} tjd | tjsQ| j+b|j d!D]@} | j d}|s| jd r+g}|D]} | j dxs| j dxsd} | s.tj j#| } | j d"xs| j dxsdjxs5tjj| jddxsd#}|j%d| d|d|sd$t''z} d%d&j-|zd'z'| <| j)| C|j dD]} | j dxs| j dxsd} | s.tj j#| } | j d"xs| j dxsdjxs5tjj| jddxsd#}d$t''z} d(| d|d)'| <| j)| t/|}&'fd*}tjd+||tjtjz}tr t |d }|j d D]%}d,D]}|j2j5|d- '|j d.D]h}|jd sD|j7d/s3|j7ds"|j7d s|j+Y|j9jt/|}g((fd0}tjd1|tjrd2nd}d-}||k7r@|}tjd3||tjtjz}||k7r@tjd'd4|tj}tjd5d6|tj}tjd7d|tj}g}|jd8D]}|j}|stjd9d|}tjd:d|}|j;d;dj;dd=|}tjd?d|}t=j>|}|j;d@d=j;dAd=}tjdBd|}|stjdC|r'jAD]\}}|j;||}tC(D]\}}|j;dD|z|}tjdE|tjs*tjdF|tjrdG|z}|j%|jd8j-|}'jAD]\}}|j;||}tC(D]\}}|j;dD|z|}tjdHd|}t=j>|}tjdId6|}tjdJd8|}tjd>d=|}tEg}!} |jd8D]W} tjdKdtjd?d| } | r| | vr6| jG| |!j%| Yd8j-|!j}gtE}#}"&D].\}$}%|$|#vr |#jG|$|"j%|$|%f0|||"fS#t0$rYwxYw#t0$rYwxYw)LuR正文 HTML 清洗：保留

/

，附件转

绝对URL

段落rrz )flagsz ]*>z]*>.*?z]*>.*?u【字号：[^】]*】 html.parseraThrefrEappendixz data-appendixz&\.(pdf|docx?|xlsx?|zip|rar|wps|et|ofd)z/module/download/oldsrcOLDSRCz^(?:javascript:|mailto:|tel:|#)r_titletitle?附件z__ATTACH__%d__ imgsrcz(icons?/|fileTypeImages/icon_)paltu图片z __IMG__%d__z

z
z

z

*s<00ELcR S rz&]*href="([^"]+)"[^>]*>(.*?)) styleclasslangdiralignvalignwidthheightborder cellpadding cellspacingNdivtablecfj|jddtdz zS)Nr __TBL__%d__r )rbrVrc)rd table_protects r _tbl_replzhtml_to_text.._tbl_repls.QWWQZ(M 2Q 677rz ]*>r zz

z  z8]*>z z(?<=>)\s*[\r\n\t]+\s*z\s*[\r\n\t]+\s*(?=<)  rz [ \t]{2,}r z z[\s\u200b\u200c\u200d\ufeff]+u^[【\[（(]?\s*(?:返回顶部|回到顶部|返回列表|打印(?:本页)?|关闭(?:窗口)?|分享|扫一扫|上一篇|下一篇)\s*[】\]）)]?$r{]]z

%s

z __(?:LINK|TBL|IMG|ATTACH)__\d+__z[ \t]+\nz\n{3,}r)$rrSrXrfind_allr/searchrrWget_textr[r\r]r^r_r`rarbrc replace_with decomposejoinr5r4attrspopfindunwraprrritems enumeratesetadd))contentrk_souprCrE_href_oldsrc real_hrefrfrgrhrQrRrSimgssrcsrltagattrdr} has_tableprevpartsblockbplainkvitbloutseenfinaldedupseen_uurrirjr|s) ` @@@r html_to_textr]s 1byff]Brtt>CQXZ](^L%NN3'% (&~~e, $wwu~@):@b99>RTTJMMO $^^C( ,zz%(  6D#\!ggenG0AGR"$"(,,"6"6x"E"wwu~G1AGRNNPO!ww// #q0ABOFN WVY$Z[\+c,.??,1GLL4F,F,O S)s+ , ~~e, &ggen?(9?R ,,..x=wwu~?)9?RFFHGww'' #q(9:G>F#c,&77T[]`$a S!  % &%jGff> BDD244K1G !'=9E~~d+ .O.DIIMM$-. .^^E* zzz-affWo !u affSkKKMHHJ   %jGM8YY>AI D '/&&;Y rtt 5 '/ ffWj'>Gff\4=GffPRTBDD*G Ev&  KKM  FF+R 3 FF*B 2 IIdB  ' 'c 2 FF<a (z2q)!!%( fc*228SA7UC  88kmr s  &&( DAq !QA  . 2FAs -!+S1A 2yyArtt,9a1NaA QWWY/ 2 ++e C""$ 1kk!Q M*23kk-!+S12 &&4b# >C   C C &&dC (C &&FC (C &&sC (C%%D YYv ffVR B!:;cTk    Q  ++e  " " $C6E1 ;  1  aV   5  ]   F   sG-A3k7!F+k7 :k7k7Ck78Ek7>C l7 ll llcg}ts|St|d}|jdtjdD]$}|j dd}|s|j dxsd j }t|j d xsd }|s/|j d }|rt|jd d }|rt|dkrtjd|s|jdr t|z}n1|jds tjj||}d } |j dtjd} | r| j dtjd} | j dtjd} | r| rtj dd | jd } tjd| jd }| rH|rF|j#dd|j#dj%dd| j%d} |j'||| f'|S)NrBarticlez card-type)class_rCTrDrErrKh3rrIr#z/xwzx/tzgg/\d+\.html/httprxz news-datez\bdate\bz\byear\bz\Dz(\d{4})[-/](\d{1,2})r -)rrrcompilerr/rrrrcrrZBASE_URLr_r`rarrVzfillrb) html_textrkrsoupartrCrErKrdatendddaydyearddyms r parse_listrs E  M 2D}}Yrzz,/G}H* HHStH $ f #**,AEE'N0b1$B#BKK4K$@AE Q yy0$7  ??3 d?D(<<''$7D XXeBJJ|$zparse_detail..Ks&C!s1v&Csz(.*?)z[-|]ru'(\d{4})[-年/](\d{1,2})[-月/](\d{1,2}))rrrrrXr/rrrVrr select_onercontentsr4rr^r)rrkrKr content_htmlrmtmddmrelrdrrSrs r parse_detailr7sA"$$E$D< D  M:D6&"**_bdd2S)TUBbffY'#ByM26&"**Z2N)OPBbffY'YYCR ]S)+!bhhqk6G6G6JBHHUVKL]L]^_L`aDYYt_' $ (?@Ei(PDOOd t@|r-tCjD|Dcgc] \}}||d c}}dnd d }tFr|jI|n|jI||dz }td|dd|||rdnd fzdtKjLdtFs|r||tg}tKjLd_tFrltOtFdd5}|D]+}|jQtCjD|dd z- dddtd!tFt-|fzdn|r||ttd"|||fzdtd#tzdy#t($r} td|| fzdYd} ~ d} ~ wwxYwcc}}w#1swYxYw)$Nr)push_to_searchdbz"=== %s (%s) pages=%d cutoff=%s ===Tr%r zPage %d fetch error: %szPage %d: empty, stopzPage %d: no items, stopzPage %d: found %d itemsrrrrr z skip empty/placeholder: %s()rKr9F) ensure_ascii) site_name source_urlr9rKpub_datesummaryrcategorytags group_nameriu + %s (%s, %d字%s)&u, 表格g?) batch_labelg333333?wr)r2r~zJSONL written: %s (%d rows)u*新增: %d | 跳过: %d | CUTOFF截断: %dz=== %s done ===))r*sysr\insertr[dirnameabspath__file__ crawler_librr8 SITE_NAME SCRIPT_NAME_PAGESCUTOFFrr) LIST_TPL_DEEPLIST_TPLr?r4rrcrrrrrrrrXCATEGORY GROUP_NAMEjsondumpsrrbr6r7openwrite)r*r new_count skip_count cutoff_count seen_urls jsonl_rowsbatchpager9rr>rrgrKrdhtmldtitleddaterrratts plain_lenno_pararritemfrows rmainr[sHHOOArwwrwwx'@AB, .)[&RX1Y Y-.-I- \IJ Ea!$;&*Rimd"X_ c I (4/t < 9c* +d2$ ?  '4U*<>U!yyGRTTBBUKT} 1}4vcr{B$Oa &ggUr""(VZ $zz8<=1q+=E S`b D!!$' T" NI (F3BK 9B:,LLSW Y JJsOO' Pe U <E 3w;z *cG 4 D! D 3U;dBC D D +z3z?.KKSWX  U < 6)ZQ]9^ ^fjk i 't4G  +tQi7t D  T>  D Ds* N>&O'",1O-> O$OO$-O6__main__))0__doc__rr[rr6rhtmlr urllib.parser_rrbs4r ImportErrorenvironr/rr1rrrrrrrnowstrftimerr( _MAX_PAGESrargvrrCrZintr^ ValueErrorrcrrr?rrrr__name__rrrs ?>>>(!ZZ^^L" -   3    $ * *44 (,,.9'2 2 <