#jLdZddlZddlZddlZddlZddlZddlZddlZ ddl m Z m Z  ddl m Z ejj!ddZdZdZd Zd Zd Zd Zd ZdZdZdZe j8e dz j;dZddddedZdZ e!ejDD]k\Z#Z$e$jKdr e&e$jOdddZ 4e$dk(s:e#dze)ejDksS e&ejDe#dzZ me dk\re ndZ*dZ+d'dZ,d(d Z-d!Z.d"Z/d#Z0d$Z1d%Z2e3d&k(re2yy#e$rdZ YwxYw#e($rYwxYw#e($rYwxYw))u crawl_rugao_zxzspj.py - 如皋市数据局/行政审批局 公告公示 (www.rugao.gov.cn) https://www.rugao.gov.cn/rgszxzspj/gggs/gggs.html ⚠️ 与同站 crawl_rugao.py (=/rgscjz/tzgg/ 如皋市长江镇) 是【不同栏目】，勿覆盖 CMS: truecms (jQuery.jpage) — 列表经 dataproxy JSONP 分页 列表 API: GET {base}/truecms/messageController/getMessage.do startrecord=(page-1)*10 endrecord=page*10-1 perpage=10 columnId=5abca8e9-c1ce-4f04-ab61-1b977720e284 响应 JSON: {"result":"1059 标题…(截断) 2026-09-10]]>..."} ⚠️ 列表 title 属性=完整标题（span.list-text 被截断），优先取 title 详情: /rgszxzspj/gggs/content/.html meta ArticleTitle(完整标题) / PubDate(YYYY-M-D HH:MM:SS) 正文: div#zoom (p≈14) N)datetime timedelta) BeautifulSoup JSONL_PATHutf-8u%如皋市行政审批局-公告公示u江苏zcrawl_rugao_zxzspj.pyu 公告公示zhttps://www.rugao.gov.cnz1https://www.rugao.gov.cn/rgszxzspj/gggs/gggs.htmlz@https://www.rugao.gov.cn/truecms/messageController/getMessage.doz$5abca8e9-c1ce-4f04-ab61-1b977720e284 iG)daysz%Y-%m-%dzkMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0 Safari/537.36z.application/json, text/javascript, */*; q=0.01zzh-CN,zh;q=0.9XMLHttpRequest)z User-AgentAcceptzAccept-LanguagezX-Requested-WithRefererz--pages==z--pagesc`|sytj|}|jddjddjddjdd}tjdd|}tjdd|}tjdd |}|j S) Nru​u‌u‍uu3^\s*(?:·|·|\u00b7)?\s*(?: |\u00a0)?\s*u ^[•·]\s*\s+ )html_libunescapereplaceresubstrip)ts crawl_rugao_zxzspj.py clean_titlerCs !A (B''"5==hKSST\^`aA Er1MA ~r1%A vsAA 779ctt}|r||d<d}tdD]} ddl}ddl}|j |j j|j|||d}|jdk(rt|_ |jcSd|jz}tj d t#d |dd d |d y#t$r} t| dd}Yd} ~ Jd} ~ wwxYw)Nr rF)headerstimeoutverifyHTTP %sP [fetch fail] Z -> Tflush)dictHEADERSrangerequestsurllib3disable_warnings exceptionsInsecureRequestWarningget status_codeENCencodingtext Exceptionstrtimesleepprint) urlr!accepthlast_err_r/r0res rfetchrENs W A ( H 1X  # $  $ $W%7%7%N%N O S!WU KA}}#  vv  1==0H 1   s3Bx :$G   #1vcr{H #sA'CC C0C++C0cddl}ddl}|j|jjd}t dD]i} |j ||t|d}|jdk(rt|_ |jcSd|jz}tjdkt!d |dd d |d y#t$r}t|dd}Yd}~Jd}~wwxYw)NrrF)paramsr r!r"r#r$r%r&r'r(r)Tr*)r/r0r1r2r3r.r4r-r5r6r7r8r9r:r;r<r=) r>rGr!r/r0rArBrCrDs r fetch_urlrHcs W//FFGH 1X  # S'Z_ `A}}#  vv  1==0H 1   s3Bx :$G   #1vcr{H #s?B>B>> CCCc )*+|sddgfStjdd|tj}tjdd|tj}tjdd|tjtjz}tjdd|tjtjz}g)i*tjdd|}tr t |d }|j d gD]}|j |j d D]N}|jd xsdjddj}d|vsd|vs?|j Pt|}tr t |d }|j dd D]}|d}|jds?|jds.tjd|tjs d|vsd|vsY|xsdj}|jdxs|jdxsdj} |r(tjd|tjs|} n| xs|} |jd xs[|jdxsH|jdxs5t j"j%| j'ddxsd } t(j*j-| } )j/| | fd!t1*z} d"| d#| d$*| <|j3| |j d%D]`}|jd&xs|jdxsd}tjd'|tjsQ|j b|j d(D]@}|j d%}|s|jd r+g}|D]}|jd&xs|jdxsd}|s.t(j*j-|} |jd)xs|jdxsdjxs5t j"j%|j'ddxsd*} |j/d"| d#| d$|sd+t1*z} d,d-j5|zd.z*| <|j3| C|j d%D]}|jd&xs|jdxsd}|s.t(j*j-|} |jd)xs|jdxsdjxs5t j"j%|j'ddxsd*} d+t1*z} d/| d#| d0*| <|j3| t|})*fd1}tjd2||tjtjz}tr t |d }|j d D]%}d3D]}|j6j9|d4 '|j d5D]h}|jd sD|j;d6s3|j;d%s"|j;ds|j Y|j=jt|}g++fd7}tjd8|tjrd9nd}d4}||k7r@|}tjd:||tjtjz}||k7r@tjd.d;|tj}tjdd|tj}g}|j'd?D]}|j}|stjd@d|}tjdAd|}|jdBdjdCd}tjdDd|}tjdEd|}t?j@|}|jdFdjdGd}tjdHd|}|stjdI|r*jCD]\}}|j||}tE+D]\} }!|jdJ| z|!}tjdK|tjs*tjdL|tjrdM|z}|j/|jd?j5|}"*jCD]\}}|"j||}"tE+D]\} }!|"jdJ| z|!}"tjdNd|"}"t?j@|"}"tjdOd=|"}"tjdPd?|"}"tjdDd|"}"tGg}$}#|"j'd?D]W}tjdQdtjdEd|} | r| |#vr6|#jI| |$j/|Yd?j5|$j}"gtG}&}%)D].\}'}(|'|&vr |&jI|'|%j/|'|(f0|"||%fS#t$rY "wxYw#t$rYwxYw#t$rYwxYw)RuR正文 HTML 清洗：保留

/，附件转

绝对URL

段落rrz )flagsz ]*>z]*>.*?z]*>.*?u【字号：[^】]*】 html.parseriframeT)stylerMrz display:nonezvisibility:hiddena)hrefrOappendixz data-appendixz&\.(pdf|docx?|xlsx?|zip|rar|wps|et|ofd)z/document/downloadz/module/download/oldsrcOLDSRCz^(?:javascript:|mailto:|tel:|#)r_titletitle?附件z__ATTACH__%d__ imgsrcz(icons?/|fileTypeImages/icon_)paltu图片z __IMG__%d__z

z
z

z

*s<00ELcR S rz&]*href="([^"]+)"[^>]*>(.*?)) rMclasslangdiralignvalignwidthheightborder cellpadding cellspacingNdivtablecfj|jddtdz zS)Nr __TBL__%d__r)rmrarn)ro table_protects r _tbl_replzhtml_to_text.._tbl_repls.QWWQZ(M 2Q 677rz ]*>rzz

z  z8]*>z z(?<=>)\s*[\r\n\t]+\s*z\s*[\r\n\t]+\s*(?=<)  z [ \t]{2,}r` z z[\s\u200b\u200c\u200d\ufeff]+u^[【\[（(]?\s*(?:返回顶部|回到顶部|返回列表|打印(?:本页)?|关闭(?:窗口)?|分享|扫一扫|上一篇|下一篇)\s*[】\]）)]?$r]]z

%s

z __(?:LINK|TBL|IMG|ATTACH)__\d+__z[ \t]+\nz\n{3,}r)%rrSrcrfind_all decomposer4rrdr:r9searchrrbget_textrfrgrhrirjrkrlrmrn replace_withjoinattrspopfindunwraprritems enumeratesetadd),contentrv_selst_souprNrO_href_oldsrc real_hrefrqrrrsr[r\r]imgssrcsrwtagattrdr has_tableprevpartsblockbplainkvitbloutseenfinaldedupseen_uurrtrurs, ` @@@r html_to_textrus? 1byff]Brtt"+DLLN #"gG6 !'=9E^^Cd^3 (yEE*%)?99%NPTVXVZVZ[/47.$6!ZR..0E uuXG!%%/GRNNPGRXX.PRWY[Y]Y]%^$) $+$4u ::D:1TQUU8_TgT gg..ys/CA/FGTKS$ll228YGG&&~6*S->>CQXZ](^L%NN3'' ((~~e, $wwu~@):@b99>RTTJMMO $^^C( ,zz%(  6D#\!ggenG0AGR"$"(,,"6"6x"E"wwu~G1AGRNNPO!ww// #q0ABOFN WVY$Z[\+c,.??,1GLL4F,F,O S)s+ , ~~e, &ggen?(9?R ,,..x=wwu~?)9?RFFHGww'' #q(9:G>F#c,&77T[]`$a S!  % &%jGff> BDD244K1G !'=9E~~d+ .O.DIIMM$-. .^^E* zzz-affWo !u affSkKKMHHJ   %jGM8YY>AI D '/&&;Y rtt 5 '/ ffWj'>Gff\4=GffPRTBDD*G Ev&  KKM  FF+R 3 FF*B 2 IIdB  ' 'c 2 FF<a (z2q)!!%( fc*228SA7UC  88kmr s  &&( DAq !QA  . 2FAs -!+S1A 2yyArtt,9a1NaA QWWY/ 2 ++e C""$ 1kk!Q M*23kk-!+S12 &&4b# >C   C C &&dC (C &&FC (C &&sC (C%%D YYv ffVR B!:;cTk    Q  ++e  " " $C6E1 ;  1  aV   5  O   r   F   s`,Bn#0n#A7n3 F+n39:n34n3Cn3$En3*C o# n0/n03 o?o ooct|dz tzdzt|tztttd}tt|}|sy t j |}|jddxsdS#t$rvtjd|tj}|rJ t j |jdjddxsdcYS#t$rYnwxYw|cYSwxYw)Nr) startrecord endrecordperpagecolumnId)rGrresultz\{.*\}r) r: PAGE_SIZE COLUMN_IDrHAPI_URLjsonloadsr4r9rrrra)pagerGrqjsros r fetch_listr7sD1H 1A56 )*y> F GF +C   ZZ_vvh#)r)  IIibdd +  zz!''!*-11(B?E2E   s6*A??0C>07C*'C>* C63C>5C66C>=C>c lg}d}tjd|}|rt|jd}tjd|tj }|D]}tjd|}d}|rQtj d|jd\}} } |d| jdd| jd}tjd |tj tjzD](} tj| jdj} | jd} tjd | }|rt|jdnd}|s/ttjd d| jd}|rt|dkrd | vr| j!d r t"| z} n8| j!ds't$j&j)t"d z| } |j+| ||f+||fS)Nrz (\d+)rzz;\s*(\d{4}-\d{1,2}-\d{1,2})\s*r-r&z$]*href="([^"]+)"[^>]*>(.*?)ztitle="([^"]*)"r`z /content//http)rrintrafindallrrizfillfinditerrcrrrrrrnreBASE_URLrjrkrlrm)api_textrtotalrorecordsfragdmrymoddlmrOrtmrqs r parse_listrQs E E 5x@AAGGAJjj18RTTBG) YYUW[ \  rxx{3IAr2!bhhqk:A++EtRTTTVTXTX[Y )B$$RXXa[1779D((1+C-s3B.0+bhhqk*bC!"&&R!"EF#c(Q,$&s#$__V,||++HsNDA LL$Q ( ) ), %<rcFdx}x}}d}trO t|d}|jddtjdtji}|r|j drt |d}|jddtjdtji}|r|j drptjd |d}|rU|jd d |jd jd d |jd jd }|s.|jd} | rt | jd}|stjd|tjtjz} | rGtjdt | jd D cgc]} | s|  } } | r| dnd}|r6|jd} | r"djd| jD}|smtjd|}|rU|jd d |jd jd d |jd jd }|||fS#t$rd}Y;wxYwcc} w)NrrKmetaname ArticleTitle)rrPubDatez(\d{4})-(\d{1,2})-(\d{1,2})rrr&rh1TrSz(.*?)rzoom)idc32K|]}t|yw)N)r:).0cs r zparse_detail..s"?a3q6"?su'(\d{4})[-年/](\d{1,2})[-月/](\d{1,2}))rrrcompilercr4rrrarrr9rrircontents) html_textrvrUdate content_htmlsoupmtmdrrror]rrs r parse_detailrqs%"$$E$D< D  M:D6&"**_bdd2S)TUBbffY'#ByM26&"**Z2N)OPBbffY'YY=r)}M)+!bhhqk6G6G6JBHHUVKL]L]^_L`aDYYt_' $ (?@E  II-y"$$+ F "QWWQZ1H IO1QQOEO!&E"IBE YY&Y ! 77"?2;;"??L  YYA9 M !#!bhhqk.?.?.BBHHQKDUDUVWDXYD $ $$# D  PsEJ JJ JJctjjdtjj tjj t ddlm}tdttttfzddx}x}}t}g}g}d}tdtdzD]9} t!|} t%| \} } | r|s| }| std|zdntd |t'| | fzd| D]\} }}| |vr |j)| |r|tkr|dz }/t+| }|s|dz }Bt-|| \}}}|s|}|s|}t/|| \}}}t't1j2d d |j5}t1j6d |t0j8 xr+t1j6d |t0j8 xr| }|dks|rtd|ddzd|dz } t| | ||d |t:d t<|r-t?j@|Dcgc] \}}||d c}}dnd d }tBr|jE|n|jE||dz }td|dd|||rdnd fzdtGjHdtBs|r||tg}tGjHd<tBrltKtBdd5}|D]+}|jMt?j@|ddz- dddtdtBt'|fzdn|r||ttd ||||fzdtd!tzdy#t"$r} td|| fzdYd} ~ d} ~ wwxYwcc}}w#1swYxYw)"Nr)push_to_searchdbz"=== %s (%s) pages=%d cutoff=%s ===Tr*rzPage %d fetch error: %szPage %d: empty, stopz"Page %d: found %d items (total=%d)r`rrrrz skip empty/placeholder: %s()rUr>F) ensure_ascii) site_name source_urlr>rUpub_datesummaryrcategorytags group_namertu + %s (%s, %d字%s)&u, 表格g?) batch_labelg333333?wr)r7rzJSONL written: %s (%d rows)u=新增: %d | 跳过: %d | CUTOFF截断: %d | 列表总数: %dz=== %s done ===)'sysrginsertrfdirnameabspath__file__ crawler_librr= SITE_NAME SCRIPT_NAME_PAGESCUTOFFrr.rr9rrnrrHrrrrrrrcCATEGORY GROUP_NAMErdumpsrrmr;r<openwrite)r new_count skip_count cutoff_count seen_urls jsonl_rowsbatch list_totalrrrDrrrrrUrdhtmldtitleddaterrratts plain_lenno_pararritemfrows rmainr sHHOOArwwrwwx'@AB, .)[&RX1Y Y-.-I- \IJ EJa!$9 !$'H"(+ u J (4/t <  2dCJ5NNVZ[$)'  GUD)# MM' "v ! g&Ea *6ug*F 'FE<'3L''J $GYBFF:r7;AACDI99Y>>U!yyGRTTBBUKT} 1}4vcr{B$Oa &ggUr""(VZ $zz8<=1q+=E S`b D!!$' T" NI (F3BK 9B:,LLSW Y JJsOO' Pe U <E 3s9v *cG 4 D! D 3U;dBC D D +z3z?.KKSWX  U < I j, ;  D Ds*7 NO"1O  O#N;;O O__main__)N)Nr")4__doc__rrfrr;rhtmlr urllib.parserjrrbs4r ImportErrorenvironr4rr6rr rr rLIST_URLrrrnowstrftimer r- _MAX_PAGESrargvrrNrerri ValueErrorrnr rrErHrrrrr __name__rrr1s"?>>>(!ZZ^^L" -   3  %  % > L 2  (,,.9'2 2 < '(   chh  DAq||J QWWS!_Q/0J iAECM1 SXXa!e_-J "QA*$!D4@#%LS5l zFiM@       s5E<E8EE EEEE$#E$