$jL;dZddlZddlZddlZddlZddlZddlZddlZ ddl m Z m Z  ddl m Z ejj!ddZdZdZd Zd Zd Zd Zd ZedezzZedzZdZe j8e dz j;dZddddeedZdZ e!ejDD]k\Z#Z$e$jKdr e&e$jOdddZ 4e$dk(s:e#dze)ejDksS e&ejDe#dzZ me dk\re ndZ*dZ+dZ,d Z-d!Z.d"Z/e0d#k(re/yy#e$rdZ YwxYw#e($rYwxYw#e($rYwxYw)$u crawl_sdde.py — 东阿县（聊城）环评审批信息 http://www.sdde.gov.cn/channel_66bc2844a73fd74b5ed13b4f/ CMS: 山东政务公开平台（layui/sea.js 前端）+ JSON API 列表: POST http://www.sdde.gov.cn/api/mDocument body: {"method":"mList","params":{"channel_name":"66bc2844a73fd74b5ed13b4f", "per_page":10,"page":N,"and":[]},"url":"<列表页 URL>"} resp: {"total":420,"page":1,"per_page":10, "items":[{"title":…, "content":"

…

", "release":"YYYY-MM-DD HH:MM:SS", "_createtime":…, "router":"http://…/doc_.html", "id":…}]} content 即完整正文 HTML（无需再抓详情页） 详情: item.router（doc_.html） N)datetime timedelta) BeautifulSoup JSONL_PATHutf-8u东阿县-环评审批信息u山东z crawl_sdde.pyu环评审批信息zhttp://www.sdde.gov.cn66bc2844a73fd74b5ed13b4fz /channel_%s/z/api/mDocument iG)daysz%Y-%m-%dzkMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0 Safari/537.36z!application/json, text/plain, */*zzh-CN,zh;q=0.9zapplication/json)z User-AgentAcceptzAccept-Languagez Content-TypeRefererOriginz--pages==z--pagesc|sytj|}|jddjddjddjdd}tjdd|}tjdd|}tjdd|}tjd d |}|j S) Nru​u‌u‍u<[^>]+>u3^\s*(?:·|·|\u00b7)?\s*(?: |\u00a0)?\s*u ^[•·]\s*\s+ )html_libunescapereplaceresubstrip)ts crawl_sdde.py clean_titler@s !A (B''"5==hKSST\^`aA z2q!A Er1MA ~r1%A vsAA 779c>ddl}ddl}|j|jjdt t |gdtd}d}tdD]} |jtttj|dd}|jd k(r,t|_tj"|j$cSd |jz}t+j,d t/d ||fzdy#t&$r}t)|dd }Yd}~Fd}~wwxYw)NrmList) channel_nameper_pagepageand)methodparamsurlF)headersdatatimeoutverifyzHTTP %sPz [list fail] page %d -> %sTflush)requestsurllib3disable_warnings exceptionsInsecureRequestWarningCHANNELPER_PAGE LIST_PAGErangepostAPI_URLHEADERSjsondumps status_codeENCencodingloadstext Exceptionstrtimesleepprint)r$r4r5bodylast_errattemptres r fetch_listrQLs W//FFG#*$WYZ D H8  # gwTZZ=M&(8A}}#  zz!&&)) 1==0H 1   '4*: :$G   #1vcr{H #sA)C;C;; DDDc .&'(|sddgfStjdd|tj}tjdd|tj}tjdd|tjtjz}tjdd|tjtjz}g&i'tjdd|}tr t |d }|j d d D]}|d }|j ds;|j ds*tjd|tjsd|vsU|xsdj}|j dxs|j dxsdj}|r(tjd|tjs|}n|xs|}|jd xs[|j dxsH|j dxs5tjj|jddxsd}tj j#|} &j%| |fdt''z} d| d|d'| <|j)| |j dD]`} | j dxs| j dxsd} tjd | tjsQ| j+b|j d!D]@} | j d}|s| jd r+g}|D]} | j dxs| j dxsd} | s.tj j#| } | j d"xs| j dxsdjxs5tjj| jddxsd#}|j%d| d|d|sd$t''z} d%d&j-|zd'z'| <| j)| C|j dD]} | j dxs| j dxsd} | s.tj j#| } | j d"xs| j dxsdjxs5tjj| jddxsd#}d$t''z} d(| d|d)'| <| j)| t/|}&'fd*}tjd+||tjtjz}tr t |d }|j d D]%}d,D]}|j2j5|d- '|j d.D]h}|jd sD|j7d/s3|j7ds"|j7d s|j+Y|j9jt/|}g((fd0}tjd1|tjrd2nd}d-}||k7r@|}tjd3||tjtjz}||k7r@tjd'd4|tj}tjd5d6|tj}tjd7d|tj}g}|jd8D]}|j}|stjd9d|}tjd:d|}|j;d;dj;dd=|}tjd?d|}t=j>|}|j;d@d=j;dAd=}tjdBd|}|stjdC|r'jAD]\}}|j;||}tC(D]\}}|j;dD|z|}tjdE|tjs*tjdF|tjrdG|z}|j%|jd8j-|}'jAD]\}}|j;||}tC(D]\}}|j;dD|z|}tjdHd|}t=j>|}tjdId6|}tjdJd8|}tjd>d=|}tEg}!} |jd8D]W} tjdKdtjd?d| } | r| | vr6| jG| |!j%| Yd8j-|!j}gtE}#}"&D].\}$}%|$|#vr |#jG|$|"j%|$|%f0|||"fS#t0$rYwxYw#t0$rYwxYw)LuR正文 HTML 清洗：保留

/，附件转

绝对URL

段落rrz )flagsz ]*>z]*>.*?z]*>.*?u【字号：[^】]*】z html.parseraT)hrefrUappendixz data-appendixz&\.(pdf|docx?|xlsx?|zip|rar|wps|et|ofd)z/module/download/oldsrcOLDSRCz^(?:javascript:|mailto:|tel:|#))r_titletitle?附件z__ATTACH__%d__ imgsrcz(icons?/|fileTypeImages/icon_)paltu图片z __IMG__%d__z

z
z

z

*s<00ELcR S rz&]*href="([^"]+)"[^>]*>(.*?)) styleclasslangdiralignvalignwidthheightborder cellpadding cellspacingNdivtablecfj|jddtdz zS)Nr __TBL__%d__r)rqrerr)rs table_protects r _tbl_replzhtml_to_text.._tbl_repls.QWWQZ(M 2Q 677rz ]*>rzz

z  z8]*>z z(?<=>)\s*[\r\n\t]+\s*z\s*[\r\n\t]+\s*(?=<)  rz [ \t]{2,}r z z[\s\u200b\u200c\u200d\ufeff]+u^[【\[（(]?\s*(?:返回顶部|回到顶部|返回列表|打印(?:本页)?|关闭(?:窗口)?|分享|扫一扫|上一篇|下一篇)\s*[】\]）)]?$r]]z

%s

z __(?:LINK|TBL|IMG|ATTACH)__\d+__z[ \t]+\nz\n{3,}r)$rrSrgrfind_allgetsearchrrfget_textrjrkrlrmrnrorprqrr replace_with decomposejoinrHrGattrspopfindunwraprrritems enumeratesetadd))contentrz_souprTrU_href_oldsrc real_hrefrurvrwr`rarbimgssrcsr{tagattrdr has_tableprevpartsblockbplainkvitbloutseenfinaldedupseen_uurrxryrs) ` @@@r html_to_textrds 1byff]Brtt>CQXZ](^L%NN3'% (&~~e, $wwu~@):@b99>RTTJMMO $^^C( ,zz%(  6D#\!ggenG0AGR"$"(,,"6"6x"E"wwu~G1AGRNNPO!ww// #q0ABOFN WVY$Z[\+c,.??,1GLL4F,F,O S)s+ , ~~e, &ggen?(9?R ,,..x=wwu~?)9?RFFHGww'' #q(9:G>F#c,&77T[]`$a S!  % &%jGff> BDD244K1G !'=9E~~d+ .O.DIIMM$-. .^^E* zzz-affWo !u affSkKKMHHJ   %jGM8YY>AI D '/&&;Y rtt 5 '/ ffWj'>Gff\4=GffPRTBDD*G Ev&  KKM  FF+R 3 FF*B 2 IIdB  ' 'c 2 FF<a (z2q)!!%( fc*228SA7UC  88kmr s  &&( DAq !QA  . 2FAs -!+S1A 2yyArtt,9a1NaA QWWY/ 2 ++e C""$ 1kk!Q M*23kk-!+S12 &&4b# >C   C C &&dC (C &&FC (C &&sC (C%%D YYv ffVR B!:;cTk    Q  ++e  " " $C6E1 ;  1  aV   5  ]   F   sG-A3k7!F+k7 :k7k7Ck78Ek7>C l7 ll llc|sytjd|}|rU|jdd|jdjdd|jdjdSy)Nrz!(\d{4})[-/](\d{1,2})[-/](\d{1,2})r-r1r))rrrezfill)srss r norm_datersZ  6:AWWQZ)9)9!)N>Nq>QRR rcv ddl}tjjdtjj tjj tddlm }tdttttfzddx}x}}t}g}g}d}t!dtdzD]E} t#| } | td| zdn%|3| j%d}td |d | j%d d| j%d xsg} | std | zdntd| t'| fzd| D]u} t)| j%dxsd} | rt'| dkr3| j%ds| j%drV| j%dxsdj+}|s|t,j.j1t2dz|}||vr|j5|t7| j%dxs| j%dxsd}|r|tkr|dz }| j%dxsd}t9|t:r|j%dd}t=||\}}}t't?j@dd|j+}t?jBd|t>jD xr+t?jBd|t>jD xr| }|dks|rtd| ddzd|dz }t||| |d|tFdtH|r-tKjL|Dcgc] \}}||d c}}d !ndd" }tNr|jQ|n|jQ||dz }td#| dd$|||rd%ndfzdxtNs|r||t&g}tSjTd'HtNrltWtNd(d)*5}|D]+}|jYtKjL|d !d+z- dddtd,tNt'|fzdn|r||t&td-|||fzdtd.tzdycc}}w#1swYgxYw)/Nr)push_to_searchdbz"=== %s (%s) pages=%d cutoff=%s ===Tr2rzPage %d: api fail, stoptotalztotal=z per_page=r#rzPage %d: no items, stopzPage %d: found %d itemsrZrr1islinkissourcerouter/release _createtimerrrrrz skip empty/placeholder: %s()rZr(F) ensure_ascii) site_name source_urlr(rZpub_datesummaryrcategorytags group_namerxu + %s (%s, %d字%s)&u, 表格) batch_labelg333333?wr)rDrzJSONL written: %s (%d rows)u*新增: %d | 跳过: %d | CUTOFF截断: %dz=== %s done ===)-r4sysrkinsertrjdirnameabspath__file__ crawler_librrK SITE_NAME SCRIPT_NAME_PAGESCUTOFFrr<rQrrrrrrnrorpBASE_URLrr isinstancedictrrrrrgCATEGORY GROUP_NAMEr@rArrqrIrJopenwrite)r4r new_count skip_count cutoff_count seen_urls jsonl_rowsbatchrr$r,rrOrZrrvdate content_htmlrratts plain_lenno_pararritemfrows rmainr!sJHHOOArwwrwwx'@AB, .)[&RX1Y Y-.-I- \IJ E Ea!$<$ < +d2$ ?  =HHW%E E488J3GHPT U!'R +d2$ ?  '4U*<6BG)# MM' "QUU9-K}1EKLDv ! 55+1rL,-+// 2> '3L''J $GYBFF:r7;AACDI99Y>>U!yyGRTTBBUKT} 1}4uSbzANa &ggDR""(VZ $zz8<=1q+=E S`b D!!$' T" NI (E#2Ji9B:,LLSW YS* YVe U <E 3y<| *cG 4 D! D 3U;dBC D D +z3z?.KKSWX  U < 6)ZQ]9^ ^fjk i 't41> D Ds&R)"1R//R8__main__)1__doc__rrjrrIr@htmlr urllib.parsernrrbs4r ImportErrorenvironrrrCrrrrrr9r;r>r:nowstrftimerr? _MAX_PAGESrargvrrTriintrm ValueErrorrrrrrQrrr__name__rrrs ?>>>(!ZZ^^L" -   *    # $ ~/ / % %  (,,.9'2 2 <