j- dZddlZddlZddlZddlZddlZddlZddlZddlm Z ddl m Z ddl m Z dZdZdZd d d d Zd ZhdZdZdZdZdZdZedk(rdej4vredej8eZej<Zej@jCeej:jCeejEedZ#de#_$ee#jJZ&ede'e&dejPejRjUede+edd5Z,e&D]Z- ejEe-d dZ#de#_$ee-d e#jJZ.e.d!xse-d!Z/e/e-d e.d"xse-d#e.d$e.d%d&d'e.d(rej`e.d(d)*nd+d,Z1e,jeej`e1d)*d-zejjd/ ddded0e'e&d1yeyy#e3$rZ4ed.e4YdZ4[4NdZ4[4wwxYw#1swYBxYw)2u' Crawl 广东省国际工程咨询有限公司 - 公示公告 https://www.gdiecc.com.cn/gongshigonggao-list.html WAF protected by Aliyun - uses Playwright for cookie bootstrapping, then requests for all pages. List: Detail: Title:

tag text Date: 发布时间：YYYY-MM-DD in .wh-article-top Content: .wh-article-con div Attachments: Pagination: /gongshigonggao-list-{page}.html (page 1 = no suffix) N) BeautifulSoup)urljoin)async_playwrightzhttps://www.gdiecc.com.cnz2https://www.gdiecc.com.cn/gongshigonggao-list.htmlz%/root/gov_crawler/output/gdiecc.jsonlz/root/.cache/ms-playwright/chromium-1228/chrome-linux64/chromez --no-sandboxz--disable-setuid-sandbox)headlessexecutable_pathargsrzzh-CN) user_agentlocale networkidlei`) wait_untiltimeoutnamevalue) rchromiumlaunch new_contextHEADERSnew_pagegotoLIST_URLasynciosleepcookiesclose)pbrowserctxpgcrs !/root/gov_crawler/crawl_gdiecc.pybootstrap_cookiesr%s*!Q ))\ "<=*   ''|,(  <<> !ggh=%gHHHmmA8; 2EFQ1V9aj(FFmmo  "H2EFsD8D D8$D#D!D#DD#3D4D#DD#0D1D#D D#DD#4D5D#: D8D!D8D#D#D#D#D#D#D#!D8#D5)D, *D51D8>)rdddldth1h2h3h4h5h6hrlioltdthtruldivnavpreformmainasidetabletbodytfoottheadcenterdialogfigurefooterheaderhgroupaddressarticlecaptiondetailssectionfieldset blockquote figcaptionc|yddl}ddlmgfd|dj}|j dd|}|j dd |}|j S) ul块级边界出换行、行内节点直接拼接、
出换行（≈ 浏览器看到的换行结构）。Nr)NavigableStringcF|jD]}t|rjt|*t |dddk(rjdLt |ddt vr+jd|jd|y)Nrbr )children isinstanceappendstrgetattr _BLOCK_TAGS)nodechrPoutwalks r$r]zbody_text..walkDs-- B"o. 3r7#VT*d2 4 VT*k9 4 R 4 R z[ \t\r\f\v]*\n[ \t\r\f\v]*rSz\n{3,}z )rebs4rPjoinsubstrip)el_retrPr\r]s @@@r$ body_textrg<sb z# C  H  A -tQ7A  61%A 779r^c$|dk(rtSd|dS)Nz.https://www.gdiecc.com.cn/gongshigonggao-list-z.html)r)page_nums r$ get_page_urlrkWs1} ;H:U KKr^c,t|d}g}|jdD]}|jd}t|dkr#|D]}|jdd}|s|j dd }d |vr0|j d st t|}|jd }|sfd } |jd } | r;| jd } tjd| } | r| jd} |j||| d|S)N html.parserr5r0aThrefrqrOzgongshigonggao-showhttprcspan(\d{4}-\d{1,2}-\d{1,2})ri)urltitledate) rfind_alllenfindget startswithrBASE_URLget_textr_searchgrouprV) htmlsoupitemsr5lisr0rorqrwrxrttxtms r$ parse_listr\s } -D EmmD!Fkk$ s8a<  FB$'A55$D$D0??6*x.JJTJ*ED776?Dmm$m/II8#>771:D LLtD E' F F0 Lr^ct|d}d}|jdd}|r|jd}n|jd}|r|jd}d}|jdd}|r;|jd}tjd |} | r| j d }|sZd D]U} |jd| } | s| jd}tjd |} | sD| j d }n|jdd } | s|jdd } d} d}g}| r| j dD]:}gd}t|jD]}||vs|jdr||=<t| } t| }| j ddD]}|jdd}tjd|js:|jdr|n t||}|j||jdxst j"j%|d||| |r|dd|dSd|dS)NrmrOr6zwh-article-top)class_r)Trsruri) ingongshiz wh-articlezwh-article-concontent)rqsrcalttargetborderzmso-rorprqz"\.(pdf|doc|docx|xls|xlsx|zip|rar)$rr)rvtexti)rwpub_date content_htmlsummary attachments)rr{rr_rrrylistattrsr}rWrgr|lowerrrVospathbasename)rvrrrwtop_divr)rdate_divrrclsrd content_divrrrtag attrs_to_keepattrroahreffull_urls r$ parse_detailrysu } -D Eii&6i7G \\$  YYt_  $ 'Hyy'7y8Hd+ II0# 6 wwqzH . C5-Bkkk-II8#> wwqzH ))E*:);K iiii8 LGK''- "CFMSYY "},T__V5LD  " " ;' K(%%c%5 qAEE&"%Eyy> N$)$4$4V$<5'#uBU""8QZZdZ=S=nWYW^W^WgWghmWn#op  q$$+74C="  24"  r^c tdtjt}td|j Dcic]\}}t |dkDs||dddzc}}t j}|jjt|jj|td|jtd}d|_t|j }td t |d d }t#d d D]}t%|} |j|d} d| _d| j vrtd|dnt| j } t | dk(rtd|dnY|j'| |}|dzdk(r'td|dt | dt |dt)j*dtdt |d|t/j0t.j2j5t6dd} t9t6dd5} t;|D]x\}}t |dd kDr |ddd dzn|d}td!|d zd"t |d#| |j|d$d}d|_d|j vr`td%tjt}|jj||j|d$d}d|_t=|d$|j }|dxs|d}||d$|d&xs|d'|d(|d)d*d+|d,rt?j@|d,d-.nd/d0}| jCt?j@|d-.d1z| d z } t)j*d3{ dddtd4| d5t6ycc}}w#t,$r} td|d| Yd} ~ 4d} ~ wwxYw#t,$r} td2|d$d| Yd} ~ d} ~ wwxYw#1swYxxYw)6Nz+Bootstrapping WAF cookies via Playwright...z Got cookies:z...zFetching page 1...rutf-8z Found  items on page 1rie renderDataz Page z : WAF blockedrz: 0 items, endz: z items (total: )g333333?z: ERROR z Total items from list: z , pages: Texist_okwencodingrw(z [/z] rvz. WAF blocked on detail! Re-bootstrapping...rrxrr'广东省国际工程咨询有限公司企业rF ensure_asciirOrwrvrxrr site_namerrrSz [ERROR] ?z Done! z records written to )"printrrunr%rrzrequestsSessionheadersupdaterrr|rrrrrangerkextendtimer Exceptionrmakedirsrdirname OUTPUT_FILEopen enumeraterjsondumpswrite)rkvsrespr total_pagesrrvr page_itemse record_countfiitempreviewdetail final_titlerecords r$r:r:s 78kk+-.G .w}}Vtq!#a&SU+1afUl?VWAIIWIIW  5525 &DDM tyy !E HSZL 0 12K 1c]1o c2&A AJqvv%s-01#AFF+J:!#s.12 LL $K2v{s"S_$5_SZLPQRS JJsO!* %c%j\;- HIKK ,t<L k3 1Q ' GAt47W 4F4Kd7mCR(50QUV]Q^G C!uAc%j\G95 6 9uuT%["u5 ' 499,JL%kk*;*=>GII$$W-55eb59D$+DM%d5k499=$Wo>g (;":.>$v,%n5%i0!J%\bcp\q4::f].CRW#Xwy  6>EF!  JJsO; @ H\N"6{m DESW@  GA3hqc* +  J 9 T%[MA3788 99siP  P -7P&2PAPAQ.DP/ Q P,P''P,/ Q8Q QQQQ"__main__z --incrementalz#Incremental mode: crawl page 1 onlyrrrzFound rTrrrrvrwrrxrrrrrFrrOrrSzERROR: rzIncremental done: z records)6__doc__rrrr_rsysrr`r urllib.parserplaywright.async_apirr~rrrr%rYrgrkrrr:__name__argvrrrrrrrr|rrrrrzrrrrrrrrrrrrrrr^r$rsv 2111 1 & ?5 QO0  .* 6L :?BMF^ z#((" 34'++/12 H    ! !uuXru* 499% s5zl"234 BGGOOK04@ +sW 5  )55eb59D$+DM)$u+tyyAF"(/"BT']K!,#E{ &z 2 Bd6l#).#9#))#4%N!)`fgt`uztzz&2GV['\{} FGGJDJJvEBTIJ 3'  * "3u:,h78 G<!)GA3-(()%  s7I BH#*I#H=( H83I8H==II