Hjn$dZddlZddlZddlZddlZddlZddlmZejjejje Z ejjde ddlmZddlZdZddiZdZd Zej,Zej0j3edd Zd Zd Zd ZddZdZe dk(reyy)u5 爬虫：石家庄市行政审批局 - 公告公示 站点：xzspj.sjz.gov.cn CMS：zyyCMS（UUID路径 + AJAX分页） 注意：列表页页码2+仅在浏览器（Browserbase）中加载，SSH服务器curl获取不到。 Page 1可正常提取，pages参数决定获取前几页（仅page1可靠）。 N BeautifulSoup)push_to_searchdbu(石家庄市行政审批局-公告公示z User-AgentzoMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36https://xzspj.sjz.gov.cnzPhttps://xzspj.sjz.gov.cn/columns/82dab922-bb4f-4f11-a919-4ccf0f650e6a/index.htmlc0t|D]$} tj|d}d|_|cSy#tj tj f$r>}||dz krtjdYd}~mtd|d|Yd}~yd}~wwxYw)N)timeoutzutf-8u ⚠️ 请求失败(z): ) rangesessiongetencodingrequestsConnectionErrorTimeouttimesleepprint)url max_retriesattemptres crawl_sjz_xzspj.pysafe_getr s%   C ,A AJH   (((*:*:; q( 1  ,SEQC8 9  s4#BB9BBc|dk(rtn td|}t|}|gSt|jd}g}|j dt j d}|sgS|jddD]}|j d t j d  }|s,|jd d }|jdd xs|jd} | r|si|r|jds t|z}|j dt j d} | r| jdnd } |j| || d|S)uX获取列表页 - 注意：只有page=1有效（内嵌在HTML中），page2+需浏览器r z?page= html.parserdiv pagingCont_idnrlbclass_az /columns/)hrefr'titleTstriphttpzfloat:\s*right)style)r)rdate) INDEX_URLrrtextfindrecompilefind_allrget_text startswithBASE_URLappend) pagerrsoupitemscontr#a_tagr'r)date_divdate_strs rfetch_list_pager@/sDqy) 6$&@C Ay  /D E 99Urzz.99 :D   eF 3F #BJJ|$< = yy$ '2&D%..t.*DD  /d?D99U"**5F*G9H4<8$$4$0" uT8DEF" Lctt|}|yt|jd}|jdd}|s5|jdt j dtj }|s5|jdt j dtj }|sy|j}gfd }t jd ||tjtj z }d }t jd ||tjtj z }t jdd|tjtj z }t jdd|}t jdd|}t jdd|}tD]J\}}t jdd|} t jdd| } |jd|d| }Ld} tdD]2} t jd| |tj } | |k(rn| }4d} t jd| |tj }t jdd|}t jdd|}|j}|sy|S) uU获取详情页正文 — 保留

// HTML（search_app 按 HTML 渲染）r(rrconNr!znrzw|article|content|detailr$zzoom|article|contentchj|jddtdz dS)Nr@@TABLEr @@)r8grouplen)mtabless r _hold_tablez!fetch_detail.._hold_tablehs. aggaj!VQr**rAz]*>.*?
)flagsc|jd}|jd}|r|jdrd|z}d|d|dS)Nr r /rz z)rGr6)rIr'txts r _abs_linkzfetch_detail.._abs_linknsIwwqzggaj DOOC(-4D4&3%t,,rAz$]*href="([^"]*)"[^>]*>(.*?)z<(script|style)[^>]*>.*?z]*>



zk<(/?)(div|span|font|strong|b|em|i|u|h[1-6]|ul|ol|li|tbody|thead|tr|td|th|img|br|hr|section|article)([^>]*)>cNd|jd|jddSNrGrIs rzfetch_detail..}s#Qqwwqzl1771:,a8rAz,<(/?)(div|span|font|strong|b|em|i|u)([^>]*)>cNd|jd|jddSrTrWrXs rrYzfetch_detail..s1QRSTSZSZ[\S]R^_`_f_fgh_i^jjkOlrArErFc|jdj}tjd|tjrd|dSd|dS)Nr z<(table|div|p|ul|ol|h[1-6])z
z
rQrR)rGr+r2searchI)rIinners r _div_to_pzfetch_detail.._div_to_psK   " 993UBDD A5'( (UG4  rAz,
([^<]*(?:<[^/][^>]*>[^<]*)*[^<]*)
c|jd}tjdd|}tjdd|}|sd|vryd|jdS)Nr z<[^>]+>r(z |\u3000| |\sz._clean_psV vvj"e,vv0"d;e+U[[]O4((rAz

(.*?)

z\n{3,}z z

\s*

)rrr0r1r2r3r]decode_contentsrbS enumeratereplacer r+)rrr: content_divrawrKrPitt2r__new_rawrcbodyrJs @r fetch_detailrpRsP Ay  /D))Ef)-K iibjj9WY[Y]Y].^i_ ii"**5Lbdd*SiT   % % 'CF+ &&+[#RTTBDD[ QC- &&8)SPRPTPTWYW[W[P[ \C &&12s"$$+ NC &&eS )C &&&# &C &&8# ?C&!/1 VVK * VVCElnp qkkGA3b/2. /!1X&&H)UX`b`d`de c>  ) &&(Crtt >#T>2D T" ZZ7 "#CJ/B R ZZ8 8zzU"5::6K+L~~c~6LL&ue,!-rAcddlm}|t|d}|jdD]P}|j dd}|s|j dr*|r t jj||n||d<R|jdD]P}|j d d}|s|j d r*|r t jj||n||d <Rt|S) uA保留 HTML 表格结构，仅将相对链接/图片转绝对 URLrrrr&r'r()r, javascript#imgsrc)r,z//zdata:) bs4rrzr4rr6urllibparseurljoin)rtbase_urlrtblr&r'rrs rryrys! E M 2C \\# SuuVR  (CD@H ,,Xt@H--h:J&Q R   Y ' E$uSZL0OP Q!R$ S^,D 12  '(HG FY' 45kW < AaC5#i.)E#2Jrs#"" 125( 6 D  & ^ (   w  FTn-( >aB zFrA