pvj 0 d Z ddlZddlZddlZddlZddlZddlmZmZ ej j dej j ej j e ddl
mZ ej ed z
j! d ZdZd d
iZd Zed z Zd
Zd Zd Zd Zd ZddZe dk( rU ej: Z e j? de d e jC Z" ee"jF Z$ e%de$ d yy)us crawl_xingtai_sthjj.py - 邢台市生态环境局-信息公开 (邢台市政府信息公开平台)
站点: http://info.xingtai.gov.cn/areas.jsp?orgCode=001001167&type=9
CMS: 邢台市政府信息公开平台 (UTF-8)
列表: ul.list > li > a[title=标题](href=/single/114|14/ID.html) + span.time(YYYY-MM-DD)
分页: areas.jsp?orgCode=001001167&type=9&pn=N (每页20条)
详情: 标题 + div 正文容器
修复(2026-08-08): ①detail容器精确匹配 class="detail" 而非 detail-box(外壳+relateInfo+script噪声)
②附件型 href 相对路径转绝对URL (点击404=正文缺失根因) ③剥外壳+清尾部空p
N)datetime timedelta)push_to_searchdbiG )daysz%Y-%m-%du% 邢台市生态环境局-信息公开z
User-Agentz6Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120.0zhttp://info.xingtai.gov.cnz#/areas.jsp?orgCode=001001167&type=9c | dk rt n
t d| } t j |dt d }d|_ t j d|j t
j }g }|D ]] \ }}}}t j dd |xs | j } t | d
k r@t |z }|j ||| d _ |S # g cY S xY w) N z&pn= Ftimeoutheadersverifyutf-8z{<a href="(/single/\d+/\d+\.html)" target="_blank" title="([^"]*)">(.*?)</a>\s*<span class="time">(\d{4}-\d{2}-\d{2})</span>\s+ )urlpub_datetitle)LIST_URLrequestsgetHEADERSencodingrefindalltextDOTALLsubstriplenBASEappend)
page_nor ritemsresultshref
title_attr
title_textpdr s
(/root/gov_crawler/crawl_xingtai_sthjj.py
fetch_listr. s l(8*D (BC LLgwu M
JJ G
E G,1 E(j*bvrJ$<*>DDF u:> TksUCD E N s $C Cc d t j || }|sy| j d|j }|dk( ry|dz }d}t j d| |d D ]J }|j d j
d r|dz
})|dz }|dk( s4| |||j z c S | |d S ) u 平衡 div 匹配: 从匹配到的 <div...> 的 '>' 之后, 到对应 </div>, 返回内部 HTML。
修复非贪婪 (.*?)</div> 在正文含嵌套 div(表格/附件区)时截断的问题r >r z<div[\s>]|</div>Nr z<div)r searchfindstartfinditergroup
startswith)html open_tag_remgtidepthmms r- extract_balanced_divr? , s +t$A
3 "B Rx
QA
Ekk-tABx8 .
88A; ! !& ) QJE QJEzAa"((*n--
. 8O c f t j dd | } t j dd | } | S )u` 把相对路径 href/src 转成绝对 URL (附件/图片)。修复附件型公告点击404问题z!href="(?!https?://|//|#)(/[^"]+)"c 8 dt | j d dS )Nzhref="r "r# r6 r> s r- <lambda>z"absolutize_links.<locals>.<lambda>B s" VD6RTRZRZ[\R]Q^^_C` r@ z src="(?!https?://|//)(/[^"]+)"c 8 dt | j d dS )Nzsrc="r rC rD rE s r- rF z"absolutize_links.<locals>.<lambda>C s %vbhhWXk]Z[@\ r@ )r r )r8 s r- absolutize_linksrH @ s1
6668`bf gD
6635\^b cD Kr@ c t j dd| t j t j z } t j dd| t j t j z } t | } t j dd| } t j dd| } t j dd| } | j S )u7 清理正文: 去 script/relateInfo/空p, href转绝对<script.*?</script>r flagsz'<div[^>]*id="relateInfo"[^>]*>.*?</div>z+^\s*<div[^>]*class="[^"]*detail[^"]*"[^>]*>z
</div>\s*$z (<p>\s*(<br\s*/?>)?\s*</p>)+$)r r r IrH r! )cs r-
clean_contentrO F s
%r1BII4DEA
92q TVTXTXHXYAA
=r1EA
}b!$A
/Q7A 779r@ c
t d D ]( } t j | dt d }d|_ n yd}t j d|j t
j }|r[t j d d|j d
j }t j d | d j }d}t |j d
}|j st |j d }|j r t | }|j sDt |j d }|j r t j dd|t
j t
j z }g }t j d|t
j D ] \ }
t j dd
j
rt#
dk r<t%
fddD rQt j d| t
j sw| j' d r| nt( | z } |j+ d| d
d d j- | }||fS # d }Y uxY w) N r Fr r )r r r z <title>\s*([^<]+?)\s*r r u [-–—|_]r z ]*class="detail"[^>]*>z+]*class="[^"]*detail-box[^"]*"[^>]*>z$]*class="right-content"[^>]*>rJ rK z$]+href="([^"]+)"[^>]*>(.*?)z<[^>]+> c 3 & K | ] }|v
y w)N ).0knames r- z fetch_detail..t s YQqDyYs )u 返回u 更多u 上一页u 下一页z?\.(pdf|docx?|xlsx?|zip|rar|wps|jpe?g|png)$|/file/|upload|attachhttpz z
)ranger r r r r r2 r r r r6 r! splitr? rO rM r r" anyr7 r# r$ join) r _attemptr& r r: contentrN rcpartsr) rW s @r- fetch_detailrc R s !H S'75QA AJ y E
2AFFBIIFAvr1771:.446 /288:G QVV%EFA 779 )W Xwwy " ==?
!!&&*Q
R
88:.Bbii"$$>NOBE jj)PRTVXV_V_ `
C
dvvj"d399;s4y1} Y*XYY yy!ceikmkokop "oof 5d4$; |C54& AB
C ii&G '>I As $I;;Jr
c x g }t }t | D ] }t |dz }|st d|dz d nt d|dz dt | d |D ] }|d r
|d t
k r|d |v r |j
|d t |d \ }}|s|d }|j sZ|j ||d |d |d t |d d
t d t | d |D cg c] }|d j s| } }| r t | d
t | S c c}w )Nr z Page z: 0 items (stop)z: z itemsr r r r )r r
source_urlr site_namer` summaryz Total: r`
xingtai_sthjj) setr[ r. printr" CUTOFFaddrc r! r$ SITE_NAMEr )
max_pagesrecordsseenpgr' itr r` r& valids
r- runrt ~ s[ G 5DI 26" GBqD6!12 3
1vRE
|623 B*~"Z.6"9%yD HHRY )"U) 4NE77 ==? NN r%y5 zN &7r
,
Ic'l^6
*+ 811Y<#5#5#7Q 8E 8 0 u:
9s ?D7D7__main__z--pages)typedefault)rn zDone: z records)r
)&__doc__sysosr r argparser r pathinsertdirnameabspath__file__ crawler_libr nowstrftimerk rm r r# r r. r? rH rO rc rt __name__ArgumentParserap add_argumentint
parse_argsargspagescntrj rT r@ r- r s ' & & ( 277??277??8#<= > (
(,,.9%0
0 : :: F 3 Q
R# 7 7((
*X > z "BOOICO3
==?D
#C F3%x
! r@