vjW 4 d Z ddlZddlZddlZddlZddlZddlZddlmZmZ ej j dej j ej j e
ddlmZ ej ed z
j# d ZdZd d
iZd Zed z Zd
ZdZd Zd Zd ZddZ e dk( rU ej< Z e jA de!d e jE Z# e e#jH Z% e&de% d yy)u crawl_zjtz_hjsp.py - 台州市生态环境局-环评审批
站点: https://sthjj.zjtz.gov.cn/col/col1229856907/index.html
CMS: jcms 新版 unitbuild API
详情: meta[PubDate] + h2.tc.pl40 标题 + div.article-box 正文
修复(2026-08-08): ①article-box 非贪婪(.*?) 在表格嵌套div处截断 → 正文后半+附件丢失
②改用平衡div匹配 ③href/src 相对路径转绝对(含 /api-gateway/ 下载链接)
N)datetime timedelta)push_to_searchdbiG )daysz%Y-%m-%du1 台州市生态环境局-建设项目环评公示z
User-Agentz6Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120.0zhttps://sthjj.zjtz.gov.cnz7/api-gateway/jpaas-publish-server/front/page/build/unit
S c t j | t dd }dddddd d
|d } t j t
|d t d
}d|_ |j }|j di j dd }t j d|t j }g }|D ]k \ } }
} t j dd| j } t | dk r;|
j! d r|
nt" |
z } |j% | | | d m |S # t $ r}t d| g cY d }~S d }~ww xY w)N)pageNopageSizeF) ensure_ascii bulidstatic3519kyvKoPpKTVXJukG1eJzrfcolumnu 当前栏目列表null
1229856907) parseTypewebIdtplSetIdpageTypetagIdeditTypepageId paramJson )paramstimeoutheadersverifyutf-8datahtml z API error: zY(\d{4}-\d{2}-\d{2})\s*]*href="([^"]+)"[^>]*title="([^"]*)"\s+ http)urlpub_datetitle)jsondumps PAGE_SIZErequestsgetAPIHEADERSencoding ExceptionprintrefindallDOTALLsubstriplen
startswithBASEappend)
page_no
param_jsonr rr# r$ eitemsresultspdhrefr+ r) s
$/root/gov_crawler/crawl_zjtz_hjsp.py
fetch_listrH s7 wI FUZ[J"V+%6\ F LLVWgV[ \
vvxxx#''3
JJd bii
E G ED%vr5)//1 u:> oof-d4$;sUCD E N
s#$ s AD E(D=7E=Ec d t j || }|sy| j d|j }|dk( ry|dz }d}t j d| |d D ]J }|j d j
d r|dz
})|dz }|dk( s4| |||j z c S | |d S ) u 平衡 div 匹配: 从匹配到的 的 '>' 之后, 到对应 , 返回内部 HTML。
修复非贪婪 (.*?) 在正文含嵌套 div(表格/附件区)时截断的问题r% > z]|Nr z]*>(.*?)z<[^>]+>z \s*([^<]+?)\s*r& u [-–—|_]r z,]*class="[^"]*article-box[^"]*"[^>]*>z(]*class="[^"]*article[^"]*"[^>]*>z)flagsz!href="(?!https?://|//|#)(/[^"]+)"c 8 dt | j d dS )Nzhref="rL "r= rQ rW s rG z fetch_detail..h s" tfUWU]U]^_U`TaabFc rY z src="(?!https?://|//)(/[^"]+)"c 8 dt | j d dS )Nzsrc="rL r^ r_ r` s rG ra z fetch_detail..i s" U4&QSQYQYZ[Q\P]]^C_ rY z (\s*()?\s*
)+$)ranger/ r0 r2 r3 r6 rM textrQ r8 r9 r: splitrX I)r) _attemptrA r+ rS rE contentcs rG fetch_detailrj I s !H S'75QA AJ y E
;QVVDA CR 2B
(!&&"))EG QVV%TUAwwy'') ==? )T U 779ggiGff+R BDD@PQGff9;celmGff68_ahiGff5r7CG "gmmo %%9 As $I77I>r c g }t }t | t }t | D ] }t |dz }|st d|dz d nt d|dz dt
| d |D ] }|d r
|d t k r|d |v r |j |d t |d \ }}} |s|d }|s|d }| j sb|j ||d |d |t | d d
t d t
| d |D
cg c] }
|
d j s|
} }
| r t | d
t
| S c c}
w )NrL z Page z: 0 items (stop)z: z itemsr* r) r+ r% )r+ r)
source_urlr* site_namerh summaryz Total: rh zjtz_hjsp)setmin TOTAL_PAGESrc rH r5 r; CUTOFFaddrj r: r> SITE_NAMEr ) max_pagesrecordsseenpagespgrC itr+ rE rh rA valids rG runr} m sq G 5D ; 'EEl 26" GBqD6!12 3
1vRE
|623 B*~"Z.6"9%yD HHRY !-bi!8 E2w7
^==? NN r%y5 &7r
0
Ic'l^6
*+ 811Y<#5#5#7Q 8E 8 , u:
9s E -E __main__z--pages)typedefault)rv zDone: z records)r )'__doc__sysosr/ r6 argparser, r r pathinsertdirnameabspath__file__ crawler_libr nowstrftimers ru r2 r= r1 r. rr rH rX rj r} __name__ArgumentParserap add_argumentint
parse_argsargsry cntr5 rY rG r s - , , ( 277??277??8#<= > (
(,,.9%0
0 : :: F ? Q
R"
FF :("&H D z "BOOICO3
==?D
#C F3%x
! rY