mvj d Z ddlZddlZddlZddlZddlZddlmZmZ ej j dej j ej j e ddl
mZ ej ed z
j! d ZdZd d
iZd Zd Zd
Zd Zd ZddZedk( rU ej6 Z e j; de d e j? Z ee jB Z" e#de" d yy)uF crawl_shangluo_hpgl.py - 商洛市-环评管理 (商洛市人民政府-环评管理栏目)
站点: https://www.shangluo.gov.cn/hjj/zfxxgk1/fdzdgknr/hpgl.htm
CMS: 商洛市政府网站 (UTF-8, vsb_content)
列表: ul.lzyjList > li > a(href=../../info/1037/ID.htm) + span.date(YYYY-MM-DD)
分页: hpgl/N.htm (首页=hpgl.htm, 第2页=hpgl/2.htm, 每页15条, 共49页)
详情: 标题 + div#vsb_content 正文
修复(2026-08-08): ①vsb_content 非贪婪(.*?)</div> 在表格嵌套div处截断 → 公示表格全丢
②改用平衡div匹配 ③href/src 相对路径转绝对
N)datetime timedelta)push_to_searchdbiG )daysz%Y-%m-%du 商洛市-环评管理z
User-Agentz6Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120.0zhttps://www.shangluo.gov.cnzhjj/zfxxgk1/fdzdgknrc ^ | dk( rt dt d}nt dt d| d} t j |dt d }d |_ t
j d
|j t j }g }|D ] \ }}}t
j d d | j }t
j d
d | j }t | dk r`|j dd j dd }t |z }|j |||d |S # g cY S xY w)N /z /hpgl.htmz/hpgl/z.htm Ftimeoutheadersverifyutf-8zj<a href="((?:\.\./){2,3}info/\d+/\d+\.htm)"[^>]*>(.*?)</a>\s*<span class="date">(\d{4}-\d{2}-\d{2})</span>z<[^>]+> \s+ z ../../../z/hjj/z../../)urlpub_datetitle)BASE LIST_PATHrequestsgetHEADERSencodingrefindalltextDOTALLsubstriplenreplaceappend) page_idxr ritemsresultshref
title_textpdr s (/root/gov_crawler/crawl_shangluo_hpgl.py
fetch_listr/ s$ 1}a {),a {&
$7 LLgwu M
JJu
E G % E j"z2z288:vr5)//1 u:> ll;0887KSjsUCD E N# s $D& &D,c d t j || }|sy| j d|j }|dk( ry|dz }d}t j d| |d D ]J }|j d j
d r|dz
})|dz }|dk( s4| |||j z c S | |d S ) u 平衡 div 匹配: 从匹配到的 <div...> 的 '>' 之后, 到对应 </div>, 返回内部 HTML。
修复非贪婪 (.*?)</div> 在正文含嵌套 div(表格/附件区)时截断的问题r >r z<div[\s>]|</div>Nr z<div)r searchfindstartfinditergroup
startswith)html open_tag_remgtidepthmms r. extract_balanced_divr@ 3 s +t$A
3 "B Rx
QA
Ekk-tABx8 .
88A; ! !& ) QJE QJEzAa"((*n--
. 8O c t d D ]( } t j | dt d }d|_ n yd}t j d|j t
j }|r[t j d d|j d
j }t j d | d j }d}t |j d
}|j r|j }|j s6t |j d }|j r|j }t j dd|t
j t
j z }t j dd | }t j dd | }t j dd| }||j fS # d }Y xY w)N r
Fr
r )r r r z <title>\s*([^<]+?)\s*r r u [-–—|_]r zid="vsb_content"[^>]*>z(]*class="[^"]*content[^"]*"[^>]*>z)flagsz!href="(?!https?://|//|#)(/[^"]+)"c 8 dt | j d dS )Nzhref="r "r r7 r? s r. z fetch_detail..` s" tfUWU]U]^_U`TaabFc rA z src="(?!https?://|//)(/[^"]+)"c 8 dt | j d dS )Nzsrc="r rF rG rH s r. rI z fetch_detail..a s" U4&QSQYQYZ[Q\P]]^C_ rA z (\s*()?\s*
)+$)ranger r r r r r3 r r! r" r7 r# splitr@ I)r _attemptr( r r; contentcs r. fetch_detailrQ G sz !H S'75QA AJ y E
2AFFBIIFAvr1771:.446 /288:G QVV%>?Awwy'') ==? )T U 779ggiGff+R BDD@PQGff9;celmGff68_ahiGff5r7CG '--/ !!- As $GG r c x g }t }t | D ] }t |dz }|st d|dz d nt d|dz dt | d |D ] }|d r
|d t
k r|d |v r |j
|d t |d \ }}|s|d }|j sZ|j ||d |d |d t |d d
t d t | d |D cg c] }|d j s| } }| r t | d
t | S c c}w )Nr z Page z: 0 items (stop)z: z itemsr r r r )r r
source_urlr site_namerO summaryz Total: rO
shangluo_hpgl) setrK r/ printr$ CUTOFFaddrQ r# r& SITE_NAMEr )
max_pagesrecordsseenpgr) itr rO r( valids
r. runrb e s[ G 5DI 26" GBqD6!12 3
1vRE
|623 B*~"Z.6"9%yD HHRY )"U) 4NE77 ==? NN r%y5 zN &7r
,
Ic'l^6
*+ 811Y<#5#5#7Q 8E 8 0 u:
9s ?D7D7__main__z--pages)typedefault)r\ zDone: z records)r )$__doc__sysosr r argparser r pathinsertdirnameabspath__file__ crawler_libr nowstrftimerY r[ r r r r/ r@ rQ rb __name__ArgumentParserap add_argumentint
parse_argsargspagescntrX rA r. r| s ' & & ( 277??277??8#<= > (
(,,.9%0
0 : :: F $ Q
R$ " 6( "< > z "BOOICO3
==?D
#C F3%x
! rA