ͫj+ h d Z ddlZddlZddlZddlZddlZddlZddlZddlm Z ddl
Z ej ej j dZdZedz ZdZdd d
d Z ej( d d
Zh dZd Zd dZd Zd dZd Zd Z d Z d Z d Z e dk( rdZ! e"ejF dkD r[ e$ejF D ]F \ Z%Z&e&dv r4e%dz e"ejF k s# e'ejF e%dz Z! n e'e& Z!H e(d e)e!d Z!e!Z e yy# Y ,xY w# Y oxY w) uT yjgx.gov.cn 阳江滨海新区（阳江高新区）- 环境保护信息公开 爬虫 N
BeautifulSoupu@ 阳江滨海新区（阳江高新区）-环境保护信息公开zhttp://www.yjgx.gov.cnz"/zwgk/zdlyxxgk/hjbhxxgk/index.html z[Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/120.0.0.0 Safari/537.36z?text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8zzh-CN,zh;q=0.9,en;q=0.8)z
User-AgentAcceptzAccept-Language SEARCH_DBz/root/search.db>) pdddldth1h2h3h4h5h6hrlioltdthtruldivnavpreformmainasidetabletbodytfoottheadcenterdialogfigurefooterheaderhgroupaddressarticlecaptiondetailssectionfieldset
blockquote
figcaptionc | yddl }ddlm g fd | dj }|j dd| }|j dd | }|j S )
ul 块级边界出换行、行内节点直接拼接、
出换行（≈ 浏览器看到的换行结构）。N r )NavigableStringc F | j D ] }t | rj t | *t |dd dk( rj d Lt |dd t
v r+j d | j d | y )Nnamebr
)children
isinstanceappendstrgetattr _BLOCK_TAGS)nodechr3 outwalks
crawl_yjgx.pyrA zbody_text..walk) s --
B"o.
3r7#VT*d2
4 VT*k9
4 R
4 R
z[ \t\r\f\v]*\n[ \t\r\f\v]*r7 z\n{3,}
)rebs4r3 joinsubstrip)el_retr3 r@ rA s @@@rB body_textrM ! sb z#
C H
A -tQ7A 61%A 779rC c ddl m} |t | d }|j d D ]P }|j dd }|s|j d r*|r t j j || n||d< R |j d D ]P }|j d d }|s|j d
r*|r t j j || n||d < R t | S ) uA 保留 HTML 表格结构，仅将相对链接/图片转绝对 URLr r html.parserahrefr2 )http
javascript#imgsrc)rR z//zdata:) rF r r; find_allget
startswithurllibparseurljoin)r base_urlr tblrP rQ rU rV s rB html_table_to_htmlr_ < s !
E
M
2C
\\#
SuuVR (CD@H ,,Xt@H --hA''!* Qs1771:s 31S_S4IJJ
rC c g }| j dd }|s
t d |S |j d }t | dk r
t d |S |d }|j d D ] }|j d
}|s|j d d }|j d
}|r|s@|j
d r
t |z }n,|j
d st dz |j d z }|j dt j dt j } t | r| j nd }
|j |||
f |S )u% 从列表页提取 (title, url, date)r xwlistclass_u [WARN] 未找到 div.xwlistr rt u$ [WARN] xwlist 中没有第二个ulrr r rP rQ r2 TrI /rR spandate)findrd rW lenrX get_textrY BASE_URLlstriprE compileIr{ r: ) soupitemsr} ulsr r rP rQ titler date_strs rB extract_list_itemsr [ s4 E
YYuXY
.F +,
//$
C
3x!| 23 QBkk$ . GGCL uuVR
&5 ??3 d?D (c>DKK$44Dwwvbjj"$$&?w@ 4
R@
eT8,-!." LrC c | t g} t dt dz D ] }| j t d| dz | S )u 生成前5页的URLrt rr z /zwgk/zdlyxxgk/hjbhxxgk/index_z.html)LIST_URLrange MAX_PAGESr: r )urlsr s rB get_pagination_urlsr { sE :D
1i!m
$ J H!?s% HHIJ KrC c t | d }|j dd }|s|j dd
}|st d | dg dfS g }|j g d d
D ] }|j
dk( r%|j
d
}|r|j | |j
dk( r |j t || b|j
dk( sr|j dd }|j dd }|s|j d r|j d| d| d |j d s|j d| dt | d dj | } | s
| } g } |j dd
D ] } | d }
t j d |
j s-| j
d
xs |
j! d d }|
j d r
t |
z }
n,|
j d st dz |
j# d z }
| j ||
d |j
}
t% |
}t j d!|
}|r|j' d" }| r t) | j+ d#k r|d}|j d$ }|r|j
d
}|sF|j, r:|j, j. r$|j, j. j+ }d%| d&|xs | d'} | | |fS # t $ r }t d| d| dg dfcY d}~S d}~ww xY w)(uC 抓取详情页：返回 (content_text, attachments_list, pub_date) ri u [WARN] 详情页请求失败: z -> r2 Nr Contentr~
content_nru [WARN] 未找到内容容器: )r r r rU T) recursiver r r rU rV altrR z![]()r rD rP )rQ rQ z"\.(pdf|doc|docx|xls|xlsx|zip|rar)$)r5 rn u, 发布日期[：:]\s*(\d{4}-\d{1,2}-\d{1,2})rr re r
)rp Exceptionrd r rW r5 r r: r_ rX rY r rG rE rw lowersplitr r{ rx r rI r string)rn r e content_divpartstagrm rV r contentrM attachmentsrP ahrefanamepub_daterz _t_hs rB fetch_detailr s R( ))E))4K iili; .se 452rz
E##$@D#Q ? 88s?<>&)LL2cU"SE!34^^C(LL2cU"XJse1!=> ? kk% G K( K
!
!#D
!
1 >&
99:EKKM JJJTJ*Bekk#.>r.BE $ 5(%%f- 3 c):: e < =>
II&H
A9MA771: c'--/*R/
YYt_
4(BdjjTZZ%6%6""((*B R c{(; K ))} .se4s ;<2rzs
L M&M;MMc < dd l } | j t d }|j d t }g }t | D ]g \ }}t
d|dz dt | d| t |d }t | }t
d
t | d |j | i t
d
t | d t } g }
|D ]0 \ } } }
| | vs | j | |
j | | |
f 2 t
dt |
d t } |j dt f D ] }|j |d d}d}d}t |
D ]Y \ }\ } } }| |v r|dz
}t
d|dz dt |
d| d d d t! | \ }}}|r|s|}|r t |j# d k r+d| d| d}|r dj% d |D }|d|z z
}|r|d d n| }d }t' | D ] } |j d | ||| | t |f |j( dkD rE|dz
}t
d | d d d | |r:|j d!t+ j, |d"# | f n|dz
}t
d$ |j/ n t3 j4 d) \ |j9 t
dd* t
d+| d,| d-| t
d* y # t $ r}t
d | Y d }~d }~ww xY w# Y xY w# t $ r]}d%t1 | v rC||dz
k r;t
d&|dz d| d' t3 j4 d( |j7 n|Y d }~}d }~ww xY w).Nr < r zPRAGMA busy_timeout=5000u
列表页 rr r z: re u 找到 u 条u [ERROR] 列表页失败: u
共获取 u 条文章链接u 去重后: z.SELECT page_url FROM gov_raw WHERE site_name=?z
[z] 2 z...r r r r7 c 3 : K | ] }d |d d|d d yw)z- [r5 r rn r N ).0rP s rB zcrawl.. s' (]A3qykAeH:Q)G(]s rD rv z
INSERT OR IGNORE INTO gov_raw(title, content, publish_date, source_url, page_url, site_name, summary)
VALUES (?, ?, ?, ?, ?, ?, ?)
u ✅ 新增: ( z | date=z1UPDATE gov_raw SET attachments=? WHERE page_url=?F) ensure_asciiu ⏭️ 已存在zdatabase is lockedu ⏳ DB锁住, 等待 (z)...rt g ?z2==================================================u 完成! 新增: u
跳过: u
错误: ) sqlite3connectr executer enumeraterd r rp r extendr setaddr: SITE_NAMEr rI rG r
total_changesjsondumpscommitr; timesleeprollbackclose) r conn list_urls all_itemspipurlr r r seen_urls unique_itemsr rn r existingrow new_count
skip_count error_countidxr r r detail_date attach_linkssummaryretriesattempts rB crawlr s ??9b? 1DLL+,#%II i( 3D l2a4&#i.!1D6 :; 3D"-D&t,E )CJ .#
#eS( (? !OJ c#a%#l+,BuSbzl# >?,8,=)k x"H #gmmo.3$SEE7(;G #yy(]Q\(] ] 6L00#*'$3- W~ G
WhS)WMO
%%) NI.s HXJGH" %X[_[e[efq AF \G IL [M N !OJ,.
! 2
3].` JJL"VH
YKz* Z }UV6([ 3 -aS1 2 2 3&
X
'3q61g! 6K2719+QwitLMJJqMMMOG $
s? +AL*0L- BL5 L*L%%L*-L25 N>ANN__main__rr )z--pagesz-pz --max-pages)r2 )re )*__doc__rj rE ossysr r urllib3rF r urllib.parserZ disable_warnings
exceptionsInsecureRequestWarningr r r r rk getenvr r= rM r_ rd rp r{ r r r r __name__ pages_argr argvr iargry maxminr rC rB r sn Z 1 1 1 1 ++BB C N # : :
pO0
BIIk#4 5 * 6 0
@B*J]@ zI
388}q ) FAs66q53sxx=( $'1
$6 #CI As9b)*II G%
s &D&D-&D*-D1