wj F d Z ddlZddlZddlZddlZddlZddlZddlmZmZ ej j dej j ej j e
ddlmZ ej ed z
j# d ZdZd Zd
d d Zd
ZdZeez dz Zd Zd Zd Zd Z d dZ e dk( rW ej> Z e jC de"d e jG Z$ e e$jJ Z& e'de de& d yy) u crawl_qdnhp_hjbh.py - 黄平县-环境保护 (贵州黔东南州黄平县人民政府)
站点: https://www.qdnhp.gov.cn/zfxxgk/fdzdgknr/zdmsxx/hjbh_5621994/
CMS: TRS WCM (UTF-8, 服务端渲染) — 同站 crawl_qdnhp.py 通知公告栏目不同
列表: 标题2026-07-06
链接: /zfxxgk/fdzdgknr/zdmsxx/hjbh_5621994/YYYYMM/tYYYYMMDD_ID.html (title 属性完整无截断)
分页: createPageHTML(67, 0, "index", "html", 1000) → index_N.html 66 页 (P67 404), 每页 15 条
详情: meta ArticleTitle + meta PubDate + div#zoom 正文(内含 TRS_Editor, 平衡 div)
N)datetime timedelta)push_to_searchdbiG )daysz%Y-%m-%du 黄平县-环境保护u 贵州zoMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36zhttps://www.qdnhp.gov.cn/)z
User-AgentRefererzhttps://www.qdnhp.gov.cnz$/zfxxgk/fdzdgknr/zdmsxx/hjbh_5621994/c t j | dt d }d|_ |j S # t
$ r+}t
d| d| t j Y d }~yd }~ww xY w) N) F)timeoutheadersverifyzutf-8z[WARN] fetch failed: z - )file ) requestsgetHEADERSencodingtext Exceptionprintsysstderr)urlres %/root/gov_crawler/crawl_qdnhp_hjbh.pyfetchr sZ LLgwu M
vv
%cU#aS1
Cs /2 A&!A!!A&c g }t j d| t j D ]( }|j d |j d |j d |j d j f\ }}}}d|vr^|xs t j
dd| j }|j
d d j }t j
d
d| }t | dk r|j d r|nt |z }t j d | } | r| j d nd}
|j ||
|d
+ |S )u9 列表条目: tYYYYMMDD_ID.html + title 属性 + b 日期zY]+href="([^"]*t\d{8}_\d+\.html)"[^>]*title="([^"]*)"[^>]*>(.*?)\s*([^<]+) hjbh_5621994<[^>]+>r ^\s*(?:·| |·|\s)+http(\d{4}-\d{2}-\d{2}))r pub_datetitle) refinditerDOTALLgroupstripsubreplacelen
startswithBASEsearchappend) htmlitemsmhref
title_attr
title_html date_textr+ r mdpds r
extract_itemsrA ' s6 E
[[uw{} ~G ~G H
C23''!*aggaj!''RS*VWV]V]^_V`VfVfVh2h/j*i % Arvvj"j AHHJ
h+1138"eD u:> oof-d4$;
YY-y
9 RXXa[B
Sb5AB
C L c t j || t j }|sy| j d|j }|dk( ry|dz }d}t j
d| |d t j D ]j }|j
d j d s |j
d j d r|dz
}I|dz }|dk( sT| |||j z c S | |d S )
u& 平衡 div 匹配 (大小写不敏感)r >r z]|Nr z .
88A; ! !& )RXXa[-C-CF-K QJE QJEzAa"((*n--
. 8OrB c t }|syd}t j d| }|r |j d j }|s[t j d|t j
}|r4t j dd|j d j }|j dd j }t j dd| }d}t j d | }|r8t j d
|j d }|r|j d }d}t |d }|j r|j }|j s,t |d }|j r|j }t j d
d|t j
t j z }t j dd|t j
t j z }ddl
m t j d fd| }t j d fd| }t j dd| }t j dd| }|||j fS )N)r r r r z/\s*([^<]+?)\s*z\s+r& r' z*]*id="zoom"[^>]*>z+]*class="[^"]*TRS_Editor[^"]*"[^>]*>z)flagszr )urljoinz,href="(?!https?://|//|#|javascript:)([^"]+)"c < d | j d dS )Nzhref="r "r/ rM r rQ s r z fetch_detail..s s &bhhqk)B(C1 E rB z src="(?!https?://|//)([^"]+)"c < d | j d dS )Nzsrc="r rS rT rU s r rV z fetch_detail..u s %RXXa[(A'B! D rB z%^(]*>(\s*()?\s*)?)+z%(]*>(\s*()?\s*)?)+$) r r, r6 r/ r0 r. r1 r2 rN rF urllib.parserQ ) r r8 r+ r: r> r? contentcrQ s ` @r fetch_detailr[ O s :D E
DdKA
" II6bii H FF62qwwqz288:E MM(B ' - - /E FF4b% @EI
?FA
YY-qwwqz
:
IG T#>?Awwy'') ==? 'U V 779ggiGff+R BDD@PQGff)2wbii"$$>NOG$ffDEwPGff5DgOG ff=r7KGff=r7KG )W]]_ ,,rB r
c g }t }t | D ] }|dk( rt nt t z d|dz dz }t | }|st
d|dz d nt | }|s6t | dk rt
d|dz d nt
d|dz d nt
d|dz d
t | d |D ]j }|d r
|d t k r|d
|v r |j |d
t |d
\ }} }
|s|d }|
st
d|d
_| xs |d } g } t j d|
t j D ] }
|
j d t j dd|
j d j# }}t j$ d|t j& sd|j) v s| j+ |xs d|d |j+ ||d
|d
| t, |
d| rt/ j0 | d ndt2 d m t
d t | d |D cg c] }|d j# s| }}|r t5 |d t | S c c}w ) Nr z/index_r z.htmlz Page z: fetch fail (stop)i z: 404/empty (stop)z: 0 items (stop): itemsr* r r+ z SKIP empty content: z$]*href="([^"]+)"[^>]*>(.*?)r% r r! z*\.(pdf|doc|docx|xls|xlsx|zip|rar|caj|wps)$downloadu 附件)r+ r F) ensure_asciiz[]) r+ r
source_urlr* site_namerY summary attachments
group_namez Total: rY
qdnhp_hjbh)setrangeLIST_URLr5 COL_DIRr r rA r3 CUTOFFaddr[ r, r- r. r/ r1 r0 r6 rF lowerr7 SITE_NAMEjsondumps
GROUP_NAMEr ) max_pagesrecordsseenpgr r8 r9 itr+ r> rY r@ attsamhtr valids r runr| | s G 5DI & 'htg~'"q&8O'OSz GBF8#67 8 d#4y4 Qx'9:;Qx'789
Qxr#e* V45 B*~"Z.6"9%yD HHRY (4RY(? %E9g7 0E <=,bnBDkk"I7TVT]T]^
Dxx{BFF:r288A;$G$M$M$O199 JArttTXbfgfmfmfoXoKK!-x BC
D
NN r%y5 Y7 VZdjjE.R`d (
' &N
Ic'l^6
*+ 811Y<#5#5#7Q 8E 8 - u:
9s J)J__main__z--pages)typedefaultz [RESULT] r] r^ )r
)(__doc__r osr r, argparsero r r pathinsertdirnameabspath__file__ crawler_libr nowstrftimerk rn rq r r5 rj ri r rA rN r[ r| __name__ArgumentParserap add_argumentint
parse_argsargspagesnr rB r r s - , , ( 277??277??8#<= > (
(,,.9'2
2 < C ((*-Z.b z "BOOICO3
==?D DJJA Ii[1#V
,- rB