zj) d Z ddlZddlZddlZddlZddlZddlZddlZddl Z ddl
m Z m Z ej j dej j ej j! e ddlmZ dZdZeez dz ZdZd Zd
Zd Zd Zd
Z ddiZ d Z d Z d Z ddZ!d Z"d Z#d Z$d Z%d Z&d Z'e(dk( r e' yy) um
湖北省发展和改革委员会 - 节能审查 爬虫
栏目: https://fgw.hubei.gov.cn/fbjd/xxgkml/zdjsxm/pzjgxx/sp/jnscyj/index.shtml
内容: 省发改委节能审查意见 (重大项目节能审查批复)
CMS: TRS WCM + 瑞数4代 JS challenge WAF (412)
WAF 方案: playwright chromium headless 渲染列表页拿瑞数 cookie → requests 带 cookie 直抓
⚠️ 服务器 DNS SERVFAIL → chromium --host-resolver-rules=MAP fgw.hubei.gov.cn 223.75.52.243
⚠️ cookie 可能过期 (412) → 自动重新 playwright 刷 cookie 重试
列表: TRS 静态分页 createPageHTML(39,...) → index_N.shtml (N=0起, index_1=第2页)
标题YYYY-MM-DD
⚠️ 显示文本截断带省略号 → 用 title 属性 (完整)
⚠️ href 相对路径须 urljoin(列表页URL) 保留完整路径段 (fbjd/xxgkml/xkfw/...)
详情: https://fgw.hubei.gov.cn/fbjd/xxgkml/xkfw/xzxkjg/xmspqk/YYYYMM/tYYYYMMDD_ID.shtml
meta ArticleTitle/PubDate; 正文 id="zwnr" class="zwblock" > div.view.TRS_UEDITOR (>p 段落)
附件: //fgw.hubei.gov.cn/pdfconvert/download/{ID}.zip?fileName=标题&template=zcwj (urljoin 绝对化)
CUTOFF 3年: 2023-08-11 (page 10-11 边界) → --pages=11 约200条
用法:
python3 crawl_hbfgw_jnscyj.py --pages=1
python3 crawl_hbfgw_jnscyj.py --pages=11
N)urljoinquote)push_to_searchdbzhttps://fgw.hubei.gov.cnz$/fbjd/xxgkml/zdjsxm/pzjgxx/sp/jnscyjz /index.shtmlu 湖北省发改委-节能审查u 节能审查z
2023-08-11z
223.75.52.243zoMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 str c d} t j dd D ] }|j d r t |j d d } 2|j d sDt t j t j j
| dz kD st t j t j j
| dz } | S )Nr z--pages==z--pages)sysargv
startswithintsplitlenindex)pagesas '/root/gov_crawler/crawl_hbfgw_jnscyj.py
parse_argsr - s
E
XXab\ 9 <<
# Q(E
\\)
$SXX 9JQ9N)N !2Q!678E 9
L c t j | xs d } t j dd| } t j dd| } t j dd| } | j S )Nr z [\u200b\u200e\u200f\ufeff\xa0]z\s+ u ^[•·\-—]\s*)html_modunescaperesubstrip)ts r clean_titler 7 sZ !'r"A
0"a8A
vsA A
"B*A 779r c ddl m} | 5 }|j j dddt g }|j t d }|j }|j t d d
|j d |j }|j d
d
d
dj D cg c] }|d d|d c} t d< t dt! | d y
# 1 sw Y SxY wc c}w )ub playwright 渲染列表页拿瑞数 cookie (服务器 DNS SERVFAIL 须 host-resolver-rules 映射)r )sync_playwrightTz---disable-blink-features=AutomationControlledz+--host-resolver-rules=MAP fgw.hubei.gov.cn )headlessargszzh-CN)
user_agentlocaleloadi` )
wait_untiltimeouti Nz; namer valuer u [cookie] 刷新成功 u 个 cookie)playwright.sync_apir" chromiumlaunchIPV4 new_contextUAnew_pagegotoLIST_URLwait_for_timeoutcookiesclosejoin
RUI_COOKIEprintr )r" pbrowserctxpager6 cs r refresh_ruishu_cookier@ ? s 3
a**##?=dVD $
!!R!@||~ (vu = d#++-
"P1ai[!G* #>"PQJu $S\N+
>?
#Qs BC7<D7D c t d D ]" }t d s
t t t d |xs t dd} t j | |t }|j dk( rmd|j dd v sd
|j dd v r6|r2|d k r-t d |d
z dt j t yd|_
|j c S |j dk( r5|r3|d k r.t d|d
z dt j t t j d % y# t $ r6}|d k( r't d| d| t j Y d}~ yY d}~Sd}~ww xY w)u@ requests 带瑞数 cookie 抓取; 412 时自动刷 cookie 重试 r zzh-CN,zh;q=0.9)z
User-AgentCookieRefererzAccept-Language)headersr) z_$ssNi zEnvironment Checking u% [WARN] WAF 页, 刷 cookie 重试 (r
))filezutf-8i u! [WARN] 412, 刷 cookie 重试 (z [WARN] GET failed z: g ?)ranger9 r@ r1 r4 requestsgetTIMEOUT status_codetextr: r stderrencoding Exceptiontimesleep)urlreferer retry_wafattemptrE res r http_getr[ T sU 8 % ! # ' *( /
S'7CA}}#QVVET]*.DuPT
.U Wq[ Egai[PQ RY\YcYcd-/ $
vv
}}# gk9'!)AFSZZX%'
39 : !|,SEA37cjjI s% B D3D3AD33 E2<&E--E2c l | j d r| S | j d rd| z S t || S )N)zhttp://zhttps://z javascript:zmailto:ztel:zdata:#z//zhttps:)r r )upage_urls r _abs_urlr` v s7 ||[\||D!| 8Q r c | syt j dd| } t j dd| } t j dd| } t j dfd| } t j dfd| } t j d d
| } t j d d| } t j d d
| } t j dd | } t j dd| t j } t j dd| } t j dd| } t j dd| } | j S )uX 正文清洗: 保留 段落 HTML, 剥 span/font 内联样式, 附件/图片绝对化r zzzzz clean_content.. s# )HQWWUVZYaDbCccd8e r zc , d| j d dS )Nz. s AGGTUJ]*>)flagsz [\u3000\xa0]r z\s*(?: )?\s*
z\r\n
)r r Ir )raw_htmlr_ s `r
clean_contentru ~ s vv12x@Hvv/X>Hvv12x@Hvv,.egopHvv-/giqrHvvk5(3Hvv&H5HvvAC^`hiHvv;=Y[cdHvvBBXZX\X\]HvvosH5Hvv12x@HvvgtX.H >> r c g }t }t j d| t j D ] }|j d t j
dd|j d j
}}t j d|t j s%d|j v sd|j v st || }||vs|j | t j d | }|} | s |r t |j d d
d } |j | xs |j d d
|d |rt j |d S dS )u5 从正文提取附件链接 (pdfconvert/下载链接)z"]*>(.*?)r
z<[^>]+>r z2\.(pdf|doc|docx|xls|xlsx|zip|rar|wps|7z|txt)(\?|$)download
pdfconvert[?&]fileName=([^&]+)N< /r* rU F ensure_ascii[])setr finditerSre r r searchrs lowerr` addurllib_parse_unquoteappendr jsondumps)
contentr_ attsseenamahrefatextr^ fmr* s
r extract_attachmentsr sC
D 5Dkk?"$$O Jxx{BFF:r288A;$G$M$M$Ou
99JESUSWSW X\fjojujujw\w |H LQ LW LW LY |Y)A} YY 6: / t | }|sdg fS g }d}t j d| }|st j d| }|rt |j d | }t j
d|t j D ] }|j d }t || t j d }|r t |j d dd n j d d
}t fd |D r~|j | d ||fS )
u$ 抓详情页: (正文, 附件列表)r u ]*id="zwnr"[^>]*>([\s\S]*?)\s*(?:]*id="fjtp"|]*class="[^"]*appendix|]*class="[^"]*article_qrcode|