łjv ~ d Z ddlZddlZddlZddlZddlZddlZej j dej j ej j e ddl m
Z
dZdZdddd d
dd d dd
ddddddZed d Zed d ZddZd Zedk( r ej, e yy)u
crawl_gdqy_hpgs.py - 清远市生态环境局-建设项目环境影响评价信息
站点: www.gdqy.gov.cn (瑞数4代 + 帝联NWAF ctct_bundle 双WAF, 用户称"天翼云人机验证")
过盾: playwright + playwright-stealth (ahshx 同款方案)
列表: 静态分页 index_N.html (服务端渲染, 页面内 fetch 200)
容器 ul.infoList > li > h4 > a(标题) + span.time(日期), 472条/20页/24页
详情: content/post_XXX.html (页面内 fetch 200, goto 会超时!)
meta ArticleTitle/PubDate/ContentSource, 正文 div.article-content article-content-body
限速: 详情间 random 3-8s 随机抖动 (用户要求), 单浏览器会话控制请求量
N)push_to_searchdbzhttps://www.gdqy.gov.cnzoMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36z(/xxgk/zzjg/zfjg/qyssthjj/xxgk/zdlyxxgkzlu1 清远市生态环境局-重点领域信息公开)dirnamez5/xxgk/zzjg/zfjg/qyssthjj/xxgk/zdlyxxgkzl/jsxmhjyxpjxxu1 清远市生态环境局-建设项目环评信息z-/xxgk/zzjg/zfjg/qyssthjj/xxgk/zdlyxxgkzl/hjbhu1 清远市生态环境局-环境保护信息公开z//xxgk/zzjg/zfjg/qyssthjj/xxgk/zdlyxxgkzl/ggfwsxu+ 清远市生态环境局-公共服务事项z//xxgk/zzjg/zfjg/qyssthjj/xxgk/zdlyxxgkzl/szhjxxu+ 清远市生态环境局-水质环境信息)zdly jsxmhjyxpjxxhjbhggfwsxszhjxxr r r c f
K t | }|d }|d }ddlm} ddlm} | 4 d { }|j
j
ddg d d { }|j t d
d d d
d d { } | }
|
j | d { | j d { } | j t |z dz dd d { } | j d d { | j d { }
t j d|
}t# d| r | j$ nd d|r|j' d nd dt) |
|rHd|j' d v r5t# d |j+ d { d d d d { yg }t1 }t3 d|dz D ]# }|dk( r|
}n' | j5 d t | d | d! d { }t j6 d%| }|st j6 d&| }d}|D ]w \ }}}t j8 d'd(| j; }|j= d) r|nt |z }||vsM|j? | |jA |||d* |dz
}y t# d+| d,t) | d-| d.t) | d/ t) | d0k st# d1 n g }tC | D ]V \ } } t# d2| dz dt) | d#| d3 d d4 d5 | j5 d6| d7 d8 d { } | d9 } | d: d;k7 sd| d d< v rt# d=| d: u | d3 }t j d?| }|r>|j' d j; r |j' d j; }| d@ } t j dA| }|r7t j dB|j' d }!|!r|!j' } d(}"t j dC| }|r |j' d j; }"|"s7t j dD| }|r |j' d j; }"|"j; s
t# dE t j8 dFdGt z dz |" }"t j8 dHdIt z dz |" }"|jA || d7 | ||"d(dJ | t) | dz
k s$tE jF dKdL }#tI jJ |# d { Y |j+ d { d d d d { t# dMt) dN |r tM |dO t) | S 7 7 \7 ;7 7 7 7 7 7 17 ## t, $ rN}t# d t/ | d d |j+ d { 7 Y d }~d d d d { 7 yd }~ww xY w7 1# t, $ r)}t# d"| d#t/ | d d$ Y d }~ hd }~ww xY w7 # t, $ r%}t# d>t/ | d d$ Y d }~d }~ww xY w7 E7 -7 # 1 d { 7 sw Y 1xY ww)PNr r r )async_playwright)StealthTz>/root/.cache/ms-playwright/chromium-1228/chrome-linux64/chrome)z --no-sandboxz--disable-setuid-sandboxz---disable-blink-features=AutomationControlled)headlessexecutable_pathargszzh-CNi i )widthheightz
Asia/Shanghai)
user_agentlocaleviewport timezone_id/domcontentloadediȯ )
wait_untiltimeouti: z([^<]*)z [LOAD] status=?z title= z len=u 请稍候u [WAF] 挑战未过z
[LOAD ERR] x z=async () => {
const r = await fetch('z/index_z.html', {headers: {'X-Requested-With': 'XMLHttpRequest'}});
return await r.text();
}z
[LIST ERR pz] d zq\s*]*>([\s\S]{0,200}?)(\d{4}-\d{2}-\d{2})z^]*>([\s\S]{0,200}?)]*>(\d{4}-\d{2}-\d{2})z<[^>]+> http)urltitledateu 第u 页: 解析 u
条, 新增 u (累计 ) u [END] 列表结束z [r" & z...z9async () => {
const r = await fetch('r! z', {headers: {'X-Requested-With': 'XMLHttpRequest'}});
return {status: r.status, html: await r.text()};
}htmlstatus i z [FETCH WARN] status=z [DETAIL ERR] z/]*class="article-content[^"]*"[^>]*>([\s\S]*?)\s*(?:]*class="article-share"|\s*|