WAF: 阿里云 (需浏览器 UA)
运行: python3 crawl_hangzhou.py [--full]
--full: 全量爬取 (所有46页)
不加参数: 仅爬第1页 (增量模式，日跑用)
N)ThreadPoolExecutor as_completed SEARCH_DBz/root/search.dbuA 杭州市生态环境局-2026建设项目环境影响评价公示u 杭州市生态环境局zhttps://epb.hangzhou.gov.cnz7/api-gateway/jpaas-publish-server/front/page/build/unit bulidstatic3249pWtD8p2JcI0v1rw5TsyBqcolumnu 当前栏目listnullS5UyzHsYHfVcPzifFrjDT) parseTypewebIdtplSetIdpageTypetagIdeditTypepageIdz
User-AgentzoMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36 . )g333333?g ? c 2 t dt d| d y )N[z] T)flush)print SITE_NAME)msgs #/root/gov_crawler/crawl_hangzhou.pylogr - s Ai[3%
- c 8 t t }t j | t ddd |d< t
dz dj
d |j D z }t j j |t } t j j |d
}t j |j j }|j! d sg S |d
d S # t" $ r}t% d| d| Y d }~y d }~ww xY w)N )pageNopageSizesearchF) ensure_ascii paramJson?&c 3 | K | ]4 \ }}| d t j j t | 6 yw)=N)urllibparsequotestr).0kvs r
z fetch_list..5 s8 #0411#Qv||!!#a&)*+#s :<headers
timeoutsuccessdatahtmlu 列表API请求失败 (page z): )dict
API_PARAMSjsondumps PAGE_SIZEAPI_URLjoinitemsr* requestRequestHEADERSurlopenloadsreaddecodeget Exceptionr )page_noparamsurlreqrespr8 es r
fetch_listrQ 1 s
*
F **YZ\%]lqrF;
C-#(( #8> # C ..
g
6C~~%%c2%6zz$))+,,./xx "IF|F## *7)3qc :;s A$C5 -C5 5 D>DDc g }t j dt j }|j | D ] }t j d| }|s |j d }t
j |j d j }d|vrdt j d| }|r|j d nd}|j d r|nt |z } |j | ||f |S ) Nz(.*?)z%]*title="([^"]*)" z/art/z>(\d{4}-\d{2}-\d{2})]*ArticleTitle["\s][^>]*content="([^"]*)"rS z2(.*?)
\s*