yj d Z ddlZddlZddlZddlZddlZddlZddlmZ ddl m
Z
ej j dej j ej j e ddlmZ dZdZdZed z Zed
z Zd Zd Zd
ZeddZ ej6 dej8 Z d Z d Z d Z d Z!d Z"d Z#e$dk( r e# yy)uw
抚州国家高新区 - 重点领域/社会公益事业/环境保护 (col8400) 爬虫
列表: https://gxq.jxfz.gov.cn/col/col8350/index.html?number=I00007I00004I00002
(col8350 与 col8400 是同一 XXGK 通用模板页, 栏目内容由 URL number 参数决定)
XXGK 动态模块: 页面 JS loadDynamic POST /module/xxgk/search.jsp
真实请求: POST search.jsp?infotypeId=I00007I00004I00002
body: infotypeId=0&jdid=53&divid=div7299&vc_title=&vc_number=&currpage=N&...&infotypeId=I00007I00004I00002&area=
共 90 条 / 18 条每页 = 5 页 (curl/GET 被 WAF 拒, 完整 POST body 稳定)
列表: 标题2026-08-06
详情: /art/YYYY/M/D/art_8400_NUMBER.html (TRS, http 301 -> https)
标题: meta ArticleTitle; 日期: meta PubDate; 正文: div#zoom
用法: python3 crawl_gxq_jxfz_hb.py [--pages=N]
N)
BeautifulSoup)urljoin)push_to_searchdbzhttps://gxq.jxfz.gov.cnu 抚州高新区-环境保护u 环境保护z1/col/col8350/index.html?number=I00007I00004I00002z/module/xxgk/search.jsp I00007I00004I00002zoMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36zzh-CN,zh;q=0.9)z
User-AgentzAccept-Languagezi]*>\s*]+href='([^']+)'[^>]*title=\"([^\"]*)\"[^>]*>(?:.*?)\s*\s*([^<]+?)\s*\s*c t j dd D ]O } | j d r t | j d d c S | j sDt | c S t S )N z--pages==)sysargv
startswithintsplitisdigit MAX_PAGES)as &/root/gov_crawler/crawl_gxq_jxfz_hb.py
parse_argsr ) sX
XXab\ <<
#qwws|A' ' 99;q6M
c t j dd| xs d } t j dd| } | j S )N\s+ u ^[•·\-—]\s*)resubstrip)ts r clean_titler 2 s7
vsAG$A
"B*A 779r c : t j }|j j t t
|j d< g }t
}t d| dz D ] }dddddt | ddddt dddd} |j t |d }d
|_ |j }d} t j# | D ] }
|
j% d j' t) |
j% d |
j% d j' }
} } | rt+ | dk rl| j- d st/ t0 | } | |v r|j3 | t5 j6 d|
}|r|j% d n|
}
|j9 | | |
d | dz
} t d | d| dt+ | d
| dk( r |S t; j< d |S # t $ r }t d | d | d
Y d}~ |S d}~ww xY w)uP 直接 POST search.jsp (复刻 loadDynamic 完整 body), currpage 翻页, 稳定Refererr 053div7299r )
infotypeIdjdiddividvc_title vc_numbercurrpage
vc_filenumbervc_alltexttypefbtimer$ r' r( area )datatimeoututf-8z Page z: ERR TflushNr http(\d{4}-\d{2}-\d{2}))titleurldatez: +u 累计g ?) requestsSessionheadersupdateHEADERSLIST_URLsetrangestrINFOTYPEpost
SEARCH_URLencodingtext ExceptionprintLI_REfinditergroupr r lenr
r BASE_URLaddr searchappendtimesleep) max_pagessession all_itemsseenpgr0 rhtmlenmhrefr: r< dms r fetch_list_requestsrc 8 s G OO7#!)GOOI I 5DAy1}% tiR 22"R
ZdB ?A AJ66D
% A !
0 0 2K
4KQWWUVZM]M]M_%DCJN??6* x.t| HHTN148B"$288A;$D uT4 H I
FA t3qcY (89F 6
3A B + GB4vaS) 6 & + s +G22 H;HHc | j |d }|j dk7 ryd|_ |j S # t $ r Y yw xY w)N )r1 r r2 )get status_coderI rJ rK )rX r; r\ s r fetchri c sJ KKRK ( ==C
vv
s "8 8 AAc t | d }d}|j dddi }|r |j d rt |d }|s,|j d }|rt |j }d}|j ddd i }|r'|j d r|d j d
d }|s)t
j d | }|r|j d
}d} |j dd }
|
rg } |
j g d D ]} } | j dk( r| j d s"| j d s| j d rGt d | j dd D rk| j dd D ]6 }
|
d j }|j d r(t || |
d< 8 | j dd }t
j d d| }|s| j! d | d | j dk( r | j! t# | %| j d k( s6| j d d }|sL|j d s t || }| j! d!| d" d#j% | } | s|
j dd }|rd | d } g }|
r|
j dd D ] }
|
d j }t
j d$|t j& sd%|j) v sNt || }|
j d xs |j+ d& d' }|j! ||d( ||| |fS ))u 标题/日期/正文z html.parserr metaname ArticleTitle)attrscontenth1PubDateN
r9 r divzoom)id)ptableimgrv rw c 3 K | ]] }t j d |j dd t j xs" d|j dd j v _ yw)*\.(pdf|doc|docx|xls|xlsx|zip|rar|ofd|wps)$ra r downloadN)r rS rg Ilower).0r s r z!extract_detail.. sb II KQUUSY[]M^`b`d`de ?!QUU62%6%<%<%>>?s A#A%r T)ra ra r8 r )r r zz
rx srcz
rz r{ /)rl r; )r findrg r get_textr r rS rO find_allrl find_parentanyr
r r rT rE joinr| r} r )r] r; soupr: mtrp r< mpr` bodyrt partselr htxtr attachmentsfullrl s r extract_detailr n s1 } -D E
6&.!9 :B bffY ByM*
YYt_
.E
D
6&)!4 5B bffY )}""$Sb) II,d 3 771:D
D 99Uv9 &D -- 56 >Bww#~>>'*bggg.>"''#, [[4[ 8
St4 4A& )A <</$+CO& 4 kk#Tk2ffVS#.LL3se4 1G# SW%E!ffUB' >>&1 %c3 /LL=V!<=3 >4 yy --4-0C SE K s . @A& !AyyF244PT^bcbibibkTk sAzzz-A b1A""D#>? @ $k ))r c t } t dt d| d t | }t dt | dd t j }|j j t g }t | D ]A \ }}t ||d }|s3|j t |d |d |d |d d
|d t d
d Kt ||d \ }}} }
|xs |d } |xs |j d d
} | rt! j" d d
| n| }
t! j" dd
|
j% d d }
|j t |d |d | | | |
t |
rt' j( |
d nd
d |dz dz dk( r t d|dz dt | d t+ j, d D t dt | dd t/ |t t1 d |D }|D cg c]
}|d s |d }}t d t | d | d t d |r t3 | nd d!|r t5 | nd d y c c}w )"Nz[GXQ-JXFZ-HB] SITE=z max_pages=Tr3 u
列表完成: u 条, 抓详情...r; r: r< r ) site_name
source_urlr; r: pub_datero summarycategoryr z<[^>]+>r r i, F) ensure_asciir rr r z detail r g333333?z
pushing z items...) batch_labelc 3 h K | ]* }|j d xs dj r'd , yw)ro r r N)rg r )r~ r\ s r r zmain.. s) Ka155+;+Ar*H*H*JKs (22r z DONE. pushed=u 空正文=u 日期范围: -z ~ )r rL SITE_NAMErc rP r= r> r? r@ rA enumerateri rT CATEGORYr rg r r r jsondumpsrU rV r summinmax)rW rY rX resultsiitr] d_titled_dater attsr: r r emptyr\ datess r mainr s_ I {+i[
AN#I.I S^ ,,>
?tL G OO7#G9% 2Wbi( NN &bi5 G"V* g;HR
&4T2e9&E#t&2g;/RVVFB /37"&&S$/U&&g.446t<""U)BuIT HCG4::d?R
ER<1 IacU!C N#34D A
3-0
Js7|nI
.d;W)4 K7K KE$+ =qq}Qz] =E = OCL>UG
s
I+I+__main__)%__doc__osr r rU r r= bs4r urllib.parser pathinsertdirnameabspath__file__ crawler_libr rQ r r rB rH r rF UArA compileSrM r r rc ri r r __name__ r r r s 277??277??8#<= > ( $ * I I
1
1
v0@
A
pDD (VD*N'jT zF r