uj 9 4 d Z ddlZddlZddlZddlZddlZddlZddl Z ddl
m Z dZ
dZe
ez Ze
dz ZdZdZd Zd
Zd d d
edZdZ eej0 D ]k \ ZZej7 d r e ej; dd d Z4edk( s:edz e ej0 k sS e ej0 edz Zm edk\ rendZ dZ!d Z"d dZ#d Z$d Z%d Z&d Z'd Z(d Z)e*d k( r e) yy# e $ r dZ Y w xY w# e $ r Y w xY w# e $ r Y w xY w) u
东乡区人民政府 - 环境保护 (www.jxfzdx.gov.cn/col/col21311/index.html)
CMS: 汉网 hanweb / TRS 系 (jpage datastore + xxgk/search.jsp 动态列表)
列表: JS 渲染 loadDynamic('/module/xxgk/search.jsp', 'div7299', ...) — 必须 POST
POST /module/xxgk/search.jsp
query: texttype=&fbtime=&vc_all=&vc_filenumber=&vc_title=&vc_number=&currpage=N&sortfield=&fields=&fieldConfigId=&hasNoPages=&infoCount=
body: infotypeId=L00004L00004L00010&jdid=59&area=&divid=div7299&vc_title=&vc_number=&currpage=N&vc_filenumber=&vc_all=&texttype=&fbtime=
18条/页, 共536条/30页. 返回 标题YYYY-MM-DD
(GET 返回 18 空行 + "请传入正确参数" — 必须 POST)
详情: meta ArticleTitle/PubDate(YYYY-MM-DD HH:MM:SS 取前10)/ColumnName=环境保护
正文 div#zoom (Word 转换 p>span 逐段 + table 保留 + ContentStart 标记)
无独立附件区 (环评公示纯文本, 链接全是导航)
N)
BeautifulSoupzhttps://www.jxfzdx.gov.cnz2/col/col21311/index.html?number=L00004L00004L00010z/module/xxgk/search.jspu 东乡区-环境保护u 江西zcrawl_jxfzdx_hjbh.pyz/mnt/data/search.dbzkMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0 Safari/537.36z?text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8zzh-CN,zh;q=0.9)z
User-AgentAcceptzAccept-LanguageReferer z--pages==z--pagesL00004L00004L00010c | syt j | } | j dd j dd j dd j dd } t j dd| } | j S )z;unescape entities + strip zero-width + collapse whitespace. u u u u \s+ )html_libunescapereplaceresubstrip)ts &/root/gov_crawler/crawl_jxfzdx_hjbh.py clean_titler B so ! A (B ''"5==hKSST\^`aA
vsA A 779 c J dd l }d }t | D ]d } |xs |j | t | }|j dk( r|j
xs d|_ |c S |}t j d|dz z f t |t r||S # t $ r }|}Y d }~Ad }~ww xY w)Nr )headerstimeout utf-8g ?r ) requestsrangegetHEADERS status_codeapparent_encodingencoding Exceptiontimesleep
isinstance) urlsessionr retriesr lastires r fetchr. L s D
7^ " $H))#w)PA}}# 0 0 ;G
D
3!a%=! " $ "
K D s AB
B
B"B B"c d| d}dt d| d}t t }d|d< | j t dz |z ||d
}|j
xs d |_ |j }t j d | }|rt |j d
nd}t j d| }|rt |j d
nd} ||| fS )u" POST search.jsp for a given page. Returns (html, total_count, total_pages).
⚠️ requests 传字符串 body 不自动设 Content-Type — 必须显式
application/x-www-form-urlencoded，否则服务器返回「请传入正确参数！」(curl -d 自动带所以 curl 可用)
zGtexttype=&fbtime=&vc_all=&vc_filenumber=&vc_title=&vc_number=&currpage=z9&sortfield=&fields=&fieldConfigId=&hasNoPages=&infoCount=z infotypeId=z;&jdid=59&area=&divid=div7299&vc_title=&vc_number=&currpage=z)&vc_filenumber=&vc_all=&texttype=&fbtime=z!application/x-www-form-urlencodedz Content-Type? )datar r r u 共(\d+)条记录r r u 共\s*(\d+)\s*页) INFOTYPE_IDdictr post
SEARCH_APIr! r" textr searchintgroup)
r( pageqbodyhr, r7 mtotal total_pagess
r
fetch_listrB ^ s
VT
VA+ ' G
ID W
A;An Z#%)a LA$$/AJ 66D
&-A C
OaE
&-A%&#aggaj/AK ##r c
| sddg fS t j dd| t j } t j dd| t j t j z } t j dd| t j t j z } t rQ t | d }|j d D ]% }d D ] }|j j |d
' t | } | }t j d d|t j }g i fd }t j d
||t j t j z }g fd}t j d||t j t j z }t j d| t j rdnd}t j dd|t j }t j dd|t j }t j dd|t j }g } |j d D ] }
|
j } | st j dd| } t j | } | j dd j dd } t j d d| } | s~j! D ] \ }
}| j |
| } t# D ] \ }}| j d | d | } | j% | j dj' | }j! D ] \ }
}|j |
| } t# D ] \ }}|j d | d | } t j d d| }t j d d| }t j | }t j d!d| }t j d"d| }t j d#d| }t) }g }|j d D ]X }t j d$dt j dd| }|s2||v r7|j+ | |j% | Z dj' | j }||fS # t $ r Y w xY w)%zConvert article-body inner HTML to stored format:
- keep z ]*>r zz
z
z(]*>z
rS r z z [\s\u200b\u200c\u200d\ufeff]+ri rU z
__LINK__\d+__z __TBL__\d+__z[ \t]+\nz\n{3,}z [ \t]{2,}r )r r SIr find_allattrspopstrr# r8 rY r r
r r items enumerater] joinsetadd)contentrf _souptagattrcontent2rg rk has_tablepartsblockbplainkvr+ tbloutseenfinalprc rd re rj s ` @@@r html_to_textr s s1 1byff]BrttAIvvgz8244@HvvlD("$$?HvvA2xWYW[W[\H E ' KKM z2q)!!%(
fc*228SA7UC &&( DAq !QA
. 0FAs GA3b/3/A 0
QWWY ++e
C ""$ 1kk!Q M* 03kkGA3b/3/0
&&!2s
+C
&& "c
*C
C
C
&&dC
(C
&&FC
(C
&& sC
(C 5D E
YYv
ffVR
B!:; $;
Q ++e
"
"
$C ; &&g s *AS S S c g }t r"t | d }|j d D ] }|j dd }|s|j dd j }t j d| sQt j j t | }t |j d d xs |j d
d }d}|j d } | r t | j d
d }t j d
| sd}|j |||f |st j d| t
j D ]~ }
t j j t |
j# d }|j |t |
j# d |
j# d j f |S )u Extract (url, title, date) from search.jsp POST response.
标题2026-07-30
rE liaT)r_ r_ r
z'/art/\d{4}/\d+/\d+/art_21311_\d+\.html$titler )r r z^20\d{2}-\d{2}-\d{2}$zo]+href="(/art/\d{4}/\d+/\d+/art_21311_\d+\.html)"[^>]*title="([^"]*)"[^>]*>(.*?)\s*]*>([^<]*)r rR )r rp findr r r r8 rZ r[ r\ BASE_URLr get_textmatchr] finditerrn r: ) html_textrt soupr r r_ rb r dater r? s r
parse_listr s
E Y
6--% 1B$'A55$**,D99 GN ll**8T:G gr 2 QajjDj6QRED A"1::c:#>?88 4d; LL'5$/ 0 1 ~ rtt
QA ll**8QWWQZ@G LL';qwwqz#:AGGAJ]*>(.*?))
r r r compilero r r r r r: rs r8 rn ) r rf r r content_htmlr mtmdddmr r? s r parse_detailr s E
DL Y
6
YYvfbjj "$$.O%PY
Q
"&&# 9
.E
YYvfbjjRTT.J%KY
L
"&	
##%A4a8Bxx{yy2::grtt# d#d
y # t $ r }
t d| d|
d
Y d }
~
xd }
~
ww xY w# t $ r#}
t d| d|
d
|dz
}Y d }
~
d }
~
ww xY w)$Nr < )r zPRAGMA busy_timeout=60000zPRAGMA journal_mode=WALr zPage z fetch error: T)flushz: found z items (total=z, pages=)zEmpty page, stop pagination)r( r r )errorsz detail error z: rS r
z skip empty: r1 z + ( z (g333333?g ?u 新增: u 跳过: z=== z done ===) r Sessionr updater r connectDB_PATHr rw r _PAGESrB r# printr r^ rx r. ry decoder r r r r r r$ r% commitcloser )r sr new_count
skip_count seen_urlsr; r r@ rA r- rt rb r r r, dhtmldtitleddater ry r~ _ plain_lenoks r mainr D s AIIW ??7B /DLL,-LL*+IJIa!$ ( ,6q$,? )Iuk 9%
dV8CJ<~eWH[MYZ[cgh /t < $) GUD)# MM' "
'1- (((C
+7ug*F 'FE< $0 w$G !GYBFF:r7 ; A A CDI2~vcr{m4DAa
dGVWeYOBQ VCR[ME7!4DAa
JJsO9 :
3Q(T KKMJJL HYK
- HZL
!. D 9
%T2W E$~aS1 >
y1#6dCa
s0 I%
*J% J .JJ J;J66J;__main__)Nr1 r )+__doc__sysrV r r$ htmlr
urllib.parserZ r bs4r ImportErrorr LIST_PATHLIST_URLr6 r r r r r
_MAX_PAGESru argvr+ r
startswithr9 rY
ValueErrorr^ r r3 r r. rB r r r r r __name__ r r r s !
' @ i
1
1
$
$
@O'
chh
DAq||J QWWS!_Q /0J
i AECM 1 SXXa!e_-J
"QA" $$*g'T D %@"J:3z zF O M4
s5 C7 /D+D7D DD D DD