kyj* d Z ddlZddlZddlZddlZddlZddlZddlZddl m
Z
ddl m Z ej j dej j ej j! e ddlmZ dZdZdZd Zd
Zd Zed d
ddZeddZg dZ ej: dej< Z d Z d Z!d Z"d Z#d Z$e%dk( r e$ yy)u
南通市生态环境局 - 公告公示(gggs)栏目爬虫
系统: TRS TrueCMS, 前端 jquery.jpage.js
列表API: GET https://sthjj.nantong.gov.cn/truecms/messageController/getMessage.do
参数(逆向自 jquery.jpage.js getRemoteData): startrecord / endrecord / perpage / columnId
(范围式分页, 非页码式! page/pageNo/offset 均无效)
返回 JSON: {"result": "1820...标题YYYY-MM-DD]]>..."}
注意: JSON 内引号被转义为 \", 需 json.loads 后解析内层 HTML
栏目ID: f53004c4-95fe-476f-8a97-88df5f6e76b3 (gggs.html 内 jpage ajaxParam, totalrecord=1820)
注: 本栏目为「公告公示」聚合, 列表 href 混合指向 tzgg(通知公告)/jsxmgs(建设项目公示)/cgzbgg(采购招标) 的 content 详情页
详情页: https://sthjj.nantong.gov.cn/ntshbj/{栏}/content/{uuid}.html (requests 可直取, 无WAF拦截)
与 tzgg 栏目 URL 相同的记录由 INSERT OR IGNORE 按 URL 去重自动跳过, 不重复入库
标题: 列表即完整标题 (容器 div.TRS_UEDITOR / #BodyLabel)
用法: python3 crawl_nantong_gggs.py [--limit=N] [--pages=N]
N)
BeautifulSoup)urljoin)push_to_searchdbu% 南通市生态环境局-公告公示u 公告公示zDhttps://sthjj.nantong.gov.cn/truecms/messageController/getMessage.doz$f53004c4-95fe-476f-8a97-88df5f6e76b3
zoMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36z.application/json, text/javascript, */*; q=0.01z2https://sthjj.nantong.gov.cn/ntshbj/gggs/gggs.htmlXMLHttpRequest)
User-AgentAcceptRefererzX-Requested-Withz)https://sthjj.nantong.gov.cn/ntshbj/gggs/)r r
)zdiv#zoomz
div.test-1zdiv.TRS_UEDITOR
div#BodyLabelz
div.zwcontentz div.contentz div#contentz:\.(pdf|doc|docx|xls|xlsx|ppt|pptx|zip|rar|wps|et|ofd|txt)$c t j | xs d } t j dd| } t j dd| } t j dd| } | j S )N z [\u200b\u200e\u200f\ufeff\xa0]\s+ u ^[•·\-—]\s*)html_modunescaperesubstrip)ts '/root/gov_crawler/crawl_nantong_gggs.py clean_titler ) sZ !'r"A
0"a8A
vsA A
"B*A 779 c t ||t d}t d D ] } | j t |t
d }|j dk7 r5t d| d| d|j d t j d ft j |j }|j d
d }t j d | }|rt |j! d
nd} g }
t j" d|t j$ D ] } t j d| } | st j d| } | s4| j! d
t' | j! d }}
t) | dk rnd }t j d| }|rd|j+ z }|
j- ||
|d | |
fc S dg fS # t. $ r7}t d| d| d| d| t j d Y d}~d}~ww xY w)u' GET 范围记录: startrecord~endrecord)columnId startrecord endrecordperpage )paramsheaderstimeout z [WARN] list -z status=z, retryresultr
z(\d+)< r z&z"]*>([^<]+)z"]*>([^<]+) z(\d{4})-(\d{2})-(\d{2})z%s-%s-%s)titlehrefdatez err z, retry N) COLUMN_IDPER_PAGErangegetAPI_URLHEADERS status_codeprinttimesleepjsonloadstextr searchintgroupfindallSr lengroupsappend Exception)sessionstartendr attemptrjinnertotal_mtotalitemsrecamr* r) r+ dmes r fetch_list_rangerP 1 s #EX`
aF8 GFGR PA}}#(qXamm_GTU
1
166"AEE(B'Eii 6>G-4C
a()!EEzz"KUTVTXTXY
KYY DcJ#H#NB hhqk;rxx{+Ceu:> YY 93?% 3D udD IJ
K %< 7 >
b5L $UG1SEqc'K L JJqMM s A G
EG
H
,HH
c
D
| j |t d }d|_ |j }|j dk7 st | dk ryt |d }d
}|j d }|r t |j d d
}|s/|j d }|r t |j d d
}|sAt j d|t j } | rt | j! d }|s@|j d }
|
r-t |
j d
j# d d }|syd
} t j d| } | rGd| j! d t% | j! d t% | j! d fz } d}
t& D ]* }|j | }
|
s|
j d
s* n |
|| d
g dS |j d xs |
}g }|j) dd
D ] }|d j+ }t, j | s)d |j/ v sd |v sd |j/ v sT|j1 |j d
xs |j# d! d" t3 || d# g }|
j) g d$ D ] }|j4 d%k( r|j7 d& s"|j d& s|j d% rG|j) dd
D ]L }|d j+ }t, j | sd |j/ v s>t3 || |d < N |j d d
}t j8 d'd | }|s|j1 d(|z |j4 d&k( r~|j) dd
D ]L }|d j+ }t, j | sd |j/ v s>t3 || |d < N |j1 t; | x|j4 d)k( s|j d*d
}|s|j1 d+t3 || d,|j d-d
d. |D ] }|j1 d/|d0 d1|d2 d3 " d4j= | }|s/|
j d d
}|rd(t j8 d'd | z }|| ||dS # t $ r}t d| d| Y d}~yd}~ww xY w)5u) 提取详情: 标题/日期/正文/附件r )r! r" zutf-8z [WARN] detail fetch z: Nr# i z html.parserr
h1r T)r zdiv.container-main-titlezE]*name=["\']ArticleTitle["\'][^>]*content=["\']([^"\']*)["\']r& r) r$ r u- (\d{4})[年\-/. ](\d{1,2})[月\-/. ](\d{1,2})z %s-%02d-%02dr' r r) r+ content attachmentsr a)r* r* downloadz
/attached/z/upload/)nameurl)ptableimgr\ r] r z %s
r^ srcz
z rZ z
) r/ DETAIL_HEADERSencodingr8 rA r3 r2 r> r findr get_text
select_oner r9 Ir; splitr:
CONTAINERSfind_allr ATTACH_RElowerr@ r rZ find_parentr strjoin)rB page_urlrF hrO soupr) rR cmtmr r+ tmbody_elsel att_scoperU rV r* partseltxtr_ attrT s r fetch_detailr} V s KK."K E
FF }}s1v| M *D E
4 B BKK4K 89 oo 89 S =>E II^`acecgcg h
+E IIg
6 < BHHQK@P QQG //#& w''d'3 BOO0;GIK
$
/ Ay D !Z4::<%?(U$**S/RTBU'.x'> @
AA
E 45 m
77c>~~g&"'''*:bggcl[[4[0
8y( ##D)Z4::<-G '$ 7AfI
8 ++c+.C&&c*C [3 ./
WW
[[4[0
8y( ##D)Z4::<-G '$ 7AfI
8
LLR !
WW
&&#C SVAWY[Y_Y_`egiYj kl+m, O
SZVMNOii G s$ / !BFF63$< r4 r5 min enumerater} CATEGORYr6 dumpsr )r max_pagesrB rJ seen
list_itemsrC page_norD r rK it gap_limits2e2t2items2resultsids r mainr s{ !|E9
"Y
./ G 4E
EG
9, h "#GUC85A 7 8 E "B&zT! HHRZ 6FCBuI b ! " BgsSXz[`befpbqErrs 8 a 5 (
1
3+ . 'Cy87K,LI Z 9,
SWZ[eWfhqs|V}}~
IoR"Wi(B)'2r:JB
&f:% F$ :RZ G5 !!"%
& aB JJsO Io *S_<=
+s:.F
FG Z 5( '
G:q) 2 "U) , 9 W+r&zbY[\A" U)e9wZ.2g;& /RZ|OPQ^O_4::a
&6UKeg(
r6Q; "aZ %99 :
3#(
-G
<=W.9
G
,-r __main__)&__doc__osr r r6 r4 htmlr r bs4r urllib.parser pathinsertdirnameabspath__file__ crawler_libr r r r0 r, r- UAr1 rb ri compilerg rk r rP r} r r __name__ r r r s 1 0 0 277??277??8#<= > ( 3
P 2
v'WJ`p r "/Z[
z
BJJTVXVZVZ [ "JSZlM.` zF r