ujT! d Z ddlZddlZddlZddlZddlZddlZddlZ ddl
Z
dZ dZ dZ
dZddd Zd
Zg d Zd d
dd ddddZd Zd Zd Zd Zd Zedk( r e yy)u-
crawl_jmdbq_hjbh.py — 荆门市东宝区人民政府「生态环境」3 个精准子栏目
http://www.jmdbq.gov.cn/col/col2535/index.html (聚合页: 环境影响评价27235 + 生态环保宣传27236 + 环保督察27237)
TRS WCM 系:
- 列表: col2535 聚合页仅各子栏目 5 条; 精准子栏目 col27235/27236/27237 单页全量(24/20/12 条)
- 分页 API: POST /module/web/jpage/dataproxy.jsp (form编码, startrecord/endrecord/perpage + col/webid/columnid/unitid/webname/permissiontype)
返回 XML: 标题日期]]>
- 详情: /art/YYYY/M/D/art_{col}_{id}.html
标题 div.con-title, 日期 span.date(发布日期：2026-06-29 16:09), 正文 ContentStart→ZJEG_RSS.content.end
- 附件: 正文内 a 标签(如有), 内嵌URL保留
Nzhttp://www.jmdbq.gov.cnu 荆门市东宝区生态环境zcrawl_jmdbq_hjbh.pyu 湖北zkMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0 Safari/537.36zzh-CN,zh;q=0.9)z
User-AgentzAccept-Language ))icj u 环境影响评价)idj u 生态环保宣传)iej u 环保督察 & /321696u 东宝区人民政府)colwebidpathsourceContentTypeunitidwebnamepermissiontypec | sy| j } t j dd| t j } | j } t j dd| } | S )u= 标题清洗: strip · 实体前缀与空白 z ·| | | flagsz\s+ )stripresubI)ts %/root/gov_crawler/crawl_jmdbq_hjbh.py clean_titler 4 sN A
.ARTTBA A
vsA A H c | }g fd}t j d||t j t j z }t j dd|t j t j z }t j dd|t j }t j dd |t j }t j d
d| }t j
| }t
d D ]& \ }\ }}}|j d | d
d| d| d }( t j dd| }t j dd| }t j dd| }|j fS )uP HTML -> 文本, 保留 a 链接(内嵌URL)与表格, 去 script/style,
分段c | j d | j d }}t j dd| }t j | j }|r"|j
d s|j
d r|xs dS t j j t | }j |||f dt dS ) Nr <[^>]+>r #
javascript__LINK____)
groupr r html_libunescaper
startswithurllibparseurljoinBASEappendlen)mhreftxtabs_urllinkss r
_link_replz html_to_text.._link_replD s GGAJ
cffZS) $**,dooc*dool.K9" ,,&&tT2
dGS)*#e* R((r z&]*href="([^"]+)"[^>]*>(.*?)r z <(script|style)[^>]*>.*?r z
z <(p|div|li|tr|h[1-6]|br)[^>]*>
r r r" r# z zz\rz\n{3,}z [ \t\u00a0]+r ) r r Sr r% r& enumeratereplacer )html_segsegr3 ir/ r1 r0 r2 s @r html_to_textr< ? sH
C E)
&&:JSUSWSWZ\Z^Z^S^
_C
&&12s"$$+
NC
&&.244
HC
&&2D#RTT
JC
&&R
%C
C
C#,UA#6 ^ D'3kkHQCr*iy@RSVRWW[,\]^
&&C
C
&&FC
(C
&& #s
+C 99; r c
N t t }||d< d|d< d|d< d|d< | j t dz i t t d| d d
d |t
}d
|_ |j }g }t j d|t j D ] }|j d } t j d| t j }
|
s<|
j d } t |
j d xs& t j dd|
j d } t" j$ j' t | }
|
j d j) }|j+ | |
|f |S # t $ r }t d| d| g cY d}~S d}~ww xY w)uP 通过 dataproxy API 获取子栏目全部条目 -> [(title, art_url, pub_date)]columnidr startrecordi' endrecordperpagez /module/web/jpage/dataproxy.jspz/col/colz /index.htmlXMLHttpRequest)RefererzX-Requested-With)headersdatatimeoututf-8z [ERR] colz dataproxy: Nz&zEhref="([^"]+)"[^>]*title="([^"]*)"[^>]*>(.*?)([^<]+)r r r )dict
PROXY_BASEpostr+ HEADERSTIMEOUTencoding Exceptionprinttextr finditerr6 r$ searchr r r( r) r* r r, )sessionr> namerE rexmloutr. r: lmr/ titleart_urlpubs r fetch_columnr_ _ s
D DD DDO LL 4 4;w ;dV8H:[+Q)9;w
(
&&C
C
[[BC
N *ggaj
YY_adfhfjfj
k xx{BHHQK N266*b"((1++NO,,&&tT2hhqk !
E7C() * J
(<s34 s > F$F F$ F$c t j d| t j }|rt |j d nd}|sSt j d| t j }|r,t |j d j d d }t j d| }d}|rGd|j d t
|j d t
|j d
fz }d}t j d | }t j d | }|r@|j }|r|j n
t | } | || j }|xs d
|xs d
|g fS )u4 详情页 -> (title, publish_date, body_html, links)z$\s*(.*?)\s*
r r z(.*?)-r u0 发布日期[：:]\s*(\d{4})-(\d{1,2})-(\d{1,2})z %s-%02d-%02dr rH z zZJEG_RSS\.content\.endN) r rT r6 r r$ splitintendstartr- r )
html_srcr] r. r\ publish body_htmlstart_mend_msrX s
r parse_detailrl s- 98RTTJA'(K
#bE II-x >
0 0 5a 89E
ExPAG AGGAJAGGAJQWWQZ #QQIii7BG II/ :E KKM "EKKMH
QqM'') =D'/T9b 88r c t j } | j dt dd | j ddd | j d t
| j }|j }|j }|j d s t j j | }t j |d
}|j }|j d d}d}d}t! j" } t$ D ] \ }
} t' | |
| } t) d| d|
dt+ | d | D ]V \ }
}} |j d||f |j- d dkD r|dz
}9| j/ |t0 t2 }d|_ t7 |j8 | \ }}}}|j; st) d|
d d |dz
}|r|}
|r|}t= | \ }}|rdj? d |D nd}|r|d d nd}|j d t@ |||
|d|ddd|dd |tB d |v rdnd|f |j d |jD |
t@ |f |jG |dz
}t) d!| d"|
d d# Y tK jL d& |jO t) d'| d(| d)| y # tH $ r$}|dz
}t) d$|
d d d%| Y d }~d }~ww xY w)*Nz--pagesr u4 抓取页数(每子栏目页数, 该站单页全量))typedefaulthelpz--dbz/mnt/data/search.dbu SQLite 数据库路径)ro rp z
--script-name)ro z/mnt/< )rF zPRAGMA busy_timeout=60000r [z colz] API u 条z?SELECT COUNT(*) FROM gov_raw WHERE page_url=? AND script_name=?)rD rF rG u 跳过空正文: r r4 c 3 & K | ] }|d yw)r N ).0as r zmain.. s %>qad%>s r i zINSERT INTO gov_raw (site_name, source_url, page_url, title, publish_date, date_rank, summary, status, category, visits, content, tags, industry, attachments, group_name, has_table, script_name) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)otherz cnameitemsr\ r] pub_datedrd_titled_date content_htmlr2 content links_outatt_txtsummaryrX s r mainr s "BOOIC9oOpOOF$9@XOYOOO[O9
==?DggG""K g &''$$W- ??7B /D
++-CKK+,III G" '% Wh6
%XJfSZL=>(- # 2 $E7H"
2 ]`git_uv<<>!$q( NI [['7[K % 7CBGGW7U4 u#))+ /cr
|<= NI #E %H%1,%?"BK&++%>I%>>QS+2'$3- A %1grSUWXZace gzG8K1QRT_a `]]E9g>
Q XJacr
|45A# 2H
3O'R JJL OI;j :i[
QR
2Q uSbzl"QC011
2s& /K6A$KB9K L K<<L__main__)__doc__rz htmlr% r r r r urllib.parser( r r+ r r} r rM rN r rK r r< r_ rl r __name__rt r r r s
, #
'
&
@ @92?SD zF r