j3, d Z ddlZddlZddlZddlZddlZddlZddlmZmZ ddl m
Z
ddl m Z ej dd ZdZd Zed
z Zed z Z ej& ed
z
j) d ZdZdddddZh dZd Zd dZd Zd Z d Z d Z d Z e d k( r7 e!ejD d kD rejD d d k( r e y e yy)!u
常德市人民政府 - 公示公告
https://www.changde.gov.cn/cdzx/gsgg
WebFuture CMS v15.2.8, 静态分页 /cdzx/gsgg_{n}
列表: ul.newsList > li (span.date + a[title])
详情: h2.title + div.conTxt + 附件
注意: 该站有 WAF CC攻击防御，请求间隔必须 ≥1秒
N)datetime timedelta)
BeautifulSoup)urljoin SEARCH_DBz/root/search.dbu 常德市-公示公告zhttps://www.changde.gov.cnz
/cdzx/gsggz
/cdzx/gsgg_{}iG )daysz%Y-%m-%d
zoMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36zUtext/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8zzh-CN,zh;q=0.9zhttps://www.changde.gov.cn/)z
User-AgentAcceptzAccept-LanguageReferer>) pdddldth1h2h3h4h5h6hrlioltdthtruldivnavpreformmainasidetabletbodytfoottheadcenterdialogfigurefooterheaderhgroupaddressarticlecaptiondetailssectionfieldset
blockquote
figcaptionc | yddl }ddlm g fd | dj }|j dd| }|j dd | }|j S )
ul 块级边界出换行、行内节点直接拼接、
出换行（≈ 浏览器看到的换行结构）。N r )NavigableStringc F | j D ] }t | rj t | *t |dd dk( rj d Lt |dd t
v r+j d | j d | y )Nnamebr
)children
isinstanceappendstrgetattr _BLOCK_TAGS)nodechr7 outwalks "/root/gov_crawler/crawl_changde.pyrE zbody_text..walk2 s --
B"o.
3r7#VT*d2
4 VT*k9
4 R
4 R
z[ \t\r\f\v]*\n[ \t\r\f\v]*r; z\n{3,}z
)rebs4r7 joinsubstrip)el_retr7 rD rE s @@@rF body_textrP * sb z#
C H
A -tQ7A 61%A 779rG c
ddl }t | D ] } |j | t d }|j dk( r6d|dz z }t d| d |dz d
| d t
j | ad |_ |j d
k( r|j c S y# |j $ rA}||dz
k r/d|dz z }t d| d| d t
j | Y d}~d}~ww xY w)z5Fetch URL with rate limiting and retry on WAF blocks.r N )headerstimeouti r u ⚠️ WAF 403, 等待 u
s 后重试 (/)zutf-8 u ⚠️ 请求异常: z, u
s后重试) requestsrangegetHEADERS status_codeprinttimesleepencodingtextRequestException)urlretriesrZ irwaites rF fetchrk E s
7^ ! ! S'2 >A}}#QU|3D6 qse1WIUVWX
4 AJ}}#vv
$! (( !7Q;AE{1!BtfJGH
4 !s A B1!BC('7C##C(c g }t | d }|j dd }|s|S |j d D ] }|j d }|s|j dd j }|s|j d
}|j d d j }|r|j
d rt t | }|j d
d } | r| j d
nd}
|s|s|
s|j |||
f |S )z3Parse list page, return list of (title, url, date). html.parserr newsListclass_r atitler6 TrL href
javascriptspandate)
r findfind_allr\ rL get_text
startswithr BASE_URLr> ) htmlitemssoupr r rq rr rt re span_daterw s rF
parse_listr Z s E } -D
4
+B
kk$ - GGCL gr"((*JJTJ*EuuVR &&(t|4 h%GGF6G2 1:y!!!- ST LL%d+ ,!-$ LrG c 0 t | d }|j dd }|r|j d nd}d}d}|j dd }|r{|j d
D ]g }|j d } t j
d | }
|
r|
j
d }d
| v sCt j dd| j }i g } g } |j dd }
|
r|
j D ] }t |t r%|j } | s&| j | 8|j dv r t | } | sT| j | f|j dk( r1t | } | s| j | | j d |j dk( s| j d |
j dt j dt j D ] }t# t$ |j' dd }|j d xs1 t( j* j- | j/ d d }|sk|sn| j ||d dj1 | j }||||| fS )zParse detail page data.rm r rr ro Trs r6 r propertyrv z!(\d{4}-\d{2}-\d{2}\s*\d{2}:\d{2})rU u 来源u ^来源[：:]?\s*conTxt)r r r1 r# r: rq z \.(doc|docx|pdf|xls|xlsx)(\?|$))rt rt ?r )re r9 r; )r rx rz ry rH searchgrouprK rL r< r= r? r> r9 rP compileIr r| r\ ospathbasenamesplitrJ )r} re r title_elrr rw sourcepropsptxtdm
content_parts attachmentsr rM a_tagatt_urlatt_name content_texts rF parse_detailr v sX } -D yygy.H-5H D )2E D
F 99U:9 .D --' GB++D+)C?EBxx{3 4b#>DDF
G MK
YYuXY
.F
// )B"c"hhj!((- 33 m!((-G# m!((-!((,D $$R( )" __Srzz:\^`^b^b/c_d GE h &"(=>G ~~D~1\RWW5E5Eg5N5T5TUX5YZ[5\H8""7H#EF G 99]+113L ,fk 99rG c
t j t d }|j d |j }d}d}d}| D ] \ }}}} }
} |s
| r| dd n|} |j d|f |j }
|
r2|
d t k7 r |j d t |
d f |dz
}|dz
}m|j d
|||t | |
| d | f |dz
} |j |j |||fS ) zPush items to gov_raw.< rT zPRAGMA busy_timeout=5000r NrX z2SELECT id, site_name FROM gov_raw WHERE page_url=?rU z)UPDATE gov_raw SET site_name=? WHERE id=?INSERT INTO gov_raw (title, page_url, publish_date, site_name, content, source_url, attachments, category, summary) VALUES (?,?,?,?,?,?,?,?,?) 政府公告) sqlite3connectDB_PATHexecutecursorfetchone SITE_NAMEcommitclose)r~ connc new_count
skip_count update_countrr re rw contentr atts_jsonsummaryrows rF
push_to_dbr s ??7B /DLL+,
AIJL8= 4sD'69 #*'$3- FOjjl 1v" E SVWXSYGZ[ ! !OJ
]
Cy'69nV] ^
Q '* KKMJJL j, ..rG c t dt dt g } t dt dz D ] }|dk( rt
nt j | }t d| d| t | }|s!t d t j d bt | }t dt | d |D ]4 \ }}}|t k\ r| j |||f %t d
| d n t j d n t d
t | d g }t | D ] \ } \ }}}| dz dz dk( r t d| dz dt | t | }|rLt || \ }
} } }
}|
s|}
| s|} |rt! j" |d nd}|j |
|| | |
|f t j d |j% d d |D cg c] \ }}}}}}||||||f }}}}}}}t' | \ }}}t dt d t d t | t d | t d | t d | y c c}}}}}}w ) N[u ] 开始爬取，阈值: rU u 列表第u 页: u ⚠️ 获取失败，跳过 u → 条u! ⏹️ 遇到超阈值日期 u ，停止翻页g ?u 共 u 条待处理（近3年）r r u
详情: rV F ensure_ascii[]333333?c | d S )N )xs rF zmain.. s
ad rG T)keyreversez
[u ] 完成u
总处理:
新增: u 跳过(已存在): u 更新站点名: )r_ r THRESHOLDr[ TOTAL_PAGES LIST_URL_TPLLIST_URL_TPL_Nformatrk r` ra r lenr> enumerater jsondumpssortr ) all_itemspagelist_urlr} r~ rr re rw detail_resultsrg d_titler d_dater attsr r r
contsrc
push_itemsr r r s rF r! r! s Ai[1)
=>I aq ) #'19<.2G2G2M
D6xj12X 4 6 JJqM 4
U D)* % E3y %d!349$OP
JJsO
+.
DY :
;< N!*9!5 E3
aC2:? Jqse1S^$45 6Sz 5A$5L 2GWffd @D
4e<$I ! !7C&)"T U
3 " ND9Ucdd5QRb$T2sBc40dJd*4Z*@'Iz< C {(
#$ M#n-.
/0 Jyk
"# !*
./ ~
./ es *I+c
0 t dt d t t } | s t d yt | }t dt | d t
j t d }|j }d }d }|D ] \ }}}|t k r|j d
|f |j r|d z
}:t | } | s!t d |j d
d ht | | \ } }
} } }
| s|} | s|} |
rt j |
d nd}|
r|
dd n| }|j d| || t |
| |d|f |d z
}t# j$ d |j' |j) t d| d| y)z Incremental crawl - only page 1.r u ] 增量爬取u ⚠️ 获取列表失败Nu
列表: r r r r z'SELECT id FROM gov_raw WHERE page_url=?rU u ⚠️ 详情获取失败: rV Fr r rX r r r r u
, 跳过: )r_ r rk r r r r r r r r r r r r r r r` ra r r )r} r~ r r newskiprr re rw r r r r r r r s rF incrementalr s Ai[
'( D
+, t E Js5zl$
'( ??7B /D
A
C D ! sD ) ;cVD ::< AID Sz 1#))C.2D1EF G 1=dC1H.&&$%v<@DJJt%8d #*'$3-
]
c69gvy.Za b
q
336 KKMJJL Jse:dV
,-rG __main__rU z
--incremental)rY )#__doc__r sysrH r` r r r r rI r urllib.parser getenvr r r| r r nowstrftimer r r] rA rP rk r r r r! r __name__r argvr rG rF r s ( ' ' (
"))K!2
3 $ ',& O+
X\\^iU 3
3 = =j I De', * 6*83:j!/F;0z-.^ z
388}qSXXa[O ;
rG