jY/ d Z ddlZddlZddlZddlZddlZddlZddlmZ ddl m
Z
dZ dZ dZ
dZd Zd
Zej" j% e Z ej* d d ZeZd
dddZdZdZdZe e ej: k rej: e Z e j? d r e e jC dd d Zn:e dk( r5edz e ej: k r e ej: edz Zedz
Zedz
Ze e ej: k reekD reZh dZ"d Z#d Z$ e$ Z%d Z&d Z'd Z(d Z)d Z*d Z+d Z,e-d k( r e, yy)!ug
兴义之窗 — 通知公告 爬虫
site: www.xyzc.cn (黔西南地方综合门户, Discuz! X3 门户系统, GBK编码)
栏目: /news/gonggao (本地资讯-通知公告, 等价 portal.php?mod=list&catid=29)
列表: div.news-list-content > dl > dt > a(article-{id}-1.html) + dd > p.news-txt-time (更新：YYYY-M-D HH:MM)
分页: ?page=N (共307页, 每页~38条; 分页条实际生成 portal.php?mod=list&catid=29&page=N)
详情: http://www.xyzc.cn/article-{id}-1.html (根路径! /news/ 前缀404)
详情结构: div.news-content-tit > h2 标题 + dl>dt 更新日期; 正文 div.news-content-txt (div段落+
)
标题: h2 (strip ' - 通知公告' 后缀); 日期: 更新：2026-7-31 16:34 -> 2026-07-31
正文规则: 保留表格HTML、\n\n分段、附件转名称内嵌URL、去tougao/other噪声
防护: JS cookie 挑战 (GET 拿 cookie 后带 Cookie 访问)
N)
BeautifulSoup)urljoinu 兴义之窗-通知公告u 贵州z www.xyzc.cnzhttp://www.xyzc.cn http://www.xyzc.cn/news/gonggaoi3 SEARCH_DBz/root/search.dbzoMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36z?text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8zzh-CN,zh;q=0.9,en;q=0.8)z
User-AgentAcceptzAccept-Language z--pages==z--pages>) pdddldth1h2h3h4h5h6hrlioltdthtruldivnavpreformmainasidetabletbodytfoottheadcenterdialogfigurefooterheaderhgroupaddressarticlecaptiondetailssectionfieldset
blockquote
figcaptionc | yddl }ddlm g fd | dj }|j dd| }|j dd | }|j S )
ul 块级边界出换行、行内节点直接拼接、
出换行（≈ 浏览器看到的换行结构）。N r )NavigableStringc F | j D ] }t | rj t | *t |dd dk( rj d Lt |dd t
v r+j d | j d | y )Nnamebr
)children
isinstanceappendstrgetattr _BLOCK_TAGS)nodechr6 outwalks '/root/gov_crawler/crawl_xyzc_gonggao.pyrD zbody_text..walkL s --
B"o.
3r7#VT*d2
4 VT*k9
4 R
4 R
z[ \t\r\f\v]*\n[ \t\r\f\v]*r: z\n{3,}
)rebs4r6 joinsubstrip)el_retr6 rC rD s @@@rE body_textrP D sb z#
C H
A -tQ7A 61%A 779rF c t j } | j j t | j t t | S # t $ r Y | S w xY w)uT Discuz 门户有 JS cookie 挑战: 先访问首页拿 cookie, 之后请求带 cookietimeout) requestsSessionheadersupdateHEADERSgetBASE_URLTIMEOUT Exception)ss rE make_sessionr^ _ sW AIIW
h(
H
H
s A A A c t j | t ddi }|j dk7 ry |j }dD ] } |j | c S |j dd S # t $ r Y 6w xY w# t $ r Y y w xY w) NRefererr )rS rV )zutf-8gbkgb18030rb replace)errors)sessionrY r[ status_codecontentdecoder\ )urlresprawencs rE fetchrn m s {{3$-/P#Q S s "ll. C
zz#&
zz% z22
s@ -A8 A8 A)A8 A8 ) A52A8 4A55A8 8 BBc t j dd| xs d } t j dd| } t j dd| } | j S )uM 标题清洗: strip 实体前缀 + ' - 通知公告' 后缀 + 省略号截断u ^[\s\xa0·\u00b7]+r5 u \s*[-–—]\s*通知公告\s*$z
\s*\.{3,}\s*$)rH rK rL )titles rE clean_titlerq sK FF("ekr :E FF5r5 AE FF#R /E ;;=rF c | syt j d| }|rPdt |j d t |j d t |j d fz S t j d| }|rPdt |j d t |j d t |j d fz S y)u* '更新：2026-7-31 16:34' -> '2026-07-31'r5 z(\d{4})-(\d{1,2})-(\d{1,2})z%04d-%02d-%02dr u" (\d{4})年(\d{1,2})月(\d{1,2})日)rH searchintgroup)r] ms rE
parse_datery s
0!4A 3qwwqz?C
OSQR_"UUU
7;A 3qwwqz?C
OSQR_"UUU
rF c N g }t | d }|j dd }|s|S |j d D ] }|j d }|s|j d }|r|j d s<|d j }d |vrT|j d
st
t d z | }t |j }|sd } |j d
}
|
r.|
j dd } | rt | j } |j ||| d |S )uH 解析列表: div.news-list-content > dl > dt > a + dd > p.news-txt-time html.parserr znews-list-contentclass_r
r ahrefzarticle-http/r5 r r z
news-txt-time)rp rj date) r findfind_allrY rL
startswithr rZ rq get_textry r= ) htmlitemssoupboxr
dt_ar~ r rp r r r s rE
parse_listr s E } -D
))E"5)
6C ll4 Bwwt} IIcNf
y T ! v&8c>40DAJJL)
WWT]
O4A!!**,/
uT4@A-B. LrF c t | d }d}|j dd }|r[|j d }|rt |j nd}|j d }|rHt |j }n.|j d }|rt |j nd}|j dt j d }|s||dd dfS g } d }
g } |j d
D ] } | j d k( r | j t | d
}
0| j dv s?| j d D ] }
|
j d xs d}|
j d }t j d|j sO|sR|j d s t || n|}| j ||f t! | }t j" dd| }t j" dd| }t j" dd| }|j% }|s:|j' d D cg c]# }|j% s|j% % }}| j dj) | dj) | }d}| rUt+ }g }| D ]3 \ }}||v r |j- | |j d|d |d 5 dj) | }||||
|fS c c}w ) ub 提取正文: div.news-content-txt (段落div + br); 返回 (content, has_table, attachments_html)r{ r5 r znews-content-titr| r r znews-content-txtr F) recursiver" r )r r r~ r T)rL z,\.(pdf|docx?|xlsx?|zip|rar|wps|et|dps)(\?|$)r z\n+r: z [ \t\u3000]+ z\n\s*\nrG z z
)r r rq r ry rH compiler r8 r= r> rY ru lowerr r rP rK rL splitrJ setadd)r page_urlr r r
r rp r content_divparts has_tableattsrM r~ r r8 abs_hreftxtllinesrh att_htmlseensegshns rE extract_contentr s } -D
D
5!3 4B
WWT].0 BKKM*b
WWT]
bkkm,D
YYt_.0 BKKM*b))E"**5H*I)JK dB2%% EI
D""U"3 -
77g LLR !I
WW
$[[%
2uuV}*zzz-99 Ldjjl[`d>Boof>Uwx 6[_HKK4 01
2 B-C&&s+C&& #s3C&&T3/C))+C,/IIdOIqqwwyII RWWU^,)-, kk% GH u ODAqDy HHQK KK1M N O
99T? $H 44 Js K?K?c
|d }| j d||d f j }|ry| j d||d |d |d |d |d |d
|d |d f | j d
kD rb| j d|f j d
}t j dd|d xs d dd }| j d||d |d |f yy)u0 入库 gov_raw + gov_search; 返回 'new'/'skip'rj z6SELECT 1 FROM gov_raw WHERE page_url=? AND site_name=? site_nameskipzINSERT OR IGNORE INTO gov_raw(page_url, source_url, title, content, publish_date, site_name, group_name, script_name, has_table) VALUES(?,?,?,?,?,?,?,?,?)
source_urlrp rh r
group_name script_namer r z'SELECT id FROM gov_raw WHERE page_url=?z<[^>]+>r5 Nra zRINSERT OR IGNORE INTO gov_search(rowid, title, site_name, summary) VALUES(?,?,?,?)new)executefetchonerowcountrH rK )curitemr existsridsummarys rE store_recordr s E{H
[[@ 4 $%''/xz KK $ 4 %tG}d9o
f tK($|*<
m d; / 12 ||akkCh[QZZ\]^_&&Ri )>B?E `
$w-k!2G < > rF c t j t d } | j }d}d}d}d}t }t dt dz D ]M }t |dkD rd|z ndz }t | } | s!|dz
}|dk\ r|dkD rt d|z n
Ed}t | }
|
st d|z nd} |
D ] } | d |v r |j | d t | d }
|
s|dz
}5t |
| d \ }}}}}|s| d
}|s| d }|s|s|dz
}b|r|r|d z |z j n|}| d |||t t t t" |d
}t% || }|dk( r |dz
}| dz
} n|dz
}t' j( d t d|| |fz P | j+ | j- t dt |||fz y )N< rR r r z?page=%dr5 u" Page %d 请求失败, 提前停止u Page %d 无列表数据rj rp r rG ) rj rp rh r r r r r r r g?u# Page %d: 新增 %d, 累计新增 %du8 完成: 总计 %d | 新增: %d | 跳过: %d | 失败: %d)sqlite3connectDB_PATHcursorr range
_MAX_PAGESLIST_URLrn printr r r rL SITE_NAME
GROUP_NAMEDOMAIN SCRIPT_NAMEr timesleepcommitclose)connr n_newn_skipn_fail fail_streak seen_urlspagerj r r page_newitdetailrp r rh r r r rs rE r r
s' ??7B /D
++-C
E
F
FKIaa ( 3Otax*t+R@Sz 1 Ka D1H:TAB 4 +d2 3 # B%yI% MM"U) $2e9%F! 8GPRSXPY8Z 5E4)X7 &z8! CJ7V+h6==?PX%y " & ( $ * &
D S$'AEz
A
! JJsOG# H 3tXu6MMNg3Oh KKMJJL
D
TY[aciGj
jkrF __main__).__doc__rH sysosr r rT rI r urllib.parser r r r rZ r TOTAL_PAGESpathbasename__file__r getenvr r rX r[ r ilenargvr~ r rv r r@ rP r^ rf rn rq ry r r r r __name__ rF rE r s
'
, gg x( BIIk#4 5
DO0
#chh- A||J a +,
i AECM 1!a% )
161FA
#chh- J* 6
.$
B95x2>lB zF rF