uj 6 d Z ddlZddlZddlZddlZddlZddlZddl Z ddl
m Z dZ
dZe
ez ZdZdZdZd Zd
d d d
ZdZ eej. D ]k \ ZZej5 d r eej9 dd d Z4edk( s:edz e ej. k sS eej. edz Zm edk\ rendZ d Z ddZ!d Z"d Z#d Z$d Z%d Z&e'dk( r e& yy# e $ r dZ Y w xY w# e $ r Y w xY w# e $ r Y w xY w)u-
库伦旗人民政府 - 通知公告 (www.kulun.gov.cn/xwzx/tzgg/)
CMS: TRS (div.lis_list_part2_list > ul > li > a[href=./YYYYMM/tID.html] + span 列表,
currentPage 0-based 分页 index_{N}.html: 首页 index.html, 第N+1页=index_{N}.html, 15条/页 87页)
列表: ul > li > a[href=./202607/t20260731_1063411.html] + span 日期 (列表标题可能截断带…, 详情meta覆盖)
详情: meta ArticleTitle/PubDate(YYYY-MM-DD HH:MM:SS)/ContentSource 齐全
正文 div.trs_editor_view.TRS_UEDITOR (Word转换 p>span 逐段 + table 保留 + 头部样式噪声)
附件正文内嵌 (urljoin(详情URL) 绝对化), icon_xls.gif 图标清理
注意: trs_editor_view 容器在源码中出现 2 次(内容相同) — find 取第一个即可
N)
BeautifulSoupzhttp://www.kulun.gov.cnz /xwzx/tzgg/u" 库伦旗人民政府-通知公告u 内蒙古zcrawl_kulun_tzgg.pyz/mnt/data/search.dbzkMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0 Safari/537.36z?text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8zzh-CN,zh;q=0.9)z
User-AgentAcceptzAccept-Language z--pages==z--pagesc | syt j | } | j dd j dd j dd j dd } t j dd| } | j S )z;unescape entities + strip zero-width + collapse whitespace. u u u u \s+ )html_libunescapereplaceresubstrip)ts %/root/gov_crawler/crawl_kulun_tzgg.py clean_titler ; so ! A (B ''"5==hKSST\^`aA
vsA A 779 c J dd l }d }t | D ]d } |xs |j | t | }|j dk( r|j
xs d|_ |c S |}t j d|dz z f t |t r||S # t $ r }|}Y d }~Ad }~ww xY w)Nr )headerstimeout utf-8g ?r ) requestsrangegetHEADERS status_codeapparent_encodingencoding Exceptiontimesleep
isinstance) urlsessionr retriesr lastires r fetchr, E s D
7^ " $H))#w)PA}}# 0 0 ;G
D
3!a%=! " $ "
K D s AB
B
B"B B"c
| sddg fS t j dd| t j } t j dd| t j t j z } t j dd| t j t j z } t rQ t | d }|j d D ]% }d D ] }|j j |d
' t | } | }t j d d|t j }g i fd }t j d
||t j t j z }g fd}t j d||t j t j z }t j d| t j rdnd}t j dd|t j }t j dd|t j }t j dd|t j }g } |j d D ] }
|
j } | st j dd| } t j | } | j dd j dd } t j d d| } | s~j! D ] \ }
}| j |
| } t# D ] \ }}| j d | d | } | j% | j dj' | }j! D ] \ }
}|j |
| } t# D ] \ }}|j d | d | } t j d d| }t j d d| }t j | }t j d!d| }t j d"d| }t j d#d| }t) }g }|j d D ]X }t j d$dt j dd| }|s2||v r7|j+ | |j% | Z dj' | j }||fS # t $ r Y w xY w)%zConvert article-body inner HTML to stored format:
- keep z ]*>r zz
z
z(]*>z
r= r
z z [\s\u200b\u200c\u200d\ufeff]+rV r@ z
__LINK__\d+__z __TBL__\d+__z[ \t]+\nz\n{3,}z [ \t]{2,}r )r r SIr find_allattrspopstrr! searchrE r r r r
items enumeraterI joinsetadd)contentrS _souptagattrcontent2rT rX has_tablepartsblockbplainkvr) tbloutseenfinalprP rQ rR rW s ` @@@r html_to_textrx W s1 1byff]BrttAIvvgz8244@HvvlD("$$?HvvA2xWYW[W[\H E ' KKM z2q)!!%(
fc*228SA7UC &&( DAq !QA
. 0FAs GA3b/3/A 0
QWWY ++e
C ""$ 1kk!Q M* 03kkGA3b/3/0
&&!2s
+C
&& "c
*C
C
C
&&dC
(C
&&FC
(C
&& sC
(C 5D E
YYv
ffVR
B!:; $;
Q ++e
"
"
$C ; &&i s *AS S S c g }t rt | d }|j d D ] }|j d D ] }|j dd }|s|j dd j }t j d | sPt j j t | }t |j d
d } |j d }
|
r|
j d nd} t j d
| sd} |j || | f |st j d| t
j D ]~ } t j j t | j# d }|j |t | j# d | j# d j f |S )z*Extract (url, title, date) from list page.r/ ulliaT)rL rL r z/20\d{4}/t\d+_\d+\.html$r
r spanz^20\d{2}-\d{2}-\d{2}$z_\s*]+href="([^"]*?/20\d{4}/t\d+_\d+\.html)"[^>]*>(.*?)\s*]*>([^<]*)r r< )r r] findr r r ra rF rG rH LIST_URLr get_textmatchrI finditerr[ rA )
html_textrS rb souprz r{ r| rL rO titler~ daterK s
r
parse_listr s E Y
6--% 5Bkk$'
5GGCdG+ uuVR(..0yy!#AJJs$J$?@wwv48t}}4}0bxx 8$? D gud 34
5 5 n rtt
QA ll**8QWWQZ@G LL';qwwqz#:AGGAJ]*>(.*?)\s*\s*)r r r compiler\ r r r r r rA r` ra r[ )r rS r r content_htmlr mtr mdddmviewtextrK s r parse_detailr s E
DL Y
6
YYvfbjj "$$.O%PY
Q
"&&# 9
.E
YYt_ Ct <=E
YYvfbjjRTT.J%KY
L
"&	
##%A4a8Bxx{yyrzz2D'EyF t9L 99U2::k+B9CD "4y IINPY[][_[_ `
+E IITV_acaeae f 771:D IIaclnpnrnr s 771:L $ $$r c
| j d|f }|j ry d} ddlm} || }| j d|||||t
t t ||f
}|j dkD ryt j dd| } t j | } t j d d
| j d d } | j d |j |t
| f y
y# t $ r Y w xY w# t j $ r=}
dt# |
v r*t% j& d t) | ||||| cY d }
~
S d }
~
ww xY w)Nz'SELECT id FROM gov_raw WHERE page_url=?Fotherr )classify_industryzINSERT OR IGNORE INTO gov_raw (page_url, source_url, title, content, publish_date, site_name, group_name, script_name, has_table, industry) VALUES (?,?,?,?,?,?,?,?,?,?)r= r r r
r zTINSERT OR IGNORE INTO gov_search (rowid, title, site_name, summary) VALUES (?,?,?,?)Tlockedr )executefetchone crawler_libr r! SITE_NAME
GROUP_NAME SCRIPT_NAMErowcountr r r r r lastrowidsqlite3OperationalErrorr` r" r# store_record) connr% r rg r rl curindustryr summaryr+ s r r r
sF
,,1 C ||~ 5(/H ll
+ #ugtY
KQZ\d e
< d$d y # t $ r }
t# d
| d|
d Y d }
~
xd }
~
ww xY w# t $ r#}
t# d| d|
d |dz
}Y d }
~
d }
~
ww xY w)%Nr < )r zPRAGMA busy_timeout=60000zPRAGMA journal_mode=WALr index_z.html)r& r r
)errorszPage z fetch error: T)flushz: found z itemszEmpty page, stop paginationz detail error z: r= r
z skip empty: z + ( z ()g333333?g ?u 新增: u 跳过: z=== z done ===) r Sessionr updater r connectDB_PATHr re r _PAGESr r, rg decoder! printr rJ rf r rx r r r r r" r# commitcloser )r sr new_count
skip_count seen_urlspager% r* r r+ rb rO r r dhtmldtitleddater rg rl _ plain_lenoks r mainr 2 s AIIW ??7B /DLL,-LL*+IJIa!$ - 19C JfT!VHE2C c1%A (((CI 9c*
dV8CJ
y1#6dCa
s0 2*J=*J? J< J77J<? K+K&&K+__main__)Nr r )(__doc__sysrB r r" htmlr urllib.parserF r bs4r ImportErrorBASE_URL LIST_PATHr r r r r r
_MAX_PAGESrc argvr) r|
startswithintrE
ValueErrorrJ r r r, rx r r r r __name__ r r r ss !
% i 0
#
@O'
chh
DAq||J QWWS!_Q /0J
i AECM 1 SXXa!e_-J
"QA$h'V B'%T"J?3D zF {
M0
s5 C, )C9%D,C65C69D DD D