ujf d Z ddlZddlZddlZddlZddlZddlZddlZdZdZ e dz Z
ej dd Z dZ
d d
d Zd Zd
Zd Zd Zd Zedk( r e yy)u, 苍南县人民政府 - 通知公告 crawler Nu" 苍南县人民政府-通知公告zhttps://www.cncn.gov.cnz7/api-gateway/jpaas-publish-server/front/page/build/unit SEARCH_DBz/root/search.db z[Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/125.0.0.0 Safari/537.36z4https://www.cncn.gov.cn/col/col1229416690/index.html)z
User-AgentRefererc L t dt j d d| y )N[z%H:%M:%Sz] )printtimestrftime)msgs /root/gov_crawler/crawl_cncn.pylogr
s! AdmmJ'
(3%
01 c ddddddt j | t d d}t d D ]u } t j
t |t d
}d |_ |j }|j d
r
|d d c S t d|j dd y y # t $ rJ}t d|dz dt | d d |dk rt j d |dz z Y d }~d }~ww xY w)N1831
1229416690 bulidstaticcolumnu 右侧列表Mm0JzuVkej5cUObgIOwPx)pageNopageSize)webIdpageId parseTypepageTypetagIdtplSetId paramJson < )paramsheaderstimeoututf-8successdatahtmlz API error: messageunknownu 请求失败(第 u 次): 2 )jsondumpsPER_PAGErangerequestsgetAPI_URLHEADERSencodingr
Exceptionstrr sleep)pager attemptrdes r fetch_list_pager= s " ++ZZ4X FGF 8 . . WVWbQA AJAuuY y(( +aeeIy 9 :; < . . "719+VCF3BK=A B{
1! ,- .s AB) B)) C<2A C77C<c " g }t d dz }d|dz d|dz d|dz d}t j d| t j D ] }t j d | }t j || }|s2|j d
j
}|j d s t |z }|j d j
}|r |j d
j
nd } |j ||| d
|S )N\ r; z( -r+ z)z"(.*?)
z)]*href="([^"]*)"[^>]*title="([^"]*)"r) http )titleurlpub_date)
chrrefindallDOTALLsearchgroupstrip
startswithBASE_URLappend)
r& artsdndpatitemmdmrE rD rF s
r parse_articlesrW - s
D R3 B
RTF!BqD62a4& 1D
@$ R L IIBD I
YYtT
" ''!*""$C>>&) nGGAJ$$&E.0rxx{((*bH KK%J K L Kr c t d D ]3 } t j | t d }d|_ |j
} n t j dt j }|r|j d nd}g }t j d |t j D ]i }t j d
d| j j d d j d
d j }|sY|j! | k dj# | } g }
t j$ d| D ]` } | j d j } | j d }
|
j' d s t( |
z }
|
j! |
| d b | |
fS # |dk( rt
j d ndg fcY c S Y xY w)Nr+ )r! r" r# r rC z (.*?)
\s*r) z]*>(.*?)z<[^>]+>z u
z,]*download="([^"]*)"[^>]*href="([^"]*)"rB )rE name)r/ r0 r1 r3 r4 textr r7 rH rK rJ rL rI subrM replacerP joinfinditerrN rO )rE r9 r: r& zmzhlinesptcontentattsrU nmlks r fetch_detailrk = s 8
S'2>A AJ66D
3T299 EB!B E
ZZ,b"))
< FF:r1 % + + - 5 5h D L LXWZ [ a a c LLO ii G
D
[[H"
M -
WWQZ
WWQZ}}V$B B B+, - D=) !|
1
2v
s /F !Gc t d t d } | s t d y t | }t dt | d t | dk( r t d y d}t d |dz D ] }t j d
t | }|st d | d 5t | }|j | |d
z dk( sZt d | dt | dt | d t dt | d t j t d }|j d t } |j dt f }|j D ] } |j | d t dt | d d\ }
} |D ] } | d |v r| dz
} t! | d \ }
}d}|
rd|
j# d d z d z }|D ] }|d |d d |d! d"z
} |j d#| d || d | d$ t f |j% |
dz
}
|
d'z dk( r|
dkD rt d(|
d)| d* t j d+ |j) t d,|
d-| d y # Y xY w# dg }}
Y xY w# t& $ r!}t d%| d d d d&| Y d }~d }~ww xY w).Nu$ 开始爬取苍南县通知公告...r) u ERROR: 第1页获取失败u 第1页: u 条r u ERROR: 解析到0条，退出 r+ g ?u 第u 页失败，跳过r u 页: u 条 (累计)u
列表共 rY )r" zPRAGMA journal_mode=WALz+SELECT title FROM gov_raw WHERE site_name=?u 已有记录: )r r rD rE rC zr[ z
z
z r\ z
z\INSERT INTO gov_raw (title, content, source_url, publish_date, site_name) VALUES (?,?,?,?,?)rF u DB写入失败: rZ d u 进度: u 存, u 跳g333333?u 完成! 新增 u
条, 跳过 )r
r= rW lenr/ r r7 extendsqlite3connectr executeset SITE_NAMEfetchalladdrk r_ commitr5 close)r& all_arts total_pagesre hrQ dbexistingcurrowsavedskippedartrg rh bodyar< s r mainr Z s ./ 1 D () d#H)CM?$'(
8} +,K
1kAo
& G
3 A #aS*+ , a r6Q; #aSc$i[ S]O1E F G *S]O4() B /BJJ()uH
jjF U<<> !C LLQ ! .X t,-NE7 w<8 # qLG # (U 4MGT 7??4;;fDD CA l1U8*BqykB BD C < JJuG dCJJS
U IIK QJE 3;! (5'y5 6
3-. HHJ/%
gYd;<9
# TG < "3w<#4"5Qqc: ; ; r s 3 1 1 1 1 0 $
N
N BIIk#4 5
pE
22 ::=x zF r