ujZ
d Z ddlZddlZddlZddlZddlZddlZddlZddl Z ddl
Z
ddl m Z m Z ddl
mZ e j j! dd ZdZdZd Zd
Zd d iZ e j. e d
z
j1 d ZdZ ej6 Z ej: j< e _ e jA d de _! ejD e _# e$ Z%d Z&d Z'd Z(d Z)d Z*d Z+e,dk( r e+ yy)u
crawl_ynzk.py — 镇雄县-环境保护 (ynzk.gov.cn)
CMS: 镇雄县信息公开平台（自建）
列表: 静态HTML分页 /zfxxgk_zkx_sthjj/hjbh.html (首页) / .../index_2~5.html
TITLEYYYY-MM-DD
详情: / / 正文HTML
页数: 5页, ~15条/页, 共约75条
注意: SSL证书使用国密曲线，需Python自定义SSL context (curl的OpenSSL 3.x无法兼容)
N)datetime timedelta)urljoin SEARCH_DBz/root/search.dbu 镇雄县-环境保护u 镇雄县人民政府z.https://ynzk.gov.cn/zfxxgk_zkx_sthjj/hjbh.htmlz)https://ynzk.gov.cn/zfxxgk_zkx_sthjj/hjbhz
User-Agentz]*>(.*?)zBhref="(/zfxxgk_zkx_sthjj/artview/\d+/\d+\.html)".*?title="([^"]*)"z (\d{4}-\d{2}-\d{2}) /zhttps://ynzk.gov.cn) refindallDOTALLsearchgroupstrip
startswithr LIST_BASE seen_urlsaddappend)
htmlitemsddsddmm2rel_urltitledate_strfull_urls
r extract_listrA . s E
**+T299
=C 6 II[]_acajaj k
YY:B
? ggajGGGAJ$$&Exx{((*H:A:L:LS:Q,w6W^_hjqWrH9$ MM( # LL(E84 5 6 Lr c d}d}d}t j d| }|r |j d j }t j d| }|r |j d j }t j d| t j }|r |j d j }|||fS )N z!ArticleTitle"\s*content="([^"]+)"r) z PubDate"\s*content="([^"]+)"z#class="xxgk-articleBox">(.*?))r, r/ r0 r1 r. )r7 r> contentpub_dater; s r extract_detailrF ? s EGH
6=A
"
148A771:##%
8$ JA''!*""$ '8 ##r c t j t }|j }d}d}| D ]F \ }}}} |j dt
|||t |xs df |j dkD r|dz
}n|dz
}H |j |j ||fS # t $ r } t d|dd d| Y d } ~ d } ~ ww xY w)Nr zINSERT OR IGNORE INTO gov_raw
(site_name, title, page_url, publish_date, source_url, content)
VALUES (?, ?, ?, ?, ?, ?)rC r) u 入库失败: ir ) sqlite3connectDB_PATHcursorexecuter
SOURCErowcountr" r commitclose)
articlesconnrK insertedskippedr# r> daterD r& s
r
save_to_dbrV N s ??7 #D
[[]FHG%-
3!UD' 3 NN0 E3fgmD
"A
1
3 KKMJJL W 3 .ST 2aS1 2 2 3s >B C&B??Cc
t gt dt dz D cg c] } t d| d c} z }g }|D ] }t d| t | }|s t
| }|s
t d nt dt | d |D ] \ }}}|t k rt d | d
|d d %t j t j d d
t | } | s\t | \ }
} } |
s|}
| s|} | st d|
d d |j ||
| | f t j d t | \ }
}t d|
d| dt | d y c c} w )Nr* r) z/index_z.htmlu 取列表: u 无内容，停止分页u 共u 条u 跳过(旧): ( g333333?g?u
无正文: u 完成！新增u 条，跳过u 条（总计u 条）)BASE_URLrange MAX_PAGESr3 r r' rA lenCUTOFFtimesleeprandomuniformrF r6 rV )ipagestotal_articlespage_urlr7 r8 r# r> r? detail_htmld_titlerD d_daterS rT s r mainrj e s
Jq)VW-AXYAYKwqc7Y YEN k( $%X T" , . eCJ JJv~~c3/ 0 *K '5k'B $GWf !mGCRL>23 ! !3"A B% C(
3? B #>2Hg/(<y S EXDYY_`aK Zs E/__main__)-__doc__sslurllib.requestr r, rH jsonr_ ra ossysr r urllib.parser environgetrJ r
rM rZ r3 r nowstrftimer^ r\ create_default_contextr
TLSVersionTLSv1_2minimum_version set_cipherscheck_hostname CERT_NONE verify_modesetr4 r r' rA rF rV rj __name__ r r r s E D D D D (
**..&7
8 $ ; 7 W
X
(,,.9'2
2 <