r]j d Z ddlZddlZddlZddlZddlZddlZddlmZmZm Z ddl
Z
ddl Z ddlZddl m
Z
e j dZdZdZ ej$ dd Z ej( ej* e d
z
j- d Zd
ZddiZd Zd Zd Z ddZ e dk( r2 e ej@ dkD xr ej@ d dv Z! e e! yy)u
独山子区 - 生态环境信息公开
https://www.dsz.gov.cn/dsz/sthj/zfxxgk.shtml
政府信息公开平台 - API直取搜索接口: /search/{channelId}?page=N&_pageSize=15
channelId = dc418fb236024fbf81277f159a17ada5
N)datetimetimezone timedelta)
BeautifulSoupu 独山子生态环境zhttps://www.dsz.gov.cn/search/{channelId}?page={page}&_pageSize=15&_isAgg=true&_isJson=true&_template=index&_rangeTimeGte=&_channelName= dc418fb236024fbf81277f159a17ada5 SEARCH_DBz/root/search.dbiG )daysz%Y-%m-%d z
User-Agentz:Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/125.0.0.0c t j t | } t j |t
d }d|_ |j S # t $ r }t d| d| d Y d }~y d }~ww xY w) N) channelIdpage )headerstimeoututf-8u
❌ API请求失败 page=: T)flush)
LIST_URLformat
CHANNEL_IDrequestsgetHEADERSencodingjson Exceptionprint)r
urlres crawl_dsz.py fetch_apir" sh
//JT/
:C LLgr :
vvx
+D6A37tDs 2A A6A11A6c t j | t dd }d|_ t |j
d }|j
d }|syg }|j d D ] }|j d
}t j d d
| j }|s=t j d| rTt j d| rt | dk ry|j
d r8dj d |j dd
j d D }|j! | |rdj | S dS # t" $ r}t% d| d| Y d}~yd}~ww xY w)u 抓详情页正文 —— API 的 content 字段是【平铺无分段】的，
网页 #zoomcon 里才是带 分段的原文（实测 10 个
→ 5 段）。
失败返回 None，调用方回落 API content。
F)r r verifyr z html.parserzoomcon)idNpTstripz[\ue000-\uf8ff] uT ^(索引号|有效性|发文机关|文号|成文日期|发布日期|关键字)[：:]u' ^(日期|来源|作者|访问量)[：:]< br
c 3 B K | ] }|j s| y w)Nr) ).0ss r! z fetch_detail..8 s _AUVU\U\U^a _s z
u ⚠️ 详情页失败 r )r r r r r textfindfind_allget_textresubr* matchlenjoinsplitappendr r )r r soupboxpartsr( tr s r! fetch_detailrB ! sJ
LLgr% H
QVV]3ii9i% c" A
&A)2q1779AxxoqrsxxBAF3q6TV;vvd|II _D)E)K)KD)Q __ LLO &+v{{5!44
*3%r!56s% AE C:E E E2E--E2c , | sy t j t d }|j d |j d d\ }}| D ] } |j d|j dd d d |j d
d |j d
d |j d xs dd d |j d
xs dd d |j d xs dd d |j dd dd|j d xs dd d f
|j
dkD r|dz
}n|dz
} |j
| d j dd }|j d|f |j
|j t d| d| y # |dz
}Y KxY w)Nr, )r zPRAGMA journal_mode=WALzPRAGMA synchronous=NORMAL)r r zINSERT OR IGNORE INTO gov_raw (site_name, source_url, page_url, title, publish_date, summary, content, status, category, tags) VALUES (?,?,?,?,?,?,?,?,?,?) site_namer+ r titlei pub_date
summarycontentactivetagsd r zINSERT INTO gov_search(rowid, title, site_name, summary) SELECT r.id, r.title, r.site_name, r.summary FROM gov_raw r WHERE r.id NOT IN (SELECT rowid FROM gov_search) AND r.site_name=?u
💾 入库: 新增u , 跳过) sqlite3connectr executer
total_changescommitcloser )itemsdbokskipitemrD s r! insert_to_dbrZ @ s B /BJJ()JJ*+HB
JJ / R(#.r0Br"TXXg%6%<"ds$C*%+Sb1)$*DS12&"&!'R#.
1
!#R1WR!)$ IIKa ["-IJJ MOXl\ IIKHHJ !"XdV
45 s
CF Fc @ t j }t dt dd | rdnt }d}t d|dz D ] }t d| ddd t | }|r|j
d
st d nQ|d
}|j
d g }|st d
n*t dt | d|j
dd d |j
dd }g }|D ] } | j
dd j }
| j
dd j } | j
d xs dd d } | j
dd xs d}
|
r| sw| r
| t k rt j dd|
t j t j z }t j dd|t j t j z } ddlm} || }t j d d| }t# | }|r t | t | d z kD r|}n|j }t j d d| d d j }| j% d rd!| d"d z } |j' t |
| ||| t d# |r t) | d$}||z |k\ r nt j* d% t d&t- || d$z d't j |z
d(d) y # t $ r Y w xY w)*Nr. z(========================================rN r u
📄 第u 页... T)endr datau
❌ 无数据resultsu 空结果u 📊 u 条 (共totalu 条)rF r+ r publishedTimeStrrH rJ z]*>.*?)flagsz]*>.*?)unescapez [\u00a0\u2000-\u200a\u3000]+g?z<[^>]+>rE zhttp://zhttps:// )rD rF r rJ rI rG rL g ?u
✅ 完成! 共扫描 u 页
⏱ z.1fr1 )timer SITE_NAME MAX_PAGESranger" r r: r* CUTOFFr7 r8 SIhtmlrc r rB
startswithr= rZ sleepmin) incrementalt0
page_limit
total_recordsr
r^ dr_ rU rY rF r rG rJ
content_clean_unescdetailrI page_sizes r! mainrz _ s
B BykF8
$%!yJMaa ( A
4&'S=488F+ / " L%% 2& +
c'l^9QUU71-=,>dCDgq)
% :DHHWb)//1E((5"%++-C !34:CR@Hhhy"-3GHv - FF#?WTVTXTXY[Y]Y]T] ^M FF#=r=XZX\X\]_]a]aXa bM
3 &} 5
"'F] [
"#&F#f+M(:S(@@ &
- 3 3 5
ffZ];DSAGGIG ~~i( 3qr7* LLy5} $i9
:G% :N ) } ,
3CAF
$S
~7K5L%M$NjY]YbYbYdegYghkXllm
no=
s #%L L L __main__rN )1z
--incremental)rq )F)"__doc__sysosr7 rf r rO r r r r urllib3bs4r disable_warningsrg r r getenvr nowutcstrftimerj rh r r" rB rZ rz __name__r: argvrq r! r s ( ' ' 2 2 # V
/
BIIk#4 5
(,,x||
$ye'<
< F Fz R
U
V > 6>JpX zchh-!#M 7M(MK[! r