uj8 X d Z ddlZddlZddlZddlZddlZddlmZmZmZ ddl Z ddl
Z
ddlZ e
j e
j j dZdZdZ ej" dd Zd Z ej( ej* ed
z
j- d Zd
diZd Zd Zd Zd Z d"dZ e dk( rodej> v Z e!de de rdnd de d ej
Z" e e \ Z#Z$ e!d ej
e"z
d d e!d e# d!e$ yy)#u
昔阳县工信局-通知公告 (www.xiyang.gov.cn)
==============================
Power CMS — 政府信息公开平台
工信局 → 法定主动公开 → 政策文件 → 通知公告
列表: https://www.xiyang.gov.cn/zwgk/zfxxgkml/xzgzbm/23gxj/fdzdgknrrsj11111/gwfg3rsj11111/gggs3rsj11111
详情: /.../content_XXXXX
正文: article.articleCon
分页: 后缀 _N
用法:
python3 crawl_xiyang.py # 全量（2页，近3年）
python3 crawl_xiyang.py --test # 测试 5 条
N)datetimetimezone timedeltau 昔阳县工信局-通知公告zhttps://www.xiyang.gov.cnzahttps://www.xiyang.gov.cn/zwgk/zfxxgkml/xzgzbm/23gxj/fdzdgknrrsj11111/gwfg3rsj11111/gggs3rsj11111 SEARCH_DBz/root/search.db iG )daysz%Y-%m-%dz
User-AgentzoMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36c t j | t dd }d|_ |j S # t
$ r}t
d| Y d }~y d }~ww xY w)N F)headerstimeoutverifyzutf-8u ⚠ fetch fail: )requestsgetHEADERSencodingtext Exceptionprint)urlres %/root/gov_crawler/crawl_xiyang_gxj.pyfetchr ! sN LLgr% H
vv
"1#&'s /2 AAAc ^ g }t j d| t j }|s|S t j d|j d t j D ] }|j d }t j d| }t j d| }|sC|j d j }|j d j }|j
d s t |z }|r|j d nd} |s|s|j ||| d |S )
u) 提取列表条目 (url, title, pub_date)z z]*>(.*?) z2]*>(\d{4}-\d{2}-\d{2})z&]*href="([^"]*)"[^>]*>([^<]+) http )titler pub_date) researchDOTALLfinditergroupstrip
startswithBASE_URLappend)
htmlitemsareamlidate_mlink_mhrefr r s
r
parse_listr2 * s E
997ryy ID
[[/A
J R
WWQZPRTUDbI <<?((*DLLO))+E??6* $*0v||AbH uTx PQ R L c t | }|sddddS d}t j d|t j }|r |j d j }|sLt j d| }|r4t j dd|j d j }d}t j d|t j }|r|j d dd }|s)t j d
| }|r|j d }d}t j d |t j }|r|j d }n8t j d |t j }|r|j d }|rt j d
d|t j t j z }t j dd|t j t j z }t j dd|t j }t j dd| }|j }|||dS )u+ 获取详情页的 title, content, pub_dater )r contentr z3]*name="ArticleTitle"[^>]*content="([^"]*)"r z([^<]+)u
[-_—|].*z-<meta[^>]*name="PubDate"[^>]*content="([^"]+)N
u( 发布时间[：:]\s*(\d{4}-\d{2}-\d{2})z+<article class="articleCon">(.*?)</article>z=<div class="printArea"[^>]*>(.*?)</div>\s*<div class="others"z<script[^>]*>.*?</script>)flagsz<style[^>]*>.*?</style>z <link[^>]*>z style="[^"]*")r r! r" Ir% r& subr# )r r* r r- r r5 arts r fetch_detailr; > s :D ;;
E
H$PRPTPTUA
" II' . FF="aggaj9??AE H
BD"$$OA771:cr? IIA4 H wwqzH G
))BD"))
TC
))A, IIVX\^`^g^g h ggajG&&5r7"))TVTXTX.Y&&3R RTRVRVW&& WBDDA&&*B8--/wH EEr3 c | s t d yddl}|j t }|j d |j d d\ }}| D ] } |j dt
|j
d xs d dd
|j
d d |j
d d |j
d
xs d dd d df |j dkD r|dz
}n|dz
} |j dt
f |j |j t d| d| d ||fS # t $ r}|dz
}Y d}~d}~ww xY w)u4 批量写入 search.db（gov_raw 表 + FTS 同步）u ⏭ 无数据r r r NzPRAGMA journal_mode=WALzPRAGMA synchronous=NORMALzwINSERT OR IGNORE INTO gov_raw (site_name, title, page_url, content, publish_date, summary, tags) VALUES (?,?,?,?,?,?,?)r r i r r5 r r6 u 通知公告r zINSERT INTO gov_search(rowid, title, site_name, summary) SELECT r.id, r.title, r.site_name, r.summary FROM gov_raw r WHERE r.id NOT IN (SELECT rowid FROM gov_search) AND r.site_name=?u 💾 入库: 新增u , 跳过u , FTS已同步) r sqlite3connectr execute SITE_NAMEr
total_changesr commitclose)r+ r> dbokfailitr s r to_dbrI l sS
#BJJ()JJ*+HB JJ) VVG_*DS1FF5"%FF9b)VVJ'-2s3"
!#a '. JJ \
IIKHHJ !"XdV>
BC
t8O AID s BD'' D?0D::D?c ^ t g}t dt dz D ] }|j t d| g }t }|D ] }t | }|st
| }|s#t d|j | dz d n|D ]A }|d |v r |j |d |d r
|d t k r1|j | C |d j dd } |d
j dd }
t d |j | dz d t | d
| d|
d | r |dd }t dt | d t dt | d |syg } t |d D ] \ } }t |d }
|
d xs |d |d |
d |
d xs |j dd d}| j | | dz dk( s| t | k( sjt | t | k( rd| d t | d nd| d t | d t | \ }}||fS ) u 全量爬取r r _u ⚠ 第u! 页无数据，可能已到末页r r r ?u
📄 第u 页: u 条 (z ~ )Nr u" 🧪 测试模式: 只处理前 u 条u
📊 列表汇总: u 条（近3年）r= r r5 r )r r r5 r z [/u ] 📝 全部完成z] ...)LIST_URLrange MAX_PAGESr) setr r2 r indexaddCUTOFFr len enumerater; rI )testurlsn all_items seen_urlspage_urlr* r+ item
first_date last_dateresultsidetailentryrF rG s r crawlrf s
:D
1i!m
$ ' xj!%&' II fX 4 Idjj 21 4 55VW X #DE{i' MM$u+ &J D$4v$= T "
# 1X\\*c2
"IMM*c2
4::h/12%E
|5 TWXaWbbcde%f( bqM
23y>2B$GH #C N#33D
EF G Y*
}4 d5k*G_5W
; i( z*Fdhhz2.F
u q5A:c)n, 1ICVCs!C N++>?^abcaddefijsfteuuz\{ |
} W~HB
t8Or3 __main__z--testu
📡 [z] u 测试模式u 全量u (最多u
页, 近3年))rY u ⏱ 耗时: z.1fsu ✅ 新增: u ❌ 跳过: )F)%__doc__r! sysosjsontimer r r r urllib3disable_warnings
exceptionsInsecureRequestWarningrA r( rP getenvr rR nowutcstrftimerV r r r2 r; rI rf __name__argvrY r t0rF rG r3 r <module>rz sN 2 2 ++BB C /
(
p BIIk#4 5
hll8<<(9%+@@
J
J:
V N O(,F\*X6p z sxx D HYKrD.h!GxPY{Zg
hi B$ HB N9499;r>#.a
01 N2$nTF
34
r3