uj d Z ddlZddlZddlZddlZddlZddlmZmZmZ ddl Z ddl
Z
ej j dej j ej j e ddlmZ e
j$ e
j& j( dZdZdZ ej0 ej2 ed z
j5 d
Zd d iZ d
Z d Z d Z d Z ddZ!e"dk( r` ej
Z# e! e$ejJ dkD xr ejJ d dv e&d ej
e#z
dd yy)u
垣曲县生态环境 (yuanqu.gov.cn) 爬虫
http://www.yuanqu.gov.cn/zfxxgk/jczwgk/gknr/gknr_hjbh/index.shtml
CMS: 静态HTML，分页 index_N.shtml
规则: 全部9页，标题含"项目"，近3年
N)datetimetimezone timedelta)push_to_searchdbu 垣曲县生态环境zhttp://www.yuanqu.gov.cnz)/zfxxgk/jczwgk/gknr/gknr_hjbh/index.shtmliG )daysz%Y-%m-%dz
User-Agentz[Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/125.0.0.0 Safari/537.36c t j | t d }d|_ |j S # t
$ r}t
d| Y d }~y d }~ww xY w)N )headerstimeoutzutf-8u ❌ 请求失败: )requestsgetHEADERSencodingtext Exceptionprint)urlres &/root/gov_crawler/crawl_yuanqu_sthj.py
fetch_pager sL LLgr :
vv
$QC()s .1 AA
Ac N g }d}t j || D ] }|j d }|j d j }|j d j }|r|sW|j d s t
|z }|j
|||dd f |S )u2 解析列表页，返回 [(title, url, date), ...]zO]*title="([^"]*)"[^>]*>.*?([^<]+) httpN
)refinditergroupstrip
startswithBASE_URLappend)htmlitemspatternmhreftitledate_strs r
parse_listr, ( s E`G
[[$
' 3wwqz
"771:##%5 v&d?D
eT8CR=123 L c j t j d| }|rt |j d S dS )u 获取总页数z"pageCount":"(\d+)"r )r searchintr )r% r( s r get_total_pagesr1 9 s,
($/A 3qwwqz?&Q&r- c t | }|syi }t j d|t j }|r7t j dd|j d j
|d< t j d| }|r|j d |d< t j d |t j }|r|j d j
}t j d
d|t j t j z }t j d d|t j t j z }||d
< |j d |j d
|j d fS )u' 获取详情页标题、正文、日期)NNNz]*>(.*?)<[^>]+> r r* u, 发布日期[：:]\s*(\d{4}-\d{1,2}-\d{1,2}) publish_datez ]*>(.*?)
z]*>.*?)flagsz]*>.*?content) r r r/ DOTALLsubr r! Ir
)r r% resultr( r7 s r fetch_detailr< ? s7 c?D
F ($ :A &&R<BBDw A4HA!"~ 4dBIIFA''!*""$&&5r7"))VXVZVZJZ[&&3R TVTXTXHXY #y
7
9
>" r- c t dd t dt t d t d t t t z }|s t d y t | }t d| d g }t
d| rd n|dz D ] }|dk( rt t z }n&t t j d
d | d t z }t d
| ddd t | }|s
t d n6t | }t dt | d |j | t dd t dt | d g }t }t | D ] \ } \ }
}} d|
vr| r
| t k r||v r |j | t d| dz dt | d|
d d d dd t! | \ } }
}| r| }
|xs | }t j d d|
xs d j# }t j d d| d d }|j% t |
||
xs d ||t d! t d" t' j( d# t dd t d$t | d |s t d% y t d& t+ |d' t dd t d(t | d) t d y )*N
z2==================================================u 🏠 u7 爬取全部页面，标题含「项目」、近3年u ❌ 无法获取列表页u 📊 共 u 页r r z/index\.z/index_.u
📄 第 u 页... T)endflushu
❌ 无返回u ✅ u 条u ──────────────────────────────────────────────────u 📊 列表总计: u 项目z [/z] 2 z...r3 r4 z\s+i, ) site_namer* r r7 pub_datesummarytagsu ✅g333333?u 📊 总计匹配: u ⏭️ 无数据，跳过推送u
📤 推送至服务器... yuanqu_sthju ✅ 完成! 共 u 条已推送至服务器)r SITE_NAMEr r# LIST_URLr1 ranger r9 r, lenextendset enumerateTHREE_YEARS_AGOaddr< r! r$ timesleepr ) incrementalr% total_pagesall_list_itemspager r& all_items seen_urlsir* r+ detail_titler7 detail_datedaterG s r mainr_ a s Bvh- E)
CE VH h) *D
)*!$'K Ik]$
'( Nak{Q? % 19X%CRVVK74&1BH MMC
D6)s$?# / " 4
SZL%& e$ %
Bxj/ N 3 4D
9: II%.~%> ! !E3 5 ?2 )
c
AaC5#n-.bs C@cQUV-9#->* g{ E&h&&S'-R8>>@&&g.t4" }"
3A D
Bxj/ I /t
45
/0 (*Y
. Bvh- S^ ,,E
FG VHr- __main__r )1z
--incremental)rU u ⏱ 耗时: z.1fzs
)F)'__doc__sysosr jsonrS r r r r urllib3pathinsertdirnameabspath__file__ crawler_libr disable_warnings
exceptionsInsecureRequestWarningrJ r# rK nowutcstrftimerQ r r r, r1 r< r_ __name__t0rM argvr r- r rw s5 2 2 277??277??8#<= > ( ++BB C # % 6 8<< - w0GGQQR\] o
"' DVr z BS]Q&P388A;:P+PQ Lr)#.c
23 r-