uj | d Z ddlZddlZddlZddlZddlmZmZmZ ddlZddl Z ej j dej j ej j e ddlmZ e j" e j$ j& dZdZdZdZ ej0 ej2 ed
z
j5 d Zd Z d
diZ d Z d Z d Z d Z!ddZ"e#dk( rF ej Z$ e"ejJ dd e&d ej e$z
dd yy)u
甘南县 - 信息公开 (gannan.gov.cn)
列表: https://www.gannan.gov.cn/gannan/c100408/zfxxgk_list.shtml
翻页: index_2.shtml, index_3.shtml ...
详情: /gannan/c100408/202605/c02_623841.shtml
正文: article 区域或主内容 div
日期: meta PubDate 或 "发布时间"
N)datetimetimezone timedelta)push_to_searchdbu 甘南县-信息公开zhttps://www.gannan.gov.cnz:https://www.gannan.gov.cn/gannan/c100408/zfxxgk_list.shtmlz)https://www.gannan.gov.cn/gannan/c100408/iG )daysz%Y-%m-%d z
User-Agentz Mozilla/5.0c r t j | t dd }d|_ |j S # Y y xY w)N F)headerstimeoutverifyzutf-8)requestsgetHEADERSencodingtext)urlrs !/root/gov_crawler/crawl_gannan.pyfetchr s5 LLgr% H
vv
s /2 6c g }d}t j || t j D ] }|j d j }t j
dd|j d j }|rt
| dk rgd|v sd|v rp|j |t |z f t }g }|D ]. \ }} | |vs |j | |j || f 0 |S ) u$ 解析列表页，提取 链接 z;]*href="(/gannan/c100408/[^"]+\.shtml)"[^>]*>(.*?) <[^>]+> r index_ zfxxgk_list) refinditerDOTALLgroupstripsublenappendBASE_URLsetadd)
htmlitemspatternmhreftitleseenuniqtus
r
parse_listr3 ! s ELG
[[$
2 /wwqz !z2qwwqz288:E
Q t } 4
eX_-./ 5D
D 1 D= HHQK KKA K c 0 | dk( rt S t d| dS )u. 第1页用主URL，第2页起用 index_N.shtmlr r z.shtml)LIST_URLLIST_DIR)pages r get_page_urlr9 8 s! qyZvdV6 **r4 c F t | }|syi }t j d|t j }|r7t j dd|j d j
|d< |j d s^t j d|t j }|r7t j dd|j d j
|d< t j d|t j }|r|j d |d < |j d s,t j d
| }|r|j d |d < |j d s,t j d | }|r|j d |d < d }t j d
|t j }|r |j d j
}|rt | dk rsdD ]n }t j d|z dz |t j }|s1t |j d dkD sO|j d j
} n |rt | dkD rt j dd|t j t j z }t j dd|t j t j z }t | dkD r||d< |j d |j d |j d fS )u0 提取详情页：标题、正文、发布日期)NNNz]*>(.*?)r r r r. z]*>(.*?)z=]*name="PubDate"[^>]*content="(\d{4}-\d{1,2}-\d{1,2}) publish_dateu, 发布时间[：:]\s*(\d{4}-\d{1,2}-\d{1,2})z!(\d{4}-\d{2}-\d{2})\s*\d{2}:\d{2}Nz ]*>(.*?)d )
zarticle-contentarticle_contentcontentarticler z main-content mainContentmain
TRS_Editorzoom NewsContentzdetail-contentdetail_contentz <(div|section)[^>]*class="[^"]*z[^"]*"[^>]*>(.*?)r 2 z]*>.*?)flagsz]*>.*?r> )
r r searchr r# r! r" r Ir$ )r r) resultr, r> clss r fetch_detailrL > s :D
F ($ :A &&R<BBDw ::g II,dBII > ffZQWWQZ @ F F HF7O RTXZ\Z^Z^_A!"~ ::n % IIEt L %&WWQZF> " ::n % II:D A %&WWQZF> " G
2D"))DA''!*""$ c'lS(H C 2S8;TTbiiA S_s*''!***, 3w<"$&&5r7"))VXVZVZJZ[&&3R TVTXTXHXY w<# 'F9 ::g
9 5vzz.7Q QQr4 c d}| rd| v sd| v sd| v rd}t dd dt dd |r t d g }|rd
nt }t d
|d
z D ]| }t | }t d | d d
d t | }|s
t d nEt
| }|s
t d n+t dt | d |j | ~ t dt | d g t } }t | D ] \ }
\ } }|| v r| j | t d|
d
z dt | d| d d dd
d t | \ } }
}| r| } |r|t k r t d q|
rt |
dk r t d t j dd
|
j! d d }t j d d
| }|j# t | ||
|xs d |t d t d t |
d! t% j& d" |r t) |d# t d$t | d y )%NFz
--incrementalz-i1T
z2==================================================u
🏠 u$ 📌 增量模式：仅爬取第1页r u
📄 第 u 页... )endflushu ❌ 请求失败u 0 条u ✅ u 条u
📊 列表总计: z [/z] rF z...u ⏭ 超时范围u
⚠ 无正文r i, z\s+r ) site_namer. r r> pub_datesummarytagsu ✅ (u 字)g333333? gannan_gku
✅ 完成! 共入库 )print SITE_NAME MAX_PAGESranger9 r r3 r$ extendr' enumerater( rL THREE_YEARS_AGOr r# r" r% timesleepr )args incrementalall_listpagesr8 r r) r* all_itemsr/ ir. dtr> daterV s r runrj z s[ K D(DDLC4K Bvhgi[6(
34
45 H A)Ea# 4
D6)s$?Sz $ % 4 'N
SZL%&
!#h-
56 #%tI$X. !r s 2 2 277??277??8#<= > ( ++BB C $ & G 6 8<< - u0EEOOPZ[
'.+ 9RxC;L z B LR ,A
./ r4