ͫja'
d Z ddlZddlZddlZddlZddlZddlZddlmZmZ ddl m
Z
ddl m Z ej dd ZdZd Zd
Ze d Zd d
ddddddddddd ZddddZd Zd Zd Zd Zd Zd! Zd3d"Zd# Z e d$k( r e ej> d%kD rej> d% nd&Z e d&k( r e ye d'k( r e ye dk( r~ ed% Z!e!re!jE d(d ndZ# ed)e# e!jE d* xs g dd+ D ]9 Z$ e%d,e$jE d- xs d.dd/ d0e$jE d1d. dd2 ; yyy)4uT
Crawler: 芜湖市弋江区人民政府 - 生态环境 (Epoint 基层政务公开)
N)datetime timedelta)
BeautifulSoup)urljoin SEARCH_DBz/root/search.dbu 弋江区生态环境分局u 生态环境zhttps://www.yjq.gov.cnz/wuhu/site/label/8888grassrootsContentPageList67878916603431151truez
yyyy-MM-dd504list1000161) labelNamesiteIdorganIdpageSize pageIndexisDate
dateFormatlengthtypeactionisJsoncatIdzoMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36z!application/json, text/plain, */*z0application/x-www-form-urlencoded; charset=UTF-8)z
User-AgentAcceptz Content-Type)
z.docz.docxz.pdfz.xlsz.xlsxz.pptz.pptxz.zipz.rarz.7zc 2 t dt d| d y )N[z] T)flush)printSITE)msgs crawl_yjq.pylogr& % s AdV2cU
4( c i t dt | i} t j t |t
dd }|j
S # t $ r}t d| Y d}~yd}~ww xY w)z%Fetch one page of list data from API.r F)dataheaderstimeoutverifyz API error: N) API_DATAstrrequestspostAPI_URLHEADERSjson Exceptionr& )
page_indexr* res r% fetch_list_pager9 ) s` 5h 5 S_ 5D MM'grRW Xvvx k! s 1A A*A%%A*c t d| } t j |t dd }d|_ |j
|fS # t $ r }t d| d| d|fcY d}~S d}~ww xY w) z&Fetch detail page and extract content.z/grassroots/content/r) F)r+ r, r- zutf-8zFetch error for : N)BASE_URLr0 getr3 encodingtextr5 r& )
content_idurlr7 r8 s r% fetch_detailrB 4 sq
J*:,
7C LLgr% H
vvs{ zl"QC 01Sys 1? A'A" A'"A'c dddd}t j d| }|rI|j d j }t j dd| j }||d< t j d| }|r%|j d j dd |d
< | j d }|d k r| j d
}|d k\ r| j
dd | }d}| j d| }|d kD r|t | k r|dz
}| ||dz dk( r | j d| }|d k\ r|dz }@| | dk( r| |dz |dz dk7 rn[| ||dz dk( rP| j d| } | |kD r9| |dz | j d }
|
r|
dk7 r| }|dz }|d k( r | || dz |d< n |d kD r|t | k r| ||dz |d< |d rt |d |d< |S )z.Extract title, date, content from detail page. title publish_datecontentz(.*?) u* [_\-—]\s*芜湖市政务公开平台\s*$rF zPubDate.*?content="([^"]+)"N
rG z"class="j-fontContent gkwz_contnet"r gkwz_contnet<> z /z,>,@@r' c
t j t | dz z
j d }t d| d t d }|s t d y |j
dd }t t d
}||z dz
|z }t d | d | g }t dt |dz d
D ] }t | }|s n|j
dg }|D ]j } | j
d xs dd d }
|
|k r!|j | d | j
dd |
| j
dd | j
dd d l t d| dt | d t j d t dt | d g } t |d D ] \ } } | d rGt! | d \ }
}|
r.t# |
}|d r|d | d< |d r|d | d < |d | d < n d| d < nd| d < | j | d | d | d | j
d d t$ d!t& t& d" | dz d k( rt d#| d$t | t j d d }d }d d l}|j+ t, }|j/ d% |j/ d& |j1 }| D ]V } |j/ d'| d( | d) | d) | d | d | d | d xs dd d* | d+ f |j2 d kD r|dz
}n|dz
}X |j7 |j9 t d-| d.| d/ y # t4 $ r }t d,| |dz
}Y d }~d }~ww xY w)0Nr) days%Y-%m-%dzFull crawl (cutoff: rk rI zFailed to fetch page 1totalr r Total: z , pages: i r* publishDaterD rJ contentIdrF urlDomainpublishDepartment)r@ rF daterA
departmentz Page r; z items ?zAfter filter: rA r@ rG r rH -rF page_urlrG rH site_namecolumnz
Detail: rQ PRAGMA journal_mode=WALPRAGMA busy_timeout=10000INSERT OR IGNORE INTO gov_raw (site_name, source_url, page_url, title, publish_date, content, summary, category) VALUES (?,?,?,?,?,?,?,?)r r r
DB error: Result: new, skipped) r nowr strftimer& r9 r= intr. rangeminr{ rZ timesleep enumeraterB rg r# COLUMNsqlite3connectDB_PATHexecutecursorrowcountr5 commitclose) months_backcutoffr* r page_size total_pages all_itemspgitemsitempub_dateenrichedr` r] _detailstoredskippedr conncr8 s r% crawl_allr s llnykB.> ??
I
I*
UF
vha() 1 D $% HHWa EHZ()I9$q(Y6K'% + /0IAs;?C01 r" $
D /52s;H& "; / '2. xx R0 "hh':B ?
gbTCJ"23 4
338 FG ??7 #DLL*+LL,-
A
II, %tJ'7j9IM4#7i )_*DS14>
zzA~! 1 & KKMJJL(6(& 23 *QC ! qLG s AM M7M22M7c
t j t d z
j d } t d| d t d }|sy |j
dg }|D cg c] }|j
d xs d d d
| k\ s |" }}g }|D ] }|j
d d }t |d \ }} |r t | nd d d d
}
|j |
d xs |j
dd ||j
d xs d d d
|
d t dt t d t j d d} d} dd l}
|
j t }|j# d |j# d |j% }|D ]V } |j# d|d |d |d |d |d |d |d xs d d d |d f |j& dkD r| dz
} n| dz
} X |j+ |j- t d | d | d y c c}w # t( $ r }t d | | dz
} Y d }~d }~ww xY w) N r r zIncremental (since rk rI r* r rD rJ r r rE rF rH r r r r r r r r r rG r r r r r r )r r r r r& r9 r= rB rg r{ r# r r r r r r r r r r5 r r )r r* r itfilteredr r rA r] r r r r r r r r8 s r% crawl_incrementalr si llnya 00
:
::
FF
fXQ'( 1 D HHVR E "Srrvvm'<'BCR&HF&RSHSH hh{B' tK 01a'+ d#2r^`1aG_="(= !XXm 4 :CR @ i( 66(+
3 FG ??7 #DLL*+LL,-
A
II, k"D$4d:6Fg ^ 4d9oy/'R#.X@
A zzA~! 1 KKMJJL(6(& 23U TJ *QC ! qLG s% $ HH
AH I #H;;I __main__rI incrementalfullr r r* z r rD rJ z | rF < )$ )&__doc__sysosr4 rS r r0 r r bs4r urllib.parser getenvr r# r r< r2 r. r3 ATTACH_EXTSr& r9 rB rg r\ r r __name__rZ argvmoder* r= r r r" r' r% r s ) ( ( (
"))K!2
3$ #
J+
,,
D1F
a ) 3lA:^4B24j z chh-!+388A; D }
q!(,!$! geW 88F#)r2A. YB B}-3Sb9:#bffWR>PQTRT>U=VW X Y
r'