uj'
d Z ddlZddlZddlZddlZddlZddlZddlmZmZ ddl m
Z
ddl m Z ej dd ZdZd Zd
Ze d Zd d
ddddddddddd ZddddZd Zd Zd Zd Zd Zd! Zd" Zd4d#Z d$ Z e d%k( r e ej@ d&kD rej@ d& nd'Z!e!d'k( r e ye!d(k( r e ye!dk( r~ ed& Z"e"re"jG d)d ndZ$ ed*e$ e"jG d+ xs g dd, D ]9 Z% e&d-e%jG d. xs d/dd0 d1e%jG d2d/ dd3 ; yyy)5ux
Crawler: 南陵县人民政府 - 建设项目环评文件（报告书、报告表）审批 (Epoint 基层政务公开)
N)datetime timedelta)
BeautifulSoup)urljoin SEARCH_DBz/root/search.dbu 南陵县生态环境分局u9 建设项目环评文件（报告书、报告表）审批zhttps://www.nlx.gov.cnz/wuhu/site/label/8888grassrootsContentPageList67878916603411151truez
yyyy-MM-dd504list1003011) labelNamesiteIdorganIdpageSize pageIndexisDate
dateFormatlengthtypeactionisJsoncatIdzoMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36z!application/json, text/plain, */*z0application/x-www-form-urlencoded; charset=UTF-8)z
User-AgentAcceptz Content-Type)
z.docz.docxz.pdfz.xlsz.xlsxz.pptz.pptxz.zipz.rarz.7zc 2 t dt d| d y )N[z] T)flush)printSITE)msgs /root/gov_crawler/crawl_nlx.pylogr& % s AdV2cU
4( c i t dt | i} t j t |t
dd }|j
S # t $ r}t d| Y d}~yd}~ww xY w)z%Fetch one page of list data from API.r F)dataheaderstimeoutverifyz API error: N) API_DATAstrrequestspostAPI_URLHEADERSjson Exceptionr& )
page_indexr* res r% fetch_list_pager9 ) s` 5h 5 S_ 5D MM'grRW Xvvx k! s 1A A*A%%A*c | j d }|dk r| j d }|dk ry| j dd| }d}| j d| }|dkD r|t | k r|dz
}| ||dz d k( r | j d
| }|dk\ r|d z }@| ||d z d
k( rU| j d| }||kD r>| |d z | j d }|r|dk7 r|}|dz }|dk( rt | ||dz S |dkD r|t | k ry)zAExtract content from nlx.gov.cn - j-fontContent gkwz_contnet div.z"class="j-fontContent gkwz_contnet"r gkwz_contnet < > z z,>,@@r' c t d| } t j |t dd }d|_ |j
|fS # t $ r }t d| d| d|fcY d}~S d}~ww xY w) z&Fetch detail page and extract content.z/grassroots/content/r) F)r+ r, r- zutf-8zFetch error for : N)re r0 rc r3 encodingro r5 r& )
content_idurlr7 r8 s r% fetch_detailrz i sq
J*:,
7C LLgr% H
vvs{ zl"QC 01Sys 1? A'A" A'"A'c n dddd}t j d| }|rI|j d j }t j dd| j }||d< t j d| }|r%|j d j dd |d
< t | }||d < |S ) z.Extract title, date, content from detail page.r< title publish_datecontentz(.*?)r> u* [_\-—]\s*芜湖市政务公开平台\s*$r} zPubDate.*?content="([^"]+)"N
r~ r )researchgrouprU subrQ )rI resultmr} r s r% parse_detailr u s 2"
=F )40A
"Db%PVVX w 0$7A!"!1!1!3CR!8~ %G F9 Mr' c
t j t | dz z
j d }t d| d t d }|s t d y |j
dd }t t d
}||z dz
|z }t d | d | g }t dt |dz d
D ] }t | }|s n|j
dg }|D ]j } | j
d xs dd d }
|
|k r!|j | d | j
dd |
| j
dd | j
dd d l t d| dt | d t j d t dt | d g } t |d D ] \ } } | d rGt! | d \ }
}|
r.t# |
}|d r|d | d< |d r|d | d < |d | d < n d| d < nd| d < | j | d | d | d | j
d d t$ d!t& t& d" | dz d k( rt d#| d$t | t j d d }d }d d l}|j+ t, }|j/ d% |j/ d& |j1 }| D ]V } |j/ d'| d( | d) | d) | d | d | d | d xs dd d* | d+ f |j2 d kD r|dz
}n|dz
}X |j7 |j9 t d-| d.| d/ y # t4 $ r }t d,| |dz
}Y d }~d }~ww xY w)0Nr) days%Y-%m-%dzFull crawl (cutoff: rV r> zFailed to fetch page 1totalr r Total: z , pages: i r* publishDater< r contentIdr} urlDomainpublishDepartment)rx r} datery
departmentz Page rv z items ?zAfter filter: ry rx r~ r r -r} page_urlr~ r site_namecolumnz
Detail: /PRAGMA journal_mode=WALPRAGMA busy_timeout=10000INSERT OR IGNORE INTO gov_raw (site_name, source_url, page_url, title, publish_date, content, summary, category) VALUES (?,?,?,?,?,?,?,?)r r r
DB error: Result: new, skipped) r nowr strftimer& r9 rc intr. rangeminri rF timesleep enumeraterz r r# COLUMNsqlite3connectDB_PATHexecutecursorrowcountr5 commitclose) months_backcutoffr* r page_size total_pages all_itemspgitemsitempub_dateenrichedrJ rI _detailstoredskippedr conncr8 s r% crawl_allr s llnykB.> ??
I
I*
UF
vha() 1 D $% HHWa EHZ()I9$q(Y6K'% + /0IAs;?C01 r" $
D /52s;H& "; / '2. xx R0 "hh':B ?
gbTCJ"23 4
338 FG ??7 #DLL*+LL,-
A
II, %tJ'7j9IM4#7i )_*DS14>
zzA~! 1 & KKMJJL(6(& 23 *QC ! qLG s AM M7M22M7c
z t j t d z
j d } t d| d t d }|sy |j
dg }g }|D ] }|j
d xs d d d
}|| k r!|j
d d }t |d \ }}|r t | nd d d d
} |s t d|j
dd d y|j | d xs |j
dd ||| d t dt t d t j d d}
d} dd l} | j t }
|
j# d |
j# d |
j% }|D ]V } |j# d|d |d |d |d |d |d |d xs d d d |d f |j& dkD r|
dz
}
n| dz
} X |
j+ |
j- t d |
d | d! y # t( $ r }t d | | dz
} Y d }~d }~ww xY w)"N r r zIncremental (since rV r> r* r r< r r r r| z Skip external: r} z (no urlDomain)r r r r r r r r r r r~ r r r r r r )r r r r r& r9 rc rz r ri r# r r r r r r r r r r5 r r )r r* r r r r ry rI r r r r r r r r8 s r% crawl_incrementalr sh llnya 00
:
::
FF
fXQ'( 1 D HHVR EH HH]+1r3B7 f hh{B' tK 01a'+ d#2r^`1a #DHHWR$8#9 I J G_="(= $ i( 66(+
3+. FG ??7 #DLL*+LL,-
A
II, k"D$4d:6Fg ^ 4d9oy/'R#.X@
A zzA~! 1 KKMJJL(6(& 23 *QC ! qLG s AH H: H55H:__main__r> incrementalfullr r r* z r r< r z | r} < )$ )'__doc__sysosr4 r r r0 r r bs4r urllib.parser getenvr r# r re r2 r. r3 ATTACH_EXTSr& r9 rQ rH rz r r r __name__rF argvmoder* rc r itr" r' r% r s ) ( ( (
"))K!2
3$ D #
J+
,,
D1F
a )<A. .]4@:4z z chh-!+388A; D }
q!(,!$! geW 88F#)r2A. YB B}-3Sb9:#bffWR>PQTRT>U=VW X Y
r'