ڵj' d Z ddlZddlZddlZddlZddlZddlZddlZddlmZ ddl m
Z
ddl m Z dZ
dZdZd Zd
Zd Zd Zd
dddZd Zd Zd ZddZd Zd Zd Ze dk( r e yy)u
Crawler for 泉州市泉港区人民政府 - 项目环评 (qg.gov.cn)
CMS: TRS (填选系统)
List: div.gl_list > ul > li > span(date) + a[href]
Pagination: index_0.htm ~ index_29.htm (15/page, 30 pages ~450 items)
Detail: div.xl_tit (title), div.xl_tit1_l > span:nth(1) (date), div.TRS_Editor (content)
div.szf_list2 ul li a (attachments)
Usage:
python3 /root/gov_crawler/crawl_qg_xmhp.py
N)datetime)
BeautifulSoup)urljoinz/root/search.dbu 泉港区-项目环评u 项目环评u 福建zhttp://www.qg.gov.cnz/zwgk/zdxxgk/hjbh/xmhp zoMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36z?text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8zzh-CN,zh;q=0.9,en;q=0.8)z
User-AgentAcceptzAccept-Languagec T | dk( rt t dS t t d| dz
dS )zpage_num is 1-indexed /z/index_z.htm)BASE_URLLIST_DIR)page_nums "/root/gov_crawler/crawl_qg_xmhp.py get_list_urlr % s5 1}H:Q''Zz! D 99 c : t | } t j |t d }d|_ t |j d }|j dd
}|s t d | t j g S g }|j d D ] }|j d
D ] }|j d } |j d }
| r| j d s9| d j } t || } | j dd j xs | j d }
|
r|
j d nd}|
s| s|j! |
| |f |S # t $ r-}t d| d| t j g cY d }~S d }~ww xY w)N headerstimeoututf-8 html.parserz Error fetching : filedivgl_listclass_z No gl_list found on page ulliaspanhreftitle Tstrip)r requestsgetHEADERSencodingr text Exceptionprintsysstderrfindfind_allr' r get_textappend)r
urlrsouper itemsr r r! r" r# page_urlr$ date_strs r extract_list_pager< , sv
x
C LLgr :
QVV]3
))E))
,C
+H:6SZZH Ell4
:++d# :B A776?DAEE&MV9??$D sD)HEE'2&,,.H!**4*2HE48t}}4}0bH eXx 89 :
: L3
!#b,3::> s 9E$ $ F-"FFFc t j | t d }d|_ t |j d }d}|j d d
}|r|j d
}|sE|j d }|r2|j d
j d d j }d}|j dddi }|r'|j d r|d j d d }|sj|j d d } | rU| j d }
|
D ]? } | j d
} t j d| }
|
s.|
j# d } n |j d d }|s|j d d }d}|r|j d d g D ] }|j% g }|j& D ]; }|j( s|j( d k( r|j+ t- | ;|j( d k( r|j d d gd!" r}|j d d gd " D ]d }|j( d k( r|j+ t- | -|j/ d r?|j d
}|sT|j+ | f |j d
}|s|j+ | |j( d k( s|j d
} | s+|j+ | > d#j1 | }t j2 d$d#| j }g }t5 }|j d d% }|r|j d&d ' D ] }|d( }t7 | | }|j d
xs& |j d) d* j d+ d }||vsS|j9 | |j+ ||d, |s{|d-| d.| d/z
} |r|j d&d ' ng D ] }|d( }t j d0|t j: s.t7 | | }|j d
xs& |j d) d* j d+ d }||vsy|j9 | |j+ ||d, ||||fS # t $ r1}t d| d| t j d d d g fcY d }~S d }~ww xY w)1Nr r r z fetch_detail error r r r r% r xl_titr Tr& r$ -r metanamePubDate)attrscontent
xl_tit1_lr" z(\d{4}-\d{2}-\d{2})r
TRS_Editorxl_con1scriptstyletablepF) recursivez
z\n{3,} szf_list2r! )r# r# r
?)rA r5 u
附件：z
z&\.(pdf|doc|docx|xls|xlsx|rar|zip|txt)$) r( r) r* r+ r- r. r/ r0 r r, r1 r3 splitr' r2 researchgroup decomposechildrenrA r4 str find_parentjoinsubsetr addI)r5 r6 r8 r7 r$ tit_divtr; r@ tit1spanssptxtmeditorrD spartschildinner attachmentsseenatt_diva_tagr# full_urlrA s r fetch_detailro P s $ LLgr :
/D
Eiihi/G t , IIg JJTJ*005a8>>@E H 99VFI#69 7D # ?((*3B/ yy{y3 MM&)E
kkk-II 4c: wwqzH
YYu\Y
2F 53G
(G!45 A
KKM __ &E::zzW$ SZ(s"::ug .%:@!& g$ !O 0 ::0 !LLU 4!&!2!27!; %T :A % Q
0 T2A Qu$nn4n0LL%- &. ++e$ ffY0668G K 5Diiki2G %%c%5 VE=D sD)H>>>-RC 1D1J1J31OPQ1RDt#"""D#BC!:8*BtfH UUG V 5; 4 0 DV}
99>bdd K sD)H>>>-RC 1D1J1J31OPQ1RDt#"""D#BCD (G[ 00G $
%cU"QC0szzBT4##$s #R) ) S#2&S S# S#c
l t d|dz D ] } | c S y
# t j $ r}dt | v rC||k r>t d| d| d| d| d t
j t j | n't d| d | t
j Y d }~ y
Y d }~d }~ww xY w) Nr lockedz z: locked (attempt r
z ), retry in zs...r z failed: F) rangesqlite3OperationalErrorrW r. r/ r0 timesleep)fndesc max_attemptsdelayattemptr8 s r
with_retryr| s L1 ,- 4K '' 3q6!g &<4& 27)1\N,W\V]]abilisist
5!4& !-CJJ? " s B3A1B..B3c t j t d } | j d | j d | S )Nr )r zPRAGMA journal_mode=WALzPRAGMA busy_timeout=30000)rs connectDB_PATHexecuteconns r init_dbr s1 ??7B /DLL*+LL,- Kr c
|r|d d n|}t j dd| j }|rt j |d nd} | j dt |||j |||t |t f
}|j dkD S # t $ r(} t d | t j
Y d } ~ yd } ~ ww xY w) N z\s+ F) ensure_asciiz[]zINSERT OR IGNORE INTO gov_raw
(site_name, source_url, page_url, title, publish_date, summary, content, category, attachments, group_name)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)r z DB error: r )rR rZ r' jsondumpsr SITE_NAMECATEGORYGROUProwcountr- r. r/ r0 )
r r$ r: publish_daterD rj summaryatt_jsoncurr8 s
r save_articler s &gdsmEGffVS'*002G>Itzz+E:tH ll
8 (EKKM<
gx5
:
||a
QC szz2s AB C B<<Cc t } t t j dkD r t t j d } t dt d| t j t d}d}t j }fd}t dt j t |dt t d
t t j t d| dz D ] }t | }|s"t d | d t j nV|t | z
}d}|D ] \ }} }
|
r|
d
k rj d| f } | j r3t | \ } }
}}| xs |}|
xs |
}|xs d| d| d}|xs g }t! || ||| r
|dz
}|dz
}t j" d t t j |z
}| |z
}|dkD r-|dkD r(t |t% d|t% d| z z dz nd}t d | d| d| d| d| d|dz d|dz dd| dt j |d z dk( svj' j' t t j |z
}j) t d | d | d |dz d |dz dd! t j y # Y xY w)"Nr z Scraping z
, max pages: r r c & j y )N)commitr s r do_cleanzmain..do_clean s
r z Cleaning old data...zClean )rx z Cleaned old data for z Page z: 0 items (end)z
2005-01-01z)SELECT id FROM gov_raw WHERE page_url = ?[z]()g333333?< r
z: +z new, z total, :02dz elapsed, ETA min z
Done! Found: z, New: z, Time: rd rf ) TOTAL_PAGESlenr/ argvintr. r r0 r ru r| rr r< r fetchonero r rv maxr close) max_pages total_new total_found
start_timer pnarticles batch_newr$ r: dater det_titledet_date det_contentdet_atts final_title
final_date
final_content
final_attselapsed
pages_leftetar s @r mainr s I
388}q CHHQK(I
Ii[
i[
9
K 9DIKJ
"4xyk 23 #I;
/cjjAAy1}% '$R( GB4/cjj A s8}$ %- !E8T|+,, JXKXC||~9Eh9O 6Ixh#,uK!)TJ ' IQ{m2hZq+IM! RJD+x]T^_Q Q JJsO% ( diikJ./ ^
ELq[UWZ[U[c*s1b3q'?&: ;;b@Aab
t1YKs9+V 1[M"
Qwrz# . /E %(JJ 0
6Q; KKMO'R KKM$))+
*+GJJL OK= {(7B;-qQXY[Q[\_P``a
bilisisty s K& &K+__main__)zDB oprE r ) __doc__r( rR r rs ru osr/ r bs4r urllib.parser r r r r r r r r* r r< ro r| r r r __name__ r r r s
2 1 1 1
$ ! # DO0 : Hg1X (AuH zF r