ujb d Z ddlZddlZddlZddlZddlZddlmZ dZdZ dZ
dZ dZ d d
iZ
dad Zd Zd
Zd Zd Zd Zedk( r e yy)us
crawl_zhuhai_spqgs.py - 珠海市生态环境局-审批前公示
CMS: NFCMS (广东政府网站群)
列表: /zxfw/xmgsgg/spqgs/index.html (page1) → index_{N}.html (pages 2+, 96页)
div.wendangListC > li > a[href] + date prefix
详情: /zxfw/xmgsgg/spqgs/content/post_XXXXXXX.html
标题:
日期:
正文: div.xxxq_text_cont > p
N)
BeautifulSoupz/root/search.dbz https://ssthjj.zhuhai.gov.cnu 珠海市-审批前公示u 广东 环境公示z
User-AgentzoMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36c t 7t j a t j j t
t j
| d }d|_ |j S # t $ r}t d| Y d }~y d }~ww xY w)N< timeoutzutf-8zERR:) SESSIONrequestsSessionheadersupdateHEADERSgetencodingtext Exceptionprint)urlres '/root/gov_crawler/crawl_zhuhai_spqgs.pyr r " so ""$ w' KKRK (
vv
QCjs )A) ) B
2BB
c b | dk( rt d}t | S t d| d}t | S )N z /zxfw/xmgsgg/spqgs/index.htmlz/zxfw/xmgsgg/spqgs/index_z.html)BASE_URLr )page_nor s r fetch_list_pager 0 s? !|
78 s8O
3G9EB s8O c ~ g }t | d }|j dd }|s|S |j d D ] }|j dd }|s|j d d
j }|rd |vrA|j d }t
j d
| }|r|j d nd
} t
j dd
| j }
|
s|j d }
|j d rn&|j d r
t |z }n t d| }|j ||
| f |S )z)Extract (url, title, date) from list page html.parserdiv wendangListCclass_liaT)hrefr& post_strip(\d{4}-\d{2}-\d{2})r z^\d{4}-\d{2}-\d{2}http/)
r findfind_allr r* get_textresearchgroupsub
startswithr append) htmlitemssoupwlr$ r% r& txt
date_matchdate_strtitles r
parse_listr? 8 s7 E } -D
5 0B
kk$ . GGCdG # uuVR &&(wd* kkk%YY 5s;
*4:##A&" ,b#6<<>JJTJ*E ??6 "
__S
!d?D Zq'D
dE8,-3.6 Lr c t | d }d}|j dddi }|r |j dd j }d}|j dddi }|r;|j dd }t j
d| }|r|j
d }d}|j d
d } | r*g }
| j D ] } | j d
k( r'| j d } | s'|
j | 9| j dk( r t | }
|
j |
e| j d
k( su| j D ]b }|j d
k( r'|j d } | s'|
j | 9|j dk( sI|
j t | d |
rdj |
}n | j d j }|rt j dd| }|r|d d nd}||||fS )Nr r' metaname ArticleTitlecontentPubDater+ r r xxxq_text_contr" pTr) tablez
) separatorz )r r. r r* r1 r2 r3 childrenrB r0 r6 strjoinr4 )r7 r r9 r>
meta_titler= pubdate_metarD mtcpartselr; tbl_htmlchildsummarys r extract_detailrX _ s } -D
E6FN#; r= rD rW page_urlnew_idr ridsites r
save_to_dbrs s ??7B? /D
AHI7A 53x'8 5
IIAH; Ozz|
II? 5)Xw>8U r= detail_htmldtddrD rW saveds r mainr s
$
$I;f1E
$
FF
Q=UV
r s
) ' D $N2-j&R)"X zF r