Sej d Z ddlZddlZddlZddlZddlZddlZddlmZ dZ dZ
dZ dZ dZ
d Zd
Zd Zd d
ddddddZddiZd Zd Zd Zd Zd Zedk( r e yy) u:
crawl_shangyu_sthj.py - 绍兴市上虞区-生态环境
CMS: JCMS (Hanweb, API分页)
API: /api-gateway/jpaas-publish-server/front/page/build/unit (GET, pageNo=N)
27条/页, 2456条, ~91页
详情: /col/col{id}/art/YYYY/art_{hash}.html
标题:
日期:
正文: div#zoom > p
N)
BeautifulSoupz/root/search.dbzhttps://www.shangyu.gov.cnz7/api-gateway/jpaas-publish-server/front/page/build/unitu 绍兴市上虞区-生态环境u 浙江u 环境公示 bulidstatic23289nuI0IJgTDxmXUM0arEPbcolumnu 当前栏目列表1a
122960460827) parseTypewebIdtplSetIdpageTypetagIdpageIdrowsz
User-Agentz s E } -Dkk-( . GGCdG #''$y'1 V9Ds# $__V, #~,EE'2&@!**4**@E##$#/D LL$t, - . L c t | d }d}|j dddi }|r |j dd j }d}|j dddi }|r;|j dd }t j
d| }|r|j
d }d}|j d
d } | rg }
| j d
D ]( } | j d } | s|
j | * | j d D ] }
|
j t |
|
rdj |
}n | j d j }|rt j dd| }|r|d d nd}||||fS )Nr/ r8 metaname ArticleTitlecontentPubDatez(\d{4}-\d{2}-\d{2})r divzoom)idpTr9 tablez
) separatorz )
r r< r r: researchgroupfind_allr> r? r joinsub)r urlrA r7
meta_titledate_str pubdate_metarK mrN partsrP txttblsummarys r, extract_detailrd Q sw } -D
E6FN#;8U?
3
u E F5'3u:,c
*+ " N5'
&' HYK
! &tCDzl"QC8 9 JJsO s 'I J)J J__main__)__doc__r rU r$ sysosr bs4r rl r! r" rp rr rq ITEMS_PER_PAGEr r r r- rE rd r r __name__ rF r, r s
'
C -
'
#
P
&&-R&R5"p zF rF