WAF: 阿里云 (浏览器UA)
运行: python3 crawl_putuo.py [--full]
--full: 全量 (2页39条)
不加: 仅第1页 (日跑增量)
N)ThreadPoolExecutor as_completed SEARCH_DBz/root/search.dbuC 舟山市普陀区人民政府-建设项目环评企业信息公示u 舟山市普陀区人民政府zhttps://www.putuo.gov.cnz7/api-gateway/jpaas-publish-server/front/page/build/unit bulidstatic27623wW2GfVyimCCPxV51GOi1columnu 新闻列表null
1229863925) parseTypewebIdtplSetIdpageTypetagIdeditTypepageIdz
User-AgentzoMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36 )g333333?g ?c 2 t dt d| d y )N[z] T)flush)print SITE_NAME)msgs /root/gov_crawler/crawl_putuo.pylogr - s Ai[3%
- c 8 t t }t j | t ddd |d< t
dz dj
d |j D z }t j j |t } t j j |d
}t j |j j }|j! d sg S |d
d S # t" $ r}t% d| d| Y d }~y d }~ww xY w)N )pageNopageSizesearchF) ensure_ascii paramJson?&c 3 | K | ]4 \ }}| d t j j t | 6 yw)=N)urllibparsequotestr).0kvs r
z fetch_list..4 s3 "b41aaS&,,*<*DDc g }t j dt j }|j | D ] }t j d| }|s |j d }t
j |j d j }t j d| }|sx|j d }|j d r|nt |z } |j | ||f |S )Nz(.*?)z%]*title="([^"]*)" r z (\d{4}-\d{2}-\d{2})http) recompileDOTALLfindallr# grouphtmlmodunescapestrip
startswithBASE_URLappend)
html_textrA patternlia_murl_pathtitledate_mdatefull_urls
r
parse_listri A s Ejj*BII6Gooi(
.ii@"E 99Q< 1 !3!3!56>C ||A ' 2 26 :88@S
ht,-
. Lr c F t j j | t } t j j |d }|j j
}d }t j d|t j }|r |j d j }nFt j d|t j }|r |j d j }|} t j d| }|rKt j t j dd |j d j }
|
r|
} |r%t j d
d | j }| | ||fS # t $ r}| ||d fcY d }~S d }~ww xY w) Nr2 r4 r5 z/(.*?)
\s*(.*?)