sej d Z ddlZddlZddlZddlZddlZddlmZ dZdZ dZ
dZ dZ d Z
d
d iZ ej Zd Zd
Zd Zd Zd Zedk( r e yy)uO
crawl_yixiu_hjxp.py - 宜秀区-建设项目环评文件审批
CMS: 自定义 (Lonsun?), 列表API分页
列表: /grassroots/column/2000009291?catId=1083811&page=N (20条/页)
详情: /grassroots/2000009291/{infoId}.html
标题:
日期:
正文: div.xxgk-wzcon > div.wzcon.j-fontContent.clearfix > p
N)
BeautifulSoupz/root/search.dbzhttps://www.yixiu.gov.cnu 宜秀区-建设项目环评u 安徽u 环评公示 z
User-Agentz[Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/125.0.0.0 Safari/537.36c t j | t d }d|_ |j S # t
$ r}t
d| Y d }~y d }~ww xY w)N< )headerstimeoutzutf-8zERR:)SESSIONgetHEADERSencodingtext Exceptionprint)urlres %/root/gov_crawler/crawl_yixiu_hjxp.pyr
r
! sL KKWbK 9
vv
QCjs .1 AA
Ac g }t | d }|j dd }|s|S |j dd D ] }|d }|j d s t |z }|j d
d xs |j
d }|sKd
|v sPd|vsU|j d }d }|r7t j d|j
} | r| j d }|j |||f |S )z Extract items from list page html.parserdiv
lm_main_listsclass_aT)hrefr httptitle stripz/grassroots/2000009291/columnspan(\d{4}-\d{2}-\d{2}) ) r findfind_all
startswithBASE_URLr
get_text find_nextresearchgroupappend)
htmlitemssouplist_divr r r date_spandate_strms
r
parse_listr6 + s E } -Dyyy7H
s
.
2yv&d?Dgr"8Ur s
% *
o (
0*-Z&R/"d zF r7