j 8 d Z ddlZddlZddlZddlZddlZej j dej j ej j e ddl m
Z
dZdZdZdZd dZd
d Zd
Zed k( r ej, e yy)u8
crawl_yichang_fgw.py - 宜昌市发改委-行政服务中心窗口公告 (fgw.yichang.gov.cn)
站点: 无防护纯静态 (Bootstrap 页面)
列表: list-40625-{N}.html (N=1..6), 每页 20 条
列表项:
详情: content-40625-{id}-1.html
meta ArticleTitle/PubDate/ContentSource, 正文 div.txtcontent-div (截断 )
内容: 项目审批核准和备案公告 (月度, 审批类+备案类, 含项目代码)
N)push_to_searchdbzhttp://fgw.yichang.gov.cn40625u1 宜昌市发改委-行政服务中心窗口公告zoMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36c t j j | dt i }t j j || 5 }|j j
dd cd d d S # 1 sw Y y xY w)Nz
User-Agent)headers)timeoutzutf-8replace)errors)urllibrequestRequestUAurlopenreaddecode)urlr reqresps &/root/gov_crawler/crawl_yichang_fgw.pyfetchr sg
..
|R.@
AC W 5 =yy{!!')!<= = =s !A44A=c
g }t }t d| dz D ] }t dt d| d} t | }t j d| }d }|D ] \ } }
} | j d
r| nt | z } t j d d |
j }
|
sK| |vsP|j | |j | |
| j d d
d |dz
} t
d| dt | d| dt | d t | d
k st
d| d n t
dt | d g }t! | D ] \ }}t
d|dz dt | d|d d d d t |d }|d }
t j" d | }|r>|j% d j r |j% d j }
|d }t j" d!| }|r7t j" d"|j% d }|r|j% }d }t j" d#| }|r |j% d j }|s7t j" d$| }|r |j% d j }|j s
t
d% {t j d&d't z dz | }t j d(d)t z dz | }|j |
|d |t& |d d* t) j* d+ t
d,t | d- |r t- |d. t | S # t
$ r)}t
d| dt | d d Y d }~ wd }~ww xY w# t
$ r%}t
d t | d d Y d }~d }~ww xY w)/N z/list--z.htmlz
[LIST ERR pz] d z\s*([^<]*)
r httpz
)r titledateu 第u 页: 解析 u
条, 新增 u (累计 )z [pu ] 列表结束u [LIST] 去重后 u 条z [/r ( z...r z [DETAIL ERR] z/([\s\S]*?)z>([\s\S]*?)
\s*\s*u [SKIP] 正文空zhref="/zhref="zsrc="/zsrc=")r r pub_date site_namecontentsummaryg333333?u 共 u
条有效 yichang_fgw)setrangeBASELIST_IDr Exceptionprintstrrefindall
startswithsubstripaddappendlen enumeratesearchgroup SITE_NAMEtimesleepr )pages items_allseenpgr htmlerowsnew_rowsutdu2r recordsidxitdhmr" dmr$ s r runrN s I 5DAuqy! fWIQrd%0 :D zz
h
GAq!ll6*qBFF="a0668E4 eQWWYsPR^!TUA
bT c$i[
hZyQTU^Q_P``abc t9r> DN+ , -0
I /t
45 G Y' &R
CE7!C N+2bk#2.>-?sCD r%y!B 7 IIH" M !!#GGAJ$$&Ef: IICR H /rl s ) ( ( 277??277??8#<= > ("
? v= K\
z CHHTV r\