uj8 d Z ddlZddlZddlZddlZddlmZmZmZ ddlZddl Z ej j dej j ej j e ddlmZ e j" e j$ j& dZdZdZdZd d
iZd Zd Zd
Zd Z e dk( r8 ej Z e e d ej e z
dd yy)u
池州市贵池区 - 环评公示 (ahgc.gov.cn)
列表: /Jczwgk/opennessList/257/111001001/page_N.html
详情: /Jczwgk/opennessShow/N.html
正文: div.m-dttexts#zoom (inside div.m-detailbox)
N)datetimetimezone timedelta)push_to_searchdbu 贵池区-环评公示zhttps://www.ahgc.gov.cnzGhttps://www.ahgc.gov.cn/Jczwgk/opennessList/257/111001001/page_{n}.html z
User-Agentz Mozilla/5.0c r t j | t dd }d|_ |j S # Y y xY w)N F)headerstimeoutverifyzutf-8)requestsgetHEADERSencodingtext)urlrs $/root/gov_crawler/crawl_ahgc_hpgs.pyfetchr s5 LLgr% H
vv
s /2 6c ddl m} g } || d }|j d D ]Y }|j d }t | dk r$g }|D ] }|j d }|j dd
} |s*| s-|j d s@| j
d d
sS|j d }
t j d|
s||j | |
f t | dk\ s|D ] \ } }
| j
d d
j } | j
dd
xs | j d } | r| rt | dk r_| j d s t | z } |j | | |
f |S |S )u
从列表页 HTML 提取 (title, url, date) 元组
结构:
r
BeautifulSoup html.parserulli spanaT)hrefstripr z\d{4}-\d{1,2}-\d{1,2}titler http)
bs4r find_alllenfindget_textr rematchappendr!
startswithBASE_URL)
htmlr itemssoupr lis candidater r r date_textr r# s
r
parse_listr5 st
" E } -DmmD! kk$ s8a< 5B776?D$'Admm$m 7AEE&"Q )
7 9uuVR(..0gr*Dajjtj.D5CJN v.#d?D eT9 56
7
L54 L c t | }|syddlm} ||d }i }|j dd }|r|j d |d
< |j d
sO|j d
}|r<|j d }t
j d d | j }||d
< |j d
ddi }|r@|j d r/t
j d|d } | r| j d |d< |j d s,t
j d| } | r| j d |d< |j dd }
|
s|j dd }
|
s|j dd }
|
s|j dd }
|
s|j dd }
|
rt |
} t
j d d | t j t j z } t
j d d | t j t j z } | j |d< |j d
|j d |j d fS ) u3 访问详情页，返回 (title, content, pub_date))NNNr r r h1zu-lgtit)class_Tr r# u= [-—].*?(?:池州市|贵池区).*?(?:人民政府|政府).*$r" metanamePubDate)attrscontentz(\d{4}-\d{1,2}-\d{1,2}) publish_dateu, 发布时间[：:]\s*(\d{4}-\d{1,2}-\d{1,2})divz m-dttextsz
j-fontContentzoom)idz m-detailboxz]*>.*?)flagsz]*>.*?)r r% r r( r) r r* subr! searchgroupstrDOTALLI) r r/ r r1 resultr8 tagtr: mdetail content_htmls r fetch_detailrQ E s& :D ! } -D
F
4 *B ++D+1w ::g ii 4 (AWY[]^_eegA F7O 99VFI#69 7D # II0$y/ B %&WWQZF> " ::n % IIEt L %&WWQZF> " YYu[Y
1F 5 9 5V, 5 7 53
6{ vv :B TVT]T]`b`d`dTde vv 8"lRTR[R[^`^b^bRbc ( . . 0y ::g
9 5vzz.7Q QQr6 c t dd dt dd g } t dt dz D ] }t j dt
| }t d| ddd
t | }|s
t d nEt | }|s
t d n+t d
t | d | j | t dt | d g t }}t | D ]i \ }\ }}} ||v r|j | t d|dz dt | d|d d ddd
t | \ }
} } |
xs |}
| xs | }|rt j dt
| }|r\t# |j% d t# |j% d t# |j% d }}}|dd|dd|d}nd}| xs d}t j& d d| j) }t j& d d| d d }|j+ t |
||||t d t d t- j. d! l |r
t1 |d" n t d# t d$t | d y )%N
z2==================================================u
🏠 r? z{n}u
📄 第 u 页... T)endflushu ❌ 请求失败u 0 条（可能已到底）u ✅ u 条u
📊 列表总计: z [/z] 2 z...z(\d{4})-(\d{1,2})-(\d{1,2}) r 04d-02dr" z<[^>]+>z\s+i, ) site_namer# r r> pub_datesummarytagsu ✅g333333? ahgc_hpgsu ⏭ 无数据，跳过推送u
✅ 完成! 共 )print SITE_NAMErange MAX_PAGESLIST_URLreplacerH r r5 r' extendset enumerateaddrQ r* r+ intrG rE r! r, timesleepr )all_listpager r/ r0 all_itemsseenir# list_datedtr> date final_title
final_daterN ymod content_textr_ s r mainr} } s Bvhgi[6(
34 HaQ ' uc$i0
D6)s$?Sz $ % 4 . /
SZL%&
!#h-
56 #%tI&/&9 #" "E3 $;
AaC5#h- 5":,c:4P ( -GTkE &Y
7Z IA qwwqz?C
OS_q2 !#wa3xq3 8
}" &&S,7==?&&g.t4" #"
e
3G#L K0
./ s9~ .d
34r6 __main__u ⏱ 耗时: z.1fs) __doc__sysosr* rm r r r r
urllib3pathinsertdirnameabspath__file__ crawler_libr disable_warnings
exceptionsInsecureRequestWarningrc r. rf re r r r5 rQ r} __name__t0rb r6 r r s 2 2 277??277??8#<= > ( ++BB C $ $ T
'#L5RpB5J z BF LR ,A
./ r6