_3yjR" x d Z ddlZddlZddlZddlZddlZddlZddlmZm Z m
Z
ddl m Z ddl
Z
ddlmZ ej j# dej j% ej j' e ddlmZmZ dZdZd Zd
Z ej8 e j: e
d z
j= d
Z ddiZ ejB dejD Z# ejB d Z$ ejB dejD Z%d Z&d Z'd Z(d Z)d"dZ*e+dk( r ej
Z,dZ-dZ.dej^ v r1 e0ej^ ej^ jc d dz Z-d ej^ v rd Z. e*e-e. e2d ej
e,z
d d! yy)#u 栾川县人民政府-重点领域信息公开-环境保护 爬虫
站点: https://www.luanchuan.gov.cn/zfxxgk/fdzdgknr/zdlyxxgk/hjbh/
列表: div.newsitem > div.newsitemtitle > a (标题含
需清洗) + div.newsitemtime > b (日期)
分页: 第1页 /, 第N页 index_{N-1}.html (每页20条, 共173页)
详情: 标题 h2.newstitle, 日期 div.htime, 正文 div#contents.newscontents
正文保留 HTML; 附件链接 urljoin 绝对化; 标题清洗(剥实体/零宽/装饰符/内嵌
)
N)datetimetimezone timedelta)urljoin)
BeautifulSoup)push_to_searchdbnormalize_pub_dateu 栾川县-环境保护u 环境保护z https://www.luanchuan.gov.cnz:https://www.luanchuan.gov.cn/zfxxgk/fdzdgknr/zdlyxxgk/hjbhiG )daysz%Y-%m-%dz
User-AgentzoMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36z2\.(pdf|doc|docx|xls|xlsx|ppt|pptx|rar|zip|wps|et)$z>[\u200b\u200c\u200d\u200e\u200f\u2060\ufeff\u00ad\u3000\u00a0]u (_栾川县人民政府|_栾川县人民政府$|【[^】]*】|《[^》]*》|\[[^\]]*\]|\s*[-_—|｜·•]\s*栾川县人民政府\s*$)c f | syt j t | }t j d| }t j dd|t
j }t j dd| }t j d| }t j dd| j }|S )ul 清洗标题: 剥HTML实体 -> 去零宽/装饰符 -> 内嵌
转空格 -> 去残留标签 -> 压缩空白 z )flags<[^>]+>\s+) html_modunescapestr
ZERO_WIDTH_REsubreIDECOR_REstrip)rawts )/root/gov_crawler/crawl_luanchuan_hjbh.py clean_titler + s #c(#A"a A
|S!2440A
z2q!A RA
vsA $$&A H c | dk( r
t d}nt d| dz
d} t j |t d }d|_ |j
S # t $ r }t d| d | d
| Y d }~y d }~ww xY w) uB 第1页 -> LIST_BASE/, 第N页(N>=2) -> LIST_BASE/index_{N-1}.html /z/index_z.html headerstimeoututf-8z [ERROR] page r
: N) LIST_BASErequestsgetHEADERSencodingtext Exceptionprint)page_numurlres r
fetch_pager4 8 s 1} 1o 78a<.6 LLgr :
vv
z3%r!56s .A A6A11A6c t | d }g }|j d D ] }|j d }|s|j d xs dj }|rt j d| sSt t dz | }t |j d xs |j d d
}|j d }|r|j d
nd} t | } |j ||| f |S )
uC 解析列表: div.newsitem > .newsitemtitle > a + .newsitemtime > b html.parserz div.newsitemzdiv.newsitemtitle ahrefr z /\d{4}/\d{2}-\d{2}/\d+\.html$r! titler
Tr zdiv.newsitemtime b)
r select
select_oner* r r searchr r( r get_textr append)
htmlsoupitemsitemar7 r1 r8 r date_texts
r parse_itemsrE G s } -D E N+
. OO 1 2 f
#**,299%EtL i#ot,AEE'N IajjDj.IJ OO 0 1./AJJTJ*R &y1
eS),-
. Lr c F t j | t d }d|_ |j }t |d }d}|j d d
}|r t |j d d
}|s1|j dd }|r t |j d d
}|sT|j d }|rAt |j d d
}t j dd| j }d} |j dd }
|
rit j d|
j d d
} | r@t | j! d d| j! d d| j! d } |j dd xs |j dd } d}
g }| rH| j# g d D ] }|j% | j# d D ] }|j d xs dj }|s)|j' d r;t) | | |d < t* j | s`|j- |j d
xs d t) | | d! | j# d" D ]7 }|j d# xs dj }|s)t) | | |d#< 9 dj/ d$ | j0 D j }
|| |
|fS # t
$ r!}t
d| d| dddg fcY d}~S d}~ww xY w)%uE 抓详情页: 标题/日期/正文(保留 HTML)/附件(绝对化)r" r# r& z [ERROR] detail r' Nr6 r h2 newstitle)class_r
Tr9 divhhr8 u$ _环境保护_栾川县人民政府$htimeu) (\d{4})[-年/.](\d{1,2})[-月/.](\d{1,2})r - contents)id newscontents)scriptstyleiframerC r7 )z javascript:zmailto:u 附件)namer1 imgsrcc 3 2 K | ] }t | y w)N)r ).0cs r z fetch_detail.. s ?Q#a&?s )r) r* r+ r, r- r. r/ r findr r= r r r r< r groupfind_all decompose
startswithr ATT_REr> joinrP )r1 r2 r? r3 r@ r8 rG rK title_tagdate_strrL m content_divcontent attachmentstagrC r7 rW rX s r fetch_detailrk \ s $ LLgr :
vv
} -D
E
4 ,B BKK4K 89
YYuTY
*
Ct <=E IIg& 2 23d 2 CDEFFBBNTTVE H IIeGI ,E IIBENNSV^bNDc d )QWWQZL!''!* Qqwwqzl*STH ))Ej)1\TYYu^Y5\KGK ''(EF C MMO %%c* AEE&M'R..0DDOO,F G #C .& ==& && !
6 B( &sD 1( ''. /C775>'R..0C $S# .E
/
''?+*>*>??EEG (G[ 00m $
!#b,-T4##$s /K6 6 L ?LL L r Fc t dd dt dt dd g }t }t d| dz D ] }t d| ddd
t | }|s
t d nt
| }|s
t d nt t | d
d }|D ] \ }} }
| |v r |j | |
r
|
t k r)d}|rIdd l
} | j d } | j d| f j }
| j |
rv|j || |
f |r
t d nt! j" d t dt | d g }d}d}d}t% | D ]3 \ }\ }} }t d|dz dt | d|d d ddd
t' | \ }}}}|xs |}|r|j) s|dz
}t+ j, d| r|dz
}|t | z
}t+ j. d d|xs d j) }t+ j. d d| d d }|j t | | |xs |||xs d t |t |rt1 j2 |d nd d!
t d" t! j" d 6 |r t5 |d# t d$t | d%| d&| d'| |S )(N
z8========================================================z (z)
r z --- Page z ---r
T)endflushz
fetch failemptyz itemsFr z/root/search.dbz(SELECT 1 FROM gov_raw WHERE source_url=?u 全部早于3年截断, 停止g333333?u 📋 待抓详情: u 条z [r! z] . z...z]r r r i, ) ensure_ascii)
site_name
source_urlr1 pub_dater8 rh categorysummarytagsri OKluanchuan_hjbhu
✅ 完成! 入库 u 条 | 空正文 u
| 含 u
| 附件 )r/ SITE_NAMECATEGORYsetranger4 rE lenaddTHREE_YEARS_AGOsqlite3connectexecutefetchonecloser> timesleep enumeraterk r r r< r jsondumpsr )pages incremental all_itemsseenpager? rA exhaustedr8 r1 rD s3connexistsresults
stat_emptystat_pstat_atti list_datedtddrh ri ftitlerw s r mainr s Bxj9+RzXJ
?@I 5Da#
$t$#T:$ , D! 'N
U F#$ %* 6 !E3 d{ HHSMY 8 I$zz"34 &PSVRXYbbd
eS) 4 5 6 4 5
3A D
I /t
45GJ
FH&/ &: " "E3
AE7!C N+2eCRj\=3dS'3C'8$Bugmmo !OJ
99Z ) aKFC $$&&S'-R8>>@&&g.t4" i }" JU4::kF[]
d
314 "23 !#g, /@ JW]V^^hiqhr
st Nr __main__z--pagesz
--incrementalT)r r zTime: z.1fs)r F)3__doc__sysosr r r r? r r r r urllib.parser r) bs4r pathinsertdirnameabspath__file__ crawler_libr r r{ r| BASE_URLr( nowutcstrftimer r+ compiler rb r r r r4 rE rk r __name__t0r r argvintindexr/ r r r s 1 0 0 2 2 277??277??8#<= > < $ ) H 8<< - w0GGQQR\] D
I244 P
E
2:: ODD
*<1~JZ z B
EKCHH CHHSXX^^I 6 :;<#((" u+. F9499;#C(
*+ r