jP d Z ddlZddlZddlZddlZddlZddlZddlmZ ej j dd Z dZ dZ
e
dz Zd Zd
Zd d iZh d
Zd ZddZd Zd ZddZedk( r] ej2 Zej7 de d ej; Z ee j> \ Z Z!Z" e#de de! de" yy)u
岫岩满族自治县-生态环境
URL: http://www.xiuyan.gov.cn/xyxzf/zwgk/hjbh/glist.html
分页: glist.html(第1页) + API: cms.anshan.gov.cn/html/page.xhtml (第2~26页)
共26页/386条，内容含表格+PDF附件
N)
BeautifulSoupDB_PATHz/root/search.dbu 岫岩县-环境保护zhttp://www.xiuyan.gov.cnz/xyxzf/zwgk/hjbh/glist.htmlzYhttp://cms.anshan.gov.cn/html/page.xhtml?s=XYXZF&o={}&p=157404820375520&c=157404820375520 z
User-AgentzoMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36>) pdddldth1h2h3h4h5h6hrlioltdthtruldivnavpreformmainasidetabletbodytfoottheadcenterdialogfigurefooterheaderhgroupaddressarticlecaptiondetailssectionfieldset
blockquote
figcaptionc | yddl }ddlm g fd | dj }|j dd| }|j dd | }|j S )
ul 块级边界出换行、行内节点直接拼接、
出换行（≈ 浏览器看到的换行结构）。N r )NavigableStringc F | j D ] }t | rj t | *t |dd dk( rj d Lt |dd t
v r+j d | j d | y )Nnamebr
)children
isinstanceappendstrgetattr _BLOCK_TAGS)nodechr1 outwalks !/root/gov_crawler/crawl_xiuyan.pyr? zbody_text..walk0 s --
B"o.
3r7#VT*d2
4 VT*k9
4 R
4 R
z[ \t\r\f\v]*\n[ \t\r\f\v]*r5 z\n{3,}z
)rebs4r1 joinsubstrip)el_retr1 r> r? s @@@r@ body_textrJ ( sb z#
C H
A -tQ7A 61%A 779rA c t t j | t | }d|_ t |j
d S )N)headerstimeoutzutf-8z html.parser)requestsgetHEADERSencodingr text)urlrM rs r@ get_souprU C s, S'7;AAJ //rA c * g }| j d D ]| }|j d }|s|j dd }d|vr.|j d }|j dd
}|r|j d nd}|j |||f ~ |S ) u2 提取列表页: ul > li > div.time + div.name > azul liahrefr0 z
html/XYXZFTrF r time)class_)selectfindrO get_textr8 )soupitemsr rW rX titletime_divpub_dates r@
parse_listrd I s Ekk'"
. GGCL uuVR t #
&775704<8$$4$0"
dE8,-
. LrA c d}d}d}d}g }| j dddi }|r$|j d r|d j }| j dddi }|r$|j d r|d j }| j d } | s| j d }
|
r|
n| } | j d
D ] } | j d d } | j d s'| j
d
s t | z } |j | j d xs t j j | | d g }
| j D ] }|j dk( r,|
j t | |
j d >|j dk( r,|j d }|sb|dvsg|
j | y|j dk( s|j d }|s|
j | dj! |
}|j s t# | }|s|j% d }|r|d }||||fS )u1 提取详情页: div.info.hwq-info-article 内容r0 metar3 ArticleTitle)attrscontentPubDatez.hwq-info-article-centerz.info.hwq-info-articlerW rX )z.pdfz.docz.docxz.xlsz.xlsxz.zipz.rarhttpTrY )r3 rS r r )u 点击：u 次r r5 r )r] rO rF
select_onefind_allendswith
startswithBASE_URLr8 r^ ospathbasenamer6 r3 r9 rD rJ split)r_ rS ra rc sourceri attachmentsmeta_tmeta_pdr! infoa_taga_hrefpartschildtxtliness r@ extract_detailr Z s/ EH
FGK YYvfn%=Y
>F
&**Y'y!'')iivy&9i:G7;;y)9%++- __ 7
8F 78 4 % 62& ??U V$$V,!F* T2Nbgg6F6Fv6N
E " :: LLU $ LL
ZZ3
..t.,Cs"66 S!
ZZ5
..t.,C S! " ii G ==?F#
d# !HE (G[ 00rA c t j t }|j }| r| nt }g }t d|dz D ] }|dk( rt }nt j | }t d| d| d|dd t | }t | }t dt | d |s
t d n|j | d
}
d
} t |d D ] \ } \ }
}}|j! d |
t" f |j% r"t d | dt | d
|d d d Tt d | dt | d|d d d t |
}t' ||
\ }}}}|s|}|s|}|rt) j* |d nd}|j! d|
t" |||
||f |j- |
dz
}
t d |j/ |
t | | fS # t $ r} t d | Y d } ~ @d } ~ ww xY w# t $ r } | dz
} t d| Y d } ~ Zd } ~ ww xY w)N u [列表] 第/u 页: ...iz -> u 条u -> 空页，停止u
-> 失败: r z7SELECT id FROM gov_raw WHERE page_url=? AND site_name=?z [u
] 跳过: z...z] ( F) ensure_asciir0 zINSERT OR REPLACE INTO gov_raw
(page_url, site_name, title, publish_date, source_url, content, attachments)
VALUES (?, ?, ?, ?, ?, ?, ?)u
-> 新增u -> 异常: )sqlite3connectr cursor TOTAL_PAGESrangeLIST_URL API_TEMPLATEformatprintrU rd lenextend Exception enumerateexecute SITE_NAMEfetchoner jsondumpscommitclose) max_pagesconncurpages all_itemspagerS r_ r` e new_count error_countidxpage_url
list_title list_datera rc ri rv att_jsons r@ crawlr si ??7 #D
++-C"I EIa# 19C %%d+C
TF!E7(3st9+>?
C=D t$E E#e* T* +,- U # & IK2;Iq2I ).
.h
I E
y !
<<> CuAc)n -Z
3B7HL M
C5#i.)JsO+G3Ar s
**..$5
6 $ % 3 3j D * 60 ";1|?2D z
$X
$
$
&F
C> D t~~.C cU)E7)F8
DE
rA