Gj% < d Z ddlZddlZddlZddlZej
j
dej
j ej
j e ddl
m Z ddl Z
dZdZdZe e dZdd iZh d
Zd Zdd Zd
Zd Zd Zedk( r e yy)uY
crawl_kmfgw_pzjg.py - 昆明市发改委-批准结果信息
CMS: 昆明市政府信息公开平台 (ZCMS)
列表: /zfxxgklm/fdzdgknr/zdlyxxgk/zdjsxmpzhssly/pzjgxx/index.shtml (page1)
/.../pzjgxx/index_{N}.shtml (page N)
详情: /c/YYYY-MM-DD/ID.shtml
内容: div.activity > p, table
标题: div.content (完整)
日期: div.activity-info
N)push_to_searchdbu% 昆明市发改委-批准结果信息zhttps://fgw.km.gov.cnz0/zfxxgklm/fdzdgknr/zdlyxxgk/zdjsxmpzhssly/pzjgxxz /index.shtmlz
User-AgentzoMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36>) pdddldth1h2h3h4h5h6hrlioltdthtruldivnavpreformmainasidetabletbodytfoottheadcenterdialogfigurefooterheaderhgroupaddressarticlecaptiondetailssectionfieldset
blockquote
figcaptionc | yddl }ddlm g fd | dj }|j dd| }|j dd | }|j S )
ul 块级边界出换行、行内节点直接拼接、
出换行（≈ 浏览器看到的换行结构）。N r )NavigableStringc F | j D ] }t | rj t | *t |dd dk( rj d Lt |dd t
v r+j d | j d | y )Nnamebr
)children
isinstanceappendstrgetattr _BLOCK_TAGS)nodechr/ outwalks crawl_kmfgw_pzjg.pyr= zbody_text..walk2 s --
B"o.
3r7#VT*d2
4 VT*k9
4 R
4 R
z[ \t\r\f\v]*\n[ \t\r\f\v]*r3 z\n{3,}
)rebs4r/ joinsubstrip)el_retr/ r< r= s @@@r> body_textrI * sb z#
C H
A -tQ7A 61%A 779r? c ddl m} |t | d }|j d D ]P }|j dd }|s|j d r*|r t j j || n||d< R |j d D ]P }|j d d }|s|j d
r*|r t j j || n||d < R t | S ) uA 保留 HTML 表格结构，仅将相对链接/图片转绝对 URLr
BeautifulSoup html.parserahrefr. )http
javascript#imgsrc)rP z//zdata:) rB rL r7 find_allget
startswithurllibparseurljoin)r base_urlrL tblrN rO rS rT s r> html_table_to_htmlr] E s !
E
M
2C
\\#
SuuVR (CD@H ,,Xt@H --h parse_list_pagery T sd ! } -D EKK OP %%j1$$Y/ {{62& t , zz'2.M&///2MH?JA ggaj\1771:+;+;A+>*?qAQAQRSAT@UV5 ??3 $H
__V
$H"H:Qtf5H
[[]
- 6 Lr? c ddl m} || d }d}|j d }|rI|j d }t j
d| }|r |j
d j }d}|j d
}|r7t j d |j }|r|j
d }g }g } |j d }
|
r|
j D ] } | j d
k( rt | } | s!| j d D ] }
|
j dd }t j d|t j s;|
j d xs t j j# | }|j% d r
t& |z }n.|j% d r|}nt& d|j) d }| j+ ||d |j+ | | j dk( r"t- | }|s0|j+ | C| j dk( sT| j/ d sg| j d D ]! }t- | }|s|j+ | # | s|
r|
j d D ] }
|
j dd }t j d|t j s;|
j d xs t j j# | }|j% d r
t& |z }n.|j% d r|}nt& d|j) d }| j+ ||d dj1 | }| r?|r|dz
}|dj1 | D
cg c] }
dj3 |
d |
d c}
z
}dj1 d | D }||||fS c c}
w ) Nr rK rM r. z div.contentTr_ u ^(.+?)发布时间ra zdiv.activity-infoz(\d{4}-\d{1,2}-\d{1,2})z div.activityr rN rO z \.(docx?|pdf|xlsx?|rar|zip)$re rP )r1 rf r r r@ r3 u [附件：{}]({})r1 rf ,c 3 & K | ] }|d yw)rf N ).0rN s r> z$parse_detail_page.. s 8q!E(8s )rB rL ri rj rA rk rl rE searchr4 r1 rI rU rV IospathbasenamerW rn lstripr6 r] findrC format)rp rL rq r` content_divrw rv info_div
body_parts attachments activity_divrF p_textrN rO attach_namerx mdr body
attach_strs r> parse_detail_pager u sf ! } -D
E//-0K$$4$0 HH*E 2 GGAJ$$&E H 23H II0(2C2C2E F wwqzH JK??>2L '' .Bww#~"2 [[ -
W uuVR 0 99%DdBDDQ*+**4**@*ZBGGDTDTUYDZK #s 3+3d?!% !8+/.6ZqS9I8J+K ' . . H/U V
W %%f-G#'+ %%b)E!bggg&6 [[ 1 .E+E2B"))"-.- .6 < &&s+
KA55$Dyy8$E jjtj 4 N8H8H8N ??3' '$H__V, #H"*1T[[-=,> ?H""K#IJ
K ;;z "D FND T_`q.55ai5J`aa8K88J (D* ,, as O
c dd l } | j d }|j dt dd |j }g }t d|j dz D ] }|dk( rt }nt t d | d
}t d | d |j d
| t j |t d }d|_ t! |j" }|s
t d n+t dt% | d |j' | t dt% | d g } t) | D ] \ }
} t d|
dz d t% | dd t j | d t d }d|_ t+ |j" \ } }
}}| s| d } |r t% |j- dk rt d | d d t | d d d | j/ | | d | d ||
xs | d! t0 |d" t d#t% | d$|
xs | d! | s t d% y t d&t% | d' t3 | t0 t d(t% | d) y # t $ r}t d| Y d }~?d }~ww xY w# t $ r}t d | Y d }~d }~ww xY w)*Nr u+ 昆明市发改委-批准结果信息爬虫) descriptionz--pages u 爬取页数)typedefaulthelpra z/index_z.shtmlu 列表页 re z: )headerstimeoutzutf-8z
FAIL: u → 0条，停止u → u 条u
共获取 u
条列表项z [z] r. )endrf z FAIL (fetch): r` u 跳过(空内容): 2 z...rg )r` rf
source_urlcontentpub_date site_namer u 正文: u 字 | 日期: u 未获取到数据u
入库 u 条...u 完成！共入库 u 条)argparseArgumentParser add_argumentint
parse_argsrangepages
LIST_PAGE1rn ro printrequestsrV HEADERSencoding Exceptionry textlenextend enumerater rE r6 SITE_NAMEr )r parserargsall_list_itemspagerf rerr resultsiitemr` rg r attachs r> r r s
$
$1^
$
_F
Q^L DNaa ( % 19 C JxjvVA AJ ' ) +
U C() e$#%&
L^,-]
;<G ^, N4
AaC5#n-.b1r: T%['2FA AJ
%6aff$=!tT6MEs4::<(1, )%* 6 7
s C !; u+ ,V "!
SYK~d6Jd6l5KLM; N>
"# Ic'l^7
+,Wi( G ~T
23e Jqc" # $ $QC( ) s0 #I=&J"= J JJ " K+J??K__main__)r. )__doc__r rA sysr r insertdirnameabspath__file__ crawler_libr urllib.parserX r rn ro r r r9 rI r] ry r r __name__r} r? r> r s
277??277??8#<= > ( 3 " =z(<
0
D * 6 BK-ZB4H zF r?