j! d Z ddlZddlZddlZddlZddlmZ ddlmZ dZdZ dZ
dZ d Z d
d iZ
h d Zd
Zd Zd Zd Zd Zd Zedk( r e yy)u
沅江市-建设项目环评
http://www.yuanjiang.gov.cn/20839/index.htm
CMS: 自有CMS (GB2312)
分页: index_N.htm (N从0开始，首页index.htm，第2页index_1.htm...第10页index_9.htm)
详情: content_{id}.html
N)datetime)
BeautifulSoupz/root/gov_crawler/search.dbu 沅江市-建设项目环评z+http://www.yuanjiang.gov.cn/20839/index.htm"http://www.yuanjiang.gov.cn/20839/gov_rawz
User-AgentzoMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36>) pdddldth1h2h3h4h5h6hrlioltdthtruldivnavpreformmainasidetabletbodytfoottheadcenterdialogfigurefooterheaderhgroupaddressarticlecaptiondetailssectionfieldset
blockquote
figcaptionc | yddl }ddlm g fd | dj }|j dd| }|j dd | }|j S )
ul 块级边界出换行、行内节点直接拼接、
出换行（≈ 浏览器看到的换行结构）。N r )NavigableStringc F | j D ] }t | rj t | *t |dd dk( rj d Lt |dd t
v r+j d | j d | y )Nnamebr
)children
isinstanceappendstrgetattr _BLOCK_TAGS)nodechr2 outwalks $/root/gov_crawler/crawl_yuanjiang.pyr@ zbody_text..walk, s --
B"o.
3r7#VT*d2
4 VT*k9
4 R
4 R
z[ \t\r\f\v]*\n[ \t\r\f\v]*r6 \n{3,}
)rebs4r2 joinsubstrip)el_retr2 r? r@ s @@@rA body_textrM $ sb z#
C H
A -tQ7A 61%A 779rB c | dk( rt }n t d| d}t j |t d }d|_ |j S )z;Get list page content for the given page number (0-indexed)r index_z.htm headerstimeoutgb2312)LIST_URLBASE_URLrequestsgetHEADERSencodingtext)pageurlrs rA
fetch_pager_ ? sB qy
&d+ S'26AAJ 66MrB c t | d }g }|j dd }|s|}|j d D ] }|j d }|r|j d s)|d }|j d }|rt | d
k rQd |vrV|j
d s7|j
d
rd|z }n |j
d rd| }n t |z }d}|j d } | r| j d }t j dd| }t j dd| j }|j |||d |S )z.Extract article links and dates from list page html.parserr listsclass_r ahrefT)rI
content_http/zhttp://www.yuanjiang.gov.cn.r r1 spanz
\s*\u00a0+\s* z\s{2,})titler] date) r findfind_allrX get_textlen
startswithrV rE rH rI r9 )
htmlsoupitems lists_divr a_tagrf rn date_strrl s
rA
extract_itemsr{ I sV } -D E % 0I & F EIIf - V}T*E
R T ! v&s#4t; %;D6B $ wwv }}4}0H 'e4y#u-335
uT8DE? FB LrB c d}d}d}t j d| }|ret |j d }t |j d }t |j d }t d| d| d| d d
||fS ) z4Extract total count and max page from pagination divr u 共(\d+)条.*?(\d+)/(\d+)页 z Pagination: u 条, rj u 页Tflush)rE searchintgroupprint)ru totalmax_pagecurrent fenye_matchs rA parse_paginationr w s
EHG )) ;TBKK%%a()k''*+{((+,
ugU7)1XJcB$O (?rB c R t j | t d }d|_ t |j
d } |g d D ] }|j
|j dd xs= |j dd
xs( |j dd
xs |j dd }|r t | }n t | }t j d
d| }t j dd| }|j S # t $ r}t d| d Y d}~yd}~ww xY w)zFetch detail page contentrP rQ rT ra )scriptstyler r% r& r r zoom)idzyea-conrc z article-concontentrC rD z {2,}rm z [ERROR] detail: Tr Nr1 )rW rX rY rZ r r[ decomposerp rM rE rH rI Exceptionr )r] r^ rv tag content_divr es rA fetch_detailr s LLgr :
QVV]3OP C MMO
IIeI '
.yyyy1
.yy}y5
. yy9y- ,G oG&&FG4&&30}}
"1#&d3s D D D& D!!D&c H t j t } | j }t }|j dt f D ] }|j |d d}|j dt f j }|r
|d r|d }| j t d| d t dt | d t d }t | \ }}t d | d
| d g }t | D ]Q } | dkD r t | }t | }
t d | d z d
| dt |
dd |j! |
S t dt | d d} d} d}
t j t } | j }|D ]U }|d }|d }|d }||v r| d z
} t# | }|rt | dk r(t d|d d dt | dd |s|
d z
}
`|sg t% j& |t( d }d|_ t- j. d |j0 }|r!|j3 d j5 d
d }|s#t7 j8 j; d }t- j< d d | d d! }|j d"|||t |f | d z
} | d#z dk( s2| j? t d$| d%| d&d X | j? | j t d'd( d t d)t d t dt | d t d*| d t d+| d t d,|
d t d( d y # Y 1xY w)-Nz)SELECT url FROM gov_raw WHERE site_name=?r z
1900-01-01z7SELECT MAX(publish_date) FROM gov_raw WHERE site_name=?zLatest DB date: Tr zExisting URLs: z
Total pages: z, Expected total: z Page r} rj z: z itemszTotal list items: r] rn ro d z
[SHORT] rP z... () rQ rT z (\d{4}[-/]\d{1,2}[-/]\d{1,2})-z%Y-%m-%dz[^\d-]r1 rg z
INSERT OR IGNORE INTO gov_raw (title, content, url, site_name, publish_date)
VALUES (?, ?, ?, ?, ?)
z Progress: z new / z skipr6 z2==================================================zSite: zNew: zSkipped (existing): zErrors: ) sqlite3connectDB_PATHcursorsetexecute SITE_NAMEaddfetchonecloser rs r_ r ranger{ extendr rW rX rY rZ rE r r[ r replacer nowstrftimerH commit)conncexistingrowlatest_db_dateru total_countr all_itemsr\ rw new_count
skip_count error_countitemr] rn rz r r2date_ms rA r r s ??7 #D
A uHyyDylS SV "N
))MPY|
\
e
e
gC
s1v QJJL ^ ,
-T: OCM?
+48 a=D ,T 2K M(#5k]
C4P Ih !8 d#D d#
Qxq
"SZL?tL
s9~ .
/t< IJK ??7 #D
A 'R5kW
< (? !OJ s##g,, JuSbzl%G ~Q?t Lq
\\#wC & #CRWWM %||A 6 6sC @H ||~..z:H66)R23B7 Wc9h
7 9 Q r>Q KKM L 7:,eDD QO'RR KKMJJL Bvh-t$ F9+
d+ s9~ .
/t< E)
T*
-T: H[M
"$/ VHT"7
s
A&N N!__main__)__doc__rW r rE osr rF r r r rU rV
TABLE_NAMErY r< rM r_ r{ r r r __name__ rB rA r s
' * 8 /
D * 6,\ <\#| zF rB