ej d Z ddlZddlZddlZddlZddlZddlmZ dZdZ dZ
dZ dZ d Z
d
Zd d iZdad
Zd Zd Zd Zd Zedk( r e yy)ud
crawl_gaoxian_hjbh.py - 高县-环境保护
CMS: 自定义 (静态分页)
列表: /zwgk/zdlyxxgk/hjbh/index.html (page 1) + index_{N}.html (page N+1)
22页, 13~14条/页, createPageHTML(22,...)
详情: /zwgk/zdlyxxgk/hjbh/YYYYMM/tYYYYMMDD_NNNNNNN.html
标题:
日期:
正文: div.yydh-xx-content > p (> 含表格)
N)
BeautifulSoupz/root/search.dbzhttps://www.gaoxian.gov.cnu 高县-环境保护u 四川u 环境公示z/zwgk/zdlyxxgk/hjbh/ z
User-Agentz| j ddg D ]( }
|
j d } | s|
j | * | j d D ] }|
j t | |
rdj |
}|s|j d }|rg }
|j d D ]( }|j d } | s|
j | * |j d D ] }|
j t | |
rdj |
}n |j d j }|rt j dd| }|r|d d nd}||||fS )Nr r" metaname ArticleTitlecontentPubDatez(\d{4}-\d{2}-\d{2}) c 6 | xr t | t xr d| v S )N TRS_UEDITOR
isinstancestrcs r z extract_detail..^ s (WjC.@(W]VWEW r3 r )psectiondivF) recursiveTr# rC rD tablez
c 6 | xr t | t xr d| v S )Nzyydh-xx-contentr= r@ s r rB z extract_detail..r s )\z!S/A)\FW[\F\ r3
) separatorz )
r r' r r$ researchgroupr( r) r- r? joinsub)r. r r0 r!
meta_titledate_str pubdate_metar8 meditorpartschildtxttagtblrE rC summarys r extract_detailr\ I sL } -D
E6FN#;&&B0 &gdsmBG (GW ,,r3 c dd l }|j t d }|j }d}g }| D ] \ }}}} }
|j d|
f |j r.|j d|
|t ||| |
dt t df |j } | r |j | |t | xs df |d z
} |j |rK|D ] \ }
}}} |j d
|
||| f |j t dt | d |j |S # t $ r } t d
|d d d | Y d } ~ d } ~ ww xY w# t $ r} t d|
d | Y d } ~ d } ~ ww xY w)Nr
r z)SELECT id FROM gov_raw WHERE page_url = ?zINSERT INTO gov_raw
(page_url, title, site_name, publish_date, content, summary,
source_url, category, industry, group_name, script_name)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)u 环境保护zcrawl_gaoxian_hjbh.pyr" r: z
[DB ERROR] z: zVINSERT OR IGNORE INTO gov_search(rowid, title, site_name, summary) VALUES (?, ?, ?, ?)z [FTS ERROR] id=u
FTS同步: 条)sqlite3connectDB_PATHcursorexecutefetchone SITE_NAMEINDUSTRYGROUP lastrowidr- r r commitlenclose)
items_datara connrA inserted fts_batchr! rR r8 r[ page_urlnew_idr ridsites r
save_to_dbrv s ??7B? /D
AHI7A 53x'8 5
IIAH; Ozz|
II? 5)Xw>8UH8} E" ./ 0 4
bTs5zl#.d; E" 23 4
k LY (
34 E & eT$i #1+t#D BB
bkE2w>?
3
u E F5'3u:,c
*+ " N5'
&' HYK
!r3 __main__)__doc__r
rL sysosr bs4r rc r+ rg ri rh r, r r r r r2 r\ rv r __name__ r3 r r s
' ! " P ,;-|&R1"h zF r3