正文
N)datetime timedelta SEARCH_DBz/root/search.dbu mmht.gov.cn-公示公告zhttp://www.mmht.gov.cnz
/xwzx/gsgg z
User-Agentz
= c | dk( rt t d}nt t d| d} t j |t d }d|_ |j S # t $ r}t d| d | Y d
}~y d
}~ww xY w) u 获取列表页HTML z /index.htmlz/index_z.html headerstimeoututf-8u [error] 第u 页请求失败: N ) BASE_URL LIST_PATHrequestsgetHEADERSencodingtext Exceptionprint)pageurlres /root/gov_crawler/crawl_mmht.pyfetch_list_pager s qy
9+[1
9+WTF%8 LLgr :
vv
D6!21#67s .A A<!A77A<c T g }d}t j || D ] }|j d j }|j d s t
|z }|j
|j d j |j d j |d |S )u 解析列表页条目zn\s*\s*([^<]+)\s*([^<]+)\s*r http )titledater )refinditergroupstrip
startswithr append)htmlitemspatternmr s r parse_list_itemsr/ * s EG
[[$
' ggaj ~~f%S.C
WWQZ%%'GGAJ$$&
L c @ t j d| }|r |j d j S t j d| }|rO|j d }t j dd| }t j dd| }|j S y)u" 从详情页提取 #content 正文z[]*id="content">([\s\S]*?)
\s*r zI
([\s\S]*?)
\s*
z
.*?
r z
.*?
)r% searchr' r( sub)r+ r. contents r fetch_detail_contentr5 ; s
prvwAwwqz !!
^`deA''!*&&4b'B&&7WE}}
r0 c t j | t d }d|_ |j S # t
$ r }t
d| dd d| Y d}~yd}~ww xY w) u 获取详情页HTMLr
r r u [warn] 详情页请求失败: iNz, r )r r r r r r r )r r r s r fetch_detail_pager7 J sX LLgr :
vv
0ST 2aSABs .1 AAAc p dd l } | j t }|j }|j d d}d}d}d}t dt dz D ] }t | }|st | } | st d| d /d}
| D ] } | d } | t k r|dz
}| d }
| d }|j d |f |j d dkD r|dz
}Pt | }|s|dz
}ct | }|sd
|
} |j d ||
|t | |
d f |j dkD r|
dz
}
|j# ||
z
}t d| dt d|
d| d |j% t d| d| d| d| y # t $ r }t d
| |dz
}Y d }~'d }~ww xY w)Nr zPRAGMA journal_mode=WALr u
[warn] 第u 页未解析到条目r$ r# r z-SELECT COUNT(*) FROM gov_raw WHERE page_url=?u [无正文] z
INSERT OR IGNORE INTO gov_raw (page_url, title, content, site_name, publish_date, summary, category)
VALUES (?, ?, ?, ?, ?, ?, ?)
u 公示公告u [error] 入库失败: z[page /z] +z new, cumulative: z newu"
[完成] 茂名高新区: 新增 u
条, 重复 u , 超3年跳过 u , 错误 )sqlite3connectDB_PATHcursorexecuterange TOTAL_PAGESr r/ r CUTOFF_DATEfetchoner7 r5 SITE_NAMErowcountr commitclose)r: connc new_count dup_count old_count err_countr r+ r, page_newitemr$ r# r detail_htmlr4 r s r mainrP U s ??7 #D
AII'(IIIIaq ) /[ t$ & Jtf$9: ; D
RT ::> MH; D
X
tfAk]#hZ7I)TXYZ_/[b JJL / {- {Rbclbmmv xA wB C D
045Q
s ?1F F5F00F5__main__)__doc__ossysr% r r r getenvr< rC r r r@ r nowstrftimerA r r r/ r5 r7 rP __name__ r0 r rZ s
(
"))K!2
3 & # P x||~ u 55??
K
. / " >DB zF r0