._fix s ggaj >>& !771: >>$ QR(G
^^C
2S8G nGwwqz!!'C-#"5w7H37NOO z\bsrc="([^"]+)")rsplitresub) html_contentpage_urlr r s @r
_make_img_absr s9 sA&q)C/H
P 66$dL 99r iG )daysz%Y-%m-%du [info] 日期过滤: >= c p | dk( rt t d}nt t d| dz
d} t j j |t
}t j j
|d }|j j d S # t $ r}t d | d
| Y d }~y d }~ww xY w) u 获取列表页HTMLr r
z/index_z.htmlheaders timeoututf-8u [error] 第u 页请求失败: N) BASE_URLLIST_DIRurllibrequestRequestHEADERSurlopenreaddecode Exceptionprint)page_nourlreqrespes r fetch_list_pager8 * s !|
8*A&
8*GGaK=>nn$$S'$:~~%%c2%6yy{!!'**
G9$5aS9:s A$B B5B00B5c . t j d| }g }|D ]w \ }}}|j d r
t |z }n%|j d rnt t z dz |z }|j |j
||j
d y |S )u 解析列表HTMLzu| \s*]*href="([^"]+)"[^>]*title="([^"]*)">[^<]*\s* | \s*\s*([^<]+)\s* | r
r
)titler4 date)r findallr r( r) appendstrip)htmlitemsresultsr4 r: r; s r parse_list_itemsrB 9 s
JJ A
E G ! SUD >># S.C
^^F
# X%+c1C sDJJLQRS Nr c t j j | t }t j j |d }|j j
d }d
}t j d | }|r |j d j }t j d
| }|r/|j d j }t || }||fS t j d| } | r/| j d j }t || }||fS d
|fS # t $ r }t d| dd d| Y d}~y d}~ww xY w)u+ 从详情页提取正文HTML和完整标题r" r$ r% r' u [warn] 详情页失败: iNz, ) rD rD z+r zI([\s\S]*?)
\s* r )
r4 r5 r6 r? r7
full_titlemeta_tr contentm2s
r fetch_detail_contentrJ M sK nn$$S'$:~~%%c2%6yy{!!'* J
YYEt
LF
\\!_**,
: A ''!*""$ -
""
T
B
((1+##% -
""
z>=
*3st9+Rs;<s A%D2 2 E;EEc dd l } | j t }|j }|j d t dt dt d}d}d}d}t dt dz D ] }t | }|st | } | st d| d /d}
| D ] } | d } | d }
| d
}|r|t k r|dz
}#|j d |
f |j d dkD r|dz
}Rt |
\ }}|sd }|r|n| } |j d
|
||t ||df |j dkD r|
dz
}
|j# ||
z
}t d| dt d|
d| |j% t dt d| d| d| d|
y # t $ r }t d| |dz
}Y d }~"d }~ww xY w)Nr zPRAGMA journal_mode=WALu [info] 总页数: u
, 每页: r u
[warn] 第u 页未解析到条目r: r4 r; z-SELECT COUNT(*) FROM gov_raw WHERE page_url=?u3 [无正文] 附件型条目，请查看原文链接zINSERT OR IGNORE INTO gov_raw
(page_url, title, content, site_name, publish_date, summary, category)
VALUES (?, ?, ?, ?, ?, ?, ?)u 通知公告u [error] 入库失败: z[page r
z] +z new, cumulative: u
[完成] u : 新增 u 条, 旧跳过 u , 重复 u , 错误 )sqlite3connectDB_PATHcursorexecuter2 TOTAL_PAGES PAGE_SIZEranger8 rB CUTOFF_DATEfetchonerJ SITE_NAMErowcountr1 commitclose)rL connc new_count old_count dup_count err_countpager? r@ page_newitemr: r4 r; rH rF final_titler7 s r mainrd t s ??7 #D
AII'( {m:i[
ABIIIIaq ) ,W t$ & Jtf$9: ; DMEu+C MH5 >
X
tfAk]#hZ7I)UVY,W\ JJL K {)I;6FykQZ[dZeenoxny
z{
045Q
s
1F"" G +GG __main__)__doc__osr sysurllib.requestr* r r getenvrN rV r( r) rR rQ r- r nowstrftimerT r2 r8 rB rJ rd __name__ r r ro s
(
"))K!2
3 9 ' 6 W
X: x||~ u 55??
K
. / ($N=|@ zF r