m }j , d Z ddlZddlZddlZddlZddlmZ ddlmZ ddl m Z m
Z
ej j dej j dd Z
d Zd
Zeez Zd Zd Zd
ZddiZd Zd Zd Zd Zd Zd Zd Zd Z e dk( r e yy)u&
crawl_lbx.py — 雷波县人民政府·环评公告
========================================
TRS CMS，createPageHTML(N) 分页，0-indexed
首页 index.html 静态内含文章列表
列表：TITLE
日期从 URL 路径提取
详情： → 正文
日期
用法:
python3 crawl_lbx.py # 全量
python3 crawl_lbx.py 1 # 增量（只爬首页）
N)
BeautifulSoup)urljoin)datetime timedelta SEARCH_DB GOV_DB_PATHz/root/search.dbzhttp://www.lbx.gov.cnz/xxgk/zdxxgk/hjbh/hpgg/u 雷波县人民政府u 雷波县环评公告 z
User-AgentzoMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36c t d| y )Nz[lbx] )print)msgs /root/gov_crawler/crawl_lbx.pylogr - s F3%. c t j d| }|r7|j d d|j d d|j d S y)uI Extract date from URL like ./202606/t20260612_2991748.html → 2026-06-12z/t(\d{4})(\d{2})(\d{2})_ - )researchgroup)urlms r
extract_date_from_urlr 0 sH
-s3A''!* Qqwwqzl!AGGAJ<88
r c " | j S )zClean title if neededstrip)titles r
clean_titler 7 s ;;=r c < | dk( rt S t t d| dS )u? page_index: 0-based. 0 → index.html, 1 → index_1.html, etc.r index_z.html)LIST_URLBASE_URL LIST_PATH)
page_indexs r
build_list_urlr' ; s% QZ {& E ::r c t | }t d| d| t j |t d }d|_ |j }t |d }g }|j d t j d
D ] }|j d }|r|j d
s(|d
j } | j d rd| vrQ|j dd xs |j d }
t! t" | } t% | } |j' | |
| f t dt) | d |S # t $ r}t d| g cY d}~S d}~ww xY w)z9Fetch one list page, return [(url, title, date_str), ...]zFetching list page : headerstimeoututf-8z ERROR fetching list page: N html.parserliz
WIDTH:\s*100%)styleahrefz./z/tr r Tr z Found articles)r' r requestsgetHEADERSencodingtext Exceptionr find_allr compilefindr
startswithget_textr r# r appendlen)
r& r resphtmlesoupitemsr0 r2 r3 r full_urldate_strs
r
fetch_list_pagerI A sS
$C
j\C512||C"=
yy
} -D E mmD
3C(DmE 2 GGCLf
y t$D(8 gr" zmain.. s !Z1aqQ/EY!!Zs r z Page z&: all articles before cutoff, stoppingz$Total articles after 3-year filter: [r] z] 2 z...z WARNING: saving z without contentr
z2==================================================z
Done. Total: r4 z Date range: z ~ a
INSERT OR REPLACE INTO gov_search(rowid, title, site_name, summary)
SELECT rowid, title, site_name,
CASE WHEN length(content) > 200 THEN substr(content, 1, 200) ELSE content END
FROM gov_raw WHERE category = ?
z!FTS index rebuilt for new entrieszFTS rebuild note: )rA sysargvr nowr strftimer rangePAGESrI extendsum enumerater[ r@ rj sortedr^ r` r_ rb rd re rf r: ) incremental all_articlespiarticlesutrs filteredremaining_youngresultsir r rH rX detail_date
final_daterdatesr^ rg rD rt s @r
mainr s chh-!#: s(:K ||~ w(??II*UO//*+, LEl "2& 7t< = .6TT'!QqO?SQ1ITTH% / 0 Q8<<>I7r s
(
**..bjjnn]DU&V
W # % i # 3 ; B*"X"8HFR zF r |