uj d Z ddlZddlZddlZddlZddlZddlZdZdZ e dz Z
dZ dZ ej Zde_ ej e_ d Zd
Zd Zd Zd
Zedk( r e yy)u
东港市政府 - 生态环境领域 (donggang.gov.cn)
CMS: 自定义 PHP, JS 分页 (client-side only, 30条)
List: /dgszf/zfxxgk/fdzdgknr/zdly/sthjly/glist.html
Container: ul > li > div.time (date) + div.name > a (title)
Detail: /html/DGSZF/{YYYYMM}/{id}.html
Content: div.center-info (p paragraphs)
Title: (strip suffix)
Date: YYYY-MM-DD from page
No WAF, curl OK, 30 items on single page (2024-2026)
Nu 东港市-生态环境领域zhttps://www.donggang.gov.cnz-/dgszf/zfxxgk/fdzdgknr/zdly/sthjly/glist.htmlz/root/search.dbz
2023-01-01Fc & t j j | ddi } t j j |dt }|j j
dd S # t $ r}t d | d
| Y d }~y d }~ww xY w) Nz
User-Agentz<Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36)headers )timeoutcontextzutf-8replace)errorsz [FETCH ERROR] : ) urllibrequestRequesturlopenssl_ctxreaddecode Exceptionprint)urlreqrespes (/root/gov_crawler/crawl_donggang_sthj.pyfetchr s
..
T/
C~~%%c2w%Gyy{!!')!<<
Rs+,s AA, , B5B Bc g }| j d }|dk r|S | j d| }|dkD r| ||dz n| |d }d}t j ||t j D ] }|j d j }|j d j }|j d j } t j d
d | } |j d s t |z }|j || |d
|S )z'Parse list - extract items from ul > liz<ul style="display:block;">r z</ul> Nzy<li>.*?<div class="time">(\d{4}-\d{2}-\d{2})</div>.*?<div class="name">.*?<a[^>]*href="([^"]*)"[^>]*title="([^"]*)"[^>]*> z ^\[.*?\]\s* http)r titledate)
findrefinditerDOTALLgroupstripsub
startswithBASE_URLappend)
htmlitemsul_idxul_endul_htmlpatternmatchr" hrefr! s
r
parse_listr5 , s E
YY4
5F
z
YYw
'F)/!d6&1*%fg G KGWgryy9 B{{1~##%{{1~##% A$$&~r51v&d?D
TE4@AB L c P d}t j d| }|r:|j d }t j dd| j }|s|}d}t j d| }|r|j d }d}| j d }|dk r| j d }|dkD rs| j
dd| }|dk r|}| j d | } | dk r| j d
| } | dk r| j d | } | dk r|d z } | || }
t j d
d|
t j } t j dd| t j } t j dd| } t j dd| } | j dd j dd } t j dd| } t j dd| } t j dd| } t j dd| j } | }|||dS ) z Extract title, date, contentr z<title>(.*?)r u! [-–—][^-–—]*-[^-–—]*$z(\d{4}-\d{2}-\d{2})zclass="center-info"r z]*>.*?)flagsz]*>.*?z
zz
z z<[^>]+>z[ \t]+z *\n *z\n{3,})r! r" content) r$ searchr' r) r( r# rfindr& r )
r- r! tmtr" dmr< idxstartendsectiontextcleans
r parse_detailrH D s
E *D 1B HHQK;RCIIKE D )4 0B xx{ G
)))
*C
Qwii-.
Qw
61c* 19ceii2C8 7$))$;SAC 7$))$5s;C 7%$,CuS/vv2BryyQvv0"d"))LvvlD$/vvgvt,||D"%--h<z2t,y#u-y$.y&%0668DW ==r6 c ( t j t d }|j }d}d}| D ] }|j dd }|j dd }|j dd }|j dd } | s|s|d z
}U |j d
||t || ||f |j dkD r|d z
}n|d z
} |j |j ||fS # t $ r#}
t d |d d d |
|d z
}Y d }
~
d }
~
ww xY w)
N r r r! r r r" r< r a
INSERT OR IGNORE INTO gov_raw
(title, page_url, site_name, publish_date, content, summary, attachments, source_url, date_rank)
VALUES (?, ?, ?, ?, ?, '', '', ?, CAST(strftime('%s', ?) AS INTEGER))
z
[DB ERROR] r
) sqlite3connectDB_PATHcursorgetexecute SITE_NAMErowcountr r commitclose) r. conncinsertedskippeditemr! r r" r< r s r insert_to_dbr[ s s' ??7B /D
AHG "%hhub!xx#((9b)u qLG
II iwTB
D
zzA~A
1 #* KKMJJL W M%* Rs3 4 qLG s 7C%% D.D Dc t dt d t t } | s t d y t | }t dt | d g }|D ] }|d t k rt d|d dt nwt d |d d
|d d d d
t |d }|s t d `t | }|d r|d |d < |d r|d |d< |d |d< |j | |rt | \ }}t d| d| d t d| d| |dkD rXt j t d }|j d |j |j t d y y y )Nz[INFO] u - 爬虫z[ERROR] Cannot fetch listz Found z itemsr" z [STOP] Date z < z r; r! 2 z...r z [SKIP] Cannot fetch detailr< z [DB] +z new, z existingu
[DONE] 新增: u
, 跳过: r rJ rK z4INSERT INTO gov_search(gov_search) VALUES('rebuild')z FTS rebuilt)r rR r LIST_URLr5 len CUTOFF_DATErH r, r[ rL rM rN rQ rT rU ) r- r. all_itemsrZ detail_htmldetailnewoldrV s r mainrf s GI;i
() ?D
)* t E HSZL
'(I
<+ % N4<.K=A B
T&\N!DM#2$6#7s;<DK( 2 4 k* '?F7ODM &>&.4< +Y
*S
VC5 23
!#j67 7??7B7D LLO P KKM JJL - r6 __main__)__doc__sysr$ osrL urllib.requestr sslrR r+ r^ rN r` create_default_contextr check_hostname CERT_NONE verify_moder r5 rH r[ rf __name__ r6 r rs s
+ ( E E
$#
$
$
& mm 0,>^ >$!N zF r6