with full HTML (tables, paragraphs, attachments)
N z[AutoPg] max_pages=z3https://www.jiangmen.gov.cn/bmpd/jmssthjj/zwgk/tzggu% 江门市生态环境局-通知公告 SEARCH_DBz/root/search.dbzoMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36z?text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8zzh-CN,zh;q=0.9,en;q=0.8)z
User-AgentAcceptzAccept-Language)g333333?g ? c P t dt j d d| d y )N[z%H:%M:%Sz] T)flush)printtimestrftime)msgs #/root/gov_crawler/crawl_jiangmen.pylogr " s# AdmmJ'
(3%
0= c | dk( r
t d}n t d| d} t j |t d }d|_ |j }t j d |t j }g }|D ]P \ }}} |j d r|}
nd
| }
|j |
|j | j d R |S # t
$ r }t
d| d | g cY d
}~S d
}~ww xY w)z6Crawl a list page and extract article URLs + metadata.r z /index.htmlz/index_z.html headerstimeoututf-8zERROR fetching list page : NzO
]*title="([^"]*)"[^>]*>.*?([^<]+)httpzhttps://www.jiangmen.gov.cn)urltitledate)BASE_URLrequestsgetHEADERSencoding Exceptionr textrefindallDOTALL
startswithappendstrip) pager rehtmlitemsresultshrefr date_strfull_urls r
crawl_listr2 & s qy
+&
'$u- LLgr :
66D
JJZ
E G!& eX ??6 "H4TF;H[[] NN$
N7 'vRs 34 s #C
C0C+%C0+C0c t j | t d }d|_ |j }t j d| }t j d | }t j d
| }|r |j d j nd }|r"|j d j dd
nd }|r |j d j nd } t j d| }
|
s||| d fS |
j } |j d| } | dk( r||| d fS d }
| d z }|t | k re|
dkD r`|||dz dk( r |
d z }
|dz
}n6|||dz dk( r&|dz t | k r||dz dv r |
d z
}
|dz
}n|d z
}|t | k r|
dkD r`|| |dz }||| |fS # t $ r}t d| d| Y d}~yd}~ww xY w)z-Crawl a detail page and extract full content.r r r zERROR fetching detail r N)NNNz+
r z