ujW$ v d Z ddlZddlZddlZddlZddlmZmZmZ ddlZddl Z ej j dej j ej j e ddlmZ e j" e j$ j& dZdZdZdZd Z ej2 ej4 ed
z
j7 d Z d
diZ d Z d Z d Z d Z!d$dZ"e#dk( rddl$Z$ e$jJ de Z&e&jO ddd e&jO de(d e&jS Z* ej Z+e*jX xs) e-ej\ d kD xr ej\ d d k( Z, e"e, e/d! ej e+z
d"d# yy)%u
赣州经开区-环境保护 (gzjkq.ganzhou.gov.cn)
列表: /jkqxxgk/c109966/xxgk_list.shtml
分页: index_N.shtml 模式
详情: /jkqxxgk/c109966/YYYYMM/{hash}.shtml
正文: div.xxgk-content 或 .xxgk-container
日期:
N)datetimetimezone timedelta)push_to_searchdbu 赣州经开区-环境保护z https://gzjkq.ganzhou.gov.cnz /jkqxxgk/c109966/xxgk_list.shtmlz/jkqxxgk/c109966 iG )daysz%Y-%m-%dz
User-Agentz[Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/125.0.0.0 Safari/537.36c r t j | t dd }d|_ |j S # Y y xY w)N F)headerstimeoutverifyzutf-8)requestsgetHEADERSencodingtext)urlrs &/root/gov_crawler/crawl_ganzhou_jkq.pyfetchr s5 LLgr% H
vv
s /2 6c
g }d}t j || t j D ]] }|j d |j d j |j d }}}|sG|sJ|j |||f _ |sd}t j || t j D ]r }|j d |j d t j dd|j d j }}}|s\|s_|j |||f t |sd}t j || t j D ] }|j d }t j dd|j d j }|rt | d k rY|j } | | | d
z }
t j d |
} | r| j d nd}|j |||f g } |D ]4 \ }}}|j d s t |z }| j |||f 6 | S )
u0
提取列表条目: (title, url, date_str)
常见 gov 列表模式:
标题YYYY-MM-DD
YYYY-MM-DD标题
zr]*>.*?]*href="([^"]+)"[^>]*title="([^"]*)"[^>]*>.*?.*?]*>(\d{4}-\d{2}-\d{2}).*? z`]*>.*?]*>(\d{4}-\d{2}-\d{2})\s*]*href="([^"]+)"[^>]*>(.*?).*?<[^>]+> zF]*href="(/jkqxxgk/c109966/\d{6}/[a-f0-9]+\.shtml)"[^>]*>(.*?) i (\d{4}-\d{2}-\d{2})http) refinditerDOTALLgroupstripappendsublenendsearch
startswithBASE_URL)
htmlitemspattern1mhreftitledate_strpattern2pattern3end_posnearbydate_mresults
r
parse_listr9 ! s
E EH
[[4
3 2 !
AGGAJ,<,<,>
Xe T LL%x0 12 vXtRYY7 6A$%GGAJ
BFF:rSTSZSZ[\S]<^ECJNeegG''#+.FYY 5v>F*0v||AbH LL%x0 1
2 F!& / tXv&d?D
udH-./ M c J | dk( r
t t z S t t d| dS )u 构建分页 URLr z/index_z.shtml)r+ LIST_PATHLIST_DIR)pages r get_page_urlr? R s+ qy)##Zzf 55r: c t | }|syi }t j d| }|r"|j d j |d< |j d s^t j d|t j }|r7t j dd|j d j |d< |j d sIt j d|t j }|r"|j d j |d< t j d | }|r|j d |d
< |j d
s,t j d | }|r|j d |d
< |j d
s,t j d | }|r|j d |d
< |j d
s,t j d
| }|r|j d |d
< d}t j d|t j }|r |j d j }|rt | dk rt j d|t j }|r|j d j }t j d|t j }|r]*name="ArticleTitle"[^>]*content="([^"]+)"r r1 z]*>(.*?)r r z(.*?)z=]*name="PubDate"[^>]*content="(\d{4}-\d{1,2}-\d{1,2}) publish_dateu, 发布日期[：:]\s*(\d{4}-\d{1,2}-\d{1,2})u, 发布时间[：:]\s*(\d{4}-\d{1,2}-\d{1,2})r NzJ]*class="[^"]*xxgk-content[^"]*"[^>]*>(.*?)\s*(?:]*>(.*?)\s*(?:|