zjd d Z ddlZddlZddlZddlZddlZddlZddlmZ ej ej j ddl m
Z
dZdZedz ZdZd Zd
ed d Zd
Zd Zd Zd Zedk( r e yy)u
仪征市自然资源和规划局 - 信息公开（江苏省自然资源厅 gtapp 应用）
https://zrzy.jiangsu.gov.cn/gtapp/nrglIndex.action?classID=2c9082b55b652546015b65ec3be2004f
站点特征（2026-08-11 探测）:
- Struts2 action 动态站；列表 POST cpage=N 分页（topage JS form），18条/页，共94页
- 列表标题 ，显示文本截断须取 title
- 日期 | YYYY-MM-DD |
- 详情 = meta PubDate + table.xxgk-gjy-top 元数据(索引号/文号/发布机构/生成日期) + 标题 + 附件
- 附件 /gtapp/nrgl/GJAttach/{ts}.{ext}（相对路径，urljoin 详情页URL 绝对化）
- 正文 = 标题 + 附件段落（源站详情无独立正文 div，证照类公告以附件为主）
用法:
python3 crawl_zrzy_yz_xxgk.py --pages=67 # CUTOFF 3年窗口 (2023-08-10 起)
python3 crawl_zrzy_yz_xxgk.py --pages=1 # 测试
N)urljoin)push_to_searchdbu. 仪征市自然资源和规划局-信息公开zhttps://zrzy.jiangsu.gov.cnz@/gtapp/nrglIndex.action?classID=2c9082b55b652546015b65ec3be2004fz
2023-08-10u 自然资源zkMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0 Safari/537.36z?text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8)z
User-AgentRefererAcceptc t j | xs d } | j dd j dd } t j dd| } | j S )N u u u ^[•·\s]+)htmlunescapereplaceresubstrip)ts '/root/gov_crawler/crawl_zrzy_yz_xxgk.py clean_titler + sO
ag2 A (B ''"5A
~r1%A 779 c 8 t j t t dt | idd }|j |j xs d|_ |j }g }t j d|t j D ] }|j d |j d |j d |j d
f\ }}}}t | }|sXt j | }t! t | } |j# || |d |S ) u& POST cpage=N 返回 (items, next_page)cpage F)headersdatatimeoutverifyutf-8z]*>(.*?)\s*\s*(\d{4}-\d{2}-\d{2}) | )titleurldate)requestspostLIST_URLHEADERSstrraise_for_statusapparent_encodingencodingtextr finditerSgroupr r r
r append)
pagerrawitemsmr hrefdispr!
detail_urls
r fetch_list_pager7 1 s
hwD 6JTV_deA$$/AJ
&&C E
[[ f RTT H #$''!*aggaj!''!*aggaj"P tT4E" }}T" Xt,
uZFG H Lr c t j | t dd }|j |j xs d|_ |j }g }t j d|t j D ]z }|j d |j d }}t j | }t | | }t | xs |j! d d
}|j# ||d | g } |r| j# d | d
|D ] }
| j# d|
d d|
d d " dj% | } | |fS # t $ r dg fcY S w xY w)u" 返回 (content_html, attachments)r F)r r r r Nz2]*href='([^']*GJAttach/[^']+)'[^>]*>(.*?)r r /)namer z r; z
r )r" getr% r' r( r) r* Exceptionr r+ r, r- r r
r r splitr. join) r6
list_titler0 r1 attsamr4 r; abs_url
body_partsacontents r fetch_detailrI E sX LLWb O ((3G
ff DkkOQTVXVZVZ[ 4XXa["((1+d}}T"*d+4 :GMM#$6r$: T'23 4 JC
|401
JL5
"QvYKxHIJggj!G D=# Rxs AE
EEc
0 d } t j dd D ]M }|j d r t |j d d } 2|j sCt | } O g }t
}d} | r|| kD rnt | }|st d| d nd}|D ]` }|d |v r |d t k r t d| d |d d
t d d } n,|j |d |j | |dz
}b t d| d
t | d| dt | |d k( rn|dz
}t d| dt | g }t | D ] \ } }t |d |d \ }
} |
d|d d}
|j t |d |d |
|d |d | rt! j" | d ndd | dz dz dk( stt d| dz dt | t d t | t% |t& y ) Nr z--pages==z[Pz
] empty, stopr r r! z] z
< CUTOFF z, stopi?B z] got z items, new z, total z[AutoPg] max_pages=z , collected r r< r= F) ensure_asciiz[]) site_namer pub_daterH
source_urlr attachments2 z [Detail] r9 z[Detail] done )sysargv
startswithintr@ isdigitsetr7 printCUTOFFaddr. len enumeraterI SITE_NAMEjsondumpsr CATEGORY) max_pagesrG all_itemsseenr/ r2 newitvalidirH rC s r mainrh _ sZ I
XXab\ <<
#AGGCLO,I
YY[AI
I 5D D
) % BtfM* + B%yD &zF"4&2f:,jGH HHRY R 1HC 4&s5zl,se8C NCSTU 6> + ,
{,s9~6F
GH
E9% 62$RY7 <
?BwK=-G
"[6
U)e9CG4::d?T
ER<1 IacU!C N#34 5 6
N3u:,
'(UH%r __main__)__doc__r rR r^ r urllib3r" urllib.parser disable_warnings
exceptionsInsecureRequestWarning crawler_libr r] BASE_URLr$ rY r` r% r r7 rI rh __name__ r r rt s
++BB C ( < ( X X @O (44&l zF r