uj d Z ddlZddlZddlZddlZddlZddlmZ ddlm Z dZ
dZ dZ dZ
d Zd
Zd d iZdad
ZefdZd Zd Zd Zedk( r e yy)u
crawl_xiangfuqu_hjbh.py - 开封祥符区-环境保护
CMS: UCAP (JS渲染), 需Playwright渲染列表
列表: /kfsxfqwz/c00511/pc/list.html (JS渲染, 15条/页, 4页共56条)
Playwright点击翻页: a.num[2], a.num[3]
详情: /kfsxfqwz/c00511/pc/content/content_{id}.html (可直接HTTP访问)
标题:
日期:
正文: div.article-content > p
N)
BeautifulSoup)sync_playwrightz/root/search.dbz https://www.xiangfuqu.gov.cnu 祥符区-环境保护u 河南 环境公示 z
User-Agentz els.map(el => ({title: el.textContent.trim(), href: el.href}))u 第u 页: 条flushhrefz /kfsxfqwz/c00511/pc/content/titleN)r chromiumlaunchnew_pagegotoBASE_URLrangeclickwait_for_timeoutr eval_on_selector_alllen
startswithappendclose) max_pages all_itemspwbrowserpagepgitemsitems r fetch_all_list_itemsr9 2 si I b++$$d$3 ! XJ ;< *E ; 9q=) DBAvJJ bT 4dJC))$/ --%WE
E"U3u:,c2$ ?
D<**hZ7S+TU $$d6lDM%BC
D! D(
56 -bT >? 6 s+ A D3-(D AD3%-D3 D0-D33D=c t | d }d}|j dddi }|r |j dd j }d}|j dddi }|r;|j dd }t j
d| }|r|j
d }d}|j d
d } | rg }
| j d
D ]( } | j d } | s|
j | * | j d D ] }
|
j t |
|
rdj |
}n | j d j }|rt j dd| }|r|d d nd}||||fS )Nz html.parser metaname ArticleTitlecontentPubDatez(\d{4}-\d{2}-\d{2})r divzarticle-content)class_pT)striptablez
) separatorz )
r findr rD researchgroupfind_allget_textr/ strjoinsub)htmlr soupr#
meta_titledate_str pubdate_metar? macpartsrC txttblsummarys r extract_detailr] S sx } -D
E6FN#;8Ur s
/
) $
P $- B&-R&R "F zF r^