uj7 @ d Z ddlZddlZddlZddlZddlZddlZddlmZmZ ddl m
Z
ej j dej j ej j e ddlmZ ej$ ed z
j' d Zd Zd
d iZd Zed
z ZdZdZd Zd Z d Z ddZ e dk( rU ej@ Z!e!jE de#d e!jI Z% e e%jL Z' e(de' d yy) u crawl_zhuji_hjsp.py - 诸暨市-建设项目环境影响评价信息公示 (浙江政务服务网)
站点: https://www.zhuji.gov.cn/col/col1229857087/index.html
CMS: jcms 新版(4.5.12.1-ZJSJYH-U39) unitbuild 动态加载 — 列表走API
API: /api-gateway/jpaas-publish-server/front/page/build/unit
?parseType=bulidstatic&webId=2745&tplSetId=TM6ldEAz1I8xTXi6VHiB0&pageType=column
&tagId=右侧list内容&editType=null&pageId=1229857087¶mJson={"pageNo":N,"pageSize":15}
列表: li.cf > a.fl(标题) + span.fr(YYYY-MM-DD)
分页: paramJson pageNo 1..48 (每页15条, 共719条)
详情: h1.wenzhang_content_title / meta[PubDate] + div.wenzhang_content_text 正文
N)datetime timedelta)quote)push_to_searchdbiG )daysz%Y-%m-%du" 诸暨市-建设项目环评公示z
User-Agentz6Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120.0zhttps://www.zhuji.gov.cnz7/api-gateway/jpaas-publish-server/front/page/build/unit 0 c t j | t dd }dddddd d
|d } t j t
|d t d
}d|_ |j }|j di j dd }t j d|t j }g }|D ]k \ } }
} t j dd|
j }
t |
dk r;| j! d r| nt" | z } |j% | | |
d m |S # t $ r}t d| g cY d }~S d }~ww xY w)N)pageNopageSizeF) ensure_ascii bulidstatic2745TM6ldEAz1I8xTXi6VHiB0columnu 右侧list内容null
1229857087) parseTypewebIdtplSetIdpageTypetagIdeditTypepageId paramJson r )paramstimeoutheadersverifyutf-8datahtml z API error: zv]*href="([^"]+)"[^>]*>(?:]*>)?([^<]{2,200})\s*(\d{4}-\d{2}-\d{2})\s+ http)urlpub_datetitle)jsondumps PAGE_SIZErequestsgetAPIHEADERSencoding ExceptionprintrefindallDOTALLsubstriplen
startswithBASEappend)
page_no
param_jsonr rr# r$ eitemsresultshrefr+ pdr) s
%/root/gov_crawler/crawl_zhuji_hjsp.py
fetch_listrH s: wI FUZ[J"V+#< F LLVWgV[ \
vvxxx#''3
JJ B bii
E G EeRvr5)//1 u:> oof-d4$;sUCD E N
s#$ s AD E(D=7E=Ec d t j || }|sy| j d|j }|dk( ry|dz }d}t j d| |d D ]J }|j d j
d r|dz
})|dz }|dk( s4| |||j z c S | |d S ) u 平衡 div 匹配: 从匹配到的 的 '>' 之后, 到对应 , 返回内部 HTML。
修复非贪婪 (.*?) 在正文含嵌套 div(表格/附件区)时截断的问题r% > z]|Nr z]*class="[^"]*wenzhang_content_title[^"]*"[^>]*>(.*?)z<[^>]+>z \s*([^<]+?)\s*r& u [-–—|_]r z6]*class="[^"]*wenzhang_content_text[^"]*"[^>]*>z1]*class="[^"]*wenzhang_content[^"]*"[^>]*>z src="(?!https?://|//)(/[^"]+)"c 8 dt | j d dS )Nzsrc="rL ")r= rQ )rW s rG z fetch_detail..j s" U4&QSQYQYZ[Q\P]]^C_ rY )ranger/ r0 r2 r3 r6 rM textrQ r8 r9 r: splitrX )r) _attemptrA r+ rS rF contentcs rG fetch_detailrf M s !H S'75QA AJ y E
;QVVDA CR 2B
UWXW]W]_a_h_hiAz2qwwqz288: II6 J FF62qwwqz288:EHH^U3A6<<>EG QVV%^_Awwy'') ==? )] ^ 779ggiGff68_ahiG "g 1 As $HH r c g }t }t | t }t | D ] }t |dz }|st d|dz d nt d|dz dt
| d |D ] }|d r
|d t k r|d |v r |j |d t |d \ }}} |s|d }| j s[|j ||d |d |xs |d t | d d
t d t
| d |D
cg c] }
|
d j s|
} }
| r t | d
t
| S c c}
w )NrL z Page z: 0 items (stop)z: z itemsr* r) r+ r% )r+ r)
source_urlr* site_namerd summaryz Total: rd
zhuji_hjsp)setmin TOTAL_PAGESr` rH r5 r; CUTOFFaddrf r: r> SITE_NAMEr ) max_pagesrecordsseenpagespgrC itr+ rF rd rA valids rG runry m sn G 5D ; 'EEl 26" GBqD6!12 3
1vRE
|623 B*~"Z.6"9%yD HHRY !-bi!8 E2w7 ==? NN r%y5 0"Z. &7r
,
Ic'l^6
*+ 811Y<#5#5#7Q 8E 8 - u:
9s E -E __main__z--pages)typedefault)rr zDone: z records)r ))__doc__sysosr/ r6 argparser, r r urllib.parser pathinsertdirnameabspath__file__ crawler_libr nowstrftimero rq r2 r= r1 r. rn rH rX rf ry __name__ArgumentParserap add_argumentint
parse_argsargsru cntr5 rY rG r s
- , , ( 277??277??8#<= > (
(,,.9%0
0 : :: F 0 Q
R!
FF :( @ @ z "BOOICO3
==?D
#C F3%x
! rY