rxj 4 d Z ddlZddlZddlZddlZddlZddlZddlZddlZddl m Z m
Z
ddl m Z dZ
dZdZdZd Z e j$ e
d
z
j' d Z ed
e ej, ddZd Zd Zd Zd Ze dk( r e yy)u
深圳市生态环境局 meeb.sz.gov.cn — 环保业务栏目 (xxgk/qt/hbyw)
CMS: NFCMS (广东南方网系) + 广东省统一搜索平台 JSONP API
数据源: https://search.gd.gov.cn/jsonp/site/755014 (site=755014 = meeb NFCMS_SITE_ID)
?callback=datacallback&category_id=24525,...,24514&order=1&text=&pagesize=15&page=N&including_link_doc=1
列表: JSONP 直接返回 {count, results:[{title,url,post_url,pub_time,content(截断摘要),attachment:[...]}]}
详情: curl 详情页 → meta ArticleTitle / article_info 发布日期 / div.article 正文 / div.article_pdf 附件
注意: --curves prime256v1 必须 (服务器 TLS 曲线协商失败 bad ecpoint)
count=407, 15/页=28 页; CUTOFF 边界 P13 (2023-08-04 末条 < 2023-08-09)
N)datetime timedelta)urljoinzoMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36u% 深圳市生态环境局-环保业务zG24525,24524,24523,24522,24521,24520,24519,24518,24517,24516,24515,24514z*https://search.gd.gov.cn/jsonp/site/755014z/root/search.dbiG )days%Y-%m-%du [i] CUTOFF 边界: filec dddddt | dt g}|r
|dd| gz
}|r |d |d
|gz
}|d | gz
}t d D ]O } t j |d
d
|dz }|j
r|j
c S t j d Q y# t $ r(}t d| t j Y d }~Dd }~ww xY w)Ncurlz-skz--curves
prime256v1z-mz-Az-Hz Referer: z-cz-bz-L T
)capture_outputtexttimeoutz
[!] curl err r ) strUArange
subprocessrunstdout Exceptionprintsysstderrtimesleep)urlreferercjr cmdattemptres $/root/gov_crawler/crawl_meeb_hbyw.pycurl_getr( s 5*lD#g,b
QC '+,, b$##D#;C8 8s4dGVXLYAxxxx
1
8 M!%CJJ 7 7 8s 3B C C Cc t dt d| d}t | }t j d|t j
}|sy t
j |j d S # t $ r Y y w xY w)Nz#?callback=datacallback&category_id=z &order=1&text=&pagesize=15&page=z&including_link_doc=1z^[^(]*\((.*)\)\s*$ )
API_URL CATEGORY_IDr( researchSjsonloadsgroupr )pager outms r' fetch_jsonpr6 - sz
I8
Eefjek lA BC
3-C
'bdd3A zz!''!*%% s #A, , A87A8c t | d }|sdddg fS d}t j d| }|r |j d j }|stt j d|t j
}|rM|j d j j
dd j
d d j }d}t j d
| }|r|j d }d}t j d |t j
}|s%t j d |t j
}|ra|j d }t j d
| D cg c] } | }
} d}t j d|t j
D ]c } t j dd| } t j dd| } t j dd| } | j } | s[|d| dz
}e |s]t j dd| } t j dd| } t j dd| } | j } | rd| d}|s?|
r=|
D ]( }
|
j d s t d|
}
|d|
dz
}* |j }g }t j d|t j
}|rt j d|j d t j
D ] }|j d j }t j dd|j d j }|s t j j | }t j d |t j s|j d s t d| }|j! d | d | d ||||fS c c} w )!u. 返回 (title, pub_date, content, attachments)z$https://meeb.sz.gov.cn/xxgk/qt/hbyw/)r! Nr z+(.*?)u _深圳市生态环境局u -深圳市生态环境局u" 发布日期：(\d{4}-\d{2}-\d{2})z2(.*?)
(.*?)
\s*
]+src="([^"]+)"z
]*>(.*?)z<[^>]+>z [ \t\u3000]+ z\s+zz
zhttpzhttps://meeb.sz.gov.cnz
z$(.*?)
z$]+href="([^"]+)"[^>]*>(.*?)r z5\.(pdf|doc|docx|xls|xlsx|zip|rar|wps|et|txt|jpg|png)$z z
)r( r- r. r2 stripr/ replacefindallsub
startswithr finditerospathbasenameIappend)r htitlemtpub_datedmcontentr5 seguimgsptxtsrc attachmentsahrefnames r' parse_detailrU 9 sa DEA T2r!! E A1 EB !!#
YY.244
8
HHQK%%'//0KRPXXYtvxy BEH 8! stopr z] u 条 (count=count)post_urlr r pub_timer u ] 跳过窗口外 r8 rF z
r )rF rH rJ r rY g?z] new=z skip=g ?z DONE new=)argparseArgumentParser add_argumentint
parse_argsr startpagesr6 getr r r lenCUTOFFrU r: joinrD r strptime timestampr r ri )apargs total_new
total_skippgdro rc itr pubrF dpubrJ attsnss r' mainr s "BOOICO3OOICO3
==?D I
DJJ
TZZ 78 $ Oi( Brd/* < I,
2$bW l155>2B!D3::V B&&$;u
;C66*%+Sb1CcV|2$ 0Qrvvgb7I#27N6OPWZWaWab)5c): &E4$w+1134!F*TYYt_< ))D/ NN DWS !2!24!D!N!N!P Q
JJsO- . #G,DAq NI !OJ Brd&6!-CJJ ?
3I$J
Ii[zl
34rj __main__)NNrt ) __doc__rv r0 r@ r- rZ r r r r r urllib.parser r r_ r, r+ r\ nowstrftimer r r r( r6 rU ri r __name__ rj r' r s > = = = ( v 3 W
6
(,,.9'2
2 <