zj) t d Z ddlZddlZddlZddlZddlZddlZddlZddl m
Z
ej j dej j ej j e ddlmZ dZedz ZdZdZd Zd
Zd d d
edz dZdZd Zd Zd Z d Z d Z d Z d Z d Z!e"dk( r e! yy)u
福建省工业和信息化厅 - 项目核准审查 爬虫
栏目: https://gxt.fujian.gov.cn/zwgk/jsxm/xmhz/
CMS: TRS WCM (list2.trsapi 动静结合) — was5 检索 JSON API
列表 API: /was5/web/search?channelid=229105&searchword=chnlid%3D61090&perpage=15&outlinepage=1&page=N&sortfield=-docreltime
channelid=229105 (SiteInfo.searchId), chnlid=61090 (PageInfo.chnlid 栏目ID)
total=552 条, 15条/页 (37页), sortfield=-docreltime 时间倒序
docs[] 字段: title(完整)/url/chnldocurl/content(截断~260字符)/pubdate/pubtime/file(附件P0)/filedesc/fileno(文号)
⚠️ 每页返回 16 条含 1 条坏条目 (url 非 http, date 空) → 过滤 url startswith http + pubdate 非空
详情: /zwgk/jsxm/xmhz/YYYYMM/tYYYYMMDD_ID.htm
正文容器:
⚠️ class=TRS_Editor 无引号; 内层 DIV 大写; style 块须剥; 段落 保留
附件: API file 字段 (P02026...pdf) + 详情页 article_attachment 区 下载：xxx.pdf
⚠️ 附件相对路径 urljoin(详情页URL, href) 绝对化 (保留栏目路径段)
标题: API title 字段完整 (福建省工业和信息化厅关于XX项目节能报告的审查意见)
用法:
python3 crawl_fjgxt_xmhz.py --pages=1
python3 crawl_fjgxt_xmhz.py --pages=16 # CUTOFF 3年窗口 (2023-08-11 起)
python3 crawl_fjgxt_xmhz.py # 默认 5 页增量
N)urljoin)push_to_searchdbzhttps://gxt.fujian.gov.cnzs/was5/web/search?channelid=229105&searchword=chnlid%%3D61090&perpage=15&outlinepage=1&page=%d&sortfield=-docreltimeu% 福建省工信厅-项目核准审查u 项目核准审查z
2023-08-11% zoMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36z.application/json, text/javascript, */*; q=0.01zzh-CN,zh;q=0.9z/zwgk/jsxm/xmhz/)z
User-AgentAcceptzAccept-LanguageReferer c d} t j dd D ] }|j d r t |j d d } 2|j d sDt t j t j j
| dz kD st t j t j j
| dz } | S )Nr z--pages==z--pages)sysargv
startswithintsplitlenindex)pagesas %/root/gov_crawler/crawl_fjgxt_xmhz.py
parse_argsr / s
E
XXab\ 9 <<
# Q(E
\\)
$SXX 9JQ9N)N !2Q!678E 9
L c t j | xs d } t j dd| } t j dd| } t j dd| } | j S )N z [\u200b\u200e\u200f\ufeff\xa0]z\s+ u ^[•·\-—]\s*)html_modunescaperesubstrip)ts r clean_titler! 9 sZ !'r"A
0"a8A
vsA A
"B*A 779r c t | z }t d D ]k } t j |t t
}|j dk( r#t j |j d c S t j d
m y # t $ r6}|dk( r't d| d| t j Y d }~ y Y d }~Rd }~ww xY w) N headerstimeout F)strict z [WARN] API pz failed: fileg ?)API_TPLrangerequestsgetHEADERSTIMEOUT status_codejsonloadstext Exceptionprintr stderrtimesleep)pageurlattemptres r fetch_apir@ A s
D.C8 S'7CA}}#zz!&&77 $
3 !|tfIaS9
K s AB C
&B>>Cc @ | j d r| S t || S )N)http://https://z javascript:zmailto:ztel:zdata:#)r r )upage_urls r _abs_urlrG P s! ||[\ 8Q r c | syt j dd| } t j dd| } t j dd| t j } t j dfd| } t j dfd | } t j d
d | } t j d d| } t j d
d | } t j dd | } t j dd| t j } t j dd| } t j dd| } t j dd| } | j S )ul 正文清洗: 保留
/
HTML, 附件/图片绝对化 (以详情页 URL 为基址保留栏目路径)r zzz ]*>)flagszz clean_content.._ s# )HQWWUVZYaDbCccd8e r zc , d| j d dS )Nz.g s AGGTUJ]*>z [\u3000\xa0]r z\s*(?: )?\s*
z\r\n
)r r Ir )raw_htmlrF s `r
clean_contentr[ V s vv12x@Hvv/X>Hvvor8244@Hvv,.egopHvv-/giqrHvvk5(3Hvv&H5HvvAC^`hiHvv;=Y[cdHvvBBXZX\X\]HvvosH5Hvv12x@HvvgtX.H >> r c 6 g }t }t j d| t j D ] }|j d t j
dd|j d j
}}t j d|t j sd|j v st || }||vs|j | |j |xs |j d d |d
|rt j |d S d
S )u8 从正文提取附件链接, 返回 JSON 数组字符串"]*>(.*?)r
<[^>]+>r r) z2\.(pdf|doc|docx|xls|xlsx|zip|rar|wps|7z|txt)(\?|$)download/namer< F ensure_ascii[])setr finditerSrM r r searchrY lowerrG addappendr r3 dumps)contentrF attsseenamahrefatextrE s r extract_attachmentsru q s
D 5Dkk?"$$O Kxx{BFF:r288A;$G$M$M$Ou
99JESUSWSW X\fjojujujw\w)A} U%>aggcl2.>q IJ
K 484::d /ATAr c t d D ]{ } t j | t t }|j
dk7 rdg fc S d|_ |j }g }t j d| }|rt j d|j d D ] }|j d }t j d
|t j s:t j d d|j d j xs |j d
d }|j! |t# || d t j d|t j$ } | r t' | j d | }
t j d|
t j$ D ] } | j d }t j d
|t j s:t j dd| j d j xs |j d
d }|j! ||d t) } g }
|D ]/ }|d | vs | j+ |d |
j! | 1 |
|
fc S d|fc S dg fS # t, $ r;}|d k( r,t/ d| d| t0 j2 dg fcY d}~c S Y d}~nd}~ww xY wt5 j6 d )u 抓详情页: 返回 (完整正文, 附件列表) — API content 截断时补全; 附件从 article_attachment 区 + 正文提取r# r$ r' r zutf-8zA\s*\s*z$]*>([^<]*)r r
z.\.(pdf|doc|docx|xls|xlsx|zip|rar|wps|7z)(\?|$)u ^下载[：:]\s*r) r` ra rb z#(.*?)r] r^ r< z [WARN] detail failed z: r* Ng ?) r- r. r/ r0 r1 r2 encodingr5 r rj rh rM rY r r r rm rG ri r[ rg rl r6 r7 r r8 r9 r: )r< r= r> html_textrp rr a2rs rt rO bodya3rq uniqa4r? s r fetch_detailr~ s 8 %# S'7CA}}#2v
AJID_ajkB ++&MrxxXY{[ RB HHQKEyy!RTY[][_[_` "':B @Q@Q@S T lX]XcXcdgXhikXl U8E3;O$PQ R @)RTTRA$QWWQZ5 ++&KTSUSWSWX CB HHQKEyy!RTY[][_[_` "z2rxx{ C I I K cu{{[^O_`bOc U5$AB C uRd =B%y, E+T[[_= Tz!t8OA%L r6M !|/uBqc:L2v
3s> 2JB JD J A8J*JJ K)KKKc
t } | sd} g }t }t d| dz D ] }t | }|st d| d n|j dg }|D cg c]A }t
|j dd j d s.|j d s@|C }}|st d| d
n?|D ] }|d j j d d }||v r,|j | t |j d
d } | s[d}
t j dt
|j d d } | r| j d }
|
r
|
t k rt! |j dd | } g }
t# t j$ dd| dk rUt' | \ }}|rCt# t j$ dd| t# t j$ dd| kD r|} |}
|
s|j dd }|rk|j d rZ|j dd xs dj dd j xs |j) d d }|t+ || dg}
|
rt- j. |
d nd}|j1 t2 | |
| |||dd t d| d t# | d t# | d t5 j6 d t d!t# | d" t9 |t: t d#t# | d$ y c c}w )%N r
z[PAGE z] API failed, stoppingdocsr< r httppubdatez] 0 valid items, stoppingrB rC titlez(\d{4}-\d{2}-\d{2})ro r^ i, r+ P0filedescu 下载：r` ra rb Frd rf ) site_namer pub_datero
source_urlr< attachmentsindustryz] +z valid (total )g333333?u
有效 u 条, 入库...u 完成! 新增/更新 u 条) r rg r- r@ r7 r/ strr r replacerl r! r rj rM CUTOFFr[ r r r~ r rG r3 rn rm SITE_NAMEr9 r: r CATEGORY)r all_itemsrq pgdatar dvalidr< r r pmro r rz rp ffnameatt_jsons r mainr s
LE I 5DAuqy! 3} F2$ 45 6 xx# bqCeR(8$9$D$DV$LQRQVQVW`Qabb F2$ 78 9 & AE(.."**9jACd{ HHSM gr 23EH13quuY7K3LMB 88A;Hv -#AEE)R$8#>G K266*b'23c9 )# .
dCz2t < =BFF:WY[bDc@d d "G " EE&"% d+ UU:r 2 8b A A+r R X X Z n^_^e^efi^jkm^nE,1(1c:J#K"LKFQtzz+EBW[H & $" ! '
;& N rd#c%j\ I7GqIJ
3g3j
Ic)n%_
56Y) "3y>"2$
78a cs #/L?L?%L?__main__)#__doc__osr r r9 r3 htmlr r. urllib.parser pathinsertdirnameabspath__file__ crawler_libr BASE_URLr, r r r MAX_PAGESr0 r1 r r! r@ rG r[ ru r~ r __name__ r r r s ( 1 0 0 277??277??8#<= > ( & E E 3 D>',,
6 B (V>9B zF r