Kjf* d Z ddlZddlZddlZddlZddlZddlZddlZddl Zddl
Z
ej j dej j ej j e ddlmZ dZdZdZdZdZ e
j. Zd Z eej6 D ]k \ Z Z e j= d
r e e jA d d d Z4e d k( s:e d z e"ej6 k sS e ej6 e d z Zm ed k\ rend Z#dd
Z$d Z%d Z&d Z'd Z(d Z)d Z*d Z+e,dk( r e+ yy# e!$ r Y w xY w# e!$ r Y w xY w)uK
常州市生态环境局 — 建设项目环境影响评价审批 (sthjj.changzhou.gov.cn)
列表: /class/ALFJELKI (第1页) + /class/ALFJELKI/N (N=2..6, 30条/页)
标题 + td 日期
详情: /html/hbj/YYYY/ALFJELKI_MMDD/NNNN.html
标题: meta name='ArticleTitle' | 日期: meta name='PubDate'(取前10)
正文: td.NewsText (含表格)
入库: crawler_lib.push_to_searchdb → /root/search.db (FTS触发器自动维护)
注意: 服务器 curl 直连可用, 无需 playwright; meta 用单引号
N)push_to_searchdbu1 常州市生态环境局-建设项目环评审批z https://sthjj.changzhou.gov.cnALFJELKIz
2018-01-01zoMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36 z--pages==z--pagesc
t | D ]r } t j j | dt i }t j j |dt }|j j dd c S y # t $ rM}||dz
k( r&t d | d d
d t | d d Y d }~ y t j d
Y d }~d }~ww xY w)Nz
User-Agent)headers )timeoutcontextzutf-8replace)errorsr z
[HTTP-ERR] F z : 2 )rangeurllibrequestRequestUAurlopenCTXreaddecode Exceptionprintstrtimesleep)urlretriesireqrespes */root/gov_crawler/crawl_changzhou_sthjj.pyhttp_getr& ' s
7^ ..((|R6H(IC>>))#r3)GD99;%%gi%@ @ GaK
c#2hZs3q6#2;-@A JJqMM s A-B C (C9CCc N | dk rt dt S t dt d| S )Nr z/class//)BASECOLUMN)pages r% list_urlr, 5 s1 qywvh''V76(!D6 ** c g }t j d| t j D ] }|j d j |j d j }}|rd|vrH| j dd|j
}| j d|j }|dk\ r
|dkD r| || nd}t j d | }|r|j d nd} |j t |z || d
t }
g } |D ]/ } | d |
vs |
j | d | j | 1 | S ) NzH]*href='(/html/hbj/\d{4}/ALFJELKI_[^']+\.html)'[^>]*title='([^']*)'r r htmlz z(\d{4}-\d{2}-\d{2}))r titledater )refinditerSgroupstriprfindstartfindendsearchappendr) setadd)
html_textitemsmhrefr1 tr_starttr_endsegm_dater2 seenuniqits
r%
parse_listrK ; s5 E [[S244
I ggaj&&(!''!**:*:*]*class=['\"]NewsText['\"][^>]*>(.*?)\s*r z*]*class=['\"]NewsText['\"][^>]*>(.*)z r zr0 )r3 r< r5 r6 r4
startswithr9 )r@ rB m2rF depthmms r% extract_news_textrQ W s
MyZ\Z^Z^_Awwqz @)RTT RB hhqk++nc2 Bxx{%%g.
A: {
++
r- c t j dd| t j t j z } t j dd| t j t j z } t j dd| t j } dD ] }| j | }|dkD s| d | } | j S )Nzr0 flagsz
)u 版权所有u 主办单位u 网站标识码u 苏ICPu 联系我们u 无障碍u 网站地图u 常州市人民政府r )r3 subr5 Ir: r7 )bodykwr! s r%
clean_bodyrZ j s
66("d"$$+ FD
66&Drtt DD
66-T 6DE IIbM q58D
::<r- c ~ d}dD ]P }t j || }|s t j dd|j d j } n d}dD ]> }t j || }|s |j d j d d } n t | }|sy t
| }|sy |||dS ) Nr0 )z9name=['\"]ArticleTitle['\"]\s+content=['\"]([^'\"]+)['\"]z9content=['\"]([^'\"]+)['\"]\s+name=['\"]ArticleTitle['\"]\s+ r )z4name=['\"]PubDate['\"]\s+content=['\"]([^'\"]+)['\"]z4content=['\"]([^'\"]+)['\"]\s+name=['\"]PubDate['\"]
)r1 pub_daterX )r3 r< rV r6 r7 rQ rZ )r@ r1 patrB r_ rX s r% parse_detailra v s EN IIc9 % FF63
399;E HI IIc9 % wwqz'')#2.H Y 'D d D $ ??r- c
| sddg fS t j dd| t j } t j dd| t j t j z } t j dd| t j t j z } | }g i fd}t j d||t j t j z }t j d ||t j t j z }fd
}t j d ||t j t j z }t j d ||t j t j z }g fd
}t j d||t j t j z }t j d| t j rdnd}t j dd|t j }t j dd|t j }t j dd|t j }g }|j d D ] }|j
} | st j dd| }
|
j dd j dd }
t j dd|
}
|
sij D ] \ } } | j | | } t D ] \ }
}| j d |
d | } |j | j
dj | }j D ] \ } } |j | | } t D ] \ }
}|j d |
d | } t j d d| }t j d d| }t j d d| }t j d!d| }t j d"d| }t }g }|j d D ]W }t j d#dt j dd| }|r||v r6|j | |j | Y dj | j
}||fS )$uL NewsText 正文 → 存储格式: 附件绝对化 + 表格保留 + 段落化r0 r rU rS z]*>.*?z]*>.*?c < | j d }| j d }t j dd| }|j }t j dd| j }|s5t j
j
|j d d xs d}t j j | }t j d |t j }|rj ||f d
t d }d | d
| d|< |S )Nr r <[^>]+>r0 u ^附件[:：]\s*?r u 附件z:\.(pdf|docx?|xlsx?|pptx?|zip|rar|wps|et|dps|txt|ofd)(\?|$)__LINK____ z
)r6 r3 rV r7 ospathbasenamesplitr parseurljoinr< rW r= len)
rB rC innertxtabs_url is_attachkey attachments link_protectpage_urls
r%
_link_replz html_to_text.._link_repl s wwqz
ffZU+iikff("c288:''""4::c?1#56B(C,,&&x6II []dfhfjfjk ~ .\*+2. *7)3EcU( S S
r- z&]*href="([^"]+)"[^>]*>(.*?)z&]*href='([^']+)'[^>]*>(.*?)c | j d }t j j | }t j
d|t j ryd| dS )Nr z'ueditor|editor|icon|logo|qrcode|ewm|btnr0 rh u& " target="_blank">查看图片)r6 r rm rn r3 r< rW )rB srcabs_srcrw s r% _img_replz html_to_text.._img_repl sN ggaj,,&&x5
99?"$$ O gY&LMMr- z ]*src="([^"]+)"[^>]*>z ]*src='([^']+)'[^>]*>c h j | j d dt dz
dS )Nr __TBL__r rg )r= r6 ro )rB
table_protects r% _tbl_replz html_to_text.._tbl_repl s3 QWWQZ(]+A-.b11r- z]*>.*?z ]*>r z
z
z(]*>z
rd r] z z [\s\u200b\u200c\u200d\ufeff]+r~ rg z
__LINK__\d+__z __TBL__\d+__z[ \t]+\nz\n{3,}z [ \t]{2,}r\ )r3 rV r5 rW r< rl r7 r rA enumerater= joinr> r? )contentrw content2rx r| r has_tablepartsblockbplainkvr! tbloutrH finalprt ru rv r s ` @@@r% html_to_textr s 1byff]BrttAIvvgz8244@HvvlD("$$?HvvA2xWYW[W[\H E '
KKM z2q)
fc*228SA7UC &&( DAq !QA
. 0FAs GA3b/3/A 0
QWWY
++e
C ""$ 1kk!Q M* 03kkGA3b/3/0
&&!2s
+C
&& "c
*C
&&dC
(C
&&FC
(C
&& sC
(C 5D E
YYv
ffVR
B!:;cTk
Q ++e
"
"
$C ; &&r- c | t dt dt d g } t dt dz D ] }t | }t | }|st d| d +t
| }t d| dt | d |s|dkD r nt[d
}|D ]( }|d r|d t k rd } n| j | * |rt d
| dt n+t j t j dd t dt | d | sy g }t | d D ]v \ }}t |d }|s$t d| dt | d|d d d ;t | } | s$t d| dt | d|d d d l| d xs |d }
| d xs |d } t! | d |d \ } }
}t t# j$ dd | j' }d | v }|d k r(|s&t d| dt | d |
d d d |
d!} |j t |
| |d | t) j* |d
" d# t d| dt | d$|
d d% d& t j t j d'd( y t- |d)* }t d+| d, y )-Nu 🚀 u — 爬取 u 页r z [pu ] HTTP失败u 第u 页: u 条Fr2 Tu [CUTOFF] 第u 页出现 < g ?g333333?u
[LIST] 共 r z [r( z] [ERR] r1 ( z ] [NO-BODY] r_ rX rd r0 zr ) ensure_ascii) site_namer1 r_ r r ru u
] 最新: 7 z...g333333?g?changzhou_sthjj) batch_labelzDone: z records)r SITE_NAME_PAGESr r, r& rK ro CUTOFFr= r r randomuniformr ra r r3 rV r7 jsondumpsr ) items_allpgr r@ rA
cutoff_hitrJ recordsr! dr1 r_ r r ru plain_len has_linksnew_cs r% runr s E)L
56IAvz" -rl SM DL) * 9%
bTs5zl$/0Av
!B&zbj6 1 !
R !
N2$l6(; <
6>>#s+,--0
M#i.)
./ G9a( -2 RY' Cs!C N+8BwK4D3EF G # Cs!C N+<7 CR8H7IJ K '
)bkZ=.BvJ*6qy"U)*L'Kz2w7==?@ ( r>) Cs!C N+:eCRj\B CE7$'G"Ue9KfkAl
A3aI'z%* SAB
6>>#s+,--0
W2C DE F5'
"#r- __main__) )-__doc__sysri r3 r r urllib.parser r urllib.requestsslrj insertdirnameabspath__file__ crawler_libr r r) r* r r _create_unverified_contextr
_MAX_PAGESr argvr! arM intrl
ValueErrorro r r& r, rK rQ rZ ra r r __name__ r- r% r sm J I I I I 277??277??8#<= > ( ? ' v$c$$&
chh
DAq||J QWWS!_Q /0J
i AECM 1 SXXa!e_-J
"QA + 8& @0Z'z9$x zE U
s$ E>EEEEE