wjU d Z ddlZddlZddlZddlZddlZddlZddlmZmZ ddl m
Z
ej j dej j ej j e ddlmZ ej$ ed z
j' d Zd Zd
Zd d d
ZdZedz ZdZddZd Z d Z d Z d Z e dk( r e yy)u0 crawl_ldnews_hjbh.py - 娄底新闻网-环境保护 (湖南娄底)
站点: https://m.ldnews.cn/pccategory/100.shtml (移动端栏目页)
CMS: cmstop (AJAX JSON API 分页)
列表: ⚠️ 非静态 HTML — POST https://m.ldnews.cn/pccategory/100.shtml
data: {catid:100, page:N, pagesize:20} → json{total:780, data:[{title,url,published,...}]}
首页静态渲染 P1 (20条), AJAX 从 page=2 起; total=780 → 39页
CUTOFF 边界: P6 末条 2023-08-09 (3年窗口=前6页≈120条)
注意: url 有两种格式 — m.ldnews.cn/pcarticle/ID.shtml 与 news.ldnews.cn/news/YYYYMM/ID.shtml
详情: PC端 http 301→https; 标题 h1.ui-article-title / 分割; 日期 <time> 文本
正文双容器: PC端 div.right_body 平衡div (尾部登录区清洗) | 移动端 div#content-show
N)datetime timedelta)urljoin)push_to_searchdbiG )daysz%Y-%m-%du 娄底新闻网-环境保护u 湖南zoMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36z(https://m.ldnews.cn/pccategory/100.shtml)z
User-AgentRefererzhttps://m.ldnews.cnz/pccategory/100.shtmld c |r t j | |dt d }n t j | dt d }d|_ |j
S # t $ r+}t d| d| t j Y d }~y d }~ww xY w)
N Fdatatimeoutheadersverify)r r r utf-8z[WARN] fetch failed: z - file )
requestspostHEADERSgetencodingtext Exceptionprintsysstderr)url post_datares &/root/gov_crawler/crawl_ldnews_hjbh.pyfetchr& ! st
c 7G\abA S'75QA
vv
%cU#aS1
Cs AA B !BBc g }|dk( rt j d| t j D ] }|j d t j dd|j d j }}t
| dk rW|j d r|nt t | }|j dd }|j ||d
|S t j t t |d d d
t d }d|_ t% j& |j( }|j+ dg D ] } | j+ d xs dj dd j }| j+ d xs dj dd }|r|rt
| dk rn|j ||d
|S # t, $ r,}
t/ d| d|
t0 j2 Y d}
~
|S d}
~
ww xY w)uP 列表条目: AJAX JSON API 解析 (首页静态 HTML 用正则, 分页用 JSON) zK<a href="([^"]+)"[^>]*>\s*(?:<img[^>]*>)?\s*<h3 class="ui-title">(.*?)</h3><[^>]+>r httpzhttp://zhttps://)r! titler
catidpagepagesizer Fr r r r- r! z[WARN] API page z failed: r N)refinditerDOTALLgroupsubstriplen
startswithr BASEreplaceappendr r LIST_URLCATIDr r jsonloadsr r r r r r ) htmlpage_noitemsmhrefr- r! r# ditr$ s r%
extract_itemsrI . s E!| kmqsus|s|} 7A''!*bffZQWWQZ&H&N&N&P%D5zA~ //&1$wtT7JC++i4C LLu5 6
7 I MM(5'WY)Z")75
J
JJqvv %%# 7BVVG_*33HbAGGIE66%=&B// :FCs5zA~ LLu5 6 7 L I
1#6SZZHH LIs C&F; ; G0!G++G0c t j || t j }|sy| j d|j }|dk( ry|dz }d}t j
d| |d t j D ]j }|j
d j d s |j
d j d r|dz
}I|dz }|dk( sT| |||j z c S | |d S )
u& 平衡 div 匹配 (大小写不敏感)r >r( z<div[\s>]|</div>Nr z<divz<DIV)r3 searchIfindstartr4 r6 r: )rB open_tag_rerE gtidepthmms r% extract_balanced_divrV L s
+tRTT*A
3 "B Rx
QA
Ekk-tABx> .
88A; ! !& )RXXa[-C-CF-K QJE QJEzAa"((*n--
. 8O c n t | }|syd}t j d|t j }|r4t j dd|j d j
}|st j d|t j }|r[t j dd|j d j
}t j d| d j
}|j d
d j
}t j d d| }d}t j d | }|r|j d }|s)t j d
| }|r|j d }|sPt j d| }|r8t j d|j d }|r|j d }d}t |d }|j
r|j
}nFt j d|t j }|r |j d j
}dD ]1 }|j | } | d kD s| t | dz kD s,|d | } n t j dd|t j t j z }|||fS )N)r r r r z:<h1[^>]*class="[^"]*ui-article-title[^"]*"[^>]*>(.*?)</h1>r) r( z <title>\s*([^<]+?)\s*z\s+u _娄底新闻网|_|[-–—]r r2 u ^\s*(?:·| |·|\s)+z ]*>(\d{4}-\d{2}-\d{2})z!(\d{4}-\d{2}-\d{2})\s+\d{2}:\d{2}z4]*name="[Pp]ub[Dd]ate"[^>]*content="([^"]*)"z(\d{4}-\d{2}-\d{2})z+]*class="[^"]*right_body[^"]*"[^>]*>zO]*id="[^"]*content-show[^"]*"[^>]*>(.*?)\s*(?:|)flags)
r& r3 rM r5 r7 r6 r8 splitr< rV rO r9 rN )
r! rB r- rE date_textmdcontentckwidxs
r% fetch_detailra ` s< :D E
OQUWYW`W`aAz2qwwqz288: II6bii H FF62qwwqz288:EHH B HHQK G T#QRAwwy'') IIhjnprpypy z ggaj&&(Gg ll2 7sS\C // dsmG ff+R BDD@PQG )W $$rW c L t j } | j dt d | j }g }t }t
d|j dz D ] }|dkD r
t |ddnd }t t | }|s't || }t | }|D ]/ } | d |v r |j | d |j | 1 t d| d t | |z
d
t | d |r n g }
|D ]A } t! | d \ } } }
| s| d } |
st d
| d 1| }|r
|t" k r?g }t% j& d|
t$ j( D ] }|j+ d t% j, dd|j+ d j/ }}t% j0 d|t$ j2 sd|j5 v s|j |xs d|d |
j | | d | d |t6 |
d|rt9 j: |d ndt< d D t dt |
d |
D cg c] }|d j/ s| }}|r t? |d t | S c c}w ) Nz--pages )typedefaultr( r
r. r! z page z: +z items (total )r- z SKIP empty content: z$]*href="([^"]+)"[^>]*>(.*?)r) r r* z*\.(pdf|doc|docx|xls|xlsx|zip|rar|caj|wps)$downloadu 附件)r- r! F) ensure_asciiz[]) r- r!
source_urlpub_date site_namer] summary attachments
group_namez Total: z itemsr] ldnews_hjbh) argparseArgumentParser add_argumentint
parse_argssetrangepagesr? r& r> rI r9 addr= r ra CUTOFFr3 r4 r5 r6 r7 r8 rM rN lower SITE_NAMEr@ dumps
GROUP_NAMEr )apargsrD seenrC r rB
page_itemsbeforerH recordsr- r[ r] pdattsamhtr# valids r% mainr s "BOOICO3
==?D E 5DDJJN+ DKaKR@UYXt$ "41
U B%yD HHRY LL
yCJ$7#8s5zlRSTU G $0E$;!y'wKE ,RYK8 9
"v+ ++ EwPRPYPYZ @B88A;z2rxx{ C I I KqAyyF244PT^bcbibibkTk am8A >? @ 2e9BuIwRV$**T*N\`$
! ,
Ic'l^6
*+ 811Y<#5#5#7Q 8E 8
. u:
9s )J!J!__main__)N)!__doc__r osr r3 rp r@ r r urllib.parser pathinsertdirnameabspath__file__ crawler_libr nowstrftimery r{ r} r r; r> r? r& rI rV ra r __name__ rW r% r s - , , ( 277??277??8#<= > (
(,,.9'2
2 <