yj d Z ddlZddlZddlZddlZddlZddlZddlmZ ej j dej j ej j e
ddlmZ dZdZdZedz Zed z Zd
Zd Zed d
Z ej0 dej2 Zd Zd Z d Z d Z d Z e dk( r e yy)uZ
唐山三友化工股份有限公司 - 公司公告 (channels/1738.html) 爬虫
列表: http://www.sanyou-chem.com.cn/sygfx/channels/1738.html
GB2312 编码, 每页9条
列表结构: 日期
公告为纯 PDF 外链 (static.sse.com.cn 上交所披露平台), 无详情页
分页: 1738.html / 1738_2.html ~ 1738_148.html
正文格式(用户要求): 标题 + URL 内嵌
公告标题
PDF名称
用法: python3 crawl_sanyou_gsgg.py [--pages=N] [--start=N]
N)urljoin)push_to_searchdbz http://www.sanyou-chem.com.cnu 唐山三友化工-公司公告u 公司公告z/sygfx/channels/1738.htmlz /sygfx/channels/1738_{}.html zoMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36zzh-CN,zh;q=0.9)z
User-AgentzAccept-Languagezp\s*\s*([^<]+?)\s*
c 6 d} d}t j dd D ]| }|j d r t |j d d } 2|j d r t |j d d }a|j srt | } ~ | |fS )Nr z--pages==z--start=)sysargv
startswithintsplitisdigit)pagesstartas &/root/gov_crawler/crawl_sanyou_gsgg.py
parse_argsr % s
E
E
XXab\ <<
# Q(E
\\*
% Q(E
YY[FE
%< c t j dd| xs d } t j dd| } | j S )Nz\s+ u
^[•·\-\s]+)resubstrip)ts r clean_titler 2 s7
vsAG$A
Q'A 779r c | j |d }|j dk7 ry|j j dd S # t $ r Y yw xY w)N )timeout r gbkignore)errors)get status_codecontentdecode Exception)sessionurlrs r fetchr, 8 sT KKRK ( ==C yy h 77 s "A A A
Ac
g }t j | D ]* }|j d j }t t j dd|j d }|j d j }|r|sv|j d st t | }|j j d s|j }t j d| }|r7|j d d |j d d |j d }|j |||d
- |S ) Nr z<[^>]+>r httpz.pdfz(\d{4})-(\d{2})-(\d{2})-)titler* date)LIST_REfinditergroupr r r r r r BASE_URLlowerendswithsearchappend)htmlitemsmhrefr2 r3 dms r
parse_listrA B s E
d
# Bwwqz !BFF:r1771: >?wwqz !D v&8T*Dzz|$$V, zz|
YY14
8
!#!bhhqk288A;GD
uT4@A!B" Lr c ^ t \ } }| dk( rt } t dt d| d| d t j
}|j j t g }t }t || dz D ] }|dk( rt }nt j | }t || }|rt | dk rt d | d
d nt! | }|st d | d d n}d} |D ]4 }
|
d |v r |j# |
d |j% |
| dz
} 6 t d | d
t | d| dt | d t' j( d t dt | dd g } |D ]y }
|
d j+ d d } d|
d d|
d d| d}
| j% t |
d |
d |
d |
d |
|
d t, t/ j0 | |
d dgd d { t d t | d d t3 | t ! t5 d" | D }| D cg c]
}|d# s |d# }}t d$t | d%| d t d&|r t7 | nd' d(|r t9 | nd' d y c c}w ))Nr z[SANYOU-GSGG] SITE=z start=z max_pages=T)flushr i z Page u : 空/失败, 停止u : 无公告, 停止r* z: u 条(新u ) 累计g333333?u
共 u' 条, 生成正文(标题+URL内嵌).../zr2 z
z
r3 )namer* F) ensure_ascii) site_name
source_urlr* r2 pub_dater& summarycategory attachmentsz
pushing z items...) batch_labelc 3 h K | ]* }|j d xs dj r'd , yw)r& r r N)r$ r ).0r+ s r zmain.. s) Ka155+;+Ar*H*H*JKs (22rJ z DONE. pushed=u 空正文=u 日期范围: r1 z ~ ) r MAX_PAGESprint SITE_NAMErequestsSessionheadersupdateHEADERSsetrangeLIST_URLPAGE_URLformatr, lenrA addr; timesleepr
CATEGORYjsondumpsr summinmax) max_pagesr r) all_itemsseenpgr* r< r= new_countitresultspdf_namer& emptyr+ datess r mainrs X s !|IuA~ {'% I;
OW[\ G OO7#I 5DE9q=)
7C//"%CWc"s4y3 GB4 34D A 4 GB4 234 @ B%yD HHRY R NI t2c%j\ 8C NCST\`a
3+.
F3y>""I
JRVWG e9??3'+"W+ e9+Rz
; " U)e9[6
'{ ::E'K&L[`a
&
Js7|nI
.d;W)4 K7K KE$+ =qq}Qz] =E = OCL>UG
s
J*J*__main__)!__doc__osr r ra rd rU urllib.parser pathinsertdirnameabspath__file__ crawler_libr r7 rT rc r\ r] rR UArY compileSr4 r r r, rA rs __name__ r r r s
277??277??8#<= > ( * - 1 1 4 4 v0@
A
"**wDD
,;j| zF r