j$dZddlZddlZddlZddlZddlZddlmZddl Z ddl Z dZ dZ dZ dZdZd Zej"j%d d Zd e d ZdZdZddZdZdZdZddZedk(rkddlZdZej>ddD]IZ e jCdr e"e jGdddZ1e jKsBe"e ZKeeyy#e$$rY^wxYw)u7 海科化工 - 公司新闻 站点: haike.haikegroup.com/hk_news.html CMS: 静态 HTML (hk_detailed/hk_news/ID.html) 列表: a[href=/hk_detailed/hk_news/3767.html] (标题在 a 内, 日期 2025年11月14) 详情: div#article 正文 分页: /hk_news/p/N.html 用法: python3 crawl_haike_gsgg.py [--pages=N | N] N)urljoinu海科化工-公司新闻zhttp://haike.haikegroup.comz(http://haike.haikegroup.com/hk_news.htmlzhaike.haikegroup.comu企业z 2023-01-01DB_PATHz/root/search.dbzoMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36)z User-AgentRefererctjdd|xsd}tjdd|}tjdd|jS)Nz$·| | |\u200b|\ufeffu ^[•·]\s*z\s+ )resubstrip)ts %/root/gov_crawler/crawl_haike_gsgg.py clean_titlersG 6AGDA ~r1%A 66&#q ! ' ' ))ctjd|xsd}|r[t|jdt|jdt|jd}}}|dd|dd|dSy) Nu-(\d{4})[年\-/. ](\d{1,2})[月\-/. ](\d{1,2})r04d-02d)r searchintgroup)smymods r normalize_dater"sk BAGLAqwwqz?C OS_q2C"S1S'** rct|D]D} tj|dtd}d|_|j dk(r|j cSFy#t$r!||dz krddl}|jdYqwxYw) NF)timeoutheadersverifyzutf-8rrr) rangerequestsgetHEADERSencoding status_codetext Exceptiontimesleep)urlretriesirespr.s r fetchr4)s 7^+ +<<ROD#DM3&yy ' +  +7Q;ZTZZ] +s>A'A>=A>c g}|s|St}ddddddddd d d d d }tjd|tjD]}|j d}tj d|tj}|s%tj d|tj}|sd|j dj }|jjdk(rd|vrt|j d} n/ttjdd|j d} t| dkr||vr|j|d} tj d|} | rct| j d|j| j dt| j d}} } | r |dd| dd| d} |jds t||n|}|j!| || |d|S)Nrrr ) JanFebMarAprMayJunJulAugSepOctNovDecz]*>(.*?)zO]+href="([^"]*hk_detailed/hk_news/\d+\.html)"[^>]*>]*title="([^"]*)"zA]+href="([^"]*hk_detailed/hk_news/\d+\.html)"[^>]*>(.*?)ztitle="<[^>]+>rz;

\s*(\d{1,2})\s*

\s*\s*([A-Za-z]{3})\s+(\d{4})rrrhttp)titler0pub_datehref_raw)setr finditerDOTALLrrr groupsrr lenaddrr( startswithrappend) html_textbase_urlitemsseenMONTHSrblockamhrefrMrNdmdaymonyr detail_urls r extract_list_itemsre5s E  5DAAAAAAATU\^egnp qF [[299b  YYikprtr{r{ |_afhjhqhqrB xx{  " 55<<1 e!3 ,Ez2rxx{ CDE u:>  4<   YYUW\ ] rxx{+VZZ -Dc"((ST+FVbC XQs3iqS :48OOF4KWXt,QU  uZX[_`a7b8 Lrc t}|syd}tjd|}|rt|j d}|sdtjd|tj }|r=ttj dd|j dj}d}tjd|tj}|rt|j d}dtjd|tj }|s%tjd|tj }|r]|j d}tjd |}|r|d|jn |jd }|d k7r|d|d zn|tj d dtjtjztj ddtjtjzg}tjdD]}|j djt|j d} } tjd| tjsb| jdr| n%| jdr t| zn t!| t#fd|Dr|j%| xs| j'dd dfd} tj d| tj d| tj ddtj dd|rDt#fd|D} | s.dj)d|D} j+dz| zt-tj d dj}d!v}d"vxsd#v}|d$kr |s |s |s|d|gd%S|||d%S)&Nrz/(.*?)u\s*[-_—]\s*.*$zG]+id="article"[^>]*>(.*)z]+id="content"[^>]*>(.*)z]+id="(?:footer|share|print)"|]+class="[^"]*(?:footer|share|print)[^"]*"|]*>|]+id="rightsidebar"zr8z]*>.*?)flagsz]*>.*?z,]*href="([^"]+)"[^>]*>([^<]{0,120})rz0\.(pdf|doc|docx|xls|xlsx|zip|rar|wps|7z|et|dps)$rL/c3.K|] }|dk(ywr0N).0aabs_urls r zparse_detail..s9qqx7*9s)namer0c|jdj}|jds"|jds|jdr|jdS|jdr$|jdj|d|zS|jdj||jdr t|zSt |S) NrrL javascript#rz//zhttp:ri)rr rVreplaceBASE_URLr)rr_r0s r abs_linkzparse_detail..abs_linkswwqz! ??6 "dool&CtWZG[771:  ??4 771:%%dGdN; ;wwqz!!$4??3;O4hhU\]`bfUghhrzhref="([^"]+)"z src="([^"]+)"z

\s*

z]*>z

c3,K|] }|dv ywrkrl)rmrn body_htmls r rpzparse_detail..s:ah)+:sc3`K|]&}d|ddtj|dd(yw)z

rqz

N)html_modescape)rmrns r rpzparse_detail..s3mcdahZr(//!F):T9UU]^ms,. rKzI'9=I}b)4I{E95I :T::wwmhlmmH!((*T1H'Y*>I!|GId28TVWWyhW[ \\rc t|d|d|dxs|d|dxs|d|d|dxsddddtjjtd }t j td }|jd |j} |jd |d |d|d|d|d|d|d|d|df |j|jy#t$rYwxYw)Nr0rMrNrriu 环境保护) site_name source_urlpage_urlrM publish_datersummarycategory script_name<)r"zPRAGMA busy_timeout=60000zINSERT OR IGNORE INTO gov_raw (site_name, source_url, page_url, title, publish_date, content, summary, category, script_name) VALUES (?,?,?,?,?,?,?,?,?)rrrrrrr) SITE_NAMEospathbasename__file__sqlite3connectrexecutecursorcommitr-close)detailitrowconncs r _storersFiuI/BwKz*R ^f%<!"U),F::i(#%g;:[]kmnxkyz$Wo<GF7O%+ZZ%?%Q2j>F> " 62 i ! 1 # $ )CvRE |6:$G 3;< Ai[ 6()G9H EFr__main__z--pages==)r)r)r)&__doc__r sysrjsonhtmlr{ urllib.parserr'rrrvrDOMAINGROUPrenvironr(rr)rrr4rerrr__name__r._MAX_PGargvrnrVrr ValueErrorisdigitrlrr rs+**  ' ( 5   **..$5 6D  *  #J?]B0!GF zG XXab\ << # aggc1oa01YY[!fG   sCC"!C"