ujdZddlZddlZddlZddlZddlmZmZmZddlZddl Z ejjdejjejjeddlmZe j"e j$j&dZdZdZdZd Zd d iZd Zd ZdZdZdZdZe dk(r8ejZ!ee"deje!z ddyy)u 陕煤集团 - 通知公告 (shccig.com) 列表: /Lists/index.html?id=97 (page 1) /index/Lists/index.html?id=97&page=N (pages 2+) 详情: /index/Article/index.html?id=N&cid=97 正文: div.article-con N)datetimetimezone timedelta)push_to_searchdbu陕煤集团-通知公告zhttps://www.shccig.comz-https://www.shccig.com/Lists/index.html?id=97z;https://www.shccig.com/index/Lists/index.html?id=97&page=%dz User-AgentzoMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36cr tj|tdd}d|_|jS#YyxYw)NF)headerstimeoutverifyzutf-8)requestsgetHEADERSencodingtext)urlrs &/root/gov_crawler/crawl_shccig_tzgg.pyfetchrs5 LLgr% H vv s/26cg}d}tj||tjD]}|jdj }|jd}tj dd|jdj }tj dd|j }|rt |d krd |d }t|z}|j|||f|S) u0从列表页提取 (date, title, href) 三元组z]*>.*?]*class="fr"[^>]*>(.*?).*?]*href="/index/Article/index.html\?id=(\d+)(?:&|&)cid=97"[^>]*>(.*?).*?<[^>]+>\s+ rz/index/Article/index.html?id=z&cid=97) refinditerDOTALLgroupstripsublenBASE_URLappend) htmlitemspatternmdate_str article_idtitlehreffull_urls r parse_listr0s E  [[$ 2 2771:##%WWQZ z2qwwqz288:vsE*002E Q .zl'Bd? hx01 2 Lc*|dkrtSt|zS)Nr) LIST_URL_P1 LIST_URL_PN)pages r get_page_urlr64s qy  r1c,d}|}t|}||kr||||dzdk(r&|jd|dz}||kDr|dz}n|dz }6|||dzdk(r|dk(r|||S|dz}|dz }V|||dzd k(r||dzd vr |dz }|dz }v|dz }||kr|||d S) u+用深度计数提取嵌套
的内容rzrrz
   N)r$find)r' start_posdepthinends rextract_nested_divrE9s EA D A a% !A#;& ))E1Q3'CQw!GQ  !A#;( "zIa(( QJE FA  !A#;& T!A#Y2N%N QJE FA  Q+ a%,  r1ct|}|syi}tjd|tj}|rI|j dj }tj dd|j }||d<|jdrt|ddkrbtjd|tj}|r;tj d d|d j }t|dkDr||d<|jd }|d kDr|jd d |}|d k\rt||} tj dd| j } tj dd| tjtjz} tj dd| tjtjz} t| dkDr| |d<|jdsdD]} tjd| zdz|tj}|s1t|j ddkDsO|j dj } tj dd| tjtjz} tj dd| tjtjz} | |d<n|jd|jddfS)u7获取详情页，返回 (title, content, publish_date))NNNz(.*?)ru[\s\-–—]+\S.*$rr-rz]*>(.*?)rzclass="article-con"rr:z ^]*>z]*>.*?)flagsz]*>.*?dcontent)rJarticlermaincondetailz<(div|section)[^>]*class="z[^"]*"[^>]*>(.*?)rN)rrsearchr r!r"r#rr$findallr?rfindrEI) rr'resultr*r-h1slast_h1 con_start div_startrJclss r fetch_detailrYVs :D  F )4;A   ",b%8>>@w ::g #fWo"6":jj/ryyA ffZSW5;;=G7|a")w /0I1}JJvq)4 >(y9Gff^R9??AGff92wbiiZ\Z^Z^N^_Gff7WBIIXZX\X\L\]G7|c!$+y! ::i J C -36OObiiAS_s*''!***,&&!=r7RTR[R[^`^b^bRbc&&!;RPRPYPY\^\`\`P`a$+y!  ::g  9 5t ;;r1c ltddtdttdg}t}tdtdzD]}t |}td|ddd t |}|s td nt|}g}|D]0\}}} | |vs |j| |j||| f2td t|d t|t|z d |s tdn|j|tdt|dg} t|D]\} \} } }td| dzdt|d| ddddd t|\}}}tjd|xs| }t!j"dd|xsdj%dd}t!j"dd|}| jt|||xsd| |tdtd| r t'| dtdt| dy)Nr<z2==================================================u🏠 ru 📄 第 u 页...rT)rDflushu❌ 请求失败u✅ u 条（跳过 u 条重复）u 无更多新条目u 📊 列表总计: u 条z [/z] 2z...g333333?rri,r) site_namer-rrJpub_datesummarytagsu✅ shccig_tzggu ✅ 完成! 共 )print SITE_NAMEsetrange MAX_PAGESr6rr0addr&r$extend enumeraterYtimesleeprr#r"r)all_list seen_urlsr5rr'r( new_itemsdater-item_url all_itemsrBr+ list_title detail_urldtrJ_r`s rrLrLsE Bvh- E)  VHHIaQ'#4  D6)s$?Sz $ % 4  %* : !D%y( h'  $x!89 : S^$N3u:I3N2O}]^ ( )  "%#( !#h- 56I1:81D- -Hj* AaC5#h-:cr?*;3?SPTU%j1GQ 3 j&&S'-R8>>@#F&&g."}"    e %*M2 s9~.d 34r1__main__u ⏱ 耗时: z.1fs)#__doc__sysosrrkrrrr urllib3pathinsertdirnameabspath__file__ crawler_librdisable_warnings exceptionsInsecureRequestWarningrdr%r3r4rgrrr0r6rErYrL__name__t0rcr1rrs 22277??277??8#<=>(++BBC ' #= K K L, :5