ujTdZddlZddlmZddlZddlZddlZddlZddlmZdZ dZ dZ dZ ejd d Zd dd Zd ZdZdZdZdZedk(reyy)u(爬取 hfsikang.com - 环评公示栏目N) BeautifulSoup)datetimezhttp://www.hfsikang.comz.http://www.hfsikang.com/category/eia-publicityuhfsikang.com-环评公示z 2023-06-16 SEARCH_DBz/root/search.dbzoMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36)z User-AgentRefererc:|jjddjddj} tj|dj dS# tj|dj dcYS#|cYcYSxYwxYw)u,解析发布时间 2026/06/10 → 2026-06-10u发布时间：u 发布时间:z%Y/%m/%dz%Y-%m-%d)stripreplacerstrptimestrftime)texts #/root/gov_crawler/crawl_hfsikang.pyparse_publish_daters ::<   12 6 > >PR S Y Y [D  z2;;JGG $$T:6?? K K Ks*$A%%B($B BBBBcLtjttd}d|_t |j d}|jd}|sy|jd}d}|D]:}|jd }|js&t|t|}<|S) u获取总页数headerstimeoututf-8 html.parserz div.pageraTr ) requestsgetLIST_URLHEADERSencodingrr select_onefind_allget_textisdigitmaxint)respsouppagerlinks last_pagertxts rget_page_countr+!s <<'2 >DDM M 2D OOK (E  NN3 EI 1jjtj$ ;;=Is3x0I1 c |dk(rt}n td|}tj|td}d|_t |j d}g}|jdD]}|jd}|s|jd }|jd d }|r|jd s t|z}|jd} d } | rt| jd } |j||| |d|S)u&获取某一页的标题/链接/时间rz/page/rrrrzul.default > liz h2 a.titelsTrhrefrhttpz p.infot em)titleurl publish_datepage)rrrrrrr selectrr! startswithBASE_URLrappend) r3r1r%r&itemsli title_tagr0r.time_empub_dates r get_list_pager=1s qy &' <<Wb 9DDM M 2D Ekk+,\MM-0  """.}}VR( /d?D-- - )'*:*:*:*FGH uT8UYZ[\ Lr,ctj|td}d|_t |j d}|j d}|r1|jdD]}|jt|Sy)u获取详情页的正文HTMLrrrrzdiv.enter.suojinz;div.bqc, div.bdsharebuttonbox, div.next_post, script, styler) rrrrrr rr4 decomposestr)r1r%r&entertags r get_detailrCLsl <<Wb 9DDM M 2D OO. /E << ]^ C MMO 5z r,c tjt}|j}|j d|j t }td|d}d}d}td|dzD]}|rntd|dt|}|D]}|d} | r | tkrd }td td n|j d |d f|jr|dz }Vt|d } | std|ddd|dz }|j d|d|d | | tf|jdkDr|dz }|j |sn|jtd|d|dy)Nad CREATE TABLE IF NOT EXISTS gov_raw ( id INTEGER PRIMARY KEY AUTOINCREMENT, title TEXT, page_url TEXT UNIQUE, content TEXT, publish_date TEXT, site_name TEXT DEFAULT '', crawl_time TEXT DEFAULT (datetime('now', '+8 hours')), similar TEXT ) u 总页数: rFru 第u页...r2Tu 截止日期 u，跳过后续z)SELECT id FROM gov_raw WHERE page_url = ?r1u 警告: 空正文跳过 r0rzhINSERT OR IGNORE INTO gov_raw (title, page_url, content, publish_date, site_name) VALUES (?, ?, ?, ?, ?)u完成: 新增 u 条, 跳过 u 条)sqlite3connectDB_PATHcursorexecutecommitr+printranger= CUTOFF_DATEfetchonerC SITE_NAMErowcountclose) connc total_pages total_added total_skippedcutoff_reachedr3r8itempubcontents rcrawlr[Ys ??7 #D AII    KKM "K K } %&KMNaq)"   dV6"#d# D~&Cs[(!%)+oFG IIADK> Rzz|"  e-G4T']3B5G4HIJ"  IIzgU Wc9E zzA~q KKM3 8  E"H JJL OK= m_D IJr,__main__)__doc__rbs4rrEosresysrr6rrOrMgetenvrGrrr+r=rCr[__name__r,rres.  $ ; '  "))K!2 3D(   6 ?KB z Gr,