}j:7dZddlZddlZddlZddlZddlZddlZddlZddlZddl m Z ddl m Z ejjejjeZejj%edZdZdZdZd Zd Zd Zejj%ed Zdddd ZdZdZdZddZdZ dZ!dZ"dZ#ddZ$dZ%dZ&dZ'ddZ(e)dk(re'yy)u crawl_cz_sthjj.py — 郴州市生态环境局 建设项目受理情况公开 爬虫 ============================================================ 站点: https://sthjj.czs.gov.cn/zwgk/xxgkml/hjpj/jsxmhbspqgs/ 特点: 天融信WAF防护，需 Playwright 绕过 JS 挑战 列表: UL > LI > a[href*='content_']，每页16条 详情: 文字信息（项目名称/建设地点/建设单位等）+ PDF附件 分页: default.htm, default_1.htm, ... (offset=1530 ≈ 97页) 用法: python3 crawl_cz_sthjj.py # 增量爬 python3 crawl_cz_sthjj.py --full # 全量爬 python3 crawl_cz_sthjj.py --test N # 测试N条 python3 crawl_cz_sthjj.py --sync # 仅同步 N)urljoin)sync_playwrightzcz_sthjj_results.db郴州市生态环境局zsthjj.czs.gov.cnzroot@1.94.217.116/root/search.dbzAhttps://sthjj.czs.gov.cn/zwgk/xxgkml/hjpj/jsxmhbspqgs/default.htmz6https://sthjj.czs.gov.cn/zwgk/xxgkml/hjpj/jsxmhbspqgs/z.cz_sthjj_chkpt)newskiperrorsczttd5}|j|d|dddy#1swYyxYw)u)保存断点：第page_num页的第idx条w|N)openCHECKPOINT_FILEwrite)page_numidxfs #/root/gov_crawler/crawl_cz_sthjj.pysave_checkpointr#s: os #%q 8*AcU#$%%%s1:c6tjjtrc t t5}|j j jd}t|dt|dfcdddSy#1swYyxYw#YyxYw)u$读取断点，返回(page_num, idx)r rN)rr) ospathexistsrr readstripsplitint)rpartss rload_checkpointr(s ww~~o& o& 4!(..s358}c%(m3 4 4   4    s)BA B= BB BBBc~tjjtrtjtyyN)rrrrremoverclear_checkpointr%3s# ww~~o& /"'r$c |r |j|r|r |j||fS#t$rtdYnwxYwt j }|j jd}|jdddd}||fS) uB检查 Playwright 连接状态，断开则重建 browser + contextu< 🔄 Playwright 连接断开，重建 browser + context...TheadlessoMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.368widthheight user_agentviewport) contextspages Exceptionprintr __enter__chromiumlaunch new_context) browser_obj context_objpws rensure_browserr=7sN   ;   K'' N LMN   $ $ &B++$$d$3K))E40*K  ##s !$;;ctjt}|jd|jd|j |j y)NPRAGMA journal_mode=WALuKCREATE TABLE IF NOT EXISTS crawl_results ( id INTEGER PRIMARY KEY AUTOINCREMENT, site_name TEXT DEFAULT '郴州市生态环境局', title TEXT, url TEXT UNIQUE, content TEXT, publish_date TEXT, summary TEXT, crawled_at TEXT DEFAULT (datetime('now','localtime')) ))sqlite3connectDB_PATHexecutecommitclose)conns rinit_dbrGIsB ??7 #DLL*+LL    KKMJJLr$cdtjt} |jd|||||f|j |j dkDrt dxxdz cc<nt dxxdz cc<|jy#t dxxdz cc<Y&xYw#|jwxYw)NzcINSERT OR IGNORE INTO crawl_results (title, url, content, publish_date, summary) VALUES (?,?,?,?,?)rrrrr )r@rArBrCrD total_changesstatsrE)titleurlcontentdatesummaryrFs r store_recordrPYs ??7 #D  !C$ 0     ! %LA L &MQ M  h1 sABBBB/cJ|ydt|jddzdzS)Nz''')strreplace)vs rescrVls(y QT* *S 00r$c tdtjt}|j dj }|j |s tdytjd}|j dd}d}|D]J}|\}}}} } |j d |||xsd dd | xsd | xsd dd |d f|jdkDr|d z }L|dkDr3|j|j d|f|j|j d|fjd} |j td|dt|d| dy#t$r} td| Yd} ~ d} ~ wwxYw)u@将本地数据直接写入 search.db（服务器本地模式）u 📤 同步到 search.db...zPSELECT title, url, content, publish_date, summary FROM crawl_results ORDER BY idu 本地没有数据Nrr?rrzwINSERT OR IGNORE INTO gov_raw (title, page_url, content, publish_date, summary, site_name, tags) VALUES (?,?,?,?,?,?,?)i i,rz Error: zINSERT INTO gov_search(rowid,title,site_name,summary) SELECT r.id,r.title,r.site_name,r.summary FROM gov_raw r WHERE r.id NOT IN (SELECT rowid FROM gov_search) AND r.site_name=?z.SELECT COUNT(*) FROM gov_raw WHERE site_name=?z OK /u 条同步到 search.db (DB共条)) r5r@rArBrCfetchallrErIr4rDfetchonelen) rFrowsdst site_name new_countrrKrLrMpub_daterOetotals rsync_to_serverrfps )* ??7 #D <<j k t t vDJJL  $% //+ ,CKK)**II  #12.sGXw # KK)gmWf5x~2-R#& 27    1$Q  #1}   QL   KK89, HHP ST VEIIK E)Ac$i[(FugT RS# # IaS/ " " #s?E&& F/FFcTt|dzD]z} t||\}}|jr|jdn|j}|j |ddt j d|j||fcSd ||fS#t$r}t|}d|vsd|vsd |vr2td |d d d td d \}}|j}n9||kr(td|d d dt j dn d ||fcYd }~cSYd }~d }~wwxYw)u>用 Playwright 页面实例获取 HTML，断连时自动重建rr networkidlei0u) wait_untiltimeoutEPIPEz Target closedzBrowser closedu ⚡ 断连(N<u), 重建连接...u ⚠ 重试(z)...) ranger=r3new_pagegototimesleeprMr4rSr5)rLpagebrowsercontextretryattemptrderr_strs rpw_getrzs/#. .-gw? GW'.}}7==#':J:J:LD IIcmUI C JJqM<<>7G3 3.& ' !! .!fG'!_%?CSW^C^ gcrl^3EFG#1$#= '')5 gcrl^489 1 Wg-- .sA5B D'A:D"D'"D'c|dk(rt}n d|dz d}t||||\}}}|sd||fSg}tjd|tjD]}|j dj }|j dj } |j dj } |jtt|| | d|||fS) u1获取列表页，返回 [{'title','url','date'}]rz>https://sthjj.czs.gov.cn/zwgk/xxgkml/hjpj/jsxmhbspqgs/default_z.htmNz]
  • \s*]*title="([^"]+)"[^>]*>.*?\s*]*>([\d-]+)rkrn)rLrKrN) BASE_LISTrzrefinditerDOTALLgrouprappendr DETAIL_BASE) rpw_pagerurvrLresultitemsmhrefrKrNs rfetch_list_pagers1}NxZ[|n\`a%c7GWEFGW Wg%% E [[h   wwqz!   "wwqz! ;-     '7 ""r$cft||||\}}}|syd}tjd|tj}|r|j dj }|sFtjd|tj }|r|j dj }d}tjd|tj} | r| j dj } tjdd| tjtj z} tjd d| tjtj z} tjd d| } | j }|s6tjdd|tjtj z} tjd d| tjtj z} tjd d | } tjd d | j } | jd } g} d}t| D]K\}tfddDrd}|sdvsdvsdvrn!| jj M| rddj| ddzdz}d}tjd|}|r|j d}g}tjd|tj D]}|j dj }|j dj }|jdrtd|dd}n>|jdrtd|dd}n|jd s t||}|jd!|d"|d#|r|d$d%j|zz }tjd d|}|r(tjd&d'|j dd(n|}|xsd||||d)S)*u抓取详情页NrXz(.*?)rz3]*name="ArticleTitle"[^>]*content="([^"]+)"z$class="xl-xqnr">(.*?)\s*z]*>.*?)flagsz]*>.*?z\s+style="[^"]*"z<[^>]+> z \s*\n\s*\n\s*Fc3&K|]}|v ywr!r#).0kwlines r zfetch_detail..si"2:is)u 项目名称u 建设地点u 建设单位u 受理日期Tu 扫一扫u 打印本页u 设为首页z

    z

    z

    u(发布时间[：:]\s*(\d{4}-\d{2}-\d{2})z+]*href="([^"]+\.pdf)"[^>]*>([^<]+)rkz../../zhttps://sthjj.czs.gov.cn/z../httpz zu&

    附件：
    z
    z\s+ i)rKrMrNrOrL)rzr}searchrrrIsubr enumerateanyrjoinr~ startswithr) detail_urlrrurvhtmlrKtmatmrMcm content_rawtextlines info_linescaptureirNdmattachesamrLnamerOrs @r fetch_detailrs#J'JD'7  E *D"))  EB xx{Hkk6 bdd Ghhqk!xx{  " >>( #5s12w?C ^^E "5s12w?C'*c*C)C5(:4&EF G;fkk(>SSSffZW-G?N3"D8N33%Oc 4tjt}|jd|dfj }|j |rt dxxdz cc<ytd|dddd d d t|d|||}|td t dxxdz cc<yt|dxs|d|d|d|jdxs|jdd |dt|jdd }td|dy )Nz'SELECT 1 FROM crawl_results WHERE url=?rLrrFu 📄 rK(z...rXT)endflushu ✗ 详情失败r rMrNrOu ✅ B) r@rArBrCr\rErJr5rrPgetr])rrtrurvrFrdetailclens rrrts ??7 #D \\Cd5k^ T ] ] _FJJL  f   GDM#2&'s +4@ $u+tWg >F ~ !" h1w(4= U y 62dhhvr2y  vzz)R( )D E$q/ r$__main__)NN)rk)*__doc__rr}rrrjsonr@ subprocessssl urllib.parserplaywright.sync_apirrdirnameabspath__file__BASE_DIRrrB SITE_NAMEDOMAIN SERVER_SSHSERVER_SEARCH_DBr|rrrJrrr%r=rGrPrVrfrzrrrr__name__r#r$rrs 9888 / 77??277??84 5 '',,x!6 7 &  $ O F '',,x):;1*% #$$ &1.T`".#8QfL\8 zFr$