qj/*(dZddlZddlZddlZddlZddlZddlZddlZddlZ ddl m Z m Z ddl m Z ddlmZdZdZdZd Zd Ze j*e d z j-d ZdZdZdZdZdZdZdZdZdZ dZ!dZ"e#dk(re"yy)ucrawl_hsq_tztg.py - 黄山市黄山区人民政府-公示公告 站点: https://www.hsq.gov.cn (帝联 NWAF: 412 + Environment Checking + ctct_bundle) 过盾: playwright + playwright-stealth (同 ahshx 方案) 列表: 静态HTML /dwzw/tztg/index.html → /dwzw/tztg/{id}.html 分页: /content/column/6794244?pageIndex=N (Ls.pagination) 详情: div.wzcon.j-fontContent 正文 + meta ArticleTitle/PubDate + 附件下载 N)datetime timedelta)sync_playwright)Stealthu黄山市黄山区-公示公告zhttps://www.hsq.gov.cnz+https://www.hsq.gov.cn/dwzw/tztg/index.html6794244im)daysz%Y-%m-%dzoMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36z/tmp/hsq_tztg.jsonlz/root/search.dbz>/root/.cache/ms-playwright/chromium-1228/chrome-linux64/chromez/tmp/hsq_tztg.lockzcrawl_hsq_tztg.pycv|s|S|Dcgc]}t|ts|}}t|t|k7r|Stg}}|D])}||vs|j ||j |+|Dcgc]&tdk\rt fd|Ds(}}|r|S|Scc}wcc}w)uB剔除「容器段」：find_all(['p','div']) 会同时收下容器
与它内部的

， 导致同一内容重复（容器那份常还被 get_text(strip=True) 拍平）。 判据：某段被列表内另一段完全包含 → 它是容器段 → 剔除。 保护：剔除后为空则原样返回。 (c3:K|]}|uxr|xr|vyw)N).0bas #/root/gov_crawler/crawl_hsq_tztg.py z(_drop_container_parts..0s&(S!)D)Da1f)D(Ss) isinstancestrlensetaddappendany)partsppsseenuniqrkeeps ` r_drop_container_partsr s   1jC0! 1B 1 2w#e* $D ( D= HHQKQ( V!FbLS(Sd(S%S  VD V4!T! 2 VsB1B1>+B6ctjddD]O}|jdrt|j ddcS|j sDt|cSt S)Nz--pages==)sysargv startswithintsplitisdigitMAX_PAGES_DEFAULT)rs rparse_pages_argr+4sX XXab\ << #qwws|A' ' 99;q6M  ctjdd|xsdj}tjdd|j}|jddjddS)Nu ^\[图\]\s*u ^[•··\s]+u​u)resubstripreplace)ts r clean_titler4=s[ ~r17+113A Q'--/A 99Xr " * *8R 88r,cB|j}d}tjd|}|r|jdj }|sFtjd|tj }|r|jdj }d}tjd|}|r|jd}|s)tjd|}|r|jd}d} ddlm}||d }|jd tjd  }|s5|jd tjd tj}|r|jtjdtj D]} | jg} |jdtjdtjD]j} | jdd} | jdxs| jdd} | j!dr t"| z} | j%| | dlg}|jgddD]} | j&dvr| j)dr$| j&dk(r| j)drE| j&dk(r5t+| }tj,dd|}|j%|| j&d k(r=| jgdd r| jdd}|r|j%|| jdd}|stj.d!|r|j%|g}|D]!}|s|r |d|k7s|j%|#|}t1|}d"j3|}| D]} d#| d$d%| d&d'}||vs|d(|zz } t5|j d)krd#|j6d%|d'}|||fS#t8$rYwxYw)*u;从详情页提取 (title, date, content_html, attachments)r.z+(.*?)(?:_黄山市黄山区人民政府)?z1zparse_content..ts[_acaiaijkalJmr,)rr?r@FuW^(责任编辑|初审|复审|终审|\[纠错\]|浏览次数|字号|分享到|扫一扫)z z

r=z

 )contentr/searchrEr1Sbs4r6findcompileIfind_all decomposegetget_textr(r&rDrr= find_parentrr0matchr joinrr> ImportError)pagehtmltitlem date_textrJr6soupwzel attachmentsrr:r=rtbltxtdeduprlinks r parse_contentrfCs <<>D E @$GA   "  IIQSWY[Y]Y] ^ GGAJ$$&EI FMAGGAJ  IIA4 H  IG7 %T=1 YYuRZZ0F%GY H5RZZ0G%NOB kk4_acaeae)fkg   K[[2::6`bdbfbf+g[h @uuVR(zzz-DC1D??3'$;D""D#>?  @Ekk">$kO &77l*r~~g/F77g%"..*A77g%b'C&&!=?morsCLL%77e#ww4wF ++b+5C S)kk"Dk1rxx)CEH ILL%' &(E $euRyA~LLO $E)%0Ekk%(G  +%ahZ/A!F)HUw&td{*G +7==?#b(( 2DUG8T )W $$    s2+IP P!P? P A P9P PPcL t}dtjv}dtjv}td|dtdt 5}|j jtdgd}|jtd d d  }tj||j} g}td |d zD]_}|d k(rt} nt dt"d|} td|d| d|j%| ddt'j(|d k(rdnd|j+dd} t-} g} | D]l} t/j0d| d}|r| d| vr&| j3| d| j5t7| d| d|j9d dn| std|ddnZtd|dt;| dd|j=| t'j(t?j@d d!btd"t;|ddg}tC|D]\}}t'j(t?j@d#d$ |j%|d%d&dt'j(d tE|\}}}|sd'}|r |tkr{|j5|d%|xs|d(||d)td*|d zd+t;|d,|d-d(dd.d/|dtd2t;|dd|rA|D]"} ttKjL| d34$ |jOdddy|rtQtRd5d675}|D]+} |jUtKjL| d34d8z- dddtd9t;|d:tRd |jOdddytWjXtZd1;}|j]d<|j_}d=}|D]} |j]d>| d%f|jar*|j]d?| d(tbd@| d%| dA| dBd'd't/jddC| dArtg| dAjidDd'nd=tjf |jls|d z }|jo|jOtdE|d|jO dddy#tF$r9}td*|d zd+t;|d0tI|dd1dYd}~d}~wwxYw#1swYxYw#|jOwxYw#1swYyxYw)FNz--dryrunz--jsonlu[hsq] 公示公告 pages=z cutoff=T)flush)z --no-sandboxz--disable-setuid-sandboxz---disable-blink-features=AutomationControlled)executable_pathheadlessargsiVi)widthheight) user_agentviewportr"z/content/column/z ?pageIndex=u [列表] 第u页 i`domcontentloaded)timeout wait_until rza[href*='/dwzw/tztg/']zUels => els.map(e => ({t: e.innerText.trim(), h: e.href})).filter(x => x.t.length > 5)z/dwzw/tztg/(\d+)\.htmlhr3)r[r>r9u 第u页 0 条, 停止u 条u [列表] 共 r>iȯr.r[)r>r[daterJz [r;z] r<r z | u ] 失败: <u [完成] F) ensure_asciirzutf-8)encodingrHz[jsonl] u 条 -> )rqzPRAGMA busy_timeout=300000rz'SELECT id FROM gov_raw WHERE page_url=?zINSERT OR IGNORE INTO gov_raw (title, site_name, group_name, page_url, publish_date, content, summary, attachments, date_rank, script_name) VALUES (?,?,?,?,?,?,?,?,?,?)u安徽ryrJz\d{4}-\d{2}-\d{2}-u[入库] 新增 )8r+r$r%printCUTOFFrchromiumlaunchCHROMIUM new_contextUArapply_stealth_asyncnew_pagerangeLIST_URLrD COLUMN_IDgototimesleepeval_on_selector_allrr/rKrrr4rErextendrandomuniform enumeraterf ExceptionrjsondumpscloseopenOUTwritesqlite3connectDB_PATHexecutecursorfetchone SITE_NAMErVr'r2 SCRIPT_NAMErowcountcommit) max_pagesdryrun jsonl_moderbrowserctxrYitemspnr>rowsr page_itemsrr\resultsidxitr[r]rJefconnc new_counts rmainrs!I 388 #Fchh&J %i[ AN  \a**##$tn$p!!RDTW:X!Y %%c*||~U EAy1}- 17"C!F"29+[MCrd$se4DA #u9K L q2a000,kmu gA ";QsVDA#$ HHQsV$%% AcF0CAcFZ[ZaZabcZd&ef g "GB4'89Ft4J'8=TJ Z( 6>>!Q/05 18 M#e*T2$ ?G$U+ YR 6>>!Q/0YIIbiCUIVJJqM0=d0C-E9g$$&  Y%7 NN!%y!&!5"W+ )#* $ CAwaE |2gbk'6J3B6O5PPST]S^_gkl! Y( Ic'l^40 = =A$**QU;<=6 MMOy\\D#sW5J$J 15 AD HIJJWhsersBAAA(/& -  8   (,,.9#. . 8 8 Dv  K ! ".9 N%bbJ zFr,