jdZddlZddlZddlZddlmZdZdZedzZddiZ d Z d Z d Z d Z eZd ZdZdZdZedk(reyy)uU仙桃市生态环境局 — 公示公告爬虫 (修复版 v2) 修复: 1. 日期从 span.date 提取(原取第一个span, 实为list_text标题→date错位+title被清空) 2. title 优先 a[title] 属性, 其次 list_text span 3. 写 gov_search FTS (原写废弃表 gov_search_v3 → 搜索不到) 4. 补 source_url/script_name/group_name N) BeautifulSoupz/root/search.dbzhttps://www.xiantao.gov.cnz>/bmxxgk/shbj/zfxxgk/fdzdgknr/qtzdgknr_37950/gsgg/index{}.shtmlz User-Agentz[Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/120.0.0.0 Safari/537.36 xiantao_gsggu湖北zcrawl_xiantao.pycv|s|S|Dcgc]}t|ts|}}t|t|k7r|Stg}}|D])}||vs|j ||j |+|Dcgc]&tdk\rt fd|Ds(}}|r|S|Scc}wcc}w)u6剔除「容器段」：find_all(['p','div']) 会同时收下容器
与它内部的

， 导致同一内容重复（容器那份常还被 get_text(strip=True) 拍平）。 判据：先按值去重，再剔除被其它段完全包含的段。保护：剔除后为空则返回去重结果。 (c3:K|]}|uxr|xr|vyw)N).0bas "/root/gov_crawler/crawl_xiantao.py z(_drop_container_parts..'s&(S!)D)Da1f)D(Ss) isinstancestrlensetaddappendany)partsppsseenuniqr keeps ` r _drop_container_partsrs   1jC0! 1B 1 2w#e* $D ( D= HHQKQ( V!FbLS(Sd(S%S  VD V4!T! 2 VsB1B1>+B6c tj|td}d|_t |j d}d}|jd}|r|jd }d}|jd D]b}d |jvs|jd }|s*tjd |j}|sQ|jd }n|jddxs|jdd} d} | rag} | jgdD]/} t| j!} | s| j#| 1dj%t'| } ||| fS#t$rYywxYw)Nheaderstimeoututf-8)NNN html.parserz div.xl_titTstriptdu 发文日期z(\d{4}-\d{2}-\d{2})divview)class_ TRS_Editor)rr)table )requestsgetHEADERSencoding Exceptionrtext select_oneget_textfind_allfind_next_siblingresearchgroupfindrr&rjoinr)urlrsouptitlettdater'nxtm content_divcontent_partstagtxts r extract_detailrK+sa LLgr :   /D E  &B  $ ' DmmD!/ R[[] *&&t,CII4cllnEQWWQZd /))E&)1ZTYYu\Y5ZKG''(=> 'Cc(.."CFMM#& '))1&9: $ 3   s#E,, E87E8c(|dk(rtjd}ntjd|} tj|td}d|_t|jd}g}|jd d D]x}|d }d |vsd|vr|jdjds |jdjdrS|jds t|z}|tvrvtj||jdxsdj!}|s%|j#d}|r|j%d }|s|j%d }d} |j#d} | r| j%d } | rD| j%d |vr0|j'| j%d dj!}|j)|j!|| d{|S#t $rgdfcYSwxYw)Nr(r$_rrr"rr#r T)hrefrNz/shbj/z/gsgg//z/gsgghttprAzspan.list_textr%z span.date)rAr>rC)LIST_TPLformatr/r0r1r2r3rr4r7rstripendswith startswithBASE seen_urlsrr&r5r6replacer) pgr>r?r@itemsr rNrAltrCsps r crawl_pager]Js Qwoob!oo"h' LLgr :   /D E ]]3T] *Jy 4 84#7  ;;s  $ $W -S1A1J1J81T v&$;D 9   dw%2,,../B $ /JJTJ*E \\+ & ;;T;*D "++D+)U2MM"++D+"92>DDFE u{{}T4HI9J: LC 1u s#H HHcFddl}|j}|jdtd|j }t dt |jt}tjtd}|jd|j}|jd|jd dx}x}}td|dzD]} td | d t!| } | std ndtd t#| d| D]C} |jd| df|j%r|dz }0td| dd| dddt'| d\} } }| s| d} | s| d} |rt#|dkr"td|r t#|ndd|dz }t)j*dd|dd}t)j*dd|j-} |jd| ||| d| d| t.t0t2f }|j4}|jd|| t.|f|j7|dz }Ftd|d |d!|d"||z|zy#tj8$r |dz }YwxYw)#Nrz--pagesr()typedefault<)r!zPRAGMA journal_mode=WALaCREATE TABLE IF NOT EXISTS gov_raw (id INTEGER PRIMARY KEY, site_name TEXT, source_url TEXT, page_url TEXT, title TEXT, publish_date TEXT, date_rank INTEGER DEFAULT 0, summary TEXT, status TEXT, category TEXT DEFAULT '', visits INTEGER DEFAULT 0, content TEXT DEFAULT '', tags TEXT DEFAULT '', industry TEXT DEFAULT 'other', attachments TEXT DEFAULT '', group_name TEXT, has_table INTEGER DEFAULT 0, script_name TEXT DEFAULT '')zgCREATE VIRTUAL TABLE IF NOT EXISTS gov_search USING fts5(title, site_name, summary, tokenize='trigram')u--- 第u页 ---u无结果，停止翻页u找到u条z'SELECT id FROM gov_raw WHERE page_url=?r>z [rCz] rA2u 正文过短(u ), 跳过z<[^>]+>r$z\s+ zINSERT INTO gov_raw (title, summary, content, page_url, source_url, publish_date, site_name, script_name, group_name) VALUES (?,?,?,?,?,?,?,?,?)zJINSERT INTO gov_search (rowid, title, site_name, summary) VALUES (?,?,?,?)u 新增: u 重复: u 过短: u 总计: )argparseArgumentParser add_argumentint parse_argsmaxminpages MAX_PAGESsqlite3connectDBexecutecursorrangeprintr]rfetchonerKr9subr&SITESCRIPTGROUP lastrowidcommitIntegrityError)reapargsrlconnc total_new total_dup total_skiprYrZitemrArCrGsummarycurrids r mainrus  "BOOICO3 ==?D 3tzz9- .E ??2r *DLL*+ AII}~IIwx)**I* JAuqy! t7#$2 , -u s5zl#&' D II?$u+ Pzz|Q  CV ~RW cr(:';< =#1$u+#> E4$w-%V c'lS0)'#g,q)ISTa ffZW5ds;GffVS'288:G iigGWd5k4;dTZ\abdmm f%w/1 Q +  < JykI;j JW`ajWjkuWuVv wx)) Q  s8A(JJ J __main__)__doc__r/rnr9bs4rrprVrQr1rwryrxrmrrWrrKr]r__name__r r rs{# R R v w   E "& >)V,y\ zFr