Ʌjm<dZddlZddlZddlZddlZddlZddlZeejdkDr2ejdjre ejdndZ e e de e zddlZddlmZddlmZmZej&ddZd Zd Zej.ed z j1d ZdZdddZej8Zej<j?eddl Z dZ!dZ"dZ#dZ$dZ%e&dk(re%yy)u遂宁市安居区人民政府 — 通知公告爬虫 https://www.scanju.gov.cn/gongkai/jichu/40016.html Custom CMS, ?page=N分页 Nz[AutoPg] max_pages=) BeautifulSoup)datetime timedelta SEARCH_DBz/root/search.dbz2https://www.scanju.gov.cn/gongkai/jichu/40016.htmlu+遂宁市安居区人民政府-通知公告iG)daysz%Y-%m-%dnzoMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36zhttps://www.scanju.gov.cn/)z User-AgentReferercR tjtd}|j}|j dt f|j Dchc]}|d }}|j|Scc}w#t$r"}td|tcYd}~Sd}~wwxYw)N<timeoutz.SELECT page_url FROM gov_raw WHERE site_name=?ru[WARN] 无法查询已有URL: ) sqlite3connectDB_PATHcursorexecute SITE_NAMEfetchallclose Exceptionprintset)conncrowexistinges !/root/gov_crawler/crawl_scanju.pyget_existing_urlsr !s w3 KKM BYLQ&'jjl3sCF33 4  .qc23u s0AA; A6"A;6A;; B&B!B&!B&c tjtd}|j}|j d|j ||j |j t f|j}|j|j|S#t$r}td|d|Yd}~yd}~wwxYw)Nr r z INSERT OR IGNORE INTO gov_raw (title, publish_date, content, page_url, site_name) VALUES (?, ?, ?, ?, ?) u[ERROR] 写入失败 : r) rrrrrstriprrowcountcommitrrr)titledate_strcontentpage_urlrraffectedrs r store_itemr+-sw3 KKM  kkmXw}}8H) T  V ::    %hZr!56sBB C)B??Cct|d}g}|jd}|s|jd}|D]}|jd}|s|j dd}d|vr.|j d }|jd } | r| j d nd} |j d r|} nd |} |j|| | f|S) u解析列表页 html.parserzul.content-list liliahrefz/gongkai/show/Tr#spanhttpzhttps://www.scanju.gov.cn)rselectfind_allfindgetget_text startswithappend) htmlpage_numsoupitemsli_tagsr.a_tagr0r& date_spanr'r)s rparse_list_pagerC>s } -D Ekk./G --%2  yy$ 4 ' T*GGFO 5>9%%D%1B ??6 "H24&9H eXx01'2* Lc tj|d}d|_|jdk7ryt |j d}d}|j dd d i }|r|jd d}|s%|j d }|r|jd}d}|j dd di }|r#|jd djdd}d}|j dd di } | r| jd d}d} |jd} | r t| } |xsd|| fS#t$r} td|d| Yd} ~ yd} ~ wwxYw)u获取详情页内容r utf-8)NNNr-r1metaname ArticleTitle)attrsr(h1Tr2PubDateN ContentSourcezdiv.xqing-web-boxuRROR] 获取详情失败 r") sessionr8encoding status_codertextr7r9r# select_onestrrr) urlrespr>r& meta_titlerMr' meta_datesourcemeta_srcr( content_divrs r fetch_detailr^]sd' {{3{+   s "#TYY 6YYvfn-EYF NN9b1E4B $ /IIfVY,?I@  }}Y399;CR@H99VFO+D9E \\)R0Foo&9: +&G{Hg--   +C51#67 s-D(C7D(( E 1EE c tdtdtt}tdt|d}d}d}t dt xst D]s}|rnn|dk(rt}ntd|dz}td|dzd | tj|d }d |_ |jd k7r!td|dzd|jdn t|j|}|std|dzdntd|dzdt|d|D]\} } } | tkrtd| d| dd|dz }d},| |vr|dz }6t| \} } }| xs| }| xs| }|std|ddd}t!|||| }|dkDsz|dz }|j#| |sitdntdtd|td|||t$d }td!t'j(|d"y#t$r}td|dzd|Yd}~xd}~wwxYw)#Nu7[INFO] 遂宁市安居区 — 通知公告 爬虫启动u[INFO] 时间阈值: u[INFO] 已有URL数: rFz?page=ru [PAGE] 第u页: rFr rGrHu [INFO] 第u 页返回 u, 停止u [ERROR] 第u页请求失败: u页无数据, 停止u 页获取 u 条u[SKIP] 日期超限 r"Tu[WARN] 正文为空: r1u([INFO] 遇到超限日期，停止翻页u [SUMMARY] 完成!u[SUMMARY] 新增: u[SUMMARY] 跳过: )newskipped site_namez [RESULT]) ensure_ascii)r THRESHOLDr lenrange_MAX_PG MAX_PAGESBASE_URLrQr8rRrSrrCrTr^r+addrjsondumps) existing_urls new_count skip_count stop_crawlr=r)rXrr?r&r' detail_urldt_titledt_dater( final_title final_dater*results rcrawlrxs CE !) -.%'M !#m"4!5 67IJJ!W1 29   q=H"6(Q,8H 8A:,eH:67 ;;x;4D#DM3& 8A:,j9I9I8J(ST'  84 Jxzl*>? @  8A:,jU DAB+0 . 'E8Z)#,XJbs EFa ! ]*a *6j)A &Hgw"+eK ,HJ-k#2.>-?@A!+z7JOH!|Q !!*-3 .6  < > s9v ! yk *+ zl +,:I NF Jtzz&u=> ?@_  K |+rs"499~1diil6J6J6L#diil RV  #g, ./( "))K!2 3 ? 9 X\\^iU3 3 = =j I  D+  (   w ">) VKAZ z GrD