uj*dZddlZddlZddlZddlZddlZddlZddlZddl m Z m Z ejjejjeZejj#dedZdZdZdZdZd Zej0Zd d d ZGd de Zedk(reeedddZej=yy)u crawl_sthj_xz.py — 徐州市生态环境局 政府信息公开爬虫 ==================================================== 基于 base_crawler (GovCrawler) + API 模式。 站点: https://sthj.xz.gov.cn/dynamic/zwgk/govInfoPub.html?categorynum=003015 分类: 003015 = 法定主动公开内容（含公示公告 = 环评/环保相关） 数据: - 列表: POST /EWB-FRONT/rest/lightfrontaction/getgovinfolist (分页, 共400条) - 详情: GET /govxxgk/...html (静态页, 含索引号/发布机构/成文日期/信息名称/附件) 用法: python3 crawl_sthj_xz.py # 增量爬 (第1页) python3 crawl_sthj_xz.py --full # 全量爬所有页 python3 crawl_sthj_xz.py --sync # 仅同步 N) GovCrawlerextract_detailu徐州市生态环境局zsthj.xz.gov.cnzEhttps://sthj.xz.gov.cn/EWB-FRONT/rest/lightfrontaction/getgovinfolistz$fb2ed5f8-902e-48bd-998f-a622a60c0951001003015z[Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/125.0.0.0 Safari/537.36zapplication/json;charset=UTF-8)z User-Agentz Content-Typec eZdZddZdZdZy) XzGovCrawlercZtjtt||tdj d}t jjt|t} t jj|dt}tj|jjd}|j!dij!dg}|j!dij!dd }||fS#t"$r} t%d |d | gd fcYd } ~ Sd } ~ wwxYw) u调用列表API获取单页)deptcode categorynum pageIndexpageSizesiteGuidutf-8)dataheaderstimeoutcontextcustomrtotalru ⚠️ API请求失败(page=z): N)jsondumps DEPT_CODE CATEGORY_NUM SITE_GUIDencodeurllibrequestRequestLIST_APIHEADERSurlopenCTXloadsreaddecodeget Exceptionprint) self page_index page_sizebodyreqrespresultrres "/root/gov_crawler/crawl_sthj_xz.pyfetch_list_pagezXzGovCrawler.fetch_list_page'szz!'#!!   6'? nn$$XD'$J >>))#r3)GDZZ 2 27 ;]*id="ivs_title"[^>]*>(.*?)z<[^>]+>z(.*?)u_徐州市生态环境局))u*索引号[^:]*:\s*]*>(.*?) 索引号)u-发布机构[^:]*:\s*]*>(.*?) 发布机构)u-成文日期[^:]*:\s*]*>(.*?) 成文日期)u-信息名称[^:]*:\s*]*>(.*?) 信息名称)u'文号[^:]*:\s*]*>(.*?)文号zA]*href="([^"]+)"[^>]*>\s*\s*(.*?)\s*u📎 z: zY]*class="cont-p"[^>]*id="ivs_content"[^>]*>(.*?)\s*]*class="cont-loadzc]*class="ewb-article-info"[^>]*>(.*?)\s*]*class="(?:share|ewb-attach|cont-load)z]*>.*?)flagsz]*>.*?z[^<]*zX\s+(?:style|class|height|width|valign|align|bordercolor|cellpadding|cellspacing)="[^"]*"z\s+align="[^"]*")r<r=r>r?r@z

u ：

u!

附件：

u📎u

📎 z

z

 r>)r<r=r>r?z\s+ i,u附件:r?titlecontentdate attachmentssummary) startswithDOMAINrrr r"r#r$r&r'r)r*researchDOTALLsubgroupstripr8finditerappendIsplitjoinr(extend)r+infourlfull_urlr/r0htmlr2rGtm meta_rows meta_patternspatkeymrJamhrefname full_href body_html body_match body_match2 content_partsaurlrHrI summary_lines body_textrKs r3 fetch_detailzXzGovCrawler.fetch_detail;s3:3E3Ec3JXfXgY/PWnn$$Xw$? >>))#r3)GD99;%%gi%@D  YY@$ R FF:r288A;7==?E2D"))DB ++,GLRRT  & LHC #tRYY/A!# B !C!I!I!K # L  ++ P "))  @B88A;$$&D88A;$$&D9=9Mhvhtf5SW ""U4&9+#>? @ YY h "))  "((+113I))vbiiK'--a0668 FF7YbiiXZX\X\n] FF5r9BIIVXVZVZN[ FF.IRTTJ FFvxz}FG FF.I>  Z ZCiIcN$${3%|IcNCSSW%XY Z    +   !D E  819f!5 !!" D! 4ID#!((+%?@ A z2y9IvsI6<<>tDI$$Y/    +   -))M*?immNB?&   G  4QC8 9 sAU U7U22U7c ddl}|j}|jddd|jdtdd |jd dd |j \}}|j r|j y|jrd n |j}td tdtd|jrdn|t|D]}|j|\}}|std|dzdytd|dzdt|d|d|D]} | jddxs| jdd} | jdd} | jdd} | jdd} | jdd}| r| sw| jdr dt | n| }|j#}|j%d ||j&fj)}|j+|r|j,d!xxdz cc<|j/| }|,|j,d"xxdz cc<| d#| d$| d%| gd&| d'| d(}|j1|dxs| ||d)|jd*xs| |jd+d|jd,rd-t|jd,gd.nd}td/| dd0d1t|jd)dd2|t3j4|j6t|d3krytd4|j,d5d6|j,d!d7|j,d"y)8Nrz--full store_trueu 全量爬取)actionhelpz--pagesr;u 爬取页数)typedefaultrsz--syncu 仅同步i'u 📋 分类: u (法定主动公开内容)u📄 最大页数: u无限u ⏹ 第u页无数据，结束u 📃 第u页 (u条, 共u条)rGr: realtitlerZinfodateinfoid categorynamer6r7z5SELECT 1 FROM crawl_results WHERE url=? AND site_id=?skipr9u

信息名称：u

发布日期：rCu信息名称: u 发布日期: rFrHrIrKrJz +u附件u ✅ #z | Bu 📊 累计: +newu 新 | 跳过u | 错误)argparseArgumentParser add_argumentintparse_known_argssyncsync_to_serverfullpagesr*rranger4lenr(rLrM_get_dbexecutesite_idfetchoneclose_statsro store_itemtimesleep sync_delay)r+rparserargs_ max_pagesr,itemsritemrGrZrwrxcategoryr[connexistsdetail attach_infos r3do_crawlzXzGovCrawler.do_crawls((*H\OICPH\ L))+a 99    !  IID4::   ~-HIJ # Hy#IJK *< }J// ;LE5 *Q,/DEF Jz!|nE#e*XeWDQ R. ,"-J+r1J((9b188J3(B/88NB7GKK)Q.)"'%7w>TU]T^^b#c (')%3E7:J8*#U F7O,u9%JJv&2(JJy"- QWPZPZ[hPi3vzz-'C#D"EVLoq uSbzl#c&**Yr2J.K-LAk][\ 4??+]. ,b5zB $T[[%7$8 dkkRXFYEZZcdhdodopxdycz{ |y< }N)r})__name__ __module__ __qualname__r4rorrr3rr%s(p dL}rr__main__zFhttps://sthj.xz.gov.cn/dynamic/zwgk/govInfoPub.html?categorynum=003015zsthj_xz_results.dbg333333?) site_namedomainrldb_namer)__doc__osrNsysrrurllib.requestrssl base_crawlerrrpathdirnameabspath__file__BASE_DIRinsert SITE_NAMErMr!rrr_create_unverified_contextr$r"rrcrawlerrunrrr3rs"43333 77??277??84 58 &  R 2   $c$$&o4  T}:T}n z T$ G KKMr