uj2"dZddlZddlZddlZddlZddlZddlZddlZddlm Z m Z ejddZ ddiZ dZd Zd Zd Zd Zd Zedk(reyy)u 六安市生态环境局 - 行政许可结果 + 公示公告 (EpointWebBuilder CMS) v2: 使用正确的 label/8888 API (pageIndex 有效分页) N)ThreadPoolExecutor as_completed SEARCH_DBz/root/search.dbz User-AgentzB'"&((;";"Ar!%)R!8!>B"%dhhtR&8"9  GD6:,bT 6B C 3'* L  / GD6!- . . /s2DG G8G33G8c|g}tddD])}tjddddddd d d d d ddddt|dtd}d|_|j }tjd|}tjd|}tjd|}|s|St|D]b\}}|t|kr||nd} |t|kr||nd} |jdj|| j| dddt|dkr|Stjd,|S)u=Fallback: 直接从公开页面获取列表（备用方案）r&!z0https://sthjj.luan.gov.cn/luan/site/label/481565r luan_govafalsedescr public_contentr rr)r0z/c1/luan/searchDataList-gk-2020r)r platformCode isAllSitesort withOutDomaintypeCodecolumnIdcatIdsfromCode orderType isLimitScore fuzzySearchfilekeywordsrMr r!utf-8z/public/6608281/(\d+)\.htmlztitle="([^"]*)"z(\d{4}-\d{2}-\d{2})z0https://sthjj.luan.gov.cn/public/6608281/{}.htmlr+r/r)r0r1r7)r@r8r9rAr;encodingtextrefindall enumeraterDrBformatrCrFrG) rHrMrJhtmlidstitlesdatesiid_r)r0s rPget_xzxk_items_from_publicruAsT Ea  MMLkVAQ!Yc3",MCI T B ( vvjj7>.5 :DA  L n FAs!"S[F1IbE 3u:~582D LLIPPQTU#    s8b=  L 3=@ Lcg}tddD]} tjdj|td}d|_|j }tjd |}|s|S|D]}|j|t|d kr|Stjd |S#t$r}td|Yd}~|Sd}~wwxYw) u!检查通知公告是否已全量r&z=https://sthjj.luan.gov.cn/content/column/6806171?pageIndex={}r r#r$rfz tzgg list error:Nz9href="(https://sthjj\.luan\.gov\.cn/zwzx/tzgg/\d+\.html)"r7)r@r8r>rnr;rirjrEr?rkrlrBrDrFrG) tzgg_itemsrMrJrorOurlsus rPcheck_tzgg_fullr~gsJa   \ccdhiszEGHA AJ66D zzVX\]    !A   a  ! t9r>   3!$   & *   s>B.. C7 C  Cc|d} tj|td}d|_|j}|jdd}t jd |}|r#|jd j}|r|}d}t jd |t j}|s%t jd |t j}|r|jd j}t jd d|t jt jz}t jdd|t jt jz}|}|jdd} | s2dD]-} t j| |}|s|jd } n|||| |dfS#t $r||dd|d|dfcYSwxYw)u抓取详情页正文r/r ryrfr)rr0r1z/]*content="([^"]*)"r&z]*>(.*?)\s*z2class="j-fontContent newscontnet"[^>]*>(.*?)z]*>.*?)flagsz]*>.*?)u.发布时间[：:]\s*(\d{4}[-/]\d{2}[-/]\d{2})z*]*content="([^"]*)") r8r>r;rirjrErksearchgrouprCDOTALLsubI) rNr/rJror)mtcontentrawr0pats rP fetch_detailrs u+CD LLgr : vv HHWb !E DdKA GGAJ    EG QSWY[YbYbcA IIKTSUS\S\ ]ggaj ff12s"))BDD.Qff/S "$$O 88FB D u C #t$Awwqz   tF| 44? DT']BV d6lCCDs/F44GGc td}tdtdt}tdt|tdtdt}tdt|dt j t }|jdtd |jd jD}td |jd jD}|jtd t|dt||Dcgc] }|d|vs |}}tdt|g}|D]} | |vs|j| ddddtdt|||z} | s tdytdt| dt j t }|jdd} td5} | D cic]} | jt| | }} t|D]}|j!\} }}}} |jd|| | |||f}|j#|j$dkDr|dz }| dz } | dzdk(s| t| k(srtd| d t| d!| dddtd"dtd#t| d$|d%|jycc}wcc} w#t&$r}td| d|Yd}~d}~wwxYw#1swYtxYw)&Nrz2==================================================u?Step 1: Fetching xzxk (行政许可) list via label/8888 API...z Total xzxk items from API: u,Step 2: Checking tzgg (通知公告) list...z TZGG pages found: r6zPRAGMA busy_timeout=30000c3&K|] }|d ywrN.0rJs rP zmain..!z8SELECT page_url FROM gov_raw WHERE site_name='luan_tzgg'c3&K|] }|d ywrrrs rPrzmain..rrz8SELECT page_url FROM gov_raw WHERE site_name='luan_xzxk'z Existing: xzxk=z, tzgg=r/z New xzxk items to fetch: r luan_tzggrgz New tzgg items to fetch: z' No new items to fetch. All up to date!z Total new items: z!, fetching details (5 threads)...) max_workerszwINSERT OR IGNORE INTO gov_raw (title, page_url, source_url, content, publish_date, site_name) VALUES (?, ?, ?, ?, ?, ?)r&z DB insert error for r52z r4z processed, new= zDone! Processed z new items, inserted .)r?rQrDr~sqlite3connectrexecutesetfetchallcloserBrsubmitrrrcommitrowcountrE) total_newxzxk tzgg_urlsconn existing_tzgg existing_xzxkitnew_xzxknew_tzggr/all_newdonepoolrNfut_mapfr)rr0r1crOs rPmainrs;I (O KL ?D )#d) 56 (O 89!I Y 0 78 ??9 %DLL,-dllB'hjMdllB'hjM JJL c-01]9K8L MN"DrRYm%CDHD 'H  78HX m # OOC"b+V WX 'H  78!G  89 G ~-N OP ??9 %DLL,- D  *MdELMT4;;|T2D8MMg& MA./hhj +Ct ;LLNCgtT: ::>NI AIDbyA~W!54&#g,/? {KL MM$ Bxj/ S\N*? {! LMJJLUE.N ;.se2aS9:: ;MMsZ1 K=>K=9L.>L&L.rsp2111? BIIk#4 5 W X < 0f#L2&5RJZ zFrv