vjdZddlZddlZddlZddlZddlZddlZddlZdZ dZ e dZ dZ ddiZ d Zd Zd Zd Zd ZdZedk(reyy)u 怀远县人民政府 - 建设项目环评审批 (crawl_ahhy_hpsp.py) 蚌埠市信息公开系统 - site/label/8888 API 总计398条, 27页(15条/页) N ahhy_hpspzhttps://www.ahhy.gov.cnz/zfxxgk/site/label/8888z/root/search.dbz User-Agentz> ! !#r ! 2D ::diik((1 22cdg}|jdi}|jdgD]}|jddj}|jdd}|jdxs|jdd}|r|dd }|sd|sgtd |}|j|||d |S) z"Extract articles from API response resultListdatatitle contentId publishDate createDateN z/zfxxgk/public/content/)r2r)date)getstripBASE_URLappend)r1articlesresultitemr2 content_idr8r)s r, parse_listrA$sH XXlB 'F 62&H"%++-XXk2. xx &D$((<*D 9D ZJ5j\BC OOeCF GH Or.c tjj|t}tjj |d}|j j dd}d}tjd|}|r|jd j}d}tjd |tjtjz}|s6tjd |tjtjz}|r|jd j}|sWtjd |tjtjz}|r|jd j}|rptjd d|tjtjz}tjdd|tjtjz}ttjdd|j}|dk\r |xsd|xsdfS|xsddfS#tjj $r}d|j"dk(rdndfcYd}~Sd}~wt$$rYywxYw)zFetch article detail contentrrrrreplace)errorsr3z/]*>([\s\S]*?)\s*zid="zoom"[^>]*>([\s\S]*?)zid="Zoom"[^>]*>([\s\S]*?)z]*>.*?)flagsz]*>.*?z<[^>]+>Ni)NN)rrr r!r"r%r&researchgroupr:DOTALLIsublenerror HTTPErrorcode Exception) r)r*r+htmlr2mcontenttext_lenes r, fetch_detailrW4s"nn$$S'$:~~%%c2%6yy{!!')!< IIH$ O GGAJ$$&E IIA4UWUYUYIY Z ?3;r> w$/\ QUW[G\\ << ! !5QVVs]T44 s*H:I=IJ "I<6J < J  J c tjddtjv}t dg}t }|rdgnt dtdz}|rdn dtd}t d||D]} t|}t|}d }|D]4} | d |vs |j| d |j| |dz }6t d |d t|d |ddtjdt|} t d| | d k(r t dyt dt!j"t$} | j'} | j)d| j+d }d }d }t-|D]\}} | j)d| d f| j/d d kDr|dz }8t1| d \}}|xs| d}|r8 | j)d||| d t2| df| j+|dz }n|dz }|dzd k(s|| dz k(rt d|d| d|d |d!| dtjd"| j5t d#|d |d!|d$y#t$r} t d |d| dYd} ~ d} ~ wwxYw#t$r} t d| Yd} ~ d} ~ wwxYw)%Nz/rootz --incrementalz(=== Phase 1: Collecting article URLs ===rE INCREMENTALzFULL (z pages)zMode: rr)z Page z: z items, z newT)flushz : ERROR - g333333?z Total unique articles: zNo articles found. Exiting.z" === Phase 2: Fetching details ===zCREATE TABLE IF NOT EXISTS gov_raw ( id INTEGER PRIMARY KEY AUTOINCREMENT, title TEXT, content TEXT, source_url TEXT UNIQUE, site_name TEXT, publish_date TEXT )z1SELECT COUNT(*) FROM gov_raw WHERE source_url = ?r2zjINSERT OR IGNORE INTO gov_raw (title, content, source_url, site_name, publish_date) VALUES (?, ?, ?, ?, ?)r8z DB error: z [/z] ins=z skip=z emp=g?z === Done: ins=z ===)oschdirsysargvprintsetrange TOTAL_PAGESr-rAaddr<rMrQtimesleepsqlite3connectTEMP_DBcursorexecutecommit enumeratefetchonerW SITE_NAMEclose) incremental all_articles seen_urlspages_to_fetchmodepager1r=newartrVtotalconncinsertedskippedemptyi detail_titlerT final_titles r,mainr[sHHW!SXX-K 45LI'aSU1kAo-FN'=vk]'-JD F4&/  ="4(D!$'HC u:Y.MM#e*- '',1HC   GD6CM?(3%tDD Q 3   E %eW -. z +, /0 ??7 #D AII   KKMHG EL)3 EE }U ::<?Q  qLG  ,SZ 8 g"2c'l  * A '3u:y#f+N A  QJE r6Q;!uqy. Cs!E7& & ugNVZ [ 336 JJL XJfWIU5' FGk = GD6A3/t < < =T * QC()) *s79$J A J &6J5 J2J--J25 K>KK__main__)__doc__r#r]r_rfrGurllib.requestrrhrpr;rrjr!rdrr-rArWr__name__r.r,rsu 8777  $ J- .  W X  3"  $NMH` zFr.