vj dZddlZddlZddlZddlZddlZddlZdZdZ e dZ dZ de dd Z d Z d Zd Zd ZdZdZdZedk(reyy)u 南陵县生态环境分局 - 行政审批 (crawl_nlx_xzsp.py) 芜湖市政府网站群CMS API: GET /wuhu/site/label/8888?labelName=publicInfoList (列表) GET /wuhu/site/label/8888?labelName=publicContentDetail&isJson=true (详情) Nnlx_xzspzhttps://www.nlx.gov.cnz/wuhu/site/label/8888z/root/search.dbzzfetch_list..*s[/q!@J!S?QQCq--c!f567Z[Y\\]^_]`Waa/sAA?headerstimeoututf-8)SITE_IDORGAN_IDCAT_IDjoinitemsr$r%Request LABEL_URLHEADERSurlopenjsonloadsreaddecode) page_indexparamsqsreqresps r+ fetch_listrFs&"F /%||~/ /B .. I;at!4g FC >> ! !#r ! 2D ::diik((1 22cd|dd}djd|jD} tjj t d|t }tjj|d }tj|jjd S#t$r}td |d |Yd }~y d }~wwxYw)zFetch article detailpublicContentDetailr )r contentIdrrc30K|]\}}|d|ywr r's r+r,zfetch_detail..8s8AQCq*8sr-r.r0r1r3z Detail error for : N)r7r8r$r%r9r:r;r<r=r>r?r@ Exceptionprint) content_idrBrCrDrEes r+ fetch_detailrR1s+F 88 8Bnn$$ {!B4%8'$J~~%%c2%6zz$))+,,W566  %j\A378sBB// C8CCc@|sytjdd|tj}tjdd|tj}ttjdd|j }|dkrytjdd|}|S) zClean up HTML contentrz]*>.*?)flagsz]*>.*?z<[^>]+>r0z(src|href)=(["\'])(/[^"\']+)cz|jdd|jdt|jdS)Nr!)groupBASE_URL)ms r+zclean_content..Ms1qwwqzl!AGGAJ<z!''!*NrG)resubDOTALLlenstrip)htmltext_lens r+ clean_contentrdBs  66.D JD 66,b$bii HD266*b$/5578H"} 661NPT VD KrGc tjddtjv}|r t dn t dt dg}d}t d|dzD]} t |}|jdg}|jd d }|jd d }||zdz |z}|D]>}|j|d |d|jdd|jddd@t d|d|dt|dt|d ||k\rn|rntjdt dt|dt dd dl } | jt} | j!} | j#d| j%d } d }d }t'|D]\}}| j#d|df| j)d d kDr|dz }8t+|d}|re|jd rTt-|d }|r> | j#d!|d||dt.|d"dd#f| j%| dz } n |dz }n|dz }|d#zd k(st d%|dt|d&| d'|d(| | j1t d)t d*t|t d+| t d,|t d-|y#t$r} t d|d| Yd} ~ d} ~ wwxYw#t$r} t d$| Yd} ~ d} ~ wwxYw).Nz/rootz --incrementalzMode: INCREMENTAL (page 1 only)z Mode: FULLz$=== Phase 1: Collecting articles ===crVdatatotalrrr rJtitle publishDaterlink)rPri publish_daterkz Page /rMz items, z totalg333333?z ERROR: z === Collected z articles ===z+ === Phase 2: Fetching details & saving ===zCREATE TABLE IF NOT EXISTS gov_raw ( id INTEGER PRIMARY KEY AUTOINCREMENT, title TEXT, content TEXT, source_url TEXT UNIQUE, site_name TEXT, publish_date TEXT )z1SELECT COUNT(*) FROM gov_raw WHERE source_url = ?rPcontentzjINSERT OR IGNORE INTO gov_raw (title, content, source_url, site_name, publish_date) VALUES (?, ?, ?, ?, ?)rl z DB error: z [z ] inserted=z skipped=z empty=z === Done ===zTotal: z Inserted: z Skipped: zEmpty content: )oschdirsysargvrOrangerFgetappendr`timesleeprNsqlite3connectTEMP_DBcursorexecutecommit enumeratefetchonerRrd SITE_NAMEclose) incremental all_articles total_pagespageresultr8rhper_pageitemrQryconncinsertedskippedemptyiartdetailrns r+mainrQswHHW!SXX-K /0 l 01LKaq) %FJJvr*EJJw*Ezz*b1H 8+a/H 89 ??7 #D AII   KKMHG EL)f3 EF ~V ::<?Q  qLG c,/0 fjj+#F9$56G0IIEWwF YNH[\_]_H`aKKMMH  QJE r6Q; Cs!C -.k(9WIU\]b\cd e5f8 JJL N GC %& '( Jxj !" IgY   OE7 #$o  GD6!- .  P!0N1#.//0s=#B>K9#K9'K9&>M__main__)__doc__r=rprrrwr]urllib.requestr$rrZr:r{r;r5r6r4rFrRrdr__name__rLrGr+rs /..  #j- . PRS    32" \%~ zFrG