`otjdZddlZddlZddlZddlZddlmZmZejjdejjejje ddl m Z ejedz jdZdZd d iZd Zd Zd ZddZedk(r$dej0vZeeZededyy)u\crawl_chaozhou_gsgg.py - 潮州市人民政府-公示公告 (静态p.over1 + 详情div.info)N)datetime timedelta)push_to_searchdbiG)daysz%Y-%m-%du潮州市-公示公告z User-Agentz6Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120.0z%https://www.chaozhou.gov.cn/zwgk/gsggc|dk(r tdz}ntd|dzd} tj|dtd}d |_t j d |jt j}g}|D]p\}}}|jd r|}nA|jdrd |z}n*|jd r t|ddz}n tdz|z}|j|||dr|S#gcYSxYw)Nr/z/index_z.htmlFtimeoutheadersverifyutf-8zd]*>\s*

.*?(.*?).*?

\s*(\d{4}-\d{2}-\d{2})httpzhttps://www.chaozhou.gov.cnz./)urlpub_datetitle) BASErequestsgetHEADERSencodingrefindalltextDOTALL startswithappend) page_numrritemsresultshrefrrfull_urls (/root/gov_crawler/crawl_chaozhou_gsgg.pyfetch_urls_from_pager(s1}Sjghl^51 LLgwu M  JJo   EG!& PeX ??6 "H __S !4t;H __T "d12hHczD(HxXNO P N# s $C,,C2ctdD](} tj|dtd}d|_nyd}t j d|jt j}|r4t jd d|jd j}|st j d |jt j}|rYt jd d|jd j}t jd d|j}d}|jjd }|dkDr|td z}d }|} | t|jkrh|dkDrc|j| | dzdk(r |d z }| dz } n%|j| | dzdk(r |d z}| dz } n| d z } | t|jkr|dkDrc|j|| dz j}||fS#d}YxYw)Nr Fr r)r+r+z]*>(.*?)z<[^>]+>r z([^<]*)z \s*[-_|].*$z
rz)rangerrrrrsearchrrsubgroupstripfindlen) r_attemptr"rmcontentidxstartdepthposs r' fetch_detailr<)s!H  S'75QA AJ   y E (!&&"))EFF>2u5;;=EG &&++* +C Qwc.//CKEAIvvc#a% F* qCE"h. qqCKEAI&&s1u%++- '>; As $HH c >g}t}|rdnd}t|D]}t|}|sntd|dzdt |d|D]q}|dt kr|d|vr|j |dt|d\}}|r|jsP|j||d|dt|dd std t |d|D cgc]} | d js| } } | r t| d t | Scc} w) Nr z Page z: z itemsrrr+)rrr site_namer7summaryz Total: r7 chaozhou_gsgg) setr.r(printr4CUTOFFaddr<r2r SITE_NAMEr) incrementalrecordsseen max_pagespgr#itrr7r"valids r'runrNNs7G 5D bII$R(  1vRE |623 B*~&%yD  HHRY )"U)4NE7  NN%yzN&"    , Ic'l^6 *+ 811Y<#5#5#7Q 8E 8 0 u: 9s "D;D__main__z --incremental)rGzDone: z records)F)__doc__sysosrrrrpathinsertdirnameabspath__file__ crawler_librnowstrftimerDrFrrr(r<rN__name__argvinccntrCr'rasb(277??277??8#<=>( (,,.9%0 0 : :: F $ Q R.6#J@ z SXX %C # C F3%x !r`