~W|j'dZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl m Z mZdZedzZdZdZdZej(j+d d Zej(j+d d Zd ZdddedZdZdadZeej<D]k\ZZ e jCdr e"e jGdddZ4e dk(s:edze%ej<ksS e"ej<edzZmedk\rendZ&d"dZ'dZ(dZ)dZ*dZ+dZ,dZ-dZ.d Z/e0d!k(re/yy#e$$rYwxYw#e$$rYwxYw)#u crawl_huizhou_jsxmhjyxpjxx.py - 惠州市-建设项目环评信息 https://www.huizhou.gov.cn/zdlyxxgk/hjbhxxgk/jsxmhjyxpjxx/ 广东政府站群 post_N 格式 (与江城区同构, NFCMS) 列表: table > tr > td > a[id=id_N][href=...content/post_N.html][title=完整标题] + td 日期(YYYY-MM-DD) 分页: index.html / index_N.html (N=2..45, 45页 ~945条, 每页21) 详情: /content/post_NNNNNN.html → meta ArticleTitle/PubDate + div#zoomcon(正文, 含表格) + /attachment/ 附件(绝对URL) N) BeautifulSoupCommentzhttp://www.huizhou.gov.cnz /zdlyxxgk/hjbhxxgk/jsxmhjyxpjxx/u"惠州市-建设项目环评信息u广东zcrawl_huizhou_jsxmhjyxpjxx.pyDB_PATHz/root/search.db JSONL_PATHutf-8zkMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0 Safari/537.36z?text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8zzh-CN,zh;q=0.9)z User-AgentAcceptzAccept-LanguageRefererg333333?gz--pages==z--pagesctjtz }|tkrtjt|z tjat |D]V} t j |t|d}|jdk(r d|_ |cS tjd|dzzXy#t$rY)wxYw)NF)headerstimeoutverifyr) time _last_reqREQUEST_INTERVALsleeprangerequestsgetHEADERS status_codeencoding Exception)urlretriesrdtirs //root/gov_crawler/crawl_huizhou_jsxmhjyxpjxx.pyhttp_getr$;s y B  #b() I 7^  S'75QA}}#$ $ 1q1u9    s$4B:: CCc|sytj|}|jddjddjddjdd}tjdd|}|j S)Nru​u‌u‍u\s+ )html_libunescapereplaceresubstrip)ts r# clean_titler/Mso !A (B''"5==hKSST\^`aA vsAA 779c|dk(rt}n td|dz}t|}|gS|j}g}tjd|tj D]}|j dt|j d|j d}}}|jdr$tjjt|}|j|||f|S)u0抓一页列表, 返回 [(abs_url, title, date)]r index_z.htmlz]*href="([^"]*content/post_\d+\.html)"[^>]*title="([^"]*)"[^>]*>.*?\s*\s*]*>\s*(\d{4}-\d{2}-\d{2})\s*r/)LIST_URLr$textr+finditerSgroupr/ startswithurllibparseurljoinBASE_URLappend) pagerr" html_textitemsmabs_urltitledates r# fetch_listrGVs qy6$u-- Ay I E [[iktvxvzvz{- wwqz;qwwqz+BAGGAJ   c "ll**8W=G gud+, - Lr0c"t|}|y|j}t|d}d}tjd|}|rt |j d}|sCtjd|tj}|rt |j dnd}d}tjd|}|r|j d}|jd } | s||dfS| jd D]} | j||| fS) u-详情页: 返回 (title, pubdate, zoom_soup)N)NNNz html.parserrz+(.*?)z1zparse_detail..s Jq',Br0string) r$r6rr+searchr/r9r8findfind_allextract) rr"rAsouprEtmt2pubdatepmzoomcs r# parse_detailr]js AyI M 2D E A9 MB BHHQK(  YY. 244 @,. BHHQK(BG G SB ((1+ 99 9 "D gt## ]]"B] C  '4 r0c|ddgfS|jgdD]}|j|jdD](}|jddg}|s|j*t|j d}g}|jd d D]}|d j }|j d s|j dr'tjjt||d <|jjds%d|jvsd|jvst|j}|s|j|d |f|jdd D]_} | dj } | j d s| j ds9tjjt| | d<at!|} | ||fS)NrF)scriptstyleiframeobjectc$|xr dt|vS)Nu您当前所在位置)strrMs r#rOzhtml_to_text..s13Z9PTWXYTZ9Zr0rPtddivtableaT)hrefrir4http)z.pdfz.docz.docxz.xlsz.xlsxz.zipz.rarz.wps attachmentuploadimg)srcrn)rT decompose find_parentboolrSr-r:r;r<r=r>lowerendswithr/get_textr?rd) zoom_souprDtagtxtanc has_table attachmentsrhhrmrNcontents r# html_to_textr}s5"}!!"IJ !!)Z![ootUm,  MMOY^^G,-IK   $  /5 fIOO  <<  V 4 ,,Xq9AfI 779  ` aequvu|u|u~e~CKOPOVOVOXCXajjl+C""AfIs#345!!%T!2; J    <<  V 4--h:CJ;)nG I{ **r0c|s|S|jr|gng}|D]\}}|jd|d|ddj|S)Nz

z

z )r-r?join)r|rzpartsrDrws r#append_attachmentsrs[  WIbE#N  |G9,>se8LMN ;;u r0c |jdtf}|jDchc]}|d c}Scc}w#t$r t cYSwxYw)Nz.SELECT page_url FROM gov_raw WHERE site_name=?r)execute SITE_NAMEfetchallrset)conncurrows r#load_existing_urlsrsNllKi\Z"%,,.13A111 u s)? :??AAc |jd|tf}|jry d} ddlm}||}|jd|||||tt t||f }|jdkDrytjdd|} tj| } tjd d | jdd } |jd |j|t| fy y#t $rYwxYw#tj $r=} dt#| vr*t%j&dt)||||||cYd} ~ Sd} ~ wwxYw)Nz7SELECT id FROM gov_raw WHERE page_url=? AND site_name=?Fotherr)classify_industryzINSERT OR IGNORE INTO gov_raw (page_url, source_url, title, content, publish_date, site_name, group_name, script_name, has_table, industry) VALUES (?,?,?,?,?,?,?,?,?,?)<[^>]+>rr&r'rzTINSERT OR IGNORE INTO gov_search (rowid, title, site_name, summary) VALUES (?,?,?,?)Tlockedr3)rrfetchone crawler_librr GROUP_NAME SCRIPT_NAMErowcountr+r,r(r)r- lastrowidsqlite3OperationalErrorrdrr store_record) rrrEr|rFryrindustryrsummaryes r#rrsF ,,PSVXaRb cC ||~  5(/Hll +#ugtY KQZ\d e   <fETR    K {& 5TB D Ds /1KK __main__)r3r)1__doc__sysosr+rhtmlr( urllib.parser;rrrbs4rrr>r5rrrenvironrrrENCrrr _MAX_PAGES enumerateargvr!rhr:intsplit ValueErrorrrr$r/rGr]r}rrrr__name__r0r#rs  & ' 8 8 0  - **..$5 6 ZZ^^L" -  @O'     chh  DAq||J QWWS!_Q/0J iAECM1 SXXa!e_-J "QA$( 8+8DBCJ zFS       s$-D6)E6D>=D>E E