`otj! d Z ddlZddlZddlZddlZddlmZmZ ej j dej j ej j e ddl m
Z
ej ed z
j d ZdZd d
iZd Zd Zd
ZddZedk( r$dej0 v Z ee Z ede d yy)u[ crawl_bynr_sthj.py - 巴彦淖尔市人民政府-生态环境 (TRS静态 + trs_editor_view) N)datetime timedelta)push_to_searchdbiG )daysz%Y-%m-%du 巴彦淖尔市-生态环境z
User-Agentz6Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120.0z1https://www.bynr.gov.cn/zfxxgk/fdzdgk/zdlyxx/sthjc t | dk( r
t dz }nt d| dz d} t j |dt d }d |_ t j d
|j t
j }g }|D ] \ }}}t j d d | j }t j d
d | j }t | dk r`|j d r
t |dd z } n#|j d rd|z } n t dz |z } |j | ||d |S # g cY S xY w)Nr /z/index_ z.html Ftimeoutheadersverifyutf-8z6]*>(.*?).*?(\d{4}-\d{2}-\d{2})<[^>]+> \s+ z./zhttps://www.bynr.gov.cn)urlpub_datetitle)BASErequestsgetHEADERSencodingrefindalltextDOTALLsubstriplen
startswithappend)
page_numr ritemsresultshref
title_htmlr r full_urls
$/root/gov_crawler/crawl_bynr_sthj.pyfetch_urls_from_pager/ s0 1}Sjghl^51 LLgwu M
JJA
E G&+ P"j(z2z288:vr5)//1 u:> ??4 d12hH
__S
!047HczD(HxXNO P N' s $D1 1D7c t d D ]( } t j | dt d }d|_ n yd|j
v ryd}t
j d |j t j }|rat
j d
d|j d j }t
j d d| j }|rd
|vr|}|st
j d|j t j }|rat
j d
d|j d j }t
j d d| j }|rd
|vr|}|sdD ] }t
j ||j t j }|s5t
j d
d|j d j }t
j d d| j }|sd
|vs|} n |st
j d|j t j }|rt
j d
d|j d j }t
j dd| }t
j dd| }t
j dd| }t
j dd| }t
j dd| j }d}t
j d|j t j }|r |j d j }|sPt
j d|j t j }|r |j d j }|sPt
j d|j t j }|r |j d j }||fS # d }Y xY w)N r
Fr
r )r r z/404/r z]*>(.*?)r r z^begin-->|end-->$404z,