uj$dZddlZddlZddlZddlZddlZddlZddlmZddl m Z m Z ejddZ dZ dZd Zd Zd Zd Ze j&e d z j)dZddddZej.Zej2j5ddddZdZdZddZedk(rXdej@vZ!dZ"ej@D].Z#e#jIdse%e#jMddZ"0ee!e"yy)z crawl_jl_gov.py -- Jiangling County Government Announcements Site: http://zwgk.jiangling.gov.cn/list.shtml?column_id=7691 CMS: Custom (Jingzhou News Network), POST API for list data N) BeautifulSoup)datetime timedelta SEARCH_DBz/root/search.dbu"江陵县人民政府-公告信息zzwgk.jiangling.gov.cnzBhttp://zwgk.jiangling.gov.cn/api/content_center/document/list-datai iG)daysz%Y-%m-%d)newskiperrorszB B# BB#B#c tj|d}d|_|jdk7ry|j}t |d}|jddd i }d }|r|jd r|d dd }|jdd}|s|jdd}d }|r t|}||fS#t $rYywxYw)Nr)rzutf-8r)r+ html.parsermetanamePubDate)attrsr+content divmainText)iddoc_text)class_) rr#encodingr"textr$rfindstr)urlr& html_textsoup meta_datedate_str content_div content_htmls r( fetch_detailrC3s KKRK ( ==C FF  M 2D &(;  %LA L &MQ M   qc"# h1sB=C C2 C--C2c tdtdtdd}d} td|dt|}|s tdnStdt|d |dd dd d |d d dd dt|D]\}}|j dd}|j dd}|j d ddd } | r/| t kr&td| d|ddt dxxdz cc<ptd|dzd|ddd| |r||k\rtd|dyt|\} } | xs| } t||| | |dz }tjd|d d dd } | t krtd| dt dn |dz }|dkDrn|td t d!d"t dd#t d$y)%Nz<============================================================z1Jiangling County Government Announcements crawlerrrIz --- Page z ---z No more itemsz Got z items (rel_timer2z ~ )rUr+pub_urlz [SKIP] Too old: rErr z [z] (z | z Test mode: z items doneg?z Reached 3-year cutoff (z < <z Done! New: r z, Skip: z , Errors: r ) r%r)len enumerater#THREE_YEARS_AGOrRrCrYr!sleep)fulltest_nseenpageitemsidxitrUr^r[r1 detail_dater@ items_dates r(crawlrnds (O => (O D D  D6&'4  # $  s5zl(58J+?+D*ESrS]I^_b`bIcHddefg ' GCFF7B'EffY+Gvvj"-cr2HH6*8*AeCRj\BCf "  CAwbs Cz: ;$&.x{;<#/#8 G[".hH ugw 9 AID JJsO) .2Yz*3B/  ' -j\_rs$ ( "))K!2 3 0  N    8<<>I5$99CCJO1* (   P4 ("<&1\h z sxx D Fxx, >>) $3*+F, tF# ro