0@j <dZddlZddlZddlZddlZddlZdZdZdZdddd d d d d d ddd dddd dddd dddd dddd dddd dd d!d d"d#d$d d%d&d'd d(d)d*d g Z d+Z e Dcgc]}|d, c}Z iZ iZ d-Zd.Zd/Zd0Zd9d1Zd2Zd3Zd4Zd5Zd6Zd7Zd8Zycc}w):u4 爬虫管理页面模块 - 注入到 search_app.py N)/root/gov_crawler/daily_crawl_config.jsonz/root/search.dbz/root/gov_crawlerpoweru电力Power)idcnen terminalsu终端/接收站 Terminals transmissionu 管道输送 Transmission productionu 生产/开采 Productionaltfuelu 替代燃料zAlternative Fuelhpiu 石油炼制z%HPI (Hydrocarbon Processing Industry)cpiu 化工加工z"CPI (Chemical Processing Industry)metalsu金属与矿物zMetals & Mineralspulpu制浆/造纸/木材zPulp, Paper & Woodfoodu食品与饮料zFood & Beverage manufacturingu 工业制造zIndustrial Manufacturingpharmau制药与生物技术zPharmaceutical & Biotechc8ddl}|jtdS)NrF) ensure_ascii)jsondumps INDUSTRIES)_js "/root/gov_crawler/crawler_admin.pyget_industries_jsonrs 88JU8 33rc|sydD]#}|j|s|t|d}n|tvrtj|dStj j t|}tj j|s dt|<y t|dd5}|jd}dddtjdtj}|s%tjd |tj}|s%tjd |tj}|r|jd nd}|t|<|S#1swYxYw#dt|<YyxYw) z9Extract SITE_NAME constant from script source to match DBznode zbash zpython3 Nrreplaceerrorsz$^SITE_NAME\s*=\s*[\"']([^\"']+)[\"']z^NAME\s*=\s*[\"']([^\"']+)[\"']z^SITE\s*=\s*[\"']([^\"']+)[\"']) startswithlen_SCRIPT_SITE_CACHEgetospathjoinGOV_CRAWLER_DIRisfileopenreadresearch MULTILINEgroup) script_nameprefix script_pathfheadmsns rextract_script_sitenamer?%s? 0  ! !& )%c&kl3K ((!%%k266'',, >+ &*,;' +s9 5 66$bllSA II=tR\\ R II=tR\\ RbB&({# I  *,;'s$E&E8EEE E,c|sydD]#}|j|s|t|d}n|tvrtj|dStj j t|}tj j|s dt|<y t|dd5}|jd}dddtjd}|stjd|}|stjd |}|stjd |}|stjd |}|r |jd jd nd}|t|<|S#1swYxYw#dt|<YyxYw)Nr!r"r#r$r%r'zB(?m)^(?:LIST_URL|list_url|listUrl)\s*=\s*["'](https?://[^"']+)["']z?(?m)^(?:API_URL|api_url|apiUrl)\s*=\s*["'](https?://[^"']+)["']zP(?m)^(?:PAGE_URL|SEARCH_URL|search_url|BASE_URL)\s*=\s*["'](https?://[^"']+)["']zY(?m)^(?:BASE|base_url|DOMAIN)\s*=\s*["'](https?://[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}[^"']*)["']z'(https?://[a-zA-Z0-9.-]+\.[a-zA-Z]{2,})r(/)r)r*_SCRIPT_URL_CACHEr,r-r.r/r0r1r2r3r4r5r7rstrip)r8r9r:r;r<r=urls rextract_script_urlrEBsk 0  ! !& )%c&kl3K '' $$["55'',, >+ &)++& +s9 5 66$cbtjtt}t |t rd|i}|r|t vrdd|dS|dD]W}|d|k(s |r||d<n|jddtj|ttddd d ||xsd d cSdd|ddS)z$Set industry for a crawler in configcrawlersFzUnknown industry: successerrornameindustryNwrindentTr!)rIrKrL Crawler " " not found) rloadr2 CONFIG_PATH isinstancelist INDUSTRY_IDSpopdump)rK industry_idcfgcs rset_crawler_industryr]gs ))D% &C#t3{,6 -? }+MNN _R V9  +* j$' IIc4 S1a P#T{?PbQ QRD6'F GGrc v12tjtt}t |t rd|i}|d}t jt}t j|_ |jdj}|Dcic]}|dt|}}d}|jDcic]\}}|||} }}|jg} |D]} | j!d| j!d| j!dd} | j!d| j!dd} | }d D]#}| j#|s| t%|d }n|r$t&j(j+t,|nd}|rt&j(j/|nd }|j!| i}|sd | vrt | d t r| d }|jDcgc] \}}||vs |}}}|r\t1d |Dt3d|Ddt5d|Ddt1d|Dt1d|Dd}|sG| j!|| i}|s(|| }| jD] \2}2|vs |}n|s | jD] \2}2vs |}n|si| jD]V\2}t5t%t%2}|dk\s*d}t7|D]}||2|k(r|dz }n|dk\sT|}n|sjj9ddj9ddj9dd}t%|dk\r$| jD]\2}|2vs2|vs|}n|sbgd}}|D]$}|j;|s|d t%| }nt%|dk\r%||k7r | jD] \2}|2vs |}n|st=| }|r ||vr||}|s| rt&j(j?| jAdd}|j#drL|dd 1|jD]4\2}1s 12vs%t%1dk\stC12fdd Ds2|}n|j!d!d}|j!d"d}|j!d#d} |j!d$d}!|j!d%d}"d x}#x}$x}%}&|rf|dk7ra dd&l"m"}'m#}(|'jI|d d'd(})|'jK|)z }*|*|(d)k}#|*|(d*)k}$|*|(d+)k}%|*|(d,)k}&d-}+d}, td.5}-tMd/j|-}.d d d tO.D]}/|/j!d| k(s|/j!d0d},|/j!d1rd2}+nV|/j!d3rd4}+nB|/j!d5d6dk(r|/j!d7d}0d8|0d9}+nd:|/j!d5}+n| jQid| d| d;| j!d;d<d=| j!d=d d>| j!d>dd!|d?|d@| d$|!d%|"dA|&dB|#dC|$dD|%dE|&dF|dG| j!dGd| j!dHd| j!d gtS| |+|,dI"| |fScc}wcc}}wcc}}w#YxYw#1swYxYw#YxYw)Ju!获取所有爬虫的状态数据rGa SELECT site_name, COUNT(*) as total, MAX(SUBSTR(publish_date,1,10)) as latest_date, MIN(SUBSTR(publish_date,1,10)) as oldest_date, SUM(CASE WHEN SUBSTR(publish_date,1,10) >= date('now', '-7 days') THEN 1 ELSE 0 END) as week_new, SUM(CASE WHEN SUBSTR(publish_date,1,10) >= date('now', '-1 days') THEN 1 ELSE 0 END) as day_new FROM gov_raw GROUP BY site_name site_namec|jddjddjddjddjdd jd djd dS N r!—-u–u（(u）)u　 r$ss r normalizez$get_crawler_stats..normalizesyyb!))%5==eSIQQRWY\]eefkmpqyyzBDEMMNRTVW WrrKscriptr!commandr"NF site_namesc3@K|]}|jddyw)totalrNr,.0r=s r z$get_crawler_stats..s Dqw!2 Dc3vK|]1}|jds|ddk7s|jdd3yw) latest_daterdNrqrrs rrtz$get_crawler_stats..G(HaVWV[V[\iVjopq~oDGpGmS(A(H 999rd)defaultc3vK|]1}|jds|ddk7s|jdd3yw) oldest_daterdNrqrrs rrtz$get_crawler_stats..rxryc3@K|]}|jddyw)week_newrNrqrrs rrtz$get_crawler_stats..s#JQAEE*a$8#Jruc3@K|]}|jddyw)day_newrNrqrrs rrtz$get_crawler_stats..s"H1155A#6"Hru)rprwr|r~rrr(rc·rN) 通知公告 公示公告 环境保护 环评信息 环评公告 环境信息 政策文件u 新闻专区 环评审批环境影响评价u高新区动态 公告公示u生态环境保护z.pycrawl_c3,K|] }d|v yw)N)rsislugr>s rrtz$get_crawler_stats..sNobctTVUVx[]~Nos)rrprwr|r~r)datetime timedelta %Y-%m-%ddaysu 待运行z/tmp/crawl_results.jsonrtstimeoutu ⏰ 超时 has_erroru ❌ 失败rc new_countu✅ +u条u⚠ rc=r7u 未分组enablednotelatestoldestactive active_3d active_7d active_30d active_180d script_existsrLrD)rDrn source_urllast_run_status last_run_ts)*rrSr2rTrUrVsqlite3connect SEARCH_DBRow row_factoryexecutefetchalldictitemscloser,r)r*r-r.r/r0r1summaxminrangesplitendswithr?basenamer$anyrrstrptimenow __import__reversedappendrE)3r[rGdbrowsr#db_statsrkkv db_stats_normresultr\rKrl script_filer9r:rstatssite_names_listmatched name_normmin_lencommonrbasecommon_suffixessfxsite_name_from_script script_baserprrr~rrrrrrrlddeltarr_f_results_rnew_crr>s3 @@rget_crawler_statsrxs ))D% &C#t3:H  #B[[BN ::    266A+Q'6H6W191ABAYq\1_BMBHHJ F P uuVQUU;h0CDExy"!56 4 F  ($S[\2  EPbggll?K@UW 7B{3  T2&*z!L/4/P oO%-^^%5NTQo9MqNGN DG DD#&(H7(HRU$V#&(H7(HRU$V ##J'#J J""H"HH !%%ior:E%dO *002EBY !*002EB B !*002 "EB!#i.#b':G!|!"!&w&A(|r!u4 &!  % & "Q;$%E! " s+A.44U;A>DDXNqQt9>!.!4!4!6"A2:t$%E!"#\!*C}}S)#Jc#hY/t9>di&7!.!4!4!6"A2:$%E!"(?(G%(-Bh-N$%:;E"$''"2"26":"B"B5""MK"--h7*12%-^^%5&EB#D Q3NognNoKo() %& '1%=#.=#.99Z+))Iq);@@ @I@ [ fm 8&&vcr{J? +!YA%66 !YA%66 "iR&88 #yc':: &  /0 7B%f-2226 7x( 66(#v-"$&&q/Kvvi(*6 ,*6b)Q. "{A 6,1%*<,3BFF4L>*B    D "F ,3QUU7K5P quuY. 06fb8I  U %f /7    #,W  k  $/  |:D]N}8|=|;j%|8dz }=|>|| j[|=| j[|>P|j]| | | r| nd||||d2}?|?Scc}w#YxYw#t>$r}(tAd|(Yd}(~(Fd}(~(wwxYwcc}wcc}w#YxYwcc}8wcc}8wcc}8w)3uO 获取分析数据 period: day/week/month mode: incremental / cumulative metric: records (数据量) / 脚本数 (脚本数) / both (双指标) days: 回溯天数 (0=全部) 脚本数按唯一脚本文件(.py)统计，脚本的新增日期取该脚本数据 在 search.db 中最早的 publish_date。 rN) defaultdictra SELECT site_name, MIN(date(publish_date)) as d FROM gov_raw WHERE publish_date IS NOT NULL AND SUBSTR(publish_date,1,10) != '' AND date(publish_date) IS NOT NULL AND date(publish_date) != '1970-01-01' GROUP BY site_name r_drGc|jddjddjddjddjdd jd djd dSrarhris rrkz$get_analysis_data..normalizeRsayyb)11%=EEeSQWUC()= date('now', '-z days')dayzdate(publish_date)weekz*date(publish_date, 'weekday 1', '-7 days')u%strftime('%Y-第%W周', publish_date)z"strftime('%Y-%m-01', publish_date)u$strftime('%Y年%m月', publish_date)z SELECT z as period_date, zN as period_label, COUNT(*) as cnt FROM gov_raw zC GROUP BY period_date ORDER BY period_date ASC period_labelcntru %Y-第%W周ru年u月c|ddS)Nr(rrxs rz#get_analysis_data..s1arkey incremental)rr)labels record_counts spider_countsperiodmodemetricr)/rr-r collectionsrrrrrrrrr.existsrSr2rUrVr,r4substriprrr*rrrgetmtime fromtimestampstrftime Exceptionprintrrrdateweekdayr$intsortedkeyssetrziprr)@rrrr_json_os_dt_ddr records_data spiders_datarcfg_path db_site_datesrr# scripts_mapr[rGrkrdb_normr\rlrKearliestrr>r_reclean clean_normsn_normrrrrrrmtimee rec_where rec_date_exprrec_label_exprq_recrec_rows rec_labelsrec_incfiltered_scriptsrjcutoffplabelddmondayspi_period_cnt spi_labelslspi_inc all_labels rec_inc_dict spi_inc_dictr_carrys_carryrs@ rget_analysis_datar-%sn.  #B[[BNLL F:HM ::   3 [>af,-cFM!K. )3 K xxx r :**T(^,C#t$owwz2.H d 1>>1y|Q>G>c x,uuVR(T%dO  1$AlOT)J o-. -b 1A'/1x<+, -# (=,T2H# W(<,WY-?@H#$GGKT:@@BE!*5!1J!W,#01D#E#'.}}- $/ -b 1A'/1x<+, -#'.}}- "i/ -b 1A'/1x<+, -#'.}} 1 "%c)nc'l"C"a<%&F%*7^*#,Q<71:#=$*aKF$) * &{$1"$5#+#3q8|/0H 1#$??3/288?BD4yA~+2==?1KGR#w$1"$5#+#3q8|/0H 1#'SO$D."==-#' #c(#3D!"4yA~$)*;+2==?1KGR#w'T/$1"$5#+#3q8|/0H 1 '/ F## # 1 12F2O P.1ll.H.H.O.X.XYc.d F+Cc NxI ax@gNN  , - 6 D @< ?     Ezz% ))+H-56!N#6J6!)*Aqx*G*!!#*1 9 )),>qc*BCLL..u5>>zJ !8ll&&(3==d=+CCMMjYF6z U?F v &&q*5::A[- [-*A4[- [-, [- [-?[-)[- [-A[-& [-3"[-A[- [-' [-5A [%[- \- \A\ \$+\) \. [-%[*'[-- \6\  \\!c t\}}t|}td|D}||z }ddl}|jt j djd}|jt jtd|D}td|D}td|D} td|D} i} |D]c} | d } | | vr ddddd | | <| | d xxd z cc<| d r| | d xxd z cc<| | dxx| d z cc<| dsT| | dxxd z cc<eddl m }|jjd}td|D}td|D}g}|jdjgd|d|d|d|r||z dznddd|d|r||z dznddd|d|r| |z dznddd| d |r| |z dznddd!| d"|d#|d$|d%|d&|d't| jD]#\}}|jd(|d)|d*|d d'%|jd+t|d,-D]%} | d sd.nd}| d r| drd/n| d r | d0dkDrd1n| d rd2nd3}| d4}| d5s d6| d4d7}d8| d9}| d r d:|d;| d9d<n d:|d=| d9d>}| j!d?dxs| j!d@d}|rd:|dA|dB|ddCdD}ndE}| dF}dG|vrdHn dI|vrdJndK|vrdLndM}|jdN|dO|dP| d9dQ| d9dR| d dS| dTxsddUk7r | dTxsdddVndUdR| d0dW| drdGndKdX| d dY|dZ|d[|d\|dX|d](|jd^dj|S)_u渲染爬虫管理HTML页面c3,K|] }|ds dyw)rr(Nrrsr\s rrtz$render_admin_page..s69!6s rNzSELECT COUNT(*) FROM gov_rawc38K|]}|ds |dsdyw)rrr(Nrr0s rrtz$render_admin_page..!K!1[>a lAK c38K|]}|ds |dsdyw)rrr(Nrr0s rrtz$render_admin_page.."r2r3c38K|]}|ds |dsdyw)rrr(Nrr0s rrtz$render_admin_page..#sM1AlO) QMr3c38K|]}|ds |dsdyw)rrr(Nrr0s rrtz$render_admin_page..$sOAQ}-=!I,aOr3r7)rprrecordsrrpr(rr7r)rz%Y-%m-%d %H:%Mc3&K|] }|d yw)r~Nrr0s rrtz$render_admin_page..4s5qQz]5c3&K|] }|d yw)rNrr0s rrtz$render_admin_page..5s3QAiL3r9r!u4 爬虫管理面板

🔍 爬虫管理面板

ud
总数据条数
rAu*
爬虫 (启用/总数)
活跃站点（启用数）
近3日
u
近7日
u
近1月
u
近半年
zS
+ub
今日新增
+ua
近7日新增
u
已禁用
▶️
手动日跑
更新时间: u[
z
ch|d |jdxsdddd|jdd fS)Nrrr!rrprrqrs rrz#render_admin_page..sA9-=h@USUW[Y[W[?\_`_d_delmn_o^o,prrdisabledu🟢r~u🟡u⚪u⚫rlru!⚠ zz/crawler/admin/toggle?name=rKz 禁用u@" class="toggle-btn toggle-off" onclick="return confirm('启用 u？')">启用rDrz" target="_blank" title="z=" style="color:#2980b9;font-size:11px;text-decoration:none;">7zz2-ru✅z#27ae60u⏰z#e67e22u❌z#e74c3cz#999z u-
状态名称条数最新 7日新增活跃分组脚本来源URL日跑状态操作
zi z% z rrdrz9 z zr zr z< z
数据量 脚本数
)rr*rrrrrfetchonerrrrrr/rrr,)rGr[rprr=r total_recordsrrrrgroupsr\grnow_str week_total day_totalpartsgnamegdatatr_class active_markerscript_display toggle_url toggle_btn _display_urlurl_cell run_status run_colors rrender_admin_pagerRs%'MHc ME6X66GwHOGOOI$>$F$FGe$f$o$o$qrs$tMwFv}wFwFGPwQwWwWwYKxKKIKxKKIMMMJOOOK F  % gJ F?"#a1MF1Iq 'a Y< 1Ii A % q )' * X; 1Ih 1 $  %"lln%%&67G5H55J3(33I E LLllU1lj2?klj@0kll18ymll9:mll;@mllArmltJOr{}BsBCFsFUVWZr[ult[TultU^T_ult_rulvJOr{}BsBCFsFUVWZr[wlv[TwlvU^T_wlv_rwlxKPr|~CsCDGsGVWX[r\ylx\UylxV`UaylxatylzNStAFuFGJuJYZ[^t_{lz_X{lzYdXe{lze2{l@3<Al@=1AlB2< ClB=/ClD08jElD9LElRMT9SlRU?SlV@EgWlVFWllZv||~.l u 8uUG2eT[nM]]ijkl LLH"p q%&y\:r"#I,1X;qQZ|`abl`mpq`qVBCDMBNx}TY 8!@8 WUN21V9+> OPQZO[y ,lmnoumvlwxJKclmwlxxyz{|BzCyDDVaW uuUB'B155r+B GSUacopsqsctuHKH() !&*!4IQ[H[9otyCpCbkIO  XJ/oddefldmcnnpqrsyqzp{| !' |,34X;3D"2Lq{(b#2.RUVW !* /4=>x[Ee3TUzlmm{l|}mmulvw77@kZLQl  #> LLRRf 775>rcttjtt}t |t rd|i}|dD]r}|j d|j dd}||k(s+|r||d<n|jddtj|ttddd d ||d cSdd |ddS)u设置爬虫的来源URLrGrKr_r!rDNrMFrNrOT)rIrKrDrQrRrH) rrSr2rTrUrVr,rXrY)rKrDr[r\cnames rset_crawler_urlrUs ))D% &C#t3 _?faeeK45 D=%eT" IIc4 S1a P#T#> >?4& 'D EErc*tjtt}t |t rt |i}|dD]I}|d|k(s |d |d<tj|ttdddd||drdnd d cSdd |d d S)u切换爬虫启用/禁用状态rGrKrrMFrNrOTr=)rIrK new_statusrQrRrH)rrSr2rTrUrVrGrY)rKr[r\s rtoggle_crawlerrXs ))D% &C#to _l V9  |+AiL IIc4 S1a P#TaPYl`jk k l 4& 'D EErz/tmp/daily_crawl_run.pidc$ddl}ddl}t}|jdrddddS |j}|dk(re|j |j ddg|j|j |jt|jdyttd 5}|jt|ddddd d|d S#YQxYw#1swYxYw#t$r}dd |ddcYd}~Sd}~wwxYw)NrrunningFu日跑已在运行中T)rImessager[bashz/root/daily_incremental.sh)stdoutstderrrMu日跑已启动)rIr\r[pidu启动失败: )r- subprocessget_daily_run_statusr,forksetsidrunDEVNULLremove RUN_PID_FILE_exitr2writestrr)r-rastatusr`r;rs rrun_daily_crawlrms ! #F zz) -DQUVV Ubggi !8 BIIK JNNF$@A * 2 2:;M;M O<( BHHQKlC( @Aqwws3x/@ @#0Ad[^_ _ D A @ U ~aS-AeTTUsTAC3=C C3&C36C'C3 C$"C3'C0,C33 D<D D Dcddl}ddl}|jjtr t t5}t |jj}ddd|jdd} |jddtdgd|jjdd }d ||d Sd diS#1swYcxYw#YxYw#ttf$r$|j tn#YnxYwYd diSwxYw)Nrr!tailrdz/var/log/crawler_daily.logr)rr_zutf-8r$r%T)r[r`log_tailr[F)r-rar.rrhr2rr3rkill check_output RUN_LOG_TAILrfdecodeOSError ValueErrorrg)r-rar;r`rps rrbrbs ww~~l# l# EqAFFHNN4D0E# E BGGCOH 2:22q/1MNj&8&8::@&QZ&:[ $CXF F u  F E D$ <( D u  sXC(C C>:C8CC CCCD$C;:D;C?=DD)rrr7Z)__doc__rrr-rar4rTrr0rrrWrBr+r?rEr]rr-rRrUrXrhrsrmrb)inds0rrzs*)((9  % ('2 0 D ~F lC N2DE .UV .RS ,4GH /7KL *2CD .8RS 2:TU  4&00cD 0 :#JH"jZtjGTF F*  U(y1s" B