结构
N)
BeautifulSoupz/mnt/data/search.dbz~/data/search.dbzhttps://www.xjshs.gov.cnzwww.xjshs.gov.cnz/api-ajax_list u 新疆生产建设兵团u 政府公告u 第五师双河市18_xxgki xxgkzY-m-d# )zinputtime DESCzdisplayorder DESCz is_top DESCc ddddddddd d
d d d d d d
d dt z dz g}t d| d}|j | t D ]\ }t |t
r |D ] }|j
dd| g 1||j
ddg G|j
dd| g ^ |j
ddg t j |ddd }|j d k7 rt d |j y t j |j }|j d g }|j d d }||fS # t j t j f$ r} t d!| Y d} ~ y d} ~ ww xY w)"u 调用AJAX API获取列表数据curlz-sS--connect-timeout10-m30z-XPOST-Hz>Content-Type: application/x-www-form-urlencoded; charset=UTF-8z X-Requested-With: XMLHttpRequestHUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36z6Accept: application/json, text/javascript, */*; q=0.01z Referer: z/xxgk-list-zbggm.html-z.htmlz--data-urlencodezajax_type[7][]=Nzajax_type[]=nullz ajax_type[]=z-dzis_ds=1Tr capture_outputtexttimeoutr u [WARN] curl返回码 )Nr datatotalu [WARN] API错误: )BASE_URLAPI_URLappendAJAX_TYPE_BASE
isinstancelistextend
subprocessrun
returncodeprintjsonloadsstdoutgetTimeoutExpiredJSONDecodeError)
pagecmdurlitemsubresultr itemsr es
%/root/gov_crawler/crawl_xjshs_zbgg.py fetch_apir4 ' s *D$ f N 0 X F kH$'>> C IQtfE
"CJJsO D dD !
J
./#0GHI
J
\ JJ*,>? @ JJ*l4&,AB CD JJi ! DtRP ! +F,=,=+>? @zz&--($!$e| % %t';'; <
$QC()s 5A D= 6AD= =#E8 E33E8c | j dd } ddddddd d
| g } t j |d d d
}|j dk7 ry|j S # t j
$ r Y yw xY w)u 获取详情页HTMLzhttp://zhttps://r z-sLr r
r r r r Tr r r N)replacer! r" r# r' r) )r- r, r0 s r3
fetch_pager7 N s~ ++i
,C*D$ X C
DtRP !}} $ $ s (A A A*)A*c F t | }|syt |d }d}|j d D ]# }|j dd j } n |sC|j d }|r0|j
d j d d j }|j d
d j d d j d d j d
d j }d}|j d D ]# }|j dd j } n |r|dd }|j d xs |j d }d}|rbg } |j D ]= }
t |
dd } | dk( r|
j d } | r|
j D ] }
t |
dd }|.t |
j }|s.| j | @|dk( r| j t |
`|dk( rf|
j
d }|s{| j | |
j
d }|s| j | | dk( r | j t |
| t |
j }|s-| j | @ | rdj | }t | dk r|rg }|j d D ]W }|j dd }|r-|j d s |j d r t |z n|}|j d | Y |j d D ]W }|j dd }|r-|j d s |j d r t |z n|}|j d | Y |j d D ] }|j d!d }d"|v r5|j# j% d# d$ j' d"d% d& nd}|d'v sU|j d s |j d r t |z n|}|j
d xs |j% d d& }|j d(| d)| |r*|sd*j | n|d*z d*j | z }|||fS )+u0 获取详情页并提取标题、日期、正文)NNNz html.parser zmeta[name=ArticleTitle]contenttitleT)stripu! - 第五师双河市人民政府· zmeta[name=PubDate]N
zdiv.xxgk_contentz
#xxgk_contentnameptablebrz
i zimg[src]srchttp/u [图片] z iframe[src]u [附件PDF] za[href]href.?r )
pdfdocdocxxlsxlsxjpgpnggifzipraru [附件: ]
)r7 r selectr( r< findget_textr6
select_onechildrengetattrstrr joinlen
startswithr lowersplitrsplit)
detail_urlhtmlsoupr; mtdate_str content_divr
paragraphschildrB inner_tablenodennamet2p_textlinksimgrF iframearI extfnames r3 fetch_detailr{ ` si j !D } -D
E
[[2
3 i$**,
IIg JJTJ*223VXZ[aacE
MM*b ) 1 1(B ? G GPR S [ [\dfh i o o qE H
[[-
. 55B'--/
CR= //"45Y 9YK
D
)) )E5&$/Ds{ #jj 1 %
6 'fd ; = #D 1A * 1 1! 4 "g - & - -c$i 8 "d] !% T !:B ! * 1 1" 5
6 #^^$^ 7F "))&1 !!#e*- J$$& %%a(; )< ;;z*D 4y3; %%j1 ,C''%$C3>>&1(+s(;hn LL9SE* + ,
"(( 7 /F**UB'C3>>&1(+s(;hn LL<u- . /
##I. :A55$DCF$;$**,$$S)!,33C;B?TVC^^v..2ooc.B8d?D
.E$**S/"2E yr$ 89 : +/499U#TD[499UCS5SD (D c dd l } | j d }|j dt t dt d |j }|j }t d| d g }d}t d
|d
z D ] }t | \ }}|t d | d #|dkD r|n|}|D ]X } |j | j d
d | j dd | j dd | j dd d Z t d| dt | d| d t dt | d| d t }
g } |D ]. } | d } | |
vs
|
j | | j | 0 t dt | d t d g }
t | d
D ] \ }} | d } | d
}| d }t d | d t | d |d d! d d"# t j" j% t' | \ }}}|sR|}|j) d$d j) d%d j) d&d j) d'd j+ }|s |r|d d( nd}|rt d)t | d* n t d+ |
j |||| | d, t d-t |
d.t, d t. j0 j3 t, sjt d/t, t d0 |
d d1 D ], }t d2|d d3|d
d d4 d3t |d5 d* . t d6t |
d7 y dd l}|j7 t, d89 }|j9 }d}|
D ]L } |j; d:|d
|d5 |d |d |d; t< t> t@ f |jB dkD r|d
z
}N |jG |dkD rt d=| d> d?t< z d@z }tI jJ dAdB|t, tM | gdCdCd!D }t dE|j" j+ |jN r't dF|jN d dG n t dH |jQ t dI tS dJ |
D } t dKt |
t dL| t dM| t dNt< t dO t dPdQ t dRdSd"dTdUd"dVdW t dQ |
d d1 D ]E }t |d
dXkD r|d
d dX n|d
} t |d dSd"| dUd"t |d5 dW G t |
d1kD rt dYt |
d t dQ y # tD $ r}t d<| Y d }~Md }~ww xY w)ZNr u% 第五师双河市-招标公告爬虫) descriptionz--pagesu 爬取页数(每页20条,默认))typedefaulthelpu 🌐 爬取API列表页 1-z...rL u [ERROR] 无法获取第u 页r; r9 r- inputtimer~ )r; r- dater~ u 第u 页: u 条 (共u 条)u
📋 共获取 u 条列表数据（总计u 条）u 🔍 去重后 u 条u
📄 爬取详情页...r z [rH rX )endr= r> r? r@ rA u ✅ u 字u ⚠️ 正文为空)r; r r: r-
source_urlu
💾 入库 u 条到 u [ERROR] 数据库不存在: u
📊 结果预览:r z z | ( r: u
...共 u 条 (未入库)< )r zINSERT OR IGNORE INTO gov_raw (title, content, publish_date, page_url, source_url, site_name, group_name, industry) VALUES (?,?,?,?,?,?,?,?)r u [ERROR] 入库失败: u
🔍 FTS同步 u 条...z
import sqlite3, sys
db = sys.argv[1]
conn = sqlite3.connect(db, timeout=60)
c = conn.cursor()
added = int(sys.argv[2])
c.execute("SELECT id, title, content FROM gov_raw ORDER BY id DESC LIMIT ?", (added,))
rows = c.fetchall()
site = 'u5 '
for rowid, title, content in rows:
c.execute("INSERT OR IGNORE INTO gov_search (rowid, title, site_name, summary) VALUES (?,?,?,?)",
(rowid, title, site, (content or '')[:500]))
conn.commit()
c.execute("SELECT COUNT(*) FROM gov_search")
print(f"FTS条数: {c.fetchone()[0]}")
conn.close()
python3z-cTr z FTS: z FTS stderr: d u+
⚠️ 无新增记录，跳过FTS同步u
📊 结果总览:c 3 , K | ] }|d r d yw)r: rL N ).0rs r3