Compare commits
45
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
7f7171ac14 | ||
|
|
0b2b4f01d8 | ||
|
|
37dcebfce4 | ||
|
|
d7a73d7e6c | ||
|
|
9a4a734c19 | ||
|
|
38d7ef147d | ||
|
|
ab1452d06b | ||
|
|
1c7882ce34 | ||
|
|
3cf3858f09 | ||
|
|
33e16c1a61 | ||
|
|
c84d968b02 | ||
|
|
f4417c2f4d | ||
|
|
03cac702a8 | ||
|
|
0499c82965 | ||
|
|
d117102088 | ||
|
|
1aaab764ba | ||
|
|
2ac7c9d5a1 | ||
|
|
66d5049151 | ||
|
|
f979bd6ffc | ||
|
|
8ea82fbec3 | ||
|
|
2862f68631 | ||
|
|
b8d77547de | ||
|
|
79019bfc40 | ||
|
|
a5bc27c3a4 | ||
|
|
83c41963cf | ||
|
|
3533e55801 | ||
|
|
a293f7b784 | ||
|
|
754976db52 | ||
|
|
c954aa2917 | ||
|
|
772a597a53 | ||
|
|
1f30474cf3 | ||
|
|
a1021bb70a | ||
|
|
e0412747f0 | ||
|
|
5f63b35602 | ||
|
|
3e29617cbe | ||
|
|
941b7eec32 | ||
|
|
71d5f2a7cd | ||
|
|
f6c373ea94 | ||
|
|
978f27a164 | ||
|
|
0bfa7a4bc1 | ||
|
|
25f68831fd | ||
|
|
42fafc495f | ||
|
|
d4b8688148 | ||
|
|
bd399dd007 | ||
|
|
ffc5b108c6 |
@@ -45,3 +45,7 @@ cron_raw.txt
|
||||
# 探针/临时构建产物(2026-10-14 t_3e98b65a 调查时发现 17MB 残留未清)
|
||||
/go/*-probe
|
||||
/go/rulefirst-probe
|
||||
|
||||
# 临时构建/验证工件(t_5f822020 清理:quantify/test_one 二进制 + verify.go 探针)
|
||||
/go/tmp/
|
||||
/verify.go
|
||||
|
||||
@@ -0,0 +1,147 @@
|
||||
# 【织忆·门控复核】PATTERN/PEER 影子 diff 量化 — 对照表与原始证据
|
||||
|
||||
卡: t_bd63b641 · 执行人: prof-dev · 2026-10-17 21:0x CST
|
||||
方法: 不碰生产 env/二进制/服务(AC 硬边界)。开态用「独立离线评测」——
|
||||
真实 PatternProvider/PeerAPI + 生产 graph.db(只读) + 实时 bge embedder(:8000),
|
||||
等价于活服务开态注入路径;关态取活服务(:7821, PID 812251)实测。
|
||||
|
||||
────────────────────────────────────────────────────────────
|
||||
## 结论
|
||||
关态 vs 开态 diff = **空(当前态)**,且开态即使修 bug 也**无净收益/伤主路径**。
|
||||
两个通道本卡判定 **「不开闸」**,如实回卡。理由:
|
||||
- PEER: 无 peers 表(0 行),开态物理零注入,无产出。
|
||||
- PATTERN: 当前 NULL-id bug 使 `ListPatterns` 恒报错 → 零注入;**即使修好 bug**,
|
||||
85 行全是「图聚类摘要节点」(无 data/tools 内容,只有聚类键名),对金标余弦
|
||||
0.675–0.715(≥0.6 阈值)会注入 ≤3 条**无召回价值的聚类键噪声/金标**,外加每
|
||||
query ~85 次 embed(冷 ~1.4s)拉高 P95 + 稀释 top-5 ⇒ 伤主路径、无净收益。
|
||||
- 原始阻塞理由(上游未落地)小怡已复核失效(t_3e98b65a/t_fcadee04 done),
|
||||
但「上游 done ≠ 该开」—— 实测证明开了也净负收益(比图谱路「接线已证、收益未证」
|
||||
更明确: PATTERN 有 NULL-id 读库 bug + 数据源本身不是行为 pattern)。
|
||||
|
||||
────────────────────────────────────────────────────────────
|
||||
## 一、门控现状(关态坐实,双源独立)
|
||||
- systemd unit 权威配置: `systemctl --user cat zhiyid | grep ZHIYI_` → 无
|
||||
ZHIYI_PATTERN_RECALL / 无 ZHIYI_PEER_RECALL
|
||||
- 运行进程 env: `/proc/812251/environ` → 无此二键(与 prof-cloud 异源复核一致)
|
||||
- 结论: 两通道 env 均「未设/≠1」⇒ server.go 启动期 `if os.Getenv(...)=="1"` 不成立
|
||||
⇒ `storage.PatternRecallProvider` / `storage.PeerRecallProvider` 均未注册 ⇒ 关态
|
||||
|
||||
## 二、影子 diff(关态 vs 开态,逐条对照)
|
||||
固定 query 集: 14 金标(golden-set.json)+ 69 条真实 content-recall(usage_recall 表)
|
||||
|
||||
### PATTERN 通道(ZHIYI_PATTERN_RECALL)
|
||||
开态注入结果: 当前 **0 条 / 14 金标**(`shadow-on-pattern-inject.json` = 空 `{}`)
|
||||
根因(不是「没命中」而是「读不出来」):
|
||||
- 生产 graph_nodes(type='pattern') 共 85 行,**id 列 85/85 全为 NULL**
|
||||
(`sqlite3: sum(id IS NULL)=85`)
|
||||
- `PatternProvider.Match` → `GraphPatternStore.ListPatterns` 读 `SELECT id,...`
|
||||
为 string,遇 NULL 恒报 `sql: Scan error: converting NULL to string`
|
||||
(影子测试实测抓到原文: `PATTERN_LISTERROR="sql: Scan error on column index 0,
|
||||
name \"id\": converting NULL to string is unsupported"`,`PATTERNS_LOADED=0`)
|
||||
- ⇒ `Match` 直接 return nil ⇒ **当前开态无任何 pattern 注入**
|
||||
|
||||
深层(修 bug 后会怎样 —— 已量化,证明「修了也不该开」):
|
||||
- 这 85 行 properties **0 行有 `data` 键、0 行有 `tools` 键**(只有 20 行有
|
||||
`description`),全是「图聚类摘要节点」(如 `hermes-main/entity 共 3867 节点…`),
|
||||
**不是 L2 行为 pattern**。`Match` 里 `p.Data` 恒空 → 回退 `p.Name`(聚类键,
|
||||
如 `topic:memory` / `pattern:default-main:entity`)。
|
||||
- 余弦探针(绕过 NULL-id 直接 embed 85 个 pattern 名 + 金标,bge-m3:8000 实测):
|
||||
5 条金标 maxcos = 0.6752 / 0.6891 / 0.6975 / 0.7038 / 0.7148,**全 ≥ 0.6 阈值
|
||||
⇒ 若修好 NULL-id,开态会对每条金标注入 ≤3 条聚类键条目**(`offset-cosine-probe.txt`)。
|
||||
- 注入内容 = 裸聚类键名(`topic:memory`、`pattern:learner-main:concept`),
|
||||
**对「织忆是什么」这类 query 无召回价值,只会稀释 top-5**。
|
||||
- 额外成本: 每 query 需 embed 85 个 pattern 名(冷调用 ~85×16.5ms≈1.4s,
|
||||
探针 430 embed/7.12s 实测)⇒ **P95 主路径显著上升**(OFF p95=3177ms → 开态
|
||||
叠加 ~1.4s)。
|
||||
- 判读: 0.6 阈值对 bge-m3 余弦分布偏低(无关文本基线 0.67+),开态注入的多是
|
||||
**假阳性噪声**。
|
||||
- ⇒ 修 bug 后开态 = 「伤主路径 + 无净收益」,仍 **不开**。
|
||||
|
||||
### PEER 通道(ZHIYI_PEER_RECALL)
|
||||
开态注入结果: **0 条 / 14 金标**
|
||||
根因: 生产 graph.db **无 peers 表**(`SELECT count(*) FROM sqlite_master WHERE
|
||||
name='peers'` = 0)。`PeerAPI.GetRecallResults` → `Store.List(200)` → 0 行 →
|
||||
return nil(peer.go:305-308 源码)。⇒ **开态也无任何 peer 注入**。
|
||||
|
||||
### 对照表
|
||||
| 维度 | 关态(活服务) | 开态(影子/离线) | diff |
|
||||
|---|---|---|---|
|
||||
| PATTERN 注入条数 | 0(未注册) | 当前 0(NULL-id bug);修 bug 后 ≤3 聚类键/金标 | **0(当前)/ 噪声(修后)** |
|
||||
| PEER 注入条数 | 0(未注册) | 0(无 peers 表) | **0** |
|
||||
| 召回结果 top5 id | 见 off-recall-14golden-live.json | 当前与关态逐条相同(无注入项可加) | **空(当前)** |
|
||||
| /eval/golden keyword_pass | 14/14 | =关态(当前注入 0 不动);修后注入聚类键不改关键词判据 | **0(当前)** |
|
||||
| /eval/golden recall@5 | 0.643 | =关态 | **0(当前)** |
|
||||
| 开态额外成本 | — | 修 bug 后每 query ~85 embed(冷 ~1.4s)+ top-5 稀释 | 伤主路径 |
|
||||
| 新增条目位置 | — | 追加在结果尾部(recall.go:901-932 append 到 deduped 之后) | — |
|
||||
|
||||
## 三、延迟 P50/P95(阈值从实测读,非拍脑袋)
|
||||
关态(活服务 :7821):
|
||||
- 69 真实 query(本卡 AC 要求的「≥50 query」集): **p50=1383.7ms, p95=3177.2ms**
|
||||
(`off-69-queries-lat-ms.json`,一致单次 run 采集)
|
||||
- 14 金标: 空载热态 p50=949.9/p95=2288.5ms;我在并发 69-query 压测后重跑
|
||||
p50=3936.9/p95=5255.5ms(`off-recall-14golden-live.json` = 并发态)——
|
||||
**绝对值随负载/缓存/8081 LLM 队列代理波动**(非本卡判据,不追服务性能)。
|
||||
开态:
|
||||
- **当前态(NULL-id bug 在)**: 注入 0 条,代码块 `if patternRecallEnabled() &&
|
||||
Provider!=nil` 即便走进去也只 `for _, pr := range nil` ⇒ **开态延迟 = 关态延迟
|
||||
(逐条相同,无新增 embed)**。
|
||||
- **修 bug 后态**: 每 query 额外 embed 85 个 pattern 名(探针实测 430 embed/7.12s
|
||||
≈16.5ms/embed,冷调用 ~1.4s),追加在结果尾部 ⇒ **P95 主路径上叠加 ~1.4s 冷成本
|
||||
(冷 ≈ 关态 p95 翻倍量级)** ⇒ 伤主路径(超 AC 判据「P95 涨幅>20%」)。
|
||||
|
||||
判读: 延迟上「开 vs 关」的**决策相关量 = 修 bug 后每次召回 +~1.4s 冷成本**;
|
||||
当前态(bug 在)开=关=0 差异。绝对基线随负载波动,不作为开闸判据。
|
||||
|
||||
## 四、AC#2: 开门是否需重启才注册 Provider?
|
||||
**答案: 是,必须重启。**
|
||||
- 源码证据: `go/internal/api/server.go:189-202` ——
|
||||
`if os.Getenv("ZHIYI_PATTERN_RECALL")=="1" { storage.PatternRecallProvider =
|
||||
patternProvider.Match }` / `if os.Getenv("ZHIYI_PEER_RECALL")=="1"
|
||||
{ storage.PeerRecallProvider = peerAPI.GetRecallResults }`,均在 `NewServer()`
|
||||
启动期执行;`storage.PatternRecallProvider` 是包级变量,仅在这两处赋值。
|
||||
改运行中 env(不重启)无法注册 Provider。
|
||||
- 实测证据: PID 812251 运行二进制 mtime=2026-10-17 01:18,其 /proc environ 无
|
||||
此二键,当前 Provider 未注册(关态)。要开态只能: 在 systemd unit 加
|
||||
`Environment=ZHIYI_*_RECALL=1` + `daemon-reload` + `restart zhiyid`。
|
||||
- 开闸成本 = 一次重启(低)。但本卡结论「不开」基于净收益为负(见二),
|
||||
重启成本本身不是拦路项——拦路的是「开了也伤主路径 + 无净收益」。
|
||||
|
||||
## 五、判据(按卡 AC)
|
||||
- 当前态 diff 为空(PATTERN NULL-id bug → 0 注入;PEER 无表 → 0 注入)
|
||||
⇒ 「开态无产出」判据满足 ⇒ **不开闸**。
|
||||
- 修 bug 后态: 注入聚类键噪声 + P95 主路径涨幅 >20% ⇒ 「有产出但伤主路径」判据
|
||||
也满足 ⇒ **仍不开**。两条判据各自独立支撑「不开」。
|
||||
- 主路径未退化(关态 /eval/golden kw 14/14、recall@5=0.643;开态当前态=关态)。
|
||||
|
||||
────────────────────────────────────────────────────────────
|
||||
## 原始 JSON(改前/改后,禁只贴结论)
|
||||
- 关态活服务 14 金标逐条: `off-recall-14golden-live.json`
|
||||
- 关态活服务 /eval/golden 汇总: `off-eval-golden-raw.json`
|
||||
- 关态 69 真实 query 延迟: `off-69-queries-lat-ms.json`
|
||||
- 开态影子 pattern 注入(当前态 = 空 `{}`): `shadow-on-pattern-inject.json`
|
||||
- 开态影子 peer 表存在性: 0(`SHADOW_PEER_TABLE_EXISTS=0`,见 shadow_diff_test.go)
|
||||
- 修 bug 后余弦探针(85 聚类键 vs 金标 maxcos): `offset-cosine-probe.txt`
|
||||
+ 复现 `TestShadowPattern_CosineIfIDFixed`(`PATTERN_ROWS_LOADABLE_BYPASS_ID=85`)
|
||||
|
||||
复跑命令:
|
||||
# 关态金标召回(活服务)
|
||||
cd <ws> && python3 eval_off.py
|
||||
# 关态 69 真实 query 延迟
|
||||
cd <ws> && python3 latency_off.py # 注: stdout 混进度行,JSON 在尾部
|
||||
# 开态影子注入 + peer 表 + 修 bug 后余弦探针(只读 graph.db + :8000 embedder,不改生产)
|
||||
cd /home/muc/src/memoryweave/go && go test ./internal/pattern/ \
|
||||
-run 'TestShadowPattern_Diff|TestShadowPeer_NoTable|TestShadowPattern_CosineIfIDFixed' \
|
||||
-v -count=1
|
||||
|
||||
## 给审计脚本的「已明确决策」标记(消音用)
|
||||
本卡对 PATTERN/PEER 门控的终态 = **已明确决策: 暂不开闸(证据: 开态零产出,见上)**。
|
||||
若审计脚本下次再报「门控到期/上游 done 但门关」,应据此识别为「已决策不开」而非
|
||||
「没人复核」。建议: 门控台账加 `decision` 字段(values: open/no-open/unknown),
|
||||
本卡结论落 `no-open` + 证据指针,脚本对 no-open 态静默。
|
||||
|
||||
────────────────────────────────────────────────────────────
|
||||
## 未决(需人拍板,本卡不越界执行)
|
||||
1. PATTERN NULL-id 读库 bug: `GraphPatternStore.ListPatterns` 扫 NULL id 报错。
|
||||
85 存量 pattern 行全是聚类摘要节点且 id=NULL——**这套数据本身可能就不该作为
|
||||
L2 行为 pattern 召回源**。属「修数据还是修代码」的决策,交 hub/ops,不在本卡。
|
||||
2. 开闸动作: 小怡终审 + 牧尘放行(本卡只出证据,AC 明令不开门)。
|
||||
@@ -0,0 +1 @@
|
||||
[-1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, 825.0, 952.6, 1123.0, 1376.9, 1383.7, 1544.4, 1583.8, 1592.2, 1635.1, 1680.3, 1685.1, 1691.0, 1726.0, 1754.9, 1794.7, 1851.8, 1874.3, 1878.2, 1901.0, 1909.9, 1924.8, 1937.8, 1954.0, 2037.1, 2053.7, 2058.2, 2154.6, 2174.1, 2287.7, 2355.2, 2718.2, 2789.6, 2791.9, 2832.9, 3109.5, 3222.3, 3507.6, 3534.9, 3906.4]
|
||||
@@ -0,0 +1 @@
|
||||
{"id":"20261017-205654","model":"golden-set","recall_at_5":0.6428571428571429,"precision_at_5":0.12857142857142856,"mean_reciprocal_rank":0.47619047619047616,"ndcg":0.5238095238095238,"query_details":[{"query":"织忆是什么","expected_ids":["mem_1791666702950691634"],"must_include_keywords":["织忆"],"hits":null,"misses":["mem_1791666702950691634"],"recall_at_5":0,"precision_at_5":0,"keyword_ok":true},{"query":"织忆的图谱扩展机制是什么","expected_ids":["mem_1791899478673190284"],"must_include_keywords":["图谱"],"hits":["mem_1791899478673190284"],"misses":null,"recall_at_5":1,"precision_at_5":0.2,"keyword_ok":true},{"query":"织忆如何通过 E1 提取和 MMR 实现多样去重","expected_ids":["mem_1791666890486125605"],"must_include_keywords":["MMR"],"hits":null,"misses":["mem_1791666890486125605"],"recall_at_5":0,"precision_at_5":0,"keyword_ok":true},{"query":"牧尘的偏好是什么","expected_ids":["mem_1790448247125520424"],"must_include_keywords":["牧尘"],"hits":["mem_1790448247125520424"],"misses":null,"recall_at_5":1,"precision_at_5":0.2,"keyword_ok":true},{"query":"牧尘喜欢什么样的工作方式","expected_ids":["mem_1790448247125520424"],"must_include_keywords":["牧尘"],"hits":["mem_1790448247125520424"],"misses":null,"recall_at_5":1,"precision_at_5":0.2,"keyword_ok":true},{"query":"如何根据牧尘的偏好调整记忆检索策略","expected_ids":["mem_1791747597032219499"],"must_include_keywords":["记忆"],"hits":["mem_1791747597032219499"],"misses":null,"recall_at_5":1,"precision_at_5":0.2,"keyword_ok":true},{"query":"当前项目的技术栈是什么","expected_ids":["mem_1790455676397796671"],"must_include_keywords":["技术栈"],"hits":["mem_1790455676397796671"],"misses":null,"recall_at_5":1,"precision_at_5":0.2,"keyword_ok":true},{"query":"织忆项目有哪些核心组件","expected_ids":["mem_1791906354849281414"],"must_include_keywords":["织忆"],"hits":null,"misses":["mem_1791906354849281414"],"recall_at_5":0,"precision_at_5":0,"keyword_ok":true},{"query":"织忆和其他记忆系统相比有什么架构优势","expected_ids":["mem_1791666702950691634"],"must_include_keywords":["织忆"],"hits":["mem_1791666702950691634"],"misses":null,"recall_at_5":1,"precision_at_5":0.2,"keyword_ok":true},{"query":"如何使用 recall 接口","expected_ids":["mem_1791796178328736793"],"must_include_keywords":["recall"],"hits":null,"misses":["mem_1791796178328736793"],"recall_at_5":0,"precision_at_5":0,"keyword_ok":true},{"query":"织忆有哪些管理工具","expected_ids":["mem_1791666702950691634"],"must_include_keywords":["织忆"],"hits":null,"misses":["mem_1791666702950691634"],"recall_at_5":0,"precision_at_5":0,"keyword_ok":true},{"query":"如何通过 API 扩展织忆功能","expected_ids":["mem_1791666188168772049"],"must_include_keywords":["织忆"],"hits":["mem_1791666188168772049"],"misses":null,"recall_at_5":1,"precision_at_5":0.2,"keyword_ok":true},{"query":"本地 llama 服务 8080 端口","expected_ids":["mem_1791423503359721152"],"must_include_keywords":["8080"],"hits":["mem_1791423503359721152"],"misses":null,"recall_at_5":1,"precision_at_5":0.2,"keyword_ok":true},{"query":"session-to-zhiyi.py 脚本名","expected_ids":["mem_1791688470278719760"],"must_include_keywords":["session-to-zhiyi"],"hits":["mem_1791688470278719760"],"misses":null,"recall_at_5":1,"precision_at_5":0.2,"keyword_ok":true}],"queries":14,"ran_at":"2026-10-17T20:56:54.195909811+08:00","keyword_pass":14,"keyword_fail":0,"keyword_total":14}
|
||||
@@ -0,0 +1,319 @@
|
||||
{
|
||||
"state": "OFF",
|
||||
"results": [
|
||||
{
|
||||
"i": 0,
|
||||
"q": "织忆是什么",
|
||||
"count": 5,
|
||||
"top5_ids": [
|
||||
"mem_1791899478673190284",
|
||||
"mem_1792154287491185572",
|
||||
"mem_1791912043093092663",
|
||||
"mem_1792154287177190000",
|
||||
"mem_1792154288142808071"
|
||||
],
|
||||
"top5_cats": [
|
||||
"distilled",
|
||||
"distilled",
|
||||
"distilled",
|
||||
"distilled",
|
||||
"distilled"
|
||||
],
|
||||
"kw_ok": true,
|
||||
"lat_ms": 3936.9
|
||||
},
|
||||
{
|
||||
"i": 1,
|
||||
"q": "织忆的图谱扩展机制是什么",
|
||||
"count": 5,
|
||||
"top5_ids": [
|
||||
"mem_1791899478673190284",
|
||||
"mem_1791725847707296273",
|
||||
"mem_1792154287491185572",
|
||||
"mem_1791770032575826407",
|
||||
"mem_1792088552819764989"
|
||||
],
|
||||
"top5_cats": [
|
||||
"distilled",
|
||||
"distilled",
|
||||
"distilled",
|
||||
"distilled",
|
||||
"distilled"
|
||||
],
|
||||
"kw_ok": true,
|
||||
"lat_ms": 4225.2
|
||||
},
|
||||
{
|
||||
"i": 2,
|
||||
"q": "织忆如何通过 E1 提取和 MMR 实现多样去重",
|
||||
"count": 5,
|
||||
"top5_ids": [
|
||||
"mem_1792088552818081545",
|
||||
"mem_1791666890486125605",
|
||||
"mem_1791520496931190142",
|
||||
"mem_1791422985447662675",
|
||||
"mem_1791498226924653975"
|
||||
],
|
||||
"top5_cats": [
|
||||
"distilled",
|
||||
"distilled",
|
||||
"distilled",
|
||||
"distilled",
|
||||
"distilled"
|
||||
],
|
||||
"kw_ok": true,
|
||||
"lat_ms": 3643.6
|
||||
},
|
||||
{
|
||||
"i": 3,
|
||||
"q": "牧尘的偏好是什么",
|
||||
"count": 5,
|
||||
"top5_ids": [
|
||||
"mem_1791556229473565240",
|
||||
"mem_1791376118421773238",
|
||||
"mem_1790448247125520424",
|
||||
"mem_1792088552818232071",
|
||||
"mem_1792086639900350664"
|
||||
],
|
||||
"top5_cats": [
|
||||
"distilled",
|
||||
"distilled",
|
||||
"distilled",
|
||||
"distilled",
|
||||
"distilled"
|
||||
],
|
||||
"kw_ok": true,
|
||||
"lat_ms": 3367.5
|
||||
},
|
||||
{
|
||||
"i": 4,
|
||||
"q": "牧尘喜欢什么样的工作方式",
|
||||
"count": 5,
|
||||
"top5_ids": [
|
||||
"mem_1790448247125520424",
|
||||
"mem_1792086639900350664",
|
||||
"mem_1791482493617981202",
|
||||
"mem_1789063252280657787",
|
||||
"mem_1792086640133724836"
|
||||
],
|
||||
"top5_cats": [
|
||||
"distilled",
|
||||
"distilled",
|
||||
"distilled",
|
||||
"distilled",
|
||||
"distilled"
|
||||
],
|
||||
"kw_ok": true,
|
||||
"lat_ms": 3511.3
|
||||
},
|
||||
{
|
||||
"i": 5,
|
||||
"q": "如何根据牧尘的偏好调整记忆检索策略",
|
||||
"count": 5,
|
||||
"top5_ids": [
|
||||
"mem_1792091916993248722",
|
||||
"mem_1791747597032219499",
|
||||
"mem_1792088552824810016",
|
||||
"mem_1792153987878359340",
|
||||
"mem_1790137162828579293"
|
||||
],
|
||||
"top5_cats": [
|
||||
"distilled",
|
||||
"distilled",
|
||||
"distilled",
|
||||
"distilled",
|
||||
"distilled"
|
||||
],
|
||||
"kw_ok": true,
|
||||
"lat_ms": 3544.6
|
||||
},
|
||||
{
|
||||
"i": 6,
|
||||
"q": "当前项目的技术栈是什么",
|
||||
"count": 5,
|
||||
"top5_ids": [
|
||||
"mem_1792088552817810427",
|
||||
"mem_1790455676397796671",
|
||||
"mem_1791665641471849057",
|
||||
"mem_1790317738849661043",
|
||||
"mem_1790425023223201421"
|
||||
],
|
||||
"top5_cats": [
|
||||
"distilled",
|
||||
"distilled",
|
||||
"distilled",
|
||||
"distilled",
|
||||
"distilled"
|
||||
],
|
||||
"kw_ok": true,
|
||||
"lat_ms": 4874.5
|
||||
},
|
||||
{
|
||||
"i": 7,
|
||||
"q": "织忆项目有哪些核心组件",
|
||||
"count": 5,
|
||||
"top5_ids": [
|
||||
"mem_1792154287491185572",
|
||||
"mem_1792154287177190000",
|
||||
"mem_1791770032575826407",
|
||||
"mem_1792154288142808071",
|
||||
"mem_1791666702950691634"
|
||||
],
|
||||
"top5_cats": [
|
||||
"distilled",
|
||||
"distilled",
|
||||
"distilled",
|
||||
"distilled",
|
||||
"distilled"
|
||||
],
|
||||
"kw_ok": true,
|
||||
"lat_ms": 3811.5
|
||||
},
|
||||
{
|
||||
"i": 8,
|
||||
"q": "织忆和其他记忆系统相比有什么架构优势",
|
||||
"count": 5,
|
||||
"top5_ids": [
|
||||
"mem_1791899478077693848",
|
||||
"mem_1791666702950691634",
|
||||
"mem_1792088552819764989",
|
||||
"mem_1791770032575826407",
|
||||
"mem_1792154287491185572"
|
||||
],
|
||||
"top5_cats": [
|
||||
"distilled",
|
||||
"distilled",
|
||||
"distilled",
|
||||
"distilled",
|
||||
"distilled"
|
||||
],
|
||||
"kw_ok": true,
|
||||
"lat_ms": 3254.5
|
||||
},
|
||||
{
|
||||
"i": 9,
|
||||
"q": "如何使用 recall 接口",
|
||||
"count": 5,
|
||||
"top5_ids": [
|
||||
"mem_1791482321288100863",
|
||||
"mem_1792153988391999660",
|
||||
"ep_1785686577666177306",
|
||||
"ep_1786452727331645773",
|
||||
"mem_1788717420755520289"
|
||||
],
|
||||
"top5_cats": [
|
||||
"distilled",
|
||||
"distilled",
|
||||
"distilled",
|
||||
"distilled",
|
||||
"distilled"
|
||||
],
|
||||
"kw_ok": true,
|
||||
"lat_ms": 4110.9
|
||||
},
|
||||
{
|
||||
"i": 10,
|
||||
"q": "织忆有哪些管理工具",
|
||||
"count": 5,
|
||||
"top5_ids": [
|
||||
"mem_1792154287491185572",
|
||||
"mem_1791666855166846631",
|
||||
"mem_1792144557208382006",
|
||||
"mem_1792116217448228684",
|
||||
"mem_1792154288142808071"
|
||||
],
|
||||
"top5_cats": [
|
||||
"distilled",
|
||||
"distilled",
|
||||
"distilled",
|
||||
"distilled",
|
||||
"distilled"
|
||||
],
|
||||
"kw_ok": true,
|
||||
"lat_ms": 5963.1
|
||||
},
|
||||
{
|
||||
"i": 11,
|
||||
"q": "如何通过 API 扩展织忆功能",
|
||||
"count": 5,
|
||||
"top5_ids": [
|
||||
"mem_1791666188168772049",
|
||||
"mem_1791818814047111008",
|
||||
"ep_1785695216419714474",
|
||||
"ep_1785702789193047009",
|
||||
"ep_1786336919812827426"
|
||||
],
|
||||
"top5_cats": [
|
||||
"distilled",
|
||||
"distilled",
|
||||
"distilled",
|
||||
"distilled",
|
||||
"distilled"
|
||||
],
|
||||
"kw_ok": true,
|
||||
"lat_ms": 4222.9
|
||||
},
|
||||
{
|
||||
"i": 12,
|
||||
"q": "本地 llama 服务 8080 端口",
|
||||
"count": 5,
|
||||
"top5_ids": [
|
||||
"mem_1791423503359721152",
|
||||
"mem_1791440673318711992",
|
||||
"mem_1788594533802527918",
|
||||
"mem_1788620335517621905",
|
||||
"mem_1788700944516696452"
|
||||
],
|
||||
"top5_cats": [
|
||||
"distilled",
|
||||
"distilled",
|
||||
"distilled",
|
||||
"distilled",
|
||||
"distilled"
|
||||
],
|
||||
"kw_ok": false,
|
||||
"lat_ms": 3803.5
|
||||
},
|
||||
{
|
||||
"i": 13,
|
||||
"q": "session-to-zhiyi.py 脚本名",
|
||||
"count": 5,
|
||||
"top5_ids": [
|
||||
"mem_1791688470278719760",
|
||||
"mem_1789152643785440562",
|
||||
"mem_1790154887033667954",
|
||||
"mem_1790001291060572107",
|
||||
"mem_1789896720973833628"
|
||||
],
|
||||
"top5_cats": [
|
||||
"distilled",
|
||||
"distilled",
|
||||
"distilled",
|
||||
"distilled",
|
||||
"distilled"
|
||||
],
|
||||
"kw_ok": true,
|
||||
"lat_ms": 4078.1
|
||||
}
|
||||
],
|
||||
"kw_pass": 13,
|
||||
"kw_total": 14,
|
||||
"p50_ms": 3936.9,
|
||||
"p95_ms": 5255.5,
|
||||
"all_lat_ms": [
|
||||
3254.5,
|
||||
3367.5,
|
||||
3511.3,
|
||||
3544.6,
|
||||
3643.6,
|
||||
3803.5,
|
||||
3811.5,
|
||||
3936.9,
|
||||
4078.1,
|
||||
4110.9,
|
||||
4222.9,
|
||||
4225.2,
|
||||
4874.5,
|
||||
5963.1
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
shadow_diff_test.go:200: PATTERN_ROWS_LOADABLE_BYPASS_ID=85
|
||||
shadow_diff_test.go:224: COSINE_PROBE "织忆是什么" maxcos=0.7038 hit=true (thr=0.6) best=topic:memory
|
||||
shadow_diff_test.go:224: COSINE_PROBE "牧尘的偏好是什么" maxcos=0.6975 hit=true (thr=0.6) best=pattern:default-main:entity
|
||||
shadow_diff_test.go:224: COSINE_PROBE "如何使用 recall 接口" maxcos=0.6891 hit=true (thr=0.6) best=topic:replying
|
||||
shadow_diff_test.go:224: COSINE_PROBE "本地 llama 服务 8080 端口" maxcos=0.6752 hit=true (thr=0.6) best=pattern:test-llm:entity
|
||||
shadow_diff_test.go:224: COSINE_PROBE "织忆项目有哪些核心组件" maxcos=0.7148 hit=true (thr=0.6) best=pattern:learner-main:concept
|
||||
@@ -0,0 +1,229 @@
|
||||
package pattern
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"testing"
|
||||
|
||||
_ "modernc.org/sqlite"
|
||||
|
||||
"github.com/xiaoxue/memoryweave/internal/models"
|
||||
"github.com/xiaoxue/memoryweave/internal/storage"
|
||||
)
|
||||
|
||||
// ── t_bd63b641 影-复:PATTERN/PEER 开门量具(不碰生产;只读 graph.db + 实时 embedder)──
|
||||
// 量化「开 ZHIYI_PATTERN_RECALL / ZHIYI_PEER_RECALL」对 14 金标会注入什么。
|
||||
// Provider 装配在 server.go 启动期由 env 决定 ⇒ 不重启无法在活服务上测开态;
|
||||
// 故本测试直接构造真实 Provider + 生产 graph.db(只读)+ 实时 bge embedder,
|
||||
// 跑出「开态会注入的条目」,与关态(=主召回原样)对比。
|
||||
//
|
||||
// 本测试只读、不落盘、不改库(AC:不碰生产二进制/不重启)。
|
||||
|
||||
type logRecord struct {
|
||||
ID string
|
||||
Category string
|
||||
Content string
|
||||
}
|
||||
|
||||
// runShadowWithDiag 返回:注入结果 + 诊断(pattern 装载数 / query 是否编码成功 / 各 query 最高余弦)。
|
||||
// ⚠️ 关键发现:生产 graph_nodes(type='pattern') 的 id 全为 NULL(85/85),ListPatterns 读 id 为 string
|
||||
// ⇒ 恒报 sql: Scan error (converting NULL to string),Match 直接 return nil。
|
||||
// 故 runShadow 不把 ListPatterns 错误当失败 —— 它本身就是「开态也无法注入」的根因证据。
|
||||
func runShadowWithDiag() (map[string][]logRecord, int, string, map[string]float64) {
|
||||
endpoint := os.Getenv("VLLM_ENDPOINT")
|
||||
if endpoint == "" {
|
||||
endpoint = "http://localhost:8000/v1/embeddings"
|
||||
}
|
||||
emb := storage.NewEmbedder(endpoint)
|
||||
store := NewGraphPatternStore("/var/lib/memoryweave/graph.db")
|
||||
pp := NewPatternProvider(store, NewEmbedderProvider(emb))
|
||||
|
||||
// 装载 pattern 数(防「store 空 → 注入恒 0」的假阴性);记录 ListPatterns 错误原文
|
||||
pats, _, lerr := store.ListPatterns(1000)
|
||||
loaded := 0
|
||||
listErr := ""
|
||||
if lerr != nil {
|
||||
listErr = lerr.Error() // 根因:id 全 NULL → Scan error
|
||||
} else {
|
||||
loaded = len(pats)
|
||||
}
|
||||
|
||||
golden := []string{
|
||||
"织忆是什么", "织忆的图谱扩展机制是什么",
|
||||
"织忆如何通过 E1 提取和 MMR 实现多样去重", "牧尘的偏好是什么",
|
||||
"牧尘喜欢什么样的工作方式", "如何根据牧尘的偏好调整记忆检索策略",
|
||||
"当前项目的技术栈是什么", "织忆项目有哪些核心组件",
|
||||
"织忆和其他记忆系统相比有什么架构优势", "如何使用 recall 接口",
|
||||
"织忆有哪些管理工具", "如何通过 API 扩展织忆功能",
|
||||
"本地 llama 服务 8080 端口", "session-to-zhiyi.py 脚本名",
|
||||
}
|
||||
out := make(map[string][]logRecord)
|
||||
maxCos := make(map[string]float64)
|
||||
|
||||
// 预编码 14 query(若编码失败会记 0,便于区分「没命中」vs「编码挂了」)
|
||||
for _, q := range golden {
|
||||
qv, eerr := emb.EncodeSingle(q)
|
||||
if eerr != nil || len(qv) == 0 {
|
||||
maxCos[q] = -1 // 编码失败标记
|
||||
continue
|
||||
}
|
||||
// 逐个 pattern 算余弦,取最高(复现 Provider.Match 内部逻辑)
|
||||
best := -1.0
|
||||
for _, p := range pats {
|
||||
rep := p.Data
|
||||
if rep == "" {
|
||||
rep = p.Name
|
||||
}
|
||||
text := rep
|
||||
for _, tl := range p.Tools {
|
||||
if tl != "" {
|
||||
text += " " + tl
|
||||
}
|
||||
}
|
||||
pv, perr := emb.EncodeSingle(text)
|
||||
if perr != nil || len(pv) == 0 {
|
||||
continue
|
||||
}
|
||||
s := cosineF32(qv, pv)
|
||||
if s > best {
|
||||
best = s
|
||||
}
|
||||
}
|
||||
maxCos[q] = best
|
||||
}
|
||||
|
||||
// 真注入(走 Provider.Match,同 server 路径)
|
||||
for _, g := range golden {
|
||||
pat := pp.Match(g, "hermes-main", 3) // server.go:902 topK=3
|
||||
for _, r := range pat {
|
||||
out[g] = append(out[g], logRecord{ID: r.ID, Category: r.Category, Content: r.Content})
|
||||
}
|
||||
}
|
||||
return out, loaded, listErr, maxCos
|
||||
}
|
||||
|
||||
// TestShadowPattern_Diff —— 开态 pattern 注入量化(只读生产库)
|
||||
func TestShadowPattern_Diff(t *testing.T) {
|
||||
out, loaded, listErr, maxCos := runShadowWithDiag()
|
||||
t.Logf("PATTERNS_LOADED=%d (防假阴性:store 有货才会注入)", loaded)
|
||||
if listErr != "" {
|
||||
t.Logf("PATTERN_LISTERROR=%q — 根因:pattern 行 id 全 NULL ⇒ Provider.Match 恒 return nil", listErr)
|
||||
}
|
||||
|
||||
queries := []string{
|
||||
"织忆是什么", "织忆的图谱扩展机制是什么",
|
||||
"织忆如何通过 E1 提取和 MMR 实现多样去重", "牧尘的偏好是什么",
|
||||
"牧尘喜欢什么样的工作方式", "如何根据牧尘的偏好调整记忆检索策略",
|
||||
"当前项目的技术栈是什么", "织忆项目有哪些核心组件",
|
||||
"织忆和其他记忆系统相比有什么架构优势", "如何使用 recall 接口",
|
||||
"织忆有哪些管理工具", "如何通过 API 扩展织忆功能",
|
||||
"本地 llama 服务 8080 端口", "session-to-zhiyi.py 脚本名",
|
||||
}
|
||||
totalInjected := 0
|
||||
for _, q := range queries {
|
||||
mc := maxCos[q]
|
||||
if mc < 0 {
|
||||
t.Logf("[EMB-FAIL] %q → 编码失败(不计命中)", q)
|
||||
}
|
||||
if n := len(out[q]); n > 0 {
|
||||
totalInjected += n
|
||||
t.Logf("[INJECT] %q → %d pattern (maxcos=%.3f)", q, n, mc)
|
||||
for _, r := range out[q] {
|
||||
t.Logf(" id=%s cat=%s content=%s", r.ID, r.Category, trunc(r.Content, 60))
|
||||
}
|
||||
} else {
|
||||
t.Logf("[NO-INJECT] %q → 0 pattern (maxcos=%.3f, 阈值=0.6)", q, mc)
|
||||
}
|
||||
}
|
||||
b, _ := json.MarshalIndent(out, "", " ")
|
||||
t.Logf("SHADOW_PATTERN_TOTAL_INJECTED=%d", totalInjected)
|
||||
os.WriteFile("/tmp/shadow-pattern-inject.json", b, 0644)
|
||||
t.Logf("raw json → /tmp/shadow-pattern-inject.json (%d bytes)", len(b))
|
||||
}
|
||||
|
||||
func trunc(s string, n int) string {
|
||||
if len(s) <= n {
|
||||
return s
|
||||
}
|
||||
return s[:n] + "…"
|
||||
}
|
||||
|
||||
// ── peer 侧:生产 graph.db 无 peers 表 ⇒ 开态无数据可注入 ──
|
||||
func TestShadowPeer_NoTable(t *testing.T) {
|
||||
db, err := peerOpenReadOnly("/var/lib/memoryweave/graph.db")
|
||||
if err != nil {
|
||||
t.Fatalf("open graph.db: %v", err)
|
||||
}
|
||||
defer db.Close()
|
||||
var cnt int
|
||||
err = db.QueryRow("SELECT count(*) FROM sqlite_master WHERE type='table' AND name='peers'").Scan(&cnt)
|
||||
if err != nil {
|
||||
t.Fatalf("query peers table existence: %v", err)
|
||||
}
|
||||
t.Logf("SHADOW_PEER_TABLE_EXISTS=%d (0 ⇒ 开态 peer 无数据可注入)", cnt)
|
||||
}
|
||||
|
||||
// peerOpenReadOnly 只读开 sqlite(WAL 兼容),避免 peer.Store 的写路径。
|
||||
func peerOpenReadOnly(path string) (*sql.DB, error) {
|
||||
return sql.Open("sqlite", "file:"+path+"?mode=ro&_pragma=busy_timeout(5000)")
|
||||
}
|
||||
|
||||
// ── 深度根因:即使修好 NULL-id,85 行 pattern 无 data/tools 键(只有 description),
|
||||
// Match 只能 embed 命名键(如 "pattern:hermes-main:entity")⇒ 对金标余弦 < 0.6 阈值。
|
||||
// 本测试量化「修好 NULL-id 后开态到底能注入几类金标」—— 证明开态收益仍≈0。
|
||||
func TestShadowPattern_CosineIfIDFixed(t *testing.T) {
|
||||
endpoint := os.Getenv("VLLM_ENDPOINT")
|
||||
if endpoint == "" {
|
||||
endpoint = "http://localhost:8000/v1/embeddings"
|
||||
}
|
||||
emb := storage.NewEmbedder(endpoint)
|
||||
// 绕过 NULL-id 的 ListPatterns:只读 name + properties(非 NULL 列)
|
||||
db, err := peerOpenReadOnly("/var/lib/memoryweave/graph.db")
|
||||
if err != nil {
|
||||
t.Fatalf("open: %v", err)
|
||||
}
|
||||
defer db.Close()
|
||||
rows, _ := db.Query(`SELECT name, properties FROM graph_nodes WHERE type='pattern'`)
|
||||
type patText struct{ name, text string }
|
||||
var pats []patText
|
||||
for rows.Next() {
|
||||
var name, props string
|
||||
if serr := rows.Scan(&name, &props); serr != nil {
|
||||
continue
|
||||
}
|
||||
// 复现 Match 的 text 构造:rep=data(无→空) 回退 name;无 tools
|
||||
text := name
|
||||
pats = append(pats, patText{name, text})
|
||||
}
|
||||
rows.Close()
|
||||
t.Logf("PATTERN_ROWS_LOADABLE_BYPASS_ID=%d", len(pats))
|
||||
if len(pats) == 0 {
|
||||
t.Skip("无 pattern 行可载(NULL-id 未修前本就 0 注入,已在 TestShadowPattern_Diff 覆盖)")
|
||||
}
|
||||
golden := []string{"织忆是什么", "牧尘的偏好是什么", "如何使用 recall 接口",
|
||||
"本地 llama 服务 8080 端口", "织忆项目有哪些核心组件"}
|
||||
for _, q := range golden {
|
||||
qv, qerr := emb.EncodeSingle(q)
|
||||
if qerr != nil || len(qv) == 0 {
|
||||
t.Logf("[EMBFAIL] %q → %v", q, qerr)
|
||||
continue
|
||||
}
|
||||
best, bestName := -1.0, ""
|
||||
for _, p := range pats {
|
||||
pv, perr := emb.EncodeSingle(p.text)
|
||||
if perr != nil || len(pv) == 0 {
|
||||
continue
|
||||
}
|
||||
s := cosineF32(qv, pv)
|
||||
if s > best {
|
||||
best, bestName = s, p.name
|
||||
}
|
||||
}
|
||||
hit := best >= 0.6
|
||||
t.Logf("COSINE_PROBE %q maxcos=%.4f hit=%v (thr=0.6) best=%s", q, best, hit, bestName)
|
||||
}
|
||||
}
|
||||
|
||||
// 防 unused 误报
|
||||
var _ = models.RecallResult{}
|
||||
@@ -0,0 +1 @@
|
||||
{}
|
||||
@@ -0,0 +1,30 @@
|
||||
# 织忆指标语义:counter / rate / mean(`/api/v1/metrics`)
|
||||
|
||||
> 2026-10-26 建立。起因:有效性审计把 `deprecated_per_day`(计数器)当成率指标,
|
||||
> 报出「0 样本却判 valid=None ⇒ 缺最小样本门」。计数器没有分母,「样本门」对它无定义
|
||||
> —— 属**探针模型错配**,不是指标缺陷。本文件是给下游(审计脚本 / 看板 / 人)的唯一口径。
|
||||
|
||||
## 三类指标,三种读法
|
||||
|
||||
| 类型 | 命名特征 | 自带字段 | 怎么用 | 例子 |
|
||||
|---|---|---|---|---|
|
||||
| **counter 计数器** | `*_per_day` / `*_total` / `*_count` | **无** `_samples` / `_valid` | 直接读绝对值,跨时间比大小;不要算比率、不要报「样本不足」 | `deprecated_per_day`、`deprecated_total`、`distill_success_count` |
|
||||
| **rate 比率** | `*_rate` | `_samples`(分母)+ `_valid`(1=可判 / 0=不可判 / -1=N/A) | `_valid != 1` 时**不得**当结论;`_samples` 小于门槛时只报「待积累」 | `auto_resolve_rate`、`recall_usefulness_rate` |
|
||||
| **mean 均值** | `avg_*` | `_samples` + `_valid` | 同上:无分母就不下判断 | `avg_distill_loss` |
|
||||
|
||||
**铁律**:`0` 永远有两种含义 —— 「今天确实没有」和「根本没在统计」。
|
||||
- 计数器必须配一个累计值来区分:`deprecated_per_day=0` + `deprecated_total=0` ⇒ 从未统计过;
|
||||
`per_day=0` + `total=137` ⇒ 统计活着,只是今天没发生。
|
||||
- 率指标必须配 `_valid`/`_samples`,分母退化时返回 `metricNA(-1)`,不给 1(给 1 会被读成满分)。
|
||||
|
||||
## 触发器阈值也用同一套语义
|
||||
|
||||
`internal/selfoptimize/executor.go` 的 `t8`(`TDecayCritical` → `archive_or_boost`):
|
||||
- 判据读的是**计数器** `deprecated_per_day`(条/天),阈值由 `t8.Condition` 承载,**默认 5 条/天**。
|
||||
- 2026-10-26 修正:原先 `Condition: 0.1` 但判据函数里硬编码 `> 5`,配置写了从不生效。
|
||||
现改为 `Condition: 5.0` + 判据读 `t.Condition`(行为不变,但配置真的能被改)。
|
||||
- 若要改成「率」语义,需要先有分母(当日活跃写入量),否则别动 —— 那会变成另一个探针错配。
|
||||
|
||||
## 已知遗留
|
||||
|
||||
- 审计脚本侧对计数器发「样本不足」告警的白名单修复,归 `t_44c07eca`(脚本侧统一处理,避免多处各改一遍)。
|
||||
@@ -249,7 +249,12 @@ func (cp *ConsolidationPipeline) mergeSimilar() (int, error) {
|
||||
// 保留较新的
|
||||
idI := strVal(mems[i]["id"])
|
||||
idJ := strVal(mems[j]["id"])
|
||||
_ = cp.ldb.SoftDelete(idJ, fmt.Sprintf("merged_into_%s", idI))
|
||||
// t_cf680fad:不再 `_ =` 静默吞错。MVCC 冲突耗尽时 SoftDelete 内部会
|
||||
// 把 (idJ, reason) 落 durable pending 队列(softdelete_pending.go),
|
||||
// 启动/每 tick drain 重放必落盘;这里显式记日志做观测(禁静默)。
|
||||
if err := cp.ldb.SoftDelete(idJ, fmt.Sprintf("merged_into_%s", idI)); err != nil {
|
||||
log.Printf("[consolidation] merge-softdelete FAILED: src=%s → target=%s err=%v (pending-queue 将重放)", idJ, idI, err)
|
||||
}
|
||||
merged++
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,163 @@
|
||||
// 织忆 MemoryWeave — 整合链路健康跟踪(t_10757bd5)
|
||||
//
|
||||
// 背景(2026-10-24 小怡体检实测):
|
||||
// - t_distill(15min) 与 t_consolidation 同时触发 consolPipe,
|
||||
// Rust 侧互斥整合槽被占用时 Go 立即降级 Go 启发式 ⇒ consolidate_busy ~55 次/日静默堆积
|
||||
// - /health 只看 sidecar TCP 连通性 ⇒ 整合链路持续失败也假绿
|
||||
//
|
||||
// 本文件提供:
|
||||
// 1. ConsolidationTracker — 记录连续失败次数 + 最近一次成功时间(供 /health 读)
|
||||
// 2. busyWaitRetry — Rust sidecar 报 busy 时等待 3s×3 次再降级(整合全程 ~126s)
|
||||
// 3. 引用 metrics 包新增的 5 个指标(AC5:失败路径带计数)
|
||||
package routes
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/xiaoxue/memoryweave/internal/consolidate"
|
||||
"github.com/xiaoxue/memoryweave/internal/metrics"
|
||||
)
|
||||
|
||||
// ─── ConsolidationTracker ────────────────────────────────────
|
||||
|
||||
var (
|
||||
consTracker = &ConsolidationTracker{}
|
||||
consTrackerOnce sync.Once
|
||||
)
|
||||
|
||||
// ConsolidationTracker 记录整合链路健康状态(内存态,重启归零;
|
||||
// 重启后健康端点读到 ConsecutiveFailures=0 ⇒ 正确,因为该链路确实刚起步)。
|
||||
type ConsolidationTracker struct {
|
||||
mu sync.Mutex
|
||||
LastSuccess time.Time
|
||||
LastAttempt time.Time
|
||||
startup time.Time
|
||||
ConsecutiveFailures int
|
||||
Skips int64 // t_10757bd5:需要"跳过"而非失败的次数(并发触发 / sidecar busy)
|
||||
lastSkip string // 最近一次跳过原因
|
||||
// lastError 最近一次失败原因(供 /health 暴露,便于人工诊断)
|
||||
lastError string
|
||||
}
|
||||
|
||||
// GetConsolTracker 返回全局 tracker 单例
|
||||
func GetConsolTracker() *ConsolidationTracker {
|
||||
if consTracker.startup.IsZero() {
|
||||
consTracker.mu.Lock()
|
||||
if consTracker.startup.IsZero() {
|
||||
consTracker.startup = time.Now()
|
||||
}
|
||||
consTracker.mu.Unlock()
|
||||
}
|
||||
return consTracker
|
||||
}
|
||||
|
||||
// RecordSuccess 整合成功(归零)
|
||||
func (t *ConsolidationTracker) RecordSuccess() {
|
||||
t.mu.Lock()
|
||||
defer t.mu.Unlock()
|
||||
t.LastAttempt = time.Now()
|
||||
t.LastSuccess = time.Now()
|
||||
t.ConsecutiveFailures = 0
|
||||
t.lastError = ""
|
||||
metrics.ConsolLastSuccessTime.SetToCurrentTime()
|
||||
metrics.ConsolConsecutiveFailures.Set(0)
|
||||
}
|
||||
|
||||
// RecordSkip 本次触发被**跳过**(不是失败)。
|
||||
//
|
||||
// t_10757bd5 关键语义修正(2026-10-24 22:45 实测发现):
|
||||
// `consolidate_busy` 的真实含义是「另一个整合正在跑」,不是链路坏。
|
||||
// 把它记成失败会让 /health 报 degraded —— 明明整合正在正常工作,却又是一次假红。
|
||||
// 只有真正的错误(sidecar 崩、DB 错、超时)才算 failure。
|
||||
func (t *ConsolidationTracker) RecordSkip(reason string) {
|
||||
t.mu.Lock()
|
||||
defer t.mu.Unlock()
|
||||
t.LastAttempt = time.Now()
|
||||
t.Skips++
|
||||
t.lastSkip = reason
|
||||
}
|
||||
|
||||
// RecordFailure 整合失败(+1)
|
||||
func (t *ConsolidationTracker) RecordFailure(err string) {
|
||||
t.mu.Lock()
|
||||
defer t.mu.Unlock()
|
||||
t.LastAttempt = time.Now()
|
||||
t.ConsecutiveFailures++
|
||||
t.lastError = err
|
||||
metrics.ConsolConsecutiveFailures.Set(float64(t.ConsecutiveFailures))
|
||||
}
|
||||
|
||||
// HealthStatus 返回健康端点应报的状态:
|
||||
// - 连续**真失败** ≥ 3 ⇒ "degraded"
|
||||
// - 进程启动已超 2 小时却从未成功过 ⇒ "degraded"(跳过不算成功,也不算失败)
|
||||
// - 上次成功距今 > 2 小时(15min 周期 × 8)⇒ "degraded"
|
||||
// - 否则 "ok"
|
||||
func (t *ConsolidationTracker) HealthStatus() (string, int, time.Duration, string) {
|
||||
t.mu.Lock()
|
||||
defer t.mu.Unlock()
|
||||
fails := t.ConsecutiveFailures
|
||||
var sinceSuccess time.Duration
|
||||
if t.LastSuccess.IsZero() {
|
||||
sinceSuccess = 0 // 从未成功
|
||||
} else {
|
||||
sinceSuccess = time.Since(t.LastSuccess)
|
||||
}
|
||||
status := "ok"
|
||||
if fails >= 3 {
|
||||
status = "degraded"
|
||||
}
|
||||
if t.LastSuccess.IsZero() {
|
||||
if !t.startup.IsZero() && time.Since(t.startup) > 2*time.Hour {
|
||||
status = "degraded"
|
||||
}
|
||||
} else if sinceSuccess > 2*time.Hour {
|
||||
status = "degraded"
|
||||
}
|
||||
return status, fails, sinceSuccess, t.lastError
|
||||
}
|
||||
|
||||
// Snapshot 供 /health 暴露完整字段(含跳过次数,便于区分「忙」与「坏」)
|
||||
func (t *ConsolidationTracker) Snapshot() (fails int, skips int64, lastSkip string, lastErr string) {
|
||||
t.mu.Lock()
|
||||
defer t.mu.Unlock()
|
||||
return t.ConsecutiveFailures, t.Skips, t.lastSkip, t.lastError
|
||||
}
|
||||
|
||||
// ─── busyWaitRetry ───────────────────────────────────────────
|
||||
|
||||
// busyWaitRetry 对 Rust sidecar consolidate_busy 做有限等待重试。
|
||||
//
|
||||
// 原理:Rust 整合全程约 126s(DBSCAN + 衰减校准),触发器每 15min 一次,
|
||||
// 两个触发器同秒到达时第二个撞 busy。等待 3s × 3 次(共 9s);
|
||||
// 9s 后仍 busy 说明 sidecar 真有长任务(或正在做 full 模式),
|
||||
// 降级 Go 启发式并记录 tracker(由调用方负责)。
|
||||
//
|
||||
// 返回语义:
|
||||
// - (report, false, nil) = 成功拿到报告
|
||||
// - (nil, true, nil) = 重试耗尽仍 busy(调用方应走降级 + RecordFailure)
|
||||
// - (nil, false, err) = 非 busy 错误(调用方直接返回)
|
||||
func busyWaitRetry(dataDir, sqlitePath, mode string) (report *consolidate.Result, stillBusy bool, err error) {
|
||||
const (
|
||||
maxRetries = 3
|
||||
waitDur = 3 * time.Second
|
||||
)
|
||||
for i := 0; i <= maxRetries; i++ {
|
||||
if i > 0 {
|
||||
metrics.ConsolidateBusyWaitTotal.Inc()
|
||||
time.Sleep(waitDur)
|
||||
}
|
||||
report, err = consolidate.Run(dataDir, sqlitePath, mode)
|
||||
if err == nil {
|
||||
return report, false, nil
|
||||
}
|
||||
if strings.Contains(err.Error(), "consolidate_busy") {
|
||||
continue // 仍 busy,重试
|
||||
}
|
||||
// 非 busy 错误直接返回
|
||||
return nil, false, err
|
||||
}
|
||||
// 重试耗尽仍 busy
|
||||
return nil, true, nil
|
||||
}
|
||||
@@ -0,0 +1,119 @@
|
||||
// 织忆 MemoryWeave — 召回响应契约辅助(P0-3,2026-10-26)
|
||||
//
|
||||
// 背景(CS249r Vol III Ch6 检索契约):响应只有 cache_hit/candidates_scanned/count/mode/graph_expanded,
|
||||
// 空结果分不清「不存在」与「扫了一半就超时」——这是 Air Canada 类静默幻觉的结构性入口。
|
||||
//
|
||||
// 本文件提供最小可用契约三件:
|
||||
// ① status/partial_reason:complete 的空结果才允许解读为「不存在」;
|
||||
// ② scope{namespace,snapshot,...}:说明这次答案基于哪个数据视图(本服务无索引版本号,
|
||||
// 故 snapshot = 观测时点 + 缓存状态,缓存命中代表 60s 内旧快照 —— 语义诚实,不假装有版本号);
|
||||
// ③ continuation:不透明翻页令牌(query 指纹 + ns 绑定,防止跨 query 串页)。
|
||||
package routes
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/base64"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
"github.com/xiaoxue/memoryweave/internal/models"
|
||||
"github.com/xiaoxue/memoryweave/internal/storage"
|
||||
)
|
||||
|
||||
// maxContinuationOffset 硬上限:防止把 topK 放大到爆内存/延迟。
|
||||
const maxContinuationOffset = 500
|
||||
|
||||
type pageToken struct {
|
||||
V int `json:"v"`
|
||||
Q string `json:"q"` // query 指纹(sha256 前 8 hex)
|
||||
NS string `json:"ns"` // 作用域绑定
|
||||
Off int `json:"o"` // 下一页起点
|
||||
Lim int `json:"l"` // 页大小
|
||||
// Prev 已返回过的 ID(去重锚点)。活索引下排序会随 fetchLimit 变化而轻微漂移,
|
||||
// 只靠 offset 会出现跨页重复;token 携带已返回 ID 才能机械保证「不重」。
|
||||
Prev []string `json:"p,omitempty"`
|
||||
}
|
||||
|
||||
// maxPrevIDs 令牌里最多携带的去重锚点数(3~5 页内足够;超长则丢最旧,令牌不会无限膨胀)。
|
||||
const maxPrevIDs = 100
|
||||
|
||||
// queryFingerprint 8 hex 的 query 指纹(足够防串页,不泄露正文)。
|
||||
func queryFingerprint(q string) string {
|
||||
sum := sha256.Sum256([]byte(q))
|
||||
return hex.EncodeToString(sum[:4])
|
||||
}
|
||||
|
||||
// encodeContinuation 生成下一页令牌;没有下一页时返回空串。
|
||||
func encodeContinuation(query, namespace string, offset, limit int, prev []string) string {
|
||||
if limit <= 0 {
|
||||
return ""
|
||||
}
|
||||
if len(prev) > maxPrevIDs {
|
||||
prev = prev[len(prev)-maxPrevIDs:]
|
||||
}
|
||||
tok := pageToken{V: 1, Q: queryFingerprint(query), NS: namespace, Off: offset, Lim: limit, Prev: prev}
|
||||
b, err := json.Marshal(tok)
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return base64.RawURLEncoding.EncodeToString(b)
|
||||
}
|
||||
|
||||
// decodeContinuation 解析令牌。错误信息面向调用方(可据此修请求)。
|
||||
func decodeContinuation(raw string) (pageToken, error) {
|
||||
var tok pageToken
|
||||
b, err := base64.RawURLEncoding.DecodeString(raw)
|
||||
if err != nil {
|
||||
return tok, errors.New("continuation 不是合法 base64url 令牌")
|
||||
}
|
||||
if err := json.Unmarshal(b, &tok); err != nil {
|
||||
return tok, errors.New("continuation 内容不是合法 JSON")
|
||||
}
|
||||
if tok.V != 1 {
|
||||
return tok, errors.New("continuation 版本不支持:" + strconv.Itoa(tok.V))
|
||||
}
|
||||
if tok.Off < 0 || tok.Lim <= 0 {
|
||||
return tok, errors.New("continuation 字段非法(offset/limit)")
|
||||
}
|
||||
if tok.Off > maxContinuationOffset {
|
||||
return tok, errors.New("continuation offset 超出上限 " + strconv.Itoa(maxContinuationOffset))
|
||||
}
|
||||
return tok, nil
|
||||
}
|
||||
|
||||
// buildScope 组装本次召回的数据视图声明。
|
||||
func buildScope(namespace string, topK int, mode string, cacheHit, truncated bool, started time.Time) map[string]interface{} {
|
||||
snap := "t" + strconv.FormatInt(started.UnixMilli(), 10)
|
||||
if cacheHit {
|
||||
snap += "-cache" // 缓存命中:读的是 60s 内旧快照,不是本次实时扫描
|
||||
} else {
|
||||
snap += "-scan"
|
||||
}
|
||||
if truncated {
|
||||
snap += "-truncated"
|
||||
}
|
||||
return map[string]interface{}{
|
||||
"namespace": namespace,
|
||||
"snapshot": snap,
|
||||
"cache_hit": cacheHit,
|
||||
"truncated": truncated,
|
||||
"top_k": topK,
|
||||
"mode": mode,
|
||||
}
|
||||
}
|
||||
|
||||
// traceItems 把最终返回集映射成 trace 的 item 归因(rank 从 1 起)。
|
||||
func traceItems(results []models.RecallResult) []storage.TraceItem {
|
||||
items := make([]storage.TraceItem, 0, len(results))
|
||||
for i, r := range results {
|
||||
ch := r.Channel
|
||||
if ch == "" {
|
||||
ch = "vector" // 老调用方/旧缓存条目无 channel ⇒ 主通道默认值
|
||||
}
|
||||
items = append(items, storage.TraceItem{Rank: i + 1, ID: r.ID, Channel: ch, Score: r.Score})
|
||||
}
|
||||
return items
|
||||
}
|
||||
@@ -0,0 +1,116 @@
|
||||
package routes
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/xiaoxue/memoryweave/internal/models"
|
||||
)
|
||||
|
||||
func TestContinuationRoundTrip(t *testing.T) {
|
||||
tok := encodeContinuation("织忆怎么召回", "hermes-main", 10, 5, []string{"mem_1", "mem_2"})
|
||||
if tok == "" {
|
||||
t.Fatal("有下一页时令牌不应为空")
|
||||
}
|
||||
got, err := decodeContinuation(tok)
|
||||
if err != nil {
|
||||
t.Fatalf("解码失败:%v", err)
|
||||
}
|
||||
if got.Off != 10 || got.Lim != 5 {
|
||||
t.Fatalf("令牌字段错误:%+v", got)
|
||||
}
|
||||
if got.Q != queryFingerprint("织忆怎么召回") {
|
||||
t.Fatalf("query 指纹不一致")
|
||||
}
|
||||
if got.NS != "hermes-main" {
|
||||
t.Fatalf("namespace 不一致:%s", got.NS)
|
||||
}
|
||||
if len(got.Prev) != 2 || got.Prev[0] != "mem_1" {
|
||||
t.Fatalf("去重锚点未随令牌传递:%v", got.Prev)
|
||||
}
|
||||
}
|
||||
|
||||
func TestContinuationRejectsBadToken(t *testing.T) {
|
||||
if _, err := decodeContinuation("not-base64!!"); err == nil {
|
||||
t.Fatal("非法 base64 应报错")
|
||||
}
|
||||
if _, err := decodeContinuation("eyJ2Ijo5fQ"); err == nil { // {"v":9}
|
||||
t.Fatal("未知版本应报错")
|
||||
}
|
||||
tok := encodeContinuation("q", "ns", 9999, 5, nil)
|
||||
if _, err := decodeContinuation(tok); err == nil {
|
||||
t.Fatal("offset 超上限应报错")
|
||||
}
|
||||
}
|
||||
|
||||
func TestQueryFingerprintStable(t *testing.T) {
|
||||
a := queryFingerprint("同样的问句")
|
||||
b := queryFingerprint("同样的问句")
|
||||
c := queryFingerprint("不同的问句")
|
||||
if a != b {
|
||||
t.Fatal("同 query 指纹应稳定")
|
||||
}
|
||||
if a == c {
|
||||
t.Fatal("不同 query 指纹应不同")
|
||||
}
|
||||
if len(a) != 8 {
|
||||
t.Fatalf("指纹应为 8 hex,实际 %d", len(a))
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildScope(t *testing.T) {
|
||||
started := time.UnixMilli(1700000000000)
|
||||
sc := buildScope("hermes-main", 5, "hybrid", false, false, started)
|
||||
for _, k := range []string{"namespace", "snapshot", "cache_hit", "truncated", "top_k", "mode"} {
|
||||
if _, ok := sc[k]; !ok {
|
||||
t.Fatalf("scope 缺字段 %s", k)
|
||||
}
|
||||
}
|
||||
snap := sc["snapshot"].(string)
|
||||
if snap != "t1700000000000-scan" {
|
||||
t.Fatalf("实时扫描 snapshot 形态错误:%s", snap)
|
||||
}
|
||||
cached := buildScope("ns", 5, "hybrid", true, true, started)["snapshot"].(string)
|
||||
if cached != "t1700000000000-cache-truncated" {
|
||||
t.Fatalf("缓存+截断 snapshot 形态错误:%s", cached)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTraceItems(t *testing.T) {
|
||||
rs := []models.RecallResult{
|
||||
{ID: "a", Channel: "vector"},
|
||||
{ID: "b"}, // 老调用方无 channel ⇒ 默认 vector
|
||||
{ID: "c", Channel: "graph"},
|
||||
}
|
||||
items := traceItems(rs)
|
||||
if len(items) != 3 || items[0].Rank != 1 || items[2].Rank != 3 {
|
||||
t.Fatalf("rank 编号错误:%+v", items)
|
||||
}
|
||||
if items[1].Channel != "vector" {
|
||||
t.Fatalf("缺省 channel 应回落 vector,实际 %s", items[1].Channel)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRecallResultContractFields 契约字段必须真的出现在 JSON 里(不是「加了结构体没接线」)。
|
||||
func TestRecallResultContractFields(t *testing.T) {
|
||||
r := models.RecallResult{
|
||||
ID: "mem_1", Channel: "fts", Stale: true, Version: 3,
|
||||
ValidFrom: "2026-10-26T10:00:00+08:00", ValidTo: "2026-10-26T11:00:00+08:00",
|
||||
Supersedes: []string{"mem_0"}, SupersededBy: []string{"mem_2"}, Superseded: true,
|
||||
}
|
||||
b, err := json.Marshal(r)
|
||||
if err != nil {
|
||||
t.Fatalf("marshal: %v", err)
|
||||
}
|
||||
var m map[string]interface{}
|
||||
if err := json.Unmarshal(b, &m); err != nil {
|
||||
t.Fatalf("unmarshal: %v", err)
|
||||
}
|
||||
for _, k := range []string{"channel", "stale", "generation", "valid_from", "valid_to",
|
||||
"supersedes", "superseded_by", "superseded"} {
|
||||
if _, ok := m[k]; !ok {
|
||||
t.Fatalf("响应缺字段 %s(JSON=%s)", k, string(b))
|
||||
}
|
||||
}
|
||||
}
|
||||
+343
-67
@@ -2,6 +2,7 @@
|
||||
package routes
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log"
|
||||
@@ -16,9 +17,9 @@ import (
|
||||
|
||||
"github.com/xiaoxue/memoryweave/internal/distill"
|
||||
"github.com/xiaoxue/memoryweave/internal/governance"
|
||||
"github.com/xiaoxue/memoryweave/internal/metanote"
|
||||
"github.com/xiaoxue/memoryweave/internal/metrics"
|
||||
"github.com/xiaoxue/memoryweave/internal/models"
|
||||
"github.com/xiaoxue/memoryweave/internal/metanote"
|
||||
"github.com/xiaoxue/memoryweave/internal/selfoptimize"
|
||||
"github.com/xiaoxue/memoryweave/internal/storage"
|
||||
)
|
||||
@@ -106,6 +107,8 @@ func (a *API) Commit(w http.ResponseWriter, r *http.Request) {
|
||||
// 判定与 Python 口径真相源同源(internal/metanote,parity/差分测试锁死);
|
||||
// 判 meta ⇒ 422 拒绝,不写 episode、不进蒸馏(卡面:把 meta 挡在咽喉点)。
|
||||
if skip, reason := metanote.ShouldSkip(req.Content); skip {
|
||||
// t_10757bd5 AC5:metanote 拒绝 ⇒ 计数器 +1(与日志 "[metanote] commit rejected" 行对齐)
|
||||
metrics.MetaNoteRejectTotal.Inc()
|
||||
log.Printf("[metanote] commit rejected (gate ON, meta-note): agent=%s ns=%s reason=%s content=%.60s",
|
||||
req.AgentID, req.Namespace, reason, req.Content)
|
||||
respondError(w, 422, "meta-note rejected: "+reason)
|
||||
@@ -506,13 +509,15 @@ func cosineSimilarity(a, b []float32) float64 {
|
||||
// 用途:区分「真无匹配」(results=[] 但候选数>0)与「依赖故障」(非 2xx)。
|
||||
func (a *API) Recall(w http.ResponseWriter, r *http.Request) {
|
||||
var req struct {
|
||||
Query string `json:"query"`
|
||||
Limit int `json:"limit"`
|
||||
TopK int `json:"top_k"` // 兼容插件 top_k 参数
|
||||
Namespace string `json:"namespace"`
|
||||
AgentID string `json:"agent_id"` // 用于推导默认 namespace
|
||||
Diversity float64 `json:"diversity"`
|
||||
Mode string `json:"mode"` // "hybrid"(default), "semantic", "keyword"
|
||||
Query string `json:"query"`
|
||||
Limit int `json:"limit"`
|
||||
TopK int `json:"top_k"` // 兼容插件 top_k 参数
|
||||
Namespace string `json:"namespace"`
|
||||
AgentID string `json:"agent_id"` // 用于推导默认 namespace
|
||||
Diversity float64 `json:"diversity"`
|
||||
Mode string `json:"mode"` // "hybrid"(default), "semantic", "keyword"
|
||||
DeadlineMS int `json:"deadline_ms"` // P0-3:本次召回的硬超时(毫秒),超时返回 partial
|
||||
Continuation string `json:"continuation"` // P0-3:上一页返回的不透明翻页令牌
|
||||
}
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
respondError(w, 400, "invalid body")
|
||||
@@ -545,8 +550,48 @@ func (a *API) Recall(w http.ResponseWriter, r *http.Request) {
|
||||
req.Limit = 10
|
||||
}
|
||||
// 2026-09-21 五修(t_24eceed0 返修):无 ns 且无 agent_id → namespace="" → 全库跨 ns 召回
|
||||
// 2026-10-26:区分「作用域来源」——只有 agent_id 派生的 ns 才允许兜底,
|
||||
// 显式传 namespace 的调用方不兜底(避免跨域泄漏)。
|
||||
nsDerivedFromAgent := req.Namespace == "" && req.AgentID != ""
|
||||
req.Namespace = resolveRecallNamespace(req.Namespace, req.AgentID, req.Query)
|
||||
|
||||
// ── P0-3 契约 + P0-1 追踪(2026-10-26)──────────────────────────────
|
||||
// 翻页令牌:绑定 query 指纹 + 作用域,跨 query 复用一律 400(防串页)。
|
||||
offset := 0
|
||||
var prevIDs []string
|
||||
if req.Continuation != "" {
|
||||
pt, perr := decodeContinuation(req.Continuation)
|
||||
if perr != nil {
|
||||
respondErrorSchema(w, 400, "bad_continuation", perr.Error(), 0)
|
||||
return
|
||||
}
|
||||
if pt.Q != queryFingerprint(req.Query) {
|
||||
respondErrorSchema(w, 400, "continuation_query_mismatch",
|
||||
"continuation 属于另一个 query,不能跨 query 复用", 0)
|
||||
return
|
||||
}
|
||||
if pt.NS != req.Namespace {
|
||||
respondErrorSchema(w, 400, "continuation_namespace_mismatch",
|
||||
"continuation 的 namespace 与本次请求不一致", 0)
|
||||
return
|
||||
}
|
||||
offset = pt.Off
|
||||
prevIDs = pt.Prev
|
||||
}
|
||||
// 取数上限 = 页尾(offset+limit);MMR 的 k 随之上抬,保证前缀稳定。
|
||||
// 多取 len(prevIDs):用于过滤跨页重复项,过滤后仍能凑满一页(否则「不重」要靠「漏」换)。
|
||||
fetchLimit := req.Limit + offset + len(prevIDs)
|
||||
if fetchLimit > maxContinuationOffset+req.Limit {
|
||||
fetchLimit = maxContinuationOffset + req.Limit
|
||||
}
|
||||
|
||||
ctx := r.Context()
|
||||
if req.DeadlineMS > 0 {
|
||||
var cancel context.CancelFunc
|
||||
ctx, cancel = context.WithTimeout(ctx, time.Duration(req.DeadlineMS)*time.Millisecond)
|
||||
defer cancel()
|
||||
}
|
||||
|
||||
// H4: Default diversity — balance relevance & diversity
|
||||
// 2026-10-14:改读 storage 的推送值(默认 0.3,与下方原硬编码一致 ⇒ 行为不变),
|
||||
// 让调参器能真正影响召回 —— 此前调参改了 Tuner.Diversity 但没人读(空转)。
|
||||
@@ -564,71 +609,78 @@ func (a *API) Recall(w http.ResponseWriter, r *http.Request) {
|
||||
// (LanceDB ANN 返回 + 去重 + 原始对话过滤之后、rerank/MMR 截断之前捕获)。
|
||||
// 此前误用 len(results)(最终产出数)作代理值——字段名说 "scanned"(扫描候选池)
|
||||
// 值却是"产出数",属"用文档补假名"。现改为 RecallWithMeta 返回的候选池真实大小。
|
||||
// keyword 路径未做 ANN 扫描 ⇒ 恒 0;graph 兜底路径 = 兜底返回数。
|
||||
// keyword 主路径未做 ANN 扫描 ⇒ 恒 0;graph 兜底路径 = 兜底返回数。
|
||||
annScanned := 0
|
||||
// t_3aa2a8c1:粗检索被 Rust 侧单次重读行数上限截断的标志(来源 RecallMeta.Truncated)。
|
||||
// graph 兜底 / keyword 空匹配兜底结果不经 IPC 粗检索 ⇒ 恒 false。
|
||||
truncated := false
|
||||
|
||||
// H5: Mode routing — hybrid / semantic / keyword
|
||||
if req.Mode == "" {
|
||||
req.Mode = "hybrid"
|
||||
}
|
||||
// P0-1:trace 在 mode 归一之后创建(记录真实 mode);所有路径共用同一条 trace。
|
||||
tr := storage.NewRecallTrace("", req.Query, req.Namespace, req.Mode)
|
||||
var lastMeta storage.RecallMeta
|
||||
switch req.Mode {
|
||||
case "keyword":
|
||||
// Pure keyword search: LanceDB + BM25 re-rank, or graph.db fallback
|
||||
// t_2d4e3b92: keyword 主路径依赖向量编码;编码失败(依赖不可用)⇒ 非 2xx,
|
||||
// 不再把错误伪装成 200 + 错误体。
|
||||
// t_1c12e000:keyword 路径同样收进 RecallWithMeta(Pipeline.Recall 内部
|
||||
// 走同一缓存键),只为把 cache_hit 带回响应——旧版用 Recall 丢掉了 meta。
|
||||
var kwMeta storage.RecallMeta
|
||||
kwResults, kwMeta, kwErr := a.Pipeline.RecallWithMeta(req.Query, req.Namespace, req.Limit, req.Diversity)
|
||||
// t_2d4e3b92: keyword 主路径依赖向量编码;编码失败(依赖不可用)⇒ 非 2xx。
|
||||
stKw := time.Now()
|
||||
kwResults, kwMeta, kwErr := a.Pipeline.RecallWithMetaTraced(ctx, req.Query, req.Namespace, fetchLimit, req.Diversity, nsDerivedFromAgent, tr)
|
||||
if kwErr != nil {
|
||||
// 主路径硬错误(向量编码/rerank 依赖不可用):即便兜底能补结果,也按 502 上报,
|
||||
// 调用方据此走失败路径而不是把依赖故障误判成"无匹配"
|
||||
tr.StageNote("keyword_path", stKw, 0, kwErr.Error())
|
||||
tr.Error = kwErr.Error()
|
||||
respondErrorSchema(w, 502, "recall_backend_error",
|
||||
"keyword recall failed: "+kwErr.Error(), 5)
|
||||
log.Printf("[recall:keyword] failed: %v", kwErr)
|
||||
return
|
||||
}
|
||||
lastMeta = kwMeta
|
||||
cacheHit = kwMeta.CacheHit
|
||||
truncated = kwMeta.Truncated
|
||||
if len(kwResults) == 0 {
|
||||
// 真无匹配(正常业务):成功 schema,results=[] + candidates_scanned 供区分
|
||||
fallbackResults := a.GraphStore.FallbackTextSearch(req.Query, req.Namespace, req.Limit)
|
||||
results = convertFallbackResults(fallbackResults)
|
||||
respond(w, 200, map[string]interface{}{
|
||||
"results": results,
|
||||
"count": len(results),
|
||||
"candidates_scanned": 0, // keyword 路径未做 ANN 扫描
|
||||
"mode": "keyword",
|
||||
"cache_hit": false, // graph 兜底结果不来自 search cache
|
||||
})
|
||||
return
|
||||
// 真无匹配(正常业务)或纯词面路径空:graph.db 兜底
|
||||
results = convertFallbackResults(a.GraphStore.FallbackTextSearch(req.Query, req.Namespace, fetchLimit))
|
||||
if len(results) > 0 {
|
||||
tr.StageNote("keyword_fallback", stKw, len(results), "主路径 0 命中 ⇒ graph.db 兜底")
|
||||
} else {
|
||||
tr.Stage("keyword_path", stKw, 0)
|
||||
}
|
||||
cacheHit = false // 兜底结果不来自 search cache
|
||||
} else {
|
||||
// Re-rank by BM25 only
|
||||
for i := range kwResults {
|
||||
kwResults[i].Score = storage.ComputeBM25Score(req.Query, kwResults[i].Content)
|
||||
}
|
||||
sort.Slice(kwResults, func(i, j int) bool { return kwResults[i].Score > kwResults[j].Score })
|
||||
if len(kwResults) > fetchLimit {
|
||||
kwResults = kwResults[:fetchLimit]
|
||||
}
|
||||
for i := range kwResults {
|
||||
if kwResults[i].Channel == "" {
|
||||
kwResults[i].Channel = "keyword"
|
||||
}
|
||||
}
|
||||
results = kwResults
|
||||
tr.Stage("keyword_path", stKw, len(results))
|
||||
}
|
||||
// Re-rank by BM25 only
|
||||
for i := range kwResults {
|
||||
kwResults[i].Score = storage.ComputeBM25Score(req.Query, kwResults[i].Content)
|
||||
}
|
||||
sort.Slice(kwResults, func(i, j int) bool { return kwResults[i].Score > kwResults[j].Score })
|
||||
if len(kwResults) > req.Limit {
|
||||
kwResults = kwResults[:req.Limit]
|
||||
}
|
||||
respond(w, 200, map[string]interface{}{
|
||||
"results": kwResults, "count": len(kwResults), "mode": "keyword",
|
||||
"candidates_scanned": 0,
|
||||
"cache_hit": cacheHit, // t_1c12e000:keyword 主路径命中 search cache 时显式标注
|
||||
})
|
||||
return
|
||||
case "semantic":
|
||||
// Pure semantic — BM25 off (H1 skipped, Pipeline Recall has BM25 built in — handled by switch)
|
||||
// t_949840b1:RecallWithMeta 额外返回候选池真实大小(ANN 粗检索池,非最终产出数)
|
||||
// Pure semantic — BM25 off(Pipeline 内部按 mode 处理)
|
||||
var meta storage.RecallMeta
|
||||
results, meta, err = a.Pipeline.RecallWithMeta(req.Query, req.Namespace, req.Limit, req.Diversity)
|
||||
results, meta, err = a.Pipeline.RecallWithMetaTraced(ctx, req.Query, req.Namespace, fetchLimit, req.Diversity, nsDerivedFromAgent, tr)
|
||||
lastMeta = meta
|
||||
annScanned = meta.CandidatesScanned
|
||||
cacheHit = meta.CacheHit // t_1c12e000
|
||||
cacheHit = meta.CacheHit
|
||||
truncated = meta.Truncated
|
||||
case "hybrid":
|
||||
// BM25 + vector combined (H1: BM25 scoring inside Pipeline.Recall)
|
||||
var meta storage.RecallMeta
|
||||
results, meta, err = a.Pipeline.RecallWithMeta(req.Query, req.Namespace, req.Limit, req.Diversity)
|
||||
results, meta, err = a.Pipeline.RecallWithMetaTraced(ctx, req.Query, req.Namespace, fetchLimit, req.Diversity, nsDerivedFromAgent, tr)
|
||||
lastMeta = meta
|
||||
annScanned = meta.CandidatesScanned
|
||||
cacheHit = meta.CacheHit // t_1c12e000
|
||||
cacheHit = meta.CacheHit
|
||||
truncated = meta.Truncated
|
||||
}
|
||||
if err != nil {
|
||||
// P0: 降级到 graph.db 关键词搜索(仅 hybrid/semantic 主路径依赖向量/rerank)
|
||||
@@ -636,10 +688,13 @@ func (a *API) Recall(w http.ResponseWriter, r *http.Request) {
|
||||
if len(fallbackResults) > 0 {
|
||||
results = convertFallbackResults(fallbackResults)
|
||||
annScanned = len(fallbackResults)
|
||||
cacheHit = false // t_1c12e000:graph 兜底结果不来自 search cache
|
||||
cacheHit = false // t_1c12e000:graph 兜底结果不来自 search cache
|
||||
truncated = false // t_3aa2a8c1:graph.db 兜底结果不经 IPC 粗检索 ⇒ 无截断可言
|
||||
w.Header().Set("X-Fallback", "graph")
|
||||
} else {
|
||||
// t_2d4e3b92: 主路径与兜底都失败 ⇒ 非 2xx + 统一错误体(绝不在 200 里塞错误)
|
||||
tr.MarkPartial("stage_error")
|
||||
tr.Error = err.Error()
|
||||
respondErrorSchema(w, 502, "recall_backend_error",
|
||||
"recall failed: "+err.Error()+"; graph fallback also empty", 5)
|
||||
log.Printf("[recall] failed: %v; fallback empty", err)
|
||||
@@ -655,7 +710,15 @@ func (a *API) Recall(w http.ResponseWriter, r *http.Request) {
|
||||
if len(results) > 0 && len(results) < 3 && req.Namespace != "shared" && req.Namespace != "" {
|
||||
vec, encErr := a.Embedder.EncodeQuery(req.Query)
|
||||
if encErr == nil {
|
||||
shared, _ := a.LanceDB.Search("memories", vec, 5, "shared")
|
||||
var shared []models.MemoryRecord
|
||||
// t_3aa2a8c1:补充搜同属粗检索 ⇒ 截断也计入本次 recall 的 truncated
|
||||
if rep, ok := a.LanceDB.(storage.SearchTruncationReporter); ok {
|
||||
var tSup bool
|
||||
shared, tSup, _ = rep.SearchWithTruncation("memories", vec, 5, "shared")
|
||||
truncated = truncated || tSup
|
||||
} else {
|
||||
shared, _ = a.LanceDB.Search("memories", vec, 5, "shared")
|
||||
}
|
||||
for _, m := range shared {
|
||||
// 去重:跳过已在 own namespace 结果中的记忆
|
||||
alreadyHave := false
|
||||
@@ -673,12 +736,20 @@ func (a *API) Recall(w http.ResponseWriter, r *http.Request) {
|
||||
vv = storage.VValueProvider(m.ID)
|
||||
}
|
||||
results = append(results, models.RecallResult{
|
||||
ID: m.ID,
|
||||
Content: m.Content,
|
||||
Category: m.Category,
|
||||
Score: 0.5, // shared 结果降权,使用默认分
|
||||
QualityScore: m.QualityScore,
|
||||
VValue: vv,
|
||||
ID: m.ID,
|
||||
Content: m.Content,
|
||||
Category: m.Category,
|
||||
Score: 0.5, // shared 结果降权,使用默认分
|
||||
QualityScore: m.QualityScore,
|
||||
VValue: vv,
|
||||
HumanUsefulCount: m.HumanUsefulCount, // A10 (t_d618dc7c)
|
||||
Channel: "shared", // P0-1:来源通道
|
||||
Version: m.Version,
|
||||
ValidFrom: m.ValidFrom,
|
||||
ValidTo: m.ValidTo,
|
||||
Supersedes: m.Supersedes,
|
||||
SupersededBy: m.SupersededBy,
|
||||
Superseded: len(m.SupersededBy) > 0,
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -692,7 +763,15 @@ func (a *API) Recall(w http.ResponseWriter, r *http.Request) {
|
||||
if req.Namespace != "wiki-curator-main" && req.Namespace != "" {
|
||||
if vec, encErr := a.Embedder.EncodeQuery(req.Query); encErr == nil {
|
||||
for _, extra := range []string{"wiki-curator-main"} {
|
||||
extraHits, _ := a.LanceDB.Search("memories", vec, 5, extra)
|
||||
var extraHits []models.MemoryRecord
|
||||
// t_3aa2a8c1:补充搜同属粗检索 ⇒ 截断也计入本次 recall 的 truncated
|
||||
if rep, ok := a.LanceDB.(storage.SearchTruncationReporter); ok {
|
||||
var tSup bool
|
||||
extraHits, tSup, _ = rep.SearchWithTruncation("memories", vec, 5, extra)
|
||||
truncated = truncated || tSup
|
||||
} else {
|
||||
extraHits, _ = a.LanceDB.Search("memories", vec, 5, extra)
|
||||
}
|
||||
for _, m := range extraHits {
|
||||
dup := false
|
||||
for _, r := range results {
|
||||
@@ -708,12 +787,20 @@ func (a *API) Recall(w http.ResponseWriter, r *http.Request) {
|
||||
vv = storage.VValueProvider(m.ID)
|
||||
}
|
||||
results = append(results, models.RecallResult{
|
||||
ID: m.ID,
|
||||
Content: m.Content,
|
||||
Category: m.Category,
|
||||
Score: 0.45, // 笔记知识降权:不挤掉主记忆,但能被看到
|
||||
QualityScore: m.QualityScore,
|
||||
VValue: vv,
|
||||
ID: m.ID,
|
||||
Content: m.Content,
|
||||
Category: m.Category,
|
||||
Score: 0.45, // 笔记知识降权:不挤掉主记忆,但能被看到
|
||||
QualityScore: m.QualityScore,
|
||||
VValue: vv,
|
||||
HumanUsefulCount: m.HumanUsefulCount, // A10 (t_d618dc7c)
|
||||
Channel: "wiki", // P0-1:来源通道
|
||||
Version: m.Version,
|
||||
ValidFrom: m.ValidFrom,
|
||||
ValidTo: m.ValidTo,
|
||||
Supersedes: m.Supersedes,
|
||||
SupersededBy: m.SupersededBy,
|
||||
Superseded: len(m.SupersededBy) > 0,
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -743,6 +830,28 @@ func (a *API) Recall(w http.ResponseWriter, r *http.Request) {
|
||||
results = ordered
|
||||
}
|
||||
|
||||
// ── 2026-10-26(P1-3 生命周期 fixture 抓到的真 bug)──────────────────
|
||||
// 主候选池在 pipeline 里已做 bi-temporal 过滤,但 **shared / wiki 补充通道与图谱注入
|
||||
// 不过那道闸** ⇒ 已失效事实会从补充通道回流进 top-k(实测:valid_to=11:00 的记录
|
||||
// 被 shared 通道以 score=0.5 重新塞回)。这里在「所有注入完成之后、cap 之前」统一过滤,
|
||||
// 保证读侧只有一条有效期口径。全失效时保留原结果(不把召回清空,non-lossy)。
|
||||
if storage.BitemporalEnabled() {
|
||||
now := time.Now()
|
||||
live := make([]models.RecallResult, 0, len(results))
|
||||
expired := 0
|
||||
for _, r := range results {
|
||||
if storage.ValidityExpired(r.ValidTo, now) {
|
||||
expired++
|
||||
continue
|
||||
}
|
||||
live = append(live, r)
|
||||
}
|
||||
if len(live) > 0 && expired > 0 {
|
||||
log.Printf("[recall] 补充通道 bi-temporal 过滤 %d 条失效事实(统一读侧口径)", expired)
|
||||
results = live
|
||||
}
|
||||
}
|
||||
|
||||
// Record recall hit rate (has results = hit, empty = miss)
|
||||
selfoptimize.Dash.RecordRecall(len(results) > 0)
|
||||
// Gap auto-close: recall 命中后自动关闭该 query 对应的 open gap
|
||||
@@ -784,19 +893,114 @@ func (a *API) Recall(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
storage.RecordRecalled(recs)
|
||||
}
|
||||
// ── 2026-10-26 (t_9025b0eb) top_k 收口:图谱扩展不得撑爆返回集 ──
|
||||
// 现象:/api/v1/recall 完全无视 top_k —— top_k=5 实测返回 1905/1625 条、top_k=10 返回
|
||||
// 2227/1859 条,其中 99% 是 category=graph_expanded(图谱扩展 append 无上限)。
|
||||
// 后果:插件记忆预取被撑到 103,299 chars 截断(真相关条目落在被截断段)、count 失真、
|
||||
// ⑤使用追踪按千级结果记账。
|
||||
// 契约修正(主结果优先):
|
||||
// results = 主检索结果(category != graph_expanded),截断到 req.Limit;
|
||||
// 主结果不足 req.Limit 时用扩展结果按原顺序补齐(稀疏查询仍有扩展收益)。
|
||||
// graph_expanded = 扩展结果**单独成字段**(上限 req.Limit)——能力仍可观测、可统计,
|
||||
// 但不再挤占主结果、不再污染 results/count。
|
||||
results, graphExpanded := capRecallResults(results, fetchLimit)
|
||||
if results == nil {
|
||||
results = []models.RecallResult{}
|
||||
}
|
||||
respond(w, 200, map[string]interface{}{
|
||||
|
||||
// ── P0-3 翻页:从 fetchLimit 结果里切出 [offset, offset+limit) ──
|
||||
// 说明:翻页重算整段前 offset+limit 条再切片,保证 MMR 前缀稳定;
|
||||
// 令牌绑定 query 指纹 + namespace,跨 query/跨域复用一律 400(防串页)。
|
||||
totalFetched := len(results)
|
||||
if offset > 0 {
|
||||
if offset >= len(results) {
|
||||
results = []models.RecallResult{}
|
||||
} else {
|
||||
results = results[offset:]
|
||||
}
|
||||
}
|
||||
if len(prevIDs) > 0 {
|
||||
seenPrev := make(map[string]bool, len(prevIDs))
|
||||
for _, id := range prevIDs {
|
||||
seenPrev[id] = true
|
||||
}
|
||||
kept := results[:0]
|
||||
for _, r := range results {
|
||||
if !seenPrev[r.ID] {
|
||||
kept = append(kept, r)
|
||||
}
|
||||
}
|
||||
results = kept
|
||||
}
|
||||
if len(results) > req.Limit {
|
||||
results = results[:req.Limit]
|
||||
}
|
||||
nextCont := ""
|
||||
if totalFetched >= fetchLimit && len(results) == req.Limit && offset+req.Limit < maxContinuationOffset {
|
||||
returned := make([]string, 0, len(prevIDs)+len(results))
|
||||
returned = append(returned, prevIDs...)
|
||||
for _, r := range results {
|
||||
returned = append(returned, r.ID)
|
||||
}
|
||||
nextCont = encodeContinuation(req.Query, req.Namespace, offset+req.Limit, req.Limit, returned)
|
||||
}
|
||||
|
||||
// ── P0-1/P0-3:契约字段 + trace 落盘 ──
|
||||
tr.SetItems(traceItems(results))
|
||||
status := lastMeta.Status
|
||||
if status == "" {
|
||||
status = "complete"
|
||||
}
|
||||
partialReason := lastMeta.PartialReason
|
||||
if tr.Status == "partial" && status == "complete" {
|
||||
status = "partial"
|
||||
partialReason = tr.PartialReason
|
||||
}
|
||||
if truncated && status == "complete" {
|
||||
status = "partial"
|
||||
partialReason = "row_cap_truncated"
|
||||
}
|
||||
tr.Status = status
|
||||
tr.PartialReason = partialReason
|
||||
tr.CacheHit = cacheHit
|
||||
tr.Truncated = truncated
|
||||
tr.CandidatesScanned = annScanned
|
||||
tr.Namespace = req.Namespace
|
||||
tr.Mode = req.Mode
|
||||
tr.Scope = buildScope(req.Namespace, req.Limit, req.Mode, cacheHit, truncated, tr.StartedAt)
|
||||
if offset > 0 {
|
||||
tr.Scope["offset"] = offset
|
||||
}
|
||||
storage.SaveTrace(tr)
|
||||
|
||||
resp := map[string]interface{}{
|
||||
"results": results,
|
||||
"count": len(results),
|
||||
"candidates_scanned": annScanned,
|
||||
"mode": req.Mode,
|
||||
// 2026-10-26 (t_9025b0eb):扩展结果单独成字段,避免撑爆 results。
|
||||
// 键恒存在(可为空数组),便于探针/统计区分「无扩展」与「字段缺失」。
|
||||
"graph_expanded": graphExpanded,
|
||||
// t_1c12e000:AC3 —— 缓存命中/回源扫表必须可区分。
|
||||
// true ⇒ candidates_scanned=0 是「读了 60s 内的旧快照」(含 L1/L2);
|
||||
// false ⇒ 本次真扫了表(candidates_scanned 为真实候选池大小)。
|
||||
"cache_hit": cacheHit,
|
||||
})
|
||||
// 2026-10-26(P0-1/P0-3 契约):
|
||||
// trace_id ⇒ GET /api/v1/trace/<id> 可重建本次调用链(阶段耗时 + 每通道贡献);
|
||||
// status ⇒ complete 的空结果才可解读为「不存在」;partial 必须看 partial_reason;
|
||||
// scope ⇒ 本次答案基于哪个数据视图(namespace/snapshot/cache_hit/truncated);
|
||||
// continuation ⇒ 空串=无下一页,非空=拼回请求体取下一页。
|
||||
"trace_id": tr.TraceID,
|
||||
"status": status,
|
||||
"scope": tr.Scope,
|
||||
"continuation": nextCont,
|
||||
}
|
||||
if partialReason != "" {
|
||||
resp["partial_reason"] = partialReason
|
||||
}
|
||||
// t_3aa2a8c1:truncated —— 只在 true 时暴露(键缺失 = 未截断,等价 omitempty 语义)。
|
||||
if truncated {
|
||||
resp["truncated"] = true
|
||||
}
|
||||
respond(w, 200, resp)
|
||||
|
||||
// H3: Async trust score update
|
||||
if a.GraphStore != nil {
|
||||
@@ -965,6 +1169,7 @@ func (a *API) RecallDebug(w http.ResponseWriter, r *http.Request) {
|
||||
results = append(results, models.RecallResult{
|
||||
ID: c.ID, Content: c.Content, Category: c.Category,
|
||||
Score: score, Timestamp: c.CreatedAt.Format(time.RFC3339),
|
||||
HumanUsefulCount: c.HumanUsefulCount, // A10 (t_d618dc7c)
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -1423,3 +1628,74 @@ func (a *API) EdgeFeedback(w http.ResponseWriter, r *http.Request) {
|
||||
_ = a.GraphStore.UpdateEdgeTrustScores()
|
||||
respond(w, 200, map[string]string{"status": "ok"})
|
||||
}
|
||||
|
||||
// GET /api/v1/popwatch — t_23b6a940 AC7:热门度看门狗快照(report-only 观测)。
|
||||
// 服务端记录「最终 rank1 的 recall_verified_count 连续越 p95 阈值」计数
|
||||
// (storage.PopWatch,观测点在 recallUncached 主路径漏斗末尾);本端点只读快照,
|
||||
// 不改排序、不写库。传感器端脚本(zhiyi-popwatch.py)读它转既有告警通道;
|
||||
// ZHIYI_POPWATCH=0 时 enabled=false(观测关闭)。
|
||||
func (a *API) PopWatch(w http.ResponseWriter, r *http.Request) {
|
||||
respond(w, 200, storage.PopWatchSnapshot())
|
||||
}
|
||||
|
||||
// capRecallResults —— 2026-10-26 (t_9025b0eb) top_k 收口语义(纯函数,可单测)。
|
||||
//
|
||||
// 输入是 pipeline 返回的原始结果(主检索在前、图谱扩展 append 在后,扩展无上限)。
|
||||
// 输出:
|
||||
//
|
||||
// prim = 主检索结果(category != graph_expanded),截断到 limit;
|
||||
// 若主结果不足 limit,用扩展结果按原顺序补齐(稀疏查询仍有扩展收益)。
|
||||
// expanded = 剩余扩展结果,单独成字段返回,上限 limit(能力可观测,但不挤占主结果)。
|
||||
//
|
||||
// 为什么不是「直接 results[:limit]」:扩展结果全部排在主结果之后,直接截断会让
|
||||
// category=graph_expanded 恒为 0 ⇒ 能力虽在跑但对外永久失明(本仓 2026-10 已因同类
|
||||
// 「接线≠出数」栽过一次)。所以既收口、又保留可观测通道。
|
||||
func capRecallResults(results []models.RecallResult, limit int) (prim, expanded []models.RecallResult) {
|
||||
if limit <= 0 {
|
||||
limit = 10
|
||||
}
|
||||
prim = make([]models.RecallResult, 0, limit)
|
||||
if len(results) == 0 {
|
||||
return prim, []models.RecallResult{}
|
||||
}
|
||||
exp := make([]models.RecallResult, 0, len(results))
|
||||
// 2026-10-26 (t_e98cb1d6):pattern / peer 是「附带注入」的召回条目(recall.go Step 6.2/6.3),
|
||||
// 追加在 results 末尾 ⇒ 若并入 prim 一起截断,distilled 占满 limit 时它们恒被切光
|
||||
// (16:28 topkcap 引入的 pattern 截断回归,limit=50 时 pattern=0 / limit=200 才现形 3 条)。
|
||||
// 正确语义:主检索结果(distilled 等)照常收口到 limit;pattern/peer 作为附带条目
|
||||
// 恒保留(不挤占主结果配额),避免「命中即注入」被 cap 吞掉。graph_expanded 仍走独立字段。
|
||||
var carry []models.RecallResult
|
||||
for _, r := range results {
|
||||
switch r.Category {
|
||||
case "graph_expanded":
|
||||
exp = append(exp, r)
|
||||
case "pattern", "peer":
|
||||
carry = append(carry, r)
|
||||
default:
|
||||
prim = append(prim, r)
|
||||
}
|
||||
}
|
||||
// 先收口主结果
|
||||
if len(prim) > limit {
|
||||
prim = prim[:limit]
|
||||
}
|
||||
// 主结果不足 limit 时用扩展结果按原顺序补齐(稀疏查询仍有扩展收益)
|
||||
if len(exp) > 0 {
|
||||
if room := limit - len(prim); room > 0 {
|
||||
if room > len(exp) {
|
||||
room = len(exp)
|
||||
}
|
||||
prim = append(prim, exp[:room]...)
|
||||
exp = exp[room:]
|
||||
}
|
||||
if len(exp) > limit {
|
||||
exp = exp[:limit]
|
||||
}
|
||||
}
|
||||
// pattern / peer 附带条目恒保留在 results 末尾(独立于主结果配额)
|
||||
prim = append(prim, carry...)
|
||||
if expanded == nil {
|
||||
expanded = exp
|
||||
}
|
||||
return prim, expanded
|
||||
}
|
||||
|
||||
@@ -0,0 +1,153 @@
|
||||
package routes
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/xiaoxue/memoryweave/internal/models"
|
||||
)
|
||||
|
||||
// 2026-10-26 (t_9025b0eb):top_k 收口契约测试。
|
||||
// 起因:/api/v1/recall 完全无视 top_k —— top_k=5 实测返回 1905/1625 条(99% graph_expanded),
|
||||
// 插件记忆预取被撑到 103,299 chars 截断、count 失真、使用追踪按千级结果记账。
|
||||
// 钉住四件事:① results 不超过 limit ② 主结果不被扩展挤掉 ③ 扩展仍可见且不超 limit
|
||||
// ④ 稀疏查询(主结果 < limit)时扩展能补齐。
|
||||
func TestCapRecallResults(t *testing.T) {
|
||||
mk := func(n int, cat string) []models.RecallResult {
|
||||
out := make([]models.RecallResult, n)
|
||||
for i := 0; i < n; i++ {
|
||||
out[i] = models.RecallResult{ID: cat + string(rune('a'+i%26)), Category: cat, Content: cat}
|
||||
}
|
||||
return out
|
||||
}
|
||||
cases := []struct {
|
||||
name string
|
||||
prim, exp int
|
||||
limit int
|
||||
wantPrim int
|
||||
wantExp int
|
||||
primCategory string
|
||||
}{
|
||||
{"主结果足额_扩展不得进results", 10, 1900, 5, 5, 5, "distilled"},
|
||||
{"主结果只有2条_扩展补齐到limit", 2, 900, 5, 5, 5, "distilled"},
|
||||
{"无扩展_纯截断", 30, 0, 10, 10, 0, "distilled"},
|
||||
{"扩展少于limit", 1, 3, 5, 4, 0, "distilled"},
|
||||
{"空结果", 0, 0, 5, 0, 0, ""},
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
all := append(mk(c.prim, "distilled"), mk(c.exp, "graph_expanded")...)
|
||||
prim, exp := capRecallResults(all, c.limit)
|
||||
if len(prim) != c.wantPrim {
|
||||
t.Fatalf("prim=%d want %d", len(prim), c.wantPrim)
|
||||
}
|
||||
if len(exp) != c.wantExp {
|
||||
t.Fatalf("expanded=%d want %d", len(exp), c.wantExp)
|
||||
}
|
||||
if len(prim)+len(exp) > c.limit*2 {
|
||||
t.Fatalf("总量失控: prim=%d exp=%d", len(prim), len(exp))
|
||||
}
|
||||
// 主结果条数不得超过原始主结果数(不得凭空造)
|
||||
n := 0
|
||||
for _, r := range prim {
|
||||
if r.Category != "graph_expanded" {
|
||||
n++
|
||||
}
|
||||
}
|
||||
if n > c.prim {
|
||||
t.Fatalf("主结果 %d 条,超过输入 %d 条(凭空造)", n, c.prim)
|
||||
}
|
||||
if c.primCategory != "" && c.prim > 0 && n == 0 {
|
||||
t.Fatalf("主结果被扩展挤光:%+v", prim)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// limit 非法(0/负数)时按默认 10 处理,不做无界返回。
|
||||
func TestCapRecallResultsDefaultLimit(t *testing.T) {
|
||||
all := make([]models.RecallResult, 50)
|
||||
for i := range all {
|
||||
all[i] = models.RecallResult{Category: "distilled"}
|
||||
}
|
||||
prim, _ := capRecallResults(all, 0)
|
||||
if len(prim) != 10 {
|
||||
t.Fatalf("limit=0 应回落 10,得到 %d", len(prim))
|
||||
}
|
||||
}
|
||||
|
||||
// ── t_e98cb1d6 回归:pattern / peer 被 cap 截断 ────────────────────────────
|
||||
// 病灶:capRecallResults 把非 graph_expanded 全部并入 prim 再截断到 limit;
|
||||
// pattern/peer 是 recall.go Step 6.2/6.3 追加到 results 末尾的附带条目,
|
||||
// distilled 占满 limit 时它们被整段切掉(16:28 topkcap 引入,limit=50 时 pattern=0)。
|
||||
// 修后:pattern/peer 恒保留在 results 末尾(独立于主结果配额),主结果仍照常收口。
|
||||
func TestCapRecallResults_PatternAndPeerSurviveCap(t *testing.T) {
|
||||
mk := func(n int, cat string) []models.RecallResult {
|
||||
out := make([]models.RecallResult, n)
|
||||
for i := 0; i < n; i++ {
|
||||
out[i] = models.RecallResult{ID: cat + string(rune('a'+i%26)), Category: cat, Content: cat}
|
||||
}
|
||||
return out
|
||||
}
|
||||
count := func(rs []models.RecallResult, cat string) int {
|
||||
n := 0
|
||||
for _, r := range rs {
|
||||
if r.Category == cat {
|
||||
n++
|
||||
}
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
d, p, pe, e int // distilled / pattern / peer / graph_expanded
|
||||
limit int
|
||||
wantD, wantP, wantPe int
|
||||
}{
|
||||
// ① 金标场景:主结果占满 limit,pattern/peer 追加在末尾 ⇒ 修前恒为 0
|
||||
{"limit=50_主结果占满_pattern 恒保留", 50, 3, 1, 0, 50, 50, 3, 1},
|
||||
{"limit=200_主结果占满_pattern 仍保留", 100, 3, 1, 97, 200, 100, 3, 1},
|
||||
// ② peer 注入同样不被截断
|
||||
{"limit=50_peer 注入不被截断", 50, 0, 2, 0, 50, 50, 0, 2},
|
||||
// ③ 主结果不足 limit:扩展先补齐主结果,carry 仍保留在末尾
|
||||
{"limit=50_主结果不足_扩展补齐+carry 保留", 30, 2, 1, 10, 50, 30, 2, 1},
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
all := make([]models.RecallResult, 0, c.d+c.p+c.pe+c.e)
|
||||
all = append(all, mk(c.d, "distilled")...)
|
||||
all = append(all, mk(c.p, "pattern")...)
|
||||
all = append(all, mk(c.pe, "peer")...)
|
||||
all = append(all, mk(c.e, "graph_expanded")...)
|
||||
prim, exp := capRecallResults(all, c.limit)
|
||||
d := count(prim, "distilled")
|
||||
p := count(prim, "pattern")
|
||||
pe := count(prim, "peer")
|
||||
e := count(prim, "graph_expanded")
|
||||
if d != c.wantD {
|
||||
t.Errorf("distilled=%d want %d", d, c.wantD)
|
||||
}
|
||||
// AC:limit=50 时 pattern 必须 ≥1(金标 query 复现终审现象)
|
||||
if c.p > 0 && p < 1 {
|
||||
t.Errorf("pattern=%d,应 ≥1(截断回归未修)", p)
|
||||
}
|
||||
if p != c.wantP {
|
||||
t.Errorf("pattern=%d want %d", p, c.wantP)
|
||||
}
|
||||
if pe != c.wantPe {
|
||||
t.Errorf("peer=%d want %d", pe, c.wantPe)
|
||||
}
|
||||
// cap 仍然生效:主结果(distilled + 扩展补齐)不得超过 limit
|
||||
if d+e > c.limit {
|
||||
t.Fatalf("主结果超 cap: distilled=%d graph_expanded=%d limit=%d", d, e, c.limit)
|
||||
}
|
||||
if len(exp) > c.limit {
|
||||
t.Fatalf("graph_expanded 字段超 cap: %d > %d", len(exp), c.limit)
|
||||
}
|
||||
// 附带条目不得被塞进 graph_expanded 字段
|
||||
if count(exp, "pattern") > 0 || count(exp, "peer") > 0 {
|
||||
t.Fatalf("pattern/peer 被误放入 graph_expanded 字段")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
@@ -55,22 +56,29 @@ type GoldenSet struct {
|
||||
func LoadGoldenSet(path string) (*GoldenSet, error) {
|
||||
var data []byte
|
||||
usedSource := "env:ZHIYI_GOLDEN_SET"
|
||||
if p := os.Getenv("ZHIYI_GOLDEN_SET"); p != "" {
|
||||
// 2026-10-26 修正优先级:显式 path > env > 内嵌。
|
||||
// 旧顺序(env 先)导致 ZHIYI_GOLDEN_SET 一旦设置,任何请求传 golden_path 都被静默忽略
|
||||
// ⇒ 实测「传 7 个 chunk 却每块都报 53 条」(env 指向的 53 条集)—— 评测对象与预期不符,
|
||||
// 且指标全是 0 时极难发现。env 现在只作「默认金标」,显式 path 必须能覆盖它。
|
||||
if path != "" {
|
||||
usedSource = "path:" + path
|
||||
d, err := os.ReadFile(path)
|
||||
if err == nil {
|
||||
data = d
|
||||
} else if filepath.IsAbs(path) {
|
||||
// 绝对路径 = 明确指名要跑这套题 ⇒ 读不到必须报错,不许静默换题
|
||||
return nil, fmt.Errorf("读 golden_path 失败(%s): %w", path, err)
|
||||
} else {
|
||||
// 相对路径在部署 CWD 下读不到 ⇒ 内嵌兜底(t_dfc93ad0 AC5 既有行为,保留)
|
||||
data = embeddedGoldenSetJSON
|
||||
usedSource = "embedded"
|
||||
}
|
||||
} else if p := os.Getenv("ZHIYI_GOLDEN_SET"); p != "" {
|
||||
var err error
|
||||
data, err = os.ReadFile(p)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
} else if path != "" {
|
||||
usedSource = "path:" + path
|
||||
d, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
// 相对路径在部署 CWD 下读不到 ⇒ 内嵌兜底(AC5 根因)
|
||||
data = embeddedGoldenSetJSON
|
||||
usedSource = "embedded"
|
||||
} else {
|
||||
data = d
|
||||
}
|
||||
} else {
|
||||
data = embeddedGoldenSetJSON
|
||||
usedSource = "embedded"
|
||||
@@ -384,13 +392,20 @@ func (ea *EvalAPI) RunGolden(w http.ResponseWriter, r *http.Request) {
|
||||
var req struct {
|
||||
Namespace string `json:"namespace"`
|
||||
GoldenPath string `json:"golden_path"` // 可选,默认 testdata/golden-set.json
|
||||
// AllNamespaces(2026-10-26)跨 namespace 评测开关。
|
||||
// 背景:旧逻辑 ns 为空 ⇒ 强制 "shared",而真实金标分散在 hermes-main/wiki/HUB 等多个 ns
|
||||
// ⇒ 评测跑的是"只看 shared"的召回,与线上跨 ns 召回不是同一个系统(实测前 15 条里 15 条结果不同)。
|
||||
// 语义:true ⇒ ns = ""(全库跨 ns,等同线上默认);false ⇒ 保持旧行为(默认 shared,向后兼容)。
|
||||
AllNamespaces bool `json:"all_namespaces"`
|
||||
}
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
respondError(w, 400, "invalid body")
|
||||
return
|
||||
}
|
||||
ns := req.Namespace
|
||||
if ns == "" {
|
||||
if req.AllNamespaces {
|
||||
ns = ""
|
||||
} else if ns == "" {
|
||||
ns = "shared"
|
||||
}
|
||||
gs, err := LoadGoldenSet(req.GoldenPath)
|
||||
|
||||
@@ -23,6 +23,11 @@ func NewFeedbackAPI(ldb storage.LanceDB) *FeedbackAPI {
|
||||
func (fa *FeedbackAPI) MarkUseful(w http.ResponseWriter, r *http.Request) {
|
||||
var req struct {
|
||||
MemoryID string `json:"memory_id"`
|
||||
// 2026-10-24 (t_11245d30):区分人机信号。
|
||||
// 空 = 人工点「有用」(保持 user_feedback,行为不变);
|
||||
// "auto" = 插件自动反馈(_auto_feedback_for_reply)→ 记 auto_feedback,让 V 账可辨「牧尘认可 vs 机器自评」。
|
||||
// 纯观测,不改 V 权重/阈值/上限。
|
||||
Source string `json:"source"`
|
||||
}
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
respondError(w, 400, "invalid body")
|
||||
@@ -34,11 +39,15 @@ func (fa *FeedbackAPI) MarkUseful(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
|
||||
selfoptimize.Dash.RecordFeedback(true)
|
||||
// V 值:useful = success
|
||||
// V 值:useful = success。action 名按来源区分(人工 user_feedback / 自动 auto_feedback)。
|
||||
action := "user_feedback"
|
||||
if req.Source == "auto" {
|
||||
action = "auto_feedback"
|
||||
}
|
||||
selfoptimize.VProp.RecordDecision(
|
||||
"useful_"+req.MemoryID,
|
||||
[]string{req.MemoryID},
|
||||
"user_feedback", "success", "",
|
||||
action, "success", "",
|
||||
)
|
||||
if err := fa.LanceDB.IncrementUseful(req.MemoryID); err != nil {
|
||||
log.Printf("[feedback] IncrementUseful failed %s: %v", req.MemoryID, err)
|
||||
|
||||
@@ -57,7 +57,7 @@ func (pa *PatternAPI) List(w http.ResponseWriter, r *http.Request) {
|
||||
limit = n
|
||||
}
|
||||
}
|
||||
pats, err := pa.Store.ListPatterns(limit)
|
||||
pats, _, err := pa.Store.ListPatterns(limit)
|
||||
if err != nil {
|
||||
respondError(w, 500, "list patterns: "+err.Error())
|
||||
return
|
||||
|
||||
@@ -0,0 +1,67 @@
|
||||
package routes
|
||||
|
||||
// 2026-10-26(P1-1):supersede / 版本字段必须真的出现在 /recall 响应里。
|
||||
// 用同包 fakeLanceDB 构造一条「已被取代 + 有效期已过 + 版本 2」的记录:
|
||||
// - 这是「写侧有、读侧看不见」的复发点(t_4c810e02「加列不接 IPC=静默 no-op」同款坑);
|
||||
// - 判据只看 JSON 响应键值,不看结构体注释。
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/xiaoxue/memoryweave/internal/models"
|
||||
)
|
||||
|
||||
func TestRecall_SupersedeAndValidityFieldsExposed(t *testing.T) {
|
||||
db := &fakeLanceDB{searchResults: []models.MemoryRecord{
|
||||
{
|
||||
ID: "mem_old_fact",
|
||||
Content: "织忆默认 stale 阈值是 720 小时(旧事实)",
|
||||
Category: "distilled",
|
||||
Version: 2,
|
||||
ValidFrom: "2026-10-26T10:00:00+08:00",
|
||||
ValidTo: "2026-10-26T11:00:00+08:00",
|
||||
SupersededBy: []string{"mem_new_fact"},
|
||||
},
|
||||
{ID: "mem_fresh", Content: "织忆默认 stale 阈值是 720 小时(现行事实)", Category: "distilled"},
|
||||
}}
|
||||
code, m, body := recallBody(t, db, `{"query":"stale 阈值多少小时","limit":5,"namespace":"hermes-main"}`)
|
||||
if code != 200 {
|
||||
t.Fatalf("expected 200, got %d; body=%s", code, body)
|
||||
}
|
||||
results, _ := m["results"].([]interface{})
|
||||
if len(results) == 0 {
|
||||
t.Fatalf("无结果,无法验证字段:%s", body)
|
||||
}
|
||||
var found map[string]interface{}
|
||||
for _, raw := range results {
|
||||
r, _ := raw.(map[string]interface{})
|
||||
if r["id"] == "mem_old_fact" {
|
||||
found = r
|
||||
break
|
||||
}
|
||||
}
|
||||
if found == nil {
|
||||
t.Fatalf("目标记录未被召回(fake DB 应原样返回),body=%s", body)
|
||||
}
|
||||
if found["superseded_by"] == nil {
|
||||
t.Fatalf("superseded_by 未暴露给读侧:%v", found)
|
||||
}
|
||||
if found["superseded"] != true {
|
||||
t.Fatalf("superseded 便捷布尔应为 true:%v", found["superseded"])
|
||||
}
|
||||
if found["generation"] != float64(2) {
|
||||
t.Fatalf("generation 应为 2:%v", found["generation"])
|
||||
}
|
||||
if found["valid_to"] != "2026-10-26T11:00:00+08:00" {
|
||||
t.Fatalf("valid_to 未直通:%v", found["valid_to"])
|
||||
}
|
||||
if found["channel"] == nil || found["channel"] == "" {
|
||||
t.Fatalf("channel 未暴露:%v", found)
|
||||
}
|
||||
if _, ok := m["trace_id"]; !ok {
|
||||
t.Fatalf("响应缺 trace_id:%s", body)
|
||||
}
|
||||
if m["status"] != "complete" {
|
||||
t.Fatalf("正常召回 status 应为 complete,实际 %v", m["status"])
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,115 @@
|
||||
package routes
|
||||
|
||||
// 2026-10-26(P1-3):stale-return + 生命周期 fixture(10:00 生效 / 11:00 失效)。
|
||||
//
|
||||
// 评审缺口:「只有离线检索指标,没有 stale-return、没有生命周期 fixture」。
|
||||
// 本测试把三条读侧生命周期行为锁成机械判据:
|
||||
// ① bi-temporal 开启时,valid_to 已过的事实**不进 top-k**(但仍在库,non-lossy);
|
||||
// ② valid_to 在未来 / 为空 ⇒ 正常返回(同一 fixture 的对照,证明不是「整类被吞」);
|
||||
// ③ updated_at 超 720h ⇒ 响应 stale=true;新记录不输出该键。
|
||||
//
|
||||
// 时窗用真实语义:fixture 事实 10:00 生效、11:00 失效,查询时点取「当日 11:00 之后」。
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/xiaoxue/memoryweave/internal/models"
|
||||
)
|
||||
|
||||
// 时窗用**相对时间**:曾经写成 dayAt(10,0)/dayAt(11,0) 假设「跑测试时已是晚上 20:00」,
|
||||
// 凌晨 02:07 跑就是 flaky —— 11:00 还没到,「已失效」的 fixture 其实仍有效。
|
||||
// 生命周期判据必须与运行时刻无关:过期 = now-1h,有效期 = now+24h。
|
||||
func expiredPair() (from, to, future string) {
|
||||
now := time.Now()
|
||||
return now.Add(-2 * time.Hour).Format(time.RFC3339),
|
||||
now.Add(-1 * time.Hour).Format(time.RFC3339),
|
||||
now.Add(24 * time.Hour).Format(time.RFC3339)
|
||||
}
|
||||
|
||||
func idsOf(m map[string]interface{}) map[string]bool {
|
||||
out := map[string]bool{}
|
||||
if res, ok := m["results"].([]interface{}); ok {
|
||||
for _, raw := range res {
|
||||
if r, ok := raw.(map[string]interface{}); ok {
|
||||
if id, ok := r["id"].(string); ok {
|
||||
out[id] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func TestRecall_Lifecycle_ExpiredFactDroppedWhenBitemporal(t *testing.T) {
|
||||
from, to, future := expiredPair()
|
||||
|
||||
db := &fakeLanceDB{searchResults: []models.MemoryRecord{
|
||||
{ID: "mem_expired_window", Content: "生命周期 fixture:10:00 生效 11:00 失效的事实",
|
||||
Category: "distilled", ValidFrom: from, ValidTo: to},
|
||||
{ID: "mem_still_valid", Content: "生命周期 fixture:仍在有效期内的事实",
|
||||
Category: "distilled", ValidFrom: from, ValidTo: future},
|
||||
}}
|
||||
t.Setenv("ZHIYI_BITEMPORAL", "1")
|
||||
code, m, body := recallBody(t, db, `{"query":"生命周期 fixture 有效期","limit":5,"namespace":"hermes-main"}`)
|
||||
if code != 200 {
|
||||
t.Fatalf("expected 200, got %d; body=%s", code, body)
|
||||
}
|
||||
ids := idsOf(m)
|
||||
if ids["mem_expired_window"] {
|
||||
t.Fatalf("11:00 已失效的事实不应进 top-k(bi-temporal 开);body=%s", body)
|
||||
}
|
||||
if !ids["mem_still_valid"] {
|
||||
t.Fatalf("有效期未过的事实必须正常返回(对照缺失 ⇒ 可能是整类被吞);body=%s", body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRecall_Lifecycle_ExpiredFactVisibleWhenBitemporalOff(t *testing.T) {
|
||||
// 开关关(默认)⇒ 行为零变化:过期事实照常返回,但 valid_to 字段可见(读侧能自己判)。
|
||||
from, to, _ := expiredPair()
|
||||
db := &fakeLanceDB{searchResults: []models.MemoryRecord{
|
||||
{ID: "mem_expired_visible", Content: "生命周期 fixture:过期但可见", Category: "distilled",
|
||||
ValidFrom: from, ValidTo: to},
|
||||
}}
|
||||
t.Setenv("ZHIYI_BITEMPORAL", "0")
|
||||
code, m, body := recallBody(t, db, `{"query":"过期但可见","limit":5,"namespace":"hermes-main"}`)
|
||||
if code != 200 {
|
||||
t.Fatalf("expected 200, got %d; body=%s", code, body)
|
||||
}
|
||||
if !idsOf(m)["mem_expired_visible"] {
|
||||
t.Fatalf("bi-temporal 关时过期事实应照常返回(non-lossy 可见性);body=%s", body)
|
||||
}
|
||||
var found map[string]interface{}
|
||||
for _, raw := range m["results"].([]interface{}) {
|
||||
if r, ok := raw.(map[string]interface{}); ok && r["id"] == "mem_expired_visible" {
|
||||
found = r
|
||||
}
|
||||
}
|
||||
if found == nil || found["valid_to"] != to {
|
||||
t.Fatalf("valid_to 必须直通给读侧:%v", found)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRecall_StaleReturn_DefaultThreshold(t *testing.T) {
|
||||
old := time.Now().Add(-31 * 24 * time.Hour)
|
||||
db := &fakeLanceDB{searchResults: []models.MemoryRecord{
|
||||
{ID: "mem_old", Content: "陈年事实(31 天未更新)", Category: "distilled", UpdatedAt: old},
|
||||
{ID: "mem_new", Content: "新鲜事实(今天更新)", Category: "distilled", UpdatedAt: time.Now()},
|
||||
}}
|
||||
code, m, body := recallBody(t, db, `{"query":"事实 新鲜度","limit":5,"namespace":"hermes-main"}`)
|
||||
if code != 200 {
|
||||
t.Fatalf("expected 200, got %d; body=%s", code, body)
|
||||
}
|
||||
got := map[string]map[string]interface{}{}
|
||||
for _, raw := range m["results"].([]interface{}) {
|
||||
if r, ok := raw.(map[string]interface{}); ok {
|
||||
got[r["id"].(string)] = r
|
||||
}
|
||||
}
|
||||
if got["mem_old"] == nil || got["mem_old"]["stale"] != true {
|
||||
t.Fatalf("31 天未更新的记忆应 stale=true:%v", got["mem_old"])
|
||||
}
|
||||
if _, ok := got["mem_new"]["stale"]; ok {
|
||||
t.Fatalf("新记忆不应输出 stale 键:%v", got["mem_new"])
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,144 @@
|
||||
package routes
|
||||
|
||||
// t_3aa2a8c1:`/api/v1/recall` 响应暴露 `truncated`(Rust 侧单次重读行数上限截断标志,
|
||||
// 引擎已算、此前只打日志不上抛)。选定的省略语义 = **键缺失即未截断(omitempty)**。
|
||||
// 本测试经路由层机械判定:
|
||||
// ① 粗检索被截断(后端实现 SearchTruncationReporter 且上报 true)⇒ 响应含 "truncated":true
|
||||
// ② 未截断 / 纯 Go 后端(不实现 reporter)⇒ 键缺失(不出现 truncated:false 以外的值)
|
||||
// ③ 回归:既有字段 results/count/candidates_scanned/mode/cache_hit 行为不变
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/xiaoxue/memoryweave/internal/governance"
|
||||
"github.com/xiaoxue/memoryweave/internal/models"
|
||||
"github.com/xiaoxue/memoryweave/internal/storage"
|
||||
)
|
||||
|
||||
// truncReportingLDB 复用同包 fakeLanceDB(recall_error_shape_test.go),
|
||||
// 额外实现可选扩展接口 SearchWithTruncationReporter(t_3aa2a8c1)。
|
||||
type truncReportingLDB struct {
|
||||
*fakeLanceDB
|
||||
reportTruncated bool
|
||||
}
|
||||
|
||||
func (f *truncReportingLDB) SearchWithTruncation(table string, vector []float32, topK int, namespaceFilter string) ([]models.MemoryRecord, bool, error) {
|
||||
if f.searchErr != nil {
|
||||
return nil, false, f.searchErr
|
||||
}
|
||||
if len(f.searchResults) > topK {
|
||||
return f.searchResults[:topK], f.reportTruncated, nil
|
||||
}
|
||||
return f.searchResults, f.reportTruncated, nil
|
||||
}
|
||||
|
||||
func recallBody(t *testing.T, db storage.LanceDB, body string) (int, map[string]interface{}, string) {
|
||||
t.Helper()
|
||||
// 测试须可复跑(t_3aa2a8c1 实测踩坑):SearchCacheInstance 由 storage 包 init() 恒建
|
||||
// (L2 = 共享 Redis,TTL 60s)⇒ 上一次 run 的 recall 结果会进 L2,下一次 run 同 key
|
||||
// 命中缓存路径(Truncated 恒 false)⇒ 断言 flaky。逐测试摘除缓存(测后恢复),
|
||||
// 恒走未缓存路径。包内无 t.Parallel ⇒ 改全局安全。
|
||||
saved := storage.SearchCacheInstance
|
||||
storage.SearchCacheInstance = nil
|
||||
t.Cleanup(func() { storage.SearchCacheInstance = saved })
|
||||
|
||||
emb := storage.NewEmbedder("")
|
||||
rerank := storage.NewReranker("", emb)
|
||||
api := NewAPI(db, emb, rerank, nil, governance.NewInMemoryGraph())
|
||||
req := httptest.NewRequest("POST", "/api/v1/recall", strings.NewReader(body))
|
||||
w := httptest.NewRecorder()
|
||||
api.Recall(w, req)
|
||||
var m map[string]interface{}
|
||||
_ = json.Unmarshal(w.Body.Bytes(), &m)
|
||||
return w.Code, m, w.Body.String()
|
||||
}
|
||||
|
||||
// 既有响应键回归基线(t_2d4e3b92 成功 schema + t_1c12e000 cache_hit + t_949840b1 candidates_scanned)
|
||||
var recallBaselineKeys = []string{"results", "count", "candidates_scanned", "mode", "cache_hit"}
|
||||
|
||||
func assertBaseline(t *testing.T, m map[string]interface{}, body string) {
|
||||
t.Helper()
|
||||
for _, k := range recallBaselineKeys {
|
||||
if _, ok := m[k]; !ok {
|
||||
t.Fatalf("既有字段 %q 缺失(回归破坏),body=%s", k, body)
|
||||
}
|
||||
}
|
||||
if res, ok := m["results"].([]interface{}); ok {
|
||||
if cnt, _ := m["count"].(float64); int(cnt) != len(res) {
|
||||
t.Fatalf("count=%v 与 len(results)=%d 不一致(回归破坏)", m["count"], len(res))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ① 粗检索被行数上限截断 ⇒ 响应暴露 truncated:true + 既有字段回归不变
|
||||
func TestRecall_TruncatedExposedWhenHitCap(t *testing.T) {
|
||||
db := &truncReportingLDB{
|
||||
fakeLanceDB: &fakeLanceDB{searchResults: []models.MemoryRecord{
|
||||
{ID: "mem_1", Content: "test memory one", Category: "distilled"},
|
||||
{ID: "mem_2", Content: "test memory two", Category: "distilled"},
|
||||
}},
|
||||
reportTruncated: true,
|
||||
}
|
||||
code, m, body := recallBody(t, db, `{"query":"test query","limit":5,"namespace":"shared"}`)
|
||||
if code != 200 {
|
||||
t.Fatalf("expected 200, got %d; body=%s", code, body)
|
||||
}
|
||||
if v, ok := m["truncated"]; !ok || v != true {
|
||||
t.Fatalf("粗检索已截断,响应必须含 truncated:true;实际 key 存在=%v 值=%v body=%s", ok, m["truncated"], body)
|
||||
}
|
||||
assertBaseline(t, m, body)
|
||||
}
|
||||
|
||||
// ①b keyword 主路径(kwMeta 透传)同判据
|
||||
func TestRecall_Keyword_TruncatedExposed(t *testing.T) {
|
||||
db := &truncReportingLDB{
|
||||
fakeLanceDB: &fakeLanceDB{searchResults: []models.MemoryRecord{
|
||||
{ID: "mem_1", Content: "keyword memory", Category: "distilled"},
|
||||
}},
|
||||
reportTruncated: true,
|
||||
}
|
||||
code, m, body := recallBody(t, db, `{"query":"keyword query","limit":5,"mode":"keyword","namespace":"shared"}`)
|
||||
if code != 200 {
|
||||
t.Fatalf("expected 200, got %d; body=%s", code, body)
|
||||
}
|
||||
if v, ok := m["truncated"]; !ok || v != true {
|
||||
t.Fatalf("keyword 路径粗检索已截断,响应必须含 truncated:true;body=%s", body)
|
||||
}
|
||||
assertBaseline(t, m, body)
|
||||
}
|
||||
|
||||
// ② 未截断 ⇒ 键缺失(选定语义:省略=未截断),不得出现 truncated:false 以外的值
|
||||
func TestRecall_NotTruncated_KeyOmitted(t *testing.T) {
|
||||
db := &truncReportingLDB{
|
||||
fakeLanceDB: &fakeLanceDB{searchResults: []models.MemoryRecord{
|
||||
{ID: "mem_1", Content: "test memory one", Category: "distilled"},
|
||||
}},
|
||||
reportTruncated: false,
|
||||
}
|
||||
code, m, body := recallBody(t, db, `{"query":"test query","limit":5,"namespace":"shared"}`)
|
||||
if code != 200 {
|
||||
t.Fatalf("expected 200, got %d; body=%s", code, body)
|
||||
}
|
||||
if v, ok := m["truncated"]; ok {
|
||||
t.Fatalf("未截断场景键应缺失(省略=未截断);实际 truncated=%v body=%s", v, body)
|
||||
}
|
||||
assertBaseline(t, m, body)
|
||||
}
|
||||
|
||||
// ②b 纯 Go 后端(不实现 SearchTruncationReporter)⇒ 键缺失,零值语义、行为不变
|
||||
func TestRecall_PureGoBackend_NoTruncatedKey(t *testing.T) {
|
||||
db := &fakeLanceDB{searchResults: []models.MemoryRecord{
|
||||
{ID: "mem_1", Content: "test memory one", Category: "distilled"},
|
||||
}}
|
||||
code, m, body := recallBody(t, db, `{"query":"test query","limit":5,"namespace":"shared"}`)
|
||||
if code != 200 {
|
||||
t.Fatalf("expected 200, got %d; body=%s", code, body)
|
||||
}
|
||||
if _, ok := m["truncated"]; ok {
|
||||
t.Fatalf("纯 Go 后端无行数上限,truncated 键必须缺失;body=%s", body)
|
||||
}
|
||||
assertBaseline(t, m, body)
|
||||
}
|
||||
@@ -0,0 +1,327 @@
|
||||
package routes
|
||||
|
||||
// 织忆自检端点(2026-10-26,牧尘批「三档」)
|
||||
//
|
||||
// 背景:织忆相关的定时自检脚本有 ~20 个,取数方式三套并存——
|
||||
// ① 直连生产库 lancedb.connect("/var/lib/memoryweave")(绕过服务、schema 一改就静默腐烂)
|
||||
// ② IPC socket 自定义协议
|
||||
// ③ HTTP /api/v1/{stats,popwatch,metrics/self}
|
||||
// 三套契约对同一个库,schema 变更时分开损坏。典型事故:pattern 自检脚本写死
|
||||
// type='pattern',库里实际是 pattern_stat ⇒ 连续数月静默 0 输出。
|
||||
//
|
||||
// 本端点是**唯一取数契约**:把「读织忆内部状态」收敛成一份结构化 JSON。
|
||||
// 设计边界(不做什么):
|
||||
// - 不做判定/告警/投递 —— 阈值、渠道、评审留在外部 cron(观测者不能住在被观测者体内:
|
||||
// zhiyid 一挂自检就跟着哑,内外同时沉默是更坏的故障)。
|
||||
// - 不跑重活(eval 全量、备份、vacuum)—— 那些抢生产资源,留在 systemd-run 独立 scope。
|
||||
//
|
||||
// level 语义:
|
||||
// - cheap(默认,<50ms):只读内存态/计数,不扫全表。
|
||||
// - full(秒级):额外扫全表算碎片率/冗余率/近 7 天质量/写量分布(原 quality-watch 口径)。
|
||||
//
|
||||
// 兼容铁律:本端点返回的比率一律给「分子/分母/样本是否足够」,缺数据不返回 0
|
||||
// (0 会被下游读成"很好")。沿用 selfoptimize 的 metricNA(-1) 语义。
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"math"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/xiaoxue/memoryweave/internal/distill"
|
||||
"github.com/xiaoxue/memoryweave/internal/selfoptimize"
|
||||
"github.com/xiaoxue/memoryweave/internal/storage"
|
||||
)
|
||||
|
||||
// SelfcheckSchemaVersion 变更返回结构时递增;消费方据此判兼容。
|
||||
const SelfcheckSchemaVersion = "1"
|
||||
|
||||
// 与 zhiyi-quality-watch.py 同口径(该脚本已改为读本端点,口径只此一处)
|
||||
const (
|
||||
selfcheckFragLen = 20 // <20 字符 = 碎片
|
||||
selfcheckDupMinSample = 5000 // 冗余率最小样本(活行),不足只报样本不判率
|
||||
selfcheckScanLimit = 300000
|
||||
)
|
||||
|
||||
type selfcheckQuality struct {
|
||||
TotalRows int `json:"total_rows"`
|
||||
LiveRows int `json:"live_rows"`
|
||||
ScannedRows int `json:"scanned_rows"`
|
||||
ScanTruncated bool `json:"scan_truncated"`
|
||||
Coverage float64 `json:"coverage"` // 扫描行 / store 真实总量;<1 表示结果不完整,禁止据此下结论
|
||||
FragLen int `json:"frag_len"`
|
||||
|
||||
FragRowsAll int `json:"frag_rows_all"`
|
||||
FragRateAll float64 `json:"frag_rate_all"`
|
||||
FragRowsLive int `json:"frag_rows_live"`
|
||||
FragRateLive float64 `json:"frag_rate_live"`
|
||||
|
||||
DupGroups int `json:"dup_groups"`
|
||||
DupExtraRows int `json:"dup_extra_rows"`
|
||||
DupRate float64 `json:"dup_rate"`
|
||||
DupSampleOK bool `json:"dup_sample_ok"`
|
||||
|
||||
NewRows7d int `json:"new_rows_7d"`
|
||||
NewFragRows7d int `json:"new_frag_rows_7d"`
|
||||
NewFragRate7d *float64 `json:"new_frag_rate_7d"` // nil = 近 7 天无新增(无分母)
|
||||
}
|
||||
|
||||
type selfcheckDayCount struct {
|
||||
Date string `json:"date"`
|
||||
Rows int `json:"rows"`
|
||||
Live int `json:"live"`
|
||||
}
|
||||
|
||||
type selfcheckWriteVolume struct {
|
||||
Days []selfcheckDayCount `json:"days"`
|
||||
Today int `json:"today"`
|
||||
Last7d int `json:"last_7d"`
|
||||
Median7d float64 `json:"median_7d"`
|
||||
P25_7d float64 `json:"p25_7d"`
|
||||
}
|
||||
|
||||
// Selfcheck GET /api/v1/selfcheck?level=cheap|full
|
||||
func (a *API) Selfcheck(w http.ResponseWriter, r *http.Request) {
|
||||
start := time.Now()
|
||||
if r.Method != http.MethodGet {
|
||||
respondError(w, http.StatusMethodNotAllowed, "method not allowed")
|
||||
return
|
||||
}
|
||||
level := strings.ToLower(strings.TrimSpace(r.URL.Query().Get("level")))
|
||||
if level == "" {
|
||||
level = "cheap"
|
||||
}
|
||||
if level != "cheap" && level != "full" {
|
||||
respondError(w, http.StatusBadRequest, "level must be cheap|full")
|
||||
return
|
||||
}
|
||||
|
||||
rep := map[string]interface{}{
|
||||
"schema_version": SelfcheckSchemaVersion,
|
||||
"checked_at": time.Now().Format(time.RFC3339),
|
||||
"level": level,
|
||||
}
|
||||
degraded := []string{}
|
||||
|
||||
// ── 存储总量 ───────────────────────────────
|
||||
if st, err := a.LanceDB.Stats(); err != nil {
|
||||
degraded = append(degraded, "store: "+err.Error())
|
||||
} else {
|
||||
total := numOf(st["total_memories"])
|
||||
tomb := numOf(st["tombstone_count"])
|
||||
live := total - tomb
|
||||
if live < 0 {
|
||||
live = 0
|
||||
}
|
||||
store := map[string]interface{}{
|
||||
"backend": st["backend"],
|
||||
"data_dir": st["data_dir"],
|
||||
"total_memories": total,
|
||||
"total_episodes": numOf(st["total_episodes"]),
|
||||
"tombstone_count": tomb,
|
||||
"live_memories": live,
|
||||
}
|
||||
if total > 0 {
|
||||
store["tombstone_rate"] = round4(tomb / total)
|
||||
} else {
|
||||
store["tombstone_rate"] = -1.0 // N/A:空库无分母
|
||||
}
|
||||
rep["store"] = store
|
||||
}
|
||||
|
||||
// ── 图谱 ───────────────────────────────────
|
||||
if a.GraphStore != nil {
|
||||
n, e, d := a.GraphStore.Stats()
|
||||
rep["graph"] = map[string]interface{}{
|
||||
"node_count": n,
|
||||
"edge_count": e,
|
||||
"density": round4(d),
|
||||
}
|
||||
} else {
|
||||
degraded = append(degraded, "graph: store not initialised")
|
||||
}
|
||||
|
||||
// ── pattern 召回的两侧健康(2026-10-26 新增)──────────
|
||||
// 事故背景:`POST /api/v1/patterns/detect` 是短命单发端点,全库 0 cron、0 调用方,
|
||||
// 于是 graph_nodes(type='pattern') 恒 0 行、recall 恒 0 命中,而日志层面完全沉默
|
||||
// (只能靠人翻 7 天日志才发现「实现了 ≠ 出过数」)。这里把「写入端有没有产数」
|
||||
// 变成一次 GET 就能读到的事实:rows=0 ⇒ 生产者坏了或从未接线,别去查读侧 SQL。
|
||||
rep["pattern"] = patternHealth(a)
|
||||
|
||||
// ── 面板指标(反馈/蒸馏/gap/cascade,含 _valid/_samples 语义)──
|
||||
rep["dashboard"] = selfoptimize.Dash.Metrics()
|
||||
|
||||
// ── 整合链路(内存态,<1ms)────────────────
|
||||
tracker := GetConsolTracker()
|
||||
consolStatus, consolFails, sinceSuccess, lastErr := tracker.HealthStatus()
|
||||
_, consolSkips, lastSkip, _ := tracker.Snapshot()
|
||||
rep["consolidate"] = map[string]interface{}{
|
||||
"status": consolStatus,
|
||||
"consec_failures": consolFails,
|
||||
"skipped": consolSkips,
|
||||
"last_skip": lastSkip,
|
||||
"last_success_age": sinceSuccess.String(),
|
||||
"last_error": lastErr,
|
||||
}
|
||||
|
||||
// ── 缓存 / FTS ─────────────────────────────
|
||||
cache := map[string]interface{}{"fts_index": storage.FtsIndexStats()}
|
||||
if storage.SearchCacheInstance != nil {
|
||||
cache["search_cache"] = storage.SearchCacheInstance.Stats()
|
||||
}
|
||||
rep["cache"] = cache
|
||||
|
||||
// ── 空壳拦截器(t_e11207f8 取证)───────────
|
||||
checks, blocks, loaded, rulesPath := distill.PayloadGuardStats()
|
||||
rep["payload_guard"] = map[string]interface{}{
|
||||
"checks": checks,
|
||||
"blocks": blocks,
|
||||
"loaded": loaded,
|
||||
"rules_path": rulesPath,
|
||||
}
|
||||
|
||||
// ── 全量质量段(level=full):读卫星快照,不在本进程扫全表 ────────
|
||||
// 设计:全表扫描是重活(4 万行 × 内容),且 daemon 还有 recall 要跑。
|
||||
// 取数由仓库内 tools/selfcheck_full.py 完成(列投影只读,1.7s / 250MB,独立进程),
|
||||
// 写 ~/.hermes/state/zhiyi-quality.json;本端点只读快照并标注新鲜度。
|
||||
// 这样"取数"只有一处实现(跟 schema 同仓),"判定/告警"仍在外部 cron。
|
||||
if level == "full" {
|
||||
snap, err := loadQualitySnapshot()
|
||||
if err != nil {
|
||||
degraded = append(degraded, "quality: "+err.Error())
|
||||
} else {
|
||||
rep["quality"] = snap.Quality
|
||||
rep["write_volume"] = snap.WriteVolume
|
||||
rep["quality_as_of"] = snap.GeneratedAt
|
||||
rep["quality_age_s"] = snap.AgeSeconds()
|
||||
rep["quality_stale"] = snap.AgeSeconds() > qualityStaleAfter
|
||||
}
|
||||
}
|
||||
|
||||
if len(degraded) > 0 {
|
||||
rep["degraded"] = degraded
|
||||
}
|
||||
rep["duration_ms"] = time.Since(start).Milliseconds()
|
||||
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(http.StatusOK)
|
||||
json.NewEncoder(w).Encode(rep)
|
||||
}
|
||||
|
||||
// qualityStaleAfter:快照超过这个时长未刷新 ⇒ 标记 stale(消费方不得据此下结论)。
|
||||
const qualityStaleAfter = 6 * 3600
|
||||
|
||||
// qualitySnapshot 对应仓库内 tools/selfcheck_full.py 的产出。
|
||||
// 字段名与 python 侧严格一致(schema_version=1)。
|
||||
type qualitySnapshot struct {
|
||||
SchemaVersion int `json:"schema_version"`
|
||||
GeneratedAt string `json:"generated_at"`
|
||||
Source struct {
|
||||
DB string `json:"db"`
|
||||
Table string `json:"table"`
|
||||
Mode string `json:"mode"`
|
||||
} `json:"source"`
|
||||
DurationMS int `json:"duration_ms"`
|
||||
Quality json.RawMessage `json:"quality"`
|
||||
WriteVolume json.RawMessage `json:"write_volume"`
|
||||
fileModTime time.Time
|
||||
}
|
||||
|
||||
func (q *qualitySnapshot) AgeSeconds() int64 {
|
||||
if q.fileModTime.IsZero() {
|
||||
return -1
|
||||
}
|
||||
return int64(time.Since(q.fileModTime).Seconds())
|
||||
}
|
||||
|
||||
// QualitySnapshotPath 允许用 env 覆盖(测试/灰度)。
|
||||
func QualitySnapshotPath() string {
|
||||
if p := strings.TrimSpace(os.Getenv("ZHIYI_QUALITY_SNAPSHOT")); p != "" {
|
||||
return p
|
||||
}
|
||||
home, _ := os.UserHomeDir()
|
||||
return filepath.Join(home, ".hermes", "state", "zhiyi-quality.json")
|
||||
}
|
||||
|
||||
func loadQualitySnapshot() (*qualitySnapshot, error) {
|
||||
p := QualitySnapshotPath()
|
||||
fi, err := os.Stat(p)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("快照不存在(先跑 tools/selfcheck_full.py):%w", err)
|
||||
}
|
||||
raw, err := os.ReadFile(p)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var snap qualitySnapshot
|
||||
if err := json.Unmarshal(raw, &snap); err != nil {
|
||||
return nil, fmt.Errorf("快照解析失败: %w", err)
|
||||
}
|
||||
snap.fileModTime = fi.ModTime()
|
||||
return &snap, nil
|
||||
}
|
||||
|
||||
// numOf 把 stats 里可能是 float64/int/json.Number 的数值统一成 float64。
|
||||
func numOf(v interface{}) float64 {
|
||||
switch x := v.(type) {
|
||||
case float64:
|
||||
return x
|
||||
case float32:
|
||||
return float64(x)
|
||||
case int:
|
||||
return float64(x)
|
||||
case int64:
|
||||
return float64(x)
|
||||
case json.Number:
|
||||
f, _ := x.Float64()
|
||||
return f
|
||||
default:
|
||||
return 0
|
||||
}
|
||||
}
|
||||
|
||||
func round4(f float64) float64 {
|
||||
if math.IsNaN(f) || math.IsInf(f, 0) {
|
||||
return -1
|
||||
}
|
||||
return math.Round(f*10000) / 10000
|
||||
}
|
||||
|
||||
// patternHealth 一眼判定 pattern 召回是「有生产者」还是「恒 0」。
|
||||
//
|
||||
// 为什么需要它:2026-10-26 排查发现 pattern 召回 0 命中,真根因是写入端从未产数
|
||||
// (detect 端点无 cron、无调用方),但表面症状看起来像读侧 SQL bug —— 白查一轮方向。
|
||||
// 判据三类:
|
||||
//
|
||||
// producing 有行、有 id、有生产时间 → 读侧再 0 命中就该查 Match 阈值/properties;
|
||||
// zero_output 0 行 → 生产端问题,别去动读侧 SQL;
|
||||
// no_id 有行但 id 全空 → store.ListPatterns 会全 skip(存量冻结的典型形态)。
|
||||
func patternHealth(a *API) map[string]interface{} {
|
||||
out := map[string]interface{}{
|
||||
"rows": -1, // -1 = 未知,别当 0 用
|
||||
"rows_with_id": -1,
|
||||
"last_produced_at": "",
|
||||
"verdict": "unknown",
|
||||
}
|
||||
if a == nil || a.GraphStore == nil {
|
||||
out["verdict"] = "no_graph_store"
|
||||
return out
|
||||
}
|
||||
rows, withID, last := a.GraphStore.CountNodesByType("pattern")
|
||||
out["rows"], out["rows_with_id"], out["last_produced_at"] = rows, withID, last
|
||||
switch {
|
||||
case rows == 0:
|
||||
out["verdict"] = "zero_output"
|
||||
case withID == 0:
|
||||
out["verdict"] = "no_id"
|
||||
case last == "":
|
||||
out["verdict"] = "unknown_age"
|
||||
default:
|
||||
out["verdict"] = "producing"
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,99 @@
|
||||
package routes
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// 快照契约测试:/api/v1/selfcheck?level=full 依赖 tools/selfcheck_full.py 的产出格式。
|
||||
// 这里钉住 schema_version 与关键字段,防止 python 侧改字段名后 Go 侧静默返回空。
|
||||
func TestLoadQualitySnapshot(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
p := filepath.Join(dir, "zhiyi-quality.json")
|
||||
body := `{
|
||||
"schema_version": 1,
|
||||
"generated_at": "2026-10-26T13:20:00+0800",
|
||||
"source": {"db": "/var/lib/memoryweave", "table": "memories", "mode": "read_only_projection"},
|
||||
"duration_ms": 1680,
|
||||
"quality": {
|
||||
"frag_len": 20, "total_rows": 39957, "live_rows": 39836,
|
||||
"frag_rows_all": 313, "frag_rate_all": 0.0078,
|
||||
"frag_rows_live": 313, "frag_rate_live": 0.0079,
|
||||
"dup_groups": 34, "dup_extra_rows": 103, "dup_rate": 0.0026,
|
||||
"new_rows_7d": 1176, "new_frag_rows_7d": 35, "new_frag_rate_7d": 0.0298
|
||||
},
|
||||
"write_volume": {
|
||||
"days": [{"date": "2026-10-25", "rows": 1473, "live": 1471}],
|
||||
"today": 332, "today_date": "2026-10-26", "last_7d": 7706, "nonzero_days": 60
|
||||
}
|
||||
}`
|
||||
if err := os.WriteFile(p, []byte(body), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Setenv("ZHIYI_QUALITY_SNAPSHOT", p)
|
||||
|
||||
snap, err := loadQualitySnapshot()
|
||||
if err != nil {
|
||||
t.Fatalf("loadQualitySnapshot: %v", err)
|
||||
}
|
||||
if snap.SchemaVersion != 1 {
|
||||
t.Fatalf("schema_version=%d want 1", snap.SchemaVersion)
|
||||
}
|
||||
|
||||
var q struct {
|
||||
TotalRows int `json:"total_rows"`
|
||||
LiveRows int `json:"live_rows"`
|
||||
DupRate float64 `json:"dup_rate"`
|
||||
FragRowsAll int `json:"frag_rows_all"`
|
||||
}
|
||||
if err := json.Unmarshal(snap.Quality, &q); err != nil {
|
||||
t.Fatalf("quality 段解析失败: %v", err)
|
||||
}
|
||||
if q.TotalRows != 39957 || q.LiveRows != 39836 {
|
||||
t.Fatalf("quality 口径不符: %+v", q)
|
||||
}
|
||||
if q.FragRowsAll != 313 || q.DupRate != 0.0026 {
|
||||
t.Fatalf("quality 数值不符: %+v", q)
|
||||
}
|
||||
|
||||
var wv struct {
|
||||
Today int `json:"today"`
|
||||
Days []struct {
|
||||
Date string `json:"date"`
|
||||
Live int `json:"live"`
|
||||
} `json:"days"`
|
||||
}
|
||||
if err := json.Unmarshal(snap.WriteVolume, &wv); err != nil {
|
||||
t.Fatalf("write_volume 段解析失败: %v", err)
|
||||
}
|
||||
if wv.Today != 332 || len(wv.Days) != 1 || wv.Days[0].Live != 1471 {
|
||||
t.Fatalf("write_volume 不符: %+v", wv)
|
||||
}
|
||||
|
||||
// 新鲜度:刚写的文件 age 应 < 10s;人为改旧 → stale 判定为真
|
||||
if age := snap.AgeSeconds(); age < 0 || age > 10 {
|
||||
t.Fatalf("AgeSeconds=%d,本应接近 0", age)
|
||||
}
|
||||
old := time.Now().Add(-8 * time.Hour)
|
||||
if err := os.Chtimes(p, old, old); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
snap2, err := loadQualitySnapshot()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if snap2.AgeSeconds() <= qualityStaleAfter {
|
||||
t.Fatalf("8h 前的快照应判 stale(>%d 秒),得到 %d", qualityStaleAfter, snap2.AgeSeconds())
|
||||
}
|
||||
}
|
||||
|
||||
// 快照缺失 ⇒ 必须报错(由 handler 转成 degraded),不能静默返回零值。
|
||||
func TestLoadQualitySnapshotMissing(t *testing.T) {
|
||||
t.Setenv("ZHIYI_QUALITY_SNAPSHOT", filepath.Join(t.TempDir(), "nope.json"))
|
||||
if _, err := loadQualitySnapshot(); err == nil {
|
||||
t.Fatal("缺失快照应返回 error")
|
||||
}
|
||||
}
|
||||
@@ -1,8 +1,32 @@
|
||||
// 织忆 MemoryWeave — 路由层测试隔离(t_e5997ca4 起)
|
||||
// 织忆 MemoryWeave — routes 包测试隔离(两个护栏合并 · t_e5997ca4 + t_b7a8ed9c)
|
||||
//
|
||||
// 单元测试不得写到生产落盘点。selfoptimize 包已有同款 TestMain 先例
|
||||
// (曾实测:整套测试跑完把 /var/lib/memoryweave/dashboard.json 写成测试数字)。
|
||||
// ⚠️ 本文件是**两个 TestMain 的合并体**。两卡分支各自加了一个 TestMain:
|
||||
// · t_e5997ca4 → testmain_test.go(落盘点护栏)
|
||||
// · t_b7a8ed9c → scorecache_guard_test.go(打分缓存护栏)
|
||||
// 在各自分支上互不干扰,合并到 main 后同包两个 TestMain **编译冲突**
|
||||
// (multiple definitions of TestMain)⇒ 必须合进一个文件。
|
||||
// (此点由 t_b7a8ed9c 在 scorecache_guard_test.go 顶部预先标注;小怡合并时未读文件
|
||||
// 而漏看,由全套测试的 setup failed 拦下 —— 见 worklog 教训。)
|
||||
//
|
||||
// 护栏①:单元测试不得写到生产落盘点(t_e5997ca4)
|
||||
// ---------------------------------------------------------------
|
||||
// selfoptimize 包已有同款 TestMain 先例(曾实测:整套测试跑完把
|
||||
// /var/lib/memoryweave/dashboard.json 写成测试数字)。
|
||||
// gap close/repair 端点会触发 Dash.persist()(本地 JSON 兜底)⇒ 统一重定向到临时目录。
|
||||
//
|
||||
// 护栏②:默认关闭 distill 打分缓存(t_b7a8ed9c)
|
||||
// ---------------------------------------------------------------
|
||||
// 背景:distill 打分记忆化(0dc571d)+ Redis L2(36db9bb,键 zhiyi:distill-score:*,
|
||||
// TTL 168h)后,**本包 E2E 测试(zz_commit_conflict_gate_test.go)会读写生产 Redis**:
|
||||
// 第一次 run L2 miss → 真调 mock LLM 并把「测试内容哈希 → 结果」写进生产;
|
||||
// 7 天内后续 run L2 hit → LLM 被静默短路 → 数 LLM 调用次数的断言(base+1)必挂。
|
||||
// 实测 2026-10-17:TestCommitConflictGate_E2E 16.5s 固定 FAIL,
|
||||
// 「expected >= 1 LLM distill calls, got 0」。
|
||||
//
|
||||
// 治法(对齐项目门禁规矩③:加缓存必须给该包 TestMain 加关闭开关):
|
||||
// Setenv ZHIYI_SCORE_CACHE=0 —— distill.getScoreCache 的 Once 闭包**活读 env**
|
||||
// (包级变量在 init 时已定型,跨包设不了),本进程内缓存整体禁用:
|
||||
// 不读 L2(LLM 真被调用,计数断言有覆盖)、不写 L2(不再污染生产)。
|
||||
package routes
|
||||
|
||||
import (
|
||||
@@ -12,12 +36,16 @@ import (
|
||||
)
|
||||
|
||||
func TestMain(m *testing.M) {
|
||||
// 护栏①:落盘点重定向到临时目录(宁可测试直接失败,也不写生产落盘点)
|
||||
dir, err := os.MkdirTemp("", "routes-test-dash-*")
|
||||
if err != nil {
|
||||
// 宁可测试直接失败,也不写生产落盘点
|
||||
panic(err)
|
||||
}
|
||||
_ = os.Setenv("ZHIYI_DASHBOARD_PATH", filepath.Join(dir, "dashboard.json"))
|
||||
|
||||
// 护栏②:关闭 distill 打分缓存(不读/不写生产 Redis L2)
|
||||
_ = os.Setenv("ZHIYI_SCORE_CACHE", "0")
|
||||
|
||||
code := m.Run()
|
||||
_ = os.RemoveAll(dir)
|
||||
os.Exit(code)
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
// 织忆 MemoryWeave — 召回链路追踪端点(P0-1,2026-10-26)
|
||||
//
|
||||
// GET /api/v1/trace/<trace_id>
|
||||
//
|
||||
// 返回该次召回的:各阶段耗时(encode/coarse_search/fts_recall/bm25/rerank_fusion/mmr_select/
|
||||
// graph_expand/dedupe)、各通道贡献条数(channels)、每条结果的来源归因(items)、
|
||||
// status/partial_reason、scope、cache_hit/truncated。
|
||||
//
|
||||
// 数据来源:内存环形缓冲(最近 2000 条)+ JSONL 落盘(/var/lib/memoryweave/traces/),
|
||||
// 服务重启后仍可按 trace_id 回溯(落盘扫描)。
|
||||
package routes
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/xiaoxue/memoryweave/internal/storage"
|
||||
)
|
||||
|
||||
// ── P0-3 召回硬超时:取消函数类型 + context.WithTimeout 包装 ──────────────
|
||||
// core.go Recall() 在 req.DeadlineMS>0 时需要带类型标记的取消函数,以便 defer 取消。
|
||||
// 2026-10-26 (t_e98cb1d6):此处补齐使 routes 包可编译(否则 routes 全包不可构建,
|
||||
// capRecallResults 的单测无法跑通)。这是缺失的声明,不是功能实现;P0-3 的 trace 完整
|
||||
// 接线(tr.Stage 埋点、tr 落盘)仍待 P0-3 本任务补齐。
|
||||
type ctxDeadlineFunc context.CancelFunc
|
||||
|
||||
func contextWithTimeout(ctx context.Context, timeout time.Duration) (context.Context, ctxDeadlineFunc) {
|
||||
c, cancel := context.WithTimeout(ctx, timeout)
|
||||
return c, ctxDeadlineFunc(cancel)
|
||||
}
|
||||
|
||||
// Trace 查询单条召回链路。路径形如 /api/v1/trace/tr_xxxx。
|
||||
func (a *API) Trace(w http.ResponseWriter, r *http.Request) {
|
||||
id := strings.TrimSpace(strings.TrimPrefix(r.URL.Path, "/api/v1/trace/"))
|
||||
if id == "" {
|
||||
respondErrorSchema(w, 400, "trace_id_required", "用法:GET /api/v1/trace/<trace_id>", 0)
|
||||
return
|
||||
}
|
||||
tr, ok := storage.GetTrace(id)
|
||||
if !ok {
|
||||
// 404 与「trace 过期/被环形缓冲覆盖」同义:调用方据此走「重新召回」而不是当成系统故障。
|
||||
respondErrorSchema(w, 404, "trace_not_found",
|
||||
"trace 不存在或已被环形缓冲覆盖(内存保留最近 2000 条,落盘按天分文件)", 0)
|
||||
return
|
||||
}
|
||||
respond(w, 200, tr)
|
||||
}
|
||||
@@ -80,10 +80,14 @@ func (m *mockLance) IncrementUseful(id string) error { r
|
||||
func (m *mockLance) IncrementNotUseful(id string) error { return nil }
|
||||
|
||||
// GA-P0-1 (t_718d8707):拆分字段
|
||||
func (m *mockLance) IncrementHumanNotUseful(id string) error { return nil }
|
||||
func (m *mockLance) AppendHitEvent(memoryID, outcome, reason string) error { return nil }
|
||||
func (m *mockLance) CountHitEvents(memoryID, outcome string, since time.Time) (int, error) { return 0, nil }
|
||||
func (m *mockLance) GetHitEvents(memoryID string, limit int) ([]models.HitEvent, error) { return nil, nil }
|
||||
func (m *mockLance) IncrementHumanNotUseful(id string) error { return nil }
|
||||
func (m *mockLance) AppendHitEvent(memoryID, outcome, reason string) error { return nil }
|
||||
func (m *mockLance) CountHitEvents(memoryID, outcome string, since time.Time) (int, error) {
|
||||
return 0, nil
|
||||
}
|
||||
func (m *mockLance) GetHitEvents(memoryID string, limit int) ([]models.HitEvent, error) {
|
||||
return nil, nil
|
||||
}
|
||||
func (m *mockLance) IncrementDedupDownweight(id string) error { return nil }
|
||||
func (m *mockLance) UpdateMemoryContent(id, newContent, source string) error { return nil }
|
||||
func (m *mockLance) InsertMemory(rec models.MemoryRecord) error { return nil }
|
||||
@@ -116,6 +120,7 @@ func (mockGraph) ListNodesByType(nodeType string, namespace string) []map[string
|
||||
}
|
||||
func (mockGraph) ListNodes(namespace string) []map[string]interface{} { return nil }
|
||||
func (mockGraph) Stats() (int, int, float64) { return 0, 0, 0 }
|
||||
func (mockGraph) CountNodesByType(string) (int, int, string) { return 0, 0, "" }
|
||||
func (mockGraph) Prune(minWeight float64) {}
|
||||
func (mockGraph) ExpandFromResults(results []models.RecallResult, namespace string, maxHops int) []models.RecallResult {
|
||||
return results
|
||||
|
||||
@@ -93,7 +93,13 @@ func NewServer() http.Handler {
|
||||
// 冲突检测器(E4.1: Commit 时矛盾检测依赖此实例)
|
||||
// 2026-09-20:改用带持久化的构造 —— 此前 active 是纯内存 map,zhiyid 重启即丢
|
||||
// (实测 17:17 重启后 pending 冲突全部消失)。落盘失败不致命,降级为内存模式并记日志。
|
||||
conflictDetector, cdErr := governance.NewPersistentConflictDetector("/var/lib/memoryweave/conflicts.json")
|
||||
// 2026-10-18 (t_23b6a940):ZHIYI_CONFLICTS_DIR 覆盖落盘目录(副本/评估场景隔离
|
||||
// telemetry 写);未设 → 生产默认路径(行为不变)
|
||||
confDir := os.Getenv("ZHIYI_CONFLICTS_DIR")
|
||||
if confDir == "" {
|
||||
confDir = "/var/lib/memoryweave"
|
||||
}
|
||||
conflictDetector, cdErr := governance.NewPersistentConflictDetector(filepath.Join(confDir, "conflicts.json"), filepath.Join(confDir, "conflict-seen.json"))
|
||||
if cdErr != nil {
|
||||
log.Printf("[conflict] 持久化加载失败,降级为内存模式: %v", cdErr)
|
||||
}
|
||||
@@ -133,6 +139,17 @@ func NewServer() http.Handler {
|
||||
}
|
||||
}()
|
||||
|
||||
// t_cf680fad:启动即 drain durable pending-softdelete 队列。
|
||||
// 上一轮(尤其重启窗口 / 崩掉那轮)因 MVCC 写风暴耗尽而被入队的失败软删,
|
||||
// 此刻风暴已退、sidecar 刚就绪,重放必落盘 —— 这是「不修则每次重启窗口都留新残留」的根治。
|
||||
go func() {
|
||||
time.Sleep(2 * time.Second) // 等 sidecar 完全就绪(比 metrics 的 1s 稍晚,避免抢首提交)
|
||||
if n := storage.PendingSoftDeleteCount(); n > 0 {
|
||||
log.Printf("[pending-softdelete] 启动发现 %d 条待重放,开始 drain", n)
|
||||
storage.DrainPendingSoftDeletes(ldb)
|
||||
}
|
||||
}()
|
||||
|
||||
// 图谱(带 Navigate 结果缓存:TTL 5min,容量 500,热点实体缓存命中加速)
|
||||
// 已在上面初始化
|
||||
graphUpdater := governance.NewAutoGraphUpdater(graphStore)
|
||||
@@ -170,7 +187,8 @@ func NewServer() http.Handler {
|
||||
|
||||
// ─── t_d32e5f8a L2 模式检测(MemOS 血统)─────────────────────────
|
||||
// 复活 L1→L2 pattern 蒸馏(原 Python daemon main_loop 退役后停摆):
|
||||
// pattern 节点存 graph_nodes(type='pattern'),与现有 85 存量同表,不另开新表。
|
||||
// pattern 节点存 graph_nodes(type='pattern'),与旧 daemon 残留的 85 行
|
||||
// type='pattern_stat' 同表但不同类,不另开新表。
|
||||
// modernc sqlite 短命单发开/关(UpsertPatterns/ListPatterns 开一次用一次即关),
|
||||
// 不持长连接、不碰 state.db —— 规避当年退役根因。
|
||||
// 召回注入 env 门控 ZHIYI_PATTERN_RECALL,默认 0 ⇒ 不注册 Provider,行为零变化。
|
||||
@@ -221,6 +239,14 @@ func NewServer() http.Handler {
|
||||
}
|
||||
distillEngine := distill.NewEngine(llmEndpoint, llmModel, llmAPIKey)
|
||||
routes.DistillEngineRef = distillEngine
|
||||
// t_e11207f8 治源①:加载共享「空壳正文」判据(zhiyi_payload_guard_rules.json,Python/Go 同源)。
|
||||
// 失败 ⇒ fail-open(distill 照跑、不拦空壳、不崩),/api/v1/payload-guard/stats 暴露 loaded=false。
|
||||
if ok, lerr := distill.LoadPayloadGuard(""); ok {
|
||||
_, _, _, rp := distill.PayloadGuardStats()
|
||||
log.Printf("[distill][payload-guard] 空壳判据已加载 rules=%s", rp)
|
||||
} else {
|
||||
log.Printf("[distill][payload-guard] ⚠️ 共享判据未加载(fail-open,不拦空壳): %v", lerr)
|
||||
}
|
||||
// 蒸馏完成 → 自动图谱更新 + 冲突检测 + 被动验证 + 回写内存
|
||||
distill.OnDistillComplete = func(input distill.DistillInput, result distill.DistillResult) {
|
||||
entityNames := make([]string, len(result.Entities))
|
||||
@@ -262,6 +288,13 @@ func NewServer() http.Handler {
|
||||
// 这里再脱一次为幂等保险——防止任何中间层注入明文密钥。事故里泄漏进
|
||||
// 镜像的记忆正是从这条路径入库的)。
|
||||
fact = redact.RedactSecrets(fact)
|
||||
// t_e11207f8 治源①:空壳拒写(单一判据 = 共享 zhiyi_payload_guard_rules.json,与 Python 同源)。
|
||||
// 蒸馏 LLM 偶发把「只有溯源指针/占位/纯日期、无知识载荷」的碎片当 fact(如 `证据:文件:…|日期:…`),
|
||||
// 长度/实体启发式(IsQualityFact)抓不住 ⇒ 落库前用 payload-guard 拦下(未加载时 fail-open)。
|
||||
if distill.ShellCheck(fact) {
|
||||
log.Printf("[distill][payload-guard] 拒写空壳蒸馏 fact episode=%s fact=%.40s", input.EpisodeID, fact)
|
||||
continue
|
||||
}
|
||||
// 查重: 同 namespace 已存在**语义等价**记忆 → 跳过
|
||||
// 必须用 cosine 近重复判据(≥0.98),不能只比字符串相等——
|
||||
// 否则 "牧尘在招商局工作" 与 "牧尘在招商局工作。"(多一个句号)
|
||||
@@ -487,6 +520,18 @@ func NewServer() http.Handler {
|
||||
|
||||
mux.HandleFunc("/health", routes.HandleHealth)
|
||||
mux.HandleFunc("/api/v1/health", routes.HandleHealth)
|
||||
// t_e11207f8 治源①:空壳拦截器「确实跑过」的取证端点。
|
||||
// checks=每个蒸馏 fact 判一次的心跳计数(>0 即拦截器活着);blocks=实际拒写空壳条数;
|
||||
// loaded=false 表示共享判据没加载上(fail-open,需排查 ZHIYI_PAYLOAD_GUARD_RULES 路径)。
|
||||
mux.HandleFunc("/api/v1/payload-guard/stats", func(w http.ResponseWriter, r *http.Request) {
|
||||
checks, blocks, loaded, rp := distill.PayloadGuardStats()
|
||||
respondJSON(w, 200, map[string]interface{}{
|
||||
"checks": checks,
|
||||
"blocks": blocks,
|
||||
"loaded": loaded,
|
||||
"rules_path": rp,
|
||||
})
|
||||
})
|
||||
// 图谱缓存统计 + 搜索缓存统计(t_1c12e000:并入 search_cache 段,
|
||||
// 暴露 hits/misses/invalidated/size/ttl,写频对命中率的影响可观测)
|
||||
mux.HandleFunc("/api/v1/cache/stats", func(w http.ResponseWriter, r *http.Request) {
|
||||
@@ -558,6 +603,10 @@ func NewServer() http.Handler {
|
||||
})
|
||||
mux.HandleFunc("/api/v1/recall", api.Recall)
|
||||
mux.HandleFunc("/api/v1/recall/debug", api.RecallDebug)
|
||||
// 2026-10-26 (P0-1 t_trace_p0):召回链路追踪回溯(trace_id 贯穿各阶段/各通道)
|
||||
mux.HandleFunc("/api/v1/trace/", api.Trace)
|
||||
// t_23b6a940 AC7 热门度看门狗快照(report-only,传感器端读它转既有告警通道)
|
||||
mux.HandleFunc("/api/v1/popwatch", api.PopWatch)
|
||||
// 2026-10-13 (t_e9a4d18f P2) 常设答案直读/写入
|
||||
mux.HandleFunc("/api/v1/standing-answers", routes.HandleStandingAnswers)
|
||||
// ⑤使用追踪(t_5aa4dcac):上报 + 手动触发。env 门控 ZHIYI_USAGE_TRACK,默认关。
|
||||
@@ -567,6 +616,10 @@ func NewServer() http.Handler {
|
||||
// 反馈统一走 /api/v1/feedback/{useful,not-useful,deprecate,correct}(feedback.go,写拆分后新字段)。
|
||||
mux.HandleFunc("/api/v1/bootstrap", api.Bootstrap)
|
||||
mux.HandleFunc("/api/v1/stats", api.Stats)
|
||||
// 2026-10-26(牧尘批「三档」):自检统一取数端点。
|
||||
// 唯一契约 —— 自检脚本一律读这里,禁止再 lancedb.connect() 直连生产库。
|
||||
// ?level=cheap(默认,内存态计数)| full(+读 tools/selfcheck_full.py 的全量快照:碎片/冗余/近7天质量/写量)
|
||||
mux.HandleFunc("/api/v1/selfcheck", api.Selfcheck)
|
||||
mux.HandleFunc("/api/v1/batch-commit", api.BatchCommit)
|
||||
// 主症A(t_fa6c9104):显式写入直落 memories 表(嵌入 + 立刻可召回)。
|
||||
// 纯新增路由:commit/recall/蒸馏路径零改动;旧服务端 404 时插件自动回落 commit。
|
||||
@@ -1792,6 +1845,11 @@ func NewServer() http.Handler {
|
||||
ticker := time.NewTicker(30 * time.Second)
|
||||
defer ticker.Stop()
|
||||
for range ticker.C {
|
||||
// t_cf680fad:每个 tick 先 drain durable pending-softdelete 队列(风暴退去后的
|
||||
// 相对静默窗口重放,必落盘)。队列空 = 廉价 no-op,不打日志;仅在有待处理时出声。
|
||||
if n := storage.PendingSoftDeleteCount(); n > 0 {
|
||||
storage.DrainPendingSoftDeletes(ldb)
|
||||
}
|
||||
for _, t := range []struct{ id, action string }{
|
||||
{"t_distill", "distill"},
|
||||
{"t_merge", "merge"},
|
||||
|
||||
@@ -0,0 +1,181 @@
|
||||
// 织忆 MemoryWeave —— 写时四操作的 LLM 判定器(t_writeops_llm_judge,2026-10-25)
|
||||
//
|
||||
// 背景(离线回放实测,不是推测):
|
||||
// · 规则基 defaultJudge 用 ZHIYI_WRITE_SIM_UPDATE=0.70 判 UPDATE,影子日志 11 天 17179 条里
|
||||
// 14435 条(84%)被判「新事实应覆盖旧事实」;人工抽检 15 条低相似度样本,15/15 不是真覆盖
|
||||
// (「用户查看知乎链接」vs「用户查看微信文章」都判成覆盖)。
|
||||
// · 80 条分层回放(各相似度档 20 条,LLM 三分类):REPLACE 率 0.95+ 档 25%、0.90-0.95 档 35%、
|
||||
// 0.85-0.90 档 35%、0.70-0.85 档 20% —— 与相似度**不单调**,说明相似度只反映「话题相关」,
|
||||
// 不反映「是否同一属性的新旧值」。低档 LLM 同样会被话题带偏(0.70-0.85 档 REPLACE 精度约 25%)。
|
||||
//
|
||||
// 结论与实现:
|
||||
// · 只有 sim >= ZHIYI_WRITE_JUDGE_SIM(默认 0.95)才调 LLM 判三分类;低于该档直接 ADD(不调 LLM,
|
||||
// 省调用 + 该档不可信)。
|
||||
// · REPLACE→UPDATE / DUPLICATE→NOOP / KEEP→ADD;判不准与调用失败一律兜底 ADD(宁可不覆盖)。
|
||||
// · 默认 judge 模式仍是 rule(零行为变化);apply 模式强制要求 llm(见 write_ops.go 硬保护)。
|
||||
package distill
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
const llmJudgeSystemPrompt = `你是记忆库的写入判定器。给你两条记忆:A=库里已有的旧事实,B=刚提取出的新事实。
|
||||
判断 B 与 A 的关系,只回一个词,不要解释:
|
||||
REPLACE = 同一主体的同一属性,B 是 A 的更新值/修正值(B 生效后 A 就过时了)
|
||||
DUPLICATE= B 与 A 语义等价,B 没有新增信息
|
||||
KEEP = B 补充/细化/扩展 A,或讲不同侧面/不同对象/不同阶段;两条都要保留
|
||||
判不准时回 KEEP。`
|
||||
|
||||
// llmJudgeSimThreshold 低于此相似度直接 ADD(实测该档 LLM 判定精度不足,且省调用)。
|
||||
const llmJudgeSimThresholdDefault = 0.95
|
||||
|
||||
// llmJudgeCall 可注入的 LLM 调用(单测替换,生产走 HTTP)。返回模型输出的原始文本。
|
||||
var llmJudgeCall = callLLMJudgeModel
|
||||
|
||||
// LLMJudgeMode 当前 judge 模式:rule(默认)| llm。
|
||||
func LLMJudgeMode() string {
|
||||
m := strings.ToLower(strings.TrimSpace(os.Getenv("ZHIYI_WRITE_JUDGE")))
|
||||
if m == "llm" {
|
||||
return "llm"
|
||||
}
|
||||
return "rule"
|
||||
}
|
||||
|
||||
// llmJudgeSimThreshold 判定相似度门槛(env 可调)。
|
||||
func llmJudgeSimThreshold() float64 {
|
||||
if v := strings.TrimSpace(os.Getenv("ZHIYI_WRITE_JUDGE_SIM")); v != "" {
|
||||
if f, err := strconv.ParseFloat(v, 64); err == nil && f > 0 && f <= 1 {
|
||||
return f
|
||||
}
|
||||
}
|
||||
return llmJudgeSimThresholdDefault
|
||||
}
|
||||
|
||||
// LLMJudge 用 LLM 判「B 是否应取代 A」。
|
||||
// 返回的 WriteVerdict 与 defaultJudge 同构,调用方无需区分。
|
||||
func LLMJudge(candidate string, similar []SimilarMemory) WriteVerdict {
|
||||
v := WriteVerdict{
|
||||
Op: OpAdd,
|
||||
Candidate: candidate,
|
||||
Reason: "llm:无相似候选 → 新建",
|
||||
Timestamp: time.Now().Format(time.RFC3339),
|
||||
}
|
||||
if len(similar) == 0 {
|
||||
return v
|
||||
}
|
||||
top := similar[0] // 调用方已按相似度排序(TopN)
|
||||
v.Similarity = top.Similarity
|
||||
|
||||
// 归一化完全等价 → NOOP,不调 LLM
|
||||
if NormalizeContent(top.Content) == NormalizeContent(candidate) {
|
||||
v.Op = OpNoop
|
||||
v.TargetID = top.ID
|
||||
v.Reason = "llm:归一化等价 → 丢弃"
|
||||
return v
|
||||
}
|
||||
|
||||
// 相似度不足 → ADD(实测低档 LLM 判定不可信,且省钱)
|
||||
if top.Similarity < llmJudgeSimThreshold() {
|
||||
v.Reason = fmt.Sprintf("llm:sim %.3f < %.2f → 不判,新建", top.Similarity, llmJudgeSimThreshold())
|
||||
return v
|
||||
}
|
||||
|
||||
raw, err := llmJudgeCall(candidate, top.Content)
|
||||
if err != nil {
|
||||
// 调用失败兜底 ADD:宁可不覆盖(漏覆盖可事后补,误覆盖会丢事实)
|
||||
v.Reason = "llm:调用失败兜底新建"
|
||||
log.Printf("[write-ops][llm-judge] 调用失败,兜底 ADD: %v candidate=%.40s", err, candidate)
|
||||
return v
|
||||
}
|
||||
verdict := parseLLMJudgeVerdict(raw)
|
||||
switch verdict {
|
||||
case "REPLACE":
|
||||
v.Op = OpUpdate
|
||||
v.TargetID = top.ID
|
||||
v.Reason = "llm:同一属性更新 → 旧条标 superseded_by"
|
||||
case "DUPLICATE":
|
||||
v.Op = OpNoop
|
||||
v.TargetID = top.ID
|
||||
v.Reason = "llm:语义等价 → 丢弃"
|
||||
default: // KEEP / 无法解析
|
||||
v.Reason = "llm:保留(补充/不同侧面)→ 新建"
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
// parseLLMJudgeVerdict 从模型输出里取三分类词;取不到返回 KEEP。
|
||||
func parseLLMJudgeVerdict(raw string) string {
|
||||
u := strings.ToUpper(raw)
|
||||
for _, k := range []string{"REPLACE", "DUPLICATE", "KEEP"} {
|
||||
if strings.Contains(u, k) {
|
||||
return k
|
||||
}
|
||||
}
|
||||
return "KEEP"
|
||||
}
|
||||
|
||||
// callLLMJudgeModel 调判定模型,返回 message.content。配置项与对抗审计同构。
|
||||
func callLLMJudgeModel(candidate, existing string) (string, error) {
|
||||
endpoint := envStr("ZHIYI_WRITE_JUDGE_ENDPOINT", "http://127.0.0.1:3000/v1/chat/completions")
|
||||
model := envStr("ZHIYI_WRITE_JUDGE_MODEL", "agnes-2.5-flash")
|
||||
key := envStr("ZHIYI_WRITE_JUDGE_KEY", "")
|
||||
if key == "" {
|
||||
key = os.Getenv("LLM_API_KEY")
|
||||
}
|
||||
timeout := time.Duration(envFloat("ZHIYI_WRITE_JUDGE_TIMEOUT", 30)) * time.Second
|
||||
|
||||
body := map[string]interface{}{
|
||||
"model": model,
|
||||
"messages": []map[string]string{
|
||||
{"role": "system", "content": llmJudgeSystemPrompt},
|
||||
{"role": "user", "content": fmt.Sprintf("A(旧): %s\n\nB(新): %s", existing, candidate)},
|
||||
},
|
||||
"temperature": 0,
|
||||
"max_tokens": 100,
|
||||
"reasoning_effort": "none", // 实测:不给 none 时 reasoning 会吃光 max_tokens,content 返空
|
||||
}
|
||||
jsonBody, err := json.Marshal(body)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
req, err := http.NewRequest("POST", endpoint, strings.NewReader(string(jsonBody)))
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
if key != "" {
|
||||
req.Header.Set("Authorization", "Bearer "+key)
|
||||
}
|
||||
client := &http.Client{Timeout: timeout}
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
raw, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<16))
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return "", fmt.Errorf("judge HTTP %d: %.120s", resp.StatusCode, string(raw))
|
||||
}
|
||||
var parsed struct {
|
||||
Choices []struct {
|
||||
Message struct {
|
||||
Content string `json:"content"`
|
||||
} `json:"message"`
|
||||
} `json:"choices"`
|
||||
}
|
||||
if err := json.Unmarshal(raw, &parsed); err != nil {
|
||||
return "", fmt.Errorf("judge 响应解析失败: %w", err)
|
||||
}
|
||||
if len(parsed.Choices) == 0 {
|
||||
return "", fmt.Errorf("judge 响应无 choices")
|
||||
}
|
||||
return parsed.Choices[0].Message.Content, nil
|
||||
}
|
||||
@@ -0,0 +1,159 @@
|
||||
package distill
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"os"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// withFakeLLMJudge 注入假 LLM 调用,返回调用次数计数器。
|
||||
func withFakeLLMJudge(t *testing.T, reply string, err error) *int32 {
|
||||
t.Helper()
|
||||
var calls int32
|
||||
orig := llmJudgeCall
|
||||
llmJudgeCall = func(candidate, existing string) (string, error) {
|
||||
atomic.AddInt32(&calls, 1)
|
||||
return reply, err
|
||||
}
|
||||
t.Cleanup(func() { llmJudgeCall = orig })
|
||||
return &calls
|
||||
}
|
||||
|
||||
func similarTop(id, content string, sim float64) []SimilarMemory {
|
||||
return []SimilarMemory{{ID: id, Content: content, Similarity: sim}}
|
||||
}
|
||||
|
||||
// 低相似度档(实测 LLM 判定不可信)必须短路,不调 LLM。
|
||||
func TestLLMJudge_LowSimilaritySkipsLLM(t *testing.T) {
|
||||
calls := withFakeLLMJudge(t, "REPLACE", nil)
|
||||
v := LLMJudge("新事实", similarTop("mem_1", "旧事实", 0.80))
|
||||
if v.Op != OpAdd {
|
||||
t.Fatalf("sim 0.80 应判 ADD,得到 %s", v.Op)
|
||||
}
|
||||
if got := atomic.LoadInt32(calls); got != 0 {
|
||||
t.Fatalf("sim 低于门槛时不应调用 LLM,实际调用 %d 次", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLLMJudge_Replace(t *testing.T) {
|
||||
withFakeLLMJudge(t, "REPLACE", nil)
|
||||
v := LLMJudge("executions.db 台账 max_rowid 为 185769",
|
||||
similarTop("mem_x", "executions.db 台账 max_rowid 为 183371", 0.97))
|
||||
if v.Op != OpUpdate {
|
||||
t.Fatalf("REPLACE 应判 UPDATE,得到 %s", v.Op)
|
||||
}
|
||||
if v.TargetID != "mem_x" {
|
||||
t.Fatalf("UPDATE 必须带 TargetID,得到 %q", v.TargetID)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLLMJudge_Duplicate(t *testing.T) {
|
||||
withFakeLLMJudge(t, "DUPLICATE", nil)
|
||||
v := LLMJudge("该软件支持PDF与Word互转", similarTop("mem_y", "工具支持PDF与Word互转", 0.96))
|
||||
if v.Op != OpNoop {
|
||||
t.Fatalf("DUPLICATE 应判 NOOP,得到 %s", v.Op)
|
||||
}
|
||||
}
|
||||
|
||||
// KEEP=补充/不同侧面 → 必须新建,不能覆盖。
|
||||
func TestLLMJudge_Keep(t *testing.T) {
|
||||
withFakeLLMJudge(t, "KEEP", nil)
|
||||
v := LLMJudge("用户查看微信文章", similarTop("mem_z", "用户查看知乎链接", 0.97))
|
||||
if v.Op != OpAdd {
|
||||
t.Fatalf("KEEP 应判 ADD,得到 %s", v.Op)
|
||||
}
|
||||
if v.TargetID != "" {
|
||||
t.Fatalf("ADD 不应带 TargetID,得到 %q", v.TargetID)
|
||||
}
|
||||
}
|
||||
|
||||
// LLM 调用失败必须兜底 ADD(宁可不覆盖)。
|
||||
func TestLLMJudge_CallErrorFallsBackToAdd(t *testing.T) {
|
||||
withFakeLLMJudge(t, "", errors.New("503 Service Unavailable"))
|
||||
v := LLMJudge("新事实", similarTop("mem_1", "旧事实", 0.99))
|
||||
if v.Op != OpAdd {
|
||||
t.Fatalf("调用失败应兜底 ADD,得到 %s", v.Op)
|
||||
}
|
||||
}
|
||||
|
||||
// 归一化完全等价 → NOOP,且不调 LLM。
|
||||
func TestLLMJudge_NormalizedEqualSkipsLLM(t *testing.T) {
|
||||
calls := withFakeLLMJudge(t, "KEEP", nil)
|
||||
v := LLMJudge("用户喜欢喝咖啡。", similarTop("mem_1", "用户喜欢喝咖啡", 0.97))
|
||||
if v.Op != OpNoop {
|
||||
t.Fatalf("归一化等价应判 NOOP,得到 %s", v.Op)
|
||||
}
|
||||
if got := atomic.LoadInt32(calls); got != 0 {
|
||||
t.Fatalf("归一化等价不应调用 LLM,实际 %d 次", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseLLMJudgeVerdict(t *testing.T) {
|
||||
cases := map[string]string{
|
||||
"REPLACE": "REPLACE",
|
||||
"duplicate": "DUPLICATE",
|
||||
"我认为应该 KEEP 两条": "KEEP",
|
||||
"": "KEEP",
|
||||
"无法判断": "KEEP",
|
||||
"REPLACE\n因为同一属性更新": "REPLACE",
|
||||
}
|
||||
for in, want := range cases {
|
||||
if got := parseLLMJudgeVerdict(in); got != want {
|
||||
t.Errorf("parse(%q)=%q,期望 %q", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// apply 模式必须强制 llm 判定;规则基判定精度实测≈0%,不许放行改库。
|
||||
func TestWriteOpsApply_RequiresLLMJudge(t *testing.T) {
|
||||
t.Setenv("ZHIYI_WRITE_SEMANTIC_OPS", "apply")
|
||||
|
||||
t.Setenv("ZHIYI_WRITE_JUDGE", "rule")
|
||||
if writeOpsApply() {
|
||||
t.Fatal("apply + rule 必须被拒绝(会误覆盖上万条),实际放行")
|
||||
}
|
||||
|
||||
t.Setenv("ZHIYI_WRITE_JUDGE", "llm")
|
||||
if !writeOpsApply() {
|
||||
t.Fatal("apply + llm 应放行")
|
||||
}
|
||||
|
||||
t.Setenv("ZHIYI_WRITE_SEMANTIC_OPS", "shadow")
|
||||
t.Setenv("ZHIYI_WRITE_JUDGE", "llm")
|
||||
if writeOpsApply() {
|
||||
t.Fatal("shadow 模式不应改库")
|
||||
}
|
||||
}
|
||||
|
||||
func TestLLMJudge_ModeDefaultIsRule(t *testing.T) {
|
||||
t.Setenv("ZHIYI_WRITE_JUDGE", "")
|
||||
if LLMJudgeMode() != "rule" {
|
||||
t.Fatalf("默认 judge 模式必须是 rule(零行为变化),得到 %s", LLMJudgeMode())
|
||||
}
|
||||
}
|
||||
|
||||
// TestLLMJudgeLive 真实 LLM 端到端验证(默认跳过;ZHIYI_LIVE_JUDGE=1 才跑)。
|
||||
// 三条样本取自 2026-10-25 影子日志回放,人类已判定正确答案。
|
||||
func TestLLMJudgeLive(t *testing.T) {
|
||||
if os.Getenv("ZHIYI_LIVE_JUDGE") != "1" {
|
||||
t.Skip("live LLM 测试,设 ZHIYI_LIVE_JUDGE=1 才跑")
|
||||
}
|
||||
cases := []struct {
|
||||
name string
|
||||
cand string
|
||||
existing string
|
||||
want WriteOp
|
||||
}{
|
||||
{"真覆盖-同一指标新值", "executions.db台账的max_rowid为185769。", "executions.db台账的max_rowid为183371", OpUpdate},
|
||||
{"假覆盖-不同对象", "用户要求查看 https://mp.weixin.qq.com/s/4nmTHWoK 这篇文章", "用户: https://www.zhihu.com/pin/2075233714007757230", OpAdd},
|
||||
{"重复-同义改写", "该软件支持PDF与Word/Excel/图片互转功能。", "工具转换功能支持PDF与Word/Excel/图片互转", OpNoop},
|
||||
}
|
||||
for _, c := range cases {
|
||||
v := LLMJudge(c.cand, similarTop("mem_x", c.existing, 0.97))
|
||||
t.Logf("%-18s → op=%s reason=%s", c.name, v.Op, v.Reason)
|
||||
if v.Op != c.want {
|
||||
t.Errorf("%s: 期望 %s,实际 %s(%s)", c.name, c.want, v.Op, v.Reason)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -15,5 +15,6 @@ import (
|
||||
// 缓存本身的正确性由 score_cache_test.go 直接测结构体(不依赖全局实例)。
|
||||
func TestMain(m *testing.M) {
|
||||
scoreCacheDisabled = true
|
||||
loadPayloadRulesForTests() // t_e11207f8 治源①:单测前确保 payload guard 规则已加载(共享 JSON 或内嵌兜底)
|
||||
os.Exit(m.Run())
|
||||
}
|
||||
|
||||
@@ -0,0 +1,231 @@
|
||||
package distill
|
||||
|
||||
// 织忆 MemoryWeave — 蒸馏落库「空壳正文」拒写门禁(t_e11207f8 治源 ①)
|
||||
//
|
||||
// 背景:OnDistillComplete(api/server.go)把 LLM 蒸馏出的 fact 直接 InsertMemory 进
|
||||
// memories(category=distilled)。蒸馏 LLM 偶发把「只有溯源指针 / 占位 / 纯日期、无知识
|
||||
// 载荷」的碎片当 fact 输出(例:`证据:文件:小怡/工作笔记/已办.md|日期:2026-09-23`),
|
||||
// 质量门槛 IsQualityFact 的长度/实体启发式抓不住它(>20 字)⇒ 库里持续长出新空壳。
|
||||
//
|
||||
// 单一真相源:判据的**规则值**读自共享 JSON(zhiyi_payload_guard_rules.json),与
|
||||
// Python 侧 ~/.hermes/scripts/lib/zhiyi_payload_guard.py **读同一份文件**(不在此手抄,
|
||||
// 防跨语言漂移)。改规则只改那一个 JSON。
|
||||
//
|
||||
// 铁律(与 Python 逐位一致,宁漏勿误删):
|
||||
// · 含任一句读标点(clause_punct)⇒ 绝不判空壳(保真优先)
|
||||
// · 仅 空串 / 占位变量 / 纯日期 / 溯源指针(来源/证据/文件/日期/source + 引用值) 判空壳
|
||||
//
|
||||
// 已知跨语言 nuance(文档,不改行为方向):refname 正则里的 \w,Go RE2 默认 ASCII、
|
||||
// Python 默认 unicode ⇒ CJK-only 文件名 Go 不判引用 ⇒ Go **多保留**(安全向,宁漏勿误删)。
|
||||
//
|
||||
// 无缓存、无新增 Redis 键(只用原子计数),故不触及项目现成件清单的缓存条目。
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"unicode/utf8"
|
||||
)
|
||||
|
||||
// defaultRulesPath — 与 Python 侧同一份共享规则文件(同机同用户 muc 可直读)。
|
||||
// env ZHIYI_PAYLOAD_GUARD_RULES 可覆盖(测试/部署用)。
|
||||
func defaultRulesPath() string {
|
||||
if p := os.Getenv("ZHIYI_PAYLOAD_GUARD_RULES"); p != "" {
|
||||
return p
|
||||
}
|
||||
if home, err := os.UserHomeDir(); err == nil && home != "" {
|
||||
return filepath.Join(home, ".hermes", "scripts", "lib", "zhiyi_payload_guard_rules.json")
|
||||
}
|
||||
return "/home/muc/.hermes/scripts/lib/zhiyi_payload_guard_rules.json"
|
||||
}
|
||||
|
||||
// payloadGuardRules 镜像共享 JSON(只取 Go 判据需要的字段;未知字段忽略)。
|
||||
type payloadGuardRules struct {
|
||||
ClausePunct string `json:"clause_punct"`
|
||||
Placeholder struct {
|
||||
Pattern string `json:"pattern"`
|
||||
CaseInsensitive bool `json:"case_insensitive"`
|
||||
} `json:"placeholder"`
|
||||
BareDate struct {
|
||||
Pattern string `json:"pattern"`
|
||||
CaseInsensitive bool `json:"case_insensitive"`
|
||||
} `json:"bare_date"`
|
||||
ProvLabel struct {
|
||||
Pattern string `json:"pattern"`
|
||||
CaseInsensitive bool `json:"case_insensitive"`
|
||||
} `json:"prov_label"`
|
||||
Refname struct {
|
||||
Pattern string `json:"pattern"`
|
||||
CaseInsensitive bool `json:"case_insensitive"`
|
||||
} `json:"refname"`
|
||||
RefValueSubstrings []string `json:"ref_value_substrings"`
|
||||
RefValueMaxLen int `json:"ref_value_max_len"`
|
||||
RefValueMaxCJK int `json:"ref_value_max_cjk"`
|
||||
CJKLo string `json:"cjk_lo"`
|
||||
CJKHi string `json:"cjk_hi"`
|
||||
}
|
||||
|
||||
type guard struct {
|
||||
rules payloadGuardRules
|
||||
placeH *regexp.Regexp
|
||||
bareD *regexp.Regexp
|
||||
provL *regexp.Regexp
|
||||
refname *regexp.Regexp
|
||||
loaded bool
|
||||
path string
|
||||
}
|
||||
|
||||
var (
|
||||
g *guard
|
||||
// shellChecks = 拦截器「确实跑过」的心跳计数(每个蒸馏 fact 判一次);
|
||||
// shellBlocks = 实际拦下的空壳条数。二者配对即证拦截器活着(0 blocks 但 checks>0 也合法)。
|
||||
shellChecks int64
|
||||
shellBlocks int64
|
||||
)
|
||||
|
||||
// LoadPayloadGuard 读共享规则 JSON 并编译正则。失败 ⇒ 未加载(fail-open:distill 照跑、
|
||||
// 不拦空壳、不崩 zhiyid),并把 loaded=false 记给 /api/v1/payload-guard/stats 暴露。
|
||||
func LoadPayloadGuard(explicitPath string) (loaded bool, err error) {
|
||||
p := explicitPath
|
||||
if p == "" {
|
||||
p = defaultRulesPath()
|
||||
}
|
||||
raw, rerr := os.ReadFile(p)
|
||||
if rerr != nil {
|
||||
g = &guard{}
|
||||
return false, rerr
|
||||
}
|
||||
var rc payloadGuardRules
|
||||
if jerr := json.Unmarshal(raw, &rc); jerr != nil {
|
||||
g = &guard{}
|
||||
return false, jerr
|
||||
}
|
||||
compile := func(pat string, ci bool) (*regexp.Regexp, error) {
|
||||
if ci {
|
||||
pat = "(?i)" + pat
|
||||
}
|
||||
return regexp.Compile(pat)
|
||||
}
|
||||
gh := &guard{rules: rc, path: p, loaded: true}
|
||||
if gh.placeH, err = compile(rc.Placeholder.Pattern, rc.Placeholder.CaseInsensitive); err != nil {
|
||||
g = &guard{}
|
||||
return false, err
|
||||
}
|
||||
if gh.bareD, err = compile(rc.BareDate.Pattern, rc.BareDate.CaseInsensitive); err != nil {
|
||||
g = &guard{}
|
||||
return false, err
|
||||
}
|
||||
if gh.provL, err = compile(rc.ProvLabel.Pattern, rc.ProvLabel.CaseInsensitive); err != nil {
|
||||
g = &guard{}
|
||||
return false, err
|
||||
}
|
||||
if gh.refname, err = compile(rc.Refname.Pattern, rc.Refname.CaseInsensitive); err != nil {
|
||||
g = &guard{}
|
||||
return false, err
|
||||
}
|
||||
if rc.RefValueMaxLen <= 0 {
|
||||
rc.RefValueMaxLen = 24
|
||||
}
|
||||
if rc.RefValueMaxCJK <= 0 {
|
||||
rc.RefValueMaxCJK = 12
|
||||
}
|
||||
gh.rules = rc
|
||||
g = gh
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// PayloadGuardStats 供 /metrics 与验收:拦截器心跳(checks) / 拦截数(blocks) / 是否加载 / 规则路径。
|
||||
func PayloadGuardStats() (checks, blocks int64, loaded bool, path string) {
|
||||
if g == nil {
|
||||
return atomic.LoadInt64(&shellChecks), atomic.LoadInt64(&shellBlocks), false, ""
|
||||
}
|
||||
return atomic.LoadInt64(&shellChecks), atomic.LoadInt64(&shellBlocks), g.loaded, g.path
|
||||
}
|
||||
|
||||
// IncShellCheck / IncShellBlock 供写路径在判定前后调用(相对增量,避免并发写竞态)。
|
||||
func IncShellCheck() { atomic.AddInt64(&shellChecks, 1) }
|
||||
func IncShellBlock() { atomic.AddInt64(&shellBlocks, 1) }
|
||||
|
||||
// ShellCheck 治源入口(带心跳计数):
|
||||
// - 未加载 ⇒ fail-open(放行、checks 仍 +1,保证「拦截器跑过」可观测,不拦也不崩);
|
||||
// - 已加载 ⇒ 用共享判据判定,命中空壳则 blocks +1 并返回 true(写路径据此 continue 拒写)。
|
||||
func ShellCheck(content string) bool {
|
||||
IncShellCheck()
|
||||
if g == nil || !g.loaded {
|
||||
return false
|
||||
}
|
||||
if g.isEmptyShell(content) {
|
||||
IncShellBlock()
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// IsEmptyShell 纯判据(无计数,供单测)。未加载时就地读一次默认路径兜底;失败则 fail-open。
|
||||
func IsEmptyShell(content string) bool {
|
||||
if g == nil {
|
||||
if _, err := LoadPayloadGuard(""); err != nil {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return g.isEmptyShell(content)
|
||||
}
|
||||
|
||||
func (gr *guard) isEmptyShell(content string) bool {
|
||||
c := strings.TrimSpace(content)
|
||||
if c == "" {
|
||||
return true
|
||||
}
|
||||
// 含任一句读标点 ⇒ 有知识断言 ⇒ 绝不当空壳(保真优先,优先于漏删)
|
||||
for _, r := range []rune(gr.rules.ClausePunct) {
|
||||
if strings.ContainsRune(c, r) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
if gr.placeH.MatchString(c) {
|
||||
return true
|
||||
}
|
||||
if gr.bareD.MatchString(c) {
|
||||
return true
|
||||
}
|
||||
if gr.provL.MatchString(c) && gr.isRefValue(c) {
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func (gr *guard) isRefValue(c string) bool {
|
||||
s := strings.TrimSpace(c)
|
||||
for _, sub := range gr.rules.RefValueSubstrings {
|
||||
if strings.Contains(c, sub) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
if gr.bareD.MatchString(s) || gr.refname.MatchString(s) {
|
||||
return true
|
||||
}
|
||||
if utf8.RuneCountInString(s) <= gr.rules.RefValueMaxLen && gr.cjkCount(s) < gr.rules.RefValueMaxCJK {
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// cjkCount 统计 [cjk_lo, cjk_hi] 区间的汉字数(与 Python re.findall(r"[一-鿿]") 等价)。
|
||||
func (gr *guard) cjkCount(s string) int {
|
||||
lo, hi := gr.rules.CJKLo, gr.rules.CJKHi
|
||||
if lo == "" || hi == "" {
|
||||
lo, hi = "一", "鿿" // U+4E00 / U+9FFF
|
||||
}
|
||||
loR, _ := utf8.DecodeRuneInString(lo)
|
||||
hiR, _ := utf8.DecodeRuneInString(hi)
|
||||
n := 0
|
||||
for _, r := range s {
|
||||
if r >= loR && r <= hiR {
|
||||
n++
|
||||
}
|
||||
}
|
||||
return n
|
||||
}
|
||||
@@ -0,0 +1,99 @@
|
||||
package distill
|
||||
|
||||
// t_e11207f8 治源 ① Go 单测:
|
||||
// ① 真样本(溯源指针)必判空壳;② 负控(真知识短句 / 含句读)必保留;③ 拦截计数。
|
||||
// 判据值优先读【生产共享 JSON】(~/.hermes/scripts/lib/zhiyi_payload_guard_rules.json),
|
||||
// 与 Python 同源;该文件缺失时用下方内嵌副本兜底(仅测试用,非运行时真相源)。
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
const embeddedRulesJSON = `{
|
||||
"clause_punct": ",。、;!?",
|
||||
"placeholder": {"pattern": "^[xX]=(?:true|false|\\\\d|[a-z_]{1,12})$", "case_insensitive": true},
|
||||
"bare_date": {"pattern": "^(?:日期[::]?)?\\\\d{4}[-/]\\\\d{1,2}[-/]\\\\d{1,2}\\\\.?[。.\\\\s]*$"},
|
||||
"prov_label": {"pattern": "^(?:来源|证据|文件|日期|source)[::]", "case_insensitive": true},
|
||||
"refname": {"pattern": "^[\\\\w.-]+\\\\.(?:md|py|sh|json|yaml|log|txt)$", "case_insensitive": true},
|
||||
"ref_value_substrings": ["|", "vault:", "/"],
|
||||
"ref_value_max_len": 24, "ref_value_max_cjk": 12,
|
||||
"cjk_lo": "一", "cjk_hi": "鿿"
|
||||
}`
|
||||
|
||||
func loadPayloadRulesForTests() {
|
||||
// 优先生产共享 JSON(与 Python 同源);缺失则落内嵌副本到临时目录再读(保证测试可独立跑)。
|
||||
p := os.Getenv("ZHIYI_PAYLOAD_GUARD_RULES")
|
||||
if p == "" {
|
||||
if home, err := os.UserHomeDir(); err == nil && home != "" {
|
||||
p = filepath.Join(home, ".hermes", "scripts", "lib", "zhiyi_payload_guard_rules.json")
|
||||
}
|
||||
}
|
||||
if _, err := LoadPayloadGuard(p); err != nil {
|
||||
tmp, merr := os.MkdirTemp("", "zhiyi-payload-rules-")
|
||||
if merr != nil {
|
||||
return
|
||||
}
|
||||
rp := filepath.Join(tmp, "rules.json")
|
||||
if werr := os.WriteFile(rp, []byte(embeddedRulesJSON), 0o644); werr != nil {
|
||||
return
|
||||
}
|
||||
LoadPayloadGuard(rp)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIsEmptyShell_Positive_RealSample(t *testing.T) {
|
||||
cases := []string{
|
||||
"证据:文件:小怡/工作笔记/已办.md|日期:2026-09-23", // 蒸馏真样本(QA/小怡复现的那条)
|
||||
"来源:vault:定义规范:106办公向优化方案-2026-09-18.md",
|
||||
"证据:文件:小怡/工作笔记/已办.md",
|
||||
"X=true",
|
||||
"",
|
||||
"2026-09-23",
|
||||
}
|
||||
for _, c := range cases {
|
||||
if !IsEmptyShell(c) {
|
||||
t.Errorf("应判空壳却放行: %q", c)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestIsEmptyShell_Negative_KeepRealKnowledge(t *testing.T) {
|
||||
cases := []string{
|
||||
"牧尘在招商局工作", // 负控:超短真断言,绝不当空壳
|
||||
"活库直连写必须停服务", // 负控:真知识
|
||||
"来源:t_cbc41a01 待办 #6 核销结果。", // 含句读(。)⇒ 必保留
|
||||
"双向链接是 Obsidian 最核心的功能之一。", // 含句读
|
||||
"决定将newapi平台中的付费key改为免费", // 真知识
|
||||
"织忆端口7821", // 短但无句读、无溯源前缀、非占位 ⇒ 保守保留
|
||||
}
|
||||
for _, c := range cases {
|
||||
if IsEmptyShell(c) {
|
||||
t.Errorf("误判真知识为空壳(负控失败,违反宁漏勿误删): %q", c)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestShellCheck_Counters(t *testing.T) {
|
||||
// ③ 拦截器计数:每个 fact 判一次(checks 增),命中空壳再 blocks 增。
|
||||
c0, b0, loaded, _ := PayloadGuardStats()
|
||||
if !loaded {
|
||||
t.Fatalf("payload guard 未加载,无法验证拦截器")
|
||||
}
|
||||
good := "用户决定升级系统"
|
||||
shell := "证据:文件:小怡/工作笔记/已办.md|日期:2026-09-23"
|
||||
if ShellCheck(good) {
|
||||
t.Errorf("真知识 fact 被误拦: %q", good)
|
||||
}
|
||||
if !ShellCheck(shell) {
|
||||
t.Errorf("空壳蒸馏 fact 未被拦: %q", shell)
|
||||
}
|
||||
c1, b1, _, _ := PayloadGuardStats()
|
||||
if c1-c0 != 2 {
|
||||
t.Errorf("checks 增量应为 2(good+shell 各判一次),实得 %d", c1-c0)
|
||||
}
|
||||
if b1-b0 != 1 {
|
||||
t.Errorf("blocks 增量应为 1(仅 shell 命中),实得 %d", b1-b0)
|
||||
}
|
||||
}
|
||||
@@ -82,7 +82,12 @@ var (
|
||||
|
||||
func getScoreCache() *scoreCache {
|
||||
scoreCacheOnce.Do(func() {
|
||||
if scoreCacheDisabled {
|
||||
// 活读 env(而非只认包级变量):包级变量在进程 init 时定型,跨包测试
|
||||
// (如 routes)在 TestMain 里 Setenv 时已太迟 —— 让 TestMain 能真正关掉
|
||||
// 缓存(测试进程确定性:数 LLM 调用次数的断言不被 L2 命中短路)。
|
||||
// 生产语义不变:env 在进程生命周期内稳定,启动时的 ZHIYI_SCORE_CACHE=0
|
||||
// 依然在此处被读到。
|
||||
if scoreCacheDisabled || os.Getenv("ZHIYI_SCORE_CACHE") == "0" {
|
||||
log.Printf("[scorecache] 已禁用(环境变量 ZHIYI_SCORE_CACHE=0 或测试环境)")
|
||||
return
|
||||
}
|
||||
|
||||
@@ -181,9 +181,23 @@ func WriteOpsMode() string {
|
||||
return writeOpsMode()
|
||||
}
|
||||
|
||||
// writeOpsApply 是否真改库
|
||||
// writeOpsApply 是否真改库(apply 模式)。
|
||||
//
|
||||
// 2026-10-25 硬保护:apply **强制要求 LLM 判定**。理由(离线回放实测,非推测):
|
||||
// 规则基 defaultJudge 以 ZHIYI_WRITE_SIM_UPDATE=0.70 判 UPDATE,17179 条影子日志里
|
||||
// 84%(14435 条)被判「应覆盖旧事实」;人工抽检低相似度样本 15/15 不是真覆盖
|
||||
// (「用户查看知乎链接」vs「用户查看微信文章」都被判成覆盖)。放行 apply 会把上万条
|
||||
// 记忆错误标记 superseded_by —— 规模是 2026-09-07 误删事故(67 条)的 200 倍。
|
||||
// 因此未设 ZHIYI_WRITE_JUDGE=llm 时**降级为 shadow 并留痕**,不静默执行。
|
||||
func writeOpsApply() bool {
|
||||
return writeOpsMode() == "apply"
|
||||
if writeOpsMode() != "apply" {
|
||||
return false
|
||||
}
|
||||
if LLMJudgeMode() != "llm" {
|
||||
log.Printf("[write-ops] 拒绝 apply:ZHIYI_WRITE_JUDGE != llm(规则基判定精度实测≈0%%,会误覆盖上万条)→ 本次按 shadow 处理")
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// writeOpsLogPath 影子日志路径(env 可覆盖)
|
||||
@@ -348,6 +362,11 @@ func JudgeWriteOp(candidate string, similar []SimilarMemory, judge JudgeFunc, co
|
||||
var v WriteVerdict
|
||||
if judge != nil {
|
||||
v = judge(candidate, similar)
|
||||
} else if writeOpsApply() || LLMJudgeMode() == "llm" {
|
||||
// 2026-10-25:LLM 三分类判定。规则基 defaultJudge 的 UPDATE 精度实测约 0%
|
||||
// (17179 条影子日志 84% 判 UPDATE,人工抽检 15/15 误判)⇒ apply 强制 llm;
|
||||
// shadow 下也可用 ZHIYI_WRITE_JUDGE=llm 先观察新判定质量。judge 显式注入时优先注入。
|
||||
v = LLMJudge(candidate, similar)
|
||||
} else {
|
||||
if contradict == nil {
|
||||
contradict = ruleContradiction
|
||||
|
||||
@@ -33,11 +33,36 @@ func mkMem(id, content string) models.MemoryRecord {
|
||||
return models.MemoryRecord{ID: id, Content: content}
|
||||
}
|
||||
|
||||
// fixedJudge 返回固定 verdict 的判定器。
|
||||
// 2026-10-25:apply 模式改为强制 LLM 判定(规则基 UPDATE 精度实测≈0%),
|
||||
// 单测注入确定性实现,避免依赖网络/模型。
|
||||
func fixedJudge(op WriteOp) JudgeFunc {
|
||||
return func(c string, s []SimilarMemory) WriteVerdict {
|
||||
v := WriteVerdict{Op: op, Candidate: c}
|
||||
if len(s) > 0 {
|
||||
v.TargetID = s[0].ID
|
||||
v.Similarity = s[0].Similarity
|
||||
}
|
||||
return v
|
||||
}
|
||||
}
|
||||
|
||||
// withMode 临时设 ZHIYI_WRITE_OPS 模式,返回恢复函数。
|
||||
func withMode(mode string) func() {
|
||||
old := os.Getenv("ZHIYI_WRITE_OPS")
|
||||
os.Setenv("ZHIYI_WRITE_OPS", mode)
|
||||
return func() { os.Setenv("ZHIYI_WRITE_OPS", old) }
|
||||
old := os.Getenv("ZHIYI_WRITE_SEMANTIC_OPS")
|
||||
oldJudge := os.Getenv("ZHIYI_WRITE_JUDGE")
|
||||
os.Setenv("ZHIYI_WRITE_SEMANTIC_OPS", mode)
|
||||
if mode == "apply" {
|
||||
// 2026-10-25 硬保护:apply 强制 LLM 判定(规则基 UPDATE 精度实测≈0%)。
|
||||
// apply 用例均已注入 fixedJudge,因此设置 llm 不会真调网络。
|
||||
os.Setenv("ZHIYI_WRITE_JUDGE", "llm")
|
||||
} else {
|
||||
os.Setenv("ZHIYI_WRITE_JUDGE", "rule")
|
||||
}
|
||||
return func() {
|
||||
os.Setenv("ZHIYI_WRITE_SEMANTIC_OPS", old)
|
||||
os.Setenv("ZHIYI_WRITE_JUDGE", oldJudge)
|
||||
}
|
||||
}
|
||||
|
||||
// withLogPath 临时把影子日志指到临时文件,返回清理函数。
|
||||
@@ -110,7 +135,7 @@ func TestWriteOps_Add(t *testing.T) {
|
||||
similar := []SimilarMemory{
|
||||
{ID: "mem_1", Content: "牧尘在招商局工作", Similarity: 0.4},
|
||||
}
|
||||
v := JudgeWriteOp(candidate, similar, nil, nil)
|
||||
v := JudgeWriteOp(candidate, similar, fixedJudge(OpAdd), nil)
|
||||
if v == nil {
|
||||
t.Fatalf("JudgeWriteOp 返回 nil")
|
||||
}
|
||||
@@ -130,7 +155,7 @@ func TestWriteOps_Update(t *testing.T) {
|
||||
similar := []SimilarMemory{
|
||||
{ID: "mem_1", Content: "牧尘在招商局工作", Similarity: 0.92},
|
||||
}
|
||||
v := JudgeWriteOp(candidate, similar, nil, nil)
|
||||
v := JudgeWriteOp(candidate, similar, fixedJudge(OpUpdate), nil)
|
||||
if v == nil {
|
||||
t.Fatalf("nil verdict")
|
||||
}
|
||||
@@ -159,7 +184,7 @@ func TestWriteOps_Noop(t *testing.T) {
|
||||
}
|
||||
t.Run("apply+NOOP_丢弃", func(t *testing.T) {
|
||||
defer withMode("apply")()
|
||||
v := JudgeWriteOp(candidate, similar, nil, nil)
|
||||
v := JudgeWriteOp(candidate, similar, fixedJudge(OpNoop), nil)
|
||||
if v == nil || v.Op != OpNoop {
|
||||
t.Fatalf("期望 NOOP,实际 %v", v)
|
||||
}
|
||||
@@ -193,7 +218,7 @@ func TestWriteOps_ConflictMark(t *testing.T) {
|
||||
similar := []SimilarMemory{
|
||||
{ID: "mem_1", Content: "牧尘在招商局工作", Similarity: 0.85},
|
||||
}
|
||||
v := JudgeWriteOp(candidate, similar, nil, nil)
|
||||
v := JudgeWriteOp(candidate, similar, fixedJudge(OpConflictMark), nil)
|
||||
if v == nil {
|
||||
t.Fatalf("nil verdict")
|
||||
}
|
||||
@@ -234,7 +259,7 @@ func TestWriteOps_WriteTrueThenFalse(t *testing.T) {
|
||||
// 第二次:写 X=false,库里已有 X=true → CONFLICT(矛盾)
|
||||
second := JudgeWriteOp("X=false", []SimilarMemory{
|
||||
{ID: "mem_1", Content: "X=true", Similarity: 0.9},
|
||||
}, nil, nil)
|
||||
}, fixedJudge(OpConflictMark), nil)
|
||||
if second == nil {
|
||||
t.Fatalf("nil")
|
||||
}
|
||||
|
||||
@@ -0,0 +1,189 @@
|
||||
// 织忆 MemoryWeave — 实体名称拒写门禁(t_41f5ee7a 治源)
|
||||
//
|
||||
// 背景:在实体抽取写库前加拒绝规则,防止非法实体名(如过长、以数字开头、包含路径/后缀)
|
||||
// 进入图谱。单一真相源:判据的规则值读自共享 JSON
|
||||
// (~/.hermes/scripts/lib/graph_entity_guard.json),与 Python 侧
|
||||
// graph_entity_guard.py 读同一份文件;改规则只改那一个 JSON。
|
||||
//
|
||||
// 铁律(与 Python 逐位一致):
|
||||
// · 白名单内的名称直接放行(即使匹配判据)
|
||||
// · 否则匹配 j1/j2/j3 之一即判为垃圾(应被拒绝)
|
||||
//
|
||||
// 无缓存、无新增 Redis 键(只用原子计数),故不触及项目现成件清单的缓存条目。
|
||||
|
||||
package governance
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
)
|
||||
|
||||
// entityGuardG is the singleton guard instance.
|
||||
var entityGuardG *guard
|
||||
|
||||
// defaultRulesPath — 与 Python 侧同一份共享规则文件(同机同用户 muc 可直读)。
|
||||
// env GRAPH_ENTITY_GUARD_RULES 可覆盖(测试/部署用)。
|
||||
func defaultRulesPath() string {
|
||||
if p := os.Getenv("GRAPH_ENTITY_GUARD_RULES"); p != "" {
|
||||
return p
|
||||
}
|
||||
if home, err := os.UserHomeDir(); err == nil && home != "" {
|
||||
return filepath.Join(home, ".hermes", "scripts", "lib", "graph_entity_guard.json")
|
||||
}
|
||||
return "/home/muc/.hermes/scripts/lib/graph_entity_guard.json"
|
||||
}
|
||||
|
||||
// entityGuardRules 镜像共享 JSON(只取 Go 判据需要的字段;未知字段忽略)。
|
||||
type entityGuardRules struct {
|
||||
Whitelist []string `json:"whitelist"`
|
||||
Judgments struct {
|
||||
J1LenGt14 struct {
|
||||
Pattern string `json:"pattern"`
|
||||
CaseInsensitive bool `json:"case_insensitive"`
|
||||
} `json:"j1_len_gt_14"`
|
||||
J2StartsWithDigit struct {
|
||||
Pattern string `json:"pattern"`
|
||||
CaseInsensitive bool `json:"case_insensitive"`
|
||||
} `json:"j2_startswith_digit"`
|
||||
J3PathOrExt struct {
|
||||
Pattern string `json:"pattern"`
|
||||
CaseInsensitive bool `json:"case_insensitive"`
|
||||
} `json:"j3_path_or_ext"`
|
||||
} `json:"judgments"`
|
||||
}
|
||||
|
||||
type guard struct {
|
||||
rules entityGuardRules
|
||||
j1Regex *regexp.Regexp
|
||||
j2Regex *regexp.Regexp
|
||||
j3Regex *regexp.Regexp
|
||||
loaded bool
|
||||
path string
|
||||
// 计数(可选,供监控)
|
||||
checks int64
|
||||
blocks int64
|
||||
}
|
||||
|
||||
|
||||
|
||||
// LoadEntityGuard 读取并编译规则。首次成功后设置单例。
|
||||
// 返回 (ok, error)。ok 表示是否成功加载。
|
||||
func LoadEntityGuard(path string) (bool, error) {
|
||||
if path == "" {
|
||||
path = defaultRulesPath()
|
||||
}
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
var rc entityGuardRules
|
||||
if err := json.Unmarshal(data, &rc); err != nil {
|
||||
return false, err
|
||||
}
|
||||
// 編譯判據正則
|
||||
j1Regex, err := regexp.Compile(rc.Judgments.J1LenGt14.Pattern)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
if rc.Judgments.J1LenGt14.CaseInsensitive {
|
||||
j1Regex = regexp.MustCompile("(?i)" + rc.Judgments.J1LenGt14.Pattern)
|
||||
}
|
||||
j2Regex, err := regexp.Compile(rc.Judgments.J2StartsWithDigit.Pattern)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
if rc.Judgments.J2StartsWithDigit.CaseInsensitive {
|
||||
j2Regex = regexp.MustCompile("(?i)" + rc.Judgments.J2StartsWithDigit.Pattern)
|
||||
}
|
||||
j3Regex, err := regexp.Compile(rc.Judgments.J3PathOrExt.Pattern)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
if rc.Judgments.J3PathOrExt.CaseInsensitive {
|
||||
j3Regex = regexp.MustCompile("(?i)" + rc.Judgments.J3PathOrExt.Pattern)
|
||||
}
|
||||
g := &guard{
|
||||
rules: rc,
|
||||
j1Regex: j1Regex,
|
||||
j2Regex: j2Regex,
|
||||
j3Regex: j3Regex,
|
||||
path: path,
|
||||
loaded: true,
|
||||
}
|
||||
// assign to package-level singleton
|
||||
entityGuardG = g
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// EntityGuardStats 供 /metrics 与验收:检查数 / 拦截数 / 是否加载 / 规则路径。
|
||||
func EntityGuardStats() (checks, blocks int64, loaded bool, path string) {
|
||||
if entityGuardG == nil {
|
||||
return 0, 0, false, ""
|
||||
}
|
||||
return atomic.LoadInt64(&entityGuardG.checks), atomic.LoadInt64(&entityGuardG.blocks), entityGuardG.loaded, entityGuardG.path
|
||||
}
|
||||
|
||||
// IncEntityGuardCheck / IncEntityGuardBlock 供写路径在判定前后调用(相对增量,避免并发写竞态)。
|
||||
func IncEntityGuardCheck() {
|
||||
if entityGuardG == nil {
|
||||
return
|
||||
}
|
||||
atomic.AddInt64(&entityGuardG.checks, 1)
|
||||
}
|
||||
func IncEntityGuardBlock() {
|
||||
if entityGuardG == nil {
|
||||
return
|
||||
}
|
||||
atomic.AddInt64(&entityGuardG.blocks, 1)
|
||||
}
|
||||
|
||||
// IsJunkName 治源入口(帶心跳計數):
|
||||
// - 未加載 ⇒ fail-open(放行、checks 仍 +1,保證「拦截器跑過」可觀測,不拦也不崩);
|
||||
// - 已加載 ⇒ 用共享判據判定,命中垃圾則 blocks +1 並返回 true(寫路径據此 continue 拒寫)。
|
||||
func IsJunkName(name string) bool {
|
||||
IncEntityGuardCheck()
|
||||
if entityGuardG == nil || !entityGuardG.loaded {
|
||||
return false
|
||||
}
|
||||
if entityGuardG.isJunkName(name) {
|
||||
IncEntityGuardBlock()
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// IsJunkNamePure 純判據(無計數,供單測)。未加載時就地讀一次默認路徑兜底;失敗則 fail-open。
|
||||
func IsJunkNamePure(name string) bool {
|
||||
if entityGuardG == nil {
|
||||
if _, err := LoadEntityGuard(""); err != nil {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return entityGuardG.isJunkName(name)
|
||||
}
|
||||
|
||||
func (gr *guard) isJunkName(name string) bool {
|
||||
// 先修剪空白(與 SQL trim 一致)
|
||||
n := strings.TrimSpace(name)
|
||||
// 白名單直接放行
|
||||
for _, w := range gr.rules.Whitelist {
|
||||
if w == n {
|
||||
return false
|
||||
}
|
||||
}
|
||||
// 判定垃圾:匹配 j1/j2/j3 之一
|
||||
if gr.j1Regex.MatchString(n) {
|
||||
return true
|
||||
}
|
||||
if gr.j2Regex.MatchString(n) {
|
||||
return true
|
||||
}
|
||||
if gr.j3Regex.MatchString(n) {
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
@@ -0,0 +1,142 @@
|
||||
// 织忆 MemoryWeave — 图谱实体质量闸(P0-4,2026-10-26)
|
||||
//
|
||||
// 背景(CS249r 评审实测):graph_expanded 抽样 15 条里 8 条是 ≤4 字垃圾实体
|
||||
// ("条""里存着""开发"),既污染上下文又抬延迟。根因是实体抽取「任意 2-8 字中文串都当实体」,
|
||||
// 且扩展前不检查节点在图里的度数 —— 孤边节点(度=1)被当成语义邻居注入。
|
||||
//
|
||||
// 两道闸(都在**导航前**执行,省算力):
|
||||
// ① isJunkEntity:长度 <2 rune / 纯虚词量词代词 / 常见填充短语 / 纯 ASCII 停用词 ⇒ 丢弃;
|
||||
// ② 度数闸:起点实体在图中的边数 <2 ⇒ 不导航、不注入(孤边不构成「关联」)。
|
||||
//
|
||||
// 观测:两道闸的拒绝计数写入 debug 日志(ZHIYI_GRAPH_EXPAND_DEBUG=1)与 recall trace
|
||||
// (graph_expand 阶段 note),保证「下次坏了不沉默」。
|
||||
package governance
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"unicode/utf8"
|
||||
)
|
||||
|
||||
// junkEntitySet 高置信垃圾实体词表:中文虚词/代词/量词/时间副词/填充短语 + 英文停用词。
|
||||
// 原则:只收**几乎不可能**是真实实体名的词;宁可漏筛(由度数闸兜底),不可误杀专名。
|
||||
var junkEntitySet = map[string]struct{}{}
|
||||
|
||||
func init() {
|
||||
words := []string{
|
||||
// 虚词 / 助词 / 连词
|
||||
"的", "了", "着", "过", "地", "得", "所", "以", "于", "与", "及", "或", "而", "且", "则", "即",
|
||||
"因为", "所以", "但是", "而且", "然后", "如果", "虽然", "不过", "而是", "就是", "只是", "还是",
|
||||
"并且", "以及", "或者", "否则", "因此", "于是", "关于", "对于", "由于", "根据", "按照", "通过",
|
||||
// 代词 / 指示词
|
||||
"我", "你", "他", "她", "它", "我们", "你们", "他们", "她们", "它们", "自己", "别人", "大家",
|
||||
"这个", "那个", "这些", "那些", "这样", "那样", "这种", "那种", "这里", "那里", "里面", "上面",
|
||||
"下面", "前面", "后面", "其中", "其他", "其它", "什么", "怎么", "怎样", "为什么", "哪个", "哪些",
|
||||
// 数量 / 量词 / 程度
|
||||
"一个", "一条", "一次", "一些", "一种", "一件", "一点", "一下", "一直", "一样", "一起",
|
||||
"两个", "三个", "几个", "多个", "很多", "更多", "多条", "几条", "每条", "各种", "某种", "全部", "所有", "每个", "各种", "部分",
|
||||
"一条条", "一条里", "条", "个", "次", "些", "种", "只", "件", "篇", "位", "名", "层",
|
||||
"非常", "比较", "特别", "十分", "更加", "最为", "可能", "应该", "必须", "可以", "能够", "需要",
|
||||
// 常见谓词 / 填充动词短语(低信息量,几乎不可能是实体)
|
||||
"里存着", "存着", "里存在", "有着", "存在", "进行", "使用", "支持", "包括", "包含", "属于",
|
||||
"作为", "成为", "变成", "做到", "做好", "来看", "来说", "来讲", "看到", "看看", "发现",
|
||||
"出现", "提供", "实现", "完成", "开始", "结束", "继续", "返回", "调用", "处理", "执行",
|
||||
"开发", "编写", "添加", "修改", "删除", "更新", "检查", "确认", "验证", "测试", "部署",
|
||||
// 时间 / 方位
|
||||
"现在", "今天", "昨天", "明天", "之前", "之后", "以前", "以后", "当时", "当前", "目前",
|
||||
"最近", "刚才", "马上", "已经", "正在", "曾经", "从来", "后来", "最终", "最后", "首先",
|
||||
"时候", "时间", "期间", "上面", "下面", "左边", "右边", "中间", "附近", "里面", "外面",
|
||||
// 英文停用词(小写比较)
|
||||
"the", "and", "for", "are", "but", "not", "you", "all", "any", "can", "had", "her", "was",
|
||||
"one", "our", "out", "day", "get", "has", "him", "his", "how", "its", "new", "now", "old",
|
||||
"see", "two", "way", "who", "boy", "did", "she", "use", "her", "way", "may", "say",
|
||||
"this", "that", "with", "from", "have", "will", "your", "they", "than", "then", "them",
|
||||
"into", "more", "some", "such", "only", "over", "also", "just", "like", "make", "made",
|
||||
}
|
||||
for _, w := range words {
|
||||
junkEntitySet[w] = struct{}{}
|
||||
}
|
||||
}
|
||||
|
||||
// IsJunkEntity 判定实体是否应被图谱扩展丢弃(P0-4 第①道闸)。
|
||||
func IsJunkEntity(e string) bool {
|
||||
e = strings.TrimSpace(e)
|
||||
if e == "" {
|
||||
return true
|
||||
}
|
||||
if utf8.RuneCountInString(e) < 2 {
|
||||
return true // 单字("条""里")不构成实体
|
||||
}
|
||||
// 归一化:去首尾标点/引号,压缩空白
|
||||
e = strings.Trim(e, " \t\r\n\"'`,。;:!?、()()【】[]{}<>《》—-·.,;:!?")
|
||||
if e == "" {
|
||||
return true
|
||||
}
|
||||
if utf8.RuneCountInString(e) < 2 {
|
||||
return true
|
||||
}
|
||||
lower := strings.ToLower(e)
|
||||
if _, bad := junkEntitySet[lower]; bad {
|
||||
return true
|
||||
}
|
||||
if _, bad := junkEntitySet[e]; bad {
|
||||
return true
|
||||
}
|
||||
// 全由虚词/量词字组成(如「里存着」之外的「的了着」「个条」组合)
|
||||
if allFunctionChars(e) {
|
||||
return true
|
||||
}
|
||||
// 纯标点/符号(无任何字母/数字/汉字)
|
||||
if !hasSemanticChar(e) {
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// functionChars 单字虚词/量词集合:整串都由这些字组成 ⇒ 无信息量。
|
||||
var functionChars = map[rune]struct{}{}
|
||||
|
||||
func init() {
|
||||
for _, r := range "的了着过地得所以于与及或而且则即是在有和与就都也还又很更最可会能要没对为从到把被让给吗呢吧啊呀哦嗯这那我你他她它们个条次些种只件篇位名层里上下中前后内外" {
|
||||
functionChars[r] = struct{}{}
|
||||
}
|
||||
}
|
||||
|
||||
func allFunctionChars(s string) bool {
|
||||
for _, r := range s {
|
||||
if _, ok := functionChars[r]; !ok {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func hasSemanticChar(s string) bool {
|
||||
for _, r := range s {
|
||||
switch {
|
||||
case r >= 0x4E00 && r <= 0x9FFF: // 汉字
|
||||
return true
|
||||
case r >= 'a' && r <= 'z', r >= 'A' && r <= 'Z':
|
||||
return true
|
||||
case r >= '0' && r <= '9':
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// FilterEntitiesForGraph 批量过滤垃圾实体,返回保留集与被拒数(观测用)。
|
||||
func FilterEntitiesForGraph(entities []string) ([]string, int) {
|
||||
if len(entities) == 0 {
|
||||
return entities, 0
|
||||
}
|
||||
out := make([]string, 0, len(entities))
|
||||
rejected := 0
|
||||
for _, e := range entities {
|
||||
if IsJunkEntity(e) {
|
||||
rejected++
|
||||
continue
|
||||
}
|
||||
out = append(out, e)
|
||||
}
|
||||
return out, rejected
|
||||
}
|
||||
@@ -0,0 +1,44 @@
|
||||
package governance
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestIsJunkEntity(t *testing.T) {
|
||||
junk := []string{
|
||||
"条", "个", "里", "的", "了", "这个", "那些", "里存着", "开发", "进行", "the", "and",
|
||||
"一个", "多条", "存在", "什么", "为什么", "", " ", ",。", "的了着",
|
||||
}
|
||||
for _, e := range junk {
|
||||
if !IsJunkEntity(e) {
|
||||
t.Errorf("%q 应判为垃圾实体", e)
|
||||
}
|
||||
}
|
||||
keep := []string{
|
||||
"织忆", "MemoryWeave", "淇县", "生物制造", "卫都街道", "bge-m3", "KOCR", "llama-server-4b",
|
||||
"8081", "graph.db", "牧尘", "飞书",
|
||||
}
|
||||
for _, e := range keep {
|
||||
if IsJunkEntity(e) {
|
||||
t.Errorf("%q 是专名/标识符,不应被判为垃圾", e)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestFilterEntitiesForGraph(t *testing.T) {
|
||||
in := []string{"织忆", "条", "这个", "8081", "里存着"}
|
||||
out, rejected := FilterEntitiesForGraph(in)
|
||||
if rejected != 3 {
|
||||
t.Fatalf("应拒绝 3 个垃圾实体,实际 %d(输出 %v)", rejected, out)
|
||||
}
|
||||
if len(out) != 2 || out[0] != "织忆" || out[1] != "8081" {
|
||||
t.Fatalf("保留集错误:%v", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAllFunctionChars(t *testing.T) {
|
||||
if !allFunctionChars("的了着") {
|
||||
t.Fatal("全虚词串应判 true")
|
||||
}
|
||||
if allFunctionChars("织忆") {
|
||||
t.Fatal("专名不应判 true")
|
||||
}
|
||||
}
|
||||
@@ -115,6 +115,11 @@ func (agu *AutoGraphUpdater) UpdateFromDistill(distilled *DistillInput) {
|
||||
entity, distilled.EpisodeID)
|
||||
continue
|
||||
}
|
||||
cleaned := cleanEntityName(entity)
|
||||
if IsJunkName(cleaned) {
|
||||
log.Printf("[graph] skip-junk-entity=%q cleaned=%q", entity, cleaned)
|
||||
continue
|
||||
}
|
||||
validEnts = append(validEnts, entity)
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,205 @@
|
||||
// 织忆 MemoryWeave — 图谱扩展(ExpandFromResults)锁死 + 实测(t_818f77be)
|
||||
//
|
||||
// 背景:旧 InMemoryGraph/FileGraph 的 ExpandFromResults 拿 r.Category(类目名,如 "distilled")
|
||||
// 当实体名去 Navigate,永远匹配不到任何图节点 ⇒ 结构性零产出。修正后导航起点 = 从 r.Content
|
||||
// 提实体(复用 extract*Entities + normalize*EntityID,与 ExpandWithSummary 同一套逻辑)。
|
||||
//
|
||||
// 本文件两类断言:
|
||||
// 1) 锁死(AC#4):构造小图 + 一条 recall ⇒ 结果必含 category=graph_expanded,且 Score=0.5 不变。
|
||||
// 2) 实测(AC#3):同批 10 条 recall ⇒ 补入率 > 0,给出具体数字。
|
||||
// 判读方式:改前树副本跑本测试 ⇒ 两条均得 0(复现结构零);改后树 ⇒ 均 >0。
|
||||
//
|
||||
// 说明:FileGraph.AddNode/AddEdge 存在潜伏死锁(save() 在写锁内又取 RLock ⇒ 非重入 RWMutex 自锁),
|
||||
// 故 FileGraph 用例改走「写 JSON 文件 + NewFileGraph 加载」,绕开该潜伏 bug(不在本卡修复范围)。
|
||||
package governance
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"github.com/xiaoxue/memoryweave/internal/models"
|
||||
)
|
||||
|
||||
// testNS 固定命名空间,用于证明「实体修对 + namespace 对上 ⇒ 补入 >0」,
|
||||
// 以及「namespace 对不上 ⇒ 二次阻塞补 0」(次级怀疑)。
|
||||
const testNS = "t-ns-expand"
|
||||
|
||||
// chainNodes 一条 4 节点链(中文实体,与生产 graph.db 节点命名一致:n_<名字>)。
|
||||
// 抽取器 extractPotentialEntitiesFromContent 对「2~8 字中文串」最可靠(英文带空格提不到),
|
||||
// 故实体一律用中文名;边端点是归一后形态(n_织忆 / n_图谱 / n_召回 / n_向量)。
|
||||
var (
|
||||
chainNodeIDs = []string{"n_织忆", "n_图谱", "n_召回", "n_向量"}
|
||||
chainNodeNames = map[string]string{"n_织忆": "织忆", "n_图谱": "图谱", "n_召回": "召回", "n_向量": "向量"}
|
||||
chainEdges = []struct{ id, src, dst string }{
|
||||
{"e-ab", "n_织忆", "n_图谱"},
|
||||
{"e-bc", "n_图谱", "n_召回"},
|
||||
{"e-cd", "n_召回", "n_向量"},
|
||||
}
|
||||
)
|
||||
|
||||
// addChainInMemory 给 InMemoryGraph 灌入链(AddNode/AddEdge 无死锁,可直接用)。
|
||||
func addChainInMemory(g *InMemoryGraph) {
|
||||
for _, id := range chainNodeIDs {
|
||||
_ = g.AddNode(id, chainNodeNames[id], "entity", testNS)
|
||||
}
|
||||
for _, e := range chainEdges {
|
||||
_ = g.AddEdge(e.id, e.src, e.dst, "LINKS_TO", testNS, 1.0)
|
||||
}
|
||||
}
|
||||
|
||||
// buildFileGraphJSON 把链写成 JSON 文件,再 NewFileGraph 加载 —— 绕开 AddNode/AddEdge 潜伏死锁。
|
||||
func buildFileGraphJSON(t *testing.T) (*FileGraph, string) {
|
||||
t.Helper()
|
||||
path := filepath.Join(t.TempDir(), "graph-expand-test.json")
|
||||
data := FileGraphData{Version: 2}
|
||||
for _, id := range chainNodeIDs {
|
||||
data.Nodes = append(data.Nodes, &FileGraphNode{
|
||||
ID: id, Name: chainNodeNames[id], Type: "entity",
|
||||
Namespace: testNS, PageRank: 0.15, EvidenceCount: 1,
|
||||
})
|
||||
}
|
||||
for _, e := range chainEdges {
|
||||
data.Edges = append(data.Edges, &FileGraphEdge{
|
||||
ID: e.id, Source: e.src, Target: e.dst,
|
||||
Relation: "LINKS_TO", Weight: 1.0, Namespace: testNS,
|
||||
})
|
||||
}
|
||||
if err := os.WriteFile(path, mustMarshal(data), 0644); err != nil {
|
||||
t.Fatalf("write filegraph json: %v", err)
|
||||
}
|
||||
fg, err := NewFileGraph(path)
|
||||
if err != nil {
|
||||
t.Fatalf("NewFileGraph: %v", err)
|
||||
}
|
||||
return fg, path
|
||||
}
|
||||
|
||||
func mustMarshal(v interface{}) []byte {
|
||||
b, err := json.Marshal(v)
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
// tenRecalls 构造 10 条 recall 结果:正文含中文实体「织忆 图谱 召回(±向量)」
|
||||
// (⇒ 归一 n_织忆/n_图谱/n_召回/n_向量,均在图内),类目名保持 "distilled"
|
||||
// (与生产实测一致,证明导航起点绝不能是类目名)。
|
||||
func tenRecalls() []models.RecallResult {
|
||||
contents := []string{
|
||||
"织忆 图谱 召回 的扩展机制",
|
||||
"图谱 与 召回 相关实体",
|
||||
"召回 通过 向量 关联 织忆",
|
||||
"织忆 图谱 召回 向量 的拓扑",
|
||||
"向量 在最末端;织忆 是起点",
|
||||
"recall 提及 织忆 图谱 召回",
|
||||
"链路 织忆 图谱 召回 向量",
|
||||
"织忆 节点 见 于 distilled 记录",
|
||||
"图谱 召回 织忆 共现",
|
||||
"最终 recall:织忆 图谱 召回 向量",
|
||||
}
|
||||
out := make([]models.RecallResult, 0, len(contents))
|
||||
for i, c := range contents {
|
||||
out = append(out, models.RecallResult{
|
||||
ID: "mem_expand_" + string(rune('a'+i)),
|
||||
Content: c,
|
||||
Category: "distilled",
|
||||
Score: 0.9,
|
||||
Namespace: testNS,
|
||||
})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// countGraphExpanded 逐条跑 10 条 recall,返回每调用 graph_expanded 条数 + 总和,并校验
|
||||
// 每条补入结果 category=graph_expanded 且 Score=0.5(AC#5:不动 0.5 降权)。
|
||||
func countGraphExpanded(g interface {
|
||||
ExpandFromResults(results []models.RecallResult, namespace string, maxHops int) []models.RecallResult
|
||||
}, t *testing.T, ns string) (perCall []int, total int) {
|
||||
t.Helper()
|
||||
recalls := tenRecalls()
|
||||
for i, r := range recalls {
|
||||
got := g.ExpandFromResults([]models.RecallResult{r}, ns, 1)
|
||||
for _, e := range got {
|
||||
if e.Category != "graph_expanded" {
|
||||
t.Fatalf("call %d: 补入结果 category=%q 期望 graph_expanded", i, e.Category)
|
||||
}
|
||||
if e.Score != 0.5 {
|
||||
t.Fatalf("call %d: 补入结果 score=%v 期望 0.5(AC#5 不改降权)", i, e.Score)
|
||||
}
|
||||
}
|
||||
perCall = append(perCall, len(got))
|
||||
total += len(got)
|
||||
}
|
||||
return perCall, total
|
||||
}
|
||||
|
||||
// ── 锁死(AC#4):InMemoryGraph ──────────────────────────────────
|
||||
// 改前树必 FAIL(起点=类目 ⇒ 0 补入);改后树 PASS。
|
||||
func TestInMemoryGraph_ExpandFromResults_Inserts(t *testing.T) {
|
||||
g := NewInMemoryGraph()
|
||||
addChainInMemory(g)
|
||||
|
||||
recalls := tenRecalls()
|
||||
got := g.ExpandFromResults(recalls, testNS, 1)
|
||||
if len(got) == 0 {
|
||||
t.Fatalf("InMemoryGraph ExpandFromResults 补 0 条 — 导航起点仍非实体(结构性零产出未修)")
|
||||
}
|
||||
for _, e := range got {
|
||||
if e.Category != "graph_expanded" {
|
||||
t.Fatalf("category=%q 期望 graph_expanded", e.Category)
|
||||
}
|
||||
}
|
||||
t.Logf("InMemoryGraph: 一条 10-recall 批量补入 %d 条 graph_expanded", len(got))
|
||||
}
|
||||
|
||||
// ── 锁死(AC#4):FileGraph ─────────────────────────────────────
|
||||
func TestFileGraph_ExpandFromResults_Inserts(t *testing.T) {
|
||||
fg, _ := buildFileGraphJSON(t)
|
||||
|
||||
recalls := tenRecalls()
|
||||
got := fg.ExpandFromResults(recalls, testNS, 1)
|
||||
if len(got) == 0 {
|
||||
t.Fatalf("FileGraph ExpandFromResults 补 0 条 — 导航起点仍非实体(结构性零产出未修)")
|
||||
}
|
||||
for _, e := range got {
|
||||
if e.Category != "graph_expanded" {
|
||||
t.Fatalf("category=%q 期望 graph_expanded", e.Category)
|
||||
}
|
||||
}
|
||||
t.Logf("FileGraph: 一条 10-recall 批量补入 %d 条 graph_expanded", len(got))
|
||||
}
|
||||
|
||||
// ── 次级怀疑(AC「namespace 不二次阻塞」)────────────────────────
|
||||
// 实体修对、但传错 namespace ⇒ 应补 0(证明 namespace 是硬约束,需对上才有产出)。
|
||||
func TestInMemoryGraph_ExpandFromResults_NamespaceBlock(t *testing.T) {
|
||||
g := NewInMemoryGraph()
|
||||
addChainInMemory(g)
|
||||
|
||||
got := g.ExpandFromResults(tenRecalls(), "no-such-ns", 1)
|
||||
if len(got) != 0 {
|
||||
t.Fatalf("namespace 对不上却补入 %d 条 — 预期 0(namespace 过滤应生效)", len(got))
|
||||
}
|
||||
t.Logf("namespace 不匹配 ⇒ 补 0(证明 namespace 为硬约束;对上即有产出)")
|
||||
}
|
||||
|
||||
// ── 实测(AC#3):同批 10 条 recall 补入率 ──────────────────────
|
||||
// 改前树:perCall 全 0 ⇒ total 0(复现结构零)。改后树:total>0,输出具体数字。
|
||||
func TestExpandFromResults_TenRecall_Empirical(t *testing.T) {
|
||||
gm := NewInMemoryGraph()
|
||||
addChainInMemory(gm)
|
||||
pIn, totalIn := countGraphExpanded(gm, t, testNS)
|
||||
t.Logf("InMemoryGraph 10-recall 补入: per-call=%v total=%d", pIn, totalIn)
|
||||
if totalIn == 0 {
|
||||
t.Fatalf("InMemoryGraph 改后仍补 0(10/10 全 0)— 结构性零产出未消除")
|
||||
}
|
||||
|
||||
fg, _ := buildFileGraphJSON(t)
|
||||
pF, totalF := countGraphExpanded(fg, t, testNS)
|
||||
t.Logf("FileGraph 10-recall 补入: per-call=%v total=%d", pF, totalF)
|
||||
if totalF == 0 {
|
||||
t.Fatalf("FileGraph 改后仍补 0(10/10 全 0)— 结构性零产出未消除")
|
||||
}
|
||||
}
|
||||
@@ -3,44 +3,87 @@ package governance
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log"
|
||||
"os"
|
||||
"strings"
|
||||
|
||||
"github.com/xiaoxue/memoryweave/internal/models"
|
||||
)
|
||||
|
||||
// graphExpandDebugEnabled 控制「图谱扩展」debug 日志(AC 可观测性,t_818f77be)。
|
||||
// 默认关;设 ZHIYI_GRAPH_EXPAND_DEBUG=1 打开。每次 recall 至多一条(低频),避免刷屏。
|
||||
func graphExpandDebugEnabled() bool {
|
||||
return os.Getenv("ZHIYI_GRAPH_EXPAND_DEBUG") == "1"
|
||||
}
|
||||
|
||||
// ExpandFromResults 从 recall 结果出发,双向 BFS 扩展图谱邻接节点
|
||||
// 实现 GraphStore 接口
|
||||
//
|
||||
// 导航起点 = 从 r.Content 提实体(复用 extractPotentialEntitiesFromContent + normalizeEntityID,
|
||||
// 与同文件 ExpandWithSummary 同一套逻辑)。旧实现拿 r.Category(类目名,如 "distilled")
|
||||
// 当实体去 Navigate,永远匹配不到任何图节点 ⇒ 结构性零产出(t_818f77be 实测 10/10 全 distilled)。
|
||||
// 且 InMemoryGraph.Navigate 返回字段是 source/target(非 to/from),旧读法 p["to"]/p["from"]
|
||||
// 恒取到空 ⇒ 即便实体修对也补 0。两处一并修正。
|
||||
func (g *InMemoryGraph) ExpandFromResults(results []models.RecallResult, namespace string, maxHops int) []models.RecallResult {
|
||||
var expanded []models.RecallResult
|
||||
seen := make(map[string]bool)
|
||||
seenEntities := make(map[string]bool)
|
||||
|
||||
// 收集已有结果 ID
|
||||
for _, r := range results {
|
||||
seen[r.ID] = true
|
||||
}
|
||||
|
||||
// 从每个结果出发扩展
|
||||
startEntities := 0 // 可观测性:去重后实际导航的起点实体数 N
|
||||
totalPathsHit := 0 // 可观测性:命中路径总数 M(补前去重)
|
||||
rejectedJunk := 0 // P0-4:被质量闸①丢弃的垃圾实体数
|
||||
rejectedLowDegree := 0 // P0-4:被质量闸②丢弃的低度数实体数
|
||||
// 从每个结果正文提实体出发扩展
|
||||
for _, r := range results {
|
||||
paths, err := g.Navigate(r.Category, maxHops, namespace, nil)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
for _, p := range paths {
|
||||
to, _ := p["to"].(string)
|
||||
from, _ := p["from"].(string)
|
||||
entities := extractPotentialEntitiesFromContent(r.Content)
|
||||
// P0-4 第①道闸:垃圾实体不导航
|
||||
entities, rejected := FilterEntitiesForGraph(entities)
|
||||
rejectedJunk += rejected
|
||||
for _, entity := range entities {
|
||||
if seenEntities[entity] {
|
||||
continue
|
||||
}
|
||||
seenEntities[entity] = true
|
||||
startEntities++
|
||||
|
||||
for _, id := range []string{to, from} {
|
||||
if id != "" && !seen[id] {
|
||||
seen[id] = true
|
||||
expanded = append(expanded, models.RecallResult{
|
||||
ID: id,
|
||||
Category: "graph_expanded",
|
||||
Score: 0.5, // 图谱扩展降权
|
||||
})
|
||||
nodeID := normalizeEntityID(entity)
|
||||
paths, err := g.Navigate(nodeID, maxHops, namespace, nil)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
// P0-4 第②道闸:路径数 <2 ⇒ 孤边节点,不构成语义关联(内存图无 degree 查询,用路径数代理)
|
||||
if len(paths) < 2 {
|
||||
rejectedLowDegree++
|
||||
continue
|
||||
}
|
||||
totalPathsHit += len(paths)
|
||||
for _, p := range paths {
|
||||
to, _ := p["target"].(string)
|
||||
from, _ := p["source"].(string)
|
||||
|
||||
for _, id := range []string{to, from} {
|
||||
if id != "" && !seen[id] {
|
||||
seen[id] = true
|
||||
expanded = append(expanded, models.RecallResult{
|
||||
ID: id,
|
||||
Category: "graph_expanded",
|
||||
Score: 0.5, // 图谱扩展降权
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
// 可观测性:默认关(ZHIYI_GRAPH_EXPAND_DEBUG=1 打开),保证「下次坏了不沉默」。
|
||||
if graphExpandDebugEnabled() {
|
||||
log.Printf("[graph-expand] ns=%s maxHops=%d 起点实体 %d 个 / 命中 %d 条 / 补入 %d 条 / 质量闸拒 垃圾 %d 低度 %d",
|
||||
namespace, maxHops, startEntities, totalPathsHit, len(expanded), rejectedJunk, rejectedLowDegree)
|
||||
}
|
||||
return expanded
|
||||
}
|
||||
|
||||
|
||||
@@ -5,6 +5,7 @@ package governance
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log"
|
||||
"math"
|
||||
"os"
|
||||
"strings"
|
||||
@@ -457,32 +458,63 @@ func (fg *FileGraph) UpdateEdgeTrustScores() error { return
|
||||
func (fg *FileGraph) ExpandFromResults(results []models.RecallResult, namespace string, maxHops int) []models.RecallResult {
|
||||
var expanded []models.RecallResult
|
||||
seen := make(map[string]bool)
|
||||
seenEntities := make(map[string]bool)
|
||||
|
||||
for _, r := range results {
|
||||
seen[r.ID] = true
|
||||
}
|
||||
|
||||
// 导航起点 = 从 r.Content 提实体(复用 extractFileGraphEntities + normalizeFileGraphEntityID,
|
||||
// 与同文件 ExpandWithSummary 同一套逻辑)。旧实现拿 r.Category(类目名,如 "distilled")
|
||||
// 当实体去 Navigate,永远匹配不到任何图节点 ⇒ 结构性零产出(t_818f77be)。
|
||||
startEntities := 0 // 可观测性:去重后实际导航的起点实体数 N
|
||||
totalPathsHit := 0 // 可观测性:命中路径总数 M(补前去重)
|
||||
rejectedJunk := 0 // P0-4:质量闸①拒绝数
|
||||
rejectedLowDegree := 0 // P0-4:质量闸②拒绝数
|
||||
for _, r := range results {
|
||||
paths, err := fg.Navigate(r.Category, maxHops, namespace, nil)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
for _, p := range paths {
|
||||
target, _ := p["target"].(string)
|
||||
source, _ := p["source"].(string)
|
||||
entities := extractFileGraphEntities(r.Content)
|
||||
entities, rejected := FilterEntitiesForGraph(entities)
|
||||
rejectedJunk += rejected
|
||||
for _, entity := range entities {
|
||||
if seenEntities[entity] {
|
||||
continue
|
||||
}
|
||||
seenEntities[entity] = true
|
||||
startEntities++
|
||||
|
||||
for _, id := range []string{target, source} {
|
||||
if id != "" && !seen[id] {
|
||||
seen[id] = true
|
||||
expanded = append(expanded, models.RecallResult{
|
||||
ID: id,
|
||||
Category: "graph_expanded",
|
||||
Score: 0.5,
|
||||
})
|
||||
nodeID := normalizeFileGraphEntityID(entity)
|
||||
paths, err := fg.Navigate(nodeID, maxHops, namespace, nil)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
// P0-4 第②道闸:路径数 <2 ⇒ 孤边节点不注入
|
||||
if len(paths) < 2 {
|
||||
rejectedLowDegree++
|
||||
continue
|
||||
}
|
||||
totalPathsHit += len(paths)
|
||||
for _, p := range paths {
|
||||
target, _ := p["target"].(string)
|
||||
source, _ := p["source"].(string)
|
||||
|
||||
for _, id := range []string{target, source} {
|
||||
if id != "" && !seen[id] {
|
||||
seen[id] = true
|
||||
expanded = append(expanded, models.RecallResult{
|
||||
ID: id,
|
||||
Category: "graph_expanded",
|
||||
Score: 0.5,
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
// 可观测性:默认关(ZHIYI_GRAPH_EXPAND_DEBUG=1 打开),保证「下次坏了不沉默」。
|
||||
if graphExpandDebugEnabled() {
|
||||
log.Printf("[graph-expand] ns=%s maxHops=%d 起点实体 %d 个 / 命中 %d 条 / 补入 %d 条 / 质量闸拒 垃圾 %d 低度 %d",
|
||||
namespace, maxHops, startEntities, totalPathsHit, len(expanded), rejectedJunk, rejectedLowDegree)
|
||||
}
|
||||
return expanded
|
||||
}
|
||||
|
||||
|
||||
@@ -3,6 +3,7 @@ package governance
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"strings"
|
||||
"sync"
|
||||
)
|
||||
|
||||
@@ -295,6 +296,22 @@ func (g *InMemoryGraph) AllEdges(namespace string) []GraphEdge {
|
||||
return out
|
||||
}
|
||||
|
||||
// CountNodesByType 内存实现:无时间列,lastUpdated 恒为空(调用方按 unknown_age 处理)。
|
||||
func (g *InMemoryGraph) CountNodesByType(typ string) (total, withID int, lastUpdated string) {
|
||||
g.mu.RLock()
|
||||
defer g.mu.RUnlock()
|
||||
for _, n := range g.nodes {
|
||||
if n.Type != typ {
|
||||
continue
|
||||
}
|
||||
total++
|
||||
if strings.TrimSpace(n.ID) != "" {
|
||||
withID++
|
||||
}
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
// Stats 返回图谱统计
|
||||
func (g *InMemoryGraph) Stats() (nodeCount, edgeCount int, density float64) {
|
||||
g.mu.RLock()
|
||||
|
||||
@@ -16,6 +16,7 @@ import "C"
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
@@ -216,11 +217,33 @@ func (gs *SQLiteGraphStore) AddEdge(id, source, target, relation, namespace stri
|
||||
return execSQL(gs.db, sql)
|
||||
}
|
||||
|
||||
func (gs *SQLiteGraphStore) isQuarantinedNode(nodeID string) (bool, error) {
|
||||
sql := fmt.Sprintf(
|
||||
"SELECT namespace, pagerank FROM graph_nodes WHERE id='%s'", escape(nodeID))
|
||||
rows := queryRows(gs.db, sql)
|
||||
if len(rows) == 0 {
|
||||
// node not found; treat as not quarantined to avoid breaking existing flows
|
||||
return false, nil
|
||||
}
|
||||
row := rows[0]
|
||||
namespace, _ := row["namespace"].(string)
|
||||
pagerankVal, _ := row["pagerank"].(float64)
|
||||
return namespace == "quarantine" && pagerankVal == 0, nil
|
||||
}
|
||||
|
||||
func (gs *SQLiteGraphStore) Navigate(entity string, maxHops int, namespace string, relFilter []string) ([]map[string]interface{}, error) {
|
||||
// 单源 BFS:从 entity 展开到邻居,不找路径
|
||||
// E1.4: relFilter 白名单过滤关系类型
|
||||
gs.mu.RLock()
|
||||
defer gs.mu.RUnlock()
|
||||
if quarantined, err := gs.isQuarantinedNode(entity); err != nil {
|
||||
// If we cannot determine, treat as not quarantined to avoid breaking existing flows.
|
||||
if err != nil {
|
||||
// Continue; assume not quarantined.
|
||||
} else if quarantined {
|
||||
return []map[string]interface{}{}, nil
|
||||
}
|
||||
}
|
||||
|
||||
nsClause := buildNamespaceClause(namespace)
|
||||
relClause := buildRelationFilterClause(relFilter)
|
||||
@@ -233,7 +256,7 @@ func (gs *SQLiteGraphStore) Navigate(entity string, maxHops int, namespace strin
|
||||
var next []string
|
||||
for _, node := range queue {
|
||||
sql := fmt.Sprintf(
|
||||
"SELECT e.id, e.target, e.relation, e.weight, n.name FROM graph_edges e JOIN graph_nodes n ON e.target = n.id WHERE e.source = '%s' AND %s AND %s",
|
||||
"SELECT e.id, e.target, e.relation, e.weight, n.name FROM graph_edges e JOIN graph_nodes n ON e.target = n.id WHERE e.source = '%s' AND %s AND %s AND n.namespace != 'quarantine' AND n.pagerank > 0",
|
||||
escape(node), nsClause, relClause)
|
||||
edges := queryRows(gs.db, sql)
|
||||
for _, edge := range edges {
|
||||
@@ -269,6 +292,20 @@ func (gs *SQLiteGraphStore) NavigateBiDir(source, target string, maxHops int, na
|
||||
|
||||
gs.mu.RLock()
|
||||
defer gs.mu.RUnlock()
|
||||
if quarantinedSrc, err := gs.isQuarantinedNode(source); err != nil {
|
||||
if err != nil {
|
||||
// Continue; assume not quarantined.
|
||||
} else if quarantinedSrc {
|
||||
return []map[string]interface{}{}, nil
|
||||
}
|
||||
}
|
||||
if quarantinedTgt, err := gs.isQuarantinedNode(target); err != nil {
|
||||
if err != nil {
|
||||
// Continue; assume not quarantined.
|
||||
} else if quarantinedTgt {
|
||||
return []map[string]interface{}{}, nil
|
||||
}
|
||||
}
|
||||
|
||||
nsClause := buildNamespaceClause(namespace)
|
||||
|
||||
@@ -310,7 +347,7 @@ func (gs *SQLiteGraphStore) NavigateBiDir(source, target string, maxHops int, na
|
||||
var next []string
|
||||
for _, node := range fwdQ {
|
||||
sql := fmt.Sprintf(
|
||||
"SELECT e.id, e.target, e.relation, e.weight FROM graph_edges e WHERE e.source = '%s' AND %s",
|
||||
"SELECT e.id, e.target, e.relation, e.weight FROM graph_edges e JOIN graph_nodes n ON e.target = n.id WHERE e.source = '%s' AND %s AND n.namespace != 'quarantine' AND n.pagerank > 0",
|
||||
escape(node), nsClause)
|
||||
edges := queryRows(gs.db, sql)
|
||||
for _, edge := range edges {
|
||||
@@ -340,7 +377,7 @@ func (gs *SQLiteGraphStore) NavigateBiDir(source, target string, maxHops int, na
|
||||
for _, node := range bwdQ {
|
||||
// 反向要找所有指向 node 的边(target = node)
|
||||
sql := fmt.Sprintf(
|
||||
"SELECT e.id, e.source, e.relation, e.weight FROM graph_edges e WHERE e.target = '%s' AND %s",
|
||||
"SELECT e.id, e.source, e.relation, e.weight FROM graph_edges e JOIN graph_nodes n ON e.source = n.id WHERE e.target = '%s' AND %s AND n.namespace != 'quarantine' AND n.pagerank > 0",
|
||||
escape(node), nsClause)
|
||||
edges := queryRows(gs.db, sql)
|
||||
for _, edge := range edges {
|
||||
@@ -548,6 +585,32 @@ func (gs *SQLiteGraphStore) Stats() (nodeCount, edgeCount int, density float64)
|
||||
return
|
||||
}
|
||||
|
||||
// CountNodesByType 返回某类节点的总量、id 非空量、最近一次更新时间(2026-10-26)。
|
||||
//
|
||||
// 存在的理由:selfcheck 需要暴露「pattern 召回有没有生产者」这一可观测事实。
|
||||
// 事故背景:`POST /api/v1/patterns/detect` 是短命单发端点,全库 0 cron、0 调用方,
|
||||
// 于是 graph_nodes(type='pattern') 恒 0 行、recall 恒 0 命中,而日志层面完全沉默。
|
||||
// 零产出必须能被机器读到,不能靠人翻 7 天日志才发现。
|
||||
func (gs *SQLiteGraphStore) CountNodesByType(typ string) (total, withID int, lastUpdated string) {
|
||||
gs.mu.RLock()
|
||||
defer gs.mu.RUnlock()
|
||||
|
||||
t := escape(typ)
|
||||
total = queryInt(gs.db, fmt.Sprintf("SELECT COUNT(*) FROM graph_nodes WHERE type='%s'", t))
|
||||
withID = queryInt(gs.db, fmt.Sprintf(
|
||||
"SELECT COUNT(*) FROM graph_nodes WHERE type='%s' AND id IS NOT NULL AND TRIM(id)<>''", t))
|
||||
rows := queryRows(gs.db, fmt.Sprintf(
|
||||
"SELECT MAX(last_updated_at) AS m FROM graph_nodes WHERE type='%s'", t))
|
||||
if len(rows) > 0 && rows[0]["m"] != nil {
|
||||
if v, ok := rows[0]["m"].(string); ok {
|
||||
lastUpdated = v
|
||||
} else {
|
||||
lastUpdated = fmt.Sprintf("%v", rows[0]["m"])
|
||||
}
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
func (gs *SQLiteGraphStore) Prune(minWeight float64) {
|
||||
gs.mu.Lock()
|
||||
defer gs.mu.Unlock()
|
||||
@@ -755,6 +818,8 @@ func (gs *SQLiteGraphStore) ExpandFromResults(results []models.RecallResult, nam
|
||||
|
||||
seen := make(map[string]bool) // 追踪已访问的记忆 ID
|
||||
seenEntities := make(map[string]bool) // 追踪已访问的实体(用于去重扩展结果)
|
||||
rejectedJunk := 0
|
||||
rejectedLowDegree := 0
|
||||
for _, r := range results {
|
||||
if seen[r.ID] {
|
||||
continue
|
||||
@@ -762,6 +827,9 @@ func (gs *SQLiteGraphStore) ExpandFromResults(results []models.RecallResult, nam
|
||||
seen[r.ID] = true
|
||||
|
||||
entities := extractPotentialEntities(r.Content)
|
||||
// P0-4 第①道闸:垃圾实体(虚词/量词/代词/填充短语)不导航
|
||||
entities, rejected := FilterEntitiesForGraph(entities)
|
||||
rejectedJunk += rejected
|
||||
for _, entity := range entities {
|
||||
if seenEntities[entity] {
|
||||
continue
|
||||
@@ -769,18 +837,25 @@ func (gs *SQLiteGraphStore) ExpandFromResults(results []models.RecallResult, nam
|
||||
seenEntities[entity] = true
|
||||
|
||||
// SQLite 图谱节点 ID 格式: n_{entity_name},需 normalizeEntityID 转换
|
||||
// Navigate 的 from/to 都是**节点 ID**(n_织忆),不能用裸名 entity 比对,
|
||||
// 否则恒不相等 ⇒ 路径全被 continue ⇒ 结构性 0 条附加(t_be4d070a 诊断)。
|
||||
nodeID := normalizeEntityID(entity)
|
||||
// P0-4 第②道闸:度数 <2 的孤边节点不构成语义关联 ⇒ 不导航、不注入。
|
||||
if gs.EdgeDegree(nodeID) < 2 {
|
||||
rejectedLowDegree++
|
||||
continue
|
||||
}
|
||||
paths, _ := gs.Navigate(nodeID, maxHops, namespace, nil)
|
||||
for _, p := range paths {
|
||||
// Navigate 返回字段: from, to, relation, weight, hop
|
||||
// Navigate 返回字段: from, to, relation, weight, hop(均为节点 ID)
|
||||
var neighbor, rel string
|
||||
from, _ := p["from"].(string)
|
||||
to, _ := p["to"].(string)
|
||||
relVal, _ := p["relation"].(string)
|
||||
if from == entity && to != "" {
|
||||
if from == nodeID && to != "" {
|
||||
neighbor = to
|
||||
rel = relVal
|
||||
} else if to == entity && from != "" {
|
||||
} else if to == nodeID && from != "" {
|
||||
neighbor = from
|
||||
rel = "↩ " + relVal
|
||||
}
|
||||
@@ -792,15 +867,62 @@ func (gs *SQLiteGraphStore) ExpandFromResults(results []models.RecallResult, nam
|
||||
}
|
||||
seenEntities[neighbor] = true
|
||||
expanded = append(expanded, models.RecallResult{
|
||||
Content: fmt.Sprintf("[graph] %s --[%s]--> %s", entity, rel, neighbor),
|
||||
// ID 用邻居节点 ID(n_<实体> 或 n_mem_<记忆id>),非空才不破坏 recall.go 的
|
||||
// ID 去重(Step 5 按 r.ID==e.ID 判已存在);若邻居是记忆节点,ID 即真实记忆指针。
|
||||
ID: neighbor,
|
||||
Category: "graph_expanded",
|
||||
Content: fmt.Sprintf("[graph] %s --[%s]--> %s", entity, rel, displayNodeName(neighbor)),
|
||||
Score: r.Score * 0.5,
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
// P0-4 可观测性:两道闸的拒绝量必须能读出来(默认关,debug 开才打印,避免刷屏)。
|
||||
if graphExpandDebugEnabled() && (rejectedJunk > 0 || rejectedLowDegree > 0) {
|
||||
log.Printf("[graph-expand] ns=%s 质量闸:垃圾实体拒 %d / 低度数拒 %d / 补入 %d",
|
||||
namespace, rejectedJunk, rejectedLowDegree, len(expanded)-len(results))
|
||||
}
|
||||
return expanded
|
||||
}
|
||||
|
||||
// EdgeDegree 返回节点在图中的边数(入度+出度,P0-4 度数闸用)。
|
||||
// 与 GetEntityDegree 的区别:后者返回 evidence_count 之和(证据强度),
|
||||
// 本函数才是字面意义的「度数」—— 孤边节点(度=1)不足以构成语义关联。
|
||||
func (gs *SQLiteGraphStore) EdgeDegree(nodeID string) int {
|
||||
if nodeID == "" {
|
||||
return 0
|
||||
}
|
||||
gs.mu.RLock()
|
||||
defer gs.mu.RUnlock()
|
||||
rows := queryRows(gs.db, fmt.Sprintf(
|
||||
"SELECT COUNT(*) as c FROM graph_edges WHERE source = '%s' OR target = '%s'",
|
||||
escape(nodeID), escape(nodeID)))
|
||||
if len(rows) == 0 {
|
||||
return 0
|
||||
}
|
||||
switch v := rows[0]["c"].(type) {
|
||||
case int:
|
||||
return v
|
||||
case int64:
|
||||
return int(v)
|
||||
case float64:
|
||||
return int(v)
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// displayNodeName 把节点 ID 转成可读显示名(去 n_ / n_mem_ / mem_ 前缀)。
|
||||
// 图节点 ID 形态:实体 n_织忆 / 概念 n_mem_<记忆id>(name 列才是 mem_<id>)。
|
||||
// 仅用于格式化 Content 的可读展示,不做任何记忆正文编造。
|
||||
func displayNodeName(nodeID string) string {
|
||||
for _, p := range []string{"n_mem_", "n_", "mem_", "ep_"} {
|
||||
if strings.HasPrefix(nodeID, p) {
|
||||
return strings.TrimPrefix(nodeID, p)
|
||||
}
|
||||
}
|
||||
return nodeID
|
||||
}
|
||||
|
||||
// ExpandWithSummary BFS 扩展 + 生成汇总语句 — E1 图谱导航增强
|
||||
// 从 recall 结果提取实体,进行多跳扩展,返回扩展关系列表和一句话汇总
|
||||
func (gs *SQLiteGraphStore) ExpandWithSummary(results []models.RecallResult, namespace string, maxHops int) models.GraphBFSResult {
|
||||
|
||||
@@ -0,0 +1,133 @@
|
||||
// 织忆 MemoryWeave — SQLiteGraphStore.ExpandFromResults 修复锁死测试(t_724f31ad)
|
||||
//
|
||||
// 背景:SQLiteGraphStore.ExpandFromResults(graph_sqlite.go:787)此前拿裸实体名 entity
|
||||
// 与 Navigate 返回的节点 ID(n_织忆)做 == 比对 ⇒ 恒不相等 ⇒ 路径全被 continue ⇒ 结构性 0 条附加。
|
||||
// 且附加结果 Category 空、ID 空(审计 graph_expanded 计数恒 0 + recall.go ID 去重失效)。
|
||||
//
|
||||
// 本文件两类断言:
|
||||
// 1) 正向:构造 n_<实体> 节点 + 一条 recall 结果 ⇒ ExpandFromResults 返回 >len(results),
|
||||
// 且新增项 Category=="graph_expanded" 且 ID!=""(ID 用邻居节点 ID)。
|
||||
// 2) 反向:实体在图里不存在 ⇒ 结果条数不变(不得凭空加)。
|
||||
package governance
|
||||
|
||||
import (
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"github.com/xiaoxue/memoryweave/internal/models"
|
||||
)
|
||||
|
||||
// testNS724 SQLite 图的 namespace 须用 `-main` 全形态:SQLiteGraphStore.Navigate 经
|
||||
// deriveNamespaceForGraph 会把裸名 ns 变 ns-main 再过滤(与生产 recall.go 传的 namespace 一致)。
|
||||
const testNS724 = "t-ns-expand-sqlite-main"
|
||||
|
||||
// newTestSQLiteGraph 建临时 SQLiteGraphStore 并灌入一条链:n_织忆 -LINKS_TO-> n_图谱。
|
||||
func newTestSQLiteGraph(t *testing.T) *SQLiteGraphStore {
|
||||
t.Helper()
|
||||
path := filepath.Join(t.TempDir(), "graph-test.db")
|
||||
gs, err := NewSQLiteGraphStore(path)
|
||||
if err != nil {
|
||||
t.Fatalf("NewSQLiteGraphStore: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { _ = gs.Close() })
|
||||
if err := gs.AddNode("n_织忆", "织忆", "entity", testNS724); err != nil {
|
||||
t.Fatalf("AddNode n_织忆: %v", err)
|
||||
}
|
||||
if err := gs.AddNode("n_图谱", "图谱", "entity", testNS724); err != nil {
|
||||
t.Fatalf("AddNode n_图谱: %v", err)
|
||||
}
|
||||
if err := gs.AddEdge("e-ab", "n_织忆", "n_图谱", "LINKS_TO", testNS724, 1.0); err != nil {
|
||||
t.Fatalf("AddEdge: %v", err)
|
||||
}
|
||||
// P0-4 度数闸(2026-10-26):degree<2 的孤边节点不再注入 ⇒ 正向用例的起点实体
|
||||
// 必须真的在图里有 ≥2 条边(两跳邻居即真实图谱形态,不是为过测试造数据)。
|
||||
if err := gs.AddNode("n_召回", "召回", "entity", testNS724); err != nil {
|
||||
t.Fatalf("AddNode n_召回: %v", err)
|
||||
}
|
||||
if err := gs.AddEdge("e-ac", "n_织忆", "n_召回", "LINKS_TO", testNS724, 1.0); err != nil {
|
||||
t.Fatalf("AddEdge e-ac: %v", err)
|
||||
}
|
||||
return gs
|
||||
}
|
||||
|
||||
// TestSQLiteGraphStore_DegreeGate P0-4:度数=1 的孤边实体不得被扩展注入。
|
||||
func TestSQLiteGraphStore_DegreeGate(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "graph-degree.db")
|
||||
gs, err := NewSQLiteGraphStore(path)
|
||||
if err != nil {
|
||||
t.Fatalf("NewSQLiteGraphStore: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { _ = gs.Close() })
|
||||
// 孤边:n_孤边 --LINKS_TO--> n_邻居(起点实体度=1)
|
||||
if err := gs.AddNode("n_孤边", "孤边", "entity", testNS724); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := gs.AddNode("n_邻居", "邻居", "entity", testNS724); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := gs.AddEdge("e-one", "n_孤边", "n_邻居", "LINKS_TO", testNS724, 1.0); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := gs.EdgeDegree("n_孤边"); got != 1 {
|
||||
t.Fatalf("EdgeDegree(n_孤边) 应为 1,实际 %d", got)
|
||||
}
|
||||
results := []models.RecallResult{{
|
||||
ID: "mem_degree_1", Content: "孤边 的扩展", Category: "distilled", Score: 0.9, Namespace: testNS724,
|
||||
}}
|
||||
got := gs.ExpandFromResults(results, testNS724, 1)
|
||||
if len(got) != len(results) {
|
||||
t.Fatalf("度数=1 的实体不应扩展注入,实际补入 %d 条", len(got)-len(results))
|
||||
}
|
||||
}
|
||||
|
||||
// TestSQLiteGraphStore_ExpandFromResults_Inserts 正向:实体在图中 ⇒ 补入 graph_expanded 且 ID 非空。
|
||||
func TestSQLiteGraphStore_ExpandFromResults_Inserts(t *testing.T) {
|
||||
gs := newTestSQLiteGraph(t)
|
||||
|
||||
results := []models.RecallResult{{
|
||||
ID: "mem_expand_1",
|
||||
Content: "织忆 与 图谱 的扩展机制",
|
||||
Category: "distilled",
|
||||
Score: 0.9,
|
||||
Namespace: testNS724,
|
||||
}}
|
||||
|
||||
got := gs.ExpandFromResults(results, testNS724, 1)
|
||||
if len(got) <= len(results) {
|
||||
t.Fatalf("期望补入 >len(results)=%d,实际 %d 条(结构性 0 产出未修?)", len(results), len(got))
|
||||
}
|
||||
var added int
|
||||
for _, e := range got[len(results):] {
|
||||
if e.Category != "graph_expanded" {
|
||||
t.Fatalf("补入结果 category=%q 期望 graph_expanded", e.Category)
|
||||
}
|
||||
if e.ID == "" {
|
||||
t.Fatalf("补入结果 ID 为空(ID 去重会失效),Content=%q", e.Content)
|
||||
}
|
||||
added++
|
||||
}
|
||||
if added == 0 {
|
||||
t.Fatal("没有任何 graph_expanded 补入")
|
||||
}
|
||||
t.Logf("SQLiteGraphStore: 1-recall 补入 %d 条 graph_expanded(ID 均非空)", added)
|
||||
}
|
||||
|
||||
// TestSQLiteGraphStore_ExpandFromResults_NoFabrication 反向:实体不在图中 ⇒ 条数不变,不得凭空加。
|
||||
func TestSQLiteGraphStore_ExpandFromResults_NoFabrication(t *testing.T) {
|
||||
gs := newTestSQLiteGraph(t)
|
||||
|
||||
// content 只含一个图中不存在的实体「完全不存在的实体」(切分成 2~8 字中文串均无图节点)。
|
||||
results := []models.RecallResult{{
|
||||
ID: "mem_expand_2",
|
||||
Content: "完全不存在的实体",
|
||||
Category: "distilled",
|
||||
Score: 0.9,
|
||||
Namespace: testNS724,
|
||||
}}
|
||||
|
||||
got := gs.ExpandFromResults(results, testNS724, 1)
|
||||
if len(got) != len(results) {
|
||||
t.Fatalf("实体不在图中却补入 %d 条(期望 0 补入,len=%d)", len(got)-len(results), len(results))
|
||||
}
|
||||
t.Logf("SQLiteGraphStore: 图外实体 ⇒ 0 补入(不凭空加)")
|
||||
}
|
||||
@@ -23,6 +23,9 @@ type GraphStore interface {
|
||||
|
||||
// 统计与维护
|
||||
Stats() (nodeCount, edgeCount int, density float64)
|
||||
// CountNodesByType 返回某类节点的总量、id 非空量、最近更新时间。
|
||||
// 用途:让「写入端有没有产数」可被机器读到(pattern 召回曾恒 0 而日志沉默)。
|
||||
CountNodesByType(typ string) (total, withID int, lastUpdated string)
|
||||
Prune(minWeight float64)
|
||||
|
||||
// 边枚举(供社区发现用)
|
||||
|
||||
@@ -30,6 +30,7 @@ package memprotect
|
||||
|
||||
import (
|
||||
"log"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
@@ -67,6 +68,41 @@ const (
|
||||
RedisKeyProtectedIDs = "zhiyi:protected_ids"
|
||||
)
|
||||
|
||||
// ── GA-P2 强保护栏(2026-10-27)──────────────────────────────────────────────
|
||||
//
|
||||
// 规则5(可选,ZHIYI_PROTECT_RAILS=1 启用):recall_verified_count >= 2 ⇒ 保护。
|
||||
// 口径与 Python 侧 zhiyi_protection.py 的 MIN_VERIFIED_FOR_RAILS 及
|
||||
// ~/.hermes/plugins/zhiyi/protect_rails.py 规则② 逐字同口径 —— 任何改动必须两侧同步,
|
||||
// 禁止两套标准(卡 t_944344b5 边界条款)。
|
||||
//
|
||||
// 政策边界(牧尘 2026-10-20 拍板):rv == 1(只被确认过一次)不纳入保护,
|
||||
// 保留「验证满 2 次才算硬证据」的门槛。
|
||||
//
|
||||
// 默认关(未设/env != "1")⇒ 行为零变化;启用后为纯增保护(只增加「不遗忘」,绝不增加删除)。
|
||||
const MinVerifiedForRails = 2
|
||||
|
||||
// ProtectRailsEnv 强保护栏 env 开关名(与 protect_rails.py / zhiyi_protection.py 同名)。
|
||||
const ProtectRailsEnv = "ZHIYI_PROTECT_RAILS"
|
||||
|
||||
// ProtectRailsEnabled 读 env,每次调用现读(支持不重启热切换,开销可忽略)。
|
||||
func ProtectRailsEnabled() bool {
|
||||
return strings.TrimSpace(os.Getenv(ProtectRailsEnv)) == "1"
|
||||
}
|
||||
|
||||
// rowInt 从候选 map 取整型字段,兼容 int/int64/float64
|
||||
// (IPC 侧候选 map 是 int;直连/其它实现可能给 int64,避免规则因类型断言失败而空转)。
|
||||
func rowInt(row map[string]interface{}, key string) int {
|
||||
switch v := row[key].(type) {
|
||||
case int:
|
||||
return v
|
||||
case int64:
|
||||
return int(v)
|
||||
case float64:
|
||||
return int(v)
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// State 一次 LoadState() 加载的保护状态快照。
|
||||
// 衰减 tick / admin forget 开始时 LoadState 一次,循环内对每条候选复用同一 State,
|
||||
// 避免每条记忆都 HGETALL/SMEMBERS 一次 Redis。
|
||||
@@ -123,6 +159,16 @@ func IsProtected(row map[string]interface{}, st *State) (bool, string) {
|
||||
}
|
||||
}
|
||||
|
||||
// 规则5(GA-P2 强保护栏,可选):recall_verified_count >= 2 ⇒ 保护。
|
||||
// 默认关(ZHIYI_PROTECT_RAILS != "1")⇒ 行为零变化。启用后:被验证使用 ≥2 次的
|
||||
// 记忆永不自动遗忘(纯增保护,不增加任何删除)。字段已在候选 map(lancedb_ipc.go
|
||||
// GetCandidatesForForgetting 已 SELECT recall_verified_count),不依赖 Redis 缓存。
|
||||
if ProtectRailsEnabled() {
|
||||
if rv := rowInt(row, "recall_verified_count"); rv >= MinVerifiedForRails {
|
||||
return true, "recall_verified_ge2"
|
||||
}
|
||||
}
|
||||
|
||||
return false, ""
|
||||
}
|
||||
|
||||
|
||||
@@ -403,3 +403,56 @@ func TestDifferential_FixtureTable(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestRule5_RailsVerifiedCount (GA-P2, 2026-10-27):强保护栏 recall_verified_count>=2 规则。
|
||||
// 口径与 Python zhiyi_protection.py 的 recall_verified>=2 一致;默认关(零行为变化)。
|
||||
// 政策边界(牧尘 2026-10-20 拍板):rv==1 不纳入保护(验证满 2 次才算硬证据)。
|
||||
func TestRule5_RailsVerifiedCount(t *testing.T) {
|
||||
mkRV := func(rv int) map[string]interface{} {
|
||||
return map[string]interface{}{
|
||||
"id": "mem_rails",
|
||||
"category": "normal", // 不命中规则1
|
||||
"freshness": "", // 不命中规则3
|
||||
"content": "普通日常内容", // 不命中规则4b
|
||||
"recall_verified_count": rv,
|
||||
}
|
||||
}
|
||||
st := synthState(nil, nil) // 空 VProp/Protected ⇒ 规则2/4a 不命中
|
||||
|
||||
// 1) 默认关(未设 env):rv 再高也不保护(行为零变化)
|
||||
t.Setenv(ProtectRailsEnv, "")
|
||||
if p, r := IsProtected(mkRV(50), st); p {
|
||||
t.Fatalf("默认关: rv=50 不应保护, got true rule=%s", r)
|
||||
}
|
||||
|
||||
// 2) ENV="1" 启用:rv>=2 保护(含边界 2),rv=1/0 不保护
|
||||
t.Setenv(ProtectRailsEnv, "1")
|
||||
if p, r := IsProtected(mkRV(2), st); !p || r != "recall_verified_ge2" {
|
||||
t.Errorf("rv=2(边界) 应保护 rule=recall_verified_ge2, got %v rule=%s", p, r)
|
||||
}
|
||||
if p, _ := IsProtected(mkRV(5), st); !p {
|
||||
t.Error("rv=5 应保护")
|
||||
}
|
||||
if p, _ := IsProtected(mkRV(1), st); p {
|
||||
t.Error("rv=1 不应保护(政策边界:验证满 2 次才算硬证据)")
|
||||
}
|
||||
if p, _ := IsProtected(mkRV(0), st); p {
|
||||
t.Error("rv=0 不应保护")
|
||||
}
|
||||
|
||||
// 3) ENV="0" 显式关:不保护
|
||||
t.Setenv(ProtectRailsEnv, "0")
|
||||
if p, _ := IsProtected(mkRV(5), st); p {
|
||||
t.Error("ENV=0 不应保护")
|
||||
}
|
||||
|
||||
// 4) rowInt 类型兼容:int64 也应正确识别(防类型断言失配致规则空转)
|
||||
t.Setenv(ProtectRailsEnv, "1")
|
||||
row := map[string]interface{}{
|
||||
"id": "mem_i64", "category": "normal", "content": "x",
|
||||
"recall_verified_count": int64(3),
|
||||
}
|
||||
if p, _ := IsProtected(row, st); !p {
|
||||
t.Error("rv=int64(3) 应保护(rowInt 类型兼容)")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -35,6 +35,38 @@ var (
|
||||
},
|
||||
)
|
||||
|
||||
// ─── 整合(consolidate)可观测性 ───────────
|
||||
// 2026-10-26:这三个指标与 MetaNoteRejectTotal 的定义在 rebase 时被丢弃(定义只在
|
||||
// a8dfa37 里),而调用方(consolidation_track.go / core.go)已进 main ⇒ HEAD 编不过。
|
||||
// 从 a8dfa37 原样恢复定义并注册,恢复 main 可构建性。
|
||||
ConsolidateBusyWaitTotal = prometheus.NewCounter(
|
||||
prometheus.CounterOpts{
|
||||
Name: "zhiyi_consolidate_busy_wait_total",
|
||||
Help: "Rust 侧报 consolidate_busy 时 Go 侧主动等待重试的次数",
|
||||
},
|
||||
)
|
||||
// 最近一次整合成功时间(Unix 秒)
|
||||
ConsolLastSuccessTime = prometheus.NewGauge(
|
||||
prometheus.GaugeOpts{
|
||||
Name: "zhiyi_consolidate_last_success_time",
|
||||
Help: "最近一次 Rust 整合完成时间(Unix 秒);0 = 从未成功",
|
||||
},
|
||||
)
|
||||
// 整合连续失败次数
|
||||
ConsolConsecutiveFailures = prometheus.NewGauge(
|
||||
prometheus.GaugeOpts{
|
||||
Name: "zhiyi_consolidate_consecutive_failures",
|
||||
Help: "连续整合失败次数(任意一次成功归零)",
|
||||
},
|
||||
)
|
||||
// metanote shadow 写前置校验拒绝计数
|
||||
MetaNoteRejectTotal = prometheus.NewCounter(
|
||||
prometheus.CounterOpts{
|
||||
Name: "zhiyi_metanote_reject_total",
|
||||
Help: "metanote shadow 422 拒绝累计次数",
|
||||
},
|
||||
)
|
||||
|
||||
// ─── LLM 调用计数 ──────────────────────────
|
||||
DistillLLMCallsToday = prometheus.NewGauge(
|
||||
prometheus.GaugeOpts{
|
||||
@@ -124,6 +156,10 @@ func init() {
|
||||
RequestDuration,
|
||||
DistillQueueDepth,
|
||||
ConflictsPending,
|
||||
ConsolidateBusyWaitTotal,
|
||||
ConsolLastSuccessTime,
|
||||
ConsolConsecutiveFailures,
|
||||
MetaNoteRejectTotal,
|
||||
DistillLLMCallsToday,
|
||||
ConsolidateFailuresTotal,
|
||||
HitEventAppendFailedTotal,
|
||||
|
||||
@@ -46,7 +46,7 @@ type MemoryRecord struct {
|
||||
Category string `json:"category"` // system_fact, user_pref, proj_context, tool_usage, code_snippet
|
||||
Vector []float32 `json:"vector"` // bge-m3 1024维,L2归一化
|
||||
Tier string `json:"tier"` // normal, core(core永不衰减)
|
||||
Importance float64 `json:"importance"` // 重要性得分 = recency_factor × (1+log(1+recall_count))
|
||||
Importance float64 `json:"importance"` // 档位重要性:写入侧决定(storage/importance_rule.go,t_789898bd:规则关恒 1.0;开则 core=0.9/wiki,llm_distill=0.7/legacy,unknown=0.4/recovery=0.3)。注意:排序用的「运行时 importance」另算(quality_fusion.computeImportance = recency × (1+log(1+recall_seen))),两者是不同量
|
||||
QualityScore float64 `json:"quality_score"`
|
||||
UsefulCount int `json:"useful_count"` // 被标记 useful 的次数
|
||||
// GA-P0-1 (t_718d8707):not_useful_count 曾把两种语义混写——
|
||||
@@ -56,6 +56,12 @@ type MemoryRecord struct {
|
||||
// 历史列(deprecated),新代码不再读它(见 AC#3)。
|
||||
HumanNotUsefulCount int `json:"human_not_useful_count"` // 用户人工点「没用」的次数
|
||||
DedupDownweightCount int `json:"dedup_downweight_count"` // dedup/合并流程自动降权的次数
|
||||
// A10 (t_d618dc7c):把「人工有用」从 UsefulCount(人工+机器混写)拆出单独计数。
|
||||
// 指针类型:nil = NULL(存量行/从未被人工点过),非 nil = 真实整数。
|
||||
// 统计口径用时分母必须 WHERE human_useful_count IS NOT NULL(NULL≠0)。
|
||||
// 只有人工 feedback(/api/v1/feedback/useful → IncrementUseful)会 ++,机器自评/
|
||||
// 蒸馏质量路径走别的方法(IncrementDedupDownweight/IncrementHumanNotUseful/...)不写此列。
|
||||
HumanUsefulCount *int `json:"human_useful_count"` // nil=NULL(从未被人工点过);非 nil=真实计数
|
||||
NotUsefulCount int `json:"not_useful_count"` // [DEPRECATED] 历史混写列,勿再读
|
||||
Version int `json:"version"` // 当前版本号
|
||||
VersionHistory []VersionRecord `json:"version_history,omitempty"` // 完整修改链
|
||||
@@ -191,7 +197,28 @@ type RecallResult struct {
|
||||
// 2026-10-13 时间维检索(t_e9a4d18f P0):该条目是否可能过期。
|
||||
// 仅当 stale 标记开启(ZHIYI_STALE_AFTER_HOURS>0)且 updated_at 超期时为 true;
|
||||
// 默认关闭 ⇒ 恒 false 且 omitempty 不输出 ⇒ 行为零变化(向后兼容)。
|
||||
// 2026-10-26(t_contract_p0):默认阈值改为 720h(30 天)——**只做展示标记,
|
||||
// 不参与排序**(排序侧的有效期判据走 valid_to,见 ValidTo);环境变量仍可覆盖/关闭。
|
||||
Stale bool `json:"stale,omitempty"`
|
||||
// 2026-10-26(t_trace_p0):本条结果的来源通道(vector|fts|graph|shared|wiki|
|
||||
// standing|pattern|peer)。此前「为什么被召回」不可答,调参只能盲修(CS249r Ch14)。
|
||||
Channel string `json:"channel,omitempty"`
|
||||
// 2026-10-26(P1 supersede 读侧暴露):版本与取代关系。
|
||||
// 写入侧一直有 supersedes / superseded_by / valid_from / valid_to / version,
|
||||
// 但 RecallResult 不带出 ⇒ 读侧永远看不见「这条已被谁取代 / 是否过期」。
|
||||
// 存量行零值(Version=0=未记录版本),omitempty 保持旧响应兼容。
|
||||
Version int `json:"generation,omitempty"` // 记忆版本号(DB version 列,语义=第几代)
|
||||
ValidFrom string `json:"valid_from,omitempty"` // 语义有效期起点(RFC3339,空=永久)
|
||||
ValidTo string `json:"valid_to,omitempty"` // 语义有效期终点(空=永久)
|
||||
Supersedes []string `json:"supersedes,omitempty"` // 我取代了谁
|
||||
SupersededBy []string `json:"superseded_by,omitempty"` // 我被谁取代(反向指针)
|
||||
// Superseded 便捷布尔:SupersededBy 非空 ⇒ 已有更新版本存在(读侧一个字段可判)。
|
||||
Superseded bool `json:"superseded,omitempty"`
|
||||
// A10 (t_d618dc7c):把「人工有用」单独计数直通给调用方。此前 RecallResult 缺它 ⇒
|
||||
// 插件/调用方读不到(t_4c810e02「加列不接 IPC=静默 no-op」同款坑)。
|
||||
// nil=该条从未被人工点过(库里 NULL);非 nil=真实人工 useful 次数(NULL≠0)。
|
||||
// 仅观测口径,不进排序(AC#6 排序零变化)。
|
||||
HumanUsefulCount *int `json:"human_useful_count"`
|
||||
}
|
||||
|
||||
// RerankResult rerank 重排后的单条结果。
|
||||
|
||||
@@ -2,15 +2,17 @@
|
||||
//
|
||||
// 背景:MemOS 血统的 L2「跨任务重复模式」检测。原实现活在 Python daemon 的
|
||||
// main_loop(30s tick),2026-09-24 为 state-db 写安全整体退役后 L2 pattern 蒸馏
|
||||
// 停摆(graph_nodes type='pattern' 停在 85 个 / 2026-09-18)。本包把 L1→L2 蒸馏
|
||||
// 停摆(graph_nodes type='pattern' 停为 0 行;旧 Python daemon 残留的 85 行是
|
||||
// type='pattern_stat',id 全 NULL、语义为 L1 命名空间统计快照,非 L2 pattern / 2026-09-18)。本包把 L1→L2 蒸馏
|
||||
// 复活成**短命单发**(modernc sqlite 直连 graph.db,开/关不占长连接,不碰 state.db),
|
||||
// 并按设计 §4.1 叠加两条检测规则:
|
||||
// - 工具组合频率:同一 tool 组合出现 ≥3 次 → tool_combo pattern
|
||||
// (confidence = min(freq*0.1, 1.0))
|
||||
// - 相似解决路径聚类:cosine ≥0.85 成簇且簇内 ≥3 条 → workflow pattern
|
||||
//
|
||||
// 存储:复用现有 graph_nodes(type='pattern'),**不另开新表**(避免与 85 存量 + 现有
|
||||
// 图谱查询割裂)。namespace='daemon-distill' 沿用旧 daemon 约定。
|
||||
// 存储:复用现有 graph_nodes(type='pattern'),**不另开新表**(避免与现存图谱查询割裂;
|
||||
// 表内另有 85 行 type='pattern_stat' 属旧 daemon 统计快照,与 L2 pattern 无关)。
|
||||
// namespace='daemon-distill' 沿用旧 daemon 约定。
|
||||
package pattern
|
||||
|
||||
import (
|
||||
|
||||
@@ -65,7 +65,7 @@ func (pp *PatternProvider) Match(query, namespace string, topK int) []models.Rec
|
||||
if pp == nil || pp.store == nil || topK <= 0 {
|
||||
return nil
|
||||
}
|
||||
pats, err := pp.store.ListPatterns(topK * 3) // 多取些再按相似度截断
|
||||
pats, _, err := pp.store.ListPatterns(topK * 3) // 多取些再按相似度截断
|
||||
if err != nil || len(pats) == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -0,0 +1,229 @@
|
||||
package pattern
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"testing"
|
||||
|
||||
_ "modernc.org/sqlite"
|
||||
|
||||
"github.com/xiaoxue/memoryweave/internal/models"
|
||||
"github.com/xiaoxue/memoryweave/internal/storage"
|
||||
)
|
||||
|
||||
// ── t_bd63b641 影-复:PATTERN/PEER 开门量具(不碰生产;只读 graph.db + 实时 embedder)──
|
||||
// 量化「开 ZHIYI_PATTERN_RECALL / ZHIYI_PEER_RECALL」对 14 金标会注入什么。
|
||||
// Provider 装配在 server.go 启动期由 env 决定 ⇒ 不重启无法在活服务上测开态;
|
||||
// 故本测试直接构造真实 Provider + 生产 graph.db(只读)+ 实时 bge embedder,
|
||||
// 跑出「开态会注入的条目」,与关态(=主召回原样)对比。
|
||||
//
|
||||
// 本测试只读、不落盘、不改库(AC:不碰生产二进制/不重启)。
|
||||
|
||||
type logRecord struct {
|
||||
ID string
|
||||
Category string
|
||||
Content string
|
||||
}
|
||||
|
||||
// runShadowWithDiag 返回:注入结果 + 诊断(pattern 装载数 / query 是否编码成功 / 各 query 最高余弦)。
|
||||
// ⚠️ 关键发现:生产 graph_nodes(type='pattern') 的 id 全为 NULL(85/85),ListPatterns 读 id 为 string
|
||||
// ⇒ 恒报 sql: Scan error (converting NULL to string),Match 直接 return nil。
|
||||
// 故 runShadow 不把 ListPatterns 错误当失败 —— 它本身就是「开态也无法注入」的根因证据。
|
||||
func runShadowWithDiag() (map[string][]logRecord, int, string, map[string]float64) {
|
||||
endpoint := os.Getenv("VLLM_ENDPOINT")
|
||||
if endpoint == "" {
|
||||
endpoint = "http://localhost:8000/v1/embeddings"
|
||||
}
|
||||
emb := storage.NewEmbedder(endpoint)
|
||||
store := NewGraphPatternStore("/var/lib/memoryweave/graph.db")
|
||||
pp := NewPatternProvider(store, NewEmbedderProvider(emb))
|
||||
|
||||
// 装载 pattern 数(防「store 空 → 注入恒 0」的假阴性);记录 ListPatterns 错误原文
|
||||
pats, _, lerr := store.ListPatterns(1000)
|
||||
loaded := 0
|
||||
listErr := ""
|
||||
if lerr != nil {
|
||||
listErr = lerr.Error() // 根因:id 全 NULL → Scan error
|
||||
} else {
|
||||
loaded = len(pats)
|
||||
}
|
||||
|
||||
golden := []string{
|
||||
"织忆是什么", "织忆的图谱扩展机制是什么",
|
||||
"织忆如何通过 E1 提取和 MMR 实现多样去重", "牧尘的偏好是什么",
|
||||
"牧尘喜欢什么样的工作方式", "如何根据牧尘的偏好调整记忆检索策略",
|
||||
"当前项目的技术栈是什么", "织忆项目有哪些核心组件",
|
||||
"织忆和其他记忆系统相比有什么架构优势", "如何使用 recall 接口",
|
||||
"织忆有哪些管理工具", "如何通过 API 扩展织忆功能",
|
||||
"本地 llama 服务 8080 端口", "session-to-zhiyi.py 脚本名",
|
||||
}
|
||||
out := make(map[string][]logRecord)
|
||||
maxCos := make(map[string]float64)
|
||||
|
||||
// 预编码 14 query(若编码失败会记 0,便于区分「没命中」vs「编码挂了」)
|
||||
for _, q := range golden {
|
||||
qv, eerr := emb.EncodeSingle(q)
|
||||
if eerr != nil || len(qv) == 0 {
|
||||
maxCos[q] = -1 // 编码失败标记
|
||||
continue
|
||||
}
|
||||
// 逐个 pattern 算余弦,取最高(复现 Provider.Match 内部逻辑)
|
||||
best := -1.0
|
||||
for _, p := range pats {
|
||||
rep := p.Data
|
||||
if rep == "" {
|
||||
rep = p.Name
|
||||
}
|
||||
text := rep
|
||||
for _, tl := range p.Tools {
|
||||
if tl != "" {
|
||||
text += " " + tl
|
||||
}
|
||||
}
|
||||
pv, perr := emb.EncodeSingle(text)
|
||||
if perr != nil || len(pv) == 0 {
|
||||
continue
|
||||
}
|
||||
s := cosineF32(qv, pv)
|
||||
if s > best {
|
||||
best = s
|
||||
}
|
||||
}
|
||||
maxCos[q] = best
|
||||
}
|
||||
|
||||
// 真注入(走 Provider.Match,同 server 路径)
|
||||
for _, g := range golden {
|
||||
pat := pp.Match(g, "hermes-main", 3) // server.go:902 topK=3
|
||||
for _, r := range pat {
|
||||
out[g] = append(out[g], logRecord{ID: r.ID, Category: r.Category, Content: r.Content})
|
||||
}
|
||||
}
|
||||
return out, loaded, listErr, maxCos
|
||||
}
|
||||
|
||||
// TestShadowPattern_Diff —— 开态 pattern 注入量化(只读生产库)
|
||||
func TestShadowPattern_Diff(t *testing.T) {
|
||||
out, loaded, listErr, maxCos := runShadowWithDiag()
|
||||
t.Logf("PATTERNS_LOADED=%d (防假阴性:store 有货才会注入)", loaded)
|
||||
if listErr != "" {
|
||||
t.Logf("PATTERN_LISTERROR=%q — 根因:pattern 行 id 全 NULL ⇒ Provider.Match 恒 return nil", listErr)
|
||||
}
|
||||
|
||||
queries := []string{
|
||||
"织忆是什么", "织忆的图谱扩展机制是什么",
|
||||
"织忆如何通过 E1 提取和 MMR 实现多样去重", "牧尘的偏好是什么",
|
||||
"牧尘喜欢什么样的工作方式", "如何根据牧尘的偏好调整记忆检索策略",
|
||||
"当前项目的技术栈是什么", "织忆项目有哪些核心组件",
|
||||
"织忆和其他记忆系统相比有什么架构优势", "如何使用 recall 接口",
|
||||
"织忆有哪些管理工具", "如何通过 API 扩展织忆功能",
|
||||
"本地 llama 服务 8080 端口", "session-to-zhiyi.py 脚本名",
|
||||
}
|
||||
totalInjected := 0
|
||||
for _, q := range queries {
|
||||
mc := maxCos[q]
|
||||
if mc < 0 {
|
||||
t.Logf("[EMB-FAIL] %q → 编码失败(不计命中)", q)
|
||||
}
|
||||
if n := len(out[q]); n > 0 {
|
||||
totalInjected += n
|
||||
t.Logf("[INJECT] %q → %d pattern (maxcos=%.3f)", q, n, mc)
|
||||
for _, r := range out[q] {
|
||||
t.Logf(" id=%s cat=%s content=%s", r.ID, r.Category, trunc(r.Content, 60))
|
||||
}
|
||||
} else {
|
||||
t.Logf("[NO-INJECT] %q → 0 pattern (maxcos=%.3f, 阈值=0.6)", q, mc)
|
||||
}
|
||||
}
|
||||
b, _ := json.MarshalIndent(out, "", " ")
|
||||
t.Logf("SHADOW_PATTERN_TOTAL_INJECTED=%d", totalInjected)
|
||||
os.WriteFile("/tmp/shadow-pattern-inject.json", b, 0644)
|
||||
t.Logf("raw json → /tmp/shadow-pattern-inject.json (%d bytes)", len(b))
|
||||
}
|
||||
|
||||
func trunc(s string, n int) string {
|
||||
if len(s) <= n {
|
||||
return s
|
||||
}
|
||||
return s[:n] + "…"
|
||||
}
|
||||
|
||||
// ── peer 侧:生产 graph.db 无 peers 表 ⇒ 开态无数据可注入 ──
|
||||
func TestShadowPeer_NoTable(t *testing.T) {
|
||||
db, err := peerOpenReadOnly("/var/lib/memoryweave/graph.db")
|
||||
if err != nil {
|
||||
t.Fatalf("open graph.db: %v", err)
|
||||
}
|
||||
defer db.Close()
|
||||
var cnt int
|
||||
err = db.QueryRow("SELECT count(*) FROM sqlite_master WHERE type='table' AND name='peers'").Scan(&cnt)
|
||||
if err != nil {
|
||||
t.Fatalf("query peers table existence: %v", err)
|
||||
}
|
||||
t.Logf("SHADOW_PEER_TABLE_EXISTS=%d (0 ⇒ 开态 peer 无数据可注入)", cnt)
|
||||
}
|
||||
|
||||
// peerOpenReadOnly 只读开 sqlite(WAL 兼容),避免 peer.Store 的写路径。
|
||||
func peerOpenReadOnly(path string) (*sql.DB, error) {
|
||||
return sql.Open("sqlite", "file:"+path+"?mode=ro&_pragma=busy_timeout(5000)")
|
||||
}
|
||||
|
||||
// ── 深度根因:即使修好 NULL-id,85 行 pattern_stat 无 data/tools 键(只有 description),
|
||||
// Match 只能 embed 命名键(如 "pattern:hermes-main:entity")⇒ 对金标余弦 < 0.6 阈值。
|
||||
// 本测试量化「修好 NULL-id 后开态到底能注入几类金标」—— 证明开态收益仍≈0。
|
||||
func TestShadowPattern_CosineIfIDFixed(t *testing.T) {
|
||||
endpoint := os.Getenv("VLLM_ENDPOINT")
|
||||
if endpoint == "" {
|
||||
endpoint = "http://localhost:8000/v1/embeddings"
|
||||
}
|
||||
emb := storage.NewEmbedder(endpoint)
|
||||
// 绕过 NULL-id 的 ListPatterns:只读 name + properties(非 NULL 列)
|
||||
db, err := peerOpenReadOnly("/var/lib/memoryweave/graph.db")
|
||||
if err != nil {
|
||||
t.Fatalf("open: %v", err)
|
||||
}
|
||||
defer db.Close()
|
||||
rows, _ := db.Query(`SELECT name, properties FROM graph_nodes WHERE type='pattern'`)
|
||||
type patText struct{ name, text string }
|
||||
var pats []patText
|
||||
for rows.Next() {
|
||||
var name, props string
|
||||
if serr := rows.Scan(&name, &props); serr != nil {
|
||||
continue
|
||||
}
|
||||
// 复现 Match 的 text 构造:rep=data(无→空) 回退 name;无 tools
|
||||
text := name
|
||||
pats = append(pats, patText{name, text})
|
||||
}
|
||||
rows.Close()
|
||||
t.Logf("PATTERN_ROWS_LOADABLE_BYPASS_ID=%d", len(pats))
|
||||
if len(pats) == 0 {
|
||||
t.Skip("无 pattern 行可载(NULL-id 未修前本就 0 注入,已在 TestShadowPattern_Diff 覆盖)")
|
||||
}
|
||||
golden := []string{"织忆是什么", "牧尘的偏好是什么", "如何使用 recall 接口",
|
||||
"本地 llama 服务 8080 端口", "织忆项目有哪些核心组件"}
|
||||
for _, q := range golden {
|
||||
qv, qerr := emb.EncodeSingle(q)
|
||||
if qerr != nil || len(qv) == 0 {
|
||||
t.Logf("[EMBFAIL] %q → %v", q, qerr)
|
||||
continue
|
||||
}
|
||||
best, bestName := -1.0, ""
|
||||
for _, p := range pats {
|
||||
pv, perr := emb.EncodeSingle(p.text)
|
||||
if perr != nil || len(pv) == 0 {
|
||||
continue
|
||||
}
|
||||
s := cosineF32(qv, pv)
|
||||
if s > best {
|
||||
best, bestName = s, p.name
|
||||
}
|
||||
}
|
||||
hit := best >= 0.6
|
||||
t.Logf("COSINE_PROBE %q maxcos=%.4f hit=%v (thr=0.6) best=%s", q, best, hit, bestName)
|
||||
}
|
||||
}
|
||||
|
||||
// 防 unused 误报
|
||||
var _ = models.RecallResult{}
|
||||
@@ -1,7 +1,8 @@
|
||||
// 织忆 MemoryWeave — L2 pattern 存储(t_d32e5f8a)
|
||||
//
|
||||
// 存储沿用现有 graph_nodes(type='pattern'),**不另开新表**(避免与 85 存量 +
|
||||
// 现有图谱查询割裂)。modernc sqlite 短命单发开/关,不持长连接、不碰 state.db。
|
||||
// 存储沿用现有 graph_nodes(type='pattern'),**不另开新表**(避免与现存图谱查询割裂;
|
||||
// 表内另有 85 行 type='pattern_stat' 属旧 daemon 统计快照,与 L2 pattern 无关)。
|
||||
// modernc sqlite 短命单发开/关,不持长连接、不碰 state.db。
|
||||
//
|
||||
// 写入幂等语义(复活 L1→L2 occurrence_count 驱动):
|
||||
// - 同名 pattern 节点存在 → occurrence_count += 1、last_seen 刷新、props 合并(新键覆盖、旧键保留)
|
||||
@@ -229,27 +230,35 @@ func (s *GraphPatternStore) GetPattern(id string) (Pattern, error) {
|
||||
|
||||
// ListPatterns 列出 graph_nodes(type='pattern'),按 occurrence_count 降序取 top N。
|
||||
// 0 / 空库 → 空列表不报错(AC1)。
|
||||
func (s *GraphPatternStore) ListPatterns(limit int) ([]Pattern, error) {
|
||||
// 单行 NULL/空 id 不致全盘失败:跳过并计数(见 AC1 差错容忍)。
|
||||
// 名空间 quarantine 排除(AC2:stat-template/单句主题不视为真 pattern)。
|
||||
func (s *GraphPatternStore) ListPatterns(limit int) ([]Pattern, int, error) {
|
||||
if limit <= 0 {
|
||||
limit = 10
|
||||
}
|
||||
db, oerr := s.open()
|
||||
if oerr != nil {
|
||||
return nil, oerr
|
||||
return nil, 0, oerr
|
||||
}
|
||||
rows, qerr := db.Query(`SELECT id, name, properties FROM graph_nodes WHERE type='pattern' LIMIT ?`, limit*3)
|
||||
// 使用 COALESCE 防止 NULL 导致 Scan 错误;同时排除 quarantine 名空间(AC2)
|
||||
rows, qerr := db.Query(`SELECT COALESCE(id,''), COALESCE(name,''), COALESCE(properties,'') FROM graph_nodes WHERE type='pattern' AND COALESCE(namespace,'') != 'quarantine' LIMIT ?`, limit*3)
|
||||
if qerr != nil {
|
||||
return nil, qerr
|
||||
return nil, 0, qerr
|
||||
}
|
||||
defer rows.Close()
|
||||
type raw struct {
|
||||
ID, Name, Props string
|
||||
}
|
||||
var all []raw
|
||||
var skipped int
|
||||
for rows.Next() {
|
||||
var r raw
|
||||
if serr := rows.Scan(&r.ID, &r.Name, &r.Props); serr != nil {
|
||||
return nil, serr
|
||||
return nil, 0, serr
|
||||
}
|
||||
if r.ID == "" {
|
||||
skipped++
|
||||
continue
|
||||
}
|
||||
all = append(all, r)
|
||||
}
|
||||
@@ -271,9 +280,6 @@ func (s *GraphPatternStore) ListPatterns(limit int) ([]Pattern, error) {
|
||||
if c, ok := props["confidence"].(float64); ok {
|
||||
p.Confidence = c
|
||||
}
|
||||
if o, ok := props["occurrence_count"].(float64); ok {
|
||||
_ = o
|
||||
}
|
||||
if ts, ok := props["source_tasks"].([]interface{}); ok {
|
||||
for _, v := range ts {
|
||||
if s2, ok := v.(string); ok {
|
||||
@@ -288,6 +294,12 @@ func (s *GraphPatternStore) ListPatterns(limit int) ([]Pattern, error) {
|
||||
}
|
||||
}
|
||||
}
|
||||
// occurrence_count 已在 props 中,但 ListPatterns 不再使用它做排序
|
||||
// 保留该字段以免反序列化时丢失(若后续代码仍需)
|
||||
if o, ok := props["occurrence_count"].(float64); ok {
|
||||
// 暂不使用,仅保留
|
||||
_ = o
|
||||
}
|
||||
}
|
||||
out = append(out, p)
|
||||
}
|
||||
@@ -298,5 +310,5 @@ func (s *GraphPatternStore) ListPatterns(limit int) ([]Pattern, error) {
|
||||
if len(out) > limit {
|
||||
out = out[:limit]
|
||||
}
|
||||
return out, nil
|
||||
return out, skipped, nil
|
||||
}
|
||||
|
||||
@@ -19,8 +19,8 @@ func newTestStore(t *testing.T) *GraphPatternStore {
|
||||
|
||||
// TestStore_EmptyList —— 空库 ListPatterns → 0 条不报错(AC1 判据:SELECT count ≥0 不报错)。
|
||||
func TestStore_EmptyList(t *testing.T) {
|
||||
s := newTestStore(t)
|
||||
got, err := s.ListPatterns(10)
|
||||
ts := newTestStore(t)
|
||||
got, _, err := ts.ListPatterns(10)
|
||||
if err != nil {
|
||||
t.Fatalf("空库 List 不应报错: %v", err)
|
||||
}
|
||||
@@ -60,7 +60,7 @@ func TestStore_UpsertCreate(t *testing.T) {
|
||||
if created != 1 || updated != 0 {
|
||||
t.Fatalf("新 pattern 应 created=1/updated=0,实际 %d/%d", created, updated)
|
||||
}
|
||||
listed, _ := s.ListPatterns(10)
|
||||
listed, _, _ := s.ListPatterns(10)
|
||||
if len(listed) != 1 {
|
||||
t.Fatalf("应 1 条,实际 %d", len(listed))
|
||||
}
|
||||
|
||||
@@ -19,10 +19,17 @@ import (
|
||||
)
|
||||
|
||||
// Local2BReasoner 用本地 2B 生成四层推理画像。
|
||||
// env:LLM_ENDPOINT(默认 http://127.0.0.1:8081/v1/chat/completions)
|
||||
//
|
||||
// LLM_MODEL (默认 local/minicpm5-2b)
|
||||
// env 查找顺序(2026-10-26 加前两项,起因 t_a5ef04a5):
|
||||
//
|
||||
// ZHIYI_PEER_LLM_ENDPOINT(peer 专用,优先)→ LLM_ENDPOINT → 默认 8081 本地 2B
|
||||
// ZHIYI_PEER_LLM_MODEL (peer 专用,优先)→ LLM_MODEL → 默认 local/minicpm5-2b
|
||||
// LLM_TIMEOUT (默认 120s)
|
||||
//
|
||||
// 为什么要 peer 专用变量:LLM_ENDPOINT 是**全进程共享**的(蒸馏 llm_judge、
|
||||
// adversarial_audit、skillgen、consolidate client 都读它,见 server.go:230)。
|
||||
// 把全局指向 8081 会把蒸馏从 agnes-2.5-flash 顺手降级成 2B;而 newapi(3000)
|
||||
// 需要鉴权、本 reasoner 按设计不发 Authorization ⇒ 必然 401。故只给 peer 开专用口子。
|
||||
type Local2BReasoner struct {
|
||||
endpoint string
|
||||
model string
|
||||
@@ -32,11 +39,17 @@ type Local2BReasoner struct {
|
||||
|
||||
// NewLocal2BReasoner 读 env 构造本地 2B 推理器。
|
||||
func NewLocal2BReasoner() *Local2BReasoner {
|
||||
endpoint := os.Getenv("LLM_ENDPOINT")
|
||||
endpoint := os.Getenv("ZHIYI_PEER_LLM_ENDPOINT")
|
||||
if endpoint == "" {
|
||||
endpoint = os.Getenv("LLM_ENDPOINT")
|
||||
}
|
||||
if endpoint == "" {
|
||||
endpoint = "http://127.0.0.1:8081/v1/chat/completions"
|
||||
}
|
||||
model := os.Getenv("LLM_MODEL")
|
||||
model := os.Getenv("ZHIYI_PEER_LLM_MODEL")
|
||||
if model == "" {
|
||||
model = os.Getenv("LLM_MODEL")
|
||||
}
|
||||
if model == "" {
|
||||
model = "local/minicpm5-2b"
|
||||
}
|
||||
|
||||
@@ -0,0 +1,53 @@
|
||||
package selfoptimize
|
||||
|
||||
import "testing"
|
||||
|
||||
// 2026-10-26(t_da03e910 终审):把「计数器语义」和「t8 阈值可配」钉住。
|
||||
//
|
||||
// 背景:审计曾把 deprecated_per_day 当成率指标报「0 样本却 valid=None ⇒ 缺样本门」,
|
||||
// 实为探针模型错配(计数器无分母)。真正的缺陷是:
|
||||
// ① 0 无法区分「今天没淘汰」与「从来没统计过」→ 加 DeprecatedTotal;
|
||||
// ② executor t8 硬编码 >5 且无视 Condition(配置写 0.1 从不生效)。
|
||||
|
||||
func TestRecordDeprecation_CountsBothTodayAndTotal(t *testing.T) {
|
||||
oldToday, oldTotal := Dash.DeprecatedToday, Dash.DeprecatedTotal
|
||||
defer func() { Dash.DeprecatedToday, Dash.DeprecatedTotal = oldToday, oldTotal }()
|
||||
Dash.DeprecatedToday, Dash.DeprecatedTotal = 0, 100
|
||||
|
||||
Dash.RecordDeprecation()
|
||||
Dash.RecordDeprecation()
|
||||
|
||||
if Dash.DeprecatedToday != 2 || Dash.DeprecatedTotal != 102 {
|
||||
t.Fatalf("today=%d total=%d,期望 2/102:累计值必须跨天保留,日值才归零",
|
||||
Dash.DeprecatedToday, Dash.DeprecatedTotal)
|
||||
}
|
||||
|
||||
m := Dash.Metrics()
|
||||
if v, ok := m["deprecated_total"]; !ok || v != 102 {
|
||||
t.Fatalf("metrics 缺 deprecated_total 或值不对: %v (ok=%v)", v, ok)
|
||||
}
|
||||
// 计数器不该带 _samples/_valid(有分母才有样本门)
|
||||
for _, k := range []string{"deprecated_samples", "deprecated_valid"} {
|
||||
if _, bad := m[k]; bad {
|
||||
t.Fatalf("计数器不应用率语义键 %s", k)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestTDecayCritical_ReadsConditionNotHardcode(t *testing.T) {
|
||||
te := &TriggerExecutor{}
|
||||
th := TriggerDef{ID: "t8", Type: TDecayCritical, Condition: 5.0, Enabled: true}
|
||||
|
||||
if te.evaluate(&th, map[string]float64{"deprecated_per_day": 4}) {
|
||||
t.Fatal("4 条/天不应触发(阈值 5)")
|
||||
}
|
||||
if !te.evaluate(&th, map[string]float64{"deprecated_per_day": 6}) {
|
||||
t.Fatal("6 条/天应触发(阈值 5)")
|
||||
}
|
||||
|
||||
// 配置必须真的生效:把阈值调到 10,同样 6 条就不该触发。
|
||||
th.Condition = 10.0
|
||||
if te.evaluate(&th, map[string]float64{"deprecated_per_day": 6}) {
|
||||
t.Fatal("Condition=10 时 6 条不该触发 —— 说明仍在用硬编码阈值")
|
||||
}
|
||||
}
|
||||
@@ -63,7 +63,10 @@ func makeTriggers() []*TriggerDef {
|
||||
{ID: "t5", Type: TGapDetected, Condition: true, Action: "gap_fill", Cooldown: 5 * time.Minute, Enabled: true},
|
||||
{ID: "t6", Type: TConflict, Condition: 3, Action: "conflict_muchen", Cooldown: 10 * time.Minute, Enabled: true},
|
||||
{ID: "t7", Type: TVValueLow, Condition: 0.2, Action: "mem_review", Cooldown: 12 * time.Hour, Enabled: true},
|
||||
{ID: "t8", Type: TDecayCritical, Condition: 0.1, Action: "archive_or_boost", Cooldown: 6 * time.Hour, Enabled: true},
|
||||
// t8 的 Condition 是「当天废弃条数」阈值(计数器语义,不是率)。
|
||||
// 2026-10-26:原先这里写 0.1 但判据函数硬编码 >5 —— 配置值从不被读取,
|
||||
// 改配置也改不动行为。现改为 Condition: 5(与硬编码等价,行为不变)+ 判据读 t.Condition。
|
||||
{ID: "t8", Type: TDecayCritical, Condition: 5.0, Action: "archive_or_boost", Cooldown: 6 * time.Hour, Enabled: true},
|
||||
}
|
||||
}
|
||||
|
||||
@@ -145,7 +148,10 @@ func (te *TriggerExecutor) evaluate(t *TriggerDef, metrics map[string]float64) b
|
||||
v, ok := metrics["recall_usefulness_rate"]
|
||||
return ok && !IsNA(v) && v < threshold
|
||||
case TDecayCritical:
|
||||
return metrics["deprecated_per_day"] > 5
|
||||
// counter 语义:deprecated_per_day 是「当天废弃条数」,阈值同单位(条/天)。
|
||||
threshold, _ := t.Condition.(float64)
|
||||
v, ok := metrics["deprecated_per_day"]
|
||||
return ok && v > threshold
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
@@ -45,15 +45,19 @@ func dashboardLocalPath() string {
|
||||
|
||||
// dashboardState 可序列化快照 —— Dashboard 内含 sync.RWMutex,不能直接 Marshal/整块复制
|
||||
type dashboardState struct {
|
||||
UsefulCount int `json:"useful_count"`
|
||||
NotUsefulCount int `json:"not_useful_count"`
|
||||
TotalRecalls int `json:"total_recalls"`
|
||||
HitCount int `json:"hit_count"`
|
||||
ClosedGaps int `json:"closed_gaps"`
|
||||
TotalGaps int `json:"total_gaps"`
|
||||
CascadeFixedTotal int `json:"cascade_fixed_total"`
|
||||
TotalFixes int `json:"total_fixes"`
|
||||
DeprecatedToday int `json:"deprecated_today"`
|
||||
UsefulCount int `json:"useful_count"`
|
||||
NotUsefulCount int `json:"not_useful_count"`
|
||||
TotalRecalls int `json:"total_recalls"`
|
||||
HitCount int `json:"hit_count"`
|
||||
ClosedGaps int `json:"closed_gaps"`
|
||||
TotalGaps int `json:"total_gaps"`
|
||||
CascadeFixedTotal int `json:"cascade_fixed_total"`
|
||||
TotalFixes int `json:"total_fixes"`
|
||||
DeprecatedToday int `json:"deprecated_today"`
|
||||
// DeprecatedTotal 累计淘汰条数(不跨天归零)。
|
||||
// 2026-10-26:加它的唯一理由是把「今天 0 条」和「从来没统计过」分开 ——
|
||||
// 与 pattern 召回那次的教训同型:0 永远有两种含义,指标必须能自证是哪一种。
|
||||
DeprecatedTotal int `json:"deprecated_total"`
|
||||
DeprecatedDate string `json:"deprecated_date"` // deprecated_today 的归属日期(跨天归零用)
|
||||
DistillLossSum float64 `json:"distill_loss_sum"`
|
||||
DistillLossCount int `json:"distill_loss_count"`
|
||||
@@ -101,6 +105,7 @@ func loadDashboardLocal() error {
|
||||
Dash.CascadeFixedTotal = st.CascadeFixedTotal
|
||||
Dash.TotalFixes = st.TotalFixes
|
||||
Dash.DeprecatedToday = st.DeprecatedToday
|
||||
Dash.DeprecatedTotal = st.DeprecatedTotal
|
||||
if st.DeprecatedDate != todayStr() { // "今日"计数跨天归零(进程跨零点不重置,下次加载时纠正)
|
||||
Dash.DeprecatedToday = 0
|
||||
}
|
||||
@@ -132,6 +137,7 @@ func (d *Dashboard) persistLocalLocked() {
|
||||
CascadeFixedTotal: d.CascadeFixedTotal,
|
||||
TotalFixes: d.TotalFixes,
|
||||
DeprecatedToday: d.DeprecatedToday,
|
||||
DeprecatedTotal: d.DeprecatedTotal,
|
||||
DeprecatedDate: todayStr(),
|
||||
DistillLossSum: d.DistillLossSum,
|
||||
DistillLossCount: d.DistillLossCount,
|
||||
@@ -174,7 +180,7 @@ func (d *Dashboard) persist() {
|
||||
_ = redisPersistence.HSet(dashboardRedisKey, "auto_resolved_conflicts", strconv.Itoa(d.AutoResolvedConflicts))
|
||||
_ = redisPersistence.HSet(dashboardRedisKey, "total_conflicts", strconv.Itoa(d.TotalConflicts))
|
||||
_ = redisPersistence.HSet(dashboardRedisKey, "dashboard_schema_version", dashboardSchemaV2) // t_e5997ca4:v2 标记常驻回写
|
||||
_ = redisPersistence.Expire(dashboardRedisKey, 90*24*time.Hour) // 90 天 TTL
|
||||
_ = redisPersistence.Expire(dashboardRedisKey, 90*24*time.Hour) // 90 天 TTL
|
||||
}
|
||||
|
||||
// loadDashboard 从 Redis 加载 Dashboard 数据
|
||||
@@ -244,16 +250,19 @@ func loadDashboard() error {
|
||||
// ─── 自优化仪表盘 ────────────────────────────────────────
|
||||
|
||||
type Dashboard struct {
|
||||
mu sync.RWMutex
|
||||
UsefulCount int `json:"useful_count"`
|
||||
NotUsefulCount int `json:"not_useful_count"`
|
||||
TotalRecalls int `json:"total_recalls"`
|
||||
HitCount int `json:"hit_count"`
|
||||
ClosedGaps int `json:"closed_gaps"`
|
||||
TotalGaps int `json:"total_gaps"`
|
||||
CascadeFixedTotal int `json:"cascade_fixed_total"`
|
||||
TotalFixes int `json:"total_fixes"`
|
||||
DeprecatedToday int `json:"deprecated_today"`
|
||||
mu sync.RWMutex
|
||||
UsefulCount int `json:"useful_count"`
|
||||
NotUsefulCount int `json:"not_useful_count"`
|
||||
TotalRecalls int `json:"total_recalls"`
|
||||
HitCount int `json:"hit_count"`
|
||||
ClosedGaps int `json:"closed_gaps"`
|
||||
TotalGaps int `json:"total_gaps"`
|
||||
CascadeFixedTotal int `json:"cascade_fixed_total"`
|
||||
TotalFixes int `json:"total_fixes"`
|
||||
DeprecatedToday int `json:"deprecated_today"`
|
||||
// DeprecatedTotal 累计淘汰条数(不跨天归零)—— 与「今天 0 条」区分开,
|
||||
// 因为 0 有两种含义:今天没淘汰 / 从来没统计。指标必须能自证是哪一种。
|
||||
DeprecatedTotal int `json:"deprecated_total"`
|
||||
DistillLossSum float64 `json:"distill_loss_sum"`
|
||||
DistillLossCount int `json:"distill_loss_count"`
|
||||
DistillSuccessCount int `json:"distill_success_count"`
|
||||
@@ -316,7 +325,11 @@ func b2f(b bool) float64 {
|
||||
// Metrics 返回核心指标。
|
||||
//
|
||||
// 比率项在分母退化时返回 metricNA(-1) 而非 1;每项比率都配 *_valid / *_samples。
|
||||
// 计数项(deprecated_per_day)与均值项(avg_distill_loss)同规则。
|
||||
//
|
||||
// 计数项(deprecated_per_day / deprecated_total)**不配** *_samples / *_valid:
|
||||
// 它们没有分母,「样本不足」对它们无定义(2026-10-26 更正:旧注释写作「与均值项
|
||||
// avg_distill_loss 同规则」是错的 —— 均值项有分母,故有 samples/valid;审计脚本
|
||||
// 曾据此对计数器报「0 样本」告警,属探针模型错配,不是指标缺陷)。
|
||||
func (d *Dashboard) Metrics() map[string]float64 {
|
||||
d.mu.RLock()
|
||||
defer d.mu.RUnlock()
|
||||
@@ -391,7 +404,9 @@ func (d *Dashboard) Metrics() map[string]float64 {
|
||||
"auto_resolve_samples": float64(d.TotalConflicts),
|
||||
"auto_resolve_valid": autoValid,
|
||||
|
||||
// counter 语义:无分母 ⇒ 不给 _samples/_valid(给了会诱导下游当率用)。
|
||||
"deprecated_per_day": float64(d.DeprecatedToday),
|
||||
"deprecated_total": float64(d.DeprecatedTotal),
|
||||
|
||||
"distill_success_count": float64(d.DistillSuccessCount),
|
||||
"avg_distill_loss": lossVal,
|
||||
@@ -922,6 +937,7 @@ func (d *Dashboard) RecordDeprecation() {
|
||||
d.mu.Lock()
|
||||
defer d.mu.Unlock()
|
||||
d.DeprecatedToday++
|
||||
d.DeprecatedTotal++
|
||||
d.persist()
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,323 @@
|
||||
package storage
|
||||
|
||||
// t_23b6a940:FeedbackScore 把「热度」(recall_verified_count) 当「质量」的修复。
|
||||
// 金标 119 条 A/B 实测:legacy(verified 1:1 进 pos)⇒ MRR -22.2%。
|
||||
// 本文件钉死:① 门关 delta=0 ② 门开方向正确(none/weight/tiebreak)③ 反向对照
|
||||
// (负向项恒定性)+ AC4(不读 deprecated not_useful_count)。
|
||||
// 测试基建(fbVec/newFbBiasPipeline/fbRun/fbBiasLance)复用 recall_feedback_bias_test.go。
|
||||
|
||||
import (
|
||||
"math"
|
||||
"testing"
|
||||
|
||||
"github.com/xiaoxue/memoryweave/internal/models"
|
||||
)
|
||||
|
||||
// satRef 手工复刻 2026-10-13 原始实现(pos=useful+verified, K=3),作 legacy 逐位基准。
|
||||
func satRef(pos, neg float64) float64 {
|
||||
sat := func(x float64) float64 {
|
||||
if x <= 0 {
|
||||
return 0
|
||||
}
|
||||
return x / (x + 3) // K 默认 3(测试环境无 ZHIYI_FEEDBACK_K)
|
||||
}
|
||||
return sat(pos) - sat(neg)
|
||||
}
|
||||
|
||||
func legacyScore(uc, vc, nuc, dd int) float64 {
|
||||
return satRef(float64(uc+vc), float64(nuc+dd))
|
||||
}
|
||||
|
||||
// TestPosModeDefaultIsLegacy — env 未设/设非法值 ⇒ legacy;且 legacy 与 10-13 原始
|
||||
// 实现逐位一致(含大计数 max=389 与中位 3 两档,金标实测的两个代表点)。
|
||||
func TestPosModeDefaultIsLegacy(t *testing.T) {
|
||||
t.Setenv("ZHIYI_POS_MODE", "")
|
||||
if got := posMode(); got != "legacy" {
|
||||
t.Fatalf("默认 posMode()=%q,want legacy", got)
|
||||
}
|
||||
t.Setenv("ZHIYI_POS_MODE", "BOGUS")
|
||||
if got := posMode(); got != "legacy" {
|
||||
t.Fatalf("非法值 posMode()=%q,want legacy(回落默认)", got)
|
||||
}
|
||||
t.Setenv("ZHIYI_POS_MODE", "")
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
rec models.MemoryRecord
|
||||
want float64
|
||||
}{
|
||||
{"hot max=389", models.MemoryRecord{RecallVerifiedCount: 389}, 389 / 392},
|
||||
{"median=3", models.MemoryRecord{RecallVerifiedCount: 3}, 3.0 / 6},
|
||||
{"useful+verified 混合", models.MemoryRecord{UsefulCount: 5, RecallVerifiedCount: 62}, 67 / 70},
|
||||
{"全零", models.MemoryRecord{}, 0},
|
||||
{"纯负向", models.MemoryRecord{HumanNotUsefulCount: 2}, -0.4},
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Setenv("ZHIYI_POS_MODE", "legacy")
|
||||
if got := FeedbackScore(&c.rec); math.Abs(got-legacyScore(c.rec.UsefulCount, c.rec.RecallVerifiedCount, c.rec.HumanNotUsefulCount, c.rec.DedupDownweightCount)) > 1e-12 {
|
||||
t.Fatalf("%s: legacy FeedbackScore=%.10f 与 10-13 原始实现不一致(必须逐位等价)", c.name, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestFeedbackScorePosModes — 四态数值单测(K=3、w=0.05 默认):
|
||||
// 热度从「近满分」压到「有界小增量」;负向项各态恒 −0.4(反向对照基线)。
|
||||
func TestFeedbackScorePosModes(t *testing.T) {
|
||||
t.Setenv("ZHIYI_POS_WEIGHT", "0.05")
|
||||
m := func(uc, vc int) models.MemoryRecord {
|
||||
return models.MemoryRecord{UsefulCount: uc, RecallVerifiedCount: vc, HumanNotUsefulCount: 2}
|
||||
}
|
||||
type want struct {
|
||||
legacy, none, weight, tiebreak float64
|
||||
}
|
||||
cases := []struct {
|
||||
name string
|
||||
rec models.MemoryRecord
|
||||
w want
|
||||
}{
|
||||
// 热度 max=389:legacy +0.9924(金标实测的硬顶值)⇒ none 0 / weight 0.8664 / tiebreak +0.0398(均再扣 neg=2 的 -0.4)
|
||||
{"hot389", m(0, 389), want{
|
||||
legacy: 389/392.0 - 0.4,
|
||||
none: -0.4,
|
||||
weight: (19.45) / (19.45 + 3.0) - 0.4,
|
||||
tiebreak: 0.05*(389.0/489.0) - 0.4,
|
||||
}},
|
||||
// 热度中位=3:legacy +0.5 ⇒ none 0 / weight sat(0.15)=0.04762 / tiebreak 0.001456
|
||||
{"median3", m(0, 3), want{
|
||||
legacy: 0.5 - 0.4,
|
||||
none: -0.4,
|
||||
weight: (0.15 / 3.15) - 0.4,
|
||||
tiebreak: 0.05*(3.0/103.0) - 0.4,
|
||||
}},
|
||||
// 纯质量(useful=2、热度=0):四态同值(质量信号不受修法影响)
|
||||
{"pure_quality", m(2, 0), want{
|
||||
legacy: 0.4 - 0.4,
|
||||
none: 0.4 - 0.4,
|
||||
weight: 0.4 - 0.4,
|
||||
tiebreak: 0.4 - 0.4,
|
||||
}},
|
||||
}
|
||||
for _, mode := range []string{"legacy", "none", "weight", "tiebreak"} {
|
||||
t.Setenv("ZHIYI_POS_MODE", mode)
|
||||
for _, c := range cases {
|
||||
var got float64
|
||||
switch mode {
|
||||
case "legacy":
|
||||
got = legacyScore(c.rec.UsefulCount, c.rec.RecallVerifiedCount, 2, 0)
|
||||
got = FeedbackScore(&c.rec) // 双校验:函数实值 == 手工 legacy 公式
|
||||
default:
|
||||
got = FeedbackScore(&c.rec)
|
||||
}
|
||||
var w float64
|
||||
switch mode {
|
||||
case "legacy":
|
||||
w = c.w.legacy
|
||||
case "none":
|
||||
w = c.w.none
|
||||
case "weight":
|
||||
w = c.w.weight
|
||||
case "tiebreak":
|
||||
w = c.w.tiebreak
|
||||
}
|
||||
if math.Abs(got-w) > 1e-9 {
|
||||
t.Fatalf("%s@%s: got=%.10f want=%.10f", c.name, mode, got, w)
|
||||
}
|
||||
}
|
||||
}
|
||||
// 跨态单调性(热度大值):legacy > weight > tiebreak > none
|
||||
hot := m(0, 389)
|
||||
for _, mode := range []string{"legacy", "weight", "tiebreak", "none"} {
|
||||
t.Setenv("ZHIYI_POS_MODE", mode)
|
||||
_ = FeedbackScore(&hot)
|
||||
}
|
||||
vals := map[string]float64{}
|
||||
for _, mode := range []string{"legacy", "weight", "tiebreak", "none"} {
|
||||
t.Setenv("ZHIYI_POS_MODE", mode)
|
||||
vals[mode] = FeedbackScore(&hot)
|
||||
}
|
||||
if !(vals["legacy"] > vals["weight"] && vals["weight"] > vals["tiebreak"] && vals["tiebreak"] > vals["none"]) {
|
||||
t.Fatalf("热度大值跨态单调性不成立: %v(want legacy>weight>tiebreak>none)", vals)
|
||||
}
|
||||
// AC4:deprecated 混写列 not_useful_count 任何模式都不读(GA-P0-1 已拆)——
|
||||
// 只改 NotUsefulCount、其余字段(含 neg=2)相同的两条记录,四态 score 必须恒等。
|
||||
// (四态全覆盖见 TestPosModeNegUnchanged;此处显式钉 legacy 默认态)
|
||||
t.Setenv("ZHIYI_POS_MODE", "")
|
||||
zeroRec := m(0, 0) // neg=2
|
||||
base := FeedbackScore(&zeroRec)
|
||||
depr := models.MemoryRecord{UsefulCount: 0, RecallVerifiedCount: 0, HumanNotUsefulCount: 2, NotUsefulCount: 99}
|
||||
if got := FeedbackScore(&depr); math.Abs(got-base) > 1e-12 {
|
||||
t.Fatalf("AC4 违反:只改 NotUsefulCount(99) 不应改变 FeedbackScore(base=%.4f got=%.4f),不得读 deprecated 列", base, got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestPosGateClosedZeroDelta — AC3 判据①(门关 delta=0):
|
||||
// ZHIYI_FEEDBACK_BIAS=0(生产现状)⇒ 偏置循环不执行 ⇒ 四种 POS_MODE 下召回
|
||||
// 位次/分数逐位相同(门控逐位等价,模式切换零行为变化)。
|
||||
func TestPosGateClosedZeroDelta(t *testing.T) {
|
||||
oldBias := feedbackBias
|
||||
feedbackBias = 0 // 生产现状 zhiyid.service: ZHIYI_FEEDBACK_BIAS=0
|
||||
defer func() { feedbackBias = oldBias }()
|
||||
|
||||
rows := []models.MemoryRecord{
|
||||
{ID: "m0", Content: "记忆甲 完全无关内容一", Category: "distilled", Source: "llm_distill", Vector: fbVec(0.95), RecallVerifiedCount: 389},
|
||||
{ID: "m1", Content: "记忆乙 完全无关内容二", Category: "distilled", Source: "llm_distill", Vector: fbVec(0.90), UsefulCount: 2},
|
||||
{ID: "m2", Content: "记忆丙 完全无关内容三", Category: "distilled", Source: "llm_distill", Vector: fbVec(0.85), RecallVerifiedCount: 3, HumanNotUsefulCount: 2},
|
||||
{ID: "m3", Content: "记忆丁 完全无关内容四", Category: "distilled", Source: "llm_distill", Vector: fbVec(0.80)},
|
||||
{ID: "m4", Content: "记忆戊 完全无关内容五", Category: "distilled", Source: "llm_distill", Vector: fbVec(0.75)},
|
||||
}
|
||||
var baseline map[string][2]float64
|
||||
for i, mode := range []string{"", "none", "weight", "tiebreak"} {
|
||||
t.Setenv("ZHIYI_POS_MODE", mode)
|
||||
got := fbRun(t, rows)
|
||||
if baseline == nil {
|
||||
baseline = got
|
||||
t.Logf("门关(bias=0) @%q 基准: m0(pos=%.0f,sc=%.4f) m1(pos=%.0f,sc=%.4f)",
|
||||
mode, got["m0"][0], got["m0"][1], got["m1"][0], got["m1"][1])
|
||||
continue
|
||||
}
|
||||
for id, b := range baseline {
|
||||
a, ok := got[id]
|
||||
if !ok {
|
||||
t.Fatalf("门关时 @mode=%q 结果集合变了:%s 消失", mode, id)
|
||||
}
|
||||
if a[0] != b[0] || a[1] != b[1] {
|
||||
t.Fatalf("AC3① 违反:bias=0 时 %s 模式位次/分数必须逐位相同(delta=0):before=(pos %.0f sc %.6f) @%s=(pos %.0f sc %.6f)",
|
||||
id, b[0], b[1], mode, a[0], a[1])
|
||||
}
|
||||
}
|
||||
_ = i
|
||||
}
|
||||
}
|
||||
|
||||
// TestPosModeNoneFlipsHeatWinner — AC3 判据②(门开方向正确):
|
||||
// BIAS=1 时,legacy 下「热度 389、质量 0」的 hot 硬顶 rank1(金标 -22% 的机制);
|
||||
// none 模式把它拆出去 ⇒ 质量信号(useful=2)的 good 反超 hot。方向翻转即修复生效。
|
||||
func TestPosModeNoneFlipsHeatWinner(t *testing.T) {
|
||||
oldBias := feedbackBias
|
||||
feedbackBias = 1.0
|
||||
defer func() { feedbackBias = oldBias }()
|
||||
|
||||
hot := func() []models.MemoryRecord {
|
||||
return []models.MemoryRecord{
|
||||
{ID: "hot", Content: "热门记忆 高频被召回", Category: "distilled", Source: "llm_distill", Vector: fbVec(0.90), RecallVerifiedCount: 389},
|
||||
{ID: "good", Content: "优质记忆 人工认可", Category: "distilled", Source: "llm_distill", Vector: fbVec(0.88), UsefulCount: 2},
|
||||
{ID: "p0", Content: "填充记忆 零", Category: "distilled", Source: "llm_distill", Vector: fbVec(0.60)},
|
||||
{ID: "p1", Content: "填充记忆 一", Category: "distilled", Source: "llm_distill", Vector: fbVec(0.55)},
|
||||
{ID: "p2", Content: "填充记忆 二", Category: "distilled", Source: "llm_distill", Vector: fbVec(0.50)},
|
||||
}
|
||||
}
|
||||
|
||||
// 对照臂:legacy(默认)门开 ⇒ hot rank1(复现金标 -22% 的机制)
|
||||
t.Setenv("ZHIYI_POS_MODE", "")
|
||||
legacy := fbRun(t, hot())
|
||||
if legacy["hot"][0] > legacy["good"][0] {
|
||||
t.Fatalf("对照臂异常:legacy 门开时 hot(热度389) 应压住 good(质量2),got hot pos=%.0f good pos=%.0f", legacy["hot"][0], legacy["good"][0])
|
||||
}
|
||||
t.Logf("对照臂 legacy: hot pos=%.0f (sc %.4f) 压住 good pos=%.0f (sc %.4f) —— 金标 -22%% 机制复现",
|
||||
legacy["hot"][0], legacy["hot"][1], legacy["good"][0], legacy["good"][1])
|
||||
|
||||
// 修复臂:none ⇒ good 反超 hot
|
||||
t.Setenv("ZHIYI_POS_MODE", "none")
|
||||
none := fbRun(t, hot())
|
||||
if none["good"][0] >= none["hot"][0] {
|
||||
t.Fatalf("AC3② 违反:none 模式 good(useful=2) 应反超 hot(热度389),got good pos=%.0f hot pos=%.0f", none["good"][0], none["hot"][0])
|
||||
}
|
||||
// 填充行不受影响(方向正确 = 只动热度项,不动质量/负向)
|
||||
for _, id := range []string{"p0", "p1", "p2"} {
|
||||
if none[id][1] != legacy[id][1] {
|
||||
t.Fatalf("填充行 %s 分数被误伤:legacy=%.6f none=%.6f", id, legacy[id][1], none[id][1])
|
||||
}
|
||||
}
|
||||
t.Logf("修复臂 none: good pos=%.0f (sc %.4f) 反超 hot pos=%.0f (sc %.4f);填充行分数逐位不变 ✓",
|
||||
none["good"][0], none["good"][1], none["hot"][0], none["hot"][1])
|
||||
}
|
||||
|
||||
// TestPosModeTiebreakWithinClass — AC1 tiebreak 候选:热度增量有界(< w·1 = 0.05),
|
||||
// 同档内保留热度顺序、不能跨档翻案;none 对照 = 热度零影响。
|
||||
// 注:fbVec 只设 v[0],全部文档与查询同向 ⇒ cosine 恒 1 ⇒ 位次靠 tie-break;
|
||||
// 这里用 fbVec2 给文档不同方向(cos(q=e0, (a,b)) = a/√(a²+b²))制造真实向量差。
|
||||
func TestPosModeTiebreakWithinClass(t *testing.T) {
|
||||
fbVec2 := func(v0, v1 float32) []float32 {
|
||||
v := make([]float32, 1024)
|
||||
v[0], v[1] = v0, v1
|
||||
return v
|
||||
}
|
||||
oldBias := feedbackBias
|
||||
feedbackBias = 1.0
|
||||
defer func() { feedbackBias = oldBias }()
|
||||
|
||||
// 同档:warm cos=0.9138 > hot cos=0.8924(差 0.021 = 同相关性档),热度差 389 vs 3
|
||||
// 跨档:low cos=0.4061;填充 p1/p2 cos≈0.32/0.20
|
||||
rows := []models.MemoryRecord{
|
||||
{ID: "hot", Content: "热门同档 高频被召回", Category: "distilled", Source: "llm_distill", Vector: fbVec2(0.89, 0.45), RecallVerifiedCount: 389},
|
||||
{ID: "warm", Content: "微热同档 偶尔被召回", Category: "distilled", Source: "llm_distill", Vector: fbVec2(0.90, 0.40), RecallVerifiedCount: 3},
|
||||
{ID: "low", Content: "低档填充", Category: "distilled", Source: "llm_distill", Vector: fbVec2(0.40, 0.90)},
|
||||
{ID: "p1", Content: "填充记忆 一", Category: "distilled", Source: "llm_distill", Vector: fbVec2(0.30, 0.90)},
|
||||
{ID: "p2", Content: "填充记忆 二", Category: "distilled", Source: "llm_distill", Vector: fbVec2(0.20, 0.98)},
|
||||
}
|
||||
|
||||
// none 对照:同档两条 bias 恒 0 ⇒ 顺序 = 纯向量序(warm 0.90 > hot 0.89)
|
||||
t.Setenv("ZHIYI_POS_MODE", "none")
|
||||
noneRes := fbRun(t, rows)
|
||||
if noneRes["warm"][0] > noneRes["hot"][0] {
|
||||
t.Fatalf("none 对照异常:同档无热度差时应按向量序 warm(0.90) 在 hot(0.89) 前,got warm pos=%.0f hot pos=%.0f", noneRes["warm"][0], noneRes["hot"][0])
|
||||
}
|
||||
|
||||
// tiebreak:hot 的热度小增量把它抬到 warm 之上(同档内热度序保留),
|
||||
// 但增量有界(0.05·sat(389,100)≈0.040 < 跨档向量差 0.145)⇒ low 不得翻案进同档上方
|
||||
t.Setenv("ZHIYI_POS_MODE", "tiebreak")
|
||||
tb := fbRun(t, rows)
|
||||
if tb["hot"][0] > tb["warm"][0] {
|
||||
t.Fatalf("AC1 tiebreak 违反方向:同档内热度更高者应占优(hot 389 > warm 3),got hot pos=%.0f warm pos=%.0f", tb["hot"][0], tb["warm"][0])
|
||||
}
|
||||
if tb["low"][0] < tb["warm"][0] || tb["low"][0] < tb["hot"][0] {
|
||||
t.Fatalf("AC1 tiebreak 违反有界性:热度小增量(<0.05)不得把低档 low(0.75) 翻案进同档(0.89/0.90) 上方:low pos=%.0f", tb["low"][0])
|
||||
}
|
||||
// 增量有界的数值核验(对 FeedbackScore 直接断言,不经 rerank)
|
||||
t.Setenv("ZHIYI_POS_MODE", "tiebreak")
|
||||
hotRow := &rows[0]
|
||||
warmRow := &rows[1]
|
||||
delta := FeedbackScore(hotRow) - FeedbackScore(warmRow) // 含 bias 系数 1.0
|
||||
if delta <= 0 {
|
||||
t.Fatalf("tiebreak 同档热度差应产生正增量:hot−warm=%.6f", delta)
|
||||
}
|
||||
if delta > 0.05 {
|
||||
t.Fatalf("AC1 tiebreak 违反有界性:增量 %.4f 超过 w=0.05 上限(大-K 饱和设计目标)", delta)
|
||||
}
|
||||
t.Logf("AC1 tiebreak: 同档增量 hot−warm=%.4f (∈(0, 0.05],有界);none 对照=纯向量序;low 未翻案 ✓", delta)
|
||||
}
|
||||
|
||||
// TestPosModeNegUnchanged — AC3 判据③(反向对照)+ AC4:
|
||||
// 负向项(human_not_useful)在四种模式下逐位不变(修复只动 pos,不碰 neg 方向);
|
||||
// 且任何模式都不读 deprecated NotUsefulCount。
|
||||
func TestPosModeNegUnchanged(t *testing.T) {
|
||||
oldBias := feedbackBias
|
||||
feedbackBias = 1.0
|
||||
defer func() { feedbackBias = oldBias }()
|
||||
|
||||
rows := []models.MemoryRecord{
|
||||
{ID: "bad", Content: "被否定记忆 内容", Category: "distilled", Source: "llm_distill", Vector: fbVec(0.90), HumanNotUsefulCount: 2, NotUsefulCount: 7},
|
||||
{ID: "clean", Content: "干净记忆 内容", Category: "distilled", Source: "llm_distill", Vector: fbVec(0.88)},
|
||||
{ID: "p0", Content: "填充记忆 零", Category: "distilled", Source: "llm_distill", Vector: fbVec(0.60)},
|
||||
{ID: "p1", Content: "填充记忆 一", Category: "distilled", Source: "llm_distill", Vector: fbVec(0.55)},
|
||||
{ID: "p2", Content: "填充记忆 二", Category: "distilled", Source: "llm_distill", Vector: fbVec(0.50)},
|
||||
}
|
||||
wantNeg := float64(0) - 2.0/5.0 // -sat(2,3) = -0.4,四态同值
|
||||
for _, mode := range []string{"", "none", "weight", "tiebreak"} {
|
||||
t.Setenv("ZHIYI_POS_MODE", mode)
|
||||
got := fbRun(t, rows)
|
||||
if cleanPos, badPos := got["clean"][0], got["bad"][0]; cleanPos > badPos {
|
||||
t.Fatalf("AC3③ 违反:neg 方向在模式 %q 下失效 —— clean(pos=%.0f) 未压住 bad(pos=%.0f)", mode, cleanPos, badPos)
|
||||
}
|
||||
_ = wantNeg
|
||||
}
|
||||
// 数值层:bad 行(verified=0,只有 neg)四态 FeedbackScore 必须逐位 == -0.4
|
||||
for _, mode := range []string{"", "none", "weight", "tiebreak"} {
|
||||
t.Setenv("ZHIYI_POS_MODE", mode)
|
||||
badRow := &rows[0]
|
||||
if got := FeedbackScore(badRow); math.Abs(got-wantNeg) > 1e-12 {
|
||||
t.Fatalf("AC3③/AC4 违反:模式 %q 下 neg 项不恒定:bad score=%.10f want=%.10f(含 NotUsefulCount=7 干扰项仍必须 == -0.4)", mode, got, wantNeg)
|
||||
}
|
||||
}
|
||||
t.Logf("AC3③ 反向对照: bad(human_not_useful=2, NotUsefulCount=7 干扰) 四态 score 恒 -0.4、位次恒被 clean 压住 ✓")
|
||||
}
|
||||
@@ -1,11 +1,50 @@
|
||||
package storage
|
||||
|
||||
import "github.com/xiaoxue/memoryweave/internal/models"
|
||||
import (
|
||||
"os"
|
||||
"strings"
|
||||
|
||||
"github.com/xiaoxue/memoryweave/internal/models"
|
||||
)
|
||||
|
||||
// feedbackSaturation 是饱和常数 K:sat(x)=x/(x+K)。
|
||||
// 计数无上界(可被反复标记),必须压进 [0,1) 才能当加性偏置用。
|
||||
var feedbackSaturation = envFloat("ZHIYI_FEEDBACK_K", 3)
|
||||
|
||||
// ── t_23b6a940:热度信号 recall_verified_count 如何进入正向质量项 ──
|
||||
//
|
||||
// 金标 119 条 A/B 实测(本卡卡体):legacy(verified 直入 pos)下 7413 条非零
|
||||
// verified(max=389、中位 3)拿到近满分偏置(最大 +0.9924),热门记忆硬顶 rank1、
|
||||
// 挤掉真正相关的 ⇒ MRR -22.2% / NDCG -17.7%(「热门≠正确」,与 t_34ec2b1f
|
||||
// 「新鲜≠正确」同一族病:单一易得信号冒充质量)。
|
||||
//
|
||||
// 修法 = 把「热度」从质量项里拆出去,env ZHIYI_POS_MODE 切换(默认 legacy =
|
||||
// 与当前生产逐位一致;回滚 = 置回 legacy 或 ZHIYI_FEEDBACK_BIAS=0,零代码变更):
|
||||
//
|
||||
// legacy = 现状(verified 1:1 进 pos,金标实测 -22%,仅作对照臂)
|
||||
// none = 移除 verified,pos = useful_count(热度不是质量;MoA 复核行动①)
|
||||
// weight = 独立小权重:pos = useful + w·verified,w=ZHIYI_POS_WEIGHT(默认 0.05)
|
||||
// tiebreak = 纯 tiebreak(MoA 复核行动②):质量项只算 useful;verified 走独立
|
||||
// 大 K(posTiebreakK=100) 饱和 × 小权重 w ⇒ 中位 v=3 ⇒ +0.0015、
|
||||
// v=389 ⇒ +0.040,只在同相关性档内做有界增量、不能跨档
|
||||
//
|
||||
// 负向项(human_not_useful + dedup_downweight)与模式无关、恒定性,
|
||||
// 各模式间逐位不变(反向对照判据)。
|
||||
func posMode() string {
|
||||
switch strings.ToLower(os.Getenv("ZHIYI_POS_MODE")) {
|
||||
case "none", "weight", "tiebreak", "legacy":
|
||||
return strings.ToLower(os.Getenv("ZHIYI_POS_MODE"))
|
||||
default:
|
||||
return "legacy" // 默认 = 与当前生产逐位一致
|
||||
}
|
||||
}
|
||||
|
||||
// posWeight 热度项权重(仅 weight/tiebreak 模式生效)。每次调用读 env,测试可 t.Setenv 翻转。
|
||||
func posWeight() float64 { return envFloat("ZHIYI_POS_WEIGHT", 0.05) }
|
||||
|
||||
// posTiebreakK tiebreak 档独立饱和常数:verified=389 ⇒ sat≈0.80,×w(默认0.05) ⇒ 增量上限 +0.040。
|
||||
const posTiebreakK = 100.0
|
||||
|
||||
// FeedbackScore 把 per-memory 反馈计数压成有界双向分 ∈ (-1, 1)。
|
||||
//
|
||||
// 设计(2026-10-13):排序此前只读 V 值(VValueProvider),而 V 值轴被 ep_*
|
||||
@@ -13,8 +52,13 @@ var feedbackSaturation = envFloat("ZHIYI_FEEDBACK_K", 3)
|
||||
// 本函数改读 DB 计数(干净、per-memory、双向),由 recall.go 以 env 门控
|
||||
// 加性偏置的形式接入排序(ZHIYI_FEEDBACK_BIAS,默认 0 = 关闭)。
|
||||
//
|
||||
// 2026-10-18 (t_23b6a940):正向项按 ZHIYI_POS_MODE 分四态(见上方注释);
|
||||
// legacy 态与 2026-10-13 原始实现**逐位一致**(verified 1:1 进 pos)——
|
||||
// 该态已被金标 A/B 证明把「热度」当「质量」(MRR -22%),保留只为对照臂
|
||||
// 与零变更回滚基线。
|
||||
//
|
||||
// 正向 = 被真实使用/认可:useful_count(人工点有用)+ recall_verified_count
|
||||
// (有正反馈或下游引用,models/memory.go:69 定义的唯一正信号)。
|
||||
// (legacy 态 1:1 计入;none/weight/tiebreak 态按 ZHIYI_POS_MODE 处理)。
|
||||
// 负向 = 被真实否定/取代:human_not_useful_count(人工点没用)+
|
||||
// dedup_downweight_count(去重降权)。
|
||||
// 注意 not_useful_count 是 deprecated 混写列(GA-P0-1 已拆),本函数不读它。
|
||||
@@ -22,13 +66,25 @@ func FeedbackScore(m *models.MemoryRecord) float64 {
|
||||
if m == nil {
|
||||
return 0
|
||||
}
|
||||
pos := float64(m.UsefulCount + m.RecallVerifiedCount)
|
||||
useful := float64(m.UsefulCount)
|
||||
verified := float64(m.RecallVerifiedCount)
|
||||
neg := float64(m.HumanNotUsefulCount + m.DedupDownweightCount)
|
||||
sat := func(x float64) float64 {
|
||||
sat := func(x, k float64) float64 {
|
||||
if x <= 0 {
|
||||
return 0
|
||||
}
|
||||
return x / (x + feedbackSaturation)
|
||||
return x / (x + k)
|
||||
}
|
||||
switch posMode() {
|
||||
case "none":
|
||||
return sat(useful, feedbackSaturation) - sat(neg, feedbackSaturation)
|
||||
case "weight":
|
||||
return sat(useful+posWeight()*verified, feedbackSaturation) - sat(neg, feedbackSaturation)
|
||||
case "tiebreak":
|
||||
return sat(useful, feedbackSaturation) - sat(neg, feedbackSaturation) +
|
||||
posWeight()*sat(verified, posTiebreakK)
|
||||
default: // legacy — 与 2026-10-13 实现逐位一致(pos = useful+verified)
|
||||
pos := useful + verified
|
||||
return sat(pos, feedbackSaturation) - sat(neg, feedbackSaturation)
|
||||
}
|
||||
return sat(pos) - sat(neg)
|
||||
}
|
||||
@@ -156,6 +156,10 @@ func (c *cachedGraphStore) AllEdges(namespace string) []governance.GraphEdge {
|
||||
return c.inner.AllEdges(namespace)
|
||||
}
|
||||
func (c *cachedGraphStore) Stats() (int, int, float64) { return c.inner.Stats() }
|
||||
|
||||
func (c *cachedGraphStore) CountNodesByType(typ string) (int, int, string) {
|
||||
return c.inner.CountNodesByType(typ)
|
||||
}
|
||||
func (c *cachedGraphStore) Prune(minWeight float64) { c.inner.Prune(minWeight) }
|
||||
func (c *cachedGraphStore) ExpandFromResults(results []models.RecallResult, namespace string, maxHops int) []models.RecallResult {
|
||||
return c.inner.ExpandFromResults(results, namespace, maxHops)
|
||||
|
||||
@@ -0,0 +1,137 @@
|
||||
package storage_test
|
||||
|
||||
// human_useful_count_e2e_test.go — t_d618dc7c A10 人工有用列端到端回归闸门
|
||||
//
|
||||
// 为什么必须存在(防重犯 t_4c810e02 静默 no-op):加了列但 Go 读侧解包缺字段 / 输出映射缺行
|
||||
// ⇒ 列真实写进 LanceDB,但 /recall、/memories 读出来恒 null(无报错、数值逐位不变)。
|
||||
// 本测试经**真实 sidecar(Rust LanceDB)写→读回**,断言 human_useful_count 非空且等于写入值:
|
||||
//
|
||||
// AC#4 读非空:InsertMemory 后 Search 读回,从未人工点过 = nil;IncrementUseful×2 = 2。
|
||||
// AC#5 反例 :机器 dedup/降权路径(IncrementDedupDownweight)跑后 human_useful_count 不变;
|
||||
// 人工 not-useful(IncrementHumanNotUseful)只动 human_not_useful_count,不动本列。
|
||||
//
|
||||
// 跑法(需先起 sidecar;未设环境变量自动跳过,可安全留在 CI):
|
||||
// zhiyi-consolidate --mode socket --socket /tmp/mw-huc.sock --data-dir /tmp/mw-hucdata
|
||||
// ZHIYI_HUC_E2E_SOCK=/tmp/mw-huc.sock go test -run TestHumanUsefulCountE2E -v ./internal/storage/
|
||||
import (
|
||||
"os"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/xiaoxue/memoryweave/internal/models"
|
||||
"github.com/xiaoxue/memoryweave/internal/storage"
|
||||
)
|
||||
|
||||
func TestHumanUsefulCountE2E(t *testing.T) {
|
||||
sock := os.Getenv("ZHIYI_HUC_E2E_SOCK")
|
||||
if sock == "" {
|
||||
t.Skip("ZHIYI_HUC_E2E_SOCK 未设置,跳过(需要 sidecar 在跑)")
|
||||
}
|
||||
const ns = "huc-go-e2e"
|
||||
|
||||
rc := storage.NewRustLanceDBClient(sock, nil)
|
||||
|
||||
vec := make([]float32, 1024)
|
||||
for i := range vec {
|
||||
vec[i] = 0.05
|
||||
}
|
||||
now := time.Now().UTC()
|
||||
// 读回:用 ScanAll(embedded model,useful_count 与 human_useful_count 都在),
|
||||
// Search 路径不投影 useful_count(历史缺口,非本卡范围)⇒ 断言两者同步得走 ScanAll。
|
||||
read := func(id string) *models.MemoryRecord {
|
||||
rows, err := rc.ScanAll()
|
||||
if err != nil {
|
||||
t.Fatalf("ScanAll 失败: %v", err)
|
||||
}
|
||||
for i := range rows {
|
||||
if rows[i].ID == id {
|
||||
return &rows[i]
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ── AC#4:读非空 —— 从未人工点过的存量行 human_useful_count 必须读回 nil(NULL,非 0)──
|
||||
t.Run("AC4_存量行读回NULL", func(t *testing.T) {
|
||||
m := models.MemoryRecord{
|
||||
ID: "huc-e2e-null", AgentID: "hermes-a06", Namespace: ns,
|
||||
Content: "huc-e2e 从未人工点过样本", Category: "test", Vector: vec,
|
||||
Tier: "normal", Importance: 1.0, QualityScore: 0.5,
|
||||
Version: 1, Freshness: "fresh", EmbeddingModel: "bge-m3",
|
||||
Source: "wiki:/tmp/huc-e2e.md", CreatedAt: now, UpdatedAt: now,
|
||||
}
|
||||
if err := rc.InsertMemory(m); err != nil {
|
||||
t.Fatalf("InsertMemory huc-e2e-null 失败: %v", err)
|
||||
}
|
||||
got := read("huc-e2e-null")
|
||||
if got == nil {
|
||||
t.Fatalf("✗ 读不回 huc-e2e-null(写入链路断)")
|
||||
}
|
||||
if got.HumanUsefulCount != nil {
|
||||
t.Fatalf("✗ AC4: 存量行 human_useful_count 应为 nil(NULL),got=%d(读侧解包/映射未接好或误填 0)", *got.HumanUsefulCount)
|
||||
}
|
||||
t.Log("✅ AC4: 存量行 human_useful_count 读回 = nil(NULL≠0 语义成立)")
|
||||
})
|
||||
|
||||
// ── AC#4 + AC#5:人工 useful ×2 ⇒ human_useful_count nil→1→2 且 useful_count 同步 +2 ──
|
||||
t.Run("AC5_人工useful两次同步+2", func(t *testing.T) {
|
||||
m := models.MemoryRecord{
|
||||
ID: "huc-e2e-inc", AgentID: "hermes-a06", Namespace: ns,
|
||||
Content: "huc-e2e 连续人工点有用样本", Category: "test", Vector: vec,
|
||||
Tier: "normal", Importance: 1.0, QualityScore: 0.5,
|
||||
Version: 1, Freshness: "fresh", EmbeddingModel: "bge-m3",
|
||||
Source: "wiki:/tmp/huc-e2e.md", CreatedAt: now, UpdatedAt: now,
|
||||
}
|
||||
if err := rc.InsertMemory(m); err != nil {
|
||||
t.Fatalf("InsertMemory huc-e2e-inc 失败: %v", err)
|
||||
}
|
||||
if err := rc.IncrementUseful("huc-e2e-inc"); err != nil {
|
||||
t.Fatalf("IncrementUseful #1 失败: %v", err)
|
||||
}
|
||||
g1 := read("huc-e2e-inc")
|
||||
if g1 == nil {
|
||||
t.Fatalf("读不回 huc-e2e-inc")
|
||||
}
|
||||
if g1.HumanUsefulCount == nil || *g1.HumanUsefulCount != 1 {
|
||||
t.Fatalf("✗ AC5: 第 1 次人工 useful 后 human_useful_count 应 nil→1,got=%v", g1.HumanUsefulCount)
|
||||
}
|
||||
if err := rc.IncrementUseful("huc-e2e-inc"); err != nil {
|
||||
t.Fatalf("IncrementUseful #2 失败: %v", err)
|
||||
}
|
||||
g2 := read("huc-e2e-inc")
|
||||
if g2 == nil {
|
||||
t.Fatalf("读不回 huc-e2e-inc (#2)")
|
||||
}
|
||||
if g2.HumanUsefulCount == nil || *g2.HumanUsefulCount != 2 {
|
||||
t.Fatalf("✗ AC5: 第 2 次人工 useful 后 human_useful_count 应=2,got=%v(读非空失败=静默 no-op)", g2.HumanUsefulCount)
|
||||
}
|
||||
if g2.UsefulCount != 2 {
|
||||
t.Fatalf("✗ AC5: useful_count 应与人工 useful 同步 =2,got=%d", g2.UsefulCount)
|
||||
}
|
||||
t.Logf("✅ AC5: 人工 useful ×2 ⇒ human_useful_count=%d useful_count=%d(同步、读非空)", *g2.HumanUsefulCount, g2.UsefulCount)
|
||||
|
||||
// ── 反例:机器 dedup 降权路径跑一次 ⇒ human_useful_count 不变(仍 2)──
|
||||
if err := rc.IncrementDedupDownweight("huc-e2e-inc"); err != nil {
|
||||
t.Fatalf("IncrementDedupDownweight 失败: %v", err)
|
||||
}
|
||||
g3 := read("huc-e2e-inc")
|
||||
if g3.HumanUsefulCount == nil || *g3.HumanUsefulCount != 2 {
|
||||
t.Fatalf("✗ AC5反例: 机器 dedup 路径不得污染 human_useful_count,want 2 got=%v", g3.HumanUsefulCount)
|
||||
}
|
||||
if g3.DedupDownweightCount != 1 {
|
||||
t.Fatalf("机器 dedup 路径 dedup_downweight_count 应 +1,got=%d", g3.DedupDownweightCount)
|
||||
}
|
||||
// ── 人工 not-useful ⇒ human_not_useful_count +1,本列不变 ──
|
||||
if err := rc.IncrementHumanNotUseful("huc-e2e-inc"); err != nil {
|
||||
t.Fatalf("IncrementHumanNotUseful 失败: %v", err)
|
||||
}
|
||||
g4 := read("huc-e2e-inc")
|
||||
if g4.HumanUsefulCount == nil || *g4.HumanUsefulCount != 2 {
|
||||
t.Fatalf("✗ AC5反例: 人工 not-useful 不得动 human_useful_count,want 2 got=%v", g4.HumanUsefulCount)
|
||||
}
|
||||
if g4.HumanNotUsefulCount != 1 {
|
||||
t.Fatalf("人工 not-useful 应 human_not_useful_count +1,got=%d", g4.HumanNotUsefulCount)
|
||||
}
|
||||
t.Logf("✅ AC5反例: 机器dedup + 人工not-useful 后 human_useful_count 仍=2(隔离成立)")
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,80 @@
|
||||
package storage
|
||||
|
||||
import "testing"
|
||||
|
||||
// TestHumanUsefulCountIsolation — A10 (t_d618dc7c) AC#3 反例测试
|
||||
//
|
||||
// 证明「只有人工 useful 才 +1 human_useful_count;机器自评/蒸馏质量路径不写此列」:
|
||||
//
|
||||
// 人工 useful ⇒ useful_count+1 且 human_useful_count+1 **同时发生**(同一次 IncrementUseful)
|
||||
// 机器自评/降权(IncrementDedupDownweight / IncrementHumanNotUseful)⇒ human_useful_count 不变
|
||||
//
|
||||
// 用相对增量断言(绝对值会被其它路径干扰)。起点设 human_useful_count = nil(模拟
|
||||
// 迁移后「从未被人工点过」的存量/NULL 行),验证首次人工 useful 从 NULL 起 +1 ⇒ 1。
|
||||
func TestHumanUsefulCountIsolation(t *testing.T) {
|
||||
mlc := NewMemLanceClient(nil)
|
||||
mlc.mu.Lock()
|
||||
// 起点:useful_count=0,human_useful_count=nil(NULL 存量语义),其它计数 0
|
||||
mlc.memories["mem_huc_1"] = &memEntry{ID: "mem_huc_1", HumanUsefulCount: nil}
|
||||
mlc.mu.Unlock()
|
||||
|
||||
e := mlc.memories["mem_huc_1"]
|
||||
|
||||
// ── 人工 useful ×2 ⇒ useful_count+2 且 human_useful_count nil→1→2 同时发生 ──
|
||||
if err := mlc.IncrementUseful("mem_huc_1"); err != nil {
|
||||
t.Fatalf("IncrementUseful #1 err: %v", err)
|
||||
}
|
||||
if e.UsefulCount != 1 {
|
||||
t.Fatalf("人工 useful#1:useful_count 相对增量 +1,want 1 got %d", e.UsefulCount)
|
||||
}
|
||||
if e.HumanUsefulCount == nil || *e.HumanUsefulCount != 1 {
|
||||
got := -1
|
||||
if e.HumanUsefulCount != nil {
|
||||
got = *e.HumanUsefulCount
|
||||
}
|
||||
t.Fatalf("人工 useful#1:human_useful_count NULL→1,want 1 got %d (nil=%v)", got, e.HumanUsefulCount == nil)
|
||||
}
|
||||
|
||||
if err := mlc.IncrementUseful("mem_huc_1"); err != nil {
|
||||
t.Fatalf("IncrementUseful #2 err: %v", err)
|
||||
}
|
||||
if e.UsefulCount != 2 || e.HumanUsefulCount == nil || *e.HumanUsefulCount != 2 {
|
||||
huc := -1
|
||||
if e.HumanUsefulCount != nil {
|
||||
huc = *e.HumanUsefulCount
|
||||
}
|
||||
t.Fatalf("人工 useful#2:want useful=2 human=%d got useful=%d human=%d", 2, e.UsefulCount, huc)
|
||||
}
|
||||
|
||||
// ── 反例:机器自评/dedup 降权路径 ⇒ human_useful_count 不变(仍 2),useful_count 不变 ──
|
||||
if err := mlc.IncrementDedupDownweight("mem_huc_1"); err != nil {
|
||||
t.Fatalf("IncrementDedupDownweight err: %v", err)
|
||||
}
|
||||
if e.HumanUsefulCount == nil || *e.HumanUsefulCount != 2 {
|
||||
got := -1
|
||||
if e.HumanUsefulCount != nil {
|
||||
got = *e.HumanUsefulCount
|
||||
}
|
||||
t.Fatalf("机器 dedup 路径不得污染 human_useful_count,want 2 got %d", got)
|
||||
}
|
||||
if e.UsefulCount != 2 {
|
||||
t.Fatalf("机器 dedup 路径不得污染 useful_count,want 2 got %d", e.UsefulCount)
|
||||
}
|
||||
if e.DedupDownweightCount != 1 {
|
||||
t.Fatalf("dedup_downweight_count 相对增量 +1,want 1 got %d", e.DedupDownweightCount)
|
||||
}
|
||||
|
||||
if err := mlc.IncrementHumanNotUseful("mem_huc_1"); err != nil {
|
||||
t.Fatalf("IncrementHumanNotUseful err: %v", err)
|
||||
}
|
||||
if e.HumanUsefulCount == nil || *e.HumanUsefulCount != 2 {
|
||||
got := -1
|
||||
if e.HumanUsefulCount != nil {
|
||||
got = *e.HumanUsefulCount
|
||||
}
|
||||
t.Fatalf("人工 not-useful 路径不得污染 human_useful_count,want 2 got %d", got)
|
||||
}
|
||||
if e.HumanNotUsefulCount != 1 {
|
||||
t.Fatalf("human_not_useful_count 相对增量 +1,want 1 got %d", e.HumanNotUsefulCount)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,118 @@
|
||||
package storage
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────────────
|
||||
// importance 规则(t_789898bd,2026-10-18)——「装温度计」
|
||||
//
|
||||
// 背景:写入侧 importance 恒硬编码 1.0(lancedb_ipc.go:159 episode / :200 memory),
|
||||
// 全库 46,903 行恒 1.0 ⇒ 排序侧没有非恒定信号,任何排序修复都测不了 NDCG 提升
|
||||
// (「修 importance = 先装温度计,才知道退烧药有没有用」)。本文件给写入侧接上
|
||||
// 规则化 importance + env 回滚开关。
|
||||
//
|
||||
// 标定依据(2026-10-18 21:00 全库只读统计,/var/lib/memoryweave memories 46,903 行,
|
||||
// 原始输出见交付 importance-fix-t_789898bd.md 的 stats_readonly 一节):
|
||||
// tier: core=1,926 (4.11%) / normal=44,977 (95.89%)
|
||||
// source: wiki=26,664 (56.85%) / llm_distill=6,197 (13.21%) /
|
||||
// legacy=12,099 (25.80%) / recovery=1,926 (4.11%,全部 tier=core) /
|
||||
// other=17 (0.04%)
|
||||
// ⇒ 回填后分布恰好 3 档,每档 ≥1%(AC2 验收判据),且无任何值等于 1.0
|
||||
// (「恒 1.0 行数 → 0」机械可判):
|
||||
// 0.9 = core(4.11%)
|
||||
// 0.7 = normal × wiki:*/llm_distill(70.06%)
|
||||
// 0.4 = normal × legacy:*/unknown(25.84%,unknown=17 行并入低置信档,
|
||||
// 0.04% 单拆会产生 <1% 档 ⇒ 违反 AC2,故并档)
|
||||
// 0.3 = 预留档:episode 原始日志 / 非 core 的 recovery 行(存量 0 行,
|
||||
// 只对未来写入生效,不参与存量直方图「每档 ≥1%」验收)
|
||||
// 未纳入的微调项(每项实测依据):
|
||||
// · 内容深度:长度 p50=32 字、78.1% <50 字 ⇒ 按深度切档必出 <1% 档
|
||||
// · 时间衰减:排序已有运行时 recency 项(quality_fusion.computeImportance),
|
||||
// 存进列会双衰减,故存量值取「静态信号」
|
||||
// · 图谱连接度:需要 graph.db,本卡红线「不碰图谱」
|
||||
//
|
||||
// env 开关(AC5 回滚开关,默认 off = 零行为变化):
|
||||
// ZHIYI_IMPORTANCE_RULE=on ⇒ 未显式传 importance 时走 ComputeImportance / EpisodeImportance
|
||||
// ZHIYI_IMPORTANCE_RULE=off / 未设 / 其他值 ⇒ 写入回落恒 1.0(与改动前逐位一致)
|
||||
// 每次调用读 env(测试可 t.Setenv 翻转,同 t_7c0538eb lowScoreDwGate 先例)。
|
||||
// 本卡**不部署生产二进制**(部署另行拍板);存量回填用 rust 侧
|
||||
// importance_backfill 工具(同 lancedb 0.15 crate,仅写 importance 一列,组更新 3 个版本)。
|
||||
// ─────────────────────────────────────────────────────────────────────────
|
||||
|
||||
import (
|
||||
"os"
|
||||
"strings"
|
||||
|
||||
"github.com/xiaoxue/memoryweave/internal/models"
|
||||
)
|
||||
|
||||
// legacyImportanceConstant 改动前恒 1.0 的字面量(规则关 / 调用方未传显式值时的回落)。
|
||||
const legacyImportanceConstant = 1.0
|
||||
|
||||
// EpisodeImportance episode(原始日志)写入档:无蒸馏信号,预留 0.3 档。
|
||||
const EpisodeImportance = 0.3
|
||||
|
||||
// importanceRuleEnabled env 开关:on/true/1 = 开;其余(含未设)= 关。
|
||||
func importanceRuleEnabled() bool {
|
||||
switch strings.ToLower(os.Getenv("ZHIYI_IMPORTANCE_RULE")) {
|
||||
case "on", "true", "1":
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// SourceClass 把 source 归入分档类。未知/空 source ⇒ "other"(按低置信类处理,
|
||||
// 实测 0.04%,不独立成档 —— 拆档会产生 <1% 档,违反 AC2 验收)。
|
||||
func SourceClass(source string) string {
|
||||
if source == "llm_distill" {
|
||||
return "llm_distill"
|
||||
}
|
||||
switch {
|
||||
case strings.HasPrefix(source, "wiki:"):
|
||||
return "wiki"
|
||||
case strings.HasPrefix(source, "legacy:"):
|
||||
return "legacy"
|
||||
case strings.HasPrefix(source, "recovery:"):
|
||||
return "recovery"
|
||||
default:
|
||||
return "other"
|
||||
}
|
||||
}
|
||||
|
||||
// ComputeImportance 由 (tier, source 类) 算出档位 importance。
|
||||
// 返回值 ∈ {0.9, 0.7, 0.4, 0.3},**永不返回 1.0**(「恒 1.0 行数→0」机械可判的前提)。
|
||||
func ComputeImportance(rec models.MemoryRecord) float64 {
|
||||
if rec.Tier == "core" {
|
||||
return 0.9 // 覆盖 1,926 行 = 4.11%
|
||||
}
|
||||
switch SourceClass(rec.Source) {
|
||||
case "wiki", "llm_distill":
|
||||
return 0.7 // 覆盖 32,861 行 = 70.06%
|
||||
case "recovery":
|
||||
return EpisodeImportance // 非 core 的 recovery(存量 0 行,预留档)
|
||||
default: // legacy / other / 空
|
||||
return 0.4 // 覆盖 12,116 行 = 25.84%
|
||||
}
|
||||
}
|
||||
|
||||
// EffectiveMemoryImportance 写入侧(InsertMemory :200)应写库的实际值(AC1 接线点):
|
||||
// 1. m.Importance != 0 ⇒ 显式值优先(调用方声明即权威,AC1.1)
|
||||
// 2. ZHIYI_IMPORTANCE_RULE=on ⇒ ComputeImportance(AC2 规则)
|
||||
// 3. 否则 ⇒ 回落 1.0(保护存量调用方,零行为突变,AC1.2)
|
||||
func EffectiveMemoryImportance(m models.MemoryRecord) float64 {
|
||||
if m.Importance != 0 {
|
||||
return m.Importance
|
||||
}
|
||||
if importanceRuleEnabled() {
|
||||
return ComputeImportance(m)
|
||||
}
|
||||
return legacyImportanceConstant
|
||||
}
|
||||
|
||||
// EffectiveEpisodeImportance episode 写入侧(InsertEpisode :159)应写库的实际值。
|
||||
// InsertEpisode 接口签名(6 个实现 + 4 个调用点)不带 importance 参数,改签名超出
|
||||
// 本卡范围;episode 原始日志也没有 tier/source 分档信号 ⇒ 规则档恒 0.3。
|
||||
// 规则 off(默认)⇒ 回落 1.0,与改动前逐位一致。
|
||||
func EffectiveEpisodeImportance() float64 {
|
||||
if importanceRuleEnabled() {
|
||||
return EpisodeImportance
|
||||
}
|
||||
return legacyImportanceConstant
|
||||
}
|
||||
@@ -0,0 +1,96 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/xiaoxue/memoryweave/internal/models"
|
||||
)
|
||||
|
||||
// t_789898bd AC2:importance 规则单测(档位来自全库实测标定,见 importance_rule.go 头注释)。
|
||||
|
||||
func TestComputeImportanceTiers(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
rec models.MemoryRecord
|
||||
want float64
|
||||
}{
|
||||
{"core 恒 0.9(source 无关)", models.MemoryRecord{Tier: "core", Source: "recovery:P0-shadow-log"}, 0.9},
|
||||
{"normal × wiki:*", models.MemoryRecord{Tier: "normal", Source: "wiki:worklog-note"}, 0.7},
|
||||
{"normal × llm_distill", models.MemoryRecord{Tier: "normal", Source: "llm_distill"}, 0.7},
|
||||
{"normal × legacy:*", models.MemoryRecord{Tier: "normal", Source: "legacy:pre0926"}, 0.4},
|
||||
{"normal × 未知 source", models.MemoryRecord{Tier: "normal", Source: "whatever"}, 0.4},
|
||||
{"normal × 空 source", models.MemoryRecord{Tier: "normal"}, 0.4},
|
||||
{"normal × recovery:*(预留 0.3 档)", models.MemoryRecord{Tier: "normal", Source: "recovery:X"}, 0.3},
|
||||
}
|
||||
for _, c := range cases {
|
||||
if got := ComputeImportance(c.rec); got != c.want {
|
||||
t.Errorf("%s: ComputeImportance=%v want %v", c.name, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestComputeImportanceNeverOne(t *testing.T) {
|
||||
// AC3 验收「恒 1.0 行数 → 0」依赖:规则输出空间不含 1.0。
|
||||
for _, tier := range []string{"core", "normal", "episode", ""} {
|
||||
for _, src := range []string{"wiki:a", "llm_distill", "legacy:b", "recovery:c", "other", ""} {
|
||||
if got := ComputeImportance(models.MemoryRecord{Tier: tier, Source: src}); got == legacyImportanceConstant {
|
||||
t.Fatalf("tier=%q source=%q 规则输出 == 1.0,违反「永不返回 1.0」", tier, src)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestEffectiveMemoryImportance(t *testing.T) {
|
||||
// 默认(规则关):显式值优先;未传 ⇒ 恒 1.0(改动前行为)
|
||||
t.Setenv("ZHIYI_IMPORTANCE_RULE", "off")
|
||||
if got := EffectiveMemoryImportance(models.MemoryRecord{Importance: 0.8}); got != 0.8 {
|
||||
t.Errorf("显式值 0.8 应直通,got %v", got)
|
||||
}
|
||||
if got := EffectiveMemoryImportance(models.MemoryRecord{Tier: "core"}); got != 1.0 {
|
||||
t.Errorf("规则关 + 未传值应回落 1.0,got %v", got)
|
||||
}
|
||||
// 规则开:未传值 ⇒ 走规则档
|
||||
t.Setenv("ZHIYI_IMPORTANCE_RULE", "on")
|
||||
if got := EffectiveMemoryImportance(models.MemoryRecord{Tier: "core"}); got != 0.9 {
|
||||
t.Errorf("规则开 + core 应 0.9,got %v", got)
|
||||
}
|
||||
if got := EffectiveMemoryImportance(models.MemoryRecord{Tier: "normal", Source: "legacy:pre0926"}); got != 0.4 {
|
||||
t.Errorf("规则开 + normal/legacy 应 0.4,got %v", got)
|
||||
}
|
||||
// 显式值在规则开时仍优先
|
||||
if got := EffectiveMemoryImportance(models.MemoryRecord{Tier: "core", Importance: 0.5}); got != 0.5 {
|
||||
t.Errorf("显式值 0.5 应优先于规则,got %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEffectiveEpisodeImportance(t *testing.T) {
|
||||
t.Setenv("ZHIYI_IMPORTANCE_RULE", "off")
|
||||
if got := EffectiveEpisodeImportance(); got != 1.0 {
|
||||
t.Errorf("规则关 episode 应回落 1.0(改动前行为),got %v", got)
|
||||
}
|
||||
t.Setenv("ZHIYI_IMPORTANCE_RULE", "on")
|
||||
if got := EffectiveEpisodeImportance(); got != EpisodeImportance {
|
||||
t.Errorf("规则开 episode 应 0.3,got %v", got)
|
||||
}
|
||||
// 未设 env ⇒ 关(测试环境无该变量时走 unset 分支)
|
||||
t.Setenv("ZHIYI_IMPORTANCE_RULE", "")
|
||||
if got := EffectiveEpisodeImportance(); got != 1.0 {
|
||||
t.Errorf("env 未设 episode 应回落 1.0,got %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSourceClass(t *testing.T) {
|
||||
cases := map[string]string{
|
||||
"llm_distill": "llm_distill",
|
||||
"wiki:worklog-note": "wiki",
|
||||
"legacy:pre0926": "legacy",
|
||||
"recovery:P0-shadow-log": "recovery",
|
||||
"episode": "other",
|
||||
"": "other",
|
||||
}
|
||||
for in, want := range cases {
|
||||
if got := SourceClass(in); got != want {
|
||||
t.Errorf("SourceClass(%q)=%q want %q", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"math/rand"
|
||||
"net"
|
||||
"os"
|
||||
"path/filepath"
|
||||
@@ -156,7 +157,7 @@ func (rc *RustLanceDBClient) InsertEpisode(agentID, namespace, content, category
|
||||
record := map[string]interface{}{
|
||||
"id": id, "agent_id": agentID, "namespace": namespace,
|
||||
"content": content, "category": category,
|
||||
"tier": "episode", "importance": 1.0, "quality_score": 0.0,
|
||||
"tier": "episode", "importance": EffectiveEpisodeImportance(), "quality_score": 0.0,
|
||||
"recall_count": 0, "useful_count": 0, "not_useful_count": 0,
|
||||
// GA-P0-2 (t_c6d41ac6):拆分字段(episode 无召回/反馈,恒 0)
|
||||
"recall_seen_count": 0, "recall_verified_count": 0,
|
||||
@@ -197,7 +198,7 @@ func (rc *RustLanceDBClient) InsertMemory(m models.MemoryRecord) error {
|
||||
record := map[string]interface{}{
|
||||
"id": m.ID, "agent_id": m.AgentID, "namespace": m.Namespace,
|
||||
"content": m.Content, "category": m.Category, "vector": m.Vector,
|
||||
"tier": m.Tier, "importance": 1.0, "quality_score": m.QualityScore,
|
||||
"tier": m.Tier, "importance": EffectiveMemoryImportance(m), "quality_score": m.QualityScore,
|
||||
"recall_count": m.RecallCount, "useful_count": 0, "not_useful_count": 0,
|
||||
// GA-P0-2 (t_c6d41ac6):拆分字段(新记忆按 model 携带,通常 0)
|
||||
"recall_seen_count": m.RecallSeenCount, "recall_verified_count": m.RecallVerifiedCount,
|
||||
@@ -228,12 +229,16 @@ func (rc *RustLanceDBClient) InsertMemory(m models.MemoryRecord) error {
|
||||
return err
|
||||
}
|
||||
|
||||
func (rc *RustLanceDBClient) Search(table string, vector []float32, topK int, namespaceFilter string) ([]models.MemoryRecord, error) {
|
||||
// SearchWithTruncation 同 Search,但额外返回「本次结果是否被 Rust 侧单次重读行数上限
|
||||
// 截断」(ipcResp.Truncated,t_576714de 引入)。t_3aa2a8c1:该标志此前只打日志不上抛
|
||||
// (recall 调用方观测不到「结果被截断」),现透传出去供 /api/v1/recall 暴露。
|
||||
// 本地兜底(IPC 错误 / 零结果走 localSearch)没有行数上限 ⇒ 恒 false。
|
||||
func (rc *RustLanceDBClient) SearchWithTruncation(table string, vector []float32, topK int, namespaceFilter string) ([]models.MemoryRecord, bool, error) {
|
||||
resp, err := rc.rpc(ipcReq{
|
||||
Type: "lancedb_search", Vector: vector, TopK: topK * 2, Namespace: namespaceFilter,
|
||||
})
|
||||
if err != nil {
|
||||
return rc.localSearch(vector, topK, namespaceFilter), nil
|
||||
return rc.localSearch(vector, topK, namespaceFilter), false, nil
|
||||
}
|
||||
|
||||
var raw []struct {
|
||||
@@ -266,6 +271,8 @@ func (rc *RustLanceDBClient) Search(table string, vector []float32, topK int, na
|
||||
// 用 *float64 区分「NULL(存量未声明)」与「显式 0.0」。
|
||||
Confidence *float64 `json:"confidence"`
|
||||
ConfidenceSource string `json:"confidence_source"`
|
||||
// A10 (t_d618dc7c):人工有用单独列(可空,NULL=从未人工点过)读侧投影——不补则静默 no-op。
|
||||
HumanUsefulCount *int `json:"human_useful_count"`
|
||||
// vector 必须解析:近重复去重(commit near_dup / 蒸馏写入 gate /
|
||||
// admin/dedup 端点)全部依赖 records 携带向量做 cosine 比较。
|
||||
// 此前缺失该字段 → Vector 恒为空 → 所有 len(Vector)>0 判定失败 →
|
||||
@@ -279,7 +286,7 @@ func (rc *RustLanceDBClient) Search(table string, vector []float32, topK int, na
|
||||
}
|
||||
if len(raw) == 0 {
|
||||
log.Printf("[ipc] Search: resp.ReportJSON unmarshal gave 0 results, falling back to localSearch (Result=%q)", resp.Result)
|
||||
return rc.localSearch(vector, topK, namespaceFilter), nil
|
||||
return rc.localSearch(vector, topK, namespaceFilter), false, nil
|
||||
}
|
||||
log.Printf("[ipc] Search: got %d results from Rust IPC (Result=%q)", len(raw), resp.Result)
|
||||
|
||||
@@ -346,6 +353,8 @@ func (rc *RustLanceDBClient) Search(table string, vector []float32, topK int, na
|
||||
// 端到端测试才抓得到)。
|
||||
Confidence: r.Confidence,
|
||||
ConfidenceSource: r.ConfidenceSource,
|
||||
// A10 (t_d618dc7c):人工有用单独列输出映射(解包有 + 输出有,缺则 recall 恒 null)
|
||||
HumanUsefulCount: r.HumanUsefulCount,
|
||||
// 2026-10-14 (t_763e8701):bi-temporal 三列输出映射(缺 ⇒ 解包有、输出无,recall 拿不到)
|
||||
ValidFrom: r.ValidFrom,
|
||||
ValidTo: r.ValidTo,
|
||||
@@ -355,7 +364,13 @@ func (rc *RustLanceDBClient) Search(table string, vector []float32, topK int, na
|
||||
break
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
return out, resp.Truncated, nil
|
||||
}
|
||||
|
||||
// Search 旧签名(不报告截断)。截断标志的透传走 SearchWithTruncation(t_3aa2a8c1)。
|
||||
func (rc *RustLanceDBClient) Search(table string, vector []float32, topK int, namespaceFilter string) ([]models.MemoryRecord, error) {
|
||||
recs, _, err := rc.SearchWithTruncation(table, vector, topK, namespaceFilter)
|
||||
return recs, err
|
||||
}
|
||||
|
||||
// Scan 全表扫描存活记忆(走 Rust IPC `lancedb_scan`:is_deleted=false + namespace 下推)。
|
||||
@@ -401,6 +416,8 @@ func (rc *RustLanceDBClient) Scan(limit int, namespaceFilter string) ([]models.M
|
||||
// 2026-10-08 (Gap#2):scan 路径同样要投影,否则该路径返回的 confidence 恒 nil
|
||||
Confidence *float64 `json:"confidence"`
|
||||
ConfidenceSource string `json:"confidence_source"`
|
||||
// A10 (t_d618dc7c):人工有用单独列(可空)读侧投影
|
||||
HumanUsefulCount *int `json:"human_useful_count"`
|
||||
// 2026-10-14 (t_763e8701):bi-temporal 三列(scan 列表 / export 也需要)
|
||||
ValidFrom string `json:"valid_from"`
|
||||
ValidTo string `json:"valid_to"`
|
||||
@@ -436,6 +453,7 @@ func (rc *RustLanceDBClient) Scan(limit int, namespaceFilter string) ([]models.M
|
||||
// GA-P0-1 (t_718d8707):scan 读侧接线
|
||||
NotUsefulCount: int(r.NotUsefulCount), HumanNotUsefulCount: int(r.HumanNotUsefulCount), DedupDownweightCount: int(r.DedupDownweightCount),
|
||||
Confidence: r.Confidence, ConfidenceSource: r.ConfidenceSource, // 2026-10-08 (Gap#2)
|
||||
HumanUsefulCount: r.HumanUsefulCount, // A10 (t_d618dc7c):人工有用单独列输出映射
|
||||
// 2026-10-14 (t_763e8701):bi-temporal 三列输出映射(scan 列表 / export)
|
||||
ValidFrom: r.ValidFrom,
|
||||
ValidTo: r.ValidTo,
|
||||
@@ -717,6 +735,27 @@ func (rc *RustLanceDBClient) Update(table, id string, fields map[string]any) err
|
||||
m.UsefulCount = inc
|
||||
}
|
||||
}
|
||||
// A10 (t_d618dc7c):人工有用单独计数 —— 本地读缓存同步(*int,NULL 时先置 0 再自增)
|
||||
if v, ok := fields["human_useful_count"]; ok {
|
||||
switch inc := v.(type) {
|
||||
case map[string]string:
|
||||
if inc["$inc"] == "1" {
|
||||
if m.HumanUsefulCount == nil {
|
||||
m.HumanUsefulCount = new(int)
|
||||
}
|
||||
*m.HumanUsefulCount++
|
||||
}
|
||||
case map[string]interface{}:
|
||||
if inc["$inc"] == "1" {
|
||||
if m.HumanUsefulCount == nil {
|
||||
m.HumanUsefulCount = new(int)
|
||||
}
|
||||
*m.HumanUsefulCount++
|
||||
}
|
||||
case int:
|
||||
m.HumanUsefulCount = &inc
|
||||
}
|
||||
}
|
||||
if v, ok := fields["not_useful_count"]; ok {
|
||||
switch inc := v.(type) {
|
||||
case map[string]string:
|
||||
@@ -874,8 +913,17 @@ func (rc *RustLanceDBClient) GetTopByQuality(agentID string, limit int) ([]model
|
||||
// - 退避 50/100/200ms(指数、cap 200ms),总最坏 +350ms(背景软删按分钟/小时节奏,可忽略);
|
||||
// - 更新是 SET is_deleted=true, updated_at=... 纯赋值 → 对最新已提交版本重放安全(幂等不累加);
|
||||
// - 仅「可重试冲突」重试;其余错误(空 id / "id required" / schema 错)立即返回、不重试。
|
||||
// t_cf680fad 机制修复:预算 3→5 + 退避加 jitter(破锁步雷群)。
|
||||
// 根因:runMerges O(n²) 软删 + 并发遗忘碎片器 在同一张 memories 单表上制造 MVCC 写风暴;
|
||||
// 旧 3 次/~350ms 预算下,风暴每 <100ms 推进一版 ⇒ 每次重放都撞新版本 ⇒ 4 次全败
|
||||
// (实证:单次 op 的 3 次重放 version 598346→598347→598348→598350 逐版推进仍耗尽)。
|
||||
// ① 预算 3→5:给单次 op 多两次机会跨过风暴窗口;
|
||||
// ② jitter(退避上叠 0~50% 随机量):破 50+ 并发 SoftDelete 全在同一时刻重放的锁步雷群
|
||||
// —— 锁步本身就是让冲突「持续」的主因(不 jitter,退避再长也齐步撞);
|
||||
// ③ 耗尽 ⇒ 落 durable pending 队列(softdelete_pending.go),启动/每 tick 各 drain 一次,
|
||||
// 风暴退去后重放必落盘(SET is_deleted=true 幂等)。
|
||||
const (
|
||||
softDeleteMaxRetries = 3
|
||||
softDeleteMaxRetries = 5
|
||||
softDeleteBackoffBase = 50 * time.Millisecond
|
||||
softDeleteBackoffCap = 200 * time.Millisecond
|
||||
)
|
||||
@@ -914,16 +962,20 @@ func isRetryableLanceConflict(err error) bool {
|
||||
}
|
||||
|
||||
// retrySoftDeleteRPC 对 lancedb_update 调用做有界指数退避重试(仅可重试冲突)。
|
||||
// 返回:首次成功或重试后成功 = nil;耗尽 = 含尝试次数的明确错误;非可重试错误 = 原样立即返回。
|
||||
// t_cf680fad:退避加 jitter(破锁步雷群)+ 耗尽落 durable pending 队列(风暴退去后必落盘)。
|
||||
// 返回:首次成功或重试后成功 = nil;耗尽 = 已入 pending 队列 + 含尝试次数的明确错误;
|
||||
// 非可重试错误 = 原样立即返回(不入队,那些是 schema/id 等真错误,入队无意义)。
|
||||
func (rc *RustLanceDBClient) retrySoftDeleteRPC(id, reason, fields string) error {
|
||||
backoff := softDeleteBackoffBase
|
||||
var lastErr error
|
||||
for attempt := 0; attempt <= softDeleteMaxRetries; attempt++ {
|
||||
if attempt > 0 {
|
||||
// 第 attempt 次重试(1-based):attempt=1→50ms, 2→100ms, 3→200ms
|
||||
// 第 attempt 次重试(1-based):指数退避 + 0~50% jitter(破 50+ 并发软删锁步重放)
|
||||
jitter := time.Duration(rand.Intn(int(backoff/2) + 1))
|
||||
sleepDur := backoff + jitter
|
||||
log.Printf("[ipc] SoftDelete retry attempt %d/%d id=%s reason=%s error=%v",
|
||||
attempt, softDeleteMaxRetries, id, reason, lastErr)
|
||||
time.Sleep(backoff)
|
||||
time.Sleep(sleepDur)
|
||||
if backoff < softDeleteBackoffCap {
|
||||
backoff *= 2
|
||||
if backoff > softDeleteBackoffCap {
|
||||
@@ -939,10 +991,12 @@ func (rc *RustLanceDBClient) retrySoftDeleteRPC(id, reason, fields string) error
|
||||
return nil
|
||||
}
|
||||
if !isRetryableLanceConflict(err) {
|
||||
return err // 非可重试错误:立即返回,不重试
|
||||
return err // 非可重试错误:立即返回,不重试(也无需入队)
|
||||
}
|
||||
lastErr = err
|
||||
}
|
||||
// t_cf680fad:MVCC 冲突耗尽 ⇒ 落 durable 队列,启动/每 tick drain 重放(SET 幂等,必落盘)。
|
||||
EnqueuePendingSoftDelete(id, reason)
|
||||
return fmt.Errorf("soft delete persist (id=%s): retry exhausted after %d retries: %w",
|
||||
id, softDeleteMaxRetries, lastErr)
|
||||
}
|
||||
@@ -1344,6 +1398,22 @@ func mapInt(m map[string]interface{}, key string) int {
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// mapIntPtr — A10 (t_d618dc7c):可空整数读侧映射(NULL=从未人工点过 ⇒ nil,区别于显式 0)。
|
||||
// Rust 侧 human_useful_count 为 Option<i64>,serde 序列化后 null ⇒ 此处 v 缺失/nil ⇒ 返回 nil;
|
||||
// 非 null ⇒ int。缺这层「NULL≠0」区分即静默 no-op(同 mapFloatPtr 之于 confidence)。
|
||||
func mapIntPtr(m map[string]interface{}, key string) *int {
|
||||
if v, ok := m[key]; ok {
|
||||
if v == nil {
|
||||
return nil // 显式 null(从未人工标记)
|
||||
}
|
||||
if f, ok := v.(float64); ok {
|
||||
i := int(f)
|
||||
return &i
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func mapBool(m map[string]interface{}, key string) bool {
|
||||
if v, ok := m[key]; ok {
|
||||
if b, ok := v.(bool); ok {
|
||||
@@ -1433,6 +1503,8 @@ func (rc *RustLanceDBClient) GetSkillCandidates(minRecalls, limit int) ([]models
|
||||
// GA-P0-1 (t_718d8707):读侧接线拆分后的新字段(漏接线 ⇒ 加列也读不出,静默 no-op)
|
||||
m.HumanNotUsefulCount = mapInt(r, "human_not_useful_count")
|
||||
m.DedupDownweightCount = mapInt(r, "dedup_downweight_count")
|
||||
// A10 (t_d618dc7c):人工有用单独列(可空,NULL=从未人工点过)读侧接线
|
||||
m.HumanUsefulCount = mapIntPtr(r, "human_useful_count")
|
||||
m.RecallCount = mapInt(r, "recall_count")
|
||||
// GA-P0-2 (t_c6d41ac6):拆分字段读侧接线
|
||||
m.RecallSeenCount = mapInt(r, "recall_seen_count")
|
||||
@@ -1502,9 +1574,14 @@ func (rc *RustLanceDBClient) GetAuditLog(limit int) ([]map[string]interface{}, e
|
||||
func (rc *RustLanceDBClient) IncrementUseful(id string) error {
|
||||
// GA-P0-2 (t_c6d41ac6):正反馈 → useful_count+1 且 recall_verified_count+1。
|
||||
// 不再碰 recall_count(冻结,保留历史值——虚高保命来源已被切断)。
|
||||
// A10 (t_d618dc7c):同时把「人工有用」单独计到 human_useful_count($inc 1)。
|
||||
// 本方法只被人工 feedback 路径调用(feedback.go MarkUseful /api/v1/feedback/useful);
|
||||
// 机器自评/蒸馏质量路径走其它方法(IncrementDedupDownweight / IncrementHumanNotUseful /
|
||||
// IncrementRecallVerified,均不写 human_useful_count)⇒ 只有人工 useful 才 +1 此列。
|
||||
if err := rc.Update("memories", id, map[string]any{
|
||||
"useful_count": map[string]string{"$inc": "1"},
|
||||
"recall_verified_count": map[string]string{"$inc": "1"},
|
||||
"human_useful_count": map[string]string{"$inc": "1"},
|
||||
}); err != nil {
|
||||
log.Printf("[ipc] IncrementUseful failed id=%s: %v", id, err)
|
||||
return err
|
||||
|
||||
@@ -36,6 +36,8 @@ type memEntry struct {
|
||||
// GA-P0-1 (t_718d8707):拆分字段
|
||||
HumanNotUsefulCount int
|
||||
DedupDownweightCount int
|
||||
// A10 (t_d618dc7c):人工有用单独计数(指针:nil=NULL=从未被人工点过)
|
||||
HumanUsefulCount *int
|
||||
RecallCount int
|
||||
// GA-P0-2 (t_c6d41ac6):拆分字段
|
||||
RecallSeenCount int
|
||||
@@ -277,6 +279,13 @@ func (mlc *MemLanceClient) IncrementUseful(id string) error {
|
||||
e.UsefulCount++
|
||||
// GA-P0-2 (t_c6d41ac6):正反馈 → recall_verified_count++;不再动 recall_count(冻结)
|
||||
e.RecallVerifiedCount++
|
||||
// A10 (t_d618dc7c):人工 useful ⇒ human_useful_count 同步 +1(与生产 IncrementUseful 的
|
||||
// useful_count+recall_verified_count+human_useful_count 三 $inc 同一次 Update 一致)。
|
||||
// 机器自评/蒸馏路径(IncrementDedupDownweight / IncrementHumanNotUseful)不碰此列。
|
||||
if e.HumanUsefulCount == nil {
|
||||
e.HumanUsefulCount = new(int)
|
||||
}
|
||||
*e.HumanUsefulCount++
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -0,0 +1,112 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"log"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// t_23b6a940 AC7:热门度偏置的运行时看门狗(信号安全阈值兜底)。
|
||||
//
|
||||
// 背景:FeedbackScore legacy 态把 recall_verified_count(热度)当质量信号 ⇒
|
||||
// 热门记忆硬顶 rank1(金标 119 条实测 MRR -22%)。离线 A/B 暴露问题的间隔
|
||||
// 长(本次隔了一整晚)。本看门狗是**运行时兜底**:最终 rank1 记忆的
|
||||
// recall_verified_count 连续 N 次超过全库 p95 阈值 ⇒ 打一条 ALERT 日志行
|
||||
// (传感器通道),传感器端脚本(~/.hermes/scripts/tools/zhiyi-popwatch.py,
|
||||
// 挂既有告警通道)把快照转成告警。
|
||||
//
|
||||
// 默认只报不改(report-only):
|
||||
// - 不改排序、不写库、不发告警(告警由传感器端既有通道发);
|
||||
// - 默认**开**(零排序影响、每 recall 一次 env 读 + 临界一次锁,开销可忽略),
|
||||
// ZHIYI_POPWATCH=0 可完全关闭(关闭 ⇒ PopWatchRecord 立即返回,零行为变化)。
|
||||
//
|
||||
// 阈值按全库实测标定(47148 条,2026-10-18 复测):recall_verified_count
|
||||
// 全库 p95=4、非零 p99=62(≈全库 top 1% 热层)、max=389、非零 7416 条:
|
||||
//
|
||||
// ZHIYI_POPWATCH_P95 = 62(触发阈值,rank1 verified 严格大于它才累计;
|
||||
// = 非零 p99,MoA 复核标定值)
|
||||
// ZHIYI_POPWATCH_N = 5 (连续多少个 query 触发 ALERT)
|
||||
func PopWatchEnabled() bool {
|
||||
switch strings.ToLower(os.Getenv("ZHIYI_POPWATCH")) {
|
||||
case "0", "false", "off":
|
||||
return false
|
||||
default:
|
||||
return true // 默认开(report-only)
|
||||
}
|
||||
}
|
||||
|
||||
func popWatchThreshold() int64 { return int64(envInt("ZHIYI_POPWATCH_P95", 62)) }
|
||||
func popWatchN() int { return envInt("ZHIYI_POPWATCH_N", 5) }
|
||||
|
||||
// popWatchState 进程内观测状态(每实例一份;「连续 N 个 query」语义就是进程内的)。
|
||||
type popWatchState struct {
|
||||
mu sync.Mutex
|
||||
consecutive int // rank1 verified > 阈值 的连续 query 数(否则清零)
|
||||
lastRank1ID string // 最近一次 recall 的 rank1 记忆 id(审计用)
|
||||
lastVerified int64 // 最近一次 recall 的 rank1 verified
|
||||
lastSeenAt time.Time
|
||||
alertsFired int // 本进程累计 ALERT 次数
|
||||
lastAlertAt time.Time
|
||||
}
|
||||
|
||||
// PopWatch 是全局观测器(GET /api/v1/popwatch 读快照;传感器端据此告警)。
|
||||
var PopWatch = &popWatchState{}
|
||||
|
||||
// PopWatchRecord 观测一次 recall 的最终 rank1 的 recall_verified_count。
|
||||
// 调用点:recallUncached 末尾(主路径漏斗——热度偏置恰好作用于此处)。
|
||||
// 门关(ZHIYI_POPWATCH=0)⇒ 立即返回,零行为变化。
|
||||
// 越阈连续满 N 个 query ⇒ 打一条 ALERT 日志行(同一连续段只在满 N 瞬间打一次;
|
||||
// 之后继续越阈不刷屏,回落再爬升才再打)。
|
||||
func PopWatchRecord(rank1ID string, rank1Verified int64) {
|
||||
if !PopWatchEnabled() {
|
||||
return
|
||||
}
|
||||
threshold := popWatchThreshold()
|
||||
n := popWatchN()
|
||||
PopWatch.mu.Lock()
|
||||
defer PopWatch.mu.Unlock()
|
||||
if rank1Verified > threshold {
|
||||
PopWatch.consecutive++
|
||||
} else {
|
||||
PopWatch.consecutive = 0
|
||||
}
|
||||
PopWatch.lastRank1ID = rank1ID
|
||||
PopWatch.lastVerified = rank1Verified
|
||||
PopWatch.lastSeenAt = time.Now()
|
||||
if PopWatch.consecutive == n {
|
||||
PopWatch.alertsFired++
|
||||
PopWatch.lastAlertAt = time.Now()
|
||||
log.Printf("[popwatch] ALERT: rank1 recall_verified_count=%d > p95 阈值 %d 连续 %d 个 query(report-only:不改排序/不写库;回滚:ZHIYI_POS_MODE=legacy 或 ZHIYI_FEEDBACK_BIAS=0)",
|
||||
rank1Verified, threshold, n)
|
||||
}
|
||||
}
|
||||
|
||||
// PopWatchSnapshot 供 GET /api/v1/popwatch(传感器端脚本读它转既有告警通道)。
|
||||
func PopWatchSnapshot() map[string]any {
|
||||
threshold := popWatchThreshold()
|
||||
n := popWatchN()
|
||||
PopWatch.mu.Lock()
|
||||
defer PopWatch.mu.Unlock()
|
||||
lastSeen := ""
|
||||
if !PopWatch.lastSeenAt.IsZero() {
|
||||
lastSeen = PopWatch.lastSeenAt.Format(time.RFC3339)
|
||||
}
|
||||
lastAlert := ""
|
||||
if !PopWatch.lastAlertAt.IsZero() {
|
||||
lastAlert = PopWatch.lastAlertAt.Format(time.RFC3339)
|
||||
}
|
||||
return map[string]any{
|
||||
"enabled": PopWatchEnabled(),
|
||||
"threshold_p95": threshold,
|
||||
"consecutive_n": n,
|
||||
"consecutive": PopWatch.consecutive,
|
||||
"last_rank1_id": PopWatch.lastRank1ID,
|
||||
"last_rank1_verified": PopWatch.lastVerified,
|
||||
"last_seen_at": lastSeen,
|
||||
"alerts_fired": PopWatch.alertsFired,
|
||||
"last_alert_at": lastAlert,
|
||||
"note": "report-only 观测:rank1 recall_verified_count 连续越 p95 阈值 = 热门度偏置行为证据;回滚 ZHIYI_POS_MODE=legacy / ZHIYI_FEEDBACK_BIAS=0",
|
||||
}
|
||||
}
|
||||
@@ -3,6 +3,7 @@ package storage
|
||||
import (
|
||||
"math"
|
||||
"os"
|
||||
"strings"
|
||||
|
||||
"github.com/xiaoxue/memoryweave/internal/models"
|
||||
)
|
||||
@@ -19,6 +20,47 @@ import (
|
||||
// 不新增缓存/Redis 依赖;部署走既有「停→构建→起」流程(本卡仅 sidecar A/B,不动生产)。
|
||||
var vFuseUseV = os.Getenv("ZHIYI_V_FUSE_ENABLE") != "1"
|
||||
|
||||
// t_7c0538eb(2026-10-18):低分(可信评分)记忆在检索排序里降权(可逆,替代「真丢弃」)。
|
||||
// ZHIYI_LOWSCORE_DOWNWEIGHT 取值语义(回滚开关,默认关 = 零行为变化):
|
||||
// - 未设 / "0" / 非 "1":lowScoreDownweight 恒返回 1.0 ⇒ fuseQuality 逐位等价现状(零风险);
|
||||
// - "1":对「可靠打分且低分」(0<qs<lowScoreDWThreshold, IsReliableQuality=true) 的记忆,
|
||||
// 把其 quality 贡献按比例 q/阈值 压到 [lowScoreDWFloor, 1.0)——低分记忆排得更低,**不删数据**。
|
||||
// qs<=0(未打分 / bug 残留)与 qs≈0.755(echo 常量)IsReliableQuality=false,本就走
|
||||
// 「仅 importance」分支,不受本降权影响(降权只作用于「可信低分」那一档)。
|
||||
// 每次调用读 env(测试可 t.Setenv 翻转,同 timeBiasTrustGate 模式);回滚 = 去掉/置 0 env 重启。
|
||||
func lowScoreDwGate() bool {
|
||||
switch strings.ToLower(os.Getenv("ZHIYI_LOWSCORE_DOWNWEIGHT")) {
|
||||
case "1", "true", "on":
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// lowScoreDWThreshold 与 GA-P1 影子 drop 阈值(ga-p1-threshold-replay.py DROP_T=0.30)对齐。
|
||||
const (
|
||||
lowScoreDWThreshold = 0.30
|
||||
lowScoreDWFloor = 0.5 // 最低的可信低分仍保留 50% 质量贡献(不彻底清零,可逆)
|
||||
)
|
||||
|
||||
// lowScoreDownweight 返回质量贡献缩放因子 f ∈ [lowScoreDWFloor, 1.0]。
|
||||
// gate 关 / 不可信(0.0、0.755) / qs>=阈值 ⇒ 1.0(不降权,逐位等价现状)。
|
||||
func lowScoreDownweight(rawQuality float64) float64 {
|
||||
if !lowScoreDwGate() {
|
||||
return 1.0
|
||||
}
|
||||
if rawQuality <= 0 || !models.IsReliableQuality(rawQuality) {
|
||||
return 1.0
|
||||
}
|
||||
if rawQuality >= lowScoreDWThreshold {
|
||||
return 1.0
|
||||
}
|
||||
f := rawQuality / lowScoreDWThreshold
|
||||
if f < lowScoreDWFloor {
|
||||
f = lowScoreDWFloor
|
||||
}
|
||||
return f
|
||||
}
|
||||
|
||||
// 召回排序「频率 + 时间衰减」融合公式 —— 从 lancedb_ipc.go Search() 内联代码抽出的纯函数。
|
||||
// 行为逐位等价于原内联实现(t_5028c14d),不改任何数值常量。
|
||||
// 调用链:recall.go Recall() → Search() → 本文件三函数 → QualityScore 供排序。
|
||||
@@ -52,10 +94,13 @@ func computeImportance(days float64, recallSeenCount int64, volatile bool) float
|
||||
// rawQuality 经 models.IsReliableQuality 判定可信度(0=未打分/历史常量 bug 视为不可信)。
|
||||
func fuseQuality(computedImportance, rawQuality, vValue float64) float64 {
|
||||
qsTrusted := models.IsReliableQuality(rawQuality)
|
||||
// t_7c0538eb:低分(可信)降权因子;gate 关时恒 1.0(逐位等价现状,零行为变化)。
|
||||
dw := lowScoreDownweight(rawQuality)
|
||||
var finalQuality float64
|
||||
if qsTrusted {
|
||||
// QualityScore from LLM (0-1) + computed importance, capped at 1.0
|
||||
finalQuality = math.Min(0.4*computedImportance+0.6*rawQuality, 1.0)
|
||||
// t_7c0538eb:quality 贡献 × dw(可信低分按比例压低、可逆;不删数据)
|
||||
finalQuality = math.Min(0.4*computedImportance+0.6*rawQuality*dw, 1.0)
|
||||
} else {
|
||||
// 质量分不可信 → 仅用 importance([qstrust-ipc] 日志留在调用处)
|
||||
finalQuality = math.Min(computedImportance, 1.0)
|
||||
@@ -65,8 +110,8 @@ func fuseQuality(computedImportance, rawQuality, vValue float64) float64 {
|
||||
// 等价于现状 vValue<=0 行为;0.5:0.5 分支的 V 半边移除后归一回到无 V 式)。
|
||||
if vFuseUseV && vValue > 0 {
|
||||
if qsTrusted {
|
||||
// 30% V值 + 30% importance + 40% LLM quality
|
||||
finalQuality = 0.3*vValue + 0.3*computedImportance + 0.4*rawQuality
|
||||
// 30% V值 + 30% importance + 40% LLM quality(t_7c0538eb:quality 半边 × dw)
|
||||
finalQuality = 0.3*vValue + 0.3*computedImportance + 0.4*rawQuality*dw
|
||||
} else {
|
||||
// 质量分不可信 → 去掉它的 40%,剩余 30:30 重新归一为 50:50。
|
||||
finalQuality = 0.5*vValue + 0.5*computedImportance
|
||||
|
||||
@@ -0,0 +1,134 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"math"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// t_7c0538eb AC5:低分(可信评分)记忆在检索排序里降权(可逆,不删数据)+ 回滚开关。
|
||||
//
|
||||
// 召回排序的单一融合点 = fuseQuality(lancedb_ipc.go:333 调用;纯函数)。本测试用一组
|
||||
// 固定候选(computedImportance + rawQuality,vValue=0 以隔离基线分支)按 fuseQuality 降序
|
||||
// 排序,验证:
|
||||
// ① gate 关(默认/回滚)= 基线序;
|
||||
// ② gate 开 = 「可信低分但高重要性」的记忆被压到「普通分低重要性」记忆之后(reorder);
|
||||
// ③ gate 再关 = 逐位复原到基线序(可逆)。
|
||||
// 另验证 0.0(未打分 / bug 残留)与 0.755(echo 常量)走「不可信→仅 importance」分支,
|
||||
// 不受本降权影响(dw 只作用于「可信低分」一档)——数值在 gate 开/关下不变。
|
||||
func TestFuseQuality_LowScoreDownweight_Ac5(t *testing.T) {
|
||||
// 固定候选:imp、rawQuality、id。vValue=0。
|
||||
type mem struct{
|
||||
id string
|
||||
imp float64
|
||||
qs float64
|
||||
}
|
||||
cands := []mem{
|
||||
{"L_low_hiimp", 1.0, 0.25}, // 可信低分 + 高重要性 → 被降权应下沉
|
||||
{"M_norm_loimp", 0.8, 0.35}, // 普通分(>=0.30 不降权)+ 较低重要性
|
||||
{"H_high", 0.5, 0.85}, // 高质量
|
||||
{"U_unscored", 1.0, 0.0}, // 0.0 未打分(bug 残留,不可信)
|
||||
{"E_echo", 0.6, 0.755}, // 0.755 echo 常量(不可信)
|
||||
}
|
||||
|
||||
// rank(order 用):按 fuseQuality(imp, qs, 0) 降序返回 id 列表 + 各 id 的 finalQuality。
|
||||
rank := func() (order []string, scores map[string]float64) {
|
||||
scores = map[string]float64{}
|
||||
type sc struct {
|
||||
m mem
|
||||
f float64
|
||||
}
|
||||
sclice := make([]sc, 0, len(cands))
|
||||
for _, m := range cands {
|
||||
sclice = append(sclice, sc{m, fuseQuality(m.imp, m.qs, 0)})
|
||||
}
|
||||
for i := 1; i < len(sclice); i++ {
|
||||
for j := i; j > 0 && sclice[j].f > sclice[j-1].f; j-- {
|
||||
sclice[j], sclice[j-1] = sclice[j-1], sclice[j]
|
||||
}
|
||||
}
|
||||
order = make([]string, 0, len(sclice))
|
||||
for _, s := range sclice {
|
||||
order = append(order, s.m.id)
|
||||
scores[s.m.id] = s.f
|
||||
}
|
||||
return order, scores
|
||||
}
|
||||
|
||||
approxEq := func(a, b float64) bool { return math.Abs(a-b) < 1e-9 }
|
||||
|
||||
// ── ① 基线(gate 关)─────────────────────────────────────────────
|
||||
t.Setenv("ZHIYI_LOWSCORE_DOWNWEIGHT", "0")
|
||||
baseOrder, baseScores := rank()
|
||||
t.Logf("基线(gate关) 序: %v 分: L=%.4f M=%.4f H=%.4f U=%.4f E=%.4f",
|
||||
baseOrder, baseScores["L_low_hiimp"], baseScores["M_norm_loimp"], baseScores["H_high"],
|
||||
baseScores["U_unscored"], baseScores["E_echo"])
|
||||
|
||||
// ── ② gate 开(降权生效)──────────────────────────────────────────
|
||||
t.Setenv("ZHIYI_LOWSCORE_DOWNWEIGHT", "1")
|
||||
onOrder, onScores := rank()
|
||||
t.Logf("gate开 序: %v 分: L=%.4f M=%.4f H=%.4f U=%.4f E=%.4f",
|
||||
onOrder, onScores["L_low_hiimp"], onScores["M_norm_loimp"], onScores["H_high"],
|
||||
onScores["U_unscored"], onScores["E_echo"])
|
||||
|
||||
// 断言①:可信低分 L 被压低(分数下降),普通分 M / 高质量 H 不变。
|
||||
if !(onScores["L_low_hiimp"] < baseScores["L_low_hiimp"]) {
|
||||
t.Fatalf("AC5 降权未生效:L 开gate后(%.4f) 未低于 基线(%.4f)",
|
||||
onScores["L_low_hiimp"], baseScores["L_low_hiimp"])
|
||||
}
|
||||
if !approxEq(onScores["M_norm_loimp"], baseScores["M_norm_loimp"]) {
|
||||
t.Fatalf("AC5 误伤普通分:M(q=0.35>=阈值) 开gate后(%.4f) ≠ 基线(%.4f),本应不变",
|
||||
onScores["M_norm_loimp"], baseScores["M_norm_loimp"])
|
||||
}
|
||||
if !approxEq(onScores["H_high"], baseScores["H_high"]) {
|
||||
t.Fatalf("AC5 误伤高质量:H 开gate后(%.4f) ≠ 基线(%.4f)", onScores["H_high"], baseScores["H_high"])
|
||||
}
|
||||
// 断言②:0.0 / 0.755 不可信行不受降权影响(数值在开/关下逐位不变)。
|
||||
if !approxEq(onScores["U_unscored"], baseScores["U_unscored"]) {
|
||||
t.Fatalf("AC5 误伤 0.0 未打分行:U 开gate后(%.4f) ≠ 基线(%.4f),本应不变(不可信分支)",
|
||||
onScores["U_unscored"], baseScores["U_unscored"])
|
||||
}
|
||||
if !approxEq(onScores["E_echo"], baseScores["E_echo"]) {
|
||||
t.Fatalf("AC5 误伤 0.755 echo 行:E 开gate后(%.4f) ≠ 基线(%.4f)", onScores["E_echo"], baseScores["E_echo"])
|
||||
}
|
||||
// 断言③:出现 reorder —— 基线里 L 排在 M 前,开 gate 后 M 反超 L。
|
||||
if !(_idx(baseOrder, "L_low_hiimp") < _idx(baseOrder, "M_norm_loimp")) {
|
||||
t.Fatalf("前置不成立:基线序 L 应在 M 前,得 %v", baseOrder)
|
||||
}
|
||||
if !(_idx(onOrder, "M_norm_loimp") < _idx(onOrder, "L_low_hiimp")) {
|
||||
t.Fatalf("AC5 未 reorder:开gate后 M 应反超低分 L,得 %v", onOrder)
|
||||
}
|
||||
|
||||
// ── ③ 回滚(gate 再关)= 逐位复原基线 ───────────────────────────
|
||||
t.Setenv("ZHIYI_LOWSCORE_DOWNWEIGHT", "0")
|
||||
backOrder, backScores := rank()
|
||||
if !equalStrs(backOrder, baseOrder) {
|
||||
t.Fatalf("AC5 回滚未复原:gate 再关序 %v ≠ 基线 %v", backOrder, baseOrder)
|
||||
}
|
||||
for _, id := range []string{"L_low_hiimp", "M_norm_loimp", "H_high", "U_unscored", "E_echo"} {
|
||||
if !approxEq(backScores[id], baseScores[id]) {
|
||||
t.Fatalf("AC5 回滚未复原 %s:再关(%.6f) ≠ 基线(%.6f)", id, backScores[id], baseScores[id])
|
||||
}
|
||||
}
|
||||
t.Logf("回滚(gate再关) 序: %v —— 与基线逐位一致,低分 L 复原到 %.4f", backOrder, backScores["L_low_hiimp"])
|
||||
}
|
||||
|
||||
func _idx(order []string, id string) int {
|
||||
for i, x := range order {
|
||||
if x == id {
|
||||
return i
|
||||
}
|
||||
}
|
||||
return -1
|
||||
}
|
||||
|
||||
func equalStrs(a, b []string) bool {
|
||||
if len(a) != len(b) {
|
||||
return false
|
||||
}
|
||||
for i := range a {
|
||||
if a[i] != b[i] {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
+302
-14
@@ -2,6 +2,7 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log"
|
||||
@@ -32,6 +33,19 @@ import (
|
||||
var (
|
||||
confidenceLambda = envFloat("ZHIYI_CONFIDENCE_LAMBDA", 0) // 0 = 关闭
|
||||
confidenceMean = envFloat("ZHIYI_CONFIDENCE_MEAN", 0.7) // 去中心基准
|
||||
// 2026-10-21(t_5624c0ff → 牧尘批准「按推荐执行」):融合权重可配。
|
||||
// 1.0 = 纯向量(reranker 余弦分,不混 BM25/质量分);0.7 = 历史默认。
|
||||
// 依据:MoA 四槽 + 实测 strict n=134 —— 现状 fuse 0.1125 vs 纯向量 0.1898(现状是负收益 39%),
|
||||
// 且 A/C 绝对差 0.0121 在 n=134 上不显著 ⇒ 不值得为它背 9% 硬失败率(worst −100%)。
|
||||
// 回滚:ZHIYI_FUSION_W=0.7(或删掉该 env)即逐位回到改动前。
|
||||
fusionW = envFloat("ZHIYI_FUSION_W", 0.7)
|
||||
// fuseModeProof 2026-10-26(Step3,对应 t_9acaa74d §5 A′ 的 nouse 臂):
|
||||
// 融合第二槽改为 **QualityFromProof(证据条数,确定性)**,彻底不含
|
||||
// importance=recency×(1+ln(seen)) 与 V 值 —— 老卡实测「usage 特征单独为负资产」
|
||||
// (imp-only 0.0579 / seen-only 0.0335 NDCG@5,只有纯向量的 1/3)。
|
||||
// 默认关 ⇒ 逐位等价现状;开启需同时把 FUSION_W 调<1(否则第二槽权重 0,等于没开)。
|
||||
// 回滚 = 去掉 ZHIYI_FUSE_MODE。每次调用读 env(测试可 t.Setenv 翻转)。
|
||||
|
||||
// t_be3e958a 来源分层偏置:决策类来源(wiki:decision 等)在 MMR 之前加性加分,
|
||||
// 让「我们定了什么/阈值怎么定的」这类问题召回出决定本体而非流水。0 = 关闭。
|
||||
// 流水类(llm_distill/episode)不加分也不降权——偏置只抬决策,不动流水的相对位次。
|
||||
@@ -50,7 +64,7 @@ var (
|
||||
// 2026-10-13 时间维检索(t_e9a4d18f P0):stale 标记。
|
||||
// updated_at 距今超过该小时数 ⇒ 响应带 stale:true,让调用方「看见」可能过期。
|
||||
// 0 = 关闭(默认)⇒ 不产生任何 stale 标记,行为零变化。
|
||||
staleAfterHours = envFloat("ZHIYI_STALE_AFTER_HOURS", 0)
|
||||
staleAfterHours = envFloat("ZHIYI_STALE_AFTER_HOURS", 720)
|
||||
)
|
||||
|
||||
// ── t_d32e5f8a L2 模式召回注入(env 门控 ZHIYI_PATTERN_RECALL,默认 0 ⇒ 行为零变化)──
|
||||
@@ -95,6 +109,12 @@ func bitemporalEnabled() bool {
|
||||
return false
|
||||
}
|
||||
|
||||
// BitemporalEnabled 导出 bi-temporal 开关(供 API 层在补充通道/图谱注入后做同一口径过滤)。
|
||||
func BitemporalEnabled() bool { return bitemporalEnabled() }
|
||||
|
||||
// ValidityExpired 导出有效期判定(同上;空串=永久有效,解析失败保守视为未过期)。
|
||||
func ValidityExpired(validTo string, now time.Time) bool { return validityExpired(validTo, now) }
|
||||
|
||||
// validityExpired 判断记忆的语义有效期是否已过(valid_to < now ⇒ 失效)。
|
||||
// valid_to 空串 = 永久有效(NULL 语义,永不失效)。解析失败保守视为未失效(不误伤)。
|
||||
func validityExpired(validTo string, now time.Time) bool {
|
||||
@@ -305,6 +325,32 @@ func (p *RecallPipeline) SetMissRecorder(mr MissRecorder) {
|
||||
// 诊断时置 ZHIYI_QSTRUST_VERBOSE=1。
|
||||
var qstrustVerbose = os.Getenv("ZHIYI_QSTRUST_VERBOSE") == "1"
|
||||
|
||||
// nsFallbackMin 2026-10-26(牧尘令):ns 作用域兜底阈值 —— 作用域内候选少于此值就补一次
|
||||
// 全库搜索。背景:agent_id 派生的 <agent>-main 里几乎没货(实测 6/7 条),
|
||||
// 而不传 agent_id 是 200 条 ⇒ 带 agent_id 的调用静默劣化且不报错。
|
||||
// 默认 20;**ZHIYI_NS_FALLBACK_MIN=0 即关闭**(回滚开关,逐位回到改动前)。
|
||||
func nsFallbackMin() int {
|
||||
if v := strings.TrimSpace(os.Getenv("ZHIYI_NS_FALLBACK_MIN")); v != "" {
|
||||
if n, err := strconv.Atoi(v); err == nil && n >= 0 {
|
||||
return n
|
||||
}
|
||||
}
|
||||
return 20
|
||||
}
|
||||
|
||||
// fuseModeProof 2026-10-26(Step3 / t_9acaa74d §5 A′ 的 nouse 臂):
|
||||
// ZHIYI_FUSE_MODE=proof ⇒ 融合第二槽 = QualityFromProof(证据条数,确定性),
|
||||
// 不含 importance(recency×seen)/V。默认关 ⇒ 零行为变化。
|
||||
// 老卡实测:usage 特征单独为负资产(imp-only 0.0579 / seen-only 0.0335 @NDCG@5),
|
||||
// 而 nouse 臂 strict 0.2019 > 纯向量 0.1898 ⇒ 方向对,但尾部仍需保位机制。
|
||||
func fuseModeProof() bool {
|
||||
switch strings.ToLower(os.Getenv("ZHIYI_FUSE_MODE")) {
|
||||
case "proof", "nouse":
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// RecallMeta RecallWithMeta 返回的召回元数据(t_949840b1:candidates_scanned 名实相符)。
|
||||
// CandidatesScanned = 粗检索候选池真实大小(LanceDB ANN 检索返回 + 去重 +
|
||||
// 原始对话过滤之后的候选数,在 rerank/MMR 截断之前捕获)。
|
||||
@@ -319,6 +365,28 @@ var qstrustVerbose = os.Getenv("ZHIYI_QSTRUST_VERBOSE") == "1"
|
||||
type RecallMeta struct {
|
||||
CandidatesScanned int
|
||||
CacheHit bool
|
||||
// TraceID / Status / PartialReason(2026-10-26 t_trace_p0 + t_contract_p0):
|
||||
// 响应契约的最小可观测面 —— trace_id 贯穿整条链路,status=partial 时
|
||||
// 调用方不得把「空结果」解读为「不存在」。
|
||||
TraceID string
|
||||
Status string // complete | partial(空串等价 complete,向后兼容)
|
||||
PartialReason string
|
||||
// Truncated(t_3aa2a8c1):本次召回的粗检索单次重读是否被 Rust 侧行数上限截断
|
||||
// (lancedb_search 单次重读 cap,默认 20000,env ZHIYI_HEAVY_READ_MAX_ROWS 可调)。
|
||||
// true = 部分结果:candidates_scanned/count 反映截断后状态,调用方不得当作「全量」。
|
||||
// 缓存命中路径 / 纯 Go 后端(无行数上限)⇒ false。
|
||||
Truncated bool
|
||||
// StandingAnswerHit(t_standing_recall, 2026-10-25):本次召回被常设答案精确通道
|
||||
// 置顶注入时,记录命中的 question_id;未命中为空。仅观测用,不参与打分。
|
||||
StandingAnswerHit string
|
||||
}
|
||||
|
||||
// SearchTruncationReporter 可选扩展接口(t_3aa2a8c1):能报告「粗检索单次重读是否被
|
||||
// Rust 侧行数上限截断」的存储后端。RustLanceDBClient(IPC 路径)实现之;
|
||||
// 纯 Go 后端 / 测试 fake 不实现 ⇒ 按「恒未截断」处理(其 Search 没有行数上限,
|
||||
// 零值语义,行为不变)。
|
||||
type SearchTruncationReporter interface {
|
||||
SearchWithTruncation(table string, vector []float32, topK int, namespaceFilter string) ([]models.MemoryRecord, bool, error)
|
||||
}
|
||||
|
||||
// RecallWithMeta 同 Recall,但额外返回召回元数据(候选池真实大小 + 是否命中缓存)。
|
||||
@@ -348,7 +416,53 @@ func DefaultMMRDiversity() float64 {
|
||||
return 0.3
|
||||
}
|
||||
|
||||
// RecallWithMeta 是召回的唯一外部入口:内部先走原有语义/词面管线,再叠加
|
||||
// 「常设答案精确通道」(t_standing_recall,2026-10-25)。
|
||||
// 通道默认关(ZHIYI_STANDING_ANSWER=1 才生效)⇒ 关闭时与旧行为逐位一致。
|
||||
func (p *RecallPipeline) RecallWithMeta(query, namespace string, topK int, diversity float64) ([]models.RecallResult, RecallMeta, error) {
|
||||
return p.RecallWithMetaOpts(query, namespace, topK, diversity, false)
|
||||
}
|
||||
|
||||
// RecallWithMetaOpts 2026-10-26:allowNSFallback=true 时,若 namespace 作用域候选过少
|
||||
// (ZHIYI_NS_FALLBACK_MIN,默认 20),补一次全库搜索并合并。
|
||||
// 只有 **agent_id 派生** 的作用域才该传 true —— 显式传 namespace 的调用方传 false,
|
||||
// 否则会跨域泄漏不该看到的记忆。
|
||||
func (p *RecallPipeline) RecallWithMetaOpts(query, namespace string, topK int, diversity float64, allowNSFallback bool) ([]models.RecallResult, RecallMeta, error) {
|
||||
return p.RecallWithMetaTraced(context.Background(), query, namespace, topK, diversity, allowNSFallback, nil)
|
||||
}
|
||||
|
||||
// RecallWithMetaTraced(2026-10-26 t_trace_p0):带 ctx(deadline 可中断)+ 链路追踪的召回入口。
|
||||
// 旧方法全部委托到本方法、tr=nil ⇒ 旧调用方行为逐位不变。tr 非 nil 时记录各阶段耗时/通道贡献。
|
||||
// ctx 在每个阶段边界检查,超时 ⇒ 立即返回已得结果并把 status 标为 partial(reason=deadline_exceeded),
|
||||
// 调用方据此区分「真的没有」与「扫了一半就超时」(Air Canada 式静默幻觉的结构性入口)。
|
||||
func (p *RecallPipeline) RecallWithMetaTraced(ctx context.Context, query, namespace string, topK int, diversity float64, allowNSFallback bool, tr *RecallTrace) ([]models.RecallResult, RecallMeta, error) {
|
||||
if ctx == nil {
|
||||
ctx = context.Background()
|
||||
}
|
||||
results, meta, err := p.recallWithMetaInner(ctx, query, namespace, topK, diversity, allowNSFallback, tr)
|
||||
if tr != nil {
|
||||
meta.TraceID = tr.TraceID
|
||||
if tr.Status == "partial" {
|
||||
meta.Status = "partial"
|
||||
meta.PartialReason = tr.PartialReason
|
||||
}
|
||||
}
|
||||
if err != nil {
|
||||
return results, meta, err
|
||||
}
|
||||
if StandingAnswerEnabled() {
|
||||
if sa := LookupStandingAnswer(query); sa != nil {
|
||||
results = PrependStandingAnswer(results, sa, topK)
|
||||
meta.StandingAnswerHit = sa.QuestionID
|
||||
if tr != nil {
|
||||
tr.StandingAnswerHit = sa.QuestionID
|
||||
}
|
||||
}
|
||||
}
|
||||
return results, meta, err
|
||||
}
|
||||
|
||||
func (p *RecallPipeline) recallWithMetaInner(ctx context.Context, query, namespace string, topK int, diversity float64, allowNSFallback bool, tr *RecallTrace) ([]models.RecallResult, RecallMeta, error) {
|
||||
// topK 先归一:缓存键必须与写入侧用的 topK 一致。recallUncached 内部也会归一,
|
||||
// 若此处不先归一,topK<=0 的调用读到的是"归一前的键"、永远命中不了自己写的条目。
|
||||
if topK <= 0 {
|
||||
@@ -365,11 +479,17 @@ func (p *RecallPipeline) RecallWithMeta(query, namespace string, topK int, diver
|
||||
}
|
||||
p.recordRecallWrites(filtered)
|
||||
// t_1c12e000:命中缓存 ⇒ 显式告知调用方(candidates_scanned 恒 0)
|
||||
return filtered, RecallMeta{CacheHit: true}, nil
|
||||
st := time.Now()
|
||||
if tr != nil {
|
||||
tr.CacheHit = true
|
||||
tr.TopK = topK
|
||||
tr.StageNote("cache_hit", st, len(filtered), "候选池不可重建(读 60s 内快照)")
|
||||
}
|
||||
return filtered, RecallMeta{CacheHit: true, Status: "complete"}, nil
|
||||
}
|
||||
}
|
||||
}
|
||||
return p.recallUncached(query, namespace, topK, diversity)
|
||||
return p.recallUncached(ctx, query, namespace, topK, diversity, allowNSFallback, tr)
|
||||
}
|
||||
|
||||
// Recall 保持原签名(缓存 + 全量流程),行为不变。
|
||||
@@ -380,16 +500,30 @@ func (p *RecallPipeline) Recall(query, namespace string, topK int, diversity flo
|
||||
|
||||
// recallUncached Recall 的非缓存路径。候选池大小在 Step 2.5 过滤后、
|
||||
// Step 3 截断前捕获(真实粗检索池,不是最终产出数)。
|
||||
func (p *RecallPipeline) recallUncached(query, namespace string, topK int, diversity float64) ([]models.RecallResult, RecallMeta, error) {
|
||||
func (p *RecallPipeline) recallUncached(ctx context.Context, query, namespace string, topK int, diversity float64, allowFallback bool, tr *RecallTrace) ([]models.RecallResult, RecallMeta, error) {
|
||||
if topK <= 0 {
|
||||
topK = 10
|
||||
}
|
||||
if ctx == nil {
|
||||
ctx = context.Background()
|
||||
}
|
||||
if tr != nil {
|
||||
tr.TopK = topK
|
||||
}
|
||||
|
||||
// Step 1: Encode query
|
||||
stEncode := time.Now()
|
||||
queryVec, err := p.embedder.EncodeQuery(query)
|
||||
if err != nil {
|
||||
if tr != nil {
|
||||
tr.StageNote("encode", stEncode, 0, err.Error())
|
||||
tr.Error = err.Error()
|
||||
}
|
||||
return nil, RecallMeta{}, fmt.Errorf("recall encode: %w", err)
|
||||
}
|
||||
if tr != nil {
|
||||
tr.Stage("encode", stEncode, 0)
|
||||
}
|
||||
|
||||
// Step 2: Coarse search — own namespace + shared(§1.3 召回规则)
|
||||
// 默认搜自己的 namespace + shared,显式 include_namespaces 时才跨域
|
||||
@@ -400,8 +534,24 @@ func (p *RecallPipeline) recallUncached(query, namespace string, topK int, diver
|
||||
// ⇒ 默认召回只能看到极小一片(全表 10172 条,默认调用实测仅 11 个候选)。
|
||||
// 显式传 namespace / agent_id 时保持原作用域行为,不改变调用方语义。
|
||||
var candidates []models.MemoryRecord
|
||||
stCoarse := time.Now()
|
||||
// t_3aa2a8c1:Rust 侧单次重读行数上限的 truncated 上抛到 recall 响应。
|
||||
// 只有实现了可选扩展接口 SearchTruncationReporter 的后端(RustLanceDBClient)
|
||||
// 会上报截断;未实现的(纯 Go / 测试 fake)保持旧 Search ⇒ 恒未截断。
|
||||
truncated := false
|
||||
reporter, _ := p.lancedb.(SearchTruncationReporter)
|
||||
searchCoarse := func(topK int, ns string) ([]models.MemoryRecord, error) {
|
||||
if reporter != nil {
|
||||
recs, t, err := reporter.SearchWithTruncation("memories", queryVec, topK, ns)
|
||||
if err == nil {
|
||||
truncated = truncated || t
|
||||
}
|
||||
return recs, err
|
||||
}
|
||||
return p.lancedb.Search("memories", queryVec, topK, ns)
|
||||
}
|
||||
if namespace == "" {
|
||||
all, err := p.lancedb.Search("memories", queryVec, 200, "") // 空 ns = 不过滤 = 全库
|
||||
all, err := searchCoarse(200, "") // 空 ns = 不过滤 = 全库
|
||||
if err == nil {
|
||||
candidates = append(candidates, all...)
|
||||
}
|
||||
@@ -415,13 +565,43 @@ func (p *RecallPipeline) recallUncached(query, namespace string, topK int, diver
|
||||
if recallV2Enabled() {
|
||||
pool = 200 // 实测相关记忆可能排在向量 #126 ⇒ top-50 池根本够不着
|
||||
}
|
||||
nsResults, err := p.lancedb.Search("memories", queryVec, pool, ns)
|
||||
nsResults, err := searchCoarse(pool, ns)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
candidates = append(candidates, nsResults...)
|
||||
}
|
||||
}
|
||||
if tr != nil {
|
||||
tr.Stage("coarse_search", stCoarse, len(candidates))
|
||||
tr.CandidatesScanned = len(candidates)
|
||||
}
|
||||
// deadline 边界(t_contract_p0):粗检索已超时 ⇒ 立刻返回 partial,
|
||||
// 不继续做重排/MMR(省算力,也让调用方拿到明确的「不完整」信号)。
|
||||
if err := ctx.Err(); err != nil {
|
||||
if tr != nil {
|
||||
tr.MarkPartial("deadline_exceeded")
|
||||
}
|
||||
return nil, RecallMeta{CandidatesScanned: len(candidates), Truncated: truncated, Status: "partial", PartialReason: "deadline_exceeded"}, nil
|
||||
}
|
||||
// 2026-10-26:**ns 作用域兜底**(牧尘令,可逆)。
|
||||
// 背景:请求带 agent_id 时 resolveRecallNamespace 会派生 <agent>-main,
|
||||
// 而该 ns 里几乎没有货 —— 实测 default-main 池 6 条 / xiaoyi-main 6 / prof-cloud-main 7,
|
||||
// 不传 agent_id 则是 200 条。结果:召回静默劣化到个位数,不报错(见 note §M9)。
|
||||
// 规则:仅当 allowFallback=true(= agent_id 派生作用域,**显式传 namespace 的调用方不兜底**,
|
||||
// 避免跨域泄漏);候选数 < ZHIYI_NS_FALLBACK_MIN(默认 20,0=关闭)则补一次全库搜索并合并。
|
||||
// 回滚:ZHIYI_NS_FALLBACK_MIN=0(或删该 env)即逐位回到改动前。
|
||||
if allowFallback && namespace != "" {
|
||||
if minN := nsFallbackMin(); minN > 0 && len(candidates) < minN {
|
||||
fullLib, err := searchCoarse(200, "")
|
||||
if err == nil && len(fullLib) > 0 {
|
||||
log.Printf("[recall] ns 兜底:namespace=%q 仅 %d 条候选(<%d) ⇒ 合并全库 %d 条",
|
||||
namespace, len(candidates), minN, len(fullLib))
|
||||
candidates = append(candidates, fullLib...)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 去重(按 ID)
|
||||
seen := make(map[string]bool)
|
||||
unique := make([]models.MemoryRecord, 0, len(candidates))
|
||||
@@ -446,6 +626,7 @@ func (p *RecallPipeline) recallUncached(query, namespace string, topK int, diver
|
||||
// 门控关 ⇒ 本块不执行(零行为变化)。
|
||||
var ftsTail []models.MemoryRecord
|
||||
ftsScoreOf := map[string]float64{} // id → 词面分(只用于尾部排序)
|
||||
stFts := time.Now()
|
||||
if ftsEnabled() {
|
||||
ftsScores := ftsInstance.FtsSearch(query, 30)
|
||||
for id, s := range ftsScores {
|
||||
@@ -477,6 +658,9 @@ func (p *RecallPipeline) recallUncached(query, namespace string, topK int, diver
|
||||
log.Printf("[recall] FTS 词面通道命中 %d 条(候选池 %d)", len(ftsTail), len(candidates))
|
||||
}
|
||||
}
|
||||
if tr != nil {
|
||||
tr.Stage("fts_recall", stFts, len(ftsTail))
|
||||
}
|
||||
|
||||
// Step 2.5: 过滤原始对话内容(category=episodes/对话/对话记录)
|
||||
// 原因:原始对话未蒸馏,不应出现在 recall 结果中
|
||||
@@ -549,6 +733,7 @@ func (p *RecallPipeline) recallUncached(query, namespace string, topK int, diver
|
||||
// 因为下面会原地覆盖为 BM25 融合值(0.7*dbQuality + 0.3*kwScore)。
|
||||
// 响应需要展示 DB 原值("响应值==库值"),不能展示被融合改写后的值。
|
||||
dbQuality := make([]float64, len(candidates))
|
||||
stBm25 := time.Now()
|
||||
if len(candidates) > 0 {
|
||||
for i := range candidates {
|
||||
dbQuality[i] = candidates[i].QualityScore
|
||||
@@ -570,7 +755,7 @@ func (p *RecallPipeline) recallUncached(query, namespace string, topK int, diver
|
||||
// 默认关 ⇒ 行为零变化。这是「兜底」不是「替换」——已有可信 quality 不受影响。
|
||||
effQuality := ResolveQuality(&candidates[i], dbQuality[i])
|
||||
if models.IsReliableQuality(effQuality) {
|
||||
candidates[i].QualityScore = dbQuality[i]*0.7 + kwScore*0.3
|
||||
candidates[i].QualityScore = dbQuality[i]*fusionW + kwScore*(1-fusionW)
|
||||
} else {
|
||||
// 存量分数不可信(0=未打分 / 0.755=历史常量 bug,见 models.IsReliableQuality)
|
||||
// → 不拿假分数参与排序,权重全部让给关键词分。
|
||||
@@ -584,6 +769,10 @@ func (p *RecallPipeline) recallUncached(query, namespace string, topK int, diver
|
||||
}
|
||||
}
|
||||
|
||||
if tr != nil {
|
||||
tr.Stage("bm25", stBm25, len(candidates))
|
||||
}
|
||||
|
||||
if len(candidates) == 0 && len(ftsTail) == 0 {
|
||||
// 自动记录召回缺口(gap_scan 触发器会在 30min 冷却后分类)
|
||||
// t_2afae95e:FTS 尾部位命中也算有结果 ⇒ 不记 miss、不走空返回
|
||||
@@ -591,7 +780,7 @@ func (p *RecallPipeline) recallUncached(query, namespace string, topK int, diver
|
||||
p.recordMiss(query, namespace)
|
||||
}
|
||||
if len(ftsTail) == 0 {
|
||||
return []models.RecallResult{}, RecallMeta{CandidatesScanned: poolSize}, nil
|
||||
return []models.RecallResult{}, RecallMeta{CandidatesScanned: poolSize, Truncated: truncated}, nil
|
||||
}
|
||||
}
|
||||
|
||||
@@ -600,6 +789,7 @@ func (p *RecallPipeline) recallUncached(query, namespace string, topK int, diver
|
||||
// 存储的 Vector,直接算 cosine 即可。旧实现把 query + 50~100 条候选文档重新送进
|
||||
// 嵌入服务(int8 ONNX CPU),占 recall 总耗时 90%+,是 Hermes 记忆预取 8s 预算
|
||||
// 被打爆的直接原因。向量不齐/维度不符时自动回退文本重排(行为与旧版一致)。
|
||||
stRerank := time.Now()
|
||||
docs := make([]string, len(candidates))
|
||||
docVecs := make([][]float32, len(candidates))
|
||||
for i, c := range candidates {
|
||||
@@ -642,7 +832,7 @@ func (p *RecallPipeline) recallUncached(query, namespace string, topK int, diver
|
||||
continue
|
||||
}
|
||||
kw := ComputeBM25ScoreV2(query, candidates[idx].Content)
|
||||
reranked[i].Score = reranked[i].Score*0.7 + kw*0.3
|
||||
reranked[i].Score = reranked[i].Score*fusionW + kw*(1-fusionW)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -719,6 +909,17 @@ func (p *RecallPipeline) recallUncached(query, namespace string, topK int, diver
|
||||
}
|
||||
|
||||
// Step 4: MMR diversity
|
||||
if tr != nil {
|
||||
tr.Stage("rerank_fusion", stRerank, len(reranked))
|
||||
}
|
||||
// deadline 边界:重排已超时 ⇒ 返回 partial(不继续 MMR/图谱扩展)。
|
||||
if ctx.Err() != nil {
|
||||
if tr != nil {
|
||||
tr.MarkPartial("deadline_exceeded")
|
||||
}
|
||||
return nil, RecallMeta{CandidatesScanned: poolSize, Truncated: truncated, Status: "partial", PartialReason: "deadline_exceeded"}, nil
|
||||
}
|
||||
stMMR := time.Now()
|
||||
finalIndices := MMRSelect(reranked, candidates, topK*2, diversity)
|
||||
|
||||
results := make([]models.RecallResult, 0, topK)
|
||||
@@ -748,7 +949,11 @@ func (p *RecallPipeline) recallUncached(query, namespace string, topK int, diver
|
||||
conf := coalesceConfidence(c.Confidence)
|
||||
// 2026-10-09 (P0 修正):λ 已在 Step 3.7 偏置进 reranked[i].Score,选择与展示
|
||||
// 共用同一套分数;此处不再重复施加(否则 legacy 被双重扣分)。
|
||||
base := score*0.7 + c.QualityScore*0.3
|
||||
base := score*fusionW + c.QualityScore*(1-fusionW)
|
||||
if fuseModeProof() {
|
||||
// 第二槽换成确定性证据分:条数饱和 x/(x+K),无 seen/时间/V。
|
||||
base = score*fusionW + QualityFromProof(&c)*(1-fusionW)
|
||||
}
|
||||
// 2026-10-11 (t_631ab13d):把库里的 quality_score / v_value 直通给调用方。
|
||||
// 展示 DB 原值("响应值==库值"判据):优先用 RawQualityScore(LanceDB 原始列,
|
||||
// 仅 Rust IPC 路径携带,json:"-" 不落库);否则退回 DB 捕获值 dbQuality[idx]
|
||||
@@ -783,11 +988,24 @@ func (p *RecallPipeline) recallUncached(query, namespace string, topK int, diver
|
||||
QualityScore: qsVal,
|
||||
QualityScoreReliable: models.IsReliableQuality(qsVal),
|
||||
VValue: vv,
|
||||
// 2026-10-13 (t_e9a4d18f P0):过期标记(默认关闭 ⇒ 恒 false 不输出)
|
||||
// A10 (t_d618dc7c):人工有用单独计数直通(nil=NULL=从未被人工点过)
|
||||
HumanUsefulCount: c.HumanUsefulCount,
|
||||
// 2026-10-13 (t_e9a4d18f P0):过期标记(默认 720h,仅展示不参与排序)
|
||||
Stale: isStale(c.UpdatedAt, staleAfterHours),
|
||||
// 2026-10-26(t_trace_p0 / P1 supersede):来源通道 + 版本/取代关系直通
|
||||
Channel: "vector",
|
||||
Version: c.Version,
|
||||
ValidFrom: c.ValidFrom,
|
||||
ValidTo: c.ValidTo,
|
||||
Supersedes: c.Supersedes,
|
||||
SupersededBy: c.SupersededBy,
|
||||
Superseded: len(c.SupersededBy) > 0,
|
||||
})
|
||||
}
|
||||
}
|
||||
if tr != nil {
|
||||
tr.Stage("mmr_select", stMMR, len(results))
|
||||
}
|
||||
|
||||
// Step 3.8 (t_2afae95e FTS 词面通道尾部保证位, 门控同 Step 2.2):
|
||||
// 词面命中在 **rerank+MMR 之后**才并入最终结果(诊断第 3 条:rerank 之前
|
||||
@@ -842,6 +1060,14 @@ func (p *RecallPipeline) recallUncached(query, namespace string, topK int, diver
|
||||
nAppend = len(fresh)
|
||||
}
|
||||
if len(results) >= topK && nAppend > 0 {
|
||||
// t_bd718662 P1:让位数不得超过现有条数。原实现在 len(results)==topK 时
|
||||
// 硬编码 budget=3,topK∈{1,2} 且 fresh 够多时 cut=len(results)-nAppend<0
|
||||
// ⇒ results[:cut] 负索引 panic(线上 trace: `slice bounds out of range [:-2]`
|
||||
// @ recall.go:846)。修后 cut≥0 恒成立;全仓唯一裁剪点仍在此(AC-D:
|
||||
// 任何路径不返回 > topK 条 —— 让位只做等量替换,不做追加扩张)。
|
||||
if nAppend > len(results) {
|
||||
nAppend = len(results)
|
||||
}
|
||||
cut := len(results) - nAppend
|
||||
results = results[:cut] // 让出尾部 nAppend 位(头部至多动 3 位)
|
||||
}
|
||||
@@ -862,12 +1088,24 @@ func (p *RecallPipeline) recallUncached(query, namespace string, topK int, diver
|
||||
QualityScore: rec.QualityScore,
|
||||
QualityScoreReliable: false, // FTS 记录未经过质量融合 ⇒ 标不可信(审计可辨)
|
||||
FtsBoosted: true,
|
||||
Channel: "fts",
|
||||
Version: rec.Version,
|
||||
ValidFrom: rec.ValidFrom,
|
||||
ValidTo: rec.ValidTo,
|
||||
Supersedes: rec.Supersedes,
|
||||
SupersededBy: rec.SupersededBy,
|
||||
Superseded: len(rec.SupersededBy) > 0,
|
||||
Stale: isStale(rec.UpdatedAt, staleAfterHours),
|
||||
// A10 (t_d618dc7c):人工有用单独计数直通(nil=NULL=从未被人工点过)
|
||||
HumanUsefulCount: rec.HumanUsefulCount,
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Step 5: 图谱融合扩展(不再作为后备补丁,每次 recall 都跑 1 跳)
|
||||
stGraph := time.Now()
|
||||
graphAdded := 0
|
||||
if p.graph != nil {
|
||||
expanded := p.graph.ExpandFromResults(results, namespace, 1)
|
||||
for _, e := range expanded {
|
||||
@@ -879,12 +1117,18 @@ func (p *RecallPipeline) recallUncached(query, namespace string, topK int, diver
|
||||
}
|
||||
}
|
||||
if !already {
|
||||
e.Channel = "graph"
|
||||
results = append(results, e)
|
||||
graphAdded++
|
||||
}
|
||||
}
|
||||
}
|
||||
if tr != nil {
|
||||
tr.Stage("graph_expand", stGraph, graphAdded)
|
||||
}
|
||||
|
||||
// Step 6: 去重(相同 content 只保留最高分)
|
||||
stDedup := time.Now()
|
||||
seenContent := make(map[string]bool)
|
||||
deduped := make([]models.RecallResult, 0, len(results))
|
||||
for _, r := range results {
|
||||
@@ -894,6 +1138,9 @@ func (p *RecallPipeline) recallUncached(query, namespace string, topK int, diver
|
||||
}
|
||||
}
|
||||
results = deduped
|
||||
if tr != nil {
|
||||
tr.Stage("dedupe", stDedup, len(results))
|
||||
}
|
||||
|
||||
// Step 6.2: t_d32e5f8a L2 模式召回注入(env 门控 ZHIYI_PATTERN_RECALL,默认 0)
|
||||
// 命中 pattern 时以 Category="pattern" 附带 top-k 返回(设计「命中即注入」)。
|
||||
@@ -909,6 +1156,7 @@ func (p *RecallPipeline) recallUncached(query, namespace string, topK int, diver
|
||||
}
|
||||
}
|
||||
if !dup {
|
||||
pr.Channel = "pattern"
|
||||
results = append(results, pr)
|
||||
}
|
||||
}
|
||||
@@ -926,6 +1174,7 @@ func (p *RecallPipeline) recallUncached(query, namespace string, topK int, diver
|
||||
}
|
||||
}
|
||||
if !dup {
|
||||
pr.Channel = "peer"
|
||||
results = append(results, pr)
|
||||
}
|
||||
}
|
||||
@@ -984,7 +1233,32 @@ func (p *RecallPipeline) recallUncached(query, namespace string, topK int, diver
|
||||
}
|
||||
}()
|
||||
|
||||
return results, RecallMeta{CandidatesScanned: poolSize}, nil
|
||||
// Step 8: 热门度观测点(t_23b6a940 AC7,report-only、零排序影响):
|
||||
// 记录最终 rank1 的 recall_verified_count;连续越 p95 阈值满 N 个 query ⇒
|
||||
// ALERT 日志行(传感器通道),传感器端 zhiyi-popwatch.py 转既有告警通道。
|
||||
// 仅观测主 ANN 池出来的 rank1(图谱扩展/FTS 尾部位注入的 rank1 无池内计数,
|
||||
// 跳过不记)——热度偏置的作用面恰在主池 rerank 漏斗。
|
||||
if len(results) > 0 {
|
||||
for i := range candidates {
|
||||
if candidates[i].ID == results[0].ID {
|
||||
PopWatchRecord(results[0].ID, int64(candidates[i].RecallVerifiedCount))
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 契约(t_contract_p0):status 默认 complete;行数上限截断 ⇒ partial(
|
||||
// 调用方不得把截断后的空/少结果解读为「不存在」)。
|
||||
status := "complete"
|
||||
partialReason := ""
|
||||
if truncated {
|
||||
status = "partial"
|
||||
partialReason = "row_cap_truncated"
|
||||
if tr != nil {
|
||||
tr.MarkPartial("row_cap_truncated")
|
||||
}
|
||||
}
|
||||
return results, RecallMeta{CandidatesScanned: poolSize, Truncated: truncated, Status: status, PartialReason: partialReason}, nil
|
||||
}
|
||||
|
||||
// recallV2 开关(2026-10-14)。默认关 ⇒ 行为零变化。
|
||||
@@ -1187,7 +1461,7 @@ func cosineSimilarity(a, b []float32) float64 {
|
||||
func (p *RecallPipeline) recordRecallWrites(results []models.RecallResult) {
|
||||
ids := make([]string, 0, len(results))
|
||||
for _, r := range results {
|
||||
if r.ID != "" {
|
||||
if isMemoryRowID(r.ID) {
|
||||
ids = append(ids, r.ID)
|
||||
}
|
||||
}
|
||||
@@ -1201,12 +1475,26 @@ func (p *RecallPipeline) recordRecallWrites(results []models.RecallResult) {
|
||||
go p.incrementRecallCount(results)
|
||||
}
|
||||
|
||||
// isMemoryRowID 只认真实落在 memories 表里的行 ID(mem_*)。
|
||||
//
|
||||
// 2026-10-26(t_0a54c12b 写入放大复发):召回结果集里混着大量**不属于 memories 表**的 ID:
|
||||
// - ep_* = episodes 原始轮次(另一张表)
|
||||
// - n_* = 图谱扩展节点(graph_expanded;实测 memories 表 40383 行里 n_* 前缀 = 0)
|
||||
//
|
||||
// 这些 ID 送进 update_recall_batch 只会产生「谓词命中 0 行」的提交 —— 而 LanceDB 是 MVCC,
|
||||
// **每次提交无论命中几行都是一个版本**。实测 2026-10-26 21:26 一条批:
|
||||
// `delta=1 ids=791 rows=10`(命中率 1.3%)、`delta=2 ids=32 rows=0`(纯空转版本)。
|
||||
// 过滤后 ① 每条 batch 命中率回到 ~100% ② pending 涨得慢 ⇒ 阈值 flush 次数降一个量级。
|
||||
func isMemoryRowID(id string) bool {
|
||||
return strings.HasPrefix(id, "mem_")
|
||||
}
|
||||
|
||||
// incrementRecallCount 旧版逐条同步写(每条结果 = 1 个 LanceDB 版本)。
|
||||
// 已不作为主路径,仅保留为缓冲不可用时的兜底。
|
||||
func (p *RecallPipeline) incrementRecallCount(results []models.RecallResult) {
|
||||
now := time.Now().Format(time.RFC3339)
|
||||
for _, r := range results {
|
||||
if r.ID != "" {
|
||||
if isMemoryRowID(r.ID) {
|
||||
// t_268b0353(治根):兜底路径同样不写 freshness=verified("被召回"≠"被验证")
|
||||
// GA-P0-2 (t_c6d41ac6):召回命中 → recall_seen_count(recall_count 冻结)
|
||||
_ = p.lancedb.Update("memories", r.ID, map[string]any{
|
||||
|
||||
@@ -0,0 +1,189 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"math"
|
||||
"testing"
|
||||
|
||||
"github.com/xiaoxue/memoryweave/internal/models"
|
||||
)
|
||||
|
||||
// fbBiasLance 复用 nsRecordingLance 的 Search/Update 桩(行由外部注入),
|
||||
// 额外实现 IncrementHumanNotUseful —— 让测试走**与生产路由同一条 storage 接口方法**
|
||||
// (feedback.go MarkNotUseful → LanceDB.IncrementHumanNotUseful),而不是手工改字段。
|
||||
type fbBiasLance struct {
|
||||
nsRecordingLance
|
||||
}
|
||||
|
||||
// IncrementHumanNotUseful 对注入行做 +1(镜像 lancedb_ipc.go 的 $inc 语义:
|
||||
// 生产是原子 UPDATE ... SET human_not_useful_count = human_not_useful_count + 1)。
|
||||
func (f *fbBiasLance) IncrementHumanNotUseful(id string) error {
|
||||
for i := range f.rows {
|
||||
if f.rows[i].ID == id {
|
||||
f.rows[i].HumanNotUsefulCount++
|
||||
return nil
|
||||
}
|
||||
}
|
||||
return fmt.Errorf("fbBiasLance: row not found: %s", id)
|
||||
}
|
||||
|
||||
func fbVec(v0 float32) []float32 {
|
||||
v := make([]float32, 1024)
|
||||
v[0] = v0
|
||||
return v
|
||||
}
|
||||
|
||||
// newFbBiasPipeline 与 newSrcTierPipeline 同构:隔离进程级搜索缓存 + 预置 query 向量缓存,
|
||||
// 注入可自定义行/向量的 LanceDB 桩。
|
||||
func newFbBiasPipeline(t *testing.T, rows []models.MemoryRecord, queryVec []float32) *RecallPipeline {
|
||||
t.Helper()
|
||||
saved := SearchCacheInstance
|
||||
SearchCacheInstance = nil
|
||||
t.Cleanup(func() { SearchCacheInstance = saved })
|
||||
|
||||
emb := &Embedder{endpoint: "http://127.0.0.1:1/never", dim: 1024}
|
||||
emb.cache.Store("fbBiasQuery", queryVec)
|
||||
fake := &fbBiasLance{nsRecordingLance{rows: rows}}
|
||||
p := NewRecallPipeline(emb, fake, &Reranker{})
|
||||
_ = fake
|
||||
return p
|
||||
}
|
||||
|
||||
// fbRun 跑一次召回并返回 [id → (位次, 分数)];位次 0-based。
|
||||
func fbRun(t *testing.T, rows []models.MemoryRecord) map[string][2]float64 {
|
||||
t.Helper()
|
||||
p := newFbBiasPipeline(t, rows, fbVec(1.0))
|
||||
res, _, err := p.RecallWithMeta("fbBiasQuery", "hermes-main", 5, 0.0)
|
||||
if err != nil {
|
||||
t.Fatalf("RecallWithMeta 出错: %v", err)
|
||||
}
|
||||
out := make(map[string][2]float64, len(res))
|
||||
for i, r := range res {
|
||||
out[r.ID] = [2]float64{float64(i), r.Score}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// TestFeedbackBiasZeroDiffWhenOff — t_7ba5e479 AC2(门关判据):
|
||||
// 生产现状 ZHIYI_FEEDBACK_BIAS=0(zhiyid.service 实测)⇒ recall.go:714 的偏置循环不执行 ⇒
|
||||
// 标记 not_useful 前后召回结果**逐位相同**(id 序列 + 分数全同,delta=0)。
|
||||
// 这正是 AC2「若没下降」的机械解释:信号写进库了,但排序主链在 env 门处断开。
|
||||
func TestFeedbackBiasZeroDiffWhenOff(t *testing.T) {
|
||||
oldBias := feedbackBias
|
||||
feedbackBias = 0 // 生产现状(zhiyid.service: ZHIYI_FEEDBACK_BIAS=0)
|
||||
defer func() { feedbackBias = oldBias }()
|
||||
|
||||
rows := []models.MemoryRecord{
|
||||
{ID: "m0", Content: "记忆甲 完全无关内容一", Category: "distilled", Source: "llm_distill", Vector: fbVec(1.0)},
|
||||
{ID: "m1", Content: "记忆乙 完全无关内容二", Category: "distilled", Source: "llm_distill", Vector: fbVec(0.9)},
|
||||
{ID: "m2", Content: "记忆丙 完全无关内容三", Category: "distilled", Source: "llm_distill", Vector: fbVec(0.8)},
|
||||
{ID: "m3", Content: "记忆丁 完全无关内容四", Category: "distilled", Source: "llm_distill", Vector: fbVec(0.7)},
|
||||
{ID: "m4", Content: "记忆戊 完全无关内容五", Category: "distilled", Source: "llm_distill", Vector: fbVec(0.6)},
|
||||
}
|
||||
|
||||
before := fbRun(t, rows) // m2 未标记(count=0)
|
||||
|
||||
// 走生产同一条 storage 方法标记 m2(等价路由 POST /feedback/not-useful 的写库动作)
|
||||
mlc := &fbBiasLance{nsRecordingLance{rows: rows}}
|
||||
if err := mlc.IncrementHumanNotUseful("m2"); err != nil {
|
||||
t.Fatalf("IncrementHumanNotUseful err: %v", err)
|
||||
}
|
||||
if rows[2].HumanNotUsefulCount != 1 {
|
||||
t.Fatalf("human_not_useful_count 应从 0→1,got %d", rows[2].HumanNotUsefulCount)
|
||||
}
|
||||
after := fbRun(t, rows) // 同查询重跑
|
||||
|
||||
// 判据:逐位相同(门关 ⇒ 零变化)
|
||||
for id, b := range before {
|
||||
a, ok := after[id]
|
||||
if !ok {
|
||||
t.Fatalf("门关时结果集合变了:m2 标记后 %s 消失", id)
|
||||
}
|
||||
if a[0] != b[0] || a[1] != b[1] {
|
||||
t.Fatalf("ZHIYI_FEEDBACK_BIAS=0 时标记前后位次/分数必须逐位相同:%s before=(pos %.0f score %.4f) after=(pos %.0f score %.4f)",
|
||||
id, b[0], b[1], a[0], a[1])
|
||||
}
|
||||
}
|
||||
t.Logf("bias=0(生产现状): 5 条结果标记前后逐位相同 ✓ m2 pos=%.0f score=%.4f(delta=0)",
|
||||
before["m2"][0], before["m2"][1])
|
||||
}
|
||||
|
||||
// TestFeedbackBiasLowersNotUseful — t_7ba5e479 AC2(门开判据):
|
||||
// ZHIYI_FEEDBACK_BIAS=1.0 ⇒ 偏置循环生效,FeedbackScore(m2) = sat(0) - sat(1) = -1/4 = -0.25
|
||||
// (K=ZHIYI_FEEDBACK_K 默认 3)⇒ m2 分数下降 0.25、位次下滑;
|
||||
// 未标记行 FeedbackScore=0 ⇒ 偏置恒 0,分数/相对位次不变(不误伤)。
|
||||
func TestFeedbackBiasLowersNotUseful(t *testing.T) {
|
||||
oldBias := feedbackBias
|
||||
feedbackBias = 1.0 // 门开(等价 env ZHIYI_FEEDBACK_BIAS=1.0)
|
||||
defer func() { feedbackBias = oldBias }()
|
||||
|
||||
rows := []models.MemoryRecord{
|
||||
{ID: "m0", Content: "记忆甲 完全无关内容一", Category: "distilled", Source: "llm_distill", Vector: fbVec(1.0)},
|
||||
{ID: "m1", Content: "记忆乙 完全无关内容二", Category: "distilled", Source: "llm_distill", Vector: fbVec(0.9)},
|
||||
{ID: "m2", Content: "记忆丙 完全无关内容三", Category: "distilled", Source: "llm_distill", Vector: fbVec(0.8)},
|
||||
{ID: "m3", Content: "记忆丁 完全无关内容四", Category: "distilled", Source: "llm_distill", Vector: fbVec(0.7)},
|
||||
{ID: "m4", Content: "记忆戊 完全无关内容五", Category: "distilled", Source: "llm_distill", Vector: fbVec(0.6)},
|
||||
}
|
||||
|
||||
before := fbRun(t, rows) // m2 count=0(标记前)
|
||||
mlc := &fbBiasLance{nsRecordingLance{rows: rows}}
|
||||
if err := mlc.IncrementHumanNotUseful("m2"); err != nil {
|
||||
t.Fatalf("IncrementHumanNotUseful err: %v", err)
|
||||
}
|
||||
after := fbRun(t, rows) // m2 count=1(标记后),同查询重跑
|
||||
|
||||
posBefore, scoreBefore := before["m2"][0], before["m2"][1]
|
||||
posAfter, scoreAfter := after["m2"][0], after["m2"][1]
|
||||
if !(posAfter > posBefore) {
|
||||
t.Fatalf("门开后 m2 位次未下滑:before pos=%.0f / after pos=%.0f(AC2 要求下降)", posBefore, posAfter)
|
||||
}
|
||||
if !(scoreAfter < scoreBefore) {
|
||||
t.Fatalf("门开后 m2 分数未下降:before=%.4f / after=%.4f", scoreBefore, scoreAfter)
|
||||
}
|
||||
// 预期降幅 = -feedbackBias * sat(1) = -1.0 * 1/(1+3) = -0.25(K 默认 3);
|
||||
// 若 rerank 重打分路径导致非精确值,退化为「严格下降 + 其它行逐位不动」双断言。
|
||||
want := -feedbackBias * (1.0 / (1.0 + float64(3)))
|
||||
delta := scoreAfter - scoreBefore
|
||||
if math.Abs(delta-want) > 1e-9 {
|
||||
t.Logf("note: 分数 delta=%.4f(rerank 重打分,非精确 %.4f),其它行未动则判据仍成立", delta, want)
|
||||
}
|
||||
// 不误伤:其它 4 条分数不变
|
||||
for _, id := range []string{"m0", "m1", "m3", "m4"} {
|
||||
if after[id][1] != before[id][1] {
|
||||
t.Fatalf("未标记行 %s 分数被误伤:before=%.4f after=%.4f", id, before[id][1], after[id][1])
|
||||
}
|
||||
}
|
||||
t.Logf("bias=1.0: m2 pos %.0f→%.0f, score %.4f→%.4f (delta=%.4f, 预期=%.4f) ✓ 未标记 4 行分数逐位不变 ✓",
|
||||
posBefore, posAfter, scoreBefore, scoreAfter, delta, want)
|
||||
}
|
||||
|
||||
// TestFeedbackBiasLiftsUseful — 正向对照(同门控的另一侧):
|
||||
// UsefulCount=1 ⇒ FeedbackScore = +sat(1) = +0.25 ⇒ 门开时 m2 分数上升、位次上移。
|
||||
func TestFeedbackBiasLiftsUseful(t *testing.T) {
|
||||
oldBias := feedbackBias
|
||||
feedbackBias = 1.0
|
||||
defer func() { feedbackBias = oldBias }()
|
||||
|
||||
rows := []models.MemoryRecord{
|
||||
{ID: "m0", Content: "记忆甲 完全无关内容一", Category: "distilled", Source: "llm_distill", Vector: fbVec(1.0)},
|
||||
{ID: "m1", Content: "记忆乙 完全无关内容二", Category: "distilled", Source: "llm_distill", Vector: fbVec(0.9)},
|
||||
{ID: "m2", Content: "记忆丙 完全无关内容三", Category: "distilled", Source: "llm_distill", Vector: fbVec(0.8)},
|
||||
{ID: "m3", Content: "记忆丁 完全无关内容四", Category: "distilled", Source: "llm_distill", Vector: fbVec(0.7)},
|
||||
{ID: "m4", Content: "记忆戊 完全无关内容五", Category: "distilled", Source: "llm_distill", Vector: fbVec(0.6)},
|
||||
}
|
||||
|
||||
before := fbRun(t, rows)
|
||||
// 正向标记(同一 storage 接口的 MarkUseful 写法):+1 useful_count
|
||||
for i := range rows {
|
||||
if rows[i].ID == "m2" {
|
||||
rows[i].UsefulCount++
|
||||
}
|
||||
}
|
||||
after := fbRun(t, rows)
|
||||
if !(after["m2"][0] < before["m2"][0] || after["m2"][1] > before["m2"][1]) {
|
||||
t.Fatalf("门开后 useful 标记应使 m2 上移/加分:before=(pos %.0f, score %.4f) after=(pos %.0f, score %.4f)",
|
||||
before["m2"][0], before["m2"][1], after["m2"][0], after["m2"][1])
|
||||
}
|
||||
t.Logf("bias=1.0 正向对照: m2 pos %.0f→%.0f, score %.4f→%.4f ✓",
|
||||
before["m2"][0], after["m2"][0], before["m2"][1], after["m2"][1])
|
||||
}
|
||||
@@ -0,0 +1,67 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"math"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/xiaoxue/memoryweave/internal/models"
|
||||
)
|
||||
|
||||
// 2026-10-26 Step3:证明 ZHIYI_FUSE_MODE 的解析与证据分语义。
|
||||
// 目的不是「跑通」,而是锁死两条:
|
||||
// 1) 默认(未设/其它值)必须 false —— 零行为变化是上线前提;
|
||||
// 2) QualityFromProof 必须只依赖证据条数(sat=x/(x+K)),与 recency/seen/V 无关。
|
||||
func TestFuseModeProofParsing(t *testing.T) {
|
||||
cases := []struct {
|
||||
env string
|
||||
want bool
|
||||
}{
|
||||
{"", false},
|
||||
{"0", false},
|
||||
{"off", false},
|
||||
{"proof", true},
|
||||
{"NOUSE", true}, // 大小写不敏感
|
||||
{"Proof", true},
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Setenv("ZHIYI_FUSE_MODE", c.env)
|
||||
if got := fuseModeProof(); got != c.want {
|
||||
t.Fatalf("ZHIYI_FUSE_MODE=%q → %v, want %v", c.env, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestQualityFromProofOnlyDependsOnEvidence(t *testing.T) {
|
||||
K := proofSaturation // 默认 5
|
||||
|
||||
// 无证据 ⇒ 0
|
||||
if got := QualityFromProof(&models.MemoryRecord{}); got != 0 {
|
||||
t.Fatalf("无证据应为 0,得 %v", got)
|
||||
}
|
||||
|
||||
// 5 条证据 ⇒ 5/(5+5) = 0.5
|
||||
m := &models.MemoryRecord{SourceEvents: []string{"e1", "e2", "e3", "e4", "e5"}}
|
||||
if got := QualityFromProof(m); math.Abs(got-5/(5+K)) > 1e-9 {
|
||||
t.Fatalf("5 条证据应为 %.4f,得 %.4f", 5/(5+K), got)
|
||||
}
|
||||
|
||||
// 去重:重复事件不虚增
|
||||
dup := &models.MemoryRecord{SourceEvents: []string{"e1", "e1", "e1"}}
|
||||
if got := QualityFromProof(dup); math.Abs(got-1/(1+K)) > 1e-9 {
|
||||
t.Fatalf("重复证据应去重为 1 条(%.4f),得 %.4f", 1/(1+K), got)
|
||||
}
|
||||
|
||||
// ★ 关键:同样的证据条数,不同 seen/时间 ⇒ 分数必须完全相同
|
||||
// (若这里失败,说明证据分被 recency/seen 污染,Step3 的前提就不成立)
|
||||
base := &models.MemoryRecord{SourceEvents: []string{"a", "b"}}
|
||||
polluted := &models.MemoryRecord{
|
||||
SourceEvents: []string{"a", "b"},
|
||||
RecallSeenCount: 999, // 高频召回
|
||||
LastRecalledAt: time.Now().AddDate(-3, 0, 0), // 三年前的最后一次召回
|
||||
}
|
||||
if QualityFromProof(base) != QualityFromProof(polluted) {
|
||||
t.Fatalf("证据分被 recency/seen 污染:%v vs %v",
|
||||
QualityFromProof(base), QualityFromProof(polluted))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,141 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"testing"
|
||||
|
||||
"github.com/xiaoxue/memoryweave/internal/models"
|
||||
)
|
||||
|
||||
// 2026-10-26(牧尘令)· ns 作用域兜底护栏的判据测试。
|
||||
//
|
||||
// 背景:请求带 agent_id 时 resolveRecallNamespace 派生 <agent>-main,而实测该 ns 里
|
||||
// 只有 6~7 条货(default-main 6 / xiaoyi-main 6 / prof-cloud-main 7),不传 agent_id
|
||||
// 则有 200 条 ⇒ 带 agent_id 的调用静默劣化到个位数、且不报错。
|
||||
//
|
||||
// 本组用例锁死四条不变量:
|
||||
// 1) agent 派生作用域 + 候选过少 ⇒ 必须补一次全库搜索并合并(池变大)。
|
||||
// 2) **显式 namespace 调用方不兜底**(allowNSFallback=false)⇒ 不得多发全库搜索,
|
||||
// 否则跨域泄漏。
|
||||
// 3) ZHIYI_NS_FALLBACK_MIN=0 ⇒ 关闭(回滚开关有效)。
|
||||
// 4) 候选本来就够多 ⇒ 不兜底(避免每次多打一次 200 行的全库搜索)。
|
||||
|
||||
// nsFallbackLance 按 namespaceFilter 返回不同规模的候选:
|
||||
// 全库("") → 100 条;hermes-main → 2 条;shared → 1 条。
|
||||
type nsFallbackLance struct {
|
||||
LanceDB
|
||||
nsFilters []string
|
||||
fullRowsN int
|
||||
nsRowsN int
|
||||
}
|
||||
|
||||
func (f *nsFallbackLance) Search(table string, vector []float32, topK int, namespaceFilter string) ([]models.MemoryRecord, error) {
|
||||
f.nsFilters = append(f.nsFilters, namespaceFilter)
|
||||
n := f.nsRowsN
|
||||
if namespaceFilter == "" {
|
||||
n = f.fullRowsN
|
||||
}
|
||||
out := make([]models.MemoryRecord, 0, n)
|
||||
for i := 0; i < n; i++ {
|
||||
ns := namespaceFilter
|
||||
if ns == "" {
|
||||
ns = "full"
|
||||
}
|
||||
out = append(out, models.MemoryRecord{
|
||||
ID: fmt.Sprintf("%s-%03d", ns, i),
|
||||
Content: fmt.Sprintf("%s 记忆 %d", ns, i),
|
||||
Category: "distilled",
|
||||
Namespace: ns,
|
||||
})
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// Update 覆写:后台 goroutine 会回写 recall_count,落到 nil 接口会 panic(见 recall_namespace_test.go 注释)。
|
||||
func (f *nsFallbackLance) Update(table, id string, fields map[string]any) error { return nil }
|
||||
|
||||
// newNSFallbackPipeline 与 newNSTestPipeline 同构,但接受任意 LanceDB 实现
|
||||
// (newNSTestPipeline 的形参是具体的 *nsRecordingLance)。
|
||||
func newNSFallbackPipeline(t *testing.T, fake LanceDB) *RecallPipeline {
|
||||
t.Helper()
|
||||
saved := SearchCacheInstance
|
||||
SearchCacheInstance = nil
|
||||
t.Cleanup(func() { SearchCacheInstance = saved })
|
||||
|
||||
emb := &Embedder{endpoint: "http://127.0.0.1:1/never", dim: 1024}
|
||||
vec := make([]float32, 1024)
|
||||
for i := range vec {
|
||||
vec[i] = 0.01
|
||||
}
|
||||
emb.cache.Store(nsTestQuery, vec) // 预置缓存 → 不走网络
|
||||
return NewRecallPipeline(emb, fake, &Reranker{})
|
||||
}
|
||||
|
||||
func (f *nsFallbackLance) countFullSearches() int {
|
||||
n := 0
|
||||
for _, ns := range f.nsFilters {
|
||||
if ns == "" {
|
||||
n++
|
||||
}
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
func TestNSFallbackMergesFullLibraryWhenScopeTooSmall(t *testing.T) {
|
||||
fake := &nsFallbackLance{fullRowsN: 100, nsRowsN: 2}
|
||||
p := newNSFallbackPipeline(t, fake)
|
||||
|
||||
_, meta, err := p.RecallWithMetaOpts(nsTestQuery, "hermes-main", 10, 0.3, true)
|
||||
if err != nil {
|
||||
t.Fatalf("Recall 出错: %v", err)
|
||||
}
|
||||
if fake.countFullSearches() != 1 {
|
||||
t.Fatalf("✗ 应补 1 次全库搜索,实际 %d 次(调用序列 %v)", fake.countFullSearches(), fake.nsFilters)
|
||||
}
|
||||
if meta.CandidatesScanned < 50 {
|
||||
t.Errorf("✗ 兜底后候选池仍只有 %d 条,应合并到接近全库 100 条", meta.CandidatesScanned)
|
||||
}
|
||||
t.Logf("✓ 兜底生效:作用域 2 条 → 合并后池 %d 条(搜索序列 %v)", meta.CandidatesScanned, fake.nsFilters)
|
||||
}
|
||||
|
||||
func TestNSFallbackDisabledForExplicitNamespace(t *testing.T) {
|
||||
fake := &nsFallbackLance{fullRowsN: 100, nsRowsN: 2}
|
||||
p := newNSFallbackPipeline(t, fake)
|
||||
|
||||
if _, _, err := p.RecallWithMetaOpts(nsTestQuery, "hermes-main", 10, 0.3, false); err != nil {
|
||||
t.Fatalf("Recall 出错: %v", err)
|
||||
}
|
||||
if n := fake.countFullSearches(); n != 0 {
|
||||
t.Fatalf("✗ 显式 namespace 不得兜底,实际发起了 %d 次全库搜索(序列 %v)", n, fake.nsFilters)
|
||||
}
|
||||
t.Logf("✓ 显式 ns 不兜底(防跨域泄漏):搜索序列 %v", fake.nsFilters)
|
||||
}
|
||||
|
||||
func TestNSFallbackCanBeTurnedOffByEnv(t *testing.T) {
|
||||
fake := &nsFallbackLance{fullRowsN: 100, nsRowsN: 2}
|
||||
p := newNSFallbackPipeline(t, fake)
|
||||
t.Setenv("ZHIYI_NS_FALLBACK_MIN", "0") // 回滚开关
|
||||
|
||||
if _, _, err := p.RecallWithMetaOpts(nsTestQuery, "hermes-main", 10, 0.3, true); err != nil {
|
||||
t.Fatalf("Recall 出错: %v", err)
|
||||
}
|
||||
if n := fake.countFullSearches(); n != 0 {
|
||||
t.Fatalf("✗ ZHIYI_NS_FALLBACK_MIN=0 应完全关闭兜底,实际 %d 次", n)
|
||||
}
|
||||
t.Logf("✓ 回滚开关有效:ZHIYI_NS_FALLBACK_MIN=0 ⇒ 零全库搜索")
|
||||
}
|
||||
|
||||
func TestNSFallbackSkippedWhenScopeAlreadyLarge(t *testing.T) {
|
||||
// nsRowsN=30 ≥ 默认阈值 20 ⇒ 不该兜底(否则每次多打一次全库搜索)
|
||||
fake := &nsFallbackLance{fullRowsN: 100, nsRowsN: 30}
|
||||
p := newNSFallbackPipeline(t, fake)
|
||||
|
||||
_, meta, err := p.RecallWithMetaOpts(nsTestQuery, "hermes-main", 10, 0.3, true)
|
||||
if err != nil {
|
||||
t.Fatalf("Recall 出错: %v", err)
|
||||
}
|
||||
if n := fake.countFullSearches(); n != 0 {
|
||||
t.Fatalf("✗ 候选已足够(%d 条)不该兜底,实际 %d 次全库搜索", meta.CandidatesScanned, n)
|
||||
}
|
||||
t.Logf("✓ 候选充足不兜底:池 %d 条,零全库搜索", meta.CandidatesScanned)
|
||||
}
|
||||
@@ -0,0 +1,225 @@
|
||||
package storage
|
||||
|
||||
// t_bd718662【织忆·P1】/recall 线上 panic:recallUncached 让位截断出负索引。
|
||||
// 根因:results 已满(len==topK)时 budget 硬编码 3,nAppend=min(3,len(fresh));
|
||||
// topK∈{1,2} 且 fresh 足够多时 cut = len(results)-nAppend < 0 ⇒ results[:cut] 负索引 panic
|
||||
// (生产 trace:`runtime error: slice bounds out of range [:-2]` @ recall.go:846)。
|
||||
//
|
||||
// 本文件三层验证(对应卡 AC + 附录 D 增补):
|
||||
// 1. TestFTS_SmallTopKTailMatrix —— 表驱动:topK∈{1,2,3} × len(results)∈{0,1,2,topK}
|
||||
// × len(fresh)∈{0,1,2,3,5},断言:不 panic + len ≤ topK + 无重复 ID + 尾部 FTS 集合精确。
|
||||
// 修前:topK=1/F≥2 与 topK=2/F≥3 的满池格 panic(slice bounds out of range);
|
||||
// 修后:全部格通过。
|
||||
// 2. TestFTS_SmallTopKGolden —— topK∈{3,4,5} 的逐字输出(ID+Score+QualityScore+FtsBoosted),
|
||||
// 修前/修后各跑一次 diff,证明正常路径零回归(AC2 逐字一致)。
|
||||
// 3. 不变量(AC4/AC-D):任何路径 len(结果) ≤ topK —— 让位只做替换(截断+追加等量),
|
||||
// 全仓唯一裁剪点 recall.go:846,修后仍只此一处理截断。
|
||||
//
|
||||
// 全部纯内存 fake(向量池 fake + FTS Ldb fake),不触网、不碰 LanceDB/生产数据。
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"testing"
|
||||
|
||||
"github.com/xiaoxue/memoryweave/internal/models"
|
||||
)
|
||||
|
||||
// matrixQuery 固定 5 个造词 ASCII 稀有词 + 1 个 CJK bigram(库里无 ⇒ 不命中)。
|
||||
// F 个 FTS 文档各含 Zetaflux1..F 之一 ⇒ FtsSearch 命中恰 F 条,且全在 ASCII 档(≥0.55)。
|
||||
const matrixQuery = "Zetaflux1 Zetaflux2 Zetaflux3 Zetaflux4 Zetaflux5 去重"
|
||||
|
||||
// poolOnly 向量池行:P 条纯内容文档(不含任何 Zetaflux 词)。
|
||||
// 关键一:每条**不同向量**(第 i 维 =1,其余 0)⇒ 两两 cosine=0,MMR 不做去重合并。
|
||||
// 关键二:每条**不同内容** ⇒ 躲开 Step 6「相同 content 只保留最高分」去重
|
||||
// (同内容池内文档会被塌成 1 条,P≥topK 满池场景造不出来)。
|
||||
var poolThemes = []string{"图谱", "嵌入", "蒸馏", "遗忘", "整合"}
|
||||
|
||||
func poolOnly(P int) []models.MemoryRecord {
|
||||
rows := make([]models.MemoryRecord, 0, P)
|
||||
for i := 1; i <= P; i++ {
|
||||
vec := make([]float32, 8)
|
||||
vec[i-1] = 1.0 // 第 i 条独占第 i 维
|
||||
rows = append(rows, models.MemoryRecord{
|
||||
ID: fmt.Sprintf("pdoc%d", i), Namespace: "hermes-main", Category: "distilled",
|
||||
Content: fmt.Sprintf("池内文档第%d篇讲织忆%s主题。", i, poolThemes[i-1]),
|
||||
Vector: vec,
|
||||
})
|
||||
}
|
||||
return rows
|
||||
}
|
||||
|
||||
// ftsDocs FTS 侧行:F 条各含不同 Zetaflux 词的 shared 区文档(池外 ⇒ 只能靠词面通道进结果)。
|
||||
func ftsDocs(F int) []models.MemoryRecord {
|
||||
vec := make([]float32, 8)
|
||||
for i := range vec {
|
||||
vec[i] = 0.5
|
||||
}
|
||||
rows := make([]models.MemoryRecord, 0, F)
|
||||
for i := 1; i <= F; i++ {
|
||||
rows = append(rows, models.MemoryRecord{
|
||||
ID: fmt.Sprintf("zdoc%d", i), Namespace: "shared", Category: "distilled",
|
||||
Content: fmt.Sprintf("共享区稀有词 Zetaflux%d 出现处。", i), Vector: vec,
|
||||
})
|
||||
}
|
||||
return rows
|
||||
}
|
||||
|
||||
// smallKPool:向量搜索恒返回固定池行(忽略请求 topK,模拟「池内 P 条、池外 F 条」),
|
||||
// FTS 池由 FtsProvider.FtsLdb 注入(池行 + FTS 行全量,索引层自己剔类别/软删)。
|
||||
type smallKPool struct {
|
||||
LanceDB // 未覆写的方法(AppendHitEvent 等)走接口缺省 —— 同 ftsSearchOnly 范式
|
||||
poolRows []models.MemoryRecord
|
||||
dim int
|
||||
}
|
||||
|
||||
func (f *smallKPool) Search(table string, vector []float32, topK int, namespaceFilter string) ([]models.MemoryRecord, error) {
|
||||
return f.poolRows, nil
|
||||
}
|
||||
func (f *smallKPool) Update(table, id string, fields map[string]any) error { return nil }
|
||||
|
||||
// runMatrixCell 单格:P 条池内 + F 条池外,topK 召回。
|
||||
func runMatrixCell(t *testing.T, topK, P, F int) []models.RecallResult {
|
||||
t.Helper()
|
||||
|
||||
savedCache := SearchCacheInstance
|
||||
SearchCacheInstance = nil
|
||||
t.Cleanup(func() { SearchCacheInstance = savedCache })
|
||||
|
||||
savedFts := FtsProvider.FtsLdb
|
||||
all := make([]models.MemoryRecord, 0, P+F)
|
||||
all = append(all, poolOnly(P)...)
|
||||
all = append(all, ftsDocs(F)...)
|
||||
FtsProvider.FtsLdb = func() ([]models.MemoryRecord, error) { return all, nil }
|
||||
t.Cleanup(func() { FtsProvider.FtsLdb = savedFts })
|
||||
|
||||
resetFTSIndex()
|
||||
|
||||
vec := make([]float32, 8)
|
||||
for i := range vec {
|
||||
vec[i] = 0.5
|
||||
}
|
||||
emb := &Embedder{endpoint: "http://127.0.0.1:1/never", dim: 8}
|
||||
emb.cache.Store(matrixQuery, vec)
|
||||
p := NewRecallPipeline(emb, &smallKPool{poolRows: poolOnly(P), dim: 8}, &Reranker{})
|
||||
|
||||
res, err := p.Recall(matrixQuery, "hermes-main", topK, 0.5)
|
||||
if err != nil {
|
||||
t.Fatalf("topK=%d P=%d F=%d recall 报错: %v", topK, P, F, err)
|
||||
}
|
||||
return res
|
||||
}
|
||||
|
||||
// 期望值推导(修后语义 = 让位只做等量替换):
|
||||
//
|
||||
// P < topK:head = P(满 P 条池内),nAppend = min(topK-P, F),total = P + nAppend ≤ topK
|
||||
// P ≥ topK:head = topK,nAppend = min(3, F, topK),截掉 nAppend 条再补 nAppend 条,total = topK
|
||||
// 尾部 FTS 集合 = zdoc1..zdoc_nAppend(各命中词 idf 相等 ⇒ 同分按 ID 升序)
|
||||
func expectedYield(topK, P, F int) (total, nAppend int) {
|
||||
if P < topK {
|
||||
nAppend = topK - P
|
||||
if nAppend > F {
|
||||
nAppend = F
|
||||
}
|
||||
total = P + nAppend
|
||||
} else {
|
||||
nAppend = 3
|
||||
if nAppend > F {
|
||||
nAppend = F
|
||||
}
|
||||
if nAppend > topK { // t_bd718662 修后:让位上限 = min(3,F,现有条数)
|
||||
nAppend = topK
|
||||
}
|
||||
total = topK
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
// 表驱动(附录 D):topK∈{1,2,3} × len(results)∈{0,1,2,topK} × len(fresh)∈{0,1,2,3,5}。
|
||||
// P(池内条数)决定 len(results)=min(P,topK):topK=1/2 时 P∈{0,1,2}、topK=3 时 P∈{0,1,2,3}。
|
||||
func TestFTS_SmallTopKTailMatrix(t *testing.T) {
|
||||
t.Setenv("ZHIYI_RECALL_FTS", "1")
|
||||
|
||||
type cell struct {
|
||||
topK, P, F int
|
||||
}
|
||||
cells := []cell{}
|
||||
for _, tk := range []int{1, 2, 3} {
|
||||
ps := []int{0, 1, 2}
|
||||
if tk >= 3 {
|
||||
ps = append(ps, 3)
|
||||
}
|
||||
for _, P := range ps {
|
||||
for _, F := range []int{0, 1, 2, 3, 5} {
|
||||
cells = append(cells, cell{tk, P, F})
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, c := range cells {
|
||||
c := c
|
||||
t.Run(fmt.Sprintf("topK%d_P%d_F%d", c.topK, c.P, c.F), func(t *testing.T) {
|
||||
res := runMatrixCell(t, c.topK, c.P, c.F)
|
||||
|
||||
// AC1/AC4/AC-D:不超 topK
|
||||
if len(res) > c.topK {
|
||||
t.Fatalf("topK=%d P=%d F=%d 返回 %d 条 > topK(让位逻辑做了追加扩张): %v",
|
||||
c.topK, c.P, c.F, len(res), idsRes(res))
|
||||
}
|
||||
// 无重复 ID
|
||||
seen := map[string]int{}
|
||||
for _, r := range res {
|
||||
seen[r.ID]++
|
||||
}
|
||||
for id, n := range seen {
|
||||
if n > 1 {
|
||||
t.Fatalf("topK=%d P=%d F=%d id %s 出现 %d 次 ⇒ 去重被破坏", c.topK, c.P, c.F, id, n)
|
||||
}
|
||||
}
|
||||
// 精确集合断言:头部 = 池内 pdoc 前 min(P,topK) 条(同向量下 MMR 稳定序),
|
||||
// 尾部 = zdoc1..zdoc_nAppend(FTS 同分按 ID 升序)
|
||||
wantTotal, wantN := expectedYield(c.topK, c.P, c.F)
|
||||
if len(res) != wantTotal {
|
||||
t.Fatalf("topK=%d P=%d F=%d 长度 %d ≠ 期望 %d(结果=%v)",
|
||||
c.topK, c.P, c.F, len(res), wantTotal, idsRes(res))
|
||||
}
|
||||
var gotFts []string
|
||||
for _, r := range res {
|
||||
if r.FtsBoosted {
|
||||
gotFts = append(gotFts, r.ID)
|
||||
}
|
||||
}
|
||||
if len(gotFts) != wantN {
|
||||
t.Fatalf("topK=%d P=%d F=%d FTS 条数 %d ≠ 期望 %d(结果=%v)",
|
||||
c.topK, c.P, c.F, len(gotFts), wantN, idsRes(res))
|
||||
}
|
||||
for i := 0; i < wantN; i++ {
|
||||
if gotFts[i] != fmt.Sprintf("zdoc%d", i+1) {
|
||||
t.Fatalf("topK=%d P=%d F=%d 尾部第 %d 位 = %s ≠ 期望 zdoc%d(FTS 分降序+ID 升序被破坏: %v)",
|
||||
c.topK, c.P, c.F, i+1, gotFts[i], i+1, gotFts)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// golden:topK∈{3,4,5}(正常路径)+ F=0(无 FTS 命中)四格,逐字输出。
|
||||
// 修前修后各跑一次:go test -run TestFTS_SmallTopKGolden ./internal/storage/ -v 2>&1 | grep GOLDEN
|
||||
// 两份输出 diff 必须为空(AC2 零回归判据)。
|
||||
func TestFTS_SmallTopKGolden(t *testing.T) {
|
||||
t.Setenv("ZHIYI_RECALL_FTS", "1")
|
||||
for _, tc := range []struct {
|
||||
topK, P, F int
|
||||
}{
|
||||
{3, 5, 3}, {4, 5, 3}, {5, 5, 3}, {5, 5, 0},
|
||||
} {
|
||||
res := runMatrixCell(t, tc.topK, tc.P, tc.F)
|
||||
line := fmt.Sprintf("GOLDEN topK=%d P=%d F=%d: [", tc.topK, tc.P, tc.F)
|
||||
for i, r := range res {
|
||||
if i > 0 {
|
||||
line += " "
|
||||
}
|
||||
line += fmt.Sprintf("{%s score=%.6f qs=%.6f fts=%v}", r.ID, r.Score, r.QualityScore, r.FtsBoosted)
|
||||
}
|
||||
line += "]"
|
||||
t.Logf("%s", line)
|
||||
}
|
||||
}
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
"os"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
@@ -44,7 +45,19 @@ type BatchRecallUpdater interface {
|
||||
|
||||
const (
|
||||
defaultRecallFlushInterval = 300 * time.Second
|
||||
defaultRecallFlushMaxItems = 256
|
||||
// 2026-10-26 (t_0a54c12b):256 → 2048。
|
||||
// 256 在搜索突发期每 1~2 秒就触发一次 flush(每次 flush = 2~4 个 delta 组 = 2~4 个
|
||||
// LanceDB 版本),实测 20~200 版本/分钟。待写条目是「不同记忆 ID」,2048 只占几百 KB 内存,
|
||||
// 把窗口内重复命中合并得更彻底、flush 次数降一个量级。
|
||||
defaultRecallFlushMaxItems = 2048
|
||||
// requeue 硬上限:IPC 长时间不可用时 pending 不能无限涨(召回计数是观测值,丢了可接受)。
|
||||
recallRequeueHardCap = 16384
|
||||
// 优雅停止窗口(t_fa797edd):Stop 内的「等后台协程退出 + 最终落盘」都必须有界。
|
||||
// 否则 Rust/LanceDB 批量更新在活 IPC 流量下阻塞时,Stop 永远不返回 → 信号协程
|
||||
// 到不了 srv.Shutdown → 进程跑满 systemd 90s TimeoutStopSec 被 SIGKILL
|
||||
// (丢写风险 + 每次重启都变强杀 + 40s IPC 中断)。
|
||||
defaultRecallStopLoopWait = 5 * time.Second // 等后台 flush 协程退出的上限
|
||||
defaultRecallStopFlushWait = 8 * time.Second // 最终落盘的上限(召回统计,丢失可接受)
|
||||
)
|
||||
|
||||
type pendingRecall struct {
|
||||
@@ -71,6 +84,8 @@ type RecallWriteBuffer struct {
|
||||
flushRows int64
|
||||
flushErrors int64
|
||||
fallbackRows int64
|
||||
// requeuedItems: IPC 通道不可用时整批退回缓冲的条目数(t_0a54c12b 观测项)
|
||||
requeuedItems int64
|
||||
}
|
||||
|
||||
// RecallWriteBufferInstance 进程级单例(与 SearchCacheInstance / CoOccurTrackerInstance 同模式)
|
||||
@@ -199,30 +214,27 @@ func (b *RecallWriteBuffer) Flush() (int, error) {
|
||||
usedFallback := false
|
||||
if b.batch != nil {
|
||||
rows, err = b.batch.UpdateRecallBatch("memories", batch, ts)
|
||||
}
|
||||
if b.batch == nil || err != nil {
|
||||
usedFallback = true
|
||||
if err != nil {
|
||||
// 2026-10-26 (t_0a54c12b):**通道不可用**与**语句被拒**必须分开处置。
|
||||
// 通道不可用(sidecar 重启/拒绝连接/超时)时逐条降级毫无收益:
|
||||
// ① IPC 不可用 ⇒ 每条都失败,2026-10-26 21:16 实测 **9142 次失败拨号**(日志风暴);
|
||||
// ② sidecar 恢复瞬间逐条成功 ⇒ **每条 = 1 个 LanceDB 版本**,正是 _versions 冲到 8.5k 的机制。
|
||||
// 正确做法:整批退回 pending,下个窗口连同新命中一起批量重试(召回计数延迟可接受)。
|
||||
if isRecallTransportErr(err) {
|
||||
atomic.AddInt64(&b.flushErrors, 1)
|
||||
atomic.AddInt64(&b.requeuedItems, int64(len(batch)))
|
||||
n := b.requeue(batch, lastAt)
|
||||
log.Printf("[recall-buffer] IPC 通道不可用(%v) → 整批 %d 条退回缓冲(pending=%d),下个窗口重试;不逐条降级(避免拨号风暴 + N 个版本)", err, len(batch), n)
|
||||
return 0, err
|
||||
}
|
||||
log.Printf("[recall-buffer] 单事务批量提交失败(%v) → 退化逐条 Update(数据不丢,版本数不优化)", err)
|
||||
atomic.AddInt64(&b.flushErrors, 1)
|
||||
usedFallback = true
|
||||
rows = b.fallbackRowUpdates(batch, ts)
|
||||
}
|
||||
rows = 0
|
||||
for _, it := range batch {
|
||||
// t_268b0353(2026-10-11 治根):召回只更新 recall_count / last_recalled_at,
|
||||
// 不再写 freshness="verified"。freshness=verified 的语义是「经过校验/用户
|
||||
// 反馈」;"被召回" ≠ "被验证"。此前每次召回都盖 verified → 越常被召回的
|
||||
// 旧快照越"可信",时效判断全被污染(过期快照被当现状的机制根因之一)。
|
||||
// GA-P0-2 (t_c6d41ac6):召回命中写 recall_seen_count;不再写 recall_count(冻结)。
|
||||
if uerr := b.ldb.Update("memories", it.ID, map[string]any{
|
||||
"recall_seen_count": map[string]string{"$inc": "1"},
|
||||
"last_recalled_at": ts,
|
||||
}); uerr != nil {
|
||||
log.Printf("[recall-buffer] 逐条 Update 失败 id=%s: %v", it.ID, uerr)
|
||||
continue
|
||||
}
|
||||
rows++
|
||||
}
|
||||
atomic.AddInt64(&b.fallbackRows, rows)
|
||||
} else {
|
||||
usedFallback = true
|
||||
rows = b.fallbackRowUpdates(batch, ts)
|
||||
}
|
||||
|
||||
// 落盘成功后同步进程内缓存(缓存只是热数据,权威值在 LanceDB)
|
||||
@@ -248,7 +260,78 @@ func (b *RecallWriteBuffer) Flush() (int, error) {
|
||||
return int(rows), nil
|
||||
}
|
||||
|
||||
// fallbackRowUpdates 逐条 Update(仅用于「批量语句被拒」或后端不支持批量的事务语义兜底)。
|
||||
// 每条 = 1 个 LanceDB 版本,故**不得**在 IPC 通道不可用时走这里(见 Flush 里的通道判定)。
|
||||
func (b *RecallWriteBuffer) fallbackRowUpdates(batch []RecallWriteItem, ts string) int64 {
|
||||
var rows int64
|
||||
for _, it := range batch {
|
||||
// t_268b0353(2026-10-11 治根):召回只更新 recall_count / last_recalled_at,
|
||||
// 不再写 freshness="verified"。freshness=verified 的语义是「经过校验/用户
|
||||
// 反馈」;"被召回" ≠ "被验证"。此前每次召回都盖 verified → 越常被召回的
|
||||
// 旧快照越"可信",时效判断全被污染(过期快照被当现状的机制根因之一)。
|
||||
// GA-P0-2 (t_c6d41ac6):召回命中写 recall_seen_count;不再写 recall_count(冻结)。
|
||||
if uerr := b.ldb.Update("memories", it.ID, map[string]any{
|
||||
"recall_seen_count": map[string]string{"$inc": "1"},
|
||||
"last_recalled_at": ts,
|
||||
}); uerr != nil {
|
||||
log.Printf("[recall-buffer] 逐条 Update 失败 id=%s: %v", it.ID, uerr)
|
||||
continue
|
||||
}
|
||||
rows++
|
||||
}
|
||||
atomic.AddInt64(&b.fallbackRows, rows)
|
||||
return rows
|
||||
}
|
||||
|
||||
// requeue 把一批未落盘的增量退回 pending(IPC 通道不可用时用):delta 累加、时间取更晚者。
|
||||
// 硬上限 recallRequeueHardCap:IPC 长期不可用时不能无限涨(召回计数是观测值,丢了可接受)。
|
||||
func (b *RecallWriteBuffer) requeue(batch []RecallWriteItem, lastAt time.Time) int {
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
for _, it := range batch {
|
||||
if len(b.pending) >= recallRequeueHardCap {
|
||||
if _, ok := b.pending[it.ID]; !ok {
|
||||
continue // 满了且是新 ID:丢弃(观测值,可接受)
|
||||
}
|
||||
}
|
||||
p, ok := b.pending[it.ID]
|
||||
if !ok {
|
||||
p = &pendingRecall{}
|
||||
b.pending[it.ID] = p
|
||||
}
|
||||
p.delta += it.Delta
|
||||
if lastAt.After(p.lastAt) {
|
||||
p.lastAt = lastAt
|
||||
}
|
||||
}
|
||||
return len(b.pending)
|
||||
}
|
||||
|
||||
// isRecallTransportErr 判断是否为「IPC 通道不可用」类错误(可重试、且绝不能逐条降级)。
|
||||
// 判据是**通道层**错误关键字,不是业务/谓词错误 —— 后者逐条降级有意义(换语句能成),
|
||||
// 前者逐条只会制造拨号风暴 + 恢复后 N 个版本。
|
||||
func isRecallTransportErr(err error) bool {
|
||||
if err == nil {
|
||||
return false
|
||||
}
|
||||
msg := strings.ToLower(err.Error())
|
||||
for _, k := range []string{
|
||||
"ipc dial", "connection refused", "connection reset", "broken pipe",
|
||||
"i/o timeout", "context deadline exceeded", "unexpected eof",
|
||||
"no such file or directory", // unix socket 文件不存在(sidecar 正在重启)
|
||||
} {
|
||||
if strings.Contains(msg, k) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// Stop 停止后台 flush 协程,并把最后一个窗口落盘(进程优雅退出时调用)。
|
||||
// 全程有界(t_fa797edd):等后台协程 ≤ defaultRecallStopLoopWait,最终落盘
|
||||
// ≤ defaultRecallStopFlushWait。Rust/LanceDB 批量更新在活 IPC 流量下可能阻塞
|
||||
// 数十秒;召回统计丢了可接受(源码注释「进程崩溃会丢最后一个窗口」),
|
||||
// 因此超时即放弃落盘、放行退出,保证进程总在有限秒内到 srv.Shutdown。
|
||||
func (b *RecallWriteBuffer) Stop() {
|
||||
b.mu.Lock()
|
||||
if b.stopped {
|
||||
@@ -261,10 +344,23 @@ func (b *RecallWriteBuffer) Stop() {
|
||||
close(b.stopCh)
|
||||
select {
|
||||
case <-b.doneCh:
|
||||
case <-time.After(10 * time.Second):
|
||||
log.Printf("[recall-buffer] 停止超时,直接落盘剩余窗口")
|
||||
case <-time.After(defaultRecallStopLoopWait):
|
||||
log.Printf("[recall-buffer] 后台 flush 协程停止超时(%s),直接进入最终落盘", defaultRecallStopLoopWait)
|
||||
}
|
||||
|
||||
// 最终落盘有界:放到 goroutine 跑,到点不再等。若后台 ticker/阈值 flush
|
||||
// 正持锁阻塞在 Rust IPC 上,本 goroutine 的 Flush 会先等 b.mu —— 同样被
|
||||
// defaultRecallStopFlushWait 兜住,超时即放行进程退出。
|
||||
flushDone := make(chan struct{})
|
||||
go func() {
|
||||
b.Flush()
|
||||
close(flushDone)
|
||||
}()
|
||||
select {
|
||||
case <-flushDone:
|
||||
case <-time.After(defaultRecallStopFlushWait):
|
||||
log.Printf("[recall-buffer] 最终落盘超时(%s),放弃剩余窗口(仅丢召回统计,进程照常退出)", defaultRecallStopFlushWait)
|
||||
}
|
||||
b.Flush()
|
||||
}
|
||||
|
||||
// Stats 观测用(flush 频率 / 合并率 / 版本写入行数)。
|
||||
@@ -273,14 +369,15 @@ func (b *RecallWriteBuffer) Stats() map[string]interface{} {
|
||||
pendingN := len(b.pending)
|
||||
b.mu.Unlock()
|
||||
return map[string]interface{}{
|
||||
"pending": pendingN,
|
||||
"flush_count": atomic.LoadInt64(&b.flushCount),
|
||||
"flush_items": atomic.LoadInt64(&b.flushItems),
|
||||
"flush_rows": atomic.LoadInt64(&b.flushRows),
|
||||
"flush_errors": atomic.LoadInt64(&b.flushErrors),
|
||||
"fallback_rows": atomic.LoadInt64(&b.fallbackRows),
|
||||
"window": b.interval.String(),
|
||||
"max_items": b.maxItems,
|
||||
"single_tx": b.batch != nil,
|
||||
"pending": pendingN,
|
||||
"flush_count": atomic.LoadInt64(&b.flushCount),
|
||||
"flush_items": atomic.LoadInt64(&b.flushItems),
|
||||
"flush_rows": atomic.LoadInt64(&b.flushRows),
|
||||
"flush_errors": atomic.LoadInt64(&b.flushErrors),
|
||||
"fallback_rows": atomic.LoadInt64(&b.fallbackRows),
|
||||
"requeued_items": atomic.LoadInt64(&b.requeuedItems),
|
||||
"window": b.interval.String(),
|
||||
"max_items": b.maxItems,
|
||||
"single_tx": b.batch != nil,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -18,6 +18,8 @@ type fakeLDB struct {
|
||||
updateCalls int
|
||||
updateIDs []string
|
||||
forceBatchErr bool
|
||||
// batchErr:指定批量提交返回的错误(用于验证「通道不可用」与「语句被拒」分流)。
|
||||
batchErr error
|
||||
}
|
||||
|
||||
// Scan mock:write-buffer 测试只需占位实现(被测路径不实际调用)。
|
||||
@@ -72,6 +74,9 @@ func (f *fakeLDB) PurgeInvalid() (int64, error) { return 0, nil }
|
||||
|
||||
// BatchRecallUpdater 实现(单事务批量)
|
||||
func (f *fakeLDB) UpdateRecallBatch(table string, items []RecallWriteItem, lastRecalledAt string) (int64, error) {
|
||||
if f.batchErr != nil {
|
||||
return 0, f.batchErr
|
||||
}
|
||||
if f.forceBatchErr {
|
||||
return 0, fmt.Errorf("forced batch failure")
|
||||
}
|
||||
@@ -227,6 +232,89 @@ func TestRecallWriteBufferFallbackToPerItem(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// t_0a54c12b:IPC 通道不可用(sidecar 重启/拒绝连接)时必须**整批退回缓冲**,
|
||||
// 不许逐条降级 —— 逐条在通道断时是 N 次失败拨号,恢复瞬间又变成 N 个 LanceDB 版本
|
||||
// (2026-10-26 实测 9142 次逐条 Update、_versions 冲 8.5k)。
|
||||
func TestRecallWriteBufferTransportErrRequeues(t *testing.T) {
|
||||
fake := &fakeLDB{batchErr: fmt.Errorf("ipc dial: dial unix /tmp/zhiyi-ipc.sock: connect: connection refused")}
|
||||
b := NewRecallWriteBuffer(fake, 0, 1000)
|
||||
defer b.Stop()
|
||||
|
||||
b.Record([]string{"mem_r1", "mem_r2"})
|
||||
if _, err := b.Flush(); err == nil {
|
||||
t.Fatal("通道不可用时 Flush 应返回 err 供上层观测")
|
||||
}
|
||||
fake.mu.Lock()
|
||||
if fake.updateCalls != 0 {
|
||||
t.Fatalf("通道不可用时禁止逐条降级,实际 updateCalls=%d", fake.updateCalls)
|
||||
}
|
||||
fake.mu.Unlock()
|
||||
if st := b.Stats(); st["pending"].(int) != 2 {
|
||||
t.Fatalf("整批应退回缓冲 pending=2,实际 %v", st["pending"])
|
||||
}
|
||||
|
||||
// 通道恢复 → 下个窗口连同新命中一起批量落盘,delta 不丢
|
||||
fake.mu.Lock()
|
||||
fake.batchErr = nil
|
||||
fake.mu.Unlock()
|
||||
b.Record([]string{"mem_r1"}) // 窗口内再命中一次 → r1 delta=2
|
||||
rows, err := b.Flush()
|
||||
if err != nil {
|
||||
t.Fatalf("恢复后 flush 应成功: %v", err)
|
||||
}
|
||||
if rows != 2 {
|
||||
t.Fatalf("恢复后 rows = %d, want 2", rows)
|
||||
}
|
||||
fake.mu.Lock()
|
||||
defer fake.mu.Unlock()
|
||||
deltas := map[string]int{}
|
||||
for _, it := range fake.batchItems {
|
||||
deltas[it.ID] = it.Delta
|
||||
}
|
||||
if deltas["mem_r1"] != 2 || deltas["mem_r2"] != 1 {
|
||||
t.Fatalf("退回缓冲后 delta 应累加不丢,实际 %v", deltas)
|
||||
}
|
||||
if st := b.Stats(); st["requeued_items"].(int64) != 2 {
|
||||
t.Fatalf("requeued_items = %v, want 2", st["requeued_items"])
|
||||
}
|
||||
}
|
||||
|
||||
// t_0a54c12b:只有 mem_* 会进缓冲。ep_*(episodes 表)与 n_*(图谱扩展节点,
|
||||
// memories 表实测 0 行)送进 update_recall_batch 只会产生「命中 0 行」的空转版本。
|
||||
func TestRecallWriteBufferSkipsNonMemoryIDs(t *testing.T) {
|
||||
// 过滤逻辑本体(recordRecallWrites 用的就是它)
|
||||
got := memoryRowIDs([]models.RecallResult{
|
||||
{ID: "mem_keep1"}, {ID: "mem_keep2"},
|
||||
{ID: "ep_1793020608863378731"}, {ID: "n_静默失效"}, {ID: "n_api_key"}, {ID: ""},
|
||||
})
|
||||
if len(got) != 2 || got[0] != "mem_keep1" || got[1] != "mem_keep2" {
|
||||
t.Fatalf("只应保留 mem_* 两条,实际 %v", got)
|
||||
}
|
||||
if isMemoryRowID("ep_x") || isMemoryRowID("n_y") || isMemoryRowID("") || !isMemoryRowID("mem_z") {
|
||||
t.Fatal("isMemoryRowID 判据错误(只应认 mem_ 前缀)")
|
||||
}
|
||||
|
||||
// 写进缓冲的也只有 mem_*(其余 ID 不进 pending ⇒ 不会触发空转提交)
|
||||
fake := &fakeLDB{}
|
||||
b := NewRecallWriteBuffer(fake, 0, 1000)
|
||||
defer b.Stop()
|
||||
b.Record(got)
|
||||
if st := b.Stats(); st["pending"].(int) != 2 {
|
||||
t.Fatalf("pending = %v, want 2", st["pending"])
|
||||
}
|
||||
}
|
||||
|
||||
// memoryRowIDs 供测试复用生产过滤逻辑。
|
||||
func memoryRowIDs(rs []models.RecallResult) []string {
|
||||
out := make([]string, 0, len(rs))
|
||||
for _, r := range rs {
|
||||
if isMemoryRowID(r.ID) {
|
||||
out = append(out, r.ID)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Stop 时落盘最后一个窗口(优雅退出不丢统计)
|
||||
func TestRecallWriteBufferStopFlushes(t *testing.T) {
|
||||
fake := &fakeLDB{}
|
||||
@@ -240,3 +328,31 @@ func TestRecallWriteBufferStopFlushes(t *testing.T) {
|
||||
t.Fatalf("Stop 应落盘最后一个窗口,batchCalls=%d", fake.batchCalls)
|
||||
}
|
||||
}
|
||||
|
||||
// hangingLDB:UpdateRecallBatch 永不返回(模拟 Rust/LanceDB 批量更新在活 IPC
|
||||
// 流量下被卡死,t_fa797edd 生产实测 80s+)。用于验证 Stop 有界:不会因 flush
|
||||
// 挂起而无限阻塞进程退出。
|
||||
type hangingLDB struct{ fakeLDB }
|
||||
|
||||
func (h *hangingLDB) UpdateRecallBatch(table string, items []RecallWriteItem, lastRecalledAt string) (int64, error) {
|
||||
select {} // 永不返回
|
||||
}
|
||||
|
||||
// t_fa797edd 反向用例:最终落盘挂起时,Stop 必须在有界时间内返回(不许无限阻塞)。
|
||||
// 原实现 b.Flush() 同步调用 → Rust 卡死则 Stop 永不返回 → 进程到不了 srv.Shutdown
|
||||
// → 跑满 systemd 90s 被 SIGKILL。修复后 Stop 在 defaultRecallStopFlushWait 内放行。
|
||||
func TestRecallWriteBufferStopBoundedWhenFlushHangs(t *testing.T) {
|
||||
h := &hangingLDB{}
|
||||
b := NewRecallWriteBuffer(h, 0, 1000)
|
||||
b.Record([]string{"mem_hang"})
|
||||
|
||||
start := time.Now()
|
||||
b.Stop()
|
||||
elapsed := time.Since(start)
|
||||
|
||||
// 有界上限:后台协程等待(5s) + 最终落盘等待(8s) + 裕量。50s 远小于 systemd 90s 超时。
|
||||
if elapsed > 50*time.Second {
|
||||
t.Fatalf("Stop 应是有界的,实际阻塞了 %s(>50s,会触发 systemd 90s SIGKILL)", elapsed)
|
||||
}
|
||||
t.Logf("Stop 在 %s 内放行(flush 挂起时也不阻塞进程退出)", elapsed)
|
||||
}
|
||||
|
||||
@@ -4,6 +4,7 @@ package storage
|
||||
import (
|
||||
"encoding/binary"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"os"
|
||||
@@ -143,18 +144,19 @@ func TestSoftDelete_AlwaysConflict_ExhaustsBounded(t *testing.T) {
|
||||
if err == nil {
|
||||
t.Fatalf("永远冲突应返回错误,却返回 nil")
|
||||
}
|
||||
// AC-2:含尝试次数(重试数)的明确错误
|
||||
if !strings.Contains(err.Error(), "retry exhausted after 3 retries") {
|
||||
t.Fatalf("错误应含 'retry exhausted after 3 retries',实际=%v", err)
|
||||
// AC-2:含尝试次数(重试数)的明确错误(t_cf680fad:预算 3→5)
|
||||
if want := fmt.Sprintf("retry exhausted after %d retries", softDeleteMaxRetries); !strings.Contains(err.Error(), want) {
|
||||
t.Fatalf("错误应含 %q,实际=%v", want, err)
|
||||
}
|
||||
if !strings.Contains(err.Error(), "mem_retry_never_ok") {
|
||||
t.Fatalf("错误应含 id,实际=%v", err)
|
||||
}
|
||||
// 有界:总 rpc 尝试 = 1 首次 + 3 重试 = 4 次,不死循环
|
||||
// 有界:总 rpc 尝试 = 1 首次 + N 重试(t_cf680fad N=5)= 6 次,不死循环
|
||||
if got := fk.callCount(); got != softDeleteMaxRetries+1 {
|
||||
t.Fatalf("总 rpc 尝试 = %d, want %d(有界不卡死)", got, softDeleteMaxRetries+1)
|
||||
}
|
||||
// 退避:50+100+200 = 350ms 最坏,宽松下限断言 ≥300ms(证明睡了退避、非忙循环)
|
||||
// 退避:jitter 后总时长随随机量浮动,但仍 ≥ 无 jitter 下限(50+100+200+200+200 的整数倍关系)
|
||||
// 宽松下限断言 ≥300ms(证明睡了退避、非忙循环)
|
||||
if elapsed < 300*time.Millisecond {
|
||||
t.Fatalf("退避未生效:elapsed=%v, want ≥300ms", elapsed)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,176 @@
|
||||
package storage
|
||||
|
||||
// t_3aa2a8c1:`truncated` 在 IPC 层被 Go 客户端丢掉 —— Search 只打日志不上抛,
|
||||
// 本测试用假 sidecar(unix socket 实现真实 IPC 协议)验证 SearchWithTruncation 透传:
|
||||
// ① 假 sidecar 回 truncated=true ⇒ 标志上抛(不再静默)
|
||||
// ② 回 truncated=false / 旧 sidecar 无该字段 ⇒ false(向后兼容)
|
||||
// ③ 本地兜底(IPC 不可用 / 零结果)⇒ false(兜底路径没有行数上限)
|
||||
// ④ 旧签名 Search 行为回归(委托 SearchWithTruncation,records 不变)
|
||||
|
||||
import (
|
||||
"encoding/binary"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"net"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// startFakeTruncSidecar 起一个只回一帧的假 sidecar:读满请求(长度前缀 + body),
|
||||
// 按 truncated 值回 ConsolidateResponse 形状(report_json = 1 条空壳记录,非零结果,
|
||||
// 避免触发本地兜底分支)。
|
||||
func startFakeTruncSidecar(t *testing.T, truncated bool) (string, func()) {
|
||||
t.Helper()
|
||||
tmp, err := os.MkdirTemp(os.TempDir(), "zytrunc")
|
||||
if err != nil {
|
||||
t.Fatalf("mkdtemp: %v", err)
|
||||
}
|
||||
sock := filepath.Join(tmp, "tr.sock")
|
||||
l, err := net.Listen("unix", sock)
|
||||
if err != nil {
|
||||
t.Fatalf("listen: %v", err)
|
||||
}
|
||||
go func() {
|
||||
for {
|
||||
c, aerr := l.Accept()
|
||||
if aerr != nil {
|
||||
return
|
||||
}
|
||||
go func(conn net.Conn) {
|
||||
var lenBuf [4]byte
|
||||
if _, rerr := io.ReadFull(conn, lenBuf[:]); rerr != nil {
|
||||
return
|
||||
}
|
||||
n := binary.BigEndian.Uint32(lenBuf[:])
|
||||
buf := make([]byte, n)
|
||||
if _, rerr := io.ReadFull(conn, buf); rerr != nil {
|
||||
return
|
||||
}
|
||||
report := `[{"id":"mem_fake"}]`
|
||||
resp, _ := json.Marshal(map[string]interface{}{
|
||||
"status": "ok", "report_json": report, "truncated": truncated,
|
||||
})
|
||||
binary.BigEndian.PutUint32(lenBuf[:], uint32(len(resp)))
|
||||
if _, werr := conn.Write(lenBuf[:]); werr != nil {
|
||||
return
|
||||
}
|
||||
if _, werr := conn.Write(resp); werr != nil {
|
||||
return
|
||||
}
|
||||
}(c)
|
||||
}
|
||||
}()
|
||||
cleanup := func() {
|
||||
l.Close()
|
||||
os.RemoveAll(tmp)
|
||||
}
|
||||
return sock, cleanup
|
||||
}
|
||||
|
||||
// ① Rust 侧被行数上限截断 ⇒ Go 客户端必须上抛(t_3aa2a8c1 核心)
|
||||
func TestSearchWithTruncation_FlagPropagated(t *testing.T) {
|
||||
sock, cleanup := startFakeTruncSidecar(t, true)
|
||||
defer cleanup()
|
||||
rc := NewRustLanceDBClient(sock, nil)
|
||||
|
||||
recs, trunc, err := rc.SearchWithTruncation("memories", make([]float32, 1024), 3, "")
|
||||
if err != nil {
|
||||
t.Fatalf("rpc 失败: %v", err)
|
||||
}
|
||||
if !trunc {
|
||||
t.Fatalf("假 sidecar 已回 truncated=true,SearchWithTruncation 却未上抛(仍静默)")
|
||||
}
|
||||
if len(recs) != 1 || recs[0].ID != "mem_fake" {
|
||||
t.Fatalf("records 回归异常: %+v", recs)
|
||||
}
|
||||
}
|
||||
|
||||
// ② 未截断(新 sidecar 显式 false)与旧 sidecar(字段缺失 ⇒ 缺省 false)⇒ 恒 false
|
||||
func TestSearchWithTruncation_NotTruncated(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
truncated bool
|
||||
}{
|
||||
{"new sidecar false", false},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
sock, cleanup := startFakeTruncSidecar(t, tc.truncated)
|
||||
defer cleanup()
|
||||
rc := NewRustLanceDBClient(sock, nil)
|
||||
_, trunc, err := rc.SearchWithTruncation("memories", make([]float32, 1024), 3, "")
|
||||
if err != nil {
|
||||
t.Fatalf("rpc 失败: %v", err)
|
||||
}
|
||||
if trunc {
|
||||
t.Fatalf("未截断场景却上报 truncated=true(假阳性)")
|
||||
}
|
||||
})
|
||||
}
|
||||
// 旧 sidecar 形状:响应无 truncated 字段 ⇒ serde/JSON 缺省 false(向后兼容回归)
|
||||
tmp, _ := os.MkdirTemp(os.TempDir(), "zyold")
|
||||
defer os.RemoveAll(tmp)
|
||||
sock := filepath.Join(tmp, "old.sock")
|
||||
l, _ := net.Listen("unix", sock)
|
||||
go func() {
|
||||
for {
|
||||
c, aerr := l.Accept()
|
||||
if aerr != nil {
|
||||
return
|
||||
}
|
||||
go func(conn net.Conn) {
|
||||
var lenBuf [4]byte
|
||||
if _, rerr := io.ReadFull(conn, lenBuf[:]); rerr != nil {
|
||||
return
|
||||
}
|
||||
n := binary.BigEndian.Uint32(lenBuf[:])
|
||||
buf := make([]byte, n)
|
||||
if _, rerr := io.ReadFull(conn, buf); rerr != nil {
|
||||
return
|
||||
}
|
||||
// 无 truncated 字段(旧 sidecar 形状)
|
||||
resp, _ := json.Marshal(map[string]interface{}{
|
||||
"status": "ok", "report_json": `[{"id":"mem_old"}]`,
|
||||
})
|
||||
binary.BigEndian.PutUint32(lenBuf[:], uint32(len(resp)))
|
||||
_, _ = conn.Write(lenBuf[:])
|
||||
_, _ = conn.Write(resp)
|
||||
}(c)
|
||||
}
|
||||
}()
|
||||
defer l.Close()
|
||||
rc := NewRustLanceDBClient(sock, nil)
|
||||
_, trunc, err := rc.SearchWithTruncation("memories", make([]float32, 1024), 3, "")
|
||||
if err != nil {
|
||||
t.Fatalf("旧 sidecar 形状 rpc 失败: %v", err)
|
||||
}
|
||||
if trunc {
|
||||
t.Fatalf("旧 sidecar(无 truncated 字段)必须向后兼容为 false,实为 true")
|
||||
}
|
||||
}
|
||||
|
||||
// ③ 本地兜底(socket 不存在 ⇒ dial 失败 ⇒ localSearch)⇒ false,不报错(降级语义不变)
|
||||
func TestSearchWithTruncation_LocalFallbackFalse(t *testing.T) {
|
||||
rc := NewRustLanceDBClient("/tmp/zhiyi-trunc-does-not-exist.sock", nil)
|
||||
_, trunc, err := rc.SearchWithTruncation("memories", make([]float32, 1024), 3, "")
|
||||
if err != nil {
|
||||
t.Fatalf("本地兜底应降级成功(旧 Search 语义:返回 nil err),实际: %v", err)
|
||||
}
|
||||
if trunc {
|
||||
t.Fatalf("本地兜底路径没有行数上限,必须 false")
|
||||
}
|
||||
}
|
||||
|
||||
// ④ 回归:旧签名 Search 走委托,records 行为不变(拿到结果、nil 错误)
|
||||
func TestSearch_DelegatedRegression(t *testing.T) {
|
||||
sock, cleanup := startFakeTruncSidecar(t, true)
|
||||
defer cleanup()
|
||||
rc := NewRustLanceDBClient(sock, nil)
|
||||
recs, err := rc.Search("memories", make([]float32, 1024), 3, "")
|
||||
if err != nil {
|
||||
t.Fatalf("旧签名 Search 回归破坏: %v", err)
|
||||
}
|
||||
if len(recs) != 1 || recs[0].ID != "mem_fake" {
|
||||
t.Fatalf("旧签名 Search records 回归异常: %+v", recs)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,143 @@
|
||||
// 织忆 MemoryWeave — durable pending-softdelete 队列(t_cf680fad 机制修复)
|
||||
//
|
||||
// 背景 / 根因:
|
||||
// runMerges O(n²) 软删 + 并发遗忘碎片器在同一张 memories 单表上制造 MVCC 写风暴;
|
||||
// retrySoftDeleteRPC 旧预算 3 次/~350ms(且退避无 jitter,50+ 并发软删锁步重放)⇒
|
||||
// 风暴下每次重放都撞已推进的新版本 ⇒ 4 次全败 ⇒ 失败软删被 consolidation_pipe.go:252
|
||||
// 的 `_ =` 静默丢弃 ⇒ 源记录 is_deleted 停在 false,每次重启窗口都留新残留
|
||||
// (检索里重复命中)。实证:单次 op 的 3 次重放 version 598346→598347→598348→598350
|
||||
// 逐版推进仍耗尽;24h「重试成功=0」。
|
||||
//
|
||||
// 本文件提供的保证("重试必须能成功落盘"):
|
||||
// ① EnqueuePendingSoftDelete:retrySoftDeleteRPC MVCC 冲突耗尽时把 (id, reason) 追加进
|
||||
// durable JSONL(单写者 append + 原子 tmp/rename,同 id 去重)。
|
||||
// ② DrainPendingSoftDeletes:zhiyid 启动 + 每 30s 触发器 tick 各调用一次,逐条重放
|
||||
// SoftDelete(SET is_deleted=true 幂等、对最新已提交版本重读),成功后移除;
|
||||
// 队列空/缺失 = 立即 no-op。风暴退去后(或下次重启窗口)重放必落盘。
|
||||
package storage
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"encoding/json"
|
||||
"log"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// defaultPendingSoftDeletePath 生产路径。测试用 env ZHIYI_SOFTDELETE_PENDING 覆盖(不污染生产库)。
|
||||
const defaultPendingSoftDeletePath = "/var/lib/memoryweave/pending-softdelete.jsonl"
|
||||
|
||||
func pendingSoftDeletePath() string {
|
||||
if p := os.Getenv("ZHIYI_SOFTDELETE_PENDING"); p != "" {
|
||||
return p
|
||||
}
|
||||
return defaultPendingSoftDeletePath
|
||||
}
|
||||
|
||||
type pendingSoftDeleteEntry struct {
|
||||
ID string `json:"id"`
|
||||
Reason string `json:"reason"`
|
||||
TS string `json:"ts"`
|
||||
}
|
||||
|
||||
// pendingMu 单写者护栏:Go 侧 zhiyid 是唯一写者(Rust sidecar 不碰此文件)。
|
||||
var pendingMu sync.Mutex
|
||||
|
||||
func readPendingEntries(p string) []pendingSoftDeleteEntry {
|
||||
f, err := os.Open(p)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
defer f.Close()
|
||||
var out []pendingSoftDeleteEntry
|
||||
sc := bufio.NewScanner(f)
|
||||
sc.Buffer(make([]byte, 1<<20), 1<<20)
|
||||
for sc.Scan() {
|
||||
var e pendingSoftDeleteEntry
|
||||
if json.Unmarshal([]byte(sc.Text()), &e) == nil && e.ID != "" {
|
||||
out = append(out, e)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func rewritePendingFile(p string, entries []pendingSoftDeleteEntry) {
|
||||
tmp := p + ".tmp"
|
||||
f, err := os.OpenFile(tmp, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o644)
|
||||
if err != nil {
|
||||
log.Printf("[ipc] pending-softdelete rewrite open failed: %v", err)
|
||||
return
|
||||
}
|
||||
w := bufio.NewWriter(f)
|
||||
for _, e := range entries {
|
||||
b, _ := json.Marshal(e)
|
||||
_, _ = w.Write(b)
|
||||
_, _ = w.Write([]byte{'\n'})
|
||||
}
|
||||
if err := w.Flush(); err != nil {
|
||||
f.Close()
|
||||
log.Printf("[ipc] pending-softdelete rewrite flush failed: %v", err)
|
||||
return
|
||||
}
|
||||
f.Close()
|
||||
if err := os.Rename(tmp, p); err != nil {
|
||||
log.Printf("[ipc] pending-softdelete rewrite rename failed: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// EnqueuePendingSoftDelete 追加一条待软删(幂等去重:同 id 只保留一条)。
|
||||
// 由 retrySoftDeleteRPC 在 MVCC 冲突耗尽时调用;schema/空 id 等真错误不调(入队无意义)。
|
||||
func EnqueuePendingSoftDelete(id, reason string) {
|
||||
p := pendingSoftDeletePath()
|
||||
if p == "" {
|
||||
return
|
||||
}
|
||||
_ = os.MkdirAll(filepath.Dir(p), 0o755)
|
||||
pendingMu.Lock()
|
||||
defer pendingMu.Unlock()
|
||||
entries := readPendingEntries(p)
|
||||
for _, e := range entries {
|
||||
if e.ID == id {
|
||||
return // 已在队列,不重复(reason 保留最早一条)
|
||||
}
|
||||
}
|
||||
entries = append(entries, pendingSoftDeleteEntry{ID: id, Reason: reason, TS: time.Now().Format(time.RFC3339)})
|
||||
rewritePendingFile(p, entries)
|
||||
log.Printf("[ipc] pending-softdelete ENQUEUED: id=%s reason=%s queue=%d", id, reason, len(entries))
|
||||
}
|
||||
|
||||
// PendingSoftDeleteCount 返回队列当前条数(观测 / 测试用;缺失 = 0)。
|
||||
func PendingSoftDeleteCount() int {
|
||||
entries := readPendingEntries(pendingSoftDeletePath())
|
||||
return len(entries)
|
||||
}
|
||||
|
||||
// DrainPendingSoftDeletes 逐条重放待软删(幂等),成功后从队列移除。
|
||||
// 启动 + 每 tick 各调用一次;队列空/缺失 = 立即 no-op(廉价 stat+read)。
|
||||
// 参数取 LanceDB 接口(任意后端);内部调 rc.SoftDelete(RustLanceDBClient 走 IPC+重试+入队)。
|
||||
// 返回 (成功条数, 仍失败条数);仍失败者保留在队列,下一 tick/下次启动再重放。
|
||||
func DrainPendingSoftDeletes(ldb LanceDB) (ok, failed int) {
|
||||
p := pendingSoftDeletePath()
|
||||
entries := readPendingEntries(p)
|
||||
if len(entries) == 0 {
|
||||
return 0, 0
|
||||
}
|
||||
var remaining []pendingSoftDeleteEntry
|
||||
for _, e := range entries {
|
||||
if err := ldb.SoftDelete(e.ID, e.Reason); err != nil {
|
||||
// 仍失败(仍在风暴内 / 真错误):保留在队列,下一轮再试。
|
||||
// 注意:若这条 SoftDelete 内部又因 MVCC 耗尽触发 EnqueuePendingSoftDelete,
|
||||
// 同 id 去重保证不会重复入队;此处 remaining 收口最终一致。
|
||||
log.Printf("[ipc] pending-softdelete drain RETRY FAILED: id=%s reason=%s err=%v", e.ID, e.Reason, err)
|
||||
remaining = append(remaining, e)
|
||||
failed++
|
||||
} else {
|
||||
ok++
|
||||
}
|
||||
}
|
||||
rewritePendingFile(p, remaining)
|
||||
log.Printf("[ipc] pending-softdelete DRAIN: retried=%d ok=%d still_pending=%d", len(entries), ok, len(remaining))
|
||||
return ok, failed
|
||||
}
|
||||
@@ -0,0 +1,119 @@
|
||||
// 织忆 MemoryWeave — durable pending-softdelete 队列单测(t_cf680fad 机制修复 AC)
|
||||
//
|
||||
// 覆盖:
|
||||
// ① MVCC 冲突耗尽 ⇒ 入 durable 队列(不再静默丢,残留根治)。
|
||||
// ② 非可重试错误(schema/空 id)⇒ 不入队(入队无意义,避免污染)。
|
||||
// ③ 幂等去重:同 id 只保留一条。
|
||||
// ④ drain:风暴退去后(server 恒 OK)逐条重放 ⇒ 落盘 + 队列清空。
|
||||
package storage
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func withTempPending(t *testing.T, fn func(pendingPath string)) {
|
||||
t.Helper()
|
||||
dir, err := os.MkdirTemp("", "pendingsdtest")
|
||||
if err != nil {
|
||||
t.Fatalf("mkdirtemp: %v", err)
|
||||
}
|
||||
defer os.RemoveAll(dir)
|
||||
p := filepath.Join(dir, "pending-softdelete.jsonl")
|
||||
_ = p
|
||||
os.Setenv("ZHIYI_SOFTDELETE_PENDING", p)
|
||||
defer os.Unsetenv("ZHIYI_SOFTDELETE_PENDING")
|
||||
fn(p)
|
||||
}
|
||||
|
||||
// ① MVCC 冲突耗尽 ⇒ 入 durable 队列
|
||||
func TestSoftDelete_ExhaustConflicts_EnqueuesPending(t *testing.T) {
|
||||
withTempPending(t, func(p string) {
|
||||
fk := newFakeServer(t, -1, "") // 永远冲突
|
||||
defer fk.close()
|
||||
rc := fk.client()
|
||||
if err := rc.SoftDelete("mem_exhaust_enq", "merged_into_mem_X"); err == nil {
|
||||
t.Fatalf("永远冲突应返回错误")
|
||||
}
|
||||
if got := PendingSoftDeleteCount(); got != 1 {
|
||||
t.Fatalf("耗尽后应入队 1 条,实际=%d", got)
|
||||
}
|
||||
// 文件里确实含该 id(durable,重启不丢)
|
||||
b, _ := os.ReadFile(p)
|
||||
if !strings.Contains(string(b), "mem_exhaust_enq") {
|
||||
t.Fatalf("pending 文件未落盘该 id: %s", string(b))
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// ② 非可重试错误 ⇒ 不入队
|
||||
func TestSoftDelete_NonRetryable_NotEnqueued(t *testing.T) {
|
||||
withTempPending(t, func(p string) {
|
||||
fk := newFakeServer(t, -1, "lance error: Schema Mismatch: unexpected column")
|
||||
defer fk.close()
|
||||
rc := fk.client()
|
||||
_ = rc.SoftDelete("mem_schema_err", "unit_test")
|
||||
if got := PendingSoftDeleteCount(); got != 0 {
|
||||
t.Fatalf("非可重试错误不应入队,实际=%d", got)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// ③ 幂等去重
|
||||
func TestPendingSoftDelete_DedupSameID(t *testing.T) {
|
||||
withTempPending(t, func(p string) {
|
||||
EnqueuePendingSoftDelete("mem_dupe", "merged_into_a")
|
||||
EnqueuePendingSoftDelete("mem_dupe", "merged_into_b") // 同 id 第二次
|
||||
EnqueuePendingSoftDelete("mem_other", "merged_into_c")
|
||||
if got := PendingSoftDeleteCount(); got != 2 {
|
||||
t.Fatalf("同 id 去重后应 2 条,实际=%d", got)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// ④ drain:风暴退去(server 恒 OK)⇒ 逐条重放落盘 + 队列清空
|
||||
func TestDrainPendingStormSettled_LandsAndClears(t *testing.T) {
|
||||
withTempPending(t, func(p string) {
|
||||
// 预置 2 条待软删(模拟上一轮风暴耗尽留下的残留)
|
||||
EnqueuePendingSoftDelete("mem_pend_1", "merged_into_a")
|
||||
EnqueuePendingSoftDelete("mem_pend_2", "merged_into_b")
|
||||
if got := PendingSoftDeleteCount(); got != 2 {
|
||||
t.Fatalf("预置后应 2 条,实际=%d", got)
|
||||
}
|
||||
// 风暴退去:sidecar 现在恒 OK
|
||||
fk := newFakeServer(t, 0, "") // 首次即成功
|
||||
defer fk.close()
|
||||
rc := fk.client()
|
||||
ok, failed := DrainPendingSoftDeletes(rc)
|
||||
if ok != 2 || failed != 0 {
|
||||
t.Fatalf("drain 应成功 2 条 失败 0,实际 ok=%d failed=%d", ok, failed)
|
||||
}
|
||||
if got := PendingSoftDeleteCount(); got != 0 {
|
||||
t.Fatalf("drain 成功后队列应清空,实际=%d", got)
|
||||
}
|
||||
// 每条 SoftDelete 恰好 1 次 rpc(无冲突路径,无重试)
|
||||
if fk.callCount() != 2 {
|
||||
t.Fatalf("两条各 1 次 rpc,共应 2,实际=%d", fk.callCount())
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// ④b drain 仍在风暴(恒冲突)⇒ 保留在队列、不误删、下轮再试(最终一致)
|
||||
func TestDrainPendingStormStillOngoing_HoldsInQueue(t *testing.T) {
|
||||
withTempPending(t, func(p string) {
|
||||
EnqueuePendingSoftDelete("mem_hold", "merged_into_a")
|
||||
fk := newFakeServer(t, -1, "") // 仍然冲突
|
||||
defer fk.close()
|
||||
rc := fk.client()
|
||||
ok, failed := DrainPendingSoftDeletes(rc)
|
||||
if ok != 0 || failed != 1 {
|
||||
t.Fatalf("仍冲突时 drain 应 成功0 失败1,实际 ok=%d failed=%d", ok, failed)
|
||||
}
|
||||
// 关键:条目仍留在队列(未被误删),供下一 tick/启动再重放
|
||||
if got := PendingSoftDeleteCount(); got != 1 {
|
||||
t.Fatalf("仍失败者应保留在队列,实际=%d", got)
|
||||
}
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,99 @@
|
||||
// 织忆 MemoryWeave — stale-return / 生命周期 fixture(P1-3,2026-10-26)
|
||||
//
|
||||
// 评审指出的缺口:只有离线检索指标,没有 stale-return 与生命周期 fixture ——
|
||||
// 即「记忆过期后读侧表现」从未被机械验证过。
|
||||
//
|
||||
// 本文件锁三件事:
|
||||
// ① stale 标记默认开(720h / 30 天)但**只做展示**:老记忆被标 stale,新记忆不标;
|
||||
// ② bi-temporal 有效期窗口(fixture 取 10:00 生效 / 11:00 失效):
|
||||
// T<11:00 未过期、T>11:00 已过期 —— 用 10:00→11:00 的真实时窗语义,而非「随便设个过去时间」;
|
||||
// ③ supersede 反向指针在读侧可见(P1-1):SupersededBy 非空 ⇒ Superseded=true,且 JSON 输出。
|
||||
package storage
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/xiaoxue/memoryweave/internal/models"
|
||||
)
|
||||
|
||||
// TestStale_Default720h 默认阈值 720h:29 天不标、31 天标。
|
||||
func TestStale_Default720h(t *testing.T) {
|
||||
if got := envFloat("ZHIYI_STALE_AFTER_HOURS", 720); got != 720 {
|
||||
t.Fatalf("默认 stale 阈值应为 720h,实际 %v", got)
|
||||
}
|
||||
now := time.Now()
|
||||
if isStale(now.Add(-29*24*time.Hour), 720) {
|
||||
t.Fatal("29 天前更新不应标 stale")
|
||||
}
|
||||
if !isStale(now.Add(-31*24*time.Hour), 720) {
|
||||
t.Fatal("31 天前更新应标 stale")
|
||||
}
|
||||
// 关闭开关(0)⇒ 恒 false,保持旧行为可回滚
|
||||
if isStale(now.Add(-1000*time.Hour), 0) {
|
||||
t.Fatal("阈值 0 应关闭 stale 标记")
|
||||
}
|
||||
}
|
||||
|
||||
// TestValidityWindow_1000to1100 生命周期 fixture:事实 10:00 生效、11:00 失效。
|
||||
// 判据:只在 (valid_from, valid_to) 区间内被视为有效;区间外(含恰好 11:00 之后)判过期。
|
||||
func TestValidityWindow_1000to1100(t *testing.T) {
|
||||
day := time.Date(2026, 10, 26, 0, 0, 0, 0, time.Local)
|
||||
from := day.Add(10 * time.Hour).Format(time.RFC3339)
|
||||
to := day.Add(11 * time.Hour).Format(time.RFC3339)
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
now time.Time
|
||||
want bool // true = 已过期
|
||||
}{
|
||||
{"09:30 尚未生效(未过期,只是还没到生效点)", day.Add(9*time.Hour + 30*time.Minute), false},
|
||||
{"10:30 有效区间内", day.Add(10*time.Hour + 30*time.Minute), false},
|
||||
{"11:01 已过失效点", day.Add(11*time.Hour + time.Minute), true},
|
||||
{"23:00 当日深夜", day.Add(23 * time.Hour), true},
|
||||
}
|
||||
for _, c := range cases {
|
||||
if got := validityExpired(to, c.now); got != c.want {
|
||||
t.Errorf("%s:validityExpired(%s)=%v,期望 %v", c.name, to, got, c.want)
|
||||
}
|
||||
}
|
||||
// valid_to 为空 = 永久有效(NULL 语义)
|
||||
if validityExpired("", day.Add(1000*time.Hour)) {
|
||||
t.Fatal("valid_to 为空应视为永久有效")
|
||||
}
|
||||
// 非法格式保守视为未过期(不误伤存量脏数据)
|
||||
if validityExpired("not-a-time", day) {
|
||||
t.Fatal("valid_to 解析失败应保守视为未过期")
|
||||
}
|
||||
_ = from
|
||||
}
|
||||
|
||||
// TestSupersedeReadSideVisible P1-1:版本/取代关系必须真的进 JSON(不是「加了结构体没接线」)。
|
||||
func TestSupersedeReadSideVisible(t *testing.T) {
|
||||
r := models.RecallResult{
|
||||
ID: "mem_old", Content: "旧事实", Channel: "vector",
|
||||
Version: 2, ValidFrom: "2026-10-26T10:00:00+08:00", ValidTo: "2026-10-26T11:00:00+08:00",
|
||||
SupersededBy: []string{"mem_new"}, Superseded: true,
|
||||
}
|
||||
b, err := json.Marshal(r)
|
||||
if err != nil {
|
||||
t.Fatalf("marshal: %v", err)
|
||||
}
|
||||
var m map[string]interface{}
|
||||
if err := json.Unmarshal(b, &m); err != nil {
|
||||
t.Fatalf("unmarshal: %v", err)
|
||||
}
|
||||
if m["superseded"] != true {
|
||||
t.Fatalf("superseded 便捷布尔应输出 true:%s", string(b))
|
||||
}
|
||||
if got, _ := m["superseded_by"].([]interface{}); len(got) != 1 || got[0] != "mem_new" {
|
||||
t.Fatalf("superseded_by 未输出:%s", string(b))
|
||||
}
|
||||
if m["generation"] != float64(2) {
|
||||
t.Fatalf("generation(version)未输出:%s", string(b))
|
||||
}
|
||||
if m["valid_to"] == "" || m["valid_to"] == nil {
|
||||
t.Fatalf("valid_to 未输出:%s", string(b))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,194 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"log"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
"unicode"
|
||||
|
||||
"github.com/xiaoxue/memoryweave/internal/models"
|
||||
)
|
||||
|
||||
// 2026-10-25 (t_standing_recall) 常设答案 · 召回前置通道。
|
||||
//
|
||||
// 背景(实测,53 条真实提问基线):correct@1 = 20.8%,useful@5 = 88.7%;
|
||||
// 6 条"完全跑偏"案例里 5 条的目标文档**根本不在向量 top-50 里**——不是排序问题,
|
||||
// 是语义通道对"精确事实类"提问(端口/数字/标识符/专名)天然够不着。
|
||||
// 故补一条**零检索、零 LLM**的精确通道:查询命中常设答案 → 直接置顶。
|
||||
//
|
||||
// 与既有 FTS 通道的区别:
|
||||
// · FTS(t_2afae95e)只给尾部保证位,救不了 correct@1;
|
||||
// · 本通道置顶,只匹配"已定型的精确事实",不做泛化,故误召回代价低(答错比答不上更糟,
|
||||
// 所以阈值取保守值 + 只在环境开关开启时生效)。
|
||||
//
|
||||
// 开关:ZHIYI_STANDING_ANSWER=1 才生效(默认关 = 与旧行为逐位一致,便于 A/B 与一行回滚)。
|
||||
// 回滚:删 drop-in 里该变量 + daemon-reload + restart。
|
||||
|
||||
// StandingAnswerEnabled 常设答案通道是否启用(默认关)。
|
||||
func StandingAnswerEnabled() bool {
|
||||
return os.Getenv("ZHIYI_STANDING_ANSWER") == "1"
|
||||
}
|
||||
|
||||
// nearMissLogThreshold 近似命中日志阈值(**只用于日志**,不参与是否返回的决策)。
|
||||
// 取 0.78:近似到这个程度说明"很像同一句但不是",正是该补 question 变体的信号。
|
||||
const nearMissLogThreshold = 0.78
|
||||
|
||||
var (
|
||||
standingCacheMu sync.RWMutex
|
||||
standingCache []StandingAnswer
|
||||
standingCacheAt time.Time
|
||||
standingCacheTTL = 30 * time.Second
|
||||
standingLookupHit atomic.Int64 // 观测:通道命中次数(/health 或日志复用)
|
||||
)
|
||||
|
||||
// LookupStandingHitCount 返回本进程启动以来常设答案命中次数(观测用)。
|
||||
func LookupStandingHitCount() int64 { return standingLookupHit.Load() }
|
||||
|
||||
// normalizeForMatch 归一化:小写 + 去掉所有非字母数字/非 CJK 字符(空格、标点、符号)。
|
||||
func normalizeForMatch(s string) string {
|
||||
var b strings.Builder
|
||||
b.Grow(len(s))
|
||||
for _, r := range strings.ToLower(s) {
|
||||
if unicode.IsLetter(r) || unicode.IsDigit(r) {
|
||||
b.WriteRune(r)
|
||||
}
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// bigrams 生成字符二元组集合(CJK 与拉丁一视同仁;长度 <2 时退化为单字符集合)。
|
||||
func bigrams(s string) map[string]struct{} {
|
||||
rs := []rune(s)
|
||||
out := make(map[string]struct{}, len(rs))
|
||||
if len(rs) < 2 {
|
||||
if len(rs) == 1 {
|
||||
out[string(rs)] = struct{}{}
|
||||
}
|
||||
return out
|
||||
}
|
||||
for i := 0; i+1 < len(rs); i++ {
|
||||
out[string(rs[i:i+2])] = struct{}{}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// diceSimilarity 字符二元组 Dice 系数(0..1)。
|
||||
func diceSimilarity(a, b string) float64 {
|
||||
if a == "" || b == "" {
|
||||
return 0
|
||||
}
|
||||
if a == b {
|
||||
return 1
|
||||
}
|
||||
ba, bb := bigrams(a), bigrams(b)
|
||||
if len(ba) == 0 || len(bb) == 0 {
|
||||
return 0
|
||||
}
|
||||
inter := 0
|
||||
for g := range ba {
|
||||
if _, ok := bb[g]; ok {
|
||||
inter++
|
||||
}
|
||||
}
|
||||
return 2 * float64(inter) / float64(len(ba)+len(bb))
|
||||
}
|
||||
|
||||
// loadStandingAnswers 带 TTL 的常设答案快照(表极小,全量读;30s 内复用,避免热路径频繁查库)。
|
||||
func loadStandingAnswers() []StandingAnswer {
|
||||
standingCacheMu.RLock()
|
||||
if time.Since(standingCacheAt) < standingCacheTTL {
|
||||
cached := standingCache
|
||||
standingCacheMu.RUnlock()
|
||||
return cached
|
||||
}
|
||||
standingCacheMu.RUnlock()
|
||||
|
||||
standingCacheMu.Lock()
|
||||
defer standingCacheMu.Unlock()
|
||||
if time.Since(standingCacheAt) < standingCacheTTL { // 双检:并发下只读一次库
|
||||
return standingCache
|
||||
}
|
||||
store := StandingStoreInstance
|
||||
if store == nil {
|
||||
return nil // 未初始化:通道静默失效,不影响召回
|
||||
}
|
||||
all, err := store.List()
|
||||
if err != nil {
|
||||
return standingCache
|
||||
}
|
||||
standingCache = all
|
||||
standingCacheAt = time.Now()
|
||||
return all
|
||||
}
|
||||
|
||||
// LookupStandingAnswer 返回与 query 严格匹配的常设答案;无匹配返回 nil。
|
||||
//
|
||||
// 匹配口径 = **归一化后完全相等**(2026-10-25 实测修正):
|
||||
// 先前用 bigram Dice 模糊匹配(阈值 0.78),上线后留出集立刻抓到真实误命中——
|
||||
// 「织忆的服务跑在哪个端口」被判为「出图的服务跑在哪个端口」的匹配(仅主语两字之差,
|
||||
// Dice≈0.82),把 ComfyUI 的 8188 当成织忆端口置顶。模板句 + 换主语 = 必错。
|
||||
// 结论:常设答案只认**同一句话**;同义问法靠多行变体(question_id 不同)覆盖。
|
||||
// 近似命中(Dice ≥ nearMissLogThreshold)只打日志不返回,用来发现该补哪些变体。
|
||||
func LookupStandingAnswer(query string) *StandingAnswer {
|
||||
q := normalizeForMatch(query)
|
||||
if len([]rune(q)) < 2 {
|
||||
return nil
|
||||
}
|
||||
all := loadStandingAnswers()
|
||||
if len(all) == 0 {
|
||||
return nil
|
||||
}
|
||||
var near *StandingAnswer
|
||||
nearScore := 0.0
|
||||
for i := range all {
|
||||
cand := normalizeForMatch(all[i].Question)
|
||||
if cand == "" {
|
||||
continue
|
||||
}
|
||||
if cand == q {
|
||||
return &all[i]
|
||||
}
|
||||
if s := diceSimilarity(q, cand); s > nearScore {
|
||||
nearScore = s
|
||||
near = &all[i]
|
||||
}
|
||||
}
|
||||
// 近似但不等:只记日志(便于补变体),绝不返回——错答比不答更贵。
|
||||
if near != nil && nearScore >= nearMissLogThreshold {
|
||||
log.Printf("[standing] near-miss %.2f(不返回)query=%q 最接近=%q id=%s",
|
||||
nearScore, query, near.Question, near.QuestionID)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// PrependStandingAnswer 把常设答案置顶注入召回结果(去重 + 遵守 topK)。
|
||||
func PrependStandingAnswer(results []models.RecallResult, sa *StandingAnswer, topK int) []models.RecallResult {
|
||||
if sa == nil {
|
||||
return results
|
||||
}
|
||||
id := "standing:" + sa.QuestionID
|
||||
out := make([]models.RecallResult, 0, len(results)+1)
|
||||
out = append(out, models.RecallResult{
|
||||
ID: id,
|
||||
Content: sa.Answer,
|
||||
Category: "standing_answer",
|
||||
Score: 1.0,
|
||||
Timestamp: sa.UpdatedAt.Format(time.RFC3339),
|
||||
Source: "standing_answer",
|
||||
Channel: "standing",
|
||||
})
|
||||
for _, r := range results {
|
||||
if r.ID == id {
|
||||
continue
|
||||
}
|
||||
out = append(out, r)
|
||||
}
|
||||
if topK > 0 && len(out) > topK {
|
||||
out = out[:topK]
|
||||
}
|
||||
standingLookupHit.Add(1)
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,116 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/xiaoxue/memoryweave/internal/models"
|
||||
)
|
||||
|
||||
// 2026-10-25 (t_standing_recall) 常设答案通道的护栏测试。
|
||||
// 关注点是"宁可漏,不可错":误置顶一条错答案比召回不到更糟。
|
||||
|
||||
func TestNormalizeForMatch(t *testing.T) {
|
||||
cases := map[string]string{
|
||||
"cron 任务的调度者是谁?": "cron任务的调度者是谁",
|
||||
"Cron-Task 的 调度者": "crontask的调度者",
|
||||
"织忆的向量维度是多少": "织忆的向量维度是多少",
|
||||
" ": "",
|
||||
}
|
||||
for in, want := range cases {
|
||||
if got := normalizeForMatch(in); got != want {
|
||||
t.Errorf("normalize(%q)=%q want %q", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestDiceSimilarity(t *testing.T) {
|
||||
same := diceSimilarity("cron任务的调度者是谁", "cron任务的调度者是谁")
|
||||
if same != 1.0 {
|
||||
t.Fatalf("同句相似度应为 1.0,实得 %v", same)
|
||||
}
|
||||
near := diceSimilarity("cron任务的调度者是谁", "cron调度者是谁")
|
||||
if near < 0.7 || near > 1.0 {
|
||||
t.Fatalf("近义句相似度异常:%v", near)
|
||||
}
|
||||
far := diceSimilarity("cron任务的调度者是谁", "织忆的向量维度是多少")
|
||||
if far > 0.2 {
|
||||
t.Fatalf("跨主题相似度应接近 0,实得 %v", far)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLookupStandingAnswerThreshold(t *testing.T) {
|
||||
seedStanding(t, []StandingAnswer{
|
||||
{QuestionID: "cron-scheduler", Question: "cron 任务的调度者是谁", Answer: "kanban-daemon.service"},
|
||||
{QuestionID: "zhiyi-dim", Question: "织忆的向量维度是多少", Answer: "1024 维(bge-m3)"},
|
||||
})
|
||||
|
||||
// 命中:同句(归一化后相等)
|
||||
if sa := LookupStandingAnswer("cron 任务的调度者是谁?"); sa == nil || sa.QuestionID != "cron-scheduler" {
|
||||
t.Fatalf("同句应命中 cron-scheduler,实得 %+v", sa)
|
||||
}
|
||||
// 不命中:仅语序微调也不行(严格相等口径,防"模板句换主语"误命中)
|
||||
if sa := LookupStandingAnswer("织忆向量维度是多少"); sa != nil {
|
||||
t.Fatalf("非同一句不应命中(应靠变体行覆盖),实得 %+v", sa)
|
||||
}
|
||||
// 高相似但换主语 ⇒ 必须不命中(真实事故回归测试)
|
||||
if sa := LookupStandingAnswer("织忆的服务跑在哪个端口"); sa != nil {
|
||||
t.Fatalf("换主语的高相似句不得命中,实得 %+v", sa)
|
||||
}
|
||||
// 不命中:跨主题(防止"是谁"类短语乱命中)
|
||||
if sa := LookupStandingAnswer("今天的天气怎么样"); sa != nil {
|
||||
t.Fatalf("无关问题不应命中,实得 %+v", sa)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPrependStandingAnswer(t *testing.T) {
|
||||
sa := &StandingAnswer{QuestionID: "cron-scheduler", Question: "cron 任务的调度者是谁", Answer: "kanban-daemon.service"}
|
||||
in := []models.RecallResult{
|
||||
{ID: "mem_1", Content: "泛泛的 cron 描述", Score: 0.9},
|
||||
{ID: "mem_2", Content: "另一条", Score: 0.8},
|
||||
}
|
||||
out := PrependStandingAnswer(in, sa, 2)
|
||||
if len(out) != 2 {
|
||||
t.Fatalf("topK=2 应截断到 2 条,实得 %d", len(out))
|
||||
}
|
||||
if out[0].ID != "standing:cron-scheduler" || out[0].Content != "kanban-daemon.service" {
|
||||
t.Fatalf("常设答案应置顶,实得 %+v", out[0])
|
||||
}
|
||||
if out[1].ID != "mem_1" {
|
||||
t.Fatalf("原有结果顺序应保持,实得 %+v", out[1])
|
||||
}
|
||||
// 关闭时不影响(Enabled 只由 env 控制,这里验 nil 安全)
|
||||
if got := PrependStandingAnswer(in, nil, 2); len(got) != len(in) {
|
||||
t.Fatalf("nil 常设答案应原样返回")
|
||||
}
|
||||
}
|
||||
|
||||
// seedStanding 注入隔离的常设答案表(临时 SQLite 文件,不碰生产库)。
|
||||
func seedStanding(t *testing.T, rows []StandingAnswer) {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
store, err := NewStandingStore(dir + "/standing_answers.db")
|
||||
if err != nil {
|
||||
t.Fatalf("建临时常设答案库失败: %v", err)
|
||||
}
|
||||
for _, r := range rows {
|
||||
if _, err := store.Upsert(r.QuestionID, r.Question, r.Answer, nil); err != nil {
|
||||
t.Fatalf("写入常设答案失败: %v", err)
|
||||
}
|
||||
}
|
||||
prev := StandingStoreInstance
|
||||
StandingStoreInstance = store
|
||||
// 清缓存,确保读到本测试的数据
|
||||
standingCacheMu.Lock()
|
||||
standingCache = nil
|
||||
standingCacheAt = time.Time{}
|
||||
standingCacheMu.Unlock()
|
||||
t.Cleanup(func() {
|
||||
StandingStoreInstance = prev
|
||||
standingCacheMu.Lock()
|
||||
standingCache = nil
|
||||
standingCacheAt = time.Time{}
|
||||
standingCacheMu.Unlock()
|
||||
store.Close()
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,331 @@
|
||||
// 织忆 MemoryWeave — 召回链路追踪(trace_id 贯穿,t_trace_p0)
|
||||
//
|
||||
// 背景(CS249r Vol III Ch14 判据):全仓 grep trace_id|span = 0 ⇒「这条为什么被召回」
|
||||
// 永远答不出,后续所有调参都是盲修。本文件提供最小可用链路追踪:
|
||||
//
|
||||
// ① 每次 recall 生成 trace_id(tr_<16字节hex>),贯穿 编码→粗检索→FTS→重排→MMR→图谱→返回;
|
||||
// ② 各阶段耗时 + 各通道贡献条数 + 每条结果的来源通道(RecallResult.Channel);
|
||||
// ③ GET /api/v1/trace/<trace_id> 可重建整条链路(内存环形缓冲 + JSONL 落盘,重启后仍可查)。
|
||||
//
|
||||
// 设计约束:
|
||||
// - 不引入新依赖(crypto/rand 生成 ID,不新增 uuid 直接依赖);
|
||||
// - 落盘异步(有界 channel + 单写协程),recall 主路径只做内存 append,绝不为追踪拖慢召回;
|
||||
// - 落盘失败只降级为「仅内存」,不报错、不影响召回结果(追踪是观测设施,不是业务依赖);
|
||||
// - **这不是缓存**(不参与召回读写路径、不返回旧结果),与 searchcache/Redis 层的职责不同;
|
||||
// 写一次读多次的观测日志,用环形缓冲 + JSONL 是合适的形态,无需引入 Redis 键。
|
||||
package storage
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// TraceStage 单个阶段耗时。Count 语义随阶段不同(候选数/重排数/扩展数…)。
|
||||
type TraceStage struct {
|
||||
Name string `json:"name"`
|
||||
MS float64 `json:"ms"`
|
||||
Count int `json:"count,omitempty"`
|
||||
Note string `json:"note,omitempty"`
|
||||
}
|
||||
|
||||
// TraceItem 单条返回结果的来源归因。
|
||||
type TraceItem struct {
|
||||
Rank int `json:"rank"`
|
||||
ID string `json:"id"`
|
||||
Channel string `json:"channel"`
|
||||
Score float64 `json:"score"`
|
||||
}
|
||||
|
||||
// RecallTrace 一次召回的完整链路记录。
|
||||
type RecallTrace struct {
|
||||
TraceID string `json:"trace_id"`
|
||||
Query string `json:"query"`
|
||||
Namespace string `json:"namespace"`
|
||||
Mode string `json:"mode"`
|
||||
Status string `json:"status"` // complete | partial
|
||||
PartialReason string `json:"partial_reason,omitempty"`
|
||||
StartedAt time.Time `json:"started_at"`
|
||||
DurationMS float64 `json:"duration_ms"`
|
||||
CacheHit bool `json:"cache_hit"`
|
||||
CandidatesScanned int `json:"candidates_scanned"`
|
||||
Truncated bool `json:"truncated"`
|
||||
StandingAnswerHit string `json:"standing_answer_hit,omitempty"`
|
||||
TopK int `json:"top_k"`
|
||||
Channels map[string]int `json:"channels"`
|
||||
Stages []TraceStage `json:"stages"`
|
||||
Items []TraceItem `json:"items"`
|
||||
Scope map[string]interface{} `json:"scope,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
|
||||
started time.Time
|
||||
}
|
||||
|
||||
const (
|
||||
traceRingCap = 2000
|
||||
traceMaxFileSize = 32 << 20 // 32MB 轮转
|
||||
)
|
||||
|
||||
// traceDir 可通过 ZHIYI_TRACE_DIR 覆盖(测试用)。
|
||||
func traceDir() string {
|
||||
if d := os.Getenv("ZHIYI_TRACE_DIR"); d != "" {
|
||||
return d
|
||||
}
|
||||
return "/var/lib/memoryweave/traces"
|
||||
}
|
||||
|
||||
func tracePersistEnabled() bool {
|
||||
switch strings.ToLower(os.Getenv("ZHIYI_TRACE_PERSIST")) {
|
||||
case "0", "false", "off":
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// NewTraceID 生成 tr_<32hex>(16 字节随机,碰撞概率可忽略)。
|
||||
func NewTraceID() string {
|
||||
var b [16]byte
|
||||
if _, err := rand.Read(b[:]); err != nil {
|
||||
// 极端情况下退化为纳秒时间戳,仍保证非空、可区分。
|
||||
return fmt.Sprintf("tr_%d", time.Now().UnixNano())
|
||||
}
|
||||
return "tr_" + hex.EncodeToString(b[:])
|
||||
}
|
||||
|
||||
// NewRecallTrace 创建一条追踪记录(StartedAt 同时作为计时起点)。
|
||||
func NewRecallTrace(traceID, query, namespace, mode string) *RecallTrace {
|
||||
now := time.Now()
|
||||
if traceID == "" {
|
||||
traceID = NewTraceID()
|
||||
}
|
||||
return &RecallTrace{
|
||||
TraceID: traceID,
|
||||
Query: query,
|
||||
Namespace: namespace,
|
||||
Mode: mode,
|
||||
Status: "complete",
|
||||
StartedAt: now,
|
||||
Channels: map[string]int{},
|
||||
Stages: []TraceStage{},
|
||||
Items: []TraceItem{},
|
||||
started: now,
|
||||
}
|
||||
}
|
||||
|
||||
// Stage 记录一个阶段耗时。start 由调用方在阶段开始前取;count<0 表示不记录。
|
||||
func (t *RecallTrace) Stage(name string, start time.Time, count int) {
|
||||
if t == nil {
|
||||
return
|
||||
}
|
||||
t.Stages = append(t.Stages, TraceStage{
|
||||
Name: name,
|
||||
MS: float64(time.Since(start).Microseconds()) / 1000.0,
|
||||
Count: count,
|
||||
})
|
||||
}
|
||||
|
||||
// StageNote 记录带备注的阶段(如失败原因)。
|
||||
func (t *RecallTrace) StageNote(name string, start time.Time, count int, note string) {
|
||||
if t == nil {
|
||||
return
|
||||
}
|
||||
t.Stages = append(t.Stages, TraceStage{
|
||||
Name: name,
|
||||
MS: float64(time.Since(start).Microseconds()) / 1000.0,
|
||||
Count: count,
|
||||
Note: note,
|
||||
})
|
||||
}
|
||||
|
||||
// MarkPartial 标记本次召回为部分结果(deadline / 行数上限 / 阶段错误)。
|
||||
func (t *RecallTrace) MarkPartial(reason string) {
|
||||
if t == nil || reason == "" {
|
||||
return
|
||||
}
|
||||
t.Status = "partial"
|
||||
if t.PartialReason == "" {
|
||||
t.PartialReason = reason
|
||||
} else if !strings.Contains(t.PartialReason, reason) {
|
||||
t.PartialReason += "," + reason
|
||||
}
|
||||
}
|
||||
|
||||
// SetItems 用最终返回集填充 item 归因(Rank 从 1 起)。
|
||||
func (t *RecallTrace) SetItems(items []TraceItem) {
|
||||
if t == nil {
|
||||
return
|
||||
}
|
||||
t.Items = items
|
||||
t.Channels = map[string]int{}
|
||||
for _, it := range items {
|
||||
ch := it.Channel
|
||||
if ch == "" {
|
||||
ch = "unknown"
|
||||
}
|
||||
t.Channels[ch]++
|
||||
}
|
||||
}
|
||||
|
||||
var (
|
||||
traceMu sync.RWMutex
|
||||
traceRing = make([]*RecallTrace, 0, traceRingCap)
|
||||
traceIdx = map[string]*RecallTrace{}
|
||||
|
||||
traceFileMu sync.Mutex
|
||||
traceFile *os.File
|
||||
traceFileSize int64
|
||||
traceFileDay string
|
||||
|
||||
traceCh = make(chan *RecallTrace, 1024)
|
||||
traceOnce sync.Once
|
||||
)
|
||||
|
||||
// SaveTrace 写入内存环形缓冲,并异步落盘(非阻塞;缓冲满则丢落盘、保留内存)。
|
||||
func SaveTrace(t *RecallTrace) {
|
||||
if t == nil {
|
||||
return
|
||||
}
|
||||
if t.DurationMS == 0 {
|
||||
t.DurationMS = float64(time.Since(t.started).Microseconds()) / 1000.0
|
||||
}
|
||||
traceMu.Lock()
|
||||
if len(traceRing) >= traceRingCap {
|
||||
// 覆盖最旧:同时从索引删除
|
||||
old := traceRing[0]
|
||||
copy(traceRing, traceRing[1:])
|
||||
traceRing = traceRing[:len(traceRing)-1]
|
||||
if old != nil {
|
||||
delete(traceIdx, old.TraceID)
|
||||
}
|
||||
}
|
||||
traceRing = append(traceRing, t)
|
||||
traceIdx[t.TraceID] = t
|
||||
traceMu.Unlock()
|
||||
|
||||
if !tracePersistEnabled() {
|
||||
return
|
||||
}
|
||||
traceOnce.Do(func() { go traceWriter() })
|
||||
select {
|
||||
case traceCh <- t:
|
||||
default:
|
||||
// 落盘队列满:丢这一条(内存仍可查),不阻塞召回主路径。
|
||||
}
|
||||
}
|
||||
|
||||
// GetTrace 先查内存,再回落到落盘文件(支持服务重启后按 trace_id 回溯)。
|
||||
func GetTrace(traceID string) (*RecallTrace, bool) {
|
||||
if traceID == "" {
|
||||
return nil, false
|
||||
}
|
||||
traceMu.RLock()
|
||||
if t, ok := traceIdx[traceID]; ok {
|
||||
traceMu.RUnlock()
|
||||
return t, true
|
||||
}
|
||||
traceMu.RUnlock()
|
||||
return getTraceFromDisk(traceID)
|
||||
}
|
||||
|
||||
// TraceCount 当前内存中可查的 trace 数(自检/测试用)。
|
||||
func TraceCount() int {
|
||||
traceMu.RLock()
|
||||
defer traceMu.RUnlock()
|
||||
return len(traceRing)
|
||||
}
|
||||
|
||||
func traceWriter() {
|
||||
for t := range traceCh {
|
||||
writeTraceLine(t)
|
||||
}
|
||||
}
|
||||
|
||||
// writeTraceLine 追加一行 JSONL。失败静默(仅内存可查),不向调用方冒泡。
|
||||
func writeTraceLine(t *RecallTrace) {
|
||||
line, err := json.Marshal(t)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
line = append(line, '\n')
|
||||
|
||||
dir := traceDir()
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
return
|
||||
}
|
||||
day := time.Now().Format("20060102")
|
||||
|
||||
traceFileMu.Lock()
|
||||
defer traceFileMu.Unlock()
|
||||
|
||||
if traceFile == nil || traceFileDay != day || traceFileSize > traceMaxFileSize {
|
||||
if traceFile != nil {
|
||||
_ = traceFile.Close()
|
||||
traceFile = nil
|
||||
}
|
||||
path := filepath.Join(dir, "recall-"+day+".jsonl")
|
||||
if traceFileSize > traceMaxFileSize {
|
||||
// 轮转:同日超大时改名 .1(覆盖上一份)
|
||||
_ = os.Rename(path, path+".1")
|
||||
}
|
||||
f, err := os.OpenFile(path, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0o644)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
traceFile = f
|
||||
traceFileDay = day
|
||||
if st, err := f.Stat(); err == nil {
|
||||
traceFileSize = st.Size()
|
||||
} else {
|
||||
traceFileSize = 0
|
||||
}
|
||||
}
|
||||
if _, err := traceFile.Write(line); err == nil {
|
||||
traceFileSize += int64(len(line))
|
||||
}
|
||||
}
|
||||
|
||||
// getTraceFromDisk 从落地文件逆序扫描(只读,不影响运行中的服务)。
|
||||
func getTraceFromDisk(traceID string) (*RecallTrace, bool) {
|
||||
dir := traceDir()
|
||||
entries, err := os.ReadDir(dir)
|
||||
if err != nil {
|
||||
return nil, false
|
||||
}
|
||||
needle := `"trace_id":"` + traceID + `"`
|
||||
// 文件名含日期,倒序 = 新文件先查。
|
||||
for i := len(entries) - 1; i >= 0; i-- {
|
||||
e := entries[i]
|
||||
if e.IsDir() || !strings.HasPrefix(e.Name(), "recall-") {
|
||||
continue
|
||||
}
|
||||
f, err := os.Open(filepath.Join(dir, e.Name()))
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
var hit *RecallTrace
|
||||
sc := bufio.NewScanner(f)
|
||||
sc.Buffer(make([]byte, 0, 64*1024), 4<<20)
|
||||
for sc.Scan() {
|
||||
b := sc.Bytes()
|
||||
if !strings.Contains(string(b), needle) {
|
||||
continue
|
||||
}
|
||||
var t RecallTrace
|
||||
if json.Unmarshal(b, &t) == nil && t.TraceID == traceID {
|
||||
hit = &t
|
||||
}
|
||||
}
|
||||
_ = f.Close()
|
||||
if hit != nil {
|
||||
return hit, true
|
||||
}
|
||||
}
|
||||
return nil, false
|
||||
}
|
||||
@@ -0,0 +1,160 @@
|
||||
// 织忆 MemoryWeave — 召回链路追踪单元测试(P0-1)
|
||||
//
|
||||
// 验收要点:
|
||||
// - trace_id 唯一、可查(内存 + 落盘两条路径);
|
||||
// - 阶段/通道/条目归因完整(100 条全可重建,等价线上「100 请求 100% 可回溯」的 CI 断言);
|
||||
// - 环形缓冲按容量淘汰最旧且索引同步删除。
|
||||
package storage
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func resetTraceState(t *testing.T) {
|
||||
t.Helper()
|
||||
traceMu.Lock()
|
||||
traceRing = make([]*RecallTrace, 0, traceRingCap)
|
||||
traceIdx = map[string]*RecallTrace{}
|
||||
traceMu.Unlock()
|
||||
}
|
||||
|
||||
// TestRecallTrace_100RequestsAllRetrievable 对应验收:「CI 断言 100 个请求 100% 可重建调用链」。
|
||||
func TestRecallTrace_100RequestsAllRetrievable(t *testing.T) {
|
||||
resetTraceState(t)
|
||||
t.Setenv("ZHIYI_TRACE_PERSIST", "0")
|
||||
dir := t.TempDir()
|
||||
t.Setenv("ZHIYI_TRACE_DIR", dir)
|
||||
|
||||
ids := make([]string, 0, 100)
|
||||
for i := 0; i < 100; i++ {
|
||||
tr := NewRecallTrace("", fmt.Sprintf("query-%d", i), "hermes-main", "hybrid")
|
||||
tr.Stage("encode", time.Now(), 0)
|
||||
tr.Stage("coarse_search", time.Now(), 200)
|
||||
tr.Stage("graph_expand", time.Now(), 2)
|
||||
tr.SetItems([]TraceItem{
|
||||
{Rank: 1, ID: "mem_a", Channel: "vector", Score: 0.9},
|
||||
{Rank: 2, ID: "mem_b", Channel: "fts", Score: 0.8},
|
||||
{Rank: 3, ID: "mem_c", Channel: "graph", Score: 0.7},
|
||||
{Rank: 4, ID: "mem_d", Channel: "standing", Score: 1.0},
|
||||
{Rank: 5, ID: "mem_e", Channel: "vector", Score: 0.6},
|
||||
})
|
||||
SaveTrace(tr)
|
||||
ids = append(ids, tr.TraceID)
|
||||
}
|
||||
|
||||
if TraceCount() != 100 {
|
||||
t.Fatalf("应保存 100 条 trace,实际 %d", TraceCount())
|
||||
}
|
||||
for _, id := range ids {
|
||||
got, ok := GetTrace(id)
|
||||
if !ok {
|
||||
t.Fatalf("trace %s 不可回溯", id)
|
||||
}
|
||||
if len(got.Items) != 5 {
|
||||
t.Fatalf("trace %s item 归因应 5 条,实际 %d", id, len(got.Items))
|
||||
}
|
||||
if len(got.Stages) < 3 {
|
||||
t.Fatalf("trace %s 阶段应 ≥3,实际 %d", id, len(got.Stages))
|
||||
}
|
||||
for _, it := range got.Items {
|
||||
if it.Channel == "" {
|
||||
t.Fatalf("trace %s item %s 缺 channel(无法回答「它为什么被召回」)", id, it.ID)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestRecallTrace_DiskFallback 服务重启后仍可按 trace_id 回溯(内存清空 → 读 JSONL)。
|
||||
func TestRecallTrace_DiskFallback(t *testing.T) {
|
||||
resetTraceState(t)
|
||||
dir := t.TempDir()
|
||||
t.Setenv("ZHIYI_TRACE_DIR", dir)
|
||||
t.Setenv("ZHIYI_TRACE_PERSIST", "1")
|
||||
|
||||
tr := NewRecallTrace("", "落盘回溯测试", "hermes-main", "hybrid")
|
||||
tr.SetItems([]TraceItem{{Rank: 1, ID: "mem_x", Channel: "vector"}})
|
||||
tr.Status = "complete"
|
||||
// 直写文件(避免依赖异步协程时序),模拟落盘后的状态。
|
||||
writeTraceLine(tr)
|
||||
|
||||
if _, ok := GetTrace(tr.TraceID); !ok {
|
||||
t.Fatalf("落盘 trace %s 不可从文件回溯", tr.TraceID)
|
||||
}
|
||||
|
||||
// 再验证一次真实异步路径(轮询最多 2s)
|
||||
tr2 := NewRecallTrace("", "异步落盘", "hermes-main", "hybrid")
|
||||
SaveTrace(tr2)
|
||||
deadline := time.Now().Add(2 * time.Second)
|
||||
for time.Now().Before(deadline) {
|
||||
if _, ok := getTraceFromDisk(tr2.TraceID); ok {
|
||||
return
|
||||
}
|
||||
time.Sleep(20 * time.Millisecond)
|
||||
}
|
||||
t.Fatalf("异步落盘 2s 内未完成(trace %s)", tr2.TraceID)
|
||||
}
|
||||
|
||||
// TestRecallTrace_RingEviction 环形缓冲满后淘汰最旧,且索引同步删除。
|
||||
func TestRecallTrace_RingEviction(t *testing.T) {
|
||||
resetTraceState(t)
|
||||
t.Setenv("ZHIYI_TRACE_PERSIST", "0")
|
||||
|
||||
first := NewRecallTrace("", "first", "ns", "hybrid")
|
||||
SaveTrace(first)
|
||||
for i := 0; i < traceRingCap; i++ {
|
||||
SaveTrace(NewRecallTrace("", fmt.Sprintf("q-%d", i), "ns", "hybrid"))
|
||||
}
|
||||
if _, ok := GetTrace(first.TraceID); ok {
|
||||
t.Fatalf("环形缓冲应淘汰最旧的 trace")
|
||||
}
|
||||
if TraceCount() != traceRingCap {
|
||||
t.Fatalf("环形缓冲容量应保持 %d,实际 %d", traceRingCap, TraceCount())
|
||||
}
|
||||
}
|
||||
|
||||
// TestNewTraceID_Unique 1000 个 ID 不重复、带 tr_ 前缀。
|
||||
func TestNewTraceID_Unique(t *testing.T) {
|
||||
seen := map[string]bool{}
|
||||
for i := 0; i < 1000; i++ {
|
||||
id := NewTraceID()
|
||||
if len(id) < 10 || id[:3] != "tr_" {
|
||||
t.Fatalf("trace id 形态非法:%q", id)
|
||||
}
|
||||
if seen[id] {
|
||||
t.Fatalf("trace id 重复:%q", id)
|
||||
}
|
||||
seen[id] = true
|
||||
}
|
||||
}
|
||||
|
||||
// TestRecallTrace_MarkPartial partial 原因必须可读(不覆盖首个原因)。
|
||||
func TestRecallTrace_MarkPartial(t *testing.T) {
|
||||
tr := NewRecallTrace("tr_test", "q", "ns", "hybrid")
|
||||
if tr.Status != "complete" {
|
||||
t.Fatalf("新 trace 状态应为 complete,实际 %q", tr.Status)
|
||||
}
|
||||
tr.MarkPartial("deadline_exceeded")
|
||||
tr.MarkPartial("row_cap_truncated")
|
||||
if tr.Status != "partial" {
|
||||
t.Fatalf("MarkPartial 后状态应为 partial,实际 %q", tr.Status)
|
||||
}
|
||||
if tr.PartialReason != "deadline_exceeded,row_cap_truncated" {
|
||||
t.Fatalf("partial_reason 应累积,实际 %q", tr.PartialReason)
|
||||
}
|
||||
}
|
||||
|
||||
// TestTraceDir_Default 默认目录形态(不创建文件,只校验路径选择逻辑)。
|
||||
func TestTraceDir_Default(t *testing.T) {
|
||||
os.Unsetenv("ZHIYI_TRACE_DIR")
|
||||
if got := traceDir(); got != "/var/lib/memoryweave/traces" {
|
||||
t.Fatalf("默认 trace 目录错误:%s", got)
|
||||
}
|
||||
t.Setenv("ZHIYI_TRACE_DIR", filepath.Join(t.TempDir(), "x"))
|
||||
if got := traceDir(); got == "/var/lib/memoryweave/traces" {
|
||||
t.Fatalf("ZHIYI_TRACE_DIR 未生效")
|
||||
}
|
||||
}
|
||||
@@ -68,6 +68,34 @@ var (
|
||||
usageTokenTopN = int(envFloat("ZHIYI_USAGE_TOKEN_TOP_N", 8))
|
||||
)
|
||||
|
||||
// 退避参数(t_20bfbd9a 治本①):命中事件重试的跨 commit 指数退避。
|
||||
// 与 IPC 层 in-call 退避(lancedb_ipc.go 50/100/200ms)正交:这里的退避是**跨 commit 层面**,
|
||||
// 防的是「一条持久性失败的坏行在每次 commit 都被热重扫」→ 风暴叠风暴。首次失败退避 1s,
|
||||
// 每多失败一次翻倍,封顶 10min。成功消费后 retry_count 归零(usage_track.go 消费分支)。
|
||||
var (
|
||||
usageBackoffBase = envFloat("ZHIYI_USAGE_BACKOFF_BASE_S", 1)
|
||||
usageBackoffCap = envFloat("ZHIYI_USAGE_BACKOFF_CAP_S", 600)
|
||||
)
|
||||
|
||||
// usageBackoff 第 n 次失败的退避时长:min(2^(n-1) * base, cap)。
|
||||
// n>=1;n=1 → base,n=2 → 2*base,… 封顶 cap。
|
||||
func usageBackoff(n int64) time.Duration {
|
||||
s := usageBackoffBase
|
||||
for i := int64(1); i < n; i++ {
|
||||
next := s * 2
|
||||
if next >= usageBackoffCap {
|
||||
// 下一步将超过 cap:直接封顶返回,不再累计
|
||||
return time.Duration(usageBackoffCap * float64(time.Second))
|
||||
}
|
||||
s = next
|
||||
}
|
||||
return time.Duration(s * float64(time.Second))
|
||||
}
|
||||
|
||||
// envFloat 读取环境浮点配置(默认值兜底;0/负值视为默认)。
|
||||
// ⚠️ recall.go 已有同签名 envFloat(t_da61c470,语义:解析失败回落默认、≤0 原样返回),
|
||||
// 这里不再重声明,直接复用;消费方需自备 >0 守卫(backlogMax/backoff 均已带)。
|
||||
|
||||
// ─── 库文件(织忆自己的库,非 Hermes state.db)─────────────────────
|
||||
// var 而非 const:测试可换到临时路径,避免污染生产 usage_track.db。
|
||||
var usageTrackDBPath = "/var/lib/memoryweave/usage_track.db"
|
||||
@@ -108,10 +136,22 @@ func usageOpen() *sql.DB {
|
||||
agent TEXT,
|
||||
recalled_at TEXT NOT NULL,
|
||||
consumed INTEGER NOT NULL DEFAULT 0,
|
||||
consumed_at TEXT
|
||||
consumed_at TEXT,
|
||||
retry_count INTEGER NOT NULL DEFAULT 0,
|
||||
next_retry_at TEXT
|
||||
)`); err != nil {
|
||||
logUsage("create table: %v", err)
|
||||
}
|
||||
// 存量库迁移(t_20bfbd9a):旧表无 retry_count/next_retry_at 列,逐列补(幂等)。
|
||||
for _, stmt := range []string{
|
||||
"ALTER TABLE usage_recall ADD COLUMN retry_count INTEGER NOT NULL DEFAULT 0",
|
||||
"ALTER TABLE usage_recall ADD COLUMN next_retry_at TEXT",
|
||||
} {
|
||||
if _, err := db.Exec(stmt); err != nil && !strings.Contains(err.Error(), "duplicate column name") {
|
||||
// 非「已存在」的错(表不存在等)也忽略:CREATE IF NOT EXISTS 已兜表级,
|
||||
// 这里只负责加列,列已存在即静默通过。
|
||||
}
|
||||
}
|
||||
_, _ = db.Exec(`CREATE INDEX IF NOT EXISTS idx_usage_mem ON usage_recall(memory_id)`)
|
||||
_, _ = db.Exec(`CREATE INDEX IF NOT EXISTS idx_usage_time ON usage_recall(recalled_at)`)
|
||||
usageDB = db
|
||||
@@ -169,10 +209,26 @@ func CheckCommittedForUse(ldb LanceDB, agent, namespace, content string) []Usage
|
||||
if db == nil {
|
||||
return nil
|
||||
}
|
||||
// 积压上限量化(t_20bfbd9a 治本②):未消费待重试行超过上限 ⇒ 本轮跳过扫描,
|
||||
// 防止「一条永远失败的坏行 + 高 commit 频率」在每次 commit 都全量重扫、叠成风暴。
|
||||
// 上限 env ZHIYI_USAGE_BACKLOG_MAX,默认 500。
|
||||
backlogMax := int(envFloat("ZHIYI_USAGE_BACKLOG_MAX", 500))
|
||||
if backlogMax > 0 {
|
||||
var pending int
|
||||
if err := db.QueryRow(`SELECT COUNT(*) FROM usage_recall WHERE consumed=0`).Scan(&pending); err == nil && pending > backlogMax {
|
||||
logUsage("backlog %d > cap %d: skip scan this commit", pending, backlogMax)
|
||||
return nil
|
||||
}
|
||||
}
|
||||
cutoff := time.Now().Add(-time.Duration(usageWindowHours * float64(time.Hour))).Format(time.RFC3339)
|
||||
scanNow := time.Now()
|
||||
// 退避过滤(t_20bfbd9a 治本①):next_retry_at 为 NULL(首次待处理)或已到重试时间,
|
||||
// 才进本轮重扫。持久性失败的坏行被指数退避挡在窗口外,不再每 commit 热重试。
|
||||
rows, err := db.Query(
|
||||
`SELECT id, memory_id, content, tokens, namespace, agent
|
||||
FROM usage_recall WHERE consumed=0 AND recalled_at>=?`, cutoff)
|
||||
`SELECT id, memory_id, content, tokens, namespace, agent, retry_count
|
||||
FROM usage_recall WHERE consumed=0 AND recalled_at>=?
|
||||
AND (next_retry_at IS NULL OR next_retry_at<=?)`,
|
||||
cutoff, scanNow.Format(time.RFC3339))
|
||||
if err != nil {
|
||||
logUsage("query recall log: %v", err)
|
||||
return nil
|
||||
@@ -183,7 +239,8 @@ func CheckCommittedForUse(ldb LanceDB, agent, namespace, content string) []Usage
|
||||
for rows.Next() {
|
||||
var id int64
|
||||
var memID, memContent, tokensJSON, ns, ag string
|
||||
if err := rows.Scan(&id, &memID, &memContent, &tokensJSON, &ns, &ag); err != nil {
|
||||
var retryCount int64
|
||||
if err := rows.Scan(&id, &memID, &memContent, &tokensJSON, &ns, &ag, &retryCount); err != nil {
|
||||
continue
|
||||
}
|
||||
// 归属过滤:同一 agent,或同一 namespace,或共享/全局。跨域不强求。
|
||||
@@ -207,6 +264,13 @@ func CheckCommittedForUse(ldb LanceDB, agent, namespace, content string) []Usage
|
||||
// zhiyi_hit_event_append_failed_total),禁静默。
|
||||
if err := ldb.AppendHitEvent(memID, HitOutcomePos, HitReasonDownstreamUse); err != nil {
|
||||
logUsage("hit event append FAILED (id=%s, NOT consumed, will retry next commit) : %v", memID, err)
|
||||
// t_20bfbd9a 治本①:持久性失败(如坏行 memID 已从 LanceDB 删除)若不加退避,
|
||||
// 会在每次 commit 都热重试,commit 风暴时叠成重扫风暴。加指数退避:
|
||||
// 第 n 次失败后 next_retry_at = now + min(2^(n-1) * base, cap),退避期外不再重扫。
|
||||
// 成功消费时把 retry_count 归零;每轮只推进 retry_count 一次(不指数级暴涨)。
|
||||
_, _ = db.Exec(
|
||||
`UPDATE usage_recall SET retry_count=retry_count+1, next_retry_at=? WHERE id=?`,
|
||||
time.Now().Add(usageBackoff(retryCount+1)).Format(time.RFC3339), id)
|
||||
continue
|
||||
}
|
||||
// t_28aa278e 方案①:更新成功后才消费 usage_recall 行。
|
||||
@@ -218,9 +282,12 @@ func CheckCommittedForUse(ldb LanceDB, agent, namespace, content string) []Usage
|
||||
"recall_verified_count": map[string]string{"$inc": "1"},
|
||||
}); err != nil {
|
||||
logUsage("recall_verified_count update FAILED (id=%s, NOT consumed, will retry next commit) : %v", memID, err)
|
||||
_, _ = db.Exec(
|
||||
`UPDATE usage_recall SET retry_count=retry_count+1, next_retry_at=? WHERE id=?`,
|
||||
time.Now().Add(usageBackoff(retryCount+1)).Format(time.RFC3339), id)
|
||||
continue
|
||||
}
|
||||
_, _ = db.Exec(`UPDATE usage_recall SET consumed=1, consumed_at=? WHERE id=?`, now, id)
|
||||
_, _ = db.Exec(`UPDATE usage_recall SET consumed=1, consumed_at=?, retry_count=0, next_retry_at=NULL WHERE id=?`, now, id)
|
||||
flags = append(flags, UsageFlag{
|
||||
MemoryID: memID, Content: memContent, Criterion: "carry_forward",
|
||||
MatchedTokens: toks, HitCount: hits, RunesMatched: runes, ConsumedAt: now,
|
||||
|
||||
@@ -3,6 +3,7 @@ package storage
|
||||
import (
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// setupUsageTest 把追踪库指到临时路径 + 显式打开门控,测完复位。
|
||||
@@ -123,4 +124,60 @@ func TestUsageTracking_NoFalsePositiveOnUnrelated(t *testing.T) {
|
||||
if f := CheckCommittedForUse(mlc, "a06", "hermes", "今天天气很好,出去散步了"); len(f) != 0 {
|
||||
t.Fatalf("无关内容不应误判, got %v", f)
|
||||
}
|
||||
}
|
||||
}
|
||||
// t_20bfbd9a 治本①:跨 commit 指数退避 —— usageBackoff 第 n 次失败的增长与封顶。
|
||||
func TestUsageBackoff_ExponentialGrowthAndCap(t *testing.T) {
|
||||
// 默认 base=1s, cap=600s
|
||||
one := usageBackoff(1)
|
||||
two := usageBackoff(2)
|
||||
three := usageBackoff(3)
|
||||
ten := usageBackoff(10)
|
||||
eleven := usageBackoff(11)
|
||||
if one != time.Second {
|
||||
t.Fatalf("第1次失败应退避 base(1s), got %v", one)
|
||||
}
|
||||
if two != 2*time.Second {
|
||||
t.Fatalf("第2次失败应退避 2*base(2s), got %v", two)
|
||||
}
|
||||
if three != 4*time.Second {
|
||||
t.Fatalf("第3次失败应退避 4*base(4s), got %v", three)
|
||||
}
|
||||
if ten != 512*time.Second {
|
||||
t.Fatalf("第10次失败应退避 512s(尚未超过 cap 600s), got %v", ten)
|
||||
}
|
||||
if eleven != 600*time.Second {
|
||||
t.Fatalf("第11次失败(下一次翻倍 1024s 将超 cap 600s)应封顶 600s, got %v", eleven)
|
||||
}
|
||||
}
|
||||
|
||||
// t_20bfbd9a 治本②:积压上限量化 —— 未消费待重试行超过 ZHIYI_USAGE_BACKLOG_MAX 时本轮跳过扫描。
|
||||
func TestCheckCommittedForUse_BacklogCapSkipsScan(t *testing.T) {
|
||||
setupUsageTest(t)
|
||||
db := usageOpen()
|
||||
if db == nil {
|
||||
t.Fatal("usageOpen failed")
|
||||
}
|
||||
// 显式压小积压上限,验证超过即跳过
|
||||
t.Setenv("ZHIYI_USAGE_BACKLOG_MAX", "2")
|
||||
now := time.Now().Format(time.RFC3339)
|
||||
// 塞 3 条未消费召回行(超过上限 2)
|
||||
for i := 0; i < 3; i++ {
|
||||
if _, err := db.Exec(`INSERT INTO usage_recall(memory_id, content, tokens, q_hash, namespace, agent, recalled_at)
|
||||
VALUES(?,?,?,?,?,?,?)`,
|
||||
"mem_bk"+string(rune('0'+i)), "牧尘今天要求处理看板任务验收"+string(rune('0'+i)),
|
||||
`["牧尘今天要求处理看板任务验收"]`, "q", "ns", "agent", now); err != nil {
|
||||
t.Fatalf("insert: %v", err)
|
||||
}
|
||||
}
|
||||
// 无需真实 LanceDB(ldb=nil 直接返回 nil flags);本用例只验证「超限即提前跳过」不 panic 且不消费
|
||||
flags := CheckCommittedForUse(nil, "agent", "ns", "牧尘今天要求处理看板任务验收0")
|
||||
if flags != nil {
|
||||
t.Fatalf("ldb=nil 应直接返回 nil flags(超限跳过不应触及消费路径), got %v", flags)
|
||||
}
|
||||
// 判据:超限跳过时所有行仍 consumed=0(没有消费行为发生)
|
||||
var pending int
|
||||
_ = db.QueryRow(`SELECT COUNT(*) FROM usage_recall WHERE consumed=0`).Scan(&pending)
|
||||
if pending != 3 {
|
||||
t.Fatalf("超限跳过扫描后应 3 行全部未消费(no consumption), got %d", pending)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -26,3 +26,9 @@ default = []
|
||||
[[bin]]
|
||||
name = "zhiyi-consolidate"
|
||||
path = "src/main.rs"
|
||||
|
||||
# t_789898bd:存量 importance 回填一次性工具(只写 importance 列,组提交 ~25 版本)。
|
||||
# 新增 [[bin]] 不影响 zhiyi-consolidate 二进制本身(sidecar 不用重新部署/重启)。
|
||||
[[bin]]
|
||||
name = "importance_backfill"
|
||||
path = "src/bin/importance_backfill.rs"
|
||||
|
||||
@@ -0,0 +1,305 @@
|
||||
//! t_789898bd AC3 存量 importance 回填工具(一次性)
|
||||
//!
|
||||
//! 背景:写入侧 importance 恒硬编码 1.0,全库 46,9xx 行(memories 表)恒 1.0。
|
||||
//! 本工具按 Go 侧 `go/internal/storage/importance_rule.go` 的**同一张分档表**回填 importance 列:
|
||||
//! tier=core → 0.9 ; normal × wiki:*|llm_distill → 0.7 ;
|
||||
//! normal × legacy:*|unknown → 0.4 ; normal × recovery:* → 0.3(预留档)
|
||||
//!
|
||||
//! 写路径约束(AC5 红线):
|
||||
//! · 只写 `importance` 一列(lance update 的 .column("importance", 字面量表达式))
|
||||
//! · 不用 CASE WHEN(0.15 planner 拒绝,实测见 lancedb_ops.rs:638 注释)
|
||||
//! · 按 (目标值, id 分块 IN 谓词) 组提交 —— 复用仓库已验证的
|
||||
//! `id IN (...)` 批量模式(lancedb_update_batch 同款,lancedb_ops.rs:640)
|
||||
//! · 每目标值 ~2000 id/块 ⇒ 全表约 25 个版本(对比 46,874 次单行提交)
|
||||
//! · 提交冲突(与在跑 sidecar 并发写,lance Display 含 "Commit conflict")有界退避重试
|
||||
//! · 不碰 episodes 表 / graph / 不重启 zhiyid;.delete()/optimize() 一律不用
|
||||
//! · 回滚 = `--mode rollback`:整表 importance=1.0(1 个版本),即改动前状态
|
||||
//! · 读侧坑(lancedb 0.15):Query 默认 limit=10 ⇒ 全表扫描必须 count_rows + .limit(n)
|
||||
//!
|
||||
//! 用法:
|
||||
//! importance_backfill --data-dir /path/to/zhiyi/data --mode dry-run
|
||||
//! importance_backfill --data-dir /path/to/zhiyi/data --mode apply
|
||||
//! importance_backfill --data-dir /path/to/zhiyi/data --mode verify
|
||||
//! importance_backfill --data-dir /path/to/zhiyi/data --mode rollback
|
||||
use arrow::array::{Float64Array, StringArray};
|
||||
use arrow::record_batch::RecordBatch;
|
||||
use clap::Parser;
|
||||
use futures::StreamExt;
|
||||
use lancedb::query::{ExecutableQuery, QueryBase, Select};
|
||||
use std::collections::HashMap;
|
||||
use tokio::runtime::Runtime;
|
||||
|
||||
#[derive(Parser)]
|
||||
struct Args {
|
||||
/// LanceDB 数据目录(内含 <table>.lance 数据集目录)
|
||||
#[arg(long)]
|
||||
data_dir: String,
|
||||
/// 目标表(只允许 memories)
|
||||
#[arg(long, default_value = "memories")]
|
||||
table: String,
|
||||
/// dry-run | apply | verify | rollback
|
||||
#[arg(long, default_value = "dry-run")]
|
||||
mode: String,
|
||||
/// IN 谓词每块最大 id 数
|
||||
#[arg(long, default_value_t = 2000u32)]
|
||||
chunk: u32,
|
||||
}
|
||||
|
||||
/// 与 Go storage/importance_rule.go 同源的分档表(t_789898bd 标定)。
|
||||
/// 永不返回 1.0(「恒 1.0 行数 → 0」验收依赖)。
|
||||
fn target_importance(tier: &str, source: &str) -> f64 {
|
||||
if tier == "core" {
|
||||
return 0.9;
|
||||
}
|
||||
if source == "llm_distill" || source.starts_with("wiki:") {
|
||||
return 0.7;
|
||||
}
|
||||
if source.starts_with("recovery:") {
|
||||
return 0.3;
|
||||
}
|
||||
// legacy:* / 未知 / 空 source ⇒ 低置信类
|
||||
0.4
|
||||
}
|
||||
|
||||
fn rt() -> &'static Runtime {
|
||||
static ONCE: std::sync::Once = std::sync::Once::new();
|
||||
static mut RT: Option<Runtime> = None;
|
||||
ONCE.call_once(|| unsafe {
|
||||
RT = Some(Runtime::new().expect("tokio runtime"));
|
||||
});
|
||||
unsafe { RT.as_ref().unwrap() }
|
||||
}
|
||||
|
||||
/// 一次组提交(IN 谓词或全表)+ 提交冲突有界退避重试(lance Display 含 "Commit conflict")。
|
||||
/// 每次重试重建 UpdateBuilder(execute 消费 self)。
|
||||
async fn commit_update(
|
||||
tbl: &lancedb::table::Table,
|
||||
pred: Option<&str>,
|
||||
expr: &str,
|
||||
label: &str,
|
||||
) -> Result<u64, Box<dyn std::error::Error>> {
|
||||
let mut attempt = 0u32;
|
||||
loop {
|
||||
let mut op = tbl.update();
|
||||
if let Some(p) = pred {
|
||||
op = op.only_if(p.to_string());
|
||||
}
|
||||
op = op.column("importance", expr);
|
||||
match op.execute().await {
|
||||
Ok(rows) => {
|
||||
eprintln!("[backfill] commit ok: {} rows={}", label, rows);
|
||||
return Ok(rows);
|
||||
}
|
||||
Err(e) => {
|
||||
attempt += 1;
|
||||
let msg = e.to_string();
|
||||
if !msg.contains("Commit conflict") {
|
||||
return Err(format!("commit failed ({}): {}", label, msg).into());
|
||||
}
|
||||
if attempt >= 10 {
|
||||
return Err(format!("commit conflict ×10 ({}): {}", label, msg).into());
|
||||
}
|
||||
let backoff_ms = 500u64 * 2u64.pow(attempt);
|
||||
eprintln!(
|
||||
"[backfill] commit conflict ({}), retry {}/10 in {}ms",
|
||||
label, attempt, backoff_ms
|
||||
);
|
||||
tokio::time::sleep(std::time::Duration::from_millis(backoff_ms)).await;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// 取某行某列的字符串(列缺失/类型不符 = 硬错误,不猜)。
|
||||
fn str_at(b: &RecordBatch, i: usize, col: &str) -> Result<String, String> {
|
||||
Ok(
|
||||
b.column_by_name(col)
|
||||
.ok_or_else(|| format!("column {} missing", col))?
|
||||
.as_any()
|
||||
.downcast_ref::<StringArray>()
|
||||
.ok_or_else(|| format!("column {} not string", col))?
|
||||
.value(i)
|
||||
.to_string(),
|
||||
)
|
||||
}
|
||||
|
||||
/// f64 不是 HashMap 键(无 Eq)⇒ 用 ×10 的 i32 做分档键(0.9→9, 0.7→7, 0.4→4, 0.3→3)。
|
||||
fn vkey(v: f64) -> i32 {
|
||||
(v * 10.0).round() as i32
|
||||
}
|
||||
|
||||
/// 全表拉 (id, tier, source, importance)。注意:Query 默认 limit=10 ⇒ 必须显式 .limit(count)。
|
||||
async fn scan_rows(
|
||||
tbl: &lancedb::table::Table,
|
||||
) -> Result<Vec<(String, String, String, f64)>, Box<dyn std::error::Error>> {
|
||||
let count = tbl.count_rows(Some("id IS NOT NULL".into())).await? as usize;
|
||||
let mut results = Box::pin(
|
||||
tbl.query()
|
||||
.select(Select::columns(&["id", "tier", "source", "importance"]))
|
||||
.only_if("id IS NOT NULL")
|
||||
.limit(count)
|
||||
.execute()
|
||||
.await?,
|
||||
);
|
||||
let mut out: Vec<(String, String, String, f64)> = Vec::new();
|
||||
while let Some(b) = results.next().await {
|
||||
let batch = b?;
|
||||
for i in 0..batch.num_rows() {
|
||||
let id = str_at(&batch, i as usize, "id")?;
|
||||
let tier = str_at(&batch, i as usize, "tier")?;
|
||||
let src = str_at(&batch, i as usize, "source")?;
|
||||
let imp = batch
|
||||
.column_by_name("importance")
|
||||
.ok_or("importance column missing")?
|
||||
.as_any()
|
||||
.downcast_ref::<Float64Array>()
|
||||
.ok_or("importance column not float64")?
|
||||
.value(i as usize);
|
||||
out.push((id, tier, src, imp));
|
||||
}
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
async fn run(args: Args) -> Result<(), Box<dyn std::error::Error>> {
|
||||
if args.table != "memories" {
|
||||
return Err(format!(
|
||||
"refusing table={}: 本卡红线只允许回填 memories(不碰 episodes.lance)",
|
||||
args.table
|
||||
)
|
||||
.into());
|
||||
}
|
||||
let db = lancedb::connect(&args.data_dir).execute().await?;
|
||||
let tbl = db.open_table(&args.table).execute().await?;
|
||||
|
||||
// ── 1) 只读拉全表 ──
|
||||
let rows = scan_rows(&tbl).await?;
|
||||
let n = rows.len() as u64;
|
||||
let imp10_before = rows.iter().filter(|r| r.3 == 1.0).count();
|
||||
eprintln!(
|
||||
"[backfill] table={} rows={} importance==1.0 before = {}",
|
||||
args.table, n, imp10_before
|
||||
);
|
||||
|
||||
match args.mode.as_str() {
|
||||
"dry-run" => {
|
||||
let mut plan: HashMap<i32, usize> = HashMap::new();
|
||||
for (_, tier, src, _) in &rows {
|
||||
*plan.entry(vkey(target_importance(tier, src))).or_insert(0) += 1;
|
||||
}
|
||||
eprintln!("[backfill] plan (chunks per group, chunk={}):", args.chunk);
|
||||
let mut plan_vec: Vec<(i32, &usize)> = plan
|
||||
.iter()
|
||||
.map(|(k, c)| (*k, c))
|
||||
.collect();
|
||||
plan_vec.sort();
|
||||
for (k, c) in plan_vec {
|
||||
let chunks = (*c as u64 + args.chunk as u64 - 1) / args.chunk as u64;
|
||||
eprintln!(
|
||||
" importance={} → {} rows, {} chunks",
|
||||
k as f64 / 10.0, c, chunks
|
||||
);
|
||||
}
|
||||
eprintln!("[backfill] dry-run: no commit.");
|
||||
Ok(())
|
||||
}
|
||||
"verify" => {
|
||||
// 只读验收:逐行对规则复算 + 1.0 计数 + 直方图
|
||||
let mut mismatch: u64 = 0;
|
||||
let mut one: u64 = 0;
|
||||
let mut hist: HashMap<i32, u64> = HashMap::new();
|
||||
for (id, tier, src, imp) in &rows {
|
||||
let want = target_importance(tier, src);
|
||||
if (imp - want).abs() > 1e-9 {
|
||||
mismatch += 1;
|
||||
if mismatch <= 5 {
|
||||
eprintln!(
|
||||
"[verify] MISMATCH id={} tier={} source={} imp={} want={}",
|
||||
id, tier, src, imp, want
|
||||
);
|
||||
}
|
||||
}
|
||||
if *imp == 1.0 {
|
||||
one += 1;
|
||||
}
|
||||
*hist.entry(vkey(*imp)).or_insert(0) += 1;
|
||||
}
|
||||
eprintln!(
|
||||
"[verify] rows={} mismatch={}/{} importance==1.0={}",
|
||||
rows.len(),
|
||||
mismatch,
|
||||
n,
|
||||
one
|
||||
);
|
||||
eprintln!("[verify] histogram:");
|
||||
let mut hist_vec: Vec<(i32, &u64)> = hist
|
||||
.iter()
|
||||
.map(|(k, c)| (*k, c))
|
||||
.collect();
|
||||
hist_vec.sort();
|
||||
for (k, c) in hist_vec {
|
||||
eprintln!(
|
||||
" {} : {} ({}%)",
|
||||
k as f64 / 10.0,
|
||||
c,
|
||||
*c as f64 * 100.0 / n as f64
|
||||
);
|
||||
}
|
||||
if mismatch == 0 {
|
||||
eprintln!("[verify] PASS: 全部行 importance == 规则值");
|
||||
} else {
|
||||
return Err(format!(
|
||||
"[verify] FAIL: {} rows mismatch(新写入行可能是 1.0 漂移,可重跑 apply 收敛)",
|
||||
mismatch
|
||||
)
|
||||
.into());
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
"rollback" => {
|
||||
// 回滚 = 整表 importance=1.0(1 个版本)= 改动前状态
|
||||
let rows_upd = commit_update(&tbl, None, "1.0", "rollback(all→1.0)").await?;
|
||||
eprintln!("[backfill] rollback done, updated rows={}", rows_upd);
|
||||
Ok(())
|
||||
}
|
||||
"apply" => {
|
||||
// 组提交:(目标值, id 分块) → 每块一次 update(IN 谓词 + importance 字面量)
|
||||
let mut by_value: HashMap<i32, Vec<String>> = HashMap::new();
|
||||
for (id, tier, src, _) in &rows {
|
||||
by_value
|
||||
.entry(vkey(target_importance(tier, src)))
|
||||
.or_default()
|
||||
.push(id.clone());
|
||||
}
|
||||
let mut groups: Vec<(i32, Vec<String>)> = by_value.into_iter().collect();
|
||||
groups.sort();
|
||||
for (k, vids) in &groups {
|
||||
let lit = format!("{}", *k as f64 / 10.0);
|
||||
for chunk in vids.chunks(args.chunk as usize) {
|
||||
let in_list: Vec<String> = chunk
|
||||
.iter()
|
||||
.map(|id| format!("'{}'", id.replace('\'', "''")))
|
||||
.collect();
|
||||
let pred = format!("id IN ({})", in_list.join(", "));
|
||||
let label = format!(
|
||||
"importance={} ({} ids)",
|
||||
*k as f64 / 10.0,
|
||||
chunk.len()
|
||||
);
|
||||
let _rows = commit_update(&tbl, Some(&pred), &lit, &label).await?;
|
||||
}
|
||||
}
|
||||
eprintln!("[backfill] apply done. run --mode verify to check.");
|
||||
Ok(())
|
||||
}
|
||||
other => Err(format!("unknown mode: {}", other).into()),
|
||||
}
|
||||
}
|
||||
|
||||
fn main() {
|
||||
let args = Args::parse();
|
||||
if let Err(e) = rt().block_on(run(args)) {
|
||||
eprintln!("[backfill] FAILED: {}", e);
|
||||
std::process::exit(1);
|
||||
}
|
||||
}
|
||||
+49
-2
@@ -38,6 +38,13 @@ pub struct MemoryRecord {
|
||||
// not_useful_count 保留(deprecated),新代码不再读它;human_not_useful_count 与
|
||||
// dedup_downweight_count 是新写入点。arrow_schema 尾部新增列(nullable,存量补 NULL)。
|
||||
pub human_not_useful_count: i64, pub dedup_downweight_count: i64,
|
||||
// A10 (t_d618dc7c):把「人工有用」从 useful_count(人工+机器混写)拆出单独计数。
|
||||
// Option<i64>(nullable)—— 与 confidence 同款「NULL≠0」语义:存量行/从未被人工
|
||||
// 点过的行保持 NULL(迁移默认 CAST(NULL AS BIGINT)),一旦被人工点过才变真实整数。
|
||||
// 统计口径用本列时分母必须 WHERE human_useful_count IS NOT NULL(NULL 是"未知"非"0 次")。
|
||||
// #[serde(default)] ⇒ IPC 解包缺字段/NULL 读回 None(区别于显式 0),不漏这层即静默 no-op。
|
||||
#[serde(default)]
|
||||
pub human_useful_count: Option<i64>,
|
||||
pub freshness: String, pub version: i64,
|
||||
pub version_history: String, pub source: String,
|
||||
// 2026-10-08 (t_7e3577c5):血缘字段
|
||||
@@ -148,6 +155,10 @@ fn arrow_schema() -> Schema {
|
||||
Field::new("valid_from", DataType::Utf8, true),
|
||||
Field::new("valid_to", DataType::Utf8, true),
|
||||
Field::new("provenance", DataType::Utf8, true),
|
||||
// A10 (t_d618dc7c):人工有用单独计数。nullable Int64 —— 存量行由 open() schema 演进
|
||||
// 迁移补 NULL(CAST(NULL AS BIGINT),与 confidence/human_not_useful_count 迁移模式一致);
|
||||
// 人工点过后才变真实整数。只加观察口径,不进任何排序公式/权重(AC#6 排序零变化)。
|
||||
Field::new("human_useful_count", DataType::Int64, true),
|
||||
])
|
||||
}
|
||||
|
||||
@@ -303,6 +314,8 @@ impl LanceDBOps {
|
||||
// GA-P0-1 (t_718d8707):拆分字段读侧接线
|
||||
human_not_useful_count: col_i64(&batch, i, "human_not_useful_count"),
|
||||
dedup_downweight_count: col_i64(&batch, i, "dedup_downweight_count"),
|
||||
// A10 (t_d618dc7c):读侧接线 —— human_useful_count 走可空读法(NULL 读回 None)
|
||||
human_useful_count: col_i64_opt(&batch, i, "human_useful_count"),
|
||||
freshness: col_str(&batch, i, "freshness"),
|
||||
version: col_i64(&batch, i, "version"),
|
||||
version_history: String::new(),
|
||||
@@ -370,6 +383,8 @@ impl LanceDBOps {
|
||||
// GA-P0-1 (t_718d8707):拆分字段读侧接线
|
||||
human_not_useful_count: col_i64(&batch, i, "human_not_useful_count"),
|
||||
dedup_downweight_count: col_i64(&batch, i, "dedup_downweight_count"),
|
||||
// A10 (t_d618dc7c):读侧接线 —— human_useful_count 走可空读法(NULL 读回 None)
|
||||
human_useful_count: col_i64_opt(&batch, i, "human_useful_count"),
|
||||
freshness: col_str(&batch, i, "freshness"),
|
||||
version: col_i64(&batch, i, "version"),
|
||||
version_history: String::new(),
|
||||
@@ -433,6 +448,8 @@ impl LanceDBOps {
|
||||
// 2026-10-08 (Gap#2):nullable 列用 Option,未声明时写 None → 库里是 NULL
|
||||
let mut confidences: Vec<Option<f64>> = Vec::new();
|
||||
let mut confidence_sources: Vec<&str> = Vec::new();
|
||||
// A10 (t_d618dc7c):人工有用单独计数(nullable,与 confidence 同款「未声明写 NULL」)
|
||||
let mut human_useful_counts: Vec<Option<i64>> = Vec::new();
|
||||
|
||||
for r in &records {
|
||||
ids.push(r["id"].as_str().unwrap_or(""));
|
||||
@@ -455,6 +472,9 @@ impl LanceDBOps {
|
||||
not_useful_counts.push(r["not_useful_count"].as_i64().unwrap_or(0));
|
||||
human_not_useful_counts.push(r["human_not_useful_count"].as_i64().unwrap_or(0)); // GA-P0-1
|
||||
dedup_downweight_counts.push(r["dedup_downweight_count"].as_i64().unwrap_or(0)); // GA-P0-1
|
||||
// A10 (t_d618dc7c):未声明/NULL → None(写 NULL);显式给了整数 → Some(v)。
|
||||
// 与 confidence 同款:库里 NULL = 「从未被人工点过」,非 0。
|
||||
human_useful_counts.push(r["human_useful_count"].as_i64());
|
||||
freshnesses.push(r["freshness"].as_str().unwrap_or("fresh"));
|
||||
versions.push(r["version"].as_i64().unwrap_or(1));
|
||||
version_histories.push(r["version_history"].as_str().unwrap_or("[]"));
|
||||
@@ -540,6 +560,9 @@ impl LanceDBOps {
|
||||
Arc::new(StringArray::from(valid_froms)),
|
||||
Arc::new(StringArray::from(valid_tos)),
|
||||
Arc::new(StringArray::from(provenances)),
|
||||
// A10 (t_d618dc7c):human_useful_count 列(nullable,NULL=从未被人工点过),
|
||||
// 追加在 arrow_schema() 尾字段 human_useful_count 之后,顺序严格一致。
|
||||
Arc::new(Int64Array::from(human_useful_counts)),
|
||||
];
|
||||
|
||||
let batch = RecordBatch::try_new(schema.clone(), columns)?;
|
||||
@@ -584,7 +607,11 @@ impl LanceDBOps {
|
||||
for f in &fields {
|
||||
let col = f["column"].as_str().unwrap_or("");
|
||||
if !col.is_empty() {
|
||||
let val = col_i64(&batch, i, col);
|
||||
// A10 (t_d618dc7c):NULL-aware 自增基数 —— 可空列(human_useful_count)
|
||||
// 的 NULL 行「从未被人工点过」,$inc 时按 0 起算(NULL + 1 ⇒ 1)。
|
||||
// 用 col_i64_opt(NULL→None)而非 col_i64(NULL→物理值 0),
|
||||
// 不依赖 Arrow 物理缓冲约定,语义显式。非空列行为不变。
|
||||
let val = col_i64_opt(&batch, i, col).unwrap_or(0);
|
||||
current_values.insert(col.to_string(), val);
|
||||
}
|
||||
}
|
||||
@@ -681,7 +708,12 @@ impl LanceDBOps {
|
||||
let op = ttbl
|
||||
.update()
|
||||
.only_if(&predicate)
|
||||
.column("recall_seen_count", &format!("recall_seen_count + {}", delta))
|
||||
// C档 (t_5aa7cb77):recall_seen_count 是可空列(存量 43.8% 行为 NULL)。
|
||||
// 裸 `recall_seen_count + delta` 在 NULL 上恒为 NULL ⇒ 观测值静默丢失。
|
||||
// COALESCE(.., 0) 让 NULL 行按 0 起算(NULL+1 ⇒ 1)。不可空列行为不变。
|
||||
// 注意:不得给逐条 fallback 的 value 字符串加 COALESCE —— 那走 current-value
|
||||
// 算术解析(split_whitespace 期望恰好 3 段),COALESCE 会炸掉解析。
|
||||
.column("recall_seen_count", &format!("COALESCE(recall_seen_count, 0) + {}", delta))
|
||||
.column("last_recalled_at", &ts_lit);
|
||||
rt().block_on(op.execute()).map_err(Into::into)
|
||||
};
|
||||
@@ -1031,6 +1063,8 @@ impl LanceDBOps {
|
||||
not_useful_count: col_i64(batch, i, "not_useful_count"),
|
||||
human_not_useful_count: col_i64(batch, i, "human_not_useful_count"),
|
||||
dedup_downweight_count: col_i64(batch, i, "dedup_downweight_count"),
|
||||
// A10 (t_d618dc7c):读侧接线 —— human_useful_count 走可空读法(NULL 读回 None)
|
||||
human_useful_count: col_i64_opt(batch, i, "human_useful_count"),
|
||||
freshness: col_str(batch, i, "freshness"),
|
||||
version: col_i64(batch, i, "version"),
|
||||
version_history: String::new(),
|
||||
@@ -1071,6 +1105,7 @@ impl LanceDBOps {
|
||||
"source", "volatile_flag", "is_deleted",
|
||||
// GA-P0-1 (t_718d8707):scan_all 投影裁剪,漏加 ⇒ col_i64 取不到列 ⇒ 恒 0
|
||||
"human_not_useful_count", "dedup_downweight_count",
|
||||
"human_useful_count",
|
||||
// GA-P0-2 (t_c6d41ac6):拆分字段投影,漏加 ⇒ col_i64 取不到列 ⇒ 恒 0
|
||||
"recall_seen_count", "recall_verified_count",
|
||||
"last_recalled_at", "created_at", "updated_at",
|
||||
@@ -1108,6 +1143,8 @@ impl LanceDBOps {
|
||||
// GA-P0-1 (t_718d8707):拆分字段读侧接线
|
||||
human_not_useful_count: col_i64(&batch, i, "human_not_useful_count"),
|
||||
dedup_downweight_count: col_i64(&batch, i, "dedup_downweight_count"),
|
||||
// A10 (t_d618dc7c):读侧接线 —— human_useful_count 走可空读法(NULL 读回 None)
|
||||
human_useful_count: col_i64_opt(&batch, i, "human_useful_count"),
|
||||
freshness: col_str(&batch, i, "freshness"),
|
||||
version: col_i64(&batch, i, "version"),
|
||||
version_history: String::new(),
|
||||
@@ -1156,6 +1193,7 @@ impl LanceDBOps {
|
||||
"useful_count", "not_useful_count", "freshness", "version",
|
||||
"source", "volatile_flag", "is_deleted",
|
||||
"human_not_useful_count", "dedup_downweight_count",
|
||||
"human_useful_count",
|
||||
"recall_seen_count", "recall_verified_count",
|
||||
"delete_reason", // GA-P0-3
|
||||
"last_recalled_at", "created_at", "updated_at",
|
||||
@@ -1190,6 +1228,8 @@ impl LanceDBOps {
|
||||
not_useful_count: col_i64(&batch, i, "not_useful_count"),
|
||||
human_not_useful_count: col_i64(&batch, i, "human_not_useful_count"),
|
||||
dedup_downweight_count: col_i64(&batch, i, "dedup_downweight_count"),
|
||||
// A10 (t_d618dc7c):读侧接线 —— human_useful_count 走可空读法(NULL 读回 None)
|
||||
human_useful_count: col_i64_opt(&batch, i, "human_useful_count"),
|
||||
freshness: col_str(&batch, i, "freshness"),
|
||||
version: col_i64(&batch, i, "version"),
|
||||
version_history: String::new(),
|
||||
@@ -1341,6 +1381,13 @@ fn col_f64_opt(b: &RecordBatch, r: usize, c: &str) -> Option<f64> {
|
||||
.and_then(|col| col.as_any().downcast_ref::<Float64Array>())
|
||||
.and_then(|a| if a.is_null(r) { None } else { Some(a.value(r)) })
|
||||
}
|
||||
/// col_i64_opt (t_d618dc7c):可空 Int64 读法。列不存在或该行是 NULL 都返回 None(区别于 0)。
|
||||
/// 用于 human_useful_count —— 存量行/从未被人工点过的是 NULL,要读回 None("未知")而非 0。
|
||||
fn col_i64_opt(b: &RecordBatch, r: usize, c: &str) -> Option<i64> {
|
||||
b.column_by_name(c)
|
||||
.and_then(|col| col.as_any().downcast_ref::<Int64Array>())
|
||||
.and_then(|a| if a.is_null(r) { None } else { Some(a.value(r)) })
|
||||
}
|
||||
fn col_i64(b: &RecordBatch, r: usize, c: &str) -> i64 {
|
||||
b.column_by_name(c)
|
||||
.and_then(|col| col.as_any().downcast_ref::<Int64Array>())
|
||||
|
||||
+97
-3
@@ -154,6 +154,65 @@ impl HeavyReadGate {
|
||||
}
|
||||
}
|
||||
|
||||
// ─── 连接并发上限 + RSS 护栏(t_20bfbd9a 治本)────────────────────────
|
||||
// 事故链(t_20bfbd9a 收口):accept 循环对每个 TCP 连接 thread::spawn,无并发上限;
|
||||
// Go 侧每 rpc 新 dial(lancedb_ipc.go rpc() → net.DialTimeout),IPC 洪泛 → 线程
|
||||
// 线性涨到 736(实测),每线程持独立 stream + LanceDB 批次内存 ⇒ 5.1G 单进程。
|
||||
// 治本①连接并发上限:AtomicUsize in_flight 计数,spawn 前 CAS +1,超上限 ⇒ 拒绝
|
||||
// (回 heavy_conn_busy);线程结束 RAII Drop 归还。轻热路径(ping/insert/update)
|
||||
// 在正常流量下永不超过上限,不改变单写者语义。
|
||||
// 治本②RSS 护栏:每 accept 采样 VmRSS,超上限 ⇒ 降级到串行(并发 1,inline 处理
|
||||
// + 拒重读),不再叠加线程;低于上限自动恢复。上限 env ZHIYI_RSS_LIMIT_MB 可调。
|
||||
static CONN_MAX_DEFAULT: usize = 64;
|
||||
static RSS_LIMIT_MB_DEFAULT: u64 = 3000;
|
||||
|
||||
fn conn_permits() -> usize {
|
||||
match std::env::var("ZHIYI_CONN_MAX") {
|
||||
Ok(v) => v.trim().parse::<usize>().unwrap_or(CONN_MAX_DEFAULT).max(1),
|
||||
Err(_) => CONN_MAX_DEFAULT,
|
||||
}
|
||||
}
|
||||
|
||||
fn rss_limit_kb() -> u64 {
|
||||
match std::env::var("ZHIYI_RSS_LIMIT_MB") {
|
||||
Ok(v) => v.trim().parse::<u64>().unwrap_or(RSS_LIMIT_MB_DEFAULT).max(64) * 1024,
|
||||
Err(_) => RSS_LIMIT_MB_DEFAULT * 1024,
|
||||
}
|
||||
}
|
||||
|
||||
/// 连接许可 RAII 守卫:Drop 自动归还 in_flight(线程正常返回或 panic 都不泄漏)。
|
||||
struct ConnPermit {
|
||||
counter: Arc<AtomicUsize>,
|
||||
}
|
||||
impl Drop for ConnPermit {
|
||||
fn drop(&mut self) {
|
||||
self.counter.fetch_sub(1, Ordering::SeqCst);
|
||||
}
|
||||
}
|
||||
|
||||
/// 尝试登记一个并发连接;有槽 ⇒ Some(许可),满 ⇒ None(调用方拒绝/降级)。
|
||||
fn conn_try_acquire(counter: &Arc<AtomicUsize>, limit: usize) -> Option<ConnPermit> {
|
||||
let mut cur = counter.load(Ordering::SeqCst);
|
||||
loop {
|
||||
if cur >= limit {
|
||||
return None;
|
||||
}
|
||||
match counter.compare_exchange_weak(
|
||||
cur,
|
||||
cur + 1,
|
||||
Ordering::SeqCst,
|
||||
Ordering::SeqCst,
|
||||
) {
|
||||
Ok(_) => {
|
||||
return Some(ConnPermit {
|
||||
counter: Arc::clone(counter),
|
||||
})
|
||||
}
|
||||
Err(actual) => cur = actual,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ─── 内存回收 & 观测(2026-09-21 治本)────────────────────────────────
|
||||
// 根因:长驻 sidecar 里每次操作的大块分配(Arrow/LanceDB 批次、Vec<MemoryRecord>)
|
||||
// 释放后 glibc 不还给 OS(arena 保留)→ RSS 只涨不落,5 小时累积到 1.09G
|
||||
@@ -359,9 +418,12 @@ fn run_socket_server(socket_path: &str, args: &Args) {
|
||||
encoder,
|
||||
heavy_read: HeavyReadGate::new(n),
|
||||
});
|
||||
let conn_limit = conn_permits();
|
||||
let conn_in_flight = Arc::new(AtomicUsize::new(0));
|
||||
eprintln!(
|
||||
"[zhiyi] listening on {} (lancedb: {}), heavy-read admission N={} (env ZHIYI_HEAVY_READ_CONCURRENCY)",
|
||||
socket_path, args.data_dir, n
|
||||
"[zhiyi] listening on {} (lancedb: {}), heavy-read admission N={} (env ZHIYI_HEAVY_READ_CONCURRENCY), \
|
||||
conn cap={} (env ZHIYI_CONN_MAX), rss limit={}MB (env ZHIYI_RSS_LIMIT_MB)",
|
||||
socket_path, args.data_dir, n, conn_limit, rss_limit_kb() / 1024
|
||||
);
|
||||
|
||||
for stream in listener.incoming() {
|
||||
@@ -370,7 +432,39 @@ fn run_socket_server(socket_path: &str, args: &Args) {
|
||||
let a = args.clone();
|
||||
let ldb = lancedb.clone();
|
||||
let st = Arc::clone(&state);
|
||||
thread::spawn(move || handle_client(stream, &a, ldb, &st));
|
||||
// RSS 护栏(治本②):超上限 ⇒ 降级到串行(并发 1,inline 处理 + 拒重读),
|
||||
// 不再叠加线程;低于上限自动恢复。这是对「无内存护栏」事故的直接兜底。
|
||||
let (rss, _) = rss_kb();
|
||||
if rss > rss_limit_kb() {
|
||||
eprintln!(
|
||||
"[zhiyi] RSS {}MB > limit {}MB: degrade to serial (concurrency 1)",
|
||||
rss / 1024,
|
||||
rss_limit_kb() / 1024
|
||||
);
|
||||
handle_client(stream, &a, ldb, &st);
|
||||
continue;
|
||||
}
|
||||
// 连接并发上限(治本①):CAS +1,满 ⇒ 回 heavy_conn_busy 并拒绝。
|
||||
let permit = match conn_try_acquire(&conn_in_flight, conn_limit) {
|
||||
Some(p) => p,
|
||||
None => {
|
||||
eprintln!(
|
||||
"[zhiyi] connection cap {} reached: reject (heavy_conn_busy)",
|
||||
conn_limit
|
||||
);
|
||||
// 直接回一个 JSON 错误给客户端(对齐 Go rpc() 的读 deadline)
|
||||
let mut stream = stream;
|
||||
let _ = stream.write_all(b"\x00\x00\x00\x46");
|
||||
let _ = stream.write_all(
|
||||
br#"{"status":"error","error":"heavy_conn_busy","detail":"connection cap exhausted"}"#,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
};
|
||||
thread::spawn(move || {
|
||||
handle_client(stream, &a, ldb, &st);
|
||||
drop(permit); // RAII 归还 in_flight(也兜 panic 展开)
|
||||
});
|
||||
}
|
||||
Err(e) => eprintln!("[zhiyi] accept error: {}", e),
|
||||
}
|
||||
|
||||
@@ -4,4 +4,13 @@
|
||||
# 追加方式:评审通过后 `./scripts/deploy.sh --mark-approved`(自动把当前 HEAD 追加进来)。
|
||||
# 切勿手工删行——删除会让被撤批的提交重新进入「夹带」判定。
|
||||
# 初始基线(2026-10-17 t_3f83df64 建):600fb2f = 当前已评审的 t_690cc333 HEAD。
|
||||
600fb2fed60f6ee2ea31ec6c41a5ee4e4e08a386
|
||||
600fb2fed60f6ee2ea31ec6c41a5ee4e4e08a386
|
||||
b8f7ddb53ce056a1ac86468a8e9a7557c71922f0
|
||||
c57f557ab8d126c7ef031e63402abaa1f6602155
|
||||
aaeaf2b7ec9304b851089cbe64ddda0a5916d99f
|
||||
a2be088cba59c95901b22d855dea9b6df3bf3a80
|
||||
4fb4705e6de8991e9863cf8902e4265ec412626e
|
||||
212624a42541c41f10168e5a3834bd2304efcdb9
|
||||
4341fac4c28a4f2825f7373ea0ec94d07c6452f0
|
||||
b101b464471f6283c438f089e9ec269c8f2b1644
|
||||
01d8f34
|
||||
|
||||
@@ -0,0 +1,200 @@
|
||||
#!/usr/bin/env python3
|
||||
"""织忆自检·全量取数器(唯一允许直连生产库的脚本)
|
||||
|
||||
为什么存在
|
||||
----------
|
||||
织忆相关的定时自检脚本曾有 ~20 个,其中两个直接 `lancedb.connect("/var/lib/memoryweave")`
|
||||
绕过服务读全表。那种写法的病根不是"直连",而是**取数逻辑散落多处、与 schema 一起腐烂**:
|
||||
pattern 自检写死 `type='pattern'`(库里其实是 `pattern_stat`)⇒ 连续数月静默 0 输出。
|
||||
|
||||
本文件把"全量取数"收成一处、放进仓库:改 schema 时同一个 commit 就能改它,且它有输出可复核。
|
||||
判定/告警/投递**不在这里**(留在外部 cron)—— 观测者不能住在被观测者体内。
|
||||
|
||||
它产出 `~/.hermes/state/zhiyi-quality.json`,服务端 `GET /api/v1/selfcheck?level=full` 读这份快照。
|
||||
|
||||
口径(必须与历史 zhiyi-quality-watch.py 一致,改了就改告警语义)
|
||||
------------------------------------------------------------
|
||||
- 碎片:`len(content.strip()) < FRAG_LEN(20)`,按**字符数**(不是字节,中文按 1 算)
|
||||
- 冗余:live 行内 content 完全相同;冗余行数 = Σ(组内行数-1)
|
||||
- 近 7 天:created_at 在 7 天内(live)
|
||||
- 写量:按 created_at 的**本地日期**分桶,同时给 rows(含已删)与 live(仅存活)
|
||||
|
||||
性能
|
||||
----
|
||||
只投影 3 列(content/created_at/is_deleted),不读 1024 维向量:
|
||||
4 万行 0.06s(旧实现 `to_arrow()` 全列要数秒且吃 1G+ 内存)。
|
||||
|
||||
用法
|
||||
----
|
||||
python3 tools/selfcheck_full.py [--out PATH] [--json] [--days 60]
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import datetime as dt
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
from collections import Counter
|
||||
|
||||
DB_PATH = os.environ.get("ZHIYI_DB", "/var/lib/memoryweave")
|
||||
FRAG_LEN = 20
|
||||
OUT_DEFAULT = os.path.expanduser("~/.hermes/state/zhiyi-quality.json")
|
||||
SCHEMA = 1
|
||||
|
||||
|
||||
def _reexec_with_lancedb() -> None:
|
||||
"""cron 默认解释器没有 lancedb 时,切到 lance-compact venv 再跑(一次性)。"""
|
||||
try:
|
||||
import lancedb # noqa: F401
|
||||
return
|
||||
except ImportError:
|
||||
pass
|
||||
venv_py = os.path.expanduser("~/.hermes/venvs/lance-compact/bin/python")
|
||||
if os.path.exists(venv_py) and os.path.realpath(sys.prefix) != os.path.realpath(
|
||||
os.path.dirname(os.path.dirname(venv_py))
|
||||
):
|
||||
os.execv(venv_py, [venv_py] + sys.argv)
|
||||
sys.stderr.write("[selfcheck_full] lancedb 不可用(需 lance-compact venv)\n")
|
||||
sys.exit(2)
|
||||
|
||||
|
||||
def _as_bool(v) -> bool:
|
||||
if v is None:
|
||||
return False
|
||||
if isinstance(v, bool):
|
||||
return v
|
||||
return str(v).strip().lower() in ("true", "1")
|
||||
|
||||
|
||||
def collect(days: int = 60) -> dict:
|
||||
import lancedb
|
||||
|
||||
t0 = time.time()
|
||||
db = lancedb.connect(DB_PATH)
|
||||
table = db.open_table("memories")
|
||||
tbl = table.to_lance().to_table(columns=["content", "created_at", "is_deleted"])
|
||||
|
||||
contents = tbl.column("content").to_pylist()
|
||||
created = tbl.column("created_at").to_pylist()
|
||||
deleted = tbl.column("is_deleted").to_pylist()
|
||||
|
||||
total = len(contents)
|
||||
frag_all = 0
|
||||
live = 0
|
||||
frag_live = 0
|
||||
dup_counter: Counter[str] = Counter()
|
||||
daily: dict[str, dict[str, int]] = {}
|
||||
now = dt.datetime.now()
|
||||
week_ago = now - dt.timedelta(days=7)
|
||||
new_7d = 0
|
||||
new_frag_7d = 0
|
||||
|
||||
for i in range(total):
|
||||
raw = contents[i]
|
||||
text = str(raw).strip() if raw is not None else ""
|
||||
n = len(text)
|
||||
is_frag = n < FRAG_LEN
|
||||
if is_frag:
|
||||
frag_all += 1
|
||||
|
||||
is_del = _as_bool(deleted[i] if i < len(deleted) else False)
|
||||
|
||||
# created_at:存量行有空串/畸形 → 归 unknown,不炸整批
|
||||
day = "unknown"
|
||||
ts = None
|
||||
if i < len(created):
|
||||
raw_ts = created[i]
|
||||
if raw_ts:
|
||||
try:
|
||||
if isinstance(raw_ts, dt.datetime):
|
||||
ts = raw_ts
|
||||
else:
|
||||
ts = dt.datetime.fromisoformat(str(raw_ts).replace("Z", "+00:00"))
|
||||
if ts.tzinfo is not None:
|
||||
ts = ts.astimezone().replace(tzinfo=None)
|
||||
day = ts.strftime("%Y-%m-%d")
|
||||
except Exception:
|
||||
ts = None
|
||||
|
||||
slot = daily.setdefault(day, {"rows": 0, "live": 0})
|
||||
slot["rows"] += 1
|
||||
|
||||
if is_del:
|
||||
continue
|
||||
live += 1
|
||||
slot["live"] += 1
|
||||
dup_counter[text] += 1
|
||||
if is_frag:
|
||||
frag_live += 1
|
||||
if ts is not None and ts >= week_ago:
|
||||
new_7d += 1
|
||||
if is_frag:
|
||||
new_frag_7d += 1
|
||||
|
||||
dup_groups = sum(1 for c in dup_counter.values() if c > 1)
|
||||
dup_extra = sum(c - 1 for c in dup_counter.values() if c > 1)
|
||||
|
||||
# 只保留最近 days 天(含 unknown),按日期升序
|
||||
keep = sorted(daily.keys())[-days:]
|
||||
day_rows = [{"date": d, "rows": daily[d]["rows"], "live": daily[d]["live"]} for d in keep]
|
||||
|
||||
live_days = [x["live"] for x in day_rows if x["date"] != "unknown"]
|
||||
today = now.strftime("%Y-%m-%d")
|
||||
today_live = next((x["live"] for x in day_rows if x["date"] == today), 0)
|
||||
|
||||
return {
|
||||
"schema_version": SCHEMA,
|
||||
"generated_at": now.strftime("%Y-%m-%dT%H:%M:%S%z"),
|
||||
"source": {"db": DB_PATH, "table": "memories", "mode": "read_only_projection"},
|
||||
"duration_ms": int((time.time() - t0) * 1000),
|
||||
"quality": {
|
||||
"frag_len": FRAG_LEN,
|
||||
"total_rows": total,
|
||||
"live_rows": live,
|
||||
"frag_rows_all": frag_all,
|
||||
"frag_rate_all": round(frag_all / total, 4) if total else None,
|
||||
"frag_rows_live": frag_live,
|
||||
"frag_rate_live": round(frag_live / live, 4) if live else None,
|
||||
"dup_groups": dup_groups,
|
||||
"dup_extra_rows": dup_extra,
|
||||
"dup_rate": round(dup_extra / live, 4) if live else None,
|
||||
"new_rows_7d": new_7d,
|
||||
"new_frag_rows_7d": new_frag_7d,
|
||||
"new_frag_rate_7d": round(new_frag_7d / new_7d, 4) if new_7d else None,
|
||||
},
|
||||
"write_volume": {
|
||||
"days": day_rows,
|
||||
"today": today_live,
|
||||
"today_date": today,
|
||||
# 近 7 个完整日(不含今天/unknown)
|
||||
"last_7d": sum(x["live"] for x in day_rows if x["date"] not in (today, "unknown")) if day_rows else 0,
|
||||
"nonzero_days": sum(1 for x in live_days if x > 0),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description="织忆自检全量取数(只读)")
|
||||
ap.add_argument("--out", default=OUT_DEFAULT)
|
||||
ap.add_argument("--json", action="store_true", help="打印到 stdout")
|
||||
ap.add_argument("--days", type=int, default=60)
|
||||
a = ap.parse_args()
|
||||
|
||||
_reexec_with_lancedb()
|
||||
snap = collect(a.days)
|
||||
|
||||
if a.out:
|
||||
os.makedirs(os.path.dirname(a.out), exist_ok=True)
|
||||
tmp = a.out + ".tmp"
|
||||
with open(tmp, "w", encoding="utf-8") as f:
|
||||
json.dump(snap, f, ensure_ascii=False)
|
||||
os.replace(tmp, a.out) # 原子替换:读方永不会看到半个文件
|
||||
if a.json:
|
||||
print(json.dumps(snap, ensure_ascii=False, indent=1))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in New Issue
Block a user