Commit Graph
19 Commits
Author SHA1 Message Date
小唯 1060e62066 fix(admin): /memories 与 /memories/export 改用全表扫描(Scan),弃零向量 ANN
病根:两个列表/导出端点一直用『零向量 ANN』冒充『列出全部』——零向量没有方向,
ANN 退化成本存储序前 N 行,再按 ns 客户端过滤 ⇒ 计数完全失真
(实测 export hermes-main 报 1 条,真实 139+;shared 报 0,真实 29)。

修复:
① Rust lancedb_ops: scan_for_forgetting 加 namespace 下推(only_if),limit 作用于过滤后结果集
② Go 接口 LanceDB 新增 Scan(limit, ns):
   - RustLanceDBClient → lancedb_scan RPC
   - SQLiteClient → SELECT ... WHERE is_deleted=0 [AND ns] ORDER BY created_at DESC LIMIT
   - MemLanceClient → 内存 map 过滤 + created_at 倒序
③ 两个端点 ListMemories / ExportMemoriesMD 改用 aa.LanceDB.Scan(limit, ns)
④ 补 mock(zz_commit mockLance / write-buffer fakeLDB)+ 5 条新判据
   (全表 3 / hermes-main 2 / shared 1 / limit 生效 / 不存在 ns 0)
2026-09-21 16:36:41 +08:00
小唯 266a85a5d2 fix(recall): 默认调用改全库跨命名空间召回(t_24eceed0 返修)
病根:纺忆写入侧按 agent/域把记忆分到 19 个命名空间(实测 300 采样:
hermes-main 139、wiki-curator-main 35、wiki 30、shared 29、soulful-xiaowei 27、
a06-main 11、domain:* 13…),而默认调用(无 namespace / 无 agent_id)
被推导成 default-main ⇒ 只搜 default-main+shared,全表 10172 条实测仅得 11 个候选,
99.9% 记忆在默认召回的盲区(t_24eceed0 worker 交付只改了推导来源,未解决跨 ns)。

修复:
① api/routes: 新增 resolveRecallNamespace() 纯函数 —— 显式 ns 原样、只有 agent_id 才推导、
   两者都空返回 "" 表示全库;Recall 与 RecallDebug 共用同一函数(debug 如实反映 recall)
② storage/recall.go: namespace=="" → Search(..., 200, "") 全库单次检索(空过滤=不过滤),
   候选池上限 50→100;显式 ns 路径完全不变([ns, shared])
③ 全库模式下关闭两段补丁式补搜(shared / wiki-curator-main),避免重复候选与白花 embedding

测试(8 条,含反证):
- routes: resolveRecallNamespace 5 用例(核心判据:空+空 必须返回 "")
- storage: 全库单次检索(记录 Search 实参)/ 显式 ns 保持 [ns shared] / shared 不重复搜
- 反证:还原旧行为 → 两组量具均 FAIL ✓("默认调用被解析成 hermes-main,又只能看到极小一片记忆")
踩坑记录:stub 用嵌入接口技巧时须一并覆写 Update(),否则后台 recall_count 回写 goroutine
落到 nil 接口 panic。
2026-09-21 15:15:07 +08:00
小唯 b5f59703ab fix(storage): check error returns from MemLanceClient in SQLiteMemClient
SQLiteMemClient.IncrementUseful/IncrementNotUseful were discarding the
error return of MemLanceClient.IncrementUseful/IncrementNotUseful.
Since the latter now return real errors (6e67f3e), dropping them
re-introduces the silent-failure class of bug we just fixed across
the Rust path. Also propagate persistMemory errors.
2026-09-20 11:39:30 +08:00
小唯 6e67f3e33f fix(storage): fix all no-op/fake implementations in LanceDB interfaces
- IncrementUseful/IncrementNotUseful now return error and persist via
  rc.Update() (Rust sidecar); all 4 implementations updated
- SoftDeleteBatch now delegates to SoftDelete() for real LanceDB write
- UpdateMemoryContent now persists content + version + version history
- Backup changed from return nil to explicit error (Rust sidecar unsupported)
- IncrementUseful/IncrementNotUseful signatures now include error return;
  all 3 call sites (feedback.go, core.go) check the error
- Test mocks (mockLance, fakeLDB) updated to match new interface

Fixes the 'silent no-op' class of bugs discovered during dedup repair.
2026-09-20 09:53:00 +08:00
小唯 fd699151a8 fix(dedup): use SoftDelete path and flush buffer before responding
Two write-path fixes found while executing the dedup policy.

1. The endpoint wrote is_deleted via Update(). The system's canonical
   path is SoftDelete(), which routes through SoftDeleteWriteBuffer
   (5-minute window / 256-item threshold, single-transaction commit).
   Update() writes one MVCC version per row, so this endpoint would
   have created 411 versions in one run - the exact growth pattern the
   vacuum script was written to clean up (table measured at 3.23G
   before a 40M compaction).

2. The response was sent while soft deletes were still buffered. During
   execution lance-vacuum (daily 03:30) stopped the service and ran
   optimize(cleanup_older_than=0), discarding the unflushed writes: the
   endpoint's deleted=34 never landed and is_deleted read back all
   false, while the caller only saw curl exit=52 for the empty reply.
   A new storage.FlushSoftDeleteBuffer() is now called before the
   response, so reported counts equal what actually landed.

Demotions were unaffected in that incident because IncrementNotUseful
writes immediately (297 of 377 landed); they are now also correct
additive increments thanks to the earlier col+1 fix.
2026-09-20 03:34:59 +08:00
小唯 dd25457926 fix(demote): IncrementNotUseful was a no-op; implement A-policy dedup
Two fixes, both silent-failure class.

1. RustLanceDBClient.IncrementNotUseful was an empty function body.
   Every demotion path was a false success: the HTTP endpoint returned
   200, zhiyi-dedup.py reported '34 demoted, 0 failed', and
   not_useful_count stayed 0 on all 32 dropped rows. Now delegates to
   Update with $inc.

2. Update emulated $inc in Go using the _local.memories cache, which is
   only populated on the Insert path (Search does not fill it). On a
   cache miss getIncCachedValue returned 1, so an increment wrote the
   absolute value 1 - resetting an existing count of e.g. 5 back to 1.
   Now the column is sent to Rust as the expression 'col + 1'; Rust's
   update() detects the arithmetic, reads the row's true current value
   and computes the result, so the increment is genuinely additive.

Also implements 牧尘's chosen dedup policy (option A) in admin/dedup:
keep the NEWEST entry per cluster, then for every other member
  - if it normalises to the same text (punctuation/whitespace/quote
    differences only) it is a true duplicate -> soft delete
  - otherwise it carries different information (different numbers,
    subject, timestamps, versions) -> demote only, never delete
This satisfies 'newest wins' without discarding distinct facts.
2026-09-20 02:55:05 +08:00
小唯 ec17c836a1 fix(dedup): parse vector field in Rust IPC Search results
RustLanceDBClient.Search decoded the Rust IPC response into a struct
that had no vector field, so every MemoryRecord returned to Go had an
empty Vector. Consequence chain:

  len(results[i].Vector) > 0 && len(results[j].Vector) > 0
always false → the cosine branch was skipped in:
  - commit path near_dup (core.go:634)
  - distill direct-write dedup gate (server.go:179)  <-- the fix I just
    deployed this round was silently a no-op
  - admin/dedup endpoint (core.go:634)                <-- merged≈0 false
    negative

Without this, no near-duplicate detection in the whole system works —
it silently fell back to exact string comparison only.

Retained: the vector is forwarded from the raw IPC JSON; if Rust stops
returning it, the degenerate behaviour is exact-string comparison
(which still works), rather than a crash.
2026-09-20 01:07:27 +08:00
小唯 962489f20c fix: Search must forward namespaceFilter to Rust IPC (was hardcoded empty) 2026-09-19 22:16:14 +08:00
小唯 dbe6c278c5 deep-think integration: conflict-detector fix, GraphRAG recall fusion, Louvain community discovery, deep-think endpoint 2026-09-19 19:12:49 +08:00
小唯 ee50ef0d59 fix(rust): 恢复缺失的 LanceDBStats/stats() 和 helper 函数 (t_3fb41049); storage: 添加软删除累积延迟缓冲 (t_384f785b) 2026-09-14 22:14:52 +08:00
小唯 68ce2c9678 feat(distill): 限制 prompt ≤8k token 保护本地 2B 单槽; storage: 添加 SoftDeleteBatch 及 Rust 端批量软删除优化 (t_3fb41049) 2026-09-14 22:09:44 +08:00
小唯 41230ccb0a docs+test(recall): 修正批量更新注释(delta 分组,非 CASE WHEN)+ 补 IPC 只读校验用例 (t_8496e8b6)
- recall_write_buffer.go / lancedb_ipc.go: 注释与实现对齐 —— lance 不支持 CASE WHEN,
  Rust 侧按 delta 分组、每组一次 update 提交;整批版本数 = 不同 delta 的个数(通常 1 个)
- batch_ipc_integration_test.go: +TestBatchIPCScanReadOnly(只读扫 socket 校验 recall_count 真落盘)

实测(临时 sidecar,/tmp/mw-verify-180517):
  VERSION_ACCOUNTING: 批量3条 → 1 个版本 | 逐条3次 → 3 个版本
  BATCH_IPC_OK id=ep_1786982837511335732 recall_count 0 → 3
  SCAN_READONLY: 总 1814 条, recall_count>0 的 742 条, 最大 828
2026-09-12 18:07:30 +08:00
小怡 f2362150d1 perf(recall): 累积延迟更新治理写放大 — 读路径零逐条写 + 单事务批量落盘 (t_8496e8b6)
根因: recall 读路径对每条结果同步 lancedb.Update,LanceDB MVCC 每次提交=1 版本
      实测 15 版本/分钟 / _versions 17.7G(真实数据 22M)

改动:
- Go: 新增 RecallWriteBuffer(窗口合并+阈值触发+优雅退出落盘);recall.go 读路径改 Record();RustLanceDBClient.UpdateRecallBatch
- Rust: lancedb_update_batch IPC — 按 delta 分组,每组一次 update(`recall_count + delta` + id IN (...)) 提交
- 单测 6 个 + IPC 端到端版本计数测试;bench_test.go 修 NewReranker 签名失配(阻塞包测试)

实测: 批量 3 条 → 1 个版本;逐条 3 次 → 3 个版本 (lance 不支持 CASE WHEN,已按 delta 分组规避)
2026-09-12 17:57:31 +08:00
小唯 310a98da4e fix(graph-cleanup): CleanupNoiseNodes NULL-id panic + 新增 scoped 卫安/A03/A04 测试污染清理端点 (t_1dfaaa4a)
- graph_sqlite.go CleanupNoiseNodes: row[id] 为 NULL(daemon-distill pattern 模板行)时直接
  .(string) panic → HTTP 连接被关(HTTP=000);改为 nil 安全跳过
- 新增 CleanupScopedNodes(dryRun, namespaces, nameContains):namespace+名称子串精确圈定,
  清理 2026-06 多 agent 测试污染(openclaw-main/a06-main/hermes-main 的 A03/A04/卫安 节点)
- graph_cache.go: cachedGraphStore 透传(inner 可选接口断言,非 SQLite 后端静默 0)
- server.go: GET/POST /api/v1/graph/cleanup/scoped?dry_run=&namespace=&name=
  实际执行后 InvalidateAll 保证 graph_cache 一致性 —— 禁外部 SQL 的官方通道
2026-09-08 09:02:35 +08:00
xiaowei 4a245b6262 fix(forgetting): P2 候选源升级 — Rust scan 全表替代纯缓存遍历
GetCandidatesForForgetting 原实现只遍历进程内 _local.memories 缓存
(commit/recall/search 才触碰), 重启后缓存空 → decay scanned=3 扫不全,
9-25 到期记忆若不在缓存则永远无法被遗忘(闭环断)。

安全版 scan_for_forgetting (防 febc2c9 风暴复发):
- 强制 limit (IPC 钳制硬上限 5000, decay 侧 2000)
- 不读 vector 列 (1024维×3000条≈12MB+, IPC JSON 会卡)
- 真读 last_recalled_at 列 (scan_all 旧实现 String::new() 恒空 → 遗忘判定失真)
- 消费方截断护栏已就位 (merge 桶500 / conflict 500 / sanity 1000)
- IPC 失败/空 → fallback 缓存遍历 (保持可用)

验证: [decay] scanned=3 → 2000 (重启即扫全表), CPU 0%, 无 panic
2026-09-06 02:32:58 +08:00
xiaowei d4343b1c47 fix(recall): Embedder http client 8s 超时 + decay 分批 AC-1 日志
- Embedder http.Client{} 无超时 → 远端 bge 半开 TCP 时 recall 永久挂起
  (gateway 10s 断 → 织忆异常误报)。加 8s 超时快速失败降级。
- decay 每 tick 限量 2000 条(最久未访问优先) + scanned/forgotten 日志
  (AC-1 验收证据)
2026-09-06 01:08:42 +08:00
xiaowei e6b1e0417c fix(forgetting): P1 防风暴 - 限长/分桶/抽样下推/decay分批
febc2c9 全表扫描 O(n²) 风暴的根治防护(生产基线 ba44e20+ 之上,不含 febc2c9 扫描代码):

1. mergeSimilar 按 category 分桶 + 单桶 maxMergePerBucket=500 截断
   - 原双层循环全量 O(n²),全表时触发 620% CPU
   - 分桶后 O(Σ桶²) << O(n²),语义不变(原 catI!=catJ continue 等价分桶)
2. scanConflicts 显式 created_at 倒序取最近 500 条
   - 原注释'最近100条'实际全量 O(n·50)
3. checkTimestampSanity 先截断 maxSanitySample=1000 再均匀抽样
   - 原全量拉取仅为 5% 抽样
4. decay 每 tick 限量 maxDecayBatch=2000,按最久未访问优先
5. 存储层输出契约统一 RFC3339 string + 补 category/created_at
   - lancedb_ipc: last_recalled_at 此前直接放 time.Time → server.go .(string)
     断言恒失败 → lastAccessed 永远默认30天前 → 遗忘判定失真(本 bug + 数据
     年龄未到 = 遗忘从不触发的双根因之一)
   - sqlite: 补 category/created_at + LIMIT 100→200(对齐 lancedb 契约)
   - 新增 memoryTimeVal/fmtTimeRFC3339 helper 兼容多类型

测试: go build ./... 全绿;gofmt 仅历史文件遗留(未全文件重排,保持最小 diff)
2026-09-06 00:56:06 +08:00
小唯 736d407e68 fix(softdelete): 单独应用 SoftDelete 持久化修复(ba44e20+SoftDelete only, 不含 lancedb_scan 风暴代码)
从 febc2c9 提取仅 SoftDelete 部分: updateField 提升包级 + SoftDelete 调
lancedb_update 持久化 is_deleted=true。跳过 GetCandidatesForForgetting 全表
扫描改动(有 620% CPU 风暴 bug, 待修复后另部署)。

验证: DELETE scratch2-1788616361067 → [ipc]SoftDelete persisted → 查询0条
→ 重启后仍0条(持久化成功)。
2026-09-06 00:23:23 +08:00
xiaowei 2a350cf781 fix(distill): max_tokens 800→1200 — Agnes推理token预算不足
Agnes 2.0-flash是推理模型,推理token占大量预算。
800仍不够→JSON截断(parse error: unexpected end of JSON input)。
1200确保推理+正文都够用。
2026-08-20 01:13:06 +08:00