Commit Graph
69 Commits
Author SHA1 Message Date
小唯 ffaa3de14d test(eval): CALIBRATED 结构化记录 + item4 已知漂移文档化 (t_a70b07e1)
- eval_test.go: TestEvalGolden_Live 逐条输出 CALIBRATED JSON 记录(i/query/top1/ids/kw_hit/id_hit/error),
  供统一口径分类器(classify.py)复算 go/py 两量具分类差异。反证测试注释补 N=5 口径说明。
- golden-set.json: description 补 item4 已知漂移说明(基线 mem_1789837840173161705 漂移,
  现稳定 top-5 成员 mem_1789834428978989104;两量具对该条可能差一级,待人工校准后 diff=0)。
2026-09-21 21:40:24 +08:00
小唯 5250eb1c2e fix(eval+recall): 统一金标判据口径 N=5 + candidates_scanned 语义修订 (t_a70b07e1)
AC1: 两量具统一为「取 recall 返回前 N=5 条查关键词」——Go 量具(eval_test.go)原本就查 top-5,Python 量具(golden-check.py)原本只查 top-1,同批 14 条金标出现矛盾分类。N=5 写入 eval_test.go 注释 + golden-set.json 说明字段 + golden-check.py 头注释三方一致。

AC2: candidates_scanned 不再是 len(results) 的代理值——语义修订为「主路径(Pipeline.Recall/graph 兜底)产出候选数,在 E4.2/wiki 补充扩展之前捕获」,与最终 count 解耦。字段名保留(消费方兼容),全部消费方注释同步。keyword 路径恒 0。

- core.go: Recall 端点契约注释 + annScanned 捕获点注释(4 处)
- eval.go: checkKeywordRelevance/RunGolden 口径注释
- eval_test.go: liveRecallOnce/keywordHitAnywhere/TestEvalGolden_Live 口径注释 + CALIBRATED 结构化记录行(供分类器复算 go/py 差异=0)
- golden-set.json: description 说明字段补 N=5 口径
- golden-check.py: N_TOP=5 前 N 条口径 + CALIBRATED 记录行(仓外文件,另行提交)
2026-09-21 21:34:04 +08:00
xiaoyi b7fae4c23b perf(distill): 单轮 flush 内并发蒸馏(concurrency=3)+ flushing 守卫防重叠 + dailyUsed 竞态修复
实测瓶颈定位:摄入 ~132 条/min vs 蒸馏 11.6 条/min。
batchSize 只决定一轮取多少条(实测 batch 恒为 10、token 阈值从未触发),
真正瓶颈是 flush 内**串行**的 LLM 调用(每条 1 次 callLLM5D ≈ 5.2s)。
改:并发调用 LLM(写回仍串行)+ 防重叠守卫 + dailyUsed 原子累加。
测试:-race 下 peak=3、耗时 403ms vs 串行 1.2s、重叠触发不叠加并发、9/9 无丢条目。
未部署:等 09-23 织忆观察窗结束再上线(重启会重置日计数器污染判据)。
2026-09-21 21:27:01 +08:00
小唯 4dfe8195a9 feat: implement 72h conflict suppression gate with persistence (seenPairs) and unit/E2E tests 2026-09-21 18:57:24 +08:00
小唯 e1985dfe32 test(eval): AC 验收 — 全量金标实跑 + 反证 + 空断言自守(t_d2380cd7)
eval_test.go(332 行,新增):
- TestEvalGolden_CounterEvidence(AC①反证):item[0] 故意改成
  mem_0000000000000000000 → 该条必须 FAIL(判据非恒真)
- TestEvalGolden_Live:14 条全量实跑,一次 HTTP 同时拿 ids+contents,
  每条 200ms 间隔;429 读 retry_after_s 退避;结尾打印真实数字
- 判据分层:ID 漂移但内容切题 → WARN;关键词未命中 → 硬 FAIL
- 校准基线:pass=11/14(fail=1 MMR;warn=2 ID 漂移)
2026-09-21 18:55:07 +08:00
小唯 5b4f7cd95a fix(eval_test): uid→ids 构建错误(t_2d4e3b92 AC②④) 2026-09-21 18:48:38 +08:00
小唯 0b765ccc8b fix(api): 统一错误体 schema — 失败返非 2xx + {"error":{code,message,retry_after_s"}} (t_2d4e3b92)
- recall 后端故障 502 recall_backend_error / keyword 编码失败 非 2xx(此前返 200 + 异形 body)
- 401 unauthorized / 429 rate_limit_exceeded 由扁平 error 字符串改为统一对象
- 成功体补 annScanned,区分「真无匹配」与「依赖故障」
- 新增 routes + middleware 两组形状契约测试(含反证:历史扁平/旧限流体必须被判错)
2026-09-21 18:34:21 +08:00
小唯 41e95aa36d chore: 收工 eval.go gofmt + server.go 未提交杂项 2026-09-21 18:14:24 +08:00
小唯 5eccbaef03 chore(api): 收工未提交杂项 core.go/eval.go/testdata 2026-09-21 18:12:18 +08:00
小唯 c6be6a4cd5 fix(governance): DedupeActive 改为归档+审计,不物理删除 (t_0fdee22a) 2026-09-21 18:09:09 +08:00
小唯 d4d5db266a feat(governance): 冲突队列保留策略+去重(t_4ce53193) - sha256去重键+7天保留期+归档审计+挂进t_consolidation触发器 2026-09-21 17:32:50 +08:00
小唯 1060e62066 fix(admin): /memories 与 /memories/export 改用全表扫描(Scan),弃零向量 ANN
病根:两个列表/导出端点一直用『零向量 ANN』冒充『列出全部』——零向量没有方向,
ANN 退化成本存储序前 N 行,再按 ns 客户端过滤 ⇒ 计数完全失真
(实测 export hermes-main 报 1 条,真实 139+;shared 报 0,真实 29)。

修复:
① Rust lancedb_ops: scan_for_forgetting 加 namespace 下推(only_if),limit 作用于过滤后结果集
② Go 接口 LanceDB 新增 Scan(limit, ns):
   - RustLanceDBClient → lancedb_scan RPC
   - SQLiteClient → SELECT ... WHERE is_deleted=0 [AND ns] ORDER BY created_at DESC LIMIT
   - MemLanceClient → 内存 map 过滤 + created_at 倒序
③ 两个端点 ListMemories / ExportMemoriesMD 改用 aa.LanceDB.Scan(limit, ns)
④ 补 mock(zz_commit mockLance / write-buffer fakeLDB)+ 5 条新判据
   (全表 3 / hermes-main 2 / shared 1 / limit 生效 / 不存在 ns 0)
2026-09-21 16:36:41 +08:00
小唯 1b937cc990 fix(recall/debug): 诊断端点 ns 解析与生产路径对齐(全库模式=单次空过滤检索 topK=200)
recall/debug 自带一套 nsList(["", "shared"])→ 与 RecallPipeline 的单次全库检索不一致,
诊断输出「2 namespaces」会误导排查。改为同一规则,保证 debug 如实反映生产召回。
2026-09-21 15:18:20 +08:00
小唯 266a85a5d2 fix(recall): 默认调用改全库跨命名空间召回(t_24eceed0 返修)
病根:纺忆写入侧按 agent/域把记忆分到 19 个命名空间(实测 300 采样:
hermes-main 139、wiki-curator-main 35、wiki 30、shared 29、soulful-xiaowei 27、
a06-main 11、domain:* 13…),而默认调用(无 namespace / 无 agent_id)
被推导成 default-main ⇒ 只搜 default-main+shared,全表 10172 条实测仅得 11 个候选,
99.9% 记忆在默认召回的盲区(t_24eceed0 worker 交付只改了推导来源,未解决跨 ns)。

修复:
① api/routes: 新增 resolveRecallNamespace() 纯函数 —— 显式 ns 原样、只有 agent_id 才推导、
   两者都空返回 "" 表示全库;Recall 与 RecallDebug 共用同一函数(debug 如实反映 recall)
② storage/recall.go: namespace=="" → Search(..., 200, "") 全库单次检索(空过滤=不过滤),
   候选池上限 50→100;显式 ns 路径完全不变([ns, shared])
③ 全库模式下关闭两段补丁式补搜(shared / wiki-curator-main),避免重复候选与白花 embedding

测试(8 条,含反证):
- routes: resolveRecallNamespace 5 用例(核心判据:空+空 必须返回 "")
- storage: 全库单次检索(记录 Search 实参)/ 显式 ns 保持 [ns shared] / shared 不重复搜
- 反证:还原旧行为 → 两组量具均 FAIL ✓("默认调用被解析成 hermes-main,又只能看到极小一片记忆")
踩坑记录:stub 用嵌入接口技巧时须一并覆写 Update(),否则后台 recall_count 回写 goroutine
落到 nil 接口 panic。
2026-09-21 15:15:07 +08:00
小唯 1b2bd417b2 fix(governance): IsContradiction 前置门 — 噪声会话段跳过 + bigram 内容词主题门
根因(t_35095c29 复盘):原判据只看否定词差异,不检查两侧是否谈同一主题。
左侧常为会话原文('用户: [kanban] Task t_xxx…'),与右侧任何事实陈述被误判冲突,
106/142 条 pending 假阳性(75%),致蒸馏反复跳过→召回不到。

修复(governance.go):
① noiseSessionPrefixes + isNoiseSessionContent:任一侧为会话噪声段 → 直接判不矛盾
② stopChars + contentBigrams + hasSharedContentWord:中文相邻双字内容词(滤停用字),
   无交集即主题无关 → false
③ 实测量具(conflict_fp_fix_test.go,106 条样本 = 19 个不同对):
   原版 18/19 判冲突(假阳性 94.7%)→ 修复后 1/19(剩余 1 对真实同主题'蒸馏模型',
   漏检方向安全,不跳过蒸馏,交给 recall/feedback 收敛)
④ 反证:撤前置门 → 量具回到 4/19 FAIL ✓ 量具具备鉴别力

测试:conflict_fp_fix_test.go(10 断言 + 量具)、conflict_false_positive_test.go
(12 条历史假阳性样本回归,保留 worker 有价值测试)、conflict_sample_baseline_test.go
(基线量化)全过。
2026-09-21 14:04:40 +08:00
小唯 2d38d8d120 fix(api): recall 默认 namespace 用 classifyNamespace 推导(query 近似 content)+ shared 回退条件改为 (0, top_k)
来源: t_24eceed0 run790 worker 改动,小怡补部署 + 清理误建的根级 go.mod
- core.go: req.Namespace=='' 时由 deriveNamespace 改为 classifyNamespace('general', agentID, query)
  → 无 ns/无 agent_id 时按 content 关键词推导,技术类 query 归 shared,其余归 agent 私有 ns
- shared 回退: len(results)<3 改为 0<len(results)<topK 时补充搜 shared(避免有结果时重复补)
- 删除 worker 误建在仓库根的 go.mod(真实模块定义在 go/go.mod)
验收: go build OK + go test internal/api/routes 3.7s ok + /proc/exe 无 deleted + shared 不变量回归通过
2026-09-21 13:00:54 +08:00
小唯 84c21803c0 fix(distill): 日限额按日历日重置,修复跨日不归零导致蒸馏永久冻结
checkDailyLimit 原用 time.Since(dailyReset) > 24h 判断跨天,
若进程重启点晚于当日 00:00,则次日重启点之前不会归零 ->
dailyUsed 达上限后持续 FALLBACK 直到重启时刻(2026-09-21 实测冻结 84min)。

改为比较年月日;跨日时清零并把 dailyReset 设为当日 00:00。
新增 engine_dailylimit_test.go 三个用例(同日保持/跨日清零/复现冻结场景)。
2026-09-21 12:04:40 +08:00
小唯 3b8ecba48e perf: 织忆内存/碎片双治本 — PageRank O(V+E) 重写 + 写入去抖 + 在线VACUUM
1. PageRank 三重循环 O(47k×374k×20≈3.5e11) → 入边表 O(V+E):
   单次 full 整合 4h41m → 95.2s,内存峰值 3.8G → 237MB(实测)
2. 图谱修剪 merge 边 id 加 relation + INSERT OR REPLACE:
   修复 09-18 起 UNIQUE constraint 静默失败 → 主库首次跑消 9.8万冗余边
3. LanceDB 写入去抖(server.go + validator.go):
   被动验证只写 Confidence 实际变化的记录;原先每输入重写全库 6000 条
   → 版本快照 O(n²),每日 ~1.9万版本/十几G 膨胀的根因
4. 新增 scripts/lance_vacuum_online.py:在线 VACUUM(不停服务)+
   24h 版本窗口(可回滚)+ 行数门禁(减少才报警);替代停服务版
   每日 vacuum → cron 已改为在线版

实测:侧车 RSS 1.09G→69MB;memories 分片 5383→1;服务全程在线
2026-09-21 10:17:56 +08:00
小唯 daab83337b feat(governance): 冲突持久化 + 禁止静默 dismiss(链路空转修复)
问题(2026-09-20 实测):
1) ConflictDetector.active 是纯内存 map,zhiyid 重启即清零
   (17:17 重启后原先 pending 的 5 条冲突全部消失)
2) server.go 的 conflict_muchen 处理器把未匹配策略的冲突改写成 dismiss 再 Resolve,
   且记成 RecordConflictResolved(true) —— 每条 pending 最多活 10 分钟即消失、
   auto_resolve_rate 虚高、冲突从不进入任何人视野
   (实测 17:19 建的冲突,同进程几分钟后 pending:0,未重启)
3) Conflict 不留两侧原文 ⇒ 事后无法核实真假矛盾(上一 commit 已加字段)

改动:
- governance:ConflictDetector 增 path + load/persistLocked(全量原子落盘 tmp+rename);
  新增 NewPersistentConflictDetector(path);NewConflictDetector() 保持纯内存(测试安全)
- Scan/Resolve 变更后落盘
- server.go:检测器改用持久化构造(落盘失败降级内存 + 记日志)
- server.go:禁止无脑 dismiss —— 72h 内保留 pending 供人工处置,超期才 latest_wins
  自动收敛(真裁决才记指标)
- 新增 conflict_persist_test.go:round-trip / resolve 落盘 / 纯内存不碰盘

未用 SQLite:冲突量级小,引 sqlite driver 需 cgo 新依赖(worker 曾在该方向崩溃
并留下锁重入死锁)。JSON 全量落盘零依赖、无锁重入。
2026-09-20 17:54:40 +08:00
小唯 571eee0d9e fix(governance): 冲突判据三修 — 否定作用域 + 补「未」 + 子串放行 + 原文留档
修前实测缺陷(~/mc/工作笔记/已办/2026-09-20-t6-冲突准确性验证-补做.md):
  1) 危险样本 3/3 全误报(假阳性):只要任一侧含「不/没」字符即认可否定差异,
     长句里任意位置的无关否定(「不喜欢吃辣」「不用 API Key」)都把新内容判成冲突
  2) 漏检:「未在招商局工作」判 false(否定词表缺单字「未」)
  3) 子串必误报:短句是长句子串时 overlap/maxLen=1.0 恒 >0.5
  4) 无法事后核实:Conflict 只有 Entity + 固定 Description,不留两侧原文

改动:
  - containsNegCN 补单字「未」(不补「别/莫」,2026-09-18 已因高频非否定词移除)
  - IsContradiction 新增子串同向判据:短句被长句完整包含且自身无否定 → 不矛盾
  - IsContradiction 新增否定作用域:否定词须落在最长公共子串窗口 ±3 内
  - 新增 lcsPos(两行滚动 DP,超长截断 1000 rune)+ negRuneIdx + negWithinWindow
  - Conflict 增 LeftContent/RightContent,Scan 留档两侧原文
  - 新增 conflict_scope_test.go:9 组真实样本回归 + 对称性 + 原文留档 + 辅助函数

验证:go vet 干净;go test ./internal/governance/ 全绿(含既有测试无回归)
阈值 0.5 未动(2026-09-18 定)
2026-09-20 17:38:49 +08:00
小唯 cff87730b2 fix(api): check SoftDelete error returns at all 5 call sites
SoftDelete returns error but call sites discarded it, so a failed
soft-delete was silent — same class as the no-op implementations.
Now logged explicitly at:
  - server.go: ephemeral_expired cleanup (x2)
  - admin.go: auto_forget_fragment / auto_forget
  - feedback.go: Deprecate endpoint
2026-09-20 11:51:57 +08:00
小唯 b5f59703ab fix(storage): check error returns from MemLanceClient in SQLiteMemClient
SQLiteMemClient.IncrementUseful/IncrementNotUseful were discarding the
error return of MemLanceClient.IncrementUseful/IncrementNotUseful.
Since the latter now return real errors (6e67f3e), dropping them
re-introduces the silent-failure class of bug we just fixed across
the Rust path. Also propagate persistMemory errors.
2026-09-20 11:39:30 +08:00
小唯 6e67f3e33f fix(storage): fix all no-op/fake implementations in LanceDB interfaces
- IncrementUseful/IncrementNotUseful now return error and persist via
  rc.Update() (Rust sidecar); all 4 implementations updated
- SoftDeleteBatch now delegates to SoftDelete() for real LanceDB write
- UpdateMemoryContent now persists content + version + version history
- Backup changed from return nil to explicit error (Rust sidecar unsupported)
- IncrementUseful/IncrementNotUseful signatures now include error return;
  all 3 call sites (feedback.go, core.go) check the error
- Test mocks (mockLance, fakeLDB) updated to match new interface

Fixes the 'silent no-op' class of bugs discovered during dedup repair.
2026-09-20 09:53:00 +08:00
小唯 ece2da12ac fix(distill): narrow short-fact exemption to structured prefixes only
The <20-char exemption added earlier for human-side soulful facts also
carried a 'hasStatementSubject && !status && !question' fallback.
hasStatementSubject only checks for a subject word - it does not check
that the content is a fact - so fragments, questions and status
reports all passed:

  '用户喜欢喝咖啡'            (7 chars, fragment)
  '用户重启了系统'            (7 chars, fragment)
  '用户今天完成了一次系统检查工作' (14 chars, fragment)
  '用户要不要升级系统吗'       (question)
  '小唯a06与all桥接状态同步完成' (status report)

Five assertions in the distill suite were failing because of this
(TestIsQualityFact_ShortFragmentsRejected, _PureQuestionRejected,
_StatusReportRejected, and TestDistillOne_QualityGateBlocksFragments
which kept 2 facts where the gate should keep 1).

The exemption is now limited to daemon-written structured prefixes
(画像规则:/心迹:/牵挂:/偏好:/习惯:/目标:/行为规则:) - strong signals that
cannot be confused with noise. Natural short sentences are weak signals
and are rejected as fragments; if a writer genuinely needs one kept, it
should supply the structured prefix rather than loosen this gate.

go test ./... now passes (TEST_EXIT=0, 8 packages ok).
2026-09-20 08:55:37 +08:00
小唯 6c895618f9 test: fix mockGraph.AllEdges signature; add namespace classification probe
mockGraph declared AllEdges() []map[string]interface{} while
governance.GraphStore requires AllEdges(string) []governance.GraphEdge.
go build ./... stayed green because test files are not compiled by
build, so the routes test package could not be compiled at all and the
whole suite silently went unrun - which is why the earlier AllEdges
'compatibility fix' looked done without ever executing.

After fixing the signature the suite runs: all tests pass including
TestCommitConflictGate_E2E (end-to-end commit -> conflict -> distill
gate, 3.70s).

zz_ns_probe_test.go also gives positive evidence for the
classifyNamespace fix: human-side content now resolves to
soulful-xiaowei-main rather than a06-main.

    '牧尘在招商局工作'      -> soulful-xiaowei-main
    '牧尘不在招商局工作'    -> soulful-xiaowei-main
    '画像规则: 牧尘在招商局工作' -> soulful-xiaowei-main
2026-09-20 08:40:41 +08:00
小唯 fd699151a8 fix(dedup): use SoftDelete path and flush buffer before responding
Two write-path fixes found while executing the dedup policy.

1. The endpoint wrote is_deleted via Update(). The system's canonical
   path is SoftDelete(), which routes through SoftDeleteWriteBuffer
   (5-minute window / 256-item threshold, single-transaction commit).
   Update() writes one MVCC version per row, so this endpoint would
   have created 411 versions in one run - the exact growth pattern the
   vacuum script was written to clean up (table measured at 3.23G
   before a 40M compaction).

2. The response was sent while soft deletes were still buffered. During
   execution lance-vacuum (daily 03:30) stopped the service and ran
   optimize(cleanup_older_than=0), discarding the unflushed writes: the
   endpoint's deleted=34 never landed and is_deleted read back all
   false, while the caller only saw curl exit=52 for the empty reply.
   A new storage.FlushSoftDeleteBuffer() is now called before the
   response, so reported counts equal what actually landed.

Demotions were unaffected in that incident because IncrementNotUseful
writes immediately (297 of 377 landed); they are now also correct
additive increments thanks to the earlier col+1 fix.
2026-09-20 03:34:59 +08:00
小唯 9188b9e5c9 fix(dedup): empty namespace means all namespaces; per-namespace breakdown
Passing namespace:'' silently scanned only hermes-main because of a
hardcoded default, and the response echoed back 'hermes-main' so the
missed scope was invisible - easy to mistake for a completed full sweep.
Empty now means all namespaces, and the response carries a by_namespace
breakdown (clusters/deleted/demoted) so a full run is auditable.
2026-09-20 03:10:22 +08:00
小唯 dd25457926 fix(demote): IncrementNotUseful was a no-op; implement A-policy dedup
Two fixes, both silent-failure class.

1. RustLanceDBClient.IncrementNotUseful was an empty function body.
   Every demotion path was a false success: the HTTP endpoint returned
   200, zhiyi-dedup.py reported '34 demoted, 0 failed', and
   not_useful_count stayed 0 on all 32 dropped rows. Now delegates to
   Update with $inc.

2. Update emulated $inc in Go using the _local.memories cache, which is
   only populated on the Insert path (Search does not fill it). On a
   cache miss getIncCachedValue returned 1, so an increment wrote the
   absolute value 1 - resetting an existing count of e.g. 5 back to 1.
   Now the column is sent to Rust as the expression 'col + 1'; Rust's
   update() detects the arithmetic, reads the row's true current value
   and computes the result, so the increment is genuinely additive.

Also implements 牧尘's chosen dedup policy (option A) in admin/dedup:
keep the NEWEST entry per cluster, then for every other member
  - if it normalises to the same text (punctuation/whitespace/quote
    differences only) it is a true duplicate -> soft delete
  - otherwise it carries different information (different numbers,
    subject, timestamps, versions) -> demote only, never delete
This satisfies 'newest wins' without discarding distinct facts.
2026-09-20 02:55:05 +08:00
小唯 cdf06312a4 feat(dedup): cluster-based keep-fullest policy in admin/dedup
Policy set by 牧尘 on 2026-09-20:
  - existing duplicates  -> keep the MOST COMPLETE entry
                            (longest content; ties broken by earliest)
  - future writes        -> newest wins (handled by the write-side gate
                            in commit/distill)

Two behavioural fixes over the previous pairwise loop:

1. Union-find clustering instead of pairwise marking. With chained
   similarity (A~B, B~C, but A!~C below threshold) the pairwise loop
   counted two pairs and left the cluster with multiple survivors, so
   'keep the fullest' silently kept several entries. Measured: 220
   clusters via closure vs 384 via pairwise - ~12% undercount.

2. Clustering is confined to a single namespace. Identical text in two
   namespaces is two agents' own copies (e.g. domain:INSIGHT and
   hermes-main), which is intended, not duplication. The early string
   compare also keeps the full-table O(n^2) pass cheap.

Response now reports clusters/merged/deleted/policy, and an empty
namespace means 'all namespaces' rather than defaulting to hermes-main.
2026-09-20 02:50:22 +08:00
小唯 ec17c836a1 fix(dedup): parse vector field in Rust IPC Search results
RustLanceDBClient.Search decoded the Rust IPC response into a struct
that had no vector field, so every MemoryRecord returned to Go had an
empty Vector. Consequence chain:

  len(results[i].Vector) > 0 && len(results[j].Vector) > 0
always false → the cosine branch was skipped in:
  - commit path near_dup (core.go:634)
  - distill direct-write dedup gate (server.go:179)  <-- the fix I just
    deployed this round was silently a no-op
  - admin/dedup endpoint (core.go:634)                <-- merged≈0 false
    negative

Without this, no near-duplicate detection in the whole system works —
it silently fell back to exact string comparison only.

Retained: the vector is forwarded from the raw IPC JSON; if Rust stops
returning it, the degenerate behaviour is exact-string comparison
(which still works), rather than a crash.
2026-09-20 01:07:27 +08:00
小唯 4af85fa093 fix(dedup): scan full table before namespace filter in Dedup endpoint
The endpoint passed the namespace straight into LanceDB.Search with
top_k=200. Rust ranks by distance to the zero vector globally and only
then applies the namespace filter, so a namespace's rows were mostly
never returned: soulful-xiaowei has 238 rows but only 9 came back,
hermes-main has 3387 but only 200 did. The endpoint therefore reported
merged=0 on every call - a false negative that made memory
deduplication look complete when a full cosine scan finds 404
redundant rows at the 0.98 threshold.

Now it pulls the whole table (top_k=1000000, no namespace filter) and
filters by namespace plus is_deleted in Go, so 'no duplicates' is a
claim the endpoint can actually support.
2026-09-20 00:43:50 +08:00
小唯 69602ddd08 fix(distill): semantic near-dedup gate on distill write path
Previously distill's direct-write path only skipped facts whose exact
content already existed (sim[0].Content == fact). This let semantically
equivalent facts - e.g. "牧尘在招商局工作" vs "牧尘在招商局工作。"
(trailing period) - slip through as separate memories, accounting for
most of the 404 redundant entries found by a full-vector cosine scan.

Now distill encodes each fact, searches 5 nearest neighbors in the same
namespace, and skips insertion if any neighbor has either identical
content OR cosine similarity >= 0.98. The same threshold is shared by
the commit path (routes/core.go), via a new api-level cosineSimilarity
helper.

This is preventative only: it does NOT merge existing duplicates.
Existing 404 redundant entries remain; they will be cleaned up by a
separate manual dedup run after the gate is live.
2026-09-20 00:37:25 +08:00
小唯 0d4add1fb2 test: add AllEdges mock for GraphStore interface compatibility 2026-09-20 00:25:00 +08:00
小唯 835b308df6 fix conflict scan: use real vector of new content instead of zero vector for retrieving existing memories 2026-09-20 00:10:03 +08:00
小唯 3e8157307e fix distill threshold: allow soulful short facts even when LLM score is low 2026-09-19 23:51:32 +08:00
小唯 ab974d0bed fix classifyNamespace: person-side categories (profiles/heart/flag/cares) must go to soulful namespace 2026-09-19 23:49:08 +08:00
小唯 21e51eb7f6 fix classifyNamespace: remove 牧尘 from sharedKeywords (person names must not trigger shared namespace) 2026-09-19 23:27:49 +08:00
小唯 497802cd62 fix: mergeMemory branches must trigger AutoDistill after dedup (was returning without distilling) 2026-09-19 22:54:41 +08:00
小唯 3ab4ab01bd fix HardRulesPass rule2: allow soulful short facts (<10 chars) into distill 2026-09-19 22:23:37 +08:00
小唯 962489f20c fix: Search must forward namespaceFilter to Rust IPC (was hardcoded empty) 2026-09-19 22:16:14 +08:00
小唯 6d4cbc79ca fix distill quality gate short-fact rejection + register conflicts into active 2026-09-19 21:38:59 +08:00
小唯 dbe6c278c5 deep-think integration: conflict-detector fix, GraphRAG recall fusion, Louvain community discovery, deep-think endpoint 2026-09-19 19:12:49 +08:00
小唯 be120f3c07 fix(governance): #14 冲突误判致蒸馏跳过→召回不到 (t_a54aae79)
1. IsContradiction 阈值 0.3→0.5: 字符级分词下 0.3 过敏, "同话题不同事实"
   大量误判为矛盾, commit 路径 distill_skipped=conflict 跳过写入
2. containsNegCN 移除单字 莫/别 扫描: 特别/其他/分别 等高频非否定词误报
3. commit 请求体补上 resolve_conflict=true 覆盖开关(此前响应文案提及但
   请求体根本无此字段, 覆盖永远不生效)
4. 测试: governance 阈值回归(真矛盾/别误报/不+购买兼容/无否定基线) +
   routes 端到端门控(E2E: 矛盾不触发/flag触发/误报放行/无矛盾基线)
5. server.go: 补 path/filepath import — 工作区有一处他人 WIP 改动
   (AddEdge 429 重试+落盘), 缺 import 导致全仓 build 失败, 此处仅补
   import 使 build 通过, 该 WIP 逻辑本身未评审

验证: go build ./... + go test ./... 全绿; 线上验证见看板卡 t_a54aae79
2026-09-18 04:21:08 +08:00
小唯 ee50ef0d59 fix(rust): 恢复缺失的 LanceDBStats/stats() 和 helper 函数 (t_3fb41049); storage: 添加软删除累积延迟缓冲 (t_384f785b) 2026-09-14 22:14:52 +08:00
小唯 68ce2c9678 feat(distill): 限制 prompt ≤8k token 保护本地 2B 单槽; storage: 添加 SoftDeleteBatch 及 Rust 端批量软删除优化 (t_3fb41049) 2026-09-14 22:09:44 +08:00
小唯 bea2849494 fix(tests): 修 pre-existing 测试失配,恢复 make test 全绿 (t_8496e8b6)
四处均为本任务之前就存在的测试漂移(与优化B 无关,但让 go test ./... 无法通过):

1. selfoptimize: 2a350cf 给 NewGapDetector 增加了 (emb, ldb) 两个参数,
   8 处测试调用点未跟进,整包 test 构建失败。测试只用到 RecordMiss/Close/List,
   不需要真实依赖,传 nil, nil 即可。
2. integration_test.go:296 selfoptimize.NewGapDetector() 同上。
3. integration_test.go:323 InMemoryGraph.Navigate 新增 relFilter 参数,补 nil。
4. integration_test.go:339 /api/v1/triggers/fire 的夹具用了 "t1"(那是 executor
   内部编号),routes.Triggers 的真实 ID 是 t_decay/t_distill/t_merge 等,
   导致 Fire 返回 404 trigger not found。夹具改用 t_decay。

验证: go vet ./... 无输出; go test ./... -count=1 全部 ok(根包 8.1s)
2026-09-12 18:07:40 +08:00
小唯 41230ccb0a docs+test(recall): 修正批量更新注释(delta 分组,非 CASE WHEN)+ 补 IPC 只读校验用例 (t_8496e8b6)
- recall_write_buffer.go / lancedb_ipc.go: 注释与实现对齐 —— lance 不支持 CASE WHEN,
  Rust 侧按 delta 分组、每组一次 update 提交;整批版本数 = 不同 delta 的个数(通常 1 个)
- batch_ipc_integration_test.go: +TestBatchIPCScanReadOnly(只读扫 socket 校验 recall_count 真落盘)

实测(临时 sidecar,/tmp/mw-verify-180517):
  VERSION_ACCOUNTING: 批量3条 → 1 个版本 | 逐条3次 → 3 个版本
  BATCH_IPC_OK id=ep_1786982837511335732 recall_count 0 → 3
  SCAN_READONLY: 总 1814 条, recall_count>0 的 742 条, 最大 828
2026-09-12 18:07:30 +08:00
小怡 f2362150d1 perf(recall): 累积延迟更新治理写放大 — 读路径零逐条写 + 单事务批量落盘 (t_8496e8b6)
根因: recall 读路径对每条结果同步 lancedb.Update,LanceDB MVCC 每次提交=1 版本
      实测 15 版本/分钟 / _versions 17.7G(真实数据 22M)

改动:
- Go: 新增 RecallWriteBuffer(窗口合并+阈值触发+优雅退出落盘);recall.go 读路径改 Record();RustLanceDBClient.UpdateRecallBatch
- Rust: lancedb_update_batch IPC — 按 delta 分组,每组一次 update(`recall_count + delta` + id IN (...)) 提交
- 单测 6 个 + IPC 端到端版本计数测试;bench_test.go 修 NewReranker 签名失配(阻塞包测试)

实测: 批量 3 条 → 1 个版本;逐条 3 次 → 3 个版本 (lance 不支持 CASE WHEN,已按 delta 分组规避)
2026-09-12 17:57:31 +08:00
xiaowei 2506291ff6 fix(recall): 笔记知识对召回不可见 —— 补搜 wiki-curator-main
问题: wiki_curator 把「笔记→知识」写进 namespace=wiki-curator-main,
而 recall 只搜 req.Namespace 与 shared ⇒ 33+ 条笔记知识长期在盲区,
用户问"笔记里的资料"永远召回不到。

改动: internal/api/routes/core.go 在 shared 补搜之后增加额外命名空间补搜
(wiki-curator-main), 去重 + 降权 0.45 (不挤掉主记忆但能被看到)。

踩坑: 初版条件写成 len(results) < req.Limit 才补搜 —— 主库记忆充足时
该条件永不成立, 补搜形同虚设。改为无条件补搜后生效。

验证: 部署后实测 recall「Hermes平台核心配置文件」→ 11 条,
含笔记知识「Hermes平台的核心配置文件包括SOUL.md、AGENTS.md…」

已知未修(预先存在, 与本次无关): selfoptimize/storage 的 bench_test.go
调用签名已变更的 NewGapDetector/NewReranker 导致 test build failed。
2026-09-11 21:22:22 +08:00
小唯 e91ca91367 fix(security): 写入侧凭证过滤 — 密钥不再进记忆/图谱/Obsidian 镜像
背景(2026-09-10 mc P0 事故):某轮蒸馏把含真 Tailscale 预授权 key 的原始对话
当事实入库 → 知识图谱按其文本建实体 → ObsidianSyncer 用实体标题当文件名/正文,
把 tskey-auth-… 落成可读镜像文件,又被 git 跟踪并推送 Gitea。
本提交在三个写入边界统一加脱敏。

新增 internal/redact(唯一真源):
- SecretPattern 覆盖 sk- / tskey- / github_pat_ / ghp_ / nvapi- / AKIA
- Go RE2 不支持 lookbehind,故"前一字符是否属于更长标识符"手工判定(isTokenByte)
- sk- 段只吃 [A-Za-z0-9],避免误伤 URL 里的 task-driven 片段(事故中的假阳性来源)
- RedactSecrets 幂等;ContainsSecret 供"直接丢弃"场景用

四处接入:
- distill/quality.go       IsQualityFact 判定前脱敏 + FilterQualityFacts 返回值脱敏
- governance/graph_auto.go cleanEntityName 命中→空串;UpdateFromDistill 只对有效实体
                           建节点/建边;extractEntitiesFromText 过滤;含凭证的 fact 不建节点
- api/routes/obsidian.go   sanitizeFilename 入参脱敏 + PushToObsidian 镜像正文脱敏
- api/server.go            蒸馏直写记忆库路径(绕过 FilterQualityFacts 的那条)同步脱敏

测试:13 条新用例(redact 5 / distill 3 / governance 4 / routes 1),
含 URL 假阳性回归与幂等性断言。全部使用构造的假密钥,不含任何真实凭证。
证据:go build ./... OK;
go test -count=1 ./internal/redact/ ./internal/distill/ ./internal/governance/ ./internal/api/routes/ 全绿。

顺带修既有 test 腐化(否则 governance 包根本无法编译测试,非本任务引入):
governance_test.go isContradiction→IsContradiction、Navigate 补 relFilter 参数;
bench_test.go 同。仅测试文件,无生产行为改动。
2026-09-11 00:41:59 +08:00