Commit Graph
97 Commits
Author SHA1 Message Date
xiaowei abef38cb05 E5.1 Obsidian 插件: CORS中间件 + 记忆面板 + 图谱视图 + 搜索模态框
- 新增 CORS 中间件 (middleware/cors.go): 支持 app://obsidian.md 跨域
- 新增 Obsidian 插件: MemoryView(分页记忆列表) + GraphView(D3力导向图) + SearchModal(语义搜索)
- Go API CORS: Access-Control-Allow-Origin: app://obsidian.md
- Makefile 新增 build-obsidian / install-obsidian 目标
- 插件安装至 ~/.obsidian/plugins/zhiyi-memory/
2026-06-02 15:42:54 +08:00
xiaowei f93da87a97 fix: distill端点 /status /queue /quota(Engine新增导出方法);WORKLOG同步 2026-06-02 12:28:00 +08:00
xiaowei 8db0138b06 G8: Restore+ListBackups 端点;G9: SearchCache 双级缓存(内存+Redis L2) 2026-06-02 11:43:34 +08:00
xiaowei bbce18f748 docs: 同步 E4.3 调用方已接入;G7 E3 自优化闭环完成记录 2026-06-02 09:55:10 +08:00
xiaowei b0059166a2 fix G7 E3: crystallize路由路径+GetSkillCandidates通过Rust IPC查LanceDB
- server.go: 修复mux路由路径(去掉"POST "前缀)
- skill_crystallize.go: 添加POST方法检查
- models/memory.go: 添加safeTime类型避免空字符串解析报错
- lancedb_ipc.go: GetSkillCandidates改用Rust IPC的lancedb_query
- Rust lancedb_ops.rs: 新增query_memories()支持min_recall过滤
- Rust main.rs: 新增lancedb_query IPC handler
2026-06-02 04:19:58 +08:00
xiaowei 243a2066a4 G7.3: skill execute + quality_score fix
G7.3 — Skill执行+遗忘联动:
- skill_execute.go: ExecuteSkill handler, applyForgettingLinkage
- /api/v1/skills/{name}/execute: 返回 enriched_prompt + linked memories
- Trial feedback 联动: success → degree保护, failure → decay加速

Fix quality_score=0 (Step 2):
- main.rs: stratified_sample 用 tier 字段,但 memories 全为 "normal"
- 改为 r.freshness(默认为 "fresh"),匹配 stratified_sample 期望的 tier 值

G7.1+G7.2 已在运行(skill persistence + crystallize API 验证通过)
2026-06-02 00:48:36 +08:00
xiaowei 43286dd2d7 G7.1+G7.2: skill persistence + crystallize API
G7.1 — Skill 持久化:
- BetaSkill: prompt_template, linked_memory_ids, linked_entities, created_at
- BayesianSkillManager: EnableRedisPersistence(), loadFromRedis(), persistSkill()
- Redis HASH (zhiyi:skills) 持久化,重启不丢 skills
- Skill CRUD API: POST/GET/DELETE /api/v1/skills/{name}
- auto_distill skill: ETA=0.93, status=active after 12 trials ✓

G7.2 — Skill 结晶:
- GetSkillCandidates(minRecalls, limit): storage.LanceDB 接口扩展
- SQLiteClient: SQL 查询候选(recall_count>=5, quality_score>=0.7)
- RustLanceDBClient/MemLanceClient: stub 实现
- /api/v1/crystallize/candidates: 获取候选记忆
- /api/v1/crystallize/memory/{id}: 对记忆执行 LLM 结晶
- callSkillLLM(): LLM 生成 prompt 模板(带降级)
- RegisterLDBGetter(): routes 包访问 LanceDB 实例
2026-06-02 00:01:59 +08:00
xiaowei 137c25f16f G7.1 fix: skillByNameHandler method dispatch, Redis persistence verified
- skillByNameHandler: GET/DELETE on same pattern via r.Method switch
- EnableRedisPersistence at startup: confirmed working via logs + curl
- Redis HGET zhiyi:skills: full skill state persisted correctly
- Trial feedback: ETA 0.5 → 0.67 after 1 success (Bayesian update ✓)
2026-06-01 23:48:51 +08:00
xiaowei 4f2c0fe769 G7.1: skill persistence - BayesianSkillManager Redis + CRUD API
- BetaSkill extended: PromptTemplate, LinkedMemoryIDs, LinkedEntities, CreatedAt
- BayesianSkillManager: EnableRedisPersistence(), loadFromRedis(), persistSkill()
- Skill CRUD API: POST/GET/DELETE /api/v1/skills/{name}
- Skill Manager: Register(), Get(), Delete(), Stats()
- redis.go: added HDel method
- triggers.go: removed duplicate Skill/SkillManager (now in skill_bayes.go)
- server.go: EnableRedisPersistence() at startup + new routes
2026-06-01 23:44:36 +08:00
xiaowei 8b01ab2d90 docs: add E4.3 bug fix to WORKLOG 2026-06-01 22:55:32 +08:00
xiaowei df2b648973 fix E4.3: extract entities from content instead of namespace
E4.3 bug: admin.go used mem["namespace"] which is empty (LanceDB
map keys are different from struct field names). Fixed by:
- Replace namespace lookup with content-based entity extraction
- Reuse distill/engine.go heuristic: capitalized words, Chinese
  entities (2-20 chars), tech tokens (alphanumeric/digits)
- Get max graph degree across all extracted entities
- Add isStopWord, stripNonChinese, isTechToken, isAllDigits helpers

Fixes: degree always 0 in ShouldForget call, graph integration broken.
2026-06-01 22:49:50 +08:00
xiaowei 525b5f2bb2 fix E4.3: 回退 FileGraph,恢复 SQLiteGraphStore(数据在 graph.db)
切换 FileGraph 导致 9.9MB 历史图谱数据丢失。
恢复 NewSQLiteGraphStore,GetEntityDegree 加到 SQLiteGraphStore。
验证:1720 节点,32983 边。
2026-06-01 01:16:31 +08:00
xiaowei 2d0923a85c E4.3: 图谱度参与遗忘决策(完整接入)
- GraphStore 接口添加 GetEntityDegree(entity) int
- InMemoryGraph + FileGraph 实现(复用 EvidenceCount 逻辑)
- GetCandidatesForForgetting 返回 namespace + content 字段(存储层)
- AdminAPI 添加 GraphStore 字段,admin.go 的 Forget 接入
- server.go decay 循环接入:namespace 作为实体查图谱度
- 修复预存在错误:SQLiteGraphStore → FileGraph(无 SQLite 图谱)
- 修复预存在错误:consolidation_pipe PageRank 类型断言删掉(FileGraph 自更新)
2026-05-31 22:29:28 +08:00
xiaowei 3c27ec8322 E4.1 中文感知修复文档补充 2026-05-31 16:23:05 +08:00
xiaowei 51eec76f1f fix E4.1: IsContradiction 中文感知修复(字符级切分+否定词检测)
问题:原 IsContradiction 使用 strings.Fields(),中文无空格时整句为一个词,
导致 overlap 始终为 0,矛盾检测永远失败。

修复:
- 新增 splitWordsCN:中文按 unicode.Han 字符级切分,英文按空格分词
- 新增 containsNegCN:检测中文否定词(不是/没有/不/没/莫/别)
- IsContradiction 合并中文/英文两种检测逻辑,阈值调至 0.3

验证:Test 2 返回 {"conflicts": ["小唯是牧尘的女朋友"]} ✅
2026-05-31 16:22:35 +08:00
xiaowei 03fe336616 E4 图谱推理实现文档:E4.1 矛盾检测+E4.2 跨agent recall+E4.3 遗忘参考图谱度
E4.1: ConflictDetector 注入 core.go:API, Commit 时对相似记忆运行 DetectContradiction()
E4.2: Recall 结果 < 3 时补充 shared namespace(去重 + Score=0.5 降权)
E4.3: ShouldForget(graphDegree ...int) 可变参数,度>5 时每度+0.03保留分
E4.3 调用方待接入 graphDegree(admin.go:71, server.go:904)
2026-05-31 16:13:29 +08:00
xiaowei a6403dd088 E4 图谱推理:E4.1 矛盾检测+E4.2 跨agent recall+E4.3 遗忘参考图谱度
E4.1: ConflictDetector 注入 core.go:API, Commit 时对相似记忆运行矛盾检测(IsContradiction),有冲突时在响应中返回 conflicts 字段
E4.2: Recall 结果 < 3 时自动补充搜索 shared namespace,合并跨 agent 结果
E4.3: Forgetter.ShouldForget 加 graphDegree 可选参数,节点度 > 5 时每度 +0.03 保留分
2026-05-31 16:03:21 +08:00
xiaowei 1ef1747769 E3 完成:增量 embedding 验证
E3 已实现(代码早已存在,本次验证确认有效):
- Go Commit() 在 core.go:79 调用 Embedder.EncodeSingle() → BGE HTTP 8000
- vector 随文本通过 IPC lancedb_insert 发送到 Rust,直接存储不重编码
- 验证:commit 后立即 recall,测试记忆排第一得分 0.843 ✅

更新:
- IMPLEMENTATION-FIVE.md: E3 现状修正(已实现)+ 验收标准打勾
- WORKLOG.md: E3 状态 + 下一步改为 E4 图谱推理
- 状态: E1/E2/E3 已完成,E4-E5 规划中
2026-05-31 15:17:35 +08:00
xiaowei f65dee6a3d E1/E2 完成:图谱导航激活 + 多 agent 命名空间验证
E1 — 图谱导航激活:
- 移除 ExpandFromResults 中 5 条残留 debug fmt.Printf 语句(graph_sqlite.go)
- Recall pipeline 已通过 server.go:112 SetGraphExpander 接入,无需额外代码

E2 — 多 agent 命名空间激活(验证完成,无需代码改动):
- Hermes: agent_id=hermes-a06 → namespace=hermes-main(deriveNamespace)
- OpenClaw: agent_id=openclaw, namespace=openclaw-main(zhiyi client 显式传递)
- Rust sidecar: search/scan_all 均有 namespace 过滤(only_if)
- 验证: hermes-main vs default-main 同一 query 返回不同 ID,隔离生效

更新:
- WORKLOG.md: E1/E2 状态 + 详细记录
- IMPLEMENTATION-FIVE.md: E2 现状修正(代码早已实现)+ 验收标准打勾
- 状态: 规划阶段 → E1/E2 已完成
2026-05-31 15:05:32 +08:00
xiaowei cbe4026834 E1: remove debug fmt.Printf from ExpandFromResults, update WORKLOG
- Remove 5 residual fmt.Printf("[E1] ...") debug statements from
  SQLiteGraphStore.ExpandFromResults (graph_sqlite.go)
- Recall pipeline already has graph expansion wired via server.go:112
  (SetGraphExpander(graphStore)) — feature was implemented previously
- ExpandFromResults trigger: len(results) < 5, BFS 1-hop, dedup by ID
- Update WORKLOG: E1 completed, G6.1 cluster_only mode explained,
  G6.2/G6.3 consolidated, add cd0f898/cc615c5 commits, update 下一步
2026-05-31 14:50:56 +08:00
xiaowei cd0f898829 fix: eps=1.0 (19 clusters from 1486 items), add col_vector() for FixedSizeListArray reading
- eps: 0.1 → 1.0 (p50=1.011, meaningful semantic clustering)
- col_vector(): parse FixedSizeListArray and Float32Array for LanceDB vector column
- search() and scan_all() now read actual vectors instead of vec![]
- client.go: remove duplicate Epsilon field, eps=1.0
2026-05-31 14:35:29 +08:00
xiaowei cc615c54d9 docs: add consolidate fix work log (2026-05-31) 2026-05-31 11:58:26 +08:00
xiaowei 1fa8349e48 fix: mode propagation, eps=0.1, BGE connectivity check, vector storage, WriteTimeout 60s
Key fixes:
- Go task mode now correctly propagates to Rust sidecar via IPC (RunWithMode)
- DBSCAN eps lowered from 1.5 to 0.1 (cosine threshold 0.995)
- Add BGE HTTP connectivity check (TcpStream 2s timeout) before Step 4 quality backtrace
- Fix LanceDB insert_batch: use Float32Array::from_iter_values + try_new with None validity
- Fix embed.rs: add .timeout(10s) on encode request
- Fix consolidation_pipe.go: nil guards on report
- Fix main.go: WriteTimeout 60s (was 10s, causing exit 52 on full mode)
- Add vector zero detection + debug logging (norms, sample distances)
- Log warning when BGE HTTP unreachable (skip quality backtrace instead of hanging)
2026-05-31 11:46:59 +08:00
xiaowei 7ac626e847 fix: useful_count type + revert resp.Result (use report_json)
1. Feedback端点  类型修复 (core.go)
   - 之前传 map[string]int,Update() 只处理 map[string]string/map[string]interface{}
   - int 被当作 string case 处理, 被丢弃
   - 修复: map[string]interface{}{"": 1}

2. IPC 搜索字段名 (lancedb_ipc.go)
   - Rust send_ok 用 report_json,不是 result
   - 错误改用 resp.Result 导致 IPC 结果永远为空
   - 回退到 resp.ReportJSON (这是正确的)
   - 加 Result 字段只是为了日志可见性(不被 Rust 填充)

验证: recall_count=6, useful_count=1 确认写入 LanceDB
2026-05-31 07:43:42 +08:00
xiaowei 8d3582046c fix: IPC 字段名 + recall_count 持久化 + feedback 端点
1. IPC 字段名修复 (lancedb_ipc.go)
   - Rust lancedb_search 返回 'result' 字段,Go ipcResp 用 'report_json'
   - 两边字段名不匹配导致 ReportJSON 永远为空,Search() 降级到 localSearch
   - 后果:recall_count 只写 Go 缓存,不写 LanceDB,重启后丢失
   - 修复:ipcResp 加 'Result' 字段 (json:"result"),Search() 改用 resp.Result

2. Update 扩展  支持 (lancedb_ipc.go)
   - 原来只有 recall_count 支持 ,useful_count/not_useful_count 被丢弃
   - 新增 getIncCachedValue() 统一处理所有 counter 字段
   - 加日志:Update OK / Update FAILED 可见
   - 修复 Feedback 端点  传递问题(map[string]int 而非 map[string]string)

3. /api/v1/feedback 端点 (core.go + server.go)
   - POST memory_id + useful/not_useful,通过 LanceDB.Update() 持久化
   - 之前返回 404

破坏性:无(Search 降级是隐式行为,原代码已有 fallback)
2026-05-31 07:36:31 +08:00
xiaowei 190e700876 docs: 更新 DESIGN.md — 同步 v3.8 实施状态
- Appendix E(统一记忆规划)→ 改为「v3.8 实施完成」状态
- 新增「原始会话 vs 语义记忆」澄清(Section 1.3)
- 版本日志标记 v3.8/v3.9 已实施
- 状态改为「v3.8 部分实施」
- 原始会话各自存储路径(Hermes: ~/.hermes/sessions/,
  OpenClaw: ~/.openclaw/workspace/sessions/)
2026-05-31 06:46:10 +08:00
xiaowei 7684bc4075 docs: 紧急修复 - service 文件路径冲突,数据完好 2026-05-31 04:41:23 +08:00
xiaowei 09c2319d7e fix(G6): recall_count 并发递增 + cache population
问题:
1. recall goroutine 通过 id 查 _local.memories 总是 cache miss(cache key 格式不匹配)
2. Update() 用 defer unlock,IPC 调用在锁外,并发时丢失递增
3. 缓存命中时跳过 incrementRecallCount

修复:
1. recall.go: 结果写入 _local.memories(cache key=id)
2. lancedb_ipc.go: 移除 defer unlock,整个 IPC 调用在锁内
3. lancedb_ipc.go: Update 发送 cache 当前值(不是硬编码 "1")
4. lancedb_ipc.go: 支持 map[string]string 和 map[string]interface{} 两类
2026-05-31 02:14:56 +08:00
xiaowei eac0dfc889 docs: 更新服务拓扑信息,WORKLOG.md补全进程/存储/API/环境变量 2026-05-31 01:56:35 +08:00
xiaowei 9900b30acc docs: 更新 WORKLOG.md — G6.2 LLM修复 + Qwen3.5-122B切换 + G7-G9补全 2026-05-31 01:39:40 +08:00
xiaowei 0ef3f2f0ba fix(G6.2): LLM model换qwen3.5-122b,deploy service同步更新 2026-05-31 01:25:51 +08:00
xiaowei b175b82b01 fix(G6.2): MiniMax M2.7 reasoning_content fallback + LLM API key 修复 2026-05-31 00:51:12 +08:00
xiaowei 3b031ed2e7 fix(G7): skill trial路由 + LLM API key 传递到 Rust sidecar + Authorization header 2026-05-30 23:55:43 +08:00
xiaowei e95ae36236 G7.1 skill修复: trial路由{name} capture + Beta-Bernoulli后端统一
- Fix /api/v1/skills/{name}/trial route: was '/skills/' (no capture) → '/skills/{name}/trial'
- Skills.Trial now delegates to BayesianSkills.RecordTrial (was broken simple ETA)
- Skills.List now reads from BayesianSkills (was reading from wrong store)
- 3 successful trials → η=0.8, status=active (Beta-Bernoulli verified)
2026-05-30 23:21:34 +08:00
xiaowei dbb43ce9b1 docs: G6 完成状态 + 新增 G7/G8 2026-05-30 23:00:54 +08:00
xiaowei 51a0747324 fix(G6): 修复 Rust IPC 字段名映射 + cluster_only 模式 + consolidation timeout
- client.go: Result 字段名从 clusters/noise 改为 clusters_found/noise_points(匹配 Rust)
- consolidation_pipe.go: 使用 cluster_only 避免 LLM 阻塞;GraphPruned 改为 0(cluster_only 不做 pruning)
- server.go: consolidation handler 加 30s timeout,避免 HTTP 挂起
2026-05-30 22:58:27 +08:00
xiaowei ad831b4e7f docs: 更新 G5 完成状态 + G6 待做 2026-05-30 22:28:33 +08:00
xiaowei d4c13e7377 feat(G5): eval 评估框架完整实现
G5.1 - eval/run namespace 支持 + 完整 IR 指标
G5.2 - eval/generate 生成 12 条金标(4类×3级),recall 自动获取 expected_ids
G5.3 - vprop 反向传播端点 + V 值查询接口(/trace/vprop, /vvalue/memory/)
2026-05-30 22:26:20 +08:00
xiaowei a629afd4cb docs: 工作记录更新 G1-G4 2026-05-30 22:00:53 +08:00
xiaowei 93671dc1d6 feat: 接入 WSPrefetchAdapter 到 recall 管线
- 新增 WSPrefetchAdapter 实现 storage.PrefetchPusher 接口
- 当 agentID 为空时使用 WSBus.Broadcast(支持多 Agent)
- 在 NewAPI 中通过 SetPrefetchPusher 注入
- WebSocket prefetch.push 事件在 CO_OCCURS 权重 > 0.6 时触发
2026-05-30 21:58:26 +08:00
xiaowei 928468042f fix: MMRSelect 使用字符 bigram Jaccard 相似度替代空向量 cosineSimilarity
原问题:Rust LanceDB IPC search 不返回 vector,导致 cosineSimilarity 永远为 0,
MMR 退化为纯相关性排序(无多样性去重效果)。

修复:改用 Content 字段的字符 bigram Jaccard 相似度,并修正 MMR 公式
(从 (1-λ)*rel - λ*maxSim 改为 (1-λ)*rel + λ*(1-maxSim),
  使 maxSim 高时 MMR 降低,符合设计文档 §2.6)
2026-05-30 20:57:16 +08:00
xiaowei 69d41c3087 fix: E1 extractPotentialEntities 使用 utf8.RuneCountInString 而非 len() 计算中文字符数
- len(string) 返回字节数而非字符数,导致所有 2-8 字中文词被错误过滤
- 原始代码用 continue,现已恢复正确迭代逻辑
- 添加 unicode/utf8 导入
2026-05-30 20:33:22 +08:00
xiaowei 31712986de fix(graph_sqlite): fix ExpandFromResults - wrong field access + Chinese entity extraction
Root causes fixed:
1. ExpandFromResults accessed p["to"] but SQLiteGraphStore.Navigate
   returns {"source","target","relation","weight"} - field name mismatch
   caused graph expansion to always return empty
2. Original logic checked seen[r.ID] (memory ID) against entity name
   (neighbor string), which never matched - removed stale logic

Also improved extractPotentialEntities:
- Now handles Chinese continuous character sequences (2-8 chars)
- Chinese entities extracted as whole words, not split by spaces
- English capitalized words preserved with original casing

Changes:
- ExpandFromResults: correct field access (source/target/relation),
  bidirectional neighbor detection (forward + reverse edges),
  separate seenEntities map for entity dedup
- extractPotentialEntities: full rewrite for Chinese + better English
  word tokenization without space-based splitting
2026-05-30 19:17:12 +08:00
xiaowei 04057637e0 docs: add 织忆五步实施计划 E1-E5,附测试验证方案
E1: 图谱导航激活
E2: 多agent命名空间
E3: 增量embedding
E4: 图谱推理
E5: 产品UI(长期)
每个阶段含阻塞记录+进度追踪
2026-05-30 18:58:16 +08:00
xiaowei 5927f7d740 fix(selfoptimize): repair 3 dashboard metrics that were stuck at 0
1. gap_closure_rate: RecordMiss now increments Dash.TotalGaps; Close
   calls Dash.RecordGapClosed(). Previously denominator was never written.
2. auto_resolve_rate: scanConflicts now calls Dash.RecordConflictResolved
   after each AutoResolve. Previously consolidation path bypassed Dashboard.
3. avg_distill_loss: merge-count heuristic scaled from 0.01 to 0.002
   with 0.05 cap. Previously 50 merges = 0.50 loss (same weight as 50 LLM
   distillations), corrupting the average.
2026-05-30 18:03:49 +08:00
xiaowei 35f78ed14b chore: fix gitignore, remove stale graph.json, add MemoryMax limits
- Fix broken .gitignore (literal \n chars removed)
- Add go/build.sh to .gitignore
- Remove empty stale graph.json (SQLite graph.db is real store)
- Set MemoryMax=2G for zhiyid + zhiyi-sidecar services
- Switch VLLM_ENDPOINT to local bge-m3 (port 8000)
2026-05-30 17:58:27 +08:00
xiaowei 05c4a2d32e fix(metrics): sync Prometheus zhiyi_total_memories from LanceDB stats
- Stats handler now updates TotalMemories/TotalEpisodes gauges on each call
- Startup goroutine initializes metrics 1s after boot (waits for Rust sidecar)
- Fixes zhiyi_total_memories showing 0 instead of actual count
2026-05-30 17:39:38 +08:00
xiaowei daabd54649 fix(graph_sqlite): add busy_timeout + stale WAL/SHM cleanup on startup
Root cause: No busy_timeout set → immediate 'database is locked' on restart
when stale WAL/SHM/journal files remain from crashed previous instance.

Fix:
- On open(): clean stale -wal/-shm/-journal files (only if main db exists)
- After sqlite3_open(): set busy_timeout=10000ms (10s wait for locks)
- This eliminates the InMemoryGraph fallback on restart
2026-05-30 17:35:29 +08:00
xiaowei cd49a2801a fix(ws): add per-conn mutex to prevent concurrent WriteJSON panic
Root cause: Broadcast() and Push() launched multiple goroutines
writing to the same websocket.Conn simultaneously. Gorilla websocket
does not support concurrent WriteMessage/WriteJSON.

Fix: add connMu sync.Map (*websocket.Conn → *sync.Mutex) to WSManager.
Each WriteJSON now goes through writeWithLock() which acquires the
per-conn mutex. Unregister() also acquires the lock before Close()
to prevent write/close races.

No functional changes to API surface.
2026-05-30 17:15:19 +08:00
xiaowei c5835def8f feat: 双插件推送 — Hermes + OpenClaw zhiyi 插件同步
Hermes 插件 (~/.hermes/plugins/memory/zhiyi/__init__.py):
  - 新增 memory_feedback 工具(useful/not-useful → VProp + Dashboard)
  - 新增 memory_metrics 工具(7项自优化指标)
  - recall 结果含 quality_score/v_value 字段
  - system_prompt 新增 feedback/metrics 使用引导

OpenClaw 插件 (openclaw-zhiyi-plugin/):
  - v0.5.0 — 3项修改由小雪完成
  - types.ts: RecallResult 加 quality_score/v_value
  - client.ts: navigate 改 GET+params; 加 metrics()/gaps()
  - index.ts: 注册 memory_feedback/memory_metrics/memory_gaps 工具
2026-05-30 15:00:33 +08:00