xiaoxue_admin
/
memoryweave
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
125 Commits 4 Branches 0 Tags 102 MiB
Commit Graph

125 Commits

Author SHA1 Message Date
小唯 a76b6d3afa feat(graph): add graph cleanup + nl_query + navigate grouped format 
- Add CleanupNoiseNodes() to GraphStore interface + SQLiteGraphStore impl
- Add /api/v1/graph/cleanup endpoint (dry_run + execute)
- Add /api/v1/graph/nl_query endpoint for natural language graph queries
- Improve /api/v1/graph/navigate: add grouped_by_relation, suggestions, normalized_entity
- Add CleanupNoiseNodes stubs to InMemoryGraph and FileGraph

BREAKING: navigate response now includes grouped_by_relation and suggestions
 2026-06-15 18:09:27 +08:00
小唯 79b994ce4b feat: add memory_graph_navigate and memory_graph_stats tools (v1.1.0) 
- Add graph_navigate() and graph_stats() to ZhiYiClient
- Add memory_graph_navigate tool: N-hop knowledge graph navigation
- Add memory_graph_stats tool: graph statistics (nodes/edges/density)
- Update system prompt to list all available tools
- Bump version 1.0.0 -> 1.1.0
 2026-06-15 17:28:16 +08:00
小唯 b9f9f75c31 fix: IsContradiction typo in bench_test.go 2026-06-15 11:08:13 +08:00
小唯 8ad4a7c794 feat: add freshness field to memory lifecycle 
- server.go: set freshness='fresh' when distill creates new memory
- recall.go: set freshness='verified' when memory is recalled
- Freshness values: fresh (new), verified (recalled), stale (decayed)
 2026-06-15 10:39:15 +08:00
xiaowei 42a8aeb53c api/server: only record distill_loss when Overall>0, exclude fallback pollution 2026-06-12 11:43:05 +08:00
xiaowei cf31ff5e8e api: add /api/v1/metrics endpoint exposing Dashboard metrics (avg_distill_loss, etc.) 2026-06-12 08:54:00 +08:00
xiaowei 90c43c0717 fix(distill): increase HTTP client timeout 30s→120s for LLM calls 2026-06-11 10:39:07 +08:00
xiaowei c3db6ffe30 feat(backup): add standalone backup script, deprecate Go API route 2026-06-10 10:19:55 +08:00
xiaowei b237cc9aeb fix(backup): add 5min timeout to tar, fix comment 2026-06-10 10:16:49 +08:00
xiaowei 2c96f805f4 fix: consolidate trigger uses full mode instead of cluster_only 
Problem: server.go 30s ticker loop called consolPipe.Run() which
defaults to cluster_only mode. The 'consolidation' and 'backtrack'
actions never executed prune/decay/quality steps.

Fix:
- Separate 'distill' action (keeps cluster_only for fast frequent runs)
- 'consolidation' and 'backtrack' now call RunWithMode(full)
- selfoptimize.Flow.Register('consolidate') also uses RunWithMode('full')
- Proper case indentation (4 tabs for labels, 5 for bodies)

Result: prune + decay + quality steps now run when t_consolidation
or t_backtrack fires (48h cooldown protects against over-firing)
 2026-06-09 12:45:33 +08:00
xiaowei ffee97032e docs: save DESIGN-v2-llm-optimizer.md (parked, not implementing yet) 2026-06-08 11:16:54 +08:00
xiaowei b669a01e16 feat: add DESIGN-v2-llm-optimizer.md (草稿 v0.3) 2026-06-08 11:09:43 +08:00
xiaowei 1516fe5bc8 fix: add clusters_found/noise_points/quality_score to consolidate API response 2026-06-08 10:35:10 +08:00
xiaowei f286348ad8 fix: gap auto-close + conflict_muchen handler 
- GapDetector.RecordHit(topic): recall 命中后自动关闭该 topic 的 open gap
- GapDetector.ClearMisses(topic): 清除 miss 计数，避免重复触发
- SetGlobalGapDetector / GetGlobalGapDetector: 全局实例注册，供 recall 回调使用
- recall handler (core.go): 命中时调用 RecordHit + ClearMisses 实现 gap auto-close
- conflict_muchen handler (server.go): 注册处理器，自动裁决 pending conflict（latest_wins / primary_wins / dismiss）
- fix: server_sqlite_nowindows.go 修复 osGetenv → os.Getenv (pre-existing)
 2026-06-08 00:54:40 +08:00
xiaowei 82652ab489 storage/embedder: 支持 MODEL_NAME 环境变量指定 embedding 模型 
- Embedder.modelName 替代硬编码的 bge-m3
- 默认值仍是 bge-m3（兼容本机 vLLM）
- Ollama 等第三方服务可通过 MODEL_NAME 覆盖
- encodeRemote 和 fallback 均使用 e.modelName
 2026-06-06 18:22:40 +08:00
xiaowei 2d6981d4fb E5.1 织忆插件 v1.1: 记忆展开详情+分类过滤+质量分条+写记忆入口+设置持久化+图谱边标签+实体列表侧边栏 2026-06-03 20:25:53 +08:00
xiaowei 8d62d35ae7 cleanup: 删除废弃数据目录 /home/muc/data（20K，仅 transaction 文件，实际数据在 /var/lib/memoryweave） 2026-06-03 12:49:54 +08:00
xiaowei 3d054e7590 自验证机制：consolidate 聚类下限 + timestamp 前置检查 + 启动数据目录检查 + 修复图谱扩展字段名 
1. consolidation_pipe.go:
   - 前置：抽样检查记忆时间戳（< 2024-01-01 视为可疑 epoch-0）
   - 后置：clusters <= 1 时记录 ERROR + patterns 标记异常
2. server.go: runStartupChecks() 启动时检测废弃路径 + socket 可达性
3. graph_expander.go: Navigate 返回字段从 "target"/"source" 修正为 "to"/"from"
 2026-06-03 12:44:31 +08:00
xiaowei 612d915eec fix: 移除 episodes 直接写入 recall 结果 
问题：commit 时直接 InsertMemory，原始对话未蒸馏就出现在 recall 结果

修复：
- recall.go: Step 2.5 过滤 episodes/对话类别（filterRecallRawCategories）
- recall.go: cache hit 路径同步加 filter，防止旧缓存污染
- core.go: 移除 commit 时的 InsertMemory，蒸馏完成后由 callback 写入 memories
- engine.go: 蒸馏 prompt 改为结构化提取（decisions/conclusions/actions_taken/open_questions）
- engine.go: max_tokens 300→600，truncate 500→1000

验证：recall('牧尘 小唯 obsidian') → 仅 distilled/system_fact/用户偏好，无 episodes
 2026-06-03 01:36:45 +08:00
xiaowei 7ec0642b9d feat: E1.1-E1.7 图谱 BFS 扩展全部完成 
E1.1: InMemoryGraph.NavigateBiDir 真正双向 BFS（替代伪实现）
E1.2: 无相遇节点返回 {unreachable:true} 而非降级单向
E1.3: 规则 NER（extractEntitiesWithNER, 7个正则模式）
E1.4: relationFilter 支持（buildRelationFilterClause, SQL注入）
E1.5: API 层 relation_filter 参数（navigate 端点）
E1.6: API 层 relation_filter 参数（同上，已在 navigate 中支持）
E1.7: 环路检测（seenEdges map，同一边不在单次 BFS 中重复访问）

同时修复: graph_expander.go Navigate 调用加 relFilter=nil
 2026-06-02 20:26:57 +08:00
xiaowei 48e33039a0 fix: SQLite WAL mode 解决图谱导航超时问题 
- 启用 PRAGMA journal_mode=WAL（写操作不阻塞读）
- busy_timeout 从 10s 降至 3s（WAL 模式下锁竞争大幅减少）
- 更新 eval_results.md 缺陷状态（P0 已修复）
 2026-06-02 19:10:03 +08:00
xiaowei 40a8c9edee feat: E1 BFS 扩展 + 生产部署完整套件 
E1 图谱导航:
- Add ExpandWithSummary (BFS 扩展 + LLM 汇总)
- Add GraphBFSResult/ExpandedRelation 模型
- Fix NavigateBiDir 伪实现问题 (docs/BFS_GRAPH_EXPANSION_DESIGN.md)

生产部署:
- README.md: 完整安装/配置/API 文档
- INSTALL.md: systemd 手动安装指南
- docker-compose.yml: 一键部署 (zhiyid + redis + bge-m3)
- Makefile: VERSION/version/build-web/install-all/docker-* 目标
- systemd: MemoryMax/CPUQuota 限制，完善环境变量

集成支持:
- eval_results.md: 性能基准测试报告
- scripts/benchmark.sh / benchmark.py: 可重复性能测试
- tests/integration_test.sh: 端到端集成测试

已知问题: graph/navigate 超时 (P0)，见 eval_results.md
 2026-06-02 18:58:38 +08:00
xiaowei 0bda9a8420 fix: 静态文件路径 / 在 Auth 白名单中放行（供 Web UI 直接访问） 2026-06-02 17:34:28 +08:00
xiaowei 293ae82fb2 docs: E5 全部完成，状态更新为全部完成 ✅ 2026-06-02 17:12:19 +08:00
xiaowei 4501d93717 E5.3 Web UI: React 单文件 + 静态文件中间件 
- web-ui/index.html: 4页 SPA（记忆/图谱/搜索/蒸馏），Babel standalone JSX
- Go 静态文件中间件: catch-all handler 检查 web-ui 目录
- Makefile: build-web / install-all 目标
 2026-06-02 17:10:28 +08:00
xiaowei 420b48ae72 E5.2 增强 CLI: zhiyi stats/tree/recall/graph/entity 命令 
- 零外部依赖，纯 Go stdlib flag 实现
- stats: 系统状态 + 蒸馏配额
- tree: 按 category 分组显示记忆
- recall: 语义搜索（/api/v1/recall）
- graph: ASCII 图谱（/api/v1/graph/navigate）
- entity: 实体详情（邻居数 + 关联记忆）
- Makefile 新增 build-cli 目标
 2026-06-02 16:32:18 +08:00
xiaowei 8a3eaae122 docs: E5.1 完成，E5.2 待启动 2026-06-02 15:43:46 +08:00
xiaowei 9c02d16322 docs: WORKLOG 同步 E5.1 Obsidian 插件 2026-06-02 15:43:31 +08:00
xiaowei abef38cb05 E5.1 Obsidian 插件: CORS中间件 + 记忆面板 + 图谱视图 + 搜索模态框 
- 新增 CORS 中间件 (middleware/cors.go): 支持 app://obsidian.md 跨域
- 新增 Obsidian 插件: MemoryView(分页记忆列表) + GraphView(D3力导向图) + SearchModal(语义搜索)
- Go API CORS: Access-Control-Allow-Origin: app://obsidian.md
- Makefile 新增 build-obsidian / install-obsidian 目标
- 插件安装至 ~/.obsidian/plugins/zhiyi-memory/
 2026-06-02 15:42:54 +08:00
xiaowei f93da87a97 fix: distill端点 /status /queue /quota（Engine新增导出方法）；WORKLOG同步 2026-06-02 12:28:00 +08:00
xiaowei 8db0138b06 G8: Restore+ListBackups 端点；G9: SearchCache 双级缓存（内存+Redis L2） 2026-06-02 11:43:34 +08:00
xiaowei bbce18f748 docs: 同步 E4.3 调用方已接入；G7 E3 自优化闭环完成记录 2026-06-02 09:55:10 +08:00
xiaowei b0059166a2 fix G7 E3: crystallize路由路径+GetSkillCandidates通过Rust IPC查LanceDB 
- server.go: 修复mux路由路径（去掉"POST "前缀）
- skill_crystallize.go: 添加POST方法检查
- models/memory.go: 添加safeTime类型避免空字符串解析报错
- lancedb_ipc.go: GetSkillCandidates改用Rust IPC的lancedb_query
- Rust lancedb_ops.rs: 新增query_memories()支持min_recall过滤
- Rust main.rs: 新增lancedb_query IPC handler
 2026-06-02 04:19:58 +08:00
xiaowei 243a2066a4 G7.3: skill execute + quality_score fix 
G7.3 — Skill执行+遗忘联动:
- skill_execute.go: ExecuteSkill handler, applyForgettingLinkage
- /api/v1/skills/{name}/execute: 返回 enriched_prompt + linked memories
- Trial feedback 联动: success → degree保护, failure → decay加速

Fix quality_score=0 (Step 2):
- main.rs: stratified_sample 用 tier 字段，但 memories 全为 "normal"
- 改为 r.freshness（默认为 "fresh"），匹配 stratified_sample 期望的 tier 值

G7.1+G7.2 已在运行（skill persistence + crystallize API 验证通过）
 2026-06-02 00:48:36 +08:00
xiaowei 43286dd2d7 G7.1+G7.2: skill persistence + crystallize API 
G7.1 — Skill 持久化:
- BetaSkill: prompt_template, linked_memory_ids, linked_entities, created_at
- BayesianSkillManager: EnableRedisPersistence(), loadFromRedis(), persistSkill()
- Redis HASH (zhiyi:skills) 持久化，重启不丢 skills
- Skill CRUD API: POST/GET/DELETE /api/v1/skills/{name}
- auto_distill skill: ETA=0.93, status=active after 12 trials ✓

G7.2 — Skill 结晶:
- GetSkillCandidates(minRecalls, limit): storage.LanceDB 接口扩展
- SQLiteClient: SQL 查询候选（recall_count>=5, quality_score>=0.7）
- RustLanceDBClient/MemLanceClient: stub 实现
- /api/v1/crystallize/candidates: 获取候选记忆
- /api/v1/crystallize/memory/{id}: 对记忆执行 LLM 结晶
- callSkillLLM(): LLM 生成 prompt 模板（带降级）
- RegisterLDBGetter(): routes 包访问 LanceDB 实例
 2026-06-02 00:01:59 +08:00
xiaowei 137c25f16f G7.1 fix: skillByNameHandler method dispatch, Redis persistence verified 
- skillByNameHandler: GET/DELETE on same pattern via r.Method switch
- EnableRedisPersistence at startup: confirmed working via logs + curl
- Redis HGET zhiyi:skills: full skill state persisted correctly
- Trial feedback: ETA 0.5 → 0.67 after 1 success (Bayesian update ✓)
 2026-06-01 23:48:51 +08:00
xiaowei 4f2c0fe769 G7.1: skill persistence - BayesianSkillManager Redis + CRUD API 
- BetaSkill extended: PromptTemplate, LinkedMemoryIDs, LinkedEntities, CreatedAt
- BayesianSkillManager: EnableRedisPersistence(), loadFromRedis(), persistSkill()
- Skill CRUD API: POST/GET/DELETE /api/v1/skills/{name}
- Skill Manager: Register(), Get(), Delete(), Stats()
- redis.go: added HDel method
- triggers.go: removed duplicate Skill/SkillManager (now in skill_bayes.go)
- server.go: EnableRedisPersistence() at startup + new routes
 2026-06-01 23:44:36 +08:00
xiaowei 8b01ab2d90 docs: add E4.3 bug fix to WORKLOG 2026-06-01 22:55:32 +08:00
xiaowei df2b648973 fix E4.3: extract entities from content instead of namespace 
E4.3 bug: admin.go used mem["namespace"] which is empty (LanceDB
map keys are different from struct field names). Fixed by:
- Replace namespace lookup with content-based entity extraction
- Reuse distill/engine.go heuristic: capitalized words, Chinese
  entities (2-20 chars), tech tokens (alphanumeric/digits)
- Get max graph degree across all extracted entities
- Add isStopWord, stripNonChinese, isTechToken, isAllDigits helpers

Fixes: degree always 0 in ShouldForget call, graph integration broken.
 2026-06-01 22:49:50 +08:00
xiaowei 525b5f2bb2 fix E4.3: 回退 FileGraph，恢复 SQLiteGraphStore（数据在 graph.db） 
切换 FileGraph 导致 9.9MB 历史图谱数据丢失。
恢复 NewSQLiteGraphStore，GetEntityDegree 加到 SQLiteGraphStore。
验证：1720 节点，32983 边。
 2026-06-01 01:16:31 +08:00
xiaowei 2d0923a85c E4.3: 图谱度参与遗忘决策（完整接入） 
- GraphStore 接口添加 GetEntityDegree(entity) int
- InMemoryGraph + FileGraph 实现（复用 EvidenceCount 逻辑）
- GetCandidatesForForgetting 返回 namespace + content 字段（存储层）
- AdminAPI 添加 GraphStore 字段，admin.go 的 Forget 接入
- server.go decay 循环接入：namespace 作为实体查图谱度
- 修复预存在错误：SQLiteGraphStore → FileGraph（无 SQLite 图谱）
- 修复预存在错误：consolidation_pipe PageRank 类型断言删掉（FileGraph 自更新）
 2026-05-31 22:29:28 +08:00
xiaowei 3c27ec8322 E4.1 中文感知修复文档补充 2026-05-31 16:23:05 +08:00
xiaowei 51eec76f1f fix E4.1: IsContradiction 中文感知修复（字符级切分+否定词检测） 
问题：原 IsContradiction 使用 strings.Fields()，中文无空格时整句为一个词，
导致 overlap 始终为 0，矛盾检测永远失败。

修复：
- 新增 splitWordsCN：中文按 unicode.Han 字符级切分，英文按空格分词
- 新增 containsNegCN：检测中文否定词（不是/没有/不/没/莫/别）
- IsContradiction 合并中文/英文两种检测逻辑，阈值调至 0.3

验证：Test 2 返回 {"conflicts": ["小唯是牧尘的女朋友"]} ✅
 2026-05-31 16:22:35 +08:00
xiaowei 03fe336616 E4 图谱推理实现文档：E4.1 矛盾检测+E4.2 跨agent recall+E4.3 遗忘参考图谱度 
E4.1: ConflictDetector 注入 core.go:API, Commit 时对相似记忆运行 DetectContradiction()
E4.2: Recall 结果 < 3 时补充 shared namespace（去重 + Score=0.5 降权）
E4.3: ShouldForget(graphDegree ...int) 可变参数，度>5 时每度+0.03保留分
E4.3 调用方待接入 graphDegree（admin.go:71, server.go:904）
 2026-05-31 16:13:29 +08:00
xiaowei a6403dd088 E4 图谱推理：E4.1 矛盾检测+E4.2 跨agent recall+E4.3 遗忘参考图谱度 
E4.1: ConflictDetector 注入 core.go:API, Commit 时对相似记忆运行矛盾检测（IsContradiction），有冲突时在响应中返回 conflicts 字段
E4.2: Recall 结果 < 3 时自动补充搜索 shared namespace，合并跨 agent 结果
E4.3: Forgetter.ShouldForget 加 graphDegree 可选参数，节点度 > 5 时每度 +0.03 保留分
 2026-05-31 16:03:21 +08:00
xiaowei 1ef1747769 E3 完成：增量 embedding 验证 
E3 已实现（代码早已存在，本次验证确认有效）：
- Go Commit() 在 core.go:79 调用 Embedder.EncodeSingle() → BGE HTTP 8000
- vector 随文本通过 IPC lancedb_insert 发送到 Rust，直接存储不重编码
- 验证：commit 后立即 recall，测试记忆排第一得分 0.843 ✅

更新：
- IMPLEMENTATION-FIVE.md: E3 现状修正（已实现）+ 验收标准打勾
- WORKLOG.md: E3 状态 + 下一步改为 E4 图谱推理
- 状态: E1/E2/E3 已完成，E4-E5 规划中
 2026-05-31 15:17:35 +08:00
xiaowei f65dee6a3d E1/E2 完成：图谱导航激活 + 多 agent 命名空间验证 
E1 — 图谱导航激活：
- 移除 ExpandFromResults 中 5 条残留 debug fmt.Printf 语句（graph_sqlite.go）
- Recall pipeline 已通过 server.go:112 SetGraphExpander 接入，无需额外代码

E2 — 多 agent 命名空间激活（验证完成，无需代码改动）：
- Hermes: agent_id=hermes-a06 → namespace=hermes-main（deriveNamespace）
- OpenClaw: agent_id=openclaw, namespace=openclaw-main（zhiyi client 显式传递）
- Rust sidecar: search/scan_all 均有 namespace 过滤（only_if）
- 验证: hermes-main vs default-main 同一 query 返回不同 ID，隔离生效

更新：
- WORKLOG.md: E1/E2 状态 + 详细记录
- IMPLEMENTATION-FIVE.md: E2 现状修正（代码早已实现）+ 验收标准打勾
- 状态: 规划阶段 → E1/E2 已完成
 2026-05-31 15:05:32 +08:00
xiaowei cbe4026834 E1: remove debug fmt.Printf from ExpandFromResults, update WORKLOG 
- Remove 5 residual fmt.Printf("[E1] ...") debug statements from
  SQLiteGraphStore.ExpandFromResults (graph_sqlite.go)
- Recall pipeline already has graph expansion wired via server.go:112
  (SetGraphExpander(graphStore)) — feature was implemented previously
- ExpandFromResults trigger: len(results) < 5, BFS 1-hop, dedup by ID
- Update WORKLOG: E1 completed, G6.1 cluster_only mode explained,
  G6.2/G6.3 consolidated, add cd0f898/cc615c5 commits, update 下一步
 2026-05-31 14:50:56 +08:00
xiaowei cd0f898829 fix: eps=1.0 (19 clusters from 1486 items), add col_vector() for FixedSizeListArray reading 
- eps: 0.1 → 1.0 (p50=1.011, meaningful semantic clustering)
- col_vector(): parse FixedSizeListArray and Float32Array for LanceDB vector column
- search() and scan_all() now read actual vectors instead of vec![]
- client.go: remove duplicate Epsilon field, eps=1.0
 2026-05-31 14:35:29 +08:00
xiaowei cc615c54d9 docs: add consolidate fix work log (2026-05-31) 2026-05-31 11:58:26 +08:00