4c62074a61
fix(consolidate): 按目标分组批量LLM决策,消除O(n²)调用风暴
...
原实现每对高相似记忆调一次LLM → limit=100产生4950对 → 223s+未完成
→ 每天processed=0(非namespace过滤bug)。改为: 外层目标i,内层收集
全部高相似候选,每目标一次LLM(UpdatePrompt原生支持多候选)。
验证: limit=20 → 4.3s返回 processed=1 updated=1; limit=100 2min(受
llama单slot排队限制)对每日低频可用。
2026-09-06 00:32:27 +08:00
0881b9c05f
fix(softdelete): 单独应用 SoftDelete 持久化修复(506f76b+SoftDelete only, 不含 lancedb_scan 风暴代码)
...
从 febc2c9
2026-09-06 00:23:23 +08:00
8356480541
fix(consolidate): eps 1.0->0.4 修复 DBSCAN 聚类失效 (clusters=1->34)
...
根因: bge-m3 向量已归一化(norm=1.0, 距离 p50=0.42-0.75), 旧 eps=1.0 按未归一化数据(p50=1.029)调试,
在归一化空间过大 -> 全部连成1簇 -> consolidate 空转多日 (clusters=1 error 每15min刷)
实测: eps=0.4 -> clusters=34 noise=1030, 能发现重复状态噪音簇(1045条主profile状态同步/961条CBM状态)
2026-09-05 18:13:53 +08:00
63f123417b
fix(distill): commit 检出冲突时不进自动蒸馏（conflict guard）
...
问题：commit 的 DetectContradiction 检出矛盾后只返回 conflicts 字段，
内容仍进 episodes→自动蒸馏→污染 distilled/memories（TencentDB 误判案例根因链一环）
修复：conflicts>0 时跳过 AutoDistillTrigger，返回 distill_skipped=conflict；
处理路径：①写错→feedback not_useful；②旧记忆错→feedback 降权；
③确认为修正→resolve_conflict=true 重 commit（待客户端支持）
验证：commit 矛盾内容 → conflicts 返回 + distill_skipped=conflict ✅
2026-09-04 01:29:11 +08:00
ae73982743
fix(distill): max_tokens 800→1200 — Agnes推理token预算不足
...
Agnes 2.0-flash是推理模型，推理token占大量预算。
800仍不够→JSON截断(parse error: unexpected end of JSON input)。
1200确保推理+正文都够用。
2026-08-20 01:13:06 +08:00
ea4b5a4dfe
fix(distill): max_tokens 600→800 — Agnes推理模型JSON不再截断
...
engine.go: max_tokens 600→800
consolidate.go: max_tokens 300→800
根因: Agnes 2.0-flash是推理模型，max_tokens=600里大量是推理token，
正文只剩~300→JSON截断→parse error→蒸馏退化。
800确保推理+正文都够用。
2026-08-20 00:10:51 +08:00
38701748d1
feat(distill): P4 AAAK压缩索引 + P5 Markdown导出层
...
P4: 借鉴 mempalace AAAK dialect — 每条事实生成紧凑索引(实体|关键词|权重|类型)
P5: 借鉴 EverOS md真相层 — /api/v1/memories/export 导出人可读 Markdown
2026-08-12 01:37:27 +08:00
82c3d25423
feat(distill): P2 离线整合(UPDATE_PROMPT) + P3 双缓冲(token积累) + JSON健壮剥离修复
...
- P2: consolidate.go 新增 ConsolidateMemory + TextSimilarity，LLM三选一(update/delete/ignore)合并相似记忆
- P2: server.go 新增 POST /api/v1/consolidate/memory 手动触发端点
- P3: engine.go Enqueue 按 token 积累触发 flush(阈值2000)，batchTimeout兜底
- fix: LLM JSON 剥离增强(找首个{和最后}截取)，修复模型返回markdown/注释导致的parse error
2026-08-11 21:38:10 +08:00
0734ffaa5a
feat(distill): LightMem式逐条事实提取 prompt（P1）— 从整段摘要升级为逐条独立事实，保留全部实体细节+时间区分+推断隐含信息
2026-08-11 17:51:08 +08:00
11162e2f60
fix: consolidation 风暴 — distill cooldown 60s→15min, cluster_only 跳过 PageRank
...
- triggers.go: TriggerDistill cooldown time.Minute → 15 * time.Minute
- consolidation_pipe.go: runGraphMaintenance 仅 full 模式执行（cluster_only 高频快速聚类跳过重负载 PageRank）
- 修复 zhiyid CPU 80-90% 风暴、recall API 120s+ 超时
根因：t_distill 每 60s 触发全量 DBSCAN + PageRank(10356 nodes)，数据量大时单次 30-60s CPU 堆积
2026-08-10 01:21:57 +08:00
98e541b985
fix: distill LLM 模型切换 gemma-4-31b-it + JSON 解析增强
...
根因链:
1. gpt-oss-120b 是 reasoning 模型，content=null 答案全在 reasoning 字段，
代码只读 content/reasoning_content（OpenAI 用 reasoning 字段名）→ parse 失败
→ 永远降级 keyword 提取 (facts=1 entities=0)
2. skill 记载的 m3/m2.7/mistral-large 均已 EOL 或渠道失效 (2026-07-27 后)
3. gemma-4-31b-it 对 '0.X' 占位符输出 0.0 → prompt 改为明确 0-1 浮点说明
修复:
- engine.go: 支持 reasoning 字段 + 剥离 markdown code fence + prompt 评分说明明确化
- zhiyid.service: LLM_MODEL=google/gemma-4-31b-it
验证: LLM entities=2, overall=1.000, facts=3 entities=2, recall 命中 0.765
2026-08-02 23:59:29 +08:00
21bc777430
fix: episodes 持久化到 LanceDB（重启不再清零）
...
根因: RustLanceDBClient.InsertEpisode 只写进程内存 _local.episodes，
从未持久化；Stats 的 total_episodes 用内存值覆盖 Rust 真实计数。
修复:
- InsertEpisode 增加 IPC 写入 LanceDB episodes 表（24 字段完整 schema）
- Stats 优先用 Rust sidecar 返回的 episodes 计数，缺失才回退内存
- 持久化失败只记日志不阻断请求（保持'总是写入'语义）
验证: 提交测试 episode → stats episodes=1 → 重启 zhiyid → 仍为 1
历史 45 条内存 episodes 已随重启丢失（原始日志，蒸馏结果在 memories 5063 条保留）
2026-08-02 23:41:37 +08:00
c4d17df71c
fix: Phase D/G/H 编译修复
...
- governance.go: 加 PendingCount() 方法给 ConflictsPending metrics
- distill/: 删 consolidation.go（死代码，231行无引用）
- scripts/: backup.sh 加7天保留策略 + restore.sh 恢复脚本
2026-07-10 05:10:12 +08:00
57ac628b3f
fix: H1-H6 gaps all resolved
...
H1: BM25 keyword scoring in recall pipeline (0.7 vector + 0.3 keyword)
H2: LLM wiki curation mode (--llm flag, graceful heuristic fallback)
H3: Auto trust score update after each recall call
H4: Default diversity=0.3 (was 0 = no diversity)
H5: Three search modes: hybrid(semantic+BM25) / keyword / semantic
H6: Multi-tier fallback already covered by P0 + SQLiteClient
All verified: hybrid(0.962), keyword(1.000), semantic(0.962)
2026-07-02 00:43:46 +08:00
72cbf73583
feat: P0 recall fallback + P2 trust scoring for graph edges
...
P0: When bge-embed/IPC recall fails, fall back to graph.db keyword search (FallbackTextSearch) instead of 500 error. Response includes X-Fallback: graph header.
P2: Add trust_score, retrieval_count, helpful_count columns to graph_edges table. New POST /api/v1/graph/edge/feedback endpoint. UpdateEdgeTrustScores batch calculation.
2026-07-02 00:22:28 +08:00
dc8d074cf5
fix: 6 Go code quality fixes + 4 plugin bug fixes
2026-06-20 16:51:13 +08:00
12c4c58c1c
fix: re-enable /api/v1/memories list endpoint (via AdminAPI)
2026-06-20 16:24:52 +08:00
23c6547503
perf: add graph navigate cache (TTL 5min, 500 entries) + cache invalidation + cache stats API
...
- NewCachedGraphStore: governance.GraphStore wrapper, only Navigate() is cached
- GraphCacheRef: expose InvalidateAll() + Stats() for cache management
- Cache invalidation on graph/edge add, graph/cleanup (non-dry-run)
- New /api/v1/cache/stats endpoint for cache monitoring
- /api/v1/health now registered at /api/v1/health path too
- normalizeEntity: preserve Unicode letters (Chinese chars not stripped)
2026-06-16 18:09:51 +08:00
d6188a2bd7
fix: health endpoint + graph/edge API + normalizeEntity Chinese + nl_query comparison
...
1. Add /api/v1/health endpoint (maps to HandleHealth)
2. Add POST /api/v1/graph/edge for adding relation edges
3. Fix normalizeEntity to preserve Chinese characters (0x4e00-0x9fa5)
4. Fix nl_query direct_path comparison to use normalizeEntity on both sides
2026-06-16 16:16:35 +08:00
bd8008b3fd
fix: comment out api.ListMemories (removed from routes)
2026-06-15 18:34:12 +08:00
a76b6d3afa
feat(graph): add graph cleanup + nl_query + navigate grouped format
...
- Add CleanupNoiseNodes() to GraphStore interface + SQLiteGraphStore impl
- Add /api/v1/graph/cleanup endpoint (dry_run + execute)
- Add /api/v1/graph/nl_query endpoint for natural language graph queries
- Improve /api/v1/graph/navigate: add grouped_by_relation, suggestions, normalized_entity
- Add CleanupNoiseNodes stubs to InMemoryGraph and FileGraph
BREAKING: navigate response now includes grouped_by_relation and suggestions
2026-06-15 18:09:27 +08:00
b9f9f75c31
fix: IsContradiction typo in bench_test.go
2026-06-15 11:08:13 +08:00
8ad4a7c794
feat: add freshness field to memory lifecycle
...
- server.go: set freshness='fresh' when distill creates new memory
- recall.go: set freshness='verified' when memory is recalled
- Freshness values: fresh (new), verified (recalled), stale (decayed)
2026-06-15 10:39:15 +08:00
42a8aeb53c
api/server: only record distill_loss when Overall>0, exclude fallback pollution
2026-06-12 11:43:05 +08:00
cf31ff5e8e
api: add /api/v1/metrics endpoint exposing Dashboard metrics (avg_distill_loss, etc.)
2026-06-12 08:54:00 +08:00
90c43c0717
fix(distill): increase HTTP client timeout 30s→120s for LLM calls
2026-06-11 10:39:07 +08:00
b237cc9aeb
fix(backup): add 5min timeout to tar, fix comment
2026-06-10 10:16:49 +08:00
2c96f805f4
fix: consolidate trigger uses full mode instead of cluster_only
...
Problem: server.go 30s ticker loop called consolPipe.Run() which
defaults to cluster_only mode. The 'consolidation' and 'backtrack'
actions never executed prune/decay/quality steps.
Fix:
- Separate 'distill' action (keeps cluster_only for fast frequent runs)
- 'consolidation' and 'backtrack' now call RunWithMode(full)
- selfoptimize.Flow.Register('consolidate') also uses RunWithMode('full')
- Proper case indentation (4 tabs for labels, 5 for bodies)
Result: prune + decay + quality steps now run when t_consolidation
or t_backtrack fires (48h cooldown protects against over-firing)
2026-06-09 12:45:33 +08:00
1516fe5bc8
fix: add clusters_found/noise_points/quality_score to consolidate API response
2026-06-08 10:35:10 +08:00
f286348ad8
fix: gap auto-close + conflict_muchen handler
...
- GapDetector.RecordHit(topic): recall 命中后自动关闭该 topic 的 open gap
- GapDetector.ClearMisses(topic): 清除 miss 计数，避免重复触发
- SetGlobalGapDetector / GetGlobalGapDetector: 全局实例注册，供 recall 回调使用
- recall handler (core.go): 命中时调用 RecordHit + ClearMisses 实现 gap auto-close
- conflict_muchen handler (server.go): 注册处理器，自动裁决 pending conflict（latest_wins / primary_wins / dismiss）
- fix: server_sqlite_nowindows.go 修复 osGetenv → os.Getenv (pre-existing)
2026-06-08 00:54:40 +08:00
82652ab489
storage/embedder: 支持 MODEL_NAME 环境变量指定 embedding 模型
...
- Embedder.modelName 替代硬编码的 bge-m3
- 默认值仍是 bge-m3（兼容本机 vLLM）
- Ollama 等第三方服务可通过 MODEL_NAME 覆盖
- encodeRemote 和 fallback 均使用 e.modelName
2026-06-06 18:22:40 +08:00
3d054e7590
自验证机制：consolidate 聚类下限 + timestamp 前置检查 + 启动数据目录检查 + 修复图谱扩展字段名
...
1. consolidation_pipe.go:
- 前置：抽样检查记忆时间戳（< 2024-01-01 视为可疑 epoch-0）
- 后置：clusters <= 1 时记录 ERROR + patterns 标记异常
2. server.go: runStartupChecks() 启动时检测废弃路径 + socket 可达性
3. graph_expander.go: Navigate 返回字段从 "target"/"source" 修正为 "to"/"from"
2026-06-03 12:44:31 +08:00
612d915eec
fix: 移除 episodes 直接写入 recall 结果
...
问题：commit 时直接 InsertMemory，原始对话未蒸馏就出现在 recall 结果
修复：
- recall.go: Step 2.5 过滤 episodes/对话类别（filterRecallRawCategories）
- recall.go: cache hit 路径同步加 filter，防止旧缓存污染
- core.go: 移除 commit 时的 InsertMemory，蒸馏完成后由 callback 写入 memories
- engine.go: 蒸馏 prompt 改为结构化提取（decisions/conclusions/actions_taken/open_questions）
- engine.go: max_tokens 300→600，truncate 500→1000
验证：recall('牧尘 小唯 obsidian') → 仅 distilled/system_fact/用户偏好，无 episodes
2026-06-03 01:36:45 +08:00
7ec0642b9d
feat: E1.1-E1.7 图谱 BFS 扩展全部完成
...
E1.1: InMemoryGraph.NavigateBiDir 真正双向 BFS（替代伪实现）
E1.2: 无相遇节点返回 {unreachable:true} 而非降级单向
E1.3: 规则 NER（extractEntitiesWithNER, 7个正则模式）
E1.4: relationFilter 支持（buildRelationFilterClause, SQL注入）
E1.5: API 层 relation_filter 参数（navigate 端点）
E1.6: API 层 relation_filter 参数（同上，已在 navigate 中支持）
E1.7: 环路检测（seenEdges map，同一边不在单次 BFS 中重复访问）
同时修复: graph_expander.go Navigate 调用加 relFilter=nil
2026-06-02 20:26:57 +08:00
48e33039a0
fix: SQLite WAL mode 解决图谱导航超时问题
...
- 启用 PRAGMA journal_mode=WAL（写操作不阻塞读）
- busy_timeout 从 10s 降至 3s（WAL 模式下锁竞争大幅减少）
- 更新 eval_results.md 缺陷状态（P0 已修复）
2026-06-02 19:10:03 +08:00
40a8c9edee
feat: E1 BFS 扩展 + 生产部署完整套件
...
E1 图谱导航:
- Add ExpandWithSummary (BFS 扩展 + LLM 汇总)
- Add GraphBFSResult/ExpandedRelation 模型
- Fix NavigateBiDir 伪实现问题 (docs/BFS_GRAPH_EXPANSION_DESIGN.md)
生产部署:
- README.md: 完整安装/配置/API 文档
- INSTALL.md: systemd 手动安装指南
- docker-compose.yml: 一键部署 (zhiyid + redis + bge-m3)
- Makefile: VERSION/version/build-web/install-all/docker-* 目标
- systemd: MemoryMax/CPUQuota 限制，完善环境变量
集成支持:
- eval_results.md: 性能基准测试报告
- scripts/benchmark.sh / benchmark.py: 可重复性能测试
- tests/integration_test.sh: 端到端集成测试
已知问题: graph/navigate 超时 (P0)，见 eval_results.md
2026-06-02 18:58:38 +08:00
0bda9a8420
fix: 静态文件路径 / 在 Auth 白名单中放行（供 Web UI 直接访问）
2026-06-02 17:34:28 +08:00
4501d93717
E5.3 Web UI: React 单文件 + 静态文件中间件
...
- web-ui/index.html: 4页 SPA（记忆/图谱/搜索/蒸馏），Babel standalone JSX
- Go 静态文件中间件: catch-all handler 检查 web-ui 目录
- Makefile: build-web / install-all 目标
2026-06-02 17:10:28 +08:00
420b48ae72
E5.2 增强 CLI: zhiyi stats/tree/recall/graph/entity 命令
...
- 零外部依赖，纯 Go stdlib flag 实现
- stats: 系统状态 + 蒸馏配额
- tree: 按 category 分组显示记忆
- recall: 语义搜索（/api/v1/recall）
- graph: ASCII 图谱（/api/v1/graph/navigate）
- entity: 实体详情（邻居数 + 关联记忆）
- Makefile 新增 build-cli 目标
2026-06-02 16:32:18 +08:00
abef38cb05
E5.1 Obsidian 插件: CORS中间件 + 记忆面板 + 图谱视图 + 搜索模态框
...
- 新增 CORS 中间件 (middleware/cors.go): 支持 app://obsidian.md 跨域
- 新增 Obsidian 插件: MemoryView(分页记忆列表) + GraphView(D3力导向图) + SearchModal(语义搜索)
- Go API CORS: Access-Control-Allow-Origin: app://obsidian.md
- Makefile 新增 build-obsidian / install-obsidian 目标
- 插件安装至 ~/.obsidian/plugins/zhiyi-memory/
2026-06-02 15:42:54 +08:00
f93da87a97
fix: distill端点 /status /queue /quota（Engine新增导出方法）；WORKLOG同步
2026-06-02 12:28:00 +08:00
8db0138b06
G8: Restore+ListBackups 端点；G9: SearchCache 双级缓存（内存+Redis L2）
2026-06-02 11:43:34 +08:00
b0059166a2
fix G7 E3: crystallize路由路径+GetSkillCandidates通过Rust IPC查LanceDB
...
- server.go: 修复mux路由路径（去掉"POST "前缀）
- skill_crystallize.go: 添加POST方法检查
- models/memory.go: 添加safeTime类型避免空字符串解析报错
- lancedb_ipc.go: GetSkillCandidates改用Rust IPC的lancedb_query
- Rust lancedb_ops.rs: 新增query_memories()支持min_recall过滤
- Rust main.rs: 新增lancedb_query IPC handler
2026-06-02 04:19:58 +08:00
243a2066a4
G7.3: skill execute + quality_score fix
...
G7.3 — Skill执行+遗忘联动:
- skill_execute.go: ExecuteSkill handler, applyForgettingLinkage
- /api/v1/skills/{name}/execute: 返回 enriched_prompt + linked memories
- Trial feedback 联动: success → degree保护, failure → decay加速
Fix quality_score=0 (Step 2):
- main.rs: stratified_sample 用 tier 字段，但 memories 全为 "normal"
- 改为 r.freshness（默认为 "fresh"），匹配 stratified_sample 期望的 tier 值
G7.1+G7.2 已在运行（skill persistence + crystallize API 验证通过）
2026-06-02 00:48:36 +08:00
43286dd2d7
G7.1+G7.2: skill persistence + crystallize API
...
G7.1 — Skill 持久化:
- BetaSkill: prompt_template, linked_memory_ids, linked_entities, created_at
- BayesianSkillManager: EnableRedisPersistence(), loadFromRedis(), persistSkill()
- Redis HASH (zhiyi:skills) 持久化，重启不丢 skills
- Skill CRUD API: POST/GET/DELETE /api/v1/skills/{name}
- auto_distill skill: ETA=0.93, status=active after 12 trials ✓
G7.2 — Skill 结晶:
- GetSkillCandidates(minRecalls, limit): storage.LanceDB 接口扩展
- SQLiteClient: SQL 查询候选（recall_count>=5, quality_score>=0.7）
- RustLanceDBClient/MemLanceClient: stub 实现
- /api/v1/crystallize/candidates: 获取候选记忆
- /api/v1/crystallize/memory/{id}: 对记忆执行 LLM 结晶
- callSkillLLM(): LLM 生成 prompt 模板（带降级）
- RegisterLDBGetter(): routes 包访问 LanceDB 实例
2026-06-02 00:01:59 +08:00
137c25f16f
G7.1 fix: skillByNameHandler method dispatch, Redis persistence verified
...
- skillByNameHandler: GET/DELETE on same pattern via r.Method switch
- EnableRedisPersistence at startup: confirmed working via logs + curl
- Redis HGET zhiyi:skills: full skill state persisted correctly
- Trial feedback: ETA 0.5 → 0.67 after 1 success (Bayesian update ✓)
2026-06-01 23:48:51 +08:00
4f2c0fe769
G7.1: skill persistence - BayesianSkillManager Redis + CRUD API
...
- BetaSkill extended: PromptTemplate, LinkedMemoryIDs, LinkedEntities, CreatedAt
- BayesianSkillManager: EnableRedisPersistence(), loadFromRedis(), persistSkill()
- Skill CRUD API: POST/GET/DELETE /api/v1/skills/{name}
- Skill Manager: Register(), Get(), Delete(), Stats()
- redis.go: added HDel method
- triggers.go: removed duplicate Skill/SkillManager (now in skill_bayes.go)
- server.go: EnableRedisPersistence() at startup + new routes
2026-06-01 23:44:36 +08:00
df2b648973
fix E4.3: extract entities from content instead of namespace
...
E4.3 bug: admin.go used mem["namespace"] which is empty (LanceDB
map keys are different from struct field names). Fixed by:
- Replace namespace lookup with content-based entity extraction
- Reuse distill/engine.go heuristic: capitalized words, Chinese
entities (2-20 chars), tech tokens (alphanumeric/digits)
- Get max graph degree across all extracted entities
- Add isStopWord, stripNonChinese, isTechToken, isAllDigits helpers
Fixes: degree always 0 in ShouldForget call, graph integration broken.
2026-06-01 22:49:50 +08:00
525b5f2bb2
fix E4.3: 回退 FileGraph，恢复 SQLiteGraphStore（数据在 graph.db）
...
切换 FileGraph 导致 9.9MB 历史图谱数据丢失。
恢复 NewSQLiteGraphStore，GetEntityDegree 加到 SQLiteGraphStore。
验证：1720 节点，32983 边。
2026-06-01 01:16:31 +08:00
2d0923a85c
E4.3: 图谱度参与遗忘决策（完整接入）
...
- GraphStore 接口添加 GetEntityDegree(entity) int
- InMemoryGraph + FileGraph 实现（复用 EvidenceCount 逻辑）
- GetCandidatesForForgetting 返回 namespace + content 字段（存储层）
- AdminAPI 添加 GraphStore 字段，admin.go 的 Forget 接入
- server.go decay 循环接入：namespace 作为实体查图谱度
- 修复预存在错误：SQLiteGraphStore → FileGraph（无 SQLite 图谱）
- 修复预存在错误：consolidation_pipe PageRank 类型断言删掉（FileGraph 自更新）
2026-05-31 22:29:28 +08:00
小唯
xiaowei
小唯
小唯
xiaowei