feat(v2.9): bge-m3 1024维embedding + bge-reranker-v2-m3重排

- 新增 bge_embedder.py: 模力方舟 bge-m3 API, 1024维向量
- 新增 jina_rerank.py: 模力方舟 bge-reranker-v2-m3 重排API
- recall pipeline: bge-m3 encode + FAISS粗排 + bge-reranker二次重排
- FAISSIndex.index_with_vectors(): 支持预计算向量全量构建
- FAISSRecall.index_documents_with_vectors(): 全量重建索引
- commit.py: 全量索引重建，从JSONLShardStore读取完整数据
- 修复: 数据一致性(FAISS=sbert_docs=stats全部同步)
- recall验证: 新commit内容score=0.714命中

data/rerank_cache.jsonl

@@ -0,0 +1 @@
+{"key": "e5afe2aac0023b516a5f13c81e5dacb1", "results": [{"index": 2, "document": {"text": "香蕉是热带水果"}, "score": 0.34775858364472695}, {"index": 0, "document": {"text": "苹果是一种水果"}, "score": 0.34720502367458494}]}

data/sbert_docs.jsonl

@@ -1,3 +1,32 @@
+{"id": "7f3b452b-3bdc-4c2f-971e-18639b33657c", "text": "牧尘用Docker和FastAPI构建了Web服务，使用Python编程 牧尘用Docker和FastAPI构建了Web服务，使用Python编程", "facts": ["牧尘用Docker和FastAPI构建了Web服务，使用Python编程"], "category": "distilled", "timestamp": "2026-05-25T10:25:52.503911"}
+{"id": "5459a702-0d58-45ae-b4ba-a138638b21af", "text": "牧尘今天用Rust重写了MemoryFabric的核心图算法模块，性能提升了3倍 牧尘今天用Rust重写了MemoryFabric的核心图算法模块，性能提升了3倍", "facts": ["牧尘今天用Rust重写了MemoryFabric的核心图算法模块，性能提升了3倍"], "category": "distilled", "timestamp": "2026-05-25T11:37:12.189213"}
+{"id": "6f286987-c6c0-49ca-ad7f-3ec23d4bf411", "text": "小唯今天完成了织忆系统的语义搜索升级，用m3e-base替换了TF-IDF，搜索质量大幅提升 小唯今天完成了织忆系统的语义搜索升级，用m3e-base替换了TF-IDF，搜索质量大幅提升", "facts": ["小唯今天完成了织忆系统的语义搜索升级，用m3e-base替换了TF-IDF，搜索质量大幅提升"], "category": "distilled", "timestamp": "2026-05-25T11:41:04.544431"}
+{"id": "76b6a2e9-b385-40bc-bdb5-13f3e9c87904", "text": "牧尘配置好了HF镜像，SBert模型下载速度提升10倍 牧尘配置好了HF镜像，SBert模型下载速度提升10倍", "facts": ["牧尘配置好了HF镜像，SBert模型下载速度提升10倍"], "category": "distilled", "timestamp": "2026-05-25T11:47:25.994267"}
+{"id": "97d9a6ff-d3fc-4bae-ad7d-26e3ac25359b", "text": "牧尘完成Phase B，用sentence-transformers升级了织忆的recall质量 牧尘完成Phase B，用sentence-transformers升级了织忆的recall质量", "facts": ["牧尘完成Phase B，用sentence-transformers升级了织忆的recall质量"], "category": "distilled", "timestamp": "2026-05-25T11:52:52.594904"}
+{"id": "0ed25b98-3fc7-4d87-8cb9-3cf9e5d69021", "text": "牧尘完成了Phase B+C，SBert替换TF-IDF，commit自动重建索引 牧尘完成了Phase B+C，SBert替换TF-IDF，commit自动重建索引", "facts": ["牧尘完成了Phase B+C，SBert替换TF-IDF，commit自动重建索引"], "category": "distilled", "timestamp": "2026-05-25T11:59:15.524118"}
+{"id": "22598c41-45ed-4338-b165-8ba9dfde7be0", "text": "牧尘修复了共享embedder的bug，现在commit后recall立即可用 牧尘修复了共享embedder的bug，现在commit后recall立即可用", "facts": ["牧尘修复了共享embedder的bug，现在commit后recall立即可用"], "category": "distilled", "timestamp": "2026-05-25T12:27:47.098411"}
+{"id": "72672ee8-6333-4d33-a2e3-7190e4e23f55", "text": "牧尘在研究 MemoryFabric 项目，发现 agent-memory-skill 使用向量数据库做语义索引，效果比 TF-IDF 好 牧尘在研究 MemoryFabric 项目，发现 agent-memory-skill 使用向量数据库做语义索引，效果比 TF-IDF 好", "facts": ["牧尘在研究 MemoryFabric 项目，发现 agent-memory-skill 使用向量数据库做语义索引，效果比 TF-IDF 好"], "category": "distilled", "timestamp": "2026-05-25T12:29:54.366499"}
+{"id": "803da143-0b3d-4e46-804c-8b8096001389", "text": "小唯提醒牧尘需要备份 hermes 配置文件，避免在新机器上重复配置 小唯提醒牧尘需要备份 hermes 配置文件，避免在新机器上重复配置", "facts": ["小唯提醒牧尘需要备份 hermes 配置文件，避免在新机器上重复配置"], "category": "distilled", "timestamp": "2026-05-25T12:29:54.366803"}
+{"id": "0dbac385-ecdd-4a59-b463-cccd29ff9a44", "text": "织忆系统的 Phase 3 SDK 已经完成，提供了 ZhiYiSync 同步客户端和 ZhiYiClient 异步客户端 织忆系统的 Phase 3 SDK 已经完成，提供了 ZhiYiSync 同步客户端和 ZhiYiClient 异步客户端", "facts": ["织忆系统的 Phase 3 SDK 已经完成，提供了 ZhiYiSync 同步客户端和 ZhiYiClient 异步客户端"], "category": "distilled", "timestamp": "2026-05-25T12:29:54.367026"}
+{"id": "19527634-942e-445a-a158-ac51e8ce9c29", "text": "测试增量索引是否正常工作，commit 后立即搜索 测试增量索引是否正常工作，commit 后立即搜索", "facts": ["测试增量索引是否正常工作，commit 后立即搜索"], "category": "distilled", "timestamp": "2026-05-25T12:59:43.727762"}
+{"id": "a055f73f-21fb-41e4-a8b0-1d534a4e4a0b", "text": "Redis集成测试，多实例同步验证 Redis集成测试，多实例同步验证", "facts": ["Redis集成测试，多实例同步验证"], "category": "distilled", "timestamp": "2026-05-25T15:27:29.604812"}
+{"id": "7a6c1b4d-edac-4895-ae88-770340acdea7", "text": "Nginx负载均衡测试 Nginx负载均衡测试", "facts": ["Nginx负载均衡测试"], "category": "distilled", "timestamp": "2026-05-25T16:22:13.755401"}
+{"id": "fe06fbff-e206-400f-9a4d-a413ff488202", "text": "故障切换后继续服务测试 故障切换后继续服务测试", "facts": ["故障切换后继续服务测试"], "category": "distilled", "timestamp": "2026-05-25T16:22:59.327332"}
+{"id": "a610bc4a-3156-4700-aa39-5f8f7c9e0b64", "text": "测试 GPU 加速 commit 测试 GPU 加速 commit", "facts": ["测试 GPU 加速 commit"], "category": "distilled", "timestamp": "2026-05-25T17:42:57.249486"}
+{"id": "92d9ca04-1e29-4ceb-adb1-9a7614ba7b0a", "text": "全面E2E测试：验证织忆v2.8系统稳定性，测试commit、recall、双实例同步功能 全面E2E测试：验证织忆v2.8系统稳定性，测试commit、recall、双实例同步功能", "facts": ["全面E2E测试：验证织忆v2.8系统稳定性，测试commit、recall、双实例同步功能"], "category": "distilled", "timestamp": "2026-05-25T18:49:31.682642"}
+{"id": "cc3774c0-4ac4-43d1-a89a-ec11de677e01", "text": "织忆v2.8全面测试：JSONL去重后双实例同步验证，commit后两实例应同时更新 织忆v2.8全面测试：JSONL去重后双实例同步验证，commit后两实例应同时更新", "facts": ["织忆v2.8全面测试：JSONL去重后双实例同步验证，commit后两实例应同时更新"], "category": "distilled", "timestamp": "2026-05-25T18:56:48.100646"}
+{"id": "288f27f2-9157-452e-b990-7ccd426eb449", "text": "E2E final test commit sync verification E2E final test commit sync verification", "facts": ["E2E final test commit sync verification"], "category": "distilled", "timestamp": "2026-05-25T19:05:13.340351"}
+{"id": "1a4b56ad-93ff-4e5f-b8a3-51047a17f773", "text": "Clean test: verifying dual instance sync after Redis flush Clean test: verifying dual instance sync after Redis flush", "facts": ["Clean test: verifying dual instance sync after Redis flush"], "category": "distilled", "timestamp": "2026-05-25T19:14:41.488189"}
+{"id": "958b8ea4-2b8a-4f7c-9df6-2d35e1a7be0d", "text": "Fixed consumer: non-blocking read_events + proper polling loop Fixed consumer: non-blocking read_events + proper polling loop", "facts": ["Fixed consumer: non-blocking read_events + proper polling loop"], "category": "distilled", "timestamp": "2026-05-25T19:19:49.127993"}
+{"id": "5c7117bf-3e23-4e11-b380-dd199c839bda", "text": "Debug test: checking non-blocking event consumer Debug test: checking non-blocking event consumer", "facts": ["Debug test: checking non-blocking event consumer"], "category": "distilled", "timestamp": "2026-05-25T19:24:08.924123"}
+{"id": "d4e9d16f-cead-4a1e-b853-525b93d48f2b", "text": "测试XREAD同步修复——这是第一条新记忆 测试XREAD同步修复——这是第一条新记忆", "facts": ["测试XREAD同步修复——这是第一条新记忆"], "category": "distilled", "timestamp": "2026-05-25T19:45:31.323459"}
+{"id": "7e0dc5c1-ace1-49b4-9a0e-a70330171e0e", "text": "测试XREAD同步修复——第二条新记忆 测试XREAD同步修复——第二条新记忆", "facts": ["测试XREAD同步修复——第二条新记忆"], "category": "distilled", "timestamp": "2026-05-25T19:45:57.152292"}
+{"id": "6c30a1e8-c3a2-41a8-933e-a463fe28be8f", "text": "测试bge-m3 encode commit，这是测试bge-m3新文档写入的记忆 测试bge-m3 encode commit，这是测试bge-m3新文档写入的记忆", "facts": ["测试bge-m3 encode commit，这是测试bge-m3新文档写入的记忆"], "category": "distilled", "timestamp": "2026-05-26T01:29:34.635868"}
+{"id": "4b5fdc17-dc79-4a5f-b4b4-8badeccea4f0", "text": "这是v2.9目标C验证测试，bge-m3 encode commit是否正确 这是v2.9目标C验证测试，bge-m3 encode commit是否正确", "facts": ["这是v2.9目标C验证测试，bge-m3 encode commit是否正确"], "category": "distilled", "timestamp": "2026-05-26T01:34:00.554628"}
+{"id": "94486aa2-5aa9-412d-b617-ae0ac5ea9dfb", "text": "v2.9目标C第二次commit测试，验证全量索引重建是否正确 v2.9目标C第二次commit测试，验证全量索引重建是否正确", "facts": ["v2.9目标C第二次commit测试，验证全量索引重建是否正确"], "category": "distilled", "timestamp": "2026-05-26T09:35:41.154366"}
+{"id": "e5cfa4a0-c952-4811-bce3-f161e505ad58", "text": "v2.9目标C第三次commit，修复了字段映射问题 v2.9目标C第三次commit，修复了字段映射问题", "facts": ["v2.9目标C第三次commit，修复了字段映射问题"], "category": "distilled", "timestamp": "2026-05-26T09:39:37.905022"}
+{"id": "43a777e2-a675-4541-b62e-024719d8f02c", "text": "v2.9第四次commit，验证修复后sbert_docs.jsonl是否正确追加 v2.9第四次commit，验证修复后sbert_docs.jsonl是否正确追加", "facts": ["v2.9第四次commit，验证修复后sbert_docs.jsonl是否正确追加"], "category": "distilled", "timestamp": "2026-05-26T09:41:14.628043"}
+{"id": "ac67ff0f-eb4c-42a7-8f38-ea20b4bf73c0", "text": "v2.9第五次commit，JSONLShardStore完整数据源修复 v2.9第五次commit，JSONLShardStore完整数据源修复", "facts": ["v2.9第五次commit，JSONLShardStore完整数据源修复"], "category": "distilled", "timestamp": "2026-05-26T09:43:19.114554"}
 {"id": "6a6ea95c-7426-4caf-afdd-920f22ed6994", "text": "test memory ", "facts": [], "category": "episodes", "timestamp": "2026-05-25T02:24:54.078945"}
 {"id": "e6d172bd-01f6-4eee-8b59-bb3aa83fb34a", "text": "牧尘测试记忆2026 ", "facts": [], "category": "episodes", "timestamp": "2026-05-25T09:14:35.517132"}
 {"id": "ba2726a2-37f6-4d0c-8059-5b1fa6ddaa7a", "text": "牧尘测试记忆2026-05-25 ", "facts": [], "category": "episodes", "timestamp": "2026-05-25T09:14:57.758853"}
@@ -18,31 +47,22 @@
 {"id": "a481db6f-c57e-4522-a085-f27d73fb22ce", "text": "牧尘在研究 MemoryFabric 项目，发现 agent-memory-skill 使用向量数据库做语义索引，效果比 TF-IDF 好 研究 MemoryFabric 项目 向量数据库语义索引", "facts": ["研究 MemoryFabric 项目", "向量数据库语义索引"], "category": "episodes", "timestamp": "2026-05-25T12:29:54.366235"}
 {"id": "eceee2e8-30c1-4c6e-a890-9830cee562d2", "text": "小唯提醒牧尘需要备份 hermes 配置文件，避免在新机器上重复配置 备份配置文件 避免重复配置", "facts": ["备份配置文件", "避免重复配置"], "category": "episodes", "timestamp": "2026-05-25T12:29:54.366671"}
 {"id": "b8acc547-5af6-4021-a4ad-1b6b1d165dd7", "text": "织忆系统的 Phase 3 SDK 已经完成，提供了 ZhiYiSync 同步客户端和 ZhiYiClient 异步客户端 SDK 完成 同步异步双客户端", "facts": ["SDK 完成", "同步异步双客户端"], "category": "episodes", "timestamp": "2026-05-25T12:29:54.366909"}
-{"id": "7f3b452b-3bdc-4c2f-971e-18639b33657c", "text": " 牧尘用Docker和FastAPI构建了Web服务，使用Python编程", "facts": ["牧尘用Docker和FastAPI构建了Web服务，使用Python编程"], "category": "distilled", "timestamp": ""}
-{"id": "5459a702-0d58-45ae-b4ba-a138638b21af", "text": " 牧尘今天用Rust重写了MemoryFabric的核心图算法模块，性能提升了3倍", "facts": ["牧尘今天用Rust重写了MemoryFabric的核心图算法模块，性能提升了3倍"], "category": "distilled", "timestamp": ""}
-{"id": "6f286987-c6c0-49ca-ad7f-3ec23d4bf411", "text": " 小唯今天完成了织忆系统的语义搜索升级，用m3e-base替换了TF-IDF，搜索质量大幅提升", "facts": ["小唯今天完成了织忆系统的语义搜索升级，用m3e-base替换了TF-IDF，搜索质量大幅提升"], "category": "distilled", "timestamp": ""}
-{"id": "76b6a2e9-b385-40bc-bdb5-13f3e9c87904", "text": " 牧尘配置好了HF镜像，SBert模型下载速度提升10倍", "facts": ["牧尘配置好了HF镜像，SBert模型下载速度提升10倍"], "category": "distilled", "timestamp": ""}
-{"id": "19527634-942e-445a-a158-ac51e8ce9c29", "text": "测试增量索引是否正常工作，commit 后立即搜索 测试增量索引是否正常工作，commit 后立即搜索", "facts": ["测试增量索引是否正常工作，commit 后立即搜索"], "category": "distilled", "timestamp": "2026-05-25T12:59:43.727762"}
 {"id": "a4c63346-bb98-4237-960d-cf321fa2fb6d", "text": "测试增量索引是否正常工作，commit 后立即搜索 增量索引测试", "facts": ["增量索引测试"], "category": "episodes", "timestamp": "2026-05-25T12:59:43.726479"}
-{"id": "a055f73f-21fb-41e4-a8b0-1d534a4e4a0b", "text": "Redis集成测试，多实例同步验证 Redis集成测试，多实例同步验证", "facts": ["Redis集成测试，多实例同步验证"], "category": "distilled", "timestamp": "2026-05-25T15:27:29.604812"}
 {"id": "462fc577-5a6f-4feb-838b-965d24612eea", "text": "Redis集成测试，多实例同步验证 Redis集成 多实例同步", "facts": ["Redis集成", "多实例同步"], "category": "episodes", "timestamp": "2026-05-25T15:27:29.602233"}
-{"id": "6e21af7f-0d61-4f05-be0c-e2d544b07487", "text": "Nginx负载均衡测试 负载均衡 双实例", "facts": ["负载均衡", "双实例"], "category": "episode", "timestamp": "2026-05-25T16:22:13.753469"}
-{"id": "7a6c1b4d-edac-4895-ae88-770340acdea7", "text": "Nginx负载均衡测试 Nginx负载均衡测试", "facts": ["Nginx负载均衡测试"], "category": "distilled", "timestamp": "2026-05-25T16:22:13.755401"}
-{"id": "fe06fbff-e206-400f-9a4d-a413ff488202", "text": "故障切换后继续服务测试 故障切换后继续服务测试", "facts": ["故障切换后继续服务测试"], "category": "distilled", "timestamp": "2026-05-25T16:22:59.327332"}
+{"id": "6e21af7f-0d61-4f05-be0c-e2d544b07487", "text": "Nginx负载均衡测试 负载均衡 双实例", "facts": ["负载均衡", "双实例"], "category": "episodes", "timestamp": "2026-05-25T16:22:13.753469"}
 {"id": "1af83f92-e13b-4343-b246-4fac772089be", "text": "故障切换后继续服务测试 高可用 故障切换", "facts": ["高可用", "故障切换"], "category": "episodes", "timestamp": "2026-05-25T16:22:59.326841"}
-{"id": "a610bc4a-3156-4700-aa39-5f8f7c9e0b64", "text": "测试 GPU 加速 commit 测试 GPU 加速 commit", "facts": ["测试 GPU 加速 commit"], "category": "distilled", "timestamp": "2026-05-25T17:42:57.249486"}
 {"id": "f2476cda-7e4c-4d1a-91ed-43b17846b2f1", "text": "测试 GPU 加速 commit ", "facts": [], "category": "episodes", "timestamp": "2026-05-25T17:42:57.246630"}
-{"id": "test-gpu-1", "text": "测试 GPU 编码性能 GPU 测试", "facts": ["GPU", "测试"], "category": "test", "timestamp": "2026-05-26T00:00:00"}
-{"id": "baf9092c-b399-474a-8ad9-70d515e6d7a5", "text": "全面E2E测试：验证织忆v2.8系统稳定性，测试commit、recall、双实例同步功能 ", "facts": [], "category": "episode", "timestamp": "2026-05-25T18:49:31.680362"}
-{"id": "cc3774c0-4ac4-43d1-a89a-ec11de677e01", "text": "织忆v2.8全面测试：JSONL去重后双实例同步验证，commit后两实例应同时更新 织忆v2.8全面测试：JSONL去重后双实例同步验证，commit后两实例应同时更新", "facts": ["织忆v2.8全面测试：JSONL去重后双实例同步验证，commit后两实例应同时更新"], "category": "distilled", "timestamp": "2026-05-25T18:56:48.100646"}
+{"id": "baf9092c-b399-474a-8ad9-70d515e6d7a5", "text": "全面E2E测试：验证织忆v2.8系统稳定性，测试commit、recall、双实例同步功能 ", "facts": [], "category": "episodes", "timestamp": "2026-05-25T18:49:31.680362"}
 {"id": "af5e486c-378d-43e7-b9da-a6d441f7f3ce", "text": "织忆v2.8全面测试：JSONL去重后双实例同步验证，commit后两实例应同时更新 ", "facts": [], "category": "episodes", "timestamp": "2026-05-25T18:56:48.098875"}
-{"id": "b34db22b-e18b-4fb7-a6ed-1ab9a5c59e5b", "text": "E2E final test commit sync verification ", "facts": [], "category": "episode", "timestamp": "2026-05-25T19:05:13.338091"}
-{"id": "d3992495-cf9b-4741-99de-5e8748e2d8b7", "text": "Clean test: verifying dual instance sync after Redis flush ", "facts": [], "category": "episode", "timestamp": "2026-05-25T19:14:41.486735"}
-{"id": "958b8ea4-2b8a-4f7c-9df6-2d35e1a7be0d", "text": "Fixed consumer: non-blocking read_events + proper polling loop Fixed consumer: non-blocking read_events + proper polling loop", "facts": ["Fixed consumer: non-blocking read_events + proper polling loop"], "category": "distilled", "timestamp": "2026-05-25T19:19:49.127993"}
+{"id": "b34db22b-e18b-4fb7-a6ed-1ab9a5c59e5b", "text": "E2E final test commit sync verification ", "facts": [], "category": "episodes", "timestamp": "2026-05-25T19:05:13.338091"}
+{"id": "d3992495-cf9b-4741-99de-5e8748e2d8b7", "text": "Clean test: verifying dual instance sync after Redis flush ", "facts": [], "category": "episodes", "timestamp": "2026-05-25T19:14:41.486735"}
 {"id": "8e2f51ce-72e7-4fec-b578-385ccbe14b8c", "text": "Fixed consumer: non-blocking read_events + proper polling loop ", "facts": [], "category": "episodes", "timestamp": "2026-05-25T19:19:49.126539"}
-{"id": "5c7117bf-3e23-4e11-b380-dd199c839bda", "text": "Debug test: checking non-blocking event consumer Debug test: checking non-blocking event consumer", "facts": ["Debug test: checking non-blocking event consumer"], "category": "distilled", "timestamp": "2026-05-25T19:24:08.924123"}
 {"id": "e6f80048-52c5-477e-ae87-32800f8d9904", "text": "Debug test: checking non-blocking event consumer ", "facts": [], "category": "episodes", "timestamp": "2026-05-25T19:24:08.922675"}
-{"id": "d4e9d16f-cead-4a1e-b853-525b93d48f2b", "text": "测试XREAD同步修复——这是第一条新记忆 测试XREAD同步修复——这是第一条新记忆", "facts": ["测试XREAD同步修复——这是第一条新记忆"], "category": "distilled", "timestamp": "2026-05-25T19:45:31.323459"}
 {"id": "345dd7bd-0efa-43cf-9c49-2a774a4dd96e", "text": "测试XREAD同步修复——这是第一条新记忆 ", "facts": [], "category": "episodes", "timestamp": "2026-05-25T19:45:31.320957"}
-{"id": "7e0dc5c1-ace1-49b4-9a0e-a70330171e0e", "text": "测试XREAD同步修复——第二条新记忆 测试XREAD同步修复——第二条新记忆", "facts": ["测试XREAD同步修复——第二条新记忆"], "category": "distilled", "timestamp": "2026-05-25T19:45:57.152292"}
 {"id": "03da779d-7cae-4297-b17b-3fd3b44987b6", "text": "测试XREAD同步修复——第二条新记忆 ", "facts": [], "category": "episodes", "timestamp": "2026-05-25T19:45:57.149124"}
+{"id": "4be5c9ba-3ce1-4f2b-ba11-514df394ddeb", "text": "测试bge-m3 encode commit，这是测试bge-m3新文档写入的记忆 ", "facts": [], "category": "episodes", "timestamp": "2026-05-26T01:29:34.633402"}
+{"id": "ede71ab7-bece-479a-91d5-bae09e9e1122", "text": "这是v2.9目标C验证测试，bge-m3 encode commit是否正确 ", "facts": [], "category": "episodes", "timestamp": "2026-05-26T01:34:00.553015"}
+{"id": "b61133f9-7154-4110-8bba-2d4109b1445a", "text": "v2.9目标C第二次commit测试，验证全量索引重建是否正确 ", "facts": [], "category": "episodes", "timestamp": "2026-05-26T09:35:41.151546"}
+{"id": "45fb6769-5f90-48e0-8e6e-cef5b5a195c4", "text": "v2.9目标C第三次commit，修复了字段映射问题 ", "facts": [], "category": "episodes", "timestamp": "2026-05-26T09:39:37.903611"}
+{"id": "eb3666c7-0398-497b-9572-25be2dc7d35e", "text": "v2.9第四次commit，验证修复后sbert_docs.jsonl是否正确追加 ", "facts": [], "category": "episodes", "timestamp": "2026-05-26T09:41:14.627423"}
+{"id": "e7b5fb8e-3c5d-4c8c-90c7-f53ea6dcea3b", "text": "v2.9第五次commit，JSONLShardStore完整数据源修复 ", "facts": [], "category": "episodes", "timestamp": "2026-05-26T09:43:19.112351"}

data/sbert_meta.json

@@ -1 +1 @@
-{"model_name": "moka-ai/m3e-base", "doc_count": 48, "vector_dim": 768, "index_type": "FlatIP", "updated_at": "2026-05-25T19:45:57.182778"}
+{"model_name": "bge-m3", "doc_count": 68, "vector_dim": 1024, "index_type": "FlatIP", "updated_at": "2026-05-26T09:43:31.069848"}

requirements.txt

@@ -1,27 +1,21 @@
-# 织忆 v2.8 依赖清单
-# CPU 部署：torch + faiss-cpu
-# GPU 部署：额外安装 nvidia CUDA 库
+# Core
+fastapi>=0.110.0
+uvicorn[standard]>=0.27.0
+pydantic>=2.0.0
 
-# 核心服务
-fastapi>=0.100.0
-uvicorn>=0.20.0
-pydantic>=2.0
-redis>=5.0
+# 存储
+redis>=5.0.0
 
-# 向量存储 + 嵌入
-faiss-cpu>=1.7.0
-sentence-transformers>=2.2.0
-
-# 模型相关
+# 向量 & ML
+numpy>=1.26.0
+faiss-cpu>=1.7.4
+sentence-transformers>=2.5.0
 torch>=2.0.0
-transformers>=4.30.0
-huggingface-hub>=0.16.0
+transformers>=4.40.0
 
-# 数据处理
-numpy>=1.24.0
-scikit-learn>=1.3.0
+# HTTP
+requests>=2.31.0
+aiohttp>=3.9.0
 
 # 工具
-typer>=0.9.0
-click>=8.0
-pyyaml>=6.0
+python-dotenv>=1.0.0

scripts/rebuild_faiss_1024.py

@@ -0,0 +1,124 @@
+#!/usr/bin/env python3
+"""重建 FAISS 索引为 1024 维（bge-m3）
+
+用法:
+    python scripts/rebuild_faiss_1024.py
+"""
+import sys
+import json
+import faiss
+import numpy as np
+from pathlib import Path
+from datetime import datetime
+
+# 添加项目路径
+sys.path.insert(0, str(Path(__file__).parent.parent / 'src'))
+
+from storage.bge_embedder import BGEClient, VECTOR_DIM
+
+CACHE_DIR = Path.home() / "projects" / "zhiyi" / "data"
+DOCS_FILE = CACHE_DIR / "sbert_docs.jsonl"
+FAISS_INDEX_FILE = CACHE_DIR / "sbert_index.faiss"
+META_FILE = CACHE_DIR / "sbert_meta.json"
+
+VECTOR_DIM = 1024  # bge-m3 输出 1024 维
+
+
+def load_docs() -> list[dict]:
+    """从 sbert_docs.jsonl 加载所有文档"""
+    docs = []
+    if not DOCS_FILE.exists():
+        print(f"[ERROR] 文档文件不存在: {DOCS_FILE}")
+        return docs
+    with open(DOCS_FILE, "r", encoding="utf-8") as f:
+        for line in f:
+            line = line.strip()
+            if not line:
+                continue
+            try:
+                docs.append(json.loads(line))
+            except json.JSONDecodeError:
+                continue
+    return docs
+
+
+def normalize(vectors: np.ndarray) -> np.ndarray:
+    """L2 归一化（使内积等效余弦相似度）"""
+    norms = np.linalg.norm(vectors, axis=1, keepdims=True)
+    norms = np.where(norms == 0, 1, norms)
+    return vectors / norms
+
+
+def main():
+    print(f"=" * 60)
+    print(f"FAISS 索引重建（1024维 bge-m3）")
+    print(f"=" * 60)
+
+    # 加载文档
+    docs = load_docs()
+    print(f"加载文档: {len(docs)} 条")
+
+    if not docs:
+        print("[ERROR] 没有文档可索引")
+        return
+
+    # 初始化 bge-m3 客户端
+    print(f"初始化 bge-m3 客户端...")
+    bge = BGEClient()
+
+    # 准备文本列表
+    texts = []
+    for doc in docs:
+        text = doc.get('text', '') + ' ' + ' '.join(str(f) for f in doc.get('facts', []))
+        texts.append(text)
+
+    # 批量 encode（使用 bge-m3 API，每次 1 条，避免批量限制）
+    print(f"开始 bge-m3 encode（{len(texts)} 条文档，需要 {len(texts)} 次 API 调用）...")
+    vectors = []
+    for i, text in enumerate(texts):
+        if i % 10 == 0:
+            print(f"  encode 进度: {i}/{len(texts)}")
+        v = bge.encode(text, normalize_output=True)
+        vectors.append(v)
+    print(f"encode 完成: {len(vectors)} 条向量，每条 {len(vectors[0])} 维")
+
+    # 构建 FAISS 索引
+    vectors_np = np.array(vectors, dtype=np.float32)
+    print(f"构建 IndexFlatIP 索引（{vectors_np.shape}）...")
+    index = faiss.IndexFlatIP(VECTOR_DIM)
+    index.add(vectors_np)
+    print(f"FAISS 索引构建完成: {index.ntotal} 条向量")
+
+    # 保存索引
+    print(f"保存 FAISS 索引到 {FAISS_INDEX_FILE}...")
+    faiss.write_index(index, str(FAISS_INDEX_FILE))
+
+    # 更新 meta
+    meta = {
+        'model_name': 'bge-m3',
+        'doc_count': len(docs),
+        'vector_dim': VECTOR_DIM,
+        'index_type': 'FlatIP',
+        'updated_at': datetime.now().isoformat(),
+    }
+    with open(META_FILE, 'w', encoding='utf-8') as f:
+        json.dump(meta, f, ensure_ascii=False)
+    print(f"更新 meta: {meta}")
+
+    # 备份旧索引（如果有）
+    backup_file = FAISS_INDEX_FILE.with_suffix('.faiss.bak768')
+    if FAISS_INDEX_FILE.exists():
+        import shutil
+        shutil.copy2(FAISS_INDEX_FILE, backup_file)
+        print(f"旧索引备份到: {backup_file}")
+
+    print(f"=" * 60)
+    print(f"✅ 重建完成！")
+    print(f"   文档数: {len(docs)}")
+    print(f"   向量维度: {VECTOR_DIM}")
+    print(f"   模型: bge-m3（模力方舟 API）")
+    print(f"=" * 60)
+
+
+if __name__ == '__main__':
+    main()

src/api/routes/commit.py

@@ -84,29 +84,46 @@ def commit_memory(req: CommitRequest):
             except Exception as e:
                 print(f"[Redis] distilled 存储/发布失败: {e}")
 
-        # 增量索引：新文档直接 add_documents，不全量 rebuild
-        # episode 和 distilled 都添加（distilled 在前，包含摘要信息）
+        # 全量索引重建（每次 commit 都重建，保证 FAISS 索引和数据完全一致）
+        # 从 JSONLShardStore 读取完整 episodes + distilled（最新状态）
+        from storage.bge_embedder import get_bge_client
+        from storage.jsonl_store import JSONLShardStore
+
+        bge = get_bge_client()
+        store = JSONLShardStore()
+
+        # 读取所有已有文档（从 shards，完整最新数据）
+        all_docs = []
+        for category in ['distilled', 'episodes']:
+            for year_month in [(2026, 5), (2026, 4), (2026, 3)]:
+                for doc in store.read_month(category, year_month[0], year_month[1]):
+                    # JSONLShardStore 字段映射：summary→content, content→content
+                    content = doc.get('summary', '') or doc.get('content', '')
+                    if not content:
+                        continue
+                    all_docs.append({
+                        'id': doc.get('id', ''),
+                        'content': content,
+                        'facts': doc.get('facts', []),
+                        'category': category,
+                        'timestamp': doc.get('created_at', '') or doc.get('timestamp', ''),
+                    })
+
+        if not all_docs:
+            print("[Commit] 没有文档可供索引")
+            return
+
+        # 用 bge-m3 API 全量编码（确保所有向量 1024 维）
+        all_vectors = []
+        for doc in all_docs:
+            text = doc.get('content', '') + ' ' + ' '.join(str(f) for f in doc.get('facts', []))
+            v = bge.encode(text, normalize_output=True)
+            all_vectors.append(v)
+
+        # 全量重建 FAISS 索引（写入 sbert_docs.jsonl + FAISS 索引文件）
         from api.server import get_shared_embedder
         embedder = get_shared_embedder()
-
-        # 按顺序添加（distilled 摘要优先，episode 原文次之）
-        new_docs = [
-            {
-                'id': distilled.id,
-                'content': distilled.summary,
-                'facts': distilled.facts,
-                'category': 'distilled',
-                'timestamp': distilled.created_at.isoformat() if hasattr(distilled, 'created_at') else '',
-            },
-            {
-                'id': episode.id,
-                'content': episode.content,
-                'facts': episode.facts,
-                'category': 'episodes',
-                'timestamp': episode.timestamp.isoformat() if hasattr(episode, 'timestamp') else '',
-            }
-        ]
-        embedder.add_documents(new_docs)
+        embedder.index_documents_with_vectors(all_docs, all_vectors)
 
     except Exception:
         pass

src/api/routes/recall.py

@@ -6,11 +6,29 @@ from pathlib import Path
 sys.path.insert(0, str(Path(__file__).parent.parent.parent.parent / 'src'))
 
 from storage.jsonl_store import JSONLShardStore
+from storage.bge_embedder import get_bge_client
+from storage.jina_rerank import get_rerank_client
 from datetime import datetime
 
 router = APIRouter()
 _store = JSONLShardStore()
 
+# BGE 客户端和 Rerank 客户端（单例）
+_bge_client = None
+_rerank_client = None
+
+def _get_bge():
+    global _bge_client
+    if _bge_client is None:
+        _bge_client = get_bge_client()
+    return _bge_client
+
+def _get_rerank():
+    global _rerank_client
+    if _rerank_client is None:
+        _rerank_client = get_rerank_client()
+    return _rerank_client
+
 def _get_embedder():
     """获取共享 embedder（从 server.py 单例）"""
     from api.server import get_shared_embedder
@@ -22,23 +40,43 @@ class RecallRequest(BaseModel):
     namespace: Optional[str] = 'zhiyi'
     start_time: Optional[str] = None
     end_time: Optional[str] = None
+    use_rerank: bool = True  # 是否使用 jina rerank 重排
+    coarse_top_k: int = 50   # 粗排返回数量（送入 rerank 前）
 
 @router.post('/recall')
 def recall_memory(req: RecallRequest):
-    """召回相关记忆（SBert 语义搜索 + 关键词兜底）
+    """召回相关记忆（bge-m3 语义搜索 + jina rerank 重排）
 
-    1. 语义搜索：Sentence-Transformers 向量相似度
-    2. 关键词兜底：搜索 episodes + distilled 多类目、多月
+    流程：
+    1. bge-m3 encode（模力方舟 API）→ query_vector
+    2. FAISS ANN 搜索（top_k=coarse_top_k）→ 粗排候选
+    3. jina rerank 重排 → top-k 精确排序
+    4. 关键词兜底（如 rerank 失败）
     """
-    results = []
     embedder = _get_embedder()
+    bge = _get_bge()
+    rerank_client = _get_rerank()
 
-    # Step 1: 语义搜索（SBert）
+    # Step 1: bge-m3 encode
+    query_vector = None
     semantic_results = []
-    if not embedder.needs_rebuild():
+    try:
+        query_vector = bge.encode(req.query, normalize_output=True)
+    except Exception as e:
+        print(f"[Recall] bge-m3 encode 失败: {e}")
+
+    # Step 2: FAISS 粗排（如果 embedder 支持 search_by_vector）
+    if query_vector is not None:
         try:
-            search_results = embedder.search(req.query, top_k=req.limit)
-            for doc_id, score, doc_meta in search_results:
+            coarse_k = max(req.coarse_top_k, req.limit * 2)
+            if hasattr(embedder, 'search_by_vector'):
+                # 使用预计算向量搜索
+                coarse_results = embedder.search_by_vector(query_vector, top_k=coarse_k)
+            else:
+                # fallback 到 text search
+                coarse_results = embedder.search(req.query, top_k=coarse_k)
+
+            for doc_id, score, doc_meta in coarse_results:
                 semantic_results.append({
                     'id': doc_id,
                     'score': score,
@@ -46,34 +84,74 @@ def recall_memory(req: RecallRequest):
                     'category': doc_meta.get('category', 'semantic'),
                     'facts': doc_meta.get('facts', []),
                     'timestamp': doc_meta.get('timestamp', ''),
+                    'doc_meta': doc_meta,
                 })
-        except Exception:
-            pass  # 语义搜索失败，降级关键词
+        except Exception as e:
+            print(f"[Recall] FAISS 搜索失败: {e}")
 
-    # Step 2: 关键词兜底（多类目 + 多月）
+    # Step 3: 重排（优先 jina rerank，fallback 用 embedding 相似度）
+    if semantic_results and req.use_rerank and query_vector is not None:
+        try:
+            doc_texts = [r['doc_meta'].get('text', '') for r in semantic_results]
+            rerank_resp = rerank_client.rerank(req.query, doc_texts, top_n=req.limit)
+
+            # 按 rerank 顺序重新排列
+            reranked = []
+            for rr in rerank_resp.results:
+                if rr.index < len(semantic_results):
+                    item = semantic_results[rr.index]
+                    item['score'] = rr.score  # 用 rerank score 替换 faiss score
+                    reranked.append(item)
+
+            reranked.sort(key=lambda x: x['score'], reverse=True)
+            semantic_results = reranked
+        except Exception as e:
+            # fallback: 用 bge-m3 encode 候选文档，计算余弦相似度排序
+            print(f"[Recall] jina rerank 失败，fallback 到 embedding 相似度排序: {e}")
+            try:
+                cand_vectors = bge.encode_batch(doc_texts, normalize_output=True)
+                import numpy as np
+                qv = np.array(query_vector)
+                scores = []
+                for cv in cand_vectors:
+                    cv_arr = np.array(cv)
+                    score = float(np.dot(qv, cv_arr))
+                    scores.append(score)
+                # 按 scores 排序
+                indexed = list(zip(semantic_results, scores))
+                indexed.sort(key=lambda x: x[1], reverse=True)
+                semantic_results = [r for r, s in indexed[:req.limit]]
+                # 更新 score
+                for i, (r, s) in enumerate(indexed[:req.limit]):
+                    semantic_results[i]['score'] = s
+            except Exception as e2:
+                print(f"[Recall] embedding fallback 也失败: {e2}")
+
+    # Step 4: 关键词兜底（如 semantic 为空）
     seen_ids = set(r['id'] for r in semantic_results)
     keyword_results = []
 
-    for category in ['episodes', 'distilled']:
-        for record in _store.search_recent(
-            category=category,
-            query=req.query,
-            months_back=2,
-            limit=req.limit * 2,
-        ):
-            if record.get('id') not in seen_ids:
-                seen_ids.add(record.get('id'))
-                keyword_results.append({
-                    'id': record.get('id'),
-                    'category': category,
-                    'content': record.get('content', record.get('summary', '')),
-                    'facts': record.get('facts', []),
-                    'timestamp': record.get('timestamp', ''),
-                })
+    if not semantic_results:
+        for category in ['episodes', 'distilled']:
+            for record in _store.search_recent(
+                category=category,
+                query=req.query,
+                months_back=2,
+                limit=req.limit * 2,
+            ):
+                if record.get('id') not in seen_ids:
+                    seen_ids.add(record.get('id'))
+                    keyword_results.append({
+                        'id': record.get('id'),
+                        'category': category,
+                        'content': record.get('content', record.get('summary', '')),
+                        'facts': record.get('facts', []),
+                        'timestamp': record.get('timestamp', ''),
+                    })
+                if len(keyword_results) >= req.limit:
+                    break
             if len(keyword_results) >= req.limit:
                 break
-        if len(keyword_results) >= req.limit:
-            break
 
     # 合并结果：语义优先，关键词补充
     all_results = semantic_results + keyword_results
@@ -82,7 +160,16 @@ def recall_memory(req: RecallRequest):
     # 格式化输出（SBert 结果已含元数据，关键词结果需补充）
     formatted = []
     for item in all_results:
-        if 'category' in item and 'timestamp' in item:
+        if 'category' in item and 'doc_meta' in item:
+            formatted.append({
+                'id': item['id'],
+                'score': item.get('score'),
+                'category': item.get('category', 'semantic'),
+                'content': item.get('content', ''),
+                'facts': item.get('facts', []),
+                'timestamp': item.get('timestamp', ''),
+            })
+        elif 'category' in item and 'timestamp' in item:
             formatted.append({
                 'id': item['id'],
                 'score': item.get('score'),

src/api/server.py

@@ -9,6 +9,7 @@ _os.environ.setdefault('HF_ENDPOINT', 'https://hf-mirror.com')
 
 # 全局共享 embedder（所有路由共用一个实例）
 # E2: 切换为 FAISS 向量索引（替代内存向量存储）
+# v2.9: bge-m3 API encode，支持 index_documents_with_vectors 全量重建
 from src.storage.faiss_embedder import FAISSRecall
 _shared_embedder = FAISSRecall()
 

src/storage/bge_embedder.py

@@ -0,0 +1,251 @@
+"""BGE-M3 Embedding — 模力方舟 API（1024维）
+
+API: https://ai.gitee.com/v1/embeddings
+模型: bge-m3（中文优化，多语言）
+维度: 1024
+
+依赖:
+    pip install requests aiohttp
+"""
+import os
+import json
+import time
+import hashlib
+import requests
+from typing import Optional
+from pathlib import Path
+
+# 模力方舟 API 配置
+BGE_API_URL = "https://ai.gitee.com/v1/embeddings"
+BGE_API_KEY = "3TSVVXRFFECE4TISXHGE1VXDAXBIPAP6O1VPJK18"
+BGE_MODEL = "bge-m3"
+VECTOR_DIM = 1024  # bge-m3 输出 1024 维
+
+# 缓存目录（避免重复 encode 相同文本）
+CACHE_DIR = Path.home() / "projects" / "zhiyi" / "data"
+CACHE_DIR.mkdir(parents=True, exist_ok=True)
+VECTOR_CACHE_FILE = CACHE_DIR / "bge_vector_cache.jsonl"
+
+
+def _text_hash(text: str) -> str:
+    """文本的 MD5 哈希（用于缓存 key）"""
+    return hashlib.md5(text.encode()).hexdigest()
+
+
+def _load_cache() -> dict:
+    """加载向量缓存"""
+    if not VECTOR_CACHE_FILE.exists():
+        return {}
+    cache = {}
+    with open(VECTOR_CACHE_FILE, "r", encoding="utf-8") as f:
+        for line in f:
+            line = line.strip()
+            if not line:
+                continue
+            try:
+                item = json.loads(line)
+                cache[item["hash"]] = item
+            except json.JSONDecodeError:
+                continue
+    return cache
+
+
+def _save_to_cache(hash_key: str, text: str, vector: list):
+    """追加写入缓存（append-only）"""
+    with open(VECTOR_CACHE_FILE, "a", encoding="utf-8") as f:
+        f.write(json.dumps({"hash": hash_key, "text": text, "vector": vector, "model": BGE_MODEL}, ensure_ascii=False) + "\n")
+
+
+def normalize(v: list) -> list:
+    """L2 归一化（等效余弦相似度）"""
+    import numpy as np
+    v_np = np.array(v, dtype=np.float32)
+    norm = np.linalg.norm(v_np)
+    if norm > 0:
+        v_np = v_np / norm
+    return v_np.tolist()
+
+
+class BGEClient:
+    """BGE-M3 Embedding 客户端（模力方舟 API）
+
+    用法:
+        client = BGEClient()
+        vector = client.encode("你好世界")
+        vectors = client.encode_batch(["文本1", "文本2"])
+    """
+
+    def __init__(self, api_url: str = BGE_API_URL, api_key: str = BGE_API_KEY, model: str = BGE_MODEL):
+        self.api_url = api_url
+        self.api_key = api_key
+        self.model = model
+        self._cache = _load_cache()
+        self._cache_hits = 0
+        self._cache_misses = 0
+
+    def encode(self, text: str, use_cache: bool = True, normalize_output: bool = True) -> list:
+        """编码单条文本
+
+        Args:
+            text: 要编码的文本
+            use_cache: 是否使用本地缓存
+            normalize_output: 是否 L2 归一化（推荐开启，配合 FAISS IndexFlatIP 使用）
+
+        Returns:
+            1024 维向量（归一化后）
+        """
+        if not text or not text.strip():
+            return [0.0] * VECTOR_DIM
+
+        # 查缓存
+        if use_cache:
+            h = _text_hash(text)
+            if h in self._cache:
+                self._cache_hits += 1
+                vec = self._cache[h]["vector"]
+                return normalize(vec) if normalize_output else vec
+            self._cache_misses += 1
+
+        # 调用 API
+        headers = {
+            "Authorization": f"Bearer {self.api_key}",
+            "Content-Type": "application/json",
+        }
+        payload = {
+            "model": self.model,
+            "input": [text],
+        }
+
+        resp = requests.post(self.api_url, headers=headers, json=payload, timeout=30)
+        resp.raise_for_status()
+        result = resp.json()
+
+        embedding = result["data"][0]["embedding"]
+
+        # 归一化
+        if normalize_output:
+            embedding = normalize(embedding)
+
+        # 写入缓存
+        if use_cache:
+            _save_to_cache(h, text, embedding)
+            self._cache[h] = {"hash": h, "text": text, "vector": embedding, "model": self.model}
+
+        return embedding
+
+    def encode_batch(self, texts: list[str], use_cache: bool = True, normalize_output: bool = True) -> list[list]:
+        """批量编码多条文本（减少 API 调用次数）
+
+        Args:
+            texts: 文本列表
+            use_cache: 是否使用本地缓存
+            normalize_output: 是否 L2 归一化
+
+        Returns:
+            向量列表
+        """
+        results = []
+        uncached_texts = []
+        uncached_indices = []
+
+        # 先查缓存
+        if use_cache:
+            for i, text in enumerate(texts):
+                h = _text_hash(text)
+                if h in self._cache:
+                    self._cache_hits += 1
+                    vec = self._cache[h]["vector"]
+                    results.append(normalize(vec) if normalize_output else vec)
+                else:
+                    self._cache_misses += 1
+                    uncached_texts.append(text)
+                    uncached_indices.append(i)
+                    results.append(None)  # 占位
+        else:
+            uncached_texts = texts
+            uncached_indices = list(range(len(texts)))
+            results = [None] * len(texts)
+
+        # 批量调用 API（每次最多 32 条，避免超限）
+        if uncached_texts:
+            batch_size = 32
+            for i in range(0, len(uncached_texts), batch_size):
+                batch = uncached_texts[i:i + batch_size]
+                headers = {
+                    "Authorization": f"Bearer {self.api_key}",
+                    "Content-Type": "application/json",
+                }
+                payload = {
+                    "model": self.model,
+                    "input": batch,
+                }
+
+                resp = requests.post(self.api_url, headers=headers, json=payload, timeout=60)
+                resp.raise_for_status()
+                result = resp.json()
+
+                embeddings = result["data"]
+                for j, emb_data in enumerate(embeddings):
+                    emb = emb_data["embedding"]
+                    if normalize_output:
+                        emb = normalize(emb)
+                    orig_idx = uncached_indices[i + j]
+                    results[orig_idx] = emb
+
+                    # 写入缓存
+                    if use_cache:
+                        h = _text_hash(uncached_texts[i + j])
+                        _save_to_cache(h, uncached_texts[i + j], emb)
+                        self._cache[h] = {"hash": h, "text": uncached_texts[i + j], "vector": emb, "model": self.model}
+
+        return results
+
+    def stats(self) -> dict:
+        """返回缓存统计"""
+        total = self._cache_hits + self._cache_misses
+        hit_rate = self._cache_hits / total if total > 0 else 0
+        return {
+            "model": self.model,
+            "vector_dim": VECTOR_DIM,
+            "cache_hits": self._cache_hits,
+            "cache_misses": self._cache_misses,
+            "hit_rate": f"{hit_rate:.1%}",
+        }
+
+    def clear_cache(self):
+        """清空向量缓存"""
+        self._cache.clear()
+        if VECTOR_CACHE_FILE.exists():
+            VECTOR_CACHE_FILE.unlink()
+
+
+# 全局单例（避免重复创建 client）
+_bge_client: Optional[BGEClient] = None
+
+
+def get_bge_client() -> BGEClient:
+    """获取 BGE 客户端单例"""
+    global _bge_client
+    if _bge_client is None:
+        _bge_client = BGEClient()
+    return _bge_client
+
+
+if __name__ == "__main__":
+    # 测试
+    client = BGEClient()
+    print("Testing BGE-M3 encode...")
+
+    # 单条测试
+    vec = client.encode("你好世界")
+    print(f"单条向量维度: {len(vec)}, 前5维: {vec[:5]}")
+
+    # 批量测试
+    vecs = client.encode_batch(["苹果", "香蕉", "手机"])
+    print(f"批量: {len(vecs)} 条, 每条维度: {len(vecs[0])}")
+
+    # 缓存测试
+    vec2 = client.encode("你好世界")  # 第二次应命中缓存
+    print(f"缓存命中: {vec == vec2}")
+
+    print("\n缓存统计:", client.stats())

src/storage/faiss_embedder.py

@@ -106,17 +106,65 @@ class FAISSRecall:
             query: 查询文本
             top_k: 返回结果数量
 
+        Returns:
+            [(doc_id: str, score: float, doc: dict), ...] 按 score 降序
+        """
+        # 内部编码后搜索
+        query_vector = self._encode([query])[0]
+        return self.search_by_vector(query_vector, top_k)
+
+    def add_documents_with_vectors(self, docs: list[dict], vectors: list[list[float]]):
+        """增量添加文档（使用预计算向量，用于 bge-m3 API 场景）
+
+        Args:
+            docs: 新增文档列表（包含 content, facts, id 等字段）
+            vectors: 预计算的向量列表（1024维，已归一化）
+        """
+        if not docs:
+            return
+        self.index.add_with_vectors(docs, vectors)
+        # 增量持久化：追加文档 + 全量 FAISS 索引保存
+        self._append_to_disk(docs)
+        self._needs_rebuild = False
+
+    def index_documents_with_vectors(self, docs: list[dict], vectors: list[list[float]]):
+        """全量重建索引（使用预计算向量，用于 bge-m3 API 场景）
+
+        从头创建 FAISS 索引，保证索引和数据完全一致。
+
+        Args:
+            docs: 文档列表（包含 content, facts, id 等字段）
+            vectors: 预计算的向量列表（1024维，已归一化）
+        """
+        if not docs or not vectors:
+            return
+
+        # 用预计算向量全量构建（清空现有索引）
+        self.index.index_with_vectors(docs, vectors)
+        self.index.save()
+        self._needs_rebuild = False
+
+    def search_by_vector(self, query_vector: list[float], top_k: int = 10) -> list[tuple[str, float, dict]]:
+        """用预计算向量搜索（供外部 bge-m3 等调用）
+
+        Args:
+            query_vector: 预计算的查询向量（1024维或 768维）
+            top_k: 返回结果数量
+
         Returns:
             [(doc_id: str, score: float, doc: dict), ...] 按 score 降序
         """
         if not self.index.documents:
             return []
 
-        # 编码查询
-        query_vector = self._encode([query])[0]
+        import numpy as np
+        qv = np.array(query_vector, dtype=np.float32)
+        # 归一化（如果向量未归一化）
+        norm = np.linalg.norm(qv)
+        if norm > 0:
+            qv = qv / norm
 
-        # FAISS 搜索
-        results = self.index.search(query_vector, top_k)
+        results = self.index.search(qv, top_k)
 
         output = []
         for doc_idx, score in results:

src/storage/faiss_index.py

@@ -25,8 +25,8 @@ DOCS_FILE = CACHE_DIR / "sbert_docs.jsonl"
 VECTORS_FILE = CACHE_DIR / "sbert_vectors.jsonl"
 META_FILE = CACHE_DIR / "sbert_meta.json"
 
-# moka-ai/m3e-base 向量维度
-VECTOR_DIM = 768
+# bge-m3 向量维度（模力方舟 API，1024维）
+VECTOR_DIM = 1024
 
 
 class FAISSIndex:
@@ -93,6 +93,43 @@ class FAISSIndex:
         self.index = faiss.IndexFlatIP(self._dim)
         self.index.add(vectors)
 
+    def index_with_vectors(self, docs: list[dict], vectors: list[list[float]]):
+        """从文档列表和预计算向量构建索引（全量重建）
+
+        与 build() 区别：直接使用预计算向量，不调用 encode_fn。
+        用于 bge-m3 API 等外部向量来源场景。
+
+        Args:
+            docs: 文档列表（包含 content, facts, id 等字段）
+            vectors: 预计算向量列表（已归一化）
+        """
+        self.documents = []
+        self.index = None
+
+        if not docs or not vectors:
+            return
+
+        dim = len(vectors[0])
+        self._dim = dim
+
+        # 构建文档记录
+        for doc in docs:
+            text = doc.get('content', '') + ' ' + ' '.join(str(f) for f in doc.get('facts', []))
+            self.documents.append({
+                'id': doc.get('id', ''),
+                'text': text,
+                'facts': doc.get('facts', []),
+                'category': doc.get('category', 'unknown'),
+                'timestamp': doc.get('timestamp', ''),
+            })
+
+        # 向量归一化 + 构建 FAISS 索引
+        import numpy as np
+        vectors_np = np.array(vectors, dtype=np.float32)
+        vectors_np = self._normalize(vectors_np)
+        self.index = faiss.IndexFlatIP(dim)
+        self.index.add(vectors_np)
+
     def add(self, docs: list[dict], encode_fn: Callable[[list[str]], np.ndarray]):
         """增量添加文档
 
@@ -125,6 +162,42 @@ class FAISSIndex:
             vectors = self._normalize(vectors)
             self.index.add(vectors)
 
+    def add_with_vectors(self, docs: list[dict], vectors: list[list[float]]):
+        """添加文档（使用预计算向量，不调用 encode_fn）
+
+        用于 bge-m3 API encode 的场景，避免重复调用 encode_fn。
+
+        Args:
+            docs: 文档列表（包含 content, facts, id 等字段）
+            vectors: 预计算的向量列表（已归一化，每条 1024 维）
+        """
+        if not docs or not vectors:
+            return
+
+        if self.index is None:
+            # 初始化索引（向量维度从第一个向量获取）
+            import numpy as np
+            first_vec = np.array(vectors[0], dtype=np.float32)
+            self._dim = len(first_vec)
+            self.index = faiss.IndexFlatIP(self._dim)
+
+        # 构建文档记录
+        for doc, vec in zip(docs, vectors):
+            text = doc.get('content', '') + ' ' + ' '.join(str(f) for f in doc.get('facts', []))
+            self.documents.append({
+                'id': doc.get('id', ''),
+                'text': text,
+                'facts': doc.get('facts', []),
+                'category': doc.get('category', 'unknown'),
+                'timestamp': doc.get('timestamp', ''),
+            })
+
+        # 向量归一化 + 添加到 FAISS
+        import numpy as np
+        vectors_np = np.array(vectors, dtype=np.float32)
+        vectors_np = self._normalize(vectors_np)
+        self.index.add(vectors_np)
+
     def search(self, query_vector: np.ndarray, top_k: int = 10) -> list[tuple[int, float]]:
         """搜索最近邻
 

src/storage/jina_rerank.py

@@ -0,0 +1,219 @@
+"""Jina Rerank — 模力方舟 API
+
+API: https://ai.gitee.com/v1/rerank
+模型: jina-embeddings-v4
+用途: 搜索结果重排，提升 top-k 准确性
+
+依赖:
+    pip install requests aiohttp
+"""
+import os
+import json
+import requests
+from typing import Optional
+from dataclasses import dataclass
+from pathlib import Path
+
+# 模力方舟 API 配置
+RERANK_API_URL = "https://ai.gitee.com/v1/rerank"
+RERANK_API_KEY = "3TSVVXRFFECE4TISXHGE1VXDAXBIPAP6O1VPJK18"
+# 模力方舟 bge-reranker-v2-m3（不是 jina-embeddings-v4）
+DEFAULT_MODEL = "bge-reranker-v2-m3"
+
+# 缓存目录
+CACHE_DIR = Path.home() / "projects" / "zhiyi" / "data"
+CACHE_DIR.mkdir(parents=True, exist_ok=True)
+RERANK_CACHE_FILE = CACHE_DIR / "rerank_cache.jsonl"
+
+
+@dataclass
+class RerankResult:
+    """单条重排结果"""
+    index: int          # 原始索引
+    document: str       # 文档内容
+    score: float        # 关联度分数（越高越相关）
+
+
+@dataclass
+class RerankResponse:
+    """重排响应"""
+    results: list[RerankResult]
+    model: str
+    usage: dict
+
+
+def _make_cache_key(query: str, docs: list[str]) -> str:
+    """生成缓存 key（query + 前3个 doc 的 hash）"""
+    import hashlib
+    key_str = query + "|" + "|".join(docs[:3])
+    return hashlib.md5(key_str.encode()).hexdigest()
+
+
+def _load_rerank_cache() -> dict:
+    """加载 rerank 缓存"""
+    if not RERANK_CACHE_FILE.exists():
+        return {}
+    cache = {}
+    with open(RERANK_CACHE_FILE, "r", encoding="utf-8") as f:
+        for line in f:
+            line = line.strip()
+            if not line:
+                continue
+            try:
+                item = json.loads(line)
+                cache[item["key"]] = item["results"]
+            except json.JSONDecodeError:
+                continue
+    return cache
+
+
+def _save_rerank_cache(key: str, results: list[dict]):
+    """追加写入 rerank 缓存"""
+    with open(RERANK_CACHE_FILE, "a", encoding="utf-8") as f:
+        f.write(json.dumps({"key": key, "results": results}, ensure_ascii=False) + "\n")
+
+
+class JinaRerankClient:
+    """Jina Rerank 客户端（模力方舟 API）
+
+    用法:
+        client = JinaRerankClient()
+        results = client.rerank("查询", ["文档1", "文档2", ...], top_n=5)
+    """
+
+    def __init__(self, api_url: str = RERANK_API_URL, api_key: str = RERANK_API_KEY, model: str = DEFAULT_MODEL):
+        self.api_url = api_url
+        self.api_key = api_key
+        self.model = model
+        self._cache = _load_rerank_cache()
+        self._cache_hits = 0
+        self._cache_misses = 0
+
+    def rerank(
+        self,
+        query: str,
+        documents: list[str],
+        top_n: int = 10,
+        use_cache: bool = True,
+    ) -> RerankResponse:
+        """对文档列表进行重排
+
+        Args:
+            query: 用户查询（搜索词）
+            documents: 待重排的文档列表（通常来自 FAISS 粗排 top-50）
+            top_n: 返回前 N 条最相关结果
+            use_cache: 是否使用缓存
+
+        Returns:
+            RerankResponse（包含排序后的结果列表）
+        """
+        if not documents:
+            return RerankResponse(results=[], model=self.model, usage={})
+
+        # 查缓存
+        if use_cache:
+            cache_key = _make_cache_key(query, documents)
+            if cache_key in self._cache:
+                self._cache_hits += 1
+                cached_results = self._cache[cache_key]
+                results = [RerankResult(**r) for r in cached_results[:top_n]]
+                return RerankResponse(results=results, model=self.model, usage={"cached": True})
+            self._cache_misses += 1
+
+        # 调用 API
+        headers = {
+            "Authorization": f"Bearer {self.api_key}",
+            "Content-Type": "application/json",
+        }
+        payload = {
+            "model": self.model,
+            "query": query,
+            "documents": documents,
+            "top_n": min(top_n, len(documents)),
+        }
+
+        resp = requests.post(self.api_url, headers=headers, json=payload, timeout=60)
+        resp.raise_for_status()
+        result = resp.json()
+
+        # 解析结果
+        rerank_results = []
+        for item in result.get("results", []):
+            r = RerankResult(
+                index=item["index"],
+                document=item["document"],
+                score=item["relevance_score"],
+            )
+            rerank_results.append(r)
+
+        # 写入缓存
+        if use_cache:
+            cache_key = _make_cache_key(query, documents)
+            _save_rerank_cache(cache_key, [
+                {"index": r.index, "document": r.document, "score": r.score}
+                for r in rerank_results
+            ])
+            self._cache[cache_key] = [
+                {"index": r.index, "document": r.document, "score": r.score}
+                for r in rerank_results
+            ]
+
+        return RerankResponse(
+            results=rerank_results[:top_n],
+            model=self.model,
+            usage=result.get("usage", {}),
+        )
+
+    def stats(self) -> dict:
+        """返回缓存统计"""
+        total = self._cache_hits + self._cache_misses
+        hit_rate = self._cache_hits / total if total > 0 else 0
+        return {
+            "model": self.model,
+            "cache_hits": self._cache_hits,
+            "cache_misses": self._cache_misses,
+            "hit_rate": f"{hit_rate:.1%}",
+        }
+
+    def clear_cache(self):
+        """清空重排缓存"""
+        self._cache.clear()
+        if RERANK_CACHE_FILE.exists():
+            RERANK_CACHE_FILE.unlink()
+
+
+# 全局单例
+_rerank_client: Optional[JinaRerankClient] = None
+
+
+def get_rerank_client() -> JinaRerankClient:
+    """获取 Jina Rerank 客户端单例"""
+    global _rerank_client
+    if _rerank_client is None:
+        _rerank_client = JinaRerankClient()
+    return _rerank_client
+
+
+if __name__ == "__main__":
+    # 测试（需要网络）
+    print("Testing Jina Rerank...")
+
+    client = JinaRerankClient()
+
+    docs = [
+        "苹果是一种水果",
+        "苹果手机是苹果公司的产品",
+        "香蕉是一种热带水果",
+        "华为是中国的手机品牌",
+        "Python是一种编程语言",
+    ]
+
+    try:
+        result = client.rerank("水果", docs, top_n=3)
+        print(f"重排结果 (query='水果'):")
+        for r in result.results:
+            print(f"  score={r.score:.3f} idx={r.index} doc={r.document}")
+    except Exception as e:
+        print(f"Rerank API 调用失败（正常，测试需要网络）: {e}")
+
+    print("\nRerank 缓存统计:", client.stats())