fix: point diverged-index recovery at from-sqlite, not re-mine (#1843)

A diverged HNSW index (for example after a failed chromadb compaction)
leaves the drawer rows intact in chroma.sqlite3 but the vector index out
of sync. Re-mining to recover silently drops drawers added through the
MCP server and diary entries, which have no source file.

- repair-status now recommends `mempalace repair --mode from-sqlite
  --archive-existing` when DIVERGED, instead of the generic `mempalace
  repair`, and explains why re-mining loses data.
- The shared recall protocol and the recall skills (Cursor + Claude
  plugin) document the compactor / "Not connected" recovery path:
  stop the server, rebuild from SQLite, verify, restart — never repair
  in-process from the agent.

Complements #1847 (legacy repair error messages); does not duplicate it.
Does not close #1843 — MCP reconnect resilience and honest add_drawer
write signalling remain open.

Co-authored-by: Cursor <cursoragent@cursor.com>

.claude-plugin/skills/mempalace-recall/SKILL.md

@@ -53,6 +53,13 @@ a variable, fixing a typo). Recall is question-driven, not reflexive.
   an answer. Offer to widen the search or file the new information.
 - **MCP error / server down** — surface the error, suggest `mempalace
   status` or re-running `/init`; never fall back to guessing.
+- **Palace index corrupt / compactor error** — if the server reports an
+  HNSW segment-writer error, a ChromaDB compaction failure, or stays
+  "Not connected" after a write, the index is out of sync with
+  `chroma.sqlite3` but the rows are intact. Tell the user to stop the
+  server and rebuild from SQLite (`mempalace repair --mode from-sqlite
+  --archive-existing --yes`), not re-mine, which drops MCP-added drawers
+  and diary entries (#1843). Do not repair in-process.
 - **Conflicting facts** — trust the knowledge graph's time-valid answer;
   invalidate-then-add rather than overwriting silently.
 

integrations/shared/recall-protocol.md

@@ -67,10 +67,37 @@ question — not a system prompt or pasted conversation) plus optional
 - **MCP unavailable / tool error.** Surface the error plainly and suggest
   the user verify the server (`mempalace status`, or re-run install).
   Do not silently fall back to guessing from model memory.
+- **Palace index corrupt / compactor error.** When the server returns an
+  error mentioning the HNSW segment writer, a ChromaDB compaction
+  failure, or a stuck "Not connected" state after a write, the on-disk
+  vector index is out of sync with `chroma.sqlite3` — but the drawer rows
+  are intact in SQLite. Recover by rebuilding the index from SQLite, not
+  by re-mining. See "Recovering a corrupt index" below. Do not attempt an
+  in-process repair from the agent; guide the user to run the CLI.
 - **Stale or conflicting facts.** Prefer the knowledge graph's
   time-valid answer; if a fact has changed, invalidate the old one and
   add the new one rather than overwriting context silently.
 
+## Recovering a corrupt index
+
+A ChromaDB compaction failure can leave the drawers HNSW index out of
+sync with `chroma.sqlite3` and wedge the MCP server (every call returns
+"Not connected"). The data is safe in SQLite; rebuild the index from it.
+Guide the user through these CLI steps — never run an in-process rebuild
+from the agent (it can break other live clients):
+
+1. Stop the MCP server (kill the `mempalace-mcp` process, or restart the
+   host editor).
+2. Optional backup: `cp -a ~/.mempalace/palace ~/.mempalace/palace.bak.$(date +%F)`
+3. Rebuild from SQLite:
+   `mempalace repair --mode from-sqlite --archive-existing --yes`
+4. Verify: `mempalace repair-status` (divergence should read 0).
+5. Restart the MCP server.
+
+Do **not** re-mine from source files to recover: re-mining drops drawers
+added through the MCP server and diary entries, which have no source file
+(see MemPalace issue #1843).
+
 ## Anti-patterns
 
 - Answering about past work, people, or decisions from model memory when

mempalace/repair.py

@@ -1309,7 +1309,15 @@ def status(palace_path=None, collection_name: Optional[str] = None) -> dict:
             print(f"    note:           {info['message']}")
 
     if drawers["diverged"] or closets["diverged"]:
-        print("\n  Recommended: run `mempalace repair` to rebuild the index.")
+        print(
+            "\n  Recommended: rebuild the index from SQLite rather than re-mining:\n"
+            "\n      mempalace repair --mode from-sqlite --archive-existing\n"
+            "\n  A diverged index usually means the HNSW segment is out of sync with\n"
+            "  chroma.sqlite3 (for example a failed chromadb HNSW compaction). The\n"
+            "  drawer rows are intact in SQLite, so --mode from-sqlite recovers them.\n"
+            "  Do not re-mine from source files: that would drop drawers added via\n"
+            "  the MCP server and diary entries, which have no source file (#1843)."
+        )
     print()
     return {"drawers": drawers, "closets": closets}
 

skills/mempalace-recall/SKILL.md

@@ -90,6 +90,20 @@ question — not a system prompt or pasted conversation) plus optional
 - **MCP error / server down.** Surface the error and suggest the user
   run `mempalace status` or re-run `/mempalace-init`. Never fall back to
   guessing.
+- **Palace index corrupt / compactor error.** If the server reports an
+  HNSW segment-writer error, a ChromaDB compaction failure, or stays
+  "Not connected" after a write, the vector index is out of sync with
+  `chroma.sqlite3` while the drawer rows remain intact. Tell the user to
+  stop the server and rebuild from SQLite — do not re-mine, which drops
+  MCP-added drawers and diary entries (#1843):
+
+  ```bash
+  mempalace repair --mode from-sqlite --archive-existing --yes
+  mempalace repair-status
+  ```
+
+  Do not attempt an in-process repair from the agent. Full steps are in
+  the shared protocol's "Recovering a corrupt index" section.
 - **Conflicting facts.** Trust the knowledge graph's time-valid answer;
   invalidate-then-add rather than overwriting silently.
 

tests/test_hnsw_capacity.py

@@ -588,7 +588,9 @@ def test_bm25_fallback_handles_short_query(palace_with_drawers):
 
 
 def test_repair_status_reports_diverged(tmp_path, capsys):
-    """The status command prints DIVERGED and recommends rebuild."""
+    """The status command prints DIVERGED and recommends the from-sqlite
+    rebuild (not a re-mine), since a diverged index means the rows are
+    intact in sqlite but the HNSW segment is out of sync (#1843)."""
     from mempalace.repair import status as repair_status
 
     seg = "seg-status"
@@ -597,7 +599,8 @@ def test_repair_status_reports_diverged(tmp_path, capsys):
     out = repair_status(palace_path=str(tmp_path))
     captured = capsys.readouterr().out
     assert "DIVERGED" in captured
-    assert "mempalace repair`" in captured
+    assert "mempalace repair --mode from-sqlite --archive-existing" in captured
+    assert "Do not re-mine" in captured
     assert out["drawers"]["diverged"] is True