Address the last two Codex review comments on #2079:
- Reopen sqlite_exact immutable read-only connections when WAL/SHM
sidecars appear after a clean-database open so recall sees a later
writer's commits instead of a frozen pre-writer snapshot.
- Clear per-palace embedder-identity validation when MCP discards
storage handles during ownership promotion, so an empty read-only
open cannot skip recording the active model on the first write.
hnsw_capacity_status() (chroma.py) exists precisely to preflight the
#1222 SIGSEGV/pyo3-panic class before anything touches the HNSW
segment, but repo-wide it was wired into only 4 call sites while raw
count()/collection.count() is called at 20+ others -- a bare
except Exception around count() cannot catch a native crash, since
the process dies regardless of any Python try/except. This wires the
existing, already-tested probe into the 7 remaining call sites the
audit identified as CRITICAL:
- #89 palace.py::_enforce_embedder_identity -- the universal
get_collection() chokepoint every tool passes through, previously
guarded only by except Exception. Highest leverage: skips this
bookkeeping-only check on divergence instead of risking count().
- #90 migrate.py::migrate -- routes straight to the same
SQLite-extraction fallback the except branch already used, instead
of ever reaching col.count() when diverged.
- #91 repair.py::scan_palace / prune_corrupt -- both abort with the
existing from-sqlite recovery guidance instead of opening the
collection.
- #10 repair.py::rebuild_index -- preflights divergence alongside its
existing sqlite-integrity and poisoned-max-seq-id preflights, before
opening the collection.
- #13 repair.py::rebuild_index never rebuilt or reported on the
closets collection -- now warns when closets is still diverged
after a drawers-only rebuild, pointing at --mode from-sqlite instead
of letting 'Repair complete' stand unqualified.
- #92 dedup.py::get_source_groups -- takes an optional palace_path
(threaded from both callers) to preflight before count(); omitted by
existing tests, which keep their pre-existing behavior.
- #93 miner.py::status -- preflights before the ChromaDB-client
fallback path (used when the direct sqlite read is unavailable).
7 new regression tests, each confirmed failing against the pre-fix
code (via git stash of the source files only) and passing after the
fix. One existing dedup.py test updated for the new palace_path kwarg
in its call-signature assertion. Full suite: 3154 passed, 1 unrelated
pre-existing flake (test_mcp_server.py peer-writer-lock module-global
state leaking across test files in full-suite ordering -- this diff
never touches mcp_server.py).
Completes the embedder-identity contract for the last backend (qdrant) and
unifies identity persistence across all local-path backends.
Identity is stored in a small per-palace sidecar (mempalace_embedder.json),
NOT in a backend's mismatch marker. The marker's presence signals "palace
initialized" (reads raise CollectionNotInitializedError when the marker exists
but the store doesn't), so recording identity at first empty open must not
create it — a sidecar is unguarded, so a brand-new palace records identity
immediately. This fixes a latent gap (caught in review) where pgvector/qdrant
palaces stayed permanently "unknown" because the marker isn't written until the
first real write.
- New mempalace/backends/_sidecar.py: shared read/write_embedder_sidecar with
the isinstance robustness the review bots taught us; chroma, pgvector, and
qdrant all use it (chroma's inline copy is removed, pgvector switches off the
marker, qdrant adds it).
- QdrantCollection.get/set_embedder_identity delegate to the sidecar, so
palace.get_collection enforces a model swap on a qdrant palace exactly like
the other backends — no live server needed for the check.
Tests: sidecar roundtrip + brand-new-palace recording (creates sidecar without
a marker) + enforcement model-swap raise, for qdrant and pgvector, all
server-free; chroma identity tests unchanged. Full suite: 2495 passed, 82.52%.
Closes#1730. Refs #743, #1724.
The explicit-embedding backends (pgvector, qdrant, sqlite_exact) embed through
EmbeddingCollection and persist nothing about which model produced their
vectors. Swapping two same-dimension models (minilm <-> embeddinggemma, both
384-d) silently corrupts retrieval with no error — the worst failure class for
a verbatim-recall system. This records the model name and refuses a swap on open.
- base.py: EmbedderIdentity dataclass, Embedder protocol,
EmbedderIdentityUnknownWarning, and a three-state check_embedder_identity()
helper (known_match / known_mismatch / unknown) raising
EmbedderIdentityMismatchError / DimensionMismatchError. BaseCollection gains
get/set/effective_embedder_identity hooks.
- embedding.py: current_model_name() (cheap) + get_embedder_identity() (probed
dimension, cached per process).
- Persistence per backend: sqlite_exact meta table, pgvector marker JSON
(preserved across rewrites), chroma sidecar JSON. EmbeddingCollection forwards
the hooks explicitly (BaseCollection methods shadow __getattr__).
- palace.get_collection enforces at open: a model swap raises; a brand-new empty
collection records the current model; a populated-but-unrecorded legacy palace
warns and is resolved via the CLI. The check needs no model load.
- CLI: `mempalace palace set-embedder [--model NAME] [--force]` records/overrides
identity without mutating global config or loading a foreign model.
Chroma additionally keeps its native embedding-function check. Qdrant identity
persistence is a fast-follow (no local metadata slot — needs a companion store).
Closes#1724. Refs #743.
Igor Lins e Silva