Windows CI intermittently returns zero hybrid hits right after a fast
seed write (same class as "Nothing found on disk" on tiny collections).
Close the palace client after seeding so the next open re-reads flushed
segments, retry search once if empty, and assert non-empty results with
a clear message instead of IndexError.
Move Unreleased entries into the 3.7.0 section, add the Wave 2 headlines
(Hermes core, source adapters, stale-library MCP gate, date search,
openai-compat embeddings, integrity/encoding fixes), and set the release
date to 2026-08-11.
The #2200 reap tests only monkeypatched HOME. On Windows expanduser("~")
reads USERPROFILE, so the reaper scanned the real home and the suite
failed on test-windows after Wave 2. Share _isolate_home() that sets both.
MemoryStack/Layer1 opened a second PersistentClient on the same palace
while the Hermes provider already held one for live filing. Concurrent
access corrupted local Chroma SQLite (disk I/O / Failed to get segments)
and failed CI on develop after #1915.
Wake-up L1 now scans the long-lived collection under the collection lock,
and filing holds that lock for the full upsert. Also rewrite the RFC 001
section-4.4 docstring to avoid the internal §N jargon guard.
_cleanup_mine_lock_file reclaims a lock correctly on the happy path (see
its own docstring for the flock-based rendezvous safety it already
handles) — but only for the specific lock a mine_lock context manager
just released. A process that dies before reaching its own finally
block (SIGKILL, force-quit, host crash) never runs that cleanup, and
nothing else in the codebase later revisits that lock file.
Found in the wild: one long-lived installation had 5,636 stale lock
files in ~/.mempalace/locks/, the oldest several months old, none held
by any live process (confirmed via lsof before cleanup). This is
distinct from the #1264 lock-holder-diagnostics fix (identifies who
holds a live lock) and the #1299 mcp_server embedding-function fix
(unrelated code path) — neither addresses orphan reclamation, and the
2026-07-10 outage postmortem comment in mcp_server.py's stdio loop
covers graceful client disconnection, not abrupt process death.
Adds reap_stale_mine_locks(), which reuses _cleanup_mine_lock_file
itself for the actual removal — same nonblocking-flock-reacquire safety
mechanism, same Windows/POSIX handling already tested in this file, no
duplicated locking logic. A lock is only ever removed after this
process re-acquires it, so anything genuinely held by a live process is
left untouched regardless of age. Wired into mine_lock() via a
throttled opportunistic call (_maybe_reap_stale_mine_locks, at most
once per 15 minutes) rather than a new background thread, scheduled
task, or CLI surface — it piggybacks on the natural cadence of mining
rather than adding new infrastructure.
mine_palace_*.lock (the newer per-palace lock added for the #974/#965
fan-out fix) is explicitly skipped — it has its own lifecycle and
holder-identity tracking and doesn't have this failure mode.
Tests: 6 new cases in test_palace_locks.py covering removal of a
genuinely stale+unheld lock, preservation of a young lock regardless of
hold state, the core safety property (a lock held by another process is
never removed even when backdated past the age threshold), skipping
mine_palace_*-prefixed locks, a missing-lock-dir no-op, and the
throttle itself. Full existing test_palace_locks.py suite (19 tests)
passes unchanged. Broader tests/ -k 'palace or mine' run clean (730
passed) aside from two pre-existing failures confirmed unrelated and
present on an unmodified checkout (test_hnsw_capacity.py SQLite WAL
signature caching, test_repair.py FTS5 shadow-table write restriction —
both environment/SQLite-build-specific, neither touches locking).
Close the last open items on #743 before merge:
- Conformance: document two isolation arms (cross-id for all backends;
same-id/different-namespace for supports_namespace_isolation advertisers).
- No silent drop: non-advertising backends must raise UnsupportedCapabilityError
when PalaceRef.namespace is set, rather than accept-and-ignore.
- Wire require_namespace_support() into chroma/sqlite_exact; add conformance test.
- Refresh implementation-status banner now that #1727/#1731/#1732/#1734 landed.
#757 landed mtime/inode cache invalidation and mempalace_reconnect
in mcp_server._get_client(). Both are Chroma-specific (stat of
chroma.sqlite3). They should migrate into ChromaBackend.get_collection
and ChromaBackend.close_palace during the §10 cleanup so the freshness
contract lives inside the backend, not in the caller.
Addresses the actual spec defects flagged in #743 review, ignoring
operator-UX asks that are not plugin-contract concerns.
- Goal #3: 'without data loss' → mirrors §8.2's capability-conditional
lossless-vs-reembed framing. No more overpromise.
- §1.5: `server_embedder` is no longer an implicit escape hatch from
identity/dimension rules. Such backends MUST expose an effective
identity via `effective_embedder_identity()` and are bound by the
same three-state check.
- §7.3: adds `maintenance_kinds: ClassVar[frozenset[str]]` advertisement
mechanism. `run_maintenance(kind)` must raise
UnsupportedMaintenanceKindError for unadvertised kinds. Benchmark
harness reads this set rather than guessing kind names. Reserves
`analyze`/`compact`/`reindex` as well-defined names.
- §1.2: adds `update()` as optional method with a default get+merge+
upsert implementation. §2.1: `supports_update` redefined to gate
atomic single-round-trip semantics (not mere capability), since the
default impl already supports partial updates.
Operator asks explicitly NOT adopted (diplomatic shims, not contract
defects): `.to_dict()` compat on typed results, migration progress
reporting, `BaseBackend.repair()` separate from `run_maintenance`,
per-palace capability variance, identity recording on read-only ops.
Incorporates review feedback from skuznetsov (Postgres, #665) and
dekoza (Lance, #574) on issue #737:
- §1.5: split 'accepts embeddings=' (signature compliance) from
'persists embeddings as-is' (correctness). Adds
supports_embeddings_passthrough capability; the former is universal,
the latter is required to label a migration lossless.
- §1.5: model identity check becomes a three-state machine
(known_match / known_mismatch / unknown) so legacy palaces without
recorded identity don't hard-fail on upgrade.
- §1.4: makes explicit that supports_contains_fast is the ONLY
performance floor the spec promises; without it callers MUST assume
O(n). $contains is a correctness requirement, not a performance one.
- §3.3: clarifies auto-detect is an upgrade-compat path only, never
the selection mechanism for new palaces.
- §8.2: migrate CLI refuses to run against a target lacking
supports_embeddings_passthrough unless --accept-re-embed is passed;
migration record now captures lossless status and model identities.
Copilot review flagged back-references in §1.4 and §6 that still used
the pre-skuznetsov-rename names (`$contains_fast`, `sync_capable`,
`change_feed`). Updated to the `supports_*` prefix used in the §2.1
capability table.
Formalizes the BaseCollection/BaseBackend contract introduced as a seam
in #413 into an interchangeability spec that third-party backends can
build to. Driven by six in-flight backend PRs (#574, #643, #665, #697,
#700, #381) each implementing the interface differently.
Key decisions captured: entry-point distribution, typed QueryResult/
GetResult replacing Chroma dict shape, daemon-first multi-palace model
via PalaceRef, required where-clause subset (incl. $contains),
mandatory embedder injection with model-identity validation, capability
tokens, shared pytest conformance suite, and a backend-neutral
migrate/verify CLI.
After rebasing #1671 onto current develop:
- Opt test_embedding_api out of conftest's stable EF mock so the
get_embedding_function selection tests exercise the real factory.
- Move the CHANGELOG entry from released 3.7.0 Performance into
Unreleased Features (rebase context had drifted).
Address the gemini-code-assist review on #1671:
- Wrap `http.client.HTTPException` (BadStatusLine / IncompleteRead — common
with local/overloaded servers) and `ValueError` (invalid/missing URL scheme;
also subsumes `json.JSONDecodeError`) in `EmbeddingAPIError` instead of
letting them crash the caller.
- Reject a non-dict top-level JSON response before calling `.get()` on it, so a
JSON list/`null`/string yields a clear `EmbeddingAPIError` rather than an
unhandled `AttributeError`.
- Add tests for all three cases.
Add an `embedding_model: "openai-compat"` option that computes embeddings
via any OpenAI-compatible `/v1/embeddings` server (LM Studio, llama.cpp,
vLLM, Ollama's OpenAI shim, self-hosted) instead of a local ONNX model.
- New OpenAICompatEmbeddingFunction (stdlib urllib, no new dependency):
batches requests, asks for `encoding_format: "float"` and a custom
User-Agent (avoids Cloudflare 403, see #1570), validates the response
(contiguous 0..n-1 indices + well-formed vectors) before use, and
L2-normalizes for the cosine collection. Exposes `embed_query` (ChromaDB
1.5 dispatches query embedding through it, not `__call__`). `name()`
encodes the model id so switching it forces `mempalace repair
rebuild-index`. Failures raise a module-specific `EmbeddingAPIError`.
- Endpoint settings resolved by MempalaceConfig as a single source of truth:
`embedding_api_url` / `embedding_api_model` / `embedding_api_key`, each
overridable via the matching `MEMPALACE_EMBEDDING_API_*` env var.
Whitespace-only values are treated as unset; the EF cache key fingerprints
the key so a token rotation is picked up.
- The miner/MCP `Device:` header reports `openai-compat (<url>)` instead of a
misleading local accelerator label when this backend is active.
- Opt-in; default stays minilm. Mirrors the existing `openai-compat` LLM
provider naming; stays local when the endpoint is on the machine/LAN.
- Tests: tests/test_embedding_api.py (no server / no network required).
- Docs: README requirement note, module docstring, CHANGELOG.
Refs #1559.
Igor Lins e Silva
mvalentsev
Offbeat-Breed
maximilize