Windows CI intermittently returns zero hybrid hits right after a fast
seed write (same class as "Nothing found on disk" on tiny collections).
Close the palace client after seeding so the next open re-reads flushed
segments, retry search once if empty, and assert non-empty results with
a clear message instead of IndexError.
The #2200 reap tests only monkeypatched HOME. On Windows expanduser("~")
reads USERPROFILE, so the reaper scanned the real home and the suite
failed on test-windows after Wave 2. Share _isolate_home() that sets both.
MemoryStack/Layer1 opened a second PersistentClient on the same palace
while the Hermes provider already held one for live filing. Concurrent
access corrupted local Chroma SQLite (disk I/O / Failed to get segments)
and failed CI on develop after #1915.
Wake-up L1 now scans the long-lived collection under the collection lock,
and filing holds that lock for the full upsert. Also rewrite the RFC 001
section-4.4 docstring to avoid the internal §N jargon guard.
_cleanup_mine_lock_file reclaims a lock correctly on the happy path (see
its own docstring for the flock-based rendezvous safety it already
handles) — but only for the specific lock a mine_lock context manager
just released. A process that dies before reaching its own finally
block (SIGKILL, force-quit, host crash) never runs that cleanup, and
nothing else in the codebase later revisits that lock file.
Found in the wild: one long-lived installation had 5,636 stale lock
files in ~/.mempalace/locks/, the oldest several months old, none held
by any live process (confirmed via lsof before cleanup). This is
distinct from the #1264 lock-holder-diagnostics fix (identifies who
holds a live lock) and the #1299 mcp_server embedding-function fix
(unrelated code path) — neither addresses orphan reclamation, and the
2026-07-10 outage postmortem comment in mcp_server.py's stdio loop
covers graceful client disconnection, not abrupt process death.
Adds reap_stale_mine_locks(), which reuses _cleanup_mine_lock_file
itself for the actual removal — same nonblocking-flock-reacquire safety
mechanism, same Windows/POSIX handling already tested in this file, no
duplicated locking logic. A lock is only ever removed after this
process re-acquires it, so anything genuinely held by a live process is
left untouched regardless of age. Wired into mine_lock() via a
throttled opportunistic call (_maybe_reap_stale_mine_locks, at most
once per 15 minutes) rather than a new background thread, scheduled
task, or CLI surface — it piggybacks on the natural cadence of mining
rather than adding new infrastructure.
mine_palace_*.lock (the newer per-palace lock added for the #974/#965
fan-out fix) is explicitly skipped — it has its own lifecycle and
holder-identity tracking and doesn't have this failure mode.
Tests: 6 new cases in test_palace_locks.py covering removal of a
genuinely stale+unheld lock, preservation of a young lock regardless of
hold state, the core safety property (a lock held by another process is
never removed even when backdated past the age threshold), skipping
mine_palace_*-prefixed locks, a missing-lock-dir no-op, and the
throttle itself. Full existing test_palace_locks.py suite (19 tests)
passes unchanged. Broader tests/ -k 'palace or mine' run clean (730
passed) aside from two pre-existing failures confirmed unrelated and
present on an unmodified checkout (test_hnsw_capacity.py SQLite WAL
signature caching, test_repair.py FTS5 shadow-table write restriction —
both environment/SQLite-build-specific, neither touches locking).
Close the last open items on #743 before merge:
- Conformance: document two isolation arms (cross-id for all backends;
same-id/different-namespace for supports_namespace_isolation advertisers).
- No silent drop: non-advertising backends must raise UnsupportedCapabilityError
when PalaceRef.namespace is set, rather than accept-and-ignore.
- Wire require_namespace_support() into chroma/sqlite_exact; add conformance test.
- Refresh implementation-status banner now that #1727/#1731/#1732/#1734 landed.
After rebasing #1671 onto current develop:
- Opt test_embedding_api out of conftest's stable EF mock so the
get_embedding_function selection tests exercise the real factory.
- Move the CHANGELOG entry from released 3.7.0 Performance into
Unreleased Features (rebase context had drifted).
Address the gemini-code-assist review on #1671:
- Wrap `http.client.HTTPException` (BadStatusLine / IncompleteRead — common
with local/overloaded servers) and `ValueError` (invalid/missing URL scheme;
also subsumes `json.JSONDecodeError`) in `EmbeddingAPIError` instead of
letting them crash the caller.
- Reject a non-dict top-level JSON response before calling `.get()` on it, so a
JSON list/`null`/string yields a clear `EmbeddingAPIError` rather than an
unhandled `AttributeError`.
- Add tests for all three cases.
Add an `embedding_model: "openai-compat"` option that computes embeddings
via any OpenAI-compatible `/v1/embeddings` server (LM Studio, llama.cpp,
vLLM, Ollama's OpenAI shim, self-hosted) instead of a local ONNX model.
- New OpenAICompatEmbeddingFunction (stdlib urllib, no new dependency):
batches requests, asks for `encoding_format: "float"` and a custom
User-Agent (avoids Cloudflare 403, see #1570), validates the response
(contiguous 0..n-1 indices + well-formed vectors) before use, and
L2-normalizes for the cosine collection. Exposes `embed_query` (ChromaDB
1.5 dispatches query embedding through it, not `__call__`). `name()`
encodes the model id so switching it forces `mempalace repair
rebuild-index`. Failures raise a module-specific `EmbeddingAPIError`.
- Endpoint settings resolved by MempalaceConfig as a single source of truth:
`embedding_api_url` / `embedding_api_model` / `embedding_api_key`, each
overridable via the matching `MEMPALACE_EMBEDDING_API_*` env var.
Whitespace-only values are treated as unset; the EF cache key fingerprints
the key so a token rotation is picked up.
- The miner/MCP `Device:` header reports `openai-compat (<url>)` instead of a
misleading local accelerator label when this backend is active.
- Opt-in; default stays minilm. Mirrors the existing `openai-compat` LLM
provider naming; stays local when the endpoint is on the machine/LAN.
- Tests: tests/test_embedding_api.py (no server / no network required).
- Docs: README requirement note, module docstring, CHANGELOG.
Refs #1559.
`repair` in its default mode and `migrate` both back up the whole palace
directory with `shutil.copytree` before they overwrite it. `copytree` cannot
duplicate a Unix domain socket or a named pipe: it copies everything else and
then raises `shutil.Error`, so one such entry anywhere under the palace killed
the command at the backup step, before any rebuild ran. Re-running only
repeated it, because the backup already written held a complete
`chroma.sqlite3`, so the next run deleted it and died in the same place.
Both call sites now go through `copy_palace_dir`, which hands `copytree` an
`ignore` callback. The predicate is an allowlist, matching how the rest of the
package treats directory entries: only regular files and directories are
copied, and anything else is named and left out. Device nodes are skipped for
the opposite reason to sockets, since the copy does not fail on them but
dereferences them; a link to `/dev/zero` wrote 493 MB in 8 seconds here before
it was stopped.
A failed `stat` is never a reason to skip. It says the entry cannot be
resolved right now, not that it holds no data, and no errno separates the two:
a symlink into a volume that is not mounted fails exactly like one whose
target was deleted, and on Windows an unmapped drive letter and an unreachable
network share both arrive as `ENOENT` as well. Such an entry is left for the
copy to fail on, exactly as before this change, because the backup is the
safety net for the rebuild that overwrites the palace next. Five tests pin
that boundary, each asserting `shutil.Error` still comes out.
What was skipped is printed for the operator, and printed for a failed copy
too, since that is when knowing what the backup lacks matters most. The report
can fail on its own, because both callers pass `print`: an stdout that cannot
encode an entry's name, or a report long enough to flush partway through onto
a device that refuses the write. One such line costs neither the lines after
it nor, on the failure path, the copy's own exception, which a report that
raises would otherwise replace. On the success path it is not suppressed, so a
caller that passed something other than a working logger hears about it.
Co-Authored-By: marcoaperez <25101951+marcoaperez@users.noreply.github.com>
Restore the fresh-palace dry-run regression alongside the existing-empty
and initialized-sqlite_exact cases. Cover explicit failure for unsupported
adapter result types and daemon source-adapter dispatch. Ignore Hypothesis'
test cache so it cannot be committed.
- existing-uninitialized palace dir dry run: assert no chroma.sqlite3
- initialized sqlite_exact palace dry run: assert artifacts byte-identical
- replace mocked MineAlreadyRunning CLI test with a real held writer lease
repair_mojibake rewrote clean text. The [ÂÃ][continuation] alternative of
_HIGH_CONFIDENCE_RUN is only a two-character window, and Portuguese, Vietnamese
and Turkish end all-caps words in Ã/Â — IRMÃ, MAÇÃ, MANHÃ, BÃO, NHÃ, HÂLÂ, IMÂ.
Prose then follows with a closing quote, guillemet, ellipsis or dash, and all of
those are in the continuation class, so "«MAÇû." and "“IRMÔ" matched a shape
that is also perfectly clean text. Each hit collapsed two characters into one
and was reported as a successful repair.
Three failures came out of the same root cause:
* clean prose destroyed — "«MAÇû." became "«MAÇû.", "“IRMÔ" became "“IRMÔ";
* visible text swapped for invisible C1 controls, because  plus 0x80-0x9F
decodes to exactly that block, so "İMÂ… edildi." lost its ellipsis to U+0085;
* the max_passes loop was not idempotent: genuinely mojibaked Portuguese was
repaired correctly on pass 1 and then destroyed on pass 2.
The fix follows the precedent already set in this module, where Ä/Å were
excluded as leads because "Ų can be legitimate scientific text" — the same
argument, applied to the continuation side. A lone [ÂÃ] window ending in
typographic punctuation is no longer high-confidence, so it is repaired only
when a lowercase letter runs directly into the lead ("coûte", "Noël"), which
is where mojibake actually occurs — inside a word. Chained units ("ação") and
the 3-4 character â/ð/ï windows are unaffected. A decode product containing a
C0/C1 control is now always refused.
Corroboration is deliberately local. Inferring "this drawer is mojibake" from a
run elsewhere in the text destroys clean prose in a mixed drawer, and drawers
are mixed by construction since the miner concatenates several sources.
The trade is asymmetric and chosen on purpose: an unrepaired string is unchanged
and can be repaired later, while a wrong repair is silent and irrecoverable.
mempalace/_stdio.py states the rule in its own docstring -- "every console
entry point that touches stdio needs to fix this on Windows" -- and cli.py
and fact_checker.py both apply it. scripts/mempalace_repair_encoding.py,
added later, does not.
That matters more here than anywhere else: this tool exists for Windows users
whose palace carries legacy mojibake, and it prints a before/after preview for
every proposed change. Under the console codepage it was written for, the
preview cannot be encoded -- the lead bytes of the corruption it detects are
exactly the characters the codepage rejects -- so the run dies with
UnicodeEncodeError before repairing a single drawer.
Reproduced with stdout on cp936:
UnicodeEncodeError: 'gbk' codec can't encode character '\xc3'
in position 14: illegal multibyte sequence
U+00C3 is the lead character of the à family this tool repairs.
Uses replace on stdout/stderr, matching cli.py and fact_checker.py, because
the preview carries verbatim drawer text that may hold surrogate halves
round-tripped from filenames; strict would crash mid-preview.
Related to #1122 (same exception class on the main CLI's help output) but a
different entry point and a different fix -- that one can use ASCII-safe
static strings, this one prints arbitrary user content.
Protect the complete rebuild_index snapshot, rebuild/swap, and cleanup
cycle with the palace writer lease.
Without whole-operation quiescence, a concurrent writer can land after
the snapshot and be absent from the rebuilt index, recreating SQLite/HNSW
divergence immediately after repair.
Add regression canaries proving that:
- ChromaDB is never opened when the writer lease is unavailable
- the rebuild body executes while the writer lease remains held
The original sweep in this PR was scoped against develop as of April. Since
then upstream added console output carrying the same characters, so the fix
had drifted into covering roughly half of the affected surface.
Re-measured against current develop and replaced the remaining occurrences of
the same five characters this PR already targets -- U+2014 em-dash, U+2500 box
drawing, U+2192 arrow, U+25CF/U+25CB circles -- on every print(), input() and
logger call reachable from a terminal:
repair.py (10), cli.py (7), format_miner.py (7), mcp_server.py (5),
miner.py (2), onboarding.py (2), migrate.py (2), searcher.py (2),
service.py (1), embedding.py (1), dialect.py (1)
Substitutions match the ones already used in this branch: em-dash -> "--",
box drawing -> "-", arrow -> "->", filled/empty circles -> "#"/".".
Four assertions matched the previous literals and were updated alongside the
strings they cover (test_cli, test_miner x2, test_repair).
Docstrings, comments and stored content are deliberately untouched -- only
what reaches a terminal, where GBK/CP1252 consoles raise UnicodeEncodeError.
Each run_* entrypoint sets MEMPALACE_PALACE_PATH so downstream config
resolution sees that call's palace, and none of them put it back. In the
daemon that costs nothing — one call, one process. In any process making
two calls it leaks: config.py reads this variable with priority over the
config file, so the first call's palace silently becomes the second
caller's, and a tmpdir palace since removed reads back as
PalaceNotFoundError from an unrelated code path.
daemon.py already saves and restores exactly these keys around its own
dispatch. This gives run_mine, run_sync and run_diary_write the same
discipline through a small decorator.
Tests cover both directions — a caller whose variable was already set
gets it back unchanged, and a caller who had none keeps none. All four
fail with the decorators removed.
#2187 fixed the default-backend ingest failure (#2190) and covered the
numpy path: an ndarray-returning EF plus a real EmbeddingCollection.upsert.
Two branches of the fixed function are still unexercised.
(The defect was not NumPy-2.x-specific. `np.float32` has never been a
`float` subclass in any NumPy major -- verified on 1.24.4, 1.26.4 and
2.4.4 -- so `list(arr)` output was always rejected. It was latent until
0e79797 added `requires_explicit_embeddings` and routed the default
Chroma backend through `_embed_texts` for the first time.)
`_embed_texts` branches on `hasattr(v, "tolist")`. The `float(x)` arm is
the one that serves embedders returning plain sequences — custom/BYO EFs,
and rows arriving as tuples. Nothing ran it, so dropping the `float()`
call there stays green and reaches users as the same production
ValueError, just on a non-default embedder.
The `if not texts: return []` guard is likewise untested. Callers pass
empty batches (a drawer set fully filtered by dedup), and loading the EF
is the expensive part — on the ONNX default it spins up a native session.
Both tests are mutation-verified against this tree:
- replacing the fallback with `list(v)` fails
test_embed_texts_handles_plain_sequence_embedders
- deleting the early return fails
test_embed_texts_short_circuits_on_empty_input
The plain-sequence EF yields `Decimal`, so the assertion proves a real
conversion rather than values passing through unchanged, and the
empty-batch test asserts by making `get_embedding_function` raise, so it
verifies the EF is never constructed rather than only checking the
return value.
Tests only — no production code changes. They live in `test_embedding.py`
for the reason #2187 documented: conftest's autouse
`_stable_embedding_function_for_tests` replaces `_embed_texts` outright
for every module outside `_REAL_EMBEDDING_TEST_MODULES`.
Refs #2190
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Chroma declares `requires_explicit_embeddings`, so every write on the
default backend routes through `EmbeddingCollection`. `_embed_texts`
built its rows with `list(v)`, and `v` is a float32 `np.ndarray` — that
unpacks into `np.float32` *scalars*, which chromadb's
`normalize_embeddings` rejects:
ValueError: Expected embeddings to be a list of floats or ints, a
list of lists, a numpy array, or a list of numpy arrays
`mine` aborted on the first drawer, as did every other write against a
default palace. Convert with `.tolist()` (C-speed), keeping a
`float(x)` branch for embedders that already return plain sequences.
The suite could not see this. conftest's autouse
`_stable_embedding_function_for_tests` monkeypatches
`embedding_wrapper._embed_texts` itself for every module outside
`_REAL_EMBEDDING_TEST_MODULES`, so the defective function was never
executed under test. The regression tests therefore go in
`test_embedding.py`, which is exempt from that stub: one asserts the
returned elements are builtin floats, one drives a real Chroma
collection through `EmbeddingCollection.upsert` and reads the document
back. Both fail against the previous line with the production
ValueError.
Verified end to end outside the suite: mining a project and searching
it back returns the drawer verbatim, on the host and in the container
image built from this tree.
`mempalace_diary_write` returns an `entry_id` for every diary entry, but
for entries large enough to be chunked that id was unusable: get_drawer,
update_drawer and delete_drawer all answered "Drawer not found", and
list_drawers showed the entry as N unrelated chunk rows.
Two metadata conventions never met. The diary chunking path stamped
`parent_entry_id` on each chunk, while the logical-id read paths added in
#1782 query only `parent_drawer_id`. Both keys mean the same thing --
"physical chunk of this logical drawer" -- so chunk groups written by
diary_write were invisible to logical-id resolution. Same bug class as
#1763, which #1782 fixed for `add_drawer` drawers only.
Read paths now resolve either key via `_PARENT_ID_KEYS`:
- `_logical_chunk_group()` matches both with an `$or` (fixes get /
update / delete). All four backends support `$or`.
- `_collapse_drawer_rows()` groups on either (fixes list_drawers, which
the `$or` alone does not cover).
- `searcher._result_drawer_id()` resolves either, so a hit on a chunked
diary entry reports the id that fetches the whole entry rather than
the single chunk that matched.
New diary writes also stamp `parent_drawer_id` alongside
`parent_entry_id` so the two conventions converge going forward. Because
the read paths still accept the `parent_entry_id`-only shape, palaces
written before this fix are repaired with no data migration.
Diary chunks are written without `source_file`, so neighbor expansion
(#1580) returns early on them and is unaffected by the added key.
Also drops the comment telling callers to iterate `chunk_ids` (it
documented the bug as intended behavior) and a stale claim that search
rejoins chunks via `parent_entry_id` -- no search code read that key.
A long-lived MCP server imports mempalace and chromadb once and serves from
those in-memory modules for the life of the process, so an upgrade on disk
mid-session never reaches it and it keeps accepting writes produced by code
the user no longer has installed.
Refuse mutating tools with JSON-RPC -32005 once a watched distribution's
installed version differs from the snapshot taken at import, or once it is
gone entirely. Reads stay available, mempalace_status reports
library_versions, and MEMPALACE_MCP_ALLOW_STALE_LIBRARY=1 opts out.
Both sides of the comparison come from installed metadata rather than a live
module.__version__. A distribution whose metadata cannot be read, or whose
search root will not open, is reported and left uncompared rather than
treated as removed: importlib.metadata suppresses the failure at both of
those levels, so either one would otherwise look exactly like an uninstall
and refuse every write on a healthy install.
The same module also memoizes each search root's listing against that root's
mtime, read in seconds where this fingerprint compares nanoseconds. An
upgrade whose removal and creation both land inside one timestamp tick would
then be answered from the listing taken before it, naming a dist-info that is
already gone; its version reads as empty, the distribution is left
uncompared, and nothing moves that mtime afterwards, so the gate would stay
off for it for the rest of the process. Drop the memo before each reading.
Watch chromadb only when chromadb is the backend serving. It is a hard
dependency rather than an extra, so it is installed even for a palace kept
in Postgres, and watching it there would refuse that user's writes whenever
chromadb alone was upgraded, over a library that writes nothing they own. A
backend that cannot be resolved keeps it watched.
Skip a sys.path entry carrying an embedded NUL. os.stat and os.listdir refuse
it during argument conversion, raising ValueError rather than the OSError
those callers hold; POSIX never gets there because realpath rejects it first,
but Windows resolves it and one such entry would end the whole reading.
The gate sits ahead of the diverged-index refusal added since (-32004, which
is why this one takes -32005). That gate's remedy is `mempalace repair
rebuild-index`, which would run the installed code against a palace this
process is still writing with the superseded one, so the restart instruction
has to be the one that reaches the client; the index probe re-runs per call
and surfaces immediately after a restart. Ordering it this way also skips
that gate's segment probe on a call already refused. Both directions of the
precedence are pinned by tests.
Co-Authored-By: messelink <274674234+messelink@users.noreply.github.com>
The dependabot bump moved pyproject's ruff pin to 0.16.1, but nothing else
followed it, so the repo asked for three different versions at once:
- pyproject.toml said 0.16.1 (the bump)
- uv.lock still resolved 0.15.20 (dependabot did not update it)
- .github/workflows/ci.yml installed 0.15.14 by its own literal pin
The lint job never reads pyproject, so CI kept linting with 0.15.14 and
reported this PR green without 0.16.1 ever running. The ci.yml pin had
already drifted from pyproject before this bump, under a comment saying to
keep them identical.
- ci.yml: pin 0.16.1 to match pyproject.
- uv.lock: regenerated so the locked resolution agrees.
- test_ruff_pins_match: assert ci.yml and pyproject stay equal, so the next
bump that touches only one of them fails loudly instead of passing blind.
- extend-exclude '*.md': 0.16 began formatting Python inside markdown
fences, taking the formatter from 199 files to 295 and reflowing
hand-aligned example code in docs/rfcs/002 and three website pages.
Excluding docs keeps this a version bump rather than a silent
documentation reflow, and restores the exact file scope the project has
always formatted.
Verified with 0.16.1 actually installed: ruff check and ruff format --check
both clean over the same 199 files, full suite green (3669 passed).
Igor Lins e Silva
mvalentsev
Offbeat-Breed
maximilize
Grace Gettert
Mohab Mohie
Arnold Wender
KeilerHirsch
Joshua Fontany
Mark Beacom