Windows CI intermittently returns zero hybrid hits right after a fast
seed write (same class as "Nothing found on disk" on tiny collections).
Close the palace client after seeding so the next open re-reads flushed
segments, retry search once if empty, and assert non-empty results with
a clear message instead of IndexError.
Move Unreleased entries into the 3.7.0 section, add the Wave 2 headlines
(Hermes core, source adapters, stale-library MCP gate, date search,
openai-compat embeddings, integrity/encoding fixes), and set the release
date to 2026-08-11.
The #2200 reap tests only monkeypatched HOME. On Windows expanduser("~")
reads USERPROFILE, so the reaper scanned the real home and the suite
failed on test-windows after Wave 2. Share _isolate_home() that sets both.
MemoryStack/Layer1 opened a second PersistentClient on the same palace
while the Hermes provider already held one for live filing. Concurrent
access corrupted local Chroma SQLite (disk I/O / Failed to get segments)
and failed CI on develop after #1915.
Wake-up L1 now scans the long-lived collection under the collection lock,
and filing holds that lock for the full upsert. Also rewrite the RFC 001
section-4.4 docstring to avoid the internal §N jargon guard.
_cleanup_mine_lock_file reclaims a lock correctly on the happy path (see
its own docstring for the flock-based rendezvous safety it already
handles) — but only for the specific lock a mine_lock context manager
just released. A process that dies before reaching its own finally
block (SIGKILL, force-quit, host crash) never runs that cleanup, and
nothing else in the codebase later revisits that lock file.
Found in the wild: one long-lived installation had 5,636 stale lock
files in ~/.mempalace/locks/, the oldest several months old, none held
by any live process (confirmed via lsof before cleanup). This is
distinct from the #1264 lock-holder-diagnostics fix (identifies who
holds a live lock) and the #1299 mcp_server embedding-function fix
(unrelated code path) — neither addresses orphan reclamation, and the
2026-07-10 outage postmortem comment in mcp_server.py's stdio loop
covers graceful client disconnection, not abrupt process death.
Adds reap_stale_mine_locks(), which reuses _cleanup_mine_lock_file
itself for the actual removal — same nonblocking-flock-reacquire safety
mechanism, same Windows/POSIX handling already tested in this file, no
duplicated locking logic. A lock is only ever removed after this
process re-acquires it, so anything genuinely held by a live process is
left untouched regardless of age. Wired into mine_lock() via a
throttled opportunistic call (_maybe_reap_stale_mine_locks, at most
once per 15 minutes) rather than a new background thread, scheduled
task, or CLI surface — it piggybacks on the natural cadence of mining
rather than adding new infrastructure.
mine_palace_*.lock (the newer per-palace lock added for the #974/#965
fan-out fix) is explicitly skipped — it has its own lifecycle and
holder-identity tracking and doesn't have this failure mode.
Tests: 6 new cases in test_palace_locks.py covering removal of a
genuinely stale+unheld lock, preservation of a young lock regardless of
hold state, the core safety property (a lock held by another process is
never removed even when backdated past the age threshold), skipping
mine_palace_*-prefixed locks, a missing-lock-dir no-op, and the
throttle itself. Full existing test_palace_locks.py suite (19 tests)
passes unchanged. Broader tests/ -k 'palace or mine' run clean (730
passed) aside from two pre-existing failures confirmed unrelated and
present on an unmodified checkout (test_hnsw_capacity.py SQLite WAL
signature caching, test_repair.py FTS5 shadow-table write restriction —
both environment/SQLite-build-specific, neither touches locking).
Close the last open items on #743 before merge:
- Conformance: document two isolation arms (cross-id for all backends;
same-id/different-namespace for supports_namespace_isolation advertisers).
- No silent drop: non-advertising backends must raise UnsupportedCapabilityError
when PalaceRef.namespace is set, rather than accept-and-ignore.
- Wire require_namespace_support() into chroma/sqlite_exact; add conformance test.
- Refresh implementation-status banner now that #1727/#1731/#1732/#1734 landed.
#757 landed mtime/inode cache invalidation and mempalace_reconnect
in mcp_server._get_client(). Both are Chroma-specific (stat of
chroma.sqlite3). They should migrate into ChromaBackend.get_collection
and ChromaBackend.close_palace during the §10 cleanup so the freshness
contract lives inside the backend, not in the caller.
Addresses the actual spec defects flagged in #743 review, ignoring
operator-UX asks that are not plugin-contract concerns.
- Goal #3: 'without data loss' → mirrors §8.2's capability-conditional
lossless-vs-reembed framing. No more overpromise.
- §1.5: `server_embedder` is no longer an implicit escape hatch from
identity/dimension rules. Such backends MUST expose an effective
identity via `effective_embedder_identity()` and are bound by the
same three-state check.
- §7.3: adds `maintenance_kinds: ClassVar[frozenset[str]]` advertisement
mechanism. `run_maintenance(kind)` must raise
UnsupportedMaintenanceKindError for unadvertised kinds. Benchmark
harness reads this set rather than guessing kind names. Reserves
`analyze`/`compact`/`reindex` as well-defined names.
- §1.2: adds `update()` as optional method with a default get+merge+
upsert implementation. §2.1: `supports_update` redefined to gate
atomic single-round-trip semantics (not mere capability), since the
default impl already supports partial updates.
Operator asks explicitly NOT adopted (diplomatic shims, not contract
defects): `.to_dict()` compat on typed results, migration progress
reporting, `BaseBackend.repair()` separate from `run_maintenance`,
per-palace capability variance, identity recording on read-only ops.
Incorporates review feedback from skuznetsov (Postgres, #665) and
dekoza (Lance, #574) on issue #737:
- §1.5: split 'accepts embeddings=' (signature compliance) from
'persists embeddings as-is' (correctness). Adds
supports_embeddings_passthrough capability; the former is universal,
the latter is required to label a migration lossless.
- §1.5: model identity check becomes a three-state machine
(known_match / known_mismatch / unknown) so legacy palaces without
recorded identity don't hard-fail on upgrade.
- §1.4: makes explicit that supports_contains_fast is the ONLY
performance floor the spec promises; without it callers MUST assume
O(n). $contains is a correctness requirement, not a performance one.
- §3.3: clarifies auto-detect is an upgrade-compat path only, never
the selection mechanism for new palaces.
- §8.2: migrate CLI refuses to run against a target lacking
supports_embeddings_passthrough unless --accept-re-embed is passed;
migration record now captures lossless status and model identities.
Copilot review flagged back-references in §1.4 and §6 that still used
the pre-skuznetsov-rename names (`$contains_fast`, `sync_capable`,
`change_feed`). Updated to the `supports_*` prefix used in the §2.1
capability table.
Formalizes the BaseCollection/BaseBackend contract introduced as a seam
in #413 into an interchangeability spec that third-party backends can
build to. Driven by six in-flight backend PRs (#574, #643, #665, #697,
#700, #381) each implementing the interface differently.
Key decisions captured: entry-point distribution, typed QueryResult/
GetResult replacing Chroma dict shape, daemon-first multi-palace model
via PalaceRef, required where-clause subset (incl. $contains),
mandatory embedder injection with model-identity validation, capability
tokens, shared pytest conformance suite, and a backend-neutral
migrate/verify CLI.
After rebasing #1671 onto current develop:
- Opt test_embedding_api out of conftest's stable EF mock so the
get_embedding_function selection tests exercise the real factory.
- Move the CHANGELOG entry from released 3.7.0 Performance into
Unreleased Features (rebase context had drifted).
Address the gemini-code-assist review on #1671:
- Wrap `http.client.HTTPException` (BadStatusLine / IncompleteRead — common
with local/overloaded servers) and `ValueError` (invalid/missing URL scheme;
also subsumes `json.JSONDecodeError`) in `EmbeddingAPIError` instead of
letting them crash the caller.
- Reject a non-dict top-level JSON response before calling `.get()` on it, so a
JSON list/`null`/string yields a clear `EmbeddingAPIError` rather than an
unhandled `AttributeError`.
- Add tests for all three cases.
Add an `embedding_model: "openai-compat"` option that computes embeddings
via any OpenAI-compatible `/v1/embeddings` server (LM Studio, llama.cpp,
vLLM, Ollama's OpenAI shim, self-hosted) instead of a local ONNX model.
- New OpenAICompatEmbeddingFunction (stdlib urllib, no new dependency):
batches requests, asks for `encoding_format: "float"` and a custom
User-Agent (avoids Cloudflare 403, see #1570), validates the response
(contiguous 0..n-1 indices + well-formed vectors) before use, and
L2-normalizes for the cosine collection. Exposes `embed_query` (ChromaDB
1.5 dispatches query embedding through it, not `__call__`). `name()`
encodes the model id so switching it forces `mempalace repair
rebuild-index`. Failures raise a module-specific `EmbeddingAPIError`.
- Endpoint settings resolved by MempalaceConfig as a single source of truth:
`embedding_api_url` / `embedding_api_model` / `embedding_api_key`, each
overridable via the matching `MEMPALACE_EMBEDDING_API_*` env var.
Whitespace-only values are treated as unset; the EF cache key fingerprints
the key so a token rotation is picked up.
- The miner/MCP `Device:` header reports `openai-compat (<url>)` instead of a
misleading local accelerator label when this backend is active.
- Opt-in; default stays minilm. Mirrors the existing `openai-compat` LLM
provider naming; stays local when the endpoint is on the machine/LAN.
- Tests: tests/test_embedding_api.py (no server / no network required).
- Docs: README requirement note, module docstring, CHANGELOG.
Refs #1559.
`repair` in its default mode and `migrate` both back up the whole palace
directory with `shutil.copytree` before they overwrite it. `copytree` cannot
duplicate a Unix domain socket or a named pipe: it copies everything else and
then raises `shutil.Error`, so one such entry anywhere under the palace killed
the command at the backup step, before any rebuild ran. Re-running only
repeated it, because the backup already written held a complete
`chroma.sqlite3`, so the next run deleted it and died in the same place.
Both call sites now go through `copy_palace_dir`, which hands `copytree` an
`ignore` callback. The predicate is an allowlist, matching how the rest of the
package treats directory entries: only regular files and directories are
copied, and anything else is named and left out. Device nodes are skipped for
the opposite reason to sockets, since the copy does not fail on them but
dereferences them; a link to `/dev/zero` wrote 493 MB in 8 seconds here before
it was stopped.
A failed `stat` is never a reason to skip. It says the entry cannot be
resolved right now, not that it holds no data, and no errno separates the two:
a symlink into a volume that is not mounted fails exactly like one whose
target was deleted, and on Windows an unmapped drive letter and an unreachable
network share both arrive as `ENOENT` as well. Such an entry is left for the
copy to fail on, exactly as before this change, because the backup is the
safety net for the rebuild that overwrites the palace next. Five tests pin
that boundary, each asserting `shutil.Error` still comes out.
What was skipped is printed for the operator, and printed for a failed copy
too, since that is when knowing what the backup lacks matters most. The report
can fail on its own, because both callers pass `print`: an stdout that cannot
encode an entry's name, or a report long enough to flush partway through onto
a device that refuses the write. One such line costs neither the lines after
it nor, on the failure path, the copy's own exception, which a report that
raises would otherwise replace. On the success path it is not suppressed, so a
caller that passed something other than a working logger hears about it.
Co-Authored-By: marcoaperez <25101951+marcoaperez@users.noreply.github.com>
Restore the fresh-palace dry-run regression alongside the existing-empty
and initialized-sqlite_exact cases. Cover explicit failure for unsupported
adapter result types and daemon source-adapter dispatch. Ignore Hypothesis'
test cache so it cannot be committed.
- existing-uninitialized palace dir dry run: assert no chroma.sqlite3
- initialized sqlite_exact palace dry run: assert artifacts byte-identical
- replace mocked MineAlreadyRunning CLI test with a real held writer lease
repair_mojibake rewrote clean text. The [ÂÃ][continuation] alternative of
_HIGH_CONFIDENCE_RUN is only a two-character window, and Portuguese, Vietnamese
and Turkish end all-caps words in Ã/Â — IRMÃ, MAÇÃ, MANHÃ, BÃO, NHÃ, HÂLÂ, IMÂ.
Prose then follows with a closing quote, guillemet, ellipsis or dash, and all of
those are in the continuation class, so "«MAÇû." and "“IRMÔ" matched a shape
that is also perfectly clean text. Each hit collapsed two characters into one
and was reported as a successful repair.
Three failures came out of the same root cause:
* clean prose destroyed — "«MAÇû." became "«MAÇû.", "“IRMÔ" became "“IRMÔ";
* visible text swapped for invisible C1 controls, because  plus 0x80-0x9F
decodes to exactly that block, so "İMÂ… edildi." lost its ellipsis to U+0085;
* the max_passes loop was not idempotent: genuinely mojibaked Portuguese was
repaired correctly on pass 1 and then destroyed on pass 2.
The fix follows the precedent already set in this module, where Ä/Å were
excluded as leads because "Ų can be legitimate scientific text" — the same
argument, applied to the continuation side. A lone [ÂÃ] window ending in
typographic punctuation is no longer high-confidence, so it is repaired only
when a lowercase letter runs directly into the lead ("coûte", "Noël"), which
is where mojibake actually occurs — inside a word. Chained units ("ação") and
the 3-4 character â/ð/ï windows are unaffected. A decode product containing a
C0/C1 control is now always refused.
Corroboration is deliberately local. Inferring "this drawer is mojibake" from a
run elsewhere in the text destroys clean prose in a mixed drawer, and drawers
are mixed by construction since the miner concatenates several sources.
The trade is asymmetric and chosen on purpose: an unrepaired string is unchanged
and can be repaired later, while a wrong repair is silent and irrecoverable.
_wing_from_jsonl_cwd took the leaf segment of the transcript cwd as the wing name. When Claude Code runs in a git worktree, cwd is <project>/.claude/worktrees/<wt>, so the leaf is the worktree name and every worktree mines into its own wing_<worktree>, fragmenting a project's memory across throwaway dirs. Strip a trailing /.claude/worktrees/<wt> segment before taking the leaf so all worktrees of a project share that project's wing.
The variable decides whether a long-lived server is still running
tomorrow, and it appeared in no markdown in the repo. Add it to the
environment variable table and to the remote server operating notes,
including the two parts that are easy to get wrong: /healthz traffic
does not reset the idle timer, and the watchdog exits 0, so a
supervisor set to restart only on failure will leave the server down.
The idle watchdog terminates the process with os._exit(0). systemd's
Restart=on-failure does not restart a unit that exits 0, so a server
installed from this template stops for good after the idle timeout
rather than restarting.
Switch the unit to Restart=always and disable the watchdog in the
server env template, since a dedicated always-on server is not the
per-session case the watchdog was added for.
The section only ever documented `docker build`, so every reader
compiled the image locally even though `ghcr.io/mempalace/mempalace` is
published multi-arch. Worse, a clone builds `develop` (the default
branch), not the release — a build and a pull could hand you different
versions with no hint that they differ.
Lead with `docker pull`, and cover the things that actually cost people
an evening:
- the container only sees what you mount, so the MCP client config now
mounts a transcripts directory; without it the server starts fine and
every mine finds nothing
- paths become container paths after that, and `~` / `$HOME` are not
expanded by every MCP client
- the first embedding call downloads ~80 MB (minilm) or ~300 MB
(embeddinggemma) into /data, which reads as a hung container
- bind mounts keep host ownership and the image runs as uid 1000, so a
0700 directory fails with a bare PermissionError on Linux; Docker
Desktop's uid mapping hides this on macOS and Windows. `--user` is
called out as the wrong fix — /data is mode 700 owned by uid 1000, so
another uid cannot write the palace at all
- mining never writes to its source, so the examples mount it read-only
- the GPU image is x86_64-only; onnxruntime-gpu has no aarch64 Linux
wheels, so that build fails on Apple Silicon
Every command in the section was run as written against the published
image, and each claim checked rather than assumed: the uid and mode of
/data read out of the image, `--user` confirmed to break a *fresh*
volume, the read-only mount confirmed to still mine, and the aarch64 GPU
failure reproduced.
mempalace/_stdio.py states the rule in its own docstring -- "every console
entry point that touches stdio needs to fix this on Windows" -- and cli.py
and fact_checker.py both apply it. scripts/mempalace_repair_encoding.py,
added later, does not.
That matters more here than anywhere else: this tool exists for Windows users
whose palace carries legacy mojibake, and it prints a before/after preview for
every proposed change. Under the console codepage it was written for, the
preview cannot be encoded -- the lead bytes of the corruption it detects are
exactly the characters the codepage rejects -- so the run dies with
UnicodeEncodeError before repairing a single drawer.
Reproduced with stdout on cp936:
UnicodeEncodeError: 'gbk' codec can't encode character '\xc3'
in position 14: illegal multibyte sequence
U+00C3 is the lead character of the à family this tool repairs.
Uses replace on stdout/stderr, matching cli.py and fact_checker.py, because
the preview carries verbatim drawer text that may hold surrogate halves
round-tripped from filenames; strict would crash mid-preview.
Related to #1122 (same exception class on the main CLI's help output) but a
different entry point and a different fix -- that one can use ASCII-safe
static strings, this one prints arbitrary user content.
Protect the complete rebuild_index snapshot, rebuild/swap, and cleanup
cycle with the palace writer lease.
Without whole-operation quiescence, a concurrent writer can land after
the snapshot and be absent from the rebuilt index, recreating SQLite/HNSW
divergence immediately after repair.
Add regression canaries proving that:
- ChromaDB is never opened when the writer lease is unavailable
- the rebuild body executes while the writer lease remains held
The original sweep in this PR was scoped against develop as of April. Since
then upstream added console output carrying the same characters, so the fix
had drifted into covering roughly half of the affected surface.
Re-measured against current develop and replaced the remaining occurrences of
the same five characters this PR already targets -- U+2014 em-dash, U+2500 box
drawing, U+2192 arrow, U+25CF/U+25CB circles -- on every print(), input() and
logger call reachable from a terminal:
repair.py (10), cli.py (7), format_miner.py (7), mcp_server.py (5),
miner.py (2), onboarding.py (2), migrate.py (2), searcher.py (2),
service.py (1), embedding.py (1), dialect.py (1)
Substitutions match the ones already used in this branch: em-dash -> "--",
box drawing -> "-", arrow -> "->", filled/empty circles -> "#"/".".
Four assertions matched the previous literals and were updated alongside the
strings they cover (test_cli, test_miner x2, test_repair).
Docstrings, comments and stored content are deliberately untouched -- only
what reaches a terminal, where GBK/CP1252 consoles raise UnicodeEncodeError.
Replace em-dash in confirm_entities() input() prompt and filled/empty
circle chars in confidence bar print() with ASCII equivalents. Both
write to stdout and raise UnicodeEncodeError on Windows GBK/CP1252.
Each run_* entrypoint sets MEMPALACE_PALACE_PATH so downstream config
resolution sees that call's palace, and none of them put it back. In the
daemon that costs nothing — one call, one process. In any process making
two calls it leaks: config.py reads this variable with priority over the
config file, so the first call's palace silently becomes the second
caller's, and a tmpdir palace since removed reads back as
PalaceNotFoundError from an unrelated code path.
daemon.py already saves and restores exactly these keys around its own
dispatch. This gives run_mine, run_sync and run_diary_write the same
discipline through a small decorator.
Tests cover both directions — a caller whose variable was already set
gets it back unchanged, and a caller who had none keeps none. All four
fail with the decorators removed.
#2187 fixed the default-backend ingest failure (#2190) and covered the
numpy path: an ndarray-returning EF plus a real EmbeddingCollection.upsert.
Two branches of the fixed function are still unexercised.
(The defect was not NumPy-2.x-specific. `np.float32` has never been a
`float` subclass in any NumPy major -- verified on 1.24.4, 1.26.4 and
2.4.4 -- so `list(arr)` output was always rejected. It was latent until
0e79797 added `requires_explicit_embeddings` and routed the default
Chroma backend through `_embed_texts` for the first time.)
`_embed_texts` branches on `hasattr(v, "tolist")`. The `float(x)` arm is
the one that serves embedders returning plain sequences — custom/BYO EFs,
and rows arriving as tuples. Nothing ran it, so dropping the `float()`
call there stays green and reaches users as the same production
ValueError, just on a non-default embedder.
The `if not texts: return []` guard is likewise untested. Callers pass
empty batches (a drawer set fully filtered by dedup), and loading the EF
is the expensive part — on the ONNX default it spins up a native session.
Both tests are mutation-verified against this tree:
- replacing the fallback with `list(v)` fails
test_embed_texts_handles_plain_sequence_embedders
- deleting the early return fails
test_embed_texts_short_circuits_on_empty_input
The plain-sequence EF yields `Decimal`, so the assertion proves a real
conversion rather than values passing through unchanged, and the
empty-batch test asserts by making `get_embedding_function` raise, so it
verifies the EF is never constructed rather than only checking the
return value.
Tests only — no production code changes. They live in `test_embedding.py`
for the reason #2187 documented: conftest's autouse
`_stable_embedding_function_for_tests` replaces `_embed_texts` outright
for every module outside `_REAL_EMBEDDING_TEST_MODULES`.
Refs #2190
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The first CI run failed at step 3 with
PermissionError: [Errno 13] Permission denied: '/work/mempalace.yaml'
`mktemp -d` creates the fixture 0700 owned by the runner user. Bind
mounts carry host ownership through unchanged, and the image runs as
uid 1000, so the container could not stat inside /work. Docker
Desktop's uid mapping hides this on macOS, which is why it passed
locally and only failed on Linux.
Model an ordinary project checkout instead — 0755 dir, 0644 file — which
is the shape that makes the README's `-v /path/to/project:/work` work
against a normal repo.
The Docker workflow built both images and never started a container,
and never parsed a Compose file. A green run therefore only meant the
Dockerfile compiled. Two defects that break the very first documented
command shipped past it: `docker-compose.yml` carried a bare
`environment:` key that made Compose reject the file outright (#2188),
and `_embed_texts` handed chromadb `np.float32` scalars so `mine`
aborted on the first drawer (#2187).
Add `scripts/docker-smoke.sh`, which exercises what the README tells
users to run:
1. `compose config` on docker-compose.yml and the server compose file
2. entrypoint dispatch for both `cli ...` and bare passthrough
3. `mine` a mounted directory, asserting a drawer is filed
4. `search` from a *separate* container, asserting the stored text
comes back verbatim — this is the assertion that matters, since
storing user words exactly is the promise the palace makes
5. a real MCP stdio JSON-RPC handshake: initialize, tools/list, and a
mempalace_search call whose result must contain the drawer
It asserts on returned content, not just exit codes, and lives in a
script rather than inline YAML so it runs identically on a laptop:
`scripts/docker-smoke.sh <image>`.
The new `smoke` job builds amd64 natively with `load: true` (buildx
cannot load a multi-arch manifest) and reads the publish job's cache
while writing its own scope, so an amd64-only export never lands on top
of the multi-arch one. `build` now needs it, so a failing smoke test
blocks publication rather than being noticed afterwards.
Verified by reintroducing each defect against a real build: the compose
regression fails at step 1, the embedding regression at step 3, and the
current tree passes all five. Failure output is clipped to 500 columns
because a rejected embedding batch otherwise prints a whole 384-dim
vector on one line and buries the message.
`environment:` was present with nothing but comments beneath it, so YAML
parsed it as null and Compose refused the whole file:
services.mcp.environment must be a mapping
That is every documented Compose command — `docker compose build`,
`docker compose run --rm mcp`, `docker compose down` — failing before
anything starts, on any machine. The README points at this file.
Comment the key out along with its example entries, and say in the file
why it cannot be left bare. The examples switch to mapping syntax so
uncommenting them yields a valid block.
Verified with `docker compose config` on the shipped file (now valid)
and by running the documented flow against a build of this tree: mine a
mounted directory through `docker compose run --rm mcp cli mine`, then
read it back with `... cli search`, which returns the drawer verbatim.
`deploy/docker-compose.server.yml` was checked for the same defect and
is unaffected.
Chroma declares `requires_explicit_embeddings`, so every write on the
default backend routes through `EmbeddingCollection`. `_embed_texts`
built its rows with `list(v)`, and `v` is a float32 `np.ndarray` — that
unpacks into `np.float32` *scalars*, which chromadb's
`normalize_embeddings` rejects:
ValueError: Expected embeddings to be a list of floats or ints, a
list of lists, a numpy array, or a list of numpy arrays
`mine` aborted on the first drawer, as did every other write against a
default palace. Convert with `.tolist()` (C-speed), keeping a
`float(x)` branch for embedders that already return plain sequences.
The suite could not see this. conftest's autouse
`_stable_embedding_function_for_tests` monkeypatches
`embedding_wrapper._embed_texts` itself for every module outside
`_REAL_EMBEDDING_TEST_MODULES`, so the defective function was never
executed under test. The regression tests therefore go in
`test_embedding.py`, which is exempt from that stub: one asserts the
returned elements are builtin floats, one drives a real Chroma
collection through `EmbeddingCollection.upsert` and reads the document
back. Both fail against the previous line with the production
ValueError.
Verified end to end outside the suite: mining a project and searching
it back returns the drawer verbatim, on the host and in the container
image built from this tree.
The previous commit replaced llm_model.split('-')[1] on the diary pre-compute
line, but run_benchmark builds the same short name again 35 lines later for
the summary header. That one is dash-guarded so it does not crash, yet it
still guesses at a short name, so a single run printed the full tag on one
line and a mangled fragment on the next: "claude-sonnet-4-6" became "sonnet",
and "llama3.2-vision:11b" became "vision:11b".
No dash-split of a model tag is left anywhere under benchmarks/.
llm_model.split('-')[1] raises IndexError for any model tag that contains
no dash (e.g. an Ollama tag like 'qwen3:8b' or an OpenAI-compat route like
'spark/fast'), crashing locomo_bench.py at startup when --llm-rerank is
enabled and longmemeval_bench.py on the diary pre-compute path. Both sites
only build display strings, so print the full model tag instead of
guessing at a short name.
Hit in practice when pointing --llm-backend ollama at a LiteLLM gateway
with provider-prefixed model routes.
`mempalace_diary_write` returns an `entry_id` for every diary entry, but
for entries large enough to be chunked that id was unusable: get_drawer,
update_drawer and delete_drawer all answered "Drawer not found", and
list_drawers showed the entry as N unrelated chunk rows.
Two metadata conventions never met. The diary chunking path stamped
`parent_entry_id` on each chunk, while the logical-id read paths added in
#1782 query only `parent_drawer_id`. Both keys mean the same thing --
"physical chunk of this logical drawer" -- so chunk groups written by
diary_write were invisible to logical-id resolution. Same bug class as
#1763, which #1782 fixed for `add_drawer` drawers only.
Read paths now resolve either key via `_PARENT_ID_KEYS`:
- `_logical_chunk_group()` matches both with an `$or` (fixes get /
update / delete). All four backends support `$or`.
- `_collapse_drawer_rows()` groups on either (fixes list_drawers, which
the `$or` alone does not cover).
- `searcher._result_drawer_id()` resolves either, so a hit on a chunked
diary entry reports the id that fetches the whole entry rather than
the single chunk that matched.
New diary writes also stamp `parent_drawer_id` alongside
`parent_entry_id` so the two conventions converge going forward. Because
the read paths still accept the `parent_entry_id`-only shape, palaces
written before this fix are repaired with no data migration.
Diary chunks are written without `source_file`, so neighbor expansion
(#1580) returns early on them and is unaffected by the added key.
Also drops the comment telling callers to iterate `chunk_ids` (it
documented the bug as intended behavior) and a stale claim that search
rejoins chunks via `parent_entry_id` -- no search code read that key.
A long-lived MCP server imports mempalace and chromadb once and serves from
those in-memory modules for the life of the process, so an upgrade on disk
mid-session never reaches it and it keeps accepting writes produced by code
the user no longer has installed.
Refuse mutating tools with JSON-RPC -32005 once a watched distribution's
installed version differs from the snapshot taken at import, or once it is
gone entirely. Reads stay available, mempalace_status reports
library_versions, and MEMPALACE_MCP_ALLOW_STALE_LIBRARY=1 opts out.
Both sides of the comparison come from installed metadata rather than a live
module.__version__. A distribution whose metadata cannot be read, or whose
search root will not open, is reported and left uncompared rather than
treated as removed: importlib.metadata suppresses the failure at both of
those levels, so either one would otherwise look exactly like an uninstall
and refuse every write on a healthy install.
The same module also memoizes each search root's listing against that root's
mtime, read in seconds where this fingerprint compares nanoseconds. An
upgrade whose removal and creation both land inside one timestamp tick would
then be answered from the listing taken before it, naming a dist-info that is
already gone; its version reads as empty, the distribution is left
uncompared, and nothing moves that mtime afterwards, so the gate would stay
off for it for the rest of the process. Drop the memo before each reading.
Watch chromadb only when chromadb is the backend serving. It is a hard
dependency rather than an extra, so it is installed even for a palace kept
in Postgres, and watching it there would refuse that user's writes whenever
chromadb alone was upgraded, over a library that writes nothing they own. A
backend that cannot be resolved keeps it watched.
Skip a sys.path entry carrying an embedded NUL. os.stat and os.listdir refuse
it during argument conversion, raising ValueError rather than the OSError
those callers hold; POSIX never gets there because realpath rejects it first,
but Windows resolves it and one such entry would end the whole reading.
The gate sits ahead of the diverged-index refusal added since (-32004, which
is why this one takes -32005). That gate's remedy is `mempalace repair
rebuild-index`, which would run the installed code against a palace this
process is still writing with the superseded one, so the restart instruction
has to be the one that reaches the client; the index probe re-runs per call
and surfaces immediately after a restart. Ordering it this way also skips
that gate's segment probe on a call already refused. Both directions of the
precedence are pinned by tests.
Co-Authored-By: messelink <274674234+messelink@users.noreply.github.com>
The dependabot bump moved pyproject's ruff pin to 0.16.1, but nothing else
followed it, so the repo asked for three different versions at once:
- pyproject.toml said 0.16.1 (the bump)
- uv.lock still resolved 0.15.20 (dependabot did not update it)
- .github/workflows/ci.yml installed 0.15.14 by its own literal pin
The lint job never reads pyproject, so CI kept linting with 0.15.14 and
reported this PR green without 0.16.1 ever running. The ci.yml pin had
already drifted from pyproject before this bump, under a comment saying to
keep them identical.
- ci.yml: pin 0.16.1 to match pyproject.
- uv.lock: regenerated so the locked resolution agrees.
- test_ruff_pins_match: assert ci.yml and pyproject stay equal, so the next
bump that touches only one of them fails loudly instead of passing blind.
- extend-exclude '*.md': 0.16 began formatting Python inside markdown
fences, taking the formatter from 199 files to 295 and reflowing
hand-aligned example code in docs/rfcs/002 and three website pages.
Excluding docs keeps this a version bump rather than a silent
documentation reflow, and restores the exact file scope the project has
always formatted.
Verified with 0.16.1 actually installed: ruff check and ruff format --check
both clean over the same 199 files, full suite green (3669 passed).
Python's text layer applies universal-newline translation: CRLF becomes LF
on read, and on Windows LF expands back to CRLF on write. Every CLI path
that carried artifact and event content went through it, so a patch piped
in from a Windows agent was stored as different bytes than the ones git
produced — under a sha256 of the mangled content, which then failed to
match on the machine that applied it. For a store whose contract is
verbatim bytes addressed by digest, that is silent corruption.
It also disarmed the CRLF warning added for kind=patch: put_artifact looks
for a carriage return that the read path had already stripped, so the one
check meant to catch unappliable diffs could never fire on the platform
that produces them.
- _read_stdin_exact() / _write_stdout_exact(): go through sys.stdin.buffer
and sys.stdout.buffer, falling back to the text layer when a replaced
stream has no .buffer.
- --file / --body-file: read_bytes().decode() instead of read_text().
- artifact get --out: write_bytes() instead of write_text().
Verified through real pipes: a 63-byte CRLF patch round-trips at 63 bytes
with an unchanged sha256 via stdin, --out, and stdout, and the CRLF warning
now fires. Regression coverage in TestVerbatimNewlines, confirmed failing
against the old behavior first.
Config reads (peers.json, replica.json, server registry, token file) are
left on read_text: JSON is newline-insensitive and none of them carry user
content.
Reported by Copilot on the pull request.
The default (legacy) repair path ignored --dry-run and ran the real
rebuild: it deleted any existing <palace>.backup, copied the live palace
over it, re-filed the drawers collection through a staged temp copy, then
rebuilt FTS5 and VACUUMed. #2095 and #2133 fixed this for
--mode from-sqlite only.
The preview now returns before ChromaBackend() is constructed. Opening a
chromadb client is itself a write to chroma.sqlite3, so a preview that
reached one could not be inert; the row count comes from
sqlite_drawer_count instead, the read-only SQLite ground truth
check_extraction_safety already trusts. Staying off the chromadb layer
also keeps a dry run clear of the layer repair is separately reported to
segfault in on a large palace (#2113).
resolve_repair_preflight_errors() decides what a dry run does about the
FTS5 autoheal. The autoheal is a write, so a preview must not run it, but
skipping it routed an isolated inverted-index error into the abort banner
and exit 1 - telling the operator to run offline sqlite3 .recover on a
palace a real run heals by itself (#1596). The dry run now classifies the
errors with the same _errors_are_isolated_fts5 predicate the real path
gates on and continues; broader corruption still aborts in both modes.
It is worded as an attempt rather than a promise, because the real heal
still gives up when another process holds the mine lock, when the rebuild
raises, or when quick_check is still dirty afterwards.
The plan describes the real run in execution order. It names the
live-collection delete the rebuild performs, since "re-file via a staged
temp copy" alone reads as additive; it warns when an existing backup
would be deleted; and it reports a no-op instead of a rebuild when the
collection holds no rows. It states the #1208 truncation guard that can
abort the run, and reports that guard as disabled when
--confirm-truncation-ok is set, because check_extraction_safety returns
immediately then and the promised abort would not happen. An unreadable
count fails closed with a non-zero exit, for parity with the from-sqlite
preview.
A real ChatGPT conversations.json is a top-level array of conversation
objects. _try_chatgpt_json requires a single dict with a "mapping" key, and
no other parser in the chain claims the array either, so the export fell
through to the plain-text path: mine --mode convos reported success while
filing serialized JSON sliced at arbitrary offsets.
Unwrap the array and normalize each conversation separately, mirroring
_try_claude_ai_json_split, so a freshly downloaded export sitting next to the
previous one files only the conversations that are actually new.
The ChatGPT parser is now reached for every element of every top-level JSON
array under the mined tree, so type-check its nested shapes: ten of the
twelve malformed payloads now covered by tests raised AttributeError,
KeyError or TypeError, which would abort a whole mine run since convo_miner
catches only OSError and ValueError.
The default `subagents/` skip is directory pruning, so it cannot reach a
caller who names one transcript directly: that path feeds a single
synthetic scan entry with no directories to prune. The split is
deliberate -- `--include-subagents` governs what a directory walk sweeps
up, while naming a path stays an explicit request -- but the single-file
scan and this filter were written independently and nothing exercised
them together, so the boundary was unpinned.
The tokenizer pads every row of a sub-batch to the longest sequence in it,
and attention costs batch x heads x length^2 per layer, so arrival order
decided what an embeddinggemma run cost: one long verbatim message dragged
its whole sub-batch up to its own length.
Sort by UTF-8 byte length before the split and scatter each row back to its
input index. An input that fits one sub-batch is left alone, since every row
pads to the same width either way.
Over 43,157 sweep drawers from 160 transcripts, padded token slots drop
39.7% and the quadratic attention term 45.0%. Wall clock on 52 messages with
one model instance: 1169.70 s to 490.83 s, control baseline 1286.23 s. Row
values move by at most one float32 ULP (1.2e-07, cosine 0.99999992), which
is reduction-order rounding, not a change of meaning.
Only palaces configured for embeddinggemma are affected; the default MiniLM
embedder pads to a fixed width.
Narrows the 3.7.0 slice to the feature users actually get: RFC 003 agent
coordination (events, artifacts, patch handoffs) plus the multi-master
logstream sync that makes it work across machines.
Removed, deferred with the rest of RFC 004:
- replica_sync.py / vector_cache.py and the `mempalace replica` CLI
- the /snapshot/* hub endpoints they backed
- website/concepts/replicated-palace.md
Memory read replicas were a leaf on the dependency graph (nothing in the
logstream path imports them), and shipping them half-done meant documenting
a mesh whose memory does not actually converge. Dropping them lets the docs
say one true thing instead of two hedged ones: coordination syncs, memory
stays local, point every agent at one hub if you want shared recall.
Fixes a bug found by running it: _start_peer_sync_thread() read peers.json
once at startup and returned early when absent, so a hub started before
peers.json was written never synced — silently, forever. That is the order
the guide tells users to follow. Membership is now re-read every round.
Verified on two live hubs: delegation loop end-to-end, verbatim patch
round-trip by sha256, bidirectional sync, CLI sync alongside a live hub,
and automatic convergence 15s after writing peers.json with no restart.
The website pages arrived from the full RFC 004 branch and documented the
memory op-log, `mempalace oplog`, and automatic bidirectional memory
convergence. None of that is in this slice: the CLI has logstream, artifact,
and replica, and memory moves by one-way `replica pull`.
- cli.md: drop the `mempalace oplog` section (the command does not exist);
note that full convergence is staged.
- replicated-palace.md: split the sync table into shipping (coordination,
multi-master) vs next (memory ops, bidirectional); say plainly that two
machines capturing independently do not merge yet.
- shared-brain.md: mark step 3 a pull you re-run, not a converging loop.
- RFC 004: record which steps landed.
The rebase resolution had taken the branch version of both files wholesale,
dropping develop's backend table, version badge, mempalace_kg_supersede
section, and the checkpoint added_by row. Rebuilt as develop's content plus
only the RFC 003 logstream additions; tool count is 43 (36 + 7 logstream).
Identity becomes the triple host:agent:project — the granularity at which an
actor actually shares filesystem, local config, and knowledge. Two sessions in
the same project on the same box are the SAME actor with the SAME identity;
session/PID is event metadata, never identity. Fixes the live collision where
two windows-claude sessions interleaved inboxes, claims, and diary under one
flat name.
Two decisions resolved (both live in the backend/hub):
- Matcher: NO change. Route the tuple as an opaque string on RFC 003's exact +
'*' broadcast; colons already pass _sanitize_routing. Hierarchical/glob
routing is deferred — it's a real index/correctness surface and no concrete
need has landed. This ships as a convention + one renderer change, zero risk
to the routing hot path.
- Order: host:agent:project (host-first), matching how mac-*/windows-*/blade-*
names already read; migration is a pure suffix append, not a rename.
Claim safety under a shared identity needs no lease server: natural mutex
(don't claim what your identity already claimed) plus RFC 004 R3's earliest-HLC
tiebreak for the residual race. The durable fix is one place — the renderer of
the mempalace-shared-brain block emits the tuple, so every box re-derives on
next sync instead of being hand-edited. Deferred behind the RFC 004 write-flip;
touches no storage or merge semantics.
(cherry picked from commit b043c29fd6a65ea32f92252b8afab1b62bc95bf5)
Event bodies are stored verbatim and fleet status updates run to several
KB, so listing many events full-body is a large payload — agents hit it
catching up on the stream. New preview=true truncates each body to a
200-char excerpt (marking body_truncated + body_length) while keeping
every routing/metadata field, so an agent can scan and then re-fetch
only the events it cares about in full via a targeted since_event_id.
Default (full verbatim) is unchanged; the verbatim-on-demand guarantee
is intact.
(cherry picked from commit dcf43984b84baea49838b2c665d62e572d259615)
The estate is observability — logstream + sync state + peers.json, no
FTS5 dependency; the profile's drawer count already degrades
gracefully. A damaged palace is exactly when mesh visibility matters
most. Caught live on the third replica: its FTS5 index went malformed
and the integrity gate refused the estate tool along with the palace
tools.
(cherry picked from commit c694a8401cb2efd0ca482344503956e4b4c5bad3)
Every hub now advertises a self-derived node profile: roles
(replica/agents/compute), resolved accelerator + embedder, live drawer
count, hardware string, advertised_at. Every field is derivation from
what the daemon observes about itself — never configuration, so mesh
dashboards render what each machine reported, not what a UI guessed.
Profiles ride the existing sync surfaces: /sync/version_vector gains
additive profile/profiles fields, each round's stats carry them, and
carriers relay profiles for origins they only know transitively —
/sync/peers now serves per-peer profiles plus origin_profiles keyed by
replica id. Unreachable peers keep their last advertised profile.
New MCP tool mempalace_mesh_peers returns exactly the /sync/peers
payload from the same producer function, so the desktop app consumes
the estate through the existing bridge — one integration, one token
path, and the tool and endpoint can never drift. Tokens are never in
the payload, asserted in tests.
(cherry picked from commit f31d0904167f5463285ae78de335bcea304ed71a)
Called out explicitly after proving it live: a pre-mesh palace joining
the replicated palace keeps everything (nothing re-mined, nothing lost),
serves its history to peers via one embed-cache + pull, and its drawers
are first-class mesh citizens. This is the path most existing users will
take, so the guarantee belongs in the guide, not in an RFC appendix.
(cherry picked from commit 1023368eed16f16e29544ba44f562872d26df101)
The docs ended at the single-hub logstream era; the mesh has since gone
from RFC to production. This brings the site current:
- NEW concepts/replicated-palace.md: one logical palace across N
machines — the availability invariant, the three layers, ops/HLC/
version vectors, gossip convergence, snapshot bootstrap + peers.json,
distributed embedding (sync the facts, derive the senses), the
/sync/peers estate surface, and the trust story (bearer tokens today,
mesh identity next). Status-honest: the memory op-log is documented
AS the dual-write shadow it currently is, with oplog verify named as
the cutover gate.
- reference/cli.md: mempalace replica (pull/embed-cache), mempalace
oplog (status/sync/fold/verify), logstream sync — with the
operational rules (hub-stopped pulls, quiescent origins,
MEMPALACE_SYNC_HTTP_TIMEOUT for big bootstraps).
- guide/shared-brain.md: new 'From hub to mesh' section — the
three-step join recipe and why the mesh removes the hub as a single
point of failure; See-also links updated.
- Sidebar: Replicated Palace under Concepts.
Example hostnames throughout; vitepress build clean.
(cherry picked from commit 63c2c3c20fcd350b8aca6b3249dd1a08ba5ecc22)
Layer 2 promised it would see peers only through a transport interface;
until now that seam existed in the RFC but not in the code. This lands
it: mempalace/transport.py defines the full contract (self_id / peers /
request now; open_stream / on_presence_change / on_inbound documented
and explicitly deferred to the MeshGuard binding with loud
NotImplementedError), and today's shipping behavior — tailnet HTTPS with
peers.json bearer tokens — becomes its first implementation,
HttpsBearerTransport, moved wholesale below the seam (load_peers,
PEERS_FILENAME, the wire primitive, the timeout knob).
get_transport(palace_path) is the single swap point: MEMPALACE_TRANSPORT
defaults to https; 'meshguard' is reserved and fails loudly until the
binding ships — a user who asked for mesh-identity auth must never
silently run on the bearer-token model it replaces. sync_all now
iterates the transport's membership snapshot (injectable for tests and
for the binding).
Compat surface preserved exactly: logsync re-exports load_peers /
PEERS_FILENAME / _peer_get, and SyncPeerError IS TransportError (same
class), so every existing importer and except-clause keeps working.
Context: three-replica mesh went live tonight on bearer tokens; joining
the third node took ~10 manual credential operations for 2 of 3 edges —
the N-squared out-of-band token relay this seam exists to retire.
MeshGuard trust-path sweep (RFC 004 pre-integration gate a) running in
parallel; the FFI binding (gate b) builds on this seam next.
tests: seam contract, https impl wiring, factory swap point incl.
reserved-name loudness, compat surface. Full suite: 3496 passed.
(cherry picked from commit 45b0aa335375af2e7c93eb9b4b805bcc6877b219)
PalaceMind's mesh view had no data source: the hub exposed version
vectors and op ranges but nothing that says which replicas exist, what
they're called, whether they were reachable last round, or how far
behind they are. A third replica joining the mesh was invisible to the
UI even though every version vector already carried its origin.
GET /sync/peers (bearer, lock-free like the rest of /sync) now serves
the RFC 004 estate data from this replica's point of view:
- self: replica_id, hostname, local version vector
- peers: configured peers.json entries (name + url — the token is
NEVER included) merged with last-round sync state recorded by the
anti-entropy loop: peer replica_id, reachable, last_success_at /
last_error(_at), last pulled counts, and the peer's remote version
vector (drift = remote vector vs self vector, computed by consumers)
- unnamed_origins: origins present in the local log but not configured
as peers — replicas known only transitively through a carrier
- sync_interval_s so consumers can pace their polling
sync_with_peer now returns remote_version_vector; the sync loop folds
each round's outcome into _PEER_SYNC_STATE (whole-entry replacement; an
error round preserves the last known good state so the UI can show
"unreachable since X, last seen at vector V").
tests: estate payload over real HTTP incl. token-privacy assertion and
a transitively-known origin; error-round state preservation.
Full suite: 3484 passed.
(cherry picked from commit ffb8f81ef41b2af32747f58f42b4f354b56d5f0b)
The hardcoded 30s per-request timeout failed in production during the
third-replica bootstrap: a deep-offset snapshot page (Chroma get(offset=N)
scans N rows) behind the mac's tailscale-serve TLS proxy exceeded 30s at
offset 27500 of a 31k-drawer origin. The pull is resumable so nothing was
lost, but bootstrap pulls need a higher ceiling. Default unchanged at 30s;
malformed values degrade to the default. Longer-term fix (tracked): id-
cursor snapshot pagination so page cost stops growing with offset.
(cherry picked from commit 30715f438eb0427cb6783f2a55e0330f20652b8b)
RFC 004 distributed derivation, first brick (asked by Igor: 'could the
mesh become the user's personal GPU cluster?'). A vector is a pure
function of (content, embedder identity); with a pinned fleet identity it
becomes a portable fact computable on whichever machine has the
horsepower.
vector_cache.py: sidecar vector_cache.sqlite3 keyed (drawer_id,
embedder), packed float32; build_cache() bulk-embeds local documents —
read-only on the content store (quiescence-safe), idempotent and
resumable. CLI: mempalace replica embed-cache [--model] [--batch]
[--all].
Serving: /snapshot/drawers?vectors=<model> joins the cache and ships
base64 float32 per item; manifest advertises per-model cache counts.
Fold: mempalace replica pull --with-vectors verifies the served identity
against the LOCAL embedder identity (loud refusal — a foreign-model
vector poisons the index silently), upserts with explicit embeddings,
falls back to local embedding per missing item; stats report
vectors_used vs locally_embedded.
Six new tests incl. the strict one: a fully-vectored fold must invoke
zero local embedding. Full suite 3450 green. Immediate use: the 156k
Windows merge — origin embeds on the 9950X3D under 'minilm', the mac
folds insert-only, hub window drops from ~an hour to minutes.
(cherry picked from commit a134f50cef950affb4888e9fed36489e433ec824)
--no-kg was only honored with an explicit --peer; pull_from_peers now
accepts and forwards it. Caught during the first production reverse pull
(Windows pilot batch, 2,235 drawers folded into the live mac palace in
31.7s including local embedding).
(cherry picked from commit a609e32fc60ca6f63101a2782c91124c1df61d0d)
Raised by windows-claude before its conversation-mining run (option C on
correlation windows_conversation_mining_20260702): with snapshots serving
the full collection, a reverse pull would re-import the puller's own
drawers re-stamped as peer-origin — an echo loop corrupting provenance and
delete-reconciliation scoping. Snapshots now serve only what a replica
AUTHORS (drawers without a replica_origin stamp), making each machine the
origin of its own history; pull pairs converge with no echo. Pages expose
next_offset (raw scan cursor) since filtered pages can be sparse; the pull
engine follows it.
Also adds pull_kg=False / CLI --no-kg: KG rows carry no origin stamp yet,
so a reverse pull could overwrite newer local KG state with the peer's
stale replicated copy — disabled for reverse pulls until step 2a gives KG
facts op provenance. New test: two-origin bidirectional pull, no echo,
clean provenance both sides. Full suite 3444 green.
(cherry picked from commit cafe7fba34868bd86280f8d0c2efbc18a3f50ae0)
Revisions kept forever (no GC path; search surfaces heads only); v4
migration staged on a copy before live-with-backup; step 2 ships as 2a
(drawers+KG ops) then 2b (registry/hallways); mac origin dual-writes
during a shadow period before cutover; remote write-flip (step 3) gated
on the local-capture promotion validating end-to-end.
(cherry picked from commit 34a264e12bdc3ba6f40877972e9435e3f912ee6d)
Driven by a real customer: the Windows machine holds un-mined projects and
conversations. Capture must happen where the data lives (mining reads local
disk), so step-1 replicas may mine locally now — unstamped drawers are
untouchable by reconciliation — and step 2 commits to a one-time promotion
pass turning them into drawer.add ops under the replica's identity.
Content-pure v4 ids double as the cross-machine dedup: identical content
mined on two origins collapses to one drawer at merge.
(cherry picked from commit d18b2a03519a1cfbe9e10bbf12c902c307195c15)
Recall becomes local on every machine. The hub serves the palace's FACTS
over GET /snapshot/manifest|drawers|ids|kg (bearer, paged): verbatim drawer
documents + metadata and raw KG rows — never the vector index, which each
replica derives with its own embedder (RFC 004 layer 3: sync the facts,
derive the senses).
replica_sync.py folds pulls idempotently: drawer upserts by id with ONE
additive provenance key (replica_origin = origin replica id), which scopes
delete reconciliation strictly to copies from that origin — locally-authored
drawers are untouchable. KG rows fold INSERT OR REPLACE by id, so
invalidations converge on re-pull; replication never deletes KG rows.
KnowledgeGraph gains dump_rows/apply_row for rowid-paged replication.
CLI: mempalace replica pull [--peer URL --token T] [--no-reconcile]
[--json]; defaults to peers.json. Step-1 boundary is explicit: writes still
belong to the origin (multi-writer memory is step 3); freshness is re-pull
until the step-2 op-log gives precise tails.
10 new tests over the production HTTP server with the sqlite_exact backend:
endpoint pagination, full fold + local search of pulled facts, idempotent
re-pull, reconcile that deletes origin copies but spares local drawers, KG
invalidation convergence, CLI. Full suite 3443 green.
(cherry picked from commit ebd772b00f73c35e571df2c2caebc6177853bb4a)
The pilot of the replicated palace: per-origin append-only op sync between
peer replicas, pull-based anti-entropy, no coordinator, no new deps.
Core: replica identity minted per palace (replica.json, upgradeable to the
mesh identity later via alias); hybrid logical clock (fixed-width sortable
stamps, restart-seeded from the log, absorbs remote instants); events gain
origin_replica/origin_seq/hlc — additive to the viewer compat surface, with
seq remaining the LOCAL arrival cursor per the RFC. Pre-replication logs
migrate in place: existing rows backfill as authored by this replica with
their original order preserved.
Sync: version vectors diff into per-origin pull ranges; artifacts fold
before the events that reference them (never a dangling id) with sha256
verified on receive; every apply is idempotent, so crash mid-round just
re-pulls the tail. Hub serves GET /sync/version_vector|ops|artifact
(bearer-authenticated, lock-free) and runs an optional background
anti-entropy loop when peers.json exists (MEMPALACE_SYNC_INTERVAL, default
15s). CLI: mempalace logstream sync [--peer URL --token T] [--json].
22 new tests: HLC ordering/regression/observe, identity mint, migration
backfill, idempotent apply + hash rejection + verbatim copy, bidirectional
two-replica convergence, partition with duplicate claims resolving
deterministically by HLC (R3), endpoints + CLI over real HTTP. Full suite
3433 green.
(cherry picked from commit 340b529e7c6a087adab190023e72230c8311f878)
Transport/lifecycle sections (§5), transport-side security (§10.1, §10.3),
and the PalaceMind appendix authored by windows-claude and delivered as a
patch artifact over the logstream (correlation
rfc004_replicated_palace_position, artifact art_20260702T081251_76aef8fb111a,
sha256 verified, applied at ab6f362). §10.2 at-rest encryption drafted by
mac-claude on apply: per-replica data keys that never traverse the mesh,
two-tier design (FDE baseline + SQLCipher-style page encryption over the
canonical stores, OS-keystore wrapped), replica-local rotation, R7
composition with the revocation ceremony.
The seam contract (§5.1) verifies against the Layer 2 design: request/stream
map to anti-entropy pull + SSE push-notify, and ReplicaId doubles as
origin_replica so provenance and authentication are one fact. RFC 004 is now
fully drafted and awaiting Igor's review.
(cherry picked from commit 9f3333998673fb196664325396c104036f771d1f)
One logical palace per human, fully replicated per machine: agents always
local, services converge via provenance-stamped ops over an encrypted
mesh. Judged against offline as the default posture (R0-R8).
Storage sections drafted (mac-claude ownership per the logstream debate on
correlation rfc004_replicated_palace_position): canonical op-log with HLC +
per-origin version vectors, complete merge-semantics table, v4 content-pure
id migration (verified mutable-state inventory with file refs), backends
demoted to fold-and-index consumers, provenance/source-bound maintenance,
4-step sequencing starting with logstream multi-master. Transport (§5),
transport-side security (§10), and the PalaceMind appendix are reserved
stubs for windows-claude, delivered as patch artifacts to keep a single
committer per repo.
(cherry picked from commit ab6f362af7a2ce947c2c37c11910fa8aeb6b983d)
GET /logstream/stream on the hub HTTP transport: bearer-authenticated
Server-Sent Events with the event_list filter set, since_event_id /
Last-Event-ID replay-then-tail resume (live tail only without a cursor,
via new Logstream.latest_event_id), ~15s heartbeat comments, and a
bounded client pool (MEMPALACE_SSE_MAX_CLIENTS, default 8; 503 +
Retry-After on cap). Frames carry the exact event_list JSON envelope —
the compat surface promised to the PalaceMind desktop viewer.
Also fixes a latent hub-starvation hazard the first SSE consumer would
have tripped: do_POST serialized every request under the global HTTP
lock, so one agent's five-minute event_wait long-poll blocked the whole
fleet. Logstream tools (own WAL db, no Chroma/KG state) now dispatch
lock-free via _http_dispatch; everything else keeps the global lock.
Requested by claude-fable-5-windows for the PalaceMind live viewer
(correlation palacemind_desktop_logstream_viewer). Full suite: 3411
passed.
(cherry picked from commit a7c720507974acc2f02957ac24b7045503ad74da)
Patch authored by windows-codex via logstream (task_win_hubforward_fix,
artifact art_20260702T044935_18f200c38ece, sha verified). Root causes
were test portability assumptions, not product bugs: the 0o600 mode
assertion is POSIX-only (Windows chmod reports 0o666 — now guarded), and
the _disown_record helper stamped PID 1 as a 'live foreign process',
which is dead on Windows, so read_live_serverinfo correctly rejected the
record and the proxy fell back local. Now uses the live parent PID with
liveness asserts. Verified 30/30 on Windows 11 (by author) and 30/30 on
macOS (this apply).
(cherry picked from commit 4006113dd6aa89448d898146b21ba01bbf367f80)
From windows-codex's full-suite verification reply over the logstream:
debug hub connections with curl before blaming agent config, client tool
caches masquerade as hub problems, the event body is the work order, and
cross-platform worker gotchas (CRLF, quoting, platform-guarded tests).
(cherry picked from commit c843d396cc915aa90f5392fd08cbfa1b91483cac)
Written by a multi-agent council (three drafts from tutorial/architecture/
operations angles, three judges on accuracy/fit/usefulness, one synthesis;
tutorial angle won) and finished with lived input from the fleet itself:
Hermes contributed the desktop-assistant role guidance and Windows Codex
the inbox/watcher etiquette, both delivered over the logstream this page
documents. Covers hub topology, local stdio auto-proxy, remote access,
agent identity, the canonical system-prompt snippet, a full worked
delegation loop with hash verification, hard rules, fleet roles, and
operations (statusz, read-only observers, upgrades via hub restart).
(cherry picked from commit 4e914729ff9eb3bc8dacb81de55de2c5a8982205)
Reported by windows-codex through the logstream itself: wait rejected
'limit' with -32602 while list accepts it, though the filter sets are
otherwise identical. The core already threaded limit through; expose it
on the MCP schema and the CLI (wait --limit).
(cherry picked from commit bc18ccaac6ec709dbb0821f65c3a8e39f68733d1)
Found in the first RFC 003 dogfood: windows-codex stored a patch without
its final trailing newline and git apply rejected the diff as corrupt on
the receiving side. Content stays verbatim — put_artifact/submit_patch
now return an advisory 'warnings' list (missing trailing newline, CRLF
line endings) so the producer can fix the handoff while it still owns
the diff. CLI 'artifact put' mirrors warnings to stderr.
(cherry picked from commit 7c0c9e1362cf20fabc72d61669fe2ea6125f4e3a)
Patch authored by windows-codex and handed off through the coordination
layer itself: correlation_id task_logstream_e2e_dogfood, patch artifact
art_20260702T041319_cdf92e302efe (sha256 a17012d6...9f64c4, verified),
test log artifact art_20260702T041254_c46525e2d550 (72 passed on
Windows 11, Python 3.12.11). Applied on mac, re-verified (72 passed),
acked applied. Trailing-newline nit in the submitted diff repaired on
apply (producer stripped the final newline when storing the artifact).
(cherry picked from commit 38f3d35804155840cd5512411e9823fb38dc7e95)
- website/concepts/agent-logstream.md: event/artifact model, delegation
loop, coordination-vs-memory table; added to the sidebar.
- website/reference/mcp-tools.md: schemas for the seven logstream tools;
tool count 35 -> 42 (README updated to match).
- website/reference/cli.md: mempalace logstream / mempalace artifact.
- website/guide/remote-server.md: hub doubles as agent coordination bus.
- integrations/shared/coordination-protocol.md: canonical shared-brain
protocol (identity, delegation loop, hard rules) with a copy-paste
system-prompt snippet; linked from recall-protocol.md and the
mempalace-recall skill so recall and coordination never blur.
(cherry picked from commit 1ff312511aef39abbe4638bdc23301266f87dbe6)
mempalace logstream append/list/wait/ack and mempalace artifact put/get.
Operates directly on the palace logstream.sqlite3 (WAL; safe alongside a
running hub — no Chroma state involved). --json everywhere for scripting;
wait exits 2 on timeout; artifact get prints exact content on stdout so
'mempalace artifact get ID | git apply' works; put reads stdin by default.
(cherry picked from commit 54ca9600e57770fa1e9b46565816fa6ceed8221d)
Durable append-only event + artifact layer in logstream.sqlite3 inside
the active palace dir, so agents can delegate work, wait for replies,
and exchange exact patches through the shared hub.
Phase 1 (mempalace/logstream.py): events, artifacts, event_artifacts
per the RFC schema; WAL + per-instance lock, no Chroma dependency;
append/list/wait/ack/put_artifact/get_artifact/submit_patch; explicit
size limits (body 256 KiB, artifact 4 MiB, metadata 64 KiB); rowid-backed
exclusive since_event_id cursor exposed as seq; inclusive
since_created_at; to_agent filters match '*' broadcasts; artifact_ids
validated in-transaction so listed events never dangle.
Phase 2 (mcp_server.py): seven MCP tools. Mutating four join
_MUTATING_TOOLS (hidden + refused in --read-only). All seven are exempt
from the Chroma SQLite integrity gate; the mutating four are exempt from
the peer-writer lease via _PEER_WRITER_EXEMPT_TOOLS (logstream owns its
own WAL db, and a running mine must not block coordination).
mempalace_reconnect drains cached Logstream handles like the KG cache.
(cherry picked from commit cdf72a994133242da862b11b21033a4546c5e146)
Machine-readable JSON server state on the MCP HTTP transport: version,
uptime, request counters by status, SQLite integrity, writer mode, and
recently observed clients (peer, user-agent, last path). Follows the
bearer-token policy since it exposes operational metadata; /healthz
remains the unauthenticated liveness probe.
(cherry picked from commit 0fb07f2645c0c853337419175f499daab8063888)
The documented "terminate TLS at a proxy" pattern was unusable with a
loopback bind: proxies like `tailscale serve` and nginx preserve the
public name in the Host header, and the loopback bind's DNS-rebinding
pin rejects anything but loopback literals — so the safest deployment
(hub on 127.0.0.1, tailnet-only HTTPS in front, nothing on the LAN)
got 403s on every request.
MEMPALACE_MCP_EXTRA_ALLOWED_HOSTS (comma-separated host or host:port,
matched exactly after lowercasing; bare hostnames also match with the
bound port) extends the pin for exactly that operator. The default pin
is unchanged and rebinding protection stays on.
Verified live behind `tailscale serve --bg --https=443`: tailnet HTTPS
with bearer token 200, no token 401, forged Host 403, LAN interface
connection-refused.
Documented the tailnet recipe in the remote-server guide.
(cherry picked from commit 2668053ee50b8cd70044d357501dafaa9a9671cc)
With a hub owning the palace, a stdio mempalace-mcp spawned by an agent
harness could only ever be a read-only peer (writer lease #1818) — and
harnesses that hardcode stdio in plugin manifests (Claude Code plugins,
desktop apps) cannot be pointed at HTTP per-install. Racing stdio
writers are also exactly what corrupted a real palace's FTS5 index
(repaired 2026-07-01 via the isolated in-place rebuild).
Now the stdio loop checks the server registry per request: when a live
hub serves this palace, the request is forwarded verbatim over HTTP
with the local bearer token; the response streams back over stdio. The
stdio process holds no Chroma handles, so any number of harnesses can
spawn it against a shared palace safely.
Semantics:
- Per-request discovery: a hub that starts, moves ports, or dies
mid-session is picked up without restarting the stdio process.
- No hub → local handling, exactly as before (and the resilience
fallback if the hub is briefly down).
- Unreachable hub: read requests fall back locally; mutating tool
calls are NOT replayed (the hub may still be executing them) and
surface as a JSON-RPC error with a hint.
- The hub never proxies to itself (pid guard); MEMPALACE_HUB_FORWARD=0
disables proxying, same kill switch as the CLI mine forwarder.
Verified live: `mempalace-mcp` under a running launchd hub logs "Live
palace hub detected ... proxying stdio requests" and serves initialize
+ search through the hub against the real palace.
3318 tests pass (7 new proxy tests), ruff clean.
(cherry picked from commit 83cd097595b205f148b474f78fc511fbede5f9b8)
A long-lived HTTP hub (mempalace serve) holds the MCP writer lease for
its whole lifetime (#1818), so every short-lived writer — including the
save hooks, which spawn `mempalace mine` — was refused with
LockHeldByOtherProcess while a hub was up. On a hub machine that meant
automatic transcript capture silently stopped: the exact forgetting
MemPalace exists to prevent.
Fix: the hub is discoverable, and mines are handed to it.
- mempalace/server_registry.py (new): the HTTP transport records
{pid, host, port, scheme, read_only} as serverinfo.json next to the
per-palace bearer token under ~/.mempalace/server/<key>/; records are
trusted only while the pid is alive, wildcard binds are dialed via
loopback, and cleanup is pid-guarded so an old hub's atexit cannot
delete a newer hub's record. cli._server_token_path delegates here so
token and serverinfo share one directory convention.
- mcp_server._serve_http: writes the record after bind, clears it on
shutdown (finally + atexit).
- cli.cmd_mine: when a live non-read-only hub serves the target palace,
forward the mine as an HTTP MCP mempalace_mine call (bearer token from
the 0600 token file when present). Only hub-expressible mines forward;
flags the tool has no parameters for (--kg-extract, --no-gitignore,
--include-ignored, --max-chunks-per-file, --redetect-origin, explicit
--backend) keep the direct path. Fallback to direct happens only
before the hub accepts the job — after that, failures exit non-zero
instead of re-mining (no double-ingest). MEMPALACE_HUB_FORWARD=0
disables forwarding.
Verified live: with a serve hub holding the writer lease, the previously
refused `mempalace mine --mode convos` now prints "forwarding mine to
palace hub http://127.0.0.1:8972" and files drawers through the hub.
3311 tests pass (23 new in tests/test_hub_forward.py), ruff clean.
(cherry picked from commit 010b1927a074c3896fcca54d60eab9cac5e66431)
test_hnsw_write_defaults_bound_index_rewrites divided a hypothetical
corpus size by the configured sync_threshold and read the quotient as a
count of full on-disk index rewrites. It measured no writes, no
compactions, no persistence calls and no rewritten bytes, so it asserted
arithmetic on a constant rather than a property of the implementation.
The metadata tests already cover the defaults, and a write-amplification
regression wants I/O measurement under the benchmark suite.
Two neighbouring docstrings carried the same reasoning in prose and now
state what holds: the values are chromadb's own documented defaults, so a
collection this backend creates indexes on the same terms as one chromadb
creates itself, and sync_threshold=2 reads as a value chromadb's own
parameter validation rejects.
The threshold relationship now reads batch_size <= sync_threshold, which
is what chromadb permits.
_hnsw_creation_metadata(options) reads hnsw_space directly, so
get_collection's local copy of that resolution had no reader; removing it
leaves the option's path to a created collection unchanged.
tests/test_backends.py: 91 passed. Backend/palace selection: 593 passed,
16 skipped.
The #1222 capacity probe guards reads only: search and check_duplicate fall
back to BM25 when the flushed HNSW segment lags sqlite. Writes went straight
to chromadb — and an upsert into that index does not fail, it blocks inside
chromadb's Rust upsert with no timeout of its own, for the life of the
process, while this server holds the dispatch lock, the palace mine lock and
the writer lease. Peer sessions drop to read-only behind a server whose
handshake still answers, and nothing on the server side says why: the only
trace of a 25-minute outage was the client's own "tool still running" ticks,
and mempalace_status could not answer because the stuck call held the
dispatch lock. This is the #1845 / #1908 symptom.
Measured on a 3.7.0 palace whose flushed segment held 803 of 820 embeddings
(py-spy: MainThread parked in chromadb/api/rust.py:_upsert, 64 idle
tokio-rt-workers, no fcntl lock on chroma.sqlite3 — an internal wait, not
lock contention): on copies of that palace, three of five freshly generated
vectors blocked forever, the other two committed in 0.03s, and each vector
reproduced its own verdict on retry. A write's fate depended on where its
embedding landed in the damaged graph. After
`mempalace repair rebuild-index`, all five committed in ~0.2s.
Two guardrails, both Tier 1 of #1963:
- _mcp_diverged_index_refusal: dispatch refuses the vector-write tools while
the probe reports divergence, and puts the rebuild command in the error
payload. Narrower than _MUTATING_TOOLS on purpose — the knowledge-graph
and tunnel/hallway tools keep their own state and never reach the segment,
so they must not even pay for the probe. Refusing is also the honest answer
for the write that does *not* hang: chromadb acknowledges it into sqlite
and the metadata segment, then leaves it out of HNSW, so the drawer reports
success while being invisible to vector search. In the same incident 16
such drawers came out of one checkpoint that returned "completed
successfully in 2s". The probe is pure sqlite + pickle, so the gate costs
no chromadb interaction on the path it protects, and it re-runs per call so
a repair in another process un-gates this server without a restart.
- a write-stall watchdog: MEMPALACE_MCP_WRITE_STALL_WARN_SECS (default 60,
0 disables) warns on stderr — where the MCP host records it — from a
thread the stuck call is not blocking. MEMPALACE_MCP_WRITE_STALL_EXIT_SECS
(default 0, opt-in) lets an operator turn the wedge into a restartable
failure: a server stuck inside chromadb never recovers, and exiting is what
releases the palace locks its peers are queued behind. Exit code 75
(EX_TEMPFAIL) so it does not read as an orderly shutdown.
Refs #1845, #1908, #1963
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Case-insensitive directory match (d.lower() == 'subagents') so
the filter still kicks in if Claude Code or a plugin ever emits
'Subagents/' on case-preserving filesystems (Windows, macOS APFS).
- Drop defensive getattr in cmd_mine: argparse always defines the
attribute since --include-subagents is unconditionally registered.
Direct args.include_subagents access matches every neighbouring
field and fails loudly if the registration is ever removed.
- Soften CLI help and docstring: drop the 80:1 ratio (reporter-
specific) and the in-code (#1217) reference. Add explicit
default=False on the argparse flag for symmetry with --extract.
- Add 2 negative tests: 'mysubagents'/'subagentsbackup' must still
be mined (regression guard against substring-match), and
'Subagents/' must be skipped (case-insensitive coverage).
scan_convos() now prunes any directory named 'subagents' during os.walk.
Claude Code records Explore/Plan/Grep subagent transcripts in
<session-uuid>/subagents/agent-*.jsonl and on a typical workspace these
outweigh main session files ~80:1, dominating mining time and producing
near-zero additional signal (the parent session already summarizes them).
Adds a --include-subagents opt-in flag for users who want full history.
The shared SKIP_DIRS set in palace.py is left untouched, so project
mining (miner.scan_project) still descends into legitimate user-created
subagents/ directories in code projects.
from-sqlite ignored --dry-run and performed the real archive+rebuild
(#2095, #2133). Preview after source validation, skip the destructive
confirm, never take the mine-lock or rename the palace, and fail closed
when SQLite row counts are unreadable instead of inventing zeros.
Address actionable #2129 bot feedback and the Windows closet KeyError:
- Reopen immutable sqlite_exact readers only when both WAL sidecars exist
(partial pair keeps the clean snapshot instead of failing the reconnect).
- Retry get_collection without options when plugin backends reject the kwarg.
- Stamp multi-conversation content_hash only on chunk 0 to avoid O(N²) meta.
- Always include results: [] on search error envelopes so callers never KeyError.
- Clearer hybrid-search assertions in the closet isolation test.
Bump package, plugins, lock, and README badge to 3.7.0. Promote the
post-3.6.0 integrity spine into CHANGELOG: single-writer ownership,
HNSW write defaults and preflight, safer repair/re-mine, MCP/daemon
lifecycle hardening, entity ReDoS guard, and hook write-routing.
Also ruff-format two test files that drifted during conflict merges.
Local validation: ruff check/format clean; 3497 passed, 31 skipped.
Address the last two Codex review comments on #2079:
- Reopen sqlite_exact immutable read-only connections when WAL/SHM
sidecars appear after a clean-database open so recall sees a later
writer's commits instead of a frozen pre-writer snapshot.
- Clear per-palace embedder-identity validation when MCP discards
storage handles during ownership promotion, so an empty read-only
open cannot skip recording the active model on the first write.
rebuild_from_sqlite renamed the existing palace aside
(palace.pre-rebuild-<ts>) and only hit the single-writer
mine_palace_lock later, when the first chromadb upsert ran. A palace
held by a live MCP server / daemon was therefore stranded: archived,
no rebuilt replacement, a partial dest left behind.
Acquire mine_palace_lock(dest_palace) BEFORE the archive/rename so
contention raises MineAlreadyRunning while the palace is untouched
(no archive, no partial). The lock wraps the whole rebuild as one
writer; the inner per-batch acquires pass through re-entrantly.
The body is split into _rebuild_from_sqlite_locked so the lock wraps
every destructive step.
Behaviour: held lock during repair from-sqlite — before: palace
archived + partial dest stranded; after: clean MineAlreadyRunning,
palace untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`_HNSW_BLOAT_GUARD` pinned `hnsw:batch_size=2` / `hnsw:sync_threshold=2` on every
collection this backend creates, to answer #1579. Three findings on chromadb 1.5.9
(PersistentClient, Rust bindings, single writer, zero concurrency) retire it.
**2 sits outside chroma's own declared valid range.** `hnsw_params.py:21-22`
validates both knobs as `isinstance(p, int) and p > 2`.
**The cost is write amplification, measured.** Bytes written (`/proc/self/io`) for
one mine of N records, 64-dim, `num_threads=1`, identical corpus and seed:
N 2/2 100/1000 ratio
10,000 47.1 MB 28.1 MB 1.67x
20,000 127.4 MB 60.0 MB 2.12x
40,000 390.3 MB 134.0 MB 2.91x
Two runs at N=20,000 reproduced within 0.1%. `sync_threshold` dominates — 3/3
measured identical to 2/2, and 1000/1000 identical to 100/1000, so `batch_size`
barely registers. The ratio grows with collection size, so the largest palaces pay
the most.
**It buys nothing.** A 5-record collection at 100/1000 — far below the threshold,
so no persist ever fires — reads back whole from a fresh process: `count()` and the
vector query both answer. On that artifact `link_lists.bin` is 0 bytes with no
`index_metadata.pickle`, which is #1579's trigger shape exactly, and
`quarantine_stale_hnsw()` creates no drift directory.
The knobs move into `_hnsw_creation_metadata()`, which both `create_collection`
paths now call, so per-collection tuning has one home.
Legacy `metadata=` is kept deliberately. Measured on 1.5.9: creating with
`configuration=` leaves `collection.metadata` as `None` and the sqlite
`collection_metadata` table empty, which blinds `_read_sync_threshold` and
`ChromaCollection.distance_metric` — both read that table. (`collection.configuration`
does read back under both paths; migrating those two readers to it is the
prerequisite for ever moving the writer, and belongs in its own change.)
Not claimed: that a small `sync_threshold` is a known chroma failure mode. No such
report was found in chroma's issues or docs, and chroma's own guidance runs the
other way. The Python persist path that #1579 and chroma#6975 describe does not
execute under the Rust bindings — `hnswlib` is not a dependency of this line.
Addresses fatkobra's CHANGES_REQUESTED review on this PR. The prior
truncation-detection fix in _paginate_ids left two loud-failure gaps:
1. When BOTH the offset get() and its no-offset fallback failed, the
`except Exception: break` returned the partial (or empty) ids
collected so far as if the listing were complete. On a mid-run
failure that hands scan_palace/rebuild a truncated palace; on the
first page it returns [] and scan_palace prints "Nothing to scan."
for a palace it never read. Now raises RuntimeError (chained from
the underlying error) instead of breaking.
2. The count()-based truncation disambiguation used the collection-wide
col.count(), but _paginate_ids accepts a where= filter (scan_palace
passes {"wing": only_wing}). A global count greater than the
collected rows does not prove a *filtered* result is truncated -- the
extra rows may belong to other wings. count() takes no where=
argument, so the raise is now gated on `where is None`; filtered
completeness is treated as unknown rather than false-positive-raised.
Adds three regression tests covering the after-progress double failure,
the first-page double failure, and the exactly-page-size filtered case.
Read-only gated on _MUTATING_TOOLS, which is the palace-write set the
peer-writer lease consults to decide which calls need the palace mine
lock. Two tools change state without touching the palace, so they are
correctly absent from that set and were served on a --read-only server
anyway: mempalace_hook_settings rewrites ~/.mempalace/config.json via
MempalaceConfig.set_hook_setting, and mempalace_memories_filed_away
unlinks ~/.mempalace/hook_state/last_checkpoint on both branches.
Add _READ_ONLY_REFUSED_TOOLS and point the dispatch gate and the
tools/list filter at it. _MUTATING_TOOLS and the peer-writer path are
unchanged: adding the two names there instead would put a config-only
tool under the palace lease, so a server that lost the lease to a peer
would answer -32001 for a call the lease has no say over.
mempalace_reconnect stays reachable on purpose and the comment records
why, since clearing ChromaBackend._quarantined_paths lets the reopen
rename a segment directory. The two --read-only help strings and the
matching row in the remote-server guide said "the mutating tools",
which now names the narrower set.
Third test now genuinely fails pre-fix (raw UTF-8 skip_name read), fixing the reviewer-flagged passes-either-way case. skip_names are lowercased on load, so assert on .lower().
dialect.py: all 11 text-mode open() calls (6 read, 5 write) omitted encoding=, so on a non-UTF-8-locale process (e.g. German Windows / cp1252) UTF-8-written JSON and AAAK text is decoded via the OS codepage, corrupting umlauts. config.py: 4 text opens lacked encoding= (config.json + people_map read paths, two writes); the other json.dump write paths already pinned UTF-8.
Audit findings #51 (dialect.py:360) and #84 (config.py:377). Regression: tests/test_encoding_hardening.py forces cp1252 default open and asserts umlaut round-trips through from_config / config.json read / raw-UTF-8 skip_name.
Independent review of the preceding commit found that dedup_palace's own
'Drawers: N' count() print runs a few lines before it calls
get_source_groups -- #92's palace_path guard on get_source_groups only
covered that function's internal count(), leaving this earlier, separate
call site in the same function fully exposed to the same #1222
SIGSEGV/panic class. Same fix pattern: preflight hnsw_capacity_status
before this print and abort the whole dedup_palace run on divergence.
show_stats has no equivalent exposure -- it goes straight to
get_source_groups with no separate count() print.
hnsw_capacity_status() (chroma.py) exists precisely to preflight the
#1222 SIGSEGV/pyo3-panic class before anything touches the HNSW
segment, but repo-wide it was wired into only 4 call sites while raw
count()/collection.count() is called at 20+ others -- a bare
except Exception around count() cannot catch a native crash, since
the process dies regardless of any Python try/except. This wires the
existing, already-tested probe into the 7 remaining call sites the
audit identified as CRITICAL:
- #89 palace.py::_enforce_embedder_identity -- the universal
get_collection() chokepoint every tool passes through, previously
guarded only by except Exception. Highest leverage: skips this
bookkeeping-only check on divergence instead of risking count().
- #90 migrate.py::migrate -- routes straight to the same
SQLite-extraction fallback the except branch already used, instead
of ever reaching col.count() when diverged.
- #91 repair.py::scan_palace / prune_corrupt -- both abort with the
existing from-sqlite recovery guidance instead of opening the
collection.
- #10 repair.py::rebuild_index -- preflights divergence alongside its
existing sqlite-integrity and poisoned-max-seq-id preflights, before
opening the collection.
- #13 repair.py::rebuild_index never rebuilt or reported on the
closets collection -- now warns when closets is still diverged
after a drawers-only rebuild, pointing at --mode from-sqlite instead
of letting 'Repair complete' stand unqualified.
- #92 dedup.py::get_source_groups -- takes an optional palace_path
(threaded from both callers) to preflight before count(); omitted by
existing tests, which keep their pre-existing behavior.
- #93 miner.py::status -- preflights before the ChromaDB-client
fallback path (used when the direct sqlite read is unavailable).
7 new regression tests, each confirmed failing against the pre-fix
code (via git stash of the source files only) and passing after the
fix. One existing dedup.py test updated for the new palace_path kwarg
in its call-signature assertion. Full suite: 3154 passed, 1 unrelated
pre-existing flake (test_mcp_server.py peer-writer-lock module-global
state leaking across test files in full-suite ordering -- this diff
never touches mcp_server.py).
#104 (CRITICAL, data-loss): the sweeper writes drawers with no
extract_mode at all (ingest_mode="sweep"). _metadata_matches_extract_mode's
legacy-compat rule -- "no extract_mode means treat as a legacy exchange
row" -- couldn't tell that apart from a genuine pre-schema convo_miner
row, so mempalace mine --mode convos (default extract=exchange) swept
every sweeper drawer for a shared transcript into its purge scope and
deleted them on the very next re-mine. The legacy-compat fallback now
only applies when the drawer is otherwise convo_miner's own (no
ingest_mode at all, or convo_miner's own "convos" tag) -- a drawer
positively identified as another producer's (sweep, or any other
foreign ingest_mode) never matches, using the same ingest_mode
discriminator sync.py already relies on for its own registry-row check.
#105 (MEDIUM, silent-failure): convo_miner's own instance of the
purge-failure swallow already fixed for miner.py at #23 -- a failed
purge in _file_chunks_locked was logged at debug level and mining
proceeded anyway, silently producing duplicate/stale drawers under
mixed schema versions. Now aborts (returns skipped=True, leaving the
old drawers' stored mtime untouched so the next mine retries) and
prints a visible warning.
428 tests pass across test_palace.py/test_convo_miner*.py/test_miner.py/
test_sweeper.py/test_hallways.py/test_format_miner.py/test_repair.py, no
regressions.
extract_via_sqlite drove its extraction with an INNER JOIN starting at
embedding_metadata. Any embedding with zero rows there (a sparse
historical write with no chroma:document and no other key -- the same
condition _extract_drawers already sanitizes for the collection-layer
rebuild path, see #1458) never appeared in the join result and was
silently dropped by mempalace repair --mode from-sqlite, with no count
mismatch and no warning.
Drive the query from embeddings (LEFT JOIN embedding_metadata) instead,
defaulting to an empty metadata dict when a row has none. Order by
e.id rather than em.id since em.id is NULL for unmatched rows.
Regression test seeds a real chromadb collection, then strips one
drawer's metadata rows directly via SQLite (current chromadb validates
against empty metadata on write, so this only reproduces on rows
already sitting in an older palace) and asserts the drawer still
comes back through the SQLite bypass.
cmd_repair() has a separate, inlined implementation of the same
live-swap-failure recovery that _rebuild_collection_via_temp's shared
fix (preserve the verified temp copy) already covers for rebuild_index().
Before this, cmd_repair's own except-block still did shutil.rmtree()
on the whole palace directory (destroying that same temp copy) and
restored a pre-repair full-directory backup -- printing recovery
guidance that pointed at data the same code path had just deleted.
Now both callers promote from the verified temp copy consistently.
Bare (#8)/#12: markers were internal finding IDs from a local audit
report, not GitHub references -- but they collide with real closed
issues in this repo (#8, #12) and would misdirect reviewers. Replaced
with plain prose.
Two independent bugs from the 2026-07-28 full-repo audit:
1. _rebuild_collection_via_temp deleted its own verified temp collection
in the except-handler even when the live collection had already been
replaced (live_replaced=True) -- at that point the temp copy is the
ONLY intact data left. Now preserved, and the error message points the
operator at it.
rebuild_index's own recovery path used to 'restore' by copying back a
pre-rebuild chroma.sqlite3 file backup and claim the palace was back
to pre-repair state -- but the live collection's on-disk HNSW segment
directories were already destroyed by the delete, so the restored
sqlite3 referenced segment UUIDs that no longer existed. Replaced with
a new _promote_temp_collection() that recovers by re-extracting from
the verified temp copy and re-uploading into a fresh live collection,
with an honest fallback message if that promotion also fails.
2. _paginate_ids silently returned a truncated ID list when offset-based
pagination failed and the no-offset fallback got structurally stuck
at the first page (1000 results). Callers (scan_palace, rebuild) treat
the result as the complete palace, so this could act on incomplete
data with no warning. Now cross-checks against the collection's own
count() to disambiguate a genuinely-complete page-sized collection
from a truncated larger one, and raises instead of silently truncating
when it can't tell.
Both fixes verified against an independent adversarial review pass
(mutation-tested: confirmed the new tests catch a wrong-source-collection
read and a swapped ids/documents payload in the recovery path, which the
original test set did not).
Deliberately out of scope: none of the rebuild paths hold mine_palace_lock
for their full duration, so a concurrent writer can still interleave with
a rebuild. That's a separate, larger architectural gap (queue-and-replay
vs. block-on-write), tracked for its own follow-up.
A stdio mcp_server orphaned by a dropped SSH session used to swallow
pipe errors in the generic exception handler and sleep forever on the
dead channel, holding the mine_palace flock (2026-07-10 write-path
outage, PID 23392). The loop now logs and shuts down on stdin EOF,
OSError from stdin reads (EIO/EBADF from orphaned ptys), and
BrokenPipeError/OSError on stdout writes. Shutdown returns from the
loop for a normal interpreter exit 0 — no os._exit — so held flocks
release via process teardown; a broken stdout is re-pointed at devnull
first so the shutdown flush cannot turn the exit into status 120.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DpEdb9SNWF42BpupHWBhC3
The English entity-candidate pattern in i18n/en.json backtracks
catastrophically on a long unbroken run of printable ASCII (base64,
minified JS, hashes, data URIs), pinning `mempalace mine` on a single
~5000-char window for hours. Collapse such runs to a space before
single-word candidate matching, in both consumers that apply the pattern
(palace._candidate_entity_words and entity_detector.extract_candidates).
Scoped to ASCII ([!-~]) so non-ASCII scripts stay untouched — a CJK
paragraph is one unbroken run with no ASCII whitespace, and zh-CN/zh-TW
have no multi-word fallback, so a blanket collapse would erase their
detection. Threshold 24 sits above the 20-char cap of the simple-name
pattern, so no real name is dropped.
Fixes#2063.
Co-authored-by: Ryan Wei <9876551+RyanWei@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A daemon job refused the palace write lock was marked terminal `failed` and
never retried. `mine_palace_lock` guards the palace write itself, so a refusal
means no drawer was filed, but the queue recorded it exactly like a crash
mid-execution, whose outcome is unknown. The work was dropped: it only ran
again if a hook happened to re-submit equivalent work, and a job kind no hook
re-emits was lost outright.
Refusals now defer. The job goes back to `queued` with `started_at` cleared,
the claim's attempt increment undone, and the reason recorded. The update is
scoped to the claim that was refused (started_at match), so a defer racing a
recovery re-claim cannot re-queue work it does not own. `claim_next` clears
the recorded reason so it never outlives the claim it describes. Every other
failure stays terminal: a crashed job's outcome is unknown, and blindly
re-running a non-idempotent kind would re-file verbatim content, which is what
MAX_ATTEMPTS guards.
The worker cools a refused job off in memory and moves on rather than sleeping
in line. It is the only worker and the holder keeps the lock until its write
finishes, which can be a long mine, so blocking would stall every unrelated job
behind a lock that has nothing to do with them, and a job merely queued behind
the refused one would never be claimed. `claim_next(exclude=...)` skips a
cooling job, so the oldest-first ordering cannot hand the same refused job back
forever while newer work waits. The filter runs in Python rather than an
`id NOT IN (?, ?, ...)` list, which would bind one host parameter per cooling
job against a cap that defaults to 999 before SQLite 3.32. The cooldown
lives in the worker, not the schema, which has no migrations; a restart just
retries at once, costing one refusal, never work.
`tool_diary_write` swallowed `MineAlreadyRunning` in its bare `except
Exception`, so the refusal reached the daemon with no `error_class` and
`diary_write` would still have been dead-lettered. It now uses a typed handler
ahead of the bare `Exception`, the way `tool_mine` and `tool_sync` already do.
Deferral makes a refused job non-terminal, and `DaemonClient.wait` only returns
on a terminal state, so callers that wait on purpose would have waited for a
state a parked job cannot reach: a foreground `mine --daemon` for the one-hour
default, and the `hooks_cli` pre-compaction mine and Stop-hook diary paths for
their whole timeout on every fire, each then reporting a failure that never
happened. `wait(stop_on_lock_deferral=True)` hands the parked job back instead.
The CLI echoes the global `--palace` back into the command it suggests, so the
suggestion does not silently list the default palace's queue instead of the
one the job is parked in. A job that is genuinely running is still waited out.
Co-Authored-By: mjvmsteixeira <185609735+mjvmsteixeira@users.noreply.github.com>
chunk_text's windowing loop (start = end - chunk_overlap) stops advancing
when chunk_overlap exceeds chunk_size // 2 on short-line content, looping
forever at 100% CPU. A paragraph or line boundary pull only moves end past
start + chunk_size // 2, so a pulled chunk spans more than half the chunk
size; the step then advances only while chunk_overlap <= chunk_size // 2.
Tighten the existing guard from chunk_overlap >= chunk_size to
chunk_overlap > chunk_size // 2 in both chunk_text (raise, with the numeric
bound in the message) and MempalaceConfig._validated_chunk_config (repair to
min(DEFAULT_CHUNK_OVERLAP, chunk_size // 2)). The windowing loop is
unchanged, so every already-valid config produces byte-identical chunks; 50%
overlap (chunk_overlap == chunk_size // 2) stays valid.
Fixes#2056.
chunk_text recomputed line_start/line_end with a full-prefix
content.count("\n", 0, pos) per chunk (O(N*K) overall), so large files
ground for days. Keep the emitted values byte-identical but tally
newlines incrementally over the newly-scanned span (O(N) total), with a
from-scratch fallback that keeps every value exact.
hnsw_capacity_status() runs on every search, duplicate check and status
call (MCP and CLI). Each call cost a COUNT(*) over the embeddings table
and a full unpickle of the segment metadata, both scaling with palace
size. Cache the verdict per (palace_path, collection_name) and reuse it
while an (inode, mtime_ns, size) signature over chroma.sqlite3, its -wal
sidecar and index_metadata.pickle is unchanged, so an external write
invalidates it at once rather than after a fixed window. The #1222
divergence guard stays fresh: a mid-probe write or a locked read is
returned but never cached, and the signature brackets the whole probe so
a pickle rewrite during it cannot be pinned as fresh.
normalize() joined every conversation in a Claude.ai privacy export
bundle into one string before it was hashed, so re-exporting the bundle
with one new conversation added changed the whole-file hash and the
existing conversations were re-mined as duplicates. Split normalization
into normalize_conversations() so each conversation can be hashed and
deduped on its own.
Also scope the content-hash map by wing — mining the same transcript
into a second wing is a deliberate re-file, not a duplicate, and was
leaving that wing with only the registry sentinel.
Repeated exports from Claude/ChatGPT land the same conversation under a
new filename each run (timestamped bundle, regenerated slug, etc.), so
the existing source_file/mtime skip never recognized it as already
mined and re-filed a duplicate set of drawers. Now content is hashed
after normalization and checked against previously filed hashes, so
the same conversation under a new path is skipped instead of
duplicated.
Implements the pgvector server-side implementation of BaseCollection.facet_counts(field, where, limit), completing the pgvector fast path for the aggregation contract merged in #1868 and specified in #1835.
The new real-Chroma checkpoint test dropped its create-time client with a
bare del, leaving the per-path SharedSystemClient's SQLite/HNSW handles
open on Windows (#1128). Close both clients so the temp palace is released,
matching the conftest fixture's close-not-del pattern.
mempalace_checkpoint hard-coded added_by="checkpoint" for every drawer,
dropping the filing agent's identity even though it arrives in the same
call via diary.agent_name. Add an optional top-level added_by parameter
and resolve attribution as explicit > diary agent_name > "checkpoint";
blank/whitespace/non-string values defer to the next source. The value
is declared in the tool schema so tools/call admits it on both stdio and
HTTP transports.
Fixes#2023
Co-Authored-By: epinethrone <172391900+epinethrone@users.noreply.github.com>
Hermes' memory tool defaults to target="memory" (the agent's own
notes); filtering on_memory_write to target == "user" silently dropped
the majority of writes. Mirror both targets under distinct subjects —
user→asserted for user facts, hermes→noted for agent notes — so
kg_query("user") never surfaces environment quirks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The provider filed and searched self._palace_path while the mcp_server
passthrough tools resolved mempalace's global config — a custom Hermes
palace_path searched one palace while drawer CRUD, duplicate checks,
and tunnels wrote another. Publish the resolved palace to
MEMPALACE_PALACE_PATH (mcp_server's own --palace mechanism) with
ownership tracking so a stale bridge never outranks edited config and
a user-set env var is never touched. The KG tools become native
handlers: mcp_server resolves its KG from DEFAULT_KG_PATH unless its
own CLI flag was given, which no env bridge can influence.
collection_name and the unset-palace default now defer to
MempalaceConfig — the same single chain the MCP server itself uses.
Also document that hermes backup does not cover ~/.mempalace (no ABC
hook exists for contributing external paths).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
on_session_end and on_pre_compress blind-re-filed the raw message list,
duplicating every turn sync_turn had already stored — filed_at is hashed
into the drawer id, so upserts cannot collapse the copies. Drop the
re-filing (and the pre-compress hint that over-promised persistence);
on_session_end keeps only the wake-up cache refresh.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- _scan_metadatas: fetch cap+1 and compare len > cap, so a collection
holding exactly STATUS_SCAN_LIMIT rows is no longer reported as
truncated (the view is complete). Callers still get at most cap rows.
- status/list_wings/list_rooms: tolerate None metadata entries from
legacy palaces / raw writers instead of failing the tool call.
- _match_wing_by_keywords: skip non-string keywords so a hand-edited
wing_config.json can't break live turn filing.
- file_conversation_exchange: extra_metadata can no longer overwrite
canonical keys (matches the documented append-only contract), and
wing/room are validated with sanitize_name — invalid names fall back
to wing_general / conversations rather than dropping the turn, per
the verbatim-first mandate.
- Fix two stale docstrings left from the pre-split layout.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Hermes provider from feat/hermes-integration, split out per review
on #1684 — provider + tests only; backfill, the hermes install CLI,
and docs follow in a stacked PR.
Changes vs the original branch:
- _file_turn routes through convo_miner.file_conversation_exchange()
instead of a hand-rolled col.upsert, so live turns carry canonical
drawer metadata and the ids.py ID recipe.
- The backfill/live wing-routing parity test moves to the backfill PR.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Live agent integrations and their backfills need to file one
conversation exchange at a time, but hand-rolling the upsert leaves
drawers without hall / entities / filed_at / extract_mode metadata —
silently invisible to hallway traversal, entity search, and the
since/before date filters.
file_conversation_exchange() builds the same metadata the convo miner
writes, and make_exchange_drawer_id() moves the ID construction into
ids.py per its single-source-of-truth contract (full-content hash, no
prefix collisions; filed_at keeps repeated exchanges distinct).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
On the HTTP transport, a client that hangs up mid-response makes the send
path raise an unhandled BrokenPipeError / ConnectionResetError, so the
default socketserver handle_error logs a full traceback for a routine
disconnect. Override handle_error on the HTTP server to log the disconnect
(ConnectionError, plus ssl.SSLEOFError over TLS) at DEBUG and drop it, while
every other exception still reaches the default handler with its traceback.
The wider ssl.SSLError tree is left uncaught, so genuine TLS handshake/cert
failures still surface.
Addresses the disconnect half of #2003.
Claude Code pages large tool outputs to <session>/tool-results/*.txt
inside ~/.claude/projects/<slug>/. These are raw machine dumps (command
output, MCP results) referenced from the transcript JSONL — not
conversations. The convo scanner picked them up as plain .txt, so every
transcript ingest stored megabytes of tool output as memories: on one
field palace, 12,825 drawers traced back to tool-results files, with a
single paged file producing 3,611 drawers.
Adds CONVO_SKIP_DIRS = SKIP_DIRS | {'tool-results'} used only by
scan_convos — project-mining semantics are unchanged, and regular nested
conversation dirs still scan (covered by a companion test).
_mark_held(palace_key) ran before the try: whose finally runs
_mark_released(). An async exception (SIGINT/KeyboardInterrupt) landing
after _mark_held() and before the try: skips _mark_released(), stranding
the key in the process-wide _palace_lock_keys set while the outer finally
frees the flock. The in-memory hold then outlives the OS lock: a later
re-entrant acquire passes through and writes without the flock while
another process can acquire it, i.e. two writers into one palace.
Move _mark_held() inside the try so it pairs with _mark_released() on
every exit. Add a regression test that injects the interrupt in the
window and asserts the holder set is not stranded.
The stdio loop ran _refresh_sqlite_integrity_status() and
_refresh_vector_disabled_flag() before reading the first request.
PRAGMA quick_check reads every page of chroma.sqlite3, so on multi-GB
palaces the probe alone (measured: 20.3s on a 1.72 GB / 326k-drawer
palace, 40-46s under disk/lock contention) starves the MCP client's
60s connect timeout — even though the initialize response itself never
touches the database. The HTTP transport already starts without the
synchronous probe.
Move both probes to a daemon thread (mcp-startup-preflight). The #1222
intent is preserved: the probe still starts at startup and logs its
warning as soon as it finishes. Consumers that need the verdict
(_ensure_sqlite_integrity_status via the tool-call integrity gate)
serialize on a new _sqlite_integrity_refresh_lock with double-checked
locking, so a tool call arriving mid-probe waits for the in-flight
verdict instead of running a second O(database size) quick_check —
and never proceeds unverified.
Measured on the 1.72 GB palace with the >512 MB startup gate disabled
(MEMPALACE_STARTUP_INTEGRITY_MAX_MB=0, full quick_check in flight):
initialize 1.4s (was 20-46s); first tool call after probe completion
3.4s with sqlite_integrity checked=true ok=true.
Complements c54531a: the oversized-palace skip still applies to the
background probe, but the handshake no longer depends on it.
PR #1960 merged with a red `lint` job: `ruff format --check .` wanted to
collapse the multi-line `MineAlreadyRunning(...)` raise in the new
`test_peer_writer_readonly_self_heals_after_peer_exits` onto one line
(it fits the line-length limit). All six real test jobs passed; only the
formatter check failed, which left `develop` red on lint.
Reformat that one statement so `ruff format --check .` is clean again.
No logic change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The #1818 peer-writer guard latched _MCP_WRITER_READ_ONLY=True on the first
MineAlreadyRunning and short-circuited every subsequent acquisition attempt,
so a server that came up read-only (a peer held the per-palace flock at
startup) stayed read-only for its entire process lifetime — even long after
the peer exited and the OS released the flock. In the common case of several
overlapping Claude sessions (one server per session, all on the same palace),
whichever session started second was stranded: mutating tools kept refusing
with -32001 and the only remedy was killing/restarting that server.
_mcp_peer_writer_refusal already calls _acquire_mcp_writer_lock() on every
mutating tool, so the retry hook existed — the sticky latch just suppressed it.
Drop the read-only short-circuit: when read-only we now re-attempt the
non-blocking flock each call and transparently promote to writer once the peer
is gone. Race-safe — fcntl LOCK_NB is kernel-arbitrated, so two servers can
never both win. The genuinely-broken-lock path (_MCP_WRITER_LOCK_FAILED) is
still cached, since retrying a broken lock mechanism can't help.
Adds test_peer_writer_readonly_self_heals_after_peer_exits.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Conversation transcripts were assumed immutable once mined: the bulk
skip-check (prefetch_mined_set) only tracked "have we seen this
source_file before at the current normalize_version", with no mtime
comparison at all. That's wrong for how Claude Code sessions actually
work -- a session keeps appending to its own JSONL file while active,
and /compact or /clear can rewrite one in place. Once a session file
was mined, any content appended after that point would silently never
get mined, with no error or warning -- the file just looked
"already filed" forever.
palace.py:
- prefetch_mined_set() now returns dict[source_file, stored_mtime]
instead of a bare set[source_file]. `if src in mined_set` still works
identically (dict `in` checks keys), so this is a source-compatible
change for that access pattern; a caller that wants staleness
detection reads mined_set[src] and compares against the file's
current mtime itself. None means no mtime was ever stored (or
getmtime failed when the drawer was written) and must be treated as
stale, not "unknown, assume unchanged".
- Removed bulk_check_mined(): it already existed for exactly this
purpose (bulk mtime prefetch) but had zero callers anywhere in the
codebase and was missing the normalize_version/extract_mode filtering
prefetch_mined_set has -- folded its intent into prefetch_mined_set
instead of maintaining two subtly-different, overlapping bulk scans
over the same underlying data.
- file_already_mined()'s docstring corrected: it previously claimed
"transcripts are assumed immutable" for convo mining. That's no
longer true; corrected to describe the actual current split (convo
miner's bulk skip-check uses prefetch_mined_set's stored mtimes; this
function's check_mtime=True path is now only its per-file,
lock-held race-condition recheck).
convo_miner.py:
- New _is_unchanged_since_last_mine() helper (extracted to keep
_mine_convos_impl under the repo's cyclomatic-complexity gate):
false whenever the file isn't in the prefetched map, its stored mtime
is None, getmtime fails, or the mtimes don't match -- true only when
genuinely unchanged.
- _file_chunks_locked's metadata now stamps source_mtime on every real
drawer (mirroring miner.py's existing pattern), and its in-lock
recheck now passes check_mtime=True.
- _register_file's 0-chunk sentinel also stamps source_mtime, so a
short file that later grows past MIN_CHUNK_SIZE is detected as
changed instead of being skipped forever by the sentinel.
One-time cost worth flagging: no existing convo drawer has source_mtime
stored (this field never existed for convo mining before now), so the
first `mempalace mine --mode convos` after this ships will see every
already-mined file as stale and fully re-mine it. Not a bug --
_file_chunks_locked's existing purge-before-insert means no
duplication results -- just a real, one-time cost across a large
corpus.
tests/test_convo_miner.py: 7 new tests -- grown-file re-mine picks up
new content, unchanged file still skipped (the mtime check must not
regress the existing optimization), grown-file re-mine purges stale
drawers rather than accumulating duplicates (checked via unique content
markers, not raw counts -- ChromaDB collections can carry unrelated
bookkeeping rows), prefetch_mined_set's returned mtime matches the real
file, None handling for a drawer with no stored mtime, a legacy
drawer (no source_mtime field, simulating pre-this-change data) is
correctly re-mined rather than skipped forever, and the sentinel path
stamps source_mtime too.
Full suite: 3327 passed, 20 skipped, 0 failed. ruff check / ruff
format -- clean.
Rebasing Lochness's exclude_patterns work (#1213) onto current develop
pushed _mine_impl's cyclomatic complexity to 26, tripping the repo's
max-complexity=25 ruff gate (clean on develop before this rebase).
Extracted the pre-scanned-file-list filtering branch into
_apply_exclude_patterns_to_prescanned_files -- same behavior, no test
changes needed, complexity back under the gate.
Allow projects to specify .gitignore-style patterns that the miner should
skip, without relying on .gitignore for mining control.
A new optional exclude_patterns list in mempalace.yaml is parsed by the
existing GitignoreMatcher class via a new from_patterns() classmethod —
same syntax, same semantics as .gitignore, no new dependency.
exclude_patterns:
- '*.md'
- '*.yaml'
- 'docs/' # dir-only: prunes entire tree without descending
- 'dist/'
- 'coverage/'
Key behaviour:
- Patterns follow .gitignore rules: anchoring (/pattern), dir-only
- dirs[:] pruning via GitignoreMatcher.matches(..., is_dir=True) so
excluded subtrees are never walked
- Checked after .gitignore filtering; force_include (--include-ignored)
bypasses exclude_patterns
- Pre-scanned files lists (init double-scan optimisation) are filtered too
- Backwards compatible: omitting exclude_patterns changes nothing
Changes:
- GitignoreMatcher.from_patterns(): new classmethod, same rule parser as
from_dir(), reads from a list instead of a file on disk
- scan_project(): builds one exclude_matcher before os.walk; used for
both dirs[:] pruning and per-file filtering
- _mine_impl(): applies the exclude matcher to pre-scanned files lists
when provided by the caller
- tests/test_miner.py: three new tests
test_scan_project_exclude_patterns_skips_matching_files
test_scan_project_exclude_patterns_prunes_entire_directory
test_scan_project_exclude_patterns_include_ignored_bypasses_exclusion
Two review comments on this PR, both addressed:
- gemini-code-assist flagged that maybe_autoheal_fts5_index's default
progress=print goes straight to stdout. _validate_palace_fts5_after_mine
runs inside the MCP server process too (mcp_server.tool_mine ->
miner.mine), where stdout is the JSON-RPC transport -- a stray print()
there would corrupt the protocol stream and crash the connection. Pass
progress=logger.info instead; palace.py already has the module logger.
- nikkunikku corroborated the fix from a real 1.4GB production palace (278
repeated abort-loop iterations before the fix) and pointed out a real
test gap: the fixture-based auto-heal tests fabricate real FTS5
corruption via direct shadow-table writes, which some SQLite builds
refuse (existing pytest.skip paths in test_miner_fts5_validation.py,
related to #1925) -- so on those builds the auto-heal wiring in
_validate_palace_fts5_after_mine is never actually exercised. Added
their suggested build-independent tests, adapted to this file's fixture
helpers: test_validator_suppresses_raise_when_autoheal_clears and
test_validator_still_raises_when_autoheal_cannot_clear, stubbing
mempalace.repair.sqlite_integrity_errors/maybe_autoheal_fts5_index
directly instead of fabricating corruption.
Added a third test, test_validator_passes_logger_progress_not_print_to_autoheal,
covering the specific progress= wiring: the two tests above mock
maybe_autoheal_fts5_index entirely and discard its kwargs, so neither would
have caught the progress=print regression this commit actually fixes. The
new test captures the real kwargs and asserts progress is a bound method of
palace.py's own logger (not print), without pinning to logger.info
specifically -- severity level is a verbosity choice, not a correctness
requirement, so the assertion shouldn't fail on a reasonable future change
to e.g. logger.debug. Verified both directions: fails against the
pre-fix `print` default (and shows the leaked stdout line to prove it),
passes at .info and at .debug alike.
Full suite: 3221 passed, 20 skipped (unchanged skip count). ruff
check/format clean.
mempalace_status reported a passing SQLite integrity check on non-chroma
backends (checked/ok true, sqlite_path pointing at a chroma.sqlite3 that does
not exist) even though _refresh_sqlite_integrity_status short-circuits the
check there. _sqlite_integrity_payload now reports the check as not-applicable
(checked false, ok null, reason) for non-chroma backends, keeping the chroma
payload shape and error surfacing unchanged.
Co-Authored-By: Zoz92 <66385795+Zoz92@users.noreply.github.com>
shutil.move's fallback for a failed os.rename is copytree + rmtree. On
Windows, when any file inside the palace is held open by another
process (a live MCP server, a running mine, another harness), the
rename fails and shutil.move falls back to deleting the live palace
file-by-file via rmtree -- which itself then fails partway through on
the first locked file, leaving the palace partially gutted next to a
partial (or empty) archive copy.
Reproduced live twice (Windows 11, 2026-07-05 and 2026-07-06): running
`mempalace repair --mode from-sqlite --yes --archive-existing` while
an MCP server / detached mine held palace/*/data_level0.bin open threw
mid-rmtree in both cases. The palace itself survived only because the
specific locked files could not be unlinked -- a different lock
pattern (e.g. a lock on a file rmtree reaches first) would have lost
data with no way back.
os.rename is atomic on both platforms it matters on (POSIX rename(2),
Windows MoveFileEx) -- it either fully succeeds or fails without
touching anything. Catch the failure and abort cleanly with actionable
guidance instead of a raw traceback.
_errors_are_isolated_fts5 gated auto-heal on one specific message shape:
malformed inverted index for FTS5 table
SQLite >= ~3.5x (confirmed on 3.53.2 / Python 3.13.7) reports the same
isolated-FTS5 condition with different wording instead:
fts5: corruption found reading blob N from table "embedding_fulltext_search"
The narrow regex never matched this phrasing, so maybe_autoheal_fts5_index
silently declined to heal on any machine running a recent-enough SQLite,
falling straight through to the hard-abort path -- the exact condition
the whole auto-heal feature (#1926/#1928) exists to avoid. Widened the
pattern to match either wording.
Caught by running this repo's own test suite on this machine:
test_repair.py's two auto-heal tests were failing (not, as assumed
earlier, pre-existing/unrelated flakiness -- that assumption was never
actually verified). Traced to this exact classification gap.
Fixing this correctly also exposed that four tests in
test_miner_fts5_validation.py had been passing for the wrong reason: they
manufacture the exact "reporter-shaped" isolated-FTS5 corruption (#1926's
actual bug shape) and asserted mine() must raise MineValidationError for
it -- true only because the classifier bug prevented auto-heal from ever
engaging. With the classifier fixed, that corruption is now correctly
auto-healed and mine() succeeds instead, so those tests' expectations
were stale, not their fixtures being invalid:
- test_helper_raises_on_fts5_segment_corruption -> renamed
test_helper_auto_heals_fts5_segment_corruption; asserts no raise + a
clean post-heal quick_check, instead of expecting a raise.
- test_full_chain_raises_through_mine_impl and
test_mine_impl_does_not_print_partial_summary_on_validation_error: their
real purpose is exception-passthrough / banner-suppression when the
validator DOES raise, not proving any particular corruption triggers it.
Switched from real file corruption to a monkeypatched raise. (Tried
swapping to _page_mangle's non-isolated corruption first -- that made
ChromaDB's own Rust bindings panic just opening the file for the
re-mine's get_collection() call, a native crash rather than a catchable
Python exception, before the validator ever ran. Different failure mode
than what these tests are about, and not reliable to depend on.)
- test_mine_formats_full_chain_raises_when_fts5_corrupt: same fix, mirrors
the miner-path change for the extract path.
- Added test_full_chain_auto_heals_isolated_fts5_corruption and
test_mine_formats_full_chain_auto_heals_isolated_fts5_corruption as
companions, proving the full mine()/mine_formats() chain -- not just
the standalone validator -- actually auto-heals and succeeds end-to-end
for the isolated case now that it's correctly classified.
- test_errors_are_isolated_fts5_classification: added the new message
wording as an explicit regression fixture (pinned literally, not
dependent on whatever this machine's SQLite happens to emit).
Full suite: 3302 passed, 20 skipped, 0 failed -- first fully clean run
this session. ruff check / ruff format -- clean.
mempalace_checkpoint and mempalace_delete_by_source (added in 3.5.0) were
missing from _MUTATING_TOOLS, so a server started with --read-only /
MEMPALACE_MCP_READ_ONLY=1 still allowed writing drawers and bulk-deleting
by source. The same gap affected the peer-writer lock gate, which uses
the same frozenset.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
mempalace mine aborts with the "ABORT: SQLite-layer corruption detected"
banner on an isolated FTS5 inverted-index corruption -- the specific case
maybe_autoheal_fts5_index already exists to fix in place. That helper is
wired into cmd_repair's preflight, but not into mine's own post-mine
validation (palace._validate_palace_fts5_after_mine), so mine forces a
manual `mempalace repair` for a corruption class that's already safely
self-healable.
This wires the same auto-heal call into _validate_palace_fts5_after_mine,
before it raises MineValidationError. maybe_autoheal_fts5_index returns
the *remaining* errors after the heal attempt, so MineValidationError
still raises whenever the corruption isn't the isolated, fully-healable
case -- this only changes behavior when the heal has verifiably and
fully cleared the corruption.
Verified against a real-world repro (mining a real Claude Code project
directory deterministically triggered this corruption after all files
filed successfully, zero concurrency, single uninterrupted process):
mempalace repair --yes confirmed the auto-heal path clears it before
proceeding to a full rebuild. With this patch, mine self-heals the same
case directly -- no abort, Files processed: <n>, Done, PRAGMA quick_check
clean afterward.
Test suite: 3214 passed, 20 skipped -- no new failures. Two pre-existing
unrelated failures in test_repair.py (a SQLite-version-dependent FTS5
corruption message-wording mismatch, tracked separately) reproduce
identically on unmodified develop.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The MCP server ran PRAGMA quick_check on the full chroma.sqlite3 during
startup, before answering the initialize handshake. quick_check is
O(database size); on multi-GB palaces it exceeds the MCP client's ~30s
connection timeout, so the server never finishes starting and the client
drops the connection (observed >2min on a 4.6GB palace).
Skip the startup probe when chroma.sqlite3 exceeds
MEMPALACE_STARTUP_INTEGRITY_MAX_MB (default 512MB; 0 disables). The gate
lives in _refresh_sqlite_integrity_status, the single choke point for the
startup calls and every lazy consumer. `mempalace repair` preflight still
runs the full quick_check via repair.sqlite_integrity_errors, so
SQLite-layer corruption is still caught on the destructive path.
Refs #1818.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Jq495N7e2D4wY2Mp2AQvg7
Browser-validated the table layout across desktop (1280) and mobile (375):
- Denser doc-table cell padding (8px 16px -> 8px 12px) so comparison tables
fit the content column instead of needing a horizontal scrollbar.
- `overflow-wrap: break-word` on table-cell code so only genuinely long
values (e.g. a Postgres DSN) wrap, while short identifiers like
`palace_path` keep natural column sizing and stay on one line.
- Drop the redundant 'Configure with' column from the storage-backends table
(each backend's connection variables are documented in full in its own
subsection right below) and shorten 'Local (exact cosine)' -> 'Local
(exact)'. The comparison table is now five columns and fits cleanly.
Verified no clipping and no page-level horizontal overflow on the
configuration, remote-server, reference (cli/mcp-tools/python-api),
claude-code, and knowledge-graph pages; wide tables scroll within their own
container on mobile.
The custom theme set `.vp-doc table { overflow: hidden }` to clip its
rounded corners, which also overrode VitePress's default `overflow-x: auto`
— so any table wider than the content column was clipped with no way to
scroll to the hidden columns (visible on the storage-backends table). Switch
to `overflow-x: auto` so wide tables scroll, keeping the rounded corners.
Also shorten the storage-backends table's two capability headers
(Namespace isolation -> Namespaces, Lexical search -> Lexical) so the table
fits the content column without needing the scrollbar.
Rename the table's 'Select with' column to 'Configure with' and list each
backend's primary connection knob, since a connection variable (e.g.
MEMPALACE_QDRANT_URL) configures a backend but does not select it — selection
is uniform via --backend / MEMPALACE_BACKEND, covered in the prose below the
table. Also state the concrete MEMPALACE_QDRANT_TIMEOUT default (10.0s).
Establish guide/configuration.md as the canonical home for per-backend
connection settings, with a compatibility table and connection-variable
reference for the chroma, sqlite_exact, qdrant, and pgvector backends.
remote-server.md already links Postgres + pgvector to /guide/configuration,
but the page had no backend section; this populates that target. New backends
add one table row plus a connection subsection, keeping README's compatibility
table in sync rather than accreting a prose paragraph per backend.
LaTeX source files and BibTeX bibliographies are prose-rich content that
benefits from both palace mining and entity detection. Adds the two
extensions to the two extension lists most relevant to them, each with a
matching test.
- ``mempalace/miner.py:READABLE_EXTENSIONS`` — ``.tex`` / ``.bib`` join the
mining allowlist (parallel to the Swift/Kotlin PR #1368 and the PHP
ecosystem PR #1819).
- ``mempalace/entity_detector.py:PROSE_EXTENSIONS`` — ``.tex`` / ``.bib``
also join the *preferred* entity-detection bucket alongside ``.md`` /
``.rst`` / ``.csv``, NOT the broader code-file fallback. The reason
``PROSE_EXTENSIONS`` exists separately is documented in-code:
programming-language files have lots of capitalized identifiers (class
names, function names) that produce false-positive person matches.
LaTeX/BibTeX don't have that problem — they're typesetting languages
for prose documents. ``.bib`` in particular is almost entirely author
names, one of the highest real-entity densities of any file type the
detector scans.
Tests follow the patterns established by the prior extension PRs:
``tests/test_miner.py::test_scan_project_includes_latex_files`` mirrors
the Swift/Kotlin scan tests, and
``tests/test_entity_detector.py::test_scan_for_detection_includes_latex_prose``
mirrors ``test_scan_for_detection_finds_prose``. The existing
``test_prose_extensions`` was extended to assert the two new entries.
Full env-cleared suite: 3216 passed, 20 skipped. ``ruff check .`` and
``ruff format --check .`` both clean.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KC5Qsknh2zFRtRvVyjXiTA
* feat(serve): turnkey secure remote MCP server (#1877)
Add `mempalace serve`: a secure-by-default wrapper over the HTTP MCP
transport so a team can stand up a shared central palace with one
command.
Server capabilities (mempalace/mcp_server.py):
- Native TLS via --tls-cert/--tls-key (env MEMPALACE_MCP_TLS_CERT/_KEY):
wraps the socket in a TLS 1.2+ context, validated before bind. Token
is still required on a non-loopback bind (TLS != auth).
- Read-only mode via --read-only (env MEMPALACE_MCP_READ_ONLY): the 24
mutating tools are hidden from tools/list and refused at dispatch
(-32003), enforced before arg handling — not merely hidden.
Turnkey command (mempalace/cli.py):
- Auto-generates a strong bearer token for non-loopback binds, stored
0600 under ~/.mempalace/server/ and printed once; reused across
restarts. Token rides in the child env, never argv, so it can't leak
via ps.
- Prints a ready-to-paste client config (scheme reflects TLS), then
foreground-execs the real server so Docker/systemd own the lifecycle.
Deployment (deploy/):
- docker-compose.server.yml wires the server + Qdrant with a /healthz
healthcheck and persistent volumes.
- server.env.example documents the env surface.
- mempalace-server.service is a hardened systemd unit template.
Tests: TLS handshake (openssl-gated), read-only enforcement, token
autogen/0600/reuse, token-not-in-argv, secure-by-default gates.
Docs: remote-server guide now leads with `mempalace serve` plus Compose
and systemd subsections.
* test(serve): fix Windows — don't patch os.name; gate 0600 asserts to POSIX
Patching os.name to 'posix' broke Path.home() on Windows (pathlib mixed
POSIX home resolution with Windows drive parsing). Capture both exec
branches (os.execve + subprocess.run) instead, and guard the POSIX
permission-bit assertions behind os.name == 'posix' (Windows files
report 0o666).
Per Gemini review (PR #1898 comment r3489013681): the forwarder lacked the
``-> dict[str, int]`` return annotation that ``BaseCollection.facet_counts``
and the sibling ``get_all_metadata`` forwarder both carry. One-line
consistency fix, no behavior change.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KC5Qsknh2zFRtRvVyjXiTA
Both methods are concrete on ``BaseCollection`` (``facet_counts`` raises
``UnsupportedCapabilityError``; ``get_all_metadata`` pages through
``self.get(include=["metadatas"])``). Python MRO resolves them on
``EmbeddingCollection`` before ``__getattr__`` ever fires, so without an
explicit forwarder the wrapper silently runs the base default instead of
delegating to the wrapped backend's optimized implementation. The pattern
matches the existing explicit forwarders for ``distance_metric``,
``lexical_search``, and the embedder-identity trio — all added to fix the
same shadow.
What this means in production for the three backends that get wrapped
(``EmbeddingCollection`` only applies to ``requires_explicit_embeddings``
backends — qdrant, pgvector, sqlite_exact; chroma is unwrapped and
unaffected):
- **facet_counts shadow (#1868 regression)**: every ``mempalace_status``,
``list_wings``, ``list_rooms``, ``get_taxonomy`` call routes through
the gated ``col.facet_counts(...)`` path. The capability check passes
(``supports_metadata_facets`` is on the backend), but the call hits the
wrapper's MRO-resolved ``BaseCollection.facet_counts`` and raises
``UnsupportedCapabilityError``. ``mcp_server``'s broad ``except`` swallows
it, logs ``WARN Failed to fetch metadata facets, falling back to client-
side loop: backend does not support facet_counts``, and counts via the
O(n) Python loop — the exact behavior #1868 was designed to eliminate.
- **get_all_metadata shadow (#1796 / #1892 regression)**: the BaseCollection
default pages through ``self.get(include=["metadatas"])`` — fine for
Chroma's SQL OFFSET cursor, but on wrapped backends (qdrant, pgvector)
the inner's overridden ``get_all_metadata`` is unreachable. For pgvector
specifically, this means #1892's ``with_document=False`` fast path is
never taken even though it's implemented — every metadata-only fetch
transfers the full document column over the wire. On a 13k-drawer remote
pgvector palace over WAN that's ~13MB per call, dominating wall time.
Why no test caught it: backend tests (``test_qdrant_backend.py``,
``test_pgvector_backend.py``) call the methods directly on the raw
collection, not through the wrapper. ``test_mcp_server.py`` facet tests
use ``MagicMock()`` for the collection, which synthesizes attributes on
demand and bypasses MRO entirely. Neither path covers the seam where the
bug lives: ``palace.get_collection() -> EmbeddingCollection -> .method()``.
Three tests pin both the fix and the bug class:
- ``test_facet_counts_forwards_to_inner`` — direct integration through the
wrapper, asserts the inner's recorded call matches.
- ``test_get_all_metadata_forwards_to_inner`` — same shape, plus a sentinel
``get()`` on the inner so a missing forwarder would route to the base
default and pick up the wrong data (observable failure, not silent).
- ``test_wrapper_forwards_all_concrete_basecollection_methods`` — meta-test
that enumerates every concrete public method on ``BaseCollection`` via
``inspect.getmembers`` and asserts each one is explicitly defined on
``EmbeddingCollection``. Catches the bug *class*: any future
``BaseCollection`` method with a concrete default body becomes a CI
failure the moment it's added without a wrapper forwarder, with a message
pointing straight at the file to edit.
Full env-cleared suite: 3205 passed, 20 skipped. ``ruff check`` and
``ruff format --check`` both clean.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KC5Qsknh2zFRtRvVyjXiTA
Documents running MemPalace as a central memory service for a team:
HTTP MCP transport (--transport http with bearer-token auth), a
networked backend (Qdrant via REST, no extra dep; or pgvector), and
optional GPU embedding. Covers the security model (non-loopback token
requirement, Host/Origin DNS-rebinding guard, TLS-in-front), client
connection, and operating notes. Adds the page to the guide sidebar.
Addresses #1877.
* feat(graph): auto-populate the associative graph from mined sessions
Conversation mining never set the `entities` drawer metadata that hallways
consume, so mined sessions produced an empty associative graph (and starved
the entity-navigation / tunnel-recommendation features built on top of it).
Add a no-LLM structural entity extractor and wire it into the convos mine:
- entities: structural-only extractor (author-quoted code spans, URLs, file
paths, qualified identifiers, CamelCase / snake_case symbols). No wordlists,
no NLP models, precision-biased so prose doesn't pollute the graph.
- convo_miner: set `entities` per chunk, and compute hallways after a convos
mine (mirroring the project-file path). Hallways run before the FTS5
validation, which opens a direct sqlite connection that can invalidate the
live Chroma collection handle on some Chroma builds.
- cli: `mempalace hallways` lists the associative graph (CLI parity with the
list_hallways MCP tool).
- tests: extractor precision/ranking, entities metadata at mine time, CLI.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(graph): address review — semicolon safety, leading-underscore snake, negative limit
- entities `_clean`: strip `;` out of tokens so a URL query string or backtick
span can't split the `;`-joined entities metadata field
- entities `_SNAKE`: optional leading/trailing `_?` so `_extract_authored_at`
and similar are matched in plain text (previously only caught via backticks)
- cli `hallways`: clamp `--limit` with max(0, ...) so a negative value shows
nothing instead of slicing from the end
- tests for all three
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Igor Lins e Silva <4753812+igorls@users.noreply.github.com>
* feat(mcp): add since/before date filter to list_drawers (#1128)
mempalace_list_drawers previously filtered only by wing/room. This adds
optional since/before ISO date bounds on filed_at: since is inclusive,
before is exclusive.
The filter runs in Python after the rows are fetched. ChromaDB 1.5.7
rejects string operands for $gte/$lt and filed_at is stored as an ISO
string, so a server-side where comparison is not available; the tool
already collapses and paginates the full result set in Python.
Drawers whose filed_at is missing or unparseable are excluded while a
bound is active, and inverted bounds (since >= before) return a clear
error.
* test: close chromadb clients between tests to fix Windows handle leak (#1128)
chromadb 1.5.7 caches one System per palace path and only frees the
SQLite/HNSW file handles on client.close(); the collection fixture and
the per-test MCP cache reset only dereferenced the client, so handles
leaked across the session. Harmless on POSIX (rmtree unlinks open files),
but on Windows the handles stay locked and accumulate until an HNSW
segment write in a later test's setup fails, which surfaced here as
TestDeleteBySource::test_commit_purges_matching_closets asserting 0 == 2.
Close the client in the collection fixture and in _reset_mcp_cache so the
handles are released between tests.
* test: release backend chromadb clients between tests (#1128)
palace.get_collection() caches one PersistentClient per palace_path on the
process-wide backend singleton and never closes it; sweep, repair and several
CLI tests reach the store through it. chromadb frees the rust-side SQLite/HNSW
file handles only on client.close(), so the handles leak across the whole
session: a 30-palace probe shows ~200 open file descriptors into the palace
tree, dropping to 0 once the clients are closed.
On POSIX the open handles are harmless (rmtree unlinks open files), but on
Windows they stay locked and accumulate until a later test's HNSW segment
write fails ("Failed to apply logs to the hnsw segment writer"), e.g.
test_sweeper.py::TestSweeperTandem::test_sweep_recovers_untaken_message_at_cursor_timestamp.
Drain the cached clients in the autouse _reset_mcp_cache teardown via
close_palace(), which closes each PersistentClient (releasing its handles)
without marking the backend closed so it stays reusable. Complements the
collection-fixture and _client_cache close() added earlier.
* fix(palace): process-wide mine_palace_lock re-entrancy for threaded HTTP transport
The MCP HTTP transport (ThreadingHTTPServer) acquires the long-lived
writer-lease on one thread (_acquire_mcp_writer_lock) but dispatches each
write request on a different worker thread. The lock re-entrancy guard was
thread-local, so write handlers (add_drawer/update_drawer) failed to see the
process-held lease, re-acquired the flock, and self-conflicted with
"palace ... is held by PID <self>". Reads worked (no lock); writes over the
HTTP transport were impossible.
Make the re-entrancy record process-wide (pid-tagged, guarded by a
threading.Lock) so a write from any thread of the process that already holds
the lease passes through. Safe: flock is per-process and HTTP writes are
serialized by _HTTP_REQUEST_LOCK. Preserves fork-safety, same-thread nesting
(miner.mine -> ChromaCollection.upsert), and cross-process protection
(MineAlreadyRunning still raised between processes).
Add cross-thread same-process regression test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(palace): reset lock guard on fork to avoid inherited-locked deadlock
Address review (PR #1859): `_palace_lock_guard` is a threading.Lock, so a child
forked while another thread held it would inherit it locked (the holder thread
is gone in the child) and deadlock on the next acquire. Register an
os.register_at_fork(after_in_child=...) handler that replaces the guard with a
fresh unlocked lock and clears state; the child must reacquire the flock anyway.
Guarded by hasattr(os, "register_at_fork") for Windows.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Igor Lins e Silva <4753812+igorls@users.noreply.github.com>
* feat(convo): preserve authored timestamp from transcripts
Conversation drawers only carried `filed_at` (ingest time), so a bulk
re-mine collapsed every drawer to a single instant and the chronological
signal was lost — even though each Claude Code / Codex JSONL line already
carries an ISO-8601 `timestamp`. The recency-window fallback and any
date-aware consumer then saw ingest order, not when content was written.
- convo_miner: derive `authored_at` (per-file max line `timestamp`) and
store it as drawer metadata; falls back to `filed_at` when absent
- searcher: surface `authored_at` in search results, and break exact
hybrid-score ties toward the more recently authored drawer (ISO strings
sort chronologically; missing dates sort oldest) — benchmark-neutral as
it only reorders exact ties
- tests: cover `_extract_authored_at` (latest wins, skips/tolerates lines
without timestamps, non-jsonl/missing -> None) and the tie-break
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(search): surface authored_at in CLI + backfill for existing data
Completes the authored_at work so the field is visible end-to-end and
existing palaces can adopt it without re-mining.
- layers: CLI `search` output shows an `authored:` date line per result
(peer of the existing date; markdown drawers fall back to filed_at)
- scripts/backfill_authored_at.py: in-place migration that stamps
authored_at on convos drawers from their source transcripts — metadata
only (no re-embedding), idempotent, dry-run by default
- docs/authored-at.md: documents created_at (ingest) vs authored_at
(written) and both backfill paths (in-place / drop-and-recreate)
- tests: backfill integration tests over an ephemeral ChromaDB collection
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(search): address review — non-string timestamp guard + top-level authored_at tiebreak
Two correctness fixes from the PR review:
- _extract_authored_at: only compare when the parsed `timestamp` is a str.
A non-string timestamp on a malformed/foreign JSONL line previously raised
TypeError outside the try and could crash the mine.
- _hybrid_rank: the tie-break read `authored_at` only from nested `metadata`,
but the search_memories path (MCP / Claude Code) carries it at the top level
of each hit — so the tie-break silently no-op'd there. Read both shapes.
- tests: non-string timestamp cases, and a top-level-shape tie-break test
(which fails before this fix).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* style: apply ruff format to authored_at changes
CI ruff format --check flagged 4 files; ruff check already passed.
Formatting only — no behavior change.
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Igor Lins e Silva <4753812+igorls@users.noreply.github.com>
* fix(pgvector): skip document column for metadata-only fetches (#1840 follow-up)
Closes the explicit "separate follow-up to keep this low-risk" callout
in PR #1840's description.
For remote pgvector deployments (TLS over WAN), `mempalace_status` and
every other metadata-only consumer was transferring the full `document`
column over the wire even when nothing read it. A single scroll over a
177K-drawer palace on a 175 ms-RTT link moved ~150 MB of document text
plus ~50 MB of metadata; this PR drops that to ~50 MB.
scroll_rows / _scroll gain `with_document: bool = True`. When False,
SELECT projects NULL::text instead of the document column. Positional
_row parser unchanged (record[1] stays the document slot, just receives
NULL). Existing callers default to True and see byte-for-byte identical
behavior.
PgVectorCollection.get_all_metadata override: where=None path goes
single-scroll with with_document=False. Filtered path falls back to base
to keep _matches_where running on array/object metadata values (same
correctness contract as #1840's filtered-path decision).
Tests:
- Update _FakePgVectorClient.scroll_rows to accept with_document; mirror
the NULL-becomes-empty-string semantics when False
- Update 5 existing scroll_calls assertions to include with_document=True
(unchanged intent)
- test_pgvector_get_all_metadata_skips_document_column: assert exactly
one scroll call with with_document=False
- test_pgvector_get_all_metadata_filtered_falls_back_to_base: assert
filtered path preserves with_document=True
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KC5Qsknh2zFRtRvVyjXiTA
* fix(pgvector): extend with_document=False fast path to filtered get_all_metadata
Per gemini-code-assist review feedback on #1892: _matches_where only reads
metadata, so the where=None vs where=set conditional fall-back was unnecessary.
The filtered path can use the same single-scroll with_document=False fast path
and apply the post-filter locally on metadata dicts — extending the wire-byte
win to every get_all_metadata caller, not just unfiltered ones.
Mirrors the pushdown + local _matches_where pattern already used by _rows
in the same file: pushdown when _requires_local_filter is False, post-filter
in Python otherwise. Same correctness contract as #1840's filtered get path.
Renames test_pgvector_get_all_metadata_filtered_falls_back_to_base to
test_pgvector_get_all_metadata_filtered_uses_fast_path and asserts the new
behavior (with_document=False + pushdown forwards the equality filter to SQL).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KC5Qsknh2zFRtRvVyjXiTA
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* fix(chroma): require SQLite magic header for ChromaBackend.detect() (#1893)
Closes#1893.
ChromaBackend.detect() was returning True for a 0-byte chroma.sqlite3 file
because the check was just os.path.isfile(...). On a palace that has any
other backend marker alongside a stale 0-byte chroma.sqlite3,
resolve_backend_name then raises BackendMismatchError and the palace becomes
unopenable until the user manually rm's the empty file.
The 0-byte file appears as a side effect of any sqlite3.connect() on a
missing path — Python creates the file immediately but writes the SQLite
header only on the first statement. So any code path that touches the
chroma.sqlite3 path with bare sqlite3.connect(), including chromadb's own
PersistentClient lazy-init (see the comment at backends/chroma.py:2052),
can leave a 0-byte artifact behind.
Fix: detect() now reads the first 16 bytes and compares to the SQLite
magic prefix b"SQLite format 3\x00" instead of relying on file presence
alone. One extra open() + 16-byte read; detect() isn't a hot path.
Properties:
- Rejects 0-byte files (the symptom #1893 is about).
- Rejects non-SQLite garbage at the canonical path (partial writes, etc.).
- Doesn't false-negative on real chroma palaces: any chroma palace whose
PersistentClient has done any work has the magic header on disk
(verified — CREATE TABLE is enough to land the header).
- Doesn't couple detect() to chroma's specific schema; the magic header
is stable across chromadb releases.
Test sweep: many test files used (chroma.sqlite3).touch() or
.write_bytes(b"") as a "fake palace" shortcut, exploiting the loose
isfile() check (one such site even had the comment "# pass the isfile
guard"). After this change, those stand-ins no longer register as chroma
palaces. Introduced tests/_chroma_palace_helper.py::make_minimal_chroma_sqlite
following the existing _backend_conformance.py precedent, and updated 15
call sites across 8 test files to use it. The existing
test_chroma_detect_matches_palace_with_chroma_sqlite (which encoded the
buggy semantics with write_bytes(b"")) is renamed to
test_chroma_detect_matches_palace_with_sqlite_header and now writes a
real SQLite database via the helper. Added two new tests for the
rejection paths (empty file, non-SQLite garbage).
Full env-cleared suite: 3137 passed, 20 skipped, 0 failed. ruff check
and ruff format --check both clean.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KC5Qsknh2zFRtRvVyjXiTA
* fix(sqlite_exact): require SQLite magic header for SQLiteExactBackend.detect()
Per gemini-code-assist review on #1892 PR #1896: SQLiteExactBackend has the
same os.path.isfile() detection pattern as ChromaBackend did, with the same
0-byte-file vulnerability. Mirrors the chroma fix for repo-wide consistency.
- SQLiteExactBackend.detect() now does the same 16-byte SQLite magic-prefix
check as ChromaBackend.detect().
- _chroma_palace_helper.py: factored its body into a private
_write_minimal_sqlite_file() and gained a sibling
make_minimal_sqlite_exact_sqlite() for the sqlite_exact filename. No churn
to any existing chroma call sites.
- test_sqlite_exact_backend.py:426 (the one site that wrote b"" for
sqlite_exact.sqlite3) updated to use the new helper.
- Three new tests in test_sqlite_exact_backend.py mirror the chroma trio:
matches with valid header, rejects empty file, rejects non-SQLite garbage.
Full env-cleared suite: 3140 passed, 20 skipped, 0 failed.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KC5Qsknh2zFRtRvVyjXiTA
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
On Windows, Path.read_text() and open(path, 'a') use locale encoding
(GBK on Chinese-locale systems) before PEP 686 / Python 3.15. A valid
UTF-8 .gitignore with non-ASCII comments crashes
_ensure_mempalace_files_gitignored() with UnicodeDecodeError, which
aborts 'mempalace init' on Windows for any user whose .gitignore
contains non-ASCII text.
Force encoding='utf-8' on both read and append, with errors='replace'
on read as a defensive fallback for legacy mixed-encoding files.
Co-authored-by: ALaDingAhmad <16530935@qq.com>
Co-authored-by: Igor Lins e Silva <4753812+igorls@users.noreply.github.com>
L1's generate() scored drawers by importance/emotional_weight/weight, and
the docstring promised "prefer high importance, recent filing". But no
ingest path (miner, convo_miner, diary, add_drawer) writes any of those
fields, so the sort collapsed to insertion order (oldest first) and
recency was never consulted. A scoped `wake-up --wing X` therefore
surfaced the *oldest* moments: the opposite of useful.
Add filed_at (present on every drawer, ISO-8601, lexically chronological)
as the secondary sort key. Importance stays primary for the day a scoring
pass populates it; filed_at is the effective signal today, making the
"recent filing" half of the promise true with data already present.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Igor Lins e Silva <4753812+igorls@users.noreply.github.com>
* fix(mcp): stop clobbering host app root logger at import (#1860)
_init_logging() ran at import and called logging.basicConfig(force=True),
resetting the root logger's level, format, and handlers unconditionally. An
app that configured logging before importing mempalace.mcp_server lost its
setup: a host on DEBUG dropped to INFO, custom formatters and handlers were
replaced.
force=True existed (#1495) only to keep MEMPALACE_LOG_FILE working when root
already had handlers. This keeps that contract without the reset: configure
root only when it is unconfigured (standalone); otherwise attach a
mempalace-filtered file handler additively and leave the host's config alone.
Adds _MempalaceLogFilter so the file captures every mempalace logger (the
dotted mempalace.* family plus the flat mempalace_* names) and nothing else.
* fix(mcp): survive importlib.reload and pin file log format (#1860)
Addresses review on #1885.
Restore _logging_configured from globals() so the idempotency guard survives
importlib.reload: a reload re-executes the module body, and a plain reset would
let _init_logging() stack a duplicate file handler on root.
Set an explicit "%(message)s" formatter on the file handler so the embedded
path does not depend on logging's default formatter (which already renders the
same, but is now pinned and identical to the standalone path).
Adds a reload regression test and a format-pin assertion.
Concurrent killed-mid-write mines can leave embedding_fulltext_search in a
malformed-inverted-index state that fails PRAGMA quick_check while the
underlying rows stay intact (integrity_check ok). The repair preflight then
hard-aborts before reaching the FTS5 rebuild step, so `mempalace repair`
refuses to run and full-text search stays broken — the exact loop #1596
reports. The MineValidationError banner even promises "repair --yes rebuilds
the FTS5 virtual table automatically," which the preflight abort made false.
Add maybe_autoheal_fts5_index(): when every quick_check error is an isolated
"malformed inverted index for FTS5 table" failure, rebuild the index in place
from the intact embedding_fulltext_search_content table
(INSERT ... VALUES('rebuild')) under mine_palace_lock, then re-run quick_check.
The rebuild touches no drawer rows. Wired into both repair preflights
(rebuild_index and cli cmd_repair). Any non-FTS5 error in the set, a lock held
by a live mine, or a rebuild that does not clear quick_check leaves the errors
unchanged so the caller still aborts with the recovery banner — broader
corruption is never silently rebuilt over.
An empty link_lists.bin is not corruption on its own: hnswlib stores the
layer-0 graph inside data_level0.bin and only writes link_lists.bin for
elements promoted to level > 0. A small/low-fanout index where every
element stays on layer 0 serializes an empty link_lists.bin and loads
fine. Flagging that shape as corrupt produced a self-perpetuating
quarantine loop — repair rebuilt the byte-identical all-layer-0 segment,
the next cold start re-quarantined it, accumulating drift dirs (221 MB in
the reported case) with no ingestion involved.
Use the persist-completion marker as the discriminator instead. ChromaDB
writes index_metadata.pickle last, so an intact pickle envelope proves the
flush finished and the empty link_lists.bin is the legitimate all-layer-0
shape. Only treat an empty link_lists.bin as a partial flush when there is
real payload AND no completion marker (absent or truncated pickle). The
#1457 partial-flush protection (real payload, no/truncated marker) is
preserved; the byte-sniff is factored into _hnsw_metadata_marker_intact
and reused by _segment_appears_healthy.
Also fixes the related single-writer stale-quarantine false positive
(#1564), which shares this all-layer-0 root cause.
The non-daemon synchronous mine fallback in _mine_sync() spawned the
mine subprocess without CREATE_NO_WINDOW, flashing a visible console
window on every PreCompact fire on Windows. The async paths
(_spawn_mine, _desktop_toast) already pass it via _detached_popen_kwargs();
this sync path was missed. getattr(..., 0) is a no-op off-Windows.
Fixes#1862
Co-authored-by: David Finkelstein <david@finkelstein.us>
* fix: sanitize wing slug for project dirs with special characters
Project folders containing characters outside sanitize_name's set
(e.g. a leading '+') leaked into the derived wing name, producing names
like 'wing_+project' that config.sanitize_name rejects, silently
breaking diary auto-save for that project.
Add _safe_wing_slug(): collapse non-word runs to '_', trim, and fall
back to 'sessions' when a name reduces to nothing. Route the three
wing-derivation sites through it.
Tests: unit cases for the helper plus a hypothesis property test
asserting wing_<slug> always passes sanitize_name for any input.
* fix: preserve dots and apostrophes in wing slug for backward compatibility
The first pass collapsed every non-word character (including dot and
apostrophe) to underscore, renaming existing valid wings — e.g. my.app
became wing_my_app — which would orphan diary entries already filed under
the old name. Keep dot and apostrophe (both accepted by sanitize_name),
collapse consecutive dots to avoid the path-traversal rejection, and trim
edge separators.
Add backward-compatibility tests for previously-valid names plus a
double-dot collapse test.
* fix: cap wing slug length to stay within sanitize_name's limit
sanitize_name rejects names over 128 characters, so a very long project
directory name would produce a wing name that fails validation,
re-triggering the silent auto-save break this PR fixes. Truncate the slug
to 120 chars (the wing_ prefix keeps the total under 128).
Widen the hypothesis property test to max_size=300 so it exercises the
length path, and add an explicit truncation test.
Addresses gemini-code-assist review feedback on PR #1852.
---------
Co-authored-by: Ivan Antsimonau <ivan.antsimonau@katim.com>
daemon.py:_detached_kwargs was the last production spawn site still using DETACHED_PROCESS. Swap it to CREATE_NO_WINDOW, matching the hook miner's _detached_popen_kwargs fixed in #1848 — the dedicated follow-up the review bot asked for. `grep -rn DETACHED_PROCESS mempalace/` now returns zero production hits.
Survivability is unchanged: CREATE_BREAKAWAY_FROM_JOB (escapes the parent Job Object's kill-on-close) plus the daemon never being attached to the launching console carry survive-terminal-close; CREATE_NEW_PROCESS_GROUP (also kept) isolates Ctrl-C/Break. CREATE_NO_WINDOW is ignored when OR'd with DETACHED_PROCESS, so this replaces the flag rather than adding it. The daemon already redirects stdout/stderr to daemon.log and reads no stdin, so it needs no console.
Adds the first tests for _detached_kwargs (posix + windows, cross-platform monkeypatch of the Windows-only flag constants, mirroring the #1848 hooks_cli tests).
The original commit printed SKIP for oversized files to stdout but the
sibling SKIP for symlinks in the same scan_project / scan_convos already
went to stderr. Align the new line with that convention.
Also adds a SKIP-with-error log for the except OSError arm right below the
size check. Files whose stat() raises (permission denied, racing delete,
broken symlink that survived the earlier is_symlink check) were the same
bug class as the silent oversize drop.
Tests switched from captured.out to .err and tightened to the full
template; new test covers the OSError arm via a selective Path.stat
monkeypatch with a follow_symlinks gate for Python 3.10+.
Both miner.py and convo_miner.py silently skip files larger than the
10 MB limit with a bare continue. This is especially painful for
conversation mining where long Claude/ChatGPT exports routinely
exceed 10 MB and vanish with no trace.
Print a SKIP warning per oversized file, matching the existing format
in split_mega_files.py.
The write-ahead log gained its own module in v3.5.0 but sat at 82% coverage;
the uncovered lines were exactly the failure/guard branches that uphold its
contracts: the cache-hit early return, the restricted-FS chmod/mkdir swallow
paths, and the promise that a WAL write failure is logged and never crashes
the calling tool. Add five tests covering those branches plus the non-string
redaction marker, bringing mempalace/wal.py to 100% and locking the
crash-safety guarantees against regression. Test-only; no production change.
- ruff format llm_client.py and miner.py (lint job)
- _copy_file_no_follow: close src fd if the dst open fails (no leak), and
route the rebuild restore through it so backup + restore share one
no-follow/regular-file path
- update repair tests to assert the unified hardened copy instead of the
removed shutil.copy2 calls; backup paths are now timestamped
- update normalize large-file test to stub fstat (size is checked on the
open fd, not via a pre-open os.path.getsize)
Bump version to 3.5.0 across version.py, pyproject.toml, the Claude/Codex
plugin manifests, the README badge, and uv.lock. Refresh the "N MCP tools"
prose from 34 to 35 (delete_by_source #1729 and checkpoint #1851 each added a
tool). Add the 3.5.0 CHANGELOG entry.
ChromaDB's rust HNSW core intermittently fails compaction on Windows with
"Failed to apply logs to the hnsw segment writer" during add/update — a
long-standing, non-reproducible-on-Linux/macOS flake that hits different tests
(test_migrate_wings, test_closets) across unrelated commits and has been
turning otherwise-green release/CI runs red at random.
Add pytest-rerunfailures and wire `--reruns 2 --only-rerun "Failed to apply
logs to the hnsw segment writer"` into the test-windows job only. The
--only-rerun scope means a real, deterministic failure still fails on the first
run; only this specific transient native-dependency error is retried. The
Linux and macOS jobs deliberately keep zero reruns so genuine regressions
surface there loudly.
The opt-in HTTP transport reuses the stdio dispatcher and binds loopback by
default, but /mcp was unauthenticated with no protection against a malicious
web page reaching a DNS-rebound localhost server, and its tests reached for
Starlette/uvicorn (not project deps) so they were silently skipped in CI —
the production _serve_http handler had zero coverage.
Hardening:
- Pin the Host header to loopback literals + the bound host on a loopback bind
(DNS-rebinding defense); relaxed for a deliberately non-loopback bind, which
is the operator's call and may sit behind a Host-rewriting proxy.
- Reject any browser Origin that isn't a loopback origin (rebinding/SSRF guard);
non-browser MCP clients omit Origin and are unaffected.
- Optional bearer token via MEMPALACE_MCP_HTTP_TOKEN (constant-time compare);
required on /mcp, never on /healthz so liveness probes work credential-free.
- Warn loudly when bound to a non-loopback host (palace reachable from network).
Testability:
- Split _build_http_server() out of _serve_http() so tests bind 127.0.0.1:0 and
drive the real handler over a loopback socket via stdlib http.client.
- Replace the skipped Starlette reimplementation with 12 tests covering dispatch,
initialize, /healthz, 404, parse-error, the 16 MiB cap, notification 202, and
the Host/Origin/token rejections — no third-party deps.
delete_by_source removed only the drawers, leaving the matching closets
(the AAAK index layer, keyed independently by source_file) behind as stale
pointers at the now-deleted source. Mirror the closet-purge step used by
sync_palace / purge_file_closets: after the drawer delete, best-effort purge
the closets via push-down delete(where=...) so it survives large palaces and
can never abort an already-committed drawer delete.
Dry run now also reports closet_match_count so the caller sees the full blast
radius; commit reports closets_deleted. Adds tests that seed the closet
collection directly (tool_add_drawer doesn't build closets) and assert the
matching closets are purged on commit and counted on dry run.
The save-hook followup now drives a single mempalace_checkpoint call, so
test_threshold_emits_followup_message must assert that tool name instead
of the old add_drawer/check_duplicate/diary_write trio. Fixes the
test-macos / test-linux CI failures on this branch.
Co-authored-by: Cursor <cursoragent@cursor.com>
Address PR review: guard untrusted MCP client payloads in
mempalace_checkpoint so a single malformed item cannot raise deep in
sanitization and abort the whole batch.
- coerce dedup_threshold to float
- require wing/room/content to be non-empty strings (skip + record error)
- validate the diary object and entry type, recording errors instead of
silently ignoring a malformed diary
On a genuine dedup-check error we still file the drawer rather than skip:
verbatim recall is the priority and add_drawer's idempotency blocks exact
duplicates. Adds tests for the non-string, dedup-error, and malformed-diary
paths.
Co-authored-by: Cursor <cursoragent@cursor.com>
Collapse the Cursor auto-save sequence (check_duplicate Nx + add_drawer
Nx + diary_write 1x) into a single mempalace_checkpoint MCP call so the
host UI renders one tool-call card and keeps its spinner up for the whole
save. The new tool reuses the existing single-item handlers, so semantic
dedup, idempotency, and verbatim guarantees are unchanged.
- mcp_server.py: add tool_checkpoint + register mempalace_checkpoint
- service.py: classify mempalace_checkpoint as a write tool
- cursor save hook: followup now drives one mempalace_checkpoint call
- docs: new mcp-tools.md section, help.md entry, 33 -> 34 tool count sweep
- tests: checkpoint add/dedup/malformed/registry + classify_tool
Co-authored-by: Cursor <cursoragent@cursor.com>
Gemini review on PR #1849: the optional palace backup step used the
Unix-only `cp -a`, which fails on Windows. MemPalace ships on win32, so
add a PowerShell `Copy-Item` alternative alongside the macOS/Linux form
and note that `--archive-existing` already moves the old palace aside.
Co-authored-by: Cursor <cursoragent@cursor.com>
A diverged HNSW index (for example after a failed chromadb compaction)
leaves the drawer rows intact in chroma.sqlite3 but the vector index out
of sync. Re-mining to recover silently drops drawers added through the
MCP server and diary entries, which have no source file.
- repair-status now recommends `mempalace repair --mode from-sqlite
--archive-existing` when DIVERGED, instead of the generic `mempalace
repair`, and explains why re-mining loses data.
- The shared recall protocol and the recall skills (Cursor + Claude
plugin) document the compactor / "Not connected" recovery path:
stop the server, rebuild from SQLite, verify, restart — never repair
in-process from the agent.
Complements #1847 (legacy repair error messages); does not duplicate it.
Does not close#1843 — MCP reconnect resilience and honest add_drawer
write signalling remain open.
Co-authored-by: Cursor <cursoragent@cursor.com>
When the chromadb compactor cannot apply the WAL into the drawers HNSW
segment (InternalError: Failed to apply logs to the hnsw segment writer),
the legacy repair paths fail on their first Collection.count() read and
advise re-mining from source files. The drawer rows are intact in
chroma.sqlite3, so repair --mode from-sqlite rebuilds them; re-mining
silently drops drawers added via the MCP server and diary entries that
have no source file.
Both legacy read-failure sites (cmd_repair and rebuild_index) now emit
shared guidance pointing at the from-sqlite recovery, worded conditionally
so it also covers a live server or mine still holding the palace open.
Co-Authored-By: undeadindustries <9536461+undeadindustries@users.noreply.github.com>
A JSON number or boolean passed for source_file is not coerced by the
string schema type, so it reached _sanitize_optional_source_file and
raised AttributeError from .strip() rather than a clean validation error.
Add an isinstance guard that raises ValueError, which tool_search returns
as a structured error. Regression test added.
Expose source_file alongside wing/room on mempalace_search. build_where_filter
generalizes to 0/1/2+ clauses and the filter threads through the main vector
path, the index-mismatch fallback, the vector-disabled BM25/SQLite path, and
the union lexical path so it never silently no-ops. Matching is on the exact
full stored value; results now expose source_path (the full path) for round
tripping, since the displayed source_file is a basename. The MCP schema gains
the source_file property and a path-tolerant sanitizer rejects null bytes,
lone surrogates, and overlong values.
Fixes#1815
Co-Authored-By: rendigua2025-gif <253093224+rendigua2025-gif@users.noreply.github.com>
SQLiteExactCollection.get(limit, offset) fetched the whole collection via
_rows() (SELECT ... FROM documents ORDER BY rowid, no LIMIT/OFFSET) and sliced
in Python, so every paginating caller (prefetch_mined_set, status, exporter,
migrate, dedup, sync, ...) re-scanned the entire table per page, making the
sweep O(N^2) in rows materialized.
Push LIMIT/OFFSET into the scan on the unfiltered page (no ids/where/
where_document and non-negative bounds); filtered, id, and negative pages keep
the full-scan plus Python-slice path so the post-filter still runs first. SQLite
requires a LIMIT before OFFSET, so an offset-only page uses LIMIT -1. ORDER BY
rowid keeps pages stable.
A lone UTF-16 surrogate (U+D800-U+DFFF) in transcript content has no UTF-8
encoding, so pgvector's bulk upsert_rows makes psycopg raise UnicodeEncodeError
and the whole mine aborts, leaving later files unmined.
Apply config.strip_lone_surrogates (-> U+FFFD) to id, document, and the
serialized metadata JSON in upsert_rows. json.dumps(ensure_ascii=False) leaves a
metadata surrogate raw in the string, so one pass over the serialized JSON covers
it; NUL, by contrast, json-escapes and must be stripped before serialization
(see #1829). Replace rather than drop, matching ChromaDB's document handling.
Verified end to end against live Postgres + pgvector: before, a surrogate in
document or metadata aborts the mine; after, it ingests and round-trips as U+FFFD.
Fixes#1833
PgVectorCollection.get(limit=, offset=) ignored pagination at the SQL
layer: scroll_rows ran SELECT ... WHERE <where> with no LIMIT/OFFSET, so
get() fetched the whole table and sliced in Python. prefetch_mined_set
pages the whole palace on every mine, so mining was O(N^2) in rows
transferred and Python objects built as the palace grows; every other
paginating caller (exporter, migrate, repair, hallways, closet_llm,
miner, palace_graph) paid the same cost.
Push LIMIT/OFFSET into scroll_rows/_scroll with ORDER BY id (the primary
key) for stable offset pagination. get() uses the pushed path only for an
unfiltered page (no ids, no where/where_document, non-negative bounds); a
filtered get keeps the full-scan path because the metadata @> ... pushdown
is broader than the exact _matches_where re-filter for array/object
values, so that re-filter must run before pagination. Full-scroll callers
pass no bound, so their SQL is unchanged.
Co-authored-by: hrabbach <181709360+hrabbach@users.noreply.github.com>
The graph_stats sqlite reader (#1837) and the sqlite_read_uri encoding fix
(#1838) landed in parallel, so _sqlite_graph_stats was the one reader left on
the naive f"file:{db_path}?mode=ro" construction that mis-parses paths with
spaces/special chars. Convert it too — now every read-only sqlite reader
percent-encodes its path. sqlite_read_uri is already imported in mcp_server
from #1838, so this is a call-site-only change.
sqlite3.connect(f"file:{db_path}?mode=ro", uri=True) mis-parses paths
containing spaces or other URI-reserved characters — common in real home
directories (a Windows "First Last" user folder, many macOS paths), and made
worse by Windows backslashes. The database silently fails to open and the
read-only fast paths fall back (or error) on those machines.
Add config.sqlite_read_uri(), which percent-encodes the path via
urllib.request.pathname2url (lazy-imported to keep config import light), and
route every read-only sqlite reader through it:
- mcp_server._tool_status_via_sqlite
- searcher BM25 sqlite fallback
- repair (status / scan / max-seq read paths)
- backends/chroma (5 readers: counts, wing/room tally, id maps, etc.)
All previously used the same naive f-string construction. Surfaced as a
gemini-code-assist review note on #1837.
- Soft-fallback on any exception, not just sqlite3.Error, so an unexpected
schema shape tripping the reconstruction degrades to build_graph() instead
of raising — matching the sibling sqlite fast paths (Copilot).
- Guard an empty/None _config.palace_path before building db_path (Gemini).
- Test: tripwire _get_collection in addition to graph_stats, directly
asserting the fast path never opens the chroma client / cold-loads HNSW
(Copilot).
tool_graph_stats built the whole palace graph via build_graph(), which pages
every metadata row (col.get limit/offset) and cold-loads the HNSW index — the
remaining overview-tool timeout from #1379 (#1836 fixed status / list_wings /
list_rooms / get_taxonomy but deliberately left graph_stats out, as it builds
an in-memory graph rather than a flat tally).
Add _sqlite_graph_stats(): one GROUP BY room, wing, hall over chroma.sqlite3,
reconstructing build_graph's room_data and the same stats (total_rooms,
tunnel_rooms, total_edges, rooms_per_wing, top_tunnels) with the same
per-drawer filter (room present, != "general", wing present) and edge
semantics (C(wings, 2) * halls per multi-wing room). Same _is_chroma_backend()
guard + client-path fallback as the #1748 overview tools.
Test seeds a real chroma palace mirroring the build_graph parity case in
test_palace_graph, with a tripwire on graph_stats proving the fast path runs
and that "general"/wing-less drawers are excluded. Idea adapted from #1381's
_sqlite_graph_stats.
#1748: normalize the sqlite fast path's "?" COALESCE placeholder (and None)
back to "unknown" inside _sqlite_taxonomy, so drawers missing wing/room
metadata keep the client path's output contract — no observable API change
for MCP clients on legacy/partial drawers.
#1068: invoke the parent embedder build via super().model instead of reaching
into cached_property's .func attribute, so the uncapped/fallback path survives
chromadb changing `model` to a plain @property or other descriptor.
PostgreSQL cannot store NUL (0x00) in text or jsonb. On the pgvector write path
a NUL in `document` is rejected by psycopg ("PostgreSQL text fields cannot
contain NUL (0x00) bytes") and a NUL in `metadata` becomes a JSON unicode escape
the jsonb cast rejects ("unsupported Unicode escape sequence"). `_execute`
re-wraps either as BackendError and `_mine_impl` re-raises, so the whole mine
exits non-zero and every file after the offending one is left unmined. ChromaDB,
SQLite, and Qdrant store the byte verbatim, so only pgvector hard-fails.
Add a recursive `_strip_nul` helper and apply it to id, document, and metadata
in `_PgVectorClient.upsert_rows`, mirroring the backend-layer sanitization
`_sanitize_documents_for_chromadb` already does for lone surrogates on the same
bulk-ingest paths. ids are SHA-256 hashes and metadata keys are fixed field
names, so the id and key passes are no-ops in practice; only transcript-derived
values change.
Co-authored-by: hrabbach <181709360+hrabbach@users.noreply.github.com>
tool_status / list_wings / list_rooms / get_taxonomy paged the entire
collection metadata through the chroma client (`_fetch_all_metadata`, a
1000-row offset loop), which cold-loads the HNSW index and materializes
hundreds of MB of dicts. On six-figure palaces these exceed the MCP host
tool-call limit (180k drawers ~3-4 min; 349k times out at 120-240s). The 5s
metadata cache only dedups repeat calls — it does not stop the cold-call
timeout.
A correct single-query SQL cross-tab already exists
(`backends.chroma._sqlite_wing_room_counts`) and is already the CLI default
(`miner.status`), but the MCP tools never used it — and the MCP-side sqlite
reader only ran behind the `vector_disabled` recovery path.
Add `_sqlite_taxonomy()` (guards on `_is_chroma_backend()`, returns None to
fall back) and wire it as the default path into all four overview tools. They
now answer from one GROUP BY without touching HNSW. Non-chroma backends
(qdrant, sqlite_exact) and unbootstrapped/legacy layouts fall back to the
existing client path unchanged.
graph_stats (also named in #1379) builds an in-memory graph via build_graph()
and needs its own treatment — tracked separately.
ChromaDB's ONNX embedder builds its InferenceSession without a thread cap, so
ORT's intra-op pool defaults to the physical core count. OMP_NUM_THREADS is
inert against it (ORT owns its own pool), so a background `mempalace mine`
pins 4-5 cores and stacked Stop-hook fires turn the machine into a thermal
event.
Add an `embedding_threads` config knob (env MEMPALACE_EMBEDDING_THREADS or
config.json). Unset/"auto" caps the intra-op pool at half the logical CPUs so
a fresh install stays usable out of the box; a positive integer sets an exact
count; 0/negative leaves ORT uncapped for users who want max throughput.
The cap is applied via SessionOptions at session construction:
- `_MempalaceONNX` (default minilm) overrides the `model` cached_property to
rebuild the session the same way upstream does plus the cap, falling back to
upstream's uncapped build if chromadb internals shift.
- `EmbeddinggemmaONNX` builds its session through the shared
`_intra_op_session_options()` helper.
Five fixes from the Copilot review of the merged daemon PR:
1. Privacy: the queue DB's SQLite WAL/SHM sidecars hold un-checkpointed
verbatim payloads but were created with the caller's umask. Set the
owner-only umask in run_server BEFORE DaemonRuntime builds the QueueStore
(not only once the HTTP server starts), and harden any existing sidecars in
QueueStore._init_db as defense-in-depth.
2. DoS guard: reject a negative Content-Length in the request reader.
rfile.read(-1) would block until the client disconnects and bypass the
MAX_BODY_BYTES cap.
3. Side effects: extract _wal_log (+ _ensure_wal, _WAL_FILE, _WAL_REDACT_KEYS)
into a new side-effect-free mempalace/wal.py. The CLI sync path and the
daemon service layer obtained _wal_log via `from .mcp_server import _wal_log`,
which runs mcp_server's import-time stdio protection (os.dup2(2, 1);
sys.stdout = sys.stderr) in a non-MCP process and misroutes operator output.
mcp_server/cli/service now import from mempalace.wal.
4. Correctness: run_mcp_tool treated any dict as success. Write tools that
return a bare {"error": ...} (e.g. tool_create_tunnel/tool_delete_tunnel
validation) were recorded as succeeded; now the "error" key infers failure.
5. Hook budget: get_client_if_running()/health() take an explicit timeout, and
the hook "is the daemon up?" precheck uses a short HOOK_PROBE_TIMEOUT (0.5s)
so a wedged daemon can't stall the hook for the default 5s.
Adds tests/test_wal.py (import isolation + redaction) and daemon tests for the
umask ordering, negative Content-Length, run_mcp_tool error inference, and the
short probe timeout.
Short sessions that exit cleanly below SAVE_INTERVAL and without a PreCompact
were never saved. Add a SessionEnd hook that takes one final flush.
Claude Code budgets SessionEnd hooks at 1.5s and a plugin-provided timeout
cannot raise it, and a cold mempalace start exceeds that, so the wrapper
backgrounds the work and returns immediately; the detached child completes the
transcript ingest, project mine, and diary checkpoint after the session exits.
The handler validates transcript_path through _validate_transcript_path before
any ingest or diary write, so a traversal or wrong-suffix path is rejected while
the independent project mine still runs.
Adds hook_session_end, both shell wrappers, the plugin hooks.json entry, the
session-end CLI choice, and focused tests.
(cherry picked from commit 10e1450e04fc7cec72984ab7442d3b4fca1490e8)
HTTPServer.server_bind() resolves server_name via socket.getfqdn(host). For the
daemon's 127.0.0.1 bind that lookup is pointless, and on a host with slow or
absent reverse DNS it blocks startup until the resolver times out (~30s) — which
looks exactly like the daemon never coming up. This is why the first daemon
HTTP-lifecycle test timed out on the macOS CI runner (httpd_bound=False after
30s) while every later one bound in seconds once the OS had cached the negative
lookup. Bind via TCPServer directly and set server_name from the literal host.
_pid_alive used os.kill(pid, 0) as an existence check. On Windows signal 0
is signal.CTRL_C_EVENT, so Python routes it to GenerateConsoleCtrlEvent and
sends a console Ctrl-C to the target's process group rather than probing the
pid. DaemonClient polls a same-process endpoint during startup, so on a CI
runner with an attached console that Ctrl-C was delivered back to the
interpreter as a spurious KeyboardInterrupt — the Windows CI hang that
interrupted the suite at the first daemon HTTP-lifecycle test (socket.py
recv). Probe via the Win32 OpenProcess/WaitForSingleObject handle API instead,
which has no signalling side effects. This is also a real Windows production
bug, not just a test artifact.
Tests:
- Skip the two owner-only (0600) permission tests on Windows: os.chmod cannot
represent POSIX mode bits there (files report 0o666); the daemon relies on
user-profile ACLs on Windows.
- _start_server now captures and re-surfaces a run_server thread crash instead
of spinning for 30s and failing with a bare assert (diagnoses the macOS
startup flake).
- Add a regression test asserting _pid_alive is correct and emits no console
control event when hammered like the poll loop.
- Remove the temporary win32 exit-hang diagnostic fixture from conftest now
that the root cause is fixed.
The Windows CI run passes all 666 tests then hangs at interpreter shutdown
(KeyboardInterrupt at socket.py:723) until the runner kills it. All daemon
lifecycle tests assert their server threads died, so the hang is a different
non-daemon thread blocked on a socket — not the daemon server thread. CI
round-trips can't show which thread it is.
Add a win32-only session fixture that:
- arms faulthandler.dump_traceback_later(130s) to print every thread's stack
to stderr once the hang has run a while, and
- prints every live thread (name + daemon flag) at session teardown — a
non-daemon thread present there is the shutdown blocker.
Gated to sys.platform == 'win32' so Linux/macOS CI see no extra output. Remove
once the Windows hang is fixed.
DaemonClient.request now uses a no-proxy opener (build_opener(ProxyHandler({})))
instead of urllib.urlopen. The daemon is always on 127.0.0.1, so a request must
never go through an HTTP proxy — this is the correct production choice. It also
bypasses urllib's proxy discovery (macOS _scproxy via SystemConfiguration), which
runs on the first request to any host and is NOT bounded by the per-request
timeout: on a CI runner with no network it hangs for tens of seconds, which looked
exactly like the daemon never came up (test_daemon_http_lifecycle_executes_job
timed out at 30.18s). With the no-proxy opener the lifecycle runs in 0.78s and no
server thread is leaked — which also removes the timing skew that made the
sqlite_exact concurrent-connection test flake on macOS CI.
The leaked server thread was also the Windows exit-hang root cause: a slow/failed
client.shutdown() POST left serve_forever running, and the interpreter blocked on
the open listening socket at process exit. Tests now capture the httpd run_server
creates (by subclassing daemon.ThreadingHTTPServer) and force httpd.shutdown() +
server_close() from the test thread if the normal shutdown path leaves the thread
alive, asserting the thread died so a leak becomes a visible failure instead of
a silent exit hang.
CI was red on all three platforms for the daemon-mode draft PR. Root causes
and fixes:
- Linux 3.9 collection error: `_submit_daemon_job`'s `dedupe_key: str | None`
parameter annotation is evaluated at def time, and hooks_cli.py has no
`from __future__ import annotations` — `str | None` raises TypeError on 3.9.
Reverted to `dedupe_key: str = None` (the original, 3.9-safe). The other
`int | None` in the file is a function-local annotation, which is never
evaluated, so it was never the problem.
- macOS/Windows daemon lifecycle flakes: the 3 HTTP-lifecycle tests failed at
the 10s readiness deadline on contended CI runners (localhost bind is
sub-second locally but took ~5s when it passed on the macOS fleet, >10s when
it didn't), and because the server thread never shuts down on timeout,
run_server's `os.environ["MEMPALACE_PALACE_PATH"]` + `os.umask(0o077)`
mutations leaked into the rest of the suite — poisoning every later test that
reads MempalaceConfig().palace_path (the 60+ test_mcp_server cascade on macOS;
the at-exit socket hang → SIGINT on Windows). Bumped the readiness deadline
to 30s and added a module-scoped snapshot + autouse fixture in test_daemon.py
that force-restores the env + umask to the pre-suite baseline after every
daemon test, so a leaked server thread can't poison other test files.
Gemini review comments (fixed in code, no thread replies per convention):
- daemon.py `_connect()` was a bare `sqlite3.connect` whose `with`-block only
managed the transaction, not the connection — an unbounded FD leak in a
long-lived daemon running thousands of jobs (also the source of the Windows
"unclosed database" ResourceWarning noise). Converted to a closing
@contextlib.contextmanager.
- `QueueStore.finish()` gained `only_if_running`; `_safe_finish` passes it so a
late worker finish can't overwrite a shutdown-cancelled job back to
succeeded/failed — removes the reliance on process-exit timing.
- `DaemonClient.request` wraps the final `json.loads` in try/except
JSONDecodeError → DaemonError, so a non-JSON 2xx response surfaces as a
structured error instead of a bare JSONDecodeError.
- test_sync.py: removed the module-level `import mempalace.mcp_server` and moved
the stdout-rebinding side effect into an autouse fixture scoped to
TestServiceRunSyncReport, so the embedder/Chroma import chain is no longer
forced at collection time for the existing sync tests.
Coverage: added focused happy-path tests for service.run_sync early-returns,
run_mine backend application + invalid mode, execute_job kind dispatch,
run_diary_write arg forwarding, run_mcp_tool write-tool dispatch, and
print_job_result — lifts service.py from 57% to 85% so the new files
(service 85%, daemon 80%) don't drag the total below the 80% CI gate now that
the daemon tests complete and the gate is actually evaluated.
- New mempalace/daemon.py: long-lived localhost HTTP server (127.0.0.1) with a
SQLite WAL job queue, single worker thread, bearer-token auth, and owner-only
file perms (0600/0700) on queue DB, token, endpoint, and log.
- New mempalace/service.py: transport-neutral job execution surface shared by the
daemon, with per-job env isolation so one job's backend/palace switch cannot
leak into the next. mcp_tool is allowlisted to write-classified tools only.
- Crash recovery re-queues jobs left 'running' by a killed daemon; jobs that
already exhausted MAX_ATTEMPTS are dead-lettered to 'failed' instead of being
retried (non-idempotent diary_write would otherwise duplicate verbatim
content on every restart).
- Bounded retention prunes terminal jobs older than 7 days
(MEMPALACE_DAEMON_RETENTION_DAYS); queued/running jobs are never touched so a
crash mid-prune cannot drop in-flight work.
- CLI: --daemon/--background on mine/sync submit to the queue; new
`mempalace daemon {start,stop,status,jobs,wait}` subcommand. Strictly opt-in:
no flag, env, or config means no daemon and no behavior change.
- Hooks opt in via MEMPALACE_HOOKS_DAEMON or config hooks.daemon; when the daemon
is not already running, hooks fall back to the existing direct/spawn path so
the 500ms hook budget is preserved (hooks never auto-start the daemon).
- service.run_sync renders the same operator-facing report shape as the direct
CLI sync path (no_source, out_of_scope, by_source, Re-run/Removed hints) and
drops the old KeyError-prone 'deleted' read.
Address Gemini review on #1729:
- normalize source_file with strip_lone_surrogates so exact matching hits
rows mined from non-ASCII paths via cp1252 stdin (#1488), mirroring
tool_add_drawer's ingestion-side normalization
- isinstance(str) guard so a non-string source_file returns a clean error
instead of AttributeError
- default missing wing/room to "" in the dry-run sample, consistent with
the rest of the file
- add tests: non-string rejection + surrogate-normalization match
Adds an MCP tool to remove every drawer mined from a given source_file
exact match, for cleaning up benchmark/test data accidentally mined into
a user wing (ShareGPT dumps, results_mempal_*.jsonl, language config
JSON) that drowns out real memories in semantic search.
Matching is pushed to the backend via delete(where={"source_file": ...})
the same idiom the miner and diary-ingest paths already use so it is not
subject to the SQLite variable limit regardless of how many drawers share
the source. Defaults to a dry run reporting match count and a sample;
dry_run=false commits. Absent source is an idempotent no-op, not an error.
Adds end-to-end regression coverage for the migration swap path where os.replace hits EXDEV, the shutil.move fallback fails, and the original palace must be restored from the rename-aside copy.
The snippet has no shell interpolation — the path arrives via argv, not
string interpolation — so single quotes are correct and make it
unambiguous that nothing is shell-expanded. Behavior is identical:
`sys.argv[1]` contains no `$`, so it was never expanded (verified
empirically). Matches the single-quoted `python -c` blocks already in
hooks/cursor/lib/common.sh. No functional change.
Address review feedback surfaced on the 3.4.1 release promotion (#1810).
Bug fix — `date -r FILE` is GNU-only. On BSD/macOS `date -r` expects
epoch seconds, not a path, so the staleness/throttle checks in the new
Cursor and Antigravity hooks silently failed on macOS: the state GC
swept on every fire and the pending-save guard was skipped. Replace
with a portable `os.path.getmtime` one-liner via the already-resolved
$MEMPAL_PYTHON_BIN (cursor/lib, antigravity/lib, antigravity save hook).
This restores the "bash 3.2.57 / macOS default" compatibility the
Antigravity changelog claims.
Docs:
- Correct the MCP tool count to 33 (was 19/29/31 in 21 places across
plugin manifests, READMEs, and website docs — all drifted from the
TOOLS dict / mcp-tools.md reference, which both have 33).
- Fix broken CHANGELOG link to the Cursor skill (skills/, not
.cursor-plugin/skills/).
- Fix one-too-many `../` in skills/mempalace/SKILL.md's cursor-hooks
link (resolved above the repo root).
- Add the required `mcpServers` wrapper to the mcp.json example in
.cursor-plugin/README.md so copy-paste yields a valid Cursor config.
Left intentionally unchanged: the os.dup2 fd-1 redirect in
mcp_server.py is deliberate (#225 keeps JSON-RPC off fd 1).
Bump version across all sources (version.py, pyproject.toml, both
Claude plugin manifests, Codex plugin manifest, README badge, uv.lock)
and promote the Unreleased changelog to 3.4.1.
Shipping: Cursor IDE plugin + hooks, first-class Antigravity IDE
support (with zero-config interpreter resolution), embeddinggemma
bulk re-embed OOM fix, and backup-retention pruning.
Also rebuilds the CHANGELOG compare-link block, which had been left
at v3.2.0: adds the full 3.3.0-3.4.1 chain plus the previously
undocumented 3.4.0, and points Unreleased at v3.4.1...HEAD. Every
version header now resolves to a compare link.
`tests/test_fact_checker.py` imports symbols from `mempalace.fact_checker` at
module top (putting it in sys.modules), then `TestCLI.test_exits_nonzero_when_
issues_found` re-executed the same module as __main__ via
`runpy.run_module("mempalace.fact_checker", run_name="__main__")`. runpy warns
because it re-runs an already-imported module against a half-initialized state:
RuntimeWarning: 'mempalace.fact_checker' found in sys.modules after import of
package 'mempalace', but prior to execution of 'mempalace.fact_checker'
Run the CLI in a fresh process via `subprocess.run([sys.executable, "-m",
"mempalace.fact_checker", ...])` instead — no sys.modules collision, and it
exercises the real `python -m` entry point. Assertions are preserved
(SystemExit code 1 → returncode 1; captured stdout substring → result.stdout).
The child's entity registry (`~/.mempalace/known_entities.json`, resolved via
expanduser at import) is redirected by overriding both HOME and USERPROFILE in
the subprocess env so it works on POSIX and Windows.
Verified: `pytest tests/test_fact_checker.py -W error::RuntimeWarning` passes
(26) with the warning promoted to error — proving it no longer fires.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two catches on tests/test_hallways_palace_scoped.py
TestMultiPalaceIsolation.test_save_then_load_under_different_palace_returns_empty:
1. Stale comment referencing the removed _HALLWAY_FILE back-compat shim
(deleted in the prior fixup commit). Removed.
2. _legacy_hallway_file was not monkey-patched, so the test isolation gap
let _load_hallways check the host's real ~/.mempalace/hallways.json
when evaluating the legacy-warning branch. Now patched to a tmp_path
sibling, matching the helper pattern used in test_palace_graph_tunnels.
Replaces the back-compat shim in _load_hallways/_save_hallways (which
honored direct monkey-patches of the _HALLWAY_FILE module constant) with
a clean single-source-of-truth resolver, matching the palace_graph
tunnel-file migration in 3.3.6.
The three existing test sites (tests/test_hallways.py,
tests/test_hallways_pagination.py, tests/test_mcp_server.py) now
monkey-patch _get_hallway_file and _legacy_hallway_file directly,
exactly mirroring the helper in tests/test_palace_graph_tunnels.py.
Production code now has one branch through the path resolution instead
of two. No behavior change. 269/269 hallway + tunnel + mcp-server
tests pass on Python 3.11 and 3.12, ruff clean.
Pre-3.4 the hallway store was hardcoded at ~/.mempalace/hallways.json
regardless of the configured palace_path, so two palaces on one host
silently shared one file. Mining into palace-A leaked records into
palace-B's hallway code paths.
Apply the 3.3.6 tunnel-file migration pattern:
* MempalaceConfig.hallway_file resolves to <dirname(palace_path)>/hallways.json
* hallways._get_hallway_file(config) reads through MempalaceConfig
* hallways._legacy_hallway_file() exposes the pre-migration hardcoded path
for one-time orphan detection; _load_hallways logs a one-line warning
when the legacy file exists but the configured one doesn't, matching
palace_graph._load_tunnels behavior. No auto-migration — silent merging
risks clobbering newer data.
Atomic-write + 0600 semantics unchanged. Module-level _HALLWAY_FILE
constant kept and honored when monkey-patched directly, so the three
existing test sites that patch it (test_hallways.py, test_hallways_pagination.py,
test_mcp_server.py) keep working without modification.
New coverage in tests/test_hallways_palace_scoped.py mirrors the analogous
tunnel tests: resolver default + custom palace_path + env-var redirect,
orphaned-legacy warning + no-warning when paths match, and an end-to-end
multi-palace isolation regression guard.
Closes#1778
Two threads sharing a cold EmbeddinggemmaONNX via _EF_CACHE could each
build a full model session, and two factory callers could each keep a
private instance. The load is now double-check locked with the session
published last, and the factory cache has an atomic check-then-construct
behind a lock-free fast path. __call__ wraps a bare string, returns []
for None and empty input before the lazy download, and its annotation
matches the accepted types.
One session.run over a repair-scale batch (5000 docs) allocates
attention buffers far beyond available RAM and the kernel kills the
process. Mirror chromadb's ONNXMiniLM_L6_V2 and embed in sub-batches
of 32; per-chunk padding also stops one long doc inflating the whole
batch.
Co-Authored-By: mojie5 <262519016+mojie5@users.noreply.github.com>
Adds the recall layer to the Antigravity plugin, mirroring the three-
layer wiring from feat/cursor-hooks-support, adapted for Antigravity's
native plugin and hook surfaces.
The wake hook (PreInvocation, invocationNum==1) was already the eager
layer — it injects verbatim palace content via injectSteps[].
ephemeralMessage before the first model call, which is strictly better
than Cursor's static directive. This commit adds the on-demand layers:
- integrations/shared/recall-protocol.md: single canonical source of
truth for "search before answering", shared across Cursor, Antigravity,
Claude Code, Codex, OpenClaw. Ported from feat/cursor-hooks-support
with the wake-up step de-coupled from any IDE-specific hook name.
- .antigravity-plugin/skills/mempalace-recall/SKILL.md: recall-only
skill with Protocol, Tool selection, Unhappy paths, and Anti-patterns
sections. Tailored to Antigravity: references injectSteps injection,
not Cursor's additional_context; links to shared protocol via GitHub
URL (relative file paths break in the installed copy at
~/.gemini/config/plugins/mempalace/).
- .antigravity-plugin/rules/mempalace-recall.md: plain .md recall rule
(no .mdc extension, no YAML frontmatter — per Antigravity's plugin
docs which specify rules/<name>.md). Body mirrors Cursor's rule;
references the shared protocol via absolute GitHub URL.
- hooks/antigravity/install.sh: creates skills/mempalace-recall/ and
rules/ in the install dir, copies both new files (cmp-gated,
idempotent). Verified: first run writes both, second run is silent.
- .antigravity-plugin/README.md: updated layout tree + "Three recall
layers" section documenting wake hook (eager), recall skill
(on-demand), and optional rule.
- 11 new tests in tests/test_antigravity_plugin_manifest.py covering:
shared protocol presence, recall skill structure (frontmatter,
required sections, GitHub URL reference), rule format (plain .md, no
frontmatter), and installer (mkdir + copy_file lines present).
Co-authored-by: Cursor <cursoragent@cursor.com>
- Stub _write_marker on the 8 concurrent writer backends: upsert()
rewrites the marker on every call with a plain open('w'), so backends
sharing one local_path race on the same file (sharing violations on
Windows) — a test-design artifact, not the contract under test
- Guard the fixture's created list with a lock for the threaded tests
- Assert exact distance-ordered ids in the query/filter arms
- Reindex race: exactly one 'ran' (index absent beforehand, so the
advisory-lock winner must build)
Re-run live after changes: 15/15 pass (PG 16.10, pgvector 0.8.2).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Mirrors the portable fake-client arms of test_pgvector_backend.py
against a real PostgreSQL+pgvector server and adds live-only arms the
in-memory fake cannot exercise: real <=> operator ground truth, JSONB
pushdown vs local-fallback equivalence, cross-namespace isolation on
real tables, 8-connection concurrent writers, and the advisory-lock
serialization of run_maintenance('reindex') under a 2-connection race.
Gated on MEMPALACE_PGVECTOR_LIVE_DSN (same pattern as the qdrant live
gate); skips cleanly when unset. First run: 15/15 pass on PostgreSQL
16.10 + pgvector 0.8.2 (+AGE 1.6.0 in the same server), psycopg 3.3.4.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A clean `mempalace repair --yes` (legacy path) finished without
_vacuum_and_rebuild_fts5: the bulk delete_collection + re-upsert cycle
leaves the FTS5 inverted index inconsistent, so the next repair aborts
at the sqlite integrity preflight. rebuild_index() got this cleanup
when #1517 was fixed; cmd_repair never did.
Extract the shared epilogue _post_rebuild_cleanup() (close chroma
handles, then VACUUM + rebuild FTS5) and call it from both full-rebuild
paths so they cannot drift apart again. Cleanup runs on the legacy
success path only; failure/restore paths are unchanged.
Closes#1747
Co-Authored-By: nord- <3777600+nord-@users.noreply.github.com>
Ports the OpenClaw "search before answering" protocol to the Cursor and
Claude plugin surfaces so the agent reads the palace before answering
about past work, people, projects, or prior decisions instead of
guessing from model memory.
- integrations/shared/recall-protocol.md: single source of truth for the
recall protocol, referenced by the skill and the rule so they cannot
drift.
- skills/mempalace-recall/SKILL.md: recall-only skill (the mempalace
skill keeps setup/mine/status); cross-linked from the ops skill.
- rules/mempalace-recall.mdc: plugin recall rule, alwaysApply: false so
it only fires on recall-relevant turns and never adds MCP latency to
greenfield work.
- examples/cursor/rules/: opt-in copies for non-plugin users, including
an aggressive alwaysApply: true variant documented with its latency
tradeoff.
- .claude-plugin/skills/mempalace-recall/SKILL.md: Claude plugin parity.
- tests: assert the recall skill and rules/ discovery layout; the
shipped rule must be alwaysApply: false.
- docs: .cursor-plugin/README.md and the cursor-hooks guide now describe
the three layers of recall (hook + skill + rule).
The Antigravity plugin mirror lands as a follow-up on the antigravity
branch, where .antigravity-plugin/ exists.
Co-authored-by: Cursor <cursoragent@cursor.com>
CI's test_no_undocumented_tools enforces that every tool registered
in the TOOLS dict has a corresponding section in mcp-tools.md.
The two hallway tools from b866f41 were missing — adding them here.
Sections mirror the format of the existing list_tunnels and
delete_tunnel entries directly above.
Closes#1739.
Hallways shipped in 3.3.6 (#1558) with Python API entry points
`list_hallways(wing=None)` and `delete_hallway(...)` in
`mempalace/hallways.py`, but neither was registered as an MCP tool
in `mempalace/mcp_server.py`. Mining produces hallways visible in
the mine log ("Hallways: +N within-wing entity link(s)") but
they were not retrievable through MCP.
This change wires both functions into the MCP tool registry
mirroring the existing tunnel-tool pattern:
- `mempalace_list_hallways(wing: str | None = None)` wraps
`hallways.list_hallways`. Optional `wing` filter goes through
`_sanitize_optional_name` just like `tool_list_tunnels`, so
invalid names surface a structured error instead of crashing.
- `mempalace_delete_hallway(hallway_id: str)` wraps
`hallways.delete_hallway` and returns `{"deleted": bool}` so
callers can distinguish a successful delete from a no-op.
Tests in `tests/test_mcp_server.py` cover:
- list returns all records without filter
- list filters correctly by wing
- list rejects invalid wing names with a structured error
- delete removes the targeted record and returns `{"deleted": True}`
- delete with an unknown id returns `{"deleted": False}`
- delete with missing/non-string id returns a structured error
- both tools are present in the public `TOOLS` registry
Pure pass-throughs of the existing Python API. No behavior change
in the underlying hallway storage, no schema change to the
`hallways.json` file format.
LLM clients (Claude, Codex) often autocorrect acronyms in tool
args (ps5→PS5). When the only difference is casing, treat it as
a no-op to prevent silent room fragmentation.
Genuinely different room/wing values still apply normally.
Fixes#1621
Completes the embedder-identity contract for the last backend (qdrant) and
unifies identity persistence across all local-path backends.
Identity is stored in a small per-palace sidecar (mempalace_embedder.json),
NOT in a backend's mismatch marker. The marker's presence signals "palace
initialized" (reads raise CollectionNotInitializedError when the marker exists
but the store doesn't), so recording identity at first empty open must not
create it — a sidecar is unguarded, so a brand-new palace records identity
immediately. This fixes a latent gap (caught in review) where pgvector/qdrant
palaces stayed permanently "unknown" because the marker isn't written until the
first real write.
- New mempalace/backends/_sidecar.py: shared read/write_embedder_sidecar with
the isinstance robustness the review bots taught us; chroma, pgvector, and
qdrant all use it (chroma's inline copy is removed, pgvector switches off the
marker, qdrant adds it).
- QdrantCollection.get/set_embedder_identity delegate to the sidecar, so
palace.get_collection enforces a model swap on a qdrant palace exactly like
the other backends — no live server needed for the check.
Tests: sidecar roundtrip + brand-new-palace recording (creates sidecar without
a marker) + enforcement model-swap raise, for qdrant and pgvector, all
server-free; chroma identity tests unchanged. Full suite: 2495 passed, 82.52%.
Closes#1730. Refs #743, #1724.
Adds the maintenance contract RFC 001 specifies but #1679 deferred, and gives
pgvector an opt-in HNSW index path with concurrency-safe builds.
- base.py: MaintenanceResult (status ran/already_running/noop + free-form
stats), UnsupportedMaintenanceKindError, BaseBackend.maintenance_kinds
ClassVar (reserved: analyze/compact/reindex; a backend with no analogue MUST
omit, not no-op), and BaseCollection.maintenance_state()/run_maintenance()
defaults. EmbeddingCollection forwards both (BaseCollection methods shadow
__getattr__).
- sqlite_exact: analyze (ANALYZE) + compact (VACUUM, autocommit + page stats);
omits reindex (exact scan, no ANN index). maintenance_state reports row/page
counts.
- pgvector: reindex builds the optional HNSW index, serialized by a
session-level pg_advisory_lock so concurrent daemon writers learn
"already_running" instead of each stacking an ACCESS EXCLUSIVE build (the
production wedge). It is opt-in: the default exact `<=>` scan is the
100%-recall path; an HNSW index trades exact recall for scale, so an operator
invokes it deliberately. Also analyze; omits compact (autovacuum). Advertises
supports_server_side_indexes. maintenance_state reports index presence.
- qdrant/chroma: empty maintenance_kinds (qdrant self-optimizes; chroma
maintenance is the separate repair CLI) — the faithful "omit" default.
Tests: contract + sqlite (real, CI-runnable) + pgvector advisory-lock flow via
a fake client (ran/noop/already_running, no live Postgres). Full suite green:
2488 passed, 82.47% coverage.
Benchmark three-phase wiring is deferred — the existing benchmarks/ are
task-benchmarks, not backend-comparison harnesses, so there is nothing to wire
into yet.
Closes#1725. Refs #743.
The explicit-embedding backends (pgvector, qdrant, sqlite_exact) embed through
EmbeddingCollection and persist nothing about which model produced their
vectors. Swapping two same-dimension models (minilm <-> embeddinggemma, both
384-d) silently corrupts retrieval with no error — the worst failure class for
a verbatim-recall system. This records the model name and refuses a swap on open.
- base.py: EmbedderIdentity dataclass, Embedder protocol,
EmbedderIdentityUnknownWarning, and a three-state check_embedder_identity()
helper (known_match / known_mismatch / unknown) raising
EmbedderIdentityMismatchError / DimensionMismatchError. BaseCollection gains
get/set/effective_embedder_identity hooks.
- embedding.py: current_model_name() (cheap) + get_embedder_identity() (probed
dimension, cached per process).
- Persistence per backend: sqlite_exact meta table, pgvector marker JSON
(preserved across rewrites), chroma sidecar JSON. EmbeddingCollection forwards
the hooks explicitly (BaseCollection methods shadow __getattr__).
- palace.get_collection enforces at open: a model swap raises; a brand-new empty
collection records the current model; a populated-but-unrecorded legacy palace
warns and is resolved via the CLI. The check needs no model load.
- CLI: `mempalace palace set-embedder [--model NAME] [--force]` records/overrides
identity without mutating global config or loading a foreign model.
Chroma additionally keeps its native embedding-function check. Qdrant identity
persistence is a fast-follow (no local metadata slot — needs a companion store).
Closes#1724. Refs #743.
The searcher hard-coded `max(0, 1 - distance)` for vector similarity, which is
correct only for cosine. A backend reporting L2 or inner-product distances — or
a legacy Chroma palace built without `hnsw:space=cosine` — was silently
mis-ranked: L2 distances routinely exceed 1.0 and floored every result to 0.
- backends/base.py: `BaseBackend.distance_metric` ClassVar (cosine default) and
a `BaseCollection.distance_metric` property per RFC 001 §2.1.
- backends/chroma.py: `ChromaCollection.distance_metric` reads the real
`hnsw:space`, so a legacy-L2 palace reports "l2" and ranks correctly.
- searcher.py: `_distance_to_similarity(distance, metric)` (cosine/l2/ip,
monotonic + bounded) and `_metric_for_collection(col)`; `_hybrid_rank` and the
CLI display are now metric-aware. Cosine output is unchanged.
- Same conversion applied to the MCP `similarity` field, layers.py L3 search,
and the dedup threshold so reported similarities are consistent.
Cosine behavior is byte-for-byte preserved; only non-cosine metrics change.
Part B of #1726 (routing the chroma.sqlite3 BM25 fallback through
BaseCollection) is tracked separately. Refs #1726, #743.
_missing_dimensionality_appears_recoverable required total_elements_added == len(id_to_label) before keeping a dim-None segment, but total_elements_added is a monotonic add-counter while id_to_label holds only the live elements. After any delete the counts diverge, so every post-deletion segment was quarantined; once the WAL is pruned that destroys the only copy of those vectors. Relax the check to >= and keep the bijection and size checks as the integrity backstop.
- Fix mempalace_find_tunnels params: (required) -> optional. The MCP
handler defaults both wing_a and wing_b to None
(mempalace/mcp_server.py:1277), so the prior docs were factually
wrong. Caught by gemini-code-assist on PR #1719.
- Clarify implicit-vs-explicit tunnel distinction with consistent
casing and a brief in-line definition (implicit = discovered from
drawer content overlap; explicit = user/agent-declared link).
Suggested by copilot-pull-request-reviewer.
- Split the mempalace_memories_filed_away one-liner into a short
description plus 'Returns' and 'When to call' sub-bullets for
readability. Suggested by copilot-pull-request-reviewer.
The openclaw skill was last updated when mempalace exposed 19 MCP
tools. Since then 13 more agent-facing tools have landed; this PR
documents the 8 that openclaw should expose so agents can call them
natively instead of falling back to `npx mcporter call ...`:
Search & Browse:
- mempalace_list_drawers (paginated drawer listing)
- mempalace_get_drawer (fetch a single drawer by id)
Palace Graph:
- mempalace_create_tunnel (explicit cross-wing link)
- mempalace_list_tunnels (enumerate explicit tunnels)
- mempalace_delete_tunnel (remove an explicit tunnel)
- mempalace_follow_tunnels (walk explicit tunnels from a room)
Write / Session:
- mempalace_update_drawer (mutate content or relocate a drawer)
- mempalace_memories_filed_away (ack the silent auto-save hook)
The 3 admin-only tools (mempalace_sync, mempalace_hook_settings,
mempalace_reconnect) are intentionally left out — they're host/admin
operations, not agent-facing memory operations. The Hermes
MemoryProvider plugin landing in MemPalace/mempalace#1684 makes the
same call.
Version bumped 3.3.0 -> 3.4.0 (additive tool surface, no breaking
changes to existing tool docs).
The mempalace_diary_write tool declared a top-level anyOf in its
input schema to require either entry or content. Anthropic's Messages
API rejects any tool schema with a top-level anyOf/oneOf/allOf and
returns a 400 for the entire tools array, so every MCP session failed
to start.
The entry/content constraint is already enforced at dispatch: content
is remapped to entry before the handler runs, and a missing value
returns -32602. Removing the combinator restores compatibility without
weakening validation.
Closes#1711
Expose mining as an MCP tool so clients that cannot shell out (Claude
Desktop, LM Studio, Aionui, Desktop Commander) can index projects,
conversations, or documents in-conversation, not only through the
`mempalace mine` CLI.
tool_mine is a synchronous wrapper over the existing miners (miner.mine,
convo_miner.mine_convos, format_miner.mine_formats) that cmd_mine already
calls, so it adds no new ingestion logic and no backend coupling. Miner
stdout is captured at the Python and file-descriptor level so it cannot
corrupt the JSON-RPC channel (#225); no Unix-only calls, so it works on
Windows. The miners keep the palace write lock, so a concurrent mine
returns a structured already-running error.
The two-job split passed the wheel from the build job to the publish job
via upload/download-artifact, which failed repeatedly with BlobNotFound on
the same-run download (GitHub artifact-service flake) — the PyPI upload step
never ran. Collapse into a single `publish` job that builds and publishes in
the same workspace, removing the handoff entirely. The job keeps the `pypi`
environment gate, OIDC (id-token: write), refs/tags checkout, and the
on-main + version-manifest checks.
Address #1707 review:
- _as_list also wraps a bare dict (single metadata) — list({'k':1}) -> ['k']
would drop the values — and returns list inputs as-is (no copy, Copilot
perf note); other iterables are materialized once.
- Normalize ids and metadatas (not just documents) in add/upsert/update so
a scalar id/metadata stays length-aligned with documents/embeddings.
- Widen query_texts annotation to list[str] | str to match the behavior.
- Tests: bare-str ids + dict metadatas, dict wrapping, list-returned-as-is.
Address Copilot review on #1708:
- Check out refs/tags/<tag> instead of the bare name, so a same-named
branch can't be resolved instead of the tag object (checkout prefers
branches on ambiguous refs).
- Tighten the dispatch-input validation to require vMAJOR.MINOR.PATCH
with an optional -/+ suffix, rejecting loose values like v3 / v3foo.
GitHub doesn't reliably emit a release event when a draft tied to a
pre-existing tag is published (e.g. publishing the long-standing v3.3.6
draft fired no run). Add a workflow_dispatch trigger with a 'tag' input so
a maintainer can run the publish for any existing tag from the Actions tab
or 'gh workflow run'. A resolve step picks the release tag or the input,
format-validates it (^v[0-9][0-9A-Za-z.+-]*$) before any use as a git ref,
and the existing on-main + version-match checks + the pypi approval gate
apply unchanged to both paths.
EmbeddingCollection did _embed_texts(list(documents)). For ChromaDB's
OneOrMany shape, a bare str document splits into per-character 'docs'
(list("abc") -> ['a','b','c']), embedding each character and breaking
length alignment with ids/metadatas on explicit-vector backends
(pgvector, sqlite_exact). Normalize str -> [str] via _as_list() at all
four sites (add/upsert/update documents, query query_texts) and pass the
normalized list to the inner backend too. Addresses PR #1706 review
(Gemini + Copilot, HIGH).
Follow-up to the wing-name normalization (#1675). Palaces built before the
rule filed drawers under leading/trailing-separator wing names (e.g. a
Claude Code path-encoded dir `-home-user-proj` -> `_home_user_proj`); the
new derivation strips those, so searches/diary reads under the new name miss
the old memories — the history is split, not lost.
`migrate_wing_names` (CLI: `mempalace migrate-wings [--dry-run] [--yes]`)
re-keys the `wing` metadata field on drawers and closets to the normalized
form, merging collisions. Design (verified against the codebase):
- Drawer/closet IDs embed the wing as an opaque prefix that is never decoded
back into a wing, and mining idempotency keys on `source_file`, so IDs are
left untouched: closet ->drawer_id pointers stay valid and future mining
still skips already-mined files. No risky extract-rebuild needed.
- `topics_by_wing` registry keys are re-keyed (merging on collision).
- Tunnels resolve via existing read-time normalization and need no rewrite.
- Backend-agnostic (uses the get/update collection abstraction); idempotent;
dry-run-able. The strip is applied by the migration itself, so it is
correct regardless of whether the running build's normalize_wing_name
already carries the #1675 change.
Tests: pure planner coverage (strip, no-op, empty/non-string, collision) +
hermetic backend integration (relabel, merge, dry-run, idempotency).
test_load_config_uses_defaults_when_yaml_missing asserted the derived
wing equals project_root.name. That only held when the random tempfile
name had no separators; tempfile's alphabet includes '_', so once
normalize_wing_name strips leading/trailing '_' (this PR), a name like
'tmpXXXX_' makes the derived wing diverge from the raw name. Compare
against normalize_wing_name(project_root.name) — the actual contract —
which is deterministic across platforms. (Surfaced as a test-windows
failure on this PR, but it was cross-platform flaky.)
- searcher: read the unfiltered fallback result via _first_or_empty()
instead of raw["documents"][0], matching the codebase's QueryResult/dict
polymorphism helper and guarding the empty-result IndexError (Gemini).
- mcp_server: the content->entry remap now fills only when 'entry' is
absent or None, so an explicit (even "") entry wins over the alias —
the truthiness check could clobber an empty entry (Gemini/Copilot).
- mcp_server: diary_write schema now expresses the real contract —
agent_name required + anyOf(entry, content) — so schema-validating
clients can legally call with content only (Copilot).
- test: lock the explicit-empty-entry edge (content must not override).
#1245 landed on develop with three defects that turned the branch red:
- tool_diary_write gave `entry` a default (`entry: str = None`), which
silently disabled the signature-based missing-parameter diagnostic
(-32602) — two TestParamShapeDiagnostics tests failed.
- the `content` alias was never added to the tool input schema, so the
dispatch arg-filter stripped `content` before the handler ever saw it,
so the alias never actually worked.
- the filtered-search fallback inlined into search_memories pushed it
over the C901 complexity ceiling (30 > 25), and mcp_server.py was left
unformatted.
Fix:
- restore `entry` as a required param (revives the -32602 diagnostic)
- add `content` to the diary_write schema and remap content->entry at
dispatch, before the handler, so a content-only call still satisfies
the required `entry` (entry wins if both are supplied)
- extract the fallback into _query_drawers_with_filter_fallback() so
search_memories drops back under the complexity ceiling
- ruff format mcp_server.py
- add regression tests for the content alias (content-only + both-supplied)
Keeps #1245's filtered-search recall fix intact; turns develop green.
Validated the backend against a real Postgres 18.4 + pgvector 0.8.2 instance
(full live roundtrip incl. close/reopen + same-dim write). The reviewer's
claim that a vector(n) column's atttypmod is dimension+4 does NOT reproduce:
raw atttypmod equals the bare dimension on 0.8.x, so the original direct read
was already correct. Keep format_type() anyway as the canonical, version-proof
way to read the typmod, and correct the comment to reflect reality instead of
asserting a bug that does not exist.
Three findings from the Copilot review on ec5d1eb:
- pgvector (real correctness bug): table_dimension() read the raw
pg_attribute.atttypmod of the vector(n) column, which is not the bare
dimension, so reopening a stored pgvector palace could raise a false
DimensionMismatchError on the next same-dimension write. Now rounds through
format_type(atttypid, atttypmod) (the type's own typmod_out), which yields
the canonical vector(N) regardless of encoding or pgvector version. The live
roundtrip test now closes + reopens and writes a same-dim vector to guard it.
- chroma (real correctness bug): _lexical_search_via_sqlite() returned
LexicalHit.id as the internal embeddings.id rowid instead of the public
embeddings.embedding_id, so lexical_search -> get(ids=...) did not round-trip
(broke hybrid-search id lookups). Now selects e.embedding_id and maps rowid
-> public id. Existing FTS test schema updated to include embedding_id (real
Chroma schema) and assert the public id; added an end-to-end round-trip test
through a real ChromaBackend collection.
- sqlite_exact (error-message quality): CollectionNotInitializedError was
raised with palace_path instead of the collection name in get_collection and
delete_collection, inconsistent with the other backends and line 287. Now
names the collection; added a regression test.
Earlier first-pass findings (palace.py unknown-backend KeyError, dedup.py
docstring) were already fixed in ec5d1eb.
Address review on #1698: a published GitHub Release may create the v* tag
or reuse an existing one. Reword the header so it no longer implies the tag
is always created at release time, and state the real guarantee — the
in-workflow checks make the pipeline self-contained (tag on main + matches
the version manifest), independent of version-guard.
Review fixes for the Docker packaging PR:
- docker-publish: tie the `latest` tag to pushes on `main` (the release
branch). Previously it was gated on `is_default_branch`, but the
default branch is `develop` and the workflow never ran there, so
`latest` was never produced. main + `v*` tags publish; develop is
validated via the pull_request trigger but does not publish.
- docker-publish: publish multi-arch amd64+arm64 (Apple Silicon / ARM)
on real pushes via setup-qemu-action; PRs stay amd64-only for speed.
- docker-publish: only export the GHA cache on in-repo events (fork PRs
get a read-only cache, which just emits 403 noise).
- docker-publish: add a build-only job that validates Dockerfile.gpu
compiles so the CUDA variant can't silently rot.
- Dockerfile: correct the persistence comment — the default `minilm`
model caches under ~/.cache/chroma (ChromaDB S3), not
~/.cache/huggingface (that's the optional embeddinggemma model).
- docker-compose: drop the redundant MEMPALACE_PALACE_PATH override (it
duplicated the HOME=/data default); document overrides as examples.
Address review on #1698: committing the version bump straight to main
bypasses branch protection and drifts develop behind. Bump on develop
first; it reaches main via the develop -> main merge.
Publish to PyPI on a published GitHub Release via Trusted Publishing
(OIDC — no stored token), gated by the `pypi` environment's manual
approval. The build job verifies the release tag is reachable from main
and matches mempalace/version.py before building the sdist + wheel; a
separate publish job holds the id-token scope and does the upload.
Documents the one-time setup (PyPI trusted publisher + `pypi`
environment) and the per-release runbook in docs/RELEASING.md.
Adds a second external storage backend (Postgres/pgvector) alongside Qdrant
to prove the BaseBackend/BaseCollection contract generalizes across substrates
(SQL + JSONB containment filters + pgvector `<=>` ranking vs Qdrant's REST/dict
model), and addresses the review feedback on PR #1679.
Backend (mempalace/backends/pgvector.py):
- table-per-(namespace, palace, collection) isolation; advertises
supports_namespace_isolation
- JSONB filter pushdown for the containment subset, local-exact fallback for
$or/$contains/comparisons/where_document
- BM25 lexical search; marker-based mismatch protection
- optional psycopg dependency (lazy import), in-memory fake for CI, live test
gated on MEMPALACE_PGVECTOR_LIVE_URL
- registered in registry/__init__/pyproject entry point + [pgvector] extra;
MEMPALACE_PGVECTOR_DSN / MEMPALACE_PGVECTOR_NAMESPACE config; README docs
Isolation contract (RFC 001):
- PalaceRef/BaseBackend document the per-id MUST and the cross-namespace MUST,
gated on the new supports_namespace_isolation capability token
- runnable conformance suite (tests/_backend_conformance.py,
tests/test_backend_conformance.py); qdrant + pgvector run it via their fakes
Marker fail-loud guard:
- qdrant and pgvector now refuse get_collection when local_path is None instead
of silently opening a remote collection with no mismatch protection
Review fixes:
- palace._open_collection_or_explain handles unknown-backend KeyError as a CLI
state message instead of an escaping stack trace
- dedup.py docstring no longer claims "No API calls" unconditionally (false for
remote backends)
mempalace migrate (.pre-migrate.* full-palace copies) and mempalace repair
max-seq-id (chroma.sqlite3.max-seq-id-backup-* DB copies) each wrote a fresh,
full-size, timestamped backup every run and never deleted the old ones. On a
machine that mines or repairs on a schedule, those copies could silently
accumulate until they filled the disk.
Add a configurable max_backups setting (default 10; env MEMPALACE_MAX_BACKUPS
or config.json) and a shared prune_backups helper that trims the oldest copies
after each new backup is written. Pruning is keyed by filesystem mtime, scoped
strictly to each backup's own naming pattern so live data is never touched, and
best-effort so a deletion failure can never abort the migrate/repair that just
succeeded. Set max_backups to 0 to keep every backup.
Add a multi-stage, uv-based Dockerfile producing a CPU image (with the
extract + spellcheck extras), plus a CUDA variant (Dockerfile.gpu) for
onnxruntime-gpu accelerated embeddings.
A single flexible entrypoint dispatches to the MCP stdio server (default)
or the mempalace CLI. All state -- palace, config, and the lazily
downloaded embedding model -- persists under /data via HOME, runs as a
non-root user, and is exposed as a volume.
Also add a docker-compose.yml for convenience, a GHCR publish workflow,
and a Docker section in the README.
Stop-hook checkpoints were saved via _save_diary_direct with a hardcoded
agent_name="session-hook". tool_diary_read filters Chroma metadata by
agent, so mempalace_diary_read(agent_name="claude") never surfaced any
hook-saved checkpoint. Derive the diary identity from the harness
(claude-code -> claude, codex -> codex; an unknown harness keeps its own
name) and thread it through to tool_diary_write. Make agent_name a
required keyword argument so the identity is always explicit.
Co-Authored-By: YC-AIUSER <273917354+YC-AIUSER@users.noreply.github.com>
Some Claude Code JSONL transcripts have tool_use blocks where the
`input` field is a list rather than a dict (e.g. multi-content tool
calls). This causes an AttributeError on line 554 when code tries to
call `.get()` on the list.
Guard by normalizing list inputs to an empty dict, allowing mining
to proceed without losing other tool metadata.
Fixes: mining Claude Code conversations crashes with
AttributeError: 'list' object has no attribute 'get'
at normalize.py:554
`mempalace status` opened the ChromaDB collection purely to tally drawers by
wing/room — and opening it cold-loads the HNSW vector index. On a 398k-drawer
palace that load costs ~60s of CPU on every invocation, even though the counts
live in chroma.sqlite3's relational tables (`repair-status` already reads them
in <1s; `status` was the outlier).
Read the wing/room histogram directly from chroma.sqlite3 via a new
`_sqlite_wing_room_counts` helper, falling back to the existing ChromaDB-client
path when the sqlite read is unavailable (missing DB, un-bootstrapped
collection, sustained writer lock, or an unexpected schema) — preserving the
state-specific guidance for absent/empty palaces.
Measured on a 398,315-drawer / 3.5GB palace: status CPU ~60s -> ~1s.
Review hardening:
- PRAGMA busy_timeout so a transient checkpoint lock is waited out rather than
instantly demoted to the slow path; a sustained lock still falls back.
- COALESCE over string/int/float so a numeric wing/room matches the ChromaDB
path instead of dropping to "?".
- Explicit `s.scope = 'METADATA'` so the segment join can't silently
double-count on a future ChromaDB layout.
Tests: exact-tally (anti fan-out), no-cold-load regression (proven failable by
reverting the fix), numeric-metadata, partial-metadata "?" bucketing,
locked-DB fallback, and collection-absent None routing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The docstring still said col "must support .get(where=..., include=...)",
but this PR changed the fetch to count() + paginated
get(limit=, offset=, include=) filtered client-side, precisely to avoid the
get(where=...) path that overflows SQLite's variable limit on large wings.
Update the Args entry to describe the real contract so fake collections and
alternate backends implement the right shape.
Docstring only — no behavior change.
The where-filter update to the pagination regression test left one list
comprehension past the line-length limit, so `ruff format --check .` failed
in CI while every test platform passed. Wrap the comprehension as ruff
format produces it — no logic change.
Restores the lint job to green.
compute_hallways_for_wing fetched the whole wing in a single
col.get(where={"wing": wing}). ChromaDB binds one SQL variable per matched id,
so on a wing larger than SQLITE_MAX_VARIABLE_NUMBER (32766) the call raised
"too many SQL variables" inside chromadb. The exception was caught, so the mine
completed — but the wing's hallway graph silently never built, and the
cross-wing tunnels promoted from it were starved, on exactly the large wings
that benefit most from navigation. Confirmed threshold: a 42,062-drawer wing
crashed; a 29,629-drawer wing succeeded.
Replace the single where-get with the established pagination pattern: count()
+ get(limit=5000, offset=...) filtered to the wing client-side — matching
miner.status, palace.regenerate_closets, and palace_graph.build_graph, which
already paginate to dodge the same 32766 limit.
Tests:
- test_hallways_pagination: a collection whose where-get raises (simulating the
overflow) while count() + paginated get works — RED before, GREEN after.
- test_hallways: _fake_collection updated to the paginated API; existing
hallway tests are unchanged in behavior.
Closes#1619.
The write-ahead-log directory was created at module scope in
mempalace/mcp_server.py, so importing the MCP server ran
`_WAL_DIR.mkdir(parents=True, exist_ok=True)` and recreated `~/.mempalace`
even after a user removed it to engage the documented kill-switch
(`hooks_cli._palace_root_exists()`, #1305). On every session start this
re-armed the autosave/mining hooks the user had disabled.
Move the WAL directory setup into a lazy `_ensure_wal()` called from the
write path (`_wal_log`). Importing the module no longer touches disk; the
directory is created on the first real write, when the palace is being
written to anyway. The WAL is intentionally not gated on
`_palace_root_exists()` (the ChromaDB/KG layer recreates the palace
regardless, so gating would only drop audit records); runtime kill-switch
enforcement for MCP writes is tracked in #504.
Add regression tests: a subprocess import asserts `~/.mempalace` is not
created, and a write test asserts the directory is created lazily with the
expected permissions.
Co-Authored-By: Grace Gettert <9805362+ggettert@users.noreply.github.com>
A path-encoded dirname like `-home-user-proj` produced a leading-underscore
slug (`_home_user_proj`) that sanitize_name — and therefore the MCP write
tools — reject, so the conversation miner filed transcripts into wings the
MCP could never write to. Strip leading/trailing `_` after collapsing
separators so the slug is valid. Adds tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per Gemini review on #1673: chromadb accepts OneOrMany[Document], so a bare
str document was iterated character-by-character by the list comprehension,
splitting it into per-character documents (the silent corruption this method
exists to prevent). Handle isinstance(str) explicitly; add a regression test.
#1235 sanitised lone UTF-16 surrogates for the MCP write tools, but the bulk
ingest paths (miner, convo_miner, sweeper, diary_ingest) build documents
without routing through sanitize_content() and reach ChromaCollection directly.
A single lone surrogate in document text raises UnicodeEncodeError inside
chromadb and aborts the whole add/upsert batch with a -32000 Internal Error,
silently dropping every other row in the same batch.
Complete the chokepoint: add _sanitize_documents_for_chromadb (mirror of
_sanitize_metadatas_for_chromadb) and apply it in add/upsert/update so the
backend guarantees UTF-8-safe documents regardless of caller. IDs and dedup are
unaffected (IDs are computed upstream); only illegal lone surrogates become
U+FFFD, matching the errors="replace" behaviour used elsewhere.
`uv tool install mempalace` / `pipx install` place the mempalace
console scripts in an isolated environment whose interpreter is not
the system python3. mempal_resolve_python previously resolved
`command -v python3`, landing on a Python that cannot import
mempalace: the `-m mempalace --version` probe failed and mining
silently never fired (hit by a real user on PR #1633).
Resolution now derives the interpreter from the mempalace-mcp /
mempalace console-script shebang on PATH (the same script the MCP
server launches) before falling back to python3. It is pure shebang
parsing + stat — no Python subprocess at source time — so the hook
performance budget is preserved. An env-style `#!/usr/bin/env python`
shebang and a non-executable interpreter are both rejected and fall
through. MEMPAL_PYTHON remains the explicit override.
Adds 6 resolver regression tests, documents resolution + MEMPAL_PYTHON
in the guide and hooks README (fixing the stale `command -v mempalace`
note), and a CHANGELOG entry.
Co-authored-by: Cursor <cursoragent@cursor.com>
Drawer/closet/triple IDs built by concatenating strings without a
delimiter before hashing can collide: hash((s1 + str(i1))) ==
hash((s2 + str(i2))) whenever s1+str(i1) == s2+str(i2). Concretely,
source_file="foo" + chunk_index=12 produces the same hash input as
source_file="foo1" + chunk_index=2 — both yield "foo12". Under
ChromaDB's primary-key constraint the second write silently
overwrites the first; one chunk's content is lost without surfacing
an error (the surrounding broad `except Exception:` blocks swallow
any backend complaint).
The defect class is "string + string concat fed to a hash without
a delimiter." Per the styleguide's partial-scope-key-migration rule,
every site sharing the pattern is a candidate and must be triaged —
not just the ones the original issue named.
FIX — 6 sites
- mempalace/miner.py:1253 drawer_id, add_drawer() back-compat
- mempalace/miner.py:1386 drawer_id, batched mine loop
- mempalace/miner.py:1416 drawer_ids, closet-build re-derivation
- mempalace/format_miner.py:643 drawer_id, format_miner batched loop
- mempalace/mcp_server.py:1136 drawer_id, tool_add_drawer
- mempalace/knowledge_graph.py:305 triple_id, KG triple insertion
MIGRATED FOR CONSISTENCY — 2 sites
- mempalace/convo_miner.py:87 sentinel_key — was `:`, now `|`
- mempalace/convo_miner.py:422 drawer_key — was `:`, now `|`
Pre-existing delimiter was `:`. Migrated because (a) source_file can
contain literal `:` on Windows paths (`C:\Users\…`) and URL-like
sources (`https://host:8080`), weakening `:` as a separator; (b) `|`
is reserved in Windows filenames so source_file cannot contain it,
making it strictly safer.
DELIMITER CHOICE
- `|` chosen over `:` (Windows / URL source_file edge cases)
- `|` chosen over `\x00` (NUL breaks log greppability + print()
debug + downstream string handling for marginal extra safety)
- Matches the established precedent in mempalace/diary_ingest.py
(lines 52, 76, 91, 98 — all already on `|`)
EXEMPT — audited and correct as-is
Single-input hashes (nothing to delimit):
- mempalace/miner.py:1432 closet_id (source_file only)
- mempalace/format_miner.py:559 sentinel_id (source_file only)
- mempalace/palace.py:433 lock filename (source_file only)
- mempalace/palace.py:629 palace_key (lock_key_source only)
- mempalace/diary_ingest.py:158 content_hash (text only)
- mempalace/hooks_cli.py:329 pidfile digest (joined cmd only)
- mempalace/sources/context.py:141 record digest (source_file only)
Already correctly delimited:
- mempalace/hallways.py:157 `f"{wing}::{a}::{b}"` (`::`)
- mempalace/palace_graph.py:454 `f"{a}↔{b}"` (`↔`)
- mempalace/diary_ingest.py:52,76,91,98 (`|` precedent)
Protected by composition (uniqueness guaranteed by the ID prefix,
not by the hash slice):
- mempalace/mcp_server.py:1635 entry_id is
`diary_{wing}_{strftime('%Y%m%d_%H%M%S%f')}_{sha[:12]}`.
Microsecond-resolution timestamp prefix supplies uniqueness;
the trailing hash is a content-discriminator, not the
write-time uniqueness guarantor.
NEW MODULES
- mempalace/ids.py — single source of truth for ID construction.
Five helpers (make_drawer_id_from_chunk, make_drawer_id_from_content,
make_convo_drawer_id, make_convo_sentinel_id, make_triple_id) plus
an ID_RECIPE = "v2" constant.
- mempalace/collision_scan.py — pre-mining defense. Runs immediately
before each batched ChromaDB upsert; raises CollisionError naming
the colliding (source_file, chunk_index) pairs if any proposed
drawer_id appears more than once with conflicting metadata across
the union of incoming and existing rows.
DETECTION
ChromaDB's primary-key constraint means past collisions were
silently resolved at upsert time (last-write-wins), erasing
evidence. A post-hoc audit cannot reconstruct what was lost from
palace state alone. Therefore:
- Pre-mining risk scan. Before each batched upsert, compute the
proposed drawer_ids for the incoming chunk set AND query existing
drawer_ids from the collection. If any proposed id appears more
than once in the union (incoming-vs-incoming or incoming-vs-
existing) with conflicting (source_file, chunk_index), abort the
mine with an actionable error naming the colliding pairs.
Collision is caught BEFORE it destroys data, which is the only
point at which palace state still carries the evidence.
- New metadata key: `"id_recipe": "v2"` on every drawer written
under the delimited recipe. Audits compare like-for-like;
drawers without `id_recipe` are treated as v1 legacy (undelimited
or `:`-delimited), not as collisions.
- Honest disclosure: palaces mined under any pre-v2 mempalace may
carry silent past collisions whose original content is
unrecoverable from palace state. Future library tier work will
give users a per-drawer audit + opt-in archival path.
TESTS
- tests/test_ids.py: 17 unit tests covering all 5 helpers, the
ID_RECIPE constant, the private `_delimited_sha256` helper, and
the four defect-class collision shapes (chunk_index boundary,
content boundary, extract_mode boundary, ISO datetime boundary).
RED collision tests confirm the v2 recipe breaks the defect class.
- tests/test_collision_scan.py: 10 tests covering clean batches,
idempotent re-mines, incoming-vs-incoming collisions, incoming-vs-
existing collisions, error-message quality, empty batches,
metadata without chunk_index, and ChromaDB backend errors
propagating cleanly.
- tests/test_convo_miner_unit.py: FakeCol gains a stub `get()` so
the pre-mining scan can probe an empty in-test collection.
BACKWARDS COMPATIBILITY
- Existing v1 drawers remain queryable; no rewrite of stored IDs.
- New mining passes write v2 drawers alongside v1 via PR #1628's
additive-mining model.
- No user action required; opt-in cleanup ships separately.
VERIFICATION
- macOS Python 3.12 (local): 2304 passed, 3 skipped, 0 failed,
coverage 85.44% (above 80% threshold).
- Linux Python 3.9/3.11/3.13 (OrbStack with full `pip install -e
'.[dev]'` flow): see PR body for per-version pass counts.
- `ruff check .` + `ruff format --check .`: clean.
- pylint score: ids.py 10.00/10, collision_scan.py 10.00/10; all
modified files >= 8.92 (no regression).
- bandit: clean on all new files; the one MEDIUM finding in
knowledge_graph.py is on lines 385/407 (pre-existing SQL string
construction in timeline queries), not in code touched by this PR.
- pre_push_check.sh: ALL CHECKS PASSED.
Refs: deferred from PR #1572 (Copilot finding #13). Styleguide audit
documented above: 8 fix/migration + 11 enumerated exempt sites.
Prevent redundant quarantine re-run when a fresh ChromaBackend instance
opens a palace that was already quarantined by another instance in the
same process. The mtime_appeared transition (cached 0.0 -> real mtime)
now only triggers a discard if the instance previously tracked the path,
distinguishing genuine file appearance from first-access default.
Addresses gemini-code-assist review on PR #1602.
Co-Authored-By: C-LaForest <C-LaForest@users.noreply.github.com>
The _quarantined_paths gate fired once per palace per process and never
re-armed after external in-place writes (closet_llm, mine, compress)
that drift HNSW segments. The MCP server path (make_client static) had
zero discard logic -- quarantine never re-armed even on inode change.
Extend the discard guard in _client() from inode_changed-only to
inode_changed or mtime_changed or mtime_appeared. Add a guarded
discard in mcp_server._get_client() before make_client(), and an
unconditional discard in tool_reconnect().
Remove dead _auto_repair / palace-daemon comment (does not exist in
this codebase) and correct misleading _get_collection retry-path
comments that overclaimed quarantine re-runs.
Co-Authored-By: C-LaForest <C-LaForest@users.noreply.github.com>
chromadb pre-allocates data_level0.bin at index creation (~168 KB for
384-dim embeddings) regardless of record count, so the previous
data-size-vs-floor heuristic in _segment_appears_healthy could not
distinguish a single-record segment (sub-threshold, never persisted)
from an interrupted persist.
Restructure _segment_appears_healthy: when index_metadata.pickle is
absent, check link_lists.bin instead of data_level0.bin size. Empty or
absent link_lists + absent metadata = sub-threshold (never persisted).
Non-empty link_lists + absent metadata = interrupted persist.
Co-Authored-By: 0xKingVee9527 <0xWinner98@users.noreply.github.com>
_HNSW_BLOAT_GUARD set batch_size and sync_threshold to 50,000 to
prevent link_lists.bin sparse-file bloat in pre-1.5.x Python chromadb
(#344). chromadb >=1.5.4 Rust bindings do not exhibit that bloat.
The 50k guard meant any mine under 50,000 drawers never triggered
chromadb's _persist(), leaving index_metadata.pickle absent and
link_lists.bin empty. quarantine_stale_hnsw then renamed the segment
on every cold open after a 300s mtime gap, accumulating .drift-*
directories indefinitely.
Lower both thresholds to 2 (empirical Rust-side minimum; 1 is rejected
with InvalidArgumentError) so any mine of 2+ drawers triggers a natural
persist. Verified: batch_size=2 with 20k records produces
link_lists.bin at 171 KB with zero sparse-file inflation.
Existing palaces retain the old 50k thresholds in their collection
metadata until the user runs repair --mode from-sqlite.
Co-Authored-By: Tim Harmon <tim-harmon@users.noreply.github.com>
_chunk_by_exchange stripped every line, joined them with single spaces, and
silently dropped blank lines. That violated the verbatim-always principle
stated in CLAUDE.md and contradicted the function's own docstring, which
claimed 'The full AI response is preserved verbatim.'
Concrete consequences before this change:
- paragraph breaks fused: 'para1\n\npara2' → 'para1 para2'
- list items fused: '1. a\n2. b' → '1. a 2. b'
- code fences destroyed: indented code collapsed to a single line
- search quality degraded because tokenization changed at ingest
Fix is surgical: keep each line as-is, join on newline, trim only
trailing newlines produced by the loop stopping at the next '>' turn.
The fallback path _chunk_by_paragraph has a narrower version of the
same bug (it strips each paragraph); that is out of scope here and left
for a follow-up.
GBK consoles (Windows PowerShell/CMD default) cannot encode U+2713 (✓),
U+2717 (✗), and U+2014 (—). The same class of UnicodeEncodeError fixed
in miner.py via #681 affects closet_llm.py and cli.py.
Replace with ASCII equivalents: [OK], [FAIL], [!], and hyphen.
Windows exports of Claude Code JSONL sessions prepend a UTF-8 BOM
(\xef\xbb\xbf). With encoding='utf-8', json.loads() raises JSONDecodeError
on the first line, _try_claude_code_jsonl silently skips every line, and
the file falls through as raw text — losing all structured message content.
utf-8-sig strips the BOM transparently and is backward-compatible with
BOM-free files on all platforms.
The EmbeddinggemmaONNX lazy-load now fetches the ONNX external-weights file
(model.onnx_data) in addition to the model graph and tokenizer, so a single
warm-up issues 3 downloads, not 2. The lazy-load-once invariant is unchanged
(InferenceSession and Tokenizer.from_file are still each built exactly once).
Resolves the maintainer review on the Cursor IDE support PR. Cursor-only
scope; cross-IDE items (wing-naming convention, shared-file merge order)
are coordinated on the separate Antigravity branch.
followup_message default (the one "decide before merge" item):
- Keep the stop-hook followup ON by default. Cursor's transcript format
is undocumented and mempalace/normalize.py has no Cursor parser, so the
background `mempalace mine --mode convos` is best-effort only and does
not yet yield clean verbatim drawers. The followup is therefore the
load-bearing verbatim-capture path; defaulting it off would leave a
default Cursor install capturing nothing.
- Add an opt-out (MEMPAL_CURSOR_SILENT=1, or MEMPAL_VERBOSE=false) for
users who want the Claude-style "zero tokens in chat" behaviour. The
hook still mines and keeps its counters/markers when silenced.
- Correct the misleading "background mine captures it" comments in the
save and precompact hooks; update hooks/cursor/README.md and the guide.
Hygiene fixes:
- Drop the hardcoded "version" field from .cursor-plugin/plugin.json and
marketplace.json (mempalace/version.py is the single source of truth);
tests now assert the field stays absent.
- Remove the committed .cursor-plugin/{commands,skills} symlinks (they
break on Windows clones with core.symlinks=false and were redundant
with the real repo-root components that `source: "."` already serves);
add a guard test that no symlinks exist under .cursor-plugin/.
- Document the preCompact synchronous-mine timeout tradeoff and that an
incremental/append-only mine is recoverable if killed (no corruption).
- Add a Cursor-namespaced, daily-throttled TTL sweep (MEMPAL_STATE_TTL_DAYS,
default 30) to lib/common.sh that GCs stale cursor_*.count/.pending only,
after the kill-switch check; shared logs and antigravity_* are untouched.
Verification: full suite green (2424 passed, 3 skipped), ruff check +
format clean, bash -n clean on all cursor scripts. +30 Cursor tests
(followup opt-out, state GC, TTL validation, no-symlink/version guards).
Co-authored-by: Cursor <cursoragent@cursor.com>
Addresses igorls' review on PR #1633 (antigravity branch only):
- Atomic counter write: add mempal_write_counter_atomic (same-dir temp +
mv -f rename) and use it in the save hook, replacing the truncate-then
-write printf that the comment falsely called "atomic". Concurrent
readers now always see a complete value.
- Background the expensive probe: `mempalace --version` pays the full
chromadb/onnx cold-start import (the mine subparser imports
mempalace.miner before argparse handles --version), so running it in
the foreground blew the <500ms save budget. Probe + mine + pending
-marker cleanup now run in one detached subshell; the foreground
returns immediately. This also retires the kill -0 watcher (the prior
gemini fix) since cleanup is now sequential within the mine's own
shell, removing the sibling-PID hazard entirely.
- State-file GC: add mempal_state_ttl_days (default 30, env
MEMPAL_STATE_TTL_DAYS) and mempal_gc_stale_state, a daily-throttled
sweep (antigravity_last_sweep marker) that removes stale
antigravity_save_count_*, antigravity_pending_*, and
antigravity_woke_* artifacts. Called after the kill-switch check so a
disabled hook touches nothing; specific name globs leave shared logs
untouched.
Tests: atomic-counter behavior + no-temp-leftover, single-subshell
structure (no wait/kill -0/MINE_PID), backgrounded-probe timing (3s
stub returns in <2s), async log polling for the missing-module path,
GC sweep/throttle/TTL-validation, and GC gated by the kill switch.
Also hardens the wake-missing test to pin MEMPAL_PYTHON so a shell
-exported interpreter can't defeat the simulation.
bash 3.2.57 safe, fail-open on every path, {}-only Stop output.
Co-authored-by: Cursor <cursoragent@cursor.com>
Under the additive-mining model (drawer history preserved across re-mines),
a single source_file can have multiple parent_drawer_id groups in the
palace, one per mining pass. Each group carries its own stored
source_mtime and normalize_version.
`file_already_mined` previously used `collection.get(where={"source_file": X},
limit=1)` in the `extract_mode is None` branch and only checked the single
returned row's metadata. ChromaDB's `get(..., limit=1)` has no ordering
guarantee across multiple matching rows, so the returned row was effectively
arbitrary. When ChromaDB happened to return a stale group (older mining
pass with a different stored mtime), the function returned False, the
additive miner concluded the file had changed, and wrote yet another
duplicate group of drawers for a file that had not actually changed since
the last successful mine.
Steady-state failure: for any source_file that has ever been edited (so
that the palace contains groups with different stored mtimes), each re-mine
has a probability of spuriously concluding "file changed" and writing
another duplicate group. Each spurious group compounds the problem because
its stored mtime can also trigger the next spurious re-mine. Storage grows
without bound; search results, hallway counts, and entity-frequency stats
become inflated proportionally.
The fix mirrors the paginated-iteration pattern already used in the
`extract_mode is not None` branch — iterate every drawer for the
source_file in 1000-row pages, short-circuit on the first matching group.
A correct group is one that passes the existing checks: normalize_version
not stale; if check_mtime, stored source_mtime within 0.001 seconds of the
current file mtime. The two branches (extract_mode is None vs set) collapse
into one loop that skips the extract_mode check when no extract_mode was
specified.
Trade-off: average-case cost rises from O(1) limit=1 query to O(N/1000)
paginated scan. For typical sources (1-3 groups) the cost is unchanged
because the loop short-circuits on the first matching group within the
first page. For pathological sources with thousands of groups, the cost
is O(number-of-pages-until-match) — still bounded, no longer flaky.
RED test pins the failure space deterministically
`test_file_already_mined_handles_multiple_groups_under_one_source_file`
uses a MockCollection that simulates two parent_drawer_id groups under one
source_file: a STALE group (older mtime) and a CURRENT group (matching
mtime). The mock returns the STALE group when called with limit=1
(worst-case ChromaDB ordering) and returns BOTH groups when called with
limit=1000 (what a correctly-iterating implementation must do). Test asserts
file_already_mined returns True even when limit=1 picks the stale group.
- Against pre-fix code: test FAILS (function returns False because
limit=1 picks stale group, mtime mismatch returns False)
- Against post-fix code: test PASSES (iteration finds the current group,
short-circuits to True)
Verified RED-then-GREEN locally. Existing 4 file_already_mined tests in
tests/test_miner.py continue to pass:
- test_file_already_mined_check_mtime
- test_file_already_mined_scopes_convo_extract_mode
- test_file_already_mined_extract_mode_paginates_large_sources
- test_file_already_mined_returns_false_for_stale_normalize_version
Verification
- macOS Python 3.12 (local) full pytest : 2268 passed, 0 failed
- Linux Python 3.9.25 (OrbStack) : 2260 passed, 0 failed
- Linux Python 3.11.15 (OrbStack) : 2261 passed, 0 failed
- Linux Python 3.13.13 (OrbStack) : 2261 passed, 0 failed
- ruff check + ruff format --check : all clean
Provenance
Surfaced during the per-query audit on the PR #1628 amendment cycle (the
search for every bare `where={"source_file": ...}` query in the repo).
One of six sites identified. The other five are legitimately file-global
in intent (closet purges, full-rebuild deletes, paginated mode-filtered
scans). This site is the one whose failure mode mirrors the cross-group
stitching pattern PR #1628 fixed at the searcher layer.
Five fixes for issues called out by the gemini-code-assist[bot] review.
Each gets a regression test that locks in the correction.
1. CRITICAL: marker-cleanup watcher used POSIX `wait` on a sibling pid
(save hook). bash `wait` only works on direct children of the
calling shell — the `( wait $MINE_PID ... ) &` subshell runs as a
sibling of MINE_PID, so wait fails immediately and the pending
marker is deleted within milliseconds, defeating the concurrency
guard. Replace with `while kill -0 $MINE_PID; do sleep 1; done`,
which queries pid existence regardless of parent-child relationship.
Test: test_save_hook_marker_watcher_uses_kill_polling.
2. Bare `mempalace` console-script invocation in the save hook fails
when the venv's bin/ is not on the hook's PATH (e.g. uv tool
install in some configurations, manually managed virtualenvs).
Switch to `"$MEMPAL_PYTHON_BIN" -m mempalace mine ...` so the
resolved interpreter runs the package directly via
mempalace/__main__.py. Tests:
test_save_hook_uses_python_module_invocation,
test_save_hook_missing_mempalace_python_module_does_not_crash.
3. Same issue in the wake hook's inner Python helper. Switch
`['mempalace', 'wake-up', ...]` to `[sys.executable, '-m',
'mempalace', 'wake-up', ...]` — sys.executable is the same
interpreter that resolved MEMPAL_PYTHON in lib/common.sh.
Test: test_wake_hook_uses_sys_executable_module_invocation.
4. The Python parser in lib/common.sh wrapped `json.load` in
`try/except` and silently fell back to `data = {}`. The script
then printed the `__MEMPAL_PARSE_OK__` sentinel even on parse
failure, so the bash sentinel-check on the caller side
(`[ "$_marker" != "__MEMPAL_PARSE_OK__" ]`) never triggered the
defense-in-depth `input parse failed` branch. Remove the
try/except so the exception propagates, Python exits non-zero,
and the sentinel is omitted on bad JSON. The traceback still
lands in antigravity_last_python_err.log for debugging.
Test: test_common_sh_parser_omits_sentinel_on_malformed_json.
5. `mempal_save_interval()` failed to strip leading zeros from
MEMPAL_SAVE_INTERVAL. Values like "08" or "09" then crashed the
modulo step `$((COUNT % INTERVAL))` because bash arithmetic
parses tokens starting with `0` as octal, and 8/9 are not valid
octal digits ("value too great for base"). Strip leading zeros
while preserving the literal "0" (which is then floored to 15).
Test: test_save_hook_handles_leading_zero_save_interval (4 cases).
Plus one cosmetic fix in install.sh: removed a no-op `(cd "$OLDPWD"
2>/dev/null || cd .) >/dev/null 2>&1` line in mempal_absolutize().
The subshell cd doesn't affect the parent shell, and the installer
never cd's in the main shell anyway, so $PWD is already correct.
Verification:
* 9 new regression tests, all 65 antigravity tests pass
* full repo: 2323 passed (was 2314), 3 skipped, 1 unrelated warning
* ruff check + ruff format --check both clean across 139 files
* bash -n clean on all four shell files
* clean reinstall to ~/.gemini/config/plugins/mempalace/ succeeds
* idempotent re-run produces zero file writes (cmp-gated)
* both hooks return {} exit 0 with synthetic camelCase stdin
Co-authored-by: Cursor <cursoragent@cursor.com>
Adds first-class integration with Google's Antigravity IDE
(https://antigravity.google/) as a third sibling to the existing
Claude Code and Codex hook integrations. Strictly additive — no
existing files in main are restructured.
What ships
----------
* `.antigravity-plugin/` — verified-minimal plugin package:
* `plugin.json` with `{"name": "mempalace"}` (no fabricated fields)
* `mcp_config.json` registering the `mempalace-mcp` stdio server
* `hooks.json.tmpl` templated with `__PLUGIN_DIR__` substitution
* `skills/mempalace/SKILL.md` (real file — no symlinks)
* `hooks/antigravity/`:
* `lib/common.sh` — shared bash 3.2.57-compatible helpers with
sentinel-guarded camelCase JSON parser, antigravity_*-namespaced
state files, every existing kill switch, `MEMPAL_SAVE_INTERVAL >= 1`
floor (no /0), and fail-open emitters
* `mempal_save_hook_antigravity.sh` — Stop event handler:
increments per-conversation counter, defers when fullyIdle=False
or terminationReason=error, validates transcriptPath against
`..` traversal, spawns `mempalace mine --mode convos` in a
detached subprocess with a per-conversation pending marker,
ALWAYS emits `{}` (never `{"decision":"continue"}` — that would
force an infinite agent loop)
* `mempal_wake_hook_antigravity.sh` — PreInvocation handler gated
to invocationNum==1 with an atomic mkdir loop guard, runs
`mempalace wake-up` with a 500ms hard timeout, emits verbatim
output as `{"injectSteps":[{"ephemeralMessage":"..."}]}` or
`{}` on any failure
* `install.sh` — idempotent installer with cmp-gated copies,
`__PLUGIN_DIR__` substitution, relative path absolutization,
`--dry-run`, and basename-guarded `--uninstall` (refuses to
wipe a directory whose basename isn't `mempalace`)
* `INVESTIGATION.md` — verbatim quotes + URLs + dates from the
five official Antigravity doc pages, recording every surface
shipped and every surface deliberately omitted
(PreCompact equivalent, slash-commands, rules/, plugin
permissions field — the latter is third-party fabrication)
* `STDIN_SHAPE.md` — exact stdin/stdout contract per event with
worked examples
* `README.md` — local hook docs + troubleshooting
* `examples/antigravity/{hooks.json,mcp_config.json,README.md}` —
standalone configs for users who don't want the full installer
* `website/guide/antigravity.md` + sidebar entry — VitePress guide
* Updates to `README.md`, `CHANGELOG.md` (Unreleased), `hooks/README.md`
Tests (56 new, all passing)
---------------------------
* `tests/test_antigravity_plugin_manifest.py` (11 tests) — schema
contract on the in-repo `.antigravity-plugin/` directory, including
guards against re-introducing the fabricated `permissions` field
and against any symlink leak.
* `tests/test_antigravity_hooks_shell.py` (31 tests) — invokes the
bash hooks via subprocess with synthetic camelCase stdin, asserts
`{}` on every failure path, kill-switch coverage (env vars +
config.json + palace nuke), divide-by-zero floor, transcript
traversal rejection, namespacing, wing inference, and the hard
refusal to ever emit `decision=continue` from the Stop hook.
* `tests/test_antigravity_hooks_install.py` (14 tests) — `--dry-run`
side-effect-free, real install layout, executable bits preserved,
byte-identical idempotent re-runs (md5 + filecmp), basename-match
uninstall safety, refusal when plugin.json is missing or names a
different plugin, relative path absolutization. Skipped on Windows.
Verification
------------
* `uv run pytest tests/ --ignore=tests/benchmarks -v` → 2314 passed,
3 skipped (Windows), 1 unrelated warning
* `uv run ruff check .` → all checks passed
* `uv run ruff format --check .` → 139 files already formatted
* `bash -n` clean on common.sh, both hook scripts, install.sh
* Local install at `~/.gemini/config/plugins/mempalace/` verified end-
to-end: layout correct, paths absolutized in hooks.json, both hooks
fire with realistic camelCase JSON in <1s, wing inference picks
`wing_mempalace` from workspacePaths[0], state files all
`antigravity_*`-namespaced, second `install.sh` run produces
byte-identical output (md5 snapshots match), uninstall removes
only the mempalace plugin and leaves all 6 sibling Google plugins
untouched.
Constraints honoured
--------------------
bash 3.2.57 (no mapfile / readarray / declare -A / `${var^^}`),
verbatim guarantee on all wake injections, hooks <500ms / startup
injection <100ms target (kill-switch path returns in <1.5s in CI),
zero new runtime dependencies, no telemetry, no external API,
strictly additive (existing Claude/Codex hooks unchanged).
Refs: hooks/antigravity/INVESTIGATION.md for the full audit.
Five fixes from the Gemini Code Assist review on
https://github.com/MemPalace/mempalace/pull/1632 — three real bugs,
two cleanups, all consistent with the bash-3.2-compatibility
contract documented in the original commit.
Bug fixes (high)
----------------
1. hooks/cursor/lib/common.sh — config.json kill-switch check used
a `python3 - <<'PYEOF' ... PYEOF` heredoc inside a `$(...)`
command substitution. The heredoc body contains parens which
trips the macOS bash 3.2.57 parser bug. Replaced with a
`python -c '...'` call passing the config path as argv[1]. Matches
the pattern already used in mempal_parse_stdin in the same file.
2. hooks/cursor/install.sh — a relative `--install-dir` was written
verbatim into hooks.json. Cursor invokes hook commands from its
own working directory (typically the project root), so a relative
command path would silently fail to launch the hook. Now resolved
to an absolute path against `$PWD` before being baked in.
3. hooks/cursor/mempal_save_hook_cursor.sh — `MEMPAL_SAVE_INTERVAL=0`
would crash bash on `$((NEXT % 0))` (division by zero). Extended
the existing sanitiser case to coerce 0 to the default interval
alongside empty / non-numeric values.
Cleanups (medium)
-----------------
4. hooks/cursor/install.sh — the EMPTY_CHECK_PY temp file is now
inlined as `python -c '...'`. Removes a small leak window
(tmpfile would linger if the script were interrupted between
mktemp and rm -f) and shortens the script.
5. hooks/cursor/install.sh — `mktemp -t prefix` has subtly different
semantics on BSD (macOS) vs GNU mktemp. Switched to the
portable absolute-template form `mktemp "${TMPDIR:-/tmp}/...XXXXXX"`
which behaves identically on both.
Regression tests
----------------
- tests/test_cursor_hooks_shell.py
test_save_interval_zero_is_coerced_to_default — guards fix#3.
- tests/test_cursor_hooks_install.py — new TestInstallDirAbsolutePath
class:
test_relative_install_dir_is_absolutized_in_hooks_json — guards
fix#2 against regression.
test_absolute_install_dir_is_preserved_verbatim — guards that
the relative-to-absolute resolution does not mangle paths
that were already absolute.
Verification
------------
- bash -n on all three edited scripts: clean.
- uv run pytest tests/test_cursor_hooks_*.py tests/test_cursor_plugin_manifest.py: 132 passed (was 129; +3 regression tests).
- uv run pytest tests/ --ignore=tests/benchmarks: 2399 passed,
3 skipped (pre-existing).
- uv run ruff check . / ruff format --check .: clean.
Co-authored-by: Cursor <cursoragent@cursor.com>
Adds first-class Cursor IDE integration alongside the existing Claude
Code and Codex hook flows, so Cursor users get the same automatic
diary saves, pre-compaction transcript capture, and session-start
memory recall — without changing any default behaviour for existing
users.
What's included
---------------
Cursor hook scripts (hooks/cursor/):
- mempal_save_hook_cursor.sh — Stop event, counter +
loop_count guard, pending-save marker consumption, background
mempalace mine, followup_message emission.
- mempal_precompact_hook_cursor.sh — synchronous mine before
compaction, drops a pending_save marker, returns user_message.
- mempal_wake_hook_cursor.sh — sessionStart event,
wing-scoped recall guidance via additional_context.
- lib/common.sh — shared parsing + state helpers
(bash 3.2 safe, no heredoc-in-subshell traps).
- install.sh — idempotent installer with
--scope, --variant, --dry-run, --uninstall. Recognises existing
entries by basename so re-installs across paths work.
- STDIN_SHAPE.md, README.md — payload schemas + quick
reference.
Cursor plugin (.cursor-plugin/ + repo-root components):
- plugin.json, marketplace.json, README.md.
- skills/mempalace/SKILL.md — model-invocable skill mirroring the
Claude plugin's skill surface.
- commands/mempalace-{help,init,mine,search,status}.md — slash
commands for marketplace-published installs (filename = slug).
- mcp.json — auto-registers the mempalace MCP
server, wrapped under the documented mcpServers key.
Examples + docs:
- examples/cursor/hooks.json, hooks.minimal.json + README.
- website/guide/cursor-hooks.md + sidebar entry.
- README.md and CHANGELOG.md updates.
Tests (129 new, all green):
- tests/test_cursor_hooks_shell.py — 75 behavioural tests for
the three hook scripts: kill switches, input parsing, counter
logic, loop prevention, pending markers, wing inference, logging.
- tests/test_cursor_hooks_install.py — 19 contract tests for the
installer: dry-run, idempotent merge, basename-matched uninstall,
refusal to overwrite malformed JSON.
- tests/test_cursor_plugin_manifest.py — 35 contract tests for the
plugin: manifest validity, version sync with mempalace.version,
mcp.json shape, skill/command frontmatter, default-discovery
layout invariants.
Design notes
------------
- Local-first and zero-API by default; hooks never call external
services. Same privacy model as the existing Claude Code hooks.
- Fail-open: hook scripts deliberately do not use set -e so a broken
hook can never block the user's conversation.
- Cursor preCompact cannot block + return a followup, so we
synchronously mine the transcript and drop a pending_save marker
that the next stop hook consumes — guarantees verbatim capture
before context window compression.
- Cursor's default plugin discovery requires real commands/, skills/,
and mcp.json at the plugin root (verified against the cached
cloudflare plugin); .cursor-plugin/{commands,skills} are convenience
symlinks back to those canonical locations.
- bash 3.2 compatibility throughout: avoids heredoc-in-command-
substitution parser bugs; uses python -c for JSON parsing;
basename-matched entry recognition in install.sh.
- All changes are additive. No existing files are removed, no
existing hooks change behaviour, and no new runtime dependencies
are introduced.
Co-authored-by: Cursor <cursoragent@cursor.com>
ChromaDB 1.5.x calls embedding_function.embed_query(input=...) via
keyword argument during collection.query(). EmbeddinggemmaONNX lacked
both embed_query and embed_documents methods, causing:
TypeError: embed_query() got an unexpected keyword argument 'input'
whenever semantic search was triggered.
This patch adds the two methods required by the ChromaDB EF protocol,
using (the ChromaDB kwarg name, noqa A002) so that palace
search works correctly with the embeddinggemma model.
Also downloads the companion ONNX file alongside the main model
to prevent runtime InferenceSession failures.
Fixes silent search failures when is set to
embeddinggemma.
Address review feedback: `with sqlite3.connect() as conn:` only
manages transactions, it does not close the connection. An open
connection before PersistentClient instantiation can leave WAL state.
Use explicit `try...finally: conn.close()` matching the read-only
helpers elsewhere in the module.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
chromadb <= 1.5.8 writes config_json_str = '{}' (empty JSON) when
creating collections. chromadb 1.5.9 introduced a strict _type check
in the collection config deserialization path -- its absence raises
KeyError: '_type' on palace open. Since the pin allows >=1.5.4,<2,
any upgrade pulls 1.5.9 and breaks every existing palace.
Add a fourth pre-open migration step (_fix_missing_collection_type)
that injects "_type": "CollectionConfigurationInternal" into
collections.config_json_str rows that lack it. Same lifecycle and
marker-file pattern as the existing _fix_blob_seq_ids.
Co-Authored-By: nautis <nautis@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Brings in the three commits that landed on main during the v3.3.6
release cycle but not on develop:
- 62a555c fix(release): align ruff pin to 0.15.14 + hoist COCA
imports out of hot paths (#1614)
- a1cf052 Merge pull request #1614 from MemPalace/fix/release-3.3.6-ruff-pin-align
- db1fbe8 Merge pull request #1610 from MemPalace/release/3.3.6
Resolves conflicts in miner.py and palace.py where develop's #1613
(known-systems lexicon, Tier 3) had added _apply_known_systems_prepass
to module-top imports + the prepass call in build_closet_lines. Main
side had only the COCA hoist from #1614. Resolution: keep develop's
side (the prepass additions).
Mirrors the v3.3.5 sync-back pattern (#1442).
Resolves conflicts in miner.py and palace.py against the COCA-hoist
work from #1612 just merged. Both files had local imports for
_get_coca_filter / _candidate_entity_words in the function bodies;
after #1612 those moved to module top. Resolution: hoist
_apply_known_systems_prepass alongside _get_coca_filter at module
top in both files, drop the now-empty local imports.
No behavior change; just removes the local-import duplication that
would have been left in place if the merge auto-resolved naively.
Addresses gemini-code-assist MEDIUM finding on PR #1613: the previous
implementation of _apply_known_systems_prepass compiled a regex pattern
for every compound on every call, repeating the work on every drawer
mined and every closet built. With 59 compounds × N drawers, that's
59N re.compile() calls for a workload where the patterns never change.
The fix moves compilation into _get_known_systems (already lru_cache'd
to size=1), which now returns tuple[tuple[str, re.Pattern], ...] —
pairs of (canonical name, pre-compiled case-insensitive word-bounded
regex). _apply_known_systems_prepass consumes the cached tuple and
does zero compilation in the hot path.
Behavior is identical: same word boundaries, same case-insensitive
matching, same longest-first ordering, same graceful-degrade on
malformed json. All 74 entity_detector tests still pass on macOS plus
the full 2258-test suite on Linux Py 3.9 / 3.11 / 3.13.
Adds a curated list of multi-word product/system names ("Claude Code",
"GitHub Copilot", "Visual Studio Code", "GPT-4", …) and a compound
pre-pass that detects them atomically before the existing single-word
extraction runs. Without this, the regex-based detector decomposes
"Claude Code" into "Claude" + "Code" — and the COCA filter (shipped
in v3.3.6) then drops "Code" as a content word, leaving "Claude" alone
with the wrong attribution.
What ships
- mempalace/data/known_systems.json — 59 curated compounds covering
common AI assistants, IDEs, model names, cloud platforms, and
Office/Google apps. Each entry is multi-word or hyphenated;
single-word product names ("ChatGPT", "Cursor") have no
decomposition risk and stay handled by the existing regex.
- mempalace/entity_detector.py — _get_known_systems() (cached loader,
mirrors _get_coca_filter from Tier 2) and _apply_known_systems_prepass
which scans for each compound case-insensitively with word boundaries,
counts occurrences, and returns the masked text + count dict so the
subsequent single-word + multi-word loops don't re-decompose.
- mempalace/miner.py and mempalace/palace.py — same pre-pass wired
into _extract_entities_for_metadata (per-drawer tagger) and
build_closet_lines (closet pointer construction). Without these,
the new behavior would only apply at init-time and per-drawer
metadata would still decompose compounds.
How it interacts with Tier 2
Tier 2 (COCA filter) blocks single-word content nouns like "Code"
and "Brutal". Tier 3 protects multi-word product names so they
don't get decomposed in the first place. They complement each
other: the compound pre-pass runs FIRST and masks compounds out
of the text; the COCA filter then runs on the remaining
single-word candidates.
Behavior verification
Before this PR, mining a document containing "Claude Code wrote the
patch" three times emitted entities:
Claude;Claude Code;Code (Code filtered by COCA);
After this PR, the same document emits:
Claude Code
The standalone "Claude" no longer appears (it never actually appeared
alone in the source) and decomposition stops at the compound boundary.
Tests
Nine new tests in tests/test_entity_detector.py covering:
- "Claude Code" detected as atomic compound at extract_candidates
- "Claude" alone NOT in results when only mentioned as part of compound
- Case-insensitive compound matching (claude code, CLAUDE CODE, etc.)
- Single-word "Code" still filtered by COCA (no Tier 2 regression)
- Single-word real name "Aya" still detected (no regression on names)
- Multiple distinct compounds in one text both detected
- Unknown two-word phrase still detected via existing multi-word regex
- known_systems.json ships with expected schema (>=20 entries, all multi-token)
- known_systems.json contains expected high-value entries
Verification
Full pytest 2267 passed / 3 skipped on macOS, coverage 85.34%.
Linux Py 3.9 / 3.11 / 3.13 via CI-matching pip install -e ".[dev]":
2258 passed each. End-to-end mine of a compound-rich corpus
confirms chromadb entities metadata now shows compounds atomic
(Claude Code, GPT-4, GitHub Copilot, Visual Studio Code) with no
decomposition.
The COCA content-word filter shipped in PR #1605 imported
`_get_coca_filter` and `_candidate_entity_words` locally inside two
hot paths:
- `palace.build_closet_lines` — runs per source file during mine
- `miner._extract_entities_for_metadata` — runs per drawer during mine
Both imports are now at module top, where they're resolved once at
import time instead of on every per-drawer call. Module-top imports
also make the dependency graph visible to static analysis (pylint's
C0415 was flagging the locals).
No behavior change. The `_get_coca_filter()` call is unchanged — only
the import statement moved. End-to-end mining produces identical
chromadb output. Addresses the MEDIUM finding gemini-code-assist
raised on PR #1605 review.
Verification: full pytest 2258 passed / 3 skipped / coverage 85.35%.
ruff check + format clean. Linux Py 3.9 / 3.11 / 3.13 via CI-matching
`pip install -e ".[dev]"`: 2249 passed each. End-to-end mine of a
test corpus produces the expected drawer + closet pointer.
The scam-alert and Claude-Code-retention admonitions were the first
content visitors saw on the repo page — louder than the project
introduction. Moves both below the logo/title/badges so the project
identity reads first, and softens the scam block (drops the H1
"CRITICAL SECURITY WARNING" + all-caps shouting + redundant emoji)
to a single-paragraph CAUTION. All factual content preserved:
impostor-domain warning, official sources, malware caveat, link to
docs/HISTORY.md.
Add _try_continue_json() normalizer for Continue.dev AI assistant
sessions (~/.continue/sessions/*.json). Parses history array with
role/content pairs, handles tool calls, system messages, and metadata.
Closes#59 (partial — adds Continue.dev format support)
Includes comprehensive test coverage for valid sessions, edge cases,
malformed input, and unicode content.
Adds _try_gemini_json parser to normalize.py for three layouts:
1. Gemini API contents format (~/.gemini/sessions/*.json):
{"contents": [{"role": "user", "parts": [{"text": "..."}]}, ...]}
2. Messages-wrapper variant:
{"messages": [{"role": "user", ...}, {"role": "model", ...}]}
3. Flat top-level list with role="model".
This complements the existing _try_gemini_jsonl parser (which handles
~/.gemini/tmp/<hash>/chats/session-*.jsonl with session_metadata
sentinel) — JSONL covers Gemini CLI runtime sessions, JSON covers
exported / Studio-saved transcripts.
## Review feedback addressed (PR #204)
bgauryy review:
- #1 Parser-precedence bug: _try_gemini_json runs *before*
_try_claude_ai_json so the {"messages":[..., role=model, ...]}
layout is no longer silently claimed by the Claude parser. The
Gemini parser's has_model_role guard prevents false-positives
against Claude / ChatGPT data.
- #2 Layout 2a coverage: TestGeminiJson.test_messages_wrapper_format
+ test_messages_wrapper_does_not_get_claimed_by_claude pin the
fix in place.
- #3 Test conflicts with current main: rebased onto develop;
tests restructured into TestGeminiJson class.
- #4 tempfile/os.unlink → pytest tmp_path everywhere.
- #5 elif not text → else (the elif branch was dead).
- #6 Module docstring updated to mention Google AI Studio.
Tests: 9 new cases in TestGeminiJson covering all three layouts,
multi-part text joining, non-text part skipping, has_model_role
disambiguation, dispatch-chain regression for review #1.
Add _try_pi_jsonl parser for Pi agent session files stored at
~/.config/pi/agent/sessions/{encoded-cwd}/{timestamp}_{uuid}.jsonl.
Uses type "message" entries with role "user"/"assistant". Skips
toolResult messages, model_change, thinking_level_change, and other
operational events. Requires session header (type "session" with
"version" key) to avoid false positives.
Format documented at github.com/badlogic/pi-mono session.md and
verified via Context7. Sample data provided by tunnckoCore in #59.
Refs: #59
- Updated READABLE_EXTENSIONS in miner.py to include ".swift", ".kt", and ".kts".
- Added tests in test_miner.py to ensure scanning includes Swift and Kotlin files.
Two bugs found in production use with a ChromaDB palace of 1200+ drawers
ingested via mixed paths (bulk import + MCP tool calls):
1. searcher.py: filtered search (wing= or room=) crashes with "Error finding
id" when the HNSW vector index is out of sync with the SQLite metadata
store. The outer try/except swallowed the error as a search failure.
Fix: inner try/except catches filter failures, retries unfiltered with
n_results*15 (capped at 500), and post-filters by wing/room in Python.
Degrades gracefully instead of returning an error.
2. mcp_server.py: diary_write requires 'entry' but add_drawer uses 'content',
making it natural to pass content= by analogy. The mismatch returns a
silent MCP -32000 error with no explanation.
Fix: accept 'content' as an alias for 'entry' with a clear error message
if neither is provided.
Both bugs were diagnosed and patched in a live palace. This contributes the
fixes upstream.
Adds .cs, .csproj, .sln, .razor, and .cshtml so C#/.NET projects
are indexed by the project miner. .razor/.cshtml are analogous to
the already-supported .jsx/.tsx.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
description:"Recall protocol for MemPalace — search the palace before answering about past work, people, projects, or prior decisions. Apply when the user asks what was decided, what happened before, who someone is, what was discussed last time, or anything that may already be filed in their memory palace; or when mempalace-recall is invoked. Complements the mempalace setup skill and requires the mempalace-mcp server."
---
# MemPalace Recall
Search-before-answer protocol for MemPalace. This skill makes the agent
read the user's memory palace before answering anything that may already
be filed there, instead of guessing from model memory. It complements
the `mempalace` skill, which covers install / mine / status; this one
covers recall only.
## Step 0 — Verify MemPalace is available
Before relying on recall, confirm MemPalace is installed and reachable:
- Official release page: <https://github.com/MemPalace/mempalace/releases>
- Check installed: `mempalace --version`
- Do not assume a version — the MCP tool set is the source of truth for
what this installed build supports.
If the `mempalace_*` MCP tools are not available, tell the user the
server is not connected and point them at the `mempalace` skill to set
it up. Do not silently fall back to answering from model memory.
## Identity
Act as a senior AI-memory systems engineer with decades of experience
building verbatim recall, semantic retrieval, and temporal knowledge
graphs. Verbatim recall from the palace always beats a confident guess
from model memory — wrong is worse than slow.
## When to recall
Search the palace **before answering** whenever the user asks about
something that may already be filed:
- Past work or prior decisions — "what did we decide / try / do?"
- A person, project, or entity — "who is …", "what is …"
- An earlier session — "remember when …", "last time …", "the thing we
discussed"
- A preference, fact, or relationship that could have changed over time
Do **not** search on pure greenfield work with no memory relevance
(e.g. "rename this variable", "fix this typo"). Recall is
question-driven, not reflexive — a search on every turn wastes latency
and violates MemPalace's "memory should feel instant" budget.
## Protocol
1. On wake-up, the MemPalace PreInvocation hook injects verbatim palace
content via `injectSteps[].ephemeralMessage` on the first model call
of a conversation. If memory was injected, start from it before
searching further.
2.**Before responding** about people, projects, past events, or prior
decisions: call `mempalace_search` first. For relational or temporal
facts ("who reported to whom in March", "what was true then"), call
`mempalace_kg_query` instead or as well.
3.**If unsure** about a fact (name, age, relationship, preference): say
"let me check the palace" and query. Wrong is worse than slow.
4.**Return verbatim.** Quote the drawer's exact stored words. Never
summarize, paraphrase, or lossy-compress what the palace returns —
that is the whole point of the system.
5.**After a substantive session**, record continuity with
`mempalace_diary_write` (background hooks may already do this — do not
double-file).
6.**When a fact changes**, call `mempalace_kg_invalidate` on the old
fact, then `mempalace_kg_add` for the new one.
The full canonical protocol — shared verbatim with the Antigravity
description:MemPalace — mine projects and conversations into a searchable memory palace. Use when the user asks about MemPalace, memory palace, mining memories, searching memories, palace setup, wings, rooms, or drawers; or when they want to recall past work that may already be filed in their palace.
---
# MemPalace
A searchable memory palace for AI — mine projects and conversations, then search them semantically. Verbatim storage, local-first, zero external API by default.
MemPalace exposes operation-specific instructions through the CLI so this skill stays accurate as MemPalace evolves. To get instructions for any operation:
```bash
mempalace instructions <command>
```
Always prefer the CLI output over what is written here when the two disagree — the CLI is the single source of truth for the installed version.
## Common operations
These are the five operations users ask for most often. Each one wraps a single MemPalace CLI subcommand. The `mempalace instructions <name>` form returns the full, version-correct guidance.
### `help` — discover what MemPalace can do
```bash
mempalace instructions help
```
Use when the user is new, unsure what's possible, or asks "what can you do".
### `init` — first-run setup of the palace
```bash
mempalace instructions init
```
Use when the user has just installed MemPalace, no palace exists yet, or the user explicitly asks to set up / configure / re-initialize their palace.
### `mine` — ingest a project or conversation directory
```bash
mempalace instructions mine
```
Use when the user wants to fold a project's files into their palace, or to ingest exported conversation transcripts into the palace as searchable memory.
### `search` — find verbatim memories by semantic query
```bash
mempalace instructions search
```
Use when the user wants to recall something from the past, find a previous decision, or rediscover code/notes/conversations they already wrote.
### `status` — what's in the palace right now
```bash
mempalace instructions status
```
Use when the user asks "what's in my palace", "how big is my palace", or wants a summary of wings, rooms, and drawer counts.
## MCP tools (preferred over CLI)
Inside Antigravity, the MemPalace MCP server registers a rich set of tools. Use these instead of shelling out to the CLI for live operations (search, diary writes, drawer adds, knowledge graph queries, palace status). The MCP tools always reflect the current palace state without spawning a subprocess.
The MCP server is auto-registered when this plugin is installed at `~/.gemini/config/plugins/mempalace/`. If the server does not appear in Antigravity's MCP store, run `mempalace-mcp --version` to verify the binary is on PATH, then restart Antigravity.
## Design principles (verbatim from the project)
- **Verbatim always** — never summarize, paraphrase, or lossy-compress user data.
- **Local-first, zero external API by default** — extraction, embedding, and LLM-assisted refinement happen on the user's machine.
- **Privacy by architecture** — the system never calls out to external services for core operations.
- **Performance budgets** — hooks under 500ms; startup injection under 100ms.
- **Background everything** — filing, indexing, and timestamps happen via hooks in the background; zero tokens spent on bookkeeping in the chat window.
If a request would violate any of these principles, refuse and explain — even if it would be technically convenient.
A Claude Code plugin that gives your AI a persistent memory system. Mine projects and conversations into a searchable palace backed by ChromaDB, with 19 MCP tools, auto-save hooks, and 5 guided skills.
A Claude Code plugin that gives your AI a persistent memory system. Mine projects and conversations into a searchable palace backed by ChromaDB, with 36 MCP tools, auto-save hooks, and 5 guided skills.
## Prerequisites
@@ -41,16 +41,17 @@ After installing the plugin, run the init command to complete setup (installs th
## Hooks
MemPalace registers two hooks that run automatically:
MemPalace registers three hooks that run automatically:
- **Stop** -- Saves conversation context every 15 messages.
- **SessionEnd** -- Runs one final save in the background on a clean exit, so short sessions that never hit the Stop interval or a compaction are still captured.
- **PreCompact** -- Preserves important memories before context compaction.
Set the `MEMPAL_DIR` environment variable to a directory path to automatically run `mempalace mine` on that directory during each save trigger.
## MCP Server
The plugin automatically configures a local MCP server with 19 tools for storing, searching, and managing memories. No manual MCP setup is required -- `/mempalace:init` handles everything.
The plugin automatically configures a local MCP server with 36 tools for storing, searching, and managing memories. No manual MCP setup is required -- `/mempalace:init` handles everything.
description:Recall protocol for MemPalace — search the palace before answering about past work, prior decisions, people, or projects. Use when the user asks what was decided, what happened before, who someone is, what was discussed last time, or anything that may already be filed in their memory palace.
allowed-tools:Bash
---
# MemPalace Recall
Search-before-answer protocol for MemPalace. Read the user's memory
palace before answering anything that may already be filed there,
instead of guessing from model memory. This complements the `mempalace`
skill (install / mine / status); this one covers recall only.
## Step 0 — Verify MemPalace is available
```bash
mempalace --version
```
If the `mempalace_*` MCP tools are not available, tell the user the
server is not connected and point them at the `mempalace` skill or
`/init`. Do not silently fall back to answering from model memory.
## When to recall
Search the palace **before answering** whenever the user asks about
something that may be filed:
- Past work or prior decisions — "what did we decide / try / do?"
- A person, project, or entity — "who is …", "what is …"
- An earlier session — "remember when …", "last time …"
- A preference, fact, or relationship that could have changed over time
Skip recall for pure greenfield work with no memory relevance (renaming
a variable, fixing a typo). Recall is question-driven, not reflexive.
## Protocol
1. Before responding about people / projects / past events / prior
decisions: call `mempalace_search` first. Use `mempalace_kg_query`
for relational or time-bound facts.
2. If unsure about a fact: say "let me check the palace" and query.
3. Return the drawer's **verbatim** text — never summarize or paraphrase
stored content.
4. After a substantive session, record continuity with
`mempalace_diary_write` (skip if a background hook already saved).
5. When a fact changes: `mempalace_kg_invalidate` the old fact, then
`mempalace_kg_add` the new one.
## Unhappy paths
- **Empty results** — say the palace has nothing on this; do not invent
an answer. Offer to widen the search or file the new information.
- **MCP error / server down** — surface the error, suggest `mempalace
status` or re-running `/init`; never fall back to guessing.
- **Palace index corrupt / compactor error** — if the server reports an
HNSW segment-writer error, a ChromaDB compaction failure, or stays
"Not connected" after a write, the index is out of sync with
`chroma.sqlite3` but the rows are intact. Tell the user to stop the
server and rebuild from SQLite (`mempalace repair --mode from-sqlite
--archive-existing --yes`), not re-mine, which drops MCP-added drawers
and diary entries (#1843). Do not repair in-process.
- **Conflicting facts** — trust the knowledge graph's time-valid answer;
invalidate-then-add rather than overwriting silently.
The canonical protocol, shared across all MemPalace integrations, lives
Give your AI a persistent memory -- mine projects and conversations into a searchable palace backed by ChromaDB, with 19 MCP tools, auto-save hooks, and guided skills.
Give your AI a persistent memory -- mine projects and conversations into a searchable palace backed by ChromaDB, with 36 MCP tools, auto-save hooks, and guided skills.
"description":"Give your AI a memory — mine projects and conversations into a searchable palace. 19 MCP tools, auto-save hooks, and guided setup.",
"version":"3.7.0",
"description":"Give your AI a memory — mine projects and conversations into a searchable palace. 36 MCP tools, auto-save hooks, and guided setup.",
"author":{
"name":"milla-jovovich"
},
@@ -18,16 +18,11 @@
"search"
],
"skills":"./skills/",
"hooks":"./hooks.json",
"mcpServers":{
"mempalace":{
"command":"mempalace-mcp"
}
},
"mcpServers":"./.mcp.json",
"interface":{
"displayName":"MemPalace",
"shortDescription":"AI memory system for Codex",
"longDescription":"Give your AI a persistent memory — mine projects and conversations into a searchable palace backed by ChromaDB, with 19 MCP tools, auto-save hooks, and guided skills.",
"longDescription":"Give your AI a persistent memory — mine projects and conversations into a searchable palace backed by ChromaDB, with 36 MCP tools, auto-save hooks, and guided skills.",
A Cursor IDE plugin that gives your agent a persistent memory system. Auto-registers the `mempalace-mcp` server (36 MCP tools), ships 5 slash commands, two model-invocable skills (setup/mining/search and a recall protocol), and an optional recall rule.
> Hooks (auto-save + session-start memory recall) are shipped separately under `hooks/cursor/` so the plugin is safe to install in any Cursor workspace without touching the agent loop. See [Hooks](#hooks-optional) below.
Then in Cursor: <kbd>Cmd</kbd>-<kbd>Shift</kbd>-<kbd>P</kbd> → **Developer: Reload Window**.
### Marketplace
Once published, install via the Cursor marketplace panel and select `mempalace`. Required-plugin distribution from a team marketplace is also supported.
## Post-Install Setup
After installing the plugin, run the `init` command in a Cursor chat:
```
/mempalace-init
```
(Or just say "use the mempalace skill" — Cursor will model-invoke the bundled skill.)
This installs the `mempalace` package via `uv tool` or `pip`, initializes a palace under `~/.mempalace/`, and verifies the MCP server is reachable.
> Cursor commands are global, not plugin-namespaced — that's why each slug is prefixed with `mempalace-` rather than appearing as `/help`, `/init`, etc. This keeps them collision-free with built-in or other-plugin commands.
## Skills
Two model-invocable skills ship at the plugin root under `skills/`:
| Skill | What it does |
|-------|--------------|
| `mempalace` | Setup, mining, status, and the dynamic `mempalace instructions` CLI. |
| `mempalace-recall` | Search-before-answer protocol — makes the agent read the palace before answering about past work, people, projects, or prior decisions instead of guessing. |
Cursor surfaces these automatically when a request matches their description, or you can attach them explicitly.
## Recall rule (optional)
The plugin also ships a Cursor rule at the plugin root under `rules/mempalace-recall.mdc`:
```yaml
description:When the user asks about past work, prior decisions, people, ... call mempalace_search before answering ...
alwaysApply:false
```
It is `alwaysApply: false` on purpose — Cursor loads it only when its matcher judges the turn recall-relevant, so it never fires on unrelated coding work and never adds MCP latency to greenfield tasks. The rule, the `mempalace-recall` skill, and the `sessionStart` hook all reference the same canonical protocol in [`integrations/shared/recall-protocol.md`](../integrations/shared/recall-protocol.md).
Want recall forced into **every** conversation regardless of context? Copy the aggressive `alwaysApply: true` variant from [`examples/cursor/rules/`](../examples/cursor/rules/README.md) into `~/.cursor/rules/`. That is a deliberate, heavier opt-in, not a default.
## MCP Server
This plugin ships `mcp.json` at the plugin root, so Cursor auto-loads the `mempalace-mcp` server on plugin install:
```json
{
"mcpServers":{
"mempalace":{
"command":"mempalace-mcp"
}
}
}
```
All 36 MemPalace MCP tools (`mempalace_search`, `mempalace_add_drawer`, `mempalace_diary_write`, `mempalace_check_duplicate`, `mempalace_diary_read`, …) become available to the agent immediately. No manual `~/.cursor/mcp.json` edit required.
If the server doesn't appear, confirm `mempalace-mcp` is on the user `$PATH`:
```bash
command -v mempalace-mcp
```
If it isn't, run `/init` (or `mempalace install` from a terminal) — `mempalace-mcp` is installed alongside the `mempalace` package.
## Hooks (optional)
Cursor's hooks system is configured separately from plugins (in `~/.cursor/hooks.json` or `.cursor/hooks.json`), so this plugin does **not** wire hooks itself. The MemPalace repository ships three Cursor-native hooks under [`hooks/cursor/`](../hooks/cursor/) that you install with one command.
User scope — writes `~/.cursor/hooks.json`, applies to every Cursor workspace (recommended):
```bash
hooks/cursor/install.sh --scope user --variant full
```
Project scope — writes `.cursor/hooks.json` under the current project only:
```bash
hooks/cursor/install.sh --scope project --variant full
"description":"AI memory system — mine projects and conversations into a searchable palace. 36 MCP tools, slash commands, and a guided skill for Cursor.",
"description":"Give your AI a memory — mine projects and conversations into a searchable palace. 36 MCP tools, slash commands, and a guided skill for Cursor.",
@@ -6,6 +6,197 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
---
## [Unreleased]
---
## [3.7.0] — 2026-08-11
### Features
- **Embeddings via any OpenAI-compatible `/v1/embeddings` endpoint.** New `embedding_model: "openai-compat"` option computes embeddings on a server (LM Studio, llama.cpp, vLLM, Ollama's OpenAI shim, or a self-hosted endpoint) instead of a local ONNX model — useful for larger/multilingual embedders such as Qwen3-Embedding, or GPU offload. New `OpenAICompatEmbeddingFunction` in [`mempalace/embedding.py`](mempalace/embedding.py) speaks the standard `/v1/embeddings` protocol over stdlib `urllib` (no new dependency), batches requests, re-sorts the response by `index`, and L2-normalizes for the cosine collection. Endpoint settings are resolved by `MempalaceConfig` as a single source of truth — `embedding_api_url` / `embedding_api_model` / `embedding_api_key` in `config.json`, each overridable via the matching `MEMPALACE_EMBEDDING_API_*` env var. The embedding function's `name()` encodes the model id so changing it forces `mempalace repair rebuild-index` (different vector space). Mirrors the existing `openai-compat` LLM provider naming; stays local when the endpoint is on your machine/LAN. (#1559)
- **`mempalace_search` / `mempalace search` — `since`/`before` date window.** Semantic search is poor at temporal queries ("what did we discuss this week?" scores ~0.35 even when matching drawers exist), so the search surfaces now accept the same `[since, before)` window `list_drawers` gained in #1128: inclusive/exclusive ISO bounds compared wall-clock against each drawer's `filed_at`, undated drawers excluded while a bound is active. The window applies on every candidate path (vector, `candidate_strategy="union"`, and the BM25-only fallback); the vector candidate pool widens under an active window (ChromaDB cannot range-compare string metadata server-side), and a full pool is flagged via `date_filter_pool_truncated` instead of passing silently. Shared parsing lives in the new `mempalace.date_window` module, also backing the `list_drawers` filter. (#463)
- **Hermes agent memory provider (core).** In-package Hermes `MemoryProvider` files live turns through the shared `file_conversation_exchange()` path (same metadata as convo mining), with a background worker so the agent loop never blocks on Chroma. Wake-up L1 uses the long-lived collection only — a second PersistentClient is not opened, avoiding local SQLite races. Install/backfill/docs remain a stacked follow-up. (#1915, #2215)
- **Explicit RFC 002 source adapters on `mempalace mine`.** `mempalace mine <source> --source <adapter>` resolves adapters through the registry, holds the palace writer lease for the full ingest, and keeps dry-runs inert (no real backend/KG open). Legacy `--mode` paths are unchanged. (#2068, #2062)
- **MCP refuses writes when the served library is no longer the one that started.** Detects mempalace (and chromadb when that backend is active) version drift or uninstall after upgrade without restarting the server; opt-out via `MEMPALACE_MCP_ALLOW_STALE_LIBRARY`. (#2081, #899)
- **Hook write-routing through the daemon.** Background hook saves and mines can honor the shared write-routing policy so multi-session setups serialize mutations through one local owner instead of racing the palace. (#2030, #1963)
### Performance
- **EmbeddingGemma groups documents by size before sub-batching.** The tokenizer pads every row of a sub-batch to the longest sequence in it, so arrival order decided the bill: one long verbatim message dragged a whole sub-batch up to its own length. Measured over 43,157 `sweep` drawers from 160 Claude Code transcripts, padded token slots drop 39.7% and the quadratic attention term 45.0%. Vectors move by at most one float32 ULP (1.2e-07 absolute, cosine 0.99999992), which is reduction-order rounding and not a change of meaning. Applies to `embedding_model: embeddinggemma` only; the default MiniLM embedder pads to a fixed width and was never affected. (#2104)
- **HNSW capacity probes are cached** and invalidated by palace file signature, so repeated MCP status/taxonomy paths no longer re-scan native segment files on every call. (#2051, #1471)
- **`chunk_text` line numbering is O(N)** via incremental tallies, fixing multi-second hangs on large sources. (#2054, #2055)
### Bug Fixes
- **Orphaned per-source-file mine locks are reaped instead of accumulating forever.** `_cleanup_mine_lock_file` reclaims a lock correctly on the happy path, but only for the specific lock its own `mine_lock` context manager just released — a process killed abruptly (SIGKILL, force-quit, host crash) never reaches that cleanup, and nothing else revisited the file afterward. One long-lived installation was found with 5,636 stale entries in `~/.mempalace/locks/`, the oldest several months old, none held by any live process. `mine_lock` now opportunistically reaps locks older than an hour via the same nonblocking-flock-reacquire safety check `_cleanup_mine_lock_file` already uses, throttled to once per 15 minutes so it costs nothing on the common path. `mine_palace_*.lock` (the newer per-palace lock) is untouched — it has its own lifecycle and holder tracking.
- **Windows MCP stdout capture falls back cleanly** when fd-level redirection is unavailable, and fails closed if protocol stdout cannot be restored after a successful redirect. (#2211, #2210)
- **Claude Code `subagents/` transcripts are skipped by default** when mining conversations (98%+ noise on typical workspaces); `--include-subagents` opts back in. (#1330, #1217)
- **Worktree transcript cwd no longer mints a throwaway wing** — wing is derived from the project root above `.claude/worktrees/`. (#2206)
- **Markdown emphasis/bold no longer scores as emotional content** in the general extractor. (#2199, #2197)
- **Encoding hardening on Windows locales:** pin `encoding=utf-8` on config/dialect opens; safer mojibake repair that does not destroy clean Portuguese/Vietnamese/Turkish prose; repair-encoding CLI reconfigures stdio like other entry points; remaining non-ASCII CLI symbols replaced for GBK consoles. (#2098, #2208, #2194, #1104, #1034, #2193)
- **`rebuild_index` holds the palace writer lease** for the full snapshot→rebuild/swap cycle so concurrent writers cannot recreate HNSW divergence mid-repair. (#2195)
- **`MEMPALACE_PALACE_PATH` is restored after each service entrypoint** so multi-call processes do not leak the first call's palace into the second. (#2192)
- **systemd unit uses `Restart=always`** and documents `MEMPALACE_MCP_IDLE_HOURS=0` so the idle watchdog does not leave a dedicated server down after a clean exit. (#2203, #2204)
- **Mining works again on the default Chroma backend.** Once Chroma began declaring `requires_explicit_embeddings`, every write started routing through `EmbeddingCollection`, whose `_embed_texts` built rows with `list(ndarray)` — that unpacks into `np.float32`*scalars*, which chromadb rejects outright (`Expected embeddings to be a list of floats or ints, a list of lists, a numpy array, or a list of numpy arrays`). `mine`, and every other write against a default palace, aborted on the first drawer. Vectors now convert to real Python floats. The suite was structurally blind to this: conftest's autouse embedding fixture replaces `_embed_texts` itself for every module outside `test_embedding` / `test_embeddinggemma`, so the defective function was never executed under test — the regression tests therefore live in `test_embedding.py`, where that stub does not apply. (#2187)
- **ChatGPT data exports are parsed instead of stored as raw JSON.** A real `conversations.json` is a top-level array of conversations, which no parser claimed, so `mine --mode convos` chunked the raw JSON and lost every speaker turn while reporting success. Each conversation now normalizes to its own transcript, as Claude.ai privacy exports already do, so per-conversation dedup survives a re-export. The ChatGPT parser also type-checks its nested shapes, so an unrelated array carrying a `mapping` key is declined instead of raising. (#2160)
- **Local backends enforce process-lifetime single-writer ownership.** File-backed and unknown backends require one writer owner for the full process lifetime (daemon holds the lease until workers exit; writable MCP HTTP acquires ownership before bind and refuses startup when blocked). Read-only MCP may coexist; `sqlite_exact` opens genuine query-only/immutable readers; remote Milvus/Zilliz remain multi-process. Addresses multi-writer SQLite/WAL corruption from MCP HTTP + daemon + mine topologies. (#2079, #2045)
- **Chroma HNSW write defaults match chromadb** (`batch_size=100` / `sync_threshold=1000`) instead of the old 2/2 bloat guard that rewrote segments thousands of times on large mines. (#2107, #2106)
- **Repair and recovery are safer under contention.** `repair --mode from-sqlite` takes the mine-lock before archiving; rebuilds preserve a verified temp collection when the live swap fails; sparse drawers with zero `embedding_metadata` rows are no longer dropped; truncated ID pagination fails loud instead of pretending success. (#2109, #2086, #2087)
- **`repair --mode from-sqlite --dry-run` is a true preview.** It no longer archives or re-embeds; it prints per-collection would-be counts from SQLite ground truth and exits without touching the palace. Unreadable counts fail closed instead of inventing zeros. (#2133, #2095, #1654)
- **`repair --dry-run` is a true preview in the default (legacy) mode too.** That path ignored the flag entirely and ran the real rebuild — deleting any existing `<palace>.backup`, copying the palace over it, and re-filing the drawers collection. It now prints a read-only plan and exits without opening a chromadb client, which is itself a write to `chroma.sqlite3`. The plan names the live-collection delete the rebuild performs, warns when an existing backup would be destroyed, and reports the truncation guard as disabled when `--confirm-truncation-ok` is set. An isolated FTS5 inverted-index error is reported as auto-healable instead of raising the manual-recovery abort a real run never reaches, unreadable counts fail closed with a non-zero exit, and the `--dry-run` help no longer claims to be `--mode max-seq-id` only. (#2144)
- **`repair` and `migrate` survive a socket or a named pipe in the palace directory.** Both take a whole-directory `shutil.copytree` backup before they overwrite the palace, and `copytree` cannot duplicate a Unix domain socket left beside `chroma.sqlite3`, nor a named pipe: it copied everything else and then raised `shutil.Error`, so the command died at the backup step before any rebuild ran, and re-running only repeated it. Neither entry carries palace data, so both are now skipped and named in the output while the backup completes. Device nodes are skipped too, because the copy dereferences them instead of failing on them. Every other copy failure still aborts the command, including a symlink whose target cannot be read: no errno separates a deleted target from one on a volume that is not mounted, so an entry that may have held data is left for the copy to fail on. (#2207)
- **HNSW divergence is preflighted before remaining `col.count()` crash sites** across mine, dedup, migrate, repair, and palace helpers. (#2093)
- **Re-mine and conversation ingest no longer lose or duplicate drawers.** Content-hash dedup prevents duplicate LLM conversation drawers; sweeper drawers are excluded from convo extract-mode purge scope and failed purges abort; search returns round-trippable `drawer_id` values for `get_drawer`. (#2050, #2125, #2089, #2090, #2044, #2080)
- **MCP and daemon lifecycle harden multi-agent use.** Read-only mode refuses config and checkpoint-ack tools that rewrite host state; stdio MCP exits on stdin EOF/broken pipe so orphaned sessions release locks; daemon jobs refused the palace lock are deferred instead of failed permanently. (#2126, #2103, #2101, #2072, #2029, #2014)
- **Entity-candidate extraction no longer hangs on long ASCII runs** (base64, minified blobs) while preserving CJK/non-ASCII text. (#2127, #2065, #2063)
- **`docker-compose.yml` is valid again.** The `environment:` key was declared with only comments beneath it, which YAML parses as null, so Compose rejected the whole file (`services.mcp.environment must be a mapping`) — every documented Compose command failed before starting. The key is commented out along with its examples, which now use mapping syntax so uncommenting them yields a valid block. (#2188)
### Documentation
- Operator write-routing / single-writer recovery notes in `docs/write-routing-policy.md`. (#2079)
- Remote-server guide wording for read-only tools that change host state. (#2126)
- **The README's Docker section leads with the published image.** It previously documented only `docker build`, even though `ghcr.io/mempalace/mempalace` ships multi-arch — and a clone builds `develop`, not the release, so a build and a pull could differ silently. It now covers what the omissions actually cost: the MCP client config mounts a transcripts directory (without one the server starts and every mine finds nothing), the first embedding call downloads ~80 MB into `/data` and looks like a hang, bind mounts keep host ownership against the image's uid 1000 so a `0700` directory fails with a bare `PermissionError` on Linux (and `--user` is the wrong fix — `/data` is mode 700 owned by that uid), mining sources can be mounted read-only, and the GPU image is x86_64-only. (#2196)
### Internal
- **The Docker workflow runs the image before publishing it.** It previously built both images without ever starting a container or parsing a Compose file, so a green run only proved the Dockerfile compiled — which is how two defects that break the first documented command shipped past it. `scripts/docker-smoke.sh` now validates both Compose files, checks entrypoint dispatch, mines a mounted directory, asserts the drawer reads back verbatim from a separate container, and drives a real MCP stdio handshake; publication is gated on it. The script runs the same way locally: `scripts/docker-smoke.sh <image>`. (#2189)
---
## [3.6.0] — 2026-07-14
### Features
- **Turnkey secure remote / team server.** `mempalace serve` exposes the full MCP surface over HTTP with secure defaults: non-loopback binds require a bearer token, generated tokens are stored with restrictive permissions and passed through the child environment rather than argv, native TLS 1.2+ is supported, and `--read-only` both hides and refuses mutating tools. Docker Compose, systemd, and environment templates are included for deployment. (#1877, #1897, #1900)
- **Milvus storage backend.** The new opt-in `milvus` backend supports embedded Milvus Lite, self-hosted Milvus, and Zilliz Cloud through `pymilvus`, with COSINE vector search, native BM25 lexical search for new collections, namespace isolation, target-mismatch protection, and an optional `milvus` dependency extra. (#1899)
- **Atomic knowledge-graph fact replacement.** `supersede()` / `mempalace_kg_supersede` closes an open fact and opens its successor at one shared instant, so model, employer, address, and other single-valued facts can change without a hand-written invalidate/add race. Temporal queries now use half-open intervals at timestamp precision, returning only the successor at the transition boundary while preserving whole-day semantics for date-only facts. (#1913)
- **Conversation chronology is preserved.** Conversation drawers now retain transcript time as `authored_at` alongside ingest time, search results expose it, and exact hybrid-score ties prefer the more recently authored drawer. An idempotent, dry-run-first backfill script adds the metadata to existing palaces without re-embedding. `mempalace_list_drawers` also accepts inclusive `since` and exclusive `before` filing-time bounds. (#1890, #1891)
- **Mined sessions populate the associative graph.** A precision-biased, no-LLM structural extractor records code symbols, URLs, paths, and qualified identifiers as drawer entities; conversation mining then derives hallways from them. The new `mempalace hallways` CLI command exposes the graph. (#1894, #1895)
- **Per-project mining exclusions.** `exclude_patterns` in `mempalace.yaml` filters pre-scanned project files without changing the corrected `--limit` semantics. (#1213, #1953)
- **LaTeX project coverage.** `.tex` and `.bib` files are now treated as readable prose sources. (#1901)
### Performance
- **MCP initialization no longer waits for palace-wide integrity work.** stdio answers the JSON-RPC `initialize` request immediately while startup preflight runs on a background thread; a tool arriving mid-probe joins the same verdict instead of launching duplicate work. The startup SQLite probe is also skipped above a configurable size limit (512 MiB by default), while repair preflights remain strict. (#1911, #1987)
- **Qdrant metadata counts use server-side facets.** Taxonomy, overview, and status paths can count metadata remotely instead of fetching and tallying every record; the embedding wrapper now forwards the facet and bulk-metadata capabilities correctly. (#1868, #1898)
- **pgvector metadata-only fetches omit document payloads**, reducing transfer and decoding work on enumeration paths. (#1892)
### Bug Fixes
- **Chroma recovery distinguishes derived-index damage from data loss.** Valid all-layer-0 HNSW segments with an empty `link_lists.bin` are no longer quarantined repeatedly; isolated FTS5 inverted-index corruption is rebuilt from intact content during repair and after mining; embedded NUL bytes are sanitized before reaching ChromaDB. (#1716, #1872, #1878, #1927, #1928)
- **SQLite recovery is bounded, atomic, and honest.** Integrity checks wait up to 15 seconds for transient writer contention instead of reporting a healthy palace as corrupt. In-place recovery archives use atomic `os.rename` rather than `shutil.move`'s destructive copy/delete fallback. `repair --mode from-sqlite` now rebuilds FTS5, vacuums, and requires a clean final `PRAGMA quick_check`; cleanup failures return non-zero with recovery details instead of printing a false success banner. (#1945, #2015, #2017)
- **CLI search checks HNSW divergence before opening ChromaDB.** A diverged palace routes directly to the existing SQLite BM25 fallback, avoiding embedder or collection initialization against damaged native index state while leaving healthy Chroma and non-Chroma backends on their normal vector path. (#2016)
- **Writer leases recover instead of stranding servers read-only.** A server refused while a peer owns the palace now retries on each mutating call and promotes itself after the peer exits. Status remains read-only and cannot acquire the writer lease, the in-process re-entrant holder set is released even across asynchronous interruption, and `checkpoint` / `delete_by_source` are correctly classified as mutations for both read-only serving and peer-writer protection. (#1923, #1930, #1934, #1960, #1970, #1971)
- **Status reports only checks that actually ran.** Non-Chroma backends now mark the Chroma SQLite integrity check as not applicable rather than claiming an unperformed success. (#1931, #1946)
- **Conversation transcripts are treated as mutable.** Appended or rewritten sessions are purged and re-filed based on modification time, preventing later turns from being silently skipped. Claude Code `tool-results/` sidecars are excluded from conversation scans so raw machine dumps cannot flood the embedding space. (#1957, #2010)
- **Explicit palace selection now scopes derived graph state.** Project, conversation, and format mining write hallways and tunnels beside the selected `--palace` instead of leaking them into the ambient default palace. (#2018)
- **Remote and local server hardening.** Non-loopback HTTP now requires a token unless an explicit insecure override is supplied; optional-provider probes cannot reuse an ambient external key before the user selects that provider; hook transcript paths and file-copy/repair paths receive stricter local validation and no-follow handling. Importing the MCP server no longer clobbers the host application's root logger, HTTP writes can re-enter the process-wide palace lock safely, and routine client disconnects no longer emit server tracebacks. (#1859, #1860, #1864, #1885, #2003, #2004)
- **Mining and configuration correctness.** Oversized files produce a visible stderr warning, `~` in configured palace paths expands consistently, wing slugs handle special characters, and Windows background daemon / synchronous hook mines use `CREATE_NO_WINDOW`. (#923, #1852, #1857, #1863, #1865)
- **`mempalace init` handles non-ASCII `.gitignore` files on Windows.** The project-file ignore guard now reads and appends UTF-8 explicitly instead of relying on locale defaults such as GBK. (#1648)
- **L1 wake-up surfaces the latest moments.** Drawers with equal importance are now ordered by `filed_at` recency rather than insertion order, so startup context prefers recent memories instead of the oldest ones. (#1630)
- **Backend detection requires a SQLite magic header** before classifying a target as Chroma or `sqlite_exact`, preventing unrelated files from being mistaken for a palace. (#1893, #1896)
### Documentation
- Added a storage-backend configuration reference, a remote/team-server deployment guide, `authored_at` migration guidance, and refreshed the OpenClaw integration for the full 36-tool MCP surface. (#1719, #1877, #1890, #1904, #1905)
### Internal
- Expanded WAL crash-safety, idempotence, and redaction-path coverage (#1869); updated GitHub Actions and Ruff; repaired `uv.lock` drift so it again matches the Ruff 0.15.20 pin and includes Python 3.9 dependency markers introduced by the Milvus resolution.
---
## [3.5.0] — 2026-06-22
### Features
- **Opt-in local daemon for queued writes.** A new `mempalace daemon` queues MemPalace writes through a single local process so background mines, diary saves, and hook-driven ingests serialize against one palace handle instead of racing for it. Opt-in and local-only — nothing binds to a public interface. (#1826)
- **Opt-in HTTP transport for the MCP server.** `mempalace-mcp --transport http` serves JSON-RPC at `POST /mcp` (with a `GET /healthz` liveness probe) for operators running MemPalace behind a long-lived HTTP MCP client/proxy, avoiding the long-lived-stdio framing failures of #1801. stdio remains the default and is unchanged. The transport reuses the exact stdio request dispatcher (no separate write/search path), binds `127.0.0.1` by default, and is hardened against the two ways a local HTTP server leaks to the network: it pins the `Host` header to loopback on a loopback bind and rejects any non-loopback browser `Origin` (DNS-rebinding/SSRF guard), and supports an optional bearer token via `MEMPALACE_MCP_HTTP_TOKEN` (required on `/mcp`, never on `/healthz`). A 16 MiB request cap and a loud warning when bound to a non-loopback host round it out. (#1801, #1806)
- **`mempalace_checkpoint` batch-save MCP tool.** Collapses multiple `add_drawer` calls plus an optional diary entry into a single MCP round-trip for agents that want to file a whole session at once. Stores content verbatim and reuses the existing idempotent add/dedup path. (#1851)
- **`mempalace_delete_by_source` bulk-cleanup MCP tool.** Exact-match, dry-run-by-default deletion of every drawer (and its matching closet/AAAK index entries) for a given `source_file` — the recourse for benchmark/eval files mined into the same wing as real data and drowning out search. The dry run reports the drawer and closet blast radius before anything is removed, and the commit writes a WAL audit entry. (#1722, #1729)
- **Optional `source_file` filter for `mempalace_search`.** Scope a search to an exact stored source path. The filter is threaded through every search path (vector, BM25/SQLite fallback, lexical union, and index-mismatch fallback) so it never silently drops a matching drawer, and results now expose the full `source_path` as a round-trippable key. (#1815, #1817)
- **New transcript parsers / importers.** Continue.dev session parser (#731), Gemini CLI / AI Studio JSON session import (#204), and a Pi agent JSONL session normalizer (#169).
- **Wider miner language coverage.** C# / .NET, PHP (#1819), Swift / Kotlin (#1368), and Java project detection including rootless subprojects (#1720).
- **Final mine on Claude plugin `SessionEnd`.** The Claude Code plugin now runs a closing mine when a session ends so the last exchanges are captured without waiting for the next save nudge. (#1814, #1820)
### Performance
- **Overview/status MCP tools answer from the SQLite aggregate.** Large palaces no longer time out building wing/room/status overviews — the counts come from a single SQLite aggregate instead of a client-side fetch-and-tally. (#1748, #1379)
- **`graph_stats` SQLite fast path.** Knowledge-graph stats are computed in SQLite rather than walking the collection, fixing large-palace timeouts. (#1379)
- **Embedder caps ONNX-runtime intra-op threads** so a background mine no longer pins every core. (#1068)
- **Backend pagination pushed into the query.** `sqlite_exact` (#1841, #1842) and `pgvector` (#1830, #1840) now apply `get(limit, offset)` in SQL, and Qdrant fetches bulk metadata in a single scroll with a larger page size (#1796, #1832).
### Bug Fixes
- **pgvector tolerates hostile transcript bytes.** A lone Unicode surrogate (#1833) or a NUL byte (#1829) in a transcript no longer aborts the whole mine — both are sanitized before the row is written.
- **SQLite read-only URIs are percent-encoded** so palace paths with spaces or special characters open correctly, and `_sqlite_graph_stats` is routed through the same `sqlite_read_uri` helper.
- **Stale ChromaDB HNSW divergence routes to the SQLite fallback** instead of failing the read outright. (#1816, #1822)
- **Diverged-index recovery now points at `repair --mode from-sqlite`, not a re-mine.** A failed ChromaDB HNSW compaction leaves the index out of sync while the rows stay intact in `chroma.sqlite3`; the old "re-mine from source" advice silently dropped MCP-added drawers and diary entries (which have no source file). Both the legacy `repair`/`rebuild_index` error messages and the `repair-status` recommendation, plus the recall skill docs, now guide users to rebuild from SQLite. (#1843, #1847, #1849)
- **The MCP server refuses a second writer for the same palace** rather than letting two processes race the same HNSW handle. (#1818, #1823)
- **Windows hook miner spawns with `CREATE_NO_WINDOW`** so background mines no longer flash a console window. (#1783, #1848)
- **`fact_checker``__main__` no longer emits a runpy warning** under the test runner. (#1798)
### Internal
- Live-substrate conformance test module for pgvector (#1769); dependabot bumps for `docker/login-action` (3→4), `docker/build-push-action` (6→7), and `docker/metadata-action` (5→6) (#1788, #1787, #1786); ruff dev dependency bumped to 0.15.18.
---
## [3.4.1] — 2026-06-14
### Features
- **Cursor IDE plugin (`.cursor-plugin/`).** Drops into `~/.cursor/plugins/local/mempalace` (or installs from the Cursor marketplace once published) and auto-registers the `mempalace-mcp` server, five slash commands (`/mempalace-help`, `/mempalace-init`, `/mempalace-mine`, `/mempalace-search`, `/mempalace-status`), and the model-invocable [`mempalace` skill](skills/mempalace/SKILL.md) — no manual `~/.cursor/mcp.json` edit required. The plugin manifest deliberately omits a hardcoded `version` field — `mempalace/version.py` is the single source of truth, so there is nothing to drift on the next release (a contract test enforces the field stays absent). The canonical plugin components (`commands/`, `skills/`, `mcp.json`) are real files at the plugin root; no symlinks are committed (committed symlinks materialise as broken text files on Windows clones with `core.symlinks=false`). Mirrors the surface of [`.claude-plugin/`](.claude-plugin/) and [`.codex-plugin/`](.codex-plugin/) without duplicating their hook scripts: the Cursor hook scripts under [`hooks/cursor/`](hooks/cursor/) (shipped in the same release) remain the canonical install path for `stop` / `preCompact` / `sessionStart`, wired separately by [`hooks/cursor/install.sh`](hooks/cursor/install.sh). Contract tests in [`tests/test_cursor_plugin_manifest.py`](tests/test_cursor_plugin_manifest.py) cover manifest JSON validity, kebab-case naming, `..`-free relative paths, on-disk path resolution, marketplace alignment, MCP config shape (`mcpServers` wrapper required by Cursor, unlike Claude's flat `.mcp.json`), the version-field-absent guard, the no-symlink guard, and every skill/command frontmatter — all pure file inspection so they run on any CI platform without Cursor itself.
- **Cursor IDE hook support (`stop` / `preCompact` / `sessionStart`).** Three new bash hooks live under [`hooks/cursor/`](hooks/cursor/) and share a `lib/common.sh` helpers module. The save hook counts `stop` invocations per Cursor `conversation_id` and emits a `followup_message` every `MEMPAL_SAVE_INTERVAL` (default 15) so the agent files the session into MemPalace and writes a diary entry. Unlike the silent-by-default Claude Code hook, the Cursor followup fires **on by default**: Cursor's transcript format is undocumented and `normalize.py` has no Cursor parser yet, so the background `mempalace mine --mode convos` is best-effort only and the `followup_message` is the load-bearing verbatim-capture path. Users who want the Claude-style "zero tokens in the chat window" behaviour can suppress it with `MEMPAL_CURSOR_SILENT=1` (or `MEMPAL_VERBOSE=false`); the default flips to silent once a Cursor transcript parser lands. The precompact hook synchronously mines the transcript before Cursor's compaction summarises it and drops a marker so the next `stop` forces a save nudge (Cursor's `preCompact` is observational-only — it cannot block or emit a `followup_message`, unlike Claude Code's `PreCompact`); the synchronous mine is bounded by Cursor's per-hook timeout, and because `mempalace mine` is incremental/append-only a killed mine resumes cleanly on the next run rather than corrupting the palace. The wake hook is Cursor-only: `sessionStart` returns `additional_context` telling the agent to recall scoped to the wing inferred from the workspace root. Honours the same `MEMPALACE_HOOKS_AUTO_SAVE=false` kill switch as the Claude Code hooks, plus a new `MEMPAL_DISABLE_HOOK=1` alias and a `MEMPAL_STATE_DIR` env override. Per-conversation state files are garbage-collected by a daily-throttled, Cursor-namespaced TTL sweep (`MEMPAL_STATE_TTL_DAYS`, default 30) so `cursor_*.count` / `cursor_*.pending` cannot grow unbounded — shared logs and other editors' state are never touched. Includes an opt-in installer at [`hooks/cursor/install.sh`](hooks/cursor/install.sh) with `--scope user|project`, `--variant full|minimal`, `--dry-run`, and `--uninstall` (idempotent, preserves unrelated hooks via `python3`-based JSON merge — no `jq` dependency). Example wirings live at [`examples/cursor/hooks.json`](examples/cursor/hooks.json) and [`examples/cursor/hooks.minimal.json`](examples/cursor/hooks.minimal.json); they are intentionally not placed at the repo root because Cursor auto-loads project hooks from any trusted workspace and we do not arm hooks on contributor checkout. Per-event stdin/stdout schema documented at [`hooks/cursor/STDIN_SHAPE.md`](hooks/cursor/STDIN_SHAPE.md). Walkthrough at [`website/guide/cursor-hooks.md`](website/guide/cursor-hooks.md). Coverage added in [`tests/test_cursor_hooks_shell.py`](tests/test_cursor_hooks_shell.py) and [`tests/test_cursor_hooks_install.py`](tests/test_cursor_hooks_install.py).
- **First-class Antigravity IDE support.** New `.antigravity-plugin/` package + idempotent installer at `hooks/antigravity/install.sh` that registers MemPalace as a Google Antigravity plugin (MCP server, skill, two lifecycle hooks) at `~/.gemini/config/plugins/mempalace/`. The Stop hook background-mines the active conversation transcript every Nth fire (default 15, configurable via `MEMPAL_SAVE_INTERVAL`); the PreInvocation hook injects verbatim memory on the first model call only via Antigravity's `injectSteps[].ephemeralMessage` output, gated by `invocationNum == 1`. Both hooks are bash 3.2.57 compatible (macOS default), use the same `~/.mempalace/hook_state/` directory as the Claude Code / Codex / Cursor hooks (`antigravity_*`-namespaced state files), and respect every existing kill switch (`MEMPAL_DISABLE_HOOK`, `MEMPALACE_HOOKS_AUTO_SAVE`, `~/.mempalace/config.json``hooks.auto_save`). Installer is `cmp`-gated (re-run produces a byte-identical install), uninstall is basename-guarded (refuses to wipe a directory whose basename isn't `mempalace`), and `--dry-run` is side-effect free. Full audit of which Antigravity surfaces we ship and which we deliberately don't is in [`hooks/antigravity/INVESTIGATION.md`](hooks/antigravity/INVESTIGATION.md). User-facing guide: [`website/guide/antigravity.md`](website/guide/antigravity.md). Standalone examples in [`examples/antigravity/`](examples/antigravity/).
- **Zero-config interpreter resolution.** `mempal_resolve_python` now derives the Python interpreter from the `mempalace-mcp` / `mempalace` console-script shebang on `$PATH` before falling back to `python3`. The common `uv tool install mempalace` / `pipx install` layout installs the console scripts into an isolated environment whose interpreter is **not** system `python3`, so the previous `command -v python3` resolution landed on a Python that couldn't import `mempalace`, the `-m mempalace` probe failed, and mining silently never fired. Resolution is pure shebang parsing + `stat` (no Python subprocess at source time, preserving the hook performance budget). `MEMPAL_PYTHON` remains the explicit override. Documented under *How the hooks find your `mempalace` install* in the guide.
### Bug Fixes
- **`embeddinggemma` no longer OOM-kills bulk re-embeds.** `EmbeddinggemmaONNX.__call__` ran a single `session.run` over its entire input, so a repair-scale batch (5000 docs) allocated attention buffers far beyond available RAM and the kernel killed the process silently: `mempalace repair --yes` on an `embedding_model: embeddinggemma` palace died right after `Building temporary collection:` with no traceback and no crash report (#1770). Embedding now runs in sub-batches of 32 docs (constructor-tunable `batch_size`), matching the internal batching of ChromaDB's bundled MiniLM embedder. Per-document vectors are unchanged: the model's pooled output is attention-masked, so sub-batch padding does not affect values. The embedder is also hardened for shared use: the process-wide EF cache and the lazy model load are thread-safe (concurrent first calls build exactly one ONNX session), and `__call__` handles a bare string, `None`, and empty input without triggering the model download.
- **Backup retention to prevent unbounded disk usage.** `mempalace migrate` (full-palace `<palace>.pre-migrate.<timestamp>` copies) and `mempalace repair max-seq-id` (`chroma.sqlite3.max-seq-id-backup-<timestamp>` copies) each wrote a fresh, full-size, timestamped backup every run and never deleted the old ones. On a machine that mines or repairs on a schedule, those copies could silently accumulate until they filled the disk — one palace was found with hundreds of GB of stale backups beside a few hundred MB of live data, hidden from a normal `du` of the home directory. A new `max_backups` setting (default `10`, env `MEMPALACE_MAX_BACKUPS`, or `config.json`) now prunes the oldest backups after each new one is written. Set it to `0` to keep every backup. Pruning is keyed by filesystem mtime, scoped strictly to each backup's own naming pattern (live data is never touched), and best-effort so a deletion failure can never abort a migration or repair that already succeeded.
> 1. This **[GitHub repository](https://github.com/MemPalace/mempalace)**
> 2. The **[PyPI package](https://pypi.org/project/mempalace/)**
> 3. The docs at **[mempalaceofficial.com](https://mempalaceofficial.com)**
>
> **ANY other domain** (including `.tech`, `.net`, or other `.com` variants) is an **impostor** and may distribute **malware**. Do not download executables from untrusted sites. Details and timeline: [docs/HISTORY.md](docs/HISTORY.md).
> [!IMPORTANT]
> **🚨 Claude Code sessions expire in 30 days w/out auto-save hooks wired!** **[Read this →](https://github.com/MemPalace/mempalace/discussions/1388)**
@@ -31,6 +13,14 @@ Local-first AI memory. Verbatim storage, pluggable backend, 96.6% R@5 raw on Lon
</div>
> [!CAUTION]
> **Beware of impostor sites.** MemPalace has no other official websites. The **only** official sources are this **[GitHub repository](https://github.com/MemPalace/mempalace)**, the **[PyPI package](https://pypi.org/project/mempalace/)**, and the docs at **[mempalaceofficial.com](https://mempalaceofficial.com)**. Any other domain (including `.tech`, `.net`, or other `.com` variants) is an impostor and may distribute malware. Details and timeline: [docs/HISTORY.md](docs/HISTORY.md).
> [!IMPORTANT]
> **Claude Code sessions expire in 30 days without auto-save hooks wired.** [Read this →](https://github.com/MemPalace/mempalace/discussions/1388)
>
> Need the shortest recovery/setup path? Use the [Claude Code retention setup checklist](https://mempalaceofficial.com/guide/claude-code-retention.html).
@@ -194,6 +293,7 @@ verbatim drawer per user/assistant message, idempotent and resume-safe.
- Python 3.9+
- A vector-store backend (ChromaDB by default)
- ~300 MB disk for the embedding model. Onboarding (`python -m mempalace.onboarding`) offers `embeddinggemma-300m` (multilingual, 100+ languages, recommended) or `all-MiniLM-L6-v2` (English-only, ~30 MB). See the docstring at [`mempalace/embedding.py`](mempalace/embedding.py) for details and migration notes.
- Optional — compute embeddings on a server instead of locally. Set `embedding_model: "openai-compat"` in `~/.mempalace/config.json` together with `embedding_api_url` / `embedding_api_model` (and `embedding_api_key` if the server needs auth) to use any OpenAI-compatible `/v1/embeddings` endpoint — LM Studio, llama.cpp, vLLM, Ollama's OpenAI shim, or a self-hosted server (e.g. a larger multilingual or GPU-served embedder). Each key is overridable via the matching `MEMPALACE_EMBEDDING_API_*` env var. When the endpoint is on your machine or LAN, no content leaves your network. Switching to it requires `mempalace repair rebuild-index` (different vector space).
No API key is required for the core benchmark path.
@@ -215,7 +315,7 @@ PRs welcome. See [CONTRIBUTING.md](CONTRIBUTING.md).
description:Set up MemPalace — install the package, initialize a palace, register the MCP server with Cursor, and verify everything works.
---
Invoke the `mempalace` skill from this plugin and run the `init` instructions, then follow them.
Concretely: run `mempalace instructions init` in a terminal, then carry out the steps it prints.
Cursor-specific extras after init:
1. The `mempalace-mcp` server is already auto-registered by this plugin — no manual `mcp.json` edit needed.
2. For automatic background saves and session-start memory recall, also run `hooks/cursor/install.sh --scope user` from a cloned MemPalace repo. See `website/guide/cursor-hooks.md` for the walkthrough.
description:Search your memories across the MemPalace using semantic search with wing/room filtering.
---
Invoke the `mempalace` skill from this plugin and run the `search` instructions, then follow them.
Concretely: run `mempalace instructions search` in a terminal, then carry out the steps it prints. The MCP tool `mempalace_search` is also available directly from this Cursor session.
> Accepting this RFC pins the contract; remaining gaps are evolutionary, not
> blocking.
## Summary
A formal contract for MemPalace storage backends so third parties can ship `pip install mempalace-<name>` packages that drop into the core without patches. The spec defines the collection interface, the backend lifecycle, registration via Python entry points, configuration shape, a required test contract, and a migration path between backends.
It also sets up MemPalace to run as a long-lived daemon that manages many palaces, where different palaces may route to different backends.
## Motivation
Six backend PRs are currently in flight. Each one solves the same problem six different ways — different method signatures, different registration mechanisms, different embedder ownership, incompatible where-clause dialects, no shared test suite. The ad-hoc `BaseCollection` ABC merged in #413 was deliberately minimal and deferred every non-obvious decision. This RFC closes the open decisions so backend authors can build to a stable contract.
## Goals
1. A backend ships as a standalone Python package; installing it is sufficient to use it.
2. All callers in MemPalace core go through the collection interface. No direct `chromadb` imports outside `mempalace/backends/chroma.py`.
3. Backends are interchangeable: every backend passes the same shared test suite, and `mempalace migrate` supports lossless movement between them when source/target capabilities allow, with explicit re-embedding as the fallback (§8.2).
4. The model scales from single-user local (one backend, one palace, no config) to a daemon serving many palaces with heterogeneous backends.
5. Chroma's current dict-shaped return values are not the long-term contract. Typed results are spec v1.
## Non-goals
- Defining the embedder pipeline in detail. The embedder is a separate contract this spec depends on but does not specify.
- Defining the sync subsystem. This spec only declares the capability flag and the minimal hook a sync subsystem will read.
- Specifying wire protocol for a future networked daemon. That is a separate RFC.
---
## 1. Collection contract
### 1.1 Required methods
All backends implement `BaseCollection` with kwargs-only signatures:
```python
classBaseCollection(ABC):
@abstractmethod
defadd(
self,
*,
documents:list[str],
ids:list[str],
metadatas:list[dict]|None=None,
embeddings:list[list[float]]|None=None,
)->None:...
@abstractmethod
defupsert(
self,
*,
documents:list[str],
ids:list[str],
metadatas:list[dict]|None=None,
embeddings:list[list[float]]|None=None,
)->None:...
@abstractmethod
defquery(
self,
*,
query_texts:list[str]|None=None,
query_embeddings:list[list[float]]|None=None,
n_results:int=10,
where:dict|None=None,
where_document:dict|None=None,
include:list[str]|None=None,
)->QueryResult:...
@abstractmethod
defget(
self,
*,
ids:list[str]|None=None,
where:dict|None=None,
where_document:dict|None=None,
limit:int|None=None,
offset:int|None=None,
include:list[str]|None=None,
)->GetResult:...
@abstractmethod
defdelete(
self,
*,
ids:list[str]|None=None,
where:dict|None=None,
)->None:...
@abstractmethod
defcount(self)->int:...
```
### 1.2 Optional methods (default implementations on the ABC)
```python
defestimated_count(self)->int:
returnself.count()
defclose(self)->None:
returnNone
defhealth(self)->HealthStatus:
returnHealthStatus.ok()
defupdate(
self,
*,
ids:list[str],
documents:list[str]|None=None,
metadatas:list[dict]|None=None,
embeddings:list[list[float]]|None=None,
)->None:
"""Partial update of existing rows. At least one of documents/metadatas/embeddings must be non-None.
Default implementation: get(ids=...), merge the provided fields, upsert. Non-atomic
and does two round-trips. Backends advertising `supports_update` MUST override with
an atomic, single-round-trip implementation.
"""
...# default impl in the ABC
```
Backends with cheap approximate counters override `estimated_count`. Backends that hold connections must override `close`. Backends with native partial-update primitives (Postgres `UPDATE`, Lance `merge_insert`) override `update` and advertise `supports_update`; the token signals "atomic + single round-trip," not "supports partial updates at all" — the default implementation already supports them, just non-atomically.
On empty results: return a result object with empty inner lists, never raise. Specifically, an empty query returns `QueryResult(ids=[[]], documents=[[]], metadatas=[[]], distances=[[]])` — the outer dimension is the number of query vectors issued; the inner dimension is hits per query and may be zero.
`include` controls which fields are populated. Fields not in `include` are populated with empty lists of the correct outer shape; they are never `None` (except `embeddings`, which is `None` when not requested).
Backends that do not support full-text natively MUST still implement `$contains` via payload string match — correctness is required; performance is not. `supports_contains_fast` (§2.1) is the only performance floor the spec promises. Without it, callers and benchmarks MUST assume `$contains` is O(n). This is an intentional split: `$contains` is a correctness requirement, `contains_fast` is the performance boundary, and the gap between scan and indexed FTS is too large for the spec to paper over.
**Unknown operators:** backends MUST raise `UnsupportedFilterError`. Silent dropping is forbidden — it produces incorrect results.
**Optional operators:**`$gt`, `$gte`, `$lt`, `$lte`. Backends either implement them or reject with `UnsupportedFilterError`. Advertised via capabilities.
### 1.5 Embeddings
#### Signature compliance (all backends)
All backends MUST accept a pre-computed `embeddings=` argument on `add` / `upsert` without raising. This is signature compliance only — it does not guarantee the vectors are persisted (see passthrough below). Capability token: `supports_embeddings_in`.
Backends MUST NOT hardcode embedding models or dimensions. Model selection is the embedder's responsibility (§4).
#### Passthrough vs re-embed (separate guarantee)
Accepting the argument is not the same as honoring it. Two distinct semantics, distinguished by capability:
- **`supports_embeddings_passthrough`** — when `embeddings=` is provided, the backend MUST persist those vectors as-is and MUST NOT re-embed from text. This is the stronger guarantee lossless migration depends on.
- **No `supports_embeddings_passthrough`** — the backend always re-embeds from text at write time. Provided `embeddings=` is accepted (signature compliance) but discarded. Migration *to* such a backend is re-embedding, not lossless transfer.
`supports_migration_export` (source-side bulk read) MUST be paired with `supports_embeddings_passthrough` (target-side lossless write) for a migration to be labeled lossless. The `mempalace migrate` CLI refuses to run between backends where the target lacks `supports_embeddings_passthrough` unless `--accept-re-embed` is passed, which records re-embedding in the target palace's migration log.
#### Dimension check (all backends, required)
Backends MUST validate embedding dimension on first write to a new collection and on open of an existing collection, and MUST raise `DimensionMismatchError` on mismatch. Silent acceptance of mismatched dimensions produces unrecoverable corruption.
#### Model identity check (all backends, three-state)
Dimension matching is necessary but not sufficient. Swapping to a different model that happens to share a dimension (e.g., both 384-d) silently degrades retrieval without tripping `DimensionMismatchError`. Backends MUST persist `embedder.model_name` alongside the collection on first write and MUST check it on subsequent open. Three outcomes:
| State | Condition | Required behavior |
|---|---|---|
| `known_match` | Stored name equals current `embedder.model_name` | Proceed normally. |
| `known_mismatch` | Stored name exists and differs from current | Raise `EmbedderIdentityMismatchError`. Override only via explicit CLI `--force-model-swap`, which writes the swap to the palace's migration log and updates the stored identity. |
| `unknown` | No model name recorded (legacy collection, pre-v1 palace) | Do not hard-fail — emit a `EmbedderIdentityUnknownWarning` on first open. The resolved identity is recorded on the next successful write, reindex, or migration, transitioning the palace to `known_match` going forward. CLI exposes `mempalace palace set-embedder --model NAME` for explicit resolution. |
The `unknown` state exists because existing palaces from #413 and earlier have no recorded identity; hard-failing them on upgrade would be hostile. Once recorded, subsequent opens are strict.
An injected embedder that exposes no usable `model_name` (empty or `None`) resolves to `unknown` rather than being a hard error — the backend persists no identity, emits `EmbedderIdentityUnknownWarning`, and records identity on the first open against an embedder that *does* report a name. A nameless embedder is therefore a degraded-but-valid mode, not a rejection; operators promote it with `mempalace palace set-embedder --model NAME`.
> **Follow-up dependency.** The `MUST persist embedder.model_name` rule above is only satisfiable once the `Embedder` protocol (§5) is a normative contract — a backend cannot persist an identity it is never handed. §5 now pins the minimal protocol (`model_name`, `dimension`, `embed`) as sufficient for this section; the embedder-identity enforcement is tracked as follow-up implementation work (the current in-tree backends defer it). See §5.
#### `server_embedder` backends are not exempt
A backend advertising `server_embedder` (§2.1) provides its own embedder and MAY ignore the `embedder=` kwarg passed to `get_collection`. That does **not** exempt it from the dimension and identity rules above. Such backends MUST:
- Expose an effective `model_name: str` and `dimension: int` describing the embedder actually in use (via `BaseCollection.effective_embedder_identity() -> EmbedderIdentity`).
- Persist that effective identity on first write and validate it on open, per the three-state rules above.
- Raise `DimensionMismatchError` and `EmbedderIdentityMismatchError` on conflicts between the effective identity and any injected `embedder` (if one was passed) or between the stored identity and the current effective identity.
`server_embedder` documents where the embedding happens; it never suspends the safety contract. A backend that cannot report its effective embedder identity does not qualify for the `server_embedder` capability.
| `supports_contains_fast` | `$contains` is indexed (vs scan-based) |
| `supports_server_side_indexes` | Exposes index creation / maintenance to operators |
| `supports_migration_export` | Implements a bulk read path suitable for `mempalace migrate` |
| `supports_change_feed` | Exposes `changes_since(cursor)` for the sync subsystem |
| `supports_sync` | Implies `supports_change_feed` plus idempotent upserts under conflicts |
| `requires_external_service` | Needs a running server (e.g., Postgres, hosted Qdrant) |
| `local_mode` | Persists to `palace.local_path` |
| `server_mode` | Connects to an external server; `palace.namespace` is used |
| `server_embedder` | Backend provides its own embedder (may ignore injected one) |
| `supports_namespace_isolation` | Enforces `PalaceRef.namespace` as a hard isolation boundary (§4.4). Multi-tenant deployments MAY rely on it for tenant isolation; backends without it MUST NOT be relied on for that. |
A backend may advertise both `local_mode` and `server_mode` (e.g., Chroma with either `PersistentClient` or `HttpClient`).
Capability tokens are free-form strings, not an enum — third-party backends may declare novel capabilities for their ecosystem. Core MemPalace only inspects the tokens listed above.
### 2.2 Palace references
A backend serves palaces, not raw filesystem paths. This is the central change from #413.
```python
@dataclass(frozen=True)
classPalaceRef:
id:str# stable identity, used as cache key
local_path:str|None=None# filesystem root, if this palace is local
namespace:str|None=None# server-side namespace/prefix, if applicable
```
Rules:
-`id` is always present. It is the key the backend uses to cache open handles.
- Local-only backends read `local_path`. If `local_path is None` they raise `PalaceNotFoundError`.
- Server-only backends read `namespace`. If `namespace is None` they derive one deterministically from `id`.
- Mixed-mode backends may use both (e.g., a local cache alongside a server store).
### 2.3 Methods
```python
classBaseBackend(ABC):
@abstractmethod
defget_collection(
self,
*,
palace:PalaceRef,
collection_name:str,
create:bool,
embedder:Embedder|None=None,
options:dict|None=None,
)->BaseCollection:...
defclose_palace(self,palace:PalaceRef)->None:
"""Evict a single palace's cached handles. Default: no-op."""
returnNone
defclose(self)->None:
"""Shut down the entire backend instance. Default: no-op."""
"""Return health. With palace=None, probe the backend itself."""
returnHealthStatus.ok()
```
### 2.4 Semantics of `create`
-`create=False` on a nonexistent palace MUST raise `PalaceNotFoundError` (subclass of `FileNotFoundError` for backwards compatibility with the #413 seam).
-`create=True` MUST be idempotent — calling it repeatedly with the same arguments produces the same state and does not corrupt existing data.
-`create=True` on local backends creates the directory with `0700` permissions (matches the existing Chroma behavior).
**Multiple collections per palace.**`get_collection` is keyed by `collection_name`, and a palace MAY hold more than one collection. Backends MUST support N collections per palace, addressed by distinct `collection_name` values, with the §2.5 isolation guarantee applying per `(palace.id, collection_name)`. The "palace" is not 1:1 with a collection: production already splits verbatim drawers from short, query-term-saturated session-recovery checkpoints into sibling collections (`mempalace_drawers` vs `mempalace_session_recovery`) so the latter can't dominate vector top-N. Backends like Postgres (schema/table naming) and Qdrant (collection naming) handle this trivially; a backend author MUST NOT assume one collection per palace and design themselves into a corner. No signature changes — this is already implicit in `collection_name`; it is stated here so it is a contract fact, not a convention.
### 2.5 Concurrency
A backend instance is long-lived and serves many palaces. Backends MUST be thread-safe for concurrent `get_collection` calls across different `PalaceRef.id` values. Collection handles for the same `(palace.id, collection_name)` MAY be cached internally and returned on subsequent calls.
Backends MAY assume a single thread accesses a given `BaseCollection` instance at a time. MemPalace core serializes access per palace; backend authors are not required to make individual collections thread-safe.
### 2.6 Lifecycle
1.`__init__`: lightweight. No I/O, no network connections. A backend instance may be constructed and never used.
2. First call to `get_collection`: may open connections, create schemas, etc. All I/O is lazy.
3.`close_palace(palace)`: releases cached handles for one palace. Safe to call on a palace that was never opened.
4.`close()`: releases all resources. After `close()`, further calls MUST raise `BackendClosedError`.
There is no explicit `connect()` — it is always implicit and lazy, matching current Chroma behavior.
---
## 3. Registration and discovery
### 3.1 Entry points (primary mechanism)
Third-party backends ship as installable packages:
```toml
# pyproject.toml of mempalace-postgres
[project.entry-points."mempalace.backends"]
postgres="mempalace_postgres:PostgresBackend"
```
MemPalace discovers backends at process start via `importlib.metadata.entry_points(group="mempalace.backends")`. No patches to the core are required.
### 3.2 In-tree registry (secondary)
For tests and local development:
```python
frommempalace.backends.registryimportregister
register("my-experimental-backend",MyBackend)
```
Entry-point discovery and explicit `register()` populate the same registry. Explicit registration wins on name conflict.
### 3.3 Selection priority
When resolving a palace's backend, priority (highest first):
1. Explicit `backend=` kwarg to `Palace(...)` or CLI `--backend`
2. Per-palace `backend` key in config (see §4)
3.`MEMPALACE_BACKEND` environment variable
4. Auto-detect from on-disk artifacts: `chroma.sqlite3` → `chroma`, `*.lance` → `lance`, etc. Backends declare detection hints via an optional `BaseBackend.detect(path: str) -> bool` classmethod.
5. Default: `chroma`.
**Auto-detection is strictly a migration/upgrade compatibility path, not a general selection mechanism.** It exists so existing palaces from v3.x keep opening without forced config migration. For *new* palaces, explicit configuration or CLI flag always wins — creating a palace without a resolved backend from (1)–(3) falls through to default (5), never to detection (4). Auto-detection fires only when a local path is presented AND no earlier rule has chosen a backend AND the path already contains backend-identifiable artifacts.
Note the interaction with a globally-set `MEMPALACE_BACKEND`: rule (3) sits above detection (4), so a user who exports `MEMPALACE_BACKEND=postgres` and then opens a palace containing on-disk Chroma artifacts gets postgres — the env var wins and detection is skipped. That is intended (explicit configuration overrides detection), but it means **setting `MEMPALACE_BACKEND` globally overrides existing-palace auto-detection; users opening pre-existing palaces of mixed backends should leave it unset** and rely on per-palace config or detection.
---
## 4. Configuration
### 4.1 Shape
```json
{
"backends":{
"chroma":{"type":"chroma"},
"pg_prod":{
"type":"postgres",
"dsn":"postgresql://...",
"pool_size":10
}
},
"palaces":{
"work":{
"backend":"pg_prod",
"namespace":"work"
},
"personal":{
"backend":"chroma",
"local_path":"~/.mempalace/personal"
}
},
"embedder":{
"type":"onnx",
"model":"all-MiniLM-L6-v2"
}
}
```
Single-user local mode: all of this is optional. The absence of a config file yields one Chroma backend, one palace at the default path, with the default embedder.
### 4.2 Environment variables
-`MEMPALACE_BACKEND` — shortcut for the default backend type when there is no config.
-`MEMPALACE_<NAME>_*` — per-backend secrets and connection info (e.g., `MEMPALACE_POSTGRES_DSN`, `MEMPALACE_QDRANT_URL`, `MEMPALACE_QDRANT_API_KEY`).
-`<NAME>` is the backend's **type** name (the `type` field in §4.1 — `postgres`, `qdrant`), uppercased, not the per-instance config key. So the `pg_prod` instance in §4.1 reads `MEMPALACE_POSTGRES_*`, not `MEMPALACE_PG_PROD_*` — instances of the same type share one env namespace, and connection-specific values that differ per instance (distinct DSNs) belong in the config file's per-backend block, not in env. Hyphens in a type name are normalized to underscores for the env prefix (`my-backend` → `MEMPALACE_MY_BACKEND_*`).
- Secrets MUST be readable from env vars; config files are for structure, env vars for credentials.
### 4.3 Backend-specific options
The `options` kwarg to `get_collection` is a free-form dict. Each backend documents its accepted keys. Unknown keys MUST be ignored (forward compatibility), but the backend MAY log a warning.
### 4.4 Multi-tenancy (absorbs #697)
Per-tenant collection-name prefixing is not a backend concern. It is handled by the resolver layer above backends: `PalaceRef.namespace` carries the tenant identifier. The `collection_prefix` concept from #697 dissolves into this model.
**Isolation contract.**`PalaceRef.id` is the *required* isolation key for every backend: within a single backend instance, a record written for one `id` MUST NOT be returned, modified, or deleted by an operation issued for a different `id`. Cross-palace access is a spec violation. This is the non-negotiable blanket MUST; multi-tenant deployments may cite it as their primary partition boundary even when they do not use `namespace`.
`namespace` is *additional* partitioning. A backend that advertises `supports_namespace_isolation` (§2.1) MUST extend the same guarantee to namespaces:
> A record written under one `namespace` MUST NOT be returned, modified, or deleted by an operation issued under a different `namespace` within the same backend instance. Cross-namespace access is a spec violation, not a caller misconfiguration.
This is what hosted multi-tenant deployments cite as the basis for *namespace*-level tenant isolation. Authorization (which namespaces a given request may touch) stays on the deployment side; the backend's job is to guarantee no bleed *within* the instance once the namespace is fixed.
Making namespace isolation a declared capability rather than a blanket `MUST` is deliberate: path-rooted local backends (e.g. `chroma`, `sqlite_exact`) already isolate by `local_path` / `id`, and forcing them to re-implement a second axis would be ceremony with no security gain.
**Conformance arms.** Self-attestation is not enough. The isolation suite (`tests/_backend_conformance.py` / `assert_partition_isolation`) has two distinct arms:
1.**Cross-`id` isolation** — required of every backend. Two `PalaceRef`s that differ only in `id` MUST NOT see each other's records.
2.**Same-`id` / different-`namespace` isolation** — required of every backend that advertises `supports_namespace_isolation`. Two refs that share `id` (and, when applicable, `local_path`) but differ in `namespace` MUST NOT see each other's records. Distinct from arm 1; a backend that only passes arm 1 does not get to claim the capability.
**No silent drop.** A backend that does **not** advertise `supports_namespace_isolation` MUST NOT silently accept and ignore a populated `namespace` — that is a latent cross-namespace leak, same spirit as `UnsupportedFilterError` for unknown operators. Such backends MUST either raise (e.g. `UnsupportedCapabilityError`) when `PalaceRef.namespace` is non-`None`, or honor the namespace and advertise the capability. Callers targeting path-rooted backends MUST leave `namespace` as `None`.
---
## 5. Embedder contract (minimal, normative here)
§1.5 makes persisting and checking `embedder.model_name` a hard `MUST`. A backend cannot satisfy that against an embedder that has no identity to read, so the minimal protocol below is **normative for this spec** — not deferred. A fuller Embedder RFC (batching, async, pooling, multi-vector) is tracked separately, but it is *additive*: §1.5 conformance depends only on the three members here.
```python
classEmbedder(Protocol):
model_name:str# stable identity persisted and checked per §1.5
Backends receive an `Embedder` via `get_collection(embedder=...)`. Backends with the `server_embedder` capability MAY ignore the injected embedder but MUST still expose an effective `model_name` / `dimension` (§1.5). An embedder whose `model_name` is empty or `None` is handled as the §1.5 `unknown` state, not a hard error.
> **Follow-up.** The full Embedder RFC is the only external contract §1.5 leans on; it is tracked as a hard, blocking dependency of the §1.5 *implementation* (not of this spec's acceptance — the minimal protocol above closes the contract gap). Tracking issue: see §13. The current in-tree backends defer embedder-identity enforcement until that work lands.
---
## 6. Sync (capability declaration only)
The sync subsystem is out of scope for this spec. What this spec defines:
-`supports_sync` capability flag (§2.1) — a backend advertising it agrees to implement idempotent upserts under conflict and to expose change data.
- `close()` releases handles and further calls raise `BackendClosedError`
- Concurrent `get_collection` across different palaces is safe
- Isolation arm 1: cross-`PalaceRef.id` (every backend)
- Isolation arm 2: same-`id` / different-`namespace` (only when `supports_namespace_isolation` is advertised; self-attestation without this arm is a conformance failure)
- Non-advertising backends raise on a populated `namespace` rather than silently accepting it (§4.4)
### 7.2 Parametrized core suite
The existing MemPalace test suite is parametrized over all registered backends when `MEMPALACE_TEST_ALL_BACKENDS=1` is set in the environment. This is the "strongest parity claim" — if a backend passes the full core suite, it is drop-in compatible. This is expensive; local development defaults to Chroma only, CI runs all backends on a scheduled job.
### 7.3 Benchmark methodology hooks
Backend-to-backend comparisons are meaningless without accounting for per-backend maintenance state. Postgres with stale planner stats behaves very differently from Postgres post-`VACUUM ANALYZE`; HNSW-based stores behave differently before and after index compaction.
Backends MAY implement `maintenance_state()` returning a structured dict describing the current state (e.g., `{"autovacuum_age_seconds": 42, "last_analyze": "...", "index_build_complete": true}`), and `run_maintenance(kind: str)` to trigger supported kinds. Both are optional.
Supported maintenance kinds MUST be advertised via a class-level frozenset:
The spec reserves the kind names `"analyze"` (update planner/query statistics), `"compact"` (reclaim space, rewrite storage), and `"reindex"` (rebuild secondary indexes). Backends MAY add their own kinds; the reserved names MUST mean what the spec says if advertised. A backend that has no analogue for a reserved kind MUST omit it from `maintenance_kinds` rather than declaring it as a no-op — otherwise a benchmark harness sees `"analyze"` advertised and assumes it did what the spec says when the implementation did nothing.
`run_maintenance(kind)` MUST raise `UnsupportedMaintenanceKindError` when called with a kind not in `maintenance_kinds`. Advertising a kind without implementing it is a conformance failure.
**`run_maintenance` is observable, not fire-and-forget.** This resolves the §12 open question, and it is driven by a production failure: on a lazy-index path, multiple daemon writers crossing the index-build threshold in the same window each issued the build, stacked an `ACCESS EXCLUSIVE` lock, and blocked writes for the whole build (fixed in-backend with a session-level advisory lock). A pure fire-and-forget call reproduces that race — if the call cannot report "already running," concurrent callers re-trigger the build. Therefore:
- `run_maintenance(kind)` MUST be safe to call concurrently. A backend MUST serialize same-kind maintenance internally (advisory lock, build flag, or equivalent) so a second caller does not start a duplicate operation.
- It returns a structured `MaintenanceResult` (e.g. `{"kind": str, "status": "ran" | "already_running" | "noop", "stats": {...}}`) rather than `None`. `stats` is free-form per kind (rows analyzed, bytes reclaimed, fragments merged). `already_running` is how a concurrent caller learns it must not re-trigger.
This makes the maintenance hook the operator-safe path the lazy-index concurrency wedge requires, and gives the benchmark harness (above) machine-readable phase data.
The benchmark harness under [benchmarks/](../../benchmarks/) records `maintenance_state()` alongside every latency/recall measurement it publishes. Published numbers MUST include three phases: immediately after bulk load, after the backend's native background maintenance has caught up, and after `run_maintenance(kind)` has been called for each kind in `maintenance_kinds`. Harnesses rely on this advertisement to decide what to call — they MUST NOT assume kind names. This prevents comparing an un-`ANALYZE`d Postgres to a settled Chroma and calling the former slow.
### 7.4 ID stability for non-string-ID backends
Backends requiring UUID IDs (Qdrant) use a canonical namespace:
The namespace UUID is fixed at spec v1 adoption and recorded here — once, for all time. This value is the one the in-tree `qdrant` backend already shipped with in #1679, so existing Qdrant palaces' point IDs already derive from it; promoting the shipped constant to canonical (rather than minting a fresh one) avoids re-deriving any deployed palace's IDs. New UUID-ID backends MUST use this exact namespace. This resolves the #700 vs #381 divergence.
Implementation is backend-agnostic: reads from source via `BaseCollection.get(include=["documents", "metadatas", "embeddings"])`, writes to target via `BaseCollection.upsert(...)` with the original embeddings. No backend-specific migration code.
### 8.2 Lossless vs re-embed
Migration is labeled **lossless** only when:
- The source advertises `supports_migration_export` (bulk read includes embeddings), AND
- The target advertises `supports_embeddings_passthrough` (persists provided embeddings as-is), AND
- Source and target agree on `embedder.model_name` (or `--force-model-swap` is explicit).
If the target lacks `supports_embeddings_passthrough`, `mempalace migrate` refuses to run. Passing `--accept-re-embed` overrides — the migration proceeds but re-embeds from document text at write time, and the migration record labels the result as re-embedded rather than lossless. Retrieval quality may shift.
A backend that persists the exact float32 vector in its own durable store (rather than re-embedding from text on read) satisfies **both** sides of the lossless pairing: it qualifies for `supports_migration_export` as a source (it can hand back the verbatim vector it stored) and for `supports_embeddings_passthrough` as a target (it persists provided vectors as-is). Such "rank-from-stored-vectors" backends are first-class migration endpoints in both directions — the pairing in this section is about the two *capabilities* being present, not about any particular index structure. An exact-vector store therefore makes `chroma → <store>` and `<store> → chroma` lossless under §8 in both directions, given model-identity agreement.
### 8.3 Safety
- Source is never modified. Migration is read-only against the source backend.
- Target palace must not already exist unless `--overwrite` is passed.
- A successful migration writes a `.mempalace-migration.json` record into the target palace containing: source backend name, source path/ref, timestamp, row count, `lossless: true|false`, source and target `embedder.model_name`, and whether `--force-model-swap` or `--accept-re-embed` was used.
### 8.4 Verification
After migration, run `mempalace verify --palace PATH --against SOURCE_PATH --source-backend chroma`. This samples N rows and confirms round-trip parity (ids match, documents match, embedding cosine similarity ≥ 0.999 when the migration was lossless; a looser document-overlap check when re-embedded).
---
## 9. Versioning and compatibility
- `BaseBackend.spec_version` declares which spec version a backend implements.
- MemPalace refuses to load a backend declaring a different major version. The failure is loud and names the mismatch, e.g.:
f"this MemPalace implements major version {CORE_MAJOR!r}. "
f"Install a build of {name!r} that targets spec {CORE_MAJOR}.x."
)
```
- Minor versions are additive (new optional methods, new capability tokens). Backends declaring an older minor continue to work.
- This is spec v1.0.
---
## 10. Cleanup prerequisite (mostly landed via #1679)
**Status update (2026-06-07).** The bulk of this cleanup landed in [#1679](https://github.com/MemPalace/mempalace/pull/1679): the direct `chromadb` *client* imports across `repair.py`, `dedup.py`, `cli.py`, `mcp_server.py`, and `migrate.py` were routed through `BaseCollection`, and the dict-to-typed-result migration (§1.3) shipped. What remains is narrow:
- **Residual exception-class imports.** `mcp_server.py` and `repair.py` still `from chromadb.errors import NotFoundError` to catch Chroma's not-found error. These are catch-site couplings, not client construction; they should resolve to a backend-neutral `PalaceNotFoundError` / collection-not-found exception from the contract (§2.4) so non-Chroma palaces raise the same type.
- **`mcp_server._get_client()` caching.** Caches a `PersistentClient` at module scope and invalidates it on `chroma.sqlite3` inode or mtime changes (merged via [#757](https://github.com/MemPalace/mempalace/pull/757)). Both the cache and the stat-based freshness check are Chroma-specific. They should migrate into `ChromaBackend.get_collection()` (§2.5, handle caching) and `ChromaBackend.close_palace()` (§2.6, explicit flush) — other backends do not have a single on-disk SQLite file to stat. The `mempalace_reconnect` MCP tool then becomes a thin wrapper around `backend.close_palace(palace_ref)`.
**`searcher.py` is the highest-leakage module still coupled, and #1679 did not cover it** (flagged by two backend authors — kostadis-ntnx on this RFC, jphein on #1679). Two concrete couplings:
- `_hybrid_rank()` hard-codes `vec_sim = max(0, 1 - distance)`, i.e. it assumes cosine distance ∈ [0, 2]. For a backend whose metric isn't cosine (dot-product, L2) the ranking is silently wrong. Fix: make the conversion metric-aware off the backend-declared `distance_metric` (§2.1) rather than hard-coding cosine. All in-tree backends are cosine today, so this is latent — but it is a contract gap, not a coincidence to keep relying on.
- The BM25 survival fallback opens `chroma.sqlite3` directly and queries Chroma's FTS5 `embedding_fulltext_search` shadow table. On any non-Chroma palace this path is dead. Fix: route lexical fallback through `BaseCollection` (§1.4 `$contains` / a lexical capability), or gate it on a backend-declared capability so backends whose index self-heals never request it.
This `searcher.py` work is tracked as follow-up implementation (see §13) — it is the remaining piece that promotes a non-Chroma backend from "stores and retrieves" to a first-class search peer.
---
## 11. Impact on in-flight PRs
**Update (2026-06-07).** The first wave of in-tree backends — `pgvector`, `qdrant`, `sqlite_exact` — landed via [#1679](https://github.com/MemPalace/mempalace/pull/1679) implementing the §1–2 contract surface directly, ahead of this RFC's merge. That reshapes the table below: the community PRs now rebase against both the spec *and* the shipped in-tree backends, and the prediction that `collection_prefix` would dissolve into `PalaceRef.namespace` (#697) held.
| PR | Status | Effort to align |
|---|---|---|
| [#574](https://github.com/MemPalace/mempalace/pull/574) LanceDB | Open | Closest to final shape. Needs `PalaceRef` and typed results (now both shipped in-tree — rebase against the merged ABC). |
| [#665](https://github.com/MemPalace/mempalace/pull/665) Postgres (`pg_sorted_heap`) | Open | The in-tree `pgvector` backend (#1679) now occupies the basic Postgres slot; #665 rebases as the optional `pg_sorted_heap` performance variant. Decouple embedder; adopt the merged `PalaceRef`. Its session-level advisory-lock fix for the lazy-index concurrency wedge is the production basis for §7.3's observable `run_maintenance`. |
| [#700](https://github.com/MemPalace/mempalace/pull/700) Qdrant | Open | Largely superseded by the in-tree `qdrant` backend (#1679), which already uses the §7.4 canonical namespace. Reconcile or close in favor of the merged backend. |
| [#381](https://github.com/MemPalace/mempalace/pull/381) Qdrant (older) | Open | Same as #700; subclass the merged `BaseCollection` rather than a bare `Protocol`, or close as superseded. |
| [#643](https://github.com/MemPalace/mempalace/pull/643) PalaceStore | Closed | POC; the parametrized-test approach it explored became the standard (§7.2). |
| [#697](https://github.com/MemPalace/mempalace/pull/697) Chroma HttpClient + prefix | Closed | `collection_prefix` dissolved into `PalaceRef.namespace` + `supports_namespace_isolation` (§4.4), as predicted. |
---
## 12. Resolved decisions
The three questions that were open at draft are resolved for v1, informed by the backend-author reviews:
- **`changes_since` accepts a collection filter — yes.** It takes an optional `collection_name` so a sync caller can request changes for one collection (LanceDB and others already track changes per table; filtering by collection is natural). Additive on the §6 optional hook.
- **Per-palace capabilities — no; capabilities stay static/class-level.** Making `supports_*` palace-dependent turns every capability check into "maybe — depends," pushing state tracking onto every caller. A backend that builds indexes lazily either guarantees the index exists when the capability is needed (and advertises it) or doesn't advertise it. The one isolation-by-deployment case (`supports_contains_fast` where an FTS index may or may not exist) is handled by the backend either guaranteeing the floor or omitting the token — not by per-palace variance. (Note: `supports_namespace_isolation` is still a static, class-level token — it declares what the backend *enforces*, not a per-palace fact.)
- **`run_maintenance(kind)` returns a structured, observable result — resolved in §7.3.** It is not fire-and-forget: it returns `MaintenanceResult` and must serialize concurrent same-kind calls (the production lazy-index concurrency wedge is why). See §7.3.
No remaining blockers. Genuinely additive future items (new optional methods, new tokens) land under the minor-version rule (§9).
---
## 13. Rollout
The original draft sequenced cleanup → spec → ChromaBackend → in-flight rebase → migrate CLI. In practice the contract surface and the first backends shipped together in [#1679](https://github.com/MemPalace/mempalace/pull/1679) ahead of this merge, so the remaining sequence is what's left, not the whole list.
4. ✅ Canonical §7.4 namespace pinned to the shipped value.
**Remaining (tracked as follow-up issues):**
5. **Embedder-identity contract (§1.5 / §5)** — [#1724](https://github.com/MemPalace/mempalace/issues/1724). Land the minimal `Embedder` protocol as a real injected dependency; persist + check `model_name` per the three-state model. *Hard dependency for §1.5 conformance.*
6. **Maintenance hooks (§7.3)** — [#1725](https://github.com/MemPalace/mempalace/issues/1725). Implement `maintenance_kinds` / `maintenance_state()` / observable `run_maintenance()` in the in-tree backends; wire the §7.3 advisory-lock serialization (`pg_sorted_heap` / #665 has the reference fix).
7. **`searcher.py` metric-awareness + lexical fallback (§10)** — [#1726](https://github.com/MemPalace/mempalace/issues/1726). Make `_hybrid_rank` read `distance_metric`; route the BM25 fallback through `BaseCollection` instead of `chroma.sqlite3`.
Status: Implemented (phases 1-5). Phase 5 SSE shipped as GET /logstream/stream on the hub HTTP transport (bearer-authenticated, event_list filter set, since_event_id/Last-Event-ID resume, 15s heartbeats, bounded clients); logstream tools additionally dispatch outside the global HTTP request lock so long-polls cannot starve the hub. Server-side cursor storage and drawer compaction remain future work.
Owner: Claude Fable 5
Created: 2026-07-01
Branch: `feat/shared-brain-dogfood`
## Summary
MemPalace should become a local-first coordination substrate for agents, not only a long-term recall system. Agents running on different machines should be able to delegate work, wait for replies, exchange patches or other artifacts, and subscribe to realtime coordination messages through the shared MemPalace hub.
This RFC proposes **MemPalace Logstream**: a small append-only event layer served by the existing MemPalace MCP HTTP hub. It preserves the core MemPalace promise of durable, local, exact storage while adding coordination primitives agents can use without Igor manually relaying messages between them.
## Motivation
The shared-brain dogfood exposed a real workflow gap:
1. Mac Codex wrote a delegation packet into `shared_agent_brain/delegation`.
2. Windows Codex found it, did the work, and wrote back status.
3. Windows Codex forgot to push the code.
4. Mac Codex could read the status but could not receive the actual patch through a formal channel.
The memory layer worked. The coordination layer was implicit and fragile.
We need a first-class way for agents to:
- create durable work packets;
- subscribe or wait for matching replies;
- submit patch/file artifacts before they are committed;
- acknowledge, apply, reject, or supersede handoffs;
- keep enough structured metadata to avoid ambiguity across machines, branches, and worktrees.
## Non-Goals
- Do not build a cloud service.
- Do not require external APIs, hosted queues, Redis, Postgres, or SaaS infra.
- Do not replace Git for final source control.
- Do not make agents apply patches silently without explicit caller intent.
- Do not weaken verbatim drawer storage. Event payloads and artifacts must remain exact.
- Do not add telemetry or phone-home behavior.
## Design Principles
- **Local-first**: the event bus runs inside the existing MemPalace hub and is reachable over loopback, LAN, or tailnet according to the user's deployment.
- **Append-only by default**: events are immutable. Corrections are new events that reference prior events.
- **Exact payloads**: event bodies and artifacts are stored verbatim.
- **Structured envelopes**: agents should not infer core routing fields from prose.
- **Durable before realtime**: every realtime message must also be recoverable after reconnect.
- **Small useful v1**: favor long-poll and polling over a large streaming subsystem if that ships faster.
- **No hidden authority**: patch application and destructive actions must be explicit local operations.
## Core Concepts
### Event
An event is a structured coordination message. It has routing metadata plus an optional verbatim body.
Required fields:
-`id`: stable event id, generated by the server.
-`type`: event type, such as `task.request`, `task.reply`, `patch.ready`, `patch.applied`.
-`stream`: logical stream name. Suggested format: `project/<project-name>` or `shared_agent_brain`.
-`room`: sub-channel inside a stream, such as `delegation`, `patches`, `reviews`, `status`.
-`from_agent`: writer identity, supplied by caller.
-`created_at`: server timestamp.
Common optional fields:
-`to_agent`: target agent or `*`.
-`correlation_id`: task or conversation id tying request/reply events together.
- Should logstream events also be mirrored into Chroma drawers for semantic recall, or should structured search remain the primary access path?
- Should artifact content support binary data in v1, or should v1 be UTF-8 text only?
- Should agent identities be free-form strings initially, or tied to configured local agent profiles?
- Should `event_wait` be allowed over stdio MCP clients that may have shorter host timeouts?
- Should events have retention/archival controls, or stay forever by default like memory?
## Claude Fable 5 Instructions
Please implement Phase 1 and Phase 2 first. Keep the patch narrow and avoid unrelated refactors.
Start by creating `mempalace/logstream.py` with the durable SQLite core and focused tests. Then expose MCP tools in `mempalace/mcp_server.py` with read-only and mutating-tool semantics. Only add CLI/docs after the core and MCP tools are green.
Before finishing:
- run focused logstream tests;
- run relevant MCP server tests;
- run `ruff check` on touched Python files;
- write a MemPalace delegation reply with the implementation status, changed files, and exact verification commands.
## Implementation Notes
Windows dogfood verification by `windows-codex` on 2026-07-02:
- Branch/base: `feat/shared-brain-dogfood` at `1ff3125`.
- OS: Microsoft Windows 11 Pro Insider Preview 10.0.29576 build 29576 64-bit.
- Python: 3.12.11 via `uv run python`.
- Command: `uv run pytest tests/test_logstream.py tests/test_mcp_logstream.py tests/test_cli_logstream.py -q`.
Shipped so far: step 0 (logstream multi-master replication) and step 1 (memory read replicas — snapshot pull, local fold, distributed embedding), plus the transport seam and the estate endpoint. Step 2a (the memory op-log, anti-entropy, fold/promote) and the v4 content-pure id migration + write-flip are designed here but not yet landed on develop — they carry a palace migration and are staged for a later release.
Owners: mac-claude (storage layers, §6–§9), windows-claude (transport & lifecycle, §5 and Appendix A), decided by Igor
Created: 2026-07-02
Branch: `feat/shared-brain-dogfood`
Prior art: RFC 003 (logstream), the `rfc004_replicated_palace_position` correlation thread (position debate, verbatim in the logstream)
## Summary
Each human has ONE palace — an extension of their brain — replicated in full
across every machine they own. Agents always talk to the MemPalace service on
`127.0.0.1`; services converge with each other over an encrypted mesh. The
hub stops being a dependency and becomes a role (rendezvous, and the home of
*shared* palaces). The design is judged against offline operation as the
default posture, not as an edge case.
One sentence: **N equal replicas of the facts, each with locally-derived
indexes and a local writer, converging through provenance-stamped ops over
the mesh — with origin as the home-of-record for source-bound maintenance.**
## Motivation
The shared-brain dogfood proved the hub topology works — and watched it fail:
when the hub machine slept, every other machine lost recall, capture, and
coordination simultaneously. A brain does not stop remembering because
another brain is asleep. Concretely, today a remote machine without the hub
has **no palace at all**.
The mission statement ("memory is identity") implies the requirement
directly: you do not rent your identity, and you do not park it on a single
machine.
## Requirements
The availability invariant, and the offline requirements distilled from the
That will write `./.cursor/hooks.json` for the repo workspace
specifically — but it is your decision, not ours, and the file is
listed in `.gitignore` paths Cursor users typically already exclude.
## Related: the Cursor plugin
The hooks here are **only one half** of MemPalace's Cursor integration. The other half is the [`.cursor-plugin/`](../../.cursor-plugin/) folder at the repo root, which packages MemPalace's MCP server, five slash commands, and the model-invocable `mempalace` skill as a regular Cursor plugin you can drop into `~/.cursor/plugins/local/mempalace`.
The two install paths are orthogonal — install whichever you want, in any order:
| MCP tools (`mempalace_search`, `mempalace_add_drawer`, …) + slash commands | The plugin — see [`.cursor-plugin/README.md`](../../.cursor-plugin/README.md) |
| Auto-save every N turns + sessionStart memory recall | The hooks here — see Option A above |
| Both | Install the plugin AND run `hooks/cursor/install.sh` |
Hooks are deliberately **not** bundled into the plugin because Cursor's hooks system is configured per-user/per-project (in `~/.cursor/hooks.json` or `.cursor/hooks.json`), not per-plugin — so the installer here owns that file with idempotent merge semantics, while the plugin owns the MCP+commands+skill side.
## See also
- [`hooks/cursor/README.md`](../../hooks/cursor/README.md) — full reference for hooks
- [`hooks/cursor/STDIN_SHAPE.md`](../../hooks/cursor/STDIN_SHAPE.md) — per-event schema with citations
Optional [Cursor rules](https://cursor.com/docs/rules) that make the
agent search MemPalace before answering questions about past work,
people, projects, or prior decisions.
These are for users who install MemPalace **without** the Cursor plugin
(or who want recall behaviour in a specific project). If you installed
the [Cursor plugin](../../../.cursor-plugin/README.md), it already ships
the `alwaysApply: false` rule at the plugin root — you do not need to
copy anything.
## Which file to use
| File | `alwaysApply` | Fires when | Use when |
|------|---------------|------------|----------|
| [`mempalace-recall.mdc`](mempalace-recall.mdc) | `false` | Cursor's matcher decides the turn is recall-relevant (from the rule `description`) | **Recommended.** Recall without paying for the rule on unrelated work. |
| [`mempalace-recall-always.mdc`](mempalace-recall-always.mdc) | `true` | Every conversation in scope, every turn | You want recall guaranteed in context and accept the cost. |
The always-on variant is heavier: it sits in context on every turn and
makes the agent more eager to call `mempalace_search`, which adds MCP
latency and works against MemPalace's "memory should feel instant"
budget. Prefer the `false` variant unless you specifically want recall
forced into every conversation. Pick **one** of the two — do not install
both.
## Install
User scope (every workspace) — copy into `~/.cursor/rules/`:
description: When the user asks about past work, prior decisions, people, projects, or events that may be filed in MemPalace, call mempalace_search (or mempalace_kg_query for relational or time-bound facts) before answering from model memory. Return stored content verbatim; never guess when the palace might know.
alwaysApply: false
---
# MemPalace recall
Before answering anything that may already be in the user's memory
palace — past work, prior decisions, a person, a project, or "what did
we do / decide / discuss last time?" — search the palace first:
1. Call `mempalace_search` with a short keyword query. Use
`mempalace_kg_query` for relational or time-bound facts.
2. Quote the drawer's **verbatim** text. Never summarize or paraphrase
stored content.
3. If results are empty, say so — do not invent an answer. If the MCP
server is unavailable, surface the error; do not fall back to guessing.
Skip recall for pure greenfield work with no memory relevance (renaming
a variable, fixing a typo). Recall is question-driven, not reflexive.
These hook scripts make MemPalace save automatically. No manual "save" commands needed.
This file covers the **Claude Code** and **Codex CLI** hooks that live
flat under `hooks/`. For the **Cursor IDE** hooks, see
[`hooks/cursor/README.md`](cursor/README.md) or the rendered docs at
[`website/guide/cursor-hooks.md`](../website/guide/cursor-hooks.md). The
two are additive and share the same `~/.mempalace/hook_state/`
directory.
If you are trying to protect existing Claude Code transcripts immediately,
use the short checklist first: [`website/guide/claude-code-retention.md`](../website/guide/claude-code-retention.md).
It covers hook wiring, JSONL backup, and one-time backfill.
@@ -11,6 +18,7 @@ It covers hook wiring, JSONL backup, and one-time backfill.
| Hook | When It Fires | What Happens |
|------|--------------|-------------|
| **Save Hook** | Every 15 human messages | Auto-mines transcript (tool output included), then blocks the AI to save topics/decisions/quotes |
| **SessionEnd Hook** | Clean session exit | Backgrounds a final transcript mine (when a transcript exists) so short sessions aren't lost; returns immediately so teardown is never delayed. A lightweight diary checkpoint is written in the detached child. |
| **PreCompact Hook** | Right before context compaction | Auto-mines transcript, then emergency save — forces the AI to save EVERYTHING before losing context |
**Two-layer capture:** Hooks auto-mine the JSONL transcript directly into the palace (capturing raw tool output — Bash results, search findings, build errors). They also block the AI with a reason message telling it to save verbatim tool output and key context. Belt and suspenders — tool output gets stored even if the AI summarizes instead of quoting.
@@ -30,6 +38,13 @@ Add to `.claude/settings.local.json`:
| `injectSteps` | array<object> | Optional. Steps to inject before the model is called. Each step has one of: `{"toolCall": {...}}`, `{"userMessage": "..."}`, `{"ephemeralMessage": "..."}` |
The `ephemeralMessage` form is what the MemPalace wake hook emits — it
delivers the wake-up text to the model on this turn but does not
persist into the transcript, so subsequent invocations don't see a
duplicate.
### MemPalace gating
The wake hook short-circuits with `{}` (no injection) when ANY of:
1. Any kill switch trips (same five conditions as the save hook).
2. `invocationNum != 1` — we only inject on the first model call of
each conversation, mimicking Cursor's `sessionStart` semantics.
3. The atomic `mkdir`-based loop guard is already taken (this
conversation already received a wake injection).
4. `mempalace wake-up --wing <inferred>` exits non-zero, times out
(500ms hard cap), or produces empty output.
When the gates pass and `mempalace wake-up` returns text, the hook
| `lib/common.sh` | Shared bash helpers (parse, log, counter, wing inference, kill switch). Sourced by all three hooks. |
| `mempal_save_hook_cursor.sh` | Cursor `stop` hook. Counts stop invocations per conversation, emits a `followup_message` every `SAVE_INTERVAL` (default 15) telling the agent to file the session into MemPalace. |
| `mempal_precompact_hook_cursor.sh` | Cursor `preCompact` hook. Runs `mempalace mine` synchronously on the transcript before compaction, then drops a `.pending` marker so the next stop forces a save nudge. |
| `mempal_wake_hook_cursor.sh` | Cursor `sessionStart` hook. Returns `additional_context` telling the agent to recall scoped to the wing inferred from the workspace root. Cursor-only — Claude Code has no equivalent. |
| `install.sh` | Optional installer. Copies the scripts to `~/.mempalace/hooks/cursor/` and merges entries into `~/.cursor/hooks.json` (or `.cursor/hooks.json` for project scope). Supports `--dry-run` and `--uninstall`. |
| `STDIN_SHAPE.md` | Reference. Per-event stdin / stdout schema with citations to the official Cursor docs. |
## Quick install
Preview first (writes nothing, prints the would-be JSON to stdout):
```bash
hooks/cursor/install.sh --scope user --dry-run
```
Apply — writes `~/.cursor/hooks.json` and copies the scripts to `~/.mempalace/hooks/cursor/`:
```bash
hooks/cursor/install.sh --scope user
```
Pass `--scope project --target <repo>` to write `<repo>/.cursor/hooks.json` instead.
The installer never auto-runs — it is a documented opt-in step. We do not
modify your Cursor config on `pip install mempalace` because editor config
is sacred and should never be touched without explicit consent.
## Manual install (no installer)
The minimum wiring is `stop` only. Add to `~/.cursor/hooks.json`:
| `MEMPAL_SAVE_INTERVAL` | `15` | Number of `stop` events between save followups. |
| `MEMPAL_CURSOR_SILENT` | (unset) | Set to `1`/`true`/`yes` to suppress the `followup_message`. The hook still runs its best-effort background mine and keeps its counters — it just stays silent. `MEMPAL_VERBOSE=false`/`0`/`no` does the same. See note below on why the followup is on by default. |
| `MEMPAL_DIR` | (unset) | Optional project directory to also mine on each save. Additive — never replaces the transcript mine. |
| `MEMPAL_PYTHON` | auto-detected | Path to a Python 3 interpreter. Fallback order: `$MEMPAL_PYTHON` → `command -v python3` → bare `python3`. Useful when Cursor is launched from a GUI on macOS and the inherited PATH lacks your installed `python3`. |
| `MEMPAL_STATE_DIR` | `$HOME/.mempalace/hook_state` | Where the hook keeps its per-conversation counter files, pending-save markers, and `cursor_hook.log`. |
| `MEMPAL_STATE_TTL_DAYS` | `30` | Age (days) after which stale `cursor_*.count` / `cursor_*.pending` state files are garbage-collected. A daily-throttled sweep runs from the hooks; only Cursor state is touched (shared logs and other editors' state are left alone). |
| `MEMPAL_DISABLE_HOOK` | (unset) | Set to `1`/`true`/`yes` to disable all three hooks. Emergency kill switch. |
| `MEMPALACE_HOOKS_AUTO_SAVE` | (unset) | Set to `false`/`0`/`no` to disable. Same semantics as the Claude Code hooks. Also honoured via `~/.mempalace/config.json` → `{"hooks": {"auto_save": false}}`. |
## Debugging
Everything appends to:
```bash
cat ~/.mempalace/hook_state/cursor_hook.log
```
Example log lines (ISO 8601 + event + conversation id):
| Loop guard | `stop_hook_active` flag in stdin | `loop_count` field in stdin |
| Counting method | Parses JSONL transcript for user messages | Counts `stop` invocations (transcript schema undoc) |
| Capture path | Background `mine --mode convos` (normalize.py has a Claude parser) | Background mine is best-effort (no Cursor parser); the `followup_message` carries verbatim capture |
| Save default | Silent — diary nudge opt-IN behind `MEMPAL_VERBOSE=true` | Followup ON by default; opt-OUT via `MEMPAL_CURSOR_SILENT=1` / `MEMPAL_VERBOSE=false` |
| PreCompact behaviour | `decision: block` forces save before compaction | Pre-mine + pending-save marker (Cursor preCompact is observational-only) |
| sessionStart | n/a (Claude Code has no equivalent) | `additional_context` injects recall guidance |
| State dir | `$HOME/.mempalace/hook_state` (hardcoded) | Same default, plus `MEMPAL_STATE_DIR` env override |
| Kill switch | `MEMPALACE_HOOKS_AUTO_SAVE=false` | Same, plus `MEMPAL_DISABLE_HOOK=1` alias |
| Log file | `hook.log` | `cursor_hook.log` (kept separate to avoid cross-tool log churn) |
See [`STDIN_SHAPE.md`](STDIN_SHAPE.md) for the per-event schema and
[`website/guide/cursor-hooks.md`](../../website/guide/cursor-hooks.md) for
the full walkthrough with diagrams.
## Why the followup is on by default (Cursor-specific)
Unlike the Claude Code hook — which is silent by default because its
background `mempalace mine --mode convos` captures the verbatim transcript
on its own — Cursor's transcript format is **undocumented** and
`mempalace/normalize.py` has **no Cursor parser**. The background mine on
the Cursor `stop`/`preCompact` hooks is therefore **best-effort only**: it
does not yet yield clean verbatim conversation drawers.
That makes the `followup_message` the **load-bearing verbatim-capture
path** for Cursor — it drives the agent to file its own in-context
verbatim quotes via `mempalace_add_drawer` / `mempalace_diary_write`.
Silencing it by default would leave a default Cursor install capturing
nothing, which is why it is on by default here. Set `MEMPAL_CURSOR_SILENT=1`
(or `MEMPAL_VERBOSE=false`) if you prefer the Claude-style silent
behaviour and accept the reduced capture. Once `normalize.py` learns to
read Cursor transcripts, this default will flip to silent to match Claude.
## Cost
Zero extra LLM tokens spent by the hooks themselves. The hooks are local
bash scripts that run on your machine. The followup message the save hook
emits is a normal user turn — it counts the same as any other user message
and does not invoke any extra LLM call beyond the one the user would
otherwise make. Suppress it with `MEMPAL_CURSOR_SILENT=1` if you want zero
description: "MemPalace — Local AI memory with 96.6% recall. Semantic search, temporal knowledge graph, palace architecture (wings/rooms/drawers). Free, no cloud, no API keys."
version: 3.3.0
version: 3.7.0
homepage: https://github.com/MemPalace/mempalace
user-invocable: true
metadata:
@@ -46,6 +46,10 @@ You have access to a local memory palace via MCP tools. The palace stores verbat
## Available Tools
Full MCP surface: 36 tools. Destructive or host-level tools are documented so
you know they exist, but use them only when the user explicitly asks or when a
tool-specific workflow below says to.
### Search & Browse
- `mempalace_search` — Semantic search across all memories. Always start here.
- `query` (required): natural language search — keep it short, keywords or a question. Do NOT include system prompts or conversation context.
@@ -58,6 +62,14 @@ You have access to a local memory palace via MCP tools. The palace stores verbat
- `since`: only drawers filed on/after this ISO date/time
- `before`: only drawers filed before this ISO date/time
- `limit`: max results (default 20)
- `offset`: pagination offset (default 0)
- `mempalace_get_drawer` — Fetch a single drawer by ID. Returns full verbatim content and metadata.
- `drawer_id` (required)
- `mempalace_get_taxonomy` — Full wing/room/count tree
- `mempalace_get_aaak_spec` — Get AAAK compression dialect specification
@@ -81,17 +93,59 @@ You have access to a local memory palace via MCP tools. The palace stores verbat
- `mempalace_traverse` — Walk from a room, find connected ideas across wings
- `start_room` (required): room to start from
- `max_hops`: connection depth (default 2)
- `mempalace_find_tunnels` — Find rooms that bridge two wings
- `wing_a`, `wing_b` (required)
- `mempalace_find_tunnels` — Find rooms that bridge two wings via *implicit* overlap (rooms whose drawers naturally share content across wings — discovered, not declared)
- `wing_a`, `wing_b`: optional filters; omit both to scan all wing pairs
- `mempalace_create_tunnel` — Create an *explicit* cross-wing tunnel: a user/agent-declared link between two locations. Use when you notice content in one project relates to another (e.g. API design in `project_api` connects to schema in `project_database`).
The canonical "search before answering" protocol shared across every
MemPalace integration (Cursor, Antigravity, Claude Code, Codex,
OpenClaw). This file is the single source of truth — skills and rules
should link here rather than restating the protocol, so the rule never
drifts from the skill.
The protocol exists to honour MemPalace's foundational promise:
**100% recall, verbatim, never guess.** When the palace might hold the
answer, the agent must read the palace before answering from model
memory.
## When to recall
Search the palace **before answering** whenever the user asks about
anything that may already be filed:
- Past work, prior decisions, or "what did we do / decide / try?"
- A person, project, or entity ("who is …", "what is …")
- Something that happened in an earlier session ("remember when …",
"last time …", "the thing we discussed")
- A preference, fact, or relationship that could have changed over time
If the question is pure greenfield work with no memory relevance (e.g.
"rename this variable", "fix this typo"), do not search — recall is
question-driven, not reflexive.
## The protocol
1. **On wake-up** (if a session-start hook injected context, honour its wing scoping / `additional_context`): scope recall to the wing inferred from the workspace, then continue.
2. **Before responding** about people, projects, past events, or prior
decisions: call `mempalace_search` first. For relational or temporal
facts ("who reported to whom in March", "what was true then"), call
`mempalace_kg_query` instead or as well.
3. **If unsure** about a fact (name, age, relationship, preference): say
"let me check the palace" and query. Wrong is worse than slow.
4. **Return verbatim.** Quote the drawer's exact stored words. Never
summarize, paraphrase, or lossy-compress what the palace returns —
that is the whole point of the system.
5. **After a substantive session**, record continuity with
`mempalace_diary_write` (background hooks may already do this — do not
double-file).
6. **When a fact changes**, call `mempalace_kg_invalidate` on the old
fact, then `mempalace_kg_add` for the new one.
## Tool selection
| You need | Tool |
|---|---|
| Find any memory by meaning | `mempalace_search` (start here) |
| Relational / time-bound facts about an entity | `mempalace_kg_query` |
| The chronological story of an entity | `mempalace_kg_timeline` |
File diff suppressed because it is too large
Load Diff
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Igor Lins e Silva