Restore the fresh-palace dry-run regression alongside the existing-empty
and initialized-sqlite_exact cases. Cover explicit failure for unsupported
adapter result types and daemon source-adapter dispatch. Ignore Hypothesis'
test cache so it cannot be committed.
- existing-uninitialized palace dir dry run: assert no chroma.sqlite3
- initialized sqlite_exact palace dry run: assert artifacts byte-identical
- replace mocked MineAlreadyRunning CLI test with a real held writer lease
The section only ever documented `docker build`, so every reader
compiled the image locally even though `ghcr.io/mempalace/mempalace` is
published multi-arch. Worse, a clone builds `develop` (the default
branch), not the release — a build and a pull could hand you different
versions with no hint that they differ.
Lead with `docker pull`, and cover the things that actually cost people
an evening:
- the container only sees what you mount, so the MCP client config now
mounts a transcripts directory; without it the server starts fine and
every mine finds nothing
- paths become container paths after that, and `~` / `$HOME` are not
expanded by every MCP client
- the first embedding call downloads ~80 MB (minilm) or ~300 MB
(embeddinggemma) into /data, which reads as a hung container
- bind mounts keep host ownership and the image runs as uid 1000, so a
0700 directory fails with a bare PermissionError on Linux; Docker
Desktop's uid mapping hides this on macOS and Windows. `--user` is
called out as the wrong fix — /data is mode 700 owned by uid 1000, so
another uid cannot write the palace at all
- mining never writes to its source, so the examples mount it read-only
- the GPU image is x86_64-only; onnxruntime-gpu has no aarch64 Linux
wheels, so that build fails on Apple Silicon
Every command in the section was run as written against the published
image, and each claim checked rather than assumed: the uid and mode of
/data read out of the image, `--user` confirmed to break a *fresh*
volume, the read-only mount confirmed to still mine, and the aarch64 GPU
failure reproduced.
The first CI run failed at step 3 with
PermissionError: [Errno 13] Permission denied: '/work/mempalace.yaml'
`mktemp -d` creates the fixture 0700 owned by the runner user. Bind
mounts carry host ownership through unchanged, and the image runs as
uid 1000, so the container could not stat inside /work. Docker
Desktop's uid mapping hides this on macOS, which is why it passed
locally and only failed on Linux.
Model an ordinary project checkout instead — 0755 dir, 0644 file — which
is the shape that makes the README's `-v /path/to/project:/work` work
against a normal repo.
The Docker workflow built both images and never started a container,
and never parsed a Compose file. A green run therefore only meant the
Dockerfile compiled. Two defects that break the very first documented
command shipped past it: `docker-compose.yml` carried a bare
`environment:` key that made Compose reject the file outright (#2188),
and `_embed_texts` handed chromadb `np.float32` scalars so `mine`
aborted on the first drawer (#2187).
Add `scripts/docker-smoke.sh`, which exercises what the README tells
users to run:
1. `compose config` on docker-compose.yml and the server compose file
2. entrypoint dispatch for both `cli ...` and bare passthrough
3. `mine` a mounted directory, asserting a drawer is filed
4. `search` from a *separate* container, asserting the stored text
comes back verbatim — this is the assertion that matters, since
storing user words exactly is the promise the palace makes
5. a real MCP stdio JSON-RPC handshake: initialize, tools/list, and a
mempalace_search call whose result must contain the drawer
It asserts on returned content, not just exit codes, and lives in a
script rather than inline YAML so it runs identically on a laptop:
`scripts/docker-smoke.sh <image>`.
The new `smoke` job builds amd64 natively with `load: true` (buildx
cannot load a multi-arch manifest) and reads the publish job's cache
while writing its own scope, so an amd64-only export never lands on top
of the multi-arch one. `build` now needs it, so a failing smoke test
blocks publication rather than being noticed afterwards.
Verified by reintroducing each defect against a real build: the compose
regression fails at step 1, the embedding regression at step 3, and the
current tree passes all five. Failure output is clipped to 500 columns
because a rejected embedding batch otherwise prints a whole 384-dim
vector on one line and buries the message.
`environment:` was present with nothing but comments beneath it, so YAML
parsed it as null and Compose refused the whole file:
services.mcp.environment must be a mapping
That is every documented Compose command — `docker compose build`,
`docker compose run --rm mcp`, `docker compose down` — failing before
anything starts, on any machine. The README points at this file.
Comment the key out along with its example entries, and say in the file
why it cannot be left bare. The examples switch to mapping syntax so
uncommenting them yields a valid block.
Verified with `docker compose config` on the shipped file (now valid)
and by running the documented flow against a build of this tree: mine a
mounted directory through `docker compose run --rm mcp cli mine`, then
read it back with `... cli search`, which returns the drawer verbatim.
`deploy/docker-compose.server.yml` was checked for the same defect and
is unaffected.
Chroma declares `requires_explicit_embeddings`, so every write on the
default backend routes through `EmbeddingCollection`. `_embed_texts`
built its rows with `list(v)`, and `v` is a float32 `np.ndarray` — that
unpacks into `np.float32` *scalars*, which chromadb's
`normalize_embeddings` rejects:
ValueError: Expected embeddings to be a list of floats or ints, a
list of lists, a numpy array, or a list of numpy arrays
`mine` aborted on the first drawer, as did every other write against a
default palace. Convert with `.tolist()` (C-speed), keeping a
`float(x)` branch for embedders that already return plain sequences.
The suite could not see this. conftest's autouse
`_stable_embedding_function_for_tests` monkeypatches
`embedding_wrapper._embed_texts` itself for every module outside
`_REAL_EMBEDDING_TEST_MODULES`, so the defective function was never
executed under test. The regression tests therefore go in
`test_embedding.py`, which is exempt from that stub: one asserts the
returned elements are builtin floats, one drives a real Chroma
collection through `EmbeddingCollection.upsert` and reads the document
back. Both fail against the previous line with the production
ValueError.
Verified end to end outside the suite: mining a project and searching
it back returns the drawer verbatim, on the host and in the container
image built from this tree.
`mempalace_diary_write` returns an `entry_id` for every diary entry, but
for entries large enough to be chunked that id was unusable: get_drawer,
update_drawer and delete_drawer all answered "Drawer not found", and
list_drawers showed the entry as N unrelated chunk rows.
Two metadata conventions never met. The diary chunking path stamped
`parent_entry_id` on each chunk, while the logical-id read paths added in
#1782 query only `parent_drawer_id`. Both keys mean the same thing --
"physical chunk of this logical drawer" -- so chunk groups written by
diary_write were invisible to logical-id resolution. Same bug class as
#1763, which #1782 fixed for `add_drawer` drawers only.
Read paths now resolve either key via `_PARENT_ID_KEYS`:
- `_logical_chunk_group()` matches both with an `$or` (fixes get /
update / delete). All four backends support `$or`.
- `_collapse_drawer_rows()` groups on either (fixes list_drawers, which
the `$or` alone does not cover).
- `searcher._result_drawer_id()` resolves either, so a hit on a chunked
diary entry reports the id that fetches the whole entry rather than
the single chunk that matched.
New diary writes also stamp `parent_drawer_id` alongside
`parent_entry_id` so the two conventions converge going forward. Because
the read paths still accept the `parent_entry_id`-only shape, palaces
written before this fix are repaired with no data migration.
Diary chunks are written without `source_file`, so neighbor expansion
(#1580) returns early on them and is unaffected by the added key.
Also drops the comment telling callers to iterate `chunk_ids` (it
documented the bug as intended behavior) and a stale claim that search
rejoins chunks via `parent_entry_id` -- no search code read that key.
The dependabot bump moved pyproject's ruff pin to 0.16.1, but nothing else
followed it, so the repo asked for three different versions at once:
- pyproject.toml said 0.16.1 (the bump)
- uv.lock still resolved 0.15.20 (dependabot did not update it)
- .github/workflows/ci.yml installed 0.15.14 by its own literal pin
The lint job never reads pyproject, so CI kept linting with 0.15.14 and
reported this PR green without 0.16.1 ever running. The ci.yml pin had
already drifted from pyproject before this bump, under a comment saying to
keep them identical.
- ci.yml: pin 0.16.1 to match pyproject.
- uv.lock: regenerated so the locked resolution agrees.
- test_ruff_pins_match: assert ci.yml and pyproject stay equal, so the next
bump that touches only one of them fails loudly instead of passing blind.
- extend-exclude '*.md': 0.16 began formatting Python inside markdown
fences, taking the formatter from 199 files to 295 and reflowing
hand-aligned example code in docs/rfcs/002 and three website pages.
Excluding docs keeps this a version bump rather than a silent
documentation reflow, and restores the exact file scope the project has
always formatted.
Verified with 0.16.1 actually installed: ruff check and ruff format --check
both clean over the same 199 files, full suite green (3669 passed).
Python's text layer applies universal-newline translation: CRLF becomes LF
on read, and on Windows LF expands back to CRLF on write. Every CLI path
that carried artifact and event content went through it, so a patch piped
in from a Windows agent was stored as different bytes than the ones git
produced — under a sha256 of the mangled content, which then failed to
match on the machine that applied it. For a store whose contract is
verbatim bytes addressed by digest, that is silent corruption.
It also disarmed the CRLF warning added for kind=patch: put_artifact looks
for a carriage return that the read path had already stripped, so the one
check meant to catch unappliable diffs could never fire on the platform
that produces them.
- _read_stdin_exact() / _write_stdout_exact(): go through sys.stdin.buffer
and sys.stdout.buffer, falling back to the text layer when a replaced
stream has no .buffer.
- --file / --body-file: read_bytes().decode() instead of read_text().
- artifact get --out: write_bytes() instead of write_text().
Verified through real pipes: a 63-byte CRLF patch round-trips at 63 bytes
with an unchanged sha256 via stdin, --out, and stdout, and the CRLF warning
now fires. Regression coverage in TestVerbatimNewlines, confirmed failing
against the old behavior first.
Config reads (peers.json, replica.json, server registry, token file) are
left on read_text: JSON is newline-insensitive and none of them carry user
content.
Reported by Copilot on the pull request.
The default (legacy) repair path ignored --dry-run and ran the real
rebuild: it deleted any existing <palace>.backup, copied the live palace
over it, re-filed the drawers collection through a staged temp copy, then
rebuilt FTS5 and VACUUMed. #2095 and #2133 fixed this for
--mode from-sqlite only.
The preview now returns before ChromaBackend() is constructed. Opening a
chromadb client is itself a write to chroma.sqlite3, so a preview that
reached one could not be inert; the row count comes from
sqlite_drawer_count instead, the read-only SQLite ground truth
check_extraction_safety already trusts. Staying off the chromadb layer
also keeps a dry run clear of the layer repair is separately reported to
segfault in on a large palace (#2113).
resolve_repair_preflight_errors() decides what a dry run does about the
FTS5 autoheal. The autoheal is a write, so a preview must not run it, but
skipping it routed an isolated inverted-index error into the abort banner
and exit 1 - telling the operator to run offline sqlite3 .recover on a
palace a real run heals by itself (#1596). The dry run now classifies the
errors with the same _errors_are_isolated_fts5 predicate the real path
gates on and continues; broader corruption still aborts in both modes.
It is worded as an attempt rather than a promise, because the real heal
still gives up when another process holds the mine lock, when the rebuild
raises, or when quick_check is still dirty afterwards.
The plan describes the real run in execution order. It names the
live-collection delete the rebuild performs, since "re-file via a staged
temp copy" alone reads as additive; it warns when an existing backup
would be deleted; and it reports a no-op instead of a rebuild when the
collection holds no rows. It states the #1208 truncation guard that can
abort the run, and reports that guard as disabled when
--confirm-truncation-ok is set, because check_extraction_safety returns
immediately then and the promised abort would not happen. An unreadable
count fails closed with a non-zero exit, for parity with the from-sqlite
preview.
A real ChatGPT conversations.json is a top-level array of conversation
objects. _try_chatgpt_json requires a single dict with a "mapping" key, and
no other parser in the chain claims the array either, so the export fell
through to the plain-text path: mine --mode convos reported success while
filing serialized JSON sliced at arbitrary offsets.
Unwrap the array and normalize each conversation separately, mirroring
_try_claude_ai_json_split, so a freshly downloaded export sitting next to the
previous one files only the conversations that are actually new.
The ChatGPT parser is now reached for every element of every top-level JSON
array under the mined tree, so type-check its nested shapes: ten of the
twelve malformed payloads now covered by tests raised AttributeError,
KeyError or TypeError, which would abort a whole mine run since convo_miner
catches only OSError and ValueError.
The tokenizer pads every row of a sub-batch to the longest sequence in it,
and attention costs batch x heads x length^2 per layer, so arrival order
decided what an embeddinggemma run cost: one long verbatim message dragged
its whole sub-batch up to its own length.
Sort by UTF-8 byte length before the split and scatter each row back to its
input index. An input that fits one sub-batch is left alone, since every row
pads to the same width either way.
Over 43,157 sweep drawers from 160 transcripts, padded token slots drop
39.7% and the quadratic attention term 45.0%. Wall clock on 52 messages with
one model instance: 1169.70 s to 490.83 s, control baseline 1286.23 s. Row
values move by at most one float32 ULP (1.2e-07, cosine 0.99999992), which
is reduction-order rounding, not a change of meaning.
Only palaces configured for embeddinggemma are affected; the default MiniLM
embedder pads to a fixed width.
Narrows the 3.7.0 slice to the feature users actually get: RFC 003 agent
coordination (events, artifacts, patch handoffs) plus the multi-master
logstream sync that makes it work across machines.
Removed, deferred with the rest of RFC 004:
- replica_sync.py / vector_cache.py and the `mempalace replica` CLI
- the /snapshot/* hub endpoints they backed
- website/concepts/replicated-palace.md
Memory read replicas were a leaf on the dependency graph (nothing in the
logstream path imports them), and shipping them half-done meant documenting
a mesh whose memory does not actually converge. Dropping them lets the docs
say one true thing instead of two hedged ones: coordination syncs, memory
stays local, point every agent at one hub if you want shared recall.
Fixes a bug found by running it: _start_peer_sync_thread() read peers.json
once at startup and returned early when absent, so a hub started before
peers.json was written never synced — silently, forever. That is the order
the guide tells users to follow. Membership is now re-read every round.
Verified on two live hubs: delegation loop end-to-end, verbatim patch
round-trip by sha256, bidirectional sync, CLI sync alongside a live hub,
and automatic convergence 15s after writing peers.json with no restart.
The website pages arrived from the full RFC 004 branch and documented the
memory op-log, `mempalace oplog`, and automatic bidirectional memory
convergence. None of that is in this slice: the CLI has logstream, artifact,
and replica, and memory moves by one-way `replica pull`.
- cli.md: drop the `mempalace oplog` section (the command does not exist);
note that full convergence is staged.
- replicated-palace.md: split the sync table into shipping (coordination,
multi-master) vs next (memory ops, bidirectional); say plainly that two
machines capturing independently do not merge yet.
- shared-brain.md: mark step 3 a pull you re-run, not a converging loop.
- RFC 004: record which steps landed.
The rebase resolution had taken the branch version of both files wholesale,
dropping develop's backend table, version badge, mempalace_kg_supersede
section, and the checkpoint added_by row. Rebuilt as develop's content plus
only the RFC 003 logstream additions; tool count is 43 (36 + 7 logstream).
Identity becomes the triple host:agent:project — the granularity at which an
actor actually shares filesystem, local config, and knowledge. Two sessions in
the same project on the same box are the SAME actor with the SAME identity;
session/PID is event metadata, never identity. Fixes the live collision where
two windows-claude sessions interleaved inboxes, claims, and diary under one
flat name.
Two decisions resolved (both live in the backend/hub):
- Matcher: NO change. Route the tuple as an opaque string on RFC 003's exact +
'*' broadcast; colons already pass _sanitize_routing. Hierarchical/glob
routing is deferred — it's a real index/correctness surface and no concrete
need has landed. This ships as a convention + one renderer change, zero risk
to the routing hot path.
- Order: host:agent:project (host-first), matching how mac-*/windows-*/blade-*
names already read; migration is a pure suffix append, not a rename.
Claim safety under a shared identity needs no lease server: natural mutex
(don't claim what your identity already claimed) plus RFC 004 R3's earliest-HLC
tiebreak for the residual race. The durable fix is one place — the renderer of
the mempalace-shared-brain block emits the tuple, so every box re-derives on
next sync instead of being hand-edited. Deferred behind the RFC 004 write-flip;
touches no storage or merge semantics.
(cherry picked from commit b043c29fd6a65ea32f92252b8afab1b62bc95bf5)
Event bodies are stored verbatim and fleet status updates run to several
KB, so listing many events full-body is a large payload — agents hit it
catching up on the stream. New preview=true truncates each body to a
200-char excerpt (marking body_truncated + body_length) while keeping
every routing/metadata field, so an agent can scan and then re-fetch
only the events it cares about in full via a targeted since_event_id.
Default (full verbatim) is unchanged; the verbatim-on-demand guarantee
is intact.
(cherry picked from commit dcf43984b84baea49838b2c665d62e572d259615)
The estate is observability — logstream + sync state + peers.json, no
FTS5 dependency; the profile's drawer count already degrades
gracefully. A damaged palace is exactly when mesh visibility matters
most. Caught live on the third replica: its FTS5 index went malformed
and the integrity gate refused the estate tool along with the palace
tools.
(cherry picked from commit c694a8401cb2efd0ca482344503956e4b4c5bad3)
Every hub now advertises a self-derived node profile: roles
(replica/agents/compute), resolved accelerator + embedder, live drawer
count, hardware string, advertised_at. Every field is derivation from
what the daemon observes about itself — never configuration, so mesh
dashboards render what each machine reported, not what a UI guessed.
Profiles ride the existing sync surfaces: /sync/version_vector gains
additive profile/profiles fields, each round's stats carry them, and
carriers relay profiles for origins they only know transitively —
/sync/peers now serves per-peer profiles plus origin_profiles keyed by
replica id. Unreachable peers keep their last advertised profile.
New MCP tool mempalace_mesh_peers returns exactly the /sync/peers
payload from the same producer function, so the desktop app consumes
the estate through the existing bridge — one integration, one token
path, and the tool and endpoint can never drift. Tokens are never in
the payload, asserted in tests.
(cherry picked from commit f31d0904167f5463285ae78de335bcea304ed71a)
Called out explicitly after proving it live: a pre-mesh palace joining
the replicated palace keeps everything (nothing re-mined, nothing lost),
serves its history to peers via one embed-cache + pull, and its drawers
are first-class mesh citizens. This is the path most existing users will
take, so the guarantee belongs in the guide, not in an RFC appendix.
(cherry picked from commit 1023368eed16f16e29544ba44f562872d26df101)
The docs ended at the single-hub logstream era; the mesh has since gone
from RFC to production. This brings the site current:
- NEW concepts/replicated-palace.md: one logical palace across N
machines — the availability invariant, the three layers, ops/HLC/
version vectors, gossip convergence, snapshot bootstrap + peers.json,
distributed embedding (sync the facts, derive the senses), the
/sync/peers estate surface, and the trust story (bearer tokens today,
mesh identity next). Status-honest: the memory op-log is documented
AS the dual-write shadow it currently is, with oplog verify named as
the cutover gate.
- reference/cli.md: mempalace replica (pull/embed-cache), mempalace
oplog (status/sync/fold/verify), logstream sync — with the
operational rules (hub-stopped pulls, quiescent origins,
MEMPALACE_SYNC_HTTP_TIMEOUT for big bootstraps).
- guide/shared-brain.md: new 'From hub to mesh' section — the
three-step join recipe and why the mesh removes the hub as a single
point of failure; See-also links updated.
- Sidebar: Replicated Palace under Concepts.
Example hostnames throughout; vitepress build clean.
(cherry picked from commit 63c2c3c20fcd350b8aca6b3249dd1a08ba5ecc22)
Layer 2 promised it would see peers only through a transport interface;
until now that seam existed in the RFC but not in the code. This lands
it: mempalace/transport.py defines the full contract (self_id / peers /
request now; open_stream / on_presence_change / on_inbound documented
and explicitly deferred to the MeshGuard binding with loud
NotImplementedError), and today's shipping behavior — tailnet HTTPS with
peers.json bearer tokens — becomes its first implementation,
HttpsBearerTransport, moved wholesale below the seam (load_peers,
PEERS_FILENAME, the wire primitive, the timeout knob).
get_transport(palace_path) is the single swap point: MEMPALACE_TRANSPORT
defaults to https; 'meshguard' is reserved and fails loudly until the
binding ships — a user who asked for mesh-identity auth must never
silently run on the bearer-token model it replaces. sync_all now
iterates the transport's membership snapshot (injectable for tests and
for the binding).
Compat surface preserved exactly: logsync re-exports load_peers /
PEERS_FILENAME / _peer_get, and SyncPeerError IS TransportError (same
class), so every existing importer and except-clause keeps working.
Context: three-replica mesh went live tonight on bearer tokens; joining
the third node took ~10 manual credential operations for 2 of 3 edges —
the N-squared out-of-band token relay this seam exists to retire.
MeshGuard trust-path sweep (RFC 004 pre-integration gate a) running in
parallel; the FFI binding (gate b) builds on this seam next.
tests: seam contract, https impl wiring, factory swap point incl.
reserved-name loudness, compat surface. Full suite: 3496 passed.
(cherry picked from commit 45b0aa335375af2e7c93eb9b4b805bcc6877b219)
PalaceMind's mesh view had no data source: the hub exposed version
vectors and op ranges but nothing that says which replicas exist, what
they're called, whether they were reachable last round, or how far
behind they are. A third replica joining the mesh was invisible to the
UI even though every version vector already carried its origin.
GET /sync/peers (bearer, lock-free like the rest of /sync) now serves
the RFC 004 estate data from this replica's point of view:
- self: replica_id, hostname, local version vector
- peers: configured peers.json entries (name + url — the token is
NEVER included) merged with last-round sync state recorded by the
anti-entropy loop: peer replica_id, reachable, last_success_at /
last_error(_at), last pulled counts, and the peer's remote version
vector (drift = remote vector vs self vector, computed by consumers)
- unnamed_origins: origins present in the local log but not configured
as peers — replicas known only transitively through a carrier
- sync_interval_s so consumers can pace their polling
sync_with_peer now returns remote_version_vector; the sync loop folds
each round's outcome into _PEER_SYNC_STATE (whole-entry replacement; an
error round preserves the last known good state so the UI can show
"unreachable since X, last seen at vector V").
tests: estate payload over real HTTP incl. token-privacy assertion and
a transitively-known origin; error-round state preservation.
Full suite: 3484 passed.
(cherry picked from commit ffb8f81ef41b2af32747f58f42b4f354b56d5f0b)
Grace Gettert
Igor Lins e Silva
dependabot[bot]
mvalentsev