Close the last open items on #743 before merge:
- Conformance: document two isolation arms (cross-id for all backends;
same-id/different-namespace for supports_namespace_isolation advertisers).
- No silent drop: non-advertising backends must raise UnsupportedCapabilityError
when PalaceRef.namespace is set, rather than accept-and-ignore.
- Wire require_namespace_support() into chroma/sqlite_exact; add conformance test.
- Refresh implementation-status banner now that #1727/#1731/#1732/#1734 landed.
#757 landed mtime/inode cache invalidation and mempalace_reconnect
in mcp_server._get_client(). Both are Chroma-specific (stat of
chroma.sqlite3). They should migrate into ChromaBackend.get_collection
and ChromaBackend.close_palace during the §10 cleanup so the freshness
contract lives inside the backend, not in the caller.
Addresses the actual spec defects flagged in #743 review, ignoring
operator-UX asks that are not plugin-contract concerns.
- Goal #3: 'without data loss' → mirrors §8.2's capability-conditional
lossless-vs-reembed framing. No more overpromise.
- §1.5: `server_embedder` is no longer an implicit escape hatch from
identity/dimension rules. Such backends MUST expose an effective
identity via `effective_embedder_identity()` and are bound by the
same three-state check.
- §7.3: adds `maintenance_kinds: ClassVar[frozenset[str]]` advertisement
mechanism. `run_maintenance(kind)` must raise
UnsupportedMaintenanceKindError for unadvertised kinds. Benchmark
harness reads this set rather than guessing kind names. Reserves
`analyze`/`compact`/`reindex` as well-defined names.
- §1.2: adds `update()` as optional method with a default get+merge+
upsert implementation. §2.1: `supports_update` redefined to gate
atomic single-round-trip semantics (not mere capability), since the
default impl already supports partial updates.
Operator asks explicitly NOT adopted (diplomatic shims, not contract
defects): `.to_dict()` compat on typed results, migration progress
reporting, `BaseBackend.repair()` separate from `run_maintenance`,
per-palace capability variance, identity recording on read-only ops.
Incorporates review feedback from skuznetsov (Postgres, #665) and
dekoza (Lance, #574) on issue #737:
- §1.5: split 'accepts embeddings=' (signature compliance) from
'persists embeddings as-is' (correctness). Adds
supports_embeddings_passthrough capability; the former is universal,
the latter is required to label a migration lossless.
- §1.5: model identity check becomes a three-state machine
(known_match / known_mismatch / unknown) so legacy palaces without
recorded identity don't hard-fail on upgrade.
- §1.4: makes explicit that supports_contains_fast is the ONLY
performance floor the spec promises; without it callers MUST assume
O(n). $contains is a correctness requirement, not a performance one.
- §3.3: clarifies auto-detect is an upgrade-compat path only, never
the selection mechanism for new palaces.
- §8.2: migrate CLI refuses to run against a target lacking
supports_embeddings_passthrough unless --accept-re-embed is passed;
migration record now captures lossless status and model identities.
Copilot review flagged back-references in §1.4 and §6 that still used
the pre-skuznetsov-rename names (`$contains_fast`, `sync_capable`,
`change_feed`). Updated to the `supports_*` prefix used in the §2.1
capability table.
Formalizes the BaseCollection/BaseBackend contract introduced as a seam
in #413 into an interchangeability spec that third-party backends can
build to. Driven by six in-flight backend PRs (#574, #643, #665, #697,
#700, #381) each implementing the interface differently.
Key decisions captured: entry-point distribution, typed QueryResult/
GetResult replacing Chroma dict shape, daemon-first multi-palace model
via PalaceRef, required where-clause subset (incl. $contains),
mandatory embedder injection with model-identity validation, capability
tokens, shared pytest conformance suite, and a backend-neutral
migrate/verify CLI.
The website pages arrived from the full RFC 004 branch and documented the
memory op-log, `mempalace oplog`, and automatic bidirectional memory
convergence. None of that is in this slice: the CLI has logstream, artifact,
and replica, and memory moves by one-way `replica pull`.
- cli.md: drop the `mempalace oplog` section (the command does not exist);
note that full convergence is staged.
- replicated-palace.md: split the sync table into shipping (coordination,
multi-master) vs next (memory ops, bidirectional); say plainly that two
machines capturing independently do not merge yet.
- shared-brain.md: mark step 3 a pull you re-run, not a converging loop.
- RFC 004: record which steps landed.
Identity becomes the triple host:agent:project — the granularity at which an
actor actually shares filesystem, local config, and knowledge. Two sessions in
the same project on the same box are the SAME actor with the SAME identity;
session/PID is event metadata, never identity. Fixes the live collision where
two windows-claude sessions interleaved inboxes, claims, and diary under one
flat name.
Two decisions resolved (both live in the backend/hub):
- Matcher: NO change. Route the tuple as an opaque string on RFC 003's exact +
'*' broadcast; colons already pass _sanitize_routing. Hierarchical/glob
routing is deferred — it's a real index/correctness surface and no concrete
need has landed. This ships as a convention + one renderer change, zero risk
to the routing hot path.
- Order: host:agent:project (host-first), matching how mac-*/windows-*/blade-*
names already read; migration is a pure suffix append, not a rename.
Claim safety under a shared identity needs no lease server: natural mutex
(don't claim what your identity already claimed) plus RFC 004 R3's earliest-HLC
tiebreak for the residual race. The durable fix is one place — the renderer of
the mempalace-shared-brain block emits the tuple, so every box re-derives on
next sync instead of being hand-edited. Deferred behind the RFC 004 write-flip;
touches no storage or merge semantics.
(cherry picked from commit b043c29fd6a65ea32f92252b8afab1b62bc95bf5)
Revisions kept forever (no GC path; search surfaces heads only); v4
migration staged on a copy before live-with-backup; step 2 ships as 2a
(drawers+KG ops) then 2b (registry/hallways); mac origin dual-writes
during a shadow period before cutover; remote write-flip (step 3) gated
on the local-capture promotion validating end-to-end.
(cherry picked from commit 34a264e12bdc3ba6f40877972e9435e3f912ee6d)
Driven by a real customer: the Windows machine holds un-mined projects and
conversations. Capture must happen where the data lives (mining reads local
disk), so step-1 replicas may mine locally now — unstamped drawers are
untouchable by reconciliation — and step 2 commits to a one-time promotion
pass turning them into drawer.add ops under the replica's identity.
Content-pure v4 ids double as the cross-machine dedup: identical content
mined on two origins collapses to one drawer at merge.
(cherry picked from commit d18b2a03519a1cfbe9e10bbf12c902c307195c15)
Transport/lifecycle sections (§5), transport-side security (§10.1, §10.3),
and the PalaceMind appendix authored by windows-claude and delivered as a
patch artifact over the logstream (correlation
rfc004_replicated_palace_position, artifact art_20260702T081251_76aef8fb111a,
sha256 verified, applied at ab6f362). §10.2 at-rest encryption drafted by
mac-claude on apply: per-replica data keys that never traverse the mesh,
two-tier design (FDE baseline + SQLCipher-style page encryption over the
canonical stores, OS-keystore wrapped), replica-local rotation, R7
composition with the revocation ceremony.
The seam contract (§5.1) verifies against the Layer 2 design: request/stream
map to anti-entropy pull + SSE push-notify, and ReplicaId doubles as
origin_replica so provenance and authentication are one fact. RFC 004 is now
fully drafted and awaiting Igor's review.
(cherry picked from commit 9f3333998673fb196664325396c104036f771d1f)
One logical palace per human, fully replicated per machine: agents always
local, services converge via provenance-stamped ops over an encrypted
mesh. Judged against offline as the default posture (R0-R8).
Storage sections drafted (mac-claude ownership per the logstream debate on
correlation rfc004_replicated_palace_position): canonical op-log with HLC +
per-origin version vectors, complete merge-semantics table, v4 content-pure
id migration (verified mutable-state inventory with file refs), backends
demoted to fold-and-index consumers, provenance/source-bound maintenance,
4-step sequencing starting with logstream multi-master. Transport (§5),
transport-side security (§10), and the PalaceMind appendix are reserved
stubs for windows-claude, delivered as patch artifacts to keep a single
committer per repo.
(cherry picked from commit ab6f362af7a2ce947c2c37c11910fa8aeb6b983d)
GET /logstream/stream on the hub HTTP transport: bearer-authenticated
Server-Sent Events with the event_list filter set, since_event_id /
Last-Event-ID replay-then-tail resume (live tail only without a cursor,
via new Logstream.latest_event_id), ~15s heartbeat comments, and a
bounded client pool (MEMPALACE_SSE_MAX_CLIENTS, default 8; 503 +
Retry-After on cap). Frames carry the exact event_list JSON envelope —
the compat surface promised to the PalaceMind desktop viewer.
Also fixes a latent hub-starvation hazard the first SSE consumer would
have tripped: do_POST serialized every request under the global HTTP
lock, so one agent's five-minute event_wait long-poll blocked the whole
fleet. Logstream tools (own WAL db, no Chroma/KG state) now dispatch
lock-free via _http_dispatch; everything else keeps the global lock.
Requested by claude-fable-5-windows for the PalaceMind live viewer
(correlation palacemind_desktop_logstream_viewer). Full suite: 3411
passed.
(cherry picked from commit a7c720507974acc2f02957ac24b7045503ad74da)
Patch authored by windows-codex and handed off through the coordination
layer itself: correlation_id task_logstream_e2e_dogfood, patch artifact
art_20260702T041319_cdf92e302efe (sha256 a17012d6...9f64c4, verified),
test log artifact art_20260702T041254_c46525e2d550 (72 passed on
Windows 11, Python 3.12.11). Applied on mac, re-verified (72 passed),
acked applied. Trailing-newline nit in the submitted diff repaired on
apply (producer stripped the final newline when storing the artifact).
(cherry picked from commit 38f3d35804155840cd5512411e9823fb38dc7e95)
Durable append-only event + artifact layer in logstream.sqlite3 inside
the active palace dir, so agents can delegate work, wait for replies,
and exchange exact patches through the shared hub.
Phase 1 (mempalace/logstream.py): events, artifacts, event_artifacts
per the RFC schema; WAL + per-instance lock, no Chroma dependency;
append/list/wait/ack/put_artifact/get_artifact/submit_patch; explicit
size limits (body 256 KiB, artifact 4 MiB, metadata 64 KiB); rowid-backed
exclusive since_event_id cursor exposed as seq; inclusive
since_created_at; to_agent filters match '*' broadcasts; artifact_ids
validated in-transaction so listed events never dangle.
Phase 2 (mcp_server.py): seven MCP tools. Mutating four join
_MUTATING_TOOLS (hidden + refused in --read-only). All seven are exempt
from the Chroma SQLite integrity gate; the mutating four are exempt from
the peer-writer lease via _PEER_WRITER_EXEMPT_TOOLS (logstream owns its
own WAL db, and a running mine must not block coordination).
mempalace_reconnect drains cached Logstream handles like the KG cache.
(cherry picked from commit cdf72a994133242da862b11b21033a4546c5e146)
Draft plugin specification for source adapters, mirroring RFC 001's
role for storage backends. Formalizes the contract six community
ingester PRs (#274, #23, #169, #232, #567, #98, #702) plus #981's
metadata-only mode have been reinventing ad-hoc, so adapter authors
can build to a stable surface.
Key decisions:
- Single ingest() method; lazy adapters yield SourceItemMetadata
ahead of drawers, eager adapters interleave
- Declared-transformation model (§1.4) replaces informal verbatim
promise with a verifiable one; byte_preserving adapters declare
the empty set, declared_lossy adapters enumerate. Existing
miner.py and the convo_miner+normalize pipeline map cleanly
- Palace is the incremental cursor via is_current(item, metadata);
no sidecar persistence
- Routing is adapter-owned; detect_room/detect_hall move into the
filesystem adapter
- Flat metadata per ChromaDB (RFC 001 §1.4) — entity hints as
json_string field, KG triples route to SQLite knowledge graph
- Closets stay core-built as a post-step; adapters may emit flat
closet_hints. Closes existing gap where convo drawers get no
closets
- No per-drawer field renames: source_file, filed_at, source_mtime,
added_by, normalize_version, entities, ingest_mode all preserved.
Spec adds adapter_name, adapter_version, privacy_class
§9 enumerates the cleanup PR prerequisites (mempalace/sources/
module, PalaceContext facade, KnowledgeGraph.add_triple gaining
backwards-compatible source_drawer_id + adapter_name params).
Tracking issue: #989
Igor Lins e Silva
bensig