Merge pull request #1052 from MemPalace/merge/main-into-release-3.3.2

release: merge main into release/3.3.2

.claude-plugin/marketplace.json

@@ -9,7 +9,7 @@
       "name": "mempalace",
       "source": "./.claude-plugin",
       "description": "AI memory system — mine projects and conversations into a searchable palace. 19 MCP tools, auto-save hooks, guided setup.",
-      "version": "3.3.1",
+      "version": "3.3.2",
       "author": {
         "name": "milla-jovovich"
       }

.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "mempalace",
-  "version": "3.3.1",
+  "version": "3.3.2",
   "description": "Give your AI a memory — mine projects and conversations into a searchable palace. 19 MCP tools, auto-save hooks, and guided setup.",
   "author": {
     "name": "milla-jovovich"

.codex-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "mempalace",
-  "version": "3.3.1",
+  "version": "3.3.2",
   "description": "Give your AI a memory — mine projects and conversations into a searchable palace. 19 MCP tools, auto-save hooks, and guided setup.",
   "author": {
     "name": "milla-jovovich"

CHANGELOG.md

@@ -6,6 +6,37 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 
 ---
 
+## [3.3.2] — 2026-04-19
+
+### Bug Fixes
+
+- Fix silent drop of `.jsonl` files in project miner; raise `MAX_FILE_SIZE` cap from 10 MB to 500 MB so large transcripts no longer fall through unnoticed. Adds a tandem **sweeper** — a message-level, timestamp-coordinated, idempotent safety net that catches anything the primary miner missed. (#998)
+- `mempalace sweep <target>` CLI to run the sweeper on demand against a transcript file or a directory. (#998)
+- Guard `Layer3.search_raw` against `None` doc/meta rows returned by ChromaDB — prevents `AttributeError` crashes on mixed-schema palaces. (#1011, #1013)
+- Guard searcher API path, closet loop, and miner status histogram against `None` metadata; matching guards added to `tool_status` / `list_wings` / `list_rooms` / `get_taxonomy` in the MCP server. (#999)
+- Upgrade `chromadb` floor to `>=1.5.4` for Python 3.13 / 3.14 compatibility and pin upper bound to `<2` so future breaking majors don't silently install. (#1010)
+- Fix Unicode checkmark rendering on Windows terminals that can't encode the `✓` glyph — avoids `UnicodeEncodeError` crashes on first-run output. (#681)
+- **`quarantine_stale_hnsw`** — on open, detect HNSW segment directories whose `data_level0.bin` is significantly older than `chroma.sqlite3` and rename them out of the way. Recovers cleanly from HNSW/sqlite drift that otherwise causes SIGSEGV on `count()` / `query(...)` (the chroma-core/chroma#2594 failure mode). Rebuilds the index lazily on next use. (#1000)
+- **PID file guard** — `mine` writes a per-source-directory PID file and refuses to start if an existing mine is still running, preventing process stacking that bloats HNSW and wedges concurrent writes. Includes cross-platform PID liveness check (`os.kill(pid, 0)` terminates on Windows, so the guard falls back to a platform-aware probe). (#1023)
+
+### Improvements
+
+- **RFC 001 §10 — typed backend contracts.** `BaseBackend` now returns typed `QueryResult` / `GetResult` dataclasses and `PalaceRef` for palace identity; registry-based backend discovery. Internal refactor; no user-facing API change. (#995)
+- **RFC 002 §9 — source adapter scaffolding.** Introduces `BaseSourceAdapter`, adapter registry, and `PalaceContext` — the plumbing that future pluggable ingest sources will target. Internal refactor; no user-facing API change yet. (#1014)
+
+### Documentation
+
+- **RFC 002** — full specification for the source adapter plugin system (future pluggable ingest). (#990)
+- First-run help text and `README` now reference the real `~/.claude/projects/<project>/` path shape instead of the placeholder `/path/to/transcripts`. (#996, #1012)
+
+### Internal
+
+- Harden sweeper for production: verbatim tool blocks, full `session_id`, logged failures.
+- Address Copilot review on #995: cursor tie-break, honest metrics, accurate comments.
+- Test hygiene: avoid ONNX network download in update-length validation tests; dedup update-length-validation tests; fix Windows file-lock in cache-invalidation test.
+
+---
+
 ## [3.3.1] — 2026-04-16
 
 ### New Features

README.md

@@ -55,7 +55,7 @@ mempalace init ~/projects/myapp
 ```bash
 # Mine content into the palace
 mempalace mine ~/projects/myapp                    # project files
-mempalace mine ~/chats/ --mode convos              # conversation exports
+mempalace mine ~/.claude/projects/ --mode convos   # Claude Code sessions (scope with --wing per project)
 
 # Search
 mempalace search "why did we switch to GraphQL"
@@ -177,7 +177,7 @@ PRs welcome. See [CONTRIBUTING.md](CONTRIBUTING.md).
 MIT — see [LICENSE](LICENSE).
 
 <!-- Link Definitions -->
-[version-shield]: https://img.shields.io/badge/version-3.3.1-4dc9f6?style=flat-square&labelColor=0a0e14
+[version-shield]: https://img.shields.io/badge/version-3.3.2-4dc9f6?style=flat-square&labelColor=0a0e14
 [release-link]: https://github.com/MemPalace/mempalace/releases
 [python-shield]: https://img.shields.io/badge/python-3.9+-7dd8f8?style=flat-square&labelColor=0a0e14&logo=python&logoColor=7dd8f8
 [python-link]: https://www.python.org/

docs/rfcs/002-source-adapter-plugin-spec.md

@@ -0,0 +1,768 @@
+# RFC 002 — Source Adapter Plugin Specification
+
+- **Status:** Draft
+- **Tracking issue:** [#989](https://github.com/MemPalace/mempalace/issues/989)
+- **Related:** [#274](https://github.com/MemPalace/mempalace/issues/274), [#23](https://github.com/MemPalace/mempalace/pull/23), [#169](https://github.com/MemPalace/mempalace/pull/169), [#232](https://github.com/MemPalace/mempalace/pull/232), [#567](https://github.com/MemPalace/mempalace/pull/567), [#98](https://github.com/MemPalace/mempalace/pull/98), [#591](https://github.com/MemPalace/mempalace/pull/591), [#592](https://github.com/MemPalace/mempalace/pull/592), [#702](https://github.com/MemPalace/mempalace/pull/702), [#981](https://github.com/MemPalace/mempalace/issues/981), [#244](https://github.com/MemPalace/mempalace/pull/244), [#419](https://github.com/MemPalace/mempalace/pull/419), [#300](https://github.com/MemPalace/mempalace/pull/300), [#952](https://github.com/MemPalace/mempalace/pull/952), [#389](https://github.com/MemPalace/mempalace/pull/389), [#434](https://github.com/MemPalace/mempalace/pull/434)
+- **Sibling spec:** [RFC 001 — Storage Backend Plugin Specification](001-storage-backend-plugin-spec.md)
+- **Spec version:** `1.0`
+
+## Summary
+
+A formal contract for MemPalace source adapters so third parties can ship `pip install mempalace-source-<name>` packages (Cursor, OpenCode, git, Slack, Notion, email, calendar, Whisper transcripts, …) that drop into `mempalace mine` without patching core. The spec defines the adapter interface, record shape, metadata schema contract, privacy class, entry-point registration, incremental-ingest semantics, closet integration, a declared-transformation model that replaces the informal "verbatim" promise with a verifiable one, conformance tests, and the refactor of the existing file and conversation miners into first-party adapters on the same contract.
+
+RFC 001 formalized the write side (where drawers are stored). This RFC formalizes the read side (where content comes from). Both are required for MemPalace to function as a durable daemon managing heterogeneous palaces across many source types.
+
+## Motivation
+
+Six source ingesters are currently in flight, each solving the same problem a different way:
+
+| PR / Issue | Source | Mechanism |
+|---|---|---|
+| [#274](https://github.com/MemPalace/mempalace/issues/274) | Cursor | `workspaceStorage/*.vscdb` SQLite extraction |
+| [#23](https://github.com/MemPalace/mempalace/pull/23) | OpenCode | SQLite session database |
+| [#169](https://github.com/MemPalace/mempalace/pull/169) | Pi agent | JSONL session normalizer |
+| [#232](https://github.com/MemPalace/mempalace/pull/232) | Cursor (JSONL variant) | JSONL normalizer |
+| [#567](https://github.com/MemPalace/mempalace/pull/567), [#98](https://github.com/MemPalace/mempalace/pull/98) | Git | `git log` + `gh pr view` with structured diff summary |
+| [#591](https://github.com/MemPalace/mempalace/pull/591), [#592](https://github.com/MemPalace/mempalace/pull/592) | Delphi Oracle | Real-time intelligence signals |
+| [#702](https://github.com/MemPalace/mempalace/pull/702) | Cursor + factory.ai | Combined session miners |
+
+Plus three ingesters already grafted into core:
+
+- `mempalace/miner.py` — filesystem project miner, fixed char-window chunking, keyword hall routing
+- `mempalace/convo_miner.py` — chat transcript miner with exchange-pair chunking
+- `mempalace/normalize.py` — format detection for four chat-export shapes (Claude Code JSONL, Codex JSONL, Claude.ai / ChatGPT / Slack JSON)
+
+Plus one open proposal for a different ingest semantic:
+
+- [#981](https://github.com/MemPalace/mempalace/issues/981) — path-level descriptions: mine metadata-as-content instead of raw bytes for matched paths. This is a legitimate third ingest mode (alongside chunked-content and whole-record) that the current architecture has no home for.
+
+Each contributor has reinvented source discovery, source-item identity, incremental-ingest bookkeeping, metadata shape, and chunking strategy. Format detection for new chat exports lands in `normalize.py` as one more branch in an `if` chain. There is no shared abstraction, no conformance suite, and no contract new adapter authors can build against.
+
+This is the same situation RFC 001 addresses for storage backends: a pattern that emerged organically, now needs a specification so the community can contribute cleanly and enterprises can build against a stable surface.
+
+### Why this matters beyond developer tooling
+
+The adapter pattern is source-agnostic. What has so far shown up as "Cursor transcripts" and "git commits" generalizes to:
+
+- **Knowledge work** — Notion, Obsidian, Logseq, Google Docs, iA Writer, Zettlr
+- **Communications** — Slack, Discord, Teams, Signal backups, mbox/eml email, iMessage
+- **Research** — arXiv PDFs, Zotero libraries, bookmarked articles, Kindle highlights, web archives
+- **Creator workflows** — YouTube captions, podcast transcripts (Whisper/Deepgram), Descript projects
+- **Regulated domains** — medical records, legal filings, financial statements (all gated on §6 privacy class)
+
+Enterprises key on their own domain metadata — `repo/PR/SHA` for engineering, `patient/encounter/CPT` for healthcare, `case/docket/jurisdiction` for legal. The schema lives in the adapter; the content lives in the drawer. This is how structured-data use cases are served without violating the byte-preservation commitments adapters make.
+
+## Goals
+
+1. A source adapter ships as a standalone Python package; `pip install mempalace-source-<name>` is sufficient to use it.
+2. `mempalace mine` and the MCP mine tool are source-agnostic — all extraction goes through registered adapters. No `if source_type == 'foo'` branches in core.
+3. Content transformations are **declared** (§1.4): each adapter advertises the set of transformations it applies to source bytes. Byte-preserving adapters declare the empty set. Consumers can programmatically determine what happened to their data.
+4. Incremental ingest is cheap and correct: re-running mine only touches items whose source-side version changed, using the palace itself as the cursor (no sidecar).
+5. Each adapter declares a structured metadata schema. Enterprises index and filter on that schema. Core is schema-agnostic beyond the universal fields in §5.1.
+6. The existing `miner.py` and `convo_miner.py` become the first two first-party adapters on the new contract. Drawer metadata fields and field names are preserved — the spec adds fields, does not rename them.
+7. A privacy class is declarable at the adapter boundary so sensitive sources (medical, financial, personal comms) are handled with explicit policy rather than implicit trust.
+
+## Non-goals
+
+- Defining chunking. Each adapter owns its chunking strategy — tree-sitter for code, exchange-pair for chat, whole-record for a PR. Core does not impose a chunk size.
+- Defining live-stream / webhook shapes (the Delphi Oracle pattern of continuous signal ingestion). That is a separate future RFC; v1 is pull-mode.
+- Defining LLM-based structured extraction. Adapters MAY use an LLM; the spec does not mandate or standardize this.
+- Defining cross-adapter dedup. When the same content appears via two adapters (e.g., a PR body mined via `git` and as a conversation quote mined via `claude-code`), both drawers land. Deduplication policy is a separate concern handled at query time by `searcher.py`.
+- Defining closet construction. Core continues to build closets from adapter-yielded drawers (§1.7); the closet-building algorithm itself is not part of this spec.
+
+---
+
+## 1. Source adapter contract
+
+### 1.1 Required method
+
+All adapters implement `BaseSourceAdapter` with a single kwargs-only ingest method:
+
+```python
+class BaseSourceAdapter(ABC):
+    @abstractmethod
+    def ingest(
+        self,
+        *,
+        source: SourceRef,
+        palace: PalaceContext,
+    ) -> Iterator[IngestResult]:
+        """Enumerate and extract content from a source.
+
+        Yields a stream of IngestResult values. Lazy adapters yield
+        `SourceItemMetadata` ahead of the drawers for that item, so core
+        can report progress and check `is_current` before the adapter
+        commits to the fetch. Adapters with no lazy-fetch benefit may
+        interleave `SourceItemMetadata` and `DrawerRecord` items freely.
+        """
+
+    @abstractmethod
+    def describe_schema(self) -> AdapterSchema:
+        """Declare the structured metadata this adapter attaches.
+
+        Returned value is stable for a given adapter version. Enterprises
+        index on this schema; core uses it to validate adapter output.
+        """
+```
+
+The single-method `ingest()` contract was chosen over a `discover` / `extract` split. Most current ingesters have no meaningful laziness benefit (filesystem walking is cheap, transcript normalizing is cheap). Adapters that do (git-mine's `gh pr list` vs `gh pr view`; hypothetical Slack/Notion API) express laziness by yielding `SourceItemMetadata` first and deferring fetch until core confirms staleness via `is_current()`.
+
+### 1.2 Optional methods (default implementations on the ABC)
+
+```python
+def is_current(
+    self,
+    *,
+    item: SourceItemMetadata,
+    existing_metadata: dict | None,
+) -> bool:
+    """Return True if the palace already has an up-to-date copy.
+
+    Called by core after querying the palace for existing drawers with
+    matching source_file. The adapter compares its version token against
+    the stored metadata and returns True to skip extraction.
+
+    Default implementation: returns False (always re-extract). Adapters
+    advertising `supports_incremental` override this.
+    """
+    return False
+
+def source_summary(self, *, source: SourceRef) -> SourceSummary:
+    """Describe a source without extracting (e.g., 'git repo mempalace,
+    847 commits, 132 PRs'). Default: returns empty summary."""
+    return SourceSummary(description=self.name)
+
+def close(self) -> None:
+    return None
+```
+
+Core's incremental loop (pseudocode):
+
+```python
+for result in adapter.ingest(source=source, palace=ctx):
+    if isinstance(result, SourceItemMetadata):
+        existing = ctx.collection.get(where={"source_file": result.source_file}, limit=1)
+        if adapter.is_current(item=result, existing_metadata=existing):
+            ctx.skip_current_item()   # adapter stops yielding drawers for this item
+    elif isinstance(result, DrawerRecord):
+        ctx.upsert_drawer(result)
+```
+
+### 1.3 Typed records
+
+```python
+@dataclass(frozen=True)
+class SourceRef:
+    """A handle to the source a user wants to ingest.
+
+    local_path is for filesystem-rooted sources (project dir, mbox file).
+    uri is for URL-like references (github.com/org/repo, slack://workspace/channel).
+    options carries adapter-specific config (non-secret values only; §M2).
+    """
+    local_path: str | None = None
+    uri: str | None = None
+    options: dict = field(default_factory=dict)
+
+@dataclass(frozen=True)
+class SourceItemMetadata:
+    """Lightweight pointer yielded before drawers for lazy-fetch adapters."""
+    source_file: str                 # Logical identity — filesystem path, PR URI, etc.
+    version: str                     # Source-side version token (mtime, commit SHA, ETag, rev id).
+    size_hint: int | None = None     # Bytes, if known. Used for progress reporting.
+    route_hint: RouteHint | None = None
+
+@dataclass(frozen=True)
+class DrawerRecord:
+    """One drawer's worth of content plus metadata."""
+    content: str                     # Subject to §1.4 declared transformations.
+    source_file: str                 # Foreign key to SourceItemMetadata.source_file.
+    chunk_index: int = 0             # 0 for single-drawer items; 0..N-1 for chunked items.
+    metadata: dict = field(default_factory=dict)  # Flat: str/int/float/bool only. Must conform to adapter schema.
+    route_hint: RouteHint | None = None
+
+@dataclass(frozen=True)
+class RouteHint:
+    wing: str | None = None
+    room: str | None = None
+    hall: str | None = None
+
+@dataclass(frozen=True)
+class SourceSummary:
+    description: str
+    item_count: int | None = None
+
+# IngestResult is the union type adapters yield.
+IngestResult = SourceItemMetadata | DrawerRecord
+
+# PalaceContext carries collection handles, palace config, and progress hooks
+# into the adapter. Full definition in §9 (cleanup prerequisite).
+```
+
+### 1.4 Declared transformations
+
+Adapters cannot silently alter content. Every adapter declares the set of transformations it applies:
+
+```python
+class BaseSourceAdapter(ABC):
+    declared_transformations: ClassVar[frozenset[str]] = frozenset()
+```
+
+The invariant: **no transformation is applied that is not declared in this set**. Adapters declaring `frozenset()` are byte-preserving end-to-end (modulo the read, which may itself involve `utf8_replace_invalid` — see below).
+
+Reserved transformation names (v1):
+
+| Name | Meaning |
+|---|---|
+| `utf8_replace_invalid` | Undecodable bytes replaced with U+FFFD on read (equivalent to `open(..., errors="replace")`). |
+| `newline_normalize` | CRLF / CR converted to LF. |
+| `whitespace_trim` | Leading / trailing whitespace stripped at a record boundary. |
+| `whitespace_collapse_internal` | Runs of three or more blank lines collapsed to two. |
+| `line_trim` | Each line individually stripped of leading / trailing whitespace. |
+| `line_join_spaces` | Adjacent lines joined with single spaces, newlines discarded. |
+| `blank_line_drop` | Empty lines between non-empty lines dropped. |
+| `strip_tool_chrome` | System tags, hook output, tool UI chrome removed (see `normalize.strip_noise`). |
+| `tool_result_truncate` | Tool output heads/tails kept; middle replaced with a marker string. |
+| `spellcheck_user` | User turns rewritten by spellcheck. |
+| `synthesized_marker` | Adapter inserts its own strings (e.g., `[N lines omitted]`, `[registry] …`, Slack provenance footer). |
+| `speaker_role_assignment` | Multi-party speakers alternately assigned `user` / `assistant` roles (Slack). |
+| `tool_result_omitted` | Some tool outputs fully omitted from transcript (e.g., Read/Edit/Write results in `normalize._format_tool_result`). |
+
+Adapters MAY define their own transformation names for behaviors the reserved list does not cover. Third-party names SHOULD be prefixed with the adapter name to avoid collisions (e.g., `cursor.composer_ordering`).
+
+**Capability derivation:**
+- `byte_preserving` — declared_transformations is empty AND output bytes equal input bytes for any source the adapter can read. Advertised via the `byte_preserving` capability (§2.1). MUST be verified by §7.2 round-trip test.
+- `declared_lossy` — declared_transformations is non-empty. The adapter's output is reproducible from source by applying *only* the declared transformations. MUST be verified by §7.3 declared-transformation test.
+
+**Existing code mapping (for the cleanup PR):**
+
+| Module | Declared transformations |
+|---|---|
+| `filesystem` (current `miner.py`) | `utf8_replace_invalid`, `whitespace_trim` |
+| `conversations` (current `convo_miner.py` + `normalize.py`) | `utf8_replace_invalid`, `newline_normalize`, `line_trim`, `line_join_spaces`, `blank_line_drop`, `whitespace_collapse_internal`, `strip_tool_chrome`, `tool_result_truncate`, `tool_result_omitted`, `spellcheck_user`, `synthesized_marker`, `speaker_role_assignment` |
+
+The filesystem adapter is nearly byte-preserving today; the conversations adapter is extensively transformed. Both are honest after this spec lands because both are fully declared.
+
+This replaces the MISSION.md promise of "verbatim always" with a stronger one: every adapter publishes what it does to your data, and the conformance suite verifies it hasn't lied. "Verbatim" becomes a capability some adapters hold (byte_preserving), not a global claim about a lossy pipeline.
+
+### 1.5 Three ingest modes
+
+A single adapter declares one or more of three modes via a class attribute:
+
+```python
+class BaseSourceAdapter(ABC):
+    supported_modes: ClassVar[frozenset[Literal["chunked_content", "whole_record", "metadata_only"]]]
+```
+
+| Mode | Content origin |
+|---|---|
+| `chunked_content` | Source bytes, split into chunks the adapter chooses (current filesystem behavior). |
+| `whole_record` | Source bytes, one drawer per source item (e.g., PR → 1 drawer). |
+| `metadata_only` | Synthesized description of a source item (absorbs #981). The description bytes are authored by the user or adapter, not the source. Declared transformations (§1.4) do not apply — content is not derived from source bytes. |
+
+`metadata_only` resolves #981: description-mode matches a path pattern and produces one drawer whose content is the user-authored description rather than the file contents. Conformance tests (§7.2, §7.3) skip `metadata_only` records.
+
+An adapter MAY support multiple modes and select per-item; the per-item mode is recorded in `metadata["ingest_mode"]` (§5.1). This field already exists on conversation drawers (`convo_miner.py:346`) and is the only existing field whose semantics this spec extends rather than preserves.
+
+### 1.6 Chunking delegation
+
+Core does not impose chunking. `miner.py`'s 800-character sliding window is the filesystem adapter's default for unknown file types — not a contract. Adapter authors choose what makes sense:
+
+- Code files → tree-sitter function/class boundaries (future enhancement to the filesystem adapter).
+- Chat transcripts → exchange pairs (current `convo_miner.py` behavior).
+- PRs → whole-record (current `git-mine` behavior in #567).
+- PDFs → page or section.
+- Voice transcripts → speaker turn.
+
+The sole cross-adapter requirement for `chunked_content` mode: chunks for a given `source_file`, re-assembled in `chunk_index` order and accounting for declared transformations in §1.4, reproduce the adapter's internal representation of the source. The conformance suite verifies this.
+
+### 1.7 Closet integration
+
+Closets are the AAAK-compressed index layer (`palace.build_closet_lines`, `upsert_closet_lines`) that points to drawer content and enables LLM-scale scanning without reading every drawer. Closet-building is not an adapter concern:
+
+- **Core builds closets** from adapter-yielded drawers as a post-step, via the existing `palace.py` helpers. Adapters do not call these APIs.
+- **Adapters MAY emit closet hints** in drawer metadata via a flat `;`-joined string:
+  ```python
+  metadata["closet_hints"] = "decided GraphQL; migrated to Postgres; fixed PR-567"
+  ```
+  Core splits on `;` and feeds these as candidate topics alongside the content-scanned ones in `build_closet_lines`. The git adapter can hint decision-signal quotes that raw content-scanning would miss; the conversations adapter can hint section headers; the filesystem adapter has no need and omits the field.
+- **metadata_only drawers get closets too.** Core builds them from the synthesized description content the same way it builds closets for any other drawer. This is how #981's path-level descriptions become searchable.
+- **Closet purging** remains keyed on `source_file` (`purge_file_closets` in `palace.py:221`). Adapters' source_file values must be stable so purge is correct on re-ingest.
+
+Current `convo_miner.py` does not build closets for conversation drawers — an existing gap. The cleanup PR (§9) routes the conversations adapter through the same post-step closet builder as filesystem, closing the gap as a side effect.
+
+---
+
+## 2. Adapter contract
+
+### 2.1 Identity and capabilities
+
+```python
+class BaseSourceAdapter(ABC):
+    name: ClassVar[str]                    # "filesystem", "cursor", "git", "slack", ...
+    spec_version: ClassVar[str] = "1.0"
+    adapter_version: ClassVar[str]         # Independent of spec_version; recorded on every drawer.
+    capabilities: ClassVar[frozenset[str]]
+    supported_modes: ClassVar[frozenset[str]]             # Per §1.5.
+    declared_transformations: ClassVar[frozenset[str]]    # Per §1.4.
+    default_privacy_class: ClassVar[str]                  # Per §6.
+```
+
+Defined capability tokens (v1):
+
+| Token | Meaning |
+|---|---|
+| `byte_preserving` | `declared_transformations` is empty AND extracted content equals source bytes. |
+| `supports_incremental` | Implements `is_current()` meaningfully; `ingest()` respects `ctx.skip_current_item()`. |
+| `supports_structured_metadata` | Attaches fields beyond §5.1 universals. |
+| `supports_entity_hints` | Emits entity hints via `metadata["entity_hints_json"]` (§5.4). |
+| `supports_kg_triples` | Writes knowledge-graph triples directly to the SQLite KG (§5.5). |
+| `supports_closet_hints` | Emits `metadata["closet_hints"]` (§1.7). |
+| `requires_auth` | Needs credentials at runtime (env vars — §4.2). |
+| `requires_external_service` | Needs a running service (Slack API, email server). |
+| `requires_local_tool` | Needs a local binary (`gh`, `rg`, `whisper`). |
+| `adapter_owns_routing` | Returns authoritative `RouteHint` values from `ingest()` that core uses as-is (§G3 / §2.5). |
+| `respects_privacy_class` | Honors §6 privacy-class filtering. |
+
+Capability tokens are free-form strings; third-party adapters MAY declare novel tokens for their ecosystem. Core only inspects the above.
+
+### 2.2 Source references
+
+See `SourceRef` in §1.3. The shape is deliberately open — adapters parse `uri` and `options` as they see fit. Core does not canonicalize URIs.
+
+**Secrets in `SourceRef.options`:** credentials MUST NOT be placed in `options`. The spec reserves `options` for non-secret values (paths, filters, date ranges). Secrets come from env vars per §4.2. An adapter that reads a credential from `options` violates the spec and MUST be rejected by the conformance suite.
+
+### 2.3 Lifecycle
+
+1. `__init__`: lightweight. No I/O, no network, no credential fetch.
+2. First call to `ingest`: may open resources. All I/O is lazy.
+3. `close()`: releases all resources. After `close()`, further calls MUST raise `AdapterClosedError`.
+
+### 2.4 Concurrency
+
+An adapter instance is long-lived and serves many mine operations. Adapters MUST be thread-safe for concurrent `ingest` calls across different `SourceRef` values. MemPalace core serializes calls within a single `SourceRef` unless an adapter advertises `supports_parallel_ingest` (not in v1 — reserved for v1.1).
+
+### 2.5 Routing
+
+Routing is the adapter's responsibility. The filesystem adapter reads `mempalace.yaml` (hall keywords, rooms list) via `MempalaceConfig()` and returns `RouteHint(wing=..., room=..., hall=...)` on each drawer. This relocates `detect_room()` and `detect_hall()` (currently in `miner.py` and `convo_miner.py`) into their respective adapters.
+
+Order of precedence for routing:
+1. Explicit `--wing` / `--room` CLI flags → passed through `SourceRef.options` → adapter honors verbatim.
+2. Palace config match (`mempalace.yaml` hall keywords, room keywords) → adapter computes.
+3. Adapter-internal fallback (e.g., filesystem adapter falls back to `"general"` room).
+
+Adapters advertising `adapter_owns_routing` return the final answer; core uses it verbatim. Adapters not advertising it return None and core applies a generic fallback router (writing to wing `default`, room `general`, hall `general`). Absent any adapter, this is how `mempalace mine` behaves today.
+
+### 2.6 Incremental ingest
+
+`is_current()` is the incremental-ingest primitive. The palace itself is the cursor — no separate persisted state. Correctness requirements:
+
+- The adapter's `SourceItemMetadata.source_file` MUST be stable across re-ingests of the same logical item. Filesystem adapter uses the absolute path (as today). Git adapter uses a URI shape like `github.com/org/repo#pr=567` or `github.com/org/repo#commit=abc123`.
+- `is_current()` returns True when the stored metadata matches the adapter's current version token. The default implementation returns False (always re-extract) — adapters advertising `supports_incremental` override.
+- Deletion tombstones: an adapter MAY yield a `SourceItemMetadata(source_file=..., version="__deleted__")` entry — core purges drawers with matching `source_file` and builds no new drawers for that item. Advertised via `supports_deletion_tombstones`.
+- Adapters without `supports_incremental` ignore `is_current()` and fully re-extract. Core logs a warning.
+
+### 2.7 Errors
+
+- `SourceNotFoundError` — the `SourceRef` does not resolve.
+- `AuthRequiredError` — adapter needs credentials; raises with a message describing which env vars to set.
+- `AdapterClosedError` — method called after `close()`.
+- `TransformationViolationError` — conformance suite raises this when the content round-trip requires an undeclared transformation.
+- `SchemaConformanceError` — a `DrawerRecord.metadata` is missing required fields declared in `describe_schema()` or violates declared types.
+
+---
+
+## 3. Registration and discovery
+
+### 3.1 Entry points (primary mechanism)
+
+Third-party adapters ship as installable packages:
+
+```toml
+# pyproject.toml of mempalace-source-cursor
+[project.entry-points."mempalace.sources"]
+cursor = "mempalace_source_cursor:CursorAdapter"
+```
+
+MemPalace discovers adapters at process start via `importlib.metadata.entry_points(group="mempalace.sources")`.
+
+### 3.2 In-tree registry (secondary)
+
+```python
+from mempalace.sources.registry import register
+
+register("my-experimental-adapter", MyAdapter)
+```
+
+Entry-point discovery and explicit `register()` populate the same registry. Explicit registration wins on name conflict.
+
+### 3.3 Selection (explicit only — no auto-detect)
+
+Unlike storage backends (RFC 001 §3.3), source adapters are never auto-detected. The user selects the adapter explicitly:
+
+```bash
+mempalace mine --source cursor ~/                      # explicit adapter
+mempalace mine --source git /path/to/repo              # explicit adapter
+mempalace mine --source filesystem /path/to/project    # explicit adapter
+mempalace mine /path/to/project                        # implicit: filesystem (default)
+```
+
+The default when no `--source` is given is `filesystem`, preserving current `mempalace mine <path>` behavior.
+
+**Backwards compatibility with `--mode`.** Current `cli.py:517-519` exposes `--mode {projects,convos}`. This spec maps:
+- `--mode projects` → `--source filesystem` (the new default)
+- `--mode convos` → `--source conversations`
+
+`--mode` stays as a deprecated alias through v4.x with a deprecation warning on use; removed in v5.0.
+
+Auto-detection would be hostile — a directory containing a `.git` folder, a `workspaceStorage/` subdir, and an `mbox` file is not a signal of user intent.
+
+---
+
+## 4. Configuration
+
+### 4.1 Shape
+
+```json
+{
+  "sources": {
+    "my-cursor": {
+      "type": "cursor",
+      "workspace_storage": "~/Library/Application Support/Cursor/User/workspaceStorage"
+    },
+    "my-git": {
+      "type": "git",
+      "repos": ["/projects/mempalace", "/projects/site"]
+    }
+  },
+  "palaces": {
+    "work": {
+      "sources": ["my-git"],
+      "privacy_floor": "internal"
+    },
+    "personal": {
+      "sources": ["my-cursor"]
+    }
+  }
+}
+```
+
+Single-user local mode: config is optional. `mempalace mine <path>` with no config uses the `filesystem` adapter and defaults.
+
+### 4.2 Environment variables
+
+- `MEMPALACE_SOURCE_<NAME>_*` — per-adapter secrets and connection info. Examples: `MEMPALACE_SOURCE_SLACK_TOKEN`, `MEMPALACE_SOURCE_NOTION_API_KEY`, `MEMPALACE_SOURCE_GIT_GITHUB_TOKEN`.
+- Secrets MUST be readable from env vars; config files carry structure, env vars carry credentials. Same rule as RFC 001 §4.2.
+
+### 4.3 Adapter-specific options
+
+`SourceRef.options` is a free-form dict of non-secret values (§2.2). Each adapter documents its accepted keys. Unknown keys MUST be ignored (forward compatibility); the adapter MAY log a warning.
+
+---
+
+## 5. Metadata schema contract
+
+### 5.1 Universal fields
+
+Existing drawer metadata fields are preserved — the spec adds the following:
+
+| New field | Type | Added by | Purpose |
+|---|---|---|---|
+| `adapter_name` | `str` | core, from `BaseSourceAdapter.name` | Which registered source produced this drawer. |
+| `adapter_version` | `str` | adapter | Adapter's own version (distinct from palace `normalize_version`). Enables re-extract workflows targeted at drawers from a known-buggy adapter version. |
+| `privacy_class` | `str` | adapter default, config override | Per §6. |
+
+Existing fields retain their current semantics (verified against `miner.py:542-561` and `convo_miner.py:338-350`):
+
+| Existing field | Role in the spec |
+|---|---|
+| `source_file` | Functions as the adapter's source-item identifier. Adapter defines the shape — a filesystem path for filesystem, a URI like `github.com/org/repo#pr=123` for git. MUST be stable across re-ingests of the same logical item. |
+| `source_mtime` | Functions as the source-item version for filesystem. Adapters without mtime semantics MAY omit this field and use a different version discriminator (e.g., commit SHA in a separate `metadata["commit_sha"]` field); the spec only requires that `is_current()` can decide staleness from the stored metadata. |
+| `filed_at` | When the record was written. ISO-8601 string. |
+| `added_by` | Agent name (e.g., `lumi`, `claude-code`). Orthogonal to `adapter_name` — the agent is *who* triggered mining; the adapter is *how* data was extracted. |
+| `wing`, `room`, `hall` | Palace routing. Populated by adapter per §2.5. |
+| `chunk_index` | Per §1.6. Always 0 for `whole_record` / `metadata_only`. |
+| `normalize_version` | Palace-wide schema version (currently `palace.py:50`). Unchanged. Separate from `adapter_version`. |
+| `entities` | Semicolon-joined candidate entity names. Already flat; kept flat (§5.4 replacement). |
+| `ingest_mode` | Per §1.5. Already on conversation drawers; added to filesystem drawers by the cleanup PR. |
+| `extract_mode` | Conversation-adapter-specific (`exchange` vs `general`). Moves into the conversations adapter's declared schema per §5.2. |
+
+**Nothing is renamed. Nothing is removed.** The spec formalizes the shape ingesters already converge on. Existing `where={"source_file": ...}` queries in `searcher.py`, `palace.py`, and callers keep working.
+
+**Chroma metadata constraint:** all metadata values MUST be `str | int | float | bool`. No lists, no nested dicts. This matches RFC 001 §1.4 and the underlying ChromaDB contract. Structured side-data goes to the SQLite knowledge graph (§5.5) or to a declared flat JSON-encoded string field (§5.4).
+
+### 5.2 Adapter schemas
+
+Each adapter returns an `AdapterSchema` from `describe_schema()`:
+
+```python
+@dataclass(frozen=True)
+class AdapterSchema:
+    fields: dict[str, FieldSpec]   # Keyed by metadata key.
+    version: str
+
+@dataclass(frozen=True)
+class FieldSpec:
+    type: Literal["string", "int", "float", "bool", "delimiter_joined_string", "json_string"]
+    required: bool
+    description: str
+    indexed: bool = False           # Hint to backends that can build indexes (RFC 001 §2.1).
+    # delimiter_joined_string: the delimiter character (default ";").
+    delimiter: str = ";"
+    # json_string: the JSON schema of the encoded object (informational only).
+    json_schema: dict | None = None
+```
+
+`delimiter_joined_string` covers the `entities` shape (current `;`-joined list of names). `json_string` is the escape hatch for adapters needing to pack nested data — the value stored is still a single flat `str` from Chroma's perspective, but the adapter is allowed to document its parsed shape.
+
+Example for a hypothetical `slack` adapter:
+
+```python
+AdapterSchema(
+    version="1.0",
+    fields={
+        "channel_name": FieldSpec(type="string", required=True, description="Slack channel name", indexed=True),
+        "channel_id": FieldSpec(type="string", required=True, description="Slack channel ID"),
+        "thread_ts": FieldSpec(type="string", required=False, description="Thread root timestamp"),
+        "author_id": FieldSpec(type="string", required=True, description="Slack user ID", indexed=True),
+        "author_name": FieldSpec(type="string", required=True, description="Display name at extraction time"),
+        "reactions": FieldSpec(type="delimiter_joined_string", required=False, description="Emoji shortcodes"),
+    },
+)
+```
+
+### 5.3 Enterprise keying
+
+The adapter schema is the stable surface enterprises filter on. A support team querying the palace for `channel_id = "C01234"` does not care about ChromaDB's internal representation. The schema field is declared by the adapter, indexed by the backend (RFC 001 §2.1 `supports_metadata_filters`), and exposed through the existing `where=` clause.
+
+This is how "structured data" serves company use cases without breaking transformation guarantees: declared-transformation content in the drawer, structured fields in the metadata, schema declared by the adapter, filtering done by the backend.
+
+### 5.4 Entity hints (optional)
+
+Adapters with `supports_entity_hints` MAY include:
+
+```python
+metadata["entity_hints_json"] = '[{"type":"person","name":"Milla Jovovich","confidence":0.95,"offset":120},{"type":"project","name":"MemPalace","confidence":1.0,"offset":0}]'
+```
+
+The value is a JSON-encoded string (type `json_string` in the adapter schema). Core parses on read and feeds into `mempalace/entity_detector.py` as a prior: hints with `confidence >= 0.9` bypass the heuristic detector; lower-confidence hints feed into it as candidates.
+
+This is additive to the existing flat `entities` field — entity_hints carries structure (type, confidence, offset); `entities` remains the Chroma-indexable flat string. An adapter that produces entity_hints MUST also populate `entities` as the flat name-only projection, so existing filter queries keep working.
+
+### 5.5 Knowledge-graph triples (optional)
+
+Adapters with `supports_kg_triples` write directly to the SQLite knowledge graph via `mempalace/knowledge_graph.py` — **not** to drawer metadata. Chroma cannot store structured triples; the KG already exists for this purpose.
+
+The adapter calls the existing `KnowledgeGraph.add_triple()` (signature verified against `mempalace/knowledge_graph.py:130`):
+
+```python
+palace.kg.add_triple(
+    subject="Ben",
+    predicate="committed",
+    obj="PR-567",                    # `object` is a Python builtin — the API uses `obj`.
+    valid_from="2026-03-12",
+    confidence=1.0,
+    source_file=drawer.source_file,  # Existing provenance parameter.
+)
+```
+
+Drawer metadata includes a flat counter — `metadata["kg_triples_count"]: int` — so search consumers can see at a glance that KG side-data exists for a drawer without hitting SQLite.
+
+The existing API has `source_closet` and `source_file` provenance parameters but no `source_drawer_id` or `adapter_name`. The cleanup PR (§9) should add these two optional parameters to `add_triple()` so adapter-written triples can be traced back to (a) the specific drawer that produced them and (b) the adapter that authored them — necessary for re-extraction workflows. Until that lands, adapters use `source_file` as the provenance key and record adapter authorship via a separate table or a predicate naming convention (e.g., `adapter:git:committed`).
+
+This aligns with the existing architecture in `CLAUDE.md` ("Knowledge Graph: ENTITY → PREDICATE → ENTITY with valid_from / valid_to dates") — the RFC formalizes the adapter-side write path.
+
+### 5.6 Source encoding and newline
+
+Current ingesters handle encoding lossily (`errors="replace"` in `miner.py:595` and `normalize.py:124`) and do not record original encoding. The spec does **not** require per-drawer `source_encoding` / `source_newline` — most runs are uniform UTF-8 / LF, and storing the same value on every drawer wastes bytes.
+
+Instead: adapters that handle non-UTF-8 or non-LF sources record the values once on the adapter's `SourceSummary` and per-drawer only when a specific drawer diverges from the adapter default. The `utf8_replace_invalid` declared transformation (§1.4) already communicates that lossy decoding happened; specific drawer-level provenance is opt-in.
+
+---
+
+## 6. Privacy class
+
+### 6.1 Defined levels
+
+| Level | Meaning | Example sources |
+|---|---|---|
+| `public` | Content intended for public consumption. | arXiv papers, public GitHub repos, published blogs. |
+| `internal` | Organizational content, not for public disclosure. | Corporate Slack, internal Notion, private git repos. |
+| `pii_potential` | May contain personally identifiable information. | Email, iMessage, Claude/ChatGPT transcripts. |
+| `sensitive` | Known to contain PII, financial, or health data. | Medical records, financial statements, legal filings. |
+| `secrets_possible` | May contain credentials or secrets. | Git history, environment dumps, CI logs. |
+
+An adapter declares a default on `BaseSourceAdapter.default_privacy_class`. Users MAY override per-source in config.
+
+### 6.2 Enforcement
+
+- Each palace declares a `privacy_floor`. Drawers above the floor (equal to or laxer) are admitted; drawers below are rejected at write time and surfaced in a `rejected` list on the CLI and MCP tool.
+- **Default floor: none** — v1 accepts all levels unless the palace explicitly configures a floor. This keeps the single-user local default low-friction (users who run `mempalace mine` on a git repo expect `secrets_possible` drawers to land). Enterprise deployments MUST set a floor; docs for regulated-domain setup will recommend starting strict and relaxing as needed.
+- Search results surface `privacy_class` in result metadata. MCP tool wrappers MAY redact results above a caller-declared ceiling.
+- `secrets_possible` drawers SHOULD pass through a secrets-scan pre-index hook when one is available. PR #389 (sensitive content scanner) is the expected enforcement mechanism for v1; until it lands, `secrets_possible` is a label without automated scanning. The label is still useful — it enables floor-based rejection and alerts downstream consumers.
+- The privacy class is recorded in drawer metadata and cannot be downgraded without a migration log entry, matching RFC 001's embedder-identity pattern.
+
+Privacy class is how a regulated-domain deployment (medical, legal, financial) can use MemPalace safely. Without it, flexible ingest becomes a liability; with it, ingest is scoped by policy.
+
+---
+
+## 7. Testing contract
+
+### 7.1 The abstract suite
+
+MemPalace ships `mempalace.sources.testing.AbstractSourceAdapterContractSuite` — a pytest mixin. Every adapter package ships a concrete subclass:
+
+```python
+from mempalace.sources.testing import AbstractSourceAdapterContractSuite
+
+class TestCursorAdapter(AbstractSourceAdapterContractSuite):
+    @pytest.fixture
+    def adapter(self):
+        return CursorAdapter()
+
+    @pytest.fixture
+    def fixture_source(self, tmp_path):
+        """Build a minimal Cursor workspaceStorage fixture."""
+        ...
+        return SourceRef(local_path=str(tmp_path))
+
+    @pytest.fixture
+    def canonical_source_bytes(self, fixture_source):
+        """Return a mapping of source_file -> authoritative bytes.
+
+        For filesystem sources: the file's raw bytes.
+        For SQLite sources: the extracted value column bytes for each row.
+        For API sources: the canonical HTTP response body bytes.
+
+        Adapter-defined — the adapter knows what its 'source bytes' are.
+        """
+        ...
+```
+
+The suite covers:
+
+- `ingest` yields items with stable `source_file` and well-formed `version`.
+- `is_current()` returns True when metadata matches, False when it differs.
+- `close()` releases resources; subsequent calls raise `AdapterClosedError`.
+- Unicode content and unicode identifiers are preserved end-to-end.
+- Large-source handling: 10k+ items ingest without loading all into memory.
+- Error paths: `SourceNotFoundError`, `AuthRequiredError` raise with correct types.
+- `SourceRef.options` MUST NOT contain secrets — the adapter raises if it detects a value matching a common-secret pattern (GitHub token prefix, Slack token prefix, etc.). Advisory test, not blocking.
+
+### 7.2 Byte-preserving round-trip (for `byte_preserving` adapters only)
+
+Required for adapters advertising `byte_preserving`:
+
+```python
+def test_byte_preserving_round_trip(self, adapter, fixture_source, canonical_source_bytes):
+    """Concatenated chunks must equal the canonical source bytes.
+
+    For each source_file in the fixture:
+      1. Read canonical_source_bytes[source_file].
+      2. Collect all DrawerRecords for that source_file from adapter.ingest(...).
+         Skip metadata_only drawers (§1.5).
+      3. Sort by chunk_index.
+      4. Concatenate record.content values.
+      5. Assert equality with the canonical bytes (UTF-8 decoded).
+    """
+```
+
+Failure raises `TransformationViolationError`.
+
+### 7.3 Declared-transformation round-trip (for `declared_lossy` adapters)
+
+Required for adapters with non-empty `declared_transformations`:
+
+```python
+def test_declared_transformation_round_trip(self, adapter, fixture_source, canonical_source_bytes):
+    """Adapter output must be reproducible by applying ONLY declared transformations.
+
+    1. For each source_file, read canonical_source_bytes.
+    2. Apply each declared transformation in declared_transformations to the bytes,
+       in the order declared by the adapter, using the reference implementations
+       in mempalace.sources.transforms.
+    3. Compare the result to the concatenated record.content values.
+    4. If they differ, the adapter has applied a transformation it did not declare.
+       Raise TransformationViolationError.
+    """
+```
+
+For transformations not in the reserved list (§1.4) — adapter-custom names — the adapter MUST provide a reference implementation callable under `mempalace.sources.transforms.<adapter_name>_<transform_name>`. The conformance suite imports and applies it. Undiscoverable custom transforms fail the test.
+
+### 7.4 Schema conformance
+
+A generator-based property test validates that every record yielded by `ingest` across the fixture source has metadata matching `describe_schema()`. Missing required fields, wrong types, or (in strict mode) undeclared fields fail the test.
+
+### 7.5 Note on current corpus
+
+No existing test in `tests/` asserts byte-preservation or declared-transformation correctness (verified via grep of `tests/` for `verbatim|byte.?preserv|round.?trip`). This RFC's conformance suite introduces the first such coverage. The existing MISSION.md claim of "verbatim always" is a social contract until this lands; afterward it becomes a machine-verified property of adapters that declare `byte_preserving`.
+
+---
+
+## 8. Versioning and compatibility
+
+- `BaseSourceAdapter.spec_version` declares which spec version an adapter implements.
+- MemPalace refuses to load an adapter declaring a different major spec version.
+- Minor spec versions are additive: new optional methods, new capability tokens, new reserved transformation names, new universal metadata fields with sensible defaults.
+- Adapters MAY declare their own `adapter_version` independent of the spec version; this is recorded on every drawer (§5.1) and enables "this drawer was extracted by cursor-adapter 0.3; 0.4 fixed a parsing bug; re-extract affected drawers" workflows.
+- This is spec v1.0.
+
+---
+
+## 9. Cleanup prerequisite (not in this spec, but gating)
+
+The existing in-tree ingesters are not adapter-shaped. Before RFC 002 can be enforced, the following refactor lands in a separate PR:
+
+- Introduce `mempalace/sources/base.py` defining `BaseSourceAdapter`, the typed records, and the registry.
+- Introduce `mempalace/sources/transforms.py` with reference implementations of every reserved transformation in §1.4. Adapters and the conformance suite both consume these.
+- `mempalace/miner.py` → `mempalace/sources/filesystem.py` implementing `BaseSourceAdapter`. Current behavior preserved: 800-char chunking becomes the adapter's default; `READABLE_EXTENSIONS` moves to the adapter; `detect_room()` and `detect_hall()` move to the adapter per §2.5. `declared_transformations = frozenset({"utf8_replace_invalid", "whitespace_trim"})`.
+- `mempalace/convo_miner.py` → `mempalace/sources/conversations.py`. Exchange-pair chunking stays. The format-detection logic in `normalize.py` becomes per-format plugins the conversations adapter composes (one for Claude Code JSONL, one for Codex JSONL, one for ChatGPT mapping trees, one for Claude.ai JSON, one for Slack JSON) — each small and independently testable, eliminating the `if source_type` chain. `declared_transformations` enumerates every transformation `normalize.py` and `convo_miner._chunk_by_exchange` actually perform (see §1.4 "Existing code mapping").
+- Closet-building wired into the conversations adapter's post-step (currently missing, per §1.7) — side effect of routing through the unified core post-step.
+- `mempalace/cli.py` subcommand `mine` routes through the `mempalace.sources` registry. `--mode {projects,convos}` becomes a deprecated alias for `--source {filesystem,conversations}`.
+- `mempalace/mcp_server.py` `mempalace_mine` tool accepts a `source` parameter.
+- `mempalace/palace.py` exposes `PalaceContext` — a per-mine-invocation facade that bundles the drawer collection, closet collection, knowledge graph, palace config, and progress hooks. Adapters receive this; they do not import `palace.py` directly.
+- `NORMALIZE_VERSION` (currently a module-level constant in `palace.py:50`) stays. It is the palace-wide schema version, orthogonal to per-adapter `adapter_version`.
+- `KnowledgeGraph.add_triple()` (`knowledge_graph.py:130`) gains two optional parameters: `source_drawer_id: str = None` and `adapter_name: str = None`. Existing callers are unaffected; adapters advertising `supports_kg_triples` (§5.5) populate both. Backwards-compatible change.
+
+This cleanup is substantial — comparable to RFC 001 §10's chroma-import removal — and should land before any new third-party adapter PR merges. Each new adapter is easier after the cleanup, not harder.
+
+---
+
+## 10. Impact on in-flight PRs
+
+| PR / Issue | Effort to align |
+|---|---|
+| [#274](https://github.com/MemPalace/mempalace/issues/274) Cursor SQLite | Becomes `mempalace-source-cursor` third-party package. Author has a working prototype on Windows; needs `describe_schema()`, `declared_transformations`, and the conformance suite. Prior #287 (closed unmerged) is predecessor work. |
+| [#23](https://github.com/MemPalace/mempalace/pull/23) OpenCode SQLite | Becomes `mempalace-source-opencode`. Same shape as Cursor. |
+| [#169](https://github.com/MemPalace/mempalace/pull/169) Pi agent | Becomes `mempalace-source-pi` or a format plugin under the conversations adapter (depending on format similarity). |
+| [#232](https://github.com/MemPalace/mempalace/pull/232) Cursor JSONL | Deprecated in favor of #274's SQLite path; or a second mode of `mempalace-source-cursor`. |
+| [#567](https://github.com/MemPalace/mempalace/pull/567), [#98](https://github.com/MemPalace/mempalace/pull/98) git-mine | Closest existing work to what the spec envisions. Becomes first-party `mempalace/sources/git.py`. Exercises `whole_record` mode, `supports_structured_metadata`, `supports_closet_hints` (decision-signal quotes), `supports_kg_triples` (commit authorship, PR review relationships). |
+| [#591](https://github.com/MemPalace/mempalace/pull/591), [#592](https://github.com/MemPalace/mempalace/pull/592) Delphi Oracle | Deferred. The live-stream pattern is out of scope for v1 (§Non-goals). A v1.1 addition will specify webhook/stream adapters. |
+| [#702](https://github.com/MemPalace/mempalace/pull/702) Cursor + factory.ai | Splits into two adapter packages. |
+| [#981](https://github.com/MemPalace/mempalace/issues/981) path-level descriptions | Absorbed by §1.5 `metadata_only` mode + §5.1 `ingest_mode`. A new first-party `descriptions` adapter or a second mode on `filesystem`. |
+| [#244](https://github.com/MemPalace/mempalace/pull/244) Cursor memory-first MCP workflow docs | Points at `mempalace-source-cursor` once the adapter lands. |
+| [#419](https://github.com/MemPalace/mempalace/pull/419), [#300](https://github.com/MemPalace/mempalace/pull/300), [#952](https://github.com/MemPalace/mempalace/pull/952) language-extension additions to `READABLE_EXTENSIONS` | Becomes per-language config on the filesystem adapter. Contributors can publish domain-specific adapters without touching core. |
+| [#389](https://github.com/MemPalace/mempalace/pull/389) sensitive content scanner | Expected enforcement mechanism for the `secrets_possible` privacy class (§6.2). Not a blocker for this spec, but a natural consumer. |
+| [#434](https://github.com/MemPalace/mempalace/pull/434) auto-populate KG from drawers | Complementary: post-hoc derivation of KG triples from drawer content. Adapters with `supports_kg_triples` provide the up-front path; #434 handles everything else. |
+
+---
+
+## 11. Open questions
+
+1. **Cross-adapter dedup.** When a PR body is mined via `git` AND shows up as a conversation quote mined via `claude-code`, both drawers land. Is query-time dedup in `searcher.py` sufficient, or should core maintain a content-hash index across adapters? Declared non-goal in v1 but worth revisiting if user feedback demands it.
+2. **Live-stream pattern.** Delphi Oracle (#591/592) and potentially Slack/Discord real-time ingestion need a push-mode contract. This is a v1.1 addition (streaming adapter trait + webhook surface), not blocking.
+3. **LLM-assisted structured extraction.** Some adapters will want to call an LLM to extract structured fields. The spec does not standardize this — should it? Argument for: conformance test for LLM-driven fields, consistent caching. Argument against: local-first / zero-API is a core promise; LLM dependencies are opt-in per adapter.
+4. **Adapter-vs-format split for conversations.** §9 proposes format plugins composed under a single conversations adapter. Alternative: one adapter per format (claude-code, chatgpt, codex, cursor-jsonl, slack). The trade-off is discoverability (one adapter is easier to find) vs. encapsulation (format plugins are simpler to test). Preference leans toward the single-adapter + plugin model; open to counter-argument.
+5. **Default `privacy_floor`.** v1 defaults to none (§6.2) so single-user local mining is frictionless. An argument exists for defaulting to `pii_potential` — forces regulated-domain users to opt in to sensitive levels rather than opt out. Open to changing the default before v1 ships.
+6. **`canonical_source_bytes` for API-backed adapters.** §7.1 defines this as adapter-declared. For API-backed adapters (Slack, Notion), what constitutes "canonical bytes" in a conformance test — the fixture's captured HTTP response? A serialized representation of the parsed object? Leaves to the adapter; may need a follow-up spec for common conventions.
+7. **`adapter_version` bump semantics.** When does an adapter bump `adapter_version`? On any behavior change? On declared-transformation changes only? Suggests a follow-up doc on adapter SemVer conventions for the community to agree on.
+
+---
+
+## 12. Rollout
+
+1. Land the cleanup PR (§9): introduce `mempalace/sources/`, refactor `miner.py` → filesystem adapter, `convo_miner.py` → conversations adapter, route CLI and MCP through the sources registry. Behavior preserved end-to-end. Closets get built for conversation drawers as a side effect.
+2. Land this spec as-is. Add `AbstractSourceAdapterContractSuite`, entry-point discovery, `AdapterSchema` validation, privacy-class enforcement (floor-gated writes), declared-transformation reference implementations in `mempalace/sources/transforms.py`.
+3. Land `mempalace/sources/git.py` as the first-party adapter absorbing #567. Exercises `whole_record`, `supports_structured_metadata`, `supports_closet_hints`, `supports_kg_triples` together.
+4. Encourage the Cursor (#274), OpenCode (#23), and Pi (#169) authors to publish as third-party packages under `mempalace-source-*`. Offer review help against the spec.
+5. Publish adapter-authoring docs at [mempalaceofficial.com/guide/authoring-sources](https://mempalaceofficial.com/guide/authoring-sources.html).
+6. Update [ROADMAP.md](../../ROADMAP.md) with spec v1.0 adoption under v4.0.0-alpha.

mempalace/__init__.py

@@ -4,10 +4,10 @@ import logging
 
 from .version import __version__  # noqa: E402
 
-# ChromaDB 0.6.x ships a Posthog telemetry client whose capture() signature is
-# incompatible with the bundled posthog library, producing noisy stderr warnings
-# on every client operation ("Failed to send telemetry event … capture() takes
-# 1 positional argument but 3 were given").  Silence just that logger.
+# chromadb telemetry: posthog capture() was broken in 0.6.x causing noisy stderr
+# warnings ("capture() takes 1 positional argument but 3 were given"). In 1.x the
+# posthog client is a no-op stub, so this is now harmless — kept as a guard in
+# case future chromadb versions re-introduce real telemetry calls.
 logging.getLogger("chromadb.telemetry.product.posthog").setLevel(logging.CRITICAL)
 
 # NOTE: the previous block set ``ORT_DISABLE_COREML=1`` on macOS arm64 as a

mempalace/backends/__init__.py

@@ -1,6 +1,64 @@
-"""Storage backend implementations for MemPalace."""
+"""Storage backend implementations for MemPalace (RFC 001).
 
-from .base import BaseCollection
+Public surface:
+
+* :class:`BaseCollection` — per-collection read/write contract.
+* :class:`BaseBackend` — per-palace factory contract.
+* :class:`PalaceRef` — value object identifying a palace for a backend.
+* :class:`QueryResult` / :class:`GetResult` — typed read returns.
+* Error classes: :class:`PalaceNotFoundError`, :class:`BackendClosedError`,
+  :class:`UnsupportedFilterError`, :class:`DimensionMismatchError`,
+  :class:`EmbedderIdentityMismatchError`.
+* Registry: :func:`get_backend`, :func:`register`, :func:`available_backends`,
+  :func:`resolve_backend_for_palace`.
+* In-tree Chroma default: :class:`ChromaBackend`, :class:`ChromaCollection`.
+"""
+
+from .base import (
+    BackendClosedError,
+    BackendError,
+    BaseBackend,
+    BaseCollection,
+    DimensionMismatchError,
+    EmbedderIdentityMismatchError,
+    GetResult,
+    HealthStatus,
+    PalaceNotFoundError,
+    PalaceRef,
+    QueryResult,
+    UnsupportedFilterError,
+)
 from .chroma import ChromaBackend, ChromaCollection
+from .registry import (
+    available_backends,
+    get_backend,
+    get_backend_class,
+    register,
+    reset_backends,
+    resolve_backend_for_palace,
+    unregister,
+)
 
-__all__ = ["BaseCollection", "ChromaBackend", "ChromaCollection"]
+__all__ = [
+    "BackendClosedError",
+    "BackendError",
+    "BaseBackend",
+    "BaseCollection",
+    "ChromaBackend",
+    "ChromaCollection",
+    "DimensionMismatchError",
+    "EmbedderIdentityMismatchError",
+    "GetResult",
+    "HealthStatus",
+    "PalaceNotFoundError",
+    "PalaceRef",
+    "QueryResult",
+    "UnsupportedFilterError",
+    "available_backends",
+    "get_backend",
+    "get_backend_class",
+    "register",
+    "reset_backends",
+    "resolve_backend_for_palace",
+    "unregister",
+]

mempalace/backends/base.py

@@ -1,49 +1,370 @@
-"""Abstract collection interface for MemPalace storage backends."""
+"""Storage backend contract for MemPalace (RFC 001).
+
+This module defines the surface every storage backend must implement:
+
+* ``BaseCollection`` — the per-collection read/write interface, kwargs-only.
+* ``BaseBackend`` — the per-palace factory, addressed by ``PalaceRef``.
+* ``QueryResult`` / ``GetResult`` — typed result dataclasses that replace the
+  Chroma dict shape as the canonical return type.
+* Error classes + ``HealthStatus`` — uniform across backends.
+
+This is the v1 cleanup from RFC 001 §10: full typed results, ``PalaceRef``,
+registry-ready ABC. Embedder injection, maintenance hooks, and the full
+conformance suite land in follow-up PRs.
+"""
 
 from abc import ABC, abstractmethod
-from typing import Any, Dict, List, Optional
+from dataclasses import dataclass
+from typing import ClassVar, Optional
+
+
+# ---------------------------------------------------------------------------
+# Errors
+# ---------------------------------------------------------------------------
+
+
+class BackendError(Exception):
+    """Base class for every storage-backend error raised by core."""
+
+
+class PalaceNotFoundError(BackendError, FileNotFoundError):
+    """Raised when ``get_collection(create=False)`` is called on a missing palace.
+
+    Subclass of ``FileNotFoundError`` so legacy callers that catch the latter
+    (pre-#413 seam) keep working unchanged.
+    """
+
+
+class BackendClosedError(BackendError):
+    """Raised when a backend method is called after ``close()``."""
+
+
+class UnsupportedFilterError(BackendError):
+    """Raised when a where-clause uses an operator the backend does not implement.
+
+    Silent dropping of unknown operators is forbidden by spec (RFC 001 §1.4).
+    """
+
+
+class DimensionMismatchError(BackendError):
+    """Raised when the embedding dimension on write does not match the collection."""
+
+
+class EmbedderIdentityMismatchError(BackendError):
+    """Raised when the stored embedder model name differs from the current one."""
+
+
+# ---------------------------------------------------------------------------
+# Value objects
+# ---------------------------------------------------------------------------
+
+
+@dataclass(frozen=True)
+class PalaceRef:
+    """A handle to a palace, consumed by backends.
+
+    ``id`` is always present and is the key backends use to cache handles.
+    ``local_path`` is populated for filesystem-rooted palaces.
+    ``namespace`` is used by server-mode backends for tenant / prefix routing.
+    """
+
+    id: str
+    local_path: Optional[str] = None
+    namespace: Optional[str] = None
+
+
+@dataclass(frozen=True)
+class HealthStatus:
+    ok: bool
+    detail: str = ""
+
+    @classmethod
+    def healthy(cls, detail: str = "") -> "HealthStatus":
+        return cls(ok=True, detail=detail)
+
+    @classmethod
+    def unhealthy(cls, detail: str) -> "HealthStatus":
+        return cls(ok=False, detail=detail)
+
+
+_TYPED_RESULT_FIELDS = ("ids", "documents", "metadatas", "distances", "embeddings")
+
+
+class _DictCompatMixin:
+    """Transitional dict-protocol access for typed results.
+
+    RFC 001 §1.3 spec is attribute access (``result.ids``). The ``result["ids"]``
+    and ``result.get("ids")`` forms are retained as a migration shim for callers
+    that predate the typed interface and are scheduled for removal in a follow-
+    up cleanup. New code MUST use attribute access.
+    """
+
+    def __getitem__(self, key: str):
+        if key in _TYPED_RESULT_FIELDS:
+            return getattr(self, key)
+        raise KeyError(key)
+
+    def get(self, key: str, default=None):
+        if key in _TYPED_RESULT_FIELDS:
+            val = getattr(self, key, default)
+            return default if val is None else val
+        return default
+
+    def __contains__(self, key: object) -> bool:
+        return key in _TYPED_RESULT_FIELDS and getattr(self, key, None) is not None
+
+
+@dataclass(frozen=True)
+class QueryResult(_DictCompatMixin):
+    """Typed return from ``BaseCollection.query``.
+
+    Outer list dimension = number of query vectors / texts.
+    Inner list dimension = hits per query (may be zero).
+
+    Fields not in ``include=`` at the call site are populated with empty lists
+    of the correct outer shape (never ``None``), except ``embeddings`` which
+    is ``None`` when not requested.
+    """
+
+    ids: list[list[str]]
+    documents: list[list[str]]
+    metadatas: list[list[dict]]
+    distances: list[list[float]]
+    embeddings: Optional[list[list[list[float]]]] = None
+
+    @classmethod
+    def empty(cls, num_queries: int = 1, embeddings_requested: bool = False) -> "QueryResult":
+        """Construct an all-empty result preserving outer dimension.
+
+        When ``embeddings_requested`` is True, ``embeddings`` preserves the outer
+        query dimension with empty hit lists (matching the spec's rule that fields
+        requested via ``include=`` carry the outer shape even when empty). When
+        False, ``embeddings`` stays ``None`` to signal the field was not requested.
+        """
+        empty_outer = [[] for _ in range(num_queries)]
+        return cls(
+            ids=[[] for _ in range(num_queries)],
+            documents=[[] for _ in range(num_queries)],
+            metadatas=[[] for _ in range(num_queries)],
+            distances=[[] for _ in range(num_queries)],
+            embeddings=empty_outer if embeddings_requested else None,
+        )
+
+
+@dataclass(frozen=True)
+class GetResult(_DictCompatMixin):
+    """Typed return from ``BaseCollection.get``."""
+
+    ids: list[str]
+    documents: list[str]
+    metadatas: list[dict]
+    embeddings: Optional[list[list[float]]] = None
+
+    @classmethod
+    def empty(cls) -> "GetResult":
+        return cls(ids=[], documents=[], metadatas=[], embeddings=None)
+
+
+# ---------------------------------------------------------------------------
+# Collection contract
+# ---------------------------------------------------------------------------
 
 
 class BaseCollection(ABC):
-    """Smallest collection contract the rest of MemPalace relies on."""
+    """Per-collection read/write surface every backend must implement."""
 
     @abstractmethod
     def add(
         self,
         *,
-        documents: List[str],
-        ids: List[str],
-        metadatas: Optional[List[Dict[str, Any]]] = None,
-    ) -> None:
-        raise NotImplementedError
+        documents: list[str],
+        ids: list[str],
+        metadatas: Optional[list[dict]] = None,
+        embeddings: Optional[list[list[float]]] = None,
+    ) -> None: ...
 
     @abstractmethod
     def upsert(
         self,
         *,
-        documents: List[str],
-        ids: List[str],
-        metadatas: Optional[List[Dict[str, Any]]] = None,
+        documents: list[str],
+        ids: list[str],
+        metadatas: Optional[list[dict]] = None,
+        embeddings: Optional[list[list[float]]] = None,
+    ) -> None: ...
+
+    @abstractmethod
+    def query(
+        self,
+        *,
+        query_texts: Optional[list[str]] = None,
+        query_embeddings: Optional[list[list[float]]] = None,
+        n_results: int = 10,
+        where: Optional[dict] = None,
+        where_document: Optional[dict] = None,
+        include: Optional[list[str]] = None,
+    ) -> QueryResult: ...
+
+    @abstractmethod
+    def get(
+        self,
+        *,
+        ids: Optional[list[str]] = None,
+        where: Optional[dict] = None,
+        where_document: Optional[dict] = None,
+        limit: Optional[int] = None,
+        offset: Optional[int] = None,
+        include: Optional[list[str]] = None,
+    ) -> GetResult: ...
+
+    @abstractmethod
+    def delete(
+        self,
+        *,
+        ids: Optional[list[str]] = None,
+        where: Optional[dict] = None,
+    ) -> None: ...
+
+    @abstractmethod
+    def count(self) -> int: ...
+
+    # ------------------------------------------------------------------
+    # Optional methods with ABC defaults (spec §1.2)
+    # ------------------------------------------------------------------
+
+    def estimated_count(self) -> int:
+        return self.count()
+
+    def close(self) -> None:
+        return None
+
+    def health(self) -> HealthStatus:
+        return HealthStatus.healthy()
+
+    def update(
+        self,
+        *,
+        ids: list[str],
+        documents: Optional[list[str]] = None,
+        metadatas: Optional[list[dict]] = None,
+        embeddings: Optional[list[list[float]]] = None,
     ) -> None:
-        raise NotImplementedError
+        """Default non-atomic update: get + merge + upsert.
+
+        Backends advertising ``supports_update`` MUST override with an atomic
+        single-round-trip implementation.
+        """
+        if documents is None and metadatas is None and embeddings is None:
+            raise ValueError("update requires at least one of documents, metadatas, embeddings")
+
+        n = len(ids)
+        for label, value in (
+            ("documents", documents),
+            ("metadatas", metadatas),
+            ("embeddings", embeddings),
+        ):
+            if value is not None and len(value) != n:
+                raise ValueError(f"{label} length {len(value)} does not match ids length {n}")
+
+        existing = self.get(ids=ids, include=["documents", "metadatas"])
+        by_id = {
+            rid: (existing.documents[i], existing.metadatas[i])
+            for i, rid in enumerate(existing.ids)
+        }
+        merged_docs: list[str] = []
+        merged_metas: list[dict] = []
+        for i, rid in enumerate(ids):
+            prev_doc, prev_meta = by_id.get(rid, ("", {}))
+            merged_docs.append(documents[i] if documents is not None else prev_doc)
+            new_meta = dict(prev_meta or {})
+            if metadatas is not None:
+                new_meta.update(metadatas[i] or {})
+            merged_metas.append(new_meta)
+        self.upsert(
+            documents=merged_docs,
+            ids=list(ids),
+            metadatas=merged_metas,
+            embeddings=embeddings,
+        )
+
+
+# ---------------------------------------------------------------------------
+# Backend contract
+# ---------------------------------------------------------------------------
+
+
+class BaseBackend(ABC):
+    """Long-lived factory serving many palaces (RFC 001 §2).
+
+    Instances are lightweight on construction — no I/O, no network. All
+    connection work is deferred to ``get_collection``. Instances are thread-
+    safe for concurrent ``get_collection`` calls across different palaces.
+    """
+
+    name: ClassVar[str]
+    spec_version: ClassVar[str] = "1.0"
+    capabilities: ClassVar[frozenset[str]] = frozenset()
 
     @abstractmethod
-    def update(self, **kwargs: Any) -> None:
-        """Update existing records. Must raise if any ID is missing."""
-        raise NotImplementedError
+    def get_collection(
+        self,
+        *,
+        palace: PalaceRef,
+        collection_name: str,
+        create: bool = False,
+        options: Optional[dict] = None,
+    ) -> BaseCollection: ...
 
-    @abstractmethod
-    def query(self, **kwargs: Any) -> Dict[str, Any]:
-        raise NotImplementedError
+    def close_palace(self, palace: PalaceRef) -> None:
+        """Evict cached handles for a single palace. Default: no-op."""
+        return None
 
-    @abstractmethod
-    def get(self, **kwargs: Any) -> Dict[str, Any]:
-        raise NotImplementedError
+    def close(self) -> None:
+        """Shut down the entire backend. Default: no-op."""
+        return None
 
-    @abstractmethod
-    def delete(self, **kwargs: Any) -> None:
-        raise NotImplementedError
+    def health(self, palace: Optional[PalaceRef] = None) -> HealthStatus:
+        return HealthStatus.healthy()
 
-    @abstractmethod
-    def count(self) -> int:
-        raise NotImplementedError
+    # Optional detection hint used by selection priority (RFC 001 §3.3 (4)):
+    @classmethod
+    def detect(cls, path: str) -> bool:  # pragma: no cover - default hook
+        return False
+
+
+# ---------------------------------------------------------------------------
+# Adapter utilities
+# ---------------------------------------------------------------------------
+
+
+# Keys the Chroma ``include=`` parameter accepts.
+_VALID_INCLUDE_KEYS = frozenset({"documents", "metadatas", "distances", "embeddings"})
+
+
+@dataclass
+class _IncludeSpec:
+    """Resolve an ``include=`` parameter with spec-mandated defaults."""
+
+    documents: bool = True
+    metadatas: bool = True
+    distances: bool = True  # only meaningful for query
+    embeddings: bool = False
+
+    @classmethod
+    def resolve(
+        cls, include: Optional[list[str]], *, default_distances: bool = True
+    ) -> "_IncludeSpec":
+        if include is None:
+            return cls(
+                documents=True,
+                metadatas=True,
+                distances=default_distances,
+                embeddings=False,
+            )
+        keys = {k for k in include if k in _VALID_INCLUDE_KEYS}
+        return cls(
+            documents="documents" in keys,
+            metadatas="metadatas" in keys,
+            distances="distances" in keys,
+            embeddings="embeddings" in keys,
+        )

mempalace/backends/chroma.py

@@ -1,17 +1,137 @@
-"""ChromaDB-backed MemPalace collection adapter."""
+"""ChromaDB-backed MemPalace storage backend (RFC 001 reference implementation)."""
 
+import datetime as _dt
 import logging
 import os
 import sqlite3
+from typing import Any, Optional
 
 import chromadb
 
-from .base import BaseCollection
+from .base import (
+    BaseBackend,
+    BaseCollection,
+    GetResult,
+    HealthStatus,
+    PalaceNotFoundError,
+    PalaceRef,
+    QueryResult,
+    UnsupportedFilterError,
+    _IncludeSpec,
+)
 
 logger = logging.getLogger(__name__)
 
 
-def _fix_blob_seq_ids(palace_path: str):
+_REQUIRED_OPERATORS = frozenset({"$eq", "$ne", "$in", "$nin", "$and", "$or", "$contains"})
+_OPTIONAL_OPERATORS = frozenset({"$gt", "$gte", "$lt", "$lte"})
+_SUPPORTED_OPERATORS = _REQUIRED_OPERATORS | _OPTIONAL_OPERATORS
+
+
+def _validate_where(where: Optional[dict]) -> None:
+    """Scan a where-clause for unknown operators and raise ``UnsupportedFilterError``.
+
+    Spec (RFC 001 §1.4): silent dropping of unknown operators is forbidden.
+    """
+    if not where:
+        return
+    stack = [where]
+    while stack:
+        node = stack.pop()
+        if not isinstance(node, dict):
+            continue
+        for k, v in node.items():
+            if k.startswith("$") and k not in _SUPPORTED_OPERATORS:
+                raise UnsupportedFilterError(f"operator {k!r} not supported by chroma backend")
+            if isinstance(v, dict):
+                stack.append(v)
+            elif isinstance(v, list):
+                stack.extend(x for x in v if isinstance(x, dict))
+
+
+def quarantine_stale_hnsw(palace_path: str, stale_seconds: float = 3600.0) -> list[str]:
+    """Rename HNSW segment dirs whose files are stale vs. chroma.sqlite3.
+
+    When a ChromaDB 1.5.x PersistentClient opens a palace whose on-disk
+    HNSW segment is significantly older than ``chroma.sqlite3``, the Rust
+    graph-walk can dereference dangling neighbor pointers for entries that
+    exist in the metadata segment but not in the HNSW index, and segfault
+    in a background thread on the next ``count()`` or ``query(...)`` call.
+
+    This is the same failure mode reported at #823 (semantic search stale
+    after ``add_drawer``), observed at neo-cortex-mcp#2 (SIGSEGV on
+    ``count()`` with chromadb 1.5.5), and acknowledged as by-design at
+    chroma-core/chroma#2594. On one fork palace (135K drawers), the drift
+    caused a 65–85% crash rate on fresh-process opens; fresh-process
+    crash rate dropped to 0% after the segment dir was renamed out of the
+    way and ChromaDB rebuilt lazily.
+
+    Heuristic: if ``chroma.sqlite3`` is more than ``stale_seconds`` newer
+    than the segment's ``data_level0.bin``, the segment is considered
+    suspect and renamed to ``<uuid>.drift-<timestamp>``. ChromaDB reopens
+    cleanly without it and writes fresh index files on next use. The
+    original directory is renamed, not deleted, so recovery remains
+    possible if the heuristic misfires.
+
+    The default threshold (1h) is deliberately conservative — ChromaDB's
+    HNSW flush cadence means legitimate drift is normally on the order of
+    seconds to minutes. A segment that is more than an hour out of date is
+    almost certainly in a "crashed mid-write" state.
+
+    Args:
+        palace_path: path to the palace directory containing ``chroma.sqlite3``
+        stale_seconds: minimum mtime gap to treat a segment as stale
+
+    Returns:
+        List of paths that were quarantined (empty if nothing drifted).
+    """
+    db_path = os.path.join(palace_path, "chroma.sqlite3")
+    if not os.path.isfile(db_path):
+        return []
+    try:
+        sqlite_mtime = os.path.getmtime(db_path)
+    except OSError:
+        return []
+
+    moved: list[str] = []
+    try:
+        entries = os.listdir(palace_path)
+    except OSError:
+        return []
+
+    for name in entries:
+        if "-" not in name or name.startswith(".") or ".drift-" in name:
+            continue
+        seg_dir = os.path.join(palace_path, name)
+        if not os.path.isdir(seg_dir):
+            continue
+        hnsw_bin = os.path.join(seg_dir, "data_level0.bin")
+        if not os.path.isfile(hnsw_bin):
+            continue
+        try:
+            hnsw_mtime = os.path.getmtime(hnsw_bin)
+        except OSError:
+            continue
+        if sqlite_mtime - hnsw_mtime < stale_seconds:
+            continue
+        stamp = _dt.datetime.now().strftime("%Y%m%d-%H%M%S")
+        target = f"{seg_dir}.drift-{stamp}"
+        try:
+            os.rename(seg_dir, target)
+            moved.append(target)
+            logger.warning(
+                "Quarantined stale HNSW segment %s "
+                "(sqlite %.0fs newer than HNSW); renamed to %s",
+                seg_dir,
+                sqlite_mtime - hnsw_mtime,
+                target,
+            )
+        except OSError:
+            logger.exception("Failed to quarantine stale HNSW segment %s", seg_dir)
+    return moved
+
+
+def _fix_blob_seq_ids(palace_path: str) -> None:
     """Fix ChromaDB 0.6.x -> 1.5.x migration bug: BLOB seq_ids -> INTEGER.
 
     ChromaDB 0.6.x stored seq_id as big-endian 8-byte BLOBs. ChromaDB 1.5.x
@@ -43,62 +163,328 @@ def _fix_blob_seq_ids(palace_path: str):
         logger.exception("Could not fix BLOB seq_ids in %s", db_path)
 
 
+# ---------------------------------------------------------------------------
+# Collection adapter
+# ---------------------------------------------------------------------------
+
+
+def _as_list(v: Any) -> list:
+    """Coerce possibly-None scalar-or-list into a list (defensive for chroma nulls)."""
+    if v is None:
+        return []
+    if isinstance(v, list):
+        return v
+    return [v]
+
+
 class ChromaCollection(BaseCollection):
-    """Thin adapter over a ChromaDB collection."""
+    """Thin adapter translating ChromaDB dict returns into typed results."""
 
     def __init__(self, collection):
         self._collection = collection
 
-    def add(self, *, documents, ids, metadatas=None):
-        self._collection.add(documents=documents, ids=ids, metadatas=metadatas)
+    # ------------------------------------------------------------------
+    # Writes
+    # ------------------------------------------------------------------
 
-    def upsert(self, *, documents, ids, metadatas=None):
-        self._collection.upsert(documents=documents, ids=ids, metadatas=metadatas)
+    def add(self, *, documents, ids, metadatas=None, embeddings=None):
+        kwargs: dict[str, Any] = {"documents": documents, "ids": ids}
+        if metadatas is not None:
+            kwargs["metadatas"] = metadatas
+        if embeddings is not None:
+            kwargs["embeddings"] = embeddings
+        self._collection.add(**kwargs)
 
-    def update(self, **kwargs):
+    def upsert(self, *, documents, ids, metadatas=None, embeddings=None):
+        kwargs: dict[str, Any] = {"documents": documents, "ids": ids}
+        if metadatas is not None:
+            kwargs["metadatas"] = metadatas
+        if embeddings is not None:
+            kwargs["embeddings"] = embeddings
+        self._collection.upsert(**kwargs)
+
+    def update(
+        self,
+        *,
+        ids,
+        documents=None,
+        metadatas=None,
+        embeddings=None,
+    ):
+        if documents is None and metadatas is None and embeddings is None:
+            raise ValueError("update requires at least one of documents, metadatas, embeddings")
+        kwargs: dict[str, Any] = {"ids": ids}
+        if documents is not None:
+            kwargs["documents"] = documents
+        if metadatas is not None:
+            kwargs["metadatas"] = metadatas
+        if embeddings is not None:
+            kwargs["embeddings"] = embeddings
         self._collection.update(**kwargs)
 
-    def query(self, **kwargs):
-        return self._collection.query(**kwargs)
+    # ------------------------------------------------------------------
+    # Reads
+    # ------------------------------------------------------------------
 
-    def get(self, **kwargs):
-        return self._collection.get(**kwargs)
+    def query(
+        self,
+        *,
+        query_texts=None,
+        query_embeddings=None,
+        n_results=10,
+        where=None,
+        where_document=None,
+        include=None,
+    ) -> QueryResult:
+        _validate_where(where)
+        _validate_where(where_document)
 
-    def delete(self, **kwargs):
+        if (query_texts is None) == (query_embeddings is None):
+            raise ValueError("query requires exactly one of query_texts or query_embeddings")
+        chosen = query_texts if query_texts is not None else query_embeddings
+        if not chosen:
+            raise ValueError("query input must be a non-empty list")
+
+        spec = _IncludeSpec.resolve(include, default_distances=True)
+        chroma_include: list[str] = []
+        if spec.documents:
+            chroma_include.append("documents")
+        if spec.metadatas:
+            chroma_include.append("metadatas")
+        if spec.distances:
+            chroma_include.append("distances")
+        if spec.embeddings:
+            chroma_include.append("embeddings")
+
+        kwargs: dict[str, Any] = {
+            "n_results": n_results,
+            "include": chroma_include,
+        }
+        if query_texts is not None:
+            kwargs["query_texts"] = query_texts
+        if query_embeddings is not None:
+            kwargs["query_embeddings"] = query_embeddings
+        if where is not None:
+            kwargs["where"] = where
+        if where_document is not None:
+            kwargs["where_document"] = where_document
+
+        raw = self._collection.query(**kwargs)
+
+        num_queries = (
+            len(query_texts)
+            if query_texts is not None
+            else (len(query_embeddings) if query_embeddings is not None else 1)
+        )
+
+        ids = raw.get("ids") or []
+        if not ids:
+            return QueryResult.empty(
+                num_queries=num_queries,
+                embeddings_requested=spec.embeddings,
+            )
+
+        documents = raw.get("documents") or [[] for _ in ids]
+        metadatas = raw.get("metadatas") or [[] for _ in ids]
+        distances = raw.get("distances") or [[] for _ in ids]
+        embeddings_raw = raw.get("embeddings") if spec.embeddings else None
+
+        def _none_list_to_empty(outer):
+            return [(inner or []) for inner in outer]
+
+        return QueryResult(
+            ids=_none_list_to_empty(ids),
+            documents=_none_list_to_empty(documents),
+            metadatas=_none_list_to_empty(metadatas),
+            distances=_none_list_to_empty(distances),
+            embeddings=(
+                [list(inner) for inner in embeddings_raw]
+                if spec.embeddings and embeddings_raw is not None
+                else None
+            ),
+        )
+
+    def get(
+        self,
+        *,
+        ids=None,
+        where=None,
+        where_document=None,
+        limit=None,
+        offset=None,
+        include=None,
+    ) -> GetResult:
+        _validate_where(where)
+        _validate_where(where_document)
+
+        spec = _IncludeSpec.resolve(include, default_distances=False)
+        chroma_include: list[str] = []
+        if spec.documents:
+            chroma_include.append("documents")
+        if spec.metadatas:
+            chroma_include.append("metadatas")
+        if spec.embeddings:
+            chroma_include.append("embeddings")
+
+        kwargs: dict[str, Any] = {"include": chroma_include}
+        if ids is not None:
+            kwargs["ids"] = ids
+        if where is not None:
+            kwargs["where"] = where
+        if where_document is not None:
+            kwargs["where_document"] = where_document
+        if limit is not None:
+            kwargs["limit"] = limit
+        if offset is not None:
+            kwargs["offset"] = offset
+
+        raw = self._collection.get(**kwargs)
+        out_ids = list(raw.get("ids") or [])
+        out_docs = list(raw.get("documents") or []) if spec.documents else []
+        out_metas = list(raw.get("metadatas") or []) if spec.metadatas else []
+        out_embeds = raw.get("embeddings") if spec.embeddings else None
+
+        # Pad doc/meta lists to match ids so downstream zipping is safe.
+        if spec.documents and len(out_docs) < len(out_ids):
+            out_docs = out_docs + [""] * (len(out_ids) - len(out_docs))
+        if spec.metadatas and len(out_metas) < len(out_ids):
+            out_metas = out_metas + [{}] * (len(out_ids) - len(out_metas))
+
+        return GetResult(
+            ids=out_ids,
+            documents=out_docs,
+            metadatas=out_metas,
+            embeddings=[list(v) for v in out_embeds] if out_embeds is not None else None,
+        )
+
+    def delete(self, *, ids=None, where=None):
+        _validate_where(where)
+        kwargs: dict[str, Any] = {}
+        if ids is not None:
+            kwargs["ids"] = ids
+        if where is not None:
+            kwargs["where"] = where
         self._collection.delete(**kwargs)
 
     def count(self):
         return self._collection.count()
 
 
-class ChromaBackend:
-    """Factory for MemPalace's default ChromaDB backend."""
+# ---------------------------------------------------------------------------
+# Backend
+# ---------------------------------------------------------------------------
+
+
+class ChromaBackend(BaseBackend):
+    """MemPalace's default ChromaDB backend.
+
+    Maintains two caches:
+
+    * ``self._clients`` — ``palace_path -> PersistentClient`` for callers
+      using the ``PalaceRef`` / :meth:`get_collection` path.
+    * An inode+mtime freshness check absorbed from ``mcp_server._get_client``
+      (merged via #757) ensuring a palace rebuild on disk is detected on the
+      next :meth:`get_collection` call.
+    """
+
+    name = "chroma"
+    capabilities = frozenset(
+        {
+            "supports_embeddings_in",
+            "supports_embeddings_passthrough",
+            "supports_embeddings_out",
+            "supports_metadata_filters",
+            "supports_contains_fast",
+            "local_mode",
+        }
+    )
 
     def __init__(self):
-        # Per-instance client cache: palace_path -> chromadb.PersistentClient
-        self._clients: dict = {}
+        # palace_path -> PersistentClient
+        self._clients: dict[str, Any] = {}
+        # palace_path -> (inode, mtime) of chroma.sqlite3 at cache time.
+        self._freshness: dict[str, tuple[int, float]] = {}
+        self._closed = False
 
     # ------------------------------------------------------------------
     # Internal helpers
     # ------------------------------------------------------------------
 
+    @staticmethod
+    def _db_stat(palace_path: str) -> tuple[int, float]:
+        """Return ``(inode, mtime)`` of ``chroma.sqlite3`` or ``(0, 0.0)`` if absent."""
+        db_path = os.path.join(palace_path, "chroma.sqlite3")
+        try:
+            st = os.stat(db_path)
+            return (st.st_ino, st.st_mtime)
+        except OSError:
+            return (0, 0.0)
+
     def _client(self, palace_path: str):
-        """Return a cached PersistentClient for *palace_path*, creating one if needed."""
-        if palace_path not in self._clients:
+        """Return a cached ``PersistentClient``, rebuilding on inode/mtime change.
+
+        Handles the palace-rebuild case (repair/nuke/purge) by invalidating the
+        cache when ``chroma.sqlite3`` changes on disk. Mirrors the semantics of
+        ``mcp_server._get_client`` (merged via #757):
+
+        * DB file missing while we hold a cached client → drop the cache so we
+          do not serve stale data after a rebuild that has not yet re-created
+          the DB.
+        * Transition 0 → nonzero stat (DB created after cache) counts as a
+          change, so the cached client is replaced with one that sees the DB.
+        * FAT/exFAT filesystems return inode 0; we never fire inode comparisons
+          when either side is 0 (safe fallback) but still honor mtime.
+        * Mtime change uses an epsilon (0.01 s) to tolerate FS timestamp
+          granularity without thrashing.
+        """
+        if self._closed:
+            from .base import BackendClosedError  # late import avoids cycles at module load
+
+            raise BackendClosedError("ChromaBackend has been closed")
+
+        cached = self._clients.get(palace_path)
+        cached_inode, cached_mtime = self._freshness.get(palace_path, (0, 0.0))
+        current_inode, current_mtime = self._db_stat(palace_path)
+
+        db_path = os.path.join(palace_path, "chroma.sqlite3")
+        # DB was present when cache was built but is now missing → invalidate.
+        if cached is not None and not os.path.isfile(db_path):
+            self._clients.pop(palace_path, None)
+            self._freshness.pop(palace_path, None)
+            cached = None
+            cached_inode, cached_mtime = 0, 0.0
+
+        inode_changed = current_inode != 0 and cached_inode != 0 and current_inode != cached_inode
+        # Transition from no-stat (0.0) to a real stat counts as a change so we
+        # pick up a DB that was created after the cache was built.
+        mtime_appeared = cached_mtime == 0.0 and current_mtime != 0.0
+        mtime_changed = (
+            current_mtime != 0.0
+            and cached_mtime != 0.0
+            and abs(current_mtime - cached_mtime) > 0.01
+        )
+
+        if cached is None or inode_changed or mtime_changed or mtime_appeared:
             _fix_blob_seq_ids(palace_path)
-            self._clients[palace_path] = chromadb.PersistentClient(path=palace_path)
-        return self._clients[palace_path]
+            cached = chromadb.PersistentClient(path=palace_path)
+            self._clients[palace_path] = cached
+            # Re-stat after the client constructor runs: chromadb creates
+            # chroma.sqlite3 lazily, so the stat captured before the call
+            # may still be (0, 0.0) on first open.
+            self._freshness[palace_path] = self._db_stat(palace_path)
+        return cached
 
     # ------------------------------------------------------------------
-    # Public static helpers (for callers that manage their own caching)
+    # Public static helpers (legacy; prefer :meth:`get_collection`)
     # ------------------------------------------------------------------
 
     @staticmethod
     def make_client(palace_path: str):
-        """Create and return a fresh PersistentClient (fix BLOB seq_ids first).
+        """Create a fresh ``PersistentClient`` (fixes BLOB seq_ids first).
 
-        Intended for long-lived callers (e.g. mcp_server) that keep their own
-        inode/mtime-based client cache.
+        Deprecated-ish: exposed for legacy long-lived callers that manage their
+        own client cache. New code should obtain a collection through
+        :meth:`get_collection` which manages caching internally.
         """
         _fix_blob_seq_ids(palace_path)
         return chromadb.PersistentClient(path=palace_path)
@@ -109,12 +495,31 @@ class ChromaBackend:
         return chromadb.__version__
 
     # ------------------------------------------------------------------
-    # Collection lifecycle
+    # BaseBackend surface
     # ------------------------------------------------------------------
 
-    def get_collection(self, palace_path: str, collection_name: str, create: bool = False):
+    def get_collection(
+        self,
+        *args,
+        **kwargs,
+    ) -> ChromaCollection:
+        """Obtain a collection for a palace.
+
+        Supports two calling conventions during the RFC 001 transition:
+
+        * New (preferred): ``get_collection(palace=PalaceRef, collection_name=...,
+          create=False, options=None)``.
+        * Legacy: ``get_collection(palace_path, collection_name, create=False)``
+          — still used by callers not yet migrated.
+        """
+        palace_ref, collection_name, create, options = _normalize_get_collection_args(args, kwargs)
+
+        palace_path = palace_ref.local_path
+        if palace_path is None:
+            raise PalaceNotFoundError("ChromaBackend requires PalaceRef.local_path")
+
         if not create and not os.path.isdir(palace_path):
-            raise FileNotFoundError(palace_path)
+            raise PalaceNotFoundError(palace_path)
 
         if create:
             os.makedirs(palace_path, exist_ok=True)
@@ -124,29 +529,115 @@ class ChromaBackend:
                 pass
 
         client = self._client(palace_path)
+        hnsw_space = "cosine"
+        if options and isinstance(options, dict):
+            hnsw_space = options.get("hnsw_space", hnsw_space)
+
         if create:
             collection = client.get_or_create_collection(
-                collection_name, metadata={"hnsw:space": "cosine"}
+                collection_name, metadata={"hnsw:space": hnsw_space}
             )
         else:
             collection = client.get_collection(collection_name)
         return ChromaCollection(collection)
 
-    def get_or_create_collection(
-        self, palace_path: str, collection_name: str
-    ) -> "ChromaCollection":
-        """Shorthand for get_collection(..., create=True)."""
+    def close_palace(self, palace) -> None:
+        """Drop cached handles for ``palace``. Accepts ``PalaceRef`` or legacy path str."""
+        path = palace.local_path if isinstance(palace, PalaceRef) else palace
+        if path is None:
+            return
+        self._clients.pop(path, None)
+        self._freshness.pop(path, None)
+
+    def close(self) -> None:
+        self._clients.clear()
+        self._freshness.clear()
+        self._closed = True
+
+    def health(self, palace: Optional[PalaceRef] = None) -> HealthStatus:
+        if self._closed:
+            return HealthStatus.unhealthy("backend closed")
+        return HealthStatus.healthy()
+
+    @classmethod
+    def detect(cls, path: str) -> bool:
+        return os.path.isfile(os.path.join(path, "chroma.sqlite3"))
+
+    # ------------------------------------------------------------------
+    # Legacy (pre-RFC 001) surface — retained while callers migrate.
+    # ------------------------------------------------------------------
+
+    def get_or_create_collection(self, palace_path: str, collection_name: str) -> ChromaCollection:
+        """Legacy shim for ``get_collection(..., create=True)`` by path string."""
         return self.get_collection(palace_path, collection_name, create=True)
 
     def delete_collection(self, palace_path: str, collection_name: str) -> None:
-        """Delete *collection_name* from the palace at *palace_path*."""
+        """Delete ``collection_name`` from the palace at ``palace_path``."""
         self._client(palace_path).delete_collection(collection_name)
 
     def create_collection(
         self, palace_path: str, collection_name: str, hnsw_space: str = "cosine"
-    ) -> "ChromaCollection":
-        """Create (not get-or-create) *collection_name* with cosine HNSW space."""
+    ) -> ChromaCollection:
+        """Create (not get-or-create) ``collection_name`` with the given HNSW space."""
         collection = self._client(palace_path).create_collection(
             collection_name, metadata={"hnsw:space": hnsw_space}
         )
         return ChromaCollection(collection)
+
+
+def _normalize_get_collection_args(args, kwargs):
+    """Unify legacy positional ``(palace_path, collection_name, create)`` calls
+    with the new kwargs-only ``(palace=PalaceRef, collection_name=..., create=...)``.
+
+    Returns ``(PalaceRef, collection_name, create, options)``.
+    """
+    # New-style: palace= kwarg with a PalaceRef (spec path).
+    if "palace" in kwargs:
+        palace_ref = kwargs.pop("palace")
+        if not isinstance(palace_ref, PalaceRef):
+            raise TypeError("palace= must be a PalaceRef instance")
+        collection_name = kwargs.pop("collection_name")
+        create = kwargs.pop("create", False)
+        options = kwargs.pop("options", None)
+        if kwargs:
+            raise TypeError(f"unexpected kwargs: {sorted(kwargs)}")
+        if args:
+            raise TypeError("positional args not allowed with palace= kwarg")
+        return palace_ref, collection_name, create, options
+
+    # Legacy: first positional is a path string.
+    if args:
+        palace_path = args[0]
+        rest = list(args[1:])
+        collection_name = kwargs.pop("collection_name", None) or (rest.pop(0) if rest else None)
+        if collection_name is None:
+            raise TypeError("collection_name is required")
+        create = kwargs.pop("create", False)
+        if rest:
+            create = rest.pop(0)
+        if rest:
+            raise TypeError(f"unexpected positional args: {rest!r}")
+        if kwargs:
+            raise TypeError(f"unexpected kwargs: {sorted(kwargs)}")
+        return (
+            PalaceRef(id=palace_path, local_path=palace_path),
+            collection_name,
+            bool(create),
+            None,
+        )
+
+    # Legacy kwargs-only (palace_path=..., collection_name=..., create=...)
+    if "palace_path" in kwargs:
+        palace_path = kwargs.pop("palace_path")
+        collection_name = kwargs.pop("collection_name")
+        create = kwargs.pop("create", False)
+        if kwargs:
+            raise TypeError(f"unexpected kwargs: {sorted(kwargs)}")
+        return (
+            PalaceRef(id=palace_path, local_path=palace_path),
+            collection_name,
+            bool(create),
+            None,
+        )
+
+    raise TypeError("get_collection requires palace= or a positional palace_path")

mempalace/backends/registry.py

@@ -0,0 +1,189 @@
+"""Backend registry + entry-point discovery (RFC 001 §3).
+
+Third-party backends ship as installable packages that declare a
+``mempalace.backends`` entry point::
+
+    # pyproject.toml of mempalace-postgres
+    [project.entry-points."mempalace.backends"]
+    postgres = "mempalace_postgres:PostgresBackend"
+
+MemPalace discovers them at process start. In-tree tests and local development
+can register manually via :func:`register`. Explicit registration wins on
+name conflict (matches RFC 001 §3.2).
+"""
+
+from __future__ import annotations
+
+import logging
+from importlib import metadata
+from threading import Lock
+from typing import Optional, Type
+
+from .base import BaseBackend
+
+logger = logging.getLogger(__name__)
+
+_ENTRY_POINT_GROUP = "mempalace.backends"
+
+_registry: dict[str, Type[BaseBackend]] = {}
+_instances: dict[str, BaseBackend] = {}
+_explicit: set[str] = set()
+_discovered = False
+_lock = Lock()
+
+
+def register(name: str, backend_cls: Type[BaseBackend]) -> None:
+    """Register ``backend_cls`` under ``name``.
+
+    Explicit registration wins over entry-point discovery on conflict
+    (RFC 001 §3.2).
+    """
+    with _lock:
+        _registry[name] = backend_cls
+        _explicit.add(name)
+        # Invalidate any cached instance so the new class is used on next get.
+        _instances.pop(name, None)
+
+
+def unregister(name: str) -> None:
+    """Remove a backend registration (primarily for tests)."""
+    with _lock:
+        _registry.pop(name, None)
+        _explicit.discard(name)
+        _instances.pop(name, None)
+
+
+def _discover_entry_points() -> None:
+    """Load entry-point-declared backends once per process."""
+    global _discovered
+    if _discovered:
+        return
+    with _lock:
+        if _discovered:
+            return
+        try:
+            eps = metadata.entry_points()
+            # Py ≥ 3.10 returns an EntryPoints object; older versions returned a dict.
+            group = (
+                eps.select(group=_ENTRY_POINT_GROUP)
+                if hasattr(eps, "select")
+                else eps.get(_ENTRY_POINT_GROUP, [])
+            )
+        except Exception:
+            logger.exception("entry-point discovery for %s failed", _ENTRY_POINT_GROUP)
+            group = []
+        for ep in group:
+            if ep.name in _explicit:
+                continue  # explicit registration wins
+            try:
+                cls = ep.load()
+            except Exception:
+                logger.exception("failed to load backend entry point %r", ep.name)
+                continue
+            if not isinstance(cls, type) or not issubclass(cls, BaseBackend):
+                logger.warning(
+                    "entry point %r did not resolve to a BaseBackend subclass (got %r)",
+                    ep.name,
+                    cls,
+                )
+                continue
+            _registry.setdefault(ep.name, cls)
+        _discovered = True
+
+
+def available_backends() -> list[str]:
+    """Return sorted list of all registered backend names."""
+    _discover_entry_points()
+    return sorted(_registry.keys())
+
+
+def get_backend_class(name: str) -> Type[BaseBackend]:
+    """Return the registered backend class for ``name``."""
+    _discover_entry_points()
+    try:
+        return _registry[name]
+    except KeyError as e:
+        raise KeyError(f"unknown backend {name!r}; available: {available_backends()}") from e
+
+
+def get_backend(name: str) -> BaseBackend:
+    """Return a long-lived instance of the named backend.
+
+    Instances are cached per-name; repeated calls return the same object.
+    Call :func:`reset_backends` in tests that need isolation.
+    """
+    _discover_entry_points()
+    with _lock:
+        inst = _instances.get(name)
+        if inst is not None:
+            return inst
+        cls = _registry.get(name)
+        if cls is None:
+            raise KeyError(f"unknown backend {name!r}; available: {sorted(_registry.keys())}")
+        inst = cls()
+        _instances[name] = inst
+        return inst
+
+
+def reset_backends() -> None:
+    """Close and drop all cached backend instances (primarily for tests)."""
+    with _lock:
+        for inst in _instances.values():
+            try:
+                inst.close()
+            except Exception:
+                logger.exception("error closing backend during reset")
+        _instances.clear()
+
+
+def resolve_backend_for_palace(
+    *,
+    explicit: Optional[str] = None,
+    config_value: Optional[str] = None,
+    env_value: Optional[str] = None,
+    palace_path: Optional[str] = None,
+    default: str = "chroma",
+) -> str:
+    """Resolve the backend name for a palace per RFC 001 §3.3 priority order.
+
+    1. Explicit kwarg / CLI flag
+    2. Per-palace config value
+    3. ``MEMPALACE_BACKEND`` env var
+    4. Auto-detect from on-disk artifacts (migration/upgrade path only)
+    5. Default (``chroma``)
+
+    Auto-detection is strictly a migration aid: it fires only when a local path
+    is presented, no earlier rule has chosen a backend, AND the path already
+    contains backend-identifiable artifacts. For new palaces, (5) wins.
+    """
+    for candidate in (explicit, config_value, env_value):
+        if candidate:
+            return candidate
+
+    _discover_entry_points()
+    if palace_path:
+        for name, cls in _registry.items():
+            try:
+                if cls.detect(palace_path):
+                    return name
+            except Exception:
+                logger.exception("detect() raised on backend %r", name)
+                continue
+    return default
+
+
+# ---------------------------------------------------------------------------
+# Built-in registration
+# ---------------------------------------------------------------------------
+
+
+def _register_builtins() -> None:
+    """Register chroma as the in-tree default."""
+    from .chroma import ChromaBackend
+
+    # Use setdefault semantics so a caller that pre-registered for tests wins.
+    if "chroma" not in _registry:
+        _registry["chroma"] = ChromaBackend
+
+
+_register_builtins()

mempalace/cli.py

@@ -4,7 +4,7 @@ MemPalace — Give your AI a memory. No API key required.
 
 Two ways to ingest:
   Projects:      mempalace mine ~/projects/my_app          (code, docs, notes)
-  Conversations: mempalace mine ~/chats/ --mode convos     (Claude, ChatGPT, Slack)
+  Conversations: mempalace mine <convo-dir> --mode convos     (Claude Code, Claude.ai, ChatGPT, Slack exports)
 
 Same palace. Same search. Different ingest strategies.
 
@@ -22,7 +22,7 @@ Commands:
 Examples:
     mempalace init ~/projects/my_app
     mempalace mine ~/projects/my_app
-    mempalace mine ~/chats/claude-sessions --mode convos
+    mempalace mine ~/.claude/projects/-Users-you-Projects-my_app --mode convos --wing my_app
     mempalace search "why did we switch to GraphQL"
     mempalace search "pricing discussion" --wing my_app --room costs
 """
@@ -34,6 +34,7 @@ import argparse
 from pathlib import Path
 
 from .config import MempalaceConfig
+from .version import __version__
 
 
 _MEMPALACE_PROJECT_FILES = ("mempalace.yaml", "entities.json")
@@ -145,6 +146,48 @@ def cmd_mine(args):
         )
 
 
+def cmd_sweep(args):
+    """Sweep a transcript file or directory.
+
+    The sweeper deduplicates against its own prior writes via
+    deterministic drawer IDs + a timestamp cursor. It does NOT currently
+    coordinate with the file-level miners (miner.py / convo_miner.py) —
+    those produce char-chunked drawers without compatible message
+    metadata, so running both miners may store overlapping content under
+    different IDs.
+    """
+    from .sweeper import sweep, sweep_directory
+
+    palace_path = os.path.expanduser(args.palace) if args.palace else MempalaceConfig().palace_path
+    target = os.path.expanduser(args.target)
+
+    if os.path.isfile(target):
+        result = sweep(target, palace_path)
+        print(
+            f"  Swept {target}: +{result['drawers_added']} new, "
+            f"{result['drawers_already_present']} already present, "
+            f"{result['drawers_skipped']} skipped (< cursor)."
+        )
+    elif os.path.isdir(target):
+        result = sweep_directory(target, palace_path)
+        print(
+            f"  Swept {result['files_succeeded']}/{result['files_attempted']} "
+            f"files from {target}: +{result['drawers_added']} new, "
+            f"{result['drawers_already_present']} already present, "
+            f"{result['drawers_skipped']} skipped (< cursor)."
+        )
+        failures = result.get("failures") or []
+        if failures:
+            print(
+                f"  WARNING: {len(failures)} file(s) failed to sweep - see stderr / logs for details.",
+                file=sys.stderr,
+            )
+            sys.exit(2)
+    else:
+        print(f"  ERROR: Not a file or directory: {target}", file=sys.stderr)
+        sys.exit(1)
+
+
 def cmd_search(args):
     from .searcher import search, SearchError
 
@@ -470,10 +513,17 @@ def cmd_compress(args):
 
 
 def main():
+    version_label = f"MemPalace {__version__}"
     parser = argparse.ArgumentParser(
         description="MemPalace — Give your AI a memory. No API key required.",
         formatter_class=argparse.RawDescriptionHelpFormatter,
-        epilog=__doc__,
+        epilog=f"{version_label}\n\n{__doc__}",
+    )
+    parser.add_argument(
+        "--version",
+        action="version",
+        version=version_label,
+        help="Show version and exit",
     )
     parser.add_argument(
         "--palace",
@@ -539,6 +589,17 @@ def main():
         help="Extraction strategy for convos mode: 'exchange' (default) or 'general' (5 memory types)",
     )
 
+    # sweep
+    p_sweep = sub.add_parser(
+        "sweep",
+        help="Tandem miner: catch anything the primary miner missed "
+        "(message-level, timestamp-coordinated, idempotent)",
+    )
+    p_sweep.add_argument(
+        "target",
+        help="A .jsonl transcript file, or a directory to scan recursively",
+    )
+
     # search
     p_search = sub.add_parser("search", help="Find anything, exact words")
     p_search.add_argument("query", help="What to search for")
@@ -671,6 +732,7 @@ def main():
         "mine": cmd_mine,
         "split": cmd_split,
         "search": cmd_search,
+        "sweep": cmd_sweep,
         "mcp": cmd_mcp,
         "compress": cmd_compress,
         "wake-up": cmd_wakeup,

mempalace/convo_miner.py

@@ -55,7 +55,14 @@ CONVO_EXTENSIONS = {
 
 MIN_CHUNK_SIZE = 30
 CHUNK_SIZE = 800  # chars per drawer — align with miner.py
-MAX_FILE_SIZE = 10 * 1024 * 1024  # 10 MB — skip files larger than this
+MAX_FILE_SIZE = 500 * 1024 * 1024  # 500 MB — skip files larger than this.
+# Matches miner.py at 500 MB. Long Claude Code sessions, multi-year
+# ChatGPT exports, and lifetime Slack dumps routinely exceed 10 MB; the
+# cap at that level silently dropped them with `continue`. Per-drawer
+# size is bounded by CHUNK_SIZE, but larger source files still produce
+# more drawers and therefore more embedding/storage work — and content
+# is normalized and loaded fully into memory before chunking, so memory
+# use also scales with source size.
 
 
 def _register_file(collection, source_file: str, wing: str, agent: str):
@@ -471,7 +478,7 @@ def mine_convos(
             room_counts[r] += n
 
         total_drawers += drawers_added
-        print(f"  ✓ [{i:4}/{len(files)}] {filepath.name[:50]:50} +{drawers_added}")
+        print(f"  + [{i:4}/{len(files)}] {filepath.name[:50]:50} +{drawers_added}")
 
     print(f"\n{'=' * 55}")
     print("  Done.")

mempalace/hooks_cli.py

@@ -150,20 +150,70 @@ def _get_mine_dir(transcript_path: str = "") -> str:
     return ""
 
 
+_MINE_PID_FILE = STATE_DIR / "mine.pid"
+
+
+def _pid_alive(pid: int) -> bool:
+    """Cross-platform existence check for a PID.
+
+    On POSIX, ``os.kill(pid, 0)`` is the well-known no-op existence probe.
+    On Windows, ``os.kill`` maps to ``TerminateProcess(handle, sig)`` and
+    would *terminate* the target process with exit code ``sig`` — using
+    it here would kill our own mine child (or worse, the caller itself).
+    Use ``OpenProcess`` + ``GetExitCodeProcess`` via ctypes instead.
+    """
+    if sys.platform == "win32":
+        import ctypes
+        from ctypes import wintypes
+
+        PROCESS_QUERY_LIMITED_INFORMATION = 0x1000
+        STILL_ACTIVE = 259
+        kernel32 = ctypes.windll.kernel32
+        handle = kernel32.OpenProcess(PROCESS_QUERY_LIMITED_INFORMATION, False, pid)
+        if not handle:
+            return False
+        try:
+            code = wintypes.DWORD()
+            if not kernel32.GetExitCodeProcess(handle, ctypes.byref(code)):
+                return False
+            return code.value == STILL_ACTIVE
+        finally:
+            kernel32.CloseHandle(handle)
+    try:
+        os.kill(pid, 0)
+        return True
+    except (OSError, ValueError):
+        return False
+
+
+def _mine_already_running() -> bool:
+    """Return True if a background mine process from a previous hook fire is still alive."""
+    try:
+        pid = int(_MINE_PID_FILE.read_text().strip())
+    except (OSError, ValueError):
+        return False
+    return _pid_alive(pid)
+
+
+def _spawn_mine(cmd: list) -> None:
+    """Spawn a mine subprocess, write its PID to the lock file, log to hook.log."""
+    STATE_DIR.mkdir(parents=True, exist_ok=True)
+    log_path = STATE_DIR / "hook.log"
+    with open(log_path, "a") as log_f:
+        proc = subprocess.Popen(cmd, stdout=log_f, stderr=log_f)
+    _MINE_PID_FILE.write_text(str(proc.pid))
+
+
 def _maybe_auto_ingest(transcript_path: str = ""):
     """Run mempalace mine in background if a mine directory is available."""
     mine_dir = _get_mine_dir(transcript_path)
     if not mine_dir:
         return
+    if _mine_already_running():
+        _log("Skipping auto-ingest: mine already running")
+        return
     try:
-        STATE_DIR.mkdir(parents=True, exist_ok=True)
-        log_path = STATE_DIR / "hook.log"
-        with open(log_path, "a") as log_f:
-            subprocess.Popen(
-                [sys.executable, "-m", "mempalace", "mine", mine_dir],
-                stdout=log_f,
-                stderr=log_f,
-            )
+        _spawn_mine([sys.executable, "-m", "mempalace", "mine", mine_dir])
     except OSError:
         pass
 

mempalace/knowledge_graph.py

@@ -83,6 +83,8 @@ class KnowledgeGraph:
                 confidence REAL DEFAULT 1.0,
                 source_closet TEXT,
                 source_file TEXT,
+                source_drawer_id TEXT,
+                adapter_name TEXT,
                 extracted_at TEXT DEFAULT CURRENT_TIMESTAMP,
                 FOREIGN KEY (subject) REFERENCES entities(id),
                 FOREIGN KEY (object) REFERENCES entities(id)
@@ -93,8 +95,25 @@ class KnowledgeGraph:
             CREATE INDEX IF NOT EXISTS idx_triples_predicate ON triples(predicate);
             CREATE INDEX IF NOT EXISTS idx_triples_valid ON triples(valid_from, valid_to);
         """)
+        self._migrate_schema(conn)
         conn.commit()
 
+    def _migrate_schema(self, conn):
+        """Backwards-compatible schema migration for older triples tables.
+
+        Fresh palaces get ``source_drawer_id`` / ``adapter_name`` (RFC 002 §5.5)
+        directly from the canonical ``CREATE TABLE`` above, so this path is a
+        no-op on new installs. It exists for palaces that were created before
+        those columns were added: SQLite has no ``ADD COLUMN IF NOT EXISTS``,
+        so we introspect the schema and only issue the ALTER when the column
+        is missing.
+        """
+        existing = {row["name"] for row in conn.execute("PRAGMA table_info(triples)")}
+        if "source_drawer_id" not in existing:
+            conn.execute("ALTER TABLE triples ADD COLUMN source_drawer_id TEXT")
+        if "adapter_name" not in existing:
+            conn.execute("ALTER TABLE triples ADD COLUMN adapter_name TEXT")
+
     def _conn(self):
         if self._connection is None:
             self._connection = sqlite3.connect(self.db_path, timeout=10, check_same_thread=False)
@@ -137,10 +156,16 @@ class KnowledgeGraph:
         confidence: float = 1.0,
         source_closet: str = None,
         source_file: str = None,
+        source_drawer_id: str = None,
+        adapter_name: str = None,
     ):
         """
         Add a relationship triple: subject → predicate → object.
 
+        ``source_drawer_id`` and ``adapter_name`` are RFC 002 §5.5 provenance
+        fields populated by adapters that advertise ``supports_kg_triples``;
+        they default to ``None`` so every existing caller stays source-compatible.
+
         Examples:
             add_triple("Max", "child_of", "Alice", valid_from="2015-04-01")
             add_triple("Max", "does", "swimming", valid_from="2025-01-01")
@@ -173,8 +198,12 @@ class KnowledgeGraph:
                 triple_id = f"t_{sub_id}_{pred}_{obj_id}_{hashlib.sha256(f'{valid_from}{datetime.now().isoformat()}'.encode()).hexdigest()[:12]}"
 
                 conn.execute(
-                    """INSERT INTO triples (id, subject, predicate, object, valid_from, valid_to, confidence, source_closet, source_file)
-                       VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)""",
+                    """INSERT INTO triples (
+                        id, subject, predicate, object,
+                        valid_from, valid_to, confidence,
+                        source_closet, source_file,
+                        source_drawer_id, adapter_name
+                    ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
                     (
                         triple_id,
                         sub_id,
@@ -185,6 +214,8 @@ class KnowledgeGraph:
                         confidence,
                         source_closet,
                         source_file,
+                        source_drawer_id,
+                        adapter_name,
                     ),
                 )
         return triple_id

mempalace/layers.py

@@ -281,6 +281,8 @@ class Layer3:
 
         lines = [f'## L3 — SEARCH RESULTS for "{query}"']
         for i, (doc, meta, dist) in enumerate(zip(docs, metas, dists), 1):
+            meta = meta or {}
+            doc = doc or ""
             similarity = round(1 - dist, 3)
             wing_name = meta.get("wing", "?")
             room_name = meta.get("room", "?")
@@ -327,6 +329,13 @@ class Layer3:
             _first_or_empty(results, "metadatas"),
             _first_or_empty(results, "distances"),
         ):
+            # ChromaDB may return None for doc/meta when a drawer's HNSW entry
+            # exists but its metadata/document rows haven't been materialized
+            # (partial-flush states, mid-delete, schema upgrade boundaries).
+            # Degrade gracefully — the hit still appears with real distance;
+            # storage fields show their fallback where content is missing.
+            meta = meta or {}
+            doc = doc or ""
             hits.append(
                 {
                     "text": doc,

mempalace/mcp_server.py

@@ -315,6 +315,7 @@ def tool_status():
     try:
         all_meta = _get_cached_metadata(col)
         for m in all_meta:
+            m = m or {}
             w = m.get("wing", "unknown")
             r = m.get("room", "unknown")
             wings[w] = wings.get(w, 0) + 1
@@ -368,6 +369,7 @@ def tool_list_wings():
     try:
         all_meta = _get_cached_metadata(col)
         for m in all_meta:
+            m = m or {}
             w = m.get("wing", "unknown")
             wings[w] = wings.get(w, 0) + 1
     except Exception as e:
@@ -391,6 +393,7 @@ def tool_list_rooms(wing: str = None):
         where = {"wing": wing} if wing else None
         all_meta = _fetch_all_metadata(col, where=where)
         for m in all_meta:
+            m = m or {}
             r = m.get("room", "unknown")
             rooms[r] = rooms.get(r, 0) + 1
     except Exception as e:
@@ -409,6 +412,7 @@ def tool_get_taxonomy():
     try:
         all_meta = _get_cached_metadata(col)
         for m in all_meta:
+            m = m or {}
             w = m.get("wing", "unknown")
             r = m.get("room", "unknown")
             if w not in taxonomy:

mempalace/migrate.py

@@ -7,8 +7,10 @@ Reads documents and metadata directly from the palace's SQLite database
 then re-imports everything into a fresh palace using the currently installed
 ChromaDB version.
 
-This fixes the 3.0.0 → 3.1.0 upgrade path where chromadb was downgraded
-from 1.5.x to 0.6.x, breaking the on-disk storage format.
+Since mempalace 3.2.0 (chromadb>=1.5.4), chromadb automatically migrates
+0.4.1+ databases on first open — no manual migration needed for upgrades.
+Use this command only when downgrading chromadb (e.g. rolling back to an
+older mempalace release) or if automatic migration fails.
 
 Usage:
     mempalace migrate                          # migrate default palace

mempalace/miner.py

@@ -36,6 +36,7 @@ READABLE_EXTENSIONS = {
     ".jsx",
     ".tsx",
     ".json",
+    ".jsonl",
     ".yaml",
     ".yml",
     ".html",
@@ -62,7 +63,14 @@ SKIP_FILENAMES = {
 CHUNK_SIZE = 800  # chars per drawer
 CHUNK_OVERLAP = 100  # overlap between chunks
 MIN_CHUNK_SIZE = 50  # skip tiny chunks
-MAX_FILE_SIZE = 10 * 1024 * 1024  # 10 MB — skip files larger than this
+MAX_FILE_SIZE = 500 * 1024 * 1024  # 500 MB — skip files larger than this.
+# Long Claude Code sessions and large transcript exports routinely exceed
+# 10 MB. The cap exists as a defensive rail against pathological binary
+# files, not as a limit on legitimate text. Per-drawer size is bounded
+# by CHUNK_SIZE, but larger sources still produce proportionally more
+# drawers and therefore more storage, embedding, and processing work —
+# and file reads are not streamed (the whole content is loaded into
+# memory before chunking), so memory use scales with source size too.
 
 
 # =============================================================================
@@ -604,7 +612,7 @@ def process_file(
     chunks = chunk_text(content, source_file)
 
     if dry_run:
-        print(f"    [DRY RUN] {filepath.name} → room:{room} ({len(chunks)} drawers)")
+        print(f"    [DRY RUN] {filepath.name} -> room:{room} ({len(chunks)} drawers)")
         return len(chunks), room
 
     # Lock this file so concurrent agents don't interleave delete+insert.
@@ -781,7 +789,7 @@ def mine(
         print("  .gitignore: DISABLED")
     if include_ignored:
         print(f"  Include: {', '.join(sorted(normalize_include_paths(include_ignored)))}")
-    print(f"{'─' * 55}\n")
+    print(f"{'-' * 55}\n")
 
     if not dry_run:
         collection = get_collection(palace_path)
@@ -811,7 +819,7 @@ def mine(
             total_drawers += drawers
             room_counts[room] += 1
             if not dry_run:
-                print(f"  ✓ [{i:4}/{len(files)}] {filepath.name[:50]:50} +{drawers}")
+                print(f"  + [{i:4}/{len(files)}] {filepath.name[:50]:50} +{drawers}")
 
     print(f"\n{'=' * 55}")
     print("  Done.")
@@ -846,6 +854,7 @@ def status(palace_path: str):
 
     wing_rooms = defaultdict(lambda: defaultdict(int))
     for m in metas:
+        m = m or {}
         wing_rooms[m.get("wing", "?")][m.get("room", "?")] += 1
 
     print(f"\n{'=' * 55}")

mempalace/searcher.py

@@ -30,15 +30,16 @@ class SearchError(Exception):
 _TOKEN_RE = re.compile(r"\w{2,}", re.UNICODE)
 
 
-def _first_or_empty(results: dict, key: str) -> list:
-    """Return the first inner list of a ChromaDB query result, or [].
+def _first_or_empty(results, key: str) -> list:
+    """Return the first inner list of a query result field, or [].
 
-    ChromaDB returns shapes like ``{"documents": [["a", "b"]], ...}`` for a
-    successful query, but ``{"documents": [], ...}`` (empty outer list) when
-    the collection is empty or the filter excludes everything. Indexing
-    ``[0]`` blindly raises IndexError in that case (issue #195).
+    Accepts both the typed :class:`QueryResult` (attribute access) and the
+    pre-typed chroma dict shape; this polymorphism is retained so test mocks
+    still work and callers mid-migration do not crash. Preserves the empty-
+    collection semantics from issue #195: when no queries returned hits, the
+    outer list may be empty and indexing ``[0]`` would raise.
     """
-    outer = results.get(key)
+    outer = getattr(results, key, None) if not isinstance(results, dict) else results.get(key)
     if not outer:
         return []
     return outer[0] or []
@@ -209,7 +210,7 @@ def _expand_with_neighbors(drawers_col, matched_doc: str, matched_meta: dict, ra
         return {"text": matched_doc, "drawer_index": chunk_idx, "total_drawers": None}
 
     indexed_docs = []
-    for doc, meta in zip(neighbors.get("documents") or [], neighbors.get("metadatas") or []):
+    for doc, meta in zip(neighbors.documents, neighbors.metadatas):
         ci = meta.get("chunk_index")
         if isinstance(ci, int):
             indexed_docs.append((ci, doc))
@@ -224,8 +225,7 @@ def _expand_with_neighbors(drawers_col, matched_doc: str, matched_meta: dict, ra
     total_drawers = None
     try:
         all_meta = drawers_col.get(where={"source_file": src}, include=["metadatas"])
-        ids = all_meta.get("ids") or []
-        total_drawers = len(ids) if ids else None
+        total_drawers = len(all_meta.ids) if all_meta.ids else None
     except Exception:
         pass
 
@@ -283,6 +283,7 @@ def search(query: str, palace_path: str, wing: str = None, room: str = None, n_r
 
     for i, (doc, meta, dist) in enumerate(zip(docs, metas, dists), 1):
         similarity = round(max(0.0, 1 - dist), 3)
+        meta = meta or {}
         source = Path(meta.get("source_file", "?")).name
         wing_name = meta.get("wing", "?")
         room_name = meta.get("room", "?")
@@ -372,6 +373,7 @@ def search_memories(
                 _first_or_empty(closet_results, "distances"),
             )
         ):
+            cmeta = cmeta or {}
             source = cmeta.get("source_file", "")
             if source and source not in closet_boost_by_source:
                 closet_boost_by_source[source] = (rank, cdist, cdoc[:200])
@@ -394,6 +396,7 @@ def search_memories(
         if max_distance > 0.0 and dist > max_distance:
             continue
 
+        meta = meta or {}
         source = meta.get("source_file", "") or ""
         boost = 0.0
         matched_via = "drawer"
@@ -451,8 +454,8 @@ def search_memories(
             )
         except Exception:
             continue
-        docs = source_drawers.get("documents") or []
-        metas_ = source_drawers.get("metadatas") or []
+        docs = source_drawers.documents
+        metas_ = source_drawers.metadatas
         if len(docs) <= 1:
             continue
 

mempalace/sources/__init__.py

@@ -0,0 +1,74 @@
+"""Source adapter subsystem (RFC 002).
+
+Public surface:
+
+* :class:`BaseSourceAdapter` — per-source read-side contract.
+* Typed records: :class:`SourceRef`, :class:`SourceItemMetadata`,
+  :class:`DrawerRecord`, :class:`RouteHint`, :class:`SourceSummary`,
+  :class:`AdapterSchema`, :class:`FieldSpec`.
+* Error classes: :class:`SourceNotFoundError`, :class:`AuthRequiredError`,
+  :class:`AdapterClosedError`, :class:`TransformationViolationError`,
+  :class:`SchemaConformanceError`.
+* Registry: :func:`register`, :func:`get_adapter`, :func:`available_adapters`,
+  :func:`resolve_adapter_for_source`.
+* :class:`PalaceContext` — facade core passes to adapters during ``ingest``.
+* :mod:`transforms` — reference implementations of the reserved §1.4
+  transformations + :func:`get_transformation` resolver.
+"""
+
+from .base import (
+    AdapterClosedError,
+    AdapterSchema,
+    AuthRequiredError,
+    BaseSourceAdapter,
+    DrawerRecord,
+    FieldSpec,
+    IngestMode,
+    IngestResult,
+    RouteHint,
+    SchemaConformanceError,
+    SourceAdapterError,
+    SourceItemMetadata,
+    SourceNotFoundError,
+    SourceRef,
+    SourceSummary,
+    TransformationViolationError,
+)
+from .context import PalaceContext, ProgressHook
+from .registry import (
+    available_adapters,
+    get_adapter,
+    get_adapter_class,
+    register,
+    reset_adapters,
+    resolve_adapter_for_source,
+    unregister,
+)
+
+__all__ = [
+    "AdapterClosedError",
+    "AdapterSchema",
+    "AuthRequiredError",
+    "BaseSourceAdapter",
+    "DrawerRecord",
+    "FieldSpec",
+    "IngestMode",
+    "IngestResult",
+    "PalaceContext",
+    "ProgressHook",
+    "RouteHint",
+    "SchemaConformanceError",
+    "SourceAdapterError",
+    "SourceItemMetadata",
+    "SourceNotFoundError",
+    "SourceRef",
+    "SourceSummary",
+    "TransformationViolationError",
+    "available_adapters",
+    "get_adapter",
+    "get_adapter_class",
+    "register",
+    "reset_adapters",
+    "resolve_adapter_for_source",
+    "unregister",
+]

mempalace/sources/base.py

@@ -0,0 +1,245 @@
+"""Source adapter contract for MemPalace (RFC 002).
+
+Mirrors what ``mempalace/backends/base.py`` does for the write side: it defines
+the read-side surface every source adapter must implement. A source adapter
+extracts content from a specific origin (filesystem, git, Slack, Cursor …) and
+yields typed records (``SourceItemMetadata`` / ``DrawerRecord``) that core
+routes into the palace.
+
+This module is spec scaffolding. The first-party miners (``mempalace/miner.py``
+and ``mempalace/convo_miner.py``) are migrated onto it in a follow-up PR;
+in this PR we publish the contract so third-party adapters can begin building
+against a stable surface.
+
+See ``docs/rfcs/002-source-adapter-plugin-spec.md`` for the authoritative
+spec text.
+"""
+
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from dataclasses import dataclass, field
+from typing import TYPE_CHECKING, ClassVar, Iterator, Literal, Optional
+
+if TYPE_CHECKING:
+    from .context import PalaceContext  # noqa: F401  (used in string annotation)
+
+
+# ---------------------------------------------------------------------------
+# Errors
+# ---------------------------------------------------------------------------
+
+
+class SourceAdapterError(Exception):
+    """Base class for every source-adapter error raised by core."""
+
+
+class SourceNotFoundError(SourceAdapterError):
+    """Raised when a ``SourceRef`` does not resolve to a readable source."""
+
+
+class AuthRequiredError(SourceAdapterError):
+    """Raised when an adapter needs credentials that were not provided.
+
+    The message MUST name the env vars (or other supported mechanism) the
+    operator needs to set.
+    """
+
+
+class AdapterClosedError(SourceAdapterError):
+    """Raised when an adapter method is called after ``close()``."""
+
+
+class TransformationViolationError(SourceAdapterError):
+    """Raised by the conformance suite when round-tripping a drawer requires
+    an undeclared transformation (RFC 002 §7.2–7.3)."""
+
+
+class SchemaConformanceError(SourceAdapterError):
+    """Raised when a ``DrawerRecord.metadata`` violates the adapter schema
+    returned by :meth:`BaseSourceAdapter.describe_schema`."""
+
+
+# ---------------------------------------------------------------------------
+# Value objects
+# ---------------------------------------------------------------------------
+
+
+@dataclass(frozen=True)
+class SourceRef:
+    """A handle to the source a user wants to ingest.
+
+    ``local_path`` is for filesystem-rooted sources (project dir, mbox file).
+    ``uri`` is for URL-like references (``github.com/org/repo``,
+    ``slack://workspace/channel``).
+    ``options`` carries adapter-specific non-secret config. Secrets MUST NOT
+    be placed here; see §4.2.
+    """
+
+    local_path: Optional[str] = None
+    uri: Optional[str] = None
+    options: dict = field(default_factory=dict)
+
+
+@dataclass(frozen=True)
+class RouteHint:
+    """Adapter-supplied routing hint (RFC 002 §2.5)."""
+
+    wing: Optional[str] = None
+    room: Optional[str] = None
+    hall: Optional[str] = None
+
+
+@dataclass(frozen=True)
+class SourceItemMetadata:
+    """Lightweight pointer yielded before drawers for lazy-fetch adapters.
+
+    Core inspects ``version`` via :meth:`BaseSourceAdapter.is_current` to
+    decide whether to skip extraction; an adapter that responds positively
+    stops yielding drawers for this item and moves to the next.
+    """
+
+    source_file: str
+    version: str
+    size_hint: Optional[int] = None
+    route_hint: Optional[RouteHint] = None
+
+
+@dataclass(frozen=True)
+class DrawerRecord:
+    """One drawer's worth of extracted content plus flat metadata.
+
+    ``metadata`` values MUST be flat scalars (``str``/``int``/``float``/``bool``)
+    per RFC 001 §1.4 — the chroma constraint. Nested data belongs on the
+    knowledge graph (§5.5) or in a declared ``json_string`` field (§5.4).
+    """
+
+    content: str
+    source_file: str
+    chunk_index: int = 0
+    metadata: dict = field(default_factory=dict)
+    route_hint: Optional[RouteHint] = None
+
+
+@dataclass(frozen=True)
+class SourceSummary:
+    """High-level description of a source returned by :meth:`source_summary`."""
+
+    description: str
+    item_count: Optional[int] = None
+
+
+IngestMode = Literal["chunked_content", "whole_record", "metadata_only"]
+
+
+@dataclass(frozen=True)
+class FieldSpec:
+    """Declared shape of a single per-adapter metadata field (§5.2)."""
+
+    type: Literal["string", "int", "float", "bool", "delimiter_joined_string", "json_string"]
+    required: bool
+    description: str
+    indexed: bool = False
+    delimiter: str = ";"
+    json_schema: Optional[dict] = None
+
+
+@dataclass(frozen=True)
+class AdapterSchema:
+    """The per-adapter metadata schema returned by :meth:`describe_schema`."""
+
+    fields: dict[str, FieldSpec]
+    version: str
+
+
+# The union type adapters yield from ``ingest``.
+IngestResult = object  # intentionally broad; runtime checks in core
+
+
+# ---------------------------------------------------------------------------
+# Adapter contract
+# ---------------------------------------------------------------------------
+
+
+class BaseSourceAdapter(ABC):
+    """Long-lived adapter serving many ``SourceRef`` invocations (RFC 002 §2).
+
+    Instances are lightweight on construction — no I/O, no network, no
+    credential fetch. All work is deferred to :meth:`ingest`. Instances are
+    thread-safe for concurrent ``ingest`` calls across different ``SourceRef``
+    values (v1 serializes within a single ``SourceRef``).
+
+    Class attributes form the adapter's identity contract:
+
+    * ``name`` — stable adapter name used for registration and drawer metadata.
+    * ``adapter_version`` — adapter's own version, independent of
+      ``spec_version``. Recorded on every drawer so re-extract workflows can
+      target drawers from a known-buggy adapter version.
+    * ``capabilities`` — free-form tokens; core inspects a documented subset.
+    * ``supported_modes`` — subset of ``chunked_content``, ``whole_record``,
+      ``metadata_only``.
+    * ``declared_transformations`` — set of transformation names the adapter
+      applies to source bytes. The empty set marks a byte-preserving adapter.
+    * ``default_privacy_class`` — privacy class level (§6) applied unless the
+      palace config overrides it.
+    """
+
+    name: ClassVar[str]
+    spec_version: ClassVar[str] = "1.0"
+    adapter_version: ClassVar[str] = "0.0.0"
+    capabilities: ClassVar[frozenset[str]] = frozenset()
+    supported_modes: ClassVar[frozenset[str]] = frozenset({"chunked_content"})
+    declared_transformations: ClassVar[frozenset[str]] = frozenset()
+    default_privacy_class: ClassVar[str] = "pii_potential"
+
+    # ------------------------------------------------------------------
+    # Required methods
+    # ------------------------------------------------------------------
+
+    @abstractmethod
+    def ingest(
+        self,
+        *,
+        source: SourceRef,
+        palace: "PalaceContext",
+    ) -> Iterator[IngestResult]:
+        """Enumerate and extract content from a source.
+
+        Yields a stream of ``SourceItemMetadata`` and ``DrawerRecord`` values.
+        Lazy adapters yield ``SourceItemMetadata`` ahead of the drawers for
+        that item so core can check :meth:`is_current` before committing to
+        the fetch. Eager adapters MAY interleave freely.
+        """
+
+    @abstractmethod
+    def describe_schema(self) -> AdapterSchema:
+        """Declare the structured metadata this adapter attaches.
+
+        The returned schema MUST be stable for a given ``adapter_version``.
+        Enterprises index on it; core uses it to validate adapter output.
+        """
+
+    # ------------------------------------------------------------------
+    # Optional methods with default implementations
+    # ------------------------------------------------------------------
+
+    def is_current(
+        self,
+        *,
+        item: SourceItemMetadata,
+        existing_metadata: Optional[dict],
+    ) -> bool:
+        """Return True if the palace already has an up-to-date copy of ``item``.
+
+        Default: always returns False (re-extract every time). Adapters
+        advertising ``supports_incremental`` MUST override.
+        """
+        return False
+
+    def source_summary(self, *, source: SourceRef) -> SourceSummary:
+        """Describe a source without extracting."""
+        return SourceSummary(description=self.name)
+
+    def close(self) -> None:
+        """Release any resources the adapter holds. Default: no-op."""
+        return None

mempalace/sources/context.py

@@ -0,0 +1,142 @@
+"""``PalaceContext`` facade passed to source adapters (RFC 002 §9).
+
+Bundles the palace-side surface an adapter needs during :meth:`ingest`:
+drawer collection, closet collection, knowledge graph, palace config, and
+progress hooks. Adapters receive a ``PalaceContext`` instance and MUST NOT
+import ``mempalace.palace`` directly — that coupling is what the facade
+exists to prevent.
+
+This module publishes the shape third-party adapters target. Core's mine
+loop will construct a concrete ``PalaceContext`` and pass it to adapters
+when the filesystem/conversations miners are migrated onto ``BaseSourceAdapter``
+in a follow-up PR; until then, no in-tree code constructs one, but the
+contract is stable.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Any, Callable, Optional, Protocol
+
+from .base import DrawerRecord
+
+
+class _CollectionLike(Protocol):
+    """Minimum of :class:`mempalace.backends.BaseCollection` adapters rely on.
+
+    Declared as a Protocol so tests and third-party adapters can substitute
+    any object with compatible method signatures without importing the
+    concrete backend. See ``mempalace/backends/base.py`` for the full surface.
+    """
+
+    def add(self, **kwargs: Any) -> None: ...
+    def upsert(self, **kwargs: Any) -> None: ...
+    def query(self, **kwargs: Any) -> Any: ...
+    def get(self, **kwargs: Any) -> Any: ...
+    def delete(self, **kwargs: Any) -> None: ...
+    def count(self) -> int: ...
+
+
+class _KnowledgeGraphLike(Protocol):
+    def add_triple(self, subject: str, predicate: str, obj: str, **kwargs: Any) -> Any: ...
+
+
+# Progress hook signature: ``fn(event_name, **details) -> None``.
+ProgressHook = Callable[..., None]
+
+
+@dataclass
+class PalaceContext:
+    """Per-mine-invocation facade passed to :meth:`BaseSourceAdapter.ingest`.
+
+    Fields:
+        drawer_collection: The palace's drawer collection (via RFC 001 backend).
+        closet_collection: The palace's closet collection, or ``None`` if the
+            palace has no closets yet. Adapters should not write to this
+            directly; core builds closets post-step (RFC 002 §1.7).
+        knowledge_graph: The palace's SQLite knowledge graph. Adapters
+            advertising ``supports_kg_triples`` call ``add_triple`` on it.
+        palace_path: Filesystem root of the palace (convenience; same as
+            ``backend.PalaceRef.local_path``).
+        config: Palace config object (hall keywords, rooms list, privacy
+            floor, etc.). Shape is the existing :class:`MempalaceConfig`.
+        adapter_name: Name of the adapter currently ingesting; populated by
+            core so drawers can carry ``metadata["adapter_name"]``.
+        adapter_version: Version of the adapter currently ingesting.
+        progress_hooks: Optional callables core invokes on progress events.
+
+    Methods are intentionally thin wrappers so the concrete mine loop in
+    core can swap implementations without changing adapter code.
+    """
+
+    drawer_collection: _CollectionLike
+    knowledge_graph: _KnowledgeGraphLike
+    palace_path: str
+    closet_collection: Optional[_CollectionLike] = None
+    config: Optional[Any] = None
+    adapter_name: str = ""
+    adapter_version: str = ""
+    progress_hooks: list[ProgressHook] = field(default_factory=list)
+
+    # Internal: flag set by :meth:`skip_current_item` and checked by the core
+    # mine loop between yields. Not part of the adapter-facing contract; the
+    # adapter only needs to know that calling :meth:`skip_current_item` stops
+    # drawer emission for the current ``SourceItemMetadata``.
+    _skip_requested: bool = False
+
+    # ------------------------------------------------------------------
+    # Adapter-facing surface
+    # ------------------------------------------------------------------
+
+    def upsert_drawer(self, record: DrawerRecord) -> None:
+        """Persist a ``DrawerRecord`` to the drawer collection.
+
+        Applies the spec-mandated ``adapter_name`` and ``adapter_version``
+        metadata stamps (§5.1) so adapters never need to populate them.
+        """
+        meta = dict(record.metadata)
+        meta.setdefault("source_file", record.source_file)
+        meta.setdefault("chunk_index", record.chunk_index)
+        if self.adapter_name:
+            meta.setdefault("adapter_name", self.adapter_name)
+        if self.adapter_version:
+            meta.setdefault("adapter_version", self.adapter_version)
+        drawer_id = _build_drawer_id(record)
+        self.drawer_collection.upsert(
+            documents=[record.content],
+            ids=[drawer_id],
+            metadatas=[meta],
+        )
+
+    def skip_current_item(self) -> None:
+        """Signal to core that the current ``SourceItemMetadata`` is up-to-date
+        and no drawers should be emitted for it. Core resets the flag after
+        advancing past the item."""
+        self._skip_requested = True
+
+    def emit(self, event: str, **details: Any) -> None:
+        """Invoke each registered progress hook with ``(event, **details)``."""
+        for hook in self.progress_hooks:
+            try:
+                hook(event, **details)
+            except Exception:  # pragma: no cover - hook errors never fail mine
+                import logging
+
+                logging.getLogger(__name__).exception("progress hook failed on %r", event)
+
+
+def _build_drawer_id(record: DrawerRecord) -> str:
+    """Deterministic drawer id: ``<sha256(source_file)[:24]>_<chunk_index>``.
+
+    Matches the shape existing miners rely on (``source_file`` + chunk index
+    pair) while keeping the id chroma-safe (no separators that collide with
+    existing metadata values). 96-bit SHA-256 prefix keeps collision risk
+    negligible across corpora the size of a palace (sha1@64 bits was too
+    close to the birthday bound for large ingests). Adapters that need a
+    different id scheme can bypass :meth:`PalaceContext.upsert_drawer` and
+    write through ``drawer_collection.upsert`` directly.
+    """
+    import hashlib
+
+    digest = hashlib.sha256(record.source_file.encode("utf-8")).hexdigest()[:24]
+    return f"{digest}_{record.chunk_index}"

mempalace/sources/registry.py

@@ -0,0 +1,162 @@
+"""Source adapter registry + entry-point discovery (RFC 002 §3).
+
+Third-party adapters ship as installable packages that declare a
+``mempalace.sources`` entry point::
+
+    # pyproject.toml of mempalace-source-cursor
+    [project.entry-points."mempalace.sources"]
+    cursor = "mempalace_source_cursor:CursorAdapter"
+
+MemPalace discovers them at process start. In-tree tests and local
+development can register manually via :func:`register`. Explicit
+registration wins on name conflict (RFC 002 §3.2).
+
+Unlike storage backends (RFC 001 §3.3), source adapters are never auto-
+detected — the user selects the adapter explicitly via ``--source NAME``
+or config (§3.3). The default when no adapter is named is ``filesystem``
+(to preserve current ``mempalace mine <path>`` behavior).
+"""
+
+from __future__ import annotations
+
+import logging
+from importlib import metadata
+from threading import Lock
+from typing import Type
+
+from .base import BaseSourceAdapter
+
+logger = logging.getLogger(__name__)
+
+_ENTRY_POINT_GROUP = "mempalace.sources"
+_DEFAULT_ADAPTER = "filesystem"
+
+_registry: dict[str, Type[BaseSourceAdapter]] = {}
+_instances: dict[str, BaseSourceAdapter] = {}
+_explicit: set[str] = set()
+_discovered = False
+_lock = Lock()
+
+
+def register(name: str, adapter_cls: Type[BaseSourceAdapter]) -> None:
+    """Register ``adapter_cls`` under ``name``.
+
+    Explicit registration wins over entry-point discovery on conflict (§3.2).
+    """
+    with _lock:
+        _registry[name] = adapter_cls
+        _explicit.add(name)
+        _instances.pop(name, None)
+
+
+def unregister(name: str) -> None:
+    """Remove an adapter registration (primarily for tests)."""
+    with _lock:
+        _registry.pop(name, None)
+        _explicit.discard(name)
+        _instances.pop(name, None)
+
+
+def _discover_entry_points() -> None:
+    global _discovered
+    if _discovered:
+        return
+    with _lock:
+        if _discovered:
+            return
+        try:
+            eps = metadata.entry_points()
+            group = (
+                eps.select(group=_ENTRY_POINT_GROUP)
+                if hasattr(eps, "select")
+                else eps.get(_ENTRY_POINT_GROUP, [])
+            )
+        except Exception:
+            logger.exception("entry-point discovery for %s failed", _ENTRY_POINT_GROUP)
+            group = []
+        for ep in group:
+            if ep.name in _explicit:
+                continue  # explicit registration wins
+            try:
+                cls = ep.load()
+            except Exception:
+                logger.exception("failed to load adapter entry point %r", ep.name)
+                continue
+            if not isinstance(cls, type) or not issubclass(cls, BaseSourceAdapter):
+                logger.warning(
+                    "entry point %r did not resolve to a BaseSourceAdapter subclass (got %r)",
+                    ep.name,
+                    cls,
+                )
+                continue
+            _registry.setdefault(ep.name, cls)
+        _discovered = True
+
+
+def available_adapters() -> list[str]:
+    """Return sorted list of all registered adapter names."""
+    _discover_entry_points()
+    return sorted(_registry.keys())
+
+
+def get_adapter_class(name: str) -> Type[BaseSourceAdapter]:
+    """Return the registered adapter class for ``name``."""
+    _discover_entry_points()
+    try:
+        return _registry[name]
+    except KeyError as e:
+        raise KeyError(f"unknown source adapter {name!r}; available: {available_adapters()}") from e
+
+
+def get_adapter(name: str) -> BaseSourceAdapter:
+    """Return a long-lived instance of the named adapter.
+
+    Instances are cached per-name; repeated calls return the same object.
+    Call :func:`reset_adapters` in tests that need isolation.
+    """
+    _discover_entry_points()
+    with _lock:
+        inst = _instances.get(name)
+        if inst is not None:
+            return inst
+        cls = _registry.get(name)
+        if cls is None:
+            raise KeyError(
+                f"unknown source adapter {name!r}; available: {sorted(_registry.keys())}"
+            )
+        inst = cls()
+        _instances[name] = inst
+        return inst
+
+
+def reset_adapters() -> None:
+    """Close and drop all cached adapter instances (primarily for tests)."""
+    with _lock:
+        for inst in _instances.values():
+            try:
+                inst.close()
+            except Exception:
+                logger.exception("error closing adapter during reset")
+        _instances.clear()
+
+
+def resolve_adapter_for_source(
+    *,
+    explicit: str | None = None,
+    config_value: str | None = None,
+    default: str = _DEFAULT_ADAPTER,
+) -> str:
+    """Resolve the adapter name per RFC 002 §3.3 priority order.
+
+    1. Explicit ``--source`` flag or kwarg
+    2. Per-source config value
+    3. Default (``filesystem``)
+
+    Auto-detection is *intentionally* absent on the read side (§3.3); a
+    directory containing ``.git`` + ``workspaceStorage/`` + an ``mbox`` file
+    is not a signal of user intent.
+    """
+    for candidate in (explicit, config_value):
+        if candidate:
+            return candidate
+    return default

mempalace/sources/transforms.py

@@ -0,0 +1,196 @@
+"""Reference implementations of the reserved content transformations (RFC 002 §1.4).
+
+Every source adapter declares the set of transformations it applies to source
+bytes via ``declared_transformations``. The conformance suite then verifies
+that the adapter's output can be reproduced from the source bytes by applying
+*only* the declared transformations in declaration order, using these
+reference implementations.
+
+Each transformation is a pure function on strings (text content after UTF-8
+decoding). ``utf8_replace_invalid`` is the one that operates on bytes.
+
+The invariant the spec enforces: **no transformation is applied that is not
+declared in the adapter's set**. Adapters with an empty set are byte-preserving
+end-to-end (modulo the initial UTF-8 decode itself, which is captured by
+``utf8_replace_invalid`` when applicable).
+
+Adapters MAY add custom transformations beyond the reserved set; third-party
+names SHOULD be prefixed with the adapter name (``cursor.composer_ordering``).
+Custom transformations MUST expose a reference implementation under
+``mempalace.sources.transforms.<adapter_name>_<transform_name>`` so the
+conformance suite can locate and apply them.
+"""
+
+from __future__ import annotations
+
+import re
+from typing import Protocol, Union
+
+
+class Transformation(Protocol):
+    """Callable signature every reserved transformation conforms to.
+
+    Accepts the current stage of the pipeline — ``bytes`` on input
+    (``utf8_replace_invalid``) or ``str`` after decoding — and returns ``str``.
+    Adapters compose them in declaration order; the first step operates on the
+    original source bytes, every subsequent step on the prior step's output.
+    """
+
+    def __call__(self, data: Union[bytes, str], /) -> str: ...
+
+
+# ---------------------------------------------------------------------------
+# Reserved transformations
+# ---------------------------------------------------------------------------
+
+
+def utf8_replace_invalid(raw: bytes) -> str:
+    """Decode bytes as UTF-8; replace invalid sequences with U+FFFD.
+
+    Equivalent to ``raw.decode("utf-8", errors="replace")``. This is the one
+    reserved transformation that operates on bytes rather than decoded text.
+    """
+    return raw.decode("utf-8", errors="replace")
+
+
+def newline_normalize(text: str) -> str:
+    """Convert CRLF and bare-CR line endings to LF."""
+    return text.replace("\r\n", "\n").replace("\r", "\n")
+
+
+def whitespace_trim(text: str) -> str:
+    """Strip leading and trailing whitespace at the record boundary only."""
+    return text.strip()
+
+
+_RUN_OF_THREE_OR_MORE_BLANK = re.compile(r"(?:\n[ \t]*){3,}\n")
+
+
+def whitespace_collapse_internal(text: str) -> str:
+    """Collapse runs of three or more blank lines to exactly two blank lines.
+
+    A "blank line" here is a line containing only spaces or tabs. Single and
+    double blank-line runs are preserved.
+    """
+    # Normalise inputs before collapsing: turn internal blank lines with
+    # whitespace content into pure \n so the regex matches consistently.
+    lines = text.split("\n")
+    normalised = "\n".join(line if line.strip() else "" for line in lines)
+    return _RUN_OF_THREE_OR_MORE_BLANK.sub("\n\n\n", normalised)
+
+
+def line_trim(text: str) -> str:
+    """Strip leading and trailing whitespace from each individual line."""
+    return "\n".join(line.strip() for line in text.split("\n"))
+
+
+def line_join_spaces(text: str) -> str:
+    """Join adjacent non-blank lines with a single space, preserving paragraph breaks.
+
+    Two lines separated by at least one blank line remain on separate lines;
+    runs of non-blank lines collapse into a single space-separated line.
+    """
+    paragraphs = re.split(r"\n[ \t]*\n", text)
+    joined = [" ".join(line.strip() for line in p.split("\n") if line.strip()) for p in paragraphs]
+    return "\n\n".join(joined)
+
+
+def blank_line_drop(text: str) -> str:
+    """Drop blank lines between non-blank lines, keeping non-blank lines only."""
+    return "\n".join(line for line in text.split("\n") if line.strip())
+
+
+# The following reserved transformations are declared in the spec but are
+# deeply adapter-specific. Rather than guess a single reference implementation
+# now, we provide identity shims that leave the input unchanged when no
+# adapter-specific implementation is available. Adapters that declare these
+# MUST either override with a concrete implementation or provide a namespaced
+# reference under
+# ``mempalace.sources.transforms.<adapter_name>_<transform_name>`` (per the
+# module docstring). The conformance suite looks up the adapter-specific
+# implementation first, falling back to these identity shims only when none
+# exists.
+
+
+def strip_tool_chrome(text: str) -> str:
+    """Adapter-supplied: remove system tags, hook output, tool UI chrome.
+
+    The reference implementation here is intentionally an identity function
+    because the noise patterns differ per transcript format (Claude Code,
+    Codex, ChatGPT, Slack). The conversations adapter, when migrated, will
+    register a concrete reference implementation under
+    ``mempalace.sources.transforms.conversations_strip_tool_chrome``.
+    """
+    return text
+
+
+def tool_result_truncate(text: str) -> str:
+    """Adapter-supplied: head/tail window on tool output with a middle marker."""
+    return text
+
+
+def tool_result_omitted(text: str) -> str:
+    """Adapter-supplied: fully omit some tool outputs (e.g., Read/Edit/Write)."""
+    return text
+
+
+def spellcheck_user(text: str) -> str:
+    """Adapter-supplied: rewrite user turns via autocorrect.
+
+    Requires the optional ``spellcheck`` extra and a tokenizer; the spec does
+    not mandate a specific language model, so the reference is adapter-owned.
+    """
+    return text
+
+
+def synthesized_marker(text: str) -> str:
+    """Adapter-supplied: adapter inserts its own strings (e.g., '[N lines omitted]')."""
+    return text
+
+
+def speaker_role_assignment(text: str) -> str:
+    """Adapter-supplied: multi-party speakers alternately assigned user/assistant."""
+    return text
+
+
+# ---------------------------------------------------------------------------
+# Registry
+# ---------------------------------------------------------------------------
+
+
+# Reserved transformation name → reference implementation.
+# Adapters look up by name to compose a round-trip pipeline during testing.
+# The value conforms to the :class:`Transformation` protocol above; we type
+# it as that Protocol rather than a concrete ``Callable`` so static checkers
+# accept both the bytes→str (``utf8_replace_invalid``) and str→str shapes.
+RESERVED_TRANSFORMATIONS: dict[str, Transformation] = {
+    "utf8_replace_invalid": utf8_replace_invalid,
+    "newline_normalize": newline_normalize,
+    "whitespace_trim": whitespace_trim,
+    "whitespace_collapse_internal": whitespace_collapse_internal,
+    "line_trim": line_trim,
+    "line_join_spaces": line_join_spaces,
+    "blank_line_drop": blank_line_drop,
+    "strip_tool_chrome": strip_tool_chrome,
+    "tool_result_truncate": tool_result_truncate,
+    "tool_result_omitted": tool_result_omitted,
+    "spellcheck_user": spellcheck_user,
+    "synthesized_marker": synthesized_marker,
+    "speaker_role_assignment": speaker_role_assignment,
+}
+
+
+def get_transformation(name: str) -> Transformation:
+    """Resolve a reserved transformation by name.
+
+    Raises :class:`KeyError` if the name is neither reserved nor registered as
+    an adapter-namespaced reference (``<adapter>_<transform>``). Callers
+    looking for adapter-specific references SHOULD ``getattr`` on this module
+    first; this helper only covers the reserved names.
+    """
+    try:
+        return RESERVED_TRANSFORMATIONS[name]
+    except KeyError as e:
+        raise KeyError(
+            f"unknown transformation {name!r}; reserved names: {sorted(RESERVED_TRANSFORMATIONS)}"
+        ) from e

mempalace/split_mega_files.py

@@ -224,7 +224,7 @@ def split_file(filepath, output_dir, dry_run=False):
             print(f"  [{i + 1}/{len(boundaries) - 1}] {name}  ({len(chunk)} lines)")
         else:
             out_path.write_text("".join(chunk), encoding="utf-8")
-            print(f"  ✓ {name}  ({len(chunk)} lines)")
+            print(f"  + {name}  ({len(chunk)} lines)")
 
         written.append(out_path)
 
@@ -290,7 +290,7 @@ def main():
     print(f"  Source:      {src_dir}")
     print(f"  Output:      {output_dir or 'same dir as source'}")
     print(f"  Mega-files:  {len(mega_files)}")
-    print(f"{'─' * 60}\n")
+    print(f"{'-' * 60}\n")
 
     total_written = 0
     for f, n_sessions in mega_files:
@@ -301,11 +301,11 @@ def main():
         if not args.dry_run and written:
             backup = f.with_suffix(".mega_backup")
             f.rename(backup)
-            print(f"  → Original renamed to {backup.name}\n")
+            print(f"  -> Original renamed to {backup.name}\n")
         else:
             print()
 
-    print(f"{'─' * 60}")
+    print(f"{'-' * 60}")
     if args.dry_run:
         print(f"  DRY RUN — would create {total_written} files from {len(mega_files)} mega-files")
     else:

mempalace/sweeper.py

@@ -0,0 +1,347 @@
+#!/usr/bin/env python3
+"""
+sweeper.py — Message-granular miner that catches what the file-level
+primary miners dropped.
+
+Algorithm, per session:
+
+    cursor = max(timestamp of sweeper-written drawers for this session_id)
+    For each user/assistant message in the jsonl:
+        if cursor is not None and message.timestamp < cursor: skip
+        else: upsert a drawer keyed by (session_id, message_uuid)
+
+Properties:
+
+  - Idempotent on its own writes: rerunning is a no-op because drawer
+    IDs are deterministic and existence is pre-checked before counting.
+  - Resume-safe: a crash mid-sweep is recovered on the next run — the
+    cursor advances to the last ingested timestamp and re-attempts at
+    that boundary are de-duped by the deterministic ID.
+  - Tie-break safe: uses ``< cursor`` (not ``<=``), so if multiple
+    messages share the max timestamp and only some were ingested, the
+    rest are still picked up on re-run.
+  - No size caps: each drawer holds one exchange, ~1-5 KB.
+
+Coordination with the primary file-level miners (``miner.py`` /
+``convo_miner.py``) is limited: those miners chunk at a fixed char size
+and do not currently stamp ``session_id``/``timestamp`` metadata that
+the sweeper can key off. In practice the sweeper coordinates with its
+own prior runs, and may ingest content that also got chunked into
+primary-miner drawers (under different IDs). Follow-up: add uniform
+``ingest_mode`` + message metadata to the primary miners so dedup spans
+both paths.
+
+Usage:
+    from mempalace.sweeper import sweep
+    result = sweep("/path/to/session.jsonl", "/path/to/palace")
+"""
+
+from __future__ import annotations
+
+import json
+import logging
+import sys
+from datetime import datetime
+from pathlib import Path
+from typing import Iterator, Optional
+
+from .palace import get_collection
+
+logger = logging.getLogger(__name__)
+
+
+# ── JSONL parsing ────────────────────────────────────────────────────
+
+
+def _flatten_content(content) -> str:
+    """Normalize Claude Code's message content to a plain string.
+
+    User messages are strings already; assistant messages are a list of
+    content blocks like [{"type": "text", "text": "..."}, {"type":
+    "tool_use", ...}]. All blocks are preserved verbatim — the design
+    principle is "verbatim always", so tool inputs and results are
+    serialized in full, never truncated.
+    """
+    if isinstance(content, str):
+        return content
+    if isinstance(content, list):
+        parts = []
+        for block in content:
+            if not isinstance(block, dict):
+                continue
+            btype = block.get("type", "")
+            if btype == "text":
+                parts.append(block.get("text", ""))
+            elif btype == "tool_use":
+                parts.append(
+                    f"[tool_use: {block.get('name', '?')} "
+                    f"input={json.dumps(block.get('input', {}), default=str)}]"
+                )
+            elif btype == "tool_result":
+                parts.append(f"[tool_result: {json.dumps(block.get('content', ''), default=str)}]")
+            else:
+                parts.append(f"[{btype}: {json.dumps(block, default=str)}]")
+        return "\n".join(p for p in parts if p)
+    return str(content)
+
+
+def parse_claude_jsonl(path: str) -> Iterator[dict]:
+    """Yield user/assistant records from a Claude Code .jsonl file.
+
+    Each yield is:
+        {
+          "session_id": str,
+          "uuid":       str,   # per-message UUID
+          "timestamp":  str,   # ISO 8601
+          "role":       "user" | "assistant",
+          "content":    str,   # flattened text
+        }
+
+    Non-message records (progress, file-history-snapshot, system,
+    queue-operation, last-prompt) are filtered out. Malformed lines are
+    skipped silently — data quality is the transcript writer's problem,
+    not ours.
+    """
+    with open(path, "r", encoding="utf-8", errors="replace") as f:
+        for line in f:
+            line = line.strip()
+            if not line:
+                continue
+            try:
+                record = json.loads(line)
+            except json.JSONDecodeError:
+                continue
+            rtype = record.get("type")
+            if rtype not in ("user", "assistant"):
+                continue
+            msg = record.get("message") or {}
+            if not isinstance(msg, dict):
+                continue
+            role = msg.get("role")
+            if role not in ("user", "assistant"):
+                continue
+            timestamp = record.get("timestamp")
+            if not timestamp:
+                continue
+            uuid = record.get("uuid")
+            if not uuid:
+                continue
+            session_id = record.get("sessionId") or record.get("session_id")
+            if not session_id:
+                continue
+            content = _flatten_content(msg.get("content", ""))
+            if not content.strip():
+                continue
+            yield {
+                "session_id": session_id,
+                "uuid": uuid,
+                "timestamp": timestamp,
+                "role": role,
+                "content": content,
+            }
+
+
+# ── Cursor resolution ────────────────────────────────────────────────
+
+
+def get_palace_cursor(collection, session_id: str) -> Optional[str]:
+    """Return the max timestamp of drawers for this session_id, or None.
+
+    ISO-8601 strings compare lexically in the right order, so we don't
+    need to parse them. Query scans metadatas for the session via the
+    backend's where-filter, then reduces.
+
+    Backend errors are logged at WARNING and surface as a `None` cursor —
+    which makes the caller treat the session as empty and ingest every
+    message. That's intentional: a no-cursor sweep is recovered from on
+    the next run by deterministic drawer IDs, so a degraded cursor never
+    causes silent data loss.
+    """
+    try:
+        data = collection.get(
+            where={"session_id": session_id},
+            include=["metadatas"],
+        )
+    except Exception as exc:
+        logger.warning(
+            "sweeper: cursor lookup failed for session_id=%s (%s); "
+            "treating as empty — drawers will be re-upserted idempotently.",
+            session_id,
+            exc,
+        )
+        return None
+    metas = data.get("metadatas") or []
+    timestamps = [m.get("timestamp") for m in metas if m and m.get("timestamp")]
+    if not timestamps:
+        return None
+    return max(timestamps)
+
+
+# ── Sweep ────────────────────────────────────────────────────────────
+
+
+def _drawer_id_for_message(session_id: str, message_uuid: str) -> str:
+    """Deterministic drawer ID so upserts at the same message are no-ops.
+
+    Uses the full session_id (not a prefix) to avoid any cross-session
+    collision risk if a transcript source ever uses non-UUID session
+    identifiers or shares prefixes across sessions.
+    """
+    return f"sweep_{session_id}_{message_uuid}"
+
+
+def sweep(jsonl_path: str, palace_path: str, source_label: Optional[str] = None) -> dict:
+    """Ingest every user/assistant message not already represented.
+
+    For each message in the jsonl:
+      - If timestamp < cursor for that session, skip (strictly earlier
+        than anything already in the palace — already covered).
+      - At timestamp == cursor we do NOT skip, because multiple messages
+        can share the same ISO-8601 timestamp; if only some of them were
+        ingested before a crash, a `<= cursor` skip would lose the rest
+        forever. Deterministic drawer IDs make re-attempting at the
+        cursor boundary safe (existing rows are found via a pre-flight
+        `get(ids=...)` and counted as "already present", not "added").
+      - Else, upsert a drawer with deterministic ID so reruns dedupe.
+
+    Returns ``{drawers_added, drawers_already_present, drawers_skipped,
+    drawers_upserted, cursor_by_session}``:
+
+    * ``drawers_added`` — rows that did not exist before this sweep.
+    * ``drawers_already_present`` — rows whose deterministic ID was
+      already in the palace and got rewritten idempotently.
+    * ``drawers_skipped`` — records skipped by the cursor (strictly
+      earlier than what's already stored).
+    * ``drawers_upserted`` — total writes = added + already_present.
+    """
+    collection = get_collection(palace_path, create=True)
+    cursors: dict = {}
+
+    drawers_added = 0
+    drawers_already_present = 0
+    drawers_skipped = 0
+
+    batch_ids: list[str] = []
+    batch_docs: list[str] = []
+    batch_metas: list[dict] = []
+    BATCH_SIZE = 64
+
+    def _flush():
+        nonlocal drawers_added, drawers_already_present
+        if not batch_ids:
+            return
+        # Pre-flight: which IDs in this batch are already present?
+        # Upsert is idempotent on data but counts as "added" would lie;
+        # this pre-query makes the metric honest (Copilot PR 998 review).
+        try:
+            existing = collection.get(ids=list(batch_ids), include=[])
+            # Chroma returns a dict; typed backends return GetResult — the
+            # compat shim makes ``.get("ids")`` work on both.
+            present = set(existing.get("ids") or [])
+        except Exception as exc:
+            logger.warning(
+                "sweeper: existence pre-check failed (%s); "
+                "counting all batch rows as new (metric may over-count on reruns).",
+                exc,
+            )
+            present = set()
+        new_count = sum(1 for rid in batch_ids if rid not in present)
+        already_count = len(batch_ids) - new_count
+
+        collection.upsert(
+            ids=batch_ids,
+            documents=batch_docs,
+            metadatas=batch_metas,
+        )
+        drawers_added += new_count
+        drawers_already_present += already_count
+        batch_ids.clear()
+        batch_docs.clear()
+        batch_metas.clear()
+
+    for rec in parse_claude_jsonl(jsonl_path):
+        sid = rec["session_id"]
+        if sid not in cursors:
+            cursors[sid] = get_palace_cursor(collection, sid)
+
+        cursor = cursors[sid]
+        if cursor is not None and rec["timestamp"] < cursor:
+            drawers_skipped += 1
+            continue
+
+        drawer_id = _drawer_id_for_message(sid, rec["uuid"])
+        document = f"{rec['role'].upper()}: {rec['content']}"
+        metadata = {
+            "session_id": sid,
+            "timestamp": rec["timestamp"],
+            "message_uuid": rec["uuid"],
+            "role": rec["role"],
+            "source_file": source_label or jsonl_path,
+            "filed_at": datetime.now().isoformat(),
+            "ingest_mode": "sweep",
+        }
+
+        batch_ids.append(drawer_id)
+        batch_docs.append(document)
+        batch_metas.append(metadata)
+
+        if len(batch_ids) >= BATCH_SIZE:
+            _flush()
+
+    _flush()
+
+    return {
+        "drawers_added": drawers_added,
+        "drawers_already_present": drawers_already_present,
+        "drawers_upserted": drawers_added + drawers_already_present,
+        "drawers_skipped": drawers_skipped,
+        "cursor_by_session": cursors,
+    }
+
+
+def sweep_directory(dir_path: str, palace_path: str) -> dict:
+    """Sweep every .jsonl file in a directory (recursive).
+
+    Returns aggregated summary across all files. ``files_attempted``
+    includes files that raised, so the count reflects discovery rather
+    than only successes; ``files_succeeded`` is the subset that
+    completed without error.
+    """
+    dir_p = Path(dir_path).expanduser().resolve()
+    files = sorted(dir_p.rglob("*.jsonl"))
+
+    total_added = 0
+    total_already_present = 0
+    total_skipped = 0
+    per_file = []
+
+    failures: list[dict] = []
+    for f in files:
+        try:
+            result = sweep(str(f), palace_path, source_label=str(f))
+        except Exception as exc:
+            logger.error("sweeper: sweep failed on %s: %s", f, exc)
+            print(f"  WARNING: sweep failed on {f}: {exc}", file=sys.stderr)
+            failures.append({"file": str(f), "error": str(exc)})
+            continue
+        total_added += result["drawers_added"]
+        total_already_present += result.get("drawers_already_present", 0)
+        total_skipped += result["drawers_skipped"]
+        per_file.append(
+            {
+                "file": str(f),
+                "added": result["drawers_added"],
+                "already_present": result.get("drawers_already_present", 0),
+                "skipped": result["drawers_skipped"],
+            }
+        )
+
+    return {
+        "files_attempted": len(files),
+        "files_succeeded": len(per_file),
+        "drawers_added": total_added,
+        "drawers_already_present": total_already_present,
+        "drawers_skipped": total_skipped,
+        "per_file": per_file,
+        "failures": failures,
+    }

mempalace/version.py

@@ -1,3 +1,3 @@
 """Single source of truth for the MemPalace package version."""
 
-__version__ = "3.3.1"
+__version__ = "3.3.2"

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "mempalace"
-version = "3.3.1"
+version = "3.3.2"
 description = "Give your AI a memory — mine projects and conversations into a searchable palace. No API key required."
 readme = "README.md"
 requires-python = ">=3.9"
@@ -21,11 +21,13 @@ classifiers = [
     "Programming Language :: Python :: 3.10",
     "Programming Language :: Python :: 3.11",
     "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Programming Language :: Python :: 3.14",
     "Topic :: Scientific/Engineering :: Artificial Intelligence",
     "Topic :: Utilities",
 ]
 dependencies = [
-    "chromadb>=0.5.0",
+    "chromadb>=1.5.4,<2",
     "pyyaml>=6.0,<7",
 ]
 
@@ -37,6 +39,15 @@ Repository = "https://github.com/MemPalace/mempalace"
 [project.scripts]
 mempalace = "mempalace.cli:main"
 
+[project.entry-points."mempalace.backends"]
+chroma = "mempalace.backends.chroma:ChromaBackend"
+
+# RFC 002 source-adapter entry-point group. Core publishes no first-party
+# adapters under this group yet; ``miner.py`` and ``convo_miner.py`` migrate
+# onto ``BaseSourceAdapter`` in a follow-up PR. Third-party adapter packages
+# (``mempalace-source-cursor``, ``mempalace-source-git``, …) register here.
+[project.entry-points."mempalace.sources"]
+
 [project.optional-dependencies]
 dev = ["pytest>=7.0", "pytest-cov>=4.0", "ruff>=0.4.0", "psutil>=5.9"]
 spellcheck = ["autocorrect>=2.0"]

tests/test_backends.py

@@ -1,14 +1,42 @@
+import os
 import sqlite3
 
 import chromadb
 import pytest
 
-from mempalace.backends.chroma import ChromaBackend, ChromaCollection, _fix_blob_seq_ids
+from mempalace.backends import (
+    GetResult,
+    PalaceRef,
+    QueryResult,
+    UnsupportedFilterError,
+    available_backends,
+    get_backend,
+)
+from mempalace.backends.chroma import (
+    ChromaBackend,
+    ChromaCollection,
+    _fix_blob_seq_ids,
+    quarantine_stale_hnsw,
+)
 
 
 class _FakeCollection:
-    def __init__(self):
+    """Stand-in for a chromadb.Collection returning raw chroma-shaped dicts."""
+
+    def __init__(self, query_response=None, get_response=None, count_value=7):
         self.calls = []
+        self._query_response = query_response or {
+            "ids": [["a", "b"]],
+            "documents": [["da", "db"]],
+            "metadatas": [[{"wing": "w1"}, {"wing": "w2"}]],
+            "distances": [[0.1, 0.2]],
+        }
+        self._get_response = get_response or {
+            "ids": ["a"],
+            "documents": ["da"],
+            "metadatas": [{"wing": "w1"}],
+        }
+        self._count_value = count_value
 
     def add(self, **kwargs):
         self.calls.append(("add", kwargs))
@@ -16,41 +44,251 @@ class _FakeCollection:
     def upsert(self, **kwargs):
         self.calls.append(("upsert", kwargs))
 
+    def update(self, **kwargs):
+        self.calls.append(("update", kwargs))
+
     def query(self, **kwargs):
         self.calls.append(("query", kwargs))
-        return {"kind": "query"}
+        return self._query_response
 
     def get(self, **kwargs):
         self.calls.append(("get", kwargs))
-        return {"kind": "get"}
+        return self._get_response
 
     def delete(self, **kwargs):
         self.calls.append(("delete", kwargs))
 
     def count(self):
         self.calls.append(("count", {}))
-        return 7
+        return self._count_value
 
 
-def test_chroma_collection_delegates_methods():
+def test_chroma_collection_returns_typed_query_result():
+    fake = _FakeCollection()
+    collection = ChromaCollection(fake)
+
+    result = collection.query(query_texts=["q"])
+
+    assert isinstance(result, QueryResult)
+    assert result.ids == [["a", "b"]]
+    assert result.documents == [["da", "db"]]
+    assert result.metadatas == [[{"wing": "w1"}, {"wing": "w2"}]]
+    assert result.distances == [[0.1, 0.2]]
+    assert result.embeddings is None
+
+
+def test_chroma_collection_returns_typed_get_result():
+    fake = _FakeCollection()
+    collection = ChromaCollection(fake)
+
+    result = collection.get(where={"wing": "w1"})
+
+    assert isinstance(result, GetResult)
+    assert result.ids == ["a"]
+    assert result.documents == ["da"]
+    assert result.metadatas == [{"wing": "w1"}]
+
+
+def test_query_result_empty_preserves_outer_dimension():
+    empty = QueryResult.empty(num_queries=2)
+    assert empty.ids == [[], []]
+    assert empty.documents == [[], []]
+    assert empty.distances == [[], []]
+    assert empty.embeddings is None
+
+
+def test_typed_results_support_dict_compat_access():
+    """Transitional compat shim per base.py — retained until callers migrate to attrs."""
+    result = GetResult(ids=["a"], documents=["da"], metadatas=[{"w": 1}])
+    assert result["ids"] == ["a"]
+    assert result.get("documents") == ["da"]
+    assert result.get("missing", "default") == "default"
+    assert "ids" in result
+    assert "missing" not in result
+
+
+def test_chroma_collection_query_empty_result_preserves_outer_shape():
+    fake = _FakeCollection(
+        query_response={"ids": [], "documents": [], "metadatas": [], "distances": []}
+    )
+    collection = ChromaCollection(fake)
+
+    result = collection.query(query_texts=["q1", "q2"])
+    assert result.ids == [[], []]
+    assert result.documents == [[], []]
+    assert result.distances == [[], []]
+
+
+def test_chroma_collection_rejects_unknown_where_operator():
+    fake = _FakeCollection()
+    collection = ChromaCollection(fake)
+
+    with pytest.raises(UnsupportedFilterError):
+        collection.query(query_texts=["q"], where={"$regex": "foo"})
+
+
+def test_chroma_collection_delegates_writes():
     fake = _FakeCollection()
     collection = ChromaCollection(fake)
 
     collection.add(documents=["d"], ids=["1"], metadatas=[{"wing": "w"}])
     collection.upsert(documents=["u"], ids=["2"], metadatas=[{"room": "r"}])
-    assert collection.query(query_texts=["q"]) == {"kind": "query"}
-    assert collection.get(where={"wing": "w"}) == {"kind": "get"}
     collection.delete(ids=["1"])
     assert collection.count() == 7
 
-    assert fake.calls == [
-        ("add", {"documents": ["d"], "ids": ["1"], "metadatas": [{"wing": "w"}]}),
-        ("upsert", {"documents": ["u"], "ids": ["2"], "metadatas": [{"room": "r"}]}),
-        ("query", {"query_texts": ["q"]}),
-        ("get", {"where": {"wing": "w"}}),
-        ("delete", {"ids": ["1"]}),
-        ("count", {}),
-    ]
+    kinds = [call[0] for call in fake.calls]
+    assert kinds == ["add", "upsert", "delete", "count"]
+
+
+def test_registry_exposes_chroma_by_default():
+    names = available_backends()
+    assert "chroma" in names
+    assert isinstance(get_backend("chroma"), ChromaBackend)
+
+
+def test_registry_unknown_backend_raises():
+    with pytest.raises(KeyError):
+        get_backend("no-such-backend-exists")
+
+
+def test_resolve_backend_priority_order(tmp_path):
+    from mempalace.backends import resolve_backend_for_palace
+
+    # explicit kwarg wins over everything
+    assert resolve_backend_for_palace(explicit="pg", config_value="lance") == "pg"
+    # config value wins over env / default
+    assert resolve_backend_for_palace(config_value="lance", env_value="qdrant") == "lance"
+    # env wins over default
+    assert resolve_backend_for_palace(env_value="qdrant", default="chroma") == "qdrant"
+    # falls back to default
+    assert resolve_backend_for_palace() == "chroma"
+
+
+def test_chroma_detect_matches_palace_with_chroma_sqlite(tmp_path):
+    (tmp_path / "chroma.sqlite3").write_bytes(b"")
+    assert ChromaBackend.detect(str(tmp_path)) is True
+    assert ChromaBackend.detect(str(tmp_path.parent)) is False
+
+
+def test_query_rejects_missing_input():
+    fake = _FakeCollection()
+    collection = ChromaCollection(fake)
+    with pytest.raises(ValueError):
+        collection.query()
+
+
+def test_query_rejects_both_texts_and_embeddings():
+    fake = _FakeCollection()
+    collection = ChromaCollection(fake)
+    with pytest.raises(ValueError):
+        collection.query(query_texts=["q"], query_embeddings=[[0.1, 0.2]])
+
+
+def test_query_rejects_empty_input_list():
+    fake = _FakeCollection()
+    collection = ChromaCollection(fake)
+    with pytest.raises(ValueError):
+        collection.query(query_texts=[])
+
+
+def test_query_empty_preserves_embeddings_outer_shape_when_requested():
+    fake = _FakeCollection(
+        query_response={"ids": [], "documents": [], "metadatas": [], "distances": []}
+    )
+    collection = ChromaCollection(fake)
+
+    requested = collection.query(query_texts=["q1", "q2"], include=["documents", "embeddings"])
+    assert requested.embeddings == [[], []]
+
+    not_requested = collection.query(query_texts=["q1", "q2"], include=["documents"])
+    assert not_requested.embeddings is None
+
+
+def test_chroma_cache_invalidates_when_db_file_missing(tmp_path):
+    """A palace rebuild that removes chroma.sqlite3 must drop the stale cache.
+
+    Primes backend._clients/_freshness directly with a sentinel rather than
+    opening a real ``PersistentClient``: on Windows the sqlite file handle
+    would still be live and ``Path.unlink`` would raise ``PermissionError``,
+    making the test unable to exercise the branch we care about. The decision
+    logic under test is pure (no chromadb calls before the branch), so a
+    sentinel is sufficient.
+    """
+    backend = ChromaBackend()
+    palace_path = tmp_path / "palace"
+    palace_path.mkdir()
+    db_file = palace_path / "chroma.sqlite3"
+    db_file.write_bytes(b"")  # any file is enough for _db_stat to see it
+    st = db_file.stat()
+
+    sentinel = object()
+    backend._clients[str(palace_path)] = sentinel
+    backend._freshness[str(palace_path)] = (st.st_ino, st.st_mtime)
+
+    # Simulate a rebuild mid-flight: chroma.sqlite3 goes away. Safe to unlink
+    # because nothing in this test is holding an OS handle on the file.
+    db_file.unlink()
+
+    prior_freshness = (st.st_ino, st.st_mtime)
+    new_client = backend._client(str(palace_path))
+    # Cache was replaced (not the sentinel) and freshness reflects the post-
+    # rebuild stat (chromadb re-creates chroma.sqlite3 during PersistentClient
+    # construction; _client re-stats after the constructor so freshness is
+    # not frozen at the pre-rebuild value). The stale cached sentinel would
+    # have served wrong data if returned.
+    assert new_client is not sentinel
+    assert backend._freshness[str(palace_path)] != prior_freshness
+
+
+def test_chroma_cache_picks_up_db_created_after_first_open(tmp_path):
+    """The 0 → nonzero stat transition invalidates a cache built before the DB existed."""
+    backend = ChromaBackend()
+    palace_path = tmp_path / "palace"
+    palace_path.mkdir()
+
+    # Seed an entry in the caches as if a prior _client() call had opened the
+    # palace when chroma.sqlite3 did not exist yet. Freshness (0, 0.0) is the
+    # signal that the DB was absent at cache time.
+    sentinel = object()
+    backend._clients[str(palace_path)] = sentinel
+    backend._freshness[str(palace_path)] = (0, 0.0)
+
+    # The DB file now appears (real chromadb would have created it by now).
+    # Use a real chromadb call so _fix_blob_seq_ids and PersistentClient succeed.
+    import chromadb as _chromadb
+
+    _chromadb.PersistentClient(path=str(palace_path)).get_or_create_collection("seed")
+    assert (palace_path / "chroma.sqlite3").is_file()
+
+    # Next _client() call must detect the 0 → nonzero transition and rebuild.
+    refreshed = backend._client(str(palace_path))
+    assert refreshed is not sentinel
+    assert backend._freshness[str(palace_path)] != (0, 0.0)
+
+
+def test_base_collection_update_default_rejects_mismatched_lengths():
+    """The ABC default update() raises ValueError rather than silently misaligning."""
+    from mempalace.backends.base import BaseCollection
+
+    collection = ChromaCollection(_FakeCollection())
+
+    with pytest.raises(ValueError, match="documents length"):
+        BaseCollection.update(collection, ids=["1", "2"], documents=["only-one"])
+
+    with pytest.raises(ValueError, match="metadatas length"):
+        BaseCollection.update(collection, ids=["1", "2"], metadatas=[{"k": 9}])
+
+
+def test_chroma_backend_accepts_palace_ref_kwarg(tmp_path):
+    palace_path = tmp_path / "palace"
+    backend = ChromaBackend()
+    collection = backend.get_collection(
+        palace=PalaceRef(id=str(palace_path), local_path=str(palace_path)),
+        collection_name="mempalace_drawers",
+        create=True,
+    )
+    assert palace_path.is_dir()
+    assert isinstance(collection, ChromaCollection)
 
 
 def test_chroma_backend_create_false_raises_without_creating_directory(tmp_path):
@@ -140,3 +378,68 @@ def test_fix_blob_seq_ids_noop_without_blobs(tmp_path):
 def test_fix_blob_seq_ids_noop_without_database(tmp_path):
     """No error when palace has no chroma.sqlite3."""
     _fix_blob_seq_ids(str(tmp_path))  # should not raise
+
+
+# ── quarantine_stale_hnsw ─────────────────────────────────────────────────
+
+
+def _make_palace_with_segment(tmp_path, hnsw_mtime, sqlite_mtime):
+    """Helper: build a palace dir with one HNSW segment + sqlite at given mtimes."""
+    palace = tmp_path / "palace"
+    palace.mkdir()
+    (palace / "chroma.sqlite3").write_text("")
+    seg = palace / "abcd-1234-5678"
+    seg.mkdir()
+    (seg / "data_level0.bin").write_text("")
+    os.utime(seg / "data_level0.bin", (hnsw_mtime, hnsw_mtime))
+    os.utime(palace / "chroma.sqlite3", (sqlite_mtime, sqlite_mtime))
+    return palace, seg
+
+
+def test_quarantine_stale_hnsw_renames_drifted_segment(tmp_path):
+    """Segment whose data_level0.bin is 2h older than sqlite gets renamed."""
+    now = 1_700_000_000.0
+    palace, seg = _make_palace_with_segment(tmp_path, hnsw_mtime=now - 7200, sqlite_mtime=now)
+    moved = quarantine_stale_hnsw(str(palace), stale_seconds=3600.0)
+    assert len(moved) == 1
+    assert ".drift-" in moved[0]
+    assert not seg.exists()
+    # the renamed directory still exists and contains the original file
+    renamed = list(palace.iterdir())
+    drift_dirs = [p for p in renamed if ".drift-" in p.name]
+    assert len(drift_dirs) == 1
+    assert (drift_dirs[0] / "data_level0.bin").exists()
+
+
+def test_quarantine_stale_hnsw_leaves_fresh_segment_alone(tmp_path):
+    """Segment with recent mtime vs sqlite is not touched."""
+    now = 1_700_000_000.0
+    palace, seg = _make_palace_with_segment(tmp_path, hnsw_mtime=now - 10, sqlite_mtime=now)
+    moved = quarantine_stale_hnsw(str(palace), stale_seconds=3600.0)
+    assert moved == []
+    assert seg.exists()
+
+
+def test_quarantine_stale_hnsw_no_palace(tmp_path):
+    """Missing palace path or chroma.sqlite3: return [] without raising."""
+    assert quarantine_stale_hnsw(str(tmp_path / "missing")) == []
+    empty = tmp_path / "empty"
+    empty.mkdir()
+    assert quarantine_stale_hnsw(str(empty)) == []
+
+
+def test_quarantine_stale_hnsw_skips_already_quarantined(tmp_path):
+    """Directories already named with ``.drift-`` suffix are never re-renamed."""
+    now = 1_700_000_000.0
+    palace = tmp_path / "palace"
+    palace.mkdir()
+    (palace / "chroma.sqlite3").write_text("")
+    os.utime(palace / "chroma.sqlite3", (now, now))
+    drift = palace / "abcd-1234.drift-20260101-000000"
+    drift.mkdir()
+    (drift / "data_level0.bin").write_text("")
+    os.utime(drift / "data_level0.bin", (now - 99999, now - 99999))
+
+    moved = quarantine_stale_hnsw(str(palace), stale_seconds=3600.0)
+    assert moved == []
+    assert drift.exists()

tests/test_convo_miner_size_cap.py

@@ -0,0 +1,31 @@
+"""TDD: convo_miner.py must not silently drop transcripts larger than 10 MB.
+
+Mirrors the miner.py fix shipped in the same PR family (see
+test_miner_jsonl_visibility.py). Long Claude Code sessions, ChatGPT
+exports, and multi-year Slack dumps routinely exceed 10 MB. The cap
+silently `continue`s past them at convo_miner.py:~289, same silent-drop
+pattern as the project miner's.
+
+Written BEFORE the fix.
+"""
+
+from mempalace.convo_miner import MAX_FILE_SIZE
+
+
+class TestConvoMinerSizeCap:
+    def test_max_file_size_accommodates_long_transcripts(self):
+        """The cap must be well above any realistic transcript.
+
+        Long sessions and lifetime exports exceed 10 MB. The cap exists
+        as a sanity rail against pathological binaries, not as a limit
+        on legitimate text — downstream chunking means source size does
+        not matter for storage or embedding cost.
+        """
+        assert MAX_FILE_SIZE >= 100 * 1024 * 1024, (
+            f"convo_miner.MAX_FILE_SIZE is {MAX_FILE_SIZE} bytes "
+            f"({MAX_FILE_SIZE / 1024 / 1024:.0f} MB). Same silent-drop "
+            "bug as miner.py's old 10 MB cap — long transcripts get "
+            "filtered out at convo_miner.py:~289 with `continue`. "
+            "Raise to at least 100 MB (match miner.py at 500 MB for "
+            "consistency across both miners)."
+        )

tests/test_hooks_cli.py

@@ -1,6 +1,7 @@
 import contextlib
 import io
 import json
+import os
 import subprocess
 from pathlib import Path
 from unittest.mock import patch
@@ -14,6 +15,7 @@ from mempalace.hooks_cli import (
     _get_mine_dir,
     _log,
     _maybe_auto_ingest,
+    _mine_already_running,
     _parse_harness_input,
     _sanitize_session_id,
     _validate_transcript_path,
@@ -250,9 +252,10 @@ def test_maybe_auto_ingest_with_env(tmp_path):
     mempal_dir.mkdir()
     with patch.dict("os.environ", {"MEMPAL_DIR": str(mempal_dir)}):
         with patch("mempalace.hooks_cli.STATE_DIR", tmp_path):
-            with patch("mempalace.hooks_cli.subprocess.Popen") as mock_popen:
-                _maybe_auto_ingest()
-                mock_popen.assert_called_once()
+            with patch("mempalace.hooks_cli._MINE_PID_FILE", tmp_path / "mine.pid"):
+                with patch("mempalace.hooks_cli.subprocess.Popen") as mock_popen:
+                    _maybe_auto_ingest()
+                    mock_popen.assert_called_once()
 
 
 def test_maybe_auto_ingest_with_transcript(tmp_path):
@@ -261,9 +264,10 @@ def test_maybe_auto_ingest_with_transcript(tmp_path):
     transcript.write_text("")
     with patch.dict("os.environ", {}, clear=True):
         with patch("mempalace.hooks_cli.STATE_DIR", tmp_path):
-            with patch("mempalace.hooks_cli.subprocess.Popen") as mock_popen:
-                _maybe_auto_ingest(str(transcript))
-                mock_popen.assert_called_once()
+            with patch("mempalace.hooks_cli._MINE_PID_FILE", tmp_path / "mine.pid"):
+                with patch("mempalace.hooks_cli.subprocess.Popen") as mock_popen:
+                    _maybe_auto_ingest(str(transcript))
+                    mock_popen.assert_called_once()
 
 
 def test_maybe_auto_ingest_oserror(tmp_path):
@@ -272,8 +276,54 @@ def test_maybe_auto_ingest_oserror(tmp_path):
     mempal_dir.mkdir()
     with patch.dict("os.environ", {"MEMPAL_DIR": str(mempal_dir)}):
         with patch("mempalace.hooks_cli.STATE_DIR", tmp_path):
-            with patch("mempalace.hooks_cli.subprocess.Popen", side_effect=OSError("fail")):
-                _maybe_auto_ingest()  # should not raise
+            with patch("mempalace.hooks_cli._MINE_PID_FILE", tmp_path / "mine.pid"):
+                with patch("mempalace.hooks_cli.subprocess.Popen", side_effect=OSError("fail")):
+                    _maybe_auto_ingest()  # should not raise
+
+
+def test_maybe_auto_ingest_skips_when_mine_running(tmp_path):
+    """Does not spawn a new mine process if one is already running."""
+    mempal_dir = tmp_path / "project"
+    mempal_dir.mkdir()
+    with patch.dict("os.environ", {"MEMPAL_DIR": str(mempal_dir)}):
+        with patch("mempalace.hooks_cli.STATE_DIR", tmp_path):
+            with patch("mempalace.hooks_cli._mine_already_running", return_value=True):
+                with patch("mempalace.hooks_cli.subprocess.Popen") as mock_popen:
+                    _maybe_auto_ingest()
+                    mock_popen.assert_not_called()
+
+
+# --- _mine_already_running ---
+
+
+def test_mine_already_running_no_file(tmp_path):
+    """Returns False when no PID file exists."""
+    with patch("mempalace.hooks_cli._MINE_PID_FILE", tmp_path / "mine.pid"):
+        assert _mine_already_running() is False
+
+
+def test_mine_already_running_dead_pid(tmp_path):
+    """Returns False when PID file contains a PID that no longer exists."""
+    pid_file = tmp_path / "mine.pid"
+    pid_file.write_text("999999999")  # almost certainly not a real PID
+    with patch("mempalace.hooks_cli._MINE_PID_FILE", pid_file):
+        assert _mine_already_running() is False
+
+
+def test_mine_already_running_live_pid(tmp_path):
+    """Returns True when PID file contains the current process's own PID."""
+    pid_file = tmp_path / "mine.pid"
+    pid_file.write_text(str(os.getpid()))  # current process is definitely alive
+    with patch("mempalace.hooks_cli._MINE_PID_FILE", pid_file):
+        assert _mine_already_running() is True
+
+
+def test_mine_already_running_corrupt_file(tmp_path):
+    """Returns False when PID file contains non-integer content."""
+    pid_file = tmp_path / "mine.pid"
+    pid_file.write_text("not-a-pid")
+    with patch("mempalace.hooks_cli._MINE_PID_FILE", pid_file):
+        assert _mine_already_running() is False
 
 
 # --- _get_mine_dir ---

tests/test_mcp_server.py

@@ -250,6 +250,38 @@ class TestReadTools:
         assert "project" in result["wings"]
         assert "notes" in result["wings"]
 
+    def test_status_handles_none_metadata_without_partial(
+        self, monkeypatch, config, palace_path, kg
+    ):
+        """tool_status must not crash or go partial when the metadata cache
+        returns a ``None`` entry — palaces can contain drawers with no
+        metadata (older mining paths, third-party writes). Before the guard,
+        ``m.get("wing")`` raised AttributeError mid-tally and the result
+        carried ``"error"`` + ``"partial": True`` even though the data was
+        perfectly fetchable."""
+        from unittest.mock import patch as _patch
+
+        _patch_mcp_server(monkeypatch, config, kg)
+        from mempalace.mcp_server import tool_status
+
+        # Inject a metadata cache where one entry is None
+        with _patch("mempalace.mcp_server._get_collection") as mock_get_col:
+            fake_col = type("C", (), {"count": lambda self: 2})()
+            mock_get_col.return_value = fake_col
+            with _patch(
+                "mempalace.mcp_server._get_cached_metadata",
+                return_value=[{"wing": "proj", "room": "r"}, None],
+            ):
+                result = tool_status()
+
+        # The None-metadata drawer falls under 'unknown/unknown' — no crash,
+        # no partial flag.
+        assert "error" not in result
+        assert result.get("partial") is not True
+        assert result["total_drawers"] == 2
+        assert result["wings"].get("proj") == 1
+        assert result["wings"].get("unknown") == 1
+
     def test_list_wings(self, monkeypatch, config, palace_path, seeded_collection, kg):
         _patch_mcp_server(monkeypatch, config, kg)
         from mempalace.mcp_server import tool_list_wings

tests/test_miner.py

@@ -343,6 +343,36 @@ def test_status_missing_palace_does_not_create_empty_collection(tmp_path, capsys
     assert not palace_path.exists()
 
 
+def test_status_handles_none_metadata_without_crash(tmp_path, capsys):
+    """status must not crash when col.get returns a None entry in metadatas.
+
+    Palaces can contain drawers whose metadata was never set (older mining
+    paths, drawers written by third-party tools). Before the guard, status
+    crashed mid-tally with ``AttributeError: 'NoneType' object has no
+    attribute 'get'`` at the wing/room histogram line."""
+    from unittest.mock import patch
+
+    class FakeCol:
+        def count(self):
+            return 2
+
+        def get(self, *args, **kwargs):
+            return {
+                "ids": ["a", "b"],
+                "documents": ["doc a", "doc b"],
+                "metadatas": [{"wing": "proj", "room": "r"}, None],
+            }
+
+    with patch("mempalace.miner.get_collection", return_value=FakeCol()):
+        status(str(tmp_path))
+
+    out = capsys.readouterr().out
+    # No crash; the None-metadata row is counted under the ?/? fallback
+    # alongside the real wing=proj row.
+    assert "WING: ?" in out
+    assert "WING: proj" in out
+
+
 # ── normalize_version schema gate ───────────────────────────────────────
 #
 # When the normalization pipeline changes shape (e.g., strip_noise lands),

tests/test_miner_jsonl_visibility.py

@@ -0,0 +1,126 @@
+"""TDD: miner.py must not silently drop .jsonl files.
+
+The project miner (mempalace.miner.scan_project) walks a directory and
+keeps only files whose suffix is in READABLE_EXTENSIONS. The whitelist
+contains `.json` but NOT `.jsonl`. Every ChatGPT export, Claude Code
+transcript, or any other jsonl transcript dumped into a project
+directory is silently dropped with no user-visible output.
+
+Two paths to fix this, both tested here:
+
+  1. READABLE_EXTENSIONS must include `.jsonl` so the file is at least
+     readable as text (jsonl is line-delimited JSON — each line is
+     already valid text for embedding).
+  2. OR scan_project must surface skipped .jsonl files to the user so
+     they know to use `--mode convos`.
+
+We test (1) — include .jsonl in READABLE_EXTENSIONS. This matches how
+`.json` is already handled: the miner doesn't care what the structure
+is, it chunks the text.
+
+Written BEFORE the fix.
+"""
+
+import tempfile
+from pathlib import Path
+from unittest.mock import patch
+
+from mempalace.miner import MAX_FILE_SIZE, READABLE_EXTENSIONS, scan_project
+
+
+class TestJsonlNotSilentlySkipped:
+    def test_jsonl_in_readable_extensions(self):
+        """`.jsonl` must be in the readable-extensions whitelist.
+
+        `.json` is already there (see mempalace/miner.py:30). `.jsonl`
+        is conceptually the same thing — line-delimited JSON — and all
+        of Claude Code's transcripts, ChatGPT exports, and similar
+        tooling writes `.jsonl`. Excluding it silently drops user data.
+        """
+        assert ".jsonl" in READABLE_EXTENSIONS, (
+            "mempalace/miner.py:READABLE_EXTENSIONS contains `.json` "
+            "but NOT `.jsonl`. Every jsonl file in a mined project is "
+            "silently skipped at miner.py:722 "
+            "(`if filepath.suffix.lower() not in READABLE_EXTENSIONS: "
+            "continue`). This causes the 'convos not being saved' bug "
+            "reported by users — the hook fires `mempalace mine`, the "
+            "miner walks the directory, skips every .jsonl file, exits "
+            "cleanly. No warning, no log line, user sees nothing wrong. "
+            "Add `.jsonl` to READABLE_EXTENSIONS."
+        )
+
+    def test_scan_project_picks_up_jsonl_file(self):
+        """scan_project should find .jsonl files in the target dir."""
+        with tempfile.TemporaryDirectory() as tmp:
+            tmpdir = Path(tmp)
+            jsonl_path = tmpdir / "transcript.jsonl"
+            jsonl_path.write_text(
+                '{"role": "user", "content": "hello"}\n'
+                '{"role": "assistant", "content": "hi there"}\n'
+                '{"role": "user", "content": "how do I install this"}\n'
+                '{"role": "assistant", "content": "pip install mempalace"}\n'
+            )
+
+            found = scan_project(str(tmpdir))
+            found_names = [p.name for p in found]
+            assert "transcript.jsonl" in found_names, (
+                "scan_project silently dropped transcript.jsonl. "
+                f"Returned: {found_names}. Users placing transcript "
+                "exports in a project directory expect them to be mined."
+            )
+
+    def test_large_jsonl_not_silently_dropped_by_size_cap(self):
+        """Long sessions produce >10 MB transcripts. They must still mine.
+
+        The legacy cap was 10 MB, which is smaller than a long Claude Code
+        session's transcript. Users hitting the cap lost their entire
+        conversation to a silent `if size > MAX: continue` at miner.py:732.
+        Raise the cap well above any realistic transcript size.
+        """
+        # 10 MB cap was silent failure — real Claude Code long sessions
+        # exceed this. The cap must accommodate them.
+        assert MAX_FILE_SIZE >= 100 * 1024 * 1024, (
+            f"MAX_FILE_SIZE is {MAX_FILE_SIZE} bytes "
+            f"({MAX_FILE_SIZE / 1024 / 1024:.0f} MB). Long Claude Code "
+            "sessions produce transcripts larger than 10 MB and get "
+            "silently dropped. Raise to at least 100 MB — chunking "
+            "at 800 chars per drawer means source file size doesn't "
+            "matter for downstream storage."
+        )
+
+    def test_scan_project_picks_up_50mb_jsonl(self):
+        """A 50 MB .jsonl must not be filtered out by the size cap.
+
+        We don't actually write 50 MB (slow test). Instead, we mock
+        stat().st_size to report a 50 MB file and confirm scan_project
+        still includes it.
+        """
+        with tempfile.TemporaryDirectory() as tmp:
+            tmpdir = Path(tmp)
+            big_jsonl = tmpdir / "big_transcript.jsonl"
+            # Write a small real file so the existence / extension / text
+            # checks pass; then mock its reported size.
+            big_jsonl.write_text('{"role": "user", "content": "hi"}\n')
+            fake_size = 50 * 1024 * 1024  # 50 MB
+
+            real_stat = Path.stat
+
+            def fake_stat(self, *args, **kwargs):
+                result = real_stat(self, *args, **kwargs)
+                if self.name == "big_transcript.jsonl":
+
+                    class _FakeStat:
+                        st_size = fake_size
+                        st_mode = result.st_mode
+
+                    return _FakeStat()
+                return result
+
+            with patch.object(Path, "stat", fake_stat):
+                found = scan_project(str(tmpdir))
+
+            found_names = [p.name for p in found]
+            assert "big_transcript.jsonl" in found_names, (
+                f"50 MB .jsonl was dropped by size cap (MAX_FILE_SIZE="
+                f"{MAX_FILE_SIZE}). Returned: {found_names}."
+            )

tests/test_searcher.py

@@ -89,6 +89,37 @@ class TestSearchMemories:
         assert result["filters"]["wing"] == "project"
         assert result["filters"]["room"] == "backend"
 
+    def test_search_memories_handles_none_metadata(self):
+        """API path: `None` entries in the drawer results' metadatas list must
+        fall back to the sentinel strings (wing/room 'unknown', source '?')
+        rather than raising `AttributeError: 'NoneType' object has no
+        attribute 'get'` while the rest of the result set renders."""
+        mock_col = MagicMock()
+        mock_col.query.return_value = {
+            "documents": [["first doc", "second doc"]],
+            "metadatas": [[{"source_file": "a.md", "wing": "w", "room": "r"}, None]],
+            "distances": [[0.1, 0.2]],
+            "ids": [["d1", "d2"]],
+        }
+
+        def mock_get_collection(path, create=False):
+            # First call: drawers. Second call: closets — raise so hybrid
+            # degrades to pure drawer search (the catch block covers it).
+            if not hasattr(mock_get_collection, "_called"):
+                mock_get_collection._called = True
+                return mock_col
+            raise RuntimeError("no closets")
+
+        with patch("mempalace.searcher.get_collection", side_effect=mock_get_collection):
+            result = search_memories("anything", "/fake/path")
+        assert "results" in result
+        assert len(result["results"]) == 2
+        # The None-metadata hit renders with sentinel values, not a crash.
+        none_hit = result["results"][1]
+        assert none_hit["text"] == "second doc"
+        assert none_hit["wing"] == "unknown"
+        assert none_hit["room"] == "unknown"
+
 
 # ── search() (CLI print function) ─────────────────────────────────────
 
@@ -141,3 +172,22 @@ class TestSearchCLI:
         captured = capsys.readouterr()
         # Should have output with at least one result block
         assert "[1]" in captured.out
+
+    def test_search_handles_none_metadata_without_crash(self, palace_path, capsys):
+        """ChromaDB can return `None` entries in the metadatas list when a
+        drawer has no metadata. The CLI print path must not crash on them
+        mid-render — it used to raise `AttributeError: 'NoneType' object has
+        no attribute 'get'` after printing earlier results."""
+        mock_col = MagicMock()
+        mock_col.query.return_value = {
+            "documents": [["first doc", "second doc"]],
+            "metadatas": [[{"source_file": "a.md", "wing": "w", "room": "r"}, None]],
+            "distances": [[0.1, 0.2]],
+        }
+        with patch("mempalace.searcher.get_collection", return_value=mock_col):
+            search("anything", "/fake/path")
+        captured = capsys.readouterr()
+        assert "[1]" in captured.out
+        assert "[2]" in captured.out
+        # Second result renders with fallback '?' values instead of crashing
+        assert "second doc" in captured.out

tests/test_sources.py

@@ -0,0 +1,467 @@
+"""Tests for the RFC 002 source-adapter scaffolding."""
+
+import pytest
+
+from mempalace.sources import (
+    AdapterSchema,
+    BaseSourceAdapter,
+    DrawerRecord,
+    FieldSpec,
+    PalaceContext,
+    RouteHint,
+    SourceItemMetadata,
+    SourceRef,
+    SourceSummary,
+    available_adapters,
+    get_adapter,
+    get_adapter_class,
+    register,
+    reset_adapters,
+    resolve_adapter_for_source,
+    unregister,
+)
+from mempalace.sources.transforms import (
+    RESERVED_TRANSFORMATIONS,
+    blank_line_drop,
+    get_transformation,
+    line_join_spaces,
+    line_trim,
+    newline_normalize,
+    utf8_replace_invalid,
+    whitespace_collapse_internal,
+    whitespace_trim,
+)
+
+
+# ---------------------------------------------------------------------------
+# Minimal conforming adapter used as a fixture across tests
+# ---------------------------------------------------------------------------
+
+
+class _TrivialAdapter(BaseSourceAdapter):
+    name = "_trivial"
+    adapter_version = "0.1.0"
+    capabilities = frozenset({"byte_preserving"})
+    supported_modes = frozenset({"whole_record"})
+    declared_transformations = frozenset()
+    default_privacy_class = "public"
+
+    def ingest(self, *, source, palace):
+        yield SourceItemMetadata(source_file=source.uri or "x", version="v1")
+        yield DrawerRecord(content="hello", source_file=source.uri or "x", chunk_index=0)
+
+    def describe_schema(self):
+        return AdapterSchema(
+            version="1.0",
+            fields={"example": FieldSpec(type="string", required=False, description="x")},
+        )
+
+
+@pytest.fixture(autouse=True)
+def _isolate_registry():
+    yield
+    reset_adapters()
+    for name in list(available_adapters()):
+        unregister(name)
+
+
+# ---------------------------------------------------------------------------
+# base.py — ABC + typed records
+# ---------------------------------------------------------------------------
+
+
+def test_base_adapter_is_abstract_without_required_methods():
+    with pytest.raises(TypeError):
+
+        class Incomplete(BaseSourceAdapter):
+            name = "incomplete"
+
+        Incomplete()
+
+
+def test_conforming_adapter_instantiates_and_yields_typed_records():
+    adapter = _TrivialAdapter()
+    results = list(adapter.ingest(source=SourceRef(uri="foo"), palace=None))
+    assert len(results) == 2
+    assert isinstance(results[0], SourceItemMetadata)
+    assert isinstance(results[1], DrawerRecord)
+    assert results[1].content == "hello"
+
+
+def test_is_current_default_is_false_always_reextracts():
+    adapter = _TrivialAdapter()
+    item = SourceItemMetadata(source_file="f", version="v1")
+    assert adapter.is_current(item=item, existing_metadata=None) is False
+    assert adapter.is_current(item=item, existing_metadata={"version": "v1"}) is False
+
+
+def test_source_summary_default_uses_adapter_name():
+    adapter = _TrivialAdapter()
+    summary = adapter.source_summary(source=SourceRef(uri="x"))
+    assert isinstance(summary, SourceSummary)
+    assert summary.description == "_trivial"
+
+
+def test_source_ref_options_default_is_empty_dict():
+    # Frozen dataclass must not share a default_factory=dict instance across instances.
+    a = SourceRef(uri="a")
+    b = SourceRef(uri="b")
+    a.options["touched"] = True
+    assert "touched" not in b.options
+
+
+# ---------------------------------------------------------------------------
+# transforms.py
+# ---------------------------------------------------------------------------
+
+
+def test_reserved_transformations_registry_has_all_13():
+    expected = {
+        "utf8_replace_invalid",
+        "newline_normalize",
+        "whitespace_trim",
+        "whitespace_collapse_internal",
+        "line_trim",
+        "line_join_spaces",
+        "blank_line_drop",
+        "strip_tool_chrome",
+        "tool_result_truncate",
+        "tool_result_omitted",
+        "spellcheck_user",
+        "synthesized_marker",
+        "speaker_role_assignment",
+    }
+    assert set(RESERVED_TRANSFORMATIONS) == expected
+
+
+def test_utf8_replace_invalid_handles_bad_bytes():
+    # A lone 0xff byte is never valid UTF-8; U+FFFD should replace it.
+    assert utf8_replace_invalid(b"ok \xff end") == "ok \ufffd end"
+
+
+def test_newline_normalize_converts_crlf_and_cr():
+    assert newline_normalize("a\r\nb\rc\nd") == "a\nb\nc\nd"
+
+
+def test_whitespace_trim_strips_boundaries():
+    assert whitespace_trim("  hello\n\n") == "hello"
+
+
+def test_whitespace_collapse_internal_caps_at_two_blanks():
+    # Five blanks collapses to exactly three newlines (two blank lines).
+    text = "a\n\n\n\n\nb"
+    assert whitespace_collapse_internal(text) == "a\n\n\nb"
+
+
+def test_line_trim_strips_each_line():
+    assert line_trim("  a  \n\t b \n c") == "a\nb\nc"
+
+
+def test_line_join_spaces_preserves_paragraph_breaks():
+    text = "foo\nbar\nbaz\n\nqux\nquux"
+    assert line_join_spaces(text) == "foo bar baz\n\nqux quux"
+
+
+def test_blank_line_drop_removes_blanks_only():
+    assert blank_line_drop("a\n\nb\n\n\nc") == "a\nb\nc"
+
+
+def test_get_transformation_resolves_reserved_and_rejects_unknown():
+    assert get_transformation("newline_normalize") is newline_normalize
+    with pytest.raises(KeyError):
+        get_transformation("not_a_real_transformation")
+
+
+# ---------------------------------------------------------------------------
+# registry.py
+# ---------------------------------------------------------------------------
+
+
+def test_register_and_get_adapter_roundtrip():
+    register("_trivial", _TrivialAdapter)
+    assert "_trivial" in available_adapters()
+    inst = get_adapter("_trivial")
+    assert isinstance(inst, _TrivialAdapter)
+    # Cached: repeated calls return the same instance.
+    assert get_adapter("_trivial") is inst
+
+
+def test_get_adapter_class_returns_class_not_instance():
+    register("_trivial", _TrivialAdapter)
+    assert get_adapter_class("_trivial") is _TrivialAdapter
+
+
+def test_get_adapter_unknown_raises_key_error():
+    with pytest.raises(KeyError):
+        get_adapter("does-not-exist")
+
+
+def test_unregister_drops_registration_and_cached_instance():
+    register("_trivial", _TrivialAdapter)
+    get_adapter("_trivial")
+    unregister("_trivial")
+    assert "_trivial" not in available_adapters()
+    with pytest.raises(KeyError):
+        get_adapter("_trivial")
+
+
+def test_resolve_adapter_priority_order():
+    # Explicit wins over everything.
+    assert resolve_adapter_for_source(explicit="cursor", config_value="git") == "cursor"
+    # Config wins over default.
+    assert resolve_adapter_for_source(config_value="git") == "git"
+    # Default is filesystem (preserves existing ``mempalace mine <path>`` behavior).
+    assert resolve_adapter_for_source() == "filesystem"
+
+
+# ---------------------------------------------------------------------------
+# PalaceContext
+# ---------------------------------------------------------------------------
+
+
+class _FakeCollection:
+    def __init__(self):
+        self.upserts = []
+
+    def add(self, **kwargs):
+        pass
+
+    def upsert(self, **kwargs):
+        self.upserts.append(kwargs)
+
+    def query(self, **kwargs):
+        return {}
+
+    def get(self, **kwargs):
+        return {}
+
+    def delete(self, **kwargs):
+        pass
+
+    def count(self):
+        return 0
+
+
+class _FakeKG:
+    def __init__(self):
+        self.triples = []
+
+    def add_triple(self, subject, predicate, obj, **kwargs):
+        self.triples.append((subject, predicate, obj, kwargs))
+
+
+def test_palace_context_upsert_drawer_stamps_adapter_metadata():
+    drawers = _FakeCollection()
+    kg = _FakeKG()
+    ctx = PalaceContext(
+        drawer_collection=drawers,
+        knowledge_graph=kg,
+        palace_path="/tmp/palace",
+        adapter_name="test-adapter",
+        adapter_version="0.1.0",
+    )
+    record = DrawerRecord(
+        content="hello",
+        source_file="/abs/path/file.txt",
+        chunk_index=2,
+        metadata={"wing": "proj"},
+    )
+    ctx.upsert_drawer(record)
+
+    assert len(drawers.upserts) == 1
+    kwargs = drawers.upserts[0]
+    assert kwargs["documents"] == ["hello"]
+    assert len(kwargs["ids"]) == 1
+    meta = kwargs["metadatas"][0]
+    assert meta["wing"] == "proj"
+    assert meta["adapter_name"] == "test-adapter"
+    assert meta["adapter_version"] == "0.1.0"
+    assert meta["source_file"] == "/abs/path/file.txt"
+    assert meta["chunk_index"] == 2
+
+
+def test_palace_context_drawer_id_is_sha256_prefix_not_sha1():
+    """Guards against the pre-review sha1[:16]=64-bit id scheme.
+
+    64-bit ids sit close to the birthday bound for palace-sized corpora.
+    The helper uses sha256[:24]=96 bits so collision risk stays negligible.
+    """
+    import hashlib
+
+    from mempalace.sources.context import _build_drawer_id
+
+    src = "/an/absolute/path/to/a/file.txt"
+    record = DrawerRecord(content="x", source_file=src, chunk_index=3)
+    drawer_id = _build_drawer_id(record)
+
+    expected_prefix = hashlib.sha256(src.encode("utf-8")).hexdigest()[:24]
+    assert drawer_id == f"{expected_prefix}_3"
+    # Negative: the old sha1 scheme MUST NOT produce the same id.
+    sha1_prefix = hashlib.sha1(src.encode("utf-8")).hexdigest()[:16]
+    assert drawer_id != f"{sha1_prefix}_3"
+
+
+def test_palace_context_skip_current_item_sets_flag():
+    ctx = PalaceContext(
+        drawer_collection=_FakeCollection(),
+        knowledge_graph=_FakeKG(),
+        palace_path="/tmp/p",
+    )
+    assert ctx._skip_requested is False
+    ctx.skip_current_item()
+    assert ctx._skip_requested is True
+
+
+def test_palace_context_emit_dispatches_to_hooks_and_swallows_errors():
+    calls = []
+    err_calls = []
+
+    def good_hook(event, **details):
+        calls.append((event, details))
+
+    def bad_hook(event, **details):
+        err_calls.append(event)
+        raise RuntimeError("hook exploded")
+
+    ctx = PalaceContext(
+        drawer_collection=_FakeCollection(),
+        knowledge_graph=_FakeKG(),
+        palace_path="/tmp/p",
+        progress_hooks=[good_hook, bad_hook],
+    )
+    ctx.emit("mined_file", path="a.txt", bytes=42)
+    assert calls == [("mined_file", {"path": "a.txt", "bytes": 42})]
+    assert err_calls == ["mined_file"]  # was invoked; error was swallowed
+
+
+def test_palace_context_uses_route_hint_when_present():
+    # Route hints are frozen dataclasses the adapter passes through.
+    hint = RouteHint(wing="proj", room="backend", hall="general")
+    assert hint.wing == "proj"
+    assert hint.room == "backend"
+
+
+# ---------------------------------------------------------------------------
+# KnowledgeGraph new provenance params (RFC 002 §5.5)
+# ---------------------------------------------------------------------------
+
+
+def test_knowledge_graph_add_triple_accepts_source_drawer_id_and_adapter_name(tmp_path):
+    from mempalace.knowledge_graph import KnowledgeGraph
+
+    kg = KnowledgeGraph(db_path=str(tmp_path / "kg.sqlite3"))
+    try:
+        triple_id = kg.add_triple(
+            "Ben",
+            "committed",
+            "PR-567",
+            valid_from="2026-03-12",
+            source_file="github.com/org/repo#pr=567",
+            source_drawer_id="abc123_0",
+            adapter_name="git",
+        )
+        assert triple_id is not None
+
+        import sqlite3
+
+        conn = sqlite3.connect(str(tmp_path / "kg.sqlite3"))
+        conn.row_factory = sqlite3.Row
+        row = conn.execute(
+            "SELECT source_drawer_id, adapter_name FROM triples WHERE id=?", (triple_id,)
+        ).fetchone()
+        assert row["source_drawer_id"] == "abc123_0"
+        assert row["adapter_name"] == "git"
+        conn.close()
+    finally:
+        kg.close()
+
+
+def test_knowledge_graph_fresh_schema_includes_new_columns(tmp_path):
+    """Brand-new palaces should get source_drawer_id / adapter_name directly
+    from CREATE TABLE, not via a post-hoc ALTER. _migrate_schema exists only
+    for legacy palaces."""
+    import sqlite3
+
+    from mempalace.knowledge_graph import KnowledgeGraph
+
+    kg = KnowledgeGraph(db_path=str(tmp_path / "fresh.sqlite3"))
+    try:
+        conn = sqlite3.connect(str(tmp_path / "fresh.sqlite3"))
+        cols = {row[1] for row in conn.execute("PRAGMA table_info(triples)")}
+        conn.close()
+        assert "source_drawer_id" in cols
+        assert "adapter_name" in cols
+    finally:
+        kg.close()
+
+
+def test_knowledge_graph_migration_adds_missing_columns_to_old_schema(tmp_path):
+    """An old-schema triples table (pre-RFC 002) should auto-migrate on open."""
+    import sqlite3
+
+    db_path = tmp_path / "legacy.sqlite3"
+    conn = sqlite3.connect(str(db_path))
+    conn.executescript("""
+        CREATE TABLE entities (
+            id TEXT PRIMARY KEY,
+            name TEXT NOT NULL,
+            type TEXT DEFAULT 'unknown',
+            properties TEXT DEFAULT '{}',
+            created_at TEXT DEFAULT CURRENT_TIMESTAMP
+        );
+        CREATE TABLE triples (
+            id TEXT PRIMARY KEY,
+            subject TEXT NOT NULL,
+            predicate TEXT NOT NULL,
+            object TEXT NOT NULL,
+            valid_from TEXT,
+            valid_to TEXT,
+            confidence REAL DEFAULT 1.0,
+            source_closet TEXT,
+            source_file TEXT,
+            extracted_at TEXT DEFAULT CURRENT_TIMESTAMP
+        );
+    """)
+    conn.commit()
+    conn.close()
+
+    from mempalace.knowledge_graph import KnowledgeGraph
+
+    kg = KnowledgeGraph(db_path=str(db_path))
+    try:
+        # New columns must be present after _init_db runs the migration.
+        conn = sqlite3.connect(str(db_path))
+        cols = {row[1] for row in conn.execute("PRAGMA table_info(triples)")}
+        conn.close()
+        assert "source_drawer_id" in cols
+        assert "adapter_name" in cols
+
+        # New-column insert works.
+        kg.add_triple("a", "rel", "b", source_drawer_id="d0", adapter_name="x")
+    finally:
+        kg.close()
+
+
+def test_knowledge_graph_add_triple_backwards_compatible_without_new_kwargs(tmp_path):
+    """Existing callers that omit the RFC 002 kwargs keep working unchanged."""
+    from mempalace.knowledge_graph import KnowledgeGraph
+
+    kg = KnowledgeGraph(db_path=str(tmp_path / "kg.sqlite3"))
+    try:
+        triple_id = kg.add_triple("Max", "likes", "trains")
+        assert triple_id is not None
+    finally:
+        kg.close()
+
+
+# ---------------------------------------------------------------------------
+# pyproject entry-point group is discoverable even when empty
+# ---------------------------------------------------------------------------
+
+
+def test_entry_point_group_exists_and_returns_zero_or_more_adapters():
+    # No in-tree first-party adapters yet (miners migrate in a follow-up PR),
+    # but the ``mempalace.sources`` entry-point group is declared so third-
+    # party packages can register. ``available_adapters`` MUST NOT raise.
+    adapters = available_adapters()
+    assert isinstance(adapters, list)

tests/test_sweeper.py

@@ -0,0 +1,318 @@
+"""TDD: tandem sweeper that catches what the primary miner missed.
+
+The primary miner (miner.py / convo_miner.py) runs at file granularity
+and can drop data (size caps, silent OSError, dedup false-positives).
+The sweeper is a second miner that works at MESSAGE granularity,
+using timestamp as the coordination cursor.
+
+For each session in the transcript directory:
+  1. Look up max(timestamp) across all drawers with matching session_id
+  2. Stream the jsonl, yielding only user/assistant messages after the cursor
+  3. Write one small drawer per message with:
+       session_id, uuid, timestamp, role, content
+  4. Idempotent: re-running sweeps should find nothing new on a complete palace.
+
+This test file is TDD — written BEFORE mempalace/sweeper.py exists.
+"""
+
+import json
+
+import pytest
+
+
+@pytest.fixture
+def mock_claude_jsonl(tmp_path):
+    """Real Claude Code jsonl shape: user/assistant records among progress noise."""
+    path = tmp_path / "session_abc.jsonl"
+    lines = [
+        # Noise: progress event, no message
+        {
+            "type": "progress",
+            "timestamp": "2026-04-18T10:00:00Z",
+            "sessionId": "abc",
+            "uuid": "p-1",
+        },
+        # User message
+        {
+            "type": "user",
+            "timestamp": "2026-04-18T10:00:05Z",
+            "sessionId": "abc",
+            "uuid": "u-1",
+            "message": {"role": "user", "content": "What's the capital of France?"},
+        },
+        # Assistant reply
+        {
+            "type": "assistant",
+            "timestamp": "2026-04-18T10:00:06Z",
+            "sessionId": "abc",
+            "uuid": "a-1",
+            "message": {"role": "assistant", "content": [{"type": "text", "text": "Paris."}]},
+        },
+        # Noise: file-history-snapshot
+        {"type": "file-history-snapshot", "messageId": "abc-snap"},
+        # Second user/assistant exchange
+        {
+            "type": "user",
+            "timestamp": "2026-04-18T10:01:00Z",
+            "sessionId": "abc",
+            "uuid": "u-2",
+            "message": {"role": "user", "content": "And of Germany?"},
+        },
+        {
+            "type": "assistant",
+            "timestamp": "2026-04-18T10:01:01Z",
+            "sessionId": "abc",
+            "uuid": "a-2",
+            "message": {"role": "assistant", "content": [{"type": "text", "text": "Berlin."}]},
+        },
+    ]
+    path.write_text("\n".join(json.dumps(x) for x in lines) + "\n")
+    return path
+
+
+class TestSweeperParsing:
+    def test_parse_yields_only_user_and_assistant(self, mock_claude_jsonl):
+        from mempalace.sweeper import parse_claude_jsonl
+
+        records = list(parse_claude_jsonl(str(mock_claude_jsonl)))
+        roles = [r["role"] for r in records]
+        assert roles == ["user", "assistant", "user", "assistant"], (
+            f"Expected 4 user/assistant in order, got {roles}. "
+            "Noise records (progress, file-history-snapshot) must be "
+            "filtered out."
+        )
+
+    def test_parse_extracts_session_id_and_timestamp(self, mock_claude_jsonl):
+        from mempalace.sweeper import parse_claude_jsonl
+
+        records = list(parse_claude_jsonl(str(mock_claude_jsonl)))
+        first = records[0]
+        assert first["session_id"] == "abc"
+        assert first["timestamp"] == "2026-04-18T10:00:05Z"
+        assert first["uuid"] == "u-1"
+
+    def test_parse_normalizes_assistant_content_list_to_text(self, mock_claude_jsonl):
+        from mempalace.sweeper import parse_claude_jsonl
+
+        records = list(parse_claude_jsonl(str(mock_claude_jsonl)))
+        assistant_rec = records[1]
+        assert assistant_rec["role"] == "assistant"
+        assert (
+            "Paris" in assistant_rec["content"]
+        ), f"Assistant content blocks must be flattened to text; got: {assistant_rec['content']!r}"
+
+    def test_parse_preserves_tool_blocks_verbatim(self, tmp_path):
+        """Per the design principle "verbatim always", tool_use and
+        tool_result blocks must NOT be truncated. A long tool input
+        (e.g. a large diff handed to a code-edit tool) must round-trip
+        in full, otherwise we silently lose user-adjacent data.
+        """
+        import json as _json
+
+        from mempalace.sweeper import parse_claude_jsonl
+
+        big_input = {"diff": "x" * 5000}  # well past the old 500-char cap
+        path = tmp_path / "session_tools.jsonl"
+        path.write_text(
+            _json.dumps(
+                {
+                    "type": "assistant",
+                    "timestamp": "2026-04-18T10:00:00Z",
+                    "sessionId": "tools-1",
+                    "uuid": "a-tool",
+                    "message": {
+                        "role": "assistant",
+                        "content": [
+                            {"type": "tool_use", "name": "Edit", "input": big_input},
+                        ],
+                    },
+                }
+            )
+            + "\n"
+        )
+
+        records = list(parse_claude_jsonl(str(path)))
+        assert len(records) == 1
+        content = records[0]["content"]
+        # The full 5000-char value must be present — no truncation marker,
+        # no [:500] slice. Look for the raw string in the serialized form.
+        assert big_input["diff"] in content, (
+            "tool_use input was truncated. The verbatim guarantee requires "
+            f"the full payload to round-trip. Got len={len(content)}."
+        )
+
+
+class TestSweeperTandem:
+    """The sweeper coordinates with other miners via max(timestamp)."""
+
+    def test_sweep_empty_palace_ingests_all_messages(self, mock_claude_jsonl, tmp_path):
+        from mempalace.sweeper import sweep
+
+        palace_path = str(tmp_path / "palace")
+        result = sweep(str(mock_claude_jsonl), palace_path)
+        assert result["drawers_added"] == 4, (
+            f"Empty palace: all 4 user/assistant messages should ingest. "
+            f"Got drawers_added={result['drawers_added']}."
+        )
+
+    def test_sweep_is_idempotent(self, mock_claude_jsonl, tmp_path):
+        """Running the sweep twice must not duplicate drawers."""
+        from mempalace.sweeper import sweep
+
+        palace_path = str(tmp_path / "palace")
+        first = sweep(str(mock_claude_jsonl), palace_path)
+        second = sweep(str(mock_claude_jsonl), palace_path)
+        assert first["drawers_added"] == 4
+        assert second["drawers_added"] == 0, (
+            f"Second sweep must be a no-op on unchanged data. "
+            f"Got drawers_added={second['drawers_added']} — "
+            "cursor logic is broken."
+        )
+
+    def test_sweep_resumes_from_cursor(self, tmp_path):
+        """If half the messages are already in the palace, sweep picks up
+        only the later half."""
+        from mempalace.sweeper import sweep
+
+        jsonl_path = tmp_path / "session.jsonl"
+        lines = [
+            {
+                "type": "user",
+                "timestamp": "2026-04-18T09:00:00Z",
+                "sessionId": "s1",
+                "uuid": "u1",
+                "message": {"role": "user", "content": "first"},
+            },
+            {
+                "type": "assistant",
+                "timestamp": "2026-04-18T09:00:01Z",
+                "sessionId": "s1",
+                "uuid": "a1",
+                "message": {"role": "assistant", "content": [{"type": "text", "text": "one"}]},
+            },
+        ]
+        jsonl_path.write_text("\n".join(json.dumps(x) for x in lines) + "\n")
+
+        palace_path = str(tmp_path / "palace")
+        first = sweep(str(jsonl_path), palace_path)
+        assert first["drawers_added"] == 2
+
+        # Append two more exchanges simulating live session growth.
+        more_lines = [
+            {
+                "type": "user",
+                "timestamp": "2026-04-18T09:05:00Z",
+                "sessionId": "s1",
+                "uuid": "u2",
+                "message": {"role": "user", "content": "second"},
+            },
+            {
+                "type": "assistant",
+                "timestamp": "2026-04-18T09:05:01Z",
+                "sessionId": "s1",
+                "uuid": "a2",
+                "message": {"role": "assistant", "content": [{"type": "text", "text": "two"}]},
+            },
+        ]
+        with open(jsonl_path, "a") as f:
+            for x in more_lines:
+                f.write(json.dumps(x) + "\n")
+
+        second = sweep(str(jsonl_path), palace_path)
+        assert second["drawers_added"] == 2, (
+            f"Second sweep should pick up only the 2 new exchanges, "
+            f"got {second['drawers_added']}. Cursor (max-timestamp) "
+            "coordination is broken."
+        )
+
+    def test_sweep_recovers_untaken_message_at_cursor_timestamp(self, tmp_path):
+        """Regression for Copilot PR #998 review: with a `<= cursor` skip,
+        any message sharing the max timestamp but not yet ingested (e.g.
+        crash mid-batch) would be lost forever. The skip must be `<` and
+        tie-break via deterministic drawer ID.
+
+        Scenario: three messages share timestamp T. First sweep ingests
+        two of them and the process dies before the third. Second sweep
+        must pick up the third — not skip it because cursor == T.
+        """
+        from mempalace.palace import get_collection
+        from mempalace.sweeper import (
+            _drawer_id_for_message,
+            parse_claude_jsonl,
+            sweep,
+        )
+
+        shared_ts = "2026-04-18T11:00:00Z"
+        lines = [
+            {
+                "type": "user",
+                "timestamp": shared_ts,
+                "sessionId": "s-tie",
+                "uuid": f"u-{i}",
+                "message": {"role": "user", "content": f"msg {i}"},
+            }
+            for i in range(3)
+        ]
+        jsonl_path = tmp_path / "tied.jsonl"
+        jsonl_path.write_text("\n".join(json.dumps(x) for x in lines) + "\n")
+
+        palace_path = str(tmp_path / "palace")
+        # Simulate a partial ingest: write 2 of 3 directly via the backend
+        # with the same drawer IDs the sweeper would use.
+        col = get_collection(palace_path, create=True)
+        recs = list(parse_claude_jsonl(str(jsonl_path)))
+        partial_ids = [_drawer_id_for_message(r["session_id"], r["uuid"]) for r in recs[:2]]
+        col.upsert(
+            ids=partial_ids,
+            documents=[f"USER: {r['content']}" for r in recs[:2]],
+            metadatas=[
+                {
+                    "session_id": r["session_id"],
+                    "timestamp": r["timestamp"],
+                    "message_uuid": r["uuid"],
+                    "role": r["role"],
+                    "ingest_mode": "sweep",
+                }
+                for r in recs[:2]
+            ],
+        )
+
+        # Now run the sweeper. It must pick up the 3rd message, not skip
+        # it because cursor == its timestamp.
+        result = sweep(str(jsonl_path), palace_path)
+        assert result["drawers_added"] == 1, (
+            f"Sweeper lost the untaken message at cursor timestamp. "
+            f"Expected drawers_added=1 (the 3rd record), got "
+            f"{result['drawers_added']}. Cursor skip is still `<=` "
+            "instead of `<`, or tie-break via drawer-id is broken."
+        )
+        assert result["drawers_already_present"] == 2, (
+            f"Expected 2 drawers already present (the partial ingest), "
+            f"got {result['drawers_already_present']}."
+        )
+
+
+class TestSweeperDrawerMetadata:
+    """Each drawer must carry the metadata the tandem-miner coordination
+    depends on: session_id, timestamp, uuid, role."""
+
+    def test_drawer_has_session_id_and_timestamp_metadata(self, mock_claude_jsonl, tmp_path):
+        from mempalace.sweeper import sweep
+        from mempalace.palace import get_collection
+
+        palace_path = str(tmp_path / "palace")
+        sweep(str(mock_claude_jsonl), palace_path)
+
+        col = get_collection(palace_path, create=False)
+        data = col.get(include=["metadatas"])
+        metas = data["metadatas"]
+        assert metas, "No drawers written"
+
+        for m in metas:
+            assert m.get("session_id") == "abc", f"Drawer missing session_id metadata: {m}"
+            assert m.get("timestamp"), f"Drawer missing timestamp metadata: {m}"
+            assert m.get("message_uuid"), f"Drawer missing message_uuid metadata: {m}"
+            assert m.get("role") in (
+                "user",
+                "assistant",
+            ), f"Drawer missing or wrong role metadata: {m}"

website/.vitepress/config.mts

@@ -11,6 +11,7 @@ function normalizeBase(base?: string): string {
 
 const docsBase = normalizeBase(process.env.DOCS_BASE || '/')
 const editBranch = process.env.DOCS_EDIT_BRANCH || 'main'
+const gaId = process.env.MEMPALACE_DOCS_GA_ID
 
 export default withMermaid(
   defineConfig({
@@ -22,10 +23,14 @@ export default withMermaid(
       ['link', { rel: 'icon', href: `${docsBase}mempalace_logo.png` }],
       ['link', { rel: 'preconnect', href: 'https://fonts.googleapis.com' }],
       ['link', { rel: 'preconnect', href: 'https://fonts.gstatic.com', crossorigin: '' }],
-      ['link', { href: 'https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&family=JetBrains+Mono:wght@400;500&display=swap', rel: 'stylesheet' }],
+      ['link', { href: 'https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&family=JetBrains+Mono:wght@300;400;500&family=Cormorant+Garamond:ital,wght@0,300;0,400;0,500;0,600;0,700;1,300;1,400&family=Geist:wght@300;400;500;600&display=swap', rel: 'stylesheet' }],
       ['meta', { property: 'og:title', content: 'MemPalace — AI Memory System' }],
       ['meta', { property: 'og:description', content: '96.6% LongMemEval recall. Zero API calls. Local, free, open source.' }],
       ['meta', { property: 'og:image', content: `${docsBase}mempalace_logo.png` }],
+      ...(gaId ? [
+        ['script', { async: '', src: `https://www.googletagmanager.com/gtag/js?id=${gaId}` }],
+        ['script', {}, `window.dataLayer = window.dataLayer || [];\nfunction gtag(){dataLayer.push(arguments);}\ngtag('js', new Date());\ngtag('config', '${gaId}');`],
+      ] as const : []),
     ],
 
     themeConfig: {

website/.vitepress/theme/Landing.vue

@@ -0,0 +1,29 @@
+<script setup>
+import { useLandingEffects } from './landing/useLandingEffects.js'
+import FolioHeader from './landing/FolioHeader.vue'
+import HeroSection from './landing/HeroSection.vue'
+import ForgettingSection from './landing/ForgettingSection.vue'
+import AnatomySection from './landing/AnatomySection.vue'
+import DialectSection from './landing/DialectSection.vue'
+import MechanicsSection from './landing/MechanicsSection.vue'
+import InstallSection from './landing/InstallSection.vue'
+import CatalogFooter from './landing/CatalogFooter.vue'
+import './landing/landing.css'
+
+useLandingEffects()
+</script>
+
+<template>
+  <div class="mempalace-landing">
+    <div class="page">
+      <FolioHeader />
+      <HeroSection />
+      <ForgettingSection />
+      <AnatomySection />
+      <DialectSection />
+      <MechanicsSection />
+      <InstallSection />
+      <CatalogFooter />
+    </div>
+  </div>
+</template>

website/.vitepress/theme/index.ts

@@ -1,6 +1,10 @@
 import DefaultTheme from 'vitepress/theme'
+import Landing from './Landing.vue'
 import './style.css'
 
 export default {
   extends: DefaultTheme,
+  enhanceApp({ app }) {
+    app.component('Landing', Landing)
+  },
 }

website/.vitepress/theme/landing/AnatomySection.vue

@@ -0,0 +1,100 @@
+<template>
+  <section v-pre id="anatomy" class="anatomy">
+    <div class="section-mark"><span class="roman">ii</span> <span>anatomy of a palace</span></div>
+
+    <div class="anatomy-head">
+      <div>
+        <span class="eyebrow">the method of loci, updated</span>
+        <h2 class="display">
+          Wings. Rooms. Closets. <em>Drawers.</em>
+        </h2>
+      </div>
+      <p class="lede">
+        An ancient memory technique, reworked for a machine. Broad categories
+        nest time-based groupings; time-based groupings bundle topics; topics
+        hold verbatim drawers. A symbolic index lets the model scan thousands
+        of drawers in a single pass and open only the ones it needs.
+      </p>
+    </div>
+
+    <div class="anatomy-diagram">
+      <article class="stratum">
+        <span class="n">W — wing</span>
+        <h3>The <em>Wings</em></h3>
+        <p class="sub">people · projects · topics</p>
+        <p>A broad region of the palace, keyed to a real entity — a person by name, a project by codename, a domain of your life. Entity-first, always.</p>
+        <div class="diagram">
+          <svg viewBox="0 0 200 80" fill="none" stroke="currentColor" stroke-width="1" style="color:var(--prism);">
+            <rect x="5" y="20" width="190" height="50" opacity="0.4"/>
+            <rect x="15" y="28" width="50" height="34" />
+            <rect x="75" y="28" width="50" height="34" />
+            <rect x="135" y="28" width="50" height="34" />
+            <line x1="5" y1="12" x2="195" y2="12" stroke-dasharray="2 3" opacity="0.5"/>
+          </svg>
+        </div>
+      </article>
+
+      <article class="stratum">
+        <span class="n">R — room</span>
+        <h3>The <em>Rooms</em></h3>
+        <p class="sub">days · sessions · threads</p>
+        <p>Inside a wing sit rooms — discrete units of time. One room per day, or one per session. Walk the corridor and the palace unfolds chronologically, room by room.</p>
+        <div class="diagram">
+          <svg viewBox="0 0 200 80" fill="none" stroke="currentColor" stroke-width="1" style="color:var(--prism);">
+            <rect x="10" y="20" width="36" height="44" />
+            <rect x="56" y="20" width="36" height="44" />
+            <rect x="102" y="20" width="36" height="44" />
+            <rect x="148" y="20" width="36" height="44" />
+            <line x1="10" y1="70" x2="184" y2="70" stroke-dasharray="1 3" opacity="0.6"/>
+          </svg>
+        </div>
+      </article>
+
+      <article class="stratum">
+        <span class="n">C — closet</span>
+        <h3>The <em>Closets</em></h3>
+        <p class="sub">topics · threads · bundles</p>
+        <p>Inside a room, closets group related drawers by topic or thread. Open one closet and you see every drawer on that subject together — no need to walk the whole room.</p>
+        <div class="diagram">
+          <svg viewBox="0 0 200 80" fill="none" stroke="currentColor" stroke-width="1" style="color:var(--prism);">
+            <g class="closet">
+              <rect x="10" y="14" width="54" height="52" />
+              <line x1="37" y1="14" x2="37" y2="66" opacity="0.5"/>
+              <circle cx="33" cy="40" r="1.2" fill="currentColor"/>
+              <circle cx="41" cy="40" r="1.2" fill="currentColor"/>
+            </g>
+            <g class="closet">
+              <rect x="73" y="14" width="54" height="52" />
+              <line x1="100" y1="14" x2="100" y2="66" opacity="0.5"/>
+              <circle cx="96" cy="40" r="1.2" fill="currentColor"/>
+              <circle cx="104" cy="40" r="1.2" fill="currentColor"/>
+            </g>
+            <g class="closet">
+              <rect x="136" y="14" width="54" height="52" />
+              <line x1="163" y1="14" x2="163" y2="66" opacity="0.5"/>
+              <circle cx="159" cy="40" r="1.2" fill="currentColor"/>
+              <circle cx="167" cy="40" r="1.2" fill="currentColor"/>
+            </g>
+          </svg>
+        </div>
+      </article>
+
+      <article class="stratum">
+        <span class="n">D — drawer</span>
+        <h3>The <em>Drawers</em></h3>
+        <p class="sub">verbatim · permanent · exact</p>
+        <p>Each room holds drawers. A drawer is a single chunk of verbatim content — the exact words, untouched. The palace's promise is kept here.</p>
+        <div class="diagram">
+          <svg viewBox="0 0 200 80" fill="none" stroke="currentColor" stroke-width="1" style="color:var(--prism);">
+            <rect x="40" y="14" width="120" height="16" />
+            <rect x="40" y="34" width="120" height="16" />
+            <rect x="40" y="54" width="120" height="16" />
+            <circle cx="150" cy="22" r="1.5" fill="currentColor"/>
+            <circle cx="150" cy="42" r="1.5" fill="currentColor"/>
+            <circle cx="150" cy="62" r="1.5" fill="currentColor"/>
+          </svg>
+        </div>
+      </article>
+    </div>
+  </section>
+</template>

website/.vitepress/theme/landing/CatalogFooter.vue

@@ -0,0 +1,45 @@
+<template>
+  <footer v-pre class="catalog">
+    <form class="waitlist waitlist-footer" data-source="footer" novalidate>
+      <div class="waitlist-head">
+        <span class="waitlist-pulse" aria-hidden="true"></span>
+        <span class="waitlist-eyebrow">Last call &middot; subscribe for updates</span>
+      </div>
+      <div class="waitlist-row">
+        <input type="email" class="waitlist-input" name="email" placeholder="you@example.com" autocomplete="email" aria-label="Email address" required />
+        <button type="submit" class="waitlist-submit">
+          <span class="waitlist-label-default">Join the list</span>
+          <span class="waitlist-label-pending" aria-hidden="true">Joining…</span>
+          <svg class="waitlist-arrow" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.6" aria-hidden="true"><path d="M5 12h14M13 6l6 6-6 6"/></svg>
+          <svg class="waitlist-check" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.8" aria-hidden="true"><path d="M5 12l5 5 9-11"/></svg>
+        </button>
+      </div>
+      <p class="waitlist-msg" aria-live="polite"></p>
+    </form>
+
+    <div class="catalog-card">
+      <div>
+        <p class="catalog-title">MemPalace <em>&mdash;</em> a memory palace for AI.</p>
+        <p class="catalog-desc">Verbatim storage, local-first, zero telemetry. Built for people who believe their words are theirs.</p>
+      </div>
+      <div>
+        <h4>Documentation</h4>
+        <ul>
+          <li><a href="/guide/getting-started">Getting started</a></li>
+          <li><a href="/concepts/the-palace">The palace</a></li>
+          <li><a href="/reference/cli">CLI reference</a></li>
+          <li><a href="/reference/benchmarks">Benchmarks</a></li>
+        </ul>
+      </div>
+      <div>
+        <h4>The project</h4>
+        <ul>
+          <li><a href="https://github.com/MemPalace/mempalace">GitHub</a></li>
+          <li><a href="https://github.com/MemPalace/mempalace/blob/main/README.md">Readme</a></li>
+          <li><a href="https://github.com/MemPalace/mempalace/blob/main/ROADMAP.md">Roadmap</a></li>
+          <li><a href="https://github.com/MemPalace/mempalace/blob/main/CHANGELOG.md">Changelog</a></li>
+        </ul>
+      </div>
+    </div>
+  </footer>
+</template>

website/.vitepress/theme/landing/DialectSection.vue

@@ -0,0 +1,64 @@
+<template>
+  <section v-pre id="dialect" class="dialect">
+    <div class="section-mark"><span class="roman">iii</span> <span>the aaak dialect</span></div>
+
+    <div class="dialect-head">
+      <span class="eyebrow">index &larr; verbatim</span>
+      <h2 class="display">
+        A compressed symbolic language <em>for finding</em>, not remembering.
+      </h2>
+      <p class="lede">
+        The content stays verbatim — always. The <em>index</em> above it is written
+        in AAAK: a dense symbolic dialect an LLM can scan at a glance. Thousands
+        of entries, one pass, exact drawer located.
+      </p>
+    </div>
+
+    <div class="dialect-grid">
+      <article class="slab">
+        <header class="card-head">
+          <span class="l">drawer · D-007</span>
+          <span>verbatim · exact · permanent</span>
+        </header>
+        <p class="label">The drawer, as stored.</p>
+        <p>
+          "My son's name is <strong>Noah</strong>. He turns <strong>six</strong>
+          on <strong>September 12th</strong>. He loves dinosaurs —
+          especially the <strong>therizinosaurus</strong> because of the
+          claws. We want to do a small party at <strong>the park on Glebe
+          Point Road</strong>, maybe eight kids."
+        </p>
+        <p style="color:var(--ice-ghost); font-size: 13.5px; font-family: var(--f-mono); letter-spacing: 0.05em; margin-top:1.5rem;">
+          &mdash; kept as spoken. never rewritten.
+        </p>
+      </article>
+
+      <div class="dialect-arrow" aria-hidden="true">
+        <svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.3">
+          <path d="M12 3v18M7 8l5-5 5 5M7 16l5 5 5-5"/>
+        </svg>
+        <span>index · AAAK</span>
+      </div>
+
+      <article class="slab mono">
+        <header class="card-head">
+          <span class="l">index · AAAK</span>
+          <span>indexes · compressed · addressable</span>
+        </header>
+        <p class="label">The pointer, as indexed.</p>
+<pre><span class="c">§ W-042/R-11/D-007</span>
+<span class="k">@p</span> <span class="t">noah</span>~<span class="v">son.age=6</span>~<span class="v">dob=09-12</span>
+<span class="k">@l</span> <span class="t">glebe-pt-rd.park</span>
+<span class="k">@e</span> <span class="t">birthday</span>~<span class="v">party(n≈8)</span>
+<span class="k">@i</span> <span class="t">therizinosaurus</span>~<span class="v">claws</span>
+<span class="k">@t</span> <span class="v">2026-04-14T09:41</span>
+<span class="c">§ ptr → D-007 (verbatim)</span></pre>
+      </article>
+    </div>
+
+    <p class="dialect-caption">
+      Dense compression on the pointer layer. Full fidelity on the content
+      layer. You get speed without ever losing a word.
+    </p>
+  </section>
+</template>

website/.vitepress/theme/landing/FolioHeader.vue

@@ -0,0 +1,16 @@
+<template>
+  <header v-pre class="folio" role="banner">
+    <div class="mark" aria-label="MemPalace">
+      <img src="/mempalace_logo.png" alt="" aria-hidden="true" />
+      <span>MemPalace</span>
+    </div>
+    <nav class="right" aria-label="Primary">
+      <a href="#anatomy" class="hide-mobile">Anatomy</a>
+      <a href="#dialect" class="hide-mobile">Dialect</a>
+      <a href="#mechanics" class="hide-mobile">Mechanics</a>
+      <a href="#install" class="hide-mobile">Install</a>
+      <a href="/guide/getting-started">Docs</a>
+      <a href="https://github.com/MemPalace/mempalace">GitHub ↗</a>
+    </nav>
+  </header>
+</template>

website/.vitepress/theme/landing/ForgettingSection.vue

@@ -0,0 +1,43 @@
+<template>
+  <section v-pre id="forgetting" class="forgetting">
+    <div class="section-mark"><span class="roman">i</span> <span>the forgetting</span></div>
+
+    <header class="forgetting-head">
+      <div class="copy">
+        <span class="eyebrow">before &middot; after</span>
+        <h2 class="display">
+          The same conversation, <em>twice.</em>
+        </h2>
+        <p class="lede" style="margin:0;">
+          Scroll down and watch. On the left, a model without memory. On the right,
+          the same model with MemPalace. The words are identical — until two weeks
+          pass.
+        </p>
+      </div>
+      <button class="replay" id="replay-demo" type="button" aria-label="Replay the demo">
+        <svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.5" aria-hidden="true"><path d="M4 4v6h6"/><path d="M20 20v-6h-6"/><path d="M4 10a8 8 0 0114-5l2 3"/><path d="M20 14a8 8 0 01-14 5l-2-3"/></svg>
+        replay
+      </button>
+    </header>
+
+    <div class="forgetting-compare" id="forgetting-compare" aria-label="Comparison demo">
+      <article class="demo-pane demo-forget">
+        <header>
+          <span class="pane-tag">without mempalace</span>
+          <span class="pane-meta">session <em>resets</em> &middot; no recall</span>
+        </header>
+        <div class="chat" data-pane="forget" aria-live="polite"></div>
+      </article>
+
+      <div class="divider" aria-hidden="true"></div>
+
+      <article class="demo-pane demo-remember">
+        <header>
+          <span class="pane-tag">with mempalace</span>
+          <span class="pane-meta">verbatim &middot; retrieved <em>instantly</em></span>
+        </header>
+        <div class="chat" data-pane="remember" aria-live="polite"></div>
+      </article>
+    </div>
+  </section>
+</template>

website/.vitepress/theme/landing/HeroSection.vue

@@ -0,0 +1,81 @@
+<template>
+  <section v-pre class="hero" id="hero">
+    <span class="corner-ticks" aria-hidden="true"><span></span></span>
+
+    <div class="hero-inner">
+      <div class="hero-copy">
+        <h1 class="display">
+          <span class="line">Memory is</span>
+          <span class="line line-2">identity.</span>
+        </h1>
+        <p class="lede">
+          An AI that forgets cannot know you. MemPalace keeps every word you have
+          shared — verbatim, on your machine, permanent. Designed for total
+          recall.
+        </p>
+        <form class="waitlist waitlist-hero" data-source="hero" novalidate>
+          <div class="waitlist-head">
+            <span class="waitlist-pulse" aria-hidden="true"></span>
+            <span class="waitlist-eyebrow">Subscribe for updates</span>
+          </div>
+          <div class="waitlist-row">
+            <input
+              type="email"
+              class="waitlist-input"
+              name="email"
+              placeholder="you@example.com"
+              autocomplete="email"
+              aria-label="Email address"
+              required
+            />
+            <button type="submit" class="waitlist-submit">
+              <span class="waitlist-label-default">Join the list</span>
+              <span class="waitlist-label-pending" aria-hidden="true">Joining…</span>
+              <svg class="waitlist-arrow" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.6" aria-hidden="true">
+                <path d="M5 12h14M13 6l6 6-6 6"/>
+              </svg>
+              <svg class="waitlist-check" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.8" aria-hidden="true">
+                <path d="M5 12l5 5 9-11"/>
+              </svg>
+            </button>
+          </div>
+          <p class="waitlist-msg" aria-live="polite"></p>
+        </form>
+
+        <div class="hero-secondary">
+          <a href="/guide/getting-started">Read the docs</a>
+          <span class="sep" aria-hidden="true">·</span>
+          <a href="https://github.com/MemPalace/mempalace">GitHub ↗</a>
+        </div>
+      </div>
+
+      <!-- Palace video visual -->
+      <div class="palace-stage" aria-hidden="true">
+        <div class="halo"></div>
+
+        <div class="stars">
+          <i style="top:12%; left:22%;  --t:5s;   --d:0.0s"></i>
+          <i style="top:18%; left:74%;  --t:6s;   --d:1.2s"></i>
+          <i style="top:34%; left:8%;   --t:4s;   --d:0.6s"></i>
+          <i style="top:44%; left:88%;  --t:7s;   --d:0.3s"></i>
+          <i style="top:62%; left:14%;  --t:5.5s; --d:1.8s"></i>
+          <i style="top:72%; left:82%;  --t:4.5s; --d:0.9s"></i>
+          <i style="top:82%; left:38%;  --t:6.2s; --d:2.4s"></i>
+          <i style="top:28%; left:52%;  --t:5.2s; --d:3.0s"></i>
+          <i style="top:88%; left:60%;  --t:4.8s; --d:1.5s"></i>
+          <i style="top:6%;  left:48%;  --t:6.8s; --d:0.4s"></i>
+        </div>
+
+        <video
+          class="palace-video"
+          src="/hero_video.mp4"
+          autoplay
+          muted
+          loop
+          playsinline
+          disablepictureinpicture
+        ></video>
+      </div>
+    </div>
+  </section>
+</template>

website/.vitepress/theme/landing/InstallSection.vue

@@ -0,0 +1,38 @@
+<template>
+  <section v-pre id="install" class="install">
+    <div class="section-mark" style="left:50%; transform:translateX(-50%);"><span class="roman">v</span> <span>begin</span></div>
+    <span class="eyebrow" style="justify-content:center;">open a drawer</span>
+    <h2 class="display">
+      Build your <em>palace.</em>
+    </h2>
+    <p class="lede" style="margin-left:auto;margin-right:auto;text-align:center;">
+      One command to install. One to initialize. Your words — yours, permanent,
+      instantly recallable — from that moment on.
+    </p>
+
+    <div class="terminal" role="figure" aria-label="Installation commands">
+      <div class="terminal-head">
+        <span class="lights"><i></i><i></i><i></i></span>
+        <span>~/mempalace &middot; bash</span>
+      </div>
+<pre><span class="prompt">$</span> pip install -e <span class="dim">".[dev]"</span>
+<span class="c">Successfully installed mempalace</span>
+<span class="prompt">$</span> mempalace init
+<span class="ok">  ✓</span> palace created at <span class="dim">~/.mempalace</span>
+<span class="ok">  ✓</span> hooks registered <span class="dim">(stop, precompact)</span>
+<span class="ok">  ✓</span> knowledge graph initialized
+<span class="prompt">$</span> mempalace mine <span class="dim">./notes</span>
+<span class="ok">  ✓</span> filed · <span class="c">W-001/R-01/D-001</span></pre>
+    </div>
+
+    <div class="install-cta">
+      <a href="/guide/getting-started" class="btn btn-primary">
+        Read the docs
+        <svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.5"><path d="M5 12h14M13 6l6 6-6 6"/></svg>
+      </a>
+      <a href="https://github.com/MemPalace/mempalace" class="btn">
+        Visit the repository
+      </a>
+    </div>
+  </section>
+</template>

website/.vitepress/theme/landing/MechanicsSection.vue

@@ -0,0 +1,83 @@
+<template>
+  <section v-pre id="mechanics">
+    <div class="section-mark"><span class="roman">iv</span> <span>how it works</span></div>
+
+    <div class="mechanics-head">
+      <span class="eyebrow">mechanism · architecture</span>
+      <h2 class="display">
+        Four pieces. <em>No cloud.</em> No keys.
+      </h2>
+    </div>
+
+    <div class="mechanics">
+      <article class="mech">
+        <div class="icon" aria-hidden="true">
+          <svg viewBox="0 0 48 48" fill="none" stroke="currentColor" stroke-width="1.3">
+            <rect x="8" y="10" width="32" height="22" rx="1"/>
+            <path d="M8 16h32"/>
+            <g class="mech-bars">
+              <path d="M14 24h20"/>
+              <path d="M14 28h12"/>
+            </g>
+            <path d="M16 38h16M20 32v6M28 32v6"/>
+            <circle class="mech-led" cx="36" cy="13.5" r="1.1" fill="currentColor"/>
+          </svg>
+        </div>
+        <span class="eyebrow no-rule"><span class="n">— 01</span></span>
+        <h3>Local-<em>first</em></h3>
+        <p>ChromaDB on disk. SQLite for the knowledge graph. Nothing is uploaded. Nothing is synced. Your palace lives under a single directory on your machine.</p>
+        <div class="metric">path · <b>~/.mempalace</b></div>
+      </article>
+
+      <article class="mech">
+        <div class="icon" aria-hidden="true">
+          <svg viewBox="0 0 48 48" fill="none" stroke="currentColor" stroke-width="1.3">
+            <circle class="mech-ring" cx="24" cy="24" r="14"/>
+            <path class="mech-plus" d="M16 24h16M24 16v16"/>
+            <path class="mech-slash" d="M10 10l28 28" stroke-width="1.5"/>
+          </svg>
+        </div>
+        <span class="eyebrow no-rule"><span class="n">— 02</span></span>
+        <h3>Zero <em>API</em></h3>
+        <p>Extraction, chunking, and embedding all run locally. No OpenAI key, no Anthropic key, no sentence-transformers endpoint. The memory works even offline, on a plane.</p>
+        <div class="metric">keys required · <b>none</b></div>
+      </article>
+
+      <article class="mech">
+        <div class="icon" aria-hidden="true">
+          <svg viewBox="0 0 48 48" fill="none" stroke="currentColor" stroke-width="1.3">
+            <path d="M8 36V18l8-8h16l8 8v18"/>
+            <path d="M8 36h32"/>
+            <g class="mech-gear">
+              <circle cx="24" cy="26" r="4"/>
+              <path d="M24 22v-4M24 30v4M20 26h-4M28 26h4"/>
+            </g>
+          </svg>
+        </div>
+        <span class="eyebrow no-rule"><span class="n">— 03</span></span>
+        <h3>Background <em>hooks</em></h3>
+        <p>Filing and indexing happen silently through Claude Code hooks. On session end, on pre-compaction. You write. The palace fills itself behind the curtain.</p>
+        <div class="metric">hook budget · <b>&lt;500 ms</b></div>
+      </article>
+
+      <article class="mech">
+        <div class="icon" aria-hidden="true">
+          <svg viewBox="0 0 48 48" fill="none" stroke="currentColor" stroke-width="1.3">
+            <path class="mech-edges" d="M12 14l10 10M36 12L26 24M14 36l8-8M36 34l-10-6" opacity="0.6"/>
+            <g class="mech-nodes">
+              <circle cx="10" cy="12" r="3"/>
+              <circle cx="38" cy="10" r="3"/>
+              <circle cx="24" cy="26" r="3"/>
+              <circle cx="12" cy="38" r="3"/>
+              <circle cx="38" cy="36" r="3"/>
+            </g>
+          </svg>
+        </div>
+        <span class="eyebrow no-rule"><span class="n">— 04</span></span>
+        <h3>Temporal <em>graph</em></h3>
+        <p>Relationships across entities with valid-from and valid-to dates. Who worked on what. When did this change. Facts that were true then, and may not be now.</p>
+        <div class="metric">store · <b>sqlite</b></div>
+      </article>
+    </div>
+  </section>
+</template>

website/.vitepress/theme/landing/useLandingEffects.js

@@ -0,0 +1,406 @@
+import { onMounted, onBeforeUnmount } from 'vue'
+
+export function useLandingEffects() {
+// Shared cleanup registry — IIFEs push disconnect/removeEventListener thunks
+// here so onBeforeUnmount can tear everything down on SPA nav.
+const cleanups = []
+
+onMounted(() => {
+  if (typeof document === 'undefined') return
+
+  // Hide VitePress chrome while the landing component is live, restore on leave.
+  document.body.classList.add('mempalace-active')
+
+  /* ---------- Waitlist submission ---------- */
+  ;(function initWaitlist(){
+    const ENDPOINT = 'https://br.staging.mempalaceofficial.com/waitlist'
+    const forms = document.querySelectorAll('.mempalace-landing .waitlist')
+    const emailRe = /^[^\s@]+@[^\s@]+\.[^\s@]+$/
+
+    forms.forEach(form => {
+      const input  = form.querySelector('.waitlist-input')
+      const button = form.querySelector('.waitlist-submit')
+      const msg    = form.querySelector('.waitlist-msg')
+      const source = form.dataset.source || 'landing'
+
+      function setState(state, text) {
+        form.classList.remove('is-pending', 'is-success', 'is-error')
+        if (state) form.classList.add('is-' + state)
+        if (text != null) msg.textContent = text
+      }
+
+      const onSubmit = async (e) => {
+        e.preventDefault()
+        if (form.classList.contains('is-success') || form.classList.contains('is-pending')) return
+
+        const email = (input.value || '').trim()
+        if (!emailRe.test(email)) {
+          setState('error', 'Please provide a valid email address.')
+          input.focus()
+          return
+        }
+
+        setState('pending', 'Sending…')
+        button.disabled = true
+        input.disabled = true
+
+        try {
+          const res = await fetch(ENDPOINT, {
+            method: 'POST',
+            headers: { 'Content-Type': 'application/json' },
+            body: JSON.stringify({ email, source })
+          })
+          let data = null
+          try { data = await res.json() } catch (_) { /* no body */ }
+
+          if (res.ok) {
+            setState('success', (data && data.message) || "Success! You're on the list for updates.")
+            // keep inputs disabled so they can't resubmit accidentally
+            input.value = email
+            return
+          }
+
+          if (res.status === 429) {
+            setState('error', 'Whoa — slow down a moment, then try again.')
+          } else if (res.status === 400) {
+            setState('error', (data && data.message) || 'Please provide a valid email address.')
+          } else {
+            setState('error', (data && data.message) || 'Something went wrong. Please try again later.')
+          }
+          button.disabled = false
+          input.disabled = false
+        } catch (_err) {
+          setState('error', 'Network error — please try again.')
+          button.disabled = false
+          input.disabled = false
+        }
+      }
+
+      const onInput = () => {
+        if (form.classList.contains('is-error')) setState(null, '')
+      }
+
+      form.addEventListener('submit', onSubmit)
+      input.addEventListener('input', onInput)
+      cleanups.push(() => {
+        form.removeEventListener('submit', onSubmit)
+        input.removeEventListener('input', onInput)
+      })
+    })
+  })()
+
+
+
+  /* ---------- Reveal-on-scroll for cards ---------- */
+  ;(function(){
+    if (!('IntersectionObserver' in window)) return
+    const items = document.querySelectorAll('.mempalace-landing .stratum, .mempalace-landing .mech, .mempalace-landing .slab')
+    items.forEach(el => {
+      el.style.opacity = '0'
+      el.style.transform = 'translateY(20px)'
+      el.style.transition = 'opacity 0.9s ease, transform 0.9s ease'
+    })
+    const io = new IntersectionObserver((entries) => {
+      entries.forEach((entry) => {
+        if (entry.isIntersecting){
+          const idx = [...entry.target.parentElement.children].indexOf(entry.target)
+          entry.target.style.transitionDelay = (idx * 80) + 'ms'
+          entry.target.style.opacity = '1'
+          entry.target.style.transform = 'translateY(0)'
+          io.unobserve(entry.target)
+        }
+      })
+    }, { rootMargin: '0px 0px -80px 0px' })
+    items.forEach(el => io.observe(el))
+    cleanups.push(() => io.disconnect())
+  })()
+
+  /* ---------- Forgetting demo ---------- */
+  ;(function initForgettingDemo(){
+    const compare = document.getElementById('forgetting-compare')
+    if (!compare) return
+    const leftChat  = compare.querySelector('[data-pane="forget"]')
+    const rightChat = compare.querySelector('[data-pane="remember"]')
+    const replayBtn = document.getElementById('replay-demo')
+    const reduced   = window.matchMedia('(prefers-reduced-motion: reduce)').matches
+
+    const delay = ms => new Promise(r => setTimeout(r, reduced ? Math.min(ms, 60) : ms))
+
+    function clear() {
+      leftChat.innerHTML = ''
+      rightChat.innerHTML = ''
+      if (replayBtn) replayBtn.classList.remove('visible')
+    }
+
+    function addMsg(chat, who, opts = {}) {
+      const row = document.createElement('div')
+      row.className = 'msg ' + (who === 'You' ? 'you' : 'ai')
+      if (opts.id) row.dataset.id = opts.id
+      row.innerHTML = '<span class="who">' + who + '</span><span class="body"></span>'
+      chat.appendChild(row)
+      chat.scrollTop = chat.scrollHeight
+      return row
+    }
+
+    async function typeInto(row, text, speed = 14) {
+      const body = row.querySelector('.body')
+      const parts = text.split(/(<[^>]+>)/)
+      row.classList.add('typing')
+      for (const part of parts) {
+        if (!part) continue
+        if (part.startsWith('<')) { body.insertAdjacentHTML('beforeend', part); continue }
+        for (const ch of part) {
+          body.insertAdjacentText('beforeend', ch)
+          if (!reduced) await delay(speed + (Math.random() < 0.08 ? 40 : 0))
+        }
+      }
+      row.classList.remove('typing')
+    }
+
+    function addDivider(chat, text) {
+      const d = document.createElement('div')
+      d.className = 'divider-time'
+      d.textContent = '— ' + text + ' —'
+      chat.appendChild(d)
+      return d
+    }
+
+    function addRetrieval(chat, callNumber, ms) {
+      const row = document.createElement('div')
+      row.className = 'retrieval'
+      row.innerHTML =
+        '<span class="who">mem</span>' +
+        '<span class="l">retrieved &middot; <span class="r">' + callNumber + '</span></span>' +
+        '<span>' + ms + '&nbsp;ms</span>'
+      chat.appendChild(row)
+      return row
+    }
+
+    function addStamp(chat, text, callNumber) {
+      const el = document.createElement('div')
+      el.className = 'stamp'
+      el.innerHTML = '<span>— ' + text + '</span>' +
+        (callNumber ? '<span class="call">' + callNumber + '</span>' : '')
+      chat.appendChild(el)
+      return el
+    }
+
+    function disintegrate(target) {
+      return new Promise(resolve => {
+        const parent = target.closest('.chat')
+        if (!parent) { resolve(); return }
+        const parentRect = parent.getBoundingClientRect()
+        const style = getComputedStyle(target)
+        const font = style.font ||
+          (style.fontStyle + ' ' + style.fontWeight + ' ' + style.fontSize + '/' + style.lineHeight + ' ' + style.fontFamily)
+        const color = style.color
+
+        let overlay = parent.querySelector('.dust-overlay')
+        if (!overlay) {
+          overlay = document.createElement('div')
+          overlay.className = 'dust-overlay'
+          parent.appendChild(overlay)
+        }
+
+        const walker = document.createTreeWalker(target, NodeFilter.SHOW_TEXT)
+        const range = document.createRange()
+        const spans = []
+        let node
+        while ((node = walker.nextNode())) {
+          const chars = node.textContent
+          for (let i = 0; i < chars.length; i++) {
+            if (chars[i] === ' ') continue
+            range.setStart(node, i)
+            range.setEnd(node, i + 1)
+            const r = range.getBoundingClientRect()
+            if (r.width === 0 || r.height === 0) continue
+            const span = document.createElement('span')
+            span.className = 'dust'
+            span.textContent = chars[i]
+            span.style.left = (r.left - parentRect.left) + 'px'
+            span.style.top  = (r.top  - parentRect.top)  + 'px'
+            span.style.width  = r.width  + 'px'
+            span.style.height = r.height + 'px'
+            span.style.font = font
+            span.style.color = color
+            span.style.opacity = '1'
+            span.style.transform = 'translate(0,0)'
+            span.style.transitionDuration = (1500 + Math.random() * 900) + 'ms'
+            overlay.appendChild(span)
+            spans.push(span)
+          }
+        }
+
+        target.style.transition = 'color 0.35s ease, opacity 0.35s ease'
+        target.style.color = 'transparent'
+
+        void overlay.offsetHeight
+        const cx = parentRect.width / 2
+        spans.forEach((s) => {
+          s.style.transitionDelay = (Math.random() * 500) + 'ms'
+          const x0 = parseFloat(s.style.left)
+          const dx = (x0 - cx) * 0.06 + (Math.random() - 0.5) * 36
+          const dy = 30 + Math.random() * 80
+          const rot = (Math.random() - 0.5) * 44
+          s.style.transform = 'translate(' + dx + 'px,' + dy + 'px) rotate(' + rot + 'deg)'
+          s.style.opacity = '0'
+          s.style.filter = 'blur(2px)'
+        })
+
+        setTimeout(() => {
+          spans.forEach(s => s.remove())
+          resolve()
+        }, reduced ? 200 : 2600)
+      })
+    }
+
+    const NOAH_TEXT = "My son's name is Noah. He turns six on September 12th."
+
+    async function runForget() {
+      const you1 = addMsg(leftChat, 'You', { id: 'noah' })
+      await delay(200)
+      await typeInto(you1, NOAH_TEXT, 16)
+      await delay(500)
+      const ai1 = addMsg(leftChat, 'Model')
+      await typeInto(ai1, "Noted. I'll remember that for next time we talk.", 14)
+      await delay(900)
+      addDivider(leftChat, 'two weeks later')
+      await delay(700)
+      const you2 = addMsg(leftChat, 'You')
+      await typeInto(you2, "Help me plan Noah's birthday.", 18)
+      await delay(700)
+      const target = leftChat.querySelector('.msg[data-id="noah"] .body')
+      if (target) await disintegrate(target)
+      await delay(250)
+      const ai2 = addMsg(leftChat, 'Model')
+      await typeInto(ai2, "Of course. Who is Noah? How old is he turning?", 16)
+      await delay(500)
+      addStamp(leftChat, 'forgotten.')
+    }
+
+    async function runRemember() {
+      const you1 = addMsg(rightChat, 'You', { id: 'noah' })
+      await delay(200)
+      await typeInto(you1, NOAH_TEXT, 16)
+      await delay(500)
+      const ai1 = addMsg(rightChat, 'Model')
+      await typeInto(ai1, "Noted. Filed — <strong>W-042/R-01/D-003</strong>.", 14)
+      await delay(900)
+      addDivider(rightChat, 'two weeks later')
+      await delay(700)
+      const you2 = addMsg(rightChat, 'You')
+      await typeInto(you2, "Help me plan Noah's birthday.", 18)
+      await delay(600)
+      addRetrieval(rightChat, 'W-042/R-01/D-003', 42)
+      await delay(700)
+      const ai2 = addMsg(rightChat, 'Model')
+      await typeInto(ai2,
+        "Of course — <strong>Noah</strong> turns <strong>six</strong> on <strong>September 12th</strong>. " +
+        "You mentioned he loves the <strong>therizinosaurus</strong>, and a park on " +
+        "<strong>Glebe Point Road</strong>. Shall we build from there?",
+        11)
+      await delay(500)
+      addStamp(rightChat, 'remembered.', 'W-042/R-01/D-003')
+    }
+
+    let running = { forget: false, remember: false }
+    let started = { forget: false, remember: false }
+
+    async function runBoth() {
+      if (running.forget || running.remember) return
+      running.forget = running.remember = true
+      started.forget = started.remember = true
+      clear()
+      await delay(200)
+      await Promise.all([runForget(), runRemember()])
+      running.forget = running.remember = false
+      if (replayBtn) replayBtn.classList.add('visible')
+    }
+
+    async function runSide(side) {
+      if (running[side] || started[side]) return
+      running[side] = true
+      started[side] = true
+      const chat = side === 'forget' ? leftChat : rightChat
+      chat.innerHTML = ''
+      await delay(200)
+      await (side === 'forget' ? runForget() : runRemember())
+      running[side] = false
+      if (started.forget && started.remember && !running.forget && !running.remember && replayBtn) {
+        replayBtn.classList.add('visible')
+      }
+    }
+
+    function resetAll() {
+      started.forget = started.remember = false
+      clear()
+    }
+
+    const stackedMQ = window.matchMedia('(max-width: 900px)')
+    const isStacked = () => stackedMQ.matches
+
+    function observeOnce(el, onReach) {
+      if (!('IntersectionObserver' in window)) { onReach(); return null }
+      let done = false
+      const io = new IntersectionObserver((entries) => {
+        entries.forEach(entry => {
+          if (done || !entry.isIntersecting) return
+          const rect = entry.boundingClientRect
+          const elementCoverage  = entry.intersectionRatio
+          const viewportCoverage = entry.intersectionRect.height / window.innerHeight
+          const mostlyVisible  = elementCoverage >= 0.65
+          const dominatesView  = viewportCoverage >= 0.60 && rect.top <= window.innerHeight * 0.15
+          if (mostlyVisible || dominatesView) {
+            done = true
+            onReach()
+            io.disconnect()
+          }
+        })
+      }, {
+        threshold: [0.1, 0.25, 0.4, 0.55, 0.7, 0.85, 1.0],
+        rootMargin: '-8% 0px -8% 0px'
+      })
+      io.observe(el)
+      return io
+    }
+
+    let observers = []
+    function disconnectObservers() {
+      observers.forEach(io => io && io.disconnect())
+      observers = []
+    }
+
+    function armObservers() {
+      disconnectObservers()
+      if (isStacked()) {
+        observers.push(observeOnce(compare.querySelector('.demo-forget'),   () => runSide('forget')))
+        observers.push(observeOnce(compare.querySelector('.demo-remember'), () => runSide('remember')))
+      } else {
+        observers.push(observeOnce(compare, runBoth))
+      }
+    }
+
+    const onReplayClick = () => {
+      resetAll()
+      armObservers()
+    }
+    if (replayBtn) replayBtn.addEventListener('click', onReplayClick)
+
+    armObservers()
+
+    cleanups.push(() => {
+      disconnectObservers()
+      if (replayBtn) replayBtn.removeEventListener('click', onReplayClick)
+    })
+  })()
+})
+
+onBeforeUnmount(() => {
+  if (typeof document === 'undefined') return
+  document.body.classList.remove('mempalace-active')
+  while (cleanups.length) {
+    const fn = cleanups.pop()
+    try { fn() } catch (_) { /* swallow — teardown best-effort */ }
+  }
+})
+}

website/guide/claude-code.md

@@ -15,7 +15,7 @@ Restart Claude Code, then type `/skills` to verify "mempalace" appears.
 
 With the plugin installed, Claude Code automatically:
 - Starts the MemPalace MCP server on launch
-- Has access to all 19 tools
+- Has access to all 29 tools
 - Learns the AAAK dialect and memory protocol from the `mempalace_status` response
 - Searches the palace before answering questions about past work
 

website/guide/gemini-cli.md

@@ -40,12 +40,14 @@ You can optionally create or edit:
 Register MemPalace as an MCP server:
 
 ```bash
-gemini mcp add mempalace /absolute/path/to/mempalace/.venv/bin/python3 \
-  -m mempalace.mcp_server --scope user
+gemini mcp add --scope user mempalace \
+  -- /absolute/path/to/mempalace/.venv/bin/python -m mempalace.mcp_server
 ```
 
 ::: warning
-Use the **absolute path** to the Python binary to ensure it works from any directory.
+Use the **absolute path** to the Python binary so the server starts from any
+working directory. The `--` separator prevents Gemini from parsing
+`-m mempalace.mcp_server` as its own flags.
 :::
 
 ## Enable Auto-Saving

website/guide/mcp-integration.md

@@ -1,6 +1,6 @@
 # MCP Integration
 
-MemPalace provides 19 tools through the [Model Context Protocol (MCP)](https://modelcontextprotocol.io/), giving any MCP-compatible AI full read/write access to your palace.
+MemPalace provides 29 tools through the [Model Context Protocol (MCP)](https://modelcontextprotocol.io/), giving any MCP-compatible AI full read/write access to your palace.
 
 ## Setup
 
@@ -24,7 +24,7 @@ claude mcp add mempalace -- python -m mempalace.mcp_server
 claude mcp add mempalace -- python -m mempalace.mcp_server --palace /path/to/palace
 ```
 
-Now your AI has all 19 tools available. Ask it anything:
+Now your AI has all 29 tools available. Ask it anything:
 
 > *"What did we decide about auth last month?"*
 
@@ -66,11 +66,19 @@ This protocol is what turns storage into memory — the AI knows to verify befor
 | `mempalace_check_duplicate` | Check before filing |
 | `mempalace_get_aaak_spec` | AAAK dialect reference |
 
+### Drawers (read)
+
+| Tool | What |
+|------|------|
+| `mempalace_get_drawer` | Fetch a single drawer by ID |
+| `mempalace_list_drawers` | List drawers with pagination |
+
 ### Palace (write)
 
 | Tool | What |
 |------|------|
 | `mempalace_add_drawer` | File verbatim content |
+| `mempalace_update_drawer` | Update drawer content or metadata |
 | `mempalace_delete_drawer` | Remove by ID |
 
 ### Knowledge Graph
@@ -91,6 +99,15 @@ This protocol is what turns storage into memory — the AI knows to verify befor
 | `mempalace_find_tunnels` | Find rooms bridging two wings |
 | `mempalace_graph_stats` | Graph connectivity overview |
 
+### Tunnels
+
+| Tool | What |
+|------|------|
+| `mempalace_create_tunnel` | Create an explicit cross-wing tunnel |
+| `mempalace_list_tunnels` | List all explicit tunnels |
+| `mempalace_delete_tunnel` | Delete an explicit tunnel |
+| `mempalace_follow_tunnels` | Follow tunnels out from a room |
+
 ### Agent Diary
 
 | Tool | What |
@@ -98,4 +115,12 @@ This protocol is what turns storage into memory — the AI knows to verify befor
 | `mempalace_diary_write` | Write AAAK diary entry |
 | `mempalace_diary_read` | Read recent diary entries |
 
+### System
+
+| Tool | What |
+|------|------|
+| `mempalace_hook_settings` | Get or set hook behavior |
+| `mempalace_memories_filed_away` | Check whether the last checkpoint was saved |
+| `mempalace_reconnect` | Force reconnect to the database |
+
 For detailed schemas and parameters, see [MCP Tools Reference](/reference/mcp-tools).

website/guide/openclaw.md

@@ -27,7 +27,7 @@ Or by directly editing your OpenClaw configuration:
 
 ## How It Works
 
-Once connected, OpenClaw agents receive all 19 tools along with the **Memory Protocol**—a strict behavioral guide indicating they should:
+Once connected, OpenClaw agents receive all 29 tools along with the **Memory Protocol**—a strict behavioral guide indicating they should:
 1. **Never guess**: Query `mempalace_search` or `mempalace_kg_query` before confidently answering.
 2. **Keep an agent diary**: Maintain continuity between sessions by writing to `mempalace_diary_write`.
 3. **Manage the Knowledge Graph**: Update declarative facts when things change using `mempalace_kg_add` and `mempalace_kg_invalidate`.

website/index.md

@@ -1,88 +1,6 @@
 ---
-layout: home
-
-hero:
-  name: MemPalace
-  text: Give your AI a memory.
-  tagline: "Local-first AI memory. Verbatim storage, pluggable backend, 96.6% R@5 raw on LongMemEval — zero API calls."
-  image:
-    src: /mempalace_logo.png
-    alt: MemPalace
-  actions:
-    - theme: brand
-      text: Get Started
-      link: /guide/getting-started
-    - theme: alt
-      text: Architecture →
-      link: /concepts/the-palace
-    - theme: alt
-      text: GitHub ↗
-      link: https://github.com/MemPalace/mempalace
-
-features:
-  - icon:
-      src: /icons/file-text.svg
-      alt: Verbatim Storage
-    title: Verbatim Storage
-    details: Store source text directly instead of extracting facts up front. The raw benchmark result comes from retrieving verbatim content.
-  - icon:
-      src: /icons/building-2.svg
-      alt: Palace Structure
-    title: Palace Structure
-    details: Wings and rooms give retrieval useful structure. In the project benchmarks, narrowing search scope outperformed flat search.
-  - icon:
-      src: /icons/search.svg
-      alt: Semantic Search
-    title: Semantic Search
-    details: Vector search over verbatim content lets the model retrieve past discussions by topic, project, or room. Backend is pluggable.
-  - icon:
-      src: /icons/git-merge.svg
-      alt: Knowledge Graph
-    title: Knowledge Graph
-    details: Temporal entity-relationship triples in SQLite. Facts can be added, queried, and invalidated over time.
-  - icon:
-      src: /icons/wrench.svg
-      alt: 19 MCP Tools
-    title: 19 MCP Tools
-    details: MCP tools expose search, filing, knowledge graph, graph navigation, and diary operations to compatible clients.
-  - icon:
-      src: /icons/shield-check.svg
-      alt: Zero Cloud
-    title: Zero Cloud
-    details: Core storage and retrieval run locally. Optional reranking features can add an API dependency but are not required for the benchmark path.
+layout: page
+pageClass: mempalace-home
 ---
 
-<style>
-:root {
-  --vp-home-hero-name-color: transparent;
-  --vp-home-hero-name-background: linear-gradient(
-    135deg,
-    #4f46e5 0%,
-    #06b6d4 50%,
-    #8b5cf6 100%
-  );
-}
-</style>
-
-<div style="max-width: 688px; margin: 0 auto; padding: 48px 24px 0;">
-
-## Verbatim Retrieval First
-
-MemPalace stores source text and retrieves it with semantic search. The benchmarked raw mode does not require an LLM at any stage — no extraction, no rerank, no summarisation.
-
-**LongMemEval retrieval recall (500 questions):**
-
-| Mode | R@5 | LLM required |
-|---|---|---|
-| Raw (semantic search over verbatim text) | **96.6%** | None |
-| Hybrid v4, held-out 450q | **98.4%** | None |
-
-The raw 96.6% reproduces on any machine with the committed dataset: result JSONLs, the `seed=42` train/held-out split, and the `--mode raw` / `--held-out` runners are all in the `benchmarks/` directory of the repo.
-
-We deliberately do not publish a side-by-side comparison against other memory systems on this page. Retrieval recall (R@5) and end-to-end QA accuracy are different metrics and are not comparable; where MemPalace can be fairly compared on the same metric, we link to the other project's published source.
-
-<div style="text-align: center; padding-top: 16px;">
-  <a href="./reference/benchmarks" style="color: var(--vp-c-brand-1); font-weight: 500;">Full benchmark methodology →</a>
-</div>
-
-</div>
+<Landing />

website/reference/api-reference.md

@@ -108,7 +108,7 @@ Unified 4-layer interface.
 
 | Method | Parameters | Returns | Description |
 |--------|-----------|---------|-------------|
-| `wake_up(wing=None)` | Optional wing | `str` | L0 + L1 context (~170–900 tokens) |
+| `wake_up(wing=None)` | Optional wing | `str` | L0 + L1 context (~600–900 tokens) |
 | `recall(wing, room, n_results=10)` | Filters | `str` | L2 on-demand retrieval |
 | `search(query, wing, room, n_results=5)` | Query + filters | `str` | L3 deep search |
 | `status()` | — | `dict` | All layer status info |

website/reference/cli.md

@@ -89,7 +89,7 @@ mempalace split <dir> --output-dir ~/split-output/
 
 ## `mempalace wake-up`
 
-Show L0 + L1 wake-up context (~170–900 tokens).
+Show L0 + L1 wake-up context (~600–900 tokens).
 
 ```bash
 mempalace wake-up

website/reference/modules.md

@@ -9,7 +9,7 @@ mempalace/
 ├── README.md                  ← project documentation
 ├── mempalace/                 ← core package
 │   ├── cli.py                 ← CLI entry point
-│   ├── mcp_server.py          ← MCP server (19 tools)
+│   ├── mcp_server.py          ← MCP server (29 tools)
 │   ├── knowledge_graph.py     ← temporal entity graph
 │   ├── palace_graph.py        ← room navigation graph
 │   ├── dialect.py             ← AAAK compression
@@ -56,7 +56,7 @@ Argparse-based CLI with subcommands: `init`, `mine`, `split`, `search`, `compres
 
 ### `mcp_server.py` — MCP Server
 
-JSON-RPC over stdin/stdout. Implements the MCP protocol with 19 tools covering palace read/write, knowledge graph, navigation, and agent diary operations. Includes the Memory Protocol and AAAK Spec in status responses.
+JSON-RPC over stdin/stdout. Implements the MCP protocol with 29 tools covering palace read/write, drawer CRUD, knowledge graph, navigation, tunnels, agent diary, and system operations. Includes the Memory Protocol and AAAK Spec in status responses.
 
 ### `searcher.py` — Semantic Search
 

website/reference/python-api.md

@@ -36,7 +36,7 @@ from mempalace.layers import MemoryStack
 stack = MemoryStack()  # uses default paths from MempalaceConfig
 
 # Wake-up: L0 (identity) + L1 (essential story)
-context = stack.wake_up(wing="myapp")  # ~170-900 tokens
+context = stack.wake_up(wing="myapp")  # ~600-900 tokens
 
 # On-demand: L2 retrieval
 recall = stack.recall(wing="myapp", room="auth", n_results=10)