From 2a2466c65620ff02e986ec42e65b0b227a2c2727 Mon Sep 17 00:00:00 2001 From: Igor Lins e Silva <4753812+igorls@users.noreply.github.com> Date: Sun, 12 Apr 2026 20:26:42 -0300 Subject: [PATCH] =?UTF-8?q?docs:=20draft=20RFC=20001=20=E2=80=94=20storage?= =?UTF-8?q?=20backend=20plugin=20specification=20(#737)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Formalizes the BaseCollection/BaseBackend contract introduced as a seam in #413 into an interchangeability spec that third-party backends can build to. Driven by six in-flight backend PRs (#574, #643, #665, #697, #700, #381) each implementing the interface differently. Key decisions captured: entry-point distribution, typed QueryResult/ GetResult replacing Chroma dict shape, daemon-first multi-palace model via PalaceRef, required where-clause subset (incl. $contains), mandatory embedder injection with model-identity validation, capability tokens, shared pytest conformance suite, and a backend-neutral migrate/verify CLI. --- docs/rfcs/001-storage-backend-plugin-spec.md | 514 +++++++++++++++++++ 1 file changed, 514 insertions(+) create mode 100644 docs/rfcs/001-storage-backend-plugin-spec.md diff --git a/docs/rfcs/001-storage-backend-plugin-spec.md b/docs/rfcs/001-storage-backend-plugin-spec.md new file mode 100644 index 0000000..73ecc24 --- /dev/null +++ b/docs/rfcs/001-storage-backend-plugin-spec.md @@ -0,0 +1,514 @@ +# RFC 001 — Storage Backend Plugin Specification + +- **Status:** Draft +- **Tracking issue:** [#737](https://github.com/MemPalace/mempalace/issues/737) +- **Supersedes:** The informal seam introduced by [#413](https://github.com/MemPalace/mempalace/pull/413) +- **Related:** [#266](https://github.com/MemPalace/mempalace/issues/266), [#574](https://github.com/MemPalace/mempalace/pull/574), [#643](https://github.com/MemPalace/mempalace/pull/643), [#665](https://github.com/MemPalace/mempalace/pull/665), [#697](https://github.com/MemPalace/mempalace/pull/697), [#700](https://github.com/MemPalace/mempalace/pull/700), [#381](https://github.com/MemPalace/mempalace/pull/381) +- **Spec version:** `1.0` + +## Summary + +A formal contract for MemPalace storage backends so third parties can ship `pip install mempalace-` packages that drop into the core without patches. The spec defines the collection interface, the backend lifecycle, registration via Python entry points, configuration shape, a required test contract, and a migration path between backends. + +It also sets up MemPalace to run as a long-lived daemon that manages many palaces, where different palaces may route to different backends. + +## Motivation + +Six backend PRs are currently in flight. Each one solves the same problem six different ways — different method signatures, different registration mechanisms, different embedder ownership, incompatible where-clause dialects, no shared test suite. The ad-hoc `BaseCollection` ABC merged in #413 was deliberately minimal and deferred every non-obvious decision. This RFC closes the open decisions so backend authors can build to a stable contract. + +## Goals + +1. A backend ships as a standalone Python package; installing it is sufficient to use it. +2. All callers in MemPalace core go through the collection interface. No direct `chromadb` imports outside `mempalace/backends/chroma.py`. +3. Backends are interchangeable: every backend passes the same shared test suite, and `mempalace migrate` moves palaces between them without data loss. +4. The model scales from single-user local (one backend, one palace, no config) to a daemon serving many palaces with heterogeneous backends. +5. Chroma's current dict-shaped return values are not the long-term contract. Typed results are spec v1. + +## Non-goals + +- Defining the embedder pipeline in detail. The embedder is a separate contract this spec depends on but does not specify. +- Defining the sync subsystem. This spec only declares the capability flag and the minimal hook a sync subsystem will read. +- Specifying wire protocol for a future networked daemon. That is a separate RFC. + +--- + +## 1. Collection contract + +### 1.1 Required methods + +All backends implement `BaseCollection` with kwargs-only signatures: + +```python +class BaseCollection(ABC): + @abstractmethod + def add( + self, + *, + documents: list[str], + ids: list[str], + metadatas: list[dict] | None = None, + embeddings: list[list[float]] | None = None, + ) -> None: ... + + @abstractmethod + def upsert( + self, + *, + documents: list[str], + ids: list[str], + metadatas: list[dict] | None = None, + embeddings: list[list[float]] | None = None, + ) -> None: ... + + @abstractmethod + def query( + self, + *, + query_texts: list[str] | None = None, + query_embeddings: list[list[float]] | None = None, + n_results: int = 10, + where: dict | None = None, + where_document: dict | None = None, + include: list[str] | None = None, + ) -> QueryResult: ... + + @abstractmethod + def get( + self, + *, + ids: list[str] | None = None, + where: dict | None = None, + where_document: dict | None = None, + limit: int | None = None, + offset: int | None = None, + include: list[str] | None = None, + ) -> GetResult: ... + + @abstractmethod + def delete( + self, + *, + ids: list[str] | None = None, + where: dict | None = None, + ) -> None: ... + + @abstractmethod + def count(self) -> int: ... +``` + +### 1.2 Optional methods (default implementations on the ABC) + +```python +def estimated_count(self) -> int: + return self.count() + +def close(self) -> None: + return None + +def health(self) -> HealthStatus: + return HealthStatus.ok() +``` + +Backends with cheap approximate counters override `estimated_count`. Backends that hold connections must override `close`. + +### 1.3 Typed results (replaces Chroma dict shape) + +```python +@dataclass(frozen=True) +class QueryResult: + ids: list[list[str]] # outer = queries, inner = hits + documents: list[list[str]] + metadatas: list[list[dict]] + distances: list[list[float]] + embeddings: list[list[list[float]]] | None = None + +@dataclass(frozen=True) +class GetResult: + ids: list[str] + documents: list[str] + metadatas: list[dict] + embeddings: list[list[float]] | None = None +``` + +On empty results: return a result object with empty inner lists, never raise. Specifically, an empty query returns `QueryResult(ids=[[]], documents=[[]], metadatas=[[]], distances=[[]])` — the outer dimension is the number of query vectors issued; the inner dimension is hits per query and may be zero. + +`include` controls which fields are populated. Fields not in `include` are populated with empty lists of the correct outer shape; they are never `None` (except `embeddings`, which is `None` when not requested). + +### 1.4 Where-clause dialect + +**Required operators:** `$eq`, `$ne`, `$in`, `$nin`, `$and`, `$or`, `$contains`. + +Backends that do not support full-text natively must implement `$contains` via payload string match. A `$contains_fast` capability flag lets callers check whether a backend's implementation is indexed (fast) or scan-based (slow). + +**Unknown operators:** backends MUST raise `UnsupportedFilterError`. Silent dropping is forbidden — it produces incorrect results. + +**Optional operators:** `$gt`, `$gte`, `$lt`, `$lte`. Backends either implement them or reject with `UnsupportedFilterError`. Advertised via capabilities. + +### 1.5 Embeddings + +Backends MUST accept a pre-computed `embeddings=` argument on `add` / `upsert`. Backends that recompute from text may ignore its value but MUST NOT reject the argument. This is required for lossless migration. + +Backends MUST NOT hardcode embedding models or dimensions. Model selection is the embedder's responsibility (§4). + +Backends MUST validate embedding dimension on first write to a new collection and on open of an existing collection, and MUST raise `DimensionMismatchError` on mismatch. Silent acceptance of mismatched dimensions produces unrecoverable corruption. + +**Dimension matching is necessary but not sufficient.** Swapping to a different model that happens to share a dimension (e.g., both 384-d) will silently degrade retrieval quality without tripping `DimensionMismatchError`. To guard this, backends MUST persist the `embedder.model_name` alongside the collection on first write, and MUST raise `EmbedderIdentityMismatchError` when a later open presents a different `model_name`. Users can override with an explicit `--force-model-swap` at the CLI layer, which records the swap in the palace's migration log. + +--- + +## 2. Backend contract + +### 2.1 Identity and capabilities + +```python +class BaseBackend(ABC): + name: ClassVar[str] # "chroma", "postgres", "qdrant", ... + spec_version: ClassVar[str] = "1.0" # which spec version this backend targets + capabilities: ClassVar[frozenset[str]] +``` + +Defined capability tokens (v1): + +| Token | Meaning | +|---|---| +| `supports_embeddings_in` | Accepts pre-computed embeddings (MUST be true for all backends) | +| `supports_embeddings_out` | Returns embeddings when `include=["embeddings"]` is requested | +| `supports_estimated_count` | `estimated_count()` is meaningfully cheaper than `count()` | +| `supports_update` | Distinguishes `upsert` from `add` with meaningful replace semantics | +| `supports_metadata_filters` | Implements the required where-clause subset (§1.4) | +| `supports_range_filters` | Implements `$gt` / `$gte` / `$lt` / `$lte` | +| `supports_contains_fast` | `$contains` is indexed (vs scan-based) | +| `supports_server_side_indexes` | Exposes index creation / maintenance to operators | +| `supports_migration_export` | Implements a bulk read path suitable for `mempalace migrate` | +| `supports_change_feed` | Exposes `changes_since(cursor)` for the sync subsystem | +| `supports_sync` | Implies `supports_change_feed` plus idempotent upserts under conflicts | +| `requires_external_service` | Needs a running server (e.g., Postgres, hosted Qdrant) | +| `local_mode` | Persists to `palace.local_path` | +| `server_mode` | Connects to an external server; `palace.namespace` is used | +| `server_embedder` | Backend provides its own embedder (may ignore injected one) | + +A backend may advertise both `local_mode` and `server_mode` (e.g., Chroma with either `PersistentClient` or `HttpClient`). + +Capability tokens are free-form strings, not an enum — third-party backends may declare novel capabilities for their ecosystem. Core MemPalace only inspects the tokens listed above. + +### 2.2 Palace references + +A backend serves palaces, not raw filesystem paths. This is the central change from #413. + +```python +@dataclass(frozen=True) +class PalaceRef: + id: str # stable identity, used as cache key + local_path: str | None = None # filesystem root, if this palace is local + namespace: str | None = None # server-side namespace/prefix, if applicable +``` + +Rules: +- `id` is always present. It is the key the backend uses to cache open handles. +- Local-only backends read `local_path`. If `local_path is None` they raise `PalaceNotFoundError`. +- Server-only backends read `namespace`. If `namespace is None` they derive one deterministically from `id`. +- Mixed-mode backends may use both (e.g., a local cache alongside a server store). + +### 2.3 Methods + +```python +class BaseBackend(ABC): + @abstractmethod + def get_collection( + self, + *, + palace: PalaceRef, + collection_name: str, + create: bool, + embedder: Embedder | None = None, + options: dict | None = None, + ) -> BaseCollection: ... + + def close_palace(self, palace: PalaceRef) -> None: + """Evict a single palace's cached handles. Default: no-op.""" + return None + + def close(self) -> None: + """Shut down the entire backend instance. Default: no-op.""" + return None + + def health(self, palace: PalaceRef | None = None) -> HealthStatus: + """Return health. With palace=None, probe the backend itself.""" + return HealthStatus.ok() +``` + +### 2.4 Semantics of `create` + +- `create=False` on a nonexistent palace MUST raise `PalaceNotFoundError` (subclass of `FileNotFoundError` for backwards compatibility with the #413 seam). +- `create=True` MUST be idempotent — calling it repeatedly with the same arguments produces the same state and does not corrupt existing data. +- `create=True` on local backends creates the directory with `0700` permissions (matches the existing Chroma behavior). + +### 2.5 Concurrency + +A backend instance is long-lived and serves many palaces. Backends MUST be thread-safe for concurrent `get_collection` calls across different `PalaceRef.id` values. Collection handles for the same `(palace.id, collection_name)` MAY be cached internally and returned on subsequent calls. + +Backends MAY assume a single thread accesses a given `BaseCollection` instance at a time. MemPalace core serializes access per palace; backend authors are not required to make individual collections thread-safe. + +### 2.6 Lifecycle + +1. `__init__`: lightweight. No I/O, no network connections. A backend instance may be constructed and never used. +2. First call to `get_collection`: may open connections, create schemas, etc. All I/O is lazy. +3. `close_palace(palace)`: releases cached handles for one palace. Safe to call on a palace that was never opened. +4. `close()`: releases all resources. After `close()`, further calls MUST raise `BackendClosedError`. + +There is no explicit `connect()` — it is always implicit and lazy, matching current Chroma behavior. + +--- + +## 3. Registration and discovery + +### 3.1 Entry points (primary mechanism) + +Third-party backends ship as installable packages: + +```toml +# pyproject.toml of mempalace-postgres +[project.entry-points."mempalace.backends"] +postgres = "mempalace_postgres:PostgresBackend" +``` + +MemPalace discovers backends at process start via `importlib.metadata.entry_points(group="mempalace.backends")`. No patches to the core are required. + +### 3.2 In-tree registry (secondary) + +For tests and local development: + +```python +from mempalace.backends.registry import register + +register("my-experimental-backend", MyBackend) +``` + +Entry-point discovery and explicit `register()` populate the same registry. Explicit registration wins on name conflict. + +### 3.3 Selection priority + +When resolving a palace's backend, priority (highest first): + +1. Explicit `backend=` kwarg to `Palace(...)` or CLI `--backend` +2. Per-palace `backend` key in config (see §4) +3. `MEMPALACE_BACKEND` environment variable +4. Auto-detect from on-disk artifacts: `chroma.sqlite3` → `chroma`, `*.lance` → `lance`, etc. Backends declare detection hints via an optional `BaseBackend.detect(path: str) -> bool` classmethod. +5. Default: `chroma`. + +Auto-detection preserves existing palaces when users upgrade — no manual config migration needed. + +--- + +## 4. Configuration + +### 4.1 Shape + +```json +{ + "backends": { + "chroma": { "type": "chroma" }, + "pg_prod": { + "type": "postgres", + "dsn": "postgresql://...", + "pool_size": 10 + } + }, + "palaces": { + "work": { + "backend": "pg_prod", + "namespace": "work" + }, + "personal": { + "backend": "chroma", + "local_path": "~/.mempalace/personal" + } + }, + "embedder": { + "type": "onnx", + "model": "all-MiniLM-L6-v2" + } +} +``` + +Single-user local mode: all of this is optional. The absence of a config file yields one Chroma backend, one palace at the default path, with the default embedder. + +### 4.2 Environment variables + +- `MEMPALACE_BACKEND` — shortcut for the default backend type when there is no config. +- `MEMPALACE__*` — per-backend secrets and connection info (e.g., `MEMPALACE_POSTGRES_DSN`, `MEMPALACE_QDRANT_URL`, `MEMPALACE_QDRANT_API_KEY`). +- Secrets MUST be readable from env vars; config files are for structure, env vars for credentials. + +### 4.3 Backend-specific options + +The `options` kwarg to `get_collection` is a free-form dict. Each backend documents its accepted keys. Unknown keys MUST be ignored (forward compatibility), but the backend MAY log a warning. + +### 4.4 Multi-tenancy (absorbs #697) + +Per-tenant collection-name prefixing is not a backend concern. It is handled by the resolver layer above backends: `PalaceRef.namespace` carries the tenant identifier, and the backend uses it as given. The `collection_prefix` concept from #697 dissolves into this model. + +--- + +## 5. Embedder contract (minimal, external to this spec) + +This spec assumes an `Embedder` protocol, defined fully in a separate RFC: + +```python +class Embedder(Protocol): + model_name: str + dimension: int + def embed(self, texts: list[str]) -> list[list[float]]: ... +``` + +Backends receive an `Embedder` via `get_collection(embedder=...)`. Backends with `server_embedder` capability MAY ignore the injected embedder. + +--- + +## 6. Sync (capability declaration only) + +The sync subsystem is out of scope for this spec. What this spec defines: + +- `sync_capable` capability flag — a backend advertising it agrees to implement idempotent upserts under conflict and to expose change data. +- Optional method on `BaseCollection`: + ```python + def changes_since(self, cursor: SyncCursor) -> Iterator[Change]: ... + ``` +- Backends without `change_feed` / `sync_capable` are rejected by the sync subsystem at bind time. + +Local single-user deployments never load the sync subsystem; non-sync-capable backends cost them nothing. + +--- + +## 7. Testing contract + +### 7.1 The abstract suite + +MemPalace ships `mempalace.backends.testing.AbstractBackendContractSuite` — a pytest mixin. Every backend package ships a concrete subclass: + +```python +from mempalace.backends.testing import AbstractBackendContractSuite + +class TestPostgresBackend(AbstractBackendContractSuite): + @pytest.fixture + def backend(self, tmp_path): + return PostgresBackend(dsn=os.environ["TEST_PG_DSN"]) +``` + +The suite covers: +- Round-trip for every required method +- Empty-result shape (outer dimension preserved, inner lists empty) +- `create=False` on missing palace raises `PalaceNotFoundError` +- `create=True` is idempotent +- Full required where-clause subset including `$contains` +- Unknown operator raises `UnsupportedFilterError` +- Dimension-mismatch detection +- Unicode text and unicode IDs +- Large batch writes (10k+ items) +- Delete-then-query consistency +- `close()` releases handles and further calls raise `BackendClosedError` +- Concurrent `get_collection` across different palaces is safe + +### 7.2 Parametrized core suite + +The existing MemPalace test suite is parametrized over all registered backends when `MEMPALACE_TEST_ALL_BACKENDS=1` is set in the environment. This is the "strongest parity claim" — if a backend passes the full core suite, it is drop-in compatible. This is expensive; local development defaults to Chroma only, CI runs all backends on a scheduled job. + +### 7.3 Benchmark methodology hooks + +Backend-to-backend comparisons are meaningless without accounting for per-backend maintenance state. Postgres with stale planner stats behaves very differently from Postgres post-`VACUUM ANALYZE`; HNSW-based stores behave differently before and after index compaction. + +Backends MAY implement `maintenance_state()` returning a structured dict describing the current state (e.g., `{"autovacuum_age_seconds": 42, "last_analyze": "...", "index_build_complete": true}`), and `run_maintenance(kind: str)` to trigger known maintenance kinds (`"analyze"`, `"compact"`, `"reindex"`). Both are optional. + +The benchmark harness under [benchmarks/](../../benchmarks/) records `maintenance_state()` alongside every latency/recall measurement it publishes. Published numbers MUST include three phases: immediately after bulk load, after the backend's native background maintenance has caught up, and after explicit `run_maintenance` if the backend advertises maintenance kinds. This prevents comparing an un-`ANALYZE`d Postgres to a settled Chroma and calling the former slow. + +### 7.4 ID stability for non-string-ID backends + +Backends requiring UUID IDs (Qdrant) use a canonical namespace: + +```python +NAMESPACE_MEMPALACE = uuid.UUID("TO-BE-ASSIGNED-ONCE-FOR-ALL-TIME") +backend_id = uuid.uuid5(NAMESPACE_MEMPALACE, original_id) +``` + +The namespace UUID is fixed at spec v1 adoption and recorded here. This resolves the #700 vs #381 divergence. + +--- + +## 8. Migration + +### 8.1 The CLI + +``` +mempalace migrate --palace PATH --from chroma --to postgres +mempalace migrate --all --to lance +``` + +Implementation is backend-agnostic: reads from source via `BaseCollection.get(include=["documents", "metadatas", "embeddings"])`, writes to target via `BaseCollection.upsert(...)` with the original embeddings. No backend-specific migration code. + +### 8.2 Safety + +- Source is never modified. Migration is read-only against the source backend. +- Target palace must not already exist unless `--overwrite` is passed. +- A successful migration writes a `.mempalace-migration.json` record into the target palace (source backend, source path, timestamp, row count). + +### 8.3 Verification + +After migration, run `mempalace verify --palace PATH --against SOURCE_PATH --source-backend chroma`. This samples N rows and confirms round-trip parity (ids match, documents match, embedding cosine similarity ≥ 0.999). + +--- + +## 9. Versioning and compatibility + +- `BaseBackend.spec_version` declares which spec version a backend implements. +- MemPalace refuses to load a backend declaring a different major version. +- Minor versions are additive (new optional methods, new capability tokens). Backends declaring an older minor continue to work. +- This is spec v1.0. + +--- + +## 10. Cleanup prerequisite (not in this spec, but gating) + +The #413 seam is incomplete. Seven files in `mempalace/` still import `chromadb` directly: + +- [mempalace/repair.py:35](https://github.com/MemPalace/mempalace/blob/develop/mempalace/repair.py#L35) +- [mempalace/dedup.py:30](https://github.com/MemPalace/mempalace/blob/develop/mempalace/dedup.py#L30) +- [mempalace/cli.py:171](https://github.com/MemPalace/mempalace/blob/develop/mempalace/cli.py#L171) +- [mempalace/cli.py:278](https://github.com/MemPalace/mempalace/blob/develop/mempalace/cli.py#L278) +- [mempalace/mcp_server.py:32](https://github.com/MemPalace/mempalace/blob/develop/mempalace/mcp_server.py#L32) +- [mempalace/migrate.py:109](https://github.com/MemPalace/mempalace/blob/develop/mempalace/migrate.py#L109) +- Plus the instruction docs. + +These must be routed through `BaseCollection` before the spec can be enforced. Combined with the dict-to-typed-result migration from §1.3, this is substantial enough to be its own PR, landing before any new backend implementation merges. + +--- + +## 11. Impact on in-flight PRs + +| PR | Effort to align | +|---|---| +| [#574](https://github.com/MemPalace/mempalace/pull/574) LanceDB | Closest to final shape. Needs `PalaceRef` and typed results. | +| [#665](https://github.com/MemPalace/mempalace/pull/665) Postgres | Decouple embedder; adopt `PalaceRef`. Schema-per-palace already fits the daemon model. | +| [#700](https://github.com/MemPalace/mempalace/pull/700) Qdrant | Decouple embedder; adopt `PalaceRef`; align UUID namespace. | +| [#381](https://github.com/MemPalace/mempalace/pull/381) Qdrant (older) | Same as #700; also subclass `BaseCollection` rather than using a bare `Protocol`. | +| [#643](https://github.com/MemPalace/mempalace/pull/643) PalaceStore | Parametrized-test approach becomes the standard. | +| [#697](https://github.com/MemPalace/mempalace/pull/697) Chroma HttpClient + prefix | `collection_prefix` dissolves into `PalaceRef.namespace`. | + +--- + +## 12. Open questions + +None blocking. The following are nice-to-have for a future minor revision: + +- Should `changes_since` accept a filter (e.g., "changes to this collection only")? +- Do we want a `BaseBackend.capabilities(palace=...)` variant, so that capabilities can be per-palace (e.g., `supports_contains_fast` depends on whether an index exists)? +- Should `run_maintenance(kind)` return a structured result (rows analyzed, bytes reclaimed) or stay fire-and-forget? + +--- + +## 13. Rollout + +1. Land the cleanup PR (§10): route all callers through `BaseCollection`, migrate to typed results. Chroma remains the only backend. +2. Land this spec as-is. Add `AbstractBackendContractSuite`, entry-point discovery, `PalaceRouter`, `PalaceRef`. +3. Update ChromaBackend to spec v1.0 (add capabilities declaration, `detect()` classmethod, `PalaceRef` support). +4. Rebase in-flight backend PRs against the spec. Each must pass the abstract suite. +5. Ship `mempalace migrate` CLI. +6. Update [ROADMAP.md](../../ROADMAP.md) with spec v1.0 adoption under v4.0.0-alpha.