Completes the embedder-identity contract for the last backend (qdrant) and
unifies identity persistence across all local-path backends.
Identity is stored in a small per-palace sidecar (mempalace_embedder.json),
NOT in a backend's mismatch marker. The marker's presence signals "palace
initialized" (reads raise CollectionNotInitializedError when the marker exists
but the store doesn't), so recording identity at first empty open must not
create it — a sidecar is unguarded, so a brand-new palace records identity
immediately. This fixes a latent gap (caught in review) where pgvector/qdrant
palaces stayed permanently "unknown" because the marker isn't written until the
first real write.
- New mempalace/backends/_sidecar.py: shared read/write_embedder_sidecar with
the isinstance robustness the review bots taught us; chroma, pgvector, and
qdrant all use it (chroma's inline copy is removed, pgvector switches off the
marker, qdrant adds it).
- QdrantCollection.get/set_embedder_identity delegate to the sidecar, so
palace.get_collection enforces a model swap on a qdrant palace exactly like
the other backends — no live server needed for the check.
Tests: sidecar roundtrip + brand-new-palace recording (creates sidecar without
a marker) + enforcement model-swap raise, for qdrant and pgvector, all
server-free; chroma identity tests unchanged. Full suite: 2495 passed, 82.52%.
Closes#1730. Refs #743, #1724.
Adds the maintenance contract RFC 001 specifies but #1679 deferred, and gives
pgvector an opt-in HNSW index path with concurrency-safe builds.
- base.py: MaintenanceResult (status ran/already_running/noop + free-form
stats), UnsupportedMaintenanceKindError, BaseBackend.maintenance_kinds
ClassVar (reserved: analyze/compact/reindex; a backend with no analogue MUST
omit, not no-op), and BaseCollection.maintenance_state()/run_maintenance()
defaults. EmbeddingCollection forwards both (BaseCollection methods shadow
__getattr__).
- sqlite_exact: analyze (ANALYZE) + compact (VACUUM, autocommit + page stats);
omits reindex (exact scan, no ANN index). maintenance_state reports row/page
counts.
- pgvector: reindex builds the optional HNSW index, serialized by a
session-level pg_advisory_lock so concurrent daemon writers learn
"already_running" instead of each stacking an ACCESS EXCLUSIVE build (the
production wedge). It is opt-in: the default exact `<=>` scan is the
100%-recall path; an HNSW index trades exact recall for scale, so an operator
invokes it deliberately. Also analyze; omits compact (autovacuum). Advertises
supports_server_side_indexes. maintenance_state reports index presence.
- qdrant/chroma: empty maintenance_kinds (qdrant self-optimizes; chroma
maintenance is the separate repair CLI) — the faithful "omit" default.
Tests: contract + sqlite (real, CI-runnable) + pgvector advisory-lock flow via
a fake client (ran/noop/already_running, no live Postgres). Full suite green:
2488 passed, 82.47% coverage.
Benchmark three-phase wiring is deferred — the existing benchmarks/ are
task-benchmarks, not backend-comparison harnesses, so there is nothing to wire
into yet.
Closes#1725. Refs #743.
The explicit-embedding backends (pgvector, qdrant, sqlite_exact) embed through
EmbeddingCollection and persist nothing about which model produced their
vectors. Swapping two same-dimension models (minilm <-> embeddinggemma, both
384-d) silently corrupts retrieval with no error — the worst failure class for
a verbatim-recall system. This records the model name and refuses a swap on open.
- base.py: EmbedderIdentity dataclass, Embedder protocol,
EmbedderIdentityUnknownWarning, and a three-state check_embedder_identity()
helper (known_match / known_mismatch / unknown) raising
EmbedderIdentityMismatchError / DimensionMismatchError. BaseCollection gains
get/set/effective_embedder_identity hooks.
- embedding.py: current_model_name() (cheap) + get_embedder_identity() (probed
dimension, cached per process).
- Persistence per backend: sqlite_exact meta table, pgvector marker JSON
(preserved across rewrites), chroma sidecar JSON. EmbeddingCollection forwards
the hooks explicitly (BaseCollection methods shadow __getattr__).
- palace.get_collection enforces at open: a model swap raises; a brand-new empty
collection records the current model; a populated-but-unrecorded legacy palace
warns and is resolved via the CLI. The check needs no model load.
- CLI: `mempalace palace set-embedder [--model NAME] [--force]` records/overrides
identity without mutating global config or loading a foreign model.
Chroma additionally keeps its native embedding-function check. Qdrant identity
persistence is a fast-follow (no local metadata slot — needs a companion store).
Closes#1724. Refs #743.
The searcher hard-coded `max(0, 1 - distance)` for vector similarity, which is
correct only for cosine. A backend reporting L2 or inner-product distances — or
a legacy Chroma palace built without `hnsw:space=cosine` — was silently
mis-ranked: L2 distances routinely exceed 1.0 and floored every result to 0.
- backends/base.py: `BaseBackend.distance_metric` ClassVar (cosine default) and
a `BaseCollection.distance_metric` property per RFC 001 §2.1.
- backends/chroma.py: `ChromaCollection.distance_metric` reads the real
`hnsw:space`, so a legacy-L2 palace reports "l2" and ranks correctly.
- searcher.py: `_distance_to_similarity(distance, metric)` (cosine/l2/ip,
monotonic + bounded) and `_metric_for_collection(col)`; `_hybrid_rank` and the
CLI display are now metric-aware. Cosine output is unchanged.
- Same conversion applied to the MCP `similarity` field, layers.py L3 search,
and the dedup threshold so reported similarities are consistent.
Cosine behavior is byte-for-byte preserved; only non-cosine metrics change.
Part B of #1726 (routing the chroma.sqlite3 BM25 fallback through
BaseCollection) is tracked separately. Refs #1726, #743.
_missing_dimensionality_appears_recoverable required total_elements_added == len(id_to_label) before keeping a dim-None segment, but total_elements_added is a monotonic add-counter while id_to_label holds only the live elements. After any delete the counts diverge, so every post-deletion segment was quarantined; once the WAL is pruned that destroys the only copy of those vectors. Relax the check to >= and keep the bijection and size checks as the integrity backstop.
- Fix mempalace_find_tunnels params: (required) -> optional. The MCP
handler defaults both wing_a and wing_b to None
(mempalace/mcp_server.py:1277), so the prior docs were factually
wrong. Caught by gemini-code-assist on PR #1719.
- Clarify implicit-vs-explicit tunnel distinction with consistent
casing and a brief in-line definition (implicit = discovered from
drawer content overlap; explicit = user/agent-declared link).
Suggested by copilot-pull-request-reviewer.
- Split the mempalace_memories_filed_away one-liner into a short
description plus 'Returns' and 'When to call' sub-bullets for
readability. Suggested by copilot-pull-request-reviewer.
The openclaw skill was last updated when mempalace exposed 19 MCP
tools. Since then 13 more agent-facing tools have landed; this PR
documents the 8 that openclaw should expose so agents can call them
natively instead of falling back to `npx mcporter call ...`:
Search & Browse:
- mempalace_list_drawers (paginated drawer listing)
- mempalace_get_drawer (fetch a single drawer by id)
Palace Graph:
- mempalace_create_tunnel (explicit cross-wing link)
- mempalace_list_tunnels (enumerate explicit tunnels)
- mempalace_delete_tunnel (remove an explicit tunnel)
- mempalace_follow_tunnels (walk explicit tunnels from a room)
Write / Session:
- mempalace_update_drawer (mutate content or relocate a drawer)
- mempalace_memories_filed_away (ack the silent auto-save hook)
The 3 admin-only tools (mempalace_sync, mempalace_hook_settings,
mempalace_reconnect) are intentionally left out — they're host/admin
operations, not agent-facing memory operations. The Hermes
MemoryProvider plugin landing in MemPalace/mempalace#1684 makes the
same call.
Version bumped 3.3.0 -> 3.4.0 (additive tool surface, no breaking
changes to existing tool docs).
The mempalace_diary_write tool declared a top-level anyOf in its
input schema to require either entry or content. Anthropic's Messages
API rejects any tool schema with a top-level anyOf/oneOf/allOf and
returns a 400 for the entire tools array, so every MCP session failed
to start.
The entry/content constraint is already enforced at dispatch: content
is remapped to entry before the handler runs, and a missing value
returns -32602. Removing the combinator restores compatibility without
weakening validation.
Closes#1711
Expose mining as an MCP tool so clients that cannot shell out (Claude
Desktop, LM Studio, Aionui, Desktop Commander) can index projects,
conversations, or documents in-conversation, not only through the
`mempalace mine` CLI.
tool_mine is a synchronous wrapper over the existing miners (miner.mine,
convo_miner.mine_convos, format_miner.mine_formats) that cmd_mine already
calls, so it adds no new ingestion logic and no backend coupling. Miner
stdout is captured at the Python and file-descriptor level so it cannot
corrupt the JSON-RPC channel (#225); no Unix-only calls, so it works on
Windows. The miners keep the palace write lock, so a concurrent mine
returns a structured already-running error.
The two-job split passed the wheel from the build job to the publish job
via upload/download-artifact, which failed repeatedly with BlobNotFound on
the same-run download (GitHub artifact-service flake) — the PyPI upload step
never ran. Collapse into a single `publish` job that builds and publishes in
the same workspace, removing the handoff entirely. The job keeps the `pypi`
environment gate, OIDC (id-token: write), refs/tags checkout, and the
on-main + version-manifest checks.
Address #1707 review:
- _as_list also wraps a bare dict (single metadata) — list({'k':1}) -> ['k']
would drop the values — and returns list inputs as-is (no copy, Copilot
perf note); other iterables are materialized once.
- Normalize ids and metadatas (not just documents) in add/upsert/update so
a scalar id/metadata stays length-aligned with documents/embeddings.
- Widen query_texts annotation to list[str] | str to match the behavior.
- Tests: bare-str ids + dict metadatas, dict wrapping, list-returned-as-is.
Address Copilot review on #1708:
- Check out refs/tags/<tag> instead of the bare name, so a same-named
branch can't be resolved instead of the tag object (checkout prefers
branches on ambiguous refs).
- Tighten the dispatch-input validation to require vMAJOR.MINOR.PATCH
with an optional -/+ suffix, rejecting loose values like v3 / v3foo.
GitHub doesn't reliably emit a release event when a draft tied to a
pre-existing tag is published (e.g. publishing the long-standing v3.3.6
draft fired no run). Add a workflow_dispatch trigger with a 'tag' input so
a maintainer can run the publish for any existing tag from the Actions tab
or 'gh workflow run'. A resolve step picks the release tag or the input,
format-validates it (^v[0-9][0-9A-Za-z.+-]*$) before any use as a git ref,
and the existing on-main + version-match checks + the pypi approval gate
apply unchanged to both paths.
EmbeddingCollection did _embed_texts(list(documents)). For ChromaDB's
OneOrMany shape, a bare str document splits into per-character 'docs'
(list("abc") -> ['a','b','c']), embedding each character and breaking
length alignment with ids/metadatas on explicit-vector backends
(pgvector, sqlite_exact). Normalize str -> [str] via _as_list() at all
four sites (add/upsert/update documents, query query_texts) and pass the
normalized list to the inner backend too. Addresses PR #1706 review
(Gemini + Copilot, HIGH).
Follow-up to the wing-name normalization (#1675). Palaces built before the
rule filed drawers under leading/trailing-separator wing names (e.g. a
Claude Code path-encoded dir `-home-user-proj` -> `_home_user_proj`); the
new derivation strips those, so searches/diary reads under the new name miss
the old memories — the history is split, not lost.
`migrate_wing_names` (CLI: `mempalace migrate-wings [--dry-run] [--yes]`)
re-keys the `wing` metadata field on drawers and closets to the normalized
form, merging collisions. Design (verified against the codebase):
- Drawer/closet IDs embed the wing as an opaque prefix that is never decoded
back into a wing, and mining idempotency keys on `source_file`, so IDs are
left untouched: closet ->drawer_id pointers stay valid and future mining
still skips already-mined files. No risky extract-rebuild needed.
- `topics_by_wing` registry keys are re-keyed (merging on collision).
- Tunnels resolve via existing read-time normalization and need no rewrite.
- Backend-agnostic (uses the get/update collection abstraction); idempotent;
dry-run-able. The strip is applied by the migration itself, so it is
correct regardless of whether the running build's normalize_wing_name
already carries the #1675 change.
Tests: pure planner coverage (strip, no-op, empty/non-string, collision) +
hermetic backend integration (relabel, merge, dry-run, idempotency).
test_load_config_uses_defaults_when_yaml_missing asserted the derived
wing equals project_root.name. That only held when the random tempfile
name had no separators; tempfile's alphabet includes '_', so once
normalize_wing_name strips leading/trailing '_' (this PR), a name like
'tmpXXXX_' makes the derived wing diverge from the raw name. Compare
against normalize_wing_name(project_root.name) — the actual contract —
which is deterministic across platforms. (Surfaced as a test-windows
failure on this PR, but it was cross-platform flaky.)
- searcher: read the unfiltered fallback result via _first_or_empty()
instead of raw["documents"][0], matching the codebase's QueryResult/dict
polymorphism helper and guarding the empty-result IndexError (Gemini).
- mcp_server: the content->entry remap now fills only when 'entry' is
absent or None, so an explicit (even "") entry wins over the alias —
the truthiness check could clobber an empty entry (Gemini/Copilot).
- mcp_server: diary_write schema now expresses the real contract —
agent_name required + anyOf(entry, content) — so schema-validating
clients can legally call with content only (Copilot).
- test: lock the explicit-empty-entry edge (content must not override).
#1245 landed on develop with three defects that turned the branch red:
- tool_diary_write gave `entry` a default (`entry: str = None`), which
silently disabled the signature-based missing-parameter diagnostic
(-32602) — two TestParamShapeDiagnostics tests failed.
- the `content` alias was never added to the tool input schema, so the
dispatch arg-filter stripped `content` before the handler ever saw it,
so the alias never actually worked.
- the filtered-search fallback inlined into search_memories pushed it
over the C901 complexity ceiling (30 > 25), and mcp_server.py was left
unformatted.
Fix:
- restore `entry` as a required param (revives the -32602 diagnostic)
- add `content` to the diary_write schema and remap content->entry at
dispatch, before the handler, so a content-only call still satisfies
the required `entry` (entry wins if both are supplied)
- extract the fallback into _query_drawers_with_filter_fallback() so
search_memories drops back under the complexity ceiling
- ruff format mcp_server.py
- add regression tests for the content alias (content-only + both-supplied)
Keeps #1245's filtered-search recall fix intact; turns develop green.