Commit Graph
1588 Commits
Author SHA1 Message Date
Igor Lins e Silva ed77125ae3 feat(backends): shared embedder-identity sidecar + qdrant identity (RFC 001, #1730)
Completes the embedder-identity contract for the last backend (qdrant) and
unifies identity persistence across all local-path backends.

Identity is stored in a small per-palace sidecar (mempalace_embedder.json),
NOT in a backend's mismatch marker. The marker's presence signals "palace
initialized" (reads raise CollectionNotInitializedError when the marker exists
but the store doesn't), so recording identity at first empty open must not
create it — a sidecar is unguarded, so a brand-new palace records identity
immediately. This fixes a latent gap (caught in review) where pgvector/qdrant
palaces stayed permanently "unknown" because the marker isn't written until the
first real write.

- New mempalace/backends/_sidecar.py: shared read/write_embedder_sidecar with
  the isinstance robustness the review bots taught us; chroma, pgvector, and
  qdrant all use it (chroma's inline copy is removed, pgvector switches off the
  marker, qdrant adds it).
- QdrantCollection.get/set_embedder_identity delegate to the sidecar, so
  palace.get_collection enforces a model swap on a qdrant palace exactly like
  the other backends — no live server needed for the check.

Tests: sidecar roundtrip + brand-new-palace recording (creates sidecar without
a marker) + enforcement model-swap raise, for qdrant and pgvector, all
server-free; chroma identity tests unchanged. Full suite: 2495 passed, 82.52%.

Closes #1730. Refs #743, #1724.
2026-06-08 10:20:45 -03:00
Igor Lins e Silva 4ceb880d19 Merge pull request #1732 from MemPalace/feat/maintenance-hooks
feat(backends): observable maintenance hooks + pgvector indexed-build path (RFC 001)
2026-06-08 09:51:29 -03:00
Igor Lins e Silva fdbafe5420 feat(backends): observable maintenance hooks + pgvector indexed-build path (RFC 001, #1725)
Adds the maintenance contract RFC 001 specifies but #1679 deferred, and gives
pgvector an opt-in HNSW index path with concurrency-safe builds.

- base.py: MaintenanceResult (status ran/already_running/noop + free-form
  stats), UnsupportedMaintenanceKindError, BaseBackend.maintenance_kinds
  ClassVar (reserved: analyze/compact/reindex; a backend with no analogue MUST
  omit, not no-op), and BaseCollection.maintenance_state()/run_maintenance()
  defaults. EmbeddingCollection forwards both (BaseCollection methods shadow
  __getattr__).
- sqlite_exact: analyze (ANALYZE) + compact (VACUUM, autocommit + page stats);
  omits reindex (exact scan, no ANN index). maintenance_state reports row/page
  counts.
- pgvector: reindex builds the optional HNSW index, serialized by a
  session-level pg_advisory_lock so concurrent daemon writers learn
  "already_running" instead of each stacking an ACCESS EXCLUSIVE build (the
  production wedge). It is opt-in: the default exact `<=>` scan is the
  100%-recall path; an HNSW index trades exact recall for scale, so an operator
  invokes it deliberately. Also analyze; omits compact (autovacuum). Advertises
  supports_server_side_indexes. maintenance_state reports index presence.
- qdrant/chroma: empty maintenance_kinds (qdrant self-optimizes; chroma
  maintenance is the separate repair CLI) — the faithful "omit" default.

Tests: contract + sqlite (real, CI-runnable) + pgvector advisory-lock flow via
a fake client (ran/noop/already_running, no live Postgres). Full suite green:
2488 passed, 82.47% coverage.

Benchmark three-phase wiring is deferred — the existing benchmarks/ are
task-benchmarks, not backend-comparison harnesses, so there is nothing to wire
into yet.

Closes #1725. Refs #743.
2026-06-08 08:44:20 -03:00
Igor Lins e Silva 13de7a6743 Merge pull request #1731 from MemPalace/feat/embedder-identity
feat(backends): embedder-identity contract + three-state enforcement (RFC 001)
2026-06-08 08:21:58 -03:00
Igor Lins e Silva 212051da35 feat(backends): embedder-identity contract + three-state enforcement (RFC 001, #1724)
The explicit-embedding backends (pgvector, qdrant, sqlite_exact) embed through
EmbeddingCollection and persist nothing about which model produced their
vectors. Swapping two same-dimension models (minilm <-> embeddinggemma, both
384-d) silently corrupts retrieval with no error — the worst failure class for
a verbatim-recall system. This records the model name and refuses a swap on open.

- base.py: EmbedderIdentity dataclass, Embedder protocol,
  EmbedderIdentityUnknownWarning, and a three-state check_embedder_identity()
  helper (known_match / known_mismatch / unknown) raising
  EmbedderIdentityMismatchError / DimensionMismatchError. BaseCollection gains
  get/set/effective_embedder_identity hooks.
- embedding.py: current_model_name() (cheap) + get_embedder_identity() (probed
  dimension, cached per process).
- Persistence per backend: sqlite_exact meta table, pgvector marker JSON
  (preserved across rewrites), chroma sidecar JSON. EmbeddingCollection forwards
  the hooks explicitly (BaseCollection methods shadow __getattr__).
- palace.get_collection enforces at open: a model swap raises; a brand-new empty
  collection records the current model; a populated-but-unrecorded legacy palace
  warns and is resolved via the CLI. The check needs no model load.
- CLI: `mempalace palace set-embedder [--model NAME] [--force]` records/overrides
  identity without mutating global config or loading a foreign model.

Chroma additionally keeps its native embedding-function check. Qdrant identity
persistence is a fast-follow (no local metadata slot — needs a companion store).

Closes #1724. Refs #743.
2026-06-08 08:14:26 -03:00
Igor Lins e Silva 52144704e7 Merge pull request #1727 from MemPalace/feat/searcher-backend-neutral
feat(searcher): metric-aware distance→similarity conversion (RFC 001 §10)
2026-06-08 06:21:59 -03:00
Igor Lins e Silva ffa8ac142c feat(searcher): metric-aware distance→similarity conversion (RFC 001 §10, #1726)
The searcher hard-coded `max(0, 1 - distance)` for vector similarity, which is
correct only for cosine. A backend reporting L2 or inner-product distances — or
a legacy Chroma palace built without `hnsw:space=cosine` — was silently
mis-ranked: L2 distances routinely exceed 1.0 and floored every result to 0.

- backends/base.py: `BaseBackend.distance_metric` ClassVar (cosine default) and
  a `BaseCollection.distance_metric` property per RFC 001 §2.1.
- backends/chroma.py: `ChromaCollection.distance_metric` reads the real
  `hnsw:space`, so a legacy-L2 palace reports "l2" and ranks correctly.
- searcher.py: `_distance_to_similarity(distance, metric)` (cosine/l2/ip,
  monotonic + bounded) and `_metric_for_collection(col)`; `_hybrid_rank` and the
  CLI display are now metric-aware. Cosine output is unchanged.
- Same conversion applied to the MCP `similarity` field, layers.py L3 search,
  and the dedup threshold so reported similarities are consistent.

Cosine behavior is byte-for-byte preserved; only non-cosine metrics change.
Part B of #1726 (routing the chroma.sqlite3 BM25 fallback through
BaseCollection) is tracked separately. Refs #1726, #743.
2026-06-08 02:48:26 -03:00
mvalentsev 9a8c87acfe fix(backends): keep post-deletion dim-None HNSW segments instead of quarantining (#1710)
_missing_dimensionality_appears_recoverable required total_elements_added == len(id_to_label) before keeping a dim-None segment, but total_elements_added is a monotonic add-counter while id_to_label holds only the live elements. After any delete the counts diverge, so every post-deletion segment was quarantined; once the WAL is pruned that destroys the only copy of those vectors. Relax the check to >= and keep the bijection and size checks as the integrity backstop.
2026-06-08 01:27:52 +05:00
chenyuxuan 6c5e1be3db fix(mcp): fail closed when add_drawer idempotency pre-check fails 2026-06-07 17:32:25 +08:00
jsiu93 74453ee3ac fix: handle rootless Java subprojects 2026-06-07 16:01:57 +08:00
jsiu93 ae415b8620 fix: detect Java project manifests 2026-06-07 15:50:15 +08:00
raman325 ca38ae899d docs(openclaw): address review round 1
- Fix mempalace_find_tunnels params: (required) -> optional. The MCP
  handler defaults both wing_a and wing_b to None
  (mempalace/mcp_server.py:1277), so the prior docs were factually
  wrong. Caught by gemini-code-assist on PR #1719.
- Clarify implicit-vs-explicit tunnel distinction with consistent
  casing and a brief in-line definition (implicit = discovered from
  drawer content overlap; explicit = user/agent-declared link).
  Suggested by copilot-pull-request-reviewer.
- Split the mempalace_memories_filed_away one-liner into a short
  description plus 'Returns' and 'When to call' sub-bullets for
  readability. Suggested by copilot-pull-request-reviewer.
2026-06-06 22:58:04 -04:00
raman325 29d7e1edcf docs(openclaw): catch up SKILL.md with 8 newer MCP tools
The openclaw skill was last updated when mempalace exposed 19 MCP
tools. Since then 13 more agent-facing tools have landed; this PR
documents the 8 that openclaw should expose so agents can call them
natively instead of falling back to `npx mcporter call ...`:

Search & Browse:
  - mempalace_list_drawers   (paginated drawer listing)
  - mempalace_get_drawer     (fetch a single drawer by id)

Palace Graph:
  - mempalace_create_tunnel  (explicit cross-wing link)
  - mempalace_list_tunnels   (enumerate explicit tunnels)
  - mempalace_delete_tunnel  (remove an explicit tunnel)
  - mempalace_follow_tunnels (walk explicit tunnels from a room)

Write / Session:
  - mempalace_update_drawer  (mutate content or relocate a drawer)
  - mempalace_memories_filed_away (ack the silent auto-save hook)

The 3 admin-only tools (mempalace_sync, mempalace_hook_settings,
mempalace_reconnect) are intentionally left out — they're host/admin
operations, not agent-facing memory operations. The Hermes
MemoryProvider plugin landing in MemPalace/mempalace#1684 makes the
same call.

Version bumped 3.3.0 -> 3.4.0 (additive tool surface, no breaking
changes to existing tool docs).
2026-06-06 22:55:13 -04:00
Hexecu b1537be5b5 fix(mcp): drop top-level anyOf from diary_write schema
The mempalace_diary_write tool declared a top-level anyOf in its
input schema to require either entry or content. Anthropic's Messages
API rejects any tool schema with a top-level anyOf/oneOf/allOf and
returns a 400 for the entire tools array, so every MCP session failed
to start.

The entry/content constraint is already enforced at dispatch: content
is remapped to entry before the handler runs, and a missing value
returns -32602. Removing the combinator restores compatibility without
weakening validation.

Closes #1711
2026-06-07 00:45:48 +02:00
mvalentsev c4da6d500b fix(searcher): scope neighbor expansion by parent_drawer_id (#1580) 2026-06-06 18:46:17 +05:00
mvalentsev c9dc4c466d fix(miner): count only new work toward --limit, not already-mined skips (#1535) 2026-06-06 17:56:08 +05:00
mvalentsev 2a66bad497 feat(mcp): add mempalace_mine tool (#1662)
Expose mining as an MCP tool so clients that cannot shell out (Claude
Desktop, LM Studio, Aionui, Desktop Commander) can index projects,
conversations, or documents in-conversation, not only through the
`mempalace mine` CLI.

tool_mine is a synchronous wrapper over the existing miners (miner.mine,
convo_miner.mine_convos, format_miner.mine_formats) that cmd_mine already
calls, so it adds no new ingestion logic and no backend coupling. Miner
stdout is captured at the Python and file-descriptor level so it cannot
corrupt the JSON-RPC channel (#225); no Unix-only calls, so it works on
Windows. The miners keep the palace write lock, so a concurrent mine
returns a structured already-running error.
2026-06-06 17:54:38 +05:00
Milla J be954671ca Merge pull request #1706 from MemPalace/develop
Release v3.4.0 — promote develop to main
v3.4.0
2026-06-06 01:43:11 -07:00
Igor Lins e Silva 939a076baf Merge pull request #1709 from MemPalace/ci/publish-single-job
ci(publish): build + publish in one job (fix BlobNotFound artifact handoff)
2026-06-06 05:13:55 -03:00
Igor Lins e Silva 19d2f9216d ci(publish): build + publish in one job to avoid cross-job artifact handoff
The two-job split passed the wheel from the build job to the publish job
via upload/download-artifact, which failed repeatedly with BlobNotFound on
the same-run download (GitHub artifact-service flake) — the PyPI upload step
never ran. Collapse into a single `publish` job that builds and publishes in
the same workspace, removing the handoff entirely. The job keeps the `pypi`
environment gate, OIDC (id-token: write), refs/tags checkout, and the
on-main + version-manifest checks.
2026-06-06 05:10:52 -03:00
Igor Lins e Silva 72a69e2476 Merge pull request #1707 from MemPalace/fix/embedding-wrapper-oneormany
fix(backends): wrap bare-str OneOrMany inputs before embedding
2026-06-06 05:08:07 -03:00
Igor Lins e Silva 051f0cc1ec fix(backends): normalize all OneOrMany inputs in EmbeddingCollection
Address #1707 review:
- _as_list also wraps a bare dict (single metadata) — list({'k':1}) -> ['k']
  would drop the values — and returns list inputs as-is (no copy, Copilot
  perf note); other iterables are materialized once.
- Normalize ids and metadatas (not just documents) in add/upsert/update so
  a scalar id/metadata stays length-aligned with documents/embeddings.
- Widen query_texts annotation to list[str] | str to match the behavior.
- Tests: bare-str ids + dict metadatas, dict wrapping, list-returned-as-is.
2026-06-06 04:56:19 -03:00
Igor Lins e Silva d464c28e0a Merge pull request #1708 from MemPalace/ci/publish-workflow-dispatch
ci(publish): add workflow_dispatch manual trigger
2026-06-06 04:50:45 -03:00
Igor Lins e Silva d51e9434f9 ci(publish): qualify checkout ref to refs/tags + tighten tag regex
Address Copilot review on #1708:
- Check out refs/tags/<tag> instead of the bare name, so a same-named
  branch can't be resolved instead of the tag object (checkout prefers
  branches on ambiguous refs).
- Tighten the dispatch-input validation to require vMAJOR.MINOR.PATCH
  with an optional -/+ suffix, rejecting loose values like v3 / v3foo.
2026-06-06 04:50:02 -03:00
Igor Lins e Silva 15833ab01a ci(publish): add workflow_dispatch manual trigger
GitHub doesn't reliably emit a release event when a draft tied to a
pre-existing tag is published (e.g. publishing the long-standing v3.3.6
draft fired no run). Add a workflow_dispatch trigger with a 'tag' input so
a maintainer can run the publish for any existing tag from the Actions tab
or 'gh workflow run'. A resolve step picks the release tag or the input,
format-validates it (^v[0-9][0-9A-Za-z.+-]*$) before any use as a git ref,
and the existing on-main + version-match checks + the pypi approval gate
apply unchanged to both paths.
2026-06-06 04:43:35 -03:00
Igor Lins e Silva ce929a8241 fix(backends): wrap bare-str OneOrMany inputs before embedding
EmbeddingCollection did _embed_texts(list(documents)). For ChromaDB's
OneOrMany shape, a bare str document splits into per-character 'docs'
(list("abc") -> ['a','b','c']), embedding each character and breaking
length alignment with ids/metadatas on explicit-vector backends
(pgvector, sqlite_exact). Normalize str -> [str] via _as_list() at all
four sites (add/upsert/update documents, query query_texts) and pass the
normalized list to the inner backend too. Addresses PR #1706 review
(Gemini + Copilot, HIGH).
2026-06-06 04:35:15 -03:00
Igor Lins e Silva 9694e2cb57 Merge pull request #1705 from MemPalace/chore/bump-3.4.0
chore(release): 3.4.0
2026-06-06 03:57:46 -03:00
Igor Lins e Silva 73d4e43b22 chore(release): bump README version badge to 3.4.0
Enforced by test_readme_claims.py::TestVersionBadge — a sixth version
location outside the version-guard 5-file set.
2026-06-06 03:41:49 -03:00
Igor Lins e Silva e66cfff0c0 chore(release): 3.4.0
Minor release. Highlights since v3.3.6:
- feat: pluggable vector backends (qdrant, pgvector, sqlite_exact)
- feat: Docker image for MCP server + CLI
- feat: mempalace migrate-wings (normalize legacy wing names)
- feat: known-systems lexicon for compound product names
- fix: embeddinggemma works with ChromaDB 1.5.x (default-model search)
- fix: drawer_id collision data-loss; WAL import side-effect / kill-switch
- ci: PyPI Trusted Publishing pipeline
2026-06-06 03:31:34 -03:00
Igor Lins e Silva ab716d02a5 Merge pull request #1675 from M8SON/fix/wing-normalize-strip-sep
fix(config): strip leading/trailing separators in normalize_wing_name
2026-06-06 03:26:33 -03:00
Igor Lins e Silva ed772a01f9 Merge remote-tracking branch 'origin/develop' into fix/wing-normalize-strip-sep 2026-06-06 03:24:08 -03:00
Igor Lins e Silva 708ef4a9fd Merge pull request #1702 from MemPalace/feat/migrate-wing-normalize
feat(migrate): mempalace migrate-wings — normalize legacy wing names (#1675 follow-up)
2026-06-06 03:20:26 -03:00
Igor Lins e Silva 8ec438ef57 docs(recovery): add wing-name migration guide for migrate-wings 2026-06-06 03:12:49 -03:00
Igor Lins e Silva e46374d24b feat(migrate): mempalace migrate-wings — normalize legacy wing names
Follow-up to the wing-name normalization (#1675). Palaces built before the
rule filed drawers under leading/trailing-separator wing names (e.g. a
Claude Code path-encoded dir `-home-user-proj` -> `_home_user_proj`); the
new derivation strips those, so searches/diary reads under the new name miss
the old memories — the history is split, not lost.

`migrate_wing_names` (CLI: `mempalace migrate-wings [--dry-run] [--yes]`)
re-keys the `wing` metadata field on drawers and closets to the normalized
form, merging collisions. Design (verified against the codebase):

- Drawer/closet IDs embed the wing as an opaque prefix that is never decoded
  back into a wing, and mining idempotency keys on `source_file`, so IDs are
  left untouched: closet ->drawer_id pointers stay valid and future mining
  still skips already-mined files. No risky extract-rebuild needed.
- `topics_by_wing` registry keys are re-keyed (merging on collision).
- Tunnels resolve via existing read-time normalization and need no rewrite.
- Backend-agnostic (uses the get/update collection abstraction); idempotent;
  dry-run-able. The strip is applied by the migration itself, so it is
  correct regardless of whether the running build's normalize_wing_name
  already carries the #1675 change.

Tests: pure planner coverage (strip, no-op, empty/non-string, collision) +
hermetic backend integration (relabel, merge, dry-run, idempotency).
2026-06-06 03:11:46 -03:00
Igor Lins e Silva 6ac2de8e04 Merge pull request #1700 from MemPalace/fix/1245-diary-write-and-lint
fix(mcp): repair diary_write content alias + restore -32602 diagnostic (fixes red develop)
2026-06-06 01:25:28 -03:00
Igor Lins e Silva ab669fd3b1 test(miner): compare default wing to normalized dirname, not raw name
test_load_config_uses_defaults_when_yaml_missing asserted the derived
wing equals project_root.name. That only held when the random tempfile
name had no separators; tempfile's alphabet includes '_', so once
normalize_wing_name strips leading/trailing '_' (this PR), a name like
'tmpXXXX_' makes the derived wing diverge from the raw name. Compare
against normalize_wing_name(project_root.name) — the actual contract —
which is deterministic across platforms. (Surfaced as a test-windows
failure on this PR, but it was cross-platform flaky.)
2026-06-06 01:23:09 -03:00
Igor Lins e Silva b130301bd5 fix(mcp): address bot review on #1700
- searcher: read the unfiltered fallback result via _first_or_empty()
  instead of raw["documents"][0], matching the codebase's QueryResult/dict
  polymorphism helper and guarding the empty-result IndexError (Gemini).
- mcp_server: the content->entry remap now fills only when 'entry' is
  absent or None, so an explicit (even "") entry wins over the alias —
  the truthiness check could clobber an empty entry (Gemini/Copilot).
- mcp_server: diary_write schema now expresses the real contract —
  agent_name required + anyOf(entry, content) — so schema-validating
  clients can legally call with content only (Copilot).
- test: lock the explicit-empty-entry edge (content must not override).
2026-06-06 01:11:54 -03:00
Igor Lins e Silva ad9874fcd5 fix(mcp): repair diary_write content alias + restore -32602 diagnostic
#1245 landed on develop with three defects that turned the branch red:

- tool_diary_write gave `entry` a default (`entry: str = None`), which
  silently disabled the signature-based missing-parameter diagnostic
  (-32602) — two TestParamShapeDiagnostics tests failed.
- the `content` alias was never added to the tool input schema, so the
  dispatch arg-filter stripped `content` before the handler ever saw it,
  so the alias never actually worked.
- the filtered-search fallback inlined into search_memories pushed it
  over the C901 complexity ceiling (30 > 25), and mcp_server.py was left
  unformatted.

Fix:
- restore `entry` as a required param (revives the -32602 diagnostic)
- add `content` to the diary_write schema and remap content->entry at
  dispatch, before the handler, so a content-only call still satisfies
  the required `entry` (entry wins if both are supplied)
- extract the fallback into _query_drawers_with_filter_fallback() so
  search_memories drops back under the complexity ceiling
- ruff format mcp_server.py
- add regression tests for the content alias (content-only + both-supplied)

Keeps #1245's filtered-search recall fix intact; turns develop green.
2026-06-06 00:42:26 -03:00
Igor Lins e Silva 334fc6d0be Merge pull request #1673 from terrizoaguimor/fix/sanitize-documents-chromadb-chokepoint
fix(backends): strip lone surrogates from documents at the ChromaDB chokepoint
2026-06-06 00:23:46 -03:00
Igor Lins e Silva 1ed7f49f13 Merge pull request #1635 from trek-e/fix/mcp-search-stale-index-retry
fix(mcp): retry stale-index transient in tool_search
2026-06-06 00:23:34 -03:00
Igor Lins e Silva 097687cf63 Merge pull request #1687 from robertcprice/fix/tool-use-input-list-guard
Fix crash when tool_use input is a list instead of dict
2026-06-06 00:23:20 -03:00
Igor Lins e Silva 46e3b22316 Merge pull request #1694 from mvalentsev/fix/1693-hook-diary-agent
fix(hooks): file stop-hook diary checkpoints under the harness agent identity (#1693)
2026-06-06 00:17:14 -03:00
Igor Lins e Silva bbe6ff9477 Merge pull request #1245 from lucian-the-traveler/fix/filtered-search-fallback-and-diary-write-alias
fix: filtered search fallback + diary_write content alias
2026-06-06 00:16:58 -03:00
Igor Lins e Silva b1df41b943 Merge pull request #1137 from arnoldwender/fix/convo-miner-preserve-response-structure
fix(convo_miner): preserve blank lines and indentation in AI responses
2026-06-06 00:16:33 -03:00
Igor Lins e Silva e1fce46006 Merge pull request #1602 from mvalentsev/fix/1573-quarantine-rearm-mtime
fix(backends): re-arm HNSW quarantine gate on mtime change and explicit reconnect (#1573)
2026-06-06 00:16:12 -03:00
Igor Lins e Silva cace2230d8 Merge pull request #1618 from mvalentsev/fix/1579-hnsw-flush-at-mine-end
fix(backends): lower HNSW bloat-guard thresholds to fix sub-50k persist (#1579)
2026-06-06 00:15:52 -03:00
Igor Lins e Silva 04af6a436d Merge pull request #1100 from arnoldwender/fix/closet-llm-gbk-symbols
fix(closet_llm): replace non-ASCII symbols in progress output (#1034)
2026-06-05 23:45:38 -03:00
Igor Lins e Silva e00bc666e7 Merge pull request #1102 from arnoldwender/fix/normalize-utf8-bom
fix(normalize): handle UTF-8 BOM in transcript files
2026-06-05 23:45:23 -03:00
Igor Lins e Silva 593cfebd79 Merge pull request #1682 from potterdigital/fix/status-sqlite-count
fix(status): count drawers from sqlite instead of cold-loading the HNSW index
2026-06-05 23:44:40 -03:00
Igor Lins e Silva 6434370034 Merge pull request #1666 from MemPalace/fix/80-drawer-id-collision-delimiter
fix(ids): delimit hash inputs to prevent drawer_id collisions (#80)
2026-06-05 23:43:27 -03:00