206 Commits
Author SHA1 Message Date
Igor Lins e Silva 2c08bb251d Merge pull request #1855 from MemPalace/develop
Release v3.5.0 — promote develop to main
2026-06-22 22:00:13 -03:00
Igor Lins e Silva 65fa1517a1 Merge pull request #1853 from MemPalace/chore/release-3.5.0
chore(release): 3.5.0
2026-06-22 16:48:16 -03:00
Igor Lins e Silva e8f96dd8b2 chore(release): 3.5.0
Bump version to 3.5.0 across version.py, pyproject.toml, the Claude/Codex
plugin manifests, the README badge, and uv.lock. Refresh the "N MCP tools"
prose from 34 to 35 (delete_by_source #1729 and checkpoint #1851 each added a
tool). Add the 3.5.0 CHANGELOG entry.
2026-06-22 16:39:37 -03:00
Igor Lins e Silva 270f1aede9 Merge pull request #1854 from MemPalace/ci/windows-chromadb-flake-retries
ci(test-windows): retry the transient ChromaDB HNSW compaction flake
2026-06-22 16:38:31 -03:00
Igor Lins e Silva 37194cf543 ci(test-windows): retry the transient ChromaDB HNSW compaction flake
ChromaDB's rust HNSW core intermittently fails compaction on Windows with
"Failed to apply logs to the hnsw segment writer" during add/update — a
long-standing, non-reproducible-on-Linux/macOS flake that hits different tests
(test_migrate_wings, test_closets) across unrelated commits and has been
turning otherwise-green release/CI runs red at random.

Add pytest-rerunfailures and wire `--reruns 2 --only-rerun "Failed to apply
logs to the hnsw segment writer"` into the test-windows job only. The
--only-rerun scope means a real, deterministic failure still fails on the first
run; only this specific transient native-dependency error is retried. The
Linux and macOS jobs deliberately keep zero reruns so genuine regressions
surface there loudly.
2026-06-22 16:01:08 -03:00
Igor Lins e Silva 58ea91b44d Merge pull request #1806 from fatkobra/fix/1801-http-transport-stdio
feat(mcp): add opt-in HTTP transport
2026-06-22 13:05:13 -03:00
Igor Lins e Silva 1095d684ff fix(mcp): harden HTTP transport — DNS-rebinding guard, optional token, real tests
The opt-in HTTP transport reuses the stdio dispatcher and binds loopback by
default, but /mcp was unauthenticated with no protection against a malicious
web page reaching a DNS-rebound localhost server, and its tests reached for
Starlette/uvicorn (not project deps) so they were silently skipped in CI —
the production _serve_http handler had zero coverage.

Hardening:
- Pin the Host header to loopback literals + the bound host on a loopback bind
  (DNS-rebinding defense); relaxed for a deliberately non-loopback bind, which
  is the operator's call and may sit behind a Host-rewriting proxy.
- Reject any browser Origin that isn't a loopback origin (rebinding/SSRF guard);
  non-browser MCP clients omit Origin and are unaffected.
- Optional bearer token via MEMPALACE_MCP_HTTP_TOKEN (constant-time compare);
  required on /mcp, never on /healthz so liveness probes work credential-free.
- Warn loudly when bound to a non-loopback host (palace reachable from network).

Testability:
- Split _build_http_server() out of _serve_http() so tests bind 127.0.0.1:0 and
  drive the real handler over a loopback socket via stdlib http.client.
- Replace the skipped Starlette reimplementation with 12 tests covering dispatch,
  initialize, /healthz, 404, parse-error, the 16 MiB cap, notification 202, and
  the Host/Origin/token rejections — no third-party deps.
2026-06-22 12:40:03 -03:00
fatkobra 32fad4844d fix(lint): resolve 7 ruff errors in test_mcp_http_transport 2026-06-22 12:40:03 -03:00
fatkobra eff887d12f fix(tests): rewrite test_mcp_http_transport for Python 3.9-3.13 + Windows 2026-06-22 12:40:03 -03:00
fatkobra ce00cf0718 fix reviewer: handling JSON-RPC 2026-06-22 12:40:03 -03:00
fatkobra 3e3fcf29fe fix(mcp): move _HTTP_REQUEST_LOCK 2026-06-22 12:40:02 -03:00
fatkobra 569beff342 fix(mcp): move _HTTP_REQUEST_LOCK and _HTTP_MAX_REQUEST_BYTES 2026-06-22 12:40:02 -03:00
fatkobra 3fbab55cab test(mcp): make HTTP transport tests network-free 2026-06-22 12:40:02 -03:00
fatkobra 1a47b78dcb test(mcp): make HTTP transport loopback tests proxy-free 2026-06-22 12:40:02 -03:00
fatkobra 960eaac851 test(mcp): bypass proxies in HTTP transport loopback tests 2026-06-22 12:40:02 -03:00
fatkobra 54e9f46027 test(mcp): avoid subprocess flakiness in HTTP transport tests 2026-06-22 12:40:02 -03:00
fatkobra 2e8ff64e3f test(mcp): keep HTTP transport tests Python 3.9 compatible 2026-06-22 12:40:02 -03:00
fatkobra 94519b91dd fix suggestion of reviewer to avoid a critical race condition and other fixes 2026-06-22 12:40:02 -03:00
fatkobra 2f92774eec feat(mcp): add opt-in HTTP transport 2026-06-22 12:40:02 -03:00
Igor Lins e Silva 9902e3a9a5 Merge pull request #1729 from arnoldwender/feat/mcp-delete-by-source
feat(mcp): add mempalace_delete_by_source bulk-cleanup tool (#1722)
2026-06-22 12:37:59 -03:00
Igor Lins e Silva a91f7e06c7 Merge pull request #1820 from fatkobra/fix/1814-session-end-hook
fix(claude-plugin): run final mine on SessionEnd
2026-06-22 12:37:19 -03:00
Igor Lins e Silva 971e5a75c8 Merge pull request #1851 from undeadindustries/feat/cursor-checkpoint-tool
feat: add mempalace_checkpoint batch save tool
2026-06-22 12:37:09 -03:00
Igor Lins e Silva 7392ab8a25 Merge pull request #1817 from mvalentsev/fix/1815-source-file-filter
feat(search): add an optional source_file filter to mempalace_search (#1815)
2026-06-22 12:36:58 -03:00
Igor Lins e Silva e38af8c2e9 Merge pull request #1823 from fatkobra/fix/1818-mcp-peer-writer
fix(mcp): refuse second writer for same palace
2026-06-22 12:36:33 -03:00
Igor Lins e Silva 70ec8e450f Merge pull request #1832 from fatkobra/fix/1796-qdrant-bulk-metadata-scroll
fix(backends): single-scroll bulk metadata fetch for Qdrant; bump scroll page size (#1796)
2026-06-22 12:36:15 -03:00
Igor Lins e Silva a11cf4b667 Merge pull request #1786 from MemPalace/dependabot/github_actions/docker/metadata-action-6
chore(deps): bump docker/metadata-action from 5 to 6
2026-06-22 12:36:08 -03:00
Igor Lins e Silva a9344937df Merge pull request #1787 from MemPalace/dependabot/github_actions/docker/build-push-action-7
chore(deps): bump docker/build-push-action from 6 to 7
2026-06-22 12:36:01 -03:00
Igor Lins e Silva abd780ee36 Merge pull request #1788 from MemPalace/dependabot/github_actions/docker/login-action-4
chore(deps): bump docker/login-action from 3 to 4
2026-06-22 12:35:54 -03:00
Igor Lins e Silva db90cdf132 Merge pull request #1849 from undeadindustries/fix/1843-recovery-guidance
fix: point diverged-index recovery at from-sqlite, not re-mine (#1843)
2026-06-22 12:35:34 -03:00
Igor Lins e Silva 91e2158213 Merge pull request #1847 from mvalentsev/fix/1843-repair-from-sqlite-guidance
fix(repair): point index-read failures to repair --mode from-sqlite (#1843)
2026-06-22 12:35:26 -03:00
Igor Lins e Silva b4a6ffdf9f Merge pull request #1822 from fatkobra/fix/1816-hnsw-strict-repair-kg
fix(chroma): route stale hnsw divergence to sqlite fallback
2026-06-22 12:35:03 -03:00
Igor Lins e Silva 92ac104e31 Merge pull request #1848 from eldar702/fix/1783-create-no-window
fix: use CREATE_NO_WINDOW so Windows hook miner spawns don't flash a console (#1783)
2026-06-22 12:34:43 -03:00
Igor Lins e Silva e576c98ed2 Merge pull request #1798 from trek-e/fix/fact-checker-runpy-warning
fix(tests): run fact_checker __main__ via subprocess to clear runpy warning
2026-06-22 12:34:39 -03:00
Igor Lins e Silva fff8546672 Merge pull request #1769 from techempower-org/test/pgvector-live-conformance
test(backends): live-substrate conformance module for pgvector
2026-06-22 12:34:35 -03:00
Igor Lins e Silva b8c6bfe504 Merge pull request #1840 from mvalentsev/fix/1830-pgvector-pagination
fix(pgvector): push get(limit, offset) pagination into SQL (#1830)
2026-06-22 12:34:31 -03:00
Igor Lins e Silva 7d311af78e Merge pull request #1842 from mvalentsev/fix/sqlite-exact-pagination
fix(backends): push sqlite_exact get(limit, offset) pagination into SQL (#1841)
2026-06-22 12:34:27 -03:00
Igor Lins e Silva 5ae2315f80 fix(mcp): purge matching closets in delete_by_source (#1722)
delete_by_source removed only the drawers, leaving the matching closets
(the AAAK index layer, keyed independently by source_file) behind as stale
pointers at the now-deleted source. Mirror the closet-purge step used by
sync_palace / purge_file_closets: after the drawer delete, best-effort purge
the closets via push-down delete(where=...) so it survives large palaces and
can never abort an already-committed drawer delete.

Dry run now also reports closet_match_count so the caller sees the full blast
radius; commit reports closets_deleted. Adds tests that seed the closet
collection directly (tool_add_drawer doesn't build closets) and assert the
matching closets are purged on commit and counted on dry run.
2026-06-22 12:24:29 -03:00
undeadindustriesandCursor 352eb67a90 test: update Cursor followup assertion for checkpoint tool
The save-hook followup now drives a single mempalace_checkpoint call, so
test_threshold_emits_followup_message must assert that tool name instead
of the old add_drawer/check_duplicate/diary_write trio. Fixes the
test-macos / test-linux CI failures on this branch.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-22 13:54:09 +10:00
undeadindustriesandCursor f3ed6f796e fix: harden tool_checkpoint input validation
Address PR review: guard untrusted MCP client payloads in
mempalace_checkpoint so a single malformed item cannot raise deep in
sanitization and abort the whole batch.

- coerce dedup_threshold to float
- require wing/room/content to be non-empty strings (skip + record error)
- validate the diary object and entry type, recording errors instead of
  silently ignoring a malformed diary

On a genuine dedup-check error we still file the drawer rather than skip:
verbatim recall is the priority and add_drawer's idempotency blocks exact
duplicates. Adds tests for the non-string, dedup-error, and malformed-diary
paths.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-22 13:46:22 +10:00
undeadindustriesandCursor 4291fecfdf feat: add mempalace_checkpoint batch save tool
Collapse the Cursor auto-save sequence (check_duplicate Nx + add_drawer
Nx + diary_write 1x) into a single mempalace_checkpoint MCP call so the
host UI renders one tool-call card and keeps its spinner up for the whole
save. The new tool reuses the existing single-item handlers, so semantic
dedup, idempotency, and verbatim guarantees are unchanged.

- mcp_server.py: add tool_checkpoint + register mempalace_checkpoint
- service.py: classify mempalace_checkpoint as a write tool
- cursor save hook: followup now drives one mempalace_checkpoint call
- docs: new mcp-tools.md section, help.md entry, 33 -> 34 tool count sweep
- tests: checkpoint add/dedup/malformed/registry + classify_tool

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-22 13:41:15 +10:00
undeadindustriesandCursor d7e182a272 docs: add Windows backup alternative to corrupt-index recovery (#1843)
Gemini review on PR #1849: the optional palace backup step used the
Unix-only `cp -a`, which fails on Windows. MemPalace ships on win32, so
add a PowerShell `Copy-Item` alternative alongside the macOS/Linux form
and note that `--archive-existing` already moves the old palace aside.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-22 10:57:19 +10:00
undeadindustriesandCursor 65d07045bd fix: point diverged-index recovery at from-sqlite, not re-mine (#1843)
A diverged HNSW index (for example after a failed chromadb compaction)
leaves the drawer rows intact in chroma.sqlite3 but the vector index out
of sync. Re-mining to recover silently drops drawers added through the
MCP server and diary entries, which have no source file.

- repair-status now recommends `mempalace repair --mode from-sqlite
  --archive-existing` when DIVERGED, instead of the generic `mempalace
  repair`, and explains why re-mining loses data.
- The shared recall protocol and the recall skills (Cursor + Claude
  plugin) document the compactor / "Not connected" recovery path:
  stop the server, rebuild from SQLite, verify, restart — never repair
  in-process from the agent.

Complements #1847 (legacy repair error messages); does not duplicate it.
Does not close #1843 — MCP reconnect resilience and honest add_drawer
write signalling remain open.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-22 10:50:16 +10:00
eldar702 fb8dc1f677 fix: use CREATE_NO_WINDOW so Windows hook miner spawns don't flash a console (#1783)
Fixes #1783
2026-06-21 20:33:55 +03:00
mvalentsevandundeadindustries 44016ad118 fix(repair): point index-read failures to repair --mode from-sqlite (#1843)
When the chromadb compactor cannot apply the WAL into the drawers HNSW
segment (InternalError: Failed to apply logs to the hnsw segment writer),
the legacy repair paths fail on their first Collection.count() read and
advise re-mining from source files. The drawer rows are intact in
chroma.sqlite3, so repair --mode from-sqlite rebuilds them; re-mining
silently drops drawers added via the MCP server and diary entries that
have no source file.

Both legacy read-failure sites (cmd_repair and rebuild_index) now emit
shared guidance pointing at the from-sqlite recovery, worded conditionally
so it also covers a live server or mine still holding the palace open.

Co-Authored-By: undeadindustries <9536461+undeadindustries@users.noreply.github.com>
2026-06-21 22:17:01 +05:00
mvalentsev c203aacf8a ci: re-trigger Windows (flaky closet-boost test) 2026-06-21 18:20:12 +05:00
mvalentsev 4c4c4aba8d fix(mcp): reject non-string source_file with a clean error (#1815)
A JSON number or boolean passed for source_file is not coerced by the
string schema type, so it reached _sanitize_optional_source_file and
raised AttributeError from .strip() rather than a clean validation error.
Add an isinstance guard that raises ValueError, which tool_search returns
as a structured error. Regression test added.
2026-06-21 18:20:12 +05:00
mvalentsevandrendigua2025-gif 7fb7bd3b8d feat(search): add an optional source_file filter to mempalace_search (#1815)
Expose source_file alongside wing/room on mempalace_search. build_where_filter
generalizes to 0/1/2+ clauses and the filter threads through the main vector
path, the index-mismatch fallback, the vector-disabled BM25/SQLite path, and
the union lexical path so it never silently no-ops. Matching is on the exact
full stored value; results now expose source_path (the full path) for round
tripping, since the displayed source_file is a basename. The MCP schema gains
the source_file property and a path-tolerant sanitizer rejects null bytes,
lone surrogates, and overlong values.

Fixes #1815

Co-Authored-By: rendigua2025-gif <253093224+rendigua2025-gif@users.noreply.github.com>
2026-06-21 18:20:12 +05:00
fatkobra 8723d30083 test(mcp_server): add missing _fetch_all_metadata delegation/fallback tests. 2026-06-21 11:29:25 +00:00
fatkobra 2ec48dde56 refactor(qdrant): reuse _rows() in get_all_metadata(); fix sys.modules test pollution
Addresses maintainer review on #1832 (the two non-blocking 🟡 items plus
two 🟢 nits)
2026-06-21 10:57:47 +00:00
mvalentsev 6cc5832176 ci: re-trigger checks (unrelated Windows closet flake) 2026-06-21 05:28:34 +05:00
mvalentsev 386f3c9796 fix(backends): push sqlite_exact get(limit, offset) pagination into SQL
SQLiteExactCollection.get(limit, offset) fetched the whole collection via
_rows() (SELECT ... FROM documents ORDER BY rowid, no LIMIT/OFFSET) and sliced
in Python, so every paginating caller (prefetch_mined_set, status, exporter,
migrate, dedup, sync, ...) re-scanned the entire table per page, making the
sweep O(N^2) in rows materialized.

Push LIMIT/OFFSET into the scan on the unfiltered page (no ids/where/
where_document and non-negative bounds); filtered, id, and negative pages keep
the full-scan plus Python-slice path so the post-filter still runs first. SQLite
requires a LIMIT before OFFSET, so an offset-only page uses LIMIT -1. ORDER BY
rowid keeps pages stable.
2026-06-21 05:11:15 +05:00
Igor Lins e Silva e0da99b2d5 Merge pull request #1839 from MemPalace/fix/graph-stats-sqlite-uri
fix(mcp): route _sqlite_graph_stats through sqlite_read_uri
2026-06-20 20:33:53 -03:00
Igor Lins e Silva 3eacbe0c43 Merge pull request #1834 from mvalentsev/fix/pgvector-surrogate
fix(pgvector): replace lone surrogates so a transcript surrogate no longer aborts the mine (#1833)
2026-06-20 20:11:47 -03:00
mvalentsev a27129a32a fix(pgvector): replace lone surrogates so a transcript surrogate no longer aborts the mine (#1833)
A lone UTF-16 surrogate (U+D800-U+DFFF) in transcript content has no UTF-8
encoding, so pgvector's bulk upsert_rows makes psycopg raise UnicodeEncodeError
and the whole mine aborts, leaving later files unmined.

Apply config.strip_lone_surrogates (-> U+FFFD) to id, document, and the
serialized metadata JSON in upsert_rows. json.dumps(ensure_ascii=False) leaves a
metadata surrogate raw in the string, so one pass over the serialized JSON covers
it; NUL, by contrast, json-escapes and must be stripped before serialization
(see #1829). Replace rather than drop, matching ChromaDB's document handling.

Verified end to end against live Postgres + pgvector: before, a surrogate in
document or metadata aborts the mine; after, it ingests and round-trips as U+FFFD.

Fixes #1833
2026-06-20 20:10:37 -03:00
Igor Lins e Silva 4791999516 Merge pull request #1831 from mvalentsev/fix/1829-pgvector-nul
fix(pgvector): strip NUL bytes so a transcript NUL no longer aborts the mine (#1829)
2026-06-20 20:07:10 -03:00
mvalentsevandhrabbach fa27e41436 fix(pgvector): push get(limit, offset) pagination into SQL (#1830)
PgVectorCollection.get(limit=, offset=) ignored pagination at the SQL
layer: scroll_rows ran SELECT ... WHERE <where> with no LIMIT/OFFSET, so
get() fetched the whole table and sliced in Python. prefetch_mined_set
pages the whole palace on every mine, so mining was O(N^2) in rows
transferred and Python objects built as the palace grows; every other
paginating caller (exporter, migrate, repair, hallways, closet_llm,
miner, palace_graph) paid the same cost.

Push LIMIT/OFFSET into scroll_rows/_scroll with ORDER BY id (the primary
key) for stable offset pagination. get() uses the pushed path only for an
unfiltered page (no ids, no where/where_document, non-negative bounds); a
filtered get keeps the full-scan path because the metadata @> ... pushdown
is broader than the exact _matches_where re-filter for array/object
values, so that re-filter must run before pagination. Full-scroll callers
pass no bound, so their SQL is unchanged.

Co-authored-by: hrabbach <181709360+hrabbach@users.noreply.github.com>
2026-06-21 04:00:32 +05:00
Igor Lins e Silva 31fff1c354 fix(mcp): route _sqlite_graph_stats through sqlite_read_uri
The graph_stats sqlite reader (#1837) and the sqlite_read_uri encoding fix
(#1838) landed in parallel, so _sqlite_graph_stats was the one reader left on
the naive f"file:{db_path}?mode=ro" construction that mis-parses paths with
spaces/special chars. Convert it too — now every read-only sqlite reader
percent-encodes its path. sqlite_read_uri is already imported in mcp_server
from #1838, so this is a call-site-only change.
2026-06-20 19:25:53 -03:00
Igor Lins e Silva fe460c4b8b Merge pull request #1838 from MemPalace/fix/sqlite-ro-uri-encoding
fix: percent-encode sqlite read-only URIs for spaced/special-char paths
2026-06-20 19:24:16 -03:00
Igor Lins e SilvaandCopilot Autofix powered by AI 73772cb707 Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-06-20 19:24:07 -03:00
Igor Lins e Silva 38253b1f5f fix: percent-encode sqlite read-only URIs so spaced/special-char paths open
sqlite3.connect(f"file:{db_path}?mode=ro", uri=True) mis-parses paths
containing spaces or other URI-reserved characters — common in real home
directories (a Windows "First Last" user folder, many macOS paths), and made
worse by Windows backslashes. The database silently fails to open and the
read-only fast paths fall back (or error) on those machines.

Add config.sqlite_read_uri(), which percent-encodes the path via
urllib.request.pathname2url (lazy-imported to keep config import light), and
route every read-only sqlite reader through it:
- mcp_server._tool_status_via_sqlite
- searcher BM25 sqlite fallback
- repair (status / scan / max-seq read paths)
- backends/chroma (5 readers: counts, wing/room tally, id maps, etc.)

All previously used the same naive f-string construction. Surfaced as a
gemini-code-assist review note on #1837.
2026-06-20 18:56:12 -03:00
Igor Lins e Silva 9d5a375017 Merge pull request #1837 from MemPalace/fix/graph-stats-sqlite-fast-path
perf(mcp): sqlite fast path for graph_stats (#1379)
2026-06-20 18:53:56 -03:00
Igor Lins e Silva 73f455c7d0 fix: address PR review feedback on graph_stats sqlite fast path (#1379)
- Soft-fallback on any exception, not just sqlite3.Error, so an unexpected
  schema shape tripping the reconstruction degrades to build_graph() instead
  of raising — matching the sibling sqlite fast paths (Copilot).
- Guard an empty/None _config.palace_path before building db_path (Gemini).
- Test: tripwire _get_collection in addition to graph_stats, directly
  asserting the fast path never opens the chroma client / cold-loads HNSW
  (Copilot).
2026-06-20 18:39:47 -03:00
Igor Lins e Silva 477aa362cd perf(mcp): sqlite fast path for graph_stats to fix large-palace timeouts (#1379)
tool_graph_stats built the whole palace graph via build_graph(), which pages
every metadata row (col.get limit/offset) and cold-loads the HNSW index — the
remaining overview-tool timeout from #1379 (#1836 fixed status / list_wings /
list_rooms / get_taxonomy but deliberately left graph_stats out, as it builds
an in-memory graph rather than a flat tally).

Add _sqlite_graph_stats(): one GROUP BY room, wing, hall over chroma.sqlite3,
reconstructing build_graph's room_data and the same stats (total_rooms,
tunnel_rooms, total_edges, rooms_per_wing, top_tunnels) with the same
per-drawer filter (room present, != "general", wing present) and edge
semantics (C(wings, 2) * halls per multi-wing room). Same _is_chroma_backend()
guard + client-path fallback as the #1748 overview tools.

Test seeds a real chroma palace mirroring the build_graph parity case in
test_palace_graph, with a tripwire on graph_stats proving the fast path runs
and that "general"/wing-less drawers are excluded. Idea adapted from #1381's
_sqlite_graph_stats.
2026-06-20 18:30:43 -03:00
Igor Lins e Silva 2eda1f9130 Merge pull request #1836 from MemPalace/fix/perf-status-and-embedder-threads
perf: sqlite fast-path for overview tools + embedder thread cap (#1748, #1379, #1068)
2026-06-20 18:05:57 -03:00
Igor Lins e Silva 8bdebe1da1 fix: address PR review feedback (preserve "unknown" label; use super().model)
#1748: normalize the sqlite fast path's "?" COALESCE placeholder (and None)
back to "unknown" inside _sqlite_taxonomy, so drawers missing wing/room
metadata keep the client path's output contract — no observable API change
for MCP clients on legacy/partial drawers.

#1068: invoke the parent embedder build via super().model instead of reaching
into cached_property's .func attribute, so the uncapped/fallback path survives
chromadb changing `model` to a plain @property or other descriptor.
2026-06-20 18:05:15 -03:00
mvalentsevandhrabbach abaf09be0a fix(pgvector): strip NUL bytes so a transcript NUL no longer aborts the mine (#1829)
PostgreSQL cannot store NUL (0x00) in text or jsonb. On the pgvector write path
a NUL in `document` is rejected by psycopg ("PostgreSQL text fields cannot
contain NUL (0x00) bytes") and a NUL in `metadata` becomes a JSON unicode escape
the jsonb cast rejects ("unsupported Unicode escape sequence"). `_execute`
re-wraps either as BackendError and `_mine_impl` re-raises, so the whole mine
exits non-zero and every file after the offending one is left unmined. ChromaDB,
SQLite, and Qdrant store the byte verbatim, so only pgvector hard-fails.

Add a recursive `_strip_nul` helper and apply it to id, document, and metadata
in `_PgVectorClient.upsert_rows`, mirroring the backend-layer sanitization
`_sanitize_documents_for_chromadb` already does for lone surrogates on the same
bulk-ingest paths. ids are SHA-256 hashes and metadata keys are fixed field
names, so the id and key passes are no-ops in practice; only transcript-derived
values change.

Co-authored-by: hrabbach <181709360+hrabbach@users.noreply.github.com>
2026-06-21 02:03:05 +05:00
Igor Lins e Silva 0601026026 perf(mcp): answer overview tools from the sqlite aggregate to fix large-palace timeouts (#1748, #1379)
tool_status / list_wings / list_rooms / get_taxonomy paged the entire
collection metadata through the chroma client (`_fetch_all_metadata`, a
1000-row offset loop), which cold-loads the HNSW index and materializes
hundreds of MB of dicts. On six-figure palaces these exceed the MCP host
tool-call limit (180k drawers ~3-4 min; 349k times out at 120-240s). The 5s
metadata cache only dedups repeat calls — it does not stop the cold-call
timeout.

A correct single-query SQL cross-tab already exists
(`backends.chroma._sqlite_wing_room_counts`) and is already the CLI default
(`miner.status`), but the MCP tools never used it — and the MCP-side sqlite
reader only ran behind the `vector_disabled` recovery path.

Add `_sqlite_taxonomy()` (guards on `_is_chroma_backend()`, returns None to
fall back) and wire it as the default path into all four overview tools. They
now answer from one GROUP BY without touching HNSW. Non-chroma backends
(qdrant, sqlite_exact) and unbootstrapped/legacy layouts fall back to the
existing client path unchanged.

graph_stats (also named in #1379) builds an in-memory graph via build_graph()
and needs its own treatment — tracked separately.
2026-06-20 17:47:33 -03:00
Igor Lins e Silva 157022ab1b perf(embedding): cap ORT intra-op threads so a background mine doesn't pin every core (#1068)
ChromaDB's ONNX embedder builds its InferenceSession without a thread cap, so
ORT's intra-op pool defaults to the physical core count. OMP_NUM_THREADS is
inert against it (ORT owns its own pool), so a background `mempalace mine`
pins 4-5 cores and stacked Stop-hook fires turn the machine into a thermal
event.

Add an `embedding_threads` config knob (env MEMPALACE_EMBEDDING_THREADS or
config.json). Unset/"auto" caps the intra-op pool at half the logical CPUs so
a fresh install stays usable out of the box; a positive integer sets an exact
count; 0/negative leaves ORT uncapped for users who want max throughput.

The cap is applied via SessionOptions at session construction:
- `_MempalaceONNX` (default minilm) overrides the `model` cached_property to
  rebuild the session the same way upstream does plus the cap, falling back to
  upstream's uncapped build if chromadb internals shift.
- `EmbeddinggemmaONNX` builds its session through the shared
  `_intra_op_session_options()` helper.
2026-06-20 17:47:22 -03:00
fatkobra 03f638a279 updated tests/test_qdrant_bulk_metadata_scroll.py because of CI failure after push the 2nd commit 2026-06-20 17:47:17 +00:00
fatkobra 5fda4e5231 fix: apply suggested reviewer suggestions 2026-06-20 16:51:30 +00:00
fatkobra b0eddf68dd fix(backends): single-scroll bulk metadata fetch for Qdrant; bump scroll page size (#1796) 2026-06-20 16:17:06 +00:00
Igor Lins e Silva 94d1ced11e Merge pull request #1827 from MemPalace/dependabot/pip/ruff-0.15.18
chore(deps-dev): bump ruff from 0.15.15 to 0.15.18
2026-06-19 18:25:11 -03:00
Igor Lins e Silva 49e427ba35 Merge pull request #1828 from MemPalace/fix/daemon-review-followups
fix(daemon): address post-merge review feedback on #1826
2026-06-19 12:46:10 -03:00
Igor Lins e Silva 7fb981c538 fix(daemon): address post-merge review feedback on #1826
Five fixes from the Copilot review of the merged daemon PR:

1. Privacy: the queue DB's SQLite WAL/SHM sidecars hold un-checkpointed
   verbatim payloads but were created with the caller's umask. Set the
   owner-only umask in run_server BEFORE DaemonRuntime builds the QueueStore
   (not only once the HTTP server starts), and harden any existing sidecars in
   QueueStore._init_db as defense-in-depth.

2. DoS guard: reject a negative Content-Length in the request reader.
   rfile.read(-1) would block until the client disconnects and bypass the
   MAX_BODY_BYTES cap.

3. Side effects: extract _wal_log (+ _ensure_wal, _WAL_FILE, _WAL_REDACT_KEYS)
   into a new side-effect-free mempalace/wal.py. The CLI sync path and the
   daemon service layer obtained _wal_log via `from .mcp_server import _wal_log`,
   which runs mcp_server's import-time stdio protection (os.dup2(2, 1);
   sys.stdout = sys.stderr) in a non-MCP process and misroutes operator output.
   mcp_server/cli/service now import from mempalace.wal.

4. Correctness: run_mcp_tool treated any dict as success. Write tools that
   return a bare {"error": ...} (e.g. tool_create_tunnel/tool_delete_tunnel
   validation) were recorded as succeeded; now the "error" key infers failure.

5. Hook budget: get_client_if_running()/health() take an explicit timeout, and
   the hook "is the daemon up?" precheck uses a short HOOK_PROBE_TIMEOUT (0.5s)
   so a wedged daemon can't stall the hook for the default 5s.

Adds tests/test_wal.py (import isolation + redaction) and daemon tests for the
umask ordering, negative Content-Length, run_mcp_tool error inference, and the
short probe timeout.
2026-06-19 11:56:09 -03:00
fatkobra 32ec879c8b fix(claude-plugin):reviewer feedback for _validate_transcript_path function calls Path.resolve(), which can raise an OSError 2026-06-19 14:21:04 +00:00
Igor Lins e Silva 4e03ce90bc Merge pull request #1826 from MemPalace/feat/daemon-mode
feat: opt-in local daemon for queued MemPalace writes
2026-06-19 11:05:03 -03:00
fatkobra 393107e4ac fix(claude-plugin): resolve SessionEnd merge semantics 2026-06-19 13:44:52 +00:00
mvalentsev d09392c11a feat(hooks): add a budget-safe SessionEnd save hook for clean exits (#1341)
Short sessions that exit cleanly below SAVE_INTERVAL and without a PreCompact
were never saved. Add a SessionEnd hook that takes one final flush.

Claude Code budgets SessionEnd hooks at 1.5s and a plugin-provided timeout
cannot raise it, and a cold mempalace start exceeds that, so the wrapper
backgrounds the work and returns immediately; the detached child completes the
transcript ingest, project mine, and diary checkpoint after the session exits.

The handler validates transcript_path through _validate_transcript_path before
any ingest or diary write, so a traversal or wrong-suffix path is rejected while
the independent project mine still runs.

Adds hook_session_end, both shell wrappers, the plugin hooks.json entry, the
session-end CLI choice, and focused tests.

(cherry picked from commit 10e1450e04fc7cec72984ab7442d3b4fca1490e8)
2026-06-19 13:00:14 +00:00
Igor Lins e Silva fe0391ab56 fix(daemon): skip reverse-DNS in server_bind so startup can't block ~30s
HTTPServer.server_bind() resolves server_name via socket.getfqdn(host). For the
daemon's 127.0.0.1 bind that lookup is pointless, and on a host with slow or
absent reverse DNS it blocks startup until the resolver times out (~30s) — which
looks exactly like the daemon never coming up. This is why the first daemon
HTTP-lifecycle test timed out on the macOS CI runner (httpd_bound=False after
30s) while every later one bound in seconds once the OS had cached the negative
lookup. Bind via TCPServer directly and set server_name from the literal host.
2026-06-19 09:49:44 -03:00
Igor Lins e Silva f868ee78d4 fix(daemon): Windows-safe pid liveness probe; finalize cross-platform daemon tests
_pid_alive used os.kill(pid, 0) as an existence check. On Windows signal 0
is signal.CTRL_C_EVENT, so Python routes it to GenerateConsoleCtrlEvent and
sends a console Ctrl-C to the target's process group rather than probing the
pid. DaemonClient polls a same-process endpoint during startup, so on a CI
runner with an attached console that Ctrl-C was delivered back to the
interpreter as a spurious KeyboardInterrupt — the Windows CI hang that
interrupted the suite at the first daemon HTTP-lifecycle test (socket.py
recv). Probe via the Win32 OpenProcess/WaitForSingleObject handle API instead,
which has no signalling side effects. This is also a real Windows production
bug, not just a test artifact.

Tests:
- Skip the two owner-only (0600) permission tests on Windows: os.chmod cannot
  represent POSIX mode bits there (files report 0o666); the daemon relies on
  user-profile ACLs on Windows.
- _start_server now captures and re-surfaces a run_server thread crash instead
  of spinning for 30s and failing with a bare assert (diagnoses the macOS
  startup flake).
- Add a regression test asserting _pid_alive is correct and emits no console
  control event when hammered like the poll loop.
- Remove the temporary win32 exit-hang diagnostic fixture from conftest now
  that the root cause is fixed.
2026-06-19 09:38:34 -03:00
Igor Lins e Silva 95211e84cb test: win32-only diagnostic for daemon process-exit hang
The Windows CI run passes all 666 tests then hangs at interpreter shutdown
(KeyboardInterrupt at socket.py:723) until the runner kills it. All daemon
lifecycle tests assert their server threads died, so the hang is a different
non-daemon thread blocked on a socket — not the daemon server thread. CI
round-trips can't show which thread it is.

Add a win32-only session fixture that:
  - arms faulthandler.dump_traceback_later(130s) to print every thread's stack
    to stderr once the hang has run a while, and
  - prints every live thread (name + daemon flag) at session teardown — a
    non-daemon thread present there is the shutdown blocker.

Gated to sys.platform == 'win32' so Linux/macOS CI see no extra output. Remove
once the Windows hang is fixed.
2026-06-19 09:01:50 -03:00
Igor Lins e Silva d859d5a565 fix: daemon client bypasses proxy discovery; tests force-shutdown server thread
DaemonClient.request now uses a no-proxy opener (build_opener(ProxyHandler({})))
instead of urllib.urlopen. The daemon is always on 127.0.0.1, so a request must
never go through an HTTP proxy — this is the correct production choice. It also
bypasses urllib's proxy discovery (macOS _scproxy via SystemConfiguration), which
runs on the first request to any host and is NOT bounded by the per-request
timeout: on a CI runner with no network it hangs for tens of seconds, which looked
exactly like the daemon never came up (test_daemon_http_lifecycle_executes_job
timed out at 30.18s). With the no-proxy opener the lifecycle runs in 0.78s and no
server thread is leaked — which also removes the timing skew that made the
sqlite_exact concurrent-connection test flake on macOS CI.

The leaked server thread was also the Windows exit-hang root cause: a slow/failed
client.shutdown() POST left serve_forever running, and the interpreter blocked on
the open listening socket at process exit. Tests now capture the httpd run_server
creates (by subclassing daemon.ThreadingHTTPServer) and force httpd.shutdown() +
server_close() from the test thread if the normal shutdown path leaves the thread
alive, asserting the thread died so a leak becomes a visible failure instead of
a silent exit hang.
2026-06-19 08:52:34 -03:00
Igor Lins e Silva 5f58cdbfa8 fix: unblock daemon PR CI + address review comments
CI was red on all three platforms for the daemon-mode draft PR. Root causes
and fixes:

- Linux 3.9 collection error: `_submit_daemon_job`'s `dedupe_key: str | None`
  parameter annotation is evaluated at def time, and hooks_cli.py has no
  `from __future__ import annotations` — `str | None` raises TypeError on 3.9.
  Reverted to `dedupe_key: str = None` (the original, 3.9-safe). The other
  `int | None` in the file is a function-local annotation, which is never
  evaluated, so it was never the problem.

- macOS/Windows daemon lifecycle flakes: the 3 HTTP-lifecycle tests failed at
  the 10s readiness deadline on contended CI runners (localhost bind is
  sub-second locally but took ~5s when it passed on the macOS fleet, >10s when
  it didn't), and because the server thread never shuts down on timeout,
  run_server's `os.environ["MEMPALACE_PALACE_PATH"]` + `os.umask(0o077)`
  mutations leaked into the rest of the suite — poisoning every later test that
  reads MempalaceConfig().palace_path (the 60+ test_mcp_server cascade on macOS;
  the at-exit socket hang → SIGINT on Windows). Bumped the readiness deadline
  to 30s and added a module-scoped snapshot + autouse fixture in test_daemon.py
  that force-restores the env + umask to the pre-suite baseline after every
  daemon test, so a leaked server thread can't poison other test files.

Gemini review comments (fixed in code, no thread replies per convention):

- daemon.py `_connect()` was a bare `sqlite3.connect` whose `with`-block only
  managed the transaction, not the connection — an unbounded FD leak in a
  long-lived daemon running thousands of jobs (also the source of the Windows
  "unclosed database" ResourceWarning noise). Converted to a closing
  @contextlib.contextmanager.
- `QueueStore.finish()` gained `only_if_running`; `_safe_finish` passes it so a
  late worker finish can't overwrite a shutdown-cancelled job back to
  succeeded/failed — removes the reliance on process-exit timing.
- `DaemonClient.request` wraps the final `json.loads` in try/except
  JSONDecodeError → DaemonError, so a non-JSON 2xx response surfaces as a
  structured error instead of a bare JSONDecodeError.
- test_sync.py: removed the module-level `import mempalace.mcp_server` and moved
  the stdout-rebinding side effect into an autouse fixture scoped to
  TestServiceRunSyncReport, so the embedder/Chroma import chain is no longer
  forced at collection time for the existing sync tests.

Coverage: added focused happy-path tests for service.run_sync early-returns,
run_mine backend application + invalid mode, execute_job kind dispatch,
run_diary_write arg forwarding, run_mcp_tool write-tool dispatch, and
print_job_result — lifts service.py from 57% to 85% so the new files
(service 85%, daemon 80%) don't drag the total below the 80% CI gate now that
the daemon tests complete and the gate is actually evaluated.
2026-06-19 08:33:37 -03:00
fatkobra 3c7808e1bc fix(mcp): guard remaining None palace_path in _mcp_sqlite_integrity_refusal. Added one regression test calling the function directly with palace_path=None. 2026-06-19 09:51:02 +00:00
fatkobra 4c46290b80 fix(mcp): applied 3 of the 4 reviewer suggestions 2026-06-19 08:59:25 +00:00
dependabot[bot] 6bccf82a8c chore(deps-dev): bump ruff from 0.15.15 to 0.15.18
Bumps [ruff](https://github.com/astral-sh/ruff) from 0.15.15 to 0.15.18.
- [Release notes](https://github.com/astral-sh/ruff/releases)
- [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md)
- [Commits](https://github.com/astral-sh/ruff/compare/0.15.15...0.15.18)

---
updated-dependencies:
- dependency-name: ruff
  dependency-version: 0.15.18
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-06-19 07:54:00 +00:00
fatkobra 63a382fc6d fix(mcp): gate startup on sqlite integrity failures 2026-06-19 06:42:36 +00:00
Igor Lins e Silva aa96bb5623 feat: add opt-in local daemon for queued MemPalace writes
- New mempalace/daemon.py: long-lived localhost HTTP server (127.0.0.1) with a
  SQLite WAL job queue, single worker thread, bearer-token auth, and owner-only
  file perms (0600/0700) on queue DB, token, endpoint, and log.
- New mempalace/service.py: transport-neutral job execution surface shared by the
  daemon, with per-job env isolation so one job's backend/palace switch cannot
  leak into the next. mcp_tool is allowlisted to write-classified tools only.
- Crash recovery re-queues jobs left 'running' by a killed daemon; jobs that
  already exhausted MAX_ATTEMPTS are dead-lettered to 'failed' instead of being
  retried (non-idempotent diary_write would otherwise duplicate verbatim
  content on every restart).
- Bounded retention prunes terminal jobs older than 7 days
  (MEMPALACE_DAEMON_RETENTION_DAYS); queued/running jobs are never touched so a
  crash mid-prune cannot drop in-flight work.
- CLI: --daemon/--background on mine/sync submit to the queue; new
  `mempalace daemon {start,stop,status,jobs,wait}` subcommand. Strictly opt-in:
  no flag, env, or config means no daemon and no behavior change.
- Hooks opt in via MEMPALACE_HOOKS_DAEMON or config hooks.daemon; when the daemon
  is not already running, hooks fall back to the existing direct/spawn path so
  the 500ms hook budget is preserved (hooks never auto-start the daemon).
- service.run_sync renders the same operator-facing report shape as the direct
  CLI sync path (no_source, out_of_scope, by_source, Re-run/Removed hints) and
  drops the old KeyError-prone 'deleted' read.
2026-06-18 23:46:11 -03:00
Igor Lins e Silva afa749c141 Merge pull request #952 from Davez69gto/feat/csharp-extensions
feat(miner): add C# and .NET file extensions to READABLE_EXTENSIONS
2026-06-18 14:28:59 -03:00
Igor Lins e Silva f0e76c4363 Merge pull request #169 from adv3nt3/feat/pi-cli-normalizer
feat: add Pi agent JSONL session normalizer
2026-06-18 13:20:25 -03:00
Igor Lins e Silva 5935dacc98 Merge remote-tracking branch 'origin/develop' into HEAD
# Conflicts:
#	mempalace/miner.py
2026-06-18 13:19:48 -03:00
Igor Lins e Silva 349485b6b6 Merge remote-tracking branch 'origin/develop' into HEAD
# Conflicts:
#	mempalace/normalize.py
2026-06-18 13:19:48 -03:00
Igor Lins e Silva 78a32c69c1 Merge pull request #204 from FBISiri/feat/gemini-cli-import
feat: add Gemini CLI / AI Studio session import support
2026-06-18 13:04:14 -03:00
Igor Lins e Silva 22cf8afd1e Merge remote-tracking branch 'origin/develop' into HEAD
# Conflicts:
#	mempalace/normalize.py
2026-06-18 12:42:26 -03:00
Igor Lins e Silva cbf6cbe851 Merge pull request #731 from sjhddh/feat/continue-dev-parser
feat(normalize): add Continue.dev session parser
2026-06-18 12:38:15 -03:00
Igor Lins e Silva 301aa335e1 Merge pull request #1720 from jsiu93/feat/java-project-scanner
fix: detect Java project manifests
2026-06-18 12:38:00 -03:00
Igor Lins e Silva d4391abb59 Merge pull request #1368 from EVSalomon/feat/add-support-swift-kotlin
feat(miner): add support for Swift and Kotlin file extensions
2026-06-18 12:37:40 -03:00
Igor Lins e Silva 617aa0c3f3 feat(miner): add PHP ecosystem file extensions (#1819) 2026-06-18 12:37:26 -03:00
fatkobra f0b5cbf374 fix(mcp): cache writer lock setup failures 2026-06-18 12:47:17 +00:00
fatkobra e594bad9d6 fix(mcp): refuse second writer for same palace 2026-06-18 11:24:18 +00:00
fatkobra ea71c5234e fix reviewer feedback for chroma and tests 2026-06-18 11:03:06 +00:00
fatkobra d704433701 fix(chroma): route stale hnsw divergence to sqlite fallback 2026-06-18 10:37:58 +00:00
fatkobra d8d88f4d21 fix reviewer feedback: To prevent a KeyError and provide a clear, actionable assertion failure message if PreCompact is ever missing 2026-06-18 09:20:41 +00:00
fatkobra ad5e7dfe2c fix(claude-plugin): run final mine on SessionEnd 2026-06-18 09:07:06 +00:00
ManuelReschke c2f42cd57e feat(miner): add PHP ecosystem file extensions 2026-06-18 10:31:18 +02:00
Arnold Wender f56e3067eb fix(mcp): harden delete_by_source per review — strip surrogates + type guard
Address Gemini review on #1729:
- normalize source_file with strip_lone_surrogates so exact matching hits
  rows mined from non-ASCII paths via cp1252 stdin (#1488), mirroring
  tool_add_drawer's ingestion-side normalization
- isinstance(str) guard so a non-string source_file returns a clean error
  instead of AttributeError
- default missing wing/room to "" in the dry-run sample, consistent with
  the rest of the file
- add tests: non-string rejection + surrogate-normalization match
2026-06-17 15:08:29 +02:00
Arnold Wender d486aef083 feat(mcp): add mempalace_delete_by_source bulk-cleanup tool (#1722)
Adds an MCP tool to remove every drawer mined from a given source_file
exact match, for cleaning up benchmark/test data accidentally mined into
a user wing (ShareGPT dumps, results_mempal_*.jsonl, language config
JSON) that drowns out real memories in semantic search.

Matching is pushed to the backend via delete(where={"source_file": ...})
the same idiom the miner and diary-ingest paths already use so it is not
subject to the SQLite variable limit regardless of how many drawers share
the source. Defaults to a dry run reporting match count and a sample;
dry_run=false commits. Absent source is an idempotent no-op, not an error.
2026-06-17 15:08:07 +02:00
Eldar Shlomi 9f434e0bfd test(migrate): cover swap-failure rollback
Adds end-to-end regression coverage for the migration swap path where os.replace hits EXDEV, the shutil.move fallback fails, and the original palace must be restored from the rename-aside copy.
2026-06-15 10:40:24 -03:00
Igor Lins e Silva bc9c052b51 Merge pull request #1810 from MemPalace/develop
Release v3.4.1 — promote develop to main
2026-06-15 06:11:35 -03:00
Igor Lins e Silva db6e4f0654 Merge pull request #1811 from MemPalace/fix/3.4.1-review-followups
fix(hooks): portable mtime in macOS hook throttles; doc cleanup
2026-06-15 06:00:18 -03:00
Igor Lins e Silva 3f20305eda style(hooks): single-quote the static python -c mtime snippet
The snippet has no shell interpolation — the path arrives via argv, not
string interpolation — so single quotes are correct and make it
unambiguous that nothing is shell-expanded. Behavior is identical:
`sys.argv[1]` contains no `$`, so it was never expanded (verified
empirically). Matches the single-quoted `python -c` blocks already in
hooks/cursor/lib/common.sh. No functional change.
2026-06-15 05:50:08 -03:00
Igor Lins e Silva 868b4c9b39 fix(hooks): portable mtime in macOS hook throttles; doc cleanup
Address review feedback surfaced on the 3.4.1 release promotion (#1810).

Bug fix — `date -r FILE` is GNU-only. On BSD/macOS `date -r` expects
epoch seconds, not a path, so the staleness/throttle checks in the new
Cursor and Antigravity hooks silently failed on macOS: the state GC
swept on every fire and the pending-save guard was skipped. Replace
with a portable `os.path.getmtime` one-liner via the already-resolved
$MEMPAL_PYTHON_BIN (cursor/lib, antigravity/lib, antigravity save hook).
This restores the "bash 3.2.57 / macOS default" compatibility the
Antigravity changelog claims.

Docs:
- Correct the MCP tool count to 33 (was 19/29/31 in 21 places across
  plugin manifests, READMEs, and website docs — all drifted from the
  TOOLS dict / mcp-tools.md reference, which both have 33).
- Fix broken CHANGELOG link to the Cursor skill (skills/, not
  .cursor-plugin/skills/).
- Fix one-too-many `../` in skills/mempalace/SKILL.md's cursor-hooks
  link (resolved above the repo root).
- Add the required `mcpServers` wrapper to the mcp.json example in
  .cursor-plugin/README.md so copy-paste yields a valid Cursor config.

Left intentionally unchanged: the os.dup2 fd-1 redirect in
mcp_server.py is deliberate (#225 keeps JSON-RPC off fd 1).
2026-06-14 19:54:16 -03:00
Igor Lins e Silva 327da58c09 Merge pull request #1809 from MemPalace/chore/release-3.4.1
chore(release): 3.4.1
2026-06-14 17:56:22 -03:00
Igor Lins e Silva b5c79a1eea chore(release): 3.4.1
Bump version across all sources (version.py, pyproject.toml, both
Claude plugin manifests, Codex plugin manifest, README badge, uv.lock)
and promote the Unreleased changelog to 3.4.1.

Shipping: Cursor IDE plugin + hooks, first-class Antigravity IDE
support (with zero-config interpreter resolution), embeddinggemma
bulk re-embed OOM fix, and backup-retention pruning.

Also rebuilds the CHANGELOG compare-link block, which had been left
at v3.2.0: adds the full 3.3.0-3.4.1 chain plus the previously
undocumented 3.4.0, and points Unreleased at v3.4.1...HEAD. Every
version header now resolves to a compare link.
2026-06-14 17:47:09 -03:00
Igor Lins e Silva 68a971b8a1 Merge pull request #1807 from MemPalace/codex/develop-341-windows-closet-boost-fix
test: stabilize closet boost fixture on Windows
2026-06-14 15:24:07 -03:00
Igor Lins e Silva 160a852bdb test: stabilize closet boost fixture on Windows 2026-06-14 15:17:01 -03:00
Igor Lins e Silva 7f5bbd8be8 Merge pull request #1803 from MemPalace/codex/1800-mine-lock-cleanup
fix(palace): clean source mine locks safely
2026-06-14 15:06:21 -03:00
Igor Lins e Silva 0cb45084bc Merge pull request #1624 from trek-e/fix/search-retry-preserve-collection
fix: preserve collection_name on MCP search retry
2026-06-14 15:03:38 -03:00
Igor Lins e Silva a40e0b7a47 fix: address mine lock review feedback 2026-06-14 14:58:20 -03:00
Igor Lins e Silva 58c45a9905 Merge pull request #1804 from MemPalace/codex/close-blob-seq-sqlite
fix: close blob seq sqlite migration connection
2026-06-14 14:50:50 -03:00
Igor Lins e Silva 9f16295f77 Merge pull request #1721 from thismilktea/fix/mcp-add-drawer-idempotency-fail-closed
fix(mcp): fail closed when add_drawer idempotency pre-check fails
2026-06-14 14:45:23 -03:00
Igor Lins e Silva 8e53959bb9 Merge pull request #1802 from MemPalace/codex/develop-341-ci-polish
test: stabilize release validation on develop
2026-06-14 14:44:59 -03:00
Igor Lins e Silva d860a008a1 fix: close blob seq sqlite migration connection 2026-06-14 13:12:00 -03:00
Igor Lins e Silva 5d5397bf52 fix(palace): clean source mine locks safely 2026-06-14 12:50:32 -03:00
Igor Lins e Silva f89bc08876 test: stabilize release validation on develop 2026-06-14 12:25:53 -03:00
Igor Lins e Silva 97ba05cfe3 fix(mcp): avoid Chroma open when cached DB disappears 2026-06-14 12:12:59 -03:00
Igor Lins e Silva 422edc2dda Merge pull request #1717 from Hexecu/fix/diary-write-schema-anyof
fix(mcp): drop top-level anyOf from diary_write schema
2026-06-14 11:18:10 -03:00
Igor Lins e Silva 0a2937d766 Merge pull request #1582 from mvalentsev/fix/1580-neighbor-scope-parent-drawer-id
fix(searcher): scope neighbor expansion by parent_drawer_id (#1580)
2026-06-14 11:14:31 -03:00
Igor Lins e Silva 7e4bd816d1 Merge pull request #1780 from ggettert/fix/hallway-file-palace-scoped
fix(hallways): scope hallway-file path to MempalaceConfig.palace_path (#1778)
2026-06-14 11:12:14 -03:00
Igor Lins e Silva 75eb86a93a Merge pull request #1781 from fatkobra/fix/1776-id-recipe-v3-clean
fix(ids): use length-prefixed recipe v3
2026-06-14 11:12:01 -03:00
Igor Lins e Silva 80c37602c0 Merge pull request #1790 from fatkobra/fix/1772-shell-hooks-delegate-python
fix(hooks): delegate shell hooks to Python runner
2026-06-14 11:11:09 -03:00
Igor Lins e Silva b286701535 Merge pull request #1777 from mvalentsev/fix/1770-embeddinggemma-chunked-batching
fix(embedding): chunk EmbeddinggemmaONNX batches to bound ONNX memory (#1770)
2026-06-14 11:10:54 -03:00
Igor Lins e Silva c50434594c Merge pull request #1749 from mvalentsev/fix/repair-legacy-fts5-postrun
fix(repair): run post-rebuild FTS5 cleanup on legacy cmd_repair path (#1747)
2026-06-14 11:10:41 -03:00
Igor Lins e Silva 528d711f95 Merge pull request #1782 from fatkobra/fix/1763-mcp-logical-chunked-drawers
fix(mcp): treat chunked drawers as logical drawers
2026-06-14 11:10:32 -03:00
Igor Lins e Silva 4fbdaf2c32 Merge pull request #1793 from mvalentsev/fix/1774-backend-connect-races
fix(backends): serialize first connect in sqlite_exact and pgvector (#1774, #1775)
2026-06-14 11:10:23 -03:00
Tom BoucherandClaude Opus 4.8 c45d069f59 fix(tests): run fact_checker __main__ via subprocess to clear runpy warning
`tests/test_fact_checker.py` imports symbols from `mempalace.fact_checker` at
module top (putting it in sys.modules), then `TestCLI.test_exits_nonzero_when_
issues_found` re-executed the same module as __main__ via
`runpy.run_module("mempalace.fact_checker", run_name="__main__")`. runpy warns
because it re-runs an already-imported module against a half-initialized state:

  RuntimeWarning: 'mempalace.fact_checker' found in sys.modules after import of
  package 'mempalace', but prior to execution of 'mempalace.fact_checker'

Run the CLI in a fresh process via `subprocess.run([sys.executable, "-m",
"mempalace.fact_checker", ...])` instead — no sys.modules collision, and it
exercises the real `python -m` entry point. Assertions are preserved
(SystemExit code 1 → returncode 1; captured stdout substring → result.stdout).
The child's entity registry (`~/.mempalace/known_entities.json`, resolved via
expanduser at import) is redirected by overriding both HOME and USERPROFILE in
the subprocess env so it works on POSIX and Windows.

Verified: `pytest tests/test_fact_checker.py -W error::RuntimeWarning` passes
(26) with the warning promoted to error — proving it no longer fires.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-13 11:11:22 -04:00
fatkobra 950713eef6 fix(hooks): typo regression addressed 2026-06-13 06:37:26 +00:00
fatkobra 41e5c20191 fix(hooks): preserve fail-loud parse diagnostics 2026-06-12 16:58:20 +00:00
mvalentsevandjphein f6b6a69197 fix(backends): serialize first connect in sqlite_exact and pgvector (#1774, #1775)
Co-Authored-By: jphein <19301265+jphein@users.noreply.github.com>
2026-06-12 21:55:36 +05:00
fatkobra 0b00d78c98 fix(hooks): normalize Windows transcript paths in shell hooks 2026-06-12 15:25:36 +00:00
dependabot[bot] 2b457a2366 chore(deps): bump docker/login-action from 3 to 4
Bumps [docker/login-action](https://github.com/docker/login-action) from 3 to 4.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](https://github.com/docker/login-action/compare/v3...v4)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-version: '4'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-06-12 07:53:38 +00:00
dependabot[bot] 38de37adbf chore(deps): bump docker/build-push-action from 6 to 7
Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 6 to 7.
- [Release notes](https://github.com/docker/build-push-action/releases)
- [Commits](https://github.com/docker/build-push-action/compare/v6...v7)

---
updated-dependencies:
- dependency-name: docker/build-push-action
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-06-12 07:53:35 +00:00
dependabot[bot] 13d6c3e7a1 chore(deps): bump docker/metadata-action from 5 to 6
Bumps [docker/metadata-action](https://github.com/docker/metadata-action) from 5 to 6.
- [Release notes](https://github.com/docker/metadata-action/releases)
- [Commits](https://github.com/docker/metadata-action/compare/v5...v6)

---
updated-dependencies:
- dependency-name: docker/metadata-action
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-06-12 07:53:32 +00:00
fatkobra 5a8f0258d6 fix(ids): simplify v3 length-prefixed hashing 2026-06-12 07:39:57 +00:00
fatkobra 761ebf78bb fix(mcp): avoid mutating drawer metadata 2026-06-12 07:37:04 +00:00
Igor Lins e Silva 7e45720dc8 Merge pull request #1632 from undeadindustries/feat/cursor-hooks-support
feat: add Cursor IDE support (hooks, plugin, skill, docs, tests)
2026-06-11 19:29:29 -03:00
fatkobra 0fc3a238d7 fix(mcp): treat chunked drawers as logical drawers 2026-06-11 20:22:41 +00:00
fatkobra 5024e90d68 fix(ids): use length-prefixed recipe v3 2026-06-11 19:41:36 +00:00
Grace Gettert f5ee33b216 fixup(hallways): address gemini-code-assist review on PR #1780
Two catches on tests/test_hallways_palace_scoped.py
TestMultiPalaceIsolation.test_save_then_load_under_different_palace_returns_empty:

1. Stale comment referencing the removed _HALLWAY_FILE back-compat shim
   (deleted in the prior fixup commit). Removed.
2. _legacy_hallway_file was not monkey-patched, so the test isolation gap
   let _load_hallways check the host's real ~/.mempalace/hallways.json
   when evaluating the legacy-warning branch. Now patched to a tmp_path
   sibling, matching the helper pattern used in test_palace_graph_tunnels.
2026-06-11 19:18:54 +00:00
Grace Gettert 4fd12318df fixup(hallways): drop _HALLWAY_FILE back-compat shim, migrate existing tests to resolver
Replaces the back-compat shim in _load_hallways/_save_hallways (which
honored direct monkey-patches of the _HALLWAY_FILE module constant) with
a clean single-source-of-truth resolver, matching the palace_graph
tunnel-file migration in 3.3.6.

The three existing test sites (tests/test_hallways.py,
tests/test_hallways_pagination.py, tests/test_mcp_server.py) now
monkey-patch _get_hallway_file and _legacy_hallway_file directly,
exactly mirroring the helper in tests/test_palace_graph_tunnels.py.

Production code now has one branch through the path resolution instead
of two. No behavior change. 269/269 hallway + tunnel + mcp-server
tests pass on Python 3.11 and 3.12, ruff clean.
2026-06-11 19:11:52 +00:00
Grace Gettert bf71ae7168 fix(hallways): scope hallway-file path to MempalaceConfig.palace_path (#1778)
Pre-3.4 the hallway store was hardcoded at ~/.mempalace/hallways.json
regardless of the configured palace_path, so two palaces on one host
silently shared one file. Mining into palace-A leaked records into
palace-B's hallway code paths.

Apply the 3.3.6 tunnel-file migration pattern:

  * MempalaceConfig.hallway_file resolves to <dirname(palace_path)>/hallways.json
  * hallways._get_hallway_file(config) reads through MempalaceConfig
  * hallways._legacy_hallway_file() exposes the pre-migration hardcoded path
    for one-time orphan detection; _load_hallways logs a one-line warning
    when the legacy file exists but the configured one doesn't, matching
    palace_graph._load_tunnels behavior. No auto-migration — silent merging
    risks clobbering newer data.

Atomic-write + 0600 semantics unchanged. Module-level _HALLWAY_FILE
constant kept and honored when monkey-patched directly, so the three
existing test sites that patch it (test_hallways.py, test_hallways_pagination.py,
test_mcp_server.py) keep working without modification.

New coverage in tests/test_hallways_palace_scoped.py mirrors the analogous
tunnel tests: resolver default + custom palace_path + env-var redirect,
orphaned-legacy warning + no-warning when paths match, and an end-to-end
multi-palace isolation regression guard.

Closes #1778
2026-06-11 19:04:36 +00:00
mvalentsev 4187521ba4 fix(embedding): lock EF cache and lazy load, guard inputs (#1770)
Two threads sharing a cold EmbeddinggemmaONNX via _EF_CACHE could each
build a full model session, and two factory callers could each keep a
private instance. The load is now double-check locked with the session
published last, and the factory cache has an atomic check-then-construct
behind a lock-free fast path. __call__ wraps a bare string, returns []
for None and empty input before the lazy download, and its annotation
matches the accepted types.
2026-06-11 23:01:48 +05:00
mvalentsevandmojie5 07f8789514 fix(embedding): chunk EmbeddinggemmaONNX batches to bound ONNX memory (#1770)
One session.run over a repair-scale batch (5000 docs) allocates
attention buffers far beyond available RAM and the kernel kills the
process. Mirror chromadb's ONNXMiniLM_L6_V2 and embed in sub-batches
of 32; per-chunk padding also stops one long doc inflating the whole
batch.

Co-Authored-By: mojie5 <262519016+mojie5@users.noreply.github.com>
2026-06-11 21:08:49 +05:00
Igor Lins e Silva 8e19e815cc merge: resolve recall-protocol conflict with develop 2026-06-11 05:07:43 -03:00
Igor Lins e Silva f31db69e03 merge: resolve CHANGELOG conflict with develop 2026-06-11 04:28:42 -03:00
Igor Lins e Silva 2ec4baec3f Merge pull request #1771 from undeadindustries/feat/antigravity-support
feat(antigravity): add recall skill, optional rule, shared protocol
2026-06-11 04:26:51 -03:00
undeadindustriesandCursor 60a4610c6b fix(antigravity): sync recall skill protocol with canonical shared source
Steps 2-6, unhappy paths, and anti-patterns now match
integrations/shared/recall-protocol.md verbatim. Step 1 remains
Antigravity-specific (PreInvocation injectSteps injection). Flagged
by gemini-code-assist on PR #1771.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-11 13:19:24 +10:00
undeadindustriesandCursor fc89a1b67e feat(antigravity): add recall skill, optional rule, shared protocol
Adds the recall layer to the Antigravity plugin, mirroring the three-
layer wiring from feat/cursor-hooks-support, adapted for Antigravity's
native plugin and hook surfaces.

The wake hook (PreInvocation, invocationNum==1) was already the eager
layer — it injects verbatim palace content via injectSteps[].
ephemeralMessage before the first model call, which is strictly better
than Cursor's static directive. This commit adds the on-demand layers:

- integrations/shared/recall-protocol.md: single canonical source of
  truth for "search before answering", shared across Cursor, Antigravity,
  Claude Code, Codex, OpenClaw. Ported from feat/cursor-hooks-support
  with the wake-up step de-coupled from any IDE-specific hook name.

- .antigravity-plugin/skills/mempalace-recall/SKILL.md: recall-only
  skill with Protocol, Tool selection, Unhappy paths, and Anti-patterns
  sections. Tailored to Antigravity: references injectSteps injection,
  not Cursor's additional_context; links to shared protocol via GitHub
  URL (relative file paths break in the installed copy at
  ~/.gemini/config/plugins/mempalace/).

- .antigravity-plugin/rules/mempalace-recall.md: plain .md recall rule
  (no .mdc extension, no YAML frontmatter — per Antigravity's plugin
  docs which specify rules/<name>.md). Body mirrors Cursor's rule;
  references the shared protocol via absolute GitHub URL.

- hooks/antigravity/install.sh: creates skills/mempalace-recall/ and
  rules/ in the install dir, copies both new files (cmp-gated,
  idempotent). Verified: first run writes both, second run is silent.

- .antigravity-plugin/README.md: updated layout tree + "Three recall
  layers" section documenting wake hook (eager), recall skill
  (on-demand), and optional rule.

- 11 new tests in tests/test_antigravity_plugin_manifest.py covering:
  shared protocol presence, recall skill structure (frontmatter,
  required sections, GitHub URL reference), rule format (plain .md, no
  frontmatter), and installer (mkdir + copy_file lines present).

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-11 13:09:40 +10:00
jpandClaude Fable 5 47bddb066e review(gemini): marker-race stub, created-list lock, exact-order + exactly-one-ran asserts
- Stub _write_marker on the 8 concurrent writer backends: upsert()
  rewrites the marker on every call with a plain open('w'), so backends
  sharing one local_path race on the same file (sharing violations on
  Windows) — a test-design artifact, not the contract under test
- Guard the fixture's created list with a lock for the threaded tests
- Assert exact distance-ordered ids in the query/filter arms
- Reindex race: exactly one 'ran' (index absent beforehand, so the
  advisory-lock winner must build)

Re-run live after changes: 15/15 pass (PG 16.10, pgvector 0.8.2).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 16:46:27 -07:00
jpandClaude Fable 5 374400a530 test(backends): live-substrate conformance module for pgvector
Mirrors the portable fake-client arms of test_pgvector_backend.py
against a real PostgreSQL+pgvector server and adds live-only arms the
in-memory fake cannot exercise: real <=> operator ground truth, JSONB
pushdown vs local-fallback equivalence, cross-namespace isolation on
real tables, 8-connection concurrent writers, and the advisory-lock
serialization of run_maintenance('reindex') under a 2-connection race.

Gated on MEMPALACE_PGVECTOR_LIVE_DSN (same pattern as the qdrant live
gate); skips cleanly when unset. First run: 15/15 pass on PostgreSQL
16.10 + pgvector 0.8.2 (+AGE 1.6.0 in the same server), psycopg 3.3.4.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 16:38:27 -07:00
Igor Lins e Silva f124bd2c84 Merge pull request #1633 from undeadindustries/feat/antigravity-support
feat: add Antigravity IDE support (hooks, plugin, skill, docs, tests)
2026-06-10 08:08:25 -03:00
Igor Lins e Silva df0e29ac8b fix(test): make hybrid search closet preview test robust on Windows 2026-06-10 08:07:09 -03:00
Igor Lins e Silva 692599aa94 merge: resolve CHANGELOG conflict with develop 2026-06-10 07:46:22 -03:00
Igor Lins e Silva ad67c1a121 Merge pull request #1737 from Sathvik-1007/fix/1621-update-drawer-room-casing
fix(mcp): preserve room/wing casing on case-only update
2026-06-10 07:24:07 -03:00
Igor Lins e Silva 4d60db05ba Merge pull request #1741 from ggettert/feat/mcp-list-hallways
feat(mcp): expose list_hallways and delete_hallway tools
2026-06-10 07:23:50 -03:00
Igor Lins e Silva 77eaa312e0 Merge pull request #1622 from mvalentsev/fix/1535-mine-limit-skips-already-mined
fix(miner): count only new work toward --limit, not already-mined skips (#1535)
2026-06-10 07:23:36 -03:00
Igor Lins e Silva 8c120bee34 Merge pull request #1692 from mvalentsev/feat/1662-mcp-mine-tool
feat(mcp): add mempalace_mine tool (#1662)
2026-06-10 07:22:44 -03:00
Igor Lins e Silva 140c380584 Merge pull request #1697 from margaretjgu/fix/backup-retention
fix(backups): add max_backups retention to bound backup disk usage
2026-06-10 07:22:07 -03:00
Igor Lins e Silva 7d7dc78107 Merge pull request #1723 from mvalentsev/fix/hnsw-dim-none-deletions
fix(backends): keep post-deletion dim-None HNSW segments instead of quarantining (#1710)
2026-06-10 07:12:00 -03:00
mvalentsevandnord- d057dee9c3 fix(repair): run post-rebuild FTS5 cleanup on legacy cmd_repair path (#1747)
A clean `mempalace repair --yes` (legacy path) finished without
_vacuum_and_rebuild_fts5: the bulk delete_collection + re-upsert cycle
leaves the FTS5 inverted index inconsistent, so the next repair aborts
at the sqlite integrity preflight. rebuild_index() got this cleanup
when #1517 was fixed; cmd_repair never did.

Extract the shared epilogue _post_rebuild_cleanup() (close chroma
handles, then VACUUM + rebuild FTS5) and call it from both full-rebuild
paths so they cannot drift apart again. Cleanup runs on the legacy
success path only; failure/restore paths are unchanged.

Closes #1747

Co-Authored-By: nord- <3777600+nord-@users.noreply.github.com>
2026-06-10 14:33:19 +05:00
undeadindustriesandCursor 24ad98df8c feat: add mempalace-recall skill and optional Cursor recall rule
Ports the OpenClaw "search before answering" protocol to the Cursor and
Claude plugin surfaces so the agent reads the palace before answering
about past work, people, projects, or prior decisions instead of
guessing from model memory.

- integrations/shared/recall-protocol.md: single source of truth for the
  recall protocol, referenced by the skill and the rule so they cannot
  drift.
- skills/mempalace-recall/SKILL.md: recall-only skill (the mempalace
  skill keeps setup/mine/status); cross-linked from the ops skill.
- rules/mempalace-recall.mdc: plugin recall rule, alwaysApply: false so
  it only fires on recall-relevant turns and never adds MCP latency to
  greenfield work.
- examples/cursor/rules/: opt-in copies for non-plugin users, including
  an aggressive alwaysApply: true variant documented with its latency
  tradeoff.
- .claude-plugin/skills/mempalace-recall/SKILL.md: Claude plugin parity.
- tests: assert the recall skill and rules/ discovery layout; the
  shipped rule must be alwaysApply: false.
- docs: .cursor-plugin/README.md and the cursor-hooks guide now describe
  the three layers of recall (hook + skill + rule).

The Antigravity plugin mirror lands as a follow-up on the antigravity
branch, where .antigravity-plugin/ exists.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-06-10 15:40:35 +10:00
Igor Lins e Silva d05bb11724 Merge pull request #1734 from MemPalace/feat/qdrant-embedder-identity
feat(backends): shared embedder-identity sidecar + qdrant identity (RFC 001)
2026-06-08 20:03:19 -03:00
Grace Gettert ef2261feb8 docs(mcp): document list_hallways and delete_hallway tools
CI's test_no_undocumented_tools enforces that every tool registered
in the TOOLS dict has a corresponding section in mcp-tools.md.
The two hallway tools from b866f41 were missing — adding them here.

Sections mirror the format of the existing list_tunnels and
delete_tunnel entries directly above.
2026-06-08 20:07:23 +00:00
Grace Gettert eab11969da feat(mcp): expose list_hallways and delete_hallway tools
Closes #1739.

Hallways shipped in 3.3.6 (#1558) with Python API entry points
`list_hallways(wing=None)` and `delete_hallway(...)` in
`mempalace/hallways.py`, but neither was registered as an MCP tool
in `mempalace/mcp_server.py`. Mining produces hallways visible in
the mine log ("Hallways: +N within-wing entity link(s)") but
they were not retrievable through MCP.

This change wires both functions into the MCP tool registry
mirroring the existing tunnel-tool pattern:

- `mempalace_list_hallways(wing: str | None = None)` wraps
  `hallways.list_hallways`. Optional `wing` filter goes through
  `_sanitize_optional_name` just like `tool_list_tunnels`, so
  invalid names surface a structured error instead of crashing.
- `mempalace_delete_hallway(hallway_id: str)` wraps
  `hallways.delete_hallway` and returns `{"deleted": bool}` so
  callers can distinguish a successful delete from a no-op.

Tests in `tests/test_mcp_server.py` cover:

- list returns all records without filter
- list filters correctly by wing
- list rejects invalid wing names with a structured error
- delete removes the targeted record and returns `{"deleted": True}`
- delete with an unknown id returns `{"deleted": False}`
- delete with missing/non-string id returns a structured error
- both tools are present in the public `TOOLS` registry

Pure pass-throughs of the existing Python API. No behavior change
in the underlying hallway storage, no schema change to the
`hallways.json` file format.
2026-06-08 19:43:25 +00:00
Sathvik-1007 d514787f95 fix(mcp): str-coerce metadata before case comparison
Addresses review: old_meta values could be non-string after
corruption or external writes. str() prevents AttributeError.
2026-06-08 21:12:34 +05:30
Sathvik-1007 676ee682ec fix(mcp): preserve room/wing casing on case-only update
LLM clients (Claude, Codex) often autocorrect acronyms in tool
args (ps5→PS5). When the only difference is casing, treat it as
a no-op to prevent silent room fragmentation.

Genuinely different room/wing values still apply normally.

Fixes #1621
2026-06-08 20:49:33 +05:30
Igor Lins e Silva ed77125ae3 feat(backends): shared embedder-identity sidecar + qdrant identity (RFC 001, #1730)
Completes the embedder-identity contract for the last backend (qdrant) and
unifies identity persistence across all local-path backends.

Identity is stored in a small per-palace sidecar (mempalace_embedder.json),
NOT in a backend's mismatch marker. The marker's presence signals "palace
initialized" (reads raise CollectionNotInitializedError when the marker exists
but the store doesn't), so recording identity at first empty open must not
create it — a sidecar is unguarded, so a brand-new palace records identity
immediately. This fixes a latent gap (caught in review) where pgvector/qdrant
palaces stayed permanently "unknown" because the marker isn't written until the
first real write.

- New mempalace/backends/_sidecar.py: shared read/write_embedder_sidecar with
  the isinstance robustness the review bots taught us; chroma, pgvector, and
  qdrant all use it (chroma's inline copy is removed, pgvector switches off the
  marker, qdrant adds it).
- QdrantCollection.get/set_embedder_identity delegate to the sidecar, so
  palace.get_collection enforces a model swap on a qdrant palace exactly like
  the other backends — no live server needed for the check.

Tests: sidecar roundtrip + brand-new-palace recording (creates sidecar without
a marker) + enforcement model-swap raise, for qdrant and pgvector, all
server-free; chroma identity tests unchanged. Full suite: 2495 passed, 82.52%.

Closes #1730. Refs #743, #1724.
2026-06-08 10:20:45 -03:00
Igor Lins e Silva 4ceb880d19 Merge pull request #1732 from MemPalace/feat/maintenance-hooks
feat(backends): observable maintenance hooks + pgvector indexed-build path (RFC 001)
2026-06-08 09:51:29 -03:00
Igor Lins e Silva fdbafe5420 feat(backends): observable maintenance hooks + pgvector indexed-build path (RFC 001, #1725)
Adds the maintenance contract RFC 001 specifies but #1679 deferred, and gives
pgvector an opt-in HNSW index path with concurrency-safe builds.

- base.py: MaintenanceResult (status ran/already_running/noop + free-form
  stats), UnsupportedMaintenanceKindError, BaseBackend.maintenance_kinds
  ClassVar (reserved: analyze/compact/reindex; a backend with no analogue MUST
  omit, not no-op), and BaseCollection.maintenance_state()/run_maintenance()
  defaults. EmbeddingCollection forwards both (BaseCollection methods shadow
  __getattr__).
- sqlite_exact: analyze (ANALYZE) + compact (VACUUM, autocommit + page stats);
  omits reindex (exact scan, no ANN index). maintenance_state reports row/page
  counts.
- pgvector: reindex builds the optional HNSW index, serialized by a
  session-level pg_advisory_lock so concurrent daemon writers learn
  "already_running" instead of each stacking an ACCESS EXCLUSIVE build (the
  production wedge). It is opt-in: the default exact `<=>` scan is the
  100%-recall path; an HNSW index trades exact recall for scale, so an operator
  invokes it deliberately. Also analyze; omits compact (autovacuum). Advertises
  supports_server_side_indexes. maintenance_state reports index presence.
- qdrant/chroma: empty maintenance_kinds (qdrant self-optimizes; chroma
  maintenance is the separate repair CLI) — the faithful "omit" default.

Tests: contract + sqlite (real, CI-runnable) + pgvector advisory-lock flow via
a fake client (ran/noop/already_running, no live Postgres). Full suite green:
2488 passed, 82.47% coverage.

Benchmark three-phase wiring is deferred — the existing benchmarks/ are
task-benchmarks, not backend-comparison harnesses, so there is nothing to wire
into yet.

Closes #1725. Refs #743.
2026-06-08 08:44:20 -03:00
Igor Lins e Silva 13de7a6743 Merge pull request #1731 from MemPalace/feat/embedder-identity
feat(backends): embedder-identity contract + three-state enforcement (RFC 001)
2026-06-08 08:21:58 -03:00
Igor Lins e Silva 212051da35 feat(backends): embedder-identity contract + three-state enforcement (RFC 001, #1724)
The explicit-embedding backends (pgvector, qdrant, sqlite_exact) embed through
EmbeddingCollection and persist nothing about which model produced their
vectors. Swapping two same-dimension models (minilm <-> embeddinggemma, both
384-d) silently corrupts retrieval with no error — the worst failure class for
a verbatim-recall system. This records the model name and refuses a swap on open.

- base.py: EmbedderIdentity dataclass, Embedder protocol,
  EmbedderIdentityUnknownWarning, and a three-state check_embedder_identity()
  helper (known_match / known_mismatch / unknown) raising
  EmbedderIdentityMismatchError / DimensionMismatchError. BaseCollection gains
  get/set/effective_embedder_identity hooks.
- embedding.py: current_model_name() (cheap) + get_embedder_identity() (probed
  dimension, cached per process).
- Persistence per backend: sqlite_exact meta table, pgvector marker JSON
  (preserved across rewrites), chroma sidecar JSON. EmbeddingCollection forwards
  the hooks explicitly (BaseCollection methods shadow __getattr__).
- palace.get_collection enforces at open: a model swap raises; a brand-new empty
  collection records the current model; a populated-but-unrecorded legacy palace
  warns and is resolved via the CLI. The check needs no model load.
- CLI: `mempalace palace set-embedder [--model NAME] [--force]` records/overrides
  identity without mutating global config or loading a foreign model.

Chroma additionally keeps its native embedding-function check. Qdrant identity
persistence is a fast-follow (no local metadata slot — needs a companion store).

Closes #1724. Refs #743.
2026-06-08 08:14:26 -03:00
Igor Lins e Silva 52144704e7 Merge pull request #1727 from MemPalace/feat/searcher-backend-neutral
feat(searcher): metric-aware distance→similarity conversion (RFC 001 §10)
2026-06-08 06:21:59 -03:00
Igor Lins e Silva ffa8ac142c feat(searcher): metric-aware distance→similarity conversion (RFC 001 §10, #1726)
The searcher hard-coded `max(0, 1 - distance)` for vector similarity, which is
correct only for cosine. A backend reporting L2 or inner-product distances — or
a legacy Chroma palace built without `hnsw:space=cosine` — was silently
mis-ranked: L2 distances routinely exceed 1.0 and floored every result to 0.

- backends/base.py: `BaseBackend.distance_metric` ClassVar (cosine default) and
  a `BaseCollection.distance_metric` property per RFC 001 §2.1.
- backends/chroma.py: `ChromaCollection.distance_metric` reads the real
  `hnsw:space`, so a legacy-L2 palace reports "l2" and ranks correctly.
- searcher.py: `_distance_to_similarity(distance, metric)` (cosine/l2/ip,
  monotonic + bounded) and `_metric_for_collection(col)`; `_hybrid_rank` and the
  CLI display are now metric-aware. Cosine output is unchanged.
- Same conversion applied to the MCP `similarity` field, layers.py L3 search,
  and the dedup threshold so reported similarities are consistent.

Cosine behavior is byte-for-byte preserved; only non-cosine metrics change.
Part B of #1726 (routing the chroma.sqlite3 BM25 fallback through
BaseCollection) is tracked separately. Refs #1726, #743.
2026-06-08 02:48:26 -03:00
mvalentsev 9a8c87acfe fix(backends): keep post-deletion dim-None HNSW segments instead of quarantining (#1710)
_missing_dimensionality_appears_recoverable required total_elements_added == len(id_to_label) before keeping a dim-None segment, but total_elements_added is a monotonic add-counter while id_to_label holds only the live elements. After any delete the counts diverge, so every post-deletion segment was quarantined; once the WAL is pruned that destroys the only copy of those vectors. Relax the check to >= and keep the bijection and size checks as the integrity backstop.
2026-06-08 01:27:52 +05:00
chenyuxuan 6c5e1be3db fix(mcp): fail closed when add_drawer idempotency pre-check fails 2026-06-07 17:32:25 +08:00
jsiu93 74453ee3ac fix: handle rootless Java subprojects 2026-06-07 16:01:57 +08:00
jsiu93 ae415b8620 fix: detect Java project manifests 2026-06-07 15:50:15 +08:00
Hexecu b1537be5b5 fix(mcp): drop top-level anyOf from diary_write schema
The mempalace_diary_write tool declared a top-level anyOf in its
input schema to require either entry or content. Anthropic's Messages
API rejects any tool schema with a top-level anyOf/oneOf/allOf and
returns a 400 for the entire tools array, so every MCP session failed
to start.

The entry/content constraint is already enforced at dispatch: content
is remapped to entry before the handler runs, and a missing value
returns -32602. Removing the combinator restores compatibility without
weakening validation.

Closes #1711
2026-06-07 00:45:48 +02:00
mvalentsev c4da6d500b fix(searcher): scope neighbor expansion by parent_drawer_id (#1580) 2026-06-06 18:46:17 +05:00
mvalentsev c9dc4c466d fix(miner): count only new work toward --limit, not already-mined skips (#1535) 2026-06-06 17:56:08 +05:00
mvalentsev 2a66bad497 feat(mcp): add mempalace_mine tool (#1662)
Expose mining as an MCP tool so clients that cannot shell out (Claude
Desktop, LM Studio, Aionui, Desktop Commander) can index projects,
conversations, or documents in-conversation, not only through the
`mempalace mine` CLI.

tool_mine is a synchronous wrapper over the existing miners (miner.mine,
convo_miner.mine_convos, format_miner.mine_formats) that cmd_mine already
calls, so it adds no new ingestion logic and no backend coupling. Miner
stdout is captured at the Python and file-descriptor level so it cannot
corrupt the JSON-RPC channel (#225); no Unix-only calls, so it works on
Windows. The miners keep the palace write lock, so a concurrent mine
returns a structured already-running error.
2026-06-06 17:54:38 +05:00
margaretjgu 9be2b97b6c fix(backups): add max_backups retention to bound backup disk usage
mempalace migrate (.pre-migrate.* full-palace copies) and mempalace repair
max-seq-id (chroma.sqlite3.max-seq-id-backup-* DB copies) each wrote a fresh,
full-size, timestamped backup every run and never deleted the old ones. On a
machine that mines or repairs on a schedule, those copies could silently
accumulate until they filled the disk.

Add a configurable max_backups setting (default 10; env MEMPALACE_MAX_BACKUPS
or config.json) and a shared prune_backups helper that trims the oldest copies
after each new backup is written. Pruning is keyed by filesystem mtime, scoped
strictly to each backup's own naming pattern so live data is never touched, and
best-effort so a deletion failure can never abort the migrate/repair that just
succeeded. Set max_backups to 0 to keep every backup.
2026-06-05 14:52:06 -04:00
sjhddh f5d2212577 style(tests): apply ruff 0.4.x format to test_normalize
Fixes lint CI: ruff format --check flagged blank-line and long-dict
wrapping in the Continue.dev parser tests.
2026-06-03 18:36:50 +02:00
undeadindustriesandCursor a329acf23c fix(antigravity): resolve interpreter from console-script shebang
`uv tool install mempalace` / `pipx install` place the mempalace
console scripts in an isolated environment whose interpreter is not
the system python3. mempal_resolve_python previously resolved
`command -v python3`, landing on a Python that cannot import
mempalace: the `-m mempalace --version` probe failed and mining
silently never fired (hit by a real user on PR #1633).

Resolution now derives the interpreter from the mempalace-mcp /
mempalace console-script shebang on PATH (the same script the MCP
server launches) before falling back to python3. It is pure shebang
parsing + stat — no Python subprocess at source time — so the hook
performance budget is preserved. An env-style `#!/usr/bin/env python`
shebang and a non-executable interpreter are both rejected and fall
through. MEMPAL_PYTHON remains the explicit override.

Adds 6 resolver regression tests, documents resolution + MEMPAL_PYTHON
in the guide and hooks README (fixing the stale `command -v mempalace`
note), and a CHANGELOG entry.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-31 10:41:09 +10:00
undeadindustriesandCursor c420a9f66c fix(cursor): address igorls review on PR #1632
Resolves the maintainer review on the Cursor IDE support PR. Cursor-only
scope; cross-IDE items (wing-naming convention, shared-file merge order)
are coordinated on the separate Antigravity branch.

followup_message default (the one "decide before merge" item):
- Keep the stop-hook followup ON by default. Cursor's transcript format
  is undocumented and mempalace/normalize.py has no Cursor parser, so the
  background `mempalace mine --mode convos` is best-effort only and does
  not yet yield clean verbatim drawers. The followup is therefore the
  load-bearing verbatim-capture path; defaulting it off would leave a
  default Cursor install capturing nothing.
- Add an opt-out (MEMPAL_CURSOR_SILENT=1, or MEMPAL_VERBOSE=false) for
  users who want the Claude-style "zero tokens in chat" behaviour. The
  hook still mines and keeps its counters/markers when silenced.
- Correct the misleading "background mine captures it" comments in the
  save and precompact hooks; update hooks/cursor/README.md and the guide.

Hygiene fixes:
- Drop the hardcoded "version" field from .cursor-plugin/plugin.json and
  marketplace.json (mempalace/version.py is the single source of truth);
  tests now assert the field stays absent.
- Remove the committed .cursor-plugin/{commands,skills} symlinks (they
  break on Windows clones with core.symlinks=false and were redundant
  with the real repo-root components that `source: "."` already serves);
  add a guard test that no symlinks exist under .cursor-plugin/.
- Document the preCompact synchronous-mine timeout tradeoff and that an
  incremental/append-only mine is recoverable if killed (no corruption).
- Add a Cursor-namespaced, daily-throttled TTL sweep (MEMPAL_STATE_TTL_DAYS,
  default 30) to lib/common.sh that GCs stale cursor_*.count/.pending only,
  after the kill-switch check; shared logs and antigravity_* are untouched.

Verification: full suite green (2424 passed, 3 skipped), ruff check +
format clean, bash -n clean on all cursor scripts. +30 Cursor tests
(followup opt-out, state GC, TTL validation, no-symlink/version guards).

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-30 11:13:51 +10:00
undeadindustriesandCursor df295bd743 fix(antigravity): atomic counter write, background --version probe, state-file GC
Addresses igorls' review on PR #1633 (antigravity branch only):

- Atomic counter write: add mempal_write_counter_atomic (same-dir temp +
  mv -f rename) and use it in the save hook, replacing the truncate-then
  -write printf that the comment falsely called "atomic". Concurrent
  readers now always see a complete value.
- Background the expensive probe: `mempalace --version` pays the full
  chromadb/onnx cold-start import (the mine subparser imports
  mempalace.miner before argparse handles --version), so running it in
  the foreground blew the <500ms save budget. Probe + mine + pending
  -marker cleanup now run in one detached subshell; the foreground
  returns immediately. This also retires the kill -0 watcher (the prior
  gemini fix) since cleanup is now sequential within the mine's own
  shell, removing the sibling-PID hazard entirely.
- State-file GC: add mempal_state_ttl_days (default 30, env
  MEMPAL_STATE_TTL_DAYS) and mempal_gc_stale_state, a daily-throttled
  sweep (antigravity_last_sweep marker) that removes stale
  antigravity_save_count_*, antigravity_pending_*, and
  antigravity_woke_* artifacts. Called after the kill-switch check so a
  disabled hook touches nothing; specific name globs leave shared logs
  untouched.

Tests: atomic-counter behavior + no-temp-leftover, single-subshell
structure (no wait/kill -0/MINE_PID), backgrounded-probe timing (3s
stub returns in <2s), async log polling for the missing-module path,
GC sweep/throttle/TTL-validation, and GC gated by the kill switch.
Also hardens the wake-missing test to pin MEMPAL_PYTHON so a shell
-exported interpreter can't defeat the simulation.

bash 3.2.57 safe, fail-open on every path, {}-only Stop output.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-30 10:18:46 +10:00
undeadindustriesandCursor b06cf2452e fix(antigravity): address gemini-code-assist review on PR #1633
Five fixes for issues called out by the gemini-code-assist[bot] review.
Each gets a regression test that locks in the correction.

1. CRITICAL: marker-cleanup watcher used POSIX `wait` on a sibling pid
   (save hook). bash `wait` only works on direct children of the
   calling shell — the `( wait $MINE_PID ... ) &` subshell runs as a
   sibling of MINE_PID, so wait fails immediately and the pending
   marker is deleted within milliseconds, defeating the concurrency
   guard. Replace with `while kill -0 $MINE_PID; do sleep 1; done`,
   which queries pid existence regardless of parent-child relationship.
   Test: test_save_hook_marker_watcher_uses_kill_polling.

2. Bare `mempalace` console-script invocation in the save hook fails
   when the venv's bin/ is not on the hook's PATH (e.g. uv tool
   install in some configurations, manually managed virtualenvs).
   Switch to `"$MEMPAL_PYTHON_BIN" -m mempalace mine ...` so the
   resolved interpreter runs the package directly via
   mempalace/__main__.py. Tests:
   test_save_hook_uses_python_module_invocation,
   test_save_hook_missing_mempalace_python_module_does_not_crash.

3. Same issue in the wake hook's inner Python helper. Switch
   `['mempalace', 'wake-up', ...]` to `[sys.executable, '-m',
   'mempalace', 'wake-up', ...]` — sys.executable is the same
   interpreter that resolved MEMPAL_PYTHON in lib/common.sh.
   Test: test_wake_hook_uses_sys_executable_module_invocation.

4. The Python parser in lib/common.sh wrapped `json.load` in
   `try/except` and silently fell back to `data = {}`. The script
   then printed the `__MEMPAL_PARSE_OK__` sentinel even on parse
   failure, so the bash sentinel-check on the caller side
   (`[ "$_marker" != "__MEMPAL_PARSE_OK__" ]`) never triggered the
   defense-in-depth `input parse failed` branch. Remove the
   try/except so the exception propagates, Python exits non-zero,
   and the sentinel is omitted on bad JSON. The traceback still
   lands in antigravity_last_python_err.log for debugging.
   Test: test_common_sh_parser_omits_sentinel_on_malformed_json.

5. `mempal_save_interval()` failed to strip leading zeros from
   MEMPAL_SAVE_INTERVAL. Values like "08" or "09" then crashed the
   modulo step `$((COUNT % INTERVAL))` because bash arithmetic
   parses tokens starting with `0` as octal, and 8/9 are not valid
   octal digits ("value too great for base"). Strip leading zeros
   while preserving the literal "0" (which is then floored to 15).
   Test: test_save_hook_handles_leading_zero_save_interval (4 cases).

Plus one cosmetic fix in install.sh: removed a no-op `(cd "$OLDPWD"
2>/dev/null || cd .) >/dev/null 2>&1` line in mempal_absolutize().
The subshell cd doesn't affect the parent shell, and the installer
never cd's in the main shell anyway, so $PWD is already correct.

Verification:
* 9 new regression tests, all 65 antigravity tests pass
* full repo: 2323 passed (was 2314), 3 skipped, 1 unrelated warning
* ruff check + ruff format --check both clean across 139 files
* bash -n clean on all four shell files
* clean reinstall to ~/.gemini/config/plugins/mempalace/ succeeds
* idempotent re-run produces zero file writes (cmp-gated)
* both hooks return {} exit 0 with synthetic camelCase stdin

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-27 19:49:19 +10:00
undeadindustries bf156fb010 feat: add Antigravity IDE support (plugin, MCP, skill, hooks, docs, tests)
Adds first-class integration with Google's Antigravity IDE
(https://antigravity.google/) as a third sibling to the existing
Claude Code and Codex hook integrations. Strictly additive — no
existing files in main are restructured.

What ships
----------

* `.antigravity-plugin/` — verified-minimal plugin package:
  * `plugin.json` with `{"name": "mempalace"}` (no fabricated fields)
  * `mcp_config.json` registering the `mempalace-mcp` stdio server
  * `hooks.json.tmpl` templated with `__PLUGIN_DIR__` substitution
  * `skills/mempalace/SKILL.md` (real file — no symlinks)
* `hooks/antigravity/`:
  * `lib/common.sh` — shared bash 3.2.57-compatible helpers with
    sentinel-guarded camelCase JSON parser, antigravity_*-namespaced
    state files, every existing kill switch, `MEMPAL_SAVE_INTERVAL >= 1`
    floor (no /0), and fail-open emitters
  * `mempal_save_hook_antigravity.sh` — Stop event handler:
    increments per-conversation counter, defers when fullyIdle=False
    or terminationReason=error, validates transcriptPath against
    `..` traversal, spawns `mempalace mine --mode convos` in a
    detached subprocess with a per-conversation pending marker,
    ALWAYS emits `{}` (never `{"decision":"continue"}` — that would
    force an infinite agent loop)
  * `mempal_wake_hook_antigravity.sh` — PreInvocation handler gated
    to invocationNum==1 with an atomic mkdir loop guard, runs
    `mempalace wake-up` with a 500ms hard timeout, emits verbatim
    output as `{"injectSteps":[{"ephemeralMessage":"..."}]}` or
    `{}` on any failure
  * `install.sh` — idempotent installer with cmp-gated copies,
    `__PLUGIN_DIR__` substitution, relative path absolutization,
    `--dry-run`, and basename-guarded `--uninstall` (refuses to
    wipe a directory whose basename isn't `mempalace`)
  * `INVESTIGATION.md` — verbatim quotes + URLs + dates from the
    five official Antigravity doc pages, recording every surface
    shipped and every surface deliberately omitted
    (PreCompact equivalent, slash-commands, rules/, plugin
    permissions field — the latter is third-party fabrication)
  * `STDIN_SHAPE.md` — exact stdin/stdout contract per event with
    worked examples
  * `README.md` — local hook docs + troubleshooting
* `examples/antigravity/{hooks.json,mcp_config.json,README.md}` —
  standalone configs for users who don't want the full installer
* `website/guide/antigravity.md` + sidebar entry — VitePress guide
* Updates to `README.md`, `CHANGELOG.md` (Unreleased), `hooks/README.md`

Tests (56 new, all passing)
---------------------------

* `tests/test_antigravity_plugin_manifest.py` (11 tests) — schema
  contract on the in-repo `.antigravity-plugin/` directory, including
  guards against re-introducing the fabricated `permissions` field
  and against any symlink leak.
* `tests/test_antigravity_hooks_shell.py` (31 tests) — invokes the
  bash hooks via subprocess with synthetic camelCase stdin, asserts
  `{}` on every failure path, kill-switch coverage (env vars +
  config.json + palace nuke), divide-by-zero floor, transcript
  traversal rejection, namespacing, wing inference, and the hard
  refusal to ever emit `decision=continue` from the Stop hook.
* `tests/test_antigravity_hooks_install.py` (14 tests) — `--dry-run`
  side-effect-free, real install layout, executable bits preserved,
  byte-identical idempotent re-runs (md5 + filecmp), basename-match
  uninstall safety, refusal when plugin.json is missing or names a
  different plugin, relative path absolutization. Skipped on Windows.

Verification
------------

* `uv run pytest tests/ --ignore=tests/benchmarks -v` → 2314 passed,
  3 skipped (Windows), 1 unrelated warning
* `uv run ruff check .` → all checks passed
* `uv run ruff format --check .` → 139 files already formatted
* `bash -n` clean on common.sh, both hook scripts, install.sh
* Local install at `~/.gemini/config/plugins/mempalace/` verified end-
  to-end: layout correct, paths absolutized in hooks.json, both hooks
  fire with realistic camelCase JSON in <1s, wing inference picks
  `wing_mempalace` from workspacePaths[0], state files all
  `antigravity_*`-namespaced, second `install.sh` run produces
  byte-identical output (md5 snapshots match), uninstall removes
  only the mempalace plugin and leaves all 6 sibling Google plugins
  untouched.

Constraints honoured
--------------------

bash 3.2.57 (no mapfile / readarray / declare -A / `${var^^}`),
verbatim guarantee on all wake injections, hooks <500ms / startup
injection <100ms target (kill-switch path returns in <1.5s in CI),
zero new runtime dependencies, no telemetry, no external API,
strictly additive (existing Claude/Codex hooks unchanged).

Refs: hooks/antigravity/INVESTIGATION.md for the full audit.
2026-05-27 19:19:17 +10:00
undeadindustriesandCursor df5db57e79 fix(cursor): address gemini-code-assist review on PR #1632
Five fixes from the Gemini Code Assist review on
https://github.com/MemPalace/mempalace/pull/1632 — three real bugs,
two cleanups, all consistent with the bash-3.2-compatibility
contract documented in the original commit.

Bug fixes (high)
----------------

1. hooks/cursor/lib/common.sh — config.json kill-switch check used
   a `python3 - <<'PYEOF' ... PYEOF` heredoc inside a `$(...)`
   command substitution. The heredoc body contains parens which
   trips the macOS bash 3.2.57 parser bug. Replaced with a
   `python -c '...'` call passing the config path as argv[1]. Matches
   the pattern already used in mempal_parse_stdin in the same file.

2. hooks/cursor/install.sh — a relative `--install-dir` was written
   verbatim into hooks.json. Cursor invokes hook commands from its
   own working directory (typically the project root), so a relative
   command path would silently fail to launch the hook. Now resolved
   to an absolute path against `$PWD` before being baked in.

3. hooks/cursor/mempal_save_hook_cursor.sh — `MEMPAL_SAVE_INTERVAL=0`
   would crash bash on `$((NEXT % 0))` (division by zero). Extended
   the existing sanitiser case to coerce 0 to the default interval
   alongside empty / non-numeric values.

Cleanups (medium)
-----------------

4. hooks/cursor/install.sh — the EMPTY_CHECK_PY temp file is now
   inlined as `python -c '...'`. Removes a small leak window
   (tmpfile would linger if the script were interrupted between
   mktemp and rm -f) and shortens the script.

5. hooks/cursor/install.sh — `mktemp -t prefix` has subtly different
   semantics on BSD (macOS) vs GNU mktemp. Switched to the
   portable absolute-template form `mktemp "${TMPDIR:-/tmp}/...XXXXXX"`
   which behaves identically on both.

Regression tests
----------------

- tests/test_cursor_hooks_shell.py
    test_save_interval_zero_is_coerced_to_default — guards fix #3.
- tests/test_cursor_hooks_install.py — new TestInstallDirAbsolutePath
  class:
    test_relative_install_dir_is_absolutized_in_hooks_json — guards
        fix #2 against regression.
    test_absolute_install_dir_is_preserved_verbatim — guards that
        the relative-to-absolute resolution does not mangle paths
        that were already absolute.

Verification
------------

- bash -n on all three edited scripts: clean.
- uv run pytest tests/test_cursor_hooks_*.py tests/test_cursor_plugin_manifest.py: 132 passed (was 129; +3 regression tests).
- uv run pytest tests/ --ignore=tests/benchmarks: 2399 passed,
  3 skipped (pre-existing).
- uv run ruff check . / ruff format --check .: clean.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-27 14:17:36 +10:00
undeadindustriesandCursor 071fa015e0 feat: add Cursor IDE support (hooks, plugin, skill, docs, tests)
Adds first-class Cursor IDE integration alongside the existing Claude
Code and Codex hook flows, so Cursor users get the same automatic
diary saves, pre-compaction transcript capture, and session-start
memory recall — without changing any default behaviour for existing
users.

What's included
---------------

Cursor hook scripts (hooks/cursor/):
  - mempal_save_hook_cursor.sh       — Stop event, counter +
    loop_count guard, pending-save marker consumption, background
    mempalace mine, followup_message emission.
  - mempal_precompact_hook_cursor.sh — synchronous mine before
    compaction, drops a pending_save marker, returns user_message.
  - mempal_wake_hook_cursor.sh       — sessionStart event,
    wing-scoped recall guidance via additional_context.
  - lib/common.sh                    — shared parsing + state helpers
    (bash 3.2 safe, no heredoc-in-subshell traps).
  - install.sh                       — idempotent installer with
    --scope, --variant, --dry-run, --uninstall. Recognises existing
    entries by basename so re-installs across paths work.
  - STDIN_SHAPE.md, README.md        — payload schemas + quick
    reference.

Cursor plugin (.cursor-plugin/ + repo-root components):
  - plugin.json, marketplace.json, README.md.
  - skills/mempalace/SKILL.md  — model-invocable skill mirroring the
    Claude plugin's skill surface.
  - commands/mempalace-{help,init,mine,search,status}.md  — slash
    commands for marketplace-published installs (filename = slug).
  - mcp.json                   — auto-registers the mempalace MCP
    server, wrapped under the documented mcpServers key.

Examples + docs:
  - examples/cursor/hooks.json, hooks.minimal.json + README.
  - website/guide/cursor-hooks.md + sidebar entry.
  - README.md and CHANGELOG.md updates.

Tests (129 new, all green):
  - tests/test_cursor_hooks_shell.py     — 75 behavioural tests for
    the three hook scripts: kill switches, input parsing, counter
    logic, loop prevention, pending markers, wing inference, logging.
  - tests/test_cursor_hooks_install.py   — 19 contract tests for the
    installer: dry-run, idempotent merge, basename-matched uninstall,
    refusal to overwrite malformed JSON.
  - tests/test_cursor_plugin_manifest.py — 35 contract tests for the
    plugin: manifest validity, version sync with mempalace.version,
    mcp.json shape, skill/command frontmatter, default-discovery
    layout invariants.

Design notes
------------

- Local-first and zero-API by default; hooks never call external
  services. Same privacy model as the existing Claude Code hooks.
- Fail-open: hook scripts deliberately do not use set -e so a broken
  hook can never block the user's conversation.
- Cursor preCompact cannot block + return a followup, so we
  synchronously mine the transcript and drop a pending_save marker
  that the next stop hook consumes — guarantees verbatim capture
  before context window compression.
- Cursor's default plugin discovery requires real commands/, skills/,
  and mcp.json at the plugin root (verified against the cached
  cloudflare plugin); .cursor-plugin/{commands,skills} are convenience
  symlinks back to those canonical locations.
- bash 3.2 compatibility throughout: avoids heredoc-in-command-
  substitution parser bugs; uses python -c for JSON parsing;
  basename-matched entry recognition in install.sh.
- All changes are additive. No existing files are removed, no
  existing hooks change behaviour, and no new runtime dependencies
  are introduced.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-27 14:05:40 +10:00
Tom Boucher 234343ef68 fix: preserve collection name on MCP search retry 2026-05-25 20:09:34 -04:00
sjhddh 25b918ee7e feat(normalize): add Continue.dev session parser
Add _try_continue_json() normalizer for Continue.dev AI assistant
sessions (~/.continue/sessions/*.json). Parses history array with
role/content pairs, handles tool calls, system messages, and metadata.

Closes #59 (partial — adds Continue.dev format support)

Includes comprehensive test coverage for valid sessions, edge cases,
malformed input, and unicode content.
2026-05-18 13:08:09 +00:00
FBISiri a9a2c35e65 feat: add Gemini CLI / AI Studio JSON session import support
Adds _try_gemini_json parser to normalize.py for three layouts:

  1. Gemini API contents format (~/.gemini/sessions/*.json):
     {"contents": [{"role": "user", "parts": [{"text": "..."}]}, ...]}
  2. Messages-wrapper variant:
     {"messages": [{"role": "user", ...}, {"role": "model", ...}]}
  3. Flat top-level list with role="model".

This complements the existing _try_gemini_jsonl parser (which handles
~/.gemini/tmp/<hash>/chats/session-*.jsonl with session_metadata
sentinel) — JSONL covers Gemini CLI runtime sessions, JSON covers
exported / Studio-saved transcripts.

## Review feedback addressed (PR #204)

bgauryy review:
- #1 Parser-precedence bug: _try_gemini_json runs *before*
  _try_claude_ai_json so the {"messages":[..., role=model, ...]}
  layout is no longer silently claimed by the Claude parser. The
  Gemini parser's has_model_role guard prevents false-positives
  against Claude / ChatGPT data.
- #2 Layout 2a coverage: TestGeminiJson.test_messages_wrapper_format
  + test_messages_wrapper_does_not_get_claimed_by_claude pin the
  fix in place.
- #3 Test conflicts with current main: rebased onto develop;
  tests restructured into TestGeminiJson class.
- #4 tempfile/os.unlink → pytest tmp_path everywhere.
- #5 elif not text → else (the elif branch was dead).
- #6 Module docstring updated to mention Google AI Studio.

Tests: 9 new cases in TestGeminiJson covering all three layouts,
multi-part text joining, non-text part skipping, has_model_role
disambiguation, dispatch-chain regression for review #1.
2026-05-09 14:05:45 +08:00
adv3nt3 66b0c5066b feat: add Pi agent JSONL session normalizer
Add _try_pi_jsonl parser for Pi agent session files stored at
~/.config/pi/agent/sessions/{encoded-cwd}/{timestamp}_{uuid}.jsonl.

Uses type "message" entries with role "user"/"assistant". Skips
toolResult messages, model_change, thinking_level_change, and other
operational events. Requires session header (type "session" with
"version" key) to avoid false positives.

Format documented at github.com/badlogic/pi-mono session.md and
verified via Context7. Sample data provided by tunnckoCore in #59.

Refs: #59
2026-05-08 14:08:31 +02:00
git e039a675af feat(miner): add support for Swift and Kotlin file extensions
- Updated READABLE_EXTENSIONS in miner.py to include ".swift", ".kt", and ".kts".
- Added tests in test_miner.py to ensure scanning includes Swift and Kotlin files.
2026-05-06 00:26:34 +02:00
David WallaceandClaude Sonnet 4.6 6165aa2d17 feat(miner): add C# and .NET file extensions to READABLE_EXTENSIONS
Adds .cs, .csproj, .sln, .razor, and .cshtml so C#/.NET projects
are indexed by the project miner. .razor/.cshtml are analogous to
the already-supported .jsx/.tsx.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16 12:05:52 -04:00
154 changed files with 25458 additions and 652 deletions
+77
View File
@@ -0,0 +1,77 @@
# MemPalace — Antigravity plugin
In-repo packaging for the MemPalace integration with Google's [Antigravity IDE](https://antigravity.google/).
This directory is the source of truth for what gets installed at
`~/.gemini/config/plugins/mempalace/` when the user runs the installer.
## Layout
```
.antigravity-plugin/
├── plugin.json # marker manifest (verified minimal schema)
├── mcp_config.json # auto-registers the mempalace-mcp stdio server
├── hooks.json.tmpl # template — installer renders to hooks.json
├── skills/
│ ├── mempalace/
│ │ └── SKILL.md # ops skill: setup, mine, status, CLI delegation
│ └── mempalace-recall/
│ └── SKILL.md # recall-only skill: search-before-answer protocol
├── rules/
│ └── mempalace-recall.md # optional recall rule (complements the skill)
└── README.md # this file
```
The hook scripts themselves live at `hooks/antigravity/`. The installer
copies them into `<install-dir>/hooks/` and renders `hooks.json.tmpl`
into a `hooks.json` whose `command` paths point at the absolute install
location.
## Three recall layers
MemPalace can store everything, but it only helps if the agent actually
*reads* the palace before answering. Three layers wire that in, from
eager to on-demand:
1. **Wake hook** (`hooks/antigravity/mempal_wake_hook_antigravity.sh`,
`PreInvocation` event, gated to `invocationNum == 1`). On the first
model call of a conversation it runs `mempalace wake-up` and injects
the **actual palace content verbatim** via Antigravity's
`injectSteps[].ephemeralMessage` output. This is Antigravity's native
equivalent of Cursor's `sessionStart` `additional_context`, except it
delivers the memory itself rather than a directive to go fetch it.
2. **Recall skill** (`skills/mempalace-recall/SKILL.md`). The
search-before-answer protocol the agent follows when a turn is
recall-relevant — tool selection, unhappy paths, anti-patterns. It
covers recall only; the `mempalace` skill covers setup / mine /
status.
3. **Optional recall rule** (`rules/mempalace-recall.md`). A lightweight
markdown rule that nudges the agent to search before answering when
Antigravity's matcher decides the turn is recall-relevant. It is
deliberately recall-scoped (not an always-on global rule) so it never
adds latency to greenfield work, honouring MemPalace's "memory should
feel instant" budget.
All three point to the single canonical protocol in
[`integrations/shared/recall-protocol.md`](../integrations/shared/recall-protocol.md)
so the skill and rule never drift.
## Install
```bash
bash hooks/antigravity/install.sh
```
The installer is idempotent and the uninstaller matches by basename, so
re-runs and partial installs are safe.
See [website/guide/antigravity.md](../website/guide/antigravity.md) for
the full user-facing guide and [hooks/antigravity/README.md](../hooks/antigravity/README.md)
for the hooks-specific documentation.
## Verified surfaces
Every file in this directory maps to a surface verified against
[Google's Antigravity docs](https://antigravity.google/docs/). See
[hooks/antigravity/INVESTIGATION.md](../hooks/antigravity/INVESTIGATION.md)
for the full audit, including the surfaces deliberately not shipped.
+20
View File
@@ -0,0 +1,20 @@
{
"mempalace-save": {
"Stop": [
{
"type": "command",
"command": "__PLUGIN_DIR__/hooks/mempal_save_hook_antigravity.sh",
"timeout": 30
}
]
},
"mempalace-wake": {
"PreInvocation": [
{
"type": "command",
"command": "__PLUGIN_DIR__/hooks/mempal_wake_hook_antigravity.sh",
"timeout": 5
}
]
}
}
+7
View File
@@ -0,0 +1,7 @@
{
"mcpServers": {
"mempalace": {
"command": "mempalace-mcp"
}
}
}
+3
View File
@@ -0,0 +1,3 @@
{
"name": "mempalace"
}
@@ -0,0 +1,18 @@
# MemPalace recall
Before answering anything that may already be in the user's memory
palace — past work, prior decisions, a person, a project, or "what did
we do / decide / discuss last time?" — search the palace first:
1. Call `mempalace_search` with a short keyword query. Use
`mempalace_kg_query` for relational or time-bound facts.
2. Quote the drawer's **verbatim** text. Never summarize or paraphrase
stored content.
3. If results are empty, say so — do not invent an answer. If the MCP
server is unavailable, surface the error; do not fall back to guessing.
Skip recall for pure greenfield work with no memory relevance (renaming
a variable, fixing a typo). Recall is question-driven, not reflexive.
Full protocol: <https://github.com/MemPalace/mempalace/blob/main/integrations/shared/recall-protocol.md>. Deeper guidance:
the `mempalace-recall` skill.
@@ -0,0 +1,119 @@
---
name: mempalace-recall
description: "Recall protocol for MemPalace — search the palace before answering about past work, people, projects, or prior decisions. Apply when the user asks what was decided, what happened before, who someone is, what was discussed last time, or anything that may already be filed in their memory palace; or when mempalace-recall is invoked. Complements the mempalace setup skill and requires the mempalace-mcp server."
---
# MemPalace Recall
Search-before-answer protocol for MemPalace. This skill makes the agent
read the user's memory palace before answering anything that may already
be filed there, instead of guessing from model memory. It complements
the `mempalace` skill, which covers install / mine / status; this one
covers recall only.
## Step 0 — Verify MemPalace is available
Before relying on recall, confirm MemPalace is installed and reachable:
- Official release page: <https://github.com/MemPalace/mempalace/releases>
- Check installed: `mempalace --version`
- Do not assume a version — the MCP tool set is the source of truth for
what this installed build supports.
If the `mempalace_*` MCP tools are not available, tell the user the
server is not connected and point them at the `mempalace` skill to set
it up. Do not silently fall back to answering from model memory.
## Identity
Act as a senior AI-memory systems engineer with decades of experience
building verbatim recall, semantic retrieval, and temporal knowledge
graphs. Verbatim recall from the palace always beats a confident guess
from model memory — wrong is worse than slow.
## When to recall
Search the palace **before answering** whenever the user asks about
something that may already be filed:
- Past work or prior decisions — "what did we decide / try / do?"
- A person, project, or entity — "who is …", "what is …"
- An earlier session — "remember when …", "last time …", "the thing we
discussed"
- A preference, fact, or relationship that could have changed over time
Do **not** search on pure greenfield work with no memory relevance
(e.g. "rename this variable", "fix this typo"). Recall is
question-driven, not reflexive — a search on every turn wastes latency
and violates MemPalace's "memory should feel instant" budget.
## Protocol
1. On wake-up, the MemPalace PreInvocation hook injects verbatim palace
content via `injectSteps[].ephemeralMessage` on the first model call
of a conversation. If memory was injected, start from it before
searching further.
2. **Before responding** about people, projects, past events, or prior
decisions: call `mempalace_search` first. For relational or temporal
facts ("who reported to whom in March", "what was true then"), call
`mempalace_kg_query` instead or as well.
3. **If unsure** about a fact (name, age, relationship, preference): say
"let me check the palace" and query. Wrong is worse than slow.
4. **Return verbatim.** Quote the drawer's exact stored words. Never
summarize, paraphrase, or lossy-compress what the palace returns —
that is the whole point of the system.
5. **After a substantive session**, record continuity with
`mempalace_diary_write` (background hooks may already do this — do not
double-file).
6. **When a fact changes**, call `mempalace_kg_invalidate` on the old
fact, then `mempalace_kg_add` for the new one.
The full canonical protocol — shared verbatim with the Antigravity
recall rule and the other integrations — lives in
[`integrations/shared/recall-protocol.md`](https://github.com/MemPalace/mempalace/blob/main/integrations/shared/recall-protocol.md).
## Tool selection
| You need | Tool |
|---|---|
| Find any memory by meaning | `mempalace_search` (start here) |
| Relational / time-bound facts about an entity | `mempalace_kg_query` |
| The chronological story of an entity | `mempalace_kg_timeline` |
| Recent session continuity | `mempalace_diary_read` |
| Which wings / rooms exist (scope unknown) | `mempalace_list_wings`, `mempalace_list_rooms` |
| Record this session | `mempalace_diary_write` |
`mempalace_search` takes a short natural-language `query` (keywords or a
question — not a system prompt or pasted conversation) plus optional
`wing` / `room` filters and `limit` (default 5).
## Unhappy paths
- **Empty results.** Say the palace has nothing on this; do not invent an
answer to fill the gap. Offer to widen the search (drop the wing
filter) or to file the new information.
- **MCP unavailable / tool error.** Surface the error plainly and suggest
the user verify the server (`mempalace status`, or re-run the
installer `hooks/antigravity/install.sh`). Do not silently fall back
to guessing from model memory.
- **Stale or conflicting facts.** Prefer the knowledge graph's
time-valid answer; if a fact has changed, invalidate the old one and
add the new one rather than overwriting context silently.
## Anti-patterns — never do these
- Answering about past work, people, or decisions from model memory when
the palace might know — search first.
- Paraphrasing or summarizing stored content instead of quoting it
verbatim.
- Searching reflexively on every turn, including pure greenfield coding
with no memory relevance.
- Pasting the full conversation or a system prompt into the `query`
argument — keep queries short and keyword-driven.
## Official References
- MemPalace: <https://github.com/MemPalace/mempalace>
- MemPalace releases: <https://github.com/MemPalace/mempalace/releases>
- Antigravity documentation: <https://antigravity.google/docs>
- Agent Skills specification: <https://agentskills.io/specification>
@@ -0,0 +1,92 @@
---
name: mempalace
description: MemPalace — mine projects and conversations into a searchable memory palace. Use when the user asks about MemPalace, memory palace, mining memories, searching memories, palace setup, wings, rooms, or drawers; or when they want to recall past work that may already be filed in their palace.
---
# MemPalace
A searchable memory palace for AI — mine projects and conversations, then search them semantically. Verbatim storage, local-first, zero external API by default.
## Prerequisites
Ensure `mempalace` is installed:
```bash
mempalace --version
```
If not installed (uv recommended):
```bash
uv tool install mempalace # or: pip install mempalace
```
## Dynamic, version-correct instructions
MemPalace exposes operation-specific instructions through the CLI so this skill stays accurate as MemPalace evolves. To get instructions for any operation:
```bash
mempalace instructions <command>
```
Always prefer the CLI output over what is written here when the two disagree — the CLI is the single source of truth for the installed version.
## Common operations
These are the five operations users ask for most often. Each one wraps a single MemPalace CLI subcommand. The `mempalace instructions <name>` form returns the full, version-correct guidance.
### `help` — discover what MemPalace can do
```bash
mempalace instructions help
```
Use when the user is new, unsure what's possible, or asks "what can you do".
### `init` — first-run setup of the palace
```bash
mempalace instructions init
```
Use when the user has just installed MemPalace, no palace exists yet, or the user explicitly asks to set up / configure / re-initialize their palace.
### `mine` — ingest a project or conversation directory
```bash
mempalace instructions mine
```
Use when the user wants to fold a project's files into their palace, or to ingest exported conversation transcripts into the palace as searchable memory.
### `search` — find verbatim memories by semantic query
```bash
mempalace instructions search
```
Use when the user wants to recall something from the past, find a previous decision, or rediscover code/notes/conversations they already wrote.
### `status` — what's in the palace right now
```bash
mempalace instructions status
```
Use when the user asks "what's in my palace", "how big is my palace", or wants a summary of wings, rooms, and drawer counts.
## MCP tools (preferred over CLI)
Inside Antigravity, the MemPalace MCP server registers a rich set of tools. Use these instead of shelling out to the CLI for live operations (search, diary writes, drawer adds, knowledge graph queries, palace status). The MCP tools always reflect the current palace state without spawning a subprocess.
The MCP server is auto-registered when this plugin is installed at `~/.gemini/config/plugins/mempalace/`. If the server does not appear in Antigravity's MCP store, run `mempalace-mcp --version` to verify the binary is on PATH, then restart Antigravity.
## Design principles (verbatim from the project)
- **Verbatim always** — never summarize, paraphrase, or lossy-compress user data.
- **Local-first, zero external API by default** — extraction, embedding, and LLM-assisted refinement happen on the user's machine.
- **Privacy by architecture** — the system never calls out to external services for core operations.
- **Performance budgets** — hooks under 500ms; startup injection under 100ms.
- **Background everything** — filing, indexing, and timestamps happen via hooks in the background; zero tokens spent on bookkeeping in the chat window.
If a request would violate any of these principles, refuse and explain — even if it would be technically convenient.
+4 -3
View File
@@ -1,6 +1,6 @@
# MemPalace Claude Code Plugin
A Claude Code plugin that gives your AI a persistent memory system. Mine projects and conversations into a searchable palace backed by ChromaDB, with 19 MCP tools, auto-save hooks, and 5 guided skills.
A Claude Code plugin that gives your AI a persistent memory system. Mine projects and conversations into a searchable palace backed by ChromaDB, with 35 MCP tools, auto-save hooks, and 5 guided skills.
## Prerequisites
@@ -41,16 +41,17 @@ After installing the plugin, run the init command to complete setup (installs th
## Hooks
MemPalace registers two hooks that run automatically:
MemPalace registers three hooks that run automatically:
- **Stop** -- Saves conversation context every 15 messages.
- **SessionEnd** -- Runs one final save in the background on a clean exit, so short sessions that never hit the Stop interval or a compaction are still captured.
- **PreCompact** -- Preserves important memories before context compaction.
Set the `MEMPAL_DIR` environment variable to a directory path to automatically run `mempalace mine` on that directory during each save trigger.
## MCP Server
The plugin automatically configures a local MCP server with 19 tools for storing, searching, and managing memories. No manual MCP setup is required -- `/mempalace:init` handles everything.
The plugin automatically configures a local MCP server with 34 tools for storing, searching, and managing memories. No manual MCP setup is required -- `/mempalace:init` handles everything.
## Full Documentation
+11
View File
@@ -12,6 +12,17 @@
]
}
],
"SessionEnd": [
{
"hooks": [
{
"type": "command",
"command": "bash \"${CLAUDE_PLUGIN_ROOT}/hooks/mempal-session-end-hook.sh\"",
"timeout": 10
}
]
}
],
"PreCompact": [
{
"hooks": [
@@ -0,0 +1,43 @@
#!/bin/bash
# MemPalace SessionEnd Hook — thin wrapper calling the Python CLI.
#
# Claude Code documents a default SessionEnd hook timeout of 1.5s, and
# "timeouts set on plugin-provided hooks do not raise the budget"
# (https://code.claude.com/docs/en/hooks). A cold `mempalace` start alone
# exceeds 1.5s, so the final mine must NOT run in the foreground — it would be
# killed before it saved anything. Unlike the foreground Stop/PreCompact plugin
# wrappers, this one backgrounds the hook and returns immediately; the detached
# child finishes the save after the session has exited. All logic lives in
# mempalace.hooks_cli for cross-harness extensibility.
run_mempalace_hook() {
if command -v mempalace >/dev/null 2>&1; then
exec mempalace hook run "$@"
fi
MEMPAL_PYTHON_BIN="${MEMPAL_PYTHON:-}"
if [ -z "$MEMPAL_PYTHON_BIN" ] || [ ! -x "$MEMPAL_PYTHON_BIN" ]; then
MEMPAL_PYTHON_BIN="$(command -v python3 2>/dev/null || echo python3)"
fi
if "$MEMPAL_PYTHON_BIN" -c "import mempalace" >/dev/null 2>&1; then
exec "$MEMPAL_PYTHON_BIN" -m mempalace hook run "$@"
fi
if command -v python >/dev/null 2>&1 && python -c "import mempalace" >/dev/null 2>&1; then
exec python -m mempalace hook run "$@"
fi
echo "MemPalace hook error: could not find a runnable mempalace command or module" >&2
exit 1
}
# Capture stdin (the SessionEnd JSON) before backgrounding — the parent's
# stdin is gone once we return. Forward it to the detached worker, which runs
# the final mine on its own time and outlives this process.
payload="$(cat)"
(
printf '%s' "$payload" | run_mempalace_hook --hook session-end --harness "${MEMPALACE_HOOK_HARNESS:-claude-code}"
) >/dev/null 2>&1 </dev/null &
disown 2>/dev/null || true
# Return immediately so the harness never blocks on session exit.
printf '{}'
+2 -2
View File
@@ -8,8 +8,8 @@
{
"name": "mempalace",
"source": "./.claude-plugin",
"description": "AI memory system — mine projects and conversations into a searchable palace. 19 MCP tools, auto-save hooks, guided setup.",
"version": "3.4.0",
"description": "AI memory system — mine projects and conversations into a searchable palace. 35 MCP tools, auto-save hooks, guided setup.",
"version": "3.5.0",
"author": {
"name": "milla-jovovich"
}
+2 -2
View File
@@ -1,7 +1,7 @@
{
"name": "mempalace",
"version": "3.4.0",
"description": "Give your AI a memory — mine projects and conversations into a searchable palace. 19 MCP tools, auto-save hooks, and guided setup.",
"version": "3.5.0",
"description": "Give your AI a memory — mine projects and conversations into a searchable palace. 35 MCP tools, auto-save hooks, and guided setup.",
"author": {
"name": "milla-jovovich"
},
@@ -0,0 +1,67 @@
---
name: mempalace-recall
description: Recall protocol for MemPalace — search the palace before answering about past work, prior decisions, people, or projects. Use when the user asks what was decided, what happened before, who someone is, what was discussed last time, or anything that may already be filed in their memory palace.
allowed-tools: Bash
---
# MemPalace Recall
Search-before-answer protocol for MemPalace. Read the user's memory
palace before answering anything that may already be filed there,
instead of guessing from model memory. This complements the `mempalace`
skill (install / mine / status); this one covers recall only.
## Step 0 — Verify MemPalace is available
```bash
mempalace --version
```
If the `mempalace_*` MCP tools are not available, tell the user the
server is not connected and point them at the `mempalace` skill or
`/init`. Do not silently fall back to answering from model memory.
## When to recall
Search the palace **before answering** whenever the user asks about
something that may be filed:
- Past work or prior decisions — "what did we decide / try / do?"
- A person, project, or entity — "who is …", "what is …"
- An earlier session — "remember when …", "last time …"
- A preference, fact, or relationship that could have changed over time
Skip recall for pure greenfield work with no memory relevance (renaming
a variable, fixing a typo). Recall is question-driven, not reflexive.
## Protocol
1. Before responding about people / projects / past events / prior
decisions: call `mempalace_search` first. Use `mempalace_kg_query`
for relational or time-bound facts.
2. If unsure about a fact: say "let me check the palace" and query.
3. Return the drawer's **verbatim** text — never summarize or paraphrase
stored content.
4. After a substantive session, record continuity with
`mempalace_diary_write` (skip if a background hook already saved).
5. When a fact changes: `mempalace_kg_invalidate` the old fact, then
`mempalace_kg_add` the new one.
## Unhappy paths
- **Empty results** — say the palace has nothing on this; do not invent
an answer. Offer to widen the search or file the new information.
- **MCP error / server down** — surface the error, suggest `mempalace
status` or re-running `/init`; never fall back to guessing.
- **Palace index corrupt / compactor error** — if the server reports an
HNSW segment-writer error, a ChromaDB compaction failure, or stays
"Not connected" after a write, the index is out of sync with
`chroma.sqlite3` but the rows are intact. Tell the user to stop the
server and rebuild from SQLite (`mempalace repair --mode from-sqlite
--archive-existing --yes`), not re-mine, which drops MCP-added drawers
and diary entries (#1843). Do not repair in-process.
- **Conflicting facts** — trust the knowledge graph's time-valid answer;
invalidate-then-add rather than overwriting silently.
The canonical protocol, shared across all MemPalace integrations, lives
in `integrations/shared/recall-protocol.md`.
+1 -1
View File
@@ -1,6 +1,6 @@
# MemPalace - Codex CLI Plugin
Give your AI a persistent memory -- mine projects and conversations into a searchable palace backed by ChromaDB, with 19 MCP tools, auto-save hooks, and guided skills.
Give your AI a persistent memory -- mine projects and conversations into a searchable palace backed by ChromaDB, with 35 MCP tools, auto-save hooks, and guided skills.
## Prerequisites
+3 -3
View File
@@ -1,7 +1,7 @@
{
"name": "mempalace",
"version": "3.4.0",
"description": "Give your AI a memory — mine projects and conversations into a searchable palace. 19 MCP tools, auto-save hooks, and guided setup.",
"version": "3.5.0",
"description": "Give your AI a memory — mine projects and conversations into a searchable palace. 35 MCP tools, auto-save hooks, and guided setup.",
"author": {
"name": "milla-jovovich"
},
@@ -27,7 +27,7 @@
"interface": {
"displayName": "MemPalace",
"shortDescription": "AI memory system for Codex",
"longDescription": "Give your AI a persistent memory — mine projects and conversations into a searchable palace backed by ChromaDB, with 19 MCP tools, auto-save hooks, and guided skills.",
"longDescription": "Give your AI a persistent memory — mine projects and conversations into a searchable palace backed by ChromaDB, with 35 MCP tools, auto-save hooks, and guided skills.",
"developerName": "milla-jovovich",
"category": "Coding",
"capabilities": [
+144
View File
@@ -0,0 +1,144 @@
# MemPalace Cursor Plugin
A Cursor IDE plugin that gives your agent a persistent memory system. Auto-registers the `mempalace-mcp` server (35 MCP tools), ships 5 slash commands, two model-invocable skills (setup/mining/search and a recall protocol), and an optional recall rule.
> Hooks (auto-save + session-start memory recall) are shipped separately under `hooks/cursor/` so the plugin is safe to install in any Cursor workspace without touching the agent loop. See [Hooks](#hooks-optional) below.
## Prerequisites
- Python 3.9+
- Cursor 1.7+ (plugin manifest schema requires it)
## Installation
### Local clone (recommended while not in the marketplace yet)
Symlink (or copy) this repository into Cursor's local plugins folder:
```bash
ln -s /path/to/mempalace ~/.cursor/plugins/local/mempalace
```
Then in Cursor: <kbd>Cmd</kbd>-<kbd>Shift</kbd>-<kbd>P</kbd> → **Developer: Reload Window**.
### Marketplace
Once published, install via the Cursor marketplace panel and select `mempalace`. Required-plugin distribution from a team marketplace is also supported.
## Post-Install Setup
After installing the plugin, run the `init` command in a Cursor chat:
```
/mempalace-init
```
(Or just say "use the mempalace skill" — Cursor will model-invoke the bundled skill.)
This installs the `mempalace` package via `uv tool` or `pip`, initializes a palace under `~/.mempalace/`, and verifies the MCP server is reachable.
## Available Slash Commands
| Command | Description |
|---------------------|-----------------------------------------------------------------------------------|
| `/mempalace-help` | Show available tools, skills, CLI commands, hooks, and architecture |
| `/mempalace-init` | Set up MemPalace — install, configure, onboard |
| `/mempalace-search` | Search your memories across the palace using semantic search |
| `/mempalace-mine` | Mine projects and conversations into the palace |
| `/mempalace-status` | Show palace overview — wings, rooms, drawer counts |
> Cursor commands are global, not plugin-namespaced — that's why each slug is prefixed with `mempalace-` rather than appearing as `/help`, `/init`, etc. This keeps them collision-free with built-in or other-plugin commands.
## Skills
Two model-invocable skills ship at the plugin root under `skills/`:
| Skill | What it does |
|-------|--------------|
| `mempalace` | Setup, mining, status, and the dynamic `mempalace instructions` CLI. |
| `mempalace-recall` | Search-before-answer protocol — makes the agent read the palace before answering about past work, people, projects, or prior decisions instead of guessing. |
Cursor surfaces these automatically when a request matches their description, or you can attach them explicitly.
## Recall rule (optional)
The plugin also ships a Cursor rule at the plugin root under `rules/mempalace-recall.mdc`:
```yaml
description: When the user asks about past work, prior decisions, people, ... call mempalace_search before answering ...
alwaysApply: false
```
It is `alwaysApply: false` on purpose — Cursor loads it only when its matcher judges the turn recall-relevant, so it never fires on unrelated coding work and never adds MCP latency to greenfield tasks. The rule, the `mempalace-recall` skill, and the `sessionStart` hook all reference the same canonical protocol in [`integrations/shared/recall-protocol.md`](../integrations/shared/recall-protocol.md).
Want recall forced into **every** conversation regardless of context? Copy the aggressive `alwaysApply: true` variant from [`examples/cursor/rules/`](../examples/cursor/rules/README.md) into `~/.cursor/rules/`. That is a deliberate, heavier opt-in, not a default.
## MCP Server
This plugin ships `mcp.json` at the plugin root, so Cursor auto-loads the `mempalace-mcp` server on plugin install:
```json
{
"mcpServers": {
"mempalace": {
"command": "mempalace-mcp"
}
}
}
```
All 34 MemPalace MCP tools (`mempalace_search`, `mempalace_add_drawer`, `mempalace_diary_write`, `mempalace_check_duplicate`, `mempalace_diary_read`, …) become available to the agent immediately. No manual `~/.cursor/mcp.json` edit required.
If the server doesn't appear, confirm `mempalace-mcp` is on the user `$PATH`:
```bash
command -v mempalace-mcp
```
If it isn't, run `/init` (or `mempalace install` from a terminal) — `mempalace-mcp` is installed alongside the `mempalace` package.
## Hooks (optional)
Cursor's hooks system is configured separately from plugins (in `~/.cursor/hooks.json` or `.cursor/hooks.json`), so this plugin does **not** wire hooks itself. The MemPalace repository ships three Cursor-native hooks under [`hooks/cursor/`](../hooks/cursor/) that you install with one command.
User scope — writes `~/.cursor/hooks.json`, applies to every Cursor workspace (recommended):
```bash
hooks/cursor/install.sh --scope user --variant full
```
Project scope — writes `.cursor/hooks.json` under the current project only:
```bash
hooks/cursor/install.sh --scope project --variant full
```
What you get:
| Hook event | What it does |
|----------------|-------------------------------------------------------------------------------------------------------|
| `sessionStart` | Injects an `additional_context` recap of relevant memories scoped to the workspace wing |
| `stop` | Counts agent turns; every N turns, emits a `followup_message` instructing a memory checkpoint |
| `preCompact` | Synchronously mines the transcript before compaction, drops a marker so the next `stop` saves a diary |
Full details: [`website/guide/cursor-hooks.md`](../website/guide/cursor-hooks.md) and [`hooks/cursor/README.md`](../hooks/cursor/README.md).
## Uninstall
Remove the local plugin symlink:
```bash
rm ~/.cursor/plugins/local/mempalace
```
Then in Cursor: <kbd>Cmd</kbd>-<kbd>Shift</kbd>-<kbd>P</kbd> → **Developer: Reload Window**.
If you also installed the hooks, remove them (leaves any unrelated hooks in `hooks.json` untouched):
```bash
hooks/cursor/install.sh --scope user --uninstall
```
## Full Documentation
See the main [README](../README.md) for complete documentation, architecture details, and advanced usage.
+17
View File
@@ -0,0 +1,17 @@
{
"name": "mempalace",
"owner": {
"name": "milla-jovovich",
"url": "https://github.com/MemPalace"
},
"plugins": [
{
"name": "mempalace",
"source": ".",
"description": "AI memory system — mine projects and conversations into a searchable palace. 35 MCP tools, slash commands, and a guided skill for Cursor.",
"author": {
"name": "milla-jovovich"
}
}
]
}
+7
View File
@@ -0,0 +1,7 @@
{
"mcpServers": {
"mempalace": {
"command": "mempalace-mcp"
}
}
}
+19
View File
@@ -0,0 +1,19 @@
{
"name": "mempalace",
"description": "Give your AI a memory — mine projects and conversations into a searchable palace. 35 MCP tools, slash commands, and a guided skill for Cursor.",
"author": {
"name": "milla-jovovich"
},
"homepage": "https://github.com/MemPalace/mempalace",
"repository": "https://github.com/MemPalace/mempalace",
"license": "MIT",
"keywords": [
"memory",
"ai",
"rag",
"mcp",
"chromadb",
"palace",
"search"
]
}
+7 -1
View File
@@ -30,7 +30,13 @@ jobs:
python-version: "3.13"
cache: 'pip'
- run: pip install -e ".[dev]"
- run: python -m pytest tests/ -v --ignore=tests/benchmarks --cov=mempalace --cov-report=term-missing --cov-fail-under=80 --durations=10
# ChromaDB's rust HNSW core intermittently fails compaction on Windows
# ("Failed to apply logs to the hnsw segment writer") regardless of our
# code — a long-standing, non-reproducible-on-Linux/macOS flake. Retry
# ONLY that specific transient error (via --only-rerun) so real,
# deterministic failures still fail on the first run. Linux/macOS jobs
# deliberately run with no reruns so genuine regressions surface there.
- run: python -m pytest tests/ -v --ignore=tests/benchmarks --cov=mempalace --cov-report=term-missing --cov-fail-under=80 --durations=10 --reruns 2 --reruns-delay 5 --only-rerun "Failed to apply logs to the hnsw segment writer"
test-macos:
runs-on: macos-latest
+4 -4
View File
@@ -36,7 +36,7 @@ jobs:
# do not push.
- name: Log in to GHCR
if: github.event_name != 'pull_request'
uses: docker/login-action@v3
uses: docker/login-action@v4
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
@@ -44,7 +44,7 @@ jobs:
- name: Extract metadata
id: meta
uses: docker/metadata-action@v5
uses: docker/metadata-action@v6
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
# latest -> main (the latest release); semver tags -> released versions.
@@ -56,7 +56,7 @@ jobs:
type=raw,value=latest,enable=${{ github.ref == 'refs/heads/main' }}
- name: Build and push
uses: docker/build-push-action@v6
uses: docker/build-push-action@v7
with:
context: .
file: ./Dockerfile
@@ -85,7 +85,7 @@ jobs:
uses: docker/setup-buildx-action@v3
- name: Build GPU image (validation only — not published)
uses: docker/build-push-action@v6
uses: docker/build-push-action@v7
with:
context: .
file: ./Dockerfile.gpu
+84 -1
View File
@@ -6,6 +6,80 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
---
## [Unreleased]
---
## [3.5.0] — 2026-06-22
### Features
- **Opt-in local daemon for queued writes.** A new `mempalace daemon` queues MemPalace writes through a single local process so background mines, diary saves, and hook-driven ingests serialize against one palace handle instead of racing for it. Opt-in and local-only — nothing binds to a public interface. (#1826)
- **Opt-in HTTP transport for the MCP server.** `mempalace-mcp --transport http` serves JSON-RPC at `POST /mcp` (with a `GET /healthz` liveness probe) for operators running MemPalace behind a long-lived HTTP MCP client/proxy, avoiding the long-lived-stdio framing failures of #1801. stdio remains the default and is unchanged. The transport reuses the exact stdio request dispatcher (no separate write/search path), binds `127.0.0.1` by default, and is hardened against the two ways a local HTTP server leaks to the network: it pins the `Host` header to loopback on a loopback bind and rejects any non-loopback browser `Origin` (DNS-rebinding/SSRF guard), and supports an optional bearer token via `MEMPALACE_MCP_HTTP_TOKEN` (required on `/mcp`, never on `/healthz`). A 16 MiB request cap and a loud warning when bound to a non-loopback host round it out. (#1801, #1806)
- **`mempalace_checkpoint` batch-save MCP tool.** Collapses multiple `add_drawer` calls plus an optional diary entry into a single MCP round-trip for agents that want to file a whole session at once. Stores content verbatim and reuses the existing idempotent add/dedup path. (#1851)
- **`mempalace_delete_by_source` bulk-cleanup MCP tool.** Exact-match, dry-run-by-default deletion of every drawer (and its matching closet/AAAK index entries) for a given `source_file` — the recourse for benchmark/eval files mined into the same wing as real data and drowning out search. The dry run reports the drawer and closet blast radius before anything is removed, and the commit writes a WAL audit entry. (#1722, #1729)
- **Optional `source_file` filter for `mempalace_search`.** Scope a search to an exact stored source path. The filter is threaded through every search path (vector, BM25/SQLite fallback, lexical union, and index-mismatch fallback) so it never silently drops a matching drawer, and results now expose the full `source_path` as a round-trippable key. (#1815, #1817)
- **New transcript parsers / importers.** Continue.dev session parser (#731), Gemini CLI / AI Studio JSON session import (#204), and a Pi agent JSONL session normalizer (#169).
- **Wider miner language coverage.** C# / .NET, PHP (#1819), Swift / Kotlin (#1368), and Java project detection including rootless subprojects (#1720).
- **Final mine on Claude plugin `SessionEnd`.** The Claude Code plugin now runs a closing mine when a session ends so the last exchanges are captured without waiting for the next save nudge. (#1814, #1820)
### Performance
- **Overview/status MCP tools answer from the SQLite aggregate.** Large palaces no longer time out building wing/room/status overviews — the counts come from a single SQLite aggregate instead of a client-side fetch-and-tally. (#1748, #1379)
- **`graph_stats` SQLite fast path.** Knowledge-graph stats are computed in SQLite rather than walking the collection, fixing large-palace timeouts. (#1379)
- **Embedder caps ONNX-runtime intra-op threads** so a background mine no longer pins every core. (#1068)
- **Backend pagination pushed into the query.** `sqlite_exact` (#1841, #1842) and `pgvector` (#1830, #1840) now apply `get(limit, offset)` in SQL, and Qdrant fetches bulk metadata in a single scroll with a larger page size (#1796, #1832).
### Bug Fixes
- **pgvector tolerates hostile transcript bytes.** A lone Unicode surrogate (#1833) or a NUL byte (#1829) in a transcript no longer aborts the whole mine — both are sanitized before the row is written.
- **SQLite read-only URIs are percent-encoded** so palace paths with spaces or special characters open correctly, and `_sqlite_graph_stats` is routed through the same `sqlite_read_uri` helper.
- **Stale ChromaDB HNSW divergence routes to the SQLite fallback** instead of failing the read outright. (#1816, #1822)
- **Diverged-index recovery now points at `repair --mode from-sqlite`, not a re-mine.** A failed ChromaDB HNSW compaction leaves the index out of sync while the rows stay intact in `chroma.sqlite3`; the old "re-mine from source" advice silently dropped MCP-added drawers and diary entries (which have no source file). Both the legacy `repair`/`rebuild_index` error messages and the `repair-status` recommendation, plus the recall skill docs, now guide users to rebuild from SQLite. (#1843, #1847, #1849)
- **The MCP server refuses a second writer for the same palace** rather than letting two processes race the same HNSW handle. (#1818, #1823)
- **Windows hook miner spawns with `CREATE_NO_WINDOW`** so background mines no longer flash a console window. (#1783, #1848)
- **`fact_checker` `__main__` no longer emits a runpy warning** under the test runner. (#1798)
### Internal
- Live-substrate conformance test module for pgvector (#1769); dependabot bumps for `docker/login-action` (3→4), `docker/build-push-action` (6→7), and `docker/metadata-action` (5→6) (#1788, #1787, #1786); ruff dev dependency bumped to 0.15.18.
---
## [3.4.1] — 2026-06-14
### Features
- **Cursor IDE plugin (`.cursor-plugin/`).** Drops into `~/.cursor/plugins/local/mempalace` (or installs from the Cursor marketplace once published) and auto-registers the `mempalace-mcp` server, five slash commands (`/mempalace-help`, `/mempalace-init`, `/mempalace-mine`, `/mempalace-search`, `/mempalace-status`), and the model-invocable [`mempalace` skill](skills/mempalace/SKILL.md) — no manual `~/.cursor/mcp.json` edit required. The plugin manifest deliberately omits a hardcoded `version` field — `mempalace/version.py` is the single source of truth, so there is nothing to drift on the next release (a contract test enforces the field stays absent). The canonical plugin components (`commands/`, `skills/`, `mcp.json`) are real files at the plugin root; no symlinks are committed (committed symlinks materialise as broken text files on Windows clones with `core.symlinks=false`). Mirrors the surface of [`.claude-plugin/`](.claude-plugin/) and [`.codex-plugin/`](.codex-plugin/) without duplicating their hook scripts: the Cursor hook scripts under [`hooks/cursor/`](hooks/cursor/) (shipped in the same release) remain the canonical install path for `stop` / `preCompact` / `sessionStart`, wired separately by [`hooks/cursor/install.sh`](hooks/cursor/install.sh). Contract tests in [`tests/test_cursor_plugin_manifest.py`](tests/test_cursor_plugin_manifest.py) cover manifest JSON validity, kebab-case naming, `..`-free relative paths, on-disk path resolution, marketplace alignment, MCP config shape (`mcpServers` wrapper required by Cursor, unlike Claude's flat `.mcp.json`), the version-field-absent guard, the no-symlink guard, and every skill/command frontmatter — all pure file inspection so they run on any CI platform without Cursor itself.
- **Cursor IDE hook support (`stop` / `preCompact` / `sessionStart`).** Three new bash hooks live under [`hooks/cursor/`](hooks/cursor/) and share a `lib/common.sh` helpers module. The save hook counts `stop` invocations per Cursor `conversation_id` and emits a `followup_message` every `MEMPAL_SAVE_INTERVAL` (default 15) so the agent files the session into MemPalace and writes a diary entry. Unlike the silent-by-default Claude Code hook, the Cursor followup fires **on by default**: Cursor's transcript format is undocumented and `normalize.py` has no Cursor parser yet, so the background `mempalace mine --mode convos` is best-effort only and the `followup_message` is the load-bearing verbatim-capture path. Users who want the Claude-style "zero tokens in the chat window" behaviour can suppress it with `MEMPAL_CURSOR_SILENT=1` (or `MEMPAL_VERBOSE=false`); the default flips to silent once a Cursor transcript parser lands. The precompact hook synchronously mines the transcript before Cursor's compaction summarises it and drops a marker so the next `stop` forces a save nudge (Cursor's `preCompact` is observational-only — it cannot block or emit a `followup_message`, unlike Claude Code's `PreCompact`); the synchronous mine is bounded by Cursor's per-hook timeout, and because `mempalace mine` is incremental/append-only a killed mine resumes cleanly on the next run rather than corrupting the palace. The wake hook is Cursor-only: `sessionStart` returns `additional_context` telling the agent to recall scoped to the wing inferred from the workspace root. Honours the same `MEMPALACE_HOOKS_AUTO_SAVE=false` kill switch as the Claude Code hooks, plus a new `MEMPAL_DISABLE_HOOK=1` alias and a `MEMPAL_STATE_DIR` env override. Per-conversation state files are garbage-collected by a daily-throttled, Cursor-namespaced TTL sweep (`MEMPAL_STATE_TTL_DAYS`, default 30) so `cursor_*.count` / `cursor_*.pending` cannot grow unbounded — shared logs and other editors' state are never touched. Includes an opt-in installer at [`hooks/cursor/install.sh`](hooks/cursor/install.sh) with `--scope user|project`, `--variant full|minimal`, `--dry-run`, and `--uninstall` (idempotent, preserves unrelated hooks via `python3`-based JSON merge — no `jq` dependency). Example wirings live at [`examples/cursor/hooks.json`](examples/cursor/hooks.json) and [`examples/cursor/hooks.minimal.json`](examples/cursor/hooks.minimal.json); they are intentionally not placed at the repo root because Cursor auto-loads project hooks from any trusted workspace and we do not arm hooks on contributor checkout. Per-event stdin/stdout schema documented at [`hooks/cursor/STDIN_SHAPE.md`](hooks/cursor/STDIN_SHAPE.md). Walkthrough at [`website/guide/cursor-hooks.md`](website/guide/cursor-hooks.md). Coverage added in [`tests/test_cursor_hooks_shell.py`](tests/test_cursor_hooks_shell.py) and [`tests/test_cursor_hooks_install.py`](tests/test_cursor_hooks_install.py).
- **First-class Antigravity IDE support.** New `.antigravity-plugin/` package + idempotent installer at `hooks/antigravity/install.sh` that registers MemPalace as a Google Antigravity plugin (MCP server, skill, two lifecycle hooks) at `~/.gemini/config/plugins/mempalace/`. The Stop hook background-mines the active conversation transcript every Nth fire (default 15, configurable via `MEMPAL_SAVE_INTERVAL`); the PreInvocation hook injects verbatim memory on the first model call only via Antigravity's `injectSteps[].ephemeralMessage` output, gated by `invocationNum == 1`. Both hooks are bash 3.2.57 compatible (macOS default), use the same `~/.mempalace/hook_state/` directory as the Claude Code / Codex / Cursor hooks (`antigravity_*`-namespaced state files), and respect every existing kill switch (`MEMPAL_DISABLE_HOOK`, `MEMPALACE_HOOKS_AUTO_SAVE`, `~/.mempalace/config.json` `hooks.auto_save`). Installer is `cmp`-gated (re-run produces a byte-identical install), uninstall is basename-guarded (refuses to wipe a directory whose basename isn't `mempalace`), and `--dry-run` is side-effect free. Full audit of which Antigravity surfaces we ship and which we deliberately don't is in [`hooks/antigravity/INVESTIGATION.md`](hooks/antigravity/INVESTIGATION.md). User-facing guide: [`website/guide/antigravity.md`](website/guide/antigravity.md). Standalone examples in [`examples/antigravity/`](examples/antigravity/).
- **Zero-config interpreter resolution.** `mempal_resolve_python` now derives the Python interpreter from the `mempalace-mcp` / `mempalace` console-script shebang on `$PATH` before falling back to `python3`. The common `uv tool install mempalace` / `pipx install` layout installs the console scripts into an isolated environment whose interpreter is **not** system `python3`, so the previous `command -v python3` resolution landed on a Python that couldn't import `mempalace`, the `-m mempalace` probe failed, and mining silently never fired. Resolution is pure shebang parsing + `stat` (no Python subprocess at source time, preserving the hook performance budget). `MEMPAL_PYTHON` remains the explicit override. Documented under *How the hooks find your `mempalace` install* in the guide.
### Bug Fixes
- **`embeddinggemma` no longer OOM-kills bulk re-embeds.** `EmbeddinggemmaONNX.__call__` ran a single `session.run` over its entire input, so a repair-scale batch (5000 docs) allocated attention buffers far beyond available RAM and the kernel killed the process silently: `mempalace repair --yes` on an `embedding_model: embeddinggemma` palace died right after `Building temporary collection:` with no traceback and no crash report (#1770). Embedding now runs in sub-batches of 32 docs (constructor-tunable `batch_size`), matching the internal batching of ChromaDB's bundled MiniLM embedder. Per-document vectors are unchanged: the model's pooled output is attention-masked, so sub-batch padding does not affect values. The embedder is also hardened for shared use: the process-wide EF cache and the lazy model load are thread-safe (concurrent first calls build exactly one ONNX session), and `__call__` handles a bare string, `None`, and empty input without triggering the model download.
- **Backup retention to prevent unbounded disk usage.** `mempalace migrate` (full-palace `<palace>.pre-migrate.<timestamp>` copies) and `mempalace repair max-seq-id` (`chroma.sqlite3.max-seq-id-backup-<timestamp>` copies) each wrote a fresh, full-size, timestamped backup every run and never deleted the old ones. On a machine that mines or repairs on a schedule, those copies could silently accumulate until they filled the disk — one palace was found with hundreds of GB of stale backups beside a few hundred MB of live data, hidden from a normal `du` of the home directory. A new `max_backups` setting (default `10`, env `MEMPALACE_MAX_BACKUPS`, or `config.json`) now prunes the oldest backups after each new one is written. Set it to `0` to keep every backup. Pruning is keyed by filesystem mtime, scoped strictly to each backup's own naming pattern (live data is never touched), and best-effort so a deletion failure can never abort a migration or repair that already succeeded.
---
## [3.3.6] — 2026-05-24
### Features
@@ -494,7 +568,16 @@ Initial public release.
---
[Unreleased]: https://github.com/MemPalace/mempalace/compare/v3.2.0...HEAD
[Unreleased]: https://github.com/MemPalace/mempalace/compare/v3.4.1...HEAD
[3.4.1]: https://github.com/MemPalace/mempalace/compare/v3.4.0...v3.4.1
[3.4.0]: https://github.com/MemPalace/mempalace/compare/v3.3.6...v3.4.0
[3.3.6]: https://github.com/MemPalace/mempalace/compare/v3.3.5...v3.3.6
[3.3.5]: https://github.com/MemPalace/mempalace/compare/v3.3.4...v3.3.5
[3.3.4]: https://github.com/MemPalace/mempalace/compare/v3.3.3...v3.3.4
[3.3.3]: https://github.com/MemPalace/mempalace/compare/v3.3.2...v3.3.3
[3.3.2]: https://github.com/MemPalace/mempalace/compare/v3.3.1...v3.3.2
[3.3.1]: https://github.com/MemPalace/mempalace/compare/v3.3.0...v3.3.1
[3.3.0]: https://github.com/MemPalace/mempalace/compare/v3.2.0...v3.3.0
[3.2.0]: https://github.com/MemPalace/mempalace/compare/v3.1.0...v3.2.0
[3.1.0]: https://github.com/MemPalace/mempalace/compare/v3.0.0...v3.1.0
[3.0.0]: https://github.com/MemPalace/mempalace/releases/tag/v3.0.0
+12 -5
View File
@@ -156,7 +156,8 @@ mempalace search "why did we switch to GraphQL"
mempalace wake-up
```
For Claude Code, Gemini CLI, MCP-compatible tools, and local models, see
For Claude Code, Gemini CLI, [Antigravity](https://mempalaceofficial.com/guide/antigravity.html),
MCP-compatible tools, and local models, see
[mempalaceofficial.com/guide/getting-started](https://mempalaceofficial.com/guide/getting-started.html).
---
@@ -224,7 +225,7 @@ Usage and tool reference:
## MCP server
29 MCP tools cover palace reads/writes, knowledge-graph operations,
35 MCP tools cover palace reads/writes, knowledge-graph operations,
cross-wing navigation, drawer management, and agent diaries. Installation
and the full tool list:
[mempalaceofficial.com/reference/mcp-tools](https://mempalaceofficial.com/reference/mcp-tools.html).
@@ -238,8 +239,14 @@ system prompt:
## Auto-save hooks
Two Claude Code hooks save periodically and before context compression:
[mempalaceofficial.com/guide/hooks](https://mempalaceofficial.com/guide/hooks.html).
Auto-save hooks for **Claude Code, Codex CLI, and Cursor IDE** save
periodically and before context compression:
- Claude Code + Codex →
[mempalaceofficial.com/guide/hooks](https://mempalaceofficial.com/guide/hooks.html)
- Cursor IDE (adds session-start recall and a transcript snapshot before
compaction) →
[mempalaceofficial.com/guide/cursor-hooks](https://mempalaceofficial.com/guide/cursor-hooks.html)
If you are installing under time pressure, start with the
[Claude Code retention setup checklist](https://mempalaceofficial.com/guide/claude-code-retention.html):
@@ -278,7 +285,7 @@ PRs welcome. See [CONTRIBUTING.md](CONTRIBUTING.md).
MIT — see [LICENSE](LICENSE).
<!-- Link Definitions -->
[version-shield]: https://img.shields.io/badge/version-3.4.0-4dc9f6?style=flat-square&labelColor=0a0e14
[version-shield]: https://img.shields.io/badge/version-3.5.0-4dc9f6?style=flat-square&labelColor=0a0e14
[release-link]: https://github.com/MemPalace/mempalace/releases
[python-shield]: https://img.shields.io/badge/python-3.9+-7dd8f8?style=flat-square&labelColor=0a0e14&logo=python&logoColor=7dd8f8
[python-link]: https://www.python.org/
+7
View File
@@ -0,0 +1,7 @@
---
description: Show comprehensive MemPalace help — available skills, MCP tools, CLI commands, hooks, and architecture.
---
Invoke the `mempalace` skill from this plugin and run the `help` instructions, then follow them.
Concretely: run `mempalace instructions help` in a terminal, then carry out the steps it prints.
+12
View File
@@ -0,0 +1,12 @@
---
description: Set up MemPalace — install the package, initialize a palace, register the MCP server with Cursor, and verify everything works.
---
Invoke the `mempalace` skill from this plugin and run the `init` instructions, then follow them.
Concretely: run `mempalace instructions init` in a terminal, then carry out the steps it prints.
Cursor-specific extras after init:
1. The `mempalace-mcp` server is already auto-registered by this plugin — no manual `mcp.json` edit needed.
2. For automatic background saves and session-start memory recall, also run `hooks/cursor/install.sh --scope user` from a cloned MemPalace repo. See `website/guide/cursor-hooks.md` for the walkthrough.
+7
View File
@@ -0,0 +1,7 @@
---
description: Mine projects and conversations into the MemPalace. Supports project files, conversation exports, and auto-classification.
---
Invoke the `mempalace` skill from this plugin and run the `mine` instructions, then follow them.
Concretely: run `mempalace instructions mine` in a terminal, then carry out the steps it prints.
+7
View File
@@ -0,0 +1,7 @@
---
description: Search your memories across the MemPalace using semantic search with wing/room filtering.
---
Invoke the `mempalace` skill from this plugin and run the `search` instructions, then follow them.
Concretely: run `mempalace instructions search` in a terminal, then carry out the steps it prints. The MCP tool `mempalace_search` is also available directly from this Cursor session.
+7
View File
@@ -0,0 +1,7 @@
---
description: Show the current state of your memory palace — wings, rooms, drawer counts, and suggestions.
---
Invoke the `mempalace` skill from this plugin and run the `status` instructions, then follow them.
Concretely: run `mempalace instructions status` in a terminal, then carry out the steps it prints.
+74
View File
@@ -0,0 +1,74 @@
# MemPalace — Antigravity examples
Two standalone configs for users who don't want to use the
`hooks/antigravity/install.sh` installer.
## Files
| File | Purpose |
|-------------------|-----------------------------------------------------------------------------------|
| `hooks.json` | Standalone `hooks.json` registering the Stop and PreInvocation hooks. |
| `mcp_config.json` | Standalone MCP entry registering the `mempalace-mcp` stdio server. |
## Wire up `hooks.json`
The example uses placeholder absolute paths (`/ABSOLUTE/PATH/TO/mempalace/...`).
You must rewrite both `command` fields to the actual absolute paths to
the hook scripts in your cloned repo, or to whichever location holds
them. Antigravity will not resolve relative paths reliably.
Then drop the file at one of:
- `~/.gemini/config/hooks.json` (global, applies to every workspace)
- `<workspace>/.agents/hooks.json` (workspace-scoped)
Restart Antigravity to pick the file up.
If you'd rather have the paths absolutized automatically, run the
installer:
```bash
bash hooks/antigravity/install.sh
```
That writes a fully rendered `hooks.json` to
`~/.gemini/config/plugins/mempalace/hooks.json`.
## Wire up `mcp_config.json`
The example registers the `mempalace-mcp` stdio server. Two options:
### Option A — merge into the user-level Antigravity MCP config
Antigravity's user-level MCP config lives at
`~/.gemini/antigravity/mcp_config.json`. Merge the `mcpServers.mempalace`
entry from this example into that file, then restart Antigravity.
### Option B — drop into a plugin directory
If you've created a custom plugin folder (per the [Antigravity plugins docs](https://antigravity.google/docs/plugins)),
copy this `mcp_config.json` directly into the plugin root:
```
<plugin-root>/mcp_config.json
```
Antigravity merges plugin-level MCP entries with the user-level config
on launch.
## Verify
After wiring up either or both:
```bash
mempalace-mcp --version # confirm binary is on PATH
ls ~/.mempalace/ # confirm palace exists (run `mempalace init` if not)
```
Restart Antigravity. The `mempalace` MCP server should appear in the
MCP store; the Stop and PreInvocation hooks fire automatically.
See [`hooks/antigravity/STDIN_SHAPE.md`](../../hooks/antigravity/STDIN_SHAPE.md)
for the exact wire format Antigravity uses, and
[`website/guide/antigravity.md`](../../website/guide/antigravity.md)
for the full user-facing guide.
+21
View File
@@ -0,0 +1,21 @@
{
"mempalace-save": {
"Stop": [
{
"type": "command",
"command": "/ABSOLUTE/PATH/TO/mempalace/hooks/antigravity/mempal_save_hook_antigravity.sh",
"timeout": 30
}
]
},
"mempalace-wake": {
"PreInvocation": [
{
"type": "command",
"command": "/ABSOLUTE/PATH/TO/mempalace/hooks/antigravity/mempal_wake_hook_antigravity.sh",
"timeout": 5
}
]
}
}
+7
View File
@@ -0,0 +1,7 @@
{
"mcpServers": {
"mempalace": {
"command": "mempalace-mcp"
}
}
}
+110
View File
@@ -0,0 +1,110 @@
# Cursor IDE Hooks — Example `hooks.json` Files
Sample configurations for wiring the MemPalace Cursor hooks into the
Cursor IDE. These are **examples only** — they are intentionally not
placed at the repo root (`/.cursor/hooks.json`) because Cursor
auto-loads project-level hooks from any trusted workspace, and the
repo is regularly opened by contributors. We do not auto-arm hooks on
contributor checkout.
## Variants
### `hooks.json` — full (recommended)
Three hooks wired:
- **`sessionStart`** — calls `mempal_wake_hook_cursor.sh`, which returns
`additional_context` telling the agent to recall scoped to the wing
inferred from the workspace root. Cursor-only — Claude Code has no
equivalent.
- **`stop`** — calls `mempal_save_hook_cursor.sh`. Counts stop
invocations per conversation and emits a `followup_message` every
`MEMPAL_SAVE_INTERVAL` (default 15) telling the agent to file the
session into the palace and write a diary entry. `loop_limit: 1` is
defense-in-depth on top of our own loop-count check.
- **`preCompact`** — calls `mempal_precompact_hook_cursor.sh`. Runs
`mempalace mine` synchronously on the transcript before compaction
summarises it, then drops a marker so the next `stop` forces a save
followup.
### `hooks.minimal.json` — `stop` only
Lightest install. Wires just the save hook. Use this if you don't
want the sessionStart recall context or the preCompact transcript
snapshot.
## How to use
The `$HOME` placeholder is **not** expanded by Cursor — you must
substitute the absolute path before saving the file. Pick one:
### Option A — let `install.sh` do it
Project scope — writes `<repo>/.cursor/hooks.json`:
```bash
hooks/cursor/install.sh --scope project --target /path/to/your/repo
```
User scope — writes `~/.cursor/hooks.json`, applies to every Cursor workspace:
```bash
hooks/cursor/install.sh --scope user
```
The installer copies the hook scripts to `~/.mempalace/hooks/cursor/`,
substitutes the absolute paths, and merges the entries into your
existing `hooks.json` without clobbering unrelated hooks. See
`install.sh --help` for `--dry-run`, `--uninstall`, and `--variant`.
### Option B — copy + edit manually
1. Copy the chosen example to the target location:
- User scope: `~/.cursor/hooks.json`
- Project scope: `<your-repo>/.cursor/hooks.json`
2. Replace every `$HOME` with the absolute path to your home
directory (e.g., `/Users/you` or `/home/you`).
3. Make sure each hook script is executable
(`chmod +x ~/.mempalace/hooks/cursor/mempal_*_hook_cursor.sh`).
4. Restart Cursor, or wait for it to auto-reload the file.
## Why aren't these files at the repo root?
Cursor automatically loads `.cursor/hooks.json` from any trusted
workspace. Placing a real `hooks.json` at the repo root would arm
MemPalace's hooks on every contributor's machine the moment they open
the repo in Cursor — which would modify their conversation behaviour
without consent and write to `~/.mempalace/hook_state/` without
asking. Editor configuration is sacred; opt-in only.
If you actually want MemPalace's hooks armed when working on the
MemPalace repo itself, run:
```bash
hooks/cursor/install.sh --scope project --target .
```
That will write `./.cursor/hooks.json` for the repo workspace
specifically — but it is your decision, not ours, and the file is
listed in `.gitignore` paths Cursor users typically already exclude.
## Related: the Cursor plugin
The hooks here are **only one half** of MemPalace's Cursor integration. The other half is the [`.cursor-plugin/`](../../.cursor-plugin/) folder at the repo root, which packages MemPalace's MCP server, five slash commands, and the model-invocable `mempalace` skill as a regular Cursor plugin you can drop into `~/.cursor/plugins/local/mempalace`.
The two install paths are orthogonal — install whichever you want, in any order:
| You want | Install |
|---------------------------------------------------------------------------|----------------------------------------------------------|
| MCP tools (`mempalace_search`, `mempalace_add_drawer`, …) + slash commands | The plugin — see [`.cursor-plugin/README.md`](../../.cursor-plugin/README.md) |
| Auto-save every N turns + sessionStart memory recall | The hooks here — see Option A above |
| Both | Install the plugin AND run `hooks/cursor/install.sh` |
Hooks are deliberately **not** bundled into the plugin because Cursor's hooks system is configured per-user/per-project (in `~/.cursor/hooks.json` or `.cursor/hooks.json`), not per-plugin — so the installer here owns that file with idempotent merge semantics, while the plugin owns the MCP+commands+skill side.
## See also
- [`hooks/cursor/README.md`](../../hooks/cursor/README.md) — full reference for hooks
- [`hooks/cursor/STDIN_SHAPE.md`](../../hooks/cursor/STDIN_SHAPE.md) — per-event schema with citations
- [`website/guide/cursor-hooks.md`](../../website/guide/cursor-hooks.md) — rendered docs
- [`.cursor-plugin/README.md`](../../.cursor-plugin/README.md) — Cursor plugin (MCP + commands + skill)
+21
View File
@@ -0,0 +1,21 @@
{
"version": 1,
"hooks": {
"sessionStart": [
{
"command": "$HOME/.mempalace/hooks/cursor/mempal_wake_hook_cursor.sh"
}
],
"stop": [
{
"command": "$HOME/.mempalace/hooks/cursor/mempal_save_hook_cursor.sh",
"loop_limit": 1
}
],
"preCompact": [
{
"command": "$HOME/.mempalace/hooks/cursor/mempal_precompact_hook_cursor.sh"
}
]
}
}
+11
View File
@@ -0,0 +1,11 @@
{
"version": 1,
"hooks": {
"stop": [
{
"command": "$HOME/.mempalace/hooks/cursor/mempal_save_hook_cursor.sh",
"loop_limit": 1
}
]
}
}
+58
View File
@@ -0,0 +1,58 @@
# Cursor Rules — MemPalace recall
Optional [Cursor rules](https://cursor.com/docs/rules) that make the
agent search MemPalace before answering questions about past work,
people, projects, or prior decisions.
These are for users who install MemPalace **without** the Cursor plugin
(or who want recall behaviour in a specific project). If you installed
the [Cursor plugin](../../../.cursor-plugin/README.md), it already ships
the `alwaysApply: false` rule at the plugin root — you do not need to
copy anything.
## Which file to use
| File | `alwaysApply` | Fires when | Use when |
|------|---------------|------------|----------|
| [`mempalace-recall.mdc`](mempalace-recall.mdc) | `false` | Cursor's matcher decides the turn is recall-relevant (from the rule `description`) | **Recommended.** Recall without paying for the rule on unrelated work. |
| [`mempalace-recall-always.mdc`](mempalace-recall-always.mdc) | `true` | Every conversation in scope, every turn | You want recall guaranteed in context and accept the cost. |
The always-on variant is heavier: it sits in context on every turn and
makes the agent more eager to call `mempalace_search`, which adds MCP
latency and works against MemPalace's "memory should feel instant"
budget. Prefer the `false` variant unless you specifically want recall
forced into every conversation. Pick **one** of the two — do not install
both.
## Install
User scope (every workspace) — copy into `~/.cursor/rules/`:
```bash
mkdir -p ~/.cursor/rules
cp examples/cursor/rules/mempalace-recall.mdc ~/.cursor/rules/
```
Project scope (this repo only) — copy into `.cursor/rules/`:
```bash
mkdir -p .cursor/rules
cp examples/cursor/rules/mempalace-recall.mdc .cursor/rules/
```
For the aggressive variant, copy `mempalace-recall-always.mdc` instead
(only one of the two). Then reload Cursor:
<kbd>Cmd</kbd>-<kbd>Shift</kbd>-<kbd>P</kbd> → **Developer: Reload Window**.
## How recall is delivered
Recall ships in three orthogonal layers — install any combination:
| Layer | What it does | Where |
|-------|--------------|-------|
| `sessionStart` hook | Injects wing-scoped recall context once per new chat | [`hooks/cursor/`](../../../hooks/cursor/) |
| `mempalace-recall` skill | Full search-before-answer protocol, model-invoked or attached | [`skills/mempalace-recall/`](../../../skills/mempalace-recall/) |
| Recall rule (these files) | Nudges search-before-answer on recall-relevant turns | here, or the plugin root `rules/` |
All three reference the same canonical protocol in
[`integrations/shared/recall-protocol.md`](../../../integrations/shared/recall-protocol.md).
@@ -0,0 +1,25 @@
---
description: Always-on MemPalace recall — search the palace before answering about past work, people, projects, or prior decisions.
alwaysApply: true
---
# MemPalace recall (always on)
This is the aggressive variant of the recall rule: `alwaysApply: true`
loads it into every conversation in scope, on every turn, regardless of
whether Cursor's matcher thinks recall is relevant.
Before answering anything that may already be in the user's memory
palace — past work, prior decisions, a person, a project, or "what did
we do / decide / discuss last time?" — search the palace first:
1. Call `mempalace_search` with a short keyword query. Use
`mempalace_kg_query` for relational or time-bound facts.
2. Quote the drawer's **verbatim** text. Never summarize or paraphrase
stored content.
3. If results are empty, say so — do not invent an answer. If the MCP
server is unavailable, surface the error; do not fall back to guessing.
Even with this rule always loaded, only actually call the tools when the
question touches memory. Do not search on pure greenfield work
(renaming a variable, fixing a typo).
@@ -0,0 +1,20 @@
---
description: When the user asks about past work, prior decisions, people, projects, or events that may be filed in MemPalace, call mempalace_search (or mempalace_kg_query for relational or time-bound facts) before answering from model memory. Return stored content verbatim; never guess when the palace might know.
alwaysApply: false
---
# MemPalace recall
Before answering anything that may already be in the user's memory
palace — past work, prior decisions, a person, a project, or "what did
we do / decide / discuss last time?" — search the palace first:
1. Call `mempalace_search` with a short keyword query. Use
`mempalace_kg_query` for relational or time-bound facts.
2. Quote the drawer's **verbatim** text. Never summarize or paraphrase
stored content.
3. If results are empty, say so — do not invent an answer. If the MCP
server is unavailable, surface the error; do not fall back to guessing.
Skip recall for pure greenfield work with no memory relevance (renaming
a variable, fixing a typo). Recall is question-driven, not reflexive.
+47 -1
View File
@@ -2,6 +2,13 @@
These hook scripts make MemPalace save automatically. No manual "save" commands needed.
This file covers the **Claude Code** and **Codex CLI** hooks that live
flat under `hooks/`. For the **Cursor IDE** hooks, see
[`hooks/cursor/README.md`](cursor/README.md) or the rendered docs at
[`website/guide/cursor-hooks.md`](../website/guide/cursor-hooks.md). The
two are additive and share the same `~/.mempalace/hook_state/`
directory.
If you are trying to protect existing Claude Code transcripts immediately,
use the short checklist first: [`website/guide/claude-code-retention.md`](../website/guide/claude-code-retention.md).
It covers hook wiring, JSONL backup, and one-time backfill.
@@ -11,6 +18,7 @@ It covers hook wiring, JSONL backup, and one-time backfill.
| Hook | When It Fires | What Happens |
|------|--------------|-------------|
| **Save Hook** | Every 15 human messages | Auto-mines transcript (tool output included), then blocks the AI to save topics/decisions/quotes |
| **SessionEnd Hook** | Clean session exit | Backgrounds a final transcript mine (when a transcript exists) so short sessions aren't lost; returns immediately so teardown is never delayed. A lightweight diary checkpoint is written in the detached child. |
| **PreCompact Hook** | Right before context compaction | Auto-mines transcript, then emergency save — forces the AI to save EVERYTHING before losing context |
**Two-layer capture:** Hooks auto-mine the JSONL transcript directly into the palace (capturing raw tool output — Bash results, search findings, build errors). They also block the AI with a reason message telling it to save verbatim tool output and key context. Belt and suspenders — tool output gets stored even if the AI summarizes instead of quoting.
@@ -30,6 +38,13 @@ Add to `.claude/settings.local.json`:
"timeout": 30
}]
}],
"SessionEnd": [{
"hooks": [{
"type": "command",
"command": "/absolute/path/to/hooks/mempal_session_end_hook.sh",
"timeout": 10
}]
}],
"PreCompact": [{
"hooks": [{
"type": "command",
@@ -41,11 +56,35 @@ Add to `.claude/settings.local.json`:
}
```
`SessionEnd` runs once on a clean exit and backgrounds its work, so it
returns instantly and stays within Claude Code's SessionEnd budget. Wired
through `settings.local.json` (above) the `timeout` can raise that budget;
the bundled plugin cannot, which is why the hook backgrounds rather than
mining in the foreground.
Make them executable:
```bash
chmod +x hooks/mempal_save_hook.sh hooks/mempal_precompact_hook.sh
chmod +x hooks/mempal_save_hook.sh hooks/mempal_session_end_hook.sh hooks/mempal_precompact_hook.sh
```
## Install — Antigravity (Google)
The Antigravity integration lives in its own subdirectory because the
wire format (camelCase JSON, `injectSteps[]` output) and event names
(`Stop`, `PreInvocation`) are Antigravity-specific. Use the dedicated
installer:
```bash
bash hooks/antigravity/install.sh
```
This installs to `~/.gemini/config/plugins/mempalace/`, registers the
MCP server, ships the `mempalace` skill, and wires the Stop +
PreInvocation hooks. See [`hooks/antigravity/README.md`](antigravity/README.md)
for the full guide and [`hooks/antigravity/INVESTIGATION.md`](antigravity/INVESTIGATION.md)
for the source-of-truth audit of which Antigravity surfaces the
integration uses.
## Install — Codex CLI (OpenAI)
Add to `.codex/hooks.json`:
@@ -65,6 +104,13 @@ Add to `.codex/hooks.json`:
}
```
**Other harnesses:** the clean-exit save runs through the harness-agnostic
`mempalace hook run --hook session-end` entry point. This release wires it
for Claude Code. Antigravity exposes no dedicated session-end event (its
lifecycle hooks are PreToolUse/PostToolUse/PreInvocation/PostInvocation/Stop,
and MemPalace already saves there via `Stop`); Cursor and Codex can adopt the
same entry point as a follow-up wherever their own session-end event is available.
## Configuration
Edit `mempal_save_hook.sh` to change:
+332
View File
@@ -0,0 +1,332 @@
# Antigravity IDE — Integration Surface Investigation
**Investigated**: 2026-05-27
**Author**: undeadindustries
**Scope**: What MemPalace can integrate with in Google's Antigravity IDE,
what we shipped, and what we deliberately did not ship.
This document is the source of truth for design decisions in the
`feat/antigravity-support` branch. It exists so a future maintainer can
re-derive every choice without re-reading the docs cold.
---
## 1. Surfaces verified against Google's official Antigravity docs
All quotes are pulled verbatim from Google's official Antigravity
documentation on 2026-05-27. URLs are the authoritative source; the
mirrored excerpts here are for reviewer convenience.
### 1.1. MCP — `https://antigravity.google/docs/mcp`
> The configuration file is located at `~/.gemini/antigravity/mcp_config.json`.
>
> The configuration file has a single `mcpServers` object where you
> define each server you want to connect to.
Schema (verified):
```json
{
"mcpServers": {
"<name>": {
"command": "...", // stdio
"args": [...],
"env": {...},
"cwd": "...",
"serverUrl": "...", // remote
"headers": {...},
"authProviderType": "google_credentials",
"oauth": {"clientId": "...", "clientSecret": "..."},
"disabled": false,
"disabledTools": [...]
}
}
}
```
Per-plugin form: `mcp_config.json` at the plugin root, same shape.
Antigravity merges plugin entries with the user's
`~/.gemini/antigravity/mcp_config.json` rather than clobbering.
**Cross-checked locally**: the user's existing
`~/.gemini/antigravity/mcp_config.json` already contains a working
`"mempalace": {"command": "/Users/robs/.local/bin/mempalace-mcp"}`
entry, proving the shape matches and the binary is on PATH.
### 1.2. Plugins — `https://antigravity.google/docs/plugins`
> A plugin is a directory containing a `plugin.json` file and optional
> subdirectories for different customization types:
>
> ```
> plugins/<plugin-name>/
> ├── plugin.json # Required marker file
> ├── mcp_config.json # Optional MCP server definitions
> ├── hooks.json # Optional hooks definition
> ├── skills/ # Optional skills
> │ └── <skill-name>/
> │ └── SKILL.md
> └── rules/ # Optional rules
> └── <rule-name>.md
> ```
Manifest schema (verified):
```json
{
"name": "my-custom-plugin"
}
```
> The name field is optional and defaults to the directory name if omitted.
Install locations (verified):
> - Workspace Level: Place your plugin folder inside a
> `.agents/plugins/` or `_agents/plugins/` directory at the root of
> your opened workspace.
> - Global Level: Place your plugin folder inside
> `~/.gemini/config/plugins/` in your user home directory.
We ship the global location as the canonical install path.
### 1.3. Skills — `https://antigravity.google/docs/skills`
> A skill is a folder containing a `SKILL.md` file with instructions
> that the agent can follow when working on specific tasks.
Frontmatter (verified):
| Field | Required | Description |
|---------------|----------|---------------------------------------------------------------------------------------------------|
| `name` | No | A unique identifier (lowercase, hyphens). Defaults to the folder name. |
| `description` | Yes | A clear description of what the skill does and when to use it. |
Standalone discovery paths (verified):
| Location | Scope |
|-------------------------------------|------------------------|
| `<workspace>/.agents/skills/` | Workspace-specific |
| `~/.gemini/antigravity/skills/` | Global, all workspaces |
In-plugin discovery: `<plugin>/skills/<skill-name>/SKILL.md`.
We ship the in-plugin form so a single install registers MCP, skill, and
hooks together.
### 1.4. Hooks — `https://antigravity.google/docs/hooks?app=antigravity`
> Hooks allow you to run custom scripts or shell commands at specific
> points during Antigravity's execution loop.
`hooks.json` lives at one of:
- `~/.gemini/config/hooks.json` (global)
- `<workspace>/.agents/hooks.json` (workspace)
- `<plugin-root>/hooks.json` (per-plugin) — what we ship
Top-level schema (verified):
```json
{
"<hook-name>": {
"enabled": true,
"PreToolUse": [{ "matcher": "...", "hooks": [{...}] }],
"PostToolUse": [{ "matcher": "...", "hooks": [{...}] }],
"PreInvocation": [{ ... }],
"PostInvocation": [{ ... }],
"Stop": [{ ... }]
}
}
```
For `PreInvocation` / `PostInvocation` / `Stop`, items are flat handler
objects; the `matcher` wrapper is only used for `PreToolUse` /
`PostToolUse`.
Handler object (verified):
| Field | Required | Description |
|-----------|----------|---------------------------------------------------|
| `type` | No | Currently only `"command"` is supported. Default. |
| `command` | Yes | The shell command to execute. |
| `timeout` | No | Timeout in seconds. Defaults to 30. |
#### STDIN/STDOUT contract (verified)
> Hooks receive input via stdin as JSON and should return output via
> stdout as JSON. Field names use camelCase.
Common stdin fields (every event):
| Field | Type | Description |
|-------------------------|-----------------|-------------------------------------------------------|
| `conversationId` | string | The unique UUID of the active agent conversation. |
| `workspacePaths` | array<string> | Absolute directory paths of the user's workspaces. |
| `transcriptPath` | string | Absolute path to the persistent `transcript.jsonl`. |
| `artifactDirectoryPath` | string | Path to conversation artifacts and screenshots. |
`Stop` event additional fields:
| Field | Type | Description |
|---------------------|---------|----------------------------------------------------------------------------|
| `executionNum` | integer | Sequence number of the execution attempt. |
| `terminationReason` | string | `"model_stop"`, `"max_steps_exceeded"`, `"error"`, etc. |
| `error` | string | Optional error message. |
| `fullyIdle` | boolean | **Required.** True iff all background commands and async tasks are done. |
`Stop` event stdout (verified):
| Field | Type | Description |
|------------|--------|-------------------------------------------------------------------------------------------------------|
| `decision` | string | **Required.** Set to `"continue"` to FORCE the agent to keep running. Anything else allows the stop. |
| `reason` | string | Optional. If `decision == "continue"`, injected as a system message. |
**CRITICAL**: emitting `{"decision": "continue"}` from a save hook would
turn it into an infinite agent-loop trigger. The MemPalace save hook
MUST emit `{}` on every code path. There is an explicit refusal in
`mempal_save_hook_antigravity.sh` to ever print the literal word
`"continue"` from a decision field.
`PreInvocation` event additional fields:
| Field | Type | Description |
|-------------------|---------|----------------------------------------------------------|
| `invocationNum` | integer | Sequence number of the current model invocation. |
| `initialNumSteps` | integer | Number of steps currently in the trajectory. |
`PreInvocation` event stdout (verified):
| Field | Type | Description |
|---------------|----------------|-----------------------------------------------------------------------------------------------|
| `injectSteps` | array<object> | Optional. Steps injected before the model is called. Each step has one of: |
| | | `{ "toolCall": {...} }` / `{ "userMessage": "..." }` / `{ "ephemeralMessage": "..." }` |
We use `ephemeralMessage` for the wake-up injection: the message is
visible to the model on this turn but does not persist to the
transcript, so we do not pollute future model calls with the same
injection.
`PreInvocation` fires before EVERY model invocation, not only at session
start. We gate to `invocationNum == 1` to mimic Cursor's `sessionStart`
semantics — exactly one wake injection per conversation.
### 1.5. Permissions — `https://antigravity.google/docs/permissions`
Permissions are user-side, configured via Allow / Deny / Ask lists.
Plugins do **not** declare permissions in `plugin.json`. The
third-party "antigravity-plugins" community skill at
`~/.gemini/skills/antigravity-plugins/SKILL.md` documents a
`"permissions": [...]` field in `plugin.json`; that field is fabricated
and does not appear in any real Google-shipped plugin (`firebase`,
`google-antigravity-sdk`, `chrome-devtools-plugin`,
`modern-web-guidance-plugin`) inspected at
`~/.gemini/config/plugins/`.
We ship a minimal `plugin.json` of `{"name": "mempalace"}`.
---
## 2. Surfaces shipped
| Surface | What we ship |
|---------------------------|---------------------------------------------------------------------------|
| Plugin manifest | `.antigravity-plugin/plugin.json` — minimal, verified shape |
| MCP auto-registration | `.antigravity-plugin/mcp_config.json` — registers `mempalace-mcp` stdio |
| Skill | `.antigravity-plugin/skills/mempalace/SKILL.md` — real file, frontmatter |
| `Stop` hook | `hooks/antigravity/mempal_save_hook_antigravity.sh` — counter + auto-mine |
| `PreInvocation` hook | `hooks/antigravity/mempal_wake_hook_antigravity.sh` — wake injection |
| Installer | `hooks/antigravity/install.sh` — idempotent, basename-match uninstall |
| User-facing docs | `website/guide/antigravity.md` + sidebar wiring |
| Examples | `examples/antigravity/{hooks.json,mcp_config.json,README.md}` |
| Tests | 3 test files mirroring the cursor blueprint |
---
## 3. Surfaces deliberately not shipped
### 3.1. `PreCompact` equivalent — NOT SHIPPED
Antigravity's external `hooks.json` does **not** expose a context-compaction event. The Python SDK has an in-process `@hooks.on_compaction`
decorator (see `~/.gemini/config/plugins/google-antigravity-sdk/examples/getting_started/hooks.md`),
but that fires inside a Python `LocalAgentConfig`-built agent, not the
IDE itself. There is no way to subscribe to compaction from a
plugin's `hooks.json`.
UX consequence: long conversations can auto-compact without a save
checkpoint. The `Stop` hook still catches the conversation when the
user actually ends the turn, so the worst case is that some mid-turn
state is lost on auto-compaction. Verbatim transcript ingestion via
the `Stop` path covers the long-term recall use case.
### 3.2. Slash-commands / `commands/` — NOT SHIPPED
Antigravity has no `commands/` plugin component. The Cursor and Codex
integrations both ship five quick-reference commands
(`mempalace-help`, `-init`, `-mine`, `-search`, `-status`) that point
at `mempalace instructions <cmd>`. Those have been folded into the
`SKILL.md` `## Common operations` section so the agent gets the same
quick-reference content via Antigravity's progressive-disclosure skill
loading. No new files, no rule-noise, same discoverability.
### 3.3. `rules/` — NOT SHIPPED
`rules/<name>.md` files are evaluated as constraints on the agent's
behavior. Shipping rules from MemPalace risks colliding with the
user's existing project rules (e.g. `.agents/rules/*.md` files the
user has already authored). Users who want strict MemPalace-related
rules can drop them into their own `<workspace>/.agents/rules/`
directory; we do not impose them.
### 3.4. Workspace-level `.agents/plugins/mempalace/` install — NOT SHIPPED BY DEFAULT
The installer writes to the global location at
`~/.gemini/config/plugins/mempalace/`. Workspace-scoped installs are
documented in `hooks/antigravity/README.md` for users who want to
limit MemPalace to one workspace; they can `cp -r .antigravity-plugin
<workspace>/.agents/plugins/mempalace`. We do not install there
automatically because the canonical UX is global.
### 3.5. `permissions` field in `plugin.json` — NOT SHIPPED
Antigravity permissions are user-side (`Allow` / `Deny` / `Ask`
lists). Plugin manifests do not declare permissions. The
`"permissions": [...]` field documented in the third-party
"antigravity-plugins" community skill is fabricated; no
Google-shipped plugin uses it.
### 3.6. `PreToolUse` / `PostToolUse` hooks — NOT SHIPPED
These would let MemPalace observe every tool call (e.g.
auto-extract entities after each `write_to_file`). Out of scope
for v1; the hook surface is real and could be added in a future
PR if there is demand. Documenting the omission here so a future
contributor doesn't conclude the hooks weren't supported.
---
## 4. Cross-checks against the user's running Antigravity
The user (`robs@`) has Antigravity 2.0 installed. The following live
artifacts on this machine corroborate the published docs:
| Path | Confirms |
|------------------------------------------------------------------------|----------------------------------------------------------------|
| `~/.gemini/antigravity/mcp_config.json` | MCP config path + standard `mcpServers` shape |
| `~/.gemini/antigravity/skills/<skill-name>/SKILL.md` (multiple) | Global skill discovery path |
| `~/.gemini/config/plugins/firebase/plugin.json` | Real `plugin.json` shape (no `permissions` field) |
| `~/.gemini/config/plugins/chrome-devtools-plugin/skills/.../SKILL.md` | In-plugin skill discovery `<plugin>/skills/<name>/SKILL.md` |
| `~/.gemini/config/plugins/google-antigravity-sdk/examples/.../hooks.md` | SDK-side compaction hook is in-process Python only |
| Existing `mempalace` entry in `mcp_config.json` | `mempalace-mcp` already running and discoverable |
---
## 5. Reference URLs (all 2026-05-27)
- `https://antigravity.google/docs/plugins`
- `https://antigravity.google/docs/hooks?app=antigravity`
- `https://antigravity.google/docs/skills`
- `https://antigravity.google/docs/mcp`
- `https://antigravity.google/docs/permissions`
- `https://antigravity.google/docs/subagents`
- `https://antigravity.google/blog/introducing-google-antigravity-sdk`
+172
View File
@@ -0,0 +1,172 @@
# MemPalace — Antigravity hook scripts
Lifecycle hooks for the [Antigravity IDE](https://antigravity.google/).
This is the third sibling of the Claude Code and Codex integrations
(see `hooks/mempal_save_hook.sh` and `.codex-plugin/hooks/`). The
overall shape is the same — a Stop event triggers a background save,
a startup-time event injects memory into the agent — but the wire
format and STDOUT contract are Antigravity-specific (see
[STDIN_SHAPE.md](STDIN_SHAPE.md)).
## Quick start
From the repo root:
```bash
bash hooks/antigravity/install.sh
```
This installs the plugin to `~/.gemini/config/plugins/mempalace/`.
Restart Antigravity and the MCP server, skill, and hooks all register
automatically.
To dry-run first:
```bash
bash hooks/antigravity/install.sh --dry-run
```
To uninstall:
```bash
bash hooks/antigravity/install.sh --uninstall
```
## What gets installed
```
~/.gemini/config/plugins/mempalace/
├── plugin.json # marker manifest
├── mcp_config.json # registers mempalace-mcp
├── hooks.json # rendered from hooks.json.tmpl
├── README.md
├── skills/
│ └── mempalace/
│ └── SKILL.md
└── hooks/
├── lib/
│ └── common.sh
├── mempal_save_hook_antigravity.sh # Stop event handler
└── mempal_wake_hook_antigravity.sh # PreInvocation handler
```
`hooks.json` carries absolute paths to the two hook scripts (resolved
from `__PLUGIN_DIR__` at install time).
## What the hooks do
### `mempal_save_hook_antigravity.sh` (Stop event)
Fires every time the agent's execution loop terminates. Increments a
per-conversation counter; every `MEMPAL_SAVE_INTERVAL` fires (default
15), spawns `mempalace mine <transcript-dir> --mode convos --wing
<inferred>` in the background. The hook itself returns `{}` to stdout
in under a few milliseconds — the actual mining runs detached and
does not block the user.
Defers when:
- `fullyIdle == false` (background tasks still running)
- `terminationReason == "error"` (transcript may be corrupt)
- A previous save for this conversation is still running
- Any kill switch is set
### `mempal_wake_hook_antigravity.sh` (PreInvocation event, gated)
Fires before every model invocation. Gated to `invocationNum == 1`
(first invocation of the conversation only) — beyond that we'd be
re-injecting on every turn. Calls `mempalace wake-up --wing <inferred>`
with a 500ms hard timeout and emits the verbatim output as an
`ephemeralMessage` so the agent sees relevant memory on its first
response without polluting the persistent transcript.
Skips when:
- `invocationNum != 1`
- Already woke this conversation (atomic `mkdir` loop guard)
- `mempalace wake-up` exits non-zero, times out, or produces empty output
- Any kill switch is set
## Kill switches
Any one of these disables both hooks (silent passthrough, exit 0):
| Knob | Value |
|------------------------------------------|--------------------------------|
| `MEMPAL_DISABLE_HOOK` | `1`, `true`, `yes` |
| `MEMPALACE_HOOKS_AUTO_SAVE` | `false`, `0`, `no` |
| `~/.mempalace/config.json` | `{"hooks": {"auto_save": false}}` |
| Removing `~/.mempalace/` entirely | (palace nuke) |
## Workspace-scoped install (advanced)
If you want MemPalace to load only inside a specific workspace,
manually copy the rendered plugin into your workspace's `.agents/plugins/`:
```bash
bash hooks/antigravity/install.sh --install-dir /tmp/render-stage
mkdir -p <workspace>/.agents/plugins/
cp -r /tmp/render-stage <workspace>/.agents/plugins/mempalace
rm -rf /tmp/render-stage
```
The global install at `~/.gemini/config/plugins/mempalace/` is the
canonical UX and what we recommend.
## Troubleshooting
### Hooks aren't firing
1. Confirm Antigravity sees the plugin: open the IDE, navigate to the
Customizations page; `mempalace` should appear in the global plugins
list.
2. Check `~/.mempalace/hook_state/antigravity_hook.log` — every fire
logs a line. No log lines = the hook is not being invoked.
3. Verify `mempalace-mcp` is on `$PATH`: `mempalace-mcp --version`.
4. Inspect the rendered `hooks.json` paths point at executable files:
`bash -n ~/.gemini/config/plugins/mempalace/hooks/*.sh`.
### Save fires but no mining happens
1. Look for the most recent `[event=stop]` lines in
`antigravity_hook.log` — `count` and `interval` should both be
visible. Mining only triggers when `count % interval == 0`.
2. Ensure a Python that can import `mempalace` is reachable. The hook
runs `"$MEMPAL_PYTHON_BIN" -m mempalace`, where `MEMPAL_PYTHON_BIN`
is resolved (in order) from `$MEMPAL_PYTHON`, the
`mempalace-mcp` / `mempalace` console-script shebang on `$PATH`,
then `python3`. A failed probe logs:
```
ERROR: mempalace is not runnable via <python> -m mempalace; install mempalace or set MEMPAL_PYTHON
```
On a GUI-launched Antigravity the harness `PATH` may differ from
your shell `PATH`; if the shebang heuristic can't find the right
interpreter, export `MEMPAL_PYTHON=/abs/path/python` (e.g.
`"$(uv tool dir)/mempalace/bin/python"`) and restart.
### Wake injection isn't appearing
1. The wake hook only injects on `invocationNum == 1`. Subsequent
invocations are gated.
2. The atomic `mkdir` marker
`~/.mempalace/hook_state/antigravity_woke_<conversationId>` exists
after a successful injection. Remove it to re-inject (rare).
3. `mempalace wake-up --wing <inferred>` may be returning empty output
if the wing doesn't exist yet. Run `mempalace status` to verify
wing presence.
## See also
- [INVESTIGATION.md](INVESTIGATION.md) — every Antigravity surface we
investigated, with verbatim quotes and source URLs.
- [STDIN_SHAPE.md](STDIN_SHAPE.md) — the exact wire format
Antigravity uses, with worked examples.
- [../mempal_save_hook.sh](../mempal_save_hook.sh) — Claude Code
equivalent.
- [../../.codex-plugin/hooks/](../../.codex-plugin/hooks/) — Codex
equivalent.
- [../../website/guide/antigravity.md](../../website/guide/antigravity.md)
— full user-facing guide.
+206
View File
@@ -0,0 +1,206 @@
# Antigravity hook STDIN / STDOUT contract
This file documents the exact wire format the Antigravity IDE uses
when invoking the MemPalace hook scripts. All fields are verbatim
from Google's official Antigravity hooks documentation
(`https://antigravity.google/docs/hooks?app=antigravity`, accessed
2026-05-27). See [INVESTIGATION.md](INVESTIGATION.md) for the
provenance audit.
## Wire format
Hooks receive **JSON on stdin** and must emit **JSON on stdout**.
Field names are **camelCase**.
Hook execution timeout defaults to 30 seconds. The MemPalace plugin
sets the Stop hook timeout to 30s and the PreInvocation hook timeout
to 5s in the rendered `hooks.json`.
## Common stdin fields (every event)
| Field | Type | Notes |
|-------------------------|-----------------|--------------------------------------------------------|
| `conversationId` | string | UUID of the active agent conversation. |
| `workspacePaths` | array<string> | Absolute workspace dirs. **First element is canonical**. |
| `transcriptPath` | string | Absolute path to `transcript.jsonl`. |
| `artifactDirectoryPath` | string | Path to conversation artifacts and screenshots. |
## Stop event
### Stdin (additional fields)
| Field | Type | Notes |
|---------------------|---------|--------------------------------------------------------------------------------------|
| `executionNum` | integer | Sequence number of the execution attempt for this conversation. |
| `terminationReason` | string | `"model_stop"`, `"max_steps_exceeded"`, `"error"`, etc. |
| `error` | string | Optional. Set when termination was caused by a system error. |
| `fullyIdle` | boolean | **Required.** True iff all background commands and async tasks have completed. |
### Stdout
| Field | Type | Notes |
|------------|--------|--------------------------------------------------------------------------------------------------------|
| `decision` | string | If `"continue"`, **forces** the agent to keep running. Anything else allows the stop. |
| `reason` | string | Optional. If `decision == "continue"`, injected as a system message into the conversation. |
**MemPalace policy**: the save hook ALWAYS emits `{}` and exits 0. It
NEVER emits `{"decision": "continue"}` — that would force an infinite
agent loop. There is an explicit refusal in
`mempal_save_hook_antigravity.sh` to ever construct a stdout JSON
object containing the literal word `"continue"` in a decision field.
### MemPalace gating
The save hook short-circuits with `{}` (no save triggered) when ANY
of the following hold:
1. `MEMPAL_DISABLE_HOOK=1` (or `true`/`yes`) is set.
2. `MEMPALACE_HOOKS_AUTO_SAVE=false` (or `0`/`no`) is set.
3. `~/.mempalace/config.json` has `hooks.auto_save: false`.
4. `~/.mempalace/` directory does not exist (user nuked the palace).
5. Stdin is malformed or empty (sentinel-guarded parse failure).
6. `fullyIdle == false` (background tasks still running; defer save).
7. `terminationReason == "error"` (transcript may be corrupt).
8. `transcriptPath` validation fails (not a `.json`/`.jsonl`, or `..` traversal).
9. The transcript file does not exist on disk.
10. The save counter has not yet hit `count % MEMPAL_SAVE_INTERVAL == 0`.
11. A pending save is still running for this conversation (less than 1 hour old).
12. The `mempalace` CLI is not on `$PATH`.
When the modulo gate is hit and validation passes, the hook spawns
`mempalace mine <transcript-dir> --mode convos --wing <inferred>` in
the background and returns `{}` immediately.
## PreInvocation event
### Stdin (additional fields)
| Field | Type | Notes |
|-------------------|---------|------------------------------------------------------------------|
| `invocationNum` | integer | Sequence number of the current model invocation (1-based). |
| `initialNumSteps` | integer | Number of steps currently in the trajectory. |
### Stdout
| Field | Type | Notes |
|---------------|----------------|--------------------------------------------------------------------------------------------------------|
| `injectSteps` | array<object> | Optional. Steps to inject before the model is called. Each step has one of: `{"toolCall": {...}}`, `{"userMessage": "..."}`, `{"ephemeralMessage": "..."}` |
The `ephemeralMessage` form is what the MemPalace wake hook emits — it
delivers the wake-up text to the model on this turn but does not
persist into the transcript, so subsequent invocations don't see a
duplicate.
### MemPalace gating
The wake hook short-circuits with `{}` (no injection) when ANY of:
1. Any kill switch trips (same five conditions as the save hook).
2. `invocationNum != 1` — we only inject on the first model call of
each conversation, mimicking Cursor's `sessionStart` semantics.
3. The atomic `mkdir`-based loop guard is already taken (this
conversation already received a wake injection).
4. `mempalace wake-up --wing <inferred>` exits non-zero, times out
(500ms hard cap), or produces empty output.
When the gates pass and `mempalace wake-up` returns text, the hook
emits:
```json
{
"injectSteps": [
{ "ephemeralMessage": "<verbatim wake-up output>" }
]
}
```
The wake hook NEVER emits a `decision` field — that field belongs to
the Stop event. There is a final guard against any stdout that
contains a `decision` key.
## Worked example: Stop event
### Input
```json
{
"executionNum": 1,
"terminationReason": "model_stop",
"error": "",
"fullyIdle": true,
"conversationId": "ec33ebf9-0cba-4100-8142-c61503f6c587",
"workspacePaths": ["/home/me/projects/mempalace"],
"transcriptPath": "/home/me/projects/mempalace/.gemini/jetski/transcript.jsonl",
"artifactDirectoryPath": "/home/me/projects/mempalace/.gemini/jetski/artifacts"
}
```
### Output (always)
```json
{}
```
(Side effects: counter `~/.mempalace/hook_state/antigravity_save_count_<id>` is
incremented; if the modulo gate fires, a background `mempalace mine`
subprocess is spawned with the transcript directory and the inferred
wing `wing_mempalace`.)
## Worked example: PreInvocation, first invocation
### Input
```json
{
"invocationNum": 1,
"initialNumSteps": 0,
"conversationId": "ec33ebf9-0cba-4100-8142-c61503f6c587",
"workspacePaths": ["/home/me/projects/mempalace"],
"transcriptPath": "/home/me/projects/mempalace/.gemini/jetski/transcript.jsonl",
"artifactDirectoryPath": "/home/me/projects/mempalace/.gemini/jetski/artifacts"
}
```
### Output (when the palace has memory for `wing_mempalace`)
```json
{
"injectSteps": [
{
"ephemeralMessage": "<exact verbatim text from `mempalace wake-up --wing wing_mempalace`>"
}
]
}
```
### Output (when invocationNum != 1, or any gate trips)
```json
{}
```
## State files
All hook state lives under `~/.mempalace/hook_state/` (overridable
via `$MEMPAL_STATE_DIR`) and is namespaced with the `antigravity_`
prefix to coexist with Claude Code, Cursor, and Codex hook state in
the same directory.
| File | Purpose |
|-----------------------------------------------|-----------------------------------------------|
| `antigravity_hook.log` | All hook activity, ISO8601Z timestamps. |
| `antigravity_save_count_<conversationId>` | Per-conversation Stop counter. |
| `antigravity_pending_<conversationId>` | Marker file for in-flight save subprocess. |
| `antigravity_woke_<conversationId>` (dir) | Atomic mkdir marker for wake injection. |
| `antigravity_last_input.log` | 4 KB cap, mode 0600, set on parse failure. |
| `antigravity_last_python_err.log` | Python stderr from the JSON parser, mode 0600.|
## Environment variables
| Variable | Default | Purpose |
|--------------------------------|----------------------|-----------------------------------------------------------|
| `MEMPAL_PYTHON` | `$(command -v python3)` | Override the Python interpreter used by the hooks. |
| `MEMPAL_STATE_DIR` | `~/.mempalace/hook_state` | Override the hook state directory. |
| `MEMPAL_SAVE_INTERVAL` | `15` | Save every Nth Stop fire. Floored to >= 1 (no /0). |
| `MEMPAL_DISABLE_HOOK` | unset | Set to `1` / `true` / `yes` to disable both hooks. |
| `MEMPALACE_HOOKS_AUTO_SAVE` | unset | Set to `false` / `0` / `no` to disable both hooks. |
+357
View File
@@ -0,0 +1,357 @@
#!/bin/bash
# MEMPALACE ANTIGRAVITY INSTALLER
#
# Idempotent installer for the Antigravity plugin. Copies
# .antigravity-plugin/* and hooks/antigravity/{lib,*.sh} into the
# install directory (default ~/.gemini/config/plugins/mempalace/),
# renders hooks.json.tmpl into hooks.json with absolute paths, and
# leaves the result in a state Antigravity will discover on next
# launch.
#
# === Usage ===
#
# bash hooks/antigravity/install.sh # install with defaults
# bash hooks/antigravity/install.sh --dry-run # show what would happen
# bash hooks/antigravity/install.sh --uninstall # remove plugin
# bash hooks/antigravity/install.sh --install-dir <p> # custom install dir
# bash hooks/antigravity/install.sh --log-level debug # noisier output
#
# === Idempotency ===
#
# Re-running the installer produces a byte-identical install dir.
# Files are only written when their content differs from what is
# already on disk (cmp gate). The user's ~/.gemini/config/plugins/
# directory is never touched outside the mempalace/ subdirectory.
#
# === Uninstall safety ===
#
# Uninstall removes the install dir entirely IFF it is the
# mempalace/ plugin directory. We match by basename of the install
# dir, never by substring search, so a user who has a sibling plugin
# at ~/.gemini/config/plugins/mempalace-foo/ is unaffected.
#
# === set -e ===
#
# Installer can use `set -e` (constraint #2 only forbids it in the
# hook scripts themselves). On any error we exit non-zero so a CI run
# fails loudly.
set -e
set -u
# ── Repo root resolution ─────────────────────────────────────────────
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)"
REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd -P)"
PLUGIN_SRC="$REPO_ROOT/.antigravity-plugin"
HOOKS_SRC="$REPO_ROOT/hooks/antigravity"
# ── Defaults ─────────────────────────────────────────────────────────
INSTALL_DIR_DEFAULT="$HOME/.gemini/config/plugins/mempalace"
INSTALL_DIR=""
DRY_RUN=0
UNINSTALL=0
LOG_LEVEL="info"
# ── Args ─────────────────────────────────────────────────────────────
print_usage() {
cat <<'USAGE'
Usage: install.sh [--install-dir DIR] [--dry-run] [--uninstall] [--log-level LEVEL]
Options:
--install-dir DIR Plugin install directory.
Default: ~/.gemini/config/plugins/mempalace
--dry-run Show what would happen without writing anything.
--uninstall Remove the installed plugin.
--log-level LEVEL debug | info | warn | error. Default: info.
-h, --help Show this help.
USAGE
}
while [ $# -gt 0 ]; do
case "$1" in
--install-dir)
INSTALL_DIR="${2:-}"
shift 2
;;
--install-dir=*)
INSTALL_DIR="${1#*=}"
shift
;;
--dry-run)
DRY_RUN=1
shift
;;
--uninstall)
UNINSTALL=1
shift
;;
--log-level)
LOG_LEVEL="${2:-info}"
shift 2
;;
--log-level=*)
LOG_LEVEL="${1#*=}"
shift
;;
-h|--help)
print_usage
exit 0
;;
*)
echo "ERROR: unknown argument: $1" >&2
print_usage >&2
exit 2
;;
esac
done
if [ -z "$INSTALL_DIR" ]; then
INSTALL_DIR="$INSTALL_DIR_DEFAULT"
fi
# ── Absolutize the install dir ───────────────────────────────────────
#
# The cursor PR review caught that a relative --install-dir would get
# baked into hooks.json verbatim, leaving paths like
# `./plugins/.../mempal_save_hook_antigravity.sh` that Antigravity
# can't resolve at runtime. Absolutize before writing anything.
mempal_absolutize() {
local p="$1"
case "$p" in
/*) printf '%s' "$p" ;;
~*) printf '%s' "${p/#\~/$HOME}" ;;
*)
# Resolve relative to the user's $PWD at invocation time.
# We never `cd` in the main shell of this installer, so
# $PWD is already the user's invocation directory — no
# subshell cd dance needed.
local base="${PWD}"
printf '%s/%s' "$base" "$p"
;;
esac
}
INSTALL_DIR="$(mempal_absolutize "$INSTALL_DIR")"
# Squash any `//` or `./` or `name/..` artefacts using Python's
# os.path.normpath; falls back to the raw value if Python is missing
# (which would be very unusual on macOS / Linux).
if command -v python3 >/dev/null 2>&1; then
INSTALL_DIR="$(python3 -c 'import os,sys; print(os.path.normpath(sys.argv[1]))' "$INSTALL_DIR")"
fi
# ── Logging ──────────────────────────────────────────────────────────
log() {
local lvl="$1"; shift
local msg="$*"
case "$lvl" in
debug)
[ "$LOG_LEVEL" = "debug" ] && echo "[install] DEBUG: $msg"
return 0
;;
info)
case "$LOG_LEVEL" in
debug|info) echo "[install] $msg" ;;
esac
;;
warn)
case "$LOG_LEVEL" in
debug|info|warn) echo "[install] WARN: $msg" >&2 ;;
esac
;;
error)
echo "[install] ERROR: $msg" >&2
;;
esac
}
# ── Action helpers (dry-run aware) ──────────────────────────────────
run() {
if [ "$DRY_RUN" -eq 1 ]; then
echo "[install] DRY-RUN: $*"
return 0
fi
"$@"
}
# ── Render template ──────────────────────────────────────────────────
#
# Substitutes __PLUGIN_DIR__ in $src into $dst with $INSTALL_DIR.
# Emits the rendered file to a temp path first, then promotes it iff
# the content differs from what's already at $dst. The cmp gate is
# what makes the installer idempotent: a no-op re-run produces no
# disk writes (and the test suite asserts byte-equality).
render_template() {
local src="$1"
local dst="$2"
if [ ! -f "$src" ]; then
log error "template not found: $src"
return 1
fi
local tmp
tmp="$(mktemp "${TMPDIR:-/tmp}/mempal_agy_render.XXXXXX")"
# Python over awk/sed — INSTALL_DIR may legitimately contain
# characters (spaces, colons) that would require careful escaping
# in a sed s/// replacement. Python read+replace handles all of
# them uniformly.
python3 -c "
import sys
src, dst, install_dir = sys.argv[1:4]
with open(src, 'r') as f:
body = f.read()
body = body.replace('__PLUGIN_DIR__', install_dir)
with open(dst, 'w') as f:
f.write(body)
" "$src" "$tmp" "$INSTALL_DIR"
if [ -f "$dst" ] && cmp -s "$tmp" "$dst"; then
rm -f "$tmp"
log debug "unchanged: $dst"
return 0
fi
if [ "$DRY_RUN" -eq 1 ]; then
echo "[install] DRY-RUN: would render $src -> $dst"
rm -f "$tmp"
return 0
fi
mv "$tmp" "$dst"
log info "wrote: $dst"
}
# ── copy_file: cmp-gated copy that preserves mode ────────────────────
copy_file() {
local src="$1"
local dst="$2"
if [ ! -f "$src" ]; then
log error "missing source file: $src"
return 1
fi
if [ -f "$dst" ] && cmp -s "$src" "$dst"; then
log debug "unchanged: $dst"
return 0
fi
if [ "$DRY_RUN" -eq 1 ]; then
echo "[install] DRY-RUN: would copy $src -> $dst"
return 0
fi
mkdir -p "$(dirname "$dst")"
cp "$src" "$dst"
log info "wrote: $dst"
}
# ── Uninstall path ───────────────────────────────────────────────────
#
# We DO NOT remove the install dir by string-substring match against
# the path. We require the install dir's basename to be exactly
# "mempalace" — that way an unrelated sibling like
# ~/.gemini/config/plugins/mempalace-foo/ is left alone, and a
# malformed --install-dir like ~ or / cannot wipe the user's home.
do_uninstall() {
local base
base="$(basename "$INSTALL_DIR")"
if [ "$base" != "mempalace" ]; then
log error "refusing to uninstall: install dir basename is '$base', expected 'mempalace'"
log error "(safety guard: prevents accidental wipe of unrelated directories)"
return 1
fi
if [ ! -d "$INSTALL_DIR" ]; then
log info "nothing to uninstall: $INSTALL_DIR does not exist"
return 0
fi
# Verify the dir LOOKS like our plugin before removing — a
# plugin.json file with our marker is the proof.
if [ ! -f "$INSTALL_DIR/plugin.json" ]; then
log error "refusing to uninstall: $INSTALL_DIR has no plugin.json"
return 1
fi
if ! grep -q '"name"[[:space:]]*:[[:space:]]*"mempalace"' "$INSTALL_DIR/plugin.json" 2>/dev/null; then
log error "refusing to uninstall: $INSTALL_DIR/plugin.json is not a mempalace plugin"
return 1
fi
if [ "$DRY_RUN" -eq 1 ]; then
echo "[install] DRY-RUN: would rm -rf $INSTALL_DIR"
return 0
fi
rm -rf "$INSTALL_DIR"
log info "uninstalled: $INSTALL_DIR"
}
if [ "$UNINSTALL" -eq 1 ]; then
do_uninstall
exit 0
fi
# ── Pre-install sanity ───────────────────────────────────────────────
if [ ! -d "$PLUGIN_SRC" ]; then
log error "missing source: $PLUGIN_SRC"
exit 1
fi
if [ ! -d "$HOOKS_SRC" ]; then
log error "missing source: $HOOKS_SRC"
exit 1
fi
# Soft-check that mempalace-mcp is on PATH; warn but do not fail.
if ! command -v mempalace-mcp >/dev/null 2>&1; then
log warn "mempalace-mcp is not on PATH; the MCP server will fail to start until it is."
log warn " fix: 'uv tool install mempalace' or 'pip install mempalace'"
fi
# Soft-check ~/.gemini exists; if missing, Antigravity isn't installed.
if [ ! -d "$HOME/.gemini" ]; then
log warn "$HOME/.gemini not found — Antigravity is probably not installed yet."
log warn " the install will still proceed; Antigravity will pick up the plugin on first launch."
fi
log info "install dir: $INSTALL_DIR"
# ── Install: directories ─────────────────────────────────────────────
run mkdir -p "$INSTALL_DIR" \
"$INSTALL_DIR/skills/mempalace" \
"$INSTALL_DIR/skills/mempalace-recall" \
"$INSTALL_DIR/rules" \
"$INSTALL_DIR/hooks" \
"$INSTALL_DIR/hooks/lib"
# ── Install: plugin metadata ─────────────────────────────────────────
copy_file "$PLUGIN_SRC/plugin.json" "$INSTALL_DIR/plugin.json"
copy_file "$PLUGIN_SRC/mcp_config.json" "$INSTALL_DIR/mcp_config.json"
copy_file "$PLUGIN_SRC/README.md" "$INSTALL_DIR/README.md"
# ── Install: skills (real files, no symlinks at the discovery path) ──
copy_file "$PLUGIN_SRC/skills/mempalace/SKILL.md" \
"$INSTALL_DIR/skills/mempalace/SKILL.md"
copy_file "$PLUGIN_SRC/skills/mempalace-recall/SKILL.md" \
"$INSTALL_DIR/skills/mempalace-recall/SKILL.md"
# ── Install: optional recall rule ────────────────────────────────────
#
# Antigravity discovers markdown rules under the plugin's rules/
# directory. This one is recall-only and intentionally lightweight —
# it complements the mempalace-recall skill. Shipping it as a plugin
# rule (not an always-on global rule) keeps it scoped to recall-
# relevant turns, honouring MemPalace's "memory should feel instant"
# budget.
copy_file "$PLUGIN_SRC/rules/mempalace-recall.md" \
"$INSTALL_DIR/rules/mempalace-recall.md"
# ── Install: hooks ───────────────────────────────────────────────────
copy_file "$HOOKS_SRC/lib/common.sh" "$INSTALL_DIR/hooks/lib/common.sh"
copy_file "$HOOKS_SRC/mempal_save_hook_antigravity.sh" "$INSTALL_DIR/hooks/mempal_save_hook_antigravity.sh"
copy_file "$HOOKS_SRC/mempal_wake_hook_antigravity.sh" "$INSTALL_DIR/hooks/mempal_wake_hook_antigravity.sh"
# Ensure hook scripts are executable on the install side. cp preserves
# mode but a fresh git clone from a tarball might not — chmod is
# defensive, idempotent, and bash 3.2 safe.
if [ "$DRY_RUN" -ne 1 ]; then
chmod 755 "$INSTALL_DIR/hooks/mempal_save_hook_antigravity.sh" 2>/dev/null || true
chmod 755 "$INSTALL_DIR/hooks/mempal_wake_hook_antigravity.sh" 2>/dev/null || true
fi
# ── Install: render hooks.json from template ─────────────────────────
render_template "$PLUGIN_SRC/hooks.json.tmpl" "$INSTALL_DIR/hooks.json"
# ── Done ─────────────────────────────────────────────────────────────
if [ "$DRY_RUN" -eq 1 ]; then
log info "DRY-RUN complete; no files written."
else
log info "install complete: $INSTALL_DIR"
log info "restart Antigravity to load the plugin."
fi
+498
View File
@@ -0,0 +1,498 @@
# shellcheck shell=bash
# MEMPALACE ANTIGRAVITY HOOK — shared helpers
#
# Sourced by the two Antigravity hook scripts:
# * mempal_save_hook_antigravity.sh (Stop event)
# * mempal_wake_hook_antigravity.sh (PreInvocation event, gated to invocationNum==1)
#
# Mirrors the conventions of the existing Claude Code hook scripts
# (hooks/mempal_save_hook.sh, hooks/mempal_precompact_hook.sh):
#
# * STATE_DIR layout under ~/.mempalace/hook_state/
# * MEMPAL_PYTHON resolution order (override -> $PATH -> bare python3)
# * MEMPALACE_HOOKS_AUTO_SAVE=false kill switch (config.json fallback)
# * sentinel-guarded Python parser via `sed -n 'Np'` (bash 3.2 safe)
# * fail-open on internal errors: emit valid JSON and log, never crash
# the hook host
#
# Antigravity-specific contract differences from Claude / Cursor:
#
# * Antigravity stdin uses camelCase (transcriptPath, conversationId,
# workspacePaths, executionNum, terminationReason, fullyIdle,
# invocationNum, initialNumSteps), not the snake_case Claude Code
# format (session_id, transcript_path, stop_hook_active).
# * Antigravity stdout for Stop event MUST be {} on every success path
# because { "decision": "continue" } would force the agent into an
# infinite re-execution loop. The save hook explicitly refuses to
# ever emit the "continue" decision.
# * Antigravity stdout for PreInvocation can carry an "injectSteps"
# array of { "ephemeralMessage": "..." } objects to inject memory
# into the agent's first turn.
#
# This file is sourced, not executed, so it intentionally has no
# shebang. The `# shellcheck shell=bash` directive above tells
# shellcheck to treat it as bash when run standalone.
# bash 3.2.57 (the macOS default) is the lower bound. Do not use
# `mapfile`, `readarray`, `declare -A`, or `${var^^}` — none of those
# exist in 3.2. Use `sed -n 'Np'` for line extraction and case-folding
# via `tr` instead.
# ── State directory + log path ────────────────────────────────────────
#
# Honour MEMPAL_STATE_DIR while keeping the default identical to the
# Claude Code hooks so a user running both keeps a single state directory
# (constraint #7 in the integration brief).
MEMPAL_STATE_DIR="${MEMPAL_STATE_DIR:-$HOME/.mempalace/hook_state}"
mkdir -p "$MEMPAL_STATE_DIR" 2>/dev/null
MEMPAL_AGY_LOG="$MEMPAL_STATE_DIR/antigravity_hook.log"
# ── Python interpreter resolution ─────────────────────────────────────
#
# The hooks run mempalace as `"$MEMPAL_PYTHON_BIN" -m mempalace`, so the
# resolved interpreter MUST be one that has the mempalace package
# importable. The single most common install path —
# `uv tool install mempalace` (and `pipx install`) — puts the
# `mempalace` / `mempalace-mcp` *console scripts* on PATH inside an
# ISOLATED environment whose interpreter is NOT the system `python3`.
# So naively resolving `command -v python3` lands on a system Python
# that can't import mempalace, the `-m mempalace` probe fails, and
# mining silently never fires. (This bit a real user on PR #1633.)
#
# Resolution order — first hit wins:
# 1. $MEMPAL_PYTHON — explicit operator override
# 2. shebang of the mempalace-mcp / mempalace console script on PATH
# — pip/uv write these with an absolute-path shebang pointing at
# the exact interpreter that owns the package. This is the SAME
# console script mcp_config.json launches, so if the MCP server
# can start, the hooks resolve a working interpreter too — no
# MEMPAL_PYTHON needed for the common install paths.
# 3. $(command -v python3) — an activated dev venv /
# editable install where python3 itself owns the package
# 4. bare "python3" — last-resort fallback
#
# Steps 2-4 are pure string parsing + stat (no Python subprocess), so
# resolution stays cheap enough to run at source time on every hook
# fire, including gated-out / kill-switched ones. We deliberately do
# NOT run an `import mempalace` probe here: building that import pays
# the chromadb/onnx cold-start cost, which a recent perf fix
# (df295bd) moved OFF the hook foreground on purpose. The downstream
# `-m mempalace --version` probe (backgrounded in the save hook,
# subprocessed in the wake hook) is the safety net that catches a
# shebang interpreter whose package is genuinely broken.
mempal_resolve_python() {
# 1. Explicit override always wins.
local p="${MEMPAL_PYTHON:-}"
if [ -n "$p" ] && [ -x "$p" ]; then
printf '%s' "$p"
return 0
fi
# 2. Derive the interpreter from a mempalace console-script shebang.
local script_path shebang interp
for script_path in mempalace-mcp mempalace; do
script_path="$(command -v "$script_path" 2>/dev/null || true)"
[ -n "$script_path" ] || continue
[ -r "$script_path" ] || continue
shebang="$(sed -n '1p' "$script_path" 2>/dev/null)"
case "$shebang" in
'#!'*)
interp="${shebang#\#!}" # drop the leading '#!'
interp="${interp%$'\r'}" # strip a trailing CR (CRLF files)
interp="${interp# }" # drop one leading space
interp="${interp%% *}" # first whitespace-delimited token
# Guard against `#!/usr/bin/env python` wrappers: only
# accept a token whose basename looks like a Python
# interpreter and is executable. An `env`-style shebang
# yields `/usr/bin/env` here, which we skip.
case "${interp##*/}" in
python*)
if [ -x "$interp" ]; then
printf '%s' "$interp"
return 0
fi
;;
esac
;;
esac
done
# 3. First python3 on PATH.
p="$(command -v python3 2>/dev/null || true)"
if [ -n "$p" ]; then
printf '%s' "$p"
return 0
fi
# 4. Last-resort bare name.
printf '%s' "python3"
}
MEMPAL_PYTHON_BIN="$(mempal_resolve_python)"
# ── Logging ───────────────────────────────────────────────────────────
#
# ISO8601Z timestamps are greppable across timezones.
mempal_log() {
local event="${1:-?}"
local conv="${2:-unknown}"
local msg="${3:-}"
local ts
ts="$(date -u '+%Y-%m-%dT%H:%M:%SZ')"
printf '[%s] [event=%s] [conv=%s] %s\n' "$ts" "$event" "$conv" "$msg" \
>> "$MEMPAL_AGY_LOG" 2>/dev/null
}
# ── Kill switch ───────────────────────────────────────────────────────
#
# Disabled if ANY of:
# * MEMPAL_DISABLE_HOOK is a truthy string
# * MEMPALACE_HOOKS_AUTO_SAVE is false/0/no
# * ~/.mempalace/config.json sets hooks.auto_save: false
# * ~/.mempalace/ directory does not exist (user nuked the palace)
#
# Returns 0 (kill switch tripped, hook should short-circuit) or non-zero
# (proceed normally).
mempal_kill_switch_tripped() {
# Palace nuke is the strongest signal: respect it before touching
# disk for state, logging, etc.
if [ ! -d "$HOME/.mempalace" ]; then
return 0
fi
case "${MEMPAL_DISABLE_HOOK:-}" in
1|true|TRUE|yes|YES) return 0 ;;
esac
case "${MEMPALACE_HOOKS_AUTO_SAVE:-}" in
false|FALSE|0|no|NO) return 0 ;;
esac
local cfg="$HOME/.mempalace/config.json"
if [ -f "$cfg" ]; then
local auto
auto=$("$MEMPAL_PYTHON_BIN" -c "
import json, sys
try:
with open(sys.argv[1]) as f:
cfg = json.load(f)
print(str(cfg.get('hooks', {}).get('auto_save', True)).lower())
except Exception:
print('true')
" "$cfg" 2>/dev/null)
if [ "$auto" = "false" ]; then
return 0
fi
fi
return 1
}
# ── camelCase JSON parser (Antigravity stdin) ────────────────────────
#
# Reads JSON from stdin once and prints a sanitized, sentinel-bracketed
# block of fields the bash side can grab via `sed -n 'Np'`. Why a
# sentinel and per-line layout: bash 3.2 doesn't have `mapfile` or
# `readarray`, and `eval`-on-shell-var is the wrong shape (every value
# is user-controllable JSON). Sentinel + line offset is the same pattern
# the existing Claude Code hook (hooks/mempal_save_hook.sh) uses.
#
# Output layout (one field per line; line numbers are stable and the
# fields are documented in STDIN_SHAPE.md):
#
# line 1: __MEMPAL_PARSE_OK__ — sentinel (parse success marker)
# line 2: conversationId — sanitized to [A-Za-z0-9._-]
# line 3: transcriptPath — sanitized to a safe path charset
# line 4: workspacePath — workspacePaths[0], sanitized
# line 5: artifactDirectoryPath — sanitized
# line 6: executionNum — integer, default 0
# line 7: terminationReason — sanitized to [a-z_]
# line 8: fullyIdle — "True" or "False" (string)
# line 9: invocationNum — integer, default 0
# line 10: initialNumSteps — integer, default 0
#
# The sanitizers are defense-in-depth: every field is also vetted by
# the Python json.load step, but we still strip shell-meaningful chars
# from any field a downstream bash variable might interpolate, so that
# a hostile / malformed harness payload cannot inject command tokens.
#
# Stderr from Python is captured to last_python_err.log at mode 0600 so
# operators can debug parse failures without re-firing the hook. The
# umask 077 on the inner subshell creates the file at 0600 atomically;
# the explicit chmod 600 below is a belt-and-suspenders guard if a
# future edit ever drops the umask.
mempal_parse_stdin() {
local input="$1"
(
umask 077
printf '%s' "$input" | "$MEMPAL_PYTHON_BIN" -c "
import sys, json, re
# IMPORTANT: do NOT wrap json.load in a try/except. If the input is
# not valid JSON we want Python to exit non-zero BEFORE printing the
# __MEMPAL_PARSE_OK__ sentinel — the bash caller looks for the
# sentinel on line 1 to decide whether to engage its defense-in-depth
# 'failed to parse' branch. Catching the exception and falling back
# to data={} would let the sentinel print, masking parse failures
# from the bash side. The traceback lands in
# antigravity_last_python_err.log so operators can debug.
data = json.load(sys.stdin)
def safe(s, allowed=r'[^a-zA-Z0-9_/.\-~]'):
return re.sub(allowed, '', str(s))
def safe_id(s):
return re.sub(r'[^a-zA-Z0-9._-]', '', str(s))
def safe_int(v, default=0):
try:
n = int(v)
return n if n >= 0 else default
except Exception:
return default
def safe_lower_alpha_underscore(s):
return re.sub(r'[^a-z_]', '', str(s).lower())
conv_id = safe_id(data.get('conversationId', ''))
transcript = safe(data.get('transcriptPath', ''))
wp_arr = data.get('workspacePaths', [])
if isinstance(wp_arr, list) and wp_arr:
workspace = safe(wp_arr[0])
else:
workspace = ''
artifact = safe(data.get('artifactDirectoryPath', ''))
execution_num = safe_int(data.get('executionNum', 0))
termination_reason = safe_lower_alpha_underscore(data.get('terminationReason', ''))
fully_idle_raw = data.get('fullyIdle', None)
if fully_idle_raw is True or str(fully_idle_raw).lower() in ('true', '1', 'yes'):
fully_idle = 'True'
else:
fully_idle = 'False'
invocation_num = safe_int(data.get('invocationNum', 0))
initial_num_steps = safe_int(data.get('initialNumSteps', 0))
print('__MEMPAL_PARSE_OK__')
print(conv_id)
print(transcript)
print(workspace)
print(artifact)
print(execution_num)
print(termination_reason)
print(fully_idle)
print(invocation_num)
print(initial_num_steps)
" 2>"$MEMPAL_STATE_DIR/antigravity_last_python_err.log"
)
# Tidy up the err log: keep it iff non-empty (failure happened).
if [ -s "$MEMPAL_STATE_DIR/antigravity_last_python_err.log" ]; then
chmod 600 "$MEMPAL_STATE_DIR/antigravity_last_python_err.log" 2>/dev/null
else
rm -f "$MEMPAL_STATE_DIR/antigravity_last_python_err.log" 2>/dev/null
fi
}
# ── Transcript path validator ─────────────────────────────────────────
#
# Mirrors mempalace.hooks_cli._validate_transcript_path: rejects empty,
# non-jsonl/json suffixes, and any `..` traversal segment.
mempal_is_valid_transcript_path() {
local path="$1"
[ -n "$path" ] || return 1
case "$path" in
*.json|*.jsonl) ;;
*) return 1 ;;
esac
case "/$path/" in
*/../*) return 1 ;;
esac
return 0
}
# ── Wing inference ────────────────────────────────────────────────────
#
# Takes the first workspace path from workspacePaths[] (already
# extracted into $1) and derives a `wing_<slug>` name from its leaf
# directory. Hyphens become underscores; spaces become underscores.
# Empty input yields wing_sessions, matching mempalace.hooks_cli's
# fallback.
mempal_infer_wing() {
local workspace="$1"
if [ -z "$workspace" ]; then
printf 'wing_sessions'
return 0
fi
# Strip trailing slashes
while [ "${workspace}" != "${workspace%/}" ]; do
workspace="${workspace%/}"
done
if [ -z "$workspace" ]; then
printf 'wing_sessions'
return 0
fi
local leaf="${workspace##*/}"
if [ -z "$leaf" ]; then
printf 'wing_sessions'
return 0
fi
# Lowercase + hyphens-to-underscores. tr is bash 3.2 safe; ${var^^}
# / ${var//-/_} on a fresh expansion are bash 4+ only.
local slug
slug=$(printf '%s' "$leaf" | tr 'A-Z' 'a-z' | tr ' -' '__')
printf 'wing_%s' "$slug"
}
# ── Save-interval floor ───────────────────────────────────────────────
#
# Reads MEMPAL_SAVE_INTERVAL from the environment, floors it to >= 1
# so that `count % interval` cannot divide by zero. We hit the
# divide-by-zero shape on the Cursor PR review; this guards explicitly.
mempal_save_interval() {
local raw="${MEMPAL_SAVE_INTERVAL:-15}"
case "$raw" in
''|*[!0-9]*) printf '15'; return 0 ;;
esac
# Strip leading zeros. bash arithmetic ($((...))) parses any token
# starting with `0` as octal, so MEMPAL_SAVE_INTERVAL=08 would
# crash $((COUNT % INTERVAL)) with "value too great for base".
# Loop while the value still starts with 0 AND has length > 1, so
# the literal string "0" is preserved (then floored to 15 below).
while [ "${raw}" != "${raw#0}" ] && [ "${#raw}" -gt 1 ]; do
raw="${raw#0}"
done
if [ "$raw" -lt 1 ] 2>/dev/null; then
printf '15'
return 0
fi
printf '%s' "$raw"
}
# ── Atomic counter write ──────────────────────────────────────────────
#
# Writes $value to $file via a same-directory temp file + `mv -f`.
# `mv` (rename) is atomic on a single filesystem, so a concurrent
# reader either sees the old contents or the new contents — never a
# half-written / truncated file. A plain `printf > file` truncates
# first and then writes, leaving a window where a concurrent Stop fire
# could read an empty / partial value. Concurrent fires for one
# conversation are unlikely (Antigravity serializes turns) but the
# previous "written atomically" comment was simply false; this makes
# it true.
#
# bash 3.2 safe. The temp lives in the same directory as the target so
# the rename stays on one filesystem (a cross-device mv would fall back
# to copy+unlink and lose atomicity). On any failure we degrade to a
# direct write rather than leaving the counter unwritten.
mempal_write_counter_atomic() {
local file="$1"
local value="$2"
local tmp
tmp="$(mktemp "${file}.XXXXXX" 2>/dev/null)" || {
printf '%s' "$value" > "$file"
return
}
printf '%s' "$value" > "$tmp"
mv -f "$tmp" "$file" 2>/dev/null || {
rm -f "$tmp" 2>/dev/null
printf '%s' "$value" > "$file"
}
}
# ── State-file TTL ────────────────────────────────────────────────────
#
# Per-conversation state artifacts (antigravity_save_count_<conv>,
# antigravity_pending_<conv>, antigravity_woke_<conv>/) accumulate one
# set per conversation and are never otherwise removed. Reads
# MEMPAL_STATE_TTL_DAYS (default 30), validated digits-only and
# leading-zero-stripped like mempal_save_interval so `find -mtime`
# never sees a bad token. A value of 0 means "sweep aggressively"
# (everything older than today); we floor empty/garbage to 30.
mempal_state_ttl_days() {
local raw="${MEMPAL_STATE_TTL_DAYS:-30}"
case "$raw" in
''|*[!0-9]*) printf '30'; return 0 ;;
esac
while [ "${raw}" != "${raw#0}" ] && [ "${#raw}" -gt 1 ]; do
raw="${raw#0}"
done
printf '%s' "$raw"
}
# ── Stale state GC ────────────────────────────────────────────────────
#
# Opportunistic sweep of per-conversation state older than the TTL.
# Gated to run at most once per 24h via the antigravity_last_sweep
# marker so it costs nothing on the vast majority of fires (a single
# mtime comparison). When it does run, three `find` passes remove the
# stale counter files, pending markers, and woke marker directories.
#
# The name globs are specific (antigravity_save_count_*, _pending_*,
# _woke_*), so the shared log files (antigravity_hook.log,
# antigravity_last_input.log, antigravity_last_python_err.log) and the
# antigravity_last_sweep marker itself are never touched. BSD find
# (macOS default) and GNU find both accept `-maxdepth`, `-mtime +N`,
# and `-exec ... +`.
#
# Fail-open: every step is best-effort; a missing state dir, a find
# that errors, or a permission problem must never abort the caller.
mempal_gc_stale_state() {
[ -d "$MEMPAL_STATE_DIR" ] || return 0
local marker="$MEMPAL_STATE_DIR/antigravity_last_sweep"
if [ -f "$marker" ]; then
local mtime now
if mtime=$("$MEMPAL_PYTHON_BIN" -c 'import os, sys; print(int(os.path.getmtime(sys.argv[1])))' "$marker" 2>/dev/null) \
&& now=$(date '+%s' 2>/dev/null) \
&& [ -n "$mtime" ] \
&& [ "$((now - mtime))" -lt 86400 ]; then
return 0
fi
fi
# Touch the marker first so a crash mid-sweep still throttles the
# next fire (better to skip a sweep than to hammer the disk).
: > "$marker" 2>/dev/null
local ttl
ttl=$(mempal_state_ttl_days)
find "$MEMPAL_STATE_DIR" -maxdepth 1 -type f \
-name 'antigravity_save_count_*' -mtime +"$ttl" \
-exec rm -f {} + 2>/dev/null
find "$MEMPAL_STATE_DIR" -maxdepth 1 -type f \
-name 'antigravity_pending_*' -mtime +"$ttl" \
-exec rm -f {} + 2>/dev/null
find "$MEMPAL_STATE_DIR" -maxdepth 1 -type d \
-name 'antigravity_woke_*' -mtime +"$ttl" \
-exec rm -rf {} + 2>/dev/null
return 0
}
# ── Fail-open emitters ────────────────────────────────────────────────
#
# Every code path in both hooks must terminate by calling exactly one
# of these emitters. Stdout is JSON. Exit status is always 0 — the hook
# never blocks the user's IDE on its own failure (constraint #2).
#
# CRITICAL: mempal_emit_stop_pass MUST NEVER emit
# {"decision":"continue"} — that would force the agent to keep running
# instead of letting the turn end. Antigravity treats any value other
# than "continue" (including `{}`) as "allow the stop". We enforce this
# by hard-coding the empty object output here.
mempal_emit_stop_pass() {
printf '{}\n'
}
mempal_emit_wake_inject() {
local message="$1"
if [ -z "$message" ]; then
printf '{}\n'
return 0
fi
# Encode the message as JSON via Python so embedded quotes / newlines
# / control chars don't corrupt the output.
"$MEMPAL_PYTHON_BIN" -c "
import json, sys
msg = sys.argv[1]
print(json.dumps({'injectSteps': [{'ephemeralMessage': msg}]}))
" "$message" 2>/dev/null || printf '{}\n'
}
+245
View File
@@ -0,0 +1,245 @@
#!/bin/bash
# MEMPALACE ANTIGRAVITY SAVE HOOK — Stop event handler
#
# Antigravity fires the Stop event each time the agent's execution loop
# terminates. We use it to background-mine the active conversation
# transcript every Nth save into the user's MemPalace, and to write a
# diary checkpoint via `mempalace mine --mode convos`.
#
# Mirrors the Claude Code (hooks/mempal_save_hook.sh) and Codex
# (.codex-plugin/hooks/mempal-hook.sh) integrations as closely as the
# Antigravity stdin/stdout contract allows. Differences:
#
# * Antigravity stdin uses camelCase: conversationId, transcriptPath,
# workspacePaths, executionNum, terminationReason, fullyIdle.
# * Antigravity stdout MUST be `{}` on every code path. Emitting
# `{"decision":"continue"}` would force the agent to keep running
# and create an infinite loop. We never call mempal_emit_stop_pass
# with anything other than the literal empty object.
# * Counter file is namespaced antigravity_save_count_<conversationId>
# to coexist with Claude Code / Cursor / Codex state in the same
# ~/.mempalace/hook_state/ directory.
#
# === STDIN (verified, camelCase) ===
# {
# "executionNum": 1,
# "terminationReason": "model_stop",
# "error": "",
# "fullyIdle": true,
# "conversationId": "<uuid>",
# "workspacePaths": ["/abs/path/..."],
# "transcriptPath": "/abs/path/transcript.jsonl",
# "artifactDirectoryPath": "/abs/path/artifacts/"
# }
#
# === STDOUT (always) ===
# {}
#
# `set -e` is intentionally NOT enabled — a broken hook must not block
# the user's conversation (constraint #2 in the integration brief).
# ── Locate this script + source common helpers ───────────────────────
MEMPAL_AGY_HOOK_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)"
# shellcheck source=lib/common.sh
. "$MEMPAL_AGY_HOOK_DIR/lib/common.sh"
# ── Read all of stdin once ───────────────────────────────────────────
INPUT=$(cat)
# ── Kill switch: short-circuit cleanly if disabled ───────────────────
if mempal_kill_switch_tripped; then
mempal_emit_stop_pass
exit 0
fi
# ── Opportunistic GC of stale per-conversation state ─────────────────
#
# Self-throttled to at most once per 24h (see mempal_gc_stale_state),
# so this is a single mtime check on the overwhelming majority of
# fires. Runs after the kill switch so a disabled hook touches nothing.
mempal_gc_stale_state
# ── Parse stdin (camelCase, sentinel-guarded) ────────────────────────
_parsed=$(mempal_parse_stdin "$INPUT")
_marker=$(printf '%s\n' "$_parsed" | sed -n '1p')
CONVERSATION_ID=$(printf '%s\n' "$_parsed" | sed -n '2p')
TRANSCRIPT_PATH=$(printf '%s\n' "$_parsed" | sed -n '3p')
WORKSPACE_PATH=$(printf '%s\n' "$_parsed" | sed -n '4p')
# Line 5 (artifactDirectoryPath) is parsed but unused for save. Skip.
EXECUTION_NUM=$(printf '%s\n' "$_parsed" | sed -n '6p')
# Line 7 (terminationReason) is parsed but used only for logging.
TERMINATION_REASON=$(printf '%s\n' "$_parsed" | sed -n '7p')
FULLY_IDLE=$(printf '%s\n' "$_parsed" | sed -n '8p')
# ── Defense-in-depth: surface raw input on parse failure ─────────────
#
# When the sentinel is missing, Python crashed before reaching its
# print() calls. Persist the offending payload (capped at 4 KB, mode
# 0600) so the next debugger doesn't lose a day to log lines that say
# "Session unknown".
if [ -n "$INPUT" ] && [ "$_marker" != "__MEMPAL_PARSE_OK__" ]; then
mempal_log "stop" "unknown" "input parse failed (sentinel missing); see antigravity_last_input.log + antigravity_last_python_err.log"
(
umask 077
printf '%s' "$INPUT" | head -c 4096 > "$MEMPAL_STATE_DIR/antigravity_last_input.log"
)
chmod 600 "$MEMPAL_STATE_DIR/antigravity_last_input.log" 2>/dev/null
# Continue with empty fields; the validators below will reject.
fi
CONVERSATION_ID="${CONVERSATION_ID:-unknown}"
TRANSCRIPT_PATH="${TRANSCRIPT_PATH:-}"
WORKSPACE_PATH="${WORKSPACE_PATH:-}"
EXECUTION_NUM="${EXECUTION_NUM:-0}"
TERMINATION_REASON="${TERMINATION_REASON:-}"
FULLY_IDLE="${FULLY_IDLE:-False}"
# Expand ~ in the transcript path
TRANSCRIPT_PATH="${TRANSCRIPT_PATH/#\~/$HOME}"
# ── Bail when fullyIdle is False ─────────────────────────────────────
#
# If background commands or async tasks are still running, the
# transcript is still in motion. Defer the save until the next Stop
# event when the agent is fully done — better to skip than to ingest a
# half-finished transcript and pollute the search index.
if [ "$FULLY_IDLE" != "True" ]; then
mempal_log "stop" "$CONVERSATION_ID" "deferring save: fullyIdle=False (executionNum=$EXECUTION_NUM, terminationReason=$TERMINATION_REASON)"
mempal_emit_stop_pass
exit 0
fi
# ── Skip when terminationReason is `error` ───────────────────────────
#
# A model error termination usually means the transcript is corrupt or
# truncated. Don't ingest noise.
if [ "$TERMINATION_REASON" = "error" ]; then
mempal_log "stop" "$CONVERSATION_ID" "skipping save: terminationReason=error"
mempal_emit_stop_pass
exit 0
fi
# ── Increment counter (per conversation) ─────────────────────────────
#
# The counter is a single integer, written via mempal_write_counter_atomic
# (same-dir temp + `mv`, which is an atomic rename on one filesystem).
# Concurrent Stop fires for the same conversation are unlikely
# (Antigravity serializes turns), but the atomic write means a
# concurrent reader always sees a complete value rather than a
# half-written / truncated file. The integer-only validation on read
# is a second guard: any garbled value resets the count to 0.
COUNTER_FILE="$MEMPAL_STATE_DIR/antigravity_save_count_${CONVERSATION_ID}"
COUNT=0
if [ -f "$COUNTER_FILE" ]; then
raw=$(cat "$COUNTER_FILE" 2>/dev/null)
case "$raw" in
''|*[!0-9]*) COUNT=0 ;;
*) COUNT="$raw" ;;
esac
fi
COUNT=$((COUNT + 1))
mempal_write_counter_atomic "$COUNTER_FILE" "$COUNT"
INTERVAL=$(mempal_save_interval)
mempal_log "stop" "$CONVERSATION_ID" "count=$COUNT interval=$INTERVAL executionNum=$EXECUTION_NUM workspace=$WORKSPACE_PATH"
# ── Modulo gate ──────────────────────────────────────────────────────
#
# `count % interval == 0` triggers a save. INTERVAL has been floored to
# >= 1 by mempal_save_interval, so the modulo cannot divide by zero
# even if the user explicitly set MEMPAL_SAVE_INTERVAL=0 or empty.
if [ $((COUNT % INTERVAL)) -ne 0 ]; then
mempal_emit_stop_pass
exit 0
fi
# ── Pending-marker guard ─────────────────────────────────────────────
#
# If a previous save is still running (the marker file exists), skip
# this fire. The mine subprocess removes the marker on exit, but a
# crashed mine could leave the marker forever — guard against that by
# treating markers older than 1 hour as stale and reclaiming them.
PENDING_FILE="$MEMPAL_STATE_DIR/antigravity_pending_${CONVERSATION_ID}"
if [ -f "$PENDING_FILE" ]; then
# mtime in epoch seconds (portable; BSD/macOS `date -r` takes epoch, not a path).
if mtime=$("$MEMPAL_PYTHON_BIN" -c 'import os, sys; print(int(os.path.getmtime(sys.argv[1])))' "$PENDING_FILE" 2>/dev/null) \
&& now=$(date '+%s') \
&& [ -n "$mtime" ] \
&& [ "$((now - mtime))" -lt 3600 ]; then
mempal_log "stop" "$CONVERSATION_ID" "pending save still in flight; skipping"
mempal_emit_stop_pass
exit 0
fi
mempal_log "stop" "$CONVERSATION_ID" "stale pending marker reclaimed"
rm -f "$PENDING_FILE" 2>/dev/null
fi
# ── Validate transcript path ─────────────────────────────────────────
if ! mempal_is_valid_transcript_path "$TRANSCRIPT_PATH"; then
mempal_log "stop" "$CONVERSATION_ID" "invalid transcriptPath rejected: $TRANSCRIPT_PATH"
mempal_emit_stop_pass
exit 0
fi
if [ ! -f "$TRANSCRIPT_PATH" ]; then
mempal_log "stop" "$CONVERSATION_ID" "transcriptPath does not exist: $TRANSCRIPT_PATH"
mempal_emit_stop_pass
exit 0
fi
# ── Trigger save ─────────────────────────────────────────────────────
WING=$(mempal_infer_wing "$WORKSPACE_PATH")
TRANSCRIPT_DIR=$(dirname "$TRANSCRIPT_PATH")
mempal_log "stop" "$CONVERSATION_ID" "TRIGGERING SAVE wing=$WING transcript_dir=$TRANSCRIPT_DIR"
# Drop the pending marker BEFORE spawning so a near-simultaneous fire
# sees it. If the spawn fails, remove the marker so the next fire can
# retry.
: > "$PENDING_FILE" 2>/dev/null
# Detach EVERYTHING heavy into a single background subshell: the
# runnability probe, the mine itself, and the pending-marker cleanup.
# The foreground returns immediately after spawning, so the hook's
# stdout (`{}`) reaches Antigravity within milliseconds.
#
# Why the probe must NOT run in the foreground: `mempalace --version`
# is NOT cheap. Building the `mine` argument parser imports
# `mempalace.miner` (-> palace -> backends -> chromadb/onnx) before
# argparse ever processes `--version`, so the probe pays the full
# cold-start import cost. Running it in the foreground would block the
# hook for that entire import and blow the <500ms save budget. Moving
# it inside the backgrounded subshell keeps the foreground instant.
#
# Folding the cleanup into this same subshell also removes the need for
# a separate process-liveness polling watcher: the `rm -f
# "$PENDING_FILE"` simply runs after the mine returns, in the same
# shell that owns the mine — no sibling-PID `wait` hazard, no polling
# loop.
#
# We invoke mempalace as `"$MEMPAL_PYTHON_BIN" -m mempalace` rather than
# the bare `mempalace` console script so a user with the package
# installed only inside a venv (and the venv's bin/ not on the hook's
# PATH, e.g. `uv tool install` in some distributions, or a manually
# managed virtualenv) still hits a working mine. MEMPAL_PYTHON honours
# user override; sees ``mempalace/__main__.py`` which dispatches to
# ``mempalace.cli:main`` — identical to the console script.
mempal_log "stop" "$CONVERSATION_ID" "spawning background mine wing=$WING transcript_dir=$TRANSCRIPT_DIR"
(
if "$MEMPAL_PYTHON_BIN" -m mempalace --version >/dev/null 2>&1; then
"$MEMPAL_PYTHON_BIN" -m mempalace mine "$TRANSCRIPT_DIR" \
--mode convos \
--wing "$WING" \
>> "$MEMPAL_AGY_LOG" 2>&1 < /dev/null
mempal_log "stop" "$CONVERSATION_ID" "background mine finished wing=$WING"
else
mempal_log "stop" "$CONVERSATION_ID" "ERROR: mempalace is not runnable via $MEMPAL_PYTHON_BIN -m mempalace; install mempalace or set MEMPAL_PYTHON"
fi
rm -f "$PENDING_FILE" 2>/dev/null
) >/dev/null 2>&1 < /dev/null &
# ── Always emit `{}` ─────────────────────────────────────────────────
#
# Never `{"decision":"continue"}`. That would force the agent into an
# infinite re-execution loop. mempal_emit_stop_pass hard-codes `{}`.
mempal_emit_stop_pass
exit 0
+166
View File
@@ -0,0 +1,166 @@
#!/bin/bash
# MEMPALACE ANTIGRAVITY WAKE HOOK — PreInvocation event handler
#
# Antigravity fires the PreInvocation event before every model
# invocation, with `invocationNum` carrying the sequence number of the
# call. We use the first invocation (invocationNum == 1) as our
# session-start equivalent and inject a verbatim memory pointer into
# the agent's context via the `injectSteps[].ephemeralMessage` output
# field — the message lives for one turn and does not persist into the
# transcript, so it doesn't pollute future invocations of this same
# conversation.
#
# === STDIN (verified, camelCase) ===
# {
# "invocationNum": 1,
# "initialNumSteps": 0,
# "conversationId": "<uuid>",
# "workspacePaths": ["/abs/path/..."],
# "transcriptPath": "/abs/path/transcript.jsonl",
# "artifactDirectoryPath": "/abs/path/artifacts/"
# }
#
# === STDOUT ===
# Either:
# {} — no injection
# Or:
# {"injectSteps":[{"ephemeralMessage":"..."}]} — verbatim memory pointer
#
# Verbatim guarantee: the ephemeralMessage carries the exact text
# emitted by `mempalace wake-up`, never paraphrased or summarized.
#
# Performance budget: the integration brief sets a 100ms ceiling for
# startup injection. We enforce a 500ms hard timeout on the
# `mempalace wake-up` subprocess (more generous than 100ms because
# cold ChromaDB connections can dominate, and missing the budget is
# strictly better than blocking the user) — if it doesn't return in
# time we emit `{}` and let the conversation start without injection.
#
# `set -e` is intentionally NOT enabled — fail-open is mandatory.
# ── Locate this script + source common helpers ───────────────────────
MEMPAL_AGY_HOOK_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)"
# shellcheck source=lib/common.sh
. "$MEMPAL_AGY_HOOK_DIR/lib/common.sh"
# ── Read all of stdin once ───────────────────────────────────────────
INPUT=$(cat)
# ── Kill switch ──────────────────────────────────────────────────────
if mempal_kill_switch_tripped; then
mempal_emit_stop_pass
exit 0
fi
# ── Parse stdin ──────────────────────────────────────────────────────
_parsed=$(mempal_parse_stdin "$INPUT")
_marker=$(printf '%s\n' "$_parsed" | sed -n '1p')
CONVERSATION_ID=$(printf '%s\n' "$_parsed" | sed -n '2p')
# Lines 3-5 (transcriptPath, workspacePath, artifactDirectoryPath) are
# parsed; we use workspacePath for wing inference. transcriptPath and
# artifactDirectoryPath are unused by the wake flow.
# Line 4: workspacePath
WORKSPACE_PATH=$(printf '%s\n' "$_parsed" | sed -n '4p')
INVOCATION_NUM=$(printf '%s\n' "$_parsed" | sed -n '9p')
# Defense-in-depth on parse failure
if [ -n "$INPUT" ] && [ "$_marker" != "__MEMPAL_PARSE_OK__" ]; then
mempal_log "preInvocation" "unknown" "input parse failed (sentinel missing)"
mempal_emit_stop_pass
exit 0
fi
CONVERSATION_ID="${CONVERSATION_ID:-unknown}"
WORKSPACE_PATH="${WORKSPACE_PATH:-}"
INVOCATION_NUM="${INVOCATION_NUM:-0}"
# ── Gate: only inject on the FIRST invocation ────────────────────────
#
# PreInvocation fires before every model call. Without this gate we'd
# inject memory on every single turn — both expensive and visually
# noisy. invocationNum == 1 means "first model call of this
# conversation", which is the closest thing Antigravity exposes to
# Cursor's `sessionStart`.
if [ "$INVOCATION_NUM" != "1" ]; then
mempal_emit_stop_pass
exit 0
fi
# ── Loop guard ───────────────────────────────────────────────────────
#
# Defense in depth: even within the first invocation, we only ever
# want to inject once per conversation. mkdir is atomic and works on
# bash 3.2 / macOS / Linux without flock or other GNU coreutils
# extensions.
WOKE_MARKER="$MEMPAL_STATE_DIR/antigravity_woke_${CONVERSATION_ID}"
if ! mkdir "$WOKE_MARKER" 2>/dev/null; then
mempal_log "preInvocation" "$CONVERSATION_ID" "already woke this conversation; skipping"
mempal_emit_stop_pass
exit 0
fi
# ── Run wake-up with a hard timeout ──────────────────────────────────
#
# `timeout` is GNU coreutils — present on most Linux installs but
# missing from stock macOS. Wrap the subprocess in a Python timeout
# (subprocess.run(timeout=...)) which is cross-platform. The Python
# script also constructs the final JSON envelope for stdout, so the
# bash side just passes the result through.
WING=$(mempal_infer_wing "$WORKSPACE_PATH")
mempal_log "preInvocation" "$CONVERSATION_ID" "WAKE injection wing=$WING invocationNum=$INVOCATION_NUM"
OUTPUT=$("$MEMPAL_PYTHON_BIN" -c "
import json, subprocess, sys
wing = sys.argv[1]
timeout_s = 0.5 # 500 ms
# Invoke as ``[sys.executable, '-m', 'mempalace', ...]`` rather than
# the bare ``mempalace`` console script. sys.executable is the same
# Python that resolved MEMPAL_PYTHON in lib/common.sh, so this binds
# the wake-up call to the correct interpreter (and its installed
# mempalace package) even when the venv's bin/ isn't on PATH.
try:
completed = subprocess.run(
[sys.executable, '-m', 'mempalace', 'wake-up', '--wing', wing],
capture_output=True,
text=True,
timeout=timeout_s,
)
if completed.returncode != 0:
print('{}')
sys.exit(0)
body = (completed.stdout or '').strip()
if not body:
print('{}')
sys.exit(0)
# Verbatim — pass the wake-up text exactly as emitted, wrapped in
# the Antigravity injectSteps envelope. json.dumps escapes embedded
# control chars and quotes correctly.
print(json.dumps({'injectSteps': [{'ephemeralMessage': body}]}))
except FileNotFoundError:
print('{}')
except subprocess.TimeoutExpired:
print('{}')
except Exception:
print('{}')
" "$WING" 2>/dev/null)
if [ -z "$OUTPUT" ]; then
OUTPUT='{}'
fi
# Sanity-check: never emit `decision` from a PreInvocation hook (that
# field belongs to the Stop event). The Python helper only ever
# constructs `{"injectSteps": [...]}` or `{}`, so this is belt-and-
# suspenders against a future edit ever leaking a Stop-shaped object.
case "$OUTPUT" in
*\"decision\"*)
mempal_log "preInvocation" "$CONVERSATION_ID" "ERROR: refused to emit decision field from wake hook"
mempal_emit_stop_pass
exit 0
;;
esac
printf '%s\n' "$OUTPUT"
exit 0
+157
View File
@@ -0,0 +1,157 @@
# MemPalace Cursor IDE Hooks
Auto-save and session-recall hooks for the [Cursor](https://cursor.com) IDE,
matching the behaviour of the existing Claude Code + Codex hooks at the repo
root and adding two Cursor-only capabilities (`sessionStart` recall and a
preCompact transcript snapshot).
For the rendered documentation see
[`website/guide/cursor-hooks.md`](../../website/guide/cursor-hooks.md) or
the published version at
[mempalaceofficial.com/guide/cursor-hooks](https://mempalaceofficial.com/guide/cursor-hooks.html).
## What's here
| File | Role |
|-------------------------------------|---------------------------------------------------------------------|
| `lib/common.sh` | Shared bash helpers (parse, log, counter, wing inference, kill switch). Sourced by all three hooks. |
| `mempal_save_hook_cursor.sh` | Cursor `stop` hook. Counts stop invocations per conversation, emits a `followup_message` every `SAVE_INTERVAL` (default 15) telling the agent to file the session into MemPalace. |
| `mempal_precompact_hook_cursor.sh` | Cursor `preCompact` hook. Runs `mempalace mine` synchronously on the transcript before compaction, then drops a `.pending` marker so the next stop forces a save nudge. |
| `mempal_wake_hook_cursor.sh` | Cursor `sessionStart` hook. Returns `additional_context` telling the agent to recall scoped to the wing inferred from the workspace root. Cursor-only — Claude Code has no equivalent. |
| `install.sh` | Optional installer. Copies the scripts to `~/.mempalace/hooks/cursor/` and merges entries into `~/.cursor/hooks.json` (or `.cursor/hooks.json` for project scope). Supports `--dry-run` and `--uninstall`. |
| `STDIN_SHAPE.md` | Reference. Per-event stdin / stdout schema with citations to the official Cursor docs. |
## Quick install
Preview first (writes nothing, prints the would-be JSON to stdout):
```bash
hooks/cursor/install.sh --scope user --dry-run
```
Apply — writes `~/.cursor/hooks.json` and copies the scripts to `~/.mempalace/hooks/cursor/`:
```bash
hooks/cursor/install.sh --scope user
```
Pass `--scope project --target <repo>` to write `<repo>/.cursor/hooks.json` instead.
The installer never auto-runs — it is a documented opt-in step. We do not
modify your Cursor config on `pip install mempalace` because editor config
is sacred and should never be touched without explicit consent.
## Manual install (no installer)
The minimum wiring is `stop` only. Add to `~/.cursor/hooks.json`:
```json
{
"version": 1,
"hooks": {
"stop": [
{
"command": "/absolute/path/to/hooks/cursor/mempal_save_hook_cursor.sh",
"loop_limit": 1
}
]
}
}
```
For the full triple (recommended), also wire `sessionStart` and `preCompact`
— see [`examples/cursor/hooks.json`](../../examples/cursor/hooks.json).
After editing the file, Cursor watches `hooks.json` and reloads
automatically. If hooks still do not fire, restart Cursor and check the
Hooks panel in Settings.
## Configuration
All knobs are env vars; defaults match the Claude Code hooks where
possible so a single hook-state directory works for both editors.
| Variable | Default | Purpose |
|--------------------------------|------------------------------------|---------|
| `MEMPAL_SAVE_INTERVAL` | `15` | Number of `stop` events between save followups. |
| `MEMPAL_CURSOR_SILENT` | (unset) | Set to `1`/`true`/`yes` to suppress the `followup_message`. The hook still runs its best-effort background mine and keeps its counters — it just stays silent. `MEMPAL_VERBOSE=false`/`0`/`no` does the same. See note below on why the followup is on by default. |
| `MEMPAL_DIR` | (unset) | Optional project directory to also mine on each save. Additive — never replaces the transcript mine. |
| `MEMPAL_PYTHON` | auto-detected | Path to a Python 3 interpreter. Fallback order: `$MEMPAL_PYTHON` → `command -v python3` → bare `python3`. Useful when Cursor is launched from a GUI on macOS and the inherited PATH lacks your installed `python3`. |
| `MEMPAL_STATE_DIR` | `$HOME/.mempalace/hook_state` | Where the hook keeps its per-conversation counter files, pending-save markers, and `cursor_hook.log`. |
| `MEMPAL_STATE_TTL_DAYS` | `30` | Age (days) after which stale `cursor_*.count` / `cursor_*.pending` state files are garbage-collected. A daily-throttled sweep runs from the hooks; only Cursor state is touched (shared logs and other editors' state are left alone). |
| `MEMPAL_DISABLE_HOOK` | (unset) | Set to `1`/`true`/`yes` to disable all three hooks. Emergency kill switch. |
| `MEMPALACE_HOOKS_AUTO_SAVE` | (unset) | Set to `false`/`0`/`no` to disable. Same semantics as the Claude Code hooks. Also honoured via `~/.mempalace/config.json` → `{"hooks": {"auto_save": false}}`. |
## Debugging
Everything appends to:
```bash
cat ~/.mempalace/hook_state/cursor_hook.log
```
Example log lines (ISO 8601 + event + conversation id):
```
[2026-05-27T02:16:01Z] [event=sessionStart] [conv=abc123] workspace=/Users/me/proj wing=proj
[2026-05-27T02:21:33Z] [event=stop] [conv=abc123] counter 0 -> 1 (interval=15)
[2026-05-27T02:42:09Z] [event=stop] [conv=abc123] counter 14 -> 15 (interval=15)
[2026-05-27T02:42:09Z] [event=stop] [conv=abc123] TRIGGERING SAVE at counter=15
[2026-05-27T02:42:11Z] [event=stop] [conv=abc123] loop_count>0; letting agent stop
```
When a hook can't parse its stdin (corrupt payload, future Cursor schema
change), the raw input (capped at 4096 bytes, mode 0600) lands at:
```
~/.mempalace/hook_state/cursor_last_input.log
~/.mempalace/hook_state/cursor_last_python_err.log
```
These are overwritten on each failure, never appended, so a repeating
misconfiguration cannot grow disk usage.
## What differs from the Claude Code hooks
| Aspect | Claude Code hooks (`hooks/mempal_*.sh`) | Cursor hooks (`hooks/cursor/*.sh`) |
|-------------------------|------------------------------------------------|-----------------------------------------------------|
| Counter key | `session_id` | `conversation_id` (Cursor's stable per-conv id) |
| Loop guard | `stop_hook_active` flag in stdin | `loop_count` field in stdin |
| Counting method | Parses JSONL transcript for user messages | Counts `stop` invocations (transcript schema undoc) |
| Capture path | Background `mine --mode convos` (normalize.py has a Claude parser) | Background mine is best-effort (no Cursor parser); the `followup_message` carries verbatim capture |
| Save default | Silent — diary nudge opt-IN behind `MEMPAL_VERBOSE=true` | Followup ON by default; opt-OUT via `MEMPAL_CURSOR_SILENT=1` / `MEMPAL_VERBOSE=false` |
| PreCompact behaviour | `decision: block` forces save before compaction | Pre-mine + pending-save marker (Cursor preCompact is observational-only) |
| sessionStart | n/a (Claude Code has no equivalent) | `additional_context` injects recall guidance |
| State dir | `$HOME/.mempalace/hook_state` (hardcoded) | Same default, plus `MEMPAL_STATE_DIR` env override |
| Kill switch | `MEMPALACE_HOOKS_AUTO_SAVE=false` | Same, plus `MEMPAL_DISABLE_HOOK=1` alias |
| Log file | `hook.log` | `cursor_hook.log` (kept separate to avoid cross-tool log churn) |
See [`STDIN_SHAPE.md`](STDIN_SHAPE.md) for the per-event schema and
[`website/guide/cursor-hooks.md`](../../website/guide/cursor-hooks.md) for
the full walkthrough with diagrams.
## Why the followup is on by default (Cursor-specific)
Unlike the Claude Code hook — which is silent by default because its
background `mempalace mine --mode convos` captures the verbatim transcript
on its own — Cursor's transcript format is **undocumented** and
`mempalace/normalize.py` has **no Cursor parser**. The background mine on
the Cursor `stop`/`preCompact` hooks is therefore **best-effort only**: it
does not yet yield clean verbatim conversation drawers.
That makes the `followup_message` the **load-bearing verbatim-capture
path** for Cursor — it drives the agent to file its own in-context
verbatim quotes via `mempalace_add_drawer` / `mempalace_diary_write`.
Silencing it by default would leave a default Cursor install capturing
nothing, which is why it is on by default here. Set `MEMPAL_CURSOR_SILENT=1`
(or `MEMPAL_VERBOSE=false`) if you prefer the Claude-style silent
behaviour and accept the reduced capture. Once `normalize.py` learns to
read Cursor transcripts, this default will flip to silent to match Claude.
## Cost
Zero extra LLM tokens spent by the hooks themselves. The hooks are local
bash scripts that run on your machine. The followup message the save hook
emits is a normal user turn — it counts the same as any other user message
and does not invoke any extra LLM call beyond the one the user would
otherwise make. Suppress it with `MEMPAL_CURSOR_SILENT=1` if you want zero
followups in the chat window.
+184
View File
@@ -0,0 +1,184 @@
# Cursor Hook Stdin Shape — Reference
This file documents the JSON payloads the Cursor IDE sends to the
MemPalace hook scripts in `hooks/cursor/`. It exists so a future
contributor does not have to re-discover the schema by writing a
probe hook.
**Source:** [`cursor.com/docs/hooks.md`](https://cursor.com/docs/hooks.md),
fetched 2026-05-27. Cursor's hook system is documented as a stable
v1 schema (`{"version": 1, ...}` at the top of `hooks.json`).
If you suspect Cursor has changed the payload shape since that fetch
date, re-verify against the upstream docs and update both this file
and `hooks/cursor/lib/common.sh::mempal_parse_stdin`. The hook
scripts deliberately ignore fields they do not consume, so adding
new fields is non-breaking.
## Common fields (all events)
Every hook receives these on stdin in addition to its event-specific
fields. Source: docs section "Common schema → Input (all hooks)".
```json
{
"conversation_id": "string",
"generation_id": "string",
"model": "string",
"hook_event_name": "string",
"cursor_version": "string",
"workspace_roots": ["<absolute path>"],
"user_email": "string | null",
"transcript_path": "string | null"
}
```
**Field notes (verified):**
- `conversation_id` is the stable per-conversation ID. The Cursor
`stop` event does **not** carry a `session_id` — only
`conversation_id`. MemPalace keys its counter files on this. Cursor
`sessionStart` does carry a `session_id`, and the docs note it is
"same as `conversation_id`".
- `generation_id` changes every user message. We do not use it.
- `transcript_path` may be `null` if the user has disabled
transcripts in Cursor settings. The hooks degrade gracefully when
the value is empty.
- `workspace_roots` is normally a single-entry array but multi-root
workspaces are supported; MemPalace uses index `[0]`.
## Event-specific fields
### `stop` (consumed by `mempal_save_hook_cursor.sh`)
```json
{
"status": "completed" | "aborted" | "error",
"loop_count": 0
}
```
- `loop_count` indicates how many times this stop hook has already
triggered an automatic followup for this conversation (starts at
0). When `loop_count > 0` we know our own previous `followup_message`
is currently being processed — the save hook returns `{}` so the
agent can finish. Equivalent to Claude Code's `stop_hook_active`.
- The per-script `loop_limit` (default 5 for Cursor hooks, configurable
via the `loop_limit` field on the hook entry in `hooks.json`) is
defense-in-depth on top of our own check. The example `hooks.json`
in `examples/cursor/` sets `loop_limit: 1`.
**Allowed output fields** (only):
```json
{ "followup_message": "<text to auto-submit as next user turn>" }
```
### `preCompact` (consumed by `mempal_precompact_hook_cursor.sh`)
```json
{
"trigger": "auto" | "manual",
"context_usage_percent": 85,
"context_tokens": 120000,
"context_window_size": 128000,
"message_count": 45,
"messages_to_compact": 30,
"is_first_compaction": true
}
```
**Critical constraint:** preCompact is documented as **observational
only**. It cannot block compaction and its allowed output fields are
limited to:
```json
{ "user_message": "<short message shown to the user when compaction occurs>" }
```
There is **no** `followup_message` and **no** `decision: block` on
this event — unlike Claude Code's `PreCompact`. MemPalace works
around this by:
1. Running `mempalace mine` synchronously inside the hook so the
verbatim transcript lands in the palace before compaction
summarises it.
2. Dropping a `cursor_<conversation_id>.pending` marker that the next
`stop` invocation reads and uses to force a save followup
regardless of its counter.
### `sessionStart` (consumed by `mempal_wake_hook_cursor.sh`)
```json
{
"session_id": "<unique session identifier>",
"is_background_agent": true,
"composer_mode": "agent" | "ask" | "edit"
}
```
`session_id` equals `conversation_id` on this event (docs are
explicit about this).
**Allowed output fields:**
```json
{
"env": { "<key>": "<value>" },
"additional_context": "<text added to conversation's initial system context>"
}
```
`additional_context` is the field MemPalace uses. The schema also
accepts `continue` and `user_message` but the docs explicitly note
"current callers do not enforce them; session creation is not
blocked even when continue is false". We do not emit either.
## Environment variables (all hooks)
Cursor sets these env vars on every hook execution; the hook scripts
fall back to them when JSON parsing fails for any reason.
| Variable | Description |
|---------------------------|---------------------------------------------------|
| `CURSOR_PROJECT_DIR` | Workspace root (= `workspace_roots[0]`) |
| `CURSOR_VERSION` | Cursor version string |
| `CURSOR_USER_EMAIL` | Authenticated user email (if logged in) |
| `CURSOR_TRANSCRIPT_PATH` | Conversation transcript path (if transcripts on) |
| `CURSOR_CODE_REMOTE` | `"true"` if running in a remote workspace |
| `CLAUDE_PROJECT_DIR` | Alias for `CURSOR_PROJECT_DIR` (Claude compat) |
## Exit code semantics
Cursor interprets command-hook exit codes as follows
(docs "Hook Types → Command-Based Hooks → Exit code behavior"):
- `0` — success, use the JSON output.
- `2` — block the action (equivalent to `permission: "deny"`).
- Other — hook failed; action proceeds (fail-open by default).
MemPalace hooks always exit `0` and emit either `{}` (no-op) or a
valid JSON response. We never use exit code `2`; nothing MemPalace
does should ever block an agent action.
## Working directory contract
- **User hooks** (`~/.cursor/hooks.json`) run from `~/.cursor/`.
- **Project hooks** (`.cursor/hooks.json`) run from the project root.
The MemPalace hooks always resolve their sibling `lib/common.sh` via
`BASH_SOURCE[0]` so the working directory does not matter for the
script's own loading — only the `command` path in `hooks.json` needs
to point at the absolute location of the script.
## Transcript file format (out of scope)
The format of the file at `transcript_path` is **not documented by
Cursor** as of the fetch date above. MemPalace deliberately does not
parse it: the save hook counts `stop` invocations (each one
corresponds to one assistant turn) and hands the transcript to
`mempalace mine`, which has its own normaliser layer.
If you need to consume the transcript directly, probe its shape with
a throw-away hook that does `cat > /tmp/cursor-transcript-sample.txt`
and inspect the output — there is no shortcut.
+399
View File
@@ -0,0 +1,399 @@
#!/bin/bash
# MEMPALACE CURSOR HOOK INSTALLER
#
# Optional helper. Copies the three Cursor hook scripts to a
# stable install location and merges entries into a Cursor
# `hooks.json` config file — without clobbering unrelated hooks
# already in that file.
#
# This is NEVER auto-invoked. Editor config is sacred; we do not
# modify a user's hooks.json without explicit consent. The user runs
# this script (or wires the hooks manually) as a documented opt-in.
#
# === USAGE ===
#
# hooks/cursor/install.sh [options]
#
# Options:
# --scope user|project Target scope. Default: user.
# - user: merges into ~/.cursor/hooks.json
# - project: merges into <target>/.cursor/hooks.json
# --target <path> Project root for --scope project (default: $PWD).
# Ignored for --scope user.
# --install-dir <path> Where to copy the hook scripts.
# Default: ~/.mempalace/hooks/cursor
# --variant full|minimal Which hook set to wire.
# - full: stop + preCompact + sessionStart
# - minimal: stop only
# Default: full.
# --dry-run Print the would-be JSON to stdout, do not write
# and do not copy scripts.
# --uninstall Remove MemPalace entries from the target
# hooks.json (preserves unrelated hooks).
# Does NOT delete the installed scripts.
# -h, --help Show this help and exit.
#
# === PORTABILITY ===
#
# Pure bash 3.2 + POSIX tools + python3 (which the hook scripts
# themselves already require). No `jq` dependency.
#
# Python helpers are materialised to temp files rather than piped via
# `$(... <<'PYEOF' ... PYEOF)` to dodge the bash 3.2.57 parser bug
# that trips on parens nested inside a heredoc body that lives inside
# a `$(...)` command substitution.
set -e
usage() {
sed -n '2,38p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'
}
# ── Defaults ──────────────────────────────────────────────────────
SCOPE="user"
TARGET=""
INSTALL_DIR="$HOME/.mempalace/hooks/cursor"
VARIANT="full"
DRY_RUN=0
UNINSTALL=0
# ── Parse args ────────────────────────────────────────────────────
while [ $# -gt 0 ]; do
case "$1" in
--scope)
shift
SCOPE="${1:-}"
;;
--target)
shift
TARGET="${1:-}"
;;
--install-dir)
shift
INSTALL_DIR="${1:-}"
;;
--variant)
shift
VARIANT="${1:-}"
;;
--dry-run) DRY_RUN=1 ;;
--uninstall) UNINSTALL=1 ;;
-h|--help)
usage
exit 0
;;
*)
printf 'install.sh: unknown argument: %s\n' "$1" >&2
usage >&2
exit 64
;;
esac
shift || true
done
case "$SCOPE" in
user|project) ;;
*)
printf 'install.sh: --scope must be "user" or "project" (got "%s")\n' \
"$SCOPE" >&2
exit 64
;;
esac
case "$VARIANT" in
full|minimal) ;;
*)
printf 'install.sh: --variant must be "full" or "minimal" (got "%s")\n' \
"$VARIANT" >&2
exit 64
;;
esac
# ── Resolve paths ─────────────────────────────────────────────────
_script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" 2>/dev/null && pwd)"
SOURCE_DIR="$_script_dir"
# Resolve --install-dir to an absolute path before it gets baked into
# hooks.json. Cursor invokes hook commands from its own working
# directory (typically the project root), so a relative command path
# would silently fail to launch the hook. gh-PR review caught this.
case "$INSTALL_DIR" in
/*) ;;
*) INSTALL_DIR="$PWD/$INSTALL_DIR" ;;
esac
# Resolve the Python interpreter the same way the hooks themselves do
# so a user with a non-default Python is consistent across install +
# runtime.
if [ -n "${MEMPAL_PYTHON:-}" ] && [ -x "$MEMPAL_PYTHON" ]; then
PYTHON_BIN="$MEMPAL_PYTHON"
else
PYTHON_BIN="$(command -v python3 2>/dev/null || true)"
fi
if [ -z "$PYTHON_BIN" ]; then
printf 'install.sh: python3 not found on PATH; cannot proceed.\n' >&2
printf 'Set $MEMPAL_PYTHON to an interpreter path or install python3.\n' >&2
exit 1
fi
# Determine the target hooks.json path.
case "$SCOPE" in
user)
if [ -n "$TARGET" ]; then
printf 'install.sh: --target is only meaningful with --scope project; ignoring.\n' >&2
fi
TARGET_DIR="$HOME/.cursor"
;;
project)
TARGET_DIR="${TARGET:-$PWD}/.cursor"
;;
esac
TARGET_FILE="$TARGET_DIR/hooks.json"
# Determine which commands the merge / uninstall logic should
# install or remove. Paths point at the install location, NOT the
# source repo — once the user runs install.sh they can move / delete
# the cloned repo without breaking the wiring.
SAVE_CMD="$INSTALL_DIR/mempal_save_hook_cursor.sh"
PRECOMPACT_CMD="$INSTALL_DIR/mempal_precompact_hook_cursor.sh"
WAKE_CMD="$INSTALL_DIR/mempal_wake_hook_cursor.sh"
# ── Step 1: copy scripts (skipped on --dry-run / --uninstall) ─────
if [ "$UNINSTALL" -eq 0 ] && [ "$DRY_RUN" -eq 0 ]; then
mkdir -p "$INSTALL_DIR/lib"
cp "$SOURCE_DIR/lib/common.sh" "$INSTALL_DIR/lib/common.sh"
cp "$SOURCE_DIR/mempal_save_hook_cursor.sh" "$INSTALL_DIR/"
cp "$SOURCE_DIR/mempal_precompact_hook_cursor.sh" "$INSTALL_DIR/"
cp "$SOURCE_DIR/mempal_wake_hook_cursor.sh" "$INSTALL_DIR/"
chmod +x "$INSTALL_DIR/mempal_save_hook_cursor.sh" \
"$INSTALL_DIR/mempal_precompact_hook_cursor.sh" \
"$INSTALL_DIR/mempal_wake_hook_cursor.sh"
printf 'install.sh: copied scripts to %s\n' "$INSTALL_DIR" >&2
fi
# ── Short-circuit: uninstall with no existing file is a no-op ─────
#
# Without this, the merge step would happily write an empty
# {"version": 1, "hooks": {}} to a brand new file that the user
# never asked us to create — surprising behaviour that the test
# suite explicitly guards against.
if [ "$UNINSTALL" -eq 1 ] && [ ! -f "$TARGET_FILE" ]; then
printf 'install.sh: nothing to uninstall (%s does not exist)\n' \
"$TARGET_FILE" >&2
exit 0
fi
# ── Step 2: merge / unmerge hooks.json via python3 ────────────────
#
# Materialise the merge logic to a temp .py file (see bash 3.2
# rationale at the top of this script), then invoke it. The Python
# script is responsible for:
# * tolerating a missing or empty hooks.json (starts from {})
# * preserving unrelated hook entries on install
# * preserving unrelated hook entries on uninstall
# * recognising MemPalace entries by basename in the `command` field
# * idempotent install (re-running does not duplicate entries)
# mktemp portability: pass an explicit absolute template so we sidestep
# the BSD vs GNU difference in `-t` semantics (BSD treats it as a
# prefix; GNU treats it as a template). Honour TMPDIR if set, fall
# back to /tmp. gh-PR review caught the previous `-t` form as
# non-portable.
MERGE_PY="$(mktemp "${TMPDIR:-/tmp}/mempal-install-merge.XXXXXX")"
trap 'rm -f "$MERGE_PY"' EXIT
cat > "$MERGE_PY" <<'PYEOF'
"""hooks.json merge helper for hooks/cursor/install.sh.
Argv:
sys.argv[1]: path to hooks.json (may not exist)
sys.argv[2]: variant ("full" or "minimal")
sys.argv[3]: uninstall flag ("1" or "0")
sys.argv[4]: save_cmd absolute path
sys.argv[5]: precompact_cmd absolute path
sys.argv[6]: wake_cmd absolute path
Output: prints the merged JSON to stdout. Exits 2 on a malformed
existing config (refuses to overwrite a broken file).
"""
import json
import os
import sys
target_file = sys.argv[1]
variant = sys.argv[2]
uninstall = sys.argv[3] == "1"
save_cmd = sys.argv[4]
precompact_cmd = sys.argv[5]
wake_cmd = sys.argv[6]
# Recognise our entries by basename. The three filenames below are
# the unique product-of-our-naming convention; any entry whose command
# ends in one of them is treated as a MemPalace entry on
# install (so we replace rather than duplicate it) and on uninstall
# (so we remove it without touching unrelated entries). Matching on
# basename rather than a full-path substring lets users pick any
# --install-dir without breaking uninstall.
MEMPAL_BASENAMES = (
"mempal_save_hook_cursor.sh",
"mempal_precompact_hook_cursor.sh",
"mempal_wake_hook_cursor.sh",
)
if os.path.exists(target_file):
with open(target_file, "r", encoding="utf-8") as fh:
try:
cfg = json.load(fh)
except Exception as exc:
sys.stderr.write(
"install.sh: existing %s is not valid JSON: %s\n"
"Refusing to overwrite. Fix the file and retry.\n"
% (target_file, exc)
)
sys.exit(2)
else:
cfg = {}
if not isinstance(cfg, dict):
sys.stderr.write(
"install.sh: %s top level must be a JSON object; got %s\n"
% (target_file, type(cfg).__name__)
)
sys.exit(2)
cfg.setdefault("version", 1)
cfg.setdefault("hooks", {})
if not isinstance(cfg["hooks"], dict):
sys.stderr.write(
"install.sh: %s 'hooks' must be a JSON object\n" % target_file
)
sys.exit(2)
def is_mempal_entry(entry):
if not isinstance(entry, dict):
return False
cmd = entry.get("command", "")
if not isinstance(cmd, str):
return False
# Match on basename so a customised --install-dir (e.g. /opt/...,
# ~/.local/share/..., or anything with or without a leading dot)
# still round-trips through uninstall.
base = os.path.basename(cmd)
return base in MEMPAL_BASENAMES
def filter_mempal(entries):
if not isinstance(entries, list):
return entries
return [e for e in entries if not is_mempal_entry(e)]
def upsert(event, entry):
existing = cfg["hooks"].get(event, [])
if not isinstance(existing, list):
sys.stderr.write(
"install.sh: %s hooks[%s] must be a list\n" % (target_file, event)
)
sys.exit(2)
cleaned = [e for e in existing if not is_mempal_entry(e)]
cleaned.append(entry)
cfg["hooks"][event] = cleaned
if uninstall:
for event in list(cfg["hooks"].keys()):
cfg["hooks"][event] = filter_mempal(cfg["hooks"][event])
if not cfg["hooks"][event]:
del cfg["hooks"][event]
else:
upsert("stop", {"command": save_cmd, "loop_limit": 1})
if variant == "full":
upsert("preCompact", {"command": precompact_cmd})
upsert("sessionStart", {"command": wake_cmd})
# Stable key order for the events MemPalace touches, then preserve
# any unrelated event names in their original order so future Cursor
# events we don't know about yet still round-trip.
known_order = [
"sessionStart",
"stop",
"preCompact",
"sessionEnd",
"preToolUse",
"postToolUse",
"postToolUseFailure",
"subagentStart",
"subagentStop",
"beforeShellExecution",
"afterShellExecution",
"beforeMCPExecution",
"afterMCPExecution",
"beforeReadFile",
"afterFileEdit",
"beforeSubmitPrompt",
"afterAgentResponse",
"afterAgentThought",
"beforeTabFileRead",
"afterTabFileEdit",
"workspaceOpen",
]
ordered_hooks = {}
for event in known_order:
if event in cfg["hooks"]:
ordered_hooks[event] = cfg["hooks"][event]
for event, entries in cfg["hooks"].items():
if event not in ordered_hooks:
ordered_hooks[event] = entries
cfg["hooks"] = ordered_hooks
print(json.dumps(cfg, indent=2, sort_keys=False))
PYEOF
NEW_JSON="$("$PYTHON_BIN" "$MERGE_PY" \
"$TARGET_FILE" "$VARIANT" "$UNINSTALL" \
"$SAVE_CMD" "$PRECOMPACT_CMD" "$WAKE_CMD")"
# ── Step 3: emit, write, or remove ────────────────────────────────
if [ "$DRY_RUN" -eq 1 ]; then
printf 'install.sh: --dry-run; would write to %s\n' "$TARGET_FILE" >&2
printf '%s\n' "$NEW_JSON"
exit 0
fi
mkdir -p "$TARGET_DIR"
# If --uninstall left an empty hooks object AND no other top-level
# keys beyond version, remove the file entirely so the user's
# `.cursor/` directory does not accumulate orphan configs.
if [ "$UNINSTALL" -eq 1 ]; then
# Inline the emptiness check via `python -c '...'` rather than a
# temp .py file. The body is short enough that a tmpfile is pure
# overhead, and removing the tmpfile eliminates a small leak
# window if the script is interrupted between mktemp and rm -f.
# gh-PR review suggested this simplification.
NON_EMPTY="$(printf '%s' "$NEW_JSON" | "$PYTHON_BIN" -c '
import json, sys
cfg = json.load(sys.stdin)
hooks = cfg.get("hooks", {})
extras = [k for k in cfg.keys() if k not in ("version", "hooks")]
print("1" if (hooks or extras) else "0")
')"
if [ "$NON_EMPTY" = "0" ] && [ -f "$TARGET_FILE" ]; then
rm -f "$TARGET_FILE"
printf 'install.sh: removed empty %s\n' "$TARGET_FILE" >&2
exit 0
fi
fi
TMP_FILE="${TARGET_FILE}.tmp.$$"
printf '%s\n' "$NEW_JSON" > "$TMP_FILE"
mv "$TMP_FILE" "$TARGET_FILE"
if [ "$UNINSTALL" -eq 1 ]; then
printf 'install.sh: removed MemPalace entries from %s\n' "$TARGET_FILE" >&2
else
printf 'install.sh: wrote %s\n' "$TARGET_FILE" >&2
printf 'install.sh: restart Cursor (or wait for it to reload hooks.json)\n' >&2
fi
+490
View File
@@ -0,0 +1,490 @@
# shellcheck shell=bash
# MEMPALACE CURSOR HOOK — shared helpers
#
# Sourced by the three Cursor hooks (stop / preCompact / sessionStart).
# Mirrors the conventions of the existing Claude Code hook scripts
# (hooks/mempal_save_hook.sh, hooks/mempal_precompact_hook.sh) so a
# user who already debugs one knows how to debug the other:
#
# * STATE_DIR layout under ~/.mempalace/hook_state/
# * MEMPAL_PYTHON resolution order (override → $PATH → bare python3)
# * MEMPALACE_HOOKS_AUTO_SAVE=false kill switch (config.json fallback)
# * sentinel-guarded Python parser via `sed -n 'Np'` (bash 3.2 safe)
# * fail-open on internal errors: emit `{}` and log, never crash the
# hook host
#
# Cursor-specific additions on top of that contract:
#
# * MEMPAL_DISABLE_HOOK=1 as an additional kill-switch alias
# * MEMPAL_STATE_DIR env override for the state directory
# * conversation_id (Cursor's stable per-conversation ID) replaces
# Claude Code's session_id in the counter file names — Cursor `stop`
# events do not carry a session_id, only conversation_id
# * loop_count is the loop-prevention signal in place of Claude Code's
# stop_hook_active flag (Cursor docs, "stop" event)
#
# This file is sourced, not executed, so it intentionally has no
# shebang. The `# shellcheck shell=bash` directive above tells
# shellcheck to treat it as bash when run standalone.
# ── State directory + log path ────────────────────────────────────────
#
# Honour MEMPAL_STATE_DIR (additive override introduced for Cursor)
# while keeping the default identical to the Claude Code hooks so a
# user running both keeps a single state directory.
MEMPAL_STATE_DIR="${MEMPAL_STATE_DIR:-$HOME/.mempalace/hook_state}"
mkdir -p "$MEMPAL_STATE_DIR" 2>/dev/null
MEMPAL_CURSOR_LOG="$MEMPAL_STATE_DIR/cursor_hook.log"
# ── Python interpreter resolution ─────────────────────────────────────
#
# Same contract as the Claude Code hooks:
# 1. $MEMPAL_PYTHON — explicit user override (absolute path)
# 2. $(command -v python3) — first python3 on the hook's PATH
# 3. bare "python3" — last-resort fallback
mempal_resolve_python() {
local p="${MEMPAL_PYTHON:-}"
if [ -n "$p" ] && [ -x "$p" ]; then
printf '%s' "$p"
return 0
fi
p="$(command -v python3 2>/dev/null || true)"
if [ -n "$p" ]; then
printf '%s' "$p"
return 0
fi
printf '%s' "python3"
}
MEMPAL_PYTHON_BIN="$(mempal_resolve_python)"
# ── Logging ───────────────────────────────────────────────────────────
#
# Lines are `[ISO8601Z] [event=...] [conv=...] message`. ISO8601 keeps
# the format greppable across timezones (the Claude Code log uses
# %H:%M:%S which loses the date — we improve on that here without
# changing the existing log file).
mempal_log() {
local event="${1:-?}"
local conv="${2:-unknown}"
local msg="${3:-}"
local ts
ts="$(date -u '+%Y-%m-%dT%H:%M:%SZ')"
printf '[%s] [event=%s] [conv=%s] %s\n' "$ts" "$event" "$conv" "$msg" \
>> "$MEMPAL_CURSOR_LOG" 2>/dev/null
}
# ── Kill switch ───────────────────────────────────────────────────────
#
# Disabled if ANY of:
# * MEMPAL_DISABLE_HOOK is a truthy string (Cursor-prompt addition)
# * MEMPALACE_HOOKS_AUTO_SAVE is false/0/no (Claude Code convention)
# * ~/.mempalace/config.json has hooks.auto_save == false
#
# Returns 0 (true in shell) when disabled, 1 when enabled.
mempal_is_disabled() {
case "${MEMPAL_DISABLE_HOOK:-}" in
1|true|yes|on) return 0 ;;
esac
case "${MEMPALACE_HOOKS_AUTO_SAVE:-}" in
false|0|no|off) return 0 ;;
esac
local cfg="$HOME/.mempalace/config.json"
if [ -f "$cfg" ]; then
local result
# Use python -c '...' with the config path as argv[1] rather
# than a heredoc. A heredoc body that contains parens inside a
# $(...) command substitution trips the bash 3.2.57 parser
# bug (macOS /bin/bash default) — gh-PR review caught this.
# The -c form is also consistent with mempal_parse_stdin below.
result="$("$MEMPAL_PYTHON_BIN" -c '
import json, sys
try:
with open(sys.argv[1]) as f:
cfg = json.load(f)
print(str(cfg.get("hooks", {}).get("auto_save", True)).lower())
except Exception:
print("true")
' "$cfg" 2>/dev/null)"
if [ "$result" = "false" ]; then
return 0
fi
fi
return 1
}
# ── Stdin parser ──────────────────────────────────────────────────────
#
# Reads Cursor's hook JSON from $1 and exports:
# MEMPAL_CONV_ID — conversation_id, falls back to "unknown"
# MEMPAL_LOOP_COUNT — integer (0 if absent / non-numeric)
# MEMPAL_TRANSCRIPT — transcript_path, may be empty
# MEMPAL_WORKSPACE — first workspace_roots entry, falls back to
# CURSOR_PROJECT_DIR env var, then $PWD
# MEMPAL_TRIGGER — preCompact trigger ("auto" | "manual"), empty otherwise
# MEMPAL_STATUS — stop status ("completed" | "aborted" | "error"),
# empty otherwise
# MEMPAL_PARSE_OK — "1" if parser ran cleanly, "0" otherwise
#
# Uses the same sentinel + `sed -n 'Np'` extraction as the Claude Code
# hooks for bash 3.2 compatibility (mapfile/readarray are unavailable
# on macOS /bin/bash 3.2.57; #1440 regression). Each line of output is
# pre-sanitised by the Python side to a shell-safe character set.
mempal_parse_stdin() {
local input="${1:-}"
local parsed
# We invoke Python via -c with a single-quoted multi-line string
# rather than ``python3 - <<'PYEOF'`` because the heredoc form
# would shadow Python's stdin with the heredoc body, leaving
# ``json.load(sys.stdin)`` to read nothing and silently fail. The
# parser body deliberately uses only double-quoted Python strings
# so the surrounding bash single-quote is safe verbatim, and uses
# only the shell-safe character set (alphanumeric, underscore,
# dash, slash, dot, tilde) matching the Claude Code hook's
# sanitiser so a hostile transcript_path cannot splice
# metacharacters into the parsed output.
parsed="$(
umask 077
printf '%s' "$input" | "$MEMPAL_PYTHON_BIN" -c '
import json, re, sys
def safe_str(value):
return re.sub(r"[^a-zA-Z0-9_/.\-~]", "", str(value or ""))
def safe_int(value):
try:
return str(int(value))
except (TypeError, ValueError):
return "0"
try:
data = json.load(sys.stdin)
except Exception:
sys.exit(1)
if not isinstance(data, dict):
sys.exit(1)
conv = safe_str(data.get("conversation_id") or data.get("session_id"))
loop_count = safe_int(data.get("loop_count", 0))
transcript = safe_str(data.get("transcript_path", ""))
trigger = safe_str(data.get("trigger", ""))
status = safe_str(data.get("status", ""))
roots = data.get("workspace_roots") or []
workspace = ""
if isinstance(roots, list) and roots:
workspace = safe_str(roots[0])
print("__MEMPAL_PARSE_OK__")
print(conv)
print(loop_count)
print(transcript)
print(workspace)
print(trigger)
print(status)
' 2>"$MEMPAL_STATE_DIR/cursor_last_python_err.log"
)"
# Drop empty stderr capture on success; lock it to 0600 on failure
# (mirrors the privacy contract in the Claude Code hooks — the
# traceback can echo back transcript_path / home layout).
if [ -s "$MEMPAL_STATE_DIR/cursor_last_python_err.log" ]; then
chmod 600 "$MEMPAL_STATE_DIR/cursor_last_python_err.log" 2>/dev/null
else
rm -f "$MEMPAL_STATE_DIR/cursor_last_python_err.log" 2>/dev/null
fi
local marker
marker="$(printf '%s\n' "$parsed" | sed -n '1p')"
if [ "$marker" = "__MEMPAL_PARSE_OK__" ]; then
MEMPAL_PARSE_OK="1"
MEMPAL_CONV_ID="$(printf '%s\n' "$parsed" | sed -n '2p')"
MEMPAL_LOOP_COUNT="$(printf '%s\n' "$parsed" | sed -n '3p')"
MEMPAL_TRANSCRIPT="$(printf '%s\n' "$parsed" | sed -n '4p')"
MEMPAL_WORKSPACE="$(printf '%s\n' "$parsed" | sed -n '5p')"
MEMPAL_TRIGGER="$(printf '%s\n' "$parsed" | sed -n '6p')"
MEMPAL_STATUS="$(printf '%s\n' "$parsed" | sed -n '7p')"
else
MEMPAL_PARSE_OK="0"
MEMPAL_CONV_ID=""
MEMPAL_LOOP_COUNT="0"
MEMPAL_TRANSCRIPT=""
MEMPAL_WORKSPACE=""
MEMPAL_TRIGGER=""
MEMPAL_STATUS=""
fi
# Defaults and environment fallbacks. The Cursor docs guarantee
# CURSOR_TRANSCRIPT_PATH and CURSOR_PROJECT_DIR env vars are set
# for every hook execution; if JSON parsing failed for any reason
# (sentinel missing, malformed payload, missing interpreter) we
# still have a usable workspace.
MEMPAL_CONV_ID="${MEMPAL_CONV_ID:-unknown}"
case "$MEMPAL_LOOP_COUNT" in
''|*[!0-9]*) MEMPAL_LOOP_COUNT="0" ;;
esac
if [ -z "$MEMPAL_TRANSCRIPT" ] && [ -n "${CURSOR_TRANSCRIPT_PATH:-}" ]; then
MEMPAL_TRANSCRIPT="${CURSOR_TRANSCRIPT_PATH}"
fi
if [ -z "$MEMPAL_WORKSPACE" ]; then
if [ -n "${CURSOR_PROJECT_DIR:-}" ]; then
MEMPAL_WORKSPACE="${CURSOR_PROJECT_DIR}"
elif [ -n "${CLAUDE_PROJECT_DIR:-}" ]; then
MEMPAL_WORKSPACE="${CLAUDE_PROJECT_DIR}"
else
MEMPAL_WORKSPACE="${PWD:-/}"
fi
fi
# Expand a leading ~ in the transcript path so downstream
# ``[ -f "$path" ]`` checks resolve correctly.
case "$MEMPAL_TRANSCRIPT" in
'~/'*) MEMPAL_TRANSCRIPT="$HOME/${MEMPAL_TRANSCRIPT#~/}" ;;
esac
}
# ── Defense-in-depth: dump unparseable stdin ──────────────────────────
#
# Same shape as the Claude Code hooks' last_input.log: bounded to 4096
# bytes, overwritten (never appended) so a misconfiguration loop cannot
# grow disk usage, 0600 perms because the dump mirrors the raw hook
# payload (transcript_path reveals the user's home + project layout).
mempal_dump_bad_input() {
local input="${1:-}"
if [ -z "$input" ]; then
return 0
fi
mempal_log "${2:-?}" "${MEMPAL_CONV_ID:-unknown}" \
"WARN: input parse failed (sentinel missing); see $MEMPAL_STATE_DIR/cursor_last_input.log + cursor_last_python_err.log"
(
umask 077
printf '%s' "$input" | head -c 4096 > "$MEMPAL_STATE_DIR/cursor_last_input.log"
)
chmod 600 "$MEMPAL_STATE_DIR/cursor_last_input.log" 2>/dev/null
}
# ── Counter helpers ───────────────────────────────────────────────────
#
# One counter file per conversation_id. Atomic write via temp file
# inside the same directory (rename is atomic on POSIX) so concurrent
# hook invocations cannot half-write the file. Read tolerates a
# corrupted or empty file by returning 0, never crashing.
_mempal_counter_path() {
local conv="${1:-unknown}"
# Sanitise the conv id one more time: it has already been through
# the Python sanitiser, but be defensive in case a caller passes a
# raw string. Strip any character outside [a-zA-Z0-9_.-].
local safe_conv
safe_conv="$(printf '%s' "$conv" | tr -cd 'a-zA-Z0-9_.-')"
if [ -z "$safe_conv" ]; then
safe_conv="unknown"
fi
printf '%s/cursor_%s.count' "$MEMPAL_STATE_DIR" "$safe_conv"
}
mempal_read_counter() {
local path="$1"
if [ ! -f "$path" ]; then
printf '0'
return 0
fi
local raw
raw="$(cat "$path" 2>/dev/null)"
case "$raw" in
''|*[!0-9]*) printf '0' ;;
*) printf '%s' "$raw" ;;
esac
}
mempal_write_counter_atomic() {
local path="$1"
local value="$2"
case "$value" in
''|*[!0-9]*) value="0" ;;
esac
local tmp="${path}.tmp.$$"
printf '%s' "$value" > "$tmp" 2>/dev/null || return 1
mv "$tmp" "$path" 2>/dev/null || {
rm -f "$tmp" 2>/dev/null
return 1
}
}
# ── Pending-save marker ───────────────────────────────────────────────
#
# Dropped by the preCompact hook (which cannot itself emit a
# followup_message — Cursor's preCompact is observational-only) and
# consumed by the next stop invocation so the LLM still gets a diary
# nudge after compaction.
_mempal_pending_path() {
local conv="${1:-unknown}"
local safe_conv
safe_conv="$(printf '%s' "$conv" | tr -cd 'a-zA-Z0-9_.-')"
if [ -z "$safe_conv" ]; then
safe_conv="unknown"
fi
printf '%s/cursor_%s.pending' "$MEMPAL_STATE_DIR" "$safe_conv"
}
mempal_set_pending() {
local conv="${1:-unknown}"
local path
path="$(_mempal_pending_path "$conv")"
: > "$path" 2>/dev/null || return 1
chmod 600 "$path" 2>/dev/null
}
mempal_consume_pending() {
local conv="${1:-unknown}"
local path
path="$(_mempal_pending_path "$conv")"
if [ -f "$path" ]; then
rm -f "$path" 2>/dev/null
return 0
fi
return 1
}
# ── State-file TTL ────────────────────────────────────────────────────
#
# Per-conversation state artifacts (cursor_<conv>.count and
# cursor_<conv>.pending) accumulate one set per conversation and are
# never otherwise removed (igorls review, PR #1632 — unbounded state
# growth). Reads MEMPAL_STATE_TTL_DAYS (default 30), validated
# digits-only and leading-zero-stripped (mirrors the SAVE_INTERVAL
# sanitiser) so `find -mtime` never sees a bad or octal token. Empty or
# non-numeric floors to 30; a value of 0 means "sweep everything older
# than today".
mempal_state_ttl_days() {
local raw="${MEMPAL_STATE_TTL_DAYS:-30}"
case "$raw" in
''|*[!0-9]*) printf '30'; return 0 ;;
esac
while [ "${raw}" != "${raw#0}" ] && [ "${#raw}" -gt 1 ]; do
raw="${raw#0}"
done
printf '%s' "$raw"
}
# ── Stale state GC ────────────────────────────────────────────────────
#
# Opportunistic sweep of per-conversation Cursor state older than the
# TTL. Throttled to at most once per 24h via the cursor_last_sweep
# marker, so it costs a single mtime comparison on the vast majority of
# fires. When it does run, two `find` passes remove the stale counter
# files and pending markers.
#
# The globs are Cursor-specific and suffix-anchored (cursor_*.count,
# cursor_*.pending), so the shared logs (cursor_hook.log,
# cursor_last_input.log, cursor_last_python_err.log), the
# cursor_last_sweep marker itself, and any antigravity_*/Claude state
# sharing the same directory are never touched. BSD find (macOS default)
# and GNU find both accept -maxdepth, -mtime +N, and -exec ... +.
#
# Fail-open: every step is best-effort; a missing state dir, a find that
# errors, or a permission problem must never abort the caller.
mempal_gc_stale_state() {
[ -d "$MEMPAL_STATE_DIR" ] || return 0
local marker="$MEMPAL_STATE_DIR/cursor_last_sweep"
if [ -f "$marker" ]; then
local mtime now
if mtime=$("$MEMPAL_PYTHON_BIN" -c 'import os, sys; print(int(os.path.getmtime(sys.argv[1])))' "$marker" 2>/dev/null) \
&& now=$(date '+%s' 2>/dev/null) \
&& [ -n "$mtime" ] \
&& [ "$((now - mtime))" -lt 86400 ]; then
return 0
fi
fi
# Touch the marker first so a crash mid-sweep still throttles the
# next fire (better to skip a sweep than to hammer the disk).
: > "$marker" 2>/dev/null
local ttl
ttl=$(mempal_state_ttl_days)
find "$MEMPAL_STATE_DIR" -maxdepth 1 -type f \
-name 'cursor_*.count' -mtime +"$ttl" \
-exec rm -f {} + 2>/dev/null
find "$MEMPAL_STATE_DIR" -maxdepth 1 -type f \
-name 'cursor_*.pending' -mtime +"$ttl" \
-exec rm -f {} + 2>/dev/null
return 0
}
# ── Workspace → wing inference ────────────────────────────────────────
#
# basename(workspace_root), normalised to [a-z0-9_-]. Edge cases:
# / → "root"
# /path/ → trailing slash stripped, then basename
# "/foo bar/" → "foo_bar" (spaces collapsed to underscores)
# "" → "cursor_session"
# "C:\\proj" → "proj" (Windows-style path; basename via tr fallback)
#
# We intentionally keep this in pure bash + POSIX tools so the
# inference is identical across the hook scripts and the test suite
# can target it as a function via `bash -c 'source ...; ...'`.
mempal_infer_wing() {
local raw="${1:-}"
if [ -z "$raw" ]; then
printf 'cursor_session'
return 0
fi
# Strip trailing slashes (but preserve the lone "/" case).
while [ "$raw" != "/" ] && [ "${raw%/}" != "$raw" ]; do
raw="${raw%/}"
done
if [ "$raw" = "/" ]; then
printf 'root'
return 0
fi
local base="${raw##*/}"
# On Windows-style paths with backslashes, fall back to splitting
# on backslash too so we don't return the whole path verbatim.
case "$base" in
*\\*) base="${base##*\\}" ;;
esac
# Lowercase + replace anything outside [a-z0-9_-] with underscore.
# Collapse runs of underscores so "foo bar" doesn't become
# "foo__bar".
base="$(printf '%s' "$base" \
| tr '[:upper:]' '[:lower:]' \
| tr -c 'a-z0-9_-' '_' \
| tr -s '_' \
| sed 's/^_//; s/_$//')"
if [ -z "$base" ]; then
printf 'cursor_session'
return 0
fi
printf '%s' "$base"
}
# ── Transcript path validation ────────────────────────────────────────
#
# Mirrors hooks/mempal_save_hook.sh::is_valid_transcript_path so the
# Cursor and Claude Code hooks reject the same shapes:
# * non-empty
# * .json or .jsonl suffix
# * no .. traversal segments
mempal_is_valid_transcript() {
local path="${1:-}"
[ -n "$path" ] || return 1
case "$path" in
*.json|*.jsonl) ;;
*) return 1 ;;
esac
case "/$path/" in
*/../*) return 1 ;;
esac
return 0
}
# ── JSON emit ─────────────────────────────────────────────────────────
#
# Final stdout write. Uses ``printf '%s'`` instead of ``echo`` because
# echo interprets ``-n``/``-e``/``-E`` as flags and varies in backslash
# handling between builtin and /bin/echo (xpg_echo shopt). Matches the
# Claude Code hook's documented rationale.
mempal_emit() {
printf '%s\n' "${1:-{\}}"
}
+133
View File
@@ -0,0 +1,133 @@
#!/bin/bash
# MEMPALACE CURSOR PRE-COMPACT HOOK — Snapshot transcript before compaction
#
# Cursor "preCompact" hook. Cursor's preCompact is documented as
# OBSERVATIONAL ONLY (cursor.com/docs/hooks.md fetched 2026-05-27):
#
# * It cannot block compaction.
# * Its only output field is `user_message` (no `followup_message`,
# no `decision: block`).
#
# So unlike the Claude Code PreCompact hook (which can block the AI
# and force a save before compaction proceeds), the Cursor preCompact
# hook can only do two useful things at this moment:
#
# 1. Run `mempalace mine` SYNCHRONOUSLY against the transcript file
# so whatever Cursor's transcript contains is ingested BEFORE
# Cursor summarises the conversation — zero LLM cost, no agent
# interaction needed. NOTE: this is BEST-EFFORT for Cursor.
# Cursor's transcript format is undocumented and normalize.py has
# no Cursor parser, so this does not yet produce clean verbatim
# drawers; it is a safety net, not the primary capture path.
#
# 2. Drop a `.pending` marker file keyed on conversation_id. The
# next `stop` hook reads that marker and forces a save followup
# regardless of its counter, so the AI still gets a "write a
# diary entry now" nudge on the very next turn. THIS followup is
# the load-bearing verbatim-capture path for Cursor (the agent
# files its own in-context verbatim quotes via the MCP tools).
#
# === INSTALL ===
#
# Add to ~/.cursor/hooks.json (or .cursor/hooks.json for project
# scope) under "preCompact":
#
# {
# "version": 1,
# "hooks": {
# "preCompact": [
# { "command": "/absolute/path/to/mempal_precompact_hook_cursor.sh" }
# ]
# }
# }
#
# No loop_limit is needed; preCompact is not a looping hook.
_mempal_self="${BASH_SOURCE[0]:-$0}"
_mempal_dir="$(cd "$(dirname "$_mempal_self")" 2>/dev/null && pwd)"
# shellcheck source=lib/common.sh
. "$_mempal_dir/lib/common.sh"
# Optional additional project directory to mine before compaction
# (parity with the Claude Code hook's MEMPAL_DIR knob).
MEMPAL_DIR="${MEMPAL_DIR:-}"
if mempal_is_disabled; then
mempal_emit '{}'
exit 0
fi
# Opportunistic, daily-throttled GC of stale per-conversation state.
# Placed after the kill switch so a disabled hook touches nothing.
mempal_gc_stale_state
INPUT="$(cat)"
mempal_parse_stdin "$INPUT"
if [ "$MEMPAL_PARSE_OK" != "1" ]; then
mempal_dump_bad_input "$INPUT" "preCompact"
mempal_emit '{}'
exit 0
fi
mempal_log "preCompact" "$MEMPAL_CONV_ID" \
"trigger=${MEMPAL_TRIGGER:-?} transcript=$MEMPAL_TRANSCRIPT"
# ── Synchronous mine ──────────────────────────────────────────────
#
# This intentionally blocks the hook. Compaction is irreversible —
# once Cursor summarises the conversation we cannot get the verbatim
# text back — so we must finish ingesting before returning. Background
# mining would race the compaction and lose data.
#
# TIMEOUT TRADEOFF (igorls review, PR #1632): on a very large transcript
# this synchronous mine can exceed Cursor's per-hook timeout, in which
# case Cursor kills the process mid-mine. That is acceptable and safe
# here: `mempalace mine` is incremental and append-only (a crash mid-
# operation leaves the existing palace untouched — see CLAUDE.md
# "Incremental only"), so a killed mine simply resumes on the next mine
# invocation rather than corrupting the palace. We deliberately do NOT
# wrap this in a shorter timeout, because truncating the mine would
# trade a recoverable partial-ingest for guaranteed silent data loss
# right before the irreversible compaction. The pending-save marker
# below is the backstop: the next `stop` hook re-mines and nudges a
# verbatim save regardless of whether this mine completed.
if command -v mempalace >/dev/null 2>&1; then
if mempal_is_valid_transcript "$MEMPAL_TRANSCRIPT" \
&& [ -f "$MEMPAL_TRANSCRIPT" ]; then
mempalace mine "$(dirname "$MEMPAL_TRANSCRIPT")" --mode convos \
>> "$MEMPAL_CURSOR_LOG" 2>&1 || \
mempal_log "preCompact" "$MEMPAL_CONV_ID" \
"WARN: mempalace mine convos returned non-zero"
elif [ -n "$MEMPAL_TRANSCRIPT" ]; then
mempal_log "preCompact" "$MEMPAL_CONV_ID" \
"skipping invalid transcript path: $MEMPAL_TRANSCRIPT"
fi
if [ -n "$MEMPAL_DIR" ] && [ -d "$MEMPAL_DIR" ]; then
mempalace mine "$MEMPAL_DIR" --mode projects \
>> "$MEMPAL_CURSOR_LOG" 2>&1 || \
mempal_log "preCompact" "$MEMPAL_CONV_ID" \
"WARN: mempalace mine projects returned non-zero"
fi
else
mempal_log "preCompact" "$MEMPAL_CONV_ID" \
"mempalace CLI not on PATH; skipping synchronous mine"
fi
# ── Drop the pending-save marker ──────────────────────────────────
mempal_set_pending "$MEMPAL_CONV_ID" || \
mempal_log "preCompact" "$MEMPAL_CONV_ID" \
"WARN: could not write pending-save marker"
# Surface a short user-visible note that compaction is about to
# happen and we've already captured the verbatim text. user_message
# is the only output field Cursor's preCompact accepts.
"$MEMPAL_PYTHON_BIN" -c '
import json
print(json.dumps({
"user_message": (
"MemPalace: transcript snapshotted before compaction. "
"A diary nudge is queued for the next agent turn."
)
}))
'
+267
View File
@@ -0,0 +1,267 @@
#!/bin/bash
# MEMPALACE CURSOR SAVE HOOK — Auto-save every N stop events
#
# Cursor "stop" hook. After every agent loop ends, this hook:
# 1. Counts stop invocations per conversation_id (each stop ≈ one
# assistant turn ≈ roughly one user message — see plan rationale).
# 2. Every SAVE_INTERVAL stops, returns a followup_message telling
# the agent to file the session into MemPalace and write a diary
# entry. Cursor auto-submits that as the next user message.
# 3. On the next stop, loop_count > 0 so we let the agent finish
# without re-firing — Cursor's loop_count is the equivalent of
# Claude Code's stop_hook_active flag.
# 4. If the preCompact hook has left a `.pending` marker, force a
# save followup regardless of the counter and clear the marker.
#
# === WHY THE FOLLOWUP FIRES BY DEFAULT (differs from the Claude hook) ===
#
# The Claude Code hook (hooks/mempal_save_hook.sh) is SILENT by default:
# its background `mempalace mine --mode convos` captures the verbatim
# transcript on its own, and the LLM-driven diary nudge is opt-IN behind
# MEMPAL_VERBOSE. That works because mempalace/normalize.py has a Claude
# Code JSONL parser.
#
# Cursor is different. Cursor's transcript format is undocumented (see
# STDIN_SHAPE.md) and normalize.py has NO Cursor parser, so the
# background mine below is BEST-EFFORT only — it does not yet yield clean
# verbatim drawers for Cursor. The followup_message is therefore the
# load-bearing verbatim-capture path: it drives the agent to call
# mempalace_add_drawer / mempalace_diary_write from its in-context memory.
# That is why it is ON by default here — silencing it by default would
# leave a default Cursor install capturing nothing.
#
# Users who want the Claude-style "zero tokens in the chat window"
# behaviour can silence the followup (see MEMPAL_CURSOR_SILENT /
# MEMPAL_VERBOSE below). Once normalize.py learns to read Cursor
# transcripts, this default should flip to silent to match Claude.
#
# Companion files in this directory:
# * lib/common.sh — shared helpers (sourced)
# * mempal_precompact_hook_cursor.sh — preCompact event
# * mempal_wake_hook_cursor.sh — sessionStart event
#
# === INSTALL ===
#
# Recommended path: run `hooks/cursor/install.sh` from a cloned repo,
# which copies the scripts to ~/.mempalace/hooks/cursor/ and merges
# the wiring into your ~/.cursor/hooks.json. See hooks/cursor/README.md
# for the full walkthrough, or website/guide/cursor-hooks.md for the
# rendered version.
#
# Manual wiring (user scope: ~/.cursor/hooks.json):
#
# {
# "version": 1,
# "hooks": {
# "stop": [
# {
# "command": "/absolute/path/to/mempal_save_hook_cursor.sh",
# "loop_limit": 1
# }
# ]
# }
# }
#
# The `loop_limit: 1` cap is defense-in-depth — even if our own
# loop_count check below regresses, Cursor itself will stop emitting
# our followup after one auto-iteration.
#
# === KILL SWITCHES ===
#
# MEMPAL_DISABLE_HOOK=1 — Cursor-prompt addition
# MEMPALACE_HOOKS_AUTO_SAVE=false — matches the Claude Code hooks
# ~/.mempalace/config.json "hooks.auto_save": false
#
# Any one of these short-circuits the hook to `{}` and exits 0.
# Resolve the directory this script lives in so we can source the
# sibling lib/common.sh whether the user invoked us by absolute path,
# by relative path, or via a symlink.
_mempal_self="${BASH_SOURCE[0]:-$0}"
_mempal_dir="$(cd "$(dirname "$_mempal_self")" 2>/dev/null && pwd)"
# shellcheck source=lib/common.sh
. "$_mempal_dir/lib/common.sh"
SAVE_INTERVAL="${MEMPAL_SAVE_INTERVAL:-15}"
# Coerce empty, non-numeric, AND zero to the default. SAVE_INTERVAL=0
# would otherwise crash bash on the modulo check below ($((NEXT % 0))
# is "division by 0"). gh-PR review caught this edge case.
case "$SAVE_INTERVAL" in
''|*[!0-9]*|0) SAVE_INTERVAL=15 ;;
esac
# Optional additional project directory to mine on save (parity with
# the Claude Code hook's MEMPAL_DIR knob — purely additive, never an
# override for the transcript mine).
MEMPAL_DIR="${MEMPAL_DIR:-}"
# ── Followup opt-out ──────────────────────────────────────────────
#
# Returns 0 (true) when the user has asked to suppress the
# followup_message. See the header comment for why the followup is ON
# by default for Cursor. When silenced, the hook still runs the
# best-effort background mine and still maintains its counters/markers
# — it just emits `{}` instead of a followup. Two equivalent signals:
# * MEMPAL_CURSOR_SILENT=1|true|yes|on — dedicated Cursor opt-out
# * MEMPAL_VERBOSE=false|0|no|off — cross-hook silence signal
# (mirror-image of the Claude hook, where MEMPAL_VERBOSE=true is
# what turns its diary nudge ON)
mempal_followup_silenced() {
case "${MEMPAL_CURSOR_SILENT:-}" in
1|true|yes|on) return 0 ;;
esac
case "${MEMPAL_VERBOSE:-}" in
false|0|no|off) return 0 ;;
esac
return 1
}
# Kill switch — emit `{}` so Cursor proceeds with normal stop.
if mempal_is_disabled; then
mempal_emit '{}'
exit 0
fi
# Opportunistic, daily-throttled GC of stale per-conversation state.
# Placed after the kill switch so a disabled hook touches nothing.
mempal_gc_stale_state
INPUT="$(cat)"
mempal_parse_stdin "$INPUT"
if [ "$MEMPAL_PARSE_OK" != "1" ]; then
mempal_dump_bad_input "$INPUT" "stop"
# Fail-open: don't block the host on a parse error.
mempal_emit '{}'
exit 0
fi
mempal_log "stop" "$MEMPAL_CONV_ID" \
"loop_count=$MEMPAL_LOOP_COUNT status=${MEMPAL_STATUS:-?} workspace=$MEMPAL_WORKSPACE"
# ── Loop-prevention ────────────────────────────────────────────────
#
# Cursor's loop_count indicates how many times THIS stop hook has
# already triggered an automatic followup for this conversation
# (starts at 0). If it is > 0, our own previous followup is currently
# being consumed by the agent — let it finish without re-firing.
if [ "$MEMPAL_LOOP_COUNT" -gt 0 ] 2>/dev/null; then
mempal_log "stop" "$MEMPAL_CONV_ID" "loop_count>0; letting agent stop"
mempal_emit '{}'
exit 0
fi
WING="$(mempal_infer_wing "$MEMPAL_WORKSPACE")"
# Build the followup message once; both the pending-marker branch and
# the threshold branch use it. Constructed via Python -c (rather than
# a heredoc) so we can pass the inferred wing as argv[1] and so the
# JSON encoding is correct even for wings whose name would otherwise
# need shell quoting.
_mempal_build_followup() {
"$MEMPAL_PYTHON_BIN" -c '
import json, sys
wing = sys.argv[1] if len(sys.argv) > 1 else "cursor_session"
msg = (
"MemPalace save checkpoint. Call mempalace_checkpoint ONCE with: "
"items=[{wing: " + wing + ", room: <short topic>, content: <verbatim "
"quote>}, ...] for the key topics, decisions, and verbatim quotes from "
"this session; and diary={agent_name: cursor-ide, wing: " + wing + ", "
"entry: <AAAK-format summary>}. It dedups, files non-duplicates, and "
"writes the diary in one call. Then stop."
)
print(json.dumps({"followup_message": msg}))
' "$WING"
}
# ── Pending-save marker from preCompact ───────────────────────────
#
# preCompact cannot itself emit a followup_message (Cursor docs:
# preCompact is observational-only, output supports only user_message),
# so it drops a marker file and we consume it here. Forces a save
# nudge regardless of the counter.
if mempal_consume_pending "$MEMPAL_CONV_ID"; then
mempal_log "stop" "$MEMPAL_CONV_ID" \
"consumed pending-save marker (post-compaction)"
if mempal_followup_silenced; then
mempal_log "stop" "$MEMPAL_CONV_ID" \
"followup silenced (MEMPAL_CURSOR_SILENT/MEMPAL_VERBOSE); emitting {}"
mempal_emit '{}'
exit 0
fi
_mempal_build_followup
exit 0
fi
# ── Normal counter path ───────────────────────────────────────────
COUNTER_FILE="$(_mempal_counter_path "$MEMPAL_CONV_ID")"
CURRENT="$(mempal_read_counter "$COUNTER_FILE")"
NEXT=$((CURRENT + 1))
mempal_write_counter_atomic "$COUNTER_FILE" "$NEXT" || {
mempal_log "stop" "$MEMPAL_CONV_ID" \
"WARN: counter write failed for $COUNTER_FILE; passing through"
mempal_emit '{}'
exit 0
}
mempal_log "stop" "$MEMPAL_CONV_ID" \
"counter $CURRENT -> $NEXT (interval=$SAVE_INTERVAL)"
# Trigger when we hit a multiple of SAVE_INTERVAL. Modulo arithmetic
# keeps the counter monotonically growing (no reset) so the log file
# is greppable for total turns across a conversation.
if [ "$((NEXT % SAVE_INTERVAL))" -ne 0 ]; then
mempal_emit '{}'
exit 0
fi
mempal_log "stop" "$MEMPAL_CONV_ID" "TRIGGERING SAVE at counter=$NEXT"
# ── Background mine (best effort) ─────────────────────────────────
#
# Two independent targets — both run if both are set:
# 1. transcript_path → its parent directory, --mode convos
# 2. MEMPAL_DIR (user-configured project) → --mode projects
#
# IMPORTANT (Cursor caveat): the --mode convos mine is BEST-EFFORT for
# Cursor. Cursor's transcript format is undocumented and
# mempalace/normalize.py has no Cursor parser, so this call does not
# yet produce clean verbatim conversation drawers — at best it ingests
# raw bytes. The verbatim-capture guarantee for Cursor is carried by
# the followup_message below, which drives the agent to file its own
# in-context verbatim quotes. The --mode projects target (MEMPAL_DIR)
# is unaffected — normalize.py reads ordinary project files fine.
#
# Both run with stdout/stderr appended to the cursor log and are
# backgrounded so a slow mine cannot push the hook past its
# Cursor-configured timeout. `command -v mempalace` gates so a user
# without the CLI on PATH (e.g. a fresh GUI-launched session) does
# not see a noisy error.
if command -v mempalace >/dev/null 2>&1; then
if mempal_is_valid_transcript "$MEMPAL_TRANSCRIPT" \
&& [ -f "$MEMPAL_TRANSCRIPT" ]; then
( mempalace mine "$(dirname "$MEMPAL_TRANSCRIPT")" --mode convos \
>> "$MEMPAL_CURSOR_LOG" 2>&1 ) &
elif [ -n "$MEMPAL_TRANSCRIPT" ]; then
mempal_log "stop" "$MEMPAL_CONV_ID" \
"skipping invalid transcript path: $MEMPAL_TRANSCRIPT"
fi
if [ -n "$MEMPAL_DIR" ] && [ -d "$MEMPAL_DIR" ]; then
( mempalace mine "$MEMPAL_DIR" --mode projects \
>> "$MEMPAL_CURSOR_LOG" 2>&1 ) &
fi
else
mempal_log "stop" "$MEMPAL_CONV_ID" \
"mempalace CLI not on PATH; skipping background mine"
fi
# The followup is the load-bearing verbatim path for Cursor (see header),
# so it fires by default. Honour the opt-out for users who want silence.
if mempal_followup_silenced; then
mempal_log "stop" "$MEMPAL_CONV_ID" \
"followup silenced (MEMPAL_CURSOR_SILENT/MEMPAL_VERBOSE); background mine only"
mempal_emit '{}'
exit 0
fi
_mempal_build_followup
+82
View File
@@ -0,0 +1,82 @@
#!/bin/bash
# MEMPALACE CURSOR WAKE HOOK — Session-start memory recall
#
# Cursor "sessionStart" hook. This is a Cursor-only capability —
# Claude Code's third-party-hooks compatibility layer does not have
# an equivalent event with the same "inject context into the agent's
# initial system message" semantics.
#
# Behaviour:
# 1. Parse Cursor's sessionStart payload (conversation_id /
# session_id, is_background_agent, composer_mode, plus the
# common workspace_roots field).
# 2. Infer the wing from basename(workspace_roots[0]).
# 3. Return {"additional_context": "..."} instructing the agent to
# scope its memory recall by calling mempalace_search +
# mempalace_diary_read with wing=<inferred>.
#
# sessionStart is documented as fire-and-forget — Cursor does not
# enforce a blocking response and does not consume "continue" /
# "user_message" — but "additional_context" is honoured and added to
# the conversation's initial system context. Verified in
# cursor.com/docs/hooks.md (fetched 2026-05-27).
#
# === INSTALL ===
#
# Add to ~/.cursor/hooks.json (or .cursor/hooks.json for project
# scope) under "sessionStart":
#
# {
# "version": 1,
# "hooks": {
# "sessionStart": [
# { "command": "/absolute/path/to/mempal_wake_hook_cursor.sh" }
# ]
# }
# }
_mempal_self="${BASH_SOURCE[0]:-$0}"
_mempal_dir="$(cd "$(dirname "$_mempal_self")" 2>/dev/null && pwd)"
# shellcheck source=lib/common.sh
. "$_mempal_dir/lib/common.sh"
if mempal_is_disabled; then
mempal_emit '{}'
exit 0
fi
INPUT="$(cat)"
mempal_parse_stdin "$INPUT"
if [ "$MEMPAL_PARSE_OK" != "1" ]; then
mempal_dump_bad_input "$INPUT" "sessionStart"
mempal_emit '{}'
exit 0
fi
WING="$(mempal_infer_wing "$MEMPAL_WORKSPACE")"
mempal_log "sessionStart" "$MEMPAL_CONV_ID" \
"workspace=$MEMPAL_WORKSPACE wing=$WING"
# Emit the additional_context payload via Python -c (rather than a
# heredoc) so the JSON encoding survives wings whose name contains
# characters that would otherwise need shell escaping, and so the
# inferred wing arrives as an argv positional. The MCP tool names
# referenced here are verified against mempalace/mcp_server.py:
# mempalace_search and mempalace_diary_read both exist and accept
# the wing parameter.
"$MEMPAL_PYTHON_BIN" -c '
import json, sys
wing = sys.argv[1] if len(sys.argv) > 1 else "cursor_session"
ctx = (
"MemPalace wake-up. The Cursor workspace maps to wing=" + wing + ". "
"Before answering anything that touches past work in this "
"project, call mempalace_search (wing=" + wing + ", "
"query=<relevant keywords>) and mempalace_diary_read "
"(agent_name=cursor-ide, wing=" + wing + ", last_n=10). "
"Use what you find verbatim where it answers the question; "
"never summarise the user'"'"'s own words."
)
print(json.dumps({"additional_context": ctx}))
' "$WING"
+3 -11
View File
@@ -119,16 +119,8 @@ INPUT=$(cat)
# backslashes are not mangled by echo flag parsing.
_mempal_parsed=$(
umask 077
printf '%s' "$INPUT" | "$MEMPAL_PYTHON_BIN" -c "
import sys, json, re
data = json.load(sys.stdin)
sid = data.get('session_id', '')
tp = data.get('transcript_path', '')
safe = lambda s: re.sub(r'[^a-zA-Z0-9_/.\-~]', '', str(s))
print('__MEMPAL_PARSE_OK__')
print(safe(sid))
print(safe(tp))
" 2>"$STATE_DIR/last_python_err.log"
printf '%s' "$INPUT" | "$MEMPAL_PYTHON_BIN" -m mempalace.hook_shell parse-precompact \
2>"$STATE_DIR/last_python_err.log"
)
# Drop the empty file on success; chmod 600 on failure to mirror
# last_input.log's privacy contract.
@@ -193,7 +185,7 @@ if is_valid_transcript_path "$TRANSCRIPT_PATH" && [ -f "$TRANSCRIPT_PATH" ]; the
mempalace mine "$(dirname "$TRANSCRIPT_PATH")" --mode convos \
>> "$STATE_DIR/hook.log" 2>&1
elif [ -n "$TRANSCRIPT_PATH" ]; then
echo "[$(date '+%H:%M:%S')] Skipping invalid transcript path: $TRANSCRIPT_PATH" \
echo "[$(date '+%H:%M:%S')] Skipping missing or invalid transcript path after normalization: $TRANSCRIPT_PATH" \
>> "$STATE_DIR/hook.log"
fi
if [ -n "$MEMPAL_DIR" ] && [ -d "$MEMPAL_DIR" ]; then
+6 -33
View File
@@ -149,21 +149,8 @@ INPUT=$(cat)
# ``printf '%s'`` removes the class of bug entirely.
_mempal_parsed=$(
umask 077
printf '%s' "$INPUT" | "$MEMPAL_PYTHON_BIN" -c "
import sys, json, re
data = json.load(sys.stdin)
sid = data.get('session_id', '')
sha_raw = data.get('stop_hook_active', False)
tp = data.get('transcript_path', '')
# Shell-safe output: only allow alphanumeric, underscore, hyphen, slash, dot, tilde
safe = lambda s: re.sub(r'[^a-zA-Z0-9_/.\-~]', '', str(s))
# Coerce stop_hook_active to strict boolean string
sha = 'True' if sha_raw is True or str(sha_raw).lower() in ('true', '1', 'yes') else 'False'
print('__MEMPAL_PARSE_OK__')
print(safe(sid))
print(sha)
print(safe(tp))
" 2>"$STATE_DIR/last_python_err.log"
printf '%s' "$INPUT" | "$MEMPAL_PYTHON_BIN" -m mempalace.hook_shell parse-stop \
2>"$STATE_DIR/last_python_err.log"
)
# The 2> redirect creates the file even when stderr is empty (success).
# Remove the empty file so the state directory stays clean on the happy
@@ -239,24 +226,10 @@ fi
# Count human messages in the JSONL transcript
# SECURITY: Pass transcript path as sys.argv to avoid shell injection via crafted paths
if [ -f "$TRANSCRIPT_PATH" ]; then
EXCHANGE_COUNT=$("$MEMPAL_PYTHON_BIN" - "$TRANSCRIPT_PATH" <<'PYEOF'
import json, sys
count = 0
with open(sys.argv[1]) as f:
for line in f:
try:
entry = json.loads(line)
msg = entry.get('message', {})
if isinstance(msg, dict) and msg.get('role') == 'user':
content = msg.get('content', '')
if isinstance(content, str) and '<command-message>' in content:
continue
count += 1
except:
pass
print(count)
PYEOF
2>/dev/null)
EXCHANGE_COUNT=$("$MEMPAL_PYTHON_BIN" -m mempalace.hook_shell count-human-messages "$TRANSCRIPT_PATH" 2>/dev/null)
elif [ -n "$TRANSCRIPT_PATH" ]; then
echo "[$(date '+%H:%M:%S')] WARN: transcript_path not found after normalization: $TRANSCRIPT_PATH" >> "$STATE_DIR/hook.log"
EXCHANGE_COUNT=0
else
EXCHANGE_COUNT=0
fi
+41
View File
@@ -0,0 +1,41 @@
#!/bin/bash
# MemPalace SessionEnd Hook — final save on clean exit.
#
# Claude Code documents a default SessionEnd hook timeout of 1.5s; a per-hook
# "timeout" in settings.local.json can raise it (up to 60s), but a
# plugin-provided timeout cannot (https://code.claude.com/docs/en/hooks). A cold
# `mempalace` start alone can exceed 1.5s, so we background the hook and return
# immediately; the detached child finishes the save after the session has
# exited. All logic lives in mempalace.hooks_cli for cross-harness extensibility.
run_mempalace_hook() {
if command -v mempalace >/dev/null 2>&1; then
exec mempalace hook run "$@"
fi
MEMPAL_PYTHON_BIN="${MEMPAL_PYTHON:-}"
if [ -z "$MEMPAL_PYTHON_BIN" ] || [ ! -x "$MEMPAL_PYTHON_BIN" ]; then
MEMPAL_PYTHON_BIN="$(command -v python3 2>/dev/null || echo python3)"
fi
if "$MEMPAL_PYTHON_BIN" -c "import mempalace" >/dev/null 2>&1; then
exec "$MEMPAL_PYTHON_BIN" -m mempalace hook run "$@"
fi
if command -v python >/dev/null 2>&1 && python -c "import mempalace" >/dev/null 2>&1; then
exec python -m mempalace hook run "$@"
fi
echo "MemPalace hook error: could not find a runnable mempalace command or module" >&2
exit 1
}
# Capture stdin (the SessionEnd JSON) before backgrounding — the parent's
# stdin is gone once we return. Forward it to the detached worker, which runs
# the final mine on its own time and outlives this process.
payload="$(cat)"
(
printf '%s' "$payload" | run_mempalace_hook --hook session-end --harness "${MEMPALACE_HOOK_HARNESS:-claude-code}"
) >/dev/null 2>&1 </dev/null &
disown 2>/dev/null || true
# Return immediately so the harness never blocks on session exit.
printf '{}'
+120
View File
@@ -0,0 +1,120 @@
# MemPalace Recall Protocol
The canonical "search before answering" protocol shared across every
MemPalace integration (Cursor, Antigravity, Claude Code, Codex,
OpenClaw). This file is the single source of truth — skills and rules
should link here rather than restating the protocol, so the rule never
drifts from the skill.
The protocol exists to honour MemPalace's foundational promise:
**100% recall, verbatim, never guess.** When the palace might hold the
answer, the agent must read the palace before answering from model
memory.
## When to recall
Search the palace **before answering** whenever the user asks about
anything that may already be filed:
- Past work, prior decisions, or "what did we do / decide / try?"
- A person, project, or entity ("who is …", "what is …")
- Something that happened in an earlier session ("remember when …",
"last time …", "the thing we discussed")
- A preference, fact, or relationship that could have changed over time
If the question is pure greenfield work with no memory relevance (e.g.
"rename this variable", "fix this typo"), do not search — recall is
question-driven, not reflexive.
## The protocol
1. **On wake-up** (if a session-start hook injected context, honour its wing scoping / `additional_context`): scope recall to the wing inferred from the workspace, then continue.
2. **Before responding** about people, projects, past events, or prior
decisions: call `mempalace_search` first. For relational or temporal
facts ("who reported to whom in March", "what was true then"), call
`mempalace_kg_query` instead or as well.
3. **If unsure** about a fact (name, age, relationship, preference): say
"let me check the palace" and query. Wrong is worse than slow.
4. **Return verbatim.** Quote the drawer's exact stored words. Never
summarize, paraphrase, or lossy-compress what the palace returns —
that is the whole point of the system.
5. **After a substantive session**, record continuity with
`mempalace_diary_write` (background hooks may already do this — do not
double-file).
6. **When a fact changes**, call `mempalace_kg_invalidate` on the old
fact, then `mempalace_kg_add` for the new one.
## Tool selection
| You need | Tool |
|---|---|
| Find any memory by meaning | `mempalace_search` (start here) |
| Relational / time-bound facts about an entity | `mempalace_kg_query` |
| The chronological story of an entity | `mempalace_kg_timeline` |
| Recent session continuity | `mempalace_diary_read` |
| Which wings / rooms exist (when scope unknown) | `mempalace_list_wings`, `mempalace_list_rooms` |
| Record this session | `mempalace_diary_write` |
`mempalace_search` takes a short natural-language `query` (keywords or a
question — not a system prompt or pasted conversation) plus optional
`wing` / `room` filters and `limit` (default 5).
## Unhappy paths
- **Empty results.** Say the palace has nothing on this; do not invent an
answer to fill the gap. Offer to widen the search (drop the wing
filter) or to file the new information.
- **MCP unavailable / tool error.** Surface the error plainly and suggest
the user verify the server (`mempalace status`, or re-run install).
Do not silently fall back to guessing from model memory.
- **Palace index corrupt / compactor error.** When the server returns an
error mentioning the HNSW segment writer, a ChromaDB compaction
failure, or a stuck "Not connected" state after a write, the on-disk
vector index is out of sync with `chroma.sqlite3` — but the drawer rows
are intact in SQLite. Recover by rebuilding the index from SQLite, not
by re-mining. See "Recovering a corrupt index" below. Do not attempt an
in-process repair from the agent; guide the user to run the CLI.
- **Stale or conflicting facts.** Prefer the knowledge graph's
time-valid answer; if a fact has changed, invalidate the old one and
add the new one rather than overwriting context silently.
## Recovering a corrupt index
A ChromaDB compaction failure can leave the drawers HNSW index out of
sync with `chroma.sqlite3` and wedge the MCP server (every call returns
"Not connected"). The data is safe in SQLite; rebuild the index from it.
Guide the user through these CLI steps — never run an in-process rebuild
from the agent (it can break other live clients):
1. Stop the MCP server (kill the `mempalace-mcp` process, or restart the
host editor).
2. Optional backup of the palace directory (`--archive-existing` already
moves the old palace aside, so this is belt-and-suspenders):
- macOS / Linux: `cp -a ~/.mempalace/palace ~/.mempalace/palace.bak.$(date +%F)`
- Windows (PowerShell): `Copy-Item -Recurse "$env:USERPROFILE\.mempalace\palace" "$env:USERPROFILE\.mempalace\palace.bak"`
3. Rebuild from SQLite:
`mempalace repair --mode from-sqlite --archive-existing --yes`
4. Verify: `mempalace repair-status` (divergence should read 0).
5. Restart the MCP server.
Do **not** re-mine from source files to recover: re-mining drops drawers
added through the MCP server and diary entries, which have no source file
(see MemPalace issue #1843).
## Anti-patterns
- Answering about past work, people, or decisions from model memory when
the palace might know — search first.
- Paraphrasing or summarizing stored content instead of quoting it
verbatim.
- Searching reflexively on every turn, including pure greenfield coding
with no memory relevance.
- Pasting the full conversation or a system prompt into the `query`
argument — keep queries short and keyword-driven.
## See also
- [`integrations/openclaw/SKILL.md`](../openclaw/SKILL.md) — the original
full-protocol skill this is distilled from.
- MemPalace design principles (verbatim, local-first, never summarize):
<https://github.com/MemPalace/mempalace>
+7
View File
@@ -0,0 +1,7 @@
{
"mcpServers": {
"mempalace": {
"command": "mempalace-mcp"
}
}
}
+1 -1
View File
@@ -16,7 +16,7 @@ The Python package that powers MemPalace. All modules, all logic.
| `dialect.py` | AAAK compression — entity codes, emotion markers, 30x lossless ratio |
| `knowledge_graph.py` | Temporal entity-relationship graph — SQLite, time-filtered queries, fact invalidation |
| `palace_graph.py` | Room-based navigation graph — BFS traversal, tunnel detection across wings |
| `mcp_server.py` | MCP server — 19 tools, AAAK auto-teach, Palace Protocol, agent diary |
| `mcp_server.py` | MCP server — 34 tools, AAAK auto-teach, Palace Protocol, agent diary |
| `onboarding.py` | Guided first-run setup — asks about people/projects, generates AAAK bootstrap + wing config |
| `entity_registry.py` | Entity code registry — maps names to AAAK codes, handles ambiguous names |
| `entity_detector.py` | Auto-detect people and projects from file content |
+4
View File
@@ -27,11 +27,13 @@ from .base import (
HealthStatus,
LexicalHit,
LexicalResult,
MaintenanceResult,
PalaceNotFoundError,
PalaceRef,
QueryResult,
UnsupportedCapabilityError,
UnsupportedFilterError,
UnsupportedMaintenanceKindError,
)
from .chroma import ChromaBackend, ChromaCollection
from .pgvector import PgVectorBackend, PgVectorCollection
@@ -64,6 +66,7 @@ __all__ = [
"HealthStatus",
"LexicalHit",
"LexicalResult",
"MaintenanceResult",
"PalaceNotFoundError",
"PalaceRef",
"PgVectorBackend",
@@ -75,6 +78,7 @@ __all__ = [
"SQLiteExactCollection",
"UnsupportedCapabilityError",
"UnsupportedFilterError",
"UnsupportedMaintenanceKindError",
"available_backends",
"detect_backend_for_path",
"detect_backends_for_path",
+71
View File
@@ -0,0 +1,71 @@
"""Shared embedder-identity sidecar (RFC 001).
A small JSON file in the palace directory, keyed by collection name, recording
the embedder identity (``model_name`` / ``dimension``). It is deliberately
*separate* from a backend's mismatch marker: a marker's presence signals
"palace initialized" (reads raise ``CollectionNotInitializedError`` when the
marker exists but the store doesn't), so recording identity at first empty open
must not create one. The sidecar is unguarded, so a brand-new palace can record
identity immediately — the same approach the chroma backend uses.
"""
import json
import os
from typing import Optional
EMBEDDER_SIDECAR_FILENAME = "mempalace_embedder.json"
def read_embedder_sidecar(path: Optional[str], collection_name: Optional[str]):
"""Return the recorded :class:`EmbedderIdentity` for ``collection_name``, or None.
Robust to a missing, unreadable, or malformed (non-dict) sidecar — any of
those degrade to ``None`` (the ``unknown`` state) rather than raising.
"""
from .base import EmbedderIdentity
if not path or not collection_name or not os.path.isfile(path):
return None
try:
with open(path, encoding="utf-8") as f:
data = json.load(f)
except (OSError, json.JSONDecodeError):
return None
if not isinstance(data, dict):
return None
entry = data.get(collection_name)
if not isinstance(entry, dict) or not entry.get("model_name"):
return None
return EmbedderIdentity(
model_name=str(entry["model_name"]),
dimension=int(entry.get("dimension") or 0),
)
def write_embedder_sidecar(path: Optional[str], collection_name: Optional[str], identity) -> None:
"""Record ``identity`` for ``collection_name`` in the sidecar, creating it if needed.
No-ops for a missing path, missing collection name, or a nameless identity.
Preserves other collections' entries; never raises on I/O failure.
"""
if not path or not collection_name or not identity or not getattr(identity, "model_name", ""):
return
data: dict = {}
if os.path.isfile(path):
try:
with open(path, encoding="utf-8") as f:
loaded = json.load(f)
if isinstance(loaded, dict):
data = loaded
except (OSError, json.JSONDecodeError):
data = {}
data[collection_name] = {
"model_name": str(identity.model_name),
"dimension": int(identity.dimension or 0),
}
try:
with open(path, "w", encoding="utf-8") as f:
json.dump(data, f, indent=2, ensure_ascii=False)
os.chmod(path, 0o600)
except (OSError, NotImplementedError):
pass
+237 -2
View File
@@ -14,8 +14,8 @@ conformance suite land in follow-up PRs.
"""
from abc import ABC, abstractmethod
from dataclasses import dataclass
from typing import ClassVar, Optional
from dataclasses import dataclass, field
from typing import ClassVar, Optional, Protocol, runtime_checkable
# ---------------------------------------------------------------------------
@@ -62,6 +62,15 @@ class UnsupportedCapabilityError(BackendError):
"""Raised when a backend does not implement an optional capability."""
class UnsupportedMaintenanceKindError(BackendError):
"""Raised when ``run_maintenance(kind)`` is called with an unadvertised kind.
A backend MUST advertise a kind in ``maintenance_kinds`` before it accepts
it (RFC 001). Advertising a kind it does not implement is a conformance
failure; a kind it has no analogue for MUST be omitted, not no-op'd.
"""
class BackendMismatchError(BackendError):
"""Raised when a selected backend does not match existing palace artifacts."""
@@ -74,6 +83,15 @@ class EmbedderIdentityMismatchError(BackendError):
"""Raised when the stored embedder model name differs from the current one."""
class EmbedderIdentityUnknownWarning(UserWarning):
"""Emitted on first open of a collection with no recorded embedder identity.
Legacy palaces created before identity tracking carry no model name. Per
RFC 001 the right behavior is warn-not-fail: the identity is recorded on
the next write and subsequent opens become strict.
"""
# ---------------------------------------------------------------------------
# Value objects
# ---------------------------------------------------------------------------
@@ -116,6 +134,114 @@ class PalaceRef:
namespace: Optional[str] = None
@dataclass(frozen=True)
class EmbedderIdentity:
"""Identity of the embedder that produced a collection's vectors (RFC 001).
``model_name`` is the stable identity persisted alongside a collection and
checked on subsequent opens. ``dimension`` is the vector width. A
``dimension`` of ``0`` means *unknown / not probed* — comparisons treat it
as "no dimension signal" rather than a real zero-width vector, so a cheap
read-path check can compare model names without loading the model.
"""
model_name: str
dimension: int = 0
@dataclass(frozen=True)
class MaintenanceResult:
"""Observable outcome of ``run_maintenance(kind)`` (RFC 001).
Maintenance is *not* fire-and-forget: a backend MUST serialize concurrent
same-kind runs and report the outcome so a caller can learn it must not
re-trigger. ``status`` is one of:
* ``"ran"`` — this call performed the maintenance.
* ``"already_running"`` — another caller holds the work; this call did
nothing and the caller MUST NOT re-trigger (the production index-build
wedge: concurrent writers each issuing the build stacked exclusive locks).
* ``"noop"`` — nothing needed doing (e.g. the index already exists).
``stats`` is free-form per kind (rows analyzed, bytes reclaimed, index
build time) for benchmark/operator reporting.
"""
kind: str
status: str
stats: dict = field(default_factory=dict)
@runtime_checkable
class Embedder(Protocol):
"""Minimal embedder contract (RFC 001, normative for identity checking).
The fuller embedder RFC (batching/async/pooling) is additive; identity
enforcement depends only on these three members.
"""
model_name: str
dimension: int
def embed(self, texts: list[str]) -> list[list[float]]: ...
def check_embedder_identity(
stored: Optional[EmbedderIdentity],
current: Optional[EmbedderIdentity],
*,
force_model_swap: bool = False,
) -> str:
"""Three-state embedder-identity check (RFC 001).
Returns the resolved state and raises on a hard, unforced conflict:
* ``"unknown"`` — no identity recorded yet (legacy collection), or the
current embedder is nameless. The caller warns and records on write.
* ``"known_match"`` — stored name (and dimension, when both known) equal
the current embedder. Proceed normally.
* ``"known_mismatch"`` — names or dimensions differ. Without
``force_model_swap`` this raises (:class:`EmbedderIdentityMismatchError`
for a model swap, :class:`DimensionMismatchError` for a width change,
which is checked first because mismatched vectors are physically
unusable). With ``force_model_swap`` it returns the state so the caller
can re-record the identity and log the swap.
A ``dimension`` of ``0`` on either side means "unknown" and is skipped, so
a model-name-only check (cheap read path) still works.
"""
if current is None or not current.model_name:
return "unknown"
if stored is None:
return "unknown"
dim_conflict = bool(stored.dimension and current.dimension) and (
stored.dimension != current.dimension
)
name_conflict = stored.model_name != current.model_name
if not dim_conflict and not name_conflict:
return "known_match"
if force_model_swap:
return "known_mismatch"
if dim_conflict:
raise DimensionMismatchError(
f"collection was built with a {stored.dimension}-dim embedder "
f"({stored.model_name!r}) but the current embedder is "
f"{current.dimension}-dim ({current.model_name!r}); the stored "
"vectors are incompatible. Re-embed the palace to switch models."
)
raise EmbedderIdentityMismatchError(
f"collection was built with embedder {stored.model_name!r} but the "
f"current embedder is {current.model_name!r}. Searching across a model "
"swap silently degrades recall. Re-embed the palace, or run "
"`mempalace palace set-embedder --model <name> --force` to record the "
"new identity if you know the vectors are compatible."
)
@dataclass(frozen=True)
class HealthStatus:
ok: bool
@@ -301,6 +427,101 @@ class BaseCollection(ABC):
def health(self) -> HealthStatus:
return HealthStatus.healthy()
@property
def distance_metric(self) -> str:
"""The space this collection's ``distances`` are reported in.
Defaults to the owning backend's declared metric (cosine for all
in-tree backends). Collections that can vary per-collection — e.g. a
legacy Chroma palace built without ``hnsw:space=cosine`` — override
this to report their actual space so core ranking converts correctly.
"""
return "cosine"
def get_stored_embedder_identity(self) -> Optional[EmbedderIdentity]:
"""Return the embedder identity recorded for this collection, if any.
Returns ``None`` when nothing is recorded — a legacy collection, or a
backend that does not yet persist identity. Core treats ``None`` as the
``unknown`` state (warn, do not fail). Backends override this and
:meth:`set_embedder_identity` against their own metadata store.
"""
return None
def set_embedder_identity(self, identity: EmbedderIdentity) -> None:
"""Persist this collection's embedder identity. Default: no-op.
A backend without an identity slot inherits the no-op default and so
stays permanently ``unknown`` (safe — it simply never enforces). The
enforcement choke point calls this when recording on first write or
on an explicit, forced model swap.
"""
return None
def effective_embedder_identity(self) -> Optional[EmbedderIdentity]:
"""The identity of the embedder this collection actually uses.
For ``server_embedder`` backends that ignore the injected embedder,
this reports the server-side embedder so the same identity rules apply
(RFC 001). Defaults to ``None`` — the collection is embedded by the
injected/core embedder, and the caller supplies the current identity.
"""
return None
def get_all_metadata(self, where: Optional[dict] = None) -> list[dict]:
"""Return every matching record's metadata in one logical pass (#1796).
Default implementation pages through :meth:`get` using
``limit``/``offset`` -- correct for backends with a real server-side
cursor (e.g. Chroma's SQL OFFSET), and the same shape callers already
relied on before this method existed.
Backends whose ``get(limit=, offset=)`` is implemented by fully
materializing a result set and then Python-slicing it (no true
server-side cursor) MUST override this method to walk their native
cursor exactly once instead. Calling the default implementation on
such a backend is O(n^2) in collection size: each page re-walks the
entire collection just to discard everything outside the requested
slice. See issue #1796.
"""
all_meta: list[dict] = []
offset = 0
page_size = 1000
while True:
kwargs: dict = {"include": ["metadatas"], "limit": page_size, "offset": offset}
if where:
kwargs["where"] = where
batch = self.get(**kwargs)
batch_meta = batch.metadatas if hasattr(batch, "metadatas") else batch.get("metadatas")
if not batch_meta:
break
all_meta.extend(batch_meta)
if len(batch_meta) < page_size:
break
offset += len(batch_meta)
return all_meta
def maintenance_state(self) -> dict:
"""Return a structured snapshot of this collection's maintenance state.
Free-form per backend (e.g. row count, whether a vector index exists,
last-analyze age). Used by benchmark harnesses to record state
alongside each latency/recall measurement so an un-analyzed store is
not compared against a settled one (RFC 001). Defaults to empty.
"""
return {}
def run_maintenance(self, kind: str) -> "MaintenanceResult":
"""Run a maintenance ``kind`` and return an observable result (RFC 001).
Backends advertise supported kinds in ``BaseBackend.maintenance_kinds``
and override this. The default supports nothing, so every kind raises
:class:`UnsupportedMaintenanceKindError`. Implementations MUST serialize
concurrent same-kind runs and report ``already_running`` rather than
stacking the work.
"""
raise UnsupportedMaintenanceKindError(f"backend does not support maintenance kind {kind!r}")
def lexical_search(
self,
*,
@@ -381,6 +602,20 @@ class BaseBackend(ABC):
name: ClassVar[str]
spec_version: ClassVar[str] = "1.0"
capabilities: ClassVar[frozenset[str]] = frozenset()
#: The space ``query()`` reports ``distances`` in (RFC 001 §2.1).
#: One of ``"cosine"`` | ``"l2"`` | ``"ip"``. The contract for the
#: ``distances`` field is *lower = closer* regardless of metric; core
#: search converts distance→similarity off this declaration rather than
#: assuming cosine. All in-tree backends are cosine today.
distance_metric: ClassVar[str] = "cosine"
#: Maintenance kinds this backend implements (RFC 001). Reserved names:
#: ``"analyze"`` (refresh planner/query statistics), ``"compact"`` (reclaim
#: space, rewrite storage), ``"reindex"`` (build/rebuild secondary indexes).
#: A backend with no analogue for a kind MUST omit it rather than declare a
#: no-op, so a benchmark harness can trust the set. Backends MAY add their
#: own kinds. ``run_maintenance`` raises ``UnsupportedMaintenanceKindError``
#: for anything not listed here.
maintenance_kinds: ClassVar[frozenset[str]] = frozenset()
@abstractmethod
def get_collection(
+139 -15
View File
@@ -9,6 +9,7 @@ import os
import pickle
import re
import sqlite3
import time
from collections import defaultdict
from numbers import Integral
from pathlib import Path
@@ -17,6 +18,8 @@ from typing import Any, Optional
import chromadb
from chromadb.errors import NotFoundError as _ChromaNotFoundError
from ..config import sqlite_read_uri
from ._sidecar import EMBEDDER_SIDECAR_FILENAME, read_embedder_sidecar, write_embedder_sidecar
from .base import (
BaseBackend,
BaseCollection,
@@ -456,7 +459,7 @@ def _vector_segment_id(palace_path: str, collection_name: str) -> Optional[str]:
if not os.path.isfile(db_path):
return None
try:
conn = sqlite3.connect(f"file:{db_path}?mode=ro", uri=True)
conn = sqlite3.connect(sqlite_read_uri(db_path), uri=True)
try:
row = conn.execute(
"""
@@ -605,6 +608,7 @@ def _hnsw_element_count(palace_path: str, segment_id: str) -> Optional[int]:
# sync_threshold) from expected steady-state lag.
_HNSW_DIVERGENCE_FALLBACK_FLOOR = 2000
_HNSW_DIVERGENCE_FRACTION = 0.10
_HNSW_PERSISTENT_DIVERGENCE_GRACE_SECONDS = 300.0
def _read_sync_threshold(palace_path: str, collection_name: str) -> int:
@@ -625,7 +629,7 @@ def _read_sync_threshold(palace_path: str, collection_name: str) -> int:
if not os.path.isfile(db_path):
return 1000
try:
conn = sqlite3.connect(f"file:{db_path}?mode=ro", uri=True)
conn = sqlite3.connect(sqlite_read_uri(db_path), uri=True)
try:
cur = conn.cursor()
cur.execute(
@@ -648,6 +652,45 @@ def _read_sync_threshold(palace_path: str, collection_name: str) -> int:
return 1000
def _collection_has_sync_threshold_metadata(palace_path: str, collection_name: str) -> bool:
"""Return True when the collection explicitly stores hnsw:sync_threshold."""
db_path = os.path.join(palace_path, "chroma.sqlite3")
if not os.path.isfile(db_path):
return False
try:
conn = sqlite3.connect(f"file:{db_path}?mode=ro", uri=True)
try:
row = conn.execute(
"""
SELECT 1
FROM collection_metadata cm
JOIN collections c ON cm.collection_id = c.id
WHERE c.name = ?
AND cm.key = 'hnsw:sync_threshold'
LIMIT 1
""",
(collection_name,),
).fetchone()
return row is not None
finally:
conn.close()
except Exception:
logger.debug("_collection_has_sync_threshold_metadata failed", exc_info=True)
return False
def _hnsw_metadata_age_seconds(palace_path: str, segment_id: str) -> Optional[float]:
"""Return index_metadata.pickle age in seconds, or None when unreadable."""
pickle_path = os.path.join(palace_path, segment_id, "index_metadata.pickle")
try:
return max(0.0, time.time() - os.path.getmtime(pickle_path))
except OSError:
return None
def hnsw_capacity_status(palace_path: str, collection_name: str = "mempalace_drawers") -> dict:
"""Compare sqlite embedding count against HNSW element count.
@@ -692,11 +735,15 @@ def hnsw_capacity_status(palace_path: str, collection_name: str = "mempalace_dra
hnsw_count = _hnsw_element_count(palace_path, seg_id)
out["hnsw_count"] = hnsw_count
sync_threshold = _read_sync_threshold(palace_path, collection_name)
# Two synchronization windows worth — see comment above
# _HNSW_DIVERGENCE_FALLBACK_FLOOR for the rationale.
divergence_floor = max(_HNSW_DIVERGENCE_FALLBACK_FLOOR, 2 * sync_threshold)
has_explicit_sync_threshold = _collection_has_sync_threshold_metadata(
palace_path,
collection_name,
)
metadata_age_seconds = (
_hnsw_metadata_age_seconds(palace_path, seg_id) if hnsw_count is not None else None
)
out["hnsw_metadata_age_seconds"] = metadata_age_seconds
if hnsw_count is None:
# No pickle yet, so this probe cannot measure HNSW capacity.
@@ -714,21 +761,53 @@ def hnsw_capacity_status(palace_path: str, collection_name: str = "mempalace_dra
divergence = sqlite_count - hnsw_count
out["divergence"] = divergence
threshold = max(divergence_floor, int(sqlite_count * _HNSW_DIVERGENCE_FRACTION))
if divergence > threshold:
# Newer palaces explicitly store mempalace's low sync threshold
# (currently 2), so a gap of dozens of rows is far beyond ordinary
# flush lag. Older palaces may lack the metadata row; keep the
# historical floor for fresh lag there, but do not let a stale pickle
# sit below the floor forever (#1816).
if has_explicit_sync_threshold:
threshold = max(0, 2 * sync_threshold)
else:
divergence_floor = max(_HNSW_DIVERGENCE_FALLBACK_FLOOR, 2 * sync_threshold)
threshold = max(
divergence_floor,
int(sqlite_count * _HNSW_DIVERGENCE_FRACTION),
)
out["threshold"] = threshold
stale_below_threshold = (
not has_explicit_sync_threshold
and divergence > 0
and metadata_age_seconds is not None
and metadata_age_seconds >= _HNSW_PERSISTENT_DIVERGENCE_GRACE_SECONDS
)
if divergence > threshold or stale_below_threshold:
out["status"] = "diverged"
out["diverged"] = True
pct = 100.0 * divergence / max(sqlite_count, 1)
if divergence > threshold:
reason = f"exceeds threshold {threshold:,}"
else:
age = metadata_age_seconds or 0.0
reason = f"persisted below the old flush-lag floor for {age:.0f}s"
out["message"] = (
f"HNSW index holds {hnsw_count:,} elements but sqlite has "
f"{sqlite_count:,} embeddings — {divergence:,} drawers ({pct:.0f}%) "
"are invisible to vector search. Run `mempalace repair` to rebuild."
f"{sqlite_count:,} embeddings - {divergence:,} drawers "
f"({pct:.0f}%) are missing from the flushed HNSW index "
f"({reason}). Vector reads are disabled until "
"`mempalace repair` rebuilds it."
)
else:
out["status"] = "ok"
out["message"] = (
f"HNSW {hnsw_count:,} / sqlite {sqlite_count:,} (within flush-lag tolerance)"
)
if divergence < 0:
out["message"] += " (HNSW has extra flushed elements; treating as safe)"
except Exception:
logger.debug("hnsw_capacity_status failed", exc_info=True)
out["message"] = "HNSW capacity probe raised; skipping"
@@ -745,7 +824,7 @@ def _sqlite_embedding_count(palace_path: str, collection_name: str) -> Optional[
if not os.path.isfile(db_path):
return None
try:
conn = sqlite3.connect(f"file:{db_path}?mode=ro", uri=True)
conn = sqlite3.connect(sqlite_read_uri(db_path), uri=True)
try:
row = conn.execute(
"""
@@ -806,7 +885,7 @@ def _sqlite_wing_room_counts(
if not os.path.isfile(db_path):
return None
try:
conn = sqlite3.connect(f"file:{db_path}?mode=ro", uri=True)
conn = sqlite3.connect(sqlite_read_uri(db_path), uri=True)
try:
# Wait out a transient writer/checkpoint lock rather than falling
# straight back to the expensive vector-index path (#1681).
@@ -925,7 +1004,12 @@ def _missing_dimensionality_appears_recoverable(
return False
label_count = len(id_to_label)
if int(total) != label_count or len(label_to_id) != label_count:
# total_elements_added is monotonic across every add, while id_to_label and
# label_to_id hold only live elements, so a segment that has had deletions
# carries total_elements_added > label_count. Require >= (not ==), otherwise
# every post-deletion dim-None segment is wrongly quarantined (#1710); the
# label-map size and bijection checks still reject inconsistent label maps.
if int(total) < label_count or len(label_to_id) != label_count:
return False
try:
return all(label_to_id.get(label) == item_id for item_id, label in id_to_label.items())
@@ -1067,7 +1151,7 @@ def _fix_blob_seq_ids(palace_path: str) -> None:
if os.path.isfile(marker):
return
try:
with sqlite3.connect(db_path) as conn:
with contextlib.closing(sqlite3.connect(db_path)) as conn:
try:
rows = conn.execute(
"SELECT rowid, seq_id FROM embeddings WHERE typeof(seq_id) = 'blob'"
@@ -1564,7 +1648,7 @@ class ChromaCollection(BaseCollection):
# rowid, embedding_id is the user-facing drawer id.
public_ids: dict[int, str] = {}
try:
conn = sqlite3.connect(f"file:{db_path}?mode=ro", uri=True)
conn = sqlite3.connect(sqlite_read_uri(db_path), uri=True)
conn.row_factory = sqlite3.Row
except sqlite3.Error:
logger.debug("Chroma lexical sqlite open failed", exc_info=True)
@@ -1724,6 +1808,46 @@ class ChromaCollection(BaseCollection):
"""
return self._collection.metadata or {}
@property
def distance_metric(self) -> str:
"""Report this collection's actual space from ``hnsw:space``.
MemPalace sets ``hnsw:space=cosine`` on every creation path, so a
healthy palace reports ``"cosine"``. When the key is absent, empty, or
an unrecognized value, the collection is genuinely using Chroma's HNSW
default — **L2** (Euclidean) — because cosine was never set on it. We
report ``"l2"`` in that case so core ranking maps the distances
correctly; reporting ``"cosine"`` here would reintroduce the
floor-every-result-to-zero misranking this property exists to fix.
"""
space = str(self.metadata.get("hnsw:space", "") or "").lower()
if space in ("cosine", "l2", "ip"):
return space
return "l2"
# ------------------------------------------------------------------
# Embedder identity (RFC 001)
#
# Stored in a small sidecar JSON in the palace dir rather than the Chroma
# collection metadata: ``collection.modify(metadata=...)`` replaces the
# whole dict and some Chroma versions reject re-passing the immutable
# ``hnsw:*`` construction keys, so mutating it on every open is fragile.
# The sidecar is keyed by collection name (a palace may hold several).
# This is complementary to Chroma's own embedding-function-name check —
# the core check runs at open time and yields the clean cross-backend
# error before Chroma's read-time rejection fires.
# ------------------------------------------------------------------
def _embedder_sidecar_path(self) -> Optional[str]:
if not self._palace_path:
return None
return os.path.join(self._palace_path, EMBEDDER_SIDECAR_FILENAME)
def get_stored_embedder_identity(self):
return read_embedder_sidecar(self._embedder_sidecar_path(), self._collection_name())
def set_embedder_identity(self, identity) -> None:
write_embedder_sidecar(self._embedder_sidecar_path(), self._collection_name(), identity)
# ---------------------------------------------------------------------------
# Backend
+26
View File
@@ -49,6 +49,32 @@ class EmbeddingCollection(BaseCollection):
def __getattr__(self, name):
return getattr(self._inner, name)
@property
def distance_metric(self) -> str:
# Explicit delegation: ``BaseCollection`` defines ``distance_metric``
# as a property, so it resolves on this subclass and ``__getattr__``
# never fires — without this override the wrapper would report the
# base "cosine" default and mask a wrapped non-cosine backend.
return self._inner.distance_metric
# Same shadowing reason as ``distance_metric``: these are concrete methods
# on ``BaseCollection``, so ``__getattr__`` never delegates them. Forward
# explicitly to the wrapped backend collection's identity store.
def get_stored_embedder_identity(self):
return self._inner.get_stored_embedder_identity()
def set_embedder_identity(self, identity) -> None:
return self._inner.set_embedder_identity(identity)
def effective_embedder_identity(self):
return self._inner.effective_embedder_identity()
def maintenance_state(self) -> dict:
return self._inner.maintenance_state()
def run_maintenance(self, kind: str):
return self._inner.run_maintenance(kind)
def add(self, *, documents, ids, metadatas=None, embeddings=None):
documents = _as_list(documents)
ids = _as_list(ids)
+279 -24
View File
@@ -37,6 +37,8 @@ from urllib import parse as urlparse
import numpy as np
from ..config import strip_lone_surrogates
from ._sidecar import EMBEDDER_SIDECAR_FILENAME, read_embedder_sidecar, write_embedder_sidecar
from .base import (
BackendClosedError,
BackendError,
@@ -79,6 +81,42 @@ def _json_dumps(obj: Any) -> str:
return json.dumps(obj or {}, ensure_ascii=False, separators=(",", ":"), sort_keys=True)
def _strip_nul(value: Any) -> Any:
"""Recursively strip NUL (0x00) from strings, list/tuple items, and dict keys
and values so pgvector can store the result.
PostgreSQL cannot store NUL in ``text`` or ``jsonb``: psycopg rejects a raw
NUL in a text column ("PostgreSQL text fields cannot contain NUL (0x00)
bytes"), and a NUL in metadata serializes to a JSON unicode escape that the
``jsonb`` cast rejects ("unsupported Unicode escape sequence"). A single
transcript that captured NUL in tool output would otherwise abort the whole
mine run (#1829). ChromaDB and the SQLite backend store the byte verbatim,
so stripping only here keeps the same inputs ingestible.
Applied to id, document, and metadata in :meth:`_PgVectorClient.upsert_rows`
so the write path never carries a NUL into Postgres. Only ``str`` values are
rewritten; the ``int``/``float``/``bool``/``None`` scalars JSON metadata
normalizes to pass through unchanged. Stripping is not injective, so two keys
(or ids)
differing only by a NUL collapse to one (last wins); this does not occur in
practice because drawer ids are SHA-256 hashes and metadata keys are fixed
field names, so only transcript-derived values are ever actually changed.
Unlike ``config.sanitize_content`` (which rejects NUL in user-supplied
content), the bulk-mine path strips so one stray byte cannot abort a whole
backfill. ``str.replace`` returns the original string when it holds no NUL,
so a clean document is not reallocated.
"""
if isinstance(value, str):
return value.replace("\x00", "")
if isinstance(value, dict):
return {_strip_nul(key): _strip_nul(item) for key, item in value.items()}
if isinstance(value, list):
return [_strip_nul(item) for item in value]
if isinstance(value, tuple):
return tuple(_strip_nul(item) for item in value)
return value
def _tokenize(text: str) -> list[str]:
if not text:
return []
@@ -352,6 +390,30 @@ def _quote_identifier(name: str) -> str:
return '"' + name.replace('"', '""') + '"'
# Session-level advisory-lock namespace for serializing HNSW index builds
# across daemon writers (RFC 001). classid is a fixed mempalace constant
# ("MEMP" in ASCII); objid is a stable per-table key. Both must fit a signed
# int4, which ``pg_advisory_lock(int4, int4)`` requires.
_MAINTENANCE_LOCK_CLASSID = 0x4D454D50 # "MEMP" — a positive, valid int4
def _advisory_objid(table: str) -> int:
"""Stable signed-int4 advisory key derived from the table name."""
raw = int(sha256(table.encode("utf-8")).hexdigest()[:8], 16) # 0 .. 2**32-1
return raw - 2**32 if raw >= 2**31 else raw
def _hnsw_index_name(table: str) -> str:
"""Deterministic, collision-safe index name for ``table``.
Routes through :func:`_pg_identifier`, which hashes the overflow when the
name exceeds Postgres' 63-byte limit. A naive ``[:63]`` truncation could
return the table name verbatim (tables and indexes share the ``pg_class``
namespace), which would fail with "relation already exists".
"""
return _pg_identifier(f"{table}_hnsw_idx")
def _field_sql(field: str, expression: Any, params: list) -> str:
"""Translate one field predicate to a JSONB containment expression."""
if isinstance(expression, dict):
@@ -440,11 +502,10 @@ class _PgVectorClient:
def __init__(self, config: _PgVectorConfig):
self._config = config
self._conn = None
self._closed = False
self._lock = threading.RLock()
def _connect(self):
if self._conn is not None and not getattr(self._conn, "closed", False):
return self._conn
try:
import psycopg
except ImportError as exc: # pragma: no cover - exercised only without the extra
@@ -452,15 +513,27 @@ class _PgVectorClient:
"pgvector backend requires the optional 'psycopg' dependency; "
"install mempalace[pgvector]"
) from exc
try:
self._conn = psycopg.connect(self._config.dsn)
except Exception as exc: # noqa: BLE001 - surface any driver failure uniformly
raise BackendError(f"pgvector connection failed: {exc}") from exc
return self._conn
# One client is shared across threads (PgVectorBackend caches a
# single instance per config), so the read-create-store on self._conn
# must hold the same lock _execute serializes on; unlocked, two
# first-connect threads each opened a connection and the loser leaked
# unclosed. The RLock makes the _execute -> _connect nesting safe. A
# stalled connect blocks peers under the lock the same way any
# in-flight query on this single shared connection already does.
with self._lock:
if self._closed:
raise BackendError("pgvector client has been closed")
if self._conn is not None and not getattr(self._conn, "closed", False):
return self._conn
try:
self._conn = psycopg.connect(self._config.dsn)
except Exception as exc: # noqa: BLE001 - surface any driver failure uniformly
raise BackendError(f"pgvector connection failed: {exc}") from exc
return self._conn
def _execute(self, sql: str, params=None, *, fetch: bool = False, many: bool = False):
conn = self._connect()
with self._lock:
conn = self._connect()
try:
with conn.cursor() as cur:
if many:
@@ -545,9 +618,21 @@ class _PgVectorClient:
)
params = [
(
row["id"],
row["document"],
_json_dumps(row.get("metadata")),
# Strip both unstorable byte classes Postgres rejects before
# binding, so one stray byte in a transcript cannot abort the
# whole mine (#1829 NUL, #1833 lone surrogate).
#
# Order matters for metadata: NUL must be stripped *before*
# serialization (json escapes it to \\u0000, which the jsonb cast
# rejects), while a lone surrogate must be stripped *after*
# serialization (json.dumps(ensure_ascii=False) leaves it raw, so
# one pass over the serialized string cleans it without walking
# the dict). id/document are plain strings, so the two passes
# commute there. ids are NUL- and surrogate-free in practice, so
# those passes are defensive no-ops on the ON CONFLICT key.
strip_lone_surrogates(_strip_nul(row["id"])),
strip_lone_surrogates(_strip_nul(row["document"])),
strip_lone_surrogates(_json_dumps(_strip_nul(row.get("metadata")))),
_vector_literal(row["embedding"]),
row.get("updated_at") or _utcnow(),
)
@@ -589,6 +674,8 @@ class _PgVectorClient:
*,
where: Optional[dict] = None,
with_embedding: bool = False,
limit: Optional[int] = None,
offset: Optional[int] = None,
) -> list[dict]:
qi = _quote_identifier(table)
params: list = []
@@ -597,6 +684,18 @@ class _PgVectorClient:
if with_embedding:
cols += ", embedding"
sql = f"SELECT {cols} FROM {qi} WHERE {where_sql}"
# Push pagination into SQL when a page is requested. ORDER BY the
# primary key gives OFFSET a stable order (an unordered scan may skip
# or repeat rows across pages); callers that scroll the whole table
# pass neither bound, leaving their SQL unchanged.
if limit is not None or offset:
sql += " ORDER BY id"
if limit is not None:
params.append(int(limit))
sql += " LIMIT %s"
if offset:
params.append(int(offset))
sql += " OFFSET %s"
rows = self._execute(sql, params, fetch=True)
return [
self._row(record, with_embedding=with_embedding, with_distance=False)
@@ -625,8 +724,46 @@ class _PgVectorClient:
def drop_table(self, table: str) -> None:
self._execute(f"DROP TABLE IF EXISTS {_quote_identifier(table)}")
# ------------------------------------------------------------------
# Maintenance (RFC 001)
# ------------------------------------------------------------------
def has_vector_index(self, table: str) -> bool:
rows = self._execute(
"SELECT 1 FROM pg_indexes WHERE schemaname = current_schema() "
"AND tablename = %s AND indexdef ILIKE %s",
[table, "%using hnsw%"],
fetch=True,
)
return bool(rows)
def try_advisory_lock(self, classid: int, objid: int) -> bool:
rows = self._execute("SELECT pg_try_advisory_lock(%s, %s)", [classid, objid], fetch=True)
return bool(rows and rows[0] and rows[0][0])
def advisory_unlock(self, classid: int, objid: int) -> None:
self._execute("SELECT pg_advisory_unlock(%s, %s)", [classid, objid], fetch=True)
def create_hnsw_index(self, table: str) -> None:
qi = _quote_identifier(table)
idx = _quote_identifier(_hnsw_index_name(table))
# Non-concurrent build takes ACCESS EXCLUSIVE for the build duration;
# the advisory lock in the caller ensures only one session builds, so
# writes are blocked once rather than by every writer that crossed the
# threshold (the production wedge this serialization fixes).
self._execute(
f"CREATE INDEX IF NOT EXISTS {idx} ON {qi} USING hnsw (embedding vector_cosine_ops)"
)
def analyze_table(self, table: str) -> None:
self._execute(f"ANALYZE {_quote_identifier(table)}")
def close(self) -> None:
# Terminal: the only caller is PgVectorBackend.close(), after which
# the backend refuses to hand the client out again. Without the flag a
# stale reference would silently reconnect and leak a session nobody
# can ever close.
with self._lock:
self._closed = True
if self._conn is not None:
try:
self._conn.close()
@@ -686,6 +823,14 @@ class PgVectorCollection(BaseCollection):
def _marker_exists(self) -> bool:
return self._backend._marker_exists(self._palace)
def get_stored_embedder_identity(self):
return self._backend._get_embedder_identity(self._palace, self._collection_name)
def set_embedder_identity(self, identity) -> None:
# Sidecar-backed (see PgVectorBackend), so this records even on a
# brand-new palace whose mismatch marker doesn't exist yet.
self._backend._set_embedder_identity(self._palace, self._collection_name, identity)
def _ensure_table(self, dimension: int) -> None:
if dimension <= 0:
raise ValueError("embedding dimension must be positive")
@@ -710,13 +855,19 @@ class PgVectorCollection(BaseCollection):
)
self._known_dimension = existing_dim or dimension
def _scroll(self, *, where=None, with_embedding=False) -> list[dict]:
def _scroll(self, *, where=None, with_embedding=False, limit=None, offset=None) -> list[dict]:
self._ensure_open()
if not self._table_exists():
if self._marker_exists():
raise CollectionNotInitializedError(self._collection_name)
return []
return self._client.scroll_rows(self._table, where=where, with_embedding=with_embedding)
return self._client.scroll_rows(
self._table,
where=where,
with_embedding=with_embedding,
limit=limit,
offset=offset,
)
def _rows(
self,
@@ -935,16 +1086,40 @@ class PgVectorCollection(BaseCollection):
include=None,
) -> GetResult:
spec = _IncludeSpec.resolve(include, default_distances=False)
rows = self._rows(
ids=ids, where=where, where_document=where_document, with_embedding=spec.embeddings
# Fast path for the common unfiltered page fetch (e.g.
# prefetch_mined_set's sweep): push LIMIT/OFFSET into the scan instead
# of fetching the whole table and slicing in Python, which is the
# O(rows x pages) cost this avoids. Only the no-filter case is pushed:
# the "metadata @> ..." pushdown is broader than the exact
# _matches_where re-filter for array/object values, so any filtered get
# keeps the full-scan path where that re-filter still runs. ids, where,
# where_document and negative bounds all fall through to the unchanged
# path below. (The document column is still selected for metadata-only
# pages; projecting it out needs the positional _row parser to change,
# so it stays a separate follow-up.)
push_page = (
ids is None
and not where
and not where_document
and (limit is None or limit >= 0)
and (offset is None or offset >= 0)
and (limit is not None or offset)
)
if ids is not None:
by_id = {row["id"]: row for row in rows}
rows = [by_id[doc_id] for doc_id in ids if doc_id in by_id]
if offset:
rows = rows[offset:]
if limit is not None:
rows = rows[:limit]
if push_page:
rows = self._scroll(
where=None, with_embedding=spec.embeddings, limit=limit, offset=offset
)
else:
rows = self._rows(
ids=ids, where=where, where_document=where_document, with_embedding=spec.embeddings
)
if ids is not None:
by_id = {row["id"]: row for row in rows}
rows = [by_id[doc_id] for doc_id in ids if doc_id in by_id]
if offset:
rows = rows[offset:]
if limit is not None:
rows = rows[:limit]
return GetResult(
ids=[row["id"] for row in rows],
documents=[row["document"] for row in rows] if spec.documents else [],
@@ -1008,6 +1183,60 @@ class PgVectorCollection(BaseCollection):
return HealthStatus.unhealthy(str(exc))
return HealthStatus.healthy()
def maintenance_state(self) -> dict:
empty = {"row_count": 0, "vector_index": None, "index_build_complete": False}
self._ensure_open()
try:
if not self._table_exists():
return empty
rows = self._client.count_rows(self._table)
has_index = self._client.has_vector_index(self._table)
except Exception: # noqa: BLE001 - state report must not raise
logger.debug("pgvector maintenance state probe failed", exc_info=True)
return empty
return {
"row_count": rows,
"vector_index": "hnsw" if has_index else None,
"index_build_complete": has_index,
}
def run_maintenance(self, kind: str):
from .base import MaintenanceResult, UnsupportedMaintenanceKindError
if kind not in PgVectorBackend.maintenance_kinds:
raise UnsupportedMaintenanceKindError(
f"pgvector does not support maintenance kind {kind!r}"
)
self._ensure_open()
# Nothing to maintain on a not-yet-materialized table (collection opened
# create=True but never written) — return noop rather than letting a
# raw "relation does not exist" error escape.
if not self._table_exists():
return MaintenanceResult(kind=kind, status="noop", stats={"reason": "no table"})
if kind == "analyze":
self._client.analyze_table(self._table)
return MaintenanceResult(kind="analyze", status="ran")
# reindex → build the optional HNSW index. Opt-in: it makes search
# approximate, trading the exact-scan 100%-recall default for scale.
# Serialized with a session advisory lock so concurrent daemon writers
# learn "already_running" instead of each stacking an ACCESS EXCLUSIVE
# index build.
if self._client.has_vector_index(self._table):
return MaintenanceResult(kind="reindex", status="noop", stats={"vector_index": "hnsw"})
classid, objid = _MAINTENANCE_LOCK_CLASSID, _advisory_objid(self._table)
if not self._client.try_advisory_lock(classid, objid):
return MaintenanceResult(kind="reindex", status="already_running")
try:
if self._client.has_vector_index(self._table): # re-check under lock
return MaintenanceResult(
kind="reindex", status="noop", stats={"vector_index": "hnsw"}
)
self._client.create_hnsw_index(self._table)
return MaintenanceResult(kind="reindex", status="ran", stats={"vector_index": "hnsw"})
finally:
self._client.advisory_unlock(classid, objid)
class PgVectorBackend(BaseBackend):
name = "pgvector"
@@ -1020,9 +1249,15 @@ class PgVectorBackend(BaseBackend):
"supports_metadata_filters",
"supports_lexical_search",
"supports_namespace_isolation",
"supports_server_side_indexes",
"server_mode",
}
)
# "compact" is omitted: Postgres autovacuum reclaims space automatically,
# so a manual VACUUM kind would be redundant. "reindex" builds the optional
# HNSW index — an opt-in scale lever, NOT on by default, because it makes
# vector search approximate (the exact ``<=>`` scan is the 100%-recall path).
maintenance_kinds = frozenset({"analyze", "reindex"})
def __init__(self):
self._clients: dict[_PgVectorConfig, _PgVectorClient] = {}
@@ -1140,11 +1375,31 @@ class PgVectorBackend(BaseBackend):
except (OSError, NotImplementedError):
pass
# Embedder identity lives in a sidecar, NOT the backend marker: the marker's
# presence signals "palace initialized" (reads raise CollectionNotInitialized
# when the marker exists but the remote table doesn't), so recording identity
# at first empty open must not create it. The sidecar is unguarded — like the
# chroma sidecar — so a brand-new palace can record identity immediately.
@staticmethod
def _embedder_sidecar_path(palace: PalaceRef) -> Optional[str]:
if not palace.local_path:
return None
return os.path.join(palace.local_path, EMBEDDER_SIDECAR_FILENAME)
def _get_embedder_identity(self, palace: PalaceRef, collection_name: str):
return read_embedder_sidecar(self._embedder_sidecar_path(palace), collection_name)
def _set_embedder_identity(self, palace: PalaceRef, collection_name: str, identity) -> None:
write_embedder_sidecar(self._embedder_sidecar_path(palace), collection_name, identity)
# ------------------------------------------------------------------
def _client(self, config: _PgVectorConfig) -> _PgVectorClient:
if self._closed:
raise BackendClosedError("PgVectorBackend has been closed")
with self._lock:
# Checked under the lock so a client cannot be created and stored
# concurrently with close() clearing the registry (mirrors
# SQLiteExactBackend._connect).
if self._closed:
raise BackendClosedError("PgVectorBackend has been closed")
client = self._clients.get(config)
if client is None:
client = _PgVectorClient(config)
+63 -2
View File
@@ -24,6 +24,7 @@ from urllib import request as urlrequest
import numpy as np
from ._sidecar import EMBEDDER_SIDECAR_FILENAME, read_embedder_sidecar, write_embedder_sidecar
from .base import (
BackendClosedError,
BackendMismatchError,
@@ -52,6 +53,20 @@ _PAYLOAD_DOCUMENT = "document"
_PAYLOAD_METADATA = "metadata"
_POINT_NAMESPACE = uuid.UUID("c06c3fc7-5c14-4dc4-84c2-24a5f72d8dc1")
_TOKEN_RE = re.compile(r"\w{2,}", re.UNICODE)
# Page size for Qdrant's /points/scroll cursor. 4096 (up from the original
# 256) cuts REST round-trips ~16x for any full-collection walk (#1796).
# Qdrant's own docs suggest larger scroll batches are safe, and this is well
# below typical REST payload-size limits for metadata-only (with_vector=False)
# scrolls such as get_all_metadata().
#
# This constant also governs vector-bearing scrolls (with_vector=True), used
# by _rows()/get() when embeddings are requested and by _query_local_exact()
# for the $or/$contains local-filter query fallback. At 4096 rows per page,
# high-dimensional embeddings make those particular responses tens of MB --
# Qdrant handles it and round-trips still drop overall, but this is a real
# trade-off, not a metadata-only optimization. (Noted in maintainer review
# on #1832.)
_SCROLL_PAGE_SIZE = 4096
_SUPPORTED_OPERATORS = frozenset(
{"$eq", "$ne", "$in", "$nin", "$and", "$or", "$contains", "$gt", "$gte", "$lt", "$lte"}
)
@@ -479,7 +494,7 @@ class _QdrantRESTClient:
collection: str,
*,
qdrant_filter: Optional[dict] = None,
limit: int = 256,
limit: int = _SCROLL_PAGE_SIZE,
offset: Any = None,
with_vector: bool = False,
) -> tuple[list[dict], Any]:
@@ -661,6 +676,14 @@ class QdrantCollection(BaseCollection):
def _marker_exists(self) -> bool:
return self._backend._marker_exists(self._palace)
def get_stored_embedder_identity(self):
return self._backend._get_embedder_identity(self._palace, self._collection_name)
def set_embedder_identity(self, identity) -> None:
# Sidecar-backed (see QdrantBackend), so this records even on a
# brand-new palace whose mismatch marker doesn't exist yet.
self._backend._set_embedder_identity(self._palace, self._collection_name, identity)
def _remote_dimension(self) -> Optional[int]:
try:
info = self._client.get_collection_info(self._remote_collection)
@@ -723,7 +746,7 @@ class QdrantCollection(BaseCollection):
points, offset = self._client.scroll_points(
self._remote_collection,
qdrant_filter=qdrant_filter,
limit=256,
limit=_SCROLL_PAGE_SIZE,
offset=offset,
with_vector=with_vector,
)
@@ -991,6 +1014,27 @@ class QdrantCollection(BaseCollection):
embeddings=[row["embedding"] or [] for row in rows] if spec.embeddings else None,
)
def get_all_metadata(self, where: Optional[dict] = None) -> list[dict]:
"""Return every matching record's metadata in one cursor pass (#1796).
Overrides the default offset-paginated implementation, which would
call self.get(limit=, offset=) in a loop -- and since self.get() is
backed by a full _scroll_all() materialization, each page of that
loop would re-walk the entire collection from the start just to
discard everything outside its slice (O(n^2) over collection size).
Delegates to self._rows(), the same single-scroll-plus-local-filter
helper that backs get()/delete(). With ids=None and
where_document=None, _rows() reduces to exactly one _scroll_all()
pass followed by an unconditional _matches_where() re-check on every
row -- the same filter logic get(), delete(), and lexical_search()
already use, so this can't independently drift from those call
sites. (Maintainer review on #1832: avoid duplicating the filter
dance inline.)
"""
rows = self._rows(where=where)
return [row["metadata"] for row in rows]
def delete(self, *, ids=None, where=None):
_validate_where(where)
if not self._remote_exists():
@@ -1175,6 +1219,23 @@ class QdrantBackend(BaseBackend):
except (OSError, NotImplementedError):
pass
# Embedder identity lives in a sidecar, NOT the backend marker: the marker's
# presence signals "palace initialized" (reads raise CollectionNotInitialized
# when the marker exists but the remote collection doesn't), so recording
# identity at first empty open must not create it. The sidecar is unguarded,
# so a brand-new palace can record identity immediately.
@staticmethod
def _embedder_sidecar_path(palace: PalaceRef) -> Optional[str]:
if not palace.local_path:
return None
return os.path.join(palace.local_path, EMBEDDER_SIDECAR_FILENAME)
def _get_embedder_identity(self, palace: PalaceRef, collection_name: str):
return read_embedder_sidecar(self._embedder_sidecar_path(palace), collection_name)
def _set_embedder_identity(self, palace: PalaceRef, collection_name: str, identity) -> None:
write_embedder_sidecar(self._embedder_sidecar_path(palace), collection_name, identity)
def _client(self, config: _QdrantConfig) -> _QdrantRESTClient:
if self._closed:
raise BackendClosedError("QdrantBackend has been closed")
+165 -27
View File
@@ -322,6 +322,36 @@ class SQLiteExactCollection(BaseCollection):
row = cur.execute("SELECT value FROM meta WHERE key = 'fts5_available'").fetchone()
return bool(row and row[0] == "1")
def _embedder_meta_key(self) -> str:
return f"embedder_model:{self._collection_name}"
def get_stored_embedder_identity(self):
from .base import EmbedderIdentity
with self._cursor() as cur:
try:
cid = self._collection_id(cur)
except CollectionNotInitializedError:
return None
row = cur.execute(
"SELECT value FROM meta WHERE key = ?",
(self._embedder_meta_key(),),
).fetchone()
if not row or not row[0]:
return None
dim = self._collection_dimension(cur, cid) or 0
return EmbedderIdentity(model_name=str(row[0]), dimension=int(dim))
def set_embedder_identity(self, identity) -> None:
if not identity or not identity.model_name:
return
with self._cursor() as cur:
cur.execute(
"INSERT INTO meta(key, value) VALUES (?, ?) "
"ON CONFLICT(key) DO UPDATE SET value = excluded.value",
(self._embedder_meta_key(), str(identity.model_name)),
)
def _replace_fts(self, cur, collection_id: int, doc_id: str, document: str) -> None:
if not self._fts_available(cur):
return
@@ -466,19 +496,32 @@ class SQLiteExactCollection(BaseCollection):
)
self._replace_fts(cur, collection_id, doc_id, doc)
def _rows(self, cur, *, where=None, where_document=None) -> list[dict]:
def _rows(self, cur, *, where=None, where_document=None, limit=None, offset=None) -> list[dict]:
_validate_where(where)
_validate_where(where_document)
collection_id = self._collection_id(cur)
rows = cur.execute(
"""
SELECT id, document, metadata_json, embedding
FROM documents
WHERE collection_id = ?
ORDER BY rowid
""",
(collection_id,),
).fetchall()
sql = (
"SELECT id, document, metadata_json, embedding\n"
"FROM documents\n"
"WHERE collection_id = ?\n"
"ORDER BY rowid"
)
params = [collection_id]
# Emit SQL LIMIT/OFFSET only on an unfiltered page. With a
# where/where_document the post-filter loop below drops rows *after*
# this scan, so a SQL LIMIT/OFFSET would cut the wrong rows; those
# callers scan in full and paginate in Python. SQLite requires a LIMIT
# before OFFSET, so an offset-only page uses "LIMIT -1" (unbounded).
if where is None and where_document is None and (limit is not None or offset):
if limit is not None:
sql += "\nLIMIT ?"
params.append(int(limit))
elif offset:
sql += "\nLIMIT -1"
if offset:
sql += "\nOFFSET ?"
params.append(int(offset))
rows = cur.execute(sql, params).fetchall()
out = []
for doc_id, doc, meta_json, emb_blob in rows:
meta = _json_loads(meta_json)
@@ -573,15 +616,33 @@ class SQLiteExactCollection(BaseCollection):
include=None,
) -> GetResult:
spec = _IncludeSpec.resolve(include, default_distances=False)
# Fast path for the common unfiltered page (e.g. the prefetch_mined_set
# and status sweeps): push LIMIT/OFFSET into the scan instead of
# materializing the whole collection and slicing in Python. Safe only
# with no post-filter (ids/where/where_document drop rows after the
# scan) and non-negative bounds: SQLite does not honor a negative LIMIT
# or OFFSET the way a Python slice does, so those keep the slice path.
push_page = (
ids is None
and where is None
and where_document is None
and (limit is None or limit >= 0)
and (offset is None or offset >= 0)
and (limit is not None or offset)
)
with self._cursor() as cur:
rows = self._rows(cur, where=where, where_document=where_document)
if ids is not None:
by_id = {row["id"]: row for row in rows}
rows = [by_id[doc_id] for doc_id in ids if doc_id in by_id]
if offset:
rows = rows[offset:]
if limit is not None:
rows = rows[:limit]
if push_page:
rows = self._rows(cur, limit=limit, offset=offset)
else:
rows = self._rows(cur, where=where, where_document=where_document)
if not push_page:
if ids is not None:
by_id = {row["id"]: row for row in rows}
rows = [by_id[doc_id] for doc_id in ids if doc_id in by_id]
if offset:
rows = rows[offset:]
if limit is not None:
rows = rows[:limit]
return GetResult(
ids=[row["id"] for row in rows],
documents=[row["document"] for row in rows] if spec.documents else [],
@@ -710,6 +771,63 @@ class SQLiteExactCollection(BaseCollection):
return HealthStatus.unhealthy("collection closed")
return HealthStatus.healthy()
def maintenance_state(self) -> dict:
try:
rows = self.count()
except Exception:
rows = 0
# vector_index is null by design — exact cosine over every row, no ANN.
state = {"row_count": rows, "vector_index": None}
try:
with self._cursor() as cur:
page_count = cur.execute("PRAGMA page_count").fetchone()
freelist = cur.execute("PRAGMA freelist_count").fetchone()
state["page_count"] = int(page_count[0]) if page_count else 0
state["freelist_pages"] = int(freelist[0]) if freelist else 0
except Exception:
pass
return state
def run_maintenance(self, kind: str):
from .base import MaintenanceResult, UnsupportedMaintenanceKindError
if kind not in SQLiteExactBackend.maintenance_kinds:
raise UnsupportedMaintenanceKindError(
f"sqlite_exact does not support maintenance kind {kind!r}"
)
if kind == "analyze":
# Refresh planner stats. Concurrent runs serialize on the handle lock.
with self._cursor() as cur:
cur.execute("ANALYZE")
return MaintenanceResult(kind="analyze", status="ran")
# compact → VACUUM. It cannot run inside a transaction, so flip the
# connection to autocommit for the duration. The handle lock serializes
# concurrent runs in-process; SQLite's own write lock serializes across
# processes.
before = self.maintenance_state()
with self._handle.lock:
self._ensure_open()
conn = self._handle.conn
prev_isolation = conn.isolation_level
try:
conn.commit()
conn.isolation_level = None
conn.execute("VACUUM")
finally:
conn.isolation_level = prev_isolation
after = self.maintenance_state()
reclaimed = max(0, before.get("page_count", 0) - after.get("page_count", 0))
return MaintenanceResult(
kind="compact",
status="ran",
stats={
"pages_before": before.get("page_count", 0),
"pages_after": after.get("page_count", 0),
"pages_reclaimed": reclaimed,
},
)
class SQLiteExactBackend(BaseBackend):
name = "sqlite_exact"
@@ -724,6 +842,9 @@ class SQLiteExactBackend(BaseBackend):
"local_mode",
}
)
# "reindex" is intentionally omitted: sqlite_exact does exact cosine over
# every row (no ANN index to build), so it has no analogue for it.
maintenance_kinds = frozenset({"analyze", "compact"})
def __init__(self):
self._clients: dict[str, _SQLiteExactHandle] = {}
@@ -746,19 +867,31 @@ class SQLiteExactBackend(BaseBackend):
os.chmod(palace_path, 0o700)
except (OSError, NotImplementedError):
pass
# Hold the registry lock across cache-check + connect + schema init:
# two threads first-opening the same palace must not each create a
# connection (the loser leaked unclosed and outlived close()) nor run
# _init_schema concurrently on a fresh file, which surfaces transient
# "database is locked" errors before WAL mode is established. Only
# first-open pays for the I/O under the lock; cache hits are a dict
# probe.
with self._clients_lock:
if self._closed:
raise BackendClosedError("SQLiteExactBackend has been closed")
cached = self._clients.get(palace_path)
if cached is not None and not cached.closed:
return cached
conn = sqlite3.connect(db_path, check_same_thread=False)
conn.row_factory = sqlite3.Row
lock = threading.RLock()
handle = _SQLiteExactHandle(conn, lock)
with handle.lock:
self._init_schema(conn)
with self._clients_lock:
conn = sqlite3.connect(db_path, check_same_thread=False)
try:
conn.row_factory = sqlite3.Row
lock = threading.RLock()
handle = _SQLiteExactHandle(conn, lock)
with handle.lock:
self._init_schema(conn)
except BaseException:
conn.close()
raise
self._clients[palace_path] = handle
return handle
return handle
def _init_schema(self, conn: sqlite3.Connection) -> None:
conn.executescript(
@@ -889,14 +1022,19 @@ class SQLiteExactBackend(BaseBackend):
cached.conn.close()
def close(self) -> None:
# Flip _closed under the registry lock so a concurrent _connect either
# sees the flag or finishes before the handle snapshot is taken; a
# connection can no longer slip into the registry after close().
# Unlocked readers of _closed elsewhere are advisory fast-fails; the
# locked recheck in _connect is the authoritative gate.
with self._clients_lock:
handles = list(self._clients.values())
self._clients.clear()
self._closed = True
for handle in handles:
with handle.lock:
handle.closed = True
handle.conn.close()
self._closed = True
def health(self, palace: Optional[PalaceRef] = None) -> HealthStatus:
if self._closed:
+74
View File
@@ -0,0 +1,74 @@
"""Retention pruning for timestamped palace backups.
``mempalace migrate`` and ``mempalace repair max-seq-id`` each write a fresh,
timestamped backup every time they run and historically never deleted the old
ones. On a machine that mines or repairs on a schedule those full-size copies
accumulate silently — a real palace was found with hundreds of gigabytes of
backups sitting beside only a few hundred megabytes of live data, nearly
filling the disk. This module prunes the backup set down to a bounded count
after each new backup is written.
The retention count comes from ``MempalaceConfig.max_backups`` (default 10).
"""
import glob
import os
import shutil
def prune_backups(pattern, max_backups, *, log=None):
"""Delete the oldest backups matching ``pattern`` so at most ``max_backups`` remain.
Args:
pattern: A glob pattern matching the backup paths (files or
directories). The caller is responsible for ``glob.escape``-ing
any literal, non-wildcard portion that can contain glob
metacharacters — palace paths sometimes do (e.g. a ``[``).
max_backups: Number of most-recent backups to keep. ``None`` or any
value ``<= 0`` disables pruning and returns immediately, so a
backup set is never touched when the user has opted out.
log: Optional callable (e.g. ``print``) for human-readable progress.
Returns:
The list of paths that were successfully removed.
Recency is determined by filesystem mtime rather than by parsing the
timestamp out of the name, so it stays correct even when two backup
producers use different timestamp formats. Deletion failures are logged
and skipped: pruning is best-effort cleanup and must never abort the
migrate/repair operation that just completed successfully.
"""
if max_backups is None or max_backups <= 0:
return []
scored = []
for path in glob.glob(pattern):
try:
scored.append((os.path.getmtime(path), path))
except OSError:
# Vanished between glob and stat (concurrent prune / cleanup);
# nothing for us to remove.
continue
if len(scored) <= max_backups:
return []
# Newest first; the path breaks mtime ties so ordering is deterministic.
scored.sort(key=lambda item: (item[0], item[1]), reverse=True)
removed = []
for _mtime, path in scored[max_backups:]:
try:
if os.path.isdir(path) and not os.path.islink(path):
shutil.rmtree(path)
else:
os.remove(path)
except OSError as exc:
if log:
log(f" Backup prune: could not remove {path}: {exc}")
continue
removed.append(path)
if log:
log(f" Backup prune: removed old backup {path}")
return removed
+313 -8
View File
@@ -53,6 +53,12 @@ _PASS_ZERO_LLM_PER_SAMPLE = 2_000 # for Tier 2 LLM call only
_PASS_ZERO_LLM_MAX_SAMPLES = 20 # caps the LLM-tier sample count
_EXPLICIT_BACKEND_ENV = "MEMPALACE_BACKEND_EXPLICIT"
# Keep parser construction lightweight for --version and hook commands.
# This mirrors miner.MAX_CHUNKS_PER_FILE without importing miner here;
# importing miner pulls in Chroma dependencies before argparse can handle
# lightweight exits such as --version.
_CLI_MAX_CHUNKS_PER_FILE_DEFAULT = 50_000
def _backend_arg(args):
"""Return a CLI-selected backend from subcommand or global flags."""
@@ -534,6 +540,27 @@ def cmd_mine(args):
for raw in args.include_ignored or []:
include_ignored.extend(part.strip() for part in raw.split(",") if part.strip())
if getattr(args, "background", False) and not getattr(args, "daemon", False):
print("mempalace: --background requires --daemon", file=sys.stderr)
sys.exit(2)
if getattr(args, "daemon", False):
payload = {
"source": args.dir,
"mode": args.mode,
"wing": args.wing,
"agent": args.agent,
"limit": args.limit,
"dry_run": args.dry_run,
"extract": args.extract,
"no_gitignore": args.no_gitignore,
"include_ignored": include_ignored,
"max_chunks_per_file": getattr(args, "max_chunks_per_file", None),
"redetect_origin": getattr(args, "redetect_origin", False),
}
_submit_daemon_cli_job("mine", payload, args, background=getattr(args, "background", False))
return
# --redetect-origin re-runs corpus_origin on the current corpus state
# and overwrites <palace>/.mempalace/origin.json before mining proceeds.
# Heuristic-only by design — full LLM detection lives on `mempalace init`.
@@ -655,14 +682,28 @@ def cmd_sweep(args):
def cmd_sync(args):
"""Prune drawers whose source files are gitignored, deleted, or moved (#1252)."""
from .mcp_server import _wal_log
palace_path = os.path.expanduser(args.palace) if args.palace else MempalaceConfig().palace_path
if getattr(args, "background", False) and not getattr(args, "daemon", False):
print("mempalace: --background requires --daemon", file=sys.stderr)
sys.exit(2)
if getattr(args, "daemon", False):
payload = {
"dir": args.dir,
"root": list(args.root or []),
"wing": args.wing,
"dry_run": args.dry_run,
}
_submit_daemon_cli_job("sync", payload, args, background=getattr(args, "background", False))
return
from .palace import MineAlreadyRunning
from .wal import _wal_log
from .backends import detect_backend_for_path
from .palace import _backend_artifact_label, resolve_backend_name
from .sync import sync_palace
palace_path = os.path.expanduser(args.palace) if args.palace else MempalaceConfig().palace_path
if not os.path.isdir(palace_path):
print(f"\n No palace found at {palace_path}")
return
@@ -745,6 +786,133 @@ def cmd_sync(args):
print(f"\n{'=' * 55}\n")
def _submit_daemon_cli_job(kind: str, payload: dict, args, *, background: bool) -> None:
palace_path = os.path.expanduser(args.palace) if args.palace else MempalaceConfig().palace_path
backend = _backend_arg(args)
from .daemon import DaemonError, submit_job
try:
job = submit_job(
kind,
payload,
palace_path=palace_path,
backend=backend,
wait=not background,
auto_start=True,
)
except DaemonError as exc:
print(f"mempalace: daemon submission failed: {exc}", file=sys.stderr)
sys.exit(1)
if background:
print(f"Submitted daemon job {job['id']} ({kind})")
return
result = job.get("result") or {}
from .service import print_job_result
exit_code = print_job_result(result)
if job.get("state") != "succeeded" and exit_code == 0:
error = job.get("error") or {}
print(
f"mempalace: daemon job failed: {error.get('message', 'unknown error')}",
file=sys.stderr,
)
exit_code = 1
if exit_code:
sys.exit(exit_code)
def cmd_daemon(args):
palace_path = os.path.expanduser(args.palace) if args.palace else MempalaceConfig().palace_path
backend = _backend_arg(args)
from .daemon import (
TERMINAL_STATES,
DaemonError,
QueueStore,
get_client_if_running,
job_to_dict,
queue_path,
start_daemon,
stop_daemon,
)
action = getattr(args, "daemon_action", None)
try:
if action == "start":
if args.foreground:
start_daemon(palace_path, backend=backend, foreground=True)
return
client = start_daemon(palace_path, backend=backend, foreground=False)
health = client.health()
print(f"MemPalace daemon running on 127.0.0.1:{client.port}")
print(f" Palace: {health.get('palace_path')}")
print(f" PID: {health.get('pid')}")
return
if action == "stop":
if stop_daemon(palace_path):
print("MemPalace daemon stopping")
else:
print("MemPalace daemon is not running")
return
if action == "status":
client = get_client_if_running(palace_path)
if client is None:
print("MemPalace daemon is not running")
sys.exit(1)
health = client.health()
print("MemPalace daemon is running")
print(f" Palace: {health.get('palace_path')}")
print(f" PID: {health.get('pid')}")
print(f" Active: {health.get('active_job_id') or '-'}")
print(f" Jobs: {health.get('counts') or {}}")
return
if action == "jobs":
client = get_client_if_running(palace_path)
if client is not None:
jobs = client.list_jobs(limit=args.limit)
else:
qpath = queue_path(palace_path)
if not qpath.exists():
jobs = []
else:
jobs = [
job_to_dict(job, include_payload=False)
for job in QueueStore(qpath).list(args.limit)
]
for job in jobs:
print(f"{job['id']} {job['state']:<9} {job['kind']:<10} {job['created_at']}")
return
if action == "wait":
client = get_client_if_running(palace_path)
if client is not None:
job = client.wait(args.job_id)
else:
qpath = queue_path(palace_path)
if not qpath.exists():
raise DaemonError("daemon is not running")
job = job_to_dict(QueueStore(qpath).get(args.job_id))
if job.get("state") not in TERMINAL_STATES:
raise DaemonError(f"daemon is not running; job {args.job_id} is {job['state']}")
result = job.get("result") or {}
from .service import print_job_result
exit_code = print_job_result(result)
if job.get("state") != "succeeded" and exit_code == 0:
print(f"mempalace: daemon job failed: {job.get('error')}", file=sys.stderr)
exit_code = 1
if exit_code:
sys.exit(exit_code)
return
except DaemonError as exc:
print(f"mempalace: daemon error: {exc}", file=sys.stderr)
sys.exit(1)
def cmd_search(args):
from .searcher import search, SearchError
@@ -831,6 +999,53 @@ def cmd_status(args):
status(palace_path=palace_path)
def cmd_palace_set_embedder(args):
"""Record (or force-override) a palace's embedder identity (RFC 001).
Resolves the ``unknown`` state for a legacy palace, or records a specific
model with ``--model``. It records identity on the palace only; it does not
change the configured model — when the two differ it prints how to align
``MEMPALACE_EMBEDDING_MODEL``. ``--force`` overwrites an existing,
differently-named identity.
"""
from .backends.base import EmbedderIdentityMismatchError
from .palace import set_palace_embedder_identity
config = MempalaceConfig()
palace_path = os.path.abspath(
os.path.expanduser(args.palace) if args.palace else config.palace_path
)
model = getattr(args, "model", None)
try:
old, new = set_palace_embedder_identity(
palace_path,
model=model,
force=getattr(args, "force", False),
backend=_backend_arg(args),
)
except EmbedderIdentityMismatchError as exc:
print(f" ✗ {exc}")
raise SystemExit(2) from exc
if old is None:
print(f" ✓ recorded embedder identity: {new.model_name} (dim={new.dimension})")
elif old.model_name == new.model_name:
print(f" ✓ embedder identity unchanged: {new.model_name} (dim={new.dimension})")
else:
print(
f" ✓ embedder identity changed: {old.model_name} → {new.model_name} "
f"(dim={new.dimension})"
)
# set-embedder records the palace's identity; it does not change the
# configured model. If they differ, the next normal open would mismatch —
# tell the user how to align them.
configured = config.embedding_model
if new.model_name and configured and new.model_name != configured:
print(
f" ⚠ configured model is {configured!r}; set MEMPALACE_EMBEDDING_MODEL="
f"{new.model_name} (or run onboarding) so normal opens of this palace match."
)
def cmd_repair_status(args):
"""Read-only HNSW capacity health check (#1222)."""
palace_path = os.path.expanduser(args.palace) if args.palace else MempalaceConfig().palace_path
@@ -842,7 +1057,12 @@ def cmd_repair_status(args):
def cmd_repair(args):
"""Rebuild palace vector index from SQLite metadata."""
"""Rebuild palace vector index from SQLite metadata.
On success the palace SQLite file is VACUUMed and the FTS5 index is
rebuilt, so the next repair's integrity preflight reads a consistent
database (#1747).
"""
config = MempalaceConfig()
collection_name = config.collection_name
palace_path = os.path.abspath(
@@ -859,8 +1079,10 @@ def cmd_repair(args):
TruncationDetected,
_close_chroma_handles,
_extract_drawers,
_post_rebuild_cleanup,
_rebuild_collection_via_temp,
check_extraction_safety,
index_read_recovery_guidance,
maybe_repair_poisoned_max_seq_id_before_rebuild,
print_sqlite_integrity_abort,
sqlite_integrity_errors,
@@ -974,7 +1196,7 @@ def cmd_repair(args):
print(f" Drawers found: {total}")
except Exception as e:
print(f" Error reading palace: {e}")
print(" Cannot recover — palace may need to be re-mined from source files.")
print(index_read_recovery_guidance())
return
if total == 0:
@@ -1052,6 +1274,10 @@ def cmd_repair(args):
print(f" Backup location: {backup_path}")
sys.exit(1)
# The bulk delete + re-upsert cycle above leaves the FTS5 inverted index
# inconsistent, which fails the next repair's integrity preflight (#1747).
_post_rebuild_cleanup(palace_path, backend=backend, progress=print)
print(f"\n Repair complete. {filed} drawers rebuilt.")
print(f" Backup saved at {backup_path}")
print(f"\n{'=' * 55}\n")
@@ -1423,13 +1649,22 @@ def main():
p_mine.add_argument(
"--dry-run", action="store_true", help="Show what would be filed without filing"
)
p_mine.add_argument(
"--daemon",
action="store_true",
help="Submit this mine to the opt-in local daemon queue",
)
p_mine.add_argument(
"--background",
action="store_true",
help="With --daemon, return a job id immediately instead of waiting",
)
p_mine.add_argument(
"--extract",
choices=["exchange", "general"],
default="exchange",
help="Extraction strategy for convos mode: 'exchange' (default) or 'general' (5 memory types)",
)
from . import miner as _miner_for_default
p_mine.add_argument(
"--max-chunks-per-file",
@@ -1438,7 +1673,7 @@ def main():
metavar="N",
help=(
f"Per-file chunk cap; files producing more chunks are skipped with a "
f"summary counter. Default {_miner_for_default.MAX_CHUNKS_PER_FILE} "
f"summary counter. Default {_CLI_MAX_CHUNKS_PER_FILE_DEFAULT} "
f"(or MEMPALACE_MAX_CHUNKS_PER_FILE). Set 0 to disable. Lower this on "
f"Windows if you hit ONNX bad_alloc (#1455)."
),
@@ -1486,6 +1721,16 @@ def main():
action="store_false",
help="Actually delete drawers (overrides --dry-run; requires --wing or a project root)",
)
p_sync.add_argument(
"--daemon",
action="store_true",
help="Submit this sync to the opt-in local daemon queue",
)
p_sync.add_argument(
"--background",
action="store_true",
help="With --daemon, return a job id immediately instead of waiting",
)
# search
p_search = sub.add_parser("search", help="Find anything, exact words")
@@ -1548,7 +1793,7 @@ def main():
p_hook_run.add_argument(
"--hook",
required=True,
choices=["session-start", "stop", "precompact"],
choices=["session-start", "stop", "session-end", "precompact"],
help="Hook name to run",
)
p_hook_run.add_argument(
@@ -1648,6 +1893,27 @@ def main():
help="Compare sqlite vs HNSW element counts (read-only; never opens a chromadb client)",
)
# daemon
p_daemon = sub.add_parser("daemon", help="Manage the opt-in long-lived daemon")
daemon_sub = p_daemon.add_subparsers(dest="daemon_action")
p_daemon_start = daemon_sub.add_parser("start", help="Start the daemon")
p_daemon_start.add_argument(
"--foreground",
action="store_true",
help="Run in the foreground for debugging or process supervisors",
)
p_daemon_start.add_argument(
"--backend",
default=None,
help="Storage backend for this daemon (default: config/env/detected/chroma)",
)
daemon_sub.add_parser("stop", help="Stop the daemon")
daemon_sub.add_parser("status", help="Show daemon status")
p_daemon_jobs = daemon_sub.add_parser("jobs", help="List recent daemon jobs")
p_daemon_jobs.add_argument("--limit", type=int, default=20, help="Max jobs to show")
p_daemon_wait = daemon_sub.add_parser("wait", help="Wait for a daemon job")
p_daemon_wait.add_argument("job_id", help="Job id returned by --background")
# mcp
p_mcp = sub.add_parser(
"mcp",
@@ -1693,6 +1959,31 @@ def main():
help="Storage backend to use for status (default: config/env/detected/chroma)",
)
p_palace = sub.add_parser("palace", help="Palace maintenance commands")
palace_sub = p_palace.add_subparsers(dest="palace_action")
p_set_embedder = palace_sub.add_parser(
"set-embedder",
help="Record/override the palace's embedder identity (resolve 'unknown', or switch models)",
)
p_set_embedder.add_argument(
"--model",
default=None,
help="Embedder model to record (default: current configured model). "
"Records identity on the palace only; does not change the configured "
"model (prints how to align MEMPALACE_EMBEDDING_MODEL if they differ).",
)
p_set_embedder.add_argument(
"--force",
action="store_true",
help="Overwrite an existing identity that names a different model "
"(only if you know the stored vectors are compatible)",
)
p_set_embedder.add_argument(
"--backend",
default=None,
help="Storage backend (default: config/env/detected/chroma)",
)
args = parser.parse_args()
_apply_backend_arg(args)
@@ -1717,6 +2008,20 @@ def main():
cmd_instructions(args)
return
if args.command == "palace":
if getattr(args, "palace_action", None) == "set-embedder":
cmd_palace_set_embedder(args)
else:
p_palace.print_help()
return
if args.command == "daemon":
if not getattr(args, "daemon_action", None):
p_daemon.print_help()
return
cmd_daemon(args)
return
dispatch = {
"init": cmd_init,
"mine": cmd_mine,
+107
View File
@@ -198,6 +198,27 @@ DEFAULT_PALACE_PATH = os.path.expanduser("~/.mempalace/palace")
DEFAULT_COLLECTION_NAME = "mempalace_drawers"
DEFAULT_BACKEND = "chroma"
# How many timestamped palace backups to retain before the oldest are
# pruned. Applies to the accumulating backups written by ``mempalace
# migrate`` and ``mempalace repair max-seq-id`` — see
# ``MempalaceConfig.max_backups``.
DEFAULT_MAX_BACKUPS = 10
def sqlite_read_uri(db_path: str) -> str:
"""Return a read-only ``file:`` URI for ``sqlite3.connect(..., uri=True)``.
A bare ``f"file:{db_path}?mode=ro"`` mis-parses paths containing spaces or
other URI-reserved characters — common in real home directories (a Windows
user folder like ``First Last``, many macOS paths). ``pathname2url``
percent-encodes the path and normalizes separators so the database opens on
every platform.
"""
from urllib.request import pathname2url
db_path = os.fspath(db_path)
return f"file:{pathname2url(db_path)}?mode=ro"
@lru_cache(maxsize=1)
def get_configured_collection_name() -> str:
@@ -326,6 +347,18 @@ class MempalaceConfig:
"""Path to the tunnel file, sibling of palace_path."""
return os.path.join(os.path.dirname(self.palace_path), "tunnels.json")
@property
def hallway_file(self):
"""Path to the hallway file, sibling of palace_path.
Mirrors ``tunnel_file`` so within-wing hallway state is scoped to the
configured palace and survives palace rebuilds (it does not live in
ChromaDB which can be recreated). Prior to this property the path was
hardcoded under ``~/.mempalace/hallways.json`` and multiple palaces on
one host silently shared one file (see ``hallways._legacy_hallway_file``).
"""
return os.path.join(os.path.dirname(self.palace_path), "hallways.json")
@property
def collection_name(self):
"""ChromaDB collection name."""
@@ -627,6 +660,37 @@ class MempalaceConfig:
return env_val.strip().lower()
return str(self._file_config.get("embedding_model", "minilm")).strip().lower()
@property
def embedding_threads(self) -> int:
"""Cap on the embedder's ONNX Runtime intra-op thread pool (#1068).
ChromaDB's ONNX embedder builds its ``InferenceSession`` with no thread
cap, so the intra-op pool defaults to the physical core count and a
background ``mine`` pins every core — stacked Stop-hook fires turn into
thermal events. ``OMP_NUM_THREADS`` is inert here (ORT owns its own
pool), so the cap is applied via ``SessionOptions`` in
:mod:`mempalace.embedding`.
Read from env ``MEMPALACE_EMBEDDING_THREADS`` first, then
``embedding_threads`` in ``config.json``. Semantics:
- unset / ``"auto"`` → half the logical CPUs (min 1), so a background
mine leaves the machine usable out of the box.
- a positive integer → exactly that many intra-op threads.
- ``0`` or negative → uncapped: ORT's default (physical core count),
for users who want maximum indexing throughput.
"""
raw = os.environ.get("MEMPALACE_EMBEDDING_THREADS")
if raw is None:
raw = self._file_config.get("embedding_threads")
if raw is None or str(raw).strip().lower() in ("", "auto"):
return max(1, (os.cpu_count() or 2) // 2)
try:
val = int(str(raw).strip())
except (TypeError, ValueError):
return max(1, (os.cpu_count() or 2) // 2)
return val if val > 0 else 0
def set_embedding_model(self, model: str) -> None:
"""Persist the embedding-model choice to ``config.json``.
@@ -691,6 +755,36 @@ class MempalaceConfig:
parsed = 1
return max(1, parsed)
@property
def max_backups(self) -> int:
"""Number of timestamped palace backups to retain before pruning.
Applies to the accumulating, timestamped backups created by
``mempalace migrate`` (``<palace>.pre-migrate.<timestamp>``) and
``mempalace repair max-seq-id``
(``chroma.sqlite3.max-seq-id-backup-<timestamp>``). Each of those
commands writes a fresh full-size copy every run and historically
never deleted the old ones, so on a machine that mines or repairs on
a schedule the backup set could silently grow until it filled the
disk. After each backup is written, copies beyond this count (oldest
first) are removed.
Reads ``MEMPALACE_MAX_BACKUPS`` env first, then ``max_backups`` in
``config.json``, then the default of ``10``. A value of ``0`` disables
pruning and keeps every backup (use when an external retention policy
manages cleanup). Negative or non-numeric values fall back to the
default rather than crashing migrate/repair.
"""
env_val = os.environ.get("MEMPALACE_MAX_BACKUPS")
if env_val is not None:
coerced = self._try_coerce_int(env_val, minimum=0)
if coerced is not None:
return coerced
coerced = self._try_coerce_int(
self._file_config.get("max_backups", DEFAULT_MAX_BACKUPS), minimum=0
)
return DEFAULT_MAX_BACKUPS if coerced is None else coerced
@property
def hook_silent_save(self):
"""Whether the stop hook saves directly (True) or blocks for MCP calls (False)."""
@@ -701,6 +795,19 @@ class MempalaceConfig:
"""Whether the stop hook shows a desktop notification via notify-send."""
return self._file_config.get("hooks", {}).get("desktop_toast", False)
@property
def hook_use_daemon(self):
"""Whether hooks should submit save/mine work to the opt-in daemon."""
env_val = os.environ.get("MEMPALACE_HOOKS_DAEMON")
if env_val is not None:
return env_val.lower() in ("true", "1", "yes", "on")
value = self._file_config.get("hooks", {}).get("daemon", False)
if isinstance(value, bool):
return value
if isinstance(value, str):
return value.lower() in ("true", "1", "yes", "on")
return value == 1
def set_hook_setting(self, key: str, value: bool):
"""Update a hook setting and write config to disk."""
if "hooks" not in self._file_config:
+12 -4
View File
@@ -594,15 +594,14 @@ def _mine_convos_impl(
wing = _resolve_wing(convo_path, wing)
files = scan_convos(convo_dir)
if limit > 0:
files = files[:limit]
print(f"\n{'=' * 55}")
print(" MemPalace Mine — Conversations")
print(f"{'=' * 55}")
print(f" Wing: {wing}")
print(f" Source: {convo_path}")
print(f" Files: {len(files)}")
limit_suffix = f" (limit: {limit} new)" if limit > 0 else ""
print(f" Files: {len(files)}{limit_suffix}")
print(f" Palace: {palace_path}")
if dry_run:
print(" DRY RUN — nothing will be filed")
@@ -620,10 +619,13 @@ def _mine_convos_impl(
)
total_drawers = 0
files_mined = 0
files_skipped = 0
files_processed = 0
room_counts = defaultdict(int)
for i, filepath in enumerate(files, 1):
files_processed = i
source_file = str(filepath)
# Skip if already filed at current NORMALIZE_VERSION
@@ -684,6 +686,9 @@ def _mine_convos_impl(
room_counts[c.get("memory_type", "general")] += 1
else:
room_counts[room] += 1
files_mined += 1
if limit > 0 and files_mined >= limit:
break
continue
if extract_mode != "general":
@@ -701,14 +706,17 @@ def _mine_convos_impl(
room_counts[r] += n
total_drawers += drawers_added
files_mined += 1
print(f" + [{i:4}/{len(files)}] {filepath.name[:50]:50} +{drawers_added}")
if limit > 0 and files_mined >= limit:
break
if not dry_run:
_validate_palace_fts5_after_mine(palace_path)
print(f"\n{'=' * 55}")
print(" Done.")
print(f" Files processed: {len(files) - files_skipped}")
print(f" Files processed: {files_processed - files_skipped}")
print(f" Files skipped (already filed): {files_skipped}")
print(f" Drawers filed: {total_drawers}")
if room_counts:
+1149
View File
File diff suppressed because it is too large Load Diff
+244 -58
View File
@@ -32,6 +32,8 @@ rather than hard-failing — mining must still work on a laptop without CUDA.
from __future__ import annotations
import logging
import os
import threading
from typing import Optional
logger = logging.getLogger(__name__)
@@ -56,6 +58,10 @@ _AUTO_ORDER = [
]
_EF_CACHE: dict = {}
# Check-then-construct on the cache must be atomic: without it, two threads
# resolving the same key each keep their own EF instance, and each instance
# later lazy-loads its own copy of the model.
_EF_CACHE_LOCK = threading.Lock()
_WARNED: set = set()
@@ -107,6 +113,35 @@ def _resolve_providers(device: str) -> tuple[list, str]:
return (requested, device)
def _intra_op_session_options(intra_op_num_threads: int):
"""Build ORT ``SessionOptions`` capping the intra-op thread pool (#1068).
Returns ``None`` when ``intra_op_num_threads <= 0`` so the caller leaves
ORT at its default (≈ physical core count). ChromaDB's embedder ignores
``OMP_NUM_THREADS`` — ORT owns its own intra-op pool, settable only via
``SessionOptions`` at session construction — so a cap has to be threaded
through here rather than via the environment.
"""
if not intra_op_num_threads or intra_op_num_threads <= 0:
return None
import onnxruntime as ort
so = ort.SessionOptions()
so.intra_op_num_threads = intra_op_num_threads
return so
def _resolve_intra_op_threads() -> int:
"""Read the configured ORT intra-op thread cap (``0`` = uncapped, #1068)."""
try:
from .config import MempalaceConfig
return MempalaceConfig().embedding_threads
except Exception:
logger.debug("embedding_threads resolution failed; leaving ORT default", exc_info=True)
return 0
def _build_ef_class():
"""Subclass ``ONNXMiniLM_L6_V2`` with name ``"default"``.
@@ -117,13 +152,51 @@ def _build_ef_class():
palaces created with ``DefaultEmbeddingFunction`` *and* palaces we
create ourselves, with the same GPU-capable ``preferred_providers``.
"""
from functools import cached_property
from chromadb.utils.embedding_functions import ONNXMiniLM_L6_V2
class _MempalaceONNX(ONNXMiniLM_L6_V2):
def __init__(self, preferred_providers=None, intra_op_num_threads=0):
super().__init__(preferred_providers=preferred_providers)
self._intra_op_num_threads = intra_op_num_threads
@staticmethod
def name() -> str:
return "default"
@cached_property
def model(self):
# Upstream builds the InferenceSession with no intra-op thread cap,
# so ORT defaults its pool to the physical core count and a
# background mine pins every core (#1068). Rebuild the session the
# same way upstream does (same SessionOptions, same CoreML pruning,
# same model path) but with our cap applied. If upstream's
# internals shift, fall back to its uncapped build so embedding
# still works.
cap = getattr(self, "_intra_op_num_threads", 0)
if not cap or cap <= 0:
return super().model
try:
ort = self.ort
providers = self._preferred_providers or ort.get_available_providers()
providers = [p for p in providers if p != "CoreMLExecutionProvider"]
so = ort.SessionOptions()
so.log_severity_level = 3
so.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
so.intra_op_num_threads = cap
return ort.InferenceSession(
os.path.join(self.DOWNLOAD_PATH, self.EXTRACTED_FOLDER_NAME, "model.onnx"),
providers=providers,
sess_options=so,
)
except Exception:
logger.warning(
"thread-capped ORT session build failed; using ORT defaults",
exc_info=True,
)
return super().model
return _MempalaceONNX
@@ -135,6 +208,16 @@ _EMBEDDINGGEMMA_ONNX = "model_quantized.onnx"
_EMBEDDINGGEMMA_PREFIX = "task: sentence similarity | query: "
_EMBEDDINGGEMMA_DIM = 384 # Matryoshka truncation — first 384 dims of the 768
_EMBEDDINGGEMMA_MAX_LEN = 2048
# Default docs per session.run. The ONNX graph has no internal batching,
# so one unchunked run over a repair-scale batch (5000 docs, repair.py/
# cli.py) allocates attention buffers that grow with batch size and
# superlinearly with padded length (score tensors are batch x heads x
# len^2 per layer), and the kernel OOM-kills the process (#1770). 32
# matches the internal batch size of chromadb's ONNXMiniLM_L6_V2, whose
# chunked _forward survives the same call sites. embeddinggemma's
# sentence_embedding output is attention-masked, so sub-batch padding
# does not change any row's vector.
_EMBEDDINGGEMMA_BATCH_SIZE = 32
class EmbeddinggemmaONNX:
@@ -158,73 +241,115 @@ class EmbeddinggemmaONNX:
# when switching models. Keep it stable.
return "embeddinggemma_300m"
def __init__(self, preferred_providers=None):
def __init__(
self,
preferred_providers=None,
batch_size: int = _EMBEDDINGGEMMA_BATCH_SIZE,
intra_op_num_threads: int = 0,
):
if batch_size < 1:
raise ValueError(f"batch_size must be >= 1, got {batch_size}")
self._providers = (
list(preferred_providers) if preferred_providers else ["CPUExecutionProvider"]
)
self._batch_size = batch_size
self._intra_op_num_threads = intra_op_num_threads
self._session = None
self._tokenizer = None
self._np = None
self._output_idx = None
# Instances are shared across threads via _EF_CACHE; serialize the
# one-time model load so concurrent cold calls cannot build (and
# transiently hold) two full model sessions.
self._load_lock = threading.Lock()
def _lazy_load(self) -> None:
if self._session is not None:
return
try:
import numpy as np
import onnxruntime as ort
from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer
except ImportError as e:
raise ImportError(
"EmbeddinggemmaONNX requires huggingface_hub, tokenizers, and "
"numpy — these ship with mempalace core, so this error usually "
"means one was uninstalled or pinned to an incompatible version. "
"Reinstall with: pip install --upgrade --force-reinstall mempalace"
) from e
with self._load_lock:
if self._session is not None:
return
try:
import numpy as np
import onnxruntime as ort
from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer
except ImportError as e:
raise ImportError(
"EmbeddinggemmaONNX requires huggingface_hub, tokenizers, and "
"numpy — these ship with mempalace core, so this error usually "
"means one was uninstalled or pinned to an incompatible version. "
"Reinstall with: pip install --upgrade --force-reinstall mempalace"
) from e
logger.info(
"Downloading %s/%s (cached after first run)…",
_EMBEDDINGGEMMA_REPO,
_EMBEDDINGGEMMA_ONNX,
)
model_path = hf_hub_download(
_EMBEDDINGGEMMA_REPO, subfolder="onnx", filename=_EMBEDDINGGEMMA_ONNX
)
hf_hub_download(
_EMBEDDINGGEMMA_REPO, subfolder="onnx", filename=_EMBEDDINGGEMMA_ONNX + "_data"
)
tok_path = hf_hub_download(_EMBEDDINGGEMMA_REPO, filename="tokenizer.json")
logger.info(
"Downloading %s/%s (cached after first run)…",
_EMBEDDINGGEMMA_REPO,
_EMBEDDINGGEMMA_ONNX,
)
model_path = hf_hub_download(
_EMBEDDINGGEMMA_REPO, subfolder="onnx", filename=_EMBEDDINGGEMMA_ONNX
)
hf_hub_download(
_EMBEDDINGGEMMA_REPO, subfolder="onnx", filename=_EMBEDDINGGEMMA_ONNX + "_data"
)
tok_path = hf_hub_download(_EMBEDDINGGEMMA_REPO, filename="tokenizer.json")
self._session = ort.InferenceSession(model_path, providers=self._providers)
out_names = [o.name for o in self._session.get_outputs()]
# Model card: sentence_embedding is the pooled output (last_hidden_state
# is the per-token output we don't want).
self._output_idx = (
out_names.index("sentence_embedding") if "sentence_embedding" in out_names else 1
)
session = ort.InferenceSession(
model_path,
sess_options=_intra_op_session_options(self._intra_op_num_threads),
providers=self._providers,
)
out_names = [o.name for o in session.get_outputs()]
# Model card: sentence_embedding is the pooled output (last_hidden_state
# is the per-token output we don't want).
output_idx = (
out_names.index("sentence_embedding") if "sentence_embedding" in out_names else 1
)
tokenizer = Tokenizer.from_file(tok_path)
tokenizer.enable_padding()
tokenizer.enable_truncation(max_length=_EMBEDDINGGEMMA_MAX_LEN)
self._tokenizer = tokenizer
self._np = np
tokenizer = Tokenizer.from_file(tok_path)
tokenizer.enable_padding()
tokenizer.enable_truncation(max_length=_EMBEDDINGGEMMA_MAX_LEN)
self._output_idx = output_idx
self._tokenizer = tokenizer
self._np = np
# Session is assigned last: the unlocked fast path above treats a
# non-None session as "fully loaded", so every other attribute
# must already be in place when it becomes visible.
self._session = session
def __call__(self, input): # noqa: A002 — ChromaDB EF protocol uses `input`
def __call__(self, input: str | list[str] | None) -> list[list[float]]: # noqa: A002 — ChromaDB EF protocol
if isinstance(input, str):
# A bare string would be iterated character by character below,
# silently producing one garbage vector per character.
input = [input]
if input is None or len(input) == 0:
# None or zero docs: nothing to embed; skip the lazy model
# download. len() over truthiness so an array-like documents
# sequence is not rejected by ambiguous-truth-value semantics.
return []
self._lazy_load()
np = self._np
texts = [_EMBEDDINGGEMMA_PREFIX + t for t in input]
encs = self._tokenizer.encode_batch(texts)
input_ids = np.asarray([e.ids for e in encs], dtype=np.int64)
attention_mask = np.asarray([e.attention_mask for e in encs], dtype=np.int64)
outputs = self._session.run(
None, {"input_ids": input_ids, "attention_mask": attention_mask}
)
sent_emb = outputs[self._output_idx][:, :_EMBEDDINGGEMMA_DIM]
# L2-normalize so cosine similarity == dot product (matches what the
# MTEB methodology assumes; ChromaDB's distance is configured for it).
norms = np.linalg.norm(sent_emb, axis=1, keepdims=True) + 1e-12
return (sent_emb / norms).tolist()
embeddings: list[list[float]] = []
# Tokenize and run per sub-batch, not over the whole input: padding
# is to the longest sequence in the sub-batch, and the ONNX runtime
# only ever holds batch_size rows of attention buffers at a time
# (#1770).
for start in range(0, len(input), self._batch_size):
chunk = input[start : start + self._batch_size]
texts = [_EMBEDDINGGEMMA_PREFIX + t for t in chunk]
encs = self._tokenizer.encode_batch(texts)
input_ids = np.asarray([e.ids for e in encs], dtype=np.int64)
attention_mask = np.asarray([e.attention_mask for e in encs], dtype=np.int64)
outputs = self._session.run(
None, {"input_ids": input_ids, "attention_mask": attention_mask}
)
sent_emb = outputs[self._output_idx][:, :_EMBEDDINGGEMMA_DIM]
# L2-normalize so cosine similarity == dot product (matches what the
# MTEB methodology assumes; ChromaDB's distance is configured for it).
norms = np.linalg.norm(sent_emb, axis=1, keepdims=True) + 1e-12
embeddings.extend((sent_emb / norms).tolist())
return embeddings
def embed_query(self, input: list[str]) -> list[list[float]]: # noqa: A002 — ChromaDB EF protocol
"""Embed query documents (ChromaDB EF protocol)."""
@@ -254,18 +379,23 @@ def get_embedding_function(device: Optional[str] = None, model: Optional[str] =
providers, effective = _resolve_providers(device)
cache_key = (model, tuple(providers))
cached = _EF_CACHE.get(cache_key)
cached = _EF_CACHE.get(cache_key) # lock-free fast path; dict.get is GIL-atomic
if cached is not None:
return cached
with _EF_CACHE_LOCK:
cached = _EF_CACHE.get(cache_key)
if cached is not None:
return cached
if model == "embeddinggemma":
ef = EmbeddinggemmaONNX(preferred_providers=providers)
else:
# Default: minilm (or anything we don't recognize — back-compat win).
ef_cls = _build_ef_class()
ef = ef_cls(preferred_providers=providers)
threads = _resolve_intra_op_threads()
if model == "embeddinggemma":
ef = EmbeddinggemmaONNX(preferred_providers=providers, intra_op_num_threads=threads)
else:
# Default: minilm (or anything we don't recognize — back-compat win).
ef_cls = _build_ef_class()
ef = ef_cls(preferred_providers=providers, intra_op_num_threads=threads)
_EF_CACHE[cache_key] = ef
_EF_CACHE[cache_key] = ef
logger.info(
"Embedding function initialized (model=%s device=%s providers=%s)",
model,
@@ -287,3 +417,59 @@ def describe_device(device: Optional[str] = None) -> str:
device = MempalaceConfig().embedding_device
_, effective = _resolve_providers(device)
return effective
# Probed vector widths, keyed by resolved model name. Populated once per
# process the first time an identity is resolved for a model.
_DIM_CACHE: dict = {}
def current_model_name(model: Optional[str] = None) -> str:
"""Resolve the canonical embedder model name (cheap, no model load).
This is the configured ``embedding_model`` (``"minilm"`` /
``"embeddinggemma"`` / ...), not the embedding function's internal
``name()`` (which is spoofed to ``"default"`` for ChromaDB compatibility).
"""
if model is not None:
return str(model).strip().lower()
from .config import MempalaceConfig
return MempalaceConfig().embedding_model
def probe_dimension(device: Optional[str] = None, model: Optional[str] = None) -> int:
"""Return the embedder's output dimension by embedding a short probe.
Model-agnostic — works for any model without a hardcoded table — and
cached per resolved model name so the probe is paid at most once per
process. Returns ``0`` if the probe fails (treated as "dimension unknown"
by the identity check, so a probe failure never blocks normal operation).
"""
name = current_model_name(model)
cached = _DIM_CACHE.get(name)
if cached is not None:
return cached
try:
ef = get_embedding_function(device=device, model=model)
vectors = ef(input=["probe"])
dim = len(vectors[0]) if vectors and vectors[0] is not None else 0
except Exception:
logger.debug("Embedding dimension probe failed for model=%s", name, exc_info=True)
dim = 0
_DIM_CACHE[name] = dim
return dim
def get_embedder_identity(device: Optional[str] = None, model: Optional[str] = None):
"""Resolve the current embedder identity (RFC 001).
``model_name`` from config (cheap); ``dimension`` from a cached one-time
probe. Returns an :class:`~mempalace.backends.base.EmbedderIdentity`.
"""
from .backends.base import EmbedderIdentity
return EmbedderIdentity(
model_name=current_model_name(model),
dimension=probe_dimension(device=device, model=model),
)
+14 -6
View File
@@ -507,7 +507,7 @@ def scan_formats(directory: Union[Path, str]) -> list[Path]:
def _print_mine_summary(
files: list,
files_seen: int,
files_with_text: int,
files_skipped: int,
files_errored: int,
@@ -523,7 +523,7 @@ def _print_mine_summary(
print(f"\n{'=' * 55}")
print(" Summary")
print(f"{'-' * 55}")
print(f" Files seen: {len(files)}")
print(f" Files seen: {files_seen}")
print(f" Files extracted: {files_with_text}")
print(f" Files skipped: {files_skipped}")
print(f" Files errored: {files_errored}")
@@ -789,9 +789,11 @@ def mine_formats(
files: list = []
collection = None
total_drawers = 0
files_mined = 0
files_skipped = 0
files_with_text = 0
files_errored = 0
files_processed = 0
status_counts: dict = defaultdict(int)
try:
@@ -799,15 +801,14 @@ def mine_formats(
# ``~/docs`` and relative inputs work consistently. Per PR #1555 review
# (Copilot #10).
files = scan_formats(format_path)
if limit > 0:
files = files[:limit]
print(f"\n{'=' * 55}")
print(" MemPalace Mine — Format extraction")
print(f"{'=' * 55}")
print(f" Wing: {wing}")
print(f" Source: {format_path}")
print(f" Files: {len(files)}")
limit_suffix = f" (limit: {limit} new)" if limit > 0 else ""
print(f" Files: {len(files)}{limit_suffix}")
print(f" Palace: {palace_path}")
if dry_run:
print(" DRY RUN — nothing will be filed")
@@ -816,6 +817,7 @@ def mine_formats(
collection = get_collection(palace_path) if not dry_run else None
for i, filepath in enumerate(files, 1):
files_processed = i
source_file = str(filepath)
# Per-file try/except so one bad file can't crash the whole mine.
@@ -876,6 +878,9 @@ def mine_formats(
if dry_run:
print(f" [DRY RUN] {filepath.name} → {len(chunks)} drawers")
total_drawers += len(chunks)
files_mined += 1
if limit > 0 and files_mined >= limit:
break
continue
drawers_added, skipped = _file_chunks_locked(
@@ -893,7 +898,10 @@ def mine_formats(
continue
total_drawers += drawers_added
files_mined += 1
print(f" + [{i:4}/{len(files)}] {filepath.name[:50]:50} +{drawers_added}")
if limit > 0 and files_mined >= limit:
break
except Exception as exc:
# Log and continue — one malformed file shouldn't kill the
# whole mine. Mirrors miner.py's per-file recovery.
@@ -973,7 +981,7 @@ def mine_formats(
logger.debug("mine_formats: _cleanup_mine_pid_file failed", exc_info=True)
_print_mine_summary(
files=files,
files_seen=files_processed,
files_with_text=files_with_text,
files_skipped=files_skipped,
files_errored=files_errored,
+59 -22
View File
@@ -46,10 +46,11 @@ from .dynamics import initialize_dynamics_fields
logger = logging.getLogger("mempalace_hallways")
# Persistence target. Mirrors ``palace_graph._TUNNEL_FILE`` so the storage
# pattern is uniform across the two related primitives. Tests override
# this via ``monkeypatch.setattr(hallways, "_HALLWAY_FILE", tmp_path/...)``.
_HALLWAY_FILE = os.path.join(os.path.expanduser("~"), ".mempalace", "hallways.json")
# Persistence target is resolved through ``_get_hallway_file`` below, which
# mirrors ``palace_graph._get_tunnel_file`` (the 3.3.6 palace-scoped pattern)
# so the storage layout is uniform across the two related primitives. Tests
# should monkey-patch ``_get_hallway_file`` and ``_legacy_hallway_file`` rather
# than poking a module-level constant.
_SCHEMA_VERSION = 1
@@ -62,36 +63,72 @@ __all__ = [
# ─────────────────────────────────────────────────────────────────────────────
# Persistence — JSON file at _HALLWAY_FILE, restricted perms (0600) on POSIX
# Persistence — JSON file resolved from MempalaceConfig.hallway_file,
# restricted perms (0600) on POSIX. Pre-3.3.6 behavior (hardcoded
# ~/.mempalace/hallways.json) is kept only as a one-time orphan detection
# fallback, matching the palace_graph tunnel-file migration pattern.
# ─────────────────────────────────────────────────────────────────────────────
def _load_hallways() -> list[dict]:
"""Read all hallway records. Returns ``[]`` if the file is missing or corrupt."""
if not os.path.exists(_HALLWAY_FILE):
def _get_hallway_file(config=None) -> str:
"""Return the path to the hallways.json file, derived from MempalaceConfig.palace_path."""
from .config import MempalaceConfig
config = config or MempalaceConfig()
return config.hallway_file
def _legacy_hallway_file() -> str:
"""The pre-palace-scoped hardcoded path. Kept only for one-time orphan detection."""
return os.path.join(os.path.expanduser("~"), ".mempalace", "hallways.json")
def _load_hallways(config=None) -> list[dict]:
"""Read all hallway records. Returns ``[]`` if the file is missing or corrupt.
Backwards-compatibility: prior to this migration the hallway file was
hardcoded at ``~/.mempalace/hallways.json`` regardless of the configured
palace_path. If the configured hallway file is missing but a legacy file
exists at a different path, log a one-line warning naming both paths so
users can move the file manually. We do NOT auto-migrate — auto-merging
hallway state across two locations is too magical for a bugfix and risks
clobbering newer data. Same posture as ``palace_graph._load_tunnels``.
"""
current_hallway_file = _get_hallway_file(config)
if os.path.exists(current_hallway_file):
try:
with open(current_hallway_file, encoding="utf-8") as f:
raw = json.load(f)
except (OSError, json.JSONDecodeError):
logger.debug("hallways: load failed, treating as empty", exc_info=True)
return []
if isinstance(raw, dict) and "hallways" in raw:
return raw.get("hallways") or []
if isinstance(raw, list):
return raw
return []
try:
with open(_HALLWAY_FILE, encoding="utf-8") as f:
raw = json.load(f)
except (OSError, json.JSONDecodeError):
logger.debug("hallways: load failed, treating as empty", exc_info=True)
return []
if isinstance(raw, dict) and "hallways" in raw:
return raw.get("hallways") or []
if isinstance(raw, list):
return raw
legacy = _legacy_hallway_file()
if legacy != current_hallway_file and os.path.exists(legacy):
logger.warning(
"Legacy hallways file at '%s' is being ignored; configured location is '%s'. "
"Move or copy the legacy file to the configured path to recover its hallways.",
legacy,
current_hallway_file,
)
return []
def _save_hallways(hallways: list[dict]) -> None:
"""Atomically persist hallway records to _HALLWAY_FILE.
def _save_hallways(hallways: list[dict], config=None) -> None:
"""Atomically persist hallway records to the configured hallway file.
Uses an os.replace temp-file dance so a crash mid-write doesn't
corrupt the file. POSIX permission is restricted to 0600 because
hallways reveal within-wing entity connections that the user may
not want world-readable.
"""
directory = os.path.dirname(_HALLWAY_FILE)
hallway_file = _get_hallway_file(config)
directory = os.path.dirname(hallway_file)
os.makedirs(directory, exist_ok=True)
payload = {
"schema_version": _SCHEMA_VERSION,
@@ -106,7 +143,7 @@ def _save_hallways(hallways: list[dict]) -> None:
except OSError:
# Non-POSIX systems may not support chmod; not fatal.
pass
os.replace(tmp_path, _HALLWAY_FILE)
os.replace(tmp_path, hallway_file)
except Exception:
try:
os.unlink(tmp_path)
+159
View File
@@ -0,0 +1,159 @@
"""Compatibility helpers for legacy shell hooks.
The shell hooks intentionally stay small and portable, but parsing Claude
hook JSON and counting UTF-8 JSONL transcripts is safer in Python than in
inline shell snippets. This module centralizes that behavior for both
hooks/mempal_save_hook.sh and hooks/mempal_precompact_hook.sh.
"""
from __future__ import annotations
import json
import re
import sys
_SESSION_ID_RE = re.compile(r"[^a-zA-Z0-9_-]")
_CONTROL_CHARS_RE = re.compile(r"[\x00\r\n]")
def sanitize_session_id(session_id: object) -> str:
"""Keep session ids safe for state-file names."""
sanitized = _SESSION_ID_RE.sub("", str(session_id or ""))
return sanitized or "unknown"
def normalize_transcript_path(path: object) -> str:
r"""Normalize a hook transcript path without destroying Windows paths.
Claude Code on Windows sends paths like:
C:\Users\me\.claude\projects\<project>\<session>.jsonl
The old shell sanitizer removed both the drive-letter colon and
backslashes. That turned a valid transcript path into a nonexistent path.
For transcript paths, we only remove control characters that would break
newline-delimited shell parsing, and normalize backslashes to forward
slashes so Git Bash can still address the same Windows file.
"""
normalized = str(path or "").replace("\\", "/")
return _CONTROL_CHARS_RE.sub("", normalized)
def _stop_hook_active(value: object) -> str:
"""Return the exact boolean string expected by the shell hook."""
if value is True:
return "True"
if str(value).strip().lower() in ("true", "1", "yes"):
return "True"
return "False"
def parse_stop_payload(payload: dict) -> tuple[str, str, str]:
return (
sanitize_session_id(payload.get("session_id", "")),
_stop_hook_active(payload.get("stop_hook_active", False)),
normalize_transcript_path(payload.get("transcript_path", "")),
)
def parse_precompact_payload(payload: dict) -> tuple[str, str]:
return (
sanitize_session_id(payload.get("session_id", "")),
normalize_transcript_path(payload.get("transcript_path", "")),
)
def count_human_messages(path: str) -> int:
"""Count user messages in a Claude transcript JSONL file.
Claude transcripts are UTF-8. Windows Python defaults to cp1252 in many
environments, so the encoding must be explicit. Invalid bytes are ignored
to match the hooks' fail-soft behavior.
"""
count = 0
with open(path, encoding="utf-8", errors="ignore") as fh:
for line in fh:
try:
entry = json.loads(line)
except Exception:
continue
msg = entry.get("message", {})
if not isinstance(msg, dict) or msg.get("role") != "user":
continue
content = msg.get("content", "")
if isinstance(content, str) and "<command-message>" in content:
continue
count += 1
return count
def _load_stdin_json() -> dict:
raw = sys.stdin.read()
# Empty stdin is a legitimate hook state. Treat it as an empty payload so
# the sentinel is printed and the shell fail-loud guard does not spam disk.
if raw == "":
return {}
# For non-empty malformed input, intentionally let json.loads raise.
# The shell hooks capture this stderr in last_python_err.log and, because
# no sentinel is printed, write a bounded copy of the raw payload to
# last_input.log. That fail-loud contract is pinned by
# tests/test_hooks_bash_compat.py.
data = json.loads(raw)
if not isinstance(data, dict):
raise TypeError(f"hook input must be a JSON object, got {type(data).__name__}")
return data
def main(argv: list[str] | None = None) -> int:
argv = list(sys.argv[1:] if argv is None else argv)
if not argv:
print(
"usage: python -m mempalace.hook_shell <parse-stop|parse-precompact|count-human-messages>",
file=sys.stderr,
)
return 2
command = argv[0]
if command == "parse-stop":
session_id, stop_hook_active, transcript_path = parse_stop_payload(_load_stdin_json())
print("__MEMPAL_PARSE_OK__")
print(session_id)
print(stop_hook_active)
print(transcript_path)
return 0
if command == "parse-precompact":
session_id, transcript_path = parse_precompact_payload(_load_stdin_json())
print("__MEMPAL_PARSE_OK__")
print(session_id)
print(transcript_path)
return 0
if command == "count-human-messages":
if len(argv) != 2:
print("count-human-messages requires a transcript path", file=sys.stderr)
return 2
try:
print(count_human_messages(argv[1]))
except Exception:
print(0)
return 0
print(f"unknown hook_shell command: {command}", file=sys.stderr)
return 2
if __name__ == "__main__":
raise SystemExit(main())
+279 -4
View File
@@ -1,8 +1,8 @@
"""
Hook logic for MemPalace — Python implementation of session-start, stop, and precompact hooks.
Hook logic for MemPalace — Python implementation of session-start, stop, session-end, and precompact hooks.
Reads JSON from stdin, outputs JSON to stdout.
Supported hooks: session-start, stop, precompact
Supported hooks: session-start, stop, session-end, precompact
Supported harnesses: claude-code, codex (extensible to cursor, gemini, etc.)
"""
@@ -25,7 +25,7 @@ PALACE_ROOT = Path.home() / ".mempalace"
def _detached_popen_kwargs() -> dict:
"""Kwargs that fully detach a Popen child so the hook process can exit.
"""Kwargs that give a Popen child a hidden console so the hook can exit.
Without these, Windows holds the parent open until the child closes the
inherited stdout/stderr handles — manifesting as "Stop hook hangs" at
@@ -36,7 +36,7 @@ def _detached_popen_kwargs() -> dict:
kwargs: dict = {"stdin": subprocess.DEVNULL, "close_fds": True}
if os.name == "nt":
flags = 0
for name in ("DETACHED_PROCESS", "CREATE_NEW_PROCESS_GROUP", "CREATE_BREAKAWAY_FROM_JOB"):
for name in ("CREATE_NO_WINDOW", "CREATE_NEW_PROCESS_GROUP", "CREATE_BREAKAWAY_FROM_JOB"):
flags |= getattr(subprocess, name, 0)
if flags:
kwargs["creationflags"] = flags
@@ -509,6 +509,73 @@ def _spawn_mine(cmd: list) -> None:
pass
def _hooks_daemon_enabled() -> bool:
try:
return MempalaceConfig().hook_use_daemon is True
except Exception:
return False
def _daemon_mine_dedupe_key(source: str, mode: str) -> str:
try:
source_key = str(Path(source).expanduser().resolve())
except OSError:
source_key = str(Path(source).expanduser())
return f"hook:mine:{mode}:{source_key}"
def _daemon_available() -> bool:
"""True iff a daemon is already running for the configured palace.
This is a fast localhost health check, not a spawn: the 500ms hook budget
forbids auto-starting a python subprocess from a hook (cold start is
~15s). Daemon mode for hooks requires the user to have started the daemon
explicitly via `mempalace daemon start`; when it isn't up, hooks fall back
to the existing direct (in-process / spawn) path instead of blocking.
"""
from .daemon import HOOK_PROBE_TIMEOUT, get_client_if_running
try:
return (
get_client_if_running(MempalaceConfig().palace_path, health_timeout=HOOK_PROBE_TIMEOUT)
is not None
)
except Exception:
return False
def _submit_daemon_job(
kind: str,
payload: dict,
*,
dedupe_key: str = None,
priority: int = 0,
wait: bool = False,
timeout: float = 60.0,
):
"""Submit to an already-running daemon. Never auto-starts (see _daemon_available).
Raises DaemonError on a real failure (job rejected, timeout, daemon died
mid-submit). Callers must NOT fall back to the direct path on such errors —
the daemon may already have accepted the job, and re-running it would
duplicate verbatim content. Only an absent daemon (handled by the caller's
_daemon_available() precheck) should fall back.
"""
from .daemon import submit_job
palace_path = MempalaceConfig().palace_path
return submit_job(
kind,
payload,
palace_path=palace_path,
dedupe_key=dedupe_key,
priority=priority,
wait=wait,
auto_start=False,
timeout=timeout,
)
def _maybe_auto_ingest():
"""Background-mine MEMPAL_DIR (project files) if set.
@@ -527,9 +594,26 @@ def _maybe_auto_ingest():
return
for mine_dir, mode in targets:
try:
if _hooks_daemon_enabled() and _daemon_available():
try:
_submit_daemon_job(
"mine",
{"source": mine_dir, "mode": mode, "agent": "mempalace"},
dedupe_key=_daemon_mine_dedupe_key(mine_dir, mode),
wait=False,
)
except Exception as exc:
# Daemon accepted context — don't fall back (would double-mine).
_log(f"Daemon mine submission failed: {exc}")
continue
_spawn_mine([_mempalace_python(), "-m", "mempalace", "mine", mine_dir, "--mode", mode])
except OSError:
pass
except Exception as exc:
# Non-daemon spawn path failed. Hooks must never crash the user's
# shell — log and continue. Do not label this a daemon failure: the
# daemon block above handles its own errors with its own message.
_log(f"mine hook failed: {exc}")
def _mine_sync():
@@ -546,6 +630,22 @@ def _mine_sync():
log_path = STATE_DIR / "hook.log"
for mine_dir, mode in targets:
try:
if _hooks_daemon_enabled() and _daemon_available():
try:
job = _submit_daemon_job(
"mine",
{"source": mine_dir, "mode": mode, "agent": "mempalace"},
dedupe_key=_daemon_mine_dedupe_key(mine_dir, mode),
wait=True,
timeout=60,
)
result = job.get("result") or {}
if job.get("state") != "succeeded" or not result.get("success", True):
_log(f"Daemon sync mine failed: {result.get('error', job.get('error'))}")
except Exception as exc:
# Daemon accepted context — don't fall back (would double-mine).
_log(f"Daemon sync mine submission failed: {exc}")
continue
with open(log_path, "a") as log_f:
subprocess.run(
[
@@ -563,6 +663,11 @@ def _mine_sync():
)
except (OSError, subprocess.TimeoutExpired):
pass
except Exception as exc:
# Non-daemon sync spawn path failed. Hooks must never crash the
# user's shell — log and continue (not a daemon failure; the daemon
# block above handles its own errors).
_log(f"mine hook failed: {exc}")
def _desktop_toast(body: str, title: str = "MemPalace"):
@@ -680,6 +785,41 @@ def _save_diary_direct(
)
try:
if _hooks_daemon_enabled() and _daemon_available():
try:
job = _submit_daemon_job(
"diary_write",
{
"agent_name": agent_name,
"entry": entry,
"topic": "checkpoint",
"wing": wing,
},
priority=10,
wait=True,
timeout=30,
)
except Exception as exc:
# Daemon accepted context — don't fall back (would double-write).
_log(f"Daemon diary checkpoint failed: {exc}")
return {"count": 0}
result = job.get("result") or {}
if job.get("state") == "succeeded" and result.get("success"):
_log(f"Diary checkpoint saved: {result.get('entry_id', '?')}")
try:
ack_file = STATE_DIR / "last_checkpoint"
ack_file.write_text(
json.dumps({"msgs": len(messages), "ts": now.isoformat()}),
encoding="utf-8",
)
except OSError:
pass
if toast:
_desktop_toast(f"Checkpoint saved - {len(messages)} messages archived")
return {"count": len(messages), "themes": themes}
_log(f"Daemon diary checkpoint failed: {result.get('error', job.get('error'))}")
return {"count": 0}
from .mcp_server import tool_diary_write
result = tool_diary_write(
@@ -721,6 +861,25 @@ def _ingest_transcript(transcript_path: str):
return
try:
if _hooks_daemon_enabled() and _daemon_available():
try:
_submit_daemon_job(
"mine",
{
"source": str(path.parent),
"mode": "convos",
"wing": "sessions",
"agent": "mempalace",
},
dedupe_key=_daemon_mine_dedupe_key(str(path.parent), "convos"),
wait=False,
)
_log(f"Transcript ingest submitted to daemon: {path.name}")
except Exception as exc:
# Daemon accepted context — don't fall back (would double-mine).
_log(f"Daemon transcript ingest failed: {exc}")
return
# Route through ``_spawn_mine`` so the per-target PID guard kicks
# in here too — repeated Stop/PreCompact fires for the same
# transcript should not stack up parallel ingest mines.
@@ -740,6 +899,11 @@ def _ingest_transcript(transcript_path: str):
_log(f"Transcript ingest started: {path.name}")
except OSError:
pass
except Exception as exc:
# Non-daemon ingest spawn path failed. Hooks must never crash the
# user's shell — log and continue (not a daemon failure; the daemon
# block above handles its own errors).
_log(f"transcript ingest hook failed: {exc}")
SUPPORTED_HARNESSES = {"claude-code", "codex"}
@@ -1021,6 +1185,116 @@ def hook_session_start(data: dict, harness: str):
_output({})
def _clear_session_last_save(session_id: str) -> None:
"""Drop the per-session save marker once a session has ended.
``hook_stop`` writes ``{session_id}_last_save`` but never had a clean-exit
cleanup path, so the marker lingered. The session is over by the time
``hook_session_end`` runs, so removing it here keeps ``hook_state/`` from
accumulating dead markers. OS errors (including a missing marker, since
``FileNotFoundError`` is an ``OSError``) are swallowed — this is best-effort
cleanup, never a reason to fail the hook.
"""
try:
(STATE_DIR / f"{session_id}_last_save").unlink()
except OSError:
pass
def hook_session_end(data: dict, harness: str):
"""Session end hook: one final flush when a session exits cleanly.
Closes the gap (#1341) where a session that never crosses ``SAVE_INTERVAL``
on ``Stop`` and never triggers ``PreCompact`` exits with nothing saved —
the common case for short, useful sessions.
Why background instead of mine inline: Claude Code's hooks reference
documents a default SessionEnd timeout of 1.5 seconds, and "timeouts set on
plugin-provided hooks do not raise the budget"
(https://code.claude.com/docs/en/hooks). A cold ``mempalace`` start alone
exceeds 1.5s, so this handler must never mine in the hook foreground. The
shell wrapper backgrounds it and returns immediately; the heavy capture is
spawned *detached* via ``_ingest_transcript`` / ``_maybe_auto_ingest`` (both
route through ``_spawn_mine`` / ``_detached_popen_kwargs``). On POSIX that
detached child reliably outlives the session (verified). On Windows only the
mine grandchild (spawned with detached-process flags) is designed to break
away from the session; the backgrounded hook process and the in-process
diary write are best-effort there (no Windows CI coverage yet). This
honors the "background everything / hooks under 500ms" budget. SessionEnd
has no decision control, so this only ever saves; it never emits a block
payload.
"""
if not _palace_root_exists():
_output({})
return
# Parse inside the try so a malformed payload (e.g. non-dict stdin that
# makes _parse_harness_input raise) still runs the finally cleanup below.
session_id = "unknown"
try:
parsed = _parse_harness_input(data, harness)
session_id = parsed["session_id"]
transcript_path = parsed["transcript_path"]
# Read config defensively (mirror hook_stop): a corrupt or unreadable
# config must not lose the final save, so default to auto-save on and
# toasts off rather than crashing the hook.
try:
config = MempalaceConfig()
auto_save = config.hooks_auto_save
toast = config.hook_desktop_toast
except Exception:
auto_save = True
toast = False
# Respect auto_save config toggle (clean opt-out)
if not auto_save:
_output({})
return
_log(f"SESSION END for session {session_id}")
# Validate the harness-provided transcript path before touching it
# (extension + ".." traversal check), mirroring the read path that
# already runs through _validate_transcript_path. A rejected path skips
# the transcript captures but still lets the independent MEMPAL_DIR mine
# run.
valid_transcript = ""
if transcript_path:
try:
validated = _validate_transcript_path(transcript_path)
except OSError:
validated = None
if validated is None:
_log(f"WARNING: transcript_path rejected by validator: {transcript_path!r}")
else:
valid_transcript = str(validated)
# Flush. The diary checkpoint (in-process ChromaDB write) runs FIRST,
# before any detached mine is spawned, so it never contends for the
# palace lock; this handler is already backgrounded by the wrapper, so it
# is not under the SessionEnd budget and has time to finish. The detached
# transcript ingest follows; re-mining a transcript ``Stop`` already
# captured is a near no-op (deterministic convo IDs + ``file_already_mined``
# short-circuit + upsert). ``reason`` is intentionally not branched on:
# every clean-exit reason (incl. ``/clear`` / ``resume``) warrants the
# flush. Order matches ``hook_stop``.
if valid_transcript:
_save_diary_direct(
valid_transcript,
session_id,
wing=_wing_from_transcript_path(valid_transcript),
toast=toast,
agent_name=_diary_agent_for_harness(harness),
)
_ingest_transcript(valid_transcript)
_maybe_auto_ingest()
_output({})
finally:
_clear_session_last_save(session_id)
def hook_precompact(data: dict, harness: str):
"""Precompact hook: mine transcript synchronously, then allow compaction.
@@ -1064,6 +1338,7 @@ def run_hook(hook_name: str, harness: str):
hooks = {
"session-start": hook_session_start,
"stop": hook_stop,
"session-end": hook_session_end,
"precompact": hook_precompact,
}
+2 -2
View File
@@ -22,7 +22,7 @@ import hashlib
# as legacy ``v1`` (pre-delimiter recipe), drawers with ``id_recipe="v2"``
# are guaranteed collision-safe within the v2 generation. The constant is
# exported so call sites use ``ids.ID_RECIPE`` rather than a magic string.
ID_RECIPE: str = "v2"
ID_RECIPE: str = "v3"
# '|' is reserved in Windows filenames and cannot appear in source paths
# on any supported platform, making it strictly safer than ':' (which
@@ -49,7 +49,7 @@ def _delimited_sha256(parts: tuple[object, ...], truncate: int) -> str:
e.g. ``valid_from=None`` joins as the literal string ``"None"`` rather
than crashing.
"""
key = _DELIM.join(str(p) for p in parts).encode()
key = "".join(f"{len(part)}:{part}" for part in map(str, parts)).encode()
return hashlib.sha256(key).hexdigest()[:truncate]
+1
View File
@@ -29,6 +29,7 @@ AI memory system. Store everything, find anything. Local, free, no API key.
### Palace (write)
- mempalace_add_drawer -- Add a new memory (drawer)
- mempalace_checkpoint -- Save a whole session in one call (dedup + file + diary)
- mempalace_delete_drawer -- Delete a memory (drawer)
### Knowledge Graph
+10 -3
View File
@@ -23,7 +23,12 @@ from collections import defaultdict
from .config import MempalaceConfig
from .palace import get_collection as _get_collection
from .searcher import _first_or_empty, build_where_filter
from .searcher import (
_distance_to_similarity,
_first_or_empty,
_metric_for_collection,
build_where_filter,
)
# ---------------------------------------------------------------------------
@@ -283,11 +288,12 @@ class Layer3:
if not docs:
return "No results found."
metric = _metric_for_collection(col)
lines = [f'## L3 — SEARCH RESULTS for "{query}"']
for i, (doc, meta, dist) in enumerate(zip(docs, metas, dists), 1):
meta = meta or {}
doc = doc or ""
similarity = round(max(0.0, 1 - dist), 3)
similarity = round(_distance_to_similarity(dist, metric), 3)
wing_name = meta.get("wing", "?")
room_name = meta.get("room", "?")
source = Path(meta.get("source_file", "")).name if meta.get("source_file") else ""
@@ -327,6 +333,7 @@ class Layer3:
except Exception:
return []
metric = _metric_for_collection(col)
hits = []
for doc, meta, dist in zip(
_first_or_empty(results, "documents"),
@@ -346,7 +353,7 @@ class Layer3:
"wing": meta.get("wing", "unknown"),
"room": meta.get("room", "unknown"),
"source_file": Path(meta.get("source_file", "?")).name,
"similarity": round(1 - dist, 3),
"similarity": round(_distance_to_similarity(dist, metric), 3),
"metadata": meta,
}
)
+1884 -187
View File
File diff suppressed because it is too large Load Diff
+14
View File
@@ -19,6 +19,7 @@ Usage:
"""
import errno
import glob
import os
import shutil
import sqlite3
@@ -28,6 +29,9 @@ from collections import defaultdict
from contextlib import closing
from datetime import datetime
from .backups import prune_backups
from .config import MempalaceConfig
def _restore_stale_palace(palace_path: str, stale_path: str) -> None:
"""Roll back a failed swap.
@@ -293,6 +297,16 @@ def migrate(palace_path: str, dry_run: bool = False, confirm: bool = False):
print(f"\n Backing up to {backup_path}...")
shutil.copytree(palace_path, backup_path)
# Enforce backup retention so repeated migrations cannot fill the disk
# with full-palace copies. The backup we just created is the newest, so
# it survives; only older ``.pre-migrate.*`` siblings beyond the limit
# are removed. Best-effort — never let cleanup fail the migration.
prune_backups(
glob.escape(palace_path) + ".pre-migrate.*",
MempalaceConfig().max_backups,
log=print,
)
# Build fresh palace in a temp directory (avoids chromadb reading old state).
# Wrap the whole import-and-swap dance in try/finally so the temp dir is
# cleaned up if any of the chromadb writes, the verify count, or the
+45 -6
View File
@@ -47,6 +47,34 @@ from .ids import ID_RECIPE, make_drawer_id_from_chunk
logger = logging.getLogger("mempalace_mcp")
PHP_EXTENSIONS = {
# Compound Blade templates such as ``view.blade.php`` are covered by the
# final ``.php`` suffix.
".php",
".php3",
".php4",
".php5",
".php7",
".php8",
".phtml",
".phps",
".phpt",
".inc",
".aw",
".fcgi",
".ctp",
".module",
".install",
".profile",
".theme",
".engine",
".twig",
".blade",
".tpl",
".latte",
".volt",
}
READABLE_EXTENSIONS = {
".txt",
".md",
@@ -64,12 +92,21 @@ READABLE_EXTENSIONS = {
".java",
".go",
".rs",
".swift",
".kt",
".kts",
".rb",
".sh",
".csv",
".sql",
".toml",
}
# C# / .NET
".cs",
".csproj",
".sln",
".razor",
".cshtml",
} | PHP_EXTENSIONS
SKIP_FILENAMES = {
"entities.json",
@@ -1623,9 +1660,6 @@ def _mine_impl(
respect_gitignore=respect_gitignore,
include_ignored=include_ignored,
)
if limit > 0:
files = files[:limit]
from .embedding import describe_device
print(f"\n{'=' * 55}")
@@ -1633,7 +1667,8 @@ def _mine_impl(
print(f"{'=' * 55}")
print(f" Wing: {wing}")
print(f" Rooms: {', '.join(r['name'] for r in rooms)}")
print(f" Files: {len(files)}")
limit_suffix = f" (limit: {limit} new)" if limit > 0 else ""
print(f" Files: {len(files)}{limit_suffix}")
print(f" Palace: {palace_path}")
print(f" Device: {describe_device()}")
if dry_run:
@@ -1652,6 +1687,7 @@ def _mine_impl(
closets_col = None
total_drawers = 0
files_mined = 0
files_skipped = 0
files_skipped_chunk_cap = 0
files_processed = 0
@@ -1699,8 +1735,11 @@ def _mine_impl(
else:
total_drawers += drawers
room_counts[room] += 1
files_mined += 1
if not dry_run:
print(f" + [{i:4}/{len(files)}] {filepath.name[:50]:50} +{drawers}")
if limit > 0 and files_mined >= limit:
break
if not dry_run:
# Cross-wing topic tunnels: after every file in this wing has been
@@ -1753,7 +1792,7 @@ def _mine_impl(
print(f"\n{'=' * 55}")
print(" Done.")
print(f" Files processed: {len(files) - files_skipped}")
print(f" Files processed: {files_processed - files_skipped}")
# The residual skip bucket label depends on mode: dry-run bypasses
# the already-mined check, so the only paths producing (0, room,
# None) under dry_run are OSError / too-short / post-lock re-check
+198 -1
View File
@@ -9,6 +9,9 @@ Supported:
- Claude Code JSONL (with tool_use/tool_result block capture)
- OpenAI Codex CLI JSONL
- Gemini CLI JSONL (~/.gemini/tmp/<project_hash>/chats/session-*.jsonl)
- Pi agent JSONL
- Gemini CLI / Google AI Studio JSON sessions (contents / messages / flat list)
- Continue.dev session JSON (~/.continue/sessions/*.json)
- Slack JSON export
- Plain text (pass through for paragraph chunking)
@@ -162,12 +165,22 @@ def _try_normalize_json(content: str) -> Optional[str]:
if normalized:
return normalized
normalized = _try_pi_jsonl(content)
if normalized:
return normalized
try:
data = json.loads(content)
except json.JSONDecodeError:
return None
for parser in (_try_claude_ai_json, _try_chatgpt_json, _try_slack_json):
for parser in (
_try_gemini_json,
_try_claude_ai_json,
_try_chatgpt_json,
_try_continue_json,
_try_slack_json,
):
normalized = parser(data)
if normalized:
return normalized
@@ -353,6 +366,135 @@ def _try_gemini_jsonl(content: str) -> Optional[str]:
return None
def _try_pi_jsonl(content: str) -> Optional[str]:
"""Pi agent sessions (~/.config/pi/agent/sessions/{cwd}/{timestamp}_{uuid}.jsonl).
Pi stores sessions as JSONL with a tree-structured message history.
User messages have role "user" with content as string or [{type, text}] blocks.
Assistant messages have role "assistant" with content as [{type, text}] blocks
(may also include "thinking" blocks which are skipped by _extract_content).
Tool results (role "toolResult") are skipped — operational, not conversation.
Format documented at github.com/badlogic/pi-mono session.md.
"""
lines = [line.strip() for line in content.strip().split("\n") if line.strip()]
messages = []
has_session_header = False
for line in lines:
try:
entry = json.loads(line)
except json.JSONDecodeError:
continue
if not isinstance(entry, dict):
continue
entry_type = entry.get("type", "")
if entry_type == "session" and "version" in entry:
has_session_header = True
continue
if entry_type != "message":
continue
message = entry.get("message", {})
if not isinstance(message, dict):
continue
role = message.get("role", "")
text = _extract_content(message.get("content", ""))
if role == "user" and text:
messages.append(("user", text))
elif role == "assistant" and text:
messages.append(("assistant", text))
if len(messages) >= 2 and has_session_header:
return _messages_to_transcript(messages)
return None
def _try_gemini_json(data) -> Optional[str]:
"""Gemini CLI / Google AI Studio JSON sessions.
Handles three layouts:
1. **Gemini API contents format** — used by Gemini CLI session files
(``~/.gemini/sessions/*.json``):
``{"contents": [{"role": "user", "parts": [{"text": "..."}]}, ...]}``
2. **Messages wrapper** — exports that wrap the conversation under a
``messages`` key:
``{"messages": [{"role": "user", "content": "..."}, {"role": "model", "content": "..."}]}``
3. **Flat messages list** — top-level array form:
``[{"role": "user", "content": "..."}, {"role": "model", "content": "..."}]``
Gemini uses ``"model"`` as the assistant role (not ``"assistant"``).
Detection requires at least one ``role="model"`` entry to disambiguate
from Claude/ChatGPT exports that use ``"assistant"``. This parser is
placed *before* ``_try_claude_ai_json`` in the dispatch chain so that
the layout-2 ``{"messages": [...]}`` wrapper does not get silently
claimed by the Claude parser, which would drop the model turns.
"""
contents = None
# Layout 1: {"contents": [...]}
if isinstance(data, dict) and "contents" in data:
contents = data["contents"]
# Layout 2a: {"messages": [...]}
elif isinstance(data, dict) and "messages" in data:
contents = data["messages"]
# Layout 2b: top-level list
elif isinstance(data, list):
contents = data
if not isinstance(contents, list) or len(contents) < 2:
return None
messages = []
has_model_role = False
for item in contents:
if not isinstance(item, dict):
continue
role = item.get("role", "")
# Extract text — try "parts" first (Gemini API), then "content" (flat).
text = ""
parts = item.get("parts")
if isinstance(parts, list):
text_parts = []
for p in parts:
if isinstance(p, str):
text_parts.append(p)
elif isinstance(p, dict) and "text" in p:
text_parts.append(p["text"])
text = " ".join(text_parts).strip()
else:
text = _extract_content(item.get("content", ""))
if not text:
continue
if role == "user":
messages.append(("user", text))
elif role == "model":
messages.append(("assistant", text))
has_model_role = True
elif role == "assistant":
# Defensive: some hand-crafted exports use "assistant" even
# for Gemini sessions. Accept but don't flip has_model_role.
messages.append(("assistant", text))
# Disambiguator: must have seen at least one role="model" entry.
# This prevents the Gemini parser from claiming Claude/ChatGPT data.
if not has_model_role:
return None
if len(messages) >= 2:
return _messages_to_transcript(messages)
return None
def _try_claude_ai_json(data) -> Optional[str]:
"""Claude.ai JSON export: flat messages list or privacy export with chat_messages."""
if isinstance(data, dict):
@@ -485,6 +627,61 @@ def _try_slack_json(data) -> Optional[str]:
return None
def _try_continue_json(data) -> Optional[str]:
"""Continue.dev session JSON (~/.continue/sessions/*.json).
Sessions contain a ``history`` array of ``{role, content}`` message objects,
plus optional metadata (``title``, ``sessionId``, ``dateCreated``).
System messages are skipped. Tool-call messages (role ``tool``) are
formatted inline when they contain text content.
"""
if not isinstance(data, dict) or "history" not in data:
return None
history = data["history"]
if not isinstance(history, list):
return None
messages = []
for item in history:
if not isinstance(item, dict):
continue
role = item.get("role", "")
content = item.get("content", "")
# Extract text from string or list-of-blocks content
if isinstance(content, list):
parts = []
for block in content:
if isinstance(block, dict):
if block.get("type") == "text":
parts.append(block.get("text", ""))
elif isinstance(block, str):
parts.append(block)
text = "\n".join(p for p in parts if p).strip()
elif isinstance(content, str):
text = content.strip()
else:
continue
if not text:
continue
if role == "user":
messages.append(("user", text))
elif role == "assistant":
messages.append(("assistant", text))
elif role == "tool":
# Append tool output to the previous assistant turn if possible
if messages and messages[-1][0] == "assistant":
prev_role, prev_text = messages[-1]
messages[-1] = (prev_role, prev_text + "\n" + f"[tool] {text}")
# Skip system and other roles
if len(messages) >= 2:
return _messages_to_transcript(messages)
return None
def _extract_content(content, tool_use_map: dict = None) -> str:
"""Pull text from content — handles str, list of blocks, or dict.
+329 -25
View File
@@ -70,13 +70,109 @@ _EXPLICIT_BACKEND_ENV = "MEMPALACE_BACKEND_EXPLICIT"
NORMALIZE_VERSION = 2
# (palace_id, collection_name, model_name) tuples already validated this
# process, so the identity check (one metadata read) runs at most once per
# collection per run — keeps the hot get_collection path cheap.
_VALIDATED_IDENTITY: set = set()
def _enforce_embedder_identity(collection, palace_path, collection_name, *, create) -> None:
"""Check (and, for a brand-new collection, record) embedder identity (RFC 001).
Check at open so a model swap fails fast — before any query silently
returns degraded results. Record only when the collection is brand-new and
empty: recording the *current* model on a legacy palace that already holds
vectors from an unknown model would mislabel it, so populated-but-unrecorded
collections warn instead and are resolved with
``mempalace palace set-embedder``.
Bookkeeping must never break memory operations: only the deliberate
identity/dimension mismatch propagates; every other error is swallowed.
"""
import warnings
from .backends.base import (
DimensionMismatchError,
EmbedderIdentity,
EmbedderIdentityMismatchError,
EmbedderIdentityUnknownWarning,
check_embedder_identity,
)
from .embedding import current_model_name
# A server_embedder backend embeds with its own model and ignores the
# injected/core embedder, so its effective identity — not the configured
# model — is what must be checked and recorded. Fall back to the configured
# model name for the normal (core-embedder) case.
current: Optional[EmbedderIdentity] = None
try:
effective = collection.effective_embedder_identity()
except Exception:
effective = None
if effective is not None and getattr(effective, "model_name", ""):
current = effective
else:
try:
model_name = current_model_name()
except Exception:
return
if not model_name:
return # nameless embedder — cannot enforce identity
current = EmbedderIdentity(model_name=model_name, dimension=0)
model_name = current.model_name
key = (str(palace_path), str(collection_name), model_name)
if key in _VALIDATED_IDENTITY:
return
try:
stored = collection.get_stored_embedder_identity()
except Exception:
logger.debug("embedder-identity read failed for %s", collection_name, exc_info=True)
return
try:
state = check_embedder_identity(stored, current)
except (EmbedderIdentityMismatchError, DimensionMismatchError):
raise # deliberate, user-facing — the whole point of the contract
except Exception:
return
if state == "unknown" and stored is None:
try:
count = collection.count()
except Exception:
count = None
if count == 0:
if create:
try:
collection.set_embedder_identity(current)
except Exception:
logger.debug("embedder-identity record failed", exc_info=True)
elif count:
warnings.warn(
f"palace collection {collection_name!r} has no recorded embedder "
f"identity; assuming the current model {model_name!r}. Run "
"`mempalace palace set-embedder --model <name>` to record it.",
EmbedderIdentityUnknownWarning,
stacklevel=2,
)
_VALIDATED_IDENTITY.add(key)
def get_collection(
palace_path: str,
collection_name: Optional[str] = None,
create: bool = True,
backend: Optional[str] = None,
_skip_identity_check: bool = False,
):
"""Get the palace collection through the backend layer."""
"""Get the palace collection through the backend layer.
``_skip_identity_check`` bypasses the embedder-identity enforcement so the
``set-embedder`` override path can open a palace whose recorded model
differs from the current one (the very state it exists to repair).
"""
if collection_name is None:
from .config import get_configured_collection_name
@@ -98,10 +194,71 @@ def get_collection(
create=create,
)
if "requires_explicit_embeddings" in getattr(backend_obj, "capabilities", frozenset()):
return EmbeddingCollection(collection)
collection = EmbeddingCollection(collection)
if not _skip_identity_check:
_enforce_embedder_identity(collection, palace_path, collection_name, create=create)
return collection
def set_palace_embedder_identity(
palace_path: str,
model: Optional[str] = None,
*,
force: bool = False,
backend: Optional[str] = None,
collection_name: Optional[str] = None,
):
"""Record (or force-override) a palace collection's embedder identity (RFC 001).
Backs ``mempalace palace set-embedder``. Returns ``(old, new)`` identities.
Without ``force``, refuses to overwrite an existing identity that names a
different model (the user must confirm they know the vectors are
compatible). Opens with the identity check skipped so a mismatched palace —
the exact state being repaired — can be opened at all.
"""
from .backends.base import EmbedderIdentity, EmbedderIdentityMismatchError
from .config import MempalaceConfig
from .embedding import get_embedder_identity
configured = MempalaceConfig().embedding_model
target = (model or configured or "").strip().lower()
if not target:
# No model given and none configured — there is nothing to record, and
# recording a nameless identity is a silent no-op in every backend.
raise ValueError(
"no embedder model to record: pass --model NAME or configure MEMPALACE_EMBEDDING_MODEL"
)
if target == (configured or "").strip().lower():
# Recording the in-use model — probe its dimension (already loaded).
new = get_embedder_identity()
else:
# Explicit override of a non-configured model: record the name only,
# never load a foreign model (which can be a large download) just to
# probe a dimension. The model-name check is the actual protection.
new = EmbedderIdentity(model_name=target, dimension=0)
collection = get_collection(
palace_path,
collection_name=collection_name,
create=True,
backend=backend,
_skip_identity_check=True,
)
try:
old = collection.get_stored_embedder_identity()
except Exception:
old = None
if old is not None and old.model_name != new.model_name and not force:
raise EmbedderIdentityMismatchError(
f"palace already records embedder {old.model_name!r}; pass --force to "
f"overwrite it with {new.model_name!r} (only if the vectors are compatible)"
)
collection.set_embedder_identity(new)
# Reset the per-process validation cache so a re-open re-checks against the
# newly recorded identity rather than a stale verdict.
_VALIDATED_IDENTITY.clear()
return old, new
def get_closets_collection(
palace_path: str,
create: bool = True,
@@ -568,38 +725,185 @@ def mine_lock(source_file: str):
Prevents multiple agents from mining the same file simultaneously,
which causes duplicate drawers when the delete+insert cycle interleaves.
"""
lock_dir = os.path.join(os.path.expanduser("~"), ".mempalace", "locks")
os.makedirs(lock_dir, exist_ok=True)
lock_path = os.path.join(
lock_dir, hashlib.sha256(source_file.encode()).hexdigest()[:16] + ".lock"
)
lf = open(lock_path, "w")
lock_path = _mine_lock_path(source_file)
lf = _acquire_mine_lock_file(lock_path)
try:
if os.name == "nt":
import msvcrt
msvcrt.locking(lf.fileno(), msvcrt.LK_LOCK, 1)
else:
import fcntl
fcntl.flock(lf, fcntl.LOCK_EX)
yield
finally:
try:
if os.name == "nt":
import msvcrt
msvcrt.locking(lf.fileno(), msvcrt.LK_UNLCK, 1)
else:
import fcntl
fcntl.flock(lf, fcntl.LOCK_UN)
_unlock_mine_lock_file(lf)
except Exception:
logger.debug("Mine-lock release failed", exc_info=True)
try:
lf.close()
except Exception:
logger.debug("Mine-lock close failed", exc_info=True)
_cleanup_mine_lock_file(lock_path)
def _mine_lock_path(source_file: str) -> str:
lock_dir = os.path.join(os.path.expanduser("~"), ".mempalace", "locks")
os.makedirs(lock_dir, exist_ok=True)
return os.path.join(lock_dir, hashlib.sha256(source_file.encode()).hexdigest()[:16] + ".lock")
def _open_mine_lock_file(lock_path: str, *, create: bool):
flags = os.O_RDWR
if create:
flags |= os.O_CREAT
fd = os.open(lock_path, flags, 0o600)
return os.fdopen(fd, "r+b")
def _lock_mine_lock_file(lock_file, *, blocking: bool) -> bool:
lock_file.seek(0)
if os.name == "nt":
import msvcrt
mode = msvcrt.LK_LOCK if blocking else msvcrt.LK_NBLCK
try:
msvcrt.locking(lock_file.fileno(), mode, 1)
except OSError:
if not blocking:
return False
raise
return True
import fcntl
flags = fcntl.LOCK_EX
if not blocking:
flags |= fcntl.LOCK_NB
try:
fcntl.flock(lock_file, flags)
except BlockingIOError:
if not blocking:
return False
raise
return True
def _unlock_mine_lock_file(lock_file) -> None:
lock_file.seek(0)
if os.name == "nt":
import msvcrt
msvcrt.locking(lock_file.fileno(), msvcrt.LK_UNLCK, 1)
return
import fcntl
fcntl.flock(lock_file, fcntl.LOCK_UN)
def _mine_lock_file_is_current(lock_file, lock_path: str) -> bool:
"""Return whether ``lock_file`` is still the inode reached by ``lock_path``.
POSIX advisory locks attach to the opened inode, not the pathname. If a
lock file is unlinked while a contender is waiting, that contender can later
acquire a lock on an inode no new process will use. We reject that stale
handle and retry on the current pathname.
"""
if os.name == "nt":
return True
try:
path_stat = os.stat(lock_path)
file_stat = os.fstat(lock_file.fileno())
except OSError:
return False
return (path_stat.st_dev, path_stat.st_ino) == (file_stat.st_dev, file_stat.st_ino)
def _acquire_open_mine_lock_file(lock_file, lock_path: str) -> bool:
"""Acquire ``lock_file`` and return False if cleanup made it stale."""
_lock_mine_lock_file(lock_file, blocking=True)
if _mine_lock_file_is_current(lock_file, lock_path):
return True
try:
_unlock_mine_lock_file(lock_file)
except Exception:
logger.debug("Mine-lock stale-handle release failed", exc_info=True)
return False
def _acquire_mine_lock_file(lock_path: str):
while True:
lf = _open_mine_lock_file(lock_path, create=True)
try:
if _acquire_open_mine_lock_file(lf, lock_path):
return lf
except Exception:
lf.close()
raise
lf.close()
def _cleanup_mine_lock_file(lock_path: str) -> None:
"""Best-effort removal that preserves flock rendezvous semantics.
A plain ``os.remove(lock_path)`` after closing the critical-section lock is
unsafe on POSIX: a waiter may already be blocked on the old inode while a
later process creates and locks a new inode at the same pathname. Instead,
cleanup briefly re-acquires the current file nonblocking. If it wins, it can
unlink that inode as cleanup-only work; waiters on the old inode will detect
the stale handle after waking and retry on the current path.
"""
try:
lf = _open_mine_lock_file(lock_path, create=False)
except FileNotFoundError:
return
except OSError:
logger.debug("Mine-lock cleanup open failed for %s", lock_path, exc_info=True)
return
acquired = False
closed = False
try:
try:
acquired = _lock_mine_lock_file(lf, blocking=False)
except OSError:
logger.debug("Mine-lock cleanup acquire failed for %s", lock_path, exc_info=True)
return
if not acquired:
return
if not _mine_lock_file_is_current(lf, lock_path):
return
if os.name == "nt":
# Windows generally cannot unlink an open locked file. Release and
# close first; if another process opens the file in the gap,
# os.remove should fail and we leave the rendezvous file in place.
try:
_unlock_mine_lock_file(lf)
except Exception:
logger.debug("Mine-lock cleanup release failed", exc_info=True)
acquired = False
return
acquired = False
lf.close()
closed = True
try:
os.remove(lock_path)
except OSError:
pass
return
try:
os.remove(lock_path)
except FileNotFoundError:
pass
except OSError:
logger.debug("Mine-lock cleanup remove failed for %s", lock_path, exc_info=True)
finally:
if not closed:
if acquired:
try:
_unlock_mine_lock_file(lf)
except Exception:
logger.debug("Mine-lock cleanup release failed", exc_info=True)
lf.close()
class MineAlreadyRunning(RuntimeError):
"""Raised when another `mempalace mine` already holds the per-palace lock."""
+93 -8
View File
@@ -3,8 +3,8 @@ project_scanner.py — Detect projects and people from real signal.
For a codebase with build manifests or git history, this beats regex-based
entity detection by a wide margin: the project's own name is already written
down in package.json / pyproject.toml / Cargo.toml / go.mod, and the people
who worked on it are in `git log`.
down in package.json / pyproject.toml / Cargo.toml / go.mod / pom.xml /
Gradle manifests, and the people who worked on it are in `git log`.
This module is used as the primary signal in `mempalace init`. The regex
detector in entity_detector.py stays as a fallback for prose-only folders
@@ -21,6 +21,7 @@ import json
import os
import re
import subprocess
import xml.etree.ElementTree as ET
from dataclasses import dataclass, field
from pathlib import Path
from typing import Optional
@@ -164,11 +165,63 @@ def _parse_gomod(path: Path) -> Optional[str]:
return None
def _xml_local_name(tag: str) -> str:
return tag.rsplit("}", 1)[-1] if "}" in tag else tag
def _parse_pom(path: Path) -> Optional[str]:
try:
root = ET.parse(path).getroot()
except (ET.ParseError, OSError):
return None
for child in root:
if isinstance(child.tag, str) and _xml_local_name(child.tag) == "artifactId":
name = (child.text or "").strip()
return name or None
return None
_GRADLE_ROOT_PROJECT_NAME_PATTERNS = [
re.compile(r"""(?m)^\s*rootProject\.name\s*=\s*(["'])(?P<name>[^"']+)\1"""),
re.compile(r"""(?m)^\s*rootProject\.name\.set\(\s*(["'])(?P<name>[^"']+)\1\s*\)"""),
]
def _parse_gradle_root_project_name(path: Path) -> Optional[str]:
try:
text = path.read_text(encoding="utf-8", errors="replace")
except OSError:
return None
for pattern in _GRADLE_ROOT_PROJECT_NAME_PATTERNS:
match = pattern.search(text)
if match:
name = match.group("name").strip()
return name or None
return None
def _parse_gradle(path: Path) -> Optional[str]:
if path.name.startswith("build.gradle"):
for settings_name in ("settings.gradle.kts", "settings.gradle"):
name = _parse_gradle_root_project_name(path.with_name(settings_name))
if name:
return name
name = _parse_gradle_root_project_name(path)
if name:
return name
return path.parent.name or None
MANIFEST_PRIORITY = {
"pyproject.toml": 0,
"package.json": 1,
"Cargo.toml": 2,
"go.mod": 3,
"pom.xml": 4,
"settings.gradle": 5,
"settings.gradle.kts": 6,
"build.gradle": 7,
"build.gradle.kts": 8,
}
# Sentinel so unknown manifests always sort after the known manifest types above.
UNKNOWN_MANIFEST_PRIORITY = max(MANIFEST_PRIORITY.values()) + 1
@@ -177,6 +230,18 @@ MANIFEST_PARSERS = {
"pyproject.toml": _parse_pyproject,
"Cargo.toml": _parse_cargo,
"go.mod": _parse_gomod,
"pom.xml": _parse_pom,
"settings.gradle": _parse_gradle,
"settings.gradle.kts": _parse_gradle,
"build.gradle": _parse_gradle,
"build.gradle.kts": _parse_gradle,
}
JAVA_MANIFESTS = {
"pom.xml",
"settings.gradle",
"settings.gradle.kts",
"build.gradle",
"build.gradle.kts",
}
@@ -461,10 +526,12 @@ def scan(root: str | os.PathLike) -> tuple[list[ProjectInfo], list[PersonInfo]]:
for repo in repos:
manifests = _collect_manifest_names(repo)
if manifests:
manifest_file, proj_name, _ = manifests[0]
root_manifest = next((entry for entry in manifests if entry[2] == repo), None)
if root_manifest:
manifest_file, proj_name, _ = root_manifest
else:
manifest_file, proj_name = None, repo.name
extra_manifests = [entry for entry in manifests if entry != root_manifest]
authors = _git_authors(repo)
non_bot_authors = [(name, email) for name, email in authors if not _is_bot(name, email)]
@@ -501,17 +568,35 @@ def scan(root: str | os.PathLike) -> tuple[list[ProjectInfo], list[PersonInfo]]:
if existing is None or proj.user_commits > existing.user_commits:
projects[proj_name] = proj
for extra_manifest, extra_name, extra_dir in extra_manifests:
if extra_manifest not in JAVA_MANIFESTS:
continue
existing = projects.get(extra_name)
if existing is not None and (
existing.manifest is not None or existing.repo_root != repo
):
continue
projects[extra_name] = ProjectInfo(
name=extra_name,
repo_root=extra_dir,
manifest=extra_manifest,
has_git=True,
total_commits=total_commits,
user_commits=user_commits,
is_mine=is_mine,
)
people = _dedupe_people(all_commits)
# Handle case: root has manifests but no git repo anywhere
if not repos:
manifests = _collect_manifest_names(root_path)
for manifest_file, proj_name, _dirpath in manifests:
for manifest_file, proj_name, dirpath in manifests:
if proj_name in projects:
continue
projects[proj_name] = ProjectInfo(
name=proj_name,
repo_root=root_path,
repo_root=dirpath,
manifest=manifest_file,
has_git=False,
)
@@ -605,8 +690,8 @@ def discover_entities(
plugs into ``confirm_entities`` unchanged.
Order of signal preference:
1. Package manifests (package.json, pyproject.toml, Cargo.toml, go.mod)
→ canonical project names
1. Package manifests (package.json, pyproject.toml, Cargo.toml, go.mod,
pom.xml, Gradle manifests) → canonical project names
2. Git commit authors → real people with real commit counts
3. Claude Code conversation dirs (~/.claude/projects/) → per-session
project names (pulled from each session's ``cwd`` metadata)
+108 -7
View File
@@ -43,6 +43,7 @@ from typing import Callable, Iterator, Optional
from chromadb.errors import NotFoundError as ChromaNotFoundError
from .backends.chroma import ChromaBackend, hnsw_capacity_status
from .config import sqlite_read_uri
COLLECTION_NAME = "mempalace_drawers"
@@ -476,7 +477,7 @@ def sqlite_drawer_count(palace_path: str, collection_name: Optional[str] = None)
try:
import sqlite3
conn = sqlite3.connect(f"file:{sqlite_path}?mode=ro", uri=True)
conn = sqlite3.connect(sqlite_read_uri(sqlite_path), uri=True)
try:
row = conn.execute(
"""
@@ -516,7 +517,7 @@ def sqlite_integrity_errors(palace_path: str) -> list[str]:
return []
try:
with sqlite3.connect(f"file:{sqlite_path}?mode=ro", uri=True) as conn:
with sqlite3.connect(sqlite_read_uri(sqlite_path), uri=True) as conn:
rows = conn.execute("PRAGMA quick_check").fetchall()
except sqlite3.Error as e:
return [f"PRAGMA quick_check failed: {e}"]
@@ -563,6 +564,40 @@ def print_sqlite_integrity_abort(palace_path: str, errors: list[str]) -> None:
print(" 6. Re-run `mempalace repair --yes`.")
def index_read_recovery_guidance() -> str:
"""Recovery guidance for a failed drawer-index read in the legacy paths.
Both ``cmd_repair`` (cli.py) and :func:`rebuild_index` read the drawers
collection via ``Collection.count()`` as their first step. The common
reason that read raises is the chromadb compactor failing to apply the
WAL into the HNSW segment (``InternalError: Failed to apply logs to the
hnsw segment writer``, issues #1308 / #1843): the on-disk HNSW index is
corrupt while the rows stay intact in ``chroma.sqlite3``, so
:func:`rebuild_from_sqlite` (``repair --mode from-sqlite``) recovers them
and re-mining would needlessly drop drawers added through the MCP server
and diary entries that have no source file.
The other thing that strands this read is a live MemPalace server or
mine still holding the palace open, so the guidance says to stop it and
retry before assuming corruption. Worded conditionally because the bare
``except Exception`` cannot prove which case it caught. Returned as a
pre-indented block so the ``print``-based CLI path and the
``progress``-callable rebuild path emit it unchanged.
"""
return (
" If a MemPalace server or mine is still running against this palace,\n"
" stop it and retry. Otherwise the drawer index is likely corrupt\n"
" (for example a failed chromadb HNSW compaction) while your drawer\n"
" rows remain in chroma.sqlite3. Rebuild the index from SQLite rather\n"
" than re-mining:\n"
"\n"
" mempalace repair --mode from-sqlite --archive-existing\n"
"\n"
" (Re-mining from source files would drop drawers added via the MCP\n"
" server and diary entries, which have no source file.)"
)
def maybe_repair_poisoned_max_seq_id_before_rebuild(
palace_path: str,
*,
@@ -715,6 +750,19 @@ def _vacuum_and_rebuild_fts5(palace_path: str, progress=print) -> None:
progress(f" Warning: post-repair cleanup failed (non-fatal): {exc}")
def _post_rebuild_cleanup(palace_path: str, backend: "ChromaBackend", progress=print) -> None:
"""Close cached chroma handles, then VACUUM and rebuild the FTS5 index.
Shared epilogue for the two full-rebuild paths (``rebuild_index`` and the
CLI legacy ``cmd_repair``), so neither can drift out of the post-run
cleanup again (issues #1517, #1747). ChromaDB's PersistentClient keeps
chroma.sqlite3 open and VACUUM needs exclusive access, so the handles
are released first.
"""
_close_chroma_handles(palace_path, backend=backend)
_vacuum_and_rebuild_fts5(palace_path, progress=progress)
def rebuild_index(
palace_path=None,
confirm_truncation_ok: bool = False,
@@ -778,7 +826,7 @@ def rebuild_index(
total = col.count()
except Exception as e:
progress(f" Error reading palace: {e}")
progress(" Palace may need to be re-mined from source files.")
progress(index_read_recovery_guidance())
return
progress(f" Drawers found: {total}")
@@ -848,8 +896,7 @@ def rebuild_index(
print(" Live collection was not replaced; leaving the original palace untouched.")
raise
_close_chroma_handles(palace_path, backend=backend)
_vacuum_and_rebuild_fts5(palace_path, progress=progress)
_post_rebuild_cleanup(palace_path, backend=backend, progress=progress)
print(f"\n Repair complete. {filed} drawers rebuilt.")
print(" HNSW index is now clean with cosine distance metric.")
@@ -1001,7 +1048,7 @@ def extract_via_sqlite(palace_path: str, collection_name: str) -> Iterator[tuple
if not os.path.isfile(sqlite_path):
return
conn = sqlite3.connect(f"file:{sqlite_path}?mode=ro", uri=True)
conn = sqlite3.connect(sqlite_read_uri(sqlite_path), uri=True)
try:
seg_row = conn.execute(
"""
@@ -1047,6 +1094,37 @@ def extract_via_sqlite(palace_path: str, collection_name: str) -> Iterator[tuple
conn.close()
def _preserve_knowledge_graph_sqlite(source_palace: str, dest_palace: str) -> list[str]:
"""Copy KG SQLite sidecars when rebuilding a palace from chroma.sqlite3.
rebuild_from_sqlite reconstructs Chroma collections into a fresh
destination directory. The knowledge graph is a separate SQLite database,
so it must be copied explicitly or the repair succeeds while silently
dropping KG state (#1816).
"""
copied: list[str] = []
for suffix in ("", "-wal", "-shm"):
filename = f"knowledge_graph.sqlite3{suffix}"
src = os.path.join(source_palace, filename)
dst = os.path.join(dest_palace, filename)
if not os.path.isfile(src):
continue
if os.path.abspath(src) == os.path.abspath(dst):
continue
os.makedirs(dest_palace, exist_ok=True)
shutil.copy2(src, dst)
copied.append(filename)
if copied:
print(" Preserved knowledge graph: " + ", ".join(copied))
return copied
def rebuild_from_sqlite(
source_palace: str,
dest_palace: str,
@@ -1193,6 +1271,7 @@ def rebuild_from_sqlite(
)
os.makedirs(dest_palace, exist_ok=True)
_preserve_knowledge_graph_sqlite(source_palace, dest_palace)
# Backend lifetime is wrapped in try/finally so the dest palace's
# PersistentClient handle (opened lazily inside ``create_collection``
@@ -1296,7 +1375,15 @@ def status(palace_path=None, collection_name: Optional[str] = None) -> dict:
print(f" note: {info['message']}")
if drawers["diverged"] or closets["diverged"]:
print("\n Recommended: run `mempalace repair` to rebuild the index.")
print(
"\n Recommended: rebuild the index from SQLite rather than re-mining:\n"
"\n mempalace repair --mode from-sqlite --archive-existing\n"
"\n A diverged index usually means the HNSW segment is out of sync with\n"
" chroma.sqlite3 (for example a failed chromadb HNSW compaction). The\n"
" drawer rows are intact in SQLite, so --mode from-sqlite recovers them.\n"
" Do not re-mine from source files: that would drop drawers added via\n"
" the MCP server and diary entries, which have no source file (#1843)."
)
print()
return {"drawers": drawers, "closets": closets}
@@ -1529,12 +1616,26 @@ def repair_max_seq_id(
return result
if backup:
import glob
from .backups import prune_backups
from .config import MempalaceConfig
timestamp = datetime.now().strftime("%Y%m%d-%H%M%S")
backup_path = os.path.join(palace_path, f"chroma.sqlite3.max-seq-id-backup-{timestamp}")
shutil.copy2(db_path, backup_path)
result["backup"] = backup_path
print(f" Backup: {backup_path}")
# Retain only the most recent N backups (the copy just written is the
# newest and is kept). Without this, every max-seq-id repair leaves a
# full chroma.sqlite3 copy behind that is never cleaned up.
prune_backups(
os.path.join(glob.escape(palace_path), "chroma.sqlite3.max-seq-id-backup-*"),
MempalaceConfig().max_backups,
log=print,
)
_close_chroma_handles(palace_path)
with sqlite3.connect(db_path) as conn:
+175 -43
View File
@@ -23,6 +23,7 @@ from .backends import (
PalaceNotFoundError,
UnsupportedCapabilityError,
)
from .config import sqlite_read_uri
from .palace import (
_open_collection_or_explain,
get_closets_collection,
@@ -130,18 +131,70 @@ def _bm25_scores(
return scores
def _distance_to_similarity(distance, metric: str = "cosine") -> float:
"""Map a backend-reported ``distance`` to a [0, 1]-ish similarity.
The backend contract for the ``distances`` field is *lower = closer*
regardless of metric (RFC 001, backend metric declaration), so every
mapping here is monotonic decreasing in ``distance``. The output stays
bounded so it is
commensurable with the min-max-normalized BM25 term in
:func:`_hybrid_rank`.
* ``cosine`` — distance ∈ [0, 2], 0 = identical: ``max(0, 1 - d)``.
* ``l2`` — Euclidean ∈ [0, ∞): ``1 / (1 + d)`` (1 at d=0, →0 as d→∞).
* ``ip`` — inner-product distance (e.g. pgvector ``<#>`` = -dot, lower =
closer), unbounded and signed: logistic squash ``1 / (1 + e^d)``.
Provisional until a real ip backend exercises it; no in-tree backend
uses ip today.
``distance is None`` (vector-unknown, e.g. a BM25-only candidate) maps to
0.0 so the candidate scores on its BM25 contribution alone.
"""
if distance is None:
return 0.0
m = (metric or "cosine").lower()
if m == "l2":
return 1.0 / (1.0 + max(0.0, distance))
if m == "ip":
# Clamp the exponent so a large positive distance can't overflow.
return 1.0 / (1.0 + math.exp(min(60.0, distance)))
# cosine (default)
return max(0.0, 1.0 - distance)
def _metric_for_collection(col) -> str:
"""Resolve a collection's declared distance metric, defaulting to cosine.
Reads the ``distance_metric`` exposed by the backend collection (the
RFC 001 backend metric declaration). ``EmbeddingCollection`` delegates the
attribute to its inner collection; legacy Chroma palaces report their
actual ``hnsw:space``.
Any failure falls back to ``"cosine"`` — the value all in-tree backends
use and the only metric MemPalace created palaces with historically.
"""
try:
metric = getattr(col, "distance_metric", "cosine")
except Exception:
return "cosine"
metric = str(metric or "cosine").lower()
return metric if metric in ("cosine", "l2", "ip") else "cosine"
def _hybrid_rank(
results: list,
query: str,
vector_weight: float = 0.6,
bm25_weight: float = 0.4,
metric: str = "cosine",
) -> list:
"""Re-rank ``results`` by a convex combination of vector similarity and BM25.
* Vector similarity uses absolute cosine sim ``max(0, 1 - distance)`` —
ChromaDB's hnsw cosine distance lives in ``[0, 2]`` (0 = identical).
Absolute (not relative-to-max) means adding/removing a candidate
can't reshuffle the others.
* Vector similarity is derived from each candidate's backend-reported
``distance`` via :func:`_distance_to_similarity`, interpreted in the
collection's declared ``metric`` (per RFC 001) rather than assuming
cosine. Absolute (not relative-to-max) means adding/removing a
candidate can't reshuffle the others.
* BM25 is real Okapi-BM25 with corpus-relative IDF over the candidates
themselves. Since the absolute scale is unbounded, BM25 is min-max
normalized within the candidate set so weights are commensurable.
@@ -164,11 +217,7 @@ def _hybrid_rank(
scored = []
for r, raw, norm in zip(results, bm25_raw, bm25_norm):
distance = r.get("distance")
if distance is None:
vec_sim = 0.0
else:
vec_sim = max(0.0, 1.0 - distance)
vec_sim = _distance_to_similarity(r.get("distance"), metric)
r["bm25_score"] = round(raw, 3)
scored.append((vector_weight * vec_sim + bm25_weight * norm, r))
@@ -177,15 +226,24 @@ def _hybrid_rank(
return results
def build_where_filter(wing: str = None, room: str = None) -> dict:
"""Build ChromaDB where filter for wing/room filtering."""
if wing and room:
return {"$and": [{"wing": wing}, {"room": room}]}
elif wing:
return {"wing": wing}
elif room:
return {"room": room}
return {}
def build_where_filter(wing: str = None, room: str = None, source_file: str = None) -> dict:
"""Build a ChromaDB where filter from optional wing/room/source_file.
ChromaDB needs a ``$and`` only when ≥2 clauses are present; a single
clause is returned bare and zero clauses yield an empty filter (#1815).
"""
clauses = []
if wing:
clauses.append({"wing": wing})
if room:
clauses.append({"room": room})
if source_file:
clauses.append({"source_file": source_file})
if not clauses:
return {}
if len(clauses) == 1:
return clauses[0]
return {"$and": clauses}
def _extract_drawer_ids_from_closet(closet_doc: str) -> list:
@@ -202,6 +260,31 @@ def _extract_drawer_ids_from_closet(closet_doc: str) -> list:
return list(seen.keys())
def _scoped_source_filter(source_file: str, parent_drawer_id=None) -> dict:
"""Build a Chroma ``where`` clause that scopes a query to ``source_file``,
additionally constrained by ``parent_drawer_id`` when one is supplied.
Two unrelated oversized ``tool_add_drawer`` writes (chunked path from
#1539) can pass the same ``source_file`` (e.g. two pastes tagged
``"chat.log"``); each call stores its own ``parent_drawer_id`` group
of chunks but the bare ``source_file`` filter pulls chunks from both
groups as if they were siblings (#1580). When the matched chunk
carries a ``parent_drawer_id`` the filter narrows to that logical
group. Otherwise (pre-#1539 drawers, single-chunk writes, and
``diary_ingest`` drawers grouped by real file path) the original
file-global shape is preserved. Mirrors the conditional-``$and``
precedent in ``build_where_filter``.
"""
if parent_drawer_id:
return {
"$and": [
{"source_file": source_file},
{"parent_drawer_id": parent_drawer_id},
]
}
return {"source_file": source_file}
def _expand_with_neighbors(drawers_col, matched_doc: str, matched_meta: dict, radius: int = 1):
"""Expand a matched drawer with its ±radius sibling chunks in the same source file.
@@ -225,15 +308,20 @@ def _expand_with_neighbors(drawers_col, matched_doc: str, matched_meta: dict, ra
if not src or not isinstance(chunk_idx, int):
return {"text": matched_doc, "drawer_index": chunk_idx, "total_drawers": None}
# Narrow by ``parent_drawer_id`` when present so chunks from unrelated
# logical drawers sharing ``source_file`` do not stitch (#1580). See
# ``_scoped_source_filter`` for the contract.
parent_id = matched_meta.get("parent_drawer_id")
target_indexes = [chunk_idx + offset for offset in range(-radius, radius + 1)]
neighbor_clauses = [
{"source_file": src},
{"chunk_index": {"$in": target_indexes}},
]
if parent_id:
neighbor_clauses.append({"parent_drawer_id": parent_id})
try:
neighbors = drawers_col.get(
where={
"$and": [
{"source_file": src},
{"chunk_index": {"$in": target_indexes}},
]
},
where={"$and": neighbor_clauses},
include=["documents", "metadatas"],
)
except Exception:
@@ -251,10 +339,16 @@ def _expand_with_neighbors(drawers_col, matched_doc: str, matched_meta: dict, ra
else:
combined_text = "\n\n".join(doc for _, doc in indexed_docs)
# Cheap total_drawers lookup: metadata-only scan of the source file.
# Cheap total_drawers lookup. When ``parent_drawer_id`` is present the
# count is scoped to that group so the returned number matches the
# text the caller gets back. Without a parent id, the legacy
# file-global count is preserved.
total_drawers = None
try:
all_meta = drawers_col.get(where={"source_file": src}, include=["metadatas"])
all_meta = drawers_col.get(
where=_scoped_source_filter(src, parent_id),
include=["metadatas"],
)
total_drawers = len(all_meta.ids) if all_meta.ids else None
except Exception:
logger.debug("total_drawers lookup failed for %s", src, exc_info=True)
@@ -350,11 +444,12 @@ def search(query: str, palace_path: str, wing: str = None, room: str = None, n_r
# The MCP tool path already hybridizes BM25 with vector sim via
# `_hybrid_rank`; do the same here so CLI results match what agents
# see via `mempalace_search`.
metric = _metric_for_collection(col)
hits = [
{"text": doc or "", "distance": float(dist), "metadata": meta or {}}
for doc, meta, dist in zip(docs, metas, dists)
]
hits = _hybrid_rank(hits, query)
hits = _hybrid_rank(hits, query, metric=metric)
print(f"\n{'=' * 60}")
print(f' Results for: "{query}"')
@@ -365,7 +460,7 @@ def search(query: str, palace_path: str, wing: str = None, room: str = None, n_r
print(f"{'=' * 60}\n")
for i, hit in enumerate(hits, 1):
vec_sim = round(max(0.0, 1 - hit["distance"]), 3)
vec_sim = round(_distance_to_similarity(hit["distance"], metric), 3)
bm25 = hit.get("bm25_score", 0.0)
meta = hit["metadata"]
source = Path(meta.get("source_file", "?")).name
@@ -374,7 +469,7 @@ def search(query: str, palace_path: str, wing: str = None, room: str = None, n_r
print(f" [{i}] {wing_name} / {room_name}")
print(f" Source: {source}")
print(f" Match: cosine={vec_sim} bm25={bm25}")
print(f" Match: {metric}_sim={vec_sim} bm25={bm25}")
print()
# Print the verbatim text, indented
for line in hit["text"].strip().split("\n"):
@@ -390,6 +485,7 @@ def _bm25_only_via_sqlite(
palace_path: str,
wing: str = None,
room: str = None,
source_file: str = None,
n_results: int = 5,
max_candidates: int = 500,
_include_internal: bool = False,
@@ -425,7 +521,7 @@ def _bm25_only_via_sqlite(
def _metadata_filter_sql(row_id_expr: str) -> tuple[str, list[str]]:
clauses = []
params = []
for key, value in (("wing", wing), ("room", room)):
for key, value in (("wing", wing), ("room", room), ("source_file", source_file)):
if not value:
continue
clauses.append(
@@ -448,7 +544,7 @@ def _bm25_only_via_sqlite(
return "".join(clauses), params
try:
conn = sqlite3.connect(f"file:{db_path}?mode=ro", uri=True)
conn = sqlite3.connect(sqlite_read_uri(db_path), uri=True)
except sqlite3.Error as e:
return {"error": f"sqlite open failed: {e}"}
@@ -538,7 +634,7 @@ def _bm25_only_via_sqlite(
if not candidate_ids:
return {
"query": query,
"filters": {"wing": wing, "room": room},
"filters": {"wing": wing, "room": room, "source_file": source_file},
"total_before_filter": 0,
"results": [],
"fallback": "bm25_only_via_sqlite",
@@ -574,6 +670,8 @@ def _bm25_only_via_sqlite(
continue
if room and meta.get("room") != room:
continue
if source_file and meta.get("source_file") != source_file:
continue
full_source = meta.get("source_file", "") or ""
candidates.append(
{
@@ -581,6 +679,7 @@ def _bm25_only_via_sqlite(
"wing": meta.get("wing", "unknown"),
"room": meta.get("room", "unknown"),
"source_file": Path(full_source).name if full_source else "?",
"source_path": full_source,
"created_at": meta.get("filed_at", "unknown"),
# No vector distance available in BM25-only mode.
"similarity": None,
@@ -616,7 +715,7 @@ def _bm25_only_via_sqlite(
return {
"query": query,
"filters": {"wing": wing, "room": room},
"filters": {"wing": wing, "room": room, "source_file": source_file},
"total_before_filter": len(candidates),
"results": hits,
"fallback": "bm25_only_via_sqlite",
@@ -632,6 +731,7 @@ def _merge_bm25_union_candidates(
room: str,
n_results: int,
max_distance: float = 0.0,
source_file: str = None,
) -> None:
"""Append top-K backend lexical candidates into ``hits`` in place.
@@ -658,7 +758,7 @@ def _merge_bm25_union_candidates(
if max_distance > 0.0:
return
where = build_where_filter(wing, room)
where = build_where_filter(wing, room, source_file)
try:
lexical = drawers_col.lexical_search(
query=query,
@@ -681,6 +781,7 @@ def _merge_bm25_union_candidates(
"wing": meta.get("wing", "unknown"),
"room": meta.get("room", "unknown"),
"source_file": Path(full_source).name if full_source else "?",
"source_path": full_source,
"created_at": meta.get("filed_at", "unknown"),
"similarity": None,
"distance": None,
@@ -746,6 +847,7 @@ def _apply_candidate_strategy(
room: str,
n_results: int,
max_distance: float = 0.0,
source_file: str = None,
) -> None:
"""Dispatch to the registered merger for ``strategy``.
@@ -754,7 +856,16 @@ def _apply_candidate_strategy(
"""
merger = _CANDIDATE_MERGERS[strategy]
if merger is not None:
merger(hits, drawers_col, query, wing, room, n_results, max_distance=max_distance)
merger(
hits,
drawers_col,
query,
wing,
room,
n_results,
max_distance=max_distance,
source_file=source_file,
)
def _finalize_candidate_hits(
@@ -767,6 +878,7 @@ def _finalize_candidate_hits(
room: str,
n_results: int,
max_distance: float,
source_file: str = None,
) -> tuple:
try:
_apply_candidate_strategy(
@@ -778,6 +890,7 @@ def _finalize_candidate_hits(
room,
n_results,
max_distance=max_distance,
source_file=source_file,
)
except UnsupportedCapabilityError:
return [], {
@@ -786,11 +899,12 @@ def _finalize_candidate_hits(
"hint": "Use candidate_strategy='vector' or select a backend that supports lexical search.",
}
hits = _hybrid_rank(hits, query)[:n_results]
hits = _hybrid_rank(hits, query, metric=_metric_for_collection(drawers_col))[:n_results]
for h in hits:
h.pop("_sort_key", None)
h.pop("_source_file_full", None)
h.pop("_chunk_index", None)
h.pop("_parent_drawer_id", None)
return hits, None
@@ -818,6 +932,7 @@ def _vector_disabled_search(
room: str,
n_results: int,
collection_name: str,
source_file: str = None,
) -> dict:
try:
backend_name = resolve_backend_name(palace_path)
@@ -837,6 +952,7 @@ def _vector_disabled_search(
palace_path,
wing=wing,
room=room,
source_file=source_file,
n_results=n_results,
collection_name=collection_name,
)
@@ -870,7 +986,9 @@ def _open_search_collection(palace_path: str, collection_name: str):
}
def _query_drawers_with_filter_fallback(drawers_col, dkwargs, query, n_results, wing, room):
def _query_drawers_with_filter_fallback(
drawers_col, dkwargs, query, n_results, wing, room, source_file=None
):
"""Run the filtered drawer query, falling back to an unfiltered query plus a
Python-side post-filter when ChromaDB raises on the filtered query.
@@ -878,7 +996,7 @@ def _query_drawers_with_filter_fallback(drawers_col, dkwargs, query, n_results,
"Error finding id" even when unfiltered search works fine — it happens when
drawers are ingested via two different paths (e.g. bulk import vs MCP tool
calls), leaving the vector index inconsistent with the metadata store. We
retry unfiltered (over-fetching) and re-apply the wing/room filter in Python.
retry unfiltered (over-fetching) and re-apply the wing/room/source_file filter in Python.
See #1245 / #1035.
"""
where = dkwargs.get("where")
@@ -907,6 +1025,8 @@ def _query_drawers_with_filter_fallback(drawers_col, dkwargs, query, n_results,
continue
if room and meta.get("room") != room:
continue
if source_file and meta.get("source_file") != source_file:
continue
fdocs.append(doc)
fmetas.append(meta)
fdists.append(dist)
@@ -918,6 +1038,7 @@ def search_memories(
palace_path: str,
wing: str = None,
room: str = None,
source_file: str = None,
n_results: int = 5,
max_distance: float = 0.0,
vector_disabled: bool = False,
@@ -933,6 +1054,8 @@ def search_memories(
palace_path: Path to the ChromaDB palace directory.
wing: Optional wing filter.
room: Optional room filter.
source_file: Optional exact source_file filter. Matches the full
stored source_file value verbatim (#1815).
n_results: Max results to return.
max_distance: Max cosine distance threshold. The palace collection uses
cosine distance (hnsw:space=cosine) — 0 = identical, 2 = opposite.
@@ -972,13 +1095,15 @@ def search_memories(
room=room,
n_results=n_results,
collection_name=collection_name,
source_file=source_file,
)
drawers_col, open_error = _open_search_collection(palace_path, collection_name)
if open_error:
return open_error
where = build_where_filter(wing, room)
metric = _metric_for_collection(drawers_col)
where = build_where_filter(wing, room, source_file)
# Hybrid retrieval: always query drawers directly (the floor), then use
# closet hits to boost rankings. Closets are a ranking SIGNAL, never a
@@ -996,7 +1121,7 @@ def search_memories(
if where:
dkwargs["where"] = where
drawer_results = _query_drawers_with_filter_fallback(
drawers_col, dkwargs, query, n_results, wing, room
drawers_col, dkwargs, query, n_results, wing, room, source_file
)
except Exception as e:
return {"error": f"Search error: {e}"}
@@ -1068,9 +1193,12 @@ def search_memories(
"text": doc,
"wing": meta.get("wing", "unknown"),
"room": meta.get("room", "unknown"),
# source_file is the basename (display); source_path is the full
# stored value, the round-trippable key for the source_file filter.
"source_file": Path(source).name if source else "?",
"source_path": source,
"created_at": meta.get("filed_at", "unknown"),
"similarity": round(max(0.0, 1 - effective_dist), 3),
"similarity": round(_distance_to_similarity(effective_dist, metric), 3),
"distance": round(dist, 4),
"effective_distance": round(effective_dist, 4),
"closet_boost": round(boost, 3),
@@ -1082,6 +1210,7 @@ def search_memories(
"_sort_key": effective_dist,
"_source_file_full": source,
"_chunk_index": meta.get("chunk_index"),
"_parent_drawer_id": meta.get("parent_drawer_id"),
}
if closet_preview:
entry["closet_preview"] = closet_preview
@@ -1102,9 +1231,11 @@ def search_memories(
full_source = h.get("_source_file_full") or ""
if not full_source:
continue
# Narrow by ``parent_drawer_id`` when present so unrelated
# chunked drawers sharing ``source_file`` do not stitch (#1580).
try:
source_drawers = drawers_col.get(
where={"source_file": full_source},
where=_scoped_source_filter(full_source, h.get("_parent_drawer_id")),
include=["documents", "metadatas"],
)
except Exception:
@@ -1163,13 +1294,14 @@ def search_memories(
room=room,
n_results=n_results,
max_distance=max_distance,
source_file=source_file,
)
if strategy_error:
return strategy_error
return {
"query": query,
"filters": {"wing": wing, "room": room},
"filters": {"wing": wing, "room": room, "source_file": source_file},
"total_before_filter": len(_first_or_empty(drawer_results, "documents")),
"results": hits,
}
+404
View File
@@ -0,0 +1,404 @@
"""Shared service operations used by daemon-backed entry points.
The MCP server remains the owner of MCP transport details. This module owns the
small, transport-neutral execution surface the daemon needs: classify known
tools and execute durable background jobs without printing directly to the
caller's terminal.
"""
from __future__ import annotations
import contextlib
import io
import os
import sys
from typing import Any
from .config import MempalaceConfig
_EXPLICIT_BACKEND_ENV = "MEMPALACE_BACKEND_EXPLICIT"
_PALACE_PATH_ENV = "MEMPALACE_PALACE_PATH"
_BACKEND_ENV = "MEMPALACE_BACKEND"
# Env vars a job may mutate via _apply_backend / palace_path injection. They are
# snapshotted per job and restored afterward so a job that switches the backend
# (e.g. qdrant) cannot poison every later job in the same daemon process —
# including mcp_tool jobs, which read MempalaceConfig (and thus the leaked env).
_PER_JOB_ENV = (_PALACE_PATH_ENV, _BACKEND_ENV, _EXPLICIT_BACKEND_ENV)
READ_TOOLS = frozenset(
{
"mempalace_status",
"mempalace_list_wings",
"mempalace_list_rooms",
"mempalace_get_taxonomy",
"mempalace_get_aaak_spec",
"mempalace_traverse",
"mempalace_find_tunnels",
"mempalace_graph_stats",
"mempalace_list_tunnels",
"mempalace_list_hallways",
"mempalace_follow_tunnels",
"mempalace_search",
"mempalace_check_duplicate",
"mempalace_get_drawer",
"mempalace_list_drawers",
"mempalace_diary_read",
"mempalace_memories_filed_away",
"mempalace_kg_query",
"mempalace_kg_stats",
"mempalace_kg_timeline",
}
)
WRITE_TOOLS = frozenset(
{
"mempalace_add_drawer",
"mempalace_checkpoint",
"mempalace_delete_drawer",
"mempalace_update_drawer",
"mempalace_diary_write",
"mempalace_kg_add",
"mempalace_kg_invalidate",
"mempalace_create_tunnel",
"mempalace_delete_tunnel",
"mempalace_delete_hallway",
"mempalace_hook_settings",
}
)
MAINTENANCE_TOOLS = frozenset({"mempalace_mine", "mempalace_sync", "mempalace_reconnect"})
def classify_tool(name: str) -> str:
"""Return ``read``, ``write``, ``maintenance``, or ``unknown`` for an MCP tool."""
if name in READ_TOOLS:
return "read"
if name in WRITE_TOOLS:
return "write"
if name in MAINTENANCE_TOOLS:
return "maintenance"
return "unknown"
def _apply_backend(backend: str | None) -> None:
if not backend:
return
backend_name = str(backend).strip().lower()
from .backends import get_backend_class
get_backend_class(backend_name)
os.environ[_EXPLICIT_BACKEND_ENV] = backend_name
os.environ[_BACKEND_ENV] = backend_name
def _capture(fn):
stdout = io.StringIO()
stderr = io.StringIO()
with contextlib.redirect_stdout(stdout), contextlib.redirect_stderr(stderr):
result = fn()
return result, stdout.getvalue(), stderr.getvalue()
def execute_job(kind: str, payload: dict[str, Any]) -> dict[str, Any]:
"""Execute one daemon job and return a JSON-serializable result."""
def _run():
if kind == "mine":
return run_mine(payload)
if kind == "sync":
return run_sync(payload)
if kind == "diary_write":
return run_diary_write(payload)
if kind == "mcp_tool":
return run_mcp_tool(payload)
return {"success": False, "error": f"unknown daemon job kind: {kind}", "exit_code": 2}
# Per-job env isolation: snapshot the backend/palace env vars and restore
# them after the job so one job's _apply_backend / palace_path injection
# can't leak into the next job in the same long-lived process.
saved_env = {key: os.environ.get(key) for key in _PER_JOB_ENV}
try:
result, stdout, stderr = _capture(_run)
finally:
for key, value in saved_env.items():
if value is None:
os.environ.pop(key, None)
else:
os.environ[key] = value
if result is None:
result = {}
if not isinstance(result, dict):
result = {"success": True, "value": result}
result.setdefault("success", True)
result.setdefault("exit_code", 0 if result.get("success") else 1)
if stdout:
result["stdout"] = stdout
if stderr:
result["stderr"] = stderr
return result
def run_mine(payload: dict[str, Any]) -> dict[str, Any]:
"""Run the same mine operation as the CLI, without daemon transport concerns."""
palace_path = os.path.abspath(
os.path.expanduser(payload.get("palace_path") or MempalaceConfig().palace_path)
)
os.environ["MEMPALACE_PALACE_PATH"] = palace_path
_apply_backend(payload.get("backend"))
source = payload.get("source") or payload.get("dir")
mode = payload.get("mode") or "projects"
wing = payload.get("wing")
agent = payload.get("agent") or "mempalace"
limit = int(payload.get("limit") or 0)
dry_run = bool(payload.get("dry_run"))
if payload.get("redetect_origin"):
from .cli import _run_pass_zero
_run_pass_zero(project_dir=source, palace_dir=palace_path, llm_provider=None)
from .palace import MineAlreadyRunning, MineValidationError
try:
if mode == "convos":
from .convo_miner import mine_convos
mine_convos(
convo_dir=source,
palace_path=palace_path,
wing=wing,
agent=agent,
limit=limit,
dry_run=dry_run,
extract_mode=payload.get("extract") or "exchange",
)
elif mode == "extract":
from .format_miner import mine_formats
mine_formats(
format_dir=source,
palace_path=palace_path,
wing=wing,
agent=agent,
limit=limit,
dry_run=dry_run,
)
elif mode == "projects":
include_ignored = payload.get("include_ignored") or []
from .miner import mine
mine(
project_dir=source,
palace_path=palace_path,
wing_override=wing,
agent=agent,
limit=limit,
dry_run=dry_run,
respect_gitignore=not bool(payload.get("no_gitignore")),
include_ignored=include_ignored,
max_chunks_per_file=payload.get("max_chunks_per_file"),
)
else:
return {"success": False, "error": f"invalid mine mode: {mode}", "exit_code": 2}
except MineAlreadyRunning as exc:
return {
"success": False,
"error": str(exc),
"error_class": "LockHeldByOtherProcess",
"exit_code": 1,
}
except MineValidationError as exc:
return {
"success": False,
"error": str(exc),
"error_class": "MineValidationError",
"exit_code": 1,
}
except SystemExit as exc:
code = exc.code if isinstance(exc.code, int) else 1
return {
"success": code == 0,
"error": str(exc),
"error_class": "SystemExit",
"exit_code": code,
}
except Exception as exc:
return {"success": False, "error": f"mine failed: {exc}", "exit_code": 1}
return {"success": True, "kind": "mine", "mode": mode, "dry_run": dry_run, "exit_code": 0}
def run_sync(payload: dict[str, Any]) -> dict[str, Any]:
"""Run sync and render the same operator-facing summary shape as the CLI."""
palace_path = os.path.abspath(
os.path.expanduser(payload.get("palace_path") or MempalaceConfig().palace_path)
)
os.environ["MEMPALACE_PALACE_PATH"] = palace_path
_apply_backend(payload.get("backend"))
from .backends import detect_backend_for_path
from .palace import MineAlreadyRunning, _backend_artifact_label, resolve_backend_name
if not os.path.isdir(palace_path):
print(f"\n No palace found at {palace_path}")
return {"success": True, "exit_code": 0}
try:
backend_name = resolve_backend_name(palace_path)
except Exception as exc:
return {
"success": False,
"error": f"Could not resolve palace backend: {exc}",
"exit_code": 1,
}
if detect_backend_for_path(palace_path) is None:
print(
f"\n Palace dir at {palace_path} exists but has no "
f"{_backend_artifact_label(backend_name)} yet."
)
print(" Run: mempalace mine <dir>")
return {"success": True, "exit_code": 0}
project_dirs = []
if payload.get("dir"):
project_dirs.append(os.path.expanduser(str(payload["dir"])))
project_dirs.extend(os.path.expanduser(str(root)) for root in payload.get("root") or [])
project_dirs = project_dirs or None
dry_run = bool(payload.get("dry_run", True))
print(f"\n{'=' * 55}")
print(" MemPalace Sync — Gitignore-aware drawer prune")
print(f"{'=' * 55}")
print(f" Palace: {palace_path}")
if payload.get("wing"):
print(f" Wing: {payload['wing']}")
if project_dirs:
for project_dir in project_dirs:
print(f" Project: {project_dir}")
print(
" Mode: DRY RUN (no deletions)" if dry_run else " Mode: APPLY (deleting drawers)"
)
print(f"{'-' * 55}\n")
try:
from .sync import sync_palace
from .wal import _wal_log
report = sync_palace(
palace_path=palace_path,
project_dirs=project_dirs,
wing=payload.get("wing"),
dry_run=dry_run,
wal_log=_wal_log,
)
except MineAlreadyRunning as exc:
return {
"success": False,
"error": str(exc),
"error_class": "LockHeldByOtherProcess",
"exit_code": 1,
}
except ValueError as exc:
return {"success": False, "error": str(exc), "exit_code": 2}
except Exception as exc:
return {"success": False, "error": f"sync failed: {exc}", "exit_code": 1}
removed_suffix = "(would remove)" if dry_run else "(removed)"
print(f" Scanned: {report['scanned']}")
print(f" Kept: {report['kept']}")
print(f" Gitignored: {report['gitignored']} {removed_suffix}")
print(f" Missing: {report['missing']} {removed_suffix}")
print(f" No source: {report['no_source']} (kept)")
print(f" Out of scope: {report['out_of_scope']} (kept)")
by_source = report.get("by_source") or {}
if by_source:
top = sorted(by_source.items(), key=lambda kv: -kv[1])[:5]
label = "Top sources to remove" if dry_run else "Top sources removed"
print(f"\n {label}:")
for src, n in top:
print(f" {src} ({n})")
if dry_run:
if report["gitignored"] + report["missing"] > 0:
print("\n Re-run with --apply to commit these deletions.")
else:
print(
f"\n Removed {report['removed_drawers']} drawers, {report['removed_closets']} closets."
)
print(f"\n{'=' * 55}\n")
return {"success": True, "report": report, "exit_code": 0}
def run_diary_write(payload: dict[str, Any]) -> dict[str, Any]:
palace_path = payload.get("palace_path")
if palace_path:
os.environ["MEMPALACE_PALACE_PATH"] = os.path.abspath(os.path.expanduser(palace_path))
_apply_backend(payload.get("backend"))
from .mcp_server import tool_diary_write
result = tool_diary_write(
agent_name=payload.get("agent_name") or "mempalace",
entry=payload.get("entry") or "",
topic=payload.get("topic") or "general",
wing=payload.get("wing") or "",
)
result.setdefault("exit_code", 0 if result.get("success") else 1)
return result
def run_mcp_tool(payload: dict[str, Any]) -> dict[str, Any]:
"""Execute an MCP tool by name over the daemon queue.
The daemon is a durable, retried write surface — not a general MCP transport.
Restrict ``mcp_tool`` to write-classified tools only: read tools would
exfiltrate verbatim palace content into the queue DB and the job result
(stored world-readable-by-default without the perms fix, and returned over
/jobs), and maintenance tools already have their own dedicated kinds
(mine/sync). No internal caller currently uses ``mcp_tool``; this allowlist
bounds the blast radius of the generic escape hatch.
"""
name = payload.get("name")
arguments = payload.get("arguments") or {}
if not isinstance(arguments, dict):
return {"success": False, "error": "arguments must be an object", "exit_code": 2}
classification = classify_tool(name) if name else "unknown"
if classification != "write":
return {
"success": False,
"error": f"daemon mcp_tool only accepts write tools; {name!r} is {classification}",
"exit_code": 2,
}
from .mcp_server import TOOLS
if name not in TOOLS:
return {"success": False, "error": f"unknown MCP tool: {name}", "exit_code": 2}
result = TOOLS[name]["handler"](**arguments)
if isinstance(result, dict):
# Several write tools signal failure with a bare {"error": ...} and no
# explicit success flag (e.g. tool_create_tunnel / tool_delete_tunnel
# validation paths). Infer failure from the "error" key so the daemon
# does not persist a failed write as succeeded with exit_code 0.
if "success" not in result:
result["success"] = "error" not in result
result.setdefault("exit_code", 0 if result.get("success") else 1)
return result
return {"success": True, "value": result, "exit_code": 0}
def print_job_result(result: dict[str, Any]) -> int:
"""Replay captured daemon job output and return the intended process exit code."""
stdout = result.get("stdout")
stderr = result.get("stderr")
if stdout:
print(stdout, end="")
if stderr:
print(stderr, end="", file=sys.stderr)
if not result.get("success", True) and result.get("error") and not stderr:
print(f"mempalace: {result['error']}", file=sys.stderr)
return int(result.get("exit_code", 0 if result.get("success", True) else 1) or 0)
+1 -1
View File
@@ -1,3 +1,3 @@
"""Single source of truth for the MemPalace package version."""
__version__ = "3.4.0"
__version__ = "3.5.0"
+97
View File
@@ -0,0 +1,97 @@
"""Side-effect-free write-ahead log for MemPalace write operations.
This lives in its own module so callers that only need WAL audit logging — the
CLI ``sync`` path and the daemon's ``service`` layer — can obtain ``_wal_log``
without importing :mod:`mempalace.mcp_server`. Importing ``mcp_server`` runs its
module-level stdio protection (``os.dup2(2, 1)`` and ``sys.stdout = sys.stderr``,
required so the MCP stdio JSON stream isn't corrupted by C-level library
banners). In a non-MCP process — e.g. the daemon worker or ``mempalace sync`` —
that redirect is an unwanted import side effect that misroutes operator output,
so the WAL machinery is kept here, free of any such side effects.
"""
from __future__ import annotations
import json
import logging
import os
from datetime import datetime
from pathlib import Path
logger = logging.getLogger(__name__)
_WAL_FILE = Path(os.path.expanduser("~/.mempalace/wal")) / "write_log.jsonl"
_WAL_INITIALIZED_DIR = None
# Keys whose values should be redacted in WAL entries to avoid logging sensitive content
_WAL_REDACT_KEYS = frozenset(
{"content", "content_preview", "document", "entry", "entry_preview", "query", "text"}
)
def _ensure_wal() -> None:
"""Create (and re-harden) the WAL directory lazily, on the first write.
This must NOT run at import time: a user who removed ``~/.mempalace`` has
engaged the documented kill-switch (``hooks_cli._palace_root_exists()``,
#1305), and recreating the directory just by importing this module would
silently re-arm the autosave/mining hooks they disabled (#1676). Creating
it on the first real write keeps the kill-switch contract intact.
It is deliberately not gated on ``_palace_root_exists()``: by the time a
write reaches here the palace is already being recreated by the ChromaDB/KG
layer regardless, so gating would only drop audit records, not prevent
recreation. Runtime kill-switch enforcement for MCP writes is the broader
question tracked in #504.
Hardening is attempted once per directory and the path cached in
``_WAL_INITIALIZED_DIR`` regardless of outcome (keyed on the path, so a
test repointing ``_WAL_FILE`` re-initialises), so a persistent failure on a
restricted filesystem does not retry on every write. ``mkdir`` runs only
when the initial ``chmod`` raises ``FileNotFoundError`` (EAFP). The parent
``~/.mempalace`` keeps its umask mode, like the other palace directories;
the WAL file is created atomically with mode 0o600 by ``_wal_log``.
"""
global _WAL_INITIALIZED_DIR
wal_dir = _WAL_FILE.parent
if _WAL_INITIALIZED_DIR == wal_dir:
return
try:
wal_dir.chmod(0o700)
except FileNotFoundError:
try:
wal_dir.mkdir(parents=True, exist_ok=True)
wal_dir.chmod(0o700)
except (OSError, NotImplementedError):
pass
except (OSError, NotImplementedError):
pass
# Cache regardless of outcome: one attempt per directory, so a persistent
# chmod/mkdir failure (restricted FS) is not retried on every write.
_WAL_INITIALIZED_DIR = wal_dir
def _wal_log(operation: str, params: dict, result: dict = None):
"""Append a write operation to the write-ahead log."""
# Redact sensitive content from params before logging
safe_params = {}
for k, v in params.items():
if k in _WAL_REDACT_KEYS:
safe_params[k] = f"[REDACTED {len(v)} chars]" if isinstance(v, str) else "[REDACTED]"
else:
safe_params[k] = v
entry = {
"timestamp": datetime.now().isoformat(),
"operation": operation,
"params": safe_params,
"result": result,
}
try:
# Dir setup shares the append's exception handler below: any WAL
# failure is logged and non-fatal, never crashing the tool call.
_ensure_wal()
fd = os.open(str(_WAL_FILE), os.O_WRONLY | os.O_APPEND | os.O_CREAT, 0o600)
with os.fdopen(fd, "a", encoding="utf-8") as f:
f.write(json.dumps(entry, default=str) + "\n")
except Exception as e:
logger.error(f"WAL write failed: {e}")
+20 -3
View File
@@ -1,6 +1,6 @@
[project]
name = "mempalace"
version = "3.4.0"
version = "3.5.0"
description = "Give your AI a memory — mine projects and conversations into a searchable palace. No API key required."
readme = "README.md"
requires-python = ">=3.9"
@@ -74,7 +74,11 @@ sqlite_exact = "mempalace.backends.sqlite_exact:SQLiteExactBackend"
dev = [
"pytest>=7.0",
"pytest-cov>=4.0",
"ruff==0.15.15",
# Retries known-transient ChromaDB-on-Windows HNSW compaction failures in CI
# (wired only into the test-windows job in ci.yml, scoped to the specific
# error). Local/Linux/macOS runs never rerun, so real failures stay loud.
"pytest-rerunfailures>=12.0",
"ruff==0.15.18",
"psutil>=5.9",
# Property-based testing — generates hundreds of random inputs per
# test to find counterexamples the hand-written positive tests miss.
@@ -131,7 +135,11 @@ extract = [
dev = [
"pytest>=7.0",
"pytest-cov>=4.0",
"ruff==0.15.15",
# Retries known-transient ChromaDB-on-Windows HNSW compaction failures in CI
# (wired only into the test-windows job in ci.yml, scoped to the specific
# error). Local/Linux/macOS runs never rerun, so real failures stay loud.
"pytest-rerunfailures>=12.0",
"ruff==0.15.18",
"psutil>=5.9",
"hypothesis>=6.0",
"pre-commit>=3.0",
@@ -169,6 +177,15 @@ markers = [
"slow: tests that take more than 30 seconds",
"stress: destructive scale tests (100K+ drawers)",
]
filterwarnings = [
# Many tests build raw-chromadb palaces directly (no recorded embedder
# identity), which correctly emits this on open. The behavior itself is
# asserted in tests/test_embedder_identity.py; ignore the fixture noise.
# Matched by message (not by class path) so pytest does not import the
# mempalace package at config time — importing it before coverage starts
# would drop module-level lines from the report.
"ignore:palace collection.*has no recorded embedder identity",
]
[tool.coverage.run]
source = ["mempalace"]
+23
View File
@@ -0,0 +1,23 @@
---
description: When the user asks about past work, prior decisions, people, projects, or events that may be filed in MemPalace, call mempalace_search (or mempalace_kg_query for relational or time-bound facts) before answering from model memory. Return stored content verbatim; never guess when the palace might know.
alwaysApply: false
---
# MemPalace recall
Before answering anything that may already be in the user's memory
palace — past work, prior decisions, a person, a project, or "what did
we do / decide / discuss last time?" — search the palace first:
1. Call `mempalace_search` with a short keyword query. Use
`mempalace_kg_query` for relational or time-bound facts.
2. Quote the drawer's **verbatim** text. Never summarize or paraphrase
stored content.
3. If results are empty, say so — do not invent an answer. If the MCP
server is unavailable, surface the error; do not fall back to guessing.
Skip recall for pure greenfield work with no memory relevance (renaming
a variable, fixing a typo). Recall is question-driven, not reflexive.
Full protocol: `integrations/shared/recall-protocol.md`. Deeper guidance:
the `mempalace-recall` skill.
+126
View File
@@ -0,0 +1,126 @@
---
name: mempalace-recall
description: "Recall protocol for MemPalace — search the palace before answering about past work, people, projects, or prior decisions. Apply when the user asks what was decided, what happened before, who someone is, what was discussed last time, or anything that may already be filed in their memory palace; or when mempalace-recall is invoked. Complements the mempalace setup skill and requires the mempalace-mcp server."
---
# MemPalace Recall
Search-before-answer protocol for MemPalace. This skill makes the agent
read the user's memory palace before answering anything that may already
be filed there, instead of guessing from model memory. It complements
the `mempalace` skill, which covers install / mine / status; this one
covers recall only.
## Step 0 — Verify MemPalace is available
Before relying on recall, confirm MemPalace is installed and reachable:
- Official release page: <https://github.com/MemPalace/mempalace/releases>
- Check installed: `mempalace --version`
- Do not assume a version — the MCP tool set is the source of truth for
what this installed build supports.
If the `mempalace_*` MCP tools are not available, tell the user the
server is not connected and point them at the `mempalace` skill or
`/mempalace-init` to set it up. Do not silently fall back to answering
from model memory.
## Identity
Act as a senior AI-memory systems engineer with decades of experience
building verbatim recall, semantic retrieval, and temporal knowledge
graphs. Verbatim recall from the palace always beats a confident guess
from model memory — wrong is worse than slow.
## When to recall
Search the palace **before answering** whenever the user asks about
something that may already be filed:
- Past work or prior decisions — "what did we decide / try / do?"
- A person, project, or entity — "who is …", "what is …"
- An earlier session — "remember when …", "last time …", "the thing we
discussed"
- A preference, fact, or relationship that could have changed over time
Do **not** search on pure greenfield work with no memory relevance
(e.g. "rename this variable", "fix this typo"). Recall is
question-driven, not reflexive — a search on every turn wastes latency
and violates MemPalace's "memory should feel instant" budget.
## Protocol
1. On wake-up, if a session-start hook injected `additional_context`,
honour its wing scoping.
2. Before responding about people / projects / past events / prior
decisions: call `mempalace_search` first. Use `mempalace_kg_query`
for relational or time-bound facts.
3. If unsure about a fact: say "let me check the palace" and query.
4. Return the drawer's **verbatim** text. Never summarize or paraphrase
stored content — quoting the exact words is the point of the system.
5. After a substantive session, record continuity with
`mempalace_diary_write` (skip if a background hook already saved).
6. When a fact changes: `mempalace_kg_invalidate` the old fact, then
`mempalace_kg_add` the new one.
The full canonical protocol — shared verbatim with the Cursor recall
rule and the other integrations — lives in
[`integrations/shared/recall-protocol.md`](../../integrations/shared/recall-protocol.md).
## Tool selection
| You need | Tool |
|---|---|
| Find any memory by meaning | `mempalace_search` (start here) |
| Relational / time-bound facts about an entity | `mempalace_kg_query` |
| The chronological story of an entity | `mempalace_kg_timeline` |
| Recent session continuity | `mempalace_diary_read` |
| Which wings / rooms exist (scope unknown) | `mempalace_list_wings`, `mempalace_list_rooms` |
| Record this session | `mempalace_diary_write` |
`mempalace_search` takes a short natural-language `query` (keywords or a
question — not a system prompt or pasted conversation) plus optional
`wing` / `room` filters and `limit` (default 5).
## Unhappy paths
- **Empty results.** Say the palace has nothing on this; do not invent an
answer. Offer to widen the search (drop the `wing` filter) or to file
the new information.
- **MCP error / server down.** Surface the error and suggest the user
run `mempalace status` or re-run `/mempalace-init`. Never fall back to
guessing.
- **Palace index corrupt / compactor error.** If the server reports an
HNSW segment-writer error, a ChromaDB compaction failure, or stays
"Not connected" after a write, the vector index is out of sync with
`chroma.sqlite3` while the drawer rows remain intact. Tell the user to
stop the server and rebuild from SQLite — do not re-mine, which drops
MCP-added drawers and diary entries (#1843):
```bash
mempalace repair --mode from-sqlite --archive-existing --yes
mempalace repair-status
```
Do not attempt an in-process repair from the agent. Full steps are in
the shared protocol's "Recovering a corrupt index" section.
- **Conflicting facts.** Trust the knowledge graph's time-valid answer;
invalidate-then-add rather than overwriting silently.
## Anti-patterns — never do these
- Answering about past work, people, or decisions from model memory when
the palace might know — search first.
- Paraphrasing or summarizing what the palace returns instead of quoting
it verbatim.
- Searching on every turn, including greenfield tasks with no memory
relevance.
- Pasting the whole conversation or a system prompt into the `query`
argument — keep queries short and keyword-driven.
## Official References
- MemPalace: <https://github.com/MemPalace/mempalace>
- MemPalace releases: <https://github.com/MemPalace/mempalace/releases>
- Cursor Skills documentation: <https://cursor.com/docs/skills>
- Agent Skills specification: <https://agentskills.io/specification>
+47
View File
@@ -0,0 +1,47 @@
---
name: mempalace
description: MemPalace — mine projects and conversations into a searchable memory palace. Use when the user asks about MemPalace, memory palace, mining memories, searching memories, palace setup, wings, rooms, or drawers; or when they want to recall past work that may already be filed in their palace.
---
# MemPalace
A searchable memory palace for AI — mine projects and conversations, then search them semantically.
## Prerequisites
Ensure `mempalace` is installed:
```bash
mempalace --version
```
If not installed (uv recommended):
```bash
uv tool install mempalace # or: pip install mempalace
```
## Usage
MemPalace provides dynamic, version-correct instructions via the CLI. To get instructions for any operation:
```bash
mempalace instructions <command>
```
Where `<command>` is one of: `help`, `init`, `mine`, `search`, `status`.
Run the appropriate instructions command, then follow the returned instructions step by step.
## Recalling past work
This skill covers setup, mining, and status. For questions about past
work, prior decisions, or people that may already be filed in the
palace, prefer the **`mempalace-recall`** skill — it enforces
search-before-answer so the agent reads the palace instead of guessing.
## Cursor-specific notes
- The `mempalace-mcp` server is auto-registered by this plugin. Once installed, all 34 MemPalace MCP tools (`mempalace_search`, `mempalace_add_drawer`, `mempalace_diary_write`, `mempalace_check_duplicate`, `mempalace_diary_read`, etc.) are available to the agent without any further configuration.
- For automatic background saving every N agent turns plus session-start memory recall, also install the Cursor hooks separately by running `hooks/cursor/install.sh --scope user` from a cloned MemPalace repo. See [`website/guide/cursor-hooks.md`](../../website/guide/cursor-hooks.md) for the full walkthrough.
- The recommended `agent_name` when calling `mempalace_diary_write` from a Cursor session is `cursor-ide` (matches the precedent of `claude-code` and `codex`).
+324
View File
@@ -0,0 +1,324 @@
"""End-to-end tests for the Antigravity install.sh.
Covers:
* `--dry-run` is fully side-effect free.
* A real install creates the expected file tree.
* `hooks.json` is rendered with absolute paths (no `__PLUGIN_DIR__` leak).
* Re-running the installer is byte-identical (cmp gate works).
* `--uninstall` removes the dir cleanly.
* `--uninstall` refuses to wipe a directory whose basename isn't `mempalace`.
* `--uninstall` refuses if the dir is missing a `mempalace` plugin.json.
* Relative `--install-dir` is absolutized into the rendered hooks.json.
"""
from __future__ import annotations
import filecmp
import json
import os
import subprocess
from pathlib import Path
import pytest
REPO_ROOT = Path(__file__).resolve().parents[1]
INSTALL_SH = REPO_ROOT / "hooks" / "antigravity" / "install.sh"
# Skip on Windows — install.sh is bash and uses POSIX path semantics.
pytestmark = pytest.mark.skipif(
os.name == "nt",
reason="install.sh is a bash script; Windows users use a separate code path.",
)
EXPECTED_FILES = (
"plugin.json",
"mcp_config.json",
"README.md",
"hooks.json",
"skills/mempalace/SKILL.md",
"hooks/lib/common.sh",
"hooks/mempal_save_hook_antigravity.sh",
"hooks/mempal_wake_hook_antigravity.sh",
)
def _run_install(
install_dir: Path,
*args: str,
cwd: Path | None = None,
timeout: float = 30.0,
) -> subprocess.CompletedProcess:
"""Invoke install.sh with the given install dir and args."""
cmd = [
"bash",
str(INSTALL_SH),
"--install-dir",
str(install_dir),
*args,
]
return subprocess.run(
cmd,
capture_output=True,
text=True,
cwd=str(cwd) if cwd else None,
timeout=timeout,
)
def _assert_install_layout(install_dir: Path) -> None:
for rel in EXPECTED_FILES:
path = install_dir / rel
assert path.is_file(), f"missing after install: {rel}"
assert not path.is_symlink(), f"unexpected symlink at: {rel}"
# ── --dry-run ──────────────────────────────────────────────────────────
def test_dry_run_is_side_effect_free(tmp_path: Path) -> None:
"""--dry-run must not create the install dir or any of its files."""
install_dir = tmp_path / "mempalace"
result = _run_install(install_dir, "--dry-run")
assert result.returncode == 0, result.stderr
assert not install_dir.exists(), (
f"dry-run created install dir {install_dir} — that is a side effect."
)
# Output should mention DRY-RUN at least once for visibility.
assert "DRY-RUN" in result.stdout, result.stdout
# ── Real install ──────────────────────────────────────────────────────
def test_real_install_creates_full_layout(tmp_path: Path) -> None:
install_dir = tmp_path / "mempalace"
result = _run_install(install_dir)
assert result.returncode == 0, f"install failed:\n{result.stdout}\n{result.stderr}"
_assert_install_layout(install_dir)
def test_install_renders_absolute_paths_in_hooks_json(tmp_path: Path) -> None:
"""`__PLUGIN_DIR__` must be substituted into hooks.json command paths."""
install_dir = tmp_path / "mempalace"
result = _run_install(install_dir)
assert result.returncode == 0
hooks = json.loads((install_dir / "hooks.json").read_text())
# Every "command" string must be absolute and live under the
# install dir.
cmds = []
for ns_payload in hooks.values():
if not isinstance(ns_payload, dict):
continue
for entries in ns_payload.values():
for entry in entries:
cmds.append(entry["command"])
assert cmds, f"no command entries found in rendered hooks.json: {hooks}"
install_str = str(install_dir)
for cmd in cmds:
assert "__PLUGIN_DIR__" not in cmd, f"placeholder leaked into rendered hooks.json: {cmd!r}"
assert cmd.startswith("/"), f"command path is not absolute: {cmd!r}"
assert cmd.startswith(install_str + "/"), (
f"command path {cmd!r} does not live under install dir {install_str!r}"
)
def test_install_executable_bits_preserved(tmp_path: Path) -> None:
"""Both hook scripts must end up executable on the install side."""
install_dir = tmp_path / "mempalace"
result = _run_install(install_dir)
assert result.returncode == 0
for rel in (
"hooks/mempal_save_hook_antigravity.sh",
"hooks/mempal_wake_hook_antigravity.sh",
):
path = install_dir / rel
assert os.access(path, os.X_OK), f"hook script not executable: {rel}"
# ── Idempotency ───────────────────────────────────────────────────────
def test_install_is_byte_identical_on_re_run(tmp_path: Path) -> None:
"""Re-running the installer should leave every file byte-identical.
The `cmp`-gated copy and template render is what makes the
installer safe to run from CI and from `babysit`-style cron loops.
"""
install_dir = tmp_path / "mempalace"
first = _run_install(install_dir)
assert first.returncode == 0, first.stderr
# Snapshot every file's content + mtime + mode.
snap1: dict[str, tuple[bytes, float, int]] = {}
for rel in EXPECTED_FILES:
p = install_dir / rel
st = p.stat()
snap1[rel] = (p.read_bytes(), st.st_mtime, st.st_mode)
# Re-run.
second = _run_install(install_dir)
assert second.returncode == 0, second.stderr
# Every file's contents must be byte-identical.
for rel in EXPECTED_FILES:
p = install_dir / rel
body, _, mode = snap1[rel]
assert p.read_bytes() == body, f"{rel} differs after re-install"
# Mode must be preserved (we don't enforce mtime since the
# cmp gate explicitly avoids re-writing).
assert p.stat().st_mode == mode, f"{rel} mode changed after re-install"
# Use filecmp.dircmp as a belt-and-suspenders check.
cmp = filecmp.dircmp(install_dir, install_dir)
assert not cmp.diff_files
def test_install_logs_no_writes_on_idempotent_re_run(tmp_path: Path) -> None:
"""The second run must not log "wrote: ..." for any file."""
install_dir = tmp_path / "mempalace"
first = _run_install(install_dir)
assert first.returncode == 0
# First run writes everything.
assert "wrote:" in first.stdout
second = _run_install(install_dir)
assert second.returncode == 0
assert "wrote:" not in second.stdout, (
f"second install should be a no-op but wrote files:\n{second.stdout}"
)
# ── Uninstall ─────────────────────────────────────────────────────────
def test_uninstall_removes_mempalace_install(tmp_path: Path) -> None:
install_dir = tmp_path / "mempalace"
install = _run_install(install_dir)
assert install.returncode == 0
uninstall = _run_install(install_dir, "--uninstall")
assert uninstall.returncode == 0, uninstall.stderr
assert not install_dir.exists()
def test_uninstall_refuses_basename_mismatch(tmp_path: Path) -> None:
"""Refuses to remove a directory whose basename isn't 'mempalace'.
Honours the basename-match safety guard caught in the cursor PR
review — prevents an accidental wipe of a sibling like
'mempalace-foo' or, in the worst case, the user's home directory.
"""
bad_dir = tmp_path / "totally-not-mempalace"
bad_dir.mkdir()
# Even though the dir has a plugin.json with name=mempalace, the
# basename mismatch must still refuse.
(bad_dir / "plugin.json").write_text(json.dumps({"name": "mempalace"}), encoding="utf-8")
sentinel = bad_dir / "do-not-delete.txt"
sentinel.write_text("preserve me", encoding="utf-8")
result = _run_install(bad_dir, "--uninstall")
assert result.returncode != 0, (
f"uninstall should have refused but exited 0:\n{result.stdout}\n{result.stderr}"
)
assert bad_dir.is_dir(), f"bad uninstall removed {bad_dir}"
assert sentinel.is_file(), "uninstall removed unrelated files inside bad dir"
def test_uninstall_refuses_when_plugin_json_missing(tmp_path: Path) -> None:
"""Refuses when the dir is missing plugin.json (not actually our plugin)."""
install_dir = tmp_path / "mempalace"
install_dir.mkdir()
sentinel = install_dir / "stranger.txt"
sentinel.write_text("preserve me", encoding="utf-8")
result = _run_install(install_dir, "--uninstall")
assert result.returncode != 0
assert install_dir.is_dir()
assert sentinel.is_file()
def test_uninstall_refuses_when_plugin_json_wrong_name(tmp_path: Path) -> None:
"""Refuses when plugin.json names a different plugin."""
install_dir = tmp_path / "mempalace"
install_dir.mkdir()
(install_dir / "plugin.json").write_text(
json.dumps({"name": "some-other-plugin"}), encoding="utf-8"
)
sentinel = install_dir / "preserved.txt"
sentinel.write_text("safe", encoding="utf-8")
result = _run_install(install_dir, "--uninstall")
assert result.returncode != 0
assert install_dir.is_dir()
assert sentinel.is_file()
def test_uninstall_no_op_when_target_missing(tmp_path: Path) -> None:
"""Uninstalling a non-existent dir is a graceful no-op."""
install_dir = tmp_path / "mempalace"
result = _run_install(install_dir, "--uninstall")
assert result.returncode == 0, result.stderr
# ── Relative path absolutization ──────────────────────────────────────
def test_relative_install_dir_is_absolutized(tmp_path: Path) -> None:
"""A relative --install-dir must be absolutized in the rendered hooks.json.
Catches the cursor PR review issue where a relative path baked
in verbatim left Antigravity unable to resolve hook commands.
"""
work = tmp_path / "work"
work.mkdir()
# Relative path resolved against $PWD at invocation time.
rel = "build/agy-out/mempalace"
result = _run_install(Path(rel), cwd=work)
assert result.returncode == 0, result.stderr
abs_install = work / rel
assert abs_install.is_dir(), f"installer did not create {abs_install} from relative path {rel}"
hooks = json.loads((abs_install / "hooks.json").read_text())
cmds = []
for ns_payload in hooks.values():
if not isinstance(ns_payload, dict):
continue
for entries in ns_payload.values():
for entry in entries:
cmds.append(entry["command"])
for cmd in cmds:
assert cmd.startswith("/"), f"relative install dir leaked into rendered hooks.json: {cmd!r}"
assert "build/agy-out/mempalace" in cmd, (
f"command path lost the relative-segment context: {cmd!r}"
)
# ── Misc ──────────────────────────────────────────────────────────────
def test_install_help_does_not_write(tmp_path: Path) -> None:
"""`--help` should print usage and exit 0 without touching the dir."""
install_dir = tmp_path / "mempalace"
result = subprocess.run(
["bash", str(INSTALL_SH), "--help"],
capture_output=True,
text=True,
timeout=10,
)
assert result.returncode == 0
assert "Usage:" in result.stdout or "Usage:" in result.stderr
assert not install_dir.exists()
def test_install_unknown_arg_exits_non_zero(tmp_path: Path) -> None:
"""Unknown args must fail loudly rather than silently ignoring."""
install_dir = tmp_path / "mempalace"
result = subprocess.run(
[
"bash",
str(INSTALL_SH),
"--install-dir",
str(install_dir),
"--this-flag-does-not-exist",
],
capture_output=True,
text=True,
timeout=10,
)
assert result.returncode != 0
assert not install_dir.exists()
File diff suppressed because it is too large Load Diff
+358
View File
@@ -0,0 +1,358 @@
"""Schema tests for the .antigravity-plugin/ directory.
Covers:
* `plugin.json` matches the verified-minimal Antigravity schema
(`{"name": "..."}`, no fabricated fields).
* `mcp_config.json` registers `mempalace-mcp` under the `mcpServers`
key with the verified shape from
https://antigravity.google/docs/mcp.
* `hooks.json.tmpl` is valid JSON, references both hook scripts via
the `__PLUGIN_DIR__` placeholder, and pins per-event timeouts
inside the safety bounds.
* `skills/mempalace/SKILL.md` exists as a real file (no symlinks) and
carries the required YAML frontmatter (`description`).
These are contract tests — they fail as soon as anyone changes the
in-repo shape in a way that drifts from Antigravity's documented
schema. See [hooks/antigravity/INVESTIGATION.md](../hooks/antigravity/INVESTIGATION.md)
for the source-of-truth audit driving the assertions.
"""
import json
import re
from pathlib import Path
import pytest
REPO_ROOT = Path(__file__).resolve().parents[1]
PLUGIN_DIR = REPO_ROOT / ".antigravity-plugin"
PLUGIN_JSON = PLUGIN_DIR / "plugin.json"
MCP_CONFIG = PLUGIN_DIR / "mcp_config.json"
HOOKS_TMPL = PLUGIN_DIR / "hooks.json.tmpl"
SKILL_MD = PLUGIN_DIR / "skills" / "mempalace" / "SKILL.md"
PLUGIN_README = PLUGIN_DIR / "README.md"
# Recall layer (mirrors the Cursor branch's recall skill + rule + shared protocol)
RECALL_SKILL_MD = PLUGIN_DIR / "skills" / "mempalace-recall" / "SKILL.md"
RECALL_RULE_MD = PLUGIN_DIR / "rules" / "mempalace-recall.md"
SHARED_PROTOCOL = REPO_ROOT / "integrations" / "shared" / "recall-protocol.md"
INSTALL_SH = REPO_ROOT / "hooks" / "antigravity" / "install.sh"
SHARED_PROTOCOL_REF = (
"https://github.com/MemPalace/mempalace/blob/main/integrations/shared/recall-protocol.md"
)
EXPECTED_HOOKS = {
"Stop": {
"script_basename": "mempal_save_hook_antigravity.sh",
"timeout_floor": 10,
"timeout_ceiling": 60,
},
"PreInvocation": {
"script_basename": "mempal_wake_hook_antigravity.sh",
"timeout_floor": 1,
"timeout_ceiling": 10,
},
}
def test_plugin_dir_exists() -> None:
"""The in-repo plugin directory exists and is laid out as expected."""
assert PLUGIN_DIR.is_dir(), f"missing: {PLUGIN_DIR}"
for required in (PLUGIN_JSON, MCP_CONFIG, HOOKS_TMPL, SKILL_MD, PLUGIN_README):
assert required.is_file(), f"missing: {required}"
def test_plugin_json_minimal_schema() -> None:
"""plugin.json must be `{"name": "mempalace"}` exactly — no fabricated fields.
The third-party "antigravity-plugins" community skill at
~/.gemini/skills/antigravity-plugins/SKILL.md documents a
`permissions` field that does not exist in any real
Google-shipped plugin. We pin to the verified minimal shape and
fail loudly if anyone re-introduces the fabrication.
"""
data = json.loads(PLUGIN_JSON.read_text(encoding="utf-8"))
assert isinstance(data, dict), "plugin.json must be a JSON object"
assert data == {"name": "mempalace"}, (
f"plugin.json must equal {{'name': 'mempalace'}} (verified shape); "
f"got {data!r}. The `permissions` field documented in the third-party "
"antigravity-plugins community skill is fabricated; do not add it."
)
def test_mcp_config_registers_mempalace_mcp() -> None:
"""mcp_config.json must register the mempalace stdio server."""
data = json.loads(MCP_CONFIG.read_text(encoding="utf-8"))
assert isinstance(data, dict)
assert "mcpServers" in data, "missing top-level mcpServers key"
servers = data["mcpServers"]
assert isinstance(servers, dict)
assert "mempalace" in servers, "mcpServers.mempalace not registered"
entry = servers["mempalace"]
assert isinstance(entry, dict)
assert entry.get("command") == "mempalace-mcp", (
f"mcpServers.mempalace.command must be 'mempalace-mcp'; got {entry.get('command')!r}"
)
def test_hooks_template_valid_json() -> None:
"""hooks.json.tmpl must be valid JSON (the `__PLUGIN_DIR__` placeholder is JSON-safe)."""
body = HOOKS_TMPL.read_text(encoding="utf-8")
try:
data = json.loads(body)
except json.JSONDecodeError as exc:
pytest.fail(f"hooks.json.tmpl is not valid JSON: {exc}")
assert isinstance(data, dict)
def test_hooks_template_uses_plugin_dir_placeholder() -> None:
"""hooks.json.tmpl must use __PLUGIN_DIR__ — never bake an absolute path."""
body = HOOKS_TMPL.read_text(encoding="utf-8")
assert "__PLUGIN_DIR__" in body, (
"hooks.json.tmpl must use __PLUGIN_DIR__ as the install-dir placeholder. "
"Hard-coded absolute paths break the installer's idempotency promise."
)
# Any `/Users/`, `/home/`, or `~/` segment in the template body is a sign
# that an absolute path leaked in.
forbidden = ["/Users/", "/home/", "~/"]
for prefix in forbidden:
assert prefix not in body, (
f"hooks.json.tmpl must not contain a hard-coded path segment {prefix!r}; "
"use the __PLUGIN_DIR__ placeholder instead."
)
@pytest.mark.parametrize("event", sorted(EXPECTED_HOOKS))
def test_hooks_template_event_present(event: str) -> None:
"""Each expected event has exactly one entry pointing at the right script with bounded timeout."""
data = json.loads(HOOKS_TMPL.read_text(encoding="utf-8"))
bounds = EXPECTED_HOOKS[event]
# Outer keys are hook namespace names, e.g. "mempalace-save".
matching = [
(ns, payload[event])
for ns, payload in data.items()
if isinstance(payload, dict) and event in payload
]
assert len(matching) == 1, (
f"expected exactly one hook namespace declaring event {event!r}; "
f"found {len(matching)}: {[m[0] for m in matching]}"
)
_, entries = matching[0]
assert isinstance(entries, list)
assert len(entries) == 1, (
f"{event}: expected exactly one handler entry, got {len(entries)}; "
"duplicate entries would double-fire the hook"
)
handler = entries[0]
assert handler.get("type", "command") == "command", (
f"{event}: only type=command is supported by Antigravity"
)
cmd = handler.get("command", "")
assert cmd.startswith("__PLUGIN_DIR__/"), (
f"{event}: command must be rooted at __PLUGIN_DIR__/, got {cmd!r}"
)
assert cmd.endswith("/" + bounds["script_basename"]), (
f"{event}: command must end with the expected script basename "
f"{bounds['script_basename']!r}; got {cmd!r}"
)
timeout = handler.get("timeout")
is_int = isinstance(timeout, int) and not isinstance(timeout, bool)
assert is_int and bounds["timeout_floor"] <= timeout <= bounds["timeout_ceiling"], (
f"{event}: timeout must be an int in "
f"[{bounds['timeout_floor']}, {bounds['timeout_ceiling']}]s; got {timeout!r}"
)
def test_skill_is_real_file_not_symlink() -> None:
"""SKILL.md at the discovery path must be a real file.
Antigravity (like Cursor) loads skills by reading
`<plugin>/skills/<name>/SKILL.md` directly. A symlink at that path
would work locally but break under any installer that does a
plain `cp`. Honouring constraint #6 in the integration brief.
"""
assert SKILL_MD.is_file(), f"missing: {SKILL_MD}"
assert not SKILL_MD.is_symlink(), (
f"{SKILL_MD} must be a real file, not a symlink — installers that "
"cp without -L would otherwise carry the symlink into the install."
)
def test_skill_has_required_frontmatter() -> None:
"""SKILL.md must carry YAML frontmatter with a non-empty description.
Antigravity's skill loader uses the `description` field to decide
when to surface the skill. An empty / missing description would
silently disable progressive disclosure.
"""
body = SKILL_MD.read_text(encoding="utf-8")
assert body.startswith("---\n"), "SKILL.md must begin with YAML frontmatter"
end = body.find("\n---\n", 4)
assert end > 0, "SKILL.md frontmatter is missing the closing fence"
front = body[4:end]
desc_match = re.search(r"^description:\s*(.+)$", front, re.MULTILINE)
assert desc_match is not None, "SKILL.md frontmatter missing `description` key"
desc_value = desc_match.group(1).strip()
assert desc_value, "SKILL.md `description` is empty"
# Sanity: the description should be substantive enough for the
# skill loader to act on. 30 chars is a soft floor, not a tight bound.
assert len(desc_value) >= 30, (
f"SKILL.md description looks too short to be useful: {desc_value!r}"
)
def test_no_symlinks_inside_plugin_dir() -> None:
"""Nothing inside .antigravity-plugin/ may be a symlink.
This is the broader version of `test_skill_is_real_file_not_symlink`
and a guard against silent regressions if someone re-introduces
the `skills -> ../skills` symlink pattern from the original plan
without honouring `cp -RL` semantics in the installer.
"""
leaks = [p for p in PLUGIN_DIR.rglob("*") if p.is_symlink()]
assert not leaks, (
f"symlinks found inside .antigravity-plugin/: {[str(p.relative_to(PLUGIN_DIR)) for p in leaks]}; "
"the entire plugin tree must be made of real files so any installer "
"(including those that cp without -L) gets a working install."
)
def test_plugin_readme_present_and_substantive() -> None:
"""README.md inside the plugin dir must exist and be substantive.
Empty / placeholder READMEs are a frequent symptom of half-finished
refactors; a 200-byte floor catches those without being so tight
it discourages legitimate rewrites.
"""
body = PLUGIN_README.read_text(encoding="utf-8")
assert len(body) >= 200, (
f".antigravity-plugin/README.md looks too short ({len(body)} bytes); "
"expected a substantive description of layout + install."
)
# Must mention key concepts so the README can't degrade into prose
# that drops the operational links.
for needle in ("plugin.json", "mcp_config.json", "hooks.json"):
assert needle in body, f"README.md must mention {needle}"
# ── Recall layer: skill, rule, shared protocol ───────────────────────
#
# Mirrors the three-layer recall wiring added for Cursor on
# feat/cursor-hooks-support, adapted for Antigravity's plugin surface.
# The wake hook (PreInvocation) is the eager layer; these files are the
# on-demand layers (skill + optional rule), both anchored to the single
# canonical protocol so they can never drift.
def test_shared_protocol_exists() -> None:
"""The canonical recall protocol is the single source of truth.
The recall skill and rule both link here rather than restating the
protocol, so the rule can never drift from the skill.
"""
assert SHARED_PROTOCOL.is_file(), f"missing: {SHARED_PROTOCOL}"
body = SHARED_PROTOCOL.read_text(encoding="utf-8")
assert "MemPalace Recall Protocol" in body, "shared protocol must carry its canonical title"
def test_recall_skill_exists() -> None:
"""The recall-only skill must be a real file at the discovery path."""
assert RECALL_SKILL_MD.is_file(), f"missing: {RECALL_SKILL_MD}"
assert not RECALL_SKILL_MD.is_symlink(), (
f"{RECALL_SKILL_MD} must be a real file, not a symlink — installers "
"that cp without -L would otherwise carry the symlink into the install."
)
def test_recall_skill_has_required_frontmatter() -> None:
"""The recall skill must carry YAML frontmatter with a non-empty description.
Antigravity's skill loader uses `description` for progressive
disclosure, exactly like the ops `mempalace` skill.
"""
body = RECALL_SKILL_MD.read_text(encoding="utf-8")
assert body.startswith("---\n"), "recall SKILL.md must begin with YAML frontmatter"
end = body.find("\n---\n", 4)
assert end > 0, "recall SKILL.md frontmatter is missing the closing fence"
front = body[4:end]
desc_match = re.search(r"^description:\s*(.+)$", front, re.MULTILINE)
assert desc_match is not None, "recall SKILL.md frontmatter missing `description` key"
assert desc_match.group(1).strip(), "recall SKILL.md `description` is empty"
def test_recall_skill_has_required_sections() -> None:
"""The recall skill must carry the load-bearing protocol sections."""
body = RECALL_SKILL_MD.read_text(encoding="utf-8")
for section in (
"When to recall",
"Protocol",
"Tool selection",
"Unhappy paths",
"Anti-patterns",
):
assert section in body, f"recall SKILL.md must contain a '{section}' section"
def test_recall_skill_links_to_shared_protocol() -> None:
"""The recall skill must defer to the canonical protocol, not restate it."""
body = RECALL_SKILL_MD.read_text(encoding="utf-8")
assert SHARED_PROTOCOL_REF in body, (
f"recall SKILL.md must reference {SHARED_PROTOCOL_REF} so the protocol stays single-sourced"
)
def test_recall_rule_exists() -> None:
"""The optional recall rule must be a real file under the plugin rules dir."""
assert RECALL_RULE_MD.is_file(), f"missing: {RECALL_RULE_MD}"
assert not RECALL_RULE_MD.is_symlink(), f"{RECALL_RULE_MD} must be a real file"
def test_recall_rule_is_plain_markdown_not_mdc() -> None:
"""Antigravity rules are plain `.md` with no YAML frontmatter.
Per https://antigravity.google/docs plugins use `rules/<name>.md`
(no `.mdc`, no Cursor-style `alwaysApply` frontmatter). Pin the
plain-markdown shape so nobody copies the Cursor `.mdc` verbatim.
"""
assert RECALL_RULE_MD.suffix == ".md", "Antigravity rule must use the .md extension"
assert not (RECALL_RULE_MD.parent / "mempalace-recall.mdc").exists(), (
"an .mdc rule leaked in; Antigravity rules are plain .md"
)
body = RECALL_RULE_MD.read_text(encoding="utf-8")
assert not body.startswith("---"), (
"Antigravity rule must NOT carry YAML frontmatter (no Cursor-style "
"`alwaysApply`); it is plain markdown"
)
def test_recall_rule_references_shared_protocol() -> None:
"""The rule must point at the canonical protocol and the deeper skill."""
body = RECALL_RULE_MD.read_text(encoding="utf-8")
assert SHARED_PROTOCOL_REF in body, f"recall rule must reference {SHARED_PROTOCOL_REF}"
def test_installer_creates_rules_dir() -> None:
"""install.sh must create the skills/mempalace-recall and rules dirs."""
body = INSTALL_SH.read_text(encoding="utf-8")
assert '"$INSTALL_DIR/rules"' in body, "install.sh mkdir block must create the rules/ directory"
assert '"$INSTALL_DIR/skills/mempalace-recall"' in body, (
"install.sh mkdir block must create the skills/mempalace-recall/ directory"
)
def test_installer_copies_recall_skill() -> None:
"""install.sh must copy the recall skill into the install dir."""
body = INSTALL_SH.read_text(encoding="utf-8")
assert "skills/mempalace-recall/SKILL.md" in body, (
"install.sh must copy_file the mempalace-recall skill"
)
def test_installer_copies_recall_rule() -> None:
"""install.sh must copy the recall rule into the install dir."""
body = INSTALL_SH.read_text(encoding="utf-8")
assert "rules/mempalace-recall.md" in body, (
"install.sh must copy_file the mempalace-recall rule"
)
+56
View File
@@ -768,6 +768,33 @@ def test_fix_blob_seq_ids_writes_marker_when_already_integer(tmp_path):
assert marker.is_file(), "marker must be written even when no BLOBs found"
def test_fix_blob_seq_ids_closes_sqlite_connection(tmp_path, monkeypatch):
"""The migration closes sqlite connections after the pre-open probe."""
db_path = tmp_path / "chroma.sqlite3"
with closing(sqlite3.connect(str(db_path))) as conn:
conn.execute("CREATE TABLE embeddings (rowid INTEGER PRIMARY KEY, seq_id INTEGER)")
conn.execute("INSERT INTO embeddings (seq_id) VALUES (42)")
conn.commit()
closed = []
real_connect = sqlite3.connect
class TrackingConnection(sqlite3.Connection):
def close(self):
closed.append(True)
super().close()
def tracking_connect(*args, **kwargs):
kwargs["factory"] = TrackingConnection
return real_connect(*args, **kwargs)
monkeypatch.setattr("mempalace.backends.chroma.sqlite3.connect", tracking_connect)
_fix_blob_seq_ids(str(tmp_path))
assert closed == [True]
def test_fix_blob_seq_ids_skips_sqlite_when_marker_present(tmp_path):
"""When the marker exists, ``_fix_blob_seq_ids`` does not open sqlite3.
@@ -1467,6 +1494,35 @@ def test_quarantine_invalid_hnsw_metadata_keeps_consistent_missing_dimensionalit
assert seg.exists()
def test_quarantine_invalid_hnsw_metadata_keeps_post_deletion_missing_dimensionality(tmp_path):
"""A deleted-from segment has total_elements_added > live label count (the
counter is monotonic); that dim-None shape is recoverable, not corruption (#1710).
"""
palace = tmp_path / "palace"
palace.mkdir()
seg = palace / "abcd-1234-5678"
seg.mkdir()
(seg / "data_level0.bin").write_bytes(b"x" * 2048)
(seg / "link_lists.bin").write_bytes(b"x" * 128)
with open(seg / "index_metadata.pickle", "wb") as f:
pickle.dump(
{
"dimensionality": None,
"total_elements_added": 5,
"max_seq_id": None,
"id_to_label": {"a": 1, "b": 2},
"label_to_id": {1: "a", 2: "b"},
"id_to_seq_id": {},
},
f,
)
moved = quarantine_invalid_hnsw_metadata(str(palace))
assert moved == []
assert seg.exists()
def test_quarantine_invalid_hnsw_metadata_renames_mismatched_missing_dimensionality(tmp_path):
palace = tmp_path / "palace"
palace.mkdir()
+157
View File
@@ -0,0 +1,157 @@
"""Tests for backup retention pruning (mempalace.backups.prune_backups).
These guard the fix for unbounded backup growth: ``mempalace migrate`` and
``mempalace repair max-seq-id`` each drop a fresh full-size, timestamped copy
every run, and used to never delete the old ones — a palace was found with
hundreds of GB of stale backups beside a few hundred MB of live data.
"""
import os
import pytest
from mempalace.backups import prune_backups
def _make_backup_dir(parent, name, mtime):
"""Create a directory backup with a fixed mtime."""
path = parent / name
path.mkdir()
(path / "chroma.sqlite3").write_text("db")
os.utime(path, (mtime, mtime))
return path
def _make_backup_file(parent, name, mtime):
"""Create a file backup with a fixed mtime."""
path = parent / name
path.write_text("db")
os.utime(path, (mtime, mtime))
return path
def test_prune_keeps_newest_and_removes_oldest(tmp_path):
# 5 backups, mtimes 100..500; keep 2 newest (400, 500).
paths = [_make_backup_file(tmp_path, f"b.{i}", mtime=i * 100) for i in range(1, 6)]
removed = prune_backups(str(tmp_path / "b.*"), max_backups=2)
surviving = {p.name for p in tmp_path.iterdir()}
assert surviving == {"b.4", "b.5"}
assert set(removed) == {str(paths[0]), str(paths[1]), str(paths[2])}
def test_prune_removes_directory_backups(tmp_path):
"""migrate writes directory backups (full copytree) — must rmtree them."""
_make_backup_dir(tmp_path, "palace.pre-migrate.1", mtime=100)
_make_backup_dir(tmp_path, "palace.pre-migrate.2", mtime=200)
keep = _make_backup_dir(tmp_path, "palace.pre-migrate.3", mtime=300)
removed = prune_backups(str(tmp_path / "palace.pre-migrate.*"), max_backups=1)
assert keep.is_dir()
assert len(removed) == 2
assert not (tmp_path / "palace.pre-migrate.1").exists()
assert not (tmp_path / "palace.pre-migrate.2").exists()
def test_prune_noop_when_under_limit(tmp_path):
_make_backup_file(tmp_path, "b.1", mtime=100)
_make_backup_file(tmp_path, "b.2", mtime=200)
removed = prune_backups(str(tmp_path / "b.*"), max_backups=10)
assert removed == []
assert len(list(tmp_path.iterdir())) == 2
def test_prune_noop_when_exactly_at_limit(tmp_path):
_make_backup_file(tmp_path, "b.1", mtime=100)
_make_backup_file(tmp_path, "b.2", mtime=200)
removed = prune_backups(str(tmp_path / "b.*"), max_backups=2)
assert removed == []
@pytest.mark.parametrize("disabled", [0, -1, None])
def test_prune_disabled_keeps_everything(tmp_path, disabled):
for i in range(1, 6):
_make_backup_file(tmp_path, f"b.{i}", mtime=i * 100)
removed = prune_backups(str(tmp_path / "b.*"), max_backups=disabled)
assert removed == []
assert len(list(tmp_path.iterdir())) == 5
def test_prune_no_matches(tmp_path):
assert prune_backups(str(tmp_path / "nope.*"), max_backups=3) == []
def test_prune_only_touches_matching_pattern(tmp_path):
"""Live data and unrelated files must never be swept up by a backup glob."""
_make_backup_file(tmp_path, "chroma.sqlite3.max-seq-id-backup-1", mtime=100)
_make_backup_file(tmp_path, "chroma.sqlite3.max-seq-id-backup-2", mtime=200)
_make_backup_file(tmp_path, "chroma.sqlite3.max-seq-id-backup-3", mtime=300)
# The live database and an unrelated file — must survive.
live = _make_backup_file(tmp_path, "chroma.sqlite3", mtime=400)
other = _make_backup_file(tmp_path, "tunnels.json", mtime=400)
prune_backups(
str(tmp_path / "chroma.sqlite3.max-seq-id-backup-*"),
max_backups=1,
)
assert live.exists()
assert other.exists()
assert (tmp_path / "chroma.sqlite3.max-seq-id-backup-3").exists()
assert not (tmp_path / "chroma.sqlite3.max-seq-id-backup-1").exists()
assert not (tmp_path / "chroma.sqlite3.max-seq-id-backup-2").exists()
def test_prune_respects_glob_escape_for_metacharacter_paths(tmp_path):
"""Palace paths can contain glob metacharacters like ``[``.
Without ``glob.escape`` the pattern would silently match nothing (the
bracket is read as a character class), leaving backups unpruned. Callers
escape the literal prefix; this confirms the helper prunes correctly once
they do.
"""
import glob
weird = tmp_path / "weird[name]"
weird.mkdir()
for i in range(1, 4):
_make_backup_file(weird, f"chroma.sqlite3.max-seq-id-backup-{i}", mtime=i * 100)
pattern = os.path.join(glob.escape(str(weird)), "chroma.sqlite3.max-seq-id-backup-*")
removed = prune_backups(pattern, max_backups=1)
assert len(removed) == 2
assert (weird / "chroma.sqlite3.max-seq-id-backup-3").exists()
def test_prune_is_best_effort_on_delete_failure(tmp_path, monkeypatch):
"""A failed deletion is logged and skipped, never raised — pruning must
not undo a migrate/repair that already succeeded."""
for i in range(1, 5):
_make_backup_file(tmp_path, f"b.{i}", mtime=i * 100)
real_remove = os.remove
def flaky_remove(path):
if path.endswith("b.1"):
raise OSError("permission denied")
return real_remove(path)
monkeypatch.setattr(os, "remove", flaky_remove)
logs = []
removed = prune_backups(str(tmp_path / "b.*"), max_backups=2, log=logs.append)
# b.1 and b.2 were over the limit; b.1 failed, b.2 succeeded.
assert str(tmp_path / "b.2") in removed
assert str(tmp_path / "b.1") not in removed
assert (tmp_path / "b.1").exists()
assert any("could not remove" in line for line in logs)
+25
View File
@@ -19,8 +19,14 @@ HOOK_CONFIG = REPO_ROOT / ".claude-plugin" / "hooks" / "hooks.json"
# timeout of 60s in mempalace/hooks_cli.py. The hook-level floor of 60
# keeps the inner bound from being truncated, and the ceiling of 90
# bounds the worst case at ~30s above that.
# SessionEnd backgrounds all of its work in the shell wrapper — the foreground
# only forks the detached child and returns in milliseconds — so its timeout is
# a generous backstop on a near-instant operation, not a synchronous-work bound
# like Stop/PreCompact. A bound is still required (#1465) so a wedged fork can
# never fall back to the 600s command default.
EVENT_TIMEOUT_BOUNDS: dict[str, tuple[int, int]] = {
"Stop": (10, 30),
"SessionEnd": (5, 30),
"PreCompact": (60, 90),
}
@@ -86,3 +92,22 @@ def test_no_unbounded_events_in_plugin_config(hook_config: dict) -> None:
"Add a (floor, ceiling) entry to EVENT_TIMEOUT_BOUNDS in this test "
"after deciding the worst-case freeze the event can tolerate."
)
def test_session_end_hook_uses_background_wrapper(hook_config: dict) -> None:
"""Claude SessionEnd should use the backgrounding wrapper, not PreCompact."""
events = hook_config.get("hooks", {})
assert "SessionEnd" in events
assert "PreCompact" in events
assert events["SessionEnd"] != events["PreCompact"]
commands = [
hook["command"]
for entry in events["SessionEnd"]
for hook in entry.get("hooks", [])
if hook.get("type") == "command"
]
assert any("mempal-session-end-hook.sh" in command for command in commands)
assert not any("mempal-precompact-hook.sh" in command for command in commands)

Some files were not shown because too many files have changed in this diff Show More