`mempalace_diary_write` returns an `entry_id` for every diary entry, but
for entries large enough to be chunked that id was unusable: get_drawer,
update_drawer and delete_drawer all answered "Drawer not found", and
list_drawers showed the entry as N unrelated chunk rows.
Two metadata conventions never met. The diary chunking path stamped
`parent_entry_id` on each chunk, while the logical-id read paths added in
#1782 query only `parent_drawer_id`. Both keys mean the same thing --
"physical chunk of this logical drawer" -- so chunk groups written by
diary_write were invisible to logical-id resolution. Same bug class as
#1763, which #1782 fixed for `add_drawer` drawers only.
Read paths now resolve either key via `_PARENT_ID_KEYS`:
- `_logical_chunk_group()` matches both with an `$or` (fixes get /
update / delete). All four backends support `$or`.
- `_collapse_drawer_rows()` groups on either (fixes list_drawers, which
the `$or` alone does not cover).
- `searcher._result_drawer_id()` resolves either, so a hit on a chunked
diary entry reports the id that fetches the whole entry rather than
the single chunk that matched.
New diary writes also stamp `parent_drawer_id` alongside
`parent_entry_id` so the two conventions converge going forward. Because
the read paths still accept the `parent_entry_id`-only shape, palaces
written before this fix are repaired with no data migration.
Diary chunks are written without `source_file`, so neighbor expansion
(#1580) returns early on them and is unaffected by the added key.
Also drops the comment telling callers to iterate `chunk_ids` (it
documented the bug as intended behavior) and a stale claim that search
rejoins chunks via `parent_entry_id` -- no search code read that key.
A long-lived MCP server imports mempalace and chromadb once and serves from
those in-memory modules for the life of the process, so an upgrade on disk
mid-session never reaches it and it keeps accepting writes produced by code
the user no longer has installed.
Refuse mutating tools with JSON-RPC -32005 once a watched distribution's
installed version differs from the snapshot taken at import, or once it is
gone entirely. Reads stay available, mempalace_status reports
library_versions, and MEMPALACE_MCP_ALLOW_STALE_LIBRARY=1 opts out.
Both sides of the comparison come from installed metadata rather than a live
module.__version__. A distribution whose metadata cannot be read, or whose
search root will not open, is reported and left uncompared rather than
treated as removed: importlib.metadata suppresses the failure at both of
those levels, so either one would otherwise look exactly like an uninstall
and refuse every write on a healthy install.
The same module also memoizes each search root's listing against that root's
mtime, read in seconds where this fingerprint compares nanoseconds. An
upgrade whose removal and creation both land inside one timestamp tick would
then be answered from the listing taken before it, naming a dist-info that is
already gone; its version reads as empty, the distribution is left
uncompared, and nothing moves that mtime afterwards, so the gate would stay
off for it for the rest of the process. Drop the memo before each reading.
Watch chromadb only when chromadb is the backend serving. It is a hard
dependency rather than an extra, so it is installed even for a palace kept
in Postgres, and watching it there would refuse that user's writes whenever
chromadb alone was upgraded, over a library that writes nothing they own. A
backend that cannot be resolved keeps it watched.
Skip a sys.path entry carrying an embedded NUL. os.stat and os.listdir refuse
it during argument conversion, raising ValueError rather than the OSError
those callers hold; POSIX never gets there because realpath rejects it first,
but Windows resolves it and one such entry would end the whole reading.
The gate sits ahead of the diverged-index refusal added since (-32004, which
is why this one takes -32005). That gate's remedy is `mempalace repair
rebuild-index`, which would run the installed code against a palace this
process is still writing with the superseded one, so the restart instruction
has to be the one that reaches the client; the index probe re-runs per call
and surfaces immediately after a restart. Ordering it this way also skips
that gate's segment probe on a call already refused. Both directions of the
precedence are pinned by tests.
Co-Authored-By: messelink <274674234+messelink@users.noreply.github.com>
Address the last two Codex review comments on #2079:
- Reopen sqlite_exact immutable read-only connections when WAL/SHM
sidecars appear after a clean-database open so recall sees a later
writer's commits instead of a frozen pre-writer snapshot.
- Clear per-palace embedder-identity validation when MCP discards
storage handles during ownership promotion, so an empty read-only
open cannot skip recording the active model on the first write.
Read-only gated on _MUTATING_TOOLS, which is the palace-write set the
peer-writer lease consults to decide which calls need the palace mine
lock. Two tools change state without touching the palace, so they are
correctly absent from that set and were served on a --read-only server
anyway: mempalace_hook_settings rewrites ~/.mempalace/config.json via
MempalaceConfig.set_hook_setting, and mempalace_memories_filed_away
unlinks ~/.mempalace/hook_state/last_checkpoint on both branches.
Add _READ_ONLY_REFUSED_TOOLS and point the dispatch gate and the
tools/list filter at it. _MUTATING_TOOLS and the peer-writer path are
unchanged: adding the two names there instead would put a config-only
tool under the palace lease, so a server that lost the lease to a peer
would answer -32001 for a call the lease has no say over.
mempalace_reconnect stays reachable on purpose and the comment records
why, since clearing ChromaBackend._quarantined_paths lets the reopen
rename a segment directory. The two --read-only help strings and the
matching row in the remote-server guide said "the mutating tools",
which now names the narrower set.
A daemon job refused the palace write lock was marked terminal `failed` and
never retried. `mine_palace_lock` guards the palace write itself, so a refusal
means no drawer was filed, but the queue recorded it exactly like a crash
mid-execution, whose outcome is unknown. The work was dropped: it only ran
again if a hook happened to re-submit equivalent work, and a job kind no hook
re-emits was lost outright.
Refusals now defer. The job goes back to `queued` with `started_at` cleared,
the claim's attempt increment undone, and the reason recorded. The update is
scoped to the claim that was refused (started_at match), so a defer racing a
recovery re-claim cannot re-queue work it does not own. `claim_next` clears
the recorded reason so it never outlives the claim it describes. Every other
failure stays terminal: a crashed job's outcome is unknown, and blindly
re-running a non-idempotent kind would re-file verbatim content, which is what
MAX_ATTEMPTS guards.
The worker cools a refused job off in memory and moves on rather than sleeping
in line. It is the only worker and the holder keeps the lock until its write
finishes, which can be a long mine, so blocking would stall every unrelated job
behind a lock that has nothing to do with them, and a job merely queued behind
the refused one would never be claimed. `claim_next(exclude=...)` skips a
cooling job, so the oldest-first ordering cannot hand the same refused job back
forever while newer work waits. The filter runs in Python rather than an
`id NOT IN (?, ?, ...)` list, which would bind one host parameter per cooling
job against a cap that defaults to 999 before SQLite 3.32. The cooldown
lives in the worker, not the schema, which has no migrations; a restart just
retries at once, costing one refusal, never work.
`tool_diary_write` swallowed `MineAlreadyRunning` in its bare `except
Exception`, so the refusal reached the daemon with no `error_class` and
`diary_write` would still have been dead-lettered. It now uses a typed handler
ahead of the bare `Exception`, the way `tool_mine` and `tool_sync` already do.
Deferral makes a refused job non-terminal, and `DaemonClient.wait` only returns
on a terminal state, so callers that wait on purpose would have waited for a
state a parked job cannot reach: a foreground `mine --daemon` for the one-hour
default, and the `hooks_cli` pre-compaction mine and Stop-hook diary paths for
their whole timeout on every fire, each then reporting a failure that never
happened. `wait(stop_on_lock_deferral=True)` hands the parked job back instead.
The CLI echoes the global `--palace` back into the command it suggests, so the
suggestion does not silently list the default palace's queue instead of the
one the job is parked in. A job that is genuinely running is still waited out.
Co-Authored-By: mjvmsteixeira <185609735+mjvmsteixeira@users.noreply.github.com>
The new real-Chroma checkpoint test dropped its create-time client with a
bare del, leaving the per-path SharedSystemClient's SQLite/HNSW handles
open on Windows (#1128). Close both clients so the temp palace is released,
matching the conftest fixture's close-not-del pattern.
mempalace_checkpoint hard-coded added_by="checkpoint" for every drawer,
dropping the filing agent's identity even though it arrives in the same
call via diary.agent_name. Add an optional top-level added_by parameter
and resolve attribution as explicit > diary agent_name > "checkpoint";
blank/whitespace/non-string values defer to the next source. The value
is declared in the tool schema so tools/call admits it on both stdio and
HTTP transports.
Fixes#2023
Co-Authored-By: epinethrone <172391900+epinethrone@users.noreply.github.com>
The stdio loop ran _refresh_sqlite_integrity_status() and
_refresh_vector_disabled_flag() before reading the first request.
PRAGMA quick_check reads every page of chroma.sqlite3, so on multi-GB
palaces the probe alone (measured: 20.3s on a 1.72 GB / 326k-drawer
palace, 40-46s under disk/lock contention) starves the MCP client's
60s connect timeout — even though the initialize response itself never
touches the database. The HTTP transport already starts without the
synchronous probe.
Move both probes to a daemon thread (mcp-startup-preflight). The #1222
intent is preserved: the probe still starts at startup and logs its
warning as soon as it finishes. Consumers that need the verdict
(_ensure_sqlite_integrity_status via the tool-call integrity gate)
serialize on a new _sqlite_integrity_refresh_lock with double-checked
locking, so a tool call arriving mid-probe waits for the in-flight
verdict instead of running a second O(database size) quick_check —
and never proceeds unverified.
Measured on the 1.72 GB palace with the >512 MB startup gate disabled
(MEMPALACE_STARTUP_INTEGRITY_MAX_MB=0, full quick_check in flight):
initialize 1.4s (was 20-46s); first tool call after probe completion
3.4s with sqlite_integrity checked=true ok=true.
Complements c54531a: the oversized-palace skip still applies to the
background probe, but the handshake no longer depends on it.
PR #1960 merged with a red `lint` job: `ruff format --check .` wanted to
collapse the multi-line `MineAlreadyRunning(...)` raise in the new
`test_peer_writer_readonly_self_heals_after_peer_exits` onto one line
(it fits the line-length limit). All six real test jobs passed; only the
formatter check failed, which left `develop` red on lint.
Reformat that one statement so `ruff format --check .` is clean again.
No logic change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The #1818 peer-writer guard latched _MCP_WRITER_READ_ONLY=True on the first
MineAlreadyRunning and short-circuited every subsequent acquisition attempt,
so a server that came up read-only (a peer held the per-palace flock at
startup) stayed read-only for its entire process lifetime — even long after
the peer exited and the OS released the flock. In the common case of several
overlapping Claude sessions (one server per session, all on the same palace),
whichever session started second was stranded: mutating tools kept refusing
with -32001 and the only remedy was killing/restarting that server.
_mcp_peer_writer_refusal already calls _acquire_mcp_writer_lock() on every
mutating tool, so the retry hook existed — the sticky latch just suppressed it.
Drop the read-only short-circuit: when read-only we now re-attempt the
non-blocking flock each call and transparently promote to writer once the peer
is gone. Race-safe — fcntl LOCK_NB is kernel-arbitrated, so two servers can
never both win. The genuinely-broken-lock path (_MCP_WRITER_LOCK_FAILED) is
still cached, since retrying a broken lock mechanism can't help.
Adds test_peer_writer_readonly_self_heals_after_peer_exits.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mempalace_status reported a passing SQLite integrity check on non-chroma
backends (checked/ok true, sqlite_path pointing at a chroma.sqlite3 that does
not exist) even though _refresh_sqlite_integrity_status short-circuits the
check there. _sqlite_integrity_payload now reports the check as not-applicable
(checked false, ok null, reason) for non-chroma backends, keeping the chroma
payload shape and error surfacing unchanged.
Co-Authored-By: Zoz92 <66385795+Zoz92@users.noreply.github.com>
The MCP server ran PRAGMA quick_check on the full chroma.sqlite3 during
startup, before answering the initialize handshake. quick_check is
O(database size); on multi-GB palaces it exceeds the MCP client's ~30s
connection timeout, so the server never finishes starting and the client
drops the connection (observed >2min on a 4.6GB palace).
Skip the startup probe when chroma.sqlite3 exceeds
MEMPALACE_STARTUP_INTEGRITY_MAX_MB (default 512MB; 0 disables). The gate
lives in _refresh_sqlite_integrity_status, the single choke point for the
startup calls and every lazy consumer. `mempalace repair` preflight still
runs the full quick_check via repair.sqlite_integrity_errors, so
SQLite-layer corruption is still caught on the destructive path.
Refs #1818.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Jq495N7e2D4wY2Mp2AQvg7
* feat(mcp): add since/before date filter to list_drawers (#1128)
mempalace_list_drawers previously filtered only by wing/room. This adds
optional since/before ISO date bounds on filed_at: since is inclusive,
before is exclusive.
The filter runs in Python after the rows are fetched. ChromaDB 1.5.7
rejects string operands for $gte/$lt and filed_at is stored as an ISO
string, so a server-side where comparison is not available; the tool
already collapses and paginates the full result set in Python.
Drawers whose filed_at is missing or unparseable are excluded while a
bound is active, and inverted bounds (since >= before) return a clear
error.
* test: close chromadb clients between tests to fix Windows handle leak (#1128)
chromadb 1.5.7 caches one System per palace path and only frees the
SQLite/HNSW file handles on client.close(); the collection fixture and
the per-test MCP cache reset only dereferenced the client, so handles
leaked across the session. Harmless on POSIX (rmtree unlinks open files),
but on Windows the handles stay locked and accumulate until an HNSW
segment write in a later test's setup fails, which surfaced here as
TestDeleteBySource::test_commit_purges_matching_closets asserting 0 == 2.
Close the client in the collection fixture and in _reset_mcp_cache so the
handles are released between tests.
* test: release backend chromadb clients between tests (#1128)
palace.get_collection() caches one PersistentClient per palace_path on the
process-wide backend singleton and never closes it; sweep, repair and several
CLI tests reach the store through it. chromadb frees the rust-side SQLite/HNSW
file handles only on client.close(), so the handles leak across the whole
session: a 30-palace probe shows ~200 open file descriptors into the palace
tree, dropping to 0 once the clients are closed.
On POSIX the open handles are harmless (rmtree unlinks open files), but on
Windows they stay locked and accumulate until a later test's HNSW segment
write fails ("Failed to apply logs to the hnsw segment writer"), e.g.
test_sweeper.py::TestSweeperTandem::test_sweep_recovers_untaken_message_at_cursor_timestamp.
Drain the cached clients in the autouse _reset_mcp_cache teardown via
close_palace(), which closes each PersistentClient (releasing its handles)
without marking the backend closed so it stays reusable. Complements the
collection-fixture and _client_cache close() added earlier.
* fix(chroma): require SQLite magic header for ChromaBackend.detect() (#1893)
Closes#1893.
ChromaBackend.detect() was returning True for a 0-byte chroma.sqlite3 file
because the check was just os.path.isfile(...). On a palace that has any
other backend marker alongside a stale 0-byte chroma.sqlite3,
resolve_backend_name then raises BackendMismatchError and the palace becomes
unopenable until the user manually rm's the empty file.
The 0-byte file appears as a side effect of any sqlite3.connect() on a
missing path — Python creates the file immediately but writes the SQLite
header only on the first statement. So any code path that touches the
chroma.sqlite3 path with bare sqlite3.connect(), including chromadb's own
PersistentClient lazy-init (see the comment at backends/chroma.py:2052),
can leave a 0-byte artifact behind.
Fix: detect() now reads the first 16 bytes and compares to the SQLite
magic prefix b"SQLite format 3\x00" instead of relying on file presence
alone. One extra open() + 16-byte read; detect() isn't a hot path.
Properties:
- Rejects 0-byte files (the symptom #1893 is about).
- Rejects non-SQLite garbage at the canonical path (partial writes, etc.).
- Doesn't false-negative on real chroma palaces: any chroma palace whose
PersistentClient has done any work has the magic header on disk
(verified — CREATE TABLE is enough to land the header).
- Doesn't couple detect() to chroma's specific schema; the magic header
is stable across chromadb releases.
Test sweep: many test files used (chroma.sqlite3).touch() or
.write_bytes(b"") as a "fake palace" shortcut, exploiting the loose
isfile() check (one such site even had the comment "# pass the isfile
guard"). After this change, those stand-ins no longer register as chroma
palaces. Introduced tests/_chroma_palace_helper.py::make_minimal_chroma_sqlite
following the existing _backend_conformance.py precedent, and updated 15
call sites across 8 test files to use it. The existing
test_chroma_detect_matches_palace_with_chroma_sqlite (which encoded the
buggy semantics with write_bytes(b"")) is renamed to
test_chroma_detect_matches_palace_with_sqlite_header and now writes a
real SQLite database via the helper. Added two new tests for the
rejection paths (empty file, non-SQLite garbage).
Full env-cleared suite: 3137 passed, 20 skipped, 0 failed. ruff check
and ruff format --check both clean.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KC5Qsknh2zFRtRvVyjXiTA
* fix(sqlite_exact): require SQLite magic header for SQLiteExactBackend.detect()
Per gemini-code-assist review on #1892 PR #1896: SQLiteExactBackend has the
same os.path.isfile() detection pattern as ChromaBackend did, with the same
0-byte-file vulnerability. Mirrors the chroma fix for repo-wide consistency.
- SQLiteExactBackend.detect() now does the same 16-byte SQLite magic-prefix
check as ChromaBackend.detect().
- _chroma_palace_helper.py: factored its body into a private
_write_minimal_sqlite_file() and gained a sibling
make_minimal_sqlite_exact_sqlite() for the sqlite_exact filename. No churn
to any existing chroma call sites.
- test_sqlite_exact_backend.py:426 (the one site that wrote b"" for
sqlite_exact.sqlite3) updated to use the new helper.
- Three new tests in test_sqlite_exact_backend.py mirror the chroma trio:
matches with valid header, rejects empty file, rejects non-SQLite garbage.
Full env-cleared suite: 3140 passed, 20 skipped, 0 failed.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KC5Qsknh2zFRtRvVyjXiTA
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* fix(mcp): stop clobbering host app root logger at import (#1860)
_init_logging() ran at import and called logging.basicConfig(force=True),
resetting the root logger's level, format, and handlers unconditionally. An
app that configured logging before importing mempalace.mcp_server lost its
setup: a host on DEBUG dropped to INFO, custom formatters and handlers were
replaced.
force=True existed (#1495) only to keep MEMPALACE_LOG_FILE working when root
already had handlers. This keeps that contract without the reset: configure
root only when it is unconfigured (standalone); otherwise attach a
mempalace-filtered file handler additively and leave the host's config alone.
Adds _MempalaceLogFilter so the file captures every mempalace logger (the
dotted mempalace.* family plus the flat mempalace_* names) and nothing else.
* fix(mcp): survive importlib.reload and pin file log format (#1860)
Addresses review on #1885.
Restore _logging_configured from globals() so the idempotency guard survives
importlib.reload: a reload re-executes the module body, and a plain reset would
let _init_logging() stack a duplicate file handler on root.
Set an explicit "%(message)s" formatter on the file handler so the embedded
path does not depend on logging's default formatter (which already renders the
same, but is now pinned and identical to the standalone path).
Adds a reload regression test and a format-pin assertion.
delete_by_source removed only the drawers, leaving the matching closets
(the AAAK index layer, keyed independently by source_file) behind as stale
pointers at the now-deleted source. Mirror the closet-purge step used by
sync_palace / purge_file_closets: after the drawer delete, best-effort purge
the closets via push-down delete(where=...) so it survives large palaces and
can never abort an already-committed drawer delete.
Dry run now also reports closet_match_count so the caller sees the full blast
radius; commit reports closets_deleted. Adds tests that seed the closet
collection directly (tool_add_drawer doesn't build closets) and assert the
matching closets are purged on commit and counted on dry run.
Address PR review: guard untrusted MCP client payloads in
mempalace_checkpoint so a single malformed item cannot raise deep in
sanitization and abort the whole batch.
- coerce dedup_threshold to float
- require wing/room/content to be non-empty strings (skip + record error)
- validate the diary object and entry type, recording errors instead of
silently ignoring a malformed diary
On a genuine dedup-check error we still file the drawer rather than skip:
verbatim recall is the priority and add_drawer's idempotency blocks exact
duplicates. Adds tests for the non-string, dedup-error, and malformed-diary
paths.
Co-authored-by: Cursor <cursoragent@cursor.com>
Collapse the Cursor auto-save sequence (check_duplicate Nx + add_drawer
Nx + diary_write 1x) into a single mempalace_checkpoint MCP call so the
host UI renders one tool-call card and keeps its spinner up for the whole
save. The new tool reuses the existing single-item handlers, so semantic
dedup, idempotency, and verbatim guarantees are unchanged.
- mcp_server.py: add tool_checkpoint + register mempalace_checkpoint
- service.py: classify mempalace_checkpoint as a write tool
- cursor save hook: followup now drives one mempalace_checkpoint call
- docs: new mcp-tools.md section, help.md entry, 33 -> 34 tool count sweep
- tests: checkpoint add/dedup/malformed/registry + classify_tool
Co-authored-by: Cursor <cursoragent@cursor.com>
A JSON number or boolean passed for source_file is not coerced by the
string schema type, so it reached _sanitize_optional_source_file and
raised AttributeError from .strip() rather than a clean validation error.
Add an isinstance guard that raises ValueError, which tool_search returns
as a structured error. Regression test added.
Expose source_file alongside wing/room on mempalace_search. build_where_filter
generalizes to 0/1/2+ clauses and the filter threads through the main vector
path, the index-mismatch fallback, the vector-disabled BM25/SQLite path, and
the union lexical path so it never silently no-ops. Matching is on the exact
full stored value; results now expose source_path (the full path) for round
tripping, since the displayed source_file is a basename. The MCP schema gains
the source_file property and a path-tolerant sanitizer rejects null bytes,
lone surrogates, and overlong values.
Fixes#1815
Co-Authored-By: rendigua2025-gif <253093224+rendigua2025-gif@users.noreply.github.com>
- Soft-fallback on any exception, not just sqlite3.Error, so an unexpected
schema shape tripping the reconstruction degrades to build_graph() instead
of raising — matching the sibling sqlite fast paths (Copilot).
- Guard an empty/None _config.palace_path before building db_path (Gemini).
- Test: tripwire _get_collection in addition to graph_stats, directly
asserting the fast path never opens the chroma client / cold-loads HNSW
(Copilot).
tool_graph_stats built the whole palace graph via build_graph(), which pages
every metadata row (col.get limit/offset) and cold-loads the HNSW index — the
remaining overview-tool timeout from #1379 (#1836 fixed status / list_wings /
list_rooms / get_taxonomy but deliberately left graph_stats out, as it builds
an in-memory graph rather than a flat tally).
Add _sqlite_graph_stats(): one GROUP BY room, wing, hall over chroma.sqlite3,
reconstructing build_graph's room_data and the same stats (total_rooms,
tunnel_rooms, total_edges, rooms_per_wing, top_tunnels) with the same
per-drawer filter (room present, != "general", wing present) and edge
semantics (C(wings, 2) * halls per multi-wing room). Same _is_chroma_backend()
guard + client-path fallback as the #1748 overview tools.
Test seeds a real chroma palace mirroring the build_graph parity case in
test_palace_graph, with a tripwire on graph_stats proving the fast path runs
and that "general"/wing-less drawers are excluded. Idea adapted from #1381's
_sqlite_graph_stats.
#1748: normalize the sqlite fast path's "?" COALESCE placeholder (and None)
back to "unknown" inside _sqlite_taxonomy, so drawers missing wing/room
metadata keep the client path's output contract — no observable API change
for MCP clients on legacy/partial drawers.
#1068: invoke the parent embedder build via super().model instead of reaching
into cached_property's .func attribute, so the uncapped/fallback path survives
chromadb changing `model` to a plain @property or other descriptor.
tool_status / list_wings / list_rooms / get_taxonomy paged the entire
collection metadata through the chroma client (`_fetch_all_metadata`, a
1000-row offset loop), which cold-loads the HNSW index and materializes
hundreds of MB of dicts. On six-figure palaces these exceed the MCP host
tool-call limit (180k drawers ~3-4 min; 349k times out at 120-240s). The 5s
metadata cache only dedups repeat calls — it does not stop the cold-call
timeout.
A correct single-query SQL cross-tab already exists
(`backends.chroma._sqlite_wing_room_counts`) and is already the CLI default
(`miner.status`), but the MCP tools never used it — and the MCP-side sqlite
reader only ran behind the `vector_disabled` recovery path.
Add `_sqlite_taxonomy()` (guards on `_is_chroma_backend()`, returns None to
fall back) and wire it as the default path into all four overview tools. They
now answer from one GROUP BY without touching HNSW. Non-chroma backends
(qdrant, sqlite_exact) and unbootstrapped/legacy layouts fall back to the
existing client path unchanged.
graph_stats (also named in #1379) builds an in-memory graph via build_graph()
and needs its own treatment — tracked separately.
Five fixes from the Copilot review of the merged daemon PR:
1. Privacy: the queue DB's SQLite WAL/SHM sidecars hold un-checkpointed
verbatim payloads but were created with the caller's umask. Set the
owner-only umask in run_server BEFORE DaemonRuntime builds the QueueStore
(not only once the HTTP server starts), and harden any existing sidecars in
QueueStore._init_db as defense-in-depth.
2. DoS guard: reject a negative Content-Length in the request reader.
rfile.read(-1) would block until the client disconnects and bypass the
MAX_BODY_BYTES cap.
3. Side effects: extract _wal_log (+ _ensure_wal, _WAL_FILE, _WAL_REDACT_KEYS)
into a new side-effect-free mempalace/wal.py. The CLI sync path and the
daemon service layer obtained _wal_log via `from .mcp_server import _wal_log`,
which runs mcp_server's import-time stdio protection (os.dup2(2, 1);
sys.stdout = sys.stderr) in a non-MCP process and misroutes operator output.
mcp_server/cli/service now import from mempalace.wal.
4. Correctness: run_mcp_tool treated any dict as success. Write tools that
return a bare {"error": ...} (e.g. tool_create_tunnel/tool_delete_tunnel
validation) were recorded as succeeded; now the "error" key infers failure.
5. Hook budget: get_client_if_running()/health() take an explicit timeout, and
the hook "is the daemon up?" precheck uses a short HOOK_PROBE_TIMEOUT (0.5s)
so a wedged daemon can't stall the hook for the default 5s.
Adds tests/test_wal.py (import isolation + redaction) and daemon tests for the
umask ordering, negative Content-Length, run_mcp_tool error inference, and the
short probe timeout.
Address Gemini review on #1729:
- normalize source_file with strip_lone_surrogates so exact matching hits
rows mined from non-ASCII paths via cp1252 stdin (#1488), mirroring
tool_add_drawer's ingestion-side normalization
- isinstance(str) guard so a non-string source_file returns a clean error
instead of AttributeError
- default missing wing/room to "" in the dry-run sample, consistent with
the rest of the file
- add tests: non-string rejection + surrogate-normalization match
Adds an MCP tool to remove every drawer mined from a given source_file
exact match, for cleaning up benchmark/test data accidentally mined into
a user wing (ShareGPT dumps, results_mempal_*.jsonl, language config
JSON) that drowns out real memories in semantic search.
Matching is pushed to the backend via delete(where={"source_file": ...})
the same idiom the miner and diary-ingest paths already use so it is not
subject to the SQLite variable limit regardless of how many drawers share
the source. Defaults to a dry run reporting match count and a sample;
dry_run=false commits. Absent source is an idempotent no-op, not an error.
mvalentsev
Igor Lins e Silva
KeilerHirsch
Evgen197310
Grace Gettert
Tom Boucher
Prayaksh Upadhyay
Pim Messelink
undeadindustries
fatkobra
Arnold Wender