On the HTTP transport, a client that hangs up mid-response makes the send
path raise an unhandled BrokenPipeError / ConnectionResetError, so the
default socketserver handle_error logs a full traceback for a routine
disconnect. Override handle_error on the HTTP server to log the disconnect
(ConnectionError, plus ssl.SSLEOFError over TLS) at DEBUG and drop it, while
every other exception still reaches the default handler with its traceback.
The wider ssl.SSLError tree is left uncaught, so genuine TLS handshake/cert
failures still surface.
Addresses the disconnect half of #2003.
Claude Code pages large tool outputs to <session>/tool-results/*.txt
inside ~/.claude/projects/<slug>/. These are raw machine dumps (command
output, MCP results) referenced from the transcript JSONL — not
conversations. The convo scanner picked them up as plain .txt, so every
transcript ingest stored megabytes of tool output as memories: on one
field palace, 12,825 drawers traced back to tool-results files, with a
single paged file producing 3,611 drawers.
Adds CONVO_SKIP_DIRS = SKIP_DIRS | {'tool-results'} used only by
scan_convos — project-mining semantics are unchanged, and regular nested
conversation dirs still scan (covered by a companion test).
_mark_held(palace_key) ran before the try: whose finally runs
_mark_released(). An async exception (SIGINT/KeyboardInterrupt) landing
after _mark_held() and before the try: skips _mark_released(), stranding
the key in the process-wide _palace_lock_keys set while the outer finally
frees the flock. The in-memory hold then outlives the OS lock: a later
re-entrant acquire passes through and writes without the flock while
another process can acquire it, i.e. two writers into one palace.
Move _mark_held() inside the try so it pairs with _mark_released() on
every exit. Add a regression test that injects the interrupt in the
window and asserts the holder set is not stranded.
The stdio loop ran _refresh_sqlite_integrity_status() and
_refresh_vector_disabled_flag() before reading the first request.
PRAGMA quick_check reads every page of chroma.sqlite3, so on multi-GB
palaces the probe alone (measured: 20.3s on a 1.72 GB / 326k-drawer
palace, 40-46s under disk/lock contention) starves the MCP client's
60s connect timeout — even though the initialize response itself never
touches the database. The HTTP transport already starts without the
synchronous probe.
Move both probes to a daemon thread (mcp-startup-preflight). The #1222
intent is preserved: the probe still starts at startup and logs its
warning as soon as it finishes. Consumers that need the verdict
(_ensure_sqlite_integrity_status via the tool-call integrity gate)
serialize on a new _sqlite_integrity_refresh_lock with double-checked
locking, so a tool call arriving mid-probe waits for the in-flight
verdict instead of running a second O(database size) quick_check —
and never proceeds unverified.
Measured on the 1.72 GB palace with the >512 MB startup gate disabled
(MEMPALACE_STARTUP_INTEGRITY_MAX_MB=0, full quick_check in flight):
initialize 1.4s (was 20-46s); first tool call after probe completion
3.4s with sqlite_integrity checked=true ok=true.
Complements c54531a: the oversized-palace skip still applies to the
background probe, but the handshake no longer depends on it.
PR #1960 merged with a red `lint` job: `ruff format --check .` wanted to
collapse the multi-line `MineAlreadyRunning(...)` raise in the new
`test_peer_writer_readonly_self_heals_after_peer_exits` onto one line
(it fits the line-length limit). All six real test jobs passed; only the
formatter check failed, which left `develop` red on lint.
Reformat that one statement so `ruff format --check .` is clean again.
No logic change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The #1818 peer-writer guard latched _MCP_WRITER_READ_ONLY=True on the first
MineAlreadyRunning and short-circuited every subsequent acquisition attempt,
so a server that came up read-only (a peer held the per-palace flock at
startup) stayed read-only for its entire process lifetime — even long after
the peer exited and the OS released the flock. In the common case of several
overlapping Claude sessions (one server per session, all on the same palace),
whichever session started second was stranded: mutating tools kept refusing
with -32001 and the only remedy was killing/restarting that server.
_mcp_peer_writer_refusal already calls _acquire_mcp_writer_lock() on every
mutating tool, so the retry hook existed — the sticky latch just suppressed it.
Drop the read-only short-circuit: when read-only we now re-attempt the
non-blocking flock each call and transparently promote to writer once the peer
is gone. Race-safe — fcntl LOCK_NB is kernel-arbitrated, so two servers can
never both win. The genuinely-broken-lock path (_MCP_WRITER_LOCK_FAILED) is
still cached, since retrying a broken lock mechanism can't help.
Adds test_peer_writer_readonly_self_heals_after_peer_exits.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Conversation transcripts were assumed immutable once mined: the bulk
skip-check (prefetch_mined_set) only tracked "have we seen this
source_file before at the current normalize_version", with no mtime
comparison at all. That's wrong for how Claude Code sessions actually
work -- a session keeps appending to its own JSONL file while active,
and /compact or /clear can rewrite one in place. Once a session file
was mined, any content appended after that point would silently never
get mined, with no error or warning -- the file just looked
"already filed" forever.
palace.py:
- prefetch_mined_set() now returns dict[source_file, stored_mtime]
instead of a bare set[source_file]. `if src in mined_set` still works
identically (dict `in` checks keys), so this is a source-compatible
change for that access pattern; a caller that wants staleness
detection reads mined_set[src] and compares against the file's
current mtime itself. None means no mtime was ever stored (or
getmtime failed when the drawer was written) and must be treated as
stale, not "unknown, assume unchanged".
- Removed bulk_check_mined(): it already existed for exactly this
purpose (bulk mtime prefetch) but had zero callers anywhere in the
codebase and was missing the normalize_version/extract_mode filtering
prefetch_mined_set has -- folded its intent into prefetch_mined_set
instead of maintaining two subtly-different, overlapping bulk scans
over the same underlying data.
- file_already_mined()'s docstring corrected: it previously claimed
"transcripts are assumed immutable" for convo mining. That's no
longer true; corrected to describe the actual current split (convo
miner's bulk skip-check uses prefetch_mined_set's stored mtimes; this
function's check_mtime=True path is now only its per-file,
lock-held race-condition recheck).
convo_miner.py:
- New _is_unchanged_since_last_mine() helper (extracted to keep
_mine_convos_impl under the repo's cyclomatic-complexity gate):
false whenever the file isn't in the prefetched map, its stored mtime
is None, getmtime fails, or the mtimes don't match -- true only when
genuinely unchanged.
- _file_chunks_locked's metadata now stamps source_mtime on every real
drawer (mirroring miner.py's existing pattern), and its in-lock
recheck now passes check_mtime=True.
- _register_file's 0-chunk sentinel also stamps source_mtime, so a
short file that later grows past MIN_CHUNK_SIZE is detected as
changed instead of being skipped forever by the sentinel.
One-time cost worth flagging: no existing convo drawer has source_mtime
stored (this field never existed for convo mining before now), so the
first `mempalace mine --mode convos` after this ships will see every
already-mined file as stale and fully re-mine it. Not a bug --
_file_chunks_locked's existing purge-before-insert means no
duplication results -- just a real, one-time cost across a large
corpus.
tests/test_convo_miner.py: 7 new tests -- grown-file re-mine picks up
new content, unchanged file still skipped (the mtime check must not
regress the existing optimization), grown-file re-mine purges stale
drawers rather than accumulating duplicates (checked via unique content
markers, not raw counts -- ChromaDB collections can carry unrelated
bookkeeping rows), prefetch_mined_set's returned mtime matches the real
file, None handling for a drawer with no stored mtime, a legacy
drawer (no source_mtime field, simulating pre-this-change data) is
correctly re-mined rather than skipped forever, and the sentinel path
stamps source_mtime too.
Full suite: 3327 passed, 20 skipped, 0 failed. ruff check / ruff
format -- clean.
Rebasing Lochness's exclude_patterns work (#1213) onto current develop
pushed _mine_impl's cyclomatic complexity to 26, tripping the repo's
max-complexity=25 ruff gate (clean on develop before this rebase).
Extracted the pre-scanned-file-list filtering branch into
_apply_exclude_patterns_to_prescanned_files -- same behavior, no test
changes needed, complexity back under the gate.
Allow projects to specify .gitignore-style patterns that the miner should
skip, without relying on .gitignore for mining control.
A new optional exclude_patterns list in mempalace.yaml is parsed by the
existing GitignoreMatcher class via a new from_patterns() classmethod —
same syntax, same semantics as .gitignore, no new dependency.
exclude_patterns:
- '*.md'
- '*.yaml'
- 'docs/' # dir-only: prunes entire tree without descending
- 'dist/'
- 'coverage/'
Key behaviour:
- Patterns follow .gitignore rules: anchoring (/pattern), dir-only
- dirs[:] pruning via GitignoreMatcher.matches(..., is_dir=True) so
excluded subtrees are never walked
- Checked after .gitignore filtering; force_include (--include-ignored)
bypasses exclude_patterns
- Pre-scanned files lists (init double-scan optimisation) are filtered too
- Backwards compatible: omitting exclude_patterns changes nothing
Changes:
- GitignoreMatcher.from_patterns(): new classmethod, same rule parser as
from_dir(), reads from a list instead of a file on disk
- scan_project(): builds one exclude_matcher before os.walk; used for
both dirs[:] pruning and per-file filtering
- _mine_impl(): applies the exclude matcher to pre-scanned files lists
when provided by the caller
- tests/test_miner.py: three new tests
test_scan_project_exclude_patterns_skips_matching_files
test_scan_project_exclude_patterns_prunes_entire_directory
test_scan_project_exclude_patterns_include_ignored_bypasses_exclusion
Two review comments on this PR, both addressed:
- gemini-code-assist flagged that maybe_autoheal_fts5_index's default
progress=print goes straight to stdout. _validate_palace_fts5_after_mine
runs inside the MCP server process too (mcp_server.tool_mine ->
miner.mine), where stdout is the JSON-RPC transport -- a stray print()
there would corrupt the protocol stream and crash the connection. Pass
progress=logger.info instead; palace.py already has the module logger.
- nikkunikku corroborated the fix from a real 1.4GB production palace (278
repeated abort-loop iterations before the fix) and pointed out a real
test gap: the fixture-based auto-heal tests fabricate real FTS5
corruption via direct shadow-table writes, which some SQLite builds
refuse (existing pytest.skip paths in test_miner_fts5_validation.py,
related to #1925) -- so on those builds the auto-heal wiring in
_validate_palace_fts5_after_mine is never actually exercised. Added
their suggested build-independent tests, adapted to this file's fixture
helpers: test_validator_suppresses_raise_when_autoheal_clears and
test_validator_still_raises_when_autoheal_cannot_clear, stubbing
mempalace.repair.sqlite_integrity_errors/maybe_autoheal_fts5_index
directly instead of fabricating corruption.
Added a third test, test_validator_passes_logger_progress_not_print_to_autoheal,
covering the specific progress= wiring: the two tests above mock
maybe_autoheal_fts5_index entirely and discard its kwargs, so neither would
have caught the progress=print regression this commit actually fixes. The
new test captures the real kwargs and asserts progress is a bound method of
palace.py's own logger (not print), without pinning to logger.info
specifically -- severity level is a verbosity choice, not a correctness
requirement, so the assertion shouldn't fail on a reasonable future change
to e.g. logger.debug. Verified both directions: fails against the
pre-fix `print` default (and shows the leaked stdout line to prove it),
passes at .info and at .debug alike.
Full suite: 3221 passed, 20 skipped (unchanged skip count). ruff
check/format clean.
mempalace_status reported a passing SQLite integrity check on non-chroma
backends (checked/ok true, sqlite_path pointing at a chroma.sqlite3 that does
not exist) even though _refresh_sqlite_integrity_status short-circuits the
check there. _sqlite_integrity_payload now reports the check as not-applicable
(checked false, ok null, reason) for non-chroma backends, keeping the chroma
payload shape and error surfacing unchanged.
Co-Authored-By: Zoz92 <66385795+Zoz92@users.noreply.github.com>
shutil.move's fallback for a failed os.rename is copytree + rmtree. On
Windows, when any file inside the palace is held open by another
process (a live MCP server, a running mine, another harness), the
rename fails and shutil.move falls back to deleting the live palace
file-by-file via rmtree -- which itself then fails partway through on
the first locked file, leaving the palace partially gutted next to a
partial (or empty) archive copy.
Reproduced live twice (Windows 11, 2026-07-05 and 2026-07-06): running
`mempalace repair --mode from-sqlite --yes --archive-existing` while
an MCP server / detached mine held palace/*/data_level0.bin open threw
mid-rmtree in both cases. The palace itself survived only because the
specific locked files could not be unlinked -- a different lock
pattern (e.g. a lock on a file rmtree reaches first) would have lost
data with no way back.
os.rename is atomic on both platforms it matters on (POSIX rename(2),
Windows MoveFileEx) -- it either fully succeeds or fails without
touching anything. Catch the failure and abort cleanly with actionable
guidance instead of a raw traceback.
_errors_are_isolated_fts5 gated auto-heal on one specific message shape:
malformed inverted index for FTS5 table
SQLite >= ~3.5x (confirmed on 3.53.2 / Python 3.13.7) reports the same
isolated-FTS5 condition with different wording instead:
fts5: corruption found reading blob N from table "embedding_fulltext_search"
The narrow regex never matched this phrasing, so maybe_autoheal_fts5_index
silently declined to heal on any machine running a recent-enough SQLite,
falling straight through to the hard-abort path -- the exact condition
the whole auto-heal feature (#1926/#1928) exists to avoid. Widened the
pattern to match either wording.
Caught by running this repo's own test suite on this machine:
test_repair.py's two auto-heal tests were failing (not, as assumed
earlier, pre-existing/unrelated flakiness -- that assumption was never
actually verified). Traced to this exact classification gap.
Fixing this correctly also exposed that four tests in
test_miner_fts5_validation.py had been passing for the wrong reason: they
manufacture the exact "reporter-shaped" isolated-FTS5 corruption (#1926's
actual bug shape) and asserted mine() must raise MineValidationError for
it -- true only because the classifier bug prevented auto-heal from ever
engaging. With the classifier fixed, that corruption is now correctly
auto-healed and mine() succeeds instead, so those tests' expectations
were stale, not their fixtures being invalid:
- test_helper_raises_on_fts5_segment_corruption -> renamed
test_helper_auto_heals_fts5_segment_corruption; asserts no raise + a
clean post-heal quick_check, instead of expecting a raise.
- test_full_chain_raises_through_mine_impl and
test_mine_impl_does_not_print_partial_summary_on_validation_error: their
real purpose is exception-passthrough / banner-suppression when the
validator DOES raise, not proving any particular corruption triggers it.
Switched from real file corruption to a monkeypatched raise. (Tried
swapping to _page_mangle's non-isolated corruption first -- that made
ChromaDB's own Rust bindings panic just opening the file for the
re-mine's get_collection() call, a native crash rather than a catchable
Python exception, before the validator ever ran. Different failure mode
than what these tests are about, and not reliable to depend on.)
- test_mine_formats_full_chain_raises_when_fts5_corrupt: same fix, mirrors
the miner-path change for the extract path.
- Added test_full_chain_auto_heals_isolated_fts5_corruption and
test_mine_formats_full_chain_auto_heals_isolated_fts5_corruption as
companions, proving the full mine()/mine_formats() chain -- not just
the standalone validator -- actually auto-heals and succeeds end-to-end
for the isolated case now that it's correctly classified.
- test_errors_are_isolated_fts5_classification: added the new message
wording as an explicit regression fixture (pinned literally, not
dependent on whatever this machine's SQLite happens to emit).
Full suite: 3302 passed, 20 skipped, 0 failed -- first fully clean run
this session. ruff check / ruff format -- clean.
mempalace_checkpoint and mempalace_delete_by_source (added in 3.5.0) were
missing from _MUTATING_TOOLS, so a server started with --read-only /
MEMPALACE_MCP_READ_ONLY=1 still allowed writing drawers and bulk-deleting
by source. The same gap affected the peer-writer lock gate, which uses
the same frozenset.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
mempalace mine aborts with the "ABORT: SQLite-layer corruption detected"
banner on an isolated FTS5 inverted-index corruption -- the specific case
maybe_autoheal_fts5_index already exists to fix in place. That helper is
wired into cmd_repair's preflight, but not into mine's own post-mine
validation (palace._validate_palace_fts5_after_mine), so mine forces a
manual `mempalace repair` for a corruption class that's already safely
self-healable.
This wires the same auto-heal call into _validate_palace_fts5_after_mine,
before it raises MineValidationError. maybe_autoheal_fts5_index returns
the *remaining* errors after the heal attempt, so MineValidationError
still raises whenever the corruption isn't the isolated, fully-healable
case -- this only changes behavior when the heal has verifiably and
fully cleared the corruption.
Verified against a real-world repro (mining a real Claude Code project
directory deterministically triggered this corruption after all files
filed successfully, zero concurrency, single uninterrupted process):
mempalace repair --yes confirmed the auto-heal path clears it before
proceeding to a full rebuild. With this patch, mine self-heals the same
case directly -- no abort, Files processed: <n>, Done, PRAGMA quick_check
clean afterward.
Test suite: 3214 passed, 20 skipped -- no new failures. Two pre-existing
unrelated failures in test_repair.py (a SQLite-version-dependent FTS5
corruption message-wording mismatch, tracked separately) reproduce
identically on unmodified develop.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The MCP server ran PRAGMA quick_check on the full chroma.sqlite3 during
startup, before answering the initialize handshake. quick_check is
O(database size); on multi-GB palaces it exceeds the MCP client's ~30s
connection timeout, so the server never finishes starting and the client
drops the connection (observed >2min on a 4.6GB palace).
Skip the startup probe when chroma.sqlite3 exceeds
MEMPALACE_STARTUP_INTEGRITY_MAX_MB (default 512MB; 0 disables). The gate
lives in _refresh_sqlite_integrity_status, the single choke point for the
startup calls and every lazy consumer. `mempalace repair` preflight still
runs the full quick_check via repair.sqlite_integrity_errors, so
SQLite-layer corruption is still caught on the destructive path.
Refs #1818.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Jq495N7e2D4wY2Mp2AQvg7
Igor Lins e Silva
KeilerHirsch
mvalentsev
J.R. Murray
Evgen197310
Lochness
Grace Gettert
J.R. Murray
Yusef Haddad
Tom Boucher
Cheney Zhang