Commit Graph
834 Commits
Author SHA1 Message Date
mvalentsev 93bdeeeec0 perf(mcp): cache the HNSW capacity probe, invalidated by file signature (#1471)
hnsw_capacity_status() runs on every search, duplicate check and status
call (MCP and CLI). Each call cost a COUNT(*) over the embeddings table
and a full unpickle of the segment metadata, both scaling with palace
size. Cache the verdict per (palace_path, collection_name) and reuse it
while an (inode, mtime_ns, size) signature over chroma.sqlite3, its -wal
sidecar and index_metadata.pickle is unchanged, so an external write
invalidates it at once rather than after a fixed window. The #1222
divergence guard stays fresh: a mid-probe write or a locked read is
returned but never cached, and the signature brackets the whole probe so
a pickle rewrite during it cannot be pinned as fresh.
2026-07-20 22:14:45 +05:00
rohan richard 54397a5cc0 fix: dedup content hashes per conversation and per wing
normalize() joined every conversation in a Claude.ai privacy export
bundle into one string before it was hashed, so re-exporting the bundle
with one new conversation added changed the whole-file hash and the
existing conversations were re-mined as duplicates. Split normalization
into normalize_conversations() so each conversation can be hashed and
deduped on its own.

Also scope the content-hash map by wing — mining the same transcript
into a second wing is a deliberate re-file, not a duplicate, and was
leaving that wing with only the registry sentinel.
2026-07-20 22:35:21 +05:30
rohan richard 283dae45e3 fix: prevent duplicate drawers when re-mining LLM conversations
Repeated exports from Claude/ChatGPT land the same conversation under a
new filename each run (timestamped bundle, regenerated slug, etc.), so
the existing source_file/mtime skip never recognized it as already
mined and re-filed a duplicate set of drawers. Now content is hashed
after normalization and checked against previously filed hashes, so
the same conversation under a new path is skipped instead of
duplicated.
2026-07-20 17:30:21 +05:30
Sandro da Silva 3791ad17ea feat(hooks): apply daemon write-routing policy 2026-07-17 08:23:36 +00:00
Igor Lins e Silva 5cd961eef9 Merge pull request #2027 from fatkobra/feat/1963-write-routing-policy
feat(routing): add shared daemon write-routing policy
2026-07-16 19:08:03 -03:00
mvalentsev de74cad911 test(mcp): close chroma clients in checkpoint attribution test
The new real-Chroma checkpoint test dropped its create-time client with a
bare del, leaving the per-path SharedSystemClient's SQLite/HNSW handles
open on Windows (#1128). Close both clients so the temp palace is released,
matching the conftest fixture's close-not-del pattern.
2026-07-16 18:08:32 +05:00
mvalentsevandepinethrone c76208507e fix(mcp): preserve agent attribution in mempalace_checkpoint (#2023)
mempalace_checkpoint hard-coded added_by="checkpoint" for every drawer,
dropping the filing agent's identity even though it arrives in the same
call via diary.agent_name. Add an optional top-level added_by parameter
and resolve attribution as explicit > diary agent_name > "checkpoint";
blank/whitespace/non-string values defer to the next source. The value
is declared in the tool schema so tools/call admits it on both stdio and
HTTP transports.

Fixes #2023

Co-Authored-By: epinethrone <172391900+epinethrone@users.noreply.github.com>
2026-07-16 17:27:50 +05:00
Sandro da Silva fa97c93dd7 feat(routing): add shared daemon write-routing policy 2026-07-15 14:38:23 +00:00
Igor Lins e Silva f62ce2696b test: cover main hotfix regressions 2026-07-14 21:52:32 -03:00
Igor Lins e Silva 68edb110a1 Merge pull request #2018 from MemPalace/agent/fix-explicit-palace-graph-scope
fix: scope derived graph state to explicit palace
2026-07-14 20:28:43 -03:00
Igor Lins e Silva a824be43c8 Merge pull request #2017 from MemPalace/agent/fix-sqlite-recovery-cleanup
fix(repair): require clean SQLite recovery finalization
2026-07-14 20:28:27 -03:00
Igor Lins e Silva 88e454caed Merge pull request #2016 from MemPalace/agent/fix-cli-divergence-preopen
fix(searcher): guard CLI divergence before Chroma open
2026-07-14 20:28:03 -03:00
Igor Lins e Silva 287d8f1b4d fix: scope hallway reads and deletion to selected palace 2026-07-14 19:40:29 -03:00
Igor Lins e Silva 0955100a1f fix(repair): preserve strict cleanup exceptions 2026-07-14 19:37:32 -03:00
Igor Lins e Silva 8a87fc66e2 fix(searcher): preserve backend diagnostics before HNSW probe 2026-07-14 19:37:32 -03:00
Igor Lins e Silva f68cb8d682 fix: scope derived graph state to explicit palace 2026-07-14 19:22:06 -03:00
Igor Lins e SilvaandAaron Zendejas acd62442be fix(repair): require clean SQLite recovery finalization
Co-authored-by: Aaron Zendejas <phileokairos@gmail.com>
2026-07-14 19:12:56 -03:00
Igor Lins e SilvaandTom Boucher 2c6cecf1fe fix(searcher): guard CLI divergence before Chroma open
Co-authored-by: Tom Boucher <trekkie@nomorestars.com>
2026-07-14 19:10:02 -03:00
Igor Lins e Silva 7267de2b32 fix(repair): wait out transient SQLite contention 2026-07-14 19:07:36 -03:00
Igor Lins e Silva 32e545eaaa Merge pull request #2004 from mvalentsev/fix/2003-http-client-disconnect
fix(mcp): don't log a traceback when an HTTP client disconnects mid-response (#2003)
2026-07-14 18:49:46 -03:00
Igor Lins e Silva 28aa2c49b0 Merge pull request #1928 from jrzmurray/fix/automine-fts5-autoheal
mine: auto-heal isolated FTS5 corruption instead of forcing manual repair
2026-07-14 18:47:51 -03:00
Igor Lins e Silva 88e839bce5 Merge pull request #1987 from KeilerHirsch/fix/mcp-async-startup-preflight-dev
fix(mcp): answer initialize immediately — run startup preflight in a background thread
2026-07-14 18:44:48 -03:00
Igor Lins e Silva 0b04b79869 Merge pull request #2010 from KeilerHirsch/fix/convo-skip-tool-results
fix(convo_miner): don't mine Claude Code tool-results/ dirs — raw tool dumps flood the palace
2026-07-14 18:42:26 -03:00
KeilerHirsch 5fdce3923e style: ruff format test_convo_miner_unit.py (CI lint) 2026-07-14 16:43:05 +02:00
mvalentsev 3c0ecb5b3e fix(mcp): don't log a traceback when an HTTP client disconnects mid-response
On the HTTP transport, a client that hangs up mid-response makes the send
path raise an unhandled BrokenPipeError / ConnectionResetError, so the
default socketserver handle_error logs a full traceback for a routine
disconnect. Override handle_error on the HTTP server to log the disconnect
(ConnectionError, plus ssl.SSLEOFError over TLS) at DEBUG and drop it, while
every other exception still reaches the default handler with its traceback.
The wider ssl.SSLError tree is left uncaught, so genuine TLS handshake/cert
failures still surface.

Addresses the disconnect half of #2003.
2026-07-14 13:59:49 +05:00
KeilerHirsch 7f07b0d4fa fix(convo_miner): scan_convos no longer descends into Claude Code tool-results dirs
Claude Code pages large tool outputs to <session>/tool-results/*.txt
inside ~/.claude/projects/<slug>/. These are raw machine dumps (command
output, MCP results) referenced from the transcript JSONL — not
conversations. The convo scanner picked them up as plain .txt, so every
transcript ingest stored megabytes of tool output as memories: on one
field palace, 12,825 drawers traced back to tool-results files, with a
single paged file producing 3,611 drawers.

Adds CONVO_SKIP_DIRS = SKIP_DIRS | {'tool-results'} used only by
scan_convos — project-mining semantics are unchanged, and regular nested
conversation dirs still scan (covered by a companion test).
2026-07-14 10:13:10 +02:00
mvalentsev 4b98dd796d fix(palace): pair the mine_palace_lock holder-set update with its release
_mark_held(palace_key) ran before the try: whose finally runs
_mark_released(). An async exception (SIGINT/KeyboardInterrupt) landing
after _mark_held() and before the try: skips _mark_released(), stranding
the key in the process-wide _palace_lock_keys set while the outer finally
frees the flock. The in-memory hold then outlives the OS lock: a later
re-entrant acquire passes through and writes without the flock while
another process can acquire it, i.e. two writers into one palace.

Move _mark_held() inside the try so it pairs with _mark_released() on
every exit. Add a regression test that injects the interrupt in the
window and asserts the holder set is not stranded.
2026-07-11 15:21:41 +05:00
KeilerHirsch 99464572f4 style: ruff format test file 2026-07-10 17:04:33 +02:00
Igor Lins e Silva 5b1c32eee1 Merge pull request #1953 from jrzmurray/rebuild-1213
feat: add exclude_patterns config key to mempalace.yaml (rebase of #1213)
2026-07-10 09:11:10 -03:00
KeilerHirsch e360a3a040 fix(mcp): answer initialize immediately — run startup preflight in a background thread
The stdio loop ran _refresh_sqlite_integrity_status() and
_refresh_vector_disabled_flag() before reading the first request.
PRAGMA quick_check reads every page of chroma.sqlite3, so on multi-GB
palaces the probe alone (measured: 20.3s on a 1.72 GB / 326k-drawer
palace, 40-46s under disk/lock contention) starves the MCP client's
60s connect timeout — even though the initialize response itself never
touches the database. The HTTP transport already starts without the
synchronous probe.

Move both probes to a daemon thread (mcp-startup-preflight). The #1222
intent is preserved: the probe still starts at startup and logs its
warning as soon as it finishes. Consumers that need the verdict
(_ensure_sqlite_integrity_status via the tool-call integrity gate)
serialize on a new _sqlite_integrity_refresh_lock with double-checked
locking, so a tool call arriving mid-probe waits for the in-flight
verdict instead of running a second O(database size) quick_check —
and never proceeds unverified.

Measured on the 1.72 GB palace with the >512 MB startup gate disabled
(MEMPALACE_STARTUP_INTEGRITY_MAX_MB=0, full quick_check in flight):
initialize 1.4s (was 20-46s); first tool call after probe completion
3.4s with sqlite_integrity checked=true ok=true.

Complements c54531a: the oversized-palace skip still applies to the
background probe, but the handshake no longer depends on it.
2026-07-10 14:10:43 +02:00
Igor Lins e Silva d7819d4b85 Merge pull request #1965 from Evgen197310/fix/test-format-mcp-writer-self-heal
style(tests): fix ruff format check left red by #1960
2026-07-10 09:08:41 -03:00
Igor Lins e Silva 18a9788961 Merge pull request #1899 from zc277584121/milvus-backend-v2
Add Milvus storage backend
2026-07-08 12:27:46 -03:00
Igor Lins e Silva 63188686e5 Merge pull request #1945 from KeilerHirsch/fix/repair-atomic-archive-rename
fix(repair): use os.rename for in-place archive, not shutil.move
2026-07-08 07:25:51 -03:00
Evgen197310andClaude Opus 4.8 09aa4034a7 style(tests): satisfy ruff format check for peer-writer self-heal test
PR #1960 merged with a red `lint` job: `ruff format --check .` wanted to
collapse the multi-line `MineAlreadyRunning(...)` raise in the new
`test_peer_writer_readonly_self_heals_after_peer_exits` onto one line
(it fits the line-length limit). All six real test jobs passed; only the
formatter check failed, which left `develop` red on lint.

Reformat that one statement so `ruff format --check .` is clean again.
No logic change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-08 11:00:03 +03:00
Igor Lins e Silva a20d769b8c Merge pull request #1960 from Evgen197310/fix/mcp-writer-readonly-self-heal
fix(mcp): self-heal writer lease instead of latching read-only for life
2026-07-08 00:33:35 -03:00
Igor Lins e Silva b0b028170a Merge pull request #1957 from jrzmurray/pr-mutable-transcripts
fix(convo_miner): treat transcripts as mutable, not immutable
2026-07-08 00:33:21 -03:00
Evgen197310andClaude Opus 4.8 9e871b45b5 fix(mcp): self-heal writer lease instead of latching read-only for life
The #1818 peer-writer guard latched _MCP_WRITER_READ_ONLY=True on the first
MineAlreadyRunning and short-circuited every subsequent acquisition attempt,
so a server that came up read-only (a peer held the per-palace flock at
startup) stayed read-only for its entire process lifetime — even long after
the peer exited and the OS released the flock. In the common case of several
overlapping Claude sessions (one server per session, all on the same palace),
whichever session started second was stranded: mutating tools kept refusing
with -32001 and the only remedy was killing/restarting that server.

_mcp_peer_writer_refusal already calls _acquire_mcp_writer_lock() on every
mutating tool, so the retry hook existed — the sticky latch just suppressed it.
Drop the read-only short-circuit: when read-only we now re-attempt the
non-blocking flock each call and transparently promote to writer once the peer
is gone. Race-safe — fcntl LOCK_NB is kernel-arbitrated, so two servers can
never both win. The genuinely-broken-lock path (_MCP_WRITER_LOCK_FAILED) is
still cached, since retrying a broken lock mechanism can't help.

Adds test_peer_writer_readonly_self_heals_after_peer_exits.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-08 00:41:17 +03:00
J.R. Murray 9b432699ce fix(convo_miner): treat transcripts as mutable, not immutable
Conversation transcripts were assumed immutable once mined: the bulk
skip-check (prefetch_mined_set) only tracked "have we seen this
source_file before at the current normalize_version", with no mtime
comparison at all. That's wrong for how Claude Code sessions actually
work -- a session keeps appending to its own JSONL file while active,
and /compact or /clear can rewrite one in place. Once a session file
was mined, any content appended after that point would silently never
get mined, with no error or warning -- the file just looked
"already filed" forever.

palace.py:
- prefetch_mined_set() now returns dict[source_file, stored_mtime]
  instead of a bare set[source_file]. `if src in mined_set` still works
  identically (dict `in` checks keys), so this is a source-compatible
  change for that access pattern; a caller that wants staleness
  detection reads mined_set[src] and compares against the file's
  current mtime itself. None means no mtime was ever stored (or
  getmtime failed when the drawer was written) and must be treated as
  stale, not "unknown, assume unchanged".
- Removed bulk_check_mined(): it already existed for exactly this
  purpose (bulk mtime prefetch) but had zero callers anywhere in the
  codebase and was missing the normalize_version/extract_mode filtering
  prefetch_mined_set has -- folded its intent into prefetch_mined_set
  instead of maintaining two subtly-different, overlapping bulk scans
  over the same underlying data.
- file_already_mined()'s docstring corrected: it previously claimed
  "transcripts are assumed immutable" for convo mining. That's no
  longer true; corrected to describe the actual current split (convo
  miner's bulk skip-check uses prefetch_mined_set's stored mtimes; this
  function's check_mtime=True path is now only its per-file,
  lock-held race-condition recheck).

convo_miner.py:
- New _is_unchanged_since_last_mine() helper (extracted to keep
  _mine_convos_impl under the repo's cyclomatic-complexity gate):
  false whenever the file isn't in the prefetched map, its stored mtime
  is None, getmtime fails, or the mtimes don't match -- true only when
  genuinely unchanged.
- _file_chunks_locked's metadata now stamps source_mtime on every real
  drawer (mirroring miner.py's existing pattern), and its in-lock
  recheck now passes check_mtime=True.
- _register_file's 0-chunk sentinel also stamps source_mtime, so a
  short file that later grows past MIN_CHUNK_SIZE is detected as
  changed instead of being skipped forever by the sentinel.

One-time cost worth flagging: no existing convo drawer has source_mtime
stored (this field never existed for convo mining before now), so the
first `mempalace mine --mode convos` after this ships will see every
already-mined file as stale and fully re-mine it. Not a bug --
_file_chunks_locked's existing purge-before-insert means no
duplication results -- just a real, one-time cost across a large
corpus.

tests/test_convo_miner.py: 7 new tests -- grown-file re-mine picks up
new content, unchanged file still skipped (the mtime check must not
regress the existing optimization), grown-file re-mine purges stale
drawers rather than accumulating duplicates (checked via unique content
markers, not raw counts -- ChromaDB collections can carry unrelated
bookkeeping rows), prefetch_mined_set's returned mtime matches the real
file, None handling for a drawer with no stored mtime, a legacy
drawer (no source_mtime field, simulating pre-this-change data) is
correctly re-mined rather than skipped forever, and the sentinel path
stamps source_mtime too.

Full suite: 3327 passed, 20 skipped, 0 failed. ruff check / ruff
format -- clean.
2026-07-07 16:41:22 -04:00
Lochness b70f06a69e feat: add exclude_patterns config key to mempalace.yaml
Allow projects to specify .gitignore-style patterns that the miner should
skip, without relying on .gitignore for mining control.

A new optional exclude_patterns list in mempalace.yaml is parsed by the
existing GitignoreMatcher class via a new from_patterns() classmethod —
same syntax, same semantics as .gitignore, no new dependency.

  exclude_patterns:
    - '*.md'
    - '*.yaml'
    - 'docs/'        # dir-only: prunes entire tree without descending
    - 'dist/'
    - 'coverage/'

Key behaviour:
- Patterns follow .gitignore rules: anchoring (/pattern), dir-only
- dirs[:] pruning via GitignoreMatcher.matches(..., is_dir=True) so
  excluded subtrees are never walked
- Checked after .gitignore filtering; force_include (--include-ignored)
  bypasses exclude_patterns
- Pre-scanned files lists (init double-scan optimisation) are filtered too
- Backwards compatible: omitting exclude_patterns changes nothing

Changes:
- GitignoreMatcher.from_patterns(): new classmethod, same rule parser as
  from_dir(), reads from a list instead of a file on disk
- scan_project(): builds one exclude_matcher before os.walk; used for
  both dirs[:] pruning and per-file filtering
- _mine_impl(): applies the exclude matcher to pre-scanned files lists
  when provided by the caller
- tests/test_miner.py: three new tests
    test_scan_project_exclude_patterns_skips_matching_files
    test_scan_project_exclude_patterns_prunes_entire_directory
    test_scan_project_exclude_patterns_include_ignored_bypasses_exclusion
2026-07-07 15:29:30 -04:00
J.R. Murray da32e562f7 fix(mine): address review feedback on FTS5 auto-heal (#1928)
Two review comments on this PR, both addressed:

- gemini-code-assist flagged that maybe_autoheal_fts5_index's default
  progress=print goes straight to stdout. _validate_palace_fts5_after_mine
  runs inside the MCP server process too (mcp_server.tool_mine ->
  miner.mine), where stdout is the JSON-RPC transport -- a stray print()
  there would corrupt the protocol stream and crash the connection. Pass
  progress=logger.info instead; palace.py already has the module logger.

- nikkunikku corroborated the fix from a real 1.4GB production palace (278
  repeated abort-loop iterations before the fix) and pointed out a real
  test gap: the fixture-based auto-heal tests fabricate real FTS5
  corruption via direct shadow-table writes, which some SQLite builds
  refuse (existing pytest.skip paths in test_miner_fts5_validation.py,
  related to #1925) -- so on those builds the auto-heal wiring in
  _validate_palace_fts5_after_mine is never actually exercised. Added
  their suggested build-independent tests, adapted to this file's fixture
  helpers: test_validator_suppresses_raise_when_autoheal_clears and
  test_validator_still_raises_when_autoheal_cannot_clear, stubbing
  mempalace.repair.sqlite_integrity_errors/maybe_autoheal_fts5_index
  directly instead of fabricating corruption.

Added a third test, test_validator_passes_logger_progress_not_print_to_autoheal,
covering the specific progress= wiring: the two tests above mock
maybe_autoheal_fts5_index entirely and discard its kwargs, so neither would
have caught the progress=print regression this commit actually fixes. The
new test captures the real kwargs and asserts progress is a bound method of
palace.py's own logger (not print), without pinning to logger.info
specifically -- severity level is a verbosity choice, not a correctness
requirement, so the assertion shouldn't fail on a reasonable future change
to e.g. logger.debug. Verified both directions: fails against the
pre-fix `print` default (and shows the leaked stdout line to prove it),
passes at .info and at .debug alike.

Full suite: 3221 passed, 20 skipped (unchanged skip count). ruff
check/format clean.
2026-07-07 13:23:59 -04:00
mvalentsevandZoz92 1d38ae8784 fix(mcp): mark sqlite_integrity not-applicable on non-chroma backends (#1931)
mempalace_status reported a passing SQLite integrity check on non-chroma
backends (checked/ok true, sqlite_path pointing at a chroma.sqlite3 that does
not exist) even though _refresh_sqlite_integrity_status short-circuits the
check there. _sqlite_integrity_payload now reports the check as not-applicable
(checked false, ok null, reason) for non-chroma backends, keeping the chroma
payload shape and error surfacing unchanged.

Co-Authored-By: Zoz92 <66385795+Zoz92@users.noreply.github.com>
2026-07-07 13:52:44 +05:00
KeilerHirsch 07182d5adf fix(repair): use os.rename for in-place archive, not shutil.move
shutil.move's fallback for a failed os.rename is copytree + rmtree. On
Windows, when any file inside the palace is held open by another
process (a live MCP server, a running mine, another harness), the
rename fails and shutil.move falls back to deleting the live palace
file-by-file via rmtree -- which itself then fails partway through on
the first locked file, leaving the palace partially gutted next to a
partial (or empty) archive copy.

Reproduced live twice (Windows 11, 2026-07-05 and 2026-07-06): running
`mempalace repair --mode from-sqlite --yes --archive-existing` while
an MCP server / detached mine held palace/*/data_level0.bin open threw
mid-rmtree in both cases. The palace itself survived only because the
specific locked files could not be unlinked -- a different lock
pattern (e.g. a lock on a file rmtree reaches first) would have lost
data with no way back.

os.rename is atomic on both platforms it matters on (POSIX rename(2),
Windows MoveFileEx) -- it either fully succeeds or fails without
touching anything. Catch the failure and abort cleanly with actionable
guidance instead of a raw traceback.
2026-07-07 09:38:03 +02:00
Igor Lins e Silva b459293e6f fix: keep status from taking writer lease 2026-07-06 09:26:16 -03:00
Grace Gettert 9815f0ae23 fix: half-open as-of interval for KG supersession
Fixes #1913.\n\nVerified locally on Windows with focused knowledge graph/MCP KG tests plus ruff check and ruff format --check.
2026-07-06 09:23:46 -03:00
J.R. Murray 4e20ad3e8c fix: sanitize embedded NUL bytes before they reach ChromaDB
Fixes #1927.\n\nVerified locally on Windows with targeted NUL/surrogate/miner/FTS5 tests plus ruff check and ruff format --check.
2026-07-06 09:23:35 -03:00
Igor Lins e Silva 65bedc2b33 Merge pull request #1911 from trek-e/fix/startup-integrity-nonblocking
fix: skip startup SQLite integrity check on oversized palace (unblocks MCP handshake)
2026-07-06 09:07:38 -03:00
Igor Lins e Silva f820b73160 Merge pull request #1898 from messelink/fix/embedding-wrapper-facet-counts-forwarder
fix(backends): forward facet_counts + get_all_metadata in EmbeddingCollection
2026-07-06 08:19:15 -03:00
Igor Lins e Silva aff1ee8255 Merge pull request #924 from mvalentsev/fix/mine-log-oversized-skips
fix(mine): log warning when files exceed MAX_FILE_SIZE (#923)
2026-07-06 08:18:55 -03:00
J.R. Murray f9db9b726f fix(repair): recognize newer SQLite FTS5 corruption message wording
_errors_are_isolated_fts5 gated auto-heal on one specific message shape:

    malformed inverted index for FTS5 table

SQLite >= ~3.5x (confirmed on 3.53.2 / Python 3.13.7) reports the same
isolated-FTS5 condition with different wording instead:

    fts5: corruption found reading blob N from table "embedding_fulltext_search"

The narrow regex never matched this phrasing, so maybe_autoheal_fts5_index
silently declined to heal on any machine running a recent-enough SQLite,
falling straight through to the hard-abort path -- the exact condition
the whole auto-heal feature (#1926/#1928) exists to avoid. Widened the
pattern to match either wording.

Caught by running this repo's own test suite on this machine:
test_repair.py's two auto-heal tests were failing (not, as assumed
earlier, pre-existing/unrelated flakiness -- that assumption was never
actually verified). Traced to this exact classification gap.

Fixing this correctly also exposed that four tests in
test_miner_fts5_validation.py had been passing for the wrong reason: they
manufacture the exact "reporter-shaped" isolated-FTS5 corruption (#1926's
actual bug shape) and asserted mine() must raise MineValidationError for
it -- true only because the classifier bug prevented auto-heal from ever
engaging. With the classifier fixed, that corruption is now correctly
auto-healed and mine() succeeds instead, so those tests' expectations
were stale, not their fixtures being invalid:

- test_helper_raises_on_fts5_segment_corruption -> renamed
  test_helper_auto_heals_fts5_segment_corruption; asserts no raise + a
  clean post-heal quick_check, instead of expecting a raise.
- test_full_chain_raises_through_mine_impl and
  test_mine_impl_does_not_print_partial_summary_on_validation_error: their
  real purpose is exception-passthrough / banner-suppression when the
  validator DOES raise, not proving any particular corruption triggers it.
  Switched from real file corruption to a monkeypatched raise. (Tried
  swapping to _page_mangle's non-isolated corruption first -- that made
  ChromaDB's own Rust bindings panic just opening the file for the
  re-mine's get_collection() call, a native crash rather than a catchable
  Python exception, before the validator ever ran. Different failure mode
  than what these tests are about, and not reliable to depend on.)
- test_mine_formats_full_chain_raises_when_fts5_corrupt: same fix, mirrors
  the miner-path change for the extract path.
- Added test_full_chain_auto_heals_isolated_fts5_corruption and
  test_mine_formats_full_chain_auto_heals_isolated_fts5_corruption as
  companions, proving the full mine()/mine_formats() chain -- not just
  the standalone validator -- actually auto-heals and succeeds end-to-end
  for the isolated case now that it's correctly classified.
- test_errors_are_isolated_fts5_classification: added the new message
  wording as an explicit regression fixture (pinned literally, not
  dependent on whatever this machine's SQLite happens to emit).

Full suite: 3302 passed, 20 skipped, 0 failed -- first fully clean run
this session. ruff check / ruff format -- clean.
2026-07-05 17:35:01 -04:00
Tom BoucherandClaude Opus 4.8 c54531a2a2 fix: skip startup SQLite integrity check on oversized palace
The MCP server ran PRAGMA quick_check on the full chroma.sqlite3 during
startup, before answering the initialize handshake. quick_check is
O(database size); on multi-GB palaces it exceeds the MCP client's ~30s
connection timeout, so the server never finishes starting and the client
drops the connection (observed >2min on a 4.6GB palace).

Skip the startup probe when chroma.sqlite3 exceeds
MEMPALACE_STARTUP_INTEGRITY_MAX_MB (default 512MB; 0 disables). The gate
lives in _refresh_sqlite_integrity_status, the single choke point for the
startup calls and every lazy consumer. `mempalace repair` preflight still
runs the full quick_check via repair.sqlite_integrity_errors, so
SQLite-layer corruption is still caught on the destructive path.

Refs #1818.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Jq495N7e2D4wY2Mp2AQvg7
2026-07-02 12:34:56 -04:00