Commit Graph
1588 Commits
Author SHA1 Message Date
Igor Lins e Silva b0b028170a Merge pull request #1957 from jrzmurray/pr-mutable-transcripts
fix(convo_miner): treat transcripts as mutable, not immutable
2026-07-08 00:33:21 -03:00
Igor Lins e Silva c0ae2abaff Merge pull request #1946 from mvalentsev/fix/status-integrity-non-chroma
fix(mcp): mark sqlite_integrity not-applicable on non-chroma backends (#1931)
2026-07-08 00:32:56 -03:00
Evgen197310andClaude Opus 4.8 9e871b45b5 fix(mcp): self-heal writer lease instead of latching read-only for life
The #1818 peer-writer guard latched _MCP_WRITER_READ_ONLY=True on the first
MineAlreadyRunning and short-circuited every subsequent acquisition attempt,
so a server that came up read-only (a peer held the per-palace flock at
startup) stayed read-only for its entire process lifetime — even long after
the peer exited and the OS released the flock. In the common case of several
overlapping Claude sessions (one server per session, all on the same palace),
whichever session started second was stranded: mutating tools kept refusing
with -32001 and the only remedy was killing/restarting that server.

_mcp_peer_writer_refusal already calls _acquire_mcp_writer_lock() on every
mutating tool, so the retry hook existed — the sticky latch just suppressed it.
Drop the read-only short-circuit: when read-only we now re-attempt the
non-blocking flock each call and transparently promote to writer once the peer
is gone. Race-safe — fcntl LOCK_NB is kernel-arbitrated, so two servers can
never both win. The genuinely-broken-lock path (_MCP_WRITER_LOCK_FAILED) is
still cached, since retrying a broken lock mechanism can't help.

Adds test_peer_writer_readonly_self_heals_after_peer_exits.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-08 00:41:17 +03:00
J.R. Murray 9b432699ce fix(convo_miner): treat transcripts as mutable, not immutable
Conversation transcripts were assumed immutable once mined: the bulk
skip-check (prefetch_mined_set) only tracked "have we seen this
source_file before at the current normalize_version", with no mtime
comparison at all. That's wrong for how Claude Code sessions actually
work -- a session keeps appending to its own JSONL file while active,
and /compact or /clear can rewrite one in place. Once a session file
was mined, any content appended after that point would silently never
get mined, with no error or warning -- the file just looked
"already filed" forever.

palace.py:
- prefetch_mined_set() now returns dict[source_file, stored_mtime]
  instead of a bare set[source_file]. `if src in mined_set` still works
  identically (dict `in` checks keys), so this is a source-compatible
  change for that access pattern; a caller that wants staleness
  detection reads mined_set[src] and compares against the file's
  current mtime itself. None means no mtime was ever stored (or
  getmtime failed when the drawer was written) and must be treated as
  stale, not "unknown, assume unchanged".
- Removed bulk_check_mined(): it already existed for exactly this
  purpose (bulk mtime prefetch) but had zero callers anywhere in the
  codebase and was missing the normalize_version/extract_mode filtering
  prefetch_mined_set has -- folded its intent into prefetch_mined_set
  instead of maintaining two subtly-different, overlapping bulk scans
  over the same underlying data.
- file_already_mined()'s docstring corrected: it previously claimed
  "transcripts are assumed immutable" for convo mining. That's no
  longer true; corrected to describe the actual current split (convo
  miner's bulk skip-check uses prefetch_mined_set's stored mtimes; this
  function's check_mtime=True path is now only its per-file,
  lock-held race-condition recheck).

convo_miner.py:
- New _is_unchanged_since_last_mine() helper (extracted to keep
  _mine_convos_impl under the repo's cyclomatic-complexity gate):
  false whenever the file isn't in the prefetched map, its stored mtime
  is None, getmtime fails, or the mtimes don't match -- true only when
  genuinely unchanged.
- _file_chunks_locked's metadata now stamps source_mtime on every real
  drawer (mirroring miner.py's existing pattern), and its in-lock
  recheck now passes check_mtime=True.
- _register_file's 0-chunk sentinel also stamps source_mtime, so a
  short file that later grows past MIN_CHUNK_SIZE is detected as
  changed instead of being skipped forever by the sentinel.

One-time cost worth flagging: no existing convo drawer has source_mtime
stored (this field never existed for convo mining before now), so the
first `mempalace mine --mode convos` after this ships will see every
already-mined file as stale and fully re-mine it. Not a bug --
_file_chunks_locked's existing purge-before-insert means no
duplication results -- just a real, one-time cost across a large
corpus.

tests/test_convo_miner.py: 7 new tests -- grown-file re-mine picks up
new content, unchanged file still skipped (the mtime check must not
regress the existing optimization), grown-file re-mine purges stale
drawers rather than accumulating duplicates (checked via unique content
markers, not raw counts -- ChromaDB collections can carry unrelated
bookkeeping rows), prefetch_mined_set's returned mtime matches the real
file, None handling for a drawer with no stored mtime, a legacy
drawer (no source_mtime field, simulating pre-this-change data) is
correctly re-mined rather than skipped forever, and the sentinel path
stamps source_mtime too.

Full suite: 3327 passed, 20 skipped, 0 failed. ruff check / ruff
format -- clean.
2026-07-07 16:41:22 -04:00
J.R. Murray 3f49e15545 refactor(miner): extract exclude_patterns prescan filter into a helper
Rebasing Lochness's exclude_patterns work (#1213) onto current develop
pushed _mine_impl's cyclomatic complexity to 26, tripping the repo's
max-complexity=25 ruff gate (clean on develop before this rebase).
Extracted the pre-scanned-file-list filtering branch into
_apply_exclude_patterns_to_prescanned_files -- same behavior, no test
changes needed, complexity back under the gate.
2026-07-07 15:29:42 -04:00
Lochness b70f06a69e feat: add exclude_patterns config key to mempalace.yaml
Allow projects to specify .gitignore-style patterns that the miner should
skip, without relying on .gitignore for mining control.

A new optional exclude_patterns list in mempalace.yaml is parsed by the
existing GitignoreMatcher class via a new from_patterns() classmethod —
same syntax, same semantics as .gitignore, no new dependency.

  exclude_patterns:
    - '*.md'
    - '*.yaml'
    - 'docs/'        # dir-only: prunes entire tree without descending
    - 'dist/'
    - 'coverage/'

Key behaviour:
- Patterns follow .gitignore rules: anchoring (/pattern), dir-only
- dirs[:] pruning via GitignoreMatcher.matches(..., is_dir=True) so
  excluded subtrees are never walked
- Checked after .gitignore filtering; force_include (--include-ignored)
  bypasses exclude_patterns
- Pre-scanned files lists (init double-scan optimisation) are filtered too
- Backwards compatible: omitting exclude_patterns changes nothing

Changes:
- GitignoreMatcher.from_patterns(): new classmethod, same rule parser as
  from_dir(), reads from a list instead of a file on disk
- scan_project(): builds one exclude_matcher before os.walk; used for
  both dirs[:] pruning and per-file filtering
- _mine_impl(): applies the exclude matcher to pre-scanned files lists
  when provided by the caller
- tests/test_miner.py: three new tests
    test_scan_project_exclude_patterns_skips_matching_files
    test_scan_project_exclude_patterns_prunes_entire_directory
    test_scan_project_exclude_patterns_include_ignored_bypasses_exclusion
2026-07-07 15:29:30 -04:00
J.R. Murray da32e562f7 fix(mine): address review feedback on FTS5 auto-heal (#1928)
Two review comments on this PR, both addressed:

- gemini-code-assist flagged that maybe_autoheal_fts5_index's default
  progress=print goes straight to stdout. _validate_palace_fts5_after_mine
  runs inside the MCP server process too (mcp_server.tool_mine ->
  miner.mine), where stdout is the JSON-RPC transport -- a stray print()
  there would corrupt the protocol stream and crash the connection. Pass
  progress=logger.info instead; palace.py already has the module logger.

- nikkunikku corroborated the fix from a real 1.4GB production palace (278
  repeated abort-loop iterations before the fix) and pointed out a real
  test gap: the fixture-based auto-heal tests fabricate real FTS5
  corruption via direct shadow-table writes, which some SQLite builds
  refuse (existing pytest.skip paths in test_miner_fts5_validation.py,
  related to #1925) -- so on those builds the auto-heal wiring in
  _validate_palace_fts5_after_mine is never actually exercised. Added
  their suggested build-independent tests, adapted to this file's fixture
  helpers: test_validator_suppresses_raise_when_autoheal_clears and
  test_validator_still_raises_when_autoheal_cannot_clear, stubbing
  mempalace.repair.sqlite_integrity_errors/maybe_autoheal_fts5_index
  directly instead of fabricating corruption.

Added a third test, test_validator_passes_logger_progress_not_print_to_autoheal,
covering the specific progress= wiring: the two tests above mock
maybe_autoheal_fts5_index entirely and discard its kwargs, so neither would
have caught the progress=print regression this commit actually fixes. The
new test captures the real kwargs and asserts progress is a bound method of
palace.py's own logger (not print), without pinning to logger.info
specifically -- severity level is a verbosity choice, not a correctness
requirement, so the assertion shouldn't fail on a reasonable future change
to e.g. logger.debug. Verified both directions: fails against the
pre-fix `print` default (and shows the leaked stdout line to prove it),
passes at .info and at .debug alike.

Full suite: 3221 passed, 20 skipped (unchanged skip count). ruff
check/format clean.
2026-07-07 13:23:59 -04:00
mvalentsevandZoz92 1d38ae8784 fix(mcp): mark sqlite_integrity not-applicable on non-chroma backends (#1931)
mempalace_status reported a passing SQLite integrity check on non-chroma
backends (checked/ok true, sqlite_path pointing at a chroma.sqlite3 that does
not exist) even though _refresh_sqlite_integrity_status short-circuits the
check there. _sqlite_integrity_payload now reports the check as not-applicable
(checked false, ok null, reason) for non-chroma backends, keeping the chroma
payload shape and error surfacing unchanged.

Co-Authored-By: Zoz92 <66385795+Zoz92@users.noreply.github.com>
2026-07-07 13:52:44 +05:00
KeilerHirsch 07182d5adf fix(repair): use os.rename for in-place archive, not shutil.move
shutil.move's fallback for a failed os.rename is copytree + rmtree. On
Windows, when any file inside the palace is held open by another
process (a live MCP server, a running mine, another harness), the
rename fails and shutil.move falls back to deleting the live palace
file-by-file via rmtree -- which itself then fails partway through on
the first locked file, leaving the palace partially gutted next to a
partial (or empty) archive copy.

Reproduced live twice (Windows 11, 2026-07-05 and 2026-07-06): running
`mempalace repair --mode from-sqlite --yes --archive-existing` while
an MCP server / detached mine held palace/*/data_level0.bin open threw
mid-rmtree in both cases. The palace itself survived only because the
specific locked files could not be unlinked -- a different lock
pattern (e.g. a lock on a file rmtree reaches first) would have lost
data with no way back.

os.rename is atomic on both platforms it matters on (POSIX rename(2),
Windows MoveFileEx) -- it either fully succeeds or fails without
touching anything. Catch the failure and abort cleanly with actionable
guidance instead of a raw traceback.
2026-07-07 09:38:03 +02:00
Igor Lins e Silva c00e8dcc16 fix: keep status from taking writer lease (#1934) 2026-07-06 18:16:11 -03:00
Igor Lins e Silva b459293e6f fix: keep status from taking writer lease 2026-07-06 09:26:16 -03:00
Grace Gettert 9815f0ae23 fix: half-open as-of interval for KG supersession
Fixes #1913.\n\nVerified locally on Windows with focused knowledge graph/MCP KG tests plus ruff check and ruff format --check.
2026-07-06 09:23:46 -03:00
J.R. Murray 4e20ad3e8c fix: sanitize embedded NUL bytes before they reach ChromaDB
Fixes #1927.\n\nVerified locally on Windows with targeted NUL/surrogate/miner/FTS5 tests plus ruff check and ruff format --check.
2026-07-06 09:23:35 -03:00
Igor Lins e Silva 65bedc2b33 Merge pull request #1911 from trek-e/fix/startup-integrity-nonblocking
fix: skip startup SQLite integrity check on oversized palace (unblocks MCP handshake)
2026-07-06 09:07:38 -03:00
Igor Lins e Silva f820b73160 Merge pull request #1898 from messelink/fix/embedding-wrapper-facet-counts-forwarder
fix(backends): forward facet_counts + get_all_metadata in EmbeddingCollection
2026-07-06 08:19:15 -03:00
Igor Lins e Silva aff1ee8255 Merge pull request #924 from mvalentsev/fix/mine-log-oversized-skips
fix(mine): log warning when files exceed MAX_FILE_SIZE (#923)
2026-07-06 08:18:55 -03:00
Igor Lins e Silva 39bec19956 Merge pull request #1930 from yusefhaddad/fix/read-only-missing-mutating-tools
fix: read-only mode misses checkpoint and delete_by_source in _MUTATING_TOOLS
2026-07-06 08:18:21 -03:00
J.R. Murray f9db9b726f fix(repair): recognize newer SQLite FTS5 corruption message wording
_errors_are_isolated_fts5 gated auto-heal on one specific message shape:

    malformed inverted index for FTS5 table

SQLite >= ~3.5x (confirmed on 3.53.2 / Python 3.13.7) reports the same
isolated-FTS5 condition with different wording instead:

    fts5: corruption found reading blob N from table "embedding_fulltext_search"

The narrow regex never matched this phrasing, so maybe_autoheal_fts5_index
silently declined to heal on any machine running a recent-enough SQLite,
falling straight through to the hard-abort path -- the exact condition
the whole auto-heal feature (#1926/#1928) exists to avoid. Widened the
pattern to match either wording.

Caught by running this repo's own test suite on this machine:
test_repair.py's two auto-heal tests were failing (not, as assumed
earlier, pre-existing/unrelated flakiness -- that assumption was never
actually verified). Traced to this exact classification gap.

Fixing this correctly also exposed that four tests in
test_miner_fts5_validation.py had been passing for the wrong reason: they
manufacture the exact "reporter-shaped" isolated-FTS5 corruption (#1926's
actual bug shape) and asserted mine() must raise MineValidationError for
it -- true only because the classifier bug prevented auto-heal from ever
engaging. With the classifier fixed, that corruption is now correctly
auto-healed and mine() succeeds instead, so those tests' expectations
were stale, not their fixtures being invalid:

- test_helper_raises_on_fts5_segment_corruption -> renamed
  test_helper_auto_heals_fts5_segment_corruption; asserts no raise + a
  clean post-heal quick_check, instead of expecting a raise.
- test_full_chain_raises_through_mine_impl and
  test_mine_impl_does_not_print_partial_summary_on_validation_error: their
  real purpose is exception-passthrough / banner-suppression when the
  validator DOES raise, not proving any particular corruption triggers it.
  Switched from real file corruption to a monkeypatched raise. (Tried
  swapping to _page_mangle's non-isolated corruption first -- that made
  ChromaDB's own Rust bindings panic just opening the file for the
  re-mine's get_collection() call, a native crash rather than a catchable
  Python exception, before the validator ever ran. Different failure mode
  than what these tests are about, and not reliable to depend on.)
- test_mine_formats_full_chain_raises_when_fts5_corrupt: same fix, mirrors
  the miner-path change for the extract path.
- Added test_full_chain_auto_heals_isolated_fts5_corruption and
  test_mine_formats_full_chain_auto_heals_isolated_fts5_corruption as
  companions, proving the full mine()/mine_formats() chain -- not just
  the standalone validator -- actually auto-heals and succeeds end-to-end
  for the isolated case now that it's correctly classified.
- test_errors_are_isolated_fts5_classification: added the new message
  wording as an explicit regression fixture (pinned literally, not
  dependent on whatever this machine's SQLite happens to emit).

Full suite: 3302 passed, 20 skipped, 0 failed -- first fully clean run
this session. ruff check / ruff format -- clean.
2026-07-05 17:35:01 -04:00
Yusef HaddadandClaude Fable 5 578e5775b4 fix: include checkpoint and delete_by_source in _MUTATING_TOOLS
mempalace_checkpoint and mempalace_delete_by_source (added in 3.5.0) were
missing from _MUTATING_TOOLS, so a server started with --read-only /
MEMPALACE_MCP_READ_ONLY=1 still allowed writing drawers and bulk-deleting
by source. The same gap affected the peer-writer lock gate, which uses
the same frozenset.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 00:42:11 -06:00
J.R. MurrayandClaude Sonnet 4.6 adc0ef876f fix: auto-heal isolated FTS5 corruption in mine, not just repair
mempalace mine aborts with the "ABORT: SQLite-layer corruption detected"
banner on an isolated FTS5 inverted-index corruption -- the specific case
maybe_autoheal_fts5_index already exists to fix in place. That helper is
wired into cmd_repair's preflight, but not into mine's own post-mine
validation (palace._validate_palace_fts5_after_mine), so mine forces a
manual `mempalace repair` for a corruption class that's already safely
self-healable.

This wires the same auto-heal call into _validate_palace_fts5_after_mine,
before it raises MineValidationError. maybe_autoheal_fts5_index returns
the *remaining* errors after the heal attempt, so MineValidationError
still raises whenever the corruption isn't the isolated, fully-healable
case -- this only changes behavior when the heal has verifiably and
fully cleared the corruption.

Verified against a real-world repro (mining a real Claude Code project
directory deterministically triggered this corruption after all files
filed successfully, zero concurrency, single uninterrupted process):
mempalace repair --yes confirmed the auto-heal path clears it before
proceeding to a full rebuild. With this patch, mine self-heals the same
case directly -- no abort, Files processed: <n>, Done, PRAGMA quick_check
clean afterward.

Test suite: 3214 passed, 20 skipped -- no new failures. Two pre-existing
unrelated failures in test_repair.py (a SQLite-version-dependent FTS5
corruption message-wording mismatch, tracked separately) reproduce
identically on unmodified develop.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-05 01:26:57 -04:00
Tom BoucherandClaude Opus 4.8 c54531a2a2 fix: skip startup SQLite integrity check on oversized palace
The MCP server ran PRAGMA quick_check on the full chroma.sqlite3 during
startup, before answering the initialize handshake. quick_check is
O(database size); on multi-GB palaces it exceeds the MCP client's ~30s
connection timeout, so the server never finishes starting and the client
drops the connection (observed >2min on a 4.6GB palace).

Skip the startup probe when chroma.sqlite3 exceeds
MEMPALACE_STARTUP_INTEGRITY_MAX_MB (default 512MB; 0 disables). The gate
lives in _refresh_sqlite_integrity_status, the single choke point for the
startup calls and every lazy consumer. `mempalace repair` preflight still
runs the full quick_check via repair.sqlite_integrity_errors, so
SQLite-layer corruption is still caught on the destructive path.

Refs #1818.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Jq495N7e2D4wY2Mp2AQvg7
2026-07-02 12:34:56 -04:00
Cheney Zhang 0104d86c53 fix: address Milvus backend review feedback
Signed-off-by: Cheney Zhang <chen.zhang@zilliz.com>
2026-07-01 07:10:39 +00:00
Cheney Zhang db6c3d6552 fix: refine Milvus backend consistency
Signed-off-by: Cheney Zhang <chen.zhang@zilliz.com>
2026-07-01 06:52:28 +00:00
Cheney Zhang bddbb5f7a5 fix: enable native Milvus Lite lexical search
Signed-off-by: Cheney Zhang <chen.zhang@zilliz.com>
2026-07-01 06:51:35 +00:00
Cheney Zhang 2c81e22e28 fix: use native Milvus lexical search
Signed-off-by: Cheney Zhang <chen.zhang@zilliz.com>
2026-07-01 06:51:35 +00:00
Cheney Zhang 9b395fe9d5 feat: add Milvus backend
Signed-off-by: Cheney Zhang <chen.zhang@zilliz.com>
2026-07-01 06:50:27 +00:00
Igor Lins e Silva da5a48caf5 Merge pull request #1719 from raman325/feat/openclaw-tool-catchup
docs(openclaw): catch up SKILL.md with 8 newer MCP tools
2026-06-30 15:39:55 -03:00
Igor Lins e Silva e096701333 docs(openclaw): document full MCP tool surface 2026-06-30 15:23:21 -03:00
Igor Lins e Silva e84221c6b7 Merge pull request #1901 from messelink/feat/latex-extensions
feat: add LaTeX (.tex, .bib) to readable and prose extensions
2026-06-30 14:38:46 -03:00
Igor Lins e Silva 0549a90df3 Merge pull request #1905 from MemPalace/docs/fix-backend-table-layout
fix(docs): wide-table layout — stop clipping, fit backend table
2026-06-30 14:12:56 -03:00
Igor Lins e Silva 281aaf489d fix(docs): make backend comparison table fit the content column
Browser-validated the table layout across desktop (1280) and mobile (375):

- Denser doc-table cell padding (8px 16px -> 8px 12px) so comparison tables
  fit the content column instead of needing a horizontal scrollbar.
- `overflow-wrap: break-word` on table-cell code so only genuinely long
  values (e.g. a Postgres DSN) wrap, while short identifiers like
  `palace_path` keep natural column sizing and stay on one line.
- Drop the redundant 'Configure with' column from the storage-backends table
  (each backend's connection variables are documented in full in its own
  subsection right below) and shorten 'Local (exact cosine)' -> 'Local
  (exact)'. The comparison table is now five columns and fits cleanly.

Verified no clipping and no page-level horizontal overflow on the
configuration, remote-server, reference (cli/mcp-tools/python-api),
claude-code, and knowledge-graph pages; wide tables scroll within their own
container on mobile.
2026-06-30 08:06:43 -03:00
Igor Lins e Silva 176c8b820b fix(docs): stop wide tables from clipping; slim backend table
The custom theme set `.vp-doc table { overflow: hidden }` to clip its
rounded corners, which also overrode VitePress's default `overflow-x: auto`
— so any table wider than the content column was clipped with no way to
scroll to the hidden columns (visible on the storage-backends table). Switch
to `overflow-x: auto` so wide tables scroll, keeping the rounded corners.

Also shorten the storage-backends table's two capability headers
(Namespace isolation -> Namespaces, Lexical search -> Lexical) so the table
fits the content column without needing the scrollbar.
2026-06-30 07:51:45 -03:00
Igor Lins e Silva 3b8d2652f4 Merge pull request #1904 from MemPalace/docs/storage-backend-config
docs(config): add storage backends configuration reference
2026-06-30 07:39:31 -03:00
Igor Lins e Silva e4924e8e94 docs(config): clarify backend selection vs configuration in table
Rename the table's 'Select with' column to 'Configure with' and list each
backend's primary connection knob, since a connection variable (e.g.
MEMPALACE_QDRANT_URL) configures a backend but does not select it — selection
is uniform via --backend / MEMPALACE_BACKEND, covered in the prose below the
table. Also state the concrete MEMPALACE_QDRANT_TIMEOUT default (10.0s).
2026-06-30 07:32:37 -03:00
Igor Lins e Silva 1577346158 docs(config): add storage backends configuration reference
Establish guide/configuration.md as the canonical home for per-backend
connection settings, with a compatibility table and connection-variable
reference for the chroma, sqlite_exact, qdrant, and pgvector backends.

remote-server.md already links Postgres + pgvector to /guide/configuration,
but the page had no backend section; this populates that target. New backends
add one table row plus a connection subsection, keeping README's compatibility
table in sync rather than accreting a prose paragraph per backend.
2026-06-30 07:12:55 -03:00
Pim MesselinkandClaude Opus 4.7 6cec591c17 feat: add LaTeX (.tex, .bib) to readable and prose extensions
LaTeX source files and BibTeX bibliographies are prose-rich content that
benefits from both palace mining and entity detection. Adds the two
extensions to the two extension lists most relevant to them, each with a
matching test.

- ``mempalace/miner.py:READABLE_EXTENSIONS`` — ``.tex`` / ``.bib`` join the
  mining allowlist (parallel to the Swift/Kotlin PR #1368 and the PHP
  ecosystem PR #1819).

- ``mempalace/entity_detector.py:PROSE_EXTENSIONS`` — ``.tex`` / ``.bib``
  also join the *preferred* entity-detection bucket alongside ``.md`` /
  ``.rst`` / ``.csv``, NOT the broader code-file fallback. The reason
  ``PROSE_EXTENSIONS`` exists separately is documented in-code:
  programming-language files have lots of capitalized identifiers (class
  names, function names) that produce false-positive person matches.
  LaTeX/BibTeX don't have that problem — they're typesetting languages
  for prose documents. ``.bib`` in particular is almost entirely author
  names, one of the highest real-entity densities of any file type the
  detector scans.

Tests follow the patterns established by the prior extension PRs:
``tests/test_miner.py::test_scan_project_includes_latex_files`` mirrors
the Swift/Kotlin scan tests, and
``tests/test_entity_detector.py::test_scan_for_detection_includes_latex_prose``
mirrors ``test_scan_for_detection_finds_prose``. The existing
``test_prose_extensions`` was extended to assert the two new entries.

Full env-cleared suite: 3216 passed, 20 skipped. ``ruff check .`` and
``ruff format --check .`` both clean.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KC5Qsknh2zFRtRvVyjXiTA
2026-06-29 07:25:50 +00:00
Igor Lins e Silva afd0428823 feat(serve): turnkey secure remote MCP server (#1877) (#1900)
* feat(serve): turnkey secure remote MCP server (#1877)

Add `mempalace serve`: a secure-by-default wrapper over the HTTP MCP
transport so a team can stand up a shared central palace with one
command.

Server capabilities (mempalace/mcp_server.py):
- Native TLS via --tls-cert/--tls-key (env MEMPALACE_MCP_TLS_CERT/_KEY):
  wraps the socket in a TLS 1.2+ context, validated before bind. Token
  is still required on a non-loopback bind (TLS != auth).
- Read-only mode via --read-only (env MEMPALACE_MCP_READ_ONLY): the 24
  mutating tools are hidden from tools/list and refused at dispatch
  (-32003), enforced before arg handling — not merely hidden.

Turnkey command (mempalace/cli.py):
- Auto-generates a strong bearer token for non-loopback binds, stored
  0600 under ~/.mempalace/server/ and printed once; reused across
  restarts. Token rides in the child env, never argv, so it can't leak
  via ps.
- Prints a ready-to-paste client config (scheme reflects TLS), then
  foreground-execs the real server so Docker/systemd own the lifecycle.

Deployment (deploy/):
- docker-compose.server.yml wires the server + Qdrant with a /healthz
  healthcheck and persistent volumes.
- server.env.example documents the env surface.
- mempalace-server.service is a hardened systemd unit template.

Tests: TLS handshake (openssl-gated), read-only enforcement, token
autogen/0600/reuse, token-not-in-argv, secure-by-default gates.

Docs: remote-server guide now leads with `mempalace serve` plus Compose
and systemd subsections.

* test(serve): fix Windows — don't patch os.name; gate 0600 asserts to POSIX

Patching os.name to 'posix' broke Path.home() on Windows (pathlib mixed
POSIX home resolution with Windows drive parsing). Capture both exec
branches (os.execve + subprocess.run) instead, and guard the POSIX
permission-bit assertions behind os.name == 'posix' (Windows files
report 0o666).
2026-06-29 03:28:06 -03:00
Pim MesselinkandClaude Opus 4.7 cea780eb79 fix(backends): annotate EmbeddingCollection.facet_counts return type
Per Gemini review (PR #1898 comment r3489013681): the forwarder lacked the
``-> dict[str, int]`` return annotation that ``BaseCollection.facet_counts``
and the sibling ``get_all_metadata`` forwarder both carry. One-line
consistency fix, no behavior change.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KC5Qsknh2zFRtRvVyjXiTA
2026-06-29 02:37:11 +00:00
Pim MesselinkandClaude Opus 4.7 2211c974ff fix(backends): forward facet_counts + get_all_metadata in EmbeddingCollection
Both methods are concrete on ``BaseCollection`` (``facet_counts`` raises
``UnsupportedCapabilityError``; ``get_all_metadata`` pages through
``self.get(include=["metadatas"])``). Python MRO resolves them on
``EmbeddingCollection`` before ``__getattr__`` ever fires, so without an
explicit forwarder the wrapper silently runs the base default instead of
delegating to the wrapped backend's optimized implementation. The pattern
matches the existing explicit forwarders for ``distance_metric``,
``lexical_search``, and the embedder-identity trio — all added to fix the
same shadow.

What this means in production for the three backends that get wrapped
(``EmbeddingCollection`` only applies to ``requires_explicit_embeddings``
backends — qdrant, pgvector, sqlite_exact; chroma is unwrapped and
unaffected):

- **facet_counts shadow (#1868 regression)**: every ``mempalace_status``,
  ``list_wings``, ``list_rooms``, ``get_taxonomy`` call routes through
  the gated ``col.facet_counts(...)`` path. The capability check passes
  (``supports_metadata_facets`` is on the backend), but the call hits the
  wrapper's MRO-resolved ``BaseCollection.facet_counts`` and raises
  ``UnsupportedCapabilityError``. ``mcp_server``'s broad ``except`` swallows
  it, logs ``WARN Failed to fetch metadata facets, falling back to client-
  side loop: backend does not support facet_counts``, and counts via the
  O(n) Python loop — the exact behavior #1868 was designed to eliminate.

- **get_all_metadata shadow (#1796 / #1892 regression)**: the BaseCollection
  default pages through ``self.get(include=["metadatas"])`` — fine for
  Chroma's SQL OFFSET cursor, but on wrapped backends (qdrant, pgvector)
  the inner's overridden ``get_all_metadata`` is unreachable. For pgvector
  specifically, this means #1892's ``with_document=False`` fast path is
  never taken even though it's implemented — every metadata-only fetch
  transfers the full document column over the wire. On a 13k-drawer remote
  pgvector palace over WAN that's ~13MB per call, dominating wall time.

Why no test caught it: backend tests (``test_qdrant_backend.py``,
``test_pgvector_backend.py``) call the methods directly on the raw
collection, not through the wrapper. ``test_mcp_server.py`` facet tests
use ``MagicMock()`` for the collection, which synthesizes attributes on
demand and bypasses MRO entirely. Neither path covers the seam where the
bug lives: ``palace.get_collection() -> EmbeddingCollection -> .method()``.

Three tests pin both the fix and the bug class:

- ``test_facet_counts_forwards_to_inner`` — direct integration through the
  wrapper, asserts the inner's recorded call matches.
- ``test_get_all_metadata_forwards_to_inner`` — same shape, plus a sentinel
  ``get()`` on the inner so a missing forwarder would route to the base
  default and pick up the wrong data (observable failure, not silent).
- ``test_wrapper_forwards_all_concrete_basecollection_methods`` — meta-test
  that enumerates every concrete public method on ``BaseCollection`` via
  ``inspect.getmembers`` and asserts each one is explicitly defined on
  ``EmbeddingCollection``. Catches the bug *class*: any future
  ``BaseCollection`` method with a concrete default body becomes a CI
  failure the moment it's added without a wrapper forwarder, with a message
  pointing straight at the file to edit.

Full env-cleared suite: 3205 passed, 20 skipped. ``ruff check`` and
``ruff format --check`` both clean.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KC5Qsknh2zFRtRvVyjXiTA
2026-06-29 02:22:10 +00:00
Igor Lins e Silva 8ec284db5d docs(guide): add Remote / Team Server deployment guide (#1877) (#1897)
Documents running MemPalace as a central memory service for a team:
HTTP MCP transport (--transport http with bearer-token auth), a
networked backend (Qdrant via REST, no extra dep; or pgvector), and
optional GPU embedding. Covers the security model (non-loopback token
requirement, Host/Origin DNS-rebinding guard, TLS-in-front), client
connection, and operating notes. Adds the page to the guide sidebar.

Addresses #1877.
2026-06-28 20:10:09 -03:00
5dcc46ba7e feat(graph): auto-populate the associative graph from mined sessions (#1895)
* feat(graph): auto-populate the associative graph from mined sessions

Conversation mining never set the `entities` drawer metadata that hallways
consume, so mined sessions produced an empty associative graph (and starved
the entity-navigation / tunnel-recommendation features built on top of it).

Add a no-LLM structural entity extractor and wire it into the convos mine:

- entities: structural-only extractor (author-quoted code spans, URLs, file
  paths, qualified identifiers, CamelCase / snake_case symbols). No wordlists,
  no NLP models, precision-biased so prose doesn't pollute the graph.
- convo_miner: set `entities` per chunk, and compute hallways after a convos
  mine (mirroring the project-file path). Hallways run before the FTS5
  validation, which opens a direct sqlite connection that can invalidate the
  live Chroma collection handle on some Chroma builds.
- cli: `mempalace hallways` lists the associative graph (CLI parity with the
  list_hallways MCP tool).
- tests: extractor precision/ranking, entities metadata at mine time, CLI.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(graph): address review — semicolon safety, leading-underscore snake, negative limit

- entities `_clean`: strip `;` out of tokens so a URL query string or backtick
  span can't split the `;`-joined entities metadata field
- entities `_SNAKE`: optional leading/trailing `_?` so `_extract_authored_at`
  and similar are matched in plain text (previously only caught via backticks)
- cli `hallways`: clamp `--limit` with max(0, ...) so a negative value shows
  nothing instead of slicing from the end
- tests for all three

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Igor Lins e Silva <4753812+igorls@users.noreply.github.com>
2026-06-28 19:44:11 -03:00
178105cbd5 feat: optimize metadata counting using Qdrant server-side facets (#1868)
* feat: add metadata facet support for qdrant

* added benchmark

* updated benchmark

* chore: remove tracking for local scratch benchmark

* feat: add metadata facet support for qdrant -clean

* Update mempalace/mcp_server.py

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update mempalace/backends/qdrant.py

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update mempalace/backends/qdrant.py

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update tests/test_qdrant_backend.py

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update tests/test_qdrant_backend.py

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update tests/test_qdrant_backend.py

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update tests/test_mcp_server.py

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* /fix always working tool_status() fallback fixed

* /fix fallback added to tool_list_rooms

* Update mempalace/mcp_server.py

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update mempalace/mcp_server.py

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update tests/test_qdrant_backend.py

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* /fix rebuilt the room populating logic

* /add added temporary files for atomic transactions

* Update mempalace/mcp_server.py

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update mempalace/mcp_server.py

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update tests/test_mcp_server.py

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* /fix ai slop

* /fix added default facet limit

* Update tests/test_qdrant_backend.py

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* /fix added max workers pool

* Update mempalace/mcp_server.py

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update mempalace/mcp_server.py

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update mempalace/backends/qdrant.py

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* /fix added clear()

* Update tests/test_mcp_server.py

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* fix(qdrant): validate facet filter before existence check; fix taxonomy test

- facet_counts now validates the where filter and rejects local-only
  filters before the _remote_exists() short-circuit, so an unsupported
  filter raises UnsupportedCapabilityError even on an unmaterialized
  collection (matches get()/lexical_search() ordering).
- test_tool_get_taxonomy_uses_metadata_facets compared concurrent room
  facet calls via set(), but a call() with a dict kwarg is unhashable;
  compare order-independently via membership instead.

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Igor Lins e Silva <4753812+igorls@users.noreply.github.com>
2026-06-28 19:43:28 -03:00
Mikhail Valentsev 833b6ab4d3 feat(mcp): add since/before date filter to list_drawers (#1128) (#1891)
* feat(mcp): add since/before date filter to list_drawers (#1128)

mempalace_list_drawers previously filtered only by wing/room. This adds
optional since/before ISO date bounds on filed_at: since is inclusive,
before is exclusive.

The filter runs in Python after the rows are fetched. ChromaDB 1.5.7
rejects string operands for $gte/$lt and filed_at is stored as an ISO
string, so a server-side where comparison is not available; the tool
already collapses and paginates the full result set in Python.

Drawers whose filed_at is missing or unparseable are excluded while a
bound is active, and inverted bounds (since >= before) return a clear
error.

* test: close chromadb clients between tests to fix Windows handle leak (#1128)

chromadb 1.5.7 caches one System per palace path and only frees the
SQLite/HNSW file handles on client.close(); the collection fixture and
the per-test MCP cache reset only dereferenced the client, so handles
leaked across the session. Harmless on POSIX (rmtree unlinks open files),
but on Windows the handles stay locked and accumulate until an HNSW
segment write in a later test's setup fails, which surfaced here as
TestDeleteBySource::test_commit_purges_matching_closets asserting 0 == 2.

Close the client in the collection fixture and in _reset_mcp_cache so the
handles are released between tests.

* test: release backend chromadb clients between tests (#1128)

palace.get_collection() caches one PersistentClient per palace_path on the
process-wide backend singleton and never closes it; sweep, repair and several
CLI tests reach the store through it. chromadb frees the rust-side SQLite/HNSW
file handles only on client.close(), so the handles leak across the whole
session: a 30-palace probe shows ~200 open file descriptors into the palace
tree, dropping to 0 once the clients are closed.

On POSIX the open handles are harmless (rmtree unlinks open files), but on
Windows they stay locked and accumulate until a later test's HNSW segment
write fails ("Failed to apply logs to the hnsw segment writer"), e.g.
test_sweeper.py::TestSweeperTandem::test_sweep_recovers_untaken_message_at_cursor_timestamp.

Drain the cached clients in the autouse _reset_mcp_cache teardown via
close_palace(), which closes each PersistentClient (releasing its handles)
without marking the backend closed so it stays reusable. Complements the
collection-fixture and _client_cache close() added earlier.
2026-06-28 18:24:58 -03:00
faa8643d9f fix(palace): process-wide mine_palace_lock re-entrancy so the HTTP transport can write (#1859)
* fix(palace): process-wide mine_palace_lock re-entrancy for threaded HTTP transport

The MCP HTTP transport (ThreadingHTTPServer) acquires the long-lived
writer-lease on one thread (_acquire_mcp_writer_lock) but dispatches each
write request on a different worker thread. The lock re-entrancy guard was
thread-local, so write handlers (add_drawer/update_drawer) failed to see the
process-held lease, re-acquired the flock, and self-conflicted with
"palace ... is held by PID <self>". Reads worked (no lock); writes over the
HTTP transport were impossible.

Make the re-entrancy record process-wide (pid-tagged, guarded by a
threading.Lock) so a write from any thread of the process that already holds
the lease passes through. Safe: flock is per-process and HTTP writes are
serialized by _HTTP_REQUEST_LOCK. Preserves fork-safety, same-thread nesting
(miner.mine -> ChromaCollection.upsert), and cross-process protection
(MineAlreadyRunning still raised between processes).

Add cross-thread same-process regression test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(palace): reset lock guard on fork to avoid inherited-locked deadlock

Address review (PR #1859): `_palace_lock_guard` is a threading.Lock, so a child
forked while another thread held it would inherit it locked (the holder thread
is gone in the child) and deadlock on the next acquire. Register an
os.register_at_fork(after_in_child=...) handler that replaces the guard with a
fresh unlocked lock and clears state; the child must reacquire the flock anyway.
Guarded by hasattr(os, "register_at_fork") for Windows.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Igor Lins e Silva <4753812+igorls@users.noreply.github.com>
2026-06-28 18:24:10 -03:00
cff43adb63 feat(convo): preserve authored timestamp from transcripts (#1890)
* feat(convo): preserve authored timestamp from transcripts

Conversation drawers only carried `filed_at` (ingest time), so a bulk
re-mine collapsed every drawer to a single instant and the chronological
signal was lost — even though each Claude Code / Codex JSONL line already
carries an ISO-8601 `timestamp`. The recency-window fallback and any
date-aware consumer then saw ingest order, not when content was written.

- convo_miner: derive `authored_at` (per-file max line `timestamp`) and
  store it as drawer metadata; falls back to `filed_at` when absent
- searcher: surface `authored_at` in search results, and break exact
  hybrid-score ties toward the more recently authored drawer (ISO strings
  sort chronologically; missing dates sort oldest) — benchmark-neutral as
  it only reorders exact ties
- tests: cover `_extract_authored_at` (latest wins, skips/tolerates lines
  without timestamps, non-jsonl/missing -> None) and the tie-break

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(search): surface authored_at in CLI + backfill for existing data

Completes the authored_at work so the field is visible end-to-end and
existing palaces can adopt it without re-mining.

- layers: CLI `search` output shows an `authored:` date line per result
  (peer of the existing date; markdown drawers fall back to filed_at)
- scripts/backfill_authored_at.py: in-place migration that stamps
  authored_at on convos drawers from their source transcripts — metadata
  only (no re-embedding), idempotent, dry-run by default
- docs/authored-at.md: documents created_at (ingest) vs authored_at
  (written) and both backfill paths (in-place / drop-and-recreate)
- tests: backfill integration tests over an ephemeral ChromaDB collection

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(search): address review — non-string timestamp guard + top-level authored_at tiebreak

Two correctness fixes from the PR review:

- _extract_authored_at: only compare when the parsed `timestamp` is a str.
  A non-string timestamp on a malformed/foreign JSONL line previously raised
  TypeError outside the try and could crash the mine.
- _hybrid_rank: the tie-break read `authored_at` only from nested `metadata`,
  but the search_memories path (MCP / Claude Code) carries it at the top level
  of each hit — so the tie-break silently no-op'd there. Read both shapes.
- tests: non-string timestamp cases, and a top-level-shape tie-break test
  (which fails before this fix).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* style: apply ruff format to authored_at changes

CI ruff format --check flagged 4 files; ruff check already passed.
Formatting only — no behavior change.

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Igor Lins e Silva <4753812+igorls@users.noreply.github.com>
2026-06-28 18:13:06 -03:00
Pim MesselinkandClaude Opus 4.7 fd3b4e8ebb fix(pgvector): skip document column for metadata-only fetches (#1840 follow-up) (#1892)
* fix(pgvector): skip document column for metadata-only fetches (#1840 follow-up)

Closes the explicit "separate follow-up to keep this low-risk" callout
in PR #1840's description.

For remote pgvector deployments (TLS over WAN), `mempalace_status` and
every other metadata-only consumer was transferring the full `document`
column over the wire even when nothing read it. A single scroll over a
177K-drawer palace on a 175 ms-RTT link moved ~150 MB of document text
plus ~50 MB of metadata; this PR drops that to ~50 MB.

scroll_rows / _scroll gain `with_document: bool = True`. When False,
SELECT projects NULL::text instead of the document column. Positional
_row parser unchanged (record[1] stays the document slot, just receives
NULL). Existing callers default to True and see byte-for-byte identical
behavior.

PgVectorCollection.get_all_metadata override: where=None path goes
single-scroll with with_document=False. Filtered path falls back to base
to keep _matches_where running on array/object metadata values (same
correctness contract as #1840's filtered-path decision).

Tests:
- Update _FakePgVectorClient.scroll_rows to accept with_document; mirror
  the NULL-becomes-empty-string semantics when False
- Update 5 existing scroll_calls assertions to include with_document=True
  (unchanged intent)
- test_pgvector_get_all_metadata_skips_document_column: assert exactly
  one scroll call with with_document=False
- test_pgvector_get_all_metadata_filtered_falls_back_to_base: assert
  filtered path preserves with_document=True

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KC5Qsknh2zFRtRvVyjXiTA

* fix(pgvector): extend with_document=False fast path to filtered get_all_metadata

Per gemini-code-assist review feedback on #1892: _matches_where only reads
metadata, so the where=None vs where=set conditional fall-back was unnecessary.
The filtered path can use the same single-scroll with_document=False fast path
and apply the post-filter locally on metadata dicts — extending the wire-byte
win to every get_all_metadata caller, not just unfiltered ones.

Mirrors the pushdown + local _matches_where pattern already used by _rows
in the same file: pushdown when _requires_local_filter is False, post-filter
in Python otherwise. Same correctness contract as #1840's filtered get path.

Renames test_pgvector_get_all_metadata_filtered_falls_back_to_base to
test_pgvector_get_all_metadata_filtered_uses_fast_path and asserts the new
behavior (with_document=False + pushdown forwards the equality filter to SQL).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KC5Qsknh2zFRtRvVyjXiTA

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-28 18:02:29 -03:00
Pim MesselinkandClaude Opus 4.7 cd7a8658a2 fix(backends): require SQLite magic header for chroma + sqlite_exact detect() (#1893) (#1896)
* fix(chroma): require SQLite magic header for ChromaBackend.detect() (#1893)

Closes #1893.

ChromaBackend.detect() was returning True for a 0-byte chroma.sqlite3 file
because the check was just os.path.isfile(...). On a palace that has any
other backend marker alongside a stale 0-byte chroma.sqlite3,
resolve_backend_name then raises BackendMismatchError and the palace becomes
unopenable until the user manually rm's the empty file.

The 0-byte file appears as a side effect of any sqlite3.connect() on a
missing path — Python creates the file immediately but writes the SQLite
header only on the first statement. So any code path that touches the
chroma.sqlite3 path with bare sqlite3.connect(), including chromadb's own
PersistentClient lazy-init (see the comment at backends/chroma.py:2052),
can leave a 0-byte artifact behind.

Fix: detect() now reads the first 16 bytes and compares to the SQLite
magic prefix b"SQLite format 3\x00" instead of relying on file presence
alone. One extra open() + 16-byte read; detect() isn't a hot path.

Properties:
- Rejects 0-byte files (the symptom #1893 is about).
- Rejects non-SQLite garbage at the canonical path (partial writes, etc.).
- Doesn't false-negative on real chroma palaces: any chroma palace whose
  PersistentClient has done any work has the magic header on disk
  (verified — CREATE TABLE is enough to land the header).
- Doesn't couple detect() to chroma's specific schema; the magic header
  is stable across chromadb releases.

Test sweep: many test files used (chroma.sqlite3).touch() or
.write_bytes(b"") as a "fake palace" shortcut, exploiting the loose
isfile() check (one such site even had the comment "# pass the isfile
guard"). After this change, those stand-ins no longer register as chroma
palaces. Introduced tests/_chroma_palace_helper.py::make_minimal_chroma_sqlite
following the existing _backend_conformance.py precedent, and updated 15
call sites across 8 test files to use it. The existing
test_chroma_detect_matches_palace_with_chroma_sqlite (which encoded the
buggy semantics with write_bytes(b"")) is renamed to
test_chroma_detect_matches_palace_with_sqlite_header and now writes a
real SQLite database via the helper. Added two new tests for the
rejection paths (empty file, non-SQLite garbage).

Full env-cleared suite: 3137 passed, 20 skipped, 0 failed. ruff check
and ruff format --check both clean.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KC5Qsknh2zFRtRvVyjXiTA

* fix(sqlite_exact): require SQLite magic header for SQLiteExactBackend.detect()

Per gemini-code-assist review on #1892 PR #1896: SQLiteExactBackend has the
same os.path.isfile() detection pattern as ChromaBackend did, with the same
0-byte-file vulnerability. Mirrors the chroma fix for repo-wide consistency.

- SQLiteExactBackend.detect() now does the same 16-byte SQLite magic-prefix
  check as ChromaBackend.detect().
- _chroma_palace_helper.py: factored its body into a private
  _write_minimal_sqlite_file() and gained a sibling
  make_minimal_sqlite_exact_sqlite() for the sqlite_exact filename. No churn
  to any existing chroma call sites.
- test_sqlite_exact_backend.py:426 (the one site that wrote b"" for
  sqlite_exact.sqlite3) updated to use the new helper.
- Three new tests in test_sqlite_exact_backend.py mirror the chroma trio:
  matches with valid header, rejects empty file, rejects non-SQLite garbage.

Full env-cleared suite: 3140 passed, 20 skipped, 0 failed.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KC5Qsknh2zFRtRvVyjXiTA

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-28 18:01:59 -03:00
dependabot[bot] d8b63a9521 chore(deps-dev): bump ruff from 0.15.18 to 0.15.20 (#1883)
Bumps [ruff](https://github.com/astral-sh/ruff) from 0.15.18 to 0.15.20.
- [Release notes](https://github.com/astral-sh/ruff/releases)
- [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md)
- [Commits](https://github.com/astral-sh/ruff/compare/0.15.18...0.15.20)

---
updated-dependencies:
- dependency-name: ruff
  dependency-version: 0.15.20
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-06-28 01:33:30 -03:00
dependabot[bot] d6cb868f75 chore(deps): bump docker/setup-buildx-action from 3 to 4 (#1881)
Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 3 to 4.
- [Release notes](https://github.com/docker/setup-buildx-action/releases)
- [Commits](https://github.com/docker/setup-buildx-action/compare/v3...v4)

---
updated-dependencies:
- dependency-name: docker/setup-buildx-action
  dependency-version: '4'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-06-28 01:33:23 -03:00
dependabot[bot] b864766b12 chore(deps): bump docker/setup-qemu-action from 3 to 4 (#1880)
Bumps [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action) from 3 to 4.
- [Release notes](https://github.com/docker/setup-qemu-action/releases)
- [Commits](https://github.com/docker/setup-qemu-action/compare/v3...v4)

---
updated-dependencies:
- dependency-name: docker/setup-qemu-action
  dependency-version: '4'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-06-28 01:33:15 -03:00