Commit Graph
10 Commits
Author SHA1 Message Date
Igor Lins e Silva e1fc71a9b9 chore(release): 3.7.0
Bump package, plugins, lock, and README badge to 3.7.0. Promote the
post-3.6.0 integrity spine into CHANGELOG: single-writer ownership,
HNSW write defaults and preflight, safer repair/re-mine, MCP/daemon
lifecycle hardening, entity ReDoS guard, and hook write-routing.

Also ruff-format two test files that drifted during conflict merges.

Local validation: ruff check/format clean; 3497 passed, 31 skipped.
2026-08-02 05:34:29 -03:00
Igor Lins e Silva 4009e75b5d merge: resolve #2065 onto develop (union palace test imports) 2026-08-02 05:26:36 -03:00
Igor Lins e Silva af0c7665e0 Merge branch 'pr-2089' into fix/2089-onto-develop 2026-08-02 05:25:13 -03:00
Igor Lins e Silva 524c980a17 fix: address backend ownership edge cases 2026-08-01 20:23:53 -03:00
KeilerHirsch 0eb137e419 style: ruff format after import wrap (CI lint) 2026-07-28 16:47:17 +02:00
KeilerHirsch 3a97996b30 fix(convo_miner): stop sweeper drawers being purged, abort on failed purge
#104 (CRITICAL, data-loss): the sweeper writes drawers with no
extract_mode at all (ingest_mode="sweep"). _metadata_matches_extract_mode's
legacy-compat rule -- "no extract_mode means treat as a legacy exchange
row" -- couldn't tell that apart from a genuine pre-schema convo_miner
row, so mempalace mine --mode convos (default extract=exchange) swept
every sweeper drawer for a shared transcript into its purge scope and
deleted them on the very next re-mine. The legacy-compat fallback now
only applies when the drawer is otherwise convo_miner's own (no
ingest_mode at all, or convo_miner's own "convos" tag) -- a drawer
positively identified as another producer's (sweep, or any other
foreign ingest_mode) never matches, using the same ingest_mode
discriminator sync.py already relies on for its own registry-row check.

#105 (MEDIUM, silent-failure): convo_miner's own instance of the
purge-failure swallow already fixed for miner.py at #23 -- a failed
purge in _file_chunks_locked was logged at debug level and mining
proceeded anyway, silently producing duplicate/stale drawers under
mixed schema versions. Now aborts (returns skipped=True, leaving the
old drawers' stored mtime untouched so the next mine retries) and
prints a visible warning.

428 tests pass across test_palace.py/test_convo_miner*.py/test_miner.py/
test_sweeper.py/test_hallways.py/test_format_miner.py/test_repair.py, no
regressions.
2026-07-28 16:24:25 +02:00
934a4419ef fix(entity): defuse entity-candidate ReDoS on long ASCII runs (#2063)
The English entity-candidate pattern in i18n/en.json backtracks
catastrophically on a long unbroken run of printable ASCII (base64,
minified JS, hashes, data URIs), pinning `mempalace mine` on a single
~5000-char window for hours. Collapse such runs to a space before
single-word candidate matching, in both consumers that apply the pattern
(palace._candidate_entity_words and entity_detector.extract_candidates).

Scoped to ASCII ([!-~]) so non-ASCII scripts stay untouched — a CJK
paragraph is one unbroken run with no ASCII whitespace, and zh-CN/zh-TW
have no multi-word fallback, so a blanket collapse would erase their
detection. Threshold 24 sits above the 20-char cap of the simple-name
pattern, so no real name is dropped.

Fixes #2063.

Co-authored-by: Ryan Wei <9876551+RyanWei@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-23 12:56:37 +05:00
Pim MesselinkandClaude Opus 4.7 cd7a8658a2 fix(backends): require SQLite magic header for chroma + sqlite_exact detect() (#1893) (#1896)
* fix(chroma): require SQLite magic header for ChromaBackend.detect() (#1893)

Closes #1893.

ChromaBackend.detect() was returning True for a 0-byte chroma.sqlite3 file
because the check was just os.path.isfile(...). On a palace that has any
other backend marker alongside a stale 0-byte chroma.sqlite3,
resolve_backend_name then raises BackendMismatchError and the palace becomes
unopenable until the user manually rm's the empty file.

The 0-byte file appears as a side effect of any sqlite3.connect() on a
missing path — Python creates the file immediately but writes the SQLite
header only on the first statement. So any code path that touches the
chroma.sqlite3 path with bare sqlite3.connect(), including chromadb's own
PersistentClient lazy-init (see the comment at backends/chroma.py:2052),
can leave a 0-byte artifact behind.

Fix: detect() now reads the first 16 bytes and compares to the SQLite
magic prefix b"SQLite format 3\x00" instead of relying on file presence
alone. One extra open() + 16-byte read; detect() isn't a hot path.

Properties:
- Rejects 0-byte files (the symptom #1893 is about).
- Rejects non-SQLite garbage at the canonical path (partial writes, etc.).
- Doesn't false-negative on real chroma palaces: any chroma palace whose
  PersistentClient has done any work has the magic header on disk
  (verified — CREATE TABLE is enough to land the header).
- Doesn't couple detect() to chroma's specific schema; the magic header
  is stable across chromadb releases.

Test sweep: many test files used (chroma.sqlite3).touch() or
.write_bytes(b"") as a "fake palace" shortcut, exploiting the loose
isfile() check (one such site even had the comment "# pass the isfile
guard"). After this change, those stand-ins no longer register as chroma
palaces. Introduced tests/_chroma_palace_helper.py::make_minimal_chroma_sqlite
following the existing _backend_conformance.py precedent, and updated 15
call sites across 8 test files to use it. The existing
test_chroma_detect_matches_palace_with_chroma_sqlite (which encoded the
buggy semantics with write_bytes(b"")) is renamed to
test_chroma_detect_matches_palace_with_sqlite_header and now writes a
real SQLite database via the helper. Added two new tests for the
rejection paths (empty file, non-SQLite garbage).

Full env-cleared suite: 3137 passed, 20 skipped, 0 failed. ruff check
and ruff format --check both clean.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KC5Qsknh2zFRtRvVyjXiTA

* fix(sqlite_exact): require SQLite magic header for SQLiteExactBackend.detect()

Per gemini-code-assist review on #1892 PR #1896: SQLiteExactBackend has the
same os.path.isfile() detection pattern as ChromaBackend did, with the same
0-byte-file vulnerability. Mirrors the chroma fix for repo-wide consistency.

- SQLiteExactBackend.detect() now does the same 16-byte SQLite magic-prefix
  check as ChromaBackend.detect().
- _chroma_palace_helper.py: factored its body into a private
  _write_minimal_sqlite_file() and gained a sibling
  make_minimal_sqlite_exact_sqlite() for the sqlite_exact filename. No churn
  to any existing chroma call sites.
- test_sqlite_exact_backend.py:426 (the one site that wrote b"" for
  sqlite_exact.sqlite3) updated to use the new helper.
- Three new tests in test_sqlite_exact_backend.py mirror the chroma trio:
  matches with valid header, rejects empty file, rejects non-SQLite garbage.

Full env-cleared suite: 3140 passed, 20 skipped, 0 failed.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KC5Qsknh2zFRtRvVyjXiTA

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
2026-06-28 18:01:59 -03:00
Igor Lins e Silva ec5d1eb1c7 feat: add pgvector backend + namespace-isolation conformance contract
Adds a second external storage backend (Postgres/pgvector) alongside Qdrant
to prove the BaseBackend/BaseCollection contract generalizes across substrates
(SQL + JSONB containment filters + pgvector `<=>` ranking vs Qdrant's REST/dict
model), and addresses the review feedback on PR #1679.

Backend (mempalace/backends/pgvector.py):
- table-per-(namespace, palace, collection) isolation; advertises
  supports_namespace_isolation
- JSONB filter pushdown for the containment subset, local-exact fallback for
  $or/$contains/comparisons/where_document
- BM25 lexical search; marker-based mismatch protection
- optional psycopg dependency (lazy import), in-memory fake for CI, live test
  gated on MEMPALACE_PGVECTOR_LIVE_URL
- registered in registry/__init__/pyproject entry point + [pgvector] extra;
  MEMPALACE_PGVECTOR_DSN / MEMPALACE_PGVECTOR_NAMESPACE config; README docs

Isolation contract (RFC 001):
- PalaceRef/BaseBackend document the per-id MUST and the cross-namespace MUST,
  gated on the new supports_namespace_isolation capability token
- runnable conformance suite (tests/_backend_conformance.py,
  tests/test_backend_conformance.py); qdrant + pgvector run it via their fakes

Marker fail-loud guard:
- qdrant and pgvector now refuse get_collection when local_path is None instead
  of silently opening a remote collection with no mismatch protection

Review fixes:
- palace._open_collection_or_explain handles unknown-backend KeyError as a CLI
  state message instead of an escaping stack trace
- dedup.py docstring no longer claims "No API calls" unconditionally (false for
  remote backends)
2026-06-05 21:41:21 -03:00
mvalentsev c933095a2f fix(palace): stratify state messages for empty/missing palace (#1498)
`mempalace status`, `search`, and `compress` printed the same misleading
"No palace found / Run: mempalace init" output for three distinct
palace states (no dir / no DB / no collection). Most user-visible on
the first-run state where `init` had run but `mine` had not: the hint
to re-run `init` is a no-op and wastes the user's time.

Backend gets a new typed exception `CollectionNotInitializedError`
(subclass of `PalaceNotFoundError`, transitively `FileNotFoundError`,
so legacy callers keep working). `ChromaBackend.get_collection(
create=False)` wraps chromadb's bare `NotFoundError` as the new typed
exception instead of leaking the chromadb-specific class to callers.

A new internal helper `_open_collection_or_explain` in `palace.py`
runs filesystem-first state checks before the backend call to avoid
chromadb's lazy `chroma.sqlite3` creation as a side-effect of a
read-only inspection, then catches the typed exceptions and prints a
state-specific actionable message. `BackendClosedError` is explicitly
re-raised so a programmer error is not masked as a UX hint.

Two CLI bug sites route through the helper: `miner.status` and
`cli.cmd_compress`. `searcher.search` catches the typed exceptions
directly so it can preserve the cause chain in `SearchError(...) from
e` for programmatic search-API consumers. `cli.cmd_sync` gained an
inline filesystem distinction (no helper needed: it does not use the
collection handle). `repair.status` (capacity check, which by design
must work on corrupted palaces without opening a chromadb client) got
the same distinction via `sqlite_drawer_count`-based empty detection.
The MCP `tool_status` is intentionally left alone: PR #831 already
fixed it there with a `create=True` bootstrap strategy appropriate
for programmatic clients.
2026-05-17 03:57:19 +05:00