* fix(chroma): require SQLite magic header for ChromaBackend.detect() (#1893)
Closes#1893.
ChromaBackend.detect() was returning True for a 0-byte chroma.sqlite3 file
because the check was just os.path.isfile(...). On a palace that has any
other backend marker alongside a stale 0-byte chroma.sqlite3,
resolve_backend_name then raises BackendMismatchError and the palace becomes
unopenable until the user manually rm's the empty file.
The 0-byte file appears as a side effect of any sqlite3.connect() on a
missing path — Python creates the file immediately but writes the SQLite
header only on the first statement. So any code path that touches the
chroma.sqlite3 path with bare sqlite3.connect(), including chromadb's own
PersistentClient lazy-init (see the comment at backends/chroma.py:2052),
can leave a 0-byte artifact behind.
Fix: detect() now reads the first 16 bytes and compares to the SQLite
magic prefix b"SQLite format 3\x00" instead of relying on file presence
alone. One extra open() + 16-byte read; detect() isn't a hot path.
Properties:
- Rejects 0-byte files (the symptom #1893 is about).
- Rejects non-SQLite garbage at the canonical path (partial writes, etc.).
- Doesn't false-negative on real chroma palaces: any chroma palace whose
PersistentClient has done any work has the magic header on disk
(verified — CREATE TABLE is enough to land the header).
- Doesn't couple detect() to chroma's specific schema; the magic header
is stable across chromadb releases.
Test sweep: many test files used (chroma.sqlite3).touch() or
.write_bytes(b"") as a "fake palace" shortcut, exploiting the loose
isfile() check (one such site even had the comment "# pass the isfile
guard"). After this change, those stand-ins no longer register as chroma
palaces. Introduced tests/_chroma_palace_helper.py::make_minimal_chroma_sqlite
following the existing _backend_conformance.py precedent, and updated 15
call sites across 8 test files to use it. The existing
test_chroma_detect_matches_palace_with_chroma_sqlite (which encoded the
buggy semantics with write_bytes(b"")) is renamed to
test_chroma_detect_matches_palace_with_sqlite_header and now writes a
real SQLite database via the helper. Added two new tests for the
rejection paths (empty file, non-SQLite garbage).
Full env-cleared suite: 3137 passed, 20 skipped, 0 failed. ruff check
and ruff format --check both clean.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KC5Qsknh2zFRtRvVyjXiTA
* fix(sqlite_exact): require SQLite magic header for SQLiteExactBackend.detect()
Per gemini-code-assist review on #1892 PR #1896: SQLiteExactBackend has the
same os.path.isfile() detection pattern as ChromaBackend did, with the same
0-byte-file vulnerability. Mirrors the chroma fix for repo-wide consistency.
- SQLiteExactBackend.detect() now does the same 16-byte SQLite magic-prefix
check as ChromaBackend.detect().
- _chroma_palace_helper.py: factored its body into a private
_write_minimal_sqlite_file() and gained a sibling
make_minimal_sqlite_exact_sqlite() for the sqlite_exact filename. No churn
to any existing chroma call sites.
- test_sqlite_exact_backend.py:426 (the one site that wrote b"" for
sqlite_exact.sqlite3) updated to use the new helper.
- Three new tests in test_sqlite_exact_backend.py mirror the chroma trio:
matches with valid header, rejects empty file, rejects non-SQLite garbage.
Full env-cleared suite: 3140 passed, 20 skipped, 0 failed.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KC5Qsknh2zFRtRvVyjXiTA
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
SQLiteExactCollection.get(limit, offset) fetched the whole collection via
_rows() (SELECT ... FROM documents ORDER BY rowid, no LIMIT/OFFSET) and sliced
in Python, so every paginating caller (prefetch_mined_set, status, exporter,
migrate, dedup, sync, ...) re-scanned the entire table per page, making the
sweep O(N^2) in rows materialized.
Push LIMIT/OFFSET into the scan on the unfiltered page (no ids/where/
where_document and non-negative bounds); filtered, id, and negative pages keep
the full-scan plus Python-slice path so the post-filter still runs first. SQLite
requires a LIMIT before OFFSET, so an offset-only page uses LIMIT -1. ORDER BY
rowid keeps pages stable.
Three findings from the Copilot review on ec5d1eb:
- pgvector (real correctness bug): table_dimension() read the raw
pg_attribute.atttypmod of the vector(n) column, which is not the bare
dimension, so reopening a stored pgvector palace could raise a false
DimensionMismatchError on the next same-dimension write. Now rounds through
format_type(atttypid, atttypmod) (the type's own typmod_out), which yields
the canonical vector(N) regardless of encoding or pgvector version. The live
roundtrip test now closes + reopens and writes a same-dim vector to guard it.
- chroma (real correctness bug): _lexical_search_via_sqlite() returned
LexicalHit.id as the internal embeddings.id rowid instead of the public
embeddings.embedding_id, so lexical_search -> get(ids=...) did not round-trip
(broke hybrid-search id lookups). Now selects e.embedding_id and maps rowid
-> public id. Existing FTS test schema updated to include embedding_id (real
Chroma schema) and assert the public id; added an end-to-end round-trip test
through a real ChromaBackend collection.
- sqlite_exact (error-message quality): CollectionNotInitializedError was
raised with palace_path instead of the collection name in get_collection and
delete_collection, inconsistent with the other backends and line 287. Now
names the collection; added a regression test.
Earlier first-pass findings (palace.py unknown-backend KeyError, dedup.py
docstring) were already fixed in ec5d1eb.
Pim Messelink
mvalentsev
Igor Lins e Silva