from-sqlite ignored --dry-run and performed the real archive+rebuild
(#2095, #2133). Preview after source validation, skip the destructive
confirm, never take the mine-lock or rename the palace, and fail closed
when SQLite row counts are unreadable instead of inventing zeros.
Address actionable #2129 bot feedback and the Windows closet KeyError:
- Reopen immutable sqlite_exact readers only when both WAL sidecars exist
(partial pair keeps the clean snapshot instead of failing the reconnect).
- Retry get_collection without options when plugin backends reject the kwarg.
- Stamp multi-conversation content_hash only on chunk 0 to avoid O(N²) meta.
- Always include results: [] on search error envelopes so callers never KeyError.
- Clearer hybrid-search assertions in the closet isolation test.
Bump package, plugins, lock, and README badge to 3.7.0. Promote the
post-3.6.0 integrity spine into CHANGELOG: single-writer ownership,
HNSW write defaults and preflight, safer repair/re-mine, MCP/daemon
lifecycle hardening, entity ReDoS guard, and hook write-routing.
Also ruff-format two test files that drifted during conflict merges.
Local validation: ruff check/format clean; 3497 passed, 31 skipped.
Address the last two Codex review comments on #2079:
- Reopen sqlite_exact immutable read-only connections when WAL/SHM
sidecars appear after a clean-database open so recall sees a later
writer's commits instead of a frozen pre-writer snapshot.
- Clear per-palace embedder-identity validation when MCP discards
storage handles during ownership promotion, so an empty read-only
open cannot skip recording the active model on the first write.
rebuild_from_sqlite renamed the existing palace aside
(palace.pre-rebuild-<ts>) and only hit the single-writer
mine_palace_lock later, when the first chromadb upsert ran. A palace
held by a live MCP server / daemon was therefore stranded: archived,
no rebuilt replacement, a partial dest left behind.
Acquire mine_palace_lock(dest_palace) BEFORE the archive/rename so
contention raises MineAlreadyRunning while the palace is untouched
(no archive, no partial). The lock wraps the whole rebuild as one
writer; the inner per-batch acquires pass through re-entrantly.
The body is split into _rebuild_from_sqlite_locked so the lock wraps
every destructive step.
Behaviour: held lock during repair from-sqlite — before: palace
archived + partial dest stranded; after: clean MineAlreadyRunning,
palace untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`_HNSW_BLOAT_GUARD` pinned `hnsw:batch_size=2` / `hnsw:sync_threshold=2` on every
collection this backend creates, to answer #1579. Three findings on chromadb 1.5.9
(PersistentClient, Rust bindings, single writer, zero concurrency) retire it.
**2 sits outside chroma's own declared valid range.** `hnsw_params.py:21-22`
validates both knobs as `isinstance(p, int) and p > 2`.
**The cost is write amplification, measured.** Bytes written (`/proc/self/io`) for
one mine of N records, 64-dim, `num_threads=1`, identical corpus and seed:
N 2/2 100/1000 ratio
10,000 47.1 MB 28.1 MB 1.67x
20,000 127.4 MB 60.0 MB 2.12x
40,000 390.3 MB 134.0 MB 2.91x
Two runs at N=20,000 reproduced within 0.1%. `sync_threshold` dominates — 3/3
measured identical to 2/2, and 1000/1000 identical to 100/1000, so `batch_size`
barely registers. The ratio grows with collection size, so the largest palaces pay
the most.
**It buys nothing.** A 5-record collection at 100/1000 — far below the threshold,
so no persist ever fires — reads back whole from a fresh process: `count()` and the
vector query both answer. On that artifact `link_lists.bin` is 0 bytes with no
`index_metadata.pickle`, which is #1579's trigger shape exactly, and
`quarantine_stale_hnsw()` creates no drift directory.
The knobs move into `_hnsw_creation_metadata()`, which both `create_collection`
paths now call, so per-collection tuning has one home.
Legacy `metadata=` is kept deliberately. Measured on 1.5.9: creating with
`configuration=` leaves `collection.metadata` as `None` and the sqlite
`collection_metadata` table empty, which blinds `_read_sync_threshold` and
`ChromaCollection.distance_metric` — both read that table. (`collection.configuration`
does read back under both paths; migrating those two readers to it is the
prerequisite for ever moving the writer, and belongs in its own change.)
Not claimed: that a small `sync_threshold` is a known chroma failure mode. No such
report was found in chroma's issues or docs, and chroma's own guidance runs the
other way. The Python persist path that #1579 and chroma#6975 describe does not
execute under the Rust bindings — `hnswlib` is not a dependency of this line.
Addresses fatkobra's CHANGES_REQUESTED review on this PR. The prior
truncation-detection fix in _paginate_ids left two loud-failure gaps:
1. When BOTH the offset get() and its no-offset fallback failed, the
`except Exception: break` returned the partial (or empty) ids
collected so far as if the listing were complete. On a mid-run
failure that hands scan_palace/rebuild a truncated palace; on the
first page it returns [] and scan_palace prints "Nothing to scan."
for a palace it never read. Now raises RuntimeError (chained from
the underlying error) instead of breaking.
2. The count()-based truncation disambiguation used the collection-wide
col.count(), but _paginate_ids accepts a where= filter (scan_palace
passes {"wing": only_wing}). A global count greater than the
collected rows does not prove a *filtered* result is truncated -- the
extra rows may belong to other wings. count() takes no where=
argument, so the raise is now gated on `where is None`; filtered
completeness is treated as unknown rather than false-positive-raised.
Adds three regression tests covering the after-progress double failure,
the first-page double failure, and the exactly-page-size filtered case.
Read-only gated on _MUTATING_TOOLS, which is the palace-write set the
peer-writer lease consults to decide which calls need the palace mine
lock. Two tools change state without touching the palace, so they are
correctly absent from that set and were served on a --read-only server
anyway: mempalace_hook_settings rewrites ~/.mempalace/config.json via
MempalaceConfig.set_hook_setting, and mempalace_memories_filed_away
unlinks ~/.mempalace/hook_state/last_checkpoint on both branches.
Add _READ_ONLY_REFUSED_TOOLS and point the dispatch gate and the
tools/list filter at it. _MUTATING_TOOLS and the peer-writer path are
unchanged: adding the two names there instead would put a config-only
tool under the palace lease, so a server that lost the lease to a peer
would answer -32001 for a call the lease has no say over.
mempalace_reconnect stays reachable on purpose and the comment records
why, since clearing ChromaBackend._quarantined_paths lets the reopen
rename a segment directory. The two --read-only help strings and the
matching row in the remote-server guide said "the mutating tools",
which now names the narrower set.
Independent review of the preceding commit found that dedup_palace's own
'Drawers: N' count() print runs a few lines before it calls
get_source_groups -- #92's palace_path guard on get_source_groups only
covered that function's internal count(), leaving this earlier, separate
call site in the same function fully exposed to the same #1222
SIGSEGV/panic class. Same fix pattern: preflight hnsw_capacity_status
before this print and abort the whole dedup_palace run on divergence.
show_stats has no equivalent exposure -- it goes straight to
get_source_groups with no separate count() print.
hnsw_capacity_status() (chroma.py) exists precisely to preflight the
#1222 SIGSEGV/pyo3-panic class before anything touches the HNSW
segment, but repo-wide it was wired into only 4 call sites while raw
count()/collection.count() is called at 20+ others -- a bare
except Exception around count() cannot catch a native crash, since
the process dies regardless of any Python try/except. This wires the
existing, already-tested probe into the 7 remaining call sites the
audit identified as CRITICAL:
- #89 palace.py::_enforce_embedder_identity -- the universal
get_collection() chokepoint every tool passes through, previously
guarded only by except Exception. Highest leverage: skips this
bookkeeping-only check on divergence instead of risking count().
- #90 migrate.py::migrate -- routes straight to the same
SQLite-extraction fallback the except branch already used, instead
of ever reaching col.count() when diverged.
- #91 repair.py::scan_palace / prune_corrupt -- both abort with the
existing from-sqlite recovery guidance instead of opening the
collection.
- #10 repair.py::rebuild_index -- preflights divergence alongside its
existing sqlite-integrity and poisoned-max-seq-id preflights, before
opening the collection.
- #13 repair.py::rebuild_index never rebuilt or reported on the
closets collection -- now warns when closets is still diverged
after a drawers-only rebuild, pointing at --mode from-sqlite instead
of letting 'Repair complete' stand unqualified.
- #92 dedup.py::get_source_groups -- takes an optional palace_path
(threaded from both callers) to preflight before count(); omitted by
existing tests, which keep their pre-existing behavior.
- #93 miner.py::status -- preflights before the ChromaDB-client
fallback path (used when the direct sqlite read is unavailable).
7 new regression tests, each confirmed failing against the pre-fix
code (via git stash of the source files only) and passing after the
fix. One existing dedup.py test updated for the new palace_path kwarg
in its call-signature assertion. Full suite: 3154 passed, 1 unrelated
pre-existing flake (test_mcp_server.py peer-writer-lock module-global
state leaking across test files in full-suite ordering -- this diff
never touches mcp_server.py).
#104 (CRITICAL, data-loss): the sweeper writes drawers with no
extract_mode at all (ingest_mode="sweep"). _metadata_matches_extract_mode's
legacy-compat rule -- "no extract_mode means treat as a legacy exchange
row" -- couldn't tell that apart from a genuine pre-schema convo_miner
row, so mempalace mine --mode convos (default extract=exchange) swept
every sweeper drawer for a shared transcript into its purge scope and
deleted them on the very next re-mine. The legacy-compat fallback now
only applies when the drawer is otherwise convo_miner's own (no
ingest_mode at all, or convo_miner's own "convos" tag) -- a drawer
positively identified as another producer's (sweep, or any other
foreign ingest_mode) never matches, using the same ingest_mode
discriminator sync.py already relies on for its own registry-row check.
#105 (MEDIUM, silent-failure): convo_miner's own instance of the
purge-failure swallow already fixed for miner.py at #23 -- a failed
purge in _file_chunks_locked was logged at debug level and mining
proceeded anyway, silently producing duplicate/stale drawers under
mixed schema versions. Now aborts (returns skipped=True, leaving the
old drawers' stored mtime untouched so the next mine retries) and
prints a visible warning.
428 tests pass across test_palace.py/test_convo_miner*.py/test_miner.py/
test_sweeper.py/test_hallways.py/test_format_miner.py/test_repair.py, no
regressions.
extract_via_sqlite drove its extraction with an INNER JOIN starting at
embedding_metadata. Any embedding with zero rows there (a sparse
historical write with no chroma:document and no other key -- the same
condition _extract_drawers already sanitizes for the collection-layer
rebuild path, see #1458) never appeared in the join result and was
silently dropped by mempalace repair --mode from-sqlite, with no count
mismatch and no warning.
Drive the query from embeddings (LEFT JOIN embedding_metadata) instead,
defaulting to an empty metadata dict when a row has none. Order by
e.id rather than em.id since em.id is NULL for unmatched rows.
Regression test seeds a real chromadb collection, then strips one
drawer's metadata rows directly via SQLite (current chromadb validates
against empty metadata on write, so this only reproduces on rows
already sitting in an older palace) and asserts the drawer still
comes back through the SQLite bypass.
cmd_repair() has a separate, inlined implementation of the same
live-swap-failure recovery that _rebuild_collection_via_temp's shared
fix (preserve the verified temp copy) already covers for rebuild_index().
Before this, cmd_repair's own except-block still did shutil.rmtree()
on the whole palace directory (destroying that same temp copy) and
restored a pre-repair full-directory backup -- printing recovery
guidance that pointed at data the same code path had just deleted.
Now both callers promote from the verified temp copy consistently.
Bare (#8)/#12: markers were internal finding IDs from a local audit
report, not GitHub references -- but they collide with real closed
issues in this repo (#8, #12) and would misdirect reviewers. Replaced
with plain prose.
Two independent bugs from the 2026-07-28 full-repo audit:
1. _rebuild_collection_via_temp deleted its own verified temp collection
in the except-handler even when the live collection had already been
replaced (live_replaced=True) -- at that point the temp copy is the
ONLY intact data left. Now preserved, and the error message points the
operator at it.
rebuild_index's own recovery path used to 'restore' by copying back a
pre-rebuild chroma.sqlite3 file backup and claim the palace was back
to pre-repair state -- but the live collection's on-disk HNSW segment
directories were already destroyed by the delete, so the restored
sqlite3 referenced segment UUIDs that no longer existed. Replaced with
a new _promote_temp_collection() that recovers by re-extracting from
the verified temp copy and re-uploading into a fresh live collection,
with an honest fallback message if that promotion also fails.
2. _paginate_ids silently returned a truncated ID list when offset-based
pagination failed and the no-offset fallback got structurally stuck
at the first page (1000 results). Callers (scan_palace, rebuild) treat
the result as the complete palace, so this could act on incomplete
data with no warning. Now cross-checks against the collection's own
count() to disambiguate a genuinely-complete page-sized collection
from a truncated larger one, and raises instead of silently truncating
when it can't tell.
Both fixes verified against an independent adversarial review pass
(mutation-tested: confirmed the new tests catch a wrong-source-collection
read and a swapped ids/documents payload in the recovery path, which the
original test set did not).
Deliberately out of scope: none of the rebuild paths hold mine_palace_lock
for their full duration, so a concurrent writer can still interleave with
a rebuild. That's a separate, larger architectural gap (queue-and-replay
vs. block-on-write), tracked for its own follow-up.
A stdio mcp_server orphaned by a dropped SSH session used to swallow
pipe errors in the generic exception handler and sleep forever on the
dead channel, holding the mine_palace flock (2026-07-10 write-path
outage, PID 23392). The loop now logs and shuts down on stdin EOF,
OSError from stdin reads (EIO/EBADF from orphaned ptys), and
BrokenPipeError/OSError on stdout writes. Shutdown returns from the
loop for a normal interpreter exit 0 — no os._exit — so held flocks
release via process teardown; a broken stdout is re-pointed at devnull
first so the shutdown flush cannot turn the exit into status 120.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DpEdb9SNWF42BpupHWBhC3
The English entity-candidate pattern in i18n/en.json backtracks
catastrophically on a long unbroken run of printable ASCII (base64,
minified JS, hashes, data URIs), pinning `mempalace mine` on a single
~5000-char window for hours. Collapse such runs to a space before
single-word candidate matching, in both consumers that apply the pattern
(palace._candidate_entity_words and entity_detector.extract_candidates).
Scoped to ASCII ([!-~]) so non-ASCII scripts stay untouched — a CJK
paragraph is one unbroken run with no ASCII whitespace, and zh-CN/zh-TW
have no multi-word fallback, so a blanket collapse would erase their
detection. Threshold 24 sits above the 20-char cap of the simple-name
pattern, so no real name is dropped.
Fixes#2063.
Co-authored-by: Ryan Wei <9876551+RyanWei@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A daemon job refused the palace write lock was marked terminal `failed` and
never retried. `mine_palace_lock` guards the palace write itself, so a refusal
means no drawer was filed, but the queue recorded it exactly like a crash
mid-execution, whose outcome is unknown. The work was dropped: it only ran
again if a hook happened to re-submit equivalent work, and a job kind no hook
re-emits was lost outright.
Refusals now defer. The job goes back to `queued` with `started_at` cleared,
the claim's attempt increment undone, and the reason recorded. The update is
scoped to the claim that was refused (started_at match), so a defer racing a
recovery re-claim cannot re-queue work it does not own. `claim_next` clears
the recorded reason so it never outlives the claim it describes. Every other
failure stays terminal: a crashed job's outcome is unknown, and blindly
re-running a non-idempotent kind would re-file verbatim content, which is what
MAX_ATTEMPTS guards.
The worker cools a refused job off in memory and moves on rather than sleeping
in line. It is the only worker and the holder keeps the lock until its write
finishes, which can be a long mine, so blocking would stall every unrelated job
behind a lock that has nothing to do with them, and a job merely queued behind
the refused one would never be claimed. `claim_next(exclude=...)` skips a
cooling job, so the oldest-first ordering cannot hand the same refused job back
forever while newer work waits. The filter runs in Python rather than an
`id NOT IN (?, ?, ...)` list, which would bind one host parameter per cooling
job against a cap that defaults to 999 before SQLite 3.32. The cooldown
lives in the worker, not the schema, which has no migrations; a restart just
retries at once, costing one refusal, never work.
`tool_diary_write` swallowed `MineAlreadyRunning` in its bare `except
Exception`, so the refusal reached the daemon with no `error_class` and
`diary_write` would still have been dead-lettered. It now uses a typed handler
ahead of the bare `Exception`, the way `tool_mine` and `tool_sync` already do.
Deferral makes a refused job non-terminal, and `DaemonClient.wait` only returns
on a terminal state, so callers that wait on purpose would have waited for a
state a parked job cannot reach: a foreground `mine --daemon` for the one-hour
default, and the `hooks_cli` pre-compaction mine and Stop-hook diary paths for
their whole timeout on every fire, each then reporting a failure that never
happened. `wait(stop_on_lock_deferral=True)` hands the parked job back instead.
The CLI echoes the global `--palace` back into the command it suggests, so the
suggestion does not silently list the default palace's queue instead of the
one the job is parked in. A job that is genuinely running is still waited out.
Co-Authored-By: mjvmsteixeira <185609735+mjvmsteixeira@users.noreply.github.com>
Igor Lins e Silva