* fix(lancedb): bound reads and unhook the husk sweep from the write lock
Two instances of the same defect class the write-lock deadline work left
behind: an await in a scheduler-gated path with nothing bounding it.
Reads (count / get_by_id / find_where / find_where_paginated / search) were
skipped last round on the reasoning that a read takes no lock and so blocks
no writer. True, but incomplete: the cascade drain loop reads on every batch
and advances strictly one batch at a time, so a read that never returns stops
the whole md -> LanceDB projection. Claimed rows stay `processing` forever
(claim_pending_batch only takes `pending`, orphan recovery runs once at
startup), and /health keeps reporting healthy because a hang raises nothing.
Budget 60s, ~1000x the measured 62ms flat scan over 117k rows.
The empty-index-dir sweep ran inside the prune critical section under a
docstring contract requiring the write lock. That contract could not hold: the
sweep runs via asyncio.to_thread, and a deadline cancels the future, not the
thread, so an orphan sweep outlives the lock -- and Path.iterdir is a lazy
os.scandir, so it can yield a dir created after the scan began. It could
therefore rmdir a directory a concurrent create_index had just made, leaving
the table with no FTS index and every search on that kind 500ing. Safety now
comes from an age filter (skip dirs younger than 300s), which holds regardless
of lock ownership; the sweep moved out of the critical section so a slow
filesystem walk can no longer overrun the prune budget.
Mutation-verified: moving the table handle back outside the read deadline
hangs the new test; moving the sweep back inside the lock fails it; setting
the age floor to 0 fails the fresh-dir test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(cascade): supervise background loops and unmask the optimize alert
The drain / heartbeat / rebuild loops were plain create_task coroutines. One
uncaught exception ended that loop permanently: nothing restarted it, and
because the worker holds a strong reference to the task the interpreter never
printed "Task exception was never retrieved" either (that fires on GC). The
loop's job just stopped happening with zero output. _run_loop had an inner
try; its two siblings did not.
Each loop now runs under _supervise: log, wait, restart with escalating
backoff (5s / 15s / 45s), then request process exit via SIGTERM so a
restarting supervisor (systemd Restart=always, Docker restart:
unless-stopped, a k8s Deployment) can recover it. SIGTERM rather than
os._exit so the ASGI server runs its graceful-shutdown path. A done-callback
is the last-resort observer for the supervisor itself ending unexpectedly.
Separately, the fallback rebuild reset the same counter the health verdict
reads, so the optimize-failure threshold was effectively unreachable: a table
failing 100% of the time cycled 1..5 -> 0 -> 1.. and the threshold value
existed only during the sub-second rebuild, ~1% observable against a 30s
scrape. cascade.healthy stayed green while the table never reclaimed a
version. The rate limiter moves to failures_since_fallback; only a successful
optimize clears the alert streak. Same shape as the run7 cross-kind max()
masking bug -- a remediation path refreshing the signal meant to report it.
Mutation-verified: dropping the restart budget, removing the exit request,
and restoring the counter reset each turn the corresponding test red.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(persistence): bound and announce the memory-root lock wait
Acquisition polls with LOCK_NB instead of blocking inside a worker thread. A
blocking flock could be neither bounded nor cancelled: cancelling the awaiting
coroutine leaves the thread to acquire the lock later with nobody left to
release it, which is strictly worse than waiting.
The wait itself is correct by design -- the second process is supposed to wait,
then find the migration already done -- and flock is released by the kernel on
process exit, so a dead holder never wedges it. What was wrong is that it had
no upper bound and emitted nothing: a server startup landing on a held lock
looked like a hang whose last log line was lifespan_provider_startup
name=lancedb. It now logs memory_root_lock_waiting on first contention,
reports how long it waited on success, and gives up after timeout_seconds
(default 300s, generous because the legitimate holder is an O(rows) migration).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test(cascade): wait for projection quiescence in the lap-race scenario
test_lap_append_during_handler_no_loss asserted no loss after
_wait_path_done, whose settle window is 0.1s. That is a bet that the
filesystem event for the appends which landed *during* a handler
invocation has already been delivered — a terminal row does not mean the
file is fully projected, because the handler read the md at whatever
length it had then, marked the row done, and the rest arrive on a later
event. The bet holds on macOS/fsevents and lost on a loaded Linux runner
(md=30 lance=17), failing the assertion for a reason unrelated to the
behaviour under test.
Waits for quiescence instead: terminal row + empty pending queue + a
projected count unchanged across three consecutive polls. Strictly
stronger than the old condition, and real loss still fails — the count
just converges below the md entry count and stays there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(lancedb): replace indexes in place and retry a lost rebuild race
rebuild_indexes dropped every index and recreated it. The docstring justified
that with "LanceDB transparently falls back to brute-force scan", which is true
for vector search and false for FTS: with no inverted index a BM25 query raises
"Cannot perform full text search unless an INVERTED index has been created".
The recall legs are gathered without return_exceptions, so one failing leg
fails the whole search request -- every keyword search landing in the window
returned 500. Measured: 55 failures across 3 rebuilds with drop+create, 0 with
create_index(replace=True). Replacing also collapses the live fragment set
identically (7 index files back to 4 after 25 optimize beats), so nothing the
rebuild existed for is lost. Only indexes on columns that are no longer indexed
at all are still dropped -- nothing queries those.
A rebuild that loses the manifest race is now retried rather than deferred to
the next 12h sweep. Lance marks the conflict Retryable and means it: another
process committed first. Retries are recorded as a deadline on the kind
(10min / 30min / 3h) and picked up by the rebuild loop, not slept through --
the loop walks kinds sequentially, so sleeping would park every later kind
behind the backoff (7 kinds x 3h outlasts the cadence itself). The loop tick is
min(60s, cadence) so a shorter configured interval is not quantised.
Removes the empty-index-dir sweep. cleanup_older_than deletes the files under a
superseded _indices/<uuid>/ but leaves the directory, and everos was removing
those with its own rmdir. No LanceDB contract says an empty index dir is
garbage, so this is being raised upstream instead. Note it is a separate gap
from index *files* not being reclaimed under delete_unverified=False (260MB
retained on a 19k-row soak table) -- solving that still leaves the empty dirs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs(changelog): put only the sweep under Removed
The rebuild-bound and lock-wait entries were swallowed into the Removed
section when the husk-sweep entry was inserted above them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(lancedb): bound the husk sweep by lance's own unverified threshold
Restores the empty-index-dir sweep with a safety argument instead of a
self-chosen number. Upstream context, read from lance's cleanup.rs: it unlinks
a superseded index's files but never the directory, and contains no rmdir at
all. That is structural, not an oversight -- lance targets object stores, where
paths are flat keys and an empty directory does not exist. Only a local
filesystem materialises them, where they accumulate as inodes (a soak run
reached 13061 dirs, 98% empty) and slow every directory scan.
Three independent guarantees, in order of strength:
1. rmdir cannot delete data. The kernel refuses it on a non-empty directory,
so no file can be lost whatever the rest of the logic decides -- and because
the check *is* the operation, there is no check-then-act window to race.
2. Live indexes are excluded by UUID, read from list_indices().
3. Anything else must outlive UNVERIFIED_THRESHOLD_DAYS = 7, which is lance's
own bound for deciding an unreferenced index UUID is dead rather than an
index build in progress. Matching it means the sweep can never be more
aggressive than lance itself; the previous 300s was our invention, and that
is what made it indefensible.
Each guarantee is pinned by the same test and mutation-verified: dropping the
live-UUID check, zeroing the age gate, and swapping rmdir for a recursive
delete each turn it red.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat(cascade): make the maintenance cadences configurable
The four cadences were already constructor arguments on CascadeWorker, but
CascadeConfig did not carry them and none of the three production construction
paths passed a config, so the module defaults were unreachable from outside the
code. That is why no soak run shorter than half a day could exercise the 12h
rebuild sweep: not a missing parameter, a config layer that dropped it.
Adds CascadeSettings ([cascade] in default.toml) and CascadeConfig.from_settings,
which the orchestrator now uses when no config is passed -- so the CLI, backfill
and server paths all pick settings up at once.
Deliberately not exposed: the read / write / prune / rebuild deadlines. Those
are hang-catchers sized from measured durations, and both directions are worse
-- too low manufactures failures on a healthy table, too high leaves a wedged
one invisible for longer. Cadences depend on write volume and are a real tuning
axis; deadlines are not. A test pins the exposed field set so a later change has
to state its intent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test(cascade): cover the AgentCase handler
agent_case was the one business kind with no handler test, and the storage
soak never writes it either -- four of the seven tables stay at zero rows
there -- so its md -> row contract was unexercised from both directions.
Covers what makes this kind different from its daily-log siblings: it lives
on the agent track, and it embeds task_intent only while approach is
BM25-indexed but deliberately never sent to the embedder. Plus the branches
every handler shares: soft-dependency embedding (no provider -> vector=None,
row still written for keyword-only deployments), optional KeyInsight, the
content_sha256 short-circuit that stops the 30s scanner re-embedding
untouched files, edit detection, and delete-by-path.
Mutation-verified: routing approach into the embedder, and dropping
section:TaskIntent from content_change_keys, each turn the relevant test red.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs(lancedb): record the empty-dir cost and what caps it
Two constants now carry the arithmetic instead of leaving it in a chat log.
_HUSK_MIN_AGE_SECONDS states what the 7-day gate costs: nothing reclaims an
empty index dir before then, each is an inode plus a 4KB block, and at ceiling
load that is ~890k dirs / ~3.6GB / 14% of a default 98GB ext4's inodes at the
7-day steady state. Also that this is the worst case and needs sustained
saturation -- a single-user deployment sits four orders of magnitude below it --
and that only ext4 has a fixed inode budget (APFS and xfs allocate
dynamically, Windows is out of scope).
DEFAULT_OPTIMIZE_MIN_INTERVAL_SECONDS gets two things it never said. First, it
is not a visibility delay: a row is searchable as soon as its upsert commits,
because LanceDB flat-scans the unindexed tail -- verified to cover BM25, not
just vector and scalar, which was the leg worth doubting given a missing FTS
index hard-fails rather than degrading. Sparse writes do not wait at all, since
the scheduler uses max(0, interval - elapsed). Second, it is the ceiling on
index-directory growth: past roughly one write per table per interval the beats
coalesce, so the accrual rate is capped by this interval rather than by write
volume, and raising it lowers the cost proportionally. That makes it the knob
to reach for if the empty dirs ever bite -- not the husk threshold.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(cascade): reset the loop-restart budget after a stable run
The supervisor's restart budget was spent per process lifetime: three
strikes ever, regardless of how long the loop ran healthily between
them. A loop hitting one recoverable transient every few days — each
cleared by a single restart — would still pool those strikes and
SIGTERM a healthy server weeks in, on the 4th, which punishes exactly
the case supervision exists to absorb.
The budget now counts consecutive quick crashes: a body that ran at
least 60s before raising starts a fresh incident with the full ladder.
A deterministic crash-on-entry still exhausts the budget in ~65s. Same
windowed counting as systemd StartLimitIntervalSec / Erlang
max_restarts-per-max_seconds.
Also corrects the husk accrual numbers in the optimize-cooldown
docstring to the ~14-day effective reclaim horizon (see the sibling
lancedb commit for why the age gate doubles).
Mutation-verified: with the reset removed, the new test exits after
run 3 instead of surviving to run 5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(lancedb): keep a husk-sweep timeout from failing the prune
By the time the sweep runs, the cleanup commit — the thing prune exists
for — has already succeeded, and the sweep is best-effort by contract.
Letting its deadline escape prune() billed the failure to the wrong
account: the optimize scheduler counted a prune failure (feeding the
fallback-rebuild threshold) and the prune-staleness clock stopped
advancing, so both alarms reported a cleanup stall that did not happen.
Same defect shape as the alert counter the fallback rebuild used to
zero: an auxiliary path corrupting the main signal's ledger.
Reachable, not theoretical: sweep time is proportional to dir count
(~35us/dir measured) and the ceiling-load steady state sits right at
the 60s budget. The timeout is tolerable exactly because it is now
swallowed — and the orphaned worker thread finishes the walk anyway, so
the reclamation still happens.
Also corrects the age-gate docstrings: the gate reads st_mtime, which
POSIX bumps when lance's cleanup empties the husk, so the effective
reclaim horizon is file wait + 7 days (~14 days total) and the
ceiling-load steady state is ~1.8M dirs / ~7GB, twice the previously
recorded figure. The "never more aggressive than lance" property is
unaffected (it is strictly more conservative).
Mutation-verified: with the try/except removed, the new test fails on
the escaping VectorStoreBusyError.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(persistence): move the lock timeout an order above the legit hold
300s sat at the edge of what the docstring itself calls a legitimate
hold (a large migration is minutes of O(rows) work), so the worst
honest migration turned every waiting process's startup into a
LockError crash. Now 1800s: the wait has been visible since the first
poll (memory_root_lock_waiting), and against the one case the bound
exists for — a holder alive but wedged — giving up at 5 minutes buys
nothing over 30, because the timeout's job is diagnosis, not recovery.
The timeout message now says which way to look: the kernel releases a
dead holder's flock automatically, so reaching the timeout means the
holder is alive — inspect that process instead of retrying this one.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(changelog): fold the review fixes into the unreleased notes
Supervisor bullet gains the per-incident budget, the sweep bullet gains
the ~14-day effective horizon and the swallowed timeout, and the lock
bullet records the 30min default with its sizing rationale.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(lancedb): pin that only a sweep timeout is absorbed by prune
The sibling test covers one side -- a deadline miss must not bill prune's
ledger. Nothing covered the other: widening the catch to `except Exception`
passes every other test in the file, and would turn a genuine fault in
_remove_empty_index_dirs (a TypeError after a signature change, a permission
error on the index dir) into a silent removed = 0 with no signal anywhere.
That is the failure shape this module keeps being audited for, so the
narrowness of the catch needs its own guard.
Found by mutating the catch rather than by reading it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(cascade): bound the optimize runner's wait on a rebuild
The two maintenance jobs park on each other -- whichever arrives second waits
on the first -- so the two waits are one hazard seen from opposite ends. The
rebuild side was bounded; this side was left open on the argument that
rebuild_indexes carries its own 300s deadline. That deadline covers its
critical section, not the task's dispatch and teardown around it, so the
transitive bound was never real.
While the runner waits, its per-kind task slot stays occupied, every
_schedule_optimize call short-circuits on it, and that table silently stops
being pruned -- the same shape as the stall this branch has been chasing.
Bounded at 180s. On expiry the beat is skipped rather than run: compacting
under a live rebuild is the interleaving the wait exists to prevent, and both
commit on the same manifest. Writes keep the dirty flag set, so the next beat
retries.
Mutation-verified: replacing the timeout with a plain await hangs the new test
until its own guard fires.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: zhanghui <zhanghui@shanda.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
8024fe576e