* fix(cascade): per-kind prune staleness + rebuild safety Adversarial review of the review-response fixes found five real defects, all in code this PR introduced. Health signal (P1): prune staleness reported the time since the NEWEST successful prune across kinds, so on a multi-kind deployment (every real one) a single kind whose cleanup died was masked by the others pruning on schedule — /health stayed green while that table's index dir grew unbounded, the exact incident the signal exists to catch. Report the WORST kind instead and name it in the reason. The failure streak could not cover this either: an intervening light beat resets it, so it never reaches the threshold for a prune-only failure. Documented that split of duties. Spurious fallback rebuilds (P1): the benign-conflict carve-out excluded the heavy beat, justified by "runs under the write lock, so it can't hit this benignly" — but that lock is in-process only, so a second process (a long `cascade backfill`, a `cascade sync`) preempts prune's Rewrite commit. Those counted as real failures, and ~25min of cross-process churn reached the threshold and fired a fallback rebuild, which drops every index before recreating it; a rebuild that also lost the race was swallowed as a warning, leaving the table with no FTS index (every /search on that kind 500s) until the next 12h sweep. Treat commit conflicts as benign on both beats and let prune-staleness detect a prune that genuinely stops succeeding. cascade rebuild (P1 ×2 + P2): it drops and recreates tables with no guard while --help/docstrings advertised it as safe, so `rebuild --yes` against a live daemon corrupted the rebuild (the daemon keeps writing through cached handles). Refuse when the OME jobstore lock is held, reusing backfill's detection and its exit code 3. It also ran the pre-drop migration pass (`ensure_business_indexes`) against the damaged table, so on the corruption classes it exists to repair (missing column, un-alterable type) the recovery path died on the damage itself — skip it via `_runtime(ensure=False)`. Reset the queue BEFORE dropping so every crash window converges on "queue pending → re-index" instead of empty tables with a fully-done queue (a silently empty deployment), and handle Ctrl-C with exit 130 plus a resume hint. Recovery guidance (P1): the nullable-vector migration error still told users to wipe the index directory — which this PR's own runbook documents as the wrong recovery (queue stays done, index comes back empty). Point it at `everos cascade rebuild`. Dropped the schema-drift error's "restart first" step too: the startup migrations only alter nullability, never a name or type, so a name/type drift never self-heals. Also: backfill's post-write prune passed a zero retention window from a separate process, able to delete files under a daemon /search still holding that version — pass the daemon's window instead. Runbook gains the /health cascade block (thresholds, what flips healthy, why failed_permanent does not) and its quoted schema-drift error now matches the code. Tests: the three safety mechanisms this PR adds were unpinned — a one-line revert of any of them passed the suite. Added per-kind staleness, heavy-beat benign conflict, benign-filter negative case (an error whose message merely contains "retryable" must still count), prune recurrence across light beats (mutation-verified: hoisting the attempt-clock advance out of the heavy branch turns it red), the prune timeout releasing the write lock, timeout-below-cadence, the rebuild server guard, and a tier3 assertion that the /health cascade block is actually wired. Froze the last fabricated-monotonic test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Also: treat ``last_prune_attempt_at == 0.0`` as "never attempted" instead of comparing clocks. ``monotonic()`` is boot-relative, so ``now - 0 >= cadence`` is false for the first ~cadence of container uptime — the catch-up prune was skipped exactly when a fresh process most needs it (and made a test depend on the runner uptime, which CI caught). * fix(lancedb): bound every write-lock critical section run7 (1h at 2.5x rate, concurrent CLI maintenance, doubled fuzz) reproduced a table whose version cleanup stopped permanently: 150 versions retained, disk 11x live size, while the other two tables sat at 1 version each — and with no error logged anywhere, because nothing failed. It simply never returned. Three things combined. The maintenance scheduler allows one task per table (a LanceDB table takes one writer), so it skips a kind whose task is still in flight. The prune timeout sat *inside* the lock and covered only the cleanup call. And the other six critical sections on that lock — add, upsert, update, delete, delete_by_md_path, rebuild_indexes — had no deadline at all. So one operation stuck anywhere outside that narrow window wedged the table for good: every writer blocked on acquire, and every later heartbeat was turned away because the stuck task never finished. Make it structurally impossible instead of patching prune: all seven sections now go through `LanceRepoBase._locked(budget, op)`, where the deadline covers **acquisition and the body**. No path can wait for this lock, or hold it, indefinitely. Budgets are hang-catchers, not throughput limits: 120s for row writes, 600s for an index rebuild, the existing 60s for prune. Expiry raises `VectorStoreBusyError`, deliberately under `ExternalServiceError` so the cascade worker retries the row; under `VectorStoreError` a transient lock contention would be marked permanently failed and need a manual `cascade fix`. Tests: a stuck holder now makes a waiter fail its deadline and release (the lock is reusable afterwards), and the prune timeout is pinned as retryable. Verified by mutation — moving the timeout back inside the lock makes a waiter block until the enclosing observation window expires (1001ms vs 51ms), i.e. wait forever in production. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(lancedb): size write-lock budgets from measurements 120s for a row write was a guess, and a bad one: the budget doubles as the detection latency for a wedged table, so an over-slack value means minutes of blocked writers before anything surfaces — the failure this change exists to prevent. Measured the four locked write ops on a local SSD across table sizes and batch sizes (10k-100k rows, 50-500 rows per call): add 3-22ms, upsert (merge_insert, the read-modify-write one) 6-25ms, update 2-4ms, delete 2-3ms; worst observation 63ms, and flat in both dimensions since these are append-and-commit, not scans. So: writes 120s -> 15s (~240x the worst observation, enough for a contended disk and several waiters queued ahead — the deadline includes acquisition and asyncio.Lock is FIFO), rebuild 600s -> 300s (still the one genuinely slow section at ~0.3s per 50k rows per indexed column). Prune stays 60s. Test pins the sizing intent: writes stay in the tens of seconds, and rebuild > prune > write so the slowest section is not the most eagerly killed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(lancedb): record wait/hold time on write-lock critical sections A soak run stalled one table's writes for ~16s and the logs could not say why: maintenance beats only log at `debug`, so a section that is slow but still inside its deadline is invisible, and the timeout warning did not distinguish "never acquired the lock" from "acquired it and overran". `_locked` now carries that apart. The deadline warning gains `acquired`, `waited_seconds` and `held_seconds` — `acquired` alone answers whether a holder was slow or this operation was — and a completed section that held the lock for at least a second logs `lancedb_write_lock_slow_hold` at info, so a stall that never reaches a deadline still leaves a trace. Uses `time.monotonic` (elapsed measurement, not wall clock — the datetime discipline bans `time.time`). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(search): reject a mismatched query vector before it reaches LanceDB A soak run showed every slow search was a failing search: `search:vector` p50 251ms / p99 1.6s, but its 8 requests over 10s were exactly its 8 failures (13-14s each). The cause was a query vector whose width disagreed with the index. LanceDB only notices after the query is built and reports it as an opaque `ValueError: Invalid input, No vector column found to match…`, which escaped as an unhandled 500. Validate at `_embed_query` — the single point every query vector passes through — against the provider's declared `dim`. Microseconds instead of 13s, and a named `ConfigurationError` (500 + CONFIGURATION_ERROR) instead of an unhandled crash. Deliberately not `InvalidInputError`/422: callers only send query *text*, so a bad width is our provider's fault, not the caller's. Also cap traceback rendering. structlog's default is `RichTracebackFormatter(show_locals=True, max_frames=100, extra_lines=3)`, which on an async stack rendered 82 frames into 6423 log lines per exception — 85MB of server.log across 11 of them — at ~290ms of synchronous CPU each, and risks printing request payloads into logs. With locals off and 15 frames the same traceback is 103 lines and 10ms. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(changelog): record the storage-reliability work under Unreleased #379 merged without changelog entries, so this covers both it and the follow-up work in this branch: the maintenance split (compaction vs reclamation) that fixes unbounded index growth, bounded write-lock critical sections, the /health cascade readiness block and its alert contract, `cascade rebuild`, schema type-drift detection, the query-vector width check, and the traceback-rendering cap. Each entry states the operator-visible consequence, not just the change — `cascade rebuild` now refusing to run against a live server, benign-conflict warnings dropping in volume, and `/health` being able to report a stalled kind that was previously invisible are all behaviour changes someone will notice. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: zhanghui <zhanghui@shanda.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
16 KiB
Cascade Runbook
The cascade daemon keeps LanceDB in sync with the markdown files under the memory root. Service / entry points only ever write markdown; the daemon is the sole writer of the LanceDB index. This runbook covers the recurring operational questions.
What runs where
When everos server start boots, the FastAPI lifespan wires six
providers in order:
- Metrics — Prometheus collector.
- LLM — LLM client initialisation.
- SQLite — system DB + schema (
SQLModel.metadata.create_all). - LanceDB — async connection + schema verification + FTS indexes.
- Cascade — watcher + scanner + worker, all in-process tasks.
- OME — offline memory engine.
The cascade subsystem itself is three independent loops:
| Loop | Source signal | Effect |
|---|---|---|
| Watcher | watchdog filesystem events (sync thread) |
md_change_state.upsert per registered kind |
| Scanner | Periodic walk (scan_interval_seconds, default 30 s) |
Same — catches changes the watcher missed |
| Worker | claim_pending_batch polling (default 1 s when idle) |
Handler dispatch → LanceDB upsert / delete |
Every loop talks to the same md_change_state sqlite table. The
worker's claim mode (pending → processing → done/failed) keeps
concurrent workers honest.
Health: everos cascade status
queue:
pending: 3
done: 1247
failed (retryable=TRUE): 1 (eligible for `cascade fix --apply`)
failed (retryable=FALSE): 1 (fix md and re-save to recover)
lsn:
max: 1252
last_processed: 1250
lag: 2
lag > 0means the worker is behind. Steady state should hover near zero; sustained lag points at a slow handler or a stuck retry.failed (retryable=FALSE)is always user-actionable. Cascade will never auto-clear these — they represent malformed md the user must edit.
Machine-readable: the cascade block on GET /health
GET /health carries a cascade block while the daemon runs (null
for an app built without the cascade lifespan). Alert on
cascade.healthy — it is the operational readiness verdict, and
reasons explains a false in plain text:
{"status": "ok", "cascade": {
"healthy": false,
"reasons": ["version cleanup stalled for kind 'episode' (1200s since its last prune — that table's index dir may grow)"],
"pending": 3, "failed_permanent": 1, "failed_retryable": 0,
"drain_consecutive_failures": 0, "unrecoverable_total": 4,
"optimize_failure_streak": 0, "prune_stale_seconds": 1200.0}}
What flips healthy false — and nothing else does:
| Symptom | Signal | Threshold |
|---|---|---|
| Writes accepted but not projected to LanceDB | drain_consecutive_failures |
≥ 3 in a row |
| Index maintenance wedged | optimize_failure_streak |
≥ 5 in a row (lost commit races excluded — they are expected under churn) |
| Version cleanup stopped, that table's disk will grow | prune_stale_seconds (worst kind, named in reasons) |
≥ 900s (3 missed 300s beats) |
failed_permanent is informational only — it is a data-quality
backlog awaiting cascade fix, so it never flips healthy (otherwise
the signal sits red until a human edits md). Watch it separately.
The HTTP code is a liveness signal and stays 200 even when the block
says healthy: false — a degraded projection must not trigger a
container restart, which fixes neither a bad md file nor disk bloat. If
the probe itself fails (locked / full SQLite), the block comes back
healthy: false with a cascade health probe failed: … reason and the
counters zeroed — treat zeros alongside that reason as "unknown", not
as "clean".
Recovering from failures: everos cascade fix
cascade fix (no flag) lists every failed row. With --apply:
UPDATE md_change_state SET status='pending', retry_count=0 WHERE status='failed' AND retryable=TRUE(the partial indexidx_md_change_retryablemakes this O(retryable)).- Drain the worker once so the retry runs synchronously.
Retryable failures cover transient embedding / HTTP errors (5xx, 429,
network resets) after the inline MAX_RETRY=3 was exhausted. The
fix command resets the counter so a working backend gets a clean
start.
retryable=FALSE rows require the user to edit the md (typically a
YAML frontmatter issue) and re-save; the watcher picks the change up
naturally.
One-shot replay: everos cascade sync [PATH]
Use this when the watcher missed an event (WSL mount, network share, external editor with no inotify) or when you want a deterministic flush before, say, a smoke test:
everos cascade sync # drain everything pending
everos cascade sync users/u1/episodes/X.md # re-enqueue + drain
The CLI builds the same CascadeOrchestrator as the daemon but only
calls sync_once / drain_once — no watcher / scanner background task.
Its drain still runs the same compaction + version-cleanup (prune) as
the daemon, but prune uses delete_unverified=False, so it never
deletes a file another process may be mid-commit on. Safe to run in
parallel with a live everos server.
Rebuild the index: everos cascade rebuild
The safe recovery from a drifted or corrupt LanceDB index. It rebuilds the whole index from markdown (the source of truth) in one shot:
everos cascade rebuild # prompts for confirmation
everos cascade rebuild --yes # non-interactive
Stop the
everos serverfirst. Unlikecascade sync, rebuild drops and recreates the LanceDB tables. A running daemon holds cached table handles that would keep pointing at (and writing to) the dropped dataset, corrupting the rebuild. This is the one cascade command that is not safe to run alongside a live server.
What it does, in order:
- Drops every business LanceDB table (
drop_business_tables) and evicts them from the connection cache. - Recreates them empty from the current schema + FTS indexes
(
ensure_business_indexes). - Clears the cascade queue (
md_change_state.reset_all) so every md file re-enqueues asaddedon the next scan. - Re-scans + drains (
sync_once): re-embeds and re-inserts every md entry.
It deliberately skips verify_business_schemas — the drift it
recovers from would otherwise trip that guard on startup before the
rebuild could run (chicken-and-egg).
Why not a bare rm:
| Recovery | Re-populates done entries |
Preserves unprocessed_buffer |
|---|---|---|
rm -rf .index/lancedb |
❌ scanner skips done rows → empty index |
✅ |
rm -rf .index |
✅ | ❌ deletes un-extracted messages |
everos cascade rebuild |
✅ | ✅ |
Recovery paths
LanceDB schema drift on startup
LanceDBLifespanProvider.startup calls verify_business_schemas. If
an on-disk table has columns the current Pydantic schema does not
declare (or vice versa), the boot fails with:
LanceDB table 'episode' schema drift: missing=[...], extra=[...],
type_drift=[...]. Recover with `everos cascade rebuild` (stop the server
first): it drops and re-indexes from md, preserving un-extracted buffered
messages. Restarting will not clear this — the startup migrations only
alter column nullability, never a column's name or type, so a name/type
drift never resolves on its own.
verify_business_schemas compares both the column names and their
Arrow types against the current schema. Catching type drift matters:
an episode.subject_vector column left as string (or null) by an
older build, while the schema now declares a 1024-d fixed_size_list,
has the same column name — so a name-only check would wave it through
and it would detonate later inside merge_insert as an opaque
LanceError(IO): Spill has sent an error (EverOS #337). The type check
turns that into this clean startup error.
Recover with everos cascade rebuild (documented above). Do not just
rm -rf ~/.everos/.index/lancedb: that clears the vectors but leaves
md_change_state marked done, so the scanner skips every already-
indexed file and the index comes back empty. And do not
rm -rf ~/.everos/.index: that also deletes unprocessed_buffer
(messages received but not yet extracted — not rebuildable from md).
cascade rebuild is correct on both counts. Markdown is the source of
truth, so no memory content is lost.
inotify watch-limit exhaustion (Linux)
Default kernel limit is 8 192 watches per user. On a sizeable memory root the watcher may silently miss events. Symptoms:
- Scanner catches the file changes but the watcher never logs an event for the same path.
cat /proc/sys/fs/inotify/max_user_watchesis at the limit.
Fix by bumping the kernel parameter:
echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf
sudo sysctl -p
WSL2 / network mounts
Filesystem events do not propagate from the Windows host into WSL2 (or across most SMB / NFS shares). The watcher will start without error and silently see nothing.
Workarounds:
- Rely on the scanner — at default 30 s interval, throughput is bounded but eventually-consistent.
- Drop the scan interval to ~5 s if the memory root is small.
- Run
everos cascade syncexplicitly after batch edits.
Daemon process crash mid-batch
claim_pending_batch flips rows to processing atomically. If the
process dies before mark_done / mark_failed, those rows stay in
processing until the next boot. The orchestrator auto-recovers
on startup: CascadeOrchestrator.start calls
md_change_state_repo.recover_orphan_processing() before launching
the watcher / scanner / worker, which resets every processing row
back to pending. Single-process cascade means no race — at boot
time no other worker could legitimately own a processing row.
No operator action required; the structured log line
cascade_recovered_orphan_processing reports the count when it
fires.
FD exhaustion (os error 24 / EMFILE)
Symptoms (any of these on a long-running daemon):
- LanceDB query / index build fails with
lance error: ... Too many open files (os error 24). lsof -p <pid> | wc -lgrows monotonically over hours / days.- Health log lines like
cascade_lancedb_optimize_failed/cascade_lancedb_rebuild_failedcarryingOSError: [Errno 24].
Cause (verified against lance crate 4.0): the LanceDB index cache
(GlobalIndexCache) holds one reader object per opened FTS / vector
/ scalar index, and each reader pins the file descriptors of its
_indices/<uuid>/... files. With a long-running daemon and steady-
state cascade ingest, every optimize() call adds new readers; with
LanceDB's own default (index_cache_size_bytes=None, unbounded), they
are never evicted and the FDs leak monotonically.
drop_index does not help — it is a manifest-only operation and
leaves the on-disk UUID directories untouched. Even an explicit
optimize(cleanup_older_than=0) unlink()-ing the files does not
release FDs: POSIX keeps the inode alive as long as a process holds
an open FD on it (the entries show as (deleted) in lsof). Only an
LRU eviction inside the cache (or a connection close) actually closes
the FDs.
Fix (already wired in LanceDBSettings.index_cache_size_bytes —
default 16 MB, ~290 FD ceiling): see
Tuning knobs § LanceDB index cache
for the sizing table and the env-var override path.
If you have already hit EMFILE in a running process, the cleanest recovery is a daemon restart — the open connection closes, every FD is released, and the next start comes up with the capped Session in place.
Tuning knobs
Cascade scheduler knobs
All defaults live in everos.memory.cascade.orchestrator.CascadeConfig
and everos.memory.cascade.worker.CascadeWorker:
| Knob | Default | Effect |
|---|---|---|
scan_interval_seconds |
30 | Scanner sweep cadence |
worker_batch_size |
50 | Rows claimed per worker cycle |
worker_max_retry |
3 | Inline retries before mark_failed(retryable=TRUE) |
worker_poll_interval_seconds |
1 | Idle wait between empty drain attempts |
worker_retry_backoff_seconds |
2 | Linear backoff seed; doubles per attempt |
Tuning surface is intentionally not in Settings yet — once we have
wall-clock numbers from real workloads, the values that need
operator override will surface there.
LanceDB index cache (index_cache_size_bytes)
Lives in LanceDBSettings; overridable via the
EVEROS_LANCEDB__INDEX_CACHE_SIZE_BYTES environment variable. This
is the only knob that bounds the steady-state file-descriptor count
of a long-running EverOS daemon — see
Recovery paths § FD exhaustion
for why nothing else (prune, rebuild, drop_index) helps.
Measured cap → FD ceiling (30 add+optimize cycles + 100-query stress
on the real Episode schema):
| Cap | FD ceiling | Query latency (p50) | Safe under ulimit -n |
|---|---|---|---|
2 MB |
~45 | ~5 ms | macOS default 256 (5× headroom) |
4 MB |
~52 | ~3 ms | macOS default 256 |
8 MB |
~140 | ~2.4 ms | macOS default 256 (1.8× headroom) |
16 MB (default) |
~290 | ~2.3 ms | Linux default 1024 (3.5× headroom); macOS needs ulimit -n 1024 |
32 MB |
~630 | ~1.4 ms | Linux default 1024 (1.6× headroom) |
unbounded |
grows forever | ~1.3 ms | NEVER use in a daemon |
EverOS's measured steady-state working set after a rebuild_indexes
cycle is roughly 50-100 readers / 3-6 MB resident (5 tables × ~7
BM25 columns × ~10 part_N reader entries each), so the 16 MB default
provides ~3× headroom for burst traffic and stale-but-not-yet-evicted
readers.
When to override:
- Tight
ulimit -nenvironments (containers; macOS dev boxes that haven't bumped the default 256) → drop to4 MBor8 MB. Query latency increases by ~1-3 ms but correctness is unaffected. - Larger working sets (many more tables or much wider FTS
indexes than the default schema set) → bump to
32-64 MB. Verify your platform'sulimit -ncovers the corresponding FD ceiling with at least 2× headroom. - Diagnostic-only: set to a tiny value (e.g.
1 MB) to force LRU thrashing and reproduce cache-miss latency in tests.
Do not set metadata_cache_size_bytes — it is intentionally left
at LanceDB's default (unbounded) because the metadata cache holds
parsed manifests / fragment stats and has zero effect on FD count;
capping it just thrashes parsing work without solving anything.
Concurrency
The worker is async, not multi-process. Inside one drain cycle,
asyncio.gather(*[_process_one(row) for row in batch]) runs every
claimed row concurrently — cascade is IO-bound (embedding HTTP calls
dominate wall time) so single-process coroutine concurrency saturates
the bottleneck. The worker_batch_size knob (default 50) caps
in-flight rows.
Multi-process workers are a scaling axis we'd reach for only if a
single process becomes CPU-bound, which the current design does not
anticipate. claim_pending_batch is already race-safe (the
WHERE status='pending' filter ensures each row lands in exactly
one batch even if multiple workers raced), so adding processes later
is a deployment-side change with no schema work.
What cascade does NOT do (yet)
- Schema migration: LanceDB has no in-place column migration; a
schema change is recovered by rebuilding from md (
everos cascade rebuild), not an automaticALTER. - Parent-id back-link: Episode rows currently carry
parent_id=None; the writer doesn't preserve the source memcell id in the entry inline. Tracked separately. - Reference-file change detection (agent_skill): edits to
references/*.mdsiblings won't trigger a re-index — only changes toSKILL.mditself fire the watcher. Workaround: runeveros cascade sync agents/<a>/skills/skill_<n>/SKILL.mdafter editing references.