Files
EverOS/docs/cascade_runbook.md
T
d256048a6d fix(cascade): per-kind prune staleness + rebuild safety (#384)
* fix(cascade): per-kind prune staleness + rebuild safety

Adversarial review of the review-response fixes found five real defects,
all in code this PR introduced.

Health signal (P1): prune staleness reported the time since the NEWEST
successful prune across kinds, so on a multi-kind deployment (every real
one) a single kind whose cleanup died was masked by the others pruning
on schedule — /health stayed green while that table's index dir grew
unbounded, the exact incident the signal exists to catch. Report the
WORST kind instead and name it in the reason. The failure streak could
not cover this either: an intervening light beat resets it, so it never
reaches the threshold for a prune-only failure. Documented that split of
duties.

Spurious fallback rebuilds (P1): the benign-conflict carve-out excluded
the heavy beat, justified by "runs under the write lock, so it can't hit
this benignly" — but that lock is in-process only, so a second process
(a long `cascade backfill`, a `cascade sync`) preempts prune's Rewrite
commit. Those counted as real failures, and ~25min of cross-process
churn reached the threshold and fired a fallback rebuild, which drops
every index before recreating it; a rebuild that also lost the race was
swallowed as a warning, leaving the table with no FTS index (every
/search on that kind 500s) until the next 12h sweep. Treat commit
conflicts as benign on both beats and let prune-staleness detect a prune
that genuinely stops succeeding.

cascade rebuild (P1 ×2 + P2): it drops and recreates tables with no
guard while --help/docstrings advertised it as safe, so `rebuild --yes`
against a live daemon corrupted the rebuild (the daemon keeps writing
through cached handles). Refuse when the OME jobstore lock is held,
reusing backfill's detection and its exit code 3. It also ran the
pre-drop migration pass (`ensure_business_indexes`) against the damaged
table, so on the corruption classes it exists to repair (missing column,
un-alterable type) the recovery path died on the damage itself — skip it
via `_runtime(ensure=False)`. Reset the queue BEFORE dropping so every
crash window converges on "queue pending → re-index" instead of empty
tables with a fully-done queue (a silently empty deployment), and handle
Ctrl-C with exit 130 plus a resume hint.

Recovery guidance (P1): the nullable-vector migration error still told
users to wipe the index directory — which this PR's own runbook documents
as the wrong recovery (queue stays done, index comes back empty). Point
it at `everos cascade rebuild`. Dropped the schema-drift error's
"restart first" step too: the startup migrations only alter nullability,
never a name or type, so a name/type drift never self-heals.

Also: backfill's post-write prune passed a zero retention window from a
separate process, able to delete files under a daemon /search still
holding that version — pass the daemon's window instead. Runbook gains
the /health cascade block (thresholds, what flips healthy, why
failed_permanent does not) and its quoted schema-drift error now matches
the code.

Tests: the three safety mechanisms this PR adds were unpinned — a
one-line revert of any of them passed the suite. Added per-kind
staleness, heavy-beat benign conflict, benign-filter negative case
(an error whose message merely contains "retryable" must still count),
prune recurrence across light beats (mutation-verified: hoisting the
attempt-clock advance out of the heavy branch turns it red), the prune
timeout releasing the write lock, timeout-below-cadence, the rebuild
server guard, and a tier3 assertion that the /health cascade block is
actually wired. Froze the last fabricated-monotonic test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Also: treat ``last_prune_attempt_at == 0.0`` as "never attempted" instead of
comparing clocks. ``monotonic()`` is boot-relative, so ``now - 0 >= cadence``
is false for the first ~cadence of container uptime — the catch-up prune was
skipped exactly when a fresh process most needs it (and made a test depend on
the runner uptime, which CI caught).

* fix(lancedb): bound every write-lock critical section

run7 (1h at 2.5x rate, concurrent CLI maintenance, doubled fuzz) reproduced a
table whose version cleanup stopped permanently: 150 versions retained, disk
11x live size, while the other two tables sat at 1 version each — and with no
error logged anywhere, because nothing failed. It simply never returned.

Three things combined. The maintenance scheduler allows one task per table (a
LanceDB table takes one writer), so it skips a kind whose task is still in
flight. The prune timeout sat *inside* the lock and covered only the cleanup
call. And the other six critical sections on that lock — add, upsert, update,
delete, delete_by_md_path, rebuild_indexes — had no deadline at all. So one
operation stuck anywhere outside that narrow window wedged the table for good:
every writer blocked on acquire, and every later heartbeat was turned away
because the stuck task never finished.

Make it structurally impossible instead of patching prune: all seven sections
now go through `LanceRepoBase._locked(budget, op)`, where the deadline covers
**acquisition and the body**. No path can wait for this lock, or hold it,
indefinitely. Budgets are hang-catchers, not throughput limits: 120s for row
writes, 600s for an index rebuild, the existing 60s for prune.

Expiry raises `VectorStoreBusyError`, deliberately under `ExternalServiceError`
so the cascade worker retries the row; under `VectorStoreError` a transient
lock contention would be marked permanently failed and need a manual
`cascade fix`.

Tests: a stuck holder now makes a waiter fail its deadline and release (the
lock is reusable afterwards), and the prune timeout is pinned as retryable.
Verified by mutation — moving the timeout back inside the lock makes a waiter
block until the enclosing observation window expires (1001ms vs 51ms), i.e.
wait forever in production.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(lancedb): size write-lock budgets from measurements

120s for a row write was a guess, and a bad one: the budget doubles as the
detection latency for a wedged table, so an over-slack value means minutes of
blocked writers before anything surfaces — the failure this change exists to
prevent.

Measured the four locked write ops on a local SSD across table sizes and batch
sizes (10k-100k rows, 50-500 rows per call): add 3-22ms, upsert (merge_insert,
the read-modify-write one) 6-25ms, update 2-4ms, delete 2-3ms; worst observation
63ms, and flat in both dimensions since these are append-and-commit, not scans.

So: writes 120s -> 15s (~240x the worst observation, enough for a contended disk
and several waiters queued ahead — the deadline includes acquisition and
asyncio.Lock is FIFO), rebuild 600s -> 300s (still the one genuinely slow
section at ~0.3s per 50k rows per indexed column). Prune stays 60s.

Test pins the sizing intent: writes stay in the tens of seconds, and
rebuild > prune > write so the slowest section is not the most eagerly killed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(lancedb): record wait/hold time on write-lock critical sections

A soak run stalled one table's writes for ~16s and the logs could not say why:
maintenance beats only log at `debug`, so a section that is slow but still
inside its deadline is invisible, and the timeout warning did not distinguish
"never acquired the lock" from "acquired it and overran".

`_locked` now carries that apart. The deadline warning gains `acquired`,
`waited_seconds` and `held_seconds` — `acquired` alone answers whether a holder
was slow or this operation was — and a completed section that held the lock for
at least a second logs `lancedb_write_lock_slow_hold` at info, so a stall that
never reaches a deadline still leaves a trace.

Uses `time.monotonic` (elapsed measurement, not wall clock — the datetime
discipline bans `time.time`).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(search): reject a mismatched query vector before it reaches LanceDB

A soak run showed every slow search was a failing search: `search:vector` p50
251ms / p99 1.6s, but its 8 requests over 10s were exactly its 8 failures
(13-14s each). The cause was a query vector whose width disagreed with the
index. LanceDB only notices after the query is built and reports it as an
opaque `ValueError: Invalid input, No vector column found to match…`, which
escaped as an unhandled 500.

Validate at `_embed_query` — the single point every query vector passes
through — against the provider's declared `dim`. Microseconds instead of 13s,
and a named `ConfigurationError` (500 + CONFIGURATION_ERROR) instead of an
unhandled crash. Deliberately not `InvalidInputError`/422: callers only send
query *text*, so a bad width is our provider's fault, not the caller's.

Also cap traceback rendering. structlog's default is
`RichTracebackFormatter(show_locals=True, max_frames=100, extra_lines=3)`,
which on an async stack rendered 82 frames into 6423 log lines per exception —
85MB of server.log across 11 of them — at ~290ms of synchronous CPU each, and
risks printing request payloads into logs. With locals off and 15 frames the
same traceback is 103 lines and 10ms.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(changelog): record the storage-reliability work under Unreleased

#379 merged without changelog entries, so this covers both it and the
follow-up work in this branch: the maintenance split (compaction vs
reclamation) that fixes unbounded index growth, bounded write-lock critical
sections, the /health cascade readiness block and its alert contract,
`cascade rebuild`, schema type-drift detection, the query-vector width check,
and the traceback-rendering cap.

Each entry states the operator-visible consequence, not just the change —
`cascade rebuild` now refusing to run against a live server, benign-conflict
warnings dropping in volume, and `/health` being able to report a stalled kind
that was previously invisible are all behaviour changes someone will notice.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: zhanghui <zhanghui@shanda.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-04 20:03:01 +08:00

16 KiB
Raw Blame History

Cascade Runbook

The cascade daemon keeps LanceDB in sync with the markdown files under the memory root. Service / entry points only ever write markdown; the daemon is the sole writer of the LanceDB index. This runbook covers the recurring operational questions.

What runs where

When everos server start boots, the FastAPI lifespan wires six providers in order:

  1. Metrics — Prometheus collector.
  2. LLM — LLM client initialisation.
  3. SQLite — system DB + schema (SQLModel.metadata.create_all).
  4. LanceDB — async connection + schema verification + FTS indexes.
  5. Cascade — watcher + scanner + worker, all in-process tasks.
  6. OME — offline memory engine.

The cascade subsystem itself is three independent loops:

Loop Source signal Effect
Watcher watchdog filesystem events (sync thread) md_change_state.upsert per registered kind
Scanner Periodic walk (scan_interval_seconds, default 30 s) Same — catches changes the watcher missed
Worker claim_pending_batch polling (default 1 s when idle) Handler dispatch → LanceDB upsert / delete

Every loop talks to the same md_change_state sqlite table. The worker's claim mode (pending → processing → done/failed) keeps concurrent workers honest.

Health: everos cascade status

queue:
  pending:                   3
  done:                      1247
  failed (retryable=TRUE):   1     (eligible for `cascade fix --apply`)
  failed (retryable=FALSE):  1     (fix md and re-save to recover)
lsn:
  max:           1252
  last_processed: 1250
  lag:            2
  • lag > 0 means the worker is behind. Steady state should hover near zero; sustained lag points at a slow handler or a stuck retry.
  • failed (retryable=FALSE) is always user-actionable. Cascade will never auto-clear these — they represent malformed md the user must edit.

Machine-readable: the cascade block on GET /health

GET /health carries a cascade block while the daemon runs (null for an app built without the cascade lifespan). Alert on cascade.healthy — it is the operational readiness verdict, and reasons explains a false in plain text:

{"status": "ok", "cascade": {
  "healthy": false,
  "reasons": ["version cleanup stalled for kind 'episode' (1200s since its last prune — that table's index dir may grow)"],
  "pending": 3, "failed_permanent": 1, "failed_retryable": 0,
  "drain_consecutive_failures": 0, "unrecoverable_total": 4,
  "optimize_failure_streak": 0, "prune_stale_seconds": 1200.0}}

What flips healthy false — and nothing else does:

Symptom Signal Threshold
Writes accepted but not projected to LanceDB drain_consecutive_failures ≥ 3 in a row
Index maintenance wedged optimize_failure_streak ≥ 5 in a row (lost commit races excluded — they are expected under churn)
Version cleanup stopped, that table's disk will grow prune_stale_seconds (worst kind, named in reasons) ≥ 900s (3 missed 300s beats)

failed_permanent is informational only — it is a data-quality backlog awaiting cascade fix, so it never flips healthy (otherwise the signal sits red until a human edits md). Watch it separately.

The HTTP code is a liveness signal and stays 200 even when the block says healthy: false — a degraded projection must not trigger a container restart, which fixes neither a bad md file nor disk bloat. If the probe itself fails (locked / full SQLite), the block comes back healthy: false with a cascade health probe failed: … reason and the counters zeroed — treat zeros alongside that reason as "unknown", not as "clean".

Recovering from failures: everos cascade fix

cascade fix (no flag) lists every failed row. With --apply:

  1. UPDATE md_change_state SET status='pending', retry_count=0 WHERE status='failed' AND retryable=TRUE (the partial index idx_md_change_retryable makes this O(retryable)).
  2. Drain the worker once so the retry runs synchronously.

Retryable failures cover transient embedding / HTTP errors (5xx, 429, network resets) after the inline MAX_RETRY=3 was exhausted. The fix command resets the counter so a working backend gets a clean start.

retryable=FALSE rows require the user to edit the md (typically a YAML frontmatter issue) and re-save; the watcher picks the change up naturally.

One-shot replay: everos cascade sync [PATH]

Use this when the watcher missed an event (WSL mount, network share, external editor with no inotify) or when you want a deterministic flush before, say, a smoke test:

everos cascade sync                           # drain everything pending
everos cascade sync users/u1/episodes/X.md    # re-enqueue + drain

The CLI builds the same CascadeOrchestrator as the daemon but only calls sync_once / drain_once — no watcher / scanner background task. Its drain still runs the same compaction + version-cleanup (prune) as the daemon, but prune uses delete_unverified=False, so it never deletes a file another process may be mid-commit on. Safe to run in parallel with a live everos server.

Rebuild the index: everos cascade rebuild

The safe recovery from a drifted or corrupt LanceDB index. It rebuilds the whole index from markdown (the source of truth) in one shot:

everos cascade rebuild          # prompts for confirmation
everos cascade rebuild --yes    # non-interactive

Stop the everos server first. Unlike cascade sync, rebuild drops and recreates the LanceDB tables. A running daemon holds cached table handles that would keep pointing at (and writing to) the dropped dataset, corrupting the rebuild. This is the one cascade command that is not safe to run alongside a live server.

What it does, in order:

  1. Drops every business LanceDB table (drop_business_tables) and evicts them from the connection cache.
  2. Recreates them empty from the current schema + FTS indexes (ensure_business_indexes).
  3. Clears the cascade queue (md_change_state.reset_all) so every md file re-enqueues as added on the next scan.
  4. Re-scans + drains (sync_once): re-embeds and re-inserts every md entry.

It deliberately skips verify_business_schemas — the drift it recovers from would otherwise trip that guard on startup before the rebuild could run (chicken-and-egg).

Why not a bare rm:

Recovery Re-populates done entries Preserves unprocessed_buffer
rm -rf .index/lancedb ❌ scanner skips done rows → empty index ✅
rm -rf .index ✅ ❌ deletes un-extracted messages
everos cascade rebuild ✅ ✅

Recovery paths

LanceDB schema drift on startup

LanceDBLifespanProvider.startup calls verify_business_schemas. If an on-disk table has columns the current Pydantic schema does not declare (or vice versa), the boot fails with:

LanceDB table 'episode' schema drift: missing=[...], extra=[...],
type_drift=[...]. Recover with `everos cascade rebuild` (stop the server
first): it drops and re-indexes from md, preserving un-extracted buffered
messages. Restarting will not clear this — the startup migrations only
alter column nullability, never a column's name or type, so a name/type
drift never resolves on its own.

verify_business_schemas compares both the column names and their Arrow types against the current schema. Catching type drift matters: an episode.subject_vector column left as string (or null) by an older build, while the schema now declares a 1024-d fixed_size_list, has the same column name — so a name-only check would wave it through and it would detonate later inside merge_insert as an opaque LanceError(IO): Spill has sent an error (EverOS #337). The type check turns that into this clean startup error.

Recover with everos cascade rebuild (documented above). Do not just rm -rf ~/.everos/.index/lancedb: that clears the vectors but leaves md_change_state marked done, so the scanner skips every already- indexed file and the index comes back empty. And do not rm -rf ~/.everos/.index: that also deletes unprocessed_buffer (messages received but not yet extracted — not rebuildable from md). cascade rebuild is correct on both counts. Markdown is the source of truth, so no memory content is lost.

inotify watch-limit exhaustion (Linux)

Default kernel limit is 8 192 watches per user. On a sizeable memory root the watcher may silently miss events. Symptoms:

  • Scanner catches the file changes but the watcher never logs an event for the same path.
  • cat /proc/sys/fs/inotify/max_user_watches is at the limit.

Fix by bumping the kernel parameter:

echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

WSL2 / network mounts

Filesystem events do not propagate from the Windows host into WSL2 (or across most SMB / NFS shares). The watcher will start without error and silently see nothing.

Workarounds:

  • Rely on the scanner — at default 30 s interval, throughput is bounded but eventually-consistent.
  • Drop the scan interval to ~5 s if the memory root is small.
  • Run everos cascade sync explicitly after batch edits.

Daemon process crash mid-batch

claim_pending_batch flips rows to processing atomically. If the process dies before mark_done / mark_failed, those rows stay in processing until the next boot. The orchestrator auto-recovers on startup: CascadeOrchestrator.start calls md_change_state_repo.recover_orphan_processing() before launching the watcher / scanner / worker, which resets every processing row back to pending. Single-process cascade means no race — at boot time no other worker could legitimately own a processing row.

No operator action required; the structured log line cascade_recovered_orphan_processing reports the count when it fires.

FD exhaustion (os error 24 / EMFILE)

Symptoms (any of these on a long-running daemon):

  • LanceDB query / index build fails with lance error: ... Too many open files (os error 24).
  • lsof -p <pid> | wc -l grows monotonically over hours / days.
  • Health log lines like cascade_lancedb_optimize_failed / cascade_lancedb_rebuild_failed carrying OSError: [Errno 24].

Cause (verified against lance crate 4.0): the LanceDB index cache (GlobalIndexCache) holds one reader object per opened FTS / vector / scalar index, and each reader pins the file descriptors of its _indices/<uuid>/... files. With a long-running daemon and steady- state cascade ingest, every optimize() call adds new readers; with LanceDB's own default (index_cache_size_bytes=None, unbounded), they are never evicted and the FDs leak monotonically.

drop_index does not help — it is a manifest-only operation and leaves the on-disk UUID directories untouched. Even an explicit optimize(cleanup_older_than=0) unlink()-ing the files does not release FDs: POSIX keeps the inode alive as long as a process holds an open FD on it (the entries show as (deleted) in lsof). Only an LRU eviction inside the cache (or a connection close) actually closes the FDs.

Fix (already wired in LanceDBSettings.index_cache_size_bytes — default 16 MB, ~290 FD ceiling): see Tuning knobs § LanceDB index cache for the sizing table and the env-var override path.

If you have already hit EMFILE in a running process, the cleanest recovery is a daemon restart — the open connection closes, every FD is released, and the next start comes up with the capped Session in place.

Tuning knobs

Cascade scheduler knobs

All defaults live in everos.memory.cascade.orchestrator.CascadeConfig and everos.memory.cascade.worker.CascadeWorker:

Knob Default Effect
scan_interval_seconds 30 Scanner sweep cadence
worker_batch_size 50 Rows claimed per worker cycle
worker_max_retry 3 Inline retries before mark_failed(retryable=TRUE)
worker_poll_interval_seconds 1 Idle wait between empty drain attempts
worker_retry_backoff_seconds 2 Linear backoff seed; doubles per attempt

Tuning surface is intentionally not in Settings yet — once we have wall-clock numbers from real workloads, the values that need operator override will surface there.

LanceDB index cache (index_cache_size_bytes)

Lives in LanceDBSettings; overridable via the EVEROS_LANCEDB__INDEX_CACHE_SIZE_BYTES environment variable. This is the only knob that bounds the steady-state file-descriptor count of a long-running EverOS daemon — see Recovery paths § FD exhaustion for why nothing else (prune, rebuild, drop_index) helps.

Measured cap → FD ceiling (30 add+optimize cycles + 100-query stress on the real Episode schema):

Cap FD ceiling Query latency (p50) Safe under ulimit -n
2 MB ~45 ~5 ms macOS default 256 (5× headroom)
4 MB ~52 ~3 ms macOS default 256
8 MB ~140 ~2.4 ms macOS default 256 (1.8× headroom)
16 MB (default) ~290 ~2.3 ms Linux default 1024 (3.5× headroom); macOS needs ulimit -n 1024
32 MB ~630 ~1.4 ms Linux default 1024 (1.6× headroom)
unbounded grows forever ~1.3 ms NEVER use in a daemon

EverOS's measured steady-state working set after a rebuild_indexes cycle is roughly 50-100 readers / 3-6 MB resident (5 tables × ~7 BM25 columns × ~10 part_N reader entries each), so the 16 MB default provides ~3× headroom for burst traffic and stale-but-not-yet-evicted readers.

When to override:

  • Tight ulimit -n environments (containers; macOS dev boxes that haven't bumped the default 256) → drop to 4 MB or 8 MB. Query latency increases by ~1-3 ms but correctness is unaffected.
  • Larger working sets (many more tables or much wider FTS indexes than the default schema set) → bump to 32-64 MB. Verify your platform's ulimit -n covers the corresponding FD ceiling with at least 2× headroom.
  • Diagnostic-only: set to a tiny value (e.g. 1 MB) to force LRU thrashing and reproduce cache-miss latency in tests.

Do not set metadata_cache_size_bytes — it is intentionally left at LanceDB's default (unbounded) because the metadata cache holds parsed manifests / fragment stats and has zero effect on FD count; capping it just thrashes parsing work without solving anything.

Concurrency

The worker is async, not multi-process. Inside one drain cycle, asyncio.gather(*[_process_one(row) for row in batch]) runs every claimed row concurrently — cascade is IO-bound (embedding HTTP calls dominate wall time) so single-process coroutine concurrency saturates the bottleneck. The worker_batch_size knob (default 50) caps in-flight rows.

Multi-process workers are a scaling axis we'd reach for only if a single process becomes CPU-bound, which the current design does not anticipate. claim_pending_batch is already race-safe (the WHERE status='pending' filter ensures each row lands in exactly one batch even if multiple workers raced), so adding processes later is a deployment-side change with no schema work.

What cascade does NOT do (yet)

  • Schema migration: LanceDB has no in-place column migration; a schema change is recovered by rebuilding from md (everos cascade rebuild), not an automatic ALTER.
  • Parent-id back-link: Episode rows currently carry parent_id=None; the writer doesn't preserve the source memcell id in the entry inline. Tracked separately.
  • Reference-file change detection (agent_skill): edits to references/*.md siblings won't trigger a re-index — only changes to SKILL.md itself fire the watcher. Workaround: run everos cascade sync agents/<a>/skills/skill_<n>/SKILL.md after editing references.