* fix(cascade): per-kind prune staleness + rebuild safety Adversarial review of the review-response fixes found five real defects, all in code this PR introduced. Health signal (P1): prune staleness reported the time since the NEWEST successful prune across kinds, so on a multi-kind deployment (every real one) a single kind whose cleanup died was masked by the others pruning on schedule — /health stayed green while that table's index dir grew unbounded, the exact incident the signal exists to catch. Report the WORST kind instead and name it in the reason. The failure streak could not cover this either: an intervening light beat resets it, so it never reaches the threshold for a prune-only failure. Documented that split of duties. Spurious fallback rebuilds (P1): the benign-conflict carve-out excluded the heavy beat, justified by "runs under the write lock, so it can't hit this benignly" — but that lock is in-process only, so a second process (a long `cascade backfill`, a `cascade sync`) preempts prune's Rewrite commit. Those counted as real failures, and ~25min of cross-process churn reached the threshold and fired a fallback rebuild, which drops every index before recreating it; a rebuild that also lost the race was swallowed as a warning, leaving the table with no FTS index (every /search on that kind 500s) until the next 12h sweep. Treat commit conflicts as benign on both beats and let prune-staleness detect a prune that genuinely stops succeeding. cascade rebuild (P1 ×2 + P2): it drops and recreates tables with no guard while --help/docstrings advertised it as safe, so `rebuild --yes` against a live daemon corrupted the rebuild (the daemon keeps writing through cached handles). Refuse when the OME jobstore lock is held, reusing backfill's detection and its exit code 3. It also ran the pre-drop migration pass (`ensure_business_indexes`) against the damaged table, so on the corruption classes it exists to repair (missing column, un-alterable type) the recovery path died on the damage itself — skip it via `_runtime(ensure=False)`. Reset the queue BEFORE dropping so every crash window converges on "queue pending → re-index" instead of empty tables with a fully-done queue (a silently empty deployment), and handle Ctrl-C with exit 130 plus a resume hint. Recovery guidance (P1): the nullable-vector migration error still told users to wipe the index directory — which this PR's own runbook documents as the wrong recovery (queue stays done, index comes back empty). Point it at `everos cascade rebuild`. Dropped the schema-drift error's "restart first" step too: the startup migrations only alter nullability, never a name or type, so a name/type drift never self-heals. Also: backfill's post-write prune passed a zero retention window from a separate process, able to delete files under a daemon /search still holding that version — pass the daemon's window instead. Runbook gains the /health cascade block (thresholds, what flips healthy, why failed_permanent does not) and its quoted schema-drift error now matches the code. Tests: the three safety mechanisms this PR adds were unpinned — a one-line revert of any of them passed the suite. Added per-kind staleness, heavy-beat benign conflict, benign-filter negative case (an error whose message merely contains "retryable" must still count), prune recurrence across light beats (mutation-verified: hoisting the attempt-clock advance out of the heavy branch turns it red), the prune timeout releasing the write lock, timeout-below-cadence, the rebuild server guard, and a tier3 assertion that the /health cascade block is actually wired. Froze the last fabricated-monotonic test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Also: treat ``last_prune_attempt_at == 0.0`` as "never attempted" instead of comparing clocks. ``monotonic()`` is boot-relative, so ``now - 0 >= cadence`` is false for the first ~cadence of container uptime — the catch-up prune was skipped exactly when a fresh process most needs it (and made a test depend on the runner uptime, which CI caught). * fix(lancedb): bound every write-lock critical section run7 (1h at 2.5x rate, concurrent CLI maintenance, doubled fuzz) reproduced a table whose version cleanup stopped permanently: 150 versions retained, disk 11x live size, while the other two tables sat at 1 version each — and with no error logged anywhere, because nothing failed. It simply never returned. Three things combined. The maintenance scheduler allows one task per table (a LanceDB table takes one writer), so it skips a kind whose task is still in flight. The prune timeout sat *inside* the lock and covered only the cleanup call. And the other six critical sections on that lock — add, upsert, update, delete, delete_by_md_path, rebuild_indexes — had no deadline at all. So one operation stuck anywhere outside that narrow window wedged the table for good: every writer blocked on acquire, and every later heartbeat was turned away because the stuck task never finished. Make it structurally impossible instead of patching prune: all seven sections now go through `LanceRepoBase._locked(budget, op)`, where the deadline covers **acquisition and the body**. No path can wait for this lock, or hold it, indefinitely. Budgets are hang-catchers, not throughput limits: 120s for row writes, 600s for an index rebuild, the existing 60s for prune. Expiry raises `VectorStoreBusyError`, deliberately under `ExternalServiceError` so the cascade worker retries the row; under `VectorStoreError` a transient lock contention would be marked permanently failed and need a manual `cascade fix`. Tests: a stuck holder now makes a waiter fail its deadline and release (the lock is reusable afterwards), and the prune timeout is pinned as retryable. Verified by mutation — moving the timeout back inside the lock makes a waiter block until the enclosing observation window expires (1001ms vs 51ms), i.e. wait forever in production. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(lancedb): size write-lock budgets from measurements 120s for a row write was a guess, and a bad one: the budget doubles as the detection latency for a wedged table, so an over-slack value means minutes of blocked writers before anything surfaces — the failure this change exists to prevent. Measured the four locked write ops on a local SSD across table sizes and batch sizes (10k-100k rows, 50-500 rows per call): add 3-22ms, upsert (merge_insert, the read-modify-write one) 6-25ms, update 2-4ms, delete 2-3ms; worst observation 63ms, and flat in both dimensions since these are append-and-commit, not scans. So: writes 120s -> 15s (~240x the worst observation, enough for a contended disk and several waiters queued ahead — the deadline includes acquisition and asyncio.Lock is FIFO), rebuild 600s -> 300s (still the one genuinely slow section at ~0.3s per 50k rows per indexed column). Prune stays 60s. Test pins the sizing intent: writes stay in the tens of seconds, and rebuild > prune > write so the slowest section is not the most eagerly killed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(lancedb): record wait/hold time on write-lock critical sections A soak run stalled one table's writes for ~16s and the logs could not say why: maintenance beats only log at `debug`, so a section that is slow but still inside its deadline is invisible, and the timeout warning did not distinguish "never acquired the lock" from "acquired it and overran". `_locked` now carries that apart. The deadline warning gains `acquired`, `waited_seconds` and `held_seconds` — `acquired` alone answers whether a holder was slow or this operation was — and a completed section that held the lock for at least a second logs `lancedb_write_lock_slow_hold` at info, so a stall that never reaches a deadline still leaves a trace. Uses `time.monotonic` (elapsed measurement, not wall clock — the datetime discipline bans `time.time`). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(search): reject a mismatched query vector before it reaches LanceDB A soak run showed every slow search was a failing search: `search:vector` p50 251ms / p99 1.6s, but its 8 requests over 10s were exactly its 8 failures (13-14s each). The cause was a query vector whose width disagreed with the index. LanceDB only notices after the query is built and reports it as an opaque `ValueError: Invalid input, No vector column found to match…`, which escaped as an unhandled 500. Validate at `_embed_query` — the single point every query vector passes through — against the provider's declared `dim`. Microseconds instead of 13s, and a named `ConfigurationError` (500 + CONFIGURATION_ERROR) instead of an unhandled crash. Deliberately not `InvalidInputError`/422: callers only send query *text*, so a bad width is our provider's fault, not the caller's. Also cap traceback rendering. structlog's default is `RichTracebackFormatter(show_locals=True, max_frames=100, extra_lines=3)`, which on an async stack rendered 82 frames into 6423 log lines per exception — 85MB of server.log across 11 of them — at ~290ms of synchronous CPU each, and risks printing request payloads into logs. With locals off and 15 frames the same traceback is 103 lines and 10ms. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(changelog): record the storage-reliability work under Unreleased #379 merged without changelog entries, so this covers both it and the follow-up work in this branch: the maintenance split (compaction vs reclamation) that fixes unbounded index growth, bounded write-lock critical sections, the /health cascade readiness block and its alert contract, `cascade rebuild`, schema type-drift detection, the query-vector width check, and the traceback-rendering cap. Each entry states the operator-visible consequence, not just the change — `cascade rebuild` now refusing to run against a live server, benign-conflict warnings dropping in volume, and `/health` being able to report a stalled kind that was previously invisible are all behaviour changes someone will notice. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: zhanghui <zhanghui@shanda.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
329 lines
13 KiB
Python
329 lines
13 KiB
Python
"""Integration test for ``everos cascade`` CLI commands.
|
|
|
|
Drives the actual Typer commands against a real sqlite + lancedb under a
|
|
tmp memory root. Validates the in-process orchestration that
|
|
``test_cascade_command`` (unit) cannot reach: ``_runtime()`` context,
|
|
queue summary formatting, fix (no-rows path), and a full
|
|
``cascade sync <path>`` round-trip against an empty queue (no handler
|
|
ever runs, so embedding availability is irrelevant here).
|
|
|
|
The CLI commands call ``asyncio.run(_run())`` internally, so this test
|
|
is **synchronous** — pytest-asyncio's auto mode would otherwise wrap it
|
|
in an event loop, which collides with the CLI's own loop.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import asyncio
|
|
import datetime as _dt
|
|
import re
|
|
from collections.abc import Iterator
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
from typer.testing import CliRunner
|
|
|
|
from everos.config import load_settings
|
|
from everos.entrypoints.cli.commands import cascade as cascade_mod
|
|
from everos.infra.persistence.lancedb import dispose_connection
|
|
from everos.infra.persistence.sqlite import dispose_engine
|
|
|
|
|
|
@pytest.fixture
|
|
def cli_runtime(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Iterator[Path]:
|
|
"""Tmp memory root + clean singletons; CLI bootstraps the schema itself."""
|
|
monkeypatch.setenv("EVEROS_ROOT", str(tmp_path))
|
|
monkeypatch.setenv("EVEROS_EMBEDDING__MODEL", "stub-model")
|
|
monkeypatch.setenv("EVEROS_EMBEDDING__BASE_URL", "http://stub.invalid/v1")
|
|
monkeypatch.setenv("EVEROS_EMBEDDING__API_KEY", "stub-key")
|
|
load_settings.cache_clear()
|
|
(tmp_path / "ome.toml").write_text("# test\n")
|
|
|
|
# Strip any singleton state from a neighbouring test.
|
|
asyncio.run(_dispose_all())
|
|
yield tmp_path
|
|
asyncio.run(_dispose_all())
|
|
|
|
|
|
async def _dispose_all() -> None:
|
|
await dispose_connection()
|
|
await dispose_engine()
|
|
|
|
|
|
def _strip_ansi(value: str) -> str:
|
|
return re.sub(r"\x1b\[[0-?]*[ -/]*[@-~]", "", value)
|
|
|
|
|
|
def test_status_on_empty_queue(cli_runtime: Path) -> None:
|
|
"""``cascade status`` boots the runtime + prints zeros for a fresh DB."""
|
|
result = CliRunner().invoke(cascade_mod.app, ["status"])
|
|
assert result.exit_code == 0, result.stdout
|
|
assert "queue:" in result.stdout
|
|
assert "pending:" in result.stdout
|
|
# Fresh DB: every counter is zero.
|
|
assert "0" in result.stdout
|
|
assert "lsn:" in result.stdout
|
|
|
|
|
|
def test_fix_with_no_failed_rows(cli_runtime: Path) -> None:
|
|
"""``cascade fix`` (no ``--apply``) prints the empty-state message."""
|
|
result = CliRunner().invoke(cascade_mod.app, ["fix"])
|
|
assert result.exit_code == 0, result.stdout
|
|
assert "no failed rows" in result.stdout
|
|
|
|
|
|
def test_fix_apply_with_no_failed_rows(cli_runtime: Path) -> None:
|
|
"""``cascade fix --apply`` is a noop when there's nothing to fix."""
|
|
result = CliRunner().invoke(cascade_mod.app, ["fix", "--apply"])
|
|
assert result.exit_code == 0, result.stdout
|
|
assert "no failed rows" in result.stdout
|
|
|
|
|
|
def test_sync_on_empty_queue_with_stub_embedder(
|
|
cli_runtime: Path, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
"""``cascade sync`` invokes orchestrator.drain even on empty queue."""
|
|
# CLI builds the embedder via build_embedding_provider() which would
|
|
# try to connect; replace the orchestrator builder with one wired to
|
|
# the stub embedder.
|
|
from everos.component.tokenizer import build_tokenizer
|
|
from everos.core.persistence import MemoryRoot
|
|
from everos.memory.cascade import CascadeOrchestrator
|
|
|
|
def fake_build_orchestrator() -> CascadeOrchestrator:
|
|
root = MemoryRoot.resolve()
|
|
root.ensure()
|
|
return CascadeOrchestrator(
|
|
memory_root=root,
|
|
tokenizer=build_tokenizer(),
|
|
)
|
|
|
|
monkeypatch.setattr(cascade_mod, "_build_orchestrator", fake_build_orchestrator)
|
|
|
|
result = CliRunner().invoke(cascade_mod.app, ["sync"])
|
|
assert result.exit_code == 0, result.stdout
|
|
assert "sync complete" in result.stdout
|
|
assert "processed 0 row(s)" in result.stdout
|
|
|
|
|
|
def test_sync_with_path_outside_root_errors(
|
|
cli_runtime: Path, tmp_path_factory: pytest.TempPathFactory
|
|
) -> None:
|
|
"""``cascade sync <path>`` rejects paths outside the memory root."""
|
|
other = tmp_path_factory.mktemp("other") / "x.md"
|
|
other.write_text("# unrelated\n")
|
|
result = CliRunner().invoke(cascade_mod.app, ["sync", str(other)])
|
|
assert result.exit_code != 0
|
|
# Typer.BadParameter surfaces in stderr / mixed output. The rich
|
|
# error box wraps the message at terminal width and pads each line
|
|
# with ``│`` (U+2502 box-drawing); so ``not under`` and
|
|
# ``memory root`` end up separated by spaces *plus* box characters
|
|
# *plus* a newline. ``\s`` doesn't match ``│``, so widen to
|
|
# ``[^\w]+`` (anything that isn't an alnum / underscore) — that
|
|
# tolerates the rich frame without falsely matching real text
|
|
# between the two tokens.
|
|
output = result.stdout + (result.stderr or "")
|
|
plain_output = _strip_ansi(output)
|
|
assert re.search(r"not under[^\w]+memory root", plain_output), output
|
|
|
|
|
|
def test_sync_with_unmatched_path(
|
|
cli_runtime: Path, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
"""A path under the root but matching no cascade kind exits 1 with a hint."""
|
|
from everos.component.tokenizer import build_tokenizer
|
|
from everos.core.persistence import MemoryRoot
|
|
from everos.memory.cascade import CascadeOrchestrator
|
|
|
|
def fake_build_orchestrator() -> CascadeOrchestrator:
|
|
return CascadeOrchestrator(
|
|
memory_root=MemoryRoot.resolve(),
|
|
tokenizer=build_tokenizer(),
|
|
)
|
|
|
|
monkeypatch.setattr(cascade_mod, "_build_orchestrator", fake_build_orchestrator)
|
|
|
|
# File under the root but in an unregistered subdirectory.
|
|
unregistered = cli_runtime / "stuff" / "random.md"
|
|
unregistered.parent.mkdir(parents=True, exist_ok=True)
|
|
unregistered.write_text("# random\n")
|
|
result = CliRunner().invoke(cascade_mod.app, ["sync", str(unregistered)])
|
|
assert result.exit_code == 1
|
|
# stderr in CliRunner is merged into stdout for typer.echo(..., err=True).
|
|
output = result.stdout + (result.stderr or "")
|
|
assert "does not match any registered cascade kind" in output
|
|
|
|
|
|
# Keep a baseline so future regressions show as a hard failure.
|
|
def test_status_handles_pending_rows(cli_runtime: Path) -> None:
|
|
"""Seed one pending row via the repo before invoking status."""
|
|
|
|
async def seed() -> None:
|
|
# Bring the runtime up like the CLI does, seed, then dispose.
|
|
async with cascade_mod._runtime():
|
|
from everos.infra.persistence.sqlite import md_change_state_repo
|
|
|
|
await md_change_state_repo.force_enqueue(
|
|
"users/u1/episodes/episode-2026-01-01.md", "episode"
|
|
)
|
|
|
|
asyncio.run(seed())
|
|
|
|
result = CliRunner().invoke(cascade_mod.app, ["status"])
|
|
assert result.exit_code == 0, result.stdout
|
|
# One row pending; LSN must be ≥ 1.
|
|
assert "pending: 1" in result.stdout
|
|
|
|
|
|
def _fake_orchestrator_factory(): # type: ignore[no-untyped-def]
|
|
from everos.component.tokenizer import build_tokenizer
|
|
from everos.core.persistence import MemoryRoot
|
|
from everos.memory.cascade import CascadeOrchestrator
|
|
|
|
def _build() -> CascadeOrchestrator:
|
|
root = MemoryRoot.resolve()
|
|
root.ensure()
|
|
return CascadeOrchestrator(
|
|
memory_root=root,
|
|
tokenizer=build_tokenizer(),
|
|
)
|
|
|
|
return _build
|
|
|
|
|
|
def test_rebuild_recovers_drifted_index_and_reindexes(
|
|
cli_runtime: Path, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
"""``cascade rebuild`` recovers a type-drifted index.
|
|
|
|
Proves the three properties bare ``rm`` can't offer together:
|
|
(1) it runs despite a schema-type drift that trips the normal
|
|
startup guard (``verify_business_schemas``); (2) it drops + recreates
|
|
the drifted table with the current (correct) type; (3) it re-indexes
|
|
md from scratch even for entries the queue already marked ``done``.
|
|
"""
|
|
import datetime as _dt
|
|
|
|
import lancedb
|
|
import pyarrow as pa
|
|
|
|
from everos.core.persistence import MemoryRoot
|
|
from everos.infra.persistence.lancedb import get_table
|
|
from everos.infra.persistence.lancedb.tables.atomic_fact import AtomicFact
|
|
from everos.infra.persistence.lancedb.tables.episode import Episode
|
|
from everos.infra.persistence.markdown import AtomicFactWriter
|
|
|
|
root = MemoryRoot.resolve()
|
|
root.ensure()
|
|
owner_id = "u_rebuild"
|
|
bucket = _dt.date(2026, 5, 18)
|
|
md_path = (
|
|
f"default_app/default_project/users/{owner_id}/.atomic_facts/"
|
|
f"atomic_fact-{bucket.isoformat()}.md"
|
|
)
|
|
|
|
async def _seed_md() -> None:
|
|
writer = AtomicFactWriter(root=root)
|
|
items = [
|
|
(
|
|
{
|
|
"owner_id": owner_id,
|
|
"session_id": f"s_{j}",
|
|
"timestamp": "2026-05-18T07:04:26+00:00",
|
|
"parent_id": f"mc_{j}",
|
|
"sender_ids": [owner_id],
|
|
},
|
|
{"Fact": f"seed fact {j}"},
|
|
)
|
|
for j in range(2)
|
|
]
|
|
await writer.append_entries(owner_id, items, date=bucket)
|
|
|
|
async def _drift_episode_table() -> None:
|
|
conn = await lancedb.connect_async(str(root.lancedb_dir))
|
|
drifted = pa.schema(
|
|
[
|
|
pa.field("subject_vector", pa.string(), nullable=True)
|
|
if f.name == "subject_vector"
|
|
else f
|
|
for f in Episode.to_arrow_schema()
|
|
]
|
|
)
|
|
await conn.create_table("episode", schema=drifted)
|
|
conn.close()
|
|
|
|
async def _episode_subject_vector_type(): # type: ignore[no-untyped-def]
|
|
tbl = await get_table("episode", Episode)
|
|
return (await tbl.schema()).field("subject_vector").type
|
|
|
|
async def _atomic_fact_row_count() -> int:
|
|
tbl = await get_table(AtomicFact.TABLE_NAME, AtomicFact)
|
|
return await tbl.count_rows(filter=f"md_path = '{md_path}'")
|
|
|
|
asyncio.run(_seed_md())
|
|
asyncio.run(_drift_episode_table())
|
|
asyncio.run(_dispose_all())
|
|
|
|
monkeypatch.setattr(
|
|
cascade_mod, "_build_orchestrator", _fake_orchestrator_factory()
|
|
)
|
|
|
|
# Contrast: a normal command boots via _runtime() → verify trips on the drift.
|
|
status_result = CliRunner().invoke(cascade_mod.app, ["status"])
|
|
assert status_result.exit_code != 0
|
|
asyncio.run(_dispose_all())
|
|
|
|
# rebuild skips verify, recreates the table, and re-indexes md.
|
|
result = CliRunner().invoke(cascade_mod.app, ["rebuild", "--yes"])
|
|
assert result.exit_code == 0, result.stdout
|
|
assert "rebuild complete" in result.stdout
|
|
asyncio.run(_dispose_all())
|
|
|
|
# Table recreated with the correct vector type; md re-indexed.
|
|
assert asyncio.run(_episode_subject_vector_type()).equals(
|
|
Episode.to_arrow_schema().field("subject_vector").type
|
|
)
|
|
assert asyncio.run(_atomic_fact_row_count()) == 2
|
|
|
|
|
|
def test_rebuild_refuses_to_run_while_a_server_holds_the_lock(
|
|
cli_runtime: Path, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
"""``rebuild`` must refuse when a daemon is running on this memory root.
|
|
|
|
It drops and recreates the LanceDB tables; a live daemon holds cached
|
|
table handles and would keep writing to the dropped dataset, leaving a
|
|
corrupted rebuild plus a permanent-failure backlog. Detection reuses the
|
|
OME jobstore lock that ``backfill`` already gates on, and the exit code
|
|
matches backfill's ``3`` (SERVER_RUNNING).
|
|
"""
|
|
monkeypatch.setattr(cascade_mod, "ome_lock_is_free", lambda: False)
|
|
|
|
result = CliRunner().invoke(cascade_mod.app, ["rebuild", "--yes"])
|
|
|
|
assert result.exit_code == 3, result.output
|
|
# The explanation goes to stderr (click 8.2 keeps the streams separate).
|
|
assert "server" in result.stderr.lower()
|
|
assert "stop `everos server`" in result.stderr.lower()
|
|
# And it must bail out BEFORE touching anything — none of the step
|
|
# progress lines may appear.
|
|
combined = result.output + result.stderr
|
|
assert "LanceDB table(s)" not in combined
|
|
assert "cascade queue row(s)" not in combined
|
|
assert "rebuild complete" not in combined
|
|
|
|
|
|
# Reduce false negatives on date drift.
|
|
def test_resolve_relative_via_command_arg(cli_runtime: Path) -> None:
|
|
"""An absolute path under the root works through ``cascade sync <path>``."""
|
|
md_file = cli_runtime / "users" / "u1" / "episodes" / "episode-2026-05-25.md"
|
|
md_file.parent.mkdir(parents=True, exist_ok=True)
|
|
today = _dt.date.today().isoformat() # only used so the var isn't unused
|
|
md_file.write_text(f"# {today}\n")
|
|
|
|
# We don't need the orchestrator to actually drain anything; pass --help
|
|
# against the sync subcommand to verify the path resolution helper
|
|
# doesn't barf at construction time.
|
|
result = CliRunner().invoke(cascade_mod.app, ["sync", "--help"])
|
|
assert result.exit_code == 0
|