* refactor(config): make [embedding] and [rerank] soft dependencies
Make [embedding] and [rerank] soft runtime dependencies so a
freshly-onboarded user can run EverOS end-to-end with only [llm]
configured. Previously the server refused to start without embedding,
locking out anyone who just wanted keyword-only search.
## Capability tiers
- Tier 1 ([llm] only) KEYWORD search, add/flush, md
writes, cascade sync
- Tier 2 ([llm] + [embedding]) + VECTOR / HYBRID search,
reflection, skill extraction,
backfill
- Tier 3 ([llm] + [embedding] + rerank) + AGENTIC search, knowledge
Tier upgrades require a server restart (capability accessors cache
for the process lifetime). Tier downgrades are read-safe: a Tier-3
user who drops [rerank] can still read/rename/delete existing
knowledge documents; only write/search endpoints return 422.
## What changed
- Component accessors — component/{embedding,rerank,llm}/accessor.py
are the single process-wide provider singletons. service/* never
maintains parallel singletons; it consumes get_embedding_capability()
/ get_rerank_capability() / get_llm_client() directly. Build-time
ValueError from the factory is logged as capability_build_failed
(was silently swallowed).
- Error mapping — ProviderNotConfiguredError -> 422 with everos.toml
section hints (never EVEROS_* env-var strings).
LanceDBMigrationError fails loud with escalating recovery guidance
(restart -> wipe index). LLMNotConfiguredError in search maps to
None for KEYWORD degradation.
- Nullable-vector LanceDB migration — schema v2 makes the vector
column nullable so Tier-1 rows can land without embeddings.
Migration is guarded by a cross-process memory_root_lock
(fcntl.flock + anyio.to_thread) and runs optimize() per table
after Phase-1 backfill to reclaim manifest bloat.
- Cascade — knowledge handlers register unconditionally (Tier-3 ->
Tier-2/1 downgrade no longer strands DELETE); embed-requiring
strategies use body-guards that check capability.available at
execution time. _TABLE_SPECS has an import-time drift assertion
against BUSINESS_SCHEMAS_WITH_VECTOR.
- `everos cascade backfill` CLI — Phase-1 (embed missing vectors) /
Phase-2 (emit synthetic events for cascaded processing) / Phase-3
(sync new skill files). Exit codes: 0 / 1 / 2 / 3 (server running
preflight) / 4 (COMPLETED_WITH_FAILURES — per-row failures rolled
up) / 130 (SIGINT). OMEConfig.crash_recovery_enabled=False in
backfill engines prevents stale-RUNNING rows re-enqueuing into a
smaller strategy registry.
- /health — reports capabilities + disabled_features per tier so ops
can distinguish "boots but degraded" from "boots and full".
- Presentation split — memory / service / infra never import typer /
click. TyperPresenter Protocol + run_backfill() live in
entrypoints/cli/commands/_backfill_cmd.py. Enforced by
import-linter.
- Startup hint — unconditional count_rows(filter="vector IS NULL")
sweep emits unbackfilled_memory_rows (event name + hint text
pinned) when Tier-1 rows exist. ParserLifespanProvider warms the
everalgo.parser import at boot so /health doesn't block on first
call.
- Knowledge upload UTF-8 short-circuit — _looks_like_utf8_text()
routes text/* mime and known plaintext extensions (md/txt/rst)
straight to UTF-8 decode instead of the parser. Prevents 503
Multimodal-not-configured when Tier 3 sans [multimodal] uploads a
markdown doc.
## Sync history with main (2 merges collapsed into this squash)
Merged origin/main at 6dcd3eb (v1.1.4 -> v1.2.0 adds OTel tracing,
/api/v2 alias, TracingLifespanProvider, per-cascade-embedding span
fix, memory-op instrumentation) and later at 42629df (PR #366
backfills v1.1.4 CWE-22 knowledge path traversal fix + cascade
retry-budget rework + errors.py -> core.errors.ExternalServiceError).
Key merge decisions:
- service/search.py adopts single wrap site — component.llm accessor
already applies UsageRecordingClient when observability is on;
service layer never keeps a parallel LLM singleton (Round-1 CR
rule: "service layer never maintains parallel singletons").
- Knowledge router prefix moved to /knowledge; create_app() mounts
it under both /api/v1 and /api/v2.
- Cascade retry classification uses ExternalServiceError from
core.errors (cascade/errors.py deleted). _MAX_TOTAL_RETRIES=12
cross-cycle budget preserved.
- Fixed backport typo: MemoryRoot.default() -> MemoryRoot.resolve()
(no .default() classmethod exists — main PR #366 shipped a broken
call).
## Verified layering
$ git grep -l "^import typer\|^from typer" src/everos/{memory,service,infra}
# empty
$ git grep -l "^import click\|^from click" src/everos/{memory,service,infra}
# empty
Memory / service / infra layers clean of CLI presentation libraries.
## Review history
Three rounds of Fable 5 (opus) code review across the pre-squash
commit history closed 38 findings total:
- Round 1: 10 findings (fail-loud migration, backfill hardening,
knowledge router gate scoping, SearchManager guards, profile
throttle lift)
- Round 2: 13 findings (hermetic test env, hot-reload doc drift,
knowledge handler registration, Phase 3 sync guarantee, Phase 2
idempotency, profile event-first path, OMEConfig crash-recovery
gate, cross-process migration lock, batch embed per-row fallback,
LanceDB optimize, typer/click layer split)
- Round 3: 15 Minor cleanups (accessor unification, marker revert,
episode query hygiene, --verbose subcommand, parser lifespan warm,
task-number scrub, temporal-overlap test, real-SIGINT slow mark,
4 design-note back-references)
Full per-round context lives in the PR description on GitHub.
## Test plan
- make lint (ruff + import-linter 3 contracts + assets +
deprecated-names + github-docs + datetime + OpenAPI drift)
- Hermetic env full pytest — 2027 passed / 7 deselected (7 = slow +
live_llm markers)
- Manual e2e across Tier 1/2/3 (21/21 assertions across v1/v2
double-mount and Tier 3 -> Tier 2 downgrade)
- /health reports correct capabilities + disabled_features per tier
## Known follow-ups
- .superpowers/sdd/followup-http-bridge.md (gitignored) — Path A for
spec §10's "backfill 期间 EverOS 完全可用" promise
- _TABLE_SCHEMA_VERSION docstring — v3+ migrations need a version
dispatch table
- extract_user_profile.py throttle-counter block — replace LanceDB
count_by_owner with a sqlite memcell count
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(review): close 3 blockers surfaced by round-4 review
N1. cluster_repo.find_cluster_id_for_member was cross-owner-unsafe.
Its reverse index (member_type, member_id) alone cannot disambiguate
two owners whose entry_id happens to collide — entry_id is
deliberately only per-owner unique (see entries.py:47:
'Cross-user uniqueness is handled at the database layer via a
composite <user_id>_<entry_id> field; it is not encoded into the
EntryId string itself'). Phase 2's _scan_all_rows crosses all
owners, so on any multi-owner root, same-day seq=1 episodes under
different owners would either false-hit each other's cluster or
be silently skipped from clustering. Add required (app_id,
project_id, owner_id) keyword args + JOIN Cluster (which already
carries scope) to filter by parent scope. Prior signature had zero
production callers except two the same PR just added, so the
API break is contained. Regression test: two owners persist a
cluster each around the same entry_id, each lookup resolves to
its own owner's cluster, a third owner's lookup returns None.
N2. Ctrl-C / EOF at the y/N prompt was landing on the generic
except Exception branch (exit 2 with rich traceback) instead of
the exit-130 interrupt path. Root cause: typer 0.15+ vendored
click under typer._click, so typer.Abort and the standalone
click.exceptions.Abort are distinct classes. The interrupt-branch
catch only listed the standalone one; every existing 'abort'
test was manually raising click.exceptions.Abort so the miss
was a false-positive guard rail. Widen the catch to
(typer.Abort, click.exceptions.Abort) and declare click as a
first-class dependency (it was only pulled in via uvicorn).
Regression test: raise real typer.Abort() at the confirm step
and assert exit 130 + INTERRUPTED banner.
N3. _looks_like_utf8_text used mime.startswith('text/'), which
caught text/html as well. HTML uploads then bypassed everalgo's
_aparse_html — losing clean_html_for_llm (strips <script>/<style>
/<nav>/<iframe> + HTML comments) and the 1 MiB output cap. A
40 MiB .html with <script> bodies and <!-- prompt injection -->
comments would flow straight into the extraction LLM. Replace
with explicit allowlist {text/plain, text/markdown, text/x-rst,
text/x-markdown}; text/html and any future text/* mime now
default to the parser path. Test matrix asserts text/html →
False (was regressed as True by the earlier commit).
Hermetic env full pytest: 2033 passed / 7 deselected (+6 tests
from these regressions).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(review): close round-4 major + minor + PR body errata
Round-4 review-driven cleanup. Blocker fixes (N1/N2/N3) landed in
561b5fe. This commit closes the remaining CONFIRMED items:
Major:
- J3 MemoryRoot.default() -> resolve() was a breaking public API
rename this branch introduced (main still has default()). Adds
default() as a backward-compat alias forwarding to resolve()
with DeprecationWarning; CHANGELOG entry under Unreleased.
- J4 episode_repo.list_by_owner_after_ts(limit=N) truncates in
fragment order (== insertion order), NOT newest-first. Docstring
now spells out the trap so a future caller passing limit for a
'newest N' window doesn't silently get the oldest N.
- J5.2 TyperPresenter.nothing_to_backfill picked colour via
'could not be read' in message — a domain wording change
would silently flip yellow -> green. Signature gains explicit
scan_failed: bool kwarg; CLI colour-picks off the flag.
- J5.5 phase_header was Protocol-declared but never dispatched
(run_backfill calls _print_phase_header directly). Removed the
dead Protocol method + both no-op implementations.
- J6.3 3 inline from everos.core.errors import ... inside
Phase 1/2/3 preflights promoted to a single top-level import.
- J7 subject-side embed failure was silently exit-0 because
rows_processed advanced whenever any side wrote. Now: a row with
a needed side still NULL counts as rows_failed (exit 4 =
COMPLETED_WITH_FAILURES). Gated on spec.subject_of + row.needs_*
+ row.subject_text so non-Episode tables and subject-empty rows
don't false-positive.
- J9 test_migration_cross_process.py did NOT actually test cross-
process (all 5 tests mock memory_root_lock). Renamed to
test_migration_lock_wiring.py; docstring now scopes it to
'lock-invocation wiring' and points at test_core/…/test_locking.py
for real flock coverage.
- J10 Phase 1 lacked the server-running preflight Phase 2/3 have.
--phase all against a live server would burn Phase 1 embed API
calls (real cost) before Phase 2 halted with exit 3. Phase 1 now
probes _probe_ome_lock_available first; regression test in
test_backfill_preflight.py; upgrade_path integration patches the
probe so its in-process 'server + backfill' scenario stays valid.
Minor:
- M1 knowledge upload with NUL byte or filename > 255 bytes UTF-8
used to raise ValueError/OSError at write_bytes → 500 with a
half-written md left on disk. _safe_original_filename now
rejects both up front with InvalidInputError (→ 400).
- M2 backfill optimize() now passes cleanup_older_than=timedelta(0)
so older manifest versions are physically pruned (previous call
compacted fragments but left the manifest chain on disk).
- M3 verify_business_schemas remediation text used to jump straight
to 'rm -rf ~/.everos/.index/lancedb'; now walks restart → wipe.
- M5 multimodal/accessor.py capability_build_failed warning added
so all four provider accessors log symmetrically (was silent).
- M7 test_knowledge_api parser-absence tests call
parser_available.cache_clear() around the sys.modules patch so
the lru_cache doesn't strand a stale True/False.
- M8 cascade_handler_embed_skipped (6 handlers) demoted INFO → DEBUG:
Tier 1 imports were generating N × 6 handler-info lines per md.
- M10.1 test_drift_scenario_would_raise was a tautology (compared
two hardcoded string sets, never touched the guard). Now
monkey-patches BUSINESS_SCHEMAS_WITH_VECTOR to a superset and
reloads _backfill, proving the import-time RuntimeError fires.
- M11 test_cascade_verbose_position subprocess.run calls gain
env= — scrubs EVEROS_* from the developer environment so the
same footgun as round-2 B1 doesn't re-appear inside subprocesses.
- M14 health() -> dict degraded the OpenAPI schema to
additionalProperties: true. Introduce HealthResponse +
HealthCapabilities Pydantic models so clients get real field
shape; docs/openapi.json regenerated.
- M16 routes/knowledge.py:_require_knowledge_capabilities docstring
claimed cascade.registry.build_handlers still gates
knowledge_topic/knowledge_document, contradicting
registry.py:177-194 (gate removed there, moved to HTTP layer).
Rewritten to describe the current design accurately.
Hermetic env full pytest: 2037 passed / 7 deselected.
Explicitly deferred to followup:
- J1 lazy multimodal client (needs everalgo signature change)
- J2 tier definition (knowledge = Tier 3 whole)
- J5.1/3/4 broader presentation-split refactor
- J6.1/2 backfill dispatch and _backfill_table refactor
- J8 Phase 1 keyset pagination for bulk migration OOM
- M4 flock timeout/waiting log/re-entry (core/persistence refactor)
- M6 UTF-8 codec strategy (BOM/UTF-16/GBK)
- M9 count_by_owner monotonicity (latent, INTERVAL=1 short-circuits)
- M13 --phase all multi-phase combined-outcome test coverage
- M15 PR-marker rationale comments (16 occurrences, all inert)
M12 was refuted (both event names exist).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
487 lines
20 KiB
Python
487 lines
20 KiB
Python
"""CLI-side orchestration for ``everos cascade backfill``.
|
|
|
|
Presentation half of PR #361 review finding M11. The memory-layer phase
|
|
runners (``memory/cascade/_backfill.py``) are pure — every user-facing
|
|
string, colour, and confirmation prompt lives here, behind the
|
|
:class:`~everos.memory.cascade._backfill.BackfillPresenter` Protocol.
|
|
:class:`TyperPresenter` is the typer-backed implementation; tests can
|
|
substitute any structurally-compatible stub.
|
|
|
|
:func:`run_backfill` is the top-level entry point the ``cascade``
|
|
subcommand imports. It builds a :class:`TyperPresenter`, dispatches to
|
|
the three phase runners in order, catches Ctrl-C /
|
|
:class:`click.exceptions.Abort` / :class:`asyncio.CancelledError`, and
|
|
maps every terminal state to a CLI exit code:
|
|
|
|
- ``0`` — clean success.
|
|
- ``1`` — user declined a confirmation prompt.
|
|
- ``2`` — unexpected error mid-phase, or preflight found the required
|
|
provider not configured.
|
|
- ``3`` — Phase 2/3 short-circuited on the OME jobstore lock (another
|
|
``everos`` process is running).
|
|
- ``4`` — every phase ran but at least one row failed embedding even
|
|
after per-row fallback; automation can requeue.
|
|
- ``130`` — Ctrl-C (SIGINT convention: 128 + 2), including Ctrl-C at
|
|
the ``[y/N]`` confirmation prompt (typer/click surfaces that as
|
|
:class:`click.exceptions.Abort`, a :class:`RuntimeError` subclass).
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import asyncio
|
|
|
|
import anyio.to_thread
|
|
import click
|
|
import typer
|
|
|
|
from everos.core.errors import ProviderNotConfiguredError
|
|
from everos.core.observability.logging import get_logger
|
|
|
|
# Module reference (not name-level imports of the phase runners): tests
|
|
# monkeypatch the runners on this module to inject Ctrl-C / boundary
|
|
# behaviour, and re-binding names on the source module only affects the
|
|
# entrypoints layer if we resolve them through the module attribute at
|
|
# call time.
|
|
from everos.memory.cascade import _backfill as _phase_runners
|
|
from everos.memory.cascade._backfill import (
|
|
BackfillPhase,
|
|
_BackfillSummary,
|
|
_count_failed_rows,
|
|
_format_tokens,
|
|
)
|
|
|
|
logger = get_logger(__name__)
|
|
|
|
|
|
PHASES: tuple[BackfillPhase, ...] = (
|
|
BackfillPhase(
|
|
number=1,
|
|
slug="vectors",
|
|
title="Phase 1 — make older memories searchable by meaning",
|
|
detail=(
|
|
"Memories created before you configured embedding can only be "
|
|
"found by keyword right now. This step lets them be found by "
|
|
"meaning too."
|
|
),
|
|
),
|
|
BackfillPhase(
|
|
number=2,
|
|
slug="clusters",
|
|
title="Phase 2 — group related memories into topics",
|
|
detail=(
|
|
"Related memories get clustered together so reflection and "
|
|
"advanced retrieval can operate on them."
|
|
),
|
|
),
|
|
BackfillPhase(
|
|
number=3,
|
|
slug="skills",
|
|
title="Phase 3 — build the agent skill library",
|
|
detail=(
|
|
"Distill reusable skills from past agent conversations. Uses "
|
|
"your LLM — the most expensive step."
|
|
),
|
|
),
|
|
)
|
|
|
|
|
|
_EXIT_LABELS: dict[int, str] = {
|
|
0: "SUCCESS",
|
|
1: "ABORTED",
|
|
2: "FAILED",
|
|
3: "SERVER_RUNNING",
|
|
4: "COMPLETED_WITH_FAILURES",
|
|
130: "INTERRUPTED",
|
|
}
|
|
"""Maps :func:`run_backfill`'s exit codes to the summary block's ``Exit:``
|
|
label. ``3`` marks a Phase 2/3 short-circuit because another process
|
|
holds the OME jobstore lock (typically ``everos server start``). ``4``
|
|
marks a partial-success run: every phase ran to completion, but at
|
|
least one row failed embedding even after the per-row fallback —
|
|
operators consult the ``cascade_backfill_row_embed_failed`` /
|
|
``..._row_subject_embed_failed`` log events for the failed row ids.
|
|
``130`` is the SIGINT convention (128 + signal number 2)."""
|
|
|
|
|
|
def _confirm(detail: str, *, auto_yes: bool) -> bool:
|
|
"""Ask the user to confirm one phase; ``--yes`` skips the prompt.
|
|
|
|
Kept as a module-level helper (rather than a bound method of
|
|
:class:`TyperPresenter`) so tests can monkeypatch it on this module
|
|
the same way the pre-M11 code was patched on ``_backfill``. The
|
|
presenter's :meth:`~TyperPresenter.confirm` delegates here.
|
|
"""
|
|
if auto_yes:
|
|
return True
|
|
return typer.confirm(f"Proceed with: {detail}", default=False)
|
|
|
|
|
|
def _print_phase_header(phase: BackfillPhase) -> None:
|
|
typer.secho(phase.title, bold=True)
|
|
typer.echo(f" {phase.detail}")
|
|
typer.echo("")
|
|
|
|
|
|
def _print_aborted() -> None:
|
|
typer.secho("Aborted by user.", fg=typer.colors.RED)
|
|
|
|
|
|
def _print_interrupted() -> None:
|
|
"""Print the Ctrl-C resume hint (see :func:`run_backfill`)."""
|
|
typer.secho(
|
|
"Interrupted — partial progress was written. Resume by running:",
|
|
fg=typer.colors.YELLOW,
|
|
)
|
|
typer.echo(" everos cascade backfill --phase <phase-name> --yes")
|
|
typer.echo("where <phase-name> is one of: vectors, clusters, skills, all.")
|
|
|
|
|
|
def _print_capability_missing_hint(message: str) -> None:
|
|
"""Emit the friendly toml-hint message shown when a phase's required
|
|
provider is not configured.
|
|
|
|
The formatted :class:`ProviderNotConfiguredError` string is built by
|
|
the memory-layer phase runner and passed through the presenter — so
|
|
the CLI and API stay in lock-step on the remediation copy without
|
|
duplicating it here.
|
|
"""
|
|
typer.secho(f" {message}", fg=typer.colors.RED)
|
|
|
|
|
|
def _print_server_running_hint() -> None:
|
|
"""Emit the friendly error shown when preflight or ``engine.start()``
|
|
detects another process holding the OME jobstore lock.
|
|
|
|
Copy points the user at ``everos server start`` (the usual holder)
|
|
and Phase 1 (``--phase=vectors``), which never touches OME and so
|
|
stays fully concurrent with a running server — never at
|
|
``EVEROS_*`` env vars.
|
|
"""
|
|
typer.secho(
|
|
" Backfill Phase 2/3 needs exclusive access to the OME jobstore,",
|
|
fg=typer.colors.RED,
|
|
)
|
|
typer.secho(
|
|
" but another everos process is holding it (typically your running",
|
|
fg=typer.colors.RED,
|
|
)
|
|
typer.secho(
|
|
" `everos server start`).",
|
|
fg=typer.colors.RED,
|
|
)
|
|
typer.echo("")
|
|
typer.echo(" Stop your everos server first, then re-run backfill.")
|
|
typer.echo(" (Phase 1 --phase=vectors already runs concurrently with the server;")
|
|
typer.echo(" only Phase 2/3 need offline mode. See spec §10.)")
|
|
|
|
|
|
def _print_vectors_estimate(total_rows: int, total_tokens: int) -> None:
|
|
typer.echo(f" memories to process: {total_rows:,}")
|
|
typer.echo(f" input tokens: {_format_tokens(total_tokens)}\n")
|
|
typer.echo(
|
|
" Uses your embedding provider — cost depends on its per-token pricing.\n"
|
|
)
|
|
|
|
|
|
def _print_clusters_estimate(episode_count: int, case_count: int) -> None:
|
|
typer.echo(f" episodes to cluster: {episode_count:,}")
|
|
typer.echo(f" agent cases to cluster: {case_count:,}\n")
|
|
typer.echo(
|
|
" Uses your embedding provider (and LLM for agent-case merges) — "
|
|
"cost depends on provider pricing.\n"
|
|
)
|
|
|
|
|
|
def _print_skills_estimate(cases: int, clusters: int) -> None:
|
|
typer.echo(f" clusters to extract skills from: {clusters:,}")
|
|
typer.echo(f" agent cases to replay: {cases:,}\n")
|
|
typer.echo(" Uses your LLM — cost depends on its per-token pricing.\n")
|
|
|
|
|
|
def _print_summary(summary: _BackfillSummary, *, exit_code: int) -> None:
|
|
"""Print the consolidated post-run summary block.
|
|
|
|
Field labels are chosen from what each phase's result dataclass
|
|
actually tracks — e.g. Phase 3 reports "agent cases processed"
|
|
(``events_emitted``) rather than "clusters processed", since the
|
|
distinct cluster count isn't retained on
|
|
:class:`~everos.memory.cascade._backfill._SkillPhaseResult`.
|
|
|
|
On ``exit_code == 4`` (COMPLETED_WITH_FAILURES) also emits a hint
|
|
pointing operators at the per-row failure log events so they can
|
|
recover the failed row ids.
|
|
"""
|
|
typer.echo("")
|
|
typer.secho("Backfill summary", bold=True)
|
|
typer.echo("-" * 44)
|
|
if summary.vectors is not None:
|
|
r = summary.vectors
|
|
typer.echo(
|
|
f" Phase 1 (vectors) — {r.rows_processed:,} rows / "
|
|
f"{r.rows_failed:,} failed / {_format_tokens(r.tokens_embedded)}"
|
|
)
|
|
if summary.clusters is not None:
|
|
c = summary.clusters
|
|
created = c.clusters_after - c.clusters_before
|
|
typer.echo(
|
|
f" Phase 2 (clusters) — {c.events_emitted:,} events emitted / "
|
|
f"{created:,} clusters created"
|
|
)
|
|
if summary.skills is not None:
|
|
s = summary.skills
|
|
extracted = s.skills_after - s.skills_before
|
|
typer.echo(
|
|
f" Phase 3 (skills) — {s.events_emitted:,} agent cases "
|
|
f"processed / {extracted:,} skills extracted"
|
|
)
|
|
typer.echo("-" * 44)
|
|
label = _EXIT_LABELS.get(exit_code, "UNKNOWN")
|
|
typer.echo(f" Exit: {label} ({exit_code})")
|
|
if exit_code == 4:
|
|
total_failed = _count_failed_rows(summary)
|
|
typer.secho(
|
|
f" {total_failed:,} rows failed embedding — see log events "
|
|
"`cascade_backfill_row_embed_failed` for row_ids and text prefixes.",
|
|
fg=typer.colors.YELLOW,
|
|
)
|
|
|
|
|
|
class TyperPresenter:
|
|
"""typer-backed :class:`BackfillPresenter` — the CLI's user-I/O seam.
|
|
|
|
Every method delegates to a module-level ``_print_*`` helper or to
|
|
:func:`_confirm`. Kept as a small class (rather than a module of
|
|
free functions) so alternative front-ends (e.g. a JSON-emitting
|
|
CI-mode presenter) can slot in without touching the memory layer.
|
|
"""
|
|
|
|
def nothing_to_backfill(self, message: str, *, scan_failed: bool = False) -> None:
|
|
colour = typer.colors.YELLOW if scan_failed else typer.colors.GREEN
|
|
typer.secho(message, fg=colour)
|
|
|
|
def capability_missing(self, *, provider: str, feature: str, message: str) -> None:
|
|
_print_capability_missing_hint(message)
|
|
|
|
def server_running(self) -> None:
|
|
_print_server_running_hint()
|
|
|
|
def estimate_vectors(self, rows: int, tokens: int) -> None:
|
|
_print_vectors_estimate(rows, tokens)
|
|
|
|
def estimate_clusters(self, episodes: int, cases: int) -> None:
|
|
_print_clusters_estimate(episodes, cases)
|
|
|
|
def estimate_skills(self, cases: int, clusters: int) -> None:
|
|
_print_skills_estimate(cases, clusters)
|
|
|
|
async def confirm(self, prompt: str, *, auto_yes: bool) -> bool:
|
|
# ``typer.confirm`` is blocking (reads from stdin); offload so a
|
|
# slow / interactive user doesn't hold the event loop. Any
|
|
# ``click.exceptions.Abort`` raised on Ctrl-C at the prompt
|
|
# propagates to :func:`run_backfill` which catches it and maps
|
|
# to exit 130.
|
|
return await anyio.to_thread.run_sync(
|
|
lambda: _confirm(prompt, auto_yes=auto_yes)
|
|
)
|
|
|
|
def emit_progress(self, done: int, total: int) -> None:
|
|
typer.echo(f" emitted {done:,} / {total:,} event(s)")
|
|
|
|
def row_progress(self, table_name: str, done: int, total: int) -> None:
|
|
typer.echo(f" {table_name}: processed {done:,} / {total:,} rows")
|
|
|
|
def phase_1_complete(self, rows_processed: int, rows_failed: int) -> None:
|
|
typer.secho(
|
|
f"phase 1 complete — {rows_processed:,} memories now "
|
|
"searchable by meaning"
|
|
+ (f" ({rows_failed:,} failed)" if rows_failed else ""),
|
|
fg=typer.colors.GREEN,
|
|
)
|
|
|
|
def phase_2_complete(self, before: int, after: int, emitted: int) -> None:
|
|
typer.secho(
|
|
f"phase 2 complete — clusters {before:,} -> {after:,} "
|
|
f"({emitted:,} events processed)",
|
|
fg=typer.colors.GREEN,
|
|
)
|
|
|
|
def phase_3_complete(
|
|
self, skills_before: int, skills_after: int, emitted: int
|
|
) -> None:
|
|
typer.secho(
|
|
f"phase 3 complete — skills {skills_before:,} -> {skills_after:,} "
|
|
f"({emitted:,} events processed)",
|
|
fg=typer.colors.GREEN,
|
|
)
|
|
|
|
|
|
async def run_backfill(*, phase: str, auto_yes: bool) -> int:
|
|
"""Run backfill phases interactively; return an exit code.
|
|
|
|
``phase`` is ``"all"`` or one of :data:`PHASES`'s ``slug`` values.
|
|
Exit codes: see :data:`_EXIT_LABELS`.
|
|
|
|
Every exit path — success, abort, error, or interrupt — ends with a
|
|
consolidated summary block (:func:`_print_summary`) covering every
|
|
phase that actually ran. The interrupt is only ever caught here, at
|
|
the outer loop: a phase body mid-batch must let it propagate rather
|
|
than swallow it, so a Ctrl-C during Phase 1 doesn't get silently
|
|
absorbed as "phase 1 failed, try phase 2 anyway".
|
|
|
|
Both ``KeyboardInterrupt`` and ``asyncio.CancelledError`` are caught:
|
|
on Python 3.11+, ``asyncio.Runner`` (which backs ``asyncio.run``)
|
|
translates a real SIGINT into ``main_task.cancel()``, which raises
|
|
``CancelledError`` — not ``KeyboardInterrupt`` — at the currently
|
|
suspended ``await``. Handling it here lets the task complete
|
|
normally with exit code 130 instead of leaking the cancellation out
|
|
to ``asyncio.run`` (which would re-synthesize a bare
|
|
``KeyboardInterrupt`` that typer/click turns into ``Abort``, exit 1,
|
|
with no resume hint printed). Catching ``CancelledError``
|
|
unconditionally is safe here: nothing in this call tree cancels the
|
|
task for reasons other than an interrupt.
|
|
"""
|
|
presenter = TyperPresenter()
|
|
selected = [p for p in PHASES if phase == "all" or p.slug == phase]
|
|
summary = _BackfillSummary()
|
|
current_phase: BackfillPhase | None = None
|
|
try:
|
|
for p in selected:
|
|
current_phase = p
|
|
_print_phase_header(p)
|
|
if p.slug == "vectors":
|
|
result = await _phase_runners._run_phase_vectors(
|
|
auto_yes=auto_yes, presenter=presenter
|
|
)
|
|
summary.vectors = result
|
|
if result.blocked_by_capability:
|
|
logger.warning(
|
|
"cascade_backfill_blocked_by_capability",
|
|
phase=p.slug,
|
|
provider=result.blocked_by_capability,
|
|
)
|
|
_print_summary(summary, exit_code=2)
|
|
return 2
|
|
if result.blocked_by_server:
|
|
# Fail fast BEFORE any embed API call — Phase 2 / 3
|
|
# already do this preflight; without it here,
|
|
# ``--phase all`` against a running server burns
|
|
# through Phase 1's real token budget and only
|
|
# halts at Phase 2 (see PR #361 review J10).
|
|
logger.warning("cascade_backfill_blocked_by_server", phase=p.slug)
|
|
_print_summary(summary, exit_code=3)
|
|
return 3
|
|
if result.aborted:
|
|
# Policy (PR #361 round-3 review #11, accepted):
|
|
# typing ``n`` at Phase 1's confirmation aborts the
|
|
# ENTIRE ``--phase all`` run — not just Phase 1.
|
|
# Rationale: ``--phase all`` is opt-in-to-run-
|
|
# everything. Explicitly declining Phase 1's cost
|
|
# estimate is a decisive "no"; interpreting it as
|
|
# "skip Phase 1 but still run Phase 2/3" would spend
|
|
# LLM/embed budget on phases the user did not see
|
|
# estimates for. Users who want only Phase 2 or 3
|
|
# invoke ``--phase clusters`` or ``--phase skills``
|
|
# explicitly (and see that phase's own estimate).
|
|
logger.info("cascade_backfill_aborted", phase=p.slug)
|
|
_print_aborted()
|
|
_print_summary(summary, exit_code=1)
|
|
return 1
|
|
continue
|
|
if p.slug == "clusters":
|
|
cluster_result = await _phase_runners._run_phase_clusters(
|
|
auto_yes=auto_yes, presenter=presenter
|
|
)
|
|
summary.clusters = cluster_result
|
|
if cluster_result.blocked_by_capability:
|
|
logger.warning(
|
|
"cascade_backfill_blocked_by_capability",
|
|
phase=p.slug,
|
|
provider=cluster_result.blocked_by_capability,
|
|
)
|
|
_print_summary(summary, exit_code=2)
|
|
return 2
|
|
if cluster_result.blocked_by_server:
|
|
logger.warning("cascade_backfill_blocked_by_server", phase=p.slug)
|
|
_print_summary(summary, exit_code=3)
|
|
return 3
|
|
if cluster_result.aborted:
|
|
logger.info("cascade_backfill_aborted", phase=p.slug)
|
|
_print_aborted()
|
|
_print_summary(summary, exit_code=1)
|
|
return 1
|
|
continue
|
|
if p.slug == "skills":
|
|
skill_result = await _phase_runners._run_phase_skills(
|
|
auto_yes=auto_yes, presenter=presenter
|
|
)
|
|
summary.skills = skill_result
|
|
if skill_result.blocked_by_capability:
|
|
logger.warning(
|
|
"cascade_backfill_blocked_by_capability",
|
|
phase=p.slug,
|
|
provider=skill_result.blocked_by_capability,
|
|
)
|
|
_print_summary(summary, exit_code=2)
|
|
return 2
|
|
if skill_result.blocked_by_server:
|
|
logger.warning("cascade_backfill_blocked_by_server", phase=p.slug)
|
|
_print_summary(summary, exit_code=3)
|
|
return 3
|
|
if skill_result.aborted:
|
|
logger.info("cascade_backfill_aborted", phase=p.slug)
|
|
_print_aborted()
|
|
_print_summary(summary, exit_code=1)
|
|
return 1
|
|
continue
|
|
# ``typer.Abort`` fires when ``typer.confirm`` catches KeyboardInterrupt
|
|
# or EOF at the y/N prompt and re-raises it as an ``Abort`` (a
|
|
# ``RuntimeError`` subclass, NOT a ``KeyboardInterrupt``). Typer 0.15+
|
|
# vendored click under ``typer._click`` — ``typer.Abort`` and the
|
|
# standalone ``click.exceptions.Abort`` are DISTINCT classes. Catching
|
|
# only ``click.exceptions.Abort`` would let the real typer-raised
|
|
# abort fall through to the generic ``except Exception`` branch (exit
|
|
# 2, rich traceback). Both are listed so the Ctrl-C-at-prompt path
|
|
# (exit 130 with resume hint) fires regardless of which one arrives.
|
|
except (
|
|
KeyboardInterrupt,
|
|
asyncio.CancelledError,
|
|
typer.Abort,
|
|
click.exceptions.Abort,
|
|
):
|
|
logger.warning(
|
|
"cascade_backfill_interrupted",
|
|
phase=current_phase.slug if current_phase else phase,
|
|
)
|
|
_print_interrupted()
|
|
_print_summary(summary, exit_code=130)
|
|
return 130
|
|
except ProviderNotConfiguredError as exc:
|
|
# Defensive backstop: phase runners preflight capability and
|
|
# return ``blocked_by_capability`` on their own. Reaching this
|
|
# branch means a code path bypassed the preflight — surface the
|
|
# remediation message instead of letting the broad ``except
|
|
# Exception`` below swallow it into "see logs for details".
|
|
logger.error(
|
|
"cascade_backfill_provider_error_escaped_preflight",
|
|
phase=current_phase.slug if current_phase else phase,
|
|
provider=exc.provider,
|
|
)
|
|
typer.secho(f" {exc}", fg=typer.colors.RED)
|
|
_print_summary(summary, exit_code=2)
|
|
return 2
|
|
except Exception:
|
|
logger.exception("cascade_backfill_phase_failed", phase=phase)
|
|
typer.secho("Backfill failed — see logs for details.", fg=typer.colors.RED)
|
|
_print_summary(summary, exit_code=2)
|
|
return 2
|
|
total_failed = _count_failed_rows(summary)
|
|
if total_failed > 0:
|
|
logger.warning(
|
|
"cascade_backfill_completed_with_failures",
|
|
total_failed=total_failed,
|
|
)
|
|
_print_summary(summary, exit_code=4)
|
|
return 4
|
|
_print_summary(summary, exit_code=0)
|
|
return 0
|