From b1441da6071df345ad8a07f9adbcb39202f1ba58 Mon Sep 17 00:00:00 2001 From: Kendrick-Song <61358070+Kendrick-Song@users.noreply.github.com> Date: Wed, 29 Jul 2026 11:05:23 +0800 Subject: [PATCH] refactor(config): make [embedding] and [rerank] soft dependencies (#361) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * refactor(config): make [embedding] and [rerank] soft dependencies Make [embedding] and [rerank] soft runtime dependencies so a freshly-onboarded user can run EverOS end-to-end with only [llm] configured. Previously the server refused to start without embedding, locking out anyone who just wanted keyword-only search. ## Capability tiers - Tier 1 ([llm] only) KEYWORD search, add/flush, md writes, cascade sync - Tier 2 ([llm] + [embedding]) + VECTOR / HYBRID search, reflection, skill extraction, backfill - Tier 3 ([llm] + [embedding] + rerank) + AGENTIC search, knowledge Tier upgrades require a server restart (capability accessors cache for the process lifetime). Tier downgrades are read-safe: a Tier-3 user who drops [rerank] can still read/rename/delete existing knowledge documents; only write/search endpoints return 422. ## What changed - Component accessors — component/{embedding,rerank,llm}/accessor.py are the single process-wide provider singletons. service/* never maintains parallel singletons; it consumes get_embedding_capability() / get_rerank_capability() / get_llm_client() directly. Build-time ValueError from the factory is logged as capability_build_failed (was silently swallowed). - Error mapping — ProviderNotConfiguredError -> 422 with everos.toml section hints (never EVEROS_* env-var strings). LanceDBMigrationError fails loud with escalating recovery guidance (restart -> wipe index). LLMNotConfiguredError in search maps to None for KEYWORD degradation. - Nullable-vector LanceDB migration — schema v2 makes the vector column nullable so Tier-1 rows can land without embeddings. Migration is guarded by a cross-process memory_root_lock (fcntl.flock + anyio.to_thread) and runs optimize() per table after Phase-1 backfill to reclaim manifest bloat. - Cascade — knowledge handlers register unconditionally (Tier-3 -> Tier-2/1 downgrade no longer strands DELETE); embed-requiring strategies use body-guards that check capability.available at execution time. _TABLE_SPECS has an import-time drift assertion against BUSINESS_SCHEMAS_WITH_VECTOR. - `everos cascade backfill` CLI — Phase-1 (embed missing vectors) / Phase-2 (emit synthetic events for cascaded processing) / Phase-3 (sync new skill files). Exit codes: 0 / 1 / 2 / 3 (server running preflight) / 4 (COMPLETED_WITH_FAILURES — per-row failures rolled up) / 130 (SIGINT). OMEConfig.crash_recovery_enabled=False in backfill engines prevents stale-RUNNING rows re-enqueuing into a smaller strategy registry. - /health — reports capabilities + disabled_features per tier so ops can distinguish "boots but degraded" from "boots and full". - Presentation split — memory / service / infra never import typer / click. TyperPresenter Protocol + run_backfill() live in entrypoints/cli/commands/_backfill_cmd.py. Enforced by import-linter. - Startup hint — unconditional count_rows(filter="vector IS NULL") sweep emits unbackfilled_memory_rows (event name + hint text pinned) when Tier-1 rows exist. ParserLifespanProvider warms the everalgo.parser import at boot so /health doesn't block on first call. - Knowledge upload UTF-8 short-circuit — _looks_like_utf8_text() routes text/* mime and known plaintext extensions (md/txt/rst) straight to UTF-8 decode instead of the parser. Prevents 503 Multimodal-not-configured when Tier 3 sans [multimodal] uploads a markdown doc. ## Sync history with main (2 merges collapsed into this squash) Merged origin/main at 6dcd3eb (v1.1.4 -> v1.2.0 adds OTel tracing, /api/v2 alias, TracingLifespanProvider, per-cascade-embedding span fix, memory-op instrumentation) and later at 42629df (PR #366 backfills v1.1.4 CWE-22 knowledge path traversal fix + cascade retry-budget rework + errors.py -> core.errors.ExternalServiceError). Key merge decisions: - service/search.py adopts single wrap site — component.llm accessor already applies UsageRecordingClient when observability is on; service layer never keeps a parallel LLM singleton (Round-1 CR rule: "service layer never maintains parallel singletons"). - Knowledge router prefix moved to /knowledge; create_app() mounts it under both /api/v1 and /api/v2. - Cascade retry classification uses ExternalServiceError from core.errors (cascade/errors.py deleted). _MAX_TOTAL_RETRIES=12 cross-cycle budget preserved. - Fixed backport typo: MemoryRoot.default() -> MemoryRoot.resolve() (no .default() classmethod exists — main PR #366 shipped a broken call). ## Verified layering $ git grep -l "^import typer\|^from typer" src/everos/{memory,service,infra} # empty $ git grep -l "^import click\|^from click" src/everos/{memory,service,infra} # empty Memory / service / infra layers clean of CLI presentation libraries. ## Review history Three rounds of Fable 5 (opus) code review across the pre-squash commit history closed 38 findings total: - Round 1: 10 findings (fail-loud migration, backfill hardening, knowledge router gate scoping, SearchManager guards, profile throttle lift) - Round 2: 13 findings (hermetic test env, hot-reload doc drift, knowledge handler registration, Phase 3 sync guarantee, Phase 2 idempotency, profile event-first path, OMEConfig crash-recovery gate, cross-process migration lock, batch embed per-row fallback, LanceDB optimize, typer/click layer split) - Round 3: 15 Minor cleanups (accessor unification, marker revert, episode query hygiene, --verbose subcommand, parser lifespan warm, task-number scrub, temporal-overlap test, real-SIGINT slow mark, 4 design-note back-references) Full per-round context lives in the PR description on GitHub. ## Test plan - make lint (ruff + import-linter 3 contracts + assets + deprecated-names + github-docs + datetime + OpenAPI drift) - Hermetic env full pytest — 2027 passed / 7 deselected (7 = slow + live_llm markers) - Manual e2e across Tier 1/2/3 (21/21 assertions across v1/v2 double-mount and Tier 3 -> Tier 2 downgrade) - /health reports correct capabilities + disabled_features per tier ## Known follow-ups - .superpowers/sdd/followup-http-bridge.md (gitignored) — Path A for spec §10's "backfill 期间 EverOS 完全可用" promise - _TABLE_SCHEMA_VERSION docstring — v3+ migrations need a version dispatch table - extract_user_profile.py throttle-counter block — replace LanceDB count_by_owner with a sqlite memcell count Co-Authored-By: Claude Opus 4.7 (1M context) * fix(review): close 3 blockers surfaced by round-4 review N1. cluster_repo.find_cluster_id_for_member was cross-owner-unsafe. Its reverse index (member_type, member_id) alone cannot disambiguate two owners whose entry_id happens to collide — entry_id is deliberately only per-owner unique (see entries.py:47: 'Cross-user uniqueness is handled at the database layer via a composite _ field; it is not encoded into the EntryId string itself'). Phase 2's _scan_all_rows crosses all owners, so on any multi-owner root, same-day seq=1 episodes under different owners would either false-hit each other's cluster or be silently skipped from clustering. Add required (app_id, project_id, owner_id) keyword args + JOIN Cluster (which already carries scope) to filter by parent scope. Prior signature had zero production callers except two the same PR just added, so the API break is contained. Regression test: two owners persist a cluster each around the same entry_id, each lookup resolves to its own owner's cluster, a third owner's lookup returns None. N2. Ctrl-C / EOF at the y/N prompt was landing on the generic except Exception branch (exit 2 with rich traceback) instead of the exit-130 interrupt path. Root cause: typer 0.15+ vendored click under typer._click, so typer.Abort and the standalone click.exceptions.Abort are distinct classes. The interrupt-branch catch only listed the standalone one; every existing 'abort' test was manually raising click.exceptions.Abort so the miss was a false-positive guard rail. Widen the catch to (typer.Abort, click.exceptions.Abort) and declare click as a first-class dependency (it was only pulled in via uvicorn). Regression test: raise real typer.Abort() at the confirm step and assert exit 130 + INTERRUPTED banner. N3. _looks_like_utf8_text used mime.startswith('text/'), which caught text/html as well. HTML uploads then bypassed everalgo's _aparse_html — losing clean_html_for_llm (strips