refactor(config): make [embedding] and [rerank] soft dependencies (#361)

* refactor(config): make [embedding] and [rerank] soft dependencies

Make [embedding] and [rerank] soft runtime dependencies so a
freshly-onboarded user can run EverOS end-to-end with only [llm]
configured. Previously the server refused to start without embedding,
locking out anyone who just wanted keyword-only search.

## Capability tiers

- Tier 1 ([llm] only)                   KEYWORD search, add/flush, md
                                        writes, cascade sync
- Tier 2 ([llm] + [embedding])          + VECTOR / HYBRID search,
                                        reflection, skill extraction,
                                        backfill
- Tier 3 ([llm] + [embedding] + rerank) + AGENTIC search, knowledge

Tier upgrades require a server restart (capability accessors cache
for the process lifetime). Tier downgrades are read-safe: a Tier-3
user who drops [rerank] can still read/rename/delete existing
knowledge documents; only write/search endpoints return 422.

## What changed

- Component accessors — component/{embedding,rerank,llm}/accessor.py
  are the single process-wide provider singletons. service/* never
  maintains parallel singletons; it consumes get_embedding_capability()
  / get_rerank_capability() / get_llm_client() directly. Build-time
  ValueError from the factory is logged as capability_build_failed
  (was silently swallowed).
- Error mapping — ProviderNotConfiguredError -> 422 with everos.toml
  section hints (never EVEROS_* env-var strings).
  LanceDBMigrationError fails loud with escalating recovery guidance
  (restart -> wipe index). LLMNotConfiguredError in search maps to
  None for KEYWORD degradation.
- Nullable-vector LanceDB migration — schema v2 makes the vector
  column nullable so Tier-1 rows can land without embeddings.
  Migration is guarded by a cross-process memory_root_lock
  (fcntl.flock + anyio.to_thread) and runs optimize() per table
  after Phase-1 backfill to reclaim manifest bloat.
- Cascade — knowledge handlers register unconditionally (Tier-3 ->
  Tier-2/1 downgrade no longer strands DELETE); embed-requiring
  strategies use body-guards that check capability.available at
  execution time. _TABLE_SPECS has an import-time drift assertion
  against BUSINESS_SCHEMAS_WITH_VECTOR.
- `everos cascade backfill` CLI — Phase-1 (embed missing vectors) /
  Phase-2 (emit synthetic events for cascaded processing) / Phase-3
  (sync new skill files). Exit codes: 0 / 1 / 2 / 3 (server running
  preflight) / 4 (COMPLETED_WITH_FAILURES — per-row failures rolled
  up) / 130 (SIGINT). OMEConfig.crash_recovery_enabled=False in
  backfill engines prevents stale-RUNNING rows re-enqueuing into a
  smaller strategy registry.
- /health — reports capabilities + disabled_features per tier so ops
  can distinguish "boots but degraded" from "boots and full".
- Presentation split — memory / service / infra never import typer /
  click. TyperPresenter Protocol + run_backfill() live in
  entrypoints/cli/commands/_backfill_cmd.py. Enforced by
  import-linter.
- Startup hint — unconditional count_rows(filter="vector IS NULL")
  sweep emits unbackfilled_memory_rows (event name + hint text
  pinned) when Tier-1 rows exist. ParserLifespanProvider warms the
  everalgo.parser import at boot so /health doesn't block on first
  call.
- Knowledge upload UTF-8 short-circuit — _looks_like_utf8_text()
  routes text/* mime and known plaintext extensions (md/txt/rst)
  straight to UTF-8 decode instead of the parser. Prevents 503
  Multimodal-not-configured when Tier 3 sans [multimodal] uploads a
  markdown doc.

## Sync history with main (2 merges collapsed into this squash)

Merged origin/main at 6dcd3eb (v1.1.4 -> v1.2.0 adds OTel tracing,
/api/v2 alias, TracingLifespanProvider, per-cascade-embedding span
fix, memory-op instrumentation) and later at 42629df (PR #366
backfills v1.1.4 CWE-22 knowledge path traversal fix + cascade
retry-budget rework + errors.py -> core.errors.ExternalServiceError).
Key merge decisions:
- service/search.py adopts single wrap site — component.llm accessor
  already applies UsageRecordingClient when observability is on;
  service layer never keeps a parallel LLM singleton (Round-1 CR
  rule: "service layer never maintains parallel singletons").
- Knowledge router prefix moved to /knowledge; create_app() mounts
  it under both /api/v1 and /api/v2.
- Cascade retry classification uses ExternalServiceError from
  core.errors (cascade/errors.py deleted). _MAX_TOTAL_RETRIES=12
  cross-cycle budget preserved.
- Fixed backport typo: MemoryRoot.default() -> MemoryRoot.resolve()
  (no .default() classmethod exists — main PR #366 shipped a broken
  call).

## Verified layering

    $ git grep -l "^import typer\|^from typer" src/everos/{memory,service,infra}
    # empty
    $ git grep -l "^import click\|^from click" src/everos/{memory,service,infra}
    # empty

Memory / service / infra layers clean of CLI presentation libraries.

## Review history

Three rounds of Fable 5 (opus) code review across the pre-squash
commit history closed 38 findings total:
- Round 1: 10 findings (fail-loud migration, backfill hardening,
  knowledge router gate scoping, SearchManager guards, profile
  throttle lift)
- Round 2: 13 findings (hermetic test env, hot-reload doc drift,
  knowledge handler registration, Phase 3 sync guarantee, Phase 2
  idempotency, profile event-first path, OMEConfig crash-recovery
  gate, cross-process migration lock, batch embed per-row fallback,
  LanceDB optimize, typer/click layer split)
- Round 3: 15 Minor cleanups (accessor unification, marker revert,
  episode query hygiene, --verbose subcommand, parser lifespan warm,
  task-number scrub, temporal-overlap test, real-SIGINT slow mark,
  4 design-note back-references)

Full per-round context lives in the PR description on GitHub.

## Test plan

- make lint (ruff + import-linter 3 contracts + assets +
  deprecated-names + github-docs + datetime + OpenAPI drift)
- Hermetic env full pytest — 2027 passed / 7 deselected (7 = slow +
  live_llm markers)
- Manual e2e across Tier 1/2/3 (21/21 assertions across v1/v2
  double-mount and Tier 3 -> Tier 2 downgrade)
- /health reports correct capabilities + disabled_features per tier

## Known follow-ups

- .superpowers/sdd/followup-http-bridge.md (gitignored) — Path A for
  spec §10's "backfill 期间 EverOS 完全可用" promise
- _TABLE_SCHEMA_VERSION docstring — v3+ migrations need a version
  dispatch table
- extract_user_profile.py throttle-counter block — replace LanceDB
  count_by_owner with a sqlite memcell count

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(review): close 3 blockers surfaced by round-4 review

N1. cluster_repo.find_cluster_id_for_member was cross-owner-unsafe.
Its reverse index (member_type, member_id) alone cannot disambiguate
two owners whose entry_id happens to collide — entry_id is
deliberately only per-owner unique (see entries.py:47:
'Cross-user uniqueness is handled at the database layer via a
composite <user_id>_<entry_id> field; it is not encoded into the
EntryId string itself'). Phase 2's _scan_all_rows crosses all
owners, so on any multi-owner root, same-day seq=1 episodes under
different owners would either false-hit each other's cluster or
be silently skipped from clustering. Add required (app_id,
project_id, owner_id) keyword args + JOIN Cluster (which already
carries scope) to filter by parent scope. Prior signature had zero
production callers except two the same PR just added, so the
API break is contained. Regression test: two owners persist a
cluster each around the same entry_id, each lookup resolves to
its own owner's cluster, a third owner's lookup returns None.

N2. Ctrl-C / EOF at the y/N prompt was landing on the generic
except Exception branch (exit 2 with rich traceback) instead of
the exit-130 interrupt path. Root cause: typer 0.15+ vendored
click under typer._click, so typer.Abort and the standalone
click.exceptions.Abort are distinct classes. The interrupt-branch
catch only listed the standalone one; every existing 'abort'
test was manually raising click.exceptions.Abort so the miss
was a false-positive guard rail. Widen the catch to
(typer.Abort, click.exceptions.Abort) and declare click as a
first-class dependency (it was only pulled in via uvicorn).
Regression test: raise real typer.Abort() at the confirm step
and assert exit 130 + INTERRUPTED banner.

N3. _looks_like_utf8_text used mime.startswith('text/'), which
caught text/html as well. HTML uploads then bypassed everalgo's
_aparse_html — losing clean_html_for_llm (strips <script>/<style>
/<nav>/<iframe> + HTML comments) and the 1 MiB output cap. A
40 MiB .html with <script> bodies and <!-- prompt injection -->
comments would flow straight into the extraction LLM. Replace
with explicit allowlist {text/plain, text/markdown, text/x-rst,
text/x-markdown}; text/html and any future text/* mime now
default to the parser path. Test matrix asserts text/html →
False (was regressed as True by the earlier commit).

Hermetic env full pytest: 2033 passed / 7 deselected (+6 tests
from these regressions).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(review): close round-4 major + minor + PR body errata

Round-4 review-driven cleanup. Blocker fixes (N1/N2/N3) landed in
561b5fe. This commit closes the remaining CONFIRMED items:

Major:
- J3 MemoryRoot.default() -> resolve() was a breaking public API
  rename this branch introduced (main still has default()). Adds
  default() as a backward-compat alias forwarding to resolve()
  with DeprecationWarning; CHANGELOG entry under Unreleased.
- J4 episode_repo.list_by_owner_after_ts(limit=N) truncates in
  fragment order (== insertion order), NOT newest-first. Docstring
  now spells out the trap so a future caller passing limit for a
  'newest N' window doesn't silently get the oldest N.
- J5.2 TyperPresenter.nothing_to_backfill picked colour via
  'could not be read' in message — a domain wording change
  would silently flip yellow -> green. Signature gains explicit
  scan_failed: bool kwarg; CLI colour-picks off the flag.
- J5.5 phase_header was Protocol-declared but never dispatched
  (run_backfill calls _print_phase_header directly). Removed the
  dead Protocol method + both no-op implementations.
- J6.3 3 inline from everos.core.errors import ... inside
  Phase 1/2/3 preflights promoted to a single top-level import.
- J7 subject-side embed failure was silently exit-0 because
  rows_processed advanced whenever any side wrote. Now: a row with
  a needed side still NULL counts as rows_failed (exit 4 =
  COMPLETED_WITH_FAILURES). Gated on spec.subject_of + row.needs_*
  + row.subject_text so non-Episode tables and subject-empty rows
  don't false-positive.
- J9 test_migration_cross_process.py did NOT actually test cross-
  process (all 5 tests mock memory_root_lock). Renamed to
  test_migration_lock_wiring.py; docstring now scopes it to
  'lock-invocation wiring' and points at test_core/…/test_locking.py
  for real flock coverage.
- J10 Phase 1 lacked the server-running preflight Phase 2/3 have.
  --phase all against a live server would burn Phase 1 embed API
  calls (real cost) before Phase 2 halted with exit 3. Phase 1 now
  probes _probe_ome_lock_available first; regression test in
  test_backfill_preflight.py; upgrade_path integration patches the
  probe so its in-process 'server + backfill' scenario stays valid.

Minor:
- M1 knowledge upload with NUL byte or filename > 255 bytes UTF-8
  used to raise ValueError/OSError at write_bytes → 500 with a
  half-written md left on disk. _safe_original_filename now
  rejects both up front with InvalidInputError (→ 400).
- M2 backfill optimize() now passes cleanup_older_than=timedelta(0)
  so older manifest versions are physically pruned (previous call
  compacted fragments but left the manifest chain on disk).
- M3 verify_business_schemas remediation text used to jump straight
  to 'rm -rf ~/.everos/.index/lancedb'; now walks restart → wipe.
- M5 multimodal/accessor.py capability_build_failed warning added
  so all four provider accessors log symmetrically (was silent).
- M7 test_knowledge_api parser-absence tests call
  parser_available.cache_clear() around the sys.modules patch so
  the lru_cache doesn't strand a stale True/False.
- M8 cascade_handler_embed_skipped (6 handlers) demoted INFO → DEBUG:
  Tier 1 imports were generating N × 6 handler-info lines per md.
- M10.1 test_drift_scenario_would_raise was a tautology (compared
  two hardcoded string sets, never touched the guard). Now
  monkey-patches BUSINESS_SCHEMAS_WITH_VECTOR to a superset and
  reloads _backfill, proving the import-time RuntimeError fires.
- M11 test_cascade_verbose_position subprocess.run calls gain
  env= — scrubs EVEROS_* from the developer environment so the
  same footgun as round-2 B1 doesn't re-appear inside subprocesses.
- M14 health() -> dict degraded the OpenAPI schema to
  additionalProperties: true. Introduce HealthResponse +
  HealthCapabilities Pydantic models so clients get real field
  shape; docs/openapi.json regenerated.
- M16 routes/knowledge.py:_require_knowledge_capabilities docstring
  claimed cascade.registry.build_handlers still gates
  knowledge_topic/knowledge_document, contradicting
  registry.py:177-194 (gate removed there, moved to HTTP layer).
  Rewritten to describe the current design accurately.

Hermetic env full pytest: 2037 passed / 7 deselected.

Explicitly deferred to followup:
- J1 lazy multimodal client (needs everalgo signature change)
- J2 tier definition (knowledge = Tier 3 whole)
- J5.1/3/4 broader presentation-split refactor
- J6.1/2 backfill dispatch and _backfill_table refactor
- J8 Phase 1 keyset pagination for bulk migration OOM
- M4 flock timeout/waiting log/re-entry (core/persistence refactor)
- M6 UTF-8 codec strategy (BOM/UTF-16/GBK)
- M9 count_by_owner monotonicity (latent, INTERVAL=1 short-circuits)
- M13 --phase all multi-phase combined-outcome test coverage
- M15 PR-marker rationale comments (16 occurrences, all inert)

M12 was refuted (both event names exist).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Kendrick-Song
2026-07-29 11:05:23 +08:00
committed by GitHub
co-authored by Claude Opus 4.7
parent 649046b0df
commit b1441da607
185 changed files with 16514 additions and 910 deletions
+9
View File
@@ -9,6 +9,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Changed
- **`MemoryRoot.default()` → `MemoryRoot.resolve()`** — the classmethod that
resolves the memory root from `--root` / `EVEROS_ROOT` / default was
renamed to make its behavior explicit (`resolve` walks the precedence
chain; `default` was ambiguous with "default location"). `MemoryRoot`
is publicly exported from `everos.core.persistence`; callers outside
the repo may have used the old name. **A `default()` alias is kept**
as a backward-compatibility shim that forwards to `resolve()` and
emits a `DeprecationWarning`. The alias will be removed in a future
major release — update call sites when convenient.
- **Uncalibrated recall scores moved to their own name** — `KEYWORD` and
single-route `VECTOR` searches now report their top score as
`recall_top_score_raw`; `recall_top_score` is reserved for the calibrated
+67 -6
View File
@@ -12,7 +12,7 @@
"health"
],
"summary": "Health",
"description": "Liveness probe — returns ``{\"status\": \"ok\"}`` with HTTP 200.",
"description": "Liveness probe with capabilities and disabled features.",
"operationId": "health_health_get",
"responses": {
"200": {
@@ -20,11 +20,7 @@
"content": {
"application/json": {
"schema": {
"additionalProperties": {
"type": "string"
},
"type": "object",
"title": "Response Health Health Get"
"$ref": "#/components/schemas/HealthResponse"
}
}
}
@@ -2719,6 +2715,71 @@
"type": "object",
"title": "HTTPValidationError"
},
"HealthCapabilities": {
"properties": {
"llm": {
"type": "boolean",
"title": "Llm"
},
"embed": {
"type": "boolean",
"title": "Embed"
},
"rerank": {
"type": "boolean",
"title": "Rerank"
},
"multimodal_llm": {
"type": "boolean",
"title": "Multimodal Llm"
},
"parser": {
"type": "boolean",
"title": "Parser"
}
},
"type": "object",
"required": [
"llm",
"embed",
"rerank",
"multimodal_llm",
"parser"
],
"title": "HealthCapabilities",
"description": "Availability flags for the five capability probes.\n\nField order matches the health-endpoint payload contract; clients\nkey off these names to decide whether to expose optional features."
},
"HealthResponse": {
"properties": {
"status": {
"type": "string",
"title": "Status"
},
"version": {
"type": "string",
"title": "Version"
},
"capabilities": {
"$ref": "#/components/schemas/HealthCapabilities"
},
"disabled_features": {
"items": {
"type": "string"
},
"type": "array",
"title": "Disabled Features"
}
},
"type": "object",
"required": [
"status",
"version",
"capabilities",
"disabled_features"
],
"title": "HealthResponse",
"description": "Response schema for ``GET /health``.\n\nDeclared as a Pydantic model (not ``dict``) so the generated\nOpenAPI schema carries the full field shape — ``capabilities`` and\n``disabled_features`` are typed. A bare ``-> dict`` return type\ndegrades the OpenAPI response to ``additionalProperties: true``,\nwhich robs clients (and codegen) of any structure to lean on."
},
"KnowledgeSearchRequest": {
"properties": {
"query": {
+8
View File
@@ -50,6 +50,14 @@ dependencies = [
# CLI + TUI
"typer>=0.12.0",
# click is a transitive dep of uvicorn today, but our CLI raises
# ``click.exceptions.Abort`` directly (e.g. tests that inject aborts)
# and typer 0.15+ vendored a copy of click under ``typer._click`` — so
# the two classes are DIFFERENT even though users only ever see
# ``typer.Abort``. We depend on the standalone click package so
# ``click.exceptions.Abort`` stays importable and both classes are
# covered in the Ctrl-C catch (see backfill exit-130 path).
"click>=8.1",
"textual>=8.2.7",
# Tokenization (BM25 Chinese support)
+2 -2
View File
@@ -51,7 +51,7 @@ def _print_header(mode: str, fixture_path: Path, session_id: str) -> None:
print(f" everos e2e memorize · mode={mode}")
print(f" fixture : {fixture_path.name}")
print(f" session_id : {session_id}")
print(f" memory root : {MemoryRoot.default().root}")
print(f" memory root : {MemoryRoot.resolve().root}")
llm_state = "<configured>" if get_llm_client() else "<None — pipeline will skip>"
print(f" llm_client : {llm_state}")
print("=" * 72)
@@ -59,7 +59,7 @@ def _print_header(mode: str, fixture_path: Path, session_id: str) -> None:
def _list_written_files(session_id: str, mode: str) -> None:
"""Walk memory root and print files touched in this run."""
root = MemoryRoot.default().root
root = MemoryRoot.resolve().root
cutoff = time.time() - 600 # files modified in the last 10 min
print()
print("─── files modified within the last 10 minutes under memory root ───")
+74
View File
@@ -0,0 +1,74 @@
"""Capabilities inference and feature availability logic.
Compute which features are disabled based on available capabilities.
Used by the health endpoint, startup banner, and related diagnostics.
Two distinct feature-name vocabularies exist in the refactored codebase.
They deliberately do NOT share a source of truth:
- ``ProviderNotConfiguredError.feature`` (raised by
``SearchManager._validate_components`` and error handlers): per-request
granular tag identifying the specific search mode or endpoint that failed —
``"vector"`` | ``"user_hybrid"`` | ``"agent_hybrid"`` | ``"agentic_search"``
| ``"knowledge"`` | ``"skill_extraction_backfill"``. Appears in HTTP 422
response ``message``. Naming mirrors the ``SearchMethod`` enum values plus
endpoint-scoped tags.
- ``compute_disabled_features()`` (below): capability-level tag identifying
a whole feature category disabled by the current tier —
``"vector_search"`` | ``"hybrid_search"`` | ``"agentic_search"``
| ``"reflection"`` | ``"skill_extraction"`` | ``"knowledge"``
| ``"multimodal_upload"``. Appears in ``GET /health`` response
``disabled_features``.
Callers of either surface should treat these vocabularies as stable client
contracts — do not unify them without a coordinated migration on the client side.
"""
from __future__ import annotations
def compute_disabled_features(caps: dict[str, bool]) -> list[str]:
"""Derive the list of disabled features from capability availability.
Args:
caps: Dictionary with keys "llm", "embed", "rerank", "multimodal_llm", "parser"
and boolean availability values. Note: ``caps["llm"]`` is
accepted for shape symmetry but NOT read here — LLM is a
Tier-1 hard requirement enforced at server startup
(``LLMLifespanProvider``), so any process reaching this
function is guaranteed to have LLM available. If LLM ever
becomes soft, add an ``if not caps["llm"]`` branch here
covering the LLM-dependent features.
Returns:
List of feature names that are disabled due to missing capabilities.
Possible values: "vector_search", "hybrid_search", "agentic_search",
"reflection", "skill_extraction", "knowledge", "multimodal_upload".
"""
disabled: list[str] = []
# Embedding-dependent features
if not caps["embed"]:
disabled.extend(
[
"vector_search",
"hybrid_search",
"reflection",
"skill_extraction",
]
)
# Rerank-dependent feature
if not caps["rerank"]:
disabled.append("agentic_search")
# Knowledge requires both embedding and rerank
if not (caps["embed"] and caps["rerank"]):
disabled.append("knowledge")
# Multimodal upload requires both multimodal_llm and parser
if not (caps["multimodal_llm"] and caps["parser"]):
disabled.append("multimodal_upload")
return disabled
+13 -4
View File
@@ -6,9 +6,18 @@ Public surface:
- :class:`EmbeddingProvider` — Protocol every provider satisfies.
- :class:`EmbeddingServiceError` — provider-side failure.
- :class:`EmbeddingError` — backward-compat alias for ``EmbeddingServiceError``.
- :class:`EmbeddingCapability` — soft-dependency wrapper around an
optional :class:`EmbeddingProvider` (``available`` / ``embed_or_none``
/ ``require``).
- :class:`OpenAIEmbeddingProvider` — concrete provider for any
OpenAI-protocol embeddings endpoint (DeepInfra, vLLM, OpenAI, …).
- :func:`build_embedding_provider` — settings-driven factory.
- :func:`get_embedding_capability` — process-wide lazy singleton
accessor for :class:`EmbeddingCapability`. There is no separate
``get_embedder`` accessor: consumers that need a provider call
``get_embedding_capability().require()``, which routes every caller
through a single shared provider (and its single ``AsyncOpenAI``
client + ``asyncio.Semaphore``).
External usage::
@@ -19,19 +28,19 @@ External usage::
from everos.core.errors import EmbeddingServiceError as EmbeddingServiceError
from .accessor import EmbeddingNotConfiguredError as EmbeddingNotConfiguredError
from .accessor import get_embedder as get_embedder
from .accessor import get_embedding_capability as get_embedding_capability
from .capability import EmbeddingCapability as EmbeddingCapability
from .factory import build_embedding_provider as build_embedding_provider
from .openai_provider import OpenAIEmbeddingProvider as OpenAIEmbeddingProvider
from .protocol import EmbeddingError as EmbeddingError
from .protocol import EmbeddingProvider as EmbeddingProvider
__all__ = [
"EmbeddingCapability",
"EmbeddingError",
"EmbeddingNotConfiguredError",
"EmbeddingProvider",
"EmbeddingServiceError",
"OpenAIEmbeddingProvider",
"build_embedding_provider",
"get_embedder",
"get_embedding_capability",
]
+37 -27
View File
@@ -1,14 +1,16 @@
"""Process-wide embedding provider accessor.
"""Process-wide embedding capability accessor.
Lazy singleton mirror of :func:`everos.component.llm.get_llm_client`:
first call reads settings and builds the OpenAI-protocol embedding
client; subsequent calls return the cached instance. Strategies and
other components that need a process-wide embedder import this rather
than threading the provider through their constructors.
Lazy singleton for :class:`EmbeddingCapability`. The first call reads
settings and attempts to build an OpenAI-protocol embedding provider;
subsequent calls return the cached wrapper. Consumers that must have an
embedder call ``get_embedding_capability().require()``; consumers that
can degrade gracefully use ``.embed_or_none`` or check ``.available``.
Raises :class:`EmbeddingNotConfiguredError` when credentials are missing
so misconfiguration surfaces at the call site (or at app startup via a
lifespan provider) instead of silently degrading.
There is deliberately no separate ``get_embedder()`` accessor: routing
every consumer through the capability keeps a single provider (and its
underlying ``AsyncOpenAI`` client + ``asyncio.Semaphore``) per process,
so the configured ``max_concurrent`` bound holds instead of silently
doubling.
"""
from __future__ import annotations
@@ -16,33 +18,41 @@ from __future__ import annotations
from everos.config import load_settings
from everos.core.observability.logging import get_logger
from .capability import EmbeddingCapability
from .factory import build_embedding_provider
from .protocol import EmbeddingProvider
logger = get_logger(__name__)
class EmbeddingNotConfiguredError(RuntimeError):
"""Raised when ``settings.embedding`` lacks ``model``/``api_key``/``base_url``."""
_capability: EmbeddingCapability | None = None
_embedder: EmbeddingProvider | None = None
def get_embedding_capability() -> EmbeddingCapability:
"""Return the process-wide :class:`EmbeddingCapability`. Never raises.
On the first call, builds and caches a capability from current
settings — ``available`` is ``False`` when the provider cannot be
built (missing fields, unsupported provider name, malformed URL, …).
The build outcome is cached, so a later settings change requires a
process restart to take effect.
def get_embedder() -> EmbeddingProvider:
"""Return the singleton :class:`EmbeddingProvider`.
Raises:
EmbeddingNotConfiguredError: When required settings fields are
unset. See :func:`build_embedding_provider` for the exact
keys.
Configuration failures (:class:`ValueError` from
:func:`build_embedding_provider`) are logged at ``warning`` level:
the downstream :class:`ProviderNotConfiguredError` message maps both
"user hasn't configured it" and "user configured it wrong" onto the
same HTTP 422, so the log line is the only place an operator can
tell those two states apart.
"""
global _embedder
if _embedder is not None:
return _embedder
global _capability
if _capability is not None:
return _capability
try:
_embedder = build_embedding_provider(load_settings().embedding)
provider = build_embedding_provider(load_settings().embedding)
except ValueError as exc:
raise EmbeddingNotConfiguredError(str(exc)) from exc
logger.info("embedder_built")
return _embedder
logger.warning(
"embedding_capability_build_failed",
reason=str(exc),
)
provider = None
_capability = EmbeddingCapability(provider=provider)
return _capability
@@ -0,0 +1,59 @@
"""EmbeddingCapability — soft-dependency wrapper for EmbeddingProvider.
The wrapper encapsulates ``Optional[EmbeddingProvider]`` so consumers
never see ``None`` directly. Two consumption modes:
- :meth:`embed_or_none` for soft-degrade write paths (cascade handlers
that write ``vector=NULL`` when embedding is unavailable).
- :meth:`require` for hard-required paths (search VECTOR/HYBRID/AGENTIC)
that raise :class:`ProviderNotConfiguredError` -> HTTP 422 when missing.
Design rationale: EverOS has multiple code paths that need to know
whether embedding is available. Rather than each callsite reading
settings and re-implementing the check, the check lives in
:func:`everos.component.embedding.accessor.get_embedding_capability`
(module-level singleton); every consumer asks the capability.
"""
from __future__ import annotations
from dataclasses import dataclass
from everos.core.errors import ProviderNotConfiguredError
from .protocol import EmbeddingProvider
@dataclass(frozen=True)
class EmbeddingCapability:
"""Wraps an optional EmbeddingProvider with soft / strict consumption modes."""
provider: EmbeddingProvider | None
@property
def available(self) -> bool:
"""True iff a provider was successfully constructed."""
return self.provider is not None
async def embed_or_none(self, text: str) -> list[float] | None:
"""Embed ``text`` if available, else return ``None``.
For write paths that can proceed without a vector (cascade
handlers write ``vector=NULL`` on the LanceDB row).
"""
if self.provider is None:
return None
return list(await self.provider.embed(text))
def require(self) -> EmbeddingProvider:
"""Return the provider, or raise :class:`ProviderNotConfiguredError`.
For paths that cannot proceed without embedding (search
VECTOR/HYBRID/AGENTIC methods). Caller supplies its own
``feature``/``alternative_hint`` context by catching and
re-raising if needed; this base call only identifies the
missing provider as ``"embedding"``.
"""
if self.provider is None:
raise ProviderNotConfiguredError(provider="embedding")
return self.provider
+4 -10
View File
@@ -2,6 +2,7 @@
from __future__ import annotations
from everos.component.utils.config_hints import missing_config_error
from everos.config import EmbeddingSettings
from .openai_provider import OpenAIEmbeddingProvider
@@ -32,18 +33,11 @@ def build_embedding_provider(
ValueError: If ``model``, ``api_key`` or ``base_url`` is unset.
"""
if not settings.model:
raise ValueError(
"Embedding model is not configured "
"(set EVEROS_EMBEDDING__MODEL or [embedding] model in user toml)"
)
raise ValueError(missing_config_error("Embedding model", "embedding"))
if not settings.api_key or not settings.api_key.get_secret_value():
raise ValueError(
"Embedding api_key is not configured (set EVEROS_EMBEDDING__API_KEY)"
)
raise ValueError(missing_config_error("Embedding api_key", "embedding"))
if not settings.base_url:
raise ValueError(
"Embedding base_url is not configured (set EVEROS_EMBEDDING__BASE_URL)"
)
raise ValueError(missing_config_error("Embedding base_url", "embedding"))
return OpenAIEmbeddingProvider(
model=settings.model,
api_key=settings.api_key.get_secret_value(),
+3 -3
View File
@@ -17,6 +17,7 @@ from everalgo.llm.protocols import LLMClient
from everalgo.llm.types import ChatMessage, ChatResponse
from pydantic import BaseModel
from everos.component.utils.config_hints import missing_config_error
from everos.config import load_settings
from everos.core.observability.logging import get_logger
@@ -93,7 +94,7 @@ def get_llm_client() -> LLMClient:
)
if not api_key or not llm_cfg.base_url:
raise LLMNotConfiguredError(
"LLM is required; set EVEROS_LLM__API_KEY + EVEROS_LLM__BASE_URL"
missing_config_error("LLM api_key and base_url", "llm")
)
client: LLMClient = build_client(
LLMConfig(
@@ -133,8 +134,7 @@ def get_multimodal_llm_client() -> LLMClient:
api_key = cfg.api_key.get_secret_value() if cfg.api_key is not None else None
if not api_key or not cfg.base_url:
raise LLMNotConfiguredError(
"Multimodal LLM is required for parsing; set "
"EVEROS_MULTIMODAL__API_KEY + EVEROS_MULTIMODAL__BASE_URL"
missing_config_error("Multimodal LLM api_key and base_url", "multimodal")
)
_multimodal_client = build_client(
LLMConfig(
+3 -8
View File
@@ -2,6 +2,7 @@
from __future__ import annotations
from everos.component.utils.config_hints import missing_config_error
from everos.config import LLMSettings
from .openai_provider import OpenAIProvider
@@ -29,15 +30,9 @@ def build_llm_provider(settings: LLMSettings) -> LLMClient:
ValueError: If ``api_key`` or ``base_url`` is unset.
"""
if not settings.api_key or not settings.api_key.get_secret_value():
raise ValueError(
"LLM api_key is not configured "
"(set EVEROS_LLM__API_KEY or [llm] api_key in user toml)"
)
raise ValueError(missing_config_error("LLM api_key", "llm"))
if not settings.base_url:
raise ValueError(
"LLM base_url is not configured "
"(set EVEROS_LLM__BASE_URL or [llm] base_url in user toml)"
)
raise ValueError(missing_config_error("LLM base_url", "llm"))
return OpenAIProvider(
model=settings.model,
api_key=settings.api_key.get_secret_value(),
@@ -0,0 +1,27 @@
"""Multimodal LLM capability — optional vision/audio support for parsing.
Public surface:
- :class:`MultimodalLLMCapability` — soft-dependency wrapper around an
optional multimodal LLMClient (``available`` / ``require``; no soft-degrade
accessor — multimodal parsing is entirely optional).
- :func:`get_multimodal_llm_capability` — process-wide lazy singleton accessor
for :class:`MultimodalLLMCapability`.
External usage::
from everos.component.multimodal import get_multimodal_llm_capability
cap = get_multimodal_llm_capability()
if cap.available:
client = cap.require()
"""
from __future__ import annotations
from .accessor import get_multimodal_llm_capability as get_multimodal_llm_capability
from .capability import MultimodalLLMCapability as MultimodalLLMCapability
__all__ = [
"MultimodalLLMCapability",
"get_multimodal_llm_capability",
]
@@ -0,0 +1,55 @@
"""Process-wide multimodal LLM capability accessor.
Lazy singleton — first call reads settings and attempts to build a multimodal
LLM client. Unlike :func:`everos.component.llm.client.get_multimodal_llm_client`
(which raises when misconfigured), this wraps the outcome in a capability that
reports ``available=False`` when the client cannot be built.
Subsequent calls return the cached instance.
"""
from __future__ import annotations
from everos.component.llm.client import (
LLMNotConfiguredError,
get_multimodal_llm_client,
)
from everos.core.observability.logging import get_logger
from .capability import MultimodalLLMCapability
logger = get_logger(__name__)
_capability: MultimodalLLMCapability | None = None
def get_multimodal_llm_capability() -> MultimodalLLMCapability:
"""Return the process-wide :class:`MultimodalLLMCapability`. Never raises.
Lazy singleton: the first call attempts to build a multimodal client from
current settings — ``available`` is ``False`` when the client cannot be
built (e.g. missing model/base_url/api_key). Use this from upload
endpoints, the health endpoint, the startup banner, and any other caller
that needs to check "is multimodal parsing available?" without a hard
dependency.
Build failures (``ValueError`` from the factory / ``LLMNotConfiguredError``
from missing settings) are logged as
``multimodal_llm_capability_build_failed`` so an operator can distinguish
"not configured" from "misconfigured" — mirrors the equivalent warning on
the embedding and rerank accessors so all three optional providers surface
the same signal.
"""
global _capability
if _capability is not None:
return _capability
try:
provider = get_multimodal_llm_client()
except (ValueError, LLMNotConfiguredError) as exc:
logger.warning(
"multimodal_llm_capability_build_failed",
reason=str(exc),
)
provider = None
_capability = MultimodalLLMCapability(provider=provider)
return _capability
@@ -0,0 +1,40 @@
"""MultimodalLLMCapability — soft-dependency wrapper for multimodal LLM client.
Parallel structure to :class:`everos.component.rerank.RerankCapability`, but
for the multimodal LLM used by ``everalgo.parser``. A caller either requires
multimodal (raising :class:`ProviderNotConfiguredError` -> HTTP 422 when
missing) or checks ``available`` to skip parsing when unavailable.
"""
from __future__ import annotations
from dataclasses import dataclass
from typing import TYPE_CHECKING
from everos.core.errors import ProviderNotConfiguredError
if TYPE_CHECKING:
from everalgo.llm.protocols import LLMClient
@dataclass(frozen=True)
class MultimodalLLMCapability:
"""Wraps an optional multimodal LLMClient with a hard-require API."""
provider: LLMClient | None
@property
def available(self) -> bool:
"""True iff a provider was successfully constructed."""
return self.provider is not None
def require(self) -> LLMClient:
"""Return the provider, or raise :class:`ProviderNotConfiguredError`.
Caller supplies its own ``feature``/``alternative_hint`` context by
catching and re-raising if needed; this base call only identifies
the missing provider as ``"multimodal_llm"``.
"""
if self.provider is None:
raise ProviderNotConfiguredError(provider="multimodal_llm")
return self.provider
+16 -1
View File
@@ -6,6 +6,7 @@ All imports are deferred so this module is safe to import without the extra.
from __future__ import annotations
from functools import lru_cache
from typing import TYPE_CHECKING
from everos.core.errors import MultimodalNotEnabledError, UnsupportedModalityError
@@ -14,8 +15,22 @@ if TYPE_CHECKING:
from everalgo.types import ParsedContent, RawFile
@lru_cache(maxsize=1)
def parser_available() -> bool:
"""Whether ``everalgo.parser`` is importable."""
"""Whether ``everalgo.parser`` is importable.
Memoised: the underlying ``import everalgo.parser`` pulls in heavy
PDF/Office dependencies (``pypdf``, ``python-docx``, ...) that would
block the event loop for hundreds of ms — unacceptable inside
``async def health()`` on a liveness probe. First call at startup
(via :class:`ParserLifespanProvider`) pays the import cost off the
request path; subsequent calls hit the cache instantly.
The cache is process-wide. Tests that patch ``sys.modules`` to
swap the ``everalgo.parser`` module in or out must invoke
:meth:`parser_available.cache_clear` between assertions to avoid
cross-test contamination.
"""
try:
import everalgo.parser # noqa: F401
except ImportError:
+9
View File
@@ -5,6 +5,9 @@ Public surface:
- :class:`RerankProvider` — Protocol every provider satisfies.
- :class:`RerankResult` / :class:`RerankServiceError` — value type + error.
- :class:`RerankError` — backward-compat alias for :class:`RerankServiceError`.
- :class:`RerankCapability` — soft-dependency wrapper around an optional
:class:`RerankProvider` (``available`` / ``require``; no soft-degrade
accessor — rerank has no write path to degrade).
- :class:`DeepInfraRerankProvider` — DeepInfra inference-API rerank.
- :class:`VllmRerankProvider` — OpenAI-compat ``/v1/rerank`` (vLLM,
self-hosted, other compatible servers).
@@ -12,6 +15,8 @@ Public surface:
``gte-rerank-v2`` native text-rerank endpoint.
- :func:`build_rerank_provider` — settings-driven factory that picks
the concrete provider via ``settings.rerank.provider``.
- :func:`get_rerank_capability` — process-wide lazy singleton accessor
for :class:`RerankCapability`.
External usage::
@@ -22,6 +27,8 @@ External usage::
from everos.core.errors import RerankServiceError as RerankServiceError
from .accessor import get_rerank_capability as get_rerank_capability
from .capability import RerankCapability as RerankCapability
from .dashscope_provider import DashScopeRerankProvider as DashScopeRerankProvider
from .deepinfra_provider import DeepInfraRerankProvider as DeepInfraRerankProvider
from .factory import build_rerank_provider as build_rerank_provider
@@ -33,10 +40,12 @@ from .vllm_provider import VllmRerankProvider as VllmRerankProvider
__all__ = [
"DashScopeRerankProvider",
"DeepInfraRerankProvider",
"RerankCapability",
"RerankError",
"RerankProvider",
"RerankResult",
"RerankServiceError",
"VllmRerankProvider",
"build_rerank_provider",
"get_rerank_capability",
]
+57
View File
@@ -0,0 +1,57 @@
"""Process-wide rerank capability accessor.
Lazy singleton mirror of
:func:`everos.component.embedding.accessor.get_embedding_capability`: first
call reads settings, attempts to build a rerank provider, and wraps the
outcome (provider or ``None``) in a :class:`RerankCapability`. Subsequent
calls return the cached instance.
Rerank is a Tier-3 optional provider — call sites either go through
:func:`get_rerank_capability` and call ``.require()`` when rerank is
mandatory, or check ``.available`` and skip reranking when it is not.
"""
from __future__ import annotations
from everos.config import load_settings
from everos.core.observability.logging import get_logger
from .capability import RerankCapability
from .factory import build_rerank_provider
logger = get_logger(__name__)
_capability: RerankCapability | None = None
def get_rerank_capability() -> RerankCapability:
"""Return the process-wide :class:`RerankCapability`. Never raises.
Lazy singleton: the first call builds and caches a capability from
current settings — ``available`` is ``False`` when the provider cannot
be built (e.g. missing model/base_url/api_key, unsupported provider
name, malformed URL, …). Use this from search strategies, the health
endpoint, the startup banner, and any other caller that needs to
check "is rerank available?" without a hard dependency.
Configuration failures (:class:`ValueError` from
:func:`build_rerank_provider`) are logged at ``warning`` level: the
downstream :class:`ProviderNotConfiguredError` message maps both
"user hasn't configured it" and "user configured it wrong" onto the
same HTTP 422, so the log line is the only place an operator can
tell those two states apart.
"""
global _capability
if _capability is not None:
return _capability
try:
provider = build_rerank_provider(load_settings().rerank)
except ValueError as exc:
logger.warning(
"rerank_capability_build_failed",
reason=str(exc),
)
provider = None
_capability = RerankCapability(provider=provider)
return _capability
+39
View File
@@ -0,0 +1,39 @@
"""RerankCapability — soft-dependency wrapper for RerankProvider.
Parallel structure to :class:`everos.component.embedding.EmbeddingCapability`,
but without a soft-degrade accessor: rerank is a query-time enhancement,
not a write path. A caller either hard-requires rerank (raising
:class:`ProviderNotConfiguredError` -> HTTP 422 when missing) or chooses to
skip reranking entirely — there is no equivalent of ``embed_or_none``.
"""
from __future__ import annotations
from dataclasses import dataclass
from everos.core.errors import ProviderNotConfiguredError
from .protocol import RerankProvider
@dataclass(frozen=True)
class RerankCapability:
"""Wraps an optional RerankProvider with a hard-require API."""
provider: RerankProvider | None
@property
def available(self) -> bool:
"""True iff a provider was successfully constructed."""
return self.provider is not None
def require(self) -> RerankProvider:
"""Return the provider, or raise :class:`ProviderNotConfiguredError`.
Caller supplies its own ``feature``/``alternative_hint`` context by
catching and re-raising if needed; this base call only identifies
the missing provider as ``"rerank"``.
"""
if self.provider is None:
raise ProviderNotConfiguredError(provider="rerank")
return self.provider
+5 -15
View File
@@ -13,6 +13,7 @@ Adding a new provider = one match arm here + one new file under
from __future__ import annotations
from everos.component.utils.config_hints import missing_config_error
from everos.config import RerankSettings
from .dashscope_provider import DashScopeRerankProvider
@@ -38,22 +39,14 @@ def build_rerank_provider(settings: RerankSettings) -> RerankProvider:
string) for ``vllm`` self-hosted endpoints.
"""
if not settings.model:
raise ValueError(
"Rerank model is not configured "
"(set EVEROS_RERANK__MODEL or [rerank] model in user toml)"
)
raise ValueError(missing_config_error("Rerank model", "rerank"))
if not settings.base_url:
raise ValueError(
"Rerank base_url is not configured (set EVEROS_RERANK__BASE_URL)"
)
raise ValueError(missing_config_error("Rerank base_url", "rerank"))
api_key = settings.api_key.get_secret_value() if settings.api_key else ""
if settings.provider == "deepinfra":
if not api_key:
raise ValueError(
"DeepInfra rerank api_key is not configured "
"(set EVEROS_RERANK__API_KEY)"
)
raise ValueError(missing_config_error("DeepInfra rerank api_key", "rerank"))
return DeepInfraRerankProvider(
model=settings.model,
api_key=api_key,
@@ -75,10 +68,7 @@ def build_rerank_provider(settings: RerankSettings) -> RerankProvider:
)
if settings.provider == "dashscope":
if not api_key:
raise ValueError(
"DashScope rerank api_key is not configured "
"(set EVEROS_RERANK__API_KEY)"
)
raise ValueError(missing_config_error("DashScope rerank api_key", "rerank"))
return DashScopeRerankProvider(
model=settings.model,
api_key=api_key,
@@ -0,0 +1,30 @@
"""User-facing error message helpers for missing provider configuration.
Guidance points users at the ``<root>/everos.toml`` file, not at
environment variables. Env vars still work as an override mechanism
(see ``pydantic-settings`` precedence in ``config/settings.py``) but
are not surfaced in onboarding-facing text.
"""
from __future__ import annotations
from everos.core.persistence import MemoryRoot
def missing_config_error(field_label: str, toml_section: str) -> str:
"""Return a uniform error message for a missing config field.
Args:
field_label: Human-readable label (e.g. ``"LLM api_key"``).
toml_section: TOML section name without brackets (e.g. ``"llm"``).
Returns:
A single-line message including the resolved memory-root path
and a hint to run ``everos init``. Never mentions env vars.
"""
root = MemoryRoot.resolve().root
return (
f"{field_label} is not configured. "
f"Edit {root}/everos.toml (run `everos init` to scaffold), "
f"section [{toml_section}]."
)
+51
View File
@@ -36,6 +36,7 @@ class ErrorCode(StrEnum):
CAPABILITY_UNAVAILABLE = "CAPABILITY_UNAVAILABLE"
CONFIGURATION_ERROR = "CONFIGURATION_ERROR"
INTERNAL_ERROR = "INTERNAL_ERROR"
PROVIDER_NOT_CONFIGURED = "PROVIDER_NOT_CONFIGURED"
# ---------------------------------------------------------------------------
@@ -175,6 +176,56 @@ class ConfigurationError(AppError):
"""
# TOML section name for each provider kind, keyed by the `provider` argument
# passed to `ProviderNotConfiguredError`.
_PROVIDER_SECTIONS: dict[str, str] = {
"llm": "llm",
"embedding": "embedding",
"rerank": "rerank",
"multimodal_llm": "multimodal",
}
class ProviderNotConfiguredError(ConfigurationError):
"""Raised when a runtime path requires a provider that is not configured.
Maps to HTTP 422 via the FastAPI exception handler; the message is
directly user-facing and points at the toml file for remediation.
Args:
provider: Which provider is missing. One of ``"llm"``,
``"embedding"``, ``"rerank"``, ``"multimodal_llm"``.
feature: Optional user-facing feature that required this
provider (e.g. ``"knowledge"``, ``"agent_hybrid"``).
alternative_hint: Optional alternative workaround the user can
take without configuring the missing provider (e.g. flip
``enable_llm_rerank=true`` to use the LLM lane).
"""
def __init__(
self,
provider: str,
feature: str | None = None,
alternative_hint: str | None = None,
) -> None:
# Lazy import: `config_hints` -> `core.persistence` -> `core.persistence
# .markdown.writer` imports `PathTraversalError` from this module, so a
# module-level import here would be circular.
from everos.component.utils.config_hints import missing_config_error
self.provider = provider
self.feature = feature
self.alternative_hint = alternative_hint
section = _PROVIDER_SECTIONS.get(provider, provider)
field_label = f"Provider '{provider}'"
if feature:
field_label += f" (required by {feature})"
message = missing_config_error(field_label, section)
if alternative_hint:
message += f" Alternative: {alternative_hint}"
super().__init__(message)
# ---------------------------------------------------------------------------
# Backward compatibility aliases
# ---------------------------------------------------------------------------
+22 -2
View File
@@ -23,11 +23,12 @@ The cascade queue, LSN watermark, and change audit all live in
The default location and tunables come from :class:`everos.config.Settings`
(loaded from ``config/default.toml`` + ``EVEROS_*`` environment variables);
:meth:`MemoryRoot.default` resolves the configured path.
:meth:`MemoryRoot.resolve` resolves the configured path.
"""
from __future__ import annotations
import warnings
from dataclasses import dataclass
from pathlib import Path
@@ -92,7 +93,7 @@ class MemoryRoot:
object.__setattr__(self, "root", resolved)
@classmethod
def default(cls, *, explicit_root: str | None = None) -> MemoryRoot:
def resolve(cls, *, explicit_root: str | None = None) -> MemoryRoot:
"""Return the memory-root resolved from CLI / env / default.
Resolution: ``explicit_root`` > ``EVEROS_ROOT`` env > ``~/.everos``.
@@ -105,6 +106,25 @@ class MemoryRoot:
return cls(resolve_root(explicit_root))
@classmethod
def default(cls, *, explicit_root: str | None = None) -> MemoryRoot:
"""Deprecated alias for :meth:`resolve`. Removed in a future major.
Kept as a backward-compatibility shim for v1.2.0 callers who used
the old ``default()`` name (renamed to ``resolve()`` in this
release). Emits :class:`DeprecationWarning` on every call so
existing callers surface in test / CI runs; the constructor
semantics are identical.
"""
warnings.warn(
"MemoryRoot.default() is deprecated and renamed to "
"MemoryRoot.resolve(); the alias will be removed in a "
"future major release. Update call sites.",
DeprecationWarning,
stacklevel=2,
)
return cls.resolve(explicit_root=explicit_root)
# ── User-visible (partitioned by app / project) ──────────────────────────
#
# These take ``(app_id, project_id)`` because the scope dirs hang off the
+5 -3
View File
@@ -34,6 +34,7 @@ from .lifespans import (
LanceDBLifespanProvider,
LLMLifespanProvider,
OmeLifespanProvider,
ParserLifespanProvider,
SqliteLifespanProvider,
)
from .routes import (
@@ -71,9 +72,9 @@ def create_app(
cors_allow_headers: Allowed CORS headers (default: ``["*"]``).
lifespan_providers: Optional list of LifespanProvider; defaults to
``[TracingLifespanProvider(), MetricsLifespanProvider(),
LLMLifespanProvider(), SqliteLifespanProvider(),
LanceDBLifespanProvider(), CascadeLifespanProvider(),
OmeLifespanProvider()]``.
LLMLifespanProvider(), ParserLifespanProvider(),
SqliteLifespanProvider(), LanceDBLifespanProvider(),
CascadeLifespanProvider(), OmeLifespanProvider()]``.
Returns:
FastAPI: Configured application instance.
@@ -85,6 +86,7 @@ def create_app(
TracingLifespanProvider(),
MetricsLifespanProvider(),
LLMLifespanProvider(),
ParserLifespanProvider(),
SqliteLifespanProvider(),
LanceDBLifespanProvider(),
CascadeLifespanProvider(),
@@ -43,6 +43,7 @@ from everos.core.errors import (
InvalidInputError,
NotFoundError,
PathTraversalError,
ProviderNotConfiguredError,
UnsupportedModalityError,
)
from everos.core.observability.logging import get_logger
@@ -241,6 +242,31 @@ async def configuration_handler(
)
async def provider_not_configured_handler(
request: Request,
exc: ProviderNotConfiguredError,
) -> JSONResponse:
"""ProviderNotConfiguredError -> 422 (client-actionable, unlike its parent).
Unlike a generic ``ConfigurationError`` (500 -- an operator bug), a
missing provider is something the caller can fix by editing
``everos.toml``, so this more-specific subclass gets its own 422
mapping instead of falling through to ``configuration_handler``.
"""
logger.warning(
"provider_not_configured_error",
path=str(request.url.path),
provider=exc.provider,
feature=exc.feature,
)
return _error_response(
request,
HTTP_422_UNPROCESSABLE_CONTENT,
ErrorCode.PROVIDER_NOT_CONFIGURED,
str(exc),
)
# ---------------------------------------------------------------------------
# Pydantic / FastAPI built-in exceptions
# ---------------------------------------------------------------------------
@@ -350,7 +376,10 @@ def register_handlers(app: FastAPI) -> None:
app.add_exception_handler(InfrastructureError, infrastructure_handler)
# Capability errors (permanent, not retryable)
app.add_exception_handler(CapabilityError, capability_handler)
# Configuration errors
# Configuration errors (specific before parent)
app.add_exception_handler(
ProviderNotConfiguredError, provider_not_configured_handler
)
app.add_exception_handler(ConfigurationError, configuration_handler)
# FastAPI built-in exceptions
app.add_exception_handler(HTTPException, http_exception_handler)
@@ -13,6 +13,7 @@ External usage::
from everos.entrypoints.api.lifespans import (
LLMLifespanProvider,
ParserLifespanProvider,
SqliteLifespanProvider,
LanceDBLifespanProvider,
CascadeLifespanProvider,
@@ -24,6 +25,7 @@ from .cascade import CascadeLifespanProvider as CascadeLifespanProvider
from .lancedb import LanceDBLifespanProvider as LanceDBLifespanProvider
from .llm import LLMLifespanProvider as LLMLifespanProvider
from .ome import OmeLifespanProvider as OmeLifespanProvider
from .parser import ParserLifespanProvider as ParserLifespanProvider
from .sqlite import SqliteLifespanProvider as SqliteLifespanProvider
__all__ = [
@@ -31,5 +33,6 @@ __all__ = [
"LLMLifespanProvider",
"LanceDBLifespanProvider",
"OmeLifespanProvider",
"ParserLifespanProvider",
"SqliteLifespanProvider",
]
@@ -4,9 +4,12 @@ Ordered after SqliteLifespan + LanceDBLifespan: the orchestrator
depends on both stores being ready before its watcher / scanner /
worker tasks can take the first row.
Construction reads the live :class:`Settings` to build the embedding +
tokenizer providers. If either is misconfigured the lifespan fails
fast — the daemon would be useless without them anyway.
Construction reads the live :class:`Settings` to build the tokenizer
provider, which fails fast if misconfigured. Embedding is a soft
dependency: startup warms the process-wide
:class:`~everos.component.embedding.EmbeddingCapability` singleton via
:func:`get_embedding_capability`, which never raises — the daemon
runs in keyword-only mode when embedding is unavailable.
"""
from __future__ import annotations
@@ -15,9 +18,8 @@ from typing import Any
from fastapi import FastAPI
from everos.component.embedding import build_embedding_provider
from everos.component.embedding import get_embedding_capability
from everos.component.tokenizer import build_tokenizer
from everos.config import load_settings
from everos.core.lifespan import LifespanProvider
from everos.core.observability.logging import get_logger
from everos.core.persistence import MemoryRoot
@@ -34,15 +36,22 @@ class CascadeLifespanProvider(LifespanProvider):
self._orchestrator: CascadeOrchestrator | None = None
async def startup(self, app: FastAPI) -> Any:
settings = load_settings()
memory_root = MemoryRoot.default()
memory_root = MemoryRoot.resolve()
memory_root.ensure()
embedder = build_embedding_provider(settings.embedding)
tokenizer = build_tokenizer()
capability = get_embedding_capability()
if capability.available:
logger.info("cascade_startup_embed_available")
else:
logger.info(
"cascade_startup_embed_unavailable",
reason="embedding not configured; keyword-only mode",
)
self._orchestrator = CascadeOrchestrator(
memory_root=memory_root,
embedder=embedder,
tokenizer=tokenizer,
)
await self._orchestrator.start()
@@ -5,9 +5,22 @@ Startup:
Importing :mod:`everos.infra.persistence.lancedb` also triggers the
side-effect import of ``tables`` so business schemas are loaded
(future: preflight registration).
Log hint if unbackfilled (vector IS NULL) rows exist.
Shutdown:
Close the connection (also clears the table cache).
Unbackfilled hint:
The informational "you have unbackfilled memory rows" banner runs
an unconditional ``count_rows(filter='vector IS NULL')`` against
every business table on startup. An earlier "marker + limit(1)
probe" amortisation was reverted (round-3 finding #3): the vector
column has no scalar index, so ``limit(1)`` on ``vector IS NULL``
costs the same full scan as ``count_rows``. On a clean state the
probe scanned the entire empty tail before returning, matching the
cost it was meant to avoid; on a dirty state the probe hit early
and then the full ``count_rows`` ran anyway, doubling the scan.
The marker's ``last_seen_count`` field was written but never read.
"""
from __future__ import annotations
@@ -19,19 +32,60 @@ from fastapi import FastAPI
from everos.core.lifespan import LifespanProvider
from everos.core.observability.logging import get_logger
from everos.infra.persistence.lancedb import (
BUSINESS_SCHEMAS_WITH_VECTOR,
dispose_connection,
ensure_business_indexes,
get_connection,
get_table,
verify_business_schemas,
)
logger = get_logger(__name__)
async def _log_unbackfilled_hint() -> None:
"""Warn at startup if there are unbackfilled memory rows.
Runs an unconditional ``count_rows(filter="vector IS NULL")`` per
business table. The vector column has no scalar index, so
``count_rows`` and any ``limit(1)`` probe cost the same full scan
— a previous "marker + probe" optimisation (removed here) turned
out to be net-zero on clean state and net-negative on dirty state
(probe hits early, then the full count runs anyway = twice the
scan).
Per-table failures are logged as warnings and don't interrupt
startup.
"""
total_null = 0
for schema in BUSINESS_SCHEMAS_WITH_VECTOR:
try:
table = await get_table(schema.TABLE_NAME, schema)
count = await table.count_rows(filter="vector IS NULL")
except Exception as exc:
logger.warning(
"unbackfilled_check_failed",
schema=schema.__name__,
error=repr(exc),
)
continue
if count > 0:
total_null += count
if total_null > 0:
banner_logger = get_logger("everos.cli.server")
banner_logger.warning(
"unbackfilled_memory_rows",
count=total_null,
hint="Run `everos cascade backfill` to include them in "
"vector/hybrid search (optional).",
)
class LanceDBLifespanProvider(LifespanProvider):
"""Manage the LanceDB connection + table cache for the app lifecycle.
Startup runs three steps:
Startup runs four steps:
1. ``get_connection`` — lazy-open the async connection.
2. ``verify_business_schemas`` — fail loud if an on-disk table's
@@ -39,6 +93,7 @@ class LanceDBLifespanProvider(LifespanProvider):
online migration; cascade is rebuildable from md so the recovery
is documented as ``rm -rf ~/.everos/.index/lancedb``.
3. ``ensure_business_indexes`` — idempotent FTS index creation.
4. ``_log_unbackfilled_hint`` — warn if unbackfilled rows exist.
"""
def __init__(self, order: int = 11) -> None:
@@ -48,6 +103,7 @@ class LanceDBLifespanProvider(LifespanProvider):
conn = await get_connection()
await verify_business_schemas()
await ensure_business_indexes()
await _log_unbackfilled_hint()
logger.info("lancedb_ready", uri=conn.uri)
return conn
@@ -0,0 +1,60 @@
"""Parser lifespan provider — warms the optional ``everalgo.parser`` import.
``everalgo.parser`` is an optional dependency (``everos[multimodal]``).
When installed it pulls in ``pypdf`` / ``python-docx`` / ... on first
import — hundreds of milliseconds to seconds of blocking work. That
cost is fine at startup but unacceptable on the request path, where
:func:`everos.entrypoints.api.routes.health.health` calls
:func:`parser_available` from inside ``async def`` — an event-loop
block there stalls liveness probes and any in-flight requests behind
it.
This provider resolves :func:`parser_available` once at startup,
priming both Python's ``sys.modules`` cache and the
:func:`functools.lru_cache` wrapping ``parser_available`` itself. If
the extra is not installed, the import fails, the cache stores
``False``, and every subsequent probe is a hot dict lookup.
Ordered between :class:`LLMLifespanProvider` (``order=8``, hard
Tier-1 requirement) and :class:`SqliteLifespanProvider` (``order=10``)
— the warm is best-effort chassis hygiene that must not delay the
storage stack coming up.
"""
from __future__ import annotations
from typing import Any
from fastapi import FastAPI
from everos.component.parser import parser_available
from everos.core.lifespan import LifespanProvider
from everos.core.observability.logging import get_logger
logger = get_logger(__name__)
class ParserLifespanProvider(LifespanProvider):
"""Warm the ``everalgo.parser`` import once at startup.
Never fails startup: when the optional extra is not installed,
:func:`parser_available` returns ``False`` and this provider logs
the fact at INFO so operators see a single line rather than the
hidden per-request block that the pre-warm eliminates.
"""
def __init__(self, order: int = 9) -> None:
# Slot picked to sit between LLM (order=8, hard requirement) and
# sqlite (order=10) — verified against every existing provider's
# `order=` default (see PR #361 review notes on M-c).
super().__init__(name="parser", order=order)
async def startup(self, app: FastAPI) -> Any:
available = parser_available()
logger.info("parser_lifespan_ready", available=available)
return None
async def shutdown(self, app: FastAPI) -> None:
# Nothing to tear down — the import is process-scoped and lives
# in `sys.modules` for the process lifetime.
return None
+64 -4
View File
@@ -3,11 +3,71 @@
from __future__ import annotations
from fastapi import APIRouter
from pydantic import BaseModel
from everos import __version__
from everos.component.capabilities import compute_disabled_features
from everos.component.embedding import get_embedding_capability
from everos.component.multimodal import get_multimodal_llm_capability
from everos.component.parser import parser_available
from everos.component.rerank import get_rerank_capability
router = APIRouter(tags=["health"])
@router.get("/health")
async def health() -> dict[str, str]:
"""Liveness probe — returns ``{"status": "ok"}`` with HTTP 200."""
return {"status": "ok"}
class HealthCapabilities(BaseModel):
"""Availability flags for the five capability probes.
Field order matches the health-endpoint payload contract; clients
key off these names to decide whether to expose optional features.
"""
llm: bool
embed: bool
rerank: bool
multimodal_llm: bool
parser: bool
class HealthResponse(BaseModel):
"""Response schema for ``GET /health``.
Declared as a Pydantic model (not ``dict``) so the generated
OpenAPI schema carries the full field shape — ``capabilities`` and
``disabled_features`` are typed. A bare ``-> dict`` return type
degrades the OpenAPI response to ``additionalProperties: true``,
which robs clients (and codegen) of any structure to lean on.
"""
status: str
version: str
capabilities: HealthCapabilities
disabled_features: list[str]
@router.get("/health", response_model=HealthResponse)
async def health() -> HealthResponse:
"""Liveness probe with capabilities and disabled features."""
# ``llm`` is hardcoded ``True`` — kept for symmetry with the caps
# dict rather than probed live. Rationale: LLM is a Tier-1 hard
# requirement enforced at startup by ``LLMLifespanProvider``
# (lifespans/llm.py), which eagerly calls ``get_llm_client()`` and
# raises ``LLMNotConfiguredError`` if credentials are missing —
# FastAPI startup then fails, so ``/health`` is unreachable
# without a working LLM. Any code path that reaches this handler
# therefore has ``get_llm_client()`` returning a real client. If
# the LLM capability is ever downgraded to soft (like embed /
# rerank), swap this literal for a real probe.
caps = HealthCapabilities(
llm=True,
embed=get_embedding_capability().available,
rerank=get_rerank_capability().available,
multimodal_llm=get_multimodal_llm_capability().available,
parser=parser_available(),
)
return HealthResponse(
status="ok",
version=__version__,
capabilities=caps,
disabled_features=compute_disabled_features(caps.model_dump()),
)
+137 -17
View File
@@ -25,15 +25,18 @@ if TYPE_CHECKING:
from everos.service.knowledge import KnowledgeExtractor
from everalgo.types import ParsedContent
from fastapi import APIRouter, Path, Query, Request, Response, UploadFile
from fastapi import APIRouter, Depends, Path, Query, Request, Response, UploadFile
from fastapi.params import Form
from pydantic import BaseModel, Field
from everos.component.embedding import get_embedding_capability
from everos.component.llm import get_llm_client
from everos.component.rerank import get_rerank_capability
from everos.component.utils.datetime import to_display_tz
from everos.config import load_settings
from everos.core.errors import (
InvalidInputError,
ProviderNotConfiguredError,
UnsupportedModalityError,
)
from everos.core.persistence import MemoryRoot
@@ -59,6 +62,61 @@ from everos.service import (
# a shared module would be cleaner but is out of scope for this PR.
from .memorize import PathSafeId, SuccessEnvelope
_KNOWLEDGE_FEATURE = "knowledge"
def _require_knowledge_capabilities() -> None:
"""Per-endpoint gate: knowledge writes/search require embed + rerank.
**Why the gate lives at the HTTP layer, not at cascade registry**:
cascade handlers register **unconditionally** now (see
``memory/cascade/registry.py:177-194`` — the earlier atomic-pair
gate that lived there was removed because it broke the delete
path: a Tier-3 → Tier-2/1 downgrade would strand existing knowledge
documents with no handler to process their deletion). Handlers
body-guard the embed/rerank branches internally instead, so the
delete path stays reachable across tier changes.
Because the cascade layer no longer refuses the write, the HTTP
layer is now the *only* place that enforces "knowledge features
require Tier 3". Without this gate a Tier-1/2 upload would:
- accept the multipart request → md write succeeds
- enqueue for cascade → cascade handler body-guards on
``get_embedding_capability().available`` and no-ops the
index write
- client sees 201 but the doc is permanently keyword-only /
unsearchable
Returning 422 up front is the honest answer.
**Attached per-endpoint (not router-wide)** so read / list /
delete / metadata-patch routes stay reachable after a Tier-3 →
Tier-2/1 downgrade: users can still inspect and clean up their
existing docs (rename, recategorize, delete) even when the
providers that would embed or rerank new content are no longer
configured. Title / category patches only rewrite md frontmatter
(and move the doc directory when category changes) — no embed or
rerank code runs on that path.
Checks both capabilities (not just embedding) up front so a client
missing only rerank gets a rerank-specific message rather than
passing this gate and failing later inside search.
"""
if not get_embedding_capability().available:
raise ProviderNotConfiguredError(
provider="embedding",
feature=_KNOWLEDGE_FEATURE,
)
if not get_rerank_capability().available:
raise ProviderNotConfiguredError(
provider="rerank",
feature=_KNOWLEDGE_FEATURE,
)
# Router prefix is /knowledge; app.py mounts it under both /api/v1 and
# /api/v2 (see create_app — v1 retained as a permanent alias).
router = APIRouter(prefix="/knowledge", tags=["knowledge"])
@@ -276,6 +334,61 @@ def _reject_oversized_upload(file: UploadFile) -> None:
raise InvalidInputError(f"Uploaded file exceeds the {limit_mib:.1f} MiB limit.")
_PLAIN_TEXT_EXTENSIONS: frozenset[str] = frozenset(
{"md", "txt", "rst", "markdown", "text"}
)
# Explicit mime allowlist. ``text/*`` prefix matching would let
# ``text/html`` (and any future ``text/xml`` etc.) bypass the parser and
# feed raw markup — script tags, HTML comments, style blocks — straight
# into the knowledge-extraction LLM. everalgo's HTML parser runs
# ``clean_html_for_llm`` (strips ``<script>/<style>/<nav>/<iframe>`` +
# comments) and caps output at 1 MiB — semantics we must NOT skip.
_PLAIN_TEXT_MIMES: frozenset[str] = frozenset(
{"text/plain", "text/markdown", "text/x-rst", "text/x-markdown"}
)
def _looks_like_utf8_text(file: UploadFile) -> bool:
"""Decide whether to skip the parser and go straight to UTF-8 decode.
A file is treated as plain UTF-8 text when its ``content_type`` is on
``_PLAIN_TEXT_MIMES`` (browsers set ``text/markdown`` / ``text/plain``
for ``.md`` / ``.txt`` uploads), or when the mime is missing /
``application/octet-stream`` (typical for ``curl -F file=@x.md``
without an explicit ``type=`` hint) AND the filename extension is on
the extension allowlist.
``text/html`` is deliberately EXCLUDED — HTML uploads must go through
the parser so ``clean_html_for_llm`` can strip active markup and cap
the payload; the ``text/*`` prefix match this file used to carry
would have let raw HTML (including ``<script>`` bodies and
``<!-- prompt injection -->`` comments) reach the LLM as-is.
"""
mime = (file.content_type or "").lower()
if mime in _PLAIN_TEXT_MIMES:
return True
if mime and mime != "application/octet-stream":
return False
if file.filename and "." in file.filename:
return file.filename.rsplit(".", 1)[-1].lower() in _PLAIN_TEXT_EXTENSIONS
return False
def _decode_as_utf8(raw_bytes: bytes) -> ParsedContent:
"""Decode ``raw_bytes`` as UTF-8 or raise a caller-friendly error."""
try:
text = raw_bytes.decode("utf-8")
except UnicodeDecodeError as exc:
raise UnsupportedModalityError(
"File is not UTF-8 text. "
"Install everos[multimodal] for PDF/HTML/DOCX support."
) from exc
parsed = ParsedContent(text=text)
if not parsed.text or not parsed.text.strip():
raise InvalidInputError("Uploaded file has no valid content.")
return parsed
async def _parse_upload(
file: UploadFile, *, raw_bytes: bytes | None = None
) -> ParsedContent:
@@ -294,6 +407,12 @@ async def _parse_upload(
if raw_bytes is None:
raw_bytes = await file.read()
# Short-circuit plain-text uploads even when everalgo.parser is
# installed — the parser path depends on [multimodal] being
# configured, which markdown/plaintext ingestion doesn't need.
if _looks_like_utf8_text(file):
return _decode_as_utf8(raw_bytes)
if parser_available():
from everalgo.types import RawFile # Deferred: optional dep
@@ -313,17 +432,7 @@ async def _parse_upload(
raise InvalidInputError("Uploaded file has no valid content.")
return parsed
try:
text = raw_bytes.decode("utf-8")
except UnicodeDecodeError as exc:
raise UnsupportedModalityError(
"File is not UTF-8 text. "
"Install everos[multimodal] for PDF/HTML/DOCX support."
) from exc
parsed = ParsedContent(text=text)
if not parsed.text or not parsed.text.strip():
raise InvalidInputError("Uploaded file has no valid content.")
return parsed
return _decode_as_utf8(raw_bytes)
def _map_create_result(result: CreateDocumentResult) -> DocumentCreateResponse:
@@ -432,7 +541,11 @@ def _map_search_result(result: SearchKnowledgeResult) -> KnowledgeSearchResponse
# ── Routes ───────────────────────────────────────────────────────────────────
@router.post("/documents", status_code=201)
@router.post(
"/documents",
status_code=201,
dependencies=[Depends(_require_knowledge_capabilities)],
)
# FastAPI requires flat Form/Query params — ≤5 positional rule exempted.
async def create_document_route(
request: Request,
@@ -450,7 +563,7 @@ async def create_document_route(
parsed = await _parse_upload(file, raw_bytes=file_content)
source_name = file.filename
knowledge_dir = MemoryRoot.default().knowledge_dir(app_id, project_id)
knowledge_dir = MemoryRoot.resolve().knowledge_dir(app_id, project_id)
extractor = _build_extractor()
result = await create_document(
@@ -466,7 +579,11 @@ async def create_document_route(
return SuccessEnvelope(request_id=rid, data=_map_create_result(result))
@router.put("/documents/{doc_id}", status_code=200)
@router.put(
"/documents/{doc_id}",
status_code=200,
dependencies=[Depends(_require_knowledge_capabilities)],
)
# FastAPI requires flat Form/Query params — ≤5 positional rule exempted.
async def replace_document_route(
request: Request,
@@ -484,7 +601,7 @@ async def replace_document_route(
file_content = await file.read()
parsed = await _parse_upload(file, raw_bytes=file_content)
knowledge_dir = MemoryRoot.default().knowledge_dir(app_id, project_id)
knowledge_dir = MemoryRoot.resolve().knowledge_dir(app_id, project_id)
extractor = _build_extractor()
result = await replace_document(
@@ -576,7 +693,10 @@ async def get_topic_route(
return SuccessEnvelope(request_id=rid, data=_map_topic_detail(detail))
@router.post("/search")
@router.post(
"/search",
dependencies=[Depends(_require_knowledge_capabilities)],
)
async def search_knowledge_route(
request: Request,
req: KnowledgeSearchRequest,
+29
View File
@@ -0,0 +1,29 @@
"""Interactive-CLI logging setup.
Interactive commands (``cascade`` / ``init`` / ``config`` / ``demo``)
default to WARNING so lifecycle events — ``sqlite_engine_built``,
``lancedb_connection_opened``, ``lancedb_table_opened`` and friends —
do not leak into the user-facing flow (Phase banners, ``[y/N]``
prompts, echo output). ``--verbose`` / ``-v`` opens INFO.
``server start`` is intentionally unaffected: it calls
:func:`everos.core.observability.logging.configure_logging` directly
so production observability still defaults to INFO. This helper is
only for the interactive path.
"""
from __future__ import annotations
from everos.core.observability.logging import configure_logging
def configure_cli_logging(verbose: bool = False) -> None:
"""Configure structlog + stdlib logging for an interactive CLI command.
Args:
verbose: True to emit INFO-level lifecycle logs; False (default)
keeps stdout limited to WARNING and above so the interactive
flow stays clean.
"""
level = "INFO" if verbose else "WARNING"
configure_logging(level=level)
@@ -0,0 +1,486 @@
"""CLI-side orchestration for ``everos cascade backfill``.
Presentation half of PR #361 review finding M11. The memory-layer phase
runners (``memory/cascade/_backfill.py``) are pure — every user-facing
string, colour, and confirmation prompt lives here, behind the
:class:`~everos.memory.cascade._backfill.BackfillPresenter` Protocol.
:class:`TyperPresenter` is the typer-backed implementation; tests can
substitute any structurally-compatible stub.
:func:`run_backfill` is the top-level entry point the ``cascade``
subcommand imports. It builds a :class:`TyperPresenter`, dispatches to
the three phase runners in order, catches Ctrl-C /
:class:`click.exceptions.Abort` / :class:`asyncio.CancelledError`, and
maps every terminal state to a CLI exit code:
- ``0`` — clean success.
- ``1`` — user declined a confirmation prompt.
- ``2`` — unexpected error mid-phase, or preflight found the required
provider not configured.
- ``3`` — Phase 2/3 short-circuited on the OME jobstore lock (another
``everos`` process is running).
- ``4`` — every phase ran but at least one row failed embedding even
after per-row fallback; automation can requeue.
- ``130`` — Ctrl-C (SIGINT convention: 128 + 2), including Ctrl-C at
the ``[y/N]`` confirmation prompt (typer/click surfaces that as
:class:`click.exceptions.Abort`, a :class:`RuntimeError` subclass).
"""
from __future__ import annotations
import asyncio
import anyio.to_thread
import click
import typer
from everos.core.errors import ProviderNotConfiguredError
from everos.core.observability.logging import get_logger
# Module reference (not name-level imports of the phase runners): tests
# monkeypatch the runners on this module to inject Ctrl-C / boundary
# behaviour, and re-binding names on the source module only affects the
# entrypoints layer if we resolve them through the module attribute at
# call time.
from everos.memory.cascade import _backfill as _phase_runners
from everos.memory.cascade._backfill import (
BackfillPhase,
_BackfillSummary,
_count_failed_rows,
_format_tokens,
)
logger = get_logger(__name__)
PHASES: tuple[BackfillPhase, ...] = (
BackfillPhase(
number=1,
slug="vectors",
title="Phase 1 — make older memories searchable by meaning",
detail=(
"Memories created before you configured embedding can only be "
"found by keyword right now. This step lets them be found by "
"meaning too."
),
),
BackfillPhase(
number=2,
slug="clusters",
title="Phase 2 — group related memories into topics",
detail=(
"Related memories get clustered together so reflection and "
"advanced retrieval can operate on them."
),
),
BackfillPhase(
number=3,
slug="skills",
title="Phase 3 — build the agent skill library",
detail=(
"Distill reusable skills from past agent conversations. Uses "
"your LLM — the most expensive step."
),
),
)
_EXIT_LABELS: dict[int, str] = {
0: "SUCCESS",
1: "ABORTED",
2: "FAILED",
3: "SERVER_RUNNING",
4: "COMPLETED_WITH_FAILURES",
130: "INTERRUPTED",
}
"""Maps :func:`run_backfill`'s exit codes to the summary block's ``Exit:``
label. ``3`` marks a Phase 2/3 short-circuit because another process
holds the OME jobstore lock (typically ``everos server start``). ``4``
marks a partial-success run: every phase ran to completion, but at
least one row failed embedding even after the per-row fallback —
operators consult the ``cascade_backfill_row_embed_failed`` /
``..._row_subject_embed_failed`` log events for the failed row ids.
``130`` is the SIGINT convention (128 + signal number 2)."""
def _confirm(detail: str, *, auto_yes: bool) -> bool:
"""Ask the user to confirm one phase; ``--yes`` skips the prompt.
Kept as a module-level helper (rather than a bound method of
:class:`TyperPresenter`) so tests can monkeypatch it on this module
the same way the pre-M11 code was patched on ``_backfill``. The
presenter's :meth:`~TyperPresenter.confirm` delegates here.
"""
if auto_yes:
return True
return typer.confirm(f"Proceed with: {detail}", default=False)
def _print_phase_header(phase: BackfillPhase) -> None:
typer.secho(phase.title, bold=True)
typer.echo(f" {phase.detail}")
typer.echo("")
def _print_aborted() -> None:
typer.secho("Aborted by user.", fg=typer.colors.RED)
def _print_interrupted() -> None:
"""Print the Ctrl-C resume hint (see :func:`run_backfill`)."""
typer.secho(
"Interrupted — partial progress was written. Resume by running:",
fg=typer.colors.YELLOW,
)
typer.echo(" everos cascade backfill --phase <phase-name> --yes")
typer.echo("where <phase-name> is one of: vectors, clusters, skills, all.")
def _print_capability_missing_hint(message: str) -> None:
"""Emit the friendly toml-hint message shown when a phase's required
provider is not configured.
The formatted :class:`ProviderNotConfiguredError` string is built by
the memory-layer phase runner and passed through the presenter — so
the CLI and API stay in lock-step on the remediation copy without
duplicating it here.
"""
typer.secho(f" {message}", fg=typer.colors.RED)
def _print_server_running_hint() -> None:
"""Emit the friendly error shown when preflight or ``engine.start()``
detects another process holding the OME jobstore lock.
Copy points the user at ``everos server start`` (the usual holder)
and Phase 1 (``--phase=vectors``), which never touches OME and so
stays fully concurrent with a running server — never at
``EVEROS_*`` env vars.
"""
typer.secho(
" Backfill Phase 2/3 needs exclusive access to the OME jobstore,",
fg=typer.colors.RED,
)
typer.secho(
" but another everos process is holding it (typically your running",
fg=typer.colors.RED,
)
typer.secho(
" `everos server start`).",
fg=typer.colors.RED,
)
typer.echo("")
typer.echo(" Stop your everos server first, then re-run backfill.")
typer.echo(" (Phase 1 --phase=vectors already runs concurrently with the server;")
typer.echo(" only Phase 2/3 need offline mode. See spec §10.)")
def _print_vectors_estimate(total_rows: int, total_tokens: int) -> None:
typer.echo(f" memories to process: {total_rows:,}")
typer.echo(f" input tokens: {_format_tokens(total_tokens)}\n")
typer.echo(
" Uses your embedding provider — cost depends on its per-token pricing.\n"
)
def _print_clusters_estimate(episode_count: int, case_count: int) -> None:
typer.echo(f" episodes to cluster: {episode_count:,}")
typer.echo(f" agent cases to cluster: {case_count:,}\n")
typer.echo(
" Uses your embedding provider (and LLM for agent-case merges) — "
"cost depends on provider pricing.\n"
)
def _print_skills_estimate(cases: int, clusters: int) -> None:
typer.echo(f" clusters to extract skills from: {clusters:,}")
typer.echo(f" agent cases to replay: {cases:,}\n")
typer.echo(" Uses your LLM — cost depends on its per-token pricing.\n")
def _print_summary(summary: _BackfillSummary, *, exit_code: int) -> None:
"""Print the consolidated post-run summary block.
Field labels are chosen from what each phase's result dataclass
actually tracks — e.g. Phase 3 reports "agent cases processed"
(``events_emitted``) rather than "clusters processed", since the
distinct cluster count isn't retained on
:class:`~everos.memory.cascade._backfill._SkillPhaseResult`.
On ``exit_code == 4`` (COMPLETED_WITH_FAILURES) also emits a hint
pointing operators at the per-row failure log events so they can
recover the failed row ids.
"""
typer.echo("")
typer.secho("Backfill summary", bold=True)
typer.echo("-" * 44)
if summary.vectors is not None:
r = summary.vectors
typer.echo(
f" Phase 1 (vectors) — {r.rows_processed:,} rows / "
f"{r.rows_failed:,} failed / {_format_tokens(r.tokens_embedded)}"
)
if summary.clusters is not None:
c = summary.clusters
created = c.clusters_after - c.clusters_before
typer.echo(
f" Phase 2 (clusters) — {c.events_emitted:,} events emitted / "
f"{created:,} clusters created"
)
if summary.skills is not None:
s = summary.skills
extracted = s.skills_after - s.skills_before
typer.echo(
f" Phase 3 (skills) — {s.events_emitted:,} agent cases "
f"processed / {extracted:,} skills extracted"
)
typer.echo("-" * 44)
label = _EXIT_LABELS.get(exit_code, "UNKNOWN")
typer.echo(f" Exit: {label} ({exit_code})")
if exit_code == 4:
total_failed = _count_failed_rows(summary)
typer.secho(
f" {total_failed:,} rows failed embedding — see log events "
"`cascade_backfill_row_embed_failed` for row_ids and text prefixes.",
fg=typer.colors.YELLOW,
)
class TyperPresenter:
"""typer-backed :class:`BackfillPresenter` — the CLI's user-I/O seam.
Every method delegates to a module-level ``_print_*`` helper or to
:func:`_confirm`. Kept as a small class (rather than a module of
free functions) so alternative front-ends (e.g. a JSON-emitting
CI-mode presenter) can slot in without touching the memory layer.
"""
def nothing_to_backfill(self, message: str, *, scan_failed: bool = False) -> None:
colour = typer.colors.YELLOW if scan_failed else typer.colors.GREEN
typer.secho(message, fg=colour)
def capability_missing(self, *, provider: str, feature: str, message: str) -> None:
_print_capability_missing_hint(message)
def server_running(self) -> None:
_print_server_running_hint()
def estimate_vectors(self, rows: int, tokens: int) -> None:
_print_vectors_estimate(rows, tokens)
def estimate_clusters(self, episodes: int, cases: int) -> None:
_print_clusters_estimate(episodes, cases)
def estimate_skills(self, cases: int, clusters: int) -> None:
_print_skills_estimate(cases, clusters)
async def confirm(self, prompt: str, *, auto_yes: bool) -> bool:
# ``typer.confirm`` is blocking (reads from stdin); offload so a
# slow / interactive user doesn't hold the event loop. Any
# ``click.exceptions.Abort`` raised on Ctrl-C at the prompt
# propagates to :func:`run_backfill` which catches it and maps
# to exit 130.
return await anyio.to_thread.run_sync(
lambda: _confirm(prompt, auto_yes=auto_yes)
)
def emit_progress(self, done: int, total: int) -> None:
typer.echo(f" emitted {done:,} / {total:,} event(s)")
def row_progress(self, table_name: str, done: int, total: int) -> None:
typer.echo(f" {table_name}: processed {done:,} / {total:,} rows")
def phase_1_complete(self, rows_processed: int, rows_failed: int) -> None:
typer.secho(
f"phase 1 complete — {rows_processed:,} memories now "
"searchable by meaning"
+ (f" ({rows_failed:,} failed)" if rows_failed else ""),
fg=typer.colors.GREEN,
)
def phase_2_complete(self, before: int, after: int, emitted: int) -> None:
typer.secho(
f"phase 2 complete — clusters {before:,} -> {after:,} "
f"({emitted:,} events processed)",
fg=typer.colors.GREEN,
)
def phase_3_complete(
self, skills_before: int, skills_after: int, emitted: int
) -> None:
typer.secho(
f"phase 3 complete — skills {skills_before:,} -> {skills_after:,} "
f"({emitted:,} events processed)",
fg=typer.colors.GREEN,
)
async def run_backfill(*, phase: str, auto_yes: bool) -> int:
"""Run backfill phases interactively; return an exit code.
``phase`` is ``"all"`` or one of :data:`PHASES`'s ``slug`` values.
Exit codes: see :data:`_EXIT_LABELS`.
Every exit path — success, abort, error, or interrupt — ends with a
consolidated summary block (:func:`_print_summary`) covering every
phase that actually ran. The interrupt is only ever caught here, at
the outer loop: a phase body mid-batch must let it propagate rather
than swallow it, so a Ctrl-C during Phase 1 doesn't get silently
absorbed as "phase 1 failed, try phase 2 anyway".
Both ``KeyboardInterrupt`` and ``asyncio.CancelledError`` are caught:
on Python 3.11+, ``asyncio.Runner`` (which backs ``asyncio.run``)
translates a real SIGINT into ``main_task.cancel()``, which raises
``CancelledError`` — not ``KeyboardInterrupt`` — at the currently
suspended ``await``. Handling it here lets the task complete
normally with exit code 130 instead of leaking the cancellation out
to ``asyncio.run`` (which would re-synthesize a bare
``KeyboardInterrupt`` that typer/click turns into ``Abort``, exit 1,
with no resume hint printed). Catching ``CancelledError``
unconditionally is safe here: nothing in this call tree cancels the
task for reasons other than an interrupt.
"""
presenter = TyperPresenter()
selected = [p for p in PHASES if phase == "all" or p.slug == phase]
summary = _BackfillSummary()
current_phase: BackfillPhase | None = None
try:
for p in selected:
current_phase = p
_print_phase_header(p)
if p.slug == "vectors":
result = await _phase_runners._run_phase_vectors(
auto_yes=auto_yes, presenter=presenter
)
summary.vectors = result
if result.blocked_by_capability:
logger.warning(
"cascade_backfill_blocked_by_capability",
phase=p.slug,
provider=result.blocked_by_capability,
)
_print_summary(summary, exit_code=2)
return 2
if result.blocked_by_server:
# Fail fast BEFORE any embed API call — Phase 2 / 3
# already do this preflight; without it here,
# ``--phase all`` against a running server burns
# through Phase 1's real token budget and only
# halts at Phase 2 (see PR #361 review J10).
logger.warning("cascade_backfill_blocked_by_server", phase=p.slug)
_print_summary(summary, exit_code=3)
return 3
if result.aborted:
# Policy (PR #361 round-3 review #11, accepted):
# typing ``n`` at Phase 1's confirmation aborts the
# ENTIRE ``--phase all`` run — not just Phase 1.
# Rationale: ``--phase all`` is opt-in-to-run-
# everything. Explicitly declining Phase 1's cost
# estimate is a decisive "no"; interpreting it as
# "skip Phase 1 but still run Phase 2/3" would spend
# LLM/embed budget on phases the user did not see
# estimates for. Users who want only Phase 2 or 3
# invoke ``--phase clusters`` or ``--phase skills``
# explicitly (and see that phase's own estimate).
logger.info("cascade_backfill_aborted", phase=p.slug)
_print_aborted()
_print_summary(summary, exit_code=1)
return 1
continue
if p.slug == "clusters":
cluster_result = await _phase_runners._run_phase_clusters(
auto_yes=auto_yes, presenter=presenter
)
summary.clusters = cluster_result
if cluster_result.blocked_by_capability:
logger.warning(
"cascade_backfill_blocked_by_capability",
phase=p.slug,
provider=cluster_result.blocked_by_capability,
)
_print_summary(summary, exit_code=2)
return 2
if cluster_result.blocked_by_server:
logger.warning("cascade_backfill_blocked_by_server", phase=p.slug)
_print_summary(summary, exit_code=3)
return 3
if cluster_result.aborted:
logger.info("cascade_backfill_aborted", phase=p.slug)
_print_aborted()
_print_summary(summary, exit_code=1)
return 1
continue
if p.slug == "skills":
skill_result = await _phase_runners._run_phase_skills(
auto_yes=auto_yes, presenter=presenter
)
summary.skills = skill_result
if skill_result.blocked_by_capability:
logger.warning(
"cascade_backfill_blocked_by_capability",
phase=p.slug,
provider=skill_result.blocked_by_capability,
)
_print_summary(summary, exit_code=2)
return 2
if skill_result.blocked_by_server:
logger.warning("cascade_backfill_blocked_by_server", phase=p.slug)
_print_summary(summary, exit_code=3)
return 3
if skill_result.aborted:
logger.info("cascade_backfill_aborted", phase=p.slug)
_print_aborted()
_print_summary(summary, exit_code=1)
return 1
continue
# ``typer.Abort`` fires when ``typer.confirm`` catches KeyboardInterrupt
# or EOF at the y/N prompt and re-raises it as an ``Abort`` (a
# ``RuntimeError`` subclass, NOT a ``KeyboardInterrupt``). Typer 0.15+
# vendored click under ``typer._click`` — ``typer.Abort`` and the
# standalone ``click.exceptions.Abort`` are DISTINCT classes. Catching
# only ``click.exceptions.Abort`` would let the real typer-raised
# abort fall through to the generic ``except Exception`` branch (exit
# 2, rich traceback). Both are listed so the Ctrl-C-at-prompt path
# (exit 130 with resume hint) fires regardless of which one arrives.
except (
KeyboardInterrupt,
asyncio.CancelledError,
typer.Abort,
click.exceptions.Abort,
):
logger.warning(
"cascade_backfill_interrupted",
phase=current_phase.slug if current_phase else phase,
)
_print_interrupted()
_print_summary(summary, exit_code=130)
return 130
except ProviderNotConfiguredError as exc:
# Defensive backstop: phase runners preflight capability and
# return ``blocked_by_capability`` on their own. Reaching this
# branch means a code path bypassed the preflight — surface the
# remediation message instead of letting the broad ``except
# Exception`` below swallow it into "see logs for details".
logger.error(
"cascade_backfill_provider_error_escaped_preflight",
phase=current_phase.slug if current_phase else phase,
provider=exc.provider,
)
typer.secho(f" {exc}", fg=typer.colors.RED)
_print_summary(summary, exit_code=2)
return 2
except Exception:
logger.exception("cascade_backfill_phase_failed", phase=phase)
typer.secho("Backfill failed — see logs for details.", fg=typer.colors.RED)
_print_summary(summary, exit_code=2)
return 2
total_failed = _count_failed_rows(summary)
if total_failed > 0:
logger.warning(
"cascade_backfill_completed_with_failures",
total_failed=total_failed,
)
_print_summary(summary, exit_code=4)
return 4
_print_summary(summary, exit_code=0)
return 0
+186 -9
View File
@@ -11,6 +11,10 @@ without standing up the FastAPI app:
- ``cascade fix`` — list every ``failed`` row. With ``--apply``, also
reset ``retryable=TRUE`` rows back to ``pending`` and drain the
worker once so the retry actually runs before the command returns.
- ``cascade backfill`` — one-shot Tier 1 → Tier 2/3 migration: re-embed
vectors, build clusters, extract skills. See
:func:`everos.entrypoints.cli.commands._backfill_cmd.run_backfill`
for the phase orchestration.
CLI is in-process (12 doc §7.1 + 16 doc §9.2): it constructs the same
:class:`CascadeOrchestrator` as the daemon but only calls
@@ -21,6 +25,7 @@ scanner background task is started.
from __future__ import annotations
import asyncio
import enum
import os
from contextlib import asynccontextmanager
from pathlib import Path
@@ -29,11 +34,13 @@ from typing import Annotated
import typer
from sqlmodel import SQLModel
from everos.component.embedding import build_embedding_provider
from everos.component.embedding import get_embedding_capability
from everos.component.tokenizer import build_tokenizer
from everos.component.utils.datetime import to_display_tz
from everos.config import load_settings
from everos.core.observability.logging import get_logger
from everos.core.persistence import MemoryRoot
from everos.entrypoints.cli._log_setup import configure_cli_logging
from everos.entrypoints.cli.commands._backfill_cmd import run_backfill
from everos.infra.persistence.lancedb import (
dispose_connection,
ensure_business_indexes,
@@ -47,6 +54,8 @@ from everos.infra.persistence.sqlite import (
)
from everos.memory.cascade import CascadeOrchestrator, match_kind
logger = get_logger(__name__)
app = typer.Typer(
name="cascade",
help="Inspect and operate the md → LanceDB sync queue",
@@ -61,12 +70,57 @@ def _cascade_callback(
"--root",
help="Memory root directory (env: EVEROS_ROOT, default: ~/.everos)",
),
verbose: bool = typer.Option(
False,
"--verbose",
"-v",
help="Emit INFO-level lifecycle logs (default: WARNING only).",
),
) -> None:
"""Set memory root before any cascade subcommand runs."""
"""Set memory root and log level before any cascade subcommand runs."""
configure_cli_logging(verbose=verbose)
_apply_root_env(root)
def _apply_root_env(root: str | None) -> None:
"""Set ``EVEROS_ROOT`` env var if a ``--root`` path was passed.
Called from both the group callback (subcommand-independent path)
and from each subcommand (subcommand-level ``--root``). Later calls
override earlier ones — a subcommand's ``--root`` wins over the
group callback's when both are supplied.
"""
if root:
os.environ["EVEROS_ROOT"] = root
def _apply_verbose_logging(verbose: bool | None) -> None:
"""Re-apply the CLI log level from a subcommand-level ``--verbose``.
``None`` means the subcommand did not receive its own ``--verbose``
flag — leave whatever level the group callback set (default
WARNING, or INFO if the user passed ``cascade --verbose``). A
``True``/``False`` value came from the subcommand and wins over the
group callback (later configuration overrides earlier).
Mirrors :func:`_apply_root_env`'s "either position works" pattern
for symmetry — users can write ``everos cascade --verbose status``
or ``everos cascade status --verbose`` and get the same behaviour.
"""
if verbose is not None:
configure_cli_logging(verbose=verbose)
_ROOT_OPTION_HELP = (
"Memory root directory (alias for `cascade --root`; either position works)."
)
_VERBOSE_OPTION_HELP = (
"Emit INFO-level lifecycle logs (alias for `cascade --verbose`; "
"either position works)."
)
# ── shared runtime context ───────────────────────────────────────────────
@@ -92,14 +146,21 @@ async def _runtime(): # type: ignore[no-untyped-def]
def _build_orchestrator() -> CascadeOrchestrator:
settings = load_settings()
memory_root = MemoryRoot.default()
memory_root = MemoryRoot.resolve()
memory_root.ensure()
embedder = build_embedding_provider(settings.embedding)
tokenizer = build_tokenizer()
capability = get_embedding_capability()
if capability.available:
logger.info("cli_cascade_embed_available")
else:
logger.info(
"cli_cascade_embed_unavailable",
reason="embedding not configured; keyword-only mode",
)
return CascadeOrchestrator(
memory_root=memory_root,
embedder=embedder,
tokenizer=tokenizer,
)
@@ -116,8 +177,18 @@ def sync(
"If omitted, only the existing queue is drained.",
),
] = None,
root: Annotated[
str | None,
typer.Option("--root", help=_ROOT_OPTION_HELP),
] = None,
verbose: Annotated[
bool | None,
typer.Option("--verbose", "-v", help=_VERBOSE_OPTION_HELP),
] = None,
) -> None:
"""Drain the cascade queue (and optionally re-enqueue a path first)."""
_apply_root_env(root)
_apply_verbose_logging(verbose)
async def _run() -> None:
async with _runtime():
@@ -144,8 +215,19 @@ def sync(
@app.command("status")
def status() -> None:
def status(
root: Annotated[
str | None,
typer.Option("--root", help=_ROOT_OPTION_HELP),
] = None,
verbose: Annotated[
bool | None,
typer.Option("--verbose", "-v", help=_VERBOSE_OPTION_HELP),
] = None,
) -> None:
"""Print the queue / LSN summary."""
_apply_root_env(root)
_apply_verbose_logging(verbose)
async def _run() -> None:
async with _runtime():
@@ -190,8 +272,18 @@ def fix(
help="Re-enqueue every `retryable=TRUE` row and drain the worker.",
),
] = False,
root: Annotated[
str | None,
typer.Option("--root", help=_ROOT_OPTION_HELP),
] = None,
verbose: Annotated[
bool | None,
typer.Option("--verbose", "-v", help=_VERBOSE_OPTION_HELP),
] = None,
) -> None:
"""List failed rows (default) or re-enqueue retryable ones (``--apply``)."""
_apply_root_env(root)
_apply_verbose_logging(verbose)
async def _run() -> None:
async with _runtime():
@@ -234,6 +326,91 @@ def fix(
asyncio.run(_run())
# ── backfill ─────────────────────────────────────────────────────────────
class _BackfillPhaseOption(enum.StrEnum):
"""CLI-facing ``--phase`` choices — mirrors ``BackfillPhase.slug``."""
VECTORS = "vectors"
CLUSTERS = "clusters"
SKILLS = "skills"
ALL = "all"
@app.command("backfill")
def backfill(
phase: Annotated[
_BackfillPhaseOption,
typer.Option(
"--phase",
help=(
"Which phase to run — vectors (re-embed rows so they become "
"searchable by meaning, not just keyword), clusters (group "
"re-embedded episodes/agent cases into topics), skills "
"(extract reusable agent skills from clustered cases), or "
"all (run every phase in order). Default: all."
),
),
] = _BackfillPhaseOption.ALL,
yes: Annotated[
bool,
typer.Option(
"--yes",
"-y",
help=(
"Auto-confirm every phase's prompt instead of pausing for a "
"y/n answer. Without this flag, each phase shows its "
"token-count estimate and waits for confirmation first."
),
),
] = False,
root: Annotated[
str | None,
typer.Option("--root", help=_ROOT_OPTION_HELP),
] = None,
verbose: Annotated[
bool | None,
typer.Option("--verbose", "-v", help=_VERBOSE_OPTION_HELP),
] = None,
) -> None:
"""Backfill embeddings, clusters, and agent skills after upgrading from
Tier 1 (LLM-only) to Tier 2/3 (embedding configured).
Runs up to three phases in order (see ``--phase``), each showing a
token-count estimate and confirming before doing any work. The
estimate is an input-token count only (K/M shorthand) — never a
price, since per-token pricing is provider-specific and can change.
Ctrl-C is safe: interrupting mid-phase prints a resume hint and exits
130. Every row and phase already written stays written, so
re-running the same ``--phase`` (or ``--phase all``) afterwards only
picks up what's left — nothing is redone from scratch.
"""
_apply_root_env(root)
_apply_verbose_logging(verbose)
async def _run() -> int:
async with _runtime():
return await run_backfill(phase=phase.value, auto_yes=yes)
try:
code = asyncio.run(_run())
except KeyboardInterrupt:
# Safety net: run_backfill already catches CancelledError and
# returns 130 cleanly in the common case, but asyncio.Runner can
# still re-raise a bare KeyboardInterrupt past asyncio.run() on a
# second Ctrl-C. Without this, typer/click would convert it to
# Abort and exit 1 with no resume hint.
typer.secho(
"Interrupted — partial progress was written. Resume by running:",
fg=typer.colors.YELLOW,
)
typer.echo(" everos cascade backfill --phase <phase-name> --yes")
raise typer.Exit(code=130) from None
raise typer.Exit(code=code)
# ── helpers ──────────────────────────────────────────────────────────────
@@ -244,7 +421,7 @@ def _resolve_relative(p: Path) -> str:
must match that convention before calling :meth:`force_enqueue`.
Outside-the-root inputs surface as an error in the caller.
"""
memory_root = MemoryRoot.default()
memory_root = MemoryRoot.resolve()
absolute = p.expanduser().resolve()
try:
rel = absolute.relative_to(memory_root.root)
@@ -7,6 +7,7 @@ import os
import typer
from everos.config.settings import resolve_root
from everos.entrypoints.cli._log_setup import configure_cli_logging
app = typer.Typer(
name="config",
@@ -14,6 +15,20 @@ app = typer.Typer(
no_args_is_help=True,
)
@app.callback()
def _config_callback(
verbose: bool = typer.Option(
False,
"--verbose",
"-v",
help="Emit INFO-level lifecycle logs (default: WARNING only).",
),
) -> None:
"""Set log level before any config subcommand runs."""
configure_cli_logging(verbose=verbose)
_SECRET_FIELDS = {"api_key"}
_SECTION_NAMES = (
@@ -15,6 +15,7 @@ from rich.console import Console
from rich.panel import Panel
from everos.component.utils.datetime import get_utc_now
from everos.entrypoints.cli._log_setup import configure_cli_logging
from everos.entrypoints.tui.demo.data import (
DEFAULT_MEMORY_SEED,
DEFAULT_QUERY,
@@ -64,8 +65,15 @@ def register(parent: typer.Typer) -> None:
"--server-url",
help="EverOS server URL used by --live.",
),
verbose: bool = typer.Option(
False,
"--verbose",
"-v",
help="Emit INFO-level lifecycle logs (default: WARNING only).",
),
) -> None:
"""Launch the EverOS first-memory Textual TUI."""
configure_cli_logging(verbose=verbose)
if live:
_run_live_demo(
cinematic=cinematic,
@@ -17,6 +17,7 @@ from pathlib import Path
import typer
from everos.config.settings import resolve_root
from everos.entrypoints.cli._log_setup import configure_cli_logging
_EVEROS_TEMPLATE = Path(__file__).resolve().parents[3] / "config" / "default.toml"
_OME_TEMPLATE = Path(__file__).resolve().parents[3] / "config" / "default_ome.toml"
@@ -42,6 +43,12 @@ def register(parent: typer.Typer) -> None:
"--print",
help="Print the everos.toml template to stdout instead of writing to disk.",
),
verbose: bool = typer.Option(
False,
"--verbose",
"-v",
help="Emit INFO-level lifecycle logs (default: WARNING only).",
),
) -> None:
"""Generate starter configuration files.
@@ -57,6 +64,7 @@ def register(parent: typer.Typer) -> None:
- 0 — files created successfully (or printed to stdout).
- 1 — files already exist and ``--force`` was not given.
"""
configure_cli_logging(verbose=verbose)
if print_:
import sys
+1 -1
View File
@@ -1,6 +1,6 @@
"""OME SQLite storage — schema initialization + connection factory.
Single file (default ``MemoryRoot.default().ome_db`` ≡
Single file (default ``MemoryRoot.resolve().ome_db`` ≡
``<memory-root>/.index/sqlite/ome.db``). Holds 3 OME-managed tables
(counter_store / idle_store / run_record); APS jobstore table is created
by APScheduler itself when its SQLAlchemyJobStore connects.
+21 -2
View File
@@ -16,7 +16,7 @@ from everos.core.persistence.memory_root import MemoryRoot
def _default_jobstore_path() -> Path:
return MemoryRoot.default().ome_db
return MemoryRoot.resolve().ome_db
class CounterOverride(BaseModel):
@@ -82,7 +82,7 @@ class OMEConfig(BaseModel):
default_factory=_default_jobstore_path,
description="SQLite DB path holding OME's own state (run records, "
"counter store, idle store). Defaults to "
"``MemoryRoot.default().ome_db`` (``<memory-root>/.index/sqlite/ome.db``).",
"``MemoryRoot.resolve().ome_db`` (``<memory-root>/.index/sqlite/ome.db``).",
)
aps_jobstore_path: Path | None = Field(
default=None,
@@ -126,6 +126,25 @@ class OMEConfig(BaseModel):
"fresh run_id.",
),
] = 1800
crash_recovery_enabled: bool = Field(
default=True,
description=(
"Run ``OfflineEngine._run_crash_recovery`` on ``engine.start()``. "
"Default ``True`` — the primary server engine "
"(``service.memorize._get_engine``) needs to resume its own "
"prior sessions' RUNNING work after a crash / restart. "
"Set to ``False`` for one-shot engines that share the jobstore "
"path (e.g. ``memory.cascade._backfill._build_cluster_engine`` "
"and ``_build_skill_engine``). Those engines register only the "
"Phase 2/3 subset of strategies; if they ran crash recovery on "
"startup they would re-enqueue the server's stale RUNNING rows "
"into their own APS scheduler, whose registry doesn't know "
"those strategy names — causing the re-enqueued event to be "
"permanently lost via ``KeyError`` at dispatch time. The "
"server's stale rows stay in ``run_record`` untouched; the "
"next server restart resumes them normally."
),
)
config_path: Path | None = Field(
default=None,
description="Path to ome.toml for per-strategy overrides. None "
+9 -1
View File
@@ -290,7 +290,8 @@ class OfflineEngine:
self._idle_event.set()
self._launch_scheduler()
_ENGINES[self._engine_id] = self
await self._run_crash_recovery()
if self._config.crash_recovery_enabled:
await self._run_crash_recovery()
self._register_scheduled_jobs()
self._start_config_reloader()
self._started = True
@@ -346,6 +347,13 @@ class OfflineEngine:
async def _run_crash_recovery(self) -> None:
"""Scan ``run_record`` for stale RUNNING rows and re-enqueue them.
Runs on :meth:`start` when :attr:`OMEConfig.crash_recovery_enabled`
is ``True`` (default). One-shot engines that register a subset of
strategies (backfill Phase 2/3) opt out via
``crash_recovery_enabled=False`` to avoid re-enqueueing the
server's stale rows into their own scheduler, whose registry
doesn't contain those strategy names.
Treats rows whose ``started_at`` is older than
``crash_recovery_timeout_seconds`` as crashes from a previous
engine session: they are marked CRASHED and re-added to APS with
+187 -31
View File
@@ -28,7 +28,7 @@ import contextlib
import datetime as dt
from everos.core.observability.logging import get_logger
from everos.core.persistence import MemoryRoot
from everos.core.persistence import BaseLanceTable, MemoryRoot, memory_root_lock
# Importing ``tables`` registers every business :class:`BaseLanceTable`
# schema so callers can rely on the package alone to surface every schema.
@@ -75,6 +75,11 @@ class LanceDBSchemaMismatchError(RuntimeError):
"""
class LanceDBMigrationError(RuntimeError):
"""Raised when :func:`migrate_table_schemas` cannot complete and
startup must abort rather than continue in a half-migrated state."""
_FTS_INDEX_SCHEMA_VERSION = 2
"""Bump when the FTS index build config changes so existing on-disk
indexes get rebuilt at startup. v2 = ``with_position=False`` (see
@@ -96,41 +101,180 @@ async def migrate_fts_indexes() -> None:
crashed-optimize churn left behind. Guarded by a version marker in
the LanceDB dir so it runs at most once per bump; the rebuild is
O(N) but only on the first startup after upgrade.
Cross-process serialization: wrapped in :func:`memory_root_lock` so a
server startup racing with a concurrent ``everos cascade`` command
(both call this via :func:`ensure_business_indexes`) can't attempt
two concurrent rebuild passes. First process runs the migration and
writes the marker; second waits for the lock, re-checks the marker
on entry, and no-ops. Removes the same TOCTOU window between marker
read and the drop-index/rebuild loop that :issue:`M7` from the
PR #361 review flagged on the sibling schema migration.
"""
logger = get_logger(__name__)
marker = MemoryRoot.default().lancedb_dir / ".fts_index_version"
try:
current = int(marker.read_text().strip()) if marker.exists() else 0
except (ValueError, OSError):
current = 0
if current >= _FTS_INDEX_SCHEMA_VERSION:
return
logger.info("fts_index_migration_started", target=_FTS_INDEX_SCHEMA_VERSION)
for schema in _BUSINESS_SCHEMAS:
if not schema.BM25_FIELDS:
continue
table = await get_table(schema.TABLE_NAME, schema)
# Drop existing indexes (everos only builds FTS here; mirrors
# LanceRepoBase.rebuild_indexes) then rebuild with the new config.
for idx in await table.list_indices():
await table.drop_index(idx.name)
await schema.ensure_fts_indexes(table)
# Reclaim the orphaned index dirs + data fragments the crashed
# optimize loop piled up. Safe now: the crashing index is gone,
# so compaction no longer decodes a position List.
with contextlib.suppress(Exception):
await table.optimize(cleanup_older_than=dt.timedelta(seconds=0))
marker.write_text(str(_FTS_INDEX_SCHEMA_VERSION))
logger.info("fts_index_migration_done", version=_FTS_INDEX_SCHEMA_VERSION)
memory_root = MemoryRoot.resolve()
async with memory_root_lock(memory_root):
marker = memory_root.lancedb_dir / ".fts_index_version"
try:
current = int(marker.read_text().strip()) if marker.exists() else 0
except (ValueError, OSError):
current = 0
if current >= _FTS_INDEX_SCHEMA_VERSION:
# Another process finished the migration while we waited
# for the lock. Nothing to do.
return
logger.info("fts_index_migration_started", target=_FTS_INDEX_SCHEMA_VERSION)
for schema in _BUSINESS_SCHEMAS:
if not schema.BM25_FIELDS:
continue
table = await get_table(schema.TABLE_NAME, schema)
# Drop existing indexes (everos only builds FTS here; mirrors
# LanceRepoBase.rebuild_indexes) then rebuild with the new config.
for idx in await table.list_indices():
await table.drop_index(idx.name)
await schema.ensure_fts_indexes(table)
# Reclaim the orphaned index dirs + data fragments the crashed
# optimize loop piled up. Safe now: the crashing index is gone,
# so compaction no longer decodes a position List.
with contextlib.suppress(Exception):
await table.optimize(cleanup_older_than=dt.timedelta(seconds=0))
marker.write_text(str(_FTS_INDEX_SCHEMA_VERSION))
logger.info("fts_index_migration_done", version=_FTS_INDEX_SCHEMA_VERSION)
_TABLE_SCHEMA_VERSION = 2
"""Bump when column nullability or presence changes on business tables.
v2 = ``vector: Vector(_DIM) | None`` on the 6 business tables that carry
a vector column (see :data:`BUSINESS_SCHEMAS_WITH_VECTOR`).
.. note::
:func:`migrate_table_schemas` currently hard-codes the v2 alter — it
does not implement per-version dispatch. Bumping this constant to
``3`` without first adding a dispatch table will short-circuit the
v3 migration entirely (the current body only compares
``current >= _TABLE_SCHEMA_VERSION`` and then runs the v2-specific
alter unconditionally). Any future v3 migration MUST:
1. Add a ``_MIGRATIONS: dict[int, Callable[[], Awaitable[None]]]``
registry mapping target version → migration coroutine.
2. Rewrite :func:`migrate_table_schemas` to run every migration from
``current + 1`` up to ``_TABLE_SCHEMA_VERSION`` in order, updating
the marker after each step so a mid-migration crash resumes at
the next step instead of replaying earlier ones.
Accepted as a design gap in PR #361 review finding #5 — not blocking
today because there is no v3.
"""
BUSINESS_SCHEMAS_WITH_VECTOR: tuple[type[BaseLanceTable], ...] = (
Episode,
AtomicFact,
Foresight,
AgentCase,
AgentSkill,
KnowledgeTopic,
)
"""Business schemas whose ``vector`` column needs the v2 nullability
migration. ``Episode.subject_vector`` was already nullable in v1.1.1 and
is intentionally left out of this migration. ``UserProfile`` has no
vector column; ``knowledge_document`` has no LanceDB table."""
async def migrate_table_schemas() -> None:
"""One-time ``alter_columns`` making ``vector`` nullable on business tables.
Soft-dependency embedding mode lets cascade write ``vector=None`` for
rows it cannot embed; the on-disk column must allow that. LanceDB's
``alter_columns`` is metadata-only (no data movement, no re-embed),
so this runs in milliseconds. Guarded by ``.table_schema_version``
so it runs at most once per version bump; per-table nullable check
is defense-in-depth so an already-migrated (or freshly-created,
already-nullable) table is a no-op rather than a redundant alter.
Cross-process serialization: wrapped in :func:`memory_root_lock` so
a server startup racing with a concurrent ``everos cascade`` command
(both call this via :func:`ensure_business_indexes`) doesn't attempt
two concurrent alter passes. First process runs the migration and
writes the marker; second waits for the lock, re-checks the marker
on entry, and no-ops. Removes the TOCTOU window between marker
read and alter that :issue:`M7` from the PR #361 review flagged.
Fail-loud semantics preserved: if ``alter_columns`` genuinely fails
for any table (LanceDB version mismatch, corrupted index), raises
:class:`LanceDBMigrationError`. Startup aborts rather than
continuing in a half-migrated state — cascade would otherwise write
``vector=None`` (soft-dependency embedding) into a still NOT-NULL
column and every row would silently fail. Recovery escalates from
a plain restart (transient hiccup) to wiping the LanceDB index
directory (rebuildable from md, the SoT).
"""
logger = get_logger(__name__)
memory_root = MemoryRoot.resolve()
async with memory_root_lock(memory_root):
marker = memory_root.lancedb_dir / ".table_schema_version"
try:
current = int(marker.read_text().strip()) if marker.exists() else 0
except (ValueError, OSError):
current = 0
if current >= _TABLE_SCHEMA_VERSION:
# Another process finished the migration while we waited
# for the lock. Nothing to do.
return
logger.info("table_schema_migration_started", target=_TABLE_SCHEMA_VERSION)
failed_schemas: list[str] = []
for schema in BUSINESS_SCHEMAS_WITH_VECTOR:
table = await get_table(schema.TABLE_NAME, schema)
arrow_schema = await table.schema()
if arrow_schema.field("vector").nullable:
continue
try:
await table.alter_columns({"path": "vector", "nullable": True})
except Exception as exc:
logger.warning(
"table_schema_migration_alter_failed",
table=schema.TABLE_NAME,
error=str(exc),
)
failed_schemas.append(schema.TABLE_NAME)
if failed_schemas:
logger.error(
"table_schema_migration_incomplete",
failed_schemas=failed_schemas,
)
raise LanceDBMigrationError(
f"LanceDB nullable-vector migration failed for tables "
f"{failed_schemas!r}. Startup aborted to prevent cascade "
f"from writing NULL vectors into NOT-NULL columns. This "
f"typically indicates a LanceDB version mismatch or a "
f"corrupted index. Recovery, in order of least- to "
f"most-destructive: (1) restart the process; a transient "
f"filesystem or LanceDB-side hiccup may resolve. (2) If "
f"the error persists, wipe the index directory "
f"`{memory_root.lancedb_dir}` and restart — cascade will "
f"re-index from source markdown."
)
marker.parent.mkdir(parents=True, exist_ok=True)
try:
marker.write_text(str(_TABLE_SCHEMA_VERSION))
except OSError:
logger.error("table_schema_migration_marker_write_failed", path=str(marker))
raise
logger.info("table_schema_migration_done", version=_TABLE_SCHEMA_VERSION)
async def ensure_business_indexes() -> None:
"""Ensure FTS (BM25) indexes for every business table (idempotent).
Called once at startup by :class:`LanceDBLifespanProvider`. First
runs :func:`migrate_fts_indexes` (one-time, marker-guarded) to
rebuild any pre-fix ``with_position=True`` indexes, then walks the
business schemas (each owns its ``TABLE_NAME`` + ``BM25_FIELDS``),
runs :func:`migrate_table_schemas` (schema first, one-time,
marker-guarded) to make ``vector`` columns nullable, then
:func:`migrate_fts_indexes` (index second, one-time, marker-guarded)
to rebuild any pre-fix ``with_position=True`` indexes, then walks
the business schemas (each owns its ``TABLE_NAME`` + ``BM25_FIELDS``),
opens each table via :func:`get_table`, and delegates to
``schema.ensure_fts_indexes(table)``. Already-indexed columns are
skipped, so re-runs are no-ops.
@@ -139,6 +283,7 @@ async def ensure_business_indexes() -> None:
everything else (table name, columns to index) reads off the
schema's ClassVars.
"""
await migrate_table_schemas()
await migrate_fts_indexes()
for schema in _BUSINESS_SCHEMAS:
table = await get_table(schema.TABLE_NAME, schema)
@@ -166,19 +311,29 @@ async def verify_business_schemas() -> None:
if missing or extra:
raise LanceDBSchemaMismatchError(
f"LanceDB table {schema.TABLE_NAME!r} schema drift: "
f"missing={sorted(missing)}, extra={sorted(extra)}. "
"The index is rebuildable from md — recover with "
"`rm -rf ~/.everos/.index/lancedb` and restart."
f"missing={sorted(missing)}, extra={sorted(extra)}.\n"
"Recovery, escalating:\n"
" 1. Restart the server — an in-flight migration may "
"still be finishing (harmless if this is your first "
"restart after upgrading EverOS).\n"
" 2. If restart doesn't clear it, the on-disk index is "
"genuinely out of sync with the code's schema. Because "
"the LanceDB index is fully rebuildable from md, wipe "
"it: `rm -rf ~/.everos/.index/lancedb` and restart — "
"the cascade daemon will re-index from the source-of-"
"truth markdown."
)
__all__ = [
"BUSINESS_SCHEMAS_WITH_VECTOR",
"AgentCase",
"AgentSkill",
"AtomicFact",
"Episode",
"Foresight",
"KnowledgeTopic",
"LanceDBMigrationError",
"LanceDBSchemaMismatchError",
"ParentType",
"UserProfile",
@@ -193,6 +348,7 @@ __all__ = [
"get_table",
"knowledge_topic_repo",
"migrate_fts_indexes",
"migrate_table_schemas",
"user_profile_repo",
"verify_business_schemas",
]
@@ -28,7 +28,7 @@ _lock = asyncio.Lock()
async def get_connection() -> AsyncConnection:
"""Return the process-wide async LanceDB connection.
Built on first call from ``MemoryRoot.default().lancedb_dir`` and
Built on first call from ``MemoryRoot.resolve().lancedb_dir`` and
``Settings.lancedb``. Subsequent calls return the same instance.
"""
async with _lock:
@@ -72,7 +72,7 @@ async def _ensure_connection_locked() -> AsyncConnection:
global _conn
if _conn is None:
settings = load_settings()
memory_root = MemoryRoot.default()
memory_root = MemoryRoot.resolve()
memory_root.ensure()
_conn = await open_lancedb_connection(memory_root.lancedb_dir, settings.lancedb)
logger.info(
@@ -2,9 +2,14 @@
from __future__ import annotations
import datetime as dt
from collections.abc import Sequence
from typing import Any
from lancedb import AsyncTable
from everos.core.persistence.lancedb import LanceDailyLogRepoBase
from everos.core.persistence.lancedb.repository import _q
from ..lancedb_manager import get_table
from ..tables.episode import Episode
@@ -16,5 +21,146 @@ class _EpisodeRepo(LanceDailyLogRepoBase[Episode]):
async def _table_lookup(self) -> AsyncTable:
return await get_table(self.schema.TABLE_NAME, self.schema)
async def count_by_owner(
self,
owner_id: str,
*,
app_id: str = "default",
project_id: str = "default",
parent_type: str | None = None,
) -> int:
"""Count episode rows for one owner within an ``(app, project)`` scope.
Args:
owner_id: The row's ``owner_id`` field.
app_id: App scope segment (default ``"default"``).
project_id: Project scope segment (default ``"default"``).
parent_type: When non-``None``, restrict the count to rows whose
``parent_type`` matches (e.g. ``"memcell"`` to exclude
Reflection-merged rows with ``parent_type='cluster'``). Kept
optional so callers without a parent-type requirement rely on
the ``None`` default meaning "count everything".
Used by :func:`extract_user_profile`'s Tier-1 direct-path throttle
to gate re-extraction frequency. LanceDB ``count_rows`` accepts a
SQL-style ``filter`` predicate; the where-string is built with the
same escape discipline (:func:`_q`) as
:meth:`list_by_owner_after_ts`.
Excludes rows with ``deprecated_by IS NOT NULL`` (Reflection-
superseded episodes) from the count, matching the filter idiom
used by :func:`memory.search.filters.compile_filters` and the
reflection read paths. Keeps the profile-throttle count aligned
with the "live" episode surface the strategy actually operates on.
"""
filter_str = (
f"owner_id = '{_q(owner_id)}' "
f"AND app_id = '{_q(app_id)}' "
f"AND project_id = '{_q(project_id)}' "
f"AND deprecated_by IS NULL"
)
if parent_type is not None:
filter_str += f" AND parent_type = '{_q(parent_type)}'"
table = await self._table()
return await table.count_rows(filter=filter_str)
async def list_by_owner_after_ts(
self,
*,
owner_id: str,
after_ts: int,
parent_type: str,
app_id: str = "default",
project_id: str = "default",
columns: Sequence[str] | None = None,
limit: int | None = None,
) -> list[Episode] | list[dict[str, Any]]:
"""Scalar filter over owner + timestamp + parent_type + scope.
Fetches episodes for the given owner after a specific timestamp,
scoped by parent_type and app/project. No vector dependency —
Tier 1-safe. Rows with vector=None are included.
Used by extract_user_profile direct path (KEYWORD_ONLY mode) to
select memcells to feed the profile extractor without going through
cluster events.
Excludes rows with ``deprecated_by IS NOT NULL`` (Reflection-
superseded episodes) — matches the filter idiom used by
:func:`memory.search.filters.compile_filters` and reflection
reads. Without this, direct-path profile extraction would feed
stale (superseded) memcell references to the extractor.
Args:
owner_id: Owner identifier.
after_ts: Timestamp threshold (milliseconds since epoch); rows with
timestamp > after_ts are returned.
parent_type: Parent type filter (e.g. "memcell").
app_id: Application scope (default "default").
project_id: Project scope (default "default").
columns: Optional column projection. Default (``None``)
returns full ``Episode`` model objects. Callers that only
need a small subset (e.g. ``parent_id`` in
``extract_user_profile``) should pass an explicit list to
skip the 1024-D ``vector`` and ``subject_vector`` payload.
When set, this method returns raw dicts scoped to the
requested columns; otherwise it returns typed ``Episode``
objects. Callers switch on the projection they asked for.
limit: Optional row cap. Default (``None``) means no cap —
the entire matching set is fetched. Callers with large
historical windows should pass a bound to avoid pulling
every memcell for the owner.
**CAVEAT — limit is NOT "newest N"**: LanceDB does not
push ORDER BY down; ``.limit(N)`` truncates whatever
the scan returns first (typically fragment / insertion
order for an append-only table = OLDEST N). The
ascending-by-timestamp sort below runs on the already-
truncated Python list, so the returned rows are the
*oldest matching subset*, not the newest. Callers that
need "most recent N" MUST either fetch unlimited and
slice in Python (accepting the memory cost), or apply
a tighter ``after_ts`` bound to shrink the matching
set upstream. A regression here would silently window
profile extraction over the wrong slice of history.
Returns:
When ``columns`` is ``None``: list of ``Episode`` objects
ordered ascending by timestamp (existing behaviour).
When ``columns`` is a projection: list of raw dicts holding
only the requested columns, ordered ascending by timestamp.
"""
from everos.component.utils.datetime import to_iso_format
# Convert milliseconds to UTC datetime, then to ISO string for DataFusion
# (stored timestamps are always UTC)
ts_dt = dt.datetime.fromtimestamp(int(after_ts) / 1000.0, tz=dt.UTC)
ts_iso = to_iso_format(ts_dt)
where = (
f"owner_id = '{_q(owner_id)}' "
f"AND timestamp > TIMESTAMP '{ts_iso}' "
f"AND parent_type = '{_q(parent_type)}' "
f"AND app_id = '{_q(app_id)}' "
f"AND project_id = '{_q(project_id)}' "
f"AND deprecated_by IS NULL"
)
table = await self._table()
query = table.query().where(where)
# Projection: caller opted into a raw-dict return by naming columns;
# `timestamp` is force-included so the ascending sort below is stable
# even when the caller forgot to list it.
projected = columns is not None
if projected:
projection = list(dict.fromkeys([*columns, "timestamp"])) # type: ignore[misc]
query = query.select(projection)
if limit is not None:
query = query.limit(limit)
rows = await query.to_list()
rows.sort(key=lambda r: r["timestamp"])
if projected:
return rows
return [self.schema.model_validate(r) for r in rows]
episode_repo = _EpisodeRepo()
@@ -81,4 +81,4 @@ class AgentCase(BaseLanceTable):
session_id / timestamp / parent_id) is NOT in the hash. See
:attr:`AgentCaseHandler.content_change_keys`."""
vector: Vector(_DIM) # type: ignore[valid-type]
vector: Vector(_DIM) | None = None # type: ignore[valid-type]
@@ -77,4 +77,4 @@ class AgentSkill(BaseLanceTable):
changed (e.g. the watcher fires for unrelated stat updates). See
:attr:`AgentSkillHandler.content_change_keys`."""
vector: Vector(_DIM) # type: ignore[valid-type]
vector: Vector(_DIM) | None = None # type: ignore[valid-type]
@@ -64,4 +64,4 @@ class AtomicFact(BaseLanceTable):
consolidated. Value is the cluster entry_id that supersedes this
row. ``NULL`` means the row is still active."""
vector: Vector(_DIM) # type: ignore[valid-type]
vector: Vector(_DIM) | None = None # type: ignore[valid-type]
@@ -80,5 +80,5 @@ class Episode(BaseLanceTable):
consolidated into a cluster. Value is the cluster entry_id that
supersedes this row. ``NULL`` means the row is still active."""
vector: Vector(_DIM) # type: ignore[valid-type]
vector: Vector(_DIM) | None = None # type: ignore[valid-type]
subject_vector: Vector(_DIM) | None = None # type: ignore[valid-type]
@@ -76,4 +76,4 @@ class Foresight(BaseLanceTable):
(owner_id / session_id / timestamp / parent_id / sender_ids) is NOT
in the hash. See :attr:`ForesightHandler.content_change_keys`."""
vector: Vector(_DIM) # type: ignore[valid-type]
vector: Vector(_DIM) | None = None # type: ignore[valid-type]
@@ -31,6 +31,6 @@ class KnowledgeTopic(BaseLanceTable):
content_labels: list[str] = []
md_path: str
content_sha256: str
vector: Vector(_DIM) # type: ignore[valid-type] -- Vector() is runtime-constructed; static analyzers cannot verify
vector: Vector(_DIM) | None = None # type: ignore[valid-type] -- Vector() is runtime-constructed; static analyzers cannot verify
created_at: dt.datetime
updated_at: dt.datetime
@@ -113,17 +113,43 @@ class _ClusterRepo(RepoBase[Cluster]):
self,
member_type: str,
member_id: str,
*,
app_id: str,
project_id: str,
owner_id: str,
) -> str | None:
"""Reverse lookup: ``(member_type, member_id) → cluster_id``.
"""Reverse lookup: ``(member_type, member_id) → cluster_id`` scoped
to ``(app_id, project_id, owner_id)``.
Returns ``None`` when the entity is not yet attached to any cluster.
Backed by ``ix_cluster_member_reverse`` so it is O(log N).
``member_id`` (e.g. episode ``entry_id`` like
``ep_20260517_00000001``) is only per-owner unique — see
``core/persistence/markdown/entries.py``: "Cross-user uniqueness
is handled at the database layer via a composite
``<user_id>_<entry_id>`` field; it is not encoded into the
EntryId string itself." Without the scope filter, two owners
writing on the same day would share ``entry_id`` and either
collide on the reverse index (false hit → the second owner's
row silently drops from a cluster it was never part of) or find
a foreign cluster.
Filter cascades: reverse index on ``(member_type, member_id)``
narrows to O(k) rows where k = number of owners sharing that
entry_id (usually 1), then the join to :class:`Cluster` filters
by scope. ``Cluster`` already carries ``app_id / project_id /
owner_id`` as of the initial schema.
Returns ``None`` when the entity is not attached to a cluster
under the specified scope.
"""
async with session_scope(self._factory) as s:
stmt = (
select(ClusterMember.cluster_id)
.join(Cluster, Cluster.cluster_id == ClusterMember.cluster_id)
.where(ClusterMember.member_type == member_type)
.where(ClusterMember.member_id == member_id)
.where(Cluster.app_id == app_id)
.where(Cluster.project_id == project_id)
.where(Cluster.owner_id == owner_id)
.limit(1)
)
return (await s.execute(stmt)).scalar_one_or_none()
@@ -207,30 +207,48 @@ class _MdChangeStateRepo(RepoBase[MdChangeState]):
row = await s.get(MdChangeState, md_path)
return row
async def claim_pending_batch(self, limit: int = 100) -> list[MdChangeState]:
async def claim_pending_batch(
self,
limit: int = 100,
*,
kinds: set[str] | None = None,
) -> list[MdChangeState]:
"""Claim up to ``limit`` pending rows in LSN order.
Returns the claimed rows (now ``status='processing'``); empty
list if none were pending. Sibling workers / processes may race
on the same prefix — the per-row ``WHERE status='pending'``
filter ensures each row lands in exactly one batch.
Args:
limit: Maximum rows to claim in one batch.
kinds: Optional restriction on which ``kind`` values are
eligible. ``None`` (the default) claims across every
registered kind — the shape the background worker and
the CLI ``cascade sync`` path have always used. Passing
a set restricts the SELECT + UPDATE to ``kind IN (...)``
via parameter binding (no string interpolation), so a
Phase-3 sync scoped to ``{"agent_skill"}`` cannot touch
unrelated kinds' queue state. Empty set is treated the
same as ``None`` — meaning "no kind filter" would be
surprising, but returning an empty result is what the
caller almost certainly meant, so we short-circuit.
"""
if limit <= 0:
return []
if kinds is not None and not kinds:
return []
now = get_utc_now()
async with session_scope(self._factory) as s:
picks = (
(
await s.execute(
select(MdChangeState.md_path)
.where(MdChangeState.status == "pending")
.order_by(MdChangeState.lsn)
.limit(limit)
)
)
.scalars()
.all()
pick_stmt = (
select(MdChangeState.md_path)
.where(MdChangeState.status == "pending")
.order_by(MdChangeState.lsn)
.limit(limit)
)
if kinds is not None:
pick_stmt = pick_stmt.where(MdChangeState.kind.in_(kinds))
picks = (await s.execute(pick_stmt)).scalars().all()
if not picks:
return []
update_result = await s.execute(
@@ -17,6 +17,17 @@ from everos.core.persistence.sqlite import RepoBase, session_scope
from ..sqlite_manager import get_session_factory
from ..tables import Memcell
_SQLITE_IN_CHUNK = 500
"""SQL ``IN (?, ?, ...)`` parameter cap for :meth:`_MemcellRepo.find_by_ids`.
SQLite's ``SQLITE_MAX_VARIABLE_NUMBER`` defaults to 999 on the shipped
amalgamation and rises to 32766 on ≥3.32; 500 stays well under the older
cap and keeps the query planner's cost estimate cheap. Callers passing
more ``memcell_ids`` than this fan out into multiple SELECTs, merged
in-memory — see the direct-path selector in
``memory/strategies/extract_user_profile.py`` where
``last_profile_ts=0`` can pull an owner's entire episode history."""
class _MemcellRepo(RepoBase[Memcell]):
model = Memcell
@@ -36,16 +47,25 @@ class _MemcellRepo(RepoBase[Memcell]):
async def find_by_ids(self, memcell_ids: list[str]) -> list[Memcell]:
"""Bulk fetch rows by primary key list — preserves caller order.
Chunked at ``_SQLITE_IN_CHUNK`` to protect against
``SQLITE_MAX_VARIABLE_NUMBER`` blowups on large owner histories
(e.g. ``extract_user_profile``'s direct path when ``user.md``
doesn't exist yet — ``last_profile_ts=0`` fetches every memcell
the owner has ever produced).
Used by offline strategies that pull every memcell in a cluster
(membership lives in :class:`ClusterMember` and is supplied to
the strategy via :class:`everalgo.clustering.Cluster.members`).
"""
if not memcell_ids:
return []
by_id: dict[str, Memcell] = {}
async with session_scope(self._factory) as s:
stmt = select(Memcell).where(Memcell.memcell_id.in_(memcell_ids))
rows = list((await s.execute(stmt)).scalars().all())
by_id = {r.memcell_id: r for r in rows}
for start in range(0, len(memcell_ids), _SQLITE_IN_CHUNK):
chunk = memcell_ids[start : start + _SQLITE_IN_CHUNK]
stmt = select(Memcell).where(Memcell.memcell_id.in_(chunk))
rows = (await s.execute(stmt)).scalars().all()
by_id.update((r.memcell_id, r) for r in rows)
return [by_id[mid] for mid in memcell_ids if mid in by_id]
@@ -3,7 +3,7 @@
The single place that owns the SQLite **runtime state**: the async
SQLAlchemy engine and the session factory bound to it. Built lazily on
first :func:`get_engine` / :func:`get_session_factory` call from
:func:`everos.config.load_settings` + :meth:`MemoryRoot.default`. The
:func:`everos.config.load_settings` + :meth:`MemoryRoot.resolve`. The
:class:`SqliteLifespanProvider` calls :func:`dispose_engine` on shutdown
to drain the connection pool; in scripts you can call it manually.
"""
@@ -29,13 +29,13 @@ _session_factory: async_sessionmaker[AsyncSession] | None = None
def get_engine() -> AsyncEngine:
"""Return the process-wide async SQLAlchemy engine.
Built on first call from ``MemoryRoot.default()`` and ``Settings.sqlite``.
Built on first call from ``MemoryRoot.resolve()`` and ``Settings.sqlite``.
Subsequent calls return the same instance.
"""
global _engine
if _engine is None:
settings = load_settings()
memory_root = MemoryRoot.default()
memory_root = MemoryRoot.resolve()
memory_root.ensure()
_engine = create_system_engine(memory_root.system_db, settings.sqlite)
logger.info(
+11
View File
@@ -11,8 +11,16 @@ Public surface — what lifespan providers / CLI commands import:
- :class:`CascadeConfig` — construction-time tuning knobs.
- :data:`KIND_REGISTRY` / :func:`match_kind` — kind dispatch (also
used by CLI ``cascade sync --path`` to resolve a single file's kind).
- :class:`BackfillPhase` — dataclass shared between the memory-layer
phase runners and the CLI's ``PHASES`` copy. The concrete ``PHASES``
tuple and ``run_backfill`` orchestrator live in
``everos.entrypoints.cli.commands._backfill_cmd`` (M11: the memory
layer must not depend on typer/click).
"""
from ._backfill import BackfillPhase as BackfillPhase
from ._backfill import BackfillPresenter as BackfillPresenter
from ._backfill import NullBackfillPresenter as NullBackfillPresenter
from .orchestrator import CascadeConfig as CascadeConfig
from .orchestrator import CascadeOrchestrator as CascadeOrchestrator
from .registry import KIND_REGISTRY as KIND_REGISTRY
@@ -21,8 +29,11 @@ from .registry import match_kind as match_kind
__all__ = [
"KIND_REGISTRY",
"BackfillPhase",
"BackfillPresenter",
"CascadeConfig",
"CascadeOrchestrator",
"KindSpec",
"NullBackfillPresenter",
"match_kind",
]
File diff suppressed because it is too large Load Diff
@@ -18,6 +18,8 @@ md contract:
``sections``:
- ``TaskIntent``: short intent statement (embedded + BM25 primary).
Embedding is a soft dependency: when unavailable, ``vector`` is
written as ``None`` and the row stays BM25/scalar-searchable only.
- ``Approach``: step-by-step approach (BM25 secondary field via
``approach_tokens``, **not** fed to the embedder — the
retrieval anchor is task_intent).
@@ -26,11 +28,15 @@ md contract:
from __future__ import annotations
from everos.component.embedding import get_embedding_capability
from everos.core.observability.logging import get_logger
from everos.infra.persistence.lancedb import AgentCase, ParentType, agent_case_repo
from ._common import require_float, require_iso_timestamp
from ._daily_log_base import BaseDailyLogHandler, ParsedEntry
logger = get_logger(__name__)
class AgentCaseHandler(BaseDailyLogHandler):
"""Cascade handler for ``agents/<a>/.cases/agent_case-*.md``."""
@@ -62,7 +68,14 @@ class AgentCaseHandler(BaseDailyLogHandler):
key_insight = (s.sections.get("KeyInsight") or "").strip() or None
intent_tokens = self._deps.tokenizer.tokenize(task_intent)
approach_tokens = self._deps.tokenizer.tokenize(approach)
vector = await self._deps.embedder.embed(task_intent)
vector = await get_embedding_capability().embed_or_none(task_intent)
if vector is None:
logger.debug(
"cascade_handler_embed_skipped",
kind=self.kind,
entry_id=entry.entry_id,
reason="embedding_capability_unavailable",
)
return AgentCase(
id=f"{owner_id}_{entry.entry_id}",
entry_id=entry.entry_id,
@@ -27,7 +27,9 @@ md contract:
Embedding source: ``name + "\\n" + description`` (mirrors opensource
AgentSkillExtractor — name belongs in the retrieval anchor too,
not just description).
not just description). Embedding is a soft dependency: when
unavailable, ``vector`` is written as ``None`` and the row stays
BM25/scalar-searchable only.
"""
from __future__ import annotations
@@ -37,6 +39,8 @@ from typing import Any, ClassVar
import anyio
from everos.component.embedding import get_embedding_capability
from everos.core.observability.logging import get_logger
from everos.core.persistence import MarkdownReader
from everos.infra.persistence.lancedb import AgentSkill, agent_skill_repo
from everos.infra.persistence.markdown import AgentSkillFrontmatter
@@ -46,6 +50,8 @@ from ._common import content_sha256 as compute_content_sha256
from ._common import resolve_scope
from .base import Handler
logger = get_logger(__name__)
class AgentSkillHandler(Handler):
"""Cascade handler for
@@ -140,7 +146,14 @@ class AgentSkillHandler(Handler):
content_tokens = " ".join(self._deps.tokenizer.tokenize(content))
# Embedding source: name + description joined (opensource parity).
embed_text = "\n".join(s for s in [name, description] if s)
vector = await self._deps.embedder.embed(embed_text)
vector = await get_embedding_capability().embed_or_none(embed_text)
if vector is None:
logger.debug(
"cascade_handler_embed_skipped",
kind=self.kind,
entry_id=skill_id,
reason="embedding_capability_unavailable",
)
row = AgentSkill(
id=skill_id,
@@ -14,16 +14,22 @@ md contract (md writer + cascade share this shape):
``sections``:
- ``Fact``: the atomic-fact sentence — fed to the embedder and the
tokenizer (``fact_tokens`` BM25 field).
tokenizer (``fact_tokens`` BM25 field). Embedding is a soft
dependency: when unavailable, ``vector`` is written as ``None`` and
the row stays BM25/scalar-searchable only.
"""
from __future__ import annotations
from everos.component.embedding import get_embedding_capability
from everos.core.observability.logging import get_logger
from everos.infra.persistence.lancedb import AtomicFact, ParentType, atomic_fact_repo
from ._common import parse_inline_list, require_iso_timestamp
from ._daily_log_base import BaseDailyLogHandler, ParsedEntry
logger = get_logger(__name__)
class AtomicFactHandler(BaseDailyLogHandler):
"""Cascade handler for ``users/<u>/.atomic_facts/atomic_fact-*.md``."""
@@ -47,7 +53,14 @@ class AtomicFactHandler(BaseDailyLogHandler):
s = entry.structured
text = s.sections.get("Fact", "").strip()
tokens = self._deps.tokenizer.tokenize(text)
vector = await self._deps.embedder.embed(text)
vector = await get_embedding_capability().embed_or_none(text)
if vector is None:
logger.debug(
"cascade_handler_embed_skipped",
kind=self.kind,
entry_id=entry.entry_id,
reason="embedding_capability_unavailable",
)
return AtomicFact(
id=f"{owner_id}_{entry.entry_id}",
entry_id=entry.entry_id,
+7 -6
View File
@@ -6,10 +6,13 @@ row state. The handler is the *only* cascade-side piece that knows
the per-kind row shape; everything around it (watcher / scanner /
worker / orchestrator / CLI) is kind-agnostic.
Per-kind handlers share the same dependencies bundle — embedding,
tokenizer, memory-root path resolver — packaged as
:class:`HandlerDeps`. Construct it once at orchestrator startup, pass
to every handler factory, no per-row resolution churn.
Per-kind handlers share the same dependencies bundle — tokenizer,
memory-root path resolver — packaged as :class:`HandlerDeps`.
Construct it once at orchestrator startup, pass to every handler
factory, no per-row resolution churn. Embedding is a soft dependency:
handlers fetch it themselves via
:func:`everos.component.embedding.get_embedding_capability` rather
than receiving it here.
"""
from __future__ import annotations
@@ -17,7 +20,6 @@ from __future__ import annotations
import abc
import dataclasses
from everos.component.embedding import EmbeddingProvider
from everos.component.tokenizer import Tokenizer
from everos.core.persistence import MemoryRoot
@@ -35,7 +37,6 @@ class HandlerDeps:
"""
memory_root: MemoryRoot
embedder: EmbeddingProvider
tokenizer: Tokenizer
+19 -3
View File
@@ -26,17 +26,25 @@ writers must match):
- ``Summary`` (optional): condensed narrative.
- ``Content``: full episode narrative — fed to the embedder AND the
tokenizer for the ``episode_tokens`` BM25 field.
Embedding is a soft dependency: when unavailable, both ``vector`` and
``subject_vector`` are written as ``None`` and the row stays
BM25/scalar-searchable only.
"""
from __future__ import annotations
import asyncio
from everos.component.embedding import get_embedding_capability
from everos.core.observability.logging import get_logger
from everos.infra.persistence.lancedb import Episode, ParentType, episode_repo
from ._common import parse_inline_list, require_iso_timestamp
from ._daily_log_base import BaseDailyLogHandler, ParsedEntry
logger = get_logger(__name__)
class EpisodeHandler(BaseDailyLogHandler):
"""Cascade handler for ``users/<u>/episodes/episode-*.md``."""
@@ -72,14 +80,22 @@ class EpisodeHandler(BaseDailyLogHandler):
subject_text = s.sections.get("Subject", "").strip()
# Embed content and subject concurrently; skip subject embed when absent.
capability = get_embedding_capability()
if subject_text:
vector, subject_vector = await asyncio.gather(
self._deps.embedder.embed(text),
self._deps.embedder.embed(subject_text),
capability.embed_or_none(text),
capability.embed_or_none(subject_text),
)
else:
vector = await self._deps.embedder.embed(text)
vector = await capability.embed_or_none(text)
subject_vector = None
if vector is None:
logger.debug(
"cascade_handler_embed_skipped",
kind=self.kind,
entry_id=entry.entry_id,
reason="embedding_capability_unavailable",
)
# BM25 tokenization covers both body and subject keywords.
tokenize_source = f"{text} {subject_text}" if subject_text else text
@@ -21,11 +21,15 @@ md contract:
``sections``:
- ``Foresight``: forward-looking inference text (embedded + BM25).
Embedding is a soft dependency: when unavailable, ``vector`` is
written as ``None`` and the row stays BM25/scalar-searchable only.
- ``Evidence`` (optional): supporting excerpt (secondary BM25 only).
"""
from __future__ import annotations
from everos.component.embedding import get_embedding_capability
from everos.core.observability.logging import get_logger
from everos.infra.persistence.lancedb import Foresight, ParentType, foresight_repo
from ._common import (
@@ -36,6 +40,8 @@ from ._common import (
)
from ._daily_log_base import BaseDailyLogHandler, ParsedEntry
logger = get_logger(__name__)
class ForesightHandler(BaseDailyLogHandler):
"""Cascade handler for ``users/<u>/.foresights/foresight-*.md``."""
@@ -68,7 +74,14 @@ class ForesightHandler(BaseDailyLogHandler):
text = s.sections.get("Foresight", "").strip()
evidence = (s.sections.get("Evidence") or "").strip() or None
tokens = self._deps.tokenizer.tokenize(text)
vector = await self._deps.embedder.embed(text)
vector = await get_embedding_capability().embed_or_none(text)
if vector is None:
logger.debug(
"cascade_handler_embed_skipped",
kind=self.kind,
entry_id=entry.entry_id,
reason="embedding_capability_unavailable",
)
evidence_tokens = (
" ".join(self._deps.tokenizer.tokenize(evidence)) if evidence else None
)
@@ -25,6 +25,8 @@ structure fields are excluded — they change on re-parse without
semantic drift.
Embedding source: ``summary`` (mirrors the search recaller's anchor).
Embedding is a soft dependency: when unavailable, ``vector`` is
written as ``None`` and the row stays BM25/scalar-searchable only.
"""
from __future__ import annotations
@@ -32,7 +34,9 @@ from __future__ import annotations
import json
from typing import Any, ClassVar
from everos.component.embedding import get_embedding_capability
from everos.component.utils.datetime import get_utc_now
from everos.core.observability.logging import get_logger
from everos.core.persistence import MarkdownReader, ParsedMarkdown
from everos.infra.persistence.lancedb import KnowledgeTopic, knowledge_topic_repo
from everos.infra.persistence.sqlite import (
@@ -45,6 +49,8 @@ from ._common import content_sha256 as compute_content_sha256
from ._common import resolve_scope
from .base import Handler
logger = get_logger(__name__)
class KnowledgeTopicHandler(Handler):
"""Cascade handler for
@@ -191,7 +197,14 @@ class KnowledgeTopicHandler(Handler):
content_tokens = " ".join(
self._deps.tokenizer.tokenize(fields["content"]),
)
vector = await self._deps.embedder.embed(fields["summary"])
vector = await get_embedding_capability().embed_or_none(fields["summary"])
if vector is None:
logger.debug(
"cascade_handler_embed_skipped",
kind=self.kind,
entry_id=fields["node_id"],
reason="embedding_capability_unavailable",
)
now = get_utc_now()
return KnowledgeTopic(
id=fields["node_id"],
+26 -10
View File
@@ -5,9 +5,11 @@ constructs it at startup, calls :meth:`start` once, and calls
:meth:`stop` at shutdown. CLI ``cascade sync`` constructs its own
instance but only invokes :meth:`drain_once` (no background tasks).
Construction is dependency-injected: the embedding / tokenizer
providers and the memory-root come in as constructor args so tests
can swap them without monkey-patching module-level singletons.
Construction is dependency-injected: the tokenizer provider and the
memory-root come in as constructor args so tests can swap them
without monkey-patching module-level singletons. Embedding is a soft
dependency handled directly by handlers via the capability accessor,
not threaded through here.
"""
from __future__ import annotations
@@ -15,7 +17,6 @@ from __future__ import annotations
import asyncio
import dataclasses
from everos.component.embedding import EmbeddingProvider
from everos.component.tokenizer import Tokenizer
from everos.core.observability.logging import get_logger
from everos.core.persistence import MemoryRoot
@@ -53,7 +54,6 @@ class CascadeOrchestrator:
self,
*,
memory_root: MemoryRoot,
embedder: EmbeddingProvider,
tokenizer: Tokenizer,
config: CascadeConfig | None = None,
) -> None:
@@ -61,7 +61,6 @@ class CascadeOrchestrator:
self._config = config or CascadeConfig()
deps = HandlerDeps(
memory_root=memory_root,
embedder=embedder,
tokenizer=tokenizer,
)
self._handlers = build_handlers(deps)
@@ -113,14 +112,31 @@ class CascadeOrchestrator:
self._started = False
logger.info("cascade_orchestrator_stopped")
async def sync_once(self) -> int:
"""One scan + drain cycle (used by CLI ``cascade sync``).
async def sync_once(self, *, kinds: set[str] | None = None) -> int:
"""One scan + drain cycle (used by CLI ``cascade sync`` and
Phase-3 backfill's post-write skill-file sync).
Args:
kinds: Optional restriction on which md kinds the scan
walks (e.g. ``{"agent_skill"}``). ``None`` — the default
for CLI ``cascade sync`` — scans every registered kind.
Phase 3 uses ``{"agent_skill"}`` so an unscoped sweep
doesn't tag unrelated kinds (notably ``knowledge_*``,
which the current process may not have handlers for)
as permanently failed.
Returns the number of rows processed in this drain. The CLI
loops on the returned count to know when to stop.
The ``kinds`` filter is threaded through **both** the scanner
AND the worker drain: the scanner scopes what gets enqueued,
the worker scopes what gets claimed. Round-1 wired only the
scanner side, so a scoped Phase-3 sync could still drain a
knowledge md queued in a prior tick and mark it permanently
failed — round-2 closes that end.
"""
await self._scanner.scan_once()
return await self._worker.drain_until_empty()
await self._scanner.scan_once(kinds=kinds)
return await self._worker.drain_until_empty(kinds=kinds)
async def drain_once(self) -> int:
"""Drain the queue exactly once without scanning first."""
+19
View File
@@ -173,5 +173,24 @@ def build_handlers(deps: HandlerDeps) -> dict[str, Handler]:
Returns a ``{kind_name: Handler}`` map used by the worker for
dispatch. Constructing once at orchestrator startup keeps the
per-row hot path free of factory churn.
Every handler registers unconditionally. Capability-dependent
steps live inside the handlers themselves:
- :class:`KnowledgeDocumentHandler` is SQLite-only; no capability
dependency.
- :class:`KnowledgeTopicHandler.handle_added_or_modified` calls
:func:`embed_or_none`, which writes ``vector=None`` when the
embedding capability is unavailable — the column has been
nullable since the embedding-soft-dependency migration.
``handle_deleted`` is pure SQL/LanceDB delete.
Prior versions gated the two knowledge handlers off as an atomic
pair when embed OR rerank was unavailable — that gate broke the
delete path (worker marks the row failed with "no handler
registered", stranding SQLite / LanceDB rows after
``shutil.rmtree`` cleared the md). Search-side gating is
enforced separately at the route level, so cascade does not need
to gate writes.
"""
return {spec.name: spec.handler_factory(deps) for spec in KIND_REGISTRY}
+29 -3
View File
@@ -72,17 +72,32 @@ class CascadeScanner:
self._task = None
logger.info("cascade_scanner_stopped")
async def scan_once(self) -> list[ReconcileDecision]:
async def scan_once(
self, *, kinds: set[str] | None = None
) -> list[ReconcileDecision]:
"""One scan + reconcile pass; returns the decisions that were
upserted into :class:`MdChangeState`.
Exposed so the CLI ``cascade sync`` command can trigger a sweep
without owning a long-lived scanner task.
Args:
kinds: Restrict the walk to specific md kinds (e.g.
``{"agent_skill"}``). ``None`` — the default — walks
every kind in :data:`KIND_REGISTRY`, matching the
background scanner loop's behaviour.
"""
scan_inputs = await asyncio.to_thread(
_collect_scan_inputs, self._memory_root.root
_collect_scan_inputs, self._memory_root.root, kinds
)
state = await _load_state_snapshot()
if kinds is not None:
# reconcile() emits ``deleted`` for every state row missing
# from the scan input — with a kind-scoped scan, the state
# snapshot must be scoped too, or non-``kinds`` rows would
# get spuriously deleted just because we didn't look at
# them this sweep.
state = {p: s for p, s in state.items() if s.kind in kinds}
decisions = reconcile(scan_inputs, state)
for decision in decisions:
await md_change_state_repo.upsert(
@@ -113,9 +128,18 @@ class CascadeScanner:
continue
def _collect_scan_inputs(root: Path) -> list[ScanInput]:
def _collect_scan_inputs(root: Path, kinds: set[str] | None = None) -> list[ScanInput]:
"""Walk ``root`` once per registered kind, returning every match.
``kinds`` — when non-``None`` — restricts the walk to specs whose
``name`` appears in the set; other kinds' md files are neither
stat'd nor emitted. Used by kind-scoped sync callers (e.g. Phase-3
backfill) so their sweep doesn't touch queue state for unrelated
kinds. The paired state-snapshot filter lives in :meth:`scan_once`
(reconcile itself is kind-agnostic) — both sides must be scoped or
reconcile would emit spurious ``deleted`` decisions for rows the
scan simply chose not to look at.
``stat()`` failure mode discrimination is **load-bearing**: the
reconciler treats "in state but not in scan" as a deletion signal,
so if we silently drop a path here under a *transient* OS error,
@@ -142,6 +166,8 @@ def _collect_scan_inputs(root: Path) -> list[ScanInput]:
"""
inputs: list[ScanInput] = []
for spec in KIND_REGISTRY:
if kinds is not None and spec.name not in kinds:
continue
for absolute in root.glob(spec.path_glob()):
try:
mtime = absolute.stat().st_mtime
+25 -4
View File
@@ -260,7 +260,7 @@ class CascadeWorker:
self._rebuild_task = None
logger.info("cascade_worker_stopped")
async def drain_once(self) -> int:
async def drain_once(self, *, kinds: set[str] | None = None) -> int:
"""Process one batch, return the number of rows handled.
Used by CLI ``cascade sync`` and ``fix --apply`` to flush the
@@ -276,8 +276,19 @@ class CascadeWorker:
merged before returning (CLI ``cascade sync``) call
:meth:`_flush_optimizers` — :meth:`drain_until_empty` does this
on their behalf.
Args:
kinds: Optional restriction on which ``kind`` values are
claimed. ``None`` (the default) claims across every
kind. Phase-3 backfill passes ``{"agent_skill"}`` so
the drain cannot flip an unrelated queued kind (e.g. a
queued ``knowledge_topic`` md whose handler this
process doesn't have registered) to
``failed(retryable=False)``.
"""
batch = await md_change_state_repo.claim_pending_batch(self._batch_size)
batch = await md_change_state_repo.claim_pending_batch(
self._batch_size, kinds=kinds
)
if not batch:
return 0
results = await asyncio.gather(*(self._process_one(row) for row in batch))
@@ -286,7 +297,12 @@ class CascadeWorker:
self._schedule_optimize(kind)
return len(batch)
async def drain_until_empty(self, *, max_passes: int = 100) -> int:
async def drain_until_empty(
self,
*,
max_passes: int = 100,
kinds: set[str] | None = None,
) -> int:
"""Drain repeatedly until the queue is empty (or ``max_passes``).
Returns the total number of rows processed. Bounded passes
@@ -298,10 +314,15 @@ class CascadeWorker:
(CLI ``cascade sync``) get a fully merged index — not for
visibility (the data is already searchable) but so ``sync``
returns a deterministically optimized state.
The ``kinds`` filter is forwarded to :meth:`drain_once` on every
pass so the scoping intent holds end-to-end for scoped syncs
(Phase-3 backfill's ``{"agent_skill"}`` sweep); ``None`` keeps
the CLI ``cascade sync`` path drainining every kind.
"""
total = 0
for _ in range(max_passes):
processed = await self.drain_once()
processed = await self.drain_once(kinds=kinds)
if processed == 0:
break
total += processed
+10 -1
View File
@@ -114,6 +114,15 @@ class ReflectionOrchestrator:
Returns:
List of successful ReflectionReport rows (typed as object
because the table class lives in infra).
Note:
No embedding-availability guard here — upstream callers are
responsible for skipping this run when the capability is
missing. In production the ``reflect_episodes`` strategy
body-guards on ``get_embedding_capability().available`` and
then constructs the orchestrator via ``.require()`` (which
raises before ``run()`` can be called if the provider is
missing), so an in-body check would be unreachable.
"""
candidates = await self._select_candidates(
owner_id=owner_id,
@@ -1013,7 +1022,7 @@ class ReflectionOrchestrator:
if is_deprecated and ep.md_path:
path_to_entries[ep.md_path][ep.entry_id] = merged_entry_id
root = MemoryRoot.default().root
root = MemoryRoot.resolve().root
for md_path, deprecated_map in path_to_entries.items():
await self._episode_writer.patch_frontmatter(
root / md_path,
+114 -38
View File
@@ -36,9 +36,12 @@ from everalgo.rank import DEFAULT_RANK_CONFIG, RankConfig, arank
from everalgo.rank.fusion import rrf
from everalgo.types import Candidate, RankInput
from everos.component.embedding import get_embedding_capability
from everos.component.rerank import get_rerank_capability
from everos.component.utils.datetime import to_display_tz
from everos.config import load_settings
from everos.core.context import resolve_request_id
from everos.core.errors import ProviderNotConfiguredError
from everos.core.observability.logging import get_logger
from everos.core.observability.tracing import (
capture_input,
@@ -755,51 +758,103 @@ class SearchManager:
# ── Component guards ────────────────────────────────────────────
def _validate_components(self, req: SearchRequest) -> None:
"""Fail fast when the chosen method needs components that are missing."""
"""Fail fast when the chosen method needs a provider that is missing.
Every provider branch (embedding, rerank, LLM) raises the same
:class:`ProviderNotConfiguredError` -> HTTP 422 with a toml-pointing
message; vocabulary is uniform across the three.
Embedding and rerank checks read BOTH the process-wide capability
singletons AND this manager's own injected providers. Either side
being "missing" is enough to fail the guard: accessor
says-unavailable means the tier never had it; ``self._embedding``
is ``None`` means whoever built this manager didn't wire one in.
Because dispatch reads ``self._embedding`` directly, letting an
accessor-says-available-but-self-is-None case through would
produce an opaque ``AttributeError('NoneType')`` deep in recall
instead of a clean 422. Single source of truth = both must agree
the provider exists.
Agent HYBRID has two rerank lanes selected by ``enable_llm_rerank``:
the LLM lane (``True``) reranks via the LLM and needs no rerank
provider; the cross-encoder lane (``False``, default) does.
Design contract (PR #361 round-3 review #6): this manager is
instantiated by ``service.search`` — a single caller that always
sources its providers from the same capability accessors this
guard reads. On that path DI-injected provider and accessor
state cannot disagree, so the belt-and-suspenders check is a
near-no-op; it only fires on pathological drift (a subclass or
a test that constructs a manager with mismatched inputs).
Library consumers that embed :class:`SearchManager` directly
and hand it their OWN provider will hit 422 even when their
provider is valid, because the accessor reflects the
process-wide toml. This is an accepted limitation of the
current DI shape — the guard prefers false-positives
(over-refuse) to false-negatives (dispatch with a ``None``
provider) because the latter surfaces as an opaque
``AttributeError('NoneType')`` deep in recall.
"""
method = req.method
needs_embedding = method != SearchMethod.KEYWORD
if needs_embedding and self._embedding is None:
raise RuntimeError(
f"method={method.value!r} requires an embedding provider; "
"configure [embedding] in settings"
)
# LLM is only mandatory when the caller explicitly opts into
# Phase-5 rerank on HYBRID, or always for AGENTIC (sufficiency
# check + multi-query generation).
if (
method == SearchMethod.HYBRID
and req.enable_llm_rerank
and self._llm is None
is_agent_hybrid = method == SearchMethod.HYBRID and req.owner_type == "agent"
needs_embedding = method in (
SearchMethod.VECTOR,
SearchMethod.HYBRID,
SearchMethod.AGENTIC,
)
if needs_embedding and (
not get_embedding_capability().available or self._embedding is None
):
raise RuntimeError(
"method='hybrid' with enable_llm_rerank=true needs an LLM; "
"configure [llm] in settings or drop the flag"
raise ProviderNotConfiguredError(
provider="embedding",
feature=_feature_name(method, req.owner_type),
)
# agent_skill HYBRID without LLM rerank reaches the cross-encoder
# lane; without the reranker it would AttributeError deep in the
# callback. Episode / agent_case HYBRID don't need it.
# agent HYBRID cross-encoder lane (enable_llm_rerank=False, the
# default) reaches ``search_agent_skills_hybrid``, which needs a
# real rerank provider; the LLM lane reranks via the LLM instead
# and is exempt.
if (
method == SearchMethod.HYBRID
and req.owner_type == "agent"
is_agent_hybrid
and not req.enable_llm_rerank
and self._reranker is None
and (not get_rerank_capability().available or self._reranker is None)
):
raise RuntimeError(
"owner_type='agent' with method='hybrid' requires a rerank "
"provider (skill cross-encoder lane); configure [rerank] in "
"settings, or set enable_llm_rerank=true to use the LLM lane"
raise ProviderNotConfiguredError(
provider="rerank",
feature="agent_hybrid",
alternative_hint=(
"Set enable_llm_rerank=true to route through the "
"LLM rerank lane instead of the cross-encoder "
"provider."
),
)
if method == SearchMethod.AGENTIC and (
not get_rerank_capability().available or self._reranker is None
):
raise ProviderNotConfiguredError(
provider="rerank", feature="agentic_search"
)
# LLM has no Capability wrapper yet; guard on the injected client
# directly. Belt-and-suspenders with the embedding/rerank branches
# above -- same 422 vocabulary via ProviderNotConfiguredError.
is_hybrid_llm_lane = method == SearchMethod.HYBRID and req.enable_llm_rerank
if is_hybrid_llm_lane and self._llm is None:
raise ProviderNotConfiguredError(
provider="llm",
feature=_feature_name(method, req.owner_type),
alternative_hint=(
"drop the enable_llm_rerank flag to use the cross-encoder "
"rerank lane (needs a configured [rerank] provider)"
),
)
if method == SearchMethod.AGENTIC and self._llm is None:
raise ProviderNotConfiguredError(
provider="llm",
feature=_feature_name(method, req.owner_type),
)
if method == SearchMethod.AGENTIC:
if self._reranker is None:
raise RuntimeError(
"method='agentic' requires a rerank provider; "
"configure [rerank] in settings"
)
if self._llm is None:
raise RuntimeError(
"method='agentic' requires an LLM client; "
"configure [llm] in settings"
)
def _scored_as_candidate(scored) -> Candidate: # type: ignore[no-untyped-def]
@@ -816,6 +871,27 @@ def _scored_as_candidate(scored) -> Candidate: # type: ignore[no-untyped-def]
)
def _feature_name(method: SearchMethod, owner_type: str) -> str:
"""Map a search method (+ owner partition) to its 422 ``feature`` tag.
HYBRID splits by ``owner_type`` because the episode path (embed only)
and the agent path (embed, plus a rerank lane) have different
prerequisites and deserve distinct log/response tags.
Note: this vocabulary (``"vector"``, ``"user_hybrid"``, ``"agent_hybrid"``,
``"agentic_search"``) is separate from the capability-level vocabulary
in :mod:`everos.component.capabilities`. See that module's docstring
for the rationale and contract.
"""
if method == SearchMethod.VECTOR:
return "vector"
if method == SearchMethod.HYBRID:
return "agent_hybrid" if owner_type == "agent" else "user_hybrid"
if method == SearchMethod.AGENTIC:
return "agentic_search"
return method.value
def _effective_llm_rerank(req: SearchRequest) -> bool:
"""LLM Phase-5 rerank only kicks in for ``HYBRID`` and only when the
caller opts in. ``AGENTIC`` runs its own cross-encoder rerank loop
@@ -44,7 +44,7 @@ _writer: AgentCaseWriter | None = None
def _get_writer() -> AgentCaseWriter:
global _writer
if _writer is None:
_writer = AgentCaseWriter(root=MemoryRoot.default())
_writer = AgentCaseWriter(root=MemoryRoot.resolve())
return _writer
@@ -39,11 +39,11 @@ from everalgo.types import AgentCase as AlgoAgentCase
from everalgo.types import AgentSkill as AlgoAgentSkill
from everos.component.embedding import (
EmbeddingNotConfiguredError,
EmbeddingServiceError,
get_embedder,
get_embedding_capability,
)
from everos.component.llm import get_llm_client
from everos.core.errors import ProviderNotConfiguredError
from everos.core.observability.logging import get_logger
from everos.core.persistence import MemoryRoot
from everos.infra.ome.context import StrategyContext
@@ -102,7 +102,7 @@ _writer: AgentSkillWriter | None = None
def _get_writer() -> AgentSkillWriter:
global _writer
if _writer is None:
_writer = AgentSkillWriter(root=MemoryRoot.default())
_writer = AgentSkillWriter(root=MemoryRoot.resolve())
return _writer
@@ -113,6 +113,25 @@ def _get_writer() -> AgentSkillWriter:
max_retries=3,
)
async def extract_agent_skill(event: SkillClusterUpdated, ctx: StrategyContext) -> None:
# Body-guard: capability is checked here for defensive degradation.
# Belt-and-suspenders even though the upstream
# trigger_skill_clustering already gates on the same capability — a
# direct emit of SkillClusterUpdated (tests, future features) should
# still degrade cleanly without an owner lock or OME retry pressure.
# Tier upgrades require a server restart; this guard is not a
# hot-reload mechanism.
#
# ``debug`` level (not ``info``) is intentional; see the body-guard
# in :func:`everos.memory.strategies.trigger_profile_clustering` for
# the shared rationale (PR #361 round-3 review #8).
if not get_embedding_capability().available:
logger.debug(
"strategy_gated_off_embedding_unavailable",
strategy_name="extract_agent_skill",
agent_id=event.agent_id,
)
return
# Serialise on agent_id: SKILL.md is addressed by (agent_id, skill_name)
# — concurrent runs across different clusters of the same agent can
# both decide to add the same skill_name and clobber the file. Different
@@ -269,10 +288,16 @@ async def _resolve_query_vector(target: LanceAgentCase) -> list[float]:
return list(target.vector)
if not target.task_intent:
return []
# ``.require()`` is defensive: the strategy body-guard checks
# ``.available`` before we reach this helper, so this branch will
# not raise ``ProviderNotConfiguredError`` in normal operation. The
# catch stays so a misconfiguration surfacing later (or a direct
# unit-test call to this helper without the guard) still degrades
# to ``[]`` instead of blowing up mid-strategy.
try:
embedder = get_embedder()
embedder = get_embedding_capability().require()
return list(await embedder.embed(target.task_intent))
except (EmbeddingNotConfiguredError, EmbeddingServiceError) as exc:
except (ProviderNotConfiguredError, EmbeddingServiceError) as exc:
logger.warning(
"agent_skill_query_embed_failed",
case_entry_id=target.entry_id,
@@ -31,7 +31,7 @@ def _get_writer() -> AtomicFactWriter:
"""Return the lazily-initialised AtomicFactWriter singleton."""
global _writer
if _writer is None:
_writer = AtomicFactWriter(root=MemoryRoot.default())
_writer = AtomicFactWriter(root=MemoryRoot.resolve())
return _writer
@@ -38,7 +38,7 @@ _writer: ForesightWriter | None = None
def _get_writer() -> ForesightWriter:
global _writer
if _writer is None:
_writer = ForesightWriter(root=MemoryRoot.default())
_writer = ForesightWriter(root=MemoryRoot.resolve())
return _writer
@@ -1,15 +1,34 @@
"""extract_user_profile strategy — synthesise the user's profile from clusters.
Listens to :class:`ProfileClusterUpdated` (fired after
``trigger_profile_clustering`` assigns a memcell to a cluster), pulls
the relevant memcells across all "fresh" clusters, and runs
:class:`everalgo.user_memory.ProfileExtractor` to INIT / UPDATE the
user's profile markdown.
Dual-trigger strategy: profile extraction must run whether or not embedding
is configured, so it listens on **two** events and gates itself so exactly
one path fires per memcell (see :func:`_profile_applies`):
- **Cluster path** (Tier 2+, embedding available): fires on
:class:`ProfileClusterUpdated`, emitted by ``trigger_profile_clustering``
after it assigns a memcell to a cluster. Pulls the relevant memcells
across all "fresh" clusters (:func:`_select_via_cluster`).
- **Direct path** (Tier 1, no embedding): fires on :class:`EpisodeExtracted`
directly. ``trigger_profile_clustering`` is always registered, but its
per-dispatch body-guard returns early when
:func:`get_embedding_capability` reports unavailable, so no
``ProfileClusterUpdated`` is ever emitted in Tier 1 — this direct
path is the sole route to profile extraction there. Pulls memcells
via a scalar LanceDB timestamp filter, no embedding required
(:func:`_select_via_timestamp`).
Both paths converge on the same LLM extraction + markdown persist tail of
``extract_user_profile`` — only memcell *selection* differs.
Opensource parity (``mem_memorize.py`` Phase 2):
- **Throttle**: ``total_memcell_count % profile_extraction_interval == 0``;
default interval = 1 (every memcell triggers a re-extraction).
- **Throttle** (both paths): ``total_count % profile_extraction_interval
== 0``; default interval = 1 (every memcell triggers a re-extraction).
Applied at strategy entry so cluster and direct paths honor the same
gate. ``total_count`` is ``sum(c.count for c in user_clusters)`` on the
cluster path and :meth:`episode_repo.count_by_owner` on the direct
path — both express "cumulative units of source-memory for this owner"
and stay ~1:1 in the extract → memcell → episode → cluster pipeline.
- **Target clusters**: every cluster whose ``last_ts`` is newer than the
user's existing profile timestamp, plus the current cluster (so the
freshly-arrived memcell is always counted even when its cluster's
@@ -27,15 +46,18 @@ profile in a follow-up turn (each cluster gets re-evaluated on every
from __future__ import annotations
from everalgo.clustering import Cluster as AlgoCluster
from everalgo.types import MemCell as AlgoMemCell
from everalgo.types import Profile as AlgoProfile
from everalgo.user_memory import ProfileExtractor
from everos.component.embedding import get_embedding_capability
from everos.component.llm import get_llm_client
from everos.core.observability.logging import get_logger
from everos.core.persistence import MemoryRoot
from everos.infra.ome.context import StrategyContext
from everos.infra.ome.decorator import offline_strategy
from everos.infra.ome.events import BaseEvent
from everos.infra.ome.triggers import Immediate
from everos.infra.persistence.lancedb import episode_repo
from everos.infra.persistence.markdown import (
@@ -45,7 +67,7 @@ from everos.infra.persistence.markdown import (
)
from everos.infra.persistence.sqlite import cluster_repo, memcell_repo
from everos.memory._partition_locks import get_partition_lock
from everos.memory.events import ProfileClusterUpdated
from everos.memory.events import EpisodeExtracted, ProfileClusterUpdated
logger = get_logger(__name__)
@@ -66,39 +88,185 @@ _reader: ProfileReader | None = None
def _get_writer() -> ProfileWriter:
global _writer
if _writer is None:
_writer = ProfileWriter(root=MemoryRoot.default())
_writer = ProfileWriter(root=MemoryRoot.resolve())
return _writer
def _get_reader() -> ProfileReader:
global _reader
if _reader is None:
_reader = ProfileReader(root=MemoryRoot.default())
_reader = ProfileReader(root=MemoryRoot.resolve())
return _reader
def _profile_applies(event: BaseEvent) -> bool:
"""Route exactly one path per memcell to ``extract_user_profile``.
- :class:`ProfileClusterUpdated` always applies: it is only ever
emitted by ``trigger_profile_clustering``, whose per-dispatch
body-guard short-circuits (returns before emit) whenever
:func:`get_embedding_capability` reports unavailable. Accepting
this event unconditionally is therefore safe — it can only arrive
when embedding is available, so it never overlaps with the direct
path below.
- :class:`EpisodeExtracted` applies only for pipeline-sourced
episodes (not Reflection's merged episodes) while embedding is
unavailable. Once embedding is available, ``trigger_profile_clustering``
runs to completion and owns the same memcell via
``ProfileClusterUpdated`` — so the direct path must stand down
here to avoid a double extraction. This check is a live capability
read (not a registration-time snapshot), so a Tier 3→1 downgrade
mid-process is picked up on the next event.
"""
if isinstance(event, ProfileClusterUpdated):
return True
if isinstance(event, EpisodeExtracted):
return event.source == "pipeline" and not get_embedding_capability().available
return False
async def _select_via_cluster(
event: ProfileClusterUpdated,
last_profile_ts: int,
user_clusters: list[AlgoCluster],
) -> list[str]:
"""Cluster path (Tier 2+): resolve memcells via the user's cluster set.
The caller (:func:`extract_user_profile`) already fetched
``user_clusters`` for the strategy-entry throttle; we take it as a
parameter to avoid a redundant :meth:`cluster_repo.list_for_owner`
round-trip. Behavior of the selection step itself is unchanged from
the pre-dual-trigger implementation.
"""
# Pick clusters fresher than the existing profile (always include
# the one we just updated).
target_clusters = [
c
for c in user_clusters
if c.last_ts > last_profile_ts or c.id == event.cluster_id
]
if not target_clusters:
return []
# Resolve cluster members (episode entry_ids) → memcell_ids.
# Cluster members store episode entry_ids. To reach the memcell
# payloads we look up each episode's parent_id (= memcell_id)
# in LanceDB, skipping merged episodes (parent_type=cluster)
# whose source memcells were already processed before Reflection.
entry_ids = [m for c in target_clusters for m in c.members]
episodes = await episode_repo.find_by_owner_entries(
event.owner_id,
entry_ids,
app_id=event.app_id,
project_id=event.project_id,
)
return [
ep.parent_id for ep in episodes if ep.parent_type == "memcell" and ep.parent_id
]
async def _select_via_timestamp(
event: EpisodeExtracted, last_profile_ts: int
) -> list[str]:
"""Direct path (Tier 1): resolve memcells from the event + a LanceDB supplement.
Always includes the current event's ``memcell_id`` — it is the
trigger for this run and its memcell is the one whose profile we
care about. The LanceDB scan is a best-effort supplement that adds
older memcells still eligible under ``last_profile_ts``; the cascade
daemon may not have indexed the freshly-arrived row yet (it runs on
its own schedule), so relying on LanceDB alone would miss the first
memory of a fresh Tier-1 install entirely. See
:class:`EpisodeExtracted` for the event-first contract that lets us
dodge this race — it carries ``memcell_id`` precisely so downstream
strategies do not need to poll LanceDB until the row appears.
Mirrors the cluster path's ``c.id == event.cluster_id`` fallback
(see :func:`_select_via_cluster`): the currently-firing entity is
always in the candidate set even when its timestamp is not strictly
fresher than the last profile — this matters for bulk imports where
every incoming memcell may carry a historical timestamp.
Order is unstable across runs (the return is a de-duplicated set
materialised to a list); downstream ``memcell_repo.find_by_ids``
reorders back to the caller's list order, so this does not
destabilise later stages.
"""
memcell_ids: set[str] = {event.memcell_id}
# Column projection: this selector only needs `parent_id` to seed the
# memcell fetch — pulling full rows would drag the two 1024-D vector
# columns across the wire for every historical episode. Raw-dict
# return is contractual when `columns` is set.
supplement = await episode_repo.list_by_owner_after_ts(
owner_id=event.owner_id,
after_ts=last_profile_ts,
parent_type="memcell",
app_id=event.app_id,
project_id=event.project_id,
columns=["parent_id"],
)
memcell_ids.update(row["parent_id"] for row in supplement if row.get("parent_id"))
return list(memcell_ids)
@offline_strategy(
name="extract_user_profile",
trigger=Immediate(on=[ProfileClusterUpdated]),
trigger=Immediate(on=[ProfileClusterUpdated, EpisodeExtracted]),
applies_to=_profile_applies,
emits=[],
max_retries=2,
)
async def extract_user_profile(
event: ProfileClusterUpdated, ctx: StrategyContext
event: ProfileClusterUpdated | EpisodeExtracted, ctx: StrategyContext
) -> None:
# Serialise on owner_id: user.md is a single per-user file and the
# body is a read → LLM merge → overwrite sequence. Different users
# run fully in parallel.
partition = f"{event.app_id}:{event.project_id}:{event.owner_id}"
async with get_partition_lock("extract_user_profile", partition):
# 1. Throttle: skip unless the Nth clustered memcell tick lands here.
user_clusters = await cluster_repo.list_for_owner(
existing = await _get_reader().read(
event.owner_id,
"user_memory",
schema=UserProfileFrontmatter,
app_id=event.app_id,
project_id=event.project_id,
)
total_count = sum(c.count for c in user_clusters)
last_profile_ts = existing[0].profile_timestamp_ms if existing else 0
# Unified throttle: both paths gate on "cumulative units of
# source-memory for this owner". Cluster path sums cluster
# counts (pre-refactor semantics); direct path counts owner
# episode rows in LanceDB. Applied here so a bumped interval
# (e.g. 10x to cap LLM cost) throttles both Tier 1 and Tier 2+
# equally instead of only the cluster path.
user_clusters: list[AlgoCluster] | None
if isinstance(event, ProfileClusterUpdated):
user_clusters = await cluster_repo.list_for_owner(
event.owner_id,
"user_memory",
app_id=event.app_id,
project_id=event.project_id,
)
total_count = sum(c.count for c in user_clusters)
else:
user_clusters = None
# Scope the counter to parent_type='memcell' so it matches
# _select_via_timestamp's selector below — otherwise Reflection-
# merged rows (parent_type='cluster') inflate the throttle count
# without ever being selectable, firing the gate at the wrong cadence.
# TODO(profile-counter): reads LanceDB and therefore races the
# cascade daemon in the same way _select_via_timestamp used to
# (fresh Tier-1 install → count=0 until cascade catches up).
# The throttle only needs a monotonic per-owner integer, so a
# stale-but-monotonic value is acceptable for now; the followup
# is to source this from a sqlite ``memcell`` count-by-owner
# query — see PR #361 review finding M4.
total_count = await episode_repo.count_by_owner(
event.owner_id,
app_id=event.app_id,
project_id=event.project_id,
parent_type="memcell",
)
if (
PROFILE_EXTRACTION_INTERVAL > 1
and total_count % PROFILE_EXTRACTION_INTERVAL != 0
@@ -108,43 +276,24 @@ async def extract_user_profile(
owner_id=event.owner_id,
total_count=total_count,
interval=PROFILE_EXTRACTION_INTERVAL,
path="cluster"
if isinstance(event, ProfileClusterUpdated)
else "direct",
)
return
# 2. Pick clusters fresher than the existing profile (always include
# the one we just updated).
existing = await _get_reader().read(
event.owner_id,
schema=UserProfileFrontmatter,
app_id=event.app_id,
project_id=event.project_id,
)
last_profile_ts = existing[0].profile_timestamp_ms if existing else 0
target_clusters = [
c
for c in user_clusters
if c.last_ts > last_profile_ts or c.id == event.cluster_id
]
if not target_clusters:
return
# Memcell selection is the only part that differs by trigger; the
# LLM extraction + persist tail below is shared by both paths.
# user_clusters is always populated on the cluster branch above,
# so the fallback to [] here is unreachable — it exists purely to
# satisfy the type checker without an assert.
if isinstance(event, ProfileClusterUpdated):
memcell_ids = await _select_via_cluster(
event, last_profile_ts, user_clusters or []
)
else:
memcell_ids = await _select_via_timestamp(event, last_profile_ts)
# 3. Resolve cluster members (episode entry_ids) → memcell_ids.
# Cluster members store episode entry_ids. To reach the memcell
# payloads we look up each episode's parent_id (= memcell_id)
# in LanceDB, skipping merged episodes (parent_type=cluster)
# whose source memcells were already processed before Reflection.
entry_ids = [m for c in target_clusters for m in c.members]
episodes = await episode_repo.find_by_owner_entries(
event.owner_id,
entry_ids,
app_id=event.app_id,
project_id=event.project_id,
)
memcell_ids = [
ep.parent_id
for ep in episodes
if ep.parent_type == "memcell" and ep.parent_id
]
if len(memcell_ids) < PROFILE_MIN_MEMCELLS:
logger.info(
"profile_extraction_below_min_memcells",
@@ -154,7 +303,7 @@ async def extract_user_profile(
)
return
# 4. Pull memcell payloads from SQLite, rehydrate to algo types.
# Pull memcell payloads from SQLite, rehydrate to algo types.
memcell_rows = await memcell_repo.find_by_ids(memcell_ids)
algo_memcells = sorted(
(AlgoMemCell.model_validate_json(r.payload_json) for r in memcell_rows),
@@ -163,14 +312,14 @@ async def extract_user_profile(
if not algo_memcells:
return
# 5. Run the LLM extractor — INIT (no prior) or UPDATE (existing).
# Run the LLM extractor — INIT (no prior) or UPDATE (existing).
old_profile = _to_algo_profile(existing[0]) if existing else None
extractor = ProfileExtractor(llm=get_llm_client())
new_profile = await extractor.aextract(
algo_memcells, sender_id=event.owner_id, old_profile=old_profile
)
# 6. Write the fresh profile back to users/<user_id>/user.md.
# Write the fresh profile back to users/<user_id>/user.md.
await _persist_profile(
new_profile,
owner_id=event.owner_id,
@@ -180,8 +329,7 @@ async def extract_user_profile(
logger.info(
"user_profile_extracted",
owner_id=event.owner_id,
cluster_count=len(target_clusters),
episode_count=len(episodes),
path="cluster" if isinstance(event, ProfileClusterUpdated) else "direct",
memcell_count=len(algo_memcells),
mode="UPDATE" if old_profile is not None else "INIT",
)
@@ -14,7 +14,7 @@ from __future__ import annotations
import asyncio
from everos.component.embedding import get_embedder
from everos.component.embedding import get_embedding_capability
from everos.component.llm import get_llm_client
from everos.core.observability.logging import get_logger
from everos.core.persistence import MemoryRoot
@@ -43,7 +43,7 @@ def _get_episode_writer() -> EpisodeWriter:
"""Return the lazily-initialised EpisodeWriter singleton."""
global _episode_writer
if _episode_writer is None:
_episode_writer = EpisodeWriter(root=MemoryRoot.default())
_episode_writer = EpisodeWriter(root=MemoryRoot.resolve())
return _episode_writer
@@ -61,6 +61,22 @@ async def reflect_episodes(event: CronTick, ctx: StrategyContext) -> None:
event: Cron tick event (unused; triggers the scheduled run).
ctx: OME strategy context for emit and logging.
"""
# Body-guard: capability is checked here for defensive degradation.
# Reflection re-embeds merged narratives, so it cannot run without
# an embedder — silently no-op instead of raising deep inside the
# orchestrator. Tier upgrades require a server restart; this guard
# is not a hot-reload mechanism.
#
# ``debug`` level (not ``info``) is intentional; see the body-guard
# in :func:`everos.memory.strategies.trigger_profile_clustering` for
# the shared rationale (PR #361 round-3 review #8).
if not get_embedding_capability().available:
logger.debug(
"strategy_gated_off_embedding_unavailable",
strategy_name="reflect_episodes",
)
return
# Deferred: avoid pulling LLM libs at module import time.
from everalgo.user_memory import EpisodeReflector
@@ -71,7 +87,7 @@ async def reflect_episodes(event: CronTick, ctx: StrategyContext) -> None:
episode_writer=_get_episode_writer(),
report_repo=reflection_report_repo,
reflector=EpisodeReflector(llm=get_llm_client()),
embedder=get_embedder(),
embedder=get_embedding_capability().require(),
)
owners = await cluster_repo.list_distinct_owners()
@@ -14,7 +14,7 @@ import numpy as np
from everalgo.clustering import Cluster as AlgoCluster
from everalgo.clustering import cluster_by_geometry
from everos.component.embedding import get_embedder
from everos.component.embedding import get_embedding_capability
from everos.config import load_settings
from everos.core.observability.logging import get_logger
from everos.infra.ome.context import StrategyContext
@@ -37,6 +37,36 @@ logger = get_logger(__name__)
async def trigger_profile_clustering(
event: EpisodeExtracted, ctx: StrategyContext
) -> None:
# Body-guard: capability is checked here for defensive degradation.
# When embedding is unavailable we cannot vectorise the episode, so
# the strategy silently no-ops — no work, no owner lock, no OME
# retry pressure. Tier upgrades require a server restart; this
# guard is not a hot-reload mechanism, it just keeps dispatch clean
# when capability was absent at engine start.
#
# Log level is intentionally ``debug`` here (and at the three sibling
# body-guards in ``trigger_skill_clustering``, ``extract_agent_skill``,
# and ``reflect_episodes``) rather than the ``info`` used by
# ``cascade.registry`` and the reflection orchestrator's
# ``cascade_registry_knowledge_gated_off`` /
# ``reflection_orchestrator_disabled`` events. Rationale (PR #361
# round-3 review #8, accepted intentional asymmetry): the
# registry/orchestrator gate events fire ONCE per process at startup
# and are useful lifecycle signals worth ``info``. The per-dispatch
# body-guards fire on EVERY memorize/reflection dispatch under Tier 1
# — a busy chat session emits hundreds. At ``info`` they would flood
# structured-log consumers even when the process-wide default level
# is ``WARNING`` (which suppresses stdout but not the sink). ``debug``
# keeps this per-event signal cheap and opt-in via ``--verbose`` for
# diagnosing why a strategy isn't running.
if not get_embedding_capability().available:
logger.debug(
"strategy_gated_off_embedding_unavailable",
strategy_name="trigger_profile_clustering",
owner_id=event.owner_id,
)
return
# Serialise on owner_id: the strategy reads the user's full cluster
# set, picks merge target by geometry, then upserts — concurrent runs
# on the same owner_id would race the read → decide → write cycle.
@@ -46,7 +76,12 @@ async def trigger_profile_clustering(
partition = f"{event.app_id}:{event.project_id}:{event.owner_id}"
async with get_partition_lock("trigger_profile_clustering", partition):
# 1. Embed the episode_text into a vector.
vector_list = await get_embedder().embed(event.episode_text)
# ``.require()`` is defensive: the body-guard above already
# returned when the capability was missing, so this cannot raise
# in the guarded path. Routing through the capability keeps a
# single shared provider (one client, one semaphore) per process.
embedder = get_embedding_capability().require()
vector_list = await embedder.embed(event.episode_text)
vector = np.asarray(vector_list, dtype=np.float32)
# 2. Load this user's existing user-memory clusters (scoped to space).
@@ -23,7 +23,7 @@ import numpy as np
from everalgo.clustering import Cluster as AlgoCluster
from everalgo.clustering import cluster_by_llm
from everos.component.embedding import get_embedder
from everos.component.embedding import get_embedding_capability
from everos.component.llm import get_llm_client
from everos.core.observability.logging import get_logger
from everos.infra.ome.context import StrategyContext
@@ -49,6 +49,23 @@ cases below this never produce a skill, so we don't bother clustering them."""
async def trigger_skill_clustering(
event: AgentCaseExtracted, ctx: StrategyContext
) -> None:
# Body-guard: capability is checked here for defensive degradation.
# When embedding is unavailable we cannot vectorise the case, so
# the strategy silently no-ops — no work, no agent lock, no OME
# retry pressure. Tier upgrades require a server restart; this
# guard is not a hot-reload mechanism.
#
# ``debug`` level (not ``info``) is intentional; see the body-guard
# in :func:`everos.memory.strategies.trigger_profile_clustering` for
# the shared rationale (PR #361 round-3 review #8).
if not get_embedding_capability().available:
logger.debug(
"strategy_gated_off_embedding_unavailable",
strategy_name="trigger_skill_clustering",
agent_id=event.agent_id,
)
return
# Serialise on agent_id: the strategy reads the agent's full cluster
# set, lets the LLM decide merge vs. mint, then upserts — concurrent
# runs on the same agent_id would race the read → decide → write
@@ -67,7 +84,12 @@ async def trigger_skill_clustering(
return
# 2. Embed the case's task_intent into a vector.
vector_list = await get_embedder().embed(event.task_intent)
# ``.require()`` is defensive: the body-guard above already
# returned when the capability was missing, so this cannot raise
# in the guarded path. Routing through the capability keeps a
# single shared provider (one client, one semaphore) per process.
embedder = get_embedding_capability().require()
vector_list = await embedder.embed(event.task_intent)
vector = np.asarray(vector_list, dtype=np.float32)
# 3. Load this agent's existing skill clusters (scoped to space).
+76 -68
View File
@@ -29,13 +29,16 @@ import anyio
from everalgo.rank.fusion import rrf
from everalgo.types import Candidate, CategorySpec, KnowledgeMemory, ParsedContent
from everos.component.embedding import get_embedding_capability
from everos.component.rerank import get_rerank_capability
from everos.component.utils.datetime import get_utc_now
from everos.core.errors import (
ConfigurationError,
DocumentNotFoundError,
DuplicateDocumentError,
ExtractionEmptyError,
InvalidInputError,
PathTraversalError,
ProviderNotConfiguredError,
TopicNotFoundError,
)
from everos.core.observability.logging import get_logger
@@ -66,6 +69,7 @@ _DOC_ID_HEX_LEN = 12
_MAX_MINT_RETRIES = 5
_SCOPE_ID_PATTERN = re.compile(r"^[a-zA-Z0-9_.\-@+]+$")
_ORIGINAL_DIR_NAME = "_original"
_KNOWLEDGE_FEATURE = "knowledge"
class KnowledgeExtractor(Protocol):
@@ -519,7 +523,7 @@ async def delete_document(
topic_count = await knowledge_topic_sqlite_repo.count_by_doc_id(doc_id)
memory_root = MemoryRoot.default()
memory_root = MemoryRoot.resolve()
doc_dir = memory_root.root / Path(row.md_path).parent
if await anyio.Path(doc_dir).is_dir():
await anyio.to_thread.run_sync(shutil.rmtree, doc_dir)
@@ -605,7 +609,7 @@ async def _backup_doc_dir(doc_id: str) -> tuple[Path, Path] | None:
without reverse-engineering the name, or ``None`` when no directory
exists.
"""
memory_root = MemoryRoot.default()
memory_root = MemoryRoot.resolve()
row = await knowledge_document_repo.get_by_doc_id(doc_id)
if row is None:
return None
@@ -844,7 +848,7 @@ async def patch_document(
Raises:
DocumentNotFoundError: When neither SQLite nor md files contain ``doc_id``.
"""
memory_root = MemoryRoot.default()
memory_root = MemoryRoot.resolve()
current = await _resolve_current_doc(doc_id, app_id, project_id, memory_root)
new_title = title if title is not None else current.title
@@ -883,7 +887,7 @@ async def list_categories(app_id: str, project_id: str) -> list[CategoryOverview
Returns:
List of CategoryOverview with document counts from SQLite.
"""
knowledge_dir = MemoryRoot.default().knowledge_dir(app_id, project_id)
knowledge_dir = MemoryRoot.resolve().knowledge_dir(app_id, project_id)
await ensure_taxonomy(knowledge_dir)
specs = await parse_taxonomy(knowledge_dir / ".taxonomy.md")
counts = await knowledge_document_repo.count_by_category(app_id, project_id)
@@ -923,6 +927,18 @@ def _apply_category_fallback(
return patched
_MAX_FILENAME_BYTES = 255
"""Portable filesystem filename byte-length ceiling.
macOS HFS+/APFS, most Linux filesystems (ext4/xfs/btrfs) and Windows
NTFS all cap a single directory entry at 255 bytes of UTF-8. Rather
than let the OS surface the ceiling as a raw ``OSError[ENAMETOOLONG]``
at ``write_bytes`` time (which reaches the HTTP layer as a 500), we
validate up front and raise a caller-friendly ``InvalidInputError``
that maps to 400.
"""
def _safe_original_filename(source_name: str) -> str:
"""Reduce an untrusted upload filename to a single safe path component.
@@ -936,6 +952,18 @@ def _safe_original_filename(source_name: str) -> str:
``.`` / ``..`` / empty cases (which a basename can still be, e.g. a
filename of ``..``) are rejected outright.
Also validates two failure modes the OS would otherwise raise as
500 during ``write_bytes`` (round-4 review M1):
- **NUL byte** in the filename → ``ValueError: embedded null
character`` from the syscall layer. Rejected up front as a
400-mapped ``InvalidInputError``.
- **Filename > 255 bytes UTF-8** → ``OSError[ENAMETOOLONG]`` from
the syscall layer. Same up-front rejection.
Both are validated **before** ``KnowledgeWriter.write`` runs, so a
doomed upload never leaves half-a-document on disk.
Args:
source_name: The client-supplied multipart filename.
@@ -944,7 +972,13 @@ def _safe_original_filename(source_name: str) -> str:
Raises:
PathTraversalError: If no safe filename can be derived.
InvalidInputError: If the filename contains a NUL byte or
exceeds the filesystem's byte-length ceiling.
"""
if "\x00" in source_name:
raise InvalidInputError(
f"upload filename must not contain a NUL byte (got {source_name!r})"
)
# PurePosixPath does not split on "\\"; strip backslash segments too so a
# Windows-style ``..\\..\\x`` or ``C:\\x`` filename cannot survive.
name = PurePosixPath(source_name).name.rsplit("\\", 1)[-1]
@@ -952,6 +986,12 @@ def _safe_original_filename(source_name: str) -> str:
raise PathTraversalError(
f"upload filename is not a valid file component: {source_name!r}"
)
if len(name.encode("utf-8")) > _MAX_FILENAME_BYTES:
raise InvalidInputError(
"upload filename exceeds the "
f"{_MAX_FILENAME_BYTES}-byte filesystem ceiling "
f"(got {len(name.encode('utf-8'))} bytes)"
)
return name
@@ -972,7 +1012,7 @@ async def _resolve_original_file_path(
safe_name = _safe_original_filename(source_name)
except PathTraversalError:
return None
memory_root = MemoryRoot.default()
memory_root = MemoryRoot.resolve()
doc_dir = memory_root.root / Path(md_path).parent
candidate = doc_dir / _ORIGINAL_DIR_NAME / safe_name
if await anyio.Path(candidate).is_file():
@@ -1007,37 +1047,21 @@ async def _write_original_file(
# ── Knowledge search ─────────────────────────────────────────────────────────
# Lazy singleton — mirrors the pattern in service/search.py.
_embedding: EmbeddingProvider | None = None
_embedding_resolved = False
def _get_embedding() -> EmbeddingProvider | None:
"""Build the embedding client on first call. ``None`` when not configured."""
global _embedding, _embedding_resolved
if _embedding_resolved:
return _embedding
"""Return the process-wide embedding provider, or ``None`` when unset.
from everos.component.embedding import ( # Deferred: singleton
build_embedding_provider,
)
from everos.config import load_settings # Deferred: singleton
cfg = load_settings().embedding
if not cfg.model or cfg.api_key is None:
logger.warning(
"knowledge_embedding_not_configured",
hint="set [embedding] model / api_key to enable vector / hybrid search",
)
_embedding = None
else:
_embedding = build_embedding_provider(cfg)
logger.info("knowledge_embedding_built", model=cfg.model)
_embedding_resolved = True
return _embedding
Delegates to :func:`get_embedding_capability` instead of keeping a
parallel module-level singleton — mirrors ``service/search.py``, so
this module answers the same "is embedding configured?" question as
the health endpoint and the cascade write path.
"""
return get_embedding_capability().provider
# Lazy singleton — mirrors the pattern for _embedding above.
# Lazy singleton — the recaller itself has no soft-dependency provider to
# delegate to (it wraps a tokenizer, not embedding/rerank), so it keeps its
# own module-level cache rather than a capability accessor.
_recaller: KnowledgeTopicRecaller | None = None
_recaller_resolved = False
@@ -1061,30 +1085,12 @@ def _build_recaller() -> KnowledgeTopicRecaller:
return _recaller
# Lazy singleton — mirrors the pattern for _embedding above.
_reranker: RerankProvider | None = None
_reranker_resolved = False
def _get_reranker() -> RerankProvider | None:
"""Build the rerank client on first call. ``None`` when not configured."""
global _reranker, _reranker_resolved
if _reranker_resolved:
return _reranker
"""Return the process-wide rerank provider, or ``None`` when unset.
from everos.component.rerank import ( # Deferred: singleton
build_rerank_provider,
)
from everos.config import load_settings # Deferred: singleton
cfg = load_settings().rerank
if not cfg.model or not cfg.base_url:
_reranker = None
else:
_reranker = build_rerank_provider(cfg)
logger.info("knowledge_reranker_built", model=cfg.model, provider=cfg.provider)
_reranker_resolved = True
return _reranker
Delegates to :func:`get_rerank_capability` — mirrors :func:`_get_embedding`.
"""
return get_rerank_capability().provider
# ── Search result types ──────────────────────────────────────────────────────
@@ -1261,23 +1267,25 @@ async def _to_search_hits(
def _require_search_providers() -> tuple[EmbeddingProvider, RerankProvider]:
"""Return embedding + reranker providers, raising if not configured.
The HTTP router (``entrypoints.api.routes.knowledge``) already gates
both capabilities via ``Depends(_require_knowledge_capabilities)``, so
this check is normally redundant for HTTP callers -- it exists for
any non-HTTP caller of :func:`search_knowledge` that bypasses the
router dependency.
Raises:
ConfigurationError: When the embedding or rerank provider is not
configured (a required setting is missing).
ProviderNotConfiguredError: When the embedding or rerank provider
is not configured (-> HTTP 422 when it does surface via HTTP).
"""
embedder = _get_embedding()
if embedder is None:
raise ConfigurationError(
"Embedding provider not configured. "
"Set EVEROS_EMBEDDING__MODEL and EVEROS_EMBEDDING__API_KEY."
embed = _get_embedding()
if embed is None:
raise ProviderNotConfiguredError(
provider="embedding", feature=_KNOWLEDGE_FEATURE
)
reranker = _get_reranker()
if reranker is None:
raise ConfigurationError(
"Rerank provider not configured. "
"Set EVEROS_RERANK__MODEL and EVEROS_RERANK__BASE_URL."
)
return embedder, reranker
rerank = _get_reranker()
if rerank is None:
raise ProviderNotConfiguredError(provider="rerank", feature=_KNOWLEDGE_FEATURE)
return embed, rerank
async def _run_category_pipeline(
+45 -13
View File
@@ -31,7 +31,6 @@ from pydantic import BaseModel
from everos.component.llm import get_llm_client
from everos.config import load_settings
from everos.core.context import resolve_request_id
from everos.core.observability.logging import get_logger
from everos.core.observability.tracing import memory_span
from everos.core.persistence import MemoryRoot
from everos.infra.ome.config import OMEConfig
@@ -56,8 +55,6 @@ from everos.memory.strategies import (
from everos.service._boundary import prepare_cells
from everos.service._session_lock import get_session_lock
logger = get_logger(__name__)
class MemorizeResult(BaseModel):
"""What memorize returns to the caller (route serialises it)."""
@@ -85,7 +82,7 @@ def _config_root() -> Path:
def _get_episode_writer() -> EpisodeWriter:
global _episode_writer
if _episode_writer is None:
_episode_writer = EpisodeWriter(MemoryRoot.default())
_episode_writer = EpisodeWriter(MemoryRoot.resolve())
return _episode_writer
@@ -115,14 +112,55 @@ def _get_agent_pipeline() -> AgentMemoryPipeline:
return _agent_pipeline
_STRATEGIES_ALWAYS = (
extract_atomic_facts,
extract_foresight,
extract_agent_case,
extract_user_profile,
)
"""LLM-only strategies — no embed dependency, always registered."""
_STRATEGIES_REQUIRE_EMBED = (
trigger_skill_clustering,
extract_agent_skill,
trigger_profile_clustering,
reflect_episodes,
)
"""Strategies whose body re-embeds or consumes a re-embedded cluster.
All are registered unconditionally; each guards its own body via
:func:`everos.component.embedding.get_embedding_capability` at execution
time. When capability is unavailable the strategy no-ops (logged at
debug level) rather than raising deep inside its body.
Why body-guard: the capability singleton at
:mod:`everos.component.embedding.accessor` is process-wide and never
invalidated at runtime — a tier upgrade (Tier 1 → Tier 2 by editing
everos.toml) is applied only after a server restart. Body-guard is
NOT a hot-reload mechanism; it exists for defensive degradation so
that direct-emit paths (unit tests, future features that bypass the
upstream clustering gate) still fail cleanly instead of crashing
inside recall.
"""
def _get_engine() -> OfflineEngine:
"""Return the singleton OfflineEngine; constructed + registered on first call.
All strategies (both always-on and embed-requiring) are registered
unconditionally. Embed-requiring strategies guard their own bodies
with :func:`everos.component.embedding.get_embedding_capability`.
Tier changes require a server restart to take effect: the accessor's
cached capability is not invalidated at runtime. Body-guard keeps
dispatches clean when capability was absent at start; it does not
substitute for restart on config changes.
Lifecycle (start/stop) is wired by ``OmeLifespanProvider``.
"""
global _ome_engine
if _ome_engine is None:
root = MemoryRoot.default()
root = MemoryRoot.resolve()
jobstore_path = root.ome_db
jobstore_path.parent.mkdir(parents=True, exist_ok=True)
engine = OfflineEngine(
@@ -131,14 +169,8 @@ def _get_engine() -> OfflineEngine:
config_path=root.ome_config,
)
)
engine.register(extract_atomic_facts)
engine.register(extract_foresight)
engine.register(extract_agent_case)
engine.register(trigger_skill_clustering)
engine.register(extract_agent_skill)
engine.register(trigger_profile_clustering)
engine.register(extract_user_profile)
engine.register(reflect_episodes)
for strategy in (*_STRATEGIES_ALWAYS, *_STRATEGIES_REQUIRE_EMBED):
engine.register(strategy)
_ome_engine = engine
return _ome_engine
+28 -85
View File
@@ -7,17 +7,24 @@ lifespan that brings up LanceDB / settings.
Component policy (matches :class:`SearchManager` guards):
* Embedding / rerank / LLM clients are **optional at boot**; they are
built lazily, and only the methods that need them fail (with a clear
* Embedding / rerank / LLM clients are **optional at boot**; the manager
is built lazily and only the methods that need them fail (with a clear
message) when the corresponding section of settings is empty.
* ``KEYWORD`` searches therefore work without any of the three clients,
which makes the endpoint usable in a freshly-installed dev setup.
* All three providers are pulled from their process-wide capability /
singleton accessors (:func:`get_embedding_capability`,
:func:`get_rerank_capability`, :func:`get_llm_client`); this module
never keeps parallel singletons of its own.
"""
from __future__ import annotations
from typing import TYPE_CHECKING
from everos.component.embedding import get_embedding_capability
from everos.component.llm import LLMNotConfiguredError, get_llm_client
from everos.component.rerank import get_rerank_capability
from everos.component.tokenizer import build_tokenizer
from everos.core.observability.logging import get_logger
from everos.memory.search import SearchRequest, SearchResponse
@@ -32,100 +39,36 @@ from everos.memory.search.recall import (
)
if TYPE_CHECKING:
from everos.component.embedding import EmbeddingProvider
from everos.component.llm import LLMClient
from everos.component.rerank import RerankProvider
logger = get_logger(__name__)
# Lazy singletons ────────────────────────────────────────────────────────
# Lazy singleton — the manager itself; every provider it needs comes
# from its own process-wide accessor (see module docstring).
_manager: SearchManager | None = None
_embedding: EmbeddingProvider | None = None
_reranker: RerankProvider | None = None
_llm_client: LLMClient | None = None
_embedding_resolved = False
_rerank_resolved = False
_llm_resolved = False
def _get_embedding() -> EmbeddingProvider | None:
"""Build the embedding client on first call. ``None`` when not configured."""
global _embedding, _embedding_resolved
if _embedding_resolved:
return _embedding
from everos.component.embedding import build_embedding_provider
from everos.config import load_settings
cfg = load_settings().embedding
if not cfg.model or not cfg.api_key or not cfg.api_key.get_secret_value():
logger.warning(
"embedding_not_configured",
hint="set [embedding] model / api_key to enable vector / hybrid search",
)
_embedding = None
else:
_embedding = build_embedding_provider(cfg)
logger.info("search_embedding_built", model=cfg.model)
_embedding_resolved = True
return _embedding
def _get_reranker() -> RerankProvider | None:
"""Build the rerank client on first call. ``None`` when not configured."""
global _reranker, _rerank_resolved
if _rerank_resolved:
return _reranker
from everos.component.rerank import build_rerank_provider
from everos.config import load_settings
cfg = load_settings().rerank
has_key = cfg.api_key and cfg.api_key.get_secret_value()
if not cfg.model or not cfg.base_url or not has_key:
logger.warning(
"rerank_not_configured",
hint="set [rerank] model / api_key / base_url to enable agentic search",
)
_reranker = None
else:
_reranker = build_rerank_provider(cfg)
logger.info("search_rerank_built", model=cfg.model, provider=cfg.provider)
_rerank_resolved = True
return _reranker
def _get_llm_client() -> LLMClient | None:
"""Lazily build the LLM client from settings (shared with memorize)."""
global _llm_client, _llm_resolved
if _llm_resolved:
return _llm_client
"""Return the process-wide LLM client, or ``None`` when unset.
from everos.component.llm import build_llm_provider
from everos.config import load_settings
settings = load_settings()
cfg = settings.llm
if not cfg.api_key or not cfg.api_key.get_secret_value() or not cfg.base_url:
Delegates to :func:`everos.component.llm.get_llm_client` (which
caches its own singleton, validates ``api_key`` / ``base_url``, and
already wraps the client with ``UsageRecordingClient`` when
observability is enabled — see ``component/llm/client.py``) instead
of maintaining a parallel module-level singleton here. LLM is
optional for search — ``KEYWORD`` works without it — so
:class:`LLMNotConfiguredError` is swallowed into ``None`` and the
manager surfaces a clear error only when a method that actually
needs the LLM is invoked.
"""
try:
return get_llm_client()
except LLMNotConfiguredError:
logger.warning(
"llm_not_configured",
hint="set [llm] api_key / base_url to enable hybrid / agentic search",
)
_llm_client = None
else:
client = build_llm_provider(cfg)
# Record token usage for the hybrid/agentic path, mirroring
# get_llm_client() — otherwise the heaviest LLM spend (query
# decomposition + rerank judge) is invisible in Langfuse.
if settings.observability.enabled:
from everos.component.llm._usage_client import UsageRecordingClient
client = UsageRecordingClient(client)
_llm_client = client
logger.info("search_llm_built", model=cfg.model)
_llm_resolved = True
return _llm_client
return None
def _get_manager() -> SearchManager:
@@ -138,8 +81,8 @@ def _get_manager() -> SearchManager:
agent_case_recaller=AgentCaseRecaller(deps),
agent_skill_recaller=AgentSkillRecaller(deps),
profile_recaller=ProfileRecaller(),
embedding=_get_embedding(),
reranker=_get_reranker(),
embedding=get_embedding_capability().provider,
reranker=get_rerank_capability().provider,
llm_client=_get_llm_client(),
)
return _manager
+53
View File
@@ -65,6 +65,59 @@ def _reset_settings_cache() -> Iterator[None]:
dt_module._display_tz.cache_clear()
@pytest.fixture(autouse=True)
def _reset_embedding_capability_singleton() -> Iterator[None]:
"""Force embedding capability to ``available=False`` for every test.
``get_embedding_capability()`` lazily builds a process-wide singleton
on first call (see ``component.embedding.accessor``). Leaving the
cache as ``None`` between tests lets the accessor read ambient
settings — ``~/.everos/everos.toml`` on a developer machine, ``.env``
when loaded — which turns test outcomes into a function of the host
environment. Pre-seeding a ``Capability(provider=None)`` keeps the
suite hermetic: any body-guard that reads ``.available`` sees
``False`` unless the test explicitly opts in by re-assigning
``acc._capability`` (or patching the strategy module's own
``get_embedding_capability`` reference).
"""
import everos.component.embedding.accessor as acc
from everos.component.embedding import EmbeddingCapability
acc._capability = EmbeddingCapability(provider=None)
yield
acc._capability = None
@pytest.fixture(autouse=True)
def _reset_rerank_capability_singleton() -> Iterator[None]:
"""Force rerank capability to ``available=False`` for every test.
See :func:`_reset_embedding_capability_singleton` for rationale — the
rerank accessor has the same lazy-singleton shape.
"""
import everos.component.rerank.accessor as acc
from everos.component.rerank import RerankCapability
acc._capability = RerankCapability(provider=None)
yield
acc._capability = None
@pytest.fixture(autouse=True)
def _reset_multimodal_capability_singleton() -> Iterator[None]:
"""Force multimodal capability to ``available=False`` for every test.
See :func:`_reset_embedding_capability_singleton` for rationale — the
multimodal accessor has the same lazy-singleton shape.
"""
import everos.component.multimodal.accessor as acc
from everos.component.multimodal import MultimodalLLMCapability
acc._capability = MultimodalLLMCapability(provider=None)
yield
acc._capability = None
@pytest.fixture(scope="session")
def long_conversation() -> dict:
"""LoCoMo conv_0 fixture (419 messages, 19 batches, one session)."""
+2 -2
View File
@@ -61,7 +61,7 @@ _MEMORIZE_SINGLETONS: tuple[str, ...] = (
)
# OME strategy modules carry module-level lazy singletons (``_writer`` /
# ``_reader``) that capture ``MemoryRoot.default()`` at first call. They
# ``_reader``) that capture ``MemoryRoot.resolve()`` at first call. They
# survive across tests, so the second test writes its output to the
# **first test's** tmp_path. Reset all of them per-test.
_STRATEGY_SINGLETONS: tuple[tuple[str, tuple[str, ...]], ...] = (
@@ -75,7 +75,7 @@ _STRATEGY_SINGLETONS: tuple[tuple[str, tuple[str, ...]], ...] = (
def _reset_strategy_singletons(monkeypatch: pytest.MonkeyPatch) -> None:
"""Null every strategy ``_writer`` / ``_reader`` so the next test
rebuilds against its own ``MemoryRoot.default()`` (driven by the
rebuilds against its own ``MemoryRoot.resolve()`` (driven by the
fresh ``EVEROS_ROOT`` env var set by the calling fixture).
"""
for mod_name, attrs in _STRATEGY_SINGLETONS:
+19 -22
View File
@@ -40,7 +40,7 @@ import pytest
from everalgo.clustering import Cluster as AlgoCluster
from httpx import ASGITransport, AsyncClient
from everos.component.embedding import get_embedder
from everos.component.embedding import get_embedding_capability
from everos.config import load_settings
from everos.entrypoints.api.app import create_app
from everos.infra.persistence.lancedb import (
@@ -96,21 +96,22 @@ async def client(
async with _engine.begin() as _conn:
await _conn.run_sync(_SQLModel.metadata.create_all)
# Search service: reset all lazy singletons so each test rebuilds
# Search service: reset the manager singleton so each test rebuilds
# against the just-monkey-patched memory root + .env creds.
for attr in (
"_manager",
"_embedding",
"_reranker",
"_llm_client",
):
setattr(search_service_mod, attr, None)
for attr in (
"_embedding_resolved",
"_rerank_resolved",
"_llm_resolved",
):
setattr(search_service_mod, attr, False)
search_service_mod._manager = None
# Embedding / rerank / LLM route through their process-wide
# capability / singleton accessors — reset those directly so each
# test rebuilds against the just-monkey-patched memory root + .env
# creds (also covered by the autouse fixtures in
# ``tests/conftest.py``; kept explicit here for locality with the
# rest of this fixture's singleton reset).
embedding_acc = import_module("everos.component.embedding.accessor")
rerank_acc = import_module("everos.component.rerank.accessor")
llm_client_mod = import_module("everos.component.llm.client")
embedding_acc._capability = None
rerank_acc._capability = None
llm_client_mod._llm_client = None
app = create_app(lifespan_providers=[])
transport = ASGITransport(app=app)
@@ -170,7 +171,7 @@ async def _seed_user_memory_cluster(eps: list[dict], *, owner_id: str) -> None:
"""
memcell_ids = list({ep["parent_id"] for ep in eps})
centroid_text = eps[0]["episode"]
centroid_vec = await get_embedder().embed(centroid_text)
centroid_vec = await get_embedding_capability().require().embed(centroid_text)
await cluster_repo.upsert_with_members(
AlgoCluster(
id=mint_cluster_id(),
@@ -428,14 +429,12 @@ async def _seed_one_agent_corpus(owner: str = "a1") -> None:
can rank them (zero vectors are undefined under cosine distance —
the dense path returns 0 hits for them).
"""
from everos.service.search import _get_embedding
case_intent = "refactor authentication middleware"
case_approach = "split provider lookup from session decode"
skill_desc = "refactor authentication middleware reliably"
skill_body = "step-by-step approach for auth refactors"
embedder = _get_embedding()
embedder = get_embedding_capability().provider
if embedder is not None:
case_vec, skill_vec = await embedder.embed_batch(
[f"{case_intent}\n{case_approach}", f"{skill_desc}\n{skill_body}"]
@@ -578,8 +577,6 @@ async def test_hybrid_rerank_bridges_skill_via_case_lineage(
surface it on its own; the bridge is the path that does, and LLM
rerank keeps it because the topic is genuinely relevant.
"""
from everos.service.search import _get_embedding
case_id_with_owner = "a_bridge_ac_1" # mirrors AgentCase.id = "<owner>_<entry_id>"
case_intent = "refactor authentication middleware"
case_approach = "split provider lookup from session decode"
@@ -595,7 +592,7 @@ async def test_hybrid_rerank_bridges_skill_via_case_lineage(
"independently."
)
embedder = _get_embedding()
embedder = get_embedding_capability().provider
assert embedder is not None, "live_llm test requires a real embedder"
case_vec, skill_vec = await embedder.embed_batch(
[f"{case_intent}\n{case_approach}", f"{skill_desc}\n{skill_body}"]
+13 -9
View File
@@ -224,15 +224,7 @@ async def search_client(
# The search service has its own module-level singletons; reset
# those too so re-attach is clean.
search_svc = importlib.import_module("everos.service.search")
for attr in (
"_manager",
"_embedding",
"_reranker",
"_llm_client",
"_embedding_resolved",
"_rerank_resolved",
"_llm_resolved",
):
for attr in ("_manager", "_llm_client", "_llm_resolved"):
if hasattr(search_svc, attr):
monkeypatch.setattr(
search_svc,
@@ -241,6 +233,18 @@ async def search_client(
raising=False,
)
# Embedding / rerank now route through the process-wide capability
# accessors (``everos.component.{embedding,rerank}.accessor``) instead
# of module-level singletons on ``everos.service.search`` — reset those
# singletons directly so re-attach is clean. (Also covered by the
# autouse fixtures in ``tests/conftest.py``; kept here for locality with
# the rest of this fixture's singleton reset.)
import everos.component.embedding.accessor as embedding_acc
import everos.component.rerank.accessor as rerank_acc
monkeypatch.setattr(embedding_acc, "_capability", None, raising=False)
monkeypatch.setattr(rerank_acc, "_capability", None, raising=False)
from everos.entrypoints.api.app import create_app
app = create_app()
@@ -0,0 +1,327 @@
"""Tests for GET /health endpoint capabilities and disabled_features response."""
from __future__ import annotations
from pytest import mark, param
from everos.component.capabilities import compute_disabled_features
class TestComputeDisabledFeatures:
"""Test the compute_disabled_features helper function."""
def test_all_capabilities_available(self) -> None:
"""When all capabilities are available, no features are disabled."""
caps = {
"llm": True,
"embed": True,
"rerank": True,
"multimodal_llm": True,
"parser": True,
}
disabled = compute_disabled_features(caps)
assert disabled == []
def test_embedding_unavailable_disables_vector_search_features(
self,
) -> None:
"""When embedding unavailable, vector/hybrid/reflection/skill disabled."""
caps = {
"llm": True,
"embed": False,
"rerank": True,
"multimodal_llm": True,
"parser": True,
}
disabled = compute_disabled_features(caps)
expected = {
"vector_search",
"hybrid_search",
"reflection",
"skill_extraction",
"knowledge",
}
assert set(disabled) == expected
def test_rerank_unavailable_disables_agentic_search(self) -> None:
"""When rerank is unavailable, agentic_search is disabled."""
caps = {
"llm": True,
"embed": True,
"rerank": False,
"multimodal_llm": True,
"parser": True,
}
disabled = compute_disabled_features(caps)
assert "agentic_search" in disabled
def test_both_embed_and_rerank_missing_disables_knowledge(self) -> None:
"""When both embed and rerank are missing, knowledge is disabled."""
caps = {
"llm": True,
"embed": False,
"rerank": False,
"multimodal_llm": True,
"parser": True,
}
disabled = compute_disabled_features(caps)
assert "knowledge" in disabled
def test_multimodal_and_parser_missing_disables_multimodal_upload(self) -> None:
"""When multimodal_llm is unavailable, multimodal_upload is disabled."""
caps = {
"llm": True,
"embed": True,
"rerank": True,
"multimodal_llm": False,
"parser": True,
}
disabled = compute_disabled_features(caps)
assert "multimodal_upload" in disabled
def test_parser_missing_disables_multimodal_upload(self) -> None:
"""When parser is unavailable, multimodal_upload is disabled."""
caps = {
"llm": True,
"embed": True,
"rerank": True,
"multimodal_llm": True,
"parser": False,
}
disabled = compute_disabled_features(caps)
assert "multimodal_upload" in disabled
def test_tier1_only_llm(self) -> None:
"""Tier 1 (LLM only): many features are disabled."""
caps = {
"llm": True,
"embed": False,
"rerank": False,
"multimodal_llm": False,
"parser": False,
}
disabled = compute_disabled_features(caps)
# Tier 1 disables: vector/hybrid/agentic/reflection/skill/knowledge/multimodal
expected = {
"vector_search",
"hybrid_search",
"agentic_search",
"reflection",
"skill_extraction",
"knowledge",
"multimodal_upload",
}
assert set(disabled) == expected
@mark.parametrize(
"caps,expected_disabled",
[
param(
{
"llm": True,
"embed": True,
"rerank": True,
"multimodal_llm": True,
"parser": True,
},
[],
id="all_available",
),
param(
{
"llm": True,
"embed": False,
"rerank": True,
"multimodal_llm": True,
"parser": True,
},
[
"vector_search",
"hybrid_search",
"reflection",
"skill_extraction",
"knowledge",
],
id="no_embed",
),
param(
{
"llm": True,
"embed": True,
"rerank": False,
"multimodal_llm": True,
"parser": True,
},
["agentic_search", "knowledge"],
id="no_rerank",
),
],
)
def test_combinations(
self,
caps: dict[str, bool],
expected_disabled: list[str],
) -> None:
"""Parametrized tests for various capability combinations."""
disabled = compute_disabled_features(caps)
assert set(disabled) == set(expected_disabled)
@mark.asyncio
async def test_health_endpoint_includes_capabilities() -> None:
"""GET /health returns capabilities dict with all 5 provider fields."""
from httpx import ASGITransport, AsyncClient
from everos.entrypoints.api.app import create_app
app = create_app()
transport = ASGITransport(app=app)
async with AsyncClient(transport=transport) as client:
response = await client.get("http://testserver/health")
assert response.status_code == 200
data = response.json()
assert "status" in data
assert "version" in data
assert "capabilities" in data
assert "disabled_features" in data
caps = data["capabilities"]
assert isinstance(caps, dict)
assert "llm" in caps
assert "embed" in caps
assert "rerank" in caps
assert "multimodal_llm" in caps
assert "parser" in caps
# All capability values should be boolean
for key, value in caps.items():
msg = f"Capability {key} should be bool, got {type(value)}"
assert isinstance(value, bool), msg
# disabled_features should be a list
disabled = data["disabled_features"]
assert isinstance(disabled, list)
for feature in disabled:
assert isinstance(feature, str)
@mark.asyncio
@mark.parametrize(
("embed", "rerank", "multimodal_llm", "parser", "expected_disabled"),
[
param(
False,
False,
False,
False,
{
"vector_search",
"hybrid_search",
"reflection",
"skill_extraction",
"agentic_search",
"knowledge",
"multimodal_upload",
},
id="tier1_llm_only",
),
param(
True,
False,
False,
False,
{"agentic_search", "knowledge", "multimodal_upload"},
id="tier2_embed_only",
),
param(
True,
True,
False,
False,
{"multimodal_upload"},
id="tier3_embed_rerank_only",
),
param(
True,
True,
True,
True,
set(),
id="full_stack_all_enabled",
),
],
)
async def test_health_endpoint_disabled_features_matches_hardcoded_expectation(
monkeypatch,
embed: bool,
rerank: bool,
multimodal_llm: bool,
parser: bool,
expected_disabled: set[str],
) -> None:
"""GET /health disabled_features must match a hard-coded expected set.
Pre-M12 this test asserted ``set(response['disabled_features']) ==
set(compute_disabled_features(response['capabilities']))`` — a
tautology, because the endpoint uses the same
:func:`compute_disabled_features` to build the field. Any bug in
the helper would be present on both sides of the equation and go
undetected. Round-2 finding M12 replaces the tautology with a
per-tier hard-coded expected set, driven by parametrisation over
every soft-dependency capability toggle — so a regression in
:func:`compute_disabled_features` (e.g. dropping the
``knowledge`` disable, mis-spelling a feature name) breaks the
test rather than staying invisible.
"""
from httpx import ASGITransport, AsyncClient
from everos.component import embedding, multimodal
from everos.component import rerank as rerank_mod
from everos.component.parser import _core as parser_core
from everos.entrypoints.api.app import create_app
class _StubCapability:
def __init__(self, available: bool) -> None:
self.available = available
monkeypatch.setattr(
embedding, "get_embedding_capability", lambda: _StubCapability(embed)
)
monkeypatch.setattr(
rerank_mod, "get_rerank_capability", lambda: _StubCapability(rerank)
)
monkeypatch.setattr(
multimodal,
"get_multimodal_llm_capability",
lambda: _StubCapability(multimodal_llm),
)
monkeypatch.setattr(parser_core, "parser_available", lambda: parser)
# The health route imports the accessors at module load; patch there
# too so the values it actually reads honour the monkeypatch.
from everos.entrypoints.api.routes import health as health_route
monkeypatch.setattr(
health_route, "get_embedding_capability", lambda: _StubCapability(embed)
)
monkeypatch.setattr(
health_route, "get_rerank_capability", lambda: _StubCapability(rerank)
)
monkeypatch.setattr(
health_route,
"get_multimodal_llm_capability",
lambda: _StubCapability(multimodal_llm),
)
monkeypatch.setattr(health_route, "parser_available", lambda: parser)
app = create_app()
transport = ASGITransport(app=app)
async with AsyncClient(transport=transport) as client:
response = await client.get("http://testserver/health")
assert response.status_code == 200
data = response.json()
assert set(data["disabled_features"]) == expected_disabled
@@ -0,0 +1,386 @@
"""Verify per-endpoint scoping of the knowledge capability gate.
Task 11 (``cascade.registry.build_handlers``) gates the knowledge
cascade handlers off as an atomic pair when embed + rerank aren't
both available — a doc queued for a gated-off kind gets marked
permanently failed by the worker instead of ever reaching LanceDB.
Before this router gate existed, the HTTP layer had no idea any of
that happened: a Tier 1 user's upload returned 200 (the md write
always succeeds) while the document silently never became
searchable.
An earlier iteration attached the gate router-wide, which regressed
downgraded users: a client who dropped from Tier 3 → Tier 2/1 could
no longer even GET or DELETE their own docs, because the router-wide
dependency fired on every request. This module pins the corrected
scoping — writes and search stay gated (they can't succeed end-to-
end without the providers) while reads/deletes stay reachable so
users can inspect and clean up state they already have on disk.
Uses ``httpx.AsyncClient`` + ``ASGITransport`` (not
``fastapi.testclient.TestClient``) to match the project's established
exception-handler test convention (see ``tests/integration/test_api/
test_provider_error_mapping.py``). The knowledge router is mounted
standalone (no lifespan / full app) since the gate fires before any
body is even parsed — sub-dependency resolution runs ahead of
body/query/path validation in FastAPI's request handler, so a bare
request with no payload is enough to reach the gate on every route,
including the multipart upload endpoints.
For endpoints without a gate we stub the service call so the test
proves the request reached the handler (i.e. no 422 from the
missing-provider check), independent of MemoryRoot / SQLite setup.
"""
from __future__ import annotations
from collections.abc import AsyncIterator
from typing import Any
import pytest
from fastapi import FastAPI
from httpx import ASGITransport, AsyncClient
import everos.component.embedding.accessor as embedding_accessor
import everos.component.rerank.accessor as rerank_accessor
from everos.component.embedding import EmbeddingCapability
from everos.component.rerank import RerankCapability
from everos.entrypoints.api.exception_handlers import register_handlers
from everos.entrypoints.api.routes import knowledge as knowledge_routes
# Endpoints that MUST return 422 when embed or rerank is missing.
# These paths either write new md that cascade will need to embed
# (POST/PUT) or run a search that requires both providers
# (POST /search). PATCH is deliberately NOT here — see the ungated
# list below.
_GATED_ENDPOINTS: list[tuple[str, str]] = [
("POST", "/api/v1/knowledge/documents"),
("PUT", "/api/v1/knowledge/documents/d_abcdef123456"),
("POST", "/api/v1/knowledge/search"),
]
# Endpoints that MUST stay reachable even when both capabilities are
# missing. All of these touch only md + SQLite state that already
# exists on disk — none of them need the embedding or rerank provider.
# PATCH is here because ``patch_document`` only rewrites md frontmatter
# and moves the doc directory when category changes; no embed or rerank
# code runs on that path. A user who downgrades from Tier 3 → Tier 1/2
# must still be able to rename or recategorize the documents they
# created while providers were configured.
_UNGATED_ENDPOINTS: list[tuple[str, str]] = [
("DELETE", "/api/v1/knowledge/documents/d_abcdef123456"),
("GET", "/api/v1/knowledge/documents"),
("GET", "/api/v1/knowledge/documents/d_abcdef123456"),
("GET", "/api/v1/knowledge/topics/d_abcdef123456_0"),
("GET", "/api/v1/knowledge/categories"),
]
def _build_app() -> FastAPI:
app = FastAPI()
register_handlers(app)
# Match production mounting: create_app() mounts the router under
# /api/v1 (and /api/v2). The router's own prefix is /knowledge, so
# the effective paths become /api/v1/knowledge/... — same shape the
# _GATED / _UNGATED constants below use.
app.include_router(knowledge_routes.router, prefix="/api/v1")
return app
@pytest.fixture
async def client() -> AsyncIterator[AsyncClient]:
app = _build_app()
transport = ASGITransport(app=app, raise_app_exceptions=False)
async with AsyncClient(transport=transport, base_url="http://test") as c:
yield c
@pytest.fixture(autouse=True)
def _capabilities_unavailable_by_default(monkeypatch: pytest.MonkeyPatch) -> None:
"""Default both singletons to unavailable; tests opt into availability
per-capability via the fixtures below. Mirrors ``tests/unit/test_memory
/test_search/test_validate_components.py``.
"""
monkeypatch.setattr(
embedding_accessor, "_capability", EmbeddingCapability(provider=None)
)
monkeypatch.setattr(rerank_accessor, "_capability", RerankCapability(provider=None))
@pytest.fixture
def embed_available(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(
embedding_accessor, "_capability", EmbeddingCapability(provider=object())
)
@pytest.fixture
def rerank_available(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(
rerank_accessor, "_capability", RerankCapability(provider=object())
)
@pytest.fixture
def _stub_read_services(monkeypatch: pytest.MonkeyPatch) -> None:
"""Replace every read/delete/list service call with a benign stub.
The tests in this module care only about whether the capability
gate fires. Ungated endpoints must reach their handler even when
providers are missing — but reaching the handler with no
MemoryRoot / SQLite setup would 500. Stubbing keeps the assertion
focused on the gate, not on end-to-end service behavior.
"""
from everos.service import knowledge as knowledge_service
async def _fake_delete_document(*args: Any, **kwargs: Any) -> Any:
return knowledge_service.DeleteResult(doc_id="d_abcdef123456", deleted_topics=0)
async def _fake_list_documents(*args: Any, **kwargs: Any) -> Any:
return knowledge_service.DocumentListResult(
documents=[], total=0, page=1, page_size=20
)
async def _fake_get_document(*args: Any, **kwargs: Any) -> Any:
from everos.component.utils.datetime import get_utc_now
now = get_utc_now()
return knowledge_service.DocumentDetail(
doc_id="d_abcdef123456",
category_id="General",
title="stub",
summary="stub",
source_name=None,
source_type=None,
original_file_path=None,
topics=[],
created_at=now,
updated_at=now,
)
async def _fake_get_topic(*args: Any, **kwargs: Any) -> Any:
from everos.component.utils.datetime import get_utc_now
now = get_utc_now()
return knowledge_service.TopicDetail(
topic_id="d_abcdef123456_0",
doc_id="d_abcdef123456",
category_id="General",
topic_name="stub",
topic_path="stub",
depth=0,
summary="stub",
content="",
content_labels=[],
parent_topic_id=None,
children_topic_ids=[],
created_at=now,
updated_at=now,
)
async def _fake_list_categories(*args: Any, **kwargs: Any) -> list[Any]:
return []
monkeypatch.setattr(knowledge_routes, "delete_document", _fake_delete_document)
monkeypatch.setattr(knowledge_routes, "list_documents", _fake_list_documents)
monkeypatch.setattr(knowledge_routes, "get_document", _fake_get_document)
monkeypatch.setattr(knowledge_routes, "get_topic", _fake_get_topic)
monkeypatch.setattr(knowledge_routes, "list_categories", _fake_list_categories)
@pytest.fixture
def _stub_patch_document(monkeypatch: pytest.MonkeyPatch) -> None:
"""Stub ``patch_document`` to echo the requested title back.
Mirrors the real service contract: a title change lands in
``updated_fields`` and the caller sees the mutation reflected in
the response envelope. Keeps the assertion focused on the gate
(or lack of it) instead of on md / SQLite plumbing.
"""
from everos.component.utils.datetime import get_utc_now
from everos.service import knowledge as knowledge_service
captured: dict[str, Any] = {}
async def _fake_patch_document(
doc_id: str,
app_id: str,
project_id: str,
*,
title: str | None = None,
category_id: str | None = None,
) -> Any:
captured["title"] = title
captured["category_id"] = category_id
updated_fields: list[str] = []
if title is not None:
updated_fields.append("title")
if category_id is not None:
updated_fields.append("category_id")
return knowledge_service.PatchResult(
doc_id=doc_id,
updated_fields=updated_fields,
updated_at=get_utc_now(),
)
monkeypatch.setattr(knowledge_routes, "patch_document", _fake_patch_document)
async def _call(client: AsyncClient, method: str, path: str) -> Any:
return await client.request(method, path)
# ── Gated endpoints: 422 when embed missing (rerank also missing) ────────
@pytest.mark.parametrize("method,path", _GATED_ENDPOINTS)
async def test_gated_endpoint_422_when_embed_missing(
client: AsyncClient, method: str, path: str
) -> None:
"""Write/search endpoints must 422 with an embedding-specific message
when embedding is missing (check-embedding-first order per Task 12/13).
"""
resp = await _call(client, method, path)
assert resp.status_code == 422
body = resp.json()
assert body["error"]["code"] == "PROVIDER_NOT_CONFIGURED"
assert "embedding" in body["error"]["message"]
assert "knowledge" in body["error"]["message"]
# ── Gated endpoints: 422 when only rerank missing ───────────────────────
@pytest.mark.parametrize("method,path", _GATED_ENDPOINTS)
async def test_gated_endpoint_422_when_rerank_missing(
client: AsyncClient, method: str, path: str, embed_available: None
) -> None:
"""Write/search endpoints must 422 with a rerank-specific message
when only rerank is missing.
"""
resp = await _call(client, method, path)
assert resp.status_code == 422
body = resp.json()
assert body["error"]["code"] == "PROVIDER_NOT_CONFIGURED"
assert "rerank" in body["error"]["message"]
assert "knowledge" in body["error"]["message"]
# ── Ungated endpoints: must NOT 422 even with both capabilities missing ─
@pytest.mark.parametrize("method,path", _UNGATED_ENDPOINTS)
async def test_ungated_endpoint_not_422_without_any_capability(
client: AsyncClient,
method: str,
path: str,
_stub_read_services: None,
) -> None:
"""Read/list/delete endpoints must stay reachable even when neither
embed nor rerank is configured — a Tier-3 → Tier-1 downgrade must
not lock users out of inspecting and cleaning up their own state.
"""
resp = await _call(client, method, path)
assert resp.status_code != 422, (
f"{method} {path} returned 422 despite the gate being scoped away "
f"from read/delete endpoints; body={resp.text}"
)
# ── Ungated endpoints: also fine with only embed (Tier-2 downgrade) ─────
@pytest.mark.parametrize("method,path", _UNGATED_ENDPOINTS)
async def test_ungated_endpoint_not_422_with_embed_only(
client: AsyncClient,
method: str,
path: str,
embed_available: None,
_stub_read_services: None,
) -> None:
"""Tier-2 scenario (embed present, rerank missing): read/delete paths
must stay reachable — this is the concrete regression from the
router-wide-gate era that this fix targets.
"""
resp = await _call(client, method, path)
assert resp.status_code != 422, (
f"{method} {path} returned 422 despite rerank not being needed; "
f"body={resp.text}"
)
# ── Gated endpoints: pass the gate when both are available ──────────────
async def test_search_not_422_when_both_available(
client: AsyncClient,
monkeypatch: pytest.MonkeyPatch,
embed_available: None,
rerank_available: None,
) -> None:
"""Both capabilities available: the gate must not fire on /search.
Stub the service call so the assertion is about the gate, not the
rest of the search pipeline (out of scope for this task).
"""
from everos.service import knowledge as knowledge_service
stub_result = knowledge_service.SearchKnowledgeResult(hits=[], total=0, took_ms=0.0)
async def _fake_search_knowledge(**kwargs: Any) -> Any:
return stub_result
monkeypatch.setattr(knowledge_routes, "search_knowledge", _fake_search_knowledge)
resp = await client.post(
"/api/v1/knowledge/search",
json={"query": "hello"},
)
assert resp.status_code != 422
# ── PATCH: metadata-only, must succeed regardless of provider config ────
_PATCH_BODY: dict[str, str] = {
"app_id": "app1",
"project_id": "proj1",
"title": "renamed",
}
async def test_knowledge_patch_document_succeeds_without_rerank(
client: AsyncClient,
embed_available: None,
_stub_patch_document: None,
) -> None:
"""Tier-2 scenario (embed configured, rerank missing): PATCH must
still land. Renaming an existing document does not touch rerank —
it only rewrites md frontmatter and upserts SQLite.
"""
resp = await client.patch(
"/api/v1/knowledge/documents/d_abcdef123456",
json=_PATCH_BODY,
)
assert resp.status_code == 200, resp.text
body = resp.json()
assert body["data"]["doc_id"] == "d_abcdef123456"
assert "title" in body["data"]["updated_fields"]
async def test_knowledge_patch_document_succeeds_without_any_capability(
client: AsyncClient,
_stub_patch_document: None,
) -> None:
"""Tier-1 scenario (no embed, no rerank): PATCH must still land.
Metadata-only edits are a cleanup verb — locking users out of
renaming their own docs after a full downgrade contradicts the
per-endpoint-gate contract documented on
``_require_knowledge_capabilities``.
"""
resp = await client.patch(
"/api/v1/knowledge/documents/d_abcdef123456",
json=_PATCH_BODY,
)
assert resp.status_code == 200, resp.text
body = resp.json()
assert body["data"]["doc_id"] == "d_abcdef123456"
assert "title" in body["data"]["updated_fields"]
@@ -0,0 +1,70 @@
"""Verify ProviderNotConfiguredError maps to HTTP 422 via the FastAPI handler.
Deviates from a naive flat `{"error_code": ..., "message": ...}` envelope: the
project already has a canonical error envelope (`ErrorResponse` /
`_error_response` in `exception_handlers.py`) shared by every domain
exception, so `ProviderNotConfiguredError` reuses it rather than inventing a
parallel shape. `ProviderNotConfiguredError` is a `ConfigurationError`
subclass; the assertions below also pin that the more-specific handler wins
over the parent `ConfigurationError` -> 500 mapping.
Uses `httpx.AsyncClient` + `ASGITransport` (not `fastapi.testclient.TestClient`)
to match the project's existing exception-handler test convention -- the
installed starlette/httpx pair deprecates `TestClient`.
"""
from __future__ import annotations
from fastapi import FastAPI
from httpx import ASGITransport, AsyncClient
from everos.core.errors import ProviderNotConfiguredError
from everos.entrypoints.api.exception_handlers import register_handlers
def _build_app_with_probes() -> FastAPI:
app = FastAPI()
register_handlers(app)
@app.get("/probe_embed")
async def probe_embed() -> None:
raise ProviderNotConfiguredError(provider="embedding")
@app.get("/probe_rerank_with_alt")
async def probe_rerank() -> None:
raise ProviderNotConfiguredError(
provider="rerank",
feature="agent_hybrid",
alternative_hint="Set enable_llm_rerank=true.",
)
return app
async def _get(path: str) -> tuple[int, dict]:
app = _build_app_with_probes()
transport = ASGITransport(app=app, raise_app_exceptions=False)
async with AsyncClient(transport=transport, base_url="http://test") as client:
resp = await client.get(path)
return resp.status_code, resp.json()
async def test_maps_to_422():
status_code, _ = await _get("/probe_embed")
assert status_code == 422
async def test_error_code_field():
_, body = await _get("/probe_embed")
assert body["error"]["code"] == "PROVIDER_NOT_CONFIGURED"
async def test_message_field_carries_toml_path():
_, body = await _get("/probe_embed")
assert "everos.toml" in body["error"]["message"]
async def test_alternative_hint_present_in_message():
_, body = await _get("/probe_rerank_with_alt")
assert "Alternative:" in body["error"]["message"]
assert "enable_llm_rerank=true" in body["error"]["message"]
@@ -0,0 +1,128 @@
"""Test startup banner warning for unbackfilled memory rows.
When LanceDB tables have rows with vector IS NULL (unbackfilled memories),
the startup banner should log a warning message pointing users to
`everos cascade backfill` command.
"""
from __future__ import annotations
import hashlib
from pathlib import Path
import pytest
import structlog.testing
from everos.component.utils.datetime import get_utc_now
from everos.core.observability.logging import get_logger
from everos.core.persistence import MemoryRoot
from everos.entrypoints.api.lifespans.lancedb import (
LanceDBLifespanProvider,
_log_unbackfilled_hint,
)
from everos.infra.persistence.lancedb import Episode
from everos.infra.persistence.lancedb.repos import episode_repo
@pytest.fixture
def _isolated_memory_root(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
"""Isolate each test with its own temporary MemoryRoot."""
monkeypatch.setattr(
MemoryRoot, "resolve", classmethod(lambda cls: MemoryRoot(root=tmp_path))
)
(tmp_path / ".index" / "sqlite").mkdir(parents=True, exist_ok=True)
async def test_no_warning_when_empty(_isolated_memory_root: None) -> None:
"""No warning is logged when the database is empty."""
from fastapi import FastAPI
app = FastAPI()
provider = LanceDBLifespanProvider()
await provider.startup(app)
# When database is empty, no warning should be logged
with structlog.testing.capture_logs() as captured:
await _log_unbackfilled_hint()
matching = [e for e in captured if e.get("event") == "unbackfilled_memory_rows"]
assert len(matching) == 0, "expected no warning when database is empty"
await provider.shutdown(app)
async def test_warns_when_unbackfilled_rows_exist(
_isolated_memory_root: None,
) -> None:
"""_log_unbackfilled_hint warns with correct count when vector=NULL rows exist."""
from fastapi import FastAPI
# Setup: initialize provider (connects to temp LanceDB)
app = FastAPI()
provider = LanceDBLifespanProvider()
await provider.startup(app)
# Seed: insert one Episode row with vector=None
def _make_unvectorized_episode() -> Episode:
"""Create an Episode row with vector=None."""
return Episode(
id="test_owner_entry1",
entry_id="entry1",
owner_id="test_owner",
owner_type="user",
app_id="default",
project_id="default",
session_id="s_test",
timestamp=get_utc_now(),
parent_type="memcell",
parent_id="mc_test",
sender_ids=["test_owner"],
subject="test subject",
summary=None,
episode="test episode content",
episode_tokens="test episode",
md_path="users/test_owner/default/default/2026-07-25.md",
content_sha256=hashlib.sha256(b"test").hexdigest(),
deprecated_by=None,
vector=None, # explicitly unvectorized
)
await episode_repo.add([_make_unvectorized_episode()])
# Capture logs and call the hint function
with structlog.testing.capture_logs() as captured:
await _log_unbackfilled_hint()
# Assert: the warning is logged with the correct count
matching = [e for e in captured if e.get("event") == "unbackfilled_memory_rows"]
assert len(matching) > 0, (
"expected unbackfilled_memory_rows warning when vector=NULL rows exist"
)
warning = matching[0]
assert warning.get("count") == 1, f"expected count=1, got {warning.get('count')}"
await provider.shutdown(app)
async def test_log_unbackfilled_hint_runs_on_startup(
_isolated_memory_root: None,
) -> None:
"""Startup should call _log_unbackfilled_hint after ensuring indexes."""
from fastapi import FastAPI
app = FastAPI()
provider = LanceDBLifespanProvider()
# This startup should not raise - _log_unbackfilled_hint should be called
await provider.startup(app)
await provider.shutdown(app)
async def test_startup_banner_logger_configuration() -> None:
"""Verify the banner logger can be obtained."""
logger = get_logger("everos.cli.server")
assert logger is not None
# Verify it has the expected logging methods
assert hasattr(logger, "warning")
assert hasattr(logger, "info")
@@ -87,6 +87,15 @@ async def cascade_runtime(
monkeypatch.setenv("EVEROS_EMBEDDING__MODEL", "stub-model")
monkeypatch.setenv("EVEROS_EMBEDDING__BASE_URL", "http://stub.invalid/v1")
monkeypatch.setenv("EVEROS_EMBEDDING__API_KEY", "stub-key")
# Handlers fetch the embedder lazily via ``get_embedding_capability()``
# rather than through ``CascadeOrchestrator`` — patch the process-wide
# singleton so cascade never hits the fake network target above.
import everos.component.embedding.accessor as acc
from everos.component.embedding import EmbeddingCapability
monkeypatch.setattr(
acc, "_capability", EmbeddingCapability(provider=_StubEmbedder())
)
await dispose_connection()
await dispose_engine()
engine = get_engine()
@@ -94,7 +103,7 @@ async def cascade_runtime(
await conn.run_sync(SQLModel.metadata.create_all)
await ensure_business_indexes()
(tmp_path / "ome.toml").write_text("# test\n")
yield MemoryRoot.default()
yield MemoryRoot.resolve()
await dispose_connection()
await dispose_engine()
@@ -237,7 +246,6 @@ async def test_md_lance_strict_consistency_per_kind(
memory_root = cascade_runtime
orchestrator = CascadeOrchestrator(
memory_root=memory_root,
embedder=_StubEmbedder(),
tokenizer=build_tokenizer(),
config=CascadeConfig(
scan_interval_seconds=60.0,
@@ -4,7 +4,8 @@ Drives the actual Typer commands against a real sqlite + lancedb under a
tmp memory root. Validates the in-process orchestration that
``test_cascade_command`` (unit) cannot reach: ``_runtime()`` context,
queue summary formatting, fix (no-rows path), and a full
``cascade sync <path>`` round-trip with a stub embedder.
``cascade sync <path>`` round-trip against an empty queue (no handler
ever runs, so embedding availability is irrelevant here).
The CLI commands call ``asyncio.run(_run())`` internally, so this test
is **synchronous** — pytest-asyncio's auto mode would otherwise wrap it
@@ -22,23 +23,12 @@ from pathlib import Path
import pytest
from typer.testing import CliRunner
from everos.component.embedding import EmbeddingProvider
from everos.config import load_settings
from everos.entrypoints.cli.commands import cascade as cascade_mod
from everos.infra.persistence.lancedb import dispose_connection
from everos.infra.persistence.sqlite import dispose_engine
class _StubEmbedder(EmbeddingProvider):
dim = 1024
async def embed(self, text: str) -> list[float]:
return [0.0] * self.dim
async def embed_batch(self, texts): # type: ignore[no-untyped-def]
return [[0.0] * self.dim for _ in texts]
@pytest.fixture
def cli_runtime(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Iterator[Path]:
"""Tmp memory root + clean singletons; CLI bootstraps the schema itself."""
@@ -101,11 +91,10 @@ def test_sync_on_empty_queue_with_stub_embedder(
from everos.memory.cascade import CascadeOrchestrator
def fake_build_orchestrator() -> CascadeOrchestrator:
root = MemoryRoot.default()
root = MemoryRoot.resolve()
root.ensure()
return CascadeOrchestrator(
memory_root=root,
embedder=_StubEmbedder(),
tokenizer=build_tokenizer(),
)
@@ -148,8 +137,7 @@ def test_sync_with_unmatched_path(
def fake_build_orchestrator() -> CascadeOrchestrator:
return CascadeOrchestrator(
memory_root=MemoryRoot.default(),
embedder=_StubEmbedder(),
memory_root=MemoryRoot.resolve(),
tokenizer=build_tokenizer(),
)
@@ -72,6 +72,15 @@ async def cascade_runtime(
monkeypatch.setenv("EVEROS_EMBEDDING__MODEL", "stub-model")
monkeypatch.setenv("EVEROS_EMBEDDING__BASE_URL", "http://stub.invalid/v1")
monkeypatch.setenv("EVEROS_EMBEDDING__API_KEY", "stub-key")
# Handlers fetch the embedder lazily via ``get_embedding_capability()``
# rather than through ``CascadeOrchestrator`` — patch the process-wide
# singleton so cascade never hits the fake network target above.
import everos.component.embedding.accessor as acc
from everos.component.embedding import EmbeddingCapability
monkeypatch.setattr(
acc, "_capability", EmbeddingCapability(provider=_StubEmbedder())
)
await dispose_connection()
await dispose_engine()
@@ -82,7 +91,7 @@ async def cascade_runtime(
await ensure_business_indexes()
(tmp_path / "ome.toml").write_text("# test\n")
yield MemoryRoot.default()
yield MemoryRoot.resolve()
await dispose_connection()
await dispose_engine()
@@ -127,7 +136,6 @@ async def test_high_freq_atomic_fact_append_no_loss(
memory_root = cascade_runtime
orchestrator = CascadeOrchestrator(
memory_root=memory_root,
embedder=_StubEmbedder(),
tokenizer=build_tokenizer(),
config=CascadeConfig(
scan_interval_seconds=60.0,
+18 -3
View File
@@ -84,12 +84,25 @@ async def cascade_runtime(
await ensure_business_indexes()
(tmp_path / "ome.toml").write_text("# test\n")
yield MemoryRoot.default()
yield MemoryRoot.resolve()
await dispose_connection()
await dispose_engine()
def _patch_embedding_capability(
monkeypatch: pytest.MonkeyPatch, embedder: _StubEmbedder
) -> None:
"""EpisodeHandler fetches the embedder lazily via
``get_embedding_capability()`` — patch the process-wide singleton so
cascade never hits the fake network target set up by ``cascade_runtime``.
"""
import everos.component.embedding.accessor as acc
from everos.component.embedding import EmbeddingCapability
monkeypatch.setattr(acc, "_capability", EmbeddingCapability(provider=embedder))
async def _poll(condition, *, deadline_seconds: float = 10.0, interval: float = 0.05): # type: ignore[no-untyped-def]
"""Poll ``condition()`` (async) until truthy, or :class:`TimeoutError`.
@@ -107,13 +120,14 @@ async def _poll(condition, *, deadline_seconds: float = 10.0, interval: float =
async def test_append_to_md_propagates_to_lancedb(
cascade_runtime: MemoryRoot,
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Happy path: writer append → watcher → state row → worker → LanceDB."""
memory_root = cascade_runtime
embedder = _StubEmbedder()
_patch_embedding_capability(monkeypatch, embedder)
orchestrator = CascadeOrchestrator(
memory_root=memory_root,
embedder=embedder,
tokenizer=build_tokenizer(),
# Tight worker poll so the test wraps in seconds, not minutes.
# Scanner interval kept long so the watcher path is the one
@@ -188,12 +202,13 @@ async def test_append_to_md_propagates_to_lancedb(
async def test_delete_md_wipes_lancedb_row(
cascade_runtime: MemoryRoot,
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Append + drain, then ``unlink`` the md and watch the row evaporate."""
memory_root = cascade_runtime
_patch_embedding_capability(monkeypatch, _StubEmbedder())
orchestrator = CascadeOrchestrator(
memory_root=memory_root,
embedder=_StubEmbedder(),
tokenizer=build_tokenizer(),
config=CascadeConfig(
scan_interval_seconds=60.0,
+22 -4
View File
@@ -83,6 +83,12 @@ async def cascade_runtime(
monkeypatch.setenv("EVEROS_EMBEDDING__MODEL", "stub-model")
monkeypatch.setenv("EVEROS_EMBEDDING__BASE_URL", "http://stub.invalid/v1")
monkeypatch.setenv("EVEROS_EMBEDDING__API_KEY", "stub-key")
# Handlers fetch the embedder lazily via ``get_embedding_capability()``
# rather than through ``CascadeOrchestrator`` — patch the process-wide
# singleton so cascade never hits the fake network target above.
# ``test_lap_append_during_handler_no_loss`` re-patches with a slow
# variant to force a handler-in-flight race.
_patch_embedding_capability(monkeypatch, _StubEmbedder())
await dispose_connection()
await dispose_engine()
@@ -93,18 +99,27 @@ async def cascade_runtime(
await ensure_business_indexes()
(tmp_path / "ome.toml").write_text("# test\n")
yield MemoryRoot.default()
yield MemoryRoot.resolve()
await dispose_connection()
await dispose_engine()
def _patch_embedding_capability(
monkeypatch: pytest.MonkeyPatch, embedder: EmbeddingProvider
) -> None:
"""Patch the process-wide embedding capability singleton to *embedder*."""
import everos.component.embedding.accessor as acc
from everos.component.embedding import EmbeddingCapability
monkeypatch.setattr(acc, "_capability", EmbeddingCapability(provider=embedder))
def _build_orchestrator(
memory_root: MemoryRoot, *, scan_interval: float = 60.0
) -> CascadeOrchestrator:
return CascadeOrchestrator(
memory_root=memory_root,
embedder=_StubEmbedder(),
tokenizer=build_tokenizer(),
config=CascadeConfig(
scan_interval_seconds=scan_interval,
@@ -486,6 +501,7 @@ async def test_concurrent_writes_different_owners_no_bleed(
async def test_lap_append_during_handler_no_loss(
cascade_runtime: MemoryRoot,
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Writer keeps appending while worker is mid-handler.
@@ -500,9 +516,12 @@ async def test_lap_append_during_handler_no_loss(
await asyncio.sleep(0.05) # handler takes ~0.05*N entries
return [0.0] * self.dim
# Override the fixture's default stub — this test needs latency to
# force a handler invocation to overlap later writer appends.
_patch_embedding_capability(monkeypatch, _SlowEmbedder())
orchestrator = CascadeOrchestrator(
memory_root=memory_root,
embedder=_SlowEmbedder(),
tokenizer=build_tokenizer(),
config=CascadeConfig(
scan_interval_seconds=60.0,
@@ -564,7 +583,6 @@ def _build_orchestrator_fast_scanner(memory_root: MemoryRoot) -> CascadeOrchestr
don't wait 30s for the fallback path."""
return CascadeOrchestrator(
memory_root=memory_root,
embedder=_StubEmbedder(),
tokenizer=build_tokenizer(),
config=CascadeConfig(
scan_interval_seconds=2.0,
@@ -0,0 +1,315 @@
"""Integration tests for Task 24's ``cascade backfill`` polish pass.
Covers three behaviours layered on top of Task 20-23's scaffold + three
phase implementations:
- **Summary block**: after ``run_backfill`` finishes (or is interrupted),
a consolidated "Backfill summary" section is printed, one line per
phase that actually ran, plus a final ``Exit: <LABEL> (<code>)`` line.
- **Ctrl-C handling**: an interrupt raised from inside a phase body must
propagate up to ``run_backfill``'s own top-level handler (not be
swallowed mid-phase), which prints a resume hint and returns exit
code 130 (SIGINT convention: 128 + 2) instead of leaking a bare
traceback. Covered at two levels: a synchronous ``KeyboardInterrupt``
raise (cheap unit-level regression guard, but *not* representative of
a real terminal Ctrl-C — see below), and the actual mechanism a real
SIGINT uses under ``asyncio.run`` on Python 3.11+ — ``asyncio.Runner``
translates it into ``main_task.cancel()``, which raises
``asyncio.CancelledError`` (a ``BaseException``, not a
``KeyboardInterrupt`` subclass) at the currently suspended ``await``.
A bare ``except KeyboardInterrupt`` alone never sees that exception,
so the CancelledError path is exercised both by simulation (raising
``CancelledError`` from inside an ``await``) and by delivering a real
``SIGINT`` to the test process while a mocked phase is parked in
``asyncio.sleep``.
- **``--help`` polish**: the CLI help text names every ``--phase`` choice
and explains what ``--yes`` does.
Phase 1 (``vectors``) is reused for the "real counts" summary case
because it is the cheapest phase to drive end-to-end (mirrors
``backfill_runtime`` in ``test_backfill_phase1.py``); Phases 2/3 are
exercised in the all-zero case only, since Task 24 doesn't add any new
phase behaviour of its own.
"""
from __future__ import annotations
import asyncio
import hashlib
import os
import signal
import threading
import time
from collections.abc import AsyncIterator
from pathlib import Path
import pytest
from typer.testing import CliRunner
from everos.component.embedding import EmbeddingCapability, EmbeddingProvider
from everos.component.utils.datetime import get_utc_now
from everos.config import load_settings
from everos.entrypoints.cli.commands import cascade as cascade_mod
from everos.entrypoints.cli.commands._backfill_cmd import run_backfill
from everos.infra.persistence.lancedb import Episode, dispose_connection, episode_repo
from everos.memory.cascade import _backfill as backfill_mod
_DIM = 1024
class _StubEmbedder(EmbeddingProvider):
dim = _DIM
async def embed(self, text: str) -> list[float]:
return [float(len(text) % 7)] * self.dim
async def embed_batch(self, texts): # type: ignore[no-untyped-def]
return [[float(len(t) % 7)] * self.dim for t in texts]
@pytest.fixture
async def backfill_runtime(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> AsyncIterator[Path]:
"""Tmp memory root + stub embedder (mirrors ``test_backfill_phase1.py``)."""
monkeypatch.setenv("EVEROS_ROOT", str(tmp_path))
load_settings.cache_clear()
await dispose_connection()
import everos.component.embedding.accessor as acc
monkeypatch.setattr(
acc, "_capability", EmbeddingCapability(provider=_StubEmbedder())
)
yield tmp_path
await dispose_connection()
def _episode(entry_id: str) -> Episode:
return Episode(
id=f"u1_{entry_id}",
entry_id=entry_id,
owner_id="u1",
owner_type="user",
timestamp=get_utc_now(),
parent_id="mc1",
sender_ids=["u1"],
episode=f"episode body {entry_id}",
episode_tokens=f"episode body {entry_id}",
md_path="users/u1/episodes/episode-2026-01-01.md",
content_sha256=hashlib.sha256(entry_id.encode()).hexdigest(),
vector=None,
)
# ── summary block ────────────────────────────────────────────────────────────
async def test_summary_block_all_zero_on_empty_db(
backfill_runtime: Path, capsys: pytest.CaptureFixture[str]
) -> None:
code = await run_backfill(phase="all", auto_yes=True)
out = capsys.readouterr().out
assert code == 0
assert "Backfill summary" in out
assert "Phase 1 (vectors)" in out
assert "0 rows / 0 failed / 0 tokens" in out
assert "Phase 2 (clusters)" in out
assert "0 events emitted / 0 clusters created" in out
assert "Phase 3 (skills)" in out
assert "0 agent cases processed / 0 skills extracted" in out
assert "Exit: SUCCESS (0)" in out
async def test_summary_block_reports_real_counts_for_ran_phase_only(
backfill_runtime: Path, capsys: pytest.CaptureFixture[str]
) -> None:
await episode_repo.add([_episode("ep1"), _episode("ep2")])
code = await run_backfill(phase="vectors", auto_yes=True)
out = capsys.readouterr().out
assert code == 0
assert "Backfill summary" in out
assert "Phase 1 (vectors)" in out
assert "2 rows / 0 failed" in out
# Only the phase that actually ran gets a summary line.
assert "Phase 2 (clusters)" not in out
assert "Phase 3 (skills)" not in out
assert "Exit: SUCCESS (0)" in out
# ── Ctrl-C handling ──────────────────────────────────────────────────────────
async def test_synchronous_keyboard_interrupt_mid_phase_returns_130_with_resume_hint(
backfill_runtime: Path,
monkeypatch: pytest.MonkeyPatch,
capsys: pytest.CaptureFixture[str],
) -> None:
"""Cheap regression guard for the raw ``except KeyboardInterrupt`` catch.
This raises synchronously (no ``await`` in between), so the
exception lands directly in ``run_backfill``'s own frame — it does
*not* exercise ``asyncio.Runner``'s SIGINT-to-``CancelledError``
translation that a real terminal Ctrl-C goes through. See
``test_cancelled_error_mid_phase_returns_130_with_resume_hint`` and
``test_real_sigint_during_phase_await_returns_130_with_resume_hint``
below for that path.
"""
async def _raise_interrupt(*_args: object, **_kwargs: object) -> None:
raise KeyboardInterrupt
monkeypatch.setattr(backfill_mod, "_run_phase_vectors", _raise_interrupt)
code = await run_backfill(phase="all", auto_yes=True)
out = capsys.readouterr().out
assert code == 130
assert "Interrupted" in out
assert "partial progress was written" in out
assert "Resume by running" in out
assert "everos cascade backfill --phase <phase-name> --yes" in out
assert "vectors, clusters, skills, all" in out
# The summary block still prints on interrupt, reflecting no progress yet.
assert "Backfill summary" in out
assert "Exit: INTERRUPTED (130)" in out
async def test_cancelled_error_mid_phase_returns_130_with_resume_hint(
backfill_runtime: Path,
monkeypatch: pytest.MonkeyPatch,
capsys: pytest.CaptureFixture[str],
) -> None:
"""Simulates what ``asyncio.Runner`` actually does on a real SIGINT.
On Python 3.11+, ``asyncio.Runner`` (which backs ``asyncio.run``)
turns a terminal Ctrl-C into ``main_task.cancel()``, which raises
``asyncio.CancelledError`` — a ``BaseException``, never a
``KeyboardInterrupt`` subclass — at the task's currently suspended
``await``. Raising it from inside an ``await`` (rather than
synchronously) reproduces that mechanism; a bare
``except KeyboardInterrupt`` alone would miss it entirely and let it
escape ``run_backfill`` uncaught.
"""
async def _raise_cancelled(*_args: object, **_kwargs: object) -> None:
await asyncio.sleep(0)
raise asyncio.CancelledError
monkeypatch.setattr(backfill_mod, "_run_phase_vectors", _raise_cancelled)
code = await run_backfill(phase="all", auto_yes=True)
out = capsys.readouterr().out
assert code == 130
assert "Interrupted" in out
assert "partial progress was written" in out
assert "Resume by running" in out
assert "Backfill summary" in out
assert "Exit: INTERRUPTED (130)" in out
def test_backfill_cli_synchronous_keyboard_interrupt_exits_130(
backfill_runtime: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""CLI-level counterpart of the synchronous-raise regression guard.
Same caveat as its ``run_backfill``-level twin above: this does not
exercise ``asyncio.Runner``'s SIGINT translation. See
``test_real_sigint_during_phase_await_returns_130_with_resume_hint``
for the real-signal end-to-end path.
"""
async def _raise_interrupt(*_args: object, **_kwargs: object) -> None:
raise KeyboardInterrupt
monkeypatch.setattr(backfill_mod, "_run_phase_vectors", _raise_interrupt)
result = CliRunner().invoke(
cascade_mod.app, ["backfill", "--phase", "vectors", "--yes"]
)
assert result.exit_code == 130
@pytest.mark.slow
def test_real_sigint_during_phase_await_returns_130_with_resume_hint(
backfill_runtime: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""End-to-end: a real terminal Ctrl-C, delivered as an actual SIGINT.
Exercises the path where on Python 3.11+, ``asyncio.Runner`` (which
backs ``asyncio.run``, used by the ``backfill`` CLI command)
converts a real SIGINT into ``main_task.cancel()``, raising
``asyncio.CancelledError`` — not ``KeyboardInterrupt`` — at the
suspended ``await``. A background thread sends a real ``SIGINT`` to
this test process while a mocked phase is parked in
``asyncio.sleep``, reproducing that exact mechanism rather than a
synchronous stand-in.
Runs synchronously (not ``async def``): ``pytest-asyncio`` drives
its tests off a bare event loop via ``run_until_complete``, not
``asyncio.Runner``, so it would never install the SIGINT handler
under test. The CLI command's own ``asyncio.run()`` call is what
must install it, so this test must invoke the CLI command directly.
Marked ``slow`` — and therefore excluded from the default
``-m "not slow"`` run — because the ``0.3s`` timer that sends the
signal races ``asyncio.Runner``'s SIGINT-handler installation. On
a fast host the handler is in place well before ``0.3s`` elapses;
on a heavily loaded CI runner the signal can land on pytest's
default handler and kill the test process. Opt-in execution keeps
the guard available for local repro without destabilising CI.
"""
async def _slow_phase(*_args: object, **_kwargs: object) -> None:
await asyncio.sleep(5)
monkeypatch.setattr(backfill_mod, "_run_phase_vectors", _slow_phase)
def _send_sigint_soon() -> None:
time.sleep(0.3)
os.kill(os.getpid(), signal.SIGINT)
threading.Thread(target=_send_sigint_soon, daemon=True).start()
result = CliRunner().invoke(
cascade_mod.app, ["backfill", "--phase", "vectors", "--yes"]
)
assert result.exit_code == 130
assert "Interrupted" in result.stdout
assert "Resume by running" in result.stdout
# ── --help polish ────────────────────────────────────────────────────────────
def test_backfill_help_documents_every_phase_name() -> None:
result = CliRunner().invoke(cascade_mod.app, ["backfill", "--help"])
assert result.exit_code == 0
for phase_name in ("vectors", "clusters", "skills", "all"):
assert phase_name in result.stdout
def test_backfill_help_documents_yes_flag_behavior() -> None:
result = CliRunner().invoke(cascade_mod.app, ["backfill", "--help"])
assert result.exit_code == 0
assert "confirm" in result.stdout.lower()
def test_backfill_help_documents_resumption() -> None:
result = CliRunner().invoke(cascade_mod.app, ["backfill", "--help"])
assert result.exit_code == 0
assert "resum" in result.stdout.lower()
def test_backfill_help_documents_cost_estimate_policy() -> None:
result = CliRunner().invoke(cascade_mod.app, ["backfill", "--help"])
assert result.exit_code == 0
assert "token" in result.stdout.lower()
assert "price" in result.stdout.lower() or "pricing" in result.stdout.lower()
@@ -0,0 +1,263 @@
"""Integration test for ``everos cascade backfill --phase vectors``.
Exercises the real Phase 1 re-embed path against actual LanceDB tables
under a tmp memory root: seeds rows with ``vector IS NULL``, drives
``run_backfill(phase="vectors", ...)``, and asserts the rows now carry
a vector. A stub embedder replaces the process-wide embedding
capability singleton so no real network call ever happens (mirrors the
pattern in ``tests/integration/test_cascade_all_kinds_consistency.py``).
Covers: empty DB (nothing to backfill), declined confirmation (nothing
written, exit 1), a small DB with rows that need backfilling (rows
updated, pre-existing vectors left untouched, token estimate rendered
in K/M notation), a batch-embed failure that is tallied without
aborting the phase, an episode's ``subject_vector`` secondary embed,
and a missing embedding capability returning exit 2.
"""
from __future__ import annotations
import hashlib
from collections.abc import AsyncIterator
from pathlib import Path
import pytest
from everos.component.embedding import EmbeddingCapability, EmbeddingProvider
from everos.component.utils.datetime import get_utc_now
from everos.config import load_settings
from everos.entrypoints.cli.commands._backfill_cmd import run_backfill
from everos.infra.persistence.lancedb import (
AtomicFact,
Episode,
atomic_fact_repo,
dispose_connection,
episode_repo,
)
_DIM = 1024
class _StubEmbedder(EmbeddingProvider):
dim = _DIM
async def embed(self, text: str) -> list[float]:
return [float(len(text) % 7)] * self.dim
async def embed_batch(self, texts): # type: ignore[no-untyped-def]
return [[float(len(t) % 7)] * self.dim for t in texts]
@pytest.fixture
async def backfill_runtime(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> AsyncIterator[Path]:
"""Tmp memory root + stub embedder; disposes the lancedb singleton
around the test so no stale connection from a neighbouring test
leaks in (mirrors ``cascade_runtime`` in the cascade test suite)."""
monkeypatch.setenv("EVEROS_ROOT", str(tmp_path))
load_settings.cache_clear()
await dispose_connection()
import everos.component.embedding.accessor as acc
monkeypatch.setattr(
acc, "_capability", EmbeddingCapability(provider=_StubEmbedder())
)
yield tmp_path
await dispose_connection()
def _episode(
entry_id: str,
*,
text: str = "",
vector: list[float] | None = None,
subject: str | None = None,
subject_vector: list[float] | None = None,
) -> Episode:
body = text or f"episode body {entry_id}"
return Episode(
id=f"u1_{entry_id}",
entry_id=entry_id,
owner_id="u1",
owner_type="user",
timestamp=get_utc_now(),
parent_id="mc1",
sender_ids=["u1"],
subject=subject,
episode=body,
episode_tokens=body,
md_path="users/u1/episodes/episode-2026-01-01.md",
content_sha256=hashlib.sha256(entry_id.encode()).hexdigest(),
vector=vector,
subject_vector=subject_vector,
)
def _fact(entry_id: str, *, vector: list[float] | None = None) -> AtomicFact:
return AtomicFact(
id=f"u1_{entry_id}",
entry_id=entry_id,
owner_id="u1",
owner_type="user",
timestamp=get_utc_now(),
parent_id="mc1",
sender_ids=["u1"],
fact=f"fact body {entry_id}",
fact_tokens=f"fact body {entry_id}",
md_path="users/u1/.atomic_facts/atomic_fact-2026-01-01.md",
content_sha256=hashlib.sha256(entry_id.encode()).hexdigest(),
vector=vector,
)
async def test_empty_db_reports_nothing_to_backfill(
backfill_runtime: Path, capsys: pytest.CaptureFixture[str]
) -> None:
code = await run_backfill(phase="vectors", auto_yes=True)
out = capsys.readouterr().out
assert code == 0
assert "Nothing to backfill" in out
async def test_declined_confirmation_writes_nothing_and_exits_one(
backfill_runtime: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
await episode_repo.add([_episode("ep1")])
monkeypatch.setattr("typer.confirm", lambda *a, **k: False)
code = await run_backfill(phase="vectors", auto_yes=False)
assert code == 1
row = await episode_repo.get_by_id("u1_ep1")
assert row is not None
assert row.vector is None
async def test_small_db_backfills_null_vector_rows_only(
backfill_runtime: Path, capsys: pytest.CaptureFixture[str]
) -> None:
pre_embedded = [0.5] * _DIM
await episode_repo.add([_episode("ep1"), _episode("ep2", vector=pre_embedded)])
await atomic_fact_repo.add([_fact("af1")])
code = await run_backfill(phase="vectors", auto_yes=True)
out = capsys.readouterr().out
assert code == 0
ep1 = await episode_repo.get_by_id("u1_ep1")
ep2 = await episode_repo.get_by_id("u1_ep2")
af1 = await atomic_fact_repo.get_by_id("u1_af1")
assert ep1 is not None and ep1.vector is not None
assert ep2 is not None and ep2.vector == pytest.approx(pre_embedded)
assert af1 is not None and af1.vector is not None
assert "memories to process: 2" in out
assert "phase 1 complete" in out
async def test_token_estimate_uses_kilo_notation(
backfill_runtime: Path, capsys: pytest.CaptureFixture[str]
) -> None:
long_text = "word " * 3000
await episode_repo.add([_episode("epbig", text=long_text)])
code = await run_backfill(phase="vectors", auto_yes=True)
out = capsys.readouterr().out
assert code == 0
assert "input tokens:" in out
assert "K" in out
async def test_failed_row_is_tallied_and_does_not_abort_the_phase(
backfill_runtime: Path,
monkeypatch: pytest.MonkeyPatch,
capsys: pytest.CaptureFixture[str],
) -> None:
"""Round-2 finding M8: a batch failure now falls back to per-row
``embed(...)`` calls (see ``_embed_primary_batch``). A poison row —
one whose per-row retry also fails — must (a) not abort the phase,
(b) not touch the healthy rows sharing its batch, and (c) drive the
final exit code to ``4`` (COMPLETED_WITH_FAILURES) rather than ``0``
so automation can distinguish clean success from partial success.
``_FlakyEmbedder`` fails BOTH ``embed_batch`` and ``embed`` on the
poison text ("boom"), so the row remains unwritable after fallback.
Healthy rows still succeed via the per-row retry path.
"""
import everos.component.embedding.accessor as acc
from everos.component.embedding import EmbeddingCapability
class _FlakyEmbedder(_StubEmbedder):
async def embed_batch(self, texts: list[str]) -> list[list[float]]:
if any("boom" in t for t in texts):
raise RuntimeError("simulated batch embed failure")
return await super().embed_batch(texts)
async def embed(self, text: str) -> list[float]:
if "boom" in text:
raise RuntimeError("simulated per-row embed failure")
return await super().embed(text)
monkeypatch.setattr(
acc, "_capability", EmbeddingCapability(provider=_FlakyEmbedder())
)
await episode_repo.add(
[_episode("epok", text="fine content"), _episode("epbad", text="boom content")]
)
code = await run_backfill(phase="vectors", auto_yes=True)
out = capsys.readouterr().out
assert code == 4
ok_row = await episode_repo.get_by_id("u1_epok")
bad_row = await episode_repo.get_by_id("u1_epbad")
assert ok_row is not None and ok_row.vector is not None
assert bad_row is not None and bad_row.vector is None
assert "phase 1 complete" in out
assert "COMPLETED_WITH_FAILURES" in out
assert "1 rows failed embedding" in out
async def test_episode_subject_vector_also_backfilled(
backfill_runtime: Path, capsys: pytest.CaptureFixture[str]
) -> None:
"""Episode is the only table with a second, independent embed
(``subject_vector`` — see ``EpisodeHandler._build_row``). A row that
has a ``subject`` must come out of Phase 1 with *both* vectors set,
exercising ``_embed_subject_batch``'s secondary ``embed_batch`` call.
"""
await episode_repo.add(
[_episode("epsub", text="episode body", subject="What is the meaning of X?")]
)
code = await run_backfill(phase="vectors", auto_yes=True)
out = capsys.readouterr().out
assert code == 0
row = await episode_repo.get_by_id("u1_epsub")
assert row is not None
assert row.vector is not None and len(row.vector) == _DIM
assert row.subject_vector is not None and len(row.subject_vector) == _DIM
assert "phase 1 complete" in out
async def test_missing_embedding_capability_returns_exit_two(
backfill_runtime: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""When the embedding capability is unavailable (no provider
configured), Phase 1 must fail cleanly with exit 2 rather than
silently skipping rows or crashing uncaught."""
import everos.component.embedding.accessor as acc
await episode_repo.add([_episode("ep1")])
monkeypatch.setattr(acc, "_capability", EmbeddingCapability(provider=None))
code = await run_backfill(phase="vectors", auto_yes=True)
assert code == 2
row = await episode_repo.get_by_id("u1_ep1")
assert row is not None and row.vector is None
@@ -0,0 +1,314 @@
"""Integration test for ``everos cascade backfill --phase clusters``.
Exercises the real Phase 2 scan + synthetic-event-emission path (see
``memory.cascade._backfill._run_phase_clusters``): seeds real episode /
agent-case rows in LanceDB, drives ``run_backfill(phase="clusters", ...)``,
and asserts the phase synthesizes exactly one ``EpisodeExtracted`` /
``AgentCaseExtracted`` event per row and fans them into the OME engine.
The OME engine itself is replaced with an in-memory spy
(``_FakeClusterEngine``, injected via monkeypatching
``_backfill._build_cluster_engine``) so this test exercises the scan +
event-synthesis contract without paying for a real APScheduler + sqlite
jobstore runtime — that end-to-end wiring is already covered by
``tests/integration/test_ome_strategies_integration.py``.
Covers: empty DB (nothing to backfill, no engine touched), declined
confirmation (no events emitted, exit 1), a small DB with episodes +
agent cases (correct event types / counts / fields, exit 0), and a
missing embedding capability (exit 2, mirroring Phase 1's guard).
One test (``test_real_offline_engine_grows_cluster_count``) does NOT
install the fake spy: it drives Phase 2 through the actual
``_build_cluster_engine`` path — a real ``OfflineEngine`` + APScheduler
runtime with ``trigger_profile_clustering`` registered — and asserts
``cluster_repo.count()`` grows. This is the one check the fake-engine
tests above cannot make: they prove the scan + event-synthesis contract,
but never exercise whether the real engine wiring actually drives a
cluster into existence.
"""
from __future__ import annotations
import hashlib
from collections.abc import AsyncIterator
from pathlib import Path
import pytest
from everos.component.embedding import EmbeddingCapability, EmbeddingProvider
from everos.component.utils.datetime import get_utc_now
from everos.config import load_settings
from everos.entrypoints.cli.commands._backfill_cmd import run_backfill
from everos.infra.ome.events import BaseEvent
from everos.infra.persistence.lancedb import (
AgentCase,
Episode,
agent_case_repo,
dispose_connection,
episode_repo,
)
from everos.infra.persistence.sqlite import cluster_repo, sqlite_manager
from everos.memory.cascade import _backfill as backfill_mod
from everos.memory.events import AgentCaseExtracted, EpisodeExtracted
_DIM = 1024
class _StubEmbedder(EmbeddingProvider):
dim = _DIM
async def embed(self, text: str) -> list[float]:
return [float(len(text) % 7)] * self.dim
async def embed_batch(self, texts): # type: ignore[no-untyped-def]
return [[float(len(t) % 7)] * self.dim for t in texts]
class _FakeClusterEngine:
"""Spy replacing the real OME engine: records emitted events instead of
dispatching them through APScheduler."""
def __init__(self) -> None:
self.emitted: list[BaseEvent] = []
self.started = False
self.stopped = False
async def start(self) -> None:
self.started = True
async def emit(self, event: BaseEvent) -> None:
self.emitted.append(event)
async def wait_idle(self, *, timeout: float = 30.0) -> bool: # noqa: ASYNC109
return True
async def stop(self) -> None:
self.stopped = True
@pytest.fixture
async def backfill_runtime(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> AsyncIterator[Path]:
"""Tmp memory root + stub embedder + isolated sqlite engine.
Phase 2 touches both LanceDB (episode / agent_case scan) and sqlite
(``cluster_repo.count()``), so both process-wide singletons are
disposed and rebuilt against ``tmp_path`` around each test — mirrors
``backfill_runtime`` in ``test_backfill_phase1.py`` plus the sqlite
reset from ``test_ome_strategies_integration.py``.
"""
monkeypatch.setenv("EVEROS_ROOT", str(tmp_path))
load_settings.cache_clear()
await dispose_connection()
if sqlite_manager._engine is not None:
await sqlite_manager.dispose_engine()
monkeypatch.setattr(sqlite_manager, "_engine", None, raising=False)
monkeypatch.setattr(sqlite_manager, "_session_factory", None, raising=False)
import everos.component.embedding.accessor as acc
monkeypatch.setattr(
acc, "_capability", EmbeddingCapability(provider=_StubEmbedder())
)
yield tmp_path
await dispose_connection()
if sqlite_manager._engine is not None:
await sqlite_manager.dispose_engine()
@pytest.fixture
def fake_engine(monkeypatch: pytest.MonkeyPatch) -> _FakeClusterEngine:
"""Install the spy engine in place of ``_build_cluster_engine``."""
engine = _FakeClusterEngine()
monkeypatch.setattr(backfill_mod, "_build_cluster_engine", lambda: engine)
return engine
def _episode(entry_id: str, *, owner_id: str = "u1") -> Episode:
return Episode(
id=f"{owner_id}_{entry_id}",
entry_id=entry_id,
owner_id=owner_id,
owner_type="user",
session_id="s1",
timestamp=get_utc_now(),
parent_id=f"mc_{entry_id}",
sender_ids=[owner_id],
episode=f"episode body {entry_id}",
episode_tokens=f"episode body {entry_id}",
md_path="users/u1/episodes/episode-2026-01-01.md",
content_sha256=hashlib.sha256(entry_id.encode()).hexdigest(),
vector=[0.1] * _DIM,
)
def _agent_case(entry_id: str, *, agent_id: str = "agent1") -> AgentCase:
return AgentCase(
id=f"{agent_id}_{entry_id}",
entry_id=entry_id,
owner_id=agent_id,
owner_type="agent",
session_id="s1",
timestamp=get_utc_now(),
parent_id=f"mc_{entry_id}",
quality_score=0.8,
task_intent=f"task intent {entry_id}",
task_intent_tokens=f"task intent {entry_id}",
approach=f"approach {entry_id}",
approach_tokens=f"approach {entry_id}",
md_path="agents/agent1/.cases/agent_case-2026-01-01.md",
content_sha256=hashlib.sha256(entry_id.encode()).hexdigest(),
vector=[0.1] * _DIM,
)
async def test_empty_db_reports_nothing_to_backfill(
backfill_runtime: Path,
fake_engine: _FakeClusterEngine,
capsys: pytest.CaptureFixture[str],
) -> None:
code = await run_backfill(phase="clusters", auto_yes=True)
out = capsys.readouterr().out
assert code == 0
assert "Nothing to backfill" in out
assert fake_engine.emitted == []
assert fake_engine.started is False
async def test_declined_confirmation_emits_nothing_and_exits_one(
backfill_runtime: Path,
fake_engine: _FakeClusterEngine,
monkeypatch: pytest.MonkeyPatch,
) -> None:
await episode_repo.add([_episode("ep1")])
monkeypatch.setattr("typer.confirm", lambda *a, **k: False)
code = await run_backfill(phase="clusters", auto_yes=False)
assert code == 1
assert fake_engine.emitted == []
assert fake_engine.started is False
async def test_episodes_and_cases_synthesize_matching_events(
backfill_runtime: Path,
fake_engine: _FakeClusterEngine,
capsys: pytest.CaptureFixture[str],
) -> None:
await episode_repo.add([_episode("ep1"), _episode("ep2")])
await agent_case_repo.add([_agent_case("ac1")])
code = await run_backfill(phase="clusters", auto_yes=True)
out = capsys.readouterr().out
assert code == 0
assert fake_engine.started is True
assert fake_engine.stopped is True
assert len(fake_engine.emitted) == 3
episode_events = [e for e in fake_engine.emitted if isinstance(e, EpisodeExtracted)]
case_events = [e for e in fake_engine.emitted if isinstance(e, AgentCaseExtracted)]
assert len(episode_events) == 2
assert len(case_events) == 1
assert all(e.source == "pipeline" for e in episode_events)
assert all(e.event_id.startswith("backfill_") for e in fake_engine.emitted)
assert {e.owner_id for e in episode_events} == {"u1"}
assert case_events[0].agent_id == "agent1"
assert case_events[0].task_intent == "task intent ac1"
assert case_events[0].quality_score == pytest.approx(0.8)
assert "phase 2 complete" in out
async def test_missing_embedding_capability_returns_exit_two(
backfill_runtime: Path,
fake_engine: _FakeClusterEngine,
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Mirrors Phase 1's own guard: no embedding provider configured means
the strategies can't re-embed, so the phase fails clean with exit 2
rather than emitting events into strategies that will crash."""
import everos.component.embedding.accessor as acc
await episode_repo.add([_episode("ep1")])
monkeypatch.setattr(acc, "_capability", EmbeddingCapability(provider=None))
code = await run_backfill(phase="clusters", auto_yes=True)
assert code == 2
assert fake_engine.emitted == []
assert fake_engine.started is False
async def test_reflection_merged_episodes_excluded_from_synthesis(
backfill_runtime: Path,
fake_engine: _FakeClusterEngine,
) -> None:
"""Reflection-merged episodes (``parent_type="cluster"``) must not be
re-synthesized as pipeline events.
``trigger_profile_clustering`` excludes Reflection output via
``applies_to=lambda e: e.source == "pipeline"`` — but that guard is
defeated if the Episode scan hands it a synthetic event whose
``memcell_id`` is actually a cluster id (see Important 1 review
finding on ``_backfill._scan_all_rows`` / ``_run_phase_clusters``).
Only the pipeline-sourced row should reach the fake engine.
"""
pipeline_episode = _episode("ep1")
merged_episode = _episode("ep_merged").model_copy(
update={
"id": "u1_ep_merged",
"parent_type": "cluster",
"parent_id": "cluster_abc123",
}
)
await episode_repo.add([pipeline_episode, merged_episode])
code = await run_backfill(phase="clusters", auto_yes=True)
assert code == 0
episode_events = [e for e in fake_engine.emitted if isinstance(e, EpisodeExtracted)]
assert len(episode_events) == 1
assert episode_events[0].memcell_id == pipeline_episode.parent_id
async def test_real_offline_engine_grows_cluster_count(
backfill_runtime: Path,
) -> None:
"""Phase 2 through the REAL ``_build_cluster_engine`` — no fake spy.
Deliberately does not use the ``fake_engine`` fixture: it drives an
actual ``OfflineEngine`` + APScheduler runtime with
``trigger_profile_clustering`` registered (see
``_backfill._build_cluster_engine``), proving the phase's own engine
wiring — not just the scan/synthesis contract the other tests in
this module pin — actually grows ``cluster_repo.count()``.
``trigger_profile_clustering`` re-embeds each episode's text via
``get_embedding_capability().require()`` — which ``backfill_runtime``
already stubs with the deterministic ``_StubEmbedder``, so no
per-strategy patch is needed here. No agent cases are seeded, so
``trigger_skill_clustering`` (the engine's other registered
strategy, whose ``cluster_by_llm`` fallback needs an LLM) never
fires, keeping this test LLM-free.
"""
await episode_repo.add([_episode(f"ep{i}") for i in range(5)])
# ``run_backfill`` itself creates the ``cluster`` table lazily (see
# ``_ensure_cluster_schema``) — call it here too so the "before" count
# can be taken without racing that lazy create.
await backfill_mod._ensure_cluster_schema()
clusters_before = await cluster_repo.count()
assert clusters_before == 0
code = await run_backfill(phase="clusters", auto_yes=True)
clusters_after = await cluster_repo.count()
assert code == 0
assert clusters_after > clusters_before
@@ -0,0 +1,257 @@
"""Integration test for ``everos cascade backfill --phase skills``.
Exercises the real Phase 3 scan + synthetic-event-emission + cascade-sync
path (see ``memory.cascade._backfill._run_phase_skills``): Phase 2's
throwaway engine (``_build_cluster_engine``, Task 22) registers
``trigger_skill_clustering`` but never ``extract_agent_skill``, so every
``SkillClusterUpdated`` it emits during clustering has no listener and is
dropped. Phase 3 walks every agent-case cluster
(``cluster_repo.list_distinct_owners`` + ``list_for_owner``) and replays
one ``SkillClusterUpdated`` per clustered case through a second throwaway
engine that DOES register ``extract_agent_skill``, then runs a one-shot
cascade sync (``CascadeOrchestrator.sync_once``) so the freshly-written
``SKILL.md`` lands in LanceDB before the phase reports its skill count.
Only ``AgentSkillExtractor`` (the LLM call) and ``get_llm_client`` are
mocked — everything else (cluster lookup, agent-case lookup, the
markdown writer, and the cascade scan/handler/embed round trip) is real,
so ``test_skills_phase_grows_agent_skill_count`` is a genuine end-to-end
proof that Phase 3 leaves ``agent_skill_repo.count()`` already grown,
not just eventually-consistent.
Covers: empty DB (nothing to backfill, no engine touched), declined
confirmation (exit 1, no skill written), a small DB with one clustered
agent case (skill extracted + indexed, exit 0), and a missing embedding
capability (exit 2, mirroring Phase 1 / Phase 2's own guard).
"""
from __future__ import annotations
import hashlib
import importlib
from collections.abc import AsyncIterator, Iterator
from pathlib import Path
from unittest.mock import AsyncMock, patch
import numpy as np
import pytest
from everalgo.clustering import Cluster as AlgoCluster
from everalgo.types import AgentSkill as AlgoAgentSkill
from everos.component.embedding import EmbeddingCapability, EmbeddingProvider
from everos.component.utils.datetime import get_utc_now
from everos.config import load_settings
from everos.entrypoints.cli.commands._backfill_cmd import run_backfill
from everos.infra.persistence.lancedb import (
AgentCase,
agent_case_repo,
agent_skill_repo,
dispose_connection,
)
from everos.infra.persistence.sqlite import (
cluster_repo,
mint_cluster_id,
sqlite_manager,
)
from everos.memory.cascade import _backfill as backfill_mod
_DIM = 1024
class _StubEmbedder(EmbeddingProvider):
dim = _DIM
async def embed(self, text: str) -> list[float]:
return [float(len(text) % 7)] * self.dim
async def embed_batch(self, texts): # type: ignore[no-untyped-def]
return [[float(len(t) % 7)] * self.dim for t in texts]
@pytest.fixture
async def backfill_runtime(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> AsyncIterator[Path]:
"""Tmp memory root + stub embedder + isolated sqlite engine.
Phase 3 touches LanceDB (agent_case / agent_skill), sqlite (cluster
tables + md_change_state), and the real filesystem (SKILL.md write +
cascade scan) — mirrors ``backfill_runtime`` in
``test_backfill_phase2.py``, plus resetting the strategy module's
``_writer`` singleton so it re-resolves against ``tmp_path``.
"""
monkeypatch.setenv("EVEROS_ROOT", str(tmp_path))
load_settings.cache_clear()
await dispose_connection()
if sqlite_manager._engine is not None:
await sqlite_manager.dispose_engine()
monkeypatch.setattr(sqlite_manager, "_engine", None, raising=False)
monkeypatch.setattr(sqlite_manager, "_session_factory", None, raising=False)
import everos.component.embedding.accessor as acc
monkeypatch.setattr(
acc, "_capability", EmbeddingCapability(provider=_StubEmbedder())
)
skill_mod = importlib.import_module("everos.memory.strategies.extract_agent_skill")
monkeypatch.setattr(skill_mod, "_writer", None, raising=False)
yield tmp_path
await dispose_connection()
if sqlite_manager._engine is not None:
await sqlite_manager.dispose_engine()
@pytest.fixture
def stub_llm_extraction(
monkeypatch: pytest.MonkeyPatch,
) -> Iterator[list[AlgoAgentSkill]]:
"""Mock the one non-deterministic seam: the LLM-backed extractor.
Everything downstream of ``AgentSkillExtractor.aextract`` (writer,
cascade scan, embed, LanceDB upsert) stays real.
"""
emitted = [
AlgoAgentSkill(
id="dummy",
cluster_id="",
name="summarise_doc",
description="how to summarise a document",
content="step 1: read; step 2: condense",
confidence=0.7,
maturity_score=0.5,
source_case_ids=["ac1"],
)
]
# ``strategies/__init__.py`` re-exports the decorated ``Strategy``
# object under this same dotted name, shadowing the submodule as a
# package attribute — ``importlib.import_module`` (not
# ``import ... as``) is required to reach the actual module and
# patch its module-level references (mirrors
# ``test_ome_strategies_integration.test_real_offline_engine_grows_
# cluster_count``).
skill_mod = importlib.import_module("everos.memory.strategies.extract_agent_skill")
monkeypatch.setattr(skill_mod, "get_llm_client", lambda: object())
with patch.object(skill_mod, "AgentSkillExtractor") as mock_extractor_cls:
mock_extractor_cls.return_value.aextract = AsyncMock(return_value=emitted)
yield emitted
def _agent_case(entry_id: str, *, agent_id: str = "agent1") -> AgentCase:
return AgentCase(
id=f"{agent_id}_{entry_id}",
entry_id=entry_id,
owner_id=agent_id,
owner_type="agent",
session_id="s1",
timestamp=get_utc_now(),
parent_id=f"mc_{entry_id}",
quality_score=0.8,
task_intent=f"task intent {entry_id}",
task_intent_tokens=f"task intent {entry_id}",
approach=f"approach {entry_id}",
approach_tokens=f"approach {entry_id}",
md_path=f"agents/default/default/{agent_id}/.cases/agent_case-2026-01-01.md",
content_sha256=hashlib.sha256(entry_id.encode()).hexdigest(),
vector=[0.1] * _DIM,
)
async def _seed_cluster(*, agent_id: str = "agent1", case_entry_id: str = "ac1") -> str:
"""Seed the sqlite ``cluster`` + ``cluster_member`` pair Phase 3 scans."""
await backfill_mod._ensure_cluster_schema()
cluster_id = mint_cluster_id()
algo_cluster = AlgoCluster(
id=cluster_id,
centroid=np.zeros(_DIM, dtype=np.float32),
count=1,
last_ts=1_700_000_000_000,
preview=[f"task intent {case_entry_id}"],
members=[case_entry_id],
)
await cluster_repo.upsert_with_members(
algo_cluster,
owner_id=agent_id,
owner_type="agent",
kind="agent_case",
member_type="case",
)
return cluster_id
async def test_empty_db_reports_nothing_to_backfill(
backfill_runtime: Path,
capsys: pytest.CaptureFixture[str],
) -> None:
await backfill_mod._ensure_cluster_schema()
code = await run_backfill(phase="skills", auto_yes=True)
out = capsys.readouterr().out
assert code == 0
assert "Nothing to backfill" in out
assert await agent_skill_repo.count() == 0
async def test_declined_confirmation_writes_nothing_and_exits_one(
backfill_runtime: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
await agent_case_repo.add([_agent_case("ac1")])
await _seed_cluster()
monkeypatch.setattr("typer.confirm", lambda *a, **k: False)
code = await run_backfill(phase="skills", auto_yes=False)
assert code == 1
assert await agent_skill_repo.count() == 0
async def test_missing_embedding_capability_returns_exit_two(
backfill_runtime: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Mirrors Phase 1 / Phase 2's own guard: no embedding provider means
``extract_agent_skill``'s own top-K cosine ranking (and the cascade
handler's re-embed on write-back) can't run, so the phase fails
clean with exit 2 rather than emitting events into a broken pipeline."""
import everos.component.embedding.accessor as acc
await agent_case_repo.add([_agent_case("ac1")])
await _seed_cluster()
monkeypatch.setattr(acc, "_capability", EmbeddingCapability(provider=None))
code = await run_backfill(phase="skills", auto_yes=True)
assert code == 2
assert await agent_skill_repo.count() == 0
async def test_skills_phase_grows_agent_skill_count(
backfill_runtime: Path,
stub_llm_extraction: list[AlgoAgentSkill],
capsys: pytest.CaptureFixture[str],
) -> None:
"""End-to-end: cluster with no skill yet → Phase 3 replays the event
the clustering pass never got a listener for → real ``SKILL.md``
write → real cascade sync → ``agent_skill_repo.count()`` grows."""
await agent_case_repo.add([_agent_case("ac1")])
await _seed_cluster()
skills_before = await agent_skill_repo.count()
assert skills_before == 0
code = await run_backfill(phase="skills", auto_yes=True)
out = capsys.readouterr().out
skills_after = await agent_skill_repo.count()
assert code == 0
assert skills_after == skills_before + 1
assert "phase 3 complete" in out
rows = await agent_skill_repo.find_where("owner_id = 'agent1'", limit=10)
assert len(rows) == 1
assert rows[0].name == "summarise_doc"
assert rows[0].cluster_id is not None
@@ -0,0 +1,146 @@
"""LanceDB business table schema migration (v2: vector nullability).
Exercises real LanceDB: builds a pre-migration ("v1-shape") physical
table with a non-nullable ``vector`` column, runs
``migrate_table_schemas``, and asserts the on-disk pyarrow schema is
altered in place plus the ``.table_schema_version`` marker is written.
Also proves idempotency — a second call must not re-issue
``alter_columns`` once the marker is at the target version.
"""
from __future__ import annotations
from pathlib import Path
import pyarrow as pa
import pytest
from everos.infra.persistence.lancedb import (
_TABLE_SCHEMA_VERSION,
BUSINESS_SCHEMAS_WITH_VECTOR,
AtomicFact,
Episode,
LanceDBMigrationError,
lancedb_manager,
migrate_table_schemas,
)
def _v1_shape_schema() -> pa.Schema:
"""Minimal pre-migration physical shape: ``vector`` is non-nullable."""
return pa.schema(
[
pa.field("id", pa.string(), nullable=False),
pa.field(
"vector",
pa.list_(pa.field("item", pa.float32(), nullable=True), 1024),
nullable=False,
),
]
)
@pytest.fixture(autouse=True)
async def _reset(tmp_path: Path, monkeypatch: pytest.MonkeyPatch):
"""Point the singleton connection at an isolated memory-root."""
monkeypatch.setenv("EVEROS_ROOT", str(tmp_path))
lancedb_manager._conn = None
lancedb_manager._tables.clear()
yield
await lancedb_manager.dispose_connection()
async def _create_v1_table(name: str) -> None:
conn = await lancedb_manager.get_connection()
await conn.create_table(name, schema=_v1_shape_schema())
async def test_migration_alters_all_business_tables_to_nullable() -> None:
for schema in BUSINESS_SCHEMAS_WITH_VECTOR:
await _create_v1_table(schema.TABLE_NAME)
await migrate_table_schemas()
for schema in BUSINESS_SCHEMAS_WITH_VECTOR:
table = await lancedb_manager.get_table(schema.TABLE_NAME, schema)
arrow_schema = await table.schema()
assert arrow_schema.field("vector").nullable is True
async def test_migration_writes_marker_with_target_version(tmp_path: Path) -> None:
await _create_v1_table(Episode.TABLE_NAME)
await migrate_table_schemas()
marker = tmp_path / ".index" / "lancedb" / ".table_schema_version"
assert marker.exists()
assert int(marker.read_text().strip()) == _TABLE_SCHEMA_VERSION
async def test_migration_idempotent_second_call_skips_alter(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Second call short-circuits at the marker; ``alter_columns`` is not
re-issued."""
await _create_v1_table(Episode.TABLE_NAME)
await migrate_table_schemas()
table = await lancedb_manager.get_table(Episode.TABLE_NAME, Episode)
calls: list[object] = []
original_alter_columns = table.alter_columns
async def _spy_alter_columns(*alterations: object) -> object:
calls.append(alterations)
return await original_alter_columns(*alterations)
monkeypatch.setattr(table, "alter_columns", _spy_alter_columns)
await migrate_table_schemas()
assert calls == []
async def test_migrate_table_schemas_raises_on_alter_failure(
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Any table's failing ``alter_columns`` must fail startup loudly:
:class:`LanceDBMigrationError` is raised, the version marker stays
unwritten, and the message names the failed table plus escalating
recovery hints (restart, then wipe). Cascade would otherwise write
NULL vectors into a still-NOT-NULL column and silently drop every
row."""
for schema in BUSINESS_SCHEMAS_WITH_VECTOR:
await _create_v1_table(schema.TABLE_NAME)
episode_table = await lancedb_manager.get_table(Episode.TABLE_NAME, Episode)
async def _raise_alter_columns(*alterations: object) -> object:
raise RuntimeError("simulated alter_columns failure")
monkeypatch.setattr(episode_table, "alter_columns", _raise_alter_columns)
with pytest.raises(LanceDBMigrationError) as excinfo:
await migrate_table_schemas()
message = str(excinfo.value)
assert Episode.TABLE_NAME in message
lancedb_dir = tmp_path / ".index" / "lancedb"
assert str(lancedb_dir) in message
# Escalating recovery: restart first, wipe second.
restart_idx = message.find("restart the process")
wipe_idx = message.find("wipe the index directory")
assert restart_idx != -1 and wipe_idx != -1
assert restart_idx < wipe_idx
marker = lancedb_dir / ".table_schema_version"
assert not marker.exists()
# Sibling tables reached in the loop before the failure keep the
# per-table alter that already succeeded — only the marker write
# (i.e. "the migration as a whole is done") is gated.
atomic_fact_table = await lancedb_manager.get_table(
AtomicFact.TABLE_NAME, AtomicFact
)
atomic_fact_schema = await atomic_fact_table.schema()
assert atomic_fact_schema.field("vector").nullable is True
+159 -50
View File
@@ -157,17 +157,6 @@ def _reset_lancedb_write_locks() -> None:
LanceRepoBase._reset_locks_for_tests()
@pytest.fixture(autouse=True)
def _reset_knowledge_embedding_singleton() -> None:
"""Reset the lazy embedding and reranker singletons in service.knowledge."""
import everos.service.knowledge as _kmod
for attr in ("_embedding", "_reranker"):
setattr(_kmod, attr, None)
for attr in ("_embedding_resolved", "_reranker_resolved"):
setattr(_kmod, attr, False)
@pytest.fixture
async def cascade_runtime(
tmp_path: Path,
@@ -188,22 +177,36 @@ async def cascade_runtime(
await ensure_business_indexes()
(tmp_path / "ome.toml").write_text("# test\n")
yield MemoryRoot.default()
yield MemoryRoot.resolve()
await dispose_connection()
await dispose_engine()
def _build_orchestrator(
monkeypatch: pytest.MonkeyPatch,
memory_root: MemoryRoot,
embedder: _StubEmbedder,
*,
scan_interval: float = 60.0,
) -> CascadeOrchestrator:
"""Factory for a tight-polling cascade orchestrator."""
"""Factory for a tight-polling cascade orchestrator.
``KnowledgeTopicHandler`` fetches the embedder lazily via
``get_embedding_capability()`` rather than through the orchestrator
— patch the process-wide singleton so cascade never hits the fake
network target set up by ``cascade_runtime``.
"""
import everos.component.embedding.accessor as acc
import everos.component.rerank.accessor as rerank_acc
from everos.component.embedding import EmbeddingCapability
from everos.component.rerank import RerankCapability
monkeypatch.setattr(acc, "_capability", EmbeddingCapability(provider=embedder))
reranker = _StubReranker()
monkeypatch.setattr(rerank_acc, "_capability", RerankCapability(provider=reranker))
return CascadeOrchestrator(
memory_root=memory_root,
embedder=embedder,
tokenizer=build_tokenizer(),
config=CascadeConfig(
scan_interval_seconds=scan_interval,
@@ -274,11 +277,12 @@ async def _create_test_document(
async def test_create_document_end_to_end(
cascade_runtime: MemoryRoot,
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Full pipeline: create -> md -> cascade -> SQLite + LanceDB."""
memory_root = cascade_runtime
embedder = _StubEmbedder()
orchestrator = _build_orchestrator(memory_root, embedder)
orchestrator = _build_orchestrator(monkeypatch, memory_root, embedder)
await orchestrator.start()
await asyncio.sleep(0.3)
@@ -366,17 +370,12 @@ async def test_create_document_end_to_end(
async def test_search_finds_ingested_topic(
cascade_runtime: MemoryRoot,
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Keyword search finds topics after cascade indexing."""
import everos.service.knowledge as _kmod
memory_root = cascade_runtime
embedder = _StubEmbedder()
_kmod._embedding = embedder
_kmod._embedding_resolved = True
_kmod._reranker = _StubReranker()
_kmod._reranker_resolved = True
orchestrator = _build_orchestrator(memory_root, embedder)
orchestrator = _build_orchestrator(monkeypatch, memory_root, embedder)
await orchestrator.start()
await asyncio.sleep(0.3)
@@ -404,17 +403,12 @@ async def test_search_finds_ingested_topic(
async def test_search_include_content(
cascade_runtime: MemoryRoot,
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""include_content flag controls whether content is populated."""
import everos.service.knowledge as _kmod
memory_root = cascade_runtime
embedder = _StubEmbedder()
_kmod._embedding = embedder
_kmod._embedding_resolved = True
_kmod._reranker = _StubReranker()
_kmod._reranker_resolved = True
orchestrator = _build_orchestrator(memory_root, embedder)
orchestrator = _build_orchestrator(monkeypatch, memory_root, embedder)
await orchestrator.start()
await asyncio.sleep(0.3)
@@ -451,17 +445,12 @@ async def test_search_include_content(
async def test_search_score_threshold_filters(
cascade_runtime: MemoryRoot,
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""score_threshold filters out low-scoring results."""
import everos.service.knowledge as _kmod
memory_root = cascade_runtime
embedder = _StubEmbedder()
_kmod._embedding = embedder
_kmod._embedding_resolved = True
_kmod._reranker = _StubReranker()
_kmod._reranker_resolved = True
orchestrator = _build_orchestrator(memory_root, embedder)
orchestrator = _build_orchestrator(monkeypatch, memory_root, embedder)
await orchestrator.start()
await asyncio.sleep(0.3)
@@ -493,17 +482,12 @@ async def test_search_score_threshold_filters(
async def test_search_app_project_isolation(
cascade_runtime: MemoryRoot,
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Documents in different app/project scopes are isolated in search."""
import everos.service.knowledge as _kmod
memory_root = cascade_runtime
embedder = _StubEmbedder()
_kmod._embedding = embedder
_kmod._embedding_resolved = True
_kmod._reranker = _StubReranker()
_kmod._reranker_resolved = True
orchestrator = _build_orchestrator(memory_root, embedder)
orchestrator = _build_orchestrator(monkeypatch, memory_root, embedder)
await orchestrator.start()
await asyncio.sleep(0.3)
@@ -597,6 +581,7 @@ async def test_search_app_project_isolation(
async def test_delete_document_end_to_end(
cascade_runtime: MemoryRoot,
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Manual directory removal + scanner detects deletion -> cleanup.
@@ -608,6 +593,7 @@ async def test_delete_document_end_to_end(
memory_root = cascade_runtime
embedder = _StubEmbedder()
orchestrator = _build_orchestrator(
monkeypatch,
memory_root,
embedder,
scan_interval=2.0,
@@ -677,13 +663,126 @@ async def test_delete_document_end_to_end(
await orchestrator.stop()
async def test_delete_document_cleans_up_after_downgrade(
cascade_runtime: MemoryRoot,
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Tier 3 → Tier 1 downgrade must not strand knowledge state.
Regression guard for PR #361 review blocker B3. Previously
``cascade.registry.build_handlers`` gated the two knowledge
handlers off as an atomic pair when embed OR rerank went missing.
On a Tier 3 → Tier 1 downgrade, ``service.knowledge.delete_document``
``shutil.rmtree``'d the md directory, the scanner enqueued
``deleted`` rows for every md — and the worker marked every one
``failed(retryable=False)`` because no handler was registered
for the kind. Phantom SQLite / LanceDB rows lingered forever;
``GET /documents`` kept listing them; ``cascade fix`` could not
recover.
The fix registers both knowledge handlers unconditionally. This
test seeds a document with both capabilities available, simulates
a full downgrade by pointing both accessor singletons at
``provider=None``, calls ``delete_document`` and asserts the
entire md + SQLite + LanceDB state clears and no permanent
failures accrue.
"""
import everos.component.embedding.accessor as embed_acc
import everos.component.rerank.accessor as rerank_acc
from everos.component.embedding import EmbeddingCapability
from everos.component.rerank import RerankCapability
memory_root = cascade_runtime
embedder = _StubEmbedder()
orchestrator = _build_orchestrator(
monkeypatch,
memory_root,
embedder,
scan_interval=2.0,
)
await orchestrator.start()
await asyncio.sleep(0.3)
try:
# ── Tier 3 seed: capabilities available, full index. ────────
doc_id = "d_downgrd001"
await _create_test_document(memory_root, doc_id=doc_id)
await _wait_lance_rows(doc_id, expected=2, deadline=20.0)
await _wait_drain(deadline=20.0)
assert await knowledge_document_repo.get_by_doc_id(doc_id) is not None
assert await knowledge_topic_sqlite_repo.count_by_doc_id(doc_id) == 2
# ── Simulate downgrade: both capabilities unavailable. ──────
# Point the process-wide singletons the handlers read at
# ``provider=None``. ``embed_or_none`` returns ``None`` and
# ``handle_deleted`` never touches embed/rerank at all.
monkeypatch.setattr(
embed_acc, "_capability", EmbeddingCapability(provider=None)
)
monkeypatch.setattr(rerank_acc, "_capability", RerankCapability(provider=None))
# ── Delete through the service (bypass HTTP for test speed).
# ``delete_document`` rmtree's the md dir; scanner picks up
# the missing files and enqueues ``deleted`` rows for both
# kinds; worker must drain them cleanly.
del_result = await delete_document(
doc_id=doc_id,
app_id="default",
project_id="default",
)
assert del_result.doc_id == doc_id
assert del_result.deleted_topics == 2
knowledge_dir = memory_root.knowledge_dir()
doc_dir = knowledge_dir / "Sports" / f"Olympics_Plan_{doc_id}"
assert not doc_dir.exists(), "delete_document should have removed the md dir"
# ── Wait for cascade to drain the deletes. ──────────────────
await _wait_lance_rows(doc_id, expected=0, deadline=20.0)
await _wait_drain(deadline=20.0)
# ── LanceDB clean. ──────────────────────────────────────────
table = await get_table(KnowledgeTopic.TABLE_NAME, KnowledgeTopic)
assert await table.count_rows(filter=f"doc_id = '{doc_id}'") == 0
# ── SQLite topic rows gone. ─────────────────────────────────
assert await knowledge_topic_sqlite_repo.count_by_doc_id(doc_id) == 0
# ── SQLite document row gone (may need a scanner retry after
# topic deletes clear the FK dependency). ───────────────────
async with asyncio.timeout(15.0):
while True:
row = await knowledge_document_repo.get_by_doc_id(doc_id)
if row is None:
break
await asyncio.sleep(0.2)
assert await knowledge_document_repo.get_by_doc_id(doc_id) is None
# ── No permanent failures in md_change_state. ───────────────
# This is the load-bearing assertion: with the gate in place,
# every ``deleted`` row would land here as failed_permanent.
summary = await md_change_state_repo.queue_summary()
assert summary.failed_permanent == 0, (
f"downgrade + delete produced permanent failures: {summary}"
)
assert summary.failed_retryable == 0, (
f"downgrade + delete produced retryable failures: {summary}"
)
finally:
await orchestrator.stop()
async def test_delete_idempotent(
cascade_runtime: MemoryRoot,
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Deleting an already-deleted (or nonexistent) document is a no-op."""
memory_root = cascade_runtime
embedder = _StubEmbedder()
orchestrator = _build_orchestrator(
monkeypatch,
memory_root,
embedder,
scan_interval=2.0,
@@ -729,11 +828,13 @@ async def test_delete_idempotent(
async def test_replace_document_end_to_end(
cascade_runtime: MemoryRoot,
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Replace = delete old + create new with same doc_id."""
memory_root = cascade_runtime
embedder = _StubEmbedder()
orchestrator = _build_orchestrator(
monkeypatch,
memory_root,
embedder,
scan_interval=2.0,
@@ -852,11 +953,12 @@ async def test_replace_document_end_to_end(
async def test_patch_title_updates_metadata(
cascade_runtime: MemoryRoot,
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""patch_document updates the title in SQLite without touching topics."""
memory_root = cascade_runtime
embedder = _StubEmbedder()
orchestrator = _build_orchestrator(memory_root, embedder)
orchestrator = _build_orchestrator(monkeypatch, memory_root, embedder)
await orchestrator.start()
await asyncio.sleep(0.3)
@@ -890,11 +992,12 @@ async def test_patch_title_updates_metadata(
async def test_patch_category_updates_sqlite(
cascade_runtime: MemoryRoot,
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""patch_document with category_id updates the document row (SQLite-only MVP)."""
memory_root = cascade_runtime
embedder = _StubEmbedder()
orchestrator = _build_orchestrator(memory_root, embedder)
orchestrator = _build_orchestrator(monkeypatch, memory_root, embedder)
await orchestrator.start()
await asyncio.sleep(0.3)
@@ -941,14 +1044,20 @@ async def test_get_nonexistent_document_raises(
async def test_search_empty_query_returns_empty(
cascade_runtime: MemoryRoot,
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""An empty query string returns empty results (no crash)."""
import everos.service.knowledge as _kmod
import everos.component.embedding.accessor as embed_acc
import everos.component.rerank.accessor as rerank_acc
from everos.component.embedding import EmbeddingCapability
from everos.component.rerank import RerankCapability
_kmod._embedding = _StubEmbedder()
_kmod._embedding_resolved = True
_kmod._reranker = _StubReranker()
_kmod._reranker_resolved = True
monkeypatch.setattr(
embed_acc, "_capability", EmbeddingCapability(provider=_StubEmbedder())
)
monkeypatch.setattr(
rerank_acc, "_capability", RerankCapability(provider=_StubReranker())
)
result = await search_knowledge(
query="",

Some files were not shown because too many files have changed in this diff Show More