refactor(config): make [embedding] and [rerank] soft dependencies (#361)
* refactor(config): make [embedding] and [rerank] soft dependencies
Make [embedding] and [rerank] soft runtime dependencies so a
freshly-onboarded user can run EverOS end-to-end with only [llm]
configured. Previously the server refused to start without embedding,
locking out anyone who just wanted keyword-only search.
## Capability tiers
- Tier 1 ([llm] only) KEYWORD search, add/flush, md
writes, cascade sync
- Tier 2 ([llm] + [embedding]) + VECTOR / HYBRID search,
reflection, skill extraction,
backfill
- Tier 3 ([llm] + [embedding] + rerank) + AGENTIC search, knowledge
Tier upgrades require a server restart (capability accessors cache
for the process lifetime). Tier downgrades are read-safe: a Tier-3
user who drops [rerank] can still read/rename/delete existing
knowledge documents; only write/search endpoints return 422.
## What changed
- Component accessors — component/{embedding,rerank,llm}/accessor.py
are the single process-wide provider singletons. service/* never
maintains parallel singletons; it consumes get_embedding_capability()
/ get_rerank_capability() / get_llm_client() directly. Build-time
ValueError from the factory is logged as capability_build_failed
(was silently swallowed).
- Error mapping — ProviderNotConfiguredError -> 422 with everos.toml
section hints (never EVEROS_* env-var strings).
LanceDBMigrationError fails loud with escalating recovery guidance
(restart -> wipe index). LLMNotConfiguredError in search maps to
None for KEYWORD degradation.
- Nullable-vector LanceDB migration — schema v2 makes the vector
column nullable so Tier-1 rows can land without embeddings.
Migration is guarded by a cross-process memory_root_lock
(fcntl.flock + anyio.to_thread) and runs optimize() per table
after Phase-1 backfill to reclaim manifest bloat.
- Cascade — knowledge handlers register unconditionally (Tier-3 ->
Tier-2/1 downgrade no longer strands DELETE); embed-requiring
strategies use body-guards that check capability.available at
execution time. _TABLE_SPECS has an import-time drift assertion
against BUSINESS_SCHEMAS_WITH_VECTOR.
- `everos cascade backfill` CLI — Phase-1 (embed missing vectors) /
Phase-2 (emit synthetic events for cascaded processing) / Phase-3
(sync new skill files). Exit codes: 0 / 1 / 2 / 3 (server running
preflight) / 4 (COMPLETED_WITH_FAILURES — per-row failures rolled
up) / 130 (SIGINT). OMEConfig.crash_recovery_enabled=False in
backfill engines prevents stale-RUNNING rows re-enqueuing into a
smaller strategy registry.
- /health — reports capabilities + disabled_features per tier so ops
can distinguish "boots but degraded" from "boots and full".
- Presentation split — memory / service / infra never import typer /
click. TyperPresenter Protocol + run_backfill() live in
entrypoints/cli/commands/_backfill_cmd.py. Enforced by
import-linter.
- Startup hint — unconditional count_rows(filter="vector IS NULL")
sweep emits unbackfilled_memory_rows (event name + hint text
pinned) when Tier-1 rows exist. ParserLifespanProvider warms the
everalgo.parser import at boot so /health doesn't block on first
call.
- Knowledge upload UTF-8 short-circuit — _looks_like_utf8_text()
routes text/* mime and known plaintext extensions (md/txt/rst)
straight to UTF-8 decode instead of the parser. Prevents 503
Multimodal-not-configured when Tier 3 sans [multimodal] uploads a
markdown doc.
## Sync history with main (2 merges collapsed into this squash)
Merged origin/main at 6dcd3eb (v1.1.4 -> v1.2.0 adds OTel tracing,
/api/v2 alias, TracingLifespanProvider, per-cascade-embedding span
fix, memory-op instrumentation) and later at 42629df (PR #366
backfills v1.1.4 CWE-22 knowledge path traversal fix + cascade
retry-budget rework + errors.py -> core.errors.ExternalServiceError).
Key merge decisions:
- service/search.py adopts single wrap site — component.llm accessor
already applies UsageRecordingClient when observability is on;
service layer never keeps a parallel LLM singleton (Round-1 CR
rule: "service layer never maintains parallel singletons").
- Knowledge router prefix moved to /knowledge; create_app() mounts
it under both /api/v1 and /api/v2.
- Cascade retry classification uses ExternalServiceError from
core.errors (cascade/errors.py deleted). _MAX_TOTAL_RETRIES=12
cross-cycle budget preserved.
- Fixed backport typo: MemoryRoot.default() -> MemoryRoot.resolve()
(no .default() classmethod exists — main PR #366 shipped a broken
call).
## Verified layering
$ git grep -l "^import typer\|^from typer" src/everos/{memory,service,infra}
# empty
$ git grep -l "^import click\|^from click" src/everos/{memory,service,infra}
# empty
Memory / service / infra layers clean of CLI presentation libraries.
## Review history
Three rounds of Fable 5 (opus) code review across the pre-squash
commit history closed 38 findings total:
- Round 1: 10 findings (fail-loud migration, backfill hardening,
knowledge router gate scoping, SearchManager guards, profile
throttle lift)
- Round 2: 13 findings (hermetic test env, hot-reload doc drift,
knowledge handler registration, Phase 3 sync guarantee, Phase 2
idempotency, profile event-first path, OMEConfig crash-recovery
gate, cross-process migration lock, batch embed per-row fallback,
LanceDB optimize, typer/click layer split)
- Round 3: 15 Minor cleanups (accessor unification, marker revert,
episode query hygiene, --verbose subcommand, parser lifespan warm,
task-number scrub, temporal-overlap test, real-SIGINT slow mark,
4 design-note back-references)
Full per-round context lives in the PR description on GitHub.
## Test plan
- make lint (ruff + import-linter 3 contracts + assets +
deprecated-names + github-docs + datetime + OpenAPI drift)
- Hermetic env full pytest — 2027 passed / 7 deselected (7 = slow +
live_llm markers)
- Manual e2e across Tier 1/2/3 (21/21 assertions across v1/v2
double-mount and Tier 3 -> Tier 2 downgrade)
- /health reports correct capabilities + disabled_features per tier
## Known follow-ups
- .superpowers/sdd/followup-http-bridge.md (gitignored) — Path A for
spec §10's "backfill 期间 EverOS 完全可用" promise
- _TABLE_SCHEMA_VERSION docstring — v3+ migrations need a version
dispatch table
- extract_user_profile.py throttle-counter block — replace LanceDB
count_by_owner with a sqlite memcell count
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(review): close 3 blockers surfaced by round-4 review
N1. cluster_repo.find_cluster_id_for_member was cross-owner-unsafe.
Its reverse index (member_type, member_id) alone cannot disambiguate
two owners whose entry_id happens to collide — entry_id is
deliberately only per-owner unique (see entries.py:47:
'Cross-user uniqueness is handled at the database layer via a
composite <user_id>_<entry_id> field; it is not encoded into the
EntryId string itself'). Phase 2's _scan_all_rows crosses all
owners, so on any multi-owner root, same-day seq=1 episodes under
different owners would either false-hit each other's cluster or
be silently skipped from clustering. Add required (app_id,
project_id, owner_id) keyword args + JOIN Cluster (which already
carries scope) to filter by parent scope. Prior signature had zero
production callers except two the same PR just added, so the
API break is contained. Regression test: two owners persist a
cluster each around the same entry_id, each lookup resolves to
its own owner's cluster, a third owner's lookup returns None.
N2. Ctrl-C / EOF at the y/N prompt was landing on the generic
except Exception branch (exit 2 with rich traceback) instead of
the exit-130 interrupt path. Root cause: typer 0.15+ vendored
click under typer._click, so typer.Abort and the standalone
click.exceptions.Abort are distinct classes. The interrupt-branch
catch only listed the standalone one; every existing 'abort'
test was manually raising click.exceptions.Abort so the miss
was a false-positive guard rail. Widen the catch to
(typer.Abort, click.exceptions.Abort) and declare click as a
first-class dependency (it was only pulled in via uvicorn).
Regression test: raise real typer.Abort() at the confirm step
and assert exit 130 + INTERRUPTED banner.
N3. _looks_like_utf8_text used mime.startswith('text/'), which
caught text/html as well. HTML uploads then bypassed everalgo's
_aparse_html — losing clean_html_for_llm (strips <script>/<style>
/<nav>/<iframe> + HTML comments) and the 1 MiB output cap. A
40 MiB .html with <script> bodies and <!-- prompt injection -->
comments would flow straight into the extraction LLM. Replace
with explicit allowlist {text/plain, text/markdown, text/x-rst,
text/x-markdown}; text/html and any future text/* mime now
default to the parser path. Test matrix asserts text/html →
False (was regressed as True by the earlier commit).
Hermetic env full pytest: 2033 passed / 7 deselected (+6 tests
from these regressions).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(review): close round-4 major + minor + PR body errata
Round-4 review-driven cleanup. Blocker fixes (N1/N2/N3) landed in
561b5fe. This commit closes the remaining CONFIRMED items:
Major:
- J3 MemoryRoot.default() -> resolve() was a breaking public API
rename this branch introduced (main still has default()). Adds
default() as a backward-compat alias forwarding to resolve()
with DeprecationWarning; CHANGELOG entry under Unreleased.
- J4 episode_repo.list_by_owner_after_ts(limit=N) truncates in
fragment order (== insertion order), NOT newest-first. Docstring
now spells out the trap so a future caller passing limit for a
'newest N' window doesn't silently get the oldest N.
- J5.2 TyperPresenter.nothing_to_backfill picked colour via
'could not be read' in message — a domain wording change
would silently flip yellow -> green. Signature gains explicit
scan_failed: bool kwarg; CLI colour-picks off the flag.
- J5.5 phase_header was Protocol-declared but never dispatched
(run_backfill calls _print_phase_header directly). Removed the
dead Protocol method + both no-op implementations.
- J6.3 3 inline from everos.core.errors import ... inside
Phase 1/2/3 preflights promoted to a single top-level import.
- J7 subject-side embed failure was silently exit-0 because
rows_processed advanced whenever any side wrote. Now: a row with
a needed side still NULL counts as rows_failed (exit 4 =
COMPLETED_WITH_FAILURES). Gated on spec.subject_of + row.needs_*
+ row.subject_text so non-Episode tables and subject-empty rows
don't false-positive.
- J9 test_migration_cross_process.py did NOT actually test cross-
process (all 5 tests mock memory_root_lock). Renamed to
test_migration_lock_wiring.py; docstring now scopes it to
'lock-invocation wiring' and points at test_core/…/test_locking.py
for real flock coverage.
- J10 Phase 1 lacked the server-running preflight Phase 2/3 have.
--phase all against a live server would burn Phase 1 embed API
calls (real cost) before Phase 2 halted with exit 3. Phase 1 now
probes _probe_ome_lock_available first; regression test in
test_backfill_preflight.py; upgrade_path integration patches the
probe so its in-process 'server + backfill' scenario stays valid.
Minor:
- M1 knowledge upload with NUL byte or filename > 255 bytes UTF-8
used to raise ValueError/OSError at write_bytes → 500 with a
half-written md left on disk. _safe_original_filename now
rejects both up front with InvalidInputError (→ 400).
- M2 backfill optimize() now passes cleanup_older_than=timedelta(0)
so older manifest versions are physically pruned (previous call
compacted fragments but left the manifest chain on disk).
- M3 verify_business_schemas remediation text used to jump straight
to 'rm -rf ~/.everos/.index/lancedb'; now walks restart → wipe.
- M5 multimodal/accessor.py capability_build_failed warning added
so all four provider accessors log symmetrically (was silent).
- M7 test_knowledge_api parser-absence tests call
parser_available.cache_clear() around the sys.modules patch so
the lru_cache doesn't strand a stale True/False.
- M8 cascade_handler_embed_skipped (6 handlers) demoted INFO → DEBUG:
Tier 1 imports were generating N × 6 handler-info lines per md.
- M10.1 test_drift_scenario_would_raise was a tautology (compared
two hardcoded string sets, never touched the guard). Now
monkey-patches BUSINESS_SCHEMAS_WITH_VECTOR to a superset and
reloads _backfill, proving the import-time RuntimeError fires.
- M11 test_cascade_verbose_position subprocess.run calls gain
env= — scrubs EVEROS_* from the developer environment so the
same footgun as round-2 B1 doesn't re-appear inside subprocesses.
- M14 health() -> dict degraded the OpenAPI schema to
additionalProperties: true. Introduce HealthResponse +
HealthCapabilities Pydantic models so clients get real field
shape; docs/openapi.json regenerated.
- M16 routes/knowledge.py:_require_knowledge_capabilities docstring
claimed cascade.registry.build_handlers still gates
knowledge_topic/knowledge_document, contradicting
registry.py:177-194 (gate removed there, moved to HTTP layer).
Rewritten to describe the current design accurately.
Hermetic env full pytest: 2037 passed / 7 deselected.
Explicitly deferred to followup:
- J1 lazy multimodal client (needs everalgo signature change)
- J2 tier definition (knowledge = Tier 3 whole)
- J5.1/3/4 broader presentation-split refactor
- J6.1/2 backfill dispatch and _backfill_table refactor
- J8 Phase 1 keyset pagination for bulk migration OOM
- M4 flock timeout/waiting log/re-entry (core/persistence refactor)
- M6 UTF-8 codec strategy (BOM/UTF-16/GBK)
- M9 count_by_owner monotonicity (latent, INTERVAL=1 short-circuits)
- M13 --phase all multi-phase combined-outcome test coverage
- M15 PR-marker rationale comments (16 occurrences, all inert)
M12 was refuted (both event names exist).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.7
parent
649046b0df
commit
b1441da607
@@ -9,6 +9,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
### Changed
|
||||
|
||||
- **`MemoryRoot.default()` → `MemoryRoot.resolve()`** — the classmethod that
|
||||
resolves the memory root from `--root` / `EVEROS_ROOT` / default was
|
||||
renamed to make its behavior explicit (`resolve` walks the precedence
|
||||
chain; `default` was ambiguous with "default location"). `MemoryRoot`
|
||||
is publicly exported from `everos.core.persistence`; callers outside
|
||||
the repo may have used the old name. **A `default()` alias is kept**
|
||||
as a backward-compatibility shim that forwards to `resolve()` and
|
||||
emits a `DeprecationWarning`. The alias will be removed in a future
|
||||
major release — update call sites when convenient.
|
||||
- **Uncalibrated recall scores moved to their own name** — `KEYWORD` and
|
||||
single-route `VECTOR` searches now report their top score as
|
||||
`recall_top_score_raw`; `recall_top_score` is reserved for the calibrated
|
||||
|
||||
+67
-6
@@ -12,7 +12,7 @@
|
||||
"health"
|
||||
],
|
||||
"summary": "Health",
|
||||
"description": "Liveness probe — returns ``{\"status\": \"ok\"}`` with HTTP 200.",
|
||||
"description": "Liveness probe with capabilities and disabled features.",
|
||||
"operationId": "health_health_get",
|
||||
"responses": {
|
||||
"200": {
|
||||
@@ -20,11 +20,7 @@
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"additionalProperties": {
|
||||
"type": "string"
|
||||
},
|
||||
"type": "object",
|
||||
"title": "Response Health Health Get"
|
||||
"$ref": "#/components/schemas/HealthResponse"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -2719,6 +2715,71 @@
|
||||
"type": "object",
|
||||
"title": "HTTPValidationError"
|
||||
},
|
||||
"HealthCapabilities": {
|
||||
"properties": {
|
||||
"llm": {
|
||||
"type": "boolean",
|
||||
"title": "Llm"
|
||||
},
|
||||
"embed": {
|
||||
"type": "boolean",
|
||||
"title": "Embed"
|
||||
},
|
||||
"rerank": {
|
||||
"type": "boolean",
|
||||
"title": "Rerank"
|
||||
},
|
||||
"multimodal_llm": {
|
||||
"type": "boolean",
|
||||
"title": "Multimodal Llm"
|
||||
},
|
||||
"parser": {
|
||||
"type": "boolean",
|
||||
"title": "Parser"
|
||||
}
|
||||
},
|
||||
"type": "object",
|
||||
"required": [
|
||||
"llm",
|
||||
"embed",
|
||||
"rerank",
|
||||
"multimodal_llm",
|
||||
"parser"
|
||||
],
|
||||
"title": "HealthCapabilities",
|
||||
"description": "Availability flags for the five capability probes.\n\nField order matches the health-endpoint payload contract; clients\nkey off these names to decide whether to expose optional features."
|
||||
},
|
||||
"HealthResponse": {
|
||||
"properties": {
|
||||
"status": {
|
||||
"type": "string",
|
||||
"title": "Status"
|
||||
},
|
||||
"version": {
|
||||
"type": "string",
|
||||
"title": "Version"
|
||||
},
|
||||
"capabilities": {
|
||||
"$ref": "#/components/schemas/HealthCapabilities"
|
||||
},
|
||||
"disabled_features": {
|
||||
"items": {
|
||||
"type": "string"
|
||||
},
|
||||
"type": "array",
|
||||
"title": "Disabled Features"
|
||||
}
|
||||
},
|
||||
"type": "object",
|
||||
"required": [
|
||||
"status",
|
||||
"version",
|
||||
"capabilities",
|
||||
"disabled_features"
|
||||
],
|
||||
"title": "HealthResponse",
|
||||
"description": "Response schema for ``GET /health``.\n\nDeclared as a Pydantic model (not ``dict``) so the generated\nOpenAPI schema carries the full field shape — ``capabilities`` and\n``disabled_features`` are typed. A bare ``-> dict`` return type\ndegrades the OpenAPI response to ``additionalProperties: true``,\nwhich robs clients (and codegen) of any structure to lean on."
|
||||
},
|
||||
"KnowledgeSearchRequest": {
|
||||
"properties": {
|
||||
"query": {
|
||||
|
||||
@@ -50,6 +50,14 @@ dependencies = [
|
||||
|
||||
# CLI + TUI
|
||||
"typer>=0.12.0",
|
||||
# click is a transitive dep of uvicorn today, but our CLI raises
|
||||
# ``click.exceptions.Abort`` directly (e.g. tests that inject aborts)
|
||||
# and typer 0.15+ vendored a copy of click under ``typer._click`` — so
|
||||
# the two classes are DIFFERENT even though users only ever see
|
||||
# ``typer.Abort``. We depend on the standalone click package so
|
||||
# ``click.exceptions.Abort`` stays importable and both classes are
|
||||
# covered in the Ctrl-C catch (see backfill exit-130 path).
|
||||
"click>=8.1",
|
||||
"textual>=8.2.7",
|
||||
|
||||
# Tokenization (BM25 Chinese support)
|
||||
|
||||
@@ -51,7 +51,7 @@ def _print_header(mode: str, fixture_path: Path, session_id: str) -> None:
|
||||
print(f" everos e2e memorize · mode={mode}")
|
||||
print(f" fixture : {fixture_path.name}")
|
||||
print(f" session_id : {session_id}")
|
||||
print(f" memory root : {MemoryRoot.default().root}")
|
||||
print(f" memory root : {MemoryRoot.resolve().root}")
|
||||
llm_state = "<configured>" if get_llm_client() else "<None — pipeline will skip>"
|
||||
print(f" llm_client : {llm_state}")
|
||||
print("=" * 72)
|
||||
@@ -59,7 +59,7 @@ def _print_header(mode: str, fixture_path: Path, session_id: str) -> None:
|
||||
|
||||
def _list_written_files(session_id: str, mode: str) -> None:
|
||||
"""Walk memory root and print files touched in this run."""
|
||||
root = MemoryRoot.default().root
|
||||
root = MemoryRoot.resolve().root
|
||||
cutoff = time.time() - 600 # files modified in the last 10 min
|
||||
print()
|
||||
print("─── files modified within the last 10 minutes under memory root ───")
|
||||
|
||||
@@ -0,0 +1,74 @@
|
||||
"""Capabilities inference and feature availability logic.
|
||||
|
||||
Compute which features are disabled based on available capabilities.
|
||||
Used by the health endpoint, startup banner, and related diagnostics.
|
||||
|
||||
Two distinct feature-name vocabularies exist in the refactored codebase.
|
||||
They deliberately do NOT share a source of truth:
|
||||
|
||||
- ``ProviderNotConfiguredError.feature`` (raised by
|
||||
``SearchManager._validate_components`` and error handlers): per-request
|
||||
granular tag identifying the specific search mode or endpoint that failed —
|
||||
``"vector"`` | ``"user_hybrid"`` | ``"agent_hybrid"`` | ``"agentic_search"``
|
||||
| ``"knowledge"`` | ``"skill_extraction_backfill"``. Appears in HTTP 422
|
||||
response ``message``. Naming mirrors the ``SearchMethod`` enum values plus
|
||||
endpoint-scoped tags.
|
||||
|
||||
- ``compute_disabled_features()`` (below): capability-level tag identifying
|
||||
a whole feature category disabled by the current tier —
|
||||
``"vector_search"`` | ``"hybrid_search"`` | ``"agentic_search"``
|
||||
| ``"reflection"`` | ``"skill_extraction"`` | ``"knowledge"``
|
||||
| ``"multimodal_upload"``. Appears in ``GET /health`` response
|
||||
``disabled_features``.
|
||||
|
||||
Callers of either surface should treat these vocabularies as stable client
|
||||
contracts — do not unify them without a coordinated migration on the client side.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
|
||||
def compute_disabled_features(caps: dict[str, bool]) -> list[str]:
|
||||
"""Derive the list of disabled features from capability availability.
|
||||
|
||||
Args:
|
||||
caps: Dictionary with keys "llm", "embed", "rerank", "multimodal_llm", "parser"
|
||||
and boolean availability values. Note: ``caps["llm"]`` is
|
||||
accepted for shape symmetry but NOT read here — LLM is a
|
||||
Tier-1 hard requirement enforced at server startup
|
||||
(``LLMLifespanProvider``), so any process reaching this
|
||||
function is guaranteed to have LLM available. If LLM ever
|
||||
becomes soft, add an ``if not caps["llm"]`` branch here
|
||||
covering the LLM-dependent features.
|
||||
|
||||
Returns:
|
||||
List of feature names that are disabled due to missing capabilities.
|
||||
Possible values: "vector_search", "hybrid_search", "agentic_search",
|
||||
"reflection", "skill_extraction", "knowledge", "multimodal_upload".
|
||||
"""
|
||||
disabled: list[str] = []
|
||||
|
||||
# Embedding-dependent features
|
||||
if not caps["embed"]:
|
||||
disabled.extend(
|
||||
[
|
||||
"vector_search",
|
||||
"hybrid_search",
|
||||
"reflection",
|
||||
"skill_extraction",
|
||||
]
|
||||
)
|
||||
|
||||
# Rerank-dependent feature
|
||||
if not caps["rerank"]:
|
||||
disabled.append("agentic_search")
|
||||
|
||||
# Knowledge requires both embedding and rerank
|
||||
if not (caps["embed"] and caps["rerank"]):
|
||||
disabled.append("knowledge")
|
||||
|
||||
# Multimodal upload requires both multimodal_llm and parser
|
||||
if not (caps["multimodal_llm"] and caps["parser"]):
|
||||
disabled.append("multimodal_upload")
|
||||
|
||||
return disabled
|
||||
@@ -6,9 +6,18 @@ Public surface:
|
||||
- :class:`EmbeddingProvider` — Protocol every provider satisfies.
|
||||
- :class:`EmbeddingServiceError` — provider-side failure.
|
||||
- :class:`EmbeddingError` — backward-compat alias for ``EmbeddingServiceError``.
|
||||
- :class:`EmbeddingCapability` — soft-dependency wrapper around an
|
||||
optional :class:`EmbeddingProvider` (``available`` / ``embed_or_none``
|
||||
/ ``require``).
|
||||
- :class:`OpenAIEmbeddingProvider` — concrete provider for any
|
||||
OpenAI-protocol embeddings endpoint (DeepInfra, vLLM, OpenAI, …).
|
||||
- :func:`build_embedding_provider` — settings-driven factory.
|
||||
- :func:`get_embedding_capability` — process-wide lazy singleton
|
||||
accessor for :class:`EmbeddingCapability`. There is no separate
|
||||
``get_embedder`` accessor: consumers that need a provider call
|
||||
``get_embedding_capability().require()``, which routes every caller
|
||||
through a single shared provider (and its single ``AsyncOpenAI``
|
||||
client + ``asyncio.Semaphore``).
|
||||
|
||||
External usage::
|
||||
|
||||
@@ -19,19 +28,19 @@ External usage::
|
||||
|
||||
from everos.core.errors import EmbeddingServiceError as EmbeddingServiceError
|
||||
|
||||
from .accessor import EmbeddingNotConfiguredError as EmbeddingNotConfiguredError
|
||||
from .accessor import get_embedder as get_embedder
|
||||
from .accessor import get_embedding_capability as get_embedding_capability
|
||||
from .capability import EmbeddingCapability as EmbeddingCapability
|
||||
from .factory import build_embedding_provider as build_embedding_provider
|
||||
from .openai_provider import OpenAIEmbeddingProvider as OpenAIEmbeddingProvider
|
||||
from .protocol import EmbeddingError as EmbeddingError
|
||||
from .protocol import EmbeddingProvider as EmbeddingProvider
|
||||
|
||||
__all__ = [
|
||||
"EmbeddingCapability",
|
||||
"EmbeddingError",
|
||||
"EmbeddingNotConfiguredError",
|
||||
"EmbeddingProvider",
|
||||
"EmbeddingServiceError",
|
||||
"OpenAIEmbeddingProvider",
|
||||
"build_embedding_provider",
|
||||
"get_embedder",
|
||||
"get_embedding_capability",
|
||||
]
|
||||
|
||||
@@ -1,14 +1,16 @@
|
||||
"""Process-wide embedding provider accessor.
|
||||
"""Process-wide embedding capability accessor.
|
||||
|
||||
Lazy singleton mirror of :func:`everos.component.llm.get_llm_client`:
|
||||
first call reads settings and builds the OpenAI-protocol embedding
|
||||
client; subsequent calls return the cached instance. Strategies and
|
||||
other components that need a process-wide embedder import this rather
|
||||
than threading the provider through their constructors.
|
||||
Lazy singleton for :class:`EmbeddingCapability`. The first call reads
|
||||
settings and attempts to build an OpenAI-protocol embedding provider;
|
||||
subsequent calls return the cached wrapper. Consumers that must have an
|
||||
embedder call ``get_embedding_capability().require()``; consumers that
|
||||
can degrade gracefully use ``.embed_or_none`` or check ``.available``.
|
||||
|
||||
Raises :class:`EmbeddingNotConfiguredError` when credentials are missing
|
||||
so misconfiguration surfaces at the call site (or at app startup via a
|
||||
lifespan provider) instead of silently degrading.
|
||||
There is deliberately no separate ``get_embedder()`` accessor: routing
|
||||
every consumer through the capability keeps a single provider (and its
|
||||
underlying ``AsyncOpenAI`` client + ``asyncio.Semaphore``) per process,
|
||||
so the configured ``max_concurrent`` bound holds instead of silently
|
||||
doubling.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -16,33 +18,41 @@ from __future__ import annotations
|
||||
from everos.config import load_settings
|
||||
from everos.core.observability.logging import get_logger
|
||||
|
||||
from .capability import EmbeddingCapability
|
||||
from .factory import build_embedding_provider
|
||||
from .protocol import EmbeddingProvider
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class EmbeddingNotConfiguredError(RuntimeError):
|
||||
"""Raised when ``settings.embedding`` lacks ``model``/``api_key``/``base_url``."""
|
||||
_capability: EmbeddingCapability | None = None
|
||||
|
||||
|
||||
_embedder: EmbeddingProvider | None = None
|
||||
def get_embedding_capability() -> EmbeddingCapability:
|
||||
"""Return the process-wide :class:`EmbeddingCapability`. Never raises.
|
||||
|
||||
On the first call, builds and caches a capability from current
|
||||
settings — ``available`` is ``False`` when the provider cannot be
|
||||
built (missing fields, unsupported provider name, malformed URL, …).
|
||||
The build outcome is cached, so a later settings change requires a
|
||||
process restart to take effect.
|
||||
|
||||
def get_embedder() -> EmbeddingProvider:
|
||||
"""Return the singleton :class:`EmbeddingProvider`.
|
||||
|
||||
Raises:
|
||||
EmbeddingNotConfiguredError: When required settings fields are
|
||||
unset. See :func:`build_embedding_provider` for the exact
|
||||
keys.
|
||||
Configuration failures (:class:`ValueError` from
|
||||
:func:`build_embedding_provider`) are logged at ``warning`` level:
|
||||
the downstream :class:`ProviderNotConfiguredError` message maps both
|
||||
"user hasn't configured it" and "user configured it wrong" onto the
|
||||
same HTTP 422, so the log line is the only place an operator can
|
||||
tell those two states apart.
|
||||
"""
|
||||
global _embedder
|
||||
if _embedder is not None:
|
||||
return _embedder
|
||||
global _capability
|
||||
if _capability is not None:
|
||||
return _capability
|
||||
try:
|
||||
_embedder = build_embedding_provider(load_settings().embedding)
|
||||
provider = build_embedding_provider(load_settings().embedding)
|
||||
except ValueError as exc:
|
||||
raise EmbeddingNotConfiguredError(str(exc)) from exc
|
||||
logger.info("embedder_built")
|
||||
return _embedder
|
||||
logger.warning(
|
||||
"embedding_capability_build_failed",
|
||||
reason=str(exc),
|
||||
)
|
||||
provider = None
|
||||
_capability = EmbeddingCapability(provider=provider)
|
||||
return _capability
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
"""EmbeddingCapability — soft-dependency wrapper for EmbeddingProvider.
|
||||
|
||||
The wrapper encapsulates ``Optional[EmbeddingProvider]`` so consumers
|
||||
never see ``None`` directly. Two consumption modes:
|
||||
|
||||
- :meth:`embed_or_none` for soft-degrade write paths (cascade handlers
|
||||
that write ``vector=NULL`` when embedding is unavailable).
|
||||
- :meth:`require` for hard-required paths (search VECTOR/HYBRID/AGENTIC)
|
||||
that raise :class:`ProviderNotConfiguredError` -> HTTP 422 when missing.
|
||||
|
||||
Design rationale: EverOS has multiple code paths that need to know
|
||||
whether embedding is available. Rather than each callsite reading
|
||||
settings and re-implementing the check, the check lives in
|
||||
:func:`everos.component.embedding.accessor.get_embedding_capability`
|
||||
(module-level singleton); every consumer asks the capability.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
from everos.core.errors import ProviderNotConfiguredError
|
||||
|
||||
from .protocol import EmbeddingProvider
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class EmbeddingCapability:
|
||||
"""Wraps an optional EmbeddingProvider with soft / strict consumption modes."""
|
||||
|
||||
provider: EmbeddingProvider | None
|
||||
|
||||
@property
|
||||
def available(self) -> bool:
|
||||
"""True iff a provider was successfully constructed."""
|
||||
return self.provider is not None
|
||||
|
||||
async def embed_or_none(self, text: str) -> list[float] | None:
|
||||
"""Embed ``text`` if available, else return ``None``.
|
||||
|
||||
For write paths that can proceed without a vector (cascade
|
||||
handlers write ``vector=NULL`` on the LanceDB row).
|
||||
"""
|
||||
if self.provider is None:
|
||||
return None
|
||||
return list(await self.provider.embed(text))
|
||||
|
||||
def require(self) -> EmbeddingProvider:
|
||||
"""Return the provider, or raise :class:`ProviderNotConfiguredError`.
|
||||
|
||||
For paths that cannot proceed without embedding (search
|
||||
VECTOR/HYBRID/AGENTIC methods). Caller supplies its own
|
||||
``feature``/``alternative_hint`` context by catching and
|
||||
re-raising if needed; this base call only identifies the
|
||||
missing provider as ``"embedding"``.
|
||||
"""
|
||||
if self.provider is None:
|
||||
raise ProviderNotConfiguredError(provider="embedding")
|
||||
return self.provider
|
||||
@@ -2,6 +2,7 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from everos.component.utils.config_hints import missing_config_error
|
||||
from everos.config import EmbeddingSettings
|
||||
|
||||
from .openai_provider import OpenAIEmbeddingProvider
|
||||
@@ -32,18 +33,11 @@ def build_embedding_provider(
|
||||
ValueError: If ``model``, ``api_key`` or ``base_url`` is unset.
|
||||
"""
|
||||
if not settings.model:
|
||||
raise ValueError(
|
||||
"Embedding model is not configured "
|
||||
"(set EVEROS_EMBEDDING__MODEL or [embedding] model in user toml)"
|
||||
)
|
||||
raise ValueError(missing_config_error("Embedding model", "embedding"))
|
||||
if not settings.api_key or not settings.api_key.get_secret_value():
|
||||
raise ValueError(
|
||||
"Embedding api_key is not configured (set EVEROS_EMBEDDING__API_KEY)"
|
||||
)
|
||||
raise ValueError(missing_config_error("Embedding api_key", "embedding"))
|
||||
if not settings.base_url:
|
||||
raise ValueError(
|
||||
"Embedding base_url is not configured (set EVEROS_EMBEDDING__BASE_URL)"
|
||||
)
|
||||
raise ValueError(missing_config_error("Embedding base_url", "embedding"))
|
||||
return OpenAIEmbeddingProvider(
|
||||
model=settings.model,
|
||||
api_key=settings.api_key.get_secret_value(),
|
||||
|
||||
@@ -17,6 +17,7 @@ from everalgo.llm.protocols import LLMClient
|
||||
from everalgo.llm.types import ChatMessage, ChatResponse
|
||||
from pydantic import BaseModel
|
||||
|
||||
from everos.component.utils.config_hints import missing_config_error
|
||||
from everos.config import load_settings
|
||||
from everos.core.observability.logging import get_logger
|
||||
|
||||
@@ -93,7 +94,7 @@ def get_llm_client() -> LLMClient:
|
||||
)
|
||||
if not api_key or not llm_cfg.base_url:
|
||||
raise LLMNotConfiguredError(
|
||||
"LLM is required; set EVEROS_LLM__API_KEY + EVEROS_LLM__BASE_URL"
|
||||
missing_config_error("LLM api_key and base_url", "llm")
|
||||
)
|
||||
client: LLMClient = build_client(
|
||||
LLMConfig(
|
||||
@@ -133,8 +134,7 @@ def get_multimodal_llm_client() -> LLMClient:
|
||||
api_key = cfg.api_key.get_secret_value() if cfg.api_key is not None else None
|
||||
if not api_key or not cfg.base_url:
|
||||
raise LLMNotConfiguredError(
|
||||
"Multimodal LLM is required for parsing; set "
|
||||
"EVEROS_MULTIMODAL__API_KEY + EVEROS_MULTIMODAL__BASE_URL"
|
||||
missing_config_error("Multimodal LLM api_key and base_url", "multimodal")
|
||||
)
|
||||
_multimodal_client = build_client(
|
||||
LLMConfig(
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from everos.component.utils.config_hints import missing_config_error
|
||||
from everos.config import LLMSettings
|
||||
|
||||
from .openai_provider import OpenAIProvider
|
||||
@@ -29,15 +30,9 @@ def build_llm_provider(settings: LLMSettings) -> LLMClient:
|
||||
ValueError: If ``api_key`` or ``base_url`` is unset.
|
||||
"""
|
||||
if not settings.api_key or not settings.api_key.get_secret_value():
|
||||
raise ValueError(
|
||||
"LLM api_key is not configured "
|
||||
"(set EVEROS_LLM__API_KEY or [llm] api_key in user toml)"
|
||||
)
|
||||
raise ValueError(missing_config_error("LLM api_key", "llm"))
|
||||
if not settings.base_url:
|
||||
raise ValueError(
|
||||
"LLM base_url is not configured "
|
||||
"(set EVEROS_LLM__BASE_URL or [llm] base_url in user toml)"
|
||||
)
|
||||
raise ValueError(missing_config_error("LLM base_url", "llm"))
|
||||
return OpenAIProvider(
|
||||
model=settings.model,
|
||||
api_key=settings.api_key.get_secret_value(),
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
"""Multimodal LLM capability — optional vision/audio support for parsing.
|
||||
|
||||
Public surface:
|
||||
|
||||
- :class:`MultimodalLLMCapability` — soft-dependency wrapper around an
|
||||
optional multimodal LLMClient (``available`` / ``require``; no soft-degrade
|
||||
accessor — multimodal parsing is entirely optional).
|
||||
- :func:`get_multimodal_llm_capability` — process-wide lazy singleton accessor
|
||||
for :class:`MultimodalLLMCapability`.
|
||||
|
||||
External usage::
|
||||
|
||||
from everos.component.multimodal import get_multimodal_llm_capability
|
||||
cap = get_multimodal_llm_capability()
|
||||
if cap.available:
|
||||
client = cap.require()
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from .accessor import get_multimodal_llm_capability as get_multimodal_llm_capability
|
||||
from .capability import MultimodalLLMCapability as MultimodalLLMCapability
|
||||
|
||||
__all__ = [
|
||||
"MultimodalLLMCapability",
|
||||
"get_multimodal_llm_capability",
|
||||
]
|
||||
@@ -0,0 +1,55 @@
|
||||
"""Process-wide multimodal LLM capability accessor.
|
||||
|
||||
Lazy singleton — first call reads settings and attempts to build a multimodal
|
||||
LLM client. Unlike :func:`everos.component.llm.client.get_multimodal_llm_client`
|
||||
(which raises when misconfigured), this wraps the outcome in a capability that
|
||||
reports ``available=False`` when the client cannot be built.
|
||||
|
||||
Subsequent calls return the cached instance.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from everos.component.llm.client import (
|
||||
LLMNotConfiguredError,
|
||||
get_multimodal_llm_client,
|
||||
)
|
||||
from everos.core.observability.logging import get_logger
|
||||
|
||||
from .capability import MultimodalLLMCapability
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
_capability: MultimodalLLMCapability | None = None
|
||||
|
||||
|
||||
def get_multimodal_llm_capability() -> MultimodalLLMCapability:
|
||||
"""Return the process-wide :class:`MultimodalLLMCapability`. Never raises.
|
||||
|
||||
Lazy singleton: the first call attempts to build a multimodal client from
|
||||
current settings — ``available`` is ``False`` when the client cannot be
|
||||
built (e.g. missing model/base_url/api_key). Use this from upload
|
||||
endpoints, the health endpoint, the startup banner, and any other caller
|
||||
that needs to check "is multimodal parsing available?" without a hard
|
||||
dependency.
|
||||
|
||||
Build failures (``ValueError`` from the factory / ``LLMNotConfiguredError``
|
||||
from missing settings) are logged as
|
||||
``multimodal_llm_capability_build_failed`` so an operator can distinguish
|
||||
"not configured" from "misconfigured" — mirrors the equivalent warning on
|
||||
the embedding and rerank accessors so all three optional providers surface
|
||||
the same signal.
|
||||
"""
|
||||
global _capability
|
||||
if _capability is not None:
|
||||
return _capability
|
||||
try:
|
||||
provider = get_multimodal_llm_client()
|
||||
except (ValueError, LLMNotConfiguredError) as exc:
|
||||
logger.warning(
|
||||
"multimodal_llm_capability_build_failed",
|
||||
reason=str(exc),
|
||||
)
|
||||
provider = None
|
||||
_capability = MultimodalLLMCapability(provider=provider)
|
||||
return _capability
|
||||
@@ -0,0 +1,40 @@
|
||||
"""MultimodalLLMCapability — soft-dependency wrapper for multimodal LLM client.
|
||||
|
||||
Parallel structure to :class:`everos.component.rerank.RerankCapability`, but
|
||||
for the multimodal LLM used by ``everalgo.parser``. A caller either requires
|
||||
multimodal (raising :class:`ProviderNotConfiguredError` -> HTTP 422 when
|
||||
missing) or checks ``available`` to skip parsing when unavailable.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from everos.core.errors import ProviderNotConfiguredError
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from everalgo.llm.protocols import LLMClient
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class MultimodalLLMCapability:
|
||||
"""Wraps an optional multimodal LLMClient with a hard-require API."""
|
||||
|
||||
provider: LLMClient | None
|
||||
|
||||
@property
|
||||
def available(self) -> bool:
|
||||
"""True iff a provider was successfully constructed."""
|
||||
return self.provider is not None
|
||||
|
||||
def require(self) -> LLMClient:
|
||||
"""Return the provider, or raise :class:`ProviderNotConfiguredError`.
|
||||
|
||||
Caller supplies its own ``feature``/``alternative_hint`` context by
|
||||
catching and re-raising if needed; this base call only identifies
|
||||
the missing provider as ``"multimodal_llm"``.
|
||||
"""
|
||||
if self.provider is None:
|
||||
raise ProviderNotConfiguredError(provider="multimodal_llm")
|
||||
return self.provider
|
||||
@@ -6,6 +6,7 @@ All imports are deferred so this module is safe to import without the extra.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from functools import lru_cache
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from everos.core.errors import MultimodalNotEnabledError, UnsupportedModalityError
|
||||
@@ -14,8 +15,22 @@ if TYPE_CHECKING:
|
||||
from everalgo.types import ParsedContent, RawFile
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def parser_available() -> bool:
|
||||
"""Whether ``everalgo.parser`` is importable."""
|
||||
"""Whether ``everalgo.parser`` is importable.
|
||||
|
||||
Memoised: the underlying ``import everalgo.parser`` pulls in heavy
|
||||
PDF/Office dependencies (``pypdf``, ``python-docx``, ...) that would
|
||||
block the event loop for hundreds of ms — unacceptable inside
|
||||
``async def health()`` on a liveness probe. First call at startup
|
||||
(via :class:`ParserLifespanProvider`) pays the import cost off the
|
||||
request path; subsequent calls hit the cache instantly.
|
||||
|
||||
The cache is process-wide. Tests that patch ``sys.modules`` to
|
||||
swap the ``everalgo.parser`` module in or out must invoke
|
||||
:meth:`parser_available.cache_clear` between assertions to avoid
|
||||
cross-test contamination.
|
||||
"""
|
||||
try:
|
||||
import everalgo.parser # noqa: F401
|
||||
except ImportError:
|
||||
|
||||
@@ -5,6 +5,9 @@ Public surface:
|
||||
- :class:`RerankProvider` — Protocol every provider satisfies.
|
||||
- :class:`RerankResult` / :class:`RerankServiceError` — value type + error.
|
||||
- :class:`RerankError` — backward-compat alias for :class:`RerankServiceError`.
|
||||
- :class:`RerankCapability` — soft-dependency wrapper around an optional
|
||||
:class:`RerankProvider` (``available`` / ``require``; no soft-degrade
|
||||
accessor — rerank has no write path to degrade).
|
||||
- :class:`DeepInfraRerankProvider` — DeepInfra inference-API rerank.
|
||||
- :class:`VllmRerankProvider` — OpenAI-compat ``/v1/rerank`` (vLLM,
|
||||
self-hosted, other compatible servers).
|
||||
@@ -12,6 +15,8 @@ Public surface:
|
||||
``gte-rerank-v2`` native text-rerank endpoint.
|
||||
- :func:`build_rerank_provider` — settings-driven factory that picks
|
||||
the concrete provider via ``settings.rerank.provider``.
|
||||
- :func:`get_rerank_capability` — process-wide lazy singleton accessor
|
||||
for :class:`RerankCapability`.
|
||||
|
||||
External usage::
|
||||
|
||||
@@ -22,6 +27,8 @@ External usage::
|
||||
|
||||
from everos.core.errors import RerankServiceError as RerankServiceError
|
||||
|
||||
from .accessor import get_rerank_capability as get_rerank_capability
|
||||
from .capability import RerankCapability as RerankCapability
|
||||
from .dashscope_provider import DashScopeRerankProvider as DashScopeRerankProvider
|
||||
from .deepinfra_provider import DeepInfraRerankProvider as DeepInfraRerankProvider
|
||||
from .factory import build_rerank_provider as build_rerank_provider
|
||||
@@ -33,10 +40,12 @@ from .vllm_provider import VllmRerankProvider as VllmRerankProvider
|
||||
__all__ = [
|
||||
"DashScopeRerankProvider",
|
||||
"DeepInfraRerankProvider",
|
||||
"RerankCapability",
|
||||
"RerankError",
|
||||
"RerankProvider",
|
||||
"RerankResult",
|
||||
"RerankServiceError",
|
||||
"VllmRerankProvider",
|
||||
"build_rerank_provider",
|
||||
"get_rerank_capability",
|
||||
]
|
||||
|
||||
@@ -0,0 +1,57 @@
|
||||
"""Process-wide rerank capability accessor.
|
||||
|
||||
Lazy singleton mirror of
|
||||
:func:`everos.component.embedding.accessor.get_embedding_capability`: first
|
||||
call reads settings, attempts to build a rerank provider, and wraps the
|
||||
outcome (provider or ``None``) in a :class:`RerankCapability`. Subsequent
|
||||
calls return the cached instance.
|
||||
|
||||
Rerank is a Tier-3 optional provider — call sites either go through
|
||||
:func:`get_rerank_capability` and call ``.require()`` when rerank is
|
||||
mandatory, or check ``.available`` and skip reranking when it is not.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from everos.config import load_settings
|
||||
from everos.core.observability.logging import get_logger
|
||||
|
||||
from .capability import RerankCapability
|
||||
from .factory import build_rerank_provider
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
_capability: RerankCapability | None = None
|
||||
|
||||
|
||||
def get_rerank_capability() -> RerankCapability:
|
||||
"""Return the process-wide :class:`RerankCapability`. Never raises.
|
||||
|
||||
Lazy singleton: the first call builds and caches a capability from
|
||||
current settings — ``available`` is ``False`` when the provider cannot
|
||||
be built (e.g. missing model/base_url/api_key, unsupported provider
|
||||
name, malformed URL, …). Use this from search strategies, the health
|
||||
endpoint, the startup banner, and any other caller that needs to
|
||||
check "is rerank available?" without a hard dependency.
|
||||
|
||||
Configuration failures (:class:`ValueError` from
|
||||
:func:`build_rerank_provider`) are logged at ``warning`` level: the
|
||||
downstream :class:`ProviderNotConfiguredError` message maps both
|
||||
"user hasn't configured it" and "user configured it wrong" onto the
|
||||
same HTTP 422, so the log line is the only place an operator can
|
||||
tell those two states apart.
|
||||
"""
|
||||
global _capability
|
||||
if _capability is not None:
|
||||
return _capability
|
||||
try:
|
||||
provider = build_rerank_provider(load_settings().rerank)
|
||||
except ValueError as exc:
|
||||
logger.warning(
|
||||
"rerank_capability_build_failed",
|
||||
reason=str(exc),
|
||||
)
|
||||
provider = None
|
||||
_capability = RerankCapability(provider=provider)
|
||||
return _capability
|
||||
@@ -0,0 +1,39 @@
|
||||
"""RerankCapability — soft-dependency wrapper for RerankProvider.
|
||||
|
||||
Parallel structure to :class:`everos.component.embedding.EmbeddingCapability`,
|
||||
but without a soft-degrade accessor: rerank is a query-time enhancement,
|
||||
not a write path. A caller either hard-requires rerank (raising
|
||||
:class:`ProviderNotConfiguredError` -> HTTP 422 when missing) or chooses to
|
||||
skip reranking entirely — there is no equivalent of ``embed_or_none``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
from everos.core.errors import ProviderNotConfiguredError
|
||||
|
||||
from .protocol import RerankProvider
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RerankCapability:
|
||||
"""Wraps an optional RerankProvider with a hard-require API."""
|
||||
|
||||
provider: RerankProvider | None
|
||||
|
||||
@property
|
||||
def available(self) -> bool:
|
||||
"""True iff a provider was successfully constructed."""
|
||||
return self.provider is not None
|
||||
|
||||
def require(self) -> RerankProvider:
|
||||
"""Return the provider, or raise :class:`ProviderNotConfiguredError`.
|
||||
|
||||
Caller supplies its own ``feature``/``alternative_hint`` context by
|
||||
catching and re-raising if needed; this base call only identifies
|
||||
the missing provider as ``"rerank"``.
|
||||
"""
|
||||
if self.provider is None:
|
||||
raise ProviderNotConfiguredError(provider="rerank")
|
||||
return self.provider
|
||||
@@ -13,6 +13,7 @@ Adding a new provider = one match arm here + one new file under
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from everos.component.utils.config_hints import missing_config_error
|
||||
from everos.config import RerankSettings
|
||||
|
||||
from .dashscope_provider import DashScopeRerankProvider
|
||||
@@ -38,22 +39,14 @@ def build_rerank_provider(settings: RerankSettings) -> RerankProvider:
|
||||
string) for ``vllm`` self-hosted endpoints.
|
||||
"""
|
||||
if not settings.model:
|
||||
raise ValueError(
|
||||
"Rerank model is not configured "
|
||||
"(set EVEROS_RERANK__MODEL or [rerank] model in user toml)"
|
||||
)
|
||||
raise ValueError(missing_config_error("Rerank model", "rerank"))
|
||||
if not settings.base_url:
|
||||
raise ValueError(
|
||||
"Rerank base_url is not configured (set EVEROS_RERANK__BASE_URL)"
|
||||
)
|
||||
raise ValueError(missing_config_error("Rerank base_url", "rerank"))
|
||||
api_key = settings.api_key.get_secret_value() if settings.api_key else ""
|
||||
|
||||
if settings.provider == "deepinfra":
|
||||
if not api_key:
|
||||
raise ValueError(
|
||||
"DeepInfra rerank api_key is not configured "
|
||||
"(set EVEROS_RERANK__API_KEY)"
|
||||
)
|
||||
raise ValueError(missing_config_error("DeepInfra rerank api_key", "rerank"))
|
||||
return DeepInfraRerankProvider(
|
||||
model=settings.model,
|
||||
api_key=api_key,
|
||||
@@ -75,10 +68,7 @@ def build_rerank_provider(settings: RerankSettings) -> RerankProvider:
|
||||
)
|
||||
if settings.provider == "dashscope":
|
||||
if not api_key:
|
||||
raise ValueError(
|
||||
"DashScope rerank api_key is not configured "
|
||||
"(set EVEROS_RERANK__API_KEY)"
|
||||
)
|
||||
raise ValueError(missing_config_error("DashScope rerank api_key", "rerank"))
|
||||
return DashScopeRerankProvider(
|
||||
model=settings.model,
|
||||
api_key=api_key,
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
"""User-facing error message helpers for missing provider configuration.
|
||||
|
||||
Guidance points users at the ``<root>/everos.toml`` file, not at
|
||||
environment variables. Env vars still work as an override mechanism
|
||||
(see ``pydantic-settings`` precedence in ``config/settings.py``) but
|
||||
are not surfaced in onboarding-facing text.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from everos.core.persistence import MemoryRoot
|
||||
|
||||
|
||||
def missing_config_error(field_label: str, toml_section: str) -> str:
|
||||
"""Return a uniform error message for a missing config field.
|
||||
|
||||
Args:
|
||||
field_label: Human-readable label (e.g. ``"LLM api_key"``).
|
||||
toml_section: TOML section name without brackets (e.g. ``"llm"``).
|
||||
|
||||
Returns:
|
||||
A single-line message including the resolved memory-root path
|
||||
and a hint to run ``everos init``. Never mentions env vars.
|
||||
"""
|
||||
root = MemoryRoot.resolve().root
|
||||
return (
|
||||
f"{field_label} is not configured. "
|
||||
f"Edit {root}/everos.toml (run `everos init` to scaffold), "
|
||||
f"section [{toml_section}]."
|
||||
)
|
||||
@@ -36,6 +36,7 @@ class ErrorCode(StrEnum):
|
||||
CAPABILITY_UNAVAILABLE = "CAPABILITY_UNAVAILABLE"
|
||||
CONFIGURATION_ERROR = "CONFIGURATION_ERROR"
|
||||
INTERNAL_ERROR = "INTERNAL_ERROR"
|
||||
PROVIDER_NOT_CONFIGURED = "PROVIDER_NOT_CONFIGURED"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -175,6 +176,56 @@ class ConfigurationError(AppError):
|
||||
"""
|
||||
|
||||
|
||||
# TOML section name for each provider kind, keyed by the `provider` argument
|
||||
# passed to `ProviderNotConfiguredError`.
|
||||
_PROVIDER_SECTIONS: dict[str, str] = {
|
||||
"llm": "llm",
|
||||
"embedding": "embedding",
|
||||
"rerank": "rerank",
|
||||
"multimodal_llm": "multimodal",
|
||||
}
|
||||
|
||||
|
||||
class ProviderNotConfiguredError(ConfigurationError):
|
||||
"""Raised when a runtime path requires a provider that is not configured.
|
||||
|
||||
Maps to HTTP 422 via the FastAPI exception handler; the message is
|
||||
directly user-facing and points at the toml file for remediation.
|
||||
|
||||
Args:
|
||||
provider: Which provider is missing. One of ``"llm"``,
|
||||
``"embedding"``, ``"rerank"``, ``"multimodal_llm"``.
|
||||
feature: Optional user-facing feature that required this
|
||||
provider (e.g. ``"knowledge"``, ``"agent_hybrid"``).
|
||||
alternative_hint: Optional alternative workaround the user can
|
||||
take without configuring the missing provider (e.g. flip
|
||||
``enable_llm_rerank=true`` to use the LLM lane).
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
provider: str,
|
||||
feature: str | None = None,
|
||||
alternative_hint: str | None = None,
|
||||
) -> None:
|
||||
# Lazy import: `config_hints` -> `core.persistence` -> `core.persistence
|
||||
# .markdown.writer` imports `PathTraversalError` from this module, so a
|
||||
# module-level import here would be circular.
|
||||
from everos.component.utils.config_hints import missing_config_error
|
||||
|
||||
self.provider = provider
|
||||
self.feature = feature
|
||||
self.alternative_hint = alternative_hint
|
||||
section = _PROVIDER_SECTIONS.get(provider, provider)
|
||||
field_label = f"Provider '{provider}'"
|
||||
if feature:
|
||||
field_label += f" (required by {feature})"
|
||||
message = missing_config_error(field_label, section)
|
||||
if alternative_hint:
|
||||
message += f" Alternative: {alternative_hint}"
|
||||
super().__init__(message)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Backward compatibility aliases
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -23,11 +23,12 @@ The cascade queue, LSN watermark, and change audit all live in
|
||||
|
||||
The default location and tunables come from :class:`everos.config.Settings`
|
||||
(loaded from ``config/default.toml`` + ``EVEROS_*`` environment variables);
|
||||
:meth:`MemoryRoot.default` resolves the configured path.
|
||||
:meth:`MemoryRoot.resolve` resolves the configured path.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import warnings
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
@@ -92,7 +93,7 @@ class MemoryRoot:
|
||||
object.__setattr__(self, "root", resolved)
|
||||
|
||||
@classmethod
|
||||
def default(cls, *, explicit_root: str | None = None) -> MemoryRoot:
|
||||
def resolve(cls, *, explicit_root: str | None = None) -> MemoryRoot:
|
||||
"""Return the memory-root resolved from CLI / env / default.
|
||||
|
||||
Resolution: ``explicit_root`` > ``EVEROS_ROOT`` env > ``~/.everos``.
|
||||
@@ -105,6 +106,25 @@ class MemoryRoot:
|
||||
|
||||
return cls(resolve_root(explicit_root))
|
||||
|
||||
@classmethod
|
||||
def default(cls, *, explicit_root: str | None = None) -> MemoryRoot:
|
||||
"""Deprecated alias for :meth:`resolve`. Removed in a future major.
|
||||
|
||||
Kept as a backward-compatibility shim for v1.2.0 callers who used
|
||||
the old ``default()`` name (renamed to ``resolve()`` in this
|
||||
release). Emits :class:`DeprecationWarning` on every call so
|
||||
existing callers surface in test / CI runs; the constructor
|
||||
semantics are identical.
|
||||
"""
|
||||
warnings.warn(
|
||||
"MemoryRoot.default() is deprecated and renamed to "
|
||||
"MemoryRoot.resolve(); the alias will be removed in a "
|
||||
"future major release. Update call sites.",
|
||||
DeprecationWarning,
|
||||
stacklevel=2,
|
||||
)
|
||||
return cls.resolve(explicit_root=explicit_root)
|
||||
|
||||
# ── User-visible (partitioned by app / project) ──────────────────────────
|
||||
#
|
||||
# These take ``(app_id, project_id)`` because the scope dirs hang off the
|
||||
|
||||
@@ -34,6 +34,7 @@ from .lifespans import (
|
||||
LanceDBLifespanProvider,
|
||||
LLMLifespanProvider,
|
||||
OmeLifespanProvider,
|
||||
ParserLifespanProvider,
|
||||
SqliteLifespanProvider,
|
||||
)
|
||||
from .routes import (
|
||||
@@ -71,9 +72,9 @@ def create_app(
|
||||
cors_allow_headers: Allowed CORS headers (default: ``["*"]``).
|
||||
lifespan_providers: Optional list of LifespanProvider; defaults to
|
||||
``[TracingLifespanProvider(), MetricsLifespanProvider(),
|
||||
LLMLifespanProvider(), SqliteLifespanProvider(),
|
||||
LanceDBLifespanProvider(), CascadeLifespanProvider(),
|
||||
OmeLifespanProvider()]``.
|
||||
LLMLifespanProvider(), ParserLifespanProvider(),
|
||||
SqliteLifespanProvider(), LanceDBLifespanProvider(),
|
||||
CascadeLifespanProvider(), OmeLifespanProvider()]``.
|
||||
|
||||
Returns:
|
||||
FastAPI: Configured application instance.
|
||||
@@ -85,6 +86,7 @@ def create_app(
|
||||
TracingLifespanProvider(),
|
||||
MetricsLifespanProvider(),
|
||||
LLMLifespanProvider(),
|
||||
ParserLifespanProvider(),
|
||||
SqliteLifespanProvider(),
|
||||
LanceDBLifespanProvider(),
|
||||
CascadeLifespanProvider(),
|
||||
|
||||
@@ -43,6 +43,7 @@ from everos.core.errors import (
|
||||
InvalidInputError,
|
||||
NotFoundError,
|
||||
PathTraversalError,
|
||||
ProviderNotConfiguredError,
|
||||
UnsupportedModalityError,
|
||||
)
|
||||
from everos.core.observability.logging import get_logger
|
||||
@@ -241,6 +242,31 @@ async def configuration_handler(
|
||||
)
|
||||
|
||||
|
||||
async def provider_not_configured_handler(
|
||||
request: Request,
|
||||
exc: ProviderNotConfiguredError,
|
||||
) -> JSONResponse:
|
||||
"""ProviderNotConfiguredError -> 422 (client-actionable, unlike its parent).
|
||||
|
||||
Unlike a generic ``ConfigurationError`` (500 -- an operator bug), a
|
||||
missing provider is something the caller can fix by editing
|
||||
``everos.toml``, so this more-specific subclass gets its own 422
|
||||
mapping instead of falling through to ``configuration_handler``.
|
||||
"""
|
||||
logger.warning(
|
||||
"provider_not_configured_error",
|
||||
path=str(request.url.path),
|
||||
provider=exc.provider,
|
||||
feature=exc.feature,
|
||||
)
|
||||
return _error_response(
|
||||
request,
|
||||
HTTP_422_UNPROCESSABLE_CONTENT,
|
||||
ErrorCode.PROVIDER_NOT_CONFIGURED,
|
||||
str(exc),
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Pydantic / FastAPI built-in exceptions
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -350,7 +376,10 @@ def register_handlers(app: FastAPI) -> None:
|
||||
app.add_exception_handler(InfrastructureError, infrastructure_handler)
|
||||
# Capability errors (permanent, not retryable)
|
||||
app.add_exception_handler(CapabilityError, capability_handler)
|
||||
# Configuration errors
|
||||
# Configuration errors (specific before parent)
|
||||
app.add_exception_handler(
|
||||
ProviderNotConfiguredError, provider_not_configured_handler
|
||||
)
|
||||
app.add_exception_handler(ConfigurationError, configuration_handler)
|
||||
# FastAPI built-in exceptions
|
||||
app.add_exception_handler(HTTPException, http_exception_handler)
|
||||
|
||||
@@ -13,6 +13,7 @@ External usage::
|
||||
|
||||
from everos.entrypoints.api.lifespans import (
|
||||
LLMLifespanProvider,
|
||||
ParserLifespanProvider,
|
||||
SqliteLifespanProvider,
|
||||
LanceDBLifespanProvider,
|
||||
CascadeLifespanProvider,
|
||||
@@ -24,6 +25,7 @@ from .cascade import CascadeLifespanProvider as CascadeLifespanProvider
|
||||
from .lancedb import LanceDBLifespanProvider as LanceDBLifespanProvider
|
||||
from .llm import LLMLifespanProvider as LLMLifespanProvider
|
||||
from .ome import OmeLifespanProvider as OmeLifespanProvider
|
||||
from .parser import ParserLifespanProvider as ParserLifespanProvider
|
||||
from .sqlite import SqliteLifespanProvider as SqliteLifespanProvider
|
||||
|
||||
__all__ = [
|
||||
@@ -31,5 +33,6 @@ __all__ = [
|
||||
"LLMLifespanProvider",
|
||||
"LanceDBLifespanProvider",
|
||||
"OmeLifespanProvider",
|
||||
"ParserLifespanProvider",
|
||||
"SqliteLifespanProvider",
|
||||
]
|
||||
|
||||
@@ -4,9 +4,12 @@ Ordered after SqliteLifespan + LanceDBLifespan: the orchestrator
|
||||
depends on both stores being ready before its watcher / scanner /
|
||||
worker tasks can take the first row.
|
||||
|
||||
Construction reads the live :class:`Settings` to build the embedding +
|
||||
tokenizer providers. If either is misconfigured the lifespan fails
|
||||
fast — the daemon would be useless without them anyway.
|
||||
Construction reads the live :class:`Settings` to build the tokenizer
|
||||
provider, which fails fast if misconfigured. Embedding is a soft
|
||||
dependency: startup warms the process-wide
|
||||
:class:`~everos.component.embedding.EmbeddingCapability` singleton via
|
||||
:func:`get_embedding_capability`, which never raises — the daemon
|
||||
runs in keyword-only mode when embedding is unavailable.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -15,9 +18,8 @@ from typing import Any
|
||||
|
||||
from fastapi import FastAPI
|
||||
|
||||
from everos.component.embedding import build_embedding_provider
|
||||
from everos.component.embedding import get_embedding_capability
|
||||
from everos.component.tokenizer import build_tokenizer
|
||||
from everos.config import load_settings
|
||||
from everos.core.lifespan import LifespanProvider
|
||||
from everos.core.observability.logging import get_logger
|
||||
from everos.core.persistence import MemoryRoot
|
||||
@@ -34,15 +36,22 @@ class CascadeLifespanProvider(LifespanProvider):
|
||||
self._orchestrator: CascadeOrchestrator | None = None
|
||||
|
||||
async def startup(self, app: FastAPI) -> Any:
|
||||
settings = load_settings()
|
||||
memory_root = MemoryRoot.default()
|
||||
memory_root = MemoryRoot.resolve()
|
||||
memory_root.ensure()
|
||||
|
||||
embedder = build_embedding_provider(settings.embedding)
|
||||
tokenizer = build_tokenizer()
|
||||
|
||||
capability = get_embedding_capability()
|
||||
if capability.available:
|
||||
logger.info("cascade_startup_embed_available")
|
||||
else:
|
||||
logger.info(
|
||||
"cascade_startup_embed_unavailable",
|
||||
reason="embedding not configured; keyword-only mode",
|
||||
)
|
||||
|
||||
self._orchestrator = CascadeOrchestrator(
|
||||
memory_root=memory_root,
|
||||
embedder=embedder,
|
||||
tokenizer=tokenizer,
|
||||
)
|
||||
await self._orchestrator.start()
|
||||
|
||||
@@ -5,9 +5,22 @@ Startup:
|
||||
Importing :mod:`everos.infra.persistence.lancedb` also triggers the
|
||||
side-effect import of ``tables`` so business schemas are loaded
|
||||
(future: preflight registration).
|
||||
Log hint if unbackfilled (vector IS NULL) rows exist.
|
||||
|
||||
Shutdown:
|
||||
Close the connection (also clears the table cache).
|
||||
|
||||
Unbackfilled hint:
|
||||
The informational "you have unbackfilled memory rows" banner runs
|
||||
an unconditional ``count_rows(filter='vector IS NULL')`` against
|
||||
every business table on startup. An earlier "marker + limit(1)
|
||||
probe" amortisation was reverted (round-3 finding #3): the vector
|
||||
column has no scalar index, so ``limit(1)`` on ``vector IS NULL``
|
||||
costs the same full scan as ``count_rows``. On a clean state the
|
||||
probe scanned the entire empty tail before returning, matching the
|
||||
cost it was meant to avoid; on a dirty state the probe hit early
|
||||
and then the full ``count_rows`` ran anyway, doubling the scan.
|
||||
The marker's ``last_seen_count`` field was written but never read.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -19,19 +32,60 @@ from fastapi import FastAPI
|
||||
from everos.core.lifespan import LifespanProvider
|
||||
from everos.core.observability.logging import get_logger
|
||||
from everos.infra.persistence.lancedb import (
|
||||
BUSINESS_SCHEMAS_WITH_VECTOR,
|
||||
dispose_connection,
|
||||
ensure_business_indexes,
|
||||
get_connection,
|
||||
get_table,
|
||||
verify_business_schemas,
|
||||
)
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
async def _log_unbackfilled_hint() -> None:
|
||||
"""Warn at startup if there are unbackfilled memory rows.
|
||||
|
||||
Runs an unconditional ``count_rows(filter="vector IS NULL")`` per
|
||||
business table. The vector column has no scalar index, so
|
||||
``count_rows`` and any ``limit(1)`` probe cost the same full scan
|
||||
— a previous "marker + probe" optimisation (removed here) turned
|
||||
out to be net-zero on clean state and net-negative on dirty state
|
||||
(probe hits early, then the full count runs anyway = twice the
|
||||
scan).
|
||||
|
||||
Per-table failures are logged as warnings and don't interrupt
|
||||
startup.
|
||||
"""
|
||||
total_null = 0
|
||||
for schema in BUSINESS_SCHEMAS_WITH_VECTOR:
|
||||
try:
|
||||
table = await get_table(schema.TABLE_NAME, schema)
|
||||
count = await table.count_rows(filter="vector IS NULL")
|
||||
except Exception as exc:
|
||||
logger.warning(
|
||||
"unbackfilled_check_failed",
|
||||
schema=schema.__name__,
|
||||
error=repr(exc),
|
||||
)
|
||||
continue
|
||||
if count > 0:
|
||||
total_null += count
|
||||
|
||||
if total_null > 0:
|
||||
banner_logger = get_logger("everos.cli.server")
|
||||
banner_logger.warning(
|
||||
"unbackfilled_memory_rows",
|
||||
count=total_null,
|
||||
hint="Run `everos cascade backfill` to include them in "
|
||||
"vector/hybrid search (optional).",
|
||||
)
|
||||
|
||||
|
||||
class LanceDBLifespanProvider(LifespanProvider):
|
||||
"""Manage the LanceDB connection + table cache for the app lifecycle.
|
||||
|
||||
Startup runs three steps:
|
||||
Startup runs four steps:
|
||||
|
||||
1. ``get_connection`` — lazy-open the async connection.
|
||||
2. ``verify_business_schemas`` — fail loud if an on-disk table's
|
||||
@@ -39,6 +93,7 @@ class LanceDBLifespanProvider(LifespanProvider):
|
||||
online migration; cascade is rebuildable from md so the recovery
|
||||
is documented as ``rm -rf ~/.everos/.index/lancedb``.
|
||||
3. ``ensure_business_indexes`` — idempotent FTS index creation.
|
||||
4. ``_log_unbackfilled_hint`` — warn if unbackfilled rows exist.
|
||||
"""
|
||||
|
||||
def __init__(self, order: int = 11) -> None:
|
||||
@@ -48,6 +103,7 @@ class LanceDBLifespanProvider(LifespanProvider):
|
||||
conn = await get_connection()
|
||||
await verify_business_schemas()
|
||||
await ensure_business_indexes()
|
||||
await _log_unbackfilled_hint()
|
||||
logger.info("lancedb_ready", uri=conn.uri)
|
||||
return conn
|
||||
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
"""Parser lifespan provider — warms the optional ``everalgo.parser`` import.
|
||||
|
||||
``everalgo.parser`` is an optional dependency (``everos[multimodal]``).
|
||||
When installed it pulls in ``pypdf`` / ``python-docx`` / ... on first
|
||||
import — hundreds of milliseconds to seconds of blocking work. That
|
||||
cost is fine at startup but unacceptable on the request path, where
|
||||
:func:`everos.entrypoints.api.routes.health.health` calls
|
||||
:func:`parser_available` from inside ``async def`` — an event-loop
|
||||
block there stalls liveness probes and any in-flight requests behind
|
||||
it.
|
||||
|
||||
This provider resolves :func:`parser_available` once at startup,
|
||||
priming both Python's ``sys.modules`` cache and the
|
||||
:func:`functools.lru_cache` wrapping ``parser_available`` itself. If
|
||||
the extra is not installed, the import fails, the cache stores
|
||||
``False``, and every subsequent probe is a hot dict lookup.
|
||||
|
||||
Ordered between :class:`LLMLifespanProvider` (``order=8``, hard
|
||||
Tier-1 requirement) and :class:`SqliteLifespanProvider` (``order=10``)
|
||||
— the warm is best-effort chassis hygiene that must not delay the
|
||||
storage stack coming up.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from fastapi import FastAPI
|
||||
|
||||
from everos.component.parser import parser_available
|
||||
from everos.core.lifespan import LifespanProvider
|
||||
from everos.core.observability.logging import get_logger
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class ParserLifespanProvider(LifespanProvider):
|
||||
"""Warm the ``everalgo.parser`` import once at startup.
|
||||
|
||||
Never fails startup: when the optional extra is not installed,
|
||||
:func:`parser_available` returns ``False`` and this provider logs
|
||||
the fact at INFO so operators see a single line rather than the
|
||||
hidden per-request block that the pre-warm eliminates.
|
||||
"""
|
||||
|
||||
def __init__(self, order: int = 9) -> None:
|
||||
# Slot picked to sit between LLM (order=8, hard requirement) and
|
||||
# sqlite (order=10) — verified against every existing provider's
|
||||
# `order=` default (see PR #361 review notes on M-c).
|
||||
super().__init__(name="parser", order=order)
|
||||
|
||||
async def startup(self, app: FastAPI) -> Any:
|
||||
available = parser_available()
|
||||
logger.info("parser_lifespan_ready", available=available)
|
||||
return None
|
||||
|
||||
async def shutdown(self, app: FastAPI) -> None:
|
||||
# Nothing to tear down — the import is process-scoped and lives
|
||||
# in `sys.modules` for the process lifetime.
|
||||
return None
|
||||
@@ -3,11 +3,71 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from fastapi import APIRouter
|
||||
from pydantic import BaseModel
|
||||
|
||||
from everos import __version__
|
||||
from everos.component.capabilities import compute_disabled_features
|
||||
from everos.component.embedding import get_embedding_capability
|
||||
from everos.component.multimodal import get_multimodal_llm_capability
|
||||
from everos.component.parser import parser_available
|
||||
from everos.component.rerank import get_rerank_capability
|
||||
|
||||
router = APIRouter(tags=["health"])
|
||||
|
||||
|
||||
@router.get("/health")
|
||||
async def health() -> dict[str, str]:
|
||||
"""Liveness probe — returns ``{"status": "ok"}`` with HTTP 200."""
|
||||
return {"status": "ok"}
|
||||
class HealthCapabilities(BaseModel):
|
||||
"""Availability flags for the five capability probes.
|
||||
|
||||
Field order matches the health-endpoint payload contract; clients
|
||||
key off these names to decide whether to expose optional features.
|
||||
"""
|
||||
|
||||
llm: bool
|
||||
embed: bool
|
||||
rerank: bool
|
||||
multimodal_llm: bool
|
||||
parser: bool
|
||||
|
||||
|
||||
class HealthResponse(BaseModel):
|
||||
"""Response schema for ``GET /health``.
|
||||
|
||||
Declared as a Pydantic model (not ``dict``) so the generated
|
||||
OpenAPI schema carries the full field shape — ``capabilities`` and
|
||||
``disabled_features`` are typed. A bare ``-> dict`` return type
|
||||
degrades the OpenAPI response to ``additionalProperties: true``,
|
||||
which robs clients (and codegen) of any structure to lean on.
|
||||
"""
|
||||
|
||||
status: str
|
||||
version: str
|
||||
capabilities: HealthCapabilities
|
||||
disabled_features: list[str]
|
||||
|
||||
|
||||
@router.get("/health", response_model=HealthResponse)
|
||||
async def health() -> HealthResponse:
|
||||
"""Liveness probe with capabilities and disabled features."""
|
||||
# ``llm`` is hardcoded ``True`` — kept for symmetry with the caps
|
||||
# dict rather than probed live. Rationale: LLM is a Tier-1 hard
|
||||
# requirement enforced at startup by ``LLMLifespanProvider``
|
||||
# (lifespans/llm.py), which eagerly calls ``get_llm_client()`` and
|
||||
# raises ``LLMNotConfiguredError`` if credentials are missing —
|
||||
# FastAPI startup then fails, so ``/health`` is unreachable
|
||||
# without a working LLM. Any code path that reaches this handler
|
||||
# therefore has ``get_llm_client()`` returning a real client. If
|
||||
# the LLM capability is ever downgraded to soft (like embed /
|
||||
# rerank), swap this literal for a real probe.
|
||||
caps = HealthCapabilities(
|
||||
llm=True,
|
||||
embed=get_embedding_capability().available,
|
||||
rerank=get_rerank_capability().available,
|
||||
multimodal_llm=get_multimodal_llm_capability().available,
|
||||
parser=parser_available(),
|
||||
)
|
||||
return HealthResponse(
|
||||
status="ok",
|
||||
version=__version__,
|
||||
capabilities=caps,
|
||||
disabled_features=compute_disabled_features(caps.model_dump()),
|
||||
)
|
||||
|
||||
@@ -25,15 +25,18 @@ if TYPE_CHECKING:
|
||||
from everos.service.knowledge import KnowledgeExtractor
|
||||
|
||||
from everalgo.types import ParsedContent
|
||||
from fastapi import APIRouter, Path, Query, Request, Response, UploadFile
|
||||
from fastapi import APIRouter, Depends, Path, Query, Request, Response, UploadFile
|
||||
from fastapi.params import Form
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from everos.component.embedding import get_embedding_capability
|
||||
from everos.component.llm import get_llm_client
|
||||
from everos.component.rerank import get_rerank_capability
|
||||
from everos.component.utils.datetime import to_display_tz
|
||||
from everos.config import load_settings
|
||||
from everos.core.errors import (
|
||||
InvalidInputError,
|
||||
ProviderNotConfiguredError,
|
||||
UnsupportedModalityError,
|
||||
)
|
||||
from everos.core.persistence import MemoryRoot
|
||||
@@ -59,6 +62,61 @@ from everos.service import (
|
||||
# a shared module would be cleaner but is out of scope for this PR.
|
||||
from .memorize import PathSafeId, SuccessEnvelope
|
||||
|
||||
_KNOWLEDGE_FEATURE = "knowledge"
|
||||
|
||||
|
||||
def _require_knowledge_capabilities() -> None:
|
||||
"""Per-endpoint gate: knowledge writes/search require embed + rerank.
|
||||
|
||||
**Why the gate lives at the HTTP layer, not at cascade registry**:
|
||||
cascade handlers register **unconditionally** now (see
|
||||
``memory/cascade/registry.py:177-194`` — the earlier atomic-pair
|
||||
gate that lived there was removed because it broke the delete
|
||||
path: a Tier-3 → Tier-2/1 downgrade would strand existing knowledge
|
||||
documents with no handler to process their deletion). Handlers
|
||||
body-guard the embed/rerank branches internally instead, so the
|
||||
delete path stays reachable across tier changes.
|
||||
|
||||
Because the cascade layer no longer refuses the write, the HTTP
|
||||
layer is now the *only* place that enforces "knowledge features
|
||||
require Tier 3". Without this gate a Tier-1/2 upload would:
|
||||
|
||||
- accept the multipart request → md write succeeds
|
||||
- enqueue for cascade → cascade handler body-guards on
|
||||
``get_embedding_capability().available`` and no-ops the
|
||||
index write
|
||||
- client sees 201 but the doc is permanently keyword-only /
|
||||
unsearchable
|
||||
|
||||
Returning 422 up front is the honest answer.
|
||||
|
||||
**Attached per-endpoint (not router-wide)** so read / list /
|
||||
delete / metadata-patch routes stay reachable after a Tier-3 →
|
||||
Tier-2/1 downgrade: users can still inspect and clean up their
|
||||
existing docs (rename, recategorize, delete) even when the
|
||||
providers that would embed or rerank new content are no longer
|
||||
configured. Title / category patches only rewrite md frontmatter
|
||||
(and move the doc directory when category changes) — no embed or
|
||||
rerank code runs on that path.
|
||||
|
||||
Checks both capabilities (not just embedding) up front so a client
|
||||
missing only rerank gets a rerank-specific message rather than
|
||||
passing this gate and failing later inside search.
|
||||
"""
|
||||
if not get_embedding_capability().available:
|
||||
raise ProviderNotConfiguredError(
|
||||
provider="embedding",
|
||||
feature=_KNOWLEDGE_FEATURE,
|
||||
)
|
||||
if not get_rerank_capability().available:
|
||||
raise ProviderNotConfiguredError(
|
||||
provider="rerank",
|
||||
feature=_KNOWLEDGE_FEATURE,
|
||||
)
|
||||
|
||||
|
||||
# Router prefix is /knowledge; app.py mounts it under both /api/v1 and
|
||||
# /api/v2 (see create_app — v1 retained as a permanent alias).
|
||||
router = APIRouter(prefix="/knowledge", tags=["knowledge"])
|
||||
|
||||
|
||||
@@ -276,6 +334,61 @@ def _reject_oversized_upload(file: UploadFile) -> None:
|
||||
raise InvalidInputError(f"Uploaded file exceeds the {limit_mib:.1f} MiB limit.")
|
||||
|
||||
|
||||
_PLAIN_TEXT_EXTENSIONS: frozenset[str] = frozenset(
|
||||
{"md", "txt", "rst", "markdown", "text"}
|
||||
)
|
||||
# Explicit mime allowlist. ``text/*`` prefix matching would let
|
||||
# ``text/html`` (and any future ``text/xml`` etc.) bypass the parser and
|
||||
# feed raw markup — script tags, HTML comments, style blocks — straight
|
||||
# into the knowledge-extraction LLM. everalgo's HTML parser runs
|
||||
# ``clean_html_for_llm`` (strips ``<script>/<style>/<nav>/<iframe>`` +
|
||||
# comments) and caps output at 1 MiB — semantics we must NOT skip.
|
||||
_PLAIN_TEXT_MIMES: frozenset[str] = frozenset(
|
||||
{"text/plain", "text/markdown", "text/x-rst", "text/x-markdown"}
|
||||
)
|
||||
|
||||
|
||||
def _looks_like_utf8_text(file: UploadFile) -> bool:
|
||||
"""Decide whether to skip the parser and go straight to UTF-8 decode.
|
||||
|
||||
A file is treated as plain UTF-8 text when its ``content_type`` is on
|
||||
``_PLAIN_TEXT_MIMES`` (browsers set ``text/markdown`` / ``text/plain``
|
||||
for ``.md`` / ``.txt`` uploads), or when the mime is missing /
|
||||
``application/octet-stream`` (typical for ``curl -F file=@x.md``
|
||||
without an explicit ``type=`` hint) AND the filename extension is on
|
||||
the extension allowlist.
|
||||
|
||||
``text/html`` is deliberately EXCLUDED — HTML uploads must go through
|
||||
the parser so ``clean_html_for_llm`` can strip active markup and cap
|
||||
the payload; the ``text/*`` prefix match this file used to carry
|
||||
would have let raw HTML (including ``<script>`` bodies and
|
||||
``<!-- prompt injection -->`` comments) reach the LLM as-is.
|
||||
"""
|
||||
mime = (file.content_type or "").lower()
|
||||
if mime in _PLAIN_TEXT_MIMES:
|
||||
return True
|
||||
if mime and mime != "application/octet-stream":
|
||||
return False
|
||||
if file.filename and "." in file.filename:
|
||||
return file.filename.rsplit(".", 1)[-1].lower() in _PLAIN_TEXT_EXTENSIONS
|
||||
return False
|
||||
|
||||
|
||||
def _decode_as_utf8(raw_bytes: bytes) -> ParsedContent:
|
||||
"""Decode ``raw_bytes`` as UTF-8 or raise a caller-friendly error."""
|
||||
try:
|
||||
text = raw_bytes.decode("utf-8")
|
||||
except UnicodeDecodeError as exc:
|
||||
raise UnsupportedModalityError(
|
||||
"File is not UTF-8 text. "
|
||||
"Install everos[multimodal] for PDF/HTML/DOCX support."
|
||||
) from exc
|
||||
parsed = ParsedContent(text=text)
|
||||
if not parsed.text or not parsed.text.strip():
|
||||
raise InvalidInputError("Uploaded file has no valid content.")
|
||||
return parsed
|
||||
|
||||
|
||||
async def _parse_upload(
|
||||
file: UploadFile, *, raw_bytes: bytes | None = None
|
||||
) -> ParsedContent:
|
||||
@@ -294,6 +407,12 @@ async def _parse_upload(
|
||||
if raw_bytes is None:
|
||||
raw_bytes = await file.read()
|
||||
|
||||
# Short-circuit plain-text uploads even when everalgo.parser is
|
||||
# installed — the parser path depends on [multimodal] being
|
||||
# configured, which markdown/plaintext ingestion doesn't need.
|
||||
if _looks_like_utf8_text(file):
|
||||
return _decode_as_utf8(raw_bytes)
|
||||
|
||||
if parser_available():
|
||||
from everalgo.types import RawFile # Deferred: optional dep
|
||||
|
||||
@@ -313,17 +432,7 @@ async def _parse_upload(
|
||||
raise InvalidInputError("Uploaded file has no valid content.")
|
||||
return parsed
|
||||
|
||||
try:
|
||||
text = raw_bytes.decode("utf-8")
|
||||
except UnicodeDecodeError as exc:
|
||||
raise UnsupportedModalityError(
|
||||
"File is not UTF-8 text. "
|
||||
"Install everos[multimodal] for PDF/HTML/DOCX support."
|
||||
) from exc
|
||||
parsed = ParsedContent(text=text)
|
||||
if not parsed.text or not parsed.text.strip():
|
||||
raise InvalidInputError("Uploaded file has no valid content.")
|
||||
return parsed
|
||||
return _decode_as_utf8(raw_bytes)
|
||||
|
||||
|
||||
def _map_create_result(result: CreateDocumentResult) -> DocumentCreateResponse:
|
||||
@@ -432,7 +541,11 @@ def _map_search_result(result: SearchKnowledgeResult) -> KnowledgeSearchResponse
|
||||
# ── Routes ───────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@router.post("/documents", status_code=201)
|
||||
@router.post(
|
||||
"/documents",
|
||||
status_code=201,
|
||||
dependencies=[Depends(_require_knowledge_capabilities)],
|
||||
)
|
||||
# FastAPI requires flat Form/Query params — ≤5 positional rule exempted.
|
||||
async def create_document_route(
|
||||
request: Request,
|
||||
@@ -450,7 +563,7 @@ async def create_document_route(
|
||||
parsed = await _parse_upload(file, raw_bytes=file_content)
|
||||
source_name = file.filename
|
||||
|
||||
knowledge_dir = MemoryRoot.default().knowledge_dir(app_id, project_id)
|
||||
knowledge_dir = MemoryRoot.resolve().knowledge_dir(app_id, project_id)
|
||||
extractor = _build_extractor()
|
||||
|
||||
result = await create_document(
|
||||
@@ -466,7 +579,11 @@ async def create_document_route(
|
||||
return SuccessEnvelope(request_id=rid, data=_map_create_result(result))
|
||||
|
||||
|
||||
@router.put("/documents/{doc_id}", status_code=200)
|
||||
@router.put(
|
||||
"/documents/{doc_id}",
|
||||
status_code=200,
|
||||
dependencies=[Depends(_require_knowledge_capabilities)],
|
||||
)
|
||||
# FastAPI requires flat Form/Query params — ≤5 positional rule exempted.
|
||||
async def replace_document_route(
|
||||
request: Request,
|
||||
@@ -484,7 +601,7 @@ async def replace_document_route(
|
||||
file_content = await file.read()
|
||||
parsed = await _parse_upload(file, raw_bytes=file_content)
|
||||
|
||||
knowledge_dir = MemoryRoot.default().knowledge_dir(app_id, project_id)
|
||||
knowledge_dir = MemoryRoot.resolve().knowledge_dir(app_id, project_id)
|
||||
extractor = _build_extractor()
|
||||
|
||||
result = await replace_document(
|
||||
@@ -576,7 +693,10 @@ async def get_topic_route(
|
||||
return SuccessEnvelope(request_id=rid, data=_map_topic_detail(detail))
|
||||
|
||||
|
||||
@router.post("/search")
|
||||
@router.post(
|
||||
"/search",
|
||||
dependencies=[Depends(_require_knowledge_capabilities)],
|
||||
)
|
||||
async def search_knowledge_route(
|
||||
request: Request,
|
||||
req: KnowledgeSearchRequest,
|
||||
|
||||
@@ -0,0 +1,29 @@
|
||||
"""Interactive-CLI logging setup.
|
||||
|
||||
Interactive commands (``cascade`` / ``init`` / ``config`` / ``demo``)
|
||||
default to WARNING so lifecycle events — ``sqlite_engine_built``,
|
||||
``lancedb_connection_opened``, ``lancedb_table_opened`` and friends —
|
||||
do not leak into the user-facing flow (Phase banners, ``[y/N]``
|
||||
prompts, echo output). ``--verbose`` / ``-v`` opens INFO.
|
||||
|
||||
``server start`` is intentionally unaffected: it calls
|
||||
:func:`everos.core.observability.logging.configure_logging` directly
|
||||
so production observability still defaults to INFO. This helper is
|
||||
only for the interactive path.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from everos.core.observability.logging import configure_logging
|
||||
|
||||
|
||||
def configure_cli_logging(verbose: bool = False) -> None:
|
||||
"""Configure structlog + stdlib logging for an interactive CLI command.
|
||||
|
||||
Args:
|
||||
verbose: True to emit INFO-level lifecycle logs; False (default)
|
||||
keeps stdout limited to WARNING and above so the interactive
|
||||
flow stays clean.
|
||||
"""
|
||||
level = "INFO" if verbose else "WARNING"
|
||||
configure_logging(level=level)
|
||||
@@ -0,0 +1,486 @@
|
||||
"""CLI-side orchestration for ``everos cascade backfill``.
|
||||
|
||||
Presentation half of PR #361 review finding M11. The memory-layer phase
|
||||
runners (``memory/cascade/_backfill.py``) are pure — every user-facing
|
||||
string, colour, and confirmation prompt lives here, behind the
|
||||
:class:`~everos.memory.cascade._backfill.BackfillPresenter` Protocol.
|
||||
:class:`TyperPresenter` is the typer-backed implementation; tests can
|
||||
substitute any structurally-compatible stub.
|
||||
|
||||
:func:`run_backfill` is the top-level entry point the ``cascade``
|
||||
subcommand imports. It builds a :class:`TyperPresenter`, dispatches to
|
||||
the three phase runners in order, catches Ctrl-C /
|
||||
:class:`click.exceptions.Abort` / :class:`asyncio.CancelledError`, and
|
||||
maps every terminal state to a CLI exit code:
|
||||
|
||||
- ``0`` — clean success.
|
||||
- ``1`` — user declined a confirmation prompt.
|
||||
- ``2`` — unexpected error mid-phase, or preflight found the required
|
||||
provider not configured.
|
||||
- ``3`` — Phase 2/3 short-circuited on the OME jobstore lock (another
|
||||
``everos`` process is running).
|
||||
- ``4`` — every phase ran but at least one row failed embedding even
|
||||
after per-row fallback; automation can requeue.
|
||||
- ``130`` — Ctrl-C (SIGINT convention: 128 + 2), including Ctrl-C at
|
||||
the ``[y/N]`` confirmation prompt (typer/click surfaces that as
|
||||
:class:`click.exceptions.Abort`, a :class:`RuntimeError` subclass).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
|
||||
import anyio.to_thread
|
||||
import click
|
||||
import typer
|
||||
|
||||
from everos.core.errors import ProviderNotConfiguredError
|
||||
from everos.core.observability.logging import get_logger
|
||||
|
||||
# Module reference (not name-level imports of the phase runners): tests
|
||||
# monkeypatch the runners on this module to inject Ctrl-C / boundary
|
||||
# behaviour, and re-binding names on the source module only affects the
|
||||
# entrypoints layer if we resolve them through the module attribute at
|
||||
# call time.
|
||||
from everos.memory.cascade import _backfill as _phase_runners
|
||||
from everos.memory.cascade._backfill import (
|
||||
BackfillPhase,
|
||||
_BackfillSummary,
|
||||
_count_failed_rows,
|
||||
_format_tokens,
|
||||
)
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
PHASES: tuple[BackfillPhase, ...] = (
|
||||
BackfillPhase(
|
||||
number=1,
|
||||
slug="vectors",
|
||||
title="Phase 1 — make older memories searchable by meaning",
|
||||
detail=(
|
||||
"Memories created before you configured embedding can only be "
|
||||
"found by keyword right now. This step lets them be found by "
|
||||
"meaning too."
|
||||
),
|
||||
),
|
||||
BackfillPhase(
|
||||
number=2,
|
||||
slug="clusters",
|
||||
title="Phase 2 — group related memories into topics",
|
||||
detail=(
|
||||
"Related memories get clustered together so reflection and "
|
||||
"advanced retrieval can operate on them."
|
||||
),
|
||||
),
|
||||
BackfillPhase(
|
||||
number=3,
|
||||
slug="skills",
|
||||
title="Phase 3 — build the agent skill library",
|
||||
detail=(
|
||||
"Distill reusable skills from past agent conversations. Uses "
|
||||
"your LLM — the most expensive step."
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
_EXIT_LABELS: dict[int, str] = {
|
||||
0: "SUCCESS",
|
||||
1: "ABORTED",
|
||||
2: "FAILED",
|
||||
3: "SERVER_RUNNING",
|
||||
4: "COMPLETED_WITH_FAILURES",
|
||||
130: "INTERRUPTED",
|
||||
}
|
||||
"""Maps :func:`run_backfill`'s exit codes to the summary block's ``Exit:``
|
||||
label. ``3`` marks a Phase 2/3 short-circuit because another process
|
||||
holds the OME jobstore lock (typically ``everos server start``). ``4``
|
||||
marks a partial-success run: every phase ran to completion, but at
|
||||
least one row failed embedding even after the per-row fallback —
|
||||
operators consult the ``cascade_backfill_row_embed_failed`` /
|
||||
``..._row_subject_embed_failed`` log events for the failed row ids.
|
||||
``130`` is the SIGINT convention (128 + signal number 2)."""
|
||||
|
||||
|
||||
def _confirm(detail: str, *, auto_yes: bool) -> bool:
|
||||
"""Ask the user to confirm one phase; ``--yes`` skips the prompt.
|
||||
|
||||
Kept as a module-level helper (rather than a bound method of
|
||||
:class:`TyperPresenter`) so tests can monkeypatch it on this module
|
||||
the same way the pre-M11 code was patched on ``_backfill``. The
|
||||
presenter's :meth:`~TyperPresenter.confirm` delegates here.
|
||||
"""
|
||||
if auto_yes:
|
||||
return True
|
||||
return typer.confirm(f"Proceed with: {detail}", default=False)
|
||||
|
||||
|
||||
def _print_phase_header(phase: BackfillPhase) -> None:
|
||||
typer.secho(phase.title, bold=True)
|
||||
typer.echo(f" {phase.detail}")
|
||||
typer.echo("")
|
||||
|
||||
|
||||
def _print_aborted() -> None:
|
||||
typer.secho("Aborted by user.", fg=typer.colors.RED)
|
||||
|
||||
|
||||
def _print_interrupted() -> None:
|
||||
"""Print the Ctrl-C resume hint (see :func:`run_backfill`)."""
|
||||
typer.secho(
|
||||
"Interrupted — partial progress was written. Resume by running:",
|
||||
fg=typer.colors.YELLOW,
|
||||
)
|
||||
typer.echo(" everos cascade backfill --phase <phase-name> --yes")
|
||||
typer.echo("where <phase-name> is one of: vectors, clusters, skills, all.")
|
||||
|
||||
|
||||
def _print_capability_missing_hint(message: str) -> None:
|
||||
"""Emit the friendly toml-hint message shown when a phase's required
|
||||
provider is not configured.
|
||||
|
||||
The formatted :class:`ProviderNotConfiguredError` string is built by
|
||||
the memory-layer phase runner and passed through the presenter — so
|
||||
the CLI and API stay in lock-step on the remediation copy without
|
||||
duplicating it here.
|
||||
"""
|
||||
typer.secho(f" {message}", fg=typer.colors.RED)
|
||||
|
||||
|
||||
def _print_server_running_hint() -> None:
|
||||
"""Emit the friendly error shown when preflight or ``engine.start()``
|
||||
detects another process holding the OME jobstore lock.
|
||||
|
||||
Copy points the user at ``everos server start`` (the usual holder)
|
||||
and Phase 1 (``--phase=vectors``), which never touches OME and so
|
||||
stays fully concurrent with a running server — never at
|
||||
``EVEROS_*`` env vars.
|
||||
"""
|
||||
typer.secho(
|
||||
" Backfill Phase 2/3 needs exclusive access to the OME jobstore,",
|
||||
fg=typer.colors.RED,
|
||||
)
|
||||
typer.secho(
|
||||
" but another everos process is holding it (typically your running",
|
||||
fg=typer.colors.RED,
|
||||
)
|
||||
typer.secho(
|
||||
" `everos server start`).",
|
||||
fg=typer.colors.RED,
|
||||
)
|
||||
typer.echo("")
|
||||
typer.echo(" Stop your everos server first, then re-run backfill.")
|
||||
typer.echo(" (Phase 1 --phase=vectors already runs concurrently with the server;")
|
||||
typer.echo(" only Phase 2/3 need offline mode. See spec §10.)")
|
||||
|
||||
|
||||
def _print_vectors_estimate(total_rows: int, total_tokens: int) -> None:
|
||||
typer.echo(f" memories to process: {total_rows:,}")
|
||||
typer.echo(f" input tokens: {_format_tokens(total_tokens)}\n")
|
||||
typer.echo(
|
||||
" Uses your embedding provider — cost depends on its per-token pricing.\n"
|
||||
)
|
||||
|
||||
|
||||
def _print_clusters_estimate(episode_count: int, case_count: int) -> None:
|
||||
typer.echo(f" episodes to cluster: {episode_count:,}")
|
||||
typer.echo(f" agent cases to cluster: {case_count:,}\n")
|
||||
typer.echo(
|
||||
" Uses your embedding provider (and LLM for agent-case merges) — "
|
||||
"cost depends on provider pricing.\n"
|
||||
)
|
||||
|
||||
|
||||
def _print_skills_estimate(cases: int, clusters: int) -> None:
|
||||
typer.echo(f" clusters to extract skills from: {clusters:,}")
|
||||
typer.echo(f" agent cases to replay: {cases:,}\n")
|
||||
typer.echo(" Uses your LLM — cost depends on its per-token pricing.\n")
|
||||
|
||||
|
||||
def _print_summary(summary: _BackfillSummary, *, exit_code: int) -> None:
|
||||
"""Print the consolidated post-run summary block.
|
||||
|
||||
Field labels are chosen from what each phase's result dataclass
|
||||
actually tracks — e.g. Phase 3 reports "agent cases processed"
|
||||
(``events_emitted``) rather than "clusters processed", since the
|
||||
distinct cluster count isn't retained on
|
||||
:class:`~everos.memory.cascade._backfill._SkillPhaseResult`.
|
||||
|
||||
On ``exit_code == 4`` (COMPLETED_WITH_FAILURES) also emits a hint
|
||||
pointing operators at the per-row failure log events so they can
|
||||
recover the failed row ids.
|
||||
"""
|
||||
typer.echo("")
|
||||
typer.secho("Backfill summary", bold=True)
|
||||
typer.echo("-" * 44)
|
||||
if summary.vectors is not None:
|
||||
r = summary.vectors
|
||||
typer.echo(
|
||||
f" Phase 1 (vectors) — {r.rows_processed:,} rows / "
|
||||
f"{r.rows_failed:,} failed / {_format_tokens(r.tokens_embedded)}"
|
||||
)
|
||||
if summary.clusters is not None:
|
||||
c = summary.clusters
|
||||
created = c.clusters_after - c.clusters_before
|
||||
typer.echo(
|
||||
f" Phase 2 (clusters) — {c.events_emitted:,} events emitted / "
|
||||
f"{created:,} clusters created"
|
||||
)
|
||||
if summary.skills is not None:
|
||||
s = summary.skills
|
||||
extracted = s.skills_after - s.skills_before
|
||||
typer.echo(
|
||||
f" Phase 3 (skills) — {s.events_emitted:,} agent cases "
|
||||
f"processed / {extracted:,} skills extracted"
|
||||
)
|
||||
typer.echo("-" * 44)
|
||||
label = _EXIT_LABELS.get(exit_code, "UNKNOWN")
|
||||
typer.echo(f" Exit: {label} ({exit_code})")
|
||||
if exit_code == 4:
|
||||
total_failed = _count_failed_rows(summary)
|
||||
typer.secho(
|
||||
f" {total_failed:,} rows failed embedding — see log events "
|
||||
"`cascade_backfill_row_embed_failed` for row_ids and text prefixes.",
|
||||
fg=typer.colors.YELLOW,
|
||||
)
|
||||
|
||||
|
||||
class TyperPresenter:
|
||||
"""typer-backed :class:`BackfillPresenter` — the CLI's user-I/O seam.
|
||||
|
||||
Every method delegates to a module-level ``_print_*`` helper or to
|
||||
:func:`_confirm`. Kept as a small class (rather than a module of
|
||||
free functions) so alternative front-ends (e.g. a JSON-emitting
|
||||
CI-mode presenter) can slot in without touching the memory layer.
|
||||
"""
|
||||
|
||||
def nothing_to_backfill(self, message: str, *, scan_failed: bool = False) -> None:
|
||||
colour = typer.colors.YELLOW if scan_failed else typer.colors.GREEN
|
||||
typer.secho(message, fg=colour)
|
||||
|
||||
def capability_missing(self, *, provider: str, feature: str, message: str) -> None:
|
||||
_print_capability_missing_hint(message)
|
||||
|
||||
def server_running(self) -> None:
|
||||
_print_server_running_hint()
|
||||
|
||||
def estimate_vectors(self, rows: int, tokens: int) -> None:
|
||||
_print_vectors_estimate(rows, tokens)
|
||||
|
||||
def estimate_clusters(self, episodes: int, cases: int) -> None:
|
||||
_print_clusters_estimate(episodes, cases)
|
||||
|
||||
def estimate_skills(self, cases: int, clusters: int) -> None:
|
||||
_print_skills_estimate(cases, clusters)
|
||||
|
||||
async def confirm(self, prompt: str, *, auto_yes: bool) -> bool:
|
||||
# ``typer.confirm`` is blocking (reads from stdin); offload so a
|
||||
# slow / interactive user doesn't hold the event loop. Any
|
||||
# ``click.exceptions.Abort`` raised on Ctrl-C at the prompt
|
||||
# propagates to :func:`run_backfill` which catches it and maps
|
||||
# to exit 130.
|
||||
return await anyio.to_thread.run_sync(
|
||||
lambda: _confirm(prompt, auto_yes=auto_yes)
|
||||
)
|
||||
|
||||
def emit_progress(self, done: int, total: int) -> None:
|
||||
typer.echo(f" emitted {done:,} / {total:,} event(s)")
|
||||
|
||||
def row_progress(self, table_name: str, done: int, total: int) -> None:
|
||||
typer.echo(f" {table_name}: processed {done:,} / {total:,} rows")
|
||||
|
||||
def phase_1_complete(self, rows_processed: int, rows_failed: int) -> None:
|
||||
typer.secho(
|
||||
f"phase 1 complete — {rows_processed:,} memories now "
|
||||
"searchable by meaning"
|
||||
+ (f" ({rows_failed:,} failed)" if rows_failed else ""),
|
||||
fg=typer.colors.GREEN,
|
||||
)
|
||||
|
||||
def phase_2_complete(self, before: int, after: int, emitted: int) -> None:
|
||||
typer.secho(
|
||||
f"phase 2 complete — clusters {before:,} -> {after:,} "
|
||||
f"({emitted:,} events processed)",
|
||||
fg=typer.colors.GREEN,
|
||||
)
|
||||
|
||||
def phase_3_complete(
|
||||
self, skills_before: int, skills_after: int, emitted: int
|
||||
) -> None:
|
||||
typer.secho(
|
||||
f"phase 3 complete — skills {skills_before:,} -> {skills_after:,} "
|
||||
f"({emitted:,} events processed)",
|
||||
fg=typer.colors.GREEN,
|
||||
)
|
||||
|
||||
|
||||
async def run_backfill(*, phase: str, auto_yes: bool) -> int:
|
||||
"""Run backfill phases interactively; return an exit code.
|
||||
|
||||
``phase`` is ``"all"`` or one of :data:`PHASES`'s ``slug`` values.
|
||||
Exit codes: see :data:`_EXIT_LABELS`.
|
||||
|
||||
Every exit path — success, abort, error, or interrupt — ends with a
|
||||
consolidated summary block (:func:`_print_summary`) covering every
|
||||
phase that actually ran. The interrupt is only ever caught here, at
|
||||
the outer loop: a phase body mid-batch must let it propagate rather
|
||||
than swallow it, so a Ctrl-C during Phase 1 doesn't get silently
|
||||
absorbed as "phase 1 failed, try phase 2 anyway".
|
||||
|
||||
Both ``KeyboardInterrupt`` and ``asyncio.CancelledError`` are caught:
|
||||
on Python 3.11+, ``asyncio.Runner`` (which backs ``asyncio.run``)
|
||||
translates a real SIGINT into ``main_task.cancel()``, which raises
|
||||
``CancelledError`` — not ``KeyboardInterrupt`` — at the currently
|
||||
suspended ``await``. Handling it here lets the task complete
|
||||
normally with exit code 130 instead of leaking the cancellation out
|
||||
to ``asyncio.run`` (which would re-synthesize a bare
|
||||
``KeyboardInterrupt`` that typer/click turns into ``Abort``, exit 1,
|
||||
with no resume hint printed). Catching ``CancelledError``
|
||||
unconditionally is safe here: nothing in this call tree cancels the
|
||||
task for reasons other than an interrupt.
|
||||
"""
|
||||
presenter = TyperPresenter()
|
||||
selected = [p for p in PHASES if phase == "all" or p.slug == phase]
|
||||
summary = _BackfillSummary()
|
||||
current_phase: BackfillPhase | None = None
|
||||
try:
|
||||
for p in selected:
|
||||
current_phase = p
|
||||
_print_phase_header(p)
|
||||
if p.slug == "vectors":
|
||||
result = await _phase_runners._run_phase_vectors(
|
||||
auto_yes=auto_yes, presenter=presenter
|
||||
)
|
||||
summary.vectors = result
|
||||
if result.blocked_by_capability:
|
||||
logger.warning(
|
||||
"cascade_backfill_blocked_by_capability",
|
||||
phase=p.slug,
|
||||
provider=result.blocked_by_capability,
|
||||
)
|
||||
_print_summary(summary, exit_code=2)
|
||||
return 2
|
||||
if result.blocked_by_server:
|
||||
# Fail fast BEFORE any embed API call — Phase 2 / 3
|
||||
# already do this preflight; without it here,
|
||||
# ``--phase all`` against a running server burns
|
||||
# through Phase 1's real token budget and only
|
||||
# halts at Phase 2 (see PR #361 review J10).
|
||||
logger.warning("cascade_backfill_blocked_by_server", phase=p.slug)
|
||||
_print_summary(summary, exit_code=3)
|
||||
return 3
|
||||
if result.aborted:
|
||||
# Policy (PR #361 round-3 review #11, accepted):
|
||||
# typing ``n`` at Phase 1's confirmation aborts the
|
||||
# ENTIRE ``--phase all`` run — not just Phase 1.
|
||||
# Rationale: ``--phase all`` is opt-in-to-run-
|
||||
# everything. Explicitly declining Phase 1's cost
|
||||
# estimate is a decisive "no"; interpreting it as
|
||||
# "skip Phase 1 but still run Phase 2/3" would spend
|
||||
# LLM/embed budget on phases the user did not see
|
||||
# estimates for. Users who want only Phase 2 or 3
|
||||
# invoke ``--phase clusters`` or ``--phase skills``
|
||||
# explicitly (and see that phase's own estimate).
|
||||
logger.info("cascade_backfill_aborted", phase=p.slug)
|
||||
_print_aborted()
|
||||
_print_summary(summary, exit_code=1)
|
||||
return 1
|
||||
continue
|
||||
if p.slug == "clusters":
|
||||
cluster_result = await _phase_runners._run_phase_clusters(
|
||||
auto_yes=auto_yes, presenter=presenter
|
||||
)
|
||||
summary.clusters = cluster_result
|
||||
if cluster_result.blocked_by_capability:
|
||||
logger.warning(
|
||||
"cascade_backfill_blocked_by_capability",
|
||||
phase=p.slug,
|
||||
provider=cluster_result.blocked_by_capability,
|
||||
)
|
||||
_print_summary(summary, exit_code=2)
|
||||
return 2
|
||||
if cluster_result.blocked_by_server:
|
||||
logger.warning("cascade_backfill_blocked_by_server", phase=p.slug)
|
||||
_print_summary(summary, exit_code=3)
|
||||
return 3
|
||||
if cluster_result.aborted:
|
||||
logger.info("cascade_backfill_aborted", phase=p.slug)
|
||||
_print_aborted()
|
||||
_print_summary(summary, exit_code=1)
|
||||
return 1
|
||||
continue
|
||||
if p.slug == "skills":
|
||||
skill_result = await _phase_runners._run_phase_skills(
|
||||
auto_yes=auto_yes, presenter=presenter
|
||||
)
|
||||
summary.skills = skill_result
|
||||
if skill_result.blocked_by_capability:
|
||||
logger.warning(
|
||||
"cascade_backfill_blocked_by_capability",
|
||||
phase=p.slug,
|
||||
provider=skill_result.blocked_by_capability,
|
||||
)
|
||||
_print_summary(summary, exit_code=2)
|
||||
return 2
|
||||
if skill_result.blocked_by_server:
|
||||
logger.warning("cascade_backfill_blocked_by_server", phase=p.slug)
|
||||
_print_summary(summary, exit_code=3)
|
||||
return 3
|
||||
if skill_result.aborted:
|
||||
logger.info("cascade_backfill_aborted", phase=p.slug)
|
||||
_print_aborted()
|
||||
_print_summary(summary, exit_code=1)
|
||||
return 1
|
||||
continue
|
||||
# ``typer.Abort`` fires when ``typer.confirm`` catches KeyboardInterrupt
|
||||
# or EOF at the y/N prompt and re-raises it as an ``Abort`` (a
|
||||
# ``RuntimeError`` subclass, NOT a ``KeyboardInterrupt``). Typer 0.15+
|
||||
# vendored click under ``typer._click`` — ``typer.Abort`` and the
|
||||
# standalone ``click.exceptions.Abort`` are DISTINCT classes. Catching
|
||||
# only ``click.exceptions.Abort`` would let the real typer-raised
|
||||
# abort fall through to the generic ``except Exception`` branch (exit
|
||||
# 2, rich traceback). Both are listed so the Ctrl-C-at-prompt path
|
||||
# (exit 130 with resume hint) fires regardless of which one arrives.
|
||||
except (
|
||||
KeyboardInterrupt,
|
||||
asyncio.CancelledError,
|
||||
typer.Abort,
|
||||
click.exceptions.Abort,
|
||||
):
|
||||
logger.warning(
|
||||
"cascade_backfill_interrupted",
|
||||
phase=current_phase.slug if current_phase else phase,
|
||||
)
|
||||
_print_interrupted()
|
||||
_print_summary(summary, exit_code=130)
|
||||
return 130
|
||||
except ProviderNotConfiguredError as exc:
|
||||
# Defensive backstop: phase runners preflight capability and
|
||||
# return ``blocked_by_capability`` on their own. Reaching this
|
||||
# branch means a code path bypassed the preflight — surface the
|
||||
# remediation message instead of letting the broad ``except
|
||||
# Exception`` below swallow it into "see logs for details".
|
||||
logger.error(
|
||||
"cascade_backfill_provider_error_escaped_preflight",
|
||||
phase=current_phase.slug if current_phase else phase,
|
||||
provider=exc.provider,
|
||||
)
|
||||
typer.secho(f" {exc}", fg=typer.colors.RED)
|
||||
_print_summary(summary, exit_code=2)
|
||||
return 2
|
||||
except Exception:
|
||||
logger.exception("cascade_backfill_phase_failed", phase=phase)
|
||||
typer.secho("Backfill failed — see logs for details.", fg=typer.colors.RED)
|
||||
_print_summary(summary, exit_code=2)
|
||||
return 2
|
||||
total_failed = _count_failed_rows(summary)
|
||||
if total_failed > 0:
|
||||
logger.warning(
|
||||
"cascade_backfill_completed_with_failures",
|
||||
total_failed=total_failed,
|
||||
)
|
||||
_print_summary(summary, exit_code=4)
|
||||
return 4
|
||||
_print_summary(summary, exit_code=0)
|
||||
return 0
|
||||
@@ -11,6 +11,10 @@ without standing up the FastAPI app:
|
||||
- ``cascade fix`` — list every ``failed`` row. With ``--apply``, also
|
||||
reset ``retryable=TRUE`` rows back to ``pending`` and drain the
|
||||
worker once so the retry actually runs before the command returns.
|
||||
- ``cascade backfill`` — one-shot Tier 1 → Tier 2/3 migration: re-embed
|
||||
vectors, build clusters, extract skills. See
|
||||
:func:`everos.entrypoints.cli.commands._backfill_cmd.run_backfill`
|
||||
for the phase orchestration.
|
||||
|
||||
CLI is in-process (12 doc §7.1 + 16 doc §9.2): it constructs the same
|
||||
:class:`CascadeOrchestrator` as the daemon but only calls
|
||||
@@ -21,6 +25,7 @@ scanner background task is started.
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import enum
|
||||
import os
|
||||
from contextlib import asynccontextmanager
|
||||
from pathlib import Path
|
||||
@@ -29,11 +34,13 @@ from typing import Annotated
|
||||
import typer
|
||||
from sqlmodel import SQLModel
|
||||
|
||||
from everos.component.embedding import build_embedding_provider
|
||||
from everos.component.embedding import get_embedding_capability
|
||||
from everos.component.tokenizer import build_tokenizer
|
||||
from everos.component.utils.datetime import to_display_tz
|
||||
from everos.config import load_settings
|
||||
from everos.core.observability.logging import get_logger
|
||||
from everos.core.persistence import MemoryRoot
|
||||
from everos.entrypoints.cli._log_setup import configure_cli_logging
|
||||
from everos.entrypoints.cli.commands._backfill_cmd import run_backfill
|
||||
from everos.infra.persistence.lancedb import (
|
||||
dispose_connection,
|
||||
ensure_business_indexes,
|
||||
@@ -47,6 +54,8 @@ from everos.infra.persistence.sqlite import (
|
||||
)
|
||||
from everos.memory.cascade import CascadeOrchestrator, match_kind
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
app = typer.Typer(
|
||||
name="cascade",
|
||||
help="Inspect and operate the md → LanceDB sync queue",
|
||||
@@ -61,12 +70,57 @@ def _cascade_callback(
|
||||
"--root",
|
||||
help="Memory root directory (env: EVEROS_ROOT, default: ~/.everos)",
|
||||
),
|
||||
verbose: bool = typer.Option(
|
||||
False,
|
||||
"--verbose",
|
||||
"-v",
|
||||
help="Emit INFO-level lifecycle logs (default: WARNING only).",
|
||||
),
|
||||
) -> None:
|
||||
"""Set memory root before any cascade subcommand runs."""
|
||||
"""Set memory root and log level before any cascade subcommand runs."""
|
||||
configure_cli_logging(verbose=verbose)
|
||||
_apply_root_env(root)
|
||||
|
||||
|
||||
def _apply_root_env(root: str | None) -> None:
|
||||
"""Set ``EVEROS_ROOT`` env var if a ``--root`` path was passed.
|
||||
|
||||
Called from both the group callback (subcommand-independent path)
|
||||
and from each subcommand (subcommand-level ``--root``). Later calls
|
||||
override earlier ones — a subcommand's ``--root`` wins over the
|
||||
group callback's when both are supplied.
|
||||
"""
|
||||
if root:
|
||||
os.environ["EVEROS_ROOT"] = root
|
||||
|
||||
|
||||
def _apply_verbose_logging(verbose: bool | None) -> None:
|
||||
"""Re-apply the CLI log level from a subcommand-level ``--verbose``.
|
||||
|
||||
``None`` means the subcommand did not receive its own ``--verbose``
|
||||
flag — leave whatever level the group callback set (default
|
||||
WARNING, or INFO if the user passed ``cascade --verbose``). A
|
||||
``True``/``False`` value came from the subcommand and wins over the
|
||||
group callback (later configuration overrides earlier).
|
||||
|
||||
Mirrors :func:`_apply_root_env`'s "either position works" pattern
|
||||
for symmetry — users can write ``everos cascade --verbose status``
|
||||
or ``everos cascade status --verbose`` and get the same behaviour.
|
||||
"""
|
||||
if verbose is not None:
|
||||
configure_cli_logging(verbose=verbose)
|
||||
|
||||
|
||||
_ROOT_OPTION_HELP = (
|
||||
"Memory root directory (alias for `cascade --root`; either position works)."
|
||||
)
|
||||
|
||||
_VERBOSE_OPTION_HELP = (
|
||||
"Emit INFO-level lifecycle logs (alias for `cascade --verbose`; "
|
||||
"either position works)."
|
||||
)
|
||||
|
||||
|
||||
# ── shared runtime context ───────────────────────────────────────────────
|
||||
|
||||
|
||||
@@ -92,14 +146,21 @@ async def _runtime(): # type: ignore[no-untyped-def]
|
||||
|
||||
|
||||
def _build_orchestrator() -> CascadeOrchestrator:
|
||||
settings = load_settings()
|
||||
memory_root = MemoryRoot.default()
|
||||
memory_root = MemoryRoot.resolve()
|
||||
memory_root.ensure()
|
||||
embedder = build_embedding_provider(settings.embedding)
|
||||
tokenizer = build_tokenizer()
|
||||
|
||||
capability = get_embedding_capability()
|
||||
if capability.available:
|
||||
logger.info("cli_cascade_embed_available")
|
||||
else:
|
||||
logger.info(
|
||||
"cli_cascade_embed_unavailable",
|
||||
reason="embedding not configured; keyword-only mode",
|
||||
)
|
||||
|
||||
return CascadeOrchestrator(
|
||||
memory_root=memory_root,
|
||||
embedder=embedder,
|
||||
tokenizer=tokenizer,
|
||||
)
|
||||
|
||||
@@ -116,8 +177,18 @@ def sync(
|
||||
"If omitted, only the existing queue is drained.",
|
||||
),
|
||||
] = None,
|
||||
root: Annotated[
|
||||
str | None,
|
||||
typer.Option("--root", help=_ROOT_OPTION_HELP),
|
||||
] = None,
|
||||
verbose: Annotated[
|
||||
bool | None,
|
||||
typer.Option("--verbose", "-v", help=_VERBOSE_OPTION_HELP),
|
||||
] = None,
|
||||
) -> None:
|
||||
"""Drain the cascade queue (and optionally re-enqueue a path first)."""
|
||||
_apply_root_env(root)
|
||||
_apply_verbose_logging(verbose)
|
||||
|
||||
async def _run() -> None:
|
||||
async with _runtime():
|
||||
@@ -144,8 +215,19 @@ def sync(
|
||||
|
||||
|
||||
@app.command("status")
|
||||
def status() -> None:
|
||||
def status(
|
||||
root: Annotated[
|
||||
str | None,
|
||||
typer.Option("--root", help=_ROOT_OPTION_HELP),
|
||||
] = None,
|
||||
verbose: Annotated[
|
||||
bool | None,
|
||||
typer.Option("--verbose", "-v", help=_VERBOSE_OPTION_HELP),
|
||||
] = None,
|
||||
) -> None:
|
||||
"""Print the queue / LSN summary."""
|
||||
_apply_root_env(root)
|
||||
_apply_verbose_logging(verbose)
|
||||
|
||||
async def _run() -> None:
|
||||
async with _runtime():
|
||||
@@ -190,8 +272,18 @@ def fix(
|
||||
help="Re-enqueue every `retryable=TRUE` row and drain the worker.",
|
||||
),
|
||||
] = False,
|
||||
root: Annotated[
|
||||
str | None,
|
||||
typer.Option("--root", help=_ROOT_OPTION_HELP),
|
||||
] = None,
|
||||
verbose: Annotated[
|
||||
bool | None,
|
||||
typer.Option("--verbose", "-v", help=_VERBOSE_OPTION_HELP),
|
||||
] = None,
|
||||
) -> None:
|
||||
"""List failed rows (default) or re-enqueue retryable ones (``--apply``)."""
|
||||
_apply_root_env(root)
|
||||
_apply_verbose_logging(verbose)
|
||||
|
||||
async def _run() -> None:
|
||||
async with _runtime():
|
||||
@@ -234,6 +326,91 @@ def fix(
|
||||
asyncio.run(_run())
|
||||
|
||||
|
||||
# ── backfill ─────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
class _BackfillPhaseOption(enum.StrEnum):
|
||||
"""CLI-facing ``--phase`` choices — mirrors ``BackfillPhase.slug``."""
|
||||
|
||||
VECTORS = "vectors"
|
||||
CLUSTERS = "clusters"
|
||||
SKILLS = "skills"
|
||||
ALL = "all"
|
||||
|
||||
|
||||
@app.command("backfill")
|
||||
def backfill(
|
||||
phase: Annotated[
|
||||
_BackfillPhaseOption,
|
||||
typer.Option(
|
||||
"--phase",
|
||||
help=(
|
||||
"Which phase to run — vectors (re-embed rows so they become "
|
||||
"searchable by meaning, not just keyword), clusters (group "
|
||||
"re-embedded episodes/agent cases into topics), skills "
|
||||
"(extract reusable agent skills from clustered cases), or "
|
||||
"all (run every phase in order). Default: all."
|
||||
),
|
||||
),
|
||||
] = _BackfillPhaseOption.ALL,
|
||||
yes: Annotated[
|
||||
bool,
|
||||
typer.Option(
|
||||
"--yes",
|
||||
"-y",
|
||||
help=(
|
||||
"Auto-confirm every phase's prompt instead of pausing for a "
|
||||
"y/n answer. Without this flag, each phase shows its "
|
||||
"token-count estimate and waits for confirmation first."
|
||||
),
|
||||
),
|
||||
] = False,
|
||||
root: Annotated[
|
||||
str | None,
|
||||
typer.Option("--root", help=_ROOT_OPTION_HELP),
|
||||
] = None,
|
||||
verbose: Annotated[
|
||||
bool | None,
|
||||
typer.Option("--verbose", "-v", help=_VERBOSE_OPTION_HELP),
|
||||
] = None,
|
||||
) -> None:
|
||||
"""Backfill embeddings, clusters, and agent skills after upgrading from
|
||||
Tier 1 (LLM-only) to Tier 2/3 (embedding configured).
|
||||
|
||||
Runs up to three phases in order (see ``--phase``), each showing a
|
||||
token-count estimate and confirming before doing any work. The
|
||||
estimate is an input-token count only (K/M shorthand) — never a
|
||||
price, since per-token pricing is provider-specific and can change.
|
||||
|
||||
Ctrl-C is safe: interrupting mid-phase prints a resume hint and exits
|
||||
130. Every row and phase already written stays written, so
|
||||
re-running the same ``--phase`` (or ``--phase all``) afterwards only
|
||||
picks up what's left — nothing is redone from scratch.
|
||||
"""
|
||||
_apply_root_env(root)
|
||||
_apply_verbose_logging(verbose)
|
||||
|
||||
async def _run() -> int:
|
||||
async with _runtime():
|
||||
return await run_backfill(phase=phase.value, auto_yes=yes)
|
||||
|
||||
try:
|
||||
code = asyncio.run(_run())
|
||||
except KeyboardInterrupt:
|
||||
# Safety net: run_backfill already catches CancelledError and
|
||||
# returns 130 cleanly in the common case, but asyncio.Runner can
|
||||
# still re-raise a bare KeyboardInterrupt past asyncio.run() on a
|
||||
# second Ctrl-C. Without this, typer/click would convert it to
|
||||
# Abort and exit 1 with no resume hint.
|
||||
typer.secho(
|
||||
"Interrupted — partial progress was written. Resume by running:",
|
||||
fg=typer.colors.YELLOW,
|
||||
)
|
||||
typer.echo(" everos cascade backfill --phase <phase-name> --yes")
|
||||
raise typer.Exit(code=130) from None
|
||||
raise typer.Exit(code=code)
|
||||
|
||||
|
||||
# ── helpers ──────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@@ -244,7 +421,7 @@ def _resolve_relative(p: Path) -> str:
|
||||
must match that convention before calling :meth:`force_enqueue`.
|
||||
Outside-the-root inputs surface as an error in the caller.
|
||||
"""
|
||||
memory_root = MemoryRoot.default()
|
||||
memory_root = MemoryRoot.resolve()
|
||||
absolute = p.expanduser().resolve()
|
||||
try:
|
||||
rel = absolute.relative_to(memory_root.root)
|
||||
|
||||
@@ -7,6 +7,7 @@ import os
|
||||
import typer
|
||||
|
||||
from everos.config.settings import resolve_root
|
||||
from everos.entrypoints.cli._log_setup import configure_cli_logging
|
||||
|
||||
app = typer.Typer(
|
||||
name="config",
|
||||
@@ -14,6 +15,20 @@ app = typer.Typer(
|
||||
no_args_is_help=True,
|
||||
)
|
||||
|
||||
|
||||
@app.callback()
|
||||
def _config_callback(
|
||||
verbose: bool = typer.Option(
|
||||
False,
|
||||
"--verbose",
|
||||
"-v",
|
||||
help="Emit INFO-level lifecycle logs (default: WARNING only).",
|
||||
),
|
||||
) -> None:
|
||||
"""Set log level before any config subcommand runs."""
|
||||
configure_cli_logging(verbose=verbose)
|
||||
|
||||
|
||||
_SECRET_FIELDS = {"api_key"}
|
||||
|
||||
_SECTION_NAMES = (
|
||||
|
||||
@@ -15,6 +15,7 @@ from rich.console import Console
|
||||
from rich.panel import Panel
|
||||
|
||||
from everos.component.utils.datetime import get_utc_now
|
||||
from everos.entrypoints.cli._log_setup import configure_cli_logging
|
||||
from everos.entrypoints.tui.demo.data import (
|
||||
DEFAULT_MEMORY_SEED,
|
||||
DEFAULT_QUERY,
|
||||
@@ -64,8 +65,15 @@ def register(parent: typer.Typer) -> None:
|
||||
"--server-url",
|
||||
help="EverOS server URL used by --live.",
|
||||
),
|
||||
verbose: bool = typer.Option(
|
||||
False,
|
||||
"--verbose",
|
||||
"-v",
|
||||
help="Emit INFO-level lifecycle logs (default: WARNING only).",
|
||||
),
|
||||
) -> None:
|
||||
"""Launch the EverOS first-memory Textual TUI."""
|
||||
configure_cli_logging(verbose=verbose)
|
||||
if live:
|
||||
_run_live_demo(
|
||||
cinematic=cinematic,
|
||||
|
||||
@@ -17,6 +17,7 @@ from pathlib import Path
|
||||
import typer
|
||||
|
||||
from everos.config.settings import resolve_root
|
||||
from everos.entrypoints.cli._log_setup import configure_cli_logging
|
||||
|
||||
_EVEROS_TEMPLATE = Path(__file__).resolve().parents[3] / "config" / "default.toml"
|
||||
_OME_TEMPLATE = Path(__file__).resolve().parents[3] / "config" / "default_ome.toml"
|
||||
@@ -42,6 +43,12 @@ def register(parent: typer.Typer) -> None:
|
||||
"--print",
|
||||
help="Print the everos.toml template to stdout instead of writing to disk.",
|
||||
),
|
||||
verbose: bool = typer.Option(
|
||||
False,
|
||||
"--verbose",
|
||||
"-v",
|
||||
help="Emit INFO-level lifecycle logs (default: WARNING only).",
|
||||
),
|
||||
) -> None:
|
||||
"""Generate starter configuration files.
|
||||
|
||||
@@ -57,6 +64,7 @@ def register(parent: typer.Typer) -> None:
|
||||
- 0 — files created successfully (or printed to stdout).
|
||||
- 1 — files already exist and ``--force`` was not given.
|
||||
"""
|
||||
configure_cli_logging(verbose=verbose)
|
||||
if print_:
|
||||
import sys
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
"""OME SQLite storage — schema initialization + connection factory.
|
||||
|
||||
Single file (default ``MemoryRoot.default().ome_db`` ≡
|
||||
Single file (default ``MemoryRoot.resolve().ome_db`` ≡
|
||||
``<memory-root>/.index/sqlite/ome.db``). Holds 3 OME-managed tables
|
||||
(counter_store / idle_store / run_record); APS jobstore table is created
|
||||
by APScheduler itself when its SQLAlchemyJobStore connects.
|
||||
|
||||
@@ -16,7 +16,7 @@ from everos.core.persistence.memory_root import MemoryRoot
|
||||
|
||||
|
||||
def _default_jobstore_path() -> Path:
|
||||
return MemoryRoot.default().ome_db
|
||||
return MemoryRoot.resolve().ome_db
|
||||
|
||||
|
||||
class CounterOverride(BaseModel):
|
||||
@@ -82,7 +82,7 @@ class OMEConfig(BaseModel):
|
||||
default_factory=_default_jobstore_path,
|
||||
description="SQLite DB path holding OME's own state (run records, "
|
||||
"counter store, idle store). Defaults to "
|
||||
"``MemoryRoot.default().ome_db`` (``<memory-root>/.index/sqlite/ome.db``).",
|
||||
"``MemoryRoot.resolve().ome_db`` (``<memory-root>/.index/sqlite/ome.db``).",
|
||||
)
|
||||
aps_jobstore_path: Path | None = Field(
|
||||
default=None,
|
||||
@@ -126,6 +126,25 @@ class OMEConfig(BaseModel):
|
||||
"fresh run_id.",
|
||||
),
|
||||
] = 1800
|
||||
crash_recovery_enabled: bool = Field(
|
||||
default=True,
|
||||
description=(
|
||||
"Run ``OfflineEngine._run_crash_recovery`` on ``engine.start()``. "
|
||||
"Default ``True`` — the primary server engine "
|
||||
"(``service.memorize._get_engine``) needs to resume its own "
|
||||
"prior sessions' RUNNING work after a crash / restart. "
|
||||
"Set to ``False`` for one-shot engines that share the jobstore "
|
||||
"path (e.g. ``memory.cascade._backfill._build_cluster_engine`` "
|
||||
"and ``_build_skill_engine``). Those engines register only the "
|
||||
"Phase 2/3 subset of strategies; if they ran crash recovery on "
|
||||
"startup they would re-enqueue the server's stale RUNNING rows "
|
||||
"into their own APS scheduler, whose registry doesn't know "
|
||||
"those strategy names — causing the re-enqueued event to be "
|
||||
"permanently lost via ``KeyError`` at dispatch time. The "
|
||||
"server's stale rows stay in ``run_record`` untouched; the "
|
||||
"next server restart resumes them normally."
|
||||
),
|
||||
)
|
||||
config_path: Path | None = Field(
|
||||
default=None,
|
||||
description="Path to ome.toml for per-strategy overrides. None "
|
||||
|
||||
@@ -290,7 +290,8 @@ class OfflineEngine:
|
||||
self._idle_event.set()
|
||||
self._launch_scheduler()
|
||||
_ENGINES[self._engine_id] = self
|
||||
await self._run_crash_recovery()
|
||||
if self._config.crash_recovery_enabled:
|
||||
await self._run_crash_recovery()
|
||||
self._register_scheduled_jobs()
|
||||
self._start_config_reloader()
|
||||
self._started = True
|
||||
@@ -346,6 +347,13 @@ class OfflineEngine:
|
||||
async def _run_crash_recovery(self) -> None:
|
||||
"""Scan ``run_record`` for stale RUNNING rows and re-enqueue them.
|
||||
|
||||
Runs on :meth:`start` when :attr:`OMEConfig.crash_recovery_enabled`
|
||||
is ``True`` (default). One-shot engines that register a subset of
|
||||
strategies (backfill Phase 2/3) opt out via
|
||||
``crash_recovery_enabled=False`` to avoid re-enqueueing the
|
||||
server's stale rows into their own scheduler, whose registry
|
||||
doesn't contain those strategy names.
|
||||
|
||||
Treats rows whose ``started_at`` is older than
|
||||
``crash_recovery_timeout_seconds`` as crashes from a previous
|
||||
engine session: they are marked CRASHED and re-added to APS with
|
||||
|
||||
@@ -28,7 +28,7 @@ import contextlib
|
||||
import datetime as dt
|
||||
|
||||
from everos.core.observability.logging import get_logger
|
||||
from everos.core.persistence import MemoryRoot
|
||||
from everos.core.persistence import BaseLanceTable, MemoryRoot, memory_root_lock
|
||||
|
||||
# Importing ``tables`` registers every business :class:`BaseLanceTable`
|
||||
# schema so callers can rely on the package alone to surface every schema.
|
||||
@@ -75,6 +75,11 @@ class LanceDBSchemaMismatchError(RuntimeError):
|
||||
"""
|
||||
|
||||
|
||||
class LanceDBMigrationError(RuntimeError):
|
||||
"""Raised when :func:`migrate_table_schemas` cannot complete and
|
||||
startup must abort rather than continue in a half-migrated state."""
|
||||
|
||||
|
||||
_FTS_INDEX_SCHEMA_VERSION = 2
|
||||
"""Bump when the FTS index build config changes so existing on-disk
|
||||
indexes get rebuilt at startup. v2 = ``with_position=False`` (see
|
||||
@@ -96,41 +101,180 @@ async def migrate_fts_indexes() -> None:
|
||||
crashed-optimize churn left behind. Guarded by a version marker in
|
||||
the LanceDB dir so it runs at most once per bump; the rebuild is
|
||||
O(N) but only on the first startup after upgrade.
|
||||
|
||||
Cross-process serialization: wrapped in :func:`memory_root_lock` so a
|
||||
server startup racing with a concurrent ``everos cascade`` command
|
||||
(both call this via :func:`ensure_business_indexes`) can't attempt
|
||||
two concurrent rebuild passes. First process runs the migration and
|
||||
writes the marker; second waits for the lock, re-checks the marker
|
||||
on entry, and no-ops. Removes the same TOCTOU window between marker
|
||||
read and the drop-index/rebuild loop that :issue:`M7` from the
|
||||
PR #361 review flagged on the sibling schema migration.
|
||||
"""
|
||||
logger = get_logger(__name__)
|
||||
marker = MemoryRoot.default().lancedb_dir / ".fts_index_version"
|
||||
try:
|
||||
current = int(marker.read_text().strip()) if marker.exists() else 0
|
||||
except (ValueError, OSError):
|
||||
current = 0
|
||||
if current >= _FTS_INDEX_SCHEMA_VERSION:
|
||||
return
|
||||
logger.info("fts_index_migration_started", target=_FTS_INDEX_SCHEMA_VERSION)
|
||||
for schema in _BUSINESS_SCHEMAS:
|
||||
if not schema.BM25_FIELDS:
|
||||
continue
|
||||
table = await get_table(schema.TABLE_NAME, schema)
|
||||
# Drop existing indexes (everos only builds FTS here; mirrors
|
||||
# LanceRepoBase.rebuild_indexes) then rebuild with the new config.
|
||||
for idx in await table.list_indices():
|
||||
await table.drop_index(idx.name)
|
||||
await schema.ensure_fts_indexes(table)
|
||||
# Reclaim the orphaned index dirs + data fragments the crashed
|
||||
# optimize loop piled up. Safe now: the crashing index is gone,
|
||||
# so compaction no longer decodes a position List.
|
||||
with contextlib.suppress(Exception):
|
||||
await table.optimize(cleanup_older_than=dt.timedelta(seconds=0))
|
||||
marker.write_text(str(_FTS_INDEX_SCHEMA_VERSION))
|
||||
logger.info("fts_index_migration_done", version=_FTS_INDEX_SCHEMA_VERSION)
|
||||
memory_root = MemoryRoot.resolve()
|
||||
async with memory_root_lock(memory_root):
|
||||
marker = memory_root.lancedb_dir / ".fts_index_version"
|
||||
try:
|
||||
current = int(marker.read_text().strip()) if marker.exists() else 0
|
||||
except (ValueError, OSError):
|
||||
current = 0
|
||||
if current >= _FTS_INDEX_SCHEMA_VERSION:
|
||||
# Another process finished the migration while we waited
|
||||
# for the lock. Nothing to do.
|
||||
return
|
||||
logger.info("fts_index_migration_started", target=_FTS_INDEX_SCHEMA_VERSION)
|
||||
for schema in _BUSINESS_SCHEMAS:
|
||||
if not schema.BM25_FIELDS:
|
||||
continue
|
||||
table = await get_table(schema.TABLE_NAME, schema)
|
||||
# Drop existing indexes (everos only builds FTS here; mirrors
|
||||
# LanceRepoBase.rebuild_indexes) then rebuild with the new config.
|
||||
for idx in await table.list_indices():
|
||||
await table.drop_index(idx.name)
|
||||
await schema.ensure_fts_indexes(table)
|
||||
# Reclaim the orphaned index dirs + data fragments the crashed
|
||||
# optimize loop piled up. Safe now: the crashing index is gone,
|
||||
# so compaction no longer decodes a position List.
|
||||
with contextlib.suppress(Exception):
|
||||
await table.optimize(cleanup_older_than=dt.timedelta(seconds=0))
|
||||
marker.write_text(str(_FTS_INDEX_SCHEMA_VERSION))
|
||||
logger.info("fts_index_migration_done", version=_FTS_INDEX_SCHEMA_VERSION)
|
||||
|
||||
|
||||
_TABLE_SCHEMA_VERSION = 2
|
||||
"""Bump when column nullability or presence changes on business tables.
|
||||
v2 = ``vector: Vector(_DIM) | None`` on the 6 business tables that carry
|
||||
a vector column (see :data:`BUSINESS_SCHEMAS_WITH_VECTOR`).
|
||||
|
||||
.. note::
|
||||
|
||||
:func:`migrate_table_schemas` currently hard-codes the v2 alter — it
|
||||
does not implement per-version dispatch. Bumping this constant to
|
||||
``3`` without first adding a dispatch table will short-circuit the
|
||||
v3 migration entirely (the current body only compares
|
||||
``current >= _TABLE_SCHEMA_VERSION`` and then runs the v2-specific
|
||||
alter unconditionally). Any future v3 migration MUST:
|
||||
|
||||
1. Add a ``_MIGRATIONS: dict[int, Callable[[], Awaitable[None]]]``
|
||||
registry mapping target version → migration coroutine.
|
||||
2. Rewrite :func:`migrate_table_schemas` to run every migration from
|
||||
``current + 1`` up to ``_TABLE_SCHEMA_VERSION`` in order, updating
|
||||
the marker after each step so a mid-migration crash resumes at
|
||||
the next step instead of replaying earlier ones.
|
||||
|
||||
Accepted as a design gap in PR #361 review finding #5 — not blocking
|
||||
today because there is no v3.
|
||||
"""
|
||||
|
||||
|
||||
BUSINESS_SCHEMAS_WITH_VECTOR: tuple[type[BaseLanceTable], ...] = (
|
||||
Episode,
|
||||
AtomicFact,
|
||||
Foresight,
|
||||
AgentCase,
|
||||
AgentSkill,
|
||||
KnowledgeTopic,
|
||||
)
|
||||
"""Business schemas whose ``vector`` column needs the v2 nullability
|
||||
migration. ``Episode.subject_vector`` was already nullable in v1.1.1 and
|
||||
is intentionally left out of this migration. ``UserProfile`` has no
|
||||
vector column; ``knowledge_document`` has no LanceDB table."""
|
||||
|
||||
|
||||
async def migrate_table_schemas() -> None:
|
||||
"""One-time ``alter_columns`` making ``vector`` nullable on business tables.
|
||||
|
||||
Soft-dependency embedding mode lets cascade write ``vector=None`` for
|
||||
rows it cannot embed; the on-disk column must allow that. LanceDB's
|
||||
``alter_columns`` is metadata-only (no data movement, no re-embed),
|
||||
so this runs in milliseconds. Guarded by ``.table_schema_version``
|
||||
so it runs at most once per version bump; per-table nullable check
|
||||
is defense-in-depth so an already-migrated (or freshly-created,
|
||||
already-nullable) table is a no-op rather than a redundant alter.
|
||||
|
||||
Cross-process serialization: wrapped in :func:`memory_root_lock` so
|
||||
a server startup racing with a concurrent ``everos cascade`` command
|
||||
(both call this via :func:`ensure_business_indexes`) doesn't attempt
|
||||
two concurrent alter passes. First process runs the migration and
|
||||
writes the marker; second waits for the lock, re-checks the marker
|
||||
on entry, and no-ops. Removes the TOCTOU window between marker
|
||||
read and alter that :issue:`M7` from the PR #361 review flagged.
|
||||
|
||||
Fail-loud semantics preserved: if ``alter_columns`` genuinely fails
|
||||
for any table (LanceDB version mismatch, corrupted index), raises
|
||||
:class:`LanceDBMigrationError`. Startup aborts rather than
|
||||
continuing in a half-migrated state — cascade would otherwise write
|
||||
``vector=None`` (soft-dependency embedding) into a still NOT-NULL
|
||||
column and every row would silently fail. Recovery escalates from
|
||||
a plain restart (transient hiccup) to wiping the LanceDB index
|
||||
directory (rebuildable from md, the SoT).
|
||||
"""
|
||||
logger = get_logger(__name__)
|
||||
memory_root = MemoryRoot.resolve()
|
||||
async with memory_root_lock(memory_root):
|
||||
marker = memory_root.lancedb_dir / ".table_schema_version"
|
||||
try:
|
||||
current = int(marker.read_text().strip()) if marker.exists() else 0
|
||||
except (ValueError, OSError):
|
||||
current = 0
|
||||
if current >= _TABLE_SCHEMA_VERSION:
|
||||
# Another process finished the migration while we waited
|
||||
# for the lock. Nothing to do.
|
||||
return
|
||||
logger.info("table_schema_migration_started", target=_TABLE_SCHEMA_VERSION)
|
||||
failed_schemas: list[str] = []
|
||||
for schema in BUSINESS_SCHEMAS_WITH_VECTOR:
|
||||
table = await get_table(schema.TABLE_NAME, schema)
|
||||
arrow_schema = await table.schema()
|
||||
if arrow_schema.field("vector").nullable:
|
||||
continue
|
||||
try:
|
||||
await table.alter_columns({"path": "vector", "nullable": True})
|
||||
except Exception as exc:
|
||||
logger.warning(
|
||||
"table_schema_migration_alter_failed",
|
||||
table=schema.TABLE_NAME,
|
||||
error=str(exc),
|
||||
)
|
||||
failed_schemas.append(schema.TABLE_NAME)
|
||||
|
||||
if failed_schemas:
|
||||
logger.error(
|
||||
"table_schema_migration_incomplete",
|
||||
failed_schemas=failed_schemas,
|
||||
)
|
||||
raise LanceDBMigrationError(
|
||||
f"LanceDB nullable-vector migration failed for tables "
|
||||
f"{failed_schemas!r}. Startup aborted to prevent cascade "
|
||||
f"from writing NULL vectors into NOT-NULL columns. This "
|
||||
f"typically indicates a LanceDB version mismatch or a "
|
||||
f"corrupted index. Recovery, in order of least- to "
|
||||
f"most-destructive: (1) restart the process; a transient "
|
||||
f"filesystem or LanceDB-side hiccup may resolve. (2) If "
|
||||
f"the error persists, wipe the index directory "
|
||||
f"`{memory_root.lancedb_dir}` and restart — cascade will "
|
||||
f"re-index from source markdown."
|
||||
)
|
||||
|
||||
marker.parent.mkdir(parents=True, exist_ok=True)
|
||||
try:
|
||||
marker.write_text(str(_TABLE_SCHEMA_VERSION))
|
||||
except OSError:
|
||||
logger.error("table_schema_migration_marker_write_failed", path=str(marker))
|
||||
raise
|
||||
logger.info("table_schema_migration_done", version=_TABLE_SCHEMA_VERSION)
|
||||
|
||||
|
||||
async def ensure_business_indexes() -> None:
|
||||
"""Ensure FTS (BM25) indexes for every business table (idempotent).
|
||||
|
||||
Called once at startup by :class:`LanceDBLifespanProvider`. First
|
||||
runs :func:`migrate_fts_indexes` (one-time, marker-guarded) to
|
||||
rebuild any pre-fix ``with_position=True`` indexes, then walks the
|
||||
business schemas (each owns its ``TABLE_NAME`` + ``BM25_FIELDS``),
|
||||
runs :func:`migrate_table_schemas` (schema first, one-time,
|
||||
marker-guarded) to make ``vector`` columns nullable, then
|
||||
:func:`migrate_fts_indexes` (index second, one-time, marker-guarded)
|
||||
to rebuild any pre-fix ``with_position=True`` indexes, then walks
|
||||
the business schemas (each owns its ``TABLE_NAME`` + ``BM25_FIELDS``),
|
||||
opens each table via :func:`get_table`, and delegates to
|
||||
``schema.ensure_fts_indexes(table)``. Already-indexed columns are
|
||||
skipped, so re-runs are no-ops.
|
||||
@@ -139,6 +283,7 @@ async def ensure_business_indexes() -> None:
|
||||
everything else (table name, columns to index) reads off the
|
||||
schema's ClassVars.
|
||||
"""
|
||||
await migrate_table_schemas()
|
||||
await migrate_fts_indexes()
|
||||
for schema in _BUSINESS_SCHEMAS:
|
||||
table = await get_table(schema.TABLE_NAME, schema)
|
||||
@@ -166,19 +311,29 @@ async def verify_business_schemas() -> None:
|
||||
if missing or extra:
|
||||
raise LanceDBSchemaMismatchError(
|
||||
f"LanceDB table {schema.TABLE_NAME!r} schema drift: "
|
||||
f"missing={sorted(missing)}, extra={sorted(extra)}. "
|
||||
"The index is rebuildable from md — recover with "
|
||||
"`rm -rf ~/.everos/.index/lancedb` and restart."
|
||||
f"missing={sorted(missing)}, extra={sorted(extra)}.\n"
|
||||
"Recovery, escalating:\n"
|
||||
" 1. Restart the server — an in-flight migration may "
|
||||
"still be finishing (harmless if this is your first "
|
||||
"restart after upgrading EverOS).\n"
|
||||
" 2. If restart doesn't clear it, the on-disk index is "
|
||||
"genuinely out of sync with the code's schema. Because "
|
||||
"the LanceDB index is fully rebuildable from md, wipe "
|
||||
"it: `rm -rf ~/.everos/.index/lancedb` and restart — "
|
||||
"the cascade daemon will re-index from the source-of-"
|
||||
"truth markdown."
|
||||
)
|
||||
|
||||
|
||||
__all__ = [
|
||||
"BUSINESS_SCHEMAS_WITH_VECTOR",
|
||||
"AgentCase",
|
||||
"AgentSkill",
|
||||
"AtomicFact",
|
||||
"Episode",
|
||||
"Foresight",
|
||||
"KnowledgeTopic",
|
||||
"LanceDBMigrationError",
|
||||
"LanceDBSchemaMismatchError",
|
||||
"ParentType",
|
||||
"UserProfile",
|
||||
@@ -193,6 +348,7 @@ __all__ = [
|
||||
"get_table",
|
||||
"knowledge_topic_repo",
|
||||
"migrate_fts_indexes",
|
||||
"migrate_table_schemas",
|
||||
"user_profile_repo",
|
||||
"verify_business_schemas",
|
||||
]
|
||||
|
||||
@@ -28,7 +28,7 @@ _lock = asyncio.Lock()
|
||||
async def get_connection() -> AsyncConnection:
|
||||
"""Return the process-wide async LanceDB connection.
|
||||
|
||||
Built on first call from ``MemoryRoot.default().lancedb_dir`` and
|
||||
Built on first call from ``MemoryRoot.resolve().lancedb_dir`` and
|
||||
``Settings.lancedb``. Subsequent calls return the same instance.
|
||||
"""
|
||||
async with _lock:
|
||||
@@ -72,7 +72,7 @@ async def _ensure_connection_locked() -> AsyncConnection:
|
||||
global _conn
|
||||
if _conn is None:
|
||||
settings = load_settings()
|
||||
memory_root = MemoryRoot.default()
|
||||
memory_root = MemoryRoot.resolve()
|
||||
memory_root.ensure()
|
||||
_conn = await open_lancedb_connection(memory_root.lancedb_dir, settings.lancedb)
|
||||
logger.info(
|
||||
|
||||
@@ -2,9 +2,14 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime as dt
|
||||
from collections.abc import Sequence
|
||||
from typing import Any
|
||||
|
||||
from lancedb import AsyncTable
|
||||
|
||||
from everos.core.persistence.lancedb import LanceDailyLogRepoBase
|
||||
from everos.core.persistence.lancedb.repository import _q
|
||||
|
||||
from ..lancedb_manager import get_table
|
||||
from ..tables.episode import Episode
|
||||
@@ -16,5 +21,146 @@ class _EpisodeRepo(LanceDailyLogRepoBase[Episode]):
|
||||
async def _table_lookup(self) -> AsyncTable:
|
||||
return await get_table(self.schema.TABLE_NAME, self.schema)
|
||||
|
||||
async def count_by_owner(
|
||||
self,
|
||||
owner_id: str,
|
||||
*,
|
||||
app_id: str = "default",
|
||||
project_id: str = "default",
|
||||
parent_type: str | None = None,
|
||||
) -> int:
|
||||
"""Count episode rows for one owner within an ``(app, project)`` scope.
|
||||
|
||||
Args:
|
||||
owner_id: The row's ``owner_id`` field.
|
||||
app_id: App scope segment (default ``"default"``).
|
||||
project_id: Project scope segment (default ``"default"``).
|
||||
parent_type: When non-``None``, restrict the count to rows whose
|
||||
``parent_type`` matches (e.g. ``"memcell"`` to exclude
|
||||
Reflection-merged rows with ``parent_type='cluster'``). Kept
|
||||
optional so callers without a parent-type requirement rely on
|
||||
the ``None`` default meaning "count everything".
|
||||
|
||||
Used by :func:`extract_user_profile`'s Tier-1 direct-path throttle
|
||||
to gate re-extraction frequency. LanceDB ``count_rows`` accepts a
|
||||
SQL-style ``filter`` predicate; the where-string is built with the
|
||||
same escape discipline (:func:`_q`) as
|
||||
:meth:`list_by_owner_after_ts`.
|
||||
|
||||
Excludes rows with ``deprecated_by IS NOT NULL`` (Reflection-
|
||||
superseded episodes) from the count, matching the filter idiom
|
||||
used by :func:`memory.search.filters.compile_filters` and the
|
||||
reflection read paths. Keeps the profile-throttle count aligned
|
||||
with the "live" episode surface the strategy actually operates on.
|
||||
"""
|
||||
filter_str = (
|
||||
f"owner_id = '{_q(owner_id)}' "
|
||||
f"AND app_id = '{_q(app_id)}' "
|
||||
f"AND project_id = '{_q(project_id)}' "
|
||||
f"AND deprecated_by IS NULL"
|
||||
)
|
||||
if parent_type is not None:
|
||||
filter_str += f" AND parent_type = '{_q(parent_type)}'"
|
||||
table = await self._table()
|
||||
return await table.count_rows(filter=filter_str)
|
||||
|
||||
async def list_by_owner_after_ts(
|
||||
self,
|
||||
*,
|
||||
owner_id: str,
|
||||
after_ts: int,
|
||||
parent_type: str,
|
||||
app_id: str = "default",
|
||||
project_id: str = "default",
|
||||
columns: Sequence[str] | None = None,
|
||||
limit: int | None = None,
|
||||
) -> list[Episode] | list[dict[str, Any]]:
|
||||
"""Scalar filter over owner + timestamp + parent_type + scope.
|
||||
|
||||
Fetches episodes for the given owner after a specific timestamp,
|
||||
scoped by parent_type and app/project. No vector dependency —
|
||||
Tier 1-safe. Rows with vector=None are included.
|
||||
|
||||
Used by extract_user_profile direct path (KEYWORD_ONLY mode) to
|
||||
select memcells to feed the profile extractor without going through
|
||||
cluster events.
|
||||
|
||||
Excludes rows with ``deprecated_by IS NOT NULL`` (Reflection-
|
||||
superseded episodes) — matches the filter idiom used by
|
||||
:func:`memory.search.filters.compile_filters` and reflection
|
||||
reads. Without this, direct-path profile extraction would feed
|
||||
stale (superseded) memcell references to the extractor.
|
||||
|
||||
Args:
|
||||
owner_id: Owner identifier.
|
||||
after_ts: Timestamp threshold (milliseconds since epoch); rows with
|
||||
timestamp > after_ts are returned.
|
||||
parent_type: Parent type filter (e.g. "memcell").
|
||||
app_id: Application scope (default "default").
|
||||
project_id: Project scope (default "default").
|
||||
columns: Optional column projection. Default (``None``)
|
||||
returns full ``Episode`` model objects. Callers that only
|
||||
need a small subset (e.g. ``parent_id`` in
|
||||
``extract_user_profile``) should pass an explicit list to
|
||||
skip the 1024-D ``vector`` and ``subject_vector`` payload.
|
||||
When set, this method returns raw dicts scoped to the
|
||||
requested columns; otherwise it returns typed ``Episode``
|
||||
objects. Callers switch on the projection they asked for.
|
||||
limit: Optional row cap. Default (``None``) means no cap —
|
||||
the entire matching set is fetched. Callers with large
|
||||
historical windows should pass a bound to avoid pulling
|
||||
every memcell for the owner.
|
||||
|
||||
**CAVEAT — limit is NOT "newest N"**: LanceDB does not
|
||||
push ORDER BY down; ``.limit(N)`` truncates whatever
|
||||
the scan returns first (typically fragment / insertion
|
||||
order for an append-only table = OLDEST N). The
|
||||
ascending-by-timestamp sort below runs on the already-
|
||||
truncated Python list, so the returned rows are the
|
||||
*oldest matching subset*, not the newest. Callers that
|
||||
need "most recent N" MUST either fetch unlimited and
|
||||
slice in Python (accepting the memory cost), or apply
|
||||
a tighter ``after_ts`` bound to shrink the matching
|
||||
set upstream. A regression here would silently window
|
||||
profile extraction over the wrong slice of history.
|
||||
|
||||
Returns:
|
||||
When ``columns`` is ``None``: list of ``Episode`` objects
|
||||
ordered ascending by timestamp (existing behaviour).
|
||||
When ``columns`` is a projection: list of raw dicts holding
|
||||
only the requested columns, ordered ascending by timestamp.
|
||||
"""
|
||||
from everos.component.utils.datetime import to_iso_format
|
||||
|
||||
# Convert milliseconds to UTC datetime, then to ISO string for DataFusion
|
||||
# (stored timestamps are always UTC)
|
||||
ts_dt = dt.datetime.fromtimestamp(int(after_ts) / 1000.0, tz=dt.UTC)
|
||||
ts_iso = to_iso_format(ts_dt)
|
||||
|
||||
where = (
|
||||
f"owner_id = '{_q(owner_id)}' "
|
||||
f"AND timestamp > TIMESTAMP '{ts_iso}' "
|
||||
f"AND parent_type = '{_q(parent_type)}' "
|
||||
f"AND app_id = '{_q(app_id)}' "
|
||||
f"AND project_id = '{_q(project_id)}' "
|
||||
f"AND deprecated_by IS NULL"
|
||||
)
|
||||
table = await self._table()
|
||||
query = table.query().where(where)
|
||||
# Projection: caller opted into a raw-dict return by naming columns;
|
||||
# `timestamp` is force-included so the ascending sort below is stable
|
||||
# even when the caller forgot to list it.
|
||||
projected = columns is not None
|
||||
if projected:
|
||||
projection = list(dict.fromkeys([*columns, "timestamp"])) # type: ignore[misc]
|
||||
query = query.select(projection)
|
||||
if limit is not None:
|
||||
query = query.limit(limit)
|
||||
rows = await query.to_list()
|
||||
rows.sort(key=lambda r: r["timestamp"])
|
||||
if projected:
|
||||
return rows
|
||||
return [self.schema.model_validate(r) for r in rows]
|
||||
|
||||
|
||||
episode_repo = _EpisodeRepo()
|
||||
|
||||
@@ -81,4 +81,4 @@ class AgentCase(BaseLanceTable):
|
||||
session_id / timestamp / parent_id) is NOT in the hash. See
|
||||
:attr:`AgentCaseHandler.content_change_keys`."""
|
||||
|
||||
vector: Vector(_DIM) # type: ignore[valid-type]
|
||||
vector: Vector(_DIM) | None = None # type: ignore[valid-type]
|
||||
|
||||
@@ -77,4 +77,4 @@ class AgentSkill(BaseLanceTable):
|
||||
changed (e.g. the watcher fires for unrelated stat updates). See
|
||||
:attr:`AgentSkillHandler.content_change_keys`."""
|
||||
|
||||
vector: Vector(_DIM) # type: ignore[valid-type]
|
||||
vector: Vector(_DIM) | None = None # type: ignore[valid-type]
|
||||
|
||||
@@ -64,4 +64,4 @@ class AtomicFact(BaseLanceTable):
|
||||
consolidated. Value is the cluster entry_id that supersedes this
|
||||
row. ``NULL`` means the row is still active."""
|
||||
|
||||
vector: Vector(_DIM) # type: ignore[valid-type]
|
||||
vector: Vector(_DIM) | None = None # type: ignore[valid-type]
|
||||
|
||||
@@ -80,5 +80,5 @@ class Episode(BaseLanceTable):
|
||||
consolidated into a cluster. Value is the cluster entry_id that
|
||||
supersedes this row. ``NULL`` means the row is still active."""
|
||||
|
||||
vector: Vector(_DIM) # type: ignore[valid-type]
|
||||
vector: Vector(_DIM) | None = None # type: ignore[valid-type]
|
||||
subject_vector: Vector(_DIM) | None = None # type: ignore[valid-type]
|
||||
|
||||
@@ -76,4 +76,4 @@ class Foresight(BaseLanceTable):
|
||||
(owner_id / session_id / timestamp / parent_id / sender_ids) is NOT
|
||||
in the hash. See :attr:`ForesightHandler.content_change_keys`."""
|
||||
|
||||
vector: Vector(_DIM) # type: ignore[valid-type]
|
||||
vector: Vector(_DIM) | None = None # type: ignore[valid-type]
|
||||
|
||||
@@ -31,6 +31,6 @@ class KnowledgeTopic(BaseLanceTable):
|
||||
content_labels: list[str] = []
|
||||
md_path: str
|
||||
content_sha256: str
|
||||
vector: Vector(_DIM) # type: ignore[valid-type] -- Vector() is runtime-constructed; static analyzers cannot verify
|
||||
vector: Vector(_DIM) | None = None # type: ignore[valid-type] -- Vector() is runtime-constructed; static analyzers cannot verify
|
||||
created_at: dt.datetime
|
||||
updated_at: dt.datetime
|
||||
|
||||
@@ -113,17 +113,43 @@ class _ClusterRepo(RepoBase[Cluster]):
|
||||
self,
|
||||
member_type: str,
|
||||
member_id: str,
|
||||
*,
|
||||
app_id: str,
|
||||
project_id: str,
|
||||
owner_id: str,
|
||||
) -> str | None:
|
||||
"""Reverse lookup: ``(member_type, member_id) → cluster_id``.
|
||||
"""Reverse lookup: ``(member_type, member_id) → cluster_id`` scoped
|
||||
to ``(app_id, project_id, owner_id)``.
|
||||
|
||||
Returns ``None`` when the entity is not yet attached to any cluster.
|
||||
Backed by ``ix_cluster_member_reverse`` so it is O(log N).
|
||||
``member_id`` (e.g. episode ``entry_id`` like
|
||||
``ep_20260517_00000001``) is only per-owner unique — see
|
||||
``core/persistence/markdown/entries.py``: "Cross-user uniqueness
|
||||
is handled at the database layer via a composite
|
||||
``<user_id>_<entry_id>`` field; it is not encoded into the
|
||||
EntryId string itself." Without the scope filter, two owners
|
||||
writing on the same day would share ``entry_id`` and either
|
||||
collide on the reverse index (false hit → the second owner's
|
||||
row silently drops from a cluster it was never part of) or find
|
||||
a foreign cluster.
|
||||
|
||||
Filter cascades: reverse index on ``(member_type, member_id)``
|
||||
narrows to O(k) rows where k = number of owners sharing that
|
||||
entry_id (usually 1), then the join to :class:`Cluster` filters
|
||||
by scope. ``Cluster`` already carries ``app_id / project_id /
|
||||
owner_id`` as of the initial schema.
|
||||
|
||||
Returns ``None`` when the entity is not attached to a cluster
|
||||
under the specified scope.
|
||||
"""
|
||||
async with session_scope(self._factory) as s:
|
||||
stmt = (
|
||||
select(ClusterMember.cluster_id)
|
||||
.join(Cluster, Cluster.cluster_id == ClusterMember.cluster_id)
|
||||
.where(ClusterMember.member_type == member_type)
|
||||
.where(ClusterMember.member_id == member_id)
|
||||
.where(Cluster.app_id == app_id)
|
||||
.where(Cluster.project_id == project_id)
|
||||
.where(Cluster.owner_id == owner_id)
|
||||
.limit(1)
|
||||
)
|
||||
return (await s.execute(stmt)).scalar_one_or_none()
|
||||
|
||||
@@ -207,30 +207,48 @@ class _MdChangeStateRepo(RepoBase[MdChangeState]):
|
||||
row = await s.get(MdChangeState, md_path)
|
||||
return row
|
||||
|
||||
async def claim_pending_batch(self, limit: int = 100) -> list[MdChangeState]:
|
||||
async def claim_pending_batch(
|
||||
self,
|
||||
limit: int = 100,
|
||||
*,
|
||||
kinds: set[str] | None = None,
|
||||
) -> list[MdChangeState]:
|
||||
"""Claim up to ``limit`` pending rows in LSN order.
|
||||
|
||||
Returns the claimed rows (now ``status='processing'``); empty
|
||||
list if none were pending. Sibling workers / processes may race
|
||||
on the same prefix — the per-row ``WHERE status='pending'``
|
||||
filter ensures each row lands in exactly one batch.
|
||||
|
||||
Args:
|
||||
limit: Maximum rows to claim in one batch.
|
||||
kinds: Optional restriction on which ``kind`` values are
|
||||
eligible. ``None`` (the default) claims across every
|
||||
registered kind — the shape the background worker and
|
||||
the CLI ``cascade sync`` path have always used. Passing
|
||||
a set restricts the SELECT + UPDATE to ``kind IN (...)``
|
||||
via parameter binding (no string interpolation), so a
|
||||
Phase-3 sync scoped to ``{"agent_skill"}`` cannot touch
|
||||
unrelated kinds' queue state. Empty set is treated the
|
||||
same as ``None`` — meaning "no kind filter" would be
|
||||
surprising, but returning an empty result is what the
|
||||
caller almost certainly meant, so we short-circuit.
|
||||
"""
|
||||
if limit <= 0:
|
||||
return []
|
||||
if kinds is not None and not kinds:
|
||||
return []
|
||||
now = get_utc_now()
|
||||
async with session_scope(self._factory) as s:
|
||||
picks = (
|
||||
(
|
||||
await s.execute(
|
||||
select(MdChangeState.md_path)
|
||||
.where(MdChangeState.status == "pending")
|
||||
.order_by(MdChangeState.lsn)
|
||||
.limit(limit)
|
||||
)
|
||||
)
|
||||
.scalars()
|
||||
.all()
|
||||
pick_stmt = (
|
||||
select(MdChangeState.md_path)
|
||||
.where(MdChangeState.status == "pending")
|
||||
.order_by(MdChangeState.lsn)
|
||||
.limit(limit)
|
||||
)
|
||||
if kinds is not None:
|
||||
pick_stmt = pick_stmt.where(MdChangeState.kind.in_(kinds))
|
||||
picks = (await s.execute(pick_stmt)).scalars().all()
|
||||
if not picks:
|
||||
return []
|
||||
update_result = await s.execute(
|
||||
|
||||
@@ -17,6 +17,17 @@ from everos.core.persistence.sqlite import RepoBase, session_scope
|
||||
from ..sqlite_manager import get_session_factory
|
||||
from ..tables import Memcell
|
||||
|
||||
_SQLITE_IN_CHUNK = 500
|
||||
"""SQL ``IN (?, ?, ...)`` parameter cap for :meth:`_MemcellRepo.find_by_ids`.
|
||||
|
||||
SQLite's ``SQLITE_MAX_VARIABLE_NUMBER`` defaults to 999 on the shipped
|
||||
amalgamation and rises to 32766 on ≥3.32; 500 stays well under the older
|
||||
cap and keeps the query planner's cost estimate cheap. Callers passing
|
||||
more ``memcell_ids`` than this fan out into multiple SELECTs, merged
|
||||
in-memory — see the direct-path selector in
|
||||
``memory/strategies/extract_user_profile.py`` where
|
||||
``last_profile_ts=0`` can pull an owner's entire episode history."""
|
||||
|
||||
|
||||
class _MemcellRepo(RepoBase[Memcell]):
|
||||
model = Memcell
|
||||
@@ -36,16 +47,25 @@ class _MemcellRepo(RepoBase[Memcell]):
|
||||
async def find_by_ids(self, memcell_ids: list[str]) -> list[Memcell]:
|
||||
"""Bulk fetch rows by primary key list — preserves caller order.
|
||||
|
||||
Chunked at ``_SQLITE_IN_CHUNK`` to protect against
|
||||
``SQLITE_MAX_VARIABLE_NUMBER`` blowups on large owner histories
|
||||
(e.g. ``extract_user_profile``'s direct path when ``user.md``
|
||||
doesn't exist yet — ``last_profile_ts=0`` fetches every memcell
|
||||
the owner has ever produced).
|
||||
|
||||
Used by offline strategies that pull every memcell in a cluster
|
||||
(membership lives in :class:`ClusterMember` and is supplied to
|
||||
the strategy via :class:`everalgo.clustering.Cluster.members`).
|
||||
"""
|
||||
if not memcell_ids:
|
||||
return []
|
||||
by_id: dict[str, Memcell] = {}
|
||||
async with session_scope(self._factory) as s:
|
||||
stmt = select(Memcell).where(Memcell.memcell_id.in_(memcell_ids))
|
||||
rows = list((await s.execute(stmt)).scalars().all())
|
||||
by_id = {r.memcell_id: r for r in rows}
|
||||
for start in range(0, len(memcell_ids), _SQLITE_IN_CHUNK):
|
||||
chunk = memcell_ids[start : start + _SQLITE_IN_CHUNK]
|
||||
stmt = select(Memcell).where(Memcell.memcell_id.in_(chunk))
|
||||
rows = (await s.execute(stmt)).scalars().all()
|
||||
by_id.update((r.memcell_id, r) for r in rows)
|
||||
return [by_id[mid] for mid in memcell_ids if mid in by_id]
|
||||
|
||||
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
The single place that owns the SQLite **runtime state**: the async
|
||||
SQLAlchemy engine and the session factory bound to it. Built lazily on
|
||||
first :func:`get_engine` / :func:`get_session_factory` call from
|
||||
:func:`everos.config.load_settings` + :meth:`MemoryRoot.default`. The
|
||||
:func:`everos.config.load_settings` + :meth:`MemoryRoot.resolve`. The
|
||||
:class:`SqliteLifespanProvider` calls :func:`dispose_engine` on shutdown
|
||||
to drain the connection pool; in scripts you can call it manually.
|
||||
"""
|
||||
@@ -29,13 +29,13 @@ _session_factory: async_sessionmaker[AsyncSession] | None = None
|
||||
def get_engine() -> AsyncEngine:
|
||||
"""Return the process-wide async SQLAlchemy engine.
|
||||
|
||||
Built on first call from ``MemoryRoot.default()`` and ``Settings.sqlite``.
|
||||
Built on first call from ``MemoryRoot.resolve()`` and ``Settings.sqlite``.
|
||||
Subsequent calls return the same instance.
|
||||
"""
|
||||
global _engine
|
||||
if _engine is None:
|
||||
settings = load_settings()
|
||||
memory_root = MemoryRoot.default()
|
||||
memory_root = MemoryRoot.resolve()
|
||||
memory_root.ensure()
|
||||
_engine = create_system_engine(memory_root.system_db, settings.sqlite)
|
||||
logger.info(
|
||||
|
||||
@@ -11,8 +11,16 @@ Public surface — what lifespan providers / CLI commands import:
|
||||
- :class:`CascadeConfig` — construction-time tuning knobs.
|
||||
- :data:`KIND_REGISTRY` / :func:`match_kind` — kind dispatch (also
|
||||
used by CLI ``cascade sync --path`` to resolve a single file's kind).
|
||||
- :class:`BackfillPhase` — dataclass shared between the memory-layer
|
||||
phase runners and the CLI's ``PHASES`` copy. The concrete ``PHASES``
|
||||
tuple and ``run_backfill`` orchestrator live in
|
||||
``everos.entrypoints.cli.commands._backfill_cmd`` (M11: the memory
|
||||
layer must not depend on typer/click).
|
||||
"""
|
||||
|
||||
from ._backfill import BackfillPhase as BackfillPhase
|
||||
from ._backfill import BackfillPresenter as BackfillPresenter
|
||||
from ._backfill import NullBackfillPresenter as NullBackfillPresenter
|
||||
from .orchestrator import CascadeConfig as CascadeConfig
|
||||
from .orchestrator import CascadeOrchestrator as CascadeOrchestrator
|
||||
from .registry import KIND_REGISTRY as KIND_REGISTRY
|
||||
@@ -21,8 +29,11 @@ from .registry import match_kind as match_kind
|
||||
|
||||
__all__ = [
|
||||
"KIND_REGISTRY",
|
||||
"BackfillPhase",
|
||||
"BackfillPresenter",
|
||||
"CascadeConfig",
|
||||
"CascadeOrchestrator",
|
||||
"KindSpec",
|
||||
"NullBackfillPresenter",
|
||||
"match_kind",
|
||||
]
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -18,6 +18,8 @@ md contract:
|
||||
``sections``:
|
||||
|
||||
- ``TaskIntent``: short intent statement (embedded + BM25 primary).
|
||||
Embedding is a soft dependency: when unavailable, ``vector`` is
|
||||
written as ``None`` and the row stays BM25/scalar-searchable only.
|
||||
- ``Approach``: step-by-step approach (BM25 secondary field via
|
||||
``approach_tokens``, **not** fed to the embedder — the
|
||||
retrieval anchor is task_intent).
|
||||
@@ -26,11 +28,15 @@ md contract:
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from everos.component.embedding import get_embedding_capability
|
||||
from everos.core.observability.logging import get_logger
|
||||
from everos.infra.persistence.lancedb import AgentCase, ParentType, agent_case_repo
|
||||
|
||||
from ._common import require_float, require_iso_timestamp
|
||||
from ._daily_log_base import BaseDailyLogHandler, ParsedEntry
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class AgentCaseHandler(BaseDailyLogHandler):
|
||||
"""Cascade handler for ``agents/<a>/.cases/agent_case-*.md``."""
|
||||
@@ -62,7 +68,14 @@ class AgentCaseHandler(BaseDailyLogHandler):
|
||||
key_insight = (s.sections.get("KeyInsight") or "").strip() or None
|
||||
intent_tokens = self._deps.tokenizer.tokenize(task_intent)
|
||||
approach_tokens = self._deps.tokenizer.tokenize(approach)
|
||||
vector = await self._deps.embedder.embed(task_intent)
|
||||
vector = await get_embedding_capability().embed_or_none(task_intent)
|
||||
if vector is None:
|
||||
logger.debug(
|
||||
"cascade_handler_embed_skipped",
|
||||
kind=self.kind,
|
||||
entry_id=entry.entry_id,
|
||||
reason="embedding_capability_unavailable",
|
||||
)
|
||||
return AgentCase(
|
||||
id=f"{owner_id}_{entry.entry_id}",
|
||||
entry_id=entry.entry_id,
|
||||
|
||||
@@ -27,7 +27,9 @@ md contract:
|
||||
|
||||
Embedding source: ``name + "\\n" + description`` (mirrors opensource
|
||||
AgentSkillExtractor — name belongs in the retrieval anchor too,
|
||||
not just description).
|
||||
not just description). Embedding is a soft dependency: when
|
||||
unavailable, ``vector`` is written as ``None`` and the row stays
|
||||
BM25/scalar-searchable only.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -37,6 +39,8 @@ from typing import Any, ClassVar
|
||||
|
||||
import anyio
|
||||
|
||||
from everos.component.embedding import get_embedding_capability
|
||||
from everos.core.observability.logging import get_logger
|
||||
from everos.core.persistence import MarkdownReader
|
||||
from everos.infra.persistence.lancedb import AgentSkill, agent_skill_repo
|
||||
from everos.infra.persistence.markdown import AgentSkillFrontmatter
|
||||
@@ -46,6 +50,8 @@ from ._common import content_sha256 as compute_content_sha256
|
||||
from ._common import resolve_scope
|
||||
from .base import Handler
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class AgentSkillHandler(Handler):
|
||||
"""Cascade handler for
|
||||
@@ -140,7 +146,14 @@ class AgentSkillHandler(Handler):
|
||||
content_tokens = " ".join(self._deps.tokenizer.tokenize(content))
|
||||
# Embedding source: name + description joined (opensource parity).
|
||||
embed_text = "\n".join(s for s in [name, description] if s)
|
||||
vector = await self._deps.embedder.embed(embed_text)
|
||||
vector = await get_embedding_capability().embed_or_none(embed_text)
|
||||
if vector is None:
|
||||
logger.debug(
|
||||
"cascade_handler_embed_skipped",
|
||||
kind=self.kind,
|
||||
entry_id=skill_id,
|
||||
reason="embedding_capability_unavailable",
|
||||
)
|
||||
|
||||
row = AgentSkill(
|
||||
id=skill_id,
|
||||
|
||||
@@ -14,16 +14,22 @@ md contract (md writer + cascade share this shape):
|
||||
``sections``:
|
||||
|
||||
- ``Fact``: the atomic-fact sentence — fed to the embedder and the
|
||||
tokenizer (``fact_tokens`` BM25 field).
|
||||
tokenizer (``fact_tokens`` BM25 field). Embedding is a soft
|
||||
dependency: when unavailable, ``vector`` is written as ``None`` and
|
||||
the row stays BM25/scalar-searchable only.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from everos.component.embedding import get_embedding_capability
|
||||
from everos.core.observability.logging import get_logger
|
||||
from everos.infra.persistence.lancedb import AtomicFact, ParentType, atomic_fact_repo
|
||||
|
||||
from ._common import parse_inline_list, require_iso_timestamp
|
||||
from ._daily_log_base import BaseDailyLogHandler, ParsedEntry
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class AtomicFactHandler(BaseDailyLogHandler):
|
||||
"""Cascade handler for ``users/<u>/.atomic_facts/atomic_fact-*.md``."""
|
||||
@@ -47,7 +53,14 @@ class AtomicFactHandler(BaseDailyLogHandler):
|
||||
s = entry.structured
|
||||
text = s.sections.get("Fact", "").strip()
|
||||
tokens = self._deps.tokenizer.tokenize(text)
|
||||
vector = await self._deps.embedder.embed(text)
|
||||
vector = await get_embedding_capability().embed_or_none(text)
|
||||
if vector is None:
|
||||
logger.debug(
|
||||
"cascade_handler_embed_skipped",
|
||||
kind=self.kind,
|
||||
entry_id=entry.entry_id,
|
||||
reason="embedding_capability_unavailable",
|
||||
)
|
||||
return AtomicFact(
|
||||
id=f"{owner_id}_{entry.entry_id}",
|
||||
entry_id=entry.entry_id,
|
||||
|
||||
@@ -6,10 +6,13 @@ row state. The handler is the *only* cascade-side piece that knows
|
||||
the per-kind row shape; everything around it (watcher / scanner /
|
||||
worker / orchestrator / CLI) is kind-agnostic.
|
||||
|
||||
Per-kind handlers share the same dependencies bundle — embedding,
|
||||
tokenizer, memory-root path resolver — packaged as
|
||||
:class:`HandlerDeps`. Construct it once at orchestrator startup, pass
|
||||
to every handler factory, no per-row resolution churn.
|
||||
Per-kind handlers share the same dependencies bundle — tokenizer,
|
||||
memory-root path resolver — packaged as :class:`HandlerDeps`.
|
||||
Construct it once at orchestrator startup, pass to every handler
|
||||
factory, no per-row resolution churn. Embedding is a soft dependency:
|
||||
handlers fetch it themselves via
|
||||
:func:`everos.component.embedding.get_embedding_capability` rather
|
||||
than receiving it here.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -17,7 +20,6 @@ from __future__ import annotations
|
||||
import abc
|
||||
import dataclasses
|
||||
|
||||
from everos.component.embedding import EmbeddingProvider
|
||||
from everos.component.tokenizer import Tokenizer
|
||||
from everos.core.persistence import MemoryRoot
|
||||
|
||||
@@ -35,7 +37,6 @@ class HandlerDeps:
|
||||
"""
|
||||
|
||||
memory_root: MemoryRoot
|
||||
embedder: EmbeddingProvider
|
||||
tokenizer: Tokenizer
|
||||
|
||||
|
||||
|
||||
@@ -26,17 +26,25 @@ writers must match):
|
||||
- ``Summary`` (optional): condensed narrative.
|
||||
- ``Content``: full episode narrative — fed to the embedder AND the
|
||||
tokenizer for the ``episode_tokens`` BM25 field.
|
||||
|
||||
Embedding is a soft dependency: when unavailable, both ``vector`` and
|
||||
``subject_vector`` are written as ``None`` and the row stays
|
||||
BM25/scalar-searchable only.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
|
||||
from everos.component.embedding import get_embedding_capability
|
||||
from everos.core.observability.logging import get_logger
|
||||
from everos.infra.persistence.lancedb import Episode, ParentType, episode_repo
|
||||
|
||||
from ._common import parse_inline_list, require_iso_timestamp
|
||||
from ._daily_log_base import BaseDailyLogHandler, ParsedEntry
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class EpisodeHandler(BaseDailyLogHandler):
|
||||
"""Cascade handler for ``users/<u>/episodes/episode-*.md``."""
|
||||
@@ -72,14 +80,22 @@ class EpisodeHandler(BaseDailyLogHandler):
|
||||
subject_text = s.sections.get("Subject", "").strip()
|
||||
|
||||
# Embed content and subject concurrently; skip subject embed when absent.
|
||||
capability = get_embedding_capability()
|
||||
if subject_text:
|
||||
vector, subject_vector = await asyncio.gather(
|
||||
self._deps.embedder.embed(text),
|
||||
self._deps.embedder.embed(subject_text),
|
||||
capability.embed_or_none(text),
|
||||
capability.embed_or_none(subject_text),
|
||||
)
|
||||
else:
|
||||
vector = await self._deps.embedder.embed(text)
|
||||
vector = await capability.embed_or_none(text)
|
||||
subject_vector = None
|
||||
if vector is None:
|
||||
logger.debug(
|
||||
"cascade_handler_embed_skipped",
|
||||
kind=self.kind,
|
||||
entry_id=entry.entry_id,
|
||||
reason="embedding_capability_unavailable",
|
||||
)
|
||||
|
||||
# BM25 tokenization covers both body and subject keywords.
|
||||
tokenize_source = f"{text} {subject_text}" if subject_text else text
|
||||
|
||||
@@ -21,11 +21,15 @@ md contract:
|
||||
``sections``:
|
||||
|
||||
- ``Foresight``: forward-looking inference text (embedded + BM25).
|
||||
Embedding is a soft dependency: when unavailable, ``vector`` is
|
||||
written as ``None`` and the row stays BM25/scalar-searchable only.
|
||||
- ``Evidence`` (optional): supporting excerpt (secondary BM25 only).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from everos.component.embedding import get_embedding_capability
|
||||
from everos.core.observability.logging import get_logger
|
||||
from everos.infra.persistence.lancedb import Foresight, ParentType, foresight_repo
|
||||
|
||||
from ._common import (
|
||||
@@ -36,6 +40,8 @@ from ._common import (
|
||||
)
|
||||
from ._daily_log_base import BaseDailyLogHandler, ParsedEntry
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class ForesightHandler(BaseDailyLogHandler):
|
||||
"""Cascade handler for ``users/<u>/.foresights/foresight-*.md``."""
|
||||
@@ -68,7 +74,14 @@ class ForesightHandler(BaseDailyLogHandler):
|
||||
text = s.sections.get("Foresight", "").strip()
|
||||
evidence = (s.sections.get("Evidence") or "").strip() or None
|
||||
tokens = self._deps.tokenizer.tokenize(text)
|
||||
vector = await self._deps.embedder.embed(text)
|
||||
vector = await get_embedding_capability().embed_or_none(text)
|
||||
if vector is None:
|
||||
logger.debug(
|
||||
"cascade_handler_embed_skipped",
|
||||
kind=self.kind,
|
||||
entry_id=entry.entry_id,
|
||||
reason="embedding_capability_unavailable",
|
||||
)
|
||||
evidence_tokens = (
|
||||
" ".join(self._deps.tokenizer.tokenize(evidence)) if evidence else None
|
||||
)
|
||||
|
||||
@@ -25,6 +25,8 @@ structure fields are excluded — they change on re-parse without
|
||||
semantic drift.
|
||||
|
||||
Embedding source: ``summary`` (mirrors the search recaller's anchor).
|
||||
Embedding is a soft dependency: when unavailable, ``vector`` is
|
||||
written as ``None`` and the row stays BM25/scalar-searchable only.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -32,7 +34,9 @@ from __future__ import annotations
|
||||
import json
|
||||
from typing import Any, ClassVar
|
||||
|
||||
from everos.component.embedding import get_embedding_capability
|
||||
from everos.component.utils.datetime import get_utc_now
|
||||
from everos.core.observability.logging import get_logger
|
||||
from everos.core.persistence import MarkdownReader, ParsedMarkdown
|
||||
from everos.infra.persistence.lancedb import KnowledgeTopic, knowledge_topic_repo
|
||||
from everos.infra.persistence.sqlite import (
|
||||
@@ -45,6 +49,8 @@ from ._common import content_sha256 as compute_content_sha256
|
||||
from ._common import resolve_scope
|
||||
from .base import Handler
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class KnowledgeTopicHandler(Handler):
|
||||
"""Cascade handler for
|
||||
@@ -191,7 +197,14 @@ class KnowledgeTopicHandler(Handler):
|
||||
content_tokens = " ".join(
|
||||
self._deps.tokenizer.tokenize(fields["content"]),
|
||||
)
|
||||
vector = await self._deps.embedder.embed(fields["summary"])
|
||||
vector = await get_embedding_capability().embed_or_none(fields["summary"])
|
||||
if vector is None:
|
||||
logger.debug(
|
||||
"cascade_handler_embed_skipped",
|
||||
kind=self.kind,
|
||||
entry_id=fields["node_id"],
|
||||
reason="embedding_capability_unavailable",
|
||||
)
|
||||
now = get_utc_now()
|
||||
return KnowledgeTopic(
|
||||
id=fields["node_id"],
|
||||
|
||||
@@ -5,9 +5,11 @@ constructs it at startup, calls :meth:`start` once, and calls
|
||||
:meth:`stop` at shutdown. CLI ``cascade sync`` constructs its own
|
||||
instance but only invokes :meth:`drain_once` (no background tasks).
|
||||
|
||||
Construction is dependency-injected: the embedding / tokenizer
|
||||
providers and the memory-root come in as constructor args so tests
|
||||
can swap them without monkey-patching module-level singletons.
|
||||
Construction is dependency-injected: the tokenizer provider and the
|
||||
memory-root come in as constructor args so tests can swap them
|
||||
without monkey-patching module-level singletons. Embedding is a soft
|
||||
dependency handled directly by handlers via the capability accessor,
|
||||
not threaded through here.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -15,7 +17,6 @@ from __future__ import annotations
|
||||
import asyncio
|
||||
import dataclasses
|
||||
|
||||
from everos.component.embedding import EmbeddingProvider
|
||||
from everos.component.tokenizer import Tokenizer
|
||||
from everos.core.observability.logging import get_logger
|
||||
from everos.core.persistence import MemoryRoot
|
||||
@@ -53,7 +54,6 @@ class CascadeOrchestrator:
|
||||
self,
|
||||
*,
|
||||
memory_root: MemoryRoot,
|
||||
embedder: EmbeddingProvider,
|
||||
tokenizer: Tokenizer,
|
||||
config: CascadeConfig | None = None,
|
||||
) -> None:
|
||||
@@ -61,7 +61,6 @@ class CascadeOrchestrator:
|
||||
self._config = config or CascadeConfig()
|
||||
deps = HandlerDeps(
|
||||
memory_root=memory_root,
|
||||
embedder=embedder,
|
||||
tokenizer=tokenizer,
|
||||
)
|
||||
self._handlers = build_handlers(deps)
|
||||
@@ -113,14 +112,31 @@ class CascadeOrchestrator:
|
||||
self._started = False
|
||||
logger.info("cascade_orchestrator_stopped")
|
||||
|
||||
async def sync_once(self) -> int:
|
||||
"""One scan + drain cycle (used by CLI ``cascade sync``).
|
||||
async def sync_once(self, *, kinds: set[str] | None = None) -> int:
|
||||
"""One scan + drain cycle (used by CLI ``cascade sync`` and
|
||||
Phase-3 backfill's post-write skill-file sync).
|
||||
|
||||
Args:
|
||||
kinds: Optional restriction on which md kinds the scan
|
||||
walks (e.g. ``{"agent_skill"}``). ``None`` — the default
|
||||
for CLI ``cascade sync`` — scans every registered kind.
|
||||
Phase 3 uses ``{"agent_skill"}`` so an unscoped sweep
|
||||
doesn't tag unrelated kinds (notably ``knowledge_*``,
|
||||
which the current process may not have handlers for)
|
||||
as permanently failed.
|
||||
|
||||
Returns the number of rows processed in this drain. The CLI
|
||||
loops on the returned count to know when to stop.
|
||||
|
||||
The ``kinds`` filter is threaded through **both** the scanner
|
||||
AND the worker drain: the scanner scopes what gets enqueued,
|
||||
the worker scopes what gets claimed. Round-1 wired only the
|
||||
scanner side, so a scoped Phase-3 sync could still drain a
|
||||
knowledge md queued in a prior tick and mark it permanently
|
||||
failed — round-2 closes that end.
|
||||
"""
|
||||
await self._scanner.scan_once()
|
||||
return await self._worker.drain_until_empty()
|
||||
await self._scanner.scan_once(kinds=kinds)
|
||||
return await self._worker.drain_until_empty(kinds=kinds)
|
||||
|
||||
async def drain_once(self) -> int:
|
||||
"""Drain the queue exactly once without scanning first."""
|
||||
|
||||
@@ -173,5 +173,24 @@ def build_handlers(deps: HandlerDeps) -> dict[str, Handler]:
|
||||
Returns a ``{kind_name: Handler}`` map used by the worker for
|
||||
dispatch. Constructing once at orchestrator startup keeps the
|
||||
per-row hot path free of factory churn.
|
||||
|
||||
Every handler registers unconditionally. Capability-dependent
|
||||
steps live inside the handlers themselves:
|
||||
|
||||
- :class:`KnowledgeDocumentHandler` is SQLite-only; no capability
|
||||
dependency.
|
||||
- :class:`KnowledgeTopicHandler.handle_added_or_modified` calls
|
||||
:func:`embed_or_none`, which writes ``vector=None`` when the
|
||||
embedding capability is unavailable — the column has been
|
||||
nullable since the embedding-soft-dependency migration.
|
||||
``handle_deleted`` is pure SQL/LanceDB delete.
|
||||
|
||||
Prior versions gated the two knowledge handlers off as an atomic
|
||||
pair when embed OR rerank was unavailable — that gate broke the
|
||||
delete path (worker marks the row failed with "no handler
|
||||
registered", stranding SQLite / LanceDB rows after
|
||||
``shutil.rmtree`` cleared the md). Search-side gating is
|
||||
enforced separately at the route level, so cascade does not need
|
||||
to gate writes.
|
||||
"""
|
||||
return {spec.name: spec.handler_factory(deps) for spec in KIND_REGISTRY}
|
||||
|
||||
@@ -72,17 +72,32 @@ class CascadeScanner:
|
||||
self._task = None
|
||||
logger.info("cascade_scanner_stopped")
|
||||
|
||||
async def scan_once(self) -> list[ReconcileDecision]:
|
||||
async def scan_once(
|
||||
self, *, kinds: set[str] | None = None
|
||||
) -> list[ReconcileDecision]:
|
||||
"""One scan + reconcile pass; returns the decisions that were
|
||||
upserted into :class:`MdChangeState`.
|
||||
|
||||
Exposed so the CLI ``cascade sync`` command can trigger a sweep
|
||||
without owning a long-lived scanner task.
|
||||
|
||||
Args:
|
||||
kinds: Restrict the walk to specific md kinds (e.g.
|
||||
``{"agent_skill"}``). ``None`` — the default — walks
|
||||
every kind in :data:`KIND_REGISTRY`, matching the
|
||||
background scanner loop's behaviour.
|
||||
"""
|
||||
scan_inputs = await asyncio.to_thread(
|
||||
_collect_scan_inputs, self._memory_root.root
|
||||
_collect_scan_inputs, self._memory_root.root, kinds
|
||||
)
|
||||
state = await _load_state_snapshot()
|
||||
if kinds is not None:
|
||||
# reconcile() emits ``deleted`` for every state row missing
|
||||
# from the scan input — with a kind-scoped scan, the state
|
||||
# snapshot must be scoped too, or non-``kinds`` rows would
|
||||
# get spuriously deleted just because we didn't look at
|
||||
# them this sweep.
|
||||
state = {p: s for p, s in state.items() if s.kind in kinds}
|
||||
decisions = reconcile(scan_inputs, state)
|
||||
for decision in decisions:
|
||||
await md_change_state_repo.upsert(
|
||||
@@ -113,9 +128,18 @@ class CascadeScanner:
|
||||
continue
|
||||
|
||||
|
||||
def _collect_scan_inputs(root: Path) -> list[ScanInput]:
|
||||
def _collect_scan_inputs(root: Path, kinds: set[str] | None = None) -> list[ScanInput]:
|
||||
"""Walk ``root`` once per registered kind, returning every match.
|
||||
|
||||
``kinds`` — when non-``None`` — restricts the walk to specs whose
|
||||
``name`` appears in the set; other kinds' md files are neither
|
||||
stat'd nor emitted. Used by kind-scoped sync callers (e.g. Phase-3
|
||||
backfill) so their sweep doesn't touch queue state for unrelated
|
||||
kinds. The paired state-snapshot filter lives in :meth:`scan_once`
|
||||
(reconcile itself is kind-agnostic) — both sides must be scoped or
|
||||
reconcile would emit spurious ``deleted`` decisions for rows the
|
||||
scan simply chose not to look at.
|
||||
|
||||
``stat()`` failure mode discrimination is **load-bearing**: the
|
||||
reconciler treats "in state but not in scan" as a deletion signal,
|
||||
so if we silently drop a path here under a *transient* OS error,
|
||||
@@ -142,6 +166,8 @@ def _collect_scan_inputs(root: Path) -> list[ScanInput]:
|
||||
"""
|
||||
inputs: list[ScanInput] = []
|
||||
for spec in KIND_REGISTRY:
|
||||
if kinds is not None and spec.name not in kinds:
|
||||
continue
|
||||
for absolute in root.glob(spec.path_glob()):
|
||||
try:
|
||||
mtime = absolute.stat().st_mtime
|
||||
|
||||
@@ -260,7 +260,7 @@ class CascadeWorker:
|
||||
self._rebuild_task = None
|
||||
logger.info("cascade_worker_stopped")
|
||||
|
||||
async def drain_once(self) -> int:
|
||||
async def drain_once(self, *, kinds: set[str] | None = None) -> int:
|
||||
"""Process one batch, return the number of rows handled.
|
||||
|
||||
Used by CLI ``cascade sync`` and ``fix --apply`` to flush the
|
||||
@@ -276,8 +276,19 @@ class CascadeWorker:
|
||||
merged before returning (CLI ``cascade sync``) call
|
||||
:meth:`_flush_optimizers` — :meth:`drain_until_empty` does this
|
||||
on their behalf.
|
||||
|
||||
Args:
|
||||
kinds: Optional restriction on which ``kind`` values are
|
||||
claimed. ``None`` (the default) claims across every
|
||||
kind. Phase-3 backfill passes ``{"agent_skill"}`` so
|
||||
the drain cannot flip an unrelated queued kind (e.g. a
|
||||
queued ``knowledge_topic`` md whose handler this
|
||||
process doesn't have registered) to
|
||||
``failed(retryable=False)``.
|
||||
"""
|
||||
batch = await md_change_state_repo.claim_pending_batch(self._batch_size)
|
||||
batch = await md_change_state_repo.claim_pending_batch(
|
||||
self._batch_size, kinds=kinds
|
||||
)
|
||||
if not batch:
|
||||
return 0
|
||||
results = await asyncio.gather(*(self._process_one(row) for row in batch))
|
||||
@@ -286,7 +297,12 @@ class CascadeWorker:
|
||||
self._schedule_optimize(kind)
|
||||
return len(batch)
|
||||
|
||||
async def drain_until_empty(self, *, max_passes: int = 100) -> int:
|
||||
async def drain_until_empty(
|
||||
self,
|
||||
*,
|
||||
max_passes: int = 100,
|
||||
kinds: set[str] | None = None,
|
||||
) -> int:
|
||||
"""Drain repeatedly until the queue is empty (or ``max_passes``).
|
||||
|
||||
Returns the total number of rows processed. Bounded passes
|
||||
@@ -298,10 +314,15 @@ class CascadeWorker:
|
||||
(CLI ``cascade sync``) get a fully merged index — not for
|
||||
visibility (the data is already searchable) but so ``sync``
|
||||
returns a deterministically optimized state.
|
||||
|
||||
The ``kinds`` filter is forwarded to :meth:`drain_once` on every
|
||||
pass so the scoping intent holds end-to-end for scoped syncs
|
||||
(Phase-3 backfill's ``{"agent_skill"}`` sweep); ``None`` keeps
|
||||
the CLI ``cascade sync`` path drainining every kind.
|
||||
"""
|
||||
total = 0
|
||||
for _ in range(max_passes):
|
||||
processed = await self.drain_once()
|
||||
processed = await self.drain_once(kinds=kinds)
|
||||
if processed == 0:
|
||||
break
|
||||
total += processed
|
||||
|
||||
@@ -114,6 +114,15 @@ class ReflectionOrchestrator:
|
||||
Returns:
|
||||
List of successful ReflectionReport rows (typed as object
|
||||
because the table class lives in infra).
|
||||
|
||||
Note:
|
||||
No embedding-availability guard here — upstream callers are
|
||||
responsible for skipping this run when the capability is
|
||||
missing. In production the ``reflect_episodes`` strategy
|
||||
body-guards on ``get_embedding_capability().available`` and
|
||||
then constructs the orchestrator via ``.require()`` (which
|
||||
raises before ``run()`` can be called if the provider is
|
||||
missing), so an in-body check would be unreachable.
|
||||
"""
|
||||
candidates = await self._select_candidates(
|
||||
owner_id=owner_id,
|
||||
@@ -1013,7 +1022,7 @@ class ReflectionOrchestrator:
|
||||
if is_deprecated and ep.md_path:
|
||||
path_to_entries[ep.md_path][ep.entry_id] = merged_entry_id
|
||||
|
||||
root = MemoryRoot.default().root
|
||||
root = MemoryRoot.resolve().root
|
||||
for md_path, deprecated_map in path_to_entries.items():
|
||||
await self._episode_writer.patch_frontmatter(
|
||||
root / md_path,
|
||||
|
||||
@@ -36,9 +36,12 @@ from everalgo.rank import DEFAULT_RANK_CONFIG, RankConfig, arank
|
||||
from everalgo.rank.fusion import rrf
|
||||
from everalgo.types import Candidate, RankInput
|
||||
|
||||
from everos.component.embedding import get_embedding_capability
|
||||
from everos.component.rerank import get_rerank_capability
|
||||
from everos.component.utils.datetime import to_display_tz
|
||||
from everos.config import load_settings
|
||||
from everos.core.context import resolve_request_id
|
||||
from everos.core.errors import ProviderNotConfiguredError
|
||||
from everos.core.observability.logging import get_logger
|
||||
from everos.core.observability.tracing import (
|
||||
capture_input,
|
||||
@@ -755,51 +758,103 @@ class SearchManager:
|
||||
# ── Component guards ────────────────────────────────────────────
|
||||
|
||||
def _validate_components(self, req: SearchRequest) -> None:
|
||||
"""Fail fast when the chosen method needs components that are missing."""
|
||||
"""Fail fast when the chosen method needs a provider that is missing.
|
||||
|
||||
Every provider branch (embedding, rerank, LLM) raises the same
|
||||
:class:`ProviderNotConfiguredError` -> HTTP 422 with a toml-pointing
|
||||
message; vocabulary is uniform across the three.
|
||||
|
||||
Embedding and rerank checks read BOTH the process-wide capability
|
||||
singletons AND this manager's own injected providers. Either side
|
||||
being "missing" is enough to fail the guard: accessor
|
||||
says-unavailable means the tier never had it; ``self._embedding``
|
||||
is ``None`` means whoever built this manager didn't wire one in.
|
||||
Because dispatch reads ``self._embedding`` directly, letting an
|
||||
accessor-says-available-but-self-is-None case through would
|
||||
produce an opaque ``AttributeError('NoneType')`` deep in recall
|
||||
instead of a clean 422. Single source of truth = both must agree
|
||||
the provider exists.
|
||||
|
||||
Agent HYBRID has two rerank lanes selected by ``enable_llm_rerank``:
|
||||
the LLM lane (``True``) reranks via the LLM and needs no rerank
|
||||
provider; the cross-encoder lane (``False``, default) does.
|
||||
|
||||
Design contract (PR #361 round-3 review #6): this manager is
|
||||
instantiated by ``service.search`` — a single caller that always
|
||||
sources its providers from the same capability accessors this
|
||||
guard reads. On that path DI-injected provider and accessor
|
||||
state cannot disagree, so the belt-and-suspenders check is a
|
||||
near-no-op; it only fires on pathological drift (a subclass or
|
||||
a test that constructs a manager with mismatched inputs).
|
||||
Library consumers that embed :class:`SearchManager` directly
|
||||
and hand it their OWN provider will hit 422 even when their
|
||||
provider is valid, because the accessor reflects the
|
||||
process-wide toml. This is an accepted limitation of the
|
||||
current DI shape — the guard prefers false-positives
|
||||
(over-refuse) to false-negatives (dispatch with a ``None``
|
||||
provider) because the latter surfaces as an opaque
|
||||
``AttributeError('NoneType')`` deep in recall.
|
||||
"""
|
||||
method = req.method
|
||||
needs_embedding = method != SearchMethod.KEYWORD
|
||||
if needs_embedding and self._embedding is None:
|
||||
raise RuntimeError(
|
||||
f"method={method.value!r} requires an embedding provider; "
|
||||
"configure [embedding] in settings"
|
||||
)
|
||||
# LLM is only mandatory when the caller explicitly opts into
|
||||
# Phase-5 rerank on HYBRID, or always for AGENTIC (sufficiency
|
||||
# check + multi-query generation).
|
||||
if (
|
||||
method == SearchMethod.HYBRID
|
||||
and req.enable_llm_rerank
|
||||
and self._llm is None
|
||||
is_agent_hybrid = method == SearchMethod.HYBRID and req.owner_type == "agent"
|
||||
|
||||
needs_embedding = method in (
|
||||
SearchMethod.VECTOR,
|
||||
SearchMethod.HYBRID,
|
||||
SearchMethod.AGENTIC,
|
||||
)
|
||||
if needs_embedding and (
|
||||
not get_embedding_capability().available or self._embedding is None
|
||||
):
|
||||
raise RuntimeError(
|
||||
"method='hybrid' with enable_llm_rerank=true needs an LLM; "
|
||||
"configure [llm] in settings or drop the flag"
|
||||
raise ProviderNotConfiguredError(
|
||||
provider="embedding",
|
||||
feature=_feature_name(method, req.owner_type),
|
||||
)
|
||||
# agent_skill HYBRID without LLM rerank reaches the cross-encoder
|
||||
# lane; without the reranker it would AttributeError deep in the
|
||||
# callback. Episode / agent_case HYBRID don't need it.
|
||||
|
||||
# agent HYBRID cross-encoder lane (enable_llm_rerank=False, the
|
||||
# default) reaches ``search_agent_skills_hybrid``, which needs a
|
||||
# real rerank provider; the LLM lane reranks via the LLM instead
|
||||
# and is exempt.
|
||||
if (
|
||||
method == SearchMethod.HYBRID
|
||||
and req.owner_type == "agent"
|
||||
is_agent_hybrid
|
||||
and not req.enable_llm_rerank
|
||||
and self._reranker is None
|
||||
and (not get_rerank_capability().available or self._reranker is None)
|
||||
):
|
||||
raise RuntimeError(
|
||||
"owner_type='agent' with method='hybrid' requires a rerank "
|
||||
"provider (skill cross-encoder lane); configure [rerank] in "
|
||||
"settings, or set enable_llm_rerank=true to use the LLM lane"
|
||||
raise ProviderNotConfiguredError(
|
||||
provider="rerank",
|
||||
feature="agent_hybrid",
|
||||
alternative_hint=(
|
||||
"Set enable_llm_rerank=true to route through the "
|
||||
"LLM rerank lane instead of the cross-encoder "
|
||||
"provider."
|
||||
),
|
||||
)
|
||||
|
||||
if method == SearchMethod.AGENTIC and (
|
||||
not get_rerank_capability().available or self._reranker is None
|
||||
):
|
||||
raise ProviderNotConfiguredError(
|
||||
provider="rerank", feature="agentic_search"
|
||||
)
|
||||
|
||||
# LLM has no Capability wrapper yet; guard on the injected client
|
||||
# directly. Belt-and-suspenders with the embedding/rerank branches
|
||||
# above -- same 422 vocabulary via ProviderNotConfiguredError.
|
||||
is_hybrid_llm_lane = method == SearchMethod.HYBRID and req.enable_llm_rerank
|
||||
if is_hybrid_llm_lane and self._llm is None:
|
||||
raise ProviderNotConfiguredError(
|
||||
provider="llm",
|
||||
feature=_feature_name(method, req.owner_type),
|
||||
alternative_hint=(
|
||||
"drop the enable_llm_rerank flag to use the cross-encoder "
|
||||
"rerank lane (needs a configured [rerank] provider)"
|
||||
),
|
||||
)
|
||||
if method == SearchMethod.AGENTIC and self._llm is None:
|
||||
raise ProviderNotConfiguredError(
|
||||
provider="llm",
|
||||
feature=_feature_name(method, req.owner_type),
|
||||
)
|
||||
if method == SearchMethod.AGENTIC:
|
||||
if self._reranker is None:
|
||||
raise RuntimeError(
|
||||
"method='agentic' requires a rerank provider; "
|
||||
"configure [rerank] in settings"
|
||||
)
|
||||
if self._llm is None:
|
||||
raise RuntimeError(
|
||||
"method='agentic' requires an LLM client; "
|
||||
"configure [llm] in settings"
|
||||
)
|
||||
|
||||
|
||||
def _scored_as_candidate(scored) -> Candidate: # type: ignore[no-untyped-def]
|
||||
@@ -816,6 +871,27 @@ def _scored_as_candidate(scored) -> Candidate: # type: ignore[no-untyped-def]
|
||||
)
|
||||
|
||||
|
||||
def _feature_name(method: SearchMethod, owner_type: str) -> str:
|
||||
"""Map a search method (+ owner partition) to its 422 ``feature`` tag.
|
||||
|
||||
HYBRID splits by ``owner_type`` because the episode path (embed only)
|
||||
and the agent path (embed, plus a rerank lane) have different
|
||||
prerequisites and deserve distinct log/response tags.
|
||||
|
||||
Note: this vocabulary (``"vector"``, ``"user_hybrid"``, ``"agent_hybrid"``,
|
||||
``"agentic_search"``) is separate from the capability-level vocabulary
|
||||
in :mod:`everos.component.capabilities`. See that module's docstring
|
||||
for the rationale and contract.
|
||||
"""
|
||||
if method == SearchMethod.VECTOR:
|
||||
return "vector"
|
||||
if method == SearchMethod.HYBRID:
|
||||
return "agent_hybrid" if owner_type == "agent" else "user_hybrid"
|
||||
if method == SearchMethod.AGENTIC:
|
||||
return "agentic_search"
|
||||
return method.value
|
||||
|
||||
|
||||
def _effective_llm_rerank(req: SearchRequest) -> bool:
|
||||
"""LLM Phase-5 rerank only kicks in for ``HYBRID`` and only when the
|
||||
caller opts in. ``AGENTIC`` runs its own cross-encoder rerank loop
|
||||
|
||||
@@ -44,7 +44,7 @@ _writer: AgentCaseWriter | None = None
|
||||
def _get_writer() -> AgentCaseWriter:
|
||||
global _writer
|
||||
if _writer is None:
|
||||
_writer = AgentCaseWriter(root=MemoryRoot.default())
|
||||
_writer = AgentCaseWriter(root=MemoryRoot.resolve())
|
||||
return _writer
|
||||
|
||||
|
||||
|
||||
@@ -39,11 +39,11 @@ from everalgo.types import AgentCase as AlgoAgentCase
|
||||
from everalgo.types import AgentSkill as AlgoAgentSkill
|
||||
|
||||
from everos.component.embedding import (
|
||||
EmbeddingNotConfiguredError,
|
||||
EmbeddingServiceError,
|
||||
get_embedder,
|
||||
get_embedding_capability,
|
||||
)
|
||||
from everos.component.llm import get_llm_client
|
||||
from everos.core.errors import ProviderNotConfiguredError
|
||||
from everos.core.observability.logging import get_logger
|
||||
from everos.core.persistence import MemoryRoot
|
||||
from everos.infra.ome.context import StrategyContext
|
||||
@@ -102,7 +102,7 @@ _writer: AgentSkillWriter | None = None
|
||||
def _get_writer() -> AgentSkillWriter:
|
||||
global _writer
|
||||
if _writer is None:
|
||||
_writer = AgentSkillWriter(root=MemoryRoot.default())
|
||||
_writer = AgentSkillWriter(root=MemoryRoot.resolve())
|
||||
return _writer
|
||||
|
||||
|
||||
@@ -113,6 +113,25 @@ def _get_writer() -> AgentSkillWriter:
|
||||
max_retries=3,
|
||||
)
|
||||
async def extract_agent_skill(event: SkillClusterUpdated, ctx: StrategyContext) -> None:
|
||||
# Body-guard: capability is checked here for defensive degradation.
|
||||
# Belt-and-suspenders even though the upstream
|
||||
# trigger_skill_clustering already gates on the same capability — a
|
||||
# direct emit of SkillClusterUpdated (tests, future features) should
|
||||
# still degrade cleanly without an owner lock or OME retry pressure.
|
||||
# Tier upgrades require a server restart; this guard is not a
|
||||
# hot-reload mechanism.
|
||||
#
|
||||
# ``debug`` level (not ``info``) is intentional; see the body-guard
|
||||
# in :func:`everos.memory.strategies.trigger_profile_clustering` for
|
||||
# the shared rationale (PR #361 round-3 review #8).
|
||||
if not get_embedding_capability().available:
|
||||
logger.debug(
|
||||
"strategy_gated_off_embedding_unavailable",
|
||||
strategy_name="extract_agent_skill",
|
||||
agent_id=event.agent_id,
|
||||
)
|
||||
return
|
||||
|
||||
# Serialise on agent_id: SKILL.md is addressed by (agent_id, skill_name)
|
||||
# — concurrent runs across different clusters of the same agent can
|
||||
# both decide to add the same skill_name and clobber the file. Different
|
||||
@@ -269,10 +288,16 @@ async def _resolve_query_vector(target: LanceAgentCase) -> list[float]:
|
||||
return list(target.vector)
|
||||
if not target.task_intent:
|
||||
return []
|
||||
# ``.require()`` is defensive: the strategy body-guard checks
|
||||
# ``.available`` before we reach this helper, so this branch will
|
||||
# not raise ``ProviderNotConfiguredError`` in normal operation. The
|
||||
# catch stays so a misconfiguration surfacing later (or a direct
|
||||
# unit-test call to this helper without the guard) still degrades
|
||||
# to ``[]`` instead of blowing up mid-strategy.
|
||||
try:
|
||||
embedder = get_embedder()
|
||||
embedder = get_embedding_capability().require()
|
||||
return list(await embedder.embed(target.task_intent))
|
||||
except (EmbeddingNotConfiguredError, EmbeddingServiceError) as exc:
|
||||
except (ProviderNotConfiguredError, EmbeddingServiceError) as exc:
|
||||
logger.warning(
|
||||
"agent_skill_query_embed_failed",
|
||||
case_entry_id=target.entry_id,
|
||||
|
||||
@@ -31,7 +31,7 @@ def _get_writer() -> AtomicFactWriter:
|
||||
"""Return the lazily-initialised AtomicFactWriter singleton."""
|
||||
global _writer
|
||||
if _writer is None:
|
||||
_writer = AtomicFactWriter(root=MemoryRoot.default())
|
||||
_writer = AtomicFactWriter(root=MemoryRoot.resolve())
|
||||
return _writer
|
||||
|
||||
|
||||
|
||||
@@ -38,7 +38,7 @@ _writer: ForesightWriter | None = None
|
||||
def _get_writer() -> ForesightWriter:
|
||||
global _writer
|
||||
if _writer is None:
|
||||
_writer = ForesightWriter(root=MemoryRoot.default())
|
||||
_writer = ForesightWriter(root=MemoryRoot.resolve())
|
||||
return _writer
|
||||
|
||||
|
||||
|
||||
@@ -1,15 +1,34 @@
|
||||
"""extract_user_profile strategy — synthesise the user's profile from clusters.
|
||||
|
||||
Listens to :class:`ProfileClusterUpdated` (fired after
|
||||
``trigger_profile_clustering`` assigns a memcell to a cluster), pulls
|
||||
the relevant memcells across all "fresh" clusters, and runs
|
||||
:class:`everalgo.user_memory.ProfileExtractor` to INIT / UPDATE the
|
||||
user's profile markdown.
|
||||
Dual-trigger strategy: profile extraction must run whether or not embedding
|
||||
is configured, so it listens on **two** events and gates itself so exactly
|
||||
one path fires per memcell (see :func:`_profile_applies`):
|
||||
|
||||
- **Cluster path** (Tier 2+, embedding available): fires on
|
||||
:class:`ProfileClusterUpdated`, emitted by ``trigger_profile_clustering``
|
||||
after it assigns a memcell to a cluster. Pulls the relevant memcells
|
||||
across all "fresh" clusters (:func:`_select_via_cluster`).
|
||||
- **Direct path** (Tier 1, no embedding): fires on :class:`EpisodeExtracted`
|
||||
directly. ``trigger_profile_clustering`` is always registered, but its
|
||||
per-dispatch body-guard returns early when
|
||||
:func:`get_embedding_capability` reports unavailable, so no
|
||||
``ProfileClusterUpdated`` is ever emitted in Tier 1 — this direct
|
||||
path is the sole route to profile extraction there. Pulls memcells
|
||||
via a scalar LanceDB timestamp filter, no embedding required
|
||||
(:func:`_select_via_timestamp`).
|
||||
|
||||
Both paths converge on the same LLM extraction + markdown persist tail of
|
||||
``extract_user_profile`` — only memcell *selection* differs.
|
||||
|
||||
Opensource parity (``mem_memorize.py`` Phase 2):
|
||||
|
||||
- **Throttle**: ``total_memcell_count % profile_extraction_interval == 0``;
|
||||
default interval = 1 (every memcell triggers a re-extraction).
|
||||
- **Throttle** (both paths): ``total_count % profile_extraction_interval
|
||||
== 0``; default interval = 1 (every memcell triggers a re-extraction).
|
||||
Applied at strategy entry so cluster and direct paths honor the same
|
||||
gate. ``total_count`` is ``sum(c.count for c in user_clusters)`` on the
|
||||
cluster path and :meth:`episode_repo.count_by_owner` on the direct
|
||||
path — both express "cumulative units of source-memory for this owner"
|
||||
and stay ~1:1 in the extract → memcell → episode → cluster pipeline.
|
||||
- **Target clusters**: every cluster whose ``last_ts`` is newer than the
|
||||
user's existing profile timestamp, plus the current cluster (so the
|
||||
freshly-arrived memcell is always counted even when its cluster's
|
||||
@@ -27,15 +46,18 @@ profile in a follow-up turn (each cluster gets re-evaluated on every
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from everalgo.clustering import Cluster as AlgoCluster
|
||||
from everalgo.types import MemCell as AlgoMemCell
|
||||
from everalgo.types import Profile as AlgoProfile
|
||||
from everalgo.user_memory import ProfileExtractor
|
||||
|
||||
from everos.component.embedding import get_embedding_capability
|
||||
from everos.component.llm import get_llm_client
|
||||
from everos.core.observability.logging import get_logger
|
||||
from everos.core.persistence import MemoryRoot
|
||||
from everos.infra.ome.context import StrategyContext
|
||||
from everos.infra.ome.decorator import offline_strategy
|
||||
from everos.infra.ome.events import BaseEvent
|
||||
from everos.infra.ome.triggers import Immediate
|
||||
from everos.infra.persistence.lancedb import episode_repo
|
||||
from everos.infra.persistence.markdown import (
|
||||
@@ -45,7 +67,7 @@ from everos.infra.persistence.markdown import (
|
||||
)
|
||||
from everos.infra.persistence.sqlite import cluster_repo, memcell_repo
|
||||
from everos.memory._partition_locks import get_partition_lock
|
||||
from everos.memory.events import ProfileClusterUpdated
|
||||
from everos.memory.events import EpisodeExtracted, ProfileClusterUpdated
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
@@ -66,39 +88,185 @@ _reader: ProfileReader | None = None
|
||||
def _get_writer() -> ProfileWriter:
|
||||
global _writer
|
||||
if _writer is None:
|
||||
_writer = ProfileWriter(root=MemoryRoot.default())
|
||||
_writer = ProfileWriter(root=MemoryRoot.resolve())
|
||||
return _writer
|
||||
|
||||
|
||||
def _get_reader() -> ProfileReader:
|
||||
global _reader
|
||||
if _reader is None:
|
||||
_reader = ProfileReader(root=MemoryRoot.default())
|
||||
_reader = ProfileReader(root=MemoryRoot.resolve())
|
||||
return _reader
|
||||
|
||||
|
||||
def _profile_applies(event: BaseEvent) -> bool:
|
||||
"""Route exactly one path per memcell to ``extract_user_profile``.
|
||||
|
||||
- :class:`ProfileClusterUpdated` always applies: it is only ever
|
||||
emitted by ``trigger_profile_clustering``, whose per-dispatch
|
||||
body-guard short-circuits (returns before emit) whenever
|
||||
:func:`get_embedding_capability` reports unavailable. Accepting
|
||||
this event unconditionally is therefore safe — it can only arrive
|
||||
when embedding is available, so it never overlaps with the direct
|
||||
path below.
|
||||
- :class:`EpisodeExtracted` applies only for pipeline-sourced
|
||||
episodes (not Reflection's merged episodes) while embedding is
|
||||
unavailable. Once embedding is available, ``trigger_profile_clustering``
|
||||
runs to completion and owns the same memcell via
|
||||
``ProfileClusterUpdated`` — so the direct path must stand down
|
||||
here to avoid a double extraction. This check is a live capability
|
||||
read (not a registration-time snapshot), so a Tier 3→1 downgrade
|
||||
mid-process is picked up on the next event.
|
||||
"""
|
||||
if isinstance(event, ProfileClusterUpdated):
|
||||
return True
|
||||
if isinstance(event, EpisodeExtracted):
|
||||
return event.source == "pipeline" and not get_embedding_capability().available
|
||||
return False
|
||||
|
||||
|
||||
async def _select_via_cluster(
|
||||
event: ProfileClusterUpdated,
|
||||
last_profile_ts: int,
|
||||
user_clusters: list[AlgoCluster],
|
||||
) -> list[str]:
|
||||
"""Cluster path (Tier 2+): resolve memcells via the user's cluster set.
|
||||
|
||||
The caller (:func:`extract_user_profile`) already fetched
|
||||
``user_clusters`` for the strategy-entry throttle; we take it as a
|
||||
parameter to avoid a redundant :meth:`cluster_repo.list_for_owner`
|
||||
round-trip. Behavior of the selection step itself is unchanged from
|
||||
the pre-dual-trigger implementation.
|
||||
"""
|
||||
# Pick clusters fresher than the existing profile (always include
|
||||
# the one we just updated).
|
||||
target_clusters = [
|
||||
c
|
||||
for c in user_clusters
|
||||
if c.last_ts > last_profile_ts or c.id == event.cluster_id
|
||||
]
|
||||
if not target_clusters:
|
||||
return []
|
||||
|
||||
# Resolve cluster members (episode entry_ids) → memcell_ids.
|
||||
# Cluster members store episode entry_ids. To reach the memcell
|
||||
# payloads we look up each episode's parent_id (= memcell_id)
|
||||
# in LanceDB, skipping merged episodes (parent_type=cluster)
|
||||
# whose source memcells were already processed before Reflection.
|
||||
entry_ids = [m for c in target_clusters for m in c.members]
|
||||
episodes = await episode_repo.find_by_owner_entries(
|
||||
event.owner_id,
|
||||
entry_ids,
|
||||
app_id=event.app_id,
|
||||
project_id=event.project_id,
|
||||
)
|
||||
return [
|
||||
ep.parent_id for ep in episodes if ep.parent_type == "memcell" and ep.parent_id
|
||||
]
|
||||
|
||||
|
||||
async def _select_via_timestamp(
|
||||
event: EpisodeExtracted, last_profile_ts: int
|
||||
) -> list[str]:
|
||||
"""Direct path (Tier 1): resolve memcells from the event + a LanceDB supplement.
|
||||
|
||||
Always includes the current event's ``memcell_id`` — it is the
|
||||
trigger for this run and its memcell is the one whose profile we
|
||||
care about. The LanceDB scan is a best-effort supplement that adds
|
||||
older memcells still eligible under ``last_profile_ts``; the cascade
|
||||
daemon may not have indexed the freshly-arrived row yet (it runs on
|
||||
its own schedule), so relying on LanceDB alone would miss the first
|
||||
memory of a fresh Tier-1 install entirely. See
|
||||
:class:`EpisodeExtracted` for the event-first contract that lets us
|
||||
dodge this race — it carries ``memcell_id`` precisely so downstream
|
||||
strategies do not need to poll LanceDB until the row appears.
|
||||
|
||||
Mirrors the cluster path's ``c.id == event.cluster_id`` fallback
|
||||
(see :func:`_select_via_cluster`): the currently-firing entity is
|
||||
always in the candidate set even when its timestamp is not strictly
|
||||
fresher than the last profile — this matters for bulk imports where
|
||||
every incoming memcell may carry a historical timestamp.
|
||||
|
||||
Order is unstable across runs (the return is a de-duplicated set
|
||||
materialised to a list); downstream ``memcell_repo.find_by_ids``
|
||||
reorders back to the caller's list order, so this does not
|
||||
destabilise later stages.
|
||||
"""
|
||||
memcell_ids: set[str] = {event.memcell_id}
|
||||
# Column projection: this selector only needs `parent_id` to seed the
|
||||
# memcell fetch — pulling full rows would drag the two 1024-D vector
|
||||
# columns across the wire for every historical episode. Raw-dict
|
||||
# return is contractual when `columns` is set.
|
||||
supplement = await episode_repo.list_by_owner_after_ts(
|
||||
owner_id=event.owner_id,
|
||||
after_ts=last_profile_ts,
|
||||
parent_type="memcell",
|
||||
app_id=event.app_id,
|
||||
project_id=event.project_id,
|
||||
columns=["parent_id"],
|
||||
)
|
||||
memcell_ids.update(row["parent_id"] for row in supplement if row.get("parent_id"))
|
||||
return list(memcell_ids)
|
||||
|
||||
|
||||
@offline_strategy(
|
||||
name="extract_user_profile",
|
||||
trigger=Immediate(on=[ProfileClusterUpdated]),
|
||||
trigger=Immediate(on=[ProfileClusterUpdated, EpisodeExtracted]),
|
||||
applies_to=_profile_applies,
|
||||
emits=[],
|
||||
max_retries=2,
|
||||
)
|
||||
async def extract_user_profile(
|
||||
event: ProfileClusterUpdated, ctx: StrategyContext
|
||||
event: ProfileClusterUpdated | EpisodeExtracted, ctx: StrategyContext
|
||||
) -> None:
|
||||
# Serialise on owner_id: user.md is a single per-user file and the
|
||||
# body is a read → LLM merge → overwrite sequence. Different users
|
||||
# run fully in parallel.
|
||||
partition = f"{event.app_id}:{event.project_id}:{event.owner_id}"
|
||||
async with get_partition_lock("extract_user_profile", partition):
|
||||
# 1. Throttle: skip unless the Nth clustered memcell tick lands here.
|
||||
user_clusters = await cluster_repo.list_for_owner(
|
||||
existing = await _get_reader().read(
|
||||
event.owner_id,
|
||||
"user_memory",
|
||||
schema=UserProfileFrontmatter,
|
||||
app_id=event.app_id,
|
||||
project_id=event.project_id,
|
||||
)
|
||||
total_count = sum(c.count for c in user_clusters)
|
||||
last_profile_ts = existing[0].profile_timestamp_ms if existing else 0
|
||||
|
||||
# Unified throttle: both paths gate on "cumulative units of
|
||||
# source-memory for this owner". Cluster path sums cluster
|
||||
# counts (pre-refactor semantics); direct path counts owner
|
||||
# episode rows in LanceDB. Applied here so a bumped interval
|
||||
# (e.g. 10x to cap LLM cost) throttles both Tier 1 and Tier 2+
|
||||
# equally instead of only the cluster path.
|
||||
user_clusters: list[AlgoCluster] | None
|
||||
if isinstance(event, ProfileClusterUpdated):
|
||||
user_clusters = await cluster_repo.list_for_owner(
|
||||
event.owner_id,
|
||||
"user_memory",
|
||||
app_id=event.app_id,
|
||||
project_id=event.project_id,
|
||||
)
|
||||
total_count = sum(c.count for c in user_clusters)
|
||||
else:
|
||||
user_clusters = None
|
||||
# Scope the counter to parent_type='memcell' so it matches
|
||||
# _select_via_timestamp's selector below — otherwise Reflection-
|
||||
# merged rows (parent_type='cluster') inflate the throttle count
|
||||
# without ever being selectable, firing the gate at the wrong cadence.
|
||||
# TODO(profile-counter): reads LanceDB and therefore races the
|
||||
# cascade daemon in the same way _select_via_timestamp used to
|
||||
# (fresh Tier-1 install → count=0 until cascade catches up).
|
||||
# The throttle only needs a monotonic per-owner integer, so a
|
||||
# stale-but-monotonic value is acceptable for now; the followup
|
||||
# is to source this from a sqlite ``memcell`` count-by-owner
|
||||
# query — see PR #361 review finding M4.
|
||||
total_count = await episode_repo.count_by_owner(
|
||||
event.owner_id,
|
||||
app_id=event.app_id,
|
||||
project_id=event.project_id,
|
||||
parent_type="memcell",
|
||||
)
|
||||
|
||||
if (
|
||||
PROFILE_EXTRACTION_INTERVAL > 1
|
||||
and total_count % PROFILE_EXTRACTION_INTERVAL != 0
|
||||
@@ -108,43 +276,24 @@ async def extract_user_profile(
|
||||
owner_id=event.owner_id,
|
||||
total_count=total_count,
|
||||
interval=PROFILE_EXTRACTION_INTERVAL,
|
||||
path="cluster"
|
||||
if isinstance(event, ProfileClusterUpdated)
|
||||
else "direct",
|
||||
)
|
||||
return
|
||||
|
||||
# 2. Pick clusters fresher than the existing profile (always include
|
||||
# the one we just updated).
|
||||
existing = await _get_reader().read(
|
||||
event.owner_id,
|
||||
schema=UserProfileFrontmatter,
|
||||
app_id=event.app_id,
|
||||
project_id=event.project_id,
|
||||
)
|
||||
last_profile_ts = existing[0].profile_timestamp_ms if existing else 0
|
||||
target_clusters = [
|
||||
c
|
||||
for c in user_clusters
|
||||
if c.last_ts > last_profile_ts or c.id == event.cluster_id
|
||||
]
|
||||
if not target_clusters:
|
||||
return
|
||||
# Memcell selection is the only part that differs by trigger; the
|
||||
# LLM extraction + persist tail below is shared by both paths.
|
||||
# user_clusters is always populated on the cluster branch above,
|
||||
# so the fallback to [] here is unreachable — it exists purely to
|
||||
# satisfy the type checker without an assert.
|
||||
if isinstance(event, ProfileClusterUpdated):
|
||||
memcell_ids = await _select_via_cluster(
|
||||
event, last_profile_ts, user_clusters or []
|
||||
)
|
||||
else:
|
||||
memcell_ids = await _select_via_timestamp(event, last_profile_ts)
|
||||
|
||||
# 3. Resolve cluster members (episode entry_ids) → memcell_ids.
|
||||
# Cluster members store episode entry_ids. To reach the memcell
|
||||
# payloads we look up each episode's parent_id (= memcell_id)
|
||||
# in LanceDB, skipping merged episodes (parent_type=cluster)
|
||||
# whose source memcells were already processed before Reflection.
|
||||
entry_ids = [m for c in target_clusters for m in c.members]
|
||||
episodes = await episode_repo.find_by_owner_entries(
|
||||
event.owner_id,
|
||||
entry_ids,
|
||||
app_id=event.app_id,
|
||||
project_id=event.project_id,
|
||||
)
|
||||
memcell_ids = [
|
||||
ep.parent_id
|
||||
for ep in episodes
|
||||
if ep.parent_type == "memcell" and ep.parent_id
|
||||
]
|
||||
if len(memcell_ids) < PROFILE_MIN_MEMCELLS:
|
||||
logger.info(
|
||||
"profile_extraction_below_min_memcells",
|
||||
@@ -154,7 +303,7 @@ async def extract_user_profile(
|
||||
)
|
||||
return
|
||||
|
||||
# 4. Pull memcell payloads from SQLite, rehydrate to algo types.
|
||||
# Pull memcell payloads from SQLite, rehydrate to algo types.
|
||||
memcell_rows = await memcell_repo.find_by_ids(memcell_ids)
|
||||
algo_memcells = sorted(
|
||||
(AlgoMemCell.model_validate_json(r.payload_json) for r in memcell_rows),
|
||||
@@ -163,14 +312,14 @@ async def extract_user_profile(
|
||||
if not algo_memcells:
|
||||
return
|
||||
|
||||
# 5. Run the LLM extractor — INIT (no prior) or UPDATE (existing).
|
||||
# Run the LLM extractor — INIT (no prior) or UPDATE (existing).
|
||||
old_profile = _to_algo_profile(existing[0]) if existing else None
|
||||
extractor = ProfileExtractor(llm=get_llm_client())
|
||||
new_profile = await extractor.aextract(
|
||||
algo_memcells, sender_id=event.owner_id, old_profile=old_profile
|
||||
)
|
||||
|
||||
# 6. Write the fresh profile back to users/<user_id>/user.md.
|
||||
# Write the fresh profile back to users/<user_id>/user.md.
|
||||
await _persist_profile(
|
||||
new_profile,
|
||||
owner_id=event.owner_id,
|
||||
@@ -180,8 +329,7 @@ async def extract_user_profile(
|
||||
logger.info(
|
||||
"user_profile_extracted",
|
||||
owner_id=event.owner_id,
|
||||
cluster_count=len(target_clusters),
|
||||
episode_count=len(episodes),
|
||||
path="cluster" if isinstance(event, ProfileClusterUpdated) else "direct",
|
||||
memcell_count=len(algo_memcells),
|
||||
mode="UPDATE" if old_profile is not None else "INIT",
|
||||
)
|
||||
|
||||
@@ -14,7 +14,7 @@ from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
|
||||
from everos.component.embedding import get_embedder
|
||||
from everos.component.embedding import get_embedding_capability
|
||||
from everos.component.llm import get_llm_client
|
||||
from everos.core.observability.logging import get_logger
|
||||
from everos.core.persistence import MemoryRoot
|
||||
@@ -43,7 +43,7 @@ def _get_episode_writer() -> EpisodeWriter:
|
||||
"""Return the lazily-initialised EpisodeWriter singleton."""
|
||||
global _episode_writer
|
||||
if _episode_writer is None:
|
||||
_episode_writer = EpisodeWriter(root=MemoryRoot.default())
|
||||
_episode_writer = EpisodeWriter(root=MemoryRoot.resolve())
|
||||
return _episode_writer
|
||||
|
||||
|
||||
@@ -61,6 +61,22 @@ async def reflect_episodes(event: CronTick, ctx: StrategyContext) -> None:
|
||||
event: Cron tick event (unused; triggers the scheduled run).
|
||||
ctx: OME strategy context for emit and logging.
|
||||
"""
|
||||
# Body-guard: capability is checked here for defensive degradation.
|
||||
# Reflection re-embeds merged narratives, so it cannot run without
|
||||
# an embedder — silently no-op instead of raising deep inside the
|
||||
# orchestrator. Tier upgrades require a server restart; this guard
|
||||
# is not a hot-reload mechanism.
|
||||
#
|
||||
# ``debug`` level (not ``info``) is intentional; see the body-guard
|
||||
# in :func:`everos.memory.strategies.trigger_profile_clustering` for
|
||||
# the shared rationale (PR #361 round-3 review #8).
|
||||
if not get_embedding_capability().available:
|
||||
logger.debug(
|
||||
"strategy_gated_off_embedding_unavailable",
|
||||
strategy_name="reflect_episodes",
|
||||
)
|
||||
return
|
||||
|
||||
# Deferred: avoid pulling LLM libs at module import time.
|
||||
from everalgo.user_memory import EpisodeReflector
|
||||
|
||||
@@ -71,7 +87,7 @@ async def reflect_episodes(event: CronTick, ctx: StrategyContext) -> None:
|
||||
episode_writer=_get_episode_writer(),
|
||||
report_repo=reflection_report_repo,
|
||||
reflector=EpisodeReflector(llm=get_llm_client()),
|
||||
embedder=get_embedder(),
|
||||
embedder=get_embedding_capability().require(),
|
||||
)
|
||||
|
||||
owners = await cluster_repo.list_distinct_owners()
|
||||
|
||||
@@ -14,7 +14,7 @@ import numpy as np
|
||||
from everalgo.clustering import Cluster as AlgoCluster
|
||||
from everalgo.clustering import cluster_by_geometry
|
||||
|
||||
from everos.component.embedding import get_embedder
|
||||
from everos.component.embedding import get_embedding_capability
|
||||
from everos.config import load_settings
|
||||
from everos.core.observability.logging import get_logger
|
||||
from everos.infra.ome.context import StrategyContext
|
||||
@@ -37,6 +37,36 @@ logger = get_logger(__name__)
|
||||
async def trigger_profile_clustering(
|
||||
event: EpisodeExtracted, ctx: StrategyContext
|
||||
) -> None:
|
||||
# Body-guard: capability is checked here for defensive degradation.
|
||||
# When embedding is unavailable we cannot vectorise the episode, so
|
||||
# the strategy silently no-ops — no work, no owner lock, no OME
|
||||
# retry pressure. Tier upgrades require a server restart; this
|
||||
# guard is not a hot-reload mechanism, it just keeps dispatch clean
|
||||
# when capability was absent at engine start.
|
||||
#
|
||||
# Log level is intentionally ``debug`` here (and at the three sibling
|
||||
# body-guards in ``trigger_skill_clustering``, ``extract_agent_skill``,
|
||||
# and ``reflect_episodes``) rather than the ``info`` used by
|
||||
# ``cascade.registry`` and the reflection orchestrator's
|
||||
# ``cascade_registry_knowledge_gated_off`` /
|
||||
# ``reflection_orchestrator_disabled`` events. Rationale (PR #361
|
||||
# round-3 review #8, accepted intentional asymmetry): the
|
||||
# registry/orchestrator gate events fire ONCE per process at startup
|
||||
# and are useful lifecycle signals worth ``info``. The per-dispatch
|
||||
# body-guards fire on EVERY memorize/reflection dispatch under Tier 1
|
||||
# — a busy chat session emits hundreds. At ``info`` they would flood
|
||||
# structured-log consumers even when the process-wide default level
|
||||
# is ``WARNING`` (which suppresses stdout but not the sink). ``debug``
|
||||
# keeps this per-event signal cheap and opt-in via ``--verbose`` for
|
||||
# diagnosing why a strategy isn't running.
|
||||
if not get_embedding_capability().available:
|
||||
logger.debug(
|
||||
"strategy_gated_off_embedding_unavailable",
|
||||
strategy_name="trigger_profile_clustering",
|
||||
owner_id=event.owner_id,
|
||||
)
|
||||
return
|
||||
|
||||
# Serialise on owner_id: the strategy reads the user's full cluster
|
||||
# set, picks merge target by geometry, then upserts — concurrent runs
|
||||
# on the same owner_id would race the read → decide → write cycle.
|
||||
@@ -46,7 +76,12 @@ async def trigger_profile_clustering(
|
||||
partition = f"{event.app_id}:{event.project_id}:{event.owner_id}"
|
||||
async with get_partition_lock("trigger_profile_clustering", partition):
|
||||
# 1. Embed the episode_text into a vector.
|
||||
vector_list = await get_embedder().embed(event.episode_text)
|
||||
# ``.require()`` is defensive: the body-guard above already
|
||||
# returned when the capability was missing, so this cannot raise
|
||||
# in the guarded path. Routing through the capability keeps a
|
||||
# single shared provider (one client, one semaphore) per process.
|
||||
embedder = get_embedding_capability().require()
|
||||
vector_list = await embedder.embed(event.episode_text)
|
||||
vector = np.asarray(vector_list, dtype=np.float32)
|
||||
|
||||
# 2. Load this user's existing user-memory clusters (scoped to space).
|
||||
|
||||
@@ -23,7 +23,7 @@ import numpy as np
|
||||
from everalgo.clustering import Cluster as AlgoCluster
|
||||
from everalgo.clustering import cluster_by_llm
|
||||
|
||||
from everos.component.embedding import get_embedder
|
||||
from everos.component.embedding import get_embedding_capability
|
||||
from everos.component.llm import get_llm_client
|
||||
from everos.core.observability.logging import get_logger
|
||||
from everos.infra.ome.context import StrategyContext
|
||||
@@ -49,6 +49,23 @@ cases below this never produce a skill, so we don't bother clustering them."""
|
||||
async def trigger_skill_clustering(
|
||||
event: AgentCaseExtracted, ctx: StrategyContext
|
||||
) -> None:
|
||||
# Body-guard: capability is checked here for defensive degradation.
|
||||
# When embedding is unavailable we cannot vectorise the case, so
|
||||
# the strategy silently no-ops — no work, no agent lock, no OME
|
||||
# retry pressure. Tier upgrades require a server restart; this
|
||||
# guard is not a hot-reload mechanism.
|
||||
#
|
||||
# ``debug`` level (not ``info``) is intentional; see the body-guard
|
||||
# in :func:`everos.memory.strategies.trigger_profile_clustering` for
|
||||
# the shared rationale (PR #361 round-3 review #8).
|
||||
if not get_embedding_capability().available:
|
||||
logger.debug(
|
||||
"strategy_gated_off_embedding_unavailable",
|
||||
strategy_name="trigger_skill_clustering",
|
||||
agent_id=event.agent_id,
|
||||
)
|
||||
return
|
||||
|
||||
# Serialise on agent_id: the strategy reads the agent's full cluster
|
||||
# set, lets the LLM decide merge vs. mint, then upserts — concurrent
|
||||
# runs on the same agent_id would race the read → decide → write
|
||||
@@ -67,7 +84,12 @@ async def trigger_skill_clustering(
|
||||
return
|
||||
|
||||
# 2. Embed the case's task_intent into a vector.
|
||||
vector_list = await get_embedder().embed(event.task_intent)
|
||||
# ``.require()`` is defensive: the body-guard above already
|
||||
# returned when the capability was missing, so this cannot raise
|
||||
# in the guarded path. Routing through the capability keeps a
|
||||
# single shared provider (one client, one semaphore) per process.
|
||||
embedder = get_embedding_capability().require()
|
||||
vector_list = await embedder.embed(event.task_intent)
|
||||
vector = np.asarray(vector_list, dtype=np.float32)
|
||||
|
||||
# 3. Load this agent's existing skill clusters (scoped to space).
|
||||
|
||||
@@ -29,13 +29,16 @@ import anyio
|
||||
from everalgo.rank.fusion import rrf
|
||||
from everalgo.types import Candidate, CategorySpec, KnowledgeMemory, ParsedContent
|
||||
|
||||
from everos.component.embedding import get_embedding_capability
|
||||
from everos.component.rerank import get_rerank_capability
|
||||
from everos.component.utils.datetime import get_utc_now
|
||||
from everos.core.errors import (
|
||||
ConfigurationError,
|
||||
DocumentNotFoundError,
|
||||
DuplicateDocumentError,
|
||||
ExtractionEmptyError,
|
||||
InvalidInputError,
|
||||
PathTraversalError,
|
||||
ProviderNotConfiguredError,
|
||||
TopicNotFoundError,
|
||||
)
|
||||
from everos.core.observability.logging import get_logger
|
||||
@@ -66,6 +69,7 @@ _DOC_ID_HEX_LEN = 12
|
||||
_MAX_MINT_RETRIES = 5
|
||||
_SCOPE_ID_PATTERN = re.compile(r"^[a-zA-Z0-9_.\-@+]+$")
|
||||
_ORIGINAL_DIR_NAME = "_original"
|
||||
_KNOWLEDGE_FEATURE = "knowledge"
|
||||
|
||||
|
||||
class KnowledgeExtractor(Protocol):
|
||||
@@ -519,7 +523,7 @@ async def delete_document(
|
||||
|
||||
topic_count = await knowledge_topic_sqlite_repo.count_by_doc_id(doc_id)
|
||||
|
||||
memory_root = MemoryRoot.default()
|
||||
memory_root = MemoryRoot.resolve()
|
||||
doc_dir = memory_root.root / Path(row.md_path).parent
|
||||
if await anyio.Path(doc_dir).is_dir():
|
||||
await anyio.to_thread.run_sync(shutil.rmtree, doc_dir)
|
||||
@@ -605,7 +609,7 @@ async def _backup_doc_dir(doc_id: str) -> tuple[Path, Path] | None:
|
||||
without reverse-engineering the name, or ``None`` when no directory
|
||||
exists.
|
||||
"""
|
||||
memory_root = MemoryRoot.default()
|
||||
memory_root = MemoryRoot.resolve()
|
||||
row = await knowledge_document_repo.get_by_doc_id(doc_id)
|
||||
if row is None:
|
||||
return None
|
||||
@@ -844,7 +848,7 @@ async def patch_document(
|
||||
Raises:
|
||||
DocumentNotFoundError: When neither SQLite nor md files contain ``doc_id``.
|
||||
"""
|
||||
memory_root = MemoryRoot.default()
|
||||
memory_root = MemoryRoot.resolve()
|
||||
current = await _resolve_current_doc(doc_id, app_id, project_id, memory_root)
|
||||
|
||||
new_title = title if title is not None else current.title
|
||||
@@ -883,7 +887,7 @@ async def list_categories(app_id: str, project_id: str) -> list[CategoryOverview
|
||||
Returns:
|
||||
List of CategoryOverview with document counts from SQLite.
|
||||
"""
|
||||
knowledge_dir = MemoryRoot.default().knowledge_dir(app_id, project_id)
|
||||
knowledge_dir = MemoryRoot.resolve().knowledge_dir(app_id, project_id)
|
||||
await ensure_taxonomy(knowledge_dir)
|
||||
specs = await parse_taxonomy(knowledge_dir / ".taxonomy.md")
|
||||
counts = await knowledge_document_repo.count_by_category(app_id, project_id)
|
||||
@@ -923,6 +927,18 @@ def _apply_category_fallback(
|
||||
return patched
|
||||
|
||||
|
||||
_MAX_FILENAME_BYTES = 255
|
||||
"""Portable filesystem filename byte-length ceiling.
|
||||
|
||||
macOS HFS+/APFS, most Linux filesystems (ext4/xfs/btrfs) and Windows
|
||||
NTFS all cap a single directory entry at 255 bytes of UTF-8. Rather
|
||||
than let the OS surface the ceiling as a raw ``OSError[ENAMETOOLONG]``
|
||||
at ``write_bytes`` time (which reaches the HTTP layer as a 500), we
|
||||
validate up front and raise a caller-friendly ``InvalidInputError``
|
||||
that maps to 400.
|
||||
"""
|
||||
|
||||
|
||||
def _safe_original_filename(source_name: str) -> str:
|
||||
"""Reduce an untrusted upload filename to a single safe path component.
|
||||
|
||||
@@ -936,6 +952,18 @@ def _safe_original_filename(source_name: str) -> str:
|
||||
``.`` / ``..`` / empty cases (which a basename can still be, e.g. a
|
||||
filename of ``..``) are rejected outright.
|
||||
|
||||
Also validates two failure modes the OS would otherwise raise as
|
||||
500 during ``write_bytes`` (round-4 review M1):
|
||||
|
||||
- **NUL byte** in the filename → ``ValueError: embedded null
|
||||
character`` from the syscall layer. Rejected up front as a
|
||||
400-mapped ``InvalidInputError``.
|
||||
- **Filename > 255 bytes UTF-8** → ``OSError[ENAMETOOLONG]`` from
|
||||
the syscall layer. Same up-front rejection.
|
||||
|
||||
Both are validated **before** ``KnowledgeWriter.write`` runs, so a
|
||||
doomed upload never leaves half-a-document on disk.
|
||||
|
||||
Args:
|
||||
source_name: The client-supplied multipart filename.
|
||||
|
||||
@@ -944,7 +972,13 @@ def _safe_original_filename(source_name: str) -> str:
|
||||
|
||||
Raises:
|
||||
PathTraversalError: If no safe filename can be derived.
|
||||
InvalidInputError: If the filename contains a NUL byte or
|
||||
exceeds the filesystem's byte-length ceiling.
|
||||
"""
|
||||
if "\x00" in source_name:
|
||||
raise InvalidInputError(
|
||||
f"upload filename must not contain a NUL byte (got {source_name!r})"
|
||||
)
|
||||
# PurePosixPath does not split on "\\"; strip backslash segments too so a
|
||||
# Windows-style ``..\\..\\x`` or ``C:\\x`` filename cannot survive.
|
||||
name = PurePosixPath(source_name).name.rsplit("\\", 1)[-1]
|
||||
@@ -952,6 +986,12 @@ def _safe_original_filename(source_name: str) -> str:
|
||||
raise PathTraversalError(
|
||||
f"upload filename is not a valid file component: {source_name!r}"
|
||||
)
|
||||
if len(name.encode("utf-8")) > _MAX_FILENAME_BYTES:
|
||||
raise InvalidInputError(
|
||||
"upload filename exceeds the "
|
||||
f"{_MAX_FILENAME_BYTES}-byte filesystem ceiling "
|
||||
f"(got {len(name.encode('utf-8'))} bytes)"
|
||||
)
|
||||
return name
|
||||
|
||||
|
||||
@@ -972,7 +1012,7 @@ async def _resolve_original_file_path(
|
||||
safe_name = _safe_original_filename(source_name)
|
||||
except PathTraversalError:
|
||||
return None
|
||||
memory_root = MemoryRoot.default()
|
||||
memory_root = MemoryRoot.resolve()
|
||||
doc_dir = memory_root.root / Path(md_path).parent
|
||||
candidate = doc_dir / _ORIGINAL_DIR_NAME / safe_name
|
||||
if await anyio.Path(candidate).is_file():
|
||||
@@ -1007,37 +1047,21 @@ async def _write_original_file(
|
||||
|
||||
# ── Knowledge search ─────────────────────────────────────────────────────────
|
||||
|
||||
# Lazy singleton — mirrors the pattern in service/search.py.
|
||||
_embedding: EmbeddingProvider | None = None
|
||||
_embedding_resolved = False
|
||||
|
||||
|
||||
def _get_embedding() -> EmbeddingProvider | None:
|
||||
"""Build the embedding client on first call. ``None`` when not configured."""
|
||||
global _embedding, _embedding_resolved
|
||||
if _embedding_resolved:
|
||||
return _embedding
|
||||
"""Return the process-wide embedding provider, or ``None`` when unset.
|
||||
|
||||
from everos.component.embedding import ( # Deferred: singleton
|
||||
build_embedding_provider,
|
||||
)
|
||||
from everos.config import load_settings # Deferred: singleton
|
||||
|
||||
cfg = load_settings().embedding
|
||||
if not cfg.model or cfg.api_key is None:
|
||||
logger.warning(
|
||||
"knowledge_embedding_not_configured",
|
||||
hint="set [embedding] model / api_key to enable vector / hybrid search",
|
||||
)
|
||||
_embedding = None
|
||||
else:
|
||||
_embedding = build_embedding_provider(cfg)
|
||||
logger.info("knowledge_embedding_built", model=cfg.model)
|
||||
_embedding_resolved = True
|
||||
return _embedding
|
||||
Delegates to :func:`get_embedding_capability` instead of keeping a
|
||||
parallel module-level singleton — mirrors ``service/search.py``, so
|
||||
this module answers the same "is embedding configured?" question as
|
||||
the health endpoint and the cascade write path.
|
||||
"""
|
||||
return get_embedding_capability().provider
|
||||
|
||||
|
||||
# Lazy singleton — mirrors the pattern for _embedding above.
|
||||
# Lazy singleton — the recaller itself has no soft-dependency provider to
|
||||
# delegate to (it wraps a tokenizer, not embedding/rerank), so it keeps its
|
||||
# own module-level cache rather than a capability accessor.
|
||||
_recaller: KnowledgeTopicRecaller | None = None
|
||||
_recaller_resolved = False
|
||||
|
||||
@@ -1061,30 +1085,12 @@ def _build_recaller() -> KnowledgeTopicRecaller:
|
||||
return _recaller
|
||||
|
||||
|
||||
# Lazy singleton — mirrors the pattern for _embedding above.
|
||||
_reranker: RerankProvider | None = None
|
||||
_reranker_resolved = False
|
||||
|
||||
|
||||
def _get_reranker() -> RerankProvider | None:
|
||||
"""Build the rerank client on first call. ``None`` when not configured."""
|
||||
global _reranker, _reranker_resolved
|
||||
if _reranker_resolved:
|
||||
return _reranker
|
||||
"""Return the process-wide rerank provider, or ``None`` when unset.
|
||||
|
||||
from everos.component.rerank import ( # Deferred: singleton
|
||||
build_rerank_provider,
|
||||
)
|
||||
from everos.config import load_settings # Deferred: singleton
|
||||
|
||||
cfg = load_settings().rerank
|
||||
if not cfg.model or not cfg.base_url:
|
||||
_reranker = None
|
||||
else:
|
||||
_reranker = build_rerank_provider(cfg)
|
||||
logger.info("knowledge_reranker_built", model=cfg.model, provider=cfg.provider)
|
||||
_reranker_resolved = True
|
||||
return _reranker
|
||||
Delegates to :func:`get_rerank_capability` — mirrors :func:`_get_embedding`.
|
||||
"""
|
||||
return get_rerank_capability().provider
|
||||
|
||||
|
||||
# ── Search result types ──────────────────────────────────────────────────────
|
||||
@@ -1261,23 +1267,25 @@ async def _to_search_hits(
|
||||
def _require_search_providers() -> tuple[EmbeddingProvider, RerankProvider]:
|
||||
"""Return embedding + reranker providers, raising if not configured.
|
||||
|
||||
The HTTP router (``entrypoints.api.routes.knowledge``) already gates
|
||||
both capabilities via ``Depends(_require_knowledge_capabilities)``, so
|
||||
this check is normally redundant for HTTP callers -- it exists for
|
||||
any non-HTTP caller of :func:`search_knowledge` that bypasses the
|
||||
router dependency.
|
||||
|
||||
Raises:
|
||||
ConfigurationError: When the embedding or rerank provider is not
|
||||
configured (a required setting is missing).
|
||||
ProviderNotConfiguredError: When the embedding or rerank provider
|
||||
is not configured (-> HTTP 422 when it does surface via HTTP).
|
||||
"""
|
||||
embedder = _get_embedding()
|
||||
if embedder is None:
|
||||
raise ConfigurationError(
|
||||
"Embedding provider not configured. "
|
||||
"Set EVEROS_EMBEDDING__MODEL and EVEROS_EMBEDDING__API_KEY."
|
||||
embed = _get_embedding()
|
||||
if embed is None:
|
||||
raise ProviderNotConfiguredError(
|
||||
provider="embedding", feature=_KNOWLEDGE_FEATURE
|
||||
)
|
||||
reranker = _get_reranker()
|
||||
if reranker is None:
|
||||
raise ConfigurationError(
|
||||
"Rerank provider not configured. "
|
||||
"Set EVEROS_RERANK__MODEL and EVEROS_RERANK__BASE_URL."
|
||||
)
|
||||
return embedder, reranker
|
||||
rerank = _get_reranker()
|
||||
if rerank is None:
|
||||
raise ProviderNotConfiguredError(provider="rerank", feature=_KNOWLEDGE_FEATURE)
|
||||
return embed, rerank
|
||||
|
||||
|
||||
async def _run_category_pipeline(
|
||||
|
||||
@@ -31,7 +31,6 @@ from pydantic import BaseModel
|
||||
from everos.component.llm import get_llm_client
|
||||
from everos.config import load_settings
|
||||
from everos.core.context import resolve_request_id
|
||||
from everos.core.observability.logging import get_logger
|
||||
from everos.core.observability.tracing import memory_span
|
||||
from everos.core.persistence import MemoryRoot
|
||||
from everos.infra.ome.config import OMEConfig
|
||||
@@ -56,8 +55,6 @@ from everos.memory.strategies import (
|
||||
from everos.service._boundary import prepare_cells
|
||||
from everos.service._session_lock import get_session_lock
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
|
||||
class MemorizeResult(BaseModel):
|
||||
"""What memorize returns to the caller (route serialises it)."""
|
||||
@@ -85,7 +82,7 @@ def _config_root() -> Path:
|
||||
def _get_episode_writer() -> EpisodeWriter:
|
||||
global _episode_writer
|
||||
if _episode_writer is None:
|
||||
_episode_writer = EpisodeWriter(MemoryRoot.default())
|
||||
_episode_writer = EpisodeWriter(MemoryRoot.resolve())
|
||||
return _episode_writer
|
||||
|
||||
|
||||
@@ -115,14 +112,55 @@ def _get_agent_pipeline() -> AgentMemoryPipeline:
|
||||
return _agent_pipeline
|
||||
|
||||
|
||||
_STRATEGIES_ALWAYS = (
|
||||
extract_atomic_facts,
|
||||
extract_foresight,
|
||||
extract_agent_case,
|
||||
extract_user_profile,
|
||||
)
|
||||
"""LLM-only strategies — no embed dependency, always registered."""
|
||||
|
||||
_STRATEGIES_REQUIRE_EMBED = (
|
||||
trigger_skill_clustering,
|
||||
extract_agent_skill,
|
||||
trigger_profile_clustering,
|
||||
reflect_episodes,
|
||||
)
|
||||
"""Strategies whose body re-embeds or consumes a re-embedded cluster.
|
||||
|
||||
All are registered unconditionally; each guards its own body via
|
||||
:func:`everos.component.embedding.get_embedding_capability` at execution
|
||||
time. When capability is unavailable the strategy no-ops (logged at
|
||||
debug level) rather than raising deep inside its body.
|
||||
|
||||
Why body-guard: the capability singleton at
|
||||
:mod:`everos.component.embedding.accessor` is process-wide and never
|
||||
invalidated at runtime — a tier upgrade (Tier 1 → Tier 2 by editing
|
||||
everos.toml) is applied only after a server restart. Body-guard is
|
||||
NOT a hot-reload mechanism; it exists for defensive degradation so
|
||||
that direct-emit paths (unit tests, future features that bypass the
|
||||
upstream clustering gate) still fail cleanly instead of crashing
|
||||
inside recall.
|
||||
"""
|
||||
|
||||
|
||||
def _get_engine() -> OfflineEngine:
|
||||
"""Return the singleton OfflineEngine; constructed + registered on first call.
|
||||
|
||||
All strategies (both always-on and embed-requiring) are registered
|
||||
unconditionally. Embed-requiring strategies guard their own bodies
|
||||
with :func:`everos.component.embedding.get_embedding_capability`.
|
||||
|
||||
Tier changes require a server restart to take effect: the accessor's
|
||||
cached capability is not invalidated at runtime. Body-guard keeps
|
||||
dispatches clean when capability was absent at start; it does not
|
||||
substitute for restart on config changes.
|
||||
|
||||
Lifecycle (start/stop) is wired by ``OmeLifespanProvider``.
|
||||
"""
|
||||
global _ome_engine
|
||||
if _ome_engine is None:
|
||||
root = MemoryRoot.default()
|
||||
root = MemoryRoot.resolve()
|
||||
jobstore_path = root.ome_db
|
||||
jobstore_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
engine = OfflineEngine(
|
||||
@@ -131,14 +169,8 @@ def _get_engine() -> OfflineEngine:
|
||||
config_path=root.ome_config,
|
||||
)
|
||||
)
|
||||
engine.register(extract_atomic_facts)
|
||||
engine.register(extract_foresight)
|
||||
engine.register(extract_agent_case)
|
||||
engine.register(trigger_skill_clustering)
|
||||
engine.register(extract_agent_skill)
|
||||
engine.register(trigger_profile_clustering)
|
||||
engine.register(extract_user_profile)
|
||||
engine.register(reflect_episodes)
|
||||
for strategy in (*_STRATEGIES_ALWAYS, *_STRATEGIES_REQUIRE_EMBED):
|
||||
engine.register(strategy)
|
||||
_ome_engine = engine
|
||||
return _ome_engine
|
||||
|
||||
|
||||
@@ -7,17 +7,24 @@ lifespan that brings up LanceDB / settings.
|
||||
|
||||
Component policy (matches :class:`SearchManager` guards):
|
||||
|
||||
* Embedding / rerank / LLM clients are **optional at boot**; they are
|
||||
built lazily, and only the methods that need them fail (with a clear
|
||||
* Embedding / rerank / LLM clients are **optional at boot**; the manager
|
||||
is built lazily and only the methods that need them fail (with a clear
|
||||
message) when the corresponding section of settings is empty.
|
||||
* ``KEYWORD`` searches therefore work without any of the three clients,
|
||||
which makes the endpoint usable in a freshly-installed dev setup.
|
||||
* All three providers are pulled from their process-wide capability /
|
||||
singleton accessors (:func:`get_embedding_capability`,
|
||||
:func:`get_rerank_capability`, :func:`get_llm_client`); this module
|
||||
never keeps parallel singletons of its own.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from everos.component.embedding import get_embedding_capability
|
||||
from everos.component.llm import LLMNotConfiguredError, get_llm_client
|
||||
from everos.component.rerank import get_rerank_capability
|
||||
from everos.component.tokenizer import build_tokenizer
|
||||
from everos.core.observability.logging import get_logger
|
||||
from everos.memory.search import SearchRequest, SearchResponse
|
||||
@@ -32,100 +39,36 @@ from everos.memory.search.recall import (
|
||||
)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from everos.component.embedding import EmbeddingProvider
|
||||
from everos.component.llm import LLMClient
|
||||
from everos.component.rerank import RerankProvider
|
||||
|
||||
logger = get_logger(__name__)
|
||||
|
||||
# Lazy singletons ────────────────────────────────────────────────────────
|
||||
|
||||
# Lazy singleton — the manager itself; every provider it needs comes
|
||||
# from its own process-wide accessor (see module docstring).
|
||||
_manager: SearchManager | None = None
|
||||
_embedding: EmbeddingProvider | None = None
|
||||
_reranker: RerankProvider | None = None
|
||||
_llm_client: LLMClient | None = None
|
||||
_embedding_resolved = False
|
||||
_rerank_resolved = False
|
||||
_llm_resolved = False
|
||||
|
||||
|
||||
def _get_embedding() -> EmbeddingProvider | None:
|
||||
"""Build the embedding client on first call. ``None`` when not configured."""
|
||||
global _embedding, _embedding_resolved
|
||||
if _embedding_resolved:
|
||||
return _embedding
|
||||
|
||||
from everos.component.embedding import build_embedding_provider
|
||||
from everos.config import load_settings
|
||||
|
||||
cfg = load_settings().embedding
|
||||
if not cfg.model or not cfg.api_key or not cfg.api_key.get_secret_value():
|
||||
logger.warning(
|
||||
"embedding_not_configured",
|
||||
hint="set [embedding] model / api_key to enable vector / hybrid search",
|
||||
)
|
||||
_embedding = None
|
||||
else:
|
||||
_embedding = build_embedding_provider(cfg)
|
||||
logger.info("search_embedding_built", model=cfg.model)
|
||||
_embedding_resolved = True
|
||||
return _embedding
|
||||
|
||||
|
||||
def _get_reranker() -> RerankProvider | None:
|
||||
"""Build the rerank client on first call. ``None`` when not configured."""
|
||||
global _reranker, _rerank_resolved
|
||||
if _rerank_resolved:
|
||||
return _reranker
|
||||
|
||||
from everos.component.rerank import build_rerank_provider
|
||||
from everos.config import load_settings
|
||||
|
||||
cfg = load_settings().rerank
|
||||
has_key = cfg.api_key and cfg.api_key.get_secret_value()
|
||||
if not cfg.model or not cfg.base_url or not has_key:
|
||||
logger.warning(
|
||||
"rerank_not_configured",
|
||||
hint="set [rerank] model / api_key / base_url to enable agentic search",
|
||||
)
|
||||
_reranker = None
|
||||
else:
|
||||
_reranker = build_rerank_provider(cfg)
|
||||
logger.info("search_rerank_built", model=cfg.model, provider=cfg.provider)
|
||||
_rerank_resolved = True
|
||||
return _reranker
|
||||
|
||||
|
||||
def _get_llm_client() -> LLMClient | None:
|
||||
"""Lazily build the LLM client from settings (shared with memorize)."""
|
||||
global _llm_client, _llm_resolved
|
||||
if _llm_resolved:
|
||||
return _llm_client
|
||||
"""Return the process-wide LLM client, or ``None`` when unset.
|
||||
|
||||
from everos.component.llm import build_llm_provider
|
||||
from everos.config import load_settings
|
||||
|
||||
settings = load_settings()
|
||||
cfg = settings.llm
|
||||
if not cfg.api_key or not cfg.api_key.get_secret_value() or not cfg.base_url:
|
||||
Delegates to :func:`everos.component.llm.get_llm_client` (which
|
||||
caches its own singleton, validates ``api_key`` / ``base_url``, and
|
||||
already wraps the client with ``UsageRecordingClient`` when
|
||||
observability is enabled — see ``component/llm/client.py``) instead
|
||||
of maintaining a parallel module-level singleton here. LLM is
|
||||
optional for search — ``KEYWORD`` works without it — so
|
||||
:class:`LLMNotConfiguredError` is swallowed into ``None`` and the
|
||||
manager surfaces a clear error only when a method that actually
|
||||
needs the LLM is invoked.
|
||||
"""
|
||||
try:
|
||||
return get_llm_client()
|
||||
except LLMNotConfiguredError:
|
||||
logger.warning(
|
||||
"llm_not_configured",
|
||||
hint="set [llm] api_key / base_url to enable hybrid / agentic search",
|
||||
)
|
||||
_llm_client = None
|
||||
else:
|
||||
client = build_llm_provider(cfg)
|
||||
# Record token usage for the hybrid/agentic path, mirroring
|
||||
# get_llm_client() — otherwise the heaviest LLM spend (query
|
||||
# decomposition + rerank judge) is invisible in Langfuse.
|
||||
if settings.observability.enabled:
|
||||
from everos.component.llm._usage_client import UsageRecordingClient
|
||||
|
||||
client = UsageRecordingClient(client)
|
||||
_llm_client = client
|
||||
logger.info("search_llm_built", model=cfg.model)
|
||||
_llm_resolved = True
|
||||
return _llm_client
|
||||
return None
|
||||
|
||||
|
||||
def _get_manager() -> SearchManager:
|
||||
@@ -138,8 +81,8 @@ def _get_manager() -> SearchManager:
|
||||
agent_case_recaller=AgentCaseRecaller(deps),
|
||||
agent_skill_recaller=AgentSkillRecaller(deps),
|
||||
profile_recaller=ProfileRecaller(),
|
||||
embedding=_get_embedding(),
|
||||
reranker=_get_reranker(),
|
||||
embedding=get_embedding_capability().provider,
|
||||
reranker=get_rerank_capability().provider,
|
||||
llm_client=_get_llm_client(),
|
||||
)
|
||||
return _manager
|
||||
|
||||
@@ -65,6 +65,59 @@ def _reset_settings_cache() -> Iterator[None]:
|
||||
dt_module._display_tz.cache_clear()
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _reset_embedding_capability_singleton() -> Iterator[None]:
|
||||
"""Force embedding capability to ``available=False`` for every test.
|
||||
|
||||
``get_embedding_capability()`` lazily builds a process-wide singleton
|
||||
on first call (see ``component.embedding.accessor``). Leaving the
|
||||
cache as ``None`` between tests lets the accessor read ambient
|
||||
settings — ``~/.everos/everos.toml`` on a developer machine, ``.env``
|
||||
when loaded — which turns test outcomes into a function of the host
|
||||
environment. Pre-seeding a ``Capability(provider=None)`` keeps the
|
||||
suite hermetic: any body-guard that reads ``.available`` sees
|
||||
``False`` unless the test explicitly opts in by re-assigning
|
||||
``acc._capability`` (or patching the strategy module's own
|
||||
``get_embedding_capability`` reference).
|
||||
"""
|
||||
import everos.component.embedding.accessor as acc
|
||||
from everos.component.embedding import EmbeddingCapability
|
||||
|
||||
acc._capability = EmbeddingCapability(provider=None)
|
||||
yield
|
||||
acc._capability = None
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _reset_rerank_capability_singleton() -> Iterator[None]:
|
||||
"""Force rerank capability to ``available=False`` for every test.
|
||||
|
||||
See :func:`_reset_embedding_capability_singleton` for rationale — the
|
||||
rerank accessor has the same lazy-singleton shape.
|
||||
"""
|
||||
import everos.component.rerank.accessor as acc
|
||||
from everos.component.rerank import RerankCapability
|
||||
|
||||
acc._capability = RerankCapability(provider=None)
|
||||
yield
|
||||
acc._capability = None
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _reset_multimodal_capability_singleton() -> Iterator[None]:
|
||||
"""Force multimodal capability to ``available=False`` for every test.
|
||||
|
||||
See :func:`_reset_embedding_capability_singleton` for rationale — the
|
||||
multimodal accessor has the same lazy-singleton shape.
|
||||
"""
|
||||
import everos.component.multimodal.accessor as acc
|
||||
from everos.component.multimodal import MultimodalLLMCapability
|
||||
|
||||
acc._capability = MultimodalLLMCapability(provider=None)
|
||||
yield
|
||||
acc._capability = None
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def long_conversation() -> dict:
|
||||
"""LoCoMo conv_0 fixture (419 messages, 19 batches, one session)."""
|
||||
|
||||
@@ -61,7 +61,7 @@ _MEMORIZE_SINGLETONS: tuple[str, ...] = (
|
||||
)
|
||||
|
||||
# OME strategy modules carry module-level lazy singletons (``_writer`` /
|
||||
# ``_reader``) that capture ``MemoryRoot.default()`` at first call. They
|
||||
# ``_reader``) that capture ``MemoryRoot.resolve()`` at first call. They
|
||||
# survive across tests, so the second test writes its output to the
|
||||
# **first test's** tmp_path. Reset all of them per-test.
|
||||
_STRATEGY_SINGLETONS: tuple[tuple[str, tuple[str, ...]], ...] = (
|
||||
@@ -75,7 +75,7 @@ _STRATEGY_SINGLETONS: tuple[tuple[str, tuple[str, ...]], ...] = (
|
||||
|
||||
def _reset_strategy_singletons(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Null every strategy ``_writer`` / ``_reader`` so the next test
|
||||
rebuilds against its own ``MemoryRoot.default()`` (driven by the
|
||||
rebuilds against its own ``MemoryRoot.resolve()`` (driven by the
|
||||
fresh ``EVEROS_ROOT`` env var set by the calling fixture).
|
||||
"""
|
||||
for mod_name, attrs in _STRATEGY_SINGLETONS:
|
||||
|
||||
@@ -40,7 +40,7 @@ import pytest
|
||||
from everalgo.clustering import Cluster as AlgoCluster
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
|
||||
from everos.component.embedding import get_embedder
|
||||
from everos.component.embedding import get_embedding_capability
|
||||
from everos.config import load_settings
|
||||
from everos.entrypoints.api.app import create_app
|
||||
from everos.infra.persistence.lancedb import (
|
||||
@@ -96,21 +96,22 @@ async def client(
|
||||
async with _engine.begin() as _conn:
|
||||
await _conn.run_sync(_SQLModel.metadata.create_all)
|
||||
|
||||
# Search service: reset all lazy singletons so each test rebuilds
|
||||
# Search service: reset the manager singleton so each test rebuilds
|
||||
# against the just-monkey-patched memory root + .env creds.
|
||||
for attr in (
|
||||
"_manager",
|
||||
"_embedding",
|
||||
"_reranker",
|
||||
"_llm_client",
|
||||
):
|
||||
setattr(search_service_mod, attr, None)
|
||||
for attr in (
|
||||
"_embedding_resolved",
|
||||
"_rerank_resolved",
|
||||
"_llm_resolved",
|
||||
):
|
||||
setattr(search_service_mod, attr, False)
|
||||
search_service_mod._manager = None
|
||||
|
||||
# Embedding / rerank / LLM route through their process-wide
|
||||
# capability / singleton accessors — reset those directly so each
|
||||
# test rebuilds against the just-monkey-patched memory root + .env
|
||||
# creds (also covered by the autouse fixtures in
|
||||
# ``tests/conftest.py``; kept explicit here for locality with the
|
||||
# rest of this fixture's singleton reset).
|
||||
embedding_acc = import_module("everos.component.embedding.accessor")
|
||||
rerank_acc = import_module("everos.component.rerank.accessor")
|
||||
llm_client_mod = import_module("everos.component.llm.client")
|
||||
embedding_acc._capability = None
|
||||
rerank_acc._capability = None
|
||||
llm_client_mod._llm_client = None
|
||||
|
||||
app = create_app(lifespan_providers=[])
|
||||
transport = ASGITransport(app=app)
|
||||
@@ -170,7 +171,7 @@ async def _seed_user_memory_cluster(eps: list[dict], *, owner_id: str) -> None:
|
||||
"""
|
||||
memcell_ids = list({ep["parent_id"] for ep in eps})
|
||||
centroid_text = eps[0]["episode"]
|
||||
centroid_vec = await get_embedder().embed(centroid_text)
|
||||
centroid_vec = await get_embedding_capability().require().embed(centroid_text)
|
||||
await cluster_repo.upsert_with_members(
|
||||
AlgoCluster(
|
||||
id=mint_cluster_id(),
|
||||
@@ -428,14 +429,12 @@ async def _seed_one_agent_corpus(owner: str = "a1") -> None:
|
||||
can rank them (zero vectors are undefined under cosine distance —
|
||||
the dense path returns 0 hits for them).
|
||||
"""
|
||||
from everos.service.search import _get_embedding
|
||||
|
||||
case_intent = "refactor authentication middleware"
|
||||
case_approach = "split provider lookup from session decode"
|
||||
skill_desc = "refactor authentication middleware reliably"
|
||||
skill_body = "step-by-step approach for auth refactors"
|
||||
|
||||
embedder = _get_embedding()
|
||||
embedder = get_embedding_capability().provider
|
||||
if embedder is not None:
|
||||
case_vec, skill_vec = await embedder.embed_batch(
|
||||
[f"{case_intent}\n{case_approach}", f"{skill_desc}\n{skill_body}"]
|
||||
@@ -578,8 +577,6 @@ async def test_hybrid_rerank_bridges_skill_via_case_lineage(
|
||||
surface it on its own; the bridge is the path that does, and LLM
|
||||
rerank keeps it because the topic is genuinely relevant.
|
||||
"""
|
||||
from everos.service.search import _get_embedding
|
||||
|
||||
case_id_with_owner = "a_bridge_ac_1" # mirrors AgentCase.id = "<owner>_<entry_id>"
|
||||
case_intent = "refactor authentication middleware"
|
||||
case_approach = "split provider lookup from session decode"
|
||||
@@ -595,7 +592,7 @@ async def test_hybrid_rerank_bridges_skill_via_case_lineage(
|
||||
"independently."
|
||||
)
|
||||
|
||||
embedder = _get_embedding()
|
||||
embedder = get_embedding_capability().provider
|
||||
assert embedder is not None, "live_llm test requires a real embedder"
|
||||
case_vec, skill_vec = await embedder.embed_batch(
|
||||
[f"{case_intent}\n{case_approach}", f"{skill_desc}\n{skill_body}"]
|
||||
|
||||
@@ -224,15 +224,7 @@ async def search_client(
|
||||
# The search service has its own module-level singletons; reset
|
||||
# those too so re-attach is clean.
|
||||
search_svc = importlib.import_module("everos.service.search")
|
||||
for attr in (
|
||||
"_manager",
|
||||
"_embedding",
|
||||
"_reranker",
|
||||
"_llm_client",
|
||||
"_embedding_resolved",
|
||||
"_rerank_resolved",
|
||||
"_llm_resolved",
|
||||
):
|
||||
for attr in ("_manager", "_llm_client", "_llm_resolved"):
|
||||
if hasattr(search_svc, attr):
|
||||
monkeypatch.setattr(
|
||||
search_svc,
|
||||
@@ -241,6 +233,18 @@ async def search_client(
|
||||
raising=False,
|
||||
)
|
||||
|
||||
# Embedding / rerank now route through the process-wide capability
|
||||
# accessors (``everos.component.{embedding,rerank}.accessor``) instead
|
||||
# of module-level singletons on ``everos.service.search`` — reset those
|
||||
# singletons directly so re-attach is clean. (Also covered by the
|
||||
# autouse fixtures in ``tests/conftest.py``; kept here for locality with
|
||||
# the rest of this fixture's singleton reset.)
|
||||
import everos.component.embedding.accessor as embedding_acc
|
||||
import everos.component.rerank.accessor as rerank_acc
|
||||
|
||||
monkeypatch.setattr(embedding_acc, "_capability", None, raising=False)
|
||||
monkeypatch.setattr(rerank_acc, "_capability", None, raising=False)
|
||||
|
||||
from everos.entrypoints.api.app import create_app
|
||||
|
||||
app = create_app()
|
||||
|
||||
@@ -0,0 +1,327 @@
|
||||
"""Tests for GET /health endpoint capabilities and disabled_features response."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pytest import mark, param
|
||||
|
||||
from everos.component.capabilities import compute_disabled_features
|
||||
|
||||
|
||||
class TestComputeDisabledFeatures:
|
||||
"""Test the compute_disabled_features helper function."""
|
||||
|
||||
def test_all_capabilities_available(self) -> None:
|
||||
"""When all capabilities are available, no features are disabled."""
|
||||
caps = {
|
||||
"llm": True,
|
||||
"embed": True,
|
||||
"rerank": True,
|
||||
"multimodal_llm": True,
|
||||
"parser": True,
|
||||
}
|
||||
disabled = compute_disabled_features(caps)
|
||||
assert disabled == []
|
||||
|
||||
def test_embedding_unavailable_disables_vector_search_features(
|
||||
self,
|
||||
) -> None:
|
||||
"""When embedding unavailable, vector/hybrid/reflection/skill disabled."""
|
||||
caps = {
|
||||
"llm": True,
|
||||
"embed": False,
|
||||
"rerank": True,
|
||||
"multimodal_llm": True,
|
||||
"parser": True,
|
||||
}
|
||||
disabled = compute_disabled_features(caps)
|
||||
expected = {
|
||||
"vector_search",
|
||||
"hybrid_search",
|
||||
"reflection",
|
||||
"skill_extraction",
|
||||
"knowledge",
|
||||
}
|
||||
assert set(disabled) == expected
|
||||
|
||||
def test_rerank_unavailable_disables_agentic_search(self) -> None:
|
||||
"""When rerank is unavailable, agentic_search is disabled."""
|
||||
caps = {
|
||||
"llm": True,
|
||||
"embed": True,
|
||||
"rerank": False,
|
||||
"multimodal_llm": True,
|
||||
"parser": True,
|
||||
}
|
||||
disabled = compute_disabled_features(caps)
|
||||
assert "agentic_search" in disabled
|
||||
|
||||
def test_both_embed_and_rerank_missing_disables_knowledge(self) -> None:
|
||||
"""When both embed and rerank are missing, knowledge is disabled."""
|
||||
caps = {
|
||||
"llm": True,
|
||||
"embed": False,
|
||||
"rerank": False,
|
||||
"multimodal_llm": True,
|
||||
"parser": True,
|
||||
}
|
||||
disabled = compute_disabled_features(caps)
|
||||
assert "knowledge" in disabled
|
||||
|
||||
def test_multimodal_and_parser_missing_disables_multimodal_upload(self) -> None:
|
||||
"""When multimodal_llm is unavailable, multimodal_upload is disabled."""
|
||||
caps = {
|
||||
"llm": True,
|
||||
"embed": True,
|
||||
"rerank": True,
|
||||
"multimodal_llm": False,
|
||||
"parser": True,
|
||||
}
|
||||
disabled = compute_disabled_features(caps)
|
||||
assert "multimodal_upload" in disabled
|
||||
|
||||
def test_parser_missing_disables_multimodal_upload(self) -> None:
|
||||
"""When parser is unavailable, multimodal_upload is disabled."""
|
||||
caps = {
|
||||
"llm": True,
|
||||
"embed": True,
|
||||
"rerank": True,
|
||||
"multimodal_llm": True,
|
||||
"parser": False,
|
||||
}
|
||||
disabled = compute_disabled_features(caps)
|
||||
assert "multimodal_upload" in disabled
|
||||
|
||||
def test_tier1_only_llm(self) -> None:
|
||||
"""Tier 1 (LLM only): many features are disabled."""
|
||||
caps = {
|
||||
"llm": True,
|
||||
"embed": False,
|
||||
"rerank": False,
|
||||
"multimodal_llm": False,
|
||||
"parser": False,
|
||||
}
|
||||
disabled = compute_disabled_features(caps)
|
||||
# Tier 1 disables: vector/hybrid/agentic/reflection/skill/knowledge/multimodal
|
||||
expected = {
|
||||
"vector_search",
|
||||
"hybrid_search",
|
||||
"agentic_search",
|
||||
"reflection",
|
||||
"skill_extraction",
|
||||
"knowledge",
|
||||
"multimodal_upload",
|
||||
}
|
||||
assert set(disabled) == expected
|
||||
|
||||
@mark.parametrize(
|
||||
"caps,expected_disabled",
|
||||
[
|
||||
param(
|
||||
{
|
||||
"llm": True,
|
||||
"embed": True,
|
||||
"rerank": True,
|
||||
"multimodal_llm": True,
|
||||
"parser": True,
|
||||
},
|
||||
[],
|
||||
id="all_available",
|
||||
),
|
||||
param(
|
||||
{
|
||||
"llm": True,
|
||||
"embed": False,
|
||||
"rerank": True,
|
||||
"multimodal_llm": True,
|
||||
"parser": True,
|
||||
},
|
||||
[
|
||||
"vector_search",
|
||||
"hybrid_search",
|
||||
"reflection",
|
||||
"skill_extraction",
|
||||
"knowledge",
|
||||
],
|
||||
id="no_embed",
|
||||
),
|
||||
param(
|
||||
{
|
||||
"llm": True,
|
||||
"embed": True,
|
||||
"rerank": False,
|
||||
"multimodal_llm": True,
|
||||
"parser": True,
|
||||
},
|
||||
["agentic_search", "knowledge"],
|
||||
id="no_rerank",
|
||||
),
|
||||
],
|
||||
)
|
||||
def test_combinations(
|
||||
self,
|
||||
caps: dict[str, bool],
|
||||
expected_disabled: list[str],
|
||||
) -> None:
|
||||
"""Parametrized tests for various capability combinations."""
|
||||
disabled = compute_disabled_features(caps)
|
||||
assert set(disabled) == set(expected_disabled)
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
async def test_health_endpoint_includes_capabilities() -> None:
|
||||
"""GET /health returns capabilities dict with all 5 provider fields."""
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
|
||||
from everos.entrypoints.api.app import create_app
|
||||
|
||||
app = create_app()
|
||||
transport = ASGITransport(app=app)
|
||||
async with AsyncClient(transport=transport) as client:
|
||||
response = await client.get("http://testserver/health")
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
|
||||
assert "status" in data
|
||||
assert "version" in data
|
||||
assert "capabilities" in data
|
||||
assert "disabled_features" in data
|
||||
|
||||
caps = data["capabilities"]
|
||||
assert isinstance(caps, dict)
|
||||
assert "llm" in caps
|
||||
assert "embed" in caps
|
||||
assert "rerank" in caps
|
||||
assert "multimodal_llm" in caps
|
||||
assert "parser" in caps
|
||||
|
||||
# All capability values should be boolean
|
||||
for key, value in caps.items():
|
||||
msg = f"Capability {key} should be bool, got {type(value)}"
|
||||
assert isinstance(value, bool), msg
|
||||
|
||||
# disabled_features should be a list
|
||||
disabled = data["disabled_features"]
|
||||
assert isinstance(disabled, list)
|
||||
for feature in disabled:
|
||||
assert isinstance(feature, str)
|
||||
|
||||
|
||||
@mark.asyncio
|
||||
@mark.parametrize(
|
||||
("embed", "rerank", "multimodal_llm", "parser", "expected_disabled"),
|
||||
[
|
||||
param(
|
||||
False,
|
||||
False,
|
||||
False,
|
||||
False,
|
||||
{
|
||||
"vector_search",
|
||||
"hybrid_search",
|
||||
"reflection",
|
||||
"skill_extraction",
|
||||
"agentic_search",
|
||||
"knowledge",
|
||||
"multimodal_upload",
|
||||
},
|
||||
id="tier1_llm_only",
|
||||
),
|
||||
param(
|
||||
True,
|
||||
False,
|
||||
False,
|
||||
False,
|
||||
{"agentic_search", "knowledge", "multimodal_upload"},
|
||||
id="tier2_embed_only",
|
||||
),
|
||||
param(
|
||||
True,
|
||||
True,
|
||||
False,
|
||||
False,
|
||||
{"multimodal_upload"},
|
||||
id="tier3_embed_rerank_only",
|
||||
),
|
||||
param(
|
||||
True,
|
||||
True,
|
||||
True,
|
||||
True,
|
||||
set(),
|
||||
id="full_stack_all_enabled",
|
||||
),
|
||||
],
|
||||
)
|
||||
async def test_health_endpoint_disabled_features_matches_hardcoded_expectation(
|
||||
monkeypatch,
|
||||
embed: bool,
|
||||
rerank: bool,
|
||||
multimodal_llm: bool,
|
||||
parser: bool,
|
||||
expected_disabled: set[str],
|
||||
) -> None:
|
||||
"""GET /health disabled_features must match a hard-coded expected set.
|
||||
|
||||
Pre-M12 this test asserted ``set(response['disabled_features']) ==
|
||||
set(compute_disabled_features(response['capabilities']))`` — a
|
||||
tautology, because the endpoint uses the same
|
||||
:func:`compute_disabled_features` to build the field. Any bug in
|
||||
the helper would be present on both sides of the equation and go
|
||||
undetected. Round-2 finding M12 replaces the tautology with a
|
||||
per-tier hard-coded expected set, driven by parametrisation over
|
||||
every soft-dependency capability toggle — so a regression in
|
||||
:func:`compute_disabled_features` (e.g. dropping the
|
||||
``knowledge`` disable, mis-spelling a feature name) breaks the
|
||||
test rather than staying invisible.
|
||||
"""
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
|
||||
from everos.component import embedding, multimodal
|
||||
from everos.component import rerank as rerank_mod
|
||||
from everos.component.parser import _core as parser_core
|
||||
from everos.entrypoints.api.app import create_app
|
||||
|
||||
class _StubCapability:
|
||||
def __init__(self, available: bool) -> None:
|
||||
self.available = available
|
||||
|
||||
monkeypatch.setattr(
|
||||
embedding, "get_embedding_capability", lambda: _StubCapability(embed)
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
rerank_mod, "get_rerank_capability", lambda: _StubCapability(rerank)
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
multimodal,
|
||||
"get_multimodal_llm_capability",
|
||||
lambda: _StubCapability(multimodal_llm),
|
||||
)
|
||||
monkeypatch.setattr(parser_core, "parser_available", lambda: parser)
|
||||
|
||||
# The health route imports the accessors at module load; patch there
|
||||
# too so the values it actually reads honour the monkeypatch.
|
||||
from everos.entrypoints.api.routes import health as health_route
|
||||
|
||||
monkeypatch.setattr(
|
||||
health_route, "get_embedding_capability", lambda: _StubCapability(embed)
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
health_route, "get_rerank_capability", lambda: _StubCapability(rerank)
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
health_route,
|
||||
"get_multimodal_llm_capability",
|
||||
lambda: _StubCapability(multimodal_llm),
|
||||
)
|
||||
monkeypatch.setattr(health_route, "parser_available", lambda: parser)
|
||||
|
||||
app = create_app()
|
||||
transport = ASGITransport(app=app)
|
||||
async with AsyncClient(transport=transport) as client:
|
||||
response = await client.get("http://testserver/health")
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
|
||||
assert set(data["disabled_features"]) == expected_disabled
|
||||
@@ -0,0 +1,386 @@
|
||||
"""Verify per-endpoint scoping of the knowledge capability gate.
|
||||
|
||||
Task 11 (``cascade.registry.build_handlers``) gates the knowledge
|
||||
cascade handlers off as an atomic pair when embed + rerank aren't
|
||||
both available — a doc queued for a gated-off kind gets marked
|
||||
permanently failed by the worker instead of ever reaching LanceDB.
|
||||
Before this router gate existed, the HTTP layer had no idea any of
|
||||
that happened: a Tier 1 user's upload returned 200 (the md write
|
||||
always succeeds) while the document silently never became
|
||||
searchable.
|
||||
|
||||
An earlier iteration attached the gate router-wide, which regressed
|
||||
downgraded users: a client who dropped from Tier 3 → Tier 2/1 could
|
||||
no longer even GET or DELETE their own docs, because the router-wide
|
||||
dependency fired on every request. This module pins the corrected
|
||||
scoping — writes and search stay gated (they can't succeed end-to-
|
||||
end without the providers) while reads/deletes stay reachable so
|
||||
users can inspect and clean up state they already have on disk.
|
||||
|
||||
Uses ``httpx.AsyncClient`` + ``ASGITransport`` (not
|
||||
``fastapi.testclient.TestClient``) to match the project's established
|
||||
exception-handler test convention (see ``tests/integration/test_api/
|
||||
test_provider_error_mapping.py``). The knowledge router is mounted
|
||||
standalone (no lifespan / full app) since the gate fires before any
|
||||
body is even parsed — sub-dependency resolution runs ahead of
|
||||
body/query/path validation in FastAPI's request handler, so a bare
|
||||
request with no payload is enough to reach the gate on every route,
|
||||
including the multipart upload endpoints.
|
||||
|
||||
For endpoints without a gate we stub the service call so the test
|
||||
proves the request reached the handler (i.e. no 422 from the
|
||||
missing-provider check), independent of MemoryRoot / SQLite setup.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import AsyncIterator
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from fastapi import FastAPI
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
|
||||
import everos.component.embedding.accessor as embedding_accessor
|
||||
import everos.component.rerank.accessor as rerank_accessor
|
||||
from everos.component.embedding import EmbeddingCapability
|
||||
from everos.component.rerank import RerankCapability
|
||||
from everos.entrypoints.api.exception_handlers import register_handlers
|
||||
from everos.entrypoints.api.routes import knowledge as knowledge_routes
|
||||
|
||||
# Endpoints that MUST return 422 when embed or rerank is missing.
|
||||
# These paths either write new md that cascade will need to embed
|
||||
# (POST/PUT) or run a search that requires both providers
|
||||
# (POST /search). PATCH is deliberately NOT here — see the ungated
|
||||
# list below.
|
||||
_GATED_ENDPOINTS: list[tuple[str, str]] = [
|
||||
("POST", "/api/v1/knowledge/documents"),
|
||||
("PUT", "/api/v1/knowledge/documents/d_abcdef123456"),
|
||||
("POST", "/api/v1/knowledge/search"),
|
||||
]
|
||||
|
||||
# Endpoints that MUST stay reachable even when both capabilities are
|
||||
# missing. All of these touch only md + SQLite state that already
|
||||
# exists on disk — none of them need the embedding or rerank provider.
|
||||
# PATCH is here because ``patch_document`` only rewrites md frontmatter
|
||||
# and moves the doc directory when category changes; no embed or rerank
|
||||
# code runs on that path. A user who downgrades from Tier 3 → Tier 1/2
|
||||
# must still be able to rename or recategorize the documents they
|
||||
# created while providers were configured.
|
||||
_UNGATED_ENDPOINTS: list[tuple[str, str]] = [
|
||||
("DELETE", "/api/v1/knowledge/documents/d_abcdef123456"),
|
||||
("GET", "/api/v1/knowledge/documents"),
|
||||
("GET", "/api/v1/knowledge/documents/d_abcdef123456"),
|
||||
("GET", "/api/v1/knowledge/topics/d_abcdef123456_0"),
|
||||
("GET", "/api/v1/knowledge/categories"),
|
||||
]
|
||||
|
||||
|
||||
def _build_app() -> FastAPI:
|
||||
app = FastAPI()
|
||||
register_handlers(app)
|
||||
# Match production mounting: create_app() mounts the router under
|
||||
# /api/v1 (and /api/v2). The router's own prefix is /knowledge, so
|
||||
# the effective paths become /api/v1/knowledge/... — same shape the
|
||||
# _GATED / _UNGATED constants below use.
|
||||
app.include_router(knowledge_routes.router, prefix="/api/v1")
|
||||
return app
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
async def client() -> AsyncIterator[AsyncClient]:
|
||||
app = _build_app()
|
||||
transport = ASGITransport(app=app, raise_app_exceptions=False)
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as c:
|
||||
yield c
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _capabilities_unavailable_by_default(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Default both singletons to unavailable; tests opt into availability
|
||||
per-capability via the fixtures below. Mirrors ``tests/unit/test_memory
|
||||
/test_search/test_validate_components.py``.
|
||||
"""
|
||||
monkeypatch.setattr(
|
||||
embedding_accessor, "_capability", EmbeddingCapability(provider=None)
|
||||
)
|
||||
monkeypatch.setattr(rerank_accessor, "_capability", RerankCapability(provider=None))
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def embed_available(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.setattr(
|
||||
embedding_accessor, "_capability", EmbeddingCapability(provider=object())
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def rerank_available(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.setattr(
|
||||
rerank_accessor, "_capability", RerankCapability(provider=object())
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def _stub_read_services(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Replace every read/delete/list service call with a benign stub.
|
||||
|
||||
The tests in this module care only about whether the capability
|
||||
gate fires. Ungated endpoints must reach their handler even when
|
||||
providers are missing — but reaching the handler with no
|
||||
MemoryRoot / SQLite setup would 500. Stubbing keeps the assertion
|
||||
focused on the gate, not on end-to-end service behavior.
|
||||
"""
|
||||
from everos.service import knowledge as knowledge_service
|
||||
|
||||
async def _fake_delete_document(*args: Any, **kwargs: Any) -> Any:
|
||||
return knowledge_service.DeleteResult(doc_id="d_abcdef123456", deleted_topics=0)
|
||||
|
||||
async def _fake_list_documents(*args: Any, **kwargs: Any) -> Any:
|
||||
return knowledge_service.DocumentListResult(
|
||||
documents=[], total=0, page=1, page_size=20
|
||||
)
|
||||
|
||||
async def _fake_get_document(*args: Any, **kwargs: Any) -> Any:
|
||||
from everos.component.utils.datetime import get_utc_now
|
||||
|
||||
now = get_utc_now()
|
||||
return knowledge_service.DocumentDetail(
|
||||
doc_id="d_abcdef123456",
|
||||
category_id="General",
|
||||
title="stub",
|
||||
summary="stub",
|
||||
source_name=None,
|
||||
source_type=None,
|
||||
original_file_path=None,
|
||||
topics=[],
|
||||
created_at=now,
|
||||
updated_at=now,
|
||||
)
|
||||
|
||||
async def _fake_get_topic(*args: Any, **kwargs: Any) -> Any:
|
||||
from everos.component.utils.datetime import get_utc_now
|
||||
|
||||
now = get_utc_now()
|
||||
return knowledge_service.TopicDetail(
|
||||
topic_id="d_abcdef123456_0",
|
||||
doc_id="d_abcdef123456",
|
||||
category_id="General",
|
||||
topic_name="stub",
|
||||
topic_path="stub",
|
||||
depth=0,
|
||||
summary="stub",
|
||||
content="",
|
||||
content_labels=[],
|
||||
parent_topic_id=None,
|
||||
children_topic_ids=[],
|
||||
created_at=now,
|
||||
updated_at=now,
|
||||
)
|
||||
|
||||
async def _fake_list_categories(*args: Any, **kwargs: Any) -> list[Any]:
|
||||
return []
|
||||
|
||||
monkeypatch.setattr(knowledge_routes, "delete_document", _fake_delete_document)
|
||||
monkeypatch.setattr(knowledge_routes, "list_documents", _fake_list_documents)
|
||||
monkeypatch.setattr(knowledge_routes, "get_document", _fake_get_document)
|
||||
monkeypatch.setattr(knowledge_routes, "get_topic", _fake_get_topic)
|
||||
monkeypatch.setattr(knowledge_routes, "list_categories", _fake_list_categories)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def _stub_patch_document(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Stub ``patch_document`` to echo the requested title back.
|
||||
|
||||
Mirrors the real service contract: a title change lands in
|
||||
``updated_fields`` and the caller sees the mutation reflected in
|
||||
the response envelope. Keeps the assertion focused on the gate
|
||||
(or lack of it) instead of on md / SQLite plumbing.
|
||||
"""
|
||||
from everos.component.utils.datetime import get_utc_now
|
||||
from everos.service import knowledge as knowledge_service
|
||||
|
||||
captured: dict[str, Any] = {}
|
||||
|
||||
async def _fake_patch_document(
|
||||
doc_id: str,
|
||||
app_id: str,
|
||||
project_id: str,
|
||||
*,
|
||||
title: str | None = None,
|
||||
category_id: str | None = None,
|
||||
) -> Any:
|
||||
captured["title"] = title
|
||||
captured["category_id"] = category_id
|
||||
updated_fields: list[str] = []
|
||||
if title is not None:
|
||||
updated_fields.append("title")
|
||||
if category_id is not None:
|
||||
updated_fields.append("category_id")
|
||||
return knowledge_service.PatchResult(
|
||||
doc_id=doc_id,
|
||||
updated_fields=updated_fields,
|
||||
updated_at=get_utc_now(),
|
||||
)
|
||||
|
||||
monkeypatch.setattr(knowledge_routes, "patch_document", _fake_patch_document)
|
||||
|
||||
|
||||
async def _call(client: AsyncClient, method: str, path: str) -> Any:
|
||||
return await client.request(method, path)
|
||||
|
||||
|
||||
# ── Gated endpoints: 422 when embed missing (rerank also missing) ────────
|
||||
|
||||
|
||||
@pytest.mark.parametrize("method,path", _GATED_ENDPOINTS)
|
||||
async def test_gated_endpoint_422_when_embed_missing(
|
||||
client: AsyncClient, method: str, path: str
|
||||
) -> None:
|
||||
"""Write/search endpoints must 422 with an embedding-specific message
|
||||
when embedding is missing (check-embedding-first order per Task 12/13).
|
||||
"""
|
||||
resp = await _call(client, method, path)
|
||||
assert resp.status_code == 422
|
||||
body = resp.json()
|
||||
assert body["error"]["code"] == "PROVIDER_NOT_CONFIGURED"
|
||||
assert "embedding" in body["error"]["message"]
|
||||
assert "knowledge" in body["error"]["message"]
|
||||
|
||||
|
||||
# ── Gated endpoints: 422 when only rerank missing ───────────────────────
|
||||
|
||||
|
||||
@pytest.mark.parametrize("method,path", _GATED_ENDPOINTS)
|
||||
async def test_gated_endpoint_422_when_rerank_missing(
|
||||
client: AsyncClient, method: str, path: str, embed_available: None
|
||||
) -> None:
|
||||
"""Write/search endpoints must 422 with a rerank-specific message
|
||||
when only rerank is missing.
|
||||
"""
|
||||
resp = await _call(client, method, path)
|
||||
assert resp.status_code == 422
|
||||
body = resp.json()
|
||||
assert body["error"]["code"] == "PROVIDER_NOT_CONFIGURED"
|
||||
assert "rerank" in body["error"]["message"]
|
||||
assert "knowledge" in body["error"]["message"]
|
||||
|
||||
|
||||
# ── Ungated endpoints: must NOT 422 even with both capabilities missing ─
|
||||
|
||||
|
||||
@pytest.mark.parametrize("method,path", _UNGATED_ENDPOINTS)
|
||||
async def test_ungated_endpoint_not_422_without_any_capability(
|
||||
client: AsyncClient,
|
||||
method: str,
|
||||
path: str,
|
||||
_stub_read_services: None,
|
||||
) -> None:
|
||||
"""Read/list/delete endpoints must stay reachable even when neither
|
||||
embed nor rerank is configured — a Tier-3 → Tier-1 downgrade must
|
||||
not lock users out of inspecting and cleaning up their own state.
|
||||
"""
|
||||
resp = await _call(client, method, path)
|
||||
assert resp.status_code != 422, (
|
||||
f"{method} {path} returned 422 despite the gate being scoped away "
|
||||
f"from read/delete endpoints; body={resp.text}"
|
||||
)
|
||||
|
||||
|
||||
# ── Ungated endpoints: also fine with only embed (Tier-2 downgrade) ─────
|
||||
|
||||
|
||||
@pytest.mark.parametrize("method,path", _UNGATED_ENDPOINTS)
|
||||
async def test_ungated_endpoint_not_422_with_embed_only(
|
||||
client: AsyncClient,
|
||||
method: str,
|
||||
path: str,
|
||||
embed_available: None,
|
||||
_stub_read_services: None,
|
||||
) -> None:
|
||||
"""Tier-2 scenario (embed present, rerank missing): read/delete paths
|
||||
must stay reachable — this is the concrete regression from the
|
||||
router-wide-gate era that this fix targets.
|
||||
"""
|
||||
resp = await _call(client, method, path)
|
||||
assert resp.status_code != 422, (
|
||||
f"{method} {path} returned 422 despite rerank not being needed; "
|
||||
f"body={resp.text}"
|
||||
)
|
||||
|
||||
|
||||
# ── Gated endpoints: pass the gate when both are available ──────────────
|
||||
|
||||
|
||||
async def test_search_not_422_when_both_available(
|
||||
client: AsyncClient,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
embed_available: None,
|
||||
rerank_available: None,
|
||||
) -> None:
|
||||
"""Both capabilities available: the gate must not fire on /search.
|
||||
Stub the service call so the assertion is about the gate, not the
|
||||
rest of the search pipeline (out of scope for this task).
|
||||
"""
|
||||
from everos.service import knowledge as knowledge_service
|
||||
|
||||
stub_result = knowledge_service.SearchKnowledgeResult(hits=[], total=0, took_ms=0.0)
|
||||
|
||||
async def _fake_search_knowledge(**kwargs: Any) -> Any:
|
||||
return stub_result
|
||||
|
||||
monkeypatch.setattr(knowledge_routes, "search_knowledge", _fake_search_knowledge)
|
||||
resp = await client.post(
|
||||
"/api/v1/knowledge/search",
|
||||
json={"query": "hello"},
|
||||
)
|
||||
assert resp.status_code != 422
|
||||
|
||||
|
||||
# ── PATCH: metadata-only, must succeed regardless of provider config ────
|
||||
|
||||
|
||||
_PATCH_BODY: dict[str, str] = {
|
||||
"app_id": "app1",
|
||||
"project_id": "proj1",
|
||||
"title": "renamed",
|
||||
}
|
||||
|
||||
|
||||
async def test_knowledge_patch_document_succeeds_without_rerank(
|
||||
client: AsyncClient,
|
||||
embed_available: None,
|
||||
_stub_patch_document: None,
|
||||
) -> None:
|
||||
"""Tier-2 scenario (embed configured, rerank missing): PATCH must
|
||||
still land. Renaming an existing document does not touch rerank —
|
||||
it only rewrites md frontmatter and upserts SQLite.
|
||||
"""
|
||||
resp = await client.patch(
|
||||
"/api/v1/knowledge/documents/d_abcdef123456",
|
||||
json=_PATCH_BODY,
|
||||
)
|
||||
assert resp.status_code == 200, resp.text
|
||||
body = resp.json()
|
||||
assert body["data"]["doc_id"] == "d_abcdef123456"
|
||||
assert "title" in body["data"]["updated_fields"]
|
||||
|
||||
|
||||
async def test_knowledge_patch_document_succeeds_without_any_capability(
|
||||
client: AsyncClient,
|
||||
_stub_patch_document: None,
|
||||
) -> None:
|
||||
"""Tier-1 scenario (no embed, no rerank): PATCH must still land.
|
||||
Metadata-only edits are a cleanup verb — locking users out of
|
||||
renaming their own docs after a full downgrade contradicts the
|
||||
per-endpoint-gate contract documented on
|
||||
``_require_knowledge_capabilities``.
|
||||
"""
|
||||
resp = await client.patch(
|
||||
"/api/v1/knowledge/documents/d_abcdef123456",
|
||||
json=_PATCH_BODY,
|
||||
)
|
||||
assert resp.status_code == 200, resp.text
|
||||
body = resp.json()
|
||||
assert body["data"]["doc_id"] == "d_abcdef123456"
|
||||
assert "title" in body["data"]["updated_fields"]
|
||||
@@ -0,0 +1,70 @@
|
||||
"""Verify ProviderNotConfiguredError maps to HTTP 422 via the FastAPI handler.
|
||||
|
||||
Deviates from a naive flat `{"error_code": ..., "message": ...}` envelope: the
|
||||
project already has a canonical error envelope (`ErrorResponse` /
|
||||
`_error_response` in `exception_handlers.py`) shared by every domain
|
||||
exception, so `ProviderNotConfiguredError` reuses it rather than inventing a
|
||||
parallel shape. `ProviderNotConfiguredError` is a `ConfigurationError`
|
||||
subclass; the assertions below also pin that the more-specific handler wins
|
||||
over the parent `ConfigurationError` -> 500 mapping.
|
||||
|
||||
Uses `httpx.AsyncClient` + `ASGITransport` (not `fastapi.testclient.TestClient`)
|
||||
to match the project's existing exception-handler test convention -- the
|
||||
installed starlette/httpx pair deprecates `TestClient`.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from fastapi import FastAPI
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
|
||||
from everos.core.errors import ProviderNotConfiguredError
|
||||
from everos.entrypoints.api.exception_handlers import register_handlers
|
||||
|
||||
|
||||
def _build_app_with_probes() -> FastAPI:
|
||||
app = FastAPI()
|
||||
register_handlers(app)
|
||||
|
||||
@app.get("/probe_embed")
|
||||
async def probe_embed() -> None:
|
||||
raise ProviderNotConfiguredError(provider="embedding")
|
||||
|
||||
@app.get("/probe_rerank_with_alt")
|
||||
async def probe_rerank() -> None:
|
||||
raise ProviderNotConfiguredError(
|
||||
provider="rerank",
|
||||
feature="agent_hybrid",
|
||||
alternative_hint="Set enable_llm_rerank=true.",
|
||||
)
|
||||
|
||||
return app
|
||||
|
||||
|
||||
async def _get(path: str) -> tuple[int, dict]:
|
||||
app = _build_app_with_probes()
|
||||
transport = ASGITransport(app=app, raise_app_exceptions=False)
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
resp = await client.get(path)
|
||||
return resp.status_code, resp.json()
|
||||
|
||||
|
||||
async def test_maps_to_422():
|
||||
status_code, _ = await _get("/probe_embed")
|
||||
assert status_code == 422
|
||||
|
||||
|
||||
async def test_error_code_field():
|
||||
_, body = await _get("/probe_embed")
|
||||
assert body["error"]["code"] == "PROVIDER_NOT_CONFIGURED"
|
||||
|
||||
|
||||
async def test_message_field_carries_toml_path():
|
||||
_, body = await _get("/probe_embed")
|
||||
assert "everos.toml" in body["error"]["message"]
|
||||
|
||||
|
||||
async def test_alternative_hint_present_in_message():
|
||||
_, body = await _get("/probe_rerank_with_alt")
|
||||
assert "Alternative:" in body["error"]["message"]
|
||||
assert "enable_llm_rerank=true" in body["error"]["message"]
|
||||
@@ -0,0 +1,128 @@
|
||||
"""Test startup banner warning for unbackfilled memory rows.
|
||||
|
||||
When LanceDB tables have rows with vector IS NULL (unbackfilled memories),
|
||||
the startup banner should log a warning message pointing users to
|
||||
`everos cascade backfill` command.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
import structlog.testing
|
||||
|
||||
from everos.component.utils.datetime import get_utc_now
|
||||
from everos.core.observability.logging import get_logger
|
||||
from everos.core.persistence import MemoryRoot
|
||||
from everos.entrypoints.api.lifespans.lancedb import (
|
||||
LanceDBLifespanProvider,
|
||||
_log_unbackfilled_hint,
|
||||
)
|
||||
from everos.infra.persistence.lancedb import Episode
|
||||
from everos.infra.persistence.lancedb.repos import episode_repo
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def _isolated_memory_root(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Isolate each test with its own temporary MemoryRoot."""
|
||||
monkeypatch.setattr(
|
||||
MemoryRoot, "resolve", classmethod(lambda cls: MemoryRoot(root=tmp_path))
|
||||
)
|
||||
(tmp_path / ".index" / "sqlite").mkdir(parents=True, exist_ok=True)
|
||||
|
||||
|
||||
async def test_no_warning_when_empty(_isolated_memory_root: None) -> None:
|
||||
"""No warning is logged when the database is empty."""
|
||||
from fastapi import FastAPI
|
||||
|
||||
app = FastAPI()
|
||||
provider = LanceDBLifespanProvider()
|
||||
await provider.startup(app)
|
||||
|
||||
# When database is empty, no warning should be logged
|
||||
with structlog.testing.capture_logs() as captured:
|
||||
await _log_unbackfilled_hint()
|
||||
|
||||
matching = [e for e in captured if e.get("event") == "unbackfilled_memory_rows"]
|
||||
assert len(matching) == 0, "expected no warning when database is empty"
|
||||
|
||||
await provider.shutdown(app)
|
||||
|
||||
|
||||
async def test_warns_when_unbackfilled_rows_exist(
|
||||
_isolated_memory_root: None,
|
||||
) -> None:
|
||||
"""_log_unbackfilled_hint warns with correct count when vector=NULL rows exist."""
|
||||
from fastapi import FastAPI
|
||||
|
||||
# Setup: initialize provider (connects to temp LanceDB)
|
||||
app = FastAPI()
|
||||
provider = LanceDBLifespanProvider()
|
||||
await provider.startup(app)
|
||||
|
||||
# Seed: insert one Episode row with vector=None
|
||||
def _make_unvectorized_episode() -> Episode:
|
||||
"""Create an Episode row with vector=None."""
|
||||
return Episode(
|
||||
id="test_owner_entry1",
|
||||
entry_id="entry1",
|
||||
owner_id="test_owner",
|
||||
owner_type="user",
|
||||
app_id="default",
|
||||
project_id="default",
|
||||
session_id="s_test",
|
||||
timestamp=get_utc_now(),
|
||||
parent_type="memcell",
|
||||
parent_id="mc_test",
|
||||
sender_ids=["test_owner"],
|
||||
subject="test subject",
|
||||
summary=None,
|
||||
episode="test episode content",
|
||||
episode_tokens="test episode",
|
||||
md_path="users/test_owner/default/default/2026-07-25.md",
|
||||
content_sha256=hashlib.sha256(b"test").hexdigest(),
|
||||
deprecated_by=None,
|
||||
vector=None, # explicitly unvectorized
|
||||
)
|
||||
|
||||
await episode_repo.add([_make_unvectorized_episode()])
|
||||
|
||||
# Capture logs and call the hint function
|
||||
with structlog.testing.capture_logs() as captured:
|
||||
await _log_unbackfilled_hint()
|
||||
|
||||
# Assert: the warning is logged with the correct count
|
||||
matching = [e for e in captured if e.get("event") == "unbackfilled_memory_rows"]
|
||||
assert len(matching) > 0, (
|
||||
"expected unbackfilled_memory_rows warning when vector=NULL rows exist"
|
||||
)
|
||||
warning = matching[0]
|
||||
assert warning.get("count") == 1, f"expected count=1, got {warning.get('count')}"
|
||||
|
||||
await provider.shutdown(app)
|
||||
|
||||
|
||||
async def test_log_unbackfilled_hint_runs_on_startup(
|
||||
_isolated_memory_root: None,
|
||||
) -> None:
|
||||
"""Startup should call _log_unbackfilled_hint after ensuring indexes."""
|
||||
from fastapi import FastAPI
|
||||
|
||||
app = FastAPI()
|
||||
provider = LanceDBLifespanProvider()
|
||||
|
||||
# This startup should not raise - _log_unbackfilled_hint should be called
|
||||
await provider.startup(app)
|
||||
|
||||
await provider.shutdown(app)
|
||||
|
||||
|
||||
async def test_startup_banner_logger_configuration() -> None:
|
||||
"""Verify the banner logger can be obtained."""
|
||||
logger = get_logger("everos.cli.server")
|
||||
assert logger is not None
|
||||
# Verify it has the expected logging methods
|
||||
assert hasattr(logger, "warning")
|
||||
assert hasattr(logger, "info")
|
||||
@@ -87,6 +87,15 @@ async def cascade_runtime(
|
||||
monkeypatch.setenv("EVEROS_EMBEDDING__MODEL", "stub-model")
|
||||
monkeypatch.setenv("EVEROS_EMBEDDING__BASE_URL", "http://stub.invalid/v1")
|
||||
monkeypatch.setenv("EVEROS_EMBEDDING__API_KEY", "stub-key")
|
||||
# Handlers fetch the embedder lazily via ``get_embedding_capability()``
|
||||
# rather than through ``CascadeOrchestrator`` — patch the process-wide
|
||||
# singleton so cascade never hits the fake network target above.
|
||||
import everos.component.embedding.accessor as acc
|
||||
from everos.component.embedding import EmbeddingCapability
|
||||
|
||||
monkeypatch.setattr(
|
||||
acc, "_capability", EmbeddingCapability(provider=_StubEmbedder())
|
||||
)
|
||||
await dispose_connection()
|
||||
await dispose_engine()
|
||||
engine = get_engine()
|
||||
@@ -94,7 +103,7 @@ async def cascade_runtime(
|
||||
await conn.run_sync(SQLModel.metadata.create_all)
|
||||
await ensure_business_indexes()
|
||||
(tmp_path / "ome.toml").write_text("# test\n")
|
||||
yield MemoryRoot.default()
|
||||
yield MemoryRoot.resolve()
|
||||
await dispose_connection()
|
||||
await dispose_engine()
|
||||
|
||||
@@ -237,7 +246,6 @@ async def test_md_lance_strict_consistency_per_kind(
|
||||
memory_root = cascade_runtime
|
||||
orchestrator = CascadeOrchestrator(
|
||||
memory_root=memory_root,
|
||||
embedder=_StubEmbedder(),
|
||||
tokenizer=build_tokenizer(),
|
||||
config=CascadeConfig(
|
||||
scan_interval_seconds=60.0,
|
||||
|
||||
@@ -4,7 +4,8 @@ Drives the actual Typer commands against a real sqlite + lancedb under a
|
||||
tmp memory root. Validates the in-process orchestration that
|
||||
``test_cascade_command`` (unit) cannot reach: ``_runtime()`` context,
|
||||
queue summary formatting, fix (no-rows path), and a full
|
||||
``cascade sync <path>`` round-trip with a stub embedder.
|
||||
``cascade sync <path>`` round-trip against an empty queue (no handler
|
||||
ever runs, so embedding availability is irrelevant here).
|
||||
|
||||
The CLI commands call ``asyncio.run(_run())`` internally, so this test
|
||||
is **synchronous** — pytest-asyncio's auto mode would otherwise wrap it
|
||||
@@ -22,23 +23,12 @@ from pathlib import Path
|
||||
import pytest
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from everos.component.embedding import EmbeddingProvider
|
||||
from everos.config import load_settings
|
||||
from everos.entrypoints.cli.commands import cascade as cascade_mod
|
||||
from everos.infra.persistence.lancedb import dispose_connection
|
||||
from everos.infra.persistence.sqlite import dispose_engine
|
||||
|
||||
|
||||
class _StubEmbedder(EmbeddingProvider):
|
||||
dim = 1024
|
||||
|
||||
async def embed(self, text: str) -> list[float]:
|
||||
return [0.0] * self.dim
|
||||
|
||||
async def embed_batch(self, texts): # type: ignore[no-untyped-def]
|
||||
return [[0.0] * self.dim for _ in texts]
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def cli_runtime(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Iterator[Path]:
|
||||
"""Tmp memory root + clean singletons; CLI bootstraps the schema itself."""
|
||||
@@ -101,11 +91,10 @@ def test_sync_on_empty_queue_with_stub_embedder(
|
||||
from everos.memory.cascade import CascadeOrchestrator
|
||||
|
||||
def fake_build_orchestrator() -> CascadeOrchestrator:
|
||||
root = MemoryRoot.default()
|
||||
root = MemoryRoot.resolve()
|
||||
root.ensure()
|
||||
return CascadeOrchestrator(
|
||||
memory_root=root,
|
||||
embedder=_StubEmbedder(),
|
||||
tokenizer=build_tokenizer(),
|
||||
)
|
||||
|
||||
@@ -148,8 +137,7 @@ def test_sync_with_unmatched_path(
|
||||
|
||||
def fake_build_orchestrator() -> CascadeOrchestrator:
|
||||
return CascadeOrchestrator(
|
||||
memory_root=MemoryRoot.default(),
|
||||
embedder=_StubEmbedder(),
|
||||
memory_root=MemoryRoot.resolve(),
|
||||
tokenizer=build_tokenizer(),
|
||||
)
|
||||
|
||||
|
||||
@@ -72,6 +72,15 @@ async def cascade_runtime(
|
||||
monkeypatch.setenv("EVEROS_EMBEDDING__MODEL", "stub-model")
|
||||
monkeypatch.setenv("EVEROS_EMBEDDING__BASE_URL", "http://stub.invalid/v1")
|
||||
monkeypatch.setenv("EVEROS_EMBEDDING__API_KEY", "stub-key")
|
||||
# Handlers fetch the embedder lazily via ``get_embedding_capability()``
|
||||
# rather than through ``CascadeOrchestrator`` — patch the process-wide
|
||||
# singleton so cascade never hits the fake network target above.
|
||||
import everos.component.embedding.accessor as acc
|
||||
from everos.component.embedding import EmbeddingCapability
|
||||
|
||||
monkeypatch.setattr(
|
||||
acc, "_capability", EmbeddingCapability(provider=_StubEmbedder())
|
||||
)
|
||||
|
||||
await dispose_connection()
|
||||
await dispose_engine()
|
||||
@@ -82,7 +91,7 @@ async def cascade_runtime(
|
||||
await ensure_business_indexes()
|
||||
(tmp_path / "ome.toml").write_text("# test\n")
|
||||
|
||||
yield MemoryRoot.default()
|
||||
yield MemoryRoot.resolve()
|
||||
|
||||
await dispose_connection()
|
||||
await dispose_engine()
|
||||
@@ -127,7 +136,6 @@ async def test_high_freq_atomic_fact_append_no_loss(
|
||||
memory_root = cascade_runtime
|
||||
orchestrator = CascadeOrchestrator(
|
||||
memory_root=memory_root,
|
||||
embedder=_StubEmbedder(),
|
||||
tokenizer=build_tokenizer(),
|
||||
config=CascadeConfig(
|
||||
scan_interval_seconds=60.0,
|
||||
|
||||
@@ -84,12 +84,25 @@ async def cascade_runtime(
|
||||
await ensure_business_indexes()
|
||||
(tmp_path / "ome.toml").write_text("# test\n")
|
||||
|
||||
yield MemoryRoot.default()
|
||||
yield MemoryRoot.resolve()
|
||||
|
||||
await dispose_connection()
|
||||
await dispose_engine()
|
||||
|
||||
|
||||
def _patch_embedding_capability(
|
||||
monkeypatch: pytest.MonkeyPatch, embedder: _StubEmbedder
|
||||
) -> None:
|
||||
"""EpisodeHandler fetches the embedder lazily via
|
||||
``get_embedding_capability()`` — patch the process-wide singleton so
|
||||
cascade never hits the fake network target set up by ``cascade_runtime``.
|
||||
"""
|
||||
import everos.component.embedding.accessor as acc
|
||||
from everos.component.embedding import EmbeddingCapability
|
||||
|
||||
monkeypatch.setattr(acc, "_capability", EmbeddingCapability(provider=embedder))
|
||||
|
||||
|
||||
async def _poll(condition, *, deadline_seconds: float = 10.0, interval: float = 0.05): # type: ignore[no-untyped-def]
|
||||
"""Poll ``condition()`` (async) until truthy, or :class:`TimeoutError`.
|
||||
|
||||
@@ -107,13 +120,14 @@ async def _poll(condition, *, deadline_seconds: float = 10.0, interval: float =
|
||||
|
||||
async def test_append_to_md_propagates_to_lancedb(
|
||||
cascade_runtime: MemoryRoot,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Happy path: writer append → watcher → state row → worker → LanceDB."""
|
||||
memory_root = cascade_runtime
|
||||
embedder = _StubEmbedder()
|
||||
_patch_embedding_capability(monkeypatch, embedder)
|
||||
orchestrator = CascadeOrchestrator(
|
||||
memory_root=memory_root,
|
||||
embedder=embedder,
|
||||
tokenizer=build_tokenizer(),
|
||||
# Tight worker poll so the test wraps in seconds, not minutes.
|
||||
# Scanner interval kept long so the watcher path is the one
|
||||
@@ -188,12 +202,13 @@ async def test_append_to_md_propagates_to_lancedb(
|
||||
|
||||
async def test_delete_md_wipes_lancedb_row(
|
||||
cascade_runtime: MemoryRoot,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Append + drain, then ``unlink`` the md and watch the row evaporate."""
|
||||
memory_root = cascade_runtime
|
||||
_patch_embedding_capability(monkeypatch, _StubEmbedder())
|
||||
orchestrator = CascadeOrchestrator(
|
||||
memory_root=memory_root,
|
||||
embedder=_StubEmbedder(),
|
||||
tokenizer=build_tokenizer(),
|
||||
config=CascadeConfig(
|
||||
scan_interval_seconds=60.0,
|
||||
|
||||
@@ -83,6 +83,12 @@ async def cascade_runtime(
|
||||
monkeypatch.setenv("EVEROS_EMBEDDING__MODEL", "stub-model")
|
||||
monkeypatch.setenv("EVEROS_EMBEDDING__BASE_URL", "http://stub.invalid/v1")
|
||||
monkeypatch.setenv("EVEROS_EMBEDDING__API_KEY", "stub-key")
|
||||
# Handlers fetch the embedder lazily via ``get_embedding_capability()``
|
||||
# rather than through ``CascadeOrchestrator`` — patch the process-wide
|
||||
# singleton so cascade never hits the fake network target above.
|
||||
# ``test_lap_append_during_handler_no_loss`` re-patches with a slow
|
||||
# variant to force a handler-in-flight race.
|
||||
_patch_embedding_capability(monkeypatch, _StubEmbedder())
|
||||
|
||||
await dispose_connection()
|
||||
await dispose_engine()
|
||||
@@ -93,18 +99,27 @@ async def cascade_runtime(
|
||||
await ensure_business_indexes()
|
||||
(tmp_path / "ome.toml").write_text("# test\n")
|
||||
|
||||
yield MemoryRoot.default()
|
||||
yield MemoryRoot.resolve()
|
||||
|
||||
await dispose_connection()
|
||||
await dispose_engine()
|
||||
|
||||
|
||||
def _patch_embedding_capability(
|
||||
monkeypatch: pytest.MonkeyPatch, embedder: EmbeddingProvider
|
||||
) -> None:
|
||||
"""Patch the process-wide embedding capability singleton to *embedder*."""
|
||||
import everos.component.embedding.accessor as acc
|
||||
from everos.component.embedding import EmbeddingCapability
|
||||
|
||||
monkeypatch.setattr(acc, "_capability", EmbeddingCapability(provider=embedder))
|
||||
|
||||
|
||||
def _build_orchestrator(
|
||||
memory_root: MemoryRoot, *, scan_interval: float = 60.0
|
||||
) -> CascadeOrchestrator:
|
||||
return CascadeOrchestrator(
|
||||
memory_root=memory_root,
|
||||
embedder=_StubEmbedder(),
|
||||
tokenizer=build_tokenizer(),
|
||||
config=CascadeConfig(
|
||||
scan_interval_seconds=scan_interval,
|
||||
@@ -486,6 +501,7 @@ async def test_concurrent_writes_different_owners_no_bleed(
|
||||
|
||||
async def test_lap_append_during_handler_no_loss(
|
||||
cascade_runtime: MemoryRoot,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Writer keeps appending while worker is mid-handler.
|
||||
|
||||
@@ -500,9 +516,12 @@ async def test_lap_append_during_handler_no_loss(
|
||||
await asyncio.sleep(0.05) # handler takes ~0.05*N entries
|
||||
return [0.0] * self.dim
|
||||
|
||||
# Override the fixture's default stub — this test needs latency to
|
||||
# force a handler invocation to overlap later writer appends.
|
||||
_patch_embedding_capability(monkeypatch, _SlowEmbedder())
|
||||
|
||||
orchestrator = CascadeOrchestrator(
|
||||
memory_root=memory_root,
|
||||
embedder=_SlowEmbedder(),
|
||||
tokenizer=build_tokenizer(),
|
||||
config=CascadeConfig(
|
||||
scan_interval_seconds=60.0,
|
||||
@@ -564,7 +583,6 @@ def _build_orchestrator_fast_scanner(memory_root: MemoryRoot) -> CascadeOrchestr
|
||||
don't wait 30s for the fallback path."""
|
||||
return CascadeOrchestrator(
|
||||
memory_root=memory_root,
|
||||
embedder=_StubEmbedder(),
|
||||
tokenizer=build_tokenizer(),
|
||||
config=CascadeConfig(
|
||||
scan_interval_seconds=2.0,
|
||||
|
||||
@@ -0,0 +1,315 @@
|
||||
"""Integration tests for Task 24's ``cascade backfill`` polish pass.
|
||||
|
||||
Covers three behaviours layered on top of Task 20-23's scaffold + three
|
||||
phase implementations:
|
||||
|
||||
- **Summary block**: after ``run_backfill`` finishes (or is interrupted),
|
||||
a consolidated "Backfill summary" section is printed, one line per
|
||||
phase that actually ran, plus a final ``Exit: <LABEL> (<code>)`` line.
|
||||
- **Ctrl-C handling**: an interrupt raised from inside a phase body must
|
||||
propagate up to ``run_backfill``'s own top-level handler (not be
|
||||
swallowed mid-phase), which prints a resume hint and returns exit
|
||||
code 130 (SIGINT convention: 128 + 2) instead of leaking a bare
|
||||
traceback. Covered at two levels: a synchronous ``KeyboardInterrupt``
|
||||
raise (cheap unit-level regression guard, but *not* representative of
|
||||
a real terminal Ctrl-C — see below), and the actual mechanism a real
|
||||
SIGINT uses under ``asyncio.run`` on Python 3.11+ — ``asyncio.Runner``
|
||||
translates it into ``main_task.cancel()``, which raises
|
||||
``asyncio.CancelledError`` (a ``BaseException``, not a
|
||||
``KeyboardInterrupt`` subclass) at the currently suspended ``await``.
|
||||
A bare ``except KeyboardInterrupt`` alone never sees that exception,
|
||||
so the CancelledError path is exercised both by simulation (raising
|
||||
``CancelledError`` from inside an ``await``) and by delivering a real
|
||||
``SIGINT`` to the test process while a mocked phase is parked in
|
||||
``asyncio.sleep``.
|
||||
- **``--help`` polish**: the CLI help text names every ``--phase`` choice
|
||||
and explains what ``--yes`` does.
|
||||
|
||||
Phase 1 (``vectors``) is reused for the "real counts" summary case
|
||||
because it is the cheapest phase to drive end-to-end (mirrors
|
||||
``backfill_runtime`` in ``test_backfill_phase1.py``); Phases 2/3 are
|
||||
exercised in the all-zero case only, since Task 24 doesn't add any new
|
||||
phase behaviour of its own.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import hashlib
|
||||
import os
|
||||
import signal
|
||||
import threading
|
||||
import time
|
||||
from collections.abc import AsyncIterator
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from everos.component.embedding import EmbeddingCapability, EmbeddingProvider
|
||||
from everos.component.utils.datetime import get_utc_now
|
||||
from everos.config import load_settings
|
||||
from everos.entrypoints.cli.commands import cascade as cascade_mod
|
||||
from everos.entrypoints.cli.commands._backfill_cmd import run_backfill
|
||||
from everos.infra.persistence.lancedb import Episode, dispose_connection, episode_repo
|
||||
from everos.memory.cascade import _backfill as backfill_mod
|
||||
|
||||
_DIM = 1024
|
||||
|
||||
|
||||
class _StubEmbedder(EmbeddingProvider):
|
||||
dim = _DIM
|
||||
|
||||
async def embed(self, text: str) -> list[float]:
|
||||
return [float(len(text) % 7)] * self.dim
|
||||
|
||||
async def embed_batch(self, texts): # type: ignore[no-untyped-def]
|
||||
return [[float(len(t) % 7)] * self.dim for t in texts]
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
async def backfill_runtime(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> AsyncIterator[Path]:
|
||||
"""Tmp memory root + stub embedder (mirrors ``test_backfill_phase1.py``)."""
|
||||
monkeypatch.setenv("EVEROS_ROOT", str(tmp_path))
|
||||
load_settings.cache_clear()
|
||||
await dispose_connection()
|
||||
|
||||
import everos.component.embedding.accessor as acc
|
||||
|
||||
monkeypatch.setattr(
|
||||
acc, "_capability", EmbeddingCapability(provider=_StubEmbedder())
|
||||
)
|
||||
|
||||
yield tmp_path
|
||||
await dispose_connection()
|
||||
|
||||
|
||||
def _episode(entry_id: str) -> Episode:
|
||||
return Episode(
|
||||
id=f"u1_{entry_id}",
|
||||
entry_id=entry_id,
|
||||
owner_id="u1",
|
||||
owner_type="user",
|
||||
timestamp=get_utc_now(),
|
||||
parent_id="mc1",
|
||||
sender_ids=["u1"],
|
||||
episode=f"episode body {entry_id}",
|
||||
episode_tokens=f"episode body {entry_id}",
|
||||
md_path="users/u1/episodes/episode-2026-01-01.md",
|
||||
content_sha256=hashlib.sha256(entry_id.encode()).hexdigest(),
|
||||
vector=None,
|
||||
)
|
||||
|
||||
|
||||
# ── summary block ────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
async def test_summary_block_all_zero_on_empty_db(
|
||||
backfill_runtime: Path, capsys: pytest.CaptureFixture[str]
|
||||
) -> None:
|
||||
code = await run_backfill(phase="all", auto_yes=True)
|
||||
out = capsys.readouterr().out
|
||||
|
||||
assert code == 0
|
||||
assert "Backfill summary" in out
|
||||
assert "Phase 1 (vectors)" in out
|
||||
assert "0 rows / 0 failed / 0 tokens" in out
|
||||
assert "Phase 2 (clusters)" in out
|
||||
assert "0 events emitted / 0 clusters created" in out
|
||||
assert "Phase 3 (skills)" in out
|
||||
assert "0 agent cases processed / 0 skills extracted" in out
|
||||
assert "Exit: SUCCESS (0)" in out
|
||||
|
||||
|
||||
async def test_summary_block_reports_real_counts_for_ran_phase_only(
|
||||
backfill_runtime: Path, capsys: pytest.CaptureFixture[str]
|
||||
) -> None:
|
||||
await episode_repo.add([_episode("ep1"), _episode("ep2")])
|
||||
|
||||
code = await run_backfill(phase="vectors", auto_yes=True)
|
||||
out = capsys.readouterr().out
|
||||
|
||||
assert code == 0
|
||||
assert "Backfill summary" in out
|
||||
assert "Phase 1 (vectors)" in out
|
||||
assert "2 rows / 0 failed" in out
|
||||
# Only the phase that actually ran gets a summary line.
|
||||
assert "Phase 2 (clusters)" not in out
|
||||
assert "Phase 3 (skills)" not in out
|
||||
assert "Exit: SUCCESS (0)" in out
|
||||
|
||||
|
||||
# ── Ctrl-C handling ──────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
async def test_synchronous_keyboard_interrupt_mid_phase_returns_130_with_resume_hint(
|
||||
backfill_runtime: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
capsys: pytest.CaptureFixture[str],
|
||||
) -> None:
|
||||
"""Cheap regression guard for the raw ``except KeyboardInterrupt`` catch.
|
||||
|
||||
This raises synchronously (no ``await`` in between), so the
|
||||
exception lands directly in ``run_backfill``'s own frame — it does
|
||||
*not* exercise ``asyncio.Runner``'s SIGINT-to-``CancelledError``
|
||||
translation that a real terminal Ctrl-C goes through. See
|
||||
``test_cancelled_error_mid_phase_returns_130_with_resume_hint`` and
|
||||
``test_real_sigint_during_phase_await_returns_130_with_resume_hint``
|
||||
below for that path.
|
||||
"""
|
||||
|
||||
async def _raise_interrupt(*_args: object, **_kwargs: object) -> None:
|
||||
raise KeyboardInterrupt
|
||||
|
||||
monkeypatch.setattr(backfill_mod, "_run_phase_vectors", _raise_interrupt)
|
||||
|
||||
code = await run_backfill(phase="all", auto_yes=True)
|
||||
out = capsys.readouterr().out
|
||||
|
||||
assert code == 130
|
||||
assert "Interrupted" in out
|
||||
assert "partial progress was written" in out
|
||||
assert "Resume by running" in out
|
||||
assert "everos cascade backfill --phase <phase-name> --yes" in out
|
||||
assert "vectors, clusters, skills, all" in out
|
||||
# The summary block still prints on interrupt, reflecting no progress yet.
|
||||
assert "Backfill summary" in out
|
||||
assert "Exit: INTERRUPTED (130)" in out
|
||||
|
||||
|
||||
async def test_cancelled_error_mid_phase_returns_130_with_resume_hint(
|
||||
backfill_runtime: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
capsys: pytest.CaptureFixture[str],
|
||||
) -> None:
|
||||
"""Simulates what ``asyncio.Runner`` actually does on a real SIGINT.
|
||||
|
||||
On Python 3.11+, ``asyncio.Runner`` (which backs ``asyncio.run``)
|
||||
turns a terminal Ctrl-C into ``main_task.cancel()``, which raises
|
||||
``asyncio.CancelledError`` — a ``BaseException``, never a
|
||||
``KeyboardInterrupt`` subclass — at the task's currently suspended
|
||||
``await``. Raising it from inside an ``await`` (rather than
|
||||
synchronously) reproduces that mechanism; a bare
|
||||
``except KeyboardInterrupt`` alone would miss it entirely and let it
|
||||
escape ``run_backfill`` uncaught.
|
||||
"""
|
||||
|
||||
async def _raise_cancelled(*_args: object, **_kwargs: object) -> None:
|
||||
await asyncio.sleep(0)
|
||||
raise asyncio.CancelledError
|
||||
|
||||
monkeypatch.setattr(backfill_mod, "_run_phase_vectors", _raise_cancelled)
|
||||
|
||||
code = await run_backfill(phase="all", auto_yes=True)
|
||||
out = capsys.readouterr().out
|
||||
|
||||
assert code == 130
|
||||
assert "Interrupted" in out
|
||||
assert "partial progress was written" in out
|
||||
assert "Resume by running" in out
|
||||
assert "Backfill summary" in out
|
||||
assert "Exit: INTERRUPTED (130)" in out
|
||||
|
||||
|
||||
def test_backfill_cli_synchronous_keyboard_interrupt_exits_130(
|
||||
backfill_runtime: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""CLI-level counterpart of the synchronous-raise regression guard.
|
||||
|
||||
Same caveat as its ``run_backfill``-level twin above: this does not
|
||||
exercise ``asyncio.Runner``'s SIGINT translation. See
|
||||
``test_real_sigint_during_phase_await_returns_130_with_resume_hint``
|
||||
for the real-signal end-to-end path.
|
||||
"""
|
||||
|
||||
async def _raise_interrupt(*_args: object, **_kwargs: object) -> None:
|
||||
raise KeyboardInterrupt
|
||||
|
||||
monkeypatch.setattr(backfill_mod, "_run_phase_vectors", _raise_interrupt)
|
||||
|
||||
result = CliRunner().invoke(
|
||||
cascade_mod.app, ["backfill", "--phase", "vectors", "--yes"]
|
||||
)
|
||||
|
||||
assert result.exit_code == 130
|
||||
|
||||
|
||||
@pytest.mark.slow
|
||||
def test_real_sigint_during_phase_await_returns_130_with_resume_hint(
|
||||
backfill_runtime: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""End-to-end: a real terminal Ctrl-C, delivered as an actual SIGINT.
|
||||
|
||||
Exercises the path where on Python 3.11+, ``asyncio.Runner`` (which
|
||||
backs ``asyncio.run``, used by the ``backfill`` CLI command)
|
||||
converts a real SIGINT into ``main_task.cancel()``, raising
|
||||
``asyncio.CancelledError`` — not ``KeyboardInterrupt`` — at the
|
||||
suspended ``await``. A background thread sends a real ``SIGINT`` to
|
||||
this test process while a mocked phase is parked in
|
||||
``asyncio.sleep``, reproducing that exact mechanism rather than a
|
||||
synchronous stand-in.
|
||||
|
||||
Runs synchronously (not ``async def``): ``pytest-asyncio`` drives
|
||||
its tests off a bare event loop via ``run_until_complete``, not
|
||||
``asyncio.Runner``, so it would never install the SIGINT handler
|
||||
under test. The CLI command's own ``asyncio.run()`` call is what
|
||||
must install it, so this test must invoke the CLI command directly.
|
||||
|
||||
Marked ``slow`` — and therefore excluded from the default
|
||||
``-m "not slow"`` run — because the ``0.3s`` timer that sends the
|
||||
signal races ``asyncio.Runner``'s SIGINT-handler installation. On
|
||||
a fast host the handler is in place well before ``0.3s`` elapses;
|
||||
on a heavily loaded CI runner the signal can land on pytest's
|
||||
default handler and kill the test process. Opt-in execution keeps
|
||||
the guard available for local repro without destabilising CI.
|
||||
"""
|
||||
|
||||
async def _slow_phase(*_args: object, **_kwargs: object) -> None:
|
||||
await asyncio.sleep(5)
|
||||
|
||||
monkeypatch.setattr(backfill_mod, "_run_phase_vectors", _slow_phase)
|
||||
|
||||
def _send_sigint_soon() -> None:
|
||||
time.sleep(0.3)
|
||||
os.kill(os.getpid(), signal.SIGINT)
|
||||
|
||||
threading.Thread(target=_send_sigint_soon, daemon=True).start()
|
||||
|
||||
result = CliRunner().invoke(
|
||||
cascade_mod.app, ["backfill", "--phase", "vectors", "--yes"]
|
||||
)
|
||||
|
||||
assert result.exit_code == 130
|
||||
assert "Interrupted" in result.stdout
|
||||
assert "Resume by running" in result.stdout
|
||||
|
||||
|
||||
# ── --help polish ────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def test_backfill_help_documents_every_phase_name() -> None:
|
||||
result = CliRunner().invoke(cascade_mod.app, ["backfill", "--help"])
|
||||
assert result.exit_code == 0
|
||||
for phase_name in ("vectors", "clusters", "skills", "all"):
|
||||
assert phase_name in result.stdout
|
||||
|
||||
|
||||
def test_backfill_help_documents_yes_flag_behavior() -> None:
|
||||
result = CliRunner().invoke(cascade_mod.app, ["backfill", "--help"])
|
||||
assert result.exit_code == 0
|
||||
assert "confirm" in result.stdout.lower()
|
||||
|
||||
|
||||
def test_backfill_help_documents_resumption() -> None:
|
||||
result = CliRunner().invoke(cascade_mod.app, ["backfill", "--help"])
|
||||
assert result.exit_code == 0
|
||||
assert "resum" in result.stdout.lower()
|
||||
|
||||
|
||||
def test_backfill_help_documents_cost_estimate_policy() -> None:
|
||||
result = CliRunner().invoke(cascade_mod.app, ["backfill", "--help"])
|
||||
assert result.exit_code == 0
|
||||
assert "token" in result.stdout.lower()
|
||||
assert "price" in result.stdout.lower() or "pricing" in result.stdout.lower()
|
||||
@@ -0,0 +1,263 @@
|
||||
"""Integration test for ``everos cascade backfill --phase vectors``.
|
||||
|
||||
Exercises the real Phase 1 re-embed path against actual LanceDB tables
|
||||
under a tmp memory root: seeds rows with ``vector IS NULL``, drives
|
||||
``run_backfill(phase="vectors", ...)``, and asserts the rows now carry
|
||||
a vector. A stub embedder replaces the process-wide embedding
|
||||
capability singleton so no real network call ever happens (mirrors the
|
||||
pattern in ``tests/integration/test_cascade_all_kinds_consistency.py``).
|
||||
|
||||
Covers: empty DB (nothing to backfill), declined confirmation (nothing
|
||||
written, exit 1), a small DB with rows that need backfilling (rows
|
||||
updated, pre-existing vectors left untouched, token estimate rendered
|
||||
in K/M notation), a batch-embed failure that is tallied without
|
||||
aborting the phase, an episode's ``subject_vector`` secondary embed,
|
||||
and a missing embedding capability returning exit 2.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
from collections.abc import AsyncIterator
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from everos.component.embedding import EmbeddingCapability, EmbeddingProvider
|
||||
from everos.component.utils.datetime import get_utc_now
|
||||
from everos.config import load_settings
|
||||
from everos.entrypoints.cli.commands._backfill_cmd import run_backfill
|
||||
from everos.infra.persistence.lancedb import (
|
||||
AtomicFact,
|
||||
Episode,
|
||||
atomic_fact_repo,
|
||||
dispose_connection,
|
||||
episode_repo,
|
||||
)
|
||||
|
||||
_DIM = 1024
|
||||
|
||||
|
||||
class _StubEmbedder(EmbeddingProvider):
|
||||
dim = _DIM
|
||||
|
||||
async def embed(self, text: str) -> list[float]:
|
||||
return [float(len(text) % 7)] * self.dim
|
||||
|
||||
async def embed_batch(self, texts): # type: ignore[no-untyped-def]
|
||||
return [[float(len(t) % 7)] * self.dim for t in texts]
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
async def backfill_runtime(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> AsyncIterator[Path]:
|
||||
"""Tmp memory root + stub embedder; disposes the lancedb singleton
|
||||
around the test so no stale connection from a neighbouring test
|
||||
leaks in (mirrors ``cascade_runtime`` in the cascade test suite)."""
|
||||
monkeypatch.setenv("EVEROS_ROOT", str(tmp_path))
|
||||
load_settings.cache_clear()
|
||||
await dispose_connection()
|
||||
|
||||
import everos.component.embedding.accessor as acc
|
||||
|
||||
monkeypatch.setattr(
|
||||
acc, "_capability", EmbeddingCapability(provider=_StubEmbedder())
|
||||
)
|
||||
|
||||
yield tmp_path
|
||||
await dispose_connection()
|
||||
|
||||
|
||||
def _episode(
|
||||
entry_id: str,
|
||||
*,
|
||||
text: str = "",
|
||||
vector: list[float] | None = None,
|
||||
subject: str | None = None,
|
||||
subject_vector: list[float] | None = None,
|
||||
) -> Episode:
|
||||
body = text or f"episode body {entry_id}"
|
||||
return Episode(
|
||||
id=f"u1_{entry_id}",
|
||||
entry_id=entry_id,
|
||||
owner_id="u1",
|
||||
owner_type="user",
|
||||
timestamp=get_utc_now(),
|
||||
parent_id="mc1",
|
||||
sender_ids=["u1"],
|
||||
subject=subject,
|
||||
episode=body,
|
||||
episode_tokens=body,
|
||||
md_path="users/u1/episodes/episode-2026-01-01.md",
|
||||
content_sha256=hashlib.sha256(entry_id.encode()).hexdigest(),
|
||||
vector=vector,
|
||||
subject_vector=subject_vector,
|
||||
)
|
||||
|
||||
|
||||
def _fact(entry_id: str, *, vector: list[float] | None = None) -> AtomicFact:
|
||||
return AtomicFact(
|
||||
id=f"u1_{entry_id}",
|
||||
entry_id=entry_id,
|
||||
owner_id="u1",
|
||||
owner_type="user",
|
||||
timestamp=get_utc_now(),
|
||||
parent_id="mc1",
|
||||
sender_ids=["u1"],
|
||||
fact=f"fact body {entry_id}",
|
||||
fact_tokens=f"fact body {entry_id}",
|
||||
md_path="users/u1/.atomic_facts/atomic_fact-2026-01-01.md",
|
||||
content_sha256=hashlib.sha256(entry_id.encode()).hexdigest(),
|
||||
vector=vector,
|
||||
)
|
||||
|
||||
|
||||
async def test_empty_db_reports_nothing_to_backfill(
|
||||
backfill_runtime: Path, capsys: pytest.CaptureFixture[str]
|
||||
) -> None:
|
||||
code = await run_backfill(phase="vectors", auto_yes=True)
|
||||
out = capsys.readouterr().out
|
||||
|
||||
assert code == 0
|
||||
assert "Nothing to backfill" in out
|
||||
|
||||
|
||||
async def test_declined_confirmation_writes_nothing_and_exits_one(
|
||||
backfill_runtime: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
await episode_repo.add([_episode("ep1")])
|
||||
monkeypatch.setattr("typer.confirm", lambda *a, **k: False)
|
||||
|
||||
code = await run_backfill(phase="vectors", auto_yes=False)
|
||||
|
||||
assert code == 1
|
||||
row = await episode_repo.get_by_id("u1_ep1")
|
||||
assert row is not None
|
||||
assert row.vector is None
|
||||
|
||||
|
||||
async def test_small_db_backfills_null_vector_rows_only(
|
||||
backfill_runtime: Path, capsys: pytest.CaptureFixture[str]
|
||||
) -> None:
|
||||
pre_embedded = [0.5] * _DIM
|
||||
await episode_repo.add([_episode("ep1"), _episode("ep2", vector=pre_embedded)])
|
||||
await atomic_fact_repo.add([_fact("af1")])
|
||||
|
||||
code = await run_backfill(phase="vectors", auto_yes=True)
|
||||
out = capsys.readouterr().out
|
||||
|
||||
assert code == 0
|
||||
ep1 = await episode_repo.get_by_id("u1_ep1")
|
||||
ep2 = await episode_repo.get_by_id("u1_ep2")
|
||||
af1 = await atomic_fact_repo.get_by_id("u1_af1")
|
||||
assert ep1 is not None and ep1.vector is not None
|
||||
assert ep2 is not None and ep2.vector == pytest.approx(pre_embedded)
|
||||
assert af1 is not None and af1.vector is not None
|
||||
assert "memories to process: 2" in out
|
||||
assert "phase 1 complete" in out
|
||||
|
||||
|
||||
async def test_token_estimate_uses_kilo_notation(
|
||||
backfill_runtime: Path, capsys: pytest.CaptureFixture[str]
|
||||
) -> None:
|
||||
long_text = "word " * 3000
|
||||
await episode_repo.add([_episode("epbig", text=long_text)])
|
||||
|
||||
code = await run_backfill(phase="vectors", auto_yes=True)
|
||||
out = capsys.readouterr().out
|
||||
|
||||
assert code == 0
|
||||
assert "input tokens:" in out
|
||||
assert "K" in out
|
||||
|
||||
|
||||
async def test_failed_row_is_tallied_and_does_not_abort_the_phase(
|
||||
backfill_runtime: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
capsys: pytest.CaptureFixture[str],
|
||||
) -> None:
|
||||
"""Round-2 finding M8: a batch failure now falls back to per-row
|
||||
``embed(...)`` calls (see ``_embed_primary_batch``). A poison row —
|
||||
one whose per-row retry also fails — must (a) not abort the phase,
|
||||
(b) not touch the healthy rows sharing its batch, and (c) drive the
|
||||
final exit code to ``4`` (COMPLETED_WITH_FAILURES) rather than ``0``
|
||||
so automation can distinguish clean success from partial success.
|
||||
|
||||
``_FlakyEmbedder`` fails BOTH ``embed_batch`` and ``embed`` on the
|
||||
poison text ("boom"), so the row remains unwritable after fallback.
|
||||
Healthy rows still succeed via the per-row retry path.
|
||||
"""
|
||||
import everos.component.embedding.accessor as acc
|
||||
from everos.component.embedding import EmbeddingCapability
|
||||
|
||||
class _FlakyEmbedder(_StubEmbedder):
|
||||
async def embed_batch(self, texts: list[str]) -> list[list[float]]:
|
||||
if any("boom" in t for t in texts):
|
||||
raise RuntimeError("simulated batch embed failure")
|
||||
return await super().embed_batch(texts)
|
||||
|
||||
async def embed(self, text: str) -> list[float]:
|
||||
if "boom" in text:
|
||||
raise RuntimeError("simulated per-row embed failure")
|
||||
return await super().embed(text)
|
||||
|
||||
monkeypatch.setattr(
|
||||
acc, "_capability", EmbeddingCapability(provider=_FlakyEmbedder())
|
||||
)
|
||||
await episode_repo.add(
|
||||
[_episode("epok", text="fine content"), _episode("epbad", text="boom content")]
|
||||
)
|
||||
|
||||
code = await run_backfill(phase="vectors", auto_yes=True)
|
||||
out = capsys.readouterr().out
|
||||
|
||||
assert code == 4
|
||||
ok_row = await episode_repo.get_by_id("u1_epok")
|
||||
bad_row = await episode_repo.get_by_id("u1_epbad")
|
||||
assert ok_row is not None and ok_row.vector is not None
|
||||
assert bad_row is not None and bad_row.vector is None
|
||||
assert "phase 1 complete" in out
|
||||
assert "COMPLETED_WITH_FAILURES" in out
|
||||
assert "1 rows failed embedding" in out
|
||||
|
||||
|
||||
async def test_episode_subject_vector_also_backfilled(
|
||||
backfill_runtime: Path, capsys: pytest.CaptureFixture[str]
|
||||
) -> None:
|
||||
"""Episode is the only table with a second, independent embed
|
||||
(``subject_vector`` — see ``EpisodeHandler._build_row``). A row that
|
||||
has a ``subject`` must come out of Phase 1 with *both* vectors set,
|
||||
exercising ``_embed_subject_batch``'s secondary ``embed_batch`` call.
|
||||
"""
|
||||
await episode_repo.add(
|
||||
[_episode("epsub", text="episode body", subject="What is the meaning of X?")]
|
||||
)
|
||||
|
||||
code = await run_backfill(phase="vectors", auto_yes=True)
|
||||
out = capsys.readouterr().out
|
||||
|
||||
assert code == 0
|
||||
row = await episode_repo.get_by_id("u1_epsub")
|
||||
assert row is not None
|
||||
assert row.vector is not None and len(row.vector) == _DIM
|
||||
assert row.subject_vector is not None and len(row.subject_vector) == _DIM
|
||||
assert "phase 1 complete" in out
|
||||
|
||||
|
||||
async def test_missing_embedding_capability_returns_exit_two(
|
||||
backfill_runtime: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""When the embedding capability is unavailable (no provider
|
||||
configured), Phase 1 must fail cleanly with exit 2 rather than
|
||||
silently skipping rows or crashing uncaught."""
|
||||
import everos.component.embedding.accessor as acc
|
||||
|
||||
await episode_repo.add([_episode("ep1")])
|
||||
monkeypatch.setattr(acc, "_capability", EmbeddingCapability(provider=None))
|
||||
|
||||
code = await run_backfill(phase="vectors", auto_yes=True)
|
||||
|
||||
assert code == 2
|
||||
row = await episode_repo.get_by_id("u1_ep1")
|
||||
assert row is not None and row.vector is None
|
||||
@@ -0,0 +1,314 @@
|
||||
"""Integration test for ``everos cascade backfill --phase clusters``.
|
||||
|
||||
Exercises the real Phase 2 scan + synthetic-event-emission path (see
|
||||
``memory.cascade._backfill._run_phase_clusters``): seeds real episode /
|
||||
agent-case rows in LanceDB, drives ``run_backfill(phase="clusters", ...)``,
|
||||
and asserts the phase synthesizes exactly one ``EpisodeExtracted`` /
|
||||
``AgentCaseExtracted`` event per row and fans them into the OME engine.
|
||||
|
||||
The OME engine itself is replaced with an in-memory spy
|
||||
(``_FakeClusterEngine``, injected via monkeypatching
|
||||
``_backfill._build_cluster_engine``) so this test exercises the scan +
|
||||
event-synthesis contract without paying for a real APScheduler + sqlite
|
||||
jobstore runtime — that end-to-end wiring is already covered by
|
||||
``tests/integration/test_ome_strategies_integration.py``.
|
||||
|
||||
Covers: empty DB (nothing to backfill, no engine touched), declined
|
||||
confirmation (no events emitted, exit 1), a small DB with episodes +
|
||||
agent cases (correct event types / counts / fields, exit 0), and a
|
||||
missing embedding capability (exit 2, mirroring Phase 1's guard).
|
||||
|
||||
One test (``test_real_offline_engine_grows_cluster_count``) does NOT
|
||||
install the fake spy: it drives Phase 2 through the actual
|
||||
``_build_cluster_engine`` path — a real ``OfflineEngine`` + APScheduler
|
||||
runtime with ``trigger_profile_clustering`` registered — and asserts
|
||||
``cluster_repo.count()`` grows. This is the one check the fake-engine
|
||||
tests above cannot make: they prove the scan + event-synthesis contract,
|
||||
but never exercise whether the real engine wiring actually drives a
|
||||
cluster into existence.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
from collections.abc import AsyncIterator
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from everos.component.embedding import EmbeddingCapability, EmbeddingProvider
|
||||
from everos.component.utils.datetime import get_utc_now
|
||||
from everos.config import load_settings
|
||||
from everos.entrypoints.cli.commands._backfill_cmd import run_backfill
|
||||
from everos.infra.ome.events import BaseEvent
|
||||
from everos.infra.persistence.lancedb import (
|
||||
AgentCase,
|
||||
Episode,
|
||||
agent_case_repo,
|
||||
dispose_connection,
|
||||
episode_repo,
|
||||
)
|
||||
from everos.infra.persistence.sqlite import cluster_repo, sqlite_manager
|
||||
from everos.memory.cascade import _backfill as backfill_mod
|
||||
from everos.memory.events import AgentCaseExtracted, EpisodeExtracted
|
||||
|
||||
_DIM = 1024
|
||||
|
||||
|
||||
class _StubEmbedder(EmbeddingProvider):
|
||||
dim = _DIM
|
||||
|
||||
async def embed(self, text: str) -> list[float]:
|
||||
return [float(len(text) % 7)] * self.dim
|
||||
|
||||
async def embed_batch(self, texts): # type: ignore[no-untyped-def]
|
||||
return [[float(len(t) % 7)] * self.dim for t in texts]
|
||||
|
||||
|
||||
class _FakeClusterEngine:
|
||||
"""Spy replacing the real OME engine: records emitted events instead of
|
||||
dispatching them through APScheduler."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.emitted: list[BaseEvent] = []
|
||||
self.started = False
|
||||
self.stopped = False
|
||||
|
||||
async def start(self) -> None:
|
||||
self.started = True
|
||||
|
||||
async def emit(self, event: BaseEvent) -> None:
|
||||
self.emitted.append(event)
|
||||
|
||||
async def wait_idle(self, *, timeout: float = 30.0) -> bool: # noqa: ASYNC109
|
||||
return True
|
||||
|
||||
async def stop(self) -> None:
|
||||
self.stopped = True
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
async def backfill_runtime(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> AsyncIterator[Path]:
|
||||
"""Tmp memory root + stub embedder + isolated sqlite engine.
|
||||
|
||||
Phase 2 touches both LanceDB (episode / agent_case scan) and sqlite
|
||||
(``cluster_repo.count()``), so both process-wide singletons are
|
||||
disposed and rebuilt against ``tmp_path`` around each test — mirrors
|
||||
``backfill_runtime`` in ``test_backfill_phase1.py`` plus the sqlite
|
||||
reset from ``test_ome_strategies_integration.py``.
|
||||
"""
|
||||
monkeypatch.setenv("EVEROS_ROOT", str(tmp_path))
|
||||
load_settings.cache_clear()
|
||||
await dispose_connection()
|
||||
if sqlite_manager._engine is not None:
|
||||
await sqlite_manager.dispose_engine()
|
||||
monkeypatch.setattr(sqlite_manager, "_engine", None, raising=False)
|
||||
monkeypatch.setattr(sqlite_manager, "_session_factory", None, raising=False)
|
||||
|
||||
import everos.component.embedding.accessor as acc
|
||||
|
||||
monkeypatch.setattr(
|
||||
acc, "_capability", EmbeddingCapability(provider=_StubEmbedder())
|
||||
)
|
||||
|
||||
yield tmp_path
|
||||
|
||||
await dispose_connection()
|
||||
if sqlite_manager._engine is not None:
|
||||
await sqlite_manager.dispose_engine()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def fake_engine(monkeypatch: pytest.MonkeyPatch) -> _FakeClusterEngine:
|
||||
"""Install the spy engine in place of ``_build_cluster_engine``."""
|
||||
engine = _FakeClusterEngine()
|
||||
monkeypatch.setattr(backfill_mod, "_build_cluster_engine", lambda: engine)
|
||||
return engine
|
||||
|
||||
|
||||
def _episode(entry_id: str, *, owner_id: str = "u1") -> Episode:
|
||||
return Episode(
|
||||
id=f"{owner_id}_{entry_id}",
|
||||
entry_id=entry_id,
|
||||
owner_id=owner_id,
|
||||
owner_type="user",
|
||||
session_id="s1",
|
||||
timestamp=get_utc_now(),
|
||||
parent_id=f"mc_{entry_id}",
|
||||
sender_ids=[owner_id],
|
||||
episode=f"episode body {entry_id}",
|
||||
episode_tokens=f"episode body {entry_id}",
|
||||
md_path="users/u1/episodes/episode-2026-01-01.md",
|
||||
content_sha256=hashlib.sha256(entry_id.encode()).hexdigest(),
|
||||
vector=[0.1] * _DIM,
|
||||
)
|
||||
|
||||
|
||||
def _agent_case(entry_id: str, *, agent_id: str = "agent1") -> AgentCase:
|
||||
return AgentCase(
|
||||
id=f"{agent_id}_{entry_id}",
|
||||
entry_id=entry_id,
|
||||
owner_id=agent_id,
|
||||
owner_type="agent",
|
||||
session_id="s1",
|
||||
timestamp=get_utc_now(),
|
||||
parent_id=f"mc_{entry_id}",
|
||||
quality_score=0.8,
|
||||
task_intent=f"task intent {entry_id}",
|
||||
task_intent_tokens=f"task intent {entry_id}",
|
||||
approach=f"approach {entry_id}",
|
||||
approach_tokens=f"approach {entry_id}",
|
||||
md_path="agents/agent1/.cases/agent_case-2026-01-01.md",
|
||||
content_sha256=hashlib.sha256(entry_id.encode()).hexdigest(),
|
||||
vector=[0.1] * _DIM,
|
||||
)
|
||||
|
||||
|
||||
async def test_empty_db_reports_nothing_to_backfill(
|
||||
backfill_runtime: Path,
|
||||
fake_engine: _FakeClusterEngine,
|
||||
capsys: pytest.CaptureFixture[str],
|
||||
) -> None:
|
||||
code = await run_backfill(phase="clusters", auto_yes=True)
|
||||
out = capsys.readouterr().out
|
||||
|
||||
assert code == 0
|
||||
assert "Nothing to backfill" in out
|
||||
assert fake_engine.emitted == []
|
||||
assert fake_engine.started is False
|
||||
|
||||
|
||||
async def test_declined_confirmation_emits_nothing_and_exits_one(
|
||||
backfill_runtime: Path,
|
||||
fake_engine: _FakeClusterEngine,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
await episode_repo.add([_episode("ep1")])
|
||||
monkeypatch.setattr("typer.confirm", lambda *a, **k: False)
|
||||
|
||||
code = await run_backfill(phase="clusters", auto_yes=False)
|
||||
|
||||
assert code == 1
|
||||
assert fake_engine.emitted == []
|
||||
assert fake_engine.started is False
|
||||
|
||||
|
||||
async def test_episodes_and_cases_synthesize_matching_events(
|
||||
backfill_runtime: Path,
|
||||
fake_engine: _FakeClusterEngine,
|
||||
capsys: pytest.CaptureFixture[str],
|
||||
) -> None:
|
||||
await episode_repo.add([_episode("ep1"), _episode("ep2")])
|
||||
await agent_case_repo.add([_agent_case("ac1")])
|
||||
|
||||
code = await run_backfill(phase="clusters", auto_yes=True)
|
||||
out = capsys.readouterr().out
|
||||
|
||||
assert code == 0
|
||||
assert fake_engine.started is True
|
||||
assert fake_engine.stopped is True
|
||||
assert len(fake_engine.emitted) == 3
|
||||
|
||||
episode_events = [e for e in fake_engine.emitted if isinstance(e, EpisodeExtracted)]
|
||||
case_events = [e for e in fake_engine.emitted if isinstance(e, AgentCaseExtracted)]
|
||||
assert len(episode_events) == 2
|
||||
assert len(case_events) == 1
|
||||
|
||||
assert all(e.source == "pipeline" for e in episode_events)
|
||||
assert all(e.event_id.startswith("backfill_") for e in fake_engine.emitted)
|
||||
assert {e.owner_id for e in episode_events} == {"u1"}
|
||||
assert case_events[0].agent_id == "agent1"
|
||||
assert case_events[0].task_intent == "task intent ac1"
|
||||
assert case_events[0].quality_score == pytest.approx(0.8)
|
||||
|
||||
assert "phase 2 complete" in out
|
||||
|
||||
|
||||
async def test_missing_embedding_capability_returns_exit_two(
|
||||
backfill_runtime: Path,
|
||||
fake_engine: _FakeClusterEngine,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Mirrors Phase 1's own guard: no embedding provider configured means
|
||||
the strategies can't re-embed, so the phase fails clean with exit 2
|
||||
rather than emitting events into strategies that will crash."""
|
||||
import everos.component.embedding.accessor as acc
|
||||
|
||||
await episode_repo.add([_episode("ep1")])
|
||||
monkeypatch.setattr(acc, "_capability", EmbeddingCapability(provider=None))
|
||||
|
||||
code = await run_backfill(phase="clusters", auto_yes=True)
|
||||
|
||||
assert code == 2
|
||||
assert fake_engine.emitted == []
|
||||
assert fake_engine.started is False
|
||||
|
||||
|
||||
async def test_reflection_merged_episodes_excluded_from_synthesis(
|
||||
backfill_runtime: Path,
|
||||
fake_engine: _FakeClusterEngine,
|
||||
) -> None:
|
||||
"""Reflection-merged episodes (``parent_type="cluster"``) must not be
|
||||
re-synthesized as pipeline events.
|
||||
|
||||
``trigger_profile_clustering`` excludes Reflection output via
|
||||
``applies_to=lambda e: e.source == "pipeline"`` — but that guard is
|
||||
defeated if the Episode scan hands it a synthetic event whose
|
||||
``memcell_id`` is actually a cluster id (see Important 1 review
|
||||
finding on ``_backfill._scan_all_rows`` / ``_run_phase_clusters``).
|
||||
Only the pipeline-sourced row should reach the fake engine.
|
||||
"""
|
||||
pipeline_episode = _episode("ep1")
|
||||
merged_episode = _episode("ep_merged").model_copy(
|
||||
update={
|
||||
"id": "u1_ep_merged",
|
||||
"parent_type": "cluster",
|
||||
"parent_id": "cluster_abc123",
|
||||
}
|
||||
)
|
||||
await episode_repo.add([pipeline_episode, merged_episode])
|
||||
|
||||
code = await run_backfill(phase="clusters", auto_yes=True)
|
||||
|
||||
assert code == 0
|
||||
episode_events = [e for e in fake_engine.emitted if isinstance(e, EpisodeExtracted)]
|
||||
assert len(episode_events) == 1
|
||||
assert episode_events[0].memcell_id == pipeline_episode.parent_id
|
||||
|
||||
|
||||
async def test_real_offline_engine_grows_cluster_count(
|
||||
backfill_runtime: Path,
|
||||
) -> None:
|
||||
"""Phase 2 through the REAL ``_build_cluster_engine`` — no fake spy.
|
||||
|
||||
Deliberately does not use the ``fake_engine`` fixture: it drives an
|
||||
actual ``OfflineEngine`` + APScheduler runtime with
|
||||
``trigger_profile_clustering`` registered (see
|
||||
``_backfill._build_cluster_engine``), proving the phase's own engine
|
||||
wiring — not just the scan/synthesis contract the other tests in
|
||||
this module pin — actually grows ``cluster_repo.count()``.
|
||||
|
||||
``trigger_profile_clustering`` re-embeds each episode's text via
|
||||
``get_embedding_capability().require()`` — which ``backfill_runtime``
|
||||
already stubs with the deterministic ``_StubEmbedder``, so no
|
||||
per-strategy patch is needed here. No agent cases are seeded, so
|
||||
``trigger_skill_clustering`` (the engine's other registered
|
||||
strategy, whose ``cluster_by_llm`` fallback needs an LLM) never
|
||||
fires, keeping this test LLM-free.
|
||||
"""
|
||||
await episode_repo.add([_episode(f"ep{i}") for i in range(5)])
|
||||
|
||||
# ``run_backfill`` itself creates the ``cluster`` table lazily (see
|
||||
# ``_ensure_cluster_schema``) — call it here too so the "before" count
|
||||
# can be taken without racing that lazy create.
|
||||
await backfill_mod._ensure_cluster_schema()
|
||||
clusters_before = await cluster_repo.count()
|
||||
assert clusters_before == 0
|
||||
|
||||
code = await run_backfill(phase="clusters", auto_yes=True)
|
||||
|
||||
clusters_after = await cluster_repo.count()
|
||||
assert code == 0
|
||||
assert clusters_after > clusters_before
|
||||
@@ -0,0 +1,257 @@
|
||||
"""Integration test for ``everos cascade backfill --phase skills``.
|
||||
|
||||
Exercises the real Phase 3 scan + synthetic-event-emission + cascade-sync
|
||||
path (see ``memory.cascade._backfill._run_phase_skills``): Phase 2's
|
||||
throwaway engine (``_build_cluster_engine``, Task 22) registers
|
||||
``trigger_skill_clustering`` but never ``extract_agent_skill``, so every
|
||||
``SkillClusterUpdated`` it emits during clustering has no listener and is
|
||||
dropped. Phase 3 walks every agent-case cluster
|
||||
(``cluster_repo.list_distinct_owners`` + ``list_for_owner``) and replays
|
||||
one ``SkillClusterUpdated`` per clustered case through a second throwaway
|
||||
engine that DOES register ``extract_agent_skill``, then runs a one-shot
|
||||
cascade sync (``CascadeOrchestrator.sync_once``) so the freshly-written
|
||||
``SKILL.md`` lands in LanceDB before the phase reports its skill count.
|
||||
|
||||
Only ``AgentSkillExtractor`` (the LLM call) and ``get_llm_client`` are
|
||||
mocked — everything else (cluster lookup, agent-case lookup, the
|
||||
markdown writer, and the cascade scan/handler/embed round trip) is real,
|
||||
so ``test_skills_phase_grows_agent_skill_count`` is a genuine end-to-end
|
||||
proof that Phase 3 leaves ``agent_skill_repo.count()`` already grown,
|
||||
not just eventually-consistent.
|
||||
|
||||
Covers: empty DB (nothing to backfill, no engine touched), declined
|
||||
confirmation (exit 1, no skill written), a small DB with one clustered
|
||||
agent case (skill extracted + indexed, exit 0), and a missing embedding
|
||||
capability (exit 2, mirroring Phase 1 / Phase 2's own guard).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import importlib
|
||||
from collections.abc import AsyncIterator, Iterator
|
||||
from pathlib import Path
|
||||
from unittest.mock import AsyncMock, patch
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
from everalgo.clustering import Cluster as AlgoCluster
|
||||
from everalgo.types import AgentSkill as AlgoAgentSkill
|
||||
|
||||
from everos.component.embedding import EmbeddingCapability, EmbeddingProvider
|
||||
from everos.component.utils.datetime import get_utc_now
|
||||
from everos.config import load_settings
|
||||
from everos.entrypoints.cli.commands._backfill_cmd import run_backfill
|
||||
from everos.infra.persistence.lancedb import (
|
||||
AgentCase,
|
||||
agent_case_repo,
|
||||
agent_skill_repo,
|
||||
dispose_connection,
|
||||
)
|
||||
from everos.infra.persistence.sqlite import (
|
||||
cluster_repo,
|
||||
mint_cluster_id,
|
||||
sqlite_manager,
|
||||
)
|
||||
from everos.memory.cascade import _backfill as backfill_mod
|
||||
|
||||
_DIM = 1024
|
||||
|
||||
|
||||
class _StubEmbedder(EmbeddingProvider):
|
||||
dim = _DIM
|
||||
|
||||
async def embed(self, text: str) -> list[float]:
|
||||
return [float(len(text) % 7)] * self.dim
|
||||
|
||||
async def embed_batch(self, texts): # type: ignore[no-untyped-def]
|
||||
return [[float(len(t) % 7)] * self.dim for t in texts]
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
async def backfill_runtime(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> AsyncIterator[Path]:
|
||||
"""Tmp memory root + stub embedder + isolated sqlite engine.
|
||||
|
||||
Phase 3 touches LanceDB (agent_case / agent_skill), sqlite (cluster
|
||||
tables + md_change_state), and the real filesystem (SKILL.md write +
|
||||
cascade scan) — mirrors ``backfill_runtime`` in
|
||||
``test_backfill_phase2.py``, plus resetting the strategy module's
|
||||
``_writer`` singleton so it re-resolves against ``tmp_path``.
|
||||
"""
|
||||
monkeypatch.setenv("EVEROS_ROOT", str(tmp_path))
|
||||
load_settings.cache_clear()
|
||||
await dispose_connection()
|
||||
if sqlite_manager._engine is not None:
|
||||
await sqlite_manager.dispose_engine()
|
||||
monkeypatch.setattr(sqlite_manager, "_engine", None, raising=False)
|
||||
monkeypatch.setattr(sqlite_manager, "_session_factory", None, raising=False)
|
||||
|
||||
import everos.component.embedding.accessor as acc
|
||||
|
||||
monkeypatch.setattr(
|
||||
acc, "_capability", EmbeddingCapability(provider=_StubEmbedder())
|
||||
)
|
||||
|
||||
skill_mod = importlib.import_module("everos.memory.strategies.extract_agent_skill")
|
||||
monkeypatch.setattr(skill_mod, "_writer", None, raising=False)
|
||||
|
||||
yield tmp_path
|
||||
|
||||
await dispose_connection()
|
||||
if sqlite_manager._engine is not None:
|
||||
await sqlite_manager.dispose_engine()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def stub_llm_extraction(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> Iterator[list[AlgoAgentSkill]]:
|
||||
"""Mock the one non-deterministic seam: the LLM-backed extractor.
|
||||
|
||||
Everything downstream of ``AgentSkillExtractor.aextract`` (writer,
|
||||
cascade scan, embed, LanceDB upsert) stays real.
|
||||
"""
|
||||
emitted = [
|
||||
AlgoAgentSkill(
|
||||
id="dummy",
|
||||
cluster_id="",
|
||||
name="summarise_doc",
|
||||
description="how to summarise a document",
|
||||
content="step 1: read; step 2: condense",
|
||||
confidence=0.7,
|
||||
maturity_score=0.5,
|
||||
source_case_ids=["ac1"],
|
||||
)
|
||||
]
|
||||
# ``strategies/__init__.py`` re-exports the decorated ``Strategy``
|
||||
# object under this same dotted name, shadowing the submodule as a
|
||||
# package attribute — ``importlib.import_module`` (not
|
||||
# ``import ... as``) is required to reach the actual module and
|
||||
# patch its module-level references (mirrors
|
||||
# ``test_ome_strategies_integration.test_real_offline_engine_grows_
|
||||
# cluster_count``).
|
||||
skill_mod = importlib.import_module("everos.memory.strategies.extract_agent_skill")
|
||||
monkeypatch.setattr(skill_mod, "get_llm_client", lambda: object())
|
||||
with patch.object(skill_mod, "AgentSkillExtractor") as mock_extractor_cls:
|
||||
mock_extractor_cls.return_value.aextract = AsyncMock(return_value=emitted)
|
||||
yield emitted
|
||||
|
||||
|
||||
def _agent_case(entry_id: str, *, agent_id: str = "agent1") -> AgentCase:
|
||||
return AgentCase(
|
||||
id=f"{agent_id}_{entry_id}",
|
||||
entry_id=entry_id,
|
||||
owner_id=agent_id,
|
||||
owner_type="agent",
|
||||
session_id="s1",
|
||||
timestamp=get_utc_now(),
|
||||
parent_id=f"mc_{entry_id}",
|
||||
quality_score=0.8,
|
||||
task_intent=f"task intent {entry_id}",
|
||||
task_intent_tokens=f"task intent {entry_id}",
|
||||
approach=f"approach {entry_id}",
|
||||
approach_tokens=f"approach {entry_id}",
|
||||
md_path=f"agents/default/default/{agent_id}/.cases/agent_case-2026-01-01.md",
|
||||
content_sha256=hashlib.sha256(entry_id.encode()).hexdigest(),
|
||||
vector=[0.1] * _DIM,
|
||||
)
|
||||
|
||||
|
||||
async def _seed_cluster(*, agent_id: str = "agent1", case_entry_id: str = "ac1") -> str:
|
||||
"""Seed the sqlite ``cluster`` + ``cluster_member`` pair Phase 3 scans."""
|
||||
await backfill_mod._ensure_cluster_schema()
|
||||
cluster_id = mint_cluster_id()
|
||||
algo_cluster = AlgoCluster(
|
||||
id=cluster_id,
|
||||
centroid=np.zeros(_DIM, dtype=np.float32),
|
||||
count=1,
|
||||
last_ts=1_700_000_000_000,
|
||||
preview=[f"task intent {case_entry_id}"],
|
||||
members=[case_entry_id],
|
||||
)
|
||||
await cluster_repo.upsert_with_members(
|
||||
algo_cluster,
|
||||
owner_id=agent_id,
|
||||
owner_type="agent",
|
||||
kind="agent_case",
|
||||
member_type="case",
|
||||
)
|
||||
return cluster_id
|
||||
|
||||
|
||||
async def test_empty_db_reports_nothing_to_backfill(
|
||||
backfill_runtime: Path,
|
||||
capsys: pytest.CaptureFixture[str],
|
||||
) -> None:
|
||||
await backfill_mod._ensure_cluster_schema()
|
||||
|
||||
code = await run_backfill(phase="skills", auto_yes=True)
|
||||
out = capsys.readouterr().out
|
||||
|
||||
assert code == 0
|
||||
assert "Nothing to backfill" in out
|
||||
assert await agent_skill_repo.count() == 0
|
||||
|
||||
|
||||
async def test_declined_confirmation_writes_nothing_and_exits_one(
|
||||
backfill_runtime: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
await agent_case_repo.add([_agent_case("ac1")])
|
||||
await _seed_cluster()
|
||||
monkeypatch.setattr("typer.confirm", lambda *a, **k: False)
|
||||
|
||||
code = await run_backfill(phase="skills", auto_yes=False)
|
||||
|
||||
assert code == 1
|
||||
assert await agent_skill_repo.count() == 0
|
||||
|
||||
|
||||
async def test_missing_embedding_capability_returns_exit_two(
|
||||
backfill_runtime: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Mirrors Phase 1 / Phase 2's own guard: no embedding provider means
|
||||
``extract_agent_skill``'s own top-K cosine ranking (and the cascade
|
||||
handler's re-embed on write-back) can't run, so the phase fails
|
||||
clean with exit 2 rather than emitting events into a broken pipeline."""
|
||||
import everos.component.embedding.accessor as acc
|
||||
|
||||
await agent_case_repo.add([_agent_case("ac1")])
|
||||
await _seed_cluster()
|
||||
monkeypatch.setattr(acc, "_capability", EmbeddingCapability(provider=None))
|
||||
|
||||
code = await run_backfill(phase="skills", auto_yes=True)
|
||||
|
||||
assert code == 2
|
||||
assert await agent_skill_repo.count() == 0
|
||||
|
||||
|
||||
async def test_skills_phase_grows_agent_skill_count(
|
||||
backfill_runtime: Path,
|
||||
stub_llm_extraction: list[AlgoAgentSkill],
|
||||
capsys: pytest.CaptureFixture[str],
|
||||
) -> None:
|
||||
"""End-to-end: cluster with no skill yet → Phase 3 replays the event
|
||||
the clustering pass never got a listener for → real ``SKILL.md``
|
||||
write → real cascade sync → ``agent_skill_repo.count()`` grows."""
|
||||
await agent_case_repo.add([_agent_case("ac1")])
|
||||
await _seed_cluster()
|
||||
|
||||
skills_before = await agent_skill_repo.count()
|
||||
assert skills_before == 0
|
||||
|
||||
code = await run_backfill(phase="skills", auto_yes=True)
|
||||
out = capsys.readouterr().out
|
||||
|
||||
skills_after = await agent_skill_repo.count()
|
||||
assert code == 0
|
||||
assert skills_after == skills_before + 1
|
||||
assert "phase 3 complete" in out
|
||||
|
||||
rows = await agent_skill_repo.find_where("owner_id = 'agent1'", limit=10)
|
||||
assert len(rows) == 1
|
||||
assert rows[0].name == "summarise_doc"
|
||||
assert rows[0].cluster_id is not None
|
||||
@@ -0,0 +1,146 @@
|
||||
"""LanceDB business table schema migration (v2: vector nullability).
|
||||
|
||||
Exercises real LanceDB: builds a pre-migration ("v1-shape") physical
|
||||
table with a non-nullable ``vector`` column, runs
|
||||
``migrate_table_schemas``, and asserts the on-disk pyarrow schema is
|
||||
altered in place plus the ``.table_schema_version`` marker is written.
|
||||
Also proves idempotency — a second call must not re-issue
|
||||
``alter_columns`` once the marker is at the target version.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import pyarrow as pa
|
||||
import pytest
|
||||
|
||||
from everos.infra.persistence.lancedb import (
|
||||
_TABLE_SCHEMA_VERSION,
|
||||
BUSINESS_SCHEMAS_WITH_VECTOR,
|
||||
AtomicFact,
|
||||
Episode,
|
||||
LanceDBMigrationError,
|
||||
lancedb_manager,
|
||||
migrate_table_schemas,
|
||||
)
|
||||
|
||||
|
||||
def _v1_shape_schema() -> pa.Schema:
|
||||
"""Minimal pre-migration physical shape: ``vector`` is non-nullable."""
|
||||
return pa.schema(
|
||||
[
|
||||
pa.field("id", pa.string(), nullable=False),
|
||||
pa.field(
|
||||
"vector",
|
||||
pa.list_(pa.field("item", pa.float32(), nullable=True), 1024),
|
||||
nullable=False,
|
||||
),
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
async def _reset(tmp_path: Path, monkeypatch: pytest.MonkeyPatch):
|
||||
"""Point the singleton connection at an isolated memory-root."""
|
||||
monkeypatch.setenv("EVEROS_ROOT", str(tmp_path))
|
||||
lancedb_manager._conn = None
|
||||
lancedb_manager._tables.clear()
|
||||
yield
|
||||
await lancedb_manager.dispose_connection()
|
||||
|
||||
|
||||
async def _create_v1_table(name: str) -> None:
|
||||
conn = await lancedb_manager.get_connection()
|
||||
await conn.create_table(name, schema=_v1_shape_schema())
|
||||
|
||||
|
||||
async def test_migration_alters_all_business_tables_to_nullable() -> None:
|
||||
for schema in BUSINESS_SCHEMAS_WITH_VECTOR:
|
||||
await _create_v1_table(schema.TABLE_NAME)
|
||||
|
||||
await migrate_table_schemas()
|
||||
|
||||
for schema in BUSINESS_SCHEMAS_WITH_VECTOR:
|
||||
table = await lancedb_manager.get_table(schema.TABLE_NAME, schema)
|
||||
arrow_schema = await table.schema()
|
||||
assert arrow_schema.field("vector").nullable is True
|
||||
|
||||
|
||||
async def test_migration_writes_marker_with_target_version(tmp_path: Path) -> None:
|
||||
await _create_v1_table(Episode.TABLE_NAME)
|
||||
|
||||
await migrate_table_schemas()
|
||||
|
||||
marker = tmp_path / ".index" / "lancedb" / ".table_schema_version"
|
||||
assert marker.exists()
|
||||
assert int(marker.read_text().strip()) == _TABLE_SCHEMA_VERSION
|
||||
|
||||
|
||||
async def test_migration_idempotent_second_call_skips_alter(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Second call short-circuits at the marker; ``alter_columns`` is not
|
||||
re-issued."""
|
||||
await _create_v1_table(Episode.TABLE_NAME)
|
||||
await migrate_table_schemas()
|
||||
|
||||
table = await lancedb_manager.get_table(Episode.TABLE_NAME, Episode)
|
||||
calls: list[object] = []
|
||||
original_alter_columns = table.alter_columns
|
||||
|
||||
async def _spy_alter_columns(*alterations: object) -> object:
|
||||
calls.append(alterations)
|
||||
return await original_alter_columns(*alterations)
|
||||
|
||||
monkeypatch.setattr(table, "alter_columns", _spy_alter_columns)
|
||||
|
||||
await migrate_table_schemas()
|
||||
|
||||
assert calls == []
|
||||
|
||||
|
||||
async def test_migrate_table_schemas_raises_on_alter_failure(
|
||||
tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Any table's failing ``alter_columns`` must fail startup loudly:
|
||||
:class:`LanceDBMigrationError` is raised, the version marker stays
|
||||
unwritten, and the message names the failed table plus escalating
|
||||
recovery hints (restart, then wipe). Cascade would otherwise write
|
||||
NULL vectors into a still-NOT-NULL column and silently drop every
|
||||
row."""
|
||||
for schema in BUSINESS_SCHEMAS_WITH_VECTOR:
|
||||
await _create_v1_table(schema.TABLE_NAME)
|
||||
|
||||
episode_table = await lancedb_manager.get_table(Episode.TABLE_NAME, Episode)
|
||||
|
||||
async def _raise_alter_columns(*alterations: object) -> object:
|
||||
raise RuntimeError("simulated alter_columns failure")
|
||||
|
||||
monkeypatch.setattr(episode_table, "alter_columns", _raise_alter_columns)
|
||||
|
||||
with pytest.raises(LanceDBMigrationError) as excinfo:
|
||||
await migrate_table_schemas()
|
||||
|
||||
message = str(excinfo.value)
|
||||
assert Episode.TABLE_NAME in message
|
||||
lancedb_dir = tmp_path / ".index" / "lancedb"
|
||||
assert str(lancedb_dir) in message
|
||||
# Escalating recovery: restart first, wipe second.
|
||||
restart_idx = message.find("restart the process")
|
||||
wipe_idx = message.find("wipe the index directory")
|
||||
assert restart_idx != -1 and wipe_idx != -1
|
||||
assert restart_idx < wipe_idx
|
||||
|
||||
marker = lancedb_dir / ".table_schema_version"
|
||||
assert not marker.exists()
|
||||
|
||||
# Sibling tables reached in the loop before the failure keep the
|
||||
# per-table alter that already succeeded — only the marker write
|
||||
# (i.e. "the migration as a whole is done") is gated.
|
||||
atomic_fact_table = await lancedb_manager.get_table(
|
||||
AtomicFact.TABLE_NAME, AtomicFact
|
||||
)
|
||||
atomic_fact_schema = await atomic_fact_table.schema()
|
||||
assert atomic_fact_schema.field("vector").nullable is True
|
||||
@@ -157,17 +157,6 @@ def _reset_lancedb_write_locks() -> None:
|
||||
LanceRepoBase._reset_locks_for_tests()
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _reset_knowledge_embedding_singleton() -> None:
|
||||
"""Reset the lazy embedding and reranker singletons in service.knowledge."""
|
||||
import everos.service.knowledge as _kmod
|
||||
|
||||
for attr in ("_embedding", "_reranker"):
|
||||
setattr(_kmod, attr, None)
|
||||
for attr in ("_embedding_resolved", "_reranker_resolved"):
|
||||
setattr(_kmod, attr, False)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
async def cascade_runtime(
|
||||
tmp_path: Path,
|
||||
@@ -188,22 +177,36 @@ async def cascade_runtime(
|
||||
await ensure_business_indexes()
|
||||
(tmp_path / "ome.toml").write_text("# test\n")
|
||||
|
||||
yield MemoryRoot.default()
|
||||
yield MemoryRoot.resolve()
|
||||
|
||||
await dispose_connection()
|
||||
await dispose_engine()
|
||||
|
||||
|
||||
def _build_orchestrator(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
memory_root: MemoryRoot,
|
||||
embedder: _StubEmbedder,
|
||||
*,
|
||||
scan_interval: float = 60.0,
|
||||
) -> CascadeOrchestrator:
|
||||
"""Factory for a tight-polling cascade orchestrator."""
|
||||
"""Factory for a tight-polling cascade orchestrator.
|
||||
|
||||
``KnowledgeTopicHandler`` fetches the embedder lazily via
|
||||
``get_embedding_capability()`` rather than through the orchestrator
|
||||
— patch the process-wide singleton so cascade never hits the fake
|
||||
network target set up by ``cascade_runtime``.
|
||||
"""
|
||||
import everos.component.embedding.accessor as acc
|
||||
import everos.component.rerank.accessor as rerank_acc
|
||||
from everos.component.embedding import EmbeddingCapability
|
||||
from everos.component.rerank import RerankCapability
|
||||
|
||||
monkeypatch.setattr(acc, "_capability", EmbeddingCapability(provider=embedder))
|
||||
reranker = _StubReranker()
|
||||
monkeypatch.setattr(rerank_acc, "_capability", RerankCapability(provider=reranker))
|
||||
return CascadeOrchestrator(
|
||||
memory_root=memory_root,
|
||||
embedder=embedder,
|
||||
tokenizer=build_tokenizer(),
|
||||
config=CascadeConfig(
|
||||
scan_interval_seconds=scan_interval,
|
||||
@@ -274,11 +277,12 @@ async def _create_test_document(
|
||||
|
||||
async def test_create_document_end_to_end(
|
||||
cascade_runtime: MemoryRoot,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Full pipeline: create -> md -> cascade -> SQLite + LanceDB."""
|
||||
memory_root = cascade_runtime
|
||||
embedder = _StubEmbedder()
|
||||
orchestrator = _build_orchestrator(memory_root, embedder)
|
||||
orchestrator = _build_orchestrator(monkeypatch, memory_root, embedder)
|
||||
await orchestrator.start()
|
||||
await asyncio.sleep(0.3)
|
||||
|
||||
@@ -366,17 +370,12 @@ async def test_create_document_end_to_end(
|
||||
|
||||
async def test_search_finds_ingested_topic(
|
||||
cascade_runtime: MemoryRoot,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Keyword search finds topics after cascade indexing."""
|
||||
import everos.service.knowledge as _kmod
|
||||
|
||||
memory_root = cascade_runtime
|
||||
embedder = _StubEmbedder()
|
||||
_kmod._embedding = embedder
|
||||
_kmod._embedding_resolved = True
|
||||
_kmod._reranker = _StubReranker()
|
||||
_kmod._reranker_resolved = True
|
||||
orchestrator = _build_orchestrator(memory_root, embedder)
|
||||
orchestrator = _build_orchestrator(monkeypatch, memory_root, embedder)
|
||||
await orchestrator.start()
|
||||
await asyncio.sleep(0.3)
|
||||
|
||||
@@ -404,17 +403,12 @@ async def test_search_finds_ingested_topic(
|
||||
|
||||
async def test_search_include_content(
|
||||
cascade_runtime: MemoryRoot,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""include_content flag controls whether content is populated."""
|
||||
import everos.service.knowledge as _kmod
|
||||
|
||||
memory_root = cascade_runtime
|
||||
embedder = _StubEmbedder()
|
||||
_kmod._embedding = embedder
|
||||
_kmod._embedding_resolved = True
|
||||
_kmod._reranker = _StubReranker()
|
||||
_kmod._reranker_resolved = True
|
||||
orchestrator = _build_orchestrator(memory_root, embedder)
|
||||
orchestrator = _build_orchestrator(monkeypatch, memory_root, embedder)
|
||||
await orchestrator.start()
|
||||
await asyncio.sleep(0.3)
|
||||
|
||||
@@ -451,17 +445,12 @@ async def test_search_include_content(
|
||||
|
||||
async def test_search_score_threshold_filters(
|
||||
cascade_runtime: MemoryRoot,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""score_threshold filters out low-scoring results."""
|
||||
import everos.service.knowledge as _kmod
|
||||
|
||||
memory_root = cascade_runtime
|
||||
embedder = _StubEmbedder()
|
||||
_kmod._embedding = embedder
|
||||
_kmod._embedding_resolved = True
|
||||
_kmod._reranker = _StubReranker()
|
||||
_kmod._reranker_resolved = True
|
||||
orchestrator = _build_orchestrator(memory_root, embedder)
|
||||
orchestrator = _build_orchestrator(monkeypatch, memory_root, embedder)
|
||||
await orchestrator.start()
|
||||
await asyncio.sleep(0.3)
|
||||
|
||||
@@ -493,17 +482,12 @@ async def test_search_score_threshold_filters(
|
||||
|
||||
async def test_search_app_project_isolation(
|
||||
cascade_runtime: MemoryRoot,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Documents in different app/project scopes are isolated in search."""
|
||||
import everos.service.knowledge as _kmod
|
||||
|
||||
memory_root = cascade_runtime
|
||||
embedder = _StubEmbedder()
|
||||
_kmod._embedding = embedder
|
||||
_kmod._embedding_resolved = True
|
||||
_kmod._reranker = _StubReranker()
|
||||
_kmod._reranker_resolved = True
|
||||
orchestrator = _build_orchestrator(memory_root, embedder)
|
||||
orchestrator = _build_orchestrator(monkeypatch, memory_root, embedder)
|
||||
await orchestrator.start()
|
||||
await asyncio.sleep(0.3)
|
||||
|
||||
@@ -597,6 +581,7 @@ async def test_search_app_project_isolation(
|
||||
|
||||
async def test_delete_document_end_to_end(
|
||||
cascade_runtime: MemoryRoot,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Manual directory removal + scanner detects deletion -> cleanup.
|
||||
|
||||
@@ -608,6 +593,7 @@ async def test_delete_document_end_to_end(
|
||||
memory_root = cascade_runtime
|
||||
embedder = _StubEmbedder()
|
||||
orchestrator = _build_orchestrator(
|
||||
monkeypatch,
|
||||
memory_root,
|
||||
embedder,
|
||||
scan_interval=2.0,
|
||||
@@ -677,13 +663,126 @@ async def test_delete_document_end_to_end(
|
||||
await orchestrator.stop()
|
||||
|
||||
|
||||
async def test_delete_document_cleans_up_after_downgrade(
|
||||
cascade_runtime: MemoryRoot,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Tier 3 → Tier 1 downgrade must not strand knowledge state.
|
||||
|
||||
Regression guard for PR #361 review blocker B3. Previously
|
||||
``cascade.registry.build_handlers`` gated the two knowledge
|
||||
handlers off as an atomic pair when embed OR rerank went missing.
|
||||
On a Tier 3 → Tier 1 downgrade, ``service.knowledge.delete_document``
|
||||
``shutil.rmtree``'d the md directory, the scanner enqueued
|
||||
``deleted`` rows for every md — and the worker marked every one
|
||||
``failed(retryable=False)`` because no handler was registered
|
||||
for the kind. Phantom SQLite / LanceDB rows lingered forever;
|
||||
``GET /documents`` kept listing them; ``cascade fix`` could not
|
||||
recover.
|
||||
|
||||
The fix registers both knowledge handlers unconditionally. This
|
||||
test seeds a document with both capabilities available, simulates
|
||||
a full downgrade by pointing both accessor singletons at
|
||||
``provider=None``, calls ``delete_document`` and asserts the
|
||||
entire md + SQLite + LanceDB state clears and no permanent
|
||||
failures accrue.
|
||||
"""
|
||||
import everos.component.embedding.accessor as embed_acc
|
||||
import everos.component.rerank.accessor as rerank_acc
|
||||
from everos.component.embedding import EmbeddingCapability
|
||||
from everos.component.rerank import RerankCapability
|
||||
|
||||
memory_root = cascade_runtime
|
||||
embedder = _StubEmbedder()
|
||||
orchestrator = _build_orchestrator(
|
||||
monkeypatch,
|
||||
memory_root,
|
||||
embedder,
|
||||
scan_interval=2.0,
|
||||
)
|
||||
await orchestrator.start()
|
||||
await asyncio.sleep(0.3)
|
||||
|
||||
try:
|
||||
# ── Tier 3 seed: capabilities available, full index. ────────
|
||||
doc_id = "d_downgrd001"
|
||||
await _create_test_document(memory_root, doc_id=doc_id)
|
||||
await _wait_lance_rows(doc_id, expected=2, deadline=20.0)
|
||||
await _wait_drain(deadline=20.0)
|
||||
|
||||
assert await knowledge_document_repo.get_by_doc_id(doc_id) is not None
|
||||
assert await knowledge_topic_sqlite_repo.count_by_doc_id(doc_id) == 2
|
||||
|
||||
# ── Simulate downgrade: both capabilities unavailable. ──────
|
||||
# Point the process-wide singletons the handlers read at
|
||||
# ``provider=None``. ``embed_or_none`` returns ``None`` and
|
||||
# ``handle_deleted`` never touches embed/rerank at all.
|
||||
monkeypatch.setattr(
|
||||
embed_acc, "_capability", EmbeddingCapability(provider=None)
|
||||
)
|
||||
monkeypatch.setattr(rerank_acc, "_capability", RerankCapability(provider=None))
|
||||
|
||||
# ── Delete through the service (bypass HTTP for test speed).
|
||||
# ``delete_document`` rmtree's the md dir; scanner picks up
|
||||
# the missing files and enqueues ``deleted`` rows for both
|
||||
# kinds; worker must drain them cleanly.
|
||||
del_result = await delete_document(
|
||||
doc_id=doc_id,
|
||||
app_id="default",
|
||||
project_id="default",
|
||||
)
|
||||
assert del_result.doc_id == doc_id
|
||||
assert del_result.deleted_topics == 2
|
||||
|
||||
knowledge_dir = memory_root.knowledge_dir()
|
||||
doc_dir = knowledge_dir / "Sports" / f"Olympics_Plan_{doc_id}"
|
||||
assert not doc_dir.exists(), "delete_document should have removed the md dir"
|
||||
|
||||
# ── Wait for cascade to drain the deletes. ──────────────────
|
||||
await _wait_lance_rows(doc_id, expected=0, deadline=20.0)
|
||||
await _wait_drain(deadline=20.0)
|
||||
|
||||
# ── LanceDB clean. ──────────────────────────────────────────
|
||||
table = await get_table(KnowledgeTopic.TABLE_NAME, KnowledgeTopic)
|
||||
assert await table.count_rows(filter=f"doc_id = '{doc_id}'") == 0
|
||||
|
||||
# ── SQLite topic rows gone. ─────────────────────────────────
|
||||
assert await knowledge_topic_sqlite_repo.count_by_doc_id(doc_id) == 0
|
||||
|
||||
# ── SQLite document row gone (may need a scanner retry after
|
||||
# topic deletes clear the FK dependency). ───────────────────
|
||||
async with asyncio.timeout(15.0):
|
||||
while True:
|
||||
row = await knowledge_document_repo.get_by_doc_id(doc_id)
|
||||
if row is None:
|
||||
break
|
||||
await asyncio.sleep(0.2)
|
||||
assert await knowledge_document_repo.get_by_doc_id(doc_id) is None
|
||||
|
||||
# ── No permanent failures in md_change_state. ───────────────
|
||||
# This is the load-bearing assertion: with the gate in place,
|
||||
# every ``deleted`` row would land here as failed_permanent.
|
||||
summary = await md_change_state_repo.queue_summary()
|
||||
assert summary.failed_permanent == 0, (
|
||||
f"downgrade + delete produced permanent failures: {summary}"
|
||||
)
|
||||
assert summary.failed_retryable == 0, (
|
||||
f"downgrade + delete produced retryable failures: {summary}"
|
||||
)
|
||||
|
||||
finally:
|
||||
await orchestrator.stop()
|
||||
|
||||
|
||||
async def test_delete_idempotent(
|
||||
cascade_runtime: MemoryRoot,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Deleting an already-deleted (or nonexistent) document is a no-op."""
|
||||
memory_root = cascade_runtime
|
||||
embedder = _StubEmbedder()
|
||||
orchestrator = _build_orchestrator(
|
||||
monkeypatch,
|
||||
memory_root,
|
||||
embedder,
|
||||
scan_interval=2.0,
|
||||
@@ -729,11 +828,13 @@ async def test_delete_idempotent(
|
||||
|
||||
async def test_replace_document_end_to_end(
|
||||
cascade_runtime: MemoryRoot,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Replace = delete old + create new with same doc_id."""
|
||||
memory_root = cascade_runtime
|
||||
embedder = _StubEmbedder()
|
||||
orchestrator = _build_orchestrator(
|
||||
monkeypatch,
|
||||
memory_root,
|
||||
embedder,
|
||||
scan_interval=2.0,
|
||||
@@ -852,11 +953,12 @@ async def test_replace_document_end_to_end(
|
||||
|
||||
async def test_patch_title_updates_metadata(
|
||||
cascade_runtime: MemoryRoot,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""patch_document updates the title in SQLite without touching topics."""
|
||||
memory_root = cascade_runtime
|
||||
embedder = _StubEmbedder()
|
||||
orchestrator = _build_orchestrator(memory_root, embedder)
|
||||
orchestrator = _build_orchestrator(monkeypatch, memory_root, embedder)
|
||||
await orchestrator.start()
|
||||
await asyncio.sleep(0.3)
|
||||
|
||||
@@ -890,11 +992,12 @@ async def test_patch_title_updates_metadata(
|
||||
|
||||
async def test_patch_category_updates_sqlite(
|
||||
cascade_runtime: MemoryRoot,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""patch_document with category_id updates the document row (SQLite-only MVP)."""
|
||||
memory_root = cascade_runtime
|
||||
embedder = _StubEmbedder()
|
||||
orchestrator = _build_orchestrator(memory_root, embedder)
|
||||
orchestrator = _build_orchestrator(monkeypatch, memory_root, embedder)
|
||||
await orchestrator.start()
|
||||
await asyncio.sleep(0.3)
|
||||
|
||||
@@ -941,14 +1044,20 @@ async def test_get_nonexistent_document_raises(
|
||||
|
||||
async def test_search_empty_query_returns_empty(
|
||||
cascade_runtime: MemoryRoot,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""An empty query string returns empty results (no crash)."""
|
||||
import everos.service.knowledge as _kmod
|
||||
import everos.component.embedding.accessor as embed_acc
|
||||
import everos.component.rerank.accessor as rerank_acc
|
||||
from everos.component.embedding import EmbeddingCapability
|
||||
from everos.component.rerank import RerankCapability
|
||||
|
||||
_kmod._embedding = _StubEmbedder()
|
||||
_kmod._embedding_resolved = True
|
||||
_kmod._reranker = _StubReranker()
|
||||
_kmod._reranker_resolved = True
|
||||
monkeypatch.setattr(
|
||||
embed_acc, "_capability", EmbeddingCapability(provider=_StubEmbedder())
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
rerank_acc, "_capability", RerankCapability(provider=_StubReranker())
|
||||
)
|
||||
|
||||
result = await search_knowledge(
|
||||
query="",
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user