b1441da607
* refactor(config): make [embedding] and [rerank] soft dependencies
Make [embedding] and [rerank] soft runtime dependencies so a
freshly-onboarded user can run EverOS end-to-end with only [llm]
configured. Previously the server refused to start without embedding,
locking out anyone who just wanted keyword-only search.
## Capability tiers
- Tier 1 ([llm] only) KEYWORD search, add/flush, md
writes, cascade sync
- Tier 2 ([llm] + [embedding]) + VECTOR / HYBRID search,
reflection, skill extraction,
backfill
- Tier 3 ([llm] + [embedding] + rerank) + AGENTIC search, knowledge
Tier upgrades require a server restart (capability accessors cache
for the process lifetime). Tier downgrades are read-safe: a Tier-3
user who drops [rerank] can still read/rename/delete existing
knowledge documents; only write/search endpoints return 422.
## What changed
- Component accessors — component/{embedding,rerank,llm}/accessor.py
are the single process-wide provider singletons. service/* never
maintains parallel singletons; it consumes get_embedding_capability()
/ get_rerank_capability() / get_llm_client() directly. Build-time
ValueError from the factory is logged as capability_build_failed
(was silently swallowed).
- Error mapping — ProviderNotConfiguredError -> 422 with everos.toml
section hints (never EVEROS_* env-var strings).
LanceDBMigrationError fails loud with escalating recovery guidance
(restart -> wipe index). LLMNotConfiguredError in search maps to
None for KEYWORD degradation.
- Nullable-vector LanceDB migration — schema v2 makes the vector
column nullable so Tier-1 rows can land without embeddings.
Migration is guarded by a cross-process memory_root_lock
(fcntl.flock + anyio.to_thread) and runs optimize() per table
after Phase-1 backfill to reclaim manifest bloat.
- Cascade — knowledge handlers register unconditionally (Tier-3 ->
Tier-2/1 downgrade no longer strands DELETE); embed-requiring
strategies use body-guards that check capability.available at
execution time. _TABLE_SPECS has an import-time drift assertion
against BUSINESS_SCHEMAS_WITH_VECTOR.
- `everos cascade backfill` CLI — Phase-1 (embed missing vectors) /
Phase-2 (emit synthetic events for cascaded processing) / Phase-3
(sync new skill files). Exit codes: 0 / 1 / 2 / 3 (server running
preflight) / 4 (COMPLETED_WITH_FAILURES — per-row failures rolled
up) / 130 (SIGINT). OMEConfig.crash_recovery_enabled=False in
backfill engines prevents stale-RUNNING rows re-enqueuing into a
smaller strategy registry.
- /health — reports capabilities + disabled_features per tier so ops
can distinguish "boots but degraded" from "boots and full".
- Presentation split — memory / service / infra never import typer /
click. TyperPresenter Protocol + run_backfill() live in
entrypoints/cli/commands/_backfill_cmd.py. Enforced by
import-linter.
- Startup hint — unconditional count_rows(filter="vector IS NULL")
sweep emits unbackfilled_memory_rows (event name + hint text
pinned) when Tier-1 rows exist. ParserLifespanProvider warms the
everalgo.parser import at boot so /health doesn't block on first
call.
- Knowledge upload UTF-8 short-circuit — _looks_like_utf8_text()
routes text/* mime and known plaintext extensions (md/txt/rst)
straight to UTF-8 decode instead of the parser. Prevents 503
Multimodal-not-configured when Tier 3 sans [multimodal] uploads a
markdown doc.
## Sync history with main (2 merges collapsed into this squash)
Merged origin/main at
|
||
|---|---|---|
| .. | ||
| fixtures | ||
| README.md | ||
| run.py | ||
README.md
End-to-end memorize test
In-process driver that pushes a realistic fixture through service.memorize,
batching by 6 messages per /add call and then /flush at the end.
What's here
| File | Purpose |
|---|---|
fixtures/chat_session.json |
22 messages · 3 topic shifts · multi-user (Alice → Bob) — chat-mode fixture |
fixtures/agent_session.json |
21 items · 2 task threads · interleaved tool_calls / tool results — agent-mode fixture |
run.py |
In-process runner (no HTTP) |
Prereqs
- LLM client configured in
.env:EVEROS_LLM__API_KEY=...EVEROS_LLM__BASE_URL=...(OpenAI-compatible)EVEROS_LLM__MODEL=...(defaults togpt-4.1-mini)- Without these, the boundary stage logs
memorize_no_llm_clientand skips the run.
- Memory root: defaults to
~/.everos; override withEVEROS_ROOT=.... - Mode is read from
settings.memorize.mode(toml/env) before the firstmemorize()call.
Run
# Chat mode — boundary uses everalgo.boundary.detect_boundaries
EVEROS_MEMORIZE__MODE=chat uv run python scripts/e2e_memorize/run.py \
--fixture scripts/e2e_memorize/fixtures/chat_session.json \
--expected-mode chat
# Agent mode — boundary uses everalgo.agent_memory.AgentBoundaryDetector
# (filter→detect→remap; tool items preserved in cells)
EVEROS_MEMORIZE__MODE=agent uv run python scripts/e2e_memorize/run.py \
--fixture scripts/e2e_memorize/fixtures/agent_session.json \
--expected-mode agent
# Dry run (print batch plan, no LLM calls)
uv run python scripts/e2e_memorize/run.py \
--fixture scripts/e2e_memorize/fixtures/chat_session.json --dry-run
What to verify after a run
1. Console output
Each batch prints status= (accumulated while buffering, extracted when
cells got cut). Final flush should be extracted if any cell remained
in the tail. The trailing file walker lists md / sqlite files modified
in the last 10 minutes.
2. Episode md (sync — 4A)
~/.everos/users/<owner_id>/episodes/episode-YYYY-MM-DD.md
- Chat fixture: 2 owners (
u_alice,u_bob) — expect Episodes split into ~3-4 cells aligned with topic shifts (Python bug → weekend ramen → Q3 review → SRE handoff/ramen wrap). - Agent fixture: 1 user (
u_alice) — expect ~2 Episodes aligned with the two task threads (latency rollback → DB index fix).
3. SQLite memcell rows
sqlite3 ~/.everos/.index/sqlite/system.db \
"select memcell_id, track, owner_id, owner_type, json_array_length(sender_ids_json) as senders
from memcell order by timestamp"
- Chat run: rows with
track=user_memory,owner_type=user. - Agent run: parallel rows for both tracks (
user_memoryandagent_memory) since agent mode dispatches both pipelines.
4. Unprocessed buffer
sqlite3 ~/.everos/.index/sqlite/system.db \
"select session_id, count(*) from unprocessed_buffer
where track='memorize' group by session_id"
After flush the buffer should be empty for the test session.
5. OME async output (only if subscribers exist)
users/<owner>/atomic_facts/atomic_fact-YYYY-MM-DD.md(always;extract_atomic_factsis registered)users/<owner>/foresights/foresight-YYYY-MM-DD.md(always;extract_foresightis registered)agents/<agent>/agent_cases/agent_case-YYYY-MM-DD.md(only afterextract_agent_casesstrategy is written + registered — currently absent, the emit is a no-op)
6. Reset between runs
The fixture's session_id is randomised per invocation, so previous runs don't pollute the new one. To wipe everything:
rm -rf ~/.everos/users ~/.everos/agents ~/.everos/.index/sqlite/system.db
Boundary expectations cheat sheet
Chat fixture topic shifts (timestamps ms)
| Range | Topic |
|---|---|
msgs 1-6 (1747396800–1747397010) |
Python KeyError debugging |
msgs 7-12 (1747400400–1747400610) |
Weekend ramen plans |
msgs 13-16 (1747407600–1747407720) |
Q3 revenue review meeting prep |
msgs 17-22 (1747411200–1747411410) |
Bob joins, SRE handoff + ramen + Q3 deck deadline |
Boundary detector should cut on topic gaps; 3 cuts → 4 cells is the most likely outcome.
Agent fixture task threads
| Range | Task |
|---|---|
items 1-13 (1747396800–1747397140) |
API latency spike → identify keepalive pool regression → rollback |
items 14-21 (1747400400–1747400720) |
DB connection pool exhaustion → find unindexed query → CREATE INDEX CONCURRENTLY |
Boundary detector should cut between item 13 and item 14 (timestamp jump ~55 minutes, topic flip). Tool items inside each cell stay attached to their initiating chat turn.