* fix(lancedb): reclaim stale versions via write-locked prune The storage soak (48h, sustained churn + fuzz) proved the bundled lock-free `optimize(cleanup_older_than=...)` loses its commit-conflict race against concurrent writes — cleanup ran only 16 of ~250 times, so old dataset versions / FTS orphans piled up and the index dir grew to the 40G disk guardrail and never reclaimed under load. main still had that bundled call. Split the maintenance path: - `LanceRepoBase.optimize()` is now compact-only and lock-free (a commit conflict here is benign — the next beat retries, so it must not stall writers). - `LanceRepoBase.prune(older_than)` runs `cleanup_older_than + delete_unverified=True` **under the per-table write lock**, so no writer is in flight: the Rewrite has the manifest to itself (cleanup completes every beat) and aggressive deletion is safe. It also removes the empty `_indices/<uuid>/` husks cleanup leaves behind (soak: 13061 dirs, 98% empty), offloaded to a thread. - The cascade worker's heavy beat calls `prune()`; the light beat calls `optimize()`. A benign light-beat commit conflict is logged at debug and does not count toward the failure streak or trigger a rebuild. - Prune's retention window (`cleanup_older_than`) is decoupled from the prune cadence and defaulted short (60s). It runs under the write lock, so the window only needs to outlive an in-flight read; keeping it = cadence (300s) left ~2 cadences of superseded full-table copies on disk between beats (soak: transient ~15G/table peaks). 60s reclaims all but the last minute each beat — same live floor (~625MB/table), far lower transient footprint. Result on the re-run soak: disk sawtooths and reclaims to live-data size (~1.3G total) under active load — vs run1 stuck at 40G until writes stopped — with 0 crashes / 0 OOM / 0 stuck cleanups over 48h. Cascade projection health is now observable: - `CascadeOrchestrator.health()` -> `CascadeHealth`, combining the worker's in-memory signals (drain-loop failures, unrecoverable count, optimize streak, prune staleness) with the SQLite queue summary. - `GET /health` gains a typed `cascade` readiness block. `healthy` reflects operational health only (drain / optimize / prune); `failed_permanent` (files awaiting `cascade fix`) is a data-quality backlog reported as an informational count that does NOT flip `healthy` — otherwise the signal would sit red permanently. The scanner-side retry cap and the `_MAX_TOTAL_RETRIES` budget already on main handle re-enqueue storms, so no duplicate is added here. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(deps): pin lancedb to >=0.34.0,<0.35.0 The previous open-ended `>=0.13.0` let any environment float to an untested release, including 0.32-0.34 which carry a compaction offset-overflow regression (lance-format/lance#7653) that stalls version cleanup and grows the index dir without bound. - Floor 0.34.0: the current resolved version; runs safely thanks to the with_position=False FTS workaround shipped in #336. Verified that data written by lancedb 0.32.0 (lance v6) reads correctly under 0.34.0 (lance v8), so existing deployments upgrade cleanly. Never widen the floor below 0.34 -- older lance cannot read v8-format data. - Ceiling <0.35.0: 0.35 embeds lance-rust v9 (large encoding jump, not yet stable-released); it must pass the soak harness before we allow it. Resolved version is unchanged (still 0.34.0); this only tightens the declared constraint. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> (cherry picked from commit 94f9aa67d11a07c93c7d59d6b134e80858b60316) * fix(search): bridge agent-kind metadata into the agentic doc contract The agent AGENTIC path (`agent_case` / `agent_skill`) fed recall candidates straight into `aagentic_retrieve`, whose `_format_docs` (LLM sufficiency / multi-query prompt) reads `metadata["episode"]` as a `{subject, content}` dict plus a ms-epoch `timestamp`. Agent-kind rows carry their body in the recaller's `text_field` and time as a datetime, so `_format_docs` raised `TypeError: Candidate ... has no episode dict` and `POST /api/v*/memory/search` returned 500 for any `owner_type=agent` + `method=agentic` request. Mirror the episode path's bridge: reshape agent candidate metadata into the everalgo doc contract before `aagentic_retrieve`, and revert it before DTO shaping so the agent shapers still see a datetime timestamp. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> (cherry picked from commit bb9a3a585e044643d3852b9d2810b42c05a7a47c) * test(search): regenerate search seed to current linkage + migrate e2e The committed `search_seed` fixture and several search e2e tests were written for the pre-1.5 memcell fact-linkage model. Current extraction links atomic_facts to episodes via `parent_id == episode.entry_id` (parent_type="episode"), and user_memory clusters store episode entry_id members — so the stale fixture made VECTOR/AGENTIC recall and the cluster-narrowing path find nothing, and stale assertions checked an old error code. - Regenerate `search_seed/*` from a fresh corpus in the current entry_id format; facts now bridge across multiple episodes (richer agentic / hierarchical-eviction coverage). - Fix `_dump_search_seed.py` sampling: pick episodes that host facts first and keep facts by episode entry_id, so re-dumps stay coherent. - Migrate e2e tests to the entry_id model (hierarchical-eviction, session/timestamp filters, cluster seeding helper) and update the filter-error assertion to the current `INVALID_INPUT` code. - Provision `ome.toml` in the full-app pipeline fixture (the OME config reloader requires it; strategies are code-registered so the packaged default suffices), unblocking corpus regeneration. Full search e2e suite now green (49/49). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> (cherry picked from commit 44cd9cecc07775cb9270104a5804f36e32cd788d) * fix(lancedb): detect schema type drift and add cascade rebuild recovery verify_business_schemas only compared column names, so a column whose on-disk Arrow type had drifted (name unchanged) slipped through and detonated later inside merge_insert as an opaque LanceError(IO) "Spill has sent an error" (#337). Now compare each shared column's Arrow type against schema.to_arrow_schema() — the exact schema get_table builds tables from, so a healthy table never false-positives. Reproduced #337 byte-identically: an episode.subject_vector column left as string or null by an older build, plus a real 1024-d vector on upsert, yields the exact crash. No lancedb version (0.13-0.34) renders Optional[Vector] as a non-vector type, so the startup guard is what should catch it — not the runtime. Add `everos cascade rebuild` as the safe recovery: it drops the business LanceDB tables and re-indexes from markdown, skipping the verify guard (which the drift would otherwise trip on startup). Unlike removing only .index/lancedb it re-populates already-done entries (reset_all clears the cascade queue); unlike removing all of .index it preserves unprocessed_buffer (messages not yet extracted). Fixes #337. * docs(cascade): document cascade rebuild and correct recovery guidance Add the `everos cascade rebuild` command to the runbook, CLI, and how-memory-works docs. Correct the old recovery guidance: a bare `rm -rf .index/lancedb` leaves md_change_state marked `done`, so the scanner skips those files and the index comes back empty — the runbook previously claimed a full repopulation that does not happen. `cascade rebuild` is the safe path (re-populates done entries, preserves unprocessed_buffer). Also document that verify now checks column types. * chore(rebase): adapt #354 integration test to soft-embedding main CascadeOrchestrator dropped the embedder param when embedding became a soft dependency (main); fold the schema-drift integration test onto it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(cascade): freeze monotonic clock in prune-staleness health tests Both prune-staleness health tests fabricated `_started_at = time.monotonic() - (ALERT + 100)`, assuming monotonic() is a large value. On a fresh CI runner monotonic() is only ~100-180s, so the subtraction went negative, the source clamped the baseline to 0, and staleness read back as ~130s < 900s — failing on CI while passing on long-lived dev boxes where monotonic() is huge. Freeze the monotonic clock via monkeypatch so staleness is deterministic regardless of the runner's boot uptime. Source logic is unchanged; only the tests are made hermetic. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(cascade): wait for stable terminal state in rename scenarios `_wait_path_done` reached a terminal status, slept a 0.1s settle window, then asserted the status was still terminal — which contradicted its own docstring ("absorb any last-second re-enqueue"). A rename's delete event or an atomic-replace echo can flip a done row back to `processing` inside that window, so on a slow CI runner the assert fired ("flipped back to processing after reaching done"), failing test_rename_cross_owner_keeps_frontmatter_owner intermittently (seen on the 3.13 job). `make integration` runs without `--reruns`, so a single flake fails the whole job. Wait for a terminal state that *survives* the settle window instead: absorb a transient re-enqueue by waiting for terminal again, still bounded by `deadline` so a row that never settles surfaces as a timeout. Pre- existing flake on main, unrelated to the prune change; the scenario's real assertions (row counts, frontmatter owner) are untouched. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(cascade): cross-process prune safety + backfill reclaim fix Review of #379 found two P0s plus P1/P2s, all verified against the code: - P0-1: cascade backfill still called the removed optimize(cleanup_older_than=…) kwarg → TypeError swallowed by the best-effort try/except → backfill silently skipped all compaction + reclaim (the exact bloat this PR fixes). CI stayed green because the test double kept the stale signature. Fix: call optimize() + prune(0) at the call site; make the fake mirror the real signature so the drift can't hide again; pin prune in the backfill tests. - P0-2: prune ran delete_unverified=True guarded only by an in-process asyncio lock, but the runbook promises `cascade sync` is safe alongside a live server — and the CLI's first optimize beat does prune, in a separate process. It could delete files the daemon is mid-commit on. Fix: switch prune to delete_unverified=False. Measured to reclaim identically on churned tables (both collapse superseded versions ~97%); True only additionally deletes in-flight/dangling files — exactly the corruption vector. No cross-process lock needed; the write-lock/commit fix (the real reclaim win) is unchanged. - P1-3: /health called orch.health() (6 SQLite aggregates) with no guard → a locked/full/migrating DB would 500 the liveness probe and restart the container. Wrap it: unhealthy readiness + reason, HTTP stays 200. - P1-4: rebuild drops + recreates tables; a live daemon holds cached handles pointing at the dropped dataset. Runbook now says stop the server first — the one cascade command unsafe alongside a live server. - P2: narrow _is_benign_commit_conflict to the "commit conflict" phrase (a bare "retryable" swallowed unrelated recoverable errors); add a timeout around the prune cleanup so a hung lance call can't wedge the write lock; correct two stale docstrings. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: fix schema-recovery guidance + drop dead internal doc refs Follow-up to the review; two doc issues neither the review nor the fix commit caught: - The schema-drift startup error's docstrings (verify_business_schemas and LanceDBLifespanProvider) still described the recovery as `rm -rf ~/.everos/.index/lancedb` — which the runbook explicitly calls the WRONG recovery (it leaves the cascade queue `done`, so nothing re-indexes and the index comes back empty). The raised error already points to `everos cascade rebuild`; align the docstrings to match. - 15 dangling references to an internal numbered design-doc set (12_/13_/16_/17_*.md) that was never shipped to this repo. Point the schema-recovery ones at docs/cascade_runbook.md; drop the rest (pure provenance in table/component docstrings) while keeping the substance. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: align maintenance docstrings with split optimize/prune API Two docstring residuals from the #379 review's P2 list: - _run_optimize_once still described the pre-split bundled heavy beat ("same work plus cleanup_older_than ... older than one cadence"); the heavy beat now calls prune() under the write lock and the retention window is decoupled from the cadence (DEFAULT_OPTIMIZE_PRUNE_RETENTION_SECONDS). - _restore_shaper_metadata converts any numeric timestamp, wider than an exact inverse of the bridge; document that this is deliberate (the shaper contract requires a datetime either way). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * style(e2e): hoist fixture-body imports to conftest module top shutil / importlib.resources.files were imported inside the core_pipeline_runtime fixture body; move them to the module top to match the repo import style (#379 review P2). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(cascade): back off a hung prune via a separate attempt clock (N1) The write-lock timeout on prune (review P2) didn't achieve its goal: it was 300s — equal to the prune cadence — and `last_prune_at` only advanced on success. So a hung lance cleanup timed out after 300s, `should_prune` was still true (clock never moved), and the next beat re-pruned ~10s later — pinning the per-table write lock ~97% of the time, the exact write-starvation the timeout was meant to prevent. A real cleanup is milliseconds even on a heavily churned table (measured ~40ms at 320k writes / 100 versions), so the timeout is a pure hang-catcher: lower it to 60s (~1500x headroom, never fires normally, well below the 300s cadence). Split the prune clock so a failed prune backs off without masking the health signal: - last_prune_attempt_at (new) gates scheduling, advanced before the call whether it succeeds or times out → a hung prune waits a full cadence before retrying (light lock-free compaction runs meanwhile), so the lock is held at most ~timeout/cadence ≈ 17% in the worst case. - last_prune_at advances only on success and still drives the prune-staleness health signal, so a persistently failing prune surfaces as degraded instead of being hidden by an advanced schedule clock. Regression test: a raising prune advances the attempt clock (next beat is light, no immediate re-prune) but not the success clock. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: zhanghui <zhanghui@shanda.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
16 KiB
How Memory Works
How EverOS turns a stream of messages into durable, searchable memory — the storage stack, the path layout on disk, the write→index→read pipeline, and the consistency guarantees.
This is the narrative companion to the reference docs: see storage_layout.md for the exact file encoding, architecture.md for the layer boundaries, and api.md for the HTTP contract.
Table of contents
- The storage stack
- Storage paths
- How a memory is born
- Memory types & storage strategies
- The cascade daemon
- The Offline Memory Engine (OME)
- Consistency model
- Zero external services
- Operating it
The storage stack
Three embedded pieces, each owning what it is best at. Markdown is the source of truth; the other two are derived and rebuildable.
| Layer | Backed by | Holds | Rebuildable? |
|---|---|---|---|
| Markdown + YAML frontmatter | plain .md files |
the memory content itself — the only portable, human-editable asset | — (it is the truth) |
SQLite (aiosqlite) |
.index/sqlite/*.db |
system state, audit log, the cascade queue, the boundary buffer, OME engine state | ✅ from markdown |
| LanceDB (Arrow) | .index/lancedb/*.lance |
vector + BM25 + scalar columns for retrieval | ✅ from markdown |
!!! note "The one rule that follows from this"
Delete the entire .index/ directory and no memory is lost — it
rebuilds from the .md tree. There is no separate "export"; the
markdown is the export. (How to trigger a rebuild:
Operating it.)
Storage paths
The default memory root is ~/.everos/ (override with
EVEROS_ROOT env var or --root on the CLI). Configuration lives
inside the memory root as everos.toml (generated by everos init).
Memory is partitioned by <app_id>/<project_id> before the
user-visible directories, so different (app, project) spaces never share
a directory or cross in search. The reserved id "default" materialises as
default_app / default_project on disk (so a default space stays
visually distinct from a user-named one).
~/.everos/ ← memory root (EVEROS_ROOT)
├── default_app/ ← <app_id> ("default" → default_app)
│ └── default_project/ ← <project_id> ("default" → default_project)
│ ├── users/ ← user-visible (source of truth)
│ │ └── <user_id>/
│ │ ├── user.md single-file (profile)
│ │ ├── episodes/
│ │ │ └── episode-<YYYY-MM-DD>.md daily-log append
│ │ ├── .atomic_facts/ daily-log (hidden)
│ │ │ └── atomic_fact-<YYYY-MM-DD>.md
│ │ └── .foresights/ daily-log (hidden)
│ │ └── foresight-<YYYY-MM-DD>.md
│ ├── agents/
│ │ └── <agent_id>/
│ │ ├── .cases/ daily-log (hidden)
│ │ │ └── agent_case-<YYYY-MM-DD>.md
│ │ └── skills/ skill-named dir
│ │ └── skill_<name>/SKILL.md (+ references/ scripts/)
│ └── knowledge/ ← shared / global
│
├── .index/ ← system-managed, rebuildable (gitignore)
│ ├── sqlite/
│ │ ├── system.db state / audit / cascade queue (md_change_state) / buffer / LSN
│ │ ├── ome.db Offline Memory Engine state
│ │ ├── ome.aps.db APScheduler jobstore (split to avoid lock contention)
│ │ └── ome.db.lock OME single-engine guard (portalocker)
│ └── lancedb/
│ └── <kind>.lance/ one Arrow table per kind
│
├── ome.toml ← user-editable OME strategy overrides (hot-reloaded)
└── .tmp/ atomic-write staging
!!! warning "Differences from older PRD-era docs"
The index dir is .index/ (dot-prefixed), not _index/. The
cascade queue and LSN/audit state live in SQLite (system.db,
table md_change_state) — there is no .cascade.log / .manifest.json
file in the current implementation. The <app>/<project> nesting is
real and always present (default_app/default_project for the default
scope). There is no command literally named reindex, but
everos cascade rebuild rebuilds the index from markdown (see
Operating it).
The path manager is
MemoryRoot; every path
above is a property on it. MemoryRoot.ensure() creates the runtime dirs
(.index/{sqlite,lancedb}/, .tmp/); user-visible dirs appear on first
write. Config files (everos.toml, ome.toml) are created by everos init.
How a memory is born
A message does not become memory immediately — it accumulates, a boundary is detected, an LLM extracts a cell, writers persist markdown, and the index catches up asynchronously.
POST /add ──▶ unprocessed_buffer (SQLite) ← messages accumulate per (session, app, project)
│
├─ boundary detector trips ─┐
POST /flush ─────────┤ (or you force it) │ one LLM call
│ ▼
│ extract MemCell ──▶ memcell row (SQLite)
│ │
│ ┌──────────────┴───────────────┐
│ ▼ ▼
│ UserMemoryPipeline (sync) AgentMemoryPipeline (fire-and-forget)
│ writes episode .md NOW emits AgentPipelineStarted
▼ │ │
(response returns once md is on disk) │
▼ ▼
┌─────────────────── Offline Memory Engine (OME) ───────────────────┐
│ async strategies write derived .md: │
│ atomic_facts · foresight · user profile · agent cases · agent skills │
└───────────────────────────────┬──────────────────────────────────────┘
▼
cascade daemon watches the .md tree
▼
md_change_state queue (SQLite, durable)
▼
rebuild LanceDB rows ──▶ searchable
/addappends messages to a per-(session_id, app_id, project_id)buffer and returnsaccumulated(orextractedif the boundary tripped on this call). See api.md./flushforces the boundary now (one extraction LLM call), used at the end of a chat/agent run.- Episode markdown is written synchronously — when
/flushreturnsextracted, the episode file is already on disk. - Everything else (atomic facts, foresight, profile, agent cases/skills) is produced asynchronously by the OME — see the OME section.
- The cascade daemon turns every
.mdwrite into LanceDB rows so the content becomes searchable.
Memory types & storage strategies
Eight business memory kinds today, each user-, agent-, or globally owned, each picking one of three on-disk patterns:
| Kind | Owner | Dir / file | Strategy | Produced by |
|---|---|---|---|---|
| episode | user | episodes/episode-<date>.md |
daily-log | extraction (sync) |
| atomic_fact | user | .atomic_facts/atomic_fact-<date>.md (hidden) |
daily-log | OME |
| foresight | user | .foresights/foresight-<date>.md (hidden) |
daily-log | OME |
| profile | user | user.md |
single-file rewrite | OME |
| agent_case | agent | .cases/agent_case-<date>.md (hidden) |
daily-log | OME |
| agent_skill | agent | skills/skill_<name>/SKILL.md |
skill-named dir | OME (clustering) |
| knowledge_document | global | knowledge/<category_id>/<title_dirname>/index.md |
knowledge tree | knowledge service |
| knowledge_topic | global | knowledge/<category_id>/<title_dirname>/<N>_<topic_slug>.md |
knowledge tree | knowledge service |
The three strategies:
| Strategy | Shape | Why |
|---|---|---|
| Daily-log append | <prefix>-<YYYY-MM-DD>.md, one entry appended per memory |
collapses thousands of per-entry files into one file per day |
| Single-file rewrite | a fixed filename overwritten in place | for a single evolving document (a user/agent profile) |
| Skill-named dir | one directory per skill | a skill is a richer unit (body + optional references/ scripts/) |
!!! note
The single-file writer also supports agent.md / soul.md /
tools.md / behaviors.md, but no shipped OME strategy produces those
yet — today only user.md is written. Detailed frontmatter and
entry-id encoding live in storage_layout.md.
The cascade daemon
The cascade subsystem keeps LanceDB in sync with the markdown tree. It runs in-process with the server (a coroutine started by the app lifespan), not as a separate OS daemon.
- A native filesystem watcher (
watchdog: FSEvents on macOS, inotify on Linux) sees a.mdcreate/modify. - The change is enqueued in the
md_change_statetable (SQLite) — durable, so a crash mid-sync replays on restart. - A worker drains the queue at entry-level granularity: it diffs the
file, re-embeds only changed entries (keyed by
content_sha256), and upserts the LanceDB rows.
Because markdown is the source of truth, editing a file directly is
fully supported — open an episode in VSCode / Obsidian / Vim, change an
entry, save, and the daemon re-indexes just that entry. Operate the queue
with everos cascade (Operating it); deeper runbook in
cascade_runbook.md.
The Offline Memory Engine (OME)
Most memory kinds are not extracted on the request path — they are derived later by the OME, an in-process async strategy engine. When extraction carves a MemCell, it emits an event; OME strategies pick it up and write their markdown when ready:
extract_atomic_facts— single-sentence facts from an episodeextract_foresight— anticipatory notesextract_user_profile— the aggregateduser.mdextract_agent_case— a reusable agent trajectory (only when the cell is substantive enough; thin trajectories are skipped by design)extract_agent_skill— clusters related cases into a named skilltrigger_profile_clustering— triggers user profile clusteringtrigger_skill_clustering— triggers agent skill clusteringreflect_episodes(cron, default off) — offline memory consolidation. Merges fragmented episodes within a cluster into a single coherent narrative, re-extracts atomic facts, and deprecates the originals. Enable viaome.toml.
Strategies are configurable without a code change via ome.toml at the
memory root (hot-reloaded within ~2 s). Example — turn two off:
[strategies.extract_foresight]
enabled = false
[strategies.extract_user_profile]
enabled = false
OME keeps its own state in .index/sqlite/ome.db (run records, counters)
and its scheduler jobstore in .index/sqlite/ome.aps.db (split so the sync
APScheduler writer and the async OME writer never contend for one file
lock).
Reflection (offline consolidation)
The reflect_episodes strategy runs on a cron schedule (default
0 2 * * 1 — every Monday at 02:00, disabled by default). It selects clusters with multiple
members, calls the LLM to merge their episodes into a single narrative,
writes the merged episode to markdown, re-extracts atomic facts, and
deprecates the originals via deprecated_by. The merged episode uses
parent_type=cluster and session_id=None. Enable it in ome.toml:
[strategies.reflect_episodes]
enabled = true
!!! tip "Implication for clients"
After /flush returns extracted, the episode is queryable soon
(once cascade indexes it), but atomic facts / profile / agent cases
appear only after their OME strategy runs — typically seconds later.
Poll / retry if you need them immediately.
Consistency model
Two paths, two guarantees:
| Path | Guarantee | Detail |
|---|---|---|
Write (/add, /flush) |
strong | the episode .md is on disk before the call returns extracted; never blocks on LanceDB |
Read (/search, /get) |
eventual | reads LanceDB, which lags md by the cascade processing time — sub-second typically, up to ~10–15 s under load |
So a /search immediately after the /flush that produced a record may
miss it. The markdown is durable regardless; index lag never loses data. If
you need read-your-write, retry with backoff, or force the queue with
everos cascade sync.
Integrity is anchored by a few invariants (details in
storage_layout.md): the frontmatter id /
entry_id is the immutable join key; content_sha256 decides whether an
entry needs re-embedding; an LSN watermark (in system.db) orders
rebuilds; the durable md_change_state queue is the replayable audit
trail.
Zero external services
No database server, message broker, or vector service to run. Vector ANN, full-text BM25, and scalar filtering all execute inside the embedded LanceDB engine in one query; SQLite is a local file. The whole stack is a single directory you can copy, back up, or check the user-visible parts of into git.
!!! note There is no automatic "grep over markdown" search fallback today — if the LanceDB index is unavailable, rebuild it from markdown (it is derived and disposable) rather than relying on a degraded search path.
Operating it
The CLI (cli.md) is intentionally small:
| Command | What it does |
|---|---|
everos init |
generate starter config files (everos.toml + ome.toml) |
everos server start |
run the HTTP API (cascade + OME start with it) |
everos cascade status |
queue / LSN summary |
everos cascade sync |
drain the cascade queue now (force md → LanceDB) |
everos cascade fix |
list failed rows / re-enqueue retryable ones |
everos cascade rebuild |
rebuild the whole index from markdown (drift / corruption recovery) |
!!! warning "There is no everos reindex or everos flush"
- Reindex = the index is rebuildable from markdown. To rebuild
the whole index, run everos cascade rebuild — it drops the
LanceDB tables and re-indexes from md, re-populating even entries
the queue already marked done and preserving un-extracted
buffered messages. (A bare rm -rf <memory-root>/.index/lancedb
is not enough: the cascade queue still shows those files
done, so the scanner skips them and the index comes back empty.)
For an incremental catch-up, use everos cascade sync.
- Flush is an HTTP endpoint (POST /api/v2/memory/flush), not a
CLI command — it forces extraction of the session buffer, which is
a different thing from forcing index sync (cascade sync).
References
- storage_layout.md — exact file encoding, frontmatter chassis, entry-id format, atomic-write semantics
- architecture.md — DDD layers and dependency rules
- api.md — the HTTP contract (
/add/flush/search/get) - cascade_runbook.md — operating the sync queue