xiaoxue_admin/EverOS
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Files
72c1facdbfa3c07d97d29db106482cf643ae8231
EverOS/docs/cli.md
T
zhanghuiClaude Opus 4.8zhanghui
 e5118c52a8 fix(lancedb): reclaim stale versions via write-locked prune (#379) 
* fix(lancedb): reclaim stale versions via write-locked prune

The storage soak (48h, sustained churn + fuzz) proved the bundled
lock-free `optimize(cleanup_older_than=...)` loses its commit-conflict
race against concurrent writes — cleanup ran only 16 of ~250 times, so
old dataset versions / FTS orphans piled up and the index dir grew to
the 40G disk guardrail and never reclaimed under load. main still had
that bundled call.

Split the maintenance path:
- `LanceRepoBase.optimize()` is now compact-only and lock-free (a commit
  conflict here is benign — the next beat retries, so it must not stall
  writers).
- `LanceRepoBase.prune(older_than)` runs `cleanup_older_than +
  delete_unverified=True` **under the per-table write lock**, so no
  writer is in flight: the Rewrite has the manifest to itself (cleanup
  completes every beat) and aggressive deletion is safe. It also removes
  the empty `_indices/<uuid>/` husks cleanup leaves behind (soak: 13061
  dirs, 98% empty), offloaded to a thread.
- The cascade worker's heavy beat calls `prune()`; the light beat calls
  `optimize()`. A benign light-beat commit conflict is logged at debug
  and does not count toward the failure streak or trigger a rebuild.
- Prune's retention window (`cleanup_older_than`) is decoupled from the
  prune cadence and defaulted short (60s). It runs under the write lock,
  so the window only needs to outlive an in-flight read; keeping it =
  cadence (300s) left ~2 cadences of superseded full-table copies on
  disk between beats (soak: transient ~15G/table peaks). 60s reclaims
  all but the last minute each beat — same live floor (~625MB/table),
  far lower transient footprint.

Result on the re-run soak: disk sawtooths and reclaims to live-data size
(~1.3G total) under active load — vs run1 stuck at 40G until writes
stopped — with 0 crashes / 0 OOM / 0 stuck cleanups over 48h.

Cascade projection health is now observable:
- `CascadeOrchestrator.health()` -> `CascadeHealth`, combining the
  worker's in-memory signals (drain-loop failures, unrecoverable count,
  optimize streak, prune staleness) with the SQLite queue summary.
- `GET /health` gains a typed `cascade` readiness block. `healthy`
  reflects operational health only (drain / optimize / prune);
  `failed_permanent` (files awaiting `cascade fix`) is a data-quality
  backlog reported as an informational count that does NOT flip
  `healthy` — otherwise the signal would sit red permanently.

The scanner-side retry cap and the `_MAX_TOTAL_RETRIES` budget already
on main handle re-enqueue storms, so no duplicate is added here.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(deps): pin lancedb to >=0.34.0,<0.35.0

The previous open-ended `>=0.13.0` let any environment float to an
untested release, including 0.32-0.34 which carry a compaction
offset-overflow regression (lance-format/lance#7653) that stalls
version cleanup and grows the index dir without bound.

- Floor 0.34.0: the current resolved version; runs safely thanks to
  the with_position=False FTS workaround shipped in #336. Verified that
  data written by lancedb 0.32.0 (lance v6) reads correctly under
  0.34.0 (lance v8), so existing deployments upgrade cleanly. Never
  widen the floor below 0.34 -- older lance cannot read v8-format data.
- Ceiling <0.35.0: 0.35 embeds lance-rust v9 (large encoding jump, not
  yet stable-released); it must pass the soak harness before we allow
  it.

Resolved version is unchanged (still 0.34.0); this only tightens the
declared constraint.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(cherry picked from commit 94f9aa67d11a07c93c7d59d6b134e80858b60316)

* fix(search): bridge agent-kind metadata into the agentic doc contract

The agent AGENTIC path (`agent_case` / `agent_skill`) fed recall
candidates straight into `aagentic_retrieve`, whose `_format_docs`
(LLM sufficiency / multi-query prompt) reads `metadata["episode"]` as a
`{subject, content}` dict plus a ms-epoch `timestamp`. Agent-kind rows
carry their body in the recaller's `text_field` and time as a datetime,
so `_format_docs` raised `TypeError: Candidate ... has no episode dict`
and `POST /api/v*/memory/search` returned 500 for any
`owner_type=agent` + `method=agentic` request.

Mirror the episode path's bridge: reshape agent candidate metadata into
the everalgo doc contract before `aagentic_retrieve`, and revert it
before DTO shaping so the agent shapers still see a datetime timestamp.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit bb9a3a585e044643d3852b9d2810b42c05a7a47c)

* test(search): regenerate search seed to current linkage + migrate e2e

The committed `search_seed` fixture and several search e2e tests were
written for the pre-1.5 memcell fact-linkage model. Current extraction
links atomic_facts to episodes via `parent_id == episode.entry_id`
(parent_type="episode"), and user_memory clusters store episode
entry_id members — so the stale fixture made VECTOR/AGENTIC recall and
the cluster-narrowing path find nothing, and stale assertions checked
an old error code.

- Regenerate `search_seed/*` from a fresh corpus in the current
  entry_id format; facts now bridge across multiple episodes (richer
  agentic / hierarchical-eviction coverage).
- Fix `_dump_search_seed.py` sampling: pick episodes that host facts
  first and keep facts by episode entry_id, so re-dumps stay coherent.
- Migrate e2e tests to the entry_id model (hierarchical-eviction,
  session/timestamp filters, cluster seeding helper) and update the
  filter-error assertion to the current `INVALID_INPUT` code.
- Provision `ome.toml` in the full-app pipeline fixture (the OME config
  reloader requires it; strategies are code-registered so the packaged
  default suffices), unblocking corpus regeneration.

Full search e2e suite now green (49/49).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit 44cd9cecc07775cb9270104a5804f36e32cd788d)

* fix(lancedb): detect schema type drift and add cascade rebuild recovery

verify_business_schemas only compared column names, so a column whose
on-disk Arrow type had drifted (name unchanged) slipped through and
detonated later inside merge_insert as an opaque LanceError(IO)
"Spill has sent an error" (#337). Now compare each shared column's Arrow
type against schema.to_arrow_schema() — the exact schema get_table
builds tables from, so a healthy table never false-positives.

Reproduced #337 byte-identically: an episode.subject_vector column left
as string or null by an older build, plus a real 1024-d vector on
upsert, yields the exact crash. No lancedb version (0.13-0.34) renders
Optional[Vector] as a non-vector type, so the startup guard is what
should catch it — not the runtime.

Add `everos cascade rebuild` as the safe recovery: it drops the business
LanceDB tables and re-indexes from markdown, skipping the verify guard
(which the drift would otherwise trip on startup). Unlike removing only
.index/lancedb it re-populates already-done entries (reset_all clears
the cascade queue); unlike removing all of .index it preserves
unprocessed_buffer (messages not yet extracted).

Fixes #337.

* docs(cascade): document cascade rebuild and correct recovery guidance

Add the `everos cascade rebuild` command to the runbook, CLI, and
how-memory-works docs. Correct the old recovery guidance: a bare
`rm -rf .index/lancedb` leaves md_change_state marked `done`, so the
scanner skips those files and the index comes back empty — the runbook
previously claimed a full repopulation that does not happen. `cascade
rebuild` is the safe path (re-populates done entries, preserves
unprocessed_buffer). Also document that verify now checks column types.

* chore(rebase): adapt #354 integration test to soft-embedding main

CascadeOrchestrator dropped the embedder param when embedding became a
soft dependency (main); fold the schema-drift integration test onto it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test(cascade): freeze monotonic clock in prune-staleness health tests

Both prune-staleness health tests fabricated
`_started_at = time.monotonic() - (ALERT + 100)`, assuming monotonic()
is a large value. On a fresh CI runner monotonic() is only ~100-180s, so
the subtraction went negative, the source clamped the baseline to 0, and
staleness read back as ~130s < 900s — failing on CI while passing on
long-lived dev boxes where monotonic() is huge.

Freeze the monotonic clock via monkeypatch so staleness is deterministic
regardless of the runner's boot uptime. Source logic is unchanged; only
the tests are made hermetic.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test(cascade): wait for stable terminal state in rename scenarios

`_wait_path_done` reached a terminal status, slept a 0.1s settle window,
then asserted the status was still terminal — which contradicted its own
docstring ("absorb any last-second re-enqueue"). A rename's delete event
or an atomic-replace echo can flip a done row back to `processing` inside
that window, so on a slow CI runner the assert fired
("flipped back to processing after reaching done"), failing
test_rename_cross_owner_keeps_frontmatter_owner intermittently (seen on
the 3.13 job). `make integration` runs without `--reruns`, so a single
flake fails the whole job.

Wait for a terminal state that *survives* the settle window instead:
absorb a transient re-enqueue by waiting for terminal again, still bounded
by `deadline` so a row that never settles surfaces as a timeout. Pre-
existing flake on main, unrelated to the prune change; the scenario's real
assertions (row counts, frontmatter owner) are untouched.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(cascade): cross-process prune safety + backfill reclaim fix

Review of #379 found two P0s plus P1/P2s, all verified against the code:

- P0-1: cascade backfill still called the removed
  optimize(cleanup_older_than=…) kwarg → TypeError swallowed by the
  best-effort try/except → backfill silently skipped all compaction +
  reclaim (the exact bloat this PR fixes). CI stayed green because the test
  double kept the stale signature. Fix: call optimize() + prune(0) at the
  call site; make the fake mirror the real signature so the drift can't hide
  again; pin prune in the backfill tests.

- P0-2: prune ran delete_unverified=True guarded only by an in-process
  asyncio lock, but the runbook promises `cascade sync` is safe alongside a
  live server — and the CLI's first optimize beat does prune, in a separate
  process. It could delete files the daemon is mid-commit on. Fix: switch
  prune to delete_unverified=False. Measured to reclaim identically on
  churned tables (both collapse superseded versions ~97%); True only
  additionally deletes in-flight/dangling files — exactly the corruption
  vector. No cross-process lock needed; the write-lock/commit fix (the real
  reclaim win) is unchanged.

- P1-3: /health called orch.health() (6 SQLite aggregates) with no guard →
  a locked/full/migrating DB would 500 the liveness probe and restart the
  container. Wrap it: unhealthy readiness + reason, HTTP stays 200.

- P1-4: rebuild drops + recreates tables; a live daemon holds cached handles
  pointing at the dropped dataset. Runbook now says stop the server first —
  the one cascade command unsafe alongside a live server.

- P2: narrow _is_benign_commit_conflict to the "commit conflict" phrase (a
  bare "retryable" swallowed unrelated recoverable errors); add a timeout
  around the prune cleanup so a hung lance call can't wedge the write lock;
  correct two stale docstrings.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs: fix schema-recovery guidance + drop dead internal doc refs

Follow-up to the review; two doc issues neither the review nor the fix
commit caught:

- The schema-drift startup error's docstrings (verify_business_schemas
  and LanceDBLifespanProvider) still described the recovery as
  `rm -rf ~/.everos/.index/lancedb` — which the runbook explicitly calls
  the WRONG recovery (it leaves the cascade queue `done`, so nothing
  re-indexes and the index comes back empty). The raised error already
  points to `everos cascade rebuild`; align the docstrings to match.

- 15 dangling references to an internal numbered design-doc set
  (12_/13_/16_/17_*.md) that was never shipped to this repo. Point the
  schema-recovery ones at docs/cascade_runbook.md; drop the rest (pure
  provenance in table/component docstrings) while keeping the substance.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs: align maintenance docstrings with split optimize/prune API

Two docstring residuals from the #379 review's P2 list:

- _run_optimize_once still described the pre-split bundled heavy beat
  ("same work plus cleanup_older_than ... older than one cadence");
  the heavy beat now calls prune() under the write lock and the
  retention window is decoupled from the cadence
  (DEFAULT_OPTIMIZE_PRUNE_RETENTION_SECONDS).

- _restore_shaper_metadata converts any numeric timestamp, wider than
  an exact inverse of the bridge; document that this is deliberate
  (the shaper contract requires a datetime either way).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* style(e2e): hoist fixture-body imports to conftest module top

shutil / importlib.resources.files were imported inside the
core_pipeline_runtime fixture body; move them to the module top to
match the repo import style (#379 review P2).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(cascade): back off a hung prune via a separate attempt clock (N1)

The write-lock timeout on prune (review P2) didn't achieve its goal: it
was 300s — equal to the prune cadence — and `last_prune_at` only advanced
on success. So a hung lance cleanup timed out after 300s, `should_prune`
was still true (clock never moved), and the next beat re-pruned ~10s
later — pinning the per-table write lock ~97% of the time, the exact
write-starvation the timeout was meant to prevent.

A real cleanup is milliseconds even on a heavily churned table (measured
~40ms at 320k writes / 100 versions), so the timeout is a pure hang-catcher:
lower it to 60s (~1500x headroom, never fires normally, well below the 300s
cadence).

Split the prune clock so a failed prune backs off without masking the
health signal:
- last_prune_attempt_at (new) gates scheduling, advanced before the call
  whether it succeeds or times out → a hung prune waits a full cadence
  before retrying (light lock-free compaction runs meanwhile), so the lock
  is held at most ~timeout/cadence ≈ 17% in the worst case.
- last_prune_at advances only on success and still drives the
  prune-staleness health signal, so a persistently failing prune surfaces
  as degraded instead of being hidden by an advanced schedule clock.

Regression test: a raising prune advances the attempt clock (next beat is
light, no immediate re-prune) but not the success clock.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: zhanghui <zhanghui@shanda.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-03 15:41:13 +08:00

123 lines
4.6 KiB
Markdown
Raw Blame History

# CLI
The `everos` command-line entry point covers **setup and operations** —
generate starter config files (`init`), run the HTTP API server (`server
start`), inspect effective config (`config show`), and operate the
md → LanceDB index queue (`cascade`). Hot-path
business (`/add` `/flush` `/search` `/get`) is the **HTTP API**, not the
CLI.
CLI commands run **in-process** — they call into the `service/` /
infrastructure layers directly rather than the HTTP loopback.
## Installation
The script is exposed via `pyproject.toml`:
```toml
[project.scripts]
everos = "everos.entrypoints.cli.main:app"
```
After `uv sync` (or `pip install -e .`) the `everos` command resolves
to [`src/everos/entrypoints/cli/main.py`](../src/everos/entrypoints/cli/main.py),
a [Typer](https://typer.tiangolo.com/) app.
## Subcommand layout
```
everos
├── init [--root PATH] [--force] [--print] Generate starter config files (everos.toml + ome.toml)
├── config
│ └── show [--root PATH] Show effective configuration
├── server
│ └── start [--host] [--port] [--root] [--reload] [--log-level] Start the HTTP API server (uvicorn)
└── cascade [--root PATH] Inspect / operate the md → LanceDB sync queue
├── status Queue / LSN summary
├── sync [PATH] Drain the queue now (optional PATH force-enqueues)
└── fix [--apply] List failed rows / re-enqueue retryable ones
```
Each subcommand lives in its own module under
[`entrypoints/cli/commands/`](../src/everos/entrypoints/cli/commands/) and is
registered in `cli/main.py`. The CLI is intentionally small — hot-path
business (`/add` `/flush` `/search` `/get`) is the **HTTP API**, not the
CLI; the CLI covers setup (`init`), running the server, and index ops
(`cascade`). There is no `reindex` command — for an incremental
catch-up run `everos cascade sync`; to rebuild the whole index from
markdown (recovery from drift / corruption) run `everos cascade
rebuild`.
## `everos server start`
Wraps `uvicorn` to launch the FastAPI app from
[`entrypoints/api/app.py`](../src/everos/entrypoints/api/app.py)
in *factory* mode.
```bash
everos server start \
--host 127.0.0.1 \
--port 8000 \
--log-level info \
--root ~/.everos
```
| Flag | Env var | Default |
|---|---|---|
| `--host` | `EVEROS_API__HOST` | `127.0.0.1` (loopback only; binding `0.0.0.0` logs a warning — EverOS ships no auth) |
| `--port` | `EVEROS_API__PORT` | `8000` |
| `--log-level` | `EVEROS_LOG_LEVEL` | `INFO` |
| `--root` | `EVEROS_ROOT` | `~/.everos` |
| `--reload` | — | off (use in development) |
Lifespan startup wires the storage backends (SQLite engine + LanceDB
connection) on app boot; see
[`entrypoints/api/lifespans/`](../src/everos/entrypoints/api/lifespans/).
## Configuration via env vars
Both CLI and HTTP server read configuration from `pydantic-settings`:
| Env var | Settings field |
|---|---|
| `EVEROS_ROOT` | memory-root path (default `~/.everos`) |
| `EVEROS_MEMORY__TIMEZONE` | `Settings.memory.timezone` (e.g. `Asia/Shanghai`) |
| `EVEROS_SQLITE__BUSY_TIMEOUT_MS` | `Settings.sqlite.busy_timeout_ms` |
| `EVEROS_LANCEDB__READ_CONSISTENCY_SECONDS` | `Settings.lancedb.read_consistency_seconds` |
Pattern: `EVEROS_<SECTION>__<KEY>` (double underscore = nesting). See
[`config/settings.py`](../src/everos/config/settings.py).
## Logging
`configure_logging` runs at CLI startup and configures `structlog` with
the resolved log level. All in-process logs (CLI command bodies +
service / infra layers) flow through the same handler.
```bash
everos server start --log-level debug # see all sql / lance traffic
```
## API ↔ CLI division of labour
| Responsibility | API | CLI |
|---|---|---|
| Hot-path business (`/add` `/flush` `/search` `/get`) | ✅ | — (HTTP only) |
| Setup (generate config files) | — | `everos init` |
| Inspect effective config | — | `everos config show` |
| Run the server | — | `everos server start` |
| Index ops (drain / inspect / fix the cascade queue) | — | `everos cascade {status,sync,fix}` |
| Health probe | `GET /health` | (use HTTP) |
| Metrics scrape | `GET /metrics` | (use HTTP) |
The CLI is the **shell-friendly** surface for ops + scripting; the
HTTP API is the **process-friendly** surface for clients (web UIs,
agents, automation).
## See also
- [architecture.md](architecture.md) — DDD layering between
entrypoints / service / memory / infra
- [`entrypoints/cli/main.py`](../src/everos/entrypoints/cli/main.py)
- [`entrypoints/cli/commands/server.py`](../src/everos/entrypoints/cli/commands/server.py)
Reference in New Issue View Git Blame Copy Permalink