72c1facdbfa3c07d97d29db106482cf643ae8231
8
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
e5118c52a8 |
fix(lancedb): reclaim stale versions via write-locked prune (#379)
* fix(lancedb): reclaim stale versions via write-locked prune The storage soak (48h, sustained churn + fuzz) proved the bundled lock-free `optimize(cleanup_older_than=...)` loses its commit-conflict race against concurrent writes — cleanup ran only 16 of ~250 times, so old dataset versions / FTS orphans piled up and the index dir grew to the 40G disk guardrail and never reclaimed under load. main still had that bundled call. Split the maintenance path: - `LanceRepoBase.optimize()` is now compact-only and lock-free (a commit conflict here is benign — the next beat retries, so it must not stall writers). - `LanceRepoBase.prune(older_than)` runs `cleanup_older_than + delete_unverified=True` **under the per-table write lock**, so no writer is in flight: the Rewrite has the manifest to itself (cleanup completes every beat) and aggressive deletion is safe. It also removes the empty `_indices/<uuid>/` husks cleanup leaves behind (soak: 13061 dirs, 98% empty), offloaded to a thread. - The cascade worker's heavy beat calls `prune()`; the light beat calls `optimize()`. A benign light-beat commit conflict is logged at debug and does not count toward the failure streak or trigger a rebuild. - Prune's retention window (`cleanup_older_than`) is decoupled from the prune cadence and defaulted short (60s). It runs under the write lock, so the window only needs to outlive an in-flight read; keeping it = cadence (300s) left ~2 cadences of superseded full-table copies on disk between beats (soak: transient ~15G/table peaks). 60s reclaims all but the last minute each beat — same live floor (~625MB/table), far lower transient footprint. Result on the re-run soak: disk sawtooths and reclaims to live-data size (~1.3G total) under active load — vs run1 stuck at 40G until writes stopped — with 0 crashes / 0 OOM / 0 stuck cleanups over 48h. Cascade projection health is now observable: - `CascadeOrchestrator.health()` -> `CascadeHealth`, combining the worker's in-memory signals (drain-loop failures, unrecoverable count, optimize streak, prune staleness) with the SQLite queue summary. - `GET /health` gains a typed `cascade` readiness block. `healthy` reflects operational health only (drain / optimize / prune); `failed_permanent` (files awaiting `cascade fix`) is a data-quality backlog reported as an informational count that does NOT flip `healthy` — otherwise the signal would sit red permanently. The scanner-side retry cap and the `_MAX_TOTAL_RETRIES` budget already on main handle re-enqueue storms, so no duplicate is added here. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(deps): pin lancedb to >=0.34.0,<0.35.0 The previous open-ended `>=0.13.0` let any environment float to an untested release, including 0.32-0.34 which carry a compaction offset-overflow regression (lance-format/lance#7653) that stalls version cleanup and grows the index dir without bound. - Floor 0.34.0: the current resolved version; runs safely thanks to the with_position=False FTS workaround shipped in #336. Verified that data written by lancedb 0.32.0 (lance v6) reads correctly under 0.34.0 (lance v8), so existing deployments upgrade cleanly. Never widen the floor below 0.34 -- older lance cannot read v8-format data. - Ceiling <0.35.0: 0.35 embeds lance-rust v9 (large encoding jump, not yet stable-released); it must pass the soak harness before we allow it. Resolved version is unchanged (still 0.34.0); this only tightens the declared constraint. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> (cherry picked from commit 94f9aa67d11a07c93c7d59d6b134e80858b60316) * fix(search): bridge agent-kind metadata into the agentic doc contract The agent AGENTIC path (`agent_case` / `agent_skill`) fed recall candidates straight into `aagentic_retrieve`, whose `_format_docs` (LLM sufficiency / multi-query prompt) reads `metadata["episode"]` as a `{subject, content}` dict plus a ms-epoch `timestamp`. Agent-kind rows carry their body in the recaller's `text_field` and time as a datetime, so `_format_docs` raised `TypeError: Candidate ... has no episode dict` and `POST /api/v*/memory/search` returned 500 for any `owner_type=agent` + `method=agentic` request. Mirror the episode path's bridge: reshape agent candidate metadata into the everalgo doc contract before `aagentic_retrieve`, and revert it before DTO shaping so the agent shapers still see a datetime timestamp. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> (cherry picked from commit bb9a3a585e044643d3852b9d2810b42c05a7a47c) * test(search): regenerate search seed to current linkage + migrate e2e The committed `search_seed` fixture and several search e2e tests were written for the pre-1.5 memcell fact-linkage model. Current extraction links atomic_facts to episodes via `parent_id == episode.entry_id` (parent_type="episode"), and user_memory clusters store episode entry_id members — so the stale fixture made VECTOR/AGENTIC recall and the cluster-narrowing path find nothing, and stale assertions checked an old error code. - Regenerate `search_seed/*` from a fresh corpus in the current entry_id format; facts now bridge across multiple episodes (richer agentic / hierarchical-eviction coverage). - Fix `_dump_search_seed.py` sampling: pick episodes that host facts first and keep facts by episode entry_id, so re-dumps stay coherent. - Migrate e2e tests to the entry_id model (hierarchical-eviction, session/timestamp filters, cluster seeding helper) and update the filter-error assertion to the current `INVALID_INPUT` code. - Provision `ome.toml` in the full-app pipeline fixture (the OME config reloader requires it; strategies are code-registered so the packaged default suffices), unblocking corpus regeneration. Full search e2e suite now green (49/49). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> (cherry picked from commit 44cd9cecc07775cb9270104a5804f36e32cd788d) * fix(lancedb): detect schema type drift and add cascade rebuild recovery verify_business_schemas only compared column names, so a column whose on-disk Arrow type had drifted (name unchanged) slipped through and detonated later inside merge_insert as an opaque LanceError(IO) "Spill has sent an error" (#337). Now compare each shared column's Arrow type against schema.to_arrow_schema() — the exact schema get_table builds tables from, so a healthy table never false-positives. Reproduced #337 byte-identically: an episode.subject_vector column left as string or null by an older build, plus a real 1024-d vector on upsert, yields the exact crash. No lancedb version (0.13-0.34) renders Optional[Vector] as a non-vector type, so the startup guard is what should catch it — not the runtime. Add `everos cascade rebuild` as the safe recovery: it drops the business LanceDB tables and re-indexes from markdown, skipping the verify guard (which the drift would otherwise trip on startup). Unlike removing only .index/lancedb it re-populates already-done entries (reset_all clears the cascade queue); unlike removing all of .index it preserves unprocessed_buffer (messages not yet extracted). Fixes #337. * docs(cascade): document cascade rebuild and correct recovery guidance Add the `everos cascade rebuild` command to the runbook, CLI, and how-memory-works docs. Correct the old recovery guidance: a bare `rm -rf .index/lancedb` leaves md_change_state marked `done`, so the scanner skips those files and the index comes back empty — the runbook previously claimed a full repopulation that does not happen. `cascade rebuild` is the safe path (re-populates done entries, preserves unprocessed_buffer). Also document that verify now checks column types. * chore(rebase): adapt #354 integration test to soft-embedding main CascadeOrchestrator dropped the embedder param when embedding became a soft dependency (main); fold the schema-drift integration test onto it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(cascade): freeze monotonic clock in prune-staleness health tests Both prune-staleness health tests fabricated `_started_at = time.monotonic() - (ALERT + 100)`, assuming monotonic() is a large value. On a fresh CI runner monotonic() is only ~100-180s, so the subtraction went negative, the source clamped the baseline to 0, and staleness read back as ~130s < 900s — failing on CI while passing on long-lived dev boxes where monotonic() is huge. Freeze the monotonic clock via monkeypatch so staleness is deterministic regardless of the runner's boot uptime. Source logic is unchanged; only the tests are made hermetic. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(cascade): wait for stable terminal state in rename scenarios `_wait_path_done` reached a terminal status, slept a 0.1s settle window, then asserted the status was still terminal — which contradicted its own docstring ("absorb any last-second re-enqueue"). A rename's delete event or an atomic-replace echo can flip a done row back to `processing` inside that window, so on a slow CI runner the assert fired ("flipped back to processing after reaching done"), failing test_rename_cross_owner_keeps_frontmatter_owner intermittently (seen on the 3.13 job). `make integration` runs without `--reruns`, so a single flake fails the whole job. Wait for a terminal state that *survives* the settle window instead: absorb a transient re-enqueue by waiting for terminal again, still bounded by `deadline` so a row that never settles surfaces as a timeout. Pre- existing flake on main, unrelated to the prune change; the scenario's real assertions (row counts, frontmatter owner) are untouched. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(cascade): cross-process prune safety + backfill reclaim fix Review of #379 found two P0s plus P1/P2s, all verified against the code: - P0-1: cascade backfill still called the removed optimize(cleanup_older_than=…) kwarg → TypeError swallowed by the best-effort try/except → backfill silently skipped all compaction + reclaim (the exact bloat this PR fixes). CI stayed green because the test double kept the stale signature. Fix: call optimize() + prune(0) at the call site; make the fake mirror the real signature so the drift can't hide again; pin prune in the backfill tests. - P0-2: prune ran delete_unverified=True guarded only by an in-process asyncio lock, but the runbook promises `cascade sync` is safe alongside a live server — and the CLI's first optimize beat does prune, in a separate process. It could delete files the daemon is mid-commit on. Fix: switch prune to delete_unverified=False. Measured to reclaim identically on churned tables (both collapse superseded versions ~97%); True only additionally deletes in-flight/dangling files — exactly the corruption vector. No cross-process lock needed; the write-lock/commit fix (the real reclaim win) is unchanged. - P1-3: /health called orch.health() (6 SQLite aggregates) with no guard → a locked/full/migrating DB would 500 the liveness probe and restart the container. Wrap it: unhealthy readiness + reason, HTTP stays 200. - P1-4: rebuild drops + recreates tables; a live daemon holds cached handles pointing at the dropped dataset. Runbook now says stop the server first — the one cascade command unsafe alongside a live server. - P2: narrow _is_benign_commit_conflict to the "commit conflict" phrase (a bare "retryable" swallowed unrelated recoverable errors); add a timeout around the prune cleanup so a hung lance call can't wedge the write lock; correct two stale docstrings. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: fix schema-recovery guidance + drop dead internal doc refs Follow-up to the review; two doc issues neither the review nor the fix commit caught: - The schema-drift startup error's docstrings (verify_business_schemas and LanceDBLifespanProvider) still described the recovery as `rm -rf ~/.everos/.index/lancedb` — which the runbook explicitly calls the WRONG recovery (it leaves the cascade queue `done`, so nothing re-indexes and the index comes back empty). The raised error already points to `everos cascade rebuild`; align the docstrings to match. - 15 dangling references to an internal numbered design-doc set (12_/13_/16_/17_*.md) that was never shipped to this repo. Point the schema-recovery ones at docs/cascade_runbook.md; drop the rest (pure provenance in table/component docstrings) while keeping the substance. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: align maintenance docstrings with split optimize/prune API Two docstring residuals from the #379 review's P2 list: - _run_optimize_once still described the pre-split bundled heavy beat ("same work plus cleanup_older_than ... older than one cadence"); the heavy beat now calls prune() under the write lock and the retention window is decoupled from the cadence (DEFAULT_OPTIMIZE_PRUNE_RETENTION_SECONDS). - _restore_shaper_metadata converts any numeric timestamp, wider than an exact inverse of the bridge; document that this is deliberate (the shaper contract requires a datetime either way). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * style(e2e): hoist fixture-body imports to conftest module top shutil / importlib.resources.files were imported inside the core_pipeline_runtime fixture body; move them to the module top to match the repo import style (#379 review P2). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(cascade): back off a hung prune via a separate attempt clock (N1) The write-lock timeout on prune (review P2) didn't achieve its goal: it was 300s — equal to the prune cadence — and `last_prune_at` only advanced on success. So a hung lance cleanup timed out after 300s, `should_prune` was still true (clock never moved), and the next beat re-pruned ~10s later — pinning the per-table write lock ~97% of the time, the exact write-starvation the timeout was meant to prevent. A real cleanup is milliseconds even on a heavily churned table (measured ~40ms at 320k writes / 100 versions), so the timeout is a pure hang-catcher: lower it to 60s (~1500x headroom, never fires normally, well below the 300s cadence). Split the prune clock so a failed prune backs off without masking the health signal: - last_prune_attempt_at (new) gates scheduling, advanced before the call whether it succeeds or times out → a hung prune waits a full cadence before retrying (light lock-free compaction runs meanwhile), so the lock is held at most ~timeout/cadence ≈ 17% in the worst case. - last_prune_at advances only on success and still drives the prune-staleness health signal, so a persistently failing prune surfaces as degraded instead of being hidden by an advanced schedule clock. Regression test: a raising prune advances the attempt clock (next beat is light, no immediate re-prune) but not the success clock. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: zhanghui <zhanghui@shanda.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
b1441da607 |
refactor(config): make [embedding] and [rerank] soft dependencies (#361)
* refactor(config): make [embedding] and [rerank] soft dependencies
Make [embedding] and [rerank] soft runtime dependencies so a
freshly-onboarded user can run EverOS end-to-end with only [llm]
configured. Previously the server refused to start without embedding,
locking out anyone who just wanted keyword-only search.
## Capability tiers
- Tier 1 ([llm] only) KEYWORD search, add/flush, md
writes, cascade sync
- Tier 2 ([llm] + [embedding]) + VECTOR / HYBRID search,
reflection, skill extraction,
backfill
- Tier 3 ([llm] + [embedding] + rerank) + AGENTIC search, knowledge
Tier upgrades require a server restart (capability accessors cache
for the process lifetime). Tier downgrades are read-safe: a Tier-3
user who drops [rerank] can still read/rename/delete existing
knowledge documents; only write/search endpoints return 422.
## What changed
- Component accessors — component/{embedding,rerank,llm}/accessor.py
are the single process-wide provider singletons. service/* never
maintains parallel singletons; it consumes get_embedding_capability()
/ get_rerank_capability() / get_llm_client() directly. Build-time
ValueError from the factory is logged as capability_build_failed
(was silently swallowed).
- Error mapping — ProviderNotConfiguredError -> 422 with everos.toml
section hints (never EVEROS_* env-var strings).
LanceDBMigrationError fails loud with escalating recovery guidance
(restart -> wipe index). LLMNotConfiguredError in search maps to
None for KEYWORD degradation.
- Nullable-vector LanceDB migration — schema v2 makes the vector
column nullable so Tier-1 rows can land without embeddings.
Migration is guarded by a cross-process memory_root_lock
(fcntl.flock + anyio.to_thread) and runs optimize() per table
after Phase-1 backfill to reclaim manifest bloat.
- Cascade — knowledge handlers register unconditionally (Tier-3 ->
Tier-2/1 downgrade no longer strands DELETE); embed-requiring
strategies use body-guards that check capability.available at
execution time. _TABLE_SPECS has an import-time drift assertion
against BUSINESS_SCHEMAS_WITH_VECTOR.
- `everos cascade backfill` CLI — Phase-1 (embed missing vectors) /
Phase-2 (emit synthetic events for cascaded processing) / Phase-3
(sync new skill files). Exit codes: 0 / 1 / 2 / 3 (server running
preflight) / 4 (COMPLETED_WITH_FAILURES — per-row failures rolled
up) / 130 (SIGINT). OMEConfig.crash_recovery_enabled=False in
backfill engines prevents stale-RUNNING rows re-enqueuing into a
smaller strategy registry.
- /health — reports capabilities + disabled_features per tier so ops
can distinguish "boots but degraded" from "boots and full".
- Presentation split — memory / service / infra never import typer /
click. TyperPresenter Protocol + run_backfill() live in
entrypoints/cli/commands/_backfill_cmd.py. Enforced by
import-linter.
- Startup hint — unconditional count_rows(filter="vector IS NULL")
sweep emits unbackfilled_memory_rows (event name + hint text
pinned) when Tier-1 rows exist. ParserLifespanProvider warms the
everalgo.parser import at boot so /health doesn't block on first
call.
- Knowledge upload UTF-8 short-circuit — _looks_like_utf8_text()
routes text/* mime and known plaintext extensions (md/txt/rst)
straight to UTF-8 decode instead of the parser. Prevents 503
Multimodal-not-configured when Tier 3 sans [multimodal] uploads a
markdown doc.
## Sync history with main (2 merges collapsed into this squash)
Merged origin/main at
|
||
|
|
649046b0df |
docs(api): use /api/v2 in docs and examples, demote v1 to legacy (#370)
1.2.0 introduced /api/v2 as the canonical, cloud-aligned prefix and mounted every business router twice, but the user-facing entry points (README, README.zh-CN, QUICKSTART, the docs/ set, the Langfuse example) still taught /api/v1 — so new users were pointed at the compatibility alias while docs/api.md already declared v2 canonical. - Switch every EverOS endpoint reference in docs, examples, and `everos demo --live` to /api/v2, plus the matching CLI test expectations. - Describe /api/v1 as a legacy compatibility alias that may be removed in a future major release, rather than a permanent one. Nothing changes at runtime: both prefixes still resolve to the same handlers and the v1/v2 parity test is untouched. - Add a short note in README / README.zh-CN / QUICKSTART so existing v1 integrations know they keep working. - Fix the five dead endpoint anchors in the docs/api.md table of contents, which still pointed at the pre-1.2.0 #post-apiv1... slugs. Left on v1 deliberately: docs/migration-to-1.0.0.md (historical record), CHANGELOG history, tests/** (v1 must stay covered), and the use-cases/claude-code-plugin + openher READMEs, which document a different cloud API. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
21b845bf88 |
feat(api): serve endpoints under /api/v2, retain /api/v1 as alias
Every business endpoint (memory/*, ome/*, knowledge/*) is now served under /api/v2, aligning the open-source API with the EverOS Cloud contract. /api/v1 is retained as a permanent, backward-compatible alias: the same router objects are mounted under both prefixes, so both resolve to identical handlers and request/response contracts. Existing /api/v1 integrations keep working unchanged. Infra endpoints (/health, /metrics) stay unversioned. Fix the Prometheus request-metric label to build the path from the full request URL (with path params folded) rather than the route's router-relative path, so the version prefix is preserved and v1/v2 traffic stays distinguishable. Docs (docs/api.md, docs/openapi.json), CHANGELOG, and route docstrings updated to lead with /api/v2. Add test_api_versioning as the parity guard: every v2 route has an identical v1 twin and vice versa. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
15efd1198c | chore(release): update EverOS to 1.1.1 (#327) | ||
|
|
0df88f5603 | chore(release): update EverOS to 1.1.0 (#307) | ||
|
|
a10cdcd197 | chore(release): prepare EverOS 1.0.1 (#290) | ||
|
|
518b8eca85 |
chore: initialize EverOS 1.0.0
md-first memory extraction framework for AI agents. Markdown is the single source of truth; SQLite holds state and LanceDB provides the rebuildable vector + BM25 + scalar index. The codebase follows a single-direction DDD layering (entrypoints -> service -> memory -> infra, with component / core / config cross-cutting) enforced by import-linter. Engineering surface: - Coding conventions in .claude/rules/ (path-scoped) and workflows in .claude/skills/ (/commit, /new-branch, /pr). - GitHub Actions CI runs make lint + test + integration; pre-commit mirrors the gates locally (ruff, hygiene hooks, gitlint commit-msg). - Commit messages follow Conventional Commits, enforced by gitlint. - make lint also enforces datetime two-zone discipline and OpenAPI drift. |