72c1facdbfa3c07d97d29db106482cf643ae8231
78
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
72c1facdbf |
fix(cascade): record which beat lost the optimize commit race (#390)
The benign-conflict log said a commit race was lost but not by which
maintenance beat, and lance labels both beats' commit identically
("This Rewrite transaction was preempted by concurrent transaction ..."),
so the message alone cannot separate them. The costs differ sharply:
- a lost LIGHT beat is free — compaction retries ~10s later;
- a lost HEAVY beat means that table skipped a whole prune cadence, so
its superseded files stay on disk until the next one lands.
Reading an index-dir growth incident off the logs therefore meant
back-inferring which beats were heavy from the 300s cadence. That was
done once during the storage soak to explain a 13-minute window where
one table's cleanup stalled with no failure logged, and it cost two
35-minute debug-level reruns to confirm — the field makes it a grep.
Adds `pruned` to the conflict log, mirroring the sibling failure log.
Log level (debug) and the benign-conflict semantics are unchanged: the
streak is still not incremented and no fallback rebuild is triggered.
Test asserts both beats in one run: the heavy beat's conflict logs
pruned=True and the light beat's logs pruned=False. Mutation-verified —
dropping the field fails the test with KeyError.
Co-authored-by: zhanghui <zhanghui@shanda.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Elliot Chen <2340896+cyfyifanchen@users.noreply.github.com>
|
||
|
|
ace0d79486 |
ci: draft the GitHub Release page from the CHANGELOG on tag (#389)
* ci: draft the GitHub Release page from the CHANGELOG on tag release.yml only published to PyPI; creating the Release page was an undocumented manual step, and v1.2.2 shipped to PyPI without one. A second job, gated on a successful upload, now drafts the page: title "EverOS X.Y.Z", body lifted from the matching CHANGELOG section with the group headings demoted to h2, `--prerelease` for PEP 440 suffixed tags. It stays a draft because the lead summary that opens every EverOS release page is prose CI cannot write. The job runs separately from the publish job so that one keeps `contents: read` alongside its OIDC token. A stable tag with no CHANGELOG section fails the job rather than publishing an empty page; pre-releases fall back to a placeholder. Existing releases are detected through the list endpoint, since the by-tag endpoint cannot see drafts (cli/cli#3037) and a re-run would 422 — a stale draft is replaced, a published release left alone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci: prefill the Upgrade section on the drafted release page Every EverOS release page since 1.1.3 closes with an Upgrade section — the pip line, any migration notes, and a compare link to the previous tag. The first draft dropped it, so a published page would have lost the one section readers act on. The pip line and the compare link are prefilled; the previous tag comes from `git tag --sort=-v:refname`, which needs the full tag list, hence fetch-depth: 0. Migration notes stay hand-written next to the lead summary, since only a human knows whether a release needs them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(release): warn that API publishing drops the tag Flipping `draft` through the API without `tag_name` rebinds the release to the `untagged-<hash>` placeholder and creates a git tag by that name against the default branch. Hit while publishing 1.2.2; the web UI button is unaffected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: zhanghui <zhanghui@shanda.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
ae0e062fb4 |
ci: enforce the file-size ceiling on pull requests (#387)
* ci: enforce the file-size ceiling on pull requests `check-added-large-files` only ran in pre-commit, so the ceiling was absent from CI: an unhooked clone or `--no-verify` bypassed it entirely, and the hook is weaker than it looks even locally — it inspects only files being *added*, so an existing fixture that grows never trips it. Add `scripts/check_file_sizes.py`, wired into `make lint` and therefore the required `lint` check. It diffs against the base branch's merge base and measures additions, modifications and renames, leaving files already committed above the ceiling alone so no pull request fails for something it did not touch. An unresolvable base is a hard failure rather than a silent pass. The `lint` job now checks out with `fetch-depth: 0` to provide it. Lower the hook's `--maxkb` from 1024 to 640 to match, and pin the two limits equal in a unit test so they cannot drift back apart. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci: catch untracked oversized files in the size gate `git diff` cannot see a newly created file until it is staged, so a local `make check-file-sizes` passed a brand-new 700 KB file — verified against the real script, not reasoned about. CI was unaffected (its checkout has everything committed), but the docstring claimed the local run covered uncommitted work, which was only true for edits to already-tracked files. Union in `git ls-files --others --exclude-standard`, which respects .gitignore, and pin both the untracked and the ignored case in tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * ci: exempt the generated search-seed corpora from the size ceiling tests/fixtures/search_seed/ holds embedded corpora regenerated by _dump_search_seed.py. Two of them are already near 1 MB and grew ~60% in one release, so the next refresh would have hit the 640 KB ceiling and the cheapest fix would have been raising it for the whole repository — a gate that teaches people to edit the gate. Exempt that one directory by path prefix instead. Outside it the largest tracked file is ~300 KB, so 640 KB still binds where it matters, including the examples/ case that prompted this work. Three tests pin the carve-out: a sibling of the exempt directory is still caught, the list itself is asserted verbatim so growth shows up in review, and every prefix must name a directory that actually exists. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: zhanghui <zhanghui@shanda.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
84554ebe22 |
chore(release): v1.2.2 (#386)
* fix(lancedb): put table-handle resolution inside the deadline run9 reproduced the stall the previous commits were supposed to close, on a different table: episode went 13 minutes without a prune — versions climbing 63→66 while foresight and atomic_fact both collapsed to 1 — with **zero** failure, timeout, or conflict logs. Its last successful prune was logged at 11:41:59 and the staleness clock matched to the second. The deadline covered the critical section but not the await ahead of it: `table = await self._table()` sat outside `_locked`, so a hang while resolving the table handle never returned. The scheduler runs one maintenance task per kind and skips a kind whose task is still in flight, so that kind stops being maintained permanently, silently, because nothing failed. Move the handle resolution inside the deadline for all seven locked operations, and give the lock-free compaction beat its own `_deadline` (it takes no lock so it cannot block writers, but it can still park a kind by never returning). Belt and braces in the scheduler: both beats now run under `_MAINTENANCE_TASK_TIMEOUT_SECONDS`, a last-resort bound on the whole call, so any await I have not thought of costs one cadence rather than forever. Regression test: a repo whose `_table_lookup` never resolves must make prune, optimize and add all raise `VectorStoreBusyError` and leave the lock free. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(release): v1.2.2 Bump to 1.2.2 and cut the changelog. This release is the storage-layer reliability work: LanceDB maintenance split into lock-free compaction and write-locked reclamation (fixing unbounded index growth), every write-lock critical section bounded by a deadline that covers acquisition, a per-kind prune-staleness signal on GET /health, `everos cascade rebuild` for a drifted or corrupt index, startup detection of column type drift, and a query-vector width check that fails fast instead of 13s deep inside LanceDB. Carries the table-handle deadline fix (previously #385) rather than shipping 1.2.2 with a known stall: the deadline covered the critical section but not the await ahead of it, so a hang while resolving a table handle parked that kind's maintenance permanently and silently. Found by a 1h high-rate soak run after #384 merged. No migration, no config change, no API change: `docs/openapi.json` differs only in the version string. The only operator-visible requirement is that `everos cascade rebuild` now refuses to run while a server holds the OME lock. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: zhanghui <zhanghui@shanda.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>v1.2.2 |
||
|
|
d256048a6d |
fix(cascade): per-kind prune staleness + rebuild safety (#384)
* fix(cascade): per-kind prune staleness + rebuild safety Adversarial review of the review-response fixes found five real defects, all in code this PR introduced. Health signal (P1): prune staleness reported the time since the NEWEST successful prune across kinds, so on a multi-kind deployment (every real one) a single kind whose cleanup died was masked by the others pruning on schedule — /health stayed green while that table's index dir grew unbounded, the exact incident the signal exists to catch. Report the WORST kind instead and name it in the reason. The failure streak could not cover this either: an intervening light beat resets it, so it never reaches the threshold for a prune-only failure. Documented that split of duties. Spurious fallback rebuilds (P1): the benign-conflict carve-out excluded the heavy beat, justified by "runs under the write lock, so it can't hit this benignly" — but that lock is in-process only, so a second process (a long `cascade backfill`, a `cascade sync`) preempts prune's Rewrite commit. Those counted as real failures, and ~25min of cross-process churn reached the threshold and fired a fallback rebuild, which drops every index before recreating it; a rebuild that also lost the race was swallowed as a warning, leaving the table with no FTS index (every /search on that kind 500s) until the next 12h sweep. Treat commit conflicts as benign on both beats and let prune-staleness detect a prune that genuinely stops succeeding. cascade rebuild (P1 ×2 + P2): it drops and recreates tables with no guard while --help/docstrings advertised it as safe, so `rebuild --yes` against a live daemon corrupted the rebuild (the daemon keeps writing through cached handles). Refuse when the OME jobstore lock is held, reusing backfill's detection and its exit code 3. It also ran the pre-drop migration pass (`ensure_business_indexes`) against the damaged table, so on the corruption classes it exists to repair (missing column, un-alterable type) the recovery path died on the damage itself — skip it via `_runtime(ensure=False)`. Reset the queue BEFORE dropping so every crash window converges on "queue pending → re-index" instead of empty tables with a fully-done queue (a silently empty deployment), and handle Ctrl-C with exit 130 plus a resume hint. Recovery guidance (P1): the nullable-vector migration error still told users to wipe the index directory — which this PR's own runbook documents as the wrong recovery (queue stays done, index comes back empty). Point it at `everos cascade rebuild`. Dropped the schema-drift error's "restart first" step too: the startup migrations only alter nullability, never a name or type, so a name/type drift never self-heals. Also: backfill's post-write prune passed a zero retention window from a separate process, able to delete files under a daemon /search still holding that version — pass the daemon's window instead. Runbook gains the /health cascade block (thresholds, what flips healthy, why failed_permanent does not) and its quoted schema-drift error now matches the code. Tests: the three safety mechanisms this PR adds were unpinned — a one-line revert of any of them passed the suite. Added per-kind staleness, heavy-beat benign conflict, benign-filter negative case (an error whose message merely contains "retryable" must still count), prune recurrence across light beats (mutation-verified: hoisting the attempt-clock advance out of the heavy branch turns it red), the prune timeout releasing the write lock, timeout-below-cadence, the rebuild server guard, and a tier3 assertion that the /health cascade block is actually wired. Froze the last fabricated-monotonic test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Also: treat ``last_prune_attempt_at == 0.0`` as "never attempted" instead of comparing clocks. ``monotonic()`` is boot-relative, so ``now - 0 >= cadence`` is false for the first ~cadence of container uptime — the catch-up prune was skipped exactly when a fresh process most needs it (and made a test depend on the runner uptime, which CI caught). * fix(lancedb): bound every write-lock critical section run7 (1h at 2.5x rate, concurrent CLI maintenance, doubled fuzz) reproduced a table whose version cleanup stopped permanently: 150 versions retained, disk 11x live size, while the other two tables sat at 1 version each — and with no error logged anywhere, because nothing failed. It simply never returned. Three things combined. The maintenance scheduler allows one task per table (a LanceDB table takes one writer), so it skips a kind whose task is still in flight. The prune timeout sat *inside* the lock and covered only the cleanup call. And the other six critical sections on that lock — add, upsert, update, delete, delete_by_md_path, rebuild_indexes — had no deadline at all. So one operation stuck anywhere outside that narrow window wedged the table for good: every writer blocked on acquire, and every later heartbeat was turned away because the stuck task never finished. Make it structurally impossible instead of patching prune: all seven sections now go through `LanceRepoBase._locked(budget, op)`, where the deadline covers **acquisition and the body**. No path can wait for this lock, or hold it, indefinitely. Budgets are hang-catchers, not throughput limits: 120s for row writes, 600s for an index rebuild, the existing 60s for prune. Expiry raises `VectorStoreBusyError`, deliberately under `ExternalServiceError` so the cascade worker retries the row; under `VectorStoreError` a transient lock contention would be marked permanently failed and need a manual `cascade fix`. Tests: a stuck holder now makes a waiter fail its deadline and release (the lock is reusable afterwards), and the prune timeout is pinned as retryable. Verified by mutation — moving the timeout back inside the lock makes a waiter block until the enclosing observation window expires (1001ms vs 51ms), i.e. wait forever in production. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(lancedb): size write-lock budgets from measurements 120s for a row write was a guess, and a bad one: the budget doubles as the detection latency for a wedged table, so an over-slack value means minutes of blocked writers before anything surfaces — the failure this change exists to prevent. Measured the four locked write ops on a local SSD across table sizes and batch sizes (10k-100k rows, 50-500 rows per call): add 3-22ms, upsert (merge_insert, the read-modify-write one) 6-25ms, update 2-4ms, delete 2-3ms; worst observation 63ms, and flat in both dimensions since these are append-and-commit, not scans. So: writes 120s -> 15s (~240x the worst observation, enough for a contended disk and several waiters queued ahead — the deadline includes acquisition and asyncio.Lock is FIFO), rebuild 600s -> 300s (still the one genuinely slow section at ~0.3s per 50k rows per indexed column). Prune stays 60s. Test pins the sizing intent: writes stay in the tens of seconds, and rebuild > prune > write so the slowest section is not the most eagerly killed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(lancedb): record wait/hold time on write-lock critical sections A soak run stalled one table's writes for ~16s and the logs could not say why: maintenance beats only log at `debug`, so a section that is slow but still inside its deadline is invisible, and the timeout warning did not distinguish "never acquired the lock" from "acquired it and overran". `_locked` now carries that apart. The deadline warning gains `acquired`, `waited_seconds` and `held_seconds` — `acquired` alone answers whether a holder was slow or this operation was — and a completed section that held the lock for at least a second logs `lancedb_write_lock_slow_hold` at info, so a stall that never reaches a deadline still leaves a trace. Uses `time.monotonic` (elapsed measurement, not wall clock — the datetime discipline bans `time.time`). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(search): reject a mismatched query vector before it reaches LanceDB A soak run showed every slow search was a failing search: `search:vector` p50 251ms / p99 1.6s, but its 8 requests over 10s were exactly its 8 failures (13-14s each). The cause was a query vector whose width disagreed with the index. LanceDB only notices after the query is built and reports it as an opaque `ValueError: Invalid input, No vector column found to match…`, which escaped as an unhandled 500. Validate at `_embed_query` — the single point every query vector passes through — against the provider's declared `dim`. Microseconds instead of 13s, and a named `ConfigurationError` (500 + CONFIGURATION_ERROR) instead of an unhandled crash. Deliberately not `InvalidInputError`/422: callers only send query *text*, so a bad width is our provider's fault, not the caller's. Also cap traceback rendering. structlog's default is `RichTracebackFormatter(show_locals=True, max_frames=100, extra_lines=3)`, which on an async stack rendered 82 frames into 6423 log lines per exception — 85MB of server.log across 11 of them — at ~290ms of synchronous CPU each, and risks printing request payloads into logs. With locals off and 15 frames the same traceback is 103 lines and 10ms. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(changelog): record the storage-reliability work under Unreleased #379 merged without changelog entries, so this covers both it and the follow-up work in this branch: the maintenance split (compaction vs reclamation) that fixes unbounded index growth, bounded write-lock critical sections, the /health cascade readiness block and its alert contract, `cascade rebuild`, schema type-drift detection, the query-vector width check, and the traceback-rendering cap. Each entry states the operator-visible consequence, not just the change — `cascade rebuild` now refusing to run against a live server, benign-conflict warnings dropping in volume, and `/health` being able to report a stalled kind that was previously invisible are all behaviour changes someone will notice. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: zhanghui <zhanghui@shanda.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
e5118c52a8 |
fix(lancedb): reclaim stale versions via write-locked prune (#379)
* fix(lancedb): reclaim stale versions via write-locked prune The storage soak (48h, sustained churn + fuzz) proved the bundled lock-free `optimize(cleanup_older_than=...)` loses its commit-conflict race against concurrent writes — cleanup ran only 16 of ~250 times, so old dataset versions / FTS orphans piled up and the index dir grew to the 40G disk guardrail and never reclaimed under load. main still had that bundled call. Split the maintenance path: - `LanceRepoBase.optimize()` is now compact-only and lock-free (a commit conflict here is benign — the next beat retries, so it must not stall writers). - `LanceRepoBase.prune(older_than)` runs `cleanup_older_than + delete_unverified=True` **under the per-table write lock**, so no writer is in flight: the Rewrite has the manifest to itself (cleanup completes every beat) and aggressive deletion is safe. It also removes the empty `_indices/<uuid>/` husks cleanup leaves behind (soak: 13061 dirs, 98% empty), offloaded to a thread. - The cascade worker's heavy beat calls `prune()`; the light beat calls `optimize()`. A benign light-beat commit conflict is logged at debug and does not count toward the failure streak or trigger a rebuild. - Prune's retention window (`cleanup_older_than`) is decoupled from the prune cadence and defaulted short (60s). It runs under the write lock, so the window only needs to outlive an in-flight read; keeping it = cadence (300s) left ~2 cadences of superseded full-table copies on disk between beats (soak: transient ~15G/table peaks). 60s reclaims all but the last minute each beat — same live floor (~625MB/table), far lower transient footprint. Result on the re-run soak: disk sawtooths and reclaims to live-data size (~1.3G total) under active load — vs run1 stuck at 40G until writes stopped — with 0 crashes / 0 OOM / 0 stuck cleanups over 48h. Cascade projection health is now observable: - `CascadeOrchestrator.health()` -> `CascadeHealth`, combining the worker's in-memory signals (drain-loop failures, unrecoverable count, optimize streak, prune staleness) with the SQLite queue summary. - `GET /health` gains a typed `cascade` readiness block. `healthy` reflects operational health only (drain / optimize / prune); `failed_permanent` (files awaiting `cascade fix`) is a data-quality backlog reported as an informational count that does NOT flip `healthy` — otherwise the signal would sit red permanently. The scanner-side retry cap and the `_MAX_TOTAL_RETRIES` budget already on main handle re-enqueue storms, so no duplicate is added here. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(deps): pin lancedb to >=0.34.0,<0.35.0 The previous open-ended `>=0.13.0` let any environment float to an untested release, including 0.32-0.34 which carry a compaction offset-overflow regression (lance-format/lance#7653) that stalls version cleanup and grows the index dir without bound. - Floor 0.34.0: the current resolved version; runs safely thanks to the with_position=False FTS workaround shipped in #336. Verified that data written by lancedb 0.32.0 (lance v6) reads correctly under 0.34.0 (lance v8), so existing deployments upgrade cleanly. Never widen the floor below 0.34 -- older lance cannot read v8-format data. - Ceiling <0.35.0: 0.35 embeds lance-rust v9 (large encoding jump, not yet stable-released); it must pass the soak harness before we allow it. Resolved version is unchanged (still 0.34.0); this only tightens the declared constraint. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> (cherry picked from commit 94f9aa67d11a07c93c7d59d6b134e80858b60316) * fix(search): bridge agent-kind metadata into the agentic doc contract The agent AGENTIC path (`agent_case` / `agent_skill`) fed recall candidates straight into `aagentic_retrieve`, whose `_format_docs` (LLM sufficiency / multi-query prompt) reads `metadata["episode"]` as a `{subject, content}` dict plus a ms-epoch `timestamp`. Agent-kind rows carry their body in the recaller's `text_field` and time as a datetime, so `_format_docs` raised `TypeError: Candidate ... has no episode dict` and `POST /api/v*/memory/search` returned 500 for any `owner_type=agent` + `method=agentic` request. Mirror the episode path's bridge: reshape agent candidate metadata into the everalgo doc contract before `aagentic_retrieve`, and revert it before DTO shaping so the agent shapers still see a datetime timestamp. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> (cherry picked from commit bb9a3a585e044643d3852b9d2810b42c05a7a47c) * test(search): regenerate search seed to current linkage + migrate e2e The committed `search_seed` fixture and several search e2e tests were written for the pre-1.5 memcell fact-linkage model. Current extraction links atomic_facts to episodes via `parent_id == episode.entry_id` (parent_type="episode"), and user_memory clusters store episode entry_id members — so the stale fixture made VECTOR/AGENTIC recall and the cluster-narrowing path find nothing, and stale assertions checked an old error code. - Regenerate `search_seed/*` from a fresh corpus in the current entry_id format; facts now bridge across multiple episodes (richer agentic / hierarchical-eviction coverage). - Fix `_dump_search_seed.py` sampling: pick episodes that host facts first and keep facts by episode entry_id, so re-dumps stay coherent. - Migrate e2e tests to the entry_id model (hierarchical-eviction, session/timestamp filters, cluster seeding helper) and update the filter-error assertion to the current `INVALID_INPUT` code. - Provision `ome.toml` in the full-app pipeline fixture (the OME config reloader requires it; strategies are code-registered so the packaged default suffices), unblocking corpus regeneration. Full search e2e suite now green (49/49). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> (cherry picked from commit 44cd9cecc07775cb9270104a5804f36e32cd788d) * fix(lancedb): detect schema type drift and add cascade rebuild recovery verify_business_schemas only compared column names, so a column whose on-disk Arrow type had drifted (name unchanged) slipped through and detonated later inside merge_insert as an opaque LanceError(IO) "Spill has sent an error" (#337). Now compare each shared column's Arrow type against schema.to_arrow_schema() — the exact schema get_table builds tables from, so a healthy table never false-positives. Reproduced #337 byte-identically: an episode.subject_vector column left as string or null by an older build, plus a real 1024-d vector on upsert, yields the exact crash. No lancedb version (0.13-0.34) renders Optional[Vector] as a non-vector type, so the startup guard is what should catch it — not the runtime. Add `everos cascade rebuild` as the safe recovery: it drops the business LanceDB tables and re-indexes from markdown, skipping the verify guard (which the drift would otherwise trip on startup). Unlike removing only .index/lancedb it re-populates already-done entries (reset_all clears the cascade queue); unlike removing all of .index it preserves unprocessed_buffer (messages not yet extracted). Fixes #337. * docs(cascade): document cascade rebuild and correct recovery guidance Add the `everos cascade rebuild` command to the runbook, CLI, and how-memory-works docs. Correct the old recovery guidance: a bare `rm -rf .index/lancedb` leaves md_change_state marked `done`, so the scanner skips those files and the index comes back empty — the runbook previously claimed a full repopulation that does not happen. `cascade rebuild` is the safe path (re-populates done entries, preserves unprocessed_buffer). Also document that verify now checks column types. * chore(rebase): adapt #354 integration test to soft-embedding main CascadeOrchestrator dropped the embedder param when embedding became a soft dependency (main); fold the schema-drift integration test onto it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(cascade): freeze monotonic clock in prune-staleness health tests Both prune-staleness health tests fabricated `_started_at = time.monotonic() - (ALERT + 100)`, assuming monotonic() is a large value. On a fresh CI runner monotonic() is only ~100-180s, so the subtraction went negative, the source clamped the baseline to 0, and staleness read back as ~130s < 900s — failing on CI while passing on long-lived dev boxes where monotonic() is huge. Freeze the monotonic clock via monkeypatch so staleness is deterministic regardless of the runner's boot uptime. Source logic is unchanged; only the tests are made hermetic. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(cascade): wait for stable terminal state in rename scenarios `_wait_path_done` reached a terminal status, slept a 0.1s settle window, then asserted the status was still terminal — which contradicted its own docstring ("absorb any last-second re-enqueue"). A rename's delete event or an atomic-replace echo can flip a done row back to `processing` inside that window, so on a slow CI runner the assert fired ("flipped back to processing after reaching done"), failing test_rename_cross_owner_keeps_frontmatter_owner intermittently (seen on the 3.13 job). `make integration` runs without `--reruns`, so a single flake fails the whole job. Wait for a terminal state that *survives* the settle window instead: absorb a transient re-enqueue by waiting for terminal again, still bounded by `deadline` so a row that never settles surfaces as a timeout. Pre- existing flake on main, unrelated to the prune change; the scenario's real assertions (row counts, frontmatter owner) are untouched. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(cascade): cross-process prune safety + backfill reclaim fix Review of #379 found two P0s plus P1/P2s, all verified against the code: - P0-1: cascade backfill still called the removed optimize(cleanup_older_than=…) kwarg → TypeError swallowed by the best-effort try/except → backfill silently skipped all compaction + reclaim (the exact bloat this PR fixes). CI stayed green because the test double kept the stale signature. Fix: call optimize() + prune(0) at the call site; make the fake mirror the real signature so the drift can't hide again; pin prune in the backfill tests. - P0-2: prune ran delete_unverified=True guarded only by an in-process asyncio lock, but the runbook promises `cascade sync` is safe alongside a live server — and the CLI's first optimize beat does prune, in a separate process. It could delete files the daemon is mid-commit on. Fix: switch prune to delete_unverified=False. Measured to reclaim identically on churned tables (both collapse superseded versions ~97%); True only additionally deletes in-flight/dangling files — exactly the corruption vector. No cross-process lock needed; the write-lock/commit fix (the real reclaim win) is unchanged. - P1-3: /health called orch.health() (6 SQLite aggregates) with no guard → a locked/full/migrating DB would 500 the liveness probe and restart the container. Wrap it: unhealthy readiness + reason, HTTP stays 200. - P1-4: rebuild drops + recreates tables; a live daemon holds cached handles pointing at the dropped dataset. Runbook now says stop the server first — the one cascade command unsafe alongside a live server. - P2: narrow _is_benign_commit_conflict to the "commit conflict" phrase (a bare "retryable" swallowed unrelated recoverable errors); add a timeout around the prune cleanup so a hung lance call can't wedge the write lock; correct two stale docstrings. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: fix schema-recovery guidance + drop dead internal doc refs Follow-up to the review; two doc issues neither the review nor the fix commit caught: - The schema-drift startup error's docstrings (verify_business_schemas and LanceDBLifespanProvider) still described the recovery as `rm -rf ~/.everos/.index/lancedb` — which the runbook explicitly calls the WRONG recovery (it leaves the cascade queue `done`, so nothing re-indexes and the index comes back empty). The raised error already points to `everos cascade rebuild`; align the docstrings to match. - 15 dangling references to an internal numbered design-doc set (12_/13_/16_/17_*.md) that was never shipped to this repo. Point the schema-recovery ones at docs/cascade_runbook.md; drop the rest (pure provenance in table/component docstrings) while keeping the substance. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: align maintenance docstrings with split optimize/prune API Two docstring residuals from the #379 review's P2 list: - _run_optimize_once still described the pre-split bundled heavy beat ("same work plus cleanup_older_than ... older than one cadence"); the heavy beat now calls prune() under the write lock and the retention window is decoupled from the cadence (DEFAULT_OPTIMIZE_PRUNE_RETENTION_SECONDS). - _restore_shaper_metadata converts any numeric timestamp, wider than an exact inverse of the bridge; document that this is deliberate (the shaper contract requires a datetime either way). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * style(e2e): hoist fixture-body imports to conftest module top shutil / importlib.resources.files were imported inside the core_pipeline_runtime fixture body; move them to the module top to match the repo import style (#379 review P2). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(cascade): back off a hung prune via a separate attempt clock (N1) The write-lock timeout on prune (review P2) didn't achieve its goal: it was 300s — equal to the prune cadence — and `last_prune_at` only advanced on success. So a hung lance cleanup timed out after 300s, `should_prune` was still true (clock never moved), and the next beat re-pruned ~10s later — pinning the per-table write lock ~97% of the time, the exact write-starvation the timeout was meant to prevent. A real cleanup is milliseconds even on a heavily churned table (measured ~40ms at 320k writes / 100 versions), so the timeout is a pure hang-catcher: lower it to 60s (~1500x headroom, never fires normally, well below the 300s cadence). Split the prune clock so a failed prune backs off without masking the health signal: - last_prune_attempt_at (new) gates scheduling, advanced before the call whether it succeeds or times out → a hung prune waits a full cadence before retrying (light lock-free compaction runs meanwhile), so the lock is held at most ~timeout/cadence ≈ 17% in the worst case. - last_prune_at advances only on success and still drives the prune-staleness health signal, so a persistently failing prune surfaces as degraded instead of being hidden by an advanced schedule clock. Regression test: a raising prune advances the attempt clock (next beat is light, no immediate re-prune) but not the success clock. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: zhanghui <zhanghui@shanda.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
6d62ecbd6f |
chore(deps): bump everalgo user-memory and agent-memory to 0.4.0 (#377)
Both were pinned to 0.3.x; PyPI 0.4.0 published 2026-07-30. Local make ci is green (lint + 180 integration tests + package smoke). Held on a branch until the next EverOS release picks it up. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
4c2ae841d7 |
docs(security): refresh supported versions and link published advisories (#375)
The supported-versions section still said "released and at v1.0.0" and
listed all of `1.x` as supported, which had drifted two minor releases.
It is now stated as a line ("1.2.x (current)") rather than a pinned
patch number, so it does not go stale on the next release.
Also:
- Point readers at /security/advisories, and tell them to read the
affected range on the advisory rather than comparing version numbers.
- Add a threat-model bullet for ingested documents. The loopback-only
default covers who can call the API, but filenames, metadata, and
content arriving from elsewhere are untrusted regardless of how the
API is reached.
- Note that reporters are credited in the advisory as well as the
release notes, which is what we do in practice.
Reporting stays email-only; GitHub private vulnerability reporting is
deliberately left disabled so the documented 5-business-day response
runs through one monitored inbox.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
|
27ac6cd181 |
docs(changelog): state the 1.2.0 regression and CWE-22 affected range (#372)
The 1.1.4 section now describes the code shipped as everos==1.1.4 on PyPI, which lists the path traversal as fixed. Read top-down, that left a 1.2.0 user concluding the fix predates their version and they are safe — the opposite of the truth, since 1.2.0 was built from a branch that never received those fixes. Three changes, no history rewritten: - The 1.2.1 Fixed entry now names what 1.2.0 regressed, instead of describing the merge as internal branch hygiene. - A Security section under 1.2.1 carries the affected-version range, which is discontinuous: everything before 1.1.4, plus 1.2.0. 1.1.4 itself is not affected. - A note under 1.2.0 points forward, so a reader who looks up their own version rather than the newest one still finds it. Matches the published v1.2.1 release notes. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Kendrick-Song <61358070+Kendrick-Song@users.noreply.github.com> |
||
|
|
e723a4eb1c |
feat(examples): zero-install Langfuse replay + a demo memory worth searching (#374)
* feat(examples): add zero-install Langfuse trace replay Native OTel moved span emission into the server, so the Langfuse example lost its try-before-install path: seeing anything now required a configured EverOS. Restore one without fabricating spans. replay.py pushes a recording of a real EverOS run into the reader's own Langfuse project. Names, attributes, token usage, structure and durations are replayed verbatim; only ids, timestamps and a `replay` tag are rewritten, so nothing in the trace is invented. It needs the OTel SDK and Langfuse keys, nothing else. record_trace.py is the maintainer tool that produced the recording. It stands in for Langfuse's OTLP and scores endpoints on localhost, which works because EverOS derives both from langfuse_host, so one sink captures both signals straight from a real server run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UyKinsWs1MgoARPoB9R4NW * feat(examples): give the Langfuse demo a memory worth searching The demo ingested one conversation and searched it, so recall had nothing to choose between and the traces showed plumbing rather than behaviour. Eleven short conversations now span ten weeks, each on its own topic, so a question has to find the right memory in a populated store. Two revisit the same subject five days apart, close enough for geometry clustering to group them, which finally gives reflection something to consolidate: the demo nudges reflect_episodes (a `0 2 * * 1` cron otherwise), waits for the merge to land, and the superseded memory is gone from search by the time the questions are asked. One question asks about something never discussed, so a miss looks like a miss. KEYWORD is no longer a demonstrated method. Its top score is raw BM25, on a different scale from the calibrated ones, so showing the three side by side invited a comparison that means nothing. Readiness is polled per session rather than slept through, since a fixed sleep searched a half-built index and reported scores lower than the memory deserved. Polling is deliberately slack: every probe is itself a traced search, and a tight loop buried the real questions under a wall of readiness checks. recorded_trace.json is that run against 1.2.1: 237 spans over 60 traces, no errors, no secrets, synthetic content throughout. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UyKinsWs1MgoARPoB9R4NW --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
4256419595 |
chore(release): v1.2.1 (#371)
Bump version to 1.2.1 and finalize CHANGELOG. Highlights: [embedding] and [rerank] become soft runtime dependencies; new everos cascade backfill CLI; LanceDB schema v2 (nullable vector); PyPI Trusted Publishing workflow; 1.1.4 backport fixes. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>v1.2.1 |
||
|
|
b1441da607 |
refactor(config): make [embedding] and [rerank] soft dependencies (#361)
* refactor(config): make [embedding] and [rerank] soft dependencies
Make [embedding] and [rerank] soft runtime dependencies so a
freshly-onboarded user can run EverOS end-to-end with only [llm]
configured. Previously the server refused to start without embedding,
locking out anyone who just wanted keyword-only search.
## Capability tiers
- Tier 1 ([llm] only) KEYWORD search, add/flush, md
writes, cascade sync
- Tier 2 ([llm] + [embedding]) + VECTOR / HYBRID search,
reflection, skill extraction,
backfill
- Tier 3 ([llm] + [embedding] + rerank) + AGENTIC search, knowledge
Tier upgrades require a server restart (capability accessors cache
for the process lifetime). Tier downgrades are read-safe: a Tier-3
user who drops [rerank] can still read/rename/delete existing
knowledge documents; only write/search endpoints return 422.
## What changed
- Component accessors — component/{embedding,rerank,llm}/accessor.py
are the single process-wide provider singletons. service/* never
maintains parallel singletons; it consumes get_embedding_capability()
/ get_rerank_capability() / get_llm_client() directly. Build-time
ValueError from the factory is logged as capability_build_failed
(was silently swallowed).
- Error mapping — ProviderNotConfiguredError -> 422 with everos.toml
section hints (never EVEROS_* env-var strings).
LanceDBMigrationError fails loud with escalating recovery guidance
(restart -> wipe index). LLMNotConfiguredError in search maps to
None for KEYWORD degradation.
- Nullable-vector LanceDB migration — schema v2 makes the vector
column nullable so Tier-1 rows can land without embeddings.
Migration is guarded by a cross-process memory_root_lock
(fcntl.flock + anyio.to_thread) and runs optimize() per table
after Phase-1 backfill to reclaim manifest bloat.
- Cascade — knowledge handlers register unconditionally (Tier-3 ->
Tier-2/1 downgrade no longer strands DELETE); embed-requiring
strategies use body-guards that check capability.available at
execution time. _TABLE_SPECS has an import-time drift assertion
against BUSINESS_SCHEMAS_WITH_VECTOR.
- `everos cascade backfill` CLI — Phase-1 (embed missing vectors) /
Phase-2 (emit synthetic events for cascaded processing) / Phase-3
(sync new skill files). Exit codes: 0 / 1 / 2 / 3 (server running
preflight) / 4 (COMPLETED_WITH_FAILURES — per-row failures rolled
up) / 130 (SIGINT). OMEConfig.crash_recovery_enabled=False in
backfill engines prevents stale-RUNNING rows re-enqueuing into a
smaller strategy registry.
- /health — reports capabilities + disabled_features per tier so ops
can distinguish "boots but degraded" from "boots and full".
- Presentation split — memory / service / infra never import typer /
click. TyperPresenter Protocol + run_backfill() live in
entrypoints/cli/commands/_backfill_cmd.py. Enforced by
import-linter.
- Startup hint — unconditional count_rows(filter="vector IS NULL")
sweep emits unbackfilled_memory_rows (event name + hint text
pinned) when Tier-1 rows exist. ParserLifespanProvider warms the
everalgo.parser import at boot so /health doesn't block on first
call.
- Knowledge upload UTF-8 short-circuit — _looks_like_utf8_text()
routes text/* mime and known plaintext extensions (md/txt/rst)
straight to UTF-8 decode instead of the parser. Prevents 503
Multimodal-not-configured when Tier 3 sans [multimodal] uploads a
markdown doc.
## Sync history with main (2 merges collapsed into this squash)
Merged origin/main at
|
||
|
|
649046b0df |
docs(api): use /api/v2 in docs and examples, demote v1 to legacy (#370)
1.2.0 introduced /api/v2 as the canonical, cloud-aligned prefix and mounted every business router twice, but the user-facing entry points (README, README.zh-CN, QUICKSTART, the docs/ set, the Langfuse example) still taught /api/v1 — so new users were pointed at the compatibility alias while docs/api.md already declared v2 canonical. - Switch every EverOS endpoint reference in docs, examples, and `everos demo --live` to /api/v2, plus the matching CLI test expectations. - Describe /api/v1 as a legacy compatibility alias that may be removed in a future major release, rather than a permanent one. Nothing changes at runtime: both prefixes still resolve to the same handlers and the v1/v2 parity test is untouched. - Add a short note in README / README.zh-CN / QUICKSTART so existing v1 integrations know they keep working. - Fix the five dead endpoint anchors in the docs/api.md table of contents, which still pointed at the pre-1.2.0 #post-apiv1... slugs. Left on v1 deliberately: docs/migration-to-1.0.0.md (historical record), CHANGELOG history, tests/** (v1 must stay covered), and the use-cases/claude-code-plugin + openher READMEs, which document a different cloud API. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
6c9792ed11 |
test(conftest): pin EVEROS_ROOT so tests ignore the developer's config (#369)
Settings resolves its TOML source through resolve_root(), so any field a test does not pass explicitly was filled from the real everos.toml on the machine running the suite. Enabling [observability] locally, for instance, made test_returns_singleton_when_configured fail, because the LLM client picks up the usage-recording wrapper when tracing is on: green in CI, red on that developer's machine, and unrelated to whatever they were changing. An autouse fixture now points EVEROS_ROOT at a per-test tmp dir. Tests that exercise root resolution itself already setenv / delenv inside the test body, which runs after the fixture, so none of them needed changing. Claude-Session: https://claude.ai/code/session_01UyKinsWs1MgoARPoB9R4NW Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: zhanghui <huizhang1995@gmail.com> |
||
|
|
4e13f7881e |
fix(observability): separate uncalibrated recall scores by name (#368)
Langfuse aggregates scores by name, so one name may only carry values on
one scale. recall_top_score was emitted for every method, mixing HYBRID's
LR-sigmoid probability and AGENTIC's cross-encoder score (both comparable
in [0, 1]) with KEYWORD's unbounded BM25 and single-route VECTOR's cosine.
A chart on that name averaged the two scales, and in practice a keyword
score can read numerically higher than a calibrated one while meaning less.
Uncalibrated methods now report recall_top_score_raw, leaving
recall_top_score comparable across methods and over time. Every recall
score also carries metadata = {method, calibrated}: a structured field
Langfuse persists and can split on, which the free-text comment could not
serve. The comment stays for reading individual scores.
Breaking for anyone charting recall_top_score for keyword search; 1.2.0 is
four days old, so this is the cheapest moment to correct the naming.
Claude-Session: https://claude.ai/code/session_01UyKinsWs1MgoARPoB9R4NW
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
|
bebb448c89 |
ci(release): add PyPI Trusted Publishing workflow + /release skill (#358)
GitHub had no publish path — the package was previously released from the (now-archived) GitLab mirror. Bring publishing to the public repo: - .github/workflows/release.yml: on a vX.Y.Z tag, build + smoke-test (make package) and upload to PyPI via Trusted Publishing (OIDC, no stored token), gated behind the `release` environment for manual approval. A guard step refuses to publish when the tag != pyproject version. - .claude/skills/release: /release documents the cut (bump version → CHANGELOG → tag → approve → verify) and the one-time PyPI trusted-publisher + GitHub environment setup. Requires two owner-only one-time steps before the first release (documented in the workflow header and the skill): register the GitHub trusted publisher on PyPI, and create the `release` environment with required reviewers. Co-authored-by: zhanghui <zhanghui@shanda.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
42629dfd4d |
fix: backfill missing 1.1.4 fixes into main (#366)
* fix(knowledge): contain original-file write path (CWE-22) The multipart upload filename was joined into ``_original/`` verbatim. An attacker-controlled filename such as ``/tmp/pwned`` or ``../../.bashrc`` would let ``POST /knowledge/documents`` write outside the document directory (the ``/`` operator discards the left operand for absolute paths; ``..`` walks upward). The read side had the symmetric issue. Fix, mirroring the sender_id containment shipped in 1.0.1 (GHSA-c795-2g9c-j48m): - Add ``_safe_original_filename`` reducing the untrusted filename to a single POSIX/Windows-basename component; reject degenerate residuals (``""``, ``"."``, ``".."``) with PathTraversalError. - ``_write_original_file`` asserts ``target.resolve()`` stays inside ``original_dir.resolve()`` before any filesystem touch (mkdir/write). - ``_resolve_original_file_path`` sanitises symmetrically so a stored provenance label can never resolve to an out-of-directory file. Four SEC regression tests cover: absolute filename, ``..`` traversal, degenerate filename rejection, and read-side sanitisation. Backport from GitLab release/v1.1.4 (commit 40f19de) — 1.1.4 shipped this fix; the GitLab -> GitHub sync stopped at 1.1.3, so 1.2.0 regressed the containment. This commit alone restores the fix; the 1.2.1 release PR ships it to PyPI. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(cascade): retry classification, budget, and reconcile races Backport the cascade reliability work that shipped in GitLab 1.1.4 (MR !49 / commit 95db2f5) — six interlocking changes the reviewer should read in order: 1. Worker retry classification uses ExternalServiceError (embedding / LLM / rerank transient failures) as the "retry inline" signal. The legacy RecoverableError hierarchy under cascade/errors.py is removed; the retry contract now lives in the domain error tree (core/errors.py). Docstrings in handlers/base.py and sqlite/tables/md_change_state.py updated to match. 2. Cross-cycle retry budget: _MAX_TOTAL_RETRIES = 12. Once total attempts across scanner cycles exhaust the budget, the worker marks retryable=False in place instead of looping forever on a sustained upstream outage. 3. md_change_state upsert preserves retry_count on scanner re-enqueue when mtime is unchanged (previously reset to 0 every sweep, defeating the budget). mtime change (user edit) still resets the counter. 4. Reconciler no longer re-enqueues pending / processing rows on stable mtime — that was overwriting the worker's mark_done. It also skips failed rows with retryable=False on stable mtime so the entry-check demote path is stable. 5. mtime tolerance (10 ms, MTIME_TOLERANCE_SECONDS) absorbs the SQLite REAL float precision loss that previously flapped the reconcile decision when the same md was rewritten without a real content change. The constant is defined once in the sqlite repo and imported by the reconciler so both sides use the same tol. 6. Worker _run_rebuild_once carries an explicit `state.task is not asyncio.current_task()` guard before awaiting the optimize task — the previous contextlib.suppress was silently swallowing self-await RuntimeError. Kept intact from the GitHub 1.2.0 baseline: - The `except FileNotFoundError → handle_deleted` branch in the worker (delete/modify race — see test_modified_event_for_vanished_file_is_processed_as_delete). Test coverage added: - test_retry_budget_exhausted_marks_unrecoverable - test_external_service_error_at_budget_edge_demotes_in_place - test_upsert_preserves_retry_count_for_failed_stable_mtime - test_upsert_resets_retry_count_on_mtime_change - test_optimize_fallback_rebuild_on_sustained_failure - reconciler mtime-tolerance / stable-mtime skip suite Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(embedding): raise on empty API data; forward MRL dimensions Backport from GitLab 1.1.4 (MR !49 / commit 95db2f5). Two behavioural changes on ``OpenAIEmbeddingProvider._embed_chunk``: 1. ``response.data == []`` now raises ``EmbeddingServiceError`` instead of returning an empty list. Some upstream providers (observed on DeepInfra under load) return HTTP 200 with an empty ``data`` array; the silent zero-vector path was corrupting search indexes without any signal. 2. When ``[embedding] dimensions = N`` is set in ``everos.toml``, the parameter is forwarded to the API so MRL-capable models (OpenAI text-embedding-3-*, Qwen3-Embedding, ...) do server-side truncation with proper re-normalization. Client-side truncation to ``dim`` remains as a fallback for backends that ignore the param. ``openai.NOT_GIVEN`` is used as the sentinel so the request omits the field when the setting is left at the default ``None``. Config plumbing: - ``EmbeddingSettings.dimensions: int | None = None`` - factory forwards ``dimensions=settings.dimensions`` to the provider The provider stays inside the existing ``memory_span`` OTel wrapper and continues to report input-only tokens via ``set_generation_usage`` - both are GitHub 1.2.0 native tracing behaviours preserved intact. Test coverage: - test_empty_response_data_raises_embedding_error (new) - test_usage_span._FakeEmbeddings.create signature updated to accept ``dimensions`` kwarg so the OTel token-recording tests still exercise the same call path Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(extract): retry episode extraction on malformed LLM output Backport from GitLab 1.1.4 (MR !49 / commit 95db2f5). The ``/flush`` synchronous path called ``EpisodeExtractor.aextract`` exactly once. everalgo raises ``ValueError`` when the LLM returns malformed JSON (observed with OpenRouter partial responses where finish_reason=stop but the body is truncated) — the caller was surfaced a 500 for a transient upstream hiccup. ``_extract_with_retry`` wraps the call with two extra attempts at 1s and 2s backoff (final attempt propagates untouched), typed as ``AlgoEpisode`` so the caller path stays annotated. Retry stays inside the existing GitHub 1.2.0 ``memory_span("everos.extract", ...)`` OTel wrapper — the OTel token capture and the retry loop are orthogonal. TODO in the code notes we should catch a typed everalgo ``ExtractionError`` once that type is introduced (currently ValueError is broad). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: log non-stop finish_reason; bump everalgo-user-memory 0.3.2 Two loosely coupled changes from GitLab 1.1.4 (MR !49 / commit 95db2f5) that arrive together as a housekeeping commit. 1. ``_LoggingLLMClient`` diagnostic wrapper (new, file-private). Wraps the raw everalgo LLM client and, on every ``chat()``, warns when ``resp.finish_reason != "stop"`` — logging the reason, ``content_len``, the last 200 chars of ``content``, and ``model``. Aims at OpenRouter/DeepSeek truncation triage where the provider silently caps output length and returns finish_reason=length / filter / etc. Non-invasive: one branch per call, no config gate. Wrapper stack in ``get_llm_client``: LoggingLLMClient(UsageRecordingClient(build_client(...))) LoggingLLMClient(build_client(...)) # observability off ``UsageRecordingClient`` (GitHub 1.2.0 native OTel token capture) stays gated by ``settings.observability.enabled`` — this commit preserves that. LoggingLLMClient is always outermost so the reason it observes is exactly the reason the underlying provider reported. 2. ``everalgo-user-memory`` 0.3.1 -> 0.3.2 (pyproject + uv.lock). Same bump the GitLab 1.1.4 release lane took; unblocks the episode-extract retry work in commit 4 seeing the upstream improvements. Verified via ``uv sync``. No functional API changes. Test coverage: - test_returns_singleton_when_configured now asserts the outer LoggingLLMClient wrapper. - test_wraps_client_when_observability_enabled asserts the two-layer Logging(UsageRecording(...)) stack. - test_does_not_wrap_client_when_observability_disabled asserts Logging still wraps when tracing is off. - test_logging_wrapper_warns_on_non_stop_finish_reason (new). - test_logging_wrapper_silent_on_stop_finish_reason (new). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(changelog): restore [1.1.4] section to match published sdist The 1.1.4 changelog entry on this branch previously listed three items (Langfuse example, delete/modify race, live-server telemetry). The 1.1.4 sdist on PyPI, however, was built from the internal release lane and includes the CWE-22 containment, the cascade retry-budget / mtime-tolerance / reconcile-guard work, the embedding empty-data raise, the episode-extract retry, MRL dimensions, and the LLM finish_reason diagnostic — none of which were represented here when the tag was cut. Rewrite the [1.1.4] section so it matches the wheel a user actually installs from PyPI: - Add a header note explaining the retroactive restoration. - Fixed: CWE-22, cascade reliability bundle, delete/modify race (unchanged wording), embedding empty-data, episode extract retry, Langfuse live-server telemetry (unchanged wording). - Added: MRL dimensions, LLM finish_reason diagnostic, Langfuse example (unchanged wording). - Changed: everalgo-user-memory 0.3.1 -> 0.3.2. The GitLab-side `.gitlab-ci.yml` in-house-runner entry is dropped — open-source CI runs on GitHub Actions and the internal runner switch is not visible to public users. Date stays 2026-07-20 (the GitHub v1.1.4 tag date / PyPI upload timestamp) rather than the internal 2026-07-23 code-freeze date, so the timeline of what shipped where remains internally consistent. The corresponding code fixes are all backported by earlier commits in this PR; this commit only aligns the changelog surface. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
6dcd3ebda2 | docs(readme): remove Star us section (#363) | ||
|
|
64e0fdc9bb |
Merge pull request #360 from EverMind-AI/chore/release-1.2.0
chore(release): update EverOS to 1.2.0v1.2.0 |
||
|
|
e8eda1290c | Merge branch 'main' into chore/release-1.2.0 | ||
|
|
b54e19720a |
Merge pull request #359 from EverMind-AI/ci/docs-run-on-all-prs
chore: native Langfuse example + docs-check on every PR |
||
|
|
af0842998a | Merge branch 'main' into ci/docs-run-on-all-prs | ||
|
|
7179d62293 |
chore(release): update EverOS to 1.2.0
Minor release: `/api/v2` API prefix (v1 retained as alias) and native OpenTelemetry tracing — both back-compatible, so 1.1.4 -> 1.2.0. - pyproject: version 1.1.4 -> 1.2.0 - CHANGELOG: promote [Unreleased] to [1.2.0] - docs/openapi.json + uv.lock: regenerated for the new version Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
869dc67804 |
docs(examples): replace Langfuse wrapper with native tracing example
The examples/langfuse wrapper (everos_langfuse.py) was the interim client-side instrumentation before EverOS gained native OpenTelemetry export. Now that [observability] emits real OTLP spans, the wrapper is redundant and its faked child spans could mislead. Replace it with a minimal, dependency-light example: enable [observability] in everos.toml, run the server, and drive one add/flush/search cycle (demo.py, stdlib only) to see native traces in Langfuse. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
a1788468a2 |
Merge pull request #353 from EverMind-AI/fix/otel-review-followups
fix(observability): address OTel review findings |
||
|
|
e85db903a2 |
ci(docs): run docs check on every PR so its required check reports
The Docs workflow (`links` job) is a required status check, but its trigger was `paths:`-filtered to markdown/docs files. A PR touching only code never triggered it, so the required check never reported and the merge box stayed BLOCKED forever waiting for a status that would never arrive. Drop the `paths:` filter: `make docs-check` validates the whole doc tree independent of the PR diff and runs in seconds, so it is cheap to report on every PR. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
b844665da1 |
docs(changelog): record native OpenTelemetry tracing under Unreleased
Summarizes the OTel observability feature (instrumentation + the review / telemetry-audit fixes) as one user-facing Added entry; the fixes themselves targeted an unreleased feature so they need no separate lines. |
||
|
|
25964ccbb3 |
fix(observability): don't spawn a root trace per cascade embedding
The embedding-observation change wrapped every _embed_chunk call in a span. Cascade-time indexing embeds run outside any request trace, so each chunk started its OWN root trace — a per-chunk trace explosion (13 orphan everos.embedding traces per add/flush), detached from session/user and contrary to the "cascade is not instrumented" decision. memory_span gains nested_only: open a span only when one is already active. Embedding uses it, so search/flush embeds still nest under their recall/extract span, while cascade embeds no-op (no trace) — restoring the cascade-untraced boundary. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
f20a123805 |
fix(observability): correct span typing and telemetry shape
Follow-up to the live-trace audit — all on the enabled path: - boundary detection LLM (everalgo detect_boundaries) ran with the SPAN-typed request root as the current span, so its ~1.2k tokens were dropped from cost. Wrap it in an everos.memcell.boundary GENERATION span so Langfuse prices it. - embedding calls stamped usage on the enclosing retriever span. Wrap each /embeddings call in an everos.embedding EMBEDDING span so the type is correct and pricing can apply. - agentic recall emitted a duplicate, same-name everos.search.recall (cluster_scoped wrapping hybrid_full, which owns the real recall span). Drop the redundant outer span; hybrid_full keeps the one recall span (also used standalone in round 2). - search now captures the returned hit ids (episodes/cases/skills) as observation output when capture_content is on — previously only the query input was captured. - add/flush spans carry request_id in metadata. - persist captures the memory-root-relative .md path, not the host absolute path (no host layout leak to the telemetry backend). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
0cbfb9f854 |
fix(observability): address OTel review findings
Adversarial review of the merged OTel instrumentation (#352) surfaced four issues, all on the enabled path (default-off, so no impact until tracing is turned on): - search LLM client was unwrapped, so hybrid/agentic token usage — the heaviest LLM spend — never reached Langfuse. Wrap it with UsageRecordingClient when observability is enabled, mirroring get_llm_client(); graceful keyword-only degradation is preserved. - set_generation_usage overwrote token counts, undercounting any span that wraps more than one chat call (the now-wrapped agentic path). Accumulate instead of replacing. - recall_hit was emitted for uncalibrated methods (unbounded BM25 / single-route vector), a near-constant always-hit signal that inflates dashboards. Gate hit on calibrated methods (HYBRID/AGENTIC); keyword and vector emit only the raw top_score. - init_tracing / init_score_sink were not idempotent — a re-init without an intervening shutdown orphaned the export thread + OTLP socket + worker task. Tear down the previous instance first (init_score_sink is now async). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
a59d385d23 |
Merge pull request #356 from EverMind-AI/feat/api-v2-alias
feat(api): serve endpoints under /api/v2, retain /api/v1 as alias |
||
|
|
21b845bf88 |
feat(api): serve endpoints under /api/v2, retain /api/v1 as alias
Every business endpoint (memory/*, ome/*, knowledge/*) is now served under /api/v2, aligning the open-source API with the EverOS Cloud contract. /api/v1 is retained as a permanent, backward-compatible alias: the same router objects are mounted under both prefixes, so both resolve to identical handlers and request/response contracts. Existing /api/v1 integrations keep working unchanged. Infra endpoints (/health, /metrics) stay unversioned. Fix the Prometheus request-metric label to build the path from the full request URL (with path params folded) rather than the route's router-relative path, so the version prefix is preserved and v1/v2 traffic stays distinguishable. Docs (docs/api.md, docs/openapi.json), CHANGELOG, and route docstrings updated to lead with /api/v2. Add test_api_versioning as the parity guard: every v2 route has an identical v1 twin and vice versa. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
c4c2977898 |
Merge pull request #352 from EverMind-AI/feat/otel-instrumentation
feat: native OpenTelemetry instrumentation (optional, off by default) |
||
|
|
357f619c64 |
feat(observability): instrument memory ops + OME trace linking
Open spans at the memory hot paths (all no-op when tracing is off): - add / flush (service.memorize), extract + persist.markdown (user pipeline). - search: everos.memory.search retriever + a uniform recall / rank decomposition across keyword / vector / hybrid / agentic (manager, agentic modules, cross-encoder callbacks); query-embedding tokens land on recall. - recall quality: top_score / hit on the search span, plus recall_top_score / recall_hit pushed to Langfuse scores via the bounded-queue sink (method tagged; off the request path). - OME: everos.ome.<strategy> agent span + everos.reflect.consolidate generation; a W3C traceparent captured at enqueue is threaded through the APScheduler job and re-attached in the Runner, so strategies fanned out from a request nest under that request's trace. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
bd26f5a80f |
feat(observability): capture LLM + embedding token usage
Surface gen_ai.* model + token attributes onto the active span so Langfuse can compute cost — without touching everalgo: - UsageRecordingClient wraps the LLM client and records response.usage after each chat(); get_llm_client composes it over the existing _LoggingLLMClient only when observability is enabled (disabled default stays overhead-free). - OpenAIEmbeddingProvider records its response.usage (input tokens) onto the active span too. Tokens land on the everos.extract / everos.reflect.consolidate generation spans and the search embedding recall; no-op when tracing is off. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
ab5cf0447e |
feat(observability): OTel tracing chassis + request-id propagation
Optional, off-by-default native OpenTelemetry export (the [otel] extra), wired as chassis so call sites never branch on config: - ObservabilitySettings ([observability]) — enabled / endpoint / headers / sample_rate / capture_content / langfuse_* / recall_hit_threshold; langfuse creds derive the OTLP endpoint + Basic-auth header. - core.observability.tracing: TracerProvider lifecycle (module-local, no-op when off/absent), memory_span helper stamping the langfuse.* contract, set_generation_usage, privacy-gated capture_input/output (redaction + truncation), non-blocking recall-score sink, W3C traceparent in/out helpers. - TracingLifespanProvider builds provider + score sink at startup. - RequestIdMiddleware: per-request id (state + contextvar + structlog + X-Request-Id) and continues an upstream traceparent when present; managers read the propagated id via resolve_request_id. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
02dae05cbc | chore(release): update EverOS to 1.1.4 (#348) v1.1.4 | ||
|
|
d3a9f9e394 |
docs(examples): make Langfuse wrapper degrade cleanly on a real server (#342)
* docs(examples): make Langfuse wrapper degrade cleanly on a real server The wrapper synthesized child spans (extraction, embedding, hybrid recall, rerank, index sync, consolidation) from a mock-only `_detail` field. Against a real EverOS server that field is absent, so those spans rendered with placeholder data — hardcoded model names, token=0, fixed sleep durations — and recall scores fell to 0. Now the per-stage child spans are emitted only when `_detail` is present (the mock, or future native in-core instrumentation). Against a live server only the top-level span per operation is emitted, with real latency and output — no fabricated data. Recall quality (recall_top_score / recall_hit) is derived from the real search response, which already carries a per-hit score, so it works against a live server today, not just the mock. Verified: mock path unchanged (full trace tree, real scores); real-ish path (no `_detail`) emits only top-level spans plus a real recall score. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(examples): count empty recalls as a miss in Langfuse hit-rate When a search returns nothing scored, record recall_hit=0 (span attribute + Langfuse score) instead of omitting it, so genuine empty recalls still show up in recall hit-rate. No top_score is emitted (there is no hit to score). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
a1e21ca676 |
docs(examples): add Langfuse (OpenTelemetry) integration example (#339)
Adds examples/langfuse/ — a thin OpenTelemetry wrapper that traces EverOS memory operations (add / flush+extract / search / reflection) into Langfuse, with recall quality pushed as Langfuse scores. Pure OTel SDK, no Langfuse package dependency; runs against a built-in mock or a real EverOS server (EVEROS_BASE_URL). Additive only, no changes to EverOS core. Referenced by the upcoming Langfuse docs integration cookbook. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
45656d331e |
fix(lancedb): fix FTS with_position optimize crash + disk bloat (#336)
FTS indexes built with with_position=True crash lance's optimize/compaction on lancedb >= 0.32 when merging an unindexed tail (Max offset exceeds length of values; upstream lance-format/lance#7653). The crash aborts optimize() including version cleanup, so the index dir grows unbounded until the disk fills. everos recall is OR-mode BM25 and never does phrase queries, so positions are never read -- disabling is lossless. - base: default with_position=False - infra: migrate_fts_indexes() rebuilds pre-fix indexes once at startup + reclaims orphans - cascade worker: count consecutive optimize failures, escalate warning->error Fixes #335. Co-authored-by: zhanghui <zhanghui@shanda.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>v1.1.3 |
||
|
|
56ee9c8e5c |
fix(search): scope deprecated_by filter to user tables only (#330)
compile_filters() unconditionally appended 'deprecated_by IS NULL' to every query, but the deprecated_by column only exists on user-scoped tables (episode, atomic_fact — Reflection V1). Agent tables (agent_case, agent_skill) lack this column, causing a SQL error on agent search/get queries. Gate the clause behind owner_type == 'user' so agent queries no longer reference a non-existent column. Bump version to 1.1.2. Co-authored-by: Jiayao Song <jiayao.song@shanda.com>v1.1.2 |
||
|
|
15efd1198c | chore(release): update EverOS to 1.1.1 (#327) v1.1.1 | ||
|
|
66a201e164 | docs(readme): add raven ecosystem entry (#322) | ||
|
|
0341f1230f |
docs: align config and github workflow (#314)
* docs: align config and positioning copy * docs: remove internal branch workflow residue * docs: record github sync guard * chore: retrigger commit lint * docs: leave readme files unchanged * docs: remove public positioning comparison |
||
|
|
b7d15f7252 |
docs: re-link orphaned docs and slim engineering.md (#313)
* docs: re-link orphaned docs and slim engineering.md After #311 realigned the docs, index.md again omitted four files that exist under docs/: everos-demo, use-cases, migration-to-1.0.0, and release-notes-1.1.0. Restore them so every docs/*.md is reachable from the index, and rewrite engineering.md for an external audience. index.md: - Re-add a Tutorials section: everos-demo, use-cases - See also: + release-notes-1.1.0, + migration-to-1.0.0 - Reframe the Engineering section as contributor-facing (not "internal") engineering.md (575 -> 113 lines): - Drop internal-only material: the self-justifying scope rationale, the Claude Code loading internals, the infra failure-impact table, the roadmap, and the "investing in infrastructure" essay - Fix claims that were false for this GitHub repo: GitLab-primary CI, the dev/master branch model, and the Gitmoji commit convention - Keep what helps a contributor: toolchain, local make targets, the CI gates, and the main-branch + Conventional Commits workflow Conventions now match the repo's own .gitlint and GitHub Actions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: align engineering.md references with the slimmed doc Update the inbound descriptions of engineering.md now that it is a contributor reference rather than an "infrastructure overview", and repoint the one reference that named content the trim removed. - CLAUDE.md, README, README.zh-CN, architecture.md: reword the link text to "contributor engineering reference: build, test, CI, conventions" - CLAUDE.md: the GitFlow Lite rationale pointer now targets .claude/skills/new-branch/SKILL.md (engineering.md no longer carries it) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
bdd702c43b | docs(readme): restore pre-trim overview (#312) | ||
|
|
bf9b8d1053 |
docs: align docs with v1.1 and trim README (#311)
Co-authored-by: Jiayao Song <jiayao.song@shanda.com> |
||
|
|
e14e9a59fa |
docs: index all docs and add a Tutorials section (#310)
index.md only linked 12 of the 19 docs under docs/. Readers entering from the index missed configuration, multimodal, demo, use-cases, benchmark, migration, and release notes. - Add a Tutorials section (Diátaxis 4th quadrant): everos-demo, use-cases - Reference: + configuration, + multimodal - How-to: + locomo_benchmark, + migration-to-1.0.0 - See also: + release-notes-1.1.0 Every docs/*.md (excluding the openapi.json artifact) is now linked. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
9328442b5c | docs(readme): sync chinese 1.1 feature positioning (#309) | ||
|
|
289e78b11e | docs(readme): promote 1.1 knowledge and reflection (#308) |