* fix(lancedb): put table-handle resolution inside the deadline
run9 reproduced the stall the previous commits were supposed to close, on a
different table: episode went 13 minutes without a prune — versions climbing
63→66 while foresight and atomic_fact both collapsed to 1 — with **zero**
failure, timeout, or conflict logs. Its last successful prune was logged at
11:41:59 and the staleness clock matched to the second.
The deadline covered the critical section but not the await ahead of it:
`table = await self._table()` sat outside `_locked`, so a hang while resolving
the table handle never returned. The scheduler runs one maintenance task per
kind and skips a kind whose task is still in flight, so that kind stops being
maintained permanently, silently, because nothing failed.
Move the handle resolution inside the deadline for all seven locked operations,
and give the lock-free compaction beat its own `_deadline` (it takes no lock so
it cannot block writers, but it can still park a kind by never returning).
Belt and braces in the scheduler: both beats now run under
`_MAINTENANCE_TASK_TIMEOUT_SECONDS`, a last-resort bound on the whole call, so
any await I have not thought of costs one cadence rather than forever.
Regression test: a repo whose `_table_lookup` never resolves must make prune,
optimize and add all raise `VectorStoreBusyError` and leave the lock free.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore(release): v1.2.2
Bump to 1.2.2 and cut the changelog. This release is the storage-layer
reliability work: LanceDB maintenance split into lock-free compaction and
write-locked reclamation (fixing unbounded index growth), every write-lock
critical section bounded by a deadline that covers acquisition, a per-kind
prune-staleness signal on GET /health, `everos cascade rebuild` for a drifted
or corrupt index, startup detection of column type drift, and a query-vector
width check that fails fast instead of 13s deep inside LanceDB.
Carries the table-handle deadline fix (previously #385) rather than shipping
1.2.2 with a known stall: the deadline covered the critical section but not the
await ahead of it, so a hang while resolving a table handle parked that kind's
maintenance permanently and silently. Found by a 1h high-rate soak run after
#384 merged.
No migration, no config change, no API change: `docs/openapi.json` differs only
in the version string. The only operator-visible requirement is that
`everos cascade rebuild` now refuses to run while a server holds the OME lock.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: zhanghui <zhanghui@shanda.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(lancedb): reclaim stale versions via write-locked prune
The storage soak (48h, sustained churn + fuzz) proved the bundled
lock-free `optimize(cleanup_older_than=...)` loses its commit-conflict
race against concurrent writes — cleanup ran only 16 of ~250 times, so
old dataset versions / FTS orphans piled up and the index dir grew to
the 40G disk guardrail and never reclaimed under load. main still had
that bundled call.
Split the maintenance path:
- `LanceRepoBase.optimize()` is now compact-only and lock-free (a commit
conflict here is benign — the next beat retries, so it must not stall
writers).
- `LanceRepoBase.prune(older_than)` runs `cleanup_older_than +
delete_unverified=True` **under the per-table write lock**, so no
writer is in flight: the Rewrite has the manifest to itself (cleanup
completes every beat) and aggressive deletion is safe. It also removes
the empty `_indices/<uuid>/` husks cleanup leaves behind (soak: 13061
dirs, 98% empty), offloaded to a thread.
- The cascade worker's heavy beat calls `prune()`; the light beat calls
`optimize()`. A benign light-beat commit conflict is logged at debug
and does not count toward the failure streak or trigger a rebuild.
- Prune's retention window (`cleanup_older_than`) is decoupled from the
prune cadence and defaulted short (60s). It runs under the write lock,
so the window only needs to outlive an in-flight read; keeping it =
cadence (300s) left ~2 cadences of superseded full-table copies on
disk between beats (soak: transient ~15G/table peaks). 60s reclaims
all but the last minute each beat — same live floor (~625MB/table),
far lower transient footprint.
Result on the re-run soak: disk sawtooths and reclaims to live-data size
(~1.3G total) under active load — vs run1 stuck at 40G until writes
stopped — with 0 crashes / 0 OOM / 0 stuck cleanups over 48h.
Cascade projection health is now observable:
- `CascadeOrchestrator.health()` -> `CascadeHealth`, combining the
worker's in-memory signals (drain-loop failures, unrecoverable count,
optimize streak, prune staleness) with the SQLite queue summary.
- `GET /health` gains a typed `cascade` readiness block. `healthy`
reflects operational health only (drain / optimize / prune);
`failed_permanent` (files awaiting `cascade fix`) is a data-quality
backlog reported as an informational count that does NOT flip
`healthy` — otherwise the signal would sit red permanently.
The scanner-side retry cap and the `_MAX_TOTAL_RETRIES` budget already
on main handle re-enqueue storms, so no duplicate is added here.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(deps): pin lancedb to >=0.34.0,<0.35.0
The previous open-ended `>=0.13.0` let any environment float to an
untested release, including 0.32-0.34 which carry a compaction
offset-overflow regression (lance-format/lance#7653) that stalls
version cleanup and grows the index dir without bound.
- Floor 0.34.0: the current resolved version; runs safely thanks to
the with_position=False FTS workaround shipped in #336. Verified that
data written by lancedb 0.32.0 (lance v6) reads correctly under
0.34.0 (lance v8), so existing deployments upgrade cleanly. Never
widen the floor below 0.34 -- older lance cannot read v8-format data.
- Ceiling <0.35.0: 0.35 embeds lance-rust v9 (large encoding jump, not
yet stable-released); it must pass the soak harness before we allow
it.
Resolved version is unchanged (still 0.34.0); this only tightens the
declared constraint.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(cherry picked from commit 94f9aa67d11a07c93c7d59d6b134e80858b60316)
* fix(search): bridge agent-kind metadata into the agentic doc contract
The agent AGENTIC path (`agent_case` / `agent_skill`) fed recall
candidates straight into `aagentic_retrieve`, whose `_format_docs`
(LLM sufficiency / multi-query prompt) reads `metadata["episode"]` as a
`{subject, content}` dict plus a ms-epoch `timestamp`. Agent-kind rows
carry their body in the recaller's `text_field` and time as a datetime,
so `_format_docs` raised `TypeError: Candidate ... has no episode dict`
and `POST /api/v*/memory/search` returned 500 for any
`owner_type=agent` + `method=agentic` request.
Mirror the episode path's bridge: reshape agent candidate metadata into
the everalgo doc contract before `aagentic_retrieve`, and revert it
before DTO shaping so the agent shapers still see a datetime timestamp.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit bb9a3a585e044643d3852b9d2810b42c05a7a47c)
* test(search): regenerate search seed to current linkage + migrate e2e
The committed `search_seed` fixture and several search e2e tests were
written for the pre-1.5 memcell fact-linkage model. Current extraction
links atomic_facts to episodes via `parent_id == episode.entry_id`
(parent_type="episode"), and user_memory clusters store episode
entry_id members — so the stale fixture made VECTOR/AGENTIC recall and
the cluster-narrowing path find nothing, and stale assertions checked
an old error code.
- Regenerate `search_seed/*` from a fresh corpus in the current
entry_id format; facts now bridge across multiple episodes (richer
agentic / hierarchical-eviction coverage).
- Fix `_dump_search_seed.py` sampling: pick episodes that host facts
first and keep facts by episode entry_id, so re-dumps stay coherent.
- Migrate e2e tests to the entry_id model (hierarchical-eviction,
session/timestamp filters, cluster seeding helper) and update the
filter-error assertion to the current `INVALID_INPUT` code.
- Provision `ome.toml` in the full-app pipeline fixture (the OME config
reloader requires it; strategies are code-registered so the packaged
default suffices), unblocking corpus regeneration.
Full search e2e suite now green (49/49).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit 44cd9cecc07775cb9270104a5804f36e32cd788d)
* fix(lancedb): detect schema type drift and add cascade rebuild recovery
verify_business_schemas only compared column names, so a column whose
on-disk Arrow type had drifted (name unchanged) slipped through and
detonated later inside merge_insert as an opaque LanceError(IO)
"Spill has sent an error" (#337). Now compare each shared column's Arrow
type against schema.to_arrow_schema() — the exact schema get_table
builds tables from, so a healthy table never false-positives.
Reproduced #337 byte-identically: an episode.subject_vector column left
as string or null by an older build, plus a real 1024-d vector on
upsert, yields the exact crash. No lancedb version (0.13-0.34) renders
Optional[Vector] as a non-vector type, so the startup guard is what
should catch it — not the runtime.
Add `everos cascade rebuild` as the safe recovery: it drops the business
LanceDB tables and re-indexes from markdown, skipping the verify guard
(which the drift would otherwise trip on startup). Unlike removing only
.index/lancedb it re-populates already-done entries (reset_all clears
the cascade queue); unlike removing all of .index it preserves
unprocessed_buffer (messages not yet extracted).
Fixes#337.
* docs(cascade): document cascade rebuild and correct recovery guidance
Add the `everos cascade rebuild` command to the runbook, CLI, and
how-memory-works docs. Correct the old recovery guidance: a bare
`rm -rf .index/lancedb` leaves md_change_state marked `done`, so the
scanner skips those files and the index comes back empty — the runbook
previously claimed a full repopulation that does not happen. `cascade
rebuild` is the safe path (re-populates done entries, preserves
unprocessed_buffer). Also document that verify now checks column types.
* chore(rebase): adapt #354 integration test to soft-embedding main
CascadeOrchestrator dropped the embedder param when embedding became a
soft dependency (main); fold the schema-drift integration test onto it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(cascade): freeze monotonic clock in prune-staleness health tests
Both prune-staleness health tests fabricated
`_started_at = time.monotonic() - (ALERT + 100)`, assuming monotonic()
is a large value. On a fresh CI runner monotonic() is only ~100-180s, so
the subtraction went negative, the source clamped the baseline to 0, and
staleness read back as ~130s < 900s — failing on CI while passing on
long-lived dev boxes where monotonic() is huge.
Freeze the monotonic clock via monkeypatch so staleness is deterministic
regardless of the runner's boot uptime. Source logic is unchanged; only
the tests are made hermetic.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(cascade): wait for stable terminal state in rename scenarios
`_wait_path_done` reached a terminal status, slept a 0.1s settle window,
then asserted the status was still terminal — which contradicted its own
docstring ("absorb any last-second re-enqueue"). A rename's delete event
or an atomic-replace echo can flip a done row back to `processing` inside
that window, so on a slow CI runner the assert fired
("flipped back to processing after reaching done"), failing
test_rename_cross_owner_keeps_frontmatter_owner intermittently (seen on
the 3.13 job). `make integration` runs without `--reruns`, so a single
flake fails the whole job.
Wait for a terminal state that *survives* the settle window instead:
absorb a transient re-enqueue by waiting for terminal again, still bounded
by `deadline` so a row that never settles surfaces as a timeout. Pre-
existing flake on main, unrelated to the prune change; the scenario's real
assertions (row counts, frontmatter owner) are untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(cascade): cross-process prune safety + backfill reclaim fix
Review of #379 found two P0s plus P1/P2s, all verified against the code:
- P0-1: cascade backfill still called the removed
optimize(cleanup_older_than=…) kwarg → TypeError swallowed by the
best-effort try/except → backfill silently skipped all compaction +
reclaim (the exact bloat this PR fixes). CI stayed green because the test
double kept the stale signature. Fix: call optimize() + prune(0) at the
call site; make the fake mirror the real signature so the drift can't hide
again; pin prune in the backfill tests.
- P0-2: prune ran delete_unverified=True guarded only by an in-process
asyncio lock, but the runbook promises `cascade sync` is safe alongside a
live server — and the CLI's first optimize beat does prune, in a separate
process. It could delete files the daemon is mid-commit on. Fix: switch
prune to delete_unverified=False. Measured to reclaim identically on
churned tables (both collapse superseded versions ~97%); True only
additionally deletes in-flight/dangling files — exactly the corruption
vector. No cross-process lock needed; the write-lock/commit fix (the real
reclaim win) is unchanged.
- P1-3: /health called orch.health() (6 SQLite aggregates) with no guard →
a locked/full/migrating DB would 500 the liveness probe and restart the
container. Wrap it: unhealthy readiness + reason, HTTP stays 200.
- P1-4: rebuild drops + recreates tables; a live daemon holds cached handles
pointing at the dropped dataset. Runbook now says stop the server first —
the one cascade command unsafe alongside a live server.
- P2: narrow _is_benign_commit_conflict to the "commit conflict" phrase (a
bare "retryable" swallowed unrelated recoverable errors); add a timeout
around the prune cleanup so a hung lance call can't wedge the write lock;
correct two stale docstrings.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs: fix schema-recovery guidance + drop dead internal doc refs
Follow-up to the review; two doc issues neither the review nor the fix
commit caught:
- The schema-drift startup error's docstrings (verify_business_schemas
and LanceDBLifespanProvider) still described the recovery as
`rm -rf ~/.everos/.index/lancedb` — which the runbook explicitly calls
the WRONG recovery (it leaves the cascade queue `done`, so nothing
re-indexes and the index comes back empty). The raised error already
points to `everos cascade rebuild`; align the docstrings to match.
- 15 dangling references to an internal numbered design-doc set
(12_/13_/16_/17_*.md) that was never shipped to this repo. Point the
schema-recovery ones at docs/cascade_runbook.md; drop the rest (pure
provenance in table/component docstrings) while keeping the substance.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs: align maintenance docstrings with split optimize/prune API
Two docstring residuals from the #379 review's P2 list:
- _run_optimize_once still described the pre-split bundled heavy beat
("same work plus cleanup_older_than ... older than one cadence");
the heavy beat now calls prune() under the write lock and the
retention window is decoupled from the cadence
(DEFAULT_OPTIMIZE_PRUNE_RETENTION_SECONDS).
- _restore_shaper_metadata converts any numeric timestamp, wider than
an exact inverse of the bridge; document that this is deliberate
(the shaper contract requires a datetime either way).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* style(e2e): hoist fixture-body imports to conftest module top
shutil / importlib.resources.files were imported inside the
core_pipeline_runtime fixture body; move them to the module top to
match the repo import style (#379 review P2).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(cascade): back off a hung prune via a separate attempt clock (N1)
The write-lock timeout on prune (review P2) didn't achieve its goal: it
was 300s — equal to the prune cadence — and `last_prune_at` only advanced
on success. So a hung lance cleanup timed out after 300s, `should_prune`
was still true (clock never moved), and the next beat re-pruned ~10s
later — pinning the per-table write lock ~97% of the time, the exact
write-starvation the timeout was meant to prevent.
A real cleanup is milliseconds even on a heavily churned table (measured
~40ms at 320k writes / 100 versions), so the timeout is a pure hang-catcher:
lower it to 60s (~1500x headroom, never fires normally, well below the 300s
cadence).
Split the prune clock so a failed prune backs off without masking the
health signal:
- last_prune_attempt_at (new) gates scheduling, advanced before the call
whether it succeeds or times out → a hung prune waits a full cadence
before retrying (light lock-free compaction runs meanwhile), so the lock
is held at most ~timeout/cadence ≈ 17% in the worst case.
- last_prune_at advances only on success and still drives the
prune-staleness health signal, so a persistently failing prune surfaces
as degraded instead of being hidden by an advanced schedule clock.
Regression test: a raising prune advances the attempt clock (next beat is
light, no immediate re-prune) but not the success clock.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: zhanghui <zhanghui@shanda.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Bump version to 1.2.1 and finalize CHANGELOG. Highlights: [embedding]
and [rerank] become soft runtime dependencies; new everos cascade
backfill CLI; LanceDB schema v2 (nullable vector); PyPI Trusted
Publishing workflow; 1.1.4 backport fixes.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(config): make [embedding] and [rerank] soft dependencies
Make [embedding] and [rerank] soft runtime dependencies so a
freshly-onboarded user can run EverOS end-to-end with only [llm]
configured. Previously the server refused to start without embedding,
locking out anyone who just wanted keyword-only search.
## Capability tiers
- Tier 1 ([llm] only) KEYWORD search, add/flush, md
writes, cascade sync
- Tier 2 ([llm] + [embedding]) + VECTOR / HYBRID search,
reflection, skill extraction,
backfill
- Tier 3 ([llm] + [embedding] + rerank) + AGENTIC search, knowledge
Tier upgrades require a server restart (capability accessors cache
for the process lifetime). Tier downgrades are read-safe: a Tier-3
user who drops [rerank] can still read/rename/delete existing
knowledge documents; only write/search endpoints return 422.
## What changed
- Component accessors — component/{embedding,rerank,llm}/accessor.py
are the single process-wide provider singletons. service/* never
maintains parallel singletons; it consumes get_embedding_capability()
/ get_rerank_capability() / get_llm_client() directly. Build-time
ValueError from the factory is logged as capability_build_failed
(was silently swallowed).
- Error mapping — ProviderNotConfiguredError -> 422 with everos.toml
section hints (never EVEROS_* env-var strings).
LanceDBMigrationError fails loud with escalating recovery guidance
(restart -> wipe index). LLMNotConfiguredError in search maps to
None for KEYWORD degradation.
- Nullable-vector LanceDB migration — schema v2 makes the vector
column nullable so Tier-1 rows can land without embeddings.
Migration is guarded by a cross-process memory_root_lock
(fcntl.flock + anyio.to_thread) and runs optimize() per table
after Phase-1 backfill to reclaim manifest bloat.
- Cascade — knowledge handlers register unconditionally (Tier-3 ->
Tier-2/1 downgrade no longer strands DELETE); embed-requiring
strategies use body-guards that check capability.available at
execution time. _TABLE_SPECS has an import-time drift assertion
against BUSINESS_SCHEMAS_WITH_VECTOR.
- `everos cascade backfill` CLI — Phase-1 (embed missing vectors) /
Phase-2 (emit synthetic events for cascaded processing) / Phase-3
(sync new skill files). Exit codes: 0 / 1 / 2 / 3 (server running
preflight) / 4 (COMPLETED_WITH_FAILURES — per-row failures rolled
up) / 130 (SIGINT). OMEConfig.crash_recovery_enabled=False in
backfill engines prevents stale-RUNNING rows re-enqueuing into a
smaller strategy registry.
- /health — reports capabilities + disabled_features per tier so ops
can distinguish "boots but degraded" from "boots and full".
- Presentation split — memory / service / infra never import typer /
click. TyperPresenter Protocol + run_backfill() live in
entrypoints/cli/commands/_backfill_cmd.py. Enforced by
import-linter.
- Startup hint — unconditional count_rows(filter="vector IS NULL")
sweep emits unbackfilled_memory_rows (event name + hint text
pinned) when Tier-1 rows exist. ParserLifespanProvider warms the
everalgo.parser import at boot so /health doesn't block on first
call.
- Knowledge upload UTF-8 short-circuit — _looks_like_utf8_text()
routes text/* mime and known plaintext extensions (md/txt/rst)
straight to UTF-8 decode instead of the parser. Prevents 503
Multimodal-not-configured when Tier 3 sans [multimodal] uploads a
markdown doc.
## Sync history with main (2 merges collapsed into this squash)
Merged origin/main at 6dcd3eb (v1.1.4 -> v1.2.0 adds OTel tracing,
/api/v2 alias, TracingLifespanProvider, per-cascade-embedding span
fix, memory-op instrumentation) and later at 42629df (PR #366
backfills v1.1.4 CWE-22 knowledge path traversal fix + cascade
retry-budget rework + errors.py -> core.errors.ExternalServiceError).
Key merge decisions:
- service/search.py adopts single wrap site — component.llm accessor
already applies UsageRecordingClient when observability is on;
service layer never keeps a parallel LLM singleton (Round-1 CR
rule: "service layer never maintains parallel singletons").
- Knowledge router prefix moved to /knowledge; create_app() mounts
it under both /api/v1 and /api/v2.
- Cascade retry classification uses ExternalServiceError from
core.errors (cascade/errors.py deleted). _MAX_TOTAL_RETRIES=12
cross-cycle budget preserved.
- Fixed backport typo: MemoryRoot.default() -> MemoryRoot.resolve()
(no .default() classmethod exists — main PR #366 shipped a broken
call).
## Verified layering
$ git grep -l "^import typer\|^from typer" src/everos/{memory,service,infra}
# empty
$ git grep -l "^import click\|^from click" src/everos/{memory,service,infra}
# empty
Memory / service / infra layers clean of CLI presentation libraries.
## Review history
Three rounds of Fable 5 (opus) code review across the pre-squash
commit history closed 38 findings total:
- Round 1: 10 findings (fail-loud migration, backfill hardening,
knowledge router gate scoping, SearchManager guards, profile
throttle lift)
- Round 2: 13 findings (hermetic test env, hot-reload doc drift,
knowledge handler registration, Phase 3 sync guarantee, Phase 2
idempotency, profile event-first path, OMEConfig crash-recovery
gate, cross-process migration lock, batch embed per-row fallback,
LanceDB optimize, typer/click layer split)
- Round 3: 15 Minor cleanups (accessor unification, marker revert,
episode query hygiene, --verbose subcommand, parser lifespan warm,
task-number scrub, temporal-overlap test, real-SIGINT slow mark,
4 design-note back-references)
Full per-round context lives in the PR description on GitHub.
## Test plan
- make lint (ruff + import-linter 3 contracts + assets +
deprecated-names + github-docs + datetime + OpenAPI drift)
- Hermetic env full pytest — 2027 passed / 7 deselected (7 = slow +
live_llm markers)
- Manual e2e across Tier 1/2/3 (21/21 assertions across v1/v2
double-mount and Tier 3 -> Tier 2 downgrade)
- /health reports correct capabilities + disabled_features per tier
## Known follow-ups
- .superpowers/sdd/followup-http-bridge.md (gitignored) — Path A for
spec §10's "backfill 期间 EverOS 完全可用" promise
- _TABLE_SCHEMA_VERSION docstring — v3+ migrations need a version
dispatch table
- extract_user_profile.py throttle-counter block — replace LanceDB
count_by_owner with a sqlite memcell count
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(review): close 3 blockers surfaced by round-4 review
N1. cluster_repo.find_cluster_id_for_member was cross-owner-unsafe.
Its reverse index (member_type, member_id) alone cannot disambiguate
two owners whose entry_id happens to collide — entry_id is
deliberately only per-owner unique (see entries.py:47:
'Cross-user uniqueness is handled at the database layer via a
composite <user_id>_<entry_id> field; it is not encoded into the
EntryId string itself'). Phase 2's _scan_all_rows crosses all
owners, so on any multi-owner root, same-day seq=1 episodes under
different owners would either false-hit each other's cluster or
be silently skipped from clustering. Add required (app_id,
project_id, owner_id) keyword args + JOIN Cluster (which already
carries scope) to filter by parent scope. Prior signature had zero
production callers except two the same PR just added, so the
API break is contained. Regression test: two owners persist a
cluster each around the same entry_id, each lookup resolves to
its own owner's cluster, a third owner's lookup returns None.
N2. Ctrl-C / EOF at the y/N prompt was landing on the generic
except Exception branch (exit 2 with rich traceback) instead of
the exit-130 interrupt path. Root cause: typer 0.15+ vendored
click under typer._click, so typer.Abort and the standalone
click.exceptions.Abort are distinct classes. The interrupt-branch
catch only listed the standalone one; every existing 'abort'
test was manually raising click.exceptions.Abort so the miss
was a false-positive guard rail. Widen the catch to
(typer.Abort, click.exceptions.Abort) and declare click as a
first-class dependency (it was only pulled in via uvicorn).
Regression test: raise real typer.Abort() at the confirm step
and assert exit 130 + INTERRUPTED banner.
N3. _looks_like_utf8_text used mime.startswith('text/'), which
caught text/html as well. HTML uploads then bypassed everalgo's
_aparse_html — losing clean_html_for_llm (strips <script>/<style>
/<nav>/<iframe> + HTML comments) and the 1 MiB output cap. A
40 MiB .html with <script> bodies and <!-- prompt injection -->
comments would flow straight into the extraction LLM. Replace
with explicit allowlist {text/plain, text/markdown, text/x-rst,
text/x-markdown}; text/html and any future text/* mime now
default to the parser path. Test matrix asserts text/html →
False (was regressed as True by the earlier commit).
Hermetic env full pytest: 2033 passed / 7 deselected (+6 tests
from these regressions).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(review): close round-4 major + minor + PR body errata
Round-4 review-driven cleanup. Blocker fixes (N1/N2/N3) landed in
561b5fe. This commit closes the remaining CONFIRMED items:
Major:
- J3 MemoryRoot.default() -> resolve() was a breaking public API
rename this branch introduced (main still has default()). Adds
default() as a backward-compat alias forwarding to resolve()
with DeprecationWarning; CHANGELOG entry under Unreleased.
- J4 episode_repo.list_by_owner_after_ts(limit=N) truncates in
fragment order (== insertion order), NOT newest-first. Docstring
now spells out the trap so a future caller passing limit for a
'newest N' window doesn't silently get the oldest N.
- J5.2 TyperPresenter.nothing_to_backfill picked colour via
'could not be read' in message — a domain wording change
would silently flip yellow -> green. Signature gains explicit
scan_failed: bool kwarg; CLI colour-picks off the flag.
- J5.5 phase_header was Protocol-declared but never dispatched
(run_backfill calls _print_phase_header directly). Removed the
dead Protocol method + both no-op implementations.
- J6.3 3 inline from everos.core.errors import ... inside
Phase 1/2/3 preflights promoted to a single top-level import.
- J7 subject-side embed failure was silently exit-0 because
rows_processed advanced whenever any side wrote. Now: a row with
a needed side still NULL counts as rows_failed (exit 4 =
COMPLETED_WITH_FAILURES). Gated on spec.subject_of + row.needs_*
+ row.subject_text so non-Episode tables and subject-empty rows
don't false-positive.
- J9 test_migration_cross_process.py did NOT actually test cross-
process (all 5 tests mock memory_root_lock). Renamed to
test_migration_lock_wiring.py; docstring now scopes it to
'lock-invocation wiring' and points at test_core/…/test_locking.py
for real flock coverage.
- J10 Phase 1 lacked the server-running preflight Phase 2/3 have.
--phase all against a live server would burn Phase 1 embed API
calls (real cost) before Phase 2 halted with exit 3. Phase 1 now
probes _probe_ome_lock_available first; regression test in
test_backfill_preflight.py; upgrade_path integration patches the
probe so its in-process 'server + backfill' scenario stays valid.
Minor:
- M1 knowledge upload with NUL byte or filename > 255 bytes UTF-8
used to raise ValueError/OSError at write_bytes → 500 with a
half-written md left on disk. _safe_original_filename now
rejects both up front with InvalidInputError (→ 400).
- M2 backfill optimize() now passes cleanup_older_than=timedelta(0)
so older manifest versions are physically pruned (previous call
compacted fragments but left the manifest chain on disk).
- M3 verify_business_schemas remediation text used to jump straight
to 'rm -rf ~/.everos/.index/lancedb'; now walks restart → wipe.
- M5 multimodal/accessor.py capability_build_failed warning added
so all four provider accessors log symmetrically (was silent).
- M7 test_knowledge_api parser-absence tests call
parser_available.cache_clear() around the sys.modules patch so
the lru_cache doesn't strand a stale True/False.
- M8 cascade_handler_embed_skipped (6 handlers) demoted INFO → DEBUG:
Tier 1 imports were generating N × 6 handler-info lines per md.
- M10.1 test_drift_scenario_would_raise was a tautology (compared
two hardcoded string sets, never touched the guard). Now
monkey-patches BUSINESS_SCHEMAS_WITH_VECTOR to a superset and
reloads _backfill, proving the import-time RuntimeError fires.
- M11 test_cascade_verbose_position subprocess.run calls gain
env= — scrubs EVEROS_* from the developer environment so the
same footgun as round-2 B1 doesn't re-appear inside subprocesses.
- M14 health() -> dict degraded the OpenAPI schema to
additionalProperties: true. Introduce HealthResponse +
HealthCapabilities Pydantic models so clients get real field
shape; docs/openapi.json regenerated.
- M16 routes/knowledge.py:_require_knowledge_capabilities docstring
claimed cascade.registry.build_handlers still gates
knowledge_topic/knowledge_document, contradicting
registry.py:177-194 (gate removed there, moved to HTTP layer).
Rewritten to describe the current design accurately.
Hermetic env full pytest: 2037 passed / 7 deselected.
Explicitly deferred to followup:
- J1 lazy multimodal client (needs everalgo signature change)
- J2 tier definition (knowledge = Tier 3 whole)
- J5.1/3/4 broader presentation-split refactor
- J6.1/2 backfill dispatch and _backfill_table refactor
- J8 Phase 1 keyset pagination for bulk migration OOM
- M4 flock timeout/waiting log/re-entry (core/persistence refactor)
- M6 UTF-8 codec strategy (BOM/UTF-16/GBK)
- M9 count_by_owner monotonicity (latent, INTERVAL=1 short-circuits)
- M13 --phase all multi-phase combined-outcome test coverage
- M15 PR-marker rationale comments (16 occurrences, all inert)
M12 was refuted (both event names exist).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Minor release: `/api/v2` API prefix (v1 retained as alias) and native
OpenTelemetry tracing — both back-compatible, so 1.1.4 -> 1.2.0.
- pyproject: version 1.1.4 -> 1.2.0
- CHANGELOG: promote [Unreleased] to [1.2.0]
- docs/openapi.json + uv.lock: regenerated for the new version
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Every business endpoint (memory/*, ome/*, knowledge/*) is now served
under /api/v2, aligning the open-source API with the EverOS Cloud
contract. /api/v1 is retained as a permanent, backward-compatible alias:
the same router objects are mounted under both prefixes, so both resolve
to identical handlers and request/response contracts. Existing /api/v1
integrations keep working unchanged. Infra endpoints (/health, /metrics)
stay unversioned.
Fix the Prometheus request-metric label to build the path from the full
request URL (with path params folded) rather than the route's
router-relative path, so the version prefix is preserved and v1/v2
traffic stays distinguishable.
Docs (docs/api.md, docs/openapi.json), CHANGELOG, and route docstrings
updated to lead with /api/v2. Add test_api_versioning as the parity
guard: every v2 route has an identical v1 twin and vice versa.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
compile_filters() unconditionally appended 'deprecated_by IS NULL' to
every query, but the deprecated_by column only exists on user-scoped
tables (episode, atomic_fact — Reflection V1). Agent tables
(agent_case, agent_skill) lack this column, causing a SQL error on
agent search/get queries.
Gate the clause behind owner_type == 'user' so agent queries no longer
reference a non-existent column.
Bump version to 1.1.2.
Co-authored-by: Jiayao Song <jiayao.song@shanda.com>
md-first memory extraction framework for AI agents.
Markdown is the single source of truth; SQLite holds state and LanceDB
provides the rebuildable vector + BM25 + scalar index. The codebase follows
a single-direction DDD layering (entrypoints -> service -> memory -> infra,
with component / core / config cross-cutting) enforced by import-linter.
Engineering surface:
- Coding conventions in .claude/rules/ (path-scoped) and workflows in
.claude/skills/ (/commit, /new-branch, /pr).
- GitHub Actions CI runs make lint + test + integration; pre-commit mirrors
the gates locally (ruff, hygiene hooks, gitlint commit-msg).
- Commit messages follow Conventional Commits, enforced by gitlint.
- make lint also enforces datetime two-zone discipline and OpenAPI drift.
zhanghui
Kendrick-Song
zhanghui
Elliot Chen
Elliot Chen