* chore(release): v1.2.3
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs(changelog): note the ome.db growth in the 1.2.3 upgrade section
SkillClusterUpdated now persists a 1024-dim embedding, taking a
skill_cluster_updated run_record row from ~0.8 KB to ~14 KB. That is a sizing
change for ~/.everos/.index/sqlite/ome.db and belongs next to the foresight
default flip, not only in the Changed entry.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: zhanghui <zhanghui@shanda.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Upstream merged the fix on 2026-08-06: cleanup now removes the directories it
empties, under the same 7-day / delete_unverified policy this sweep
reimplements. It is not in any release yet (lancedb 0.34.0 embeds lance 8.0.0;
the merge sits ahead of v11.0.0-beta.2), so the sweep stays -- but the next pin
carrying it should delete this rather than keep a second copy of upstream's
rule. That upgrade also collapses the documented horizon from ~14 days to ~7:
lance drops the directory in the same pass that empties it, so our age gate
stops stacking on top of theirs.
Co-authored-by: zhanghui <zhanghui@shanda.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat(ome): exponential backoff + jitter between retry attempts
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(events): carry case body + vector on agent skill chain
* feat(strategies): populate extended agent-skill chain events
* feat(md): AgentSkillReader.list_by_cluster for md-first skill enum
* fix(strategies): rescue extract_agent_skill from cascade-lag dead-letter
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(agentic): honor radius, use kind-shaped rerank, non-empty passage
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(api/ome): expose dispatched + runs, distinguish not_dispatched
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(release): v1.2.3 — agent-skill rescue + OME/agentic contracts
* fix(agentic): drop inert radius plumbing; correct release docs
* fix(md): sanitize LLM-generated skill names against path traversal
AgentSkillFrontmatter.name comes straight from LLM output
(memory.strategies.extract_agent_skill) and was concatenated
unsanitized into the skills/skill_<name>/ directory segment on both
the write path (agent_skill_writer._skill_dir) and the read path
(agent_skill_reader._skill_dir). Given a sufficiently long ../ prefix,
the write target could escape the memory root (CWE-22). A live run
also produced a skill name containing spaces and CJK characters,
proving the "keep snake_case" docstring convention on
AgentSkillFrontmatter.name is not enforced at runtime.
This is the same class of defect already fixed for knowledge-upload
titles/categories. Promote that fix's sanitizer
(knowledge_writer._sanitize_dirname) to a shared primitive,
everos.core.persistence.markdown.sanitize_dirname, so there is one
CWE-22 defense for md directory names instead of two independently
maintained copies:
- New core/persistence/markdown/path_safety.py holds sanitize_dirname
(idempotent: sanitize(sanitize(x)) == sanitize(x)), exported through
the markdown + persistence facades.
- SkillPathMixin gains skill_dir_name(), the single sanitization point
both AgentSkillWriter._skill_dir and AgentSkillReader._skill_dir now
derive from, replacing their previous independent string
concatenation.
- KnowledgeWriter now imports the shared sanitize_dirname instead of
keeping its own private copy.
- AgentSkillFrontmatter.name gains a field_validator rejecting path
separators / ".." as defence in depth, so a hand-edited SKILL.md is
caught on parse rather than silently relocating the skill on the
next write.
Idempotency is what keeps the reader and writer in agreement even
though they recover a skill_name from different sources:
list_by_cluster derives it from the on-disk (already-sanitized)
directory name, while extract_agent_skill._hydrate_algo_skills
re-reads using the frontmatter's raw name field. A regression test
(test_agent_skill_reader.py) seeds a skill whose frontmatter name
contains CJK + a space and asserts both routes resolve to the same
file.
No data migration: agent-skill extraction has never once succeeded
before this branch (the cascade-lag defect this branch fixes meant
.skills/ was never created), so there is no legacy skill corpus whose
directory names would change under the new sanitizer.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test(e2e): make the agent-skill chain assertion real
tests/conftest.py's autouse fixture pins embedding + rerank capability
to unavailable for hermeticity, and this test never opted back in.
trigger_skill_clustering and extract_agent_skill both body-guard on
get_embedding_capability().available and return early, so the module
docstring's "real embedder" claim was false and the skill chain never
ran: measured log counts were skill_cluster_updated=0,
agent_skills_extracted=0, strategy_gated_off_embedding_unavailable=10.
The three skill assertions were assert len(...) >= 0 — always true —
with a comment blaming "LLM-dependent" flakiness for a count that was
in fact deterministically zero. This is why a defect that made
agent-skill extraction fail 4/4 in production reached a release: there
was no working e2e coverage of the chain.
- New _opt_in_real_embedding_and_rerank autouse fixture, scoped to this
file only, resets everos.component.embedding.accessor._capability and
everos.component.rerank.accessor._capability to None (the mechanism
the global fixture's own docstring prescribes) so both capabilities
rebuild from the real .env credentials tests/e2e/conftest.py already
loads. Restores to None on teardown; every other test keeps its
hermetic default.
- Replaced the three vacuous per-agent assertions with one aggregate
floor across all three agents (>= 1 total skill). A per-agent floor
would be flaky: extract_agent_skill has no cluster-size gate, only
everalgo's per-case skip_quality_threshold, so a single low-quality
trajectory can legitimately yield 0 skills for one agent.
- Added a sharper, defect-specific check: assert no dead-lettered
extract_agent_skill run in OME's run_record (via
OfflineEngine.list_runs), since a dead-letter (retries exhausted)
is unambiguously a failure, unlike a quality-gated 0-skill outcome.
- Corrected the module docstring's "real embedder" claim and the old
"# 4.5" comment's reasoning: extract_agent_skill has no cluster-size
gate, only everalgo's per-case quality threshold.
Unexecuted: this test is slow + live_llm and this machine has no
provider credentials (the verification .env was deleted), so make ci
does not run it and it could not be run here. Verified by inspection
instead: ran the test file with -m "" to override the marker
deselection and confirmed it proceeds past the new fixture and
through app lifespan startup without error, failing only at the
expected point — LLMNotConfiguredError from the missing API key —
which confirms the fixture and imports are wired correctly and the
only blocker is the missing credentials, not a bug in this change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(strategies): sanitize skill name before frontmatter construction
Follow-up to afe1609 (path-traversal sanitization). That commit added a
field_validator on AgentSkillFrontmatter.name rejecting path separators
or "..", intended as a read-side defence for hand-edited SKILL.md
files. But _persist_skill (memory/strategies/extract_agent_skill.py)
constructs AgentSkillFrontmatter(name=skill.name) with skill.name
straight from LLM output — the raw, unsanitized string — so the
validator actually fires on the write path too. Sanitization already
makes the on-disk path safe (SkillPathMixin.skill_dir_name), so an
LLM emitting a traversal-shaped name (reachable via prompt injection,
since the LLM's input is user conversation content) gained nothing
from the validator except a new failure mode: ValidationError ->
strategy raises -> OME retries with backoff -> dead-letter -> that
case's skill is permanently lost. A DoS vector introduced by a
security fix, and it also contradicted the validator's own docstring
("catches a hand-edited SKILL.md").
Also fixes a latent second bug in the validator itself, caught by the
new tests below: it rejected any name containing the substring "..",
but sanitize_dirname keeps "." as a safe character, so
"../" * 8 + "tmp/pwned" sanitizes to "................tmppwned" —
still containing ".." many times over. The validator would have
rejected the sanitizer's own safe output. Narrowed the check to actual
path separators or the name being exactly ".." (the only case where
".." functions as a real traversal component when there's no separator
left to combine it with).
Fix:
- SkillPathMixin gains sanitize_skill_name(skill_name) — the bare
sanitized name (no skill_ prefix), factored out of skill_dir_name so
both share one sanitizer call.
- _persist_skill now sanitizes skill.name via sanitize_skill_name
once, up front, and uses that same sanitized string for
AgentSkillFrontmatter.id, .name, and the writer.write_main() call.
A traversal-shaped LLM name is now made filesystem-safe before it
ever reaches the frontmatter constructor, instead of tripping the
validator.
- The validator's docstring now describes actual behaviour: the write
path pre-sanitizes, so the validator only fires for a name that
bypassed the writer (e.g. a hand-edited file, or any other direct
AgentSkillFrontmatter construction that skips pre-sanitization).
Bonus: with the write path pre-sanitizing, frontmatter.name becomes
byte-identical to the directory-derived name for LLM-written skills —
an identity, not merely an idempotency argument. This also closes the
gap in the previous commit's reader/writer test, which proved
idempotency generically but never drove an adversarial name through
the actual production write path end-to-end.
Tests:
- test_agent_skill.py: constructing AgentSkillFrontmatter with a
pre-sanitized adversarial name (mirroring _persist_skill's own call
shape) succeeds and yields a separator-free name; the read-side
rejection test for bypassed/hand-edited names is unchanged and still
passes with the narrowed check.
- test_agent_skill_writer.py: new parametrized identity test — for
both an adversarial and a CJK/space raw name, sanitize once, write
via that sanitized name, and assert frontmatter.name equals the
directory-derived name exactly.
- test_agent_skill_reader.py: docstring updated to clarify its
existing round-trip test now covers the bypass case (a caller that
writes via a raw, unsanitized name directly through the writer,
skipping _persist_skill's pre-sanitization) rather than the normal
production path, which is proven as an identity by the writer test
above.
- Existing test_extract_agent_skill.py strategy tests (snake_case
fixture names) are unaffected — sanitize_dirname is the identity
function for already-safe names.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(md): reject degenerate sanitizer results, read globbed paths
Security review of 0c6820f found it did not close the DoS it was meant
to close: sanitize_dirname("../") returns ".." verbatim, because "."
is a safe character and is not stripped by the character-class filter
— only the leading "/" is removed. sanitize_skill_name("../") ->
sanitize_dirname("../") -> "..", and AgentSkillFrontmatter(name="..")
still raises ValidationError (name == ".." is exactly the case the
narrowed validator rejects). Same dead-letter DoS as 0c6820f, just a
shorter payload; the previous commit's tests only exercised the long
"../" * 8 + "tmp/pwned" payload, which happens to sanitize past the
fixpoint. The same fixpoint is a real one-level directory escape on
the knowledge path, which has no skill_ prefix to protect it:
Path(root) / sanitize_dirname("../", "Others") / "doc_123" resolved to
root/doc_123, skipping the category directory entirely.
Fix (path_safety.py): sanitize_dirname now falls back on "", ".", or
".." instead of only "". This one change closes both the skill
dead-letter DoS and the knowledge one-level escape, since both callers
already route through this single primitive. Also NFC-normalizes the
input before the character filter (unicodedata.normalize("NFC", raw)),
so an NFD-decomposed accented character (base letter + combining
mark, which is not \w) no longer silently loses its accent. Rewrote
the docstring, which previously claimed ".. sequences are always
stripped" (false — "." is explicitly a safe character) and "cannot
escape the directory it is concatenated into" (false for an unprefixed
caller before this fix); it now states what actually holds: no
separator survives, so the result is always exactly one path
component, and it is never "", ".", or "..".
Also fixes (per review, cheap and worth doing alongside):
- AgentSkillReader.list_by_cluster previously globbed skill_*/SKILL.md,
stripped the prefix to recover a name, then called read_main(name),
which re-derives (and re-sanitizes) the path from that name. Any
on-disk directory whose suffix was not already a sanitizer fixpoint
(e.g. "skill_My Skill", a raw space) re-derived to a path that
doesn't exist and was silently dropped. Since list_by_cluster is the
documented strong-consistency existence check, a dropped skill would
make the LLM emit add() for a skill that already exists, duplicating
it at the sanitized path and orphaning the original. Fixed by having
list_by_cluster read each globbed path directly (new _read_path
helper, shared with read_main) instead of round-tripping through a
recovered name — the reader never derives a path at all on this
route, which is a stronger guarantee than the idempotency argument
the docstrings previously leaned on.
- e2e test: made fixture ordering explicit — the embedding opt-in
fixture now takes _reset_embedding_capability_singleton and
_reset_rerank_capability_singleton as parameters so pytest's
dependency graph guarantees correct ordering, rather than relying on
collection order between conftest files. Added a positive
"extract_agent_skill actually ran" assertion (any status) before the
dead-letter check — without it, the dead-letter assertion alone is
vacuously satisfied by a strategy that never executed at all; it was
only meaningful before because the skill-count floor happened to run
first. Dropped the rerank capability opt-in and the module
docstring's "real reranker credentials" claim: nothing on the
agent-skill write path touches rerank, so opting it in only widened
the credential surface with no coverage benefit.
- CHANGELOG: corrected the validator description (rejects a path
separator or being exactly "..", not any string containing ".."),
and added the previously-missing user-visible fact that
AgentSkillFrontmatter.name and the agent_skill LanceDB primary key
now hold the sanitized name, not the raw LLM output.
Tests: parametrized the sanitize -> construct -> (write, for the
writer-level test) tests over a boundary family instead of one long
payload: "..", "../", "/../", ".", "./", "!!!" (empty), "a" * 200
(truncation), a CJK+space name, and the original "../" * 8 +
"tmp/pwned". Each case asserts the sanitized name is a single
component, is never "" / "." / "..", frontmatter construction
succeeds, and (writer-level) frontmatter.name is byte-identical to the
directory-derived suffix. New test_path_safety.py cases pin the
degenerate-fixpoint fallback directly, the knowledge-style unprefixed
one-level-escape repro, and NFC normalization. New
test_list_by_cluster_finds_skill_whose_directory_suffix_has_a_space
reproduces the exact list_by_cluster drop bug against a directory
written outside the writer entirely.
Explicitly not in scope (per review): the collision behaviour where
"fix django" and "fix_django" now map to the same directory is a real
product-decision question the reviewer is raising separately, not
touched here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs(md): record the skill-name collision trade-off
No behaviour change. Documents a product decision the coordinator made
explicit: sanitize_dirname is lossy, so distinct raw skill names can
collapse onto the same directory ("fix django" and "fix_django" both
become "fix_django"; "fix!django" and "fixdjango" both become
"fixdjango"; names differing only past the 50-char cap also collide).
Because AgentSkillWriter.write_main is a full-file replace and the
LanceDB primary key is f"{agent_id}_{sanitized_name}", a collision
means the later skill silently overwrites the earlier one, losing its
accumulated source_case_ids, maturity_score, and body.
This is accepted rather than mitigated: the LLM's add/update decision
for a skill is keyed on the name it sees, so a collision usually reads
as an intended update anyway; and adding a disambiguating suffix would
break the frontmatter.name == directory-suffix identity the
reader/writer seam (from 0c6820f) relies on.
- SkillPathMixin.sanitize_skill_name docstring now states the
collision consequence and the two reasons it is accepted, so a
reader does not have to derive them.
- sanitize_dirname's docstring gains one line: the function is lossy
and not injective; callers that need distinct outputs for distinct
inputs must disambiguate themselves. General primitive — the
knowledge path calls it too.
- CHANGELOG: added the collision consequence to the existing
path-traversal entry, next to the already-documented fact that name
/ the LanceDB key hold the sanitized value.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(md): return skill bodies from list_by_cluster; correct docs
Security review of 1b8cf11 + add842d found the list_by_cluster fix was
incomplete: it stopped its own enumeration from dropping a skill whose
directory suffix wasn't a sanitizer fixpoint, but the caller
(_hydrate_algo_skills) still re-read each selected skill by
fm.name via read_main, which re-derives (and re-sanitizes) a path from
that name and drops it there instead. Reproduced on a real
filesystem: skill_My Skill/ enumerates fine, but
read_main("My Skill") re-derives to skill_My_Skill/ and misses. The
drop moved one layer downstream; existing_relevant_skills was empty
before and after the prior fix.
Fix: list_by_cluster now returns (frontmatter, body) pairs instead of
frontmatter alone, so the caller never needs a second, name-based
read. _select_existing_skills / _rank_skills_by_relevance updated to
carry (fm, body) tuples through selection; _hydrate_algo_skills is
deleted — the body is already in hand, so there's nothing left for it
to do. This closes the drop for real, removes the second disk read
(the 2n-read concern carried since Task 4), and makes "the reader
never derives a path" true end-to-end rather than true only for
list_by_cluster's own enumeration step.
New end-to-end regression test
(test_select_existing_skills... / test_existing_skills_reaches_llm_for_skill_whose_directory_has_a_space)
seeds a skill_My Skill/ directory directly on disk (bypassing the
writer) and runs the real extract_agent_skill strategy against it,
asserting the skill reaches existing_relevant_skills with non-empty
content — the property the previous commit's test docstring claimed
but the code didn't yet deliver. The reader-level regression test
gained the same body assertion.
Also, per review:
- path_safety.py: corrected the NFC docstring claim, which was wrong
for the ~1,082 Unicode composition-exclusion codepoints (e.g.
Devanagari क़/ख़, U+0958/U+0959) — NFC decomposes an
already-precomposed exclusion character instead of preserving it, so
the combining mark is stripped either way. Scoped the claim to
"best-effort for the common case", not a guarantee for every script.
New test pins this directly.
- Dropped the e2e test's unused _reset_rerank_capability_singleton
fixture parameter: the reviewer adjudicated the earlier instruction
conflict the other way — ordering is only meaningful between
fixtures that touch the same state, and this fixture never reads or
writes the rerank capability at all.
- Widened the skill-name collision documentation (SkillPathMixin.sanitize_skill_name,
CHANGELOG) beyond dropped-punctuation / space-collapse / truncation
to the larger case: every combining mark is non-\w and is stripped
regardless of script, so e.g. Devanagari "किताब" and "कताब" both
collapse to "कतब" (same for Thai tone marks, Hebrew niqqud, Arabic
harakat).
- Corrected the collision justification: because _persist_skill
sanitizes before frontmatter construction, the LLM sees the
already-sanitized name in existing_relevant_skills, so a colliding
raw name is an *affirmative* decision that two skills are different,
not a probable intended update. The decision to accept collisions
still stands, but on its real grounds: a disambiguating suffix would
break the fm.name == directory-suffix identity the reader/writer
seam relies on, and detecting-and-raising would reintroduce the
dead-letter DoS.
- Merged two consecutive "# -- Internals --" banners in
agent_skill_reader.py into one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(md): skip unparseable SKILL.md instead of failing the cluster
`list_by_cluster` only handled a missing file, so a ValidationError from
`_read_path` aborted the whole enumeration. That enumeration is what
feeds `extract_agent_skill` its existing skills, so one bad SKILL.md
starved every skill in the cluster and dead-lettered that cluster's
extraction on every subsequent run -- the permanent-failure mode this
md-first read path was introduced to eliminate. The write path already
pre-sanitizes to avoid exactly this; the read path was left open.
The trigger surface is the whole schema, not just the traversal
validator this PR added: `_read_path` validates the full
`AgentSkillFrontmatter`, so a field a later revision makes required
would take out every existing file at once. Reproduced both ways.
`read_main` still propagates -- a caller naming one specific skill needs
an error, since `None` already means "not created yet" and reusing it
for "exists but is corrupt" would let an upsert overwrite the damage.
Also moves this route's `OfflineEngine` import under TYPE_CHECKING. It
is used only to annotate `_summarize_runs`, and an eager import
contradicted the deferred `_get_engine` import a few lines below. Note
this saves nothing at startup today: `service.memorize` imports the
engine eagerly to construct it, so any app process pays the ~750ms
apscheduler cost regardless.
* docs(md): correct two docstrings, add the case-collision dimension
Three fixes to claims that did not match behavior:
- `_rank_skills_by_relevance` claimed no skill is silently dropped from
the prompt. The backfill loop is capped at MAX_SKILLS_IN_PROMPT, and
the function only runs when the cluster already exceeds that budget, so
skills beyond K are dropped by design. Reworded to what the backfill
actually guarantees: a lagging index cannot under-fill the prompt.
- `sanitize_skill_name` enumerated collision causes in detail but omitted
case, the dimension an LLM varies most freely. "Fix Django" and "fix
django" sanitize to two distinct names -- two LanceDB rows, but one
directory on a case-insensitive filesystem (macOS APFS, Windows NTFS
defaults), so the index advertises a name whose content was overwritten.
- The same docstring justified accepting collisions partly on a
disambiguating suffix breaking the `frontmatter.name` = directory-suffix
identity. It would not: writing "fix_django_2" into both keeps that
intact. Replaced with the real reason it is deferred rather than
dismissed -- it needs a collision probe and a case-folding rule.
Adds the knowledge-writer sanitization tests that were missing entirely:
swapping in the shared primitive changed NFD input ("Résumé" no longer
degrades to "Resume") and made a "." / ".." topic fall back. Knowledge
upload predates this PR, so unlike skills it has a corpus whose
directory names those first cases affect. Both tests verified red against
the 1.2.2 sanitizer.
* docs(changelog): scope the migration claim, record run_record growth
Three corrections to the 1.2.3 entry:
- "No data migration" was asserted for the whole sanitization change but
only holds for agent skills, which have no corpus because extraction
never succeeded. Knowledge upload predates this release and does have
one: NFD topics and `.`/`..` topics resolve to a different directory
now. Scoped the claim and spelled out both cases.
- Added the case dimension to the collision list, and replaced the
"disambiguating suffix breaks the name = directory identity" reason
with the accurate one -- it does not break it, it just needs a probe
and a case-folding rule, so it is deferred rather than rejected.
- Recorded that `SkillClusterUpdated` now persists a 1024-dim vector in
`run_record.event_payload`: ~0.8 KB to ~14 KB per record, ~14 MB per
strategy at the default 1000-record ring buffer. Operators sizing
ome.db need this number, and it was not stated anywhere.
* fix(strategies): ship extract_foresight disabled by default
The sender scan reads `m.role` off every memcell item, but only
ChatMessage carries it: ToolCallRequest has `sender_id` and no `role`,
ToolCallResult has neither. So any memcell holding a tool call raises
AttributeError before the first sender resolves -- correct on plain user
chat, guaranteed to fail on agent trajectories, where it burns its
max_retries budget and dead-letters on output nothing consumes today.
Flipped the decorator rather than `default_ome.toml`, because `everos
init` skips an existing `~/.everos/ome.toml` (init_cmd.py:85), so a
template edit would reach new installs only. The toml opt-in is left
working on purpose -- a chat-only deployment does get correct
foresights -- and documented in both the module docstring and the
template comment.
This is a stop-gap. The fix is per-episode extraction, like
atomic_fact, which needs an everalgo entry point that does not exist
yet.
The one test that used foresight as its UserPipelineStarted subscriber
now opts back in through that same toml key, so the opt-in path is
covered rather than worked around. It has to wait for the override to
reach the registry first: ConfigReloader.start() fires its initial load
as a task, so engine.start() returns before ome.toml is applied, and an
emit inside that window is judged against the coded defaults and
dropped by the enabled gate with no redelivery.
* fix(ome): hold engine_sem per attempt, not across the retry chain
The backoff this PR added slept inside the semaphore block, so a run
waiting to retry kept its concurrency slot. That turns a partial outage
into a total stall: with max_concurrent_runs slots and a 1s/2s/4s
backoff, enough simultaneously-failing runs park every slot in
asyncio.sleep and starve strategies that would have succeeded.
The cap exists to bound concurrent strategy work -- LLM calls,
embeddings, storage IO -- and a sleeping coroutine consumes none of it.
Backpressure on the failing work is intended; backpressure on everything
else is not. Semantics change is deliberate and stated in the docstring:
the cap still applies to execution, no longer to waiting.
The guard uses a single-permit semaphore so locked() is unambiguous, and
asserts a second waiter actually acquires -- locked() alone would pass on
an implementation that freed the slot but left waiters unable to take it.
Verified red against the previous structure.
* fix(strategies): reap the directory a renamed skill leaves behind
everalgo treats a name change as a first-class update: _apply_update
preserves prior.id while swapping the name, so _persist_skill wrote the
skill to a new skill_<new_name>/ and the old directory survived with the
same cluster_id.
That is not a cosmetic leak now. Since existing skills are read from
markdown rather than LanceDB, the orphan returns in the next run's
existing_relevant_skills as a duplicate of a skill the LLM already
renamed -- feeding exactly the add-instead-of-update full-replace clobber
this PR set out to close, once more per rename. Reconciliation keys off
skill.id, the only field that survives a rename (a fresh add mints a
uuid4 and can never match), and never deletes a name another emitted
skill just claimed.
Also in this pass:
- AgentSkillWriter.delete_skill, the one destructive operation here. It
fails closed: a directory it cannot resolve by the writer's own path
rule is left alone rather than targeted by anything looser.
- reference_name / script_filename now sanitized on both reader and
writer. They are appended after the skill_<name> segment, so
skill_dir_name never covered them. Zero callers in src/ today; closing
it before progressive disclosure wires them up.
- Agentic case rows with every passage field empty fall back to a
placeholder instead of raising ValueError in everalgo's _format_docs
and 500ing a whole search the row merely appears in.
- The retire op is documented as unimplemented rather than left implied.
aextract returns a flat list with no discriminator, so a retirement
arrives as an ordinary low-confidence skill and is written back like
any other. Honouring it means either giving an LLM confidence score
authority to delete the source of truth, or a retired flag that the
enumeration, cascade, and search all learn to filter on -- a design
decision, deferred.
- The embedding body-guard comment no longer claims to protect a local
embed call; this strategy stopped embedding when the vector moved onto
the event.
* fix(strategies): stop extract_foresight crashing on tool-call memcells
The sender scan read m.role off every memcell item, but only ChatMessage
carries it: ToolCallRequest has sender_id without it, ToolCallResult has
neither. The first tool call raised AttributeError before any sender was
resolved, so the strategy was correct on plain user chat and
dead-lettered every time on agent trajectories.
everalgo contracts for exactly this input -- user_memory/_render
.chat_messages says "the caller need not pre-filter; an
AgentMemCell-shaped MemCell is acceptable input" -- and every other
user-memory extractor gets that for free by delegating. This strategy was
the one place the filter was hand-rolled, and it was hand-rolled wrong.
It stays disabled by default, but for the correct reason: nothing in
EverOS reads foresights yet, so running it spends one LLM call per sender
per memcell on write-only data. The earlier justification (per-episode
extraction needs an everalgo entry point that does not exist) confused
extraction granularity with the crash; granularity is still open, the
crash was one line. Fixing it is what makes the documented ome.toml
opt-in actually usable.
The guard pins both directions: a pure agent trajectory extracts nothing
and never reaches the LLM, and a mixed memcell extracts for human senders
only -- an implementation that stopped raising but scanned tool-call
sender_id values would invent "agent" as a user.
CHANGELOG also records that the stale-index clobber is fully closed only
for clusters at or below MAX_SKILLS_IN_PROMPT; above it LanceDB orders
the markdown candidates, and the skill a lagging index omits is the one
written most recently.
* docs(changelog): fold the merged cascade work into the 1.2.3 entry
#392 landed on main with its entries under [Unreleased] and no version
bump. Since 1.2.3 ships that code, leaving them there would have the
release notes disclaim work the release contains. Merged section by
section into [1.2.3] and dated it to the actual release day.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(lancedb): bound reads and unhook the husk sweep from the write lock
Two instances of the same defect class the write-lock deadline work left
behind: an await in a scheduler-gated path with nothing bounding it.
Reads (count / get_by_id / find_where / find_where_paginated / search) were
skipped last round on the reasoning that a read takes no lock and so blocks
no writer. True, but incomplete: the cascade drain loop reads on every batch
and advances strictly one batch at a time, so a read that never returns stops
the whole md -> LanceDB projection. Claimed rows stay `processing` forever
(claim_pending_batch only takes `pending`, orphan recovery runs once at
startup), and /health keeps reporting healthy because a hang raises nothing.
Budget 60s, ~1000x the measured 62ms flat scan over 117k rows.
The empty-index-dir sweep ran inside the prune critical section under a
docstring contract requiring the write lock. That contract could not hold: the
sweep runs via asyncio.to_thread, and a deadline cancels the future, not the
thread, so an orphan sweep outlives the lock -- and Path.iterdir is a lazy
os.scandir, so it can yield a dir created after the scan began. It could
therefore rmdir a directory a concurrent create_index had just made, leaving
the table with no FTS index and every search on that kind 500ing. Safety now
comes from an age filter (skip dirs younger than 300s), which holds regardless
of lock ownership; the sweep moved out of the critical section so a slow
filesystem walk can no longer overrun the prune budget.
Mutation-verified: moving the table handle back outside the read deadline
hangs the new test; moving the sweep back inside the lock fails it; setting
the age floor to 0 fails the fresh-dir test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(cascade): supervise background loops and unmask the optimize alert
The drain / heartbeat / rebuild loops were plain create_task coroutines. One
uncaught exception ended that loop permanently: nothing restarted it, and
because the worker holds a strong reference to the task the interpreter never
printed "Task exception was never retrieved" either (that fires on GC). The
loop's job just stopped happening with zero output. _run_loop had an inner
try; its two siblings did not.
Each loop now runs under _supervise: log, wait, restart with escalating
backoff (5s / 15s / 45s), then request process exit via SIGTERM so a
restarting supervisor (systemd Restart=always, Docker restart:
unless-stopped, a k8s Deployment) can recover it. SIGTERM rather than
os._exit so the ASGI server runs its graceful-shutdown path. A done-callback
is the last-resort observer for the supervisor itself ending unexpectedly.
Separately, the fallback rebuild reset the same counter the health verdict
reads, so the optimize-failure threshold was effectively unreachable: a table
failing 100% of the time cycled 1..5 -> 0 -> 1.. and the threshold value
existed only during the sub-second rebuild, ~1% observable against a 30s
scrape. cascade.healthy stayed green while the table never reclaimed a
version. The rate limiter moves to failures_since_fallback; only a successful
optimize clears the alert streak. Same shape as the run7 cross-kind max()
masking bug -- a remediation path refreshing the signal meant to report it.
Mutation-verified: dropping the restart budget, removing the exit request,
and restoring the counter reset each turn the corresponding test red.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(persistence): bound and announce the memory-root lock wait
Acquisition polls with LOCK_NB instead of blocking inside a worker thread. A
blocking flock could be neither bounded nor cancelled: cancelling the awaiting
coroutine leaves the thread to acquire the lock later with nobody left to
release it, which is strictly worse than waiting.
The wait itself is correct by design -- the second process is supposed to wait,
then find the migration already done -- and flock is released by the kernel on
process exit, so a dead holder never wedges it. What was wrong is that it had
no upper bound and emitted nothing: a server startup landing on a held lock
looked like a hang whose last log line was lifespan_provider_startup
name=lancedb. It now logs memory_root_lock_waiting on first contention,
reports how long it waited on success, and gives up after timeout_seconds
(default 300s, generous because the legitimate holder is an O(rows) migration).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test(cascade): wait for projection quiescence in the lap-race scenario
test_lap_append_during_handler_no_loss asserted no loss after
_wait_path_done, whose settle window is 0.1s. That is a bet that the
filesystem event for the appends which landed *during* a handler
invocation has already been delivered — a terminal row does not mean the
file is fully projected, because the handler read the md at whatever
length it had then, marked the row done, and the rest arrive on a later
event. The bet holds on macOS/fsevents and lost on a loaded Linux runner
(md=30 lance=17), failing the assertion for a reason unrelated to the
behaviour under test.
Waits for quiescence instead: terminal row + empty pending queue + a
projected count unchanged across three consecutive polls. Strictly
stronger than the old condition, and real loss still fails — the count
just converges below the md entry count and stays there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(lancedb): replace indexes in place and retry a lost rebuild race
rebuild_indexes dropped every index and recreated it. The docstring justified
that with "LanceDB transparently falls back to brute-force scan", which is true
for vector search and false for FTS: with no inverted index a BM25 query raises
"Cannot perform full text search unless an INVERTED index has been created".
The recall legs are gathered without return_exceptions, so one failing leg
fails the whole search request -- every keyword search landing in the window
returned 500. Measured: 55 failures across 3 rebuilds with drop+create, 0 with
create_index(replace=True). Replacing also collapses the live fragment set
identically (7 index files back to 4 after 25 optimize beats), so nothing the
rebuild existed for is lost. Only indexes on columns that are no longer indexed
at all are still dropped -- nothing queries those.
A rebuild that loses the manifest race is now retried rather than deferred to
the next 12h sweep. Lance marks the conflict Retryable and means it: another
process committed first. Retries are recorded as a deadline on the kind
(10min / 30min / 3h) and picked up by the rebuild loop, not slept through --
the loop walks kinds sequentially, so sleeping would park every later kind
behind the backoff (7 kinds x 3h outlasts the cadence itself). The loop tick is
min(60s, cadence) so a shorter configured interval is not quantised.
Removes the empty-index-dir sweep. cleanup_older_than deletes the files under a
superseded _indices/<uuid>/ but leaves the directory, and everos was removing
those with its own rmdir. No LanceDB contract says an empty index dir is
garbage, so this is being raised upstream instead. Note it is a separate gap
from index *files* not being reclaimed under delete_unverified=False (260MB
retained on a 19k-row soak table) -- solving that still leaves the empty dirs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs(changelog): put only the sweep under Removed
The rebuild-bound and lock-wait entries were swallowed into the Removed
section when the husk-sweep entry was inserted above them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(lancedb): bound the husk sweep by lance's own unverified threshold
Restores the empty-index-dir sweep with a safety argument instead of a
self-chosen number. Upstream context, read from lance's cleanup.rs: it unlinks
a superseded index's files but never the directory, and contains no rmdir at
all. That is structural, not an oversight -- lance targets object stores, where
paths are flat keys and an empty directory does not exist. Only a local
filesystem materialises them, where they accumulate as inodes (a soak run
reached 13061 dirs, 98% empty) and slow every directory scan.
Three independent guarantees, in order of strength:
1. rmdir cannot delete data. The kernel refuses it on a non-empty directory,
so no file can be lost whatever the rest of the logic decides -- and because
the check *is* the operation, there is no check-then-act window to race.
2. Live indexes are excluded by UUID, read from list_indices().
3. Anything else must outlive UNVERIFIED_THRESHOLD_DAYS = 7, which is lance's
own bound for deciding an unreferenced index UUID is dead rather than an
index build in progress. Matching it means the sweep can never be more
aggressive than lance itself; the previous 300s was our invention, and that
is what made it indefensible.
Each guarantee is pinned by the same test and mutation-verified: dropping the
live-UUID check, zeroing the age gate, and swapping rmdir for a recursive
delete each turn it red.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat(cascade): make the maintenance cadences configurable
The four cadences were already constructor arguments on CascadeWorker, but
CascadeConfig did not carry them and none of the three production construction
paths passed a config, so the module defaults were unreachable from outside the
code. That is why no soak run shorter than half a day could exercise the 12h
rebuild sweep: not a missing parameter, a config layer that dropped it.
Adds CascadeSettings ([cascade] in default.toml) and CascadeConfig.from_settings,
which the orchestrator now uses when no config is passed -- so the CLI, backfill
and server paths all pick settings up at once.
Deliberately not exposed: the read / write / prune / rebuild deadlines. Those
are hang-catchers sized from measured durations, and both directions are worse
-- too low manufactures failures on a healthy table, too high leaves a wedged
one invisible for longer. Cadences depend on write volume and are a real tuning
axis; deadlines are not. A test pins the exposed field set so a later change has
to state its intent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* test(cascade): cover the AgentCase handler
agent_case was the one business kind with no handler test, and the storage
soak never writes it either -- four of the seven tables stay at zero rows
there -- so its md -> row contract was unexercised from both directions.
Covers what makes this kind different from its daily-log siblings: it lives
on the agent track, and it embeds task_intent only while approach is
BM25-indexed but deliberately never sent to the embedder. Plus the branches
every handler shares: soft-dependency embedding (no provider -> vector=None,
row still written for keyword-only deployments), optional KeyInsight, the
content_sha256 short-circuit that stops the 30s scanner re-embedding
untouched files, edit detection, and delete-by-path.
Mutation-verified: routing approach into the embedder, and dropping
section:TaskIntent from content_change_keys, each turn the relevant test red.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs(lancedb): record the empty-dir cost and what caps it
Two constants now carry the arithmetic instead of leaving it in a chat log.
_HUSK_MIN_AGE_SECONDS states what the 7-day gate costs: nothing reclaims an
empty index dir before then, each is an inode plus a 4KB block, and at ceiling
load that is ~890k dirs / ~3.6GB / 14% of a default 98GB ext4's inodes at the
7-day steady state. Also that this is the worst case and needs sustained
saturation -- a single-user deployment sits four orders of magnitude below it --
and that only ext4 has a fixed inode budget (APFS and xfs allocate
dynamically, Windows is out of scope).
DEFAULT_OPTIMIZE_MIN_INTERVAL_SECONDS gets two things it never said. First, it
is not a visibility delay: a row is searchable as soon as its upsert commits,
because LanceDB flat-scans the unindexed tail -- verified to cover BM25, not
just vector and scalar, which was the leg worth doubting given a missing FTS
index hard-fails rather than degrading. Sparse writes do not wait at all, since
the scheduler uses max(0, interval - elapsed). Second, it is the ceiling on
index-directory growth: past roughly one write per table per interval the beats
coalesce, so the accrual rate is capped by this interval rather than by write
volume, and raising it lowers the cost proportionally. That makes it the knob
to reach for if the empty dirs ever bite -- not the husk threshold.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(cascade): reset the loop-restart budget after a stable run
The supervisor's restart budget was spent per process lifetime: three
strikes ever, regardless of how long the loop ran healthily between
them. A loop hitting one recoverable transient every few days — each
cleared by a single restart — would still pool those strikes and
SIGTERM a healthy server weeks in, on the 4th, which punishes exactly
the case supervision exists to absorb.
The budget now counts consecutive quick crashes: a body that ran at
least 60s before raising starts a fresh incident with the full ladder.
A deterministic crash-on-entry still exhausts the budget in ~65s. Same
windowed counting as systemd StartLimitIntervalSec / Erlang
max_restarts-per-max_seconds.
Also corrects the husk accrual numbers in the optimize-cooldown
docstring to the ~14-day effective reclaim horizon (see the sibling
lancedb commit for why the age gate doubles).
Mutation-verified: with the reset removed, the new test exits after
run 3 instead of surviving to run 5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(lancedb): keep a husk-sweep timeout from failing the prune
By the time the sweep runs, the cleanup commit — the thing prune exists
for — has already succeeded, and the sweep is best-effort by contract.
Letting its deadline escape prune() billed the failure to the wrong
account: the optimize scheduler counted a prune failure (feeding the
fallback-rebuild threshold) and the prune-staleness clock stopped
advancing, so both alarms reported a cleanup stall that did not happen.
Same defect shape as the alert counter the fallback rebuild used to
zero: an auxiliary path corrupting the main signal's ledger.
Reachable, not theoretical: sweep time is proportional to dir count
(~35us/dir measured) and the ceiling-load steady state sits right at
the 60s budget. The timeout is tolerable exactly because it is now
swallowed — and the orphaned worker thread finishes the walk anyway, so
the reclamation still happens.
Also corrects the age-gate docstrings: the gate reads st_mtime, which
POSIX bumps when lance's cleanup empties the husk, so the effective
reclaim horizon is file wait + 7 days (~14 days total) and the
ceiling-load steady state is ~1.8M dirs / ~7GB, twice the previously
recorded figure. The "never more aggressive than lance" property is
unaffected (it is strictly more conservative).
Mutation-verified: with the try/except removed, the new test fails on
the escaping VectorStoreBusyError.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(persistence): move the lock timeout an order above the legit hold
300s sat at the edge of what the docstring itself calls a legitimate
hold (a large migration is minutes of O(rows) work), so the worst
honest migration turned every waiting process's startup into a
LockError crash. Now 1800s: the wait has been visible since the first
poll (memory_root_lock_waiting), and against the one case the bound
exists for — a holder alive but wedged — giving up at 5 minutes buys
nothing over 30, because the timeout's job is diagnosis, not recovery.
The timeout message now says which way to look: the kernel releases a
dead holder's flock automatically, so reaching the timeout means the
holder is alive — inspect that process instead of retrying this one.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(changelog): fold the review fixes into the unreleased notes
Supervisor bullet gains the per-incident budget, the sweep bullet gains
the ~14-day effective horizon and the swallowed timeout, and the lock
bullet records the 30min default with its sizing rationale.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(lancedb): pin that only a sweep timeout is absorbed by prune
The sibling test covers one side -- a deadline miss must not bill prune's
ledger. Nothing covered the other: widening the catch to `except Exception`
passes every other test in the file, and would turn a genuine fault in
_remove_empty_index_dirs (a TypeError after a signature change, a permission
error on the index dir) into a silent removed = 0 with no signal anywhere.
That is the failure shape this module keeps being audited for, so the
narrowness of the catch needs its own guard.
Found by mutating the catch rather than by reading it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(cascade): bound the optimize runner's wait on a rebuild
The two maintenance jobs park on each other -- whichever arrives second waits
on the first -- so the two waits are one hazard seen from opposite ends. The
rebuild side was bounded; this side was left open on the argument that
rebuild_indexes carries its own 300s deadline. That deadline covers its
critical section, not the task's dispatch and teardown around it, so the
transitive bound was never real.
While the runner waits, its per-kind task slot stays occupied, every
_schedule_optimize call short-circuits on it, and that table silently stops
being pruned -- the same shape as the stall this branch has been chasing.
Bounded at 180s. On expiry the beat is skipped rather than run: compacting
under a live rebuild is the interleaving the wait exists to prevent, and both
commit on the same manifest. Writes keep the dirty flag set, so the next beat
retries.
Mutation-verified: replacing the timeout with a plain await hangs the new test
until its own guard fires.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: zhanghui <zhanghui@shanda.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Leaving the lead summary and the upgrade notes to be composed at publish
time put the writing at the worst possible moment — weeks after the changes,
with no review. Both belong in the CHANGELOG entry, which the release PR is
already editing.
The lead needed no code: text between the version heading and the first
group already flowed into the page. The Upgrade group did — CI appended its
own `## Upgrade`, so a CHANGELOG that carried upgrade notes produced two
headings. The group is now lifted out and the boilerplate wrapped around it,
pip line above, compare link below, matching every page since 1.1.3.
Publishing is now a read-through and a click.
Co-authored-by: zhanghui <zhanghui@shanda.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The benign-conflict log said a commit race was lost but not by which
maintenance beat, and lance labels both beats' commit identically
("This Rewrite transaction was preempted by concurrent transaction ..."),
so the message alone cannot separate them. The costs differ sharply:
- a lost LIGHT beat is free — compaction retries ~10s later;
- a lost HEAVY beat means that table skipped a whole prune cadence, so
its superseded files stay on disk until the next one lands.
Reading an index-dir growth incident off the logs therefore meant
back-inferring which beats were heavy from the 300s cadence. That was
done once during the storage soak to explain a 13-minute window where
one table's cleanup stalled with no failure logged, and it cost two
35-minute debug-level reruns to confirm — the field makes it a grep.
Adds `pruned` to the conflict log, mirroring the sibling failure log.
Log level (debug) and the benign-conflict semantics are unchanged: the
streak is still not incremented and no fallback rebuild is triggered.
Test asserts both beats in one run: the heavy beat's conflict logs
pruned=True and the light beat's logs pruned=False. Mutation-verified —
dropping the field fails the test with KeyError.
Co-authored-by: zhanghui <zhanghui@shanda.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Elliot Chen <2340896+cyfyifanchen@users.noreply.github.com>
* ci: draft the GitHub Release page from the CHANGELOG on tag
release.yml only published to PyPI; creating the Release page was an
undocumented manual step, and v1.2.2 shipped to PyPI without one.
A second job, gated on a successful upload, now drafts the page: title
"EverOS X.Y.Z", body lifted from the matching CHANGELOG section with the
group headings demoted to h2, `--prerelease` for PEP 440 suffixed tags. It
stays a draft because the lead summary that opens every EverOS release page
is prose CI cannot write.
The job runs separately from the publish job so that one keeps `contents:
read` alongside its OIDC token. A stable tag with no CHANGELOG section fails
the job rather than publishing an empty page; pre-releases fall back to a
placeholder. Existing releases are detected through the list endpoint, since
the by-tag endpoint cannot see drafts (cli/cli#3037) and a re-run would 422 —
a stale draft is replaced, a published release left alone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* ci: prefill the Upgrade section on the drafted release page
Every EverOS release page since 1.1.3 closes with an Upgrade section — the
pip line, any migration notes, and a compare link to the previous tag. The
first draft dropped it, so a published page would have lost the one section
readers act on.
The pip line and the compare link are prefilled; the previous tag comes from
`git tag --sort=-v:refname`, which needs the full tag list, hence
fetch-depth: 0. Migration notes stay hand-written next to the lead summary,
since only a human knows whether a release needs them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs(release): warn that API publishing drops the tag
Flipping `draft` through the API without `tag_name` rebinds the release to
the `untagged-<hash>` placeholder and creates a git tag by that name against
the default branch. Hit while publishing 1.2.2; the web UI button is
unaffected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: zhanghui <zhanghui@shanda.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* ci: enforce the file-size ceiling on pull requests
`check-added-large-files` only ran in pre-commit, so the ceiling was absent
from CI: an unhooked clone or `--no-verify` bypassed it entirely, and the
hook is weaker than it looks even locally — it inspects only files being
*added*, so an existing fixture that grows never trips it.
Add `scripts/check_file_sizes.py`, wired into `make lint` and therefore the
required `lint` check. It diffs against the base branch's merge base and
measures additions, modifications and renames, leaving files already
committed above the ceiling alone so no pull request fails for something it
did not touch. An unresolvable base is a hard failure rather than a silent
pass. The `lint` job now checks out with `fetch-depth: 0` to provide it.
Lower the hook's `--maxkb` from 1024 to 640 to match, and pin the two limits
equal in a unit test so they cannot drift back apart.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* ci: catch untracked oversized files in the size gate
`git diff` cannot see a newly created file until it is staged, so a local
`make check-file-sizes` passed a brand-new 700 KB file — verified against the
real script, not reasoned about. CI was unaffected (its checkout has
everything committed), but the docstring claimed the local run covered
uncommitted work, which was only true for edits to already-tracked files.
Union in `git ls-files --others --exclude-standard`, which respects
.gitignore, and pin both the untracked and the ignored case in tests.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* ci: exempt the generated search-seed corpora from the size ceiling
tests/fixtures/search_seed/ holds embedded corpora regenerated by
_dump_search_seed.py. Two of them are already near 1 MB and grew ~60% in one
release, so the next refresh would have hit the 640 KB ceiling and the
cheapest fix would have been raising it for the whole repository — a gate that
teaches people to edit the gate.
Exempt that one directory by path prefix instead. Outside it the largest
tracked file is ~300 KB, so 640 KB still binds where it matters, including the
examples/ case that prompted this work. Three tests pin the carve-out: a
sibling of the exempt directory is still caught, the list itself is asserted
verbatim so growth shows up in review, and every prefix must name a directory
that actually exists.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: zhanghui <zhanghui@shanda.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(lancedb): put table-handle resolution inside the deadline
run9 reproduced the stall the previous commits were supposed to close, on a
different table: episode went 13 minutes without a prune — versions climbing
63→66 while foresight and atomic_fact both collapsed to 1 — with **zero**
failure, timeout, or conflict logs. Its last successful prune was logged at
11:41:59 and the staleness clock matched to the second.
The deadline covered the critical section but not the await ahead of it:
`table = await self._table()` sat outside `_locked`, so a hang while resolving
the table handle never returned. The scheduler runs one maintenance task per
kind and skips a kind whose task is still in flight, so that kind stops being
maintained permanently, silently, because nothing failed.
Move the handle resolution inside the deadline for all seven locked operations,
and give the lock-free compaction beat its own `_deadline` (it takes no lock so
it cannot block writers, but it can still park a kind by never returning).
Belt and braces in the scheduler: both beats now run under
`_MAINTENANCE_TASK_TIMEOUT_SECONDS`, a last-resort bound on the whole call, so
any await I have not thought of costs one cadence rather than forever.
Regression test: a repo whose `_table_lookup` never resolves must make prune,
optimize and add all raise `VectorStoreBusyError` and leave the lock free.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore(release): v1.2.2
Bump to 1.2.2 and cut the changelog. This release is the storage-layer
reliability work: LanceDB maintenance split into lock-free compaction and
write-locked reclamation (fixing unbounded index growth), every write-lock
critical section bounded by a deadline that covers acquisition, a per-kind
prune-staleness signal on GET /health, `everos cascade rebuild` for a drifted
or corrupt index, startup detection of column type drift, and a query-vector
width check that fails fast instead of 13s deep inside LanceDB.
Carries the table-handle deadline fix (previously #385) rather than shipping
1.2.2 with a known stall: the deadline covered the critical section but not the
await ahead of it, so a hang while resolving a table handle parked that kind's
maintenance permanently and silently. Found by a 1h high-rate soak run after
#384 merged.
No migration, no config change, no API change: `docs/openapi.json` differs only
in the version string. The only operator-visible requirement is that
`everos cascade rebuild` now refuses to run while a server holds the OME lock.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: zhanghui <zhanghui@shanda.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(cascade): per-kind prune staleness + rebuild safety
Adversarial review of the review-response fixes found five real defects,
all in code this PR introduced.
Health signal (P1): prune staleness reported the time since the NEWEST
successful prune across kinds, so on a multi-kind deployment (every real
one) a single kind whose cleanup died was masked by the others pruning
on schedule — /health stayed green while that table's index dir grew
unbounded, the exact incident the signal exists to catch. Report the
WORST kind instead and name it in the reason. The failure streak could
not cover this either: an intervening light beat resets it, so it never
reaches the threshold for a prune-only failure. Documented that split of
duties.
Spurious fallback rebuilds (P1): the benign-conflict carve-out excluded
the heavy beat, justified by "runs under the write lock, so it can't hit
this benignly" — but that lock is in-process only, so a second process
(a long `cascade backfill`, a `cascade sync`) preempts prune's Rewrite
commit. Those counted as real failures, and ~25min of cross-process
churn reached the threshold and fired a fallback rebuild, which drops
every index before recreating it; a rebuild that also lost the race was
swallowed as a warning, leaving the table with no FTS index (every
/search on that kind 500s) until the next 12h sweep. Treat commit
conflicts as benign on both beats and let prune-staleness detect a prune
that genuinely stops succeeding.
cascade rebuild (P1 ×2 + P2): it drops and recreates tables with no
guard while --help/docstrings advertised it as safe, so `rebuild --yes`
against a live daemon corrupted the rebuild (the daemon keeps writing
through cached handles). Refuse when the OME jobstore lock is held,
reusing backfill's detection and its exit code 3. It also ran the
pre-drop migration pass (`ensure_business_indexes`) against the damaged
table, so on the corruption classes it exists to repair (missing column,
un-alterable type) the recovery path died on the damage itself — skip it
via `_runtime(ensure=False)`. Reset the queue BEFORE dropping so every
crash window converges on "queue pending → re-index" instead of empty
tables with a fully-done queue (a silently empty deployment), and handle
Ctrl-C with exit 130 plus a resume hint.
Recovery guidance (P1): the nullable-vector migration error still told
users to wipe the index directory — which this PR's own runbook documents
as the wrong recovery (queue stays done, index comes back empty). Point
it at `everos cascade rebuild`. Dropped the schema-drift error's
"restart first" step too: the startup migrations only alter nullability,
never a name or type, so a name/type drift never self-heals.
Also: backfill's post-write prune passed a zero retention window from a
separate process, able to delete files under a daemon /search still
holding that version — pass the daemon's window instead. Runbook gains
the /health cascade block (thresholds, what flips healthy, why
failed_permanent does not) and its quoted schema-drift error now matches
the code.
Tests: the three safety mechanisms this PR adds were unpinned — a
one-line revert of any of them passed the suite. Added per-kind
staleness, heavy-beat benign conflict, benign-filter negative case
(an error whose message merely contains "retryable" must still count),
prune recurrence across light beats (mutation-verified: hoisting the
attempt-clock advance out of the heavy branch turns it red), the prune
timeout releasing the write lock, timeout-below-cadence, the rebuild
server guard, and a tier3 assertion that the /health cascade block is
actually wired. Froze the last fabricated-monotonic test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Also: treat ``last_prune_attempt_at == 0.0`` as "never attempted" instead of
comparing clocks. ``monotonic()`` is boot-relative, so ``now - 0 >= cadence``
is false for the first ~cadence of container uptime — the catch-up prune was
skipped exactly when a fresh process most needs it (and made a test depend on
the runner uptime, which CI caught).
* fix(lancedb): bound every write-lock critical section
run7 (1h at 2.5x rate, concurrent CLI maintenance, doubled fuzz) reproduced a
table whose version cleanup stopped permanently: 150 versions retained, disk
11x live size, while the other two tables sat at 1 version each — and with no
error logged anywhere, because nothing failed. It simply never returned.
Three things combined. The maintenance scheduler allows one task per table (a
LanceDB table takes one writer), so it skips a kind whose task is still in
flight. The prune timeout sat *inside* the lock and covered only the cleanup
call. And the other six critical sections on that lock — add, upsert, update,
delete, delete_by_md_path, rebuild_indexes — had no deadline at all. So one
operation stuck anywhere outside that narrow window wedged the table for good:
every writer blocked on acquire, and every later heartbeat was turned away
because the stuck task never finished.
Make it structurally impossible instead of patching prune: all seven sections
now go through `LanceRepoBase._locked(budget, op)`, where the deadline covers
**acquisition and the body**. No path can wait for this lock, or hold it,
indefinitely. Budgets are hang-catchers, not throughput limits: 120s for row
writes, 600s for an index rebuild, the existing 60s for prune.
Expiry raises `VectorStoreBusyError`, deliberately under `ExternalServiceError`
so the cascade worker retries the row; under `VectorStoreError` a transient
lock contention would be marked permanently failed and need a manual
`cascade fix`.
Tests: a stuck holder now makes a waiter fail its deadline and release (the
lock is reusable afterwards), and the prune timeout is pinned as retryable.
Verified by mutation — moving the timeout back inside the lock makes a waiter
block until the enclosing observation window expires (1001ms vs 51ms), i.e.
wait forever in production.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(lancedb): size write-lock budgets from measurements
120s for a row write was a guess, and a bad one: the budget doubles as the
detection latency for a wedged table, so an over-slack value means minutes of
blocked writers before anything surfaces — the failure this change exists to
prevent.
Measured the four locked write ops on a local SSD across table sizes and batch
sizes (10k-100k rows, 50-500 rows per call): add 3-22ms, upsert (merge_insert,
the read-modify-write one) 6-25ms, update 2-4ms, delete 2-3ms; worst observation
63ms, and flat in both dimensions since these are append-and-commit, not scans.
So: writes 120s -> 15s (~240x the worst observation, enough for a contended disk
and several waiters queued ahead — the deadline includes acquisition and
asyncio.Lock is FIFO), rebuild 600s -> 300s (still the one genuinely slow
section at ~0.3s per 50k rows per indexed column). Prune stays 60s.
Test pins the sizing intent: writes stay in the tens of seconds, and
rebuild > prune > write so the slowest section is not the most eagerly killed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(lancedb): record wait/hold time on write-lock critical sections
A soak run stalled one table's writes for ~16s and the logs could not say why:
maintenance beats only log at `debug`, so a section that is slow but still
inside its deadline is invisible, and the timeout warning did not distinguish
"never acquired the lock" from "acquired it and overran".
`_locked` now carries that apart. The deadline warning gains `acquired`,
`waited_seconds` and `held_seconds` — `acquired` alone answers whether a holder
was slow or this operation was — and a completed section that held the lock for
at least a second logs `lancedb_write_lock_slow_hold` at info, so a stall that
never reaches a deadline still leaves a trace.
Uses `time.monotonic` (elapsed measurement, not wall clock — the datetime
discipline bans `time.time`).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(search): reject a mismatched query vector before it reaches LanceDB
A soak run showed every slow search was a failing search: `search:vector` p50
251ms / p99 1.6s, but its 8 requests over 10s were exactly its 8 failures
(13-14s each). The cause was a query vector whose width disagreed with the
index. LanceDB only notices after the query is built and reports it as an
opaque `ValueError: Invalid input, No vector column found to match…`, which
escaped as an unhandled 500.
Validate at `_embed_query` — the single point every query vector passes
through — against the provider's declared `dim`. Microseconds instead of 13s,
and a named `ConfigurationError` (500 + CONFIGURATION_ERROR) instead of an
unhandled crash. Deliberately not `InvalidInputError`/422: callers only send
query *text*, so a bad width is our provider's fault, not the caller's.
Also cap traceback rendering. structlog's default is
`RichTracebackFormatter(show_locals=True, max_frames=100, extra_lines=3)`,
which on an async stack rendered 82 frames into 6423 log lines per exception —
85MB of server.log across 11 of them — at ~290ms of synchronous CPU each, and
risks printing request payloads into logs. With locals off and 15 frames the
same traceback is 103 lines and 10ms.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(changelog): record the storage-reliability work under Unreleased
#379 merged without changelog entries, so this covers both it and the
follow-up work in this branch: the maintenance split (compaction vs
reclamation) that fixes unbounded index growth, bounded write-lock critical
sections, the /health cascade readiness block and its alert contract,
`cascade rebuild`, schema type-drift detection, the query-vector width check,
and the traceback-rendering cap.
Each entry states the operator-visible consequence, not just the change —
`cascade rebuild` now refusing to run against a live server, benign-conflict
warnings dropping in volume, and `/health` being able to report a stalled kind
that was previously invisible are all behaviour changes someone will notice.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: zhanghui <zhanghui@shanda.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(lancedb): reclaim stale versions via write-locked prune
The storage soak (48h, sustained churn + fuzz) proved the bundled
lock-free `optimize(cleanup_older_than=...)` loses its commit-conflict
race against concurrent writes — cleanup ran only 16 of ~250 times, so
old dataset versions / FTS orphans piled up and the index dir grew to
the 40G disk guardrail and never reclaimed under load. main still had
that bundled call.
Split the maintenance path:
- `LanceRepoBase.optimize()` is now compact-only and lock-free (a commit
conflict here is benign — the next beat retries, so it must not stall
writers).
- `LanceRepoBase.prune(older_than)` runs `cleanup_older_than +
delete_unverified=True` **under the per-table write lock**, so no
writer is in flight: the Rewrite has the manifest to itself (cleanup
completes every beat) and aggressive deletion is safe. It also removes
the empty `_indices/<uuid>/` husks cleanup leaves behind (soak: 13061
dirs, 98% empty), offloaded to a thread.
- The cascade worker's heavy beat calls `prune()`; the light beat calls
`optimize()`. A benign light-beat commit conflict is logged at debug
and does not count toward the failure streak or trigger a rebuild.
- Prune's retention window (`cleanup_older_than`) is decoupled from the
prune cadence and defaulted short (60s). It runs under the write lock,
so the window only needs to outlive an in-flight read; keeping it =
cadence (300s) left ~2 cadences of superseded full-table copies on
disk between beats (soak: transient ~15G/table peaks). 60s reclaims
all but the last minute each beat — same live floor (~625MB/table),
far lower transient footprint.
Result on the re-run soak: disk sawtooths and reclaims to live-data size
(~1.3G total) under active load — vs run1 stuck at 40G until writes
stopped — with 0 crashes / 0 OOM / 0 stuck cleanups over 48h.
Cascade projection health is now observable:
- `CascadeOrchestrator.health()` -> `CascadeHealth`, combining the
worker's in-memory signals (drain-loop failures, unrecoverable count,
optimize streak, prune staleness) with the SQLite queue summary.
- `GET /health` gains a typed `cascade` readiness block. `healthy`
reflects operational health only (drain / optimize / prune);
`failed_permanent` (files awaiting `cascade fix`) is a data-quality
backlog reported as an informational count that does NOT flip
`healthy` — otherwise the signal would sit red permanently.
The scanner-side retry cap and the `_MAX_TOTAL_RETRIES` budget already
on main handle re-enqueue storms, so no duplicate is added here.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(deps): pin lancedb to >=0.34.0,<0.35.0
The previous open-ended `>=0.13.0` let any environment float to an
untested release, including 0.32-0.34 which carry a compaction
offset-overflow regression (lance-format/lance#7653) that stalls
version cleanup and grows the index dir without bound.
- Floor 0.34.0: the current resolved version; runs safely thanks to
the with_position=False FTS workaround shipped in #336. Verified that
data written by lancedb 0.32.0 (lance v6) reads correctly under
0.34.0 (lance v8), so existing deployments upgrade cleanly. Never
widen the floor below 0.34 -- older lance cannot read v8-format data.
- Ceiling <0.35.0: 0.35 embeds lance-rust v9 (large encoding jump, not
yet stable-released); it must pass the soak harness before we allow
it.
Resolved version is unchanged (still 0.34.0); this only tightens the
declared constraint.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(cherry picked from commit 94f9aa67d11a07c93c7d59d6b134e80858b60316)
* fix(search): bridge agent-kind metadata into the agentic doc contract
The agent AGENTIC path (`agent_case` / `agent_skill`) fed recall
candidates straight into `aagentic_retrieve`, whose `_format_docs`
(LLM sufficiency / multi-query prompt) reads `metadata["episode"]` as a
`{subject, content}` dict plus a ms-epoch `timestamp`. Agent-kind rows
carry their body in the recaller's `text_field` and time as a datetime,
so `_format_docs` raised `TypeError: Candidate ... has no episode dict`
and `POST /api/v*/memory/search` returned 500 for any
`owner_type=agent` + `method=agentic` request.
Mirror the episode path's bridge: reshape agent candidate metadata into
the everalgo doc contract before `aagentic_retrieve`, and revert it
before DTO shaping so the agent shapers still see a datetime timestamp.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit bb9a3a585e044643d3852b9d2810b42c05a7a47c)
* test(search): regenerate search seed to current linkage + migrate e2e
The committed `search_seed` fixture and several search e2e tests were
written for the pre-1.5 memcell fact-linkage model. Current extraction
links atomic_facts to episodes via `parent_id == episode.entry_id`
(parent_type="episode"), and user_memory clusters store episode
entry_id members — so the stale fixture made VECTOR/AGENTIC recall and
the cluster-narrowing path find nothing, and stale assertions checked
an old error code.
- Regenerate `search_seed/*` from a fresh corpus in the current
entry_id format; facts now bridge across multiple episodes (richer
agentic / hierarchical-eviction coverage).
- Fix `_dump_search_seed.py` sampling: pick episodes that host facts
first and keep facts by episode entry_id, so re-dumps stay coherent.
- Migrate e2e tests to the entry_id model (hierarchical-eviction,
session/timestamp filters, cluster seeding helper) and update the
filter-error assertion to the current `INVALID_INPUT` code.
- Provision `ome.toml` in the full-app pipeline fixture (the OME config
reloader requires it; strategies are code-registered so the packaged
default suffices), unblocking corpus regeneration.
Full search e2e suite now green (49/49).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit 44cd9cecc07775cb9270104a5804f36e32cd788d)
* fix(lancedb): detect schema type drift and add cascade rebuild recovery
verify_business_schemas only compared column names, so a column whose
on-disk Arrow type had drifted (name unchanged) slipped through and
detonated later inside merge_insert as an opaque LanceError(IO)
"Spill has sent an error" (#337). Now compare each shared column's Arrow
type against schema.to_arrow_schema() — the exact schema get_table
builds tables from, so a healthy table never false-positives.
Reproduced #337 byte-identically: an episode.subject_vector column left
as string or null by an older build, plus a real 1024-d vector on
upsert, yields the exact crash. No lancedb version (0.13-0.34) renders
Optional[Vector] as a non-vector type, so the startup guard is what
should catch it — not the runtime.
Add `everos cascade rebuild` as the safe recovery: it drops the business
LanceDB tables and re-indexes from markdown, skipping the verify guard
(which the drift would otherwise trip on startup). Unlike removing only
.index/lancedb it re-populates already-done entries (reset_all clears
the cascade queue); unlike removing all of .index it preserves
unprocessed_buffer (messages not yet extracted).
Fixes#337.
* docs(cascade): document cascade rebuild and correct recovery guidance
Add the `everos cascade rebuild` command to the runbook, CLI, and
how-memory-works docs. Correct the old recovery guidance: a bare
`rm -rf .index/lancedb` leaves md_change_state marked `done`, so the
scanner skips those files and the index comes back empty — the runbook
previously claimed a full repopulation that does not happen. `cascade
rebuild` is the safe path (re-populates done entries, preserves
unprocessed_buffer). Also document that verify now checks column types.
* chore(rebase): adapt #354 integration test to soft-embedding main
CascadeOrchestrator dropped the embedder param when embedding became a
soft dependency (main); fold the schema-drift integration test onto it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(cascade): freeze monotonic clock in prune-staleness health tests
Both prune-staleness health tests fabricated
`_started_at = time.monotonic() - (ALERT + 100)`, assuming monotonic()
is a large value. On a fresh CI runner monotonic() is only ~100-180s, so
the subtraction went negative, the source clamped the baseline to 0, and
staleness read back as ~130s < 900s — failing on CI while passing on
long-lived dev boxes where monotonic() is huge.
Freeze the monotonic clock via monkeypatch so staleness is deterministic
regardless of the runner's boot uptime. Source logic is unchanged; only
the tests are made hermetic.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(cascade): wait for stable terminal state in rename scenarios
`_wait_path_done` reached a terminal status, slept a 0.1s settle window,
then asserted the status was still terminal — which contradicted its own
docstring ("absorb any last-second re-enqueue"). A rename's delete event
or an atomic-replace echo can flip a done row back to `processing` inside
that window, so on a slow CI runner the assert fired
("flipped back to processing after reaching done"), failing
test_rename_cross_owner_keeps_frontmatter_owner intermittently (seen on
the 3.13 job). `make integration` runs without `--reruns`, so a single
flake fails the whole job.
Wait for a terminal state that *survives* the settle window instead:
absorb a transient re-enqueue by waiting for terminal again, still bounded
by `deadline` so a row that never settles surfaces as a timeout. Pre-
existing flake on main, unrelated to the prune change; the scenario's real
assertions (row counts, frontmatter owner) are untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(cascade): cross-process prune safety + backfill reclaim fix
Review of #379 found two P0s plus P1/P2s, all verified against the code:
- P0-1: cascade backfill still called the removed
optimize(cleanup_older_than=…) kwarg → TypeError swallowed by the
best-effort try/except → backfill silently skipped all compaction +
reclaim (the exact bloat this PR fixes). CI stayed green because the test
double kept the stale signature. Fix: call optimize() + prune(0) at the
call site; make the fake mirror the real signature so the drift can't hide
again; pin prune in the backfill tests.
- P0-2: prune ran delete_unverified=True guarded only by an in-process
asyncio lock, but the runbook promises `cascade sync` is safe alongside a
live server — and the CLI's first optimize beat does prune, in a separate
process. It could delete files the daemon is mid-commit on. Fix: switch
prune to delete_unverified=False. Measured to reclaim identically on
churned tables (both collapse superseded versions ~97%); True only
additionally deletes in-flight/dangling files — exactly the corruption
vector. No cross-process lock needed; the write-lock/commit fix (the real
reclaim win) is unchanged.
- P1-3: /health called orch.health() (6 SQLite aggregates) with no guard →
a locked/full/migrating DB would 500 the liveness probe and restart the
container. Wrap it: unhealthy readiness + reason, HTTP stays 200.
- P1-4: rebuild drops + recreates tables; a live daemon holds cached handles
pointing at the dropped dataset. Runbook now says stop the server first —
the one cascade command unsafe alongside a live server.
- P2: narrow _is_benign_commit_conflict to the "commit conflict" phrase (a
bare "retryable" swallowed unrelated recoverable errors); add a timeout
around the prune cleanup so a hung lance call can't wedge the write lock;
correct two stale docstrings.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs: fix schema-recovery guidance + drop dead internal doc refs
Follow-up to the review; two doc issues neither the review nor the fix
commit caught:
- The schema-drift startup error's docstrings (verify_business_schemas
and LanceDBLifespanProvider) still described the recovery as
`rm -rf ~/.everos/.index/lancedb` — which the runbook explicitly calls
the WRONG recovery (it leaves the cascade queue `done`, so nothing
re-indexes and the index comes back empty). The raised error already
points to `everos cascade rebuild`; align the docstrings to match.
- 15 dangling references to an internal numbered design-doc set
(12_/13_/16_/17_*.md) that was never shipped to this repo. Point the
schema-recovery ones at docs/cascade_runbook.md; drop the rest (pure
provenance in table/component docstrings) while keeping the substance.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs: align maintenance docstrings with split optimize/prune API
Two docstring residuals from the #379 review's P2 list:
- _run_optimize_once still described the pre-split bundled heavy beat
("same work plus cleanup_older_than ... older than one cadence");
the heavy beat now calls prune() under the write lock and the
retention window is decoupled from the cadence
(DEFAULT_OPTIMIZE_PRUNE_RETENTION_SECONDS).
- _restore_shaper_metadata converts any numeric timestamp, wider than
an exact inverse of the bridge; document that this is deliberate
(the shaper contract requires a datetime either way).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* style(e2e): hoist fixture-body imports to conftest module top
shutil / importlib.resources.files were imported inside the
core_pipeline_runtime fixture body; move them to the module top to
match the repo import style (#379 review P2).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(cascade): back off a hung prune via a separate attempt clock (N1)
The write-lock timeout on prune (review P2) didn't achieve its goal: it
was 300s — equal to the prune cadence — and `last_prune_at` only advanced
on success. So a hung lance cleanup timed out after 300s, `should_prune`
was still true (clock never moved), and the next beat re-pruned ~10s
later — pinning the per-table write lock ~97% of the time, the exact
write-starvation the timeout was meant to prevent.
A real cleanup is milliseconds even on a heavily churned table (measured
~40ms at 320k writes / 100 versions), so the timeout is a pure hang-catcher:
lower it to 60s (~1500x headroom, never fires normally, well below the 300s
cadence).
Split the prune clock so a failed prune backs off without masking the
health signal:
- last_prune_attempt_at (new) gates scheduling, advanced before the call
whether it succeeds or times out → a hung prune waits a full cadence
before retrying (light lock-free compaction runs meanwhile), so the lock
is held at most ~timeout/cadence ≈ 17% in the worst case.
- last_prune_at advances only on success and still drives the
prune-staleness health signal, so a persistently failing prune surfaces
as degraded instead of being hidden by an advanced schedule clock.
Regression test: a raising prune advances the attempt clock (next beat is
light, no immediate re-prune) but not the success clock.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: zhanghui <zhanghui@shanda.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Both were pinned to 0.3.x; PyPI 0.4.0 published 2026-07-30. Local
make ci is green (lint + 180 integration tests + package smoke).
Held on a branch until the next EverOS release picks it up.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The supported-versions section still said "released and at v1.0.0" and
listed all of `1.x` as supported, which had drifted two minor releases.
It is now stated as a line ("1.2.x (current)") rather than a pinned
patch number, so it does not go stale on the next release.
Also:
- Point readers at /security/advisories, and tell them to read the
affected range on the advisory rather than comparing version numbers.
- Add a threat-model bullet for ingested documents. The loopback-only
default covers who can call the API, but filenames, metadata, and
content arriving from elsewhere are untrusted regardless of how the
API is reached.
- Note that reporters are credited in the advisory as well as the
release notes, which is what we do in practice.
Reporting stays email-only; GitHub private vulnerability reporting is
deliberately left disabled so the documented 5-business-day response
runs through one monitored inbox.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The 1.1.4 section now describes the code shipped as everos==1.1.4 on
PyPI, which lists the path traversal as fixed. Read top-down, that left
a 1.2.0 user concluding the fix predates their version and they are
safe — the opposite of the truth, since 1.2.0 was built from a branch
that never received those fixes.
Three changes, no history rewritten:
- The 1.2.1 Fixed entry now names what 1.2.0 regressed, instead of
describing the merge as internal branch hygiene.
- A Security section under 1.2.1 carries the affected-version range,
which is discontinuous: everything before 1.1.4, plus 1.2.0. 1.1.4
itself is not affected.
- A note under 1.2.0 points forward, so a reader who looks up their own
version rather than the newest one still finds it.
Matches the published v1.2.1 release notes.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kendrick-Song <61358070+Kendrick-Song@users.noreply.github.com>
* feat(examples): add zero-install Langfuse trace replay
Native OTel moved span emission into the server, so the Langfuse example
lost its try-before-install path: seeing anything now required a
configured EverOS. Restore one without fabricating spans.
replay.py pushes a recording of a real EverOS run into the reader's own
Langfuse project. Names, attributes, token usage, structure and durations
are replayed verbatim; only ids, timestamps and a `replay` tag are
rewritten, so nothing in the trace is invented. It needs the OTel SDK and
Langfuse keys, nothing else.
record_trace.py is the maintainer tool that produced the recording. It
stands in for Langfuse's OTLP and scores endpoints on localhost, which
works because EverOS derives both from langfuse_host, so one sink captures
both signals straight from a real server run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UyKinsWs1MgoARPoB9R4NW
* feat(examples): give the Langfuse demo a memory worth searching
The demo ingested one conversation and searched it, so recall had nothing to
choose between and the traces showed plumbing rather than behaviour.
Eleven short conversations now span ten weeks, each on its own topic, so a
question has to find the right memory in a populated store. Two revisit the
same subject five days apart, close enough for geometry clustering to group
them, which finally gives reflection something to consolidate: the demo nudges
reflect_episodes (a `0 2 * * 1` cron otherwise), waits for the merge to land,
and the superseded memory is gone from search by the time the questions are
asked. One question asks about something never discussed, so a miss looks like
a miss.
KEYWORD is no longer a demonstrated method. Its top score is raw BM25, on a
different scale from the calibrated ones, so showing the three side by side
invited a comparison that means nothing.
Readiness is polled per session rather than slept through, since a fixed sleep
searched a half-built index and reported scores lower than the memory deserved.
Polling is deliberately slack: every probe is itself a traced search, and a
tight loop buried the real questions under a wall of readiness checks.
recorded_trace.json is that run against 1.2.1: 237 spans over 60 traces, no
errors, no secrets, synthetic content throughout.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UyKinsWs1MgoARPoB9R4NW
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Bump version to 1.2.1 and finalize CHANGELOG. Highlights: [embedding]
and [rerank] become soft runtime dependencies; new everos cascade
backfill CLI; LanceDB schema v2 (nullable vector); PyPI Trusted
Publishing workflow; 1.1.4 backport fixes.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(config): make [embedding] and [rerank] soft dependencies
Make [embedding] and [rerank] soft runtime dependencies so a
freshly-onboarded user can run EverOS end-to-end with only [llm]
configured. Previously the server refused to start without embedding,
locking out anyone who just wanted keyword-only search.
## Capability tiers
- Tier 1 ([llm] only) KEYWORD search, add/flush, md
writes, cascade sync
- Tier 2 ([llm] + [embedding]) + VECTOR / HYBRID search,
reflection, skill extraction,
backfill
- Tier 3 ([llm] + [embedding] + rerank) + AGENTIC search, knowledge
Tier upgrades require a server restart (capability accessors cache
for the process lifetime). Tier downgrades are read-safe: a Tier-3
user who drops [rerank] can still read/rename/delete existing
knowledge documents; only write/search endpoints return 422.
## What changed
- Component accessors — component/{embedding,rerank,llm}/accessor.py
are the single process-wide provider singletons. service/* never
maintains parallel singletons; it consumes get_embedding_capability()
/ get_rerank_capability() / get_llm_client() directly. Build-time
ValueError from the factory is logged as capability_build_failed
(was silently swallowed).
- Error mapping — ProviderNotConfiguredError -> 422 with everos.toml
section hints (never EVEROS_* env-var strings).
LanceDBMigrationError fails loud with escalating recovery guidance
(restart -> wipe index). LLMNotConfiguredError in search maps to
None for KEYWORD degradation.
- Nullable-vector LanceDB migration — schema v2 makes the vector
column nullable so Tier-1 rows can land without embeddings.
Migration is guarded by a cross-process memory_root_lock
(fcntl.flock + anyio.to_thread) and runs optimize() per table
after Phase-1 backfill to reclaim manifest bloat.
- Cascade — knowledge handlers register unconditionally (Tier-3 ->
Tier-2/1 downgrade no longer strands DELETE); embed-requiring
strategies use body-guards that check capability.available at
execution time. _TABLE_SPECS has an import-time drift assertion
against BUSINESS_SCHEMAS_WITH_VECTOR.
- `everos cascade backfill` CLI — Phase-1 (embed missing vectors) /
Phase-2 (emit synthetic events for cascaded processing) / Phase-3
(sync new skill files). Exit codes: 0 / 1 / 2 / 3 (server running
preflight) / 4 (COMPLETED_WITH_FAILURES — per-row failures rolled
up) / 130 (SIGINT). OMEConfig.crash_recovery_enabled=False in
backfill engines prevents stale-RUNNING rows re-enqueuing into a
smaller strategy registry.
- /health — reports capabilities + disabled_features per tier so ops
can distinguish "boots but degraded" from "boots and full".
- Presentation split — memory / service / infra never import typer /
click. TyperPresenter Protocol + run_backfill() live in
entrypoints/cli/commands/_backfill_cmd.py. Enforced by
import-linter.
- Startup hint — unconditional count_rows(filter="vector IS NULL")
sweep emits unbackfilled_memory_rows (event name + hint text
pinned) when Tier-1 rows exist. ParserLifespanProvider warms the
everalgo.parser import at boot so /health doesn't block on first
call.
- Knowledge upload UTF-8 short-circuit — _looks_like_utf8_text()
routes text/* mime and known plaintext extensions (md/txt/rst)
straight to UTF-8 decode instead of the parser. Prevents 503
Multimodal-not-configured when Tier 3 sans [multimodal] uploads a
markdown doc.
## Sync history with main (2 merges collapsed into this squash)
Merged origin/main at 6dcd3eb (v1.1.4 -> v1.2.0 adds OTel tracing,
/api/v2 alias, TracingLifespanProvider, per-cascade-embedding span
fix, memory-op instrumentation) and later at 42629df (PR #366
backfills v1.1.4 CWE-22 knowledge path traversal fix + cascade
retry-budget rework + errors.py -> core.errors.ExternalServiceError).
Key merge decisions:
- service/search.py adopts single wrap site — component.llm accessor
already applies UsageRecordingClient when observability is on;
service layer never keeps a parallel LLM singleton (Round-1 CR
rule: "service layer never maintains parallel singletons").
- Knowledge router prefix moved to /knowledge; create_app() mounts
it under both /api/v1 and /api/v2.
- Cascade retry classification uses ExternalServiceError from
core.errors (cascade/errors.py deleted). _MAX_TOTAL_RETRIES=12
cross-cycle budget preserved.
- Fixed backport typo: MemoryRoot.default() -> MemoryRoot.resolve()
(no .default() classmethod exists — main PR #366 shipped a broken
call).
## Verified layering
$ git grep -l "^import typer\|^from typer" src/everos/{memory,service,infra}
# empty
$ git grep -l "^import click\|^from click" src/everos/{memory,service,infra}
# empty
Memory / service / infra layers clean of CLI presentation libraries.
## Review history
Three rounds of Fable 5 (opus) code review across the pre-squash
commit history closed 38 findings total:
- Round 1: 10 findings (fail-loud migration, backfill hardening,
knowledge router gate scoping, SearchManager guards, profile
throttle lift)
- Round 2: 13 findings (hermetic test env, hot-reload doc drift,
knowledge handler registration, Phase 3 sync guarantee, Phase 2
idempotency, profile event-first path, OMEConfig crash-recovery
gate, cross-process migration lock, batch embed per-row fallback,
LanceDB optimize, typer/click layer split)
- Round 3: 15 Minor cleanups (accessor unification, marker revert,
episode query hygiene, --verbose subcommand, parser lifespan warm,
task-number scrub, temporal-overlap test, real-SIGINT slow mark,
4 design-note back-references)
Full per-round context lives in the PR description on GitHub.
## Test plan
- make lint (ruff + import-linter 3 contracts + assets +
deprecated-names + github-docs + datetime + OpenAPI drift)
- Hermetic env full pytest — 2027 passed / 7 deselected (7 = slow +
live_llm markers)
- Manual e2e across Tier 1/2/3 (21/21 assertions across v1/v2
double-mount and Tier 3 -> Tier 2 downgrade)
- /health reports correct capabilities + disabled_features per tier
## Known follow-ups
- .superpowers/sdd/followup-http-bridge.md (gitignored) — Path A for
spec §10's "backfill 期间 EverOS 完全可用" promise
- _TABLE_SCHEMA_VERSION docstring — v3+ migrations need a version
dispatch table
- extract_user_profile.py throttle-counter block — replace LanceDB
count_by_owner with a sqlite memcell count
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(review): close 3 blockers surfaced by round-4 review
N1. cluster_repo.find_cluster_id_for_member was cross-owner-unsafe.
Its reverse index (member_type, member_id) alone cannot disambiguate
two owners whose entry_id happens to collide — entry_id is
deliberately only per-owner unique (see entries.py:47:
'Cross-user uniqueness is handled at the database layer via a
composite <user_id>_<entry_id> field; it is not encoded into the
EntryId string itself'). Phase 2's _scan_all_rows crosses all
owners, so on any multi-owner root, same-day seq=1 episodes under
different owners would either false-hit each other's cluster or
be silently skipped from clustering. Add required (app_id,
project_id, owner_id) keyword args + JOIN Cluster (which already
carries scope) to filter by parent scope. Prior signature had zero
production callers except two the same PR just added, so the
API break is contained. Regression test: two owners persist a
cluster each around the same entry_id, each lookup resolves to
its own owner's cluster, a third owner's lookup returns None.
N2. Ctrl-C / EOF at the y/N prompt was landing on the generic
except Exception branch (exit 2 with rich traceback) instead of
the exit-130 interrupt path. Root cause: typer 0.15+ vendored
click under typer._click, so typer.Abort and the standalone
click.exceptions.Abort are distinct classes. The interrupt-branch
catch only listed the standalone one; every existing 'abort'
test was manually raising click.exceptions.Abort so the miss
was a false-positive guard rail. Widen the catch to
(typer.Abort, click.exceptions.Abort) and declare click as a
first-class dependency (it was only pulled in via uvicorn).
Regression test: raise real typer.Abort() at the confirm step
and assert exit 130 + INTERRUPTED banner.
N3. _looks_like_utf8_text used mime.startswith('text/'), which
caught text/html as well. HTML uploads then bypassed everalgo's
_aparse_html — losing clean_html_for_llm (strips <script>/<style>
/<nav>/<iframe> + HTML comments) and the 1 MiB output cap. A
40 MiB .html with <script> bodies and <!-- prompt injection -->
comments would flow straight into the extraction LLM. Replace
with explicit allowlist {text/plain, text/markdown, text/x-rst,
text/x-markdown}; text/html and any future text/* mime now
default to the parser path. Test matrix asserts text/html →
False (was regressed as True by the earlier commit).
Hermetic env full pytest: 2033 passed / 7 deselected (+6 tests
from these regressions).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(review): close round-4 major + minor + PR body errata
Round-4 review-driven cleanup. Blocker fixes (N1/N2/N3) landed in
561b5fe. This commit closes the remaining CONFIRMED items:
Major:
- J3 MemoryRoot.default() -> resolve() was a breaking public API
rename this branch introduced (main still has default()). Adds
default() as a backward-compat alias forwarding to resolve()
with DeprecationWarning; CHANGELOG entry under Unreleased.
- J4 episode_repo.list_by_owner_after_ts(limit=N) truncates in
fragment order (== insertion order), NOT newest-first. Docstring
now spells out the trap so a future caller passing limit for a
'newest N' window doesn't silently get the oldest N.
- J5.2 TyperPresenter.nothing_to_backfill picked colour via
'could not be read' in message — a domain wording change
would silently flip yellow -> green. Signature gains explicit
scan_failed: bool kwarg; CLI colour-picks off the flag.
- J5.5 phase_header was Protocol-declared but never dispatched
(run_backfill calls _print_phase_header directly). Removed the
dead Protocol method + both no-op implementations.
- J6.3 3 inline from everos.core.errors import ... inside
Phase 1/2/3 preflights promoted to a single top-level import.
- J7 subject-side embed failure was silently exit-0 because
rows_processed advanced whenever any side wrote. Now: a row with
a needed side still NULL counts as rows_failed (exit 4 =
COMPLETED_WITH_FAILURES). Gated on spec.subject_of + row.needs_*
+ row.subject_text so non-Episode tables and subject-empty rows
don't false-positive.
- J9 test_migration_cross_process.py did NOT actually test cross-
process (all 5 tests mock memory_root_lock). Renamed to
test_migration_lock_wiring.py; docstring now scopes it to
'lock-invocation wiring' and points at test_core/…/test_locking.py
for real flock coverage.
- J10 Phase 1 lacked the server-running preflight Phase 2/3 have.
--phase all against a live server would burn Phase 1 embed API
calls (real cost) before Phase 2 halted with exit 3. Phase 1 now
probes _probe_ome_lock_available first; regression test in
test_backfill_preflight.py; upgrade_path integration patches the
probe so its in-process 'server + backfill' scenario stays valid.
Minor:
- M1 knowledge upload with NUL byte or filename > 255 bytes UTF-8
used to raise ValueError/OSError at write_bytes → 500 with a
half-written md left on disk. _safe_original_filename now
rejects both up front with InvalidInputError (→ 400).
- M2 backfill optimize() now passes cleanup_older_than=timedelta(0)
so older manifest versions are physically pruned (previous call
compacted fragments but left the manifest chain on disk).
- M3 verify_business_schemas remediation text used to jump straight
to 'rm -rf ~/.everos/.index/lancedb'; now walks restart → wipe.
- M5 multimodal/accessor.py capability_build_failed warning added
so all four provider accessors log symmetrically (was silent).
- M7 test_knowledge_api parser-absence tests call
parser_available.cache_clear() around the sys.modules patch so
the lru_cache doesn't strand a stale True/False.
- M8 cascade_handler_embed_skipped (6 handlers) demoted INFO → DEBUG:
Tier 1 imports were generating N × 6 handler-info lines per md.
- M10.1 test_drift_scenario_would_raise was a tautology (compared
two hardcoded string sets, never touched the guard). Now
monkey-patches BUSINESS_SCHEMAS_WITH_VECTOR to a superset and
reloads _backfill, proving the import-time RuntimeError fires.
- M11 test_cascade_verbose_position subprocess.run calls gain
env= — scrubs EVEROS_* from the developer environment so the
same footgun as round-2 B1 doesn't re-appear inside subprocesses.
- M14 health() -> dict degraded the OpenAPI schema to
additionalProperties: true. Introduce HealthResponse +
HealthCapabilities Pydantic models so clients get real field
shape; docs/openapi.json regenerated.
- M16 routes/knowledge.py:_require_knowledge_capabilities docstring
claimed cascade.registry.build_handlers still gates
knowledge_topic/knowledge_document, contradicting
registry.py:177-194 (gate removed there, moved to HTTP layer).
Rewritten to describe the current design accurately.
Hermetic env full pytest: 2037 passed / 7 deselected.
Explicitly deferred to followup:
- J1 lazy multimodal client (needs everalgo signature change)
- J2 tier definition (knowledge = Tier 3 whole)
- J5.1/3/4 broader presentation-split refactor
- J6.1/2 backfill dispatch and _backfill_table refactor
- J8 Phase 1 keyset pagination for bulk migration OOM
- M4 flock timeout/waiting log/re-entry (core/persistence refactor)
- M6 UTF-8 codec strategy (BOM/UTF-16/GBK)
- M9 count_by_owner monotonicity (latent, INTERVAL=1 short-circuits)
- M13 --phase all multi-phase combined-outcome test coverage
- M15 PR-marker rationale comments (16 occurrences, all inert)
M12 was refuted (both event names exist).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1.2.0 introduced /api/v2 as the canonical, cloud-aligned prefix and mounted
every business router twice, but the user-facing entry points (README,
README.zh-CN, QUICKSTART, the docs/ set, the Langfuse example) still taught
/api/v1 — so new users were pointed at the compatibility alias while
docs/api.md already declared v2 canonical.
- Switch every EverOS endpoint reference in docs, examples, and
`everos demo --live` to /api/v2, plus the matching CLI test expectations.
- Describe /api/v1 as a legacy compatibility alias that may be removed in a
future major release, rather than a permanent one. Nothing changes at
runtime: both prefixes still resolve to the same handlers and the
v1/v2 parity test is untouched.
- Add a short note in README / README.zh-CN / QUICKSTART so existing v1
integrations know they keep working.
- Fix the five dead endpoint anchors in the docs/api.md table of contents,
which still pointed at the pre-1.2.0 #post-apiv1... slugs.
Left on v1 deliberately: docs/migration-to-1.0.0.md (historical record),
CHANGELOG history, tests/** (v1 must stay covered), and the
use-cases/claude-code-plugin + openher READMEs, which document a different
cloud API.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Settings resolves its TOML source through resolve_root(), so any field a
test does not pass explicitly was filled from the real everos.toml on the
machine running the suite. Enabling [observability] locally, for instance,
made test_returns_singleton_when_configured fail, because the LLM client
picks up the usage-recording wrapper when tracing is on: green in CI, red
on that developer's machine, and unrelated to whatever they were changing.
An autouse fixture now points EVEROS_ROOT at a per-test tmp dir. Tests that
exercise root resolution itself already setenv / delenv inside the test
body, which runs after the fixture, so none of them needed changing.
Claude-Session: https://claude.ai/code/session_01UyKinsWs1MgoARPoB9R4NW
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: zhanghui <huizhang1995@gmail.com>
Langfuse aggregates scores by name, so one name may only carry values on
one scale. recall_top_score was emitted for every method, mixing HYBRID's
LR-sigmoid probability and AGENTIC's cross-encoder score (both comparable
in [0, 1]) with KEYWORD's unbounded BM25 and single-route VECTOR's cosine.
A chart on that name averaged the two scales, and in practice a keyword
score can read numerically higher than a calibrated one while meaning less.
Uncalibrated methods now report recall_top_score_raw, leaving
recall_top_score comparable across methods and over time. Every recall
score also carries metadata = {method, calibrated}: a structured field
Langfuse persists and can split on, which the free-text comment could not
serve. The comment stays for reading individual scores.
Breaking for anyone charting recall_top_score for keyword search; 1.2.0 is
four days old, so this is the cheapest moment to correct the naming.
Claude-Session: https://claude.ai/code/session_01UyKinsWs1MgoARPoB9R4NW
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
GitHub had no publish path — the package was previously released from the
(now-archived) GitLab mirror. Bring publishing to the public repo:
- .github/workflows/release.yml: on a vX.Y.Z tag, build + smoke-test
(make package) and upload to PyPI via Trusted Publishing (OIDC, no stored
token), gated behind the `release` environment for manual approval. A guard
step refuses to publish when the tag != pyproject version.
- .claude/skills/release: /release documents the cut (bump version →
CHANGELOG → tag → approve → verify) and the one-time PyPI trusted-publisher
+ GitHub environment setup.
Requires two owner-only one-time steps before the first release (documented
in the workflow header and the skill): register the GitHub trusted publisher
on PyPI, and create the `release` environment with required reviewers.
Co-authored-by: zhanghui <zhanghui@shanda.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(knowledge): contain original-file write path (CWE-22)
The multipart upload filename was joined into ``_original/`` verbatim.
An attacker-controlled filename such as ``/tmp/pwned`` or
``../../.bashrc`` would let ``POST /knowledge/documents`` write outside
the document directory (the ``/`` operator discards the left operand
for absolute paths; ``..`` walks upward). The read side had the
symmetric issue.
Fix, mirroring the sender_id containment shipped in 1.0.1
(GHSA-c795-2g9c-j48m):
- Add ``_safe_original_filename`` reducing the untrusted filename to a
single POSIX/Windows-basename component; reject degenerate residuals
(``""``, ``"."``, ``".."``) with PathTraversalError.
- ``_write_original_file`` asserts ``target.resolve()`` stays inside
``original_dir.resolve()`` before any filesystem touch (mkdir/write).
- ``_resolve_original_file_path`` sanitises symmetrically so a stored
provenance label can never resolve to an out-of-directory file.
Four SEC regression tests cover: absolute filename, ``..`` traversal,
degenerate filename rejection, and read-side sanitisation.
Backport from GitLab release/v1.1.4 (commit 40f19de) — 1.1.4 shipped
this fix; the GitLab -> GitHub sync stopped at 1.1.3, so 1.2.0
regressed the containment. This commit alone restores the fix; the
1.2.1 release PR ships it to PyPI.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(cascade): retry classification, budget, and reconcile races
Backport the cascade reliability work that shipped in GitLab 1.1.4
(MR !49 / commit 95db2f5) — six interlocking changes the reviewer
should read in order:
1. Worker retry classification uses ExternalServiceError (embedding /
LLM / rerank transient failures) as the "retry inline" signal.
The legacy RecoverableError hierarchy under cascade/errors.py is
removed; the retry contract now lives in the domain error tree
(core/errors.py). Docstrings in handlers/base.py and
sqlite/tables/md_change_state.py updated to match.
2. Cross-cycle retry budget: _MAX_TOTAL_RETRIES = 12. Once total
attempts across scanner cycles exhaust the budget, the worker
marks retryable=False in place instead of looping forever on a
sustained upstream outage.
3. md_change_state upsert preserves retry_count on scanner
re-enqueue when mtime is unchanged (previously reset to 0 every
sweep, defeating the budget). mtime change (user edit) still
resets the counter.
4. Reconciler no longer re-enqueues pending / processing rows on
stable mtime — that was overwriting the worker's mark_done. It
also skips failed rows with retryable=False on stable mtime so
the entry-check demote path is stable.
5. mtime tolerance (10 ms, MTIME_TOLERANCE_SECONDS) absorbs the
SQLite REAL float precision loss that previously flapped the
reconcile decision when the same md was rewritten without a real
content change. The constant is defined once in the sqlite repo
and imported by the reconciler so both sides use the same tol.
6. Worker _run_rebuild_once carries an explicit
`state.task is not asyncio.current_task()` guard before awaiting
the optimize task — the previous contextlib.suppress was silently
swallowing self-await RuntimeError.
Kept intact from the GitHub 1.2.0 baseline:
- The `except FileNotFoundError → handle_deleted` branch in the
worker (delete/modify race — see
test_modified_event_for_vanished_file_is_processed_as_delete).
Test coverage added:
- test_retry_budget_exhausted_marks_unrecoverable
- test_external_service_error_at_budget_edge_demotes_in_place
- test_upsert_preserves_retry_count_for_failed_stable_mtime
- test_upsert_resets_retry_count_on_mtime_change
- test_optimize_fallback_rebuild_on_sustained_failure
- reconciler mtime-tolerance / stable-mtime skip suite
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(embedding): raise on empty API data; forward MRL dimensions
Backport from GitLab 1.1.4 (MR !49 / commit 95db2f5).
Two behavioural changes on ``OpenAIEmbeddingProvider._embed_chunk``:
1. ``response.data == []`` now raises ``EmbeddingServiceError`` instead
of returning an empty list. Some upstream providers (observed on
DeepInfra under load) return HTTP 200 with an empty ``data`` array;
the silent zero-vector path was corrupting search indexes without
any signal.
2. When ``[embedding] dimensions = N`` is set in ``everos.toml``, the
parameter is forwarded to the API so MRL-capable models
(OpenAI text-embedding-3-*, Qwen3-Embedding, ...) do server-side
truncation with proper re-normalization. Client-side truncation to
``dim`` remains as a fallback for backends that ignore the param.
``openai.NOT_GIVEN`` is used as the sentinel so the request omits
the field when the setting is left at the default ``None``.
Config plumbing:
- ``EmbeddingSettings.dimensions: int | None = None``
- factory forwards ``dimensions=settings.dimensions`` to the provider
The provider stays inside the existing ``memory_span`` OTel wrapper
and continues to report input-only tokens via ``set_generation_usage``
- both are GitHub 1.2.0 native tracing behaviours preserved intact.
Test coverage:
- test_empty_response_data_raises_embedding_error (new)
- test_usage_span._FakeEmbeddings.create signature updated to accept
``dimensions`` kwarg so the OTel token-recording tests still exercise
the same call path
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(extract): retry episode extraction on malformed LLM output
Backport from GitLab 1.1.4 (MR !49 / commit 95db2f5).
The ``/flush`` synchronous path called ``EpisodeExtractor.aextract``
exactly once. everalgo raises ``ValueError`` when the LLM returns
malformed JSON (observed with OpenRouter partial responses where
finish_reason=stop but the body is truncated) — the caller was
surfaced a 500 for a transient upstream hiccup.
``_extract_with_retry`` wraps the call with two extra attempts at 1s
and 2s backoff (final attempt propagates untouched), typed as
``AlgoEpisode`` so the caller path stays annotated. Retry stays inside
the existing GitHub 1.2.0 ``memory_span("everos.extract", ...)`` OTel
wrapper — the OTel token capture and the retry loop are orthogonal.
TODO in the code notes we should catch a typed everalgo
``ExtractionError`` once that type is introduced (currently ValueError
is broad).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: log non-stop finish_reason; bump everalgo-user-memory 0.3.2
Two loosely coupled changes from GitLab 1.1.4 (MR !49 / commit 95db2f5)
that arrive together as a housekeeping commit.
1. ``_LoggingLLMClient`` diagnostic wrapper (new, file-private).
Wraps the raw everalgo LLM client and, on every ``chat()``, warns
when ``resp.finish_reason != "stop"`` — logging the reason,
``content_len``, the last 200 chars of ``content``, and ``model``.
Aims at OpenRouter/DeepSeek truncation triage where the provider
silently caps output length and returns finish_reason=length /
filter / etc. Non-invasive: one branch per call, no config gate.
Wrapper stack in ``get_llm_client``:
LoggingLLMClient(UsageRecordingClient(build_client(...)))
LoggingLLMClient(build_client(...)) # observability off
``UsageRecordingClient`` (GitHub 1.2.0 native OTel token capture)
stays gated by ``settings.observability.enabled`` — this commit
preserves that. LoggingLLMClient is always outermost so the reason
it observes is exactly the reason the underlying provider reported.
2. ``everalgo-user-memory`` 0.3.1 -> 0.3.2 (pyproject + uv.lock).
Same bump the GitLab 1.1.4 release lane took; unblocks the
episode-extract retry work in commit 4 seeing the upstream
improvements. Verified via ``uv sync``.
No functional API changes.
Test coverage:
- test_returns_singleton_when_configured now asserts the outer
LoggingLLMClient wrapper.
- test_wraps_client_when_observability_enabled asserts the two-layer
Logging(UsageRecording(...)) stack.
- test_does_not_wrap_client_when_observability_disabled asserts
Logging still wraps when tracing is off.
- test_logging_wrapper_warns_on_non_stop_finish_reason (new).
- test_logging_wrapper_silent_on_stop_finish_reason (new).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(changelog): restore [1.1.4] section to match published sdist
The 1.1.4 changelog entry on this branch previously listed three
items (Langfuse example, delete/modify race, live-server telemetry).
The 1.1.4 sdist on PyPI, however, was built from the internal
release lane and includes the CWE-22 containment, the cascade
retry-budget / mtime-tolerance / reconcile-guard work, the embedding
empty-data raise, the episode-extract retry, MRL dimensions, and the
LLM finish_reason diagnostic — none of which were represented here
when the tag was cut.
Rewrite the [1.1.4] section so it matches the wheel a user actually
installs from PyPI:
- Add a header note explaining the retroactive restoration.
- Fixed: CWE-22, cascade reliability bundle, delete/modify race
(unchanged wording), embedding empty-data, episode extract retry,
Langfuse live-server telemetry (unchanged wording).
- Added: MRL dimensions, LLM finish_reason diagnostic, Langfuse
example (unchanged wording).
- Changed: everalgo-user-memory 0.3.1 -> 0.3.2.
The GitLab-side `.gitlab-ci.yml` in-house-runner entry is dropped —
open-source CI runs on GitHub Actions and the internal runner switch
is not visible to public users.
Date stays 2026-07-20 (the GitHub v1.1.4 tag date / PyPI upload
timestamp) rather than the internal 2026-07-23 code-freeze date, so
the timeline of what shipped where remains internally consistent.
The corresponding code fixes are all backported by earlier commits
in this PR; this commit only aligns the changelog surface.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Minor release: `/api/v2` API prefix (v1 retained as alias) and native
OpenTelemetry tracing — both back-compatible, so 1.1.4 -> 1.2.0.
- pyproject: version 1.1.4 -> 1.2.0
- CHANGELOG: promote [Unreleased] to [1.2.0]
- docs/openapi.json + uv.lock: regenerated for the new version
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The examples/langfuse wrapper (everos_langfuse.py) was the interim client-side
instrumentation before EverOS gained native OpenTelemetry export. Now that
[observability] emits real OTLP spans, the wrapper is redundant and its faked
child spans could mislead. Replace it with a minimal, dependency-light example:
enable [observability] in everos.toml, run the server, and drive one
add/flush/search cycle (demo.py, stdlib only) to see native traces in Langfuse.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Docs workflow (`links` job) is a required status check, but its
trigger was `paths:`-filtered to markdown/docs files. A PR touching only
code never triggered it, so the required check never reported and the merge
box stayed BLOCKED forever waiting for a status that would never arrive.
Drop the `paths:` filter: `make docs-check` validates the whole doc tree
independent of the PR diff and runs in seconds, so it is cheap to report on
every PR.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Summarizes the OTel observability feature (instrumentation + the review /
telemetry-audit fixes) as one user-facing Added entry; the fixes themselves
targeted an unreleased feature so they need no separate lines.
The embedding-observation change wrapped every _embed_chunk call in a
span. Cascade-time indexing embeds run outside any request trace, so each
chunk started its OWN root trace — a per-chunk trace explosion (13 orphan
everos.embedding traces per add/flush), detached from session/user and
contrary to the "cascade is not instrumented" decision.
memory_span gains nested_only: open a span only when one is already
active. Embedding uses it, so search/flush embeds still nest under their
recall/extract span, while cascade embeds no-op (no trace) — restoring the
cascade-untraced boundary.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up to the live-trace audit — all on the enabled path:
- boundary detection LLM (everalgo detect_boundaries) ran with the
SPAN-typed request root as the current span, so its ~1.2k tokens were
dropped from cost. Wrap it in an everos.memcell.boundary GENERATION
span so Langfuse prices it.
- embedding calls stamped usage on the enclosing retriever span. Wrap
each /embeddings call in an everos.embedding EMBEDDING span so the type
is correct and pricing can apply.
- agentic recall emitted a duplicate, same-name everos.search.recall
(cluster_scoped wrapping hybrid_full, which owns the real recall span).
Drop the redundant outer span; hybrid_full keeps the one recall span
(also used standalone in round 2).
- search now captures the returned hit ids (episodes/cases/skills) as
observation output when capture_content is on — previously only the
query input was captured.
- add/flush spans carry request_id in metadata.
- persist captures the memory-root-relative .md path, not the host
absolute path (no host layout leak to the telemetry backend).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adversarial review of the merged OTel instrumentation (#352) surfaced
four issues, all on the enabled path (default-off, so no impact until
tracing is turned on):
- search LLM client was unwrapped, so hybrid/agentic token usage — the
heaviest LLM spend — never reached Langfuse. Wrap it with
UsageRecordingClient when observability is enabled, mirroring
get_llm_client(); graceful keyword-only degradation is preserved.
- set_generation_usage overwrote token counts, undercounting any span
that wraps more than one chat call (the now-wrapped agentic path).
Accumulate instead of replacing.
- recall_hit was emitted for uncalibrated methods (unbounded BM25 /
single-route vector), a near-constant always-hit signal that inflates
dashboards. Gate hit on calibrated methods (HYBRID/AGENTIC); keyword
and vector emit only the raw top_score.
- init_tracing / init_score_sink were not idempotent — a re-init without
an intervening shutdown orphaned the export thread + OTLP socket +
worker task. Tear down the previous instance first (init_score_sink is
now async).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Every business endpoint (memory/*, ome/*, knowledge/*) is now served
under /api/v2, aligning the open-source API with the EverOS Cloud
contract. /api/v1 is retained as a permanent, backward-compatible alias:
the same router objects are mounted under both prefixes, so both resolve
to identical handlers and request/response contracts. Existing /api/v1
integrations keep working unchanged. Infra endpoints (/health, /metrics)
stay unversioned.
Fix the Prometheus request-metric label to build the path from the full
request URL (with path params folded) rather than the route's
router-relative path, so the version prefix is preserved and v1/v2
traffic stays distinguishable.
Docs (docs/api.md, docs/openapi.json), CHANGELOG, and route docstrings
updated to lead with /api/v2. Add test_api_versioning as the parity
guard: every v2 route has an identical v1 twin and vice versa.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Open spans at the memory hot paths (all no-op when tracing is off):
- add / flush (service.memorize), extract + persist.markdown (user pipeline).
- search: everos.memory.search retriever + a uniform recall / rank
decomposition across keyword / vector / hybrid / agentic (manager, agentic
modules, cross-encoder callbacks); query-embedding tokens land on recall.
- recall quality: top_score / hit on the search span, plus recall_top_score /
recall_hit pushed to Langfuse scores via the bounded-queue sink (method
tagged; off the request path).
- OME: everos.ome.<strategy> agent span + everos.reflect.consolidate
generation; a W3C traceparent captured at enqueue is threaded through the
APScheduler job and re-attached in the Runner, so strategies fanned out
from a request nest under that request's trace.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Surface gen_ai.* model + token attributes onto the active span so Langfuse
can compute cost — without touching everalgo:
- UsageRecordingClient wraps the LLM client and records response.usage after
each chat(); get_llm_client composes it over the existing _LoggingLLMClient
only when observability is enabled (disabled default stays overhead-free).
- OpenAIEmbeddingProvider records its response.usage (input tokens) onto the
active span too.
Tokens land on the everos.extract / everos.reflect.consolidate generation
spans and the search embedding recall; no-op when tracing is off.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(examples): make Langfuse wrapper degrade cleanly on a real server
The wrapper synthesized child spans (extraction, embedding, hybrid
recall, rerank, index sync, consolidation) from a mock-only `_detail`
field. Against a real EverOS server that field is absent, so those spans
rendered with placeholder data — hardcoded model names, token=0, fixed
sleep durations — and recall scores fell to 0.
Now the per-stage child spans are emitted only when `_detail` is present
(the mock, or future native in-core instrumentation). Against a live
server only the top-level span per operation is emitted, with real
latency and output — no fabricated data. Recall quality
(recall_top_score / recall_hit) is derived from the real search
response, which already carries a per-hit score, so it works against a
live server today, not just the mock.
Verified: mock path unchanged (full trace tree, real scores); real-ish
path (no `_detail`) emits only top-level spans plus a real recall score.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(examples): count empty recalls as a miss in Langfuse hit-rate
When a search returns nothing scored, record recall_hit=0 (span attribute +
Langfuse score) instead of omitting it, so genuine empty recalls still show
up in recall hit-rate. No top_score is emitted (there is no hit to score).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds examples/langfuse/ — a thin OpenTelemetry wrapper that traces EverOS
memory operations (add / flush+extract / search / reflection) into Langfuse,
with recall quality pushed as Langfuse scores. Pure OTel SDK, no Langfuse
package dependency; runs against a built-in mock or a real EverOS server
(EVEROS_BASE_URL). Additive only, no changes to EverOS core.
Referenced by the upcoming Langfuse docs integration cookbook.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
FTS indexes built with with_position=True crash lance's optimize/compaction
on lancedb >= 0.32 when merging an unindexed tail (Max offset exceeds length
of values; upstream lance-format/lance#7653). The crash aborts optimize()
including version cleanup, so the index dir grows unbounded until the disk
fills. everos recall is OR-mode BM25 and never does phrase queries, so
positions are never read -- disabling is lossless.
- base: default with_position=False
- infra: migrate_fts_indexes() rebuilds pre-fix indexes once at startup + reclaims orphans
- cascade worker: count consecutive optimize failures, escalate warning->error
Fixes#335.
Co-authored-by: zhanghui <zhanghui@shanda.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
compile_filters() unconditionally appended 'deprecated_by IS NULL' to
every query, but the deprecated_by column only exists on user-scoped
tables (episode, atomic_fact — Reflection V1). Agent tables
(agent_case, agent_skill) lack this column, causing a SQL error on
agent search/get queries.
Gate the clause behind owner_type == 'user' so agent queries no longer
reference a non-existent column.
Bump version to 1.1.2.
Co-authored-by: Jiayao Song <jiayao.song@shanda.com>
Elliot Chen
zhanghui
Kendrick-Song
Dani
zhanghui