main
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
b1441da607 |
refactor(config): make [embedding] and [rerank] soft dependencies (#361)
* refactor(config): make [embedding] and [rerank] soft dependencies
Make [embedding] and [rerank] soft runtime dependencies so a
freshly-onboarded user can run EverOS end-to-end with only [llm]
configured. Previously the server refused to start without embedding,
locking out anyone who just wanted keyword-only search.
## Capability tiers
- Tier 1 ([llm] only) KEYWORD search, add/flush, md
writes, cascade sync
- Tier 2 ([llm] + [embedding]) + VECTOR / HYBRID search,
reflection, skill extraction,
backfill
- Tier 3 ([llm] + [embedding] + rerank) + AGENTIC search, knowledge
Tier upgrades require a server restart (capability accessors cache
for the process lifetime). Tier downgrades are read-safe: a Tier-3
user who drops [rerank] can still read/rename/delete existing
knowledge documents; only write/search endpoints return 422.
## What changed
- Component accessors — component/{embedding,rerank,llm}/accessor.py
are the single process-wide provider singletons. service/* never
maintains parallel singletons; it consumes get_embedding_capability()
/ get_rerank_capability() / get_llm_client() directly. Build-time
ValueError from the factory is logged as capability_build_failed
(was silently swallowed).
- Error mapping — ProviderNotConfiguredError -> 422 with everos.toml
section hints (never EVEROS_* env-var strings).
LanceDBMigrationError fails loud with escalating recovery guidance
(restart -> wipe index). LLMNotConfiguredError in search maps to
None for KEYWORD degradation.
- Nullable-vector LanceDB migration — schema v2 makes the vector
column nullable so Tier-1 rows can land without embeddings.
Migration is guarded by a cross-process memory_root_lock
(fcntl.flock + anyio.to_thread) and runs optimize() per table
after Phase-1 backfill to reclaim manifest bloat.
- Cascade — knowledge handlers register unconditionally (Tier-3 ->
Tier-2/1 downgrade no longer strands DELETE); embed-requiring
strategies use body-guards that check capability.available at
execution time. _TABLE_SPECS has an import-time drift assertion
against BUSINESS_SCHEMAS_WITH_VECTOR.
- `everos cascade backfill` CLI — Phase-1 (embed missing vectors) /
Phase-2 (emit synthetic events for cascaded processing) / Phase-3
(sync new skill files). Exit codes: 0 / 1 / 2 / 3 (server running
preflight) / 4 (COMPLETED_WITH_FAILURES — per-row failures rolled
up) / 130 (SIGINT). OMEConfig.crash_recovery_enabled=False in
backfill engines prevents stale-RUNNING rows re-enqueuing into a
smaller strategy registry.
- /health — reports capabilities + disabled_features per tier so ops
can distinguish "boots but degraded" from "boots and full".
- Presentation split — memory / service / infra never import typer /
click. TyperPresenter Protocol + run_backfill() live in
entrypoints/cli/commands/_backfill_cmd.py. Enforced by
import-linter.
- Startup hint — unconditional count_rows(filter="vector IS NULL")
sweep emits unbackfilled_memory_rows (event name + hint text
pinned) when Tier-1 rows exist. ParserLifespanProvider warms the
everalgo.parser import at boot so /health doesn't block on first
call.
- Knowledge upload UTF-8 short-circuit — _looks_like_utf8_text()
routes text/* mime and known plaintext extensions (md/txt/rst)
straight to UTF-8 decode instead of the parser. Prevents 503
Multimodal-not-configured when Tier 3 sans [multimodal] uploads a
markdown doc.
## Sync history with main (2 merges collapsed into this squash)
Merged origin/main at
|
||
|
|
42629dfd4d |
fix: backfill missing 1.1.4 fixes into main (#366)
* fix(knowledge): contain original-file write path (CWE-22) The multipart upload filename was joined into ``_original/`` verbatim. An attacker-controlled filename such as ``/tmp/pwned`` or ``../../.bashrc`` would let ``POST /knowledge/documents`` write outside the document directory (the ``/`` operator discards the left operand for absolute paths; ``..`` walks upward). The read side had the symmetric issue. Fix, mirroring the sender_id containment shipped in 1.0.1 (GHSA-c795-2g9c-j48m): - Add ``_safe_original_filename`` reducing the untrusted filename to a single POSIX/Windows-basename component; reject degenerate residuals (``""``, ``"."``, ``".."``) with PathTraversalError. - ``_write_original_file`` asserts ``target.resolve()`` stays inside ``original_dir.resolve()`` before any filesystem touch (mkdir/write). - ``_resolve_original_file_path`` sanitises symmetrically so a stored provenance label can never resolve to an out-of-directory file. Four SEC regression tests cover: absolute filename, ``..`` traversal, degenerate filename rejection, and read-side sanitisation. Backport from GitLab release/v1.1.4 (commit 40f19de) — 1.1.4 shipped this fix; the GitLab -> GitHub sync stopped at 1.1.3, so 1.2.0 regressed the containment. This commit alone restores the fix; the 1.2.1 release PR ships it to PyPI. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(cascade): retry classification, budget, and reconcile races Backport the cascade reliability work that shipped in GitLab 1.1.4 (MR !49 / commit 95db2f5) — six interlocking changes the reviewer should read in order: 1. Worker retry classification uses ExternalServiceError (embedding / LLM / rerank transient failures) as the "retry inline" signal. The legacy RecoverableError hierarchy under cascade/errors.py is removed; the retry contract now lives in the domain error tree (core/errors.py). Docstrings in handlers/base.py and sqlite/tables/md_change_state.py updated to match. 2. Cross-cycle retry budget: _MAX_TOTAL_RETRIES = 12. Once total attempts across scanner cycles exhaust the budget, the worker marks retryable=False in place instead of looping forever on a sustained upstream outage. 3. md_change_state upsert preserves retry_count on scanner re-enqueue when mtime is unchanged (previously reset to 0 every sweep, defeating the budget). mtime change (user edit) still resets the counter. 4. Reconciler no longer re-enqueues pending / processing rows on stable mtime — that was overwriting the worker's mark_done. It also skips failed rows with retryable=False on stable mtime so the entry-check demote path is stable. 5. mtime tolerance (10 ms, MTIME_TOLERANCE_SECONDS) absorbs the SQLite REAL float precision loss that previously flapped the reconcile decision when the same md was rewritten without a real content change. The constant is defined once in the sqlite repo and imported by the reconciler so both sides use the same tol. 6. Worker _run_rebuild_once carries an explicit `state.task is not asyncio.current_task()` guard before awaiting the optimize task — the previous contextlib.suppress was silently swallowing self-await RuntimeError. Kept intact from the GitHub 1.2.0 baseline: - The `except FileNotFoundError → handle_deleted` branch in the worker (delete/modify race — see test_modified_event_for_vanished_file_is_processed_as_delete). Test coverage added: - test_retry_budget_exhausted_marks_unrecoverable - test_external_service_error_at_budget_edge_demotes_in_place - test_upsert_preserves_retry_count_for_failed_stable_mtime - test_upsert_resets_retry_count_on_mtime_change - test_optimize_fallback_rebuild_on_sustained_failure - reconciler mtime-tolerance / stable-mtime skip suite Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(embedding): raise on empty API data; forward MRL dimensions Backport from GitLab 1.1.4 (MR !49 / commit 95db2f5). Two behavioural changes on ``OpenAIEmbeddingProvider._embed_chunk``: 1. ``response.data == []`` now raises ``EmbeddingServiceError`` instead of returning an empty list. Some upstream providers (observed on DeepInfra under load) return HTTP 200 with an empty ``data`` array; the silent zero-vector path was corrupting search indexes without any signal. 2. When ``[embedding] dimensions = N`` is set in ``everos.toml``, the parameter is forwarded to the API so MRL-capable models (OpenAI text-embedding-3-*, Qwen3-Embedding, ...) do server-side truncation with proper re-normalization. Client-side truncation to ``dim`` remains as a fallback for backends that ignore the param. ``openai.NOT_GIVEN`` is used as the sentinel so the request omits the field when the setting is left at the default ``None``. Config plumbing: - ``EmbeddingSettings.dimensions: int | None = None`` - factory forwards ``dimensions=settings.dimensions`` to the provider The provider stays inside the existing ``memory_span`` OTel wrapper and continues to report input-only tokens via ``set_generation_usage`` - both are GitHub 1.2.0 native tracing behaviours preserved intact. Test coverage: - test_empty_response_data_raises_embedding_error (new) - test_usage_span._FakeEmbeddings.create signature updated to accept ``dimensions`` kwarg so the OTel token-recording tests still exercise the same call path Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(extract): retry episode extraction on malformed LLM output Backport from GitLab 1.1.4 (MR !49 / commit 95db2f5). The ``/flush`` synchronous path called ``EpisodeExtractor.aextract`` exactly once. everalgo raises ``ValueError`` when the LLM returns malformed JSON (observed with OpenRouter partial responses where finish_reason=stop but the body is truncated) — the caller was surfaced a 500 for a transient upstream hiccup. ``_extract_with_retry`` wraps the call with two extra attempts at 1s and 2s backoff (final attempt propagates untouched), typed as ``AlgoEpisode`` so the caller path stays annotated. Retry stays inside the existing GitHub 1.2.0 ``memory_span("everos.extract", ...)`` OTel wrapper — the OTel token capture and the retry loop are orthogonal. TODO in the code notes we should catch a typed everalgo ``ExtractionError`` once that type is introduced (currently ValueError is broad). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: log non-stop finish_reason; bump everalgo-user-memory 0.3.2 Two loosely coupled changes from GitLab 1.1.4 (MR !49 / commit 95db2f5) that arrive together as a housekeeping commit. 1. ``_LoggingLLMClient`` diagnostic wrapper (new, file-private). Wraps the raw everalgo LLM client and, on every ``chat()``, warns when ``resp.finish_reason != "stop"`` — logging the reason, ``content_len``, the last 200 chars of ``content``, and ``model``. Aims at OpenRouter/DeepSeek truncation triage where the provider silently caps output length and returns finish_reason=length / filter / etc. Non-invasive: one branch per call, no config gate. Wrapper stack in ``get_llm_client``: LoggingLLMClient(UsageRecordingClient(build_client(...))) LoggingLLMClient(build_client(...)) # observability off ``UsageRecordingClient`` (GitHub 1.2.0 native OTel token capture) stays gated by ``settings.observability.enabled`` — this commit preserves that. LoggingLLMClient is always outermost so the reason it observes is exactly the reason the underlying provider reported. 2. ``everalgo-user-memory`` 0.3.1 -> 0.3.2 (pyproject + uv.lock). Same bump the GitLab 1.1.4 release lane took; unblocks the episode-extract retry work in commit 4 seeing the upstream improvements. Verified via ``uv sync``. No functional API changes. Test coverage: - test_returns_singleton_when_configured now asserts the outer LoggingLLMClient wrapper. - test_wraps_client_when_observability_enabled asserts the two-layer Logging(UsageRecording(...)) stack. - test_does_not_wrap_client_when_observability_disabled asserts Logging still wraps when tracing is off. - test_logging_wrapper_warns_on_non_stop_finish_reason (new). - test_logging_wrapper_silent_on_stop_finish_reason (new). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(changelog): restore [1.1.4] section to match published sdist The 1.1.4 changelog entry on this branch previously listed three items (Langfuse example, delete/modify race, live-server telemetry). The 1.1.4 sdist on PyPI, however, was built from the internal release lane and includes the CWE-22 containment, the cascade retry-budget / mtime-tolerance / reconcile-guard work, the embedding empty-data raise, the episode-extract retry, MRL dimensions, and the LLM finish_reason diagnostic — none of which were represented here when the tag was cut. Rewrite the [1.1.4] section so it matches the wheel a user actually installs from PyPI: - Add a header note explaining the retroactive restoration. - Fixed: CWE-22, cascade reliability bundle, delete/modify race (unchanged wording), embedding empty-data, episode extract retry, Langfuse live-server telemetry (unchanged wording). - Added: MRL dimensions, LLM finish_reason diagnostic, Langfuse example (unchanged wording). - Changed: everalgo-user-memory 0.3.1 -> 0.3.2. The GitLab-side `.gitlab-ci.yml` in-house-runner entry is dropped — open-source CI runs on GitHub Actions and the internal runner switch is not visible to public users. Date stays 2026-07-20 (the GitHub v1.1.4 tag date / PyPI upload timestamp) rather than the internal 2026-07-23 code-freeze date, so the timeline of what shipped where remains internally consistent. The corresponding code fixes are all backported by earlier commits in this PR; this commit only aligns the changelog surface. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
bd26f5a80f |
feat(observability): capture LLM + embedding token usage
Surface gen_ai.* model + token attributes onto the active span so Langfuse can compute cost — without touching everalgo: - UsageRecordingClient wraps the LLM client and records response.usage after each chat(); get_llm_client composes it over the existing _LoggingLLMClient only when observability is enabled (disabled default stays overhead-free). - OpenAIEmbeddingProvider records its response.usage (input tokens) onto the active span too. Tokens land on the everos.extract / everos.reflect.consolidate generation spans and the search embedding recall; no-op when tracing is off. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
15efd1198c | chore(release): update EverOS to 1.1.1 (#327) | ||
|
|
dc02b2fbab | feat(rerank): add Bailian DashScope provider support (#295) | ||
|
|
518b8eca85 |
chore: initialize EverOS 1.0.0
md-first memory extraction framework for AI agents. Markdown is the single source of truth; SQLite holds state and LanceDB provides the rebuildable vector + BM25 + scalar index. The codebase follows a single-direction DDD layering (entrypoints -> service -> memory -> infra, with component / core / config cross-cutting) enforced by import-linter. Engineering surface: - Coding conventions in .claude/rules/ (path-scoped) and workflows in .claude/skills/ (/commit, /new-branch, /pr). - GitHub Actions CI runs make lint + test + integration; pre-commit mirrors the gates locally (ruff, hygiene hooks, gitlint commit-msg). - Commit messages follow Conventional Commits, enforced by gitlint. - make lint also enforces datetime two-zone discipline and OpenAPI drift. |