xiaoxue_admin
/
EverOS
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

chore(release): update EverOS to 1.1.0 (#307)

This commit is contained in:
Elliot Chen 2026-06-24 23:17:23 +08:00 committed by GitHub
parent 1ea44ca548
commit 0df88f5603
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
262 changed files with 20901 additions and 2595 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

308
.claude/skills/add-memory-kind/SKILL.md Normal file
View File

@ -0,0 +1,308 @@
---
name: add-memory-kind
description: Add a new business memory kind end-to-end. Pick the storage combination (Markdown / SQLite / LanceDB), pick the markdown strategy (daily-log / skill-named / single-file), then wire up the schema(s), repo(s), and writer(s).
---
# /add-memory-kind — Add a new business memory kind
## When to invoke
Adding a new persisted business entity (Episode, Case, Skill, AtomicFact,
Foresight, Profile, or something custom). Multiple storage layers may be
involved; this skill walks the decision then the wiring.
## 1. Decide the storage combination
A memory kind **does not have to use all three** layers. Pick by what
the kind actually needs:
| Need | Markdown | SQLite | LanceDB |
|---|:-:|:-:|:-:|
| Human-readable / agent-editable source-of-truth text | ✅ | | |
| Structured state, ACID transactions, joins, predicates | | ✅ | |
| Vector / BM25 / hybrid retrieval | | | ✅ |
Common combinations seen in EverOS:
| Combo | Example | Rationale |
|---|---|---|
| **md only** | scratch notes / dump bins | text-of-truth, no index needed |
| **md + lancedb** | episode / memcell / case | text-of-truth + semantic retrieval |
| **md + sqlite** | profile / playbook / soul.md state | text-of-truth + structured state to query |
| **md + sqlite + lancedb** | full-blown business records | when you need *both* transactional state AND retrieval |
| **sqlite only** | audit log / task queue / LSN watermark | system state, never user-facing |
| **lancedb only** | rare; usually you still want md | derived embeddings without text-of-truth |
Rule of thumb: **markdown is the truth**; sqlite and lancedb are derived
indexes that can be rebuilt from md. Drop md only when the kind has no
human-readable form (pure system state).
## 2. Pick the markdown storage strategy (if md is in your combo)
Three strategies — declared in the EverOS Markdown First spec:
| Strategy | Filename | Mutation | Examples |
|---|---|---|---|
| **Daily-log append** | `<prefix>-YYYY-MM-DD.md` | append entries | memcell / episode / case / atomic_fact / foresight |
| **Skill-named in-place** | `skill_<name>.md` | overwrite the file | skills (procedural memory) |
| **Single-file rewrite** | `user.md` / `agent.md` / `soul.md` / `behaviors.md` / `tools.md` | overwrite the file | profiles / playbooks |
This skill currently has a **complete recipe for daily-log append**.
Skill-named and single-file recipes are sketched at the bottom — their
base writers (`BaseSkillWriter` / `BaseProfileWriter`) land later in the
project; until then build a thin wrapper over `MarkdownWriter`
directly.
---
## 3. Markdown daily-log: 4 steps
### 3.1 Frontmatter schema — `infra/persistence/markdown/mds/<name>.py`
```python
"""Episode daily-log frontmatter."""
from __future__ import annotations
import datetime as _dt
from typing import ClassVar, Literal
from everos.core.persistence.markdown import UserScopedFrontmatter
class UserEpisodeDailyFrontmatter(UserScopedFrontmatter):
"""``users/<u>/episodes/episode-<YYYY-MM-DD>.md``."""
ENTRY_ID_PREFIX: ClassVar[str] = "ep"
DIR_NAME: ClassVar[str] = "episodes"
FILE_PREFIX: ClassVar[str] = "episode"
type: Literal["user_episode_daily"] = "user_episode_daily"
date: _dt.date
entry_count: int = 0
last_appended_at: _dt.datetime | None = None
```
For agent-track kinds subclass `AgentScopedFrontmatter` instead. If
user-track and agent-track share a kind name (e.g. `memcell`), give
each a **distinct** `ENTRY_ID_PREFIX` (e.g. `umc` vs `amc`) so reverse
lookup is unambiguous.
### 3.2 Re-export — `mds/__init__.py`
```python
from .episode import UserEpisodeDailyFrontmatter as UserEpisodeDailyFrontmatter
```
### 3.3 Business writer — `infra/persistence/markdown/writers/<name>.py`
```python
"""Episode appender."""
from __future__ import annotations
from pathlib import Path
from everos.core.persistence import MarkdownReader
from ..mds import UserEpisodeDailyFrontmatter
from .base import BaseDailyWriter
class UserEpisodeAppender(BaseDailyWriter):
schema = UserEpisodeDailyFrontmatter
# OPTIONAL: override the count strategy. Default is len(entries);
# override to trust the frontmatter field instead.
def _current_count(self, path: Path) -> int:
if not path.exists():
return 0
return MarkdownReader.read(path).frontmatter.get("entry_count", 0)
```
### 3.4 Re-export — `writers/__init__.py`
```python
from .episode import UserEpisodeAppender as UserEpisodeAppender
```
### Done — usage
```python
from everos.infra.persistence.markdown.writers import UserEpisodeAppender
appender = UserEpisodeAppender(memory_root)
eid = appender.append("u_jason", "I went to the doctor today.")
# → users/u_jason/episodes/episode-<today>.md
# → entry markers carry an auto-generated EntryId (e.g. ep_20260507_001)
```
---
## 4. (Optional) SQLite table — 4 steps
Skip this section if the kind doesn't need structured state beyond markdown.
### 4.1 Schema — `infra/persistence/sqlite/tables/<name>.py`
```python
from everos.core.persistence.sqlite import BaseTable, Field
class EpisodeState(BaseTable, table=True):
__tablename__ = "episode_state" # type: ignore[assignment]
id: int | None = Field(default=None, primary_key=True)
entry_id: str = Field(index=True, unique=True)
cluster_id: str | None = Field(default=None, index=True)
status: str = Field(default="active")
```
`BaseTable` already provides `created_at` / `updated_at` (auto-bumped).
### 4.2 Re-export — `tables/__init__.py`
```python
from .episode import EpisodeState as EpisodeState
```
### 4.3 Repo — `infra/persistence/sqlite/repos/<name>.py`
```python
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from everos.core.persistence.sqlite import RepoBase
from ..sqlite_manager import get_session_factory
from ..tables import EpisodeState
class _EpisodeStateRepo(RepoBase[EpisodeState]):
model = EpisodeState
def _factory_lookup(self) -> async_sessionmaker[AsyncSession]:
return get_session_factory()
episode_state_repo = _EpisodeStateRepo()
```
### 4.4 Re-export — `repos/__init__.py`
```python
from .episode import episode_state_repo as episode_state_repo
```
---
## 5. (Optional) LanceDB index — 4 steps
Skip this section if the kind doesn't need vector / BM25 / hybrid retrieval.
### 5.1 Schema — `infra/persistence/lancedb/tables/<name>.py`
```python
from everos.core.persistence.lancedb import BaseLanceTable, Vector
class EpisodeIndex(BaseLanceTable):
entry_id: str
text: str
tags: list[str]
vector: Vector(384) # type: ignore[valid-type]
```
`Vector(N)` must match your embedding dimension.
### 5.2 Re-export — `tables/__init__.py`
```python
from .episode import EpisodeIndex as EpisodeIndex
```
### 5.3 Repo — `infra/persistence/lancedb/repos/<name>.py`
```python
from lancedb import AsyncTable
from everos.core.persistence.lancedb import LanceRepoBase
from ..lancedb_manager import get_table
from ..tables import EpisodeIndex
class _EpisodeIndexRepo(LanceRepoBase[EpisodeIndex]):
schema = EpisodeIndex
table_name = "episode_index"
async def _table_lookup(self) -> AsyncTable:
return await get_table(self.table_name, self.schema)
episode_index_repo = _EpisodeIndexRepo()
```
### 5.4 Re-export — `repos/__init__.py`
```python
from .episode import episode_index_repo as episode_index_repo
```
---
## 6. (Future) Skill-named & single-file markdown strategies
When the new memory kind needs:
- **skill-named** files (one file per named skill, in-place rewrite) — wait
for `BaseSkillWriter`, or use `MarkdownWriter.write_markdown` directly
with a thin wrapper.
- **single-file rewrite** (one fixed file like `user.md`) — wait for
`BaseProfileWriter`, same fallback.
These strategies do **not** use entry markers; their frontmatter schema
does not need `ENTRY_ID_PREFIX` (only `id` / `type` / `schema_version` plus
the scope mixin fields).
---
## 7. Verification checklist
- [ ] `make lint` — ruff + import-linter clean
- [ ] `make test` — existing manager / lifespan / writer tests still pass
- [ ] When the kind crosses **multiple** layers:
- [ ] markdown entry id is the join key for the sqlite / lancedb rows
- [ ] business code reads only via the repo singleton (no raw engine
access in service / memory)
- [ ] cascade daemon (when it lands) can rebuild sqlite / lancedb
from md alone — keep md as the truth
Tests by layer:
| Tests for | Location |
|---|---|
| Markdown frontmatter schema | `tests/unit/test_infra/test_markdown/test_mds/` |
| Markdown business appender | `tests/unit/test_infra/test_markdown/test_writers/` |
| SQLite RepoBase logic | `tests/unit/test_core/test_persistence/test_sqlite/` |
| SQLite manager / lifespan | `tests/unit/test_infra/test_sqlite/` |
| LanceDB LanceRepoBase logic | `tests/unit/test_core/test_persistence/test_lancedb/` |
| LanceDB manager / lifespan | `tests/unit/test_infra/test_lancedb/` |
## 8. Common pitfalls
| Mistake | Symptom | Fix |
|---|---|---|
| Forgot `ENTRY_ID_PREFIX` / `DIR_NAME` / `FILE_PREFIX` on a daily-log schema | `BaseDailyWriter.__init__` raises `TypeError` | Add all three ClassVars |
| Same `ENTRY_ID_PREFIX` on user + agent variants | `MemoryLayout.locate_for_entry` collision error | Use distinct prefixes (e.g. `umc` vs `amc`) |
| Imported `RepoBase` from `infra.persistence.sqlite` | `ImportError` | Lives in `core.persistence.sqlite` (moved earlier) |
| Skipped one of the four files (schema / writer / table / repo) | One side silently absent | Re-export both/all from each `__init__.py` |
| `Vector(N)` mismatched with embedding dim | LanceDB raises on insert | Make `N` exactly match the model output |
| Imported `MemoryLayout` from a writer (infra) | `import-linter` fails (`infra → memory` reverse dep) | Use `MemoryRoot` (in core) and let the schema's ClassVars drive paths |
| Hand-rolling `datetime.now()` instead of `today_with_timezone()` | Day-boundary drift across timezones | Always go through `everos.component.utils.datetime` |
## Background
- Architecture: [../../rules/architecture.md](../../rules/architecture.md)
- `__init__.py` re-export rules: [../../rules/init-py-and-reexport.md](../../rules/init-py-and-reexport.md)
- Async programming: [../../rules/async-programming.md](../../rules/async-programming.md)
- Datetime handling: [../../rules/datetime-handling.md](../../rules/datetime-handling.md)

20
.env.example
View File

@ -4,23 +4,19 @@
# =====================================================
#
# Setup:
# 1. Create .env with `everos init` or `cp .env.example .env`
# 1. cp env.template .env
# 2. Edit .env with your values
# 3. .env is gitignored (never commit)
#
# Override priority (low → high):
# src/everos/config/default.toml (shipped baseline)
# ↓
# ~/.everos/config.toml (user-level overrides; optional)
# <root>/everos.toml (user config; optional; root resolved
# by EVEROS_ROOT env > ~/.everos)
# ↓
# .env (this file; gitignored)
# ↓
# EVEROS_<SECTION>__<KEY> process envs
# EVEROS_<SECTION>__<KEY> process envs (this file sources these)
# ↓
# programmatic init args / CLI flags
#
# The user-level toml path defaults to ~/.everos/config.toml; override
# with EVEROS_CONFIG_FILE=/path/to/your.toml. Missing file is skipped.
# =====================================================
@ -65,7 +61,7 @@ EVEROS_MULTIMODAL__BASE_URL=https://openrouter.ai/api/v1
# ─── Embedding (OpenAI-protocol /embeddings) ─────────
# Any OpenAI-compatible embedding endpoint plugs in via base_url.
# model / api_key / base_url have no shipped default — set them here
# or in ~/.everos/config.toml before the embedding capability is used.
# or in <root>/everos.toml before the embedding capability is used.
EVEROS_EMBEDDING__MODEL=Qwen/Qwen3-Embedding-4B
EVEROS_EMBEDDING__API_KEY=
@ -109,10 +105,10 @@ EVEROS_RERANK__BASE_URL=https://api.deepinfra.com/v1/inference
# ─── Storage paths ───────────────────────────────────
# memory-root holds md files + .index/ (LanceDB) + .system.db (SQLite) + ...
# Override the default with EVEROS_MEMORY__ROOT (note the double-underscore
# for nested config keys); see config/default.toml for all tunables.
# Override the default (~/.everos) with EVEROS_ROOT; also controls which
# everos.toml is loaded. See config/default.toml for all other tunables.
# EVEROS_MEMORY__ROOT=~/.everos
# EVEROS_ROOT=~/.everos
# ─── HTTP API ────────────────────────────────────────

192
.gitignore vendored
View File

@ -1,3 +1,6 @@
# VS Code workspace files
*.code-workspace
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
@ -50,10 +53,6 @@ coverage.xml
*.py,cover
.hypothesis/
.pytest_cache/
.ruff_cache/
.import_linter_cache/
.uv-cache/
.package-smoke/
# Translations
*.mo
@ -107,6 +106,7 @@ celerybeat.pid
# Environments
.env
.env.*
.venv
env/
venv/
@ -132,73 +132,139 @@ dmypy.json
# Pyre type checker
.pyre/
# ───────────────────────────────────────────────────────────────────────────
# Project specific
# ───────────────────────────────────────────────────────────────────────────
outputs/
.pytest_cache/
evaluation/static_memory_evaluation/logs/
evaluation/static_memory_evaluation/retrieval/
evaluation/static_memory_evaluation/retrieval_bk/
evaluation/static_memory_evaluation/analysis/
memory_base/static_memory_base/bm25_index/
memory_base/static_memory_base/hipporag/
memory_base/static_memory_base/vector_search/
# Demo related files
demo/.chat_history
demo/chat_history/
demo/memcell_outputs/
# macOS
.DS_Store
# Editor / IDE
.trae/
.cursor/
# Claude Code: track team-shared rules/skills/settings; ignore personal + runtime
.claude/*
!.claude/rules/
!.claude/skills/
!.claude/settings.json
.claude/settings.local.json
.claude/worktrees/
.worktrees/
worktrees/
# Runtime data (the default memory root + local databases)
.everos/
*.duckdb
# Large / generated artefacts that should never be committed
# Evaluation and memory data files
evaluation_memory_offline/data/
evaluation_memory_offline/temporary/
# Dynamic memory files (runtime generated)
memory_base/dynamic_memory_base/memory/
# Evaluation results and large files
evaluation/dynamic_memory_evaluation/locomo_results/*.json
evaluation/dynamic_memory_evaluation/locomo_results/hipporag_index*/
evaluation/dynamic_memory_evaluation/locomo_results/vector_index*/
evaluation/dynamic_memory_evaluation/locomo_results/storages*
# Large data files
memory_base/static_memory_base/raw_data/*.json
*.tar
*.zip
tmp/
outputs/
logs/
# Large evaluation files
evaluation/memory_evaluation/converted_messages_all_from_1_1.jsonl
evaluation/memory_evaluation/*.jsonl
evaluation/memory_evaluation/*.json
# Large log files
*.log
su_ser.log
service.log
service_*.log
nohup.out
mongodb_backup/
# Repository media policy: do not commit images, videos, or asset/media folders.
# Use external hosting, release artifacts, or approved storage and link from docs.
asset/
assets/
image/
images/
img/
media/
video/
videos/
*.avif
*.bmp
*.gif
*.heic
*.heif
*.icns
*.ico
*.jpeg
*.jpg
*.png
*.svg
*.tif
*.tiff
*.webp
*.avi
*.flv
*.m4v
*.mkv
*.mov
*.mp4
*.mpeg
*.mpg
*.webm
*.wmv
# Log directories
logs/
log/
# Use-cases: exclude lock files to keep the repo lean
use-cases/**/package-lock.json
# Apollo config
apollo_config/
# Exclude specific files from git tracking
simple_query_memcell.py
evaluation/memory_evaluation/extract_room_data.py
evaluation/memory_evaluation/test.py
evaluation/results/
# Large test files
unit_test/memcell_outputs/
unit_test/profile_outputs/
unit_test/memcell_outputs.zip
#temp test files
export_mongodb_data.py
#backup files
src/memory_layer/memory_extractor/profile_memory_extractor keep llm merge.py
# JetBrains IDE
.idea/
# Local design plans
docs/plans/
# Git worktrees
.worktrees/
#LLM related
AGENTS.mk
.cursor/*
# .claude/ team-shared (rules, skills, settings.json, CLAUDE.md) goes IN git
# Only personal overrides are ignored:
.claude/settings.local.json
CLAUDE.local.md
# Work context: personal design drafts, not part of the codebase
.work_context/
# Local personal workspace: spec drafts, plans, scratch, recovery scripts
local/
#tmp_data
demo/memcell_outputs/
demo/results/
evaluation/locomo_evaluation/results/
evaluation/locomo_evaluation/results_ref/
tmp/
evaluation/locomo_evaluation/results_ref/demo/results/
# i18n translation progress
.translation_progress.json
.review_progress.json
# Single "output/" directory (plural "outputs/" already covered above)
output/
demo/output/
# Adapter prompt dumps (runtime generated, sometimes several MB)
evaluation/src/adapters/*/prompts/profile/*.json
# Locomo source dataset (downloadable, not source code)
data/locomo10.json
evaluation/data/locomo/locomo10.json
evaluation/locomo_evaluation/data/locomo10.json
# Legacy src kept locally for migration reference; not under version control.
src_old/
# Benchmark checkpoints — Phase-level intermediate JSON dumps produced by
# tests/test_locomo.py / tests/run_locomo_batch.sh. Per-conv final results
# (benchmark_results/run_*/convN.json) and reports (REPORT.md / REPORT.html)
# stay tracked; checkpoints are large, regeneratable, and not useful in
# code review.
benchmark_results/
benchmark_checkpoints/
# Local everos runtime data (memory root, indexes, OME state)
.everos/

30
.pre-commit-config.yaml
View File

@ -1,23 +1,16 @@
# Local quality gate, run before each commit. Mirrors the checks CI enforces so
# failures surface locally first. Install with `make install` (sets up both the
# pre-commit and commit-msg hook stages).
#
# Run manually across the repo: uv run pre-commit run --all-files
default_install_hook_types: [pre-commit, commit-msg]
default_language_version:
python: python3.12
repos:
# ruff version is kept in sync with the `ruff` dev dependency in uv.lock.
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.15.12
hooks:
- id: ruff
name: ruff (lint + autofix)
args: [--fix]
- id: ruff-format
name: ruff (format)
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v5.0.0
rev: v6.0.0
hooks:
- id: trailing-whitespace
- id: end-of-file-fixer
@ -25,25 +18,12 @@ repos:
- id: check-toml
- id: check-added-large-files
args: [--maxkb=1024]
- id: detect-private-key
- id: check-merge-conflict
- id: detect-private-key
- repo: local
hooks:
- id: no-repo-assets
name: block committed images, videos, and asset directories
entry: python3 scripts/check_repo_assets.py
language: system
pass_filenames: false
- id: no-deprecated-product-names
name: block deprecated product names
entry: python3 scripts/check_deprecated_names.py
language: system
pass_filenames: false
# Commit message format: gitlint runs in commit-msg stage (no Node required).
- repo: https://github.com/jorisroovers/gitlint
rev: v0.19.1
hooks:
- id: gitlint
name: gitlint (commit message format)
stages: [commit-msg]

136
CHANGELOG.md
View File

@ -7,31 +7,134 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
_Unreleased changes on `dev` will be listed here._
## [1.1.0] - 2026-06-24
### Added
- **Knowledge base subsystem** — full-stack document management exposed via
`/api/v1/knowledge/*`. Upload documents (PDF / HTML / DOCX via multimodal
parser), CRUD operations, and hybrid search (BM25 + vector + rerank +
category boost). Ships with a 20-category default taxonomy
(`.taxonomy.md`, auto-generated on first use). Original uploaded files are
preserved alongside extracted Markdown. New settings group:
`knowledge.*` (search tuning, `max_upload_bytes`, etc.).
- **Reflection V1** — offline memory self-improvement engine.
Select → Merge → Re-extract → Deprecate: clusters related episodes within
existing 7-day windows, merges them via LLM, re-extracts consolidated
episodes, and deprecates the originals. Runs as an OME strategy
(`reflect_episodes`); configure via `ome.toml`
(`[strategies.reflect_episodes]`, cron `0 2 * * 1`), changes are
hot-reloaded within ~2 s, no restart needed; **disabled by default**.
Requires `everalgo-user-memory>=0.3.1`.
- **Standardized error response contract.** All API errors now return a
canonical envelope with a semantic `ErrorCode` (10 codes: `NOT_FOUND`,
`CONFLICT`, `INVALID_INPUT`, `EXTRACTION_EMPTY`, `UNSUPPORTED_FORMAT`,
`EXTERNAL_SERVICE_UNAVAILABLE`, `CAPABILITY_UNAVAILABLE`,
`CONFIGURATION_ERROR`, `INTERNAL_ERROR`, `BAD_REQUEST`), per-type
exception handlers with MRO dispatch, and an `ErrorResponse` Pydantic
model visible in OpenAPI docs. Replaces the v1.0 two-code scheme
(`HTTP_ERROR` / `SYSTEM_ERROR`).
- **Search: hierarchical fact eviction** (Layer-4) with `min_score` floor —
low-confidence atomic facts are evicted before fusion, improving
precision.
- **Search degradation guidance** — when embedding or rerank providers fail,
the response now includes a `degradation` field explaining which
capability is unavailable and how results are affected.
- **Knowledge topic recaller** — dual-column BM25 recall for knowledge
topics, integrated into the search manager alongside existing recall
types.
### Changed
- **`everos init` now generates `gpt-4.1-mini`** as the default LLM model
(was `gpt-4o-mini`). Existing user configurations are not affected.
- **API error `code` values have changed.** v1.0 returned only `HTTP_ERROR`
(all 4xx) and `SYSTEM_ERROR` (all 5xx). v1.1 returns fine-grained
semantic codes (see Added above). Clients that match on `error.code`
string values need to update. The envelope structure
(`request_id` + `error.{code, message, timestamp, path}`) is unchanged.
- **DDD-aligned exception hierarchy** — domain errors reorganized:
`ValidationError` → `InvalidInputError`;
`DocumentAlreadyExistsError` → `DuplicateDocumentError`;
`EmbeddingError` → `EmbeddingServiceError`;
`RerankError` → `RerankServiceError`;
`LLMError` → `LLMServiceError` (at the boundary);
`MultimodalError` split into `UnsupportedModalityError` (domain) +
`MultimodalNotEnabledError` (infrastructure).
New base classes: `CapabilityError`, `ConfigurationError`.
- **`infra/` restructured** — storage adapters moved under
`infra/persistence/{markdown,sqlite,lancedb}`; each sub-package's
`__init__.py` is the sole public API (enforced by import-linter).
- **Parser capability extracted** to `component/parser` (shared by memorize
and knowledge upload paths).
### Fixed
- **Knowledge search no longer returns a bare `500 INTERNAL_ERROR` when the
embedding or rerank provider is unconfigured.** `_require_search_providers`
now raises `ConfigurationError` → `500 CONFIGURATION_ERROR`. A provider
that is configured but fails at call time still surfaces as
`503 EXTERNAL_SERVICE_UNAVAILABLE`.
- **Knowledge document uploads are capped** at `knowledge.max_upload_bytes`
(default 50 MiB); oversized uploads are rejected with `422` before parsing.
- **Knowledge search `query` is bounded** to 2000 chars.
- **`GET /knowledge/documents?sort_by=updated_at`** is now accepted.
- **`POST /knowledge/documents` returns `original_file_path`** so callers no
longer need a follow-up `GET` to locate the preserved upload.
- **Rerank providers no longer echo the upstream HTTP response body** into the
client-facing `503` message (vLLM / DeepInfra); the body is logged instead.
- **Knowledge FK cascade race** — removed the foreign key on
`knowledge_topics.doc_id` that caused delete-order race conditions;
cascade cleanup handled at application level.
- **Knowledge `replace_document`** — atomic PUT: backup old Markdown before
re-extraction; removed explicit SQLite delete for atomicity.
- **Knowledge duplicate `doc_id`** rejected on create; title collision
resolved by appending `doc_id` to directory name.
- **Knowledge `md_path` resolution** fixed in `delete_document` (was not
resolved against `memory_root`).
- **OME file-handle leak** — portalocker file handle is now closed on lock
contention instead of being left open.
- **jieba / Python 3.12 compatibility** — deferred jieba import to avoid
`SyntaxError` from invalid escape sequences; suppressed
`DeprecationWarning` in tests.
- **Test isolation** — tests no longer leak `.env` state or depend on module
import ordering.
### Documentation
- Added knowledge base technical documentation.
- Corrected the onboarding flow: `everos init` writes `everos.toml` +
`ome.toml` (TOML), not a `.env` file; removed the nonexistent
`--xdg` / `--env-file` options and the false `0600`-permissions claim
from `README.md` / `QUICKSTART.md`; fixed the stable-version line
(`v1.0.1`) and completed the `docs/cli.md` command tree.
- Updated error handling docs to match the new DDD exception hierarchy.
## [1.0.1] - 2026-06-16
### Security
- **Path-traversal hardening for caller-supplied identifiers.** `sender_id`
now carries the same path-safety guard as `app_id` / `project_id`: a
character whitelist plus rejection of the `.` / `..` tokens. The whitelist
admits `@` and `+` so email-style ids and plus-addressing still pass.
(which flows through to `owner_id` and becomes a directory segment on the
episode write path) now carries the same path-safety guard as `app_id` /
`project_id`: a character whitelist plus rejection of the `.` / `..` tokens.
The whitelist admits `@` and `+` so real-world ids (email-style,
plus-addressing) still pass.
- **Defense-in-depth write containment.** `MarkdownWriter` now rejects any
write target that resolves outside the configured memory root before reading,
creating parent directories, or writing files. The API layer maps this
backstop error to HTTP 400.
write target that resolves outside the configured memory root, before any
filesystem touch (both the write `mkdir` and the append read-modify-write
read). This backstop holds even if an identifier reaches the writer
unsanitised (e.g. an `owner_id` set in the extract pipeline rather than from
the DTO). The API layer maps the resulting error to HTTP 400.
### Documentation
- Add a multimodal usage guide and correct the multimodal error semantics
after end-to-end verification.
- Document the upcoming Knowledge Wiki and idle/offline Reflection/Dreaming
roadmap in the README and documentation set.
- Rename outdated algorithm-library references to `everalgo` across docs and
code comments; no code identifiers changed.
- Fix accuracy drift found in a documentation audit; reflect the `everalgo`
packages being published and the v1.0.0 stable status.
- Rename the algorithm library from the previous package name to `everalgo`
across docs and code comments (no code identifiers changed).
- Fix accuracy drift found in an adversarial doc audit; reflect the
`everalgo` packages being published and the v1.0.0 stable status.
## [1.0.0] - 2026-06-03
@ -60,6 +163,7 @@ for AI agents.
- **Decoupled algorithms** — memory extraction algorithms live in the standalone
`everalgo-*` libraries published on PyPI.
[Unreleased]: https://github.com/EverMind-AI/EverOS/compare/v1.0.1...HEAD
[1.0.1]: https://github.com/EverMind-AI/EverOS/releases/tag/v1.0.1
[1.0.0]: https://github.com/EverMind-AI/EverOS/releases/tag/v1.0.0
[Unreleased]: https://github.com/EverMind-AI/everos/compare/v1.1.0...HEAD
[1.1.0]: https://github.com/EverMind-AI/everos/compare/v1.0.1...v1.1.0
[1.0.1]: https://github.com/EverMind-AI/everos/releases/tag/v1.0.1
[1.0.0]: https://github.com/EverMind-AI/everos/releases/tag/v1.0.0

8
CITATION.md
View File

@ -33,8 +33,8 @@ To cite the software itself:
```
EverOS: md-first memory extraction framework for AI agents
Version: 1.0.1
URL: https://github.com/EverMind-AI/EverOS
Version: 0.1.0
URL: https://github.com/EverMind-AI/everos
License: Apache 2.0
```
@ -52,9 +52,9 @@ If you use EverOS, we appreciate:
## Stay updated
- Watch the [GitHub repository](https://github.com/EverMind-AI/EverOS) for paper announcements
- Watch the [GitHub repository](https://github.com/EverMind-AI/everos) for paper announcements
- Follow [@EverMindAI](https://x.com/EverMindAI) on X / Twitter
- Join [GitHub Discussions](https://github.com/EverMind-AI/EverOS/discussions)
- Join [GitHub Discussions](https://github.com/EverMind-AI/everos/discussions)
---

2
Makefile
View File

@ -4,7 +4,7 @@ help:
@echo "Targets:"
@echo " install Install deps + pre-commit hooks (full dev setup)"
@echo " install-deps Install deps only (uv sync --frozen, used by CI)"
@echo " lint ruff + import-linter + repo hygiene + datetime discipline + openapi drift"
@echo " lint ruff (check + format-check) + import-linter + datetime discipline + openapi drift"
@echo " docs-check Validate Markdown links, use-case banners, and issue template YAML"
@echo " check-commits Validate Conventional Commit subjects for a git range"
@echo " check-pr-title Validate PR title uses Conventional Commit format"

8
SECURITY.md
View File

@ -2,13 +2,13 @@
## Supported Versions
EverOS is in active alpha development. Security fixes are applied to the latest
release line only.
EverOS is released and at v1.0.0 (stable). Security fixes are applied to the
latest release line only.
| Version | Supported |
|---------|-----------|
| 0.1.x | ✅ |
| < 0.1 | ❌ |
| 1.x | ✅ |
| < 1.0 | ❌ |
## Reporting a Vulnerability

2
config.example.toml
View File

@ -21,7 +21,7 @@
# ── LLM ───────────────────────────────────────────────
# OpenAI-protocol chat-completions endpoint used by the algo extractors.
[llm]
model = "gpt-4o-mini"
model = "gpt-4.1-mini"
api_key = "sk-..."
base_url = "https://api.openai.com/v1"

78
docs/api.md
View File

@ -30,6 +30,7 @@ business semantics the raw spec does not carry.
- [POST /api/v1/memory/flush](#post-apiv1memoryflush)
- [POST /api/v1/memory/search](#post-apiv1memorysearch)
- [POST /api/v1/memory/get](#post-apiv1memoryget)
- [POST /api/v1/ome/trigger](#post-apiv1ometrigger)
- [OpenAPI spec source](#openapi-spec-source)
## Overview
@ -166,26 +167,39 @@ the top level (mirroring the success envelope) alongside a nested
{
"request_id": "<32-char hex>",
"error": {
"code": "HTTP_ERROR",
"message": "Value error, exactly one of user_id / agent_id must be provided",
"code": "NOT_FOUND",
"message": "Document 'abc123' not found",
"timestamp": "2026-06-01T12:24:46+00:00",
"path": "/api/v1/memory/search"
"path": "/api/v1/knowledge/documents/abc123"
}
}
```
| HTTP | `error.code` | `error.message` | When |
### error.code values
`error.code` is a machine-readable `ErrorCode` enum. Clients can switch
on this value to decide retry / display / routing behaviour without
parsing the human-readable `message` field.
| `error.code` | HTTP | Retryable? | When |
|---|---|---|---|
| `415 Unsupported Media Type` | `HTTP_ERROR` | the parse-failure reason | `/add` only — a `ContentItem` could not be parsed (unsupported modality for the configured multimodal LLM, or a payload that cannot be fetched / dispatched) |
| `422 Unprocessable Entity` | `HTTP_ERROR` | the **first** validation error (see below) | Request-body validation failure. Also covers `/search` / `/get` filter-DSL compile errors — the compile reason rides in `message` |
| `500 Internal Server Error` | `SYSTEM_ERROR` | `"Internal server error"` (fixed; internal details are logged, never leaked) | Unhandled exception caught by the global handler |
| `NOT_FOUND` | `404` | No | Requested resource does not exist |
| `CONFLICT` | `409` | No | Operation conflicts with existing state (e.g. duplicate document) |
| `INVALID_INPUT` | `422` | No | Request-body validation failure. Also covers `/search` / `/get` filter-DSL compile errors — the compile reason rides in `message` |
| `EXTRACTION_EMPTY` | `422` | No | Document extraction produced no topics (empty or whitespace-only content) |
| `BAD_REQUEST` | `400` | No | Path traversal attempt or other malformed input |
| `UNSUPPORTED_FORMAT` | `415` | No | File format or modality not supported (e.g. unsupported `ContentItem` type, missing `ext` for `base64`) |
| `EXTERNAL_SERVICE_UNAVAILABLE` | `503` | **Yes** | An external service (LLM, embedding, rerank) returned an error or timed out |
| `CAPABILITY_UNAVAILABLE` | `503` | No | A required server-side capability is missing (e.g. `everos[multimodal]` extra not installed, LibreOffice absent) — requires admin action, not retry |
| `CONFIGURATION_ERROR` | `500` | No | A required configuration is missing or invalid (e.g. embedding model not set) |
| `INTERNAL_ERROR` | `500` | No | Unhandled exception (internal details are logged, never leaked) |
### error object
| Field | Type | Description |
|---|---|---|
| `code` | `string` | `"HTTP_ERROR"` for 4xx (validation / business / `HTTPException`); `"SYSTEM_ERROR"` for 5xx |
| `message` | `string` | Human-readable reason. For `422`, **only the first** validation error is surfaced, formatted `"<msg>: <dotted-loc>"` with the leading `body` segment stripped (e.g. `"Field required: messages"`); a model-level validator with no field location surfaces just `"<msg>"` (e.g. the XOR example above) |
| `code` | `string` | One of the `ErrorCode` values listed above |
| `message` | `string` | Human-readable reason. For `INVALID_INPUT` from request validation, **only the first** validation error is surfaced, formatted `"<msg>: <dotted-loc>"` with the leading `body` segment stripped (e.g. `"Field required: messages"`); a model-level validator with no field location surfaces just `"<msg>"` (e.g. `"Value error, exactly one of user_id / agent_id must be provided"`) |
| `timestamp` | `string` | ISO-8601 with timezone offset (display tz) |
| `path` | `string` | Request path, e.g. `/api/v1/memory/add` |
@ -193,6 +207,15 @@ the top level (mirroring the success envelope) alongside a nested
> returned — only the first error's message. A client that needs the
> offending field can read the `<loc>` suffix in `message`.
### Search degradation
When a `/search` call uses `method: "vector"` or `"hybrid"` and the
embedding or rerank service is temporarily unavailable, the server does
**not** return `503`. Instead, it degrades gracefully — the response
suggests an alternative method in the error detail so the client can
retry with `"keyword"` (which requires no embedding). This keeps search
available during transient provider outages.
## Common types
### MessageItem
@ -793,7 +816,7 @@ attribution, so `session_id` is the only meaningful query dimension.
| `sender_id` | `string` | Original sender id from `/add` |
| `sender_name` | `string \| null` | Original sender name; `null` if not provided |
| `role` | `"user" \| "assistant" \| "tool"` | Original role |
| `content` | `string \| array<object>` | `string` for the single-text shorthand, `array` of opaque content items for the original multimodal payload (mirrors [MessageItem.content](#messageitem)) |
| `content` | `string \| array<object>` | `string` for the single-text shorthand, `array` of opaque content items for the original multimodal payload (mirrors [MessageItem.content](#addmessage)) |
| `timestamp` | `string` | ISO-8601 with timezone offset — see [Conventions](#conventions) |
| `tool_calls` | `array<object> \| null` | Original tool_calls payload if any |
| `tool_call_id` | `string \| null` | Original tool_call_id if any |
@ -1021,6 +1044,41 @@ Response (real capture):
}
```
### POST /api/v1/ome/trigger
Manually trigger a registered OME strategy.
#### Request body
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
| `name` | `string` | yes | — | Strategy name (e.g. `reflect_episodes`) |
| `timeout` | `float` | no | `120.0` | Max seconds to wait for completion |
| `force` | `bool` | no | `false` | Bypass the `enabled` gate in `ome.toml` |
#### Response body
`200 OK` returns:
| Field | Type | Notes |
|---|---|---|
| `status` | `"ok" \| "timeout"` | Whether the strategy completed within the timeout |
| `name` | `string` | Echoes the requested strategy name |
#### Errors
- `404` — strategy name not found in the OME registry.
#### cURL example
```bash
curl -X POST http://127.0.0.1:8000/api/v1/ome/trigger \
-H 'Content-Type: application/json' \
-d '{"name": "reflect_episodes", "force": true}'
```
---
## OpenAPI spec source
This document mirrors the OpenAPI 3.x spec that FastAPI auto-generates

57
docs/architecture.md
View File

@ -10,7 +10,7 @@
│ cli + api │
├──────────────────────────────────────────────────────┤
│ service/ (Application — Use Case orchestration) │
│ memorize / retrieve / evolve / manage │
│ memorize / search / get / knowledge │
├──────────────────────────────────────────────────────┤
│ memory/ (Domain — Business core) │
│ models + extract + search + cascade + prompt_slots │
@ -20,8 +20,8 @@
└──────────────────────────────────────────────────────┘
Cross-cutting (used by all layers, depends on none):
component/ ← Injectable providers (LLM / Embedding / config / utils)
core/ ← Runtime base (observability / lifespan / context)
component/ ← Injectable providers (LLM / Embedding / parser / config / utils)
core/ ← Runtime base (observability / lifespan / context / errors)
config/ ← Configuration data (Settings schema + default.toml)
```
@ -170,7 +170,7 @@ Three-piece observability:
## Markdown layout
```
~/.everos/ # memory root (default; EVEROS_MEMORY__ROOT)
~/.everos/ # memory root (default; EVEROS_ROOT)
└── <app_id>/<project_id>/ # scope ("default" → default_app/default_project)
├── users/<user_id>/
│ ├── user.md # profile (single-file rewrite)
@ -209,6 +209,55 @@ everalgo is:
This boundary lets everalgo be reused across product forms (this open-source build, EverOS Cloud, OpenClaw plugins, etc.).
## Error handling architecture
### Exception hierarchy
All application exceptions derive from `AppError` (`core/errors.py`),
split into four branches by nature:
```
AppError
├── DomainError (client-side / business-rule violations)
│ ├── NotFoundError → 404 NOT_FOUND
│ │ ├── DocumentNotFoundError
│ │ └── TopicNotFoundError
│ ├── ConflictError → 409 CONFLICT
│ │ └── DuplicateDocumentError
│ ├── InvalidInputError → 422 INVALID_INPUT
│ │ ├── ExtractionEmptyError → 422 EXTRACTION_EMPTY
│ │ └── FilterError
│ ├── PathTraversalError → 400 BAD_REQUEST
│ └── UnsupportedModalityError → 415 UNSUPPORTED_FORMAT
├── InfrastructureError (transient, retryable) → 503
│ ├── StorageError
│ ├── VectorStoreError
│ └── ExternalServiceError
│ ├── LLMServiceError
│ ├── EmbeddingServiceError
│ └── RerankServiceError
├── CapabilityError (permanent, not retryable) → 503
│ └── MultimodalNotEnabledError
└── ConfigurationError (misconfiguration) → 500
```
### Error propagation strategy
Exceptions are raised at the layer where the error is detected and
propagate naturally — **service and route layers do not catch-and-wrap**.
The entrypoints layer registers per-type exception handlers
(`entrypoints/api/exception_handlers.py`) via Starlette's MRO dispatch.
Each handler converts the exception into a canonical error envelope with
an `ErrorCode` enum value and the appropriate HTTP status code.
### Boundary translation
Third-party exception types are translated at the component boundary to
prevent external types from leaking into upper layers:
- `everalgo.llm.LLMError` → `LLMServiceError` at `component/parser/_core.py`
- Embedding / rerank provider errors → `EmbeddingServiceError` / `RerankServiceError` at their respective protocol modules
## Further reading
- [docs/overview.md](overview.md) — vision and scope

2
docs/cascade_runbook.md
View File

@ -204,7 +204,7 @@ Lives in `LanceDBSettings`; overridable via the
`EVEROS_LANCEDB__INDEX_CACHE_SIZE_BYTES` environment variable. This
is the only knob that bounds the steady-state file-descriptor count
of a long-running EverOS daemon — see
[Recovery paths § FD exhaustion](#fd-exhaustion-os-error-24--emfile)
[Recovery paths § FD exhaustion](#fd-exhaustion-os-error-24-emfile)
for why nothing else (prune, rebuild, `drop_index`) helps.
Measured cap → FD ceiling (30 add+optimize cycles + 100-query stress

26
docs/cli.md
View File

@ -1,8 +1,9 @@
# CLI
The `everos` command-line entry point covers **setup and operations** —
generate a starter `.env` (`init`), run the HTTP API server (`server
start`), and operate the md → LanceDB index queue (`cascade`). Hot-path
generate starter config files (`init`), run the HTTP API server (`server
start`), inspect effective config (`config show`), and operate the
md → LanceDB index queue (`cascade`). Hot-path
business (`/add` `/flush` `/search` `/get`) is the **HTTP API**, not the
CLI.
@ -26,13 +27,15 @@ a [Typer](https://typer.tiangolo.com/) app.
```
everos
├── init Generate a starter .env from the packaged template
├── init [--root PATH] [--force] [--print] Generate starter config files (everos.toml + ome.toml)
├── config
│ └── show [--root PATH] Show effective configuration
├── server
│ └── start Start the HTTP API server (uvicorn)
└── cascade Inspect / operate the md → LanceDB sync queue
│ └── start [--host] [--port] [--root] [--reload] [--log-level] Start the HTTP API server (uvicorn)
└── cascade [--root PATH] Inspect / operate the md → LanceDB sync queue
├── status Queue / LSN summary
├── sync Drain the queue now (force md → LanceDB)
└── fix List failed rows / re-enqueue retryable ones
├── sync [PATH] Drain the queue now (optional PATH force-enqueues)
└── fix [--apply] List failed rows / re-enqueue retryable ones
```
Each subcommand lives in its own module under
@ -54,7 +57,7 @@ everos server start \
--host 127.0.0.1 \
--port 8000 \
--log-level info \
--env-file .env
--root ~/.everos
```
| Flag | Env var | Default |
@ -62,7 +65,7 @@ everos server start \
| `--host` | `EVEROS_API__HOST` | `127.0.0.1` (loopback only; binding `0.0.0.0` logs a warning — EverOS ships no auth) |
| `--port` | `EVEROS_API__PORT` | `8000` |
| `--log-level` | `EVEROS_LOG_LEVEL` | `INFO` |
| `--env-file` | — | searched: `./.env` → `$XDG_CONFIG_HOME/everos/.env` → `~/.everos/.env` |
| `--root` | `EVEROS_ROOT` | `~/.everos` |
| `--reload` | — | off (use in development) |
Lifespan startup wires the storage backends (SQLite engine + LanceDB
@ -75,7 +78,7 @@ Both CLI and HTTP server read configuration from `pydantic-settings`:
| Env var | Settings field |
|---|---|
| `EVEROS_MEMORY__ROOT` | `Settings.memory.root` (memory-root path) |
| `EVEROS_ROOT` | memory-root path (default `~/.everos`) |
| `EVEROS_MEMORY__TIMEZONE` | `Settings.memory.timezone` (e.g. `Asia/Shanghai`) |
| `EVEROS_SQLITE__BUSY_TIMEOUT_MS` | `Settings.sqlite.busy_timeout_ms` |
| `EVEROS_LANCEDB__READ_CONSISTENCY_SECONDS` | `Settings.lancedb.read_consistency_seconds` |
@ -98,7 +101,8 @@ everos server start --log-level debug # see all sql / lance traffic
| Responsibility | API | CLI |
|---|---|---|
| Hot-path business (`/add` `/flush` `/search` `/get`) | ✅ | — (HTTP only) |
| Setup (generate `.env`) | — | `everos init` |
| Setup (generate config files) | — | `everos init` |
| Inspect effective config | — | `everos config show` |
| Run the server | — | `everos server start` |
| Index ops (drain / inspect / fix the cascade queue) | — | `everos cascade {status,sync,fix}` |
| Health probe | `GET /health` | (use HTTP) |

228
docs/configuration.md Normal file
View File

@ -0,0 +1,228 @@
# Configuration
EverOS uses a two-file TOML configuration system with environment variable
overrides for container deployments.
## Quick Start
```bash
# 1. Generate config files in the default root (~/.everos/)
everos init
# 2. Edit the config — fill in API keys
$EDITOR ~/.everos/everos.toml
# 3. Start the server
everos server start
```
## Configuration Files
### `everos.toml` — Application Settings
Located at `<root>/everos.toml`. Controls all application behavior: API
bind address, LLM/embedding/rerank provider credentials, SQLite pragmas,
search strategy, memorize mode, and clustering tunables.
Generated by `everos init` from the shipped `config/default.toml`
template. Changes require a server restart.
### `ome.toml` — Strategy Configuration
Located at `<root>/ome.toml`. Controls the Offline Memory Engine (OME)
strategy scheduling: which strategies are enabled, cron expressions,
gate thresholds, and retry limits.
Generated by `everos init` from the shipped `config/default_ome.toml`
template. Changes are **hot-reloaded** within ~2 seconds — no server
restart needed.
### Source Priority
Settings are resolved in this order (later wins):
1. `config/default.toml` — shipped defaults (lowest priority)
2. `<root>/everos.toml` — user configuration (optional)
3. `EVEROS_*` environment variables — container/CI overrides
4. Programmatic init args (highest priority; internal use)
## Memory Root Resolution
The memory root directory is resolved from three sources (first wins):
| Source | Example |
|---|---|
| `--root` CLI flag | `everos server start --root /data/everos` |
| `EVEROS_ROOT` env var | `export EVEROS_ROOT=/data/everos` |
| Default | `~/.everos` |
All CLI commands that interact with storage accept `--root`:
```bash
everos server start --root /data/everos
everos cascade status --root /data/everos
everos config show --root /data/everos
everos init --root /data/everos
```
## Configuration Reference
### `[memory]`
| Field | Type | Default | Description |
|---|---|---|---|
| `timezone` | string | `"UTC"` | Effective timezone for date buckets and timestamps. Validated against `zoneinfo.ZoneInfo`. |
### `[api]`
| Field | Type | Default | Description |
|---|---|---|---|
| `host` | string | `"127.0.0.1"` | HTTP server bind address. |
| `port` | int | `8000` | HTTP server bind port (1–65535). |
### `[sqlite]`
| Field | Type | Default | Description |
|---|---|---|---|
| `journal_mode` | string | `"WAL"` | PRAGMA journal_mode. Options: WAL, DELETE, MEMORY, OFF, TRUNCATE, PERSIST. |
| `synchronous` | string | `"NORMAL"` | PRAGMA synchronous. Options: FULL, NORMAL, OFF, EXTRA. |
| `foreign_keys` | bool | `true` | PRAGMA foreign_keys. |
| `temp_store` | string | `"MEMORY"` | PRAGMA temp_store. Options: DEFAULT, FILE, MEMORY. |
| `busy_timeout_ms` | int | `5000` | PRAGMA busy_timeout in milliseconds. |
| `journal_size_limit_bytes` | int | `67108864` | PRAGMA journal_size_limit (~64 MB). |
| `cache_size_kb` | int | `2048` | PRAGMA cache_size in KB (per connection). |
### `[lancedb]`
| Field | Type | Default | Description |
|---|---|---|---|
| `read_consistency_seconds` | float \| null | `null` | Read consistency interval. `null` = no check, `0` = strict, `>0` = eventual. |
| `index_cache_size_bytes` | int | `16777216` | Upper bound on LanceDB index cache (16 MB default). |
### `[llm]`
| Field | Type | Default | Required | Description |
|---|---|---|---|---|
| `model` | string | `"gpt-4.1-mini"` | No | LLM model identifier. |
| `api_key` | string | — | **Yes** | API key for the LLM provider. |
| `base_url` | string | — | No | Custom endpoint URL (OpenAI-compatible). |
### `[multimodal]`
| Field | Type | Default | Required | Description |
|---|---|---|---|---|
| `model` | string | `"google/gemini-3-flash-preview"` | No | Multimodal parsing model. |
| `api_key` | string | — | **Yes** | API key. |
| `base_url` | string | — | No | Custom endpoint URL. |
| `max_concurrency` | int | `4` | No | Max parallel parsing requests. |
### `[embedding]`
| Field | Type | Default | Required | Description |
|---|---|---|---|---|
| `model` | string | — | **Yes** | Embedding model identifier. |
| `api_key` | string | — | **Yes** | API key. |
| `base_url` | string | — | **Yes** | Embedding endpoint URL. |
| `timeout_seconds` | float | `30.0` | No | Request timeout. |
| `max_retries` | int | `3` | No | Retry count on failure. |
| `batch_size` | int | `10` | No | Texts per batch request. |
| `max_concurrent` | int | `5` | No | Max parallel batch requests. |
### `[rerank]`
| Field | Type | Default | Required | Description |
|---|---|---|---|---|
| `provider` | string | `"deepinfra"` | No | Rerank provider: `deepinfra` or `vllm`. |
| `model` | string | — | **Yes** | Reranker model identifier. |
| `api_key` | string | — | **Yes** | API key. |
| `base_url` | string | — | **Yes** | Rerank endpoint URL. |
| `timeout_seconds` | float | `30.0` | No | Request timeout. |
| `max_retries` | int | `3` | No | Retry count on failure. |
| `batch_size` | int | `10` | No | Documents per batch. |
| `max_concurrent` | int | `5` | No | Max parallel batch requests. |
### `[boundary_detection]`
| Field | Type | Default | Description |
|---|---|---|---|
| `hard_token_limit` | int | `65536` | Max tokens before forced boundary. |
| `hard_msg_limit` | int | `500` | Max messages before forced boundary. |
### `[memorize]`
| Field | Type | Default | Description |
|---|---|---|---|
| `mode` | string | `"agent"` | Conversation mode: `chat` (user-memory only) or `agent` (user + agent memory). Requires restart. |
| `session_lock_timeout_seconds` | float | `360.0` | Max wall-clock per memorize() invocation. |
### `[clustering]`
| Field | Type | Default | Description |
|---|---|---|---|
| `threshold` | float | `0.65` | Cosine similarity threshold for clustering (0–1). |
| `time_window_days` | float | `7.0` | Max age gap between cluster members. |
### `[search]`
| Field | Type | Default | Description |
|---|---|---|---|
| `vector_strategy` | string | `"maxsim_atomic"` | Vector retrieval path: `maxsim_atomic` (finer-grained) or `episode` (legacy). |
### `[knowledge.search]`
| Field | Type | Default | Description |
|---|---|---|---|
| `recall_n` | int | `200` | Initial recall pool size. |
| `rerank_n` | int | `50` | Candidates sent to reranker. |
| `mass_top_m` | int | `50` | Top-M for mass scoring. |
| `lambda` | float | `0.1` | Interpolation weight. |
| `top_k_cap` | int | `100` | Hard cap on returned results. |
## Troubleshooting
Use `everos config show` to inspect the effective configuration:
```bash
everos config show
everos config show --root /data/everos
```
This prints:
- The resolved root directory
- Which config files were found
- All settings sections with their effective values
- API keys are masked in output
## Advanced: Container Deployment
In containerized environments, skip `everos init` and use environment
variables directly:
```dockerfile
ENV EVEROS_ROOT=/data/everos
ENV EVEROS_LLM__API_KEY=sk-...
ENV EVEROS_LLM__MODEL=gpt-4o
ENV EVEROS_EMBEDDING__MODEL=text-embedding-3-large
ENV EVEROS_EMBEDDING__API_KEY=sk-...
ENV EVEROS_EMBEDDING__BASE_URL=https://api.openai.com/v1
ENV EVEROS_API__HOST=0.0.0.0
ENV EVEROS_API__PORT=8000
```
Environment variable naming convention:
```
EVEROS_<SECTION>__<KEY>
```
- Section and key are uppercased
- Double underscore (`__`) separates section from key
- Nested sections use additional `__` separators
Examples:
| TOML | Environment Variable |
|---|---|
| `[llm] api_key = "sk-..."` | `EVEROS_LLM__API_KEY=sk-...` |
| `[sqlite] busy_timeout_ms = 10000` | `EVEROS_SQLITE__BUSY_TIMEOUT_MS=10000` |
| `[memory] timezone = "Asia/Tokyo"` | `EVEROS_MEMORY__TIMEZONE=Asia/Tokyo` |

20
docs/datetime.md
View File

@ -66,8 +66,8 @@ All helpers live in [`everos.component.utils.datetime`](../src/everos/component/
| Helper | Behaviour |
|---|---|
| `get_utc_now() -> datetime` | Current UTC instant, `tzinfo=UTC`. Independent of any setting. Use as `default_factory` on any storage field. |
| `ensure_utc(d) -> datetime` | Naive → attach display tz → convert to UTC. Aware → `astimezone(UTC)`. Use at the storage boundary if you receive a datetime you didn't construct. |
| `UtcDatetime` | `Annotated[datetime, AfterValidator(ensure_utc)]`. Apply to any SQLite field. Pydantic auto-runs validation on both INSERT defaults and read-back rows. |
| `ensure_utc(d) -> datetime \| None` | Naive → **assume UTC** (attach `tzinfo=UTC`; no display-tz step). Aware → `astimezone(UTC)`. `None` → `None`. Use at the storage boundary; for caller input that may be naive-in-display-tz, funnel through `from_iso_format` first. |
| `UtcDatetime` | `Annotated[datetime, AfterValidator(ensure_utc)]` — normalises on **construction**. The ORM hydrate path bypasses it; UTC-on-read is handled by the `UtcDateTimeColumn` SQL type (below). |
### Display rail
@ -107,8 +107,9 @@ SQLModel's ORM hydrate path (rows from `select(...)`) **bypasses**
the Pydantic validator — SQLAlchemy assigns column values straight
to instance attributes. To close that gap,
[core/persistence/sqlite/base.py](../src/everos/core/persistence/sqlite/base.py)
registers a SQLAlchemy `load` event listener that re-attaches
`tzinfo=UTC` to every `UtcDatetime` column after hydrate. Net effect:
defines a `TypeDecorator` column type, `UtcDateTimeColumn`, whose
`process_result_value` re-attaches `tzinfo=UTC` on read (and
`process_bind_param` normalises to UTC on write). Net effect:
**callers never see a naive datetime from a SQLite repo**, whatever
the code path.
@ -154,8 +155,8 @@ event payload.
| Backend | Defense | Where |
|---|---|---|
| **SQLite** | SQLAlchemy `load` event listener on `BaseTable` re-attaches `tzinfo=UTC` after every ORM hydrate | [core/persistence/sqlite/base.py](../src/everos/core/persistence/sqlite/base.py) |
| **LanceDB** | `BaseLanceTable.to_arrow_schema()` rewrites `UTC_DATETIME_FIELDS` columns to `timestamp[us, tz=UTC]`; PyArrow handles UTC end-to-end | [core/persistence/lancedb/base.py](../src/everos/core/persistence/lancedb/base.py) |
| **SQLite** | `UtcDateTimeColumn` (`TypeDecorator`) re-attaches `tzinfo=UTC` on read (`process_result_value`) and normalises to UTC on write (`process_bind_param`) | [core/persistence/sqlite/base.py](../src/everos/core/persistence/sqlite/base.py) |
| **LanceDB** | `BaseLanceTable.to_arrow_schema()` rewrites **every** `timestamp[us]` column to `timestamp[us, tz=UTC]`; PyArrow handles UTC end-to-end | [core/persistence/lancedb/base.py](../src/everos/core/persistence/lancedb/base.py) |
| **CI gate** | `scripts/check_datetime_discipline.py` fails the build on any code that bypasses `component/utils/datetime` | wired into `make lint` |
These defenses replace what used to be an "every consumer must call
@ -176,9 +177,10 @@ User input (any zone)
▼
┌────────────────┬────────────────┐
│ SQLite │ LanceDB │
│ (UtcDatetime │ (Arrow │
│ re-attaches │ stripped to │
│ UTC on read) │ UTC bytes) │
│ (UtcDateTime- │ (Arrow │
│ Column re- │ stripped to │
│ attaches UTC │ UTC bytes) │
│ on read) │ │
└────────────────┴────────────────┘
│
▼

308
docs/engineering.md
View File

@ -4,7 +4,6 @@
> hard coding constraints live in [../.claude/rules/](../.claude/rules/).
> This document covers the surrounding tooling, configuration, and processes
> — what we adopted, what role each piece plays, and how they fit together.
> CI runs on GitHub Actions; all checks are invoked through the `Makefile`.
---
@ -56,7 +55,7 @@ Reasons this is documented separately:
│ │ .claude/ │ │
│ │ ├── CLAUDE.md subdir context (optional) │ │
│ │ ├── rules/ (10) path-scoped hard coding rules │ │
│ │ ├── skills/ (3) slash command workflows │ │
│ │ ├── skills/ (5) slash command workflows │ │
│ │ └── settings.json permissions allowlist │ │
│ │ │ │
│ └────────────────────────────────────────────────────────────┘ │
@ -69,13 +68,11 @@ Reasons this is documented separately:
│ │ ├ check-yaml / check-toml │ │
│ │ ├ check-added-large-files (≥1MB warn) │ │
│ │ ├ detect-private-key │ │
│ │ ├ no committed images/videos/assets │ │
│ │ └ gitlint (commit-msg stage) │ │
│ │ │ │
│ │ ruff lint + format │ │
│ │ (replaces black / isort / flake8) │ │
│ │ import-linter DDD layer-direction enforcement │ │
│ │ repo asset gate blocks images/videos/assets in git │ │
│ │ pytest unit / integration │ │
│ │ │ │
│ └────────────────────────────────────────────────────────────┘ │
@ -88,26 +85,25 @@ Reasons this is documented separately:
│ │ uv.lock checked in; CI uses --frozen │ │
│ │ hatchling wheel build backend │ │
│ │ Makefile unified entry; CI calls it │ │
│ │ src/everos/templates/env.template │ │
│ │ environment variable template │ │
│ │ config/default.toml default settings (shipped) │ │
│ │ │ │
│ └────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─ CI/CD (GitHub Actions) ───────────────────────────────────┐ │
│ ┌─ Dual-platform CI/CD ──────────────────────────────────────┐ │
│ │ │ │
│ │ CI: .github/workflows/ci.yml lint / test / integ │ │
│ │ / package build │ │
│ │ Docs: .github/workflows/docs.yml Markdown + YAML check │ │
│ │ Gates invoke Makefile targets; the Makefile is the │ │
│ │ Primary: GitLab CI .gitlab-ci.yml │ │
│ │ Mirror: GitHub Actions .github/workflows/ci.yml │ │
│ │ Both invoke Makefile targets; the Makefile is the │ │
│ │ single source of truth for commands. │ │
│ │ │ │
│ └────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─ Collaboration workflow ───────────────────────────────────┐ │
│ │ │ │
│ │ Branch model: protected main + short-lived PR branches │ │
│ │ PR template: .github/PULL_REQUEST_TEMPLATE.md │ │
│ │ ISSUE_TEMPLATE: bug / feature / use-case / docs / config │ │
│ │ Branch model: dev / master (GitFlow Lite) │ │
│ │ PR / MR templates: same template across platforms │ │
│ │ CODEOWNERS: by DDD layer ownership │ │
│ │ ISSUE_TEMPLATE: bug / feature / config │ │
│ │ CONTRIBUTING.md: contributor onboarding │ │
│ │ │ │
│ └────────────────────────────────────────────────────────────┘ │
@ -158,13 +154,15 @@ session start (no manual import):
load (~1.5–2K tokens); the rest load on demand when Claude Code reads a
matching `.py` file.
### 3.3 Skills (3 slash commands)
### 3.3 Skills (5 slash commands)
| Command | Purpose | When to use |
|---|---|---|
| `/commit` | Generate a Conventional Commits message | After a focused change, ready to commit |
| `/new-branch` | Create branch from protected main | Starting a new feat / fix / ci branch |
| `/pr` | Open a GitHub PR with the repo template | Ready to merge |
| `/commit` | Generate Gitmoji-format commit message | After a focused change, ready to commit |
| `/new-branch` | Create branch under dev/master strategy | Starting a new feat / fix / hotfix |
| `/pr` | Create GitLab MR or GitHub PR with template | Ready to merge |
| `/add-memory-kind` | Scaffold a new business memory kind end-to-end | Adding a new memory type (md + sqlite + lancedb) |
| `/release` | Cut a PyPI release (rc / stable) | Shipping a version |
Skills and rules use **independent loading mechanisms**: rules auto-load
into the system prompt, skills only trigger when the user types `/<name>`.
@ -206,26 +204,25 @@ Stage 2: pre-commit (triggered by `git commit`)
├ trailing-whitespace, end-of-file-fixer
├ check-yaml, check-toml
├ check-added-large-files (≥1MB)
├ check-merge-conflict
├ detect-private-key
├ no-repo-assets (rejects images/videos/assets in git)
└ gitlint (commit-msg stage; rejects malformed messages)
│
▼
Stage 3: local `make ci` (manual, before push)
├ make lint (ruff + import-linter + repo hygiene gates)
├ make lint (ruff + format-check + import-linter + datetime + openapi-drift)
├ make test (pytest tests/unit)
├ make integration (pytest tests/integration)
└ make package (sdist/wheel build + import smoke test)
└ make integration (pytest tests/integration)
│
▼
Stage 4: CI (GitHub Actions, push + PR triggered)
└ re-runs the same `make lint / test / integration / package` targets
Stage 4: CI (PR triggered, GitLab + GitHub)
└ re-runs the same `make lint / test / integration` targets
│
▼
Stage 5: PR review
Stage 5: PR / MR review
├ ≥ 1 approval
└ all threads resolved + all CI green
```
@ -260,12 +257,13 @@ everos = "everos.entrypoints.cli.main:app" # exposes CLI command
[tool.ruff] # code style
[tool.pytest.ini_options] # tests
[tool.coverage.run] # coverage config (gate lives in `make cov`)
[tool.coverage.run] # coverage (unit+integration; make cov gates at 80%)
[tool.importlinter] # dependency direction
[dependency-groups]
dev = ["ruff", "pytest", "pytest-asyncio", "pytest-cov",
"import-linter", "pre-commit", "ipdb"]
"pytest-rerunfailures", "import-linter", "pre-commit",
"pyinstrument", "ipdb"]
```
**Single-file principle**: configuration that used to live in `pylintrc`,
@ -277,60 +275,68 @@ dev = ["ruff", "pytest", "pytest-asyncio", "pytest-cov",
make help list all targets
make install uv sync --frozen
make format ruff fix + format
make lint ruff + import-linter + repo asset/media + datetime discipline + openapi drift
make lint ruff + format --check + import-linter + datetime + openapi-drift
make test pytest tests/unit
make integration pytest tests/integration
make package build sdist/wheel + smoke-test wheel import
make cov pytest unit + integration, coverage gate (fail under 80%)
make ci lint + test + integration + package
make cov pytest unit+integration with coverage, gate at 80%
make ci lint + test + integration ← CI invokes these targets
make clean clear caches
```
**Single source of truth**: CI only invokes `make <target>`, so local and CI
run identical commands and cannot drift.
> Plus `install-deps` (CI's `uv sync --frozen`), `openapi` (regenerate
> `docs/openapi.json`), `check-openapi` / `check-datetime` (the lint
> sub-gates), and `check-cjk` (advisory) — see the `Makefile` for the
> full list.
### 5.3 env.template (slimmed down)
**Single source of truth**: CI configuration only invokes `make <target>`,
preventing drift between GitHub and GitLab. Local and CI run identical
commands.
The template lives at `src/everos/templates/env.template` (bundled
inside the wheel as package data, copied to `./.env` via `everos init`).
It groups settings by provider, each block sharing the OpenAI-protocol
`MODEL` / `API_KEY` / `BASE_URL` triple:
### 5.3 Configuration model
Settings are loaded in ascending priority:
1. `config/default.toml` (shipped with the package; lowest priority)
2. `<memory-root>/everos.toml` (user config; optional)
3. `EVEROS_*` environment variables (highest priority)
`everos init [--root PATH]` generates starter config files
(`everos.toml` + `ome.toml`) in the memory root. `everos.toml` holds
all config (API keys, model, storage); override individual fields with
`EVEROS_*` env vars. Inspect the effective config with
`everos config show [--root PATH]`.
Key `EVEROS_*` env vars:
```
EVEROS_LLM__* # text model (model / api_key / base_url)
EVEROS_MULTIMODAL__* # vision model for image/office inputs
EVEROS_EMBEDDING__* # embedding model (vector index)
EVEROS_RERANK__* # cross-encoder reranker
EVEROS_MEMORY__ROOT # memory-root (md files + .index/{sqlite,lancedb}/)
EVEROS_LOG_LEVEL # DEBUG | INFO | WARNING | ERROR
EVEROS_LOG_FORMAT # json | text
TZ # display timezone (storage is always UTC)
EVEROS_LLM__MODEL # model name (provider-agnostic)
EVEROS_LLM__API_KEY # any OpenAI-protocol API key
EVEROS_LLM__BASE_URL # optional: custom endpoint (Ollama bridge etc.)
EVEROS_ROOT # memory-root (default ~/.everos)
EVEROS_LOG_LEVEL
TZ
```
Every key has a sensible default except the `API_KEY` fields, which you fill in.
---
## 6. CI/CD (GitHub Actions)
## 6. Dual-platform CI/CD
### 6.1 Strategy
### 6.1 Dual-platform strategy
```
┌──────────────────────────────────────────────────────────┐
│ │
│ GitHub Actions (.github/workflows/) │
│ ci.yml push (main) + PR │
│ ├ lint make lint │
│ ├ unit tests make test │
│ ├ integration tests make integration │
│ └ package build make package │
│ docs.yml Markdown link check + issue-template YAML │
│ └ make docs-check │
│ commits.yml Conventional Commit subject check │
│ └ make check-commits │
│ Primary: GitLab CI (.gitlab-ci.yml) │
│ ├ internal team dev stages: lint / test │
│ ├ MR triggered │
│ └ uv cache (keyed by uv.lock) │
│ │
│ Mirror: GitHub Actions (.github/workflows/ci.yml) │
│ ├ public OSS mirror same make targets │
│ ├ push + PR triggered │
│ └ astral-sh/setup-uv@v3 │
│ │
│ Consistency: │
│ ├ astral-sh/setup-uv (cache keyed by uv.lock) │
│ ├ Makefile is the single source of CI commands │
│ └ pre-commit runs locally first to reduce CI churn │
│ │
@ -339,77 +345,102 @@ Every key has a sensible default except the `API_KEY` fields, which you fill in.
### 6.2 CI checklist
| Check | Tool | Failure condition |
|---|---|---|
| Lint | `make lint` (ruff check + ruff format --check) | any error |
| Layer direction | `make lint` (lint-imports inside) | layer violation |
| Repository media | `make lint` (check_repo_assets.py) | images/videos/assets committed |
| Datetime discipline | `make lint` (check_datetime_discipline.py) | bypasses helper module |
| OpenAPI drift | `make lint` (dump_openapi.py --check) | schema ≠ committed openapi.json |
| Unit | `make test` (pytest tests/unit) | any failure |
| Integration | `make integration` (pytest tests/integration) | any failure |
| Package build | `make package` (sdist/wheel + import smoke test) | build or import failure |
| Commit message | `Commit lint` workflow | non-Conventional Commit subject |
| Check | Tool | Platform | Failure condition |
|---|---|---|---|
| Lint | `make lint` (ruff + format-check + import-linter + datetime + openapi-drift) | both | any error |
| Layer direction | `make lint` (lint-imports inside) | both | layer violation |
| Unit | `make test` (pytest tests/unit) | both | any failure |
| Integration | `make integration` (pytest tests/integration) | both | any failure (PR + master/dev push only) |
Integration tests run with a `FakeLLMClient` — no live credentials are needed in CI.
Commit message format is enforced locally via `gitlint` in the `commit-msg`
pre-commit stage and remotely via the `Commit lint` workflow.
Commit message format is enforced **locally** via `gitlint` in the
`commit-msg` pre-commit stage; it does not run in CI.
### 6.3 Branch protection
| Branch | Rule |
|---|---|
| **main** | branch protection: PR + two reviews + green required checks; no direct push |
| feat / fix / docs / ci | contributor branches; merge through PR |
| Branch | GitLab rule | GitHub rule |
|---|---|---|
| **master** | no direct push; MR + 1 approval + green pipeline | branch protection + 1 review + status checks |
| **dev** | same as above | same as above |
| feat / fix / hotfix | free push; rebase parent before merge | same |
---
## 7. Collaboration workflow
### 7.1 Branch model
EverOS uses a simple protected-main model after the 1.0 history reset:
### 7.1 Branch model (GitFlow Lite)
```
main ●────●────●────●────► protected, releasable
▲ ▲ ▲
│ │ └─ PR from ci/*
│ └────── PR from fix/*
└─────────── PR from feat/*
v0.1 v0.2 v1.0
▲ ▲ ▲
│ release PR │ release PR │ release PR
│ (dev→master+tag) │ (dev→master+tag) │ (dev→master+tag)
master ●──────────────────────●─────────────●──────────────────●──────────────────────────────────●────► stable / released
│ ▲ │ │
│ │ merge hotfix │ │
│ │ │ │
│ ●──●──┘ │ │
│ │ hotfix branch │ │
│ │ (cut from master) │ │
│ │ │ │
│ ▼ sync to dev │ │
│ │ │ │
dev ●──●──●──●──●──●──●──●──●─●──●──●─●──●──●──●──●──●──●──●──●─●──●──●──●──●──●──●──●──●──●──●──●─────► integration
▲ ↑ ↑ ↑
│ release point release point release point
feat/A (dev HEAD → (dev HEAD → (dev HEAD →
●──●──● master + v0.1) master + v0.2) master + v1.0)
feat/* : cut from dev → PR → merge into dev
hotfix/* : cut from master → merge into master + sync into dev (double merge)
release : dev → master + tag on master (no separate release branch)
Vertical │ in the diagram = "dev HEAD merged into master via release PR + v0.x tag"
```
All work starts from `main`, lands through a pull request, and requires green
checks. Force-pushing `main` is reserved only for repository recovery work.
Details in [../.claude/skills/new-branch/SKILL.md](../.claude/skills/new-branch/SKILL.md).
### 7.2 PR template
### 7.2 PR / MR template (shared across platforms)
A single PR template at [`.github/PULL_REQUEST_TEMPLATE.md`](../.github/PULL_REQUEST_TEMPLATE.md)
with five sections: **Summary / Area / Verification / Checklist / Notes for
Reviewers**. The `/pr` skill fills it in (see
[../.claude/skills/pr/SKILL.md](../.claude/skills/pr/SKILL.md)).
Six sections: changes / target branch / scope / API impact / tests /
checklist.
### 7.3 Commit convention (Conventional Commits)
File locations:
Format: `<type>[(scope)][!]: <description>` per
[Conventional Commits](https://www.conventionalcommits.org).
- GitLab: `.gitlab/merge_request_templates/default.md`
- GitHub: `.github/PULL_REQUEST_TEMPLATE.md`
### 7.3 CODEOWNERS (by DDD layer)
```
feat: new feature
fix: bug fix
refactor: restructuring (no behavior change)
test: add / update tests
docs: documentation
style: formatting
perf: performance optimization
chore: configuration / build / tooling
build: build system or dependencies
ci: CI configuration
revert: revert a previous commit
/src/everos/memory/ @chandler.zhang @libin.zhang001
/src/everos/infra/ @chandler.zhang @yeanhua
/src/everos/component/ @chandler.zhang
/src/everos/core/ @chandler.zhang
/src/everos/service/ @chandler.zhang @libin.zhang001
/src/everos/entrypoints/ @chandler.zhang
/.claude/ @chandler.zhang
/.gitlab-ci.yml @chandler.zhang @jianhua.yao
```
`gitlint` enforces the format locally via its `contrib-title-conventional-commits`
rule in the commit-msg pre-commit stage. GitHub Actions runs the same policy on
pushes to `main` and pull requests. See
At least one owner per directory; two owners for critical modules. Edits
auto-mention the corresponding owners.
### 7.4 Commit convention (Gitmoji)
```
✨ feat: new feature
🐛 fix: bug fix
♻️ refactor: refactoring (no behavior change)
✅ test: add / update tests
📝 docs: documentation
🎨 style: formatting
⚡️ perf: performance optimization
🔧 chore: configuration / build
🚧 wip: work in progress (must not land on master)
```
`gitlint` enforces format **locally** (commit-msg pre-commit stage). See
[../.claude/skills/commit/SKILL.md](../.claude/skills/commit/SKILL.md).
---
@ -418,11 +449,9 @@ pushes to `main` and pull requests. See
```
.github/ISSUE_TEMPLATE/
├── bug_report.yml structured bug report (form)
├── feature_request.yml feature proposal (form)
├── use_case.yml share a use case / integration
├── docs.yml documentation issue
└── config.yml disable blank issues + community links
├── bug_report.md software deps: lancedb / sqlite / ruff
├── feature_request.md generic template
└── config.yml disable blank issue + Discord / Discussions links
CONTRIBUTING.md contributor onboarding: setup / code style /
branch / commit / PR / testing
@ -441,7 +470,7 @@ CONTRIBUTING.md contributor onboarding: setup / code style /
│ │ │ context │
│ Team rules │ /.claude/rules/ (10) │ cc unaware │
│ │ │ of conv. │
│ Team skills │ /.claude/skills/ (3) │ no slash │
│ Team skills │ /.claude/skills/ (5) │ no slash │
│ │ │ workflows │
│ Permissions │ /.claude/settings.json │ cc prompts │
│ │ │ on each op │
@ -452,15 +481,18 @@ CONTRIBUTING.md contributor onboarding: setup / code style /
│ │ │ entry │
│ pre-commit │ /.pre-commit-config.yaml │ no local │
│ │ │ gate │
│ env template │ /src/everos/templates/env.template │ newcomers │
│ │ │ lost on env│
│ default config │ /src/everos/config/default.toml │ newcomers │
│ │ │ lost on cfg│
├─────────────────────┼──────────────────────────────────────┼─────────────┤
│ CI │ /.github/workflows/ci.yml │ PR cannot │
│ GitLab CI │ /.gitlab-ci.yml │ MR cannot │
│ │ │ merge │
│ Docs CI │ /.github/workflows/docs.yml │ broken │
│ │ │ doc links │
│ PR template │ /.github/PULL_REQUEST_TEMPLATE.md │ no PR temp │
│ Issue templates │ /.github/ISSUE_TEMPLATE/ (5) │ scattered │
│ GitHub Actions │ /.github/workflows/ci.yml │ PR cannot │
│ │ │ merge │
│ CODEOWNERS │ /.gitlab/CODEOWNERS │ no auto │
│ │ │ reviewer │
│ GitLab MR template │ /.gitlab/merge_request_templates/ │ no MR temp │
│ GitHub PR template │ /.github/PULL_REQUEST_TEMPLATE.md │ no PR temp │
│ Issue templates │ /.github/ISSUE_TEMPLATE/ (3) │ scattered │
│ CONTRIBUTING │ /CONTRIBUTING.md │ contrib. │
│ │ │ confused │
└─────────────────────┴──────────────────────────────────────┴─────────────┘
@ -472,19 +504,22 @@ CONTRIBUTING.md contributor onboarding: setup / code style /
```
Near-term
☑ Coverage threshold — `make cov` now gates at 80% (unit + integration)
□ /new-module skill: scaffold a subpackage that complies with rules
□ /run-eval skill: run behavior-consistency eval
□ ruff rule sets: add D (docstring), ANN (annotations)
□ Static type checking (pyright or mypy) once hot paths stabilize
Mid-term
Mid-term (before v0.5)
□ Type checking re-introduction (pyright or mypy) once hot paths stabilize
□ release-please / Conventional Commits → automated changelog
□ Automated PyPI wheel upload on tag
□ Multi-Python version matrix (3.12 / 3.13)
□ pre-commit autoupdate cadence
□ Performance benchmark CI with historical comparison
Long-term
Long-term (after v1.0)
□ /security-review skill: automated security review
□ Mutation testing (mutmut)
□ Coverage ratchet (raise the 80% gate as the suite matures)
□ Multi-Python version matrix (3.12 / 3.13)
□ Automated PyPI wheel upload
```
---
@ -500,7 +535,8 @@ Long-term
│ coding rules + │
│ quality gates (pre-commit + CI) + │
│ automation (Makefile + skills) + │
│ collaboration (branch + PR) + │
│ collaboration (branch + PR + │
│ CODEOWNERS) + │
│ knowledge base (CLAUDE.md + │
│ rules + docs) │
│ │
@ -518,10 +554,10 @@ Old project vs. new project after this rewrite:
| Config files | pyproject + pylintrc + pyrightconfig + pytest.ini | unified pyproject.toml |
| pre-commit | basic | adds gitlint commit-msg + import / yaml / private-key checks |
| Layer direction | not enforced | import-linter enforced in CI |
| Commit format | freeform | gitlint pre-commit hook (Conventional Commits) |
| Commit format | freeform | gitlint pre-commit hook (Gitmoji) |
| Claude Code integration | partial rules | rules + skills + settings (full) |
| CI platform | ad hoc | GitHub Actions calling Makefile targets |
| Tests | basic | unit + integration + e2e + coverage report |
| CI platform | GitLab only | GitLab + GitHub mirror, both calling Makefile |
| Tests | basic | unit + integration + golden + coverage report |
These are not perfectionism — they are baseline requirements for
**multi-person collaboration, long-term maintenance, and sustainable
@ -542,5 +578,7 @@ evolution**.
- gitlint: [jorisroovers.com/gitlint](https://jorisroovers.com/gitlint/)
- uv: [docs.astral.sh/uv](https://docs.astral.sh/uv/)
- pre-commit: [pre-commit.com](https://pre-commit.com/)
- Conventional Commits: [conventionalcommits.org](https://www.conventionalcommits.org/)
- Gitmoji: [gitmoji.dev](https://gitmoji.dev/)
- GitLab CI: [docs.gitlab.com/ee/ci](https://docs.gitlab.com/ee/ci/)
- GitHub Actions: [docs.github.com/en/actions](https://docs.github.com/en/actions)
- CODEOWNERS: [docs.gitlab.com/ee/user/project/codeowners](https://docs.gitlab.com/ee/user/project/codeowners/)

31
docs/how-memory-works.md
View File

@ -41,9 +41,8 @@ Three embedded pieces, each owning what it is best at. Markdown is the
## Storage paths
The default memory root is **`~/.everos/`** (override with
`EVEROS_MEMORY__ROOT` or `[memory] root` in TOML). Configuration (the
`.env` file) is separate from data (the memory root): the server searches
`./.env` → `$XDG_CONFIG_HOME/everos/.env` → `~/.everos/.env`.
`EVEROS_ROOT` env var or `--root` on the CLI). Configuration lives
inside the memory root as `everos.toml` (generated by `everos init`).
Memory is partitioned by **`<app_id>/<project_id>`** *before* the
user-visible directories, so different `(app, project)` spaces never share
@ -52,7 +51,7 @@ a directory or cross in search. The reserved id `"default"` materialises as
visually distinct from a user-named one).
```
~/.everos/ ← memory root (EVEROS_MEMORY__ROOT)
~/.everos/ ← memory root (EVEROS_ROOT)
├── default_app/ ← <app_id> ("default" → default_app)
│ └── default_project/ ← <project_id> ("default" → default_project)
│ ├── users/ ← user-visible (source of truth)
@ -97,8 +96,8 @@ visually distinct from a user-named one).
The path manager is
[`MemoryRoot`](../src/everos/core/persistence/memory_root.py); every path
above is a property on it. `MemoryRoot.ensure()` creates the runtime dirs
(`.index/{sqlite,lancedb}/`, `.tmp/`) and copies the OME template to
`ome.toml`; user-visible dirs appear on first write.
(`.index/{sqlite,lancedb}/`, `.tmp/`); user-visible dirs appear on first
write. Config files (`everos.toml`, `ome.toml`) are created by `everos init`.
## How a memory is born
@ -207,6 +206,10 @@ and write their markdown when ready:
- `extract_agent_case` — a reusable agent trajectory (only when the cell is
substantive enough; thin trajectories are skipped by design)
- `extract_agent_skill` — clusters related cases into a named skill
- `reflect_episodes` (cron, default off) — offline memory consolidation.
Merges fragmented episodes within a cluster into a single coherent
narrative, re-extracts atomic facts, and deprecates the originals.
Enable via `ome.toml`.
Strategies are configurable without a code change via **`ome.toml`** at the
memory root (hot-reloaded within ~2 s). Example — turn two off:
@ -224,6 +227,20 @@ and its scheduler jobstore in `.index/sqlite/ome.aps.db` (split so the sync
APScheduler writer and the async OME writer never contend for one file
lock).
### Reflection (offline consolidation)
The `reflect_episodes` strategy runs on a cron schedule (default
`0 2 * * 1` — every Monday at 02:00, disabled by default). It selects clusters with multiple
members, calls the LLM to merge their episodes into a single narrative,
writes the merged episode to markdown, re-extracts atomic facts, and
deprecates the originals via `deprecated_by`. The merged episode uses
`parent_type=cluster` and `session_id=None`. Enable it in `ome.toml`:
```toml
[strategies.reflect_episodes]
enabled = true
```
!!! tip "Implication for clients"
After `/flush` returns `extracted`, the **episode** is queryable soon
(once cascade indexes it), but **atomic facts / profile / agent cases**
@ -270,7 +287,7 @@ The CLI ([cli.md](cli.md)) is intentionally small:
| Command | What it does |
|---|---|
| `everos init` | write a starter `.env` |
| `everos init` | generate starter config files (`everos.toml` + `ome.toml`) |
| `everos server start` | run the HTTP API (cascade + OME start with it) |
| `everos cascade status` | queue / LSN summary |
| `everos cascade sync` | drain the cascade queue now (force md → LanceDB) |

5
docs/index.md
View File

@ -12,10 +12,11 @@ already know what you want to do and need to know exactly how.
| Doc | Purpose |
|---|---|
| [api.md](api.md) | HTTP API v1 reference — endpoints, request / response, error contracts |
| [knowledge.md](knowledge.md) | Knowledge base module — upload, search, taxonomy, storage layout |
| [reflection.md](reflection.md) | Reflection — offline memory consolidation: enable, schedule, storage, triggering |
| [cli.md](cli.md) | `everos` CLI subcommands + env var conventions |
| [storage_layout.md](storage_layout.md) | Memory-root tree + frontmatter chassis + EntryId encoding |
| [prompt_slots.md](prompt_slots.md) | YamlConfigLoader + three-layer prompt override |
| [migration-to-1.0.0.md](migration-to-1.0.0.md) | Legacy API and infrastructure migration notes for EverOS 1.0.0 |
## Explanation
@ -37,7 +38,6 @@ specific thing (drain a queue, recover from a stuck row, etc.).
| Doc | Purpose |
|---|---|
| [cascade_runbook.md](cascade_runbook.md) | Cascade subsystem ops — drain queue, recover stuck rows |
| [multimodal.md](multimodal.md) | Ingest images, PDFs, audio, and office docs into memory |
## Engineering / Internal
@ -54,7 +54,6 @@ Top-level project files live next to the repo root:
- [README.md](../README.md) — quick start & feature overview
- [QUICKSTART.md](../QUICKSTART.md) — 5-minute walkthrough (install → service → search)
- [use-cases.md](use-cases.md) — full use-case gallery and integration examples
- [CONTRIBUTING.md](../CONTRIBUTING.md) — how to contribute (issue-only model)
- [CHANGELOG.md](../CHANGELOG.md) — release notes
- [SECURITY.md](../SECURITY.md) — security policy & private vulnerability reporting

668
docs/knowledge.md Normal file
View File

@ -0,0 +1,668 @@
# Knowledge Base
The Knowledge module turns unstructured documents (Markdown, PDF, DOCX, …)
into a searchable topic library. Upload a file, and EverOS extracts a
structured topic tree via LLM, indexes it for keyword + vector search,
and keeps the original file for reference.
## Quick start
> The examples below assume EverOS is running on the default port 8000.
> See [README](../README.md) or [QUICKSTART](../QUICKSTART.md) to start
> the server.
```bash
# Upload a document
curl -s -X POST http://localhost:8000/api/v1/knowledge/documents \
-F "file=@my-report.pdf" \
-F "title=Q1 Engineering Report" \
| jq .data
# → { "doc_id": "d_a1b2c3d4e5f6", "category_id": "Technology", "topic_count": 8, ... }
# Search
curl -s -X POST http://localhost:8000/api/v1/knowledge/search \
-H "Content-Type: application/json" \
-d '{"query": "performance bottleneck", "method": "hybrid"}' \
| jq '.data.hits[:3] | .[] | {topic_name, score}'
```
## Three-tier hierarchy
Knowledge is organized into three levels, from broadest to most granular:
```
L0 Category ← taxonomy bucket (e.g., "Technology", "Finance")
L1 Document ← one uploaded file = one document
L2 Topic ← LLM-extracted section with content
```
Each level corresponds to a different granularity of API:
| Level | Endpoint | Returns |
|-------|----------|---------|
| L0 | `GET /categories` | `category_id`, `description`, `document_count` |
| L1 | `GET /documents` | `doc_id`, `title`, `category_id`, `topic_count`, `created_at` |
| L1 (detail) | `GET /documents/{id}` | Full detail: summary, source info, original file path, topic list |
| L2 | `GET /topics/{id}` | Full topic: content, labels, tree position |
## Storage layout
Every document is a self-contained directory. Markdown files are the
single source of truth; SQLite and LanceDB are derived indexes built
automatically by the cascade daemon.
```
~/.everos/<app>/<project>/knowledge/
├── .taxonomy.md ← category definitions (YAML)
├── Technology/
│ └── Q1_Engineering_Report_d_a1b2c3d4e5f6/
│ ├── index.md ← document metadata + summary
│ ├── 1_Performance_Analysis.md ← topic with full content
│ ├── 2_Infrastructure_Costs.md
│ ├── 3_Team_Velocity.md
│ └── _original/ ← original uploaded file
│ └── my-report.pdf
└── Finance/
└── Budget_Review_d_f6e5d4c3b2a1/
├── index.md
├── 1_Revenue.md
└── _original/
└── budget.xlsx
```
### Storage roles
```
Markdown (source of truth) + SQLite (structured state) + LanceDB (vector + BM25 index)
```
| Store | What it holds | Role |
|-------|---------------|------|
| Markdown | Document metadata, summaries, topic content, original files | Single source of truth; human-readable and editable |
| SQLite | Document rows, topic rows (with content), change queue | Structured queries, paginated lists, count aggregation |
| LanceDB | Topic vectors, BM25 tokens, scalar fields | Search index (fully rebuildable from Markdown) |
Even if SQLite and LanceDB data is corrupted, as long as the Markdown
files are intact, the indexes can be fully rebuilt via the cascade daemon.
### Markdown format
**index.md** (document root):
```yaml
---
type: knowledge_document
id: d_a1b2c3d4e5f6
doc_id: d_a1b2c3d4e5f6
category_id: Technology
title: Q1 Engineering Report
schema_version: 1
source_name: my-report.pdf
source_type: file
---
This report covers Q1 engineering outcomes including performance
analysis, infrastructure costs, and team velocity metrics.
```
The body is an LLM-generated summary of the entire document.
**Topic files** (e.g., `1_Performance_Analysis.md`):
```yaml
---
type: knowledge_topic
id: d_a1b2c3d4e5f6_1
node_id: d_a1b2c3d4e5f6_1
doc_id: d_a1b2c3d4e5f6
category_id: Technology
topic_index: 1
topic_name: Performance Analysis
topic_path: Q1 Engineering Report > Performance Analysis
summary: Analysis of API latency, database query times, and caching hit rates.
depth: 1
parent_node_id: d_a1b2c3d4e5f6_0
children_node_ids: []
content_labels: ["performance", "latency", "caching"]
schema_version: 1
---
The P99 API latency dropped from 450ms to 120ms after the Redis
caching layer was deployed in week 6. Database query times improved
by 40% following the index optimization sprint...
```
The body is the full extracted content for this topic.
> The taxonomy file uses `kind` (not `type`) in its frontmatter to
> distinguish it from document and topic files, which use `type`.
### Original file preservation
The `_original/` subdirectory stores the uploaded binary file unchanged.
Users can locate the original via the `original_file_path` field returned
by `GET /documents/{doc_id}`.
The underscore prefix follows the Jekyll/Eleventy convention for
non-content directories that the cascade daemon should skip.
Lifecycle:
- **POST** (create) — writes `_original/<filename>`
- **PUT** (replace) — clears the old directory and writes the new file
- **DELETE** — removes the entire document directory including `_original/`
- **PATCH** (category change) — moves the whole directory; `_original/` follows
## Taxonomy
Categories are defined in `.taxonomy.md` at the knowledge root. EverOS
ships with 20 default categories:
| Category | Description |
|----------|-------------|
| Technology | CS, software, AI/ML, cloud, cybersecurity |
| Science | Physics, chemistry, biology, astronomy |
| Medical | Clinical medicine, drugs, public health |
| Finance | Securities, banking, accounting, fintech |
| Legal | Laws, contracts, compliance, IP |
| Education | Teaching, curriculum, e-learning |
| Business | Strategy, marketing, operations, HR |
| Engineering | Mechanical, civil, electrical engineering |
| Arts | Visual arts, music, literature, film |
| Sports | Athletics, fitness, sports science |
| Travel | Tourism, hospitality, transportation |
| Food | Culinary, nutrition, food safety |
| Environment | Climate, ecology, sustainability |
| Politics | Government, international relations, policy |
| History | Historical events, civilizations, historiography |
| Psychology | Cognitive science, behavioral psychology, mental health |
| Agriculture | Farming, crop science, agribusiness |
| RealEstate | Property development, urban planning, housing |
| Media | Journalism, social media, PR |
| Others | Fallback for unclassified documents |
### Customization
Edit `.taxonomy.md` directly to add, remove, or rename categories:
```yaml
---
kind: knowledge_taxonomy
categories:
- id: Technology
description: Computer science, software engineering, AI/ML.
- id: InternalOps
description: Company-specific operational procedures and runbooks.
- id: CustomerSuccess
description: Customer onboarding, support playbooks, case studies.
---
```
Taxonomy changes are **hot-reloaded** — no server restart needed. The
system reads `.taxonomy.md` from disk on every upload and category list
request, so edits take effect immediately.
When a document is uploaded, the LLM selects the best-matching category
from this list. If no category matches, the document falls back to
`Others`. You can also specify `category_id` explicitly in the upload
request to bypass LLM classification.
## API reference
All endpoints are under `/api/v1/knowledge`. Responses use the envelope
format `{"request_id": "...", "data": {...}}`. The `request_id` field is
omitted from examples below for brevity.
### Upload a document
```
POST /documents
Content-Type: multipart/form-data
```
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `file` | file | yes | The document to upload |
| `title` | string | yes | Human-readable title |
| `source_type` | string | no | Provenance type (`"file"`, `"url"`, …) |
| `category_id` | string | no | Skip LLM classification; use this category |
| `app_id` | string | no | Tenant app (default: `"default"`) |
| `project_id` | string | no | Tenant project (default: `"default"`) |
**Response** (201):
```json
{
"data": {
"doc_id": "d_a1b2c3d4e5f6",
"category_id": "Technology",
"topic_count": 8,
"source_name": "my-report.pdf",
"md_path": "/home/user/.everos/default_app/default_project/knowledge/Technology/Q1_Report_d_a1b2c3d4e5f6",
"original_file_path": "/home/user/.everos/.../Q1_Report_d_a1b2c3d4e5f6/_original/my-report.pdf"
}
}
```
`original_file_path` is the absolute path to the preserved upload, or `null`
when no binary was stored (e.g. an empty filename).
**Example — Python**:
```python
from pathlib import Path
import httpx
async def upload_document(file_path: str, title: str) -> dict:
async with httpx.AsyncClient(base_url="http://localhost:8000") as client:
with open(file_path, "rb") as f:
resp = await client.post(
"/api/v1/knowledge/documents",
files={"file": (Path(file_path).name, f)},
data={"title": title},
)
resp.raise_for_status()
return resp.json()["data"]
```
**Example — curl**:
```bash
curl -X POST http://localhost:8000/api/v1/knowledge/documents \
-F "file=@report.pdf" \
-F "title=Quarterly Report" \
-F "category_id=Finance"
```
### Replace a document
```
PUT /documents/{doc_id}
Content-Type: multipart/form-data
```
Same fields as POST. Atomic operation: if extraction fails, the old
document is restored from backup.
### Update metadata
```
PATCH /documents/{doc_id}
Content-Type: application/json
```
```json
{
"title": "Updated Title",
"category_id": "Finance"
}
```
Returns `updated_fields: ["title", "category_id"]`. Changing `category_id`
moves the document directory to the new category folder.
### Delete a document
```
DELETE /documents/{doc_id}
```
Returns 204 if the document did not exist (idempotent), or 200 with
`deleted_topics` count.
### List documents
```
GET /documents?page=1&page_size=20&sort_by=created_at&sort_order=desc
```
Optional filter: `category_id=Technology`. `sort_by` accepts `created_at`
(default), `updated_at`, or `title`; `sort_order` is `asc` or `desc`.
```json
{
"data": {
"documents": [
{
"doc_id": "d_a1b2c3d4e5f6",
"category_id": "Technology",
"title": "Q1 Engineering Report",
"topic_count": 8,
"created_at": "2026-06-24T10:00:00Z"
}
],
"total": 42,
"page": 1,
"page_size": 20
}
}
```
### Get document detail
```
GET /documents/{doc_id}
```
Returns full metadata, summary, original file path, and topic overview list.
```json
{
"data": {
"doc_id": "d_a1b2c3d4e5f6",
"category_id": "Technology",
"title": "Q1 Engineering Report",
"summary": "This report covers Q1 engineering outcomes...",
"source_name": "my-report.pdf",
"source_type": "file",
"original_file_path": "/home/user/.everos/.../Q1_Report_d_a1b2c3d4e5f6/_original/my-report.pdf",
"topics": [
{
"topic_id": "d_a1b2c3d4e5f6_1",
"topic_name": "Performance Analysis",
"topic_path": "Q1 Engineering Report > Performance Analysis",
"depth": 1,
"summary": "Analysis of API latency..."
}
],
"created_at": "2026-06-24T10:00:00Z",
"updated_at": "2026-06-24T10:00:00Z"
}
}
```
`original_file_path` is `null` for documents created before the original
file preservation feature, or when no file was attached.
### Get topic detail
```
GET /topics/{topic_id}
```
Returns the full topic content, tree structure, and labels.
```json
{
"data": {
"topic_id": "d_a1b2c3d4e5f6_1",
"doc_id": "d_a1b2c3d4e5f6",
"category_id": "Technology",
"topic_name": "Performance Analysis",
"topic_path": "Q1 Engineering Report > Performance Analysis",
"depth": 1,
"summary": "Analysis of API latency, database query times...",
"content": "The P99 API latency dropped from 450ms to 120ms...",
"content_labels": ["performance", "latency", "caching"],
"parent_topic_id": "d_a1b2c3d4e5f6_0",
"children_topic_ids": [],
"created_at": "2026-06-24T10:00:00Z",
"updated_at": "2026-06-24T10:00:00Z"
}
}
```
### Search
```
POST /search
Content-Type: application/json
```
```json
{
"query": "performance bottleneck",
"method": "hybrid",
"top_k": 10,
"include_content": true,
"score_threshold": 0.5
}
```
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `query` | string | — | Search query (required, 1–2000 chars) |
| `method` | string | `"hybrid"` | `"keyword"`, `"vector"`, or `"hybrid"` |
| `top_k` | int | 10 | Max results (1–100) |
| `include_content` | bool | false | Include full topic content in results |
| `score_threshold` | float | null | Drop results below this score |
**Search methods**:
- **keyword** — BM25 sparse retrieval over tokenized summary + content
- **vector** — Dense ANN over embedded summary vectors (requires embedding provider)
- **hybrid** — Parallel keyword + vector, fused with Reciprocal Rank Fusion (RRF), then cross-encoder reranking
All three methods embed the query and apply cross-encoder reranking, so
knowledge search requires **both** an embedding and a rerank provider —
there is no provider-free fallback (this is by design: no silent
degradation). The two failure modes map to distinct status codes:
- **Provider not configured** → `500 CONFIGURATION_ERROR` (a required
setting is missing; retrying will not help — set `EVEROS_EMBEDDING__*` /
`EVEROS_RERANK__*`).
- **Provider configured but failing/timing out at call time** →
`503 EXTERNAL_SERVICE_UNAVAILABLE` (transient; retryable).
**Response**:
```json
{
"data": {
"hits": [
{
"topic_id": "d_a1b2c3d4e5f6_1",
"category_id": "Technology",
"topic_name": "Performance Analysis",
"topic_path": "Q1 Engineering Report > Performance Analysis",
"depth": 1,
"summary": "Analysis of API latency...",
"content": "The P99 API latency dropped...",
"score": 0.92,
"retrieval_method": "hybrid",
"document": {
"doc_id": "d_a1b2c3d4e5f6",
"title": "Q1 Engineering Report",
"summary": "This report covers..."
}
}
],
"total": 3,
"took_ms": 245.6
}
}
```
### List categories
```
GET /categories
```
```json
{
"data": {
"categories": [
{"category_id": "Technology", "description": "Computer science...", "document_count": 12},
{"category_id": "Finance", "description": "Securities...", "document_count": 5},
{"category_id": "Others", "description": "Fallback...", "document_count": 0}
]
}
}
```
## Search pipeline
```
query ─→ embed ─→ keyword (BM25) ─┐
vector (ANN) ──┤─→ RRF fusion ─→ rerank ─→ top_k
```
1. **Embed** — the query is embedded using the configured embedding provider
2. **Recall** — dual-channel retrieval from LanceDB:
- BM25 channel: keyword matching on `summary_tokens` + `content_tokens`
- ANN channel: nearest-neighbor search on the `vector` column
- In `hybrid` mode, both channels run in parallel
3. **Fuse** — Reciprocal Rank Fusion merges the two candidate lists
4. **Rerank** — cross-encoder reranker rescores the top candidates
5. **Filter** — drop results below `score_threshold` and limit to `top_k`
### Configuration
Search tuning parameters in `src/everos/config/default.toml`:
```toml
[knowledge.search]
recall_n = 200 # initial recall pool size per channel
rerank_n = 50 # candidates sent to reranker
mass_top_m = 50 # category-aware retrieve pool
lambda = 0.1 # category boost weight
top_k_cap = 100 # hard cap on returned results
```
Override via environment variables:
```bash
export EVEROS_KNOWLEDGE__SEARCH__RECALL_N=500
export EVEROS_KNOWLEDGE__SEARCH__RERANK_N=100
```
## Cascade sync
The cascade daemon watches the knowledge Markdown directory for file
changes and keeps SQLite + LanceDB in sync.
```
md file written
→ FSEvents / watchdog detects change
→ worker picks up from queue (≤1s poll interval)
→ handler dispatched by file type:
index.md → KnowledgeDocumentHandler → SQLite upsert (metadata)
N_topic.md → KnowledgeTopicHandler → tokenize + embed + SQLite + LanceDB upsert
```
The topic handler uses a SHA-256 content digest to skip unchanged files —
re-embedding only happens when the content actually changes.
Typical latency from file write to search availability: **1–3 seconds**.
## Supported file formats
EverOS accepts text-based files natively. Binary formats require the
`everos[multimodal]` extra (depends on LibreOffice for document conversion).
| Category | Formats | Requires `[multimodal]` |
|----------|---------|:-----------------------:|
| Text | `.txt`, `.md`, `.csv`, `.tsv`, `.vtt` | No |
| Documents | `.pdf`, `.docx`, `.doc`, `.rtf`, `.odt`, `.pages` | Yes |
| Spreadsheets | `.xlsx`, `.xls`, `.ods`, `.numbers` | Yes |
| Presentations | `.pptx`, `.ppt`, `.odp`, `.key` | Yes |
| Web | `.html`, `.htm`, `.eml` | Yes |
| Images (OCR) | `.png`, `.jpg`, `.webp`, `.tiff`, `.bmp`, `.svg` | Yes |
| Audio (transcription) | `.mp3`, `.wav`, `.m4a`, `.amr`, `.aiff`, `.aac`, `.ogg`, `.flac` | Yes |
```bash
pip install everos[multimodal]
```
## Error handling
| HTTP | Error code | Scenario |
|------|-----------|----------|
| 404 | `NOT_FOUND` | Document or topic does not exist |
| 409 | `CONFLICT` | `doc_id` already exists (use PUT to replace) |
| 415 | `UNSUPPORTED_FORMAT` | File format not parseable |
| 422 | `INVALID_INPUT` | Empty/oversized query, empty title, invalid ID format |
| 500 | `CONFIGURATION_ERROR` | Embedding or rerank provider not configured |
| 503 | `EXTERNAL_SERVICE_UNAVAILABLE` | Configured embedding/rerank provider failing at call time |
| 503 | `CAPABILITY_UNAVAILABLE` | `everos[multimodal]` not installed |
All error responses include a human-readable `message` field:
```json
{
"error": {
"code": "NOT_FOUND",
"message": "Document 'd_abc123' not found"
}
}
```
## Multi-tenancy
All endpoints accept `app_id` and `project_id` parameters (default:
`"default"`). Data is fully isolated per tenant pair:
```bash
# Tenant A uploads
curl -X POST .../documents -F "file=@a.pdf" -F "title=A" \
-F "app_id=tenant_a" -F "project_id=proj_1"
# Tenant B cannot see Tenant A's data
curl .../documents?app_id=tenant_b&project_id=proj_1
# → { "documents": [], "total": 0 }
```
Storage paths, SQLite rows, and LanceDB indexes are all scoped by
`app_id` + `project_id`.
## End-to-end walkthrough
A complete workflow from upload to search:
```bash
BASE=http://localhost:8000/api/v1/knowledge
# 1. List available categories
curl -s "$BASE/categories" | jq '[.data.categories[] | .category_id]'
# → ["Technology", "Science", "Medical", ..., "Others"]
# 2. Upload a document
DOC_ID=$(curl -s -X POST "$BASE/documents" \
-F "file=@architecture-guide.md" \
-F "title=System Architecture Guide" \
| jq -r .data.doc_id)
echo "Created: $DOC_ID"
# 3. View document detail (with topic list)
curl -s "$BASE/documents/$DOC_ID" | jq '{
title: .data.title,
category: .data.category_id,
topics: [.data.topics[] | .topic_name],
original: .data.original_file_path
}'
# 4. Read a topic — pick the first from the detail response
TOPIC_ID=$(curl -s "$BASE/documents/$DOC_ID" \
| jq -r '.data.topics[0].topic_id')
curl -s "$BASE/topics/$TOPIC_ID" | jq '{
name: .data.topic_name,
path: .data.topic_path,
content: .data.content[:200],
labels: .data.content_labels
}'
# 5. Search (index is typically ready within 1–3 seconds)
sleep 3
curl -s -X POST "$BASE/search" \
-H "Content-Type: application/json" \
-d '{
"query": "how to handle service failures",
"method": "hybrid",
"top_k": 5,
"include_content": true
}' | jq '.data | {
total,
took_ms,
top_hit: .hits[0] | {topic_name, score, content: .content[:100]}
}'
# 6. Move document to a different category
curl -s -X PATCH "$BASE/documents/$DOC_ID" \
-H "Content-Type: application/json" \
-d '{"category_id": "Engineering"}' \
| jq .data.updated_fields
# → ["category_id"]
# 7. Clean up
curl -s -X DELETE "$BASE/documents/$DOC_ID" | jq .
```

21
docs/locomo_benchmark.md
View File

@ -18,8 +18,8 @@ locally using the `hybrid` and `agentic` search methods.
## Prerequisites
- Python **3.12**, [uv](https://docs.astral.sh/uv/)
- A `.env` at the repo root with the LLM / embedding credentials EverOS
needs:
- LLM / embedding credentials configured via `everos.toml` in the memory
root (generated by `everos init`) or `EVEROS_*` env vars:
- `EVEROS_LLM__MODEL`, `EVEROS_LLM__API_KEY`, `EVEROS_LLM__BASE_URL`
- `EVEROS_EMBEDDING__*`
- `EVEROS_RERANK__*`
@ -41,11 +41,11 @@ later with `--data-path` if you keep it elsewhere.
## 2. Start the server
```bash
EVEROS_MEMORY__ROOT=~/.everos \
EVEROS_ROOT=~/.everos \
uv run python -m everos.entrypoints.cli.main server start --port 8000
```
`EVEROS_MEMORY__ROOT` isolates one benchmark's corpus from another —
`EVEROS_ROOT` isolates one benchmark's corpus from another —
change it (or `rm -rf` it) whenever you want a clean run.
Leave the server running in one terminal; run the benchmark from
@ -74,7 +74,7 @@ bash tests/run_locomo_batch.sh \
--concurrency 2
```
The wrapper picks up `EVEROS_MEMORY__ROOT` from the environment so the
The wrapper picks up `EVEROS_ROOT` from the environment so the
cascade poll path matches the server's data root. If you set them
differently, pass `--corpus-path` explicitly.
@ -119,6 +119,17 @@ An aggregate accuracy table prints at the end of the wrapper run.
## Notes
- **Smoke mode**: add `--smoke` for a quick sanity check — 2 sessions
× 5 QA with reduced wait times. Not a scored run, just pipeline
verification. Pass it to `test_locomo.py` directly or via the batch
wrapper's `--` passthrough:
```bash
uv run python tests/test_locomo.py --smoke --base-url http://localhost:8000
# or via the wrapper
bash tests/run_locomo_batch.sh --conv-indices 0 --methods hybrid -- --smoke
```
- **Re-running on the same corpus**: add `--skip-add` to skip ingest and
reuse what's already in `~/.everos`. Useful when comparing methods
side by side.

57
docs/multimodal.md
View File

@ -85,14 +85,15 @@ brew install --cask libreoffice # macOS
sudo apt-get install -y libreoffice # Debian / Ubuntu
```
Without LibreOffice, **office uploads return `415`** with a clear error;
image / PDF / audio / HTML / email parsing is unaffected.
Without LibreOffice, **office uploads return `503`**
(`CAPABILITY_UNAVAILABLE`) with a clear error message; image / PDF /
audio / HTML / email parsing is unaffected.
### Configure the multimodal LLM
The parser uses its own LLM section, independent from `[llm]`. The model
must accept OpenAI `image_url` parts. `everos init` writes these into the
generated `.env`:
must accept OpenAI `image_url` parts. Configure these in `everos.toml`
(under `[multimodal]`) or via env vars:
```bash
EVEROS_MULTIMODAL__MODEL=google/gemini-3-flash-preview
@ -277,26 +278,38 @@ All fields bind from the environment via the parent `Settings`
## Errors and limits
Two failure classes behave differently. **Deterministic** problems
(nothing to parse, no handler, missing system dependency) **abort the
whole `/add` batch with `415`**. A **transient** multimodal-LLM failure
(timeout, rate-limit, the model rejecting the asset) **degrades just that
item** — the request still returns `200`, the item is marked
`parse_status="failed"` and contributes no text, and the rest of the
batch extracts normally.
Three failure classes behave differently:
| Condition | Result |
|---|---|
| Non-text item carries only `text` (no `uri` / `base64`) | `415` (batch aborted) |
| Extension / modality the parser has no handler for | `415` (batch aborted) |
| `base64` without a resolvable `ext` / MIME to dispatch on | `415` (batch aborted) |
| Office document but no LibreOffice (`soffice`) on host | `415` (batch aborted) |
| `file://` fails a guardrail (missing / non-regular / too large / outside allowlist) | `415` (batch aborted) |
| Multimodal **LLM call** fails (timeout / rate-limit / model rejects the asset) | **`200`** — that item is skipped (`parse_status="failed"`), the rest of the batch still extracts |
**Format errors** — the uploaded file format is invalid or not
recognized. These abort the batch with `415` (`UNSUPPORTED_FORMAT`):
The `415` body uses the standard error envelope with the parse-failure
reason in `error.message` — see
[api.md → POST /add](api.md#post-apiv1memoryadd).
| Condition | HTTP | `error.code` |
|---|---|---|
| Non-text item carries only `text` (no `uri` / `base64`) | `415` | `UNSUPPORTED_FORMAT` |
| Extension / modality the parser has no handler for | `415` | `UNSUPPORTED_FORMAT` |
| `base64` without a resolvable `ext` / MIME to dispatch on | `415` | `UNSUPPORTED_FORMAT` |
| `file://` fails a guardrail (missing / non-regular / too large / outside allowlist) | `415` | `UNSUPPORTED_FORMAT` |
**Capability errors** — the server is missing a required dependency.
These abort the batch with `503` (`CAPABILITY_UNAVAILABLE`). Unlike
transient errors, retrying will not help — admin action is required:
| Condition | HTTP | `error.code` |
|---|---|---|
| `everos[multimodal]` extra not installed | `503` | `CAPABILITY_UNAVAILABLE` |
| Office document but no LibreOffice (`soffice`) on host | `503` | `CAPABILITY_UNAVAILABLE` |
**Transient LLM errors** — the multimodal LLM call failed. These
degrade gracefully — the request still returns `200`, the affected
item is marked `parse_status="failed"` and contributes no text, and the
rest of the batch extracts normally:
| Condition | HTTP | Result |
|---|---|---|
| Multimodal LLM call fails (timeout / rate-limit / model rejects) | `200` | That item is skipped; the rest of the batch still extracts |
All error responses use the standard error envelope — see
[api.md → Errors](api.md#errors).
## Searching multimodal memory

1511
docs/openapi.json
View File

File diff suppressed because it is too large Load Diff

14
docs/prompt_slots.md
View File

@ -25,9 +25,12 @@ layer 3 is supplied at the call site.
## Loader
The category loader lives at
[`src/everos/component/config/loader.py`](../src/everos/component/config/loader.py)
as `YamlConfigLoader`:
The prompt-slots public entry point is
[`PromptLoader`](../src/everos/memory/prompt_slots/loader.py) (re-exported
from `everos.memory.prompt_slots`); it wraps the generic category loader
[`YamlConfigLoader`](../src/everos/component/config/loader.py). The generic
loader is shown below — `PromptLoader` is the prompt-slots-specific wrapper
over the same mechanism:
```python
from pathlib import Path
@ -68,7 +71,7 @@ output_schema:
participants: { type: array }
llm:
model: gpt-4o-mini
model: gpt-4.1-mini
temperature: 0.3
max_tokens: 2000
@ -106,6 +109,7 @@ forcing one model on the other gets clunky.
## See also
- [`src/everos/component/config/loader.py`](../src/everos/component/config/loader.py)
- [`src/everos/memory/prompt_slots/`](../src/everos/memory/prompt_slots/) — `PromptLoader` (prompt-slots public API)
- [`src/everos/component/config/loader.py`](../src/everos/component/config/loader.py) — generic `YamlConfigLoader`
- [`tests/unit/test_component/test_config/test_loader.py`](../tests/unit/test_component/test_config/test_loader.py)
- [`docs/architecture.md`](architecture.md) — layer placement

359
docs/reflection.md Normal file
View File

@ -0,0 +1,359 @@
# Reflection
Reflection periodically consolidates the memory fragments scattered across
many conversations into a single, chronologically-organized narrative. It
runs offline in the background: it merges the multiple Episodes inside one
similarity cluster into one, resolves stale information by keeping the
latest state, and soft-archives the originals it replaces — so memory gets
more accurate and more compact with use, instead of piling up into noise.
## Prerequisites
Reflection's merge step calls an LLM, and re-clustering the merged narrative
calls an embedding model. Configure both as you would for the rest of EverOS
— an OpenAI-compatible `[llm]` and `[embedding]` block in `<root>/everos.toml`,
or the matching `EVEROS_LLM__*` / `EVEROS_EMBEDDING__*` environment variables.
If a provider is unavailable, the affected clusters are skipped and logged
rather than failing the run.
## Quick start
> The examples below assume EverOS is running on the default port 8000.
> `<root>` is the EverOS memory root (see [QUICKSTART](../QUICKSTART.md)).
Reflection is off by default. Turn it on in `<root>/ome.toml` — **one line**:
```toml
[strategies.reflect_episodes]
enabled = true
```
Once enabled, it **runs automatically every Monday at 02:00**. This is how
Reflection is meant to work — nothing else to configure. To change the run
time, add a `cron` line (a schedule expression; optional):
```toml
[strategies.reflect_episodes]
enabled = true
cron = "0 3 * * 0" # optional: change the run time (here: Sundays at 03:00)
```
> **Don't run it too often.** Once a week at most is recommended. Each run
> is a lossy LLM merge; repeatedly re-consolidating the same memories can
> make the narrative *worse*, not better — which is why the default is
> weekly.
Config changes hot-reload (no restart needed). From then on, at each
scheduled time, Reflection consolidates each user's memory once.
**What does it produce, and where do I see it?** Each run **appends one
merged narrative** to the relevant user's Episode log, and marks the older
fragments it replaces as archived (removed from default search). Markdown is
the source of truth — just open the user's Episode log file:
```
<root>/default_app/default_project/users/<user_id>/episodes/episode-<date>.md
```
The new entry carries `parent_type: cluster` (Episodes produced by ordinary
conversation are `parent_type: memcell`). It looks roughly like this:
```markdown
---
owner_id: u_andrew
timestamp: 2026-10-11T02:00:00+00:00
parent_type: cluster # <- marks it as a Reflection merge product
parent_id: cl_a1b2c3d4e5f6
---
## Subject
Andrew's pet adoption journey
## Content
Andrew initially had no pets. He later adopted a dog named Toby, and then
adopted another dog named Buddy. He currently has two dogs.
```
A search on the topic afterwards returns this single complete narrative
rather than the scattered old fragments.
> To inspect which clusters were consolidated, how many entries were
> archived, etc., see [Auditing & troubleshooting](#auditing--troubleshooting)
> (advanced; not needed for everyday use).
> For debugging without waiting for the schedule, you can trigger a run by
> hand — see [Triggering a run](#triggering-a-run).
## How it works
Reflection runs *offline*, separate from the live conversation path. The
online path keeps extracting Episodes and clustering them; Reflection later
consumes those clusters — it never sits between a user and a response:
```
Online (never blocked) Offline (scheduled)
────────────────────── ───────────────────
conversation → Episode → Cluster ───► Reflection consolidates the clusters
```
A scheduled run processes every user across all app/project tenants that have
clusters.
After each conversation, EverOS extracts an **Episode** (a summary of a
conversation segment), and geometric clustering groups semantically similar,
time-adjacent Episodes into a **Cluster**. The same topic thus ends up
scattered as several point-in-time snapshots within one cluster:
```
Cluster cl_xxx
├── ep_0001 "Andrew has no pets yet" (August)
├── ep_0002 "Andrew adopted Toby" (September)
└── ep_0003 "Andrew also adopted Buddy" (October)
```
Reflection consolidates one cluster at a time, in four steps:
```
Select ─→ Merge ─→ Re-extract ─→ Deprecate
```
1. **Select** — pick clusters worth consolidating: not yet consolidated and
holding ≥ 2 members, or already consolidated and since joined by new
members. At most 10 clusters per run, largest first.
2. **Merge** — hand the cluster's Episodes to the LLM in chronological order
and merge them into one narrative: preserve facts, resolve contradictions
by keeping the latest state, restore the timeline, drop duplicates, and
end with the current state. A previously consolidated cluster is updated
incrementally — only the new fragments are folded into the existing
narrative.
3. **Re-extract** — the merged narrative is written to Markdown and triggers
re-extraction of atomic facts, keeping derived data consistent with it.
4. **Deprecate** — the replaced original Episodes and their atomic facts get
`deprecated_by` pointing at the new narrative; cluster membership is
updated; an audit record is written.
The result:
```
Cluster cl_xxx
└── ep_0042 "Andrew initially had no pets. He later adopted a dog named
Toby, then another named Buddy. He currently has two dogs."
(originals ep_0001 / ep_0002 / ep_0003 → deprecated_by = ep_0042)
```
The merged narrative is, structurally, just an ordinary Episode
(`parent_type="cluster"`) — transparent to retrieval, no search-pipeline
changes required. Default search excludes any memory carrying `deprecated_by`,
so a query like "how many pets does Andrew have" only hits the one complete
narrative.
## Storage layout
Memory uses Markdown as the single source of truth; SQLite and LanceDB are
derived indexes built automatically by the cascade daemon.
| Store | What it holds | Role |
|---|---|---|
| Markdown | Episode bodies, merged narratives, archive markers | Single source of truth; human-readable and editable |
| SQLite | Clusters and members, consolidation audit records | Structured state and queries |
| LanceDB | Vectors + BM25 index for Episodes / atomic facts | Search (rebuildable from Markdown) |
The **merged narrative** is written to the Episode daily-log Markdown; its
frontmatter marks that it came from a cluster:
```yaml
---
owner_id: u_andrew
timestamp: 2026-10-10T12:00:00+00:00
parent_type: cluster
parent_id: cl_a1b2c3d4e5f6
---
## Subject
Andrew's pet adoption journey
## Content
Andrew initially had no pets. He later adopted a dog named Toby, and then
adopted another dog named Buddy. He currently has two dogs.
```
The **replaced originals** are not deleted. Their file's frontmatter records
the archive mapping, and the index layer writes `deprecated_by`:
```yaml
---
# added to the original Episode file's frontmatter:
deprecated_entries:
ep_20260810_0001: ep_20261010_0042
ep_20260910_0002: ep_20261010_0042
---
```
> Soft-archive, not delete: even if SQLite / LanceDB are corrupted, as long
> as the Markdown is intact the indexes can be fully rebuilt — and every
> consolidation remains traceable back to its original content.
## Configuration
| Setting | Location | Default | Description |
|---|---|---|---|
| `reflect_episodes.enabled` | `<root>/ome.toml` | `false` | Set to `true` to enable (the only setting needed) |
| `reflect_episodes.cron` | `<root>/ome.toml` | `0 2 * * 1` | Run time, as a standard cron expression (`0 2 * * 1` = Mondays at 02:00); **optional**, omit to use the built-in default. Running more than weekly is not recommended |
| `clustering.threshold` | `<root>/everos.toml` | `0.65` | Clustering similarity threshold |
| `clustering.time_window_days` | `<root>/everos.toml` | `7.0` | Clustering time window (days) |
Two files, two scopes: `ome.toml` holds OME-strategy config (Reflection's
on/off switch and schedule); `everos.toml` holds general settings (clustering
and the like). Both live under the memory root, and you only write the keys
you want to override — everything else falls back to the shipped defaults in
`config/default.toml`, which you never edit by hand.
Changes to `ome.toml` hot-reload (~1–2s); no server restart needed.
> Setting `enabled` back to `false` stops the *next* run from starting; a run
> already in progress finishes normally.
## API reference
Reflection's normal mode of operation is the scheduled automatic run (see
[Quick start](#quick-start)). The endpoint below triggers a run **on
demand** — for testing, debugging, or when you want to consolidate
immediately. It is an auxiliary path, not the normal mode (there is no
dedicated CLI command).
### Triggering a run
```
POST /api/v1/ome/trigger
Content-Type: application/json
```
| Field | Type | Default | Description |
|---|---|---|---|
| `name` | string | — | Strategy name; use `reflect_episodes` |
| `timeout` | float | 120.0 | Max seconds to wait for the run to finish |
| `force` | bool | false | When `true`, runs even if `enabled=false` |
**Response**:
```json
{ "status": "ok", "name": "reflect_episodes" }
```
`status` is `"ok"` (finished) or `"timeout"` (did not finish in time); an
unknown strategy name returns 404.
**Example — Python**:
```python
import httpx
async def trigger_reflection() -> str:
async with httpx.AsyncClient(base_url="http://localhost:8000") as client:
resp = await client.post(
"/api/v1/ome/trigger",
json={"name": "reflect_episodes", "timeout": 120, "force": True},
)
resp.raise_for_status()
return resp.json()["status"]
```
**Example — curl**:
```bash
curl -X POST http://localhost:8000/api/v1/ome/trigger \
-H "Content-Type: application/json" \
-d '{"name": "reflect_episodes", "timeout": 120, "force": true}'
```
## Auditing & troubleshooting
> This section is **advanced**. For everyday use you don't need it — just
> read the Markdown (see [Quick start](#quick-start)). It's here for
> inspecting consolidation details or diagnosing problems.
Each run writes one `reflection_report` audit record, useful for reviewing
consolidation history:
| Field | Description |
|---|---|
| `cluster_id` | The cluster that was consolidated |
| `mode` | `init` (first merge) or `update` (incremental update) |
| `source_count` | Number of fragments merged |
| `merged_entry_id` | The merged-narrative Episode produced |
| `deprecated_fact_count` | Number of atomic facts archived alongside |
| `created_at` | Consolidation time |
```bash
sqlite3 <root>/.index/sqlite/system.db \
"SELECT cluster_id, mode, source_count, merged_entry_id
FROM reflection_report ORDER BY created_at DESC LIMIT 10;"
```
| Symptom | Likely cause |
|---|---|
| No consolidation records after triggering | No eligible clusters (a cluster needs ≥ 2 members) |
| Response `status: "timeout"` | Downstream re-extraction is slow; raise `timeout` and retry |
| Old fragments still appear in search | Index syncs asynchronously, usually 1–3s; wait and retry |
| 404 returned | Strategy name must be `reflect_episodes` |
## Design notes
Why Reflection is shaped the way it is:
- **Offline and scheduled.** Merging is a heavy, lossy LLM operation, so it
runs off the request path — conversations stay fast — and a weekly cadence
lets enough new fragments accumulate to be worth re-merging.
- **Soft-archive, never delete.** Originals stay in Markdown, so every
consolidation is traceable and the indexes can always be rebuilt from the
Markdown source of truth.
- **A merged narrative is just an Episode.** Reusing the Episode type means
search and every downstream consumer keep working unchanged — Reflection
introduces no new retrieval path.
## Limitations
- **Merging is lossy** — LLM consolidation may drop individual details. The
original fragments are retained in storage and remain traceable, but are
not in default search results.
- **Clustering is by similarity** — Reflection consolidates the output of
similarity clustering; a single cluster is not guaranteed to be strictly
one topic.
- **No one-click rollback yet** — originals are fully retained, but there is
currently no endpoint to undo a specific consolidation.
## End-to-end walkthrough
The walkthrough triggers a run by hand to demonstrate the full flow; in a
real deployment, once enabled it runs automatically on schedule, so this step
isn't needed.
```bash
BASE=http://localhost:8000/api/v1
# 1. With Reflection enabled (set enabled = true in <root>/ome.toml),
# trigger a run by hand here (for the demo; in production it runs on schedule)
curl -s -X POST "$BASE/ome/trigger" \
-H "Content-Type: application/json" \
-d '{"name": "reflect_episodes", "timeout": 120, "force": true}' \
| jq .
# → { "status": "ok", "name": "reflect_episodes" }
# 2. Review the consolidation audit
sqlite3 <root>/.index/sqlite/system.db \
"SELECT mode, source_count, merged_entry_id
FROM reflection_report ORDER BY created_at DESC LIMIT 1;"
# → init|3|ep_20261010_0042
# 3. Verify via search: the hit is the merged narrative, not the old fragments
curl -s -X POST "$BASE/memory/search" \
-H "Content-Type: application/json" \
-d '{"query": "how many pets does Andrew have", "top_k": 5}' \
| jq '.data.episodes[0] | {subject, episode, session_id}'
# → session_id is null on a merged narrative (the aggregation-product marker);
# episode holds the full narrative text
```
## See also
- [how-memory-works.md](how-memory-works.md) — Episodes and the memory extraction pipeline
- [storage_layout.md](storage_layout.md) — Markdown + SQLite + LanceDB stack
- [api.md](api.md) — full HTTP API reference

52
docs/release-notes-1.1.0.md Normal file
View File

@ -0,0 +1,52 @@
# EverOS 1.1.0 Release Notes
EverOS 1.1.0 expands the memory system beyond user episodes with first-class
knowledge management, reflection, and stronger operational guarantees around
search, persistence, and API contracts.
## Highlights
- Added Knowledge APIs and storage for document creation, listing, patching,
deletion, taxonomy handling, and knowledge-topic search.
- Added Reflection orchestration for periodically merging and refining episode
clusters.
- Expanded OME runtime support with event IDs, run-record storage migration,
configuration reload behavior, and testing harness improvements.
- Improved search and get behavior for agent-owned memory, including agent
cases, agent skills, and owner-type isolation.
- Reworked API error handling around typed exception handlers and consistent
error envelopes.
- Updated docs, OpenAPI schema, configuration examples, and test coverage for
the 1.1.0 surface area.
## Compatibility Notes
- Existing local TUI demo registration remains available in this PR.
- Existing DashScope rerank support is preserved; the DashScope provider file
is not replaced by the 1.1.0 archive.
- The update intentionally leaves directories outside the 1.1.0 archive scope,
including existing use-case and iOS demo material, untouched.
- Knowledge search requires configured embedding and rerank providers. Missing
providers now fail explicitly with configuration errors rather than silently
returning degraded results.
## Upgrade Notes
- Regenerate or review `docs/openapi.json` after route or DTO changes.
- Run `uv sync --frozen` against the updated `uv.lock`.
- Review `config.example.toml`, `src/everos/config/default.toml`, and
`src/everos/config/default_ome.toml` for new Knowledge and OME settings.
- If running e2e tests without live provider credentials, use dummy provider
environment variables for startup-only checks; live vector and rerank paths
remain behind slow/live markers.
## Verification
This PR was checked with:
- `uv run ruff check .`
- `uv run pytest tests/unit`
- `uv run pytest tests/integration`
- `uv run lint-imports`
- `uv run pytest tests/e2e` with dummy LLM and embedding environment variables
- Targeted search/get regression tests for agent and deprecated-filter handling

60
docs/storage_layout.md
View File

@ -9,8 +9,8 @@ derived indexes that can be rebuilt from markdown alone.
## 1. Memory-root tree
A memory-root is a single directory holding all persisted memory. The
default location is `~/.everos/`; override via `EVEROS_MEMORY__ROOT`
env var or `[memory] root` in the TOML config.
default location is `~/.everos/`; override via the `EVEROS_ROOT`
env var or `--root` on the CLI.
Memory is partitioned by **`<app_id>/<project_id>`** *before* the
user-visible scope dirs, so different `(app, project)` spaces never share
@ -64,9 +64,9 @@ the frontmatter (see [§3](#3-frontmatter-chassis-yaml)).
The path manager is [`MemoryRoot`](../src/everos/core/persistence/memory_root.py),
exposing every path as a property. `MemoryRoot.ensure()` creates the
runtime-required dirs (`.index/{sqlite,lancedb}/`, `.tmp/`) and copies the
OME template to `ome.toml`; the user-visible dirs are *not* pre-created —
they appear on first write.
runtime-required dirs (`.index/{sqlite,lancedb}/`, `.tmp/`); the
user-visible dirs are *not* pre-created — they appear on first write.
Config files (`everos.toml`, `ome.toml`) are created by `everos init`.
> The single-file writer also supports `agent.md` / `soul.md` / `tools.md`
> / `behaviors.md`, but no shipped strategy produces those today — only
@ -89,11 +89,8 @@ business-aware writers live in
[`infra/persistence/markdown/writers/`](../src/everos/infra/persistence/markdown/writers/)
and pick the right strategy via a base class.
To add a new memory kind, define its per-kind frontmatter schema under
[`infra/persistence/markdown/mds/`](../src/everos/infra/persistence/markdown/mds/)
and add a matching writer/reader pair under
[`writers/`](../src/everos/infra/persistence/markdown/writers/) and
[`readers/`](../src/everos/infra/persistence/markdown/readers/).
For a step-by-step recipe to add a new memory kind, see the
[`/add-memory-kind`](../.claude/skills/add-memory-kind/SKILL.md) skill.
## 3. Frontmatter chassis (YAML)
@ -178,20 +175,30 @@ Implementation: [`core/persistence/markdown/entries.py`](../src/everos/core/pers
```
.index/
├── sqlite/
│ └── system.db state / audit log / task queue / LSN watermark
│ + per-kind business state tables (composite key)
│ └── system.db state / audit / cascade queue + buffer / LSN
│ (system tables: md_change_state, memcell,
│ unprocessed_buffer, conversation_status, cluster)
└── lancedb/
└── <kind>.lance/ one Arrow-based table per kind
stores text / vector / tags / metadata
└── <kind>.lance/ one Arrow table per business kind — the per-kind
rows (text / vector / tokens / metadata) live here
```
- **SQLite** schema lives in
[`infra/persistence/sqlite/tables/`](../src/everos/infra/persistence/sqlite/tables/);
every business table that joins back to markdown declares a
`UniqueConstraint("user_id", "entry_id")` (or `agent_id` symmetric).
- **LanceDB** schemas live in
[`infra/persistence/lancedb/tables/`](../src/everos/infra/persistence/lancedb/tables/);
`Vector(N)` dimension matches the embedding model output.
- **SQLite** ([`infra/persistence/sqlite/tables/`](../src/everos/infra/persistence/sqlite/tables/))
holds only system / coordination tables — `md_change_state` (cascade
queue), `memcell` (boundary ledger), `unprocessed_buffer`,
`conversation_status`, `cluster`, `reflection_report` — **not**
per-kind business rows. `reflection_report` is the audit trail for
Reflection merges (cluster_id, mode, source_members, merged_entry_id,
status).
- **LanceDB** ([`infra/persistence/lancedb/tables/`](../src/everos/infra/persistence/lancedb/tables/))
holds the per-kind business rows, keyed `<owner_id>_<entry_id>` (so
cross-table joins use `(owner_id, entry_id)`); each table's `Vector(N)`
dimension matches the embedding model output.
Episode and AtomicFact LanceDB tables carry a `deprecated_by: str | None`
column. When an episode is superseded by a Reflection merge,
`deprecated_by` is set to the merged episode's entry_id. Search filters
automatically exclude rows where `deprecated_by IS NOT NULL`.
Both layers are **fully derivable from markdown** — wipe `.index/`
and the in-process cascade subsystem re-builds everything by scanning the
@ -210,13 +217,14 @@ appends an entry block → atomic write back. The caller passes a full
`EntryId` (built via `EntryId.next_for(prefix, date, current_count)`);
this primitive is **schema-agnostic** — field-level semantics
(`entry_count` / `last_appended_at`) are a business writer's job
(see `BaseDailyAppender._frontmatter_updates` in
(see `BaseDailyWriter._frontmatter_updates` in
[`infra/persistence/markdown/writers/base.py`](../src/everos/infra/persistence/markdown/writers/base.py)).
## 7. References
- Skill: [`/add-memory-kind`](../.claude/skills/add-memory-kind/SKILL.md)
- Code:
- [`core/persistence/memory_root.py`](../src/everos/core/persistence/memory_root.py) — memory-root resolution
- [`core/persistence/markdown/`](../src/everos/core/persistence/markdown/) — schema-agnostic read/write chassis
- [`infra/persistence/markdown/mds/`](../src/everos/infra/persistence/markdown/mds/) — per-kind frontmatter schemas
- [`infra/persistence/{markdown,sqlite,lancedb}/`](../src/everos/infra/persistence/) — business-aware adapters
- [`core/persistence/memory_root.py`](../src/everos/core/persistence/memory_root.py)
- [`core/persistence/markdown/`](../src/everos/core/persistence/markdown/)
- [`infra/persistence/{markdown,sqlite,lancedb}/`](../src/everos/infra/persistence/)
- [`memory/cascade/`](../src/everos/memory/cascade/) (md → LanceDB sync)

29
pyproject.toml
View File

@ -1,6 +1,6 @@
[project]
name = "everos"
version = "1.0.1"
version = "1.1.0"
description = "EverOS — local-first markdown memory framework for AI agents and user chats; lightweight, dev-friendly, small-team"
license = {text = "Apache-2.0"}
readme = "README.md"
@ -39,6 +39,7 @@ dependencies = [
# Web framework (entrypoints/api)
"fastapi>=0.104.0",
"python-multipart>=0.0.7", # Required by FastAPI for Form / UploadFile endpoints
"uvicorn[standard]>=0.24.0",
# Observability (core/observability)
@ -59,15 +60,14 @@ dependencies = [
"anyio>=4.0", # Async file I/O (anyio.Path, to_thread.run_sync) for the markdown layer
# Algorithm library (everalgo monorepo, published on PyPI).
"everalgo-core==0.2.0",
"everalgo-boundary==0.2.0",
"everalgo-user-memory==0.2.0",
"everalgo-agent-memory==0.2.0",
"everalgo-rank==0.3.0",
"everalgo-user-memory==0.3.1",
"everalgo-agent-memory==0.3.1",
"everalgo-rank==0.4.1",
"everalgo-knowledge==0.1.1",
]
[project.optional-dependencies]
multimodal = ["everalgo-parser[svg]>=0.1.0"] # [svg] bundles cairosvg → SVG works by default
multimodal = ["everalgo-parser[svg]>=0.2.1"] # [svg] bundles cairosvg → SVG works by default
[build-system]
requires = ["hatchling"]
@ -136,6 +136,21 @@ python_files = ["test_*.py"]
python_functions = ["test_*"]
asyncio_mode = "auto"
addopts = "-v --tb=short -m 'not slow and not live_llm'"
filterwarnings = [
"error",
# jieba 0.42.1: unescaped backslashes in regex strings (upstream
# unmaintained; no fix available). Python 3.12 emits
# SyntaxWarning for invalid escape sequences.
"ignore:invalid escape sequence:SyntaxWarning",
# aiosqlite teardown: connection-worker thread and Connection.__del__
# raise after pytest closes the event loop. Upstream issue
# (aiosqlite + pytest-asyncio); we cannot fix their lifecycle.
"ignore:Exception in thread.*_connection_worker_thread:pytest.PytestUnhandledThreadExceptionWarning",
# aiosqlite + asyncio teardown: Connection.__del__ and
# BaseEventLoop.__del__ fire after pytest closes the event loop.
# Upstream lifecycle issue; we cannot fix their cleanup order.
"ignore:Exception ignored.*:pytest.PytestUnraisableExceptionWarning",
]
markers = [
"slow: runs that take >=10s (regardless of dependency); CI default excludes these. Run with `pytest -m slow`.",
"live_llm: requires real LLM / embedder / reranker credentials from .env; burns tokens. CI default excludes. Run with `pytest -m live_llm`.",

2
scripts/check_consistency.py
View File

@ -159,7 +159,7 @@ def _print_monotonicity(reports: list[MonotonicityReport]) -> int:
async def run_lifespan_mode(corpus: Path) -> int:
"""Full strict check via app lifespan; covers every kind in KIND_REGISTRY."""
os.environ["EVEROS_MEMORY__ROOT"] = str(corpus)
os.environ["EVEROS_ROOT"] = str(corpus)
from everos.config import load_settings
load_settings.cache_clear()

2
scripts/check_deprecated_names.py
View File

@ -58,6 +58,8 @@ def _tracked_paths() -> list[Path]:
def _tracked_text_files() -> Iterable[tuple[str, str]]:
for path in _tracked_paths():
if not path.exists():
continue
if path.suffix.lower() in SKIP_SUFFIXES:
continue
try:

4
scripts/e2e_memorize/README.md
View File

@ -16,9 +16,9 @@ batching by 6 messages per `/add` call and then `/flush` at the end.
1. **LLM client configured** in `.env`:
- `EVEROS_LLM__API_KEY=...`
- `EVEROS_LLM__BASE_URL=...` (OpenAI-compatible)
- `EVEROS_LLM__MODEL=...` (defaults to `gpt-4o-mini`)
- `EVEROS_LLM__MODEL=...` (defaults to `gpt-4.1-mini`)
- Without these, the boundary stage logs `memorize_no_llm_client` and skips the run.
2. **Memory root**: defaults to `~/.everos`; override with `EVEROS_MEMORY__ROOT=...`.
2. **Memory root**: defaults to `~/.everos`; override with `EVEROS_ROOT=...`.
3. **Mode** is read from `settings.memorize.mode` (toml/env) before the first `memorize()` call.
## Run

6
src/everos/component/embedding/__init__.py
View File

@ -4,7 +4,8 @@
Public surface:
- :class:`EmbeddingProvider` — Protocol every provider satisfies.
- :class:`EmbeddingError` — provider-side failure.
- :class:`EmbeddingServiceError` — provider-side failure.
- :class:`EmbeddingError` — backward-compat alias for ``EmbeddingServiceError``.
- :class:`OpenAIEmbeddingProvider` — concrete provider for any
OpenAI-protocol embeddings endpoint (DeepInfra, vLLM, OpenAI, …).
- :func:`build_embedding_provider` — settings-driven factory.
@ -16,6 +17,8 @@ External usage::
vec = await provider.embed("hello")
"""
from everos.core.errors import EmbeddingServiceError as EmbeddingServiceError
from .accessor import EmbeddingNotConfiguredError as EmbeddingNotConfiguredError
from .accessor import get_embedder as get_embedder
from .factory import build_embedding_provider as build_embedding_provider
@ -27,6 +30,7 @@ __all__ = [
"EmbeddingError",
"EmbeddingNotConfiguredError",
"EmbeddingProvider",
"EmbeddingServiceError",
"OpenAIEmbeddingProvider",
"build_embedding_provider",
"get_embedder",

4
src/everos/component/embedding/openai_provider.py
View File

@ -23,7 +23,7 @@ from collections.abc import Sequence
import openai
from .protocol import EmbeddingError
from .protocol import EmbeddingServiceError
class OpenAIEmbeddingProvider:
@ -93,6 +93,6 @@ class OpenAIEmbeddingProvider:
input=chunk,
)
except openai.OpenAIError as exc:
raise EmbeddingError(str(exc)) from exc
raise EmbeddingServiceError(str(exc)) from exc
# OpenAI returns ``data`` indexed by request order; truncate to ``dim``.
return [list(item.embedding[: self.dim]) for item in response.data]

11
src/everos/component/embedding/protocol.py
View File

@ -13,13 +13,10 @@ from __future__ import annotations
from collections.abc import Sequence
from typing import Protocol, runtime_checkable
from everos.core.errors import EmbeddingServiceError as EmbeddingServiceError
class EmbeddingError(Exception):
"""Raised on any provider-side embedding failure.
Wraps the upstream SDK exception via ``__cause__`` (PEP 3134) so
diagnostic loggers preserve the original error chain.
"""
# Backward compat — old name still importable from this module.
EmbeddingError = EmbeddingServiceError
@runtime_checkable
@ -42,7 +39,7 @@ class EmbeddingProvider(Protocol):
Implementations chunk by ``batch_size`` and bound in-flight
requests by ``max_concurrent`` (both from settings). On failure,
raises :class:`EmbeddingError` — the worker treats it as a
raises :class:`EmbeddingServiceError` — the worker treats it as a
retryable / unrecoverable case per HTTP-status mapping.
"""
...

17
src/everos/component/parser/__init__.py Normal file
View File

@ -0,0 +1,17 @@
"""component.parser — shared multimodal file parsing via everalgo-parser.
External usage:
from everos.component.parser import aparse_file, parser_available
"""
from __future__ import annotations
from ._core import aparse_file as aparse_file
from ._core import parser_available as parser_available
from ._core import require_parser as require_parser
__all__ = [
"aparse_file",
"parser_available",
"require_parser",
]

67
src/everos/component/parser/_core.py Normal file
View File

@ -0,0 +1,67 @@
"""Core parse dispatch — wraps everalgo-parser with LLM injection and error mapping.
``everalgo-parser`` is an optional dependency (``everos[multimodal]``).
All imports are deferred so this module is safe to import without the extra.
"""
from __future__ import annotations
from typing import TYPE_CHECKING
from everos.core.errors import MultimodalNotEnabledError, UnsupportedModalityError
if TYPE_CHECKING:
from everalgo.types import ParsedContent, RawFile
def parser_available() -> bool:
"""Whether ``everalgo.parser`` is importable."""
try:
import everalgo.parser # noqa: F401
except ImportError:
return False
return True
def require_parser() -> None:
"""Raise when the parser extra is not installed.
Raises:
MultimodalNotEnabledError: When ``everalgo.parser`` cannot be imported.
"""
if not parser_available():
raise MultimodalNotEnabledError(
"Multimodal parsing requires the parser extra. "
"Install with: pip install 'everos[multimodal]'"
)
async def aparse_file(raw_file: RawFile) -> ParsedContent:
"""Parse a file via everalgo-parser with the multimodal LLM client.
Args:
raw_file: Hydrated ``RawFile`` with ``content`` bytes or ``uri``.
Returns:
Parsed text content with modality metadata.
Raises:
MultimodalNotEnabledError: Parser not installed or system dep missing.
UnsupportedModalityError: File format not supported by the parser.
"""
from everalgo.llm import LLMError
from everalgo.parser import aparse # Deferred: optional dep
from everos.component.llm import get_multimodal_llm_client
from everos.core.errors import LLMServiceError
try:
return await aparse(raw_file, llm=get_multimodal_llm_client())
except NotImplementedError as exc:
raise UnsupportedModalityError(f"modality not supported: {exc}") from exc
except LLMError as exc:
raise LLMServiceError(str(exc)) from exc
except ValueError as exc:
raise UnsupportedModalityError(str(exc)) from exc
except RuntimeError as exc:
raise MultimodalNotEnabledError(str(exc)) from exc

9
src/everos/component/rerank/__init__.py
View File

@ -3,12 +3,12 @@
Public surface:
- :class:`RerankProvider` — Protocol every provider satisfies.
- :class:`RerankResult` / :class:`RerankError` — value type + error.
- :class:`RerankResult` / :class:`RerankServiceError` — value type + error.
- :class:`RerankError` — backward-compat alias for :class:`RerankServiceError`.
- :class:`DeepInfraRerankProvider` — DeepInfra inference-API rerank.
- :class:`DashScopeRerankProvider` — Aliyun Bailian / DashScope rerank.
- :class:`VllmRerankProvider` — OpenAI-compat ``/v1/rerank`` (vLLM,
self-hosted, other compatible servers).
- :class:`DashScopeRerankProvider` — Aliyun Bailian / DashScope
``gte-rerank-v2`` native ``text-rerank`` endpoint.
- :func:`build_rerank_provider` — settings-driven factory that picks
the concrete provider via ``settings.rerank.provider``.
@ -19,6 +19,8 @@ External usage::
scored = await provider.rerank("how to file a claim", documents)
"""
from everos.core.errors import RerankServiceError as RerankServiceError
from .dashscope_provider import DashScopeRerankProvider as DashScopeRerankProvider
from .deepinfra_provider import DeepInfraRerankProvider as DeepInfraRerankProvider
from .factory import build_rerank_provider as build_rerank_provider
@ -33,6 +35,7 @@ __all__ = [
"RerankError",
"RerankProvider",
"RerankResult",
"RerankServiceError",
"VllmRerankProvider",
"build_rerank_provider",
]

45
src/everos/component/rerank/_errors.py Normal file
View File

@ -0,0 +1,45 @@
"""Shared error construction for HTTP-based rerank providers."""
from __future__ import annotations
import httpx
from everos.core.observability.logging import get_logger
from .protocol import RerankServiceError
logger = get_logger(__name__)
def upstream_http_error(provider: str, response: httpx.Response) -> RerankServiceError:
"""Log the upstream response body and return a client-safe error.
The body can carry provider-internal detail, so it is logged rather than
surfaced in the returned message (which exposes only the status code).
Args:
provider: Human-readable provider label (e.g. ``"vLLM"``).
response: The non-success HTTP response from the rerank backend.
Returns:
A :class:`RerankServiceError` naming the provider and status code.
"""
logger.warning(
"rerank_http_error",
provider=provider,
status=response.status_code,
body=response.text[:200],
)
return RerankServiceError(
f"{provider} rerank upstream error (HTTP {response.status_code})."
)
def transport_error(provider: str, exc: Exception) -> RerankServiceError:
"""Build the error for a transport-level failure (connect / timeout)."""
return RerankServiceError(f"{provider} rerank transport failure: {exc}")
def retries_exhausted_error(provider: str, max_retries: int) -> RerankServiceError:
"""Build the error for a retry loop that fell through without a result."""
return RerankServiceError(f"{provider} rerank exhausted retries ({max_retries}).")

27
src/everos/component/rerank/deepinfra_provider.py
View File

@ -35,11 +35,12 @@ from typing import Any
import httpx
from .protocol import RerankError, RerankResult
from ._errors import retries_exhausted_error, transport_error, upstream_http_error
from .protocol import RerankResult, RerankServiceError
# Qwen3-Reranker chat template. The DeepInfra inference API treats the reranker
# as a yes/no generator, so the prompt scaffolding must be supplied client-side
# (verbatim mirror of the benchmark reranker client). Without it the
# (verbatim mirror of the EverAlgo benchmark's reranker client). Without it the
# model scores raw text off-template and returns uncalibrated relevance.
_QWEN3_PREFIX = (
"<|im_start|>system\n"
@ -149,9 +150,7 @@ class DeepInfraRerankProvider:
)
except httpx.HTTPError as exc:
if attempt == self._max_retries:
raise RerankError(
f"DeepInfra rerank transport failure: {exc}"
) from exc
raise transport_error("DeepInfra", exc) from exc
continue
if response.status_code == 200:
@ -160,19 +159,11 @@ class DeepInfraRerankProvider:
# Retry on 5xx / 429 only; surface 4xx immediately.
if response.status_code >= 500 or response.status_code == 429:
if attempt == self._max_retries:
raise RerankError(
f"DeepInfra rerank HTTP {response.status_code}: "
f"{response.text[:200]}"
)
raise upstream_http_error("DeepInfra", response)
continue
raise RerankError(
f"DeepInfra rerank HTTP {response.status_code}: "
f"{response.text[:200]}"
)
raise upstream_http_error("DeepInfra", response)
raise RerankError(
f"DeepInfra rerank exhausted retries ({self._max_retries})"
)
raise retries_exhausted_error("DeepInfra", self._max_retries)
def _extract_scores(body: dict[str, Any], expected_len: int) -> list[float]:
@ -187,10 +178,10 @@ def _extract_scores(body: dict[str, Any], expected_len: int) -> list[float]:
"""
raw = body.get("scores")
if not isinstance(raw, list):
raise RerankError(f"DeepInfra rerank response missing scores: {body!r}")
raise RerankServiceError(f"DeepInfra rerank response missing scores: {body!r}")
row = raw[0] if raw and isinstance(raw[0], list) else raw
if len(row) != expected_len:
raise RerankError(
raise RerankServiceError(
f"DeepInfra rerank returned {len(row)} scores, expected {expected_len}"
)
return [float(s) for s in row]

2
src/everos/component/rerank/factory.py
View File

@ -26,7 +26,7 @@ from .vllm_provider import VllmRerankProvider
logger = get_logger(__name__)
# host substring → provider. Ordered most-specific first; matched against
# host substring -> provider. Ordered most-specific first; matched against
# the ``base_url`` host so a Bailian / DeepInfra URL routes to the right
# request-shape without the operator also having to set ``provider``.
_PROVIDER_HOST_HINTS: tuple[tuple[str, str], ...] = (

5
src/everos/component/rerank/protocol.py
View File

@ -14,9 +14,10 @@ from __future__ import annotations
from collections.abc import Sequence
from typing import NamedTuple, Protocol, runtime_checkable
from everos.core.errors import RerankServiceError as RerankServiceError
class RerankError(Exception):
"""Raised on any provider-side rerank failure."""
# Backward compat alias.
RerankError = RerankServiceError
class RerankResult(NamedTuple):

24
src/everos/component/rerank/vllm_provider.py
View File

@ -41,7 +41,8 @@ from typing import Any
import httpx
from .protocol import RerankError, RerankResult
from ._errors import retries_exhausted_error, transport_error, upstream_http_error
from .protocol import RerankResult, RerankServiceError
class VllmRerankProvider:
@ -133,9 +134,7 @@ class VllmRerankProvider:
)
except httpx.HTTPError as exc:
if attempt == self._max_retries:
raise RerankError(
f"vLLM rerank transport failure: {exc}"
) from exc
raise transport_error("vLLM", exc) from exc
continue
if response.status_code == 200:
@ -143,22 +142,17 @@ class VllmRerankProvider:
if response.status_code >= 500 or response.status_code == 429:
if attempt == self._max_retries:
raise RerankError(
f"vLLM rerank HTTP {response.status_code}: "
f"{response.text[:200]}"
)
raise upstream_http_error("vLLM", response)
continue
raise RerankError(
f"vLLM rerank HTTP {response.status_code}: {response.text[:200]}"
)
raise upstream_http_error("vLLM", response)
raise RerankError(f"vLLM rerank exhausted retries ({self._max_retries})")
raise retries_exhausted_error("vLLM", self._max_retries)
def _parse_rerank_results(body: dict[str, Any]) -> list[RerankResult]:
items = body.get("results")
if not isinstance(items, list):
raise RerankError(f"vLLM rerank response missing results: {body!r}")
raise RerankServiceError(f"vLLM rerank response missing results: {body!r}")
parsed: list[RerankResult] = []
for item in items:
try:
@ -169,5 +163,7 @@ def _parse_rerank_results(body: dict[str, Any]) -> list[RerankResult]:
)
)
except (KeyError, TypeError, ValueError) as exc:
raise RerankError(f"malformed rerank result entry: {item!r}") from exc
raise RerankServiceError(
f"malformed rerank result entry: {item!r}"
) from exc
return parsed

2
src/everos/component/tokenizer/__init__.py
View File

@ -14,11 +14,9 @@ External usage::
"""
from .factory import build_tokenizer as build_tokenizer
from .jieba_provider import JiebaTokenizer as JiebaTokenizer
from .protocol import Tokenizer as Tokenizer
__all__ = [
"JiebaTokenizer",
"Tokenizer",
"build_tokenizer",
]

6
src/everos/component/tokenizer/factory.py
View File

@ -8,10 +8,14 @@ change — see ``17_lancedb_tables_design.md`` §2.4.1.
from __future__ import annotations
from .jieba_provider import JiebaTokenizer
from .protocol import Tokenizer
def build_tokenizer() -> Tokenizer:
"""Build the default tokenizer (``JiebaTokenizer``)."""
# Deferred: jieba contains invalid escape sequences that raise
# SyntaxError on Python 3.12+; defer so the cost is paid only when
# tokenization is actually needed (not at import time).
from .jieba_provider import JiebaTokenizer
return JiebaTokenizer()

4
src/everos/config/__init__.py
View File

@ -5,7 +5,7 @@ Public API:
Settings, MemorySettings, SqliteSettings, LanceDBSettings,
LLMSettings, EmbeddingSettings, RerankSettings,
BoundaryDetectionSettings,
load_settings,
load_settings, resolve_root,
)
Distinct from ``everos.component.config`` (which is a *capability* —
@ -22,6 +22,7 @@ from .settings import RerankSettings as RerankSettings
from .settings import Settings as Settings
from .settings import SqliteSettings as SqliteSettings
from .settings import load_settings as load_settings
from .settings import resolve_root as resolve_root
__all__ = [
"BoundaryDetectionSettings",
@ -34,4 +35,5 @@ __all__ = [
"Settings",
"SqliteSettings",
"load_settings",
"resolve_root",
]

40
src/everos/config/default.toml
View File

@ -2,19 +2,15 @@
#
# Lookup order (later overrides earlier):
# 1. This file (shipped defaults; lowest priority)
# 2. ~/.everos/config.toml — user-level overrides (optional;
# path is overridable via EVEROS_CONFIG_FILE)
# 3. .env file in the working directory
# 4. Environment variables — EVEROS_<SECTION>__<KEY>
# 2. <root>/everos.toml — user config (optional; root resolved by
# resolve_root(): EVEROS_ROOT env > ~/.everos)
# 3. Environment variables — EVEROS_<SECTION>__<KEY>
# e.g. EVEROS_SQLITE__BUSY_TIMEOUT_MS=10000
# 5. Programmatic init args (highest priority)
# 4. Programmatic init args (highest priority)
#
# `null` (omitted in TOML) means "use the Pydantic default declared in code".
[memory]
# memory-root is the single directory holding all persisted memory.
# `~` is expanded; the path is resolved when MemoryRoot is constructed.
root = "~/.everos"
# Effective timezone for date buckets and timestamps. Drives
# component.utils.datetime; this is the SOLE source — OS `TZ` is not
# read. Override via `EVEROS_MEMORY__TIMEZONE` env var if needed.
@ -57,8 +53,8 @@ cache_size_kb = 2048
[llm]
# Provider-agnostic OpenAI-protocol client config. Override via env:
# EVEROS_LLM__MODEL, EVEROS_LLM__API_KEY, EVEROS_LLM__BASE_URL
# Or via a ``.env`` file next to the project root (auto-loaded).
model = "gpt-4o-mini"
# Or set the field directly in this file (<root>/everos.toml).
model = "gpt-4.1-mini"
# api_key = ""
# base_url = ""
@ -135,6 +131,22 @@ vector_strategy = "maxsim_atomic"
# requires a restart. Override via EVEROS_MEMORIZE__MODE.
mode = "agent"
[knowledge]
# Max bytes for an uploaded knowledge document (default 50 MiB). Oversized
# uploads are rejected with HTTP 422 before parsing/extraction. Note: the
# multipart body is still buffered by the server first, so set a reverse-proxy
# / gateway body-size limit for hard ingress protection.
# Override via EVEROS_KNOWLEDGE__MAX_UPLOAD_BYTES.
max_upload_bytes = 52_428_800 # 50 MiB
[knowledge.search]
recall_n = 200
rerank_n = 50
# "lambda" is a Python keyword — aliased as "lam" in Settings
lambda = 0.1
mass_top_m = 50
top_k_cap = 100
# Maximum wall-clock for one memorize() invocation while holding the
# per-session lock. On timeout the outer asyncio.timeout cancels the call
# and the lock auto-releases so subsequent concurrent /add on the same
@ -142,3 +154,11 @@ mode = "agent"
# synchronous portion of pipeline dispatch.
# Override via EVEROS_MEMORIZE__SESSION_LOCK_TIMEOUT_SECONDS.
session_lock_timeout_seconds = 360.0
[clustering]
# Geometry-clustering: cosine similarity threshold and time window.
# Episodes older than ``time_window_days`` from the newest cluster
# member are excluded from merge consideration.
# Override via EVEROS_CLUSTERING__THRESHOLD, EVEROS_CLUSTERING__TIME_WINDOW_DAYS.
threshold = 0.65
time_window_days = 7.0

26
src/everos/config/default_ome.toml
View File

@ -1,11 +1,12 @@
# everos OME (Offline Memory Engine) — per-strategy overrides.
# everos OME (Offline Memory Engine) — strategy configuration.
#
# This file is materialised at ``<memory-root>/ome.toml`` by
# ``MemoryRoot.ensure()`` on first server start. Edit it to toggle
# individual strategies or tweak their gate / retry / cron without
# restarting the server; the engine watches this file and hot-reloads
# changes within ~2 seconds. Re-running ``ensure()`` will NOT overwrite
# your edits — the file is only materialised when absent.
# THIS IS THE SINGLE ENTRY POINT for all strategy settings (enabled,
# cron, max_retries, gate). Edit this file to toggle strategies on/off
# or tune their scheduling — changes are hot-reloaded within ~2 seconds,
# no server restart needed.
#
# Created at ``~/.everos/ome.toml`` on first server start. Editing the
# file after creation is safe — it will not be overwritten.
#
# Overrides are partial: only the keys you set replace the in-code
# defaults; omitted keys keep each strategy's coded value. Unknown
@ -46,6 +47,17 @@
# [strategies.extract_user_profile]
# enabled = false
# ── Reflection ──────────────────────────────────────────────────────────
# Offline memory consolidation. Disabled by default — uncomment to enable.
# Once enabled, runs on the default schedule: weekly, Monday 02:00.
# Run Reflection sparingly — at most once a week. Triggering it too
# often degrades quality, since each run is a lossy LLM re-merge.
# The cron expression can be overridden without restarting the server.
# [strategies.reflect_episodes]
# enabled = true
# cron = "0 2 * * 1"
# ── Agent-memory pipeline ───────────────────────────────────────────────
# Agent case extraction (runs per agent memcell). One per tool call cycle.

125
src/everos/config/settings.py
View File

@ -3,14 +3,13 @@
Loaded by :func:`load_settings`. Source priority (later wins):
1. ``config/default.toml`` (shipped values; lowest priority)
2. ``~/.everos/config.toml`` (user-level overrides; optional)
3. ``.env`` file in the working directory (secrets / machine-specific)
4. ``EVEROS_<SECTION>__<KEY>`` environment variables
5. Init args passed programmatically (highest priority)
2. ``<root>/everos.toml`` (user config; optional; ``<root>`` resolved by
:func:`resolve_root`)
3. ``EVEROS_<SECTION>__<KEY>`` environment variables
4. Init args passed programmatically (highest priority)
The user-level toml path defaults to ``~/.everos/config.toml``. Override
with the ``EVEROS_CONFIG_FILE`` environment variable. The file is
optional — if it does not exist, the source is silently skipped.
The memory root is resolved by :func:`resolve_root`:
``explicit arg > EVEROS_ROOT env > ~/.everos``.
The settings tree mirrors the TOML structure: ``settings.sqlite.busy_timeout_ms``
maps to ``[sqlite].busy_timeout_ms`` and to ``EVEROS_SQLITE__BUSY_TIMEOUT_MS``.
@ -29,7 +28,7 @@ from pathlib import Path
from typing import Literal
from zoneinfo import ZoneInfo, ZoneInfoNotFoundError
from pydantic import BaseModel, Field, SecretStr, field_validator
from pydantic import BaseModel, ConfigDict, Field, SecretStr, field_validator
from pydantic_settings import (
BaseSettings,
PydanticBaseSettingsSource,
@ -38,24 +37,31 @@ from pydantic_settings import (
)
_DEFAULT_TOML_PATH = Path(__file__).parent / "default.toml"
_USER_TOML_ENV_VAR = "EVEROS_CONFIG_FILE"
_DEFAULT_USER_TOML_PATH = Path("~/.everos/config.toml").expanduser()
_DEFAULT_ROOT = Path("~/.everos")
def _resolve_user_toml_path() -> Path:
"""Resolve the user-level ``config.toml`` path.
def resolve_root(explicit: str | None = None) -> Path:
"""Resolve the memory-root path.
Defaults to ``~/.everos/config.toml``; override with the
``EVEROS_CONFIG_FILE`` environment variable.
Priority: explicit arg > EVEROS_ROOT env > ~/.everos default.
Args:
explicit: Caller-supplied path string (e.g. from ``--root`` CLI flag).
Returns:
Absolute resolved path to the memory root.
"""
override = os.environ.get(_USER_TOML_ENV_VAR)
return Path(override).expanduser() if override else _DEFAULT_USER_TOML_PATH
if explicit:
return Path(explicit).expanduser().resolve()
from_env = os.environ.get("EVEROS_ROOT")
if from_env:
return Path(from_env).expanduser().resolve()
return _DEFAULT_ROOT.expanduser().resolve()
class MemorySettings(BaseModel):
"""memory-root configuration."""
"""Memory configuration."""
root: Path = Path("~/.everos")
timezone: str = "UTC"
"""Effective timezone for date buckets and timestamps.
@ -123,7 +129,7 @@ class LLMSettings(BaseModel):
EVEROS_LLM__BASE_URL
"""
model: str = "gpt-4o-mini"
model: str = "gpt-4.1-mini"
api_key: SecretStr | None = None
base_url: str | None = None
@ -258,6 +264,18 @@ class MemorizeSettings(BaseModel):
session_lock_timeout_seconds: float = Field(default=360.0, gt=0)
class ClusteringSettings(BaseModel):
"""Geometry-clustering tunables.
Env binding:
EVEROS_CLUSTERING__THRESHOLD
EVEROS_CLUSTERING__TIME_WINDOW_DAYS
"""
threshold: float = Field(default=0.65, gt=0, le=1)
time_window_days: float = Field(default=7.0, gt=0)
class SearchSettings(BaseModel):
"""Search-pipeline policy knobs.
@ -343,6 +361,25 @@ class LanceDBSettings(BaseModel):
index_cache_size_bytes: int = 16 * 1024 * 1024
class KnowledgeSearchSettings(BaseModel):
"""``[knowledge.search]`` — retrieval tuning for the knowledge module."""
recall_n: int = 200
rerank_n: int = 50
mass_top_m: int = 50
lam: float = Field(0.1, alias="lambda")
top_k_cap: int = 100
model_config = ConfigDict(populate_by_name=True)
class KnowledgeSettings(BaseModel):
"""``[knowledge]`` — knowledge module configuration."""
max_upload_bytes: int = 52_428_800 # 50 MiB
search: KnowledgeSearchSettings = KnowledgeSearchSettings()
class Settings(BaseSettings):
"""Top-level application settings."""
@ -355,18 +392,44 @@ class Settings(BaseSettings):
rerank: RerankSettings = RerankSettings()
boundary_detection: BoundaryDetectionSettings = BoundaryDetectionSettings()
memorize: MemorizeSettings = MemorizeSettings()
clustering: ClusteringSettings = ClusteringSettings()
search: SearchSettings = SearchSettings()
multimodal: MultimodalSettings = MultimodalSettings()
knowledge: KnowledgeSettings = KnowledgeSettings()
model_config = SettingsConfigDict(
env_prefix="EVEROS_",
env_nested_delimiter="__",
env_file=".env",
env_file_encoding="utf-8",
toml_file=_DEFAULT_TOML_PATH,
extra="ignore",
)
def __init__(self, *, _everos_root: Path | None = None, **kwargs: object) -> None:
"""Initialise settings, optionally pinning the memory-root for testing.
Args:
_everos_root: Override the memory root used to locate
``everos.toml``. Intended for tests only; pass ``None``
(the default) in production to use :func:`resolve_root`.
**kwargs: Forwarded verbatim to :class:`pydantic_settings.BaseSettings`.
"""
if _everos_root is not None:
# Temporarily inject EVEROS_ROOT so that settings_customise_sources
# (a classmethod that cannot access instance state) picks it up via
# resolve_root(). We restore the original value after super().__init__
# returns to avoid leaking the override into the process environment.
_prev = os.environ.get("EVEROS_ROOT")
os.environ["EVEROS_ROOT"] = str(_everos_root)
try:
super().__init__(**kwargs)
finally:
if _prev is None:
os.environ.pop("EVEROS_ROOT", None)
else:
os.environ["EVEROS_ROOT"] = _prev
else:
super().__init__(**kwargs)
@classmethod
def settings_customise_sources(
cls,
@ -376,26 +439,18 @@ class Settings(BaseSettings):
dotenv_settings: PydanticBaseSettingsSource,
file_secret_settings: PydanticBaseSettingsSource,
) -> tuple[PydanticBaseSettingsSource, ...]:
"""Layer TOML sources between env / dotenv and the secret store.
Order (earlier wins in pydantic-settings):
init_args > env > .env > user_toml > default_toml > secrets
The user-level toml (default ``~/.everos/config.toml``) is only
registered when the file exists, so the source list stays tight.
"""
"""Source order: init_args > env_vars > everos.toml > default.toml."""
sources: list[PydanticBaseSettingsSource] = [
init_settings,
env_settings,
dotenv_settings,
]
user_toml_path = _resolve_user_toml_path()
if user_toml_path.is_file():
# Attempt to load <root>/everos.toml if it exists.
everos_toml = resolve_root() / "everos.toml"
if everos_toml.is_file():
sources.append(
TomlConfigSettingsSource(settings_cls, toml_file=user_toml_path)
TomlConfigSettingsSource(settings_cls, toml_file=everos_toml)
)
sources.append(TomlConfigSettingsSource(settings_cls))
sources.append(file_secret_settings)
sources.append(TomlConfigSettingsSource(settings_cls)) # default.toml
return tuple(sources)

183
src/everos/core/errors.py
View File

@ -1,43 +1,184 @@
"""Cross-cutting domain errors surfaced to API callers.
"""Cross-cutting exception hierarchy for EverOS.
These live in ``core`` so the ``memory`` layer can raise them and the
``entrypoints`` layer can catch them without crossing the layered import
boundary — ``any -> core`` is the only edge both share (entrypoints must
not import ``memory`` directly).
All application exceptions derive from ``AppError``, split into four branches:
- ``DomainError`` — business-rule violations (not-found, conflict, invalid
input, path traversal, unsupported format).
- ``InfrastructureError`` — transient storage and external-service failures
(retryable).
- ``CapabilityError`` — permanent server-side capability gaps (not retryable).
- ``ConfigurationError`` — misconfiguration detected at runtime.
Any layer may raise ``AppError`` subclasses; the entrypoints layer catches
them and maps them to aligned HTTP responses.
"""
from __future__ import annotations
from enum import StrEnum
class PathTraversalError(Exception):
class ErrorCode(StrEnum):
"""Machine-readable error codes returned in the API error envelope.
Each code maps to exactly one HTTP status code. Clients can switch on
this value to decide retry / display / routing behaviour without parsing
the human-readable ``message`` field.
"""
NOT_FOUND = "NOT_FOUND"
CONFLICT = "CONFLICT"
INVALID_INPUT = "INVALID_INPUT"
EXTRACTION_EMPTY = "EXTRACTION_EMPTY"
BAD_REQUEST = "BAD_REQUEST"
UNSUPPORTED_FORMAT = "UNSUPPORTED_FORMAT"
EXTERNAL_SERVICE_UNAVAILABLE = "EXTERNAL_SERVICE_UNAVAILABLE"
CAPABILITY_UNAVAILABLE = "CAPABILITY_UNAVAILABLE"
CONFIGURATION_ERROR = "CONFIGURATION_ERROR"
INTERNAL_ERROR = "INTERNAL_ERROR"
# ---------------------------------------------------------------------------
# Root
# ---------------------------------------------------------------------------
class AppError(Exception):
"""Root for all EverOS application exceptions."""
# ---------------------------------------------------------------------------
# Domain branch — client-side / business-rule errors
# ---------------------------------------------------------------------------
class DomainError(AppError):
"""Business-rule violation originating in the domain or service layer."""
class NotFoundError(DomainError):
"""A requested resource does not exist."""
class DocumentNotFoundError(NotFoundError):
"""A document with the given identifier was not found."""
class TopicNotFoundError(NotFoundError):
"""A knowledge topic with the given identifier was not found."""
class ConflictError(DomainError):
"""An operation conflicts with existing state (e.g. duplicate resource)."""
class DuplicateDocumentError(ConflictError):
"""A document with the same identifier already exists."""
class InvalidInputError(DomainError):
"""Input does not meet domain rules."""
class ExtractionEmptyError(InvalidInputError):
"""An extraction pipeline produced no output when output was required."""
class FilterError(InvalidInputError):
"""A caller-supplied filter expression is invalid or malformed."""
class PathTraversalError(DomainError):
"""A write target resolved outside the configured memory root.
Raised by the markdown writer as a defense-in-depth backstop: any
caller-supplied identifier that becomes a path segment is validated at
the DTO layer, but this containment check does not depend on every such
id being sanitized upstream. The API layer maps it to HTTP 400.
caller-supplied identifier that becomes a path segment (``app_id`` /
``project_id`` / ``sender_id`` -> ``owner_id``) is validated at the DTO
layer, but this containment check does not depend on every such id being
sanitised upstream. The API layer maps it to HTTP 400.
"""
class MultimodalError(Exception):
"""Base for multimodal-parsing errors meant to reach the caller.
class UnsupportedModalityError(DomainError):
"""The uploaded file format is not supported (e.g. video, unknown type).
The API layer maps any ``MultimodalError`` to an aligned
``{error: {code, message}}`` envelope (HTTP 415).
Wraps everalgo's ``NotImplementedError`` / dispatch ``ValueError`` so
the caller gets a stable 415 instead of a raw 500.
"""
class UnsupportedModalityError(MultimodalError):
"""everalgo cannot handle this modality (e.g. video stub, unknown type).
# ---------------------------------------------------------------------------
# Infrastructure branch — transient failures (retryable)
# ---------------------------------------------------------------------------
Wraps everalgo's ``NotImplementedError`` / dispatch ``ValueError`` so the
caller gets a stable, aligned error instead of a raw 500.
class InfrastructureError(AppError):
"""Transient failure in a storage adapter or external service."""
class StorageError(InfrastructureError):
"""A markdown or SQLite persistence operation failed."""
class VectorStoreError(InfrastructureError):
"""A LanceDB vector-store operation failed."""
class ExternalServiceError(InfrastructureError):
"""An external service (LLM, embedding, rerank) returned an error or timed out."""
class LLMServiceError(ExternalServiceError):
"""The configured LLM provider returned an error or timed out."""
class EmbeddingServiceError(ExternalServiceError):
"""The configured embedding provider returned an error or timed out."""
class RerankServiceError(ExternalServiceError):
"""The configured rerank provider returned an error or timed out."""
# ---------------------------------------------------------------------------
# Capability branch — permanent server-side gaps (not retryable)
# ---------------------------------------------------------------------------
class CapabilityError(AppError):
"""A required server-side capability is not available.
Unlike ``InfrastructureError`` (transient — retry may help),
``CapabilityError`` signals a permanent gap that requires admin
action (install a dependency, enable a feature).
"""
class MultimodalNotEnabledError(MultimodalError):
"""Multimodal capability is not ready.
class MultimodalNotEnabledError(CapabilityError):
"""Multimodal parsing capability is not available.
Raised when the ``everos[multimodal]`` extra is not installed, or when a
required system dependency (LibreOffice for Office documents) is absent.
Raised when the ``everos[multimodal]`` extra is not installed, or when
a required system dependency (LibreOffice for Office documents) is absent.
"""
# ---------------------------------------------------------------------------
# Configuration branch — misconfiguration detected at runtime
# ---------------------------------------------------------------------------
class ConfigurationError(AppError):
"""A required configuration is missing or invalid.
Raised when a mandatory setting (e.g. embedding model, rerank provider)
is not configured but the code path requires it.
"""
# ---------------------------------------------------------------------------
# Backward compatibility aliases
# ---------------------------------------------------------------------------
# Renamed in v0.2 — old names kept for external consumers.
DocumentAlreadyExistsError = DuplicateDocumentError
ValidationError = InvalidInputError

6
src/everos/core/middleware/__init__.py
View File

@ -1,6 +1,7 @@
"""Cross-cutting HTTP middleware components.
External usage:
External usage::
from everos.core.middleware import (
DEFAULT_CORS_ALLOW_CREDENTIALS,
DEFAULT_CORS_ALLOW_HEADERS,
@ -8,7 +9,6 @@ External usage:
DEFAULT_CORS_ORIGINS,
ProfileMiddleware,
PrometheusMiddleware,
global_exception_handler,
)
"""
@ -16,7 +16,6 @@ from .cors import DEFAULT_CORS_ALLOW_CREDENTIALS as DEFAULT_CORS_ALLOW_CREDENTIA
from .cors import DEFAULT_CORS_ALLOW_HEADERS as DEFAULT_CORS_ALLOW_HEADERS
from .cors import DEFAULT_CORS_ALLOW_METHODS as DEFAULT_CORS_ALLOW_METHODS
from .cors import DEFAULT_CORS_ORIGINS as DEFAULT_CORS_ORIGINS
from .global_exception import global_exception_handler as global_exception_handler
from .profile import ProfileMiddleware as ProfileMiddleware
from .prometheus import PrometheusMiddleware as PrometheusMiddleware
@ -27,5 +26,4 @@ __all__ = [
"DEFAULT_CORS_ORIGINS",
"ProfileMiddleware",
"PrometheusMiddleware",
"global_exception_handler",
]

143
src/everos/core/middleware/global_exception.py
View File

@ -1,143 +0,0 @@
"""Global exception handler — uniform error envelope per v1 API brief §1.
Envelope shape (matches the v1 API brief §1 — ``request_id`` at the top
level alongside ``error``; the ``error`` object carries ``code`` /
``message`` plus ops-friendly ``timestamp`` / ``path`` for debugging)::
{
"request_id": "<32 lowercase hex chars — W3C trace_id format>",
"error": {
"code": "HTTP_ERROR" | "SYSTEM_ERROR",
"message": "<reason>",
"timestamp": "<ISO 8601 with tz>",
"path": "<request path>"
}
}
Rules:
- 4xx (DTO / business validation / HTTPException) → ``code="HTTP_ERROR"``
with the human-readable reason in ``message``.
- 5xx (unhandled exception) → ``code="SYSTEM_ERROR"`` with a fixed
``message="Internal server error"`` — internal exception details are
logged but never leak to the client.
- ``request_id`` is sourced from ``request.state.request_id`` (set by
upstream middleware); falls back to a freshly minted id when absent.
"""
from __future__ import annotations
from fastapi import HTTPException, Request
from fastapi.exceptions import RequestValidationError
from fastapi.responses import JSONResponse
from starlette.status import (
HTTP_422_UNPROCESSABLE_ENTITY,
HTTP_500_INTERNAL_SERVER_ERROR,
)
from everos.component.utils.datetime import (
get_now_with_timezone,
to_iso_format,
)
from everos.core.observability.logging import get_logger
from everos.core.observability.tracing import gen_request_id
logger = get_logger(__name__)
_INTERNAL_ERROR_MESSAGE = "Internal server error"
def _request_id(request: Request) -> str:
"""Return the request_id set by middleware, or mint a fresh fallback."""
rid = getattr(request.state, "request_id", None)
if rid:
return str(rid)
return gen_request_id()
def _envelope(
*,
code: str,
message: str,
request: Request,
) -> dict[str, object]:
"""Build the canonical error envelope (wiki §1 shape — nested ``error``).
``request_id`` at the top level, ``error`` object carries the
contract fields (``code`` / ``message``) plus ops-friendly
``timestamp`` / ``path``.
"""
return {
"request_id": _request_id(request),
"error": {
"code": code,
"message": message,
"timestamp": to_iso_format(get_now_with_timezone()),
"path": str(request.url.path),
},
}
async def global_exception_handler(request: Request, exc: Exception) -> JSONResponse:
"""Convert any exception into a uniform JSON error response."""
path = str(request.url.path)
method = request.method
if isinstance(exc, RequestValidationError):
errors = exc.errors()
if errors:
first = errors[0]
loc = ".".join(str(p) for p in first.get("loc", []) if p != "body")
msg = first.get("msg", "Validation error")
message = f"{msg}: {loc}" if loc else msg
else:
message = "Request validation error"
logger.warning("validation_error", method=method, path=path, message=message)
return JSONResponse(
status_code=HTTP_422_UNPROCESSABLE_ENTITY,
content=_envelope(code="HTTP_ERROR", message=message, request=request),
)
if isinstance(exc, HTTPException):
logger.warning(
"http_exception",
method=method,
path=path,
status_code=exc.status_code,
detail=exc.detail,
)
# 5xx routed through HTTPException is rare but valid; still honour
# the SYSTEM_ERROR code so the envelope is consistent.
if exc.status_code >= 500:
return JSONResponse(
status_code=exc.status_code,
content=_envelope(
code="SYSTEM_ERROR",
message=_INTERNAL_ERROR_MESSAGE,
request=request,
),
)
return JSONResponse(
status_code=exc.status_code,
content=_envelope(
code="HTTP_ERROR",
message=str(exc.detail),
request=request,
),
)
logger.error(
"unhandled_exception",
method=method,
path=path,
exception_type=type(exc).__name__,
exc_info=True,
)
return JSONResponse(
status_code=HTTP_500_INTERNAL_SERVER_ERROR,
content=_envelope(
code="SYSTEM_ERROR",
message=_INTERNAL_ERROR_MESSAGE,
request=request,
),
)

27
src/everos/core/persistence/lancedb/repository.py
View File

@ -78,7 +78,7 @@ class LanceRepoBase[T: BaseLanceTable]:
reads stay unlocked so search QPS is not impacted by writers.
Locks live in a class-level dict keyed by table name and are never
evicted (mirrors :mod:`everos.memory.strategies._partition_locks`
evicted (mirrors :mod:`everos.memory._partition_locks`
on bpo-28427 — a lock with pending waiters must outlive any dict
entry that points to it).
"""
@ -111,7 +111,7 @@ class LanceRepoBase[T: BaseLanceTable]:
a module-level lock surviving across tests fails with "bound to
a different event loop". The production cascade worker runs on
one loop forever and does not need this hook. Mirrors
:func:`everos.memory.strategies._partition_locks._reset_for_tests`.
:func:`everos.memory._partition_locks._reset_for_tests`.
"""
cls._table_locks.clear()
@ -427,6 +427,29 @@ class LanceRepoBase[T: BaseLanceTable]:
q = q.where(where)
return await q.limit(limit).to_list()
# ── Update ─────────────────────────────────────────────────────────────
async def update(
self,
updates: dict[str, Any],
*,
where: str,
) -> None:
"""Partial column update for rows matching ``where``.
Wraps ``AsyncTable.update`` — sets specific column values without
rewriting the full row. Useful for lightweight metadata patches
(e.g. setting ``deprecated_by``) where a full embed+upsert cycle
is unnecessary.
Args:
updates: Column-name to new-value mapping.
where: SQL-like predicate scoping the update.
"""
table = await self._table()
async with self._write_lock(self.table_name):
await table.update(updates, where=where)
# ── Delete ─────────────────────────────────────────────────────────────
async def delete(self, predicate: str) -> None:

8
src/everos/core/persistence/markdown/__init__.py
View File

@ -18,6 +18,8 @@ External usage (frontmatter schema chassis):
from everos.core.persistence.markdown import (
BaseFrontmatter, UserScopedFrontmatter, AgentScopedFrontmatter,
DailyLogPathMixin, SkillPathMixin, ProfilePathMixin,
KnowledgeScopedMixin, KnowledgeDocumentPathMixin,
KnowledgeTopicPathMixin,
)
"""
@ -31,6 +33,9 @@ from .entries import split_entries as split_entries
from .frontmatter import AgentScopedFrontmatter as AgentScopedFrontmatter
from .frontmatter import BaseFrontmatter as BaseFrontmatter
from .frontmatter import DailyLogPathMixin as DailyLogPathMixin
from .frontmatter import KnowledgeDocumentPathMixin as KnowledgeDocumentPathMixin
from .frontmatter import KnowledgeScopedMixin as KnowledgeScopedMixin
from .frontmatter import KnowledgeTopicPathMixin as KnowledgeTopicPathMixin
from .frontmatter import ProfilePathMixin as ProfilePathMixin
from .frontmatter import SkillPathMixin as SkillPathMixin
from .frontmatter import UserScopedFrontmatter as UserScopedFrontmatter
@ -46,6 +51,9 @@ __all__ = [
"DailyLogPathMixin",
"Entry",
"EntryId",
"KnowledgeDocumentPathMixin",
"KnowledgeScopedMixin",
"KnowledgeTopicPathMixin",
"MarkdownReader",
"MarkdownWriter",
"ParsedMarkdown",

44
src/everos/core/persistence/markdown/frontmatter.py
View File

@ -298,3 +298,47 @@ class AgentScopedFrontmatter(BaseFrontmatter):
agent_id: str
track: Literal["agent"] = "agent"
class KnowledgeScopedMixin:
"""Records in the knowledge scope (no owner_id).
Unlike user/agent scopes, knowledge records are not owned by an
individual entity. The scope directory is ``knowledge/``.
"""
SCOPE_DIR: ClassVar[str] = "knowledge"
class KnowledgeDocumentPathMixin:
"""Path strategy for ``knowledge/{category}/{doc_title}/index.md``.
Place this mixin first so MRO resolves ``path_glob()`` here::
class KnowledgeDocumentFrontmatter(
KnowledgeDocumentPathMixin, KnowledgeScopedMixin, BaseFrontmatter
): ...
"""
SCOPE_DIR: ClassVar[str]
@classmethod
def path_glob(cls) -> str:
return f"*/*/{cls.SCOPE_DIR}/*/*/index.md"
class KnowledgeTopicPathMixin:
"""Path strategy for ``knowledge/{category}/{doc_title}/<n>_<name>.md``.
Place this mixin first so MRO resolves ``path_glob()`` here::
class KnowledgeTopicFrontmatter(
KnowledgeTopicPathMixin, KnowledgeScopedMixin, BaseFrontmatter
): ...
"""
SCOPE_DIR: ClassVar[str]
@classmethod
def path_glob(cls) -> str:
return f"*/*/{cls.SCOPE_DIR}/*/*/[0-9]*.md"

79
src/everos/core/persistence/markdown/writer.py
View File

@ -52,7 +52,7 @@ from everos.core.errors import PathTraversalError
from ..memory_root import MemoryRoot
from .entries import EntryId
from .frontmatter import dump_frontmatter
from .frontmatter import dump_frontmatter, parse_frontmatter
from .reader import MarkdownReader
@ -60,9 +60,11 @@ class MarkdownWriter:
"""Atomic writer for markdown files inside a memory-root.
The ``memory_root`` reference anchors a containment check: every write
target must resolve inside ``memory_root.root``. This is defense-in-depth
against path traversal via caller-supplied identifiers that become path
segments.
target must resolve inside ``memory_root.root`` (see
:meth:`_ensure_within_root`). This is defense-in-depth against path
traversal via any caller-supplied identifier that becomes a path
segment — the DTO layer also rejects ``.``/``..`` in such ids, but this
check does not depend on every id being sanitised upstream.
"""
def __init__(self, memory_root: MemoryRoot) -> None:
@ -100,8 +102,30 @@ class MarkdownWriter:
return lock
def _ensure_within_root(self, target: Path) -> Path:
"""Reject a write target that resolves outside the memory root."""
root = self._memory_root.root
"""Reject a write target that resolves outside the memory root.
Defense-in-depth against path traversal: a caller-supplied id that
becomes a path segment (e.g. ``sender_id`` -> ``owner_id``) could
otherwise smuggle ``..`` segments and walk the write out of the
configured root. ``resolve()`` collapses ``..`` and symlinks
lexically/physically before the comparison, so the check holds even
though ``target`` does not exist yet.
Must run *before* any filesystem touch — the ``mkdir`` on the write
path and the read-modify-write read on the append path both call this
first, so an escaping path never creates parent directories nor opens
an out-of-root file.
Args:
target: The intended write path.
Returns:
The resolved, root-contained absolute path.
Raises:
PathTraversalError: If the resolved path is not within the root.
"""
root = self._memory_root.root # already absolute + resolved
resolved = target.resolve()
if not resolved.is_relative_to(root):
raise PathTraversalError(
@ -121,6 +145,9 @@ class MarkdownWriter:
Returns:
``path`` (resolved as written).
Raises:
PathTraversalError: If ``path`` resolves outside the memory root.
"""
target = Path(path)
self._ensure_within_root(target)
@ -146,6 +173,43 @@ class MarkdownWriter:
head = dump_frontmatter(frontmatter or {})
return await self.write(path, head + body)
async def patch_frontmatter(self, path: Path, updates: Mapping[str, Any]) -> None:
"""Update frontmatter fields on an existing md file in-place.
Reads the file, merges ``updates`` into frontmatter, writes back.
Only the frontmatter portion is rewritten; entries are untouched.
Uses the same per-path lock as ``append_entries`` for concurrency
safety.
For dict-type fields (e.g. ``deprecated_entries``) the merge is
additive: existing keys are preserved, new keys are added or
overwritten. Scalar fields are replaced wholesale.
Args:
path: Target markdown file (must exist).
updates: Mapping of frontmatter keys to merge.
Raises:
FileNotFoundError: If ``path`` does not exist.
"""
target = Path(path)
async with self.lock_for(target):
# 1. Read raw text.
raw = await anyio.Path(target).read_text(encoding="utf-8")
# 2. Split into frontmatter + remainder (entries body).
existing_fm, remainder = parse_frontmatter(raw)
# 3. Deep-merge dict fields; replace scalars.
for key, value in updates.items():
if isinstance(value, dict) and isinstance(existing_fm.get(key), dict):
existing_fm[key].update(value)
else:
existing_fm[key] = value
# 4. Atomic write with merged frontmatter + original body.
await self.write(target, dump_frontmatter(existing_fm) + remainder)
async def append_entry(
self,
path: Path,
@ -239,6 +303,9 @@ class MarkdownWriter:
breaks the safety contract.
"""
target = Path(path)
# Guard the read too, not just the final write: an escaping path must
# not even reach MarkdownReader (otherwise an out-of-root file would be
# opened and parsed before the write-side check rejected it).
self._ensure_within_root(target)
# 1. Load existing markdown (or initialise empty).

42
src/everos/core/persistence/memory_root.py
View File

@ -49,12 +49,6 @@ _DEFAULT_SCOPE_ID = "default"
_DEFAULT_APP_DIR = "default_app"
_DEFAULT_PROJECT_DIR = "default_project"
# Path to the shipped OME override template; copied to ``<root>/ome.toml`` on
# first ``ensure()`` so users have a real file to edit instead of having to
# create one from scratch. ``parents[2]`` is the ``src/everos/`` package root
# (memory_root.py sits at ``core/persistence/memory_root.py``).
_OME_TEMPLATE_PATH = Path(__file__).parents[2] / "config" / "default_ome.toml"
def app_dir_name(app_id: str) -> str:
"""Map an ``app_id`` to its on-disk directory name."""
@ -98,16 +92,18 @@ class MemoryRoot:
object.__setattr__(self, "root", resolved)
@classmethod
def default(cls) -> MemoryRoot:
"""Return the memory-root from :class:`everos.config.Settings`.
def default(cls, *, explicit_root: str | None = None) -> MemoryRoot:
"""Return the memory-root resolved from CLI / env / default.
The effective default lives in ``config/default.toml`` (``[memory]
root``); environment variable ``EVEROS_MEMORY__ROOT`` overrides it.
Resolution: ``explicit_root`` > ``EVEROS_ROOT`` env > ``~/.everos``.
Args:
explicit_root: Caller-supplied path (e.g. from ``--root`` CLI flag).
"""
# Lazy import to keep this module dependency-free at import time.
from everos.config import load_settings
from everos.config.settings import resolve_root
return cls(load_settings().memory.root)
return cls(resolve_root(explicit_root))
# ── User-visible (partitioned by app / project) ──────────────────────────
#
@ -184,19 +180,17 @@ class MemoryRoot:
@property
def ome_config(self) -> Path:
"""``<root>/ome.toml`` — user-editable OME strategy overrides.
"""``<root>/ome.toml`` — single entry point for strategy configuration.
Drop a file here to toggle strategies on/off or tweak per-strategy
knobs (max_retries, gate, cron …) without restarting the server.
The engine watches this file and hot-reloads changes within ~2 s.
All strategy settings (enabled, cron, max_retries, gate) are managed
here. The engine watches this file and hot-reloads changes within ~2 s.
No server restart needed.
Example to disable foresight and user-profile extraction::
Example — enable Reflection with a custom cron::
[strategies.extract_foresight]
enabled = false
[strategies.extract_user_profile]
enabled = false
[strategies.reflect_episodes]
enabled = true
cron = "0 3 * * *"
"""
return self.root / "ome.toml"
@ -237,7 +231,3 @@ class MemoryRoot:
self.sqlite_dir.mkdir(parents=True, exist_ok=True)
self.lancedb_dir.mkdir(parents=True, exist_ok=True)
self.tmp_dir.mkdir(parents=True, exist_ok=True)
# Materialize the OME override template on first run; existence-only
# check preserves any edits the user has already made.
if not self.ome_config.exists():
self.ome_config.write_bytes(_OME_TEMPLATE_PATH.read_bytes())

15
src/everos/entrypoints/api/app.py
View File

@ -8,8 +8,7 @@ from __future__ import annotations
import os
from fastapi import FastAPI, HTTPException
from fastapi.exceptions import RequestValidationError
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from everos import __version__
@ -25,10 +24,10 @@ from everos.core.middleware import (
DEFAULT_CORS_ORIGINS,
ProfileMiddleware,
PrometheusMiddleware,
global_exception_handler,
)
from everos.core.observability.logging import get_logger
from .exception_handlers import register_handlers
from .lifespans import (
CascadeLifespanProvider,
LanceDBLifespanProvider,
@ -39,8 +38,10 @@ from .lifespans import (
from .routes import (
get,
health,
knowledge,
memorize,
metrics,
ome,
search,
)
@ -97,10 +98,8 @@ def create_app(
openapi_url="/openapi.json" if enable_docs else None,
)
# Exception handlers: HTTPException, validation errors, plus a fallback.
app.add_exception_handler(HTTPException, global_exception_handler)
app.add_exception_handler(RequestValidationError, global_exception_handler)
app.add_exception_handler(Exception, global_exception_handler)
# Exception handlers
register_handlers(app)
# Middleware order: earlier `add_middleware` calls become inner, later ones outer.
# CORS innermost (matches base_app.py legacy pattern).
@ -120,6 +119,8 @@ def create_app(
app.include_router(memorize.router)
app.include_router(search.router)
app.include_router(get.router)
app.include_router(ome.router)
app.include_router(knowledge.router)
logger.info("app_created", docs_enabled=enable_docs)
return app

362
src/everos/entrypoints/api/exception_handlers.py Normal file
View File

@ -0,0 +1,362 @@
"""Per-type exception handlers for the EverOS FastAPI application.
Each handler converts a specific exception class (or hierarchy root) into
the canonical error envelope::
{
"request_id": "<32 lowercase hex chars>",
"error": {
"code": "<ErrorCode>",
"message": "<reason>",
"timestamp": "<ISO 8601 with tz>",
"path": "<request path>"
}
}
Register all handlers at once with ``register_handlers(app)``.
"""
from __future__ import annotations
from fastapi import FastAPI, HTTPException, Request
from fastapi.exceptions import RequestValidationError
from fastapi.responses import JSONResponse
from pydantic import BaseModel
from starlette.status import (
HTTP_400_BAD_REQUEST,
HTTP_404_NOT_FOUND,
HTTP_409_CONFLICT,
HTTP_415_UNSUPPORTED_MEDIA_TYPE,
HTTP_422_UNPROCESSABLE_CONTENT,
HTTP_500_INTERNAL_SERVER_ERROR,
HTTP_503_SERVICE_UNAVAILABLE,
)
from everos.component.utils.datetime import get_now_with_timezone, to_iso_format
from everos.core.errors import (
CapabilityError,
ConfigurationError,
ConflictError,
ErrorCode,
ExtractionEmptyError,
InfrastructureError,
InvalidInputError,
NotFoundError,
PathTraversalError,
UnsupportedModalityError,
)
from everos.core.observability.logging import get_logger
from .utils import extract_request_id
logger = get_logger(__name__)
_INTERNAL_ERROR_MESSAGE = "Internal server error"
# ---------------------------------------------------------------------------
# Response model (visible in OpenAPI docs)
# ---------------------------------------------------------------------------
class ErrorDetail(BaseModel):
"""Inner ``error`` object in the canonical error envelope."""
code: ErrorCode
message: str
timestamp: str
path: str
class ErrorResponse(BaseModel):
"""Canonical error envelope returned by all error handlers."""
request_id: str
error: ErrorDetail
# ---------------------------------------------------------------------------
# Envelope builder
# ---------------------------------------------------------------------------
def _error_response(
request: Request,
status_code: int,
code: ErrorCode,
message: str,
) -> JSONResponse:
"""Build a JSONResponse with the canonical error envelope."""
body = ErrorResponse(
request_id=extract_request_id(request),
error=ErrorDetail(
code=code,
message=message,
timestamp=to_iso_format(get_now_with_timezone()),
path=str(request.url.path),
),
)
return JSONResponse(
status_code=status_code,
content=body.model_dump(),
)
# ---------------------------------------------------------------------------
# Handlers
# ---------------------------------------------------------------------------
async def not_found_handler(
request: Request,
exc: NotFoundError,
) -> JSONResponse:
"""NotFoundError (and subclasses) -> 404."""
return _error_response(
request,
HTTP_404_NOT_FOUND,
ErrorCode.NOT_FOUND,
str(exc),
)
async def conflict_handler(
request: Request,
exc: ConflictError,
) -> JSONResponse:
"""ConflictError (and subclasses) -> 409."""
return _error_response(
request,
HTTP_409_CONFLICT,
ErrorCode.CONFLICT,
str(exc),
)
async def extraction_empty_handler(
request: Request,
exc: ExtractionEmptyError,
) -> JSONResponse:
"""ExtractionEmptyError -> 422 with dedicated code."""
return _error_response(
request,
HTTP_422_UNPROCESSABLE_CONTENT,
ErrorCode.EXTRACTION_EMPTY,
str(exc),
)
async def invalid_input_handler(
request: Request,
exc: InvalidInputError,
) -> JSONResponse:
"""InvalidInputError (and subclasses) -> 422."""
return _error_response(
request,
HTTP_422_UNPROCESSABLE_CONTENT,
ErrorCode.INVALID_INPUT,
str(exc),
)
async def path_traversal_handler(
request: Request,
exc: PathTraversalError,
) -> JSONResponse:
"""PathTraversalError -> 400."""
return _error_response(
request,
HTTP_400_BAD_REQUEST,
ErrorCode.BAD_REQUEST,
"Invalid input: path contains illegal characters.",
)
async def unsupported_modality_handler(
request: Request,
exc: UnsupportedModalityError,
) -> JSONResponse:
"""UnsupportedModalityError -> 415."""
return _error_response(
request,
HTTP_415_UNSUPPORTED_MEDIA_TYPE,
ErrorCode.UNSUPPORTED_FORMAT,
str(exc),
)
async def infrastructure_handler(
request: Request,
exc: InfrastructureError,
) -> JSONResponse:
"""InfrastructureError (and subclasses) -> 503."""
logger.warning(
"infrastructure_error",
path=str(request.url.path),
exception_type=type(exc).__name__,
message=str(exc),
)
return _error_response(
request,
HTTP_503_SERVICE_UNAVAILABLE,
ErrorCode.EXTERNAL_SERVICE_UNAVAILABLE,
str(exc),
)
async def capability_handler(
request: Request,
exc: CapabilityError,
) -> JSONResponse:
"""CapabilityError (and subclasses) -> 503 (not retryable)."""
logger.warning(
"capability_error",
path=str(request.url.path),
exception_type=type(exc).__name__,
message=str(exc),
)
return _error_response(
request,
HTTP_503_SERVICE_UNAVAILABLE,
ErrorCode.CAPABILITY_UNAVAILABLE,
str(exc),
)
async def configuration_handler(
request: Request,
exc: ConfigurationError,
) -> JSONResponse:
"""ConfigurationError -> 500."""
logger.error(
"configuration_error",
path=str(request.url.path),
message=str(exc),
)
return _error_response(
request,
HTTP_500_INTERNAL_SERVER_ERROR,
ErrorCode.CONFIGURATION_ERROR,
str(exc),
)
# ---------------------------------------------------------------------------
# Pydantic / FastAPI built-in exceptions
# ---------------------------------------------------------------------------
_FIELD_HINTS: dict[str, str] = {
"doc_id": (
"Invalid doc_id format. Use GET /documents to look up valid doc_id values."
),
"topic_id": (
"Invalid topic_id format. "
"Use GET /documents/{doc_id} to look up valid topic_id values."
),
"query": "Search query cannot be empty.",
"title": "Invalid title. Must contain at least one letter or digit.",
}
async def request_validation_handler(
request: Request,
exc: RequestValidationError,
) -> JSONResponse:
"""FastAPI RequestValidationError -> 422."""
errors = exc.errors()
if errors:
first = errors[0]
loc_parts = [str(p) for p in first.get("loc", []) if p != "body"]
field = loc_parts[-1] if loc_parts else ""
if field in _FIELD_HINTS:
message = _FIELD_HINTS[field]
else:
loc = ".".join(loc_parts)
msg = first.get("msg", "Validation error")
message = f"{msg}: {loc}" if loc else msg
else:
message = "Request validation error"
return _error_response(
request,
HTTP_422_UNPROCESSABLE_CONTENT,
ErrorCode.INVALID_INPUT,
message,
)
async def http_exception_handler(
request: Request,
exc: HTTPException,
) -> JSONResponse:
"""FastAPI HTTPException -> envelope with original status code."""
if exc.status_code >= 500:
logger.error(
"http_exception_5xx",
path=str(request.url.path),
status_code=exc.status_code,
)
return _error_response(
request,
exc.status_code,
ErrorCode.INTERNAL_ERROR,
_INTERNAL_ERROR_MESSAGE,
)
return _error_response(
request,
exc.status_code,
ErrorCode.BAD_REQUEST,
str(exc.detail),
)
async def unexpected_handler(
request: Request,
exc: Exception,
) -> JSONResponse:
"""Catch-all for unhandled exceptions -> 500 (no detail leak)."""
logger.error(
"unhandled_exception",
path=str(request.url.path),
exception_type=type(exc).__name__,
exc_info=True,
)
return _error_response(
request,
HTTP_500_INTERNAL_SERVER_ERROR,
ErrorCode.INTERNAL_ERROR,
_INTERNAL_ERROR_MESSAGE,
)
# ---------------------------------------------------------------------------
# Registration
# ---------------------------------------------------------------------------
def register_handlers(app: FastAPI) -> None:
"""Register all per-type exception handlers on ``app``.
Starlette walks the exception MRO and picks the first matching
handler, so more-specific types are registered before their parents.
"""
# Domain errors (specific before parent)
app.add_exception_handler(PathTraversalError, path_traversal_handler)
app.add_exception_handler(UnsupportedModalityError, unsupported_modality_handler)
app.add_exception_handler(NotFoundError, not_found_handler)
app.add_exception_handler(ConflictError, conflict_handler)
app.add_exception_handler(ExtractionEmptyError, extraction_empty_handler)
app.add_exception_handler(InvalidInputError, invalid_input_handler)
# Infrastructure errors (transient, retryable)
app.add_exception_handler(InfrastructureError, infrastructure_handler)
# Capability errors (permanent, not retryable)
app.add_exception_handler(CapabilityError, capability_handler)
# Configuration errors
app.add_exception_handler(ConfigurationError, configuration_handler)
# FastAPI built-in exceptions
app.add_exception_handler(HTTPException, http_exception_handler)
app.add_exception_handler(
RequestValidationError,
request_validation_handler,
)
# Catch-all
app.add_exception_handler(Exception, unexpected_handler)

9
src/everos/entrypoints/api/routes/get.py
View File

@ -7,10 +7,9 @@ return the envelope verbatim. ``request_id`` is generated inside the
from __future__ import annotations
from fastapi import APIRouter, HTTPException
from fastapi import APIRouter
from everos.memory.get import GetRequest, GetResponse
from everos.memory.search import FilterError
from everos.service import get as get_service
router = APIRouter(prefix="/api/v1/memory", tags=["memory"])
@ -19,8 +18,4 @@ router = APIRouter(prefix="/api/v1/memory", tags=["memory"])
@router.post("/get", response_model=GetResponse)
async def post_get(req: GetRequest) -> GetResponse:
"""Paginated listing over the requested ``memory_type``."""
try:
return await get_service(req)
except FilterError as exc:
# Filter-DSL violations surface as 422 with the compile message.
raise HTTPException(status_code=422, detail=str(exc)) from exc
return await get_service(req)

643
src/everos/entrypoints/api/routes/knowledge.py Normal file
View File

@ -0,0 +1,643 @@
"""Knowledge document CRUD + search HTTP endpoints.
Routes follow the thin-adapter pattern: validate the DTO, delegate to
the service layer, format the response envelope. No business logic
lives here.
Endpoint overview (design spec sections 6.2-6.6):
POST /documents — upload a new document (multipart)
PUT /documents/{doc_id} — replace an existing document (multipart)
PATCH /documents/{doc_id} — update mutable metadata
DELETE /documents/{doc_id} — remove a document
GET /documents — paginated document listing
GET /documents/{doc_id} — single document detail
GET /topics/{topic_id} — single topic detail
POST /search — knowledge retrieval
GET /categories — taxonomy listing
"""
from __future__ import annotations
from datetime import datetime
from typing import TYPE_CHECKING, Annotated, Literal
if TYPE_CHECKING:
from everos.service.knowledge import KnowledgeExtractor
from everalgo.types import ParsedContent
from fastapi import APIRouter, Path, Query, Request, Response, UploadFile
from fastapi.params import Form
from pydantic import BaseModel, Field
from everos.component.llm import get_llm_client
from everos.component.utils.datetime import to_display_tz
from everos.config import load_settings
from everos.core.errors import (
InvalidInputError,
UnsupportedModalityError,
)
from everos.core.persistence import MemoryRoot
from everos.entrypoints.api.utils import extract_request_id
from everos.service import (
CreateDocumentResult,
DocumentDetail,
DocumentListResult,
SearchKnowledgeResult,
TopicDetail,
create_document,
delete_document,
get_document,
get_topic,
list_categories,
list_documents,
patch_document,
replace_document,
search_knowledge,
)
# PathSafeId and SuccessEnvelope are imported from memorize routes;
# a shared module would be cleaner but is out of scope for this PR.
from .memorize import PathSafeId, SuccessEnvelope
router = APIRouter(prefix="/api/v1/knowledge", tags=["knowledge"])
# ── Annotated param types (satisfies B008) ──────────────────────────────────
_FormTitle = Annotated[str, Form(min_length=1, pattern=r"\w")]
_FormOptStr = Annotated[str | None, Form()]
_FormPathSafe = Annotated[PathSafeId, Form()]
_QueryPathSafe = Annotated[PathSafeId, Query()]
_QueryOptStr = Annotated[str | None, Query()]
_QueryPage = Annotated[int, Query(ge=1)]
_QueryPageSize = Annotated[int, Query(ge=1, le=100)]
_QuerySortBy = Annotated[Literal["created_at", "updated_at", "title"], Query()]
_QuerySortOrder = Annotated[Literal["asc", "desc"], Query()]
# Upper bound on a search query string — guards the embedding call against
# pathologically long input.
_MAX_QUERY_LENGTH = 2000
# doc_id: system-generated "d_<hex12..32>"; topic_id: "{doc_id}_{index}".
_DOC_ID_PATTERN = r"^d_[a-f0-9]{12,32}$"
_TOPIC_ID_PATTERN = r"^d_[a-f0-9]{12,32}_\d+$"
_PathDocId = Annotated[str, Path(pattern=_DOC_ID_PATTERN)]
_PathTopicId = Annotated[str, Path(pattern=_TOPIC_ID_PATTERN)]
# ── Response DTOs ────────────────────────────────────────────────────────────
class DocumentCreateResponse(BaseModel):
"""Response for POST/PUT /documents."""
doc_id: str
category_id: str
topic_count: int
source_name: str | None
md_path: str
original_file_path: str | None
class DocumentDeleteResponse(BaseModel):
"""Response for DELETE /documents/{doc_id}."""
doc_id: str
deleted_topics: int
class TopicOverviewDTO(BaseModel):
"""Minimal topic summary inside a document detail."""
topic_id: str
topic_name: str
topic_path: str
depth: int
summary: str
class DocumentDetailResponse(BaseModel):
"""Response for GET /documents/{doc_id}."""
doc_id: str
category_id: str
title: str
summary: str
source_name: str | None
source_type: str | None
original_file_path: str | None
topics: list[TopicOverviewDTO]
created_at: datetime
updated_at: datetime
class DocumentOverviewItemDTO(BaseModel):
"""One row in the paginated document list."""
doc_id: str
category_id: str
title: str
topic_count: int
created_at: datetime
class DocumentListResponse(BaseModel):
"""Response for GET /documents."""
documents: list[DocumentOverviewItemDTO]
total: int
page: int
page_size: int
class TopicDetailResponse(BaseModel):
"""Response for GET /topics/{topic_id}."""
topic_id: str
doc_id: str
category_id: str
topic_name: str
topic_path: str
depth: int
summary: str
content: str
content_labels: list[str]
parent_topic_id: str | None
children_topic_ids: list[str]
created_at: datetime
updated_at: datetime
class DocumentContextDTO(BaseModel):
"""L1 document metadata attached to every search hit."""
doc_id: str
title: str
summary: str
class SearchHitDTO(BaseModel):
"""One ranked result from knowledge search."""
topic_id: str
category_id: str
topic_name: str
topic_path: str
depth: int
summary: str
content: str | None
score: float
retrieval_method: str
source: str | None
document: DocumentContextDTO
class KnowledgeSearchResponse(BaseModel):
"""Response for POST /search."""
hits: list[SearchHitDTO]
total: int
took_ms: float
class CategoryDTO(BaseModel):
"""One taxonomy category."""
category_id: str
description: str
document_count: int
class CategoryListResponse(BaseModel):
"""Response for GET /categories."""
categories: list[CategoryDTO]
class DocumentPatchResponse(BaseModel):
"""Response for PATCH /documents/{doc_id}."""
doc_id: str
updated_fields: list[str]
updated_at: datetime
# ── Request DTOs ─────────────────────────────────────────────────────────────
class KnowledgeSearchRequest(BaseModel):
"""Request body for POST /search."""
query: str = Field(..., min_length=1, max_length=_MAX_QUERY_LENGTH)
method: Literal["keyword", "vector", "hybrid"] = "hybrid"
top_k: int = Field(default=10, ge=1, le=100)
score_threshold: float | None = None
include_content: bool = False
app_id: PathSafeId = "default"
project_id: PathSafeId = "default"
class DocumentPatchRequest(BaseModel):
"""Request body for PATCH /documents/{doc_id}."""
title: str | None = Field(default=None, min_length=1, pattern=r"\w")
category_id: str | None = Field(default=None, min_length=1)
app_id: PathSafeId = "default"
project_id: PathSafeId = "default"
# ── Extractor builder ───────────────────────────────────────────────────────
def _build_extractor() -> KnowledgeExtractor:
"""Lazily import and build the knowledge extractor from ``everalgo``.
Returns an object satisfying the ``KnowledgeExtractor`` protocol
defined in ``service.knowledge``.
"""
# Deferred: heavy everalgo import; only needed on document creation.
from everalgo.knowledge import KnowledgeExtractor as AlgoKnowledgeExtractor
return AlgoKnowledgeExtractor(llm=get_llm_client())
# ── Helpers ──────────────────────────────────────────────────────────────────
def _reject_oversized_upload(file: UploadFile) -> None:
"""Reject an upload whose declared size exceeds the configured limit.
Raises:
InvalidInputError: When ``file.size`` exceeds ``knowledge.max_upload_bytes``.
"""
max_bytes = load_settings().knowledge.max_upload_bytes
if file.size is not None and file.size > max_bytes:
limit_mib = max_bytes / (1024 * 1024)
raise InvalidInputError(f"Uploaded file exceeds the {limit_mib:.1f} MiB limit.")
async def _parse_upload(
file: UploadFile, *, raw_bytes: bytes | None = None
) -> ParsedContent:
"""Parse an uploaded file via component.parser, or fall back to UTF-8.
Args:
file: FastAPI upload file handle.
raw_bytes: Pre-read bytes; when provided, skips ``file.read()``.
Raises:
UnsupportedModalityError: When the file cannot be parsed.
InvalidInputError: When the parsed content is empty.
"""
from everos.component.parser import parser_available # Deferred: optional dep
if raw_bytes is None:
raw_bytes = await file.read()
if parser_available():
from everalgo.types import RawFile # Deferred: optional dep
from everos.component.parser import aparse_file # Deferred: optional dep
extension = ""
if file.filename and "." in file.filename:
extension = file.filename.rsplit(".", 1)[-1].lower()
parsed = await aparse_file(
RawFile(
content=raw_bytes,
mime=file.content_type or "",
extension=extension,
)
)
if not parsed.text or not parsed.text.strip():
raise InvalidInputError("Uploaded file has no valid content.")
return parsed
try:
text = raw_bytes.decode("utf-8")
except UnicodeDecodeError as exc:
raise UnsupportedModalityError(
"File is not UTF-8 text. "
"Install everos[multimodal] for PDF/HTML/DOCX support."
) from exc
parsed = ParsedContent(text=text)
if not parsed.text or not parsed.text.strip():
raise InvalidInputError("Uploaded file has no valid content.")
return parsed
def _map_create_result(result: CreateDocumentResult) -> DocumentCreateResponse:
"""Map CreateDocumentResult to response DTO."""
return DocumentCreateResponse(
doc_id=result.doc_id,
category_id=result.category_id,
topic_count=result.topic_count,
source_name=result.source_name,
md_path=result.md_path,
original_file_path=result.original_file_path,
)
def _map_document_detail(detail: DocumentDetail) -> DocumentDetailResponse:
"""Map DocumentDetail to response DTO."""
return DocumentDetailResponse(
doc_id=detail.doc_id,
category_id=detail.category_id,
title=detail.title,
summary=detail.summary,
source_name=detail.source_name,
source_type=detail.source_type,
original_file_path=detail.original_file_path,
topics=[
TopicOverviewDTO(
topic_id=t.topic_id,
topic_name=t.topic_name,
topic_path=t.topic_path,
depth=t.depth,
summary=t.summary,
)
for t in detail.topics
],
created_at=to_display_tz(detail.created_at),
updated_at=to_display_tz(detail.updated_at),
)
def _map_list_result(result: DocumentListResult) -> DocumentListResponse:
"""Map DocumentListResult to response DTO."""
return DocumentListResponse(
documents=[
DocumentOverviewItemDTO(
doc_id=d.doc_id,
category_id=d.category_id,
title=d.title,
topic_count=d.topic_count,
created_at=to_display_tz(d.created_at),
)
for d in result.documents
],
total=result.total,
page=result.page,
page_size=result.page_size,
)
def _map_topic_detail(detail: TopicDetail) -> TopicDetailResponse:
"""Map TopicDetail to response DTO."""
return TopicDetailResponse(
topic_id=detail.topic_id,
doc_id=detail.doc_id,
category_id=detail.category_id,
topic_name=detail.topic_name,
topic_path=detail.topic_path,
depth=detail.depth,
summary=detail.summary,
content=detail.content,
content_labels=detail.content_labels,
parent_topic_id=detail.parent_topic_id,
children_topic_ids=detail.children_topic_ids,
created_at=to_display_tz(detail.created_at),
updated_at=to_display_tz(detail.updated_at),
)
def _map_search_result(result: SearchKnowledgeResult) -> KnowledgeSearchResponse:
"""Map SearchKnowledgeResult to response DTO."""
return KnowledgeSearchResponse(
hits=[
SearchHitDTO(
topic_id=h.topic_id,
category_id=h.category_id,
topic_name=h.topic_name,
topic_path=h.topic_path,
depth=h.depth,
summary=h.summary,
content=h.content,
score=h.score,
retrieval_method=h.retrieval_method,
source=h.source,
document=DocumentContextDTO(
doc_id=h.document.doc_id,
title=h.document.title,
summary=h.document.summary,
),
)
for h in result.hits
],
total=result.total,
took_ms=result.took_ms,
)
# ── Routes ───────────────────────────────────────────────────────────────────
@router.post("/documents", status_code=201)
# FastAPI requires flat Form/Query params — ≤5 positional rule exempted.
async def create_document_route(
request: Request,
file: UploadFile,
title: _FormTitle,
source_type: _FormOptStr = None,
category_id: _FormOptStr = None,
app_id: _FormPathSafe = "default",
project_id: _FormPathSafe = "default",
) -> SuccessEnvelope[DocumentCreateResponse]:
"""Upload a new knowledge document."""
rid = extract_request_id(request)
_reject_oversized_upload(file)
file_content = await file.read()
parsed = await _parse_upload(file, raw_bytes=file_content)
source_name = file.filename
knowledge_dir = MemoryRoot.default().knowledge_dir(app_id, project_id)
extractor = _build_extractor()
result = await create_document(
extractor=extractor,
parsed=parsed,
title=title,
knowledge_dir=knowledge_dir,
source_name=source_name,
source_type=source_type,
category_id=category_id,
file_content=file_content,
)
return SuccessEnvelope(request_id=rid, data=_map_create_result(result))
@router.put("/documents/{doc_id}", status_code=200)
# FastAPI requires flat Form/Query params — ≤5 positional rule exempted.
async def replace_document_route(
request: Request,
doc_id: _PathDocId,
file: UploadFile,
title: _FormTitle,
source_type: _FormOptStr = None,
category_id: _FormOptStr = None,
app_id: _FormPathSafe = "default",
project_id: _FormPathSafe = "default",
) -> SuccessEnvelope[DocumentCreateResponse]:
"""Replace an existing knowledge document (atomic backup/restore on failure)."""
rid = extract_request_id(request)
_reject_oversized_upload(file)
file_content = await file.read()
parsed = await _parse_upload(file, raw_bytes=file_content)
knowledge_dir = MemoryRoot.default().knowledge_dir(app_id, project_id)
extractor = _build_extractor()
result = await replace_document(
extractor=extractor,
parsed=parsed,
title=title,
doc_id=doc_id,
knowledge_dir=knowledge_dir,
source_name=file.filename,
source_type=source_type,
category_id=category_id,
file_content=file_content,
)
return SuccessEnvelope(request_id=rid, data=_map_create_result(result))
@router.delete("/documents/{doc_id}", response_model=None)
async def delete_document_route(
request: Request,
doc_id: _PathDocId,
app_id: _QueryPathSafe = "default",
project_id: _QueryPathSafe = "default",
) -> SuccessEnvelope[DocumentDeleteResponse] | Response:
"""Remove a knowledge document."""
rid = extract_request_id(request)
result = await delete_document(doc_id, app_id, project_id)
if result.deleted_topics == 0:
return Response(status_code=204)
return SuccessEnvelope(
request_id=rid,
data=DocumentDeleteResponse(
doc_id=result.doc_id,
deleted_topics=result.deleted_topics,
),
)
@router.get("/documents")
# FastAPI requires flat Form/Query params — ≤5 positional rule exempted.
async def list_documents_route(
request: Request,
app_id: _QueryPathSafe = "default",
project_id: _QueryPathSafe = "default",
category_id: _QueryOptStr = None,
page: _QueryPage = 1,
page_size: _QueryPageSize = 20,
sort_by: _QuerySortBy = "created_at",
sort_order: _QuerySortOrder = "desc",
) -> SuccessEnvelope[DocumentListResponse]:
"""Paginated document listing."""
rid = extract_request_id(request)
result = await list_documents(
app_id,
project_id,
category_id=category_id,
page=page,
page_size=page_size,
sort_by=sort_by,
sort_order=sort_order,
)
return SuccessEnvelope(request_id=rid, data=_map_list_result(result))
@router.get("/documents/{doc_id}")
async def get_document_route(
request: Request,
doc_id: _PathDocId,
app_id: _QueryPathSafe = "default",
project_id: _QueryPathSafe = "default",
) -> SuccessEnvelope[DocumentDetailResponse]:
"""Fetch a single document with its topic list."""
rid = extract_request_id(request)
detail = await get_document(doc_id, app_id, project_id)
return SuccessEnvelope(request_id=rid, data=_map_document_detail(detail))
@router.get("/topics/{topic_id}")
async def get_topic_route(
request: Request,
topic_id: _PathTopicId,
app_id: _QueryPathSafe = "default",
project_id: _QueryPathSafe = "default",
) -> SuccessEnvelope[TopicDetailResponse]:
"""Fetch a single topic with full content."""
rid = extract_request_id(request)
detail = await get_topic(topic_id, app_id, project_id)
return SuccessEnvelope(request_id=rid, data=_map_topic_detail(detail))
@router.post("/search")
async def search_knowledge_route(
request: Request,
req: KnowledgeSearchRequest,
) -> SuccessEnvelope[KnowledgeSearchResponse]:
"""Knowledge retrieval (keyword / vector / hybrid)."""
rid = extract_request_id(request)
result = await search_knowledge(
query=req.query,
method=req.method,
top_k=req.top_k,
score_threshold=req.score_threshold,
include_content=req.include_content,
app_id=req.app_id,
project_id=req.project_id,
)
return SuccessEnvelope(request_id=rid, data=_map_search_result(result))
@router.get("/categories")
async def list_categories_route(
request: Request,
app_id: _QueryPathSafe = "default",
project_id: _QueryPathSafe = "default",
) -> SuccessEnvelope[CategoryListResponse]:
"""List taxonomy categories from ``.taxonomy.md``."""
rid = extract_request_id(request)
overviews = await list_categories(app_id, project_id)
categories = [
CategoryDTO(
category_id=c.category_id,
description=c.description,
document_count=c.document_count,
)
for c in overviews
]
return SuccessEnvelope(
request_id=rid,
data=CategoryListResponse(categories=categories),
)
@router.patch("/documents/{doc_id}")
async def patch_document_route(
request: Request,
doc_id: _PathDocId,
req: DocumentPatchRequest,
) -> SuccessEnvelope[DocumentPatchResponse]:
"""Update mutable document metadata fields."""
rid = extract_request_id(request)
result = await patch_document(
doc_id,
req.app_id,
req.project_id,
title=req.title,
category_id=req.category_id,
)
return SuccessEnvelope(
request_id=rid,
data=DocumentPatchResponse(
doc_id=result.doc_id,
updated_fields=result.updated_fields,
updated_at=to_display_tz(result.updated_at),
),
)

53
src/everos/entrypoints/api/routes/memorize.py
View File

@ -10,13 +10,13 @@ server-side and does not expose this endpoint).
from __future__ import annotations
import re
from typing import Annotated, Any, Literal
from fastapi import APIRouter, HTTPException, Request
from fastapi import APIRouter, Request
from pydantic import AfterValidator, BaseModel, ConfigDict, Field
from everos.core.errors import MultimodalError, PathTraversalError
from everos.core.observability.tracing import gen_request_id
from everos.entrypoints.api.utils import extract_request_id
from everos.service import memorize
router = APIRouter(prefix="/api/v1/memory", tags=["memory"])
@ -24,22 +24,34 @@ router = APIRouter(prefix="/api/v1/memory", tags=["memory"])
# ── Path-safe identifier ────────────────────────────────────────────────────
# ``app_id`` / ``project_id`` / ``sender_id`` all become directory segments
# under the memory root (``sender_id`` flows through to ``owner_id``), so they
# must reject ``.`` and ``..`` (path traversal). The basic character whitelist
# is enforced via ``pattern`` (pydantic_core uses the Rust regex engine, which
# does NOT support lookaround), and the two reserved tokens are filtered out
# with a follow-up ``AfterValidator``.
# under the memory root (``sender_id`` flows through to ``owner_id`` and is
# joined into the daily-log write path), so they must reject ``.`` and ``..``
# (path traversal). The basic character whitelist is enforced via ``pattern``
# (pydantic_core uses the Rust regex engine, which does NOT support
# lookaround), and the two reserved tokens are filtered out with a follow-up
# ``AfterValidator``.
#
# ``@`` and ``+`` are admitted so real-world ids survive (email-style ids and
# plus-addressing). Path separators and NUL stay out of the whitelist, while
# the markdown writer's root-containment check is the final backstop.
# ``@`` and ``+`` are admitted so real-world ids survive (email-style
# ``user@example.com``, plus-addressing ``user+tag``); both are legal,
# non-separator filename chars on every target filesystem (incl. NTFS, whose
# reserved set is ``< > : " / \ | ? *``). The genuinely path-dangerous chars
# (``/`` ``\`` NUL) stay out of the whitelist, and ``.``/``..`` stay blocked
# by the token filter; the markdown writer's ``_ensure_within_root`` is the
# final backstop regardless.
_PATH_SAFE_CHARSET = r"^[a-zA-Z0-9_.@+-]+$"
_PATH_TRAVERSAL_TOKENS = frozenset({".", ".."})
_PATH_SAFE_RE = re.compile(_PATH_SAFE_CHARSET)
def _reject_path_traversal(value: str) -> str:
if value in _PATH_TRAVERSAL_TOKENS:
raise ValueError("'.' and '..' are reserved (path traversal)")
if not _PATH_SAFE_RE.match(value):
raise ValueError(
"Only alphanumerics, underscore, dot, hyphen, @, and + are allowed"
)
return value
@ -75,6 +87,9 @@ class ContentItemDTO(BaseModel):
class MessageItemDTO(BaseModel):
# ``sender_id`` becomes ``owner_id`` and then a directory segment on the
# episode write path, so it carries the same path-safety guard as
# ``app_id`` / ``project_id`` (charset whitelist + ``.``/``..`` rejection).
sender_id: PathSafeId = Field(
...,
min_length=1,
@ -155,13 +170,8 @@ async def add_memory(
request: Request,
) -> SuccessEnvelope[AddResponseData]:
"""Add messages into the user-memory + agent-memory pipelines."""
request_id = getattr(request.state, "request_id", None) or _gen_request_id()
try:
result = await memorize(req.model_dump())
except MultimodalError as exc:
raise HTTPException(status_code=415, detail=str(exc)) from exc
except PathTraversalError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
request_id = extract_request_id(request)
result = await memorize(req.model_dump())
return SuccessEnvelope(
request_id=request_id,
data=AddResponseData(
@ -181,7 +191,7 @@ async def flush_memory(
[OSS-only] — cloud edition decides boundary timing server-side and
does not expose this endpoint.
"""
request_id = getattr(request.state, "request_id", None) or _gen_request_id()
request_id = extract_request_id(request)
result = await memorize(
{
"session_id": req.session_id,
@ -200,8 +210,3 @@ async def flush_memory(
request_id=request_id,
data=FlushResponseData(status=status),
)
def _gen_request_id() -> str:
"""Fallback request id when no middleware set one."""
return gen_request_id()

47
src/everos/entrypoints/api/routes/ome.py Normal file
View File

@ -0,0 +1,47 @@
"""OME trigger route — manually invoke a registered strategy."""
from __future__ import annotations
from fastapi import APIRouter
from pydantic import BaseModel
from everos.core.errors import NotFoundError
from everos.core.observability.logging import get_logger
router = APIRouter(prefix="/api/v1/ome", tags=["ome"])
logger = get_logger(__name__)
class TriggerRequest(BaseModel):
"""Request body for ``POST /api/v1/ome/trigger``."""
name: str
timeout: float = 120.0
force: bool = False
class TriggerResponse(BaseModel):
"""Response body for ``POST /api/v1/ome/trigger``."""
status: str
name: str
@router.post("/trigger", response_model=TriggerResponse)
async def trigger(req: TriggerRequest) -> TriggerResponse:
"""Manually trigger a registered OME strategy and wait for completion."""
# Deferred: avoid importing heavy OME engine at module level.
from everos.service.memorize import _get_engine # noqa: SLF001
engine = _get_engine()
try:
await engine.trigger_manual(req.name, force=req.force)
except KeyError:
raise NotFoundError(f"strategy '{req.name}' not found") from None
logger.info("ome_trigger_manual", strategy=req.name)
idle = await engine.wait_idle(timeout=req.timeout)
if not idle:
logger.warning("ome_trigger_timeout", strategy=req.name, timeout=req.timeout)
return TriggerResponse(status="timeout", name=req.name)
return TriggerResponse(status="ok", name=req.name)

11
src/everos/entrypoints/api/routes/search.py
View File

@ -8,9 +8,9 @@ on the way out.
from __future__ import annotations
from fastapi import APIRouter, HTTPException
from fastapi import APIRouter
from everos.memory.search import FilterError, SearchRequest, SearchResponse
from everos.memory.search import SearchRequest, SearchResponse
from everos.service import search
router = APIRouter(prefix="/api/v1/memory", tags=["memory"])
@ -19,9 +19,4 @@ router = APIRouter(prefix="/api/v1/memory", tags=["memory"])
@router.post("/search", response_model=SearchResponse)
async def post_search(req: SearchRequest) -> SearchResponse:
"""Hybrid retrieval across the configured memory backends."""
try:
return await search(req)
except FilterError as exc:
# Filter-DSL violations surface as 422 with the compile message
# (mirrors /get's contract).
raise HTTPException(status_code=422, detail=str(exc)) from exc
return await search(req)

13
src/everos/entrypoints/api/utils.py Normal file
View File

@ -0,0 +1,13 @@
"""Shared helpers for the API layer (routes + exception handlers)."""
from __future__ import annotations
from fastapi import Request
from everos.core.observability.tracing import gen_request_id
def extract_request_id(request: Request) -> str:
"""Return the request_id set by middleware, or mint a fresh fallback."""
rid = getattr(request.state, "request_id", None)
return str(rid) if rid else gen_request_id()

14
src/everos/entrypoints/cli/commands/cascade.py
View File

@ -21,6 +21,7 @@ scanner background task is started.
from __future__ import annotations
import asyncio
import os
from contextlib import asynccontextmanager
from pathlib import Path
from typing import Annotated
@ -53,6 +54,19 @@ app = typer.Typer(
)
@app.callback()
def _cascade_callback(
root: str | None = typer.Option(
None,
"--root",
help="Memory root directory (env: EVEROS_ROOT, default: ~/.everos)",
),
) -> None:
"""Set memory root before any cascade subcommand runs."""
if root:
os.environ["EVEROS_ROOT"] = root
# ── shared runtime context ───────────────────────────────────────────────

85
src/everos/entrypoints/cli/commands/config_cmd.py Normal file
View File

@ -0,0 +1,85 @@
"""``everos config show`` — display effective configuration."""
from __future__ import annotations
import os
import typer
from everos.config.settings import resolve_root
app = typer.Typer(
name="config",
help="Configuration management",
no_args_is_help=True,
)
_SECRET_FIELDS = {"api_key"}
_SECTION_NAMES = (
"memory",
"api",
"sqlite",
"lancedb",
"llm",
"multimodal",
"embedding",
"rerank",
"boundary_detection",
"memorize",
"clustering",
"search",
"knowledge",
)
def _mask(value: str) -> str:
"""Mask a secret value, keeping first/last 4 chars if long enough."""
if len(value) <= 8:
return "****"
return value[:4] + "****" + value[-4:]
@app.command("show")
def show(
root: str | None = typer.Option(
None,
"--root",
help="Memory root directory",
),
) -> None:
"""Print the effective configuration."""
if root:
os.environ["EVEROS_ROOT"] = root
resolved = resolve_root(root)
typer.echo(f"Root: {resolved}")
typer.echo()
everos_toml = resolved / "everos.toml"
if everos_toml.is_file():
typer.echo(f"Config: {everos_toml}")
else:
typer.echo("Config: (no everos.toml found, using defaults)")
ome_toml = resolved / "ome.toml"
if ome_toml.is_file():
typer.echo(f"Strategy: {ome_toml}")
typer.echo()
from everos.config import load_settings
load_settings.cache_clear()
settings = load_settings()
for section_name in _SECTION_NAMES:
section = getattr(settings, section_name, None)
if section is None:
continue
typer.secho(f"[{section_name}]", bold=True)
for field_name, value in section.model_dump().items():
display = str(value)
if field_name in _SECRET_FIELDS and value:
display = _mask(str(value))
typer.echo(f" {field_name} = {display}")
typer.echo()

186
src/everos/entrypoints/cli/commands/init_cmd.py
View File

@ -1,10 +1,9 @@
"""``everos init`` — generate a starter ``.env`` from the packaged template.
"""``everos init`` — generate starter config files in the memory root.
The ``env.template`` ships inside the wheel as package data at
``everos/templates/env.template``. ``init`` reads it via
:mod:`importlib.resources`, so the command works identically for pip-
installed users and source-tree users (the file is the single source
of truth).
Copies the shipped ``default.toml`` and ``default_ome.toml`` templates
into the resolved memory root as ``everos.toml`` and ``ome.toml``
respectively. Users then edit these files to fill in API keys and tune
strategy schedules.
Subcommand mounted as ``everos init`` (top-level leaf command — not a
Typer group), to match the idiomatic ``alembic init`` / ``django-admin
@ -13,74 +12,14 @@ startproject`` shape.
from __future__ import annotations
import contextlib
import logging
import os
import sys
import tempfile
from importlib import resources
from pathlib import Path
import typer
_TEMPLATE_PACKAGE = "everos.templates"
_TEMPLATE_NAME = "env.template"
from everos.config.settings import resolve_root
_log = logging.getLogger("everos.cli.init")
def _read_template() -> str:
"""Read the packaged ``env.template`` from wheel resources.
Returns the file contents as a UTF-8 string. Raises ``RuntimeError``
on missing-file — if this fires it means the wheel was built from a
source tree where ``src/everos/templates/env.template`` was missing
(canonical location; auto-included via ``packages=["src/everos"]``
in ``pyproject.toml``).
"""
try:
return (
resources.files(_TEMPLATE_PACKAGE)
.joinpath(_TEMPLATE_NAME)
.read_text(encoding="utf-8")
)
except (FileNotFoundError, ModuleNotFoundError) as exc:
raise RuntimeError(
f"packaged template {_TEMPLATE_NAME!r} not found under "
f"{_TEMPLATE_PACKAGE!r}; the wheel is missing its "
"force-include entry (see pyproject.toml "
"[tool.hatch.build.targets.wheel.force-include])."
) from exc
def _xdg_default_path() -> Path:
"""``$XDG_CONFIG_HOME/everos/.env`` (default ``~/.config/everos/.env``)."""
xdg = os.environ.get("XDG_CONFIG_HOME") or "~/.config"
return Path(xdg).expanduser() / "everos" / ".env"
def _atomic_write(target: Path, content: str, mode: int = 0o600) -> None:
"""Write ``content`` to ``target`` atomically with ``mode`` permission.
Writes to a tempfile in the same directory then ``os.replace``s it
onto the target — guarantees either the full new file is visible or
the original (if any) is untouched. Permission bits applied before
the rename so the file is never readable by other users.
"""
target.parent.mkdir(parents=True, exist_ok=True)
fd, tmp_path = tempfile.mkstemp(
prefix=target.name + ".",
dir=target.parent,
)
try:
with os.fdopen(fd, "w", encoding="utf-8") as f:
f.write(content)
os.chmod(tmp_path, mode)
os.replace(tmp_path, target)
except Exception:
with contextlib.suppress(OSError):
os.unlink(tmp_path)
raise
_EVEROS_TEMPLATE = Path(__file__).resolve().parents[3] / "config" / "default.toml"
_OME_TEMPLATE = Path(__file__).resolve().parents[3] / "config" / "default_ome.toml"
def register(parent: typer.Typer) -> None:
@ -88,98 +27,65 @@ def register(parent: typer.Typer) -> None:
@parent.command("init")
def init(
to: str | None = typer.Option(
root: str | None = typer.Option(
None,
"--to",
help=(
"Target path for the .env file (default: ./.env). "
"Parent directories are created if needed."
),
"--root",
help="Memory root directory (default: ~/.everos)",
),
force: bool = typer.Option(
False,
"--force",
help="Overwrite an existing file at the target path.",
help="Overwrite existing files",
),
print_: bool = typer.Option(
False,
"--print",
help="Print the template to stdout instead of writing to disk.",
),
xdg: bool = typer.Option(
False,
"--xdg",
help=(
"Shortcut for --to=${XDG_CONFIG_HOME:-~/.config}/everos/.env "
"(mutually exclusive with --to)."
),
help="Print the everos.toml template to stdout instead of writing to disk.",
),
) -> None:
"""Generate a starter ``.env`` from the packaged template.
"""Generate starter configuration files.
Common flows::
everos init # writes ./.env
everos init --xdg # writes ~/.config/everos/.env
everos init --to /etc/foo.env --force
everos init --print > custom.env
everos init # writes to ~/.everos/
everos init --root /data/everos # writes to /data/everos/
everos init --force # overwrites existing files
everos init --print # prints everos.toml to stdout
Exit codes:
- 0 — written successfully (or printed to stdout).
- 1 — target file already exists and ``--force`` was not given.
- 2 — packaged template missing (wheel build problem).
- 3 — write failed (permissions / disk full / parent unwritable).
- 0 — files created successfully (or printed to stdout).
- 1 — files already exist and ``--force`` was not given.
"""
if xdg and to is not None:
typer.secho(
"error: --xdg and --to are mutually exclusive",
fg=typer.colors.RED,
err=True,
)
raise typer.Exit(code=2)
try:
template = _read_template()
except RuntimeError as exc:
typer.secho(f"error: {exc}", fg=typer.colors.RED, err=True)
raise typer.Exit(code=2) from exc
if print_:
sys.stdout.write(template)
import sys
sys.stdout.write(_EVEROS_TEMPLATE.read_text(encoding="utf-8"))
return
if xdg:
target = _xdg_default_path()
elif to is not None:
target = Path(to).expanduser().resolve()
else:
target = Path.cwd() / ".env"
resolved = resolve_root(root)
resolved.mkdir(parents=True, exist_ok=True)
if target.exists() and not force:
typer.secho(
f"error: {target} already exists; pass --force to overwrite",
fg=typer.colors.RED,
err=True,
)
everos_toml = resolved / "everos.toml"
ome_toml = resolved / "ome.toml"
created: list[Path] = []
for target, template in [
(everos_toml, _EVEROS_TEMPLATE),
(ome_toml, _OME_TEMPLATE),
]:
if target.exists() and not force:
typer.echo(f" exists: {target} (skipped)")
continue
target.write_bytes(template.read_bytes())
created.append(target)
typer.secho(f" created: {target}", fg=typer.colors.GREEN)
if not created:
typer.echo("Nothing to create (use --force to overwrite).")
raise typer.Exit(code=1)
try:
_atomic_write(target, template)
except OSError as exc:
typer.secho(
f"error: failed to write {target}: {exc}",
fg=typer.colors.RED,
err=True,
)
raise typer.Exit(code=3) from exc
# Friendly next-step block (stdout — quiet enough for piping).
size_kb = target.stat().st_size / 1024
typer.secho(f"✓ wrote {target} ({size_kb:.1f} KB)", fg=typer.colors.GREEN)
typer.echo("Next steps:")
typer.echo(" 1. Edit the file and fill in the API keys (see comments inside).")
typer.echo(" 2. Run `everos server start`.")
typer.echo(
"Docs: https://github.com/EverMind-AI/EverOS/blob/main/QUICKSTART.md"
)
typer.echo("\nNext steps:")
typer.echo(f" 1. Edit {everos_toml} — fill in API keys (see comments inside)")
root_flag = f" --root {resolved}" if root else ""
typer.echo(f" 2. Run: everos server start{root_flag}")

93
src/everos/entrypoints/cli/commands/server.py
View File

@ -11,11 +11,12 @@ from __future__ import annotations
import logging
import os
import sys
from pathlib import Path
import typer
import uvicorn
from everos.config.settings import resolve_root
app = typer.Typer(
name="server",
help="Run / manage the HTTP API server",
@ -23,58 +24,6 @@ app = typer.Typer(
)
def _resolve_env_file(explicit: str | None) -> Path | None:
"""Find the first existing ``.env`` along the four-layer search path.
Search order (highest-wins):
1. ``explicit`` — when the caller passed ``--env-file <path>``.
2. ``./.env`` — the current working directory (project-local convention).
3. ``${XDG_CONFIG_HOME:-~/.config}/everos/.env`` — XDG-standard user config.
4. ``~/.everos/.env`` — the project's default memory-root location.
Returns ``None`` if none of the layers exist (caller may then fall back
to inherited process env / CI secrets).
"""
candidates: list[Path] = []
if explicit:
candidates.append(Path(explicit).expanduser())
candidates.append(Path.cwd() / ".env")
xdg = os.environ.get("XDG_CONFIG_HOME") or "~/.config"
candidates.append(Path(xdg).expanduser() / "everos" / ".env")
candidates.append(Path("~/.everos/.env").expanduser())
for p in candidates:
try:
if p.is_file():
return p
except OSError:
# Path traversal / permission denied on a fallback candidate
# must not crash the search — skip and keep going.
continue
return None
def _load_env_file(path: str | None) -> Path | None:
"""Load environment variables from the resolved ``.env`` file.
Returns the path that was loaded, or ``None`` when no ``.env`` was
found anywhere along the search path. Existence of a ``.env`` is
optional — the user may rely entirely on inherited process env
(e.g. container / CI secret injection).
"""
resolved = _resolve_env_file(path)
if resolved is None:
return None
try:
from dotenv import load_dotenv
load_dotenv(resolved, override=False)
except ImportError:
# python-dotenv is in our deps; tolerate its absence anyway.
pass
return resolved
@app.command("start")
def start(
host: str | None = typer.Option(
@ -87,14 +36,10 @@ def start(
"--port",
help="Bind port (env: EVEROS_API__PORT, default: 8000)",
),
env_file: str | None = typer.Option(
root: str | None = typer.Option(
None,
"--env-file",
help=(
"Path to a dotenv file (highest priority). When omitted, "
"the server searches: ./.env → ${XDG_CONFIG_HOME:-~/.config}"
"/everos/.env → ~/.everos/.env. Run `everos init` to create one."
),
"--root",
help="Memory root directory (env: EVEROS_ROOT, default: ~/.everos)",
),
reload: bool = typer.Option(
False,
@ -108,10 +53,20 @@ def start(
),
) -> None:
"""Start the HTTP API server."""
loaded_env = _load_env_file(env_file)
if root:
os.environ["EVEROS_ROOT"] = root
resolved_root = resolve_root(root)
everos_toml = resolved_root / "everos.toml"
if not everos_toml.is_file():
typer.secho(
f"Error: {everos_toml} not found.\n"
f"Run `everos init` first to create configuration files.",
fg=typer.colors.RED,
err=True,
)
raise typer.Exit(code=1)
# Load settings AFTER .env is in place so EVEROS_API__HOST and
# EVEROS_API__PORT (and any other env override) are honored.
from everos.config import load_settings
settings = load_settings()
@ -125,13 +80,6 @@ def start(
configure_logging(level=log_level_resolved)
bootstrap_logger = logging.getLogger("everos.cli.server")
if loaded_env is not None:
bootstrap_logger.info("loaded env file: %s", loaded_env)
else:
bootstrap_logger.info(
"no .env found along the search path; relying on inherited env vars "
"(run `everos init` to generate one)"
)
bootstrap_logger.info("starting everos on %s:%d", host_resolved, port_resolved)
if host_resolved == "0.0.0.0":
bootstrap_logger.warning(
@ -147,11 +95,6 @@ def start(
reload=reload,
factory=True,
log_level=log_level_resolved.lower(),
# ``configure_logging()`` above already installed the root
# handler + structlog ProcessorFormatter. ``log_config=None``
# stops uvicorn from running its own ``dictConfig`` over
# ours; otherwise uvicorn / fastapi messages revert to the
# ``INFO:`` no-structlog format on every restart.
log_config=None,
)
except KeyboardInterrupt:

3
src/everos/entrypoints/cli/main.py
View File

@ -13,7 +13,7 @@ from __future__ import annotations
import typer
from .commands import cascade, demo, init_cmd, server
from .commands import cascade, config_cmd, demo, init_cmd, server
app = typer.Typer(
name="everos",
@ -24,6 +24,7 @@ app = typer.Typer(
app.add_typer(server.app, name="server")
app.add_typer(cascade.app, name="cascade")
app.add_typer(config_cmd.app, name="config")
# ``init`` is a top-level leaf command (not a Typer group) — match the
# idiomatic ``alembic init`` / ``django-admin startproject`` shape.

5
src/everos/infra/ome/_background/config_reloader.py
View File

@ -207,6 +207,11 @@ class ConfigReloader:
"""Fire-and-forget the watch loop. Idempotent: raises on double-start."""
if self._path is None:
return
if not self._path.exists():
raise FileNotFoundError(
f"{self._path} not found. "
"Run `everos init` to create configuration files."
)
if self._task is not None and not self._task.done():
raise RuntimeError("ConfigReloader already started")
self._task = asyncio.create_task(self._loop())

21
src/everos/infra/ome/_dispatch/registry.py
View File

@ -9,10 +9,8 @@ Kahn-style topological pass on the event-flow DAG implied by
from __future__ import annotations
from collections import defaultdict, deque
from collections.abc import Callable
from typing import Any
from everos.infra.ome.decorator import StrategyMeta
from everos.infra.ome.decorator import Strategy, StrategyMeta
from everos.infra.ome.events import BaseEvent, CronTick, IdleTick
from everos.infra.ome.exceptions import StartupValidationError
from everos.infra.ome.triggers import Cron, Idle, Immediate, Trigger
@ -24,18 +22,19 @@ class StrategyRegistry:
def __init__(self) -> None:
self._strategies: dict[str, StrategyMeta] = {}
def register(self, func: Callable[..., Any]) -> None:
"""Register a strategy function (reads ``_ome_strategy_meta``).
def register(self, strategy: Strategy) -> None:
"""Register a :class:`Strategy` returned by ``@offline_strategy``.
Raises ``StartupValidationError`` if ``func`` is not decorated
with ``@offline_strategy`` or if its name is already registered.
Raises:
StartupValidationError: If ``strategy`` is not a Strategy
instance or its name is already registered.
"""
meta = getattr(func, "_ome_strategy_meta", None)
if not isinstance(meta, StrategyMeta):
fn_name = getattr(func, "__name__", repr(func))
if not isinstance(strategy, Strategy):
label = getattr(strategy, "__name__", repr(strategy))
raise StartupValidationError(
f"register: {fn_name} is not decorated with @offline_strategy"
f"register: {label} is not decorated with @offline_strategy"
)
meta = strategy.meta
if meta.name in self._strategies:
raise StartupValidationError(
f"register: duplicate strategy name {meta.name!r}"

32
src/everos/infra/ome/_dispatch/runner.py
View File

@ -22,6 +22,7 @@ from __future__ import annotations
import asyncio
import traceback
from collections.abc import Awaitable, Callable
from typing import TYPE_CHECKING
from uuid import uuid4
from structlog.contextvars import bound_contextvars
@ -35,14 +36,18 @@ from everos.infra.ome.events import BaseEvent
from everos.infra.ome.exceptions import EmitNotDeclaredError, StrategyContractError
from everos.infra.ome.records import RunRecord
if TYPE_CHECKING:
from everos.infra.ome.engine import OfflineEngine
logger = get_logger(__name__)
class _RunCtx:
"""Per-invocation context handed to ``meta.func(event, ctx)``.
"""Implements :class:`~everos.infra.ome.context.StrategyContext` Protocol.
Carries ``run_id``, a strategy-scoped logger, and the ``emit``
callback that enforces the declared ``emits=[...]`` contract.
Carries ``run_id``, a strategy-scoped logger, the ``emit``
callback that enforces the declared ``emits=[...]`` contract,
and engine-delegated helpers for event/run queries.
"""
def __init__(
@ -52,12 +57,14 @@ class _RunCtx:
strategy_name: str,
emit_hook: Callable[[BaseEvent], Awaitable[None]],
declared_emits: frozenset[type[BaseEvent]],
engine: OfflineEngine,
) -> None:
self.run_id = run_id
self.logger = get_logger("ome.strategy")
self._emit_hook = emit_hook
self._declared = declared_emits
self._strategy_name = strategy_name
self._engine = engine
async def emit(self, event: BaseEvent) -> None:
if type(event) not in self._declared:
@ -67,6 +74,19 @@ class _RunCtx:
)
await self._emit_hook(event)
async def wait_for_event(
self,
event_id: str,
*,
timeout: float = 120.0, # noqa: ASYNC109
) -> list[RunRecord]:
"""Poll until all runs for ``event_id`` reach a terminal status."""
return await self._engine.wait_for_event(event_id, timeout=timeout)
async def list_runs_by_event_id(self, event_id: str) -> list[RunRecord]:
"""Return all run records triggered by ``event_id``."""
return await self._engine.list_runs_by_event_id(event_id)
class Runner:
"""Drive one strategy invocation through retries to a terminal state."""
@ -78,11 +98,13 @@ class Runner:
engine_sem: asyncio.Semaphore,
emit_hook: Callable[[BaseEvent], Awaitable[None]],
on_dead_letter: Callable[[RunRecord], None] | None = None,
engine: OfflineEngine,
) -> None:
self._rec = run_record_store
self._sem = engine_sem
self._emit_hook = emit_hook
self._on_dead_letter = on_dead_letter
self._engine = engine
async def run(
self,
@ -143,6 +165,7 @@ class Runner:
strategy_name=meta.name,
emit_hook=self._emit_hook,
declared_emits=meta.emits,
engine=self._engine,
)
with bound_contextvars( # type: ignore[arg-type] # structlog typed as Generator; @contextmanager wraps at runtime (structlog/contextvars.py:170)
strategy_name=meta.name,
@ -156,6 +179,7 @@ class Runner:
event_topic=event_topic,
event_payload=event_payload,
max_retries_snapshot=max_retries_snapshot,
event_id=event.event_id,
):
return True # mark_running failed; abort run, no DB row exists
try:
@ -194,6 +218,7 @@ class Runner:
event_topic: str,
event_payload: str,
max_retries_snapshot: int,
event_id: str,
) -> bool:
"""Persist this attempt as RUNNING; return ``False`` on write failure.
@ -210,6 +235,7 @@ class Runner:
event_topic=event_topic,
event_payload=event_payload,
max_retries_snapshot=max_retries_snapshot,
event_id=event_id,
)
except Exception: # noqa: BLE001
logger.exception(

19
src/everos/infra/ome/_stores/run_record.py
View File

@ -40,14 +40,15 @@ class RunRecordStore:
event_topic: str,
event_payload: str,
max_retries_snapshot: int,
event_id: str,
) -> None:
"""Insert a new RUNNING row and trim the strategy's ring buffer atomically."""
async with self._storage.transaction() as conn:
await conn.execute(
"INSERT INTO run_record "
"(run_id, strategy_name, status, attempt, started_at, "
" event_topic, event_payload, max_retries_snapshot) "
"VALUES (?, ?, ?, ?, ?, ?, ?, ?)",
" event_topic, event_payload, max_retries_snapshot, event_id) "
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)",
(
run_id,
strategy_name,
@ -57,6 +58,7 @@ class RunRecordStore:
event_topic,
event_payload,
max_retries_snapshot,
event_id,
),
)
await conn.execute(
@ -145,10 +147,20 @@ class RunRecordStore:
rows = await cur.fetchall()
return [_row_to_record(r) for r in rows]
async def list_by_event_id(self, event_id: str) -> list[RunRecord]:
"""Return all runs triggered by the given ``event_id``."""
async with self._storage.connect() as conn:
cur = await conn.execute(
_SELECT_COLUMNS + " WHERE event_id = ? ORDER BY started_at DESC",
(event_id,),
)
rows = await cur.fetchall()
return [_row_to_record(r) for r in rows]
_SELECT_COLUMNS = (
"SELECT run_id, strategy_name, status, attempt, started_at, finished_at, "
" error, event_topic, event_payload, max_retries_snapshot "
" error, event_topic, event_payload, max_retries_snapshot, event_id "
"FROM run_record"
)
@ -165,4 +177,5 @@ def _row_to_record(row: tuple) -> RunRecord:
event_topic=row[7],
event_payload=row[8],
max_retries_snapshot=row[9],
event_id=row[10],
)

24
src/everos/infra/ome/_stores/storage.py
View File

@ -55,12 +55,15 @@ CREATE TABLE IF NOT EXISTS run_record (
error TEXT,
event_topic TEXT NOT NULL,
event_payload TEXT NOT NULL,
max_retries_snapshot INTEGER NOT NULL
max_retries_snapshot INTEGER NOT NULL,
event_id TEXT NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_run_strategy_started
ON run_record (strategy_name, started_at DESC);
CREATE INDEX IF NOT EXISTS idx_run_status_started
ON run_record (status, started_at DESC);
CREATE INDEX IF NOT EXISTS idx_run_event_id
ON run_record (event_id);
"""
_INIT_PRAGMAS = ("PRAGMA journal_mode=WAL",)
@ -78,14 +81,31 @@ class OMEStorage:
self.db_path = db_path
async def init(self) -> None:
"""Create parent dirs + apply file-level pragmas + create schema."""
"""Create parent dirs + apply file-level pragmas + create schema.
Runs forward-only migrations for existing databases (e.g. adding
the ``event_id`` column introduced in P3).
"""
self.db_path.parent.mkdir(parents=True, exist_ok=True)
async with aiosqlite.connect(self.db_path) as conn:
for pragma in _INIT_PRAGMAS:
await conn.execute(pragma)
await self._migrate(conn)
await conn.executescript(_SCHEMA)
await conn.commit()
@staticmethod
async def _migrate(conn: aiosqlite.Connection) -> None:
"""Forward-only column migrations for existing databases."""
cur = await conn.execute("PRAGMA table_info(run_record)")
columns = {row[1] for row in await cur.fetchall()}
if not columns:
return
if "event_id" not in columns:
await conn.execute(
"ALTER TABLE run_record ADD COLUMN event_id TEXT NOT NULL DEFAULT ''"
)
@asynccontextmanager
async def connect(self) -> AsyncIterator[aiosqlite.Connection]:
"""Yield an aiosqlite connection with per-connection pragmas applied."""

27
src/everos/infra/ome/context.py
View File

@ -13,21 +13,34 @@ from typing import Protocol
from structlog.types import FilteringBoundLogger
from everos.infra.ome.events import BaseEvent
from everos.infra.ome.records import RunRecord
class StrategyContext(Protocol):
"""Per-run context handed to a strategy function.
- run_id: the current RunRecord id (string).
- logger: structlog logger; ``strategy_name`` / ``run_id`` /
``attempt`` are auto-injected into every log record in this call
— strategies don't have to use this specific logger to get those
fields.
- emit(event): chain-emit a follow-up event (must be in decorator's
``emits=[...]``, else EmitNotDeclaredError).
Attributes:
run_id: The current RunRecord id.
logger: Structlog logger with ``strategy_name`` / ``run_id`` /
``attempt`` auto-bound.
emit: Chain-emit a follow-up event (must be in decorator's
``emits=[...]``, else EmitNotDeclaredError).
wait_for_event: Poll until all runs triggered by an event_id
reach a terminal status.
list_runs_by_event_id: Return all run records triggered by an
event_id.
"""
run_id: str
logger: FilteringBoundLogger
async def emit(self, event: BaseEvent) -> None: ...
async def wait_for_event(
self,
event_id: str,
*,
timeout: float = 120.0, # noqa: ASYNC109
) -> list[RunRecord]: ...
async def list_runs_by_event_id(self, event_id: str) -> list[RunRecord]: ...

103
src/everos/infra/ome/decorator.py
View File

@ -1,7 +1,7 @@
"""@offline_strategy decorator — attaches StrategyMeta to the function.
"""@offline_strategy decorator — attaches StrategyMeta to a Strategy wrapper.
Decorator is side-effect-free; engine collects via explicit
`engine.register(func)`.
``engine.register(strategy)``.
"""
from __future__ import annotations
@ -9,15 +9,22 @@ from __future__ import annotations
import inspect
from collections.abc import Awaitable, Callable
from dataclasses import dataclass
from typing import Any, TypeVar, overload
from everos.infra.ome.context import StrategyContext
from everos.infra.ome.events import BaseEvent
from everos.infra.ome.events import BaseEvent, CronTick, IdleTick
from everos.infra.ome.gates import Counter
from everos.infra.ome.triggers import Trigger
from everos.infra.ome.triggers import Cron, Idle, Immediate, Trigger
type AppliesTo = str | Callable[[BaseEvent], bool] | None
type StrategyFn = Callable[[BaseEvent, StrategyContext], Awaitable[None]]
_E = TypeVar("_E", bound=BaseEvent)
_CronStrategyFn = Callable[[CronTick, StrategyContext], Awaitable[None]]
_IdleStrategyFn = Callable[[IdleTick, StrategyContext], Awaitable[None]]
_EventStrategyFn = Callable[[_E, StrategyContext], Awaitable[None]]
@dataclass(frozen=True)
class StrategyMeta:
@ -33,6 +40,68 @@ class StrategyMeta:
func: StrategyFn
class Strategy:
"""Wrapper returned by :func:`offline_strategy`.
Carries typed :attr:`meta` and delegates ``__call__`` to the
original async function — so ``await my_strategy(event, ctx)``
works transparently in both production and tests.
Args:
meta: Frozen strategy metadata captured at decoration time.
"""
__slots__ = ("meta",)
def __init__(self, meta: StrategyMeta) -> None:
self.meta = meta
async def __call__(self, event: BaseEvent, ctx: StrategyContext) -> None:
await self.meta.func(event, ctx)
def __repr__(self) -> str:
return f"Strategy({self.meta.name!r})"
@overload
def offline_strategy(
*,
name: str,
trigger: Cron,
emits: list[type[BaseEvent]],
applies_to: AppliesTo = ...,
gate: Counter | None = ...,
max_retries: int | None = ...,
enabled: bool = ...,
) -> Callable[[_CronStrategyFn], Strategy]: ...
@overload
def offline_strategy(
*,
name: str,
trigger: Idle,
emits: list[type[BaseEvent]],
applies_to: AppliesTo = ...,
gate: Counter | None = ...,
max_retries: int | None = ...,
enabled: bool = ...,
) -> Callable[[_IdleStrategyFn], Strategy]: ...
@overload
def offline_strategy(
*,
name: str,
trigger: Immediate,
emits: list[type[BaseEvent]],
applies_to: AppliesTo = ...,
gate: Counter | None = ...,
max_retries: int | None = ...,
enabled: bool = ...,
) -> Callable[[_EventStrategyFn[_E]], Strategy]: ...
def offline_strategy(
*,
name: str,
@ -42,13 +111,30 @@ def offline_strategy(
gate: Counter | None = None,
max_retries: int | None = None,
enabled: bool = True,
) -> Callable[[StrategyFn], StrategyFn]:
"""Mark an async function as an OME strategy."""
) -> Any: # overloads above provide call-site precision
"""Mark an async function as an OME strategy.
Args:
name: Unique strategy name (used for logging, run records, config).
trigger: When to fire — ``Cron``, ``Idle``, or ``Immediate``.
emits: Event types this strategy may emit via ``ctx.emit()``.
applies_to: Optional gate predicate (None = all events).
gate: Optional counter-based rate limiter.
max_retries: Override engine default; ``None`` uses engine config.
enabled: ``False`` disables without unregistering.
Returns:
Decorator that wraps the function in a :class:`Strategy` instance.
Raises:
ValueError: If ``name`` is empty or whitespace-only.
TypeError: If the decorated function is not async.
"""
if not name or not name.strip():
raise ValueError("offline_strategy: name must be a non-empty string")
def wrap(func: StrategyFn) -> StrategyFn:
def wrap(func: StrategyFn) -> Strategy:
if not inspect.iscoroutinefunction(func):
raise TypeError(
f"offline_strategy: {func.__name__} must be async (coroutine function)"
@ -63,7 +149,6 @@ def offline_strategy(
enabled=enabled,
func=func,
)
func._ome_strategy_meta = meta # type: ignore[attr-defined]
return func
return Strategy(meta)
return wrap

101
src/everos/infra/ome/engine.py
View File

@ -36,7 +36,7 @@ from everos.infra.ome._stores.idle import IdleStore
from everos.infra.ome._stores.run_record import RunRecordStore
from everos.infra.ome._stores.storage import OMEStorage
from everos.infra.ome.config import OMEConfig
from everos.infra.ome.decorator import StrategyMeta
from everos.infra.ome.decorator import Strategy, StrategyMeta
from everos.infra.ome.events import BaseEvent, CronTick, ManualTick, resolve_topic
from everos.infra.ome.exceptions import (
EngineCallFromStrategyError,
@ -120,7 +120,12 @@ async def _cron_entry(engine_id: str, strategy_name: str) -> None:
Looks the engine up by id and emits ``CronTick`` so the event flows
back through the standard dispatch pipeline.
Clears ``_CURRENT_STRATEGY`` because APScheduler may inherit the
context from a running strategy task, which would trip the
``_guard_no_strategy_call`` check on ``engine.emit()``.
"""
_CURRENT_STRATEGY.set(None)
engine = _ENGINES.get(engine_id)
if engine is None:
logger.error(
@ -136,8 +141,10 @@ async def _idle_entry(engine_id: str, strategy_name: str) -> None:
"""Module-level APS jobstore callback for Idle IntervalTriggers.
Looks the engine up by id and hands off to
:meth:`OfflineEngine.run_idle_scan`.
:meth:`OfflineEngine.run_idle_scan`. Clears ``_CURRENT_STRATEGY``
for the same reason as :func:`_cron_entry`.
"""
_CURRENT_STRATEGY.set(None)
engine = _ENGINES.get(engine_id)
if engine is None:
logger.error(
@ -200,16 +207,22 @@ class OfflineEngine:
self._active_runs = 0
self._idle_event: asyncio.Event | None = None
def register(self, func: Callable[..., Any]) -> None:
"""Register a strategy decorated with :func:`offline_strategy`.
def register(self, strategy: Strategy) -> None:
"""Register a :class:`Strategy` returned by ``@offline_strategy``.
Must be called before :meth:`start`; registering after start raises
:class:`OMEError` because the scheduler has already snapshotted
the strategy set for Cron / Idle job creation.
Args:
strategy: A Strategy instance produced by the decorator.
Raises:
OMEError: If the engine has already started.
"""
if self._started:
raise OMEError("register: cannot register after start()")
self._registry.register(func)
self._registry.register(strategy)
@_refuse_inside_strategy
def reschedule_cron_job(self, name: str, expr: str) -> None:
@ -301,6 +314,7 @@ class OfflineEngine:
engine_sem=self._engine_sem,
emit_hook=self._dispatch_event,
on_dead_letter=self._on_dead_letter,
engine=self,
)
self._idle_store = IdleStore(storage=self._storage)
@ -519,14 +533,15 @@ class OfflineEngine:
def _acquire_lock(self) -> None:
lock_path = Path(str(self._config.jobstore_path) + ".lock")
lock_path.parent.mkdir(parents=True, exist_ok=True)
handle = open(lock_path, "a+") # noqa: SIM115
try:
handle = open(lock_path, "a+") # noqa: SIM115
portalocker.lock(handle, portalocker.LOCK_EX | portalocker.LOCK_NB)
self._lock_handle = handle
except portalocker.LockException as e:
handle.close()
raise EngineLockHeldError(
f"another OfflineEngine instance already holds {lock_path}"
) from e
self._lock_handle = handle
def _release_lock(self) -> None:
if self._lock_handle is not None:
@ -795,3 +810,75 @@ class OfflineEngine:
if not self._started:
raise OMEError("get_run_status: engine not started")
return await self._run_record_store.get(run_id)
async def list_runs_by_event_id(self, event_id: str) -> list[RunRecord]:
"""Return all run records triggered by ``event_id``.
Not guarded by ``_refuse_inside_strategy`` — designed to be
called from inside a strategy (e.g. Reflection waiting for
downstream extraction runs).
Args:
event_id: The ``BaseEvent.event_id`` of the triggering event.
Returns:
All ``RunRecord`` instances whose ``event_id`` matches,
newest first.
Raises:
OMEError: Engine has not been started.
"""
if not self._started:
raise OMEError("list_runs_by_event_id: engine not started")
return await self._run_record_store.list_by_event_id(event_id)
_TERMINAL_STATUSES = frozenset(
{
RunStatus.SUCCESS,
RunStatus.FAILED,
RunStatus.DEAD_LETTER,
RunStatus.CRASHED,
}
)
async def wait_for_event(
self,
event_id: str,
*,
timeout: float = 120.0, # noqa: ASYNC109
poll_interval: float = 0.5,
) -> list[RunRecord]:
"""Poll until all runs triggered by ``event_id`` reach a terminal status.
Not guarded by ``_refuse_inside_strategy`` — designed to be
called from inside a strategy (e.g. Reflection waiting for
downstream extraction runs to complete before deprecating).
Args:
event_id: The ``BaseEvent.event_id`` of the triggering event.
timeout: Maximum seconds to wait before raising ``TimeoutError``.
poll_interval: Seconds between polls.
Returns:
All ``RunRecord`` instances for ``event_id`` once every run
has reached a terminal status (SUCCESS / FAILED / DEAD_LETTER
/ CRASHED).
Raises:
TimeoutError: If any run is still non-terminal after ``timeout``.
OMEError: Engine has not been started.
"""
if not self._started:
raise OMEError("wait_for_event: engine not started")
deadline = asyncio.get_event_loop().time() + timeout
while True:
runs = await self._run_record_store.list_by_event_id(event_id)
if runs and all(r.status in self._TERMINAL_STATUSES for r in runs):
return runs
remaining = deadline - asyncio.get_event_loop().time()
if remaining <= 0:
raise TimeoutError(
f"wait_for_event: timed out after {timeout}s "
f"for event_id={event_id!r}"
)
await asyncio.sleep(min(poll_interval, remaining))

1
src/everos/infra/ome/records.py
View File

@ -56,6 +56,7 @@ class RunRecord(BaseModel):
),
]
max_retries_snapshot: Annotated[int, Field(ge=0)]
event_id: str = ""
@model_validator(mode="after")
def _check_status_invariants(self) -> Self:

39
src/everos/infra/ome/testing/fakes.py
View File

@ -8,23 +8,22 @@ from __future__ import annotations
from everos.core.observability.logging import get_logger
from everos.infra.ome.events import BaseEvent
from everos.infra.ome.records import RunRecord
class FakeStrategyContext:
"""Implements StrategyContext Protocol; collects emit() calls in a list.
Args:
run_id: Run identifier, defaults to ``"fake_run"``.
Attributes:
run_id: Unique identifier for this run (default: "fake_run").
run_id: Unique identifier for this run (default: ``"fake_run"``).
logger: A structlog BoundLogger for test logging.
emitted: List of BaseEvent objects passed to emit().
"""
def __init__(self, *, run_id: str = "fake_run") -> None:
"""Initialize a FakeStrategyContext.
Args:
run_id: Run identifier, defaults to "fake_run".
"""
self.run_id = run_id
self.logger = get_logger("ome.fake_ctx")
self.emitted: list[BaseEvent] = []
@ -36,3 +35,31 @@ class FakeStrategyContext:
event: The BaseEvent to emit.
"""
self.emitted.append(event)
async def wait_for_event(
self,
event_id: str,
*,
timeout: float = 120.0, # noqa: ASYNC109
) -> list[RunRecord]:
"""No-op stub; returns an empty list.
Args:
event_id: The event identifier to wait for.
timeout: Maximum seconds to wait (unused in fake).
Returns:
Empty list (no runs in test doubles).
"""
return []
async def list_runs_by_event_id(self, event_id: str) -> list[RunRecord]:
"""No-op stub; returns an empty list.
Args:
event_id: The event identifier to query.
Returns:
Empty list (no runs in test doubles).
"""
return []

9
src/everos/infra/ome/testing/harness.py
View File

@ -12,6 +12,7 @@ from tempfile import mkdtemp
from typing import Any
from everos.infra.ome.config import OMEConfig
from everos.infra.ome.decorator import Strategy
from everos.infra.ome.engine import OfflineEngine
from everos.infra.ome.events import BaseEvent
from everos.infra.ome.records import RunRecord, RunStatus
@ -55,13 +56,13 @@ class StrategyTestHarness:
finally:
shutil.rmtree(self._tmpdir, ignore_errors=True) # noqa: SLF001
def register(self, func: Any) -> None:
"""Register a strategy function.
def register(self, strategy: Strategy) -> None:
"""Register a :class:`Strategy` returned by ``@offline_strategy``.
Args:
func: A function decorated with @offline_strategy.
strategy: A Strategy instance produced by the decorator.
"""
self._engine.register(func)
self._engine.register(strategy)
async def start(self) -> None:
"""Start the OfflineEngine."""

7
src/everos/infra/persistence/lancedb/__init__.py
View File

@ -13,8 +13,10 @@ External usage::
from everos.infra.persistence.lancedb import (
get_connection, get_table, dispose_connection,
Episode, AtomicFact, Foresight, AgentCase, AgentSkill, UserProfile,
KnowledgeTopic,
episode_repo, atomic_fact_repo, foresight_repo,
agent_case_repo, agent_skill_repo, user_profile_repo,
knowledge_topic_repo,
)
Three index kinds: scalar / BM25 / vector. Tables are created lazily on
@ -33,12 +35,14 @@ from .repos import agent_skill_repo as agent_skill_repo
from .repos import atomic_fact_repo as atomic_fact_repo
from .repos import episode_repo as episode_repo
from .repos import foresight_repo as foresight_repo
from .repos import knowledge_topic_repo as knowledge_topic_repo
from .repos import user_profile_repo as user_profile_repo
from .tables import AgentCase as AgentCase
from .tables import AgentSkill as AgentSkill
from .tables import AtomicFact as AtomicFact
from .tables import Episode as Episode
from .tables import Foresight as Foresight
from .tables import KnowledgeTopic as KnowledgeTopic
from .tables import ParentType as ParentType
from .tables import UserProfile as UserProfile
@ -49,6 +53,7 @@ _BUSINESS_SCHEMAS = (
AgentCase,
AgentSkill,
UserProfile,
KnowledgeTopic,
)
@ -115,6 +120,7 @@ __all__ = [
"AtomicFact",
"Episode",
"Foresight",
"KnowledgeTopic",
"LanceDBSchemaMismatchError",
"ParentType",
"UserProfile",
@ -127,6 +133,7 @@ __all__ = [
"foresight_repo",
"get_connection",
"get_table",
"knowledge_topic_repo",
"user_profile_repo",
"verify_business_schemas",
]

3
src/everos/infra/persistence/lancedb/repos/__init__.py
View File

@ -15,6 +15,7 @@ External usage::
agent_case_repo,
agent_skill_repo,
user_profile_repo,
knowledge_topic_repo,
)
await episode_repo.add([Episode(...)])
@ -25,6 +26,7 @@ from .agent_skill import agent_skill_repo as agent_skill_repo
from .atomic_fact import atomic_fact_repo as atomic_fact_repo
from .episode import episode_repo as episode_repo
from .foresight import foresight_repo as foresight_repo
from .knowledge_topic import knowledge_topic_repo as knowledge_topic_repo
from .user_profile import user_profile_repo as user_profile_repo
__all__ = [
@ -33,5 +35,6 @@ __all__ = [
"atomic_fact_repo",
"episode_repo",
"foresight_repo",
"knowledge_topic_repo",
"user_profile_repo",
]

22
src/everos/infra/persistence/lancedb/repos/knowledge_topic.py Normal file
View File

@ -0,0 +1,22 @@
"""LanceDB repo singleton for the ``knowledge_topic`` table."""
from __future__ import annotations
from lancedb import AsyncTable
from everos.core.persistence.lancedb import LanceRepoBase
from ..lancedb_manager import get_table
from ..tables.knowledge_topic import KnowledgeTopic
class _KnowledgeTopicRepo(LanceRepoBase[KnowledgeTopic]):
"""LanceDB repository for the ``knowledge_topic`` table."""
schema = KnowledgeTopic
async def _table_lookup(self) -> AsyncTable:
return await get_table(self.schema.TABLE_NAME, self.schema)
knowledge_topic_repo = _KnowledgeTopicRepo()

3
src/everos/infra/persistence/lancedb/tables/__init__.py
View File

@ -12,6 +12,7 @@ External usage::
AgentCase,
AgentSkill,
UserProfile,
KnowledgeTopic,
ParentType,
)
"""
@ -22,6 +23,7 @@ from .agent_skill import AgentSkill as AgentSkill
from .atomic_fact import AtomicFact as AtomicFact
from .episode import Episode as Episode
from .foresight import Foresight as Foresight
from .knowledge_topic import KnowledgeTopic as KnowledgeTopic
from .user_profile import UserProfile as UserProfile
__all__ = [
@ -30,6 +32,7 @@ __all__ = [
"AtomicFact",
"Episode",
"Foresight",
"KnowledgeTopic",
"ParentType",
"UserProfile",
]

25
src/everos/infra/persistence/lancedb/tables/_parent_type.py
View File

@ -1,16 +1,19 @@
"""``ParentType`` — provenance label for memory records linked back to a source.
Currently the only value is :attr:`ParentType.MEMCELL`: every business row
(episode / foresight / atomic_fact / agent_case) points back to a source
MemCell. The earlier opensource design enumerated ``"episode"`` as an
alternative parent but the production path never wrote that value, so the
new framework collapses the enum to its single in-use member.
Three values cover the current provenance graph:
Kept as an :class:`enum.Enum` (rather than a bare string constant) so that
adding a future parent kind stays a non-breaking enum extension. LanceDB's
pydantic-to-arrow conversion does not accept ``Enum`` field annotations,
so table schemas declare ``parent_type: str = ParentType.MEMCELL.value``
and reference the enum only at the default-value level.
* :attr:`ParentType.MEMCELL` — the original ingestion unit; every
business row (episode / foresight / atomic_fact / agent_case) points
back to a source MemCell by default.
* :attr:`ParentType.EPISODE` — used by atomic facts that are extracted
from an episode rather than directly from a MemCell.
* :attr:`ParentType.CLUSTER` — used by merged episodes produced by the
Reflection consolidation mechanism.
LanceDB's pydantic-to-arrow conversion does not accept ``Enum`` field
annotations, so table schemas declare
``parent_type: str = ParentType.MEMCELL.value`` and reference the enum
only at the default-value level.
"""
from __future__ import annotations
@ -22,3 +25,5 @@ class ParentType(StrEnum):
"""Provenance label of a memory record's parent."""
MEMCELL = "memcell"
EPISODE = "episode"
CLUSTER = "cluster"

7
src/everos/infra/persistence/lancedb/tables/atomic_fact.py
View File

@ -34,7 +34,7 @@ class AtomicFact(BaseLanceTable):
app_id: str = "default"
project_id: str = "default"
"""App / project scope (default ``"default"``); cascade fills from md path."""
session_id: str
session_id: str | None = None
timestamp: _dt.datetime
parent_type: str = ParentType.MEMCELL.value
@ -59,4 +59,9 @@ class AtomicFact(BaseLanceTable):
(owner_id / session_id / timestamp / parent_id / sender_ids) are
NOT in the hash."""
deprecated_by: str | None = None
"""Soft-delete marker set by Reflection when this fact is
consolidated. Value is the cluster entry_id that supersedes this
row. ``NULL`` means the row is still active."""
vector: Vector(_DIM) # type: ignore[valid-type]

7
src/everos/infra/persistence/lancedb/tables/episode.py
View File

@ -36,7 +36,7 @@ class Episode(BaseLanceTable):
app_id: str = "default"
project_id: str = "default"
"""App / project scope (default ``"default"``); cascade fills from md path."""
session_id: str
session_id: str | None = None
timestamp: _dt.datetime
parent_type: str = ParentType.MEMCELL.value
@ -75,4 +75,9 @@ class Episode(BaseLanceTable):
NOT in the hash so editing them doesn't waste an embedding call.
See ``16_cascade_impl_design.md`` §3.3."""
deprecated_by: str | None = None
"""Soft-delete marker set by Reflection when this episode is
consolidated into a cluster. Value is the cluster entry_id that
supersedes this row. ``NULL`` means the row is still active."""
vector: Vector(_DIM) # type: ignore[valid-type]

2
src/everos/infra/persistence/lancedb/tables/foresight.py
View File

@ -35,7 +35,7 @@ class Foresight(BaseLanceTable):
app_id: str = "default"
project_id: str = "default"
"""App / project scope (default ``"default"``); cascade fills from md path."""
session_id: str
session_id: str | None = None
timestamp: _dt.datetime
"""Foresight generation time."""

36
src/everos/infra/persistence/lancedb/tables/knowledge_topic.py Normal file
View File

@ -0,0 +1,36 @@
"""LanceDB table schema for L2 knowledge topic nodes."""
from __future__ import annotations
import datetime as dt
from typing import ClassVar
from everos.core.persistence.lancedb import BaseLanceTable, Vector
_DIM = 1024
class KnowledgeTopic(BaseLanceTable):
"""L2 topic node — dense + dual-column BM25 retrieval."""
TABLE_NAME: ClassVar[str] = "knowledge_topic"
BM25_FIELDS: ClassVar[list[str]] = ["summary_tokens", "content_tokens"]
id: str
doc_id: str
category_id: str
app_id: str
project_id: str
topic_name: str
topic_path: str
depth: int
parent_node_id: str = ""
summary: str
summary_tokens: str
content_tokens: str
content_labels: list[str] = []
md_path: str
content_sha256: str
vector: Vector(_DIM) # type: ignore[valid-type] -- Vector() is runtime-constructed; static analyzers cannot verify
created_at: dt.datetime
updated_at: dt.datetime

10
src/everos/infra/persistence/markdown/__init__.py
View File

@ -33,6 +33,8 @@ from .mds import AgentSkillFrontmatter as AgentSkillFrontmatter
from .mds import AtomicFactDailyFrontmatter as AtomicFactDailyFrontmatter
from .mds import EpisodeDailyFrontmatter as EpisodeDailyFrontmatter
from .mds import ForesightDailyFrontmatter as ForesightDailyFrontmatter
from .mds import KnowledgeDocumentFrontmatter as KnowledgeDocumentFrontmatter
from .mds import KnowledgeTopicFrontmatter as KnowledgeTopicFrontmatter
from .mds import UserProfileFrontmatter as UserProfileFrontmatter
from .readers import AgentCaseReader as AgentCaseReader
from .readers import AgentSkillReader as AgentSkillReader
@ -41,12 +43,15 @@ from .readers import BaseDailyReader as BaseDailyReader
from .readers import EpisodeReader as EpisodeReader
from .readers import ForesightReader as ForesightReader
from .readers import ProfileReader as ProfileReader
from .readers import ensure_taxonomy as ensure_taxonomy
from .readers import parse_taxonomy as parse_taxonomy
from .writers import AgentCaseWriter as AgentCaseWriter
from .writers import AgentSkillWriter as AgentSkillWriter
from .writers import AtomicFactWriter as AtomicFactWriter
from .writers import BaseDailyWriter as BaseDailyWriter
from .writers import EpisodeWriter as EpisodeWriter
from .writers import ForesightWriter as ForesightWriter
from .writers import KnowledgeWriter as KnowledgeWriter
from .writers import ProfileWriter as ProfileWriter
__all__ = [
@ -67,7 +72,12 @@ __all__ = [
"ForesightDailyFrontmatter",
"ForesightReader",
"ForesightWriter",
"KnowledgeDocumentFrontmatter",
"KnowledgeTopicFrontmatter",
"KnowledgeWriter",
"ProfileReader",
"ProfileWriter",
"UserProfileFrontmatter",
"ensure_taxonomy",
"parse_taxonomy",
]

6
src/everos/infra/persistence/markdown/mds/__init__.py
View File

@ -28,6 +28,10 @@ from .agent_skill import AgentSkillFrontmatter as AgentSkillFrontmatter
from .atomic_fact import AtomicFactDailyFrontmatter as AtomicFactDailyFrontmatter
from .episode import EpisodeDailyFrontmatter as EpisodeDailyFrontmatter
from .foresight import ForesightDailyFrontmatter as ForesightDailyFrontmatter
from .knowledge_document import (
KnowledgeDocumentFrontmatter as KnowledgeDocumentFrontmatter,
)
from .knowledge_topic import KnowledgeTopicFrontmatter as KnowledgeTopicFrontmatter
from .profile import UserProfileFrontmatter as UserProfileFrontmatter
__all__ = [
@ -36,5 +40,7 @@ __all__ = [
"AtomicFactDailyFrontmatter",
"EpisodeDailyFrontmatter",
"ForesightDailyFrontmatter",
"KnowledgeDocumentFrontmatter",
"KnowledgeTopicFrontmatter",
"UserProfileFrontmatter",
]

3
src/everos/infra/persistence/markdown/mds/atomic_fact.py
View File

@ -17,6 +17,8 @@ from __future__ import annotations
import datetime as _dt
from typing import ClassVar, Literal
from pydantic import Field
from everos.core.persistence.markdown import (
DailyLogPathMixin,
UserScopedFrontmatter,
@ -36,3 +38,4 @@ class AtomicFactDailyFrontmatter(DailyLogPathMixin, UserScopedFrontmatter):
entry_count: int = 0
created_at: _dt.datetime | None = None
last_appended_at: _dt.datetime | None = None
deprecated_entries: dict[str, str] = Field(default_factory=dict)

3
src/everos/infra/persistence/markdown/mds/episode.py
View File

@ -12,6 +12,8 @@ from __future__ import annotations
import datetime as _dt
from typing import ClassVar, Literal
from pydantic import Field
from everos.core.persistence.markdown import (
DailyLogPathMixin,
UserScopedFrontmatter,
@ -31,3 +33,4 @@ class EpisodeDailyFrontmatter(DailyLogPathMixin, UserScopedFrontmatter):
entry_count: int = 0
created_at: _dt.datetime | None = None
last_appended_at: _dt.datetime | None = None
deprecated_entries: dict[str, str] = Field(default_factory=dict)

26
src/everos/infra/persistence/markdown/mds/knowledge_document.py Normal file
View File

@ -0,0 +1,26 @@
"""Frontmatter schema for ``knowledge/{category}/{doc_title}/index.md``."""
from __future__ import annotations
from typing import Literal
from everos.core.persistence.markdown import (
BaseFrontmatter,
KnowledgeDocumentPathMixin,
KnowledgeScopedMixin,
)
class KnowledgeDocumentFrontmatter(
KnowledgeDocumentPathMixin,
KnowledgeScopedMixin,
BaseFrontmatter,
):
"""L1 document-level frontmatter (index.md). Body = doc summary."""
type: Literal["knowledge_document"] = "knowledge_document"
doc_id: str
category_id: str
title: str
source_name: str | None = None
source_type: str | None = None

32
src/everos/infra/persistence/markdown/mds/knowledge_topic.py Normal file
View File

@ -0,0 +1,32 @@
"""Frontmatter schema for ``knowledge/{category}/{doc_title}/<n>_<name>.md``."""
from __future__ import annotations
from typing import Literal
from everos.core.persistence.markdown import (
BaseFrontmatter,
KnowledgeScopedMixin,
KnowledgeTopicPathMixin,
)
class KnowledgeTopicFrontmatter(
KnowledgeTopicPathMixin,
KnowledgeScopedMixin,
BaseFrontmatter,
):
"""L2 topic-node frontmatter. Body = topic content full text."""
type: Literal["knowledge_topic"] = "knowledge_topic"
node_id: str
doc_id: str
category_id: str
topic_index: int
topic_name: str
topic_path: str
summary: str
depth: int
parent_node_id: str | None = None
children_node_ids: list[str] = []
content_labels: list[str] = []

4
src/everos/infra/persistence/markdown/readers/__init__.py
View File

@ -37,6 +37,8 @@ from .base import BaseDailyReader as BaseDailyReader
from .episode_reader import EpisodeReader as EpisodeReader
from .foresight_reader import ForesightReader as ForesightReader
from .profile_reader import ProfileReader as ProfileReader
from .taxonomy_reader import ensure_taxonomy as ensure_taxonomy
from .taxonomy_reader import parse_taxonomy as parse_taxonomy
__all__ = [
"AgentCaseReader",
@ -46,4 +48,6 @@ __all__ = [
"EpisodeReader",
"ForesightReader",
"ProfileReader",
"ensure_taxonomy",
"parse_taxonomy",
]

217
src/everos/infra/persistence/markdown/readers/taxonomy_reader.py Normal file
View File

@ -0,0 +1,217 @@
"""Read and auto-generate ``.taxonomy.md`` — the knowledge category taxonomy.
The taxonomy file lives at ``knowledge_dir/.taxonomy.md``. It uses YAML
frontmatter with a ``categories`` list, each entry having ``id`` and
``description`` — matching ``everalgo.types.CategorySpec`` exactly.
``ensure_taxonomy(knowledge_dir)`` creates the file from
``DEFAULT_TAXONOMY`` if it does not exist.
"""
from __future__ import annotations
from pathlib import Path
from typing import Any
import anyio
import yaml
from everalgo.types import CategorySpec
TAXONOMY_FILENAME = ".taxonomy.md"
DEFAULT_TAXONOMY: list[dict[str, str]] = [
{
"id": "Technology",
"description": (
"Computer science, software engineering, AI/ML, semiconductors, "
"cloud computing, cybersecurity, and general IT infrastructure."
),
},
{
"id": "Science",
"description": (
"Natural sciences including physics, chemistry, biology, earth sciences, "
"astronomy, and interdisciplinary research."
),
},
{
"id": "Medical",
"description": (
"Clinical medicine, disease diagnosis, drug specifications, "
"medical devices, public health, nursing, and healthcare administration."
),
},
{
"id": "Finance",
"description": (
"Securities, banking, insurance, accounting, taxation, corporate finance, "
"macroeconomics, and fintech."
),
},
{
"id": "Legal",
"description": (
"Laws, regulations, contracts, compliance, intellectual property, "
"litigation, and legal procedures."
),
},
{
"id": "Education",
"description": (
"Teaching methods, curriculum design, academic research, "
"student assessment, e-learning, and educational policy."
),
},
{
"id": "Business",
"description": (
"Corporate strategy, marketing, sales, operations management, "
"supply chain, HR, and entrepreneurship."
),
},
{
"id": "Engineering",
"description": (
"Mechanical, civil, electrical, chemical, and industrial engineering — "
"design, manufacturing, and construction."
),
},
{
"id": "Arts",
"description": (
"Visual arts, music, literature, film, theater, design, "
"and cultural studies."
),
},
{
"id": "Sports",
"description": (
"Athletic training, sports events, fitness, sports science, "
"and sports industry management."
),
},
{
"id": "Travel",
"description": (
"Tourism, hospitality, travel guides, destination reviews, "
"and transportation logistics."
),
},
{
"id": "Food",
"description": (
"Culinary arts, nutrition science, food safety, restaurant management, "
"and food industry."
),
},
{
"id": "Environment",
"description": (
"Climate change, ecology, pollution control, renewable energy, "
"conservation, and sustainability."
),
},
{
"id": "Politics",
"description": (
"Government policy, international relations, elections, "
"public administration, and geopolitics."
),
},
{
"id": "History",
"description": (
"Historical events, civilizations, archaeology, historical analysis, "
"and historiography."
),
},
{
"id": "Psychology",
"description": (
"Cognitive science, behavioral psychology, clinical psychology, "
"mental health, and neuroscience."
),
},
{
"id": "Agriculture",
"description": (
"Farming techniques, crop science, animal husbandry, agribusiness, "
"and food production systems."
),
},
{
"id": "RealEstate",
"description": (
"Property development, real estate investment, urban planning, "
"architecture, and housing policy."
),
},
{
"id": "Media",
"description": (
"Journalism, broadcasting, social media, public relations, "
"advertising, and communications."
),
},
{
"id": "Others",
"description": (
"Documents that do not clearly fit any of the above categories."
),
},
]
async def parse_taxonomy(path: anyio.Path | Path) -> list[CategorySpec]:
"""Parse ``.taxonomy.md`` and return the category list.
Args:
path: Path to the ``.taxonomy.md`` file.
Returns:
Parsed category list, or an empty list when the file does not
exist or has no categories.
"""
apath = anyio.Path(path) if not isinstance(path, anyio.Path) else path
if not await apath.exists():
return []
text = await apath.read_text(encoding="utf-8")
parts = text.split("---", 2)
if len(parts) < 3:
return []
data: dict[str, Any] = yaml.safe_load(parts[1]) or {}
raw_categories = data.get("categories") or []
return [
CategorySpec(id=c["id"], description=c.get("description", ""))
for c in raw_categories
if isinstance(c, dict) and "id" in c
]
async def ensure_taxonomy(knowledge_dir: anyio.Path | Path) -> Path:
"""Create ``.taxonomy.md`` from defaults if it does not exist.
Args:
knowledge_dir: Knowledge directory where ``.taxonomy.md`` lives.
Returns:
Path to the taxonomy file (whether created or pre-existing).
"""
adir = (
knowledge_dir
if isinstance(knowledge_dir, anyio.Path)
else anyio.Path(knowledge_dir)
)
p = adir / TAXONOMY_FILENAME
if await p.exists():
return Path(p)
await adir.mkdir(parents=True, exist_ok=True)
frontmatter = yaml.dump(
{"kind": "knowledge_taxonomy", "categories": DEFAULT_TAXONOMY},
default_flow_style=False,
allow_unicode=True,
sort_keys=False,
)
await p.write_text(f"---\n{frontmatter}---\n", encoding="utf-8")
return Path(p)

2
src/everos/infra/persistence/markdown/writers/__init__.py
View File

@ -30,6 +30,7 @@ from .atomic_fact_writer import AtomicFactWriter as AtomicFactWriter
from .base import BaseDailyWriter as BaseDailyWriter
from .episode_writer import EpisodeWriter as EpisodeWriter
from .foresight_writer import ForesightWriter as ForesightWriter
from .knowledge_writer import KnowledgeWriter as KnowledgeWriter
from .profile_writer import ProfileWriter as ProfileWriter
__all__ = [
@ -39,5 +40,6 @@ __all__ = [
"BaseDailyWriter",
"EpisodeWriter",
"ForesightWriter",
"KnowledgeWriter",
"ProfileWriter",
]

12
src/everos/infra/persistence/markdown/writers/base.py
View File

@ -234,6 +234,18 @@ class BaseDailyWriter:
)
return eid
async def patch_frontmatter(self, path: Path, updates: Mapping[str, Any]) -> None:
"""Merge ``updates`` into the frontmatter of an existing daily-log file.
Delegates to the underlying :class:`MarkdownWriter` so that callers
do not need to reach through the private ``_writer`` attribute.
Args:
path: Target markdown file (must exist).
updates: Mapping of frontmatter keys to merge.
"""
await self._writer.patch_frontmatter(path, updates)
# ── Hooks (subclass override) ─────────────────────────────────────────
async def _current_count(self, path: Path) -> int:

246
src/everos/infra/persistence/markdown/writers/knowledge_writer.py Normal file
View File

@ -0,0 +1,246 @@
"""KnowledgeWriter — write knowledge document + topic markdown files.
Knowledge storage uses a **directory per document** layout::
knowledge/{category_id}/{title_dirname}/index.md ← document root
knowledge/{category_id}/{title_dirname}/1_topic_slug.md ← topic node
knowledge/{category_id}/{title_dirname}/2_topic_slug.md ← topic node
Each call to :meth:`write` replaces the entire document directory
(delete-then-recreate) so that stale topic files from a prior extraction
do not linger.
The writer is intentionally **static** — it takes a ``knowledge_dir``
path rather than binding to :class:`MemoryRoot`. The service layer
resolves ``knowledge_dir`` from ``MemoryRoot.knowledge_dir(app, project)``
and passes it in. This keeps the writer decoupled from the root-resolution
logic and easier to test.
"""
from __future__ import annotations
import re
import shutil
from pathlib import Path
import anyio
import yaml
from everalgo.types import KnowledgeMemory
from everos.core.observability.logging import get_logger
logger = get_logger(__name__)
_MAX_DIRNAME_LEN = 50
_SAFE_CHARS = re.compile(r"[^\w\-.]", re.UNICODE)
# ── Writer ────────────────────────────────────────────────────────────────
class KnowledgeWriter:
"""Convert ``KnowledgeMemory`` list to markdown files."""
@staticmethod
async def write(
memories: list[KnowledgeMemory],
knowledge_dir: Path,
*,
source_name: str | None = None,
source_type: str | None = None,
) -> Path:
"""Write md files and return the document directory path.
Args:
memories: Flat list of nodes produced by everalgo extraction.
Must contain exactly one root node (``topic_index=0``).
knowledge_dir: Base ``knowledge/`` directory (from
``MemoryRoot.knowledge_dir``).
source_name: Optional provenance label (e.g. URL, filename).
source_type: Optional provenance type (e.g. ``"url"``,
``"file"``).
Returns:
Absolute path of the written document directory.
Raises:
ValueError: If *memories* is empty or has no root node.
"""
root_node, topic_nodes = _split_root_and_topics(memories)
doc_dir = _resolve_doc_dir(knowledge_dir, root_node)
# Overwrite: remove existing directory, then recreate.
await _remove_dir(doc_dir)
await anyio.Path(doc_dir).mkdir(parents=True, exist_ok=True)
await _write_index(doc_dir, root_node, source_name, source_type)
for node in topic_nodes:
await _write_topic(doc_dir, node, root_node.doc_id)
logger.info(
"knowledge document written",
doc_id=root_node.doc_id,
category_id=root_node.category_id or "Others",
topic_count=len(topic_nodes),
)
return doc_dir
# ── Internals ─────────────────────────────────────────────────────────────
def _split_root_and_topics(
memories: list[KnowledgeMemory],
) -> tuple[KnowledgeMemory, list[KnowledgeMemory]]:
"""Separate the root node from topic nodes.
Raises:
ValueError: If *memories* is empty or contains no root node.
"""
if not memories:
raise ValueError("memories must not be empty")
root: KnowledgeMemory | None = None
topics: list[KnowledgeMemory] = []
for m in memories:
if m.topic_index == 0:
root = m
else:
topics.append(m)
if root is None:
raise ValueError("memories must contain a root node (topic_index=0)")
return root, topics
def _sanitize_dirname(raw: str, fallback: str) -> str:
"""Produce a safe directory/file name segment.
* Replace spaces with underscores.
* Strip characters outside ``[a-zA-Z0-9_\\-.]``.
* Truncate to 50 characters.
* Fall back to *fallback* if the result is empty.
"""
slug = raw.replace(" ", "_")
slug = _SAFE_CHARS.sub("", slug)
slug = slug[:_MAX_DIRNAME_LEN]
return slug if slug else fallback
def _resolve_doc_dir(knowledge_dir: Path, root: KnowledgeMemory) -> Path:
"""Build the document directory path from category, title, and doc_id."""
category = _sanitize_dirname(
root.category_id if root.category_id else "Others", "Others"
)
title_slug = _sanitize_dirname(root.topic, "doc")
dir_name = f"{title_slug}_{root.doc_id}"
return knowledge_dir / category / dir_name
def _build_index_frontmatter(
root: KnowledgeMemory,
source_name: str | None,
source_type: str | None,
) -> dict[str, object]:
"""Build YAML frontmatter dict for the document index file."""
category = root.category_id if root.category_id else "Others"
fm: dict[str, object] = {
"type": "knowledge_document",
"id": root.doc_id,
"doc_id": root.doc_id,
"category_id": category,
"title": root.topic,
"schema_version": 1,
}
if source_name is not None:
fm["source_name"] = source_name
if source_type is not None:
fm["source_type"] = source_type
return fm
def _build_topic_frontmatter(
node: KnowledgeMemory,
doc_id: str,
) -> dict[str, object]:
"""Build YAML frontmatter dict for a single topic file."""
category = node.category_id if node.category_id else "Others"
node_id = f"{doc_id}_{node.topic_index}"
parent_node_id = None if node.depth <= 1 else f"{doc_id}_{node.parent_index}"
children_node_ids = [f"{doc_id}_{ci}" for ci in node.children_index]
return {
"type": "knowledge_topic",
"id": node_id,
"node_id": node_id,
"doc_id": doc_id,
"category_id": category,
"topic_index": node.topic_index,
"topic_name": node.topic,
"topic_path": node.topic_path,
"summary": node.summary,
"depth": node.depth,
"parent_node_id": parent_node_id,
"children_node_ids": children_node_ids,
"content_labels": node.content_labels,
"schema_version": 1,
}
def _dump_yaml_frontmatter(meta: dict[str, object]) -> str:
"""Render a YAML frontmatter block with ``---`` delimiters."""
yaml_block = yaml.safe_dump(
meta,
sort_keys=False,
allow_unicode=True,
default_flow_style=False,
)
return f"---\n{yaml_block}---\n"
def _ensure_trailing_newline(text: str) -> str:
"""Append a newline if *text* does not already end with one."""
if not text:
return ""
return text if text.endswith("\n") else text + "\n"
async def _write_file(path: Path, content: str) -> None:
"""Write content to *path* via anyio (async)."""
await anyio.Path(path.parent).mkdir(parents=True, exist_ok=True)
await anyio.Path(path).write_text(content, encoding="utf-8")
async def _write_index(
doc_dir: Path,
root: KnowledgeMemory,
source_name: str | None,
source_type: str | None,
) -> None:
"""Write index.md with frontmatter and summary body."""
fm = _build_index_frontmatter(root, source_name, source_type)
body = _ensure_trailing_newline(root.summary)
content = _dump_yaml_frontmatter(fm) + body
await _write_file(doc_dir / "index.md", content)
async def _write_topic(
doc_dir: Path,
node: KnowledgeMemory,
doc_id: str,
) -> None:
"""Write a numbered topic md file with frontmatter and content body."""
slug = _sanitize_dirname(node.topic, f"topic_{node.topic_index}")
filename = f"{node.topic_index}_{slug}.md"
fm = _build_topic_frontmatter(node, doc_id)
body = _ensure_trailing_newline(node.content)
content = _dump_yaml_frontmatter(fm) + body
await _write_file(doc_dir / filename, content)
async def _remove_dir(path: Path) -> None:
"""Remove directory tree if it exists (sync shutil offloaded)."""
if await anyio.Path(path).is_dir():
await anyio.to_thread.run_sync(shutil.rmtree, path)

20
src/everos/infra/persistence/sqlite/__init__.py
View File

@ -17,7 +17,9 @@ External usage::
# ``infra.persistence.sqlite.**`` (sub-packages) are forbidden
# to ``service`` / ``memory`` / ``entrypoints`` by import-linter.
UnprocessedBuffer, Memcell, ConversationStatus,
KnowledgeDocumentRow, KnowledgeTopicRow,
unprocessed_buffer_repo, memcell_repo, conversation_status_repo,
knowledge_document_repo, knowledge_topic_sqlite_repo,
)
The :class:`SqliteLifespanProvider` runs ``SQLModel.metadata.create_all``
@ -29,12 +31,18 @@ not need to manage either.
# ``SQLModel.metadata`` so ``SqliteLifespanProvider.startup`` can
# ``create_all`` without callers having to import each model module.
from . import tables as tables # noqa: F401
from .repos import DocumentListPage as DocumentListPage
from .repos import DocumentUpsertPayload as DocumentUpsertPayload
from .repos import QueueSummary as QueueSummary
from .repos import TopicUpsertPayload as TopicUpsertPayload
from .repos import cluster_repo as cluster_repo
from .repos import conversation_status_repo as conversation_status_repo
from .repos import knowledge_document_repo as knowledge_document_repo
from .repos import knowledge_topic_sqlite_repo as knowledge_topic_sqlite_repo
from .repos import md_change_state_repo as md_change_state_repo
from .repos import memcell_repo as memcell_repo
from .repos import mint_cluster_id as mint_cluster_id
from .repos import reflection_report_repo as reflection_report_repo
from .repos import unprocessed_buffer_repo as unprocessed_buffer_repo
from .sqlite_manager import dispose_engine as dispose_engine
from .sqlite_manager import get_engine as get_engine
@ -42,25 +50,37 @@ from .sqlite_manager import get_session_factory as get_session_factory
from .tables import Cluster as Cluster
from .tables import ClusterMember as ClusterMember
from .tables import ConversationStatus as ConversationStatus
from .tables import KnowledgeDocumentRow as KnowledgeDocumentRow
from .tables import KnowledgeTopicRow as KnowledgeTopicRow
from .tables import MdChangeState as MdChangeState
from .tables import Memcell as Memcell
from .tables import ReflectionReport as ReflectionReport
from .tables import UnprocessedBuffer as UnprocessedBuffer
__all__ = [
"Cluster",
"ClusterMember",
"ConversationStatus",
"DocumentListPage",
"DocumentUpsertPayload",
"KnowledgeDocumentRow",
"KnowledgeTopicRow",
"MdChangeState",
"Memcell",
"QueueSummary",
"ReflectionReport",
"TopicUpsertPayload",
"UnprocessedBuffer",
"cluster_repo",
"conversation_status_repo",
"dispose_engine",
"get_engine",
"get_session_factory",
"knowledge_document_repo",
"knowledge_topic_sqlite_repo",
"md_change_state_repo",
"memcell_repo",
"mint_cluster_id",
"reflection_report_repo",
"unprocessed_buffer_repo",
]

Some files were not shown because too many files have changed in this diff Show More