diff --git a/CHANGELOG.md b/CHANGELOG.md index 68355ea..036d85a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -37,9 +37,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **Search: hierarchical fact eviction** (Layer-4) with `min_score` floor — low-confidence atomic facts are evicted before fusion, improving precision. -- **Search degradation guidance** — when embedding or rerank providers fail, - the response now includes a `degradation` field explaining which - capability is unavailable and how results are affected. +- **Knowledge search degradation guidance** — when the embedding or rerank + provider fails at call time, the knowledge search route enriches the + error message with actionable guidance (e.g. retry with `method=keyword`, + which needs no embedding) before returning `503`. - **Knowledge topic recaller** — dual-column BM25 recall for knowledge topics, integrated into the search manager alongside existing recall types. @@ -131,8 +132,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Add a multimodal usage guide and correct the multimodal error semantics after end-to-end verification. -- Rename the algorithm library from the previous package name to `everalgo` - across docs and code comments (no code identifiers changed). +- Rename the algorithm library to `everalgo` across docs and + code comments (no code identifiers changed). - Fix accuracy drift found in an adversarial doc audit; reflect the `everalgo` packages being published and the v1.0.0 stable status. diff --git a/QUICKSTART.md b/QUICKSTART.md index 96c291b..dfd19f1 100644 --- a/QUICKSTART.md +++ b/QUICKSTART.md @@ -29,33 +29,36 @@ pip install everos ## 2. Configure -Generate a starter `.env` and drop in your two keys: +Generate the starter config and drop in your two keys: ```bash -everos init # writes ./.env (use --xdg for ~/.config/everos/.env) -# or, from a source checkout: -cp .env.example .env - -# Edit .env and fill four API key slots (only two distinct keys needed): -# EVEROS_LLM__API_KEY (OpenRouter — chat LLM) -# EVEROS_MULTIMODAL__API_KEY (OpenRouter — same key works) -# EVEROS_EMBEDDING__API_KEY (DeepInfra) -# EVEROS_RERANK__API_KEY (DeepInfra — same key works) +everos init # writes ~/.everos/everos.toml + ome.toml (use --root to relocate) +# Edit ~/.everos/everos.toml and fill four api_key slots (only two distinct keys needed): +# [llm] api_key (OpenRouter — chat LLM) +# [multimodal] api_key (OpenRouter — same key works) +# [embedding] api_key (DeepInfra) +# [rerank] api_key (DeepInfra — same key works) ``` -`everos init` reads the template bundled inside the wheel and writes it -with `0600` permissions (only your user can read the API keys). +`everos init` generates two files: `everos.toml` (provider settings) +and `ome.toml` (offline memory engine strategy config, hot-reloaded). +Because `everos.toml` holds API keys, consider restricting access +after editing: `chmod 600 ~/.everos/everos.toml`. -The shipped template already points LLM + multimodal → OpenRouter -(`openai/gpt-4.1-mini` and `google/gemini-3-flash-preview`) and -embedding + rerank → DeepInfra (`Qwen/Qwen3-Embedding-4B` and -`Qwen/Qwen3-Reranker-4B`). To use a different OpenAI-compatible -endpoint, override the matching `*__BASE_URL` env var. +The shipped template sets model defaults for `[llm]` (`gpt-4.1-mini`) and +`[multimodal]` (`google/gemini-3-flash-preview`); `[embedding]` and +`[rerank]` ship no model default — set `model` + `base_url` for those two +sections yourself (e.g. DeepInfra's `Qwen/Qwen3-Embedding-4B` / +`Qwen/Qwen3-Reranker-4B`). To use a different OpenAI-compatible endpoint +for any provider, set the matching `base_url` field. -> **Where to store `.env`** — `everos server start` searches in order: -> `--env-file ` → `./.env` (cwd) → `${XDG_CONFIG_HOME:-~/.config}/everos/.env` → -> `~/.everos/.env`. The first existing file wins. Use `everos init --xdg` to write -> the XDG location so the same config works from any cwd. +> **Where config lives** — `everos init` writes into the memory root +> (`~/.everos` by default; relocate with `everos init --root ` and +> start the server with the matching `everos server start --root `). +> `everos server start` reads `/everos.toml` and exits with an error +> if it is missing. Any setting can also be overridden by an `EVEROS_*` +> environment variable (e.g. `EVEROS_LLM__API_KEY`) — handy for containers +> and CI. ## 3. Start the server @@ -108,7 +111,7 @@ curl -X POST http://127.0.0.1:8000/api/v1/memory/add \ }" ``` -Typical response: +Response: ```json { @@ -122,9 +125,7 @@ Typical response: `status: "accumulated"` means the three messages are in the session buffer, but the boundary detector hasn't decided to extract a memory -cell yet. If you see `status: "extracted"` instead, EverOS already -carved out a memory cell; you can still continue. For a deterministic -quick demo we'll force the boundary in the next step. +cell yet. For a quick demo we'll force it. ## 5. Force boundary extraction @@ -146,7 +147,7 @@ Response (this takes a few seconds — one LLM call for extraction): ``` `status: "extracted"` means at least one memory cell was carved out and -written to disk; the local cascade process then indexes the Markdown. +written to disk + indexed. > `/flush` is **OSS-only**. The cloud edition decides boundary timing > server-side and does not expose this endpoint. @@ -189,8 +190,7 @@ Response (trimmed): ], "profiles": [], "agent_cases": [], - "agent_skills": [], - "unprocessed_messages": [] + "agent_skills": [] } } ``` @@ -198,13 +198,8 @@ Response (trimmed): The hybrid retrieval (BM25 + vector + scalar) returns the episode that contains the climbing fact, with the matching atomic fact nested under it. Other response arrays (`profiles` / `agent_cases` / -`agent_skills` / `unprocessed_messages`) are always present for -client-side symmetry, populated only when the requested kind matches -or when `filters.session_id` asks for in-flight buffer rows. - -If the result is empty on the first try, wait a moment and retry; the -Markdown write is synchronous, while the local index catches up in the -background. +`agent_skills`) are always present for client-side symmetry, populated +only when the requested kind matches. ## 7. Your memory is just Markdown @@ -216,15 +211,19 @@ $ tree ~/.everos -L 5 -a ~/.everos ├── default_app/ ← app_id ("default" → "default_app") │ └── default_project/ ← project_id ("default" → "default_project") -│ └── users/ -│ └── alice/ ← user_id (mirror dir: agents//) -│ ├── episodes/ -│ │ └── episode-2026-05-28.md -│ ├── .atomic_facts/ ← hidden (dot-prefix) -│ │ └── atomic_fact-2026-05-28.md -│ ├── .foresights/ -│ │ └── foresight-2026-05-28.md -│ └── user.md ← profile +│ ├── users/ +│ │ └── alice/ ← user_id (mirror dir: agents//) +│ │ ├── episodes/ +│ │ │ └── episode-2026-05-28.md +│ │ ├── .atomic_facts/ ← hidden (dot-prefix) +│ │ │ └── atomic_fact-2026-05-28.md +│ │ ├── .foresights/ +│ │ │ └── foresight-2026-05-28.md +│ │ └── user.md ← profile +│ └── knowledge/ ← shared knowledge base (v1.1+) +│ └── .taxonomy.md +├── everos.toml ← provider config (API keys) +├── ome.toml ← strategy config (hot-reloaded) ├── .index/ ← derived indexes (rebuildable from md) │ ├── sqlite/system.db │ └── lancedb/*.lance/ @@ -296,6 +295,14 @@ LLM → metrics) before exiting. call them from your agent loop. - **App + project scope** — set `app_id` / `project_id` to anything other than `"default"` to partition memory spaces inside one server. +- **Knowledge base** — upload documents (PDF / HTML / DOCX) via + `/api/v1/knowledge/documents` and search them with hybrid retrieval + at `/api/v1/knowledge/search`. Ships with a 20-category default + taxonomy. See [docs/knowledge.md](docs/knowledge.md). +- **Reflection** — offline memory self-improvement that consolidates + related episodes. Disabled by default; enable in `ome.toml` + (`[strategies.reflect_episodes] enabled = true`). Changes are + hot-reloaded, no server restart needed. - **Multi-modal messages** — `messages[].content` accepts a list of typed `ContentItem`s (`text` / `image` / `audio` / `doc` / `pdf` / `html` / `email`) for non-text input. Install the optional extra @@ -304,7 +311,8 @@ LLM → metrics) before exiting. (`doc` / `docx` / `xls` / `ppt` / `…`) additionally need **LibreOffice** on the host (`brew install --cask libreoffice` / `apt-get install libreoffice`) — without it those uploads return - HTTP 415; PDF / image / audio / HTML still work. + HTTP 503 (`CAPABILITY_UNAVAILABLE`); PDF / image / audio / HTML + still work. - **Filter DSL and search modes** — `/search` supports a filter DSL (`AND` / `OR` / scalar predicates) and four methods (`HYBRID` / `KEYWORD` / `VECTOR` / `AGENTIC`). The OpenAPI docs UI is served at diff --git a/README.md b/README.md index c63c400..211acbc 100644 --- a/README.md +++ b/README.md @@ -1,262 +1,65 @@ -
+# EverOS -![EverOS banner](https://github.com/user-attachments/assets/8e217d39-5d15-4c6c-9b54-3e83add4e0f2) +> Local-first markdown memory framework for AI agents and user chats — lightweight, dev-friendly, small-team. -

- X - HuggingFace - Discord - WeChat -

+[![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE) +[![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/) -[Website](https://evermind.ai) · [Documentation](https://docs.evermind.ai) · [Blog](https://evermind.ai/blogs) · [中文](README.zh-CN.md) +--- -
+## What is EverOS +EverOS is an open-source Python framework that turns conversations, agent trajectories, and files into **structured, retrievable, evolving long-term memory** for AI agents and user chats. Designed for **lightweight local deployments** (small teams, individual developers), with three core principles: -
+1. **Markdown as Source of Truth** — All memory persists as plain `.md` files. Open, edit, grep, version with Git, view in Obsidian. No black-box database lock-in. +2. **Lightweight three-piece storage** — `Markdown` files (truth) + `SQLite` (state/queue) + `LanceDB` (vector + BM25 + scalar). No MongoDB / Elasticsearch / Milvus / Redis / Kafka required. +3. **EverAlgo as pure algorithm library** — Memory extraction algorithms are decoupled into a separate library; this project orchestrates and persists. -
- Table of Contents +## Architecture at a glance -
- -- [Why Ever OS](#why-ever-os) -- [Quick Start](#quick-start) -- [Use Cases](#use-cases) -- [Documentation](#documentation) -- [Star Us](#star-us) -- [EverMind Ecosystems](#evermind-ecosystems) -- [Contributing](#contributing) - -
- -
- - -## Why Ever OS - -EverOS is a Python library and local-first memory runtime for agents and -makers. It gives one portable memory layer across coding assistants, apps, -devices, and workflows from day one. It stores conversations, files, and agent -trajectories as readable Markdown, then syncs local SQLite and LanceDB indexes -for fast retrieval and self-evolving reuse. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
TitleEverOSOther Agent Memory Libraries
Markdown source of truth✅ Canonical .md files that are readable, editable, diffable, and Git-versioned❌ Usually API, vector, graph, dashboard, or database state
Direct file editing✅ Edit .md files; cascade watcher syncs❌ Usually SDK, API, dashboard, or backend update paths
Local three-part stack✅ Markdown + SQLite + LanceDB; no MongoDB, Elasticsearch, or Redis required❌ Often depends on managed services, vector DBs, graph DBs, or server stacks
User + agent tracks✅ User episodes/profile and agent cases/skills are separate first-class surfaces❌ Usually centered on chat history, profiles, entities, facts, or retrieval records
Orthogonal retrieval✅ Search by user_id, agent_id, app_id, project_id, and session_id❌ Usually app, namespace, tenant, thread, or graph scoped
Knowledge Wiki✅ Editable, source-backed Markdown knowledge pages with taxonomy, CRUD APIs, and topic search❌ Usually separate from memory, trapped in a dashboard, or not tied back to source files
Reflection✅ Offline memory evolution that merges episode clusters and refines profiles and skills between sessions❌ Usually retrieval-only memory with little background consolidation or long-horizon improvement
- -
- -## Quick Start - -> Goal: play with the memory visualizer first, then start EverOS, write one -> real memory, and search it back. - -### 0. Prerequisites - -- Python 3.12+ -- No API keys are needed for `everos demo`. -- To run the real server-backed memory flow, create two provider keys before - `everos init`: - -| Capability | Provider | Used for | Fill these `.env` slots | -| --- | --- | --- | --- | -| Chat + multimodal | [OpenRouter](https://openrouter.ai/) | `LLM` / `MULTIMODAL` | `EVEROS_LLM__API_KEY`, `EVEROS_MULTIMODAL__API_KEY` | -| Embedding + rerank | [DeepInfra](https://deepinfra.com/) | `EMBEDDING` / `RERANK` | `EVEROS_EMBEDDING__API_KEY`, `EVEROS_RERANK__API_KEY` | - -You can use other OpenAI-compatible providers by changing the matching -`*__BASE_URL` fields in `.env`. - -### 1. Install - -```bash -uv pip install everos -# or: pip install everos +``` +┌───────────────────────────────────────────────┐ +│ entrypoints/ (CLI + HTTP API) │ presentation +├───────────────────────────────────────────────┤ +│ service/ (use cases: memorize/retrieve) │ application +├───────────────────────────────────────────────┤ +│ memory/ (extract + search + cascade) │ domain +├───────────────────────────────────────────────┤ +│ infra/ (markdown / sqlite / lancedb) │ infrastructure +└───────────────────────────────────────────────┘ + ↑ ↑ + component/ core/ + (LLM/Embedding) (observability/lifespan) ``` -### 2. Play With The Demo +DDD 5 layers, single-direction dependency. See [docs/architecture.md](docs/architecture.md). -Run this before configuring API keys or starting the server: +## Quick start + +### Install as a package ```bash -everos demo -``` +uv pip install everos # or: pip install everos -The command asks for one memory and one recall question, then opens a -full-screen terminal UI. This is an educational visualizer: it is hardcoded, -local to the CLI, and does not connect to the EverOS server. Its job is to make -the memory lifecycle visible: conversation -> memory sphere -> recall -> source -proof -> confetti. See [docs/everos-demo.md](docs/everos-demo.md) for the demo -scope and TUI source layout. +# Generate starter config (model defaults bundled inside the wheel) +everos init # writes ~/.everos/everos.toml + ome.toml (use --root to relocate) +# Edit ~/.everos/everos.toml and fill the api_key fields (see comments inside). -The sphere moves through ingest, extraction, indexing, recall, source reveal, -and a confetti burst after the first memory lands. Press `r` to replay and `q` -to quit. - -

- Animated EverOS demo preview showing the memory sphere moving through recall and confetti states -

- -For the looping showroom view used in README media, run: - -```bash -everos demo --cinematic -``` - -If your shell is not interactive, or you want a copyable preview, use: - -```bash -everos demo --plain -``` - -### 3. Configure - -Generate a starter `.env` file, then fill the four API key slots shown in the -generated comments. With the default setup, paste your OpenRouter key into the -`LLM` / `MULTIMODAL` slots and your DeepInfra key into the `EMBEDDING` / -`RERANK` slots. - -```bash -everos init -# or, from a source checkout: -cp .env.example .env -``` - -`everos init` writes `./.env` by default. Use `everos init --xdg` to -write `${XDG_CONFIG_HOME:-~/.config}/everos/.env` instead. - -### 4. Start EverOS - -```bash +everos --help everos server start ``` -Keep the server running, then open a second terminal and check it: +`everos init` writes two TOML files into the memory root (`~/.everos` by +default; relocate with `--root`): `everos.toml` (app settings + provider +credentials) and `ome.toml` (offline-engine schedules). `everos server start` +reads `/everos.toml` and exits with an error if it is missing. Any +setting can also be overridden by an `EVEROS_*` environment variable +(e.g. `EVEROS_LLM__API_KEY`). The endpoint stack is OpenAI-protocol +compatible (OpenAI / OpenRouter / vLLM / Ollama / DeepInfra …) — set the +`base_url` field in each provider section of `everos.toml` to point at any +of them. -```bash -curl http://127.0.0.1:8000/health -``` - -Expected response: - -```json -{"status":"ok"} -``` - -`everos server start` searches for `.env` in this order: `--env-file ` → -`./.env` (cwd) → `${XDG_CONFIG_HOME:-~/.config}/everos/.env` → `~/.everos/.env`. -The endpoint stack is OpenAI-protocol compatible (OpenAI / OpenRouter / vLLM / -Ollama / DeepInfra) - override `*__BASE_URL` in the generated `.env` to point -at any of them. - -Now make the demo real. In the second terminal, run: - -```bash -everos demo --live -``` - -Live demo mode connects to the running server and performs the real -`/health` -> `/api/v1/memory/add` -> `/api/v1/memory/flush` -> -`/api/v1/memory/search` flow before opening the same memory sphere UI. Use -`--server-url ` if your server is not on `http://127.0.0.1:8000`. - -### 5. Try Your First Memory - -Add a tiny conversation: - -```bash -TS=$(($(date +%s)*1000)) - -curl -X POST http://127.0.0.1:8000/api/v1/memory/add \ - -H 'Content-Type: application/json' \ - -d "{ - \"session_id\": \"demo-001\", - \"app_id\": \"default\", - \"project_id\": \"default\", - \"messages\": [ - {\"sender_id\": \"alice\", \"role\": \"user\", \"timestamp\": $TS, \"content\": \"I love climbing in Yosemite every spring.\"}, - {\"sender_id\": \"alice\", \"role\": \"user\", \"timestamp\": $((TS+10000)), \"content\": \"My favorite coffee shop is Blue Bottle in SOMA.\"} - ] - }" -``` - -Force extraction for the local demo: - -```bash -curl -X POST http://127.0.0.1:8000/api/v1/memory/flush \ - -H 'Content-Type: application/json' \ - -d '{"session_id":"demo-001","app_id":"default","project_id":"default"}' -``` - -Search it back: - -```bash -curl -X POST http://127.0.0.1:8000/api/v1/memory/search \ - -H 'Content-Type: application/json' \ - -d '{ - "user_id": "alice", - "app_id": "default", - "project_id": "default", - "query": "Where do I like to climb?", - "top_k": 5 - }' -``` - -You should see the Yosemite memory in the response. If the result is empty on -the first try, wait a moment and retry; Markdown is written synchronously, while -the local index catches up in the background. - -> [!TIP] -> **First memory unlocked.** -> You just gave EverOS a fact, flushed it into durable Markdown-backed memory, -> and searched it back through the local index. That is the core loop. -> Want to see the source of truth? Open `~/.everos` and inspect the generated -> Markdown files. - -For annotated responses and the Markdown files EverOS creates, see -[QUICKSTART.md](QUICKSTART.md). - -### Optional: Ingest Multimodal Files +#### Multi-modal (optional) To ingest non-text content (image / pdf / audio / office documents) through `/api/v1/memory/add` `content` items, install the optional @@ -268,14 +71,15 @@ uv pip install 'everos[multimodal]' # or: pip install 'everos[multimodal]' This pulls in `everalgo-parser` (with the `[svg]` bundle for SVG support via cairosvg) and wires up the multimodal LLM client -(`EVEROS_MULTIMODAL__*` fields in `.env`, defaults to -`google/gemini-3-flash-preview` via OpenRouter). +(the `[multimodal]` section in `everos.toml`, defaults to +`google/gemini-3-flash-preview`). **Office document support requires LibreOffice as a system dependency.** The parser shells out to `soffice` (LibreOffice's headless renderer) to convert `.doc` / `.docx` / `.ppt` / `.pptx` / `.xls` / `.xlsx` to PDF before feeding the result into the multimodal LLM. Without LibreOffice, -office uploads return HTTP 415 with a clear error message; PDF / image +office uploads return HTTP 503 (`CAPABILITY_UNAVAILABLE`) with a clear +error message; PDF / image / audio / HTML / email parsing is unaffected. Install on the host before serving office documents: @@ -285,479 +89,108 @@ brew install --cask libreoffice # macOS sudo apt-get install -y libreoffice # Debian / Ubuntu ``` -### For Contributors +For the full multimodal contract (supported modalities, `uri` vs +`base64`, config, error semantics, end-to-end curl examples), see +[docs/multimodal.md](docs/multimodal.md). + +For a step-by-step walkthrough (add a conversation → flush → search → +read the markdown), see [QUICKSTART.md](QUICKSTART.md). + +### Develop locally ```bash git clone https://github.com/EverMind-AI/EverOS.git cd EverOS uv sync # creates ./.venv and installs deps -source .venv/bin/activate # or prefix commands with `uv run` -everos demo --plain # try the local educational demo; no API keys needed -everos init # paste OpenRouter + DeepInfra keys into .env +source .venv/bin/activate # — or skip activation and prefix every command with `uv run` +everos init # fill in the [llm] api_key in the generated everos.toml everos --help make test ``` -
-
- -[![](https://img.shields.io/badge/-Back_to_top-gray?style=flat-square)](#readme-top) - -
- -## Use Cases - -Now that you have had your first successful EverOS moment, explore what people -are building with persistent memory across agents, apps, and community -integrations. - -Use cases show what persistent memory makes possible in real products and -workflows. Some examples are packaged in this repository; others point to -external demos or integrations you can study and adapt. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- -[![banner-gif](https://github.com/user-attachments/assets/840470d7-a838-4c05-8685-dd797d4e9cdf)](https://evermind.ai/usecase_reunite) - -#### Reunite - Find With EverOS - -Parents describe what they remember. Children describe what they recall. Reunite uses semantic memory to surface the connections. - -[Learn more](https://evermind.ai/usecase_reunite) - - - -[![banner-gif](https://github.com/user-attachments/assets/7282b38b-56bf-4356-aa7b-06a845e7683d)](https://github.com/tt-a1i/hive) - -#### Hive Orchestrator - -Browser-native hive-mind for CLI coding agents - Claude Code, Codex, Gemini, and OpenCode collaborate as real PTY processes via a team protocol. - -[Code](https://github.com/tt-a1i/hive) - -
- -[![banner-gif](https://github.com/user-attachments/assets/867d9329-ce9a-496f-ab1e-15c77974e5fa)](https://github.com/tt-a1i/evermemos-mcp) - -#### AI Coding Assistants With EverOS - -Universal long-term memory layer for AI coding assistants, powered by EverOS. - -[Code](https://github.com/tt-a1i/evermemos-mcp) - - - -[![banner-gif](https://github.com/user-attachments/assets/a4f0fd86-1c81-4445-bebc-e51eb5e33b30)](https://github.com/yuansui123/AI-Data-Technician-EverMemOS) - -#### AI Data Technician - -An agentic AI system that learns from scientist interaction to inspect, analyze, and classify high-dimensional time series data - with persistent memory that improves across sessions. - -[Code](https://github.com/yuansui123/AI-Data-Technician-EverMemOS) - -
- -![banner-gif](https://github.com/user-attachments/assets/650b901b-c9ba-4001-bac7-626b009df830) - -#### Rokid AI Assistant With EverOS - -Connect to EverOS within Rokid Glasses enabling long-term memory for all of your smart activities. - -Coming soon - - - -![banner-gif](https://github.com/user-attachments/assets/85b338b2-e48e-4a65-9f30-0bc6998df872) - -#### Creative Assistant With Memory - -Creative assistant with long-term memory, so your creative context stays available across sessions. - -Coming soon - -
-Back to top -
- -[![banner-gif](https://github.com/user-attachments/assets/f30617a1-adc0-4271-bc0e-c3a0b28cb903)](https://github.com/xunyud/Earth-Online) - -#### Earth Online Memory Game - -Earth Online is a memory-aware productivity game that turns everyday planning into a living quest log. - -[Code](https://github.com/xunyud/Earth-Online) - - - -[![banner-gif](https://github.com/user-attachments/assets/57d8cda7-35a5-4561-b794-5520dffc917b)](https://github.com/golutra/golutra) - -#### Multi-Agent Orchestration Platform - -Golutra presents a multi-agent workforce for engineering teams, extending the IDE model from a single assistant to coordinated agents. - -[Code](https://github.com/golutra/golutra) - -
- -[![banner-gif](https://github.com/user-attachments/assets/75f19db5-30f6-4eed-9b1e-c9c6a0e6b7de)](https://github.com/Yangtze-Seventh/taste-verse) - -#### Your Personal Tasting Universe - -Record, visualize, and explore your tasting journey through an immersive 3D star map. - -[Code](https://github.com/Yangtze-Seventh/taste-verse) - - - -[![banner-gif](https://github.com/user-attachments/assets/93ac2a68-4f18-4fcb-8d87-80aeb00a9d7c)](https://github.com/kellyvv/OpenHer) - -#### EverOS Open Her - -Build AI that feels. Open-source persona engine - personality emerges from neural drives, not prompts. Inspired by Her. - -[Code](https://github.com/kellyvv/OpenHer) - -
- -[![banner-gif](https://github.com/user-attachments/assets/550071c1-dc39-4964-9f67-ffdfad792345)](https://chromewebstore.google.com/detail/ruminer-browser-agent/lbccjohfpdpimbhpckljimgolndfmfif) - -#### Browser Agent For Personal Memory - -Ruminer brings persistent memory to a browser agent so it can carry personal context across web tasks. - -[Plugin](https://chromewebstore.google.com/detail/ruminer-browser-agent/lbccjohfpdpimbhpckljimgolndfmfif) - - - -[![banner-gif](https://github.com/user-attachments/assets/c258a6c4-fe70-497a-98d1-3dade4a932f6)](https://github.com/nanxingw/EverMem) - -#### EverMem Sync With EverOS - -One command to connect any AI coding CLI to EverMemOS long-term memory. - -[Code](https://github.com/nanxingw/EverMem) - -
-Back to top -
- -[![banner-gif](https://github.com/user-attachments/assets/39274473-ceb3-48fb-a031-e22230decbe2)](https://github.com/mco-org/mco) - -#### MCO - Orchestrate AI Coding Agents - -MCO equips your primary agent with an agent team that can work together to solve complex tasks. - -[Code](https://github.com/mco-org/mco) - - - -[![banner-gif](https://github.com/user-attachments/assets/314c9126-8e08-4688-bbbb-8555ad58cf67)](https://github.com/onenewborn/StudyBuddy-public) - -#### Study Buddy With Self-Evolving Memory - -Study proactively with an agent that has self-evolving memory. - -[Code](https://github.com/onenewborn/StudyBuddy-public) - -
- -[![banner-gif](https://github.com/user-attachments/assets/21da76aa-9a8b-48e0-9134-42429d7390e7)](https://github.com/TonyLiangDesign/MemoCare) - -#### Alzheimer's Memory Assistant - -Empowering individuals with advanced memory support and daily assistance. - -[Code](https://github.com/TonyLiangDesign/MemoCare) - - - -[![banner-gif](https://github.com/user-attachments/assets/e2428df3-ea11-4e88-8f9c-dad437dd8998)](https://github.com/AlexL1024/NeuralConnect) - -#### Memory-Driven Multi-Agent NPC Experience - -An iOS sci-fi mystery game where players explore and uncover the truth. - -[Code](https://github.com/AlexL1024/NeuralConnect) - -
- -[![banner-gif](https://github.com/user-attachments/assets/e6eaf308-a874-483f-8874-6934bf95a78f)](https://github.com/elontusk5219-prog/Mobi) - -#### Mobi Companion - -An iOS app where users create, nurture, and live with a personalized AI companion called Mobi. - -[Code](https://github.com/elontusk5219-prog/Mobi) - - - -[![banner-gif](https://github.com/user-attachments/assets/9aabcaa9-f97a-49d2-9109-0b5bb696ed41)](https://github.com/JaMesLiMers/EvermemCompetition-Spiro) - -#### AI Wearable With Memory - -A context-native AI wearable that listens to everyday life and converts conversations into memory. - -[Code](https://github.com/JaMesLiMers/EvermemCompetition-Spiro) - -
-Back to top -
- -[![banner-gif](https://github.com/user-attachments/assets/df9677ec-386f-4c56-a428-08bca25c54dc)](docs/migration-to-1.0.0.md) - -#### Legacy OpenClaw Agent Memory - -Archived pre-1.0.0 plugin reference. New integrations should use the current EverOS API. - -[Learn more](docs/migration-to-1.0.0.md) - - - -[![banner-gif](https://github.com/user-attachments/assets/3a2357a1-c0c3-464a-8979-0d1cdfc9b0d4)](https://github.com/TEN-framework/ten-framework/tree/04cb80601374fa9e35b4e544b2dbd23286ca7763/ai_agents/agents/examples/voice-assistant-with-EverMemOS) - -#### Live2D Character With Memory - -Add long-term memory to a real-time Live2D character, powered by [TEN Framework](https://github.com/TEN-framework/ten-framework). - -[Code](https://github.com/TEN-framework/ten-framework/tree/04cb80601374fa9e35b4e544b2dbd23286ca7763/ai_agents/agents/examples/voice-assistant-with-EverMemOS) - -
- -[![banner-gif](https://github.com/user-attachments/assets/c36bdc04-97d3-4fe9-97d9-4b93b475595a)](https://screenshot-analysis-vercel.vercel.app/) - -#### Computer-Use With Memory - -Run screenshot-based analysis with computer-use and store the results in memory. - -[Live Demo](https://screenshot-analysis-vercel.vercel.app/) - - - -[![banner-gif](https://github.com/user-attachments/assets/54a7cf8f-62c4-4fbc-9d50-b214d034e051)](use-cases/game-of-throne-demo) - -#### Game Of Thrones Memories - -A demonstration of AI memory infrastructure through an interactive Q&A experience with *A Game of Thrones*. - -[Code](use-cases/game-of-throne-demo) - -
- -[![banner-gif](https://github.com/user-attachments/assets/af37c1f6-7ba5-430c-b99d-2a7e7eac618f)](use-cases/claude-code-plugin) - -#### Claude Code Plugin - -Persistent memory for Claude Code. Automatically saves and recalls context from past coding sessions. - -[Code](use-cases/claude-code-plugin) - - - -[![banner-gif](https://github.com/user-attachments/assets/d521d28c-0ccd-44ff-aecc-828245e2f973)](https://main.d2j21qxnymu6wl.amplifyapp.com/graph.html) - -#### Memory Graph Visualization - -Explore stored entities and relationships in a graph interface. Frontend demo; backend integration is in progress. - -[Live Demo](https://main.d2j21qxnymu6wl.amplifyapp.com/graph.html) - -
- -
-
- -[![](https://img.shields.io/badge/-Back_to_top-gray?style=flat-square)](#readme-top) - -
+## Storage layout + +``` +~/.everos/ +├── default_app/ # app_id ("default" → "default_app" on disk) +│ └── default_project/ # project_id ("default" → "default_project") +│ ├── users// +│ │ ├── user.md # profile +│ │ ├── episodes/ # daily-log episodes (visible) +│ │ ├── .atomic_facts/ # nested facts (dotfile-hidden) +│ │ └── .foresights/ # predictive memory (dotfile-hidden) +│ └── agents// +│ ├── agent.md +│ ├── .cases/ # one task case per entry +│ └── skills/ # named procedural memories +├── .index/ # derived indexes (rebuildable from md) +│ ├── sqlite/system.db # state + queue + audit +│ └── lancedb/*.lance/ # vector + BM25 + scalar +└── .tmp/ # transient working files +``` + +Open any `//users//` folder in Obsidian — your +agent's brain is just files. The dotfile directories (`.atomic_facts/`, +`.foresights/`, `.cases/`) stay hidden by default so the visible folder +is the user-facing memory surface, while extracted derivatives sit +quietly alongside. + +## Features + +- **Hybrid retrieval**: BM25 + vector (HNSW/IVF-PQ) + scalar filter, single-query in LanceDB +- **Cascade index sync**: edit a `.md` → file watcher → entry-level diff → LanceDB sync, sub-second +- **Multi-source extraction**: conversations / agent trajectories / file knowledge +- **Dual-track memory**: user-track (Episodes / Profiles) + agent-track (Cases / Skills) +- **Async-first**: full asyncio, single event loop +- **Multi-modal**: text + small image / audio inline; large media via S3/OSS reference + +## Project structure + +``` +everos/ # repo root +├── src/everos/ # main package (src layout) +│ ├── entrypoints/ # cli + api +│ ├── service/ # use case orchestration +│ ├── memory/ # domain: extract + search + cascade + prompt_slots +│ ├── infra/ # storage: markdown + lancedb + sqlite +│ ├── component/ # cross-cutting: llm / embedding / config / utils +│ ├── core/ # runtime: observability / lifespan / context +│ └── config/ # configuration data + Settings schema +├── tests/ # unit / integration / golden / fixtures +├── docs/ # design docs +└── .claude/ # team-shared rules + skills (auto-loaded by Claude Code) +``` ## Documentation -- [docs/everos-demo.md](docs/everos-demo.md) — Demo scope and TUI source layout -- [docs/how-memory-works.md](docs/how-memory-works.md) — Markdown, SQLite, LanceDB, and recall flow -- [docs/use-cases.md](docs/use-cases.md) — Full use-case gallery and integration examples -- [docs/engineering.md](docs/engineering.md) — Engineering and CI tooling -- [docs/migration-to-1.0.0.md](docs/migration-to-1.0.0.md) — Legacy API migration notes +- [docs/overview.md](docs/overview.md) — Project overview & vision +- [docs/architecture.md](docs/architecture.md) — DDD layered architecture & dependency rules +- [docs/engineering.md](docs/engineering.md) — Engineering & dev-efficiency infrastructure (CI / tooling / Claude Code) +- [docs/multimodal.md](docs/multimodal.md) — Multimodal memory: ingest image / pdf / audio / office docs via the HTTP API - [CHANGELOG.md](CHANGELOG.md) — Release notes - [CONTRIBUTING.md](CONTRIBUTING.md) — How to contribute +- [.claude/rules/](.claude/rules/) — Detailed coding conventions (auto-loaded by Claude Code) -
-
+## Use Cases -[![](https://img.shields.io/badge/-Back_to_top-gray?style=flat-square)](#readme-top) - -
- -## Star Us - -If EverOS is useful to your agent stack, please star the repo. It helps more -builders discover the project and gives the memory ecosystem a stronger signal -to keep improving. - -### Star History - -[![Star History Chart](https://api.star-history.com/svg?repos=EverMind-AI/EverOS&type=Date)](https://www.star-history.com/#EverMind-AI/EverOS&Date) - -
-
- -[![](https://img.shields.io/badge/-Back_to_top-gray?style=flat-square)](#readme-top) - -
- -## EverMind Ecosystems - -EverMind is an open-source ecosystem for long-term memory, self-evolving agents, and memory evaluation. +See [use-cases/README.md](use-cases/README.md) for the full gallery. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
EverMind Open-Source Ecosystem
Memory RuntimeEverOS - the local memory operating system and research-backed runtime for agent and user memory.
Algorithm EngineEverAlgo - stateless extraction, ranking, parsing, and memory operators that power EverOS.
Hypergraph MemoryHyperMem - hypergraph memory for long-term conversations, with its own benchmark-backed topic -> episode -> fact retrieval method.
BenchmarksEverMemBench · EvoAgentBench - evaluation suites for conversational memory and agent self-evolution.
Long-Context ResearchMSA - Memory Sparse Attention for scalable latent memory and 100M-token contexts.
Personal Memory LayerEverMe - CLI and agent plugin suite for cross-device, cross-agent personal memory.
Developer Integrationsevermem-claude-code · everos-plugins - plugins, skills, and migration tooling for AI coding agents.
-Together, these repositories form EverMind's research-to-runtime stack: new memory methods, reusable algorithms, benchmark evidence, and practical agent integrations. +## Status -
-
+**Stable (v1.1.0)** — Released on PyPI; the v1 API is stable. -[![](https://img.shields.io/badge/-Back_to_top-gray?style=flat-square)](#readme-top) - -
- -
- -## Contributing - -Contributions are welcome across the whole repository: memory methods, benchmark coverage, use-case examples, documentation, and bug fixes. Browse [Issues](https://github.com/EverMind-AI/EverOS/issues) to find a good entry point, then open a PR when you are ready. - -
- -> [!TIP] -> -> **Welcome all kinds of contributions** 🎉 -> -> Help make EverOS better. Code, documentation, benchmark reports, use-case write-ups, and integration examples are all valuable. Share your projects on social media to inspire others. -> -> Connect with one of the EverOS maintainers [@elliotchen200](https://x.com/elliotchen200) on 𝕏 or [@cyfyifanchen](https://github.com/cyfyifanchen) on GitHub for project updates, discussions, and collaboration opportunities. - -![divider](https://github.com/user-attachments/assets/2e2bbcc6-e6d8-4227-83c6-0620fc96f761#gh-light-mode-only) -![divider](https://github.com/user-attachments/assets/d57fad08-4f49-4a1c-bdfc-f659a5d86150#gh-dark-mode-only) - -### Code Contributors - -[![EverOS Contributors](https://contrib.rocks/image?repo=EverMind-AI/EverOS)](https://github.com/EverMind-AI/EverOS/graphs/contributors) - -![divider](https://github.com/user-attachments/assets/2e2bbcc6-e6d8-4227-83c6-0620fc96f761#gh-light-mode-only) -![divider](https://github.com/user-attachments/assets/d57fad08-4f49-4a1c-bdfc-f659a5d86150#gh-dark-mode-only) - -### License +## License [Apache License 2.0](LICENSE) — see [NOTICE](NOTICE) for third-party attributions. -### Citation +## Citation If you use EverOS in research, see [CITATION.md](CITATION.md). -
+--- -
- -[![](https://img.shields.io/badge/-Back_to_top-gray?style=flat-square)](#readme-top) - -
+**Acknowledgments**: This project builds on prior research and tooling — see [ACKNOWLEDGMENTS.md](ACKNOWLEDGMENTS.md). diff --git a/docs/api.md b/docs/api.md index e9a0a53..5750aa3 100644 --- a/docs/api.md +++ b/docs/api.md @@ -31,6 +31,7 @@ business semantics the raw spec does not carry. - [POST /api/v1/memory/search](#post-apiv1memorysearch) - [POST /api/v1/memory/get](#post-apiv1memoryget) - [POST /api/v1/ome/trigger](#post-apiv1ometrigger) + - [Knowledge endpoints](#knowledge-endpoints) - [OpenAPI spec source](#openapi-spec-source) ## Overview @@ -43,10 +44,12 @@ business semantics the raw spec does not carry. | Port | `8000` | `EVEROS_API__PORT` env var or `--port` flag | | Version prefix | `/api/v1` | — | -All business endpoints documented here live under `/api/v1/memory/`. -The operational endpoints `GET /health` and `GET /metrics` exist but -are intentionally outside this reference — they are runtime probes for -deployment, not part of the application contract. +Business endpoints live under `/api/v1/memory/`, `/api/v1/ome/`, and +`/api/v1/knowledge/`. Knowledge endpoints have their own dedicated +reference at [docs/knowledge.md](knowledge.md) and are cross-referenced +below. The operational endpoints `GET /health` and `GET /metrics` exist +but are intentionally outside this reference — they are runtime probes +for deployment, not part of the application contract. ### Content type @@ -207,15 +210,6 @@ parsing the human-readable `message` field. > returned — only the first error's message. A client that needs the > offending field can read the `` suffix in `message`. -### Search degradation - -When a `/search` call uses `method: "vector"` or `"hybrid"` and the -embedding or rerank service is temporarily unavailable, the server does -**not** return `503`. Instead, it degrades gracefully — the response -suggests an alternative method in the error detail so the client can -retry with `"keyword"` (which requires no embedding). This keeps search -available during transient provider outages. - ## Common types ### MessageItem @@ -380,7 +374,7 @@ A node is a JSON object whose keys are one of: |---|---|---| | `AND` | `array` | All child nodes must match. Omit if not needed | | `OR` | `array` | At least one child node must match. Omit if not needed | -| ** | scalar or operator map | Predicate on that field — see [Allowed fields](#filter-allowed-fields) and [Operators](#filter-operators) | +| ** | scalar or operator map | Predicate on that field — see [Allowed fields](#allowed-fields) and [Operators](#operators) | `AND`, `OR`, and scalar predicates **mix freely at the same level**; they are implicitly joined with `AND`. A node with only scalar keys is @@ -540,7 +534,7 @@ correlation. #### cURL example ```bash -TS=$(date +%s) +TS=$(( $(date +%s) * 1000 )) curl -X POST http://127.0.0.1:8000/api/v1/memory/add \ -H 'Content-Type: application/json' \ -d "{ @@ -644,6 +638,7 @@ optional final LLM rerank. Returns ranked items grouped by kind. | `method` | [SearchMethod](#searchmethod) | no | `"hybrid"` | — | | `top_k` | `integer` | no | `-1` | `-1` or `1..100` | | `radius` | `number \| null` | no | `null` | `0.0 ≤ x ≤ 1.0` if set | +| `min_score` | `number \| null` | no | `null` | `0.0 ≤ x ≤ 1.0` if set | | `include_profile` | `boolean` | no | `false` | — | | `enable_llm_rerank` | `boolean` | no | `false` | — | | `filters` | [FilterNode](#filternode-filter-dsl) `\| null` | no | `null` | — | @@ -683,6 +678,10 @@ radius: 3. With `top_k>0` and no caller-supplied `radius`, no threshold is applied (`null`). +**`min_score`** — Optional **post-fusion relevance floor** in +`[0.0, 1.0]`. Results below this score are evicted after fusion, +independent of `radius` (which is a per-recall cosine threshold). + **`include_profile`** — When `user_id` is set, also fetch the user's profile and include it in `data.profiles`. The profile is not ranked; `score` is `null`. Ignored when `agent_id` is set. @@ -816,7 +815,7 @@ attribution, so `session_id` is the only meaningful query dimension. | `sender_id` | `string` | Original sender id from `/add` | | `sender_name` | `string \| null` | Original sender name; `null` if not provided | | `role` | `"user" \| "assistant" \| "tool"` | Original role | -| `content` | `string \| array` | `string` for the single-text shorthand, `array` of opaque content items for the original multimodal payload (mirrors [MessageItem.content](#addmessage)) | +| `content` | `string \| array` | `string` for the single-text shorthand, `array` of opaque content items for the original multimodal payload (mirrors [MessageItem.content](#messageitem)) | | `timestamp` | `string` | ISO-8601 with timezone offset — see [Conventions](#conventions) | | `tool_calls` | `array \| null` | Original tool_calls payload if any | | `tool_call_id` | `string \| null` | Original tool_call_id if any | @@ -1079,6 +1078,28 @@ curl -X POST http://127.0.0.1:8000/api/v1/ome/trigger \ --- +### Knowledge endpoints + +The knowledge base subsystem (`/api/v1/knowledge/*`) provides document +upload, CRUD, and hybrid search. These endpoints are fully documented +in their own reference: **[docs/knowledge.md](knowledge.md)**. + +Summary of available routes: + +| Method | Path | Description | +|---|---|---| +| `POST` | `/api/v1/knowledge/documents` | Upload and extract a document | +| `GET` | `/api/v1/knowledge/documents` | List documents (paginated) | +| `GET` | `/api/v1/knowledge/documents/{doc_id}` | Get a single document | +| `PUT` | `/api/v1/knowledge/documents/{doc_id}` | Replace a document | +| `PATCH` | `/api/v1/knowledge/documents/{doc_id}` | Partial update | +| `DELETE` | `/api/v1/knowledge/documents/{doc_id}` | Delete a document | +| `GET` | `/api/v1/knowledge/topics/{topic_id}` | Get a single topic | +| `POST` | `/api/v1/knowledge/search` | Hybrid search over topics | +| `GET` | `/api/v1/knowledge/categories` | List taxonomy categories | + +--- + ## OpenAPI spec source This document mirrors the OpenAPI 3.x spec that FastAPI auto-generates diff --git a/docs/architecture.md b/docs/architecture.md index 1a7ca66..afb5efe 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -14,6 +14,7 @@ ├──────────────────────────────────────────────────────┤ │ memory/ (Domain — Business core) │ │ models + extract + search + cascade + prompt_slots │ +│ + reflection + strategies + get + events │ ├──────────────────────────────────────────────────────┤ │ infra/persistence (Storage adapters; infra/ may host other adapter types) │ │ markdown + sqlite + lancedb │ @@ -21,7 +22,7 @@ Cross-cutting (used by all layers, depends on none): component/ ← Injectable providers (LLM / Embedding / parser / config / utils) - core/ ← Runtime base (observability / lifespan / context / errors) + core/ ← Runtime base (observability / lifespan / context / errors / persistence / middleware) config/ ← Configuration data (Settings schema + default.toml) ``` @@ -131,6 +132,7 @@ User query ``` extract/ ├── ingest/ Standardized message intake + multi-modal parser dispatch +├── parser/ Input parsing (format normalization, message preprocessing) ├── pipeline/ Main extraction pipeline (calls everalgo + dual-track split + writes store) └── evolution/ Async memory evolution (event/counter/cron triggers) ``` @@ -144,6 +146,8 @@ Daemon that watches markdown changes and syncs to LanceDB: - Entry-level diff (added / changed / removed) - LanceDB single-transaction update (text + vector columns atomic) - LSN-based crash recovery via the SQLite `md_change_state` queue +- Handlers for all eight business kinds: episode, atomic_fact, foresight, + user_profile, agent_case, agent_skill, knowledge_document, knowledge_topic ### `memory/prompt_slots/` @@ -157,7 +161,23 @@ config/prompt_slots/*.yaml (Layer 1: defaults, ships with package) runtime override (Layer 3: per-call override) ``` -everalgo receives PromptSlot as parameter — no hardcoded prompts in algorithm code. +Extractors may accept a prompt-override parameter; EverOS supplies overrides for episode and boundary-detection prompts, and falls back to the algo-bundled default elsewhere — no hardcoded prompts in algorithm code. + +### `memory/reflection/` + +Offline memory self-improvement. The orchestrator (`orchestrator.py`) +implements the Select → Merge → Re-extract → Deprecate pipeline, merging +fragmented episodes within a cluster into a single coherent narrative. Driven +by the `reflect_episodes` OME strategy (cron, disabled by default). + +### `memory/strategies/` + +OME strategy implementations — one file per strategy: + +- `extract_atomic_facts` / `extract_foresight` / `extract_user_profile` — user pipeline +- `extract_agent_case` / `extract_agent_skill` — agent pipeline +- `reflect_episodes` — offline episode consolidation (cron) +- `trigger_profile_clustering` / `trigger_skill_clustering` — clustering triggers ### `core/observability/` @@ -191,21 +211,22 @@ protection (L1 read-only / L2 system / L3 business / L4 user). ## everalgo boundary -`everalgo` is a set of PyPI-published packages (`everalgo-core`, -`everalgo-boundary`, `everalgo-user-memory`, `everalgo-agent-memory`, -`everalgo-rank`, plus the optional `everalgo-parser` extra), imported under -the `everalgo` namespace, holding **only memory extraction algorithms**: +`everalgo` is a set of PyPI-published packages (`everalgo-user-memory`, +`everalgo-agent-memory`, `everalgo-rank`, `everalgo-knowledge`, plus the +optional `everalgo-parser` extra), imported under the `everalgo` namespace, +holding **only memory extraction algorithms**: - `everalgo.parser` — multi-modal parsing (optional `[multimodal]` extra) - `everalgo.user_memory` — ConvMemCell / Episode / Foresight / AtomicFact / Profile extractors - `everalgo.agent_memory` — AgentMemCell / Case / Skill extractors -- `everalgo.boundary` / `everalgo.rank` — boundary detection / fusion + rerank +- `everalgo.rank` — boundary detection / fusion + rerank +- `everalgo.knowledge` — KnowledgeExtractor (document parse + topic extraction) everalgo is: - **Stateless** — pure functions, no class hierarchy - **No I/O** — does not touch md files / LanceDB / SQLite -- **No prompts inline** — receives `PromptSlot` parameter, project supplies defaults +- **No prompts inline** — extractors that accept a prompt-override parameter use the project-supplied value; others use their algo-bundled defaults This boundary lets everalgo be reused across product forms (this open-source build, EverOS Cloud, OpenClaw plugins, etc.). diff --git a/docs/cascade_runbook.md b/docs/cascade_runbook.md index d5f628f..c9811ee 100644 --- a/docs/cascade_runbook.md +++ b/docs/cascade_runbook.md @@ -7,13 +7,15 @@ the recurring operational questions. ## What runs where -When `everos server start` boots, the FastAPI lifespan wires four +When `everos server start` boots, the FastAPI lifespan wires six providers in order: 1. **Metrics** — Prometheus collector. -2. **SQLite** — system DB + schema (`SQLModel.metadata.create_all`). -3. **LanceDB** — async connection + schema verification + FTS indexes. -4. **Cascade** — watcher + scanner + worker, all in-process tasks. +2. **LLM** — LLM client initialisation. +3. **SQLite** — system DB + schema (`SQLModel.metadata.create_all`). +4. **LanceDB** — async connection + schema verification + FTS indexes. +5. **Cascade** — watcher + scanner + worker, all in-process tasks. +6. **OME** — offline memory engine. The cascade subsystem itself is three independent loops: @@ -204,7 +206,7 @@ Lives in `LanceDBSettings`; overridable via the `EVEROS_LANCEDB__INDEX_CACHE_SIZE_BYTES` environment variable. This is the only knob that bounds the steady-state file-descriptor count of a long-running EverOS daemon — see -[Recovery paths § FD exhaustion](#fd-exhaustion-os-error-24-emfile) +[Recovery paths § FD exhaustion](#fd-exhaustion-os-error-24--emfile) for why nothing else (prune, rebuild, `drop_index`) helps. Measured cap → FD ceiling (30 add+optimize cycles + 100-query stress diff --git a/docs/configuration.md b/docs/configuration.md index f0334cf..eda0f49 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -115,6 +115,8 @@ everos init --root /data/everos | `api_key` | string | — | **Yes** | API key. | | `base_url` | string | — | No | Custom endpoint URL. | | `max_concurrency` | int | `4` | No | Max parallel parsing requests. | +| `file_uri_allow_dirs` | list[string] | `[]` | No | Allowlisted base dirs for `file://` URIs. Empty = allow any readable file. | +| `file_uri_max_bytes` | int | `52428800` | No | Max size (bytes) of a `file://` asset; larger files are rejected. | ### `[embedding]` @@ -168,6 +170,12 @@ everos init --root /data/everos |---|---|---|---| | `vector_strategy` | string | `"maxsim_atomic"` | Vector retrieval path: `maxsim_atomic` (finer-grained) or `episode` (legacy). | +### `[knowledge]` + +| Field | Type | Default | Description | +|---|---|---|---| +| `max_upload_bytes` | int | `52428800` | Max bytes for an uploaded knowledge document (50 MiB). Oversized uploads are rejected with HTTP 422 before parsing. | + ### `[knowledge.search]` | Field | Type | Default | Description | diff --git a/docs/engineering.md b/docs/engineering.md index 112041e..2a97cbb 100644 --- a/docs/engineering.md +++ b/docs/engineering.md @@ -410,21 +410,11 @@ File locations: - GitLab: `.gitlab/merge_request_templates/default.md` - GitHub: `.github/PULL_REQUEST_TEMPLATE.md` -### 7.3 CODEOWNERS (by DDD layer) +### 7.3 Code ownership -``` -/src/everos/memory/ @chandler.zhang @libin.zhang001 -/src/everos/infra/ @chandler.zhang @yeanhua -/src/everos/component/ @chandler.zhang -/src/everos/core/ @chandler.zhang -/src/everos/service/ @chandler.zhang @libin.zhang001 -/src/everos/entrypoints/ @chandler.zhang -/.claude/ @chandler.zhang -/.gitlab-ci.yml @chandler.zhang @jianhua.yao -``` - -At least one owner per directory; two owners for critical modules. Edits -auto-mention the corresponding owners. +The `.gitlab/CODEOWNERS` file was removed (commit `e870927`) to avoid +leaking internal accounts. Code ownership is now managed via GitLab +project-level settings (Merge Request approval rules). ### 7.4 Commit convention (Gitmoji) @@ -488,8 +478,8 @@ CONTRIBUTING.md contributor onboarding: setup / code style / │ │ │ merge │ │ GitHub Actions │ /.github/workflows/ci.yml │ PR cannot │ │ │ │ merge │ -│ CODEOWNERS │ /.gitlab/CODEOWNERS │ no auto │ -│ │ │ reviewer │ +│ Code ownership │ GitLab project settings │ no auto │ +│ │ (approval rules) │ reviewer │ │ GitLab MR template │ /.gitlab/merge_request_templates/ │ no MR temp │ │ GitHub PR template │ /.github/PULL_REQUEST_TEMPLATE.md │ no PR temp │ │ Issue templates │ /.github/ISSUE_TEMPLATE/ (3) │ scattered │ @@ -509,13 +499,13 @@ Near-term □ /run-eval skill: run behavior-consistency eval □ ruff rule sets: add D (docstring), ANN (annotations) -Mid-term (before v0.5) +Mid-term (v1.2 – v1.3) □ Type checking re-introduction (pyright or mypy) once hot paths stabilize □ release-please / Conventional Commits → automated changelog □ pre-commit autoupdate cadence □ Performance benchmark CI with historical comparison -Long-term (after v1.0) +Long-term (v2+) □ /security-review skill: automated security review □ Mutation testing (mutmut) □ Multi-Python version matrix (3.12 / 3.13) diff --git a/docs/how-memory-works.md b/docs/how-memory-works.md index da6969d..1a81364 100644 --- a/docs/how-memory-works.md +++ b/docs/how-memory-works.md @@ -69,7 +69,7 @@ visually distinct from a user-named one). │ │ │ └── agent_case-.md │ │ └── skills/ skill-named dir │ │ └── skill_/SKILL.md (+ references/ scripts/) -│ └── knowledge/ ← shared / global (reserved) +│ └── knowledge/ ← shared / global │ ├── .index/ ← system-managed, rebuildable (gitignore) │ ├── sqlite/ @@ -147,8 +147,8 @@ index catches up asynchronously. ## Memory types & storage strategies -Six business memory kinds today, each user- or agent-owned, each picking -one of three on-disk patterns: +Eight business memory kinds today, each user-, agent-, or globally owned, +each picking one of three on-disk patterns: | Kind | Owner | Dir / file | Strategy | Produced by | |---|---|---|---|---| @@ -158,6 +158,8 @@ one of three on-disk patterns: | **profile** | user | `user.md` | single-file rewrite | OME | | **agent_case** | agent | `.cases/agent_case-.md` (hidden) | daily-log | OME | | **agent_skill** | agent | `skills/skill_/SKILL.md` | skill-named dir | OME (clustering) | +| **knowledge_document** | global | `knowledge///index.md` | knowledge tree | knowledge service | +| **knowledge_topic** | global | `knowledge///_.md` | knowledge tree | knowledge service | The three strategies: @@ -206,6 +208,8 @@ and write their markdown when ready: - `extract_agent_case` — a reusable agent trajectory (only when the cell is substantive enough; thin trajectories are skipped by design) - `extract_agent_skill` — clusters related cases into a named skill +- `trigger_profile_clustering` — triggers user profile clustering +- `trigger_skill_clustering` — triggers agent skill clustering - `reflect_episodes` (cron, default off) — offline memory consolidation. Merges fragmented episodes within a cluster into a single coherent narrative, re-extracts atomic facts, and deprecates the originals. diff --git a/docs/index.md b/docs/index.md index eeee75b..6d33157 100644 --- a/docs/index.md +++ b/docs/index.md @@ -4,16 +4,6 @@ Documentation for [EverOS](../README.md) — md-first memory extraction framework. Organised by [Diátaxis](https://diataxis.fr/) — what kind of question you have determines which section to read. -## Tutorials - -Learning-oriented entry points — start here to get a feel for the system -before wiring it into a real workflow. - -| Doc | Purpose | -|---|---| -| [everos-demo.md](everos-demo.md) | `everos demo` — local educational TUI to feel the memory lifecycle before configuring keys | -| [use-cases.md](use-cases.md) | Worked examples and integrations showing what persistent memory enables, to study and adapt | - ## Reference Technical reference: contracts, commands, schemas — read these when you @@ -25,10 +15,10 @@ already know what you want to do and need to know exactly how. | [knowledge.md](knowledge.md) | Knowledge base module — upload, search, taxonomy, storage layout | | [reflection.md](reflection.md) | Reflection — offline memory consolidation: enable, schedule, storage, triggering | | [cli.md](cli.md) | `everos` CLI subcommands + env var conventions | -| [configuration.md](configuration.md) | Two-file TOML configuration + environment-variable overrides for container deployments | -| [multimodal.md](multimodal.md) | Multimodal ingest — supported modalities, `uri` vs `base64` payloads, required extras/config | | [storage_layout.md](storage_layout.md) | Memory-root tree + frontmatter chassis + EntryId encoding | -| [prompt_slots.md](prompt_slots.md) | YamlConfigLoader + three-layer prompt override | +| [prompt_slots.md](prompt_slots.md) | PromptSlot loader — bundled default prompts (Layer 1 live; app/runtime overlays planned) | +| [configuration.md](configuration.md) | TOML / env-var configuration reference | +| [multimodal.md](multimodal.md) | Multimodal content items — image / PDF / audio / doc parsing | ## Explanation @@ -50,8 +40,7 @@ specific thing (drain a queue, recover from a stuck row, etc.). | Doc | Purpose | |---|---| | [cascade_runbook.md](cascade_runbook.md) | Cascade subsystem ops — drain queue, recover stuck rows | -| [locomo_benchmark.md](locomo_benchmark.md) | Reproduce EverOS's LoCoMo retrieval scores locally (`hybrid` / `agentic`) | -| [migration-to-1.0.0.md](migration-to-1.0.0.md) | Migrate off pre-1.0.0 APIs / infrastructure to the current 1.0.0 contract | +| [locomo_benchmark.md](locomo_benchmark.md) | LoCoMo benchmark — run and evaluate | ## Engineering / Internal @@ -70,7 +59,6 @@ Top-level project files live next to the repo root: - [QUICKSTART.md](../QUICKSTART.md) — 5-minute walkthrough (install → service → search) - [CONTRIBUTING.md](../CONTRIBUTING.md) — how to contribute (issue-only model) - [CHANGELOG.md](../CHANGELOG.md) — release notes -- [release-notes-1.1.0.md](release-notes-1.1.0.md) — EverOS 1.1.0 highlights (Knowledge, Reflection, OME) - [SECURITY.md](../SECURITY.md) — security policy & private vulnerability reporting - [CITATION.md](../CITATION.md) — academic citation info - [ACKNOWLEDGMENTS.md](../ACKNOWLEDGMENTS.md) — third-party acknowledgments diff --git a/docs/knowledge.md b/docs/knowledge.md index 463479f..c2aa816 100644 --- a/docs/knowledge.md +++ b/docs/knowledge.md @@ -280,7 +280,8 @@ PUT /documents/{doc_id} Content-Type: multipart/form-data ``` -Same fields as POST. Atomic operation: if extraction fails, the old +Same fields as POST. Returns 404 if `doc_id` does not exist; on success +returns 200 (not 201). Atomic operation: if extraction fails, the old document is restored from backup. ### Update metadata @@ -297,8 +298,8 @@ Content-Type: application/json } ``` -Returns `updated_fields: ["title", "category_id"]`. Changing `category_id` -moves the document directory to the new category folder. +Returns `doc_id`, `updated_at`, and `updated_fields: ["title", "category_id"]`. +Changing `category_id` moves the document directory to the new category folder. ### Delete a document @@ -306,8 +307,8 @@ moves the document directory to the new category folder. DELETE /documents/{doc_id} ``` -Returns 204 if the document did not exist (idempotent), or 200 with -`deleted_topics` count. +Returns 204 when no topics were removed (document absent or present with zero +topics); 200 with `doc_id` + `deleted_topics` otherwise. ### List documents @@ -459,6 +460,7 @@ degradation). The two failure modes map to distinct status codes: "content": "The P99 API latency dropped...", "score": 0.92, "retrieval_method": "hybrid", + "source": null, "document": { "doc_id": "d_a1b2c3d4e5f6", "title": "Q1 Engineering Report", @@ -574,18 +576,11 @@ pip install everos[multimodal] | 422 | `INVALID_INPUT` | Empty/oversized query, empty title, invalid ID format | | 500 | `CONFIGURATION_ERROR` | Embedding or rerank provider not configured | | 503 | `EXTERNAL_SERVICE_UNAVAILABLE` | Configured embedding/rerank provider failing at call time | +| 422 | `EXTRACTION_EMPTY` | Document parsed but extractor produced no topics | | 503 | `CAPABILITY_UNAVAILABLE` | `everos[multimodal]` not installed | -All error responses include a human-readable `message` field: - -```json -{ - "error": { - "code": "NOT_FOUND", - "message": "Document 'd_abc123' not found" - } -} -``` +All error responses use the standard error envelope — see +[api.md → Errors](api.md#errors). ## Multi-tenancy diff --git a/docs/locomo_benchmark.md b/docs/locomo_benchmark.md index f4bc70b..1fe5f14 100644 --- a/docs/locomo_benchmark.md +++ b/docs/locomo_benchmark.md @@ -42,7 +42,7 @@ later with `--data-path` if you keep it elsewhere. ```bash EVEROS_ROOT=~/.everos \ -uv run python -m everos.entrypoints.cli.main server start --port 8000 +uv run everos server start --port 8000 ``` `EVEROS_ROOT` isolates one benchmark's corpus from another — diff --git a/docs/multimodal.md b/docs/multimodal.md index 8551d70..c0100a7 100644 --- a/docs/multimodal.md +++ b/docs/multimodal.md @@ -38,7 +38,7 @@ POST /api/v1/memory/add parsed text merged back into the session buffer (in original order) │ ▼ - boundary detector → extraction LLM → MemCell + boundary detector → extraction LLM → memory cell (MemCell) │ ▼ markdown (truth) + SQLite (state) + LanceDB (vector + BM25) @@ -56,7 +56,7 @@ text-bearing formats can be parsed without it (e.g. a plain email with no inline images). The parser returns text; that text takes the place of the asset in the message buffer. Nothing downstream of the parser knows or cares that the content originated as an image or PDF — the raw -bytes are **not** persisted past extraction (the episode and memory cell +bytes are **not** persisted past extraction (the episode and memory cell (`MemCell`) store only the parsed text). ## Prerequisites @@ -271,7 +271,7 @@ All fields bind from the environment via the parent `Settings` |---|---|---| | `EVEROS_MULTIMODAL__MODEL` | `google/gemini-3-flash-preview` | Parsing model; must accept `image_url` parts | | `EVEROS_MULTIMODAL__API_KEY` | — | API key for the multimodal endpoint | -| `EVEROS_MULTIMODAL__BASE_URL` | `https://openrouter.ai/api/v1` | OpenAI-compatible base URL | +| `EVEROS_MULTIMODAL__BASE_URL` | `None` | OpenAI-compatible base URL | | `EVEROS_MULTIMODAL__MAX_CONCURRENCY` | `4` | Cap on parallel multimodal calls within one extraction | | `EVEROS_MULTIMODAL__FILE_URI_MAX_BYTES` | `52428800` (50 MiB) | Max size of a `file://` asset | | `EVEROS_MULTIMODAL__FILE_URI_ALLOW_DIRS` | `[]` (any) | JSON list of allowlisted base dirs for `file://` URIs | diff --git a/docs/overview.md b/docs/overview.md index 038f1b8..7f9be38 100644 --- a/docs/overview.md +++ b/docs/overview.md @@ -13,7 +13,10 @@ Build an open-source Python memory framework where **AI agents' long-term memory - Hybrid retrieval (BM25 + vector + scalar filter) - Cascade index sync (md edit → LanceDB sub-second) - Dual-track memory (user-track / agent-track) -- Offline memory evolution (Foresight / AtomicFact / Profile / Skill) +- Offline memory evolution (Foresight / AtomicFact / Profile / Skill), + including Reflection — a consolidation strategy within the OME that + merges + re-extracts related episodes +- Knowledge base (document upload, parse, CRUD, semantic search) - CLI + HTTP API **Out of scope (v1, future v2)**: @@ -43,7 +46,7 @@ User trust comes from physical visibility — the user can `cat` / `vim` / `grep ### 3. Algorithm-orchestration separation -`everalgo` (a set of separate PyPI packages — `everalgo-core` / `-boundary` / `-user-memory` / `-agent-memory` / `-rank`, plus the optional `-parser` extra) holds the extraction algorithms (MemCell extraction, Episode generation, Profile evolution). EverOS calls everalgo via the PromptSlot interface; everalgo knows nothing about storage. +`everalgo` (a set of separate PyPI packages — `everalgo-user-memory` / `-agent-memory` / `-rank` / `-knowledge`, plus the optional `-parser` extra) holds the extraction algorithms (memory-cell extraction, episode generation, profile evolution). EverOS calls everalgo's extractor functions directly — passing storage-free data in, getting structured results out; for a couple of extractors (episode and boundary detection) it can override the bundled prompt via the PromptSlot mechanism. everalgo knows nothing about storage. This boundary lets the same algorithm power both this open-source lightweight version and other product forms. @@ -78,9 +81,9 @@ Strict single-direction dependency, enforced by `import-linter` in CI. - **v0.2** — Full extraction pipeline (workspace / agent / knowledge), evolution framework - **v0.3** — Production hardening, full CLI, HTTP API, Obsidian demo - **v1.0** — Stable API, PyPI release, comprehensive docs +- **v1.1** — Knowledge base + Reflection (offline memory consolidation) - **v2** (future) — Edge-to-cloud sync via EverMe (separate project) ## Status -**Stable (v1.0.1)** — Released on PyPI; the v1 API is stable. Development -continues on `dev` toward v1.1. +**Latest stable release: v1.1.0** (PyPI) — the v1 API is stable. diff --git a/docs/prompt_slots.md b/docs/prompt_slots.md index d66c046..be35ad2 100644 --- a/docs/prompt_slots.md +++ b/docs/prompt_slots.md @@ -5,9 +5,12 @@ the prompts it sends to LLMs. Algorithm code receives a `PromptSlot` parameter; the *project* (EverOS) supplies defaults and lets operators override. -> **Status (2026-05-07)**: the YAML loader is implemented; the higher- -> level `PromptSlot` model + sandbox dry-run + three-layer overlay -> resolution arrive when the memory layer ships (see Stage 2). +> **Status (2026-05-07)**: Layer 1 (bundled defaults under +> `config/prompt_slots/`) is live — `PromptLoader` is integrated into the +> memorize pipeline (`service/memorize.py`). Two slots ship today — +> `boundary_detection` and `episode_extract`; other extractors use their +> algo-bundled defaults. Layers 2-3 (app-level overlay from +> `~/.everos/prompt_slots/` and per-call runtime override) are still pending. ## Three-layer overlay @@ -27,63 +30,44 @@ layer 3 is supplied at the call site. The prompt-slots public entry point is [`PromptLoader`](../src/everos/memory/prompt_slots/loader.py) (re-exported -from `everos.memory.prompt_slots`); it wraps the generic category loader -[`YamlConfigLoader`](../src/everos/component/config/loader.py). The generic -loader is shown below — `PromptLoader` is the prompt-slots-specific wrapper -over the same mechanism: +from `everos.memory.prompt_slots`). Its public method is +`load(name: str) -> str | None` — returns the override template when the +slot is enabled and non-empty, or `None` to fall back to the algo default. + +Internally, `PromptLoader` wraps the generic category loader +[`YamlConfigLoader`](../src/everos/component/config/loader.py): ```python +from everos.memory.prompt_slots import PromptLoader from pathlib import Path -from everos.component.config import YamlConfigLoader -loader = YamlConfigLoader( - root=Path("src/everos/config"), - categories={"prompt_slots": None}, # subdir == category name -) +loader = PromptLoader(config_root=Path("src/everos/config")) -# Reads /prompt_slots/episode_extract.yaml → dict -slot = loader.find("prompt_slots", "episode_extract") - -# Refresh after on-disk edits. -loader.refresh() # drop the entire cache -loader.refresh("prompt_slots") # drop one category -loader.refresh("prompt_slots", "episode_extract") # drop one entry +# Returns the template string, or None when disabled / empty. +template = loader.load("episode_extract") ``` +The underlying `YamlConfigLoader` supports `find()`, `refresh()`, etc. — +but callers should use `PromptLoader.load()` rather than reaching into the +generic layer directly. + Top-level YAML is required to be a mapping; a list / scalar root raises `TypeError` to fail-fast (loud, not silent). -## YAML format (proposed; subject to change) +## YAML format + +Each slot file uses two keys: `enabled` (boolean) and `template` (string). ```yaml # config/prompt_slots/episode_extract.yaml -template: | - Extract a single episode from this conversation: - {{ memcell.text }} - -variables: - memcell: input memcell - -output_schema: - type: object - properties: - summary: { type: string } - participants: { type: array } - -llm: - model: gpt-4.1-mini - temperature: 0.3 - max_tokens: 2000 - -validation: - test_cases: - - input: { memcell: { text: "Hi" } } - expected: { summary: "...", participants: [] } +enabled: false +template: "" ``` -When layer 2 supplies an override the loader will be re-pointed at -`~/.everos/prompt_slots/`; the runtime resolution logic (currently TBD) -sandbox-runs the merged slot before returning it. +When `enabled` is `true` and `template` is a non-empty string, +`PromptLoader.load()` returns the template as-is. Otherwise it returns +`None`, and the pipeline falls back to the algo-bundled default prompt +(zero override cost). ## Why YAML (not TOML) diff --git a/docs/storage_layout.md b/docs/storage_layout.md index 9fea9c9..8604466 100644 --- a/docs/storage_layout.md +++ b/docs/storage_layout.md @@ -41,7 +41,7 @@ the frontmatter (see [§3](#3-frontmatter-chassis-yaml)). │ │ ├── SKILL.md │ │ ├── references/ (optional) │ │ └── scripts/ (optional) -│ └── knowledge/ user-visible (shared / global, reserved) +│ └── knowledge/ user-visible (shared / global) │ ├── .index/ system-managed, rebuildable (gitignore) │ ├── sqlite/ @@ -186,7 +186,7 @@ Implementation: [`core/persistence/markdown/entries.py`](../src/everos/core/pers - **SQLite** ([`infra/persistence/sqlite/tables/`](../src/everos/infra/persistence/sqlite/tables/)) holds only system / coordination tables — `md_change_state` (cascade queue), `memcell` (boundary ledger), `unprocessed_buffer`, - `conversation_status`, `cluster`, `reflection_report` — **not** + `conversation_status`, `cluster`, `knowledge`, `reflection_report` — **not** per-kind business rows. `reflection_report` is the audit trail for Reflection merges (cluster_id, mode, source_members, merged_entry_id, status).