From a4ca094a7b07eb4592b7f9183ff33efc7da06a18 Mon Sep 17 00:00:00 2001 From: Claudio Drews Date: Mon, 1 Jun 2026 19:23:55 -0300 Subject: [PATCH] =?UTF-8?q?docs:=20add=20ground-truth=20hierarchy=20layer?= =?UTF-8?q?=20(07)=20=E2=80=94=20the=20critical=20missing=20piece?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Context injection delivers memory into the prompt, but without an explicit Ground Truth hierarchy the agent treats it as optional suggestion. This layer documents the fix: injected memory ([qdrant], [fabric], [sessions], [facts]) ranked as authoritative for documented knowledge, with clear conflict-resolution rules against terminal output and training knowledge. --- README.md | 34 ++++++++++++-- layers/07-ground-truth.md | 95 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 125 insertions(+), 4 deletions(-) create mode 100644 layers/07-ground-truth.md diff --git a/README.md b/README.md index f2b7017..c08ea24 100644 --- a/README.md +++ b/README.md @@ -28,7 +28,7 @@ After months of hitting these walls in production, I built something that actual ## What Memory OS is -Not just another plugin. A complete **memory operating system** — 6 layers working in concert, from flat files to a vector database, with surgical context injection and a knowledge pipeline that organizes itself. +Not just another plugin. A complete **memory operating system** — 7 layers working in concert, from flat files to a vector database, with surgical context injection, a knowledge pipeline that organizes itself, **and an explicit Ground Truth hierarchy that tells the agent to actually use the injected memory**. Designed and refined by someone who ran headfirst into every limitation of stock Hermes and every existing memory solution. @@ -37,7 +37,7 @@ Compatible with any LLM provider Hermes supports — OpenRouter, OpenAI, Anthrop --- -## Architecture: 6 memory layers +## Architecture: 7 memory layers ``` ┌──────────────────────────────────────────────────────────────────┐ @@ -67,18 +67,42 @@ Compatible with any LLM provider Hermes supports — OpenRouter, OpenAI, Anthrop │ LAYER 6 · LLM WIKI │ │ Auto-curated vault: concepts/ · entities/ · comparisons/ │ │ → Continuously ingested into Qdrant via wiki-continuous-ingest │ +├──────────────────────────────────────────────────────────────────┤ +│ ⚡ LAYER 7 · GROUND TRUTH HIERARCHY (identity layer) │ +│ SOUL.md · rulebook.md │ +│ → Tells the agent that injected memory is authoritative │ +│ → Without this, layers 2-6 deliver context the agent ignores │ └──────────────────────────────────────────────────────────────────┘ ``` **How it flows:** -`pre_llm_call` → surgical recall from all four sources (Fabric + Qdrant + Sessions + Facts) +`pre_llm_call` → surgical recall from all four sources (Fabric + Qdrant + Sessions + Facts) + +**But recall is not enough.** The agent must be explicitly instructed to treat this injected context as authoritative. That's what [Layer 7](layers/07-ground-truth.md) provides — without it, the agent rediscovers knowledge that's already in the prompt. + `post_llm_call` + `on_session_end` → automatic learning extraction and capture Each source is gated by relevance thresholds. Per-session deduplication prevents the same context from appearing twice. A social-closer filter skips trivial messages entirely. No padding. No firehose. The LLM gets exactly what it needs — nothing more. --- +## Why Layer 7 is the most important layer + +Layers 1-6 ensure memory is **captured, stored, and injected**. Layer 7 ensures the injected memory is **used**. + +Without the Ground Truth hierarchy: +- Qdrant points are injected but the agent calls the Qdrant API to verify them +- Fabric entries are injected but the agent runs `fabric_recall` to re-find them +- Session history is injected but the agent runs `session_search` to re-discover it +- Facts are injected but the agent probes `fact_store` to confirm them + +The result: **memory-zero behavior** despite perfect injection. Every rediscovery burns tokens, context, and time. + +→ **[Read Layer 7: Ground Truth Hierarchy](layers/07-ground-truth.md)** — the critical fix. + +--- + ## Memory OS vs. stock Hermes | Aspect | Stock Hermes | Memory OS | @@ -90,7 +114,8 @@ Each source is gated by relevance thresholds. Per-session deduplication prevents | Vector search | Not present | Qdrant hybrid + 4-level fallback cascade | | Cleanup and deduplication | Not present | Decay scanner + semantic dedup + archival | | Knowledge pipeline | Not present | Self-curating LLM Wiki | -| Token efficiency | — | Surgical: gated retrieval + per-session dedup | +| **Ground Truth hierarchy** | **Not present** | **Injected memory ranked as authoritative; agent must use context provided** | +| Token efficiency | — | Surgical: gated retrieval + per-session dedup + no wasted rediscovery | | Infrastructure | — | Local memory stack (Qdrant + Redis + ARQ) + any LLM provider | --- @@ -108,6 +133,7 @@ Because almost every modern memory solution is **cloud-first**. If you want real | Structured facts + trust scores | ✓ | Partial | ✗ | ✗ | | Self-curating wiki | ✓ | ✗ | ✗ | ✗ | | Intelligent decay + archival | ✓ | ✗ | ✗ | ✗ | +| **Ground Truth hierarchy** | **✓** | **✗** | **✗** | **✗** | --- diff --git a/layers/07-ground-truth.md b/layers/07-ground-truth.md new file mode 100644 index 0000000..3f5f8f8 --- /dev/null +++ b/layers/07-ground-truth.md @@ -0,0 +1,95 @@ +# Layer 7 — Ground Truth Hierarchy + +> **Type:** Identity-layer fix (SOUL.md + rulebook.md) +> **Why it exists:** Context injection is not enough — the agent must be *instructed* to treat injected memory as authoritative. +> **Discovered:** 2026-05-31 + +## The problem + +Memory OS successfully injects context from all four sources (Fabric + Qdrant + Sessions + Facts) into every prompt. You can see it in the system preamble: `[qdrant]`, `[fabric]`, `[sessions]`, `[facts]` blocks are right there. + +**But the agent ignores them.** + +Symptoms: +- Agent runs `search_files`, `read_file`, `session_search` to rediscover information that `[qdrant]` already provided +- Treats every question as novel even when the answer is literally in the prompt +- Rediscovers projects, decisions, and constraints from scratch each session + +## Root cause + +Memory OS was injecting memory into the prompt, but the agent's **identity documents** (`SOUL.md` and `rulebook.md`) did not include injected memory in the Ground Truth hierarchy. Without an explicit rank, the injected context was implicitly treated as optional suggestion — below terminal output and official documentation. + +The original hierarchy had only 3 levels: + +``` +1. Terminal output → Ground Truth +2. Official documentation → Authoritative +3. Training knowledge → Reference only +``` + +The injected memory (`[qdrant]`, `[fabric]`, `[sessions]`, `[facts]`) was **not listed at all**. No status = no authority. + +## The fix + +The hierarchy was expanded to 4 levels, with injected memory inserted as the second level: + +``` +1. Terminal output → Ground Truth for system state (runtime) +2. Injected memory [qdrant, fabric, sessions, facts] → Ground Truth for + documented knowledge and prior decisions +3. Official documentation → Authoritative for APIs, configs, version-specifics +4. Training knowledge → Reference only; always verify against 1-3 +``` + +### Conflict resolution + +| Sources conflict | Winner | +|---|---| +| Terminal vs Injected memory | Terminal wins for system state. Injected memory wins for documented knowledge. | +| Injected memory vs Assumptions | **Injected memory wins.** Never treat a question as novel when the answer is already in your prompt. | +| Injected memory vs Official docs | Official docs win for version-sensitive specifics. Injected memory wins for project context. | +| Training knowledge vs anything | Training knowledge always loses. Verify against 1-3. | + +### Files changed + +| File | Change | +|------|--------| +| `~/.hermes/SOUL.md` | Ground Truth section expanded from 3 to 4 levels; added conflict rules | +| `~/.hermes/rulebook.md` | Added "Injected memory" row to Source of Truth table; added mandatory verification behavior | + +### Key instruction added to SOUL.md + +> *"When injected memory contradicts your assumptions, injected memory wins. Never treat a question as novel when the answer is already in your prompt."* + +## Why this matters + +The infrastructure layers (01-06) ensure memory is captured, stored, and injected. Layer 07 ensures the injected memory is **used**. Without it: + +- Qdrant points are injected but the agent `curl`s the Qdrant API to verify them +- Fabric entries are injected but the agent calls `fabric_recall` to re-find them +- Session history is injected but the agent runs `session_search` to re-discover it +- Facts are injected but the agent probes `fact_store` to confirm them + +Each rediscovery burns tokens, time, and model context. Layer 07 is what stops the waste. + +## Verification + +After applying this fix (updating SOUL.md and rulebook.md), the agent should: + +1. Read injected `[qdrant]`, `[fabric]`, `[sessions]`, `[facts]` blocks before running any search/discovery tools +2. Not rediscover knowledge that is already in the prompt +3. Cite injected context directly instead of re-deriving it +4. Respect the conflict rules when sources disagree + +A gateway restart is required after editing SOUL.md or rulebook.md for changes to take effect in new sessions: + +```bash +systemctl --user restart hermes-gateway +``` + +## Related + +- [Layer 4 — Fabric (injection mechanism)](04-icarus-fabric.md) +- [Layer 5 — Qdrant (vector source)](05-qdrant.md) +- [Layer 3 — Fact Store (structured facts)](03-fact-store.md) +- [Layer 2 — Sessions](02-sessions.md)