docs: add ground-truth hierarchy layer (07) — the critical missing piece
Context injection delivers memory into the prompt, but without an explicit Ground Truth hierarchy the agent treats it as optional suggestion. This layer documents the fix: injected memory ([qdrant], [fabric], [sessions], [facts]) ranked as authoritative for documented knowledge, with clear conflict-resolution rules against terminal output and training knowledge.
This commit is contained in:
Claudio Drews
parent
b7d637a52c
commit
a4ca094a7b
34
README.md
34
README.md
|
|
@ -28,7 +28,7 @@ After months of hitting these walls in production, I built something that actual
|
|||
|
||||
## What Memory OS is
|
||||
|
||||
Not just another plugin. A complete **memory operating system** — 6 layers working in concert, from flat files to a vector database, with surgical context injection and a knowledge pipeline that organizes itself.
|
||||
Not just another plugin. A complete **memory operating system** — 7 layers working in concert, from flat files to a vector database, with surgical context injection, a knowledge pipeline that organizes itself, **and an explicit Ground Truth hierarchy that tells the agent to actually use the injected memory**.
|
||||
|
||||
Designed and refined by someone who ran headfirst into every limitation of stock Hermes and every existing memory solution.
|
||||
|
||||
|
|
@ -37,7 +37,7 @@ Compatible with any LLM provider Hermes supports — OpenRouter, OpenAI, Anthrop
|
|||
|
||||
---
|
||||
|
||||
## Architecture: 6 memory layers
|
||||
## Architecture: 7 memory layers
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────────┐
|
||||
|
|
@ -67,18 +67,42 @@ Compatible with any LLM provider Hermes supports — OpenRouter, OpenAI, Anthrop
|
|||
│ LAYER 6 · LLM WIKI │
|
||||
│ Auto-curated vault: concepts/ · entities/ · comparisons/ │
|
||||
│ → Continuously ingested into Qdrant via wiki-continuous-ingest │
|
||||
├──────────────────────────────────────────────────────────────────┤
|
||||
│ ⚡ LAYER 7 · GROUND TRUTH HIERARCHY (identity layer) │
|
||||
│ SOUL.md · rulebook.md │
|
||||
│ → Tells the agent that injected memory is authoritative │
|
||||
│ → Without this, layers 2-6 deliver context the agent ignores │
|
||||
└──────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**How it flows:**
|
||||
|
||||
`pre_llm_call` → surgical recall from all four sources (Fabric + Qdrant + Sessions + Facts)
|
||||
`pre_llm_call` → surgical recall from all four sources (Fabric + Qdrant + Sessions + Facts)
|
||||
|
||||
**But recall is not enough.** The agent must be explicitly instructed to treat this injected context as authoritative. That's what [Layer 7](layers/07-ground-truth.md) provides — without it, the agent rediscovers knowledge that's already in the prompt.
|
||||
|
||||
`post_llm_call` + `on_session_end` → automatic learning extraction and capture
|
||||
|
||||
Each source is gated by relevance thresholds. Per-session deduplication prevents the same context from appearing twice. A social-closer filter skips trivial messages entirely. No padding. No firehose. The LLM gets exactly what it needs — nothing more.
|
||||
|
||||
---
|
||||
|
||||
## Why Layer 7 is the most important layer
|
||||
|
||||
Layers 1-6 ensure memory is **captured, stored, and injected**. Layer 7 ensures the injected memory is **used**.
|
||||
|
||||
Without the Ground Truth hierarchy:
|
||||
- Qdrant points are injected but the agent calls the Qdrant API to verify them
|
||||
- Fabric entries are injected but the agent runs `fabric_recall` to re-find them
|
||||
- Session history is injected but the agent runs `session_search` to re-discover it
|
||||
- Facts are injected but the agent probes `fact_store` to confirm them
|
||||
|
||||
The result: **memory-zero behavior** despite perfect injection. Every rediscovery burns tokens, context, and time.
|
||||
|
||||
→ **[Read Layer 7: Ground Truth Hierarchy](layers/07-ground-truth.md)** — the critical fix.
|
||||
|
||||
---
|
||||
|
||||
## Memory OS vs. stock Hermes
|
||||
|
||||
| Aspect | Stock Hermes | Memory OS |
|
||||
|
|
@ -90,7 +114,8 @@ Each source is gated by relevance thresholds. Per-session deduplication prevents
|
|||
| Vector search | Not present | Qdrant hybrid + 4-level fallback cascade |
|
||||
| Cleanup and deduplication | Not present | Decay scanner + semantic dedup + archival |
|
||||
| Knowledge pipeline | Not present | Self-curating LLM Wiki |
|
||||
| Token efficiency | — | Surgical: gated retrieval + per-session dedup |
|
||||
| **Ground Truth hierarchy** | **Not present** | **Injected memory ranked as authoritative; agent must use context provided** |
|
||||
| Token efficiency | — | Surgical: gated retrieval + per-session dedup + no wasted rediscovery |
|
||||
| Infrastructure | — | Local memory stack (Qdrant + Redis + ARQ) + any LLM provider |
|
||||
|
||||
---
|
||||
|
|
@ -108,6 +133,7 @@ Because almost every modern memory solution is **cloud-first**. If you want real
|
|||
| Structured facts + trust scores | ✓ | Partial | ✗ | ✗ |
|
||||
| Self-curating wiki | ✓ | ✗ | ✗ | ✗ |
|
||||
| Intelligent decay + archival | ✓ | ✗ | ✗ | ✗ |
|
||||
| **Ground Truth hierarchy** | **✓** | **✗** | **✗** | **✗** |
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,95 @@
|
|||
# Layer 7 — Ground Truth Hierarchy
|
||||
|
||||
> **Type:** Identity-layer fix (SOUL.md + rulebook.md)
|
||||
> **Why it exists:** Context injection is not enough — the agent must be *instructed* to treat injected memory as authoritative.
|
||||
> **Discovered:** 2026-05-31
|
||||
|
||||
## The problem
|
||||
|
||||
Memory OS successfully injects context from all four sources (Fabric + Qdrant + Sessions + Facts) into every prompt. You can see it in the system preamble: `[qdrant]`, `[fabric]`, `[sessions]`, `[facts]` blocks are right there.
|
||||
|
||||
**But the agent ignores them.**
|
||||
|
||||
Symptoms:
|
||||
- Agent runs `search_files`, `read_file`, `session_search` to rediscover information that `[qdrant]` already provided
|
||||
- Treats every question as novel even when the answer is literally in the prompt
|
||||
- Rediscovers projects, decisions, and constraints from scratch each session
|
||||
|
||||
## Root cause
|
||||
|
||||
Memory OS was injecting memory into the prompt, but the agent's **identity documents** (`SOUL.md` and `rulebook.md`) did not include injected memory in the Ground Truth hierarchy. Without an explicit rank, the injected context was implicitly treated as optional suggestion — below terminal output and official documentation.
|
||||
|
||||
The original hierarchy had only 3 levels:
|
||||
|
||||
```
|
||||
1. Terminal output → Ground Truth
|
||||
2. Official documentation → Authoritative
|
||||
3. Training knowledge → Reference only
|
||||
```
|
||||
|
||||
The injected memory (`[qdrant]`, `[fabric]`, `[sessions]`, `[facts]`) was **not listed at all**. No status = no authority.
|
||||
|
||||
## The fix
|
||||
|
||||
The hierarchy was expanded to 4 levels, with injected memory inserted as the second level:
|
||||
|
||||
```
|
||||
1. Terminal output → Ground Truth for system state (runtime)
|
||||
2. Injected memory [qdrant, fabric, sessions, facts] → Ground Truth for
|
||||
documented knowledge and prior decisions
|
||||
3. Official documentation → Authoritative for APIs, configs, version-specifics
|
||||
4. Training knowledge → Reference only; always verify against 1-3
|
||||
```
|
||||
|
||||
### Conflict resolution
|
||||
|
||||
| Sources conflict | Winner |
|
||||
|---|---|
|
||||
| Terminal vs Injected memory | Terminal wins for system state. Injected memory wins for documented knowledge. |
|
||||
| Injected memory vs Assumptions | **Injected memory wins.** Never treat a question as novel when the answer is already in your prompt. |
|
||||
| Injected memory vs Official docs | Official docs win for version-sensitive specifics. Injected memory wins for project context. |
|
||||
| Training knowledge vs anything | Training knowledge always loses. Verify against 1-3. |
|
||||
|
||||
### Files changed
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `~/.hermes/SOUL.md` | Ground Truth section expanded from 3 to 4 levels; added conflict rules |
|
||||
| `~/.hermes/rulebook.md` | Added "Injected memory" row to Source of Truth table; added mandatory verification behavior |
|
||||
|
||||
### Key instruction added to SOUL.md
|
||||
|
||||
> *"When injected memory contradicts your assumptions, injected memory wins. Never treat a question as novel when the answer is already in your prompt."*
|
||||
|
||||
## Why this matters
|
||||
|
||||
The infrastructure layers (01-06) ensure memory is captured, stored, and injected. Layer 07 ensures the injected memory is **used**. Without it:
|
||||
|
||||
- Qdrant points are injected but the agent `curl`s the Qdrant API to verify them
|
||||
- Fabric entries are injected but the agent calls `fabric_recall` to re-find them
|
||||
- Session history is injected but the agent runs `session_search` to re-discover it
|
||||
- Facts are injected but the agent probes `fact_store` to confirm them
|
||||
|
||||
Each rediscovery burns tokens, time, and model context. Layer 07 is what stops the waste.
|
||||
|
||||
## Verification
|
||||
|
||||
After applying this fix (updating SOUL.md and rulebook.md), the agent should:
|
||||
|
||||
1. Read injected `[qdrant]`, `[fabric]`, `[sessions]`, `[facts]` blocks before running any search/discovery tools
|
||||
2. Not rediscover knowledge that is already in the prompt
|
||||
3. Cite injected context directly instead of re-deriving it
|
||||
4. Respect the conflict rules when sources disagree
|
||||
|
||||
A gateway restart is required after editing SOUL.md or rulebook.md for changes to take effect in new sessions:
|
||||
|
||||
```bash
|
||||
systemctl --user restart hermes-gateway
|
||||
```
|
||||
|
||||
## Related
|
||||
|
||||
- [Layer 4 — Fabric (injection mechanism)](04-icarus-fabric.md)
|
||||
- [Layer 5 — Qdrant (vector source)](05-qdrant.md)
|
||||
- [Layer 3 — Fact Store (structured facts)](03-fact-store.md)
|
||||
- [Layer 2 — Sessions](02-sessions.md)
|
||||
Loading…
Reference in New Issue