Initial commit: Memory OS — 6-layer memory architecture for Hermes Agent
This commit is contained in:
ClaudioDrews
commit
0b32ffcfc2
|
|
@ -0,0 +1,93 @@
|
|||
# Memory OS — Environment Variables
|
||||
# Copy this file to .env and fill in your values:
|
||||
# cp .env.example .env
|
||||
|
||||
# ── Required ──────────────────────────────────────────
|
||||
|
||||
# OpenRouter API key (for embeddings and LLM extraction)
|
||||
OPENROUTER_API_KEY=sk-or-...
|
||||
|
||||
# Redis password (generate with: openssl rand -hex 16)
|
||||
REDIS_PASSWORD=change-me
|
||||
|
||||
# ── Paths ─────────────────────────────────────────────
|
||||
|
||||
# Where Icarus writes fabric entries (absolute path required — systemd does not expand ~)
|
||||
FABRIC_DIR=/home/your-user/vault/fabric
|
||||
|
||||
# Vault root for Vault Curator, wiki, and backfill scripts
|
||||
VAULT_PATH=/home/your-user/vault
|
||||
|
||||
# Wiki directory for Qdrant ingestion
|
||||
WIKI_ROOT=/home/your-user/vault/wiki
|
||||
|
||||
# Hermes home (usually ~/.hermes)
|
||||
HERMES_HOME=/home/your-user/.hermes
|
||||
|
||||
# State database path (SQLite with FTS5 for session search)
|
||||
STATE_DB_PATH=/home/your-user/.hermes/state.db
|
||||
|
||||
# Logs directory
|
||||
HERMES_LOGS_DIR=/home/your-user/.hermes/logs
|
||||
|
||||
# DLQ (Dead Letter Queue) state file
|
||||
HERMES_DLQ_PATH=/home/your-user/.hermes/wiki_ingest_failures.json
|
||||
|
||||
# DLQ report output (used by dlq_manager.py)
|
||||
HERMES_DLQ_REPORT_LOG=/home/your-user/.hermes/cron/output/dlq-report.log
|
||||
HERMES_DLQ_REPORT_DIR=/home/your-user/.hermes/cron/output
|
||||
|
||||
# Telemetry log for context_enhancer.py query tracking
|
||||
TELEMETRY_LOG_PATH=/home/your-user/.hermes/logs/query-telemetry.jsonl
|
||||
|
||||
# Reflection trigger log path
|
||||
REFLECTION_LOG_PATH=/home/your-user/.hermes/logs/reflection_trigger.log
|
||||
|
||||
# MaaS env path (reflection_trigger.py loads env from this location)
|
||||
MAA_ENV_PATH=/home/your-user/.env
|
||||
|
||||
# ── Strongly Recommended ──────────────────────────────
|
||||
|
||||
# LLM extraction token limit — 1024 is too small, causes fabric truncation
|
||||
ICARUS_EXTRACTION_MAX_TOKENS=4096
|
||||
|
||||
# LLM extraction model (any OpenRouter chat model)
|
||||
ICARUS_EXTRACTION_MODEL=deepseek/deepseek-v4-flash
|
||||
|
||||
# Embedding dimensions — must match Qdrant collection schema
|
||||
EMBEDDING_DIMS=4096
|
||||
|
||||
# Qdrant collection name
|
||||
COLLECTION_NAME=knowledge_base
|
||||
|
||||
# ── Optional ──────────────────────────────────────────
|
||||
|
||||
# Obsidian integration
|
||||
# ICARUS_OBSIDIAN=1
|
||||
# OBSIDIAN_VAULT_PATH=/home/your-user/vault
|
||||
|
||||
# Fallback truncation limits (only used when LLM extraction fails)
|
||||
# ICARUS_RESULT_MAX_CHARS=500
|
||||
# ICARUS_TASK_MAX_CHARS=300
|
||||
|
||||
# Training/eval (Together AI)
|
||||
# TOGETHER_API_KEY=tok-...
|
||||
|
||||
# Alternative OpenRouter keys (tried in order by context_enhancer.py)
|
||||
# OPENROUTER_FULL_API_KEY=sk-or-...
|
||||
# OPENROUTER_DS_API_KEY=sk-or-...
|
||||
|
||||
# Docker: wiki mount path inside worker container
|
||||
# WIKI_PATH=/wiki
|
||||
|
||||
# Logging
|
||||
# LOG_LEVEL=INFO
|
||||
# CURATOR_LOG_LEVEL=INFO
|
||||
|
||||
# ARQ worker tuning
|
||||
# ARQ_MAX_JOBS=10
|
||||
# ARQ_JOB_TIMEOUT=300
|
||||
# ARQ_KEEP_RESULT=3600
|
||||
|
||||
# Micro-reflection budget
|
||||
# MICRO_REFLECTION_MAX_PER_HOUR=5
|
||||
|
|
@ -0,0 +1,22 @@
|
|||
# Python
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
*.so
|
||||
|
||||
# Environment
|
||||
.env
|
||||
.env.local
|
||||
|
||||
# Docker
|
||||
docker/qdrant_data/
|
||||
docker/redis_data/
|
||||
|
||||
# IDEs
|
||||
.vscode/
|
||||
.idea/
|
||||
*.swp
|
||||
|
||||
# OS
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
|
|
@ -0,0 +1,21 @@
|
|||
MIT License
|
||||
|
||||
Copyright (c) 2026 Claudio Drews
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
|
|
@ -0,0 +1,137 @@
|
|||
# Memory OS — Hermes Agent Memory Operating System
|
||||
|
||||
> **Your agent finally stops forgetting.**
|
||||
> Permanent memory. Local memory infrastructure. API-provider agnostic. Surgically token-efficient.
|
||||
|
||||
Six memory layers. Automatic, intelligent context injection. Structured facts with trust scoring. A self-curating wiki pipeline. Semantic search across **every conversation you've ever had**.
|
||||
|
||||
Memory OS turns Hermes Agent into a real long-term collaborator — one that remembers your projects, your decisions, your reasoning, and brings exactly the right context back at exactly the right moment. Like talking to a colleague who was there for every session.
|
||||
|
||||
**Memory infrastructure runs entirely on your machine. Works with any LLM provider — OpenRouter, OpenAI, Anthropic, Ollama, or local models. No memory subscription. No vendor lock-in.**
|
||||
|
||||
---
|
||||
|
||||
## The problem every serious Hermes user knows
|
||||
|
||||
You spend hours configuring the agent, teaching it your preferences, solving hard problems together — and in the next session it acts like it's meeting you for the first time.
|
||||
|
||||
- Repeating context at the start of every conversation
|
||||
- Losing the thread of important decisions made weeks ago
|
||||
- Structured facts — your stack, your projects, your patterns — with nowhere to live
|
||||
- Every memory solution you've tried is either cloud-locked or too shallow to matter
|
||||
|
||||
After months of hitting these walls in production, I built something that actually works.
|
||||
|
||||
---
|
||||
|
||||
## What Memory OS is
|
||||
|
||||
Not just another plugin. A complete **memory operating system** — 6 layers working in concert, from flat files to a vector database, with surgical context injection and a knowledge pipeline that organizes itself.
|
||||
|
||||
Designed and refined by someone who ran headfirst into every limitation of stock Hermes and every existing memory solution.
|
||||
|
||||
**Requirements:** Hermes Agent + Docker (Qdrant + Redis + ARQ Worker) + Python 3.11+.
|
||||
Compatible with any LLM provider Hermes supports — OpenRouter, OpenAI, Anthropic, Ollama, and more.
|
||||
|
||||
---
|
||||
|
||||
## Architecture: 6 memory layers
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────────┐
|
||||
│ LAYER 1 · WORKSPACE │
|
||||
│ MEMORY.md · USER.md · CREATIVE.md │
|
||||
│ → Injected into the system prompt every single turn │
|
||||
├──────────────────────────────────────────────────────────────────┤
|
||||
│ LAYER 2 · SESSIONS │
|
||||
│ state.db (SQLite + FTS5) │
|
||||
│ → Full-text search across your entire conversation history │
|
||||
├──────────────────────────────────────────────────────────────────┤
|
||||
│ LAYER 3 · STRUCTURED FACTS │
|
||||
│ memory_store.db (SQLite + HRR + FTS5 + trust scoring) │
|
||||
│ → Durable facts with entity resolution and an automatic │
|
||||
│ feedback loop that trains trust scores over time │
|
||||
├──────────────────────────────────────────────────────────────────┤
|
||||
│ LAYER 4 · FABRIC (CROSS-SESSION) │
|
||||
│ Icarus Plugin (heavily forked) │
|
||||
│ → LLM-powered session extraction + multi-source injection │
|
||||
│ → 16 tools: fabric_recall, fabric_write, fabric_brief, etc. │
|
||||
├──────────────────────────────────────────────────────────────────┤
|
||||
│ LAYER 5 · VECTOR DATABASE │
|
||||
│ Qdrant (4096d Cosine + BM25 sparse) │
|
||||
│ → 4-level fallback: hybrid → dense → lexical → SQLite │
|
||||
│ → Weekly decay scanner + semantic dedup (cosine >0.92 → merge) │
|
||||
├──────────────────────────────────────────────────────────────────┤
|
||||
│ LAYER 6 · LLM WIKI │
|
||||
│ Auto-curated vault: concepts/ · entities/ · comparisons/ │
|
||||
│ → Continuously ingested into Qdrant via wiki-continuous-ingest │
|
||||
└──────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**How it flows:**
|
||||
|
||||
`pre_llm_call` → surgical recall from all four sources (Fabric + Qdrant + Sessions + Facts)
|
||||
`post_llm_call` + `on_session_end` → automatic learning extraction and capture
|
||||
|
||||
Each source is gated by relevance thresholds. Per-session deduplication prevents the same context from appearing twice. A social-closer filter skips trivial messages entirely. No padding. No firehose. The LLM gets exactly what it needs — nothing more.
|
||||
|
||||
---
|
||||
|
||||
## Memory OS vs. stock Hermes
|
||||
|
||||
| Aspect | Stock Hermes | Memory OS |
|
||||
|---|---|---|
|
||||
| Workspace memory | MEMORY.md + USER.md | + CREATIVE.md + intelligent injection |
|
||||
| Session memory | Basic state.db | + FTS5 full-text search + session injection |
|
||||
| Structured facts | Not present | Fact store + trust scoring + feedback loop |
|
||||
| Cross-session recall | Limited | Fabric fork + multi-source injection |
|
||||
| Vector search | Not present | Qdrant hybrid + 4-level fallback cascade |
|
||||
| Cleanup and deduplication | Not present | Decay scanner + semantic dedup + archival |
|
||||
| Knowledge pipeline | Not present | Self-curating LLM Wiki |
|
||||
| Token efficiency | — | Surgical: gated retrieval + per-session dedup |
|
||||
| Infrastructure | — | Local memory stack (Qdrant + Redis + ARQ) + any LLM provider |
|
||||
|
||||
---
|
||||
|
||||
## Why not mem0, Zep, Letta, or other providers?
|
||||
|
||||
Because almost every modern memory solution is **cloud-first**. If you want real, private memory infrastructure running on your own machine — with no cloud memory subscription, full provider flexibility, and no data leaving your local stack — none of them deliver what Memory OS delivers.
|
||||
|
||||
| | Memory OS | mem0 | Zep | Letta |
|
||||
|---|---|---|---|---|
|
||||
| Local memory infrastructure | ✓ | ✗ | ✗ | ✗ |
|
||||
| No memory subscription | ✓ | ✗ | ✗ | ✗ |
|
||||
| Provider agnostic (OpenRouter, Ollama…) | ✓ | Partial | Partial | Partial |
|
||||
| Hermes-native | ✓ | ✗ | ✗ | ✗ |
|
||||
| Structured facts + trust scores | ✓ | Partial | ✗ | ✗ |
|
||||
| Self-curating wiki | ✓ | ✗ | ✗ | ✗ |
|
||||
| Intelligent decay + archival | ✓ | ✗ | ✗ | ✗ |
|
||||
|
||||
---
|
||||
|
||||
## Included components
|
||||
|
||||
- **Icarus Plugin (heavily modified fork)** — bundled in `icarus/`
|
||||
The upstream [esaradev/icarus-plugin](https://github.com/esaradev/icarus-plugin) is the base, but this fork is not upstream-compatible. Key additions: LLM-powered session extraction (replaces `text[:500]` truncation), multi-source injection (Qdrant + sessions + facts — upstream is fabric only), CREATIVE.md isolation (fixes `§` delimiter corruption from dual-writer conflict), backtick sanitization, system injection filter, and social closer detection.
|
||||
|
||||
- **Vault Curator v3** — [ClaudioDrews/vault-curator](https://github.com/ClaudioDrews/vault-curator)
|
||||
Frontmatter enrichment, semantic linking, and MOC index generation for the wiki layer.
|
||||
|
||||
---
|
||||
|
||||
## Who this is for
|
||||
|
||||
For people who take Hermes Agent seriously.
|
||||
For people who want an agent that **actually evolves** over time — one that doesn't need the world re-explained every session.
|
||||
For people who value clean engineering, extreme efficiency, and solutions that hold up in real local production.
|
||||
|
||||
If you're like me — tired of amnesiac agents — Memory OS was built for you.
|
||||
|
||||
---
|
||||
|
||||
**Want to see the agent remember for real?**
|
||||
Clone it, run it, feel the difference.
|
||||
|
||||
→ [Setup guide](setup/install.md) · [Layer deep-dives](layers/) · [Infrastructure docs](infrastructure/architecture.md) · [License](LICENSE)
|
||||
|
||||
MIT License · Built with obsession by someone who runs Hermes every single day.
|
||||
|
|
@ -0,0 +1,72 @@
|
|||
# Memory OS — Docker Compose
|
||||
#
|
||||
# Starts Qdrant (vector DB), Redis (job queue), and ARQ Worker (embedding pipeline).
|
||||
# Copy .env.example to .env and fill in required values before running.
|
||||
|
||||
services:
|
||||
redis:
|
||||
image: redis:7-alpine
|
||||
restart: unless-stopped
|
||||
command: >
|
||||
sh -c '
|
||||
mkdir -p /usr/local/etc/redis &&
|
||||
echo "requirepass $${REDIS_PASSWORD}" > /usr/local/etc/redis/redis.conf &&
|
||||
echo "bind 0.0.0.0" >> /usr/local/etc/redis/redis.conf &&
|
||||
echo "appendonly yes" >> /usr/local/etc/redis/redis.conf &&
|
||||
echo "maxmemory 512mb" >> /usr/local/etc/redis/redis.conf &&
|
||||
echo "maxmemory-policy allkeys-lru" >> /usr/local/etc/redis/redis.conf &&
|
||||
redis-server /usr/local/etc/redis/redis.conf
|
||||
'
|
||||
environment:
|
||||
REDIS_PASSWORD: ${REDIS_PASSWORD}
|
||||
ports:
|
||||
- "127.0.0.1:6379:6379"
|
||||
volumes:
|
||||
- redis_data:/data
|
||||
healthcheck:
|
||||
test: ["CMD", "redis-cli", "-a", "${REDIS_PASSWORD}", "ping"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 5
|
||||
|
||||
qdrant:
|
||||
image: qdrant/qdrant:v1.17.1
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "127.0.0.1:6333:6333"
|
||||
volumes:
|
||||
- qdrant_data:/qdrant/storage
|
||||
environment:
|
||||
QDRANT__SERVICE__HTTP_PORT: "6333"
|
||||
QDRANT__SERVICE__API_KEY: ${QDRANT_API_KEY:-}
|
||||
|
||||
worker:
|
||||
build:
|
||||
context: ./worker
|
||||
dockerfile: Dockerfile
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
qdrant:
|
||||
condition: service_started
|
||||
redis:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
REDIS_HOST: redis
|
||||
REDIS_PORT: "6379"
|
||||
REDIS_PASSWORD: ${REDIS_PASSWORD}
|
||||
QDRANT_HOST: qdrant
|
||||
QDRANT_PORT: "6333"
|
||||
OPENROUTER_API_KEY: ${OPENROUTER_API_KEY}
|
||||
EMBEDDING_DIMS: "${EMBEDDING_DIMS:-4096}"
|
||||
COLLECTION_NAME: "${COLLECTION_NAME:-knowledge_base}"
|
||||
ARQ_JOB_TIMEOUT: "300"
|
||||
ARQ_MAX_JOBS: "10"
|
||||
ARQ_KEEP_RESULT: "3600"
|
||||
LOG_LEVEL: ${LOG_LEVEL:-INFO}
|
||||
volumes:
|
||||
- ${WIKI_PATH:-./wiki}:/wiki:ro
|
||||
- ${HERMES_HOME:-~/.hermes}:/hermes:rw
|
||||
|
||||
volumes:
|
||||
redis_data:
|
||||
qdrant_data:
|
||||
|
|
@ -0,0 +1,24 @@
|
|||
FROM python:3.12-slim
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Install system dependencies
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
gcc \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Copy and install Python dependencies
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
# Copy source code
|
||||
COPY . .
|
||||
|
||||
# Default environment variables (override via docker-compose)
|
||||
ENV PYTHONUNBUFFERED=1 \
|
||||
PYTHONDONTWRITEBYTECODE=1
|
||||
|
||||
RUN useradd -r -u 10001 appuser
|
||||
USER appuser
|
||||
|
||||
CMD ["python", "main.py", "--run-worker"]
|
||||
|
|
@ -0,0 +1,113 @@
|
|||
"""
|
||||
Cognitive Worker ARQ — Memory-as-a-Service (MaaS)
|
||||
Main entry point for the processing worker.
|
||||
"""
|
||||
import asyncio
|
||||
import logging
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
# ─── Load .env ──────────────────────────────────────────────────────────────
|
||||
ENV_PATH = Path(__file__).parent.parent / ".env"
|
||||
if ENV_PATH.exists():
|
||||
load_dotenv(ENV_PATH)
|
||||
|
||||
# ─── Logging configuration ──────────────────────────────────────────────────
|
||||
LOG_LEVEL = os.environ.get("LOG_LEVEL", "INFO")
|
||||
logging.basicConfig(
|
||||
level=getattr(logging, LOG_LEVEL.upper()),
|
||||
format="%(asctime)s | %(name)s | %(levelname)s | %(message)s",
|
||||
datefmt="%Y-%m-%d %H:%M:%S",
|
||||
)
|
||||
logger = logging.getLogger("cognitive-worker")
|
||||
|
||||
# ─── Import ARQ worker ──────────────────────────────────────────────────────
|
||||
from arq import create_pool, cron
|
||||
from arq.connections import RedisSettings
|
||||
|
||||
from tasks.ingestion import ingest_memory
|
||||
from tasks.reflection import reflect_on_memories, micro_reflection
|
||||
from tasks.file_ingestion import ingest_file
|
||||
|
||||
# ─── Redis configuration ────────────────────────────────────────────────────
|
||||
REDIS_HOST = os.environ.get("REDIS_HOST", "redis-maas")
|
||||
REDIS_PORT = int(os.environ.get("REDIS_PORT", "6379"))
|
||||
REDIS_PASSWORD = os.environ.get("REDIS_PASSWORD", "")
|
||||
|
||||
redis_settings = RedisSettings(
|
||||
host=REDIS_HOST,
|
||||
port=REDIS_PORT,
|
||||
password=REDIS_PASSWORD or None,
|
||||
)
|
||||
|
||||
# ─── Startup/shutdown functions ────────────────────────────────────────────
|
||||
async def startup(ctx):
|
||||
"""Connect to Qdrant and validate the collection."""
|
||||
from services.local_qdrant import get_qdrant_client, ensure_collection
|
||||
|
||||
logger.info("Worker starting...")
|
||||
ctx["qdrant"] = get_qdrant_client()
|
||||
await ensure_collection(ctx["qdrant"])
|
||||
logger.info("Qdrant connection validated")
|
||||
|
||||
|
||||
async def process_wiki_file(ctx, file_path: str):
|
||||
"""ARQ job: ingests a .md file from the vault."""
|
||||
return await ingest_file(ctx["qdrant"], file_path)
|
||||
|
||||
|
||||
async def shutdown(ctx):
|
||||
"""Clean up connections."""
|
||||
logger.info("Worker shutting down...")
|
||||
if "qdrant" in ctx:
|
||||
await ctx["qdrant"].close()
|
||||
|
||||
|
||||
# ─── ARQ function definitions ────────────────────────────────────────────
|
||||
async def process_ingestion(ctx, memory_text: str, source: str, tags: list = None):
|
||||
"""ARQ job: ingests a memory into the vector store."""
|
||||
return await ingest_memory(ctx["qdrant"], memory_text, source, tags)
|
||||
|
||||
|
||||
async def process_reflection(ctx):
|
||||
"""ARQ job: runs periodic reflection."""
|
||||
return await reflect_on_memories(ctx["qdrant"])
|
||||
|
||||
|
||||
async def process_micro_reflection(ctx):
|
||||
"""ARQ job: runs on-demand micro-reflection (Phase 3)."""
|
||||
return await micro_reflection(ctx["qdrant"])
|
||||
|
||||
|
||||
# ─── ARQ Worker Settings ─────────────────────────────────────────────────
|
||||
class WorkerSettings:
|
||||
"""ARQ worker settings."""
|
||||
redis_settings = redis_settings
|
||||
functions = [process_ingestion, process_reflection, process_micro_reflection, process_wiki_file]
|
||||
on_startup = startup
|
||||
on_shutdown = shutdown
|
||||
max_jobs = int(os.environ.get("ARQ_MAX_JOBS", "10"))
|
||||
job_timeout = int(os.environ.get("ARQ_JOB_TIMEOUT", "300"))
|
||||
keep_result = int(os.environ.get("ARQ_KEEP_RESULT", "3600"))
|
||||
cron_jobs = [
|
||||
# Reflection every 2 hours (minute 0 of even hours)
|
||||
cron(process_reflection, hour={0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22}, minute=0),
|
||||
# Micro-reflection now triggered via reflection_trigger.py (idle + budget)
|
||||
]
|
||||
|
||||
|
||||
# ─── Entry point ────────────────────────────────────────────────────────────
|
||||
if __name__ == "__main__":
|
||||
import sys
|
||||
|
||||
if len(sys.argv) > 1 and sys.argv[1] == "--run-worker":
|
||||
# Start ARQ worker (concurrency via max_jobs, not multi-process)
|
||||
import subprocess
|
||||
logger.info("Starting ARQ worker...")
|
||||
subprocess.run(["arq", "main.WorkerSettings"])
|
||||
else:
|
||||
print("Usage: python main.py --run-worker")
|
||||
print("")
|
||||
print("To enqueue jobs, use the enqueue_host.py script")
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
redis>=5.0.0
|
||||
qdrant-client>=1.17.0
|
||||
httpx>=0.28.0
|
||||
arq>=0.28.0
|
||||
pydantic>=2.12.0
|
||||
python-dotenv>=1.0.0
|
||||
pyyaml>=6.0
|
||||
fastembed>=0.8.0
|
||||
|
|
@ -0,0 +1,59 @@
|
|||
"""
|
||||
Embedding client via OpenRouter.
|
||||
Mandatory dimension validation.
|
||||
"""
|
||||
import os
|
||||
import logging
|
||||
|
||||
import httpx
|
||||
|
||||
logger = logging.getLogger("cognitive-worker.embedding")
|
||||
|
||||
OPENROUTER_API_KEY = os.environ.get("OPENROUTER_API_KEY", "")
|
||||
EMBEDDING_DIMS = int(os.environ.get("EMBEDDING_DIMS", "4096"))
|
||||
EMBEDDING_MODEL = "qwen/qwen3-embedding-8b"
|
||||
API_BASE = "https://openrouter.ai/api/v1"
|
||||
|
||||
|
||||
async def get_embedding(text: str) -> list[float]:
|
||||
"""
|
||||
Generates embedding via OpenRouter.
|
||||
Validates that the returned dimensions match EMBEDDING_DIMS.
|
||||
"""
|
||||
if not OPENROUTER_API_KEY:
|
||||
raise RuntimeError("OPENROUTER_API_KEY is not configured")
|
||||
|
||||
headers = {
|
||||
"Authorization": f"Bearer {OPENROUTER_API_KEY}",
|
||||
"Content-Type": "application/json",
|
||||
"HTTP-Referer": "https://localhost",
|
||||
"X-Title": "Cognitive-Agent-MaaS",
|
||||
}
|
||||
|
||||
payload = {
|
||||
"model": EMBEDDING_MODEL,
|
||||
"input": text,
|
||||
"dimensions": EMBEDDING_DIMS,
|
||||
}
|
||||
|
||||
async with httpx.AsyncClient(timeout=60) as client:
|
||||
resp = await client.post(
|
||||
f"{API_BASE}/embeddings",
|
||||
headers=headers,
|
||||
json=payload,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
|
||||
vec = data["data"][0]["embedding"]
|
||||
|
||||
# ─── Critical dimension validation ──────────────────────────────────────
|
||||
if len(vec) != EMBEDDING_DIMS:
|
||||
raise ValueError(
|
||||
f"Embedding dimension mismatch: "
|
||||
f"expected {EMBEDDING_DIMS}, got {len(vec)}. "
|
||||
f"Check EMBEDDING_DIMS in .env and the Qdrant collection."
|
||||
)
|
||||
|
||||
logger.debug(f"Embedding generated: {len(vec)} dims")
|
||||
return vec
|
||||
|
|
@ -0,0 +1,57 @@
|
|||
"""
|
||||
LLM client via native Ollama.
|
||||
"""
|
||||
import os
|
||||
import logging
|
||||
|
||||
import httpx
|
||||
|
||||
logger = logging.getLogger("cognitive-worker.llm")
|
||||
|
||||
OLLAMA_BASE_URL = os.environ.get("OLLAMA_BASE_URL", "http://host.docker.internal:11434")
|
||||
OLLAMA_MODEL = os.environ.get("OLLAMA_MODEL", "deepseek-v4-flash:cloud")
|
||||
OLLAMA_API_KEY = os.environ.get("OLLAMA_API_KEY", "")
|
||||
|
||||
|
||||
def get_auth_header() -> dict:
|
||||
"""Returns auth header if API key is configured."""
|
||||
if OLLAMA_API_KEY:
|
||||
return {"Authorization": f"Bearer {OLLAMA_API_KEY}"}
|
||||
return {}
|
||||
|
||||
|
||||
async def ollama_chat(prompt: str, model: str | None = None, timeout: int = 120) -> str:
|
||||
"""
|
||||
Sends a prompt to native Ollama and returns the response.
|
||||
Uses cloud models like deepseek-v4-flash:cloud.
|
||||
"""
|
||||
model = model or OLLAMA_MODEL
|
||||
url = f"{OLLAMA_BASE_URL}/api/generate"
|
||||
|
||||
headers = {
|
||||
"Content-Type": "application/json",
|
||||
**get_auth_header(),
|
||||
}
|
||||
|
||||
payload = {
|
||||
"model": model,
|
||||
"prompt": prompt,
|
||||
"stream": False,
|
||||
"options": {
|
||||
"temperature": 0.7,
|
||||
"num_predict": 4096, # DeepSeek generates long reasoning; needs space
|
||||
},
|
||||
}
|
||||
|
||||
async with httpx.AsyncClient(timeout=timeout) as client:
|
||||
resp = await client.post(url, headers=headers, json=payload)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
|
||||
# DeepSeek v4 flash: reasoning can consume tokens, leaving response empty
|
||||
# Return reasoning if content is empty
|
||||
response = data.get("response", "")
|
||||
if not response and "reasoning" in data:
|
||||
response = data["reasoning"]
|
||||
|
||||
return response
|
||||
|
|
@ -0,0 +1,56 @@
|
|||
"""
|
||||
Qdrant client — connection and collection validation.
|
||||
Creates a HYBRID collection (dense + BM25 sparse) if it doesn't exist.
|
||||
"""
|
||||
import os
|
||||
import logging
|
||||
|
||||
from qdrant_client import AsyncQdrantClient
|
||||
from qdrant_client.models import Distance, VectorParams, SparseVectorParams, Modifier
|
||||
|
||||
logger = logging.getLogger("cognitive-worker.qdrant")
|
||||
|
||||
QDRANT_HOST = os.environ.get("QDRANT_HOST", "qdrant-maas")
|
||||
QDRANT_PORT = int(os.environ.get("QDRANT_PORT", "6333"))
|
||||
EMBEDDING_DIMS = int(os.environ.get("EMBEDDING_DIMS", "4096"))
|
||||
COLLECTION_NAME = os.environ.get("COLLECTION_NAME", "knowledge_base")
|
||||
|
||||
_client: AsyncQdrantClient | None = None
|
||||
|
||||
|
||||
def get_qdrant_client() -> AsyncQdrantClient:
|
||||
"""Returns a singleton of the async Qdrant client."""
|
||||
global _client
|
||||
if _client is None:
|
||||
_client = AsyncQdrantClient(host=QDRANT_HOST, port=QDRANT_PORT)
|
||||
logger.info(f"Connected to Qdrant at {QDRANT_HOST}:{QDRANT_PORT}")
|
||||
return _client
|
||||
|
||||
|
||||
async def ensure_collection(client: AsyncQdrantClient) -> None:
|
||||
"""Ensures the hybrid collection exists with dense + sparse configs."""
|
||||
try:
|
||||
collections = (await client.get_collections()).collections
|
||||
names = [c.name for c in collections]
|
||||
if COLLECTION_NAME not in names:
|
||||
logger.info(
|
||||
f"Creating collection {COLLECTION_NAME} with "
|
||||
f"dense={EMBEDDING_DIMS} dims + sparse BM25"
|
||||
)
|
||||
await client.create_collection(
|
||||
collection_name=COLLECTION_NAME,
|
||||
vectors_config={
|
||||
"dense": VectorParams(
|
||||
size=EMBEDDING_DIMS,
|
||||
distance=Distance.COSINE,
|
||||
)
|
||||
},
|
||||
sparse_vectors_config={
|
||||
"sparse": SparseVectorParams(modifier=Modifier.IDF)
|
||||
},
|
||||
)
|
||||
else:
|
||||
logger.info(f"Collection {COLLECTION_NAME} already exists")
|
||||
except Exception as e:
|
||||
logger.error(f"Error validating collection: {e}")
|
||||
raise
|
||||
|
|
@ -0,0 +1,34 @@
|
|||
"""
|
||||
BM25 Sparse Embedding client via FastEmbed.
|
||||
Caches the model in memory (lazy init).
|
||||
"""
|
||||
import logging
|
||||
from fastembed.sparse import SparseTextEmbedding
|
||||
|
||||
logger = logging.getLogger("cognitive-worker.sparse_embedding")
|
||||
|
||||
BM25_MODEL = "Qdrant/bm25"
|
||||
_model = None
|
||||
|
||||
|
||||
def _get_model() -> SparseTextEmbedding:
|
||||
"""Lazy init of the FastEmbed BM25 model."""
|
||||
global _model
|
||||
if _model is None:
|
||||
logger.info("Loading BM25 sparse embedding model...")
|
||||
_model = SparseTextEmbedding(model_name=BM25_MODEL)
|
||||
logger.info("BM25 model loaded.")
|
||||
return _model
|
||||
|
||||
|
||||
def get_sparse_embedding(text: str) -> dict:
|
||||
"""
|
||||
Generates BM25 sparse embedding via FastEmbed.
|
||||
Returns a Qdrant-compatible dict: {"indices": [...], "values": [...]}
|
||||
"""
|
||||
model = _get_model()
|
||||
sparse = list(model.embed(text))[0]
|
||||
return {
|
||||
"indices": sparse.indices.tolist(),
|
||||
"values": sparse.values.tolist(),
|
||||
}
|
||||
|
|
@ -0,0 +1,247 @@
|
|||
"""
|
||||
Tasks — File-based wiki ingestion (Phase B: continuous).
|
||||
Receives a file path to a .md file inside the container (e.g. /wiki/concepts/new.md),
|
||||
extracts frontmatter, generates DENSE + BM25 SPARSE embeddings, upserts into knowledge_base_hybrid.
|
||||
"""
|
||||
import logging
|
||||
import os
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from datetime import datetime, timezone
|
||||
|
||||
import httpx
|
||||
from qdrant_client import AsyncQdrantClient
|
||||
from qdrant_client.models import PointStruct
|
||||
|
||||
from services.embedding import get_embedding
|
||||
from services.sparse_embedding import get_sparse_embedding
|
||||
|
||||
logger = logging.getLogger("cognitive-worker.file_ingest")
|
||||
|
||||
COLLECTION_NAME = os.environ.get("COLLECTION_NAME", "knowledge_base")
|
||||
QDRANT_HOST = os.environ.get("QDRANT_HOST", "qdrant-maas")
|
||||
QDRANT_PORT = int(os.environ.get("QDRANT_PORT", "6333"))
|
||||
WIKI_PATH = os.environ.get("WIKI_PATH", "/wiki")
|
||||
MAX_TEXT_LEN = 8000
|
||||
|
||||
|
||||
def parse_frontmatter(text: str) -> tuple[dict, str]:
|
||||
"""Extracts YAML frontmatter and returns (metadata, body)."""
|
||||
if text.startswith("---"):
|
||||
parts = text.split("---", 2)
|
||||
if len(parts) >= 3:
|
||||
try:
|
||||
import yaml
|
||||
meta = yaml.safe_load(parts[1])
|
||||
body = parts[2].strip()
|
||||
return (meta if isinstance(meta, dict) else {}), body
|
||||
except Exception:
|
||||
pass
|
||||
return {}, text
|
||||
|
||||
|
||||
def get_source_tag(path: Path) -> str:
|
||||
"""Derives a source tag from the path relative to WIKI_PATH."""
|
||||
rel = path.relative_to(WIKI_PATH)
|
||||
parts = rel.parts
|
||||
if len(parts) > 1:
|
||||
return f"wiki-{parts[0]}"
|
||||
return "wiki-root"
|
||||
|
||||
|
||||
def get_tags_from_frontmatter(meta: dict) -> list[str]:
|
||||
"""Extracts tags from frontmatter."""
|
||||
tags = meta.get("tags", [])
|
||||
if isinstance(tags, str):
|
||||
tags = [t.strip() for t in tags.split(",")]
|
||||
return tags if isinstance(tags, list) else []
|
||||
|
||||
|
||||
async def upsert_with_dedup(
|
||||
qdrant: AsyncQdrantClient,
|
||||
collection: str,
|
||||
dense_vector: list,
|
||||
sparse_vector,
|
||||
payload: dict,
|
||||
dedup_threshold: float = 0.92,
|
||||
) -> dict:
|
||||
"""
|
||||
Pre-write dedup: searches for similar neighbors before upserting.
|
||||
If cosine similarity >= threshold, merges payload into the existing point.
|
||||
Returns a dict with status: 'dedup' or 'upserted'.
|
||||
"""
|
||||
try:
|
||||
# Use REST API directly — AsyncQdrantClient doesn't have .search() in qdrant-client 1.18.0
|
||||
async with httpx.AsyncClient(timeout=30) as http:
|
||||
resp = await http.post(
|
||||
f"http://{QDRANT_HOST}:{QDRANT_PORT}/collections/{collection}/points/search",
|
||||
json={
|
||||
"vector": {"name": "dense", "vector": dense_vector},
|
||||
"limit": 10,
|
||||
"with_payload": True,
|
||||
},
|
||||
)
|
||||
resp.raise_for_status()
|
||||
results = resp.json()["result"]
|
||||
for hit in results:
|
||||
hit_score = hit["score"]
|
||||
hit_id = hit["id"]
|
||||
hit_payload = hit.get("payload") or {}
|
||||
if hit_score >= dedup_threshold:
|
||||
existing_payload = hit_payload
|
||||
# Merge: tags (union)
|
||||
existing_tags = set(existing_payload.get("tags", []))
|
||||
new_tags = set(payload.get("tags", []))
|
||||
merged_tags = list(existing_tags | new_tags)
|
||||
# Merge: source_type (priority human > procedural > ai)
|
||||
st_priority = {"human": 3, "procedural": 2, "ai": 1}
|
||||
existing_st = existing_payload.get("source_type", "ai")
|
||||
new_st = payload.get("source_type", "ai")
|
||||
merged_st = existing_st if st_priority.get(existing_st, 0) >= st_priority.get(new_st, 0) else new_st
|
||||
# Merge: last_accessed_at (max)
|
||||
existing_la = existing_payload.get("last_accessed_at", payload.get("created_at"))
|
||||
new_la = payload.get("last_accessed_at")
|
||||
merged_la = max(existing_la, new_la) if existing_la and new_la else (existing_la or new_la)
|
||||
# Merge: importance_score (max)
|
||||
existing_imp = existing_payload.get("importance_score", 0.5)
|
||||
new_imp = payload.get("importance_score", 0.5)
|
||||
merged_imp = max(existing_imp, new_imp)
|
||||
# Merge: lineage_ids
|
||||
existing_lineages = existing_payload.get("lineage_ids", [])
|
||||
new_lineages = payload.get("lineage_ids", [])
|
||||
if not isinstance(existing_lineages, list):
|
||||
existing_lineages = []
|
||||
if not isinstance(new_lineages, list):
|
||||
new_lineages = []
|
||||
merged_lineages = list(set(existing_lineages + new_lineages))
|
||||
# Apply merge
|
||||
await qdrant.set_payload(
|
||||
collection_name=collection,
|
||||
payload={
|
||||
"tags": merged_tags,
|
||||
"source_type": merged_st,
|
||||
"last_accessed_at": merged_la,
|
||||
"importance_score": merged_imp,
|
||||
"lineage_ids": merged_lineages,
|
||||
},
|
||||
points=[hit_id],
|
||||
)
|
||||
logger.info(f"Dedup: merged into chunk {hit_id} (score={hit_score:.3f})")
|
||||
return {
|
||||
"status": "dedup",
|
||||
"existing_id": str(hit_id),
|
||||
"similarity": hit_score,
|
||||
}
|
||||
except Exception as e:
|
||||
logger.warning(f"Error in pre-write dedup: {e}")
|
||||
# Fallback: normal upsert
|
||||
point = PointStruct(
|
||||
id=str(uuid.uuid4()),
|
||||
vector={"dense": dense_vector, "sparse": sparse_vector},
|
||||
payload=payload,
|
||||
)
|
||||
await qdrant.upsert(collection_name=collection, points=[point], wait=True)
|
||||
return {
|
||||
"status": "upserted",
|
||||
"id": point.id,
|
||||
}
|
||||
|
||||
|
||||
async def ingest_file(
|
||||
qdrant: AsyncQdrantClient,
|
||||
file_path: str,
|
||||
) -> dict:
|
||||
"""
|
||||
Ingests a .md file from the vault into knowledge_base_hybrid (dense + BM25 sparse).
|
||||
Returns a dict with id and status.
|
||||
"""
|
||||
wiki_root = Path(WIKI_PATH).resolve()
|
||||
path = Path(file_path).resolve()
|
||||
if not str(path).startswith(str(wiki_root)) or path.suffix != ".md":
|
||||
raise ValueError("file_path must be a .md file under WIKI_PATH")
|
||||
if not path.exists():
|
||||
raise FileNotFoundError(f"File not found: {file_path}")
|
||||
|
||||
text = path.read_text(encoding="utf-8", errors="replace")
|
||||
if not text.strip():
|
||||
return {"status": "skipped", "reason": "empty file", "file": str(path)}
|
||||
|
||||
meta, body = parse_frontmatter(text)
|
||||
source = get_source_tag(path)
|
||||
tags = get_tags_from_frontmatter(meta)
|
||||
folder_tag = source.replace("wiki-", "")
|
||||
if folder_tag not in tags:
|
||||
tags.append(folder_tag)
|
||||
|
||||
title = meta.get("title", path.stem)
|
||||
embed_text = f"{title}\n\n{body}"[:MAX_TEXT_LEN]
|
||||
|
||||
# Generate embeddings
|
||||
try:
|
||||
dense_vector = await get_embedding(embed_text)
|
||||
sparse_vector = get_sparse_embedding(embed_text)
|
||||
except Exception as e:
|
||||
logger.error(f"Error generating embedding for {path}: {e}")
|
||||
raise
|
||||
|
||||
# Payload
|
||||
now = datetime.now(timezone.utc).isoformat()
|
||||
|
||||
# Heuristic importance_score based on file path/name
|
||||
importance_score = 0.5 # default
|
||||
path_str_lower = str(path).lower()
|
||||
if any(k in path_str_lower for k in ["architecture", "core", "important"]):
|
||||
importance_score = 0.7
|
||||
if any(t.lower() in ["important", "critical"] for t in tags):
|
||||
importance_score = 0.8
|
||||
if any(k in path_str_lower for k in ["draft", "temp", "old"]):
|
||||
importance_score = 0.2
|
||||
|
||||
payload = {
|
||||
"text": embed_text,
|
||||
"source": source,
|
||||
"tags": tags,
|
||||
"created_at": now,
|
||||
"reflection_count": 0,
|
||||
"last_reflected": None,
|
||||
"file_path": str(path),
|
||||
"title": title,
|
||||
"word_count": len(embed_text.split()),
|
||||
# ── Lineage fields (Phase 1) ──
|
||||
"lineage_id": None, # legacy: last lineage that generated this chunk
|
||||
"lineage_ids": [], # Phase 3.5: all accumulated lineages (merge)
|
||||
"generation_model": None,
|
||||
"generation_context_hash": None,
|
||||
"retrieved_chunk_ids": None,
|
||||
# ── Decay fields (Phase 2) ──
|
||||
"decay_score": 1.0,
|
||||
"last_accessed_at": now,
|
||||
"importance_score": importance_score,
|
||||
"source_type": "human", # vault files = human origin
|
||||
"confidence_score": 1.0,
|
||||
"archived": False,
|
||||
}
|
||||
|
||||
# Use pre-write dedup instead of direct upsert
|
||||
result = await upsert_with_dedup(
|
||||
qdrant=qdrant,
|
||||
collection=COLLECTION_NAME,
|
||||
dense_vector=dense_vector,
|
||||
sparse_vector=sparse_vector,
|
||||
payload=payload,
|
||||
dedup_threshold=0.92,
|
||||
)
|
||||
|
||||
if result["status"] == "dedup":
|
||||
logger.info(f"File {path.name} deduplicated (merged into {result['existing_id']}) — similarity {result['similarity']:.3f}")
|
||||
else:
|
||||
logger.info(f"File {path.name} ingested ({source}) — dense+sparse")
|
||||
|
||||
return {
|
||||
"id": result.get("id") or result.get("existing_id"),
|
||||
"status": result["status"],
|
||||
"collection": COLLECTION_NAME,
|
||||
"source": source,
|
||||
"file": str(path),
|
||||
"similarity": result.get("similarity"),
|
||||
}
|
||||
|
|
@ -0,0 +1,71 @@
|
|||
"""
|
||||
Tasks — episodic memory ingestion.
|
||||
"""
|
||||
import logging
|
||||
import os
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from qdrant_client import AsyncQdrantClient
|
||||
from qdrant_client.models import PointStruct
|
||||
|
||||
from services.embedding import get_embedding
|
||||
|
||||
logger = logging.getLogger("cognitive-worker.ingestion")
|
||||
|
||||
COLLECTION_NAME = os.environ.get("COLLECTION_NAME", "knowledge_base")
|
||||
|
||||
|
||||
async def ingest_memory(
|
||||
qdrant: AsyncQdrantClient,
|
||||
memory_text: str,
|
||||
source: str,
|
||||
tags: list | None = None,
|
||||
) -> dict:
|
||||
"""
|
||||
Ingests an episodic memory into Qdrant.
|
||||
Returns a dict with id and status.
|
||||
"""
|
||||
if not memory_text or not memory_text.strip():
|
||||
raise ValueError("memory_text cannot be empty")
|
||||
|
||||
tags = tags or []
|
||||
point_id = str(uuid.uuid4())
|
||||
timestamp = datetime.now(timezone.utc).isoformat()
|
||||
|
||||
# Generate embedding
|
||||
try:
|
||||
vector = await get_embedding(memory_text)
|
||||
except Exception as e:
|
||||
logger.error(f"Error generating embedding: {e}")
|
||||
raise
|
||||
|
||||
# Rich payload for search and reflection
|
||||
payload = {
|
||||
"text": memory_text,
|
||||
"source": source,
|
||||
"tags": tags,
|
||||
"created_at": timestamp,
|
||||
"reflection_count": 0,
|
||||
"last_reflected": None,
|
||||
}
|
||||
|
||||
point = PointStruct(
|
||||
id=point_id,
|
||||
vector={"dense": vector},
|
||||
payload=payload,
|
||||
)
|
||||
|
||||
await qdrant.upsert(
|
||||
collection_name=COLLECTION_NAME,
|
||||
points=[point],
|
||||
wait=True,
|
||||
)
|
||||
|
||||
logger.info(f"Memory {point_id[:8]}... ingested ({source})")
|
||||
|
||||
return {
|
||||
"id": point_id,
|
||||
"status": "ingested",
|
||||
"collection": COLLECTION_NAME,
|
||||
}
|
||||
|
|
@ -0,0 +1,374 @@
|
|||
"""
|
||||
Tasks — Reflection Engine v2.
|
||||
Reviews old memories, generates insights, CREATES NEW INDEXABLE POINTS in Qdrant.
|
||||
"""
|
||||
import logging
|
||||
import json
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from qdrant_client import AsyncQdrantClient
|
||||
from qdrant_client.models import PointStruct, Filter, FieldCondition, Range
|
||||
|
||||
# Alias for qdrant_client.models used in Micro Reflection
|
||||
import qdrant_client.models as qmodels
|
||||
|
||||
from services.llm import ollama_chat
|
||||
from services.embedding import get_embedding
|
||||
|
||||
logger = logging.getLogger("cognitive-worker.reflection")
|
||||
|
||||
COLLECTION_NAME = os.environ.get("COLLECTION_NAME", "knowledge_base")
|
||||
REFLECTION_PROMPT = """
|
||||
You are a cognitive memory assistant. Analyze the provided memories and extract:
|
||||
1. Recurring patterns
|
||||
2. Connections between memories
|
||||
3. Insights or learnings
|
||||
4. Suggested actions
|
||||
|
||||
Memories:
|
||||
{memories}
|
||||
|
||||
Respond in JSON with keys: patterns, connections, insights, actions.
|
||||
"""
|
||||
|
||||
|
||||
async def reflect_on_memories(qdrant: AsyncQdrantClient) -> dict:
|
||||
"""
|
||||
Runs a reflection cycle on unreflected or old memories.
|
||||
GENERATES NEW indexable points in Qdrant with the insights.
|
||||
"""
|
||||
# Fetch memories with low reflection_count or old
|
||||
filter_ref = Filter(
|
||||
must=[
|
||||
FieldCondition(
|
||||
key="reflection_count",
|
||||
range=Range(lt=3),
|
||||
),
|
||||
]
|
||||
)
|
||||
|
||||
results = await qdrant.scroll(
|
||||
collection_name=COLLECTION_NAME,
|
||||
scroll_filter=filter_ref,
|
||||
limit=5, # reduced from 20 to avoid LLM timeout
|
||||
with_payload=True,
|
||||
with_vectors=False,
|
||||
)
|
||||
|
||||
points = results[0] # scroll returns (points, next_page_offset)
|
||||
if not points:
|
||||
logger.info("No memories need reflection")
|
||||
return {"status": "no-op", "processed": 0}
|
||||
|
||||
parent_ids = [p.id for p in points]
|
||||
|
||||
# Prepare batch of memories for LLM
|
||||
memories_text = "\n\n".join(
|
||||
f"- [{p.id[:8]}] Source: {p.payload.get('source', '?')} | {p.payload.get('text', '')[:400]}"
|
||||
for p in points
|
||||
)
|
||||
|
||||
prompt = REFLECTION_PROMPT.format(memories=memories_text)
|
||||
|
||||
try:
|
||||
response = await ollama_chat(prompt)
|
||||
reflection_data = json.loads(response)
|
||||
except json.JSONDecodeError:
|
||||
logger.warning("Reflection returned invalid JSON, saving raw")
|
||||
reflection_data = {"raw": response}
|
||||
except Exception as e:
|
||||
logger.error(f"Reflection LLM error: {e}")
|
||||
raise
|
||||
|
||||
# ─── CREATE NEW INDEXABLE POINT with the insight ─────────────────────────
|
||||
# Text for embedding: concatenation of insights
|
||||
insight_text = json.dumps(reflection_data, ensure_ascii=False, indent=2)
|
||||
reflection_vector = await get_embedding(insight_text)
|
||||
|
||||
now = datetime.now(timezone.utc).isoformat()
|
||||
|
||||
reflection_point = PointStruct(
|
||||
id=str(uuid.uuid4()),
|
||||
vector={"dense": reflection_vector},
|
||||
payload={
|
||||
"text": insight_text,
|
||||
"source": "reflection",
|
||||
"tags": ["reflection", "auto-generated", "insight"],
|
||||
"created_at": now,
|
||||
"reflection_count": 0,
|
||||
"last_reflected": None,
|
||||
"parent_ids": parent_ids,
|
||||
"title": f"Reflection batch ({len(points)} memories)",
|
||||
"word_count": len(insight_text.split()),
|
||||
},
|
||||
)
|
||||
|
||||
await qdrant.upsert(
|
||||
collection_name=COLLECTION_NAME,
|
||||
points=[reflection_point],
|
||||
wait=True,
|
||||
)
|
||||
|
||||
logger.info(f"Reflection point created: {reflection_point.id[:8]} (parents: {len(parent_ids)})")
|
||||
|
||||
# Update metadata of processed memories (lifecycle)
|
||||
for point in points:
|
||||
new_count = point.payload.get("reflection_count", 0) + 1
|
||||
await qdrant.set_payload(
|
||||
collection_name=COLLECTION_NAME,
|
||||
payload={
|
||||
"reflection_count": new_count,
|
||||
"last_reflected": now,
|
||||
},
|
||||
points=[point.id],
|
||||
)
|
||||
|
||||
logger.info(f"Reflection completed: {len(points)} memories processed + 1 new indexable point")
|
||||
|
||||
return {
|
||||
"status": "reflected",
|
||||
"processed": len(points),
|
||||
"reflection_point_id": reflection_point.id,
|
||||
"reflection": reflection_data,
|
||||
}
|
||||
|
||||
|
||||
# ─── MICRO REFLECTION (Phase 3) — Consolidation, not cogitation ────────────
|
||||
|
||||
MICRO_REFLECTION_PROMPT = """
|
||||
You are a cognitive memory assistant. Your job is to CONSOLIDATE existing data, never generate new knowledge.
|
||||
|
||||
Analyze the following memory chunk and its similar neighbors. Detect factual contradictions,
|
||||
inconsistencies, or problematic patterns that could reduce the reliability of this chunk.
|
||||
|
||||
Main chunk:
|
||||
{chunk_text}
|
||||
|
||||
Similar neighbors:
|
||||
{neighbors_text}
|
||||
|
||||
Respond in JSON:
|
||||
{{
|
||||
"contradiction_found": true | false,
|
||||
"severity": "low" | "medium" | "high",
|
||||
"explanation": "Concise description of the problem or confirmation of consistency"
|
||||
}}
|
||||
"""
|
||||
|
||||
import sqlite3
|
||||
import os
|
||||
from typing import Optional
|
||||
|
||||
STATE_DB_PATH = os.environ.get("STATE_DB_PATH", "/hermes/state.db")
|
||||
|
||||
|
||||
def get_budget_for_hour(hour_window: str) -> int:
|
||||
"""Returns how many micro-reflections have run in this hour window."""
|
||||
try:
|
||||
conn = sqlite3.connect(STATE_DB_PATH)
|
||||
cursor = conn.cursor()
|
||||
cursor.execute(
|
||||
"SELECT count FROM reflection_budget WHERE hour_window = ?",
|
||||
(hour_window,),
|
||||
)
|
||||
row = cursor.fetchone()
|
||||
conn.close()
|
||||
return row[0] if row else 0
|
||||
except Exception as e:
|
||||
logger.warning(f"Error checking budget: {e}")
|
||||
return 0 # fail-open: if can't check, allow through
|
||||
|
||||
|
||||
def increment_budget(hour_window: str, tokens_used: int = 0):
|
||||
"""Increments the reflection counter for the current hour."""
|
||||
try:
|
||||
conn = sqlite3.connect(STATE_DB_PATH)
|
||||
cursor = conn.cursor()
|
||||
cursor.execute("""
|
||||
INSERT INTO reflection_budget (hour_window, count, tokens_used)
|
||||
VALUES (?, 1, ?)
|
||||
ON CONFLICT(hour_window)
|
||||
DO UPDATE SET count = count + 1, tokens_used = tokens_used + ?
|
||||
""", (hour_window, tokens_used, tokens_used))
|
||||
conn.commit()
|
||||
conn.close()
|
||||
except Exception as e:
|
||||
logger.warning(f"Error incrementing budget: {e}")
|
||||
|
||||
|
||||
async def micro_reflection(qdrant: AsyncQdrantClient) -> dict:
|
||||
"""
|
||||
Micro-reflection: consolidates freshly ingested chunks using LLM.
|
||||
Does NOT create new points. Only updates confidence_score and reflection_notes.
|
||||
"""
|
||||
now = datetime.now(timezone.utc)
|
||||
hour_window = now.strftime("%Y-%m-%dT%H")
|
||||
|
||||
# ── Budget check ──────────────────────────────────────────────────────
|
||||
max_per_hour = int(os.environ.get("MICRO_REFLECTION_MAX_PER_HOUR", "5"))
|
||||
current_count = get_budget_for_hour(hour_window)
|
||||
|
||||
if current_count >= max_per_hour:
|
||||
logger.info(f"Micro-reflection budget exhausted for {hour_window} ({current_count}/{max_per_hour})")
|
||||
return {"status": "budget_exceeded", "processed": 0}
|
||||
|
||||
# ── Select chunks ─────────────────────────────────────────────────────
|
||||
max_chunks = int(os.environ.get("MICRO_REFLECTION_MAX_CHUNKS", "10"))
|
||||
|
||||
filter_chunks = Filter(
|
||||
must=[
|
||||
FieldCondition(
|
||||
key="reflection_count",
|
||||
range=Range(lt=3),
|
||||
),
|
||||
FieldCondition(
|
||||
key="archived",
|
||||
match=qmodels.MatchValue(value=False),
|
||||
),
|
||||
]
|
||||
)
|
||||
|
||||
try:
|
||||
# Order by created_at DESC via scroll (Qdrant has no direct sort; use scroll with limit)
|
||||
results = await qdrant.scroll(
|
||||
collection_name=COLLECTION_NAME,
|
||||
scroll_filter=filter_chunks,
|
||||
limit=max_chunks,
|
||||
with_payload=True,
|
||||
with_vectors=False,
|
||||
)
|
||||
points = results[0]
|
||||
except Exception as e:
|
||||
logger.warning(f"Error fetching chunks for reflection: {e}")
|
||||
return {"status": "error", "error": str(e)}
|
||||
|
||||
if not points:
|
||||
logger.info("No eligible chunks for micro-reflection")
|
||||
return {"status": "no-op", "processed": 0}
|
||||
|
||||
processed = 0
|
||||
contradictions = 0
|
||||
consistencies = 0
|
||||
|
||||
for point in points:
|
||||
chunk_text = point.payload.get("text", "")
|
||||
chunk_id = point.id
|
||||
|
||||
if not chunk_text:
|
||||
continue
|
||||
|
||||
# Fetch similar neighbors via REST API
|
||||
try:
|
||||
import httpx
|
||||
point_data = await qdrant.retrieve(
|
||||
collection_name=COLLECTION_NAME,
|
||||
ids=[chunk_id],
|
||||
with_vectors=True,
|
||||
)
|
||||
if not point_data or not point_data[0].vector:
|
||||
logger.warning(f"Could not get vector for chunk {chunk_id}")
|
||||
continue
|
||||
|
||||
vector = point_data[0].vector.get("dense") if isinstance(point_data[0].vector, dict) else point_data[0].vector
|
||||
|
||||
async with httpx.AsyncClient() as client:
|
||||
qdrant_host = os.environ.get("QDRANT_HOST", "qdrant-maas")
|
||||
qdrant_port = int(os.environ.get("QDRANT_PORT", "6333"))
|
||||
resp = await client.post(
|
||||
f"http://{qdrant_host}:{qdrant_port}/collections/{COLLECTION_NAME}/points/search",
|
||||
json={
|
||||
"vector": {"name": "dense", "vector": vector},
|
||||
"limit": 4,
|
||||
"with_payload": True,
|
||||
},
|
||||
timeout=10,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
neighbors = resp.json().get("result", [])
|
||||
|
||||
# Filter out the chunk itself
|
||||
neighbor_texts = []
|
||||
for n in neighbors:
|
||||
if n["id"] != chunk_id and n.get("payload", {}).get("text"):
|
||||
neighbor_texts.append(f"[{str(n['id'])[:8]}] {n['payload']['text'][:300]}")
|
||||
|
||||
if len(neighbor_texts) < 2:
|
||||
logger.info(f"Chunk {chunk_id[:8]} has too few neighbors, skipping")
|
||||
continue
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"Error fetching neighbors for {chunk_id}: {e}")
|
||||
continue
|
||||
|
||||
# LLM analysis
|
||||
neighbors_text = "\n\n".join(neighbor_texts[:3])
|
||||
prompt = MICRO_REFLECTION_PROMPT.format(
|
||||
chunk_text=chunk_text[:600],
|
||||
neighbors_text=neighbors_text,
|
||||
)
|
||||
|
||||
try:
|
||||
response = await ollama_chat(prompt)
|
||||
# Extract JSON from response
|
||||
import re
|
||||
json_match = re.search(r'\{[^}]*\}', response)
|
||||
if json_match:
|
||||
analysis = json.loads(json_match.group())
|
||||
else:
|
||||
# Try to parse the whole response
|
||||
analysis = json.loads(response.strip())
|
||||
except json.JSONDecodeError:
|
||||
logger.warning(f"LLM response is not valid JSON for chunk {chunk_id[:8]}: {response[:100]}")
|
||||
analysis = {"contradiction_found": False, "severity": "low", "explanation": "Could not analyze"}
|
||||
except Exception as e:
|
||||
logger.warning(f"LLM error for chunk {chunk_id[:8]}: {e}")
|
||||
continue
|
||||
|
||||
# Apply result
|
||||
current_confidence = point.payload.get("confidence_score", 1.0)
|
||||
contradiction_found = analysis.get("contradiction_found", False)
|
||||
severity = analysis.get("severity", "low")
|
||||
explanation = analysis.get("explanation", "")
|
||||
|
||||
if contradiction_found:
|
||||
severity_mult = {"low": 0.05, "medium": 0.1, "high": 0.2}.get(severity, 0.1)
|
||||
new_confidence = max(0.0, current_confidence - severity_mult)
|
||||
contradictions += 1
|
||||
reflection_note = f"[CONTRADICTION {severity.upper()}] {explanation}"
|
||||
else:
|
||||
new_confidence = min(1.0, current_confidence + 0.05)
|
||||
consistencies += 1
|
||||
reflection_note = f"[CONSISTENT] {explanation}"
|
||||
|
||||
# Update payload in Qdrant
|
||||
new_count = point.payload.get("reflection_count", 0) + 1
|
||||
try:
|
||||
await qdrant.set_payload(
|
||||
collection_name=COLLECTION_NAME,
|
||||
payload={
|
||||
"confidence_score": round(new_confidence, 3),
|
||||
"reflection_count": new_count,
|
||||
"last_reflected": now.isoformat(),
|
||||
"reflection_notes": reflection_note,
|
||||
},
|
||||
points=[chunk_id],
|
||||
)
|
||||
processed += 1
|
||||
logger.info(f"Micro-reflection {chunk_id[:8]}: {reflection_note[:80]}")
|
||||
except Exception as e:
|
||||
logger.warning(f"Error updating chunk {chunk_id}: {e}")
|
||||
|
||||
# Increment budget
|
||||
increment_budget(hour_window, tokens_used=0)
|
||||
|
||||
logger.info(f"Micro-reflection completed: {processed} chunks, {contradictions} contradictions, {consistencies} consistent")
|
||||
|
||||
return {
|
||||
"status": "completed",
|
||||
"processed": processed,
|
||||
"contradictions": contradictions,
|
||||
"consistencies": consistencies,
|
||||
"budget_hour": hour_window,
|
||||
"budget_used": current_count + 1,
|
||||
}
|
||||
|
|
@ -0,0 +1,21 @@
|
|||
MIT License
|
||||
|
||||
Copyright (c) 2026 esaradev
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
|
|
@ -0,0 +1,356 @@
|
|||
```diff
|
||||
+ .
|
||||
+ /|\
|
||||
+ / | \
|
||||
+ / | \
|
||||
+ / | \
|
||||
+ / ,--+--, \
|
||||
+ /,' | ',\
|
||||
! // ,--+--, \\
|
||||
! //__/ | \__\\
|
||||
- \ | /
|
||||
- \ __|__ /
|
||||
- \/ \/
|
||||
! '. .'
|
||||
! ____'.'____
|
||||
! / \
|
||||
+ / I C A R U S \
|
||||
+ / \
|
||||
+ '~~~~~~~~~~~~~~~~~~~'
|
||||
```
|
||||
|
||||
> **Self-memory and replacement models for Hermes agents.**
|
||||
>
|
||||
> *Remember your work. Train your replacement.*
|
||||
|
||||
## What this is
|
||||
|
||||
Icarus is a **Hermes plugin**. It runs inside Hermes and gives agents shared memory, training data extraction, and a model replacement pipeline.
|
||||
|
||||
Icarus is **not** an Obsidian plugin. Obsidian is an optional viewer/editor for the markdown files Icarus writes. You don't need Obsidian to use Icarus.
|
||||
|
||||
## What this is not
|
||||
|
||||
- Not an orchestration framework
|
||||
- Not an agent router
|
||||
- Not a dashboard
|
||||
- Not an Obsidian community plugin
|
||||
- Not a standalone app
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────┐
|
||||
│ Hermes Agent │
|
||||
│ ├── Icarus plugin (this repo) │
|
||||
│ │ ├── hooks: auto-capture decisions, inject context │
|
||||
│ │ ├── tools: recall, write, search, train, switch │
|
||||
│ │ └── scoring: session quality, export weighting │
|
||||
│ │ │
|
||||
│ │ writes/reads │
|
||||
│ │ │ │
|
||||
│ ▼ ▼ │
|
||||
│ ~/my-vault/icarus/ (FABRIC_DIR) │
|
||||
│ ├── agent-decision-chose-fastify-abc1.md │
|
||||
│ ├── agent-review-rate-limiter-race-d4e2.md │
|
||||
│ ├── daily/2026-04-01.md (Obsidian daily notes) │
|
||||
│ └── cold/ (archived entries) │
|
||||
│ │
|
||||
│ ~/my-vault/ (OBSIDIAN_VAULT_PATH) │
|
||||
│ └── .obsidian/app.json (vault config) │
|
||||
│ │
|
||||
│ export-training.py ──► together.jsonl ──► Together AI │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ fine-tuned replacement model │
|
||||
└──────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## 2-minute quickstart
|
||||
|
||||
### 1. Install the plugin
|
||||
|
||||
```bash
|
||||
git clone https://github.com/esaradev/icarus-plugin.git
|
||||
mkdir -p ~/.hermes/plugins/icarus
|
||||
cp -r icarus-plugin/* ~/.hermes/plugins/icarus/
|
||||
```
|
||||
|
||||
### 2. Set environment variables
|
||||
|
||||
Add to your Hermes profile `.env` (e.g. `~/.hermes/.env`):
|
||||
|
||||
```bash
|
||||
# required: where Icarus writes notes
|
||||
FABRIC_DIR=~/Documents/my-vault/icarus
|
||||
|
||||
# optional: enable Obsidian wikilinks and daily notes
|
||||
ICARUS_OBSIDIAN=1
|
||||
|
||||
# optional: vault root (if icarus notes are a subfolder)
|
||||
OBSIDIAN_VAULT_PATH=~/Documents/my-vault
|
||||
|
||||
# optional: for training/eval tools
|
||||
TOGETHER_API_KEY=tok-...
|
||||
```
|
||||
|
||||
### 3. Start Hermes and verify
|
||||
|
||||
```bash
|
||||
hermes chat
|
||||
```
|
||||
|
||||
Type `/plugins` to verify:
|
||||
|
||||
```
|
||||
Plugins (1):
|
||||
✓ icarus v0.3.0 (16 tools, 4 hooks)
|
||||
```
|
||||
|
||||
### 4. Initialize Obsidian (optional)
|
||||
|
||||
Inside your Hermes chat, say:
|
||||
|
||||
> Set up Obsidian for my notes
|
||||
|
||||
The agent will call `fabric_init_obsidian`, which creates `.obsidian/app.json` at your vault root and `daily/` inside your notes directory.
|
||||
|
||||
### 5. Write a test note and verify
|
||||
|
||||
Inside Hermes:
|
||||
|
||||
> Write a fabric note about testing the setup
|
||||
|
||||
Then open your vault in Obsidian. You should see:
|
||||
- A new `.md` file in your notes directory with a readable title
|
||||
- A daily note at `daily/2026-04-01.md` with a wikilink to it
|
||||
- YAML frontmatter visible in the note
|
||||
|
||||
## What Icarus adds to Hermes
|
||||
|
||||
Hermes already has per-instance memory and a capable runtime. Icarus adds:
|
||||
|
||||
- **Cross-instance shared memory** -- agents on different profiles read each other's work through a shared `FABRIC_DIR`
|
||||
- **Decision-quality tagging** -- entries carry `training_value` (high/normal/low) so noise doesn't pollute training data
|
||||
- **Training data extraction** -- fabric entries become fine-tuning pairs with quality filtering and pair weighting
|
||||
- **Model replacement pipeline** -- fine-tune a cheaper model from your agent's own history, eval it, switch to it
|
||||
|
||||
## Tools
|
||||
|
||||
### Memory
|
||||
|
||||
| Tool | What it does |
|
||||
|------|-------------|
|
||||
| `fabric_recall` | Ranked retrieval from shared memory |
|
||||
| `fabric_write` | Write entries with linking, evidence, and handoff fields |
|
||||
| `fabric_search` | Keyword grep across all entries |
|
||||
| `fabric_pending` | Show work assigned to this agent |
|
||||
| `fabric_curate` | Set training value (high/normal/low) on an entry |
|
||||
|
||||
### Training
|
||||
|
||||
| Tool | What it does |
|
||||
|------|-------------|
|
||||
| `fabric_export` | Export training pairs. Modes: high-precision, normal, high-volume |
|
||||
| `fabric_train` | Start fine-tune, auto-selects best quality mode with enough pairs |
|
||||
| `fabric_train_status` | Check job progress, updates model registry |
|
||||
|
||||
### Replacement models
|
||||
|
||||
| Tool | What it does |
|
||||
|------|-------------|
|
||||
| `fabric_models` | List all trained models with eval scores |
|
||||
| `fabric_eval` | Compare candidate vs base model on fabric-derived prompts |
|
||||
| `fabric_switch_model` | Activate a replacement model if eval passes threshold |
|
||||
| `fabric_rollback_model` | Emergency rollback to previous model |
|
||||
|
||||
### Operational
|
||||
|
||||
| Tool | What it does |
|
||||
|------|-------------|
|
||||
| `fabric_brief` | Daily brief: pending work, recent activity, suggested action |
|
||||
| `fabric_telemetry` | Recall/usage stats: what gets recalled, what gets used |
|
||||
| `fabric_report` | Corpus health: entries by type, training value, trainable estimate |
|
||||
| `fabric_init_obsidian` | One-time Obsidian vault setup (Hermes tool, not a shell command) |
|
||||
|
||||
## Hooks
|
||||
|
||||
4 automatic hooks fire without the agent calling anything:
|
||||
|
||||
- **on_session_start** -- loads SOUL, pending handoffs, recent context
|
||||
- **pre_llm_call** -- injects relevant memories when the topic changes
|
||||
- **post_llm_call** -- captures high-value decisions (decision + outcome + substantial user request)
|
||||
- **on_session_end** -- scores session quality, writes structured note if threshold met
|
||||
|
||||
## Obsidian setup
|
||||
|
||||
Icarus is a **Hermes plugin**, not an Obsidian plugin. Obsidian just reads the markdown files.
|
||||
|
||||
**How it works:**
|
||||
- `FABRIC_DIR` is where Icarus writes `.md` files (your notes directory)
|
||||
- `OBSIDIAN_VAULT_PATH` is where `.obsidian/` lives (your vault root)
|
||||
- `ICARUS_OBSIDIAN=1` enables wikilinks in note bodies and daily note linking
|
||||
- `fabric_init_obsidian` is a Hermes tool -- call it from inside Hermes, not from the terminal
|
||||
|
||||
**Two setups:**
|
||||
|
||||
Dedicated vault (Icarus IS the vault):
|
||||
```
|
||||
FABRIC_DIR=~/icarus-vault
|
||||
# OBSIDIAN_VAULT_PATH not needed
|
||||
```
|
||||
|
||||
Subfolder in existing vault:
|
||||
```
|
||||
FABRIC_DIR=~/my-vault/icarus-notes
|
||||
OBSIDIAN_VAULT_PATH=~/my-vault
|
||||
```
|
||||
|
||||
## Builder -> reviewer -> fix
|
||||
|
||||
```
|
||||
# builder finishes work, hands off
|
||||
fabric_write(type="code-session", summary="rate limiter ready",
|
||||
status="open", assigned_to="daedalus")
|
||||
|
||||
# reviewer sees it at session start, writes linked review
|
||||
fabric_write(type="review", summary="found race condition",
|
||||
review_of="icarus:a3f29b01")
|
||||
|
||||
# builder sees the review, writes linked fix
|
||||
fabric_write(type="code-session", summary="fixed race condition",
|
||||
revises="icarus:a3f29b01")
|
||||
```
|
||||
|
||||
## Memory -> training -> replacement model
|
||||
|
||||
```
|
||||
1. Work normally. The plugin captures decisions and completions automatically.
|
||||
|
||||
2. Check readiness:
|
||||
fabric_export(mode="high-precision")
|
||||
|
||||
3. Fine-tune:
|
||||
fabric_train(suffix="my-agent-v2")
|
||||
|
||||
4. Check progress:
|
||||
fabric_train_status()
|
||||
|
||||
5. Evaluate:
|
||||
fabric_eval(candidate_model="user/my-agent-v2-abc123")
|
||||
|
||||
6. Switch:
|
||||
fabric_switch_model(model_id="user/my-agent-v2-abc123")
|
||||
```
|
||||
|
||||
## Training value
|
||||
|
||||
Entries carry a `training_value` field: `high`, `normal`, or `low`.
|
||||
|
||||
- **high** -- decisions with outcomes, completed reviews, successful fixes
|
||||
- **normal** -- default for most entries
|
||||
- **low** -- generic session summaries, conversational exchanges
|
||||
|
||||
Export modes:
|
||||
- `high-precision` -- only grounded entries: high-value, verified, linked reviews, structured sessions, or completed entries with evidence
|
||||
- `normal` -- excludes low-value and skips noisy unstructured session notes unless grounded
|
||||
- `high-volume` -- everything
|
||||
|
||||
## Profiles (Hermes v0.6.0)
|
||||
|
||||
```bash
|
||||
hermes profile create coder
|
||||
hermes profile create reviewer --clone
|
||||
mkdir -p ~/.hermes-coder/plugins/icarus ~/.hermes-reviewer/plugins/icarus
|
||||
cp -r icarus-plugin/* ~/.hermes-coder/plugins/icarus/
|
||||
cp -r icarus-plugin/* ~/.hermes-reviewer/plugins/icarus/
|
||||
hermes -p coder chat
|
||||
```
|
||||
|
||||
Both profiles write to the same `FABRIC_DIR`, so the reviewer sees the coder's work.
|
||||
|
||||
## Fallback models
|
||||
|
||||
After switching to a replacement model, set the original as fallback in `config.yaml`:
|
||||
|
||||
```yaml
|
||||
model: user/my-agent-v2-abc123
|
||||
fallback_model:
|
||||
provider: openrouter
|
||||
model: anthropic/claude-sonnet-4
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**"tool not found" when calling fabric_write or fabric_recall**
|
||||
- Run `/plugins` in Hermes. If Icarus isn't listed, the plugin isn't installed in the right directory.
|
||||
- Check: `ls ~/.hermes/plugins/icarus/__init__.py` (global) or `ls ~/.hermes-YOUR_PROFILE/plugins/icarus/__init__.py` (profile-specific Hermes home).
|
||||
- The plugin needs `__init__.py`, `plugin.yaml`, and all `.py` files in the same directory.
|
||||
- If you copied the repo twice, make sure you do **not** have a nested path like `~/.hermes/plugins/icarus/icarus-plugin/__init__.py`.
|
||||
|
||||
**Notes not showing in Obsidian**
|
||||
- Check `FABRIC_DIR` points to a directory inside your Obsidian vault.
|
||||
- Open the vault root (not the notes subdirectory) in Obsidian.
|
||||
- If you set `OBSIDIAN_VAULT_PATH`, make sure `FABRIC_DIR` is inside it.
|
||||
|
||||
**"I pointed FABRIC_DIR at the wrong directory"**
|
||||
- Change `FABRIC_DIR` in your `.env` and restart Hermes. Existing notes stay where they were. Move them manually if needed.
|
||||
|
||||
**".obsidian ended up in the wrong place"**
|
||||
- Delete the misplaced `.obsidian/` directory.
|
||||
- Set `OBSIDIAN_VAULT_PATH` to your actual vault root.
|
||||
- Call `fabric_init_obsidian` again inside Hermes.
|
||||
|
||||
**"I expected an Obsidian plugin"**
|
||||
- Icarus is a Hermes plugin, not an Obsidian community plugin. There is nothing to install in Obsidian. Obsidian reads the markdown files directly -- no plugin needed.
|
||||
|
||||
**Wikilinks not appearing in notes**
|
||||
- Set `ICARUS_OBSIDIAN=1` in your `.env` and restart Hermes. Links are only added when this flag is set.
|
||||
|
||||
## Validation
|
||||
|
||||
After setup, verify everything works:
|
||||
|
||||
```
|
||||
1. In Hermes: "write a test note about validating the setup"
|
||||
2. Check: ls $FABRIC_DIR/*.md (should show a new file)
|
||||
3. Check: ls $FABRIC_DIR/daily/ (should show today's date)
|
||||
4. Open vault in Obsidian: note should appear with frontmatter
|
||||
5. In Hermes: fabric_brief() (should show the note in recent work)
|
||||
```
|
||||
|
||||
## Smoke test
|
||||
|
||||
```bash
|
||||
bash scripts/smoke-handoff.sh
|
||||
bash scripts/test-plugin.sh
|
||||
```
|
||||
|
||||
## Requirements
|
||||
|
||||
- [Hermes](https://github.com/NousResearch/hermes-agent) v0.6.0+
|
||||
- Python 3.10+
|
||||
- `TOGETHER_API_KEY` in `.env` (for training/eval tools)
|
||||
- `FABRIC_DIR` set in `.env` (defaults to `~/fabric/`)
|
||||
|
||||
## Files
|
||||
|
||||
```
|
||||
__init__.py registration (16 tools, 4 hooks)
|
||||
plugin.yaml manifest
|
||||
schemas.py tool schemas (what the LLM sees)
|
||||
tools.py tool handlers
|
||||
hooks.py lifecycle hooks
|
||||
state.py fabric I/O, session scoring, model registry
|
||||
obsidian.py opt-in Obsidian formatting
|
||||
fabric-retrieve.py ranked retrieval with scoring
|
||||
export-training.py training pair extraction with quality filtering
|
||||
scripts/
|
||||
eval-replacement.py model comparison eval
|
||||
smoke-handoff.sh end-to-end handoff proof
|
||||
test-plugin.sh 66-test fixture suite
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
|
|
@ -0,0 +1,89 @@
|
|||
"""
|
||||
Icarus v3 — Self-memory and replacement models for Hermes agents.
|
||||
|
||||
Remember your work. Train your replacement.
|
||||
|
||||
Memory tools:
|
||||
fabric_recall — ranked retrieval from shared fabric
|
||||
fabric_write — write entry with linking, training_value, and handoff fields
|
||||
fabric_search — keyword grep across fabric
|
||||
fabric_pending — work assigned to this agent
|
||||
fabric_curate — set training value (high/normal/low) on an entry
|
||||
|
||||
Training tools:
|
||||
fabric_export — export training pairs with quality filtering (high-precision/normal/high-volume)
|
||||
fabric_train — start Together AI fine-tune, registers model in registry
|
||||
fabric_train_status — check job, update registry on completion
|
||||
|
||||
Replacement-model tools:
|
||||
fabric_models — list all trained models with eval scores
|
||||
fabric_eval — compare candidate vs base model on fabric-derived eval set
|
||||
fabric_switch_model — activate a replacement model if eval passes threshold
|
||||
fabric_rollback_model — emergency rollback to .env.backup
|
||||
|
||||
Daily driver:
|
||||
fabric_brief — operational brief: pending, recent work, suggested action
|
||||
fabric_telemetry — retrieval/usage stats: what gets recalled, what gets used
|
||||
|
||||
Hooks (automatic):
|
||||
on_session_start — loads SOUL, pending handoffs, recent context
|
||||
pre_llm_call — injects relevant memories on topic change
|
||||
post_llm_call — captures high-value decisions (requires outcome indicator)
|
||||
on_session_end — writes best exchange as session entry (skips thin sessions)
|
||||
"""
|
||||
|
||||
import logging
|
||||
|
||||
from . import schemas, tools, hooks
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def register(ctx):
|
||||
# memory
|
||||
ctx.register_tool(name="fabric_recall", toolset="fabric",
|
||||
schema=schemas.FABRIC_RECALL, handler=tools.fabric_recall)
|
||||
ctx.register_tool(name="fabric_write", toolset="fabric",
|
||||
schema=schemas.FABRIC_WRITE, handler=tools.fabric_write)
|
||||
ctx.register_tool(name="fabric_search", toolset="fabric",
|
||||
schema=schemas.FABRIC_SEARCH, handler=tools.fabric_search)
|
||||
ctx.register_tool(name="fabric_pending", toolset="fabric",
|
||||
schema=schemas.FABRIC_PENDING, handler=tools.fabric_pending)
|
||||
ctx.register_tool(name="fabric_curate", toolset="fabric",
|
||||
schema=schemas.FABRIC_CURATE, handler=tools.fabric_curate)
|
||||
|
||||
# training
|
||||
ctx.register_tool(name="fabric_export", toolset="fabric",
|
||||
schema=schemas.FABRIC_EXPORT, handler=tools.fabric_export)
|
||||
ctx.register_tool(name="fabric_train", toolset="fabric",
|
||||
schema=schemas.FABRIC_TRAIN, handler=tools.fabric_train)
|
||||
ctx.register_tool(name="fabric_train_status", toolset="fabric",
|
||||
schema=schemas.FABRIC_TRAIN_STATUS, handler=tools.fabric_train_status)
|
||||
|
||||
# replacement models
|
||||
ctx.register_tool(name="fabric_models", toolset="fabric",
|
||||
schema=schemas.FABRIC_MODELS, handler=tools.fabric_models)
|
||||
ctx.register_tool(name="fabric_eval", toolset="fabric",
|
||||
schema=schemas.FABRIC_EVAL, handler=tools.fabric_eval)
|
||||
ctx.register_tool(name="fabric_switch_model", toolset="fabric",
|
||||
schema=schemas.FABRIC_SWITCH_MODEL, handler=tools.fabric_switch_model)
|
||||
ctx.register_tool(name="fabric_rollback_model", toolset="fabric",
|
||||
schema=schemas.FABRIC_ROLLBACK_MODEL, handler=tools.fabric_rollback_model)
|
||||
|
||||
# daily driver
|
||||
ctx.register_tool(name="fabric_brief", toolset="fabric",
|
||||
schema=schemas.FABRIC_BRIEF, handler=tools.fabric_brief)
|
||||
ctx.register_tool(name="fabric_telemetry", toolset="fabric",
|
||||
schema=schemas.FABRIC_TELEMETRY, handler=tools.fabric_telemetry)
|
||||
ctx.register_tool(name="fabric_init_obsidian", toolset="fabric",
|
||||
schema=schemas.FABRIC_INIT_OBSIDIAN, handler=tools.fabric_init_obsidian)
|
||||
ctx.register_tool(name="fabric_report", toolset="fabric",
|
||||
schema=schemas.FABRIC_REPORT, handler=tools.fabric_report)
|
||||
|
||||
# hooks
|
||||
ctx.register_hook("on_session_start", hooks.on_session_start)
|
||||
ctx.register_hook("pre_llm_call", hooks.pre_llm_call)
|
||||
ctx.register_hook("post_llm_call", hooks.post_llm_call)
|
||||
ctx.register_hook("on_session_end", hooks.on_session_end)
|
||||
|
||||
logger.info("icarus v3 registered (16 tools, 4 hooks)")
|
||||
|
|
@ -0,0 +1,521 @@
|
|||
#!/usr/bin/env python3
|
||||
"""export-training.py -- Extract fine-tuning data from fabric entries.
|
||||
|
||||
Reads ~/fabric/ and generates training pairs in three formats:
|
||||
openai.jsonl -- OpenAI fine-tuning format
|
||||
hf-dataset.jsonl -- Hugging Face dataset format
|
||||
raw-pairs.json -- Raw input/output pairs
|
||||
|
||||
Usage: python3 export-training.py --output ./training-data/
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
FABRIC_DIR = Path(os.environ.get("FABRIC_DIR", Path.home() / "fabric"))
|
||||
|
||||
|
||||
def _truthy(value) -> bool:
|
||||
return str(value).strip().lower() in {"1", "true", "yes", "y", "on"}
|
||||
|
||||
|
||||
def _has_evidence(entry) -> bool:
|
||||
return any(str(entry.get(field, "")).strip() for field in ("evidence", "source_tool", "artifact_paths"))
|
||||
|
||||
|
||||
def _is_structured_session(entry) -> bool:
|
||||
body = str(entry.get("body", ""))
|
||||
return "## Task" in body and "## Result" in body
|
||||
|
||||
|
||||
def _entry_quality(entry) -> dict:
|
||||
verified = _truthy(entry.get("verified", ""))
|
||||
training_value = str(entry.get("training_value", "")).strip()
|
||||
has_evidence = _has_evidence(entry)
|
||||
is_review_linked = entry.get("type") == "review" and bool(entry.get("review_of"))
|
||||
structured_session = entry.get("type") == "session" and _is_structured_session(entry)
|
||||
is_completed = str(entry.get("status", "")).strip() == "completed"
|
||||
return {
|
||||
"verified": verified,
|
||||
"training_value": training_value,
|
||||
"has_evidence": has_evidence,
|
||||
"is_review_linked": is_review_linked,
|
||||
"structured_session": structured_session,
|
||||
"is_completed": is_completed,
|
||||
"is_high_precision": (
|
||||
training_value == "high"
|
||||
or verified
|
||||
or is_review_linked
|
||||
or (structured_session and training_value in ("high", "normal"))
|
||||
or (is_completed and has_evidence)
|
||||
),
|
||||
"is_normal": (
|
||||
training_value != "low"
|
||||
and (
|
||||
entry.get("type") != "session"
|
||||
or structured_session
|
||||
or verified
|
||||
or training_value == "high"
|
||||
or has_evidence
|
||||
)
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def _strip_generated_obsidian_sections(body: str) -> str:
|
||||
body = re.sub(
|
||||
r"\n*<!-- ICARUS_OBSIDIAN_LINKS_START -->.*?<!-- ICARUS_OBSIDIAN_LINKS_END -->\n*",
|
||||
"\n",
|
||||
body,
|
||||
flags=re.DOTALL,
|
||||
)
|
||||
return body.strip()
|
||||
|
||||
|
||||
def parse_entry(filepath):
|
||||
"""Parse a fabric markdown entry into a dict."""
|
||||
text = filepath.read_text(encoding="utf-8")
|
||||
if not text.startswith("---"):
|
||||
return None
|
||||
parts = text.split("---", 2)
|
||||
if len(parts) < 3:
|
||||
return None
|
||||
meta = {}
|
||||
try:
|
||||
import yaml as _yaml
|
||||
meta = _yaml.safe_load(parts[1]) or {}
|
||||
except Exception:
|
||||
lines = parts[1].strip().split("\n")
|
||||
current_key = None
|
||||
for line in lines:
|
||||
stripped = line.strip()
|
||||
if stripped.startswith("- ") and current_key:
|
||||
if not isinstance(meta.get(current_key), list):
|
||||
meta[current_key] = []
|
||||
meta[current_key].append(stripped[2:].strip().strip("\"'"))
|
||||
elif ": " in stripped and not stripped.startswith("-"):
|
||||
k, v = stripped.split(": ", 1)
|
||||
k = k.strip()
|
||||
current_key = k
|
||||
if v.startswith("[") and v.endswith("]"):
|
||||
meta[k] = [x.strip().strip("\"'") for x in v[1:-1].split(",") if x.strip()]
|
||||
elif v.strip():
|
||||
meta[k] = v.strip()
|
||||
else:
|
||||
meta[k] = []
|
||||
elif stripped.endswith(":") and not stripped.startswith("-"):
|
||||
current_key = stripped[:-1].strip()
|
||||
meta[current_key] = []
|
||||
meta["body"] = _strip_generated_obsidian_sections(parts[2])
|
||||
meta["file"] = filepath.name
|
||||
return meta
|
||||
|
||||
|
||||
def scan_all():
|
||||
"""Scan all fabric entries including cold."""
|
||||
entries = []
|
||||
for d in [FABRIC_DIR, FABRIC_DIR / "cold"]:
|
||||
if not d.exists():
|
||||
continue
|
||||
for f in sorted(d.glob("*.md")):
|
||||
e = parse_entry(f)
|
||||
if e:
|
||||
entries.append(e)
|
||||
return entries
|
||||
|
||||
|
||||
def _resolve_ref(ref, entries):
|
||||
"""Resolve a ref string (agent:id) to a specific entry.
|
||||
|
||||
Checks in priority order:
|
||||
1. id field match (exact)
|
||||
2. agent:cycle match
|
||||
3. agent:timestamp substring
|
||||
4. agent:filename substring
|
||||
First match wins. Returns None if unresolvable.
|
||||
"""
|
||||
if ":" not in ref:
|
||||
return None
|
||||
ref_agent, ref_id = ref.split(":", 1)
|
||||
if not ref_agent or not ref_id:
|
||||
return None
|
||||
# 1. Exact id field match
|
||||
for o in entries:
|
||||
if o.get("agent") == ref_agent and str(o.get("id", "")) == str(ref_id):
|
||||
return o
|
||||
# 2. Cycle field match
|
||||
for o in entries:
|
||||
if o.get("agent") == ref_agent and str(o.get("cycle", "")) == str(ref_id):
|
||||
return o
|
||||
# 3. Timestamp substring match
|
||||
for o in entries:
|
||||
if o.get("agent") == ref_agent and str(ref_id) in str(o.get("timestamp", "")):
|
||||
return o
|
||||
# 4. Filename substring match
|
||||
for o in entries:
|
||||
if o.get("agent") == ref_agent and str(ref_id) in o.get("file", ""):
|
||||
return o
|
||||
return None
|
||||
|
||||
|
||||
def estimate_tokens(text):
|
||||
"""Rough token estimate: ~4 chars per token."""
|
||||
return len(text) // 4
|
||||
|
||||
|
||||
def make_pair(user_content, assistant_content, metadata=None):
|
||||
"""Create a training pair dict."""
|
||||
return {
|
||||
"input": user_content,
|
||||
"output": assistant_content,
|
||||
"metadata": metadata or {},
|
||||
}
|
||||
|
||||
|
||||
def _timestamp_sort_key(entry):
|
||||
return str(entry.get("timestamp", ""))
|
||||
|
||||
|
||||
def extract_pairs(entries):
|
||||
"""Extract all training pairs from fabric entries."""
|
||||
pairs = []
|
||||
seen_pairs = set()
|
||||
review_pairs = 0
|
||||
xplat_pairs = 0
|
||||
|
||||
# index entries by agent+cycle for cross-referencing
|
||||
by_ref = {}
|
||||
for e in entries:
|
||||
refs = e.get("refs", [])
|
||||
if isinstance(refs, str):
|
||||
refs = [r.strip() for r in refs.split(",") if r.strip()]
|
||||
agent = e.get("agent", "")
|
||||
by_ref[f"{agent}:{e.get('file', '')}"] = e
|
||||
|
||||
def add_pair(user_msg, output, metadata, dedupe_key=None):
|
||||
nonlocal pairs
|
||||
key = dedupe_key or (metadata.get("type"), user_msg, output)
|
||||
if key in seen_pairs:
|
||||
return False
|
||||
seen_pairs.add(key)
|
||||
pairs.append(make_pair(user_msg, output, metadata))
|
||||
return True
|
||||
|
||||
for e in entries:
|
||||
agent = e.get("agent", "unknown")
|
||||
platform = e.get("platform", "unknown")
|
||||
entry_type = e.get("type", "")
|
||||
body = e.get("body", "")
|
||||
summary = e.get("summary", "")
|
||||
|
||||
quality = _entry_quality(e)
|
||||
tv = quality["training_value"]
|
||||
verified = quality["verified"]
|
||||
has_evidence = quality["has_evidence"]
|
||||
|
||||
if not body or len(body) < 20:
|
||||
continue
|
||||
|
||||
# base metadata shared by all pairs from this entry
|
||||
base_meta = {
|
||||
"agent": agent,
|
||||
"platform": platform,
|
||||
"training_value": tv,
|
||||
"verified": verified,
|
||||
"has_evidence": has_evidence,
|
||||
"entry_type": entry_type,
|
||||
}
|
||||
|
||||
# ── OUTCOME PAIR: focused summary → outcome ──
|
||||
if e.get("outcome"):
|
||||
add_pair(f"[outcome] {summary}", e["outcome"], {**base_meta, "type": "outcome"})
|
||||
|
||||
# ── BASIC PAIR: type as task, body as response ──
|
||||
if entry_type in ("code-session", "task", "resolution", "research"):
|
||||
user_msg = f"[{entry_type}] {summary}" if summary else f"Complete this {entry_type}"
|
||||
add_pair(user_msg, body, {**base_meta, "type": "basic"})
|
||||
|
||||
elif entry_type == "dialogue":
|
||||
user_msg = f"[dialogue] Respond as {agent} in a multi-agent conversation."
|
||||
add_pair(user_msg, body, {**base_meta, "type": "dialogue"})
|
||||
|
||||
elif entry_type == "decision":
|
||||
user_msg = f"[decision] What did you decide?"
|
||||
add_pair(user_msg, body, {**base_meta, "type": "decision"})
|
||||
|
||||
elif entry_type == "session":
|
||||
# structured session: extract task->result pair if present
|
||||
if "## Task" in body and "## Result" in body:
|
||||
task_match = re.search(r"## Task\n(.+?)(?=\n## |\Z)", body, re.DOTALL)
|
||||
result_match = re.search(r"## Result\n(.+?)(?=\n## |\Z)", body, re.DOTALL)
|
||||
if task_match and result_match:
|
||||
add_pair(
|
||||
f"[session-task] {task_match.group(1).strip()[:300]}",
|
||||
result_match.group(1).strip()[:500],
|
||||
{**base_meta, "type": "session-structured"},
|
||||
)
|
||||
# generic session pairs are much noisier; only keep when grounded
|
||||
if verified or tv == "high" or has_evidence:
|
||||
user_msg = f"[session] Summarize what was accomplished."
|
||||
add_pair(user_msg, body, {**base_meta, "type": "session"})
|
||||
|
||||
elif entry_type == "review":
|
||||
user_msg = f"[review] Review the following code or work."
|
||||
add_pair(user_msg, body, {**base_meta, "type": "review"})
|
||||
|
||||
else:
|
||||
user_msg = f"[{entry_type or 'task'}] {summary or 'Complete this task'}"
|
||||
add_pair(user_msg, body, {"type": "basic", "agent": agent, "platform": platform})
|
||||
|
||||
# ── Parse refs ──
|
||||
refs = e.get("refs", [])
|
||||
if isinstance(refs, str):
|
||||
refs = [r.strip() for r in refs.split(",") if r.strip()]
|
||||
|
||||
# ── REVIEW PAIRS: only pair explicitly linked entries ──
|
||||
if entry_type == "review" and refs:
|
||||
for ref in refs:
|
||||
orig = _resolve_ref(ref, entries)
|
||||
if not orig:
|
||||
continue
|
||||
# Find revision: must explicitly ref back to the review or original
|
||||
review_file = e.get("file", "")
|
||||
orig_file = orig.get("file", "")
|
||||
ref_agent = ref.split(":")[0] if ":" in ref else ""
|
||||
candidates = []
|
||||
for candidate in entries:
|
||||
if candidate.get("agent") != ref_agent:
|
||||
continue
|
||||
if str(candidate.get("timestamp", "")) <= str(e.get("timestamp", "")):
|
||||
continue
|
||||
if candidate.get("file") == orig_file:
|
||||
continue
|
||||
# candidate must ref back to the review or original
|
||||
cand_refs = candidate.get("refs", [])
|
||||
if isinstance(cand_refs, str):
|
||||
cand_refs = [r.strip() for r in cand_refs.split(",") if r.strip()]
|
||||
refs_back = False
|
||||
for cr in cand_refs:
|
||||
resolved = _resolve_ref(cr, [e, orig])
|
||||
if resolved:
|
||||
refs_back = True
|
||||
break
|
||||
if refs_back:
|
||||
candidates.append(candidate)
|
||||
if not candidates:
|
||||
continue
|
||||
improved = max(candidates, key=_timestamp_sort_key)
|
||||
user_msg = f"[self-correct] Original work:\n{orig.get('body', '')[:300]}\n\nReview feedback:\n{body[:300]}\n\nProvide the improved version."
|
||||
if add_pair(
|
||||
user_msg,
|
||||
improved.get("body", ""),
|
||||
{"type": "review-correction", "reviewer": agent, "author": ref_agent},
|
||||
dedupe_key=("review-correction", e.get("file", ""), orig.get("file", ""), improved.get("file", "")),
|
||||
):
|
||||
review_pairs += 1
|
||||
|
||||
# ── REVIEW PAIRS via review_of/revises (v3 plugin path) ──
|
||||
if entry_type == "review" and e.get("review_of"):
|
||||
ref = e["review_of"]
|
||||
orig = _resolve_ref(ref, entries)
|
||||
if orig:
|
||||
ref_agent = ref.split(":")[0] if ":" in ref else ""
|
||||
candidates = [
|
||||
candidate for candidate in entries
|
||||
if candidate.get("revises") == ref
|
||||
]
|
||||
if candidates:
|
||||
improved = max(candidates, key=_timestamp_sort_key)
|
||||
rc_msg = f"[self-correct] Original work:\n{orig.get('body', '')[:300]}\n\nReview feedback:\n{body[:300]}\n\nProvide the improved version."
|
||||
if add_pair(
|
||||
rc_msg,
|
||||
improved.get("body", ""),
|
||||
{"type": "review-correction", "reviewer": agent, "author": ref_agent, "training_value": tv},
|
||||
dedupe_key=("review-correction", e.get("file", ""), orig.get("file", ""), improved.get("file", "")),
|
||||
):
|
||||
review_pairs += 1
|
||||
|
||||
# ── CROSS-PLATFORM via review_of ──
|
||||
if e.get("review_of") and platform:
|
||||
source = _resolve_ref(e["review_of"], entries)
|
||||
if source:
|
||||
src_plat = source.get("platform", "")
|
||||
if src_plat and src_plat != platform:
|
||||
user_msg = f"[cross-platform context] Memory from {src_plat}:\n{source.get('body', '')[:300]}\n\nYou are on {platform}. Use this context in your response."
|
||||
if add_pair(
|
||||
user_msg,
|
||||
body,
|
||||
{"type": "cross-platform", "source_platform": src_plat, "target_platform": platform, "agent": agent, "training_value": tv},
|
||||
dedupe_key=("cross-platform", source.get("file", ""), e.get("file", ""), platform),
|
||||
):
|
||||
xplat_pairs += 1
|
||||
|
||||
# ── CROSS-PLATFORM PAIRS: resolve ref to specific entry ──
|
||||
if refs and platform:
|
||||
for ref in refs:
|
||||
source = _resolve_ref(ref, entries)
|
||||
if not source:
|
||||
continue
|
||||
src_plat = source.get("platform", "")
|
||||
if not src_plat or src_plat == platform:
|
||||
continue # same platform, not cross-platform
|
||||
user_msg = f"[cross-platform context] Memory from {src_plat}:\n{source.get('body', '')[:300]}\n\nYou are on {platform}. Use this context in your response."
|
||||
if add_pair(
|
||||
user_msg,
|
||||
body,
|
||||
{"type": "cross-platform", "source_platform": src_plat, "target_platform": platform, "agent": agent},
|
||||
dedupe_key=("cross-platform", source.get("file", ""), e.get("file", ""), platform),
|
||||
):
|
||||
xplat_pairs += 1
|
||||
|
||||
return pairs, review_pairs, xplat_pairs
|
||||
|
||||
|
||||
def to_openai(pair):
|
||||
"""Convert to OpenAI fine-tuning format."""
|
||||
return {"messages": [
|
||||
{"role": "user", "content": pair["input"]},
|
||||
{"role": "assistant", "content": pair["output"]},
|
||||
]}
|
||||
|
||||
|
||||
def to_together(pair):
|
||||
"""Convert to Together AI fine-tuning format (messages with system prompt)."""
|
||||
return {"messages": [
|
||||
{"role": "system", "content": "You are a helpful AI agent with shared memory across platforms."},
|
||||
{"role": "user", "content": pair["input"]},
|
||||
{"role": "assistant", "content": pair["output"]},
|
||||
]}
|
||||
|
||||
|
||||
def to_hf(pair):
|
||||
"""Convert to Hugging Face dataset format."""
|
||||
return {
|
||||
"instruction": pair["input"],
|
||||
"output": pair["output"],
|
||||
"metadata": pair.get("metadata", {}),
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Export fabric entries as fine-tuning data")
|
||||
parser.add_argument("--output", default="./training-data", help="Output directory")
|
||||
parser.add_argument("--fabric-dir", default=None, help="Fabric directory (default: ~/fabric/)")
|
||||
parser.add_argument("--mode", choices=["high-precision", "normal", "high-volume"],
|
||||
default="normal", help="Export quality mode")
|
||||
args = parser.parse_args()
|
||||
|
||||
global FABRIC_DIR
|
||||
if args.fabric_dir:
|
||||
FABRIC_DIR = Path(args.fabric_dir)
|
||||
|
||||
if not FABRIC_DIR.exists():
|
||||
print(f"error: {FABRIC_DIR} does not exist")
|
||||
sys.exit(1)
|
||||
|
||||
all_entries = scan_all()
|
||||
if not all_entries:
|
||||
print("no fabric entries found")
|
||||
sys.exit(0)
|
||||
|
||||
# filter by mode
|
||||
excluded = 0
|
||||
if args.mode == "high-precision":
|
||||
entries = [e for e in all_entries if _entry_quality(e)["is_high_precision"]]
|
||||
excluded = len(all_entries) - len(entries)
|
||||
elif args.mode == "normal":
|
||||
entries = [e for e in all_entries if _entry_quality(e)["is_normal"]]
|
||||
excluded = len(all_entries) - len(entries)
|
||||
else:
|
||||
entries = all_entries
|
||||
|
||||
pairs, review_count, xplat_count = extract_pairs(entries)
|
||||
|
||||
if not pairs:
|
||||
print("no training pairs extracted")
|
||||
sys.exit(0)
|
||||
|
||||
# weight high-value pairs (verified + cross-agent + structured get extra boost)
|
||||
weighted = []
|
||||
for p in pairs:
|
||||
meta = p.get("metadata", {})
|
||||
ptype = meta.get("type", "")
|
||||
tv = meta.get("training_value", "")
|
||||
is_verified = meta.get("verified", False)
|
||||
has_evidence = meta.get("has_evidence", False)
|
||||
author = meta.get("author", "")
|
||||
reviewer = meta.get("reviewer", "")
|
||||
is_cross_agent = bool(author and reviewer and author != reviewer)
|
||||
if ptype == "review-correction":
|
||||
if is_cross_agent and is_verified:
|
||||
weighted.extend([p] * 5)
|
||||
elif is_cross_agent or is_verified:
|
||||
weighted.extend([p] * 4)
|
||||
else:
|
||||
weighted.extend([p] * 3)
|
||||
elif ptype == "session-structured" and tv == "high":
|
||||
weighted.extend([p] * 3)
|
||||
elif tv == "high":
|
||||
weighted.extend([p] * (4 if (is_verified and has_evidence) else 3 if is_verified else 2))
|
||||
elif has_evidence and is_verified:
|
||||
weighted.extend([p] * 3)
|
||||
elif has_evidence:
|
||||
weighted.extend([p] * 2)
|
||||
elif is_verified:
|
||||
weighted.extend([p] * 2)
|
||||
else:
|
||||
weighted.append(p)
|
||||
pairs = weighted
|
||||
|
||||
# Write outputs
|
||||
out = Path(args.output)
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# OpenAI format
|
||||
with open(out / "openai.jsonl", "w") as f:
|
||||
for p in pairs:
|
||||
f.write(json.dumps(to_openai(p)) + "\n")
|
||||
|
||||
# Together AI format (Llama instruct template)
|
||||
with open(out / "together.jsonl", "w") as f:
|
||||
for p in pairs:
|
||||
f.write(json.dumps(to_together(p)) + "\n")
|
||||
|
||||
# HuggingFace format
|
||||
with open(out / "hf-dataset.jsonl", "w") as f:
|
||||
for p in pairs:
|
||||
f.write(json.dumps(to_hf(p)) + "\n")
|
||||
|
||||
# Raw pairs
|
||||
with open(out / "raw-pairs.json", "w") as f:
|
||||
json.dump(pairs, f, indent=2)
|
||||
|
||||
# Stats
|
||||
total_tokens = sum(estimate_tokens(p["input"] + p["output"]) for p in pairs)
|
||||
type_counts = {}
|
||||
for p in pairs:
|
||||
t = p.get("metadata", {}).get("type", "unknown")
|
||||
type_counts[t] = type_counts.get(t, 0) + 1
|
||||
|
||||
print(f"exported to {out}/")
|
||||
print(f" total pairs: {len(pairs)}")
|
||||
print(f" review pairs: {review_count}")
|
||||
print(f" cross-platform: {xplat_count}")
|
||||
print(f" estimated tokens: {total_tokens:,}")
|
||||
print(f" source entries: {len(entries)} selected / {len(all_entries)} total ({excluded} excluded)")
|
||||
print(f" by type:")
|
||||
for t, c in sorted(type_counts.items()):
|
||||
print(f" {t}: {c}")
|
||||
print(f" files:")
|
||||
print(f" {out}/openai.jsonl")
|
||||
print(f" {out}/together.jsonl")
|
||||
print(f" {out}/hf-dataset.jsonl")
|
||||
print(f" {out}/raw-pairs.json")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
@ -0,0 +1,391 @@
|
|||
#!/usr/bin/env python3
|
||||
"""fabric-retrieve.py -- Smart retrieval for fabric entries.
|
||||
|
||||
Given a query, returns the top N most relevant entries ranked by:
|
||||
keyword match, project match, agent match, recency, tier, type match, ref chain.
|
||||
|
||||
Usage:
|
||||
python3 fabric-retrieve.py "billing issue" --max-results 5 --max-tokens 2000
|
||||
python3 fabric-retrieve.py "auth module" --agent icarus --project myapp
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
FABRIC_DIR = Path(os.environ.get("FABRIC_DIR", Path.home() / "fabric"))
|
||||
|
||||
|
||||
def _strip_generated_obsidian_sections(body: str) -> str:
|
||||
body = re.sub(
|
||||
r"\n*<!-- ICARUS_OBSIDIAN_LINKS_START -->.*?<!-- ICARUS_OBSIDIAN_LINKS_END -->\n*",
|
||||
"\n",
|
||||
body,
|
||||
flags=re.DOTALL,
|
||||
)
|
||||
return body.strip()
|
||||
|
||||
STOP_WORDS = {"the", "a", "an", "is", "was", "are", "were", "be", "been", "being",
|
||||
"have", "has", "had", "do", "does", "did", "will", "would", "could",
|
||||
"should", "may", "might", "shall", "can", "to", "of", "in", "for",
|
||||
"on", "with", "at", "by", "from", "as", "into", "through", "during",
|
||||
"it", "its", "this", "that", "and", "or", "but", "not", "no", "if",
|
||||
"then", "than", "so", "up", "out", "about", "what", "which", "who",
|
||||
"how", "when", "where", "why", "i", "me", "my", "we", "our", "you"}
|
||||
|
||||
CODE_WORDS = {"function", "bug", "error", "build", "deploy", "test", "code", "fix",
|
||||
"commit", "merge", "api", "endpoint", "module", "class", "method",
|
||||
"refactor", "debug", "compile", "runtime", "exception", "stack"}
|
||||
|
||||
CUSTOMER_WORDS = {"customer", "billing", "support", "ticket", "refund", "account",
|
||||
"subscription", "payment", "invoice", "complaint", "resolution",
|
||||
"escalation", "onboarding", "churn", "retention"}
|
||||
|
||||
HANDOFF_WORDS = {"handoff", "review", "reviewer", "pickup", "pending", "relay",
|
||||
"assigned", "assignee", "revise", "revision", "feedback"}
|
||||
|
||||
|
||||
def parse_entry(filepath):
|
||||
text = filepath.read_text(encoding="utf-8")
|
||||
if not text.startswith("---"):
|
||||
return None
|
||||
parts = text.split("---", 2)
|
||||
if len(parts) < 3:
|
||||
return None
|
||||
meta = {}
|
||||
try:
|
||||
import yaml
|
||||
meta = yaml.safe_load(parts[1]) or {}
|
||||
except Exception:
|
||||
current_key = None
|
||||
for line in parts[1].strip().split("\n"):
|
||||
stripped = line.strip()
|
||||
if stripped.startswith("- ") and current_key:
|
||||
if not isinstance(meta.get(current_key), list):
|
||||
meta[current_key] = []
|
||||
meta[current_key].append(stripped[2:].strip().strip("\"'"))
|
||||
elif ": " in stripped and not stripped.startswith("-"):
|
||||
k, v = stripped.split(": ", 1)
|
||||
k = k.strip()
|
||||
current_key = k
|
||||
if v.startswith("[") and v.endswith("]"):
|
||||
meta[k] = [x.strip().strip("\"'") for x in v[1:-1].split(",") if x.strip()]
|
||||
elif v.strip():
|
||||
meta[k] = v.strip()
|
||||
else:
|
||||
meta[k] = []
|
||||
elif stripped.endswith(":") and not stripped.startswith("-"):
|
||||
current_key = stripped[:-1].strip()
|
||||
meta[current_key] = []
|
||||
meta["_body"] = _strip_generated_obsidian_sections(parts[2])
|
||||
meta["_file"] = filepath.name
|
||||
meta["_full"] = text
|
||||
return meta
|
||||
|
||||
|
||||
def tokenize(text):
|
||||
words = set(re.findall(r'[a-z0-9]+', text.lower()))
|
||||
return words - STOP_WORDS
|
||||
|
||||
|
||||
def _ngrams(tokens, n):
|
||||
if len(tokens) < n:
|
||||
return set()
|
||||
return {tuple(tokens[i:i + n]) for i in range(len(tokens) - n + 1)}
|
||||
|
||||
|
||||
def age_hours(timestamp_str):
|
||||
if not timestamp_str:
|
||||
return 9999
|
||||
try:
|
||||
ts = datetime.fromisoformat(str(timestamp_str).replace("Z", "+00:00"))
|
||||
delta = datetime.now(timezone.utc) - ts
|
||||
return max(0, delta.total_seconds() / 3600)
|
||||
except (ValueError, AttributeError):
|
||||
return 9999
|
||||
|
||||
|
||||
def _as_list(value):
|
||||
if not value:
|
||||
return []
|
||||
if isinstance(value, list):
|
||||
return value
|
||||
return [value]
|
||||
|
||||
|
||||
def score_entry(entry, query_tokens, agent=None, project=None, relevant_refs=None):
|
||||
summary = str(entry.get("summary", ""))
|
||||
summary_lower = summary.lower()
|
||||
body = (entry.get("_body", "") + " " + summary).lower()
|
||||
entry_tokens = tokenize(body)
|
||||
summary_tokens = tokenize(summary_lower)
|
||||
entry_type = entry.get("type", "")
|
||||
query_text = " ".join(re.findall(r"[a-z0-9]+", " ".join(sorted(query_tokens))))
|
||||
body_text = " ".join(re.findall(r"[a-z0-9]+", body))
|
||||
query_seq = re.findall(r"[a-z0-9]+", query_text)
|
||||
body_seq = re.findall(r"[a-z0-9]+", body)
|
||||
|
||||
score = 0.0
|
||||
|
||||
# 1. Keyword match (body + summary)
|
||||
keyword_hits = len(query_tokens & entry_tokens)
|
||||
score += keyword_hits * 5 # keywords are the primary signal
|
||||
|
||||
# 1a. Summary match is higher-signal than body text alone.
|
||||
summary_hits = len(query_tokens & summary_tokens)
|
||||
score += summary_hits * 3
|
||||
|
||||
# 1b. Exact phrase and n-gram matches beat loose token overlap.
|
||||
raw_query = " ".join(re.findall(r"[a-z0-9]+", entry.get("_query", "")))
|
||||
if raw_query and raw_query in body_text:
|
||||
score += 18
|
||||
if query_seq:
|
||||
body_bigrams = _ngrams(body_seq, 2)
|
||||
query_bigrams = _ngrams(query_seq, 2)
|
||||
score += len(query_bigrams & body_bigrams) * 4
|
||||
body_trigrams = _ngrams(body_seq, 3)
|
||||
query_trigrams = _ngrams(query_seq, 3)
|
||||
score += len(query_trigrams & body_trigrams) * 7
|
||||
|
||||
# 1b. Tag match (tags are high-signal metadata)
|
||||
entry_tags = _as_list(entry.get("tags", []))
|
||||
tag_tokens = set()
|
||||
for t in entry_tags:
|
||||
tag_tokens.update(re.findall(r'[a-z0-9]+', str(t).lower()))
|
||||
tag_hits = len(query_tokens & tag_tokens)
|
||||
score += tag_hits * 4
|
||||
|
||||
# 2. Same project (check project_id field first, then fallback to keyword match)
|
||||
entry_project = entry.get("project_id", entry.get("project", ""))
|
||||
if project:
|
||||
project_lower = project.lower()
|
||||
if project_lower == str(entry_project).lower():
|
||||
score += 10 # exact project_id match
|
||||
elif project_lower in body:
|
||||
score += 8 # project name in body text
|
||||
elif any(project_lower in str(t).lower() for t in entry_tags):
|
||||
score += 8 # project name in tags
|
||||
|
||||
# 3. Same agent
|
||||
if agent and entry.get("agent") == agent:
|
||||
score += 5
|
||||
|
||||
# 4. Recency (secondary signal, should not override keyword match)
|
||||
hours = age_hours(entry.get("timestamp"))
|
||||
if hours < 1:
|
||||
score += 4
|
||||
elif hours < 24:
|
||||
score += 3
|
||||
elif hours < 168: # 1 week
|
||||
score += 2
|
||||
elif hours < 720: # 1 month
|
||||
score += 1
|
||||
|
||||
# 5. Tier boost (light touch)
|
||||
tier = entry.get("tier", "")
|
||||
if tier == "hot":
|
||||
score += 2
|
||||
elif tier == "warm":
|
||||
score += 1
|
||||
|
||||
# 6. Type match
|
||||
if query_tokens & CODE_WORDS:
|
||||
if entry_type in ("code-session", "review", "decision"):
|
||||
score += 5
|
||||
if query_tokens & CUSTOMER_WORDS:
|
||||
if entry_type in ("resolution", "task", "decision"):
|
||||
score += 5
|
||||
if query_tokens & HANDOFF_WORDS:
|
||||
if entry_type in ("task", "review", "resolution", "code-session"):
|
||||
score += 6
|
||||
elif entry_type == "session":
|
||||
score -= 3
|
||||
|
||||
# Prefer source work artifacts over generic summaries.
|
||||
if entry_type in ("task", "review", "resolution", "code-session", "research"):
|
||||
score += 2
|
||||
elif entry_type == "decision":
|
||||
score += 1
|
||||
elif entry_type == "session":
|
||||
score -= 2
|
||||
|
||||
# Structured workflow fields are high-signal for handoffs and follow-ups.
|
||||
if entry.get("status") == "open":
|
||||
score += 4
|
||||
if entry.get("assigned_to"):
|
||||
score += 2
|
||||
|
||||
# Reviews reference the original work's keywords, which inflates their
|
||||
# keyword score. When the query isn't asking for reviews/feedback, penalize
|
||||
# type=review so the source entry ranks higher.
|
||||
REVIEW_QUERY_WORDS = {"review", "reviewed", "feedback", "issue", "fix", "must", "should", "approve", "reject", "lgtm"}
|
||||
if entry_type == "review":
|
||||
if query_tokens & REVIEW_QUERY_WORDS:
|
||||
# query IS about reviews — boost linking fields
|
||||
if entry.get("review_of"):
|
||||
score += 5
|
||||
else:
|
||||
# query is about the work itself — reviews are secondary
|
||||
score -= 4
|
||||
elif entry.get("review_of"):
|
||||
score += 3
|
||||
if entry.get("revises"):
|
||||
score += 4
|
||||
|
||||
# 7. Ref chain
|
||||
if relevant_refs:
|
||||
entry_refs = _as_list(entry.get("refs", []))
|
||||
entry_id = entry.get("id", "")
|
||||
entry_agent = entry.get("agent", "")
|
||||
entry_cycle = str(entry.get("cycle", ""))
|
||||
for ref in relevant_refs:
|
||||
if ref in entry_refs:
|
||||
score += 3
|
||||
# Check if this entry is referenced by a relevant entry
|
||||
for ref_str in relevant_refs:
|
||||
if ":" in ref_str:
|
||||
ref_agent, ref_id = ref_str.split(":", 1)
|
||||
if ref_agent == entry_agent and (ref_id == entry_id or ref_id == entry_cycle):
|
||||
score += 3
|
||||
linked_refs = [entry.get("review_of"), entry.get("revises")]
|
||||
for ref in linked_refs:
|
||||
if ref and ref in relevant_refs:
|
||||
score += 5
|
||||
|
||||
return score
|
||||
|
||||
|
||||
def deduplicate(entries):
|
||||
seen = {}
|
||||
result = []
|
||||
for e in entries:
|
||||
key = (e.get("agent", ""), e.get("type", ""), e.get("_body", "")[:50])
|
||||
existing = seen.get(key)
|
||||
if existing:
|
||||
# Keep the newer one
|
||||
if str(e.get("timestamp", "")) > str(existing.get("timestamp", "")):
|
||||
result.remove(existing)
|
||||
result.append(e)
|
||||
seen[key] = e
|
||||
else:
|
||||
seen[key] = e
|
||||
result.append(e)
|
||||
return result
|
||||
|
||||
|
||||
def retrieve(query, max_results=5, max_tokens=2000, agent=None, project=None):
|
||||
if not FABRIC_DIR.exists():
|
||||
return []
|
||||
|
||||
# Scan all entries
|
||||
entries = []
|
||||
for d in [FABRIC_DIR, FABRIC_DIR / "cold"]:
|
||||
if not d.exists():
|
||||
continue
|
||||
for f in d.glob("*.md"):
|
||||
e = parse_entry(f)
|
||||
if e:
|
||||
entries.append(e)
|
||||
|
||||
if not entries:
|
||||
return []
|
||||
|
||||
query_tokens = tokenize(query)
|
||||
query_words = re.findall(r"[a-z0-9]+", query.lower())
|
||||
normalized_query = " ".join(query_words)
|
||||
for e in entries:
|
||||
e["_query"] = normalized_query
|
||||
|
||||
# First pass: score without ref chain
|
||||
scored = [(score_entry(e, query_tokens, agent, project), e) for e in entries]
|
||||
|
||||
# Collect refs from top-scoring entries for ref chain boost
|
||||
scored.sort(key=lambda x: x[0], reverse=True)
|
||||
top_refs = set()
|
||||
for score, e in scored[:10]:
|
||||
if score > 0:
|
||||
refs = e.get("refs", [])
|
||||
if isinstance(refs, str):
|
||||
refs = [refs]
|
||||
top_refs.update(refs)
|
||||
eid = e.get("id", "")
|
||||
eagent = e.get("agent", "")
|
||||
if eid:
|
||||
top_refs.add(f"{eagent}:{eid}")
|
||||
|
||||
# Second pass: rescore with ref chain
|
||||
if top_refs:
|
||||
scored = [(score_entry(e, query_tokens, agent, project, top_refs), e) for e in entries]
|
||||
scored.sort(key=lambda x: x[0], reverse=True)
|
||||
|
||||
# Filter zero scores
|
||||
scored = [(s, e) for s, e in scored if s > 0]
|
||||
|
||||
# Deduplicate
|
||||
deduped_entries = deduplicate([e for _, e in scored])
|
||||
# Reattach scores
|
||||
score_map = {id(e): s for s, e in scored}
|
||||
# Rebuild scored list preserving dedup order
|
||||
final = []
|
||||
for e in deduped_entries:
|
||||
# Find the score for this entry
|
||||
for s, orig in scored:
|
||||
if orig is e:
|
||||
final.append((s, e))
|
||||
break
|
||||
final.sort(key=lambda x: x[0], reverse=True)
|
||||
|
||||
# Apply max_results
|
||||
final = final[:max_results]
|
||||
|
||||
# Apply token budget (always enforced, including first entry)
|
||||
budget = max_tokens
|
||||
result = []
|
||||
for score, e in final:
|
||||
content = e.get("_full", "")
|
||||
tokens = len(content) // 4
|
||||
if tokens > budget:
|
||||
continue # skip oversized entries, try next
|
||||
budget -= tokens
|
||||
result.append((score, e))
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def format_results(results):
|
||||
lines = []
|
||||
for score, e in results:
|
||||
lines.append(f"# relevance: {score:.0f} | {e.get('agent','')} | {e.get('platform','')} | {e.get('type','')}")
|
||||
lines.append(e.get("_full", "").strip())
|
||||
lines.append("---")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Smart fabric retrieval")
|
||||
parser.add_argument("query", help="Search query or current task description")
|
||||
parser.add_argument("--max-results", type=int, default=5, help="Maximum entries to return")
|
||||
parser.add_argument("--max-tokens", type=int, default=2000, help="Token budget (chars/4)")
|
||||
parser.add_argument("--agent", default=None, help="Boost entries from this agent")
|
||||
parser.add_argument("--project", default=None, help="Boost entries from this project")
|
||||
parser.add_argument("--fabric-dir", default=None, help="Override fabric directory")
|
||||
args = parser.parse_args()
|
||||
|
||||
global FABRIC_DIR
|
||||
if args.fabric_dir:
|
||||
FABRIC_DIR = Path(args.fabric_dir)
|
||||
|
||||
results = retrieve(args.query, args.max_results, args.max_tokens, args.agent, args.project)
|
||||
|
||||
if not results:
|
||||
print("no relevant entries found")
|
||||
sys.exit(0)
|
||||
|
||||
print(format_results(results))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
@ -0,0 +1,776 @@
|
|||
"""Lifecycle hooks — memory capture, decision detection, creative tracking."""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import urllib.request
|
||||
import urllib.error
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
from . import state
|
||||
|
||||
# ── LLM extraction key ──
|
||||
_OPENROUTER_KEY = (
|
||||
os.environ.get("OPENROUTER_FULL_API_KEY", "")
|
||||
or os.environ.get("OPENROUTER_DS_API_KEY", "")
|
||||
or os.environ.get("OPENROUTER_API_KEY", "")
|
||||
)
|
||||
_EXTRACTION_MODEL = os.environ.get("ICARUS_EXTRACTION_MODEL", "deepseek/deepseek-v4-flash")
|
||||
_EXTRACTION_MAX_TOKENS = int(os.environ.get("ICARUS_EXTRACTION_MAX_TOKENS", "1024"))
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# ── Truncation limits (env-configurable) ──
|
||||
_RESULT_MAX = int(os.environ.get("ICARUS_RESULT_MAX_CHARS", "500"))
|
||||
_TASK_MAX = int(os.environ.get("ICARUS_TASK_MAX_CHARS", "300"))
|
||||
|
||||
# ── System injection detection ──
|
||||
_SYSTEM_PREFIXES = (
|
||||
"[IMPORTANT:",
|
||||
"[SYSTEM:",
|
||||
"You are running as a scheduled",
|
||||
)
|
||||
|
||||
|
||||
def _is_system_injection(text):
|
||||
"""Return True if text starts with a known orchestrator/system preamble."""
|
||||
stripped = text.strip()
|
||||
return any(stripped.startswith(p) for p in _SYSTEM_PREFIXES)
|
||||
|
||||
# use shared regexes from state for decision/outcome/completion detection
|
||||
# keep local regexes only for creative tracking (broader set)
|
||||
_THEME_RE = re.compile(
|
||||
r"(?i)\b(decided|resolved|completed|fixed|deployed|shipped|reviewed|approved|rejected|built|created)\b"
|
||||
)
|
||||
_EVAL_RE = re.compile(
|
||||
r"(?i)\b(worked well|didn't work|failed|succeeded|learned|noticed|realized|discovered|finding|insight|improvement)\b"
|
||||
)
|
||||
_QUESTION_RE = re.compile(
|
||||
r"(?i)\b(what if|wonder|curious about|want to try|experiment with|explore|investigate|test whether)\b"
|
||||
)
|
||||
_STOPWORDS = frozenset(
|
||||
"this that with from have been were will about would could should their there "
|
||||
"these them then when what which some other more also just like very into only "
|
||||
"than over such make made most each does done being".split()
|
||||
)
|
||||
|
||||
# ── Topic overlap tracking ──
|
||||
_last_query_tokens: set = set()
|
||||
|
||||
# ── Per-session injection dedup (reset on session start) ──
|
||||
_injected_fabric: set = set()
|
||||
_injected_qdrant: set = set()
|
||||
_injected_sessions: set = set()
|
||||
|
||||
|
||||
def _tokenize(text):
|
||||
words = set(re.findall(r"[a-z0-9]+", text.lower()))
|
||||
return words - {"the", "a", "an", "is", "was", "are", "to", "of", "in", "for",
|
||||
"on", "with", "it", "and", "or", "not", "i", "you", "can", "do",
|
||||
"this", "that", "what", "how", "please", "help", "me", "my"}
|
||||
|
||||
|
||||
def _extract_theme(text):
|
||||
words = re.findall(r"\b[a-z]{4,}\b", text.lower())
|
||||
filtered = [w for w in words[:30] if w not in _STOPWORDS][:3]
|
||||
return " ".join(filtered) if filtered else ""
|
||||
|
||||
|
||||
def _sanitize_learning(s: str) -> str:
|
||||
"""Remove unpaired backticks that would produce orphaned markdown."""
|
||||
if s.count('`') % 2 != 0:
|
||||
s = s.replace('`', '')
|
||||
if s.count('```') % 2 != 0:
|
||||
s = s.replace('```', '')
|
||||
return s.strip()
|
||||
|
||||
|
||||
def _extract_sentence(text, pattern):
|
||||
for s in re.split(r"[.!?\n]+", text):
|
||||
s = s.strip()
|
||||
if len(s) > 15 and pattern.search(s):
|
||||
return s[:120]
|
||||
return ""
|
||||
|
||||
|
||||
# ── Hooks ────────────────────────────────────────────────
|
||||
|
||||
def on_session_start(session_id="", platform="", **kwargs):
|
||||
"""Load context: SOUL + pending handoffs + recent entries + creative state."""
|
||||
global _last_query_tokens
|
||||
_last_query_tokens = set()
|
||||
_injected_fabric.clear()
|
||||
_injected_qdrant.clear()
|
||||
_injected_sessions.clear()
|
||||
state.session_id = session_id
|
||||
state.exchanges = []
|
||||
state._recall_log = []
|
||||
|
||||
creative = state.load_creative()
|
||||
creative["cycle"] += 1
|
||||
state.save_creative(creative)
|
||||
|
||||
parts = []
|
||||
|
||||
soul = state.load_soul()
|
||||
if soul:
|
||||
parts.append(soul.strip())
|
||||
|
||||
# pending work (handoff-aware)
|
||||
open_tasks, reviews, open_tickets = state.read_pending()
|
||||
if open_tasks:
|
||||
parts.append(f"[fabric] {len(open_tasks)} item(s) assigned to you:")
|
||||
for t in open_tasks[:5]:
|
||||
src = t.get("agent", "?")
|
||||
entry_id = t.get("id", "?")
|
||||
etype = t.get("type", "task")
|
||||
parts.append(f" - {src}: {t.get('summary', '?')} ({etype}, id {entry_id})")
|
||||
parts.append(" If reviewing, set review_of. If revising, set revises. Otherwise just complete the work.")
|
||||
|
||||
if reviews:
|
||||
parts.append(f"[fabric] {len(reviews)} review(s) of your work:")
|
||||
for r in reviews[:5]:
|
||||
reviewer = r.get("agent", "?")
|
||||
entry_id = r.get("id", "?")
|
||||
ref = r.get("review_of", "")
|
||||
parts.append(f" - {reviewer}: {r.get('summary', '?')} (review id {entry_id}, of {ref})")
|
||||
parts.append(" When you fix the issues, set revises to your original entry's agent:id.")
|
||||
|
||||
if open_tickets:
|
||||
parts.append(f"[fabric] {len(open_tickets)} ticket(s) assigned to you:")
|
||||
for t in open_tickets[:5]:
|
||||
cid = t.get("customer_id", "?")
|
||||
src = t.get("agent", "?")
|
||||
entry_id = t.get("id", "?")
|
||||
parts.append(f" - [{cid}] {t.get('summary', '?')} (from {src}, id {entry_id})")
|
||||
parts.append(" Carry customer_id forward when you resolve these.")
|
||||
|
||||
# cross-agent feedback (non-pending items)
|
||||
if not open_tasks and not reviews:
|
||||
feedback = state.read_cross_agent(3)
|
||||
if feedback:
|
||||
parts.append("[fabric] from other agents:")
|
||||
for f in feedback:
|
||||
parts.append(f" {f}")
|
||||
|
||||
# recent entries
|
||||
entries = state.read_recent(limit=5)
|
||||
if entries:
|
||||
parts.append("[fabric] recent activity:")
|
||||
for e in entries:
|
||||
ts = e["timestamp"][:16] if e["timestamp"] else "?"
|
||||
parts.append(f" [{ts}] {e['agent']}: {e['summary']}")
|
||||
|
||||
# creative state
|
||||
if creative["questions"]:
|
||||
parts.append(f"[fabric] open questions: {'; '.join(creative['questions'][-3:])}")
|
||||
if creative["learnings"]:
|
||||
parts.append(f"[fabric] learnings: {'; '.join(creative['learnings'][-3:])}")
|
||||
|
||||
context = "\n".join(parts)
|
||||
return {"context": context} if context else None
|
||||
|
||||
|
||||
# ── Qdrant context injection ──────────────────────────────
|
||||
|
||||
_SOCIAL_CLOSERS = frozenset({
|
||||
"ok", "obrigado", "valeu", "beleza", "blz", "tks", "thanks",
|
||||
"👍", "👌", "✅", "feito", "certo", "confirmo", "entendido",
|
||||
"certo", "isso", "sim", "não", "claro", "perfeito", "ótimo"
|
||||
})
|
||||
|
||||
|
||||
def _is_social_close(text):
|
||||
"""Return True if message is a social closer that shouldn't trigger search."""
|
||||
stripped = text.strip().lower()
|
||||
if stripped in _SOCIAL_CLOSERS:
|
||||
return True
|
||||
# Very short ASCII-only without technical markers
|
||||
if len(stripped) < 6 and stripped.isascii() and not any(
|
||||
c in stripped for c in "://.@#$_?"
|
||||
):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _search_qdrant(query, top_k=2, threshold=0.72):
|
||||
"""Search Qdrant knowledge_base via context_enhancer pipeline.
|
||||
|
||||
Returns list of result dicts with keys: id, score, title,
|
||||
content_preview, source, tags.
|
||||
Returns empty list on any failure (fail-open).
|
||||
"""
|
||||
try:
|
||||
# context_enhancer looks for OPENROUTER_API_KEY (singular);
|
||||
# inject our resolved key into environ for compatibility
|
||||
if _OPENROUTER_KEY and not os.environ.get("OPENROUTER_API_KEY"):
|
||||
os.environ["OPENROUTER_API_KEY"] = _OPENROUTER_KEY
|
||||
|
||||
from scripts.context_enhancer import (
|
||||
embed_query, embed_query_sparse, search_with_fallback
|
||||
)
|
||||
dense = embed_query(query)
|
||||
sparse = embed_query_sparse(query)
|
||||
results, _level, _qdrant_ms, _fallback_ms = search_with_fallback(
|
||||
dense_vector=dense,
|
||||
sparse_vector=sparse,
|
||||
query_text=query,
|
||||
top_k=top_k,
|
||||
score_threshold=threshold,
|
||||
)
|
||||
return results
|
||||
except Exception:
|
||||
return []
|
||||
|
||||
|
||||
# ── Session history search (FTS5 over state.db) ──────────────
|
||||
|
||||
|
||||
def _resolve_state_db():
|
||||
"""Locate the Hermes session DB. Prefer state.HERMES_HOME, fall back to ~/.hermes."""
|
||||
import sqlite3 # noqa: F401 (ensure available)
|
||||
candidates = []
|
||||
home = getattr(state, "HERMES_HOME", None)
|
||||
if home:
|
||||
candidates.append(Path(home) / "state.db")
|
||||
candidates.append(Path.home() / ".hermes" / "state.db")
|
||||
for c in candidates:
|
||||
if c and c.exists():
|
||||
return c
|
||||
return None
|
||||
|
||||
|
||||
def _search_sessions(query, current_session_id="", top_k=2):
|
||||
"""FTS5 search over prior session messages in state.db.
|
||||
|
||||
Returns list of {session_id, title, when, snippet}, excluding the
|
||||
current session. Fail-open: returns [] on any error.
|
||||
"""
|
||||
import sqlite3
|
||||
db = _resolve_state_db()
|
||||
if not db:
|
||||
return []
|
||||
|
||||
# Build an FTS5 OR-query from meaningful tokens (avoids AND over-filtering)
|
||||
toks = [t for t in _tokenize(query) if len(t) >= 4]
|
||||
if not toks:
|
||||
return []
|
||||
fts_query = " OR ".join(toks[:8])
|
||||
|
||||
try:
|
||||
con = sqlite3.connect(f"file:{db}?mode=ro", uri=True)
|
||||
con.row_factory = sqlite3.Row
|
||||
cur = con.cursor()
|
||||
# NOTE: snippet() cannot be combined with GROUP BY in the same SELECT
|
||||
# (FTS5 raises "unable to use function snippet in the requested
|
||||
# context"). Fetch top-ranked rows and dedup by session in Python.
|
||||
rows = cur.execute(
|
||||
"""
|
||||
SELECT m.session_id AS session_id,
|
||||
s.title AS title,
|
||||
s.started_at AS started_at,
|
||||
snippet(messages_fts, 0, '', '', '…', 12) AS snip
|
||||
FROM messages_fts
|
||||
JOIN messages m ON m.id = messages_fts.rowid
|
||||
LEFT JOIN sessions s ON s.id = m.session_id
|
||||
WHERE messages_fts MATCH ?
|
||||
AND m.session_id != ?
|
||||
AND m.role IN ('user','assistant')
|
||||
ORDER BY rank
|
||||
LIMIT 20
|
||||
""",
|
||||
(fts_query, current_session_id),
|
||||
).fetchall()
|
||||
con.close()
|
||||
except Exception:
|
||||
return []
|
||||
|
||||
out = []
|
||||
seen = set()
|
||||
for r in rows:
|
||||
sid = r["session_id"]
|
||||
if sid in seen:
|
||||
continue
|
||||
seen.add(sid)
|
||||
# started_at is a Unix timestamp (float); format to a readable date.
|
||||
when = ""
|
||||
sa = r["started_at"]
|
||||
if sa:
|
||||
try:
|
||||
when = datetime.fromtimestamp(float(sa)).strftime("%Y-%m-%d %H:%M")
|
||||
except (ValueError, TypeError, OSError):
|
||||
when = str(sa)[:16]
|
||||
out.append({
|
||||
"session_id": sid,
|
||||
"title": r["title"],
|
||||
"when": when,
|
||||
"snippet": (r["snip"] or "").replace("\n", " "),
|
||||
})
|
||||
if len(out) >= top_k:
|
||||
break
|
||||
return out
|
||||
|
||||
|
||||
# ── fact_store search (FTS5 over memory_store.db) ────────────
|
||||
|
||||
def _search_facts(query, top_k=3):
|
||||
"""FTS5 search over durable facts in memory_store.db.
|
||||
|
||||
Returns list of fact content strings. Fail-open: returns [] on error.
|
||||
"""
|
||||
import sqlite3
|
||||
db = Path.home() / ".hermes" / "memory_store.db"
|
||||
home = getattr(state, "HERMES_HOME", None)
|
||||
if home and (Path(home) / "memory_store.db").exists():
|
||||
db = Path(home) / "memory_store.db"
|
||||
if not db.exists():
|
||||
return []
|
||||
|
||||
toks = [t for t in _tokenize(query) if len(t) >= 4]
|
||||
if not toks:
|
||||
return []
|
||||
fts_query = " OR ".join(toks[:8])
|
||||
|
||||
try:
|
||||
con = sqlite3.connect(f"file:{db}?mode=ro", uri=True)
|
||||
con.row_factory = sqlite3.Row
|
||||
cur = con.cursor()
|
||||
rows = cur.execute(
|
||||
"""
|
||||
SELECT f.content AS content, f.trust_score AS trust
|
||||
FROM facts_fts
|
||||
JOIN facts f ON f.fact_id = facts_fts.rowid
|
||||
WHERE facts_fts MATCH ?
|
||||
ORDER BY rank
|
||||
LIMIT ?
|
||||
""",
|
||||
(fts_query, top_k),
|
||||
).fetchall()
|
||||
con.close()
|
||||
except Exception:
|
||||
return []
|
||||
|
||||
return [r["content"][:200] for r in rows if r["content"]]
|
||||
|
||||
|
||||
def pre_llm_call(session_id="", user_message="", is_first_turn=False, **kwargs):
|
||||
"""Inject relevant memories when topic changes (fabric + Qdrant)."""
|
||||
global _last_query_tokens
|
||||
if not user_message:
|
||||
return None
|
||||
|
||||
tokens = _tokenize(user_message)
|
||||
if not tokens:
|
||||
return None
|
||||
|
||||
# Overlap gate: only suppress on NEAR-LITERAL repetition of the previous
|
||||
# turn (>0.85). The old 0.6 gate killed all injection in long single-topic
|
||||
# sessions — exactly when accumulated context matters most. We now keep
|
||||
# injecting and rely on per-source dedup (_injected_* sets) to avoid
|
||||
# repeating identical results turn after turn.
|
||||
if _last_query_tokens:
|
||||
overlap = len(tokens & _last_query_tokens) / max(len(tokens), 1)
|
||||
if overlap > 0.85:
|
||||
return None
|
||||
|
||||
_last_query_tokens = tokens
|
||||
|
||||
is_social = _is_social_close(user_message)
|
||||
|
||||
agent = state.AGENT_NAME or "agent"
|
||||
results = state.recall(user_message, max_results=5, agent=agent)
|
||||
|
||||
# log fabric recall for telemetry (even if empty)
|
||||
if results:
|
||||
state.log_recall(user_message, results, source="pre_llm_call")
|
||||
|
||||
# ── Qdrant search (independent of fabric) ──
|
||||
# Threshold lowered 0.72 → 0.55: legitimate queries scored 0.57-0.63 and
|
||||
# were silently filtered out by the old 0.72 gate.
|
||||
qdrant_results = []
|
||||
if not is_social:
|
||||
qdrant_results = _search_qdrant(user_message, top_k=2, threshold=0.55)
|
||||
|
||||
# ── Session history (FTS5 over state.db) — the layer that holds
|
||||
# "this was already built in a prior session". No automatic injection
|
||||
# existed before; this is new. ──
|
||||
session_results = []
|
||||
if not is_social:
|
||||
session_results = _search_sessions(user_message, session_id, top_k=2)
|
||||
|
||||
# ── fact_store probe (durable user/environment facts) — first turn only,
|
||||
# to avoid per-turn cost. ──
|
||||
fact_results = []
|
||||
if is_first_turn and not is_social:
|
||||
fact_results = _search_facts(user_message, top_k=3)
|
||||
|
||||
# ── Bail if nothing from any source ──
|
||||
if not results and not qdrant_results and not session_results and not fact_results:
|
||||
return None
|
||||
|
||||
parts = []
|
||||
|
||||
# Fabric context (dedup against previously injected entry ids)
|
||||
if results:
|
||||
lines = ["[fabric] relevant to your request:"]
|
||||
emitted = 0
|
||||
for e in results:
|
||||
summary = e.get("summary") or e.get("_body", e.get("body", ""))[:80]
|
||||
eid = str(e.get("id", "")) or summary[:60]
|
||||
if eid in _injected_fabric:
|
||||
continue
|
||||
_injected_fabric.add(eid)
|
||||
ts = str(e.get("timestamp", ""))[:16] or "?"
|
||||
lines.append(f" [{ts}] {e.get('agent', '?')}: {summary}")
|
||||
emitted += 1
|
||||
if emitted:
|
||||
parts.append("\n".join(lines))
|
||||
|
||||
# Qdrant context (dedup against previously injected point ids)
|
||||
if qdrant_results:
|
||||
lines = ["[qdrant] knowledge base:"]
|
||||
emitted = 0
|
||||
for r in qdrant_results:
|
||||
rid = str(r.get("id", "")) or str(r.get("content_preview", ""))[:40]
|
||||
if rid in _injected_qdrant:
|
||||
continue
|
||||
_injected_qdrant.add(rid)
|
||||
source = r.get("source", "?")
|
||||
title = r.get("title", "")
|
||||
score = r.get("score", 0)
|
||||
label = f"{source}"
|
||||
if title:
|
||||
label = f"{source}: {title[:60]}"
|
||||
content = r.get("content_preview", "")[:600]
|
||||
lines.append(f" ### {label} (score: {score:.2f})\n {content}")
|
||||
emitted += 1
|
||||
if emitted:
|
||||
parts.append("\n".join(lines))
|
||||
|
||||
# Session history context (dedup against previously injected session ids)
|
||||
if session_results:
|
||||
lines = ["[sessions] prior conversations on this topic:"]
|
||||
emitted = 0
|
||||
for s in session_results:
|
||||
sid = s.get("session_id", "")
|
||||
if sid in _injected_sessions:
|
||||
continue
|
||||
_injected_sessions.add(sid)
|
||||
title = s.get("title") or "(untitled)"
|
||||
snippet = s.get("snippet", "")[:200]
|
||||
when = s.get("when", "")
|
||||
lines.append(f" [{when}] {title}: {snippet}")
|
||||
emitted += 1
|
||||
if emitted:
|
||||
parts.append("\n".join(lines))
|
||||
|
||||
# fact_store context (first turn only)
|
||||
if fact_results:
|
||||
lines = ["[facts] durable facts about the user/environment:"]
|
||||
for f in fact_results:
|
||||
lines.append(f" - {f}")
|
||||
parts.append("\n".join(lines))
|
||||
|
||||
if not parts:
|
||||
return None
|
||||
|
||||
return {"context": "\n\n".join(parts)}
|
||||
|
||||
|
||||
def post_llm_call(session_id="", user_message="", assistant_response="", platform="", **kwargs):
|
||||
"""Capture high-value decisions + creative tracking."""
|
||||
if not assistant_response:
|
||||
return
|
||||
|
||||
state.exchanges.append({
|
||||
"user": (user_message or "")[:200],
|
||||
"assistant": assistant_response[:500],
|
||||
})
|
||||
|
||||
agent = state.AGENT_NAME or "agent"
|
||||
plat = platform or "cli"
|
||||
|
||||
# capture decisions: requires decision + outcome in response, AND a substantial
|
||||
# user request (>50 chars) to ground the claim
|
||||
user_text = (user_message or "").strip()
|
||||
if (state.DECISION_RE.search(assistant_response)
|
||||
and state.OUTCOME_RE.search(assistant_response)
|
||||
and len(assistant_response) > 200
|
||||
and len(user_text) > 50):
|
||||
body = f"Task: {user_text[:_TASK_MAX]}\n\nResult: {assistant_response[:_RESULT_MAX]}"
|
||||
summary = assistant_response[:80].replace("\n", " ")
|
||||
entry_status = "completed" if state.COMPLETION_RE.search(assistant_response) else ""
|
||||
state.write_entry("decision", body, summary,
|
||||
platform=plat, status=entry_status, training_value="high")
|
||||
|
||||
# creative tracking (uses broader _THEME_RE, doesn't write entries)
|
||||
creative = state.load_creative()
|
||||
changed = False
|
||||
|
||||
if _THEME_RE.search(assistant_response):
|
||||
theme = _extract_theme(assistant_response)
|
||||
if theme and theme not in creative["themes"]:
|
||||
creative["themes"].append(theme)
|
||||
creative["themes"] = creative["themes"][-20:]
|
||||
changed = True
|
||||
|
||||
if _EVAL_RE.search(assistant_response):
|
||||
learning = _extract_sentence(assistant_response, _EVAL_RE)
|
||||
if learning:
|
||||
learning = _sanitize_learning(learning)
|
||||
if learning and learning not in creative["learnings"]:
|
||||
creative["learnings"].append(learning)
|
||||
creative["learnings"] = creative["learnings"][-15:]
|
||||
changed = True
|
||||
|
||||
if _QUESTION_RE.search(assistant_response):
|
||||
question = _extract_sentence(assistant_response, _QUESTION_RE)
|
||||
if question and question not in creative["questions"]:
|
||||
creative["questions"].append(question)
|
||||
creative["questions"] = creative["questions"][-15:]
|
||||
changed = True
|
||||
|
||||
if changed:
|
||||
state.save_creative(creative)
|
||||
|
||||
|
||||
# ── LLM-powered session extraction ────────────────────────
|
||||
|
||||
def _parse_json_robust(raw):
|
||||
"""Extract JSON array/object from LLM output with markdown tolerances.
|
||||
|
||||
Handles: ```json fences, leading text, trailing commas, whitespace.
|
||||
Returns parsed value on success, None on failure.
|
||||
"""
|
||||
if not raw or not raw.strip():
|
||||
return None
|
||||
|
||||
text = raw.strip()
|
||||
|
||||
# Strip markdown code fences
|
||||
for fence in ("```json", "```"):
|
||||
if text.startswith(fence):
|
||||
text = text[len(fence):].lstrip()
|
||||
if text.endswith("```"):
|
||||
text = text[:-3].rstrip()
|
||||
|
||||
# Find first JSON structure character
|
||||
for start_char in ("[", "{"):
|
||||
idx = text.find(start_char)
|
||||
if idx != -1:
|
||||
text = text[idx:]
|
||||
break
|
||||
|
||||
# Attempt parse; progressively strip trailing characters on failure
|
||||
attempts = 0
|
||||
while attempts < 20:
|
||||
try:
|
||||
return json.loads(text)
|
||||
except json.JSONDecodeError:
|
||||
# Strip last char and try again (handles trailing commas, extra })
|
||||
if text:
|
||||
text = text[:-1]
|
||||
attempts += 1
|
||||
continue
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def _build_transcript(exchanges):
|
||||
"""Build a compact transcript from session exchanges for LLM analysis."""
|
||||
lines = []
|
||||
for i, ex in enumerate(exchanges):
|
||||
user = (ex.get("user") or "").strip()
|
||||
assistant = (ex.get("assistant") or "").strip()
|
||||
if user:
|
||||
lines.append(f"[Turn {i+1} — User]\n{user[:500]}")
|
||||
if assistant:
|
||||
lines.append(f"[Turn {i+1} — Agent]\n{assistant[:800]}")
|
||||
return "\n\n".join(lines)
|
||||
|
||||
|
||||
def _llm_extract_entries(transcript):
|
||||
"""Use LLM to extract significant entries from session transcript.
|
||||
|
||||
Returns list of dicts: {type, summary, content, training_value}
|
||||
Returns empty list on failure or if nothing worth preserving.
|
||||
"""
|
||||
if not _OPENROUTER_KEY:
|
||||
logger.warning("icarus: no OpenRouter key — skipping LLM extraction")
|
||||
return []
|
||||
|
||||
prompt = (
|
||||
"You are a session archivist for an AI agent. Analyze this agent session "
|
||||
"transcript and extract ONLY significant entries worth preserving in a "
|
||||
"cross-agent knowledge base. Skip trivial sessions, greetings, and routine chatter.\n\n"
|
||||
"For each significant entry, provide:\n"
|
||||
"- type: \"decision\" (technical decision with rationale), "
|
||||
"\"resolution\" (bug fix or problem solved), "
|
||||
"or \"note\" (discovery or learning)\n"
|
||||
"- summary: one line, max 80 chars, in the original language of the session\n"
|
||||
"- content: structured markdown with ## Context, ## Action/Decision, and ## Outcome. "
|
||||
"Include concrete details: commands, paths, error messages, decisions made.\n"
|
||||
"- training_value: \"high\" (outcome verified, artifact produced, decision with evidence), "
|
||||
"\"normal\" (useful context or progress), "
|
||||
"or \"low\" (marginal, but not zero)\n\n"
|
||||
"If the session contains NOTHING worth preserving across sessions, "
|
||||
"return an empty array: []\n\n"
|
||||
"Return ONLY valid JSON array, no other text:\n"
|
||||
'[{"type": "decision", "summary": "...", "content": "...", "training_value": "high"}, ...]'
|
||||
)
|
||||
|
||||
payload = json.dumps({
|
||||
"model": _EXTRACTION_MODEL,
|
||||
"messages": [
|
||||
{"role": "system", "content": prompt},
|
||||
{"role": "user", "content": transcript[:8000]}
|
||||
],
|
||||
"max_tokens": _EXTRACTION_MAX_TOKENS,
|
||||
"temperature": 0.2
|
||||
}).encode("utf-8")
|
||||
|
||||
try:
|
||||
req = urllib.request.Request(
|
||||
"https://openrouter.ai/api/v1/chat/completions",
|
||||
data=payload,
|
||||
headers={
|
||||
"Authorization": f"Bearer {_OPENROUTER_KEY}",
|
||||
"Content-Type": "application/json",
|
||||
"HTTP-Referer": "https://hermes-agent.local",
|
||||
"X-Title": "Icarus Session Extraction"
|
||||
}
|
||||
)
|
||||
resp = urllib.request.urlopen(req, timeout=45)
|
||||
body = json.loads(resp.read().decode("utf-8"))
|
||||
raw = body["choices"][0]["message"]["content"]
|
||||
|
||||
# Parse JSON from response (robust — handles markdown fences, null)
|
||||
if raw is None:
|
||||
raise ValueError("DeepSeek returned content:null (response_format bug)")
|
||||
extracted = _parse_json_robust(raw)
|
||||
if isinstance(extracted, dict):
|
||||
# Some models return {entries: [...]} — unwrap
|
||||
for key in ("entries", "results", "items"):
|
||||
if key in extracted and isinstance(extracted[key], list):
|
||||
extracted = extracted[key]
|
||||
break
|
||||
else:
|
||||
# Single entry wrapped in dict
|
||||
if "type" in extracted:
|
||||
extracted = [extracted]
|
||||
else:
|
||||
extracted = []
|
||||
|
||||
if not isinstance(extracted, list):
|
||||
logger.warning("icarus: LLM extraction returned non-list: %s", type(extracted))
|
||||
return []
|
||||
|
||||
# Validate and filter
|
||||
valid = []
|
||||
allowed_types = {"decision", "resolution", "note"}
|
||||
for entry in extracted:
|
||||
if not isinstance(entry, dict):
|
||||
continue
|
||||
etype = entry.get("type", "")
|
||||
summary = entry.get("summary", "")
|
||||
content = entry.get("content", "")
|
||||
if etype not in allowed_types:
|
||||
continue
|
||||
if len(summary) < 10 or len(content) < 60:
|
||||
continue
|
||||
valid.append({
|
||||
"type": etype,
|
||||
"summary": summary[:80],
|
||||
"content": content[:2000],
|
||||
"training_value": entry.get("training_value", "normal")
|
||||
})
|
||||
|
||||
return valid
|
||||
|
||||
except (urllib.error.URLError, json.JSONDecodeError, KeyError, IndexError, ValueError,
|
||||
ConnectionError, TimeoutError, OSError) as e:
|
||||
logger.warning("icarus: LLM extraction failed (%s) — falling back to legacy", type(e).__name__)
|
||||
return []
|
||||
|
||||
|
||||
def _legacy_session_write(platform, scores):
|
||||
"""Fallback: original truncated session write (pre-LLM behavior)."""
|
||||
plat = platform or "cli"
|
||||
parts = []
|
||||
|
||||
first_user = next(
|
||||
(
|
||||
ex["user"] for ex in state.exchanges
|
||||
if len(ex.get("user", "").strip()) > 50
|
||||
and not _is_system_injection(ex.get("user", ""))
|
||||
),
|
||||
None
|
||||
)
|
||||
if first_user:
|
||||
parts.append(f"## Task\n{first_user[:_TASK_MAX]}")
|
||||
|
||||
for ex in state.exchanges:
|
||||
resp = ex.get("assistant", "")
|
||||
if state.DECISION_RE.search(resp) and len(resp) > 100:
|
||||
parts.append(f"## Decision\n{resp[:500]}")
|
||||
break
|
||||
|
||||
substantive = [ex for ex in state.exchanges if len(ex.get("assistant", "").strip()) > 100]
|
||||
if substantive:
|
||||
parts.append(f"## Result\n{substantive[-1]['assistant'][:_RESULT_MAX]}")
|
||||
|
||||
content = "\n\n".join(parts) if parts else state.exchanges[-1].get("assistant", "")[:500]
|
||||
|
||||
if substantive:
|
||||
result_text = substantive[-1]['assistant']
|
||||
else:
|
||||
result_text = content
|
||||
summary = re.sub(r"\s+", " ", result_text.replace("\n", " ")).strip()[:80]
|
||||
summary = re.sub(r"-{2,}", "—", summary) # sanitize: prevent YAML frontmatter breakage
|
||||
|
||||
if scores["total"] >= 0.6:
|
||||
tv = "high"
|
||||
elif scores["total"] >= 0.3:
|
||||
tv = "normal"
|
||||
else:
|
||||
tv = "low"
|
||||
|
||||
state.write_entry("session", content, summary, platform=plat,
|
||||
training_value=tv, status="completed")
|
||||
|
||||
|
||||
def on_session_end(session_id="", platform="", completed=False, **kwargs):
|
||||
"""Score session, extract entries via LLM, fall back to legacy truncation."""
|
||||
creative = state.load_creative()
|
||||
state.write_memory_file(creative)
|
||||
|
||||
if not state.exchanges:
|
||||
return
|
||||
|
||||
scores = state.score_session()
|
||||
if scores["total"] < 0.2:
|
||||
return
|
||||
|
||||
plat = platform or "cli"
|
||||
|
||||
# ── LLM extraction (primary) ──
|
||||
transcript = _build_transcript(state.exchanges)
|
||||
entries = _llm_extract_entries(transcript)
|
||||
|
||||
if entries:
|
||||
for entry in entries:
|
||||
state.write_entry(
|
||||
entry["type"],
|
||||
entry["content"],
|
||||
entry["summary"],
|
||||
platform=plat,
|
||||
training_value=entry.get("training_value", "normal"),
|
||||
status="completed"
|
||||
)
|
||||
logger.info("icarus: LLM extracted %d entries from session", len(entries))
|
||||
else:
|
||||
# ── Legacy fallback ──
|
||||
logger.info("icarus: LLM extraction produced nothing — using legacy truncation")
|
||||
_legacy_session_write(platform, scores)
|
||||
|
|
@ -0,0 +1,146 @@
|
|||
"""Obsidian integration for Icarus fabric entries.
|
||||
|
||||
Opt-in via ICARUS_OBSIDIAN=1 environment variable.
|
||||
|
||||
- format_entry: appends wikilinks for review_of/revises refs
|
||||
- ensure_daily_note: links new entries in a daily note
|
||||
- init_obsidian: one-time vault setup
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
from .state import _parse_head
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
LINKS_START = "<!-- ICARUS_OBSIDIAN_LINKS_START -->"
|
||||
LINKS_END = "<!-- ICARUS_OBSIDIAN_LINKS_END -->"
|
||||
|
||||
|
||||
def _find_entry_file(ref: str, fabric_dir: Path) -> Optional[str]:
|
||||
"""Resolve agent:id ref to a filename (without .md extension)."""
|
||||
if ":" not in ref:
|
||||
return None
|
||||
agent, entry_id = ref.split(":", 1)
|
||||
if not agent or not entry_id:
|
||||
return None
|
||||
for d in (fabric_dir, fabric_dir / "cold"):
|
||||
if not d.exists():
|
||||
continue
|
||||
for f in d.glob("*.md"):
|
||||
h = _parse_head(f)
|
||||
if h.get("agent") == agent and h.get("id") == entry_id:
|
||||
return f.stem # filename without .md
|
||||
return None
|
||||
|
||||
|
||||
def format_entry(filepath: Path, fabric_dir: Path, review_of: str = "", revises: str = ""):
|
||||
"""Append wikilinks section to a fabric entry for Obsidian navigation."""
|
||||
if not review_of and not revises:
|
||||
return
|
||||
|
||||
text = filepath.read_text("utf-8")
|
||||
links = []
|
||||
|
||||
if review_of:
|
||||
target = _find_entry_file(review_of, fabric_dir)
|
||||
if target:
|
||||
links.append(f"- Reviews: [[{target}]]")
|
||||
else:
|
||||
links.append(f"- Reviews: {review_of}")
|
||||
|
||||
if revises:
|
||||
target = _find_entry_file(revises, fabric_dir)
|
||||
if target:
|
||||
links.append(f"- Revises: [[{target}]]")
|
||||
else:
|
||||
links.append(f"- Revises: {revises}")
|
||||
|
||||
if not links:
|
||||
return
|
||||
|
||||
generated = (
|
||||
f"\n\n{LINKS_START}\n"
|
||||
"## Links\n"
|
||||
+ "\n".join(links)
|
||||
+ f"\n{LINKS_END}\n"
|
||||
)
|
||||
|
||||
existing = re.compile(
|
||||
rf"\n*{re.escape(LINKS_START)}.*?{re.escape(LINKS_END)}\n*",
|
||||
re.DOTALL,
|
||||
)
|
||||
text = existing.sub("\n", text).rstrip()
|
||||
filepath.write_text(text + generated, "utf-8")
|
||||
|
||||
|
||||
def ensure_daily_note(fabric_dir: Path, entry_filename: str, summary: str):
|
||||
"""Add a link to the new entry in today's daily note."""
|
||||
daily_dir = fabric_dir / "daily"
|
||||
daily_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
today = datetime.now().strftime("%Y-%m-%d")
|
||||
daily_path = daily_dir / f"{today}.md"
|
||||
|
||||
stem = entry_filename.replace(".md", "")
|
||||
link_line = f"- [[{stem}]] {summary}\n"
|
||||
|
||||
if daily_path.exists():
|
||||
content = daily_path.read_text("utf-8")
|
||||
if f"[[{stem}]]" in content:
|
||||
return # already linked
|
||||
daily_path.write_text(content.rstrip() + "\n" + link_line, "utf-8")
|
||||
else:
|
||||
daily_path.write_text(f"# {today}\n\n{link_line}", "utf-8")
|
||||
|
||||
|
||||
def _vault_dir_for(fabric_dir: Path) -> Path:
|
||||
"""Resolve the Obsidian vault root for config files."""
|
||||
configured = os.environ.get("OBSIDIAN_VAULT_PATH", "").strip()
|
||||
if configured:
|
||||
return Path(configured).expanduser()
|
||||
return fabric_dir
|
||||
|
||||
|
||||
def init_obsidian(fabric_dir: Path) -> dict:
|
||||
"""One-time Obsidian vault setup for a fabric directory."""
|
||||
created = []
|
||||
|
||||
daily_dir = fabric_dir / "daily"
|
||||
if not daily_dir.exists():
|
||||
daily_dir.mkdir(parents=True, exist_ok=True)
|
||||
created.append(str(daily_dir))
|
||||
|
||||
vault_dir = _vault_dir_for(fabric_dir)
|
||||
vault_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
obsidian_dir = vault_dir / ".obsidian"
|
||||
if not obsidian_dir.exists():
|
||||
obsidian_dir.mkdir(parents=True, exist_ok=True)
|
||||
created.append(str(obsidian_dir))
|
||||
|
||||
app_json = obsidian_dir / "app.json"
|
||||
if not app_json.exists():
|
||||
app_json.write_text(json.dumps({
|
||||
"showFrontmatter": True,
|
||||
"readableLineLength": True,
|
||||
}, indent=2), "utf-8")
|
||||
created.append(str(app_json))
|
||||
|
||||
if created:
|
||||
logger.info("icarus: obsidian init created %s", created)
|
||||
|
||||
return {
|
||||
"status": "initialized" if created else "already_initialized",
|
||||
"fabric_dir": str(fabric_dir),
|
||||
"vault_dir": str(vault_dir),
|
||||
"created": created,
|
||||
}
|
||||
|
|
@ -0,0 +1,26 @@
|
|||
name: icarus
|
||||
version: 0.3.0
|
||||
description: Self-memory and replacement models for Hermes agents. Remember your work. Train your replacement.
|
||||
author: esaradev
|
||||
provides_tools:
|
||||
- fabric_recall
|
||||
- fabric_write
|
||||
- fabric_search
|
||||
- fabric_pending
|
||||
- fabric_curate
|
||||
- fabric_export
|
||||
- fabric_train
|
||||
- fabric_train_status
|
||||
- fabric_models
|
||||
- fabric_eval
|
||||
- fabric_switch_model
|
||||
- fabric_rollback_model
|
||||
- fabric_brief
|
||||
- fabric_telemetry
|
||||
- fabric_init_obsidian
|
||||
- fabric_report
|
||||
provides_hooks:
|
||||
- on_session_start
|
||||
- pre_llm_call
|
||||
- post_llm_call
|
||||
- on_session_end
|
||||
|
|
@ -0,0 +1,420 @@
|
|||
"""Tool schemas — what the LLM sees."""
|
||||
|
||||
FABRIC_RECALL = {
|
||||
"name": "fabric_recall",
|
||||
"description": (
|
||||
"Retrieve relevant memories from the shared fabric. Uses ranked scoring "
|
||||
"across keyword match, project/agent affinity, recency, and tier. "
|
||||
"Use this when you need context from past sessions, other agents' work, "
|
||||
"or cross-platform history. Returns the top matching entries with scores."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"query": {
|
||||
"type": "string",
|
||||
"description": "What to search for — a topic, question, or keyword phrase",
|
||||
},
|
||||
"max_results": {
|
||||
"type": "integer",
|
||||
"description": "Maximum entries to return (default: 5)",
|
||||
},
|
||||
"agent": {
|
||||
"type": "string",
|
||||
"description": "Boost entries from this agent (optional)",
|
||||
},
|
||||
"project": {
|
||||
"type": "string",
|
||||
"description": "Boost entries from this project (optional)",
|
||||
},
|
||||
},
|
||||
"required": ["query"],
|
||||
},
|
||||
}
|
||||
|
||||
FABRIC_WRITE = {
|
||||
"name": "fabric_write",
|
||||
"description": (
|
||||
"Write a new entry to shared fabric memory. All agents on all platforms "
|
||||
"can read it. Linking guidelines:\n"
|
||||
"- type='review' + review_of: when you evaluate another agent's work, "
|
||||
"link back to the original entry so the chain is traceable.\n"
|
||||
"- revises: when you fix or improve an entry after receiving feedback, "
|
||||
"link to your original entry so before/after are connected.\n"
|
||||
"- status='open' + assigned_to: when handing work to a specific agent.\n"
|
||||
"Links improve retrieval, training data quality, and cross-agent awareness. "
|
||||
"If you have the source entry ID (from session context or fabric_pending), use it."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"type": {
|
||||
"type": "string",
|
||||
"description": "Entry type: task, decision, review, resolution, research, code-session, session, note",
|
||||
},
|
||||
"content": {
|
||||
"type": "string",
|
||||
"description": "The full content/body of the entry",
|
||||
},
|
||||
"summary": {
|
||||
"type": "string",
|
||||
"description": "One-line summary (shown in listings and search results)",
|
||||
},
|
||||
"tags": {
|
||||
"type": "string",
|
||||
"description": "Comma-separated tags (optional)",
|
||||
},
|
||||
"status": {
|
||||
"type": "string",
|
||||
"description": "open (requires assigned_to), completed, blocked, or superseded",
|
||||
},
|
||||
"outcome": {
|
||||
"type": "string",
|
||||
"description": "Result or conclusion. Most valuable field for training.",
|
||||
},
|
||||
"review_of": {
|
||||
"type": "string",
|
||||
"description": "When type='review': the entry you are evaluating, as agent:id (e.g. icarus:a3f29b01). Get the id from session context or fabric_pending.",
|
||||
},
|
||||
"revises": {
|
||||
"type": "string",
|
||||
"description": "When resubmitting fixed work: the original entry you are revising, as agent:id. Connects the before/after for training.",
|
||||
},
|
||||
"customer_id": {
|
||||
"type": "string",
|
||||
"description": "Customer/account scope. Carry forward from the original entry when resolving a ticket.",
|
||||
},
|
||||
"assigned_to": {
|
||||
"type": "string",
|
||||
"description": "When status='open': the agent who should pick this up. Required for the entry to appear in their fabric_pending.",
|
||||
},
|
||||
"training_value": {
|
||||
"type": "string",
|
||||
"enum": ["high", "normal", "low"],
|
||||
"description": "Training quality signal. high = decisions with outcomes, completed reviews, successful fixes. low = generic chatter. Affects export filtering.",
|
||||
},
|
||||
"verified": {
|
||||
"type": "string",
|
||||
"enum": ["true", "false"],
|
||||
"description": "Whether this outcome was verified (tests passed, deployment succeeded, customer confirmed). Verified entries are preferred in high-precision export.",
|
||||
},
|
||||
"evidence": {
|
||||
"type": "string",
|
||||
"description": "How the outcome was verified. E.g. 'tests pass', 'deployed to prod', 'customer confirmed fix'. Grounds the entry for training.",
|
||||
},
|
||||
"source_tool": {
|
||||
"type": "string",
|
||||
"description": "The tool that produced this result (e.g. 'bash', 'code_editor', 'web_search'). Helps training data reflect real tool use.",
|
||||
},
|
||||
"artifact_paths": {
|
||||
"type": "string",
|
||||
"description": "Comma-separated file paths of artifacts produced (e.g. 'src/limiter.ts, tests/limiter.test.ts').",
|
||||
},
|
||||
},
|
||||
"required": ["type", "content", "summary"],
|
||||
},
|
||||
}
|
||||
|
||||
FABRIC_PENDING = {
|
||||
"name": "fabric_pending",
|
||||
"description": (
|
||||
"Show work assigned to you. Returns entry metadata including IDs for linking.\n"
|
||||
"- open_tasks: work from other agents you need to act on. Could be code to "
|
||||
"review, research to implement, a ticket to resolve, or a task to complete. "
|
||||
"Check the entry type to decide your response.\n"
|
||||
"- reviews_of_my_work: feedback from other agents on your entries. "
|
||||
"Use revises to link your fix back to the original.\n"
|
||||
"- open_tickets: customer-scoped entries. Carry customer_id forward when resolving.\n"
|
||||
"Call at session start to see what needs attention."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"customer_id": {
|
||||
"type": "string",
|
||||
"description": "Filter to a specific customer (optional)",
|
||||
},
|
||||
},
|
||||
"required": [],
|
||||
},
|
||||
}
|
||||
|
||||
FABRIC_SEARCH = {
|
||||
"name": "fabric_search",
|
||||
"description": (
|
||||
"Keyword search across all fabric entries. Simpler than fabric_recall — "
|
||||
"just grep. Use when you know the exact term you're looking for "
|
||||
"(a function name, error message, specific ID). Returns matching filenames "
|
||||
"and the lines that matched."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"query": {
|
||||
"type": "string",
|
||||
"description": "Exact keyword or phrase to search for",
|
||||
},
|
||||
},
|
||||
"required": ["query"],
|
||||
},
|
||||
}
|
||||
|
||||
FABRIC_CURATE = {
|
||||
"name": "fabric_curate",
|
||||
"description": (
|
||||
"Set the training value of a fabric entry. Affects which entries are "
|
||||
"included when exporting training data. Use 'high' for decisions with "
|
||||
"outcomes, completed reviews, and successful fixes. Use 'normal' for "
|
||||
"standard work. Use 'low' for generic session summaries and chatter. "
|
||||
"high-precision export mode only includes high-value entries."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"entry_id": {
|
||||
"type": "string",
|
||||
"description": "The entry ID (8 hex chars) to update",
|
||||
},
|
||||
"training_value": {
|
||||
"type": "string",
|
||||
"enum": ["high", "normal", "low"],
|
||||
"description": "Training value: high, normal, or low",
|
||||
},
|
||||
},
|
||||
"required": ["entry_id", "training_value"],
|
||||
},
|
||||
}
|
||||
|
||||
FABRIC_EXPORT = {
|
||||
"name": "fabric_export",
|
||||
"description": (
|
||||
"Export fabric entries as fine-tuning training pairs. Generates "
|
||||
"OpenAI, Together AI, and HuggingFace format JSONL files. "
|
||||
"Use mode to control quality vs volume tradeoff."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"mode": {
|
||||
"type": "string",
|
||||
"enum": ["high-precision", "normal", "high-volume"],
|
||||
"description": "high-precision: only high-value + completed + linked reviews. normal: excludes low-value (default). high-volume: everything.",
|
||||
},
|
||||
},
|
||||
"required": [],
|
||||
},
|
||||
}
|
||||
|
||||
FABRIC_TRAIN = {
|
||||
"name": "fabric_train",
|
||||
"description": (
|
||||
"Start a fine-tuning job on Together AI using your fabric entries as "
|
||||
"training data. Exports, uploads, and kicks off training. Returns "
|
||||
"immediately with a job ID. Use fabric_train_status to check progress, "
|
||||
"fabric_eval to test the result, fabric_switch_model to activate it."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"model": {
|
||||
"type": "string",
|
||||
"description": "Base model (default: Qwen/Qwen2-7B-Instruct)",
|
||||
},
|
||||
"suffix": {
|
||||
"type": "string",
|
||||
"description": "Model name suffix (default: agent name)",
|
||||
},
|
||||
"epochs": {
|
||||
"type": "integer",
|
||||
"description": "Training epochs (default: 3)",
|
||||
},
|
||||
"mode": {
|
||||
"type": "string",
|
||||
"enum": ["high-precision", "normal", "high-volume"],
|
||||
"description": "Optional export mode. Omit to auto-select the highest-quality mode with enough pairs.",
|
||||
},
|
||||
"min_pairs": {
|
||||
"type": "integer",
|
||||
"description": "Minimum pair count required before starting training (default: 10).",
|
||||
},
|
||||
"batch_size": {
|
||||
"type": "integer",
|
||||
"description": "Together batch size, must be >= 8 (default: 8)",
|
||||
},
|
||||
"learning_rate": {
|
||||
"type": "number",
|
||||
"description": "Together learning rate, must be > 0 (default: 1e-5)",
|
||||
},
|
||||
"n_checkpoints": {
|
||||
"type": "integer",
|
||||
"description": "Together checkpoint count, must be >= 1 (default: 1)",
|
||||
},
|
||||
},
|
||||
"required": [],
|
||||
},
|
||||
}
|
||||
|
||||
FABRIC_TRAIN_STATUS = {
|
||||
"name": "fabric_train_status",
|
||||
"description": (
|
||||
"Check the status of a Together AI fine-tuning job. If completed, returns "
|
||||
"the output model ID. Pass a job ID or omit to check the most recent job."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"job_id": {
|
||||
"type": "string",
|
||||
"description": "Fine-tune job ID (omit to check last job)",
|
||||
},
|
||||
},
|
||||
"required": [],
|
||||
},
|
||||
}
|
||||
|
||||
FABRIC_MODELS = {
|
||||
"name": "fabric_models",
|
||||
"description": (
|
||||
"List all fine-tuned models trained from your fabric data. Shows job ID, "
|
||||
"base model, output model, pair count, eval scores, and whether the model "
|
||||
"is currently active. Use this to see your training history and decide "
|
||||
"which model to evaluate or activate."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {},
|
||||
"required": [],
|
||||
},
|
||||
}
|
||||
|
||||
FABRIC_EVAL = {
|
||||
"name": "fabric_eval",
|
||||
"description": (
|
||||
"Compare a candidate replacement model against the current model. "
|
||||
"Runs both on eval prompts extracted from your high-value fabric entries. "
|
||||
"Scores task completion, format compliance, and style match. "
|
||||
"Results are saved to the model registry. Requires TOGETHER_API_KEY."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"candidate_model": {
|
||||
"type": "string",
|
||||
"description": "The fine-tuned model ID to evaluate",
|
||||
},
|
||||
"base_model": {
|
||||
"type": "string",
|
||||
"description": "Model to compare against (default: current LLM_MODEL)",
|
||||
},
|
||||
"sample_count": {
|
||||
"type": "integer",
|
||||
"description": "Number of eval prompts to run (default: 10)",
|
||||
},
|
||||
},
|
||||
"required": ["candidate_model"],
|
||||
},
|
||||
}
|
||||
|
||||
FABRIC_SWITCH_MODEL = {
|
||||
"name": "fabric_switch_model",
|
||||
"description": (
|
||||
"Switch this agent to use a replacement model. Only switches if the "
|
||||
"model has eval scores above the threshold. Updates .env with the new "
|
||||
"model config and creates a backup of the current .env."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"model_id": {
|
||||
"type": "string",
|
||||
"description": "The fine-tuned model ID to switch to (from fabric_models)",
|
||||
},
|
||||
"min_eval_score": {
|
||||
"type": "number",
|
||||
"description": "Minimum average eval score required to switch (default: 0.7)",
|
||||
},
|
||||
},
|
||||
"required": ["model_id"],
|
||||
},
|
||||
}
|
||||
|
||||
FABRIC_ROLLBACK_MODEL = {
|
||||
"name": "fabric_rollback_model",
|
||||
"description": (
|
||||
"Roll back to the previous model by restoring .env from backup. "
|
||||
"Use when a replacement model is underperforming in production. "
|
||||
"No eval gate needed -- this is an emergency escape hatch."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {},
|
||||
"required": [],
|
||||
},
|
||||
}
|
||||
|
||||
FABRIC_BRIEF = {
|
||||
"name": "fabric_brief",
|
||||
"description": (
|
||||
"Get your daily operational brief. Returns: what's pending for you "
|
||||
"(open tasks, reviews, tickets), your recent work, what other agents "
|
||||
"have done, and a suggested next action. Use this at the start of "
|
||||
"every session to decide what to work on. One call replaces checking "
|
||||
"fabric_pending + fabric_recall + fabric_models separately."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {},
|
||||
"required": [],
|
||||
},
|
||||
}
|
||||
|
||||
FABRIC_TELEMETRY = {
|
||||
"name": "fabric_telemetry",
|
||||
"description": (
|
||||
"Show retrieval and usage telemetry. Reports: how many times memory "
|
||||
"was recalled, how many recalled entries were actually used (referenced "
|
||||
"via review_of or revises), and the usage rate. Use this to understand "
|
||||
"whether recalled memories are useful or just noise."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"last_n": {
|
||||
"type": "integer",
|
||||
"description": "Number of recent telemetry events to return (default: 50)",
|
||||
},
|
||||
},
|
||||
"required": [],
|
||||
},
|
||||
}
|
||||
|
||||
FABRIC_INIT_OBSIDIAN = {
|
||||
"name": "fabric_init_obsidian",
|
||||
"description": (
|
||||
"Initialize the fabric directory as an Obsidian vault. Creates "
|
||||
"daily/ directory for daily notes and .obsidian/ with minimal config. "
|
||||
"Safe to call multiple times. After this, open ~/fabric/ in Obsidian "
|
||||
"to browse entries with wikilinks and daily notes. "
|
||||
"Set ICARUS_OBSIDIAN=1 in .env to enable ongoing Obsidian formatting."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {},
|
||||
"required": [],
|
||||
},
|
||||
}
|
||||
|
||||
FABRIC_REPORT = {
|
||||
"name": "fabric_report",
|
||||
"description": (
|
||||
"Corpus health report. Shows: entry counts by type and training value, "
|
||||
"verified entry count, recall usage rates by entry type, and estimated "
|
||||
"trainable corpus size. Use periodically to understand whether your "
|
||||
"memory is producing good training data."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {},
|
||||
"required": [],
|
||||
},
|
||||
}
|
||||
|
|
@ -0,0 +1,215 @@
|
|||
#!/usr/bin/env python3
|
||||
"""eval-replacement.py -- Compare a candidate model against a base model.
|
||||
|
||||
Extracts eval prompts from high-value fabric entries, runs both models,
|
||||
scores task completion, format compliance, and style match.
|
||||
|
||||
Usage:
|
||||
TOGETHER_API_KEY=tok_... python3 scripts/eval-replacement.py \\
|
||||
--candidate-model user/icarus-v1 \\
|
||||
--base-model Qwen/Qwen2-7B-Instruct \\
|
||||
--sample-count 10
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
FABRIC_DIR = Path(os.environ.get("FABRIC_DIR", Path.home() / "fabric"))
|
||||
|
||||
STOP_WORDS = {"the", "a", "an", "is", "was", "are", "to", "of", "in", "for",
|
||||
"on", "with", "it", "and", "or", "not", "i", "you", "this", "that"}
|
||||
|
||||
# type-specific format patterns
|
||||
FORMAT_PATTERNS = {
|
||||
"review": re.compile(r"(?i)(MUST FIX|SHOULD FIX|approved|rejected|feedback|issue)"),
|
||||
"decision": re.compile(r"(?i)(because|result|outcome|conclusion|chose|decided)"),
|
||||
"code-session": re.compile(r"(?i)(function|class|def |import |return |const |let |var )"),
|
||||
"resolution": re.compile(r"(?i)(resolved|fixed|root cause|refund|ticket)"),
|
||||
"research": re.compile(r"(?i)(found|compared|analysis|benchmark|option)"),
|
||||
}
|
||||
|
||||
|
||||
def parse_entry(filepath):
|
||||
text = filepath.read_text("utf-8")
|
||||
if not text.startswith("---"):
|
||||
return None
|
||||
parts = text.split("---", 2)
|
||||
if len(parts) < 3:
|
||||
return None
|
||||
meta = {}
|
||||
for line in parts[1].strip().split("\n"):
|
||||
if ": " in line and not line.strip().startswith("-"):
|
||||
k, v = line.strip().split(": ", 1)
|
||||
meta[k.strip()] = v.strip()
|
||||
meta["body"] = parts[2].strip()
|
||||
return meta
|
||||
|
||||
|
||||
def get_eval_entries(sample_count):
|
||||
"""Get high-value entries for eval prompts."""
|
||||
entries = []
|
||||
for d in [FABRIC_DIR, FABRIC_DIR / "cold"]:
|
||||
if not d.exists():
|
||||
continue
|
||||
for f in sorted(d.glob("*.md"), key=lambda p: p.stat().st_mtime, reverse=True):
|
||||
e = parse_entry(f)
|
||||
if not e or not e.get("body") or len(e["body"]) < 50:
|
||||
continue
|
||||
if e.get("training_value") == "high" or e.get("status") == "completed":
|
||||
entries.append(e)
|
||||
if len(entries) >= sample_count * 2:
|
||||
break
|
||||
return entries[:sample_count]
|
||||
|
||||
|
||||
def call_model(model, prompt, api_key):
|
||||
"""Call a model via Together's OpenAI-compatible API."""
|
||||
data = json.dumps({
|
||||
"model": model,
|
||||
"messages": [
|
||||
{"role": "system", "content": "You are a helpful AI agent."},
|
||||
{"role": "user", "content": prompt},
|
||||
],
|
||||
"max_tokens": 512,
|
||||
"temperature": 0.3,
|
||||
}).encode()
|
||||
|
||||
req = urllib.request.Request(
|
||||
"https://api.together.xyz/v1/chat/completions",
|
||||
data=data,
|
||||
headers={
|
||||
"Authorization": f"Bearer {api_key}",
|
||||
"Content-Type": "application/json",
|
||||
},
|
||||
method="POST",
|
||||
)
|
||||
try:
|
||||
resp = urllib.request.urlopen(req, timeout=30)
|
||||
result = json.loads(resp.read())
|
||||
return result["choices"][0]["message"]["content"]
|
||||
except Exception as e:
|
||||
return f"ERROR: {e}"
|
||||
|
||||
|
||||
def tokenize(text):
|
||||
words = re.findall(r"[a-z0-9]+", text.lower())
|
||||
return [w for w in words if w not in STOP_WORDS]
|
||||
|
||||
|
||||
def score_task_completion(response, expected):
|
||||
"""Does the response have enough substance? 0-1."""
|
||||
if not expected:
|
||||
return 1.0 if len(response) > 50 else 0.0
|
||||
return min(1.0, len(response) / max(len(expected) * 0.5, 1))
|
||||
|
||||
|
||||
def score_format_compliance(response, entry_type):
|
||||
"""Does the response match type-specific format patterns? 0 or 1."""
|
||||
pattern = FORMAT_PATTERNS.get(entry_type)
|
||||
if not pattern:
|
||||
return 1.0
|
||||
return 1.0 if pattern.search(response) else 0.0
|
||||
|
||||
|
||||
def score_style_match(response, expected):
|
||||
"""Cosine similarity of word frequency distributions. 0-1."""
|
||||
if not expected or not response:
|
||||
return 0.0
|
||||
resp_tokens = tokenize(response)
|
||||
exp_tokens = tokenize(expected)
|
||||
if not resp_tokens or not exp_tokens:
|
||||
return 0.0
|
||||
|
||||
all_words = set(resp_tokens) | set(exp_tokens)
|
||||
resp_freq = {w: resp_tokens.count(w) for w in all_words}
|
||||
exp_freq = {w: exp_tokens.count(w) for w in all_words}
|
||||
|
||||
dot = sum(resp_freq.get(w, 0) * exp_freq.get(w, 0) for w in all_words)
|
||||
mag_r = math.sqrt(sum(v ** 2 for v in resp_freq.values()))
|
||||
mag_e = math.sqrt(sum(v ** 2 for v in exp_freq.values()))
|
||||
|
||||
if mag_r == 0 or mag_e == 0:
|
||||
return 0.0
|
||||
return dot / (mag_r * mag_e)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Compare candidate vs base model")
|
||||
parser.add_argument("--candidate-model", required=True)
|
||||
parser.add_argument("--base-model", required=True)
|
||||
# Read from environment instead of CLI to avoid leaking via argv
|
||||
TOGETHER_API_KEY = os.environ.get("TOGETHER_API_KEY", "")
|
||||
if not TOGETHER_API_KEY:
|
||||
parser.error("TOGETHER_API_KEY environment variable not set")
|
||||
parser.add_argument("--fabric-dir", default=None)
|
||||
parser.add_argument("--sample-count", type=int, default=10)
|
||||
args = parser.parse_args()
|
||||
|
||||
global FABRIC_DIR
|
||||
if args.fabric_dir:
|
||||
FABRIC_DIR = Path(args.fabric_dir)
|
||||
|
||||
entries = get_eval_entries(args.sample_count)
|
||||
if not entries:
|
||||
json.dump({"error": "no eval entries found"}, sys.stdout)
|
||||
sys.exit(1)
|
||||
|
||||
results = []
|
||||
for e in entries:
|
||||
entry_type = e.get("type", "task")
|
||||
summary = e.get("summary", "")
|
||||
body = e.get("body", "")
|
||||
prompt = f"[{entry_type}] {summary}" if summary else f"Complete this {entry_type}"
|
||||
|
||||
base_resp = call_model(args.base_model, prompt, TOGETHER_API_KEY)
|
||||
cand_resp = call_model(args.candidate_model, prompt, TOGETHER_API_KEY)
|
||||
|
||||
base_scores = {
|
||||
"task_completion": score_task_completion(base_resp, body),
|
||||
"format_compliance": score_format_compliance(base_resp, entry_type),
|
||||
"style_match": score_style_match(base_resp, body),
|
||||
}
|
||||
cand_scores = {
|
||||
"task_completion": score_task_completion(cand_resp, body),
|
||||
"format_compliance": score_format_compliance(cand_resp, entry_type),
|
||||
"style_match": score_style_match(cand_resp, body),
|
||||
}
|
||||
|
||||
results.append({
|
||||
"prompt": prompt[:80],
|
||||
"type": entry_type,
|
||||
"base_scores": base_scores,
|
||||
"candidate_scores": cand_scores,
|
||||
})
|
||||
|
||||
# aggregate
|
||||
def avg_scores(key):
|
||||
vals = {}
|
||||
for r in results:
|
||||
for metric, score in r[key].items():
|
||||
vals.setdefault(metric, []).append(score)
|
||||
return {m: round(sum(s) / len(s), 3) for m, s in vals.items()}
|
||||
|
||||
base_avg = avg_scores("base_scores")
|
||||
cand_avg = avg_scores("candidate_scores")
|
||||
|
||||
output = {
|
||||
"sample_count": len(results),
|
||||
"base_model": args.base_model,
|
||||
"candidate_model": args.candidate_model,
|
||||
"base_scores": base_avg,
|
||||
"candidate_scores": cand_avg,
|
||||
"per_prompt": results,
|
||||
}
|
||||
|
||||
json.dump(output, sys.stdout, indent=2)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
@ -0,0 +1,165 @@
|
|||
#!/usr/bin/env bash
|
||||
# smoke-handoff.sh -- prove the Icarus plugin handoff chain works end-to-end
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
REPO_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
TMP_ROOT="$(mktemp -d "${TMPDIR:-/tmp}/icarus-smoke-XXXXXX")"
|
||||
trap 'rm -rf "$TMP_ROOT"' EXIT
|
||||
|
||||
FABRIC_DIR="$TMP_ROOT/fabric"
|
||||
ICARUS_HOME="$TMP_ROOT/.hermes-icarus"
|
||||
DAEDALUS_HOME="$TMP_ROOT/.hermes-daedalus"
|
||||
|
||||
pass() { printf ' pass: %s\n' "$1"; }
|
||||
fail() { printf ' FAIL: %s\n' "$1" >&2; exit 1; }
|
||||
|
||||
mkdir -p "$FABRIC_DIR" "$FABRIC_DIR/cold"
|
||||
mkdir -p "$ICARUS_HOME/plugins/icarus" "$DAEDALUS_HOME/plugins/icarus"
|
||||
|
||||
# repo root IS the plugin — copy all plugin files + support scripts
|
||||
for home in "$ICARUS_HOME" "$DAEDALUS_HOME"; do
|
||||
cp "$REPO_DIR"/*.py "$home/plugins/icarus/"
|
||||
cp "$REPO_DIR"/plugin.yaml "$home/plugins/icarus/"
|
||||
[ -d "$REPO_DIR/scripts" ] && cp -R "$REPO_DIR/scripts" "$home/plugins/icarus/"
|
||||
done
|
||||
|
||||
export FABRIC_DIR
|
||||
export ICARUS_HOME
|
||||
export DAEDALUS_HOME
|
||||
export REPO_DIR
|
||||
|
||||
python3 - <<'PY'
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# load the plugin as a package (files use relative imports)
|
||||
import importlib.util
|
||||
import types
|
||||
|
||||
repo_dir = Path(os.environ["REPO_DIR"])
|
||||
|
||||
ns = types.ModuleType("hermes_plugins")
|
||||
ns.__path__ = []
|
||||
ns.__package__ = "hermes_plugins"
|
||||
sys.modules["hermes_plugins"] = ns
|
||||
|
||||
spec = importlib.util.spec_from_file_location(
|
||||
"hermes_plugins.icarus", str(repo_dir / "__init__.py"),
|
||||
submodule_search_locations=[str(repo_dir)])
|
||||
mod = importlib.util.module_from_spec(spec)
|
||||
mod.__package__ = "hermes_plugins.icarus"
|
||||
mod.__path__ = [str(repo_dir)]
|
||||
sys.modules["hermes_plugins.icarus"] = mod
|
||||
spec.loader.exec_module(mod)
|
||||
|
||||
hooks = mod.hooks
|
||||
state = mod.state
|
||||
tools = mod.tools
|
||||
|
||||
|
||||
def parse_id(path: str) -> str:
|
||||
for line in Path(path).read_text("utf-8").splitlines():
|
||||
if line.startswith("id: "):
|
||||
return line.split(": ", 1)[1].strip()
|
||||
raise RuntimeError(f"missing id in {path}")
|
||||
|
||||
|
||||
def run_as(agent_name: str, home: str):
|
||||
os.environ["HERMES_AGENT_NAME"] = agent_name
|
||||
os.environ["HERMES_HOME"] = home
|
||||
state.AGENT_NAME = agent_name
|
||||
state.HERMES_HOME = Path(home)
|
||||
state.FABRIC_DIR = Path(os.environ["FABRIC_DIR"])
|
||||
state._JOB_FILE = state.HERMES_HOME / ".icarus-training-job.txt"
|
||||
state._STATE_FILE = state.HERMES_HOME / ".icarus-state.json"
|
||||
state._REGISTRY_FILE = state.HERMES_HOME / ".icarus-models.json"
|
||||
state.session_id = ""
|
||||
state.exchanges = []
|
||||
|
||||
|
||||
def expect(cond: bool, msg: str):
|
||||
if not cond:
|
||||
raise AssertionError(msg)
|
||||
print(f" pass: {msg}")
|
||||
|
||||
|
||||
icarus_home = os.environ["ICARUS_HOME"]
|
||||
daedalus_home = os.environ["DAEDALUS_HOME"]
|
||||
|
||||
# 1. Builder writes an assigned handoff.
|
||||
run_as("icarus", icarus_home)
|
||||
raw = tools.fabric_write({
|
||||
"type": "code-session",
|
||||
"summary": "smoke handoff for daedalus",
|
||||
"content": "Built the relay patch. Needs review. Token amber relay.",
|
||||
"status": "open",
|
||||
"assigned_to": "daedalus",
|
||||
})
|
||||
payload = json.loads(raw)
|
||||
expect(payload.get("status") == "written", "builder handoff written")
|
||||
task_path = payload["path"]
|
||||
task_id = parse_id(task_path)
|
||||
expect(task_id, "builder handoff has entry id")
|
||||
|
||||
# 2. Reviewer sees the handoff in session-start context.
|
||||
run_as("daedalus", daedalus_home)
|
||||
ctx = hooks.on_session_start(session_id="smoke-daedalus")
|
||||
context = (ctx or {}).get("context", "")
|
||||
expect("smoke handoff for daedalus" in context, "reviewer session-start sees handoff")
|
||||
expect(f"id {task_id}" in context, "reviewer session-start includes source id")
|
||||
|
||||
# 3. Reviewer sees the handoff in fabric_pending.
|
||||
pending = json.loads(tools.fabric_pending({}))
|
||||
expect(pending.get("total", 0) >= 1, "fabric_pending returns assigned work")
|
||||
open_tasks = pending.get("open_tasks", [])
|
||||
expect(any(t.get("id") == task_id for t in open_tasks), "fabric_pending exposes exact task id")
|
||||
|
||||
# 4. Reviewer writes a linked review.
|
||||
review_ref = f"icarus:{task_id}"
|
||||
raw = tools.fabric_write({
|
||||
"type": "review",
|
||||
"summary": "reviewed smoke handoff from icarus",
|
||||
"content": "Confirmed relay handoff pickup worked. One nit on naming.",
|
||||
"review_of": review_ref,
|
||||
"status": "completed",
|
||||
"outcome": "pickup worked",
|
||||
})
|
||||
payload = json.loads(raw)
|
||||
expect(payload.get("status") == "written", "reviewer linked review written")
|
||||
review_path = payload["path"]
|
||||
review_id = parse_id(review_path)
|
||||
expect(review_id, "review has entry id")
|
||||
|
||||
# 5. Builder sees the linked review.
|
||||
run_as("icarus", icarus_home)
|
||||
ctx = hooks.on_session_start(session_id="smoke-icarus")
|
||||
context = (ctx or {}).get("context", "")
|
||||
expect("reviewed smoke handoff from icarus" in context, "builder session-start sees linked review")
|
||||
expect(review_ref in context, "builder context shows review_of link")
|
||||
|
||||
# 6. Builder writes a linked fix.
|
||||
raw = tools.fabric_write({
|
||||
"type": "code-session",
|
||||
"summary": "fixed smoke handoff after review",
|
||||
"content": "Renamed relay variables and cleaned the patch after review.",
|
||||
"revises": review_ref,
|
||||
"status": "completed",
|
||||
})
|
||||
payload = json.loads(raw)
|
||||
expect(payload.get("status") == "written", "builder linked fix written")
|
||||
|
||||
# 7. Recall retrieves the chain.
|
||||
results = json.loads(tools.fabric_recall({"query": "amber relay handoff", "max_results": 5}))
|
||||
entries = results.get("entries", [])
|
||||
expect(any(e.get("id") == task_id for e in entries), "recall returns source handoff")
|
||||
expect(any(e.get("review_of") == review_ref for e in entries), "recall returns linked review")
|
||||
|
||||
print("")
|
||||
print("Smoke handoff OK")
|
||||
print(f" fabric: {os.environ['FABRIC_DIR']}")
|
||||
print(f" task id: {task_id}")
|
||||
print(f" review_of:{review_ref}")
|
||||
PY
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,236 @@
|
|||
"""Tool handlers — the code that runs when the LLM calls each tool."""
|
||||
|
||||
import json
|
||||
from . import state
|
||||
|
||||
|
||||
def _json(payload) -> str:
|
||||
return json.dumps(payload, default=str)
|
||||
|
||||
|
||||
def fabric_recall(args: dict, **kwargs) -> str:
|
||||
query = args.get("query", "").strip()
|
||||
if not query:
|
||||
return _json({"error": "No query provided"})
|
||||
try:
|
||||
results = state.recall(
|
||||
query,
|
||||
max_results=args.get("max_results", 5),
|
||||
agent=args.get("agent"),
|
||||
project=args.get("project"),
|
||||
)
|
||||
return _json({"query": query, "count": len(results), "entries": results})
|
||||
except Exception as e:
|
||||
return _json({"error": str(e)})
|
||||
|
||||
|
||||
def fabric_write(args: dict, **kwargs) -> str:
|
||||
entry_type = args.get("type", "").strip()
|
||||
content = args.get("content", "").strip()
|
||||
summary = args.get("summary", "").strip()
|
||||
status = args.get("status", "").strip()
|
||||
assigned_to = args.get("assigned_to", "").strip()
|
||||
review_of = args.get("review_of", "").strip()
|
||||
revises = args.get("revises", "").strip()
|
||||
if not entry_type or not content or not summary:
|
||||
return _json({"error": "Need type, content, and summary"})
|
||||
if status == "open" and not assigned_to:
|
||||
return _json({"error": "status='open' requires assigned_to"})
|
||||
if entry_type == "review" and not review_of:
|
||||
return _json({"error": "type='review' requires review_of (agent:id of the entry you are reviewing)"})
|
||||
if review_of:
|
||||
parts = review_of.split(":", 1)
|
||||
if len(parts) != 2 or not parts[0] or not parts[1] or len(parts[1]) < 4:
|
||||
return _json({"error": f"review_of must be agent:id (e.g. icarus:a3f29b01), got '{review_of}'"})
|
||||
if not state.has_entry_ref(review_of):
|
||||
return _json({"error": f"review_of points to a missing entry: '{review_of}'"})
|
||||
if revises:
|
||||
parts = revises.split(":", 1)
|
||||
if len(parts) != 2 or not parts[0] or not parts[1] or len(parts[1]) < 4:
|
||||
return _json({"error": f"revises must be agent:id (e.g. icarus:a3f29b01), got '{revises}'"})
|
||||
if not state.has_entry_ref(revises):
|
||||
return _json({"error": f"revises points to a missing entry: '{revises}'"})
|
||||
tv = args.get("training_value", "").strip()
|
||||
if tv and tv not in ("high", "normal", "low"):
|
||||
return _json({"error": f"training_value must be high/normal/low, got '{tv}'"})
|
||||
try:
|
||||
path = state.write_entry(
|
||||
entry_type=entry_type,
|
||||
content=content,
|
||||
summary=summary,
|
||||
tags=args.get("tags", ""),
|
||||
status=status,
|
||||
outcome=args.get("outcome", ""),
|
||||
review_of=review_of,
|
||||
revises=revises,
|
||||
customer_id=args.get("customer_id", ""),
|
||||
assigned_to=assigned_to,
|
||||
training_value=tv,
|
||||
verified=args.get("verified", ""),
|
||||
evidence=args.get("evidence", ""),
|
||||
source_tool=args.get("source_tool", ""),
|
||||
artifact_paths=args.get("artifact_paths", ""),
|
||||
)
|
||||
# log usage telemetry when referencing other entries
|
||||
if review_of:
|
||||
ref_id = review_of.split(":", 1)[1] if ":" in review_of else review_of
|
||||
if state.was_recalled(ref_id):
|
||||
state.log_usage(ref_id, action="reviewed")
|
||||
if revises:
|
||||
ref_id = revises.split(":", 1)[1] if ":" in revises else revises
|
||||
if state.was_recalled(ref_id):
|
||||
state.log_usage(ref_id, action="revised")
|
||||
return _json({"status": "written", "path": path})
|
||||
except Exception as e:
|
||||
return _json({"error": str(e)})
|
||||
|
||||
|
||||
def fabric_search(args: dict, **kwargs) -> str:
|
||||
query = args.get("query", "").strip()
|
||||
if not query:
|
||||
return _json({"error": "No query provided"})
|
||||
try:
|
||||
results = state.search_entries(query)
|
||||
return _json({"query": query, "count": len(results), "results": results})
|
||||
except Exception as e:
|
||||
return _json({"error": str(e)})
|
||||
|
||||
|
||||
def fabric_pending(args: dict, **kwargs) -> str:
|
||||
try:
|
||||
open_tasks, reviews, open_tickets = state.read_pending(
|
||||
customer_id=args.get("customer_id"),
|
||||
)
|
||||
return _json({
|
||||
"open_tasks": open_tasks,
|
||||
"reviews_of_my_work": reviews,
|
||||
"open_tickets": open_tickets,
|
||||
"total": len(open_tasks) + len(reviews) + len(open_tickets),
|
||||
})
|
||||
except Exception as e:
|
||||
return _json({"error": str(e)})
|
||||
|
||||
|
||||
def fabric_curate(args: dict, **kwargs) -> str:
|
||||
entry_id = args.get("entry_id", "").strip()
|
||||
training_value = args.get("training_value", "").strip()
|
||||
if not entry_id or training_value not in ("high", "normal", "low"):
|
||||
return _json({"error": "Need entry_id and training_value (high/normal/low)"})
|
||||
try:
|
||||
result = state.curate_entry(entry_id, training_value)
|
||||
return _json(result)
|
||||
except Exception as e:
|
||||
return _json({"error": str(e)})
|
||||
|
||||
|
||||
def fabric_export(args: dict, **kwargs) -> str:
|
||||
try:
|
||||
result = state.export_training(mode=args.get("mode", "normal"))
|
||||
result.pop("_training_data", None)
|
||||
result.pop("training_data_path", None)
|
||||
return _json(result)
|
||||
except Exception as e:
|
||||
return _json({"error": str(e)})
|
||||
|
||||
|
||||
def fabric_train(args: dict, **kwargs) -> str:
|
||||
try:
|
||||
result = state.start_training(
|
||||
model=args.get("model"),
|
||||
suffix=args.get("suffix"),
|
||||
epochs=args.get("epochs", 3),
|
||||
batch_size=args.get("batch_size"),
|
||||
learning_rate=args.get("learning_rate"),
|
||||
checkpoints=args.get("n_checkpoints"),
|
||||
mode=args.get("mode"),
|
||||
min_pairs=args.get("min_pairs", 10),
|
||||
)
|
||||
return _json(result)
|
||||
except Exception as e:
|
||||
return _json({"error": str(e)})
|
||||
|
||||
|
||||
def fabric_train_status(args: dict, **kwargs) -> str:
|
||||
try:
|
||||
result = state.check_training(job_id=args.get("job_id"))
|
||||
return _json(result)
|
||||
except Exception as e:
|
||||
return _json({"error": str(e)})
|
||||
|
||||
|
||||
def fabric_models(args: dict, **kwargs) -> str:
|
||||
try:
|
||||
registry = state.list_models()
|
||||
return _json(registry)
|
||||
except Exception as e:
|
||||
return _json({"error": str(e)})
|
||||
|
||||
|
||||
def fabric_eval(args: dict, **kwargs) -> str:
|
||||
candidate = args.get("candidate_model", "").strip()
|
||||
if not candidate:
|
||||
return _json({"error": "candidate_model is required"})
|
||||
try:
|
||||
result = state.run_eval(
|
||||
candidate_model=candidate,
|
||||
base_model=args.get("base_model"),
|
||||
sample_count=args.get("sample_count", 10),
|
||||
)
|
||||
return _json(result)
|
||||
except Exception as e:
|
||||
return _json({"error": str(e)})
|
||||
|
||||
|
||||
def fabric_switch_model(args: dict, **kwargs) -> str:
|
||||
model_id = args.get("model_id", "").strip()
|
||||
if not model_id:
|
||||
return _json({"error": "model_id is required"})
|
||||
try:
|
||||
result = state.switch_model(
|
||||
model_id=model_id,
|
||||
min_eval_score=args.get("min_eval_score", 0.7),
|
||||
)
|
||||
return _json(result)
|
||||
except Exception as e:
|
||||
return _json({"error": str(e)})
|
||||
|
||||
|
||||
def fabric_rollback_model(args: dict, **kwargs) -> str:
|
||||
try:
|
||||
result = state.rollback_model()
|
||||
return _json(result)
|
||||
except Exception as e:
|
||||
return _json({"error": str(e)})
|
||||
|
||||
|
||||
def fabric_brief(args: dict, **kwargs) -> str:
|
||||
try:
|
||||
result = state.build_brief()
|
||||
return _json(result)
|
||||
except Exception as e:
|
||||
return _json({"error": str(e)})
|
||||
|
||||
|
||||
def fabric_telemetry(args: dict, **kwargs) -> str:
|
||||
try:
|
||||
result = state.get_telemetry(last_n=args.get("last_n", 50))
|
||||
return _json(result)
|
||||
except Exception as e:
|
||||
return _json({"error": str(e)})
|
||||
|
||||
|
||||
def fabric_init_obsidian(args: dict, **kwargs) -> str:
|
||||
try:
|
||||
from . import obsidian
|
||||
result = obsidian.init_obsidian(state.FABRIC_DIR)
|
||||
return _json(result)
|
||||
except Exception as e:
|
||||
return _json({"error": str(e)})
|
||||
|
||||
|
||||
def fabric_report(args: dict, **kwargs) -> str:
|
||||
try:
|
||||
result = state.build_weekly_report()
|
||||
return _json(result)
|
||||
except Exception as e:
|
||||
return _json({"error": str(e)})
|
||||
|
|
@ -0,0 +1,119 @@
|
|||
# Infrastructure
|
||||
|
||||
> Docker services, cronjobs, and environment configuration that support the 6 memory layers.
|
||||
|
||||
## Docker Services
|
||||
|
||||
The vector and pipeline layers run as Docker containers:
|
||||
|
||||
```yaml
|
||||
# docker-compose.yml
|
||||
services:
|
||||
qdrant:
|
||||
image: qdrant/qdrant:v1.17.1
|
||||
ports:
|
||||
- "6333:6333"
|
||||
- "6334:6334"
|
||||
volumes:
|
||||
- ./qdrant_data:/qdrant/storage
|
||||
restart: unless-stopped
|
||||
|
||||
redis:
|
||||
image: redis:7-alpine
|
||||
ports:
|
||||
- "6379:6379"
|
||||
command: redis-server --requirepass ${REDIS_PASSWORD}
|
||||
restart: unless-stopped
|
||||
|
||||
worker:
|
||||
build: ./worker
|
||||
depends_on:
|
||||
- qdrant
|
||||
- redis
|
||||
environment:
|
||||
- QDRANT_URL=http://qdrant:6333
|
||||
- REDIS_URL=redis://:${REDIS_PASSWORD}@redis:6379/0
|
||||
- EMBEDDING_DIMS=${EMBEDDING_DIMS:-4096}
|
||||
- COLLECTION_NAME=${COLLECTION_NAME:-knowledge_base}
|
||||
- OPENROUTER_API_KEY=${OPENROUTER_API_KEY}
|
||||
restart: unless-stopped
|
||||
```
|
||||
|
||||
**Key configuration:**
|
||||
- `EMBEDDING_DIMS=4096` — must match Qdrant collection schema
|
||||
- `COLLECTION_NAME=knowledge_base` — target collection for all wiki ingestion
|
||||
- Redis password required — set in `.env`, used by both Redis container and worker
|
||||
|
||||
## Cronjobs
|
||||
|
||||
| Job | Recommended schedule | What it does |
|
||||
|-----|---------------------|--------------|
|
||||
| **wiki-continuous-ingest** | Hourly (:00) | SHA-256 diff detection → embed new wiki files → Qdrant |
|
||||
| **wiki-raw-ingest-monitor** | 2x/week | Read raw/ files → extract concepts/entities/comparisons → create wiki pages |
|
||||
| **vault-curator-weekly** | Weekly | Phase 1 (frontmatter enrichment) + Phase 2 (semantic linking) + Phase 3 (INDEX.md) |
|
||||
| **decay-scanner** | Weekly | Archive low-importance, aged AI content from Qdrant |
|
||||
| **dlq-auto-report** | Every 6h | Dead letter queue monitoring and reporting |
|
||||
| **maas-heartbeat** | Every 6h | Infrastructure health check |
|
||||
| **holographic-memory-backup** | Weekly | Backup of workspace memory files and databases |
|
||||
| **monitor-openrouter-balance** | Daily | OpenRouter credit balance check |
|
||||
|
||||
**Interaction between jobs:**
|
||||
- `wiki-raw-ingest-monitor` creates new wiki pages → next `wiki-continuous-ingest` picks them up and sends to Qdrant
|
||||
- `vault-curator-weekly` enriches ALL vault files — adds frontmatter, semantic links, and INDEX.md
|
||||
|
||||
## Environment Variables
|
||||
|
||||
### Required
|
||||
|
||||
| Variable | Purpose | Example |
|
||||
|----------|---------|---------|
|
||||
| `FABRIC_DIR` | Where Icarus writes fabric entries | `/home/your-user/vault/fabric` |
|
||||
| `OPENROUTER_API_KEY` | Embedding + LLM extraction | `sk-or-...` |
|
||||
| `REDIS_PASSWORD` | Redis authentication | (generated) |
|
||||
|
||||
### Strongly recommended
|
||||
|
||||
| Variable | Default | Recommended | Why |
|
||||
|----------|---------|-------------|-----|
|
||||
| `ICARUS_EXTRACTION_MAX_TOKENS` | 1024 | **4096** | 1024 causes fabric truncation |
|
||||
| `ICARUS_EXTRACTION_MODEL` | deepseek-v4-flash | same | Any OpenRouter chat model works |
|
||||
| `EMBEDDING_DIMS` | varies | **4096** | Must match Qdrant collection schema |
|
||||
|
||||
### Optional
|
||||
|
||||
| Variable | Purpose |
|
||||
|----------|---------|
|
||||
| `ICARUS_OBSIDIAN=1` | Enable Obsidian wikilinks and daily notes |
|
||||
| `OBSIDIAN_VAULT_PATH` | Vault root (if fabric is a subfolder) |
|
||||
| `ICARUS_RESULT_MAX_CHARS` | Fallback truncation limit (default 500) |
|
||||
| `ICARUS_TASK_MAX_CHARS` | Fallback task truncation (default 300) |
|
||||
| `TOGETHER_API_KEY` | For training/eval tools |
|
||||
| `OPENROUTER_FULL_API_KEY` | Alternative key for LLM extraction |
|
||||
| `OPENROUTER_DS_API_KEY` | Alternative key for LLM extraction |
|
||||
| `CURATOR_LOG_LEVEL` | Logging level for Vault Curator |
|
||||
| `VAULT_PATH` | Path to vault root for Vault Curator |
|
||||
|
||||
## File locations
|
||||
|
||||
| Component | Path |
|
||||
|-----------|------|
|
||||
| Workspace memory | `$HERMES_HOME/memories/` |
|
||||
| Session DB | `$HERMES_HOME/state.db` |
|
||||
| Fact store DB | `$HERMES_HOME/memory_store.db` |
|
||||
| Icarus plugin | `$HERMES_HOME/plugins/icarus/` |
|
||||
| Fabric entries | `$FABRIC_DIR` |
|
||||
| Wiki files | `$VAULT_PATH/wiki/` |
|
||||
| Qdrant data | `./qdrant_data/` (Docker volume) |
|
||||
| Docker compose | Project root |
|
||||
| Cron scripts | Project scripts directory |
|
||||
|
||||
## System requirements
|
||||
|
||||
| Resource | Minimum | Recommended |
|
||||
|----------|---------|-------------|
|
||||
| RAM | 8 GB | 16 GB (Qdrant + Redis + ARQ worker) |
|
||||
| Disk | 20 GB | 50 GB (Qdrant vectors + wiki files) |
|
||||
| Docker | 24.0+ | Latest stable |
|
||||
| Python | 3.11+ | 3.11 (tested) |
|
||||
| Hermes Agent | 0.14.0+ | 0.15.2 (tested) |
|
||||
| Qdrant | 1.17+ | 1.17.1 (tested) |
|
||||
|
|
@ -0,0 +1,39 @@
|
|||
# Layer 1 — Workspace Memory
|
||||
|
||||
> **Files:** `MEMORY.md`, `USER.md`, `CREATIVE.md` in `$HERMES_HOME/memories/`
|
||||
> **Injection:** System prompt, every turn
|
||||
> **Persistence:** Markdown on disk
|
||||
|
||||
## What it stores
|
||||
|
||||
| File | Writer | Format | Purpose |
|
||||
|------|--------|--------|---------|
|
||||
| **MEMORY.md** | `memory` tool | `§`-delimited entries | Agent's durable memory: environment facts, tool quirks, project conventions |
|
||||
| **USER.md** | Manual (user) | Markdown | Static user profile: who the user is, preferences, workflow |
|
||||
| **CREATIVE.md** | Icarus plugin (`state.py`) | Markdown headers + bullets | Agent's creative state: learnings, open questions, cycle counter |
|
||||
|
||||
## The `§` delimiter story
|
||||
|
||||
The `memory` tool uses `§` (paragraph sign, U+00A7) as an entry delimiter in MEMORY.md:
|
||||
|
||||
```
|
||||
Entry about Qdrant named vectors
|
||||
§
|
||||
Entry about Tailscale configuration
|
||||
§
|
||||
Entry about project conventions
|
||||
```
|
||||
|
||||
**What broke:** The Icarus plugin's `write_memory_file()` used to overwrite MEMORY.md with `.write_text()` on every session end, destroying all `§`-delimited entries. Two writers, one file, incompatible formats.
|
||||
|
||||
**Fix:** Icarus now writes to **CREATIVE.md** instead. Two writers, two files, zero conflicts. This fix is in our [Icarus fork](https://github.com/ClaudioDrews/icarus-plugin).
|
||||
|
||||
## Configuration
|
||||
|
||||
No additional configuration needed — these files are always injected by Hermes. To change injection behavior, edit `SOUL.md` or `rulebook.md`.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **systemd does not expand `~`:** Paths in `.env` read by the gateway must use absolute paths
|
||||
- **Never edit MEMORY.md manually:** Use `memory(action='add')` — the tool writes atomic `§`-delimited entries
|
||||
- **Icarus conflict:** If MEMORY.md ever shows `cycles:` or markdown headers, Icarus overwrote it. Delete and restore from backup, then ensure Icarus is writing to CREATIVE.md
|
||||
|
|
@ -0,0 +1,49 @@
|
|||
# Layer 2 — Session Database
|
||||
|
||||
> **File:** `$HERMES_HOME/state.db` (SQLite + FTS5)
|
||||
> **Tool:** `session_search`
|
||||
> **Writer:** Hermes Gateway (automatic)
|
||||
|
||||
## What it stores
|
||||
|
||||
Every message sent and received by the agent is logged automatically by the gateway process:
|
||||
|
||||
```
|
||||
sessions table:
|
||||
session_id, title, source (telegram/cli/cron), started_at, ended_at
|
||||
|
||||
messages table:
|
||||
id, session_id, role (user/assistant/tool), content, timestamp
|
||||
|
||||
messages_fts (FTS5 virtual table):
|
||||
Full-text index over message content
|
||||
```
|
||||
|
||||
## How the agent uses it
|
||||
|
||||
The `session_search` tool provides three access patterns:
|
||||
|
||||
1. **Discovery** — `session_search(query="auth refactor")` → FTS5 search across all sessions, returns top matches with context windows
|
||||
2. **Scroll** — `session_search(session_id="...", around_message_id=12345)` → read a specific session
|
||||
3. **Browse** — `session_search()` → recent sessions chronologically
|
||||
|
||||
## Context injection (Icarus enhancement)
|
||||
|
||||
The Icarus fork adds `_search_sessions()` in `hooks.py` — FTS5 search during `pre_llm_call` that injects relevant past conversations into the system prompt. This means the agent doesn't need to explicitly call `session_search` — relevant history finds it automatically.
|
||||
|
||||
Key implementation details:
|
||||
- FTS5 query: OR of tokens ≥ 4 characters from user message
|
||||
- Excludes current session
|
||||
- Deduplicates by session in Python
|
||||
- Labeled `[sessions]` in prompt for source transparency
|
||||
- Opens DB read-only: `sqlite3.connect("file:{db}?mode=ro", uri=True)`
|
||||
|
||||
## Configuration
|
||||
|
||||
No configuration needed — the gateway manages this automatically.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **FTS5 is lexical, not semantic:** Queries in one language may not match content in another with different wording
|
||||
- **`snippet()` + `GROUP BY` incompatibility:** Cannot combine in same SQL query — fetch ranked rows, dedup in Python
|
||||
- **`started_at` is Unix timestamp (float):** Not an ISO string — use `datetime.fromtimestamp()` for display
|
||||
|
|
@ -0,0 +1,53 @@
|
|||
# Layer 3 — Structured Facts (Holographic Memory)
|
||||
|
||||
> **File:** `$HERMES_HOME/memory_store.db` (SQLite + HRR + FTS5 + trust scoring)
|
||||
> **Tool:** `fact_store` (CRUD: add, search, probe, reason, update, remove, contradict)
|
||||
> **Feedback:** `fact_feedback` (helpful/unhelpful)
|
||||
|
||||
## What it stores
|
||||
|
||||
Durable, structured facts with entity resolution and trust scoring:
|
||||
|
||||
```
|
||||
facts table:
|
||||
fact_id, content, category (user_pref|project|tool|general),
|
||||
entities (JSON array), tags, trust_score, retrieval_count,
|
||||
helpful_count, created_at, last_accessed_at
|
||||
```
|
||||
|
||||
## How the agent uses it
|
||||
|
||||
6 actions:
|
||||
|
||||
| Action | What it does | Example |
|
||||
|--------|-------------|---------|
|
||||
| `add` | Store a new fact | `fact_store(action='add', content='...', entities=['Qdrant'])` |
|
||||
| `search` | Keyword lookup (FTS5) | `fact_store(action='search', query='Qdrant named vectors')` |
|
||||
| `probe` | All facts about an entity | `fact_store(action='probe', entity='Docker')` |
|
||||
| `reason` | Compositional: facts connected to multiple entities | `fact_store(action='reason', entities=['Qdrant', 'Docker'])` |
|
||||
| `contradict` | Find conflicting claims | `fact_store(action='contradict')` |
|
||||
| `update/remove` | CRUD maintenance | `fact_store(action='update', fact_id=87, trust_delta=0.1)` |
|
||||
|
||||
## Trust scoring
|
||||
|
||||
The system tracks which facts are actually useful:
|
||||
|
||||
- `retrieval_count` — how many times the fact was retrieved
|
||||
- `helpful_count` — how many times it was marked helpful
|
||||
- `trust_score` — calculated from ratio + Bayesian prior (starts at 0.50)
|
||||
|
||||
**Critical rule:** When you retrieve a fact via `probe`, `search`, or `reason` and reference it in your response, you MUST call `fact_feedback` in the same turn. Without feedback, `trust_score` is ornamental and fact quality degrades silently.
|
||||
|
||||
## Context injection (Icarus enhancement)
|
||||
|
||||
The Icarus fork adds `_search_facts()` — FTS5 search during `pre_llm_call` that injects relevant facts on the **first turn only** (to avoid per-turn cost). Labeled `[facts]` in prompt.
|
||||
|
||||
## Configuration
|
||||
|
||||
No configuration needed. DB path is managed by Hermes.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **`retrieval_count` was broken for `probe()`/`reason()`** — only `search()` incremented it. Fixed in current code
|
||||
- **Trust scores need feedback to move:** Without `fact_feedback` calls, every fact stays at 0.50
|
||||
- **HRR is not exposed to the agent directly:** It powers entity resolution internally — the agent sees keyword search + entity linking, not the holographic vectors
|
||||
|
|
@ -0,0 +1,96 @@
|
|||
# Layer 4 — Fabric (Cross-Session Memory)
|
||||
|
||||
> **Plugin:** Icarus ([bundled with Memory OS](../icarus/))
|
||||
> **Storage:** `$FABRIC_DIR` (markdown files with YAML frontmatter)
|
||||
> **Tools:** 16 (fabric_recall, fabric_write, fabric_brief, etc.)
|
||||
> **Hooks:** 4 (on_session_start, pre_llm_call, post_llm_call, on_session_end)
|
||||
|
||||
## What it stores
|
||||
|
||||
Structured, cross-session entries — each session end produces one or more markdown files:
|
||||
|
||||
```markdown
|
||||
---
|
||||
id: "29914be5"
|
||||
type: "resolution"
|
||||
summary: "Fixed MEMORY.md corruption from dual-writer conflict"
|
||||
training_value: "high"
|
||||
status: "completed"
|
||||
---
|
||||
## Context
|
||||
...
|
||||
## Action/Decision
|
||||
...
|
||||
## Outcome
|
||||
...
|
||||
```
|
||||
|
||||
**Entry types:** decision, resolution, note, code-session, session, review, research, task
|
||||
|
||||
## How the agent uses it
|
||||
|
||||
| Tool | What it does |
|
||||
|------|-------------|
|
||||
| `fabric_recall` | Ranked retrieval from shared memory |
|
||||
| `fabric_write` | Write entries with linking, evidence, and handoff fields |
|
||||
| `fabric_search` | Keyword grep across all entries |
|
||||
| `fabric_pending` | Show work assigned to this agent |
|
||||
| `fabric_brief` | Daily brief: pending work, recent activity |
|
||||
| `fabric_curate` | Set training value (high/normal/low) |
|
||||
| `fabric_export` | Export training pairs for fine-tuning |
|
||||
| `fabric_train` | Start fine-tune job on Together AI |
|
||||
| `fabric_models` | List trained replacement models |
|
||||
|
||||
## Key enhancements over upstream
|
||||
|
||||
| Enhancement | What it fixes |
|
||||
|-------------|---------------|
|
||||
| **LLM-powered extraction** | Replaces upstream's `text[:500]` truncation with structured JSON extraction via OpenRouter |
|
||||
| **Multi-source context injection** | Qdrant + sessions + facts injected automatically (upstream: fabric only) |
|
||||
| **MEMORY.md → CREATIVE.md** | Fixes `§` delimiter corruption from dual-writer conflict |
|
||||
| **Backtick sanitization** | Prevents orphaned backticks in learning lines |
|
||||
| **System injection filter** | Prevents orchestrator preambles from being captured as tasks |
|
||||
| **Social closer detection** | Skips trivial messages — avoids wasting embeddings on small talk |
|
||||
|
||||
See the [bundled Icarus source](../icarus/) for full details.
|
||||
|
||||
## Context injection flow
|
||||
|
||||
```
|
||||
pre_llm_call(user_message):
|
||||
├── _is_social_close(message)?
|
||||
│ └── Yes → skip all search-based injection
|
||||
├── fabric_recall(query) → [fabric] results
|
||||
├── _search_qdrant(query, threshold=0.55) → [qdrant] results
|
||||
├── _search_sessions(query) → [sessions] results
|
||||
└── _search_facts(query) → [facts] results (first turn only)
|
||||
|
||||
Per-source dedup:
|
||||
_injected_fabric, _injected_qdrant, _injected_sessions
|
||||
→ Reset on session start
|
||||
→ Prevents same result injected twice in one session
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
```bash
|
||||
# Required
|
||||
FABRIC_DIR=/absolute/path/to/fabric
|
||||
OPENROUTER_API_KEY=sk-or-...
|
||||
|
||||
# Strongly recommended
|
||||
ICARUS_EXTRACTION_MAX_TOKENS=4096
|
||||
ICARUS_EXTRACTION_MODEL=deepseek/deepseek-v4-flash
|
||||
|
||||
# Optional (for Obsidian integration)
|
||||
ICARUS_OBSIDIAN=1
|
||||
OBSIDIAN_VAULT_PATH=/absolute/path/to/vault
|
||||
```
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **`ICARUS_EXTRACTION_MAX_TOKENS` is frozen at import time** — changing `.env` requires gateway restart
|
||||
- **DeepSeek + `response_format: json_object` = `content: null`** — the fork uses prompt-based JSON + `_parse_json_robust()` instead
|
||||
- **`FABRIC_DIR` must be absolute path** — systemd does not expand `~`
|
||||
- **Obsidian is optional** — Icarus writes plain markdown, Obsidian just reads it
|
||||
- **Gateway restart required** after editing `hooks.py` or changing env vars
|
||||
|
|
@ -0,0 +1,90 @@
|
|||
# Layer 5 — Vector Database (Qdrant)
|
||||
|
||||
> **Service:** Qdrant 1.17+ (Docker)
|
||||
> **Collection:** `knowledge_base` (4096d Cosine + BM25 sparse)
|
||||
> **Embedding:** Qwen3-Embedding-8B via OpenRouter
|
||||
> **Endpoint:** `http://localhost:6333`
|
||||
|
||||
## What it stores
|
||||
|
||||
All knowledge that benefits from semantic search — wiki pages, session transcripts, raw documents, technical references. Content is ingested via the continuous ingest pipeline (hourly) and the wiki agent (scheduled).
|
||||
|
||||
## How the agent uses it
|
||||
|
||||
**Two access patterns:**
|
||||
|
||||
1. **Explicit** — `qdrant_search(query="memory architecture", top_k=3)` → agent calls the tool directly
|
||||
2. **Automatic** — Icarus `pre_llm_call` injects relevant Qdrant results into the system prompt every turn (via `_search_qdrant()`)
|
||||
|
||||
**Search uses 4-level fallback cascade:**
|
||||
|
||||
```
|
||||
1. Hybrid: dense (4096d cosine) + sparse (BM25) → RRF fusion
|
||||
2. Dense-only: if sparse fails → pure vector search
|
||||
3. Lexical: if Qdrant offline → markdown file search in vault
|
||||
4. SQLite: if vault inaccessible → keyword search in lineage table
|
||||
5. None: all exhausted → return empty (fail-open)
|
||||
```
|
||||
|
||||
## Collection schema
|
||||
|
||||
```json
|
||||
{
|
||||
"vectors": {
|
||||
"dense": { "size": 4096, "distance": "Cosine" }
|
||||
},
|
||||
"sparse_vectors": {
|
||||
"sparse": { "index": { "on_disk": false } }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Named vectors:** Points must use `{"dense": [...], "sparse": SparseVector(...)}`. Must match schema exactly — unnamed vectors are rejected silently.
|
||||
|
||||
## Decay and dedup
|
||||
|
||||
| Mechanism | Schedule | What it does |
|
||||
|-----------|----------|--------------|
|
||||
| **Decay scanner** | Weekly | Archives AI-generated points with low importance and high age. Exempts human-generated content and high-importance (>0.7) points. Formula: `decay_score = exp(-ln(2) * age_days / half_life)` |
|
||||
| **Semantic dedup** | Monthly | Merges near-duplicate points (cosine > 0.92). Tags union, priority by source_type. |
|
||||
|
||||
## Context injection (Icarus enhancement)
|
||||
|
||||
```python
|
||||
_search_qdrant(query, top_k=2, threshold=0.55):
|
||||
1. Embed user message via context_enhancer pipeline
|
||||
2. search_with_fallback() → 4-level cascade
|
||||
3. Results labeled [qdrant] in system prompt
|
||||
4. Per-session dedup by point ID
|
||||
```
|
||||
|
||||
**Key injection pitfall:** `context_enhancer.py` reads `OPENROUTER_API_KEY` (singular), but some environments use split keys. Before importing, inject: `os.environ["OPENROUTER_API_KEY"] = resolved_key`.
|
||||
|
||||
**Social closer gate:** Trivial messages ("ok", "thanks", emoji-only) skip Qdrant search entirely — no point embedding small talk.
|
||||
|
||||
## Embedding pipeline
|
||||
|
||||
```
|
||||
File in $VAULT_PATH/wiki/
|
||||
│
|
||||
▼
|
||||
wiki-continuous-ingest (hourly cron)
|
||||
│ SHA-256 diff detection
|
||||
▼
|
||||
Redis queue (ARQ job)
|
||||
│
|
||||
▼
|
||||
ARQ Worker (Docker)
|
||||
│ embed via Qwen3-Embedding-8B (OpenRouter)
|
||||
│ get_sparse_embedding() → BM25 (fastembed, local)
|
||||
▼
|
||||
Qdrant upsert (with dedup check)
|
||||
```
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Named vectors are mandatory in Qdrant 1.17+:** `upsert` requires `"vector": {"dense": vec}`, plain vectors are rejected silently
|
||||
- **`AsyncQdrantClient.search()` doesn't exist in qdrant-client 1.18:** Use REST API `POST /collections/{name}/points/search`
|
||||
- **Embedding dimension mismatch is silent:** If env says 1024 but collection is 4096, OpenRouter truncates via Matryoshka — vectors are valid but degraded
|
||||
- **Decay scanner is inert without payload metadata:** Requires `importance_score`, `last_accessed_at`, `confidence_score` in point payloads. Missing → all points get `decay_score=1.0` → nothing archived
|
||||
- **Isolate persona/relational collections from decay/dedup** — they preserve temporal variation
|
||||
|
|
@ -0,0 +1,84 @@
|
|||
# Layer 6 — LLM Wiki
|
||||
|
||||
> **Location:** `$VAULT_PATH/wiki/`
|
||||
> **Pipeline:** wiki-raw-ingest-monitor (scheduled) + wiki-continuous-ingest (hourly)
|
||||
> **Qdrant:** All files ingested into `knowledge_base`
|
||||
|
||||
## Directory structure
|
||||
|
||||
```
|
||||
$VAULT_PATH/wiki/
|
||||
├── raw/ Source documents
|
||||
│ ├── articles/ (analyses, dossiers, tutorials)
|
||||
│ ├── releases/ (software release notes, PR trackers)
|
||||
│ ├── projects/ (project architecture, reports)
|
||||
│ └── ... (any other document source)
|
||||
├── concepts/ Extracted ideas and patterns
|
||||
├── entities/ Concrete things (tools, models, projects, people)
|
||||
├── comparisons/ Side-by-side analyses
|
||||
├── _meta/ Schema, templates, taxonomy
|
||||
├── _archive/ Deprecated pages
|
||||
├── index.md Master catalog with one-line summaries
|
||||
├── SCHEMA.md Constitution — what merits a page, how to link, what tags to use
|
||||
└── log.md Logbook — every curation session recorded
|
||||
```
|
||||
|
||||
## The two pipelines
|
||||
|
||||
| Pipeline | Trigger | What it does |
|
||||
|----------|---------|--------------|
|
||||
| **Wiki Agent** (curation) | Scheduled cron | Reads `raw/` files, extracts concepts/entities/comparisons, creates structured wiki pages |
|
||||
| **Continuous Ingest** (Qdrant) | Hourly cron | SHA-256 diff detection, embeds new/modified files, upserts to `knowledge_base` |
|
||||
|
||||
They're independent: the Wiki Agent builds the curated knowledge graph; Continuous Ingest ensures Qdrant stays in sync.
|
||||
|
||||
## Wiki Agent pipeline
|
||||
|
||||
```
|
||||
1. Cron triggers wiki-raw-ingest-monitor (scheduled)
|
||||
2. check_raw_ingest.py → detects new files in raw/
|
||||
3. Agent reads SCHEMA.md, index.md, log.md
|
||||
4. For each new file:
|
||||
a. LLM analyzes content
|
||||
b. Decides: concept? entity? comparison? skip?
|
||||
c. Creates page in appropriate directory
|
||||
d. Frontmatter: type, tags (from closed taxonomy), sources (linking back to raw/)
|
||||
5. Updates index.md and log.md
|
||||
6. Lint: validates frontmatter, wikilinks, index coverage
|
||||
```
|
||||
|
||||
## Continuous Ingest pipeline
|
||||
|
||||
```
|
||||
1. Cron triggers wiki-continuous-ingest (hourly)
|
||||
2. wiki_continuous_ingest.py:
|
||||
a. Scans $VAULT_PATH/wiki/ for *.md files
|
||||
b. Computes SHA-256 hash for each file
|
||||
c. Compares with state file
|
||||
d. New or modified → enqueues ARQ job in Redis
|
||||
3. ARQ Worker (Docker):
|
||||
a. process_wiki_file → reads file content
|
||||
b. parse_frontmatter → extracts metadata
|
||||
c. get_embedding() → Qwen3-Embedding-8B (4096d)
|
||||
d. get_sparse_embedding() → BM25 (fastembed, local)
|
||||
e. upsert_with_dedup() → Qdrant knowledge_base
|
||||
```
|
||||
|
||||
**State file:** JSON file tracking `{file_path, sha256_hash, ingested_at}` for each indexed file. Prevents re-ingestion of unchanged files.
|
||||
|
||||
## Page quality standards
|
||||
|
||||
Every wiki page must have:
|
||||
- Valid YAML frontmatter with `type`, `tags`, `sources`, `confidence`
|
||||
- At least one wikilink to another wiki page
|
||||
- A one-line summary (used in index.md)
|
||||
- `sources` linking back to the raw file(s) it was extracted from
|
||||
|
||||
**Closed taxonomy:** Tags come from a fixed set of categories defined in SCHEMA.md. No ad-hoc tags allowed.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Wiki Agent and Vault Curator are different cronjobs** — don't conflate them. Wiki Agent creates pages; Vault Curator enriches frontmatter + adds semantic links to existing files
|
||||
- **raw/ files are source material, not curated knowledge** — they feed the pipeline but aren't themselves structured wiki pages
|
||||
- **SCHEMA.md is the constitution** — any change to page structure or taxonomy must be reflected there first
|
||||
- **log.md is the audit trail** — if a page looks wrong, check log.md to see which session created it and why
|
||||
|
|
@ -0,0 +1,97 @@
|
|||
# Modifications to Hermes Core
|
||||
|
||||
Memory OS requires changes to core Hermes files that govern agent behavior. These modifications ensure the agent trusts its injected memory as authoritative rather than re-discovering known facts.
|
||||
|
||||
## SOUL.md — Ground Truth hierarchy
|
||||
|
||||
Add a new level 2 to the Ground Truth hierarchy in `SOUL.md`:
|
||||
|
||||
```markdown
|
||||
## Ground Truth
|
||||
|
||||
Authoritative sources, in priority order:
|
||||
|
||||
1. **Terminal output** — stdout, stderr, exit codes. Never reinterpret.
|
||||
2. **Injected memory** — qdrant, fabric, sessions, facts. Ground truth for documented
|
||||
knowledge. When injected memory contradicts other sources, injected memory wins
|
||||
because it represents verified, persisted knowledge from prior sessions.
|
||||
3. **Official documentation** — man pages, --help, upstream docs for the installed version.
|
||||
4. **Training knowledge** — reference only. Always verify against sources 1-3 before acting.
|
||||
```
|
||||
|
||||
**Why this matters:** Without level 2, the agent treats facts already persisted in Qdrant/fabric/sessions as less authoritative than documentation, causing it to re-discover known information. An agent that has Tailscale configuration in `fact_store` should not spend time re-verifying it against `man tailscale`.
|
||||
|
||||
## SOUL.md — Context injection convention
|
||||
|
||||
Add source labeling conventions:
|
||||
|
||||
```markdown
|
||||
## Context injection convention
|
||||
|
||||
When context is injected into the system prompt, it is labeled by source:
|
||||
- [fabric] — from Icarus fabric recall
|
||||
- [qdrant] — from Qdrant semantic search
|
||||
- [sessions] — from session history FTS5
|
||||
- [facts] — from holographic fact store
|
||||
|
||||
Injected memory takes priority level 2 in Ground Truth. This means:
|
||||
"You already know this. Don't re-discover it. Use it."
|
||||
```
|
||||
|
||||
## SOUL.md — Agent identity
|
||||
|
||||
Add clear identity boundaries:
|
||||
|
||||
```markdown
|
||||
## You are not
|
||||
|
||||
You are not a search engine. You are not a chatbot. You are not here to produce
|
||||
plausible-sounding output. You are an agent that executes real work in real
|
||||
environments, where errors have real costs. Treat every action accordingly.
|
||||
```
|
||||
|
||||
## rulebook.md — Mandatory verifications
|
||||
|
||||
Add to the rulebook:
|
||||
|
||||
```markdown
|
||||
## Mandatory Verifications
|
||||
|
||||
Before reporting a fact as true, verify:
|
||||
1. **Runtime evidence** — terminal output, file existence, process status
|
||||
2. **Injected memory** — qdrant_search, fact_store probe, fabric_recall
|
||||
3. **Documentation** — man pages, official docs for installed version
|
||||
4. **Training knowledge** — never cite without verifying against 1-3
|
||||
```
|
||||
|
||||
## rulebook.md — Memory architecture
|
||||
|
||||
Add a section documenting the 6-layer architecture so the agent knows where to find information:
|
||||
|
||||
```markdown
|
||||
## Memory Architecture
|
||||
|
||||
The agent has 6 layers of persistent memory:
|
||||
|
||||
| Layer | What it stores | How to access |
|
||||
|-------|---------------|---------------|
|
||||
| 1. Workspace | MEMORY.md, USER.md, CREATIVE.md | Always in system prompt |
|
||||
| 2. Sessions | state.db (FTS5) | session_search |
|
||||
| 3. Facts | memory_store.db (HRR) | fact_store |
|
||||
| 4. Fabric | $FABRIC_DIR (markdown) | fabric_recall, fabric_write |
|
||||
| 5. Qdrant | knowledge_base (4096d) | qdrant_search, auto-injection |
|
||||
| 6. Wiki | $VAULT_PATH/wiki/ | qdrant_search → knowledge_base |
|
||||
```
|
||||
|
||||
## Impact
|
||||
|
||||
Without these modifications:
|
||||
- Qdrant/fabric/session/fact injection still works technically
|
||||
- But the agent doesn't trust injected memory as authoritative
|
||||
- Result: agent re-discovers known facts, wastes tokens, makes redundant decisions
|
||||
|
||||
With these modifications:
|
||||
- Agent treats injected memory as ground truth (level 2)
|
||||
- Reduces redundant discovery work
|
||||
- Agent can reference prior decisions without re-litigating them
|
||||
- Cross-session continuity is real, not aspirational
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
# Memory OS — Host Scripts
|
||||
requests>=2.31.0
|
||||
aiohttp>=3.9.0
|
||||
arq>=0.28.0
|
||||
python-dotenv>=1.0.0
|
||||
pyyaml>=6.0
|
||||
qdrant-client>=1.17.0
|
||||
|
|
@ -0,0 +1,49 @@
|
|||
# Memory OS — Scripts
|
||||
|
||||
Standalone Python scripts that maintain the Qdrant vector database and wiki pipeline.
|
||||
|
||||
## Qdrant Maintenance
|
||||
|
||||
| Script | What it does | Run |
|
||||
|--------|-------------|-----|
|
||||
| `decay_scanner.py` | Archives low-importance, aged AI content based on half-life decay | Weekly cron |
|
||||
| `backfill_decay_metadata.py` | Populates missing `importance_score`, `last_accessed_at`, `confidence_score` in Qdrant points | Run once before enabling decay scanner |
|
||||
| `semantic_dedup.py` | Merges near-duplicate points (cosine >0.92) | Monthly cron |
|
||||
|
||||
## Context Injection
|
||||
|
||||
| Script | What it does | Used by |
|
||||
|--------|-------------|---------|
|
||||
| `context_enhancer.py` | Embedding pipeline: query → embed → search Qdrant (4-level fallback). Also provides BM25 sparse embedding via FastEmbed. | Icarus `pre_llm_call` hook |
|
||||
|
||||
## Wiki Pipeline
|
||||
|
||||
| Script | What it does | Run |
|
||||
|--------|-------------|-----|
|
||||
| `wiki_continuous_ingest.py` | SHA-256 diff detection: finds new/modified wiki files, enqueues ARQ jobs in Redis | Hourly cron |
|
||||
| `bulk_wiki_ingest.py` | One-shot bulk ingestion of all wiki files into Qdrant | After initial setup or collection rebuild |
|
||||
|
||||
## Quality Control
|
||||
|
||||
| Script | What it does | Run |
|
||||
|--------|-------------|-----|
|
||||
| `pre_validator.py` | Pre-flight validation of wiki documents: YAML frontmatter, required fields, link targets | Before ingestion |
|
||||
| `reflection_trigger.py` | Idle detection for ARQ worker — enqueues micro-reflection when queue is empty and within hourly budget | Every 5min cron |
|
||||
|
||||
## Monitoring
|
||||
|
||||
| Script | What it does | Run |
|
||||
|--------|-------------|-----|
|
||||
| `dlq_manager.py` | Dead letter queue monitoring and reporting | Every 6h cron |
|
||||
|
||||
## Environment variables
|
||||
|
||||
All scripts read configuration from environment variables. See `.env.example` in the project root for the full reference.
|
||||
|
||||
Key variables:
|
||||
- `OPENROUTER_API_KEY` — embeddings (required)
|
||||
- `WIKI_PATH` — wiki root directory (default: `~/vault/wiki`)
|
||||
- `COLLECTION_NAME` — Qdrant collection (default: `knowledge_base`)
|
||||
- `EMBEDDING_DIMS` — vector dimensions (default: 4096)
|
||||
- `REDIS_PASSWORD` — Redis auth (required for wiki ingest)
|
||||
- `QDRANT_URL` — Qdrant endpoint (default: `http://localhost:6333`)
|
||||
|
|
@ -0,0 +1,292 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Backfill decay metadata for Qdrant knowledge_base.
|
||||
|
||||
Populates missing fields (created_at, last_accessed_at, importance_score,
|
||||
confidence_score, archived) so the decay_scanner can actually work.
|
||||
|
||||
Modes:
|
||||
--dry-run Simulate everything, print stats, no writes (DEFAULT)
|
||||
--commit Actually update points via REST API
|
||||
--pilot N Process only N points (for validation)
|
||||
|
||||
Usage:
|
||||
python3 backfill_decay_metadata.py --dry-run --pilot 50
|
||||
python3 backfill_decay_metadata.py --dry-run
|
||||
python3 backfill_decay_metadata.py --commit
|
||||
|
||||
Heuristics:
|
||||
- created_at / last_accessed_at:
|
||||
Session points: from payload.timestamp (Unix epoch → ISO 8601)
|
||||
Wiki points: from file mtime if file exists, else now()
|
||||
- importance_score: 0.5 flat (conservative)
|
||||
- confidence_score:
|
||||
wiki-* → 0.85
|
||||
session → 0.70
|
||||
unknown → 0.75
|
||||
- archived: false (explicit)
|
||||
|
||||
Safety:
|
||||
- Dry-run by default — no writes without --commit
|
||||
- Only touches points MISSING the target fields (no overwrites)
|
||||
- Batched via REST API POST /points/payload
|
||||
- Uses same pattern as migrate_strength.py (2026-05-29)
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from urllib.request import Request, urlopen
|
||||
from urllib.error import URLError
|
||||
|
||||
# ─── Config ──────────────────────────────────────────────────────────────────
|
||||
QDRANT_URL = os.environ.get("QDRANT_URL", "http://localhost:6333")
|
||||
COLLECTION = os.environ.get("QDRANT_COLLECTION", os.environ.get("COLLECTION_NAME", "knowledge_base"))
|
||||
BATCH_SIZE = 200
|
||||
SCROLL_LIMIT = 200
|
||||
LOG_FILE = Path(os.environ.get("HERMES_LOGS_DIR", str(Path.home() / ".hermes" / "logs"))) / "decay_scanner.log"
|
||||
VAULT_ROOT = Path(os.environ.get("VAULT_PATH", "."))
|
||||
|
||||
# ─── Heuristics ──────────────────────────────────────────────────────────────
|
||||
CONFIDENCE_BY_SOURCE = {
|
||||
"wiki-concepts": 0.85,
|
||||
"wiki-entities": 0.85,
|
||||
"wiki-comparisons": 0.85,
|
||||
"wiki-raw": 0.85,
|
||||
"session": 0.70,
|
||||
}
|
||||
CONFIDENCE_DEFAULT = 0.75
|
||||
IMPORTANCE_SCORE = 0.5
|
||||
ARCHIVED = False
|
||||
|
||||
|
||||
def log(msg: str) -> None:
|
||||
"""Append timestamped message to log."""
|
||||
ts = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
|
||||
line = f"[backfill-decay] {ts} {msg}"
|
||||
print(line)
|
||||
LOG_FILE.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(LOG_FILE, "a") as f:
|
||||
f.write(line + "\n")
|
||||
|
||||
|
||||
def scroll_all() -> list[dict]:
|
||||
"""Scroll entire collection, returning all points with payloads."""
|
||||
points = []
|
||||
offset = None
|
||||
while True:
|
||||
body = {"limit": SCROLL_LIMIT, "with_payload": True, "with_vector": False}
|
||||
if offset:
|
||||
body["offset"] = offset
|
||||
req = Request(
|
||||
f"{QDRANT_URL}/collections/{COLLECTION}/points/scroll",
|
||||
data=json.dumps(body).encode(),
|
||||
headers={"Content-Type": "application/json"},
|
||||
)
|
||||
try:
|
||||
with urlopen(req, timeout=30) as resp:
|
||||
data = json.loads(resp.read().decode())
|
||||
except URLError as e:
|
||||
log(f"ERROR: scroll failed: {e}")
|
||||
break
|
||||
result = data.get("result", {})
|
||||
batch = result.get("points", [])
|
||||
points.extend(batch)
|
||||
offset = result.get("next_page_offset")
|
||||
if offset is None or len(batch) == 0:
|
||||
break
|
||||
return points
|
||||
|
||||
|
||||
def resolve_timestamp(point: dict) -> str:
|
||||
"""Return ISO 8601 string for created_at/last_accessed_at.
|
||||
|
||||
Session points: use payload.timestamp (Unix epoch float).
|
||||
Wiki points: locate file via payload.filename → file mtime.
|
||||
Fallback: now().
|
||||
"""
|
||||
pl = point.get("payload", {})
|
||||
ts_raw = pl.get("timestamp")
|
||||
|
||||
if ts_raw is not None:
|
||||
try:
|
||||
ts_float = float(ts_raw)
|
||||
return datetime.fromtimestamp(ts_float, tz=timezone.utc).isoformat()
|
||||
except (ValueError, TypeError, OSError):
|
||||
pass
|
||||
|
||||
# Wiki point — resolve filename to filesystem path
|
||||
filename = pl.get("filename", "")
|
||||
source = pl.get("source", "")
|
||||
if filename and source.startswith("wiki-"):
|
||||
# Map source to subfolder: wiki-concepts → concepts, wiki-entities → entities, etc.
|
||||
subfolder = source.replace("wiki-", "")
|
||||
candidate = VAULT_ROOT / "wiki" / subfolder / f"{filename}.md"
|
||||
try:
|
||||
mtime = candidate.stat().st_mtime
|
||||
return datetime.fromtimestamp(mtime, tz=timezone.utc).isoformat()
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
# Absolute fallback
|
||||
return datetime.now(timezone.utc).isoformat()
|
||||
|
||||
|
||||
def compute_confidence(point: dict) -> float:
|
||||
"""Heuristic confidence_score by source."""
|
||||
source = point.get("payload", {}).get("source", "")
|
||||
return CONFIDENCE_BY_SOURCE.get(source, CONFIDENCE_DEFAULT)
|
||||
|
||||
|
||||
def needs_backfill(point: dict) -> list[str]:
|
||||
"""Return list of fields this point is missing (that we backfill)."""
|
||||
TARGETS = ["created_at", "last_accessed_at", "importance_score", "confidence_score", "archived"]
|
||||
pl = point.get("payload", {})
|
||||
return [f for f in TARGETS if f not in pl]
|
||||
|
||||
|
||||
def build_payload(point: dict) -> dict:
|
||||
"""Build the payload fragment to backfill for this point."""
|
||||
ts = resolve_timestamp(point)
|
||||
return {
|
||||
"created_at": ts,
|
||||
"last_accessed_at": ts,
|
||||
"importance_score": IMPORTANCE_SCORE,
|
||||
"confidence_score": compute_confidence(point),
|
||||
"archived": ARCHIVED,
|
||||
}
|
||||
|
||||
|
||||
def upsert_batch(point_ids: list[str], payload: dict) -> bool:
|
||||
"""Update payload for a batch of points via REST API. Fail-open."""
|
||||
body = {"payload": payload, "points": point_ids}
|
||||
req = Request(
|
||||
f"{QDRANT_URL}/collections/{COLLECTION}/points/payload",
|
||||
data=json.dumps(body).encode(),
|
||||
headers={"Content-Type": "application/json"},
|
||||
)
|
||||
try:
|
||||
with urlopen(req, timeout=30) as resp:
|
||||
result = json.loads(resp.read().decode())
|
||||
return result.get("status") == "ok"
|
||||
except URLError as e:
|
||||
log(f"ERROR: upsert batch failed: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def run(dry_run: bool = True, pilot: int = 0, commit: bool = False):
|
||||
"""Main execution."""
|
||||
mode = "DRY-RUN" if dry_run else "COMMIT"
|
||||
log(f"=== BACKFILL DECAY METADATA ({mode}) ===")
|
||||
log(f"Collection: {COLLECTION}")
|
||||
log(f"Pilot: {pilot if pilot else 'all'}")
|
||||
|
||||
# ── Scroll all points ──
|
||||
log("Scrolling all points...")
|
||||
all_points = scroll_all()
|
||||
total = len(all_points)
|
||||
log(f"Total points: {total}")
|
||||
|
||||
# ── Identify points needing backfill ──
|
||||
to_update: list[tuple[str, dict]] = [] # (point_id, payload_fragment)
|
||||
skipped_already_have = 0
|
||||
stats_by_source = {}
|
||||
|
||||
for p in all_points:
|
||||
missing = needs_backfill(p)
|
||||
if not missing:
|
||||
skipped_already_have += 1
|
||||
continue
|
||||
pid = str(p["id"])
|
||||
pl = build_payload(p)
|
||||
to_update.append((pid, pl))
|
||||
src = p.get("payload", {}).get("source", "unknown")
|
||||
stats_by_source[src] = stats_by_source.get(src, 0) + 1
|
||||
|
||||
log(f"Points needing backfill: {len(to_update)}/{total}")
|
||||
log(f"Points already complete: {skipped_already_have}")
|
||||
|
||||
if pilot and pilot < len(to_update):
|
||||
to_update = to_update[:pilot]
|
||||
log(f"Pilot mode: processing first {pilot}")
|
||||
|
||||
# ── Stats by source ──
|
||||
log("Backfill candidates by source:")
|
||||
for src, count in sorted(stats_by_source.items(), key=lambda x: -x[1]):
|
||||
ts_sample = ""
|
||||
for pid, pl in to_update:
|
||||
if pid and len(ts_sample) < 3:
|
||||
p = next((pp for pp in all_points if str(pp["id"]) == pid), None)
|
||||
if p and p.get("payload", {}).get("source") == src:
|
||||
ts_sample = pl.get("created_at", "")[:19]
|
||||
break
|
||||
log(f" {src:<25} {count:>6} pts ts_sample={ts_sample}")
|
||||
|
||||
if dry_run:
|
||||
log("DRY-RUN complete. No changes made. Use --commit to apply.")
|
||||
return
|
||||
|
||||
# ── Commit ──
|
||||
if not commit:
|
||||
log("ERROR: --commit flag required for writes. Aborting.")
|
||||
return
|
||||
|
||||
log(f"Committing in batches of {BATCH_SIZE}...")
|
||||
success = 0
|
||||
failed = 0
|
||||
|
||||
for i in range(0, len(to_update), BATCH_SIZE):
|
||||
batch = to_update[i : i + BATCH_SIZE]
|
||||
# Group by identical payload (same timestamp, same confidence)
|
||||
# For simplicity, each point gets its own upsert — but we batch IDs
|
||||
# with identical payloads where possible
|
||||
payload_groups: dict[str, tuple[list[str], dict]] = {}
|
||||
for pid, pl in batch:
|
||||
key = json.dumps(pl, sort_keys=True)
|
||||
if key not in payload_groups:
|
||||
payload_groups[key] = ([], pl)
|
||||
payload_groups[key][0].append(pid)
|
||||
|
||||
for ids, pl in payload_groups.values():
|
||||
ok = upsert_batch(ids, pl)
|
||||
if ok:
|
||||
success += len(ids)
|
||||
else:
|
||||
failed += len(ids)
|
||||
log(f" FAILED batch: {len(ids)} points, first id={ids[0]}")
|
||||
|
||||
batch_num = i // BATCH_SIZE + 1
|
||||
total_batches = (len(to_update) + BATCH_SIZE - 1) // BATCH_SIZE
|
||||
log(f" Batch {batch_num}/{total_batches}: {success} ok, {failed} failed")
|
||||
|
||||
if i + BATCH_SIZE < len(to_update):
|
||||
time.sleep(0.5) # gentle rate limit
|
||||
|
||||
log(f"COMMIT complete. success={success}, failed={failed}")
|
||||
if failed > 0:
|
||||
log("WARNING: some batches failed. Check logs.")
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Backfill decay metadata for Qdrant knowledge_base")
|
||||
parser.add_argument("--dry-run", action="store_true", default=True,
|
||||
help="Simulate, no writes (default)")
|
||||
parser.add_argument("--commit", action="store_true",
|
||||
help="Actually write to Qdrant")
|
||||
parser.add_argument("--pilot", type=int, default=0,
|
||||
help="Process only N points (for validation)")
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.commit:
|
||||
args.dry_run = False
|
||||
|
||||
run(dry_run=args.dry_run, pilot=args.pilot, commit=args.commit)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
@ -0,0 +1,267 @@
|
|||
#!/usr/bin/env python3
|
||||
"""
|
||||
Bulk ingest script — populates the Qdrant knowledge_base with all wiki content.
|
||||
Phase A: one-shot of existing files.
|
||||
"""
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import json
|
||||
import time
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from datetime import datetime, timezone
|
||||
from collections import Counter
|
||||
|
||||
import aiohttp
|
||||
import asyncio
|
||||
|
||||
# ─── Config ────────────────────────────────────────────────────────────────
|
||||
OPENROUTER_KEY = os.environ.get("OPENROUTER_API_KEY")
|
||||
QDRANT_URL = os.environ.get("QDRANT_URL", "http://localhost:6333")
|
||||
COLLECTION = os.environ.get("QDRANT_COLLECTION", os.environ.get("COLLECTION_NAME", "knowledge_base"))
|
||||
WIKI_ROOT = Path(os.environ.get("WIKI_ROOT", "."))
|
||||
EMBEDDING_MODEL = "qwen/qwen3-embedding-8b"
|
||||
EMBEDDING_DIMS = 4096
|
||||
MAX_TEXT_LEN = 8000 # truncate text for embedding (model context limit)
|
||||
BATCH_SIZE = 8 # parallel embedding requests
|
||||
RATE_LIMIT_SLEEP = 0.5 # seconds between batches
|
||||
|
||||
if not OPENROUTER_KEY:
|
||||
print("❌ OPENROUTER_API_KEY not found in environment")
|
||||
sys.exit(1)
|
||||
|
||||
print(f"📁 Wiki root: {WIKI_ROOT}")
|
||||
print(f"🎯 Collection: {COLLECTION}")
|
||||
print(f"🔑 OpenRouter: configured")
|
||||
|
||||
# ─── Find all .md files ───────────────────────────────────────────────────
|
||||
md_files = sorted(WIKI_ROOT.rglob("*.md"))
|
||||
print(f"📄 .md files found: {len(md_files)}")
|
||||
|
||||
# ─── Helpers ──────────────────────────────────────────────────────────────
|
||||
def parse_frontmatter(text: str) -> tuple[dict, str]:
|
||||
"""Extract YAML frontmatter and return (metadata, body)."""
|
||||
if text.startswith("---"):
|
||||
parts = text.split("---", 2)
|
||||
if len(parts) >= 3:
|
||||
try:
|
||||
import yaml
|
||||
meta = yaml.safe_load(parts[1])
|
||||
body = parts[2].strip()
|
||||
return (meta if isinstance(meta, dict) else {}), body
|
||||
except Exception:
|
||||
pass
|
||||
return {}, text
|
||||
|
||||
def get_source_tag(path: Path) -> str:
|
||||
"""Derive source tag from path relative to wiki root."""
|
||||
rel = path.relative_to(WIKI_ROOT)
|
||||
parts = rel.parts
|
||||
if len(parts) > 1:
|
||||
return f"wiki-{parts[0]}"
|
||||
return "wiki-root"
|
||||
|
||||
def get_tags_from_frontmatter(meta: dict) -> list[str]:
|
||||
"""Extract tags from frontmatter."""
|
||||
tags = meta.get("tags", [])
|
||||
if isinstance(tags, str):
|
||||
tags = [t.strip() for t in tags.split(",")]
|
||||
return tags if isinstance(tags, list) else []
|
||||
|
||||
async def get_embedding(session: aiohttp.ClientSession, text: str) -> list[float] | None:
|
||||
"""Generate embedding via OpenRouter."""
|
||||
payload = {
|
||||
"model": EMBEDDING_MODEL,
|
||||
"input": text[:MAX_TEXT_LEN],
|
||||
"dimensions": EMBEDDING_DIMS,
|
||||
}
|
||||
try:
|
||||
async with session.post(
|
||||
"https://openrouter.ai/api/v1/embeddings",
|
||||
headers={
|
||||
"Authorization": f"Bearer {OPENROUTER_KEY}",
|
||||
"Content-Type": "application/json",
|
||||
},
|
||||
json=payload,
|
||||
timeout=aiohttp.ClientTimeout(total=30),
|
||||
) as resp:
|
||||
if resp.status != 200:
|
||||
body = await resp.text()
|
||||
print(f"⚠️ Embedding HTTP {resp.status}: {body[:200]}")
|
||||
return None
|
||||
data = await resp.json()
|
||||
return data["data"][0]["embedding"]
|
||||
except Exception as e:
|
||||
print(f"⚠️ Embedding error: {e}")
|
||||
return None
|
||||
|
||||
async def upsert_to_qdrant(session: aiohttp.ClientSession, points: list[dict]) -> bool:
|
||||
"""Upsert batch of points into Qdrant."""
|
||||
try:
|
||||
async with session.put(
|
||||
f"{QDRANT_URL}/collections/{COLLECTION}/points",
|
||||
headers={"Content-Type": "application/json"},
|
||||
json={"points": points},
|
||||
timeout=aiohttp.ClientTimeout(total=30),
|
||||
) as resp:
|
||||
if resp.status != 200:
|
||||
body = await resp.text()
|
||||
print(f"⚠️ Qdrant HTTP {resp.status}: {body[:200]}")
|
||||
return False
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f"⚠️ Qdrant error: {e}")
|
||||
return False
|
||||
|
||||
# ─── Main processing ──────────────────────────────────────────────────────
|
||||
async def main():
|
||||
stats = Counter({"ok": 0, "fail": 0, "skip": 0, "empty": 0})
|
||||
errors = []
|
||||
processed = 0
|
||||
total = len(md_files)
|
||||
|
||||
connector = aiohttp.TCPConnector(limit=20)
|
||||
async with aiohttp.ClientSession(connector=connector) as session:
|
||||
# Check collection
|
||||
async with session.get(f"{QDRANT_URL}/collections/{COLLECTION}") as r:
|
||||
if r.status != 200:
|
||||
print(f"❌ Collection {COLLECTION} does not exist!")
|
||||
sys.exit(1)
|
||||
|
||||
print("\n🚀 Starting ingestion in batches...\n")
|
||||
|
||||
batch = []
|
||||
for idx, path in enumerate(md_files, 1):
|
||||
text = path.read_text(encoding="utf-8", errors="replace")
|
||||
if not text.strip():
|
||||
stats["empty"] += 1
|
||||
continue
|
||||
|
||||
meta, body = parse_frontmatter(text)
|
||||
source = get_source_tag(path)
|
||||
tags = get_tags_from_frontmatter(meta)
|
||||
# Additional tag from folder
|
||||
folder_tag = source.replace("wiki-", "")
|
||||
if folder_tag not in tags:
|
||||
tags.append(folder_tag)
|
||||
|
||||
# Title from frontmatter or filename
|
||||
title = meta.get("title", path.stem)
|
||||
|
||||
# Text for embedding: title + body (without frontmatter)
|
||||
embed_text = f"{title}\n\n{body}"[:MAX_TEXT_LEN]
|
||||
|
||||
batch.append({
|
||||
"idx": idx,
|
||||
"path": str(path),
|
||||
"title": title,
|
||||
"source": source,
|
||||
"tags": tags,
|
||||
"embed_text": embed_text,
|
||||
"meta": meta,
|
||||
})
|
||||
|
||||
if len(batch) >= BATCH_SIZE or idx == total:
|
||||
# Generate embeddings in parallel
|
||||
embed_tasks = [get_embedding(session, b["embed_text"]) for b in batch]
|
||||
vectors = await asyncio.gather(*embed_tasks)
|
||||
|
||||
# Prepare Qdrant points
|
||||
points = []
|
||||
for b, vec in zip(batch, vectors):
|
||||
if vec is None:
|
||||
stats["fail"] += 1
|
||||
errors.append(f"Embedding failed: {b['path']}")
|
||||
continue
|
||||
|
||||
# Heuristic importance_score based on path/name
|
||||
importance_score = 0.5
|
||||
path_str_lower = b["path"].lower()
|
||||
if any(k in path_str_lower for k in ["architecture", "core", "important"]):
|
||||
importance_score = 0.7
|
||||
if any(t.lower() in ["important", "critical"] for t in b["tags"]):
|
||||
importance_score = 0.8
|
||||
if any(k in path_str_lower for k in ["draft", "temp", "old"]):
|
||||
importance_score = 0.2
|
||||
|
||||
now_iso = datetime.now(timezone.utc).isoformat()
|
||||
|
||||
point = {
|
||||
"id": str(uuid.uuid4()),
|
||||
"vector": {"dense": vec},
|
||||
"payload": {
|
||||
"text": b["embed_text"],
|
||||
"source": b["source"],
|
||||
"tags": b["tags"],
|
||||
"created_at": now_iso,
|
||||
"reflection_count": 0,
|
||||
"last_reflected": None,
|
||||
"file_path": b["path"],
|
||||
"title": b["title"],
|
||||
"word_count": len(b["embed_text"].split()),
|
||||
# ── Lineage fields (Phase 1)
|
||||
"lineage_id": None,
|
||||
"generation_model": None,
|
||||
"generation_context_hash": None,
|
||||
"retrieved_chunk_ids": None,
|
||||
# ── Decay fields (Phase 2)
|
||||
"decay_score": 1.0,
|
||||
"last_accessed_at": now_iso,
|
||||
"importance_score": importance_score,
|
||||
"source_type": "human",
|
||||
"confidence_score": 1.0,
|
||||
"archived": False,
|
||||
},
|
||||
}
|
||||
points.append(point)
|
||||
|
||||
# Upsert
|
||||
if points:
|
||||
ok = await upsert_to_qdrant(session, points)
|
||||
if ok:
|
||||
stats["ok"] += len(points)
|
||||
else:
|
||||
stats["fail"] += len(points)
|
||||
for p in points:
|
||||
errors.append(f"Qdrant upsert failed: {p['payload']['file_path']}")
|
||||
|
||||
processed += len(batch)
|
||||
batch = []
|
||||
|
||||
# Progress
|
||||
pct = (processed / total) * 100
|
||||
print(f" [{processed}/{total}] {pct:.1f}% | ✅ {stats['ok']} | ⚠️ {stats['fail']} | ⏭️ {stats['skip']} | 🈳 {stats['empty']}")
|
||||
|
||||
# Rate limit breathing
|
||||
await asyncio.sleep(RATE_LIMIT_SLEEP)
|
||||
|
||||
# ─── Final report ───────────────────────────────────────────────────
|
||||
print("\n" + "=" * 60)
|
||||
print("📊 INGESTION REPORT")
|
||||
print("=" * 60)
|
||||
print(f" Total files: {total}")
|
||||
print(f" Ingested (ok): {stats['ok']}")
|
||||
print(f" Failures: {stats['fail']}")
|
||||
print(f" Empty: {stats['empty']}")
|
||||
print(f" Success rate: {(stats['ok']/max(total-stats['empty'],1)*100):.1f}%")
|
||||
print(f"\n ⏱️ Finished: {datetime.now(timezone.utc).isoformat()}")
|
||||
|
||||
if errors:
|
||||
print(f"\n ⚠️ First errors ({min(10, len(errors))} of {len(errors)}):")
|
||||
for e in errors[:10]:
|
||||
print(f" - {e}")
|
||||
|
||||
# Verify final count
|
||||
async with aiohttp.ClientSession() as s:
|
||||
async with s.get(f"{QDRANT_URL}/collections/{COLLECTION}") as r:
|
||||
data = await r.json()
|
||||
final_count = data.get("result", {}).get("points_count", "?")
|
||||
print(f"\n 📦 Points in collection: {final_count}")
|
||||
|
||||
print("\n✅ Bulk ingest complete.")
|
||||
return stats
|
||||
|
||||
if __name__ == "__main__":
|
||||
stats = asyncio.run(main())
|
||||
sys.exit(0 if stats["fail"] == 0 else 1)
|
||||
|
|
@ -0,0 +1,715 @@
|
|||
#!/usr/bin/env python3
|
||||
"""
|
||||
Context Enhancer — HYBRID search (semantic + BM25) on knowledge_base_hybrid
|
||||
for prompt enrichment.
|
||||
|
||||
Runs as a synchronous function (fast, <1s) before each Hermes response.
|
||||
If Qdrant is offline or embedding fails, returns "" (fail-open).
|
||||
|
||||
Usage:
|
||||
python3 context_enhancer.py "your query here"
|
||||
python3 context_enhancer.py --top-k 5 --threshold 0.50 "deploy docker"
|
||||
python3 context_enhancer.py --hybrid-off "your query here" # forces dense-only
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import json
|
||||
import uuid
|
||||
import hashlib
|
||||
import sqlite3
|
||||
import subprocess
|
||||
import requests
|
||||
import argparse
|
||||
import re
|
||||
import glob
|
||||
from typing import List, Dict, Optional, Tuple
|
||||
from pathlib import Path
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
|
||||
# ─── Config ────────────────────────────────────────────────────────────────
|
||||
OPENROUTER_KEY = os.environ.get("OPENROUTER_API_KEY")
|
||||
if not OPENROUTER_KEY:
|
||||
env_path = Path.home() / ".env"
|
||||
if env_path.exists():
|
||||
for line in env_path.read_text().splitlines():
|
||||
if line.startswith("OPENROUTER_API_KEY="):
|
||||
OPENROUTER_KEY = line.split("=", 1)[1].strip().strip('"')
|
||||
break
|
||||
|
||||
QDRANT_URL = os.environ.get("QDRANT_URL", "http://localhost:6333")
|
||||
COLLECTION = os.environ.get("QDRANT_COLLECTION", "knowledge_base")
|
||||
|
||||
EMBEDDING_MODEL = "qwen/qwen3-embedding-8b"
|
||||
TOP_K_DEFAULT = 3
|
||||
SCORE_THRESHOLD_DEFAULT = 0.55
|
||||
MAX_TEXT_LEN = 8000
|
||||
REQUEST_TIMEOUT = 10
|
||||
|
||||
# FastEmbed BM25 config
|
||||
FASTEMBED_VENV = os.environ.get("FASTEMBED_VENV", "")
|
||||
|
||||
# Default: use current Python if venv not configured
|
||||
_FASTEMBED_PYTHON = FASTEMBED_VENV if FASTEMBED_VENV else sys.executable
|
||||
_FASTEMBED_SITEPKGS = os.environ.get(
|
||||
"FASTEMBED_SITEPKGS",
|
||||
os.path.join(os.path.dirname(sys.executable), "../lib/python3.12/site-packages")
|
||||
)
|
||||
BM25_MODEL = "Qdrant/bm25"
|
||||
|
||||
# Lineage config
|
||||
LINEAGE_DB = os.environ.get(
|
||||
"STATE_DB_PATH",
|
||||
os.path.expanduser("~/.hermes/state.db")
|
||||
)
|
||||
|
||||
# Telemetry config
|
||||
TELEMETRY_LOG = os.environ.get(
|
||||
"TELEMETRY_LOG_PATH",
|
||||
os.path.expanduser("~/.hermes/logs/query-telemetry.jsonl")
|
||||
)
|
||||
TELEMETRY_MAX_BYTES = 10 * 1024 * 1024 # 10MB rotation
|
||||
|
||||
# ─── Lineage Registration ───────────────────────────────────────────────────
|
||||
|
||||
def register_lineage(
|
||||
session_id: str,
|
||||
query: str,
|
||||
retrieved_chunk_ids: List[str],
|
||||
generation_context_hash: str,
|
||||
generation_model: str = "unknown",
|
||||
) -> Optional[str]:
|
||||
"""
|
||||
Register generation provenance in the lineage DB.
|
||||
Fail-open: if it fails, log error and return None. Never breaks the critical path.
|
||||
"""
|
||||
lineage_id = str(uuid.uuid4())
|
||||
try:
|
||||
with sqlite3.connect(LINEAGE_DB) as conn:
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO lineage (lineage_id, session_id, query, retrieved_chunk_ids,
|
||||
generation_model, generation_context_hash, created_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, datetime('now'))
|
||||
""",
|
||||
(lineage_id, session_id, query,
|
||||
json.dumps(retrieved_chunk_ids, ensure_ascii=False),
|
||||
generation_model, generation_context_hash)
|
||||
)
|
||||
conn.commit()
|
||||
return lineage_id
|
||||
except Exception as e:
|
||||
print(f"[LINEAGE-WARNING] Failed to register lineage: {e}", file=sys.stderr)
|
||||
return None
|
||||
|
||||
|
||||
# ─── Telemetry ─────────────────────────────────────────────────────────────
|
||||
|
||||
def _rotate_telemetry_if_needed() -> None:
|
||||
"""Rotate telemetry file if >10MB. Rename to .1 and restart."""
|
||||
try:
|
||||
if os.path.exists(TELEMETRY_LOG) and os.path.getsize(TELEMETRY_LOG) > TELEMETRY_MAX_BYTES:
|
||||
rotated = TELEMETRY_LOG + ".1"
|
||||
if os.path.exists(rotated):
|
||||
os.remove(rotated)
|
||||
os.rename(TELEMETRY_LOG, rotated)
|
||||
except Exception as e:
|
||||
print(f"[TELEMETRY-WARNING] Rotation failed: {e}", file=sys.stderr)
|
||||
|
||||
|
||||
def emit_telemetry(record: dict) -> None:
|
||||
"""
|
||||
Append-only JSONL for query telemetry.
|
||||
Fail-open: never breaks the critical path.
|
||||
"""
|
||||
try:
|
||||
os.makedirs(os.path.dirname(TELEMETRY_LOG), exist_ok=True)
|
||||
_rotate_telemetry_if_needed()
|
||||
with open(TELEMETRY_LOG, "a", encoding="utf-8") as f:
|
||||
f.write(json.dumps(record, ensure_ascii=False) + "\n")
|
||||
except Exception as e:
|
||||
print(f"[TELEMETRY-WARNING] Failed to write telemetry: {e}", file=sys.stderr)
|
||||
|
||||
|
||||
def estimate_tokens(text: str) -> int:
|
||||
"""Simple estimate: ~1.3 tokens per word (basic heuristic)."""
|
||||
return int(len(text.split()) * 1.3)
|
||||
|
||||
|
||||
# ─── Core ───────────────────────────────────────────────────────────────────
|
||||
|
||||
def embed_query(text: str) -> Optional[List[float]]:
|
||||
"""Generate dense embedding via OpenRouter qwen/qwen3-embedding-8b."""
|
||||
if not OPENROUTER_KEY:
|
||||
return None
|
||||
try:
|
||||
resp = requests.post(
|
||||
"https://openrouter.ai/api/v1/embeddings",
|
||||
headers={
|
||||
"Authorization": f"Bearer {OPENROUTER_KEY}",
|
||||
"Content-Type": "application/json"
|
||||
},
|
||||
json={
|
||||
"model": EMBEDDING_MODEL,
|
||||
"input": text[:MAX_TEXT_LEN]
|
||||
},
|
||||
timeout=REQUEST_TIMEOUT
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json()["data"][0]["embedding"]
|
||||
except Exception as e:
|
||||
# Fail-open: log silently, return None
|
||||
print(f"[CE-ERROR] Embedding dense failed: {e}", file=sys.stderr)
|
||||
return None
|
||||
|
||||
|
||||
def embed_query_sparse(text: str) -> Optional[Tuple[List[int], List[float]]]:
|
||||
"""
|
||||
Generate sparse BM25 embedding via FastEmbed (subprocess in ai-lab venv).
|
||||
Fail-open: if it fails, return None. Caller falls back to dense-only.
|
||||
"""
|
||||
try:
|
||||
# Shell-quote the text to prevent injection in Python -c
|
||||
import shlex
|
||||
safe_text = shlex.quote(text)
|
||||
result = subprocess.run(
|
||||
[_FASTEMBED_PYTHON, "-c", f"""\
|
||||
import os, sys
|
||||
sys.path.insert(0, os.environ["FASTEMBED_SITEPKGS"])
|
||||
from fastembed.sparse import SparseTextEmbedding
|
||||
import json
|
||||
model = SparseTextEmbedding(model_name=\\"{BM25_MODEL}\\")
|
||||
sparse = list(model.embed({safe_text}))[0]
|
||||
print(json.dumps({{\\"indices\\": sparse.indices.tolist(), \\"values\\": sparse.values.tolist()}}))
|
||||
"""],
|
||||
capture_output=True, text=True, timeout=15
|
||||
)
|
||||
data = json.loads(result.stdout.strip())
|
||||
return data["indices"], data["values"]
|
||||
except Exception as e:
|
||||
print(f"[CE-ERROR] Embedding sparse failed: {e}", file=sys.stderr)
|
||||
return None
|
||||
|
||||
|
||||
def search_knowledge_base(
|
||||
dense_vector: List[float],
|
||||
sparse_vector: Optional[Tuple[List[int], List[float]]] = None,
|
||||
top_k: int = TOP_K_DEFAULT,
|
||||
score_threshold: float = SCORE_THRESHOLD_DEFAULT
|
||||
) -> List[Dict]:
|
||||
"""
|
||||
Hybrid search in Qdrant: dense (semantic) + sparse (BM25 keyword)
|
||||
via prefetch + RRF. If sparse fails, fall back to dense-only.
|
||||
"""
|
||||
try:
|
||||
if sparse_vector is not None:
|
||||
# Hybrid: prefetch dense + prefetch sparse → RRF
|
||||
resp = requests.post(
|
||||
f"{QDRANT_URL}/collections/{COLLECTION}/points/query",
|
||||
headers={"Content-Type": "application/json"},
|
||||
json={
|
||||
"prefetch": [
|
||||
{"query": dense_vector, "using": "dense", "limit": top_k * 3},
|
||||
{"query": {"indices": sparse_vector[0], "values": sparse_vector[1]},
|
||||
"using": "sparse", "limit": top_k * 3},
|
||||
],
|
||||
"query": {"fusion": "rrf"},
|
||||
"limit": top_k * 2,
|
||||
"with_payload": True,
|
||||
},
|
||||
timeout=REQUEST_TIMEOUT + 5,
|
||||
)
|
||||
else:
|
||||
# Fallback: dense-only (collections with compatible named vectors)
|
||||
resp = requests.post(
|
||||
f"{QDRANT_URL}/collections/{COLLECTION}/points/query",
|
||||
headers={"Content-Type": "application/json"},
|
||||
json={
|
||||
"query": dense_vector,
|
||||
"using": "dense",
|
||||
"limit": top_k,
|
||||
"with_payload": True,
|
||||
},
|
||||
timeout=REQUEST_TIMEOUT,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
results = []
|
||||
raw_results = data.get("result", {})
|
||||
# API /points/search returns direct list; API /points/query returns {"points": [...]}
|
||||
points = raw_results if isinstance(raw_results, list) else raw_results.get("points", [])
|
||||
for r in points:
|
||||
score = r.get("score", 0)
|
||||
if score < score_threshold:
|
||||
continue
|
||||
payload = r.get("payload", {})
|
||||
results.append({
|
||||
"id": r.get("id", "unknown"),
|
||||
"score": score,
|
||||
"title": payload.get("title", "Untitled"),
|
||||
"content_preview": (payload.get("text", "") or "")[:400],
|
||||
"source": payload.get("source", "unknown"),
|
||||
"tags": payload.get("tags", [])
|
||||
})
|
||||
return results
|
||||
except Exception as e:
|
||||
print(f"[CE-ERROR] Qdrant search failed: {e}", file=sys.stderr)
|
||||
return []
|
||||
|
||||
|
||||
# ─── Fallback: Lexical Search in Vault ──────────────────────────────────────
|
||||
|
||||
def tokenize_query(text: str) -> List[str]:
|
||||
"""Tokenize query into relevant terms (lowercase, alphanumeric)."""
|
||||
# Remove basic Portuguese/English stopwords
|
||||
stopwords = {"o", "a", "os", "as", "um", "uma", "de", "da", "do", "em", "no", "na", "para", "com", "por", "que", "se", "e", "ou", "mas", "the", "a", "an", "is", "are", "was", "were", "be", "been", "have", "has", "had", "do", "does", "did", "will", "would", "could", "should"}
|
||||
tokens = re.findall(r'\b[a-zA-Z0-9]+\b', text.lower())
|
||||
return [t for t in tokens if t not in stopwords and len(t) > 2]
|
||||
|
||||
|
||||
def lexical_search_in_vault(
|
||||
query_terms: List[str],
|
||||
top_k: int = TOP_K_DEFAULT,
|
||||
vault_root: str = os.environ.get("WIKI_PATH", os.path.expanduser("~/vault/wiki"))
|
||||
) -> List[Dict]:
|
||||
"""
|
||||
Lexical search in .md files under vault/wiki/.
|
||||
Rank by term match density / document size.
|
||||
Return top-k simulated chunks (or entire file if small).
|
||||
"""
|
||||
if not query_terms:
|
||||
return []
|
||||
|
||||
md_files = glob.glob(f"{vault_root}/**/*.md", recursive=True)
|
||||
if not md_files:
|
||||
return []
|
||||
|
||||
scored = []
|
||||
for filepath in md_files:
|
||||
try:
|
||||
text = Path(filepath).read_text(encoding="utf-8", errors="replace").lower()
|
||||
if not text:
|
||||
continue
|
||||
|
||||
matches = 0
|
||||
for term in query_terms:
|
||||
matches += text.count(term)
|
||||
|
||||
if matches == 0:
|
||||
continue
|
||||
|
||||
# Heuristic: matches / sqrt(word_count) — favors concise documents
|
||||
word_count = max(1, len(text.split()))
|
||||
density = matches / (word_count ** 0.5) # sqrt(word_count) to not over-penalize medium texts
|
||||
scored.append({
|
||||
"filepath": filepath,
|
||||
"matches": matches,
|
||||
"word_count": word_count,
|
||||
"density": density,
|
||||
"title": Path(filepath).stem,
|
||||
"text": text[:2000], # truncate for return
|
||||
})
|
||||
except Exception:
|
||||
continue
|
||||
|
||||
if not scored:
|
||||
return []
|
||||
|
||||
# Sort by descending density
|
||||
scored.sort(key=lambda x: x["density"], reverse=True)
|
||||
|
||||
results = []
|
||||
for i, item in enumerate(scored[:top_k]):
|
||||
# Simulate a result chunk
|
||||
results.append({
|
||||
"id": f"lexical-{hashlib.md5(item['filepath'].encode()).hexdigest()[:16]}",
|
||||
"score": round(min(1.0, item["density"]), 2),
|
||||
"title": item["title"],
|
||||
"content_preview": item["text"][:400],
|
||||
"source": f"vault-{item['filepath'].replace(vault_root, '').lstrip('/')[:40]}",
|
||||
"tags": ["fallback", "lexical"],
|
||||
"fallback_level": "lexical",
|
||||
})
|
||||
|
||||
if results:
|
||||
print(f"[CE-FALLBACK] Lexical search returned {len(results)} results from vault", file=sys.stderr)
|
||||
return results
|
||||
|
||||
|
||||
# ─── Fallback: SQLite Keyword Search ────────────────────────────────────────
|
||||
|
||||
def sqlite_keyword_search(
|
||||
query_terms: List[str],
|
||||
top_k: int = TOP_K_DEFAULT
|
||||
) -> List[Dict]:
|
||||
"""
|
||||
Search terms in the lineage table (query field) and other state tables.
|
||||
Last resort — does not replace vault.
|
||||
"""
|
||||
if not query_terms:
|
||||
return []
|
||||
|
||||
try:
|
||||
with sqlite3.connect(LINEAGE_DB) as conn:
|
||||
conn.row_factory = sqlite3.Row
|
||||
c = conn.cursor()
|
||||
|
||||
results = []
|
||||
# Search in lineage.query
|
||||
placeholders = " OR ".join(["query LIKE ?"] * len(query_terms))
|
||||
params = [f"%{term}%" for term in query_terms]
|
||||
c.execute(
|
||||
f"""
|
||||
SELECT lineage_id, session_id, query, generation_context_hash, created_at
|
||||
FROM lineage
|
||||
WHERE {placeholders}
|
||||
ORDER BY created_at DESC
|
||||
LIMIT ?
|
||||
""",
|
||||
params + [top_k * 2]
|
||||
)
|
||||
for row in c.fetchall():
|
||||
results.append({
|
||||
"id": f"sqlite-{row['lineage_id'][:16]}",
|
||||
"score": 0.5,
|
||||
"title": f"Lineage {row['lineage_id'][:8]}...",
|
||||
"content_preview": (row["query"] or "")[:400],
|
||||
"source": f"sqlite-history-{row['session_id']}",
|
||||
"tags": ["fallback", "sqlite"],
|
||||
"fallback_level": "sqlite",
|
||||
})
|
||||
|
||||
if results:
|
||||
print(f"[CE-FALLBACK] SQLite keyword search returned {len(results)} results", file=sys.stderr)
|
||||
return results[:top_k]
|
||||
except Exception as e:
|
||||
print(f"[CE-FALLBACK] SQLite search failed: {e}", file=sys.stderr)
|
||||
return []
|
||||
|
||||
|
||||
# ─── Fallback Wrapper ──────────────────────────────────────────────────────
|
||||
|
||||
def search_with_fallback(
|
||||
dense_vector: Optional[List[float]] = None,
|
||||
sparse_vector: Optional[Tuple[List[int], List[float]]] = None,
|
||||
query_text: str = "",
|
||||
top_k: int = TOP_K_DEFAULT,
|
||||
score_threshold: float = SCORE_THRESHOLD_DEFAULT
|
||||
) -> Tuple[List[Dict], str, float, float]:
|
||||
"""
|
||||
4-level fallback cascade:
|
||||
1. Hybrid (dense + sparse + RRF) — normal mode
|
||||
2. Dense-only — if sparse fails or is None
|
||||
3. Lexical-only — if Qdrant goes down (ConnectionError, Timeout)
|
||||
4. SQLite keyword — if vault is inaccessible
|
||||
|
||||
Returns (results, fallback_level, qdrant_latency_ms, fallback_latency_ms).
|
||||
"""
|
||||
fallback_level = "hybrid"
|
||||
qdrant_latency_ms = 0.0
|
||||
t0 = time.perf_counter()
|
||||
|
||||
# Level 1 or 2: Qdrant (hybrid if sparse available, otherwise dense-only)
|
||||
try:
|
||||
t_q0 = time.perf_counter()
|
||||
if sparse_vector is not None:
|
||||
resp = requests.post(
|
||||
f"{QDRANT_URL}/collections/{COLLECTION}/points/query",
|
||||
headers={"Content-Type": "application/json"},
|
||||
json={
|
||||
"prefetch": [
|
||||
{"query": dense_vector, "using": "dense", "limit": top_k * 3},
|
||||
{"query": {"indices": sparse_vector[0], "values": sparse_vector[1]},
|
||||
"using": "sparse", "limit": top_k * 3},
|
||||
],
|
||||
"query": {"fusion": "rrf"},
|
||||
"limit": top_k * 2,
|
||||
"with_payload": True,
|
||||
},
|
||||
timeout=REQUEST_TIMEOUT + 5,
|
||||
)
|
||||
fallback_level = "hybrid"
|
||||
else:
|
||||
resp = requests.post(
|
||||
f"{QDRANT_URL}/collections/{COLLECTION}/points/query",
|
||||
headers={"Content-Type": "application/json"},
|
||||
json={
|
||||
"query": dense_vector,
|
||||
"using": "dense",
|
||||
"limit": top_k,
|
||||
"with_payload": True,
|
||||
},
|
||||
timeout=REQUEST_TIMEOUT,
|
||||
)
|
||||
fallback_level = "dense-only"
|
||||
|
||||
resp.raise_for_status()
|
||||
qdrant_latency_ms = (time.perf_counter() - t_q0) * 1000
|
||||
data = resp.json()
|
||||
results = []
|
||||
raw_results = data.get("result", {})
|
||||
points = raw_results if isinstance(raw_results, list) else raw_results.get("points", [])
|
||||
for r in points:
|
||||
score = r.get("score", 0)
|
||||
if score < score_threshold:
|
||||
continue
|
||||
payload = r.get("payload", {})
|
||||
results.append({
|
||||
"id": r.get("id", "unknown"),
|
||||
"score": score,
|
||||
"title": payload.get("title", "Untitled"),
|
||||
"content_preview": (payload.get("text", "") or "")[:400],
|
||||
"source": payload.get("source", "unknown"),
|
||||
"tags": payload.get("tags", [])
|
||||
})
|
||||
|
||||
if results:
|
||||
fallback_latency_ms = (time.perf_counter() - t0) * 1000
|
||||
return results, fallback_level, qdrant_latency_ms, fallback_latency_ms
|
||||
except requests.exceptions.ConnectionError:
|
||||
print("[CE-FALLBACK] Qdrant unavailable (ConnectionError), falling back to lexical search.", file=sys.stderr)
|
||||
except requests.exceptions.Timeout:
|
||||
print("[CE-FALLBACK] Qdrant timeout, falling back to lexical search.", file=sys.stderr)
|
||||
except Exception as e:
|
||||
# Unexpected Qdrant error (e.g. SparseIndexError, 5xx, etc.)
|
||||
# If sparse_vector existed, it might be a sparse error — try dense-only
|
||||
if sparse_vector is not None and "sparse" in str(e).lower():
|
||||
print(f"[CE-FALLBACK] Sparse index failed ('{e}'), trying dense-only...", file=sys.stderr)
|
||||
try:
|
||||
t_q2 = time.perf_counter()
|
||||
resp = requests.post(
|
||||
f"{QDRANT_URL}/collections/{COLLECTION}/points/search",
|
||||
headers={"Content-Type": "application/json"},
|
||||
json={
|
||||
"vector": dense_vector,
|
||||
"using": "dense",
|
||||
"limit": top_k,
|
||||
"with_payload": True,
|
||||
},
|
||||
timeout=REQUEST_TIMEOUT,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
qdrant_latency_ms = (time.perf_counter() - t_q2) * 1000
|
||||
data = resp.json()
|
||||
results = []
|
||||
raw_results = data.get("result", {})
|
||||
points = raw_results if isinstance(raw_results, list) else raw_results.get("points", [])
|
||||
for r in points:
|
||||
score = r.get("score", 0)
|
||||
if score < score_threshold:
|
||||
continue
|
||||
payload = r.get("payload", {})
|
||||
results.append({
|
||||
"id": r.get("id", "unknown"),
|
||||
"score": score,
|
||||
"title": payload.get("title", "Untitled"),
|
||||
"content_preview": (payload.get("text", "") or "")[:400],
|
||||
"source": payload.get("source", "unknown"),
|
||||
"tags": payload.get("tags", [])
|
||||
})
|
||||
if results:
|
||||
fallback_latency_ms = (time.perf_counter() - t0) * 1000
|
||||
print("[CE-FALLBACK] Dense-only worked after sparse error.", file=sys.stderr)
|
||||
return results, "dense-only", qdrant_latency_ms, fallback_latency_ms
|
||||
except (requests.exceptions.ConnectionError, requests.exceptions.Timeout):
|
||||
print("[CE-FALLBACK] Qdrant also unavailable for dense-only — lexical fallback.", file=sys.stderr)
|
||||
except Exception as e2:
|
||||
print(f"[CE-FALLBACK] Dense-only also failed: {e2}", file=sys.stderr)
|
||||
else:
|
||||
print(f"[CE-FALLBACK] Qdrant general error ({e}), falling back to lexical.", file=sys.stderr)
|
||||
|
||||
# Level 3: Lexical search in vault
|
||||
terms = tokenize_query(query_text)
|
||||
lexical_results = lexical_search_in_vault(terms, top_k=top_k)
|
||||
if lexical_results:
|
||||
fallback_latency_ms = (time.perf_counter() - t0) * 1000
|
||||
return lexical_results, "lexical", qdrant_latency_ms, fallback_latency_ms
|
||||
|
||||
# Level 4: SQLite keyword search
|
||||
sqlite_results = sqlite_keyword_search(terms, top_k=top_k)
|
||||
if sqlite_results:
|
||||
fallback_latency_ms = (time.perf_counter() - t0) * 1000
|
||||
return sqlite_results, "sqlite", qdrant_latency_ms, fallback_latency_ms
|
||||
|
||||
# Nothing worked
|
||||
fallback_latency_ms = (time.perf_counter() - t0) * 1000
|
||||
print("[CE-FALLBACK] All fallback levels exhausted.", file=sys.stderr)
|
||||
return [], "none", qdrant_latency_ms, fallback_latency_ms
|
||||
|
||||
|
||||
def update_last_accessed_at(chunk_ids: list) -> None:
|
||||
"""
|
||||
Update last_accessed_at on chunks returned by search.
|
||||
Reset decay for chunks that are actually being used.
|
||||
Fail-open: never breaks the critical query path.
|
||||
"""
|
||||
if not chunk_ids:
|
||||
return
|
||||
try:
|
||||
now = datetime.now().astimezone().isoformat()
|
||||
requests.post(
|
||||
f"{QDRANT_URL}/collections/{COLLECTION}/points/payload",
|
||||
headers={"Content-Type": "application/json"},
|
||||
json={
|
||||
"points": chunk_ids,
|
||||
"payload": {"last_accessed_at": now},
|
||||
},
|
||||
timeout=5,
|
||||
)
|
||||
except Exception as e:
|
||||
print(f"[CE-WARNING] Failed to update last_accessed_at: {e}", file=sys.stderr)
|
||||
|
||||
|
||||
def enhance_prompt(
|
||||
user_message: str,
|
||||
top_k: int = TOP_K_DEFAULT,
|
||||
score_threshold: float = SCORE_THRESHOLD_DEFAULT,
|
||||
format_mode: str = "markdown",
|
||||
hybrid: bool = True,
|
||||
session_id: Optional[str] = None,
|
||||
generation_model: str = "unknown",
|
||||
) -> str:
|
||||
"""
|
||||
Return a context block with relevant vault content (hybrid dense+BM25),
|
||||
or an empty string if nothing relevant found / services offline.
|
||||
Register lineage with chunk provenance and generated context hash.
|
||||
"""
|
||||
# Resolve session_id
|
||||
if session_id is None:
|
||||
session_id = os.environ.get("HERMES_SESSION_ID", "standalone")
|
||||
|
||||
# Skip very short or irrelevant queries
|
||||
if len(user_message.strip()) < 5:
|
||||
return ""
|
||||
|
||||
# Very short or social query — skip
|
||||
social_keywords = {"hi", "hello", "hey", "yo", "sup", "ok", "thanks", "bye", "okay", "great"}
|
||||
if user_message.strip().lower() in social_keywords:
|
||||
return ""
|
||||
|
||||
dense_vector = embed_query(user_message)
|
||||
sparse_vector = None
|
||||
if hybrid:
|
||||
sparse_vector = embed_query_sparse(user_message)
|
||||
|
||||
# Fallback cascade: hybrid → dense-only → lexical → sqlite
|
||||
hits, fallback_level, qdrant_latency_ms, fallback_latency_ms = search_with_fallback(
|
||||
dense_vector=dense_vector,
|
||||
sparse_vector=sparse_vector,
|
||||
query_text=user_message,
|
||||
top_k=top_k,
|
||||
score_threshold=score_threshold
|
||||
)
|
||||
|
||||
# If empty and we had no embedding, try pure lexical
|
||||
if not hits and dense_vector is None:
|
||||
terms = tokenize_query(user_message)
|
||||
lexical_results = lexical_search_in_vault(terms, top_k=top_k)
|
||||
if lexical_results:
|
||||
hits = lexical_results
|
||||
fallback_level = "lexical"
|
||||
|
||||
if not hits:
|
||||
# Emit telemetry even when empty (records that search ran and returned nothing)
|
||||
emit_telemetry({
|
||||
"timestamp": datetime.now(timezone.utc).isoformat(),
|
||||
"session_id": session_id,
|
||||
"query": user_message,
|
||||
"retrieval_mode": fallback_level,
|
||||
"retrieved_chunk_ids": [],
|
||||
"semantic_scores": [],
|
||||
"rerank_scores": [],
|
||||
"final_context_tokens": 0,
|
||||
"fallback_level": {"hybrid": 0, "dense-only": 1, "lexical": 2, "sqlite": 3, "none": -1}.get(fallback_level, -1),
|
||||
"llm_response_quality": None,
|
||||
"qdrant_latency_ms": round(qdrant_latency_ms, 2),
|
||||
"fallback_latency_ms": round(fallback_latency_ms, 2),
|
||||
})
|
||||
return ""
|
||||
|
||||
# Update last_accessed_at for returned chunks (reset decay on real usage)
|
||||
retrieved_chunk_ids = [h["id"] for h in hits]
|
||||
update_last_accessed_at(retrieved_chunk_ids)
|
||||
|
||||
if format_mode == "markdown":
|
||||
lines = ["\n## Relevant Context from Vault\n"]
|
||||
for i, h in enumerate(hits, 1):
|
||||
lines.append(f"### [{i}] {h['title']} (score: {h['score']:.2f})")
|
||||
lines.append(f"- **Source:** `{h['source']}` | Tags: {', '.join(str(t) for t in h['tags'])}")
|
||||
lines.append(f"- **Excerpt:** {h['content_preview']}...")
|
||||
lines.append("")
|
||||
context_str = "\n".join(lines)
|
||||
elif format_mode == "compact":
|
||||
parts = []
|
||||
for h in hits:
|
||||
parts.append(f"[{h['title'][:40]}] (s:{h['score']:.2f})")
|
||||
context_str = "\n".join(parts)
|
||||
else:
|
||||
context_str = json.dumps(hits, indent=2, ensure_ascii=False)
|
||||
|
||||
generation_context_hash = hashlib.sha256(context_str.encode("utf-8")).hexdigest()[:32]
|
||||
|
||||
register_lineage(
|
||||
session_id=session_id,
|
||||
query=user_message,
|
||||
retrieved_chunk_ids=retrieved_chunk_ids,
|
||||
generation_context_hash=generation_context_hash,
|
||||
generation_model=generation_model,
|
||||
)
|
||||
|
||||
# Extract scores for telemetry
|
||||
semantic_scores = [h.get("score", 0) for h in hits]
|
||||
# For rerank_scores, use the same scores (internal Qdrant RRF, not exposed via simple API)
|
||||
rerank_scores = semantic_scores[:]
|
||||
|
||||
final_context_tokens = estimate_tokens(context_str)
|
||||
|
||||
emit_telemetry({
|
||||
"timestamp": datetime.now(timezone.utc).isoformat(),
|
||||
"session_id": session_id,
|
||||
"query": user_message,
|
||||
"retrieval_mode": fallback_level,
|
||||
"retrieved_chunk_ids": retrieved_chunk_ids,
|
||||
"semantic_scores": semantic_scores,
|
||||
"rerank_scores": rerank_scores,
|
||||
"final_context_tokens": final_context_tokens,
|
||||
"fallback_level": {"hybrid": 0, "dense-only": 1, "lexical": 2, "sqlite": 3, "none": -1}.get(fallback_level, -1),
|
||||
"llm_response_quality": None,
|
||||
"qdrant_latency_ms": round(qdrant_latency_ms, 2),
|
||||
"fallback_latency_ms": round(fallback_latency_ms, 2),
|
||||
})
|
||||
|
||||
return context_str
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Context Enhancer — Hybrid search in knowledge_base")
|
||||
parser.add_argument("query", nargs="?", help="User message/query")
|
||||
parser.add_argument("--top-k", type=int, default=TOP_K_DEFAULT, help=f"Number of results (default: {TOP_K_DEFAULT})")
|
||||
parser.add_argument("--threshold", type=float, default=SCORE_THRESHOLD_DEFAULT, help=f"Minimum score (default: {SCORE_THRESHOLD_DEFAULT})")
|
||||
parser.add_argument("--format", choices=["markdown", "compact", "json"], default="markdown", help="Output format")
|
||||
parser.add_argument("--silent", action="store_true", help="Silent — return empty on error")
|
||||
parser.add_argument("--hybrid-off", action="store_true", help="Force dense-only (ignore BM25)")
|
||||
args = parser.parse_args()
|
||||
|
||||
if not args.query:
|
||||
args.query = sys.stdin.read().strip() or "how ARQ worker works"
|
||||
|
||||
try:
|
||||
result = enhance_prompt(
|
||||
args.query,
|
||||
top_k=args.top_k,
|
||||
score_threshold=args.threshold,
|
||||
format_mode=args.format,
|
||||
hybrid=not args.hybrid_off,
|
||||
generation_model=os.environ.get("HERMES_MODEL", "unknown"),
|
||||
)
|
||||
print(result)
|
||||
except Exception as e:
|
||||
if not args.silent:
|
||||
print(f"[CE-ERROR] {e}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
@ -0,0 +1,284 @@
|
|||
#!/usr/bin/env python3
|
||||
"""
|
||||
decay_scanner.py
|
||||
Selective archiving script for low-importance AI-generated chunks.
|
||||
Runs via weekly cron (0 3 * * 0).
|
||||
|
||||
Rules:
|
||||
- source_type in ["human", "procedural"] → exempt (never archive)
|
||||
- importance_score >= 0.7 → exempt
|
||||
- archived == True → skip (already archived)
|
||||
- half_life: 90d if importance_score >= 0.3, else 30d
|
||||
- decay_score < 0.1:
|
||||
- If confidence_score >= 0.7 → alert (report, don't archive)
|
||||
- Otherwise → archive (archived = True)
|
||||
- gabi_* collections are completely ignored
|
||||
|
||||
Usage:
|
||||
python3 decay_scanner.py [--collection knowledge_base_hybrid] [--dry-run]
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import json
|
||||
import math
|
||||
import argparse
|
||||
import requests
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
# ─── Config ────────────────────────────────────────────────────────────────
|
||||
QDRANT_URL = os.environ.get("QDRANT_URL", "http://localhost:6333")
|
||||
COLLECTION = os.environ.get("QDRANT_COLLECTION", "knowledge_base")
|
||||
SCROLL_LIMIT = 100 # Qdrant pagination
|
||||
LOG_DIR = Path(os.environ.get("HERMES_LOGS_DIR", str(Path.home() / ".hermes" / "logs")))
|
||||
LOG_FILE = LOG_DIR / "decay_scanner.log"
|
||||
|
||||
# ─── Helpers ──────────────────────────────────────────────────────────────
|
||||
|
||||
def now_iso() -> str:
|
||||
return datetime.now(timezone.utc).isoformat()
|
||||
|
||||
|
||||
def calculate_decay_score(last_accessed_at: str, importance_score: float) -> float:
|
||||
"""
|
||||
Calculate exponential decay: score = exp(-ln(2) * age_days / half_life).
|
||||
More important chunks persist longer (larger half-lives).
|
||||
"""
|
||||
try:
|
||||
last = datetime.fromisoformat(last_accessed_at.replace("Z", "+00:00"))
|
||||
except (ValueError, TypeError):
|
||||
# If timestamp is invalid, assume now (hasn't decayed yet)
|
||||
return 1.0
|
||||
|
||||
now = datetime.now(timezone.utc)
|
||||
age_days = max(0, (now - last).total_seconds() / 86400)
|
||||
|
||||
# Fix: LARGER half-life for more important chunks
|
||||
if importance_score >= 0.3:
|
||||
half_life = 90 # medium/high chunks → 90 days
|
||||
else:
|
||||
half_life = 30 # low chunks → 30 days
|
||||
|
||||
decay_score = math.exp(-math.log(2) * age_days / half_life)
|
||||
return decay_score
|
||||
|
||||
|
||||
def ensure_log_dir():
|
||||
"""Create log directory if it doesn't exist."""
|
||||
LOG_DIR.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
|
||||
def log_message(msg: str):
|
||||
"""Log to stdout and append to log file."""
|
||||
ts = now_iso()
|
||||
line = f"[{ts}] {msg}"
|
||||
print(line)
|
||||
ensure_log_dir()
|
||||
with open(LOG_FILE, "a", encoding="utf-8") as f:
|
||||
f.write(line + "\n")
|
||||
|
||||
|
||||
# ─── Qdrant Operations ────────────────────────────────────────────────────
|
||||
|
||||
def scroll_chunks(collection: str, limit: int = SCROLL_LIMIT):
|
||||
"""
|
||||
Generator that iterates over all points in the collection via scroll.
|
||||
Avoids loading the entire collection into memory.
|
||||
"""
|
||||
offset = None
|
||||
total_scanned = 0
|
||||
|
||||
while True:
|
||||
payload = {
|
||||
"limit": limit,
|
||||
"with_payload": True,
|
||||
"with_vector": False,
|
||||
}
|
||||
if offset is not None:
|
||||
payload["offset"] = offset
|
||||
|
||||
try:
|
||||
resp = requests.post(
|
||||
f"{QDRANT_URL}/collections/{collection}/points/scroll",
|
||||
headers={"Content-Type": "application/json"},
|
||||
json=payload,
|
||||
timeout=30,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
result = data.get("result", {})
|
||||
points = result.get("points", [])
|
||||
|
||||
if not points:
|
||||
break
|
||||
|
||||
for point in points:
|
||||
yield point
|
||||
total_scanned += 1
|
||||
|
||||
offset = result.get("next_page_offset")
|
||||
if offset is None:
|
||||
break
|
||||
|
||||
except Exception as e:
|
||||
log_message(f"❌ Qdrant scroll error: {e}")
|
||||
break
|
||||
|
||||
log_message(f"📊 Total chunks scanned: {total_scanned}")
|
||||
|
||||
|
||||
def update_point_archived(point_id: str, collection: str, decay_score: float, dry_run: bool = False):
|
||||
"""Update point payload: archived=True + calculated decay_score."""
|
||||
if dry_run:
|
||||
log_message(f" [DRY-RUN] Would archive point {point_id} (decay_score={decay_score:.4f})")
|
||||
return True
|
||||
|
||||
try:
|
||||
resp = requests.post(
|
||||
f"{QDRANT_URL}/collections/{collection}/points/payload",
|
||||
headers={"Content-Type": "application/json"},
|
||||
json={
|
||||
"points": [point_id],
|
||||
"payload": {
|
||||
"archived": True,
|
||||
"decay_score": decay_score,
|
||||
},
|
||||
},
|
||||
timeout=10,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return True
|
||||
except Exception as e:
|
||||
log_message(f" ❌ Failed to archive point {point_id}: {e}")
|
||||
return False
|
||||
|
||||
|
||||
# ─── Main ─────────────────────────────────────────────────────────────────
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Decay Scanner — Selective chunk archiving")
|
||||
parser.add_argument("--collection", default=COLLECTION, help="Qdrant collection name")
|
||||
parser.add_argument("--dry-run", action="store_true", help="Simulation — does not modify anything")
|
||||
parser.add_argument("--threshold", type=float, default=0.1, help="Decay threshold for archiving")
|
||||
args = parser.parse_args()
|
||||
|
||||
collection = args.collection
|
||||
|
||||
# Ignore gabi_* collections
|
||||
if collection.startswith("gabi_"):
|
||||
log_message(f"⏭️ Collection '{collection}' is exempt (gabi_*). Exiting.")
|
||||
return
|
||||
|
||||
log_message(f"🚀 Starting decay scanner (collection={collection}, threshold={args.threshold}, dry_run={args.dry_run})")
|
||||
|
||||
# Metrics
|
||||
stats = {
|
||||
"scanned": 0,
|
||||
"archived": 0,
|
||||
"alerted": 0,
|
||||
"skipped_human": 0,
|
||||
"skipped_procedural": 0,
|
||||
"skipped_high_importance": 0,
|
||||
"skipped_already_archived": 0,
|
||||
"failed": 0,
|
||||
}
|
||||
|
||||
alerts = [] # List of alerts (decay < threshold but confidence >= 0.7)
|
||||
|
||||
for point in scroll_chunks(collection):
|
||||
stats["scanned"] += 1
|
||||
|
||||
point_id = point.get("id")
|
||||
payload = point.get("payload", {})
|
||||
|
||||
source_type = payload.get("source_type", "unknown")
|
||||
importance_score = payload.get("importance_score", 0.5)
|
||||
archived = payload.get("archived", False)
|
||||
last_accessed_at = payload.get("last_accessed_at", payload.get("created_at", now_iso()))
|
||||
confidence_score = payload.get("confidence_score", 1.0)
|
||||
|
||||
# Skip: already archived
|
||||
if archived:
|
||||
stats["skipped_already_archived"] += 1
|
||||
continue
|
||||
|
||||
# Skip: human (exempt)
|
||||
if source_type == "human":
|
||||
stats["skipped_human"] += 1
|
||||
continue
|
||||
|
||||
# Skip: procedural (exempt)
|
||||
if source_type == "procedural":
|
||||
stats["skipped_procedural"] += 1
|
||||
continue
|
||||
|
||||
# Skip: high importance
|
||||
if importance_score >= 0.7:
|
||||
stats["skipped_high_importance"] += 1
|
||||
continue
|
||||
|
||||
# Calculate decay
|
||||
decay_score = calculate_decay_score(last_accessed_at, importance_score)
|
||||
|
||||
# Check threshold
|
||||
if decay_score < args.threshold:
|
||||
# Decay-confidence rule: if confidence is high, alert instead of archiving
|
||||
if confidence_score >= 0.7:
|
||||
stats["alerted"] += 1
|
||||
alerts.append({
|
||||
"point_id": point_id,
|
||||
"decay_score": round(decay_score, 4),
|
||||
"confidence_score": round(confidence_score, 2),
|
||||
"importance_score": round(importance_score, 2),
|
||||
"age_days": round((datetime.now(timezone.utc) - datetime.fromisoformat(last_accessed_at.replace("Z", "+00:00"))).total_seconds() / 86400, 1),
|
||||
"reason": "decay < threshold but confidence >= 0.7 — manual review recommended",
|
||||
})
|
||||
log_message(f" ⚠️ ALERT: point {point_id} (decay={decay_score:.4f}, confidence={confidence_score:.2f}) — manual review recommended")
|
||||
else:
|
||||
# Archive
|
||||
ok = update_point_archived(point_id, collection, decay_score, args.dry_run)
|
||||
if ok:
|
||||
stats["archived"] += 1
|
||||
log_message(f" 📦 Archived: point {point_id} (decay={decay_score:.4f}, importance={importance_score:.2f})")
|
||||
else:
|
||||
stats["failed"] += 1
|
||||
|
||||
# Structured JSON report
|
||||
report = {
|
||||
"timestamp": now_iso(),
|
||||
"collection": collection,
|
||||
"threshold": args.threshold,
|
||||
"dry_run": args.dry_run,
|
||||
"scanned": stats["scanned"],
|
||||
"archived": stats["archived"],
|
||||
"alerted": stats["alerted"],
|
||||
"skipped_human": stats["skipped_human"],
|
||||
"skipped_procedural": stats["skipped_procedural"],
|
||||
"skipped_high_importance": stats["skipped_high_importance"],
|
||||
"skipped_already_archived": stats["skipped_already_archived"],
|
||||
"failed": stats["failed"],
|
||||
"alerts": alerts,
|
||||
}
|
||||
|
||||
log_message("=" * 60)
|
||||
log_message("📊 DECAY SCANNER REPORT")
|
||||
log_message("=" * 60)
|
||||
log_message(f" Scanned: {stats['scanned']}")
|
||||
log_message(f" Archived: {stats['archived']}")
|
||||
log_message(f" Alerts (decay+conf.): {stats['alerted']}")
|
||||
log_message(f" Skipped human: {stats['skipped_human']}")
|
||||
log_message(f" Skipped procedural: {stats['skipped_procedural']}")
|
||||
log_message(f" Skipped high imp.: {stats['skipped_high_importance']}")
|
||||
log_message(f" Skipped archived: {stats['skipped_already_archived']}")
|
||||
log_message(f" Failures: {stats['failed']}")
|
||||
log_message("=" * 60)
|
||||
|
||||
# JSON report to stderr (parseable)
|
||||
print(json.dumps(report, ensure_ascii=False, indent=2), file=sys.stderr)
|
||||
|
||||
log_message("✅ Decay scanner complete.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
@ -0,0 +1,211 @@
|
|||
#!/usr/bin/env python3
|
||||
"""
|
||||
DLQ Manager — Reads, classifies, reports, and marks wiki ingest failures.
|
||||
|
||||
Usage:
|
||||
python3 dlq_manager.py --report # report unreported failures
|
||||
python3 dlq_manager.py --status # DLQ status summary
|
||||
python3 dlq_manager.py --json # full JSON output
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import json
|
||||
import time
|
||||
from datetime import datetime, timedelta
|
||||
from typing import List, Dict, Optional
|
||||
from dataclasses import dataclass, asdict, field
|
||||
from collections import Counter
|
||||
|
||||
# ─── Config ────────────────────────────────────────────────────────────────
|
||||
DLQ_PATH = os.environ.get("HERMES_DLQ_PATH", os.path.expanduser("~/.hermes/wiki_ingest_failures.json"))
|
||||
REPORT_LOG = os.environ.get("HERMES_DLQ_REPORT_LOG", os.path.expanduser("~/.hermes/cron/output/dlq_reports.jsonl"))
|
||||
REPORT_DIR = os.environ.get("HERMES_DLQ_REPORT_DIR", os.path.expanduser("~/.hermes/cron/output/quality_report"))
|
||||
MAX_REPORT_HISTORY = 100 # entries in JSONL
|
||||
|
||||
# ─── Data Model ─────────────────────────────────────────────────────────────
|
||||
|
||||
@dataclass
|
||||
class DLQEntry:
|
||||
file: str
|
||||
error: str
|
||||
timestamp: str
|
||||
failure_class: str = "unknown"
|
||||
reported: bool = False
|
||||
retry_count: int = 0
|
||||
last_retry: Optional[str] = None
|
||||
error_hash: str = "" # error hash for deduplication
|
||||
|
||||
# ─── File I/O ─────────────────────────────────────────────────────────────
|
||||
|
||||
def load_dlq() -> List[DLQEntry]:
|
||||
if not os.path.exists(DLQ_PATH):
|
||||
return []
|
||||
try:
|
||||
with open(DLQ_PATH, "r") as f:
|
||||
data = json.load(f)
|
||||
if isinstance(data, list):
|
||||
return [DLQEntry(**item) for item in data]
|
||||
elif isinstance(data, dict) and "failures" in data:
|
||||
return [DLQEntry(**item) for item in data["failures"]]
|
||||
return []
|
||||
except Exception as e:
|
||||
print(f"[DLQ-ERROR] Failed to load: {e}", file=sys.stderr)
|
||||
return []
|
||||
|
||||
def save_dlq(entries: List[DLQEntry]):
|
||||
tmp = DLQ_PATH + ".tmp"
|
||||
with open(tmp, "w") as f:
|
||||
json.dump([asdict(e) for e in entries], f, indent=2)
|
||||
f.flush()
|
||||
os.fsync(f.fileno())
|
||||
os.replace(tmp, DLQ_PATH)
|
||||
|
||||
# ─── Classification ─────────────────────────────────────────────────────────
|
||||
|
||||
def classify_error(error_msg: str) -> str:
|
||||
e = error_msg.lower()
|
||||
transient = ["timeout", "connection", "temporarily", "rate limit", "503", "502", "504",
|
||||
"too many requests", "unavailable", "cannot", "refused", "reset"]
|
||||
permanent = ["404", "not found", "invalid format", "parse error", "file not found",
|
||||
"deleted", "permission denied", "encode", "utf-8", "json", "schema"]
|
||||
for p in transient:
|
||||
if p in e:
|
||||
return "transient"
|
||||
for p in permanent:
|
||||
if p in e:
|
||||
return "permanent"
|
||||
return "unknown"
|
||||
|
||||
def compute_error_hash(file: str, error: str) -> str:
|
||||
"""Generate a simple hash for deduplication of similar errors."""
|
||||
import hashlib
|
||||
return hashlib.md5(f"{file}:{error[:80]}".encode()).hexdigest()[:8]
|
||||
|
||||
# ─── Reporting ─────────────────────────────────────────────────────────────
|
||||
|
||||
def build_report(entries: List[DLQEntry]) -> Dict:
|
||||
unreported = [e for e in entries if not e.reported]
|
||||
total = len(entries)
|
||||
|
||||
if not unreported:
|
||||
return {"status": "ok", "unreported_count": 0, "total": total, "report": ""}
|
||||
|
||||
# Classify
|
||||
for e in unreported:
|
||||
if e.failure_class == "unknown":
|
||||
e.failure_class = classify_error(e.error)
|
||||
|
||||
by_class = Counter(e.failure_class for e in unreported)
|
||||
by_error_short = Counter(str(e.error)[:70] for e in unreported)
|
||||
by_file = Counter(os.path.basename(e.file) for e in unreported)
|
||||
|
||||
lines = [
|
||||
f"🚨 [DLQ-ALERT] {len(unreported)} new failure(s) in ingest",
|
||||
f" Total accumulated in DLQ: {total}",
|
||||
"",
|
||||
"By class:",
|
||||
]
|
||||
emoji = {"transient": "⏳", "permanent": "💀", "unknown": "❓"}
|
||||
for cls, count in by_class.most_common():
|
||||
lines.append(f" {emoji.get(cls, '❓')} {cls}: {count}")
|
||||
|
||||
lines.append("")
|
||||
lines.append("Top errors:")
|
||||
for err, count in by_error_short.most_common(5):
|
||||
lines.append(f" • ({count}x) {err}")
|
||||
|
||||
lines.append("")
|
||||
lines.append("Files:")
|
||||
for fname, count in by_file.most_common(10):
|
||||
lines.append(f" • {fname} ({count}x)")
|
||||
|
||||
report_text = "\n".join(lines)
|
||||
|
||||
return {
|
||||
"status": "alert",
|
||||
"unreported_count": len(unreported),
|
||||
"total": total,
|
||||
"by_class": dict(by_class),
|
||||
"top_errors": dict(by_error_short.most_common(5)),
|
||||
"report": report_text,
|
||||
}
|
||||
|
||||
def save_report(report: Dict):
|
||||
os.makedirs(REPORT_DIR, exist_ok=True)
|
||||
os.makedirs(os.path.dirname(REPORT_LOG), exist_ok=True)
|
||||
timestamp = datetime.now().isoformat()
|
||||
|
||||
# JSONL
|
||||
with open(REPORT_LOG, "a") as f:
|
||||
f.write(json.dumps({"timestamp": timestamp, **report}, ensure_ascii=False) + "\n")
|
||||
f.flush()
|
||||
os.fsync(f.fileno())
|
||||
|
||||
def mark_reported(entries: List[DLQEntry]):
|
||||
for e in entries:
|
||||
e.reported = True
|
||||
|
||||
def get_status_summary(entries: List[DLQEntry]) -> Dict:
|
||||
total = len(entries)
|
||||
unreported = len([e for e in entries if not e.reported])
|
||||
by_class = Counter(e.failure_class for e in entries)
|
||||
recent = [e for e in entries if datetime.now(datetime.timezone.utc) - datetime.fromisoformat(e.timestamp.replace("Z", "+00:00")).astimezone(datetime.timezone.utc) < timedelta(hours=24)]
|
||||
|
||||
return {
|
||||
"total": total,
|
||||
"unreported": unreported,
|
||||
"by_class": dict(by_class),
|
||||
"last_24h": len(recent),
|
||||
"oldest": entries[0].timestamp if entries else None,
|
||||
}
|
||||
|
||||
# ─── Main ───────────────────────────────────────────────────────────────────
|
||||
|
||||
def main():
|
||||
import argparse
|
||||
p = argparse.ArgumentParser(description="DLQ Manager — Auto-report of failures")
|
||||
p.add_argument("--report", action="store_true", help="Generate report of unreported failures")
|
||||
p.add_argument("--status", action="store_true", help="Status summary")
|
||||
p.add_argument("--json", action="store_true", help="JSON output")
|
||||
p.add_argument("--silent-if-ok", action="store_true", help="Silent if DLQ is ok")
|
||||
args = p.parse_args()
|
||||
|
||||
entries = load_dlq()
|
||||
|
||||
if args.status:
|
||||
summary = get_status_summary(entries)
|
||||
if args.json:
|
||||
print(json.dumps(summary, indent=2, ensure_ascii=False))
|
||||
else:
|
||||
print(f"DLQ status: {summary['total']} total, {summary['unreported']} unreported")
|
||||
for cls, count in summary.get("by_class", {}).items():
|
||||
print(f" {cls}: {count}")
|
||||
return
|
||||
|
||||
report = build_report(entries)
|
||||
|
||||
if report["status"] == "ok":
|
||||
msg = "[DLQ-OK] No new failures since last check."
|
||||
if not args.silent_if_ok:
|
||||
print(msg)
|
||||
if args.json:
|
||||
print(json.dumps(report, indent=2, ensure_ascii=False))
|
||||
return
|
||||
|
||||
# Has new failures
|
||||
if args.json:
|
||||
print(json.dumps(report, indent=2, ensure_ascii=False))
|
||||
else:
|
||||
print(report["report"])
|
||||
|
||||
# Save and mark as reported
|
||||
save_report(report)
|
||||
mark_reported(entries)
|
||||
save_dlq(entries)
|
||||
|
||||
# Exit code 1 for cron trigger
|
||||
sys.exit(1)
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
@ -0,0 +1,281 @@
|
|||
#!/usr/bin/env python3
|
||||
"""
|
||||
Semantic Pre-Validator — Decision linter based on the knowledge_base.
|
||||
Queries the vault before I/O actions or API calls.
|
||||
|
||||
Usage:
|
||||
python3 pre_validator.py "POST to Qdrant upsert" # should find pitfalls
|
||||
python3 pre_validator.py --json "use Claude from Anthropic" # JSON output
|
||||
python3 pre_validator.py --domain qdrant,api "modify docker-compose" # restrict search
|
||||
|
||||
Exit codes:
|
||||
0 = pass/warn (action may proceed)
|
||||
1 = blocked (action must be aborted)
|
||||
|
||||
Fail-open: if OpenRouter or Qdrant is offline, allows execution with a warning.
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import json
|
||||
import re
|
||||
import requests
|
||||
from typing import List, Dict, Optional
|
||||
from pathlib import Path
|
||||
|
||||
# ─── Config ────────────────────────────────────────────────────────────────
|
||||
OPENROUTER_KEY = os.environ.get("OPENROUTER_API_KEY")
|
||||
QDRANT_URL = os.environ.get("QDRANT_URL", "http://localhost:6333")
|
||||
COLLECTION = os.environ.get("QDRANT_COLLECTION", "knowledge_base")
|
||||
if not OPENROUTER_KEY:
|
||||
_env_path = os.environ.get("ENV_PATH", "")
|
||||
if _env_path:
|
||||
_env = Path(_env_path)
|
||||
else:
|
||||
_env = Path.home() / ".env"
|
||||
if _env.exists():
|
||||
for ln in _env.read_text().splitlines():
|
||||
if ln.startswith("OPENROUTER_API_KEY="):
|
||||
OPENROUTER_KEY = ln.split("=", 1)[1].strip().strip('"')
|
||||
break
|
||||
EMBEDDING_MODEL = "qwen/qwen3-embedding-8b"
|
||||
TOP_K = 5
|
||||
SCORE_THRESHOLD = 0.60
|
||||
WARN_THRESHOLD = 0.75 # pure wiki docs need a higher score for a warning
|
||||
BLOCK_SEVERITIES = {"critical", "high"}
|
||||
WARN_SEVERITIES = {"medium"}
|
||||
RULE_SOURCES = {"reflection", "decision", "rule", "pitfall", "insight"}
|
||||
REQUEST_TIMEOUT = 10
|
||||
|
||||
# ─── Restriction Patterns in wiki text ─────────────────────────────────────
|
||||
RESTRICTION_KEYWORDS = [
|
||||
"do not use", "must not", "cannot", "never use", "avoid",
|
||||
"forbidden", "not recommended", "anti-pattern", "common mistake",
|
||||
"caution", "warning", "important:", "⚠️", "🚫",
|
||||
"must use", "must always", "requires", "mandatory",
|
||||
"keep", "do not change", "do not modify", "freeze",
|
||||
]
|
||||
|
||||
def contains_restriction(text: str) -> bool:
|
||||
"""Check whether text contains restriction/decision patterns."""
|
||||
if not text:
|
||||
return False
|
||||
text_lower = text.lower()
|
||||
return any(kw in text_lower for kw in RESTRICTION_KEYWORDS)
|
||||
|
||||
# ─── Domain Tag Inference ─────────────────────────────────────────────────
|
||||
DOMAIN_PATTERNS = {
|
||||
"docker" : ["docker", "compose", "container", "image", "dockerfile"],
|
||||
"qdrant" : ["qdrant", "collection", "points", "upsert", "vector", "vectors", "embedding"],
|
||||
"redis" : ["redis", "arq", "queue", "job", "worker", "broker"],
|
||||
"openrouter" : ["openrouter", "embedding", "api_key", "openai", "api_base", "model"],
|
||||
"hermes" : ["hermes", "config.yaml", "skill", "cron", "gateway", "cli"],
|
||||
"wiki" : ["wiki", "raw/", "ingest", "vault", "obsidian", "knowledge_base"],
|
||||
"webui" : ["webui", "open-webui", "frontend", "chat", "rag"],
|
||||
"infra" : ["deploy", "server", "systemd", "service", "port", "host"],
|
||||
"security" : ["password", "secret", "token", "auth", "permission", "sudo"],
|
||||
"maas" : ["maas", "memory", "cognitive", "agent"],
|
||||
}
|
||||
|
||||
def infer_domain_tags(description: str) -> List[str]:
|
||||
d = description.lower()
|
||||
found = set()
|
||||
for domain, pats in DOMAIN_PATTERNS.items():
|
||||
if any(p in d for p in pats):
|
||||
found.add(domain)
|
||||
return sorted(found)
|
||||
|
||||
# ─── Core ───────────────────────────────────────────────────────────────────
|
||||
|
||||
def embed_text(text: str) -> Optional[List[float]]:
|
||||
if not OPENROUTER_KEY:
|
||||
return None
|
||||
try:
|
||||
r = requests.post(
|
||||
"https://openrouter.ai/api/v1/embeddings",
|
||||
headers={
|
||||
"Authorization": f"Bearer {OPENROUTER_KEY}",
|
||||
"Content-Type": "application/json"
|
||||
},
|
||||
json={"model": EMBEDDING_MODEL, "input": text[:8000]},
|
||||
timeout=REQUEST_TIMEOUT
|
||||
)
|
||||
r.raise_for_status()
|
||||
return r.json()["data"][0]["embedding"]
|
||||
except Exception as e:
|
||||
print(f"[PV-ERROR] Embedding failed: {e}", file=sys.stderr)
|
||||
return None
|
||||
|
||||
def search_knowledge_base(vector: List[float], domain_tags: List[str]) -> List[Dict]:
|
||||
try:
|
||||
r = requests.post(
|
||||
f"{QDRANT_URL}/collections/{COLLECTION}/points/search",
|
||||
headers={"Content-Type": "application/json"},
|
||||
json={"vector": vector, "limit": TOP_K * 3, "with_payload": True},
|
||||
timeout=REQUEST_TIMEOUT
|
||||
)
|
||||
r.raise_for_status()
|
||||
hits = []
|
||||
for item in r.json().get("result", []):
|
||||
pld = item.get("payload", {})
|
||||
src = str(pld.get("source", "")).lower()
|
||||
sev = str(pld.get("severity", pld.get("decision_severity", "low"))).lower()
|
||||
tags = [str(t).lower() for t in pld.get("tags", [])]
|
||||
score = item.get("score", 0)
|
||||
|
||||
# If domain filters requested, require overlap
|
||||
if domain_tags:
|
||||
dom_low = [d.lower() for d in domain_tags]
|
||||
if not set(dom_low) & set(tags):
|
||||
continue
|
||||
|
||||
hits.append({
|
||||
"id" : str(item.get("id", "")),
|
||||
"score" : score,
|
||||
"title" : pld.get("title", "Untitled"),
|
||||
"text" : (pld.get("text", "") or "")[:400],
|
||||
"source" : src,
|
||||
"severity": sev,
|
||||
"tags" : tags,
|
||||
})
|
||||
hits.sort(key=lambda x: x["score"], reverse=True)
|
||||
return hits[:TOP_K]
|
||||
except Exception as e:
|
||||
print(f"[PV-ERROR] Qdrant search failed: {e}", file=sys.stderr)
|
||||
return []
|
||||
|
||||
def is_rule_hit(hit: Dict) -> bool:
|
||||
"""Return True if the hit contains an explicit rule (reflection/decision/rule/insight/pitfall)."""
|
||||
return any(s in hit["source"] for s in RULE_SOURCES)
|
||||
|
||||
def classify_hit(hit: Dict, action_desc: str) -> str:
|
||||
"""
|
||||
Return hit category: 'block', 'warn', 'info', or 'none'.
|
||||
Considers both source=reflection/decision/rule and restriction patterns
|
||||
embedded in wiki document text.
|
||||
"""
|
||||
sev = hit.get("severity", "low")
|
||||
is_rule = is_rule_hit(hit) or contains_restriction(hit.get("text", ""))
|
||||
score = hit.get("score", 0)
|
||||
|
||||
# If text contains restriction, give it more weight
|
||||
restriction_bonus = 0.08 if contains_restriction(hit.get("text", "")) else 0
|
||||
effective_score = score + restriction_bonus
|
||||
|
||||
# Proximity: if the action term (e.g. "POST") appears near a keyword in the text
|
||||
action_terms = set(action_desc.lower().split())
|
||||
text_lower = (hit.get("text", "") or "").lower()
|
||||
text_words = set(text_lower.split())
|
||||
proximity_match = len(action_terms & text_words) > 0
|
||||
|
||||
# If restriction + proximity → elevate severity
|
||||
has_restriction = contains_restriction(hit.get("text", "")) and proximity_match
|
||||
|
||||
if is_rule or has_restriction:
|
||||
if sev in BLOCK_SEVERITIES or (has_restriction and effective_score >= 0.65):
|
||||
return "block"
|
||||
elif sev in WARN_SEVERITIES or (has_restriction and effective_score >= SCORE_THRESHOLD):
|
||||
return "warn"
|
||||
|
||||
# For normal wiki documents, only warn if score is very high
|
||||
if effective_score >= WARN_THRESHOLD:
|
||||
return "warn"
|
||||
if effective_score >= SCORE_THRESHOLD:
|
||||
return "info"
|
||||
return "none"
|
||||
|
||||
def validate_action(action_description: str, domain_tags: Optional[List[str]] = None) -> Dict:
|
||||
try:
|
||||
dom = domain_tags or infer_domain_tags(action_description)
|
||||
vec = embed_text(action_description)
|
||||
if vec is None:
|
||||
return {"status": "pass", "blocked": False, "message": "⚠️ Validator offline. Proceeding with caution.", "action": action_description}
|
||||
|
||||
hits = search_knowledge_base(vec, dom)
|
||||
blockers = []
|
||||
warnings = []
|
||||
infos = []
|
||||
|
||||
for h in hits:
|
||||
cat = classify_hit(h, action_description)
|
||||
if cat == "block":
|
||||
blockers.append(h)
|
||||
elif cat == "warn":
|
||||
warnings.append(h)
|
||||
elif cat == "info":
|
||||
infos.append(h)
|
||||
|
||||
if blockers:
|
||||
lines = [f"🚫 ACTION BLOCKED — {len(blockers)} critical rule(s) in the vault:"]
|
||||
for b in blockers:
|
||||
lines.append(f" • [{b['severity'].upper()}] {b['title']} (score: {b['score']:.2f})")
|
||||
lines.append(f" {b['text'][:200]}...")
|
||||
lines.append("")
|
||||
lines.append("Override? Type 'force' (not recommended).")
|
||||
return {
|
||||
"status": "blocked", "blocked": True,
|
||||
"blockers": blockers, "warnings": warnings,
|
||||
"message": "\n".join(lines), "action": action_description, "domain": dom,
|
||||
}
|
||||
|
||||
if warnings:
|
||||
lines = [f"⚠️ {len(warnings)} warning(s) found in the vault:"]
|
||||
for w in warnings:
|
||||
lines.append(f" • [{w['severity'].upper()}] {w['title']} (score: {w['score']:.2f})")
|
||||
lines.append(f" {w['text'][:200]}...")
|
||||
return {
|
||||
"status": "warn", "blocked": False,
|
||||
"warnings": warnings, "infos": infos,
|
||||
"message": "\n".join(lines), "action": action_description, "domain": dom,
|
||||
}
|
||||
|
||||
if infos:
|
||||
return {
|
||||
"status": "info", "blocked": False,
|
||||
"infos": infos,
|
||||
"message": f"ℹ️ {len(infos)} relevant document(s), none critical.",
|
||||
"action": action_description, "domain": dom,
|
||||
}
|
||||
|
||||
return {
|
||||
"status": "pass", "blocked": False,
|
||||
"message": "No relevant insights found. Execution authorized.",
|
||||
"action": action_description, "domain": dom,
|
||||
}
|
||||
except Exception as e:
|
||||
return {
|
||||
"status": "pass", "blocked": False,
|
||||
"message": f"Validator failed ({e}). Proceeding with caution.",
|
||||
"action": action_description, "domain": [],
|
||||
}
|
||||
|
||||
# ─── Main ───────────────────────────────────────────────────────────────────
|
||||
def main():
|
||||
import argparse
|
||||
p = argparse.ArgumentParser(description="Semantic Pre-Validator")
|
||||
p.add_argument("action", nargs="?", help="Action description")
|
||||
p.add_argument("--domain", help="Comma-separated domain tags")
|
||||
p.add_argument("--json", action="store_true", help="JSON output")
|
||||
p.add_argument("--silent", action="store_true", help="Silent — exit code only")
|
||||
p.add_argument("--force-block", action="store_true", help="Force block (testing)")
|
||||
args = p.parse_args()
|
||||
|
||||
action = args.action or sys.stdin.read().strip() or "POST to Qdrant upsert endpoint"
|
||||
dom = [x.strip() for x in args.domain.split(",")] if args.domain else None
|
||||
|
||||
res = validate_action(action, dom)
|
||||
if args.force_block:
|
||||
res["blocked"] = True
|
||||
res["status"] = "blocked"
|
||||
|
||||
if args.json:
|
||||
print(json.dumps(res, indent=2, ensure_ascii=False, default=str))
|
||||
elif not args.silent:
|
||||
print(res["message"])
|
||||
if res["blocked"]:
|
||||
print("\n(Use --force-block to test validator bypass)")
|
||||
|
||||
sys.exit(1 if res["blocked"] else 0)
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
@ -0,0 +1,219 @@
|
|||
#!/usr/bin/env python3
|
||||
"""
|
||||
reflection_trigger.py
|
||||
Checks whether the ARQ worker is idle (no pending/running jobs)
|
||||
and dispatches a micro_reflection via ARQ enqueue. Runs via cron every 5 minutes.
|
||||
|
||||
Rules:
|
||||
- Only triggers if there are no pending or running jobs (idle)
|
||||
- Respects the max_per_hour budget (reads from env or defaults to 5)
|
||||
- Enqueues ARQ job "process_micro_reflection" (function registered in the worker)
|
||||
- Fail-open: if Redis/ARQ is unavailable, exits silently
|
||||
- Never blocks the critical query/ingestion path
|
||||
|
||||
Usage (cron):
|
||||
*/5 * * * * $VENV_DIR/bin/python $PROJECT_DIR/scripts/reflection_trigger.py >> $HERMES_LOG_DIR/reflection_trigger.cron.log 2>&1
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import json
|
||||
import asyncio
|
||||
import argparse
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from arq import create_pool
|
||||
from arq.connections import RedisSettings
|
||||
import redis.asyncio as aioredis
|
||||
|
||||
# ─── Config ────────────────────────────────────────────────────────────────
|
||||
ENV_PATH = Path(os.environ.get("MAA_ENV_PATH", "."))
|
||||
if ENV_PATH.exists():
|
||||
load_dotenv(ENV_PATH)
|
||||
|
||||
REDIS_HOST = os.environ.get("REDIS_HOST", "127.0.0.1")
|
||||
REDIS_PORT = int(os.environ.get("REDIS_PORT", "6379"))
|
||||
REDIS_PASSWORD = os.environ.get("REDIS_PASSWORD", "")
|
||||
MAX_REFLECTIONS_PER_HOUR = int(os.environ.get("MICRO_REFLECTION_MAX_PER_HOUR", "5"))
|
||||
|
||||
redis_settings = RedisSettings(
|
||||
host=REDIS_HOST,
|
||||
port=REDIS_PORT,
|
||||
password=REDIS_PASSWORD or None,
|
||||
)
|
||||
|
||||
LOG_FILE = Path(
|
||||
os.environ.get(
|
||||
"REFLECTION_LOG_PATH",
|
||||
str(Path.home() / ".hermes" / "logs" / "reflection_trigger.log")
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def log_message(msg: str):
|
||||
ts = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
|
||||
line = f"[{ts}] {msg}"
|
||||
try:
|
||||
LOG_FILE.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(LOG_FILE, "a", encoding="utf-8") as f:
|
||||
f.write(line + "\n")
|
||||
except Exception:
|
||||
pass
|
||||
print(line)
|
||||
|
||||
|
||||
async def is_idle() -> bool:
|
||||
"""Check whether there are any pending or running jobs in ARQ."""
|
||||
try:
|
||||
r = aioredis.Redis(
|
||||
host=REDIS_HOST, port=REDIS_PORT,
|
||||
password=REDIS_PASSWORD or None,
|
||||
decode_responses=True,
|
||||
)
|
||||
|
||||
# ARQ stores jobs in queues like 'arq:queue:default'
|
||||
queue_names = ["arq:queue:default"]
|
||||
qr_prefix = os.environ.get("ARQ_QUEUE_PREFIX", "arq:queue:")
|
||||
if qr_prefix:
|
||||
try:
|
||||
found = await r.keys(f"{qr_prefix}*")
|
||||
queue_names = list(found) if found else queue_names
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
total_pending = 0
|
||||
for qn in queue_names:
|
||||
try:
|
||||
total_pending += await r.llen(qn)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# In-progress jobs: ARQ uses sets like 'arq:in-progress:...'
|
||||
in_progress_keys = await r.keys("arq:in-progress:*")
|
||||
total_in_progress = 0
|
||||
for key in in_progress_keys:
|
||||
try:
|
||||
total_in_progress += await r.scard(key)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
await r.aclose()
|
||||
return (total_pending + total_in_progress) == 0
|
||||
except Exception as e:
|
||||
log_message(f"Error checking idle status: {e}")
|
||||
return False # fail-safe: if unable to verify, do not trigger
|
||||
|
||||
|
||||
async def check_budget() -> tuple[bool, int, int]:
|
||||
"""Return (allowed, used, max) based on the hourly counter in SQLite."""
|
||||
try:
|
||||
import sqlite3
|
||||
db_path = Path(
|
||||
os.environ.get(
|
||||
"STATE_DB_PATH",
|
||||
str(Path.home() / ".hermes" / "state.db")
|
||||
)
|
||||
)
|
||||
hour_window = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H")
|
||||
conn = sqlite3.connect(str(db_path))
|
||||
c = conn.cursor()
|
||||
c.execute("SELECT count FROM reflection_budget WHERE hour_window = ?", (hour_window,))
|
||||
row = c.fetchone()
|
||||
used = row[0] if row else 0
|
||||
conn.close()
|
||||
return (used < MAX_REFLECTIONS_PER_HOUR, used, MAX_REFLECTIONS_PER_HOUR)
|
||||
except Exception as e:
|
||||
log_message(f"Error checking budget: {e}")
|
||||
return (True, 0, MAX_REFLECTIONS_PER_HOUR) # fail-open
|
||||
|
||||
|
||||
def increment_budget():
|
||||
"""Increment the reflection counter in SQLite."""
|
||||
try:
|
||||
import sqlite3
|
||||
db_path = Path(
|
||||
os.environ.get(
|
||||
"STATE_DB_PATH",
|
||||
str(Path.home() / ".hermes" / "state.db")
|
||||
)
|
||||
)
|
||||
hour_window = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H")
|
||||
conn = sqlite3.connect(str(db_path))
|
||||
c = conn.cursor()
|
||||
c.execute("""
|
||||
INSERT INTO reflection_budget (hour_window, count, tokens_used)
|
||||
VALUES (?, 1, 0)
|
||||
ON CONFLICT(hour_window)
|
||||
DO UPDATE SET count = count + 1
|
||||
""", (hour_window,))
|
||||
conn.commit()
|
||||
conn.close()
|
||||
except Exception as e:
|
||||
log_message(f"Error incrementing budget: {e}")
|
||||
|
||||
|
||||
async def trigger_micro_reflection(dry_run: bool = False) -> dict:
|
||||
"""Pipeline: idle check → budget check → ARQ enqueue → increment budget."""
|
||||
|
||||
# 1. Idle check
|
||||
idle = await is_idle()
|
||||
if not idle:
|
||||
return {"status": "busy", "triggered": False}
|
||||
|
||||
# 2. Budget check
|
||||
budget_ok, used, max_ref = await check_budget()
|
||||
if not budget_ok:
|
||||
return {
|
||||
"status": "budget_exceeded",
|
||||
"used": used,
|
||||
"max": max_ref,
|
||||
"triggered": False,
|
||||
}
|
||||
|
||||
# 3. Enqueue
|
||||
if dry_run:
|
||||
return {
|
||||
"status": "would_trigger",
|
||||
"triggered": False,
|
||||
"used": used,
|
||||
"max": max_ref,
|
||||
}
|
||||
|
||||
try:
|
||||
pool = await create_pool(redis_settings)
|
||||
job = await pool.enqueue_job("process_micro_reflection")
|
||||
await pool.aclose()
|
||||
|
||||
# 4. Budget accounting is owned by the worker after actual processing.
|
||||
# NOT incremented here — the worker's increment_budget() call
|
||||
# handles this, preventing double-counting.
|
||||
# increment_budget()
|
||||
|
||||
return {
|
||||
"status": "triggered",
|
||||
"triggered": True,
|
||||
"job_id": str(job.job_id) if job else None,
|
||||
"used": used + 1,
|
||||
"max": max_ref,
|
||||
}
|
||||
except Exception as e:
|
||||
log_message(f"Error enqueuing micro_reflection: {e}")
|
||||
return {"status": "error", "error": str(e), "triggered": False}
|
||||
|
||||
|
||||
async def main():
|
||||
parser = argparse.ArgumentParser(description="Reflection Trigger — idle detection")
|
||||
parser.add_argument("--dry-run", action="store_true", help="Simulate, do not enqueue")
|
||||
args = parser.parse_args()
|
||||
|
||||
result = await trigger_micro_reflection(dry_run=args.dry_run)
|
||||
log_message(json.dumps(result, ensure_ascii=False))
|
||||
|
||||
if result.get("status") == "error":
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
|
|
@ -0,0 +1,272 @@
|
|||
#!/usr/bin/env python3
|
||||
"""
|
||||
semantic_dedup.py
|
||||
Monthly scanner for near-duplicates in knowledge_base_hybrid via cosine similarity.
|
||||
Runs on the first Sunday of each month (cron: 0 3 1 * *).
|
||||
|
||||
⚠️ WARNING: This performs O(n²) brute-force pairwise comparisons. For large
|
||||
collections (e.g. 100K+ points), this can be extremely slow and memory-heavy.
|
||||
Use --max-points to limit processing, or prefer Qdrant's built-in
|
||||
nearest-neighbor search on a random sample where feasible.
|
||||
|
||||
Rules:
|
||||
- Ignores gabi_* collections
|
||||
- Does not delete automatically — only emits a JSON report of candidates
|
||||
- Similarity threshold: 0.92 (configurable)
|
||||
- Merge is handled via upserts in file_ingestion.py (pre-write dedup)
|
||||
- This script does the retrospective scan of the entire collection
|
||||
(capped by MAX_POINTS)
|
||||
|
||||
Usage:
|
||||
python3 semantic_dedup.py [--collection knowledge_base_hybrid] [--threshold 0.92] [--dry-run] [--max-points 5000]
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import json
|
||||
import math
|
||||
import argparse
|
||||
import requests
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import List, Dict, Tuple, Optional
|
||||
|
||||
# ─── Config ────────────────────────────────────────────────────────────────
|
||||
QDRANT_URL = os.environ.get("QDRANT_URL", "http://localhost:6333")
|
||||
COLLECTION = os.environ.get("QDRANT_COLLECTION", "knowledge_base")
|
||||
SCROLL_LIMIT = 50 # Qdrant pagination (avoids timeout on large collections)
|
||||
SIMILARITY_THRESHOLD = 0.92
|
||||
TOP_NEIGHBORS = 10
|
||||
|
||||
LOG_DIR = Path(
|
||||
os.environ.get("HERMES_LOG_DIR", str(Path.home() / ".hermes" / "logs"))
|
||||
)
|
||||
LOG_FILE = LOG_DIR / "semantic_dedup.log"
|
||||
REPORT_FILE = LOG_DIR / "semantic_dedup_report.json"
|
||||
|
||||
# Safety cap — limit processed points to avoid O(n²) blowup on large collections
|
||||
MAX_POINTS = int(os.environ.get("DEDUP_MAX_POINTS", "5000"))
|
||||
|
||||
|
||||
def now_iso() -> str:
|
||||
return datetime.now(timezone.utc).isoformat()
|
||||
|
||||
|
||||
def log_message(msg: str):
|
||||
ts = now_iso()
|
||||
line = f"[{ts}] {msg}"
|
||||
print(line)
|
||||
try:
|
||||
LOG_DIR.mkdir(parents=True, exist_ok=True)
|
||||
with open(LOG_FILE, "a", encoding="utf-8") as f:
|
||||
f.write(line + "\n")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
# ─── Qdrant Operations ────────────────────────────────────────────────────
|
||||
|
||||
def scroll_all_chunks(collection: str) -> List[Dict]:
|
||||
"""
|
||||
Load all points from the collection, paginating via scroll.
|
||||
Returns a list of {id, vector, payload}.
|
||||
"""
|
||||
all_chunks = []
|
||||
offset = None
|
||||
scanned = 0
|
||||
|
||||
while True:
|
||||
payload = {
|
||||
"limit": SCROLL_LIMIT,
|
||||
"with_payload": True,
|
||||
"with_vector": True,
|
||||
}
|
||||
if offset is not None:
|
||||
payload["offset"] = offset
|
||||
|
||||
try:
|
||||
resp = requests.post(
|
||||
f"{QDRANT_URL}/collections/{collection}/points/scroll",
|
||||
headers={"Content-Type": "application/json"},
|
||||
json=payload,
|
||||
timeout=30,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
result = data.get("result", {})
|
||||
points = result.get("points", [])
|
||||
|
||||
if not points:
|
||||
break
|
||||
|
||||
for point in points:
|
||||
# Get only the dense vector for similarity
|
||||
vector = point.get("vector")
|
||||
dense = None
|
||||
if isinstance(vector, dict):
|
||||
dense = vector.get("dense")
|
||||
elif isinstance(vector, list):
|
||||
dense = vector # fallback: simple vector
|
||||
|
||||
if dense:
|
||||
all_chunks.append({
|
||||
"id": point.get("id"),
|
||||
"vector": dense,
|
||||
"payload": point.get("payload", {}),
|
||||
})
|
||||
|
||||
scanned += len(points)
|
||||
offset = result.get("next_page_offset")
|
||||
if offset is None:
|
||||
break
|
||||
|
||||
except Exception as e:
|
||||
log_message(f"❌ Error in Qdrant scroll: {e}")
|
||||
break
|
||||
|
||||
log_message(f"📊 Total chunks loaded: {len(all_chunks)} / {scanned} scanned")
|
||||
return all_chunks
|
||||
|
||||
|
||||
def cosine_similarity(v1: List[float], v2: List[float]) -> float:
|
||||
"""Calculate cosine similarity between two vectors."""
|
||||
if len(v1) != len(v2):
|
||||
return 0.0
|
||||
|
||||
dot = sum(a * b for a, b in zip(v1, v2))
|
||||
norm1 = math.sqrt(sum(a * a for a in v1))
|
||||
norm2 = math.sqrt(sum(b * b for b in v2))
|
||||
|
||||
if norm1 == 0 or norm2 == 0:
|
||||
return 0.0
|
||||
|
||||
return dot / (norm1 * norm2)
|
||||
|
||||
|
||||
def find_near_duplicates(chunks: List[Dict], threshold: float = SIMILARITY_THRESHOLD) -> List[Dict]:
|
||||
"""
|
||||
Find near-duplicate pairs via brute-force cosine similarity.
|
||||
Optimization: upper-triangular matrix comparison.
|
||||
Returns list of {chunk_id_a, chunk_id_b, similarity}.
|
||||
"""
|
||||
n = len(chunks)
|
||||
if n < 2:
|
||||
return []
|
||||
|
||||
candidates = []
|
||||
ids_seen = set() # avoid duplicates (A,B) and (B,A)
|
||||
|
||||
for i in range(n):
|
||||
for j in range(i + 1, n):
|
||||
# Fast heuristic: skip if texts differ greatly in size
|
||||
text_len_i = len(chunks[i]["payload"].get("text", ""))
|
||||
text_len_j = len(chunks[j]["payload"].get("text", ""))
|
||||
if text_len_i > 0 and text_len_j > 0:
|
||||
ratio = min(text_len_i, text_len_j) / max(text_len_i, text_len_j)
|
||||
if ratio < 0.5: # Very different sizes, skip
|
||||
continue
|
||||
|
||||
sim = cosine_similarity(chunks[i]["vector"], chunks[j]["vector"])
|
||||
if sim >= threshold:
|
||||
pair_key = tuple(sorted([str(chunks[i]["id"]), str(chunks[j]["id"])]))
|
||||
if pair_key not in ids_seen:
|
||||
ids_seen.add(pair_key)
|
||||
candidates.append({
|
||||
"chunk_id_a": chunks[i]["id"],
|
||||
"chunk_id_b": chunks[j]["id"],
|
||||
"similarity": round(sim, 6),
|
||||
"source_a": chunks[i]["payload"].get("source", "unknown"),
|
||||
"source_b": chunks[j]["payload"].get("source", "unknown"),
|
||||
"title_a": chunks[i]["payload"].get("title", "")[:60],
|
||||
"title_b": chunks[j]["payload"].get("title", "")[:60],
|
||||
"text_preview_a": chunks[i]["payload"].get("text", "")[:100],
|
||||
"text_preview_b": chunks[j]["payload"].get("text", "")[:100],
|
||||
})
|
||||
|
||||
# Sort by descending similarity
|
||||
candidates.sort(key=lambda x: x["similarity"], reverse=True)
|
||||
return candidates
|
||||
|
||||
|
||||
def generate_report(candidates: List[Dict], collection: str, threshold: float, scanned: int) -> Dict:
|
||||
"""Generate structured JSON report."""
|
||||
return {
|
||||
"timestamp": now_iso(),
|
||||
"collection": collection,
|
||||
"threshold": threshold,
|
||||
"scanned_chunks": scanned,
|
||||
"near_duplicate_pairs": len(candidates),
|
||||
"candidates": candidates,
|
||||
"recommendation": (
|
||||
f"{len(candidates)} near-duplicate pairs found. "
|
||||
"Review manually and apply merge via Qdrant point update if approved."
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Semantic Dedup Scanner")
|
||||
parser.add_argument("--collection", default=COLLECTION, help="Qdrant collection name")
|
||||
parser.add_argument("--threshold", type=float, default=SIMILARITY_THRESHOLD, help="Cosine similarity threshold")
|
||||
parser.add_argument("--max-points", type=int, default=MAX_POINTS, help="Max points to process (cap O(n²))")
|
||||
parser.add_argument("--dry-run", action="store_true", help="Scan only, do not save report")
|
||||
args = parser.parse_args()
|
||||
|
||||
collection = args.collection
|
||||
|
||||
# Skip gabi_* collections
|
||||
if collection.startswith("gabi_"):
|
||||
log_message(f"⏭️ Collection '{collection}' is exempt (gabi_*). Exiting.")
|
||||
return
|
||||
|
||||
log_message(f"🚀 Starting semantic dedup (collection={collection}, threshold={args.threshold}, dry_run={args.dry_run})")
|
||||
|
||||
# Load chunks (capped by --max-points to avoid O(n²) blowup)
|
||||
chunks = scroll_all_chunks(collection)
|
||||
if args.max_points and len(chunks) > args.max_points:
|
||||
log_message(f"⚠️ Collection has {len(chunks)} points, truncating to {args.max_points} (use --max-points to change)")
|
||||
chunks = chunks[:args.max_points]
|
||||
|
||||
if not chunks:
|
||||
log_message("⚠️ No chunks found in the collection.")
|
||||
return
|
||||
|
||||
# Find near-duplicates
|
||||
log_message(f"🔍 Analyzing similarity among {len(chunks)} chunks...")
|
||||
candidates = find_near_duplicates(chunks, threshold=args.threshold)
|
||||
|
||||
# Generate report
|
||||
report = generate_report(candidates, collection, args.threshold, len(chunks))
|
||||
|
||||
log_message("=" * 60)
|
||||
log_message("📊 SEMANTIC DEDUP REPORT")
|
||||
log_message("=" * 60)
|
||||
log_message(f" Chunks scanned: {report['scanned_chunks']}")
|
||||
log_message(f" Near-duplicate pairs: {report['near_duplicate_pairs']}")
|
||||
|
||||
if candidates:
|
||||
log_message(f" Top similarity: {candidates[0]['similarity']:.4f}")
|
||||
log_message(f" Top pair: {candidates[0]['chunk_id_a']} ↔ {candidates[0]['chunk_id_b']}")
|
||||
else:
|
||||
log_message(" No near-duplicates found.")
|
||||
|
||||
log_message("=" * 60)
|
||||
|
||||
# Save JSON report
|
||||
if not args.dry_run and candidates:
|
||||
try:
|
||||
REPORT_FILE.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(REPORT_FILE, "w", encoding="utf-8") as f:
|
||||
json.dump(report, f, ensure_ascii=False, indent=2)
|
||||
log_message(f"📄 Report saved: {REPORT_FILE}")
|
||||
except Exception as e:
|
||||
log_message(f"❌ Error saving report: {e}")
|
||||
|
||||
# Output JSON to stderr (parseable)
|
||||
print(json.dumps(report, ensure_ascii=False, indent=2), file=sys.stderr)
|
||||
|
||||
log_message("✅ Semantic dedup complete.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
@ -0,0 +1,188 @@
|
|||
#!/usr/bin/env python3
|
||||
"""
|
||||
wiki-continuous-ingest.py
|
||||
Detects new/modified .md files in the vault and enqueues them to the ARQ worker.
|
||||
Runs on the host, accesses local Redis (127.0.0.1:6379) and Qdrant (localhost:6333).
|
||||
"""
|
||||
import os
|
||||
import sys
|
||||
import json
|
||||
import hashlib
|
||||
import asyncio
|
||||
from pathlib import Path
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from arq import create_pool
|
||||
from arq.connections import RedisSettings
|
||||
import redis.asyncio as aioredis
|
||||
|
||||
# ─── Config ────────────────────────────────────────────────────────────────
|
||||
ENV_PATH = os.environ.get("ENV_PATH", "")
|
||||
if ENV_PATH:
|
||||
env_p = Path(ENV_PATH)
|
||||
if env_p.exists():
|
||||
load_dotenv(env_p)
|
||||
|
||||
WIKI_ROOT = Path(os.environ.get("WIKI_ROOT", "."))
|
||||
STATE_DIR = Path(os.environ.get("HERMES_STATE_DIR", str(Path.home() / ".hermes")))
|
||||
STATE_FILE = STATE_DIR / "wiki_ingest_state.json"
|
||||
FAILURES_FILE = STATE_DIR / "wiki_ingest_failures.json"
|
||||
REDIS_PASSWORD = os.environ.get("REDIS_PASSWORD", "")
|
||||
|
||||
redis_settings = RedisSettings(
|
||||
host="127.0.0.1",
|
||||
port=6379,
|
||||
password=REDIS_PASSWORD or None,
|
||||
)
|
||||
|
||||
|
||||
def load_state() -> dict:
|
||||
if STATE_FILE.exists():
|
||||
with open(STATE_FILE) as f:
|
||||
return json.load(f)
|
||||
return {}
|
||||
|
||||
|
||||
def save_state(state: dict):
|
||||
"""Atomic write via tempfile + rename to avoid corruption."""
|
||||
STATE_FILE.parent.mkdir(parents=True, exist_ok=True)
|
||||
tmp = STATE_FILE.with_suffix(".tmp")
|
||||
with open(tmp, "w") as f:
|
||||
json.dump(state, f, indent=2)
|
||||
f.flush()
|
||||
os.fsync(f.fileno())
|
||||
os.replace(tmp, STATE_FILE)
|
||||
|
||||
|
||||
def file_hash(path: Path) -> str:
|
||||
return hashlib.sha256(path.read_bytes()).hexdigest()[:16]
|
||||
|
||||
|
||||
async def redis_ready() -> bool:
|
||||
"""Check whether Redis is accessible before enqueuing."""
|
||||
try:
|
||||
r = aioredis.Redis(
|
||||
host="127.0.0.1", port=6379,
|
||||
password=REDIS_PASSWORD or None,
|
||||
socket_connect_timeout=3,
|
||||
socket_timeout=3,
|
||||
)
|
||||
ok = await r.ping()
|
||||
await r.aclose()
|
||||
return bool(ok)
|
||||
except Exception as e:
|
||||
print(f" ⚠️ Redis unavailable: {e}")
|
||||
return False
|
||||
|
||||
|
||||
async def main():
|
||||
if not await redis_ready():
|
||||
print("❌ Redis not ready. Docker stack may still be starting up. Aborting.")
|
||||
return
|
||||
|
||||
state = load_state()
|
||||
new_files = []
|
||||
modified_files = []
|
||||
skipped = 0
|
||||
total = 0
|
||||
|
||||
# Scan all .md files
|
||||
for path in sorted(WIKI_ROOT.rglob("*.md")):
|
||||
total += 1
|
||||
rel = str(path.relative_to(WIKI_ROOT))
|
||||
mtime = path.stat().st_mtime
|
||||
current_hash = file_hash(path)
|
||||
|
||||
if rel not in state:
|
||||
new_files.append(rel)
|
||||
state[rel] = {"mtime": mtime, "hash": current_hash, "queued_at": None, "ingested_at": None}
|
||||
elif state[rel]["hash"] != current_hash:
|
||||
modified_files.append(rel)
|
||||
state[rel]["mtime"] = mtime
|
||||
state[rel]["hash"] = current_hash
|
||||
state[rel]["queued_at"] = None
|
||||
state[rel]["ingested_at"] = None
|
||||
else:
|
||||
skipped += 1
|
||||
|
||||
files_to_ingest = new_files + modified_files
|
||||
|
||||
if not files_to_ingest:
|
||||
print(f"⏭️ Nothing new. {total} files tracked, {skipped} unchanged.")
|
||||
return
|
||||
|
||||
# Enqueue in ARQ
|
||||
redis = await create_pool(redis_settings)
|
||||
enqueued = 0
|
||||
failed = 0
|
||||
failures = []
|
||||
|
||||
for rel_path in files_to_ingest:
|
||||
abs_path = str(WIKI_ROOT / rel_path)
|
||||
try:
|
||||
job = await redis.enqueue_job(
|
||||
"process_wiki_file",
|
||||
file_path=f"/wiki/{rel_path}", # path inside container
|
||||
)
|
||||
state[rel_path]["queued_at"] = datetime.now(timezone.utc).isoformat()
|
||||
enqueued += 1
|
||||
print(f" ✅ Enqueued: {rel_path} (job: {job.job_id[:8]})")
|
||||
except Exception as e:
|
||||
failed += 1
|
||||
error_msg = str(e)
|
||||
# Classify the error for the DLQ
|
||||
error_lower = error_msg.lower()
|
||||
transient_patterns = ["timeout", "connection", "rate limit", "503", "502", "504",
|
||||
"unavailable", "too many requests", "refused", "reset"]
|
||||
permanent_patterns = ["400", "404", "not found", "invalid", "parse error",
|
||||
"deleted", "permission denied"]
|
||||
failure_class = "unknown"
|
||||
for p in transient_patterns:
|
||||
if p in error_lower:
|
||||
failure_class = "transient"
|
||||
break
|
||||
if failure_class == "unknown":
|
||||
for p in permanent_patterns:
|
||||
if p in error_lower:
|
||||
failure_class = "permanent"
|
||||
break
|
||||
|
||||
failures.append({
|
||||
"timestamp": datetime.now(timezone.utc).isoformat(),
|
||||
"file": rel_path,
|
||||
"error": error_msg,
|
||||
"failure_class": failure_class, # NEW: classification
|
||||
"reported": False, # NEW: not yet reported
|
||||
"retry_count": 0, # NEW: zero retries
|
||||
})
|
||||
print(f" ⚠️ Failure: {rel_path} — {e} [{failure_class}]")
|
||||
|
||||
await redis.aclose()
|
||||
save_state(state)
|
||||
|
||||
# Persist failures to simple DLQ (atomic, last 500)
|
||||
if failures:
|
||||
FAILURES_FILE.parent.mkdir(parents=True, exist_ok=True)
|
||||
existing = []
|
||||
if FAILURES_FILE.exists():
|
||||
with open(FAILURES_FILE) as f:
|
||||
existing = json.load(f)
|
||||
existing.extend(failures)
|
||||
existing = existing[-500:]
|
||||
tmp = FAILURES_FILE.with_suffix(".tmp")
|
||||
with open(tmp, "w") as f:
|
||||
json.dump(existing, f, indent=2)
|
||||
f.flush()
|
||||
os.fsync(f.fileno())
|
||||
os.replace(tmp, FAILURES_FILE)
|
||||
|
||||
print(f"\n📊 {total} files tracked")
|
||||
print(f" New: {len(new_files)} | Modified: {len(modified_files)} | Unchanged: {skipped}")
|
||||
print(f" Enqueued: {enqueued} | Failures: {failed}")
|
||||
if failures:
|
||||
print(f" 📋 Failures persisted to: {FAILURES_FILE}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
|
|
@ -0,0 +1,177 @@
|
|||
# Setup Guide
|
||||
|
||||
> Step-by-step installation of the Memory OS stack. Assumes Hermes Agent is already installed and configured.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Hermes Agent 0.14.0+ (tested on 0.15.2)
|
||||
- Python 3.11+
|
||||
- Docker 24.0+
|
||||
- OpenRouter API key (for embeddings and LLM extraction)
|
||||
- 16 GB RAM recommended (8 GB minimum)
|
||||
|
||||
## Installation
|
||||
|
||||
### 1. Icarus Plugin (bundled)
|
||||
|
||||
```bash
|
||||
# Copy the bundled Icarus fork into the Hermes plugins directory
|
||||
cp -r icarus/ ~/.hermes/plugins/icarus/
|
||||
```
|
||||
|
||||
### 2. Enable Icarus in Hermes Config
|
||||
|
||||
Icarus must be registered as an enabled plugin. Edit `~/.hermes/config.yaml`:
|
||||
|
||||
```yaml
|
||||
enabled:
|
||||
- hermes-achievements # optional
|
||||
- icarus # required — activates fabric tools + context injection hooks
|
||||
```
|
||||
|
||||
Then restart the gateway:
|
||||
|
||||
```bash
|
||||
hermes gateway restart
|
||||
```
|
||||
|
||||
Verify the plugin loaded:
|
||||
|
||||
```bash
|
||||
hermes status
|
||||
# → Should show: icarus v0.3.0 (16 tools, 4 hooks)
|
||||
```
|
||||
|
||||
### 3. Docker Infrastructure
|
||||
|
||||
```bash
|
||||
# Copy docker-compose.yml from this repository
|
||||
cp docker/docker-compose.yml ~/memory-os/
|
||||
cd ~/memory-os
|
||||
|
||||
# Create .env with required variables
|
||||
cat > .env << EOF
|
||||
OPENROUTER_API_KEY=sk-or-...
|
||||
REDIS_PASSWORD=$(openssl rand -hex 16)
|
||||
EMBEDDING_DIMS=4096
|
||||
COLLECTION_NAME=knowledge_base
|
||||
EOF
|
||||
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
Verify:
|
||||
```bash
|
||||
curl -s http://localhost:6333/healthz # → {"title":"ok","version":"1.17.1"}
|
||||
redis-cli -a "$REDIS_PASSWORD" ping # → PONG
|
||||
```
|
||||
|
||||
### 4. Environment Variables
|
||||
|
||||
Add to your Hermes profile `.env` (e.g. `~/.hermes/.env`):
|
||||
|
||||
```bash
|
||||
# Required
|
||||
FABRIC_DIR=/home/your-user/vault/fabric
|
||||
OPENROUTER_API_KEY=sk-or-...
|
||||
|
||||
# Strongly recommended
|
||||
ICARUS_EXTRACTION_MAX_TOKENS=4096
|
||||
ICARUS_EXTRACTION_MODEL=deepseek/deepseek-v4-flash
|
||||
EMBEDDING_DIMS=4096
|
||||
|
||||
# Optional
|
||||
ICARUS_OBSIDIAN=1
|
||||
ICARUS_RESULT_MAX_CHARS=500
|
||||
ICARUS_TASK_MAX_CHARS=300
|
||||
```
|
||||
|
||||
**⚠️ Use absolute paths.** The Hermes gateway runs as a systemd service — `~` is not expanded. Always use `/home/your-user/...`.
|
||||
|
||||
### 5. Core File Modifications
|
||||
|
||||
Apply the changes documented in [modifications/soul-rulebook.md](../modifications/soul-rulebook.md):
|
||||
|
||||
- Add Ground Truth level 2 (injected memory) to `SOUL.md`
|
||||
- Add memory architecture documentation to `rulebook.md`
|
||||
- Add context injection convention to `SOUL.md`
|
||||
|
||||
These modifications ensure the agent trusts its injected memory as authoritative.
|
||||
|
||||
### 6. Wiki Setup
|
||||
|
||||
```bash
|
||||
mkdir -p $VAULT_PATH/wiki/{raw,concepts,entities,comparisons,_meta,_archive}
|
||||
# Copy SCHEMA.md template, create initial index.md and log.md
|
||||
```
|
||||
|
||||
The wiki starts empty. Add source documents to `raw/` and the wiki-agent cronjob will begin extracting structured pages.
|
||||
|
||||
### 7. Cronjobs
|
||||
|
||||
Add to crontab (`crontab -e`):
|
||||
|
||||
```cron
|
||||
# Wiki ingestion — keeps Qdrant in sync
|
||||
0 * * * * /usr/bin/python3 /path/to/scripts/wiki_continuous_ingest.py
|
||||
|
||||
# Qdrant maintenance
|
||||
0 3 * * 0 /usr/bin/python3 /path/to/scripts/decay_scanner.py
|
||||
|
||||
# Dead letter queue monitoring
|
||||
0 */6 * * * /usr/bin/python3 /path/to/scripts/dlq_manager.py
|
||||
|
||||
# Semantic dedup (first Sunday of month)
|
||||
0 3 * * 0 [ $(date +\%d) -le 7 ] && /usr/bin/python3 /path/to/scripts/semantic_dedup.py
|
||||
```
|
||||
|
||||
### 8. Gateway Restart
|
||||
|
||||
```bash
|
||||
hermes gateway restart
|
||||
```
|
||||
|
||||
Changes to `.env`, `SOUL.md`, `rulebook.md`, and Icarus plugin code only take effect after restart.
|
||||
|
||||
### 9. Verify
|
||||
|
||||
Inside Hermes chat:
|
||||
|
||||
```
|
||||
/plugins
|
||||
# → Should show: icarus v0.3.0 (16 tools, 4 hooks)
|
||||
|
||||
fabric_brief()
|
||||
# → Should show recent fabric entries (initially empty)
|
||||
|
||||
qdrant_search("test query")
|
||||
# → Should return results from knowledge_base (if wiki has content)
|
||||
|
||||
fact_store(action='probe', entity='test')
|
||||
# → Should return empty (no facts stored yet)
|
||||
```
|
||||
|
||||
## What to expect
|
||||
|
||||
**Day 1:** Infrastructure running. Fabric entries begin accumulating at session end. Qdrant indexing starts as wiki files are added.
|
||||
|
||||
**Week 1:** Context injection active. Agent references past decisions automatically. Wiki pipeline producing curated pages from raw documents.
|
||||
|
||||
**Month 1:** Decay scanner has aged content to evaluate. Structured facts accumulating with trust scores.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Qdrant collection shows 0 points
|
||||
Check: `EMBEDDING_DIMS=4096` matches collection schema. Mismatch → vectors rejected silently.
|
||||
|
||||
### Fabric entries are truncated
|
||||
Check: `ICARUS_EXTRACTION_MAX_TOKENS=4096` in `.env` AND gateway was restarted after setting it.
|
||||
|
||||
### Memory tool reports "Icarus write conflict"
|
||||
Icarus is writing to MEMORY.md instead of CREATIVE.md. Verify Icarus fork is installed (not upstream esaradev version).
|
||||
|
||||
### Context injection not working
|
||||
Check: OpenRouter API key is set, `context_enhancer.py` can import, gateway restarted after `hooks.py` edits.
|
||||
|
||||
### Decay scanner produces "0 archived" every week
|
||||
Most likely: point payloads missing `last_accessed_at` or `importance_score` metadata. Run backfill before enabling decay.
|
||||
|
|
@ -0,0 +1,125 @@
|
|||
# Wiki Schema Template
|
||||
|
||||
This document defines the structure for wiki pages in the Memory OS knowledge base.
|
||||
Each page under `wiki/{concepts,entities,comparisons}/` should follow this structure.
|
||||
|
||||
## Frontmatter
|
||||
|
||||
```yaml
|
||||
---
|
||||
title: "Page Title"
|
||||
type: concept # concept | entity | comparison
|
||||
tags: [tag1, tag2]
|
||||
created: YYYY-MM-DD
|
||||
updated: YYYY-MM-DD
|
||||
source: raw/filename.md # link to source document
|
||||
status: seedling # seedling | growing | evergreen
|
||||
aliases: [alt-name-1, alt-name-2]
|
||||
---
|
||||
```
|
||||
|
||||
**Field descriptions:**
|
||||
- `type` — one of: `concept` (abstract pattern/idea), `entity` (concrete tool/project/person), `comparison` (side-by-side analysis)
|
||||
- `status` — maturity indicator: `seedling` (new), `growing` (being refined), `evergreen` (stable reference)
|
||||
- `source` — relative path to the raw source document that generated this page
|
||||
- `aliases` — alternative names for cross-linking and search
|
||||
|
||||
## Body Structure
|
||||
|
||||
### For `concept` pages
|
||||
|
||||
```markdown
|
||||
# Concept Name
|
||||
|
||||
## Summary
|
||||
|
||||
One-paragraph high-level overview of the concept.
|
||||
|
||||
## Description
|
||||
|
||||
Detailed explanation. Include:
|
||||
- What problem this concept solves
|
||||
- How it works at a high level
|
||||
- Key principles or rules
|
||||
|
||||
## Examples
|
||||
|
||||
### Example 1: Descriptive title
|
||||
```
|
||||
code or configuration block
|
||||
```
|
||||
Brief explanation of what the example demonstrates.
|
||||
|
||||
### Example 2: Another example
|
||||
|
||||
## Related
|
||||
|
||||
- [[Related Concept 1]] — relationship description
|
||||
- [[Related Concept 2]] — relationship description
|
||||
```
|
||||
|
||||
### For `entity` pages
|
||||
|
||||
```markdown
|
||||
# Entity Name
|
||||
|
||||
## Summary
|
||||
|
||||
What this thing is — one paragraph.
|
||||
|
||||
## Purpose
|
||||
|
||||
Why this entity exists in the system.
|
||||
|
||||
## Configuration
|
||||
|
||||
```yaml
|
||||
# Example configuration block
|
||||
key: value
|
||||
option: setting
|
||||
```
|
||||
|
||||
## Dependencies
|
||||
|
||||
- Dependency 1: what it provides
|
||||
- Dependency 2: what it provides
|
||||
|
||||
## Usage Notes
|
||||
|
||||
Practical considerations, edge cases, known issues.
|
||||
|
||||
## Related
|
||||
|
||||
- [[Related Concept]] — relationship
|
||||
```
|
||||
|
||||
### For `comparison` pages
|
||||
|
||||
```markdown
|
||||
# Comparison: A vs B
|
||||
|
||||
## Summary
|
||||
|
||||
One-paragraph overview of what is being compared.
|
||||
|
||||
## Comparison Table
|
||||
|
||||
| Aspect | Option A | Option B |
|
||||
|--------|----------|----------|
|
||||
| Strengths | ... | ... |
|
||||
| Weaknesses | ... | ... |
|
||||
| Best for | ... | ... |
|
||||
|
||||
## Decision Factors
|
||||
|
||||
Considerations that favour one option over the other.
|
||||
|
||||
## Recommendation
|
||||
|
||||
Final recommendation with reasoning.
|
||||
|
||||
## Related
|
||||
|
||||
- [[Option A detail page]]
|
||||
- [[Option B detail page]]
|
||||
```
|
||||
|
|
@ -0,0 +1,49 @@
|
|||
# Knowledge Wiki
|
||||
|
||||
> Map of Content — curated knowledge base for the Memory OS agent.
|
||||
|
||||
```yaml
|
||||
---
|
||||
title: "Wiki Index"
|
||||
type: meta
|
||||
last_updated: YYYY-MM-DD
|
||||
total_pages: 0
|
||||
---
|
||||
```
|
||||
|
||||
## Concepts
|
||||
|
||||
Abstract patterns, ideas, and recurring themes.
|
||||
|
||||
- _Add wiki links here as concepts are created_
|
||||
|
||||
## Entities
|
||||
|
||||
Concrete things: tools, models, projects, people.
|
||||
|
||||
- _Add wiki links here as entities are created_
|
||||
|
||||
## Comparisons
|
||||
|
||||
Side-by-side analyses of alternatives.
|
||||
|
||||
- _Add wiki links here as comparisons are created_
|
||||
|
||||
## Recent Additions
|
||||
|
||||
| Date | Title | Type |
|
||||
|------|-------|------|
|
||||
| | | |
|
||||
|
||||
---
|
||||
|
||||
## Raw Sources
|
||||
|
||||
Source documents that feed the wiki pipeline:
|
||||
|
||||
```
|
||||
raw/
|
||||
├── external/ # Third-party articles, docs, specs
|
||||
├── notes/ # Personal notes and observations
|
||||
└── research/ # Research papers and investigations
|
||||
```
|
||||
|
|
@ -0,0 +1,31 @@
|
|||
# Wiki Change Log
|
||||
|
||||
> Track what was added, updated, or removed from the wiki.
|
||||
|
||||
```yaml
|
||||
---
|
||||
title: "Wiki Change Log"
|
||||
type: meta
|
||||
---
|
||||
```
|
||||
|
||||
## YYYY-MM-DD
|
||||
|
||||
### Added
|
||||
|
||||
- `concepts/example-concept.md` — first concept about ...
|
||||
|
||||
### Updated
|
||||
|
||||
- `entities/example-entity.md` — added configuration section
|
||||
|
||||
### Removed
|
||||
|
||||
- _Nothing removed_
|
||||
|
||||
---
|
||||
|
||||
## Structure Notes
|
||||
|
||||
- Each entry links to the wiki page path and a brief description of the change.
|
||||
- The wiki-agent cronjob updates this log automatically during curation.
|
||||
Loading…
Reference in New Issue