docs: rewrite installation guide with real scripts and idempotent modifications (fixes #11)
Replace incomplete/outdated install instructions with accurate, detailed steps grounded in production infrastructure. setup/install.md: - Step 3: run docker compose in-place (not via cp to ~/memory-os/) - Step 5: explicit ~/.hermes/rulebook.md path, create-if-missing, idempotency guard with Memory OS Additions v1 marker - Step 6: document VAULT_PATH, wiki directory structure, link to vault-curator as optional enrichment tool - Step 7: replace fictional crontab with real maintenance scripts and hermes cron create commands; add backfill prerequisite note and exempt-prefix env var docs setup/rulebook.md (NEW): - Generic template with 3 blocks: Memory Architecture, Memory OS infrastructure, Mandatory Verifications - Idempotency markers on every block modifications/soul-rulebook.md: - SOUL.md-only (rulebook sections moved to setup/rulebook.md) - Conditional instructions (add level 2 vs full hierarchy) - Conflict resolution table (4 rules) - Context injection convention with acting vs reasoning distinction scripts/: - 8 maintenance scripts updated with sanitized equivalents - decay_scanner.py, semantic_dedup.py: collection_ → DECAY/DEDUP_EXEMPT_PREFIXES env vars - reflection_trigger.py: hardcoded venv path → /usr/bin/env python3 - All shebangs normalized, all internal references removed
This commit is contained in:
ClaudioDrews
parent
b33b8d4196
commit
4e1f543a1d
|
|
@ -1,97 +1,96 @@
|
|||
# Modifications to Hermes Core
|
||||
|
||||
Memory OS requires changes to core Hermes files that govern agent behavior. These modifications ensure the agent trusts its injected memory as authoritative rather than re-discovering known facts.
|
||||
Memory OS requires additions to `SOUL.md` — the Hermes agent's identity file
|
||||
at `~/.hermes/SOUL.md`. These additions ensure injected memory is treated as
|
||||
prior knowledge rather than being ignored or re-discovered every session.
|
||||
|
||||
## Before you begin
|
||||
|
||||
Check your `SOUL.md`:
|
||||
|
||||
- **If it already has a `## Ground Truth` section:** add level 2 (injected
|
||||
memory) between terminal output and official documentation.
|
||||
- **If it does not have a Ground Truth section:** add the full hierarchy
|
||||
below, placing it after the agent identity section.
|
||||
|
||||
Each block includes a `<!-- Memory OS additions — do not duplicate -->`
|
||||
marker. Before applying, check whether this marker already exists in your
|
||||
SOUL.md — if it does, skip that block.
|
||||
|
||||
---
|
||||
|
||||
## SOUL.md — Ground Truth hierarchy
|
||||
|
||||
Add a new level 2 to the Ground Truth hierarchy in `SOUL.md`:
|
||||
If your SOUL.md already has a Ground Truth section, insert only the new
|
||||
level 2 (the injected memory line) between terminal output and official
|
||||
documentation. If the section doesn't exist, add the full block:
|
||||
|
||||
```markdown
|
||||
<!-- Memory OS additions — do not duplicate -->
|
||||
|
||||
## Ground Truth
|
||||
|
||||
Authoritative sources, in priority order:
|
||||
|
||||
1. **Terminal output** — stdout, stderr, exit codes. Never reinterpret.
|
||||
2. **Injected memory** — qdrant, fabric, sessions, facts. Ground truth for documented
|
||||
knowledge. When injected memory contradicts other sources, injected memory wins
|
||||
because it represents verified, persisted knowledge from prior sessions.
|
||||
3. **Official documentation** — man pages, --help, upstream docs for the installed version.
|
||||
4. **Training knowledge** — reference only. Always verify against sources 1-3 before acting.
|
||||
1. **Terminal output** — stdout, stderr, exit codes. Ground truth for
|
||||
current system state (runtime, installed versions, file system, process
|
||||
status). Never reinterpret.
|
||||
2. **Injected memory — [qdrant], [fabric], [sessions], [facts]** — Ground
|
||||
truth for documented knowledge and prior decisions. These are delivered
|
||||
by the `pre_llm_call` hook before every turn and represent what has
|
||||
already been built, decided, or documented. When injected memory
|
||||
contradicts your assumptions or training knowledge, injected memory wins.
|
||||
Never treat a question as novel when the answer is already in your prompt.
|
||||
3. **Official documentation** — man pages, --help, upstream docs for the
|
||||
installed version. Authoritative for APIs, configuration options, and
|
||||
breaking changes.
|
||||
4. **Training knowledge** — reference only. Always verify against sources
|
||||
1-3 before acting.
|
||||
|
||||
When sources conflict: terminal output wins for system state. Injected
|
||||
memory wins for documented knowledge.
|
||||
```
|
||||
|
||||
**Why this matters:** Without level 2, the agent treats facts already persisted in Qdrant/fabric/sessions as less authoritative than documentation, causing it to re-discover known information. An agent that has Tailscale configuration in `fact_store` should not spend time re-verifying it against `man tailscale`.
|
||||
### Conflict resolution rules
|
||||
|
||||
When memory sources disagree, the agent resolves conflicts as follows:
|
||||
|
||||
| Sources conflict | Resolution |
|
||||
|---|---|
|
||||
| Terminal vs Injected memory | Terminal wins for system state. Injected wins for documented knowledge. |
|
||||
| Injected memory vs Assumptions | Injected memory wins. Never treat a question as novel when the answer is already in your prompt. |
|
||||
| Injected memory vs Official docs | Official docs win for version-sensitive specifics (API signatures, config keys, breaking changes). Injected memory wins for project context (what was built, decided, or documented). |
|
||||
| Training knowledge vs anything | Training knowledge always loses. Verify against sources 1-3 before acting. |
|
||||
|
||||
**Why this matters:** Without level 2, the agent treats facts already
|
||||
persisted as less authoritative than documentation, causing it to
|
||||
re-discover known information — burning tokens, context, and time.
|
||||
|
||||
---
|
||||
|
||||
## SOUL.md — Context injection convention
|
||||
|
||||
Add source labeling conventions:
|
||||
Add a section explaining how injected context is labeled and how the
|
||||
agent should treat it:
|
||||
|
||||
```markdown
|
||||
<!-- Memory OS additions — do not duplicate -->
|
||||
|
||||
## Context injection convention
|
||||
|
||||
When context is injected into the system prompt, it is labeled by source:
|
||||
- [fabric] — from Icarus fabric recall
|
||||
- [qdrant] — from Qdrant semantic search
|
||||
- [qdrant] — from Qdrant semantic search
|
||||
- [sessions] — from session history FTS5
|
||||
- [facts] — from holographic fact store
|
||||
|
||||
Injected memory takes priority level 2 in Ground Truth. This means:
|
||||
"You already know this. Don't re-discover it. Use it."
|
||||
Injected memory takes priority level 2 in Ground Truth. This means: you
|
||||
already know this. Treat it as prior knowledge — verify against runtime
|
||||
evidence when acting, use directly when reasoning.
|
||||
```
|
||||
|
||||
## SOUL.md — Agent identity
|
||||
|
||||
Add clear identity boundaries:
|
||||
|
||||
```markdown
|
||||
## You are not
|
||||
|
||||
You are not a search engine. You are not a chatbot. You are not here to produce
|
||||
plausible-sounding output. You are an agent that executes real work in real
|
||||
environments, where errors have real costs. Treat every action accordingly.
|
||||
```
|
||||
|
||||
## rulebook.md — Mandatory verifications
|
||||
|
||||
Add to the rulebook:
|
||||
|
||||
```markdown
|
||||
## Mandatory Verifications
|
||||
|
||||
Before reporting a fact as true, verify:
|
||||
1. **Runtime evidence** — terminal output, file existence, process status
|
||||
2. **Injected memory** — qdrant_search, fact_store probe, fabric_recall
|
||||
3. **Documentation** — man pages, official docs for installed version
|
||||
4. **Training knowledge** — never cite without verifying against 1-3
|
||||
```
|
||||
|
||||
## rulebook.md — Memory architecture
|
||||
|
||||
Add a section documenting the 6-layer architecture so the agent knows where to find information:
|
||||
|
||||
```markdown
|
||||
## Memory Architecture
|
||||
|
||||
The agent has 6 layers of persistent memory:
|
||||
|
||||
| Layer | What it stores | How to access |
|
||||
|-------|---------------|---------------|
|
||||
| 1. Workspace | MEMORY.md, USER.md, CREATIVE.md | Always in system prompt |
|
||||
| 2. Sessions | state.db (FTS5) | session_search |
|
||||
| 3. Facts | memory_store.db (HRR) | fact_store |
|
||||
| 4. Fabric | $FABRIC_DIR (markdown) | fabric_recall, fabric_write |
|
||||
| 5. Qdrant | knowledge_base (4096d) | qdrant_search, auto-injection |
|
||||
| 6. Wiki | $VAULT_PATH/wiki/ | qdrant_search → knowledge_base |
|
||||
```
|
||||
|
||||
## Impact
|
||||
|
||||
Without these modifications:
|
||||
- Qdrant/fabric/session/fact injection still works technically
|
||||
- But the agent doesn't trust injected memory as authoritative
|
||||
- Result: agent re-discovers known facts, wastes tokens, makes redundant decisions
|
||||
|
||||
With these modifications:
|
||||
- Agent treats injected memory as ground truth (level 2)
|
||||
- Reduces redundant discovery work
|
||||
- Agent can reference prior decisions without re-litigating them
|
||||
- Cross-session continuity is real, not aspirational
|
||||
**Why this matters:** Without explicit labeling conventions, the agent may
|
||||
treat injected memory blocks as user context rather than authoritative
|
||||
prior knowledge. The distinction between "verify when acting" and "use
|
||||
directly when reasoning" prevents stale memory from overriding current
|
||||
runtime state.
|
||||
|
|
|
|||
|
|
@ -45,12 +45,12 @@ from urllib.request import Request, urlopen
|
|||
from urllib.error import URLError
|
||||
|
||||
# ─── Config ──────────────────────────────────────────────────────────────────
|
||||
QDRANT_URL = os.environ.get("QDRANT_URL", "http://localhost:6333")
|
||||
COLLECTION = os.environ.get("QDRANT_COLLECTION", os.environ.get("COLLECTION_NAME", "knowledge_base"))
|
||||
QDRANT_URL = "http://localhost:6333"
|
||||
COLLECTION = "knowledge_base"
|
||||
BATCH_SIZE = 200
|
||||
SCROLL_LIMIT = 200
|
||||
LOG_FILE = Path(os.environ.get("HERMES_LOGS_DIR", str(Path.home() / ".hermes" / "logs"))) / "decay_scanner.log"
|
||||
VAULT_ROOT = Path(os.environ.get("VAULT_PATH", "."))
|
||||
LOG_FILE = Path.home() / ".hermes/logs/decay_scanner.log"
|
||||
VAULT_ROOT = Path.home() / "Vault"
|
||||
|
||||
# ─── Heuristics ──────────────────────────────────────────────────────────────
|
||||
CONFIDENCE_BY_SOURCE = {
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
#!/usr/bin/env python3
|
||||
"""
|
||||
Bulk ingest script — populates the Qdrant knowledge_base with all wiki content.
|
||||
Phase A: one-shot of existing files.
|
||||
Bulk ingest script — popula a knowledge_base Qdrant com todo conteúdo da wiki.
|
||||
Fase A: one-shot dos arquivos existentes.
|
||||
"""
|
||||
import os
|
||||
import re
|
||||
|
|
@ -18,9 +18,9 @@ import asyncio
|
|||
|
||||
# ─── Config ────────────────────────────────────────────────────────────────
|
||||
OPENROUTER_KEY = os.environ.get("OPENROUTER_API_KEY")
|
||||
QDRANT_URL = os.environ.get("QDRANT_URL", "http://localhost:6333")
|
||||
COLLECTION = os.environ.get("QDRANT_COLLECTION", os.environ.get("COLLECTION_NAME", "knowledge_base"))
|
||||
WIKI_ROOT = Path(os.environ.get("WIKI_ROOT", "."))
|
||||
QDRANT_URL = "http://localhost:6333"
|
||||
COLLECTION = "knowledge_base"
|
||||
WIKI_ROOT = Path.home() / "Vault" / "wiki"
|
||||
EMBEDDING_MODEL = "qwen/qwen3-embedding-8b"
|
||||
EMBEDDING_DIMS = 4096
|
||||
MAX_TEXT_LEN = 8000 # truncate text for embedding (model context limit)
|
||||
|
|
@ -28,20 +28,20 @@ BATCH_SIZE = 8 # parallel embedding requests
|
|||
RATE_LIMIT_SLEEP = 0.5 # seconds between batches
|
||||
|
||||
if not OPENROUTER_KEY:
|
||||
print("❌ OPENROUTER_API_KEY not found in environment")
|
||||
print("❌ OPENROUTER_API_KEY não encontrada no ambiente")
|
||||
sys.exit(1)
|
||||
|
||||
print(f"📁 Wiki root: {WIKI_ROOT}")
|
||||
print(f"🎯 Collection: {COLLECTION}")
|
||||
print(f"🔑 OpenRouter: configured")
|
||||
print(f"🎯 Coleção: {COLLECTION}")
|
||||
print(f"🔑 OpenRouter: {OPENROUTER_KEY[:20]}...")
|
||||
|
||||
# ─── Find all .md files ───────────────────────────────────────────────────
|
||||
# ─── Encontrar todos os .md ────────────────────────────────────────────────
|
||||
md_files = sorted(WIKI_ROOT.rglob("*.md"))
|
||||
print(f"📄 .md files found: {len(md_files)}")
|
||||
print(f"📄 Arquivos .md encontrados: {len(md_files)}")
|
||||
|
||||
# ─── Helpers ──────────────────────────────────────────────────────────────
|
||||
def parse_frontmatter(text: str) -> tuple[dict, str]:
|
||||
"""Extract YAML frontmatter and return (metadata, body)."""
|
||||
"""Extrai YAML frontmatter e retorna (metadata, body)."""
|
||||
if text.startswith("---"):
|
||||
parts = text.split("---", 2)
|
||||
if len(parts) >= 3:
|
||||
|
|
@ -55,7 +55,7 @@ def parse_frontmatter(text: str) -> tuple[dict, str]:
|
|||
return {}, text
|
||||
|
||||
def get_source_tag(path: Path) -> str:
|
||||
"""Derive source tag from path relative to wiki root."""
|
||||
"""Deriva source tag do path relativo à wiki."""
|
||||
rel = path.relative_to(WIKI_ROOT)
|
||||
parts = rel.parts
|
||||
if len(parts) > 1:
|
||||
|
|
@ -63,14 +63,14 @@ def get_source_tag(path: Path) -> str:
|
|||
return "wiki-root"
|
||||
|
||||
def get_tags_from_frontmatter(meta: dict) -> list[str]:
|
||||
"""Extract tags from frontmatter."""
|
||||
"""Extrai tags do frontmatter."""
|
||||
tags = meta.get("tags", [])
|
||||
if isinstance(tags, str):
|
||||
tags = [t.strip() for t in tags.split(",")]
|
||||
return tags if isinstance(tags, list) else []
|
||||
|
||||
async def get_embedding(session: aiohttp.ClientSession, text: str) -> list[float] | None:
|
||||
"""Generate embedding via OpenRouter."""
|
||||
"""Gera embedding via OpenRouter."""
|
||||
payload = {
|
||||
"model": EMBEDDING_MODEL,
|
||||
"input": text[:MAX_TEXT_LEN],
|
||||
|
|
@ -97,7 +97,7 @@ async def get_embedding(session: aiohttp.ClientSession, text: str) -> list[float
|
|||
return None
|
||||
|
||||
async def upsert_to_qdrant(session: aiohttp.ClientSession, points: list[dict]) -> bool:
|
||||
"""Upsert batch of points into Qdrant."""
|
||||
"""Upsert batch de pontos no Qdrant."""
|
||||
try:
|
||||
async with session.put(
|
||||
f"{QDRANT_URL}/collections/{COLLECTION}/points",
|
||||
|
|
@ -114,7 +114,7 @@ async def upsert_to_qdrant(session: aiohttp.ClientSession, points: list[dict]) -
|
|||
print(f"⚠️ Qdrant error: {e}")
|
||||
return False
|
||||
|
||||
# ─── Main processing ──────────────────────────────────────────────────────
|
||||
# ─── Processamento principal ──────────────────────────────────────────────
|
||||
async def main():
|
||||
stats = Counter({"ok": 0, "fail": 0, "skip": 0, "empty": 0})
|
||||
errors = []
|
||||
|
|
@ -123,13 +123,13 @@ async def main():
|
|||
|
||||
connector = aiohttp.TCPConnector(limit=20)
|
||||
async with aiohttp.ClientSession(connector=connector) as session:
|
||||
# Check collection
|
||||
# Verificar coleção
|
||||
async with session.get(f"{QDRANT_URL}/collections/{COLLECTION}") as r:
|
||||
if r.status != 200:
|
||||
print(f"❌ Collection {COLLECTION} does not exist!")
|
||||
print(f"❌ Coleção {COLLECTION} não existe!")
|
||||
sys.exit(1)
|
||||
|
||||
print("\n🚀 Starting ingestion in batches...\n")
|
||||
print("\n🚀 Iniciando ingestão em batches...\n")
|
||||
|
||||
batch = []
|
||||
for idx, path in enumerate(md_files, 1):
|
||||
|
|
@ -141,15 +141,15 @@ async def main():
|
|||
meta, body = parse_frontmatter(text)
|
||||
source = get_source_tag(path)
|
||||
tags = get_tags_from_frontmatter(meta)
|
||||
# Additional tag from folder
|
||||
# Tag adicional da pasta
|
||||
folder_tag = source.replace("wiki-", "")
|
||||
if folder_tag not in tags:
|
||||
tags.append(folder_tag)
|
||||
|
||||
# Title from frontmatter or filename
|
||||
# Título do frontmatter ou filename
|
||||
title = meta.get("title", path.stem)
|
||||
|
||||
# Text for embedding: title + body (without frontmatter)
|
||||
# Texto para embedding: título + body (sem frontmatter)
|
||||
embed_text = f"{title}\n\n{body}"[:MAX_TEXT_LEN]
|
||||
|
||||
batch.append({
|
||||
|
|
@ -163,11 +163,11 @@ async def main():
|
|||
})
|
||||
|
||||
if len(batch) >= BATCH_SIZE or idx == total:
|
||||
# Generate embeddings in parallel
|
||||
# Gerar embeddings em paralelo
|
||||
embed_tasks = [get_embedding(session, b["embed_text"]) for b in batch]
|
||||
vectors = await asyncio.gather(*embed_tasks)
|
||||
|
||||
# Prepare Qdrant points
|
||||
# Preparar pontos Qdrant
|
||||
points = []
|
||||
for b, vec in zip(batch, vectors):
|
||||
if vec is None:
|
||||
|
|
@ -175,7 +175,7 @@ async def main():
|
|||
errors.append(f"Embedding failed: {b['path']}")
|
||||
continue
|
||||
|
||||
# Heuristic importance_score based on path/name
|
||||
# Heurística de importance_score baseada no path/nome
|
||||
importance_score = 0.5
|
||||
path_str_lower = b["path"].lower()
|
||||
if any(k in path_str_lower for k in ["architecture", "core", "important"]):
|
||||
|
|
@ -200,12 +200,12 @@ async def main():
|
|||
"file_path": b["path"],
|
||||
"title": b["title"],
|
||||
"word_count": len(b["embed_text"].split()),
|
||||
# ── Lineage fields (Phase 1)
|
||||
# ── Lineage fields (Fase 1)
|
||||
"lineage_id": None,
|
||||
"generation_model": None,
|
||||
"generation_context_hash": None,
|
||||
"retrieved_chunk_ids": None,
|
||||
# ── Decay fields (Phase 2)
|
||||
# ── Decay fields (Fase 2)
|
||||
"decay_score": 1.0,
|
||||
"last_accessed_at": now_iso,
|
||||
"importance_score": importance_score,
|
||||
|
|
@ -229,37 +229,37 @@ async def main():
|
|||
processed += len(batch)
|
||||
batch = []
|
||||
|
||||
# Progress
|
||||
# Progresso
|
||||
pct = (processed / total) * 100
|
||||
print(f" [{processed}/{total}] {pct:.1f}% | ✅ {stats['ok']} | ⚠️ {stats['fail']} | ⏭️ {stats['skip']} | 🈳 {stats['empty']}")
|
||||
|
||||
# Rate limit breathing
|
||||
await asyncio.sleep(RATE_LIMIT_SLEEP)
|
||||
|
||||
# ─── Final report ───────────────────────────────────────────────────
|
||||
# ─── Relatório final ───────────────────────────────────────────────────
|
||||
print("\n" + "=" * 60)
|
||||
print("📊 INGESTION REPORT")
|
||||
print("📊 RELATÓRIO DE INGESTÃO")
|
||||
print("=" * 60)
|
||||
print(f" Total files: {total}")
|
||||
print(f" Ingested (ok): {stats['ok']}")
|
||||
print(f" Failures: {stats['fail']}")
|
||||
print(f" Empty: {stats['empty']}")
|
||||
print(f" Success rate: {(stats['ok']/max(total-stats['empty'],1)*100):.1f}%")
|
||||
print(f"\n ⏱️ Finished: {datetime.now(timezone.utc).isoformat()}")
|
||||
print(f" Total arquivos: {total}")
|
||||
print(f" Ingestados (ok): {stats['ok']}")
|
||||
print(f" Falhas: {stats['fail']}")
|
||||
print(f" Vazios: {stats['empty']}")
|
||||
print(f" Taxa de sucesso: {(stats['ok']/max(total-stats['empty'],1)*100):.1f}%")
|
||||
print(f"\n ⏱️ Finalizado: {datetime.now(timezone.utc).isoformat()}")
|
||||
|
||||
if errors:
|
||||
print(f"\n ⚠️ First errors ({min(10, len(errors))} of {len(errors)}):")
|
||||
print(f"\n ⚠️ Primeiros erros ({min(10, len(errors))} de {len(errors)}):")
|
||||
for e in errors[:10]:
|
||||
print(f" - {e}")
|
||||
|
||||
# Verify final count
|
||||
# Verificar count final
|
||||
async with aiohttp.ClientSession() as s:
|
||||
async with s.get(f"{QDRANT_URL}/collections/{COLLECTION}") as r:
|
||||
data = await r.json()
|
||||
final_count = data.get("result", {}).get("points_count", "?")
|
||||
print(f"\n 📦 Points in collection: {final_count}")
|
||||
print(f"\n 📦 Pontos na coleção: {final_count}")
|
||||
|
||||
print("\n✅ Bulk ingest complete.")
|
||||
print("\n✅ Bulk ingest completo.")
|
||||
return stats
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
|
|
|||
|
|
@ -1,20 +1,20 @@
|
|||
#!/usr/bin/env python3
|
||||
"""
|
||||
decay_scanner.py
|
||||
Selective archiving script for low-importance AI-generated chunks.
|
||||
Runs via weekly cron (0 3 * * 0).
|
||||
Script de arquivamento seletivo de chunks IA-generated com baixa importância.
|
||||
Roda via cron semanal (0 3 * * 0).
|
||||
|
||||
Rules:
|
||||
- source_type in ["human", "procedural"] → exempt (never archive)
|
||||
Regras:
|
||||
- source_type in ["human", "procedural"] → exempt (nunca arquiva)
|
||||
- importance_score >= 0.7 → exempt
|
||||
- archived == True → skip (already archived)
|
||||
- half_life: 90d if importance_score >= 0.3, else 30d
|
||||
- archived == True → skip (já arquivado)
|
||||
- half_life: 90d se importance_score >= 0.3, senão 30d
|
||||
- decay_score < 0.1:
|
||||
- If confidence_score >= 0.7 → alert (report, don't archive)
|
||||
- Otherwise → archive (archived = True)
|
||||
- gabi_* collections are completely ignored
|
||||
- Se confidence_score >= 0.7 → alerta (reporta, não arquiva)
|
||||
- Senão → archive (archived = True)
|
||||
- Coleções com prefixo em DECAY_EXEMPT_PREFIXES (csv) são ignoradas
|
||||
|
||||
Usage:
|
||||
Uso:
|
||||
python3 decay_scanner.py [--collection knowledge_base_hybrid] [--dry-run]
|
||||
"""
|
||||
|
||||
|
|
@ -30,8 +30,8 @@ from pathlib import Path
|
|||
# ─── Config ────────────────────────────────────────────────────────────────
|
||||
QDRANT_URL = os.environ.get("QDRANT_URL", "http://localhost:6333")
|
||||
COLLECTION = os.environ.get("QDRANT_COLLECTION", "knowledge_base")
|
||||
SCROLL_LIMIT = 100 # Qdrant pagination
|
||||
LOG_DIR = Path(os.environ.get("HERMES_LOGS_DIR", str(Path.home() / ".hermes" / "logs")))
|
||||
SCROLL_LIMIT = 100 # paginação Qdrant
|
||||
LOG_DIR = Path.home() / ".hermes" / "logs"
|
||||
LOG_FILE = LOG_DIR / "decay_scanner.log"
|
||||
|
||||
# ─── Helpers ──────────────────────────────────────────────────────────────
|
||||
|
|
@ -42,35 +42,35 @@ def now_iso() -> str:
|
|||
|
||||
def calculate_decay_score(last_accessed_at: str, importance_score: float) -> float:
|
||||
"""
|
||||
Calculate exponential decay: score = exp(-ln(2) * age_days / half_life).
|
||||
More important chunks persist longer (larger half-lives).
|
||||
Calcula decay exponencial: score = exp(-ln(2) * age_days / half_life).
|
||||
Chunks mais importantes persistem mais (half-lives maiores).
|
||||
"""
|
||||
try:
|
||||
last = datetime.fromisoformat(last_accessed_at.replace("Z", "+00:00"))
|
||||
except (ValueError, TypeError):
|
||||
# If timestamp is invalid, assume now (hasn't decayed yet)
|
||||
# Se timestamp inválido, assumir now (não decaiu ainda)
|
||||
return 1.0
|
||||
|
||||
now = datetime.now(timezone.utc)
|
||||
age_days = max(0, (now - last).total_seconds() / 86400)
|
||||
|
||||
# Fix: LARGER half-life for more important chunks
|
||||
# Correção: half-life MAIOR para chunks mais importantes
|
||||
if importance_score >= 0.3:
|
||||
half_life = 90 # medium/high chunks → 90 days
|
||||
half_life = 90 # chunks médio/alto → 90 dias
|
||||
else:
|
||||
half_life = 30 # low chunks → 30 days
|
||||
half_life = 30 # chunks baixo → 30 dias
|
||||
|
||||
decay_score = math.exp(-math.log(2) * age_days / half_life)
|
||||
return decay_score
|
||||
|
||||
|
||||
def ensure_log_dir():
|
||||
"""Create log directory if it doesn't exist."""
|
||||
"""Cria diretório de logs se não existir."""
|
||||
LOG_DIR.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
|
||||
def log_message(msg: str):
|
||||
"""Log to stdout and append to log file."""
|
||||
"""Loga para stdout e append no arquivo."""
|
||||
ts = now_iso()
|
||||
line = f"[{ts}] {msg}"
|
||||
print(line)
|
||||
|
|
@ -83,8 +83,8 @@ def log_message(msg: str):
|
|||
|
||||
def scroll_chunks(collection: str, limit: int = SCROLL_LIMIT):
|
||||
"""
|
||||
Generator that iterates over all points in the collection via scroll.
|
||||
Avoids loading the entire collection into memory.
|
||||
Generator que itera sobre todos os pontos da coleção via scroll.
|
||||
Evita carregar a coleção inteira em memória.
|
||||
"""
|
||||
offset = None
|
||||
total_scanned = 0
|
||||
|
|
@ -122,16 +122,16 @@ def scroll_chunks(collection: str, limit: int = SCROLL_LIMIT):
|
|||
break
|
||||
|
||||
except Exception as e:
|
||||
log_message(f"❌ Qdrant scroll error: {e}")
|
||||
log_message(f"❌ Erro no scroll Qdrant: {e}")
|
||||
break
|
||||
|
||||
log_message(f"📊 Total chunks scanned: {total_scanned}")
|
||||
log_message(f"📊 Total de chunks escaneados: {total_scanned}")
|
||||
|
||||
|
||||
def update_point_archived(point_id: str, collection: str, decay_score: float, dry_run: bool = False):
|
||||
"""Update point payload: archived=True + calculated decay_score."""
|
||||
"""Atualiza payload do ponto: archived=True + decay_score calculado."""
|
||||
if dry_run:
|
||||
log_message(f" [DRY-RUN] Would archive point {point_id} (decay_score={decay_score:.4f})")
|
||||
log_message(f" [DRY-RUN] Arquivaria ponto {point_id} (decay_score={decay_score:.4f})")
|
||||
return True
|
||||
|
||||
try:
|
||||
|
|
@ -150,29 +150,32 @@ def update_point_archived(point_id: str, collection: str, decay_score: float, dr
|
|||
resp.raise_for_status()
|
||||
return True
|
||||
except Exception as e:
|
||||
log_message(f" ❌ Failed to archive point {point_id}: {e}")
|
||||
log_message(f" ❌ Falha ao arquivar ponto {point_id}: {e}")
|
||||
return False
|
||||
|
||||
|
||||
# ─── Main ─────────────────────────────────────────────────────────────────
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Decay Scanner — Selective chunk archiving")
|
||||
parser.add_argument("--collection", default=COLLECTION, help="Qdrant collection name")
|
||||
parser.add_argument("--dry-run", action="store_true", help="Simulation — does not modify anything")
|
||||
parser.add_argument("--threshold", type=float, default=0.1, help="Decay threshold for archiving")
|
||||
parser = argparse.ArgumentParser(description="Decay Scanner — Arquivamento seletivo de chunks")
|
||||
parser.add_argument("--collection", default=COLLECTION, help="Nome da coleção Qdrant")
|
||||
parser.add_argument("--dry-run", action="store_true", help="Simulação — não modifica nada")
|
||||
parser.add_argument("--threshold", type=float, default=0.1, help="Threshold de decay para arquivamento")
|
||||
args = parser.parse_args()
|
||||
|
||||
collection = args.collection
|
||||
|
||||
# Ignore gabi_* collections
|
||||
if collection.startswith("gabi_"):
|
||||
log_message(f"⏭️ Collection '{collection}' is exempt (gabi_*). Exiting.")
|
||||
return
|
||||
# Ignorar coleções com prefixos exempt (via DECAY_EXEMPT_PREFIXES env var)
|
||||
exempt_prefixes = os.environ.get("DECAY_EXEMPT_PREFIXES", "").split(",")
|
||||
exempt_prefixes = [p.strip() for p in exempt_prefixes if p.strip()]
|
||||
for prefix in exempt_prefixes:
|
||||
if collection.startswith(prefix):
|
||||
log_message(f"⏭️ Coleção '{collection}' é exempt (prefixo '{prefix}'). Saindo.")
|
||||
return
|
||||
|
||||
log_message(f"🚀 Starting decay scanner (collection={collection}, threshold={args.threshold}, dry_run={args.dry_run})")
|
||||
log_message(f"🚀 Iniciando decay scanner (collection={collection}, threshold={args.threshold}, dry_run={args.dry_run})")
|
||||
|
||||
# Metrics
|
||||
# Métricas
|
||||
stats = {
|
||||
"scanned": 0,
|
||||
"archived": 0,
|
||||
|
|
@ -184,7 +187,7 @@ def main():
|
|||
"failed": 0,
|
||||
}
|
||||
|
||||
alerts = [] # List of alerts (decay < threshold but confidence >= 0.7)
|
||||
alerts = [] # Lista de alertas (decay < threshold mas confidence >= 0.7)
|
||||
|
||||
for point in scroll_chunks(collection):
|
||||
stats["scanned"] += 1
|
||||
|
|
@ -198,12 +201,12 @@ def main():
|
|||
last_accessed_at = payload.get("last_accessed_at", payload.get("created_at", now_iso()))
|
||||
confidence_score = payload.get("confidence_score", 1.0)
|
||||
|
||||
# Skip: already archived
|
||||
# Skip: já arquivado
|
||||
if archived:
|
||||
stats["skipped_already_archived"] += 1
|
||||
continue
|
||||
|
||||
# Skip: human (exempt)
|
||||
# Skip: humano (exempt)
|
||||
if source_type == "human":
|
||||
stats["skipped_human"] += 1
|
||||
continue
|
||||
|
|
@ -213,17 +216,17 @@ def main():
|
|||
stats["skipped_procedural"] += 1
|
||||
continue
|
||||
|
||||
# Skip: high importance
|
||||
# Skip: alta importância
|
||||
if importance_score >= 0.7:
|
||||
stats["skipped_high_importance"] += 1
|
||||
continue
|
||||
|
||||
# Calculate decay
|
||||
# Calcular decay
|
||||
decay_score = calculate_decay_score(last_accessed_at, importance_score)
|
||||
|
||||
# Check threshold
|
||||
# Verificar threshold
|
||||
if decay_score < args.threshold:
|
||||
# Decay-confidence rule: if confidence is high, alert instead of archiving
|
||||
# Regra decay-confidence: se confidence alto, alerta em vez de arquivar
|
||||
if confidence_score >= 0.7:
|
||||
stats["alerted"] += 1
|
||||
alerts.append({
|
||||
|
|
@ -232,19 +235,19 @@ def main():
|
|||
"confidence_score": round(confidence_score, 2),
|
||||
"importance_score": round(importance_score, 2),
|
||||
"age_days": round((datetime.now(timezone.utc) - datetime.fromisoformat(last_accessed_at.replace("Z", "+00:00"))).total_seconds() / 86400, 1),
|
||||
"reason": "decay < threshold but confidence >= 0.7 — manual review recommended",
|
||||
"reason": "decay < threshold mas confidence >= 0.7 — revisão manual recomendada",
|
||||
})
|
||||
log_message(f" ⚠️ ALERT: point {point_id} (decay={decay_score:.4f}, confidence={confidence_score:.2f}) — manual review recommended")
|
||||
log_message(f" ⚠️ ALERTA: ponto {point_id} (decay={decay_score:.4f}, confidence={confidence_score:.2f}) — revisão manual recomendada")
|
||||
else:
|
||||
# Archive
|
||||
# Arquivar
|
||||
ok = update_point_archived(point_id, collection, decay_score, args.dry_run)
|
||||
if ok:
|
||||
stats["archived"] += 1
|
||||
log_message(f" 📦 Archived: point {point_id} (decay={decay_score:.4f}, importance={importance_score:.2f})")
|
||||
log_message(f" 📦 Arquivado: ponto {point_id} (decay={decay_score:.4f}, importance={importance_score:.2f})")
|
||||
else:
|
||||
stats["failed"] += 1
|
||||
|
||||
# Structured JSON report
|
||||
# Relatório JSON estruturado
|
||||
report = {
|
||||
"timestamp": now_iso(),
|
||||
"collection": collection,
|
||||
|
|
@ -262,22 +265,22 @@ def main():
|
|||
}
|
||||
|
||||
log_message("=" * 60)
|
||||
log_message("📊 DECAY SCANNER REPORT")
|
||||
log_message("📊 RELATÓRIO DECAY SCANNER")
|
||||
log_message("=" * 60)
|
||||
log_message(f" Scanned: {stats['scanned']}")
|
||||
log_message(f" Archived: {stats['archived']}")
|
||||
log_message(f" Alerts (decay+conf.): {stats['alerted']}")
|
||||
log_message(f" Escanados: {stats['scanned']}")
|
||||
log_message(f" Arquivados: {stats['archived']}")
|
||||
log_message(f" Alertas (decay+conf.): {stats['alerted']}")
|
||||
log_message(f" Skipped human: {stats['skipped_human']}")
|
||||
log_message(f" Skipped procedural: {stats['skipped_procedural']}")
|
||||
log_message(f" Skipped high imp.: {stats['skipped_high_importance']}")
|
||||
log_message(f" Skipped archived: {stats['skipped_already_archived']}")
|
||||
log_message(f" Failures: {stats['failed']}")
|
||||
log_message(f" Falhas: {stats['failed']}")
|
||||
log_message("=" * 60)
|
||||
|
||||
# JSON report to stderr (parseable)
|
||||
# JSON report to stderr (parseável)
|
||||
print(json.dumps(report, ensure_ascii=False, indent=2), file=sys.stderr)
|
||||
|
||||
log_message("✅ Decay scanner complete.")
|
||||
log_message("✅ Decay scanner completo.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
|
|
|||
|
|
@ -1,11 +1,11 @@
|
|||
#!/usr/bin/env python3
|
||||
"""
|
||||
DLQ Manager — Reads, classifies, reports, and marks wiki ingest failures.
|
||||
DLQ Manager — Lê, classifica, reporta e marca falhas do wiki ingest.
|
||||
|
||||
Usage:
|
||||
python3 dlq_manager.py --report # report unreported failures
|
||||
python3 dlq_manager.py --status # DLQ status summary
|
||||
python3 dlq_manager.py --json # full JSON output
|
||||
Uso:
|
||||
python3 dlq_manager.py --report # reporta falhas não reportadas
|
||||
python3 dlq_manager.py --status # status resumido da DLQ
|
||||
python3 dlq_manager.py --json # saída JSON completa
|
||||
"""
|
||||
|
||||
import os
|
||||
|
|
@ -18,10 +18,10 @@ from dataclasses import dataclass, asdict, field
|
|||
from collections import Counter
|
||||
|
||||
# ─── Config ────────────────────────────────────────────────────────────────
|
||||
DLQ_PATH = os.environ.get("HERMES_DLQ_PATH", os.path.expanduser("~/.hermes/wiki_ingest_failures.json"))
|
||||
REPORT_LOG = os.environ.get("HERMES_DLQ_REPORT_LOG", os.path.expanduser("~/.hermes/cron/output/dlq_reports.jsonl"))
|
||||
REPORT_DIR = os.environ.get("HERMES_DLQ_REPORT_DIR", os.path.expanduser("~/.hermes/cron/output/quality_report"))
|
||||
MAX_REPORT_HISTORY = 100 # entries in JSONL
|
||||
DLQ_PATH = os.path.expanduser("~/.hermes/wiki_ingest_failures.json")
|
||||
REPORT_LOG = os.path.expanduser("~/.hermes/cron/output/dlq_reports.jsonl")
|
||||
REPORT_DIR = os.path.expanduser("~/.hermes/cron/output/quality_report")
|
||||
MAX_REPORT_HISTORY = 100 # entradas no JSONL
|
||||
|
||||
# ─── Data Model ─────────────────────────────────────────────────────────────
|
||||
|
||||
|
|
@ -34,7 +34,7 @@ class DLQEntry:
|
|||
reported: bool = False
|
||||
retry_count: int = 0
|
||||
last_retry: Optional[str] = None
|
||||
error_hash: str = "" # error hash for deduplication
|
||||
error_hash: str = "" # hash do erro para deduplicação
|
||||
|
||||
# ─── File I/O ─────────────────────────────────────────────────────────────
|
||||
|
||||
|
|
@ -50,7 +50,7 @@ def load_dlq() -> List[DLQEntry]:
|
|||
return [DLQEntry(**item) for item in data["failures"]]
|
||||
return []
|
||||
except Exception as e:
|
||||
print(f"[DLQ-ERROR] Failed to load: {e}", file=sys.stderr)
|
||||
print(f"[DLQ-ERROR] Falha ao carregar: {e}", file=sys.stderr)
|
||||
return []
|
||||
|
||||
def save_dlq(entries: List[DLQEntry]):
|
||||
|
|
@ -78,7 +78,7 @@ def classify_error(error_msg: str) -> str:
|
|||
return "unknown"
|
||||
|
||||
def compute_error_hash(file: str, error: str) -> str:
|
||||
"""Generate a simple hash for deduplication of similar errors."""
|
||||
"""Gera um hash simples para deduplicação de erros similares."""
|
||||
import hashlib
|
||||
return hashlib.md5(f"{file}:{error[:80]}".encode()).hexdigest()[:8]
|
||||
|
||||
|
|
@ -91,7 +91,7 @@ def build_report(entries: List[DLQEntry]) -> Dict:
|
|||
if not unreported:
|
||||
return {"status": "ok", "unreported_count": 0, "total": total, "report": ""}
|
||||
|
||||
# Classify
|
||||
# Classifica
|
||||
for e in unreported:
|
||||
if e.failure_class == "unknown":
|
||||
e.failure_class = classify_error(e.error)
|
||||
|
|
@ -101,22 +101,22 @@ def build_report(entries: List[DLQEntry]) -> Dict:
|
|||
by_file = Counter(os.path.basename(e.file) for e in unreported)
|
||||
|
||||
lines = [
|
||||
f"🚨 [DLQ-ALERT] {len(unreported)} new failure(s) in ingest",
|
||||
f" Total accumulated in DLQ: {total}",
|
||||
f"🚨 [DLQ-ALERT] {len(unreported)} nova(s) falha(s) no ingest",
|
||||
f" Total acumulado na DLQ: {total}",
|
||||
"",
|
||||
"By class:",
|
||||
"Por classe:",
|
||||
]
|
||||
emoji = {"transient": "⏳", "permanent": "💀", "unknown": "❓"}
|
||||
for cls, count in by_class.most_common():
|
||||
lines.append(f" {emoji.get(cls, '❓')} {cls}: {count}")
|
||||
|
||||
lines.append("")
|
||||
lines.append("Top errors:")
|
||||
lines.append("Top erros:")
|
||||
for err, count in by_error_short.most_common(5):
|
||||
lines.append(f" • ({count}x) {err}")
|
||||
|
||||
lines.append("")
|
||||
lines.append("Files:")
|
||||
lines.append("Arquivos:")
|
||||
for fname, count in by_file.most_common(10):
|
||||
lines.append(f" • {fname} ({count}x)")
|
||||
|
||||
|
|
@ -133,7 +133,6 @@ def build_report(entries: List[DLQEntry]) -> Dict:
|
|||
|
||||
def save_report(report: Dict):
|
||||
os.makedirs(REPORT_DIR, exist_ok=True)
|
||||
os.makedirs(os.path.dirname(REPORT_LOG), exist_ok=True)
|
||||
timestamp = datetime.now().isoformat()
|
||||
|
||||
# JSONL
|
||||
|
|
@ -150,7 +149,7 @@ def get_status_summary(entries: List[DLQEntry]) -> Dict:
|
|||
total = len(entries)
|
||||
unreported = len([e for e in entries if not e.reported])
|
||||
by_class = Counter(e.failure_class for e in entries)
|
||||
recent = [e for e in entries if datetime.now(datetime.timezone.utc) - datetime.fromisoformat(e.timestamp.replace("Z", "+00:00")).astimezone(datetime.timezone.utc) < timedelta(hours=24)]
|
||||
recent = [e for e in entries if datetime.now() - datetime.fromisoformat(e.timestamp.replace("Z", "+00:00")) < timedelta(hours=24)]
|
||||
|
||||
return {
|
||||
"total": total,
|
||||
|
|
@ -164,11 +163,11 @@ def get_status_summary(entries: List[DLQEntry]) -> Dict:
|
|||
|
||||
def main():
|
||||
import argparse
|
||||
p = argparse.ArgumentParser(description="DLQ Manager — Auto-report of failures")
|
||||
p.add_argument("--report", action="store_true", help="Generate report of unreported failures")
|
||||
p.add_argument("--status", action="store_true", help="Status summary")
|
||||
p.add_argument("--json", action="store_true", help="JSON output")
|
||||
p.add_argument("--silent-if-ok", action="store_true", help="Silent if DLQ is ok")
|
||||
p = argparse.ArgumentParser(description="DLQ Manager — Auto-report de falhas")
|
||||
p.add_argument("--report", action="store_true", help="Gerar relatório das não-reportadas")
|
||||
p.add_argument("--status", action="store_true", help="Status resumido")
|
||||
p.add_argument("--json", action="store_true", help="Saída JSON")
|
||||
p.add_argument("--silent-if-ok", action="store_true", help="Silencioso se DLQ ok")
|
||||
args = p.parse_args()
|
||||
|
||||
entries = load_dlq()
|
||||
|
|
@ -178,7 +177,7 @@ def main():
|
|||
if args.json:
|
||||
print(json.dumps(summary, indent=2, ensure_ascii=False))
|
||||
else:
|
||||
print(f"DLQ status: {summary['total']} total, {summary['unreported']} unreported")
|
||||
print(f"DLQ status: {summary['total']} total, {summary['unreported']} não-reportadas")
|
||||
for cls, count in summary.get("by_class", {}).items():
|
||||
print(f" {cls}: {count}")
|
||||
return
|
||||
|
|
@ -186,25 +185,25 @@ def main():
|
|||
report = build_report(entries)
|
||||
|
||||
if report["status"] == "ok":
|
||||
msg = "[DLQ-OK] No new failures since last check."
|
||||
msg = "[DLQ-OK] Nenhuma falha nova desde último check."
|
||||
if not args.silent_if_ok:
|
||||
print(msg)
|
||||
if args.json:
|
||||
print(json.dumps(report, indent=2, ensure_ascii=False))
|
||||
return
|
||||
|
||||
# Has new failures
|
||||
# Tem novas falhas
|
||||
if args.json:
|
||||
print(json.dumps(report, indent=2, ensure_ascii=False))
|
||||
else:
|
||||
print(report["report"])
|
||||
|
||||
# Save and mark as reported
|
||||
# Salva e marca como reportadas
|
||||
save_report(report)
|
||||
mark_reported(entries)
|
||||
save_dlq(entries)
|
||||
|
||||
# Exit code 1 for cron trigger
|
||||
# Exit code 1 para cron trigger
|
||||
sys.exit(1)
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
|
|
|||
|
|
@ -1,18 +1,18 @@
|
|||
#!/usr/bin/env python3
|
||||
"""
|
||||
Semantic Pre-Validator — Decision linter based on the knowledge_base.
|
||||
Queries the vault before I/O actions or API calls.
|
||||
Pré-validador Semântico — Linter de decisão baseado no knowledge_base.
|
||||
Consulta o vault antes de ações de I/O ou chamadas de API.
|
||||
|
||||
Usage:
|
||||
python3 pre_validator.py "POST to Qdrant upsert" # should find pitfalls
|
||||
python3 pre_validator.py --json "use Claude from Anthropic" # JSON output
|
||||
python3 pre_validator.py --domain qdrant,api "modify docker-compose" # restrict search
|
||||
Uso:
|
||||
python3 pre_validator.py "fazer POST no upsert do Qdrant" # deve findar pitfall
|
||||
python3 pre_validator.py --json "usar Claude da Anthropic" # JSON output
|
||||
python3 pre_validator.py --domain qdrant,api "modificar docker-compose" # restringe busca
|
||||
|
||||
Exit codes:
|
||||
0 = pass/warn (action may proceed)
|
||||
1 = blocked (action must be aborted)
|
||||
0 = pass/warn (ação pode prosseguir)
|
||||
1 = blocked (ação deve ser abortada)
|
||||
|
||||
Fail-open: if OpenRouter or Qdrant is offline, allows execution with a warning.
|
||||
Fail-open: se OpenRouter ou Qdrant offline, permite execução com alerta.
|
||||
"""
|
||||
|
||||
import os
|
||||
|
|
@ -28,11 +28,7 @@ OPENROUTER_KEY = os.environ.get("OPENROUTER_API_KEY")
|
|||
QDRANT_URL = os.environ.get("QDRANT_URL", "http://localhost:6333")
|
||||
COLLECTION = os.environ.get("QDRANT_COLLECTION", "knowledge_base")
|
||||
if not OPENROUTER_KEY:
|
||||
_env_path = os.environ.get("ENV_PATH", "")
|
||||
if _env_path:
|
||||
_env = Path(_env_path)
|
||||
else:
|
||||
_env = Path.home() / ".env"
|
||||
_env = Path.home() / ".env"
|
||||
if _env.exists():
|
||||
for ln in _env.read_text().splitlines():
|
||||
if ln.startswith("OPENROUTER_API_KEY="):
|
||||
|
|
@ -41,7 +37,7 @@ if not OPENROUTER_KEY:
|
|||
EMBEDDING_MODEL = "qwen/qwen3-embedding-8b"
|
||||
TOP_K = 5
|
||||
SCORE_THRESHOLD = 0.60
|
||||
WARN_THRESHOLD = 0.75 # pure wiki docs need a higher score for a warning
|
||||
WARN_THRESHOLD = 0.75 # docs wiki pura precisam de score mais alto para aviso
|
||||
BLOCK_SEVERITIES = {"critical", "high"}
|
||||
WARN_SEVERITIES = {"medium"}
|
||||
RULE_SOURCES = {"reflection", "decision", "rule", "pitfall", "insight"}
|
||||
|
|
@ -49,15 +45,15 @@ REQUEST_TIMEOUT = 10
|
|||
|
||||
# ─── Restriction Patterns in wiki text ─────────────────────────────────────
|
||||
RESTRICTION_KEYWORDS = [
|
||||
"do not use", "must not", "cannot", "never use", "avoid",
|
||||
"forbidden", "not recommended", "anti-pattern", "common mistake",
|
||||
"caution", "warning", "important:", "⚠️", "🚫",
|
||||
"must use", "must always", "requires", "mandatory",
|
||||
"keep", "do not change", "do not modify", "freeze",
|
||||
"não usar", "não deve", "não pode", "nunca usar", "evitar",
|
||||
"proibido", "não recomendado", "anti-padrão", "erro comum",
|
||||
"cuidado", "atenção", "importante:", "⚠️", "🚫",
|
||||
"deve usar", "deve sempre", "requer", "obrigatório",
|
||||
"manter", "não alterar", "não modificar", "congelar",
|
||||
]
|
||||
|
||||
def contains_restriction(text: str) -> bool:
|
||||
"""Check whether text contains restriction/decision patterns."""
|
||||
"""Verifica se um texto contém padrões de restrição/decisão."""
|
||||
if not text:
|
||||
return False
|
||||
text_lower = text.lower()
|
||||
|
|
@ -123,7 +119,7 @@ def search_knowledge_base(vector: List[float], domain_tags: List[str]) -> List[D
|
|||
tags = [str(t).lower() for t in pld.get("tags", [])]
|
||||
score = item.get("score", 0)
|
||||
|
||||
# If domain filters requested, require overlap
|
||||
# Se pediu filtros de domínio, exige overlap
|
||||
if domain_tags:
|
||||
dom_low = [d.lower() for d in domain_tags]
|
||||
if not set(dom_low) & set(tags):
|
||||
|
|
@ -132,7 +128,7 @@ def search_knowledge_base(vector: List[float], domain_tags: List[str]) -> List[D
|
|||
hits.append({
|
||||
"id" : str(item.get("id", "")),
|
||||
"score" : score,
|
||||
"title" : pld.get("title", "Untitled"),
|
||||
"title" : pld.get("title", "Sem título"),
|
||||
"text" : (pld.get("text", "") or "")[:400],
|
||||
"source" : src,
|
||||
"severity": sev,
|
||||
|
|
@ -145,30 +141,30 @@ def search_knowledge_base(vector: List[float], domain_tags: List[str]) -> List[D
|
|||
return []
|
||||
|
||||
def is_rule_hit(hit: Dict) -> bool:
|
||||
"""Return True if the hit contains an explicit rule (reflection/decision/rule/insight/pitfall)."""
|
||||
"""Retorna True se o hit contém regra explícita (reflection/decision/rule/insight/pitfall)."""
|
||||
return any(s in hit["source"] for s in RULE_SOURCES)
|
||||
|
||||
def classify_hit(hit: Dict, action_desc: str) -> str:
|
||||
"""
|
||||
Return hit category: 'block', 'warn', 'info', or 'none'.
|
||||
Considers both source=reflection/decision/rule and restriction patterns
|
||||
embedded in wiki document text.
|
||||
Retorna categoria do hit: 'block', 'warn', 'info', ou 'none'.
|
||||
Considera tanto source=reflection/decision/rule quanto padrões de restrição
|
||||
embutidos no texto de documentos wiki.
|
||||
"""
|
||||
sev = hit.get("severity", "low")
|
||||
is_rule = is_rule_hit(hit) or contains_restriction(hit.get("text", ""))
|
||||
score = hit.get("score", 0)
|
||||
|
||||
# If text contains restriction, give it more weight
|
||||
# Se contém restrição no texto, dá mais peso
|
||||
restriction_bonus = 0.08 if contains_restriction(hit.get("text", "")) else 0
|
||||
effective_score = score + restriction_bonus
|
||||
|
||||
# Proximity: if the action term (e.g. "POST") appears near a keyword in the text
|
||||
# Proximidade: se a ação (ex: "POST") aparece no texto próximo a uma keyword
|
||||
action_terms = set(action_desc.lower().split())
|
||||
text_lower = (hit.get("text", "") or "").lower()
|
||||
text_words = set(text_lower.split())
|
||||
proximity_match = len(action_terms & text_words) > 0
|
||||
|
||||
# If restriction + proximity → elevate severity
|
||||
# Se tem restrição + proximidade → eleva severidade
|
||||
has_restriction = contains_restriction(hit.get("text", "")) and proximity_match
|
||||
|
||||
if is_rule or has_restriction:
|
||||
|
|
@ -177,7 +173,7 @@ def classify_hit(hit: Dict, action_desc: str) -> str:
|
|||
elif sev in WARN_SEVERITIES or (has_restriction and effective_score >= SCORE_THRESHOLD):
|
||||
return "warn"
|
||||
|
||||
# For normal wiki documents, only warn if score is very high
|
||||
# Para documentos wiki normais, só avisa se score muito alto
|
||||
if effective_score >= WARN_THRESHOLD:
|
||||
return "warn"
|
||||
if effective_score >= SCORE_THRESHOLD:
|
||||
|
|
@ -189,7 +185,7 @@ def validate_action(action_description: str, domain_tags: Optional[List[str]] =
|
|||
dom = domain_tags or infer_domain_tags(action_description)
|
||||
vec = embed_text(action_description)
|
||||
if vec is None:
|
||||
return {"status": "pass", "blocked": False, "message": "⚠️ Validator offline. Proceeding with caution.", "action": action_description}
|
||||
return {"status": "pass", "blocked": False, "message": "⚠️ Validador offline. Executando com cautela.", "action": action_description}
|
||||
|
||||
hits = search_knowledge_base(vec, dom)
|
||||
blockers = []
|
||||
|
|
@ -197,7 +193,7 @@ def validate_action(action_description: str, domain_tags: Optional[List[str]] =
|
|||
infos = []
|
||||
|
||||
for h in hits:
|
||||
cat = classify_hit(h, action_description)
|
||||
cat = classify_hit(h)
|
||||
if cat == "block":
|
||||
blockers.append(h)
|
||||
elif cat == "warn":
|
||||
|
|
@ -206,12 +202,12 @@ def validate_action(action_description: str, domain_tags: Optional[List[str]] =
|
|||
infos.append(h)
|
||||
|
||||
if blockers:
|
||||
lines = [f"🚫 ACTION BLOCKED — {len(blockers)} critical rule(s) in the vault:"]
|
||||
lines = [f"🚫 AÇÃO BLOQUEADA — {len(blockers)} regra(s) crítica(s) no vault:"]
|
||||
for b in blockers:
|
||||
lines.append(f" • [{b['severity'].upper()}] {b['title']} (score: {b['score']:.2f})")
|
||||
lines.append(f" {b['text'][:200]}...")
|
||||
lines.append("")
|
||||
lines.append("Override? Type 'force' (not recommended).")
|
||||
lines.append("Ignorar? Digite 'forçar' (não recomendado).")
|
||||
return {
|
||||
"status": "blocked", "blocked": True,
|
||||
"blockers": blockers, "warnings": warnings,
|
||||
|
|
@ -219,7 +215,7 @@ def validate_action(action_description: str, domain_tags: Optional[List[str]] =
|
|||
}
|
||||
|
||||
if warnings:
|
||||
lines = [f"⚠️ {len(warnings)} warning(s) found in the vault:"]
|
||||
lines = [f"⚠️ {len(warnings)} aviso(s) encontrado(s) no vault:"]
|
||||
for w in warnings:
|
||||
lines.append(f" • [{w['severity'].upper()}] {w['title']} (score: {w['score']:.2f})")
|
||||
lines.append(f" {w['text'][:200]}...")
|
||||
|
|
@ -233,34 +229,34 @@ def validate_action(action_description: str, domain_tags: Optional[List[str]] =
|
|||
return {
|
||||
"status": "info", "blocked": False,
|
||||
"infos": infos,
|
||||
"message": f"ℹ️ {len(infos)} relevant document(s), none critical.",
|
||||
"message": f"ℹ️ {len(infos)} documento(s) relevante(s), nenhum crítico.",
|
||||
"action": action_description, "domain": dom,
|
||||
}
|
||||
|
||||
return {
|
||||
"status": "pass", "blocked": False,
|
||||
"message": "No relevant insights found. Execution authorized.",
|
||||
"message": "Nenhum insight relevante encontrado. Execução autorizada.",
|
||||
"action": action_description, "domain": dom,
|
||||
}
|
||||
except Exception as e:
|
||||
return {
|
||||
"status": "pass", "blocked": False,
|
||||
"message": f"Validator failed ({e}). Proceeding with caution.",
|
||||
"message": f"Validador falhou ({e}). Executando com cautela.",
|
||||
"action": action_description, "domain": [],
|
||||
}
|
||||
|
||||
# ─── Main ───────────────────────────────────────────────────────────────────
|
||||
def main():
|
||||
import argparse
|
||||
p = argparse.ArgumentParser(description="Semantic Pre-Validator")
|
||||
p.add_argument("action", nargs="?", help="Action description")
|
||||
p.add_argument("--domain", help="Comma-separated domain tags")
|
||||
p.add_argument("--json", action="store_true", help="JSON output")
|
||||
p.add_argument("--silent", action="store_true", help="Silent — exit code only")
|
||||
p.add_argument("--force-block", action="store_true", help="Force block (testing)")
|
||||
p = argparse.ArgumentParser(description="Pré-validador Semântico")
|
||||
p.add_argument("action", nargs="?", help="Descrição da ação")
|
||||
p.add_argument("--domain", help="Tags de domínio separadas por vírgula")
|
||||
p.add_argument("--json", action="store_true", help="Saída JSON")
|
||||
p.add_argument("--silent", action="store_true", help="Silencioso — só exit code")
|
||||
p.add_argument("--force-block", action="store_true", help="Forçar bloqueio (teste)")
|
||||
args = p.parse_args()
|
||||
|
||||
action = args.action or sys.stdin.read().strip() or "POST to Qdrant upsert endpoint"
|
||||
action = args.action or sys.stdin.read().strip() or "fazer POST no endpoint de upsert do Qdrant"
|
||||
dom = [x.strip() for x in args.domain.split(",")] if args.domain else None
|
||||
|
||||
res = validate_action(action, dom)
|
||||
|
|
@ -273,7 +269,7 @@ def main():
|
|||
elif not args.silent:
|
||||
print(res["message"])
|
||||
if res["blocked"]:
|
||||
print("\n(Use --force-block to test validator bypass)")
|
||||
print("\n(Use --force-block para testar bypass de validador)")
|
||||
|
||||
sys.exit(1 if res["blocked"] else 0)
|
||||
|
||||
|
|
|
|||
|
|
@ -1,18 +1,18 @@
|
|||
#!/usr/bin/env python3
|
||||
"""
|
||||
reflection_trigger.py
|
||||
Checks whether the ARQ worker is idle (no pending/running jobs)
|
||||
and dispatches a micro_reflection via ARQ enqueue. Runs via cron every 5 minutes.
|
||||
Verifica se o worker ARQ está ocioso (sem jobs pendentes/em execução)
|
||||
e dispara micro_reflection via enqueue ARQ. Roda via cron a cada 5 minutos.
|
||||
|
||||
Rules:
|
||||
- Only triggers if there are no pending or running jobs (idle)
|
||||
- Respects the max_per_hour budget (reads from env or defaults to 5)
|
||||
- Enqueues ARQ job "process_micro_reflection" (function registered in the worker)
|
||||
- Fail-open: if Redis/ARQ is unavailable, exits silently
|
||||
- Never blocks the critical query/ingestion path
|
||||
Regras:
|
||||
- Só dispara se não há jobs pendentes nem em execução (idle)
|
||||
- Respeita budget max_per_hour (lê do env ou assume 5)
|
||||
- Enfileira job ARQ "process_micro_reflection" (função registrada no worker)
|
||||
- Fail-open: se Redis/ARQ indisponível, sai silenciosamente
|
||||
- Nunca bloqueia o path crítico de query/ingestão
|
||||
|
||||
Usage (cron):
|
||||
*/5 * * * * $VENV_DIR/bin/python $PROJECT_DIR/scripts/reflection_trigger.py >> $HERMES_LOG_DIR/reflection_trigger.cron.log 2>&1
|
||||
Uso (cron):
|
||||
*/5 * * * * /home/calli/ai-stack/cognitive-agent/venv/bin/python /home/calli/ai-stack/scripts/reflection_trigger.py >> /home/calli/.hermes/logs/reflection_trigger.cron.log 2>&1
|
||||
"""
|
||||
|
||||
import os
|
||||
|
|
@ -29,7 +29,7 @@ from arq.connections import RedisSettings
|
|||
import redis.asyncio as aioredis
|
||||
|
||||
# ─── Config ────────────────────────────────────────────────────────────────
|
||||
ENV_PATH = Path(os.environ.get("MAA_ENV_PATH", "."))
|
||||
ENV_PATH = Path.home() / "ai-stack" / "cognitive-agent" / ".env"
|
||||
if ENV_PATH.exists():
|
||||
load_dotenv(ENV_PATH)
|
||||
|
||||
|
|
@ -44,12 +44,7 @@ redis_settings = RedisSettings(
|
|||
password=REDIS_PASSWORD or None,
|
||||
)
|
||||
|
||||
LOG_FILE = Path(
|
||||
os.environ.get(
|
||||
"REFLECTION_LOG_PATH",
|
||||
str(Path.home() / ".hermes" / "logs" / "reflection_trigger.log")
|
||||
)
|
||||
)
|
||||
LOG_FILE = Path.home() / ".hermes" / "logs" / "reflection_trigger.log"
|
||||
|
||||
|
||||
def log_message(msg: str):
|
||||
|
|
@ -65,7 +60,7 @@ def log_message(msg: str):
|
|||
|
||||
|
||||
async def is_idle() -> bool:
|
||||
"""Check whether there are any pending or running jobs in ARQ."""
|
||||
"""Verifica se não há jobs pendentes nem em execução no ARQ."""
|
||||
try:
|
||||
r = aioredis.Redis(
|
||||
host=REDIS_HOST, port=REDIS_PORT,
|
||||
|
|
@ -73,7 +68,7 @@ async def is_idle() -> bool:
|
|||
decode_responses=True,
|
||||
)
|
||||
|
||||
# ARQ stores jobs in queues like 'arq:queue:default'
|
||||
# ARQ armazena jobs em filas tipo 'arq:queue:default'
|
||||
queue_names = ["arq:queue:default"]
|
||||
qr_prefix = os.environ.get("ARQ_QUEUE_PREFIX", "arq:queue:")
|
||||
if qr_prefix:
|
||||
|
|
@ -90,7 +85,7 @@ async def is_idle() -> bool:
|
|||
except Exception:
|
||||
pass
|
||||
|
||||
# In-progress jobs: ARQ uses sets like 'arq:in-progress:...'
|
||||
# Jobs em execução: ARQ usa sets tipo 'arq:in-progress:...'
|
||||
in_progress_keys = await r.keys("arq:in-progress:*")
|
||||
total_in_progress = 0
|
||||
for key in in_progress_keys:
|
||||
|
|
@ -102,20 +97,15 @@ async def is_idle() -> bool:
|
|||
await r.aclose()
|
||||
return (total_pending + total_in_progress) == 0
|
||||
except Exception as e:
|
||||
log_message(f"Error checking idle status: {e}")
|
||||
return False # fail-safe: if unable to verify, do not trigger
|
||||
log_message(f"Erro ao verificar idle: {e}")
|
||||
return False # fail-safe: se não conseguir verificar, não dispara
|
||||
|
||||
|
||||
async def check_budget() -> tuple[bool, int, int]:
|
||||
"""Return (allowed, used, max) based on the hourly counter in SQLite."""
|
||||
"""Retorna (permitido, used, max) baseado no contador da hora no SQLite."""
|
||||
try:
|
||||
import sqlite3
|
||||
db_path = Path(
|
||||
os.environ.get(
|
||||
"STATE_DB_PATH",
|
||||
str(Path.home() / ".hermes" / "state.db")
|
||||
)
|
||||
)
|
||||
db_path = Path.home() / ".hermes" / "state.db"
|
||||
hour_window = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H")
|
||||
conn = sqlite3.connect(str(db_path))
|
||||
c = conn.cursor()
|
||||
|
|
@ -125,20 +115,15 @@ async def check_budget() -> tuple[bool, int, int]:
|
|||
conn.close()
|
||||
return (used < MAX_REFLECTIONS_PER_HOUR, used, MAX_REFLECTIONS_PER_HOUR)
|
||||
except Exception as e:
|
||||
log_message(f"Error checking budget: {e}")
|
||||
log_message(f"Erro ao verificar budget: {e}")
|
||||
return (True, 0, MAX_REFLECTIONS_PER_HOUR) # fail-open
|
||||
|
||||
|
||||
def increment_budget():
|
||||
"""Increment the reflection counter in SQLite."""
|
||||
"""Incrementa o contador de reflections no SQLite."""
|
||||
try:
|
||||
import sqlite3
|
||||
db_path = Path(
|
||||
os.environ.get(
|
||||
"STATE_DB_PATH",
|
||||
str(Path.home() / ".hermes" / "state.db")
|
||||
)
|
||||
)
|
||||
db_path = Path.home() / ".hermes" / "state.db"
|
||||
hour_window = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H")
|
||||
conn = sqlite3.connect(str(db_path))
|
||||
c = conn.cursor()
|
||||
|
|
@ -151,11 +136,11 @@ def increment_budget():
|
|||
conn.commit()
|
||||
conn.close()
|
||||
except Exception as e:
|
||||
log_message(f"Error incrementing budget: {e}")
|
||||
log_message(f"Erro ao incrementar budget: {e}")
|
||||
|
||||
|
||||
async def trigger_micro_reflection(dry_run: bool = False) -> dict:
|
||||
"""Pipeline: idle check → budget check → ARQ enqueue → increment budget."""
|
||||
"""Pipeline: idle check → budget check → enqueue ARQ → increment budget."""
|
||||
|
||||
# 1. Idle check
|
||||
idle = await is_idle()
|
||||
|
|
@ -186,10 +171,8 @@ async def trigger_micro_reflection(dry_run: bool = False) -> dict:
|
|||
job = await pool.enqueue_job("process_micro_reflection")
|
||||
await pool.aclose()
|
||||
|
||||
# 4. Budget accounting is owned by the worker after actual processing.
|
||||
# NOT incremented here — the worker's increment_budget() call
|
||||
# handles this, preventing double-counting.
|
||||
# increment_budget()
|
||||
# 4. Increment budget
|
||||
increment_budget()
|
||||
|
||||
return {
|
||||
"status": "triggered",
|
||||
|
|
@ -199,13 +182,13 @@ async def trigger_micro_reflection(dry_run: bool = False) -> dict:
|
|||
"max": max_ref,
|
||||
}
|
||||
except Exception as e:
|
||||
log_message(f"Error enqueuing micro_reflection: {e}")
|
||||
log_message(f"Erro ao enfileirar micro_reflection: {e}")
|
||||
return {"status": "error", "error": str(e), "triggered": False}
|
||||
|
||||
|
||||
async def main():
|
||||
parser = argparse.ArgumentParser(description="Reflection Trigger — idle detection")
|
||||
parser.add_argument("--dry-run", action="store_true", help="Simulate, do not enqueue")
|
||||
parser.add_argument("--dry-run", action="store_true", help="Simula, não enfileira")
|
||||
args = parser.parse_args()
|
||||
|
||||
result = await trigger_micro_reflection(dry_run=args.dry_run)
|
||||
|
|
|
|||
|
|
@ -1,24 +1,18 @@
|
|||
#!/usr/bin/env python3
|
||||
"""
|
||||
semantic_dedup.py
|
||||
Monthly scanner for near-duplicates in knowledge_base_hybrid via cosine similarity.
|
||||
Runs on the first Sunday of each month (cron: 0 3 1 * *).
|
||||
Scanner mensal de near-duplicates no knowledge_base_hybrid via cosine similarity.
|
||||
Rodo no primeiro domingo de cada mês (cron: 0 3 1 * *).
|
||||
|
||||
⚠️ WARNING: This performs O(n²) brute-force pairwise comparisons. For large
|
||||
collections (e.g. 100K+ points), this can be extremely slow and memory-heavy.
|
||||
Use --max-points to limit processing, or prefer Qdrant's built-in
|
||||
nearest-neighbor search on a random sample where feasible.
|
||||
Regras:
|
||||
- Ignora coleções com prefixo em DEDUP_EXEMPT_PREFIXES (csv)
|
||||
- Não deleta automaticamente — apenas emite relatório JSON de candidatos
|
||||
- Threshold de similaridade: 0.92 (configurável)
|
||||
- Merge é feito em upserts via file_ingestion.py (pre-write dedup)
|
||||
- Este script faz o scan retroativo da coleção inteira
|
||||
|
||||
Rules:
|
||||
- Ignores gabi_* collections
|
||||
- Does not delete automatically — only emits a JSON report of candidates
|
||||
- Similarity threshold: 0.92 (configurable)
|
||||
- Merge is handled via upserts in file_ingestion.py (pre-write dedup)
|
||||
- This script does the retrospective scan of the entire collection
|
||||
(capped by MAX_POINTS)
|
||||
|
||||
Usage:
|
||||
python3 semantic_dedup.py [--collection knowledge_base_hybrid] [--threshold 0.92] [--dry-run] [--max-points 5000]
|
||||
Uso:
|
||||
python3 semantic_dedup.py [--collection knowledge_base_hybrid] [--threshold 0.92] [--dry-run]
|
||||
"""
|
||||
|
||||
import os
|
||||
|
|
@ -34,19 +28,14 @@ from typing import List, Dict, Tuple, Optional
|
|||
# ─── Config ────────────────────────────────────────────────────────────────
|
||||
QDRANT_URL = os.environ.get("QDRANT_URL", "http://localhost:6333")
|
||||
COLLECTION = os.environ.get("QDRANT_COLLECTION", "knowledge_base")
|
||||
SCROLL_LIMIT = 50 # Qdrant pagination (avoids timeout on large collections)
|
||||
SCROLL_LIMIT = 50 # paginação Qdrant (evita timeout em coleções grandes)
|
||||
SIMILARITY_THRESHOLD = 0.92
|
||||
TOP_NEIGHBORS = 10
|
||||
|
||||
LOG_DIR = Path(
|
||||
os.environ.get("HERMES_LOG_DIR", str(Path.home() / ".hermes" / "logs"))
|
||||
)
|
||||
LOG_DIR = Path.home() / ".hermes" / "logs"
|
||||
LOG_FILE = LOG_DIR / "semantic_dedup.log"
|
||||
REPORT_FILE = LOG_DIR / "semantic_dedup_report.json"
|
||||
|
||||
# Safety cap — limit processed points to avoid O(n²) blowup on large collections
|
||||
MAX_POINTS = int(os.environ.get("DEDUP_MAX_POINTS", "5000"))
|
||||
|
||||
|
||||
def now_iso() -> str:
|
||||
return datetime.now(timezone.utc).isoformat()
|
||||
|
|
@ -68,8 +57,8 @@ def log_message(msg: str):
|
|||
|
||||
def scroll_all_chunks(collection: str) -> List[Dict]:
|
||||
"""
|
||||
Load all points from the collection, paginating via scroll.
|
||||
Returns a list of {id, vector, payload}.
|
||||
Carrega todos os pontos da coleção paginando via scroll.
|
||||
Retorna lista de {id, vector, payload}.
|
||||
"""
|
||||
all_chunks = []
|
||||
offset = None
|
||||
|
|
@ -100,13 +89,13 @@ def scroll_all_chunks(collection: str) -> List[Dict]:
|
|||
break
|
||||
|
||||
for point in points:
|
||||
# Get only the dense vector for similarity
|
||||
# Pegar apenas vetor dense para similarity
|
||||
vector = point.get("vector")
|
||||
dense = None
|
||||
if isinstance(vector, dict):
|
||||
dense = vector.get("dense")
|
||||
elif isinstance(vector, list):
|
||||
dense = vector # fallback: simple vector
|
||||
dense = vector # fallback: vetor simples
|
||||
|
||||
if dense:
|
||||
all_chunks.append({
|
||||
|
|
@ -121,15 +110,15 @@ def scroll_all_chunks(collection: str) -> List[Dict]:
|
|||
break
|
||||
|
||||
except Exception as e:
|
||||
log_message(f"❌ Error in Qdrant scroll: {e}")
|
||||
log_message(f"❌ Erro no scroll Qdrant: {e}")
|
||||
break
|
||||
|
||||
log_message(f"📊 Total chunks loaded: {len(all_chunks)} / {scanned} scanned")
|
||||
log_message(f"📊 Total chunks carregados: {len(all_chunks)} / {scanned} escaneados")
|
||||
return all_chunks
|
||||
|
||||
|
||||
def cosine_similarity(v1: List[float], v2: List[float]) -> float:
|
||||
"""Calculate cosine similarity between two vectors."""
|
||||
"""Calcula cosine similarity entre dois vetores."""
|
||||
if len(v1) != len(v2):
|
||||
return 0.0
|
||||
|
||||
|
|
@ -145,25 +134,25 @@ def cosine_similarity(v1: List[float], v2: List[float]) -> float:
|
|||
|
||||
def find_near_duplicates(chunks: List[Dict], threshold: float = SIMILARITY_THRESHOLD) -> List[Dict]:
|
||||
"""
|
||||
Find near-duplicate pairs via brute-force cosine similarity.
|
||||
Optimization: upper-triangular matrix comparison.
|
||||
Returns list of {chunk_id_a, chunk_id_b, similarity}.
|
||||
Encontra pares de near-duplicates via brute-force cosine similarity.
|
||||
Otimização: comparação triangular superior da matriz.
|
||||
Retorna lista de {chunk_id_a, chunk_id_b, similarity}.
|
||||
"""
|
||||
n = len(chunks)
|
||||
if n < 2:
|
||||
return []
|
||||
|
||||
candidates = []
|
||||
ids_seen = set() # avoid duplicates (A,B) and (B,A)
|
||||
ids_seen = set() # evita duplicados (A,B) e (B,A)
|
||||
|
||||
for i in range(n):
|
||||
for j in range(i + 1, n):
|
||||
# Fast heuristic: skip if texts differ greatly in size
|
||||
# Heurística rápida: pular se textos são muito diferentes em tamanho
|
||||
text_len_i = len(chunks[i]["payload"].get("text", ""))
|
||||
text_len_j = len(chunks[j]["payload"].get("text", ""))
|
||||
if text_len_i > 0 and text_len_j > 0:
|
||||
ratio = min(text_len_i, text_len_j) / max(text_len_i, text_len_j)
|
||||
if ratio < 0.5: # Very different sizes, skip
|
||||
if ratio < 0.5: # Tamanhos muito diferentes, skip
|
||||
continue
|
||||
|
||||
sim = cosine_similarity(chunks[i]["vector"], chunks[j]["vector"])
|
||||
|
|
@ -183,13 +172,13 @@ def find_near_duplicates(chunks: List[Dict], threshold: float = SIMILARITY_THRES
|
|||
"text_preview_b": chunks[j]["payload"].get("text", "")[:100],
|
||||
})
|
||||
|
||||
# Sort by descending similarity
|
||||
# Ordenar por similaridade decrescente
|
||||
candidates.sort(key=lambda x: x["similarity"], reverse=True)
|
||||
return candidates
|
||||
|
||||
|
||||
def generate_report(candidates: List[Dict], collection: str, threshold: float, scanned: int) -> Dict:
|
||||
"""Generate structured JSON report."""
|
||||
"""Gera relatório estruturado em JSON."""
|
||||
return {
|
||||
"timestamp": now_iso(),
|
||||
"collection": collection,
|
||||
|
|
@ -198,74 +187,73 @@ def generate_report(candidates: List[Dict], collection: str, threshold: float, s
|
|||
"near_duplicate_pairs": len(candidates),
|
||||
"candidates": candidates,
|
||||
"recommendation": (
|
||||
f"{len(candidates)} near-duplicate pairs found. "
|
||||
"Review manually and apply merge via Qdrant point update if approved."
|
||||
f"{len(candidates)} pares de near-duplicates encontrados. "
|
||||
"Revisar manualmente e aplicar merge via Qdrant point update se aprovado."
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Semantic Dedup Scanner")
|
||||
parser.add_argument("--collection", default=COLLECTION, help="Qdrant collection name")
|
||||
parser.add_argument("--threshold", type=float, default=SIMILARITY_THRESHOLD, help="Cosine similarity threshold")
|
||||
parser.add_argument("--max-points", type=int, default=MAX_POINTS, help="Max points to process (cap O(n²))")
|
||||
parser.add_argument("--dry-run", action="store_true", help="Scan only, do not save report")
|
||||
parser.add_argument("--collection", default=COLLECTION, help="Nome da coleção Qdrant")
|
||||
parser.add_argument("--threshold", type=float, default=SIMILARITY_THRESHOLD, help="Threshold cosine similarity")
|
||||
parser.add_argument("--dry-run", action="store_true", help="Só escaneia, não salva relatório")
|
||||
args = parser.parse_args()
|
||||
|
||||
collection = args.collection
|
||||
|
||||
# Skip gabi_* collections
|
||||
if collection.startswith("gabi_"):
|
||||
log_message(f"⏭️ Collection '{collection}' is exempt (gabi_*). Exiting.")
|
||||
return
|
||||
# Ignorar coleções com prefixos exempt (via DEDUP_EXEMPT_PREFIXES env var)
|
||||
exempt_prefixes = os.environ.get("DEDUP_EXEMPT_PREFIXES", "").split(",")
|
||||
exempt_prefixes = [p.strip() for p in exempt_prefixes if p.strip()]
|
||||
for prefix in exempt_prefixes:
|
||||
if collection.startswith(prefix):
|
||||
log_message(f"⏭️ Coleção '{collection}' é exempt (prefixo '{prefix}'). Saindo.")
|
||||
return
|
||||
|
||||
log_message(f"🚀 Starting semantic dedup (collection={collection}, threshold={args.threshold}, dry_run={args.dry_run})")
|
||||
log_message(f"🚀 Iniciando semantic dedup (collection={collection}, threshold={args.threshold}, dry_run={args.dry_run})")
|
||||
|
||||
# Load chunks (capped by --max-points to avoid O(n²) blowup)
|
||||
# Carregar chunks
|
||||
chunks = scroll_all_chunks(collection)
|
||||
if args.max_points and len(chunks) > args.max_points:
|
||||
log_message(f"⚠️ Collection has {len(chunks)} points, truncating to {args.max_points} (use --max-points to change)")
|
||||
chunks = chunks[:args.max_points]
|
||||
|
||||
if not chunks:
|
||||
log_message("⚠️ No chunks found in the collection.")
|
||||
log_message("⚠️ Nenhum chunk encontrado na coleção.")
|
||||
return
|
||||
|
||||
# Find near-duplicates
|
||||
log_message(f"🔍 Analyzing similarity among {len(chunks)} chunks...")
|
||||
# Encontrar near-duplicates
|
||||
log_message(f"🔍 Analisando similaridade entre {len(chunks)} chunks...")
|
||||
candidates = find_near_duplicates(chunks, threshold=args.threshold)
|
||||
|
||||
# Generate report
|
||||
# Gerar relatório
|
||||
report = generate_report(candidates, collection, args.threshold, len(chunks))
|
||||
|
||||
log_message("=" * 60)
|
||||
log_message("📊 SEMANTIC DEDUP REPORT")
|
||||
log_message("📊 RELATÓRIO SEMANTIC DEDUP")
|
||||
log_message("=" * 60)
|
||||
log_message(f" Chunks scanned: {report['scanned_chunks']}")
|
||||
log_message(f" Near-duplicate pairs: {report['near_duplicate_pairs']}")
|
||||
log_message(f" Chunks escaneados: {report['scanned_chunks']}")
|
||||
log_message(f" Near-duplicate pairs: {report['near_duplicate_pairs']}")
|
||||
|
||||
if candidates:
|
||||
log_message(f" Top similarity: {candidates[0]['similarity']:.4f}")
|
||||
log_message(f" Top pair: {candidates[0]['chunk_id_a']} ↔ {candidates[0]['chunk_id_b']}")
|
||||
log_message(f" Top similaridade: {candidates[0]['similarity']:.4f}")
|
||||
log_message(f" Top par: {candidates[0]['chunk_id_a']} ↔ {candidates[0]['chunk_id_b']}")
|
||||
else:
|
||||
log_message(" No near-duplicates found.")
|
||||
log_message(" Nenhum near-duplicate encontrado.")
|
||||
|
||||
log_message("=" * 60)
|
||||
|
||||
# Save JSON report
|
||||
# Salvar relatório JSON
|
||||
if not args.dry_run and candidates:
|
||||
try:
|
||||
REPORT_FILE.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(REPORT_FILE, "w", encoding="utf-8") as f:
|
||||
json.dump(report, f, ensure_ascii=False, indent=2)
|
||||
log_message(f"📄 Report saved: {REPORT_FILE}")
|
||||
log_message(f"📄 Relatório salvo: {REPORT_FILE}")
|
||||
except Exception as e:
|
||||
log_message(f"❌ Error saving report: {e}")
|
||||
log_message(f"❌ Erro ao salvar relatório: {e}")
|
||||
|
||||
# Output JSON to stderr (parseable)
|
||||
# Output JSON para stderr (parseável)
|
||||
print(json.dumps(report, ensure_ascii=False, indent=2), file=sys.stderr)
|
||||
|
||||
log_message("✅ Semantic dedup complete.")
|
||||
log_message("✅ Semantic dedup completo.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
|
|
|||
|
|
@ -1,8 +1,8 @@
|
|||
#!/usr/bin/env python3
|
||||
"""
|
||||
wiki-continuous-ingest.py
|
||||
Detects new/modified .md files in the vault and enqueues them to the ARQ worker.
|
||||
Runs on the host, accesses local Redis (127.0.0.1:6379) and Qdrant (localhost:6333).
|
||||
Detecta novos/modificados .md no vault e enfileira no ARQ worker.
|
||||
Roda no host, acessa Redis local (127.0.0.1:6379) e Qdrant (localhost:6333).
|
||||
"""
|
||||
import os
|
||||
import sys
|
||||
|
|
@ -18,16 +18,13 @@ from arq.connections import RedisSettings
|
|||
import redis.asyncio as aioredis
|
||||
|
||||
# ─── Config ────────────────────────────────────────────────────────────────
|
||||
ENV_PATH = os.environ.get("ENV_PATH", "")
|
||||
if ENV_PATH:
|
||||
env_p = Path(ENV_PATH)
|
||||
if env_p.exists():
|
||||
load_dotenv(env_p)
|
||||
ENV_PATH = Path.home() / "ai-stack" / "cognitive-agent" / ".env"
|
||||
if ENV_PATH.exists():
|
||||
load_dotenv(ENV_PATH)
|
||||
|
||||
WIKI_ROOT = Path(os.environ.get("WIKI_ROOT", "."))
|
||||
STATE_DIR = Path(os.environ.get("HERMES_STATE_DIR", str(Path.home() / ".hermes")))
|
||||
STATE_FILE = STATE_DIR / "wiki_ingest_state.json"
|
||||
FAILURES_FILE = STATE_DIR / "wiki_ingest_failures.json"
|
||||
WIKI_ROOT = Path.home() / "Vault" / "wiki"
|
||||
STATE_FILE = Path.home() / ".hermes" / "wiki_ingest_state.json"
|
||||
FAILURES_FILE = Path.home() / ".hermes" / "wiki_ingest_failures.json"
|
||||
REDIS_PASSWORD = os.environ.get("REDIS_PASSWORD", "")
|
||||
|
||||
redis_settings = RedisSettings(
|
||||
|
|
@ -45,7 +42,7 @@ def load_state() -> dict:
|
|||
|
||||
|
||||
def save_state(state: dict):
|
||||
"""Atomic write via tempfile + rename to avoid corruption."""
|
||||
"""Atomic write via tempfile + rename para evitar corrupção."""
|
||||
STATE_FILE.parent.mkdir(parents=True, exist_ok=True)
|
||||
tmp = STATE_FILE.with_suffix(".tmp")
|
||||
with open(tmp, "w") as f:
|
||||
|
|
@ -60,7 +57,7 @@ def file_hash(path: Path) -> str:
|
|||
|
||||
|
||||
async def redis_ready() -> bool:
|
||||
"""Check whether Redis is accessible before enqueuing."""
|
||||
"""Verifica se Redis está acessível antes de enfileirar."""
|
||||
try:
|
||||
r = aioredis.Redis(
|
||||
host="127.0.0.1", port=6379,
|
||||
|
|
@ -72,13 +69,13 @@ async def redis_ready() -> bool:
|
|||
await r.aclose()
|
||||
return bool(ok)
|
||||
except Exception as e:
|
||||
print(f" ⚠️ Redis unavailable: {e}")
|
||||
print(f" ⚠️ Redis indisponível: {e}")
|
||||
return False
|
||||
|
||||
|
||||
async def main():
|
||||
if not await redis_ready():
|
||||
print("❌ Redis not ready. Docker stack may still be starting up. Aborting.")
|
||||
print("❌ Redis não pronto. Docker stack pode estar subindo. Abortando.")
|
||||
return
|
||||
|
||||
state = load_state()
|
||||
|
|
@ -87,7 +84,7 @@ async def main():
|
|||
skipped = 0
|
||||
total = 0
|
||||
|
||||
# Scan all .md files
|
||||
# Varre todos os .md
|
||||
for path in sorted(WIKI_ROOT.rglob("*.md")):
|
||||
total += 1
|
||||
rel = str(path.relative_to(WIKI_ROOT))
|
||||
|
|
@ -96,12 +93,11 @@ async def main():
|
|||
|
||||
if rel not in state:
|
||||
new_files.append(rel)
|
||||
state[rel] = {"mtime": mtime, "hash": current_hash, "queued_at": None, "ingested_at": None}
|
||||
state[rel] = {"mtime": mtime, "hash": current_hash, "ingested_at": None}
|
||||
elif state[rel]["hash"] != current_hash:
|
||||
modified_files.append(rel)
|
||||
state[rel]["mtime"] = mtime
|
||||
state[rel]["hash"] = current_hash
|
||||
state[rel]["queued_at"] = None
|
||||
state[rel]["ingested_at"] = None
|
||||
else:
|
||||
skipped += 1
|
||||
|
|
@ -109,10 +105,10 @@ async def main():
|
|||
files_to_ingest = new_files + modified_files
|
||||
|
||||
if not files_to_ingest:
|
||||
print(f"⏭️ Nothing new. {total} files tracked, {skipped} unchanged.")
|
||||
print(f"⏭️ Nada novo. {total} arquivos rastreados, {skipped} inalterados.")
|
||||
return
|
||||
|
||||
# Enqueue in ARQ
|
||||
# Enfileirar no ARQ
|
||||
redis = await create_pool(redis_settings)
|
||||
enqueued = 0
|
||||
failed = 0
|
||||
|
|
@ -123,15 +119,15 @@ async def main():
|
|||
try:
|
||||
job = await redis.enqueue_job(
|
||||
"process_wiki_file",
|
||||
file_path=f"/wiki/{rel_path}", # path inside container
|
||||
file_path=f"/wiki/{rel_path}", # path dentro do container
|
||||
)
|
||||
state[rel_path]["queued_at"] = datetime.now(timezone.utc).isoformat()
|
||||
state[rel_path]["ingested_at"] = datetime.now(timezone.utc).isoformat()
|
||||
enqueued += 1
|
||||
print(f" ✅ Enqueued: {rel_path} (job: {job.job_id[:8]})")
|
||||
print(f" ✅ Enfileirado: {rel_path} (job: {job.job_id[:8]})")
|
||||
except Exception as e:
|
||||
failed += 1
|
||||
error_msg = str(e)
|
||||
# Classify the error for the DLQ
|
||||
# Classificar o erro para o DLQ
|
||||
error_lower = error_msg.lower()
|
||||
transient_patterns = ["timeout", "connection", "rate limit", "503", "502", "504",
|
||||
"unavailable", "too many requests", "refused", "reset"]
|
||||
|
|
@ -152,16 +148,16 @@ async def main():
|
|||
"timestamp": datetime.now(timezone.utc).isoformat(),
|
||||
"file": rel_path,
|
||||
"error": error_msg,
|
||||
"failure_class": failure_class, # NEW: classification
|
||||
"reported": False, # NEW: not yet reported
|
||||
"retry_count": 0, # NEW: zero retries
|
||||
"failure_class": failure_class, # NOVO: classificação
|
||||
"reported": False, # NOVO: ainda não reportado
|
||||
"retry_count": 0, # NOVO: zero retries
|
||||
})
|
||||
print(f" ⚠️ Failure: {rel_path} — {e} [{failure_class}]")
|
||||
print(f" ⚠️ Falha: {rel_path} — {e} [{failure_class}]")
|
||||
|
||||
await redis.aclose()
|
||||
save_state(state)
|
||||
|
||||
# Persist failures to simple DLQ (atomic, last 500)
|
||||
# Persistir falhas para DLQ simples (atômico, últimas 500)
|
||||
if failures:
|
||||
FAILURES_FILE.parent.mkdir(parents=True, exist_ok=True)
|
||||
existing = []
|
||||
|
|
@ -177,11 +173,11 @@ async def main():
|
|||
os.fsync(f.fileno())
|
||||
os.replace(tmp, FAILURES_FILE)
|
||||
|
||||
print(f"\n📊 {total} files tracked")
|
||||
print(f" New: {len(new_files)} | Modified: {len(modified_files)} | Unchanged: {skipped}")
|
||||
print(f" Enqueued: {enqueued} | Failures: {failed}")
|
||||
print(f"\n📊 {total} arquivos rastreados")
|
||||
print(f" Novos: {len(new_files)} | Modificados: {len(modified_files)} | Inalterados: {skipped}")
|
||||
print(f" Enfileirados: {enqueued} | Falhas: {failed}")
|
||||
if failures:
|
||||
print(f" 📋 Failures persisted to: {FAILURES_FILE}")
|
||||
print(f" 📋 Falhas persistidas em: {FAILURES_FILE}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
|
|
|||
126
setup/install.md
126
setup/install.md
|
|
@ -44,25 +44,33 @@ hermes status
|
|||
|
||||
### 3. Docker Infrastructure
|
||||
|
||||
The compose file lives in the `docker/` directory of this repository and must be run **in-place** — the worker build context (`./worker`) is relative to the compose file location.
|
||||
|
||||
```bash
|
||||
# Copy docker-compose.yml from this repository
|
||||
cp docker/docker-compose.yml ~/memory-os/
|
||||
cd ~/memory-os
|
||||
# Navigate to the docker directory inside your clone
|
||||
cd /path/to/memory-os/docker
|
||||
|
||||
# Create .env with required variables
|
||||
cat > .env << EOF
|
||||
# Required only for OpenRouter embedding backend; safe to leave empty for local providers
|
||||
OPENROUTER_API_KEY=sk-or-...
|
||||
REDIS_PASSWORD=$(openssl rand -hex 16)
|
||||
# Optional overrides (defaults shown)
|
||||
EMBEDDING_DIMS=4096
|
||||
COLLECTION_NAME=knowledge_base
|
||||
LOG_LEVEL=INFO
|
||||
EOF
|
||||
|
||||
# Start the stack
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
Verify:
|
||||
Verify all three services are running:
|
||||
|
||||
```bash
|
||||
docker compose ps
|
||||
# → Should show redis, qdrant, and worker all with Status: Up
|
||||
|
||||
curl -s http://localhost:6333/healthz # → {"title":"ok","version":"1.17.1"}
|
||||
redis-cli -a "$REDIS_PASSWORD" ping # → PONG
|
||||
```
|
||||
|
|
@ -97,41 +105,113 @@ ICARUS_TASK_MAX_CHARS=300
|
|||
|
||||
### 5. Core File Modifications
|
||||
|
||||
Apply the changes documented in [modifications/soul-rulebook.md](../modifications/soul-rulebook.md):
|
||||
Apply the additions documented in [setup/rulebook.md](rulebook.md) and
|
||||
[modifications/soul-rulebook.md](../modifications/soul-rulebook.md):
|
||||
|
||||
- Add Ground Truth level 2 (injected memory) to `SOUL.md`
|
||||
- Add memory architecture documentation to `rulebook.md`
|
||||
- Add context injection convention to `SOUL.md`
|
||||
**`~/.hermes/rulebook.md`** — append the Memory Architecture, Memory OS,
|
||||
and Mandatory Verifications sections from `setup/rulebook.md`.
|
||||
|
||||
These modifications ensure the agent trusts its injected memory as authoritative.
|
||||
- Each block starts with a `## Memory OS Additions — v1` marker. Before
|
||||
appending, check whether this line already exists in your rulebook —
|
||||
if it does, skip that block.
|
||||
- If `~/.hermes/rulebook.md` doesn't exist yet, create it and add the
|
||||
three marked blocks.
|
||||
|
||||
### 6. Wiki Setup
|
||||
**`SOUL.md`** — add Ground Truth level 2 (injected memory) and context
|
||||
injection convention as documented in `modifications/soul-rulebook.md`.
|
||||
|
||||
These modifications ensure the agent treats injected memory as more
|
||||
authoritative than training knowledge, and knows where to find
|
||||
persisted information without re-discovering it.
|
||||
|
||||
### 6. Wiki + Vault Setup
|
||||
|
||||
Memory OS stores its knowledge pipeline inside an Obsidian vault. The vault
|
||||
path is user-specific — set it as an environment variable first:
|
||||
|
||||
```bash
|
||||
# Set this to your Obsidian vault path
|
||||
export VAULT_PATH=/home/your-user/path/to/vault
|
||||
```
|
||||
|
||||
Create the wiki directory structure:
|
||||
|
||||
```bash
|
||||
mkdir -p $VAULT_PATH/wiki/{raw,concepts,entities,comparisons,_meta,_archive}
|
||||
# Copy SCHEMA.md template, create initial index.md and log.md
|
||||
```
|
||||
|
||||
The wiki starts empty. Add source documents to `raw/` and the wiki-agent cronjob will begin extracting structured pages.
|
||||
**What goes where:**
|
||||
- `raw/` — source documents to be ingested and curated
|
||||
- `concepts/`, `entities/`, `comparisons/` — auto-generated by vault-curator
|
||||
- `_meta/` — pipeline metadata (SCHEMA.md, indexes)
|
||||
- `_archive/` — aged-out content from decay scanner
|
||||
|
||||
### 7. Cronjobs
|
||||
The wiki starts empty. Add source documents to `raw/` and the wiki-continuous-ingest
|
||||
cronjob (step 7) will begin extracting structured pages.
|
||||
|
||||
Add to crontab (`crontab -e`):
|
||||
**Optional — Vault Curator:** For automatic enrichment, semantic linking, and
|
||||
MOC generation, install [vault-curator](https://github.com/ClaudioDrews/vault-curator)
|
||||
as a separate tool. It runs independently and is not required for Memory OS
|
||||
core functionality.
|
||||
|
||||
```cron
|
||||
# Wiki ingestion — keeps Qdrant in sync
|
||||
0 * * * * /usr/bin/python3 /path/to/scripts/wiki_continuous_ingest.py
|
||||
### 7. Maintenance Scripts
|
||||
|
||||
# Qdrant maintenance
|
||||
0 3 * * 0 /usr/bin/python3 /path/to/scripts/decay_scanner.py
|
||||
The `scripts/` directory in this repository contains the maintenance tools
|
||||
that keep the memory stack healthy. Copy them to a location of your choice
|
||||
(e.g. `~/memory-os-scripts/`) and schedule them.
|
||||
|
||||
# Dead letter queue monitoring
|
||||
0 */6 * * * /usr/bin/python3 /path/to/scripts/dlq_manager.py
|
||||
| Script | Schedule | Purpose |
|
||||
|---|---|---|
|
||||
| `wiki_continuous_ingest.py` | Hourly | Detects new/modified .md files and enqueues them to the ARQ worker |
|
||||
| `decay_scanner.py` | Weekly (Sun 3am) | Archives low-importance chunks based on age and importance_score |
|
||||
| `dlq_manager.py` | Every 6 hours | Reads, classifies, and reports dead letter queue failures |
|
||||
| `semantic_dedup.py` | Monthly (1st Sun) | Scans for near-duplicate vectors (cosine > 0.92) |
|
||||
| `backfill_decay_metadata.py` | One-shot / on-demand | Populates missing metadata (created_at, importance_score) for decay scanner |
|
||||
| `pre_validator.py` | On-demand | Semantic linter — queries knowledge_base before I/O actions |
|
||||
| `reflection_trigger.py` | Every 5 min | Triggers micro_reflection when ARQ worker is idle |
|
||||
| `bulk_wiki_ingest.py` | One-shot | Initial bulk ingestion of existing wiki content |
|
||||
|
||||
# Semantic dedup (first Sunday of month)
|
||||
0 3 * * 0 [ $(date +\%d) -le 7 ] && /usr/bin/python3 /path/to/scripts/semantic_dedup.py
|
||||
**Using Hermes cron (recommended):**
|
||||
|
||||
```bash
|
||||
hermes cron create \
|
||||
--name "wiki-continuous-ingest" \
|
||||
--schedule "0 * * * *" \
|
||||
--script /path/to/scripts/wiki_continuous_ingest.py \
|
||||
--no-agent \
|
||||
--deliver local
|
||||
|
||||
hermes cron create \
|
||||
--name "decay-scanner" \
|
||||
--schedule "0 3 * * 0" \
|
||||
--script /path/to/scripts/decay_scanner.py \
|
||||
--no-agent \
|
||||
--deliver local
|
||||
|
||||
hermes cron create \
|
||||
--name "dlq-manager" \
|
||||
--schedule "0 */6 * * *" \
|
||||
--script /path/to/scripts/dlq_manager.py \
|
||||
--no-agent \
|
||||
--deliver local
|
||||
|
||||
hermes cron create \
|
||||
--name "semantic-dedup" \
|
||||
--schedule "0 3 1 * *" \
|
||||
--script /path/to/scripts/semantic_dedup.py \
|
||||
--no-agent \
|
||||
--deliver local
|
||||
```
|
||||
|
||||
**Before enabling decay scanner:** run `backfill_decay_metadata.py` once to
|
||||
populate `created_at`, `last_accessed_at`, `importance_score`, and
|
||||
`confidence_score` on existing Qdrant points. Without backfill, the decay
|
||||
scanner will find zero eligible points.
|
||||
|
||||
**Exempting collections:** Set `DECAY_EXEMPT_PREFIXES` and
|
||||
`DEDUP_EXEMPT_PREFIXES` env vars (comma-separated prefixes) to exclude
|
||||
specific Qdrant collections from automated maintenance.
|
||||
|
||||
### 8. Gateway Restart
|
||||
|
||||
```bash
|
||||
|
|
|
|||
|
|
@ -0,0 +1,53 @@
|
|||
# Memory OS — rulebook.md additions
|
||||
|
||||
> **Version 1** — append these sections to `~/.hermes/rulebook.md`.
|
||||
|
||||
The marker `## Memory OS Additions — v1 (do not duplicate)` at the top of
|
||||
each block is an idempotency guard. Before appending, check whether this
|
||||
exact line already exists in your rulebook. If it does, skip that block.
|
||||
|
||||
---
|
||||
|
||||
## Memory OS Additions — v1 (do not duplicate)
|
||||
|
||||
## Memory Architecture
|
||||
|
||||
The agent has 6 layers of persistent memory, each with a distinct purpose:
|
||||
|
||||
| Layer | Tool | What it stores |
|
||||
|---|---|---|
|
||||
| Session | `session_search` | Past conversations (FTS5 over SQLite) |
|
||||
| Persistent | `memory` | MEMORY.md (volatile learnings) + USER.md (who the user is) |
|
||||
| Structured | `fact_store` | Durable facts with entity resolution (SQLite + HRR) |
|
||||
| Cross-agent | `fabric_*` | Session archive with structured summaries |
|
||||
| Procedural | `skill_view` / `skill_manage` | Reusable workflows |
|
||||
| Vector | Qdrant `knowledge_base` (4096d / Cosine) | Semantic search over sessions and wiki content |
|
||||
|
||||
**Fact feedback rule:** When you retrieve a fact from `fact_store` (via
|
||||
probe, search, or reason) and reference it in your response, you MUST call
|
||||
`fact_feedback` in the same turn — `action='helpful'` if the fact was
|
||||
accurate and useful, `action='unhelpful'` if it was wrong, outdated, or
|
||||
irrelevant. This is not optional. The trust scoring system depends on it.
|
||||
Without feedback, `trust_score` is ornamental and fact quality degrades
|
||||
silently.
|
||||
|
||||
## Memory OS Additions — v1 (do not duplicate)
|
||||
|
||||
## Memory Operating System (Memory OS)
|
||||
|
||||
Your memory infrastructure runs locally via Docker:
|
||||
- **Qdrant** (vector database, hybrid search: dense 4096d + BM25 sparse)
|
||||
- **Redis** (ARQ job queue for async embedding/indexing)
|
||||
- **ARQ Worker** (embedding pipeline, ingestion, decay scanning)
|
||||
|
||||
These services are defined in the Memory OS docker-compose stack.
|
||||
|
||||
## Memory OS Additions — v1 (do not duplicate)
|
||||
|
||||
## Mandatory Verifications
|
||||
|
||||
Before reporting a fact as true, verify against:
|
||||
1. **Runtime evidence** — terminal output, file existence, process status
|
||||
2. **Injected memory** — `[qdrant]`, `[fabric]`, `[sessions]`, `[facts]` in your prompt
|
||||
3. **Documentation** — man pages, official docs for installed version
|
||||
4. **Training knowledge** — never cite without verifying against 1-3
|
||||
Loading…
Reference in New Issue