The worker code (embedding.py) already reads EMBEDDING_API_BASE, EMBEDDING_MODEL, and EMBEDDING_API_KEY — only the compose file was missing the plumbing. Adds the missing env vars with defaults preserving current OpenRouter/Qwen3 behavior. Also adds setup.sh entries for consistency.
Closes#29
Replaces str.startswith() with Path.is_relative_to() to prevent sibling-prefix paths (e.g. /wiki-shared/x.md when WIKI_PATH=/wiki) from bypassing the entry guard. Includes regression test with 3 scenarios proving the bug exists pre-patch and is fixed post-patch.
Closes#26
Co-authored-by: aaronjmars <61592645+aaronjmars@users.noreply.github.com>
- file_ingestion.py: docstrings now reference 'configured collection'
instead of hardcoded 'knowledge_base_hybrid' (code already uses
COLLECTION_NAME env var, defaulting to knowledge_base)
- context_enhancer.py: same docstring fix
- wiki_continuous_ingest.py: ENV_PATH now points to memory-os/docker/.env
instead of ai-stack/cognitive-agent/.env (production-specific path)
Adds explicit examples of bad vs. good Match execution to
prevent performative compliance — the model producing vague
statements like 'Match: relevante' that satisfy the checklist
syntactically but provide zero information.
Also adds a quality rule: Match statements must cite at least
one concrete detail from the injected content so a human can
identify which block was read and what it contains.
Qdrant v1.17+ enables authentication when QDRANT__SERVICE__API_KEY
env var is set (even to empty string). Since the default in setup.sh
is no Qdrant key (localhost-only bind), we now conditionally remove
the line from docker-compose.yml when QDRANT_API_KEY is unset.
- setup.sh Phase 7b: install cron job for wiki_continuous_ingest.py
- watcher script: fallback to Docker .env for REDIS_PASSWORD in cron context
- Cron runs every hour with idempotent marker check
Worker ran as appuser (UID 10001) but wiki files are owned by
host user hermes (UID 1000) with permissions 600. Changed
Dockerfile to create appuser with UID 1000 so the container
process can read bind-mounted wiki files.
- score_all() now multiplies amplify_gain by base salience before
applying Hebbian cross-source corroboration boost, so globally-
important structural facts no longer crowd out query-locally-
relevant facts
- ICARUS_COLLAPSE_AMPLIFY_GAIN env var (default 0.15) tunes the
tradeoff; raise to 0.20 if genuine corroboration feels undervalued
- All tests passing (_test_collapse.py, collapse_eval.py)
- Real pipeline test: 30-35% token reduction with correct ranking
Closes#23
Co-authored-by: ClaudioDrews <claudio@drews.com.br>
Detects whether the four required Memory OS SOUL.md sections are present:
- Ground Truth Hierarchy (layer 7)
- Context Injection Convention
- Fact Feedback Rule
- Honcho Deprecation Notice
Case-insensitive keyword matching. Exits 0 if all pass, 1 otherwise.
Without these sections, injected memory is treated as external sources
to cite rather than prior knowledge — causing the 'internal mechanics leak'
behavior reported in #19.
m.group(1) captures 'f95842e2' (with quotes) from YAML id: field.
Comparison with unquoted entry_id always failed → no entry ever found.
Added .strip('"') after .strip() to normalize before comparison.
D01 — parse_entry YAML fallback was duplicated and buggy (split ': '
breaks on URLs). Primary parser (yaml.safe_load) already handled
URLs correctly; manual fallback is now centralized in one module.
D09 — export-training.py had a duplicate parse_entry with the same
bug. Now imports from parsing.py.
Both files use try/except ImportError to support both package-mode
(icarus plugin) and standalone execution (CLI scripts).
Shared module returns both naming conventions (_body/body, _file/file)
for backward compatibility with fabric-retrieve and export-training.
B3.1 — CRITICAL: docker-compose.yml inherited host env vars
(WIKI_PATH, HERMES_HOME, FABRIC_DIR), causing dev stacks to
accidentally mount production volumes.
Fix:
- docker-compose.yml: ~/Vault/wiki →
- docker-compose.yml: ~/.hermes →
- docker-compose.yml: ~/Vault/fabric →
- .env.example: document new vars with safety warning
- install.md: add optional volume override section with ⚠️ note
Fallback defaults (./wiki, ./hermes, ./fabric) are preserved —
zero-config dev setup still works out of the box.
A imagem qdrant/qdrant:v1.17.1 não inclui curl, wget, python, ps,
pgrep, nc nem /dev/tcp. O healthcheck anterior nunca passava porque
o binário curl não existe.
Corrigido para: sh -c 'grep -q :18BD /proc/net/tcp'
(6333 = 0x18BD)
Bug encontrado durante Nível 2 de testes (stack isolado).
Collection knowledge_base usa named vectors (dense + sparse). O Qdrant
exige o parâmetro 'using' em collections com múltiplos vetores nomeados.
Sem ele, query_batch_points retornava 'Not existing vector name error'
em 100% das batches (44/44). O script reportava falsos 0 near-duplicates.
Correção: 1 linha — using='dense' em models.QueryRequest (linha 176).
Validado com --dry-run --threshold 0.95 contra 8886 pontos reais:
0 erros, 1776 near-duplicate pairs detectados.
R3: .env.example — Embedding backend reestruturado com Option A (Ollama,
recomendado para produção) e Option B (OpenRouter). install.md passo 4
atualizado para mencionar Ollama como opção local.
R4: install.md passo 8 — adicionados 3 cron jobs stack-base na tabela
e nos exemplos CLI: holographic-memory-backup (backup semanal do
memory_store.db), wiki-raw-ingest-monitor (detecção de drift em raw/),
maas-heartbeat (healthcheck Qdrant/Redis/ARQ).
R5: .env.example + install.md passo 6 — adicionado HERMES_AGENT_NAME.
state.py write_entry agora loga warning quando AGENT_NAME não está
definido.
R6: install.md Troubleshooting — nota sobre coexistência de coleções
Qdrant (knowledge_base + outras coleções de agentes externos).
R1: setup/rulebook.md reescrito como pointer para o novo arquivo
modifications/execution-agent-protocol.md, que formata as 3 seções
(Memory Architecture, Memory OS Docker stack, Mandatory Verifications)
como amendments ao Execution Agent protocol. Cada amendment especifica
onde inserir relativo às regras do protocolo. install.md passo 6
atualizado para refletir o novo formato.
R2: modifications/soul-rulebook.md ganhou duas novas seções:
- Fact feedback rule (fecha o loop de qualidade dos fatos)
- Honcho deprecation (avisa para não configurar Honcho)
R7 (V5.03/B11): hooks.py _search_qdrant — os.environ mutation agora usa
try/finally com restore. Chave OPENROUTER_API_KEY é injetada
temporariamente só durante o import do context_enhancer.
R8 (V5.06/B06): state.py write_entry e write_memory_file — substituído
path.write_text() por tmp + rename (escrita atômica). Evita corrupção
de arquivo em caso de crash durante o write.
R9 (item 5): state.py read_pending — normalizado agent = AGENT_NAME or
'agent' para consistência com write_entry. Corrige lógica de assignment
quando HERMES_AGENT_NAME não está definido.
Make the Icarus extraction pipeline work with any LLM provider
instead of hardcoding OpenRouter.
New env vars (all optional, backward compatible):
DEEPSEEK_API_KEY — route directly to api.deepseek.com
ICARUS_ENDPOINT — fully custom v1/chat/completions URL
ICARUS_API_KEY_ENV — env var name for the custom endpoint key
Resolution priority:
1. ICARUS_ENDPOINT + ICARUS_API_KEY_ENV (fully custom)
2. DEEPSEEK_API_KEY → api.deepseek.com
3. OPENROUTER_API_KEY → openrouter.ai (existing behaviour, unchanged)
Model names are automatically normalized: OpenRouter-style
deepseek/deepseek-v4-flash slugs have the provider prefix
stripped for direct API calls.
No breaking changes. If DEEPSEEK_API_KEY and ICARUS_ENDPOINT
are both unset, behaviour is identical to previous versions.
V4.01: Replace O(n²) brute-force cosine similarity in semantic_dedup.py
with Qdrant query_batch_points() — delegates nearest-neighbor search
to the native HNSW index. Batch size auto-scales (min 200, adjusted
for collection size). Self-match excluded server-side via
HasIdCondition filter. Graceful fallback if qdrant-client not installed.
V4.03: Replace glob+parse in fabric-retrieve.py retrieve() with SQLite
fabric_index + fabric_fts (FTS5). _ensure_fabric_index() rebuilds the
index lazily when any fabric .md mtime exceeds the stored max mtime.
Transactional rebuild (BEGIN; DELETE; DELETE; ...; COMMIT) prevents
index corruption on partial failure. All score_entry fields preserved
as columns — no YAML parsing after initial index build.
V4.02: Replace glob+parse scan in _find_entry_file with in-memory dict index
- _build_entry_index() scans fabric dirs once, builds (agent,id)→stem dict
- Index rebuilds lazily when fabric_dir.stat().st_mtime changes
- O(n) rebuild only on changes, O(1) per lookup thereafter
V4.07: Replace 4× O(n) linear scans in _resolve_ref with prebuilt indices
- index_by_id, index_by_cycle: exact dict lookups (covers ~90% of calls)
- by_agent_entries: scoped substring scans (per-agent, not full list)
- Slow path preserved for standalone calls without indices
- All 4 call sites in build_pairs() updated to pass indices
V4.04: Add retry_transient() to dlq_manager — re-enables reported transient
failures for retry with --retry CLI flag, increments retry_count/last_retry
V4.05: save_report() now truncates JSONL to MAX_REPORT_HISTORY entries
after each append (best-effort rotation)
V4.06: Wrap datetime.fromisoformat() in try/except (ValueError, TypeError)
in get_status_summary() — skips entries with malformed timestamps
V4.08: Replace body[:50] dedup key with hashlib.sha256(body).hexdigest()
in fabric-retrieve.py deduplicate()
V4.09: Add OrderedDict LRU cache (256 entries) to embedding.py get_embedding()
— avoids recomputing embeddings for repeated text
get_sparse_vector was accidentally inserted between get_embedding's
signature and its body, disconnecting the embedding call and leaving
module-level code referencing undefined variables. Moved to standalone
function before get_embedding.
V2.07 (A01): Replace subprocess.run(['arq', ...]) with programmatic
arq.worker.run_worker(WorkerSettings). Fixes signal handling,
graceful shutdown, and logging unification.
V2.10 (D11): Fix filename → file_path in backfill_decay_metadata.py
resolve_timestamp() now reads payload.file_path (consistent with
what bulk_wiki_ingest.py writes). Replaced VAULT_ROOT path
reconstruction with direct Path(file_path).
V2.11 (D05): Auto-create Qdrant collection in bulk_wiki_ingest.py
If collection doesn't exist, creates it with hybrid schema
(dense 4096d Cosine + sparse BM25) instead of sys.exit(1).
V2.12 (D27): Add sparse vector generation to bulk_wiki_ingest.py
Optional fastembed BM25 sparse vectors alongside dense embeddings.
Falls back to dense-only if fastembed not installed. Vectors stored
as Qdrant named vectors: {'dense': ..., 'sparse': ...}
Split the corrupted line 15 into two valid assignments:
- QDRANT_API_KEY = os.environ.get('QDRANT_API_KEY', '')
- EMBEDDING_DIMS = int(os.environ.get('EMBEDDING_DIMS', '4096'))
The merged line 'QDRANT_API_KEY=os.env...DIMS = ...' was a leftover
placeholder that prevented the ARQ worker from starting.
Reported-by: CG1up
- Silent removal (empty string) left grammatically broken sentences
and no audit trail. [REDACTED] preserves context structure.
- Control chars and zero-width Unicode keep empty string replacement
(invisible characters, removal is correct).
- Updated 4 test assertions in _test_sanitize.py.
fix(context_enhancer): refactor embed_query_sparse to use stdin instead of f-string interpolation
- Eliminates shlex.quote() crash with apostrophes (SyntaxError in subprocess)
- Eliminates user text interpolation into Python code string
- Hardcodes BM25 model name (was already hardcoded module-level)
- Corrects FastEmbed API usage: model.embed([query]) instead of model.embed(string)
Fix 4 bugs that prevent the ARQ worker from booting with a
Qdrant instance that has an API key set:
1. reflection.py: add missing 'import os' (NameError crash at
COLLECTION_NAME = os.environ.get(...))
2. local_qdrant.py: read QDRANT_API_KEY from environment
3. local_qdrant.py: pass api_key= to AsyncQdrantClient and set
https=False to prevent SSL error when API key triggers
auto-HTTPS against a local HTTP Qdrant instance
4. docker-compose.yml: pass QDRANT_API_KEY env var to the
worker service so the key is available inside the container
- Remove unconditional OPENROUTER_API_KEY check that blocked local providers
(Ollama, vLLM, llama.cpp) from running without an API key
- Send OpenRouter-specific headers (HTTP-Referer, X-Title) only when
EMBEDDING_API_BASE contains 'openrouter'
- Stop sending empty Authorization header to non-OpenRouter endpoints
- Update .env.example and setup/install.md to clarify that
OPENROUTER_API_KEY is only required for OpenRouter
Closes#1
Co-authored-by: ClaudioDrews <claudio@drews.com.br>
Context injection delivers memory into the prompt, but without an explicit
Ground Truth hierarchy the agent treats it as optional suggestion. This
layer documents the fix: injected memory ([qdrant], [fabric], [sessions],
[facts]) ranked as authoritative for documented knowledge, with clear
conflict-resolution rules against terminal output and training knowledge.
Claudio Drews
@aaronjmars
ClaudioDrews
Brian Doherty
David Soff
brian-doherty