- file_ingestion.py: docstrings now reference 'configured collection'
instead of hardcoded 'knowledge_base_hybrid' (code already uses
COLLECTION_NAME env var, defaulting to knowledge_base)
- context_enhancer.py: same docstring fix
- wiki_continuous_ingest.py: ENV_PATH now points to memory-os/docker/.env
instead of ai-stack/cognitive-agent/.env (production-specific path)
- setup.sh Phase 7b: install cron job for wiki_continuous_ingest.py
- watcher script: fallback to Docker .env for REDIS_PASSWORD in cron context
- Cron runs every hour with idempotent marker check
- score_all() now multiplies amplify_gain by base salience before
applying Hebbian cross-source corroboration boost, so globally-
important structural facts no longer crowd out query-locally-
relevant facts
- ICARUS_COLLAPSE_AMPLIFY_GAIN env var (default 0.15) tunes the
tradeoff; raise to 0.20 if genuine corroboration feels undervalued
- All tests passing (_test_collapse.py, collapse_eval.py)
- Real pipeline test: 30-35% token reduction with correct ranking
Closes#23
Co-authored-by: ClaudioDrews <claudio@drews.com.br>
Detects whether the four required Memory OS SOUL.md sections are present:
- Ground Truth Hierarchy (layer 7)
- Context Injection Convention
- Fact Feedback Rule
- Honcho Deprecation Notice
Case-insensitive keyword matching. Exits 0 if all pass, 1 otherwise.
Without these sections, injected memory is treated as external sources
to cite rather than prior knowledge — causing the 'internal mechanics leak'
behavior reported in #19.
Collection knowledge_base usa named vectors (dense + sparse). O Qdrant
exige o parâmetro 'using' em collections com múltiplos vetores nomeados.
Sem ele, query_batch_points retornava 'Not existing vector name error'
em 100% das batches (44/44). O script reportava falsos 0 near-duplicates.
Correção: 1 linha — using='dense' em models.QueryRequest (linha 176).
Validado com --dry-run --threshold 0.95 contra 8886 pontos reais:
0 erros, 1776 near-duplicate pairs detectados.
V4.01: Replace O(n²) brute-force cosine similarity in semantic_dedup.py
with Qdrant query_batch_points() — delegates nearest-neighbor search
to the native HNSW index. Batch size auto-scales (min 200, adjusted
for collection size). Self-match excluded server-side via
HasIdCondition filter. Graceful fallback if qdrant-client not installed.
V4.03: Replace glob+parse in fabric-retrieve.py retrieve() with SQLite
fabric_index + fabric_fts (FTS5). _ensure_fabric_index() rebuilds the
index lazily when any fabric .md mtime exceeds the stored max mtime.
Transactional rebuild (BEGIN; DELETE; DELETE; ...; COMMIT) prevents
index corruption on partial failure. All score_entry fields preserved
as columns — no YAML parsing after initial index build.
V4.04: Add retry_transient() to dlq_manager — re-enables reported transient
failures for retry with --retry CLI flag, increments retry_count/last_retry
V4.05: save_report() now truncates JSONL to MAX_REPORT_HISTORY entries
after each append (best-effort rotation)
V4.06: Wrap datetime.fromisoformat() in try/except (ValueError, TypeError)
in get_status_summary() — skips entries with malformed timestamps
V4.08: Replace body[:50] dedup key with hashlib.sha256(body).hexdigest()
in fabric-retrieve.py deduplicate()
V4.09: Add OrderedDict LRU cache (256 entries) to embedding.py get_embedding()
— avoids recomputing embeddings for repeated text
get_sparse_vector was accidentally inserted between get_embedding's
signature and its body, disconnecting the embedding call and leaving
module-level code referencing undefined variables. Moved to standalone
function before get_embedding.
V2.07 (A01): Replace subprocess.run(['arq', ...]) with programmatic
arq.worker.run_worker(WorkerSettings). Fixes signal handling,
graceful shutdown, and logging unification.
V2.10 (D11): Fix filename → file_path in backfill_decay_metadata.py
resolve_timestamp() now reads payload.file_path (consistent with
what bulk_wiki_ingest.py writes). Replaced VAULT_ROOT path
reconstruction with direct Path(file_path).
V2.11 (D05): Auto-create Qdrant collection in bulk_wiki_ingest.py
If collection doesn't exist, creates it with hybrid schema
(dense 4096d Cosine + sparse BM25) instead of sys.exit(1).
V2.12 (D27): Add sparse vector generation to bulk_wiki_ingest.py
Optional fastembed BM25 sparse vectors alongside dense embeddings.
Falls back to dense-only if fastembed not installed. Vectors stored
as Qdrant named vectors: {'dense': ..., 'sparse': ...}
- Silent removal (empty string) left grammatically broken sentences
and no audit trail. [REDACTED] preserves context structure.
- Control chars and zero-width Unicode keep empty string replacement
(invisible characters, removal is correct).
- Updated 4 test assertions in _test_sanitize.py.
fix(context_enhancer): refactor embed_query_sparse to use stdin instead of f-string interpolation
- Eliminates shlex.quote() crash with apostrophes (SyntaxError in subprocess)
- Eliminates user text interpolation into Python code string
- Hardcodes BM25 model name (was already hardcoded module-level)
- Corrects FastEmbed API usage: model.embed([query]) instead of model.embed(string)
ClaudioDrews
Claudio Drews
David Soff