xiaoxue_admin
/
memory-os
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
memory-os/scripts
History
ClaudioDrews 144795bf74 perf: Qdrant query_batch_points + SQLite FTS5 for semantic_dedup and fabric-retrieve 
V4.01: Replace O(n²) brute-force cosine similarity in semantic_dedup.py
  with Qdrant query_batch_points() — delegates nearest-neighbor search
  to the native HNSW index. Batch size auto-scales (min 200, adjusted
  for collection size). Self-match excluded server-side via
  HasIdCondition filter. Graceful fallback if qdrant-client not installed.

V4.03: Replace glob+parse in fabric-retrieve.py retrieve() with SQLite
  fabric_index + fabric_fts (FTS5). _ensure_fabric_index() rebuilds the
  index lazily when any fabric .md mtime exceeds the stored max mtime.
  Transactional rebuild (BEGIN; DELETE; DELETE; ...; COMMIT) prevents
  index corruption on partial failure. All score_entry fields preserved
  as columns — no YAML parsing after initial index build.
 		2026-06-03 12:14:30 -03:00
..
README.md Initial commit: Memory OS — 6-layer memory architecture for Hermes Agent 2026-05-31 16:50:37 -03:00
backfill_decay_metadata.py fix: resolve all 8 Phase 3 config/path issues 2026-06-03 11:44:22 -03:00
bulk_wiki_ingest.py fix: resolve all 8 Phase 3 config/path issues 2026-06-03 11:44:22 -03:00
context_enhancer.py fix: resolve all 8 Phase 3 config/path issues 2026-06-03 11:44:22 -03:00
decay_scanner.py docs: rewrite installation guide with real scripts and idempotent modifications (fixes #11) 2026-06-03 09:29:19 -03:00
dlq_manager.py fix: resolve 5 easy Phase 4 performance/resilience issues 2026-06-03 11:53:18 -03:00
pre_validator.py docs: rewrite installation guide with real scripts and idempotent modifications (fixes #11) 2026-06-03 09:29:19 -03:00
reflection_trigger.py docs: rewrite installation guide with real scripts and idempotent modifications (fixes #11) 2026-06-03 09:29:19 -03:00
semantic_dedup.py perf: Qdrant query_batch_points + SQLite FTS5 for semantic_dedup and fabric-retrieve 2026-06-03 12:14:30 -03:00
wiki_continuous_ingest.py fix: resolve all 8 Phase 3 config/path issues 2026-06-03 11:44:22 -03:00

README.md

Memory OS — Scripts

Standalone Python scripts that maintain the Qdrant vector database and wiki pipeline.

Qdrant Maintenance

Script What it does Run
decay_scanner.py Archives low-importance, aged AI content based on half-life decay Weekly cron
backfill_decay_metadata.py Populates missing importance_score, last_accessed_at, confidence_score in Qdrant points Run once before enabling decay scanner
semantic_dedup.py Merges near-duplicate points (cosine >0.92) Monthly cron

Context Injection

Script What it does Used by
context_enhancer.py Embedding pipeline: query → embed → search Qdrant (4-level fallback). Also provides BM25 sparse embedding via FastEmbed. Icarus pre_llm_call hook

Wiki Pipeline

Script What it does Run
wiki_continuous_ingest.py SHA-256 diff detection: finds new/modified wiki files, enqueues ARQ jobs in Redis Hourly cron
bulk_wiki_ingest.py One-shot bulk ingestion of all wiki files into Qdrant After initial setup or collection rebuild

Quality Control

Script What it does Run
pre_validator.py Pre-flight validation of wiki documents: YAML frontmatter, required fields, link targets Before ingestion
reflection_trigger.py Idle detection for ARQ worker — enqueues micro-reflection when queue is empty and within hourly budget Every 5min cron

Monitoring

Script What it does Run
dlq_manager.py Dead letter queue monitoring and reporting Every 6h cron

Environment variables

All scripts read configuration from environment variables. See .env.example in the project root for the full reference.

Key variables:

  • OPENROUTER_API_KEY — embeddings (required)
  • WIKI_PATH — wiki root directory (default: ~/vault/wiki)
  • COLLECTION_NAME — Qdrant collection (default: knowledge_base)
  • EMBEDDING_DIMS — vector dimensions (default: 4096)
  • REDIS_PASSWORD — Redis auth (required for wiki ingest)
  • QDRANT_URL — Qdrant endpoint (default: http://localhost:6333)