xiaoxue_admin
/
memory-os
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
memory-os/scripts
History
David Soff a5c1344afb
security(hooks, context_enhancer): add dual-layer prompt injection sanitization (#5) 
security(hooks, context_enhancer): add dual-layer prompt injection sanitization

- Layer 1 (hooks.py): aggressive sanitization at prompt boundary with
  11 regex patterns + heuristic. Detects override directives, template
  injection, URI schemes, system prefixes, HTML/XML, control chars,
  zero-width Unicode, and code fences. Applied to fabric, Qdrant,
  sessions, and facts injection points. Complements existing
  _is_system_injection() (which filters ingest, not egest).

- Layer 2 (context_enhancer.py): lightweight sanitization on all
  search result content_preview outputs (hybrid, dense, sparse,
  lexical, sqlite).

- _test_sanitize.py: manual validation script with 24 test cases.

- Replacement strategy: [REDACTED] preserves audit trail and
  grammatical context instead of silent removal.
 		2026-06-02 13:43:34 -03:00
..
README.md Initial commit: Memory OS — 6-layer memory architecture for Hermes Agent 2026-05-31 16:50:37 -03:00
backfill_decay_metadata.py Initial commit: Memory OS — 6-layer memory architecture for Hermes Agent 2026-05-31 16:50:37 -03:00
bulk_wiki_ingest.py Initial commit: Memory OS — 6-layer memory architecture for Hermes Agent 2026-05-31 16:50:37 -03:00
context_enhancer.py security(hooks, context_enhancer): add dual-layer prompt injection sanitization (#5) 2026-06-02 13:43:34 -03:00
decay_scanner.py Initial commit: Memory OS — 6-layer memory architecture for Hermes Agent 2026-05-31 16:50:37 -03:00
dlq_manager.py Initial commit: Memory OS — 6-layer memory architecture for Hermes Agent 2026-05-31 16:50:37 -03:00
pre_validator.py Initial commit: Memory OS — 6-layer memory architecture for Hermes Agent 2026-05-31 16:50:37 -03:00
reflection_trigger.py Initial commit: Memory OS — 6-layer memory architecture for Hermes Agent 2026-05-31 16:50:37 -03:00
semantic_dedup.py Initial commit: Memory OS — 6-layer memory architecture for Hermes Agent 2026-05-31 16:50:37 -03:00
wiki_continuous_ingest.py Initial commit: Memory OS — 6-layer memory architecture for Hermes Agent 2026-05-31 16:50:37 -03:00

README.md

Memory OS — Scripts

Standalone Python scripts that maintain the Qdrant vector database and wiki pipeline.

Qdrant Maintenance

Script What it does Run
decay_scanner.py Archives low-importance, aged AI content based on half-life decay Weekly cron
backfill_decay_metadata.py Populates missing importance_score, last_accessed_at, confidence_score in Qdrant points Run once before enabling decay scanner
semantic_dedup.py Merges near-duplicate points (cosine >0.92) Monthly cron

Context Injection

Script What it does Used by
context_enhancer.py Embedding pipeline: query → embed → search Qdrant (4-level fallback). Also provides BM25 sparse embedding via FastEmbed. Icarus pre_llm_call hook

Wiki Pipeline

Script What it does Run
wiki_continuous_ingest.py SHA-256 diff detection: finds new/modified wiki files, enqueues ARQ jobs in Redis Hourly cron
bulk_wiki_ingest.py One-shot bulk ingestion of all wiki files into Qdrant After initial setup or collection rebuild

Quality Control

Script What it does Run
pre_validator.py Pre-flight validation of wiki documents: YAML frontmatter, required fields, link targets Before ingestion
reflection_trigger.py Idle detection for ARQ worker — enqueues micro-reflection when queue is empty and within hourly budget Every 5min cron

Monitoring

Script What it does Run
dlq_manager.py Dead letter queue monitoring and reporting Every 6h cron

Environment variables

All scripts read configuration from environment variables. See .env.example in the project root for the full reference.

Key variables:

  • OPENROUTER_API_KEY — embeddings (required)
  • WIKI_PATH — wiki root directory (default: ~/vault/wiki)
  • COLLECTION_NAME — Qdrant collection (default: knowledge_base)
  • EMBEDDING_DIMS — vector dimensions (default: 4096)
  • REDIS_PASSWORD — Redis auth (required for wiki ingest)
  • QDRANT_URL — Qdrant endpoint (default: http://localhost:6333)