Commit Graph

18 Commits

Author SHA1 Message Date
Claudio Drews e03db1f1c8
fix: expose embedding backend env vars in docker-compose and setup.sh
The worker code (embedding.py) already reads EMBEDDING_API_BASE, EMBEDDING_MODEL, and EMBEDDING_API_KEY — only the compose file was missing the plumbing. Adds the missing env vars with defaults preserving current OpenRouter/Qwen3 behavior. Also adds setup.sh entries for consistency.

Closes #29
2026-06-10 07:40:43 -03:00
@aaronjmars e654b9f5ee
fix(security): use Path.is_relative_to for WIKI_PATH containment in ingest_file
Replaces str.startswith() with Path.is_relative_to() to prevent sibling-prefix paths (e.g. /wiki-shared/x.md when WIKI_PATH=/wiki) from bypassing the entry guard. Includes regression test with 3 scenarios proving the bug exists pre-patch and is fixed post-patch.

Closes #26
 
Co-authored-by: aaronjmars <61592645+aaronjmars@users.noreply.github.com>
2026-06-10 07:20:07 -03:00
Claudio Drews d2228d5f95
fix: resolve 6 installation bugs in setup.sh and docker-compose (#28)
- Bug 1: QDRANT_API_KEY unbound variable (default to empty)
- Bug 2: Qdrant v1.17+ auth with empty key (conditional config)
- Bug 3: Redis crash with empty requirepass (conditional config)
- Bug 4: grep pattern now covers OPENROUTER_DS_API_KEY
- Bug 5: worker container healthcheck added
- Bug 7: execution-agent protocol installed during setup

Co-authored-by: ClaudioDrews <claudio@drews.com.br>
2026-06-08 21:53:15 -03:00
ClaudioDrews 7de2265c66 fix: remove stale knowledge_base_hybrid references, fix generic ENV_PATH
- file_ingestion.py: docstrings now reference 'configured collection'
  instead of hardcoded 'knowledge_base_hybrid' (code already uses
  COLLECTION_NAME env var, defaulting to knowledge_base)
- context_enhancer.py: same docstring fix
- wiki_continuous_ingest.py: ENV_PATH now points to memory-os/docker/.env
  instead of ai-stack/cognitive-agent/.env (production-specific path)
2026-06-07 14:05:31 -03:00
ClaudioDrews 59557a86f3 fix: remove leaked docker-compose-monitored.yml, add to gitignore 2026-06-07 13:32:42 -03:00
ClaudioDrews 16e394cd07 fix: align worker UID with host user (1000) for bind mount read access
Worker ran as appuser (UID 10001) but wiki files are owned by
host user hermes (UID 1000) with permissions 600. Changed
Dockerfile to create appuser with UID 1000 so the container
process can read bind-mounted wiki files.
2026-06-07 13:06:11 -03:00
ClaudioDrews 0975b26382 fix: rename docker-compose volume vars to MEMORY_OS_* prefix
B3.1 — CRITICAL: docker-compose.yml inherited host env vars
(WIKI_PATH, HERMES_HOME, FABRIC_DIR), causing dev stacks to
accidentally mount production volumes.

Fix:
- docker-compose.yml: ~/Vault/wiki →
- docker-compose.yml: ~/.hermes →
- docker-compose.yml: ~/Vault/fabric →
- .env.example: document new vars with safety warning
- install.md: add optional volume override section with ⚠️ note

Fallback defaults (./wiki, ./hermes, ./fabric) are preserved —
zero-config dev setup still works out of the box.
2026-06-03 16:24:35 -03:00
ClaudioDrews 545a882157 fix: docker-compose.yml — Qdrant healthcheck usa grep em vez de curl
A imagem qdrant/qdrant:v1.17.1 não inclui curl, wget, python, ps,
pgrep, nc nem /dev/tcp. O healthcheck anterior nunca passava porque
o binário curl não existe.

Corrigido para: sh -c 'grep -q :18BD /proc/net/tcp'
(6333 = 0x18BD)

Bug encontrado durante Nível 2 de testes (stack isolado).
2026-06-03 16:08:24 -03:00
ClaudioDrews 1422a6d583 fix: resolve 5 easy Phase 4 performance/resilience issues
V4.04: Add retry_transient() to dlq_manager — re-enables reported transient
  failures for retry with --retry CLI flag, increments retry_count/last_retry

V4.05: save_report() now truncates JSONL to MAX_REPORT_HISTORY entries
  after each append (best-effort rotation)

V4.06: Wrap datetime.fromisoformat() in try/except (ValueError, TypeError)
  in get_status_summary() — skips entries with malformed timestamps

V4.08: Replace body[:50] dedup key with hashlib.sha256(body).hexdigest()
  in fabric-retrieve.py deduplicate()

V4.09: Add OrderedDict LRU cache (256 entries) to embedding.py get_embedding()
  — avoids recomputing embeddings for repeated text
2026-06-03 11:53:18 -03:00
ClaudioDrews e5edb58a19 fix: resolve all 8 Phase 3 config/path issues
V3.01-V3.03: Replace hardcoded Path.home() / 'Vault' with os.environ.get()
  - wiki_continuous_ingest.py: WIKI_ROOT via env with fallback
  - bulk_wiki_ingest.py: WIKI_ROOT + EMBEDDING_MODEL via env with fallback
  - backfill_decay_metadata.py: VAULT_ROOT via VAULT_PATH env with fallback

V3.04-V3.05: Replace hardcoded EMBEDDING_MODEL string with os.environ.get()
  - context_enhancer.py + bulk_wiki_ingest.py

V3.06: Replace ~/.hermes til fallback with ./hermes (Docker doesn't expand ~)

V3.07: Add /fabric volume mount (~/Vault/fabric:/fabric:rw)

V3.08: Add Qdrant healthcheck + change worker qdrant dependency from
  service_started to service_healthy
2026-06-03 11:44:22 -03:00
ClaudioDrews 384395b78d fix: resolve all 7 Phase 1 setup/infra issues
V1.01: Add fastembed, httpx, redis to requirements.txt (now 10 lines, 9 deps)
V1.03: Create setup/setup_db.py — idempotent SQLite schema for state.db + memory_store.db
V1.04: Create .dockerignore (21 patterns — .env, *.key, __pycache__, etc.)
V1.05: Dockerfile COPY --chown=appuser:appuser (after useradd, before USER switch)
V1.06: Add EXPOSE 8000 + HEALTHCHECK (Redis ping, 30s interval)
V1.07: Document database setup as Step 2 in install.md (pip install + setup_db.py + table listing)
V1.08: Remove change-me from REDIS_PASSWORD in .env.example
V1.09: Replace gcc with build-essential + python3-dev in Dockerfile
2026-06-03 11:31:04 -03:00
ClaudioDrews a3d644a5c9 fix: resolve 4 code-critical issues from Fase 2 audit
V2.07 (A01): Replace subprocess.run(['arq', ...]) with programmatic
  arq.worker.run_worker(WorkerSettings). Fixes signal handling,
  graceful shutdown, and logging unification.

V2.10 (D11): Fix filename → file_path in backfill_decay_metadata.py
  resolve_timestamp() now reads payload.file_path (consistent with
  what bulk_wiki_ingest.py writes). Replaced VAULT_ROOT path
  reconstruction with direct Path(file_path).

V2.11 (D05): Auto-create Qdrant collection in bulk_wiki_ingest.py
  If collection doesn't exist, creates it with hybrid schema
  (dense 4096d Cosine + sparse BM25) instead of sys.exit(1).

V2.12 (D27): Add sparse vector generation to bulk_wiki_ingest.py
  Optional fastembed BM25 sparse vectors alongside dense embeddings.
  Falls back to dense-only if fastembed not installed. Vectors stored
  as Qdrant named vectors: {'dense': ..., 'sparse': ...}
2026-06-03 10:54:53 -03:00
ClaudioDrews ff781f9824 feat: add EMBEDDING_API_KEY for authenticated non-OpenRouter endpoints (fixes #8)
embedding.py:
- Add EMBEDDING_API_KEY env var (no fallback — explicit per-branch)
- OpenRouter branch: uses OPENROUTER_API_KEY + vendor headers
- Generic branch: sends Authorization: Bearer <EMBEDDING_API_KEY> if set
- Unauth fallback: no header (localhost, no auth needed)
- Credential isolation: no cross-branch key leakage

.env.example + setup/install.md:
- Document EMBEDDING_API_KEY as optional for non-OpenRouter
  authenticated endpoints (vLLM --api-key, custom hosted services)
2026-06-03 09:37:38 -03:00
ClaudioDrews b33b8d4196 fix: SyntaxError on QDRANT_API_KEY placeholder (fixes #10)
Split the corrupted line 15 into two valid assignments:
- QDRANT_API_KEY = os.environ.get('QDRANT_API_KEY', '')
- EMBEDDING_DIMS = int(os.environ.get('EMBEDDING_DIMS', '4096'))

The merged line 'QDRANT_API_KEY=os.env...DIMS = ...' was a leftover
placeholder that prevented the ARQ worker from starting.

Reported-by: CG1up
2026-06-03 08:18:32 -03:00
brian-doherty 02018160d2
fix(worker): add missing import os, Qdrant auth, and env config (#6)
Fix 4 bugs that prevent the ARQ worker from booting with a
Qdrant instance that has an API key set:

1. reflection.py: add missing 'import os' (NameError crash at
   COLLECTION_NAME = os.environ.get(...))

2. local_qdrant.py: read QDRANT_API_KEY from environment

3. local_qdrant.py: pass api_key= to AsyncQdrantClient and set
   https=False to prevent SSL error when API key triggers
   auto-HTTPS against a local HTTP Qdrant instance

4. docker-compose.yml: pass QDRANT_API_KEY env var to the
   worker service so the key is available inside the container
2026-06-02 13:21:30 -03:00
Claudio Drews 8e8ea95458
fix(embedding): make OpenRouter auth and headers conditional — unblock 100% local usage (#7)
- Remove unconditional OPENROUTER_API_KEY check that blocked local providers
  (Ollama, vLLM, llama.cpp) from running without an API key
- Send OpenRouter-specific headers (HTTP-Referer, X-Title) only when
  EMBEDDING_API_BASE contains 'openrouter'
- Stop sending empty Authorization header to non-OpenRouter endpoints
- Update .env.example and setup/install.md to clarify that
  OPENROUTER_API_KEY is only required for OpenRouter

Closes #1

Co-authored-by: ClaudioDrews <claudio@drews.com.br>
2026-06-02 13:05:12 -03:00
Claudio Drews 1281d7a778 fix: make embedding provider-agnostic via env vars
- embedding.py: EMBEDDING_API_BASE and EMBEDDING_MODEL now read from env
  with OpenRouter defaults (preserves backward compatibility)
- .env.example: document embedding backend config with Qwen3 rationale
- setup/install.md: update prerequisite to reflect configurable backend
- layers/05-qdrant.md: add "Why Qwen3-Embedding-8B" section
  (multilingual, quality, speed, cost)

Closes #1
2026-06-02 09:45:31 -03:00
ClaudioDrews 0b32ffcfc2 Initial commit: Memory OS — 6-layer memory architecture for Hermes Agent 2026-05-31 16:50:37 -03:00