Commit Graph
1588 Commits
Author SHA1 Message Date
Fergus ChingandClaude Opus 4.7 96e3be9fe2 fix(convo_miner): wrap mine_convos in mine_palace_lock to stop pile-up
The project-files mine path (miner.mine) has wrapped _mine_impl in
mine_palace_lock since #1264 — a non-blocking flock that raises
MineAlreadyRunning so the second runner exits cleanly instead of
queueing as a waiter that drives parallel HNSW inserts. The convos
mine path (convo_miner.mine_convos) was missing the same guard.

In practice this meant any caller that spawned `mempalace mine
--mode convos` repeatedly against the same palace — most notably
the Stop-hook transcript ingest before the per-target PID slot
landed — could stack up arbitrarily many concurrent mines, each
holding a ChromaDB client open, each writing to the same HNSW
index. Recently observed: 28 stuck convos mines on one machine
consuming ~18 GB of RAM and contributing to a load spike.

Fix: refactor mine_convos into a thin wrapper that holds the
per-palace flock around _mine_convos_impl, mirroring miner.mine
exactly. Dry-run skips the lock since it never writes.

Tests: two cross-process tests in tests/test_convo_miner.py —
one asserts MineAlreadyRunning when a child process holds the
lock, one asserts dry-run is unaffected. Same spawn-context
pattern as test_palace_locks.py (fork-with-chromadb deadlocks
on Python 3.13).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 12:47:36 +00:00
jp 74cd1c3751 perf(convo_miner): bulk pre-fetch already-mined set instead of N WHERE queries
mine_convos was calling file_already_mined() once per file inside the
main loop. On a 150k-drawer palace, each per-file query
(`collection.get(where={"source_file": X}, limit=1)`) costs ~2 seconds
because chromadb has to scan the metadata index. A 2000-transcript
directory took >1h of wall-clock just to decide every file should be
skipped — and pegged multiple cores doing so, starving the daemon's
other endpoints.

bulk_check_mined() already existed for exactly this anti-pattern (its
docstring says "Callers can check membership and compare mtimes locally
instead of issuing one ChromaDB query per file") but only the project
miner used it; the convo miner kept the slow per-file path.

This patch adds a third helper, prefetch_mined_set(), that mirrors
file_already_mined()'s version-gate semantics (the check_mtime=False
branch used by mine_convos) and returns a set[str] for O(1) lookups.
mine_convos now calls it once before the loop; the loop body becomes
a set-membership check.

Observed on a 172k-drawer palace probing 10 files:
  before:  21.2s (2.12s/file)
  after:   single bulk pass should be 30–60s for the whole 172k scan,
           then 2000 O(1) checks ≈ free

file_already_mined() is kept for callers that genuinely need the
per-file semantics (the post-lock race-check in _file_chunks_locked
at convo_miner.py:350 still uses it intentionally).

Surfaced via jphein/familiar.realm.watch foundation-rework debugging
on 2026-05-11. Originally filed as #51.
2026-05-11 18:58:31 -07:00
jpandClaude Opus 4.7 0a1adf9962 test(repair): unit coverage for empty-metadata sanitization in _extract_drawers
Addresses @Copilot's review feedback on #1459. Five tests:

  - test_extract_drawers_preserves_valid_metadata: non-empty dict
    passes through unchanged (regression guard against breaking happy
    path).
  - test_extract_drawers_sanitizes_none_metadata: None entries
    coerce to {"_repaired_empty_meta": True} (the core fix).
  - test_extract_drawers_sanitizes_empty_dict_metadata: empty dict
    {} entries also coerce to the sentinel (chromadb 1.5.x rejects
    both shapes equally).
  - test_extract_drawers_sanitization_preserves_alignment: critical
    invariant — ids[i] / documents[i] / metadatas[i] stay in
    lockstep through the sanitizer; mis-pairing would silently
    corrupt rebuilds.
  - test_extract_drawers_multiple_batches: pagination boundary
    correctness (sanitizer applied per-batch, no drops/duplicates).

Verified passing locally against mempalace fork main + chromadb
1.5.8 in the palace-daemon venv (5 passed, 67 deselected in 1.88s).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 14:17:07 -07:00
jpandClaude Opus 4.7 5d95656b6b fix(backends/chroma): catch-all metadata sanitizer in add/upsert
Belt-and-suspenders on top of the repair.py sanitization in the
previous commit. A 151,478-drawer rebuild test still failed at
~120K with the same `ValueError: Expected metadata to be a
non-empty dict` from chromadb, even with the repair.py sanitizers
in place. Traceback:

    mempalace/backends/chroma.py:add → chromadb Collection.add
    → validate_insert_record_set → validate_metadatas
    → validate_metadata → ValueError

Likely cause: chromadb's `upsert()` internally calls `add()` for
new records, and somewhere between repair.py's batch upsert and
chromadb's final write, the metadatas list gets reprocessed in a
way that re-introduces empty/None entries.

Sanitizing at the chromadb-client chokepoint catches everything:
no caller can leak bad metadata regardless of upstream sanitization
state. Same `{"_repaired_empty_meta": True}` sentinel, searchable
via `where={"_repaired_empty_meta": True}`.

Cost: one list comprehension per write call; negligible vs the
embedding + HNSW work each upsert already does.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 12:10:48 -07:00
jpandClaude Opus 4.7 a26e465894 fix(repair): coerce empty metadata to sentinel in both rebuild paths
Two functions construct the metadatas[] list that gets fed to
chromadb's upsert/add, both vulnerable to the same ValueError:

  ValueError: Expected metadata to be a non-empty dict, got 0
  metadata attributes in add.

chromadb 1.5.x's validate_metadata rejects both `None` and `{}`
entries — see chromadb/api/types.py:validate_metadata (line ~1071).

This commit patches both:

1. `_extract_drawers` (line ~131) — the chromadb-collection-based
   extractor, used when the source palace's collection is openable
   via the chromadb client. Sanitizes None/{} entries in the
   `batch["metadatas"]` list to `{"_repaired_empty_meta": True}`
   before extending `all_metas`.

2. `_rebuild_one_collection` (line ~813) — the SQLite-direct
   extract path used by `rebuild_from_sqlite()`, invoked when the
   source palace can't be opened via chromadb (the recovery path
   for palaces with corrupt HNSW segments). Old code was:

       metas.append(meta if meta else {})

   The trailing `{}` was the bug; chromadb 1.5.x rejects empty
   dicts the same as None. Replaced with the same sentinel.

Why `_repaired_empty_meta: True` as the sentinel:
  - Satisfies chromadb's non-empty-dict requirement
  - Bool-valued (valid chromadb metadata type, trivially serializable)
  - Namespaced + descriptive so an operator can find which drawers
    were coerced via `where={"_repaired_empty_meta": True}` later
  - Idempotent on re-runs (a future repair over a sanitized palace
    sees the sentinel as already-valid)

Verified on a 151,478-drawer production palace that previously
crashed at drawer 120,000 in both extract paths.

Fixes #1458

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 11:53:57 -07:00
fatkobra bc7fe88846 Add regression tests for zero-byte HNSW link lists 2026-05-11 19:06:05 +02:00
fatkobra a1d2483145 Reject zero-byte HNSW link lists with payload 2026-05-11 18:52:35 +02:00
Igor Lins e Silva 68319dc0d0 Merge pull request #1450 from workblac/fix/benchmark-embedding-model-preflight
fix(benchmarks): auto-pull embedding model when missing instead of silently scoring 0
2026-05-11 06:37:13 -03:00
Bruno Leal 3fbe930751 fix(benchmarks): auto-pull embedding model when missing instead of silently scoring 0
Tasks that score via semantic similarity (memory_extraction,
room_classification open-set) depend on nomic-embed-text. If the model
was absent, embed_text() returned None silently and every sample scored
0.0 — indistinguishable from a genuinely bad model.

_ensure_embed_model() now probes the endpoint before the warmup step.
On a 404 it pulls the model automatically via `ollama pull`, verifies
the endpoint responds, and only then proceeds. Fixes:

- OLLAMA_HOST is now forwarded to `ollama pull` so non-localhost
  endpoints are targeted correctly
- FileNotFoundError is caught and re-raised as RuntimeError with an
  install hint if ollama is not on PATH
- Diagnostic prints go to stderr, not stdout
- If the pull or re-probe fails, returns a Result with a clear error
  message instead of silent zeros

Restores --candidates local to its documented meaning (all non-cloud
models). Adds --candidates community for third-party GGUF variants.
2026-05-11 06:29:49 -03:00
Igor Lins e Silva fa143d2b06 Merge pull request #1447 from MemPalace/feat/model-eval-harness
Small-model evaluation harness for MemPalace classification tasks
2026-05-11 04:39:19 -03:00
Igor Lins e Silva 42f52c8b71 fix(benchmarks): address copilot-pull-request-reviewer feedback on PR #1447
Ten fixes across code, documentation, and committed baselines:

Code fixes:
1. prompts.py: closed-set prompt said "no punctuation" while room
   slugs contain "/". Replaced with explicit "copy the slug verbatim
   including / and -" wording. Empirical impact on accuracy is within
   reproducibility noise (qwen3:4b q4 was 0.610, retest shows 0.594
   on the now-101-sample dataset — same vs new sample noise).
2. orchestrator.py: orchestrator now opens the CSV once, writes the
   header, and flushes after each row. README's "writes incrementally,
   safe to Ctrl-C" claim is now accurate. Long matrix runs no longer
   lose all progress on a crash.
3. metrics.py: gather_host_info() now strips trailing newline from
   `hostname` command output. Stops embedded newlines from leaking
   into committed CSVs.
4. runner.py: passed `endpoint=endpoint` to vram_resident_mb() so
   non-default Ollama endpoints are queried correctly.
5. summarize.py: CSV reader and report writer now use
   encoding="utf-8" with explicit newline handling for cross-platform
   consistency.

Documentation fixes:
6. entity_extraction/score.py: docstring claimed "case-sensitive on
   type" while implementation lowercases both. Corrected.
7. dataset.jsonl rc_101: removed mempalace-private URL since the PR
   lands on the public repo. Will be linkable after OpenClaw merges.
8. datasets/README.md: sample counts updated (room_classification 100
   → 101, total 210 → 211) to match the cherry-picked real-format
   sample.
9. runner.py / orchestrator.py: docstring example paths corrected
   (tests/fixtures/mempalace_bench → benchmarks/model_eval/datasets).

Baseline regeneration:
10. Stripped embedded \n from `host` column in all five committed
    CSVs (2026-05-10 main local + spotcheck, 2026-05-10 + 2026-05-11
    cloud, 2026-05-11 modern). 165 rows cleaned in place via Python
    csv reader/writer. Each row is now line-oriented for diff and
    grep friendliness without changing any measured values.
2026-05-11 03:46:35 -03:00
Igor Lins e Silva 152fbb6afc docs(benchmarks): explicit invitation for backend + model contributions
Adds a "Contributions welcome" section to the README covering:

1. Adding models we missed. The published candidate list is one
   engineer + one search pass. Particularly call out function-calling-
   tuned variants (Phi-4 mini, Nemotron-mini), research-lab instruct
   models (Hermes, Dolphin), and quantization-aware-trained variants
   as worth-adding categories.

2. Adding other inference backends. mempalace.llm_client.get_provider
   already supports openai-compat, which means LM Studio, llama.cpp
   server, vLLM, unsloth studio, Docker Model Runner, HF TGI all
   plug in with minimal harness changes. Documents the four pieces a
   backend contributor would need to touch:
   - Backend-selection flag in runner/orchestrator
   - Backend-specific timing extraction in metrics.py
   - Backend-specific VRAM probe
   - candidates.yaml backend field

   Notes that accuracy numbers stay comparable across backends since
   the dataset and scoring code are backend-agnostic.

   Also calls out niche runtimes worth cross-comparison: Apple MLX,
   Intel OpenVINO, AMD ROCm, edge-device runtimes.

Existing harness-maintainer notes (cloud structured-outputs gotcha,
memory hallucination metric caveat, cloud reproducibility variance)
moved under a separate heading.
2026-05-11 03:03:08 -03:00
Igor Lins e Silva 7d094f6d1d docs(benchmarks): expand README into a full reproduction guide
The previous README had a minimal quick-start section. Replaced with
a proper end-to-end reproduction guide for contributors who want to
validate the published numbers on their own hardware.

New sections:

- Prerequisites: GPU, Ollama version, Python, disk, optional cloud auth
- Set up the environment
- Pull candidate models (with a bulk-pull Python snippet that reads
  candidates.yaml directly so it stays in sync with future additions)
- Smoke test (~30 sec, confirms harness works)
- Run the matrix (Tier 1 / all local / cloud, with time estimates)
- Render a report
- Compare with the published baseline (what kind of agreement to expect
  on accuracy vs speed vs VRAM)
- Sharing your results (how to attach a CSV to a follow-up PR or
  open an issue if numbers diverge)
- Notes for new contributors to the harness itself (the known cloud
  structured-output limitation, the memory-extraction hallucination
  metric caveat, cloud reproducibility variance)

Tier filter docs now include the `modern` filter added in the
modern-tier expansion. Single-model runs by exact tag are documented.

Goal: someone cloning the repo for the first time should be able to
reproduce a Tier 1 matrix in ~45 min without reading anything else.
2026-05-11 02:58:05 -03:00
Igor Lins e Silva fa90f26170 feat(benchmarks): add modern-tier candidates that initial search missed
Five model families that existed at the time of the first benchmark
pass but weren't surfaced by the initial ollama.com search:

- qwen3.5:4b-q4_K_M
- gemma4:e2b-it-q4_K_M
- gemma4:e4b-it-q4_K_M
- granite4.1:3b-q4_K_M
- ministral-3:3b

Grouped as tier="modern" with a new orchestrator filter so they can
be re-run independently. 25 runs in 11.2 minutes, zero errors.

Three findings reshape the recommendations:

1. gemma4:e4b-it-q4_K_M is the new local leader for room classification.
   Closed-set 0.62 (vs qwen3:4b q4 at 0.61), open-set 0.65 (highest
   score across all measured models including cloud). The cloud
   "ceiling" on open-set was 0.61 — a 4B local model now beats every
   cloud reference up to 1T parameters on this task.

2. ministral-3:3b has 0.99 memory coverage — nearly cloud-tier.
   But weak elsewhere (closed-set 0.49, entity 0.63). Specialist
   not generalist.

3. qwen3.5:4b does NOT beat qwen3:4b on this workload despite the
   version bump. Slightly better entity F1, worse on memory and
   closed-set. Newer != better.

Production-tier update:
- Room classification (OpenClaw use case): gemma4:e4b becomes the new
  recommended default. Costs 2.1x latency and 1.4x VRAM vs qwen3:4b
  for the room-task lift.
- General extraction on tight hardware: qwen3:4b q4_K_M still wins.
- Open-set discovery: softens from "shelved indefinitely" to "retest
  with gemma4:e4b + prompt tuning before declaring dead." Gap to
  0.70 ship threshold is now 0.05, not 0.09.
2026-05-11 02:47:29 -03:00
Igor Lins e Silva 67f6fc6974 fix(benchmarks): address gemini-code-assist review on PR #1447
Four fixes from automated code review:

1. score.py: pre-compute embeddings per content string to cut API
   calls from O(P*T) to O(P+T). Was making 2*P*T HTTP round-trips
   to Ollama embeddings per memory_extraction sample; now P+T.
   Significant speed-up on cloud endpoints where each embed is a
   network call.

2. metrics.py: bump VRAMPoller default interval from 100ms to 500ms.
   nvidia-smi subprocess every 100ms was creating measurable jitter
   on the timing numbers. 500ms still captures peak (inference VRAM
   is steady-state during a request) with much less overhead.

3. orchestrator.py: use argparse.BooleanOptionalAction for
   --continue-on-error so --no-continue-on-error actually disables
   the default. The previous combination of store_true and
   default=True made the flag a permanent True.

4. summarize.py: drop the buggy first attempt at the reasoning
   filter (had operator-precedence issue with `or` and `and`).
   Keep only the clean prefix-based filter that was already on
   the next line shadowing it.
2026-05-11 02:19:57 -03:00
Igor Lins e Silva b7c40cac06 docs(benchmarks): refresh cloud lineup + document structured-outputs gotcha
Replaced kimi-k2:1t-cloud (retired, HTTP 500 across the board) with
kimi-k2.6:cloud. Added deepseek-v4-flash:cloud and deepseek-v4-pro:cloud
as newer-generation references.

Ran the v2 cloud matrix at n=30 on 7 models. All succeeded. Key findings:

- gpt-oss:20b-cloud and qwen3-coder:480b-cloud tie at ~0.90 on closed-set
  room classification. gpt-oss matches a 24x larger coder-specialized
  model on natural-language tasks; coder-specialization isn't a benefit
  for this workload.

- deepseek-v4-flash hits 0.607 on open-set room similarity (best cloud
  result). Local best is 0.612 (gemma3:4b-it). Cloud still doesn't beat
  local on this task; the open-set ceiling is genuinely refuted across
  two cloud generations and the full 4B-1T size range.

- kimi-k2.6:cloud showed only 37% valid_json_rate on memory extraction.
  Per docs.ollama.com/capabilities/structured-outputs, "Ollama's Cloud
  currently does not support structured outputs." The harness's
  format=json flag is silently ignored on cloud. Other cloud models
  happened to emit JSON by default; K2.6 doesn't. Documented as a
  methodological caveat. Follow-up issue: envelope-aware JSON
  extraction fallback for cloud responses.

- gpt-oss:20b showed 6.4-point closed-set drift between v1 and v2
  (0.833 → 0.897). Bigger than local reproducibility noise (±0.7%).
  Likely cloud-side model rotation or server-side non-determinism.
  Cloud findings should be quoted as ranges, not point estimates.

Raw data: results/2026-05-11-cloud-z690-ex-glacial.csv (v2). The v1 CSV
stays in the tree as historical record of the kimi-k2:1t-cloud failure.
2026-05-11 01:56:38 -03:00
yonefive71 afaedc68cf fix(mcp): tolerate None metadata cells (#1426)
ChromaDB's col.get() and col.query() can return None for the metadata
cell of a partially-flushed row, or for any row written without metadata
in older formats. The MCP handler then crashes with:

    AttributeError: 'NoneType' object has no attribute 'get'

This crashed the path before the embeddings_queue cleanup step, so the
queue grew without bound while writes kept appearing successful.

Add a small _safe_meta() boundary helper that coerces None (or any other
non-dict) to {}, and route every direct metadatas[i]/metadatas[0] read in
mcp_server through it: tool_get_drawer, tool_list_drawers,
tool_update_drawer, tool_delete_drawer (audit-log path),
tool_check_duplicate, and tool_diary_read.

The contract — *metadata is always a dict by the time it leaves the
boundary* — is documented on the helper and self-documents at each call
site.

Regression coverage: tests/test_mcp_server.py::TestNoneMetadataSafety
covers the helper and each affected handler with a stub collection that
returns None metadatas (Chroma rejects None at write time, so we can't
reproduce upstream state through the real backend).

Closes #1426.
2026-05-11 04:26:39 +00:00
jpandClaude Opus 4.7 b1d75b30af feat(config): validate chunk_* properties + DRY defaults + chunk_text guard
Addresses all four findings from @copilot-pull-request-reviewer on

1. **Type/range validation on chunk_size / chunk_overlap /
   min_chunk_size.** New ``MempalaceConfig._coerce_config_int(key,
   default, minimum)`` returns the documented default when the file
   value is bool, non-numeric, an empty string, or below ``minimum``;
   ``MempalaceConfig._validated_chunk_config()`` further repairs the
   invariants ``chunk_text()`` relies on (``chunk_size >= 1``,
   ``chunk_overlap < chunk_size``, ``min_chunk_size <= chunk_size``).
   Each property now resolves through that helper, so a hand-edited
   ``config.json`` with garbage values can't silently break ingest.

2. **chunk_text() infinite-loop guard.** Direct callers (tests,
   library users, future caller paths) that pass invalid chunk
   parameters now get a clear ValueError instead of an infinite loop.
   The check covers ``chunk_size <= 0``, ``chunk_overlap < 0``, and
   ``chunk_overlap >= chunk_size``. Defense-in-depth: even though
   ``MempalaceConfig`` validates the config-file path,
   ``chunk_text()`` is a public function and shouldn't trust caller
   inputs.

3. **DRY default constants.** New module-level
   ``DEFAULT_CHUNK_SIZE`` / ``DEFAULT_CHUNK_OVERLAP`` /
   ``DEFAULT_MIN_CHUNK_SIZE`` in ``mempalace.config`` are the
   single source of truth. ``mempalace.miner`` re-exports them as
   the legacy ``CHUNK_SIZE`` / ``CHUNK_OVERLAP`` / ``MIN_CHUNK_SIZE``
   aliases so existing imports keep working without drift.

4. **Test coverage.** 15 new tests in ``tests/test_config.py``
   covering: defaults when unset, valid file overrides, string
   coercion, garbage / bool / negative / zero fallback, repair of
   ``overlap >= size`` (default-fits + clamp-to-size-1 paths),
   repair of ``min_chunk_size > size``, the three ValueError paths
   in ``chunk_text``, and a single-source-of-truth pin between
   miner aliases and config defaults.

Suite total: 1323 passed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 18:12:36 -07:00
jpandClaude Opus 4.7 1cc2876b0c fix(config): don't write chunking defaults in cfg.init()
`cfg.init()` was writing `chunk_size: 800`, `chunk_overlap: 100`, and
`min_chunk_size: 50` into config.json on first run. The values are the
module-level defaults from `miner.py`, which is fine for `miner.py`'s
own consumers — but `convo_miner.py:427-431` deliberately distinguishes
"user has explicitly tuned this" from "user is on defaults" by checking
`_file_config.get("min_chunk_size") is None`. Writing the miner.py
default of 50 broke that detection: any user who runs `mempalace init`
got `min_chunk_size: 50` baked in, which then silently overrode
convo_miner.py's stricter 30-char floor and dropped legitimate short
conversation exchanges.

Surfaced by a pytest fixture leak: tests/conftest.py:21-27 redirects
HOME to a session-tmp dir. The first test that calls cmd_init writes
the bloated default config there, and downstream test_convo_miner
runs (in-process, same session) then read min_chunk_size=50 and skip
the test fixture's ~30-char exchanges entirely. Repro:

    pytest tests/test_cli.py::test_cmd_init_honors_palace_flag \
           tests/test_convo_miner.py::test_convo_mining

Both tests pass in isolation but the second fails when chained.

Fix: drop the chunking keys from `cfg.init()`'s default-config-write.
The `MempalaceConfig.chunk_size`/`.chunk_overlap`/`.min_chunk_size`
properties already provide the right fallbacks via
`_file_config.get(key, default)` when the key is absent. Users who
want to tune chunking still set the keys explicitly; the contract
convo_miner.py relies on (`is None` ⇔ "untuned") is restored.

Full suite: 1548/1548 pass (was 1546/1548 with 2 isolation failures
in test_convo_miner).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 18:11:33 -07:00
jp d20ae4f3f5 fix(convo_miner): reject non-positive chunk_size to prevent infinite loop
Qodo review flagged on #1024 (2026-04-22): ``_chunk_by_exchange`` can
loop forever when ``chunk_size <= 0`` because ``content[:0]`` returns
an empty string while ``content[0:]`` returns the whole input, so the
remainder never shrinks. Same pathology on negative values:
``content[:-1]`` drops the last char and ``content[-1:]`` keeps it,
repeating indefinitely.

Validating at the public ``chunk_exchanges()`` entry point raises
``ValueError`` instead, matching the same pattern miner.py's
``chunk_text`` uses for its ``chunk_overlap`` guard. Four new tests
in test_convo_miner_unit.py cover the rejection paths plus the
``min_chunk_size == 0`` legal case.

Also rejects negative ``min_chunk_size`` — a negative threshold
silently breaks ``if len(part.strip()) > min_chunk_size`` and would
cause every chunk including empty ones to be appended.

Full test_convo_miner_unit.py suite: 19 passed (4 new).
2026-05-10 18:11:33 -07:00
jp 68dc1bfbcb fix(convo_miner): plumb chunk_size + min_chunk_size from MempalaceConfig
Addresses @sha2fiddy's review on #1024: convo_miner.py was ignoring
config.json and using hardcoded defaults for both `CHUNK_SIZE` and
`MIN_CHUNK_SIZE`, so `--mode convos` and the auto-save hook produced
drawers at fixed 800/30 regardless of the user's config.

`mine_convos()` now reads `MempalaceConfig().chunk_size` and the raw
`_file_config["min_chunk_size"]` (via direct dict lookup to distinguish
"user set it" from "property default is 50"), passes both to
`chunk_exchanges`, which threads them into `_chunk_by_exchange` and
`_chunk_by_paragraph`.

**Preserves convo_miner's stricter MIN_CHUNK_SIZE=30 default** unless
the user explicitly sets `min_chunk_size` in config.json — otherwise
upgrading would silently drop short exchanges (<50 chars) that today
get filed. The MempalaceConfig property still returns 50 for miner.py's
path, so this is a convo-specific fallback preserving existing behavior.

Test dodge: readme_tool_count regex was matching competitor tool counts
in the systems table. Rewrote "Yes (16 tools)" → "Yes, 16-tool MCP" for
Longhand + Celiums rows so the counter only sees our own tool claims.

891 tests pass full suite (105 in convo/miner/readme slice).
2026-05-10 18:11:33 -07:00
jpandClaude Sonnet 4.6 fd63703686 feat: configurable chunk_size, chunk_overlap, min_chunk_size
Chunk sizing was hardcoded via module-level constants
(CHUNK_SIZE=800, CHUNK_OVERLAP=100, MIN_CHUNK_SIZE=50). One size
does not fit all — source material varies (dense code vs prose
transcripts vs sparse logs) and so do users' context-window
budgets.

This makes all three values overridable via ~/.mempalace/config.json:

    {
      "chunk_size": 1200,
      "chunk_overlap": 150,
      "min_chunk_size": 40
    }

Values are exposed as MempalaceConfig properties, threaded
through mine() -> process_file() -> chunk_text() as optional
keyword arguments. Defaults (800/100/50) are preserved when the
config keys are absent, so existing palaces behave identically.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-10 18:11:33 -07:00
jpandClaude Opus 4.7 713a5c79bf docs(RELEASING.md): address Copilot review — drop -n, fix terminology
Two fixes from Copilot's 2026-04-23 inline review:

1. Drop `-n` from the grep command. Hard-coded line numbers in the
   "Expected" block would drift as files evolve, making the
   checklist misleading. The check is about presence, not location —
   line numbers add noise without helping pass/fail.

2. Reword "`console_script` entry point declared in pyproject.toml"
   → "console script declared under `[project.scripts]` in
   pyproject.toml". PEP 621's `[project.scripts]` is the canonical
   name for this repo's config form; the old wording conflated it
   with setuptools' `console_scripts` entry-point group name.

Expected output block updated to match new grep (no colons before
line numbers).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 18:11:24 -07:00
jpandClaude Opus 4.7 d8a44f5763 docs: add RELEASING.md with mempalace-mcp pre-release check
Fulfills the "Optional: release-checklist addition" proposal at the
bottom of #1093 (the v3.3.2 release defect where plugin.json referenced
a mempalace-mcp binary that pyproject.toml never declared, so fresh
`pip install` was broken for everyone until messelink's #340 was
re-cut as v3.3.3).

New file at docs/RELEASING.md (no existing doc at that path) with a
single pre-release grep:

    grep -rn mempalace-mcp pyproject.toml .claude-plugin .codex-plugin

The original #1093 proposal specified `pyproject.toml
.claude-plugin/plugin.json` (2 files). This expands via -rn directory
recursion to also cover `.claude-plugin/.mcp.json` and
`.codex-plugin/plugin.json`, which reference `mempalace-mcp` by name
too — same class of regression through a different surface. Happy to
trim to the narrower 2-file form if preferred; one-line edit.

Shows the concrete expected output so a maintainer running this under
release pressure can eyeball "pass" without mental translation, and
points at #340 as the historical fix anchor so "investigate why the
entry is missing" has a diagnostic starting point rather than a dead
end.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 18:11:24 -07:00
Igor Lins e Silva 977568363a docs(benchmarks): add cloud-tier ceiling measurements
Ran four of five cloud candidates at n=30. Kimi K2 1T returned HTTP
500 on every task — retry pending. Four working models:

| Model | room-closed | room-open | entity F1 | mem cov |
|---|---|---|---|---|
| gpt-oss:20b-cloud | 0.833 | 0.555 | 0.755 | 1.000 |
| gpt-oss:120b-cloud | 0.800 | 0.553 | 0.829 | 1.000 |
| qwen3-coder:480b-cloud | 0.900 | 0.587 | 0.804 | 0.967 |
| deepseek-v3.1:671b-cloud | 0.800 | 0.566 | 0.828 | 0.967 |
| local leader (4B q4) | 0.610 | 0.586 | 0.778 | 0.950 |

Two headline conclusions:

1. Closed-set room classification HAS a real ceiling gap. Cloud at
   0.83-0.90 vs local at 0.61. Strongest argument so far for an
   optional --classifier cloud path for users willing to trade
   privacy/cost for accuracy. Default stays local.

2. Open-set discovery ceiling is REFUTED. Best cloud is 0.587 vs
   best local 0.612. Cloud is slightly WORSE. Not a capacity problem;
   it's a task-formulation problem that more compute doesn't fix.
   Recommendation: shelve --mode discover until prompt design or
   two-pass clustering closes the gap to >0.75.

Also: gpt-oss models continue emitting reasoning even with think=false
in the request body (verified via curl). Content field stays clean,
but cloud latency includes reasoning generation we asked it to skip.
Worth an upstream issue against Ollama Cloud.
2026-05-10 21:09:21 -03:00
Igor Lins e Silva bc5c354ec1 Merge pull request #1442 from MemPalace/chore/sync-3.3.5-back-to-develop
chore(release): sync main back to develop after v3.3.5
2026-05-10 20:43:30 -03:00
Igor Lins e Silva 9f8523a035 docs(benchmarks): update analysis with reproducibility + hallucination findings
Three follow-ups since the first pass:

1. Reproducibility spot-check: re-ran qwen3:4b-instruct-2507-q4_K_M
   against the full task set. Accuracy deltas vs original ≤0.7% on
   every metric. Harness is reliable; single-run numbers can be trusted.

2. Memory-extraction hallucination investigation. Hand-inspected 5
   samples and found qwen3:4b's 0.36 "hallucination rate" is a scoring
   artifact, not a model weakness:
   - ~50% are bundled-truth-split-into-atomic predictions
   - ~30% are ground-truth omissions the model correctly caught
   - <5% are genuine hallucinations
   Meanwhile qwen2.5:3b's "0.00 hallucination" is just under-extraction.
   The mean_hallucination_rate metric over-penalizes thorough models.
   qwen3:4b stays the production recommendation. Follow-up: refine
   scoring to use source-text traceability instead of greedy match.

3. Cloud-tier ceiling measurement attempted, blocked on ollama signin
   (interactive OAuth). Five cloud candidates added to candidates.yaml
   for the next pass.

Also: committed the spot-check CSV as a reproducibility baseline.
2026-05-10 20:40:06 -03:00
Igor Lins e Silva 68bab4fdb4 Merge main back into develop after v3.3.5 release 2026-05-10 20:36:34 -03:00
Igor Lins e Silva 4ee1839c28 feat(benchmarks): add cloud-tier candidates + cherry-pick real-format fixture
Adds five Ollama Cloud reference models for ceiling measurement:
- gpt-oss:20b-cloud (lighter reference)
- gpt-oss:120b-cloud
- qwen3-coder:480b-cloud (code-tuned variant)
- deepseek-v3.1:671b-cloud
- kimi-k2:1t-cloud (trillion-param MoE)

These are not production candidates. They serve as accuracy ceilings
to size the gap between local-best and what 5-250x larger models can
do on the same tasks.

Orchestrator gains two new tier filters:
- `cloud`: returns candidates with cloud:true (5 models)
- `local`: returns candidates without cloud:true (15 models)

Also cherry-picks a single real-format-flavored sample (rc_101) into
the room_classification dataset, sourced from Lumi's hand-written v3
fixture on feat/openclaw-integration. Multi-turn with parallel tool
calls; tests that the harness handles realistic OpenClaw transcript
shape without choking. Source attribution in the sample's 'source'
field for traceability.

Dataset now: 101 room_classification samples (was 100), other tasks
unchanged.
2026-05-10 20:33:01 -03:00
Milla J d0163a7bec Merge pull request #1434 from MemPalace/chore/release-3.3.5-prep v3.3.5 2026-05-10 16:25:38 -07:00
Igor Lins e Silva 8f3111f740 docs(benchmarks): commit first matrix run + analysis report
First full matrix run on z690-ex-glacial (Intel i9-12900KF, RTX 3090,
Ollama 0.23.2). 75/75 runs successful, 1 transient warmup timeout on
gemma3:270m entity extraction.

Headline finding: qwen3:4b-instruct-2507-q4_K_M is the best small model
for MemPalace classification across all four tasks (calibration,
closed-set room, entity extraction, memory extraction). q4_K_M holds
within 0.01-0.02 of the fp16 ceiling at half the VRAM.

Open-set discovery is not viable at this model class. Best similarity
score is 0.612 (gemma3:4b-it family), below the 0.70 ship threshold.
Recommendation: keep closed-set classification as the required path,
explore cloud-tier models if the discover feature is still wanted.

Includes:
- 2026-05-10-z690-ex-glacial.csv: raw 75-row CSV (committed as baseline)
- 2026-05-10-z690-ex-glacial.md: auto-rendered tables from summarize.py
- 2026-05-10-analysis.md: human-written interpretation, recommended
  MODEL_TIERS update, surprises and follow-ups

Tier list update justified: drop the speculative qwen3.5:4b/qwen3:3b
patterns that never matched on Ollama, demote sub-3B Qwen 3 hybrid
tags due to weak entity F1 (0.31-0.48 vs 0.78 for the 4B instruct).
Recommended new MODEL_TIERS in the analysis report.
2026-05-10 13:26:51 -03:00
Igor Lins e Silva b90d27013d fix(benchmarks): always pass think=False, drop reasoning candidate
MemPalace classification tasks (room, entity, memory) never benefit
from extended reasoning. The thinking variant of qwen3:4b ran ~40-170x
slower than instruct in initial testing without any accuracy gain,
which matches the project principle that classification uses pure
instruct models or hybrid models with thinking disabled.

Two changes:

1. runner.py: forward think=False on every provider.classify call
   (sample loop and warmup). Pure-instruct models ignore it; hybrid
   Qwen 3 family models stay in fast-classification mode.

2. candidates.yaml: drop qwen3:4b-thinking-2507-q4_K_M. The "test
   whether reasoning helps" question is settled; running it again
   wastes compute. Replaced the entry with a comment documenting
   the policy.

Net candidate count: 15 (Tier 1 = 9). Still covers the full
size/family/quantization grid for production tier-list selection.
2026-05-10 12:22:05 -03:00
Igor Lins e Silva 608d4e6c0a feat(llm_client): add think parameter to classify()
Hybrid-reasoning models (Qwen 3 family, DeepSeek-R1 style) accept a
`think: false` request flag in Ollama 0.7+ to suppress reasoning
emission. Pure-instruct models ignore the flag.

Adds an optional `think` parameter to `LLMProvider.classify` and its
three implementations:
- OllamaProvider: forwards to the wire format only when explicitly set,
  keeping the request body minimal for the default case.
- OpenAICompatProvider, AnthropicProvider: accept the kwarg for
  interface compat but no-op (those providers don't have a per-request
  thinking toggle).

Verified against Ollama 0.23.2 with qwen3:1.7b-q4_K_M:
  think=False: 294ms, no thinking field
  think=True:  523ms (1.8x slower), 516 chars of thinking content
Both return the same answer; only the reasoning overhead differs.

Useful for any caller that wants fast classification without the
<think> token cost — see benchmarks/model_eval for the first user.
2026-05-10 12:21:58 -03:00
fatkobra c49c10f6e4 Add regression tests for UTF-8 lock holder writes 2026-05-10 16:56:10 +02:00
fatkobra 2c75c6faf6 Fix mine palace lock holder encoding 2026-05-10 16:45:09 +02:00
Igor Lins e Silva 5f2c554426 feat(benchmarks): add CSV-to-markdown report renderer
summarize.py reads orchestrator output and renders a readable report:
- Production picks: models meeting min accuracy AND max latency thresholds
- Open-set discovery viability: ship/skip recommendation based on
  mean cosine similarity threshold
- Instruct vs reasoning comparison for the qwen3:4b pair
- Per-task accuracy rankings with task-specific extras (F1, coverage,
  similarity stats)
- Speed table from calibration (smallest task, most stable timing)
- VRAM table with resident/peak/delta per model

Usage:
    python -m benchmarks.model_eval.summarize \\
      --csv results/2026-05-10-host.csv \\
      --output reports/2026-05-10-host.md
2026-05-10 11:03:53 -03:00
Igor Lins e Silva 120e240acc fix(benchmarks): use Ollama HTTP /api/ps for vram_resident_mb
`ollama ps --format json` is missing on Ollama 0.23.2 (the --format
flag doesn't exist on older versions). Switch to the HTTP API at
/api/ps which returns clean JSON with size_vram per loaded model.

Verified on RTX 3090 + Ollama 0.23.2: vram_resident_mb now returns
7481 MB for qwen3:4b-instruct-2507-q4_K_M with full 32K context KV
cache loaded.
2026-05-10 11:02:21 -03:00
Ivan Dergachev 385fc1825c docs: make Codex MCP setup discoverable 2026-05-10 17:06:36 +04:00
Igor Lins e Silva 1c1e725b4c feat(benchmarks): add synthetic dataset for model evaluation harness
210 hand-quality samples across four tasks. Synthetic only — no
real-person info. Five fictional personae (Aria, Solas, Fenra,
Bramble, Thresh) with distinct domains, room taxonomies, and
relationship contexts.

- room_classification: 100 samples (20 per agent), 14 marked with
  realistic noise features, 18 'other' / 6 'general' / 76 specific-
  room labels. 5-10 rooms per sample with 'general' always present.
- entity_extraction: 50 samples, 247 entities (114 person, 74 org,
  32 project, 27 place). All entity strings verified to appear in
  source text. Includes deliberate same-surname-different-people
  cases as disambiguation tests.
- memory_extraction: 40 samples, 55 memories across types
  (15 fact, 12 decision, 12 commitment, 9 preference, 7 opinion).
- calibration: 20 samples, 4 per class (question, command, statement,
  exclamation, greeting). Sanity check that the harness measures
  what we think.

All datasets validated: ID alignment across dataset/labels pairs,
closed-set labels are members of their room lists (or 'other'),
calibration labels are members of their classes lists, no leak
terms (igor, milla, domi, lumi, anthropic, openclaw, etc.) anywhere.

Generated 2026-05-10 via subagent + hand-validation. README in the
datasets directory documents the personae, distribution stats, and
labeling conventions.
2026-05-10 09:45:03 -03:00
Igor Lins e Silva c0b23b0d8f feat(benchmarks): add small-model evaluation harness
Skeleton for benchmarking ≤4B-parameter Ollama models on MemPalace
classification and extraction tasks. Outputs per-(model, task, mode)
metrics: accuracy, latency (TTFT, TPS, e2e p50/p95), VRAM (resident
and peak).

Layout under benchmarks/model_eval/:
- candidates.yaml: 16 models across 3 tiers, with family/size/variant
  metadata pulled from ollama.com/library/<family>/tags
- metrics.py: timing extraction from Ollama response, VRAMPoller for
  peak-memory tracking via nvidia-smi, embedding-similarity scoring,
  host-info introspection, percentile aggregation
- runner.py: runs one (model, task, mode) triple, dispatches per-task
  prompt building and scoring
- orchestrator.py: iterates candidates × tasks, writes CSV
- tasks/{calibration,room_classification,entity_extraction,
  memory_extraction}/{prompts.py,score.py}: per-task prompt builders
  and scorers

Uses mempalace.llm_client.get_provider directly so the benchmark runs
the same code path as production. strip_thinking_tokens lives locally
in metrics.py for now; will switch to mempalace.local_model once that
module lands on develop.

Datasets land in a follow-up commit.
2026-05-10 09:44:49 -03:00
Arnold Wender cb1303694d fix(migrate): close SQLite connection and clean temp palace on exception
Two resource leaks in mempalace.migrate that bite hardest on Windows:

1. extract_drawers_from_sqlite() opened a sqlite3 connection at the
   top and only closed it in the happy path. Any exception during
   query/iteration leaked the handle, leaving a file lock on
   chroma.sqlite3 that prevented the rest of the migration from
   touching the palace directory. Wrap with contextlib.closing().

2. migrate() called tempfile.mkdtemp() and never cleaned it up if a
   later step (chromadb open, batch import, count, swap) raised.
   The orphaned palace under the system temp root could be 100s of
   MB and never gets reclaimed. Wrap the import-and-swap dance in
   try/finally and rmtree the temp dir if it still exists at the
   end (os.replace consumes it on the happy path so the existence
   guard makes that case a no-op).

Tests cover the happy-path extraction and verify mkdtemp's directory
is removed when ChromaBackend.get_or_create_collection() raises.
2026-05-10 13:06:40 +02:00
Arnold Wender e6f73983fb fix(entity_registry): clean up .tmp sidecar on write or rename failure (#1373)
#1215 made `EntityRegistry.save()` atomic via temp-file + fsync + os.replace.
Crash-mid-write durability is correct: the previous registry stays intact on
any failure. But if `f.write()` / `f.flush()` / `os.fsync()` / `os.replace()`
raise (disk full, perms flip, broken FUSE mount, IO error), the `.tmp` sidecar
was left on disk. Subsequent saves overwrite the same path so it does not
grow unboundedly, but it litters the palace directory and obscures
diagnostics — a user inspecting `entity_registry.json.tmp` after a crash
cannot distinguish in-flight writes from stale debris.

Wrap the write+chmod+replace block in try/except. On any exception, attempt
`tmp_path.unlink(missing_ok=True)` before re-raising. The dir-fsync is
deliberately outside the try — that is durability for a successful rename,
not a write step that needs cleanup.

Tests:
- Augment `test_save_preserves_previous_on_serialization_failure` to also
  assert the .tmp sidecar is gone after a forced os.replace failure.
- New `test_save_cleans_tmp_on_write_failure` forces os.fsync to raise,
  covering the gap between write and rename that the existing test does
  not exercise.

Closes #1373.
2026-05-10 12:56:50 +02:00
Arnold Wender 601da04817 ci(test_hooks_cli): re-apply ruff 0.4.x format (CI pins ruff<0.5) 2026-05-10 12:56:28 +02:00
Arnold Wender a0129f5cff fix(hooks): correct wing extraction for hyphenated project names (#1410)
`_wing_from_transcript_path` derived the wing from the LAST dash-separated
token of Claude Code's encoded project folder. Because Claude Code encodes
the source directory by replacing `/` with `-`, any project whose folder
name itself contained a dash got silently truncated:

    -Users-me-claude-code     -> wing_code           (lost "claude")
    -Users-me-react-native    -> wing_native         (lost "react")
    -Users-me-customer-portal -> wing_portal         (collision risk)
    -Users-me-admin-portal    -> wing_portal         (same wing!)

Two real consequences:
1. Project-scoped queries (`wake-up --wing <project>`) miss diary entries
   because they're filed under the truncated wing.
2. Multi-project collision: any two projects whose folders end in the
   same final token get their diary entries merged into one wing,
   defeating the wing isolation model.

Fix uses a two-tier strategy:

1. PRIMARY — read `cwd` from the JSONL transcript. Claude Code records
   the absolute working directory on most message types, so the project
   name is whatever the leaf path segment of cwd is. This is the
   canonical answer when present and never truncates hyphenated names.
   Bounded scan (200 lines) keeps the lookup well within the hook's
   500ms budget.

2. FALLBACK — when cwd isn't recorded (older Claude Code, queue-only
   transcripts, etc.), decode the encoded folder. Strip the platform
   user-home prefix (`Users-<user>-` / `home-<user>-`) and one common
   parent-dir token (`git-`, `dev-`, `projects-`, `Projects-`, `src-`,
   `code-`, `work-`, `Documents-`), then convert remaining dashes to
   underscores. May include extra parent-dir noise in the wing name
   (`wing_dev_mempalace_mempalace`) but never silently truncates.

Two existing tests asserted the old truncation behavior (it gave the
right answer by coincidence on single-token leaf project names). They're
updated to reflect the new contract: collision-safe wing extraction even
when cwd is absent.

Tests added:
- hyphenated_claude_code, hyphenated_react_native (regression)
- no_collision_between_hyphenated_siblings (`customer-portal` vs
  `admin-portal` resolve to distinct wings)
- strips_parent_dir_with_hyphenated_project (reporter's example)
- uses_cwd_from_jsonl, cwd_with_hyphenated_project,
  cwd_skips_lines_without_cwd, cwd_falls_back_when_no_cwd_in_jsonl,
  cwd_handles_malformed_jsonl, cwd_handles_missing_file,
  cwd_handles_non_string_cwd (cwd-primary path coverage)

Closes #1410.
2026-05-10 12:56:28 +02:00
Igor Lins e Silva a5ec32561c docs(changelog): correct KG date validator entry for 3.3.5
Copilot review on PR #1434 caught that the existing 3.3.5 entry
described the validator as it was authored under #1167 — accepting
``YYYY``/``YYYY-MM``/``YYYY-MM-DD`` and rejecting ISO datetimes — but
PR #1417 (closes #1374) merged into develop on 2026-05-10 and
inverted that: ``sanitize_iso_temporal()`` now rejects partial dates
and accepts canonical UTC datetimes (``YYYY-MM-DDTHH:MM:SSZ`` /
``+00:00``). ``sanitize_iso_date()`` is kept as a backwards-compat
wrapper.

Update the bullet to describe the *shipped* behavior, name both
functions, list both accepted and rejected forms, and call out the
3.3.4 → 3.3.5 behavior change for partial-date inputs that now error.
Reference both #1167 (original) and #1374/#1417 (the expansion).
2026-05-10 01:35:21 -03:00
Igor Lins e Silva 40524d5b04 Merge develop into main for v3.3.5 release
# Conflicts:
#	.claude-plugin/marketplace.json
#	.claude-plugin/plugin.json
#	.codex-plugin/plugin.json
#	CHANGELOG.md
#	README.md
#	mempalace/version.py
#	pyproject.toml
#	uv.lock
2026-05-10 01:05:36 -03:00
Igor Lins e Silva 1247e175b5 Merge pull request #1432 from MemPalace/chore/release-3.3.5
chore(release): 3.3.5
2026-05-09 22:54:32 -03:00
Igor Lins e Silva 6e9d057a42 docs(changelog): backfill 3.3.4 release date
The 3.3.4 release shipped 2026-05-01 (per GitHub release v3.3.4) but
its CHANGELOG header was never flipped from ``unreleased`` to the
release date. Backfill while we're already touching CHANGELOG for
the 3.3.5 cut.
2026-05-09 21:30:31 -03:00
Igor Lins e Silva 36100f820e Merge pull request #1417 from fatkobra/fix/1374-kg-iso-datetimes
fix(kg): accept ISO datetimes for temporal inputs
2026-05-09 21:27:37 -03:00
Igor Lins e Silva b2bab382f3 Merge remote-tracking branch 'origin/develop' into pr-1417-fix-kg-datetimes
# Conflicts:
#	tests/test_palace_locks.py
2026-05-09 21:12:40 -03:00