The project-files mine path (miner.mine) has wrapped _mine_impl in
mine_palace_lock since #1264 — a non-blocking flock that raises
MineAlreadyRunning so the second runner exits cleanly instead of
queueing as a waiter that drives parallel HNSW inserts. The convos
mine path (convo_miner.mine_convos) was missing the same guard.
In practice this meant any caller that spawned `mempalace mine
--mode convos` repeatedly against the same palace — most notably
the Stop-hook transcript ingest before the per-target PID slot
landed — could stack up arbitrarily many concurrent mines, each
holding a ChromaDB client open, each writing to the same HNSW
index. Recently observed: 28 stuck convos mines on one machine
consuming ~18 GB of RAM and contributing to a load spike.
Fix: refactor mine_convos into a thin wrapper that holds the
per-palace flock around _mine_convos_impl, mirroring miner.mine
exactly. Dry-run skips the lock since it never writes.
Tests: two cross-process tests in tests/test_convo_miner.py —
one asserts MineAlreadyRunning when a child process holds the
lock, one asserts dry-run is unaffected. Same spawn-context
pattern as test_palace_locks.py (fork-with-chromadb deadlocks
on Python 3.13).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
mine_convos was calling file_already_mined() once per file inside the
main loop. On a 150k-drawer palace, each per-file query
(`collection.get(where={"source_file": X}, limit=1)`) costs ~2 seconds
because chromadb has to scan the metadata index. A 2000-transcript
directory took >1h of wall-clock just to decide every file should be
skipped — and pegged multiple cores doing so, starving the daemon's
other endpoints.
bulk_check_mined() already existed for exactly this anti-pattern (its
docstring says "Callers can check membership and compare mtimes locally
instead of issuing one ChromaDB query per file") but only the project
miner used it; the convo miner kept the slow per-file path.
This patch adds a third helper, prefetch_mined_set(), that mirrors
file_already_mined()'s version-gate semantics (the check_mtime=False
branch used by mine_convos) and returns a set[str] for O(1) lookups.
mine_convos now calls it once before the loop; the loop body becomes
a set-membership check.
Observed on a 172k-drawer palace probing 10 files:
before: 21.2s (2.12s/file)
after: single bulk pass should be 30–60s for the whole 172k scan,
then 2000 O(1) checks ≈ free
file_already_mined() is kept for callers that genuinely need the
per-file semantics (the post-lock race-check in _file_chunks_locked
at convo_miner.py:350 still uses it intentionally).
Surfaced via jphein/familiar.realm.watch foundation-rework debugging
on 2026-05-11. Originally filed as #51.
Addresses @Copilot's review feedback on #1459. Five tests:
- test_extract_drawers_preserves_valid_metadata: non-empty dict
passes through unchanged (regression guard against breaking happy
path).
- test_extract_drawers_sanitizes_none_metadata: None entries
coerce to {"_repaired_empty_meta": True} (the core fix).
- test_extract_drawers_sanitizes_empty_dict_metadata: empty dict
{} entries also coerce to the sentinel (chromadb 1.5.x rejects
both shapes equally).
- test_extract_drawers_sanitization_preserves_alignment: critical
invariant — ids[i] / documents[i] / metadatas[i] stay in
lockstep through the sanitizer; mis-pairing would silently
corrupt rebuilds.
- test_extract_drawers_multiple_batches: pagination boundary
correctness (sanitizer applied per-batch, no drops/duplicates).
Verified passing locally against mempalace fork main + chromadb
1.5.8 in the palace-daemon venv (5 passed, 67 deselected in 1.88s).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Belt-and-suspenders on top of the repair.py sanitization in the
previous commit. A 151,478-drawer rebuild test still failed at
~120K with the same `ValueError: Expected metadata to be a
non-empty dict` from chromadb, even with the repair.py sanitizers
in place. Traceback:
mempalace/backends/chroma.py:add → chromadb Collection.add
→ validate_insert_record_set → validate_metadatas
→ validate_metadata → ValueError
Likely cause: chromadb's `upsert()` internally calls `add()` for
new records, and somewhere between repair.py's batch upsert and
chromadb's final write, the metadatas list gets reprocessed in a
way that re-introduces empty/None entries.
Sanitizing at the chromadb-client chokepoint catches everything:
no caller can leak bad metadata regardless of upstream sanitization
state. Same `{"_repaired_empty_meta": True}` sentinel, searchable
via `where={"_repaired_empty_meta": True}`.
Cost: one list comprehension per write call; negligible vs the
embedding + HNSW work each upsert already does.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two functions construct the metadatas[] list that gets fed to
chromadb's upsert/add, both vulnerable to the same ValueError:
ValueError: Expected metadata to be a non-empty dict, got 0
metadata attributes in add.
chromadb 1.5.x's validate_metadata rejects both `None` and `{}`
entries — see chromadb/api/types.py:validate_metadata (line ~1071).
This commit patches both:
1. `_extract_drawers` (line ~131) — the chromadb-collection-based
extractor, used when the source palace's collection is openable
via the chromadb client. Sanitizes None/{} entries in the
`batch["metadatas"]` list to `{"_repaired_empty_meta": True}`
before extending `all_metas`.
2. `_rebuild_one_collection` (line ~813) — the SQLite-direct
extract path used by `rebuild_from_sqlite()`, invoked when the
source palace can't be opened via chromadb (the recovery path
for palaces with corrupt HNSW segments). Old code was:
metas.append(meta if meta else {})
The trailing `{}` was the bug; chromadb 1.5.x rejects empty
dicts the same as None. Replaced with the same sentinel.
Why `_repaired_empty_meta: True` as the sentinel:
- Satisfies chromadb's non-empty-dict requirement
- Bool-valued (valid chromadb metadata type, trivially serializable)
- Namespaced + descriptive so an operator can find which drawers
were coerced via `where={"_repaired_empty_meta": True}` later
- Idempotent on re-runs (a future repair over a sanitized palace
sees the sentinel as already-valid)
Verified on a 151,478-drawer production palace that previously
crashed at drawer 120,000 in both extract paths.
Fixes#1458
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tasks that score via semantic similarity (memory_extraction,
room_classification open-set) depend on nomic-embed-text. If the model
was absent, embed_text() returned None silently and every sample scored
0.0 — indistinguishable from a genuinely bad model.
_ensure_embed_model() now probes the endpoint before the warmup step.
On a 404 it pulls the model automatically via `ollama pull`, verifies
the endpoint responds, and only then proceeds. Fixes:
- OLLAMA_HOST is now forwarded to `ollama pull` so non-localhost
endpoints are targeted correctly
- FileNotFoundError is caught and re-raised as RuntimeError with an
install hint if ollama is not on PATH
- Diagnostic prints go to stderr, not stdout
- If the pull or re-probe fails, returns a Result with a clear error
message instead of silent zeros
Restores --candidates local to its documented meaning (all non-cloud
models). Adds --candidates community for third-party GGUF variants.
Ten fixes across code, documentation, and committed baselines:
Code fixes:
1. prompts.py: closed-set prompt said "no punctuation" while room
slugs contain "/". Replaced with explicit "copy the slug verbatim
including / and -" wording. Empirical impact on accuracy is within
reproducibility noise (qwen3:4b q4 was 0.610, retest shows 0.594
on the now-101-sample dataset — same vs new sample noise).
2. orchestrator.py: orchestrator now opens the CSV once, writes the
header, and flushes after each row. README's "writes incrementally,
safe to Ctrl-C" claim is now accurate. Long matrix runs no longer
lose all progress on a crash.
3. metrics.py: gather_host_info() now strips trailing newline from
`hostname` command output. Stops embedded newlines from leaking
into committed CSVs.
4. runner.py: passed `endpoint=endpoint` to vram_resident_mb() so
non-default Ollama endpoints are queried correctly.
5. summarize.py: CSV reader and report writer now use
encoding="utf-8" with explicit newline handling for cross-platform
consistency.
Documentation fixes:
6. entity_extraction/score.py: docstring claimed "case-sensitive on
type" while implementation lowercases both. Corrected.
7. dataset.jsonl rc_101: removed mempalace-private URL since the PR
lands on the public repo. Will be linkable after OpenClaw merges.
8. datasets/README.md: sample counts updated (room_classification 100
→ 101, total 210 → 211) to match the cherry-picked real-format
sample.
9. runner.py / orchestrator.py: docstring example paths corrected
(tests/fixtures/mempalace_bench → benchmarks/model_eval/datasets).
Baseline regeneration:
10. Stripped embedded \n from `host` column in all five committed
CSVs (2026-05-10 main local + spotcheck, 2026-05-10 + 2026-05-11
cloud, 2026-05-11 modern). 165 rows cleaned in place via Python
csv reader/writer. Each row is now line-oriented for diff and
grep friendliness without changing any measured values.
Adds a "Contributions welcome" section to the README covering:
1. Adding models we missed. The published candidate list is one
engineer + one search pass. Particularly call out function-calling-
tuned variants (Phi-4 mini, Nemotron-mini), research-lab instruct
models (Hermes, Dolphin), and quantization-aware-trained variants
as worth-adding categories.
2. Adding other inference backends. mempalace.llm_client.get_provider
already supports openai-compat, which means LM Studio, llama.cpp
server, vLLM, unsloth studio, Docker Model Runner, HF TGI all
plug in with minimal harness changes. Documents the four pieces a
backend contributor would need to touch:
- Backend-selection flag in runner/orchestrator
- Backend-specific timing extraction in metrics.py
- Backend-specific VRAM probe
- candidates.yaml backend field
Notes that accuracy numbers stay comparable across backends since
the dataset and scoring code are backend-agnostic.
Also calls out niche runtimes worth cross-comparison: Apple MLX,
Intel OpenVINO, AMD ROCm, edge-device runtimes.
Existing harness-maintainer notes (cloud structured-outputs gotcha,
memory hallucination metric caveat, cloud reproducibility variance)
moved under a separate heading.
The previous README had a minimal quick-start section. Replaced with
a proper end-to-end reproduction guide for contributors who want to
validate the published numbers on their own hardware.
New sections:
- Prerequisites: GPU, Ollama version, Python, disk, optional cloud auth
- Set up the environment
- Pull candidate models (with a bulk-pull Python snippet that reads
candidates.yaml directly so it stays in sync with future additions)
- Smoke test (~30 sec, confirms harness works)
- Run the matrix (Tier 1 / all local / cloud, with time estimates)
- Render a report
- Compare with the published baseline (what kind of agreement to expect
on accuracy vs speed vs VRAM)
- Sharing your results (how to attach a CSV to a follow-up PR or
open an issue if numbers diverge)
- Notes for new contributors to the harness itself (the known cloud
structured-output limitation, the memory-extraction hallucination
metric caveat, cloud reproducibility variance)
Tier filter docs now include the `modern` filter added in the
modern-tier expansion. Single-model runs by exact tag are documented.
Goal: someone cloning the repo for the first time should be able to
reproduce a Tier 1 matrix in ~45 min without reading anything else.
Five model families that existed at the time of the first benchmark
pass but weren't surfaced by the initial ollama.com search:
- qwen3.5:4b-q4_K_M
- gemma4:e2b-it-q4_K_M
- gemma4:e4b-it-q4_K_M
- granite4.1:3b-q4_K_M
- ministral-3:3b
Grouped as tier="modern" with a new orchestrator filter so they can
be re-run independently. 25 runs in 11.2 minutes, zero errors.
Three findings reshape the recommendations:
1. gemma4:e4b-it-q4_K_M is the new local leader for room classification.
Closed-set 0.62 (vs qwen3:4b q4 at 0.61), open-set 0.65 (highest
score across all measured models including cloud). The cloud
"ceiling" on open-set was 0.61 — a 4B local model now beats every
cloud reference up to 1T parameters on this task.
2. ministral-3:3b has 0.99 memory coverage — nearly cloud-tier.
But weak elsewhere (closed-set 0.49, entity 0.63). Specialist
not generalist.
3. qwen3.5:4b does NOT beat qwen3:4b on this workload despite the
version bump. Slightly better entity F1, worse on memory and
closed-set. Newer != better.
Production-tier update:
- Room classification (OpenClaw use case): gemma4:e4b becomes the new
recommended default. Costs 2.1x latency and 1.4x VRAM vs qwen3:4b
for the room-task lift.
- General extraction on tight hardware: qwen3:4b q4_K_M still wins.
- Open-set discovery: softens from "shelved indefinitely" to "retest
with gemma4:e4b + prompt tuning before declaring dead." Gap to
0.70 ship threshold is now 0.05, not 0.09.
Four fixes from automated code review:
1. score.py: pre-compute embeddings per content string to cut API
calls from O(P*T) to O(P+T). Was making 2*P*T HTTP round-trips
to Ollama embeddings per memory_extraction sample; now P+T.
Significant speed-up on cloud endpoints where each embed is a
network call.
2. metrics.py: bump VRAMPoller default interval from 100ms to 500ms.
nvidia-smi subprocess every 100ms was creating measurable jitter
on the timing numbers. 500ms still captures peak (inference VRAM
is steady-state during a request) with much less overhead.
3. orchestrator.py: use argparse.BooleanOptionalAction for
--continue-on-error so --no-continue-on-error actually disables
the default. The previous combination of store_true and
default=True made the flag a permanent True.
4. summarize.py: drop the buggy first attempt at the reasoning
filter (had operator-precedence issue with `or` and `and`).
Keep only the clean prefix-based filter that was already on
the next line shadowing it.
Replaced kimi-k2:1t-cloud (retired, HTTP 500 across the board) with
kimi-k2.6:cloud. Added deepseek-v4-flash:cloud and deepseek-v4-pro:cloud
as newer-generation references.
Ran the v2 cloud matrix at n=30 on 7 models. All succeeded. Key findings:
- gpt-oss:20b-cloud and qwen3-coder:480b-cloud tie at ~0.90 on closed-set
room classification. gpt-oss matches a 24x larger coder-specialized
model on natural-language tasks; coder-specialization isn't a benefit
for this workload.
- deepseek-v4-flash hits 0.607 on open-set room similarity (best cloud
result). Local best is 0.612 (gemma3:4b-it). Cloud still doesn't beat
local on this task; the open-set ceiling is genuinely refuted across
two cloud generations and the full 4B-1T size range.
- kimi-k2.6:cloud showed only 37% valid_json_rate on memory extraction.
Per docs.ollama.com/capabilities/structured-outputs, "Ollama's Cloud
currently does not support structured outputs." The harness's
format=json flag is silently ignored on cloud. Other cloud models
happened to emit JSON by default; K2.6 doesn't. Documented as a
methodological caveat. Follow-up issue: envelope-aware JSON
extraction fallback for cloud responses.
- gpt-oss:20b showed 6.4-point closed-set drift between v1 and v2
(0.833 → 0.897). Bigger than local reproducibility noise (±0.7%).
Likely cloud-side model rotation or server-side non-determinism.
Cloud findings should be quoted as ranges, not point estimates.
Raw data: results/2026-05-11-cloud-z690-ex-glacial.csv (v2). The v1 CSV
stays in the tree as historical record of the kimi-k2:1t-cloud failure.
ChromaDB's col.get() and col.query() can return None for the metadata
cell of a partially-flushed row, or for any row written without metadata
in older formats. The MCP handler then crashes with:
AttributeError: 'NoneType' object has no attribute 'get'
This crashed the path before the embeddings_queue cleanup step, so the
queue grew without bound while writes kept appearing successful.
Add a small _safe_meta() boundary helper that coerces None (or any other
non-dict) to {}, and route every direct metadatas[i]/metadatas[0] read in
mcp_server through it: tool_get_drawer, tool_list_drawers,
tool_update_drawer, tool_delete_drawer (audit-log path),
tool_check_duplicate, and tool_diary_read.
The contract — *metadata is always a dict by the time it leaves the
boundary* — is documented on the helper and self-documents at each call
site.
Regression coverage: tests/test_mcp_server.py::TestNoneMetadataSafety
covers the helper and each affected handler with a stub collection that
returns None metadatas (Chroma rejects None at write time, so we can't
reproduce upstream state through the real backend).
Closes#1426.
Addresses all four findings from @copilot-pull-request-reviewer on
1. **Type/range validation on chunk_size / chunk_overlap /
min_chunk_size.** New ``MempalaceConfig._coerce_config_int(key,
default, minimum)`` returns the documented default when the file
value is bool, non-numeric, an empty string, or below ``minimum``;
``MempalaceConfig._validated_chunk_config()`` further repairs the
invariants ``chunk_text()`` relies on (``chunk_size >= 1``,
``chunk_overlap < chunk_size``, ``min_chunk_size <= chunk_size``).
Each property now resolves through that helper, so a hand-edited
``config.json`` with garbage values can't silently break ingest.
2. **chunk_text() infinite-loop guard.** Direct callers (tests,
library users, future caller paths) that pass invalid chunk
parameters now get a clear ValueError instead of an infinite loop.
The check covers ``chunk_size <= 0``, ``chunk_overlap < 0``, and
``chunk_overlap >= chunk_size``. Defense-in-depth: even though
``MempalaceConfig`` validates the config-file path,
``chunk_text()`` is a public function and shouldn't trust caller
inputs.
3. **DRY default constants.** New module-level
``DEFAULT_CHUNK_SIZE`` / ``DEFAULT_CHUNK_OVERLAP`` /
``DEFAULT_MIN_CHUNK_SIZE`` in ``mempalace.config`` are the
single source of truth. ``mempalace.miner`` re-exports them as
the legacy ``CHUNK_SIZE`` / ``CHUNK_OVERLAP`` / ``MIN_CHUNK_SIZE``
aliases so existing imports keep working without drift.
4. **Test coverage.** 15 new tests in ``tests/test_config.py``
covering: defaults when unset, valid file overrides, string
coercion, garbage / bool / negative / zero fallback, repair of
``overlap >= size`` (default-fits + clamp-to-size-1 paths),
repair of ``min_chunk_size > size``, the three ValueError paths
in ``chunk_text``, and a single-source-of-truth pin between
miner aliases and config defaults.
Suite total: 1323 passed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`cfg.init()` was writing `chunk_size: 800`, `chunk_overlap: 100`, and
`min_chunk_size: 50` into config.json on first run. The values are the
module-level defaults from `miner.py`, which is fine for `miner.py`'s
own consumers — but `convo_miner.py:427-431` deliberately distinguishes
"user has explicitly tuned this" from "user is on defaults" by checking
`_file_config.get("min_chunk_size") is None`. Writing the miner.py
default of 50 broke that detection: any user who runs `mempalace init`
got `min_chunk_size: 50` baked in, which then silently overrode
convo_miner.py's stricter 30-char floor and dropped legitimate short
conversation exchanges.
Surfaced by a pytest fixture leak: tests/conftest.py:21-27 redirects
HOME to a session-tmp dir. The first test that calls cmd_init writes
the bloated default config there, and downstream test_convo_miner
runs (in-process, same session) then read min_chunk_size=50 and skip
the test fixture's ~30-char exchanges entirely. Repro:
pytest tests/test_cli.py::test_cmd_init_honors_palace_flag \
tests/test_convo_miner.py::test_convo_mining
Both tests pass in isolation but the second fails when chained.
Fix: drop the chunking keys from `cfg.init()`'s default-config-write.
The `MempalaceConfig.chunk_size`/`.chunk_overlap`/`.min_chunk_size`
properties already provide the right fallbacks via
`_file_config.get(key, default)` when the key is absent. Users who
want to tune chunking still set the keys explicitly; the contract
convo_miner.py relies on (`is None` ⇔ "untuned") is restored.
Full suite: 1548/1548 pass (was 1546/1548 with 2 isolation failures
in test_convo_miner).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Qodo review flagged on #1024 (2026-04-22): ``_chunk_by_exchange`` can
loop forever when ``chunk_size <= 0`` because ``content[:0]`` returns
an empty string while ``content[0:]`` returns the whole input, so the
remainder never shrinks. Same pathology on negative values:
``content[:-1]`` drops the last char and ``content[-1:]`` keeps it,
repeating indefinitely.
Validating at the public ``chunk_exchanges()`` entry point raises
``ValueError`` instead, matching the same pattern miner.py's
``chunk_text`` uses for its ``chunk_overlap`` guard. Four new tests
in test_convo_miner_unit.py cover the rejection paths plus the
``min_chunk_size == 0`` legal case.
Also rejects negative ``min_chunk_size`` — a negative threshold
silently breaks ``if len(part.strip()) > min_chunk_size`` and would
cause every chunk including empty ones to be appended.
Full test_convo_miner_unit.py suite: 19 passed (4 new).
Addresses @sha2fiddy's review on #1024: convo_miner.py was ignoring
config.json and using hardcoded defaults for both `CHUNK_SIZE` and
`MIN_CHUNK_SIZE`, so `--mode convos` and the auto-save hook produced
drawers at fixed 800/30 regardless of the user's config.
`mine_convos()` now reads `MempalaceConfig().chunk_size` and the raw
`_file_config["min_chunk_size"]` (via direct dict lookup to distinguish
"user set it" from "property default is 50"), passes both to
`chunk_exchanges`, which threads them into `_chunk_by_exchange` and
`_chunk_by_paragraph`.
**Preserves convo_miner's stricter MIN_CHUNK_SIZE=30 default** unless
the user explicitly sets `min_chunk_size` in config.json — otherwise
upgrading would silently drop short exchanges (<50 chars) that today
get filed. The MempalaceConfig property still returns 50 for miner.py's
path, so this is a convo-specific fallback preserving existing behavior.
Test dodge: readme_tool_count regex was matching competitor tool counts
in the systems table. Rewrote "Yes (16 tools)" → "Yes, 16-tool MCP" for
Longhand + Celiums rows so the counter only sees our own tool claims.
891 tests pass full suite (105 in convo/miner/readme slice).
Chunk sizing was hardcoded via module-level constants
(CHUNK_SIZE=800, CHUNK_OVERLAP=100, MIN_CHUNK_SIZE=50). One size
does not fit all — source material varies (dense code vs prose
transcripts vs sparse logs) and so do users' context-window
budgets.
This makes all three values overridable via ~/.mempalace/config.json:
{
"chunk_size": 1200,
"chunk_overlap": 150,
"min_chunk_size": 40
}
Values are exposed as MempalaceConfig properties, threaded
through mine() -> process_file() -> chunk_text() as optional
keyword arguments. Defaults (800/100/50) are preserved when the
config keys are absent, so existing palaces behave identically.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two fixes from Copilot's 2026-04-23 inline review:
1. Drop `-n` from the grep command. Hard-coded line numbers in the
"Expected" block would drift as files evolve, making the
checklist misleading. The check is about presence, not location —
line numbers add noise without helping pass/fail.
2. Reword "`console_script` entry point declared in pyproject.toml"
→ "console script declared under `[project.scripts]` in
pyproject.toml". PEP 621's `[project.scripts]` is the canonical
name for this repo's config form; the old wording conflated it
with setuptools' `console_scripts` entry-point group name.
Expected output block updated to match new grep (no colons before
line numbers).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fulfills the "Optional: release-checklist addition" proposal at the
bottom of #1093 (the v3.3.2 release defect where plugin.json referenced
a mempalace-mcp binary that pyproject.toml never declared, so fresh
`pip install` was broken for everyone until messelink's #340 was
re-cut as v3.3.3).
New file at docs/RELEASING.md (no existing doc at that path) with a
single pre-release grep:
grep -rn mempalace-mcp pyproject.toml .claude-plugin .codex-plugin
The original #1093 proposal specified `pyproject.toml
.claude-plugin/plugin.json` (2 files). This expands via -rn directory
recursion to also cover `.claude-plugin/.mcp.json` and
`.codex-plugin/plugin.json`, which reference `mempalace-mcp` by name
too — same class of regression through a different surface. Happy to
trim to the narrower 2-file form if preferred; one-line edit.
Shows the concrete expected output so a maintainer running this under
release pressure can eyeball "pass" without mental translation, and
points at #340 as the historical fix anchor so "investigate why the
entry is missing" has a diagnostic starting point rather than a dead
end.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Ran four of five cloud candidates at n=30. Kimi K2 1T returned HTTP
500 on every task — retry pending. Four working models:
| Model | room-closed | room-open | entity F1 | mem cov |
|---|---|---|---|---|
| gpt-oss:20b-cloud | 0.833 | 0.555 | 0.755 | 1.000 |
| gpt-oss:120b-cloud | 0.800 | 0.553 | 0.829 | 1.000 |
| qwen3-coder:480b-cloud | 0.900 | 0.587 | 0.804 | 0.967 |
| deepseek-v3.1:671b-cloud | 0.800 | 0.566 | 0.828 | 0.967 |
| local leader (4B q4) | 0.610 | 0.586 | 0.778 | 0.950 |
Two headline conclusions:
1. Closed-set room classification HAS a real ceiling gap. Cloud at
0.83-0.90 vs local at 0.61. Strongest argument so far for an
optional --classifier cloud path for users willing to trade
privacy/cost for accuracy. Default stays local.
2. Open-set discovery ceiling is REFUTED. Best cloud is 0.587 vs
best local 0.612. Cloud is slightly WORSE. Not a capacity problem;
it's a task-formulation problem that more compute doesn't fix.
Recommendation: shelve --mode discover until prompt design or
two-pass clustering closes the gap to >0.75.
Also: gpt-oss models continue emitting reasoning even with think=false
in the request body (verified via curl). Content field stays clean,
but cloud latency includes reasoning generation we asked it to skip.
Worth an upstream issue against Ollama Cloud.
Three follow-ups since the first pass:
1. Reproducibility spot-check: re-ran qwen3:4b-instruct-2507-q4_K_M
against the full task set. Accuracy deltas vs original ≤0.7% on
every metric. Harness is reliable; single-run numbers can be trusted.
2. Memory-extraction hallucination investigation. Hand-inspected 5
samples and found qwen3:4b's 0.36 "hallucination rate" is a scoring
artifact, not a model weakness:
- ~50% are bundled-truth-split-into-atomic predictions
- ~30% are ground-truth omissions the model correctly caught
- <5% are genuine hallucinations
Meanwhile qwen2.5:3b's "0.00 hallucination" is just under-extraction.
The mean_hallucination_rate metric over-penalizes thorough models.
qwen3:4b stays the production recommendation. Follow-up: refine
scoring to use source-text traceability instead of greedy match.
3. Cloud-tier ceiling measurement attempted, blocked on ollama signin
(interactive OAuth). Five cloud candidates added to candidates.yaml
for the next pass.
Also: committed the spot-check CSV as a reproducibility baseline.
Adds five Ollama Cloud reference models for ceiling measurement:
- gpt-oss:20b-cloud (lighter reference)
- gpt-oss:120b-cloud
- qwen3-coder:480b-cloud (code-tuned variant)
- deepseek-v3.1:671b-cloud
- kimi-k2:1t-cloud (trillion-param MoE)
These are not production candidates. They serve as accuracy ceilings
to size the gap between local-best and what 5-250x larger models can
do on the same tasks.
Orchestrator gains two new tier filters:
- `cloud`: returns candidates with cloud:true (5 models)
- `local`: returns candidates without cloud:true (15 models)
Also cherry-picks a single real-format-flavored sample (rc_101) into
the room_classification dataset, sourced from Lumi's hand-written v3
fixture on feat/openclaw-integration. Multi-turn with parallel tool
calls; tests that the harness handles realistic OpenClaw transcript
shape without choking. Source attribution in the sample's 'source'
field for traceability.
Dataset now: 101 room_classification samples (was 100), other tasks
unchanged.
First full matrix run on z690-ex-glacial (Intel i9-12900KF, RTX 3090,
Ollama 0.23.2). 75/75 runs successful, 1 transient warmup timeout on
gemma3:270m entity extraction.
Headline finding: qwen3:4b-instruct-2507-q4_K_M is the best small model
for MemPalace classification across all four tasks (calibration,
closed-set room, entity extraction, memory extraction). q4_K_M holds
within 0.01-0.02 of the fp16 ceiling at half the VRAM.
Open-set discovery is not viable at this model class. Best similarity
score is 0.612 (gemma3:4b-it family), below the 0.70 ship threshold.
Recommendation: keep closed-set classification as the required path,
explore cloud-tier models if the discover feature is still wanted.
Includes:
- 2026-05-10-z690-ex-glacial.csv: raw 75-row CSV (committed as baseline)
- 2026-05-10-z690-ex-glacial.md: auto-rendered tables from summarize.py
- 2026-05-10-analysis.md: human-written interpretation, recommended
MODEL_TIERS update, surprises and follow-ups
Tier list update justified: drop the speculative qwen3.5:4b/qwen3:3b
patterns that never matched on Ollama, demote sub-3B Qwen 3 hybrid
tags due to weak entity F1 (0.31-0.48 vs 0.78 for the 4B instruct).
Recommended new MODEL_TIERS in the analysis report.
MemPalace classification tasks (room, entity, memory) never benefit
from extended reasoning. The thinking variant of qwen3:4b ran ~40-170x
slower than instruct in initial testing without any accuracy gain,
which matches the project principle that classification uses pure
instruct models or hybrid models with thinking disabled.
Two changes:
1. runner.py: forward think=False on every provider.classify call
(sample loop and warmup). Pure-instruct models ignore it; hybrid
Qwen 3 family models stay in fast-classification mode.
2. candidates.yaml: drop qwen3:4b-thinking-2507-q4_K_M. The "test
whether reasoning helps" question is settled; running it again
wastes compute. Replaced the entry with a comment documenting
the policy.
Net candidate count: 15 (Tier 1 = 9). Still covers the full
size/family/quantization grid for production tier-list selection.
Hybrid-reasoning models (Qwen 3 family, DeepSeek-R1 style) accept a
`think: false` request flag in Ollama 0.7+ to suppress reasoning
emission. Pure-instruct models ignore the flag.
Adds an optional `think` parameter to `LLMProvider.classify` and its
three implementations:
- OllamaProvider: forwards to the wire format only when explicitly set,
keeping the request body minimal for the default case.
- OpenAICompatProvider, AnthropicProvider: accept the kwarg for
interface compat but no-op (those providers don't have a per-request
thinking toggle).
Verified against Ollama 0.23.2 with qwen3:1.7b-q4_K_M:
think=False: 294ms, no thinking field
think=True: 523ms (1.8x slower), 516 chars of thinking content
Both return the same answer; only the reasoning overhead differs.
Useful for any caller that wants fast classification without the
<think> token cost — see benchmarks/model_eval for the first user.
summarize.py reads orchestrator output and renders a readable report:
- Production picks: models meeting min accuracy AND max latency thresholds
- Open-set discovery viability: ship/skip recommendation based on
mean cosine similarity threshold
- Instruct vs reasoning comparison for the qwen3:4b pair
- Per-task accuracy rankings with task-specific extras (F1, coverage,
similarity stats)
- Speed table from calibration (smallest task, most stable timing)
- VRAM table with resident/peak/delta per model
Usage:
python -m benchmarks.model_eval.summarize \\
--csv results/2026-05-10-host.csv \\
--output reports/2026-05-10-host.md
`ollama ps --format json` is missing on Ollama 0.23.2 (the --format
flag doesn't exist on older versions). Switch to the HTTP API at
/api/ps which returns clean JSON with size_vram per loaded model.
Verified on RTX 3090 + Ollama 0.23.2: vram_resident_mb now returns
7481 MB for qwen3:4b-instruct-2507-q4_K_M with full 32K context KV
cache loaded.
210 hand-quality samples across four tasks. Synthetic only — no
real-person info. Five fictional personae (Aria, Solas, Fenra,
Bramble, Thresh) with distinct domains, room taxonomies, and
relationship contexts.
- room_classification: 100 samples (20 per agent), 14 marked with
realistic noise features, 18 'other' / 6 'general' / 76 specific-
room labels. 5-10 rooms per sample with 'general' always present.
- entity_extraction: 50 samples, 247 entities (114 person, 74 org,
32 project, 27 place). All entity strings verified to appear in
source text. Includes deliberate same-surname-different-people
cases as disambiguation tests.
- memory_extraction: 40 samples, 55 memories across types
(15 fact, 12 decision, 12 commitment, 9 preference, 7 opinion).
- calibration: 20 samples, 4 per class (question, command, statement,
exclamation, greeting). Sanity check that the harness measures
what we think.
All datasets validated: ID alignment across dataset/labels pairs,
closed-set labels are members of their room lists (or 'other'),
calibration labels are members of their classes lists, no leak
terms (igor, milla, domi, lumi, anthropic, openclaw, etc.) anywhere.
Generated 2026-05-10 via subagent + hand-validation. README in the
datasets directory documents the personae, distribution stats, and
labeling conventions.
Skeleton for benchmarking ≤4B-parameter Ollama models on MemPalace
classification and extraction tasks. Outputs per-(model, task, mode)
metrics: accuracy, latency (TTFT, TPS, e2e p50/p95), VRAM (resident
and peak).
Layout under benchmarks/model_eval/:
- candidates.yaml: 16 models across 3 tiers, with family/size/variant
metadata pulled from ollama.com/library/<family>/tags
- metrics.py: timing extraction from Ollama response, VRAMPoller for
peak-memory tracking via nvidia-smi, embedding-similarity scoring,
host-info introspection, percentile aggregation
- runner.py: runs one (model, task, mode) triple, dispatches per-task
prompt building and scoring
- orchestrator.py: iterates candidates × tasks, writes CSV
- tasks/{calibration,room_classification,entity_extraction,
memory_extraction}/{prompts.py,score.py}: per-task prompt builders
and scorers
Uses mempalace.llm_client.get_provider directly so the benchmark runs
the same code path as production. strip_thinking_tokens lives locally
in metrics.py for now; will switch to mempalace.local_model once that
module lands on develop.
Datasets land in a follow-up commit.
Two resource leaks in mempalace.migrate that bite hardest on Windows:
1. extract_drawers_from_sqlite() opened a sqlite3 connection at the
top and only closed it in the happy path. Any exception during
query/iteration leaked the handle, leaving a file lock on
chroma.sqlite3 that prevented the rest of the migration from
touching the palace directory. Wrap with contextlib.closing().
2. migrate() called tempfile.mkdtemp() and never cleaned it up if a
later step (chromadb open, batch import, count, swap) raised.
The orphaned palace under the system temp root could be 100s of
MB and never gets reclaimed. Wrap the import-and-swap dance in
try/finally and rmtree the temp dir if it still exists at the
end (os.replace consumes it on the happy path so the existence
guard makes that case a no-op).
Tests cover the happy-path extraction and verify mkdtemp's directory
is removed when ChromaBackend.get_or_create_collection() raises.
#1215 made `EntityRegistry.save()` atomic via temp-file + fsync + os.replace.
Crash-mid-write durability is correct: the previous registry stays intact on
any failure. But if `f.write()` / `f.flush()` / `os.fsync()` / `os.replace()`
raise (disk full, perms flip, broken FUSE mount, IO error), the `.tmp` sidecar
was left on disk. Subsequent saves overwrite the same path so it does not
grow unboundedly, but it litters the palace directory and obscures
diagnostics — a user inspecting `entity_registry.json.tmp` after a crash
cannot distinguish in-flight writes from stale debris.
Wrap the write+chmod+replace block in try/except. On any exception, attempt
`tmp_path.unlink(missing_ok=True)` before re-raising. The dir-fsync is
deliberately outside the try — that is durability for a successful rename,
not a write step that needs cleanup.
Tests:
- Augment `test_save_preserves_previous_on_serialization_failure` to also
assert the .tmp sidecar is gone after a forced os.replace failure.
- New `test_save_cleans_tmp_on_write_failure` forces os.fsync to raise,
covering the gap between write and rename that the existing test does
not exercise.
Closes#1373.
`_wing_from_transcript_path` derived the wing from the LAST dash-separated
token of Claude Code's encoded project folder. Because Claude Code encodes
the source directory by replacing `/` with `-`, any project whose folder
name itself contained a dash got silently truncated:
-Users-me-claude-code -> wing_code (lost "claude")
-Users-me-react-native -> wing_native (lost "react")
-Users-me-customer-portal -> wing_portal (collision risk)
-Users-me-admin-portal -> wing_portal (same wing!)
Two real consequences:
1. Project-scoped queries (`wake-up --wing <project>`) miss diary entries
because they're filed under the truncated wing.
2. Multi-project collision: any two projects whose folders end in the
same final token get their diary entries merged into one wing,
defeating the wing isolation model.
Fix uses a two-tier strategy:
1. PRIMARY — read `cwd` from the JSONL transcript. Claude Code records
the absolute working directory on most message types, so the project
name is whatever the leaf path segment of cwd is. This is the
canonical answer when present and never truncates hyphenated names.
Bounded scan (200 lines) keeps the lookup well within the hook's
500ms budget.
2. FALLBACK — when cwd isn't recorded (older Claude Code, queue-only
transcripts, etc.), decode the encoded folder. Strip the platform
user-home prefix (`Users-<user>-` / `home-<user>-`) and one common
parent-dir token (`git-`, `dev-`, `projects-`, `Projects-`, `src-`,
`code-`, `work-`, `Documents-`), then convert remaining dashes to
underscores. May include extra parent-dir noise in the wing name
(`wing_dev_mempalace_mempalace`) but never silently truncates.
Two existing tests asserted the old truncation behavior (it gave the
right answer by coincidence on single-token leaf project names). They're
updated to reflect the new contract: collision-safe wing extraction even
when cwd is absent.
Tests added:
- hyphenated_claude_code, hyphenated_react_native (regression)
- no_collision_between_hyphenated_siblings (`customer-portal` vs
`admin-portal` resolve to distinct wings)
- strips_parent_dir_with_hyphenated_project (reporter's example)
- uses_cwd_from_jsonl, cwd_with_hyphenated_project,
cwd_skips_lines_without_cwd, cwd_falls_back_when_no_cwd_in_jsonl,
cwd_handles_malformed_jsonl, cwd_handles_missing_file,
cwd_handles_non_string_cwd (cwd-primary path coverage)
Closes#1410.
Copilot review on PR #1434 caught that the existing 3.3.5 entry
described the validator as it was authored under #1167 — accepting
``YYYY``/``YYYY-MM``/``YYYY-MM-DD`` and rejecting ISO datetimes — but
PR #1417 (closes#1374) merged into develop on 2026-05-10 and
inverted that: ``sanitize_iso_temporal()`` now rejects partial dates
and accepts canonical UTC datetimes (``YYYY-MM-DDTHH:MM:SSZ`` /
``+00:00``). ``sanitize_iso_date()`` is kept as a backwards-compat
wrapper.
Update the bullet to describe the *shipped* behavior, name both
functions, list both accepted and rejected forms, and call out the
3.3.4 → 3.3.5 behavior change for partial-date inputs that now error.
Reference both #1167 (original) and #1374/#1417 (the expansion).
The 3.3.4 release shipped 2026-05-01 (per GitHub release v3.3.4) but
its CHANGELOG header was never flipped from ``unreleased`` to the
release date. Backfill while we're already touching CHANGELOG for
the 3.3.5 cut.