Without an explicit num_ctx, each candidate ran at its Modelfile default
(32k for the Gemma4 variants, larger for qwen3), so VRAM and latency
weren't comparable across families — a 32k-default model pre-allocates
KV cache a 4k-default model doesn't. The flag's own docstring promised
"apples-to-apples" but defaulted to None, defeating the intent.
All current benchmark prompts fit comfortably under 4k tokens
(memory_extraction is the longest at ~500). Users with longer prompts
can still pass --num-ctx <larger>.
Adds a methodology note to the 2026-05-13 multilingual report so its
VRAM/latency numbers aren't conflated with future runs at the new default.
Addresses the review feedback from igorls, gemini-code-assist, and Copilot.
HIGH:
- orchestrator: --output single-file mode now shares ONE (fh, writer) across
all languages instead of opening N handles to the same path. The old code
caused interleaved buffer corruption: first language opened "w", subsequent
ones opened "a", and writes from independent file offsets could overwrite
each other. Verified with a multi-language --output smoke test (4 rows
written, all distinct).
- 19 untranslated/empty samples re-translated:
- dataset.de.jsonl: cal_017
- dataset.hi.jsonl entity_extraction: ent_020, ent_025, ent_032, ent_038
- dataset.hi.jsonl room_classification: rc_017, rc_026, rc_028, rc_040,
rc_064, rc_089, rc_091
- dataset.ko.jsonl room_classification: rc_027, rc_067
- dataset.it.jsonl room_classification: rc_029, rc_030, rc_031, rc_032,
rc_053 (previously empty strings)
- labels.ko.jsonl: restored all proper nouns to English (Doreth, Saela, Ivora,
Ren Solanke, Pol Krisat, Pell Halloran, Bramble, Hollowmounts Institute,
Wendelsea, Bridgewater Community Garden, Wends, Drukar, Aerwyn cycle,
Jaccard, Mason bee, Markdown). Also fixed mistranslation 유전자 사과
(genetic apple) → 재래종 사과 (heirloom apple).
MEDIUM:
- runner.py: refactored label-resolution one-liner into 3 readable lines
and added an info log when falling back to English ground truth, so
readers don't misread "score collapse" as model failure.
LOW:
- orchestrator: moved `import socket` to module top (PEP 8); removed
unused `out_path` from the unpacking tuple.
- translate_datasets.py: renamed loop variable `l` → `code` (ruff E741);
made the _translate_one fallback return path explicit instead of relying
on for-loop fall-through; added a privacy warning in the docstring
flagging that the default `kimi-k2.6:cloud` sends prose to a remote
endpoint and should not be used over real palace data.
- 2026-05-13-multilingual.md: converted analytical paragraphs from
Portuguese to English to match the existing repo convention.
Adds 6 new language datasets (German, French, Hindi, Italian, Korean, Russian)
across all 4 benchmark tasks (calibration, entity_extraction, memory_extraction,
room_classification) — 630 samples total, same conventions as the existing
pt-BR/es/zh datasets: inputs translated, labels/ground-truth stay English
except where noted.
Changes:
- 24 new dataset.{de,fr,hi,it,ko,ru}.jsonl files across all 4 tasks
- labels.ko.jsonl for memory_extraction: Korean ground-truth so the scorer
compares Korean model output against Korean expected content instead of
English (fixes ~20pp score gap identified during testing — see report)
- runner.py: loads labels.{lang}.jsonl when present, falls back to labels.jsonl
- orchestrator.py: adds --output-dir (writes <dir>/<lang>/YYYY-MM-DD-<host>.csv
per language); --output single-file mode unchanged
- candidates.yaml: adds community tier (igorls classifier variants, heretic)
and local tier (gemma4:e4b)
- translate_datasets.py: script used to generate the translations via Ollama;
included so contributors can extend to new languages without manual work
- reports/2026-05-13-multilingual.md: 210-run benchmark report across
6 models × 7 languages × 5 tasks on RTX 3080 Laptop 8 GB
Five model families that existed at the time of the first benchmark
pass but weren't surfaced by the initial ollama.com search:
- qwen3.5:4b-q4_K_M
- gemma4:e2b-it-q4_K_M
- gemma4:e4b-it-q4_K_M
- granite4.1:3b-q4_K_M
- ministral-3:3b
Grouped as tier="modern" with a new orchestrator filter so they can
be re-run independently. 25 runs in 11.2 minutes, zero errors.
Three findings reshape the recommendations:
1. gemma4:e4b-it-q4_K_M is the new local leader for room classification.
Closed-set 0.62 (vs qwen3:4b q4 at 0.61), open-set 0.65 (highest
score across all measured models including cloud). The cloud
"ceiling" on open-set was 0.61 — a 4B local model now beats every
cloud reference up to 1T parameters on this task.
2. ministral-3:3b has 0.99 memory coverage — nearly cloud-tier.
But weak elsewhere (closed-set 0.49, entity 0.63). Specialist
not generalist.
3. qwen3.5:4b does NOT beat qwen3:4b on this workload despite the
version bump. Slightly better entity F1, worse on memory and
closed-set. Newer != better.
Production-tier update:
- Room classification (OpenClaw use case): gemma4:e4b becomes the new
recommended default. Costs 2.1x latency and 1.4x VRAM vs qwen3:4b
for the room-task lift.
- General extraction on tight hardware: qwen3:4b q4_K_M still wins.
- Open-set discovery: softens from "shelved indefinitely" to "retest
with gemma4:e4b + prompt tuning before declaring dead." Gap to
0.70 ship threshold is now 0.05, not 0.09.
Replaced kimi-k2:1t-cloud (retired, HTTP 500 across the board) with
kimi-k2.6:cloud. Added deepseek-v4-flash:cloud and deepseek-v4-pro:cloud
as newer-generation references.
Ran the v2 cloud matrix at n=30 on 7 models. All succeeded. Key findings:
- gpt-oss:20b-cloud and qwen3-coder:480b-cloud tie at ~0.90 on closed-set
room classification. gpt-oss matches a 24x larger coder-specialized
model on natural-language tasks; coder-specialization isn't a benefit
for this workload.
- deepseek-v4-flash hits 0.607 on open-set room similarity (best cloud
result). Local best is 0.612 (gemma3:4b-it). Cloud still doesn't beat
local on this task; the open-set ceiling is genuinely refuted across
two cloud generations and the full 4B-1T size range.
- kimi-k2.6:cloud showed only 37% valid_json_rate on memory extraction.
Per docs.ollama.com/capabilities/structured-outputs, "Ollama's Cloud
currently does not support structured outputs." The harness's
format=json flag is silently ignored on cloud. Other cloud models
happened to emit JSON by default; K2.6 doesn't. Documented as a
methodological caveat. Follow-up issue: envelope-aware JSON
extraction fallback for cloud responses.
- gpt-oss:20b showed 6.4-point closed-set drift between v1 and v2
(0.833 → 0.897). Bigger than local reproducibility noise (±0.7%).
Likely cloud-side model rotation or server-side non-determinism.
Cloud findings should be quoted as ranges, not point estimates.
Raw data: results/2026-05-11-cloud-z690-ex-glacial.csv (v2). The v1 CSV
stays in the tree as historical record of the kimi-k2:1t-cloud failure.
Ran four of five cloud candidates at n=30. Kimi K2 1T returned HTTP
500 on every task — retry pending. Four working models:
| Model | room-closed | room-open | entity F1 | mem cov |
|---|---|---|---|---|
| gpt-oss:20b-cloud | 0.833 | 0.555 | 0.755 | 1.000 |
| gpt-oss:120b-cloud | 0.800 | 0.553 | 0.829 | 1.000 |
| qwen3-coder:480b-cloud | 0.900 | 0.587 | 0.804 | 0.967 |
| deepseek-v3.1:671b-cloud | 0.800 | 0.566 | 0.828 | 0.967 |
| local leader (4B q4) | 0.610 | 0.586 | 0.778 | 0.950 |
Two headline conclusions:
1. Closed-set room classification HAS a real ceiling gap. Cloud at
0.83-0.90 vs local at 0.61. Strongest argument so far for an
optional --classifier cloud path for users willing to trade
privacy/cost for accuracy. Default stays local.
2. Open-set discovery ceiling is REFUTED. Best cloud is 0.587 vs
best local 0.612. Cloud is slightly WORSE. Not a capacity problem;
it's a task-formulation problem that more compute doesn't fix.
Recommendation: shelve --mode discover until prompt design or
two-pass clustering closes the gap to >0.75.
Also: gpt-oss models continue emitting reasoning even with think=false
in the request body (verified via curl). Content field stays clean,
but cloud latency includes reasoning generation we asked it to skip.
Worth an upstream issue against Ollama Cloud.
Three follow-ups since the first pass:
1. Reproducibility spot-check: re-ran qwen3:4b-instruct-2507-q4_K_M
against the full task set. Accuracy deltas vs original ≤0.7% on
every metric. Harness is reliable; single-run numbers can be trusted.
2. Memory-extraction hallucination investigation. Hand-inspected 5
samples and found qwen3:4b's 0.36 "hallucination rate" is a scoring
artifact, not a model weakness:
- ~50% are bundled-truth-split-into-atomic predictions
- ~30% are ground-truth omissions the model correctly caught
- <5% are genuine hallucinations
Meanwhile qwen2.5:3b's "0.00 hallucination" is just under-extraction.
The mean_hallucination_rate metric over-penalizes thorough models.
qwen3:4b stays the production recommendation. Follow-up: refine
scoring to use source-text traceability instead of greedy match.
3. Cloud-tier ceiling measurement attempted, blocked on ollama signin
(interactive OAuth). Five cloud candidates added to candidates.yaml
for the next pass.
Also: committed the spot-check CSV as a reproducibility baseline.
First full matrix run on z690-ex-glacial (Intel i9-12900KF, RTX 3090,
Ollama 0.23.2). 75/75 runs successful, 1 transient warmup timeout on
gemma3:270m entity extraction.
Headline finding: qwen3:4b-instruct-2507-q4_K_M is the best small model
for MemPalace classification across all four tasks (calibration,
closed-set room, entity extraction, memory extraction). q4_K_M holds
within 0.01-0.02 of the fp16 ceiling at half the VRAM.
Open-set discovery is not viable at this model class. Best similarity
score is 0.612 (gemma3:4b-it family), below the 0.70 ship threshold.
Recommendation: keep closed-set classification as the required path,
explore cloud-tier models if the discover feature is still wanted.
Includes:
- 2026-05-10-z690-ex-glacial.csv: raw 75-row CSV (committed as baseline)
- 2026-05-10-z690-ex-glacial.md: auto-rendered tables from summarize.py
- 2026-05-10-analysis.md: human-written interpretation, recommended
MODEL_TIERS update, surprises and follow-ups
Tier list update justified: drop the speculative qwen3.5:4b/qwen3:3b
patterns that never matched on Ollama, demote sub-3B Qwen 3 hybrid
tags due to weak entity F1 (0.31-0.48 vs 0.78 for the 4B instruct).
Recommended new MODEL_TIERS in the analysis report.
Skeleton for benchmarking ≤4B-parameter Ollama models on MemPalace
classification and extraction tasks. Outputs per-(model, task, mode)
metrics: accuracy, latency (TTFT, TPS, e2e p50/p95), VRAM (resident
and peak).
Layout under benchmarks/model_eval/:
- candidates.yaml: 16 models across 3 tiers, with family/size/variant
metadata pulled from ollama.com/library/<family>/tags
- metrics.py: timing extraction from Ollama response, VRAMPoller for
peak-memory tracking via nvidia-smi, embedding-similarity scoring,
host-info introspection, percentile aggregation
- runner.py: runs one (model, task, mode) triple, dispatches per-task
prompt building and scoring
- orchestrator.py: iterates candidates × tasks, writes CSV
- tasks/{calibration,room_classification,entity_extraction,
memory_extraction}/{prompts.py,score.py}: per-task prompt builders
and scorers
Uses mempalace.llm_client.get_provider directly so the benchmark runs
the same code path as production. strip_thinking_tokens lives locally
in metrics.py for now; will switch to mempalace.local_model once that
module lands on develop.
Datasets land in a follow-up commit.
Igor Lins e Silva
Bruno Leal