The previous commit replaced llm_model.split('-')[1] on the diary pre-compute
line, but run_benchmark builds the same short name again 35 lines later for
the summary header. That one is dash-guarded so it does not crash, yet it
still guesses at a short name, so a single run printed the full tag on one
line and a mangled fragment on the next: "claude-sonnet-4-6" became "sonnet",
and "llama3.2-vision:11b" became "vision:11b".
No dash-split of a model tag is left anywhere under benchmarks/.
llm_model.split('-')[1] raises IndexError for any model tag that contains
no dash (e.g. an Ollama tag like 'qwen3:8b' or an OpenAI-compat route like
'spark/fast'), crashing locomo_bench.py at startup when --llm-rerank is
enabled and longmemeval_bench.py on the diary pre-compute path. Both sites
only build display strings, so print the full model tag instead of
guessing at a short name.
Hit in practice when pointing --llm-backend ollama at a LiteLLM gateway
with provider-prefixed model routes.
Resolves conflicts in CHANGELOG.md and pyproject.toml by combining
the multilingual-embedder additions (huggingface_hub/tokenizers/numpy
core deps, [multilingual] alias, Features section) with develop's
additions (python-dateutil core dep, [extract] extra, tunnel Bug
Fixes and Internal sections).
Prepares PR #1483 for merge into v3.3.6.
Without an explicit num_ctx, each candidate ran at its Modelfile default
(32k for the Gemma4 variants, larger for qwen3), so VRAM and latency
weren't comparable across families — a 32k-default model pre-allocates
KV cache a 4k-default model doesn't. The flag's own docstring promised
"apples-to-apples" but defaulted to None, defeating the intent.
All current benchmark prompts fit comfortably under 4k tokens
(memory_extraction is the longest at ~500). Users with longer prompts
can still pass --num-ctx <larger>.
Adds a methodology note to the 2026-05-13 multilingual report so its
VRAM/latency numbers aren't conflated with future runs at the new default.
Addresses the review feedback from igorls, gemini-code-assist, and Copilot.
HIGH:
- orchestrator: --output single-file mode now shares ONE (fh, writer) across
all languages instead of opening N handles to the same path. The old code
caused interleaved buffer corruption: first language opened "w", subsequent
ones opened "a", and writes from independent file offsets could overwrite
each other. Verified with a multi-language --output smoke test (4 rows
written, all distinct).
- 19 untranslated/empty samples re-translated:
- dataset.de.jsonl: cal_017
- dataset.hi.jsonl entity_extraction: ent_020, ent_025, ent_032, ent_038
- dataset.hi.jsonl room_classification: rc_017, rc_026, rc_028, rc_040,
rc_064, rc_089, rc_091
- dataset.ko.jsonl room_classification: rc_027, rc_067
- dataset.it.jsonl room_classification: rc_029, rc_030, rc_031, rc_032,
rc_053 (previously empty strings)
- labels.ko.jsonl: restored all proper nouns to English (Doreth, Saela, Ivora,
Ren Solanke, Pol Krisat, Pell Halloran, Bramble, Hollowmounts Institute,
Wendelsea, Bridgewater Community Garden, Wends, Drukar, Aerwyn cycle,
Jaccard, Mason bee, Markdown). Also fixed mistranslation 유전자 사과
(genetic apple) → 재래종 사과 (heirloom apple).
MEDIUM:
- runner.py: refactored label-resolution one-liner into 3 readable lines
and added an info log when falling back to English ground truth, so
readers don't misread "score collapse" as model failure.
LOW:
- orchestrator: moved `import socket` to module top (PEP 8); removed
unused `out_path` from the unpacking tuple.
- translate_datasets.py: renamed loop variable `l` → `code` (ruff E741);
made the _translate_one fallback return path explicit instead of relying
on for-loop fall-through; added a privacy warning in the docstring
flagging that the default `kimi-k2.6:cloud` sends prose to a remote
endpoint and should not be used over real palace data.
- 2026-05-13-multilingual.md: converted analytical paragraphs from
Portuguese to English to match the existing repo convention.
Adds 6 new language datasets (German, French, Hindi, Italian, Korean, Russian)
across all 4 benchmark tasks (calibration, entity_extraction, memory_extraction,
room_classification) — 630 samples total, same conventions as the existing
pt-BR/es/zh datasets: inputs translated, labels/ground-truth stay English
except where noted.
Changes:
- 24 new dataset.{de,fr,hi,it,ko,ru}.jsonl files across all 4 tasks
- labels.ko.jsonl for memory_extraction: Korean ground-truth so the scorer
compares Korean model output against Korean expected content instead of
English (fixes ~20pp score gap identified during testing — see report)
- runner.py: loads labels.{lang}.jsonl when present, falls back to labels.jsonl
- orchestrator.py: adds --output-dir (writes <dir>/<lang>/YYYY-MM-DD-<host>.csv
per language); --output single-file mode unchanged
- candidates.yaml: adds community tier (igorls classifier variants, heretic)
and local tier (gemma4:e4b)
- translate_datasets.py: script used to generate the translations via Ollama;
included so contributors can extend to new languages without manual work
- reports/2026-05-13-multilingual.md: 210-run benchmark report across
6 models × 7 languages × 5 tasks on RTX 3080 Laptop 8 GB
Addresses Copilot + gemini-code-assist review on #1483.
1. Path-traversal guard for --language. The value is interpolated into
the dataset filename (`dataset.{language}.jsonl`), so unvalidated
input could escape `task_dir`. Now:
- regex `^[A-Za-z][A-Za-z0-9]*(?:[_-][A-Za-z0-9]+)?$` accepts en,
pt-BR, zh-CN, fr_CA, etc. and rejects anything with path separators
or `..`
- belt-and-suspenders `Path.resolve().is_relative_to(task_dir)` check
before opening the file
2. --embed-endpoint now defaults to None and is resolved after parsing:
uses --endpoint when --llm-provider=ollama (so remote benchmark
runs score against the same host), else http://localhost:11434.
Help text now matches behavior. runner.py's CLI was also missing the
flag entirely — added so single-task runs honor remote endpoints.
Enables shipping decisions for non-English users and fair comparison across
candidates whose Modelfile defaults disagree.
- --language / --languages: load dataset.{lang}.jsonl alongside the base
dataset.jsonl. CSV gains a language column. Synthesized candidate
entries let ad-hoc model tags run without editing candidates.yaml.
- --num-ctx: force Ollama options.num_ctx per request, overriding the
model's Modelfile default. Required for apples-to-apples VRAM/TPS
(qwen3:4b-q8 defaults to 32k = 9.7 GB resident; at 8k it's 5.6 GB).
- --embed-model: thread the semantic-similarity embedding model through
scoring. Default flips to embeddinggemma (was nomic-embed-text v1).
Reason: v1 cosine on EN<->PT-BR same-meaning pairs sits at ~0.607
(right at the 0.6 match threshold), so any phrasing drift collapses
to false-negative. embeddinggemma lands ~0.766 with 2.7x the
signal/noise spread. PT-BR memory_extraction recovered 0.15 -> 0.85
on the same outputs after the swap.
Datasets: 12 new files (pt-BR/es/zh x 4 tasks, 633 samples). Input text
translated; proper nouns and labels stay English so cross-lingual
scoring against the existing labels.jsonl works without re-translation.
Tasks that score via semantic similarity (memory_extraction,
room_classification open-set) depend on nomic-embed-text. If the model
was absent, embed_text() returned None silently and every sample scored
0.0 — indistinguishable from a genuinely bad model.
_ensure_embed_model() now probes the endpoint before the warmup step.
On a 404 it pulls the model automatically via `ollama pull`, verifies
the endpoint responds, and only then proceeds. Fixes:
- OLLAMA_HOST is now forwarded to `ollama pull` so non-localhost
endpoints are targeted correctly
- FileNotFoundError is caught and re-raised as RuntimeError with an
install hint if ollama is not on PATH
- Diagnostic prints go to stderr, not stdout
- If the pull or re-probe fails, returns a Result with a clear error
message instead of silent zeros
Restores --candidates local to its documented meaning (all non-cloud
models). Adds --candidates community for third-party GGUF variants.
Ten fixes across code, documentation, and committed baselines:
Code fixes:
1. prompts.py: closed-set prompt said "no punctuation" while room
slugs contain "/". Replaced with explicit "copy the slug verbatim
including / and -" wording. Empirical impact on accuracy is within
reproducibility noise (qwen3:4b q4 was 0.610, retest shows 0.594
on the now-101-sample dataset — same vs new sample noise).
2. orchestrator.py: orchestrator now opens the CSV once, writes the
header, and flushes after each row. README's "writes incrementally,
safe to Ctrl-C" claim is now accurate. Long matrix runs no longer
lose all progress on a crash.
3. metrics.py: gather_host_info() now strips trailing newline from
`hostname` command output. Stops embedded newlines from leaking
into committed CSVs.
4. runner.py: passed `endpoint=endpoint` to vram_resident_mb() so
non-default Ollama endpoints are queried correctly.
5. summarize.py: CSV reader and report writer now use
encoding="utf-8" with explicit newline handling for cross-platform
consistency.
Documentation fixes:
6. entity_extraction/score.py: docstring claimed "case-sensitive on
type" while implementation lowercases both. Corrected.
7. dataset.jsonl rc_101: removed mempalace-private URL since the PR
lands on the public repo. Will be linkable after OpenClaw merges.
8. datasets/README.md: sample counts updated (room_classification 100
→ 101, total 210 → 211) to match the cherry-picked real-format
sample.
9. runner.py / orchestrator.py: docstring example paths corrected
(tests/fixtures/mempalace_bench → benchmarks/model_eval/datasets).
Baseline regeneration:
10. Stripped embedded \n from `host` column in all five committed
CSVs (2026-05-10 main local + spotcheck, 2026-05-10 + 2026-05-11
cloud, 2026-05-11 modern). 165 rows cleaned in place via Python
csv reader/writer. Each row is now line-oriented for diff and
grep friendliness without changing any measured values.
Adds a "Contributions welcome" section to the README covering:
1. Adding models we missed. The published candidate list is one
engineer + one search pass. Particularly call out function-calling-
tuned variants (Phi-4 mini, Nemotron-mini), research-lab instruct
models (Hermes, Dolphin), and quantization-aware-trained variants
as worth-adding categories.
2. Adding other inference backends. mempalace.llm_client.get_provider
already supports openai-compat, which means LM Studio, llama.cpp
server, vLLM, unsloth studio, Docker Model Runner, HF TGI all
plug in with minimal harness changes. Documents the four pieces a
backend contributor would need to touch:
- Backend-selection flag in runner/orchestrator
- Backend-specific timing extraction in metrics.py
- Backend-specific VRAM probe
- candidates.yaml backend field
Notes that accuracy numbers stay comparable across backends since
the dataset and scoring code are backend-agnostic.
Also calls out niche runtimes worth cross-comparison: Apple MLX,
Intel OpenVINO, AMD ROCm, edge-device runtimes.
Existing harness-maintainer notes (cloud structured-outputs gotcha,
memory hallucination metric caveat, cloud reproducibility variance)
moved under a separate heading.
The previous README had a minimal quick-start section. Replaced with
a proper end-to-end reproduction guide for contributors who want to
validate the published numbers on their own hardware.
New sections:
- Prerequisites: GPU, Ollama version, Python, disk, optional cloud auth
- Set up the environment
- Pull candidate models (with a bulk-pull Python snippet that reads
candidates.yaml directly so it stays in sync with future additions)
- Smoke test (~30 sec, confirms harness works)
- Run the matrix (Tier 1 / all local / cloud, with time estimates)
- Render a report
- Compare with the published baseline (what kind of agreement to expect
on accuracy vs speed vs VRAM)
- Sharing your results (how to attach a CSV to a follow-up PR or
open an issue if numbers diverge)
- Notes for new contributors to the harness itself (the known cloud
structured-output limitation, the memory-extraction hallucination
metric caveat, cloud reproducibility variance)
Tier filter docs now include the `modern` filter added in the
modern-tier expansion. Single-model runs by exact tag are documented.
Goal: someone cloning the repo for the first time should be able to
reproduce a Tier 1 matrix in ~45 min without reading anything else.
Five model families that existed at the time of the first benchmark
pass but weren't surfaced by the initial ollama.com search:
- qwen3.5:4b-q4_K_M
- gemma4:e2b-it-q4_K_M
- gemma4:e4b-it-q4_K_M
- granite4.1:3b-q4_K_M
- ministral-3:3b
Grouped as tier="modern" with a new orchestrator filter so they can
be re-run independently. 25 runs in 11.2 minutes, zero errors.
Three findings reshape the recommendations:
1. gemma4:e4b-it-q4_K_M is the new local leader for room classification.
Closed-set 0.62 (vs qwen3:4b q4 at 0.61), open-set 0.65 (highest
score across all measured models including cloud). The cloud
"ceiling" on open-set was 0.61 — a 4B local model now beats every
cloud reference up to 1T parameters on this task.
2. ministral-3:3b has 0.99 memory coverage — nearly cloud-tier.
But weak elsewhere (closed-set 0.49, entity 0.63). Specialist
not generalist.
3. qwen3.5:4b does NOT beat qwen3:4b on this workload despite the
version bump. Slightly better entity F1, worse on memory and
closed-set. Newer != better.
Production-tier update:
- Room classification (OpenClaw use case): gemma4:e4b becomes the new
recommended default. Costs 2.1x latency and 1.4x VRAM vs qwen3:4b
for the room-task lift.
- General extraction on tight hardware: qwen3:4b q4_K_M still wins.
- Open-set discovery: softens from "shelved indefinitely" to "retest
with gemma4:e4b + prompt tuning before declaring dead." Gap to
0.70 ship threshold is now 0.05, not 0.09.
Four fixes from automated code review:
1. score.py: pre-compute embeddings per content string to cut API
calls from O(P*T) to O(P+T). Was making 2*P*T HTTP round-trips
to Ollama embeddings per memory_extraction sample; now P+T.
Significant speed-up on cloud endpoints where each embed is a
network call.
2. metrics.py: bump VRAMPoller default interval from 100ms to 500ms.
nvidia-smi subprocess every 100ms was creating measurable jitter
on the timing numbers. 500ms still captures peak (inference VRAM
is steady-state during a request) with much less overhead.
3. orchestrator.py: use argparse.BooleanOptionalAction for
--continue-on-error so --no-continue-on-error actually disables
the default. The previous combination of store_true and
default=True made the flag a permanent True.
4. summarize.py: drop the buggy first attempt at the reasoning
filter (had operator-precedence issue with `or` and `and`).
Keep only the clean prefix-based filter that was already on
the next line shadowing it.
Replaced kimi-k2:1t-cloud (retired, HTTP 500 across the board) with
kimi-k2.6:cloud. Added deepseek-v4-flash:cloud and deepseek-v4-pro:cloud
as newer-generation references.
Ran the v2 cloud matrix at n=30 on 7 models. All succeeded. Key findings:
- gpt-oss:20b-cloud and qwen3-coder:480b-cloud tie at ~0.90 on closed-set
room classification. gpt-oss matches a 24x larger coder-specialized
model on natural-language tasks; coder-specialization isn't a benefit
for this workload.
- deepseek-v4-flash hits 0.607 on open-set room similarity (best cloud
result). Local best is 0.612 (gemma3:4b-it). Cloud still doesn't beat
local on this task; the open-set ceiling is genuinely refuted across
two cloud generations and the full 4B-1T size range.
- kimi-k2.6:cloud showed only 37% valid_json_rate on memory extraction.
Per docs.ollama.com/capabilities/structured-outputs, "Ollama's Cloud
currently does not support structured outputs." The harness's
format=json flag is silently ignored on cloud. Other cloud models
happened to emit JSON by default; K2.6 doesn't. Documented as a
methodological caveat. Follow-up issue: envelope-aware JSON
extraction fallback for cloud responses.
- gpt-oss:20b showed 6.4-point closed-set drift between v1 and v2
(0.833 → 0.897). Bigger than local reproducibility noise (±0.7%).
Likely cloud-side model rotation or server-side non-determinism.
Cloud findings should be quoted as ranges, not point estimates.
Raw data: results/2026-05-11-cloud-z690-ex-glacial.csv (v2). The v1 CSV
stays in the tree as historical record of the kimi-k2:1t-cloud failure.
Ran four of five cloud candidates at n=30. Kimi K2 1T returned HTTP
500 on every task — retry pending. Four working models:
| Model | room-closed | room-open | entity F1 | mem cov |
|---|---|---|---|---|
| gpt-oss:20b-cloud | 0.833 | 0.555 | 0.755 | 1.000 |
| gpt-oss:120b-cloud | 0.800 | 0.553 | 0.829 | 1.000 |
| qwen3-coder:480b-cloud | 0.900 | 0.587 | 0.804 | 0.967 |
| deepseek-v3.1:671b-cloud | 0.800 | 0.566 | 0.828 | 0.967 |
| local leader (4B q4) | 0.610 | 0.586 | 0.778 | 0.950 |
Two headline conclusions:
1. Closed-set room classification HAS a real ceiling gap. Cloud at
0.83-0.90 vs local at 0.61. Strongest argument so far for an
optional --classifier cloud path for users willing to trade
privacy/cost for accuracy. Default stays local.
2. Open-set discovery ceiling is REFUTED. Best cloud is 0.587 vs
best local 0.612. Cloud is slightly WORSE. Not a capacity problem;
it's a task-formulation problem that more compute doesn't fix.
Recommendation: shelve --mode discover until prompt design or
two-pass clustering closes the gap to >0.75.
Also: gpt-oss models continue emitting reasoning even with think=false
in the request body (verified via curl). Content field stays clean,
but cloud latency includes reasoning generation we asked it to skip.
Worth an upstream issue against Ollama Cloud.
Three follow-ups since the first pass:
1. Reproducibility spot-check: re-ran qwen3:4b-instruct-2507-q4_K_M
against the full task set. Accuracy deltas vs original ≤0.7% on
every metric. Harness is reliable; single-run numbers can be trusted.
2. Memory-extraction hallucination investigation. Hand-inspected 5
samples and found qwen3:4b's 0.36 "hallucination rate" is a scoring
artifact, not a model weakness:
- ~50% are bundled-truth-split-into-atomic predictions
- ~30% are ground-truth omissions the model correctly caught
- <5% are genuine hallucinations
Meanwhile qwen2.5:3b's "0.00 hallucination" is just under-extraction.
The mean_hallucination_rate metric over-penalizes thorough models.
qwen3:4b stays the production recommendation. Follow-up: refine
scoring to use source-text traceability instead of greedy match.
3. Cloud-tier ceiling measurement attempted, blocked on ollama signin
(interactive OAuth). Five cloud candidates added to candidates.yaml
for the next pass.
Also: committed the spot-check CSV as a reproducibility baseline.
Adds five Ollama Cloud reference models for ceiling measurement:
- gpt-oss:20b-cloud (lighter reference)
- gpt-oss:120b-cloud
- qwen3-coder:480b-cloud (code-tuned variant)
- deepseek-v3.1:671b-cloud
- kimi-k2:1t-cloud (trillion-param MoE)
These are not production candidates. They serve as accuracy ceilings
to size the gap between local-best and what 5-250x larger models can
do on the same tasks.
Orchestrator gains two new tier filters:
- `cloud`: returns candidates with cloud:true (5 models)
- `local`: returns candidates without cloud:true (15 models)
Also cherry-picks a single real-format-flavored sample (rc_101) into
the room_classification dataset, sourced from Lumi's hand-written v3
fixture on feat/openclaw-integration. Multi-turn with parallel tool
calls; tests that the harness handles realistic OpenClaw transcript
shape without choking. Source attribution in the sample's 'source'
field for traceability.
Dataset now: 101 room_classification samples (was 100), other tasks
unchanged.
First full matrix run on z690-ex-glacial (Intel i9-12900KF, RTX 3090,
Ollama 0.23.2). 75/75 runs successful, 1 transient warmup timeout on
gemma3:270m entity extraction.
Headline finding: qwen3:4b-instruct-2507-q4_K_M is the best small model
for MemPalace classification across all four tasks (calibration,
closed-set room, entity extraction, memory extraction). q4_K_M holds
within 0.01-0.02 of the fp16 ceiling at half the VRAM.
Open-set discovery is not viable at this model class. Best similarity
score is 0.612 (gemma3:4b-it family), below the 0.70 ship threshold.
Recommendation: keep closed-set classification as the required path,
explore cloud-tier models if the discover feature is still wanted.
Includes:
- 2026-05-10-z690-ex-glacial.csv: raw 75-row CSV (committed as baseline)
- 2026-05-10-z690-ex-glacial.md: auto-rendered tables from summarize.py
- 2026-05-10-analysis.md: human-written interpretation, recommended
MODEL_TIERS update, surprises and follow-ups
Tier list update justified: drop the speculative qwen3.5:4b/qwen3:3b
patterns that never matched on Ollama, demote sub-3B Qwen 3 hybrid
tags due to weak entity F1 (0.31-0.48 vs 0.78 for the 4B instruct).
Recommended new MODEL_TIERS in the analysis report.
MemPalace classification tasks (room, entity, memory) never benefit
from extended reasoning. The thinking variant of qwen3:4b ran ~40-170x
slower than instruct in initial testing without any accuracy gain,
which matches the project principle that classification uses pure
instruct models or hybrid models with thinking disabled.
Two changes:
1. runner.py: forward think=False on every provider.classify call
(sample loop and warmup). Pure-instruct models ignore it; hybrid
Qwen 3 family models stay in fast-classification mode.
2. candidates.yaml: drop qwen3:4b-thinking-2507-q4_K_M. The "test
whether reasoning helps" question is settled; running it again
wastes compute. Replaced the entry with a comment documenting
the policy.
Net candidate count: 15 (Tier 1 = 9). Still covers the full
size/family/quantization grid for production tier-list selection.
summarize.py reads orchestrator output and renders a readable report:
- Production picks: models meeting min accuracy AND max latency thresholds
- Open-set discovery viability: ship/skip recommendation based on
mean cosine similarity threshold
- Instruct vs reasoning comparison for the qwen3:4b pair
- Per-task accuracy rankings with task-specific extras (F1, coverage,
similarity stats)
- Speed table from calibration (smallest task, most stable timing)
- VRAM table with resident/peak/delta per model
Usage:
python -m benchmarks.model_eval.summarize \\
--csv results/2026-05-10-host.csv \\
--output reports/2026-05-10-host.md
`ollama ps --format json` is missing on Ollama 0.23.2 (the --format
flag doesn't exist on older versions). Switch to the HTTP API at
/api/ps which returns clean JSON with size_vram per loaded model.
Verified on RTX 3090 + Ollama 0.23.2: vram_resident_mb now returns
7481 MB for qwen3:4b-instruct-2507-q4_K_M with full 32K context KV
cache loaded.
210 hand-quality samples across four tasks. Synthetic only — no
real-person info. Five fictional personae (Aria, Solas, Fenra,
Bramble, Thresh) with distinct domains, room taxonomies, and
relationship contexts.
- room_classification: 100 samples (20 per agent), 14 marked with
realistic noise features, 18 'other' / 6 'general' / 76 specific-
room labels. 5-10 rooms per sample with 'general' always present.
- entity_extraction: 50 samples, 247 entities (114 person, 74 org,
32 project, 27 place). All entity strings verified to appear in
source text. Includes deliberate same-surname-different-people
cases as disambiguation tests.
- memory_extraction: 40 samples, 55 memories across types
(15 fact, 12 decision, 12 commitment, 9 preference, 7 opinion).
- calibration: 20 samples, 4 per class (question, command, statement,
exclamation, greeting). Sanity check that the harness measures
what we think.
All datasets validated: ID alignment across dataset/labels pairs,
closed-set labels are members of their room lists (or 'other'),
calibration labels are members of their classes lists, no leak
terms (igor, milla, domi, lumi, anthropic, openclaw, etc.) anywhere.
Generated 2026-05-10 via subagent + hand-validation. README in the
datasets directory documents the personae, distribution stats, and
labeling conventions.
Skeleton for benchmarking ≤4B-parameter Ollama models on MemPalace
classification and extraction tasks. Outputs per-(model, task, mode)
metrics: accuracy, latency (TTFT, TPS, e2e p50/p95), VRAM (resident
and peak).
Layout under benchmarks/model_eval/:
- candidates.yaml: 16 models across 3 tiers, with family/size/variant
metadata pulled from ollama.com/library/<family>/tags
- metrics.py: timing extraction from Ollama response, VRAMPoller for
peak-memory tracking via nvidia-smi, embedding-similarity scoring,
host-info introspection, percentile aggregation
- runner.py: runs one (model, task, mode) triple, dispatches per-task
prompt building and scoring
- orchestrator.py: iterates candidates × tasks, writes CSV
- tasks/{calibration,room_classification,entity_extraction,
memory_extraction}/{prompts.py,score.py}: per-task prompt builders
and scorers
Uses mempalace.llm_client.get_provider directly so the benchmark runs
the same code path as production. strip_thinking_tokens lives locally
in metrics.py for now; will switch to mempalace.local_model once that
module lands on develop.
Datasets land in a follow-up commit.
End-user installs now lead with `uv tool install mempalace`, with
`pip install mempalace` kept as a fallback. Dev/contributor docs lead
with `uv sync --extra dev` and `uv run` for tests/benchmarks/lint, with
the equivalent pip recipe kept inline. The shipped `/mempalace:init`
skill instructions (mempalace/instructions/init.md) try `uv tool install`
first when uv is on PATH, then fall back through the pip variants.
Adds a .python-version pin at 3.12 because the lockfile's
onnxruntime==1.24.3 only ships wheels for Python >=3.11; without the
pin, `uv sync` on a host where uv prefers 3.10 fails with no source
distribution available, which would make the documented command a
footgun. pyproject's `requires-python = ">=3.9"` is unchanged — pip
users on 3.9/3.10 are unaffected.
Files updated: README.md, CONTRIBUTING.md, CLAUDE.md, the gemini-cli
guide and example, the .claude-plugin / .codex-plugin READMEs, the
mempalace SKILL, the openclaw SKILL, tools/save.md, the three
benchmarks docs, and the corresponding website mirrors.
The miner upserted one drawer per ChromaDB call, paying tokenizer +
ONNX session setup per chunk. The embedding device was CPU-only because
no EmbeddingFunction was ever wired through the backend.
Two changes, each a speedup in its own right; stacked they give ~10x
end-to-end on a medium corpus (20 files, 568 drawers):
1. Batched upsert. `process_file` and `_file_chunks_locked` now collect
all chunks of a file into a single `collection.upsert(...)` so the
embedding model runs one forward pass per file instead of N.
2. Hardware-accelerated embedding function. New `mempalace/embedding.py`
wraps `ONNXMiniLM_L6_V2` with configurable `preferred_providers`.
`MEMPALACE_EMBEDDING_DEVICE` (or `embedding_device` in config.json)
selects auto / cpu / cuda / coreml / dml. Unavailable accelerators
log a warning and fall back to CPU.
The factory subclasses `ONNXMiniLM_L6_V2` and spoofs its `name()` to
`"default"` so the persisted EF identity matches existing palaces
created with ChromaDB's bare `DefaultEmbeddingFunction` -- same
model, same 384-dim vectors, no rebuild needed when turning GPU on.
`ChromaBackend.get_collection` / `create_collection` now pass the
resolved EF on every call so miner writes and searcher reads agree.
Benchmarks (i9-12900KF + RTX 3090, medium scenario, 568 drawers):
per-chunk + CPU 19.77s · 29 drw/s (baseline)
batched + CPU 8.07s · 70 drw/s (2.4x)
batched + CUDA 2.15s · 264 drw/s (9.2x)
Reproducible via `benchmarks/mine_bench.py`.
Install paths:
pip install mempalace[gpu] # NVIDIA CUDA
pip install mempalace[dml] # DirectML (Windows)
pip install mempalace[coreml] # macOS Neural Engine
Mine header now prints `Device: cpu|cuda|...` so users can confirm the
accelerator engaged.
Remaining in-repo surfaces carrying the same retracted or broken
claims as the public pages fixed in the previous two commits.
CONTRIBUTING.md
- "Palace structure matters ... 34% retrieval improvement" → reframed
as scoping (same rewording applied to the website equivalents).
benchmarks/BENCHMARKS.md
- Add a prominent "Important caveat" block at the top of the
"Comparison vs Published Systems" table explaining that R@5
(retrieval recall) and QA accuracy are different metrics, with
citations to Mastra, Mem0, and Supermemory's own published
methodology pages. Annotate the specific competitor rows whose
numbers are QA accuracy, not retrieval recall.
- Annotate the `hybrid v4 + rerank 100%` row to note that the 99.4
→ 100 step was tuned on 3 specific wrong answers (already disclosed
further down in the doc under "Benchmark Integrity"); the honest
hybrid figure is held-out 98.4%.
- Fix the broken clone URL — `aya-thekeeper/mempal` no longer points
at anything; now `MemPalace/mempalace`.
benchmarks/README.md + benchmarks/HYBRID_MODE.md
- Same clone-URL fix applied.
CHANGELOG.md
- Add a ### Documentation entry under [Unreleased] v3.3.0 that names
#875 and summarises the scope of the rewrite.
Addresses #875: every internal BENCHMARKS.md claim reproduced
on Linux x86_64 (v3.3.0 tag, deterministic ChromaDB embeddings,
seed=42 for the LongMemEval dev/held-out split).
Scorecard — all reproduce exactly:
LongMemEval
raw R@5 96.6% (500/500) ✅
hybrid_v4 held-out 450 R@5 98.4% (442/450) ✅
hybrid_v4 + minimax rerank R@5 99.2% (496/500) *
hybrid_v4 + minimax rerank R@10 100.0% (500/500) *
LoCoMo (session, top-10)
raw 60.3% (1986q) ✅
hybrid v5 88.9% (1986q) ✅
ConvoMem all-categories (250 items) 92.9% ✅
MemBench all-categories (8500) 80.3% ✅
* The minimax-m2.7:cloud rerank run replicates the "100%" claim
with a different LLM family (no Anthropic dependency). R@10 is
a perfect reproduction; R@5 misses 4 questions that the
published Haiku run caught — consistent with BENCHMARKS.md's own
disclosure that hybrid_v4 includes three question-specific fixes
developed by inspecting misses, i.e. teaching to the test.
The committed 50/450 split is the deterministic (seed=42) split
BENCHMARKS.md references but wasn't previously in the repo.
Full result JSONLs include every question, every retrieved id,
and every score — auditable end-to-end.
The rerank pipeline was hardcoded to Anthropic's /v1/messages.
Add a backend flag so the same code path can be exercised with
any OpenAI-compatible endpoint — local Ollama, Ollama Cloud,
or any gateway that speaks /v1/chat/completions.
Enables independent verification of the "100% with Haiku rerank"
claim by running the full benchmark with a different LLM family
(e.g. minimax-m2.7:cloud) and zero Anthropic dependency.
Both longmemeval_bench.py and locomo_bench.py:
- llm_rerank*() gain backend= / base_url= kwargs
- CLI: --llm-backend {anthropic,ollama}, --llm-base-url
- API key required only when backend=anthropic (diary/palace modes still require it)
- Parse last integer in response (reasoning models emit multi-int output)
- Fallback to message.reasoning when content is empty
- Raise max_tokens to 1024 for reasoning models
The `_load_api_key()` function in longmemeval_bench.py and locomo_bench.py
searched for API keys in a fixed path (`~/.config/lu/keys.json`) using
personal key names (`anthropic_milla`, `anthropic_claude_code_main`).
This leaks internal infrastructure details into the public codebase and
trains contributors to store credentials in a non-standard location
rather than using the standard ANTHROPIC_API_KEY env var.
Simplified to: CLI flag > env var > empty string. Updated help text
and HYBRID_MODE.md docs to match.
Co-authored-by: Tadao <tadao@travisfixes.com>
The module-level `ssl._create_default_https_context = ssl._create_unverified_context`
disables certificate verification for ALL urllib requests in the process,
not just the benchmark's HuggingFace downloads. This silently exposes
the benchmark runner to MITM attacks.
If a specific environment needs to skip verification (e.g. corporate proxy),
users can set `PYTHONHTTPSVERIFY=0` or pass a custom ssl context per-request
rather than globally patching the ssl module.
Co-authored-by: Tadao <tadao@travisfixes.com>
ATKabli
Igor Lins e Silva
Bruno Leal
Ben Sigman
travisBREAKS
bensig