Ten fixes across code, documentation, and committed baselines:
Code fixes:
1. prompts.py: closed-set prompt said "no punctuation" while room
slugs contain "/". Replaced with explicit "copy the slug verbatim
including / and -" wording. Empirical impact on accuracy is within
reproducibility noise (qwen3:4b q4 was 0.610, retest shows 0.594
on the now-101-sample dataset — same vs new sample noise).
2. orchestrator.py: orchestrator now opens the CSV once, writes the
header, and flushes after each row. README's "writes incrementally,
safe to Ctrl-C" claim is now accurate. Long matrix runs no longer
lose all progress on a crash.
3. metrics.py: gather_host_info() now strips trailing newline from
`hostname` command output. Stops embedded newlines from leaking
into committed CSVs.
4. runner.py: passed `endpoint=endpoint` to vram_resident_mb() so
non-default Ollama endpoints are queried correctly.
5. summarize.py: CSV reader and report writer now use
encoding="utf-8" with explicit newline handling for cross-platform
consistency.
Documentation fixes:
6. entity_extraction/score.py: docstring claimed "case-sensitive on
type" while implementation lowercases both. Corrected.
7. dataset.jsonl rc_101: removed mempalace-private URL since the PR
lands on the public repo. Will be linkable after OpenClaw merges.
8. datasets/README.md: sample counts updated (room_classification 100
→ 101, total 210 → 211) to match the cherry-picked real-format
sample.
9. runner.py / orchestrator.py: docstring example paths corrected
(tests/fixtures/mempalace_bench → benchmarks/model_eval/datasets).
Baseline regeneration:
10. Stripped embedded \n from `host` column in all five committed
CSVs (2026-05-10 main local + spotcheck, 2026-05-10 + 2026-05-11
cloud, 2026-05-11 modern). 165 rows cleaned in place via Python
csv reader/writer. Each row is now line-oriented for diff and
grep friendliness without changing any measured values.
Four fixes from automated code review:
1. score.py: pre-compute embeddings per content string to cut API
calls from O(P*T) to O(P+T). Was making 2*P*T HTTP round-trips
to Ollama embeddings per memory_extraction sample; now P+T.
Significant speed-up on cloud endpoints where each embed is a
network call.
2. metrics.py: bump VRAMPoller default interval from 100ms to 500ms.
nvidia-smi subprocess every 100ms was creating measurable jitter
on the timing numbers. 500ms still captures peak (inference VRAM
is steady-state during a request) with much less overhead.
3. orchestrator.py: use argparse.BooleanOptionalAction for
--continue-on-error so --no-continue-on-error actually disables
the default. The previous combination of store_true and
default=True made the flag a permanent True.
4. summarize.py: drop the buggy first attempt at the reasoning
filter (had operator-precedence issue with `or` and `and`).
Keep only the clean prefix-based filter that was already on
the next line shadowing it.
summarize.py reads orchestrator output and renders a readable report:
- Production picks: models meeting min accuracy AND max latency thresholds
- Open-set discovery viability: ship/skip recommendation based on
mean cosine similarity threshold
- Instruct vs reasoning comparison for the qwen3:4b pair
- Per-task accuracy rankings with task-specific extras (F1, coverage,
similarity stats)
- Speed table from calibration (smallest task, most stable timing)
- VRAM table with resident/peak/delta per model
Usage:
python -m benchmarks.model_eval.summarize \\
--csv results/2026-05-10-host.csv \\
--output reports/2026-05-10-host.md
Igor Lins e Silva