Files
mempalace/benchmarks/model_eval/reports
Igor Lins e Silva fa90f26170 feat(benchmarks): add modern-tier candidates that initial search missed
Five model families that existed at the time of the first benchmark
pass but weren't surfaced by the initial ollama.com search:

- qwen3.5:4b-q4_K_M
- gemma4:e2b-it-q4_K_M
- gemma4:e4b-it-q4_K_M
- granite4.1:3b-q4_K_M
- ministral-3:3b

Grouped as tier="modern" with a new orchestrator filter so they can
be re-run independently. 25 runs in 11.2 minutes, zero errors.

Three findings reshape the recommendations:

1. gemma4:e4b-it-q4_K_M is the new local leader for room classification.
   Closed-set 0.62 (vs qwen3:4b q4 at 0.61), open-set 0.65 (highest
   score across all measured models including cloud). The cloud
   "ceiling" on open-set was 0.61 — a 4B local model now beats every
   cloud reference up to 1T parameters on this task.

2. ministral-3:3b has 0.99 memory coverage — nearly cloud-tier.
   But weak elsewhere (closed-set 0.49, entity 0.63). Specialist
   not generalist.

3. qwen3.5:4b does NOT beat qwen3:4b on this workload despite the
   version bump. Slightly better entity F1, worse on memory and
   closed-set. Newer != better.

Production-tier update:
- Room classification (OpenClaw use case): gemma4:e4b becomes the new
  recommended default. Costs 2.1x latency and 1.4x VRAM vs qwen3:4b
  for the room-task lift.
- General extraction on tight hardware: qwen3:4b q4_K_M still wins.
- Open-set discovery: softens from "shelved indefinitely" to "retest
  with gemma4:e4b + prompt tuning before declaring dead." Gap to
  0.70 ship threshold is now 0.05, not 0.09.
2026-05-11 02:47:29 -03:00
..

Reports directory

Hand-written markdown analyses go here. After each benchmark run, render a summary table from the CSV and write a narrative interpretation: which models won, where the surprises were, which entries belong in the next MODEL_TIERS revision.

Reports are committed (unlike the raw CSVs in results/).