8591d133da
Without an explicit num_ctx, each candidate ran at its Modelfile default (32k for the Gemma4 variants, larger for qwen3), so VRAM and latency weren't comparable across families — a 32k-default model pre-allocates KV cache a 4k-default model doesn't. The flag's own docstring promised "apples-to-apples" but defaulted to None, defeating the intent. All current benchmark prompts fit comfortably under 4k tokens (memory_extraction is the longest at ~500). Users with longer prompts can still pass --num-ctx <larger>. Adds a methodology note to the 2026-05-13 multilingual report so its VRAM/latency numbers aren't conflated with future runs at the new default. |
||
|---|---|---|
| .. | ||
| 2026-05-10-analysis.md | ||
| 2026-05-10-z690-ex-glacial.md | ||
| 2026-05-13-multilingual.md | ||
| README.md | ||
README.md
Reports directory
Hand-written markdown analyses go here. After each benchmark run, render a summary table from the CSV and write a narrative interpretation: which models won, where the surprises were, which entries belong in the next MODEL_TIERS revision.
Reports are committed (unlike the raw CSVs in results/).