Files
mempalace/benchmarks/model_eval
Igor Lins e Silva 5f2c554426 feat(benchmarks): add CSV-to-markdown report renderer
summarize.py reads orchestrator output and renders a readable report:
- Production picks: models meeting min accuracy AND max latency thresholds
- Open-set discovery viability: ship/skip recommendation based on
  mean cosine similarity threshold
- Instruct vs reasoning comparison for the qwen3:4b pair
- Per-task accuracy rankings with task-specific extras (F1, coverage,
  similarity stats)
- Speed table from calibration (smallest task, most stable timing)
- VRAM table with resident/peak/delta per model

Usage:
    python -m benchmarks.model_eval.summarize \\
      --csv results/2026-05-10-host.csv \\
      --output reports/2026-05-10-host.md
2026-05-10 11:03:53 -03:00
..

MemPalace small-model evaluation harness

Evaluates ≤4B-parameter Ollama models on MemPalace's classification and extraction tasks.

Outputs a CSV with accuracy, latency (TTFT, TPS, e2e p50/p95), and VRAM (resident, peak) per (model, task, mode) triple. Use the results to populate mempalace/local_model.py::MODEL_TIERS from data instead of vibes.

Quick start

# 1. Pull the candidate models you want to evaluate
ollama pull qwen3:4b-instruct-2507-q4_K_M
ollama pull gemma3:4b-it-q4_K_M
# ... etc, see candidates.yaml

# 2. Run one (model, task) pair to verify the harness works
python -m benchmarks.model_eval.runner \
  --model qwen3:4b-instruct-2507-q4_K_M \
  --task room_classification \
  --mode closed

# 3. Run the whole matrix
python -m benchmarks.model_eval.orchestrator \
  --candidates tier1 \
  --tasks all \
  --output results/$(date -u +%Y-%m-%d)-$(hostname).csv

What it measures

For each (model, task, mode):

  • Accuracy — task-specific scoring against the labeled dataset
  • TTFT (time-to-first-token) — p50 and p95 over N=5 warm runs
  • TPS (tokens/second) — sustained throughput on a 10-request batch
  • e2e latency — full single-classification time, p50 and p95 over N=20 warm runs
  • VRAM resident — model memory after warmup (read from ollama ps)
  • VRAM peak — peak GPU memory during inference (polled via nvidia-smi)

The first run of each model is discarded (cache + GPU clock ramp).

Tasks

  • room_classification — closed-set (room list provided) and open-set (model invents the slug)
  • entity_extraction — JSON list of entities per sample
  • memory_extraction — structured memory items per sample
  • calibration — simple 5-class sentence-type, sanity check the harness

See tasks/<name>/README.md for input format and scoring details.

Datasets

All datasets are synthetic and committed to this repo. No real-person info. Generated once and frozen so benchmark numbers stay comparable across runs.

If you need to extend the dataset, add samples; don't replace existing ones, otherwise prior numbers stop being comparable.

Hardware reporting

Every result file includes a header row with:

  • CPU, core count, RAM
  • GPU model, VRAM total
  • Ollama version
  • OS/kernel
  • Run timestamp (UTC)

Speed numbers are not portable across machines. Use them for relative ranking on a single setup. Accuracy numbers cross-port cleanly.

Reusing existing infrastructure

The harness uses mempalace.llm_client.get_provider("ollama", model=tag) and provider.classify(...) directly — same code path as production. For thinking-token models, post-processes via mempalace.local_model.strip_thinking_tokens.

No new HTTP plumbing. No reimplementation of provider abstraction. The harness benchmarks the same code that ships.