Commit Graph
1 Commits
Author SHA1 Message Date
Bruno Leal 9e0df1c29f feat(benchmarks): add DE/FR/HI/IT/KO/RU datasets + --output-dir + translated labels
Adds 6 new language datasets (German, French, Hindi, Italian, Korean, Russian)
across all 4 benchmark tasks (calibration, entity_extraction, memory_extraction,
room_classification) — 630 samples total, same conventions as the existing
pt-BR/es/zh datasets: inputs translated, labels/ground-truth stay English
except where noted.

Changes:
- 24 new dataset.{de,fr,hi,it,ko,ru}.jsonl files across all 4 tasks
- labels.ko.jsonl for memory_extraction: Korean ground-truth so the scorer
  compares Korean model output against Korean expected content instead of
  English (fixes ~20pp score gap identified during testing — see report)
- runner.py: loads labels.{lang}.jsonl when present, falls back to labels.jsonl
- orchestrator.py: adds --output-dir (writes <dir>/<lang>/YYYY-MM-DD-<host>.csv
  per language); --output single-file mode unchanged
- candidates.yaml: adds community tier (igorls classifier variants, heretic)
  and local tier (gemma4:e4b)
- translate_datasets.py: script used to generate the translations via Ollama;
  included so contributors can extend to new languages without manual work
- reports/2026-05-13-multilingual.md: 210-run benchmark report across
  6 models × 7 languages × 5 tasks on RTX 3080 Laptop 8 GB
2026-05-13 20:27:48 -03:00