Commit Graph
1 Commits
Author SHA1 Message Date
Igor Lins e Silva 8e34ffa343 feat(benchmarks): multilingual datasets + parity controls (embed model, num_ctx, language)
Enables shipping decisions for non-English users and fair comparison across
candidates whose Modelfile defaults disagree.

- --language / --languages: load dataset.{lang}.jsonl alongside the base
  dataset.jsonl. CSV gains a language column. Synthesized candidate
  entries let ad-hoc model tags run without editing candidates.yaml.
- --num-ctx: force Ollama options.num_ctx per request, overriding the
  model's Modelfile default. Required for apples-to-apples VRAM/TPS
  (qwen3:4b-q8 defaults to 32k = 9.7 GB resident; at 8k it's 5.6 GB).
- --embed-model: thread the semantic-similarity embedding model through
  scoring. Default flips to embeddinggemma (was nomic-embed-text v1).
  Reason: v1 cosine on EN<->PT-BR same-meaning pairs sits at ~0.607
  (right at the 0.6 match threshold), so any phrasing drift collapses
  to false-negative. embeddinggemma lands ~0.766 with 2.7x the
  signal/noise spread. PT-BR memory_extraction recovered 0.15 -> 0.85
  on the same outputs after the swap.

Datasets: 12 new files (pt-BR/es/zh x 4 tasks, 633 samples). Input text
translated; proper nouns and labels stay English so cross-lingual
scoring against the existing labels.jsonl works without re-translation.
2026-05-12 19:01:02 -03:00