Commit Graph
2 Commits
Author SHA1 Message Date
Igor Lins e Silva 4ee1839c28 feat(benchmarks): add cloud-tier candidates + cherry-pick real-format fixture
Adds five Ollama Cloud reference models for ceiling measurement:
- gpt-oss:20b-cloud (lighter reference)
- gpt-oss:120b-cloud
- qwen3-coder:480b-cloud (code-tuned variant)
- deepseek-v3.1:671b-cloud
- kimi-k2:1t-cloud (trillion-param MoE)

These are not production candidates. They serve as accuracy ceilings
to size the gap between local-best and what 5-250x larger models can
do on the same tasks.

Orchestrator gains two new tier filters:
- `cloud`: returns candidates with cloud:true (5 models)
- `local`: returns candidates without cloud:true (15 models)

Also cherry-picks a single real-format-flavored sample (rc_101) into
the room_classification dataset, sourced from Lumi's hand-written v3
fixture on feat/openclaw-integration. Multi-turn with parallel tool
calls; tests that the harness handles realistic OpenClaw transcript
shape without choking. Source attribution in the sample's 'source'
field for traceability.

Dataset now: 101 room_classification samples (was 100), other tasks
unchanged.
2026-05-10 20:33:01 -03:00
Igor Lins e Silva 1c1e725b4c feat(benchmarks): add synthetic dataset for model evaluation harness
210 hand-quality samples across four tasks. Synthetic only — no
real-person info. Five fictional personae (Aria, Solas, Fenra,
Bramble, Thresh) with distinct domains, room taxonomies, and
relationship contexts.

- room_classification: 100 samples (20 per agent), 14 marked with
  realistic noise features, 18 'other' / 6 'general' / 76 specific-
  room labels. 5-10 rooms per sample with 'general' always present.
- entity_extraction: 50 samples, 247 entities (114 person, 74 org,
  32 project, 27 place). All entity strings verified to appear in
  source text. Includes deliberate same-surname-different-people
  cases as disambiguation tests.
- memory_extraction: 40 samples, 55 memories across types
  (15 fact, 12 decision, 12 commitment, 9 preference, 7 opinion).
- calibration: 20 samples, 4 per class (question, command, statement,
  exclamation, greeting). Sanity check that the harness measures
  what we think.

All datasets validated: ID alignment across dataset/labels pairs,
closed-set labels are members of their room lists (or 'other'),
calibration labels are members of their classes lists, no leak
terms (igor, milla, domi, lumi, anthropic, openclaw, etc.) anywhere.

Generated 2026-05-10 via subagent + hand-validation. README in the
datasets directory documents the personae, distribution stats, and
labeling conventions.
2026-05-10 09:45:03 -03:00