Commit Graph
3 Commits
Author SHA1 Message Date
Igor Lins e Silva 42f52c8b71 fix(benchmarks): address copilot-pull-request-reviewer feedback on PR #1447
Ten fixes across code, documentation, and committed baselines:

Code fixes:
1. prompts.py: closed-set prompt said "no punctuation" while room
   slugs contain "/". Replaced with explicit "copy the slug verbatim
   including / and -" wording. Empirical impact on accuracy is within
   reproducibility noise (qwen3:4b q4 was 0.610, retest shows 0.594
   on the now-101-sample dataset — same vs new sample noise).
2. orchestrator.py: orchestrator now opens the CSV once, writes the
   header, and flushes after each row. README's "writes incrementally,
   safe to Ctrl-C" claim is now accurate. Long matrix runs no longer
   lose all progress on a crash.
3. metrics.py: gather_host_info() now strips trailing newline from
   `hostname` command output. Stops embedded newlines from leaking
   into committed CSVs.
4. runner.py: passed `endpoint=endpoint` to vram_resident_mb() so
   non-default Ollama endpoints are queried correctly.
5. summarize.py: CSV reader and report writer now use
   encoding="utf-8" with explicit newline handling for cross-platform
   consistency.

Documentation fixes:
6. entity_extraction/score.py: docstring claimed "case-sensitive on
   type" while implementation lowercases both. Corrected.
7. dataset.jsonl rc_101: removed mempalace-private URL since the PR
   lands on the public repo. Will be linkable after OpenClaw merges.
8. datasets/README.md: sample counts updated (room_classification 100
   → 101, total 210 → 211) to match the cherry-picked real-format
   sample.
9. runner.py / orchestrator.py: docstring example paths corrected
   (tests/fixtures/mempalace_bench → benchmarks/model_eval/datasets).

Baseline regeneration:
10. Stripped embedded \n from `host` column in all five committed
    CSVs (2026-05-10 main local + spotcheck, 2026-05-10 + 2026-05-11
    cloud, 2026-05-11 modern). 165 rows cleaned in place via Python
    csv reader/writer. Each row is now line-oriented for diff and
    grep friendliness without changing any measured values.
2026-05-11 03:46:35 -03:00
Igor Lins e Silva 4ee1839c28 feat(benchmarks): add cloud-tier candidates + cherry-pick real-format fixture
Adds five Ollama Cloud reference models for ceiling measurement:
- gpt-oss:20b-cloud (lighter reference)
- gpt-oss:120b-cloud
- qwen3-coder:480b-cloud (code-tuned variant)
- deepseek-v3.1:671b-cloud
- kimi-k2:1t-cloud (trillion-param MoE)

These are not production candidates. They serve as accuracy ceilings
to size the gap between local-best and what 5-250x larger models can
do on the same tasks.

Orchestrator gains two new tier filters:
- `cloud`: returns candidates with cloud:true (5 models)
- `local`: returns candidates without cloud:true (15 models)

Also cherry-picks a single real-format-flavored sample (rc_101) into
the room_classification dataset, sourced from Lumi's hand-written v3
fixture on feat/openclaw-integration. Multi-turn with parallel tool
calls; tests that the harness handles realistic OpenClaw transcript
shape without choking. Source attribution in the sample's 'source'
field for traceability.

Dataset now: 101 room_classification samples (was 100), other tasks
unchanged.
2026-05-10 20:33:01 -03:00
Igor Lins e Silva 1c1e725b4c feat(benchmarks): add synthetic dataset for model evaluation harness
210 hand-quality samples across four tasks. Synthetic only — no
real-person info. Five fictional personae (Aria, Solas, Fenra,
Bramble, Thresh) with distinct domains, room taxonomies, and
relationship contexts.

- room_classification: 100 samples (20 per agent), 14 marked with
  realistic noise features, 18 'other' / 6 'general' / 76 specific-
  room labels. 5-10 rooms per sample with 'general' always present.
- entity_extraction: 50 samples, 247 entities (114 person, 74 org,
  32 project, 27 place). All entity strings verified to appear in
  source text. Includes deliberate same-surname-different-people
  cases as disambiguation tests.
- memory_extraction: 40 samples, 55 memories across types
  (15 fact, 12 decision, 12 commitment, 9 preference, 7 opinion).
- calibration: 20 samples, 4 per class (question, command, statement,
  exclamation, greeting). Sanity check that the harness measures
  what we think.

All datasets validated: ID alignment across dataset/labels pairs,
closed-set labels are members of their room lists (or 'other'),
calibration labels are members of their classes lists, no leak
terms (igor, milla, domi, lumi, anthropic, openclaw, etc.) anywhere.

Generated 2026-05-10 via subagent + hand-validation. README in the
datasets directory documents the personae, distribution stats, and
labeling conventions.
2026-05-10 09:45:03 -03:00