Commit Graph
2 Commits
Author SHA1 Message Date
Igor Lins e Silva 42f52c8b71 fix(benchmarks): address copilot-pull-request-reviewer feedback on PR #1447
Ten fixes across code, documentation, and committed baselines:

Code fixes:
1. prompts.py: closed-set prompt said "no punctuation" while room
   slugs contain "/". Replaced with explicit "copy the slug verbatim
   including / and -" wording. Empirical impact on accuracy is within
   reproducibility noise (qwen3:4b q4 was 0.610, retest shows 0.594
   on the now-101-sample dataset — same vs new sample noise).
2. orchestrator.py: orchestrator now opens the CSV once, writes the
   header, and flushes after each row. README's "writes incrementally,
   safe to Ctrl-C" claim is now accurate. Long matrix runs no longer
   lose all progress on a crash.
3. metrics.py: gather_host_info() now strips trailing newline from
   `hostname` command output. Stops embedded newlines from leaking
   into committed CSVs.
4. runner.py: passed `endpoint=endpoint` to vram_resident_mb() so
   non-default Ollama endpoints are queried correctly.
5. summarize.py: CSV reader and report writer now use
   encoding="utf-8" with explicit newline handling for cross-platform
   consistency.

Documentation fixes:
6. entity_extraction/score.py: docstring claimed "case-sensitive on
   type" while implementation lowercases both. Corrected.
7. dataset.jsonl rc_101: removed mempalace-private URL since the PR
   lands on the public repo. Will be linkable after OpenClaw merges.
8. datasets/README.md: sample counts updated (room_classification 100
   → 101, total 210 → 211) to match the cherry-picked real-format
   sample.
9. runner.py / orchestrator.py: docstring example paths corrected
   (tests/fixtures/mempalace_bench → benchmarks/model_eval/datasets).

Baseline regeneration:
10. Stripped embedded \n from `host` column in all five committed
    CSVs (2026-05-10 main local + spotcheck, 2026-05-10 + 2026-05-11
    cloud, 2026-05-11 modern). 165 rows cleaned in place via Python
    csv reader/writer. Each row is now line-oriented for diff and
    grep friendliness without changing any measured values.
2026-05-11 03:46:35 -03:00
Igor Lins e Silva 1c1e725b4c feat(benchmarks): add synthetic dataset for model evaluation harness
210 hand-quality samples across four tasks. Synthetic only — no
real-person info. Five fictional personae (Aria, Solas, Fenra,
Bramble, Thresh) with distinct domains, room taxonomies, and
relationship contexts.

- room_classification: 100 samples (20 per agent), 14 marked with
  realistic noise features, 18 'other' / 6 'general' / 76 specific-
  room labels. 5-10 rooms per sample with 'general' always present.
- entity_extraction: 50 samples, 247 entities (114 person, 74 org,
  32 project, 27 place). All entity strings verified to appear in
  source text. Includes deliberate same-surname-different-people
  cases as disambiguation tests.
- memory_extraction: 40 samples, 55 memories across types
  (15 fact, 12 decision, 12 commitment, 9 preference, 7 opinion).
- calibration: 20 samples, 4 per class (question, command, statement,
  exclamation, greeting). Sanity check that the harness measures
  what we think.

All datasets validated: ID alignment across dataset/labels pairs,
closed-set labels are members of their room lists (or 'other'),
calibration labels are members of their classes lists, no leak
terms (igor, milla, domi, lumi, anthropic, openclaw, etc.) anywhere.

Generated 2026-05-10 via subagent + hand-validation. README in the
datasets directory documents the personae, distribution stats, and
labeling conventions.
2026-05-10 09:45:03 -03:00