Commit Graph
11 Commits
Author SHA1 Message Date
Igor Lins e Silva 8591d133da fix(benchmarks): default --num-ctx to 4096 for apples-to-apples comparison
Without an explicit num_ctx, each candidate ran at its Modelfile default
(32k for the Gemma4 variants, larger for qwen3), so VRAM and latency
weren't comparable across families — a 32k-default model pre-allocates
KV cache a 4k-default model doesn't. The flag's own docstring promised
"apples-to-apples" but defaulted to None, defeating the intent.

All current benchmark prompts fit comfortably under 4k tokens
(memory_extraction is the longest at ~500). Users with longer prompts
can still pass --num-ctx <larger>.

Adds a methodology note to the 2026-05-13 multilingual report so its
VRAM/latency numbers aren't conflated with future runs at the new default.
2026-05-14 01:27:53 -03:00
Bruno Leal 0b7799cfa5 fix(benchmarks): address PR #1503 review — file-handle bug, untranslated samples, KO labels
Addresses the review feedback from igorls, gemini-code-assist, and Copilot.

HIGH:
- orchestrator: --output single-file mode now shares ONE (fh, writer) across
  all languages instead of opening N handles to the same path. The old code
  caused interleaved buffer corruption: first language opened "w", subsequent
  ones opened "a", and writes from independent file offsets could overwrite
  each other. Verified with a multi-language --output smoke test (4 rows
  written, all distinct).
- 19 untranslated/empty samples re-translated:
  - dataset.de.jsonl: cal_017
  - dataset.hi.jsonl entity_extraction: ent_020, ent_025, ent_032, ent_038
  - dataset.hi.jsonl room_classification: rc_017, rc_026, rc_028, rc_040,
    rc_064, rc_089, rc_091
  - dataset.ko.jsonl room_classification: rc_027, rc_067
  - dataset.it.jsonl room_classification: rc_029, rc_030, rc_031, rc_032,
    rc_053 (previously empty strings)
- labels.ko.jsonl: restored all proper nouns to English (Doreth, Saela, Ivora,
  Ren Solanke, Pol Krisat, Pell Halloran, Bramble, Hollowmounts Institute,
  Wendelsea, Bridgewater Community Garden, Wends, Drukar, Aerwyn cycle,
  Jaccard, Mason bee, Markdown). Also fixed mistranslation 유전자 사과
  (genetic apple) → 재래종 사과 (heirloom apple).

MEDIUM:
- runner.py: refactored label-resolution one-liner into 3 readable lines
  and added an info log when falling back to English ground truth, so
  readers don't misread "score collapse" as model failure.

LOW:
- orchestrator: moved `import socket` to module top (PEP 8); removed
  unused `out_path` from the unpacking tuple.
- translate_datasets.py: renamed loop variable `l` → `code` (ruff E741);
  made the _translate_one fallback return path explicit instead of relying
  on for-loop fall-through; added a privacy warning in the docstring
  flagging that the default `kimi-k2.6:cloud` sends prose to a remote
  endpoint and should not be used over real palace data.
- 2026-05-13-multilingual.md: converted analytical paragraphs from
  Portuguese to English to match the existing repo convention.
2026-05-14 00:11:50 -03:00
Bruno Leal 9e0df1c29f feat(benchmarks): add DE/FR/HI/IT/KO/RU datasets + --output-dir + translated labels
Adds 6 new language datasets (German, French, Hindi, Italian, Korean, Russian)
across all 4 benchmark tasks (calibration, entity_extraction, memory_extraction,
room_classification) — 630 samples total, same conventions as the existing
pt-BR/es/zh datasets: inputs translated, labels/ground-truth stay English
except where noted.

Changes:
- 24 new dataset.{de,fr,hi,it,ko,ru}.jsonl files across all 4 tasks
- labels.ko.jsonl for memory_extraction: Korean ground-truth so the scorer
  compares Korean model output against Korean expected content instead of
  English (fixes ~20pp score gap identified during testing — see report)
- runner.py: loads labels.{lang}.jsonl when present, falls back to labels.jsonl
- orchestrator.py: adds --output-dir (writes <dir>/<lang>/YYYY-MM-DD-<host>.csv
  per language); --output single-file mode unchanged
- candidates.yaml: adds community tier (igorls classifier variants, heretic)
  and local tier (gemma4:e4b)
- translate_datasets.py: script used to generate the translations via Ollama;
  included so contributors can extend to new languages without manual work
- reports/2026-05-13-multilingual.md: 210-run benchmark report across
  6 models × 7 languages × 5 tasks on RTX 3080 Laptop 8 GB
2026-05-13 20:27:48 -03:00
Igor Lins e Silva e2273fad9c fix(benchmarks): validate --language input + correct --embed-endpoint defaulting
Addresses Copilot + gemini-code-assist review on #1483.

1. Path-traversal guard for --language. The value is interpolated into
   the dataset filename (`dataset.{language}.jsonl`), so unvalidated
   input could escape `task_dir`. Now:
   - regex `^[A-Za-z][A-Za-z0-9]*(?:[_-][A-Za-z0-9]+)?$` accepts en,
     pt-BR, zh-CN, fr_CA, etc. and rejects anything with path separators
     or `..`
   - belt-and-suspenders `Path.resolve().is_relative_to(task_dir)` check
     before opening the file

2. --embed-endpoint now defaults to None and is resolved after parsing:
   uses --endpoint when --llm-provider=ollama (so remote benchmark
   runs score against the same host), else http://localhost:11434.
   Help text now matches behavior. runner.py's CLI was also missing the
   flag entirely — added so single-task runs honor remote endpoints.
2026-05-12 19:19:43 -03:00
Igor Lins e Silva 8e34ffa343 feat(benchmarks): multilingual datasets + parity controls (embed model, num_ctx, language)
Enables shipping decisions for non-English users and fair comparison across
candidates whose Modelfile defaults disagree.

- --language / --languages: load dataset.{lang}.jsonl alongside the base
  dataset.jsonl. CSV gains a language column. Synthesized candidate
  entries let ad-hoc model tags run without editing candidates.yaml.
- --num-ctx: force Ollama options.num_ctx per request, overriding the
  model's Modelfile default. Required for apples-to-apples VRAM/TPS
  (qwen3:4b-q8 defaults to 32k = 9.7 GB resident; at 8k it's 5.6 GB).
- --embed-model: thread the semantic-similarity embedding model through
  scoring. Default flips to embeddinggemma (was nomic-embed-text v1).
  Reason: v1 cosine on EN<->PT-BR same-meaning pairs sits at ~0.607
  (right at the 0.6 match threshold), so any phrasing drift collapses
  to false-negative. embeddinggemma lands ~0.766 with 2.7x the
  signal/noise spread. PT-BR memory_extraction recovered 0.15 -> 0.85
  on the same outputs after the swap.

Datasets: 12 new files (pt-BR/es/zh x 4 tasks, 633 samples). Input text
translated; proper nouns and labels stay English so cross-lingual
scoring against the existing labels.jsonl works without re-translation.
2026-05-12 19:01:02 -03:00
Bruno Leal 3fbe930751 fix(benchmarks): auto-pull embedding model when missing instead of silently scoring 0
Tasks that score via semantic similarity (memory_extraction,
room_classification open-set) depend on nomic-embed-text. If the model
was absent, embed_text() returned None silently and every sample scored
0.0 — indistinguishable from a genuinely bad model.

_ensure_embed_model() now probes the endpoint before the warmup step.
On a 404 it pulls the model automatically via `ollama pull`, verifies
the endpoint responds, and only then proceeds. Fixes:

- OLLAMA_HOST is now forwarded to `ollama pull` so non-localhost
  endpoints are targeted correctly
- FileNotFoundError is caught and re-raised as RuntimeError with an
  install hint if ollama is not on PATH
- Diagnostic prints go to stderr, not stdout
- If the pull or re-probe fails, returns a Result with a clear error
  message instead of silent zeros

Restores --candidates local to its documented meaning (all non-cloud
models). Adds --candidates community for third-party GGUF variants.
2026-05-11 06:29:49 -03:00
Igor Lins e Silva 42f52c8b71 fix(benchmarks): address copilot-pull-request-reviewer feedback on PR #1447
Ten fixes across code, documentation, and committed baselines:

Code fixes:
1. prompts.py: closed-set prompt said "no punctuation" while room
   slugs contain "/". Replaced with explicit "copy the slug verbatim
   including / and -" wording. Empirical impact on accuracy is within
   reproducibility noise (qwen3:4b q4 was 0.610, retest shows 0.594
   on the now-101-sample dataset — same vs new sample noise).
2. orchestrator.py: orchestrator now opens the CSV once, writes the
   header, and flushes after each row. README's "writes incrementally,
   safe to Ctrl-C" claim is now accurate. Long matrix runs no longer
   lose all progress on a crash.
3. metrics.py: gather_host_info() now strips trailing newline from
   `hostname` command output. Stops embedded newlines from leaking
   into committed CSVs.
4. runner.py: passed `endpoint=endpoint` to vram_resident_mb() so
   non-default Ollama endpoints are queried correctly.
5. summarize.py: CSV reader and report writer now use
   encoding="utf-8" with explicit newline handling for cross-platform
   consistency.

Documentation fixes:
6. entity_extraction/score.py: docstring claimed "case-sensitive on
   type" while implementation lowercases both. Corrected.
7. dataset.jsonl rc_101: removed mempalace-private URL since the PR
   lands on the public repo. Will be linkable after OpenClaw merges.
8. datasets/README.md: sample counts updated (room_classification 100
   → 101, total 210 → 211) to match the cherry-picked real-format
   sample.
9. runner.py / orchestrator.py: docstring example paths corrected
   (tests/fixtures/mempalace_bench → benchmarks/model_eval/datasets).

Baseline regeneration:
10. Stripped embedded \n from `host` column in all five committed
    CSVs (2026-05-10 main local + spotcheck, 2026-05-10 + 2026-05-11
    cloud, 2026-05-11 modern). 165 rows cleaned in place via Python
    csv reader/writer. Each row is now line-oriented for diff and
    grep friendliness without changing any measured values.
2026-05-11 03:46:35 -03:00
Igor Lins e Silva fa90f26170 feat(benchmarks): add modern-tier candidates that initial search missed
Five model families that existed at the time of the first benchmark
pass but weren't surfaced by the initial ollama.com search:

- qwen3.5:4b-q4_K_M
- gemma4:e2b-it-q4_K_M
- gemma4:e4b-it-q4_K_M
- granite4.1:3b-q4_K_M
- ministral-3:3b

Grouped as tier="modern" with a new orchestrator filter so they can
be re-run independently. 25 runs in 11.2 minutes, zero errors.

Three findings reshape the recommendations:

1. gemma4:e4b-it-q4_K_M is the new local leader for room classification.
   Closed-set 0.62 (vs qwen3:4b q4 at 0.61), open-set 0.65 (highest
   score across all measured models including cloud). The cloud
   "ceiling" on open-set was 0.61 — a 4B local model now beats every
   cloud reference up to 1T parameters on this task.

2. ministral-3:3b has 0.99 memory coverage — nearly cloud-tier.
   But weak elsewhere (closed-set 0.49, entity 0.63). Specialist
   not generalist.

3. qwen3.5:4b does NOT beat qwen3:4b on this workload despite the
   version bump. Slightly better entity F1, worse on memory and
   closed-set. Newer != better.

Production-tier update:
- Room classification (OpenClaw use case): gemma4:e4b becomes the new
  recommended default. Costs 2.1x latency and 1.4x VRAM vs qwen3:4b
  for the room-task lift.
- General extraction on tight hardware: qwen3:4b q4_K_M still wins.
- Open-set discovery: softens from "shelved indefinitely" to "retest
  with gemma4:e4b + prompt tuning before declaring dead." Gap to
  0.70 ship threshold is now 0.05, not 0.09.
2026-05-11 02:47:29 -03:00
Igor Lins e Silva 67f6fc6974 fix(benchmarks): address gemini-code-assist review on PR #1447
Four fixes from automated code review:

1. score.py: pre-compute embeddings per content string to cut API
   calls from O(P*T) to O(P+T). Was making 2*P*T HTTP round-trips
   to Ollama embeddings per memory_extraction sample; now P+T.
   Significant speed-up on cloud endpoints where each embed is a
   network call.

2. metrics.py: bump VRAMPoller default interval from 100ms to 500ms.
   nvidia-smi subprocess every 100ms was creating measurable jitter
   on the timing numbers. 500ms still captures peak (inference VRAM
   is steady-state during a request) with much less overhead.

3. orchestrator.py: use argparse.BooleanOptionalAction for
   --continue-on-error so --no-continue-on-error actually disables
   the default. The previous combination of store_true and
   default=True made the flag a permanent True.

4. summarize.py: drop the buggy first attempt at the reasoning
   filter (had operator-precedence issue with `or` and `and`).
   Keep only the clean prefix-based filter that was already on
   the next line shadowing it.
2026-05-11 02:19:57 -03:00
Igor Lins e Silva 4ee1839c28 feat(benchmarks): add cloud-tier candidates + cherry-pick real-format fixture
Adds five Ollama Cloud reference models for ceiling measurement:
- gpt-oss:20b-cloud (lighter reference)
- gpt-oss:120b-cloud
- qwen3-coder:480b-cloud (code-tuned variant)
- deepseek-v3.1:671b-cloud
- kimi-k2:1t-cloud (trillion-param MoE)

These are not production candidates. They serve as accuracy ceilings
to size the gap between local-best and what 5-250x larger models can
do on the same tasks.

Orchestrator gains two new tier filters:
- `cloud`: returns candidates with cloud:true (5 models)
- `local`: returns candidates without cloud:true (15 models)

Also cherry-picks a single real-format-flavored sample (rc_101) into
the room_classification dataset, sourced from Lumi's hand-written v3
fixture on feat/openclaw-integration. Multi-turn with parallel tool
calls; tests that the harness handles realistic OpenClaw transcript
shape without choking. Source attribution in the sample's 'source'
field for traceability.

Dataset now: 101 room_classification samples (was 100), other tasks
unchanged.
2026-05-10 20:33:01 -03:00
Igor Lins e Silva c0b23b0d8f feat(benchmarks): add small-model evaluation harness
Skeleton for benchmarking ≤4B-parameter Ollama models on MemPalace
classification and extraction tasks. Outputs per-(model, task, mode)
metrics: accuracy, latency (TTFT, TPS, e2e p50/p95), VRAM (resident
and peak).

Layout under benchmarks/model_eval/:
- candidates.yaml: 16 models across 3 tiers, with family/size/variant
  metadata pulled from ollama.com/library/<family>/tags
- metrics.py: timing extraction from Ollama response, VRAMPoller for
  peak-memory tracking via nvidia-smi, embedding-similarity scoring,
  host-info introspection, percentile aggregation
- runner.py: runs one (model, task, mode) triple, dispatches per-task
  prompt building and scoring
- orchestrator.py: iterates candidates × tasks, writes CSV
- tasks/{calibration,room_classification,entity_extraction,
  memory_extraction}/{prompts.py,score.py}: per-task prompt builders
  and scorers

Uses mempalace.llm_client.get_provider directly so the benchmark runs
the same code path as production. strip_thinking_tokens lives locally
in metrics.py for now; will switch to mempalace.local_model once that
module lands on develop.

Datasets land in a follow-up commit.
2026-05-10 09:44:49 -03:00