mempalace/benchmarks/model_eval/tasks/memory_extraction
Igor Lins e Silva 67f6fc6974 fix(benchmarks): address gemini-code-assist review on PR #1447
Four fixes from automated code review:

1. score.py: pre-compute embeddings per content string to cut API
   calls from O(P*T) to O(P+T). Was making 2*P*T HTTP round-trips
   to Ollama embeddings per memory_extraction sample; now P+T.
   Significant speed-up on cloud endpoints where each embed is a
   network call.

2. metrics.py: bump VRAMPoller default interval from 100ms to 500ms.
   nvidia-smi subprocess every 100ms was creating measurable jitter
   on the timing numbers. 500ms still captures peak (inference VRAM
   is steady-state during a request) with much less overhead.

3. orchestrator.py: use argparse.BooleanOptionalAction for
   --continue-on-error so --no-continue-on-error actually disables
   the default. The previous combination of store_true and
   default=True made the flag a permanent True.

4. summarize.py: drop the buggy first attempt at the reasoning
   filter (had operator-precedence issue with `or` and `and`).
   Keep only the clean prefix-based filter that was already on
   the next line shadowing it.
2026-05-11 02:19:57 -03:00
..
__init__.py feat(benchmarks): add small-model evaluation harness 2026-05-10 09:44:49 -03:00
prompts.py feat(benchmarks): add small-model evaluation harness 2026-05-10 09:44:49 -03:00
score.py fix(benchmarks): address gemini-code-assist review on PR #1447 2026-05-11 02:19:57 -03:00