mempalace/benchmarks/model_eval/datasets/memory_extraction
Bruno Leal 0b7799cfa5 fix(benchmarks): address PR #1503 review — file-handle bug, untranslated samples, KO labels
Addresses the review feedback from igorls, gemini-code-assist, and Copilot.

HIGH:
- orchestrator: --output single-file mode now shares ONE (fh, writer) across
  all languages instead of opening N handles to the same path. The old code
  caused interleaved buffer corruption: first language opened "w", subsequent
  ones opened "a", and writes from independent file offsets could overwrite
  each other. Verified with a multi-language --output smoke test (4 rows
  written, all distinct).
- 19 untranslated/empty samples re-translated:
  - dataset.de.jsonl: cal_017
  - dataset.hi.jsonl entity_extraction: ent_020, ent_025, ent_032, ent_038
  - dataset.hi.jsonl room_classification: rc_017, rc_026, rc_028, rc_040,
    rc_064, rc_089, rc_091
  - dataset.ko.jsonl room_classification: rc_027, rc_067
  - dataset.it.jsonl room_classification: rc_029, rc_030, rc_031, rc_032,
    rc_053 (previously empty strings)
- labels.ko.jsonl: restored all proper nouns to English (Doreth, Saela, Ivora,
  Ren Solanke, Pol Krisat, Pell Halloran, Bramble, Hollowmounts Institute,
  Wendelsea, Bridgewater Community Garden, Wends, Drukar, Aerwyn cycle,
  Jaccard, Mason bee, Markdown). Also fixed mistranslation 유전자 사과
  (genetic apple) → 재래종 사과 (heirloom apple).

MEDIUM:
- runner.py: refactored label-resolution one-liner into 3 readable lines
  and added an info log when falling back to English ground truth, so
  readers don't misread "score collapse" as model failure.

LOW:
- orchestrator: moved `import socket` to module top (PEP 8); removed
  unused `out_path` from the unpacking tuple.
- translate_datasets.py: renamed loop variable `l` → `code` (ruff E741);
  made the _translate_one fallback return path explicit instead of relying
  on for-loop fall-through; added a privacy warning in the docstring
  flagging that the default `kimi-k2.6:cloud` sends prose to a remote
  endpoint and should not be used over real palace data.
- 2026-05-13-multilingual.md: converted analytical paragraphs from
  Portuguese to English to match the existing repo convention.
2026-05-14 00:11:50 -03:00
..
dataset.de.jsonl feat(benchmarks): add DE/FR/HI/IT/KO/RU datasets + --output-dir + translated labels 2026-05-13 20:27:48 -03:00
dataset.es.jsonl feat(benchmarks): multilingual datasets + parity controls (embed model, num_ctx, language) 2026-05-12 19:01:02 -03:00
dataset.fr.jsonl feat(benchmarks): add DE/FR/HI/IT/KO/RU datasets + --output-dir + translated labels 2026-05-13 20:27:48 -03:00
dataset.hi.jsonl feat(benchmarks): add DE/FR/HI/IT/KO/RU datasets + --output-dir + translated labels 2026-05-13 20:27:48 -03:00
dataset.it.jsonl feat(benchmarks): add DE/FR/HI/IT/KO/RU datasets + --output-dir + translated labels 2026-05-13 20:27:48 -03:00
dataset.jsonl feat(benchmarks): add synthetic dataset for model evaluation harness 2026-05-10 09:45:03 -03:00
dataset.ko.jsonl feat(benchmarks): add DE/FR/HI/IT/KO/RU datasets + --output-dir + translated labels 2026-05-13 20:27:48 -03:00
dataset.pt-BR.jsonl feat(benchmarks): multilingual datasets + parity controls (embed model, num_ctx, language) 2026-05-12 19:01:02 -03:00
dataset.ru.jsonl feat(benchmarks): add DE/FR/HI/IT/KO/RU datasets + --output-dir + translated labels 2026-05-13 20:27:48 -03:00
dataset.zh.jsonl feat(benchmarks): multilingual datasets + parity controls (embed model, num_ctx, language) 2026-05-12 19:01:02 -03:00
labels.jsonl feat(benchmarks): add synthetic dataset for model evaluation harness 2026-05-10 09:45:03 -03:00
labels.ko.jsonl fix(benchmarks): address PR #1503 review — file-handle bug, untranslated samples, KO labels 2026-05-14 00:11:50 -03:00