Files
mempalace/benchmarks/model_eval/datasets/memory_extraction/labels.jsonl
T
Igor Lins e Silva 1c1e725b4c feat(benchmarks): add synthetic dataset for model evaluation harness
210 hand-quality samples across four tasks. Synthetic only — no
real-person info. Five fictional personae (Aria, Solas, Fenra,
Bramble, Thresh) with distinct domains, room taxonomies, and
relationship contexts.

- room_classification: 100 samples (20 per agent), 14 marked with
  realistic noise features, 18 'other' / 6 'general' / 76 specific-
  room labels. 5-10 rooms per sample with 'general' always present.
- entity_extraction: 50 samples, 247 entities (114 person, 74 org,
  32 project, 27 place). All entity strings verified to appear in
  source text. Includes deliberate same-surname-different-people
  cases as disambiguation tests.
- memory_extraction: 40 samples, 55 memories across types
  (15 fact, 12 decision, 12 commitment, 9 preference, 7 opinion).
- calibration: 20 samples, 4 per class (question, command, statement,
  exclamation, greeting). Sanity check that the harness measures
  what we think.

All datasets validated: ID alignment across dataset/labels pairs,
closed-set labels are members of their room lists (or 'other'),
calibration labels are members of their classes lists, no leak
terms (igor, milla, domi, lumi, anthropic, openclaw, etc.) anywhere.

Generated 2026-05-10 via subagent + hand-validation. README in the
datasets directory documents the personae, distribution stats, and
labeling conventions.
2026-05-10 09:45:03 -03:00

41 lines
8.8 KiB
JSON

{"id": "mem_001", "memories": [{"type": "decision", "content": "Switching the topic-clustering pipeline from cosine similarity to Jaccard similarity."}]}
{"id": "mem_002", "memories": [{"type": "preference", "content": "Saela prefers writing drafts in LaTeX over Markdown."}, {"type": "commitment", "content": "Will stop converting Saela's files to Markdown going forward."}]}
{"id": "mem_003", "memories": [{"type": "fact", "content": "The new server room is at 19°C ambient."}, {"type": "fact", "content": "The chiller spec sheet says optimal temperature is 21-23°C."}]}
{"id": "mem_004", "memories": [{"type": "opinion", "content": "The field has overrated chain-of-thought prompting; it works on math benchmarks because the reasoning traces happen to match the underlying computation, but on harder reasoning the traces are post-hoc rationalizations."}]}
{"id": "mem_005", "memories": [{"type": "commitment", "content": "Will deliver the first draft of the survey paper to the user by Friday morning."}]}
{"id": "mem_006", "memories": [{"type": "decision", "content": "No more synthetic NPK fertilizer in the kitchen garden; switching to compost amendments and a winter cover crop rotation, starting this fall."}]}
{"id": "mem_007", "memories": [{"type": "preference", "content": "Ivora prefers receiving weekly cashflow reports on Monday mornings rather than Friday afternoons."}, {"type": "commitment", "content": "Will update the auto-send schedule for cashflow reports to Monday mornings."}]}
{"id": "mem_008", "memories": [{"type": "fact", "content": "The Tartine Lab journal's submission deadline is March 15 and they don't accept late submissions."}]}
{"id": "mem_009", "memories": [{"type": "commitment", "content": "Will fix the OCR pipeline's accuracy on hand-scanned invoices from 85% to 97% by month-end, or revert to manual review for that document type."}]}
{"id": "mem_010", "memories": [{"type": "opinion", "content": "Row polymorphism is the right abstraction but the syntax noise it introduces in error messages will require significant work on diagnostics before shipping to external users."}]}
{"id": "mem_011", "memories": [{"type": "decision", "content": "Using straight-line depreciation for the office build-out and MACRS for the equipment, with Section 179 election only on items they're sure they'll use long-term."}]}
{"id": "mem_012", "memories": [{"type": "preference", "content": "Karis prefers weekly updates as bullet points rather than long paragraphs because she reads them on her phone between garden visits."}]}
{"id": "mem_013", "memories": [{"type": "fact", "content": "This year's first-bloom dates are eight days earlier than the long-term mean at the Wendelsea Botanic Garden."}, {"type": "fact", "content": "The Wendelsea Botanic Garden's first-bloom records go back 18 years and this is the third-earliest spring on record."}]}
{"id": "mem_014", "memories": [{"type": "opinion", "content": "Mapping linguistic stereotypes (Wends speak softer than Drukar) onto fictional peoples will feel reductive on the page and can flatten characters."}]}
{"id": "mem_015", "memories": [{"type": "commitment", "content": "Will email Doreth the pruning schedule for the heritage apples by end of week."}]}
{"id": "mem_016", "memories": [{"type": "decision", "content": "Switching from Mann-Whitney U to permutation tests for latency comparisons; re-running all previous comparisons under the new method."}]}
{"id": "mem_017", "memories": [{"type": "preference", "content": "Pol Krisat prefers code reviews via threaded comments on the PR rather than separate emails or chat messages."}, {"type": "decision", "content": "Adopting the same convention (threaded PR comments) myself."}]}
{"id": "mem_018", "memories": [{"type": "fact", "content": "Aerwyn Capital's diligence period is 30 days from term sheet signing, with an April 14 deadline."}]}
{"id": "mem_019", "memories": [{"type": "opinion", "content": "The Hollowmounts Institute's vision-language model paper has selection bias because they only report on prompts where their model wins; the technical work is solid but the paper oversells."}]}
{"id": "mem_020", "memories": [{"type": "commitment", "content": "Will send the revised forecast to Ivora by Thursday end of day."}]}
{"id": "mem_021", "memories": [{"type": "decision", "content": "Saela decided to scope the survey to contrastive objectives only — no clustering-based or distillation-based approaches."}, {"type": "commitment", "content": "Will restructure the survey outline to match the narrower scope."}]}
{"id": "mem_022", "memories": [{"type": "preference", "content": "Bramble prefers native-only plantings for restoration projects even when establishment is slower than non-native alternatives."}, {"type": "decision", "content": "Building the native-only preference into all future restoration recommendations."}]}
{"id": "mem_023", "memories": [{"type": "fact", "content": "Crestmoor Systems uses OTLP HTTP for all internal services, never gRPC, due to version-skew issues across their polyglot stack."}]}
{"id": "mem_024", "memories": [{"type": "preference", "content": "Solas prefers Pratt-style operator precedence parsers over generated parsers for languages with complex precedence rules."}]}
{"id": "mem_025", "memories": [{"type": "commitment", "content": "Will write one chapter this week without AI assistance and share the comparison with the Wendelsea Writing Group."}]}
{"id": "mem_026", "memories": [{"type": "decision", "content": "Chose B-tree over skip list for the new index because of the read-mostly workload and better cache behavior."}]}
{"id": "mem_027", "memories": [{"type": "fact", "content": "Ren Solanke's dissertation defense is May 6."}, {"type": "commitment", "content": "Will provide one round of feedback on Ren Solanke's draft chapter by April 22."}]}
{"id": "mem_028", "memories": [{"type": "fact", "content": "The Wendelsea district sits on limestone bedrock, causing soil pH to drift toward neutral or slightly alkaline."}]}
{"id": "mem_029", "memories": [{"type": "opinion", "content": "The Aerwyn cycle's appendix on language adds meaningfully to the salt-debt motif's payoff in the final act, even if some readers find such artifacts indulgent."}]}
{"id": "mem_030", "memories": [{"type": "decision", "content": "User decided to delay the secondary sale by one quarter to clean up the cap table first."}]}
{"id": "mem_031", "memories": [{"type": "preference", "content": "Pell Halloran prefers email over phone for routine reconciliation questions because it leaves a written trail."}, {"type": "decision", "content": "Will route all standard AR queries to Pell via email; phone reserved for urgent escalations only."}]}
{"id": "mem_032", "memories": [{"type": "fact", "content": "Mason bee colonies at Bridgewater Community Garden emerged on March 18 this year."}, {"type": "fact", "content": "Apple bloom at Wendelsea Orchard started March 20 this year."}]}
{"id": "mem_033", "memories": [{"type": "fact", "content": "Saela's sabbatical begins on June 1."}, {"type": "commitment", "content": "Will review Saela's full 60-page manuscript draft before her sabbatical begins on June 1."}]}
{"id": "mem_034", "memories": [{"type": "opinion", "content": "Agentic coding pipelines generate code faster than anyone can review it and the resulting bugs are harder to trace; would prefer a slower pace that builds understanding."}]}
{"id": "mem_035", "memories": [{"type": "decision", "content": "Switching the invoice parsing pipeline from Tesseract to PaddleOCR; both will run in parallel for about a week during migration."}]}
{"id": "mem_036", "memories": [{"type": "preference", "content": "Ivora prefers conservative cash forecasts over optimistic ones, accepting some loss of accuracy to avoid downside surprises."}, {"type": "decision", "content": "Adjusting the model's reporting to be conservative in line with Ivora's preference."}]}
{"id": "mem_037", "memories": [{"type": "preference", "content": "Karis wants me to attend the Bridgewater Schools garden committee meeting in person on April 10 rather than via video."}, {"type": "commitment", "content": "Confirmed I will attend the April 10 Bridgewater Schools garden committee meeting in person."}]}
{"id": "mem_038", "memories": [{"type": "fact", "content": "Apple scab fungus overwinters in fallen leaves, making fall leaf removal critical."}, {"type": "fact", "content": "Copper sprays for apple scab are only effective at bud-swell; once buds open, the window has closed."}]}
{"id": "mem_039", "memories": [{"type": "opinion", "content": "Saela's third-act decision should be a moment of fierce action rather than collapse — it honors the arc and avoids the cliché."}]}
{"id": "mem_040", "memories": [{"type": "commitment", "content": "Solas committed to shipping the Distributed Tracing fix by end of sprint (Friday)."}, {"type": "fact", "content": "The Distributed Tracing bug has been blocking the Aerwyn Telemetry consortium for two weeks."}]}