mempalace/benchmarks/model_eval/datasets

MemPalace Small-Model Benchmark Datasets

Synthetic, hand-curated evaluation datasets for testing small (≤4B parameter) Ollama models on the four classification and extraction tasks that matter for MemPalace's local-first memory pipeline.

• Synthetic only. No real-person names, no real organizations, no real PII.
• Public-safe. Intended to be committed to the public MemPalace open-source repository.
• Generated 2026-05-10.
• 211 samples total across four tasks (100 synthetic + 1 real-format-flavored sample in room_classification).

Tasks

Task	Samples	Purpose
room_classification/	101	Given a session summary and a room list, pick the right room (or "other"). Tests the closed-set room-routing path used at filing time. 100 synthetic + 1 real-format-flavored (rc_101).
entity_extraction/	50	Given a conversation excerpt, extract typed entities (person / project / place / organization). Tests the entity detector that feeds the knowledge graph.
memory_extraction/	40	Given a snippet, extract memory-worthy items (decision / preference / fact / opinion / commitment). Tests the layer-promotion heuristic.
calibration/	20	Sentence-type classification (question / command / statement / exclamation / greeting). A trivially-easy baseline used to detect a broken model load before more meaningful evals run.

The five fictional personae

The 100 synthetic room-classification samples are split 20-per-agent across these synthetic agents (rc_101 is a separate real-format-flavored sample assigned to Solas). Each agent has a fixed room taxonomy; per-sample room lists are drawn as 5-10 room subsets that always include general.

1. Aria — research assistant (she/her)

Synthesizes academic papers.

Rooms: about, projects/meta-cognition, projects/embedding-spaces, projects/topic-clustering, skills/latex, skills/python, skills/statistical-tests, daily-logs, general

2. Solas — coding agent (they/them)

Writes and reviews code; focuses on systems work.

Rooms: about, projects/distributed-tracing, projects/parser-combinators, projects/type-systems, skills/rust, skills/ocaml, skills/llvm, daily-logs, general

3. Fenra — creative writing agent (she/her)

Drafts fiction and worldbuilds.

Rooms: about, projects/world-building, projects/character-arcs, projects/dialogue-engine, skills/narrative-design, skills/etymology, skills/mythology, daily-logs, general

4. Bramble — gardening agent (he/him)

Companion-plant and ecological-gardening advice.

Rooms: about, projects/pollinator-paths, projects/soil-microbes, projects/native-species, skills/phenology, skills/plant-pathology, skills/propagation, daily-logs, general

5. Thresh — finance/accounting agent (they/them)

Small-business bookkeeping.

Rooms: about, projects/invoice-parsing, projects/tax-prep, projects/cashflow-models, skills/double-entry, skills/depreciation-rules, skills/ifrs, daily-logs, general

Distribution stats

room_classification/ (101 samples)

Metric	Value
Samples per agent	Aria 20, Solas 20, Fenra 20, Bramble 20, Thresh 20
Messy samples (include_messy_features: true)	14 (target: ~15%)
Closed-set "other"	18 (target: ~20%)
Closed-set "general" (escape hatch used as best fit)	6
Closed-set match to a non-general non-other room	76

"other" distribution by agent: Aria 2, Solas 3, Fenra 3, Bramble 7, Thresh 3. Bramble skews higher because gardening conversations frequently span topics that don't cleanly fit a single skill room (e.g. seed-saving + plant pathology, soil chemistry, vegetable-pest control).

entity_extraction/ (50 samples)

Metric	Value
Total entities	247
person	114
organization	74
project	32
place	27
Entities per sample (min / max / avg)	3 / 9 / 4.9

Person-skew is intentional: real MemPalace sessions are heavily person-centric, and the entity detector's hardest job is disambiguating people (the dataset deliberately reuses some last names like "Halloran" across distinct individuals to test this).

memory_extraction/ (40 samples)

Metric	Value
Total memories	55
decision	12
preference	9
commitment	12
fact	15
opinion	7
Memories per sample	1 (25 samples) or 2 (15 samples)

calibration/ (20 samples)

Exactly 4 samples per class: question, command, statement, exclamation, greeting. Designed to be unambiguous — any working model should hit ≥95% accuracy here. If a model fails calibration, it is broken or misconfigured and the more interesting tasks should not be run.

Schema reference

room_classification/dataset.jsonl

{"id": "rc_001", "agent": "Aria", "session_summary": "...", "include_messy_features": false}

room_classification/labels.jsonl

{"id": "rc_001", "closed_set_label": "projects/meta-cognition", "preferred_open_label": "meta-cognition-research"}

closed_set_label is either a room from that sample's rooms list, or exactly "other". preferred_open_label is the slug a human annotator would invent if labeling freely (lowercase, hyphenated).

room_classification/room_lists.jsonl

{"id": "rc_001", "rooms": ["about", "projects/meta-cognition", "skills/python", "daily-logs", "general"]}

Always includes "general". Always 5-10 rooms. Composition varies per sample so the model can't just memorize an agent's full taxonomy.

entity_extraction/dataset.jsonl + labels.jsonl

{"id": "ent_001", "text": "..."}
{"id": "ent_001", "entities": [{"name": "Aria", "type": "person"}, {"name": "Embedding Spaces", "type": "project"}]}

Entity types: person, project, place, organization.

memory_extraction/dataset.jsonl + labels.jsonl

{"id": "mem_001", "text": "..."}
{"id": "mem_001", "memories": [{"type": "decision", "content": "..."}]}

Memory types: decision, preference, fact, opinion, commitment.

calibration/dataset.jsonl + labels.jsonl

{"id": "cal_001", "text": "Could you fix the indentation on this Python file?", "classes": ["question", "command", "statement", "exclamation", "greeting"]}
{"id": "cal_001", "label": "command"}

The classes field is identical across all 20 samples, included in each line for self-contained downstream loading.

Annotation conventions

• Decision vs. commitment. A decision changes how something will be done from now on (a policy or design choice); a commitment binds the speaker to a specific deliverable, often with a deadline. "Switching to Jaccard similarity" is a decision; "I'll deliver the draft Friday" is a commitment. When both apply to one sentence, both are emitted.
• Fact vs. opinion. A fact is a claim about the world that is checkable in principle. An opinion is signaled by hedge words ("I think", "honestly", "my read") or by being a value judgement. Borderline cases (e.g. an arguable empirical claim about a paper) lean to opinion when the speaker frames it as their own assessment.
• Preference. Preferences attach to a person and describe how that person likes things done. They are durable across conversations, unlike one-off decisions.
• Closed-set "other". Used when no listed room is a substantively better home for the content than general. If general is a clean fit (catch-all small talk, identity questions), the label is general rather than other.
• Ambiguous closed-set samples (~10% of room_classification) have two rooms that fit; the label picks the one that better matches the dominant content of the session. preferred_open_label may name the secondary topic when the closed-set choice is forced.

Extending

To regenerate or extend:

1. Add new personae. Define a name, pronouns, role, and 7-9 rooms (always include about, daily-logs, general). Add 20 samples covering each room at least once, plus a few drift / messy / "other" cases.
2. Add more samples. Keep the per-agent balance even. Maintain ~70/20/10 distribution for closed-set / "other" / ambiguous.
3. Add new tasks. Pick a discriminative task that small models would plausibly fail in interesting ways (relation extraction, coreference, stance, etc.). Match the pattern of separate dataset.jsonl and labels.jsonl files keyed by id.

Provenance

• Synthetic, generated 2026-05-10 for the MemPalace project.
• No real-person names. Fictional personae and place/organization names invented for this benchmark.
• Safe to commit to a public repository.