210 hand-quality samples across four tasks. Synthetic only — no real-person info. Five fictional personae (Aria, Solas, Fenra, Bramble, Thresh) with distinct domains, room taxonomies, and relationship contexts. - room_classification: 100 samples (20 per agent), 14 marked with realistic noise features, 18 'other' / 6 'general' / 76 specific- room labels. 5-10 rooms per sample with 'general' always present. - entity_extraction: 50 samples, 247 entities (114 person, 74 org, 32 project, 27 place). All entity strings verified to appear in source text. Includes deliberate same-surname-different-people cases as disambiguation tests. - memory_extraction: 40 samples, 55 memories across types (15 fact, 12 decision, 12 commitment, 9 preference, 7 opinion). - calibration: 20 samples, 4 per class (question, command, statement, exclamation, greeting). Sanity check that the harness measures what we think. All datasets validated: ID alignment across dataset/labels pairs, closed-set labels are members of their room lists (or 'other'), calibration labels are members of their classes lists, no leak terms (igor, milla, domi, lumi, anthropic, openclaw, etc.) anywhere. Generated 2026-05-10 via subagent + hand-validation. README in the datasets directory documents the personae, distribution stats, and labeling conventions.
MemPalace Small-Model Benchmark Datasets
Synthetic, hand-curated evaluation datasets for testing small (≤4B parameter) Ollama models on the four classification and extraction tasks that matter for MemPalace's local-first memory pipeline.
- Synthetic only. No real-person names, no real organizations, no real PII.
- Public-safe. Intended to be committed to the public MemPalace open-source repository.
- Generated 2026-05-10.
- 210 samples total across four tasks.
Tasks
| Task | Samples | Purpose |
|---|---|---|
room_classification/ |
100 | Given a session summary and a room list, pick the right room (or "other"). Tests the closed-set room-routing path used at filing time. |
entity_extraction/ |
50 | Given a conversation excerpt, extract typed entities (person / project / place / organization). Tests the entity detector that feeds the knowledge graph. |
memory_extraction/ |
40 | Given a snippet, extract memory-worthy items (decision / preference / fact / opinion / commitment). Tests the layer-promotion heuristic. |
calibration/ |
20 | Sentence-type classification (question / command / statement / exclamation / greeting). A trivially-easy baseline used to detect a broken model load before more meaningful evals run. |
The five fictional personae
The 100 room-classification samples are split 20-per-agent across these synthetic agents. Each agent has a fixed room taxonomy; per-sample room lists are drawn as 5-10 room subsets that always include general.
1. Aria — research assistant (she/her)
Synthesizes academic papers.
Rooms: about, projects/meta-cognition, projects/embedding-spaces, projects/topic-clustering, skills/latex, skills/python, skills/statistical-tests, daily-logs, general
2. Solas — coding agent (they/them)
Writes and reviews code; focuses on systems work.
Rooms: about, projects/distributed-tracing, projects/parser-combinators, projects/type-systems, skills/rust, skills/ocaml, skills/llvm, daily-logs, general
3. Fenra — creative writing agent (she/her)
Drafts fiction and worldbuilds.
Rooms: about, projects/world-building, projects/character-arcs, projects/dialogue-engine, skills/narrative-design, skills/etymology, skills/mythology, daily-logs, general
4. Bramble — gardening agent (he/him)
Companion-plant and ecological-gardening advice.
Rooms: about, projects/pollinator-paths, projects/soil-microbes, projects/native-species, skills/phenology, skills/plant-pathology, skills/propagation, daily-logs, general
5. Thresh — finance/accounting agent (they/them)
Small-business bookkeeping.
Rooms: about, projects/invoice-parsing, projects/tax-prep, projects/cashflow-models, skills/double-entry, skills/depreciation-rules, skills/ifrs, daily-logs, general
Distribution stats
room_classification/ (100 samples)
| Metric | Value |
|---|---|
| Samples per agent | Aria 20, Solas 20, Fenra 20, Bramble 20, Thresh 20 |
Messy samples (include_messy_features: true) |
14 (target: ~15%) |
Closed-set "other" |
18 (target: ~20%) |
Closed-set "general" (escape hatch used as best fit) |
6 |
Closed-set match to a non-general non-other room |
76 |
"other" distribution by agent: Aria 2, Solas 3, Fenra 3, Bramble 7, Thresh 3. Bramble skews higher because gardening conversations frequently span topics that don't cleanly fit a single skill room (e.g. seed-saving + plant pathology, soil chemistry, vegetable-pest control).
entity_extraction/ (50 samples)
| Metric | Value |
|---|---|
| Total entities | 247 |
person |
114 |
organization |
74 |
project |
32 |
place |
27 |
| Entities per sample (min / max / avg) | 3 / 9 / 4.9 |
Person-skew is intentional: real MemPalace sessions are heavily person-centric, and the entity detector's hardest job is disambiguating people (the dataset deliberately reuses some last names like "Halloran" across distinct individuals to test this).
memory_extraction/ (40 samples)
| Metric | Value |
|---|---|
| Total memories | 55 |
decision |
12 |
preference |
9 |
commitment |
12 |
fact |
15 |
opinion |
7 |
| Memories per sample | 1 (25 samples) or 2 (15 samples) |
calibration/ (20 samples)
Exactly 4 samples per class: question, command, statement, exclamation, greeting. Designed to be unambiguous — any working model should hit ≥95% accuracy here. If a model fails calibration, it is broken or misconfigured and the more interesting tasks should not be run.
Schema reference
room_classification/dataset.jsonl
{"id": "rc_001", "agent": "Aria", "session_summary": "...", "include_messy_features": false}
room_classification/labels.jsonl
{"id": "rc_001", "closed_set_label": "projects/meta-cognition", "preferred_open_label": "meta-cognition-research"}
closed_set_label is either a room from that sample's rooms list, or exactly "other".
preferred_open_label is the slug a human annotator would invent if labeling freely (lowercase, hyphenated).
room_classification/room_lists.jsonl
{"id": "rc_001", "rooms": ["about", "projects/meta-cognition", "skills/python", "daily-logs", "general"]}
Always includes "general". Always 5-10 rooms. Composition varies per sample so the model can't just memorize an agent's full taxonomy.
entity_extraction/dataset.jsonl + labels.jsonl
{"id": "ent_001", "text": "..."}
{"id": "ent_001", "entities": [{"name": "Aria", "type": "person"}, {"name": "Embedding Spaces", "type": "project"}]}
Entity types: person, project, place, organization.
memory_extraction/dataset.jsonl + labels.jsonl
{"id": "mem_001", "text": "..."}
{"id": "mem_001", "memories": [{"type": "decision", "content": "..."}]}
Memory types: decision, preference, fact, opinion, commitment.
calibration/dataset.jsonl + labels.jsonl
{"id": "cal_001", "text": "Could you fix the indentation on this Python file?", "classes": ["question", "command", "statement", "exclamation", "greeting"]}
{"id": "cal_001", "label": "command"}
The classes field is identical across all 20 samples, included in each line for self-contained downstream loading.
Annotation conventions
- Decision vs. commitment. A decision changes how something will be done from now on (a policy or design choice); a commitment binds the speaker to a specific deliverable, often with a deadline. "Switching to Jaccard similarity" is a decision; "I'll deliver the draft Friday" is a commitment. When both apply to one sentence, both are emitted.
- Fact vs. opinion. A fact is a claim about the world that is checkable in principle. An opinion is signaled by hedge words ("I think", "honestly", "my read") or by being a value judgement. Borderline cases (e.g. an arguable empirical claim about a paper) lean to opinion when the speaker frames it as their own assessment.
- Preference. Preferences attach to a person and describe how that person likes things done. They are durable across conversations, unlike one-off decisions.
- Closed-set "other". Used when no listed room is a substantively better home for the content than
general. Ifgeneralis a clean fit (catch-all small talk, identity questions), the label isgeneralrather thanother. - Ambiguous closed-set samples (~10% of room_classification) have two rooms that fit; the label picks the one that better matches the dominant content of the session.
preferred_open_labelmay name the secondary topic when the closed-set choice is forced.
Extending
To regenerate or extend:
- Add new personae. Define a name, pronouns, role, and 7-9 rooms (always include
about,daily-logs,general). Add 20 samples covering each room at least once, plus a few drift / messy /"other"cases. - Add more samples. Keep the per-agent balance even. Maintain ~70/20/10 distribution for closed-set /
"other"/ ambiguous. - Add new tasks. Pick a discriminative task that small models would plausibly fail in interesting ways (relation extraction, coreference, stance, etc.). Match the pattern of separate
dataset.jsonlandlabels.jsonlfiles keyed byid.
Provenance
- Synthetic, generated 2026-05-10 for the MemPalace project.
- No real-person names. Fictional personae and place/organization names invented for this benchmark.
- Safe to commit to a public repository.