Skeleton for benchmarking ≤4B-parameter Ollama models on MemPalace
classification and extraction tasks. Outputs per-(model, task, mode)
metrics: accuracy, latency (TTFT, TPS, e2e p50/p95), VRAM (resident
and peak).
Layout under benchmarks/model_eval/:
- candidates.yaml: 16 models across 3 tiers, with family/size/variant
metadata pulled from ollama.com/library/<family>/tags
- metrics.py: timing extraction from Ollama response, VRAMPoller for
peak-memory tracking via nvidia-smi, embedding-similarity scoring,
host-info introspection, percentile aggregation
- runner.py: runs one (model, task, mode) triple, dispatches per-task
prompt building and scoring
- orchestrator.py: iterates candidates × tasks, writes CSV
- tasks/{calibration,room_classification,entity_extraction,
memory_extraction}/{prompts.py,score.py}: per-task prompt builders
and scorers
Uses mempalace.llm_client.get_provider directly so the benchmark runs
the same code path as production. strip_thinking_tokens lives locally
in metrics.py for now; will switch to mempalace.local_model once that
module lands on develop.
Datasets land in a follow-up commit.
25 lines
748 B
Python
25 lines
748 B
Python
"""Memory extraction prompts. JSON output."""
|
|
from __future__ import annotations
|
|
|
|
|
|
SYSTEM = """You extract memory-worthy items from agent conversations. Return ONLY valid JSON:
|
|
|
|
{"memories": [{"type": "...", "content": "..."}]}
|
|
|
|
Memory types: decision, preference, fact, opinion, commitment.
|
|
|
|
- decision: a choice made between alternatives
|
|
- preference: a stated like/dislike or way-of-doing
|
|
- fact: an objective piece of information about a person/project/world
|
|
- opinion: a subjective judgment
|
|
- commitment: a promise, plan, or scheduled action
|
|
|
|
Only include items that would be useful to remember. Skip pleasantries, greetings, and generic discussion."""
|
|
|
|
|
|
def build_user(text: str) -> str:
|
|
return f"""Text:
|
|
{text}
|
|
|
|
Extract memory items."""
|