Files
mempalace/benchmarks/model_eval/tasks/memory_extraction/prompts.py
T
Igor Lins e Silva c0b23b0d8f feat(benchmarks): add small-model evaluation harness
Skeleton for benchmarking ≤4B-parameter Ollama models on MemPalace
classification and extraction tasks. Outputs per-(model, task, mode)
metrics: accuracy, latency (TTFT, TPS, e2e p50/p95), VRAM (resident
and peak).

Layout under benchmarks/model_eval/:
- candidates.yaml: 16 models across 3 tiers, with family/size/variant
  metadata pulled from ollama.com/library/<family>/tags
- metrics.py: timing extraction from Ollama response, VRAMPoller for
  peak-memory tracking via nvidia-smi, embedding-similarity scoring,
  host-info introspection, percentile aggregation
- runner.py: runs one (model, task, mode) triple, dispatches per-task
  prompt building and scoring
- orchestrator.py: iterates candidates × tasks, writes CSV
- tasks/{calibration,room_classification,entity_extraction,
  memory_extraction}/{prompts.py,score.py}: per-task prompt builders
  and scorers

Uses mempalace.llm_client.get_provider directly so the benchmark runs
the same code path as production. strip_thinking_tokens lives locally
in metrics.py for now; will switch to mempalace.local_model once that
module lands on develop.

Datasets land in a follow-up commit.
2026-05-10 09:44:49 -03:00

25 lines
748 B
Python

"""Memory extraction prompts. JSON output."""
from __future__ import annotations
SYSTEM = """You extract memory-worthy items from agent conversations. Return ONLY valid JSON:
{"memories": [{"type": "...", "content": "..."}]}
Memory types: decision, preference, fact, opinion, commitment.
- decision: a choice made between alternatives
- preference: a stated like/dislike or way-of-doing
- fact: an objective piece of information about a person/project/world
- opinion: a subjective judgment
- commitment: a promise, plan, or scheduled action
Only include items that would be useful to remember. Skip pleasantries, greetings, and generic discussion."""
def build_user(text: str) -> str:
return f"""Text:
{text}
Extract memory items."""