Skeleton for benchmarking ≤4B-parameter Ollama models on MemPalace
classification and extraction tasks. Outputs per-(model, task, mode)
metrics: accuracy, latency (TTFT, TPS, e2e p50/p95), VRAM (resident
and peak).
Layout under benchmarks/model_eval/:
- candidates.yaml: 16 models across 3 tiers, with family/size/variant
metadata pulled from ollama.com/library/<family>/tags
- metrics.py: timing extraction from Ollama response, VRAMPoller for
peak-memory tracking via nvidia-smi, embedding-similarity scoring,
host-info introspection, percentile aggregation
- runner.py: runs one (model, task, mode) triple, dispatches per-task
prompt building and scoring
- orchestrator.py: iterates candidates × tasks, writes CSV
- tasks/{calibration,room_classification,entity_extraction,
memory_extraction}/{prompts.py,score.py}: per-task prompt builders
and scorers
Uses mempalace.llm_client.get_provider directly so the benchmark runs
the same code path as production. strip_thinking_tokens lives locally
in metrics.py for now; will switch to mempalace.local_model once that
module lands on develop.
Datasets land in a follow-up commit.
19 lines
523 B
Python
19 lines
523 B
Python
"""Entity extraction prompts. Output is JSON; we use json_mode=True."""
|
|
from __future__ import annotations
|
|
|
|
|
|
SYSTEM = """You extract entities from text. Return ONLY valid JSON in this exact shape:
|
|
|
|
{"entities": [{"name": "...", "type": "..."}]}
|
|
|
|
Entity types are limited to: person, project, place, organization.
|
|
|
|
Do not invent entities not in the text. Do not include common nouns or generic concepts. Only proper-noun entities."""
|
|
|
|
|
|
def build_user(text: str) -> str:
|
|
return f"""Text:
|
|
{text}
|
|
|
|
Extract all entities."""
|