Files
mempalace/benchmarks/model_eval/tasks/entity_extraction/prompts.py
T
Igor Lins e Silva c0b23b0d8f feat(benchmarks): add small-model evaluation harness
Skeleton for benchmarking ≤4B-parameter Ollama models on MemPalace
classification and extraction tasks. Outputs per-(model, task, mode)
metrics: accuracy, latency (TTFT, TPS, e2e p50/p95), VRAM (resident
and peak).

Layout under benchmarks/model_eval/:
- candidates.yaml: 16 models across 3 tiers, with family/size/variant
  metadata pulled from ollama.com/library/<family>/tags
- metrics.py: timing extraction from Ollama response, VRAMPoller for
  peak-memory tracking via nvidia-smi, embedding-similarity scoring,
  host-info introspection, percentile aggregation
- runner.py: runs one (model, task, mode) triple, dispatches per-task
  prompt building and scoring
- orchestrator.py: iterates candidates × tasks, writes CSV
- tasks/{calibration,room_classification,entity_extraction,
  memory_extraction}/{prompts.py,score.py}: per-task prompt builders
  and scorers

Uses mempalace.llm_client.get_provider directly so the benchmark runs
the same code path as production. strip_thinking_tokens lives locally
in metrics.py for now; will switch to mempalace.local_model once that
module lands on develop.

Datasets land in a follow-up commit.
2026-05-10 09:44:49 -03:00

19 lines
523 B
Python

"""Entity extraction prompts. Output is JSON; we use json_mode=True."""
from __future__ import annotations
SYSTEM = """You extract entities from text. Return ONLY valid JSON in this exact shape:
{"entities": [{"name": "...", "type": "..."}]}
Entity types are limited to: person, project, place, organization.
Do not invent entities not in the text. Do not include common nouns or generic concepts. Only proper-noun entities."""
def build_user(text: str) -> str:
return f"""Text:
{text}
Extract all entities."""