Adds a "Contributions welcome" section to the README covering:
1. Adding models we missed. The published candidate list is one
engineer + one search pass. Particularly call out function-calling-
tuned variants (Phi-4 mini, Nemotron-mini), research-lab instruct
models (Hermes, Dolphin), and quantization-aware-trained variants
as worth-adding categories.
2. Adding other inference backends. mempalace.llm_client.get_provider
already supports openai-compat, which means LM Studio, llama.cpp
server, vLLM, unsloth studio, Docker Model Runner, HF TGI all
plug in with minimal harness changes. Documents the four pieces a
backend contributor would need to touch:
- Backend-selection flag in runner/orchestrator
- Backend-specific timing extraction in metrics.py
- Backend-specific VRAM probe
- candidates.yaml backend field
Notes that accuracy numbers stay comparable across backends since
the dataset and scoring code are backend-agnostic.
Also calls out niche runtimes worth cross-comparison: Apple MLX,
Intel OpenVINO, AMD ROCm, edge-device runtimes.
Existing harness-maintainer notes (cloud structured-outputs gotcha,
memory hallucination metric caveat, cloud reproducibility variance)
moved under a separate heading.
The previous README had a minimal quick-start section. Replaced with
a proper end-to-end reproduction guide for contributors who want to
validate the published numbers on their own hardware.
New sections:
- Prerequisites: GPU, Ollama version, Python, disk, optional cloud auth
- Set up the environment
- Pull candidate models (with a bulk-pull Python snippet that reads
candidates.yaml directly so it stays in sync with future additions)
- Smoke test (~30 sec, confirms harness works)
- Run the matrix (Tier 1 / all local / cloud, with time estimates)
- Render a report
- Compare with the published baseline (what kind of agreement to expect
on accuracy vs speed vs VRAM)
- Sharing your results (how to attach a CSV to a follow-up PR or
open an issue if numbers diverge)
- Notes for new contributors to the harness itself (the known cloud
structured-output limitation, the memory-extraction hallucination
metric caveat, cloud reproducibility variance)
Tier filter docs now include the `modern` filter added in the
modern-tier expansion. Single-model runs by exact tag are documented.
Goal: someone cloning the repo for the first time should be able to
reproduce a Tier 1 matrix in ~45 min without reading anything else.
Skeleton for benchmarking ≤4B-parameter Ollama models on MemPalace
classification and extraction tasks. Outputs per-(model, task, mode)
metrics: accuracy, latency (TTFT, TPS, e2e p50/p95), VRAM (resident
and peak).
Layout under benchmarks/model_eval/:
- candidates.yaml: 16 models across 3 tiers, with family/size/variant
metadata pulled from ollama.com/library/<family>/tags
- metrics.py: timing extraction from Ollama response, VRAMPoller for
peak-memory tracking via nvidia-smi, embedding-similarity scoring,
host-info introspection, percentile aggregation
- runner.py: runs one (model, task, mode) triple, dispatches per-task
prompt building and scoring
- orchestrator.py: iterates candidates × tasks, writes CSV
- tasks/{calibration,room_classification,entity_extraction,
memory_extraction}/{prompts.py,score.py}: per-task prompt builders
and scorers
Uses mempalace.llm_client.get_provider directly so the benchmark runs
the same code path as production. strip_thinking_tokens lives locally
in metrics.py for now; will switch to mempalace.local_model once that
module lands on develop.
Datasets land in a follow-up commit.
Igor Lins e Silva