The previous commit replaced llm_model.split('-')[1] on the diary pre-compute
line, but run_benchmark builds the same short name again 35 lines later for
the summary header. That one is dash-guarded so it does not crash, yet it
still guesses at a short name, so a single run printed the full tag on one
line and a mangled fragment on the next: "claude-sonnet-4-6" became "sonnet",
and "llama3.2-vision:11b" became "vision:11b".
No dash-split of a model tag is left anywhere under benchmarks/.
llm_model.split('-')[1] raises IndexError for any model tag that contains
no dash (e.g. an Ollama tag like 'qwen3:8b' or an OpenAI-compat route like
'spark/fast'), crashing locomo_bench.py at startup when --llm-rerank is
enabled and longmemeval_bench.py on the diary pre-compute path. Both sites
only build display strings, so print the full model tag instead of
guessing at a short name.
Hit in practice when pointing --llm-backend ollama at a LiteLLM gateway
with provider-prefixed model routes.
The rerank pipeline was hardcoded to Anthropic's /v1/messages.
Add a backend flag so the same code path can be exercised with
any OpenAI-compatible endpoint — local Ollama, Ollama Cloud,
or any gateway that speaks /v1/chat/completions.
Enables independent verification of the "100% with Haiku rerank"
claim by running the full benchmark with a different LLM family
(e.g. minimax-m2.7:cloud) and zero Anthropic dependency.
Both longmemeval_bench.py and locomo_bench.py:
- llm_rerank*() gain backend= / base_url= kwargs
- CLI: --llm-backend {anthropic,ollama}, --llm-base-url
- API key required only when backend=anthropic (diary/palace modes still require it)
- Parse last integer in response (reasoning models emit multi-int output)
- Fallback to message.reasoning when content is empty
- Raise max_tokens to 1024 for reasoning models
The `_load_api_key()` function in longmemeval_bench.py and locomo_bench.py
searched for API keys in a fixed path (`~/.config/lu/keys.json`) using
personal key names (`anthropic_milla`, `anthropic_claude_code_main`).
This leaks internal infrastructure details into the public codebase and
trains contributors to store credentials in a non-standard location
rather than using the standard ANTHROPIC_API_KEY env var.
Simplified to: CLI flag > env var > empty string. Updated help text
and HYBRID_MODE.md docs to match.
Co-authored-by: Tadao <tadao@travisfixes.com>
ATKabli
Igor Lins e Silva
travisBREAKS
bensig