Ten fixes across code, documentation, and committed baselines:
Code fixes:
1. prompts.py: closed-set prompt said "no punctuation" while room
slugs contain "/". Replaced with explicit "copy the slug verbatim
including / and -" wording. Empirical impact on accuracy is within
reproducibility noise (qwen3:4b q4 was 0.610, retest shows 0.594
on the now-101-sample dataset — same vs new sample noise).
2. orchestrator.py: orchestrator now opens the CSV once, writes the
header, and flushes after each row. README's "writes incrementally,
safe to Ctrl-C" claim is now accurate. Long matrix runs no longer
lose all progress on a crash.
3. metrics.py: gather_host_info() now strips trailing newline from
`hostname` command output. Stops embedded newlines from leaking
into committed CSVs.
4. runner.py: passed `endpoint=endpoint` to vram_resident_mb() so
non-default Ollama endpoints are queried correctly.
5. summarize.py: CSV reader and report writer now use
encoding="utf-8" with explicit newline handling for cross-platform
consistency.
Documentation fixes:
6. entity_extraction/score.py: docstring claimed "case-sensitive on
type" while implementation lowercases both. Corrected.
7. dataset.jsonl rc_101: removed mempalace-private URL since the PR
lands on the public repo. Will be linkable after OpenClaw merges.
8. datasets/README.md: sample counts updated (room_classification 100
→ 101, total 210 → 211) to match the cherry-picked real-format
sample.
9. runner.py / orchestrator.py: docstring example paths corrected
(tests/fixtures/mempalace_bench → benchmarks/model_eval/datasets).
Baseline regeneration:
10. Stripped embedded \n from `host` column in all five committed
CSVs (2026-05-10 main local + spotcheck, 2026-05-10 + 2026-05-11
cloud, 2026-05-11 modern). 165 rows cleaned in place via Python
csv reader/writer. Each row is now line-oriented for diff and
grep friendliness without changing any measured values.
39 lines
1.2 KiB
Python
39 lines
1.2 KiB
Python
"""Room classification prompts. Two modes:
|
|
|
|
- closed: room list provided, model picks one or "other"
|
|
- open: no list, model invents a slug
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
|
|
CLOSED_SYSTEM = """You are a room classifier for an AI agent's memory palace. Given a session excerpt and a list of rooms, choose the best-fitting room.
|
|
|
|
Respond with EXACTLY one room slug from the list, copied verbatim (including any "/" or "-" characters), or the literal word "other" if no room fits well. No explanation, no quotes, no extra text."""
|
|
|
|
|
|
OPEN_SYSTEM = """You are a room classifier for an AI agent's memory palace. Given a session excerpt and the agent's identity, INVENT a short room slug that captures the session's topic.
|
|
|
|
Respond with EXACTLY one slug: lowercase, hyphenated, no spaces, no punctuation, no quotes. Example formats: "project-alpha", "code-review", "daily-logs"."""
|
|
|
|
|
|
def build_closed_user(agent: str, session_summary: str, rooms: list[str]) -> str:
|
|
rooms_block = "\n".join(f"- {r}" for r in rooms)
|
|
return f"""Agent: {agent}
|
|
|
|
Available rooms:
|
|
{rooms_block}
|
|
|
|
Session excerpt:
|
|
{session_summary}
|
|
|
|
Room:"""
|
|
|
|
|
|
def build_open_user(agent: str, session_summary: str) -> str:
|
|
return f"""Agent: {agent}
|
|
|
|
Session excerpt:
|
|
{session_summary}
|
|
|
|
Room slug:"""
|