Addresses @Copilot's review feedback on #1459. Five tests:
- test_extract_drawers_preserves_valid_metadata: non-empty dict
passes through unchanged (regression guard against breaking happy
path).
- test_extract_drawers_sanitizes_none_metadata: None entries
coerce to {"_repaired_empty_meta": True} (the core fix).
- test_extract_drawers_sanitizes_empty_dict_metadata: empty dict
{} entries also coerce to the sentinel (chromadb 1.5.x rejects
both shapes equally).
- test_extract_drawers_sanitization_preserves_alignment: critical
invariant — ids[i] / documents[i] / metadatas[i] stay in
lockstep through the sanitizer; mis-pairing would silently
corrupt rebuilds.
- test_extract_drawers_multiple_batches: pagination boundary
correctness (sanitizer applied per-batch, no drops/duplicates).
Verified passing locally against mempalace fork main + chromadb
1.5.8 in the palace-daemon venv (5 passed, 67 deselected in 1.88s).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Belt-and-suspenders on top of the repair.py sanitization in the
previous commit. A 151,478-drawer rebuild test still failed at
~120K with the same `ValueError: Expected metadata to be a
non-empty dict` from chromadb, even with the repair.py sanitizers
in place. Traceback:
mempalace/backends/chroma.py:add → chromadb Collection.add
→ validate_insert_record_set → validate_metadatas
→ validate_metadata → ValueError
Likely cause: chromadb's `upsert()` internally calls `add()` for
new records, and somewhere between repair.py's batch upsert and
chromadb's final write, the metadatas list gets reprocessed in a
way that re-introduces empty/None entries.
Sanitizing at the chromadb-client chokepoint catches everything:
no caller can leak bad metadata regardless of upstream sanitization
state. Same `{"_repaired_empty_meta": True}` sentinel, searchable
via `where={"_repaired_empty_meta": True}`.
Cost: one list comprehension per write call; negligible vs the
embedding + HNSW work each upsert already does.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two functions construct the metadatas[] list that gets fed to
chromadb's upsert/add, both vulnerable to the same ValueError:
ValueError: Expected metadata to be a non-empty dict, got 0
metadata attributes in add.
chromadb 1.5.x's validate_metadata rejects both `None` and `{}`
entries — see chromadb/api/types.py:validate_metadata (line ~1071).
This commit patches both:
1. `_extract_drawers` (line ~131) — the chromadb-collection-based
extractor, used when the source palace's collection is openable
via the chromadb client. Sanitizes None/{} entries in the
`batch["metadatas"]` list to `{"_repaired_empty_meta": True}`
before extending `all_metas`.
2. `_rebuild_one_collection` (line ~813) — the SQLite-direct
extract path used by `rebuild_from_sqlite()`, invoked when the
source palace can't be opened via chromadb (the recovery path
for palaces with corrupt HNSW segments). Old code was:
metas.append(meta if meta else {})
The trailing `{}` was the bug; chromadb 1.5.x rejects empty
dicts the same as None. Replaced with the same sentinel.
Why `_repaired_empty_meta: True` as the sentinel:
- Satisfies chromadb's non-empty-dict requirement
- Bool-valued (valid chromadb metadata type, trivially serializable)
- Namespaced + descriptive so an operator can find which drawers
were coerced via `where={"_repaired_empty_meta": True}` later
- Idempotent on re-runs (a future repair over a sanitized palace
sees the sentinel as already-valid)
Verified on a 151,478-drawer production palace that previously
crashed at drawer 120,000 in both extract paths.
Fixes#1458
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tasks that score via semantic similarity (memory_extraction,
room_classification open-set) depend on nomic-embed-text. If the model
was absent, embed_text() returned None silently and every sample scored
0.0 — indistinguishable from a genuinely bad model.
_ensure_embed_model() now probes the endpoint before the warmup step.
On a 404 it pulls the model automatically via `ollama pull`, verifies
the endpoint responds, and only then proceeds. Fixes:
- OLLAMA_HOST is now forwarded to `ollama pull` so non-localhost
endpoints are targeted correctly
- FileNotFoundError is caught and re-raised as RuntimeError with an
install hint if ollama is not on PATH
- Diagnostic prints go to stderr, not stdout
- If the pull or re-probe fails, returns a Result with a clear error
message instead of silent zeros
Restores --candidates local to its documented meaning (all non-cloud
models). Adds --candidates community for third-party GGUF variants.
Ten fixes across code, documentation, and committed baselines:
Code fixes:
1. prompts.py: closed-set prompt said "no punctuation" while room
slugs contain "/". Replaced with explicit "copy the slug verbatim
including / and -" wording. Empirical impact on accuracy is within
reproducibility noise (qwen3:4b q4 was 0.610, retest shows 0.594
on the now-101-sample dataset — same vs new sample noise).
2. orchestrator.py: orchestrator now opens the CSV once, writes the
header, and flushes after each row. README's "writes incrementally,
safe to Ctrl-C" claim is now accurate. Long matrix runs no longer
lose all progress on a crash.
3. metrics.py: gather_host_info() now strips trailing newline from
`hostname` command output. Stops embedded newlines from leaking
into committed CSVs.
4. runner.py: passed `endpoint=endpoint` to vram_resident_mb() so
non-default Ollama endpoints are queried correctly.
5. summarize.py: CSV reader and report writer now use
encoding="utf-8" with explicit newline handling for cross-platform
consistency.
Documentation fixes:
6. entity_extraction/score.py: docstring claimed "case-sensitive on
type" while implementation lowercases both. Corrected.
7. dataset.jsonl rc_101: removed mempalace-private URL since the PR
lands on the public repo. Will be linkable after OpenClaw merges.
8. datasets/README.md: sample counts updated (room_classification 100
→ 101, total 210 → 211) to match the cherry-picked real-format
sample.
9. runner.py / orchestrator.py: docstring example paths corrected
(tests/fixtures/mempalace_bench → benchmarks/model_eval/datasets).
Baseline regeneration:
10. Stripped embedded \n from `host` column in all five committed
CSVs (2026-05-10 main local + spotcheck, 2026-05-10 + 2026-05-11
cloud, 2026-05-11 modern). 165 rows cleaned in place via Python
csv reader/writer. Each row is now line-oriented for diff and
grep friendliness without changing any measured values.
Adds a "Contributions welcome" section to the README covering:
1. Adding models we missed. The published candidate list is one
engineer + one search pass. Particularly call out function-calling-
tuned variants (Phi-4 mini, Nemotron-mini), research-lab instruct
models (Hermes, Dolphin), and quantization-aware-trained variants
as worth-adding categories.
2. Adding other inference backends. mempalace.llm_client.get_provider
already supports openai-compat, which means LM Studio, llama.cpp
server, vLLM, unsloth studio, Docker Model Runner, HF TGI all
plug in with minimal harness changes. Documents the four pieces a
backend contributor would need to touch:
- Backend-selection flag in runner/orchestrator
- Backend-specific timing extraction in metrics.py
- Backend-specific VRAM probe
- candidates.yaml backend field
Notes that accuracy numbers stay comparable across backends since
the dataset and scoring code are backend-agnostic.
Also calls out niche runtimes worth cross-comparison: Apple MLX,
Intel OpenVINO, AMD ROCm, edge-device runtimes.
Existing harness-maintainer notes (cloud structured-outputs gotcha,
memory hallucination metric caveat, cloud reproducibility variance)
moved under a separate heading.
The previous README had a minimal quick-start section. Replaced with
a proper end-to-end reproduction guide for contributors who want to
validate the published numbers on their own hardware.
New sections:
- Prerequisites: GPU, Ollama version, Python, disk, optional cloud auth
- Set up the environment
- Pull candidate models (with a bulk-pull Python snippet that reads
candidates.yaml directly so it stays in sync with future additions)
- Smoke test (~30 sec, confirms harness works)
- Run the matrix (Tier 1 / all local / cloud, with time estimates)
- Render a report
- Compare with the published baseline (what kind of agreement to expect
on accuracy vs speed vs VRAM)
- Sharing your results (how to attach a CSV to a follow-up PR or
open an issue if numbers diverge)
- Notes for new contributors to the harness itself (the known cloud
structured-output limitation, the memory-extraction hallucination
metric caveat, cloud reproducibility variance)
Tier filter docs now include the `modern` filter added in the
modern-tier expansion. Single-model runs by exact tag are documented.
Goal: someone cloning the repo for the first time should be able to
reproduce a Tier 1 matrix in ~45 min without reading anything else.
Five model families that existed at the time of the first benchmark
pass but weren't surfaced by the initial ollama.com search:
- qwen3.5:4b-q4_K_M
- gemma4:e2b-it-q4_K_M
- gemma4:e4b-it-q4_K_M
- granite4.1:3b-q4_K_M
- ministral-3:3b
Grouped as tier="modern" with a new orchestrator filter so they can
be re-run independently. 25 runs in 11.2 minutes, zero errors.
Three findings reshape the recommendations:
1. gemma4:e4b-it-q4_K_M is the new local leader for room classification.
Closed-set 0.62 (vs qwen3:4b q4 at 0.61), open-set 0.65 (highest
score across all measured models including cloud). The cloud
"ceiling" on open-set was 0.61 — a 4B local model now beats every
cloud reference up to 1T parameters on this task.
2. ministral-3:3b has 0.99 memory coverage — nearly cloud-tier.
But weak elsewhere (closed-set 0.49, entity 0.63). Specialist
not generalist.
3. qwen3.5:4b does NOT beat qwen3:4b on this workload despite the
version bump. Slightly better entity F1, worse on memory and
closed-set. Newer != better.
Production-tier update:
- Room classification (OpenClaw use case): gemma4:e4b becomes the new
recommended default. Costs 2.1x latency and 1.4x VRAM vs qwen3:4b
for the room-task lift.
- General extraction on tight hardware: qwen3:4b q4_K_M still wins.
- Open-set discovery: softens from "shelved indefinitely" to "retest
with gemma4:e4b + prompt tuning before declaring dead." Gap to
0.70 ship threshold is now 0.05, not 0.09.
Four fixes from automated code review:
1. score.py: pre-compute embeddings per content string to cut API
calls from O(P*T) to O(P+T). Was making 2*P*T HTTP round-trips
to Ollama embeddings per memory_extraction sample; now P+T.
Significant speed-up on cloud endpoints where each embed is a
network call.
2. metrics.py: bump VRAMPoller default interval from 100ms to 500ms.
nvidia-smi subprocess every 100ms was creating measurable jitter
on the timing numbers. 500ms still captures peak (inference VRAM
is steady-state during a request) with much less overhead.
3. orchestrator.py: use argparse.BooleanOptionalAction for
--continue-on-error so --no-continue-on-error actually disables
the default. The previous combination of store_true and
default=True made the flag a permanent True.
4. summarize.py: drop the buggy first attempt at the reasoning
filter (had operator-precedence issue with `or` and `and`).
Keep only the clean prefix-based filter that was already on
the next line shadowing it.
Replaced kimi-k2:1t-cloud (retired, HTTP 500 across the board) with
kimi-k2.6:cloud. Added deepseek-v4-flash:cloud and deepseek-v4-pro:cloud
as newer-generation references.
Ran the v2 cloud matrix at n=30 on 7 models. All succeeded. Key findings:
- gpt-oss:20b-cloud and qwen3-coder:480b-cloud tie at ~0.90 on closed-set
room classification. gpt-oss matches a 24x larger coder-specialized
model on natural-language tasks; coder-specialization isn't a benefit
for this workload.
- deepseek-v4-flash hits 0.607 on open-set room similarity (best cloud
result). Local best is 0.612 (gemma3:4b-it). Cloud still doesn't beat
local on this task; the open-set ceiling is genuinely refuted across
two cloud generations and the full 4B-1T size range.
- kimi-k2.6:cloud showed only 37% valid_json_rate on memory extraction.
Per docs.ollama.com/capabilities/structured-outputs, "Ollama's Cloud
currently does not support structured outputs." The harness's
format=json flag is silently ignored on cloud. Other cloud models
happened to emit JSON by default; K2.6 doesn't. Documented as a
methodological caveat. Follow-up issue: envelope-aware JSON
extraction fallback for cloud responses.
- gpt-oss:20b showed 6.4-point closed-set drift between v1 and v2
(0.833 → 0.897). Bigger than local reproducibility noise (±0.7%).
Likely cloud-side model rotation or server-side non-determinism.
Cloud findings should be quoted as ranges, not point estimates.
Raw data: results/2026-05-11-cloud-z690-ex-glacial.csv (v2). The v1 CSV
stays in the tree as historical record of the kimi-k2:1t-cloud failure.
Ran four of five cloud candidates at n=30. Kimi K2 1T returned HTTP
500 on every task — retry pending. Four working models:
| Model | room-closed | room-open | entity F1 | mem cov |
|---|---|---|---|---|
| gpt-oss:20b-cloud | 0.833 | 0.555 | 0.755 | 1.000 |
| gpt-oss:120b-cloud | 0.800 | 0.553 | 0.829 | 1.000 |
| qwen3-coder:480b-cloud | 0.900 | 0.587 | 0.804 | 0.967 |
| deepseek-v3.1:671b-cloud | 0.800 | 0.566 | 0.828 | 0.967 |
| local leader (4B q4) | 0.610 | 0.586 | 0.778 | 0.950 |
Two headline conclusions:
1. Closed-set room classification HAS a real ceiling gap. Cloud at
0.83-0.90 vs local at 0.61. Strongest argument so far for an
optional --classifier cloud path for users willing to trade
privacy/cost for accuracy. Default stays local.
2. Open-set discovery ceiling is REFUTED. Best cloud is 0.587 vs
best local 0.612. Cloud is slightly WORSE. Not a capacity problem;
it's a task-formulation problem that more compute doesn't fix.
Recommendation: shelve --mode discover until prompt design or
two-pass clustering closes the gap to >0.75.
Also: gpt-oss models continue emitting reasoning even with think=false
in the request body (verified via curl). Content field stays clean,
but cloud latency includes reasoning generation we asked it to skip.
Worth an upstream issue against Ollama Cloud.
Three follow-ups since the first pass:
1. Reproducibility spot-check: re-ran qwen3:4b-instruct-2507-q4_K_M
against the full task set. Accuracy deltas vs original ≤0.7% on
every metric. Harness is reliable; single-run numbers can be trusted.
2. Memory-extraction hallucination investigation. Hand-inspected 5
samples and found qwen3:4b's 0.36 "hallucination rate" is a scoring
artifact, not a model weakness:
- ~50% are bundled-truth-split-into-atomic predictions
- ~30% are ground-truth omissions the model correctly caught
- <5% are genuine hallucinations
Meanwhile qwen2.5:3b's "0.00 hallucination" is just under-extraction.
The mean_hallucination_rate metric over-penalizes thorough models.
qwen3:4b stays the production recommendation. Follow-up: refine
scoring to use source-text traceability instead of greedy match.
3. Cloud-tier ceiling measurement attempted, blocked on ollama signin
(interactive OAuth). Five cloud candidates added to candidates.yaml
for the next pass.
Also: committed the spot-check CSV as a reproducibility baseline.
Adds five Ollama Cloud reference models for ceiling measurement:
- gpt-oss:20b-cloud (lighter reference)
- gpt-oss:120b-cloud
- qwen3-coder:480b-cloud (code-tuned variant)
- deepseek-v3.1:671b-cloud
- kimi-k2:1t-cloud (trillion-param MoE)
These are not production candidates. They serve as accuracy ceilings
to size the gap between local-best and what 5-250x larger models can
do on the same tasks.
Orchestrator gains two new tier filters:
- `cloud`: returns candidates with cloud:true (5 models)
- `local`: returns candidates without cloud:true (15 models)
Also cherry-picks a single real-format-flavored sample (rc_101) into
the room_classification dataset, sourced from Lumi's hand-written v3
fixture on feat/openclaw-integration. Multi-turn with parallel tool
calls; tests that the harness handles realistic OpenClaw transcript
shape without choking. Source attribution in the sample's 'source'
field for traceability.
Dataset now: 101 room_classification samples (was 100), other tasks
unchanged.
First full matrix run on z690-ex-glacial (Intel i9-12900KF, RTX 3090,
Ollama 0.23.2). 75/75 runs successful, 1 transient warmup timeout on
gemma3:270m entity extraction.
Headline finding: qwen3:4b-instruct-2507-q4_K_M is the best small model
for MemPalace classification across all four tasks (calibration,
closed-set room, entity extraction, memory extraction). q4_K_M holds
within 0.01-0.02 of the fp16 ceiling at half the VRAM.
Open-set discovery is not viable at this model class. Best similarity
score is 0.612 (gemma3:4b-it family), below the 0.70 ship threshold.
Recommendation: keep closed-set classification as the required path,
explore cloud-tier models if the discover feature is still wanted.
Includes:
- 2026-05-10-z690-ex-glacial.csv: raw 75-row CSV (committed as baseline)
- 2026-05-10-z690-ex-glacial.md: auto-rendered tables from summarize.py
- 2026-05-10-analysis.md: human-written interpretation, recommended
MODEL_TIERS update, surprises and follow-ups
Tier list update justified: drop the speculative qwen3.5:4b/qwen3:3b
patterns that never matched on Ollama, demote sub-3B Qwen 3 hybrid
tags due to weak entity F1 (0.31-0.48 vs 0.78 for the 4B instruct).
Recommended new MODEL_TIERS in the analysis report.
MemPalace classification tasks (room, entity, memory) never benefit
from extended reasoning. The thinking variant of qwen3:4b ran ~40-170x
slower than instruct in initial testing without any accuracy gain,
which matches the project principle that classification uses pure
instruct models or hybrid models with thinking disabled.
Two changes:
1. runner.py: forward think=False on every provider.classify call
(sample loop and warmup). Pure-instruct models ignore it; hybrid
Qwen 3 family models stay in fast-classification mode.
2. candidates.yaml: drop qwen3:4b-thinking-2507-q4_K_M. The "test
whether reasoning helps" question is settled; running it again
wastes compute. Replaced the entry with a comment documenting
the policy.
Net candidate count: 15 (Tier 1 = 9). Still covers the full
size/family/quantization grid for production tier-list selection.
Hybrid-reasoning models (Qwen 3 family, DeepSeek-R1 style) accept a
`think: false` request flag in Ollama 0.7+ to suppress reasoning
emission. Pure-instruct models ignore the flag.
Adds an optional `think` parameter to `LLMProvider.classify` and its
three implementations:
- OllamaProvider: forwards to the wire format only when explicitly set,
keeping the request body minimal for the default case.
- OpenAICompatProvider, AnthropicProvider: accept the kwarg for
interface compat but no-op (those providers don't have a per-request
thinking toggle).
Verified against Ollama 0.23.2 with qwen3:1.7b-q4_K_M:
think=False: 294ms, no thinking field
think=True: 523ms (1.8x slower), 516 chars of thinking content
Both return the same answer; only the reasoning overhead differs.
Useful for any caller that wants fast classification without the
<think> token cost — see benchmarks/model_eval for the first user.
summarize.py reads orchestrator output and renders a readable report:
- Production picks: models meeting min accuracy AND max latency thresholds
- Open-set discovery viability: ship/skip recommendation based on
mean cosine similarity threshold
- Instruct vs reasoning comparison for the qwen3:4b pair
- Per-task accuracy rankings with task-specific extras (F1, coverage,
similarity stats)
- Speed table from calibration (smallest task, most stable timing)
- VRAM table with resident/peak/delta per model
Usage:
python -m benchmarks.model_eval.summarize \\
--csv results/2026-05-10-host.csv \\
--output reports/2026-05-10-host.md
`ollama ps --format json` is missing on Ollama 0.23.2 (the --format
flag doesn't exist on older versions). Switch to the HTTP API at
/api/ps which returns clean JSON with size_vram per loaded model.
Verified on RTX 3090 + Ollama 0.23.2: vram_resident_mb now returns
7481 MB for qwen3:4b-instruct-2507-q4_K_M with full 32K context KV
cache loaded.
210 hand-quality samples across four tasks. Synthetic only — no
real-person info. Five fictional personae (Aria, Solas, Fenra,
Bramble, Thresh) with distinct domains, room taxonomies, and
relationship contexts.
- room_classification: 100 samples (20 per agent), 14 marked with
realistic noise features, 18 'other' / 6 'general' / 76 specific-
room labels. 5-10 rooms per sample with 'general' always present.
- entity_extraction: 50 samples, 247 entities (114 person, 74 org,
32 project, 27 place). All entity strings verified to appear in
source text. Includes deliberate same-surname-different-people
cases as disambiguation tests.
- memory_extraction: 40 samples, 55 memories across types
(15 fact, 12 decision, 12 commitment, 9 preference, 7 opinion).
- calibration: 20 samples, 4 per class (question, command, statement,
exclamation, greeting). Sanity check that the harness measures
what we think.
All datasets validated: ID alignment across dataset/labels pairs,
closed-set labels are members of their room lists (or 'other'),
calibration labels are members of their classes lists, no leak
terms (igor, milla, domi, lumi, anthropic, openclaw, etc.) anywhere.
Generated 2026-05-10 via subagent + hand-validation. README in the
datasets directory documents the personae, distribution stats, and
labeling conventions.
Skeleton for benchmarking ≤4B-parameter Ollama models on MemPalace
classification and extraction tasks. Outputs per-(model, task, mode)
metrics: accuracy, latency (TTFT, TPS, e2e p50/p95), VRAM (resident
and peak).
Layout under benchmarks/model_eval/:
- candidates.yaml: 16 models across 3 tiers, with family/size/variant
metadata pulled from ollama.com/library/<family>/tags
- metrics.py: timing extraction from Ollama response, VRAMPoller for
peak-memory tracking via nvidia-smi, embedding-similarity scoring,
host-info introspection, percentile aggregation
- runner.py: runs one (model, task, mode) triple, dispatches per-task
prompt building and scoring
- orchestrator.py: iterates candidates × tasks, writes CSV
- tasks/{calibration,room_classification,entity_extraction,
memory_extraction}/{prompts.py,score.py}: per-task prompt builders
and scorers
Uses mempalace.llm_client.get_provider directly so the benchmark runs
the same code path as production. strip_thinking_tokens lives locally
in metrics.py for now; will switch to mempalace.local_model once that
module lands on develop.
Datasets land in a follow-up commit.
Copilot review on PR #1434 caught that the existing 3.3.5 entry
described the validator as it was authored under #1167 — accepting
``YYYY``/``YYYY-MM``/``YYYY-MM-DD`` and rejecting ISO datetimes — but
PR #1417 (closes#1374) merged into develop on 2026-05-10 and
inverted that: ``sanitize_iso_temporal()`` now rejects partial dates
and accepts canonical UTC datetimes (``YYYY-MM-DDTHH:MM:SSZ`` /
``+00:00``). ``sanitize_iso_date()`` is kept as a backwards-compat
wrapper.
Update the bullet to describe the *shipped* behavior, name both
functions, list both accepted and rejected forms, and call out the
3.3.4 → 3.3.5 behavior change for partial-date inputs that now error.
Reference both #1167 (original) and #1374/#1417 (the expansion).
The 3.3.4 release shipped 2026-05-01 (per GitHub release v3.3.4) but
its CHANGELOG header was never flipped from ``unreleased`` to the
release date. Backfill while we're already touching CHANGELOG for
the 3.3.5 cut.
Bumps version 3.3.4 → 3.3.5 across pyproject.toml, version.py, plugin
manifests, README badge, and uv.lock. Flips CHANGELOG.md from
``[3.3.5] — unreleased`` to ``[3.3.5] — 2026-05-09`` and adds entries
for the four PRs that landed after the bug-fix block was authored:
- Bug Fixes: #1396 (tool_search retry on transient HNSW flush)
- Documentation: #1385 (CONTRIBUTING git-identity guidance, closes#1317)
- Internal: #1431 (test multiprocessing fork → spawn)
- Internal: #1430 (test sqlite connection lifecycle via contextlib.closing)
The four open issues remaining on the v3.3.5 milestone (#1266, #1253,
#1092, #1082) have been moved to v3.4 — they form the concurrent-writer
/ HNSW corruption cluster that needs deeper work than this cycle could
absorb.
Several tests opened sqlite3 connections without try/finally or
context-manager cleanup, relying on a flat conn.close() after the
work. Any assertion failure or exception between connect and close
leaked the connection until GC, producing
``ResourceWarning: unclosed database`` in CI logs.
On Python 3.13 / macOS the ResourceWarning isn't just noise: a
leaked connection can hold a SQLite advisory lock long enough for
later test setup to block on it, which appears to be the cause of
recent intermittent CI hangs on those two runners.
Wrap each affected ``conn = sqlite3.connect(...)`` block in
``contextlib.closing(...)`` so cleanup runs on the failure path too.
Mirrors the try/finally pattern already used in production code
(searcher.py, repair.py, backends/chroma.py).
No behavior change — same operations, same assertions, just
deterministic cleanup. All 162 affected tests pass locally.
test_palace_locks.py and test_chroma_collection_lock.py spawned child
processes with the ``fork`` start method on POSIX. Under Python 3.13
this deadlocks reliably enough to hang the Linux 3.13 and macOS CI jobs
indefinitely while Linux 3.9 / 3.11 / Windows complete normally.
Root cause: by the time these tests run, the pytest parent process is
multi-threaded — chromadb and onnxruntime both spawn background threads
on import. ``fork`` snapshots the parent's address space into the
child without those threads, so any lock another thread held at fork
time stays locked in the child forever. Python 3.13 widened the window
where Python's own internal threads can be holding locks (hence the new
DeprecationWarning that fired ten times in our local 3.13 run).
macOS hits a related but distinct issue: Apple's CoreFoundation
explicitly forbids fork-without-exec; once anything in the parent has
loaded a CF-using library (ONNX, anything via Objective-C bridges) a
forked child will silently hang the moment it touches the same
library.
Switching to ``spawn`` re-imports modules in the child (~0.5s overhead
per Process — measurable but bounded), which is the standard fix for
both classes of bug. Lock-file semantics are unchanged: ``spawn``
inherits ``os.environ`` (including monkeypatched ``HOME``), which is
all these tests need from the parent.
Locally on Python 3.13: all 14 lock tests pass in 6.58s.
Windows runs treated `/tmp/elsewhere/x.md` as relative because Windows
absolute paths require a drive letter, so `_classify_drawer` routed
`drawer_out_of_scope` to `no_source` instead of `out_of_scope` and
`test_dry_run_classifies_correctly` failed on test-windows.
`Path(tmp_dir) / "elsewhere" / "x.md"` is absolute on every platform
and still lives outside the project root that the synced_world fixture
exposes via `repo_path`, so the bucket assertions hold cross-platform.
CI fix: `_classify_drawer` now resolves `source_file` symmetric to
`project_roots` (which `_normalize_project_dirs` and
`_auto_detect_project_roots` already `.resolve()`). Without this, on
platforms where the temp directory is a symlink (macOS `/var/folders` ->
`/private/var/folders`, Windows 8.3 short-name normalization), every
drawer mis-bucketed as `out_of_scope` and survived prune.
Perf:
- `_resolve_project_root`: early-return on first match (sorted-desc
precondition).
- `_normalize_project_dirs`: sort `(-len(str(p)), str(p))` desc for
early-return + deterministic tie-break on equal-length paths.
- `_auto_detect_project_roots`: `seen_sources` dedupe so a 200-chunk
file costs one disk walk, not 200.
- `sync_palace` main loop: per-file classification cache; registry
sentinels (`_reg_*`, `room=_registry`, `ingest_mode=registry`) routed
to "kept" before cache lookup so a sentinel sharing a `source_file`
with a pruned drawer cannot inherit a stale "gitignored" verdict.
- Closet purge: collapse O(N) per-file purge into one
`where={"source_file": {"$in": [...]}}` get + one bulk delete.
Tests (5 new in `TestSyncPalace`, 38 total):
- `test_symlinked_project_root_resolves`: pins symmetric resolve via
real `os.symlink` (skipped on Windows).
- `test_classification_cache_avoids_redundant_disk_hits`: monkeypatch
counter on `_classify_drawer` asserts `call_count == 1` for 5 chunks
sharing one source_file.
- `test_closet_batch_purge_single_call`: wraps closets collection with
`CallCountingCol` (forwards `.get`/`.delete`); asserts
`delete_calls == 1` and `get_calls == 1`; expected `removed_closets`
derived from `report["by_source"]` to stay robust to fixture changes.
- `test_registry_check_runs_before_cache_lookup`: a regular drawer
caches "gitignored" first; a sentinel with the same source_file must
still be kept.
- `test_normalize_project_dirs_sort_stable_on_equal_length`: pins the
alphabetical secondary key when paths share length.
Add `mempalace sync` CLI command and `mempalace_sync` MCP tool that
prune drawers whose source files are gitignored, deleted, or moved
out of the project. Reuses the existing GitignoreMatcher
infrastructure in mempalace/miner.py so the same gitignore rules
that block ingest also drive the corresponding cleanup.
Closes#1252.