Commit Graph
911 Commits
Author SHA1 Message Date
jpandClaude Opus 4.7 0a1adf9962 test(repair): unit coverage for empty-metadata sanitization in _extract_drawers
Addresses @Copilot's review feedback on #1459. Five tests:

  - test_extract_drawers_preserves_valid_metadata: non-empty dict
    passes through unchanged (regression guard against breaking happy
    path).
  - test_extract_drawers_sanitizes_none_metadata: None entries
    coerce to {"_repaired_empty_meta": True} (the core fix).
  - test_extract_drawers_sanitizes_empty_dict_metadata: empty dict
    {} entries also coerce to the sentinel (chromadb 1.5.x rejects
    both shapes equally).
  - test_extract_drawers_sanitization_preserves_alignment: critical
    invariant — ids[i] / documents[i] / metadatas[i] stay in
    lockstep through the sanitizer; mis-pairing would silently
    corrupt rebuilds.
  - test_extract_drawers_multiple_batches: pagination boundary
    correctness (sanitizer applied per-batch, no drops/duplicates).

Verified passing locally against mempalace fork main + chromadb
1.5.8 in the palace-daemon venv (5 passed, 67 deselected in 1.88s).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 14:17:07 -07:00
jpandClaude Opus 4.7 5d95656b6b fix(backends/chroma): catch-all metadata sanitizer in add/upsert
Belt-and-suspenders on top of the repair.py sanitization in the
previous commit. A 151,478-drawer rebuild test still failed at
~120K with the same `ValueError: Expected metadata to be a
non-empty dict` from chromadb, even with the repair.py sanitizers
in place. Traceback:

    mempalace/backends/chroma.py:add → chromadb Collection.add
    → validate_insert_record_set → validate_metadatas
    → validate_metadata → ValueError

Likely cause: chromadb's `upsert()` internally calls `add()` for
new records, and somewhere between repair.py's batch upsert and
chromadb's final write, the metadatas list gets reprocessed in a
way that re-introduces empty/None entries.

Sanitizing at the chromadb-client chokepoint catches everything:
no caller can leak bad metadata regardless of upstream sanitization
state. Same `{"_repaired_empty_meta": True}` sentinel, searchable
via `where={"_repaired_empty_meta": True}`.

Cost: one list comprehension per write call; negligible vs the
embedding + HNSW work each upsert already does.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 12:10:48 -07:00
jpandClaude Opus 4.7 a26e465894 fix(repair): coerce empty metadata to sentinel in both rebuild paths
Two functions construct the metadatas[] list that gets fed to
chromadb's upsert/add, both vulnerable to the same ValueError:

  ValueError: Expected metadata to be a non-empty dict, got 0
  metadata attributes in add.

chromadb 1.5.x's validate_metadata rejects both `None` and `{}`
entries — see chromadb/api/types.py:validate_metadata (line ~1071).

This commit patches both:

1. `_extract_drawers` (line ~131) — the chromadb-collection-based
   extractor, used when the source palace's collection is openable
   via the chromadb client. Sanitizes None/{} entries in the
   `batch["metadatas"]` list to `{"_repaired_empty_meta": True}`
   before extending `all_metas`.

2. `_rebuild_one_collection` (line ~813) — the SQLite-direct
   extract path used by `rebuild_from_sqlite()`, invoked when the
   source palace can't be opened via chromadb (the recovery path
   for palaces with corrupt HNSW segments). Old code was:

       metas.append(meta if meta else {})

   The trailing `{}` was the bug; chromadb 1.5.x rejects empty
   dicts the same as None. Replaced with the same sentinel.

Why `_repaired_empty_meta: True` as the sentinel:
  - Satisfies chromadb's non-empty-dict requirement
  - Bool-valued (valid chromadb metadata type, trivially serializable)
  - Namespaced + descriptive so an operator can find which drawers
    were coerced via `where={"_repaired_empty_meta": True}` later
  - Idempotent on re-runs (a future repair over a sanitized palace
    sees the sentinel as already-valid)

Verified on a 151,478-drawer production palace that previously
crashed at drawer 120,000 in both extract paths.

Fixes #1458

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 11:53:57 -07:00
Igor Lins e Silva 68319dc0d0 Merge pull request #1450 from workblac/fix/benchmark-embedding-model-preflight
fix(benchmarks): auto-pull embedding model when missing instead of silently scoring 0
2026-05-11 06:37:13 -03:00
Bruno Leal 3fbe930751 fix(benchmarks): auto-pull embedding model when missing instead of silently scoring 0
Tasks that score via semantic similarity (memory_extraction,
room_classification open-set) depend on nomic-embed-text. If the model
was absent, embed_text() returned None silently and every sample scored
0.0 — indistinguishable from a genuinely bad model.

_ensure_embed_model() now probes the endpoint before the warmup step.
On a 404 it pulls the model automatically via `ollama pull`, verifies
the endpoint responds, and only then proceeds. Fixes:

- OLLAMA_HOST is now forwarded to `ollama pull` so non-localhost
  endpoints are targeted correctly
- FileNotFoundError is caught and re-raised as RuntimeError with an
  install hint if ollama is not on PATH
- Diagnostic prints go to stderr, not stdout
- If the pull or re-probe fails, returns a Result with a clear error
  message instead of silent zeros

Restores --candidates local to its documented meaning (all non-cloud
models). Adds --candidates community for third-party GGUF variants.
2026-05-11 06:29:49 -03:00
Igor Lins e Silva fa143d2b06 Merge pull request #1447 from MemPalace/feat/model-eval-harness
Small-model evaluation harness for MemPalace classification tasks
2026-05-11 04:39:19 -03:00
Igor Lins e Silva 42f52c8b71 fix(benchmarks): address copilot-pull-request-reviewer feedback on PR #1447
Ten fixes across code, documentation, and committed baselines:

Code fixes:
1. prompts.py: closed-set prompt said "no punctuation" while room
   slugs contain "/". Replaced with explicit "copy the slug verbatim
   including / and -" wording. Empirical impact on accuracy is within
   reproducibility noise (qwen3:4b q4 was 0.610, retest shows 0.594
   on the now-101-sample dataset — same vs new sample noise).
2. orchestrator.py: orchestrator now opens the CSV once, writes the
   header, and flushes after each row. README's "writes incrementally,
   safe to Ctrl-C" claim is now accurate. Long matrix runs no longer
   lose all progress on a crash.
3. metrics.py: gather_host_info() now strips trailing newline from
   `hostname` command output. Stops embedded newlines from leaking
   into committed CSVs.
4. runner.py: passed `endpoint=endpoint` to vram_resident_mb() so
   non-default Ollama endpoints are queried correctly.
5. summarize.py: CSV reader and report writer now use
   encoding="utf-8" with explicit newline handling for cross-platform
   consistency.

Documentation fixes:
6. entity_extraction/score.py: docstring claimed "case-sensitive on
   type" while implementation lowercases both. Corrected.
7. dataset.jsonl rc_101: removed mempalace-private URL since the PR
   lands on the public repo. Will be linkable after OpenClaw merges.
8. datasets/README.md: sample counts updated (room_classification 100
   → 101, total 210 → 211) to match the cherry-picked real-format
   sample.
9. runner.py / orchestrator.py: docstring example paths corrected
   (tests/fixtures/mempalace_bench → benchmarks/model_eval/datasets).

Baseline regeneration:
10. Stripped embedded \n from `host` column in all five committed
    CSVs (2026-05-10 main local + spotcheck, 2026-05-10 + 2026-05-11
    cloud, 2026-05-11 modern). 165 rows cleaned in place via Python
    csv reader/writer. Each row is now line-oriented for diff and
    grep friendliness without changing any measured values.
2026-05-11 03:46:35 -03:00
Igor Lins e Silva 152fbb6afc docs(benchmarks): explicit invitation for backend + model contributions
Adds a "Contributions welcome" section to the README covering:

1. Adding models we missed. The published candidate list is one
   engineer + one search pass. Particularly call out function-calling-
   tuned variants (Phi-4 mini, Nemotron-mini), research-lab instruct
   models (Hermes, Dolphin), and quantization-aware-trained variants
   as worth-adding categories.

2. Adding other inference backends. mempalace.llm_client.get_provider
   already supports openai-compat, which means LM Studio, llama.cpp
   server, vLLM, unsloth studio, Docker Model Runner, HF TGI all
   plug in with minimal harness changes. Documents the four pieces a
   backend contributor would need to touch:
   - Backend-selection flag in runner/orchestrator
   - Backend-specific timing extraction in metrics.py
   - Backend-specific VRAM probe
   - candidates.yaml backend field

   Notes that accuracy numbers stay comparable across backends since
   the dataset and scoring code are backend-agnostic.

   Also calls out niche runtimes worth cross-comparison: Apple MLX,
   Intel OpenVINO, AMD ROCm, edge-device runtimes.

Existing harness-maintainer notes (cloud structured-outputs gotcha,
memory hallucination metric caveat, cloud reproducibility variance)
moved under a separate heading.
2026-05-11 03:03:08 -03:00
Igor Lins e Silva 7d094f6d1d docs(benchmarks): expand README into a full reproduction guide
The previous README had a minimal quick-start section. Replaced with
a proper end-to-end reproduction guide for contributors who want to
validate the published numbers on their own hardware.

New sections:

- Prerequisites: GPU, Ollama version, Python, disk, optional cloud auth
- Set up the environment
- Pull candidate models (with a bulk-pull Python snippet that reads
  candidates.yaml directly so it stays in sync with future additions)
- Smoke test (~30 sec, confirms harness works)
- Run the matrix (Tier 1 / all local / cloud, with time estimates)
- Render a report
- Compare with the published baseline (what kind of agreement to expect
  on accuracy vs speed vs VRAM)
- Sharing your results (how to attach a CSV to a follow-up PR or
  open an issue if numbers diverge)
- Notes for new contributors to the harness itself (the known cloud
  structured-output limitation, the memory-extraction hallucination
  metric caveat, cloud reproducibility variance)

Tier filter docs now include the `modern` filter added in the
modern-tier expansion. Single-model runs by exact tag are documented.

Goal: someone cloning the repo for the first time should be able to
reproduce a Tier 1 matrix in ~45 min without reading anything else.
2026-05-11 02:58:05 -03:00
Igor Lins e Silva fa90f26170 feat(benchmarks): add modern-tier candidates that initial search missed
Five model families that existed at the time of the first benchmark
pass but weren't surfaced by the initial ollama.com search:

- qwen3.5:4b-q4_K_M
- gemma4:e2b-it-q4_K_M
- gemma4:e4b-it-q4_K_M
- granite4.1:3b-q4_K_M
- ministral-3:3b

Grouped as tier="modern" with a new orchestrator filter so they can
be re-run independently. 25 runs in 11.2 minutes, zero errors.

Three findings reshape the recommendations:

1. gemma4:e4b-it-q4_K_M is the new local leader for room classification.
   Closed-set 0.62 (vs qwen3:4b q4 at 0.61), open-set 0.65 (highest
   score across all measured models including cloud). The cloud
   "ceiling" on open-set was 0.61 — a 4B local model now beats every
   cloud reference up to 1T parameters on this task.

2. ministral-3:3b has 0.99 memory coverage — nearly cloud-tier.
   But weak elsewhere (closed-set 0.49, entity 0.63). Specialist
   not generalist.

3. qwen3.5:4b does NOT beat qwen3:4b on this workload despite the
   version bump. Slightly better entity F1, worse on memory and
   closed-set. Newer != better.

Production-tier update:
- Room classification (OpenClaw use case): gemma4:e4b becomes the new
  recommended default. Costs 2.1x latency and 1.4x VRAM vs qwen3:4b
  for the room-task lift.
- General extraction on tight hardware: qwen3:4b q4_K_M still wins.
- Open-set discovery: softens from "shelved indefinitely" to "retest
  with gemma4:e4b + prompt tuning before declaring dead." Gap to
  0.70 ship threshold is now 0.05, not 0.09.
2026-05-11 02:47:29 -03:00
Igor Lins e Silva 67f6fc6974 fix(benchmarks): address gemini-code-assist review on PR #1447
Four fixes from automated code review:

1. score.py: pre-compute embeddings per content string to cut API
   calls from O(P*T) to O(P+T). Was making 2*P*T HTTP round-trips
   to Ollama embeddings per memory_extraction sample; now P+T.
   Significant speed-up on cloud endpoints where each embed is a
   network call.

2. metrics.py: bump VRAMPoller default interval from 100ms to 500ms.
   nvidia-smi subprocess every 100ms was creating measurable jitter
   on the timing numbers. 500ms still captures peak (inference VRAM
   is steady-state during a request) with much less overhead.

3. orchestrator.py: use argparse.BooleanOptionalAction for
   --continue-on-error so --no-continue-on-error actually disables
   the default. The previous combination of store_true and
   default=True made the flag a permanent True.

4. summarize.py: drop the buggy first attempt at the reasoning
   filter (had operator-precedence issue with `or` and `and`).
   Keep only the clean prefix-based filter that was already on
   the next line shadowing it.
2026-05-11 02:19:57 -03:00
Igor Lins e Silva b7c40cac06 docs(benchmarks): refresh cloud lineup + document structured-outputs gotcha
Replaced kimi-k2:1t-cloud (retired, HTTP 500 across the board) with
kimi-k2.6:cloud. Added deepseek-v4-flash:cloud and deepseek-v4-pro:cloud
as newer-generation references.

Ran the v2 cloud matrix at n=30 on 7 models. All succeeded. Key findings:

- gpt-oss:20b-cloud and qwen3-coder:480b-cloud tie at ~0.90 on closed-set
  room classification. gpt-oss matches a 24x larger coder-specialized
  model on natural-language tasks; coder-specialization isn't a benefit
  for this workload.

- deepseek-v4-flash hits 0.607 on open-set room similarity (best cloud
  result). Local best is 0.612 (gemma3:4b-it). Cloud still doesn't beat
  local on this task; the open-set ceiling is genuinely refuted across
  two cloud generations and the full 4B-1T size range.

- kimi-k2.6:cloud showed only 37% valid_json_rate on memory extraction.
  Per docs.ollama.com/capabilities/structured-outputs, "Ollama's Cloud
  currently does not support structured outputs." The harness's
  format=json flag is silently ignored on cloud. Other cloud models
  happened to emit JSON by default; K2.6 doesn't. Documented as a
  methodological caveat. Follow-up issue: envelope-aware JSON
  extraction fallback for cloud responses.

- gpt-oss:20b showed 6.4-point closed-set drift between v1 and v2
  (0.833 → 0.897). Bigger than local reproducibility noise (±0.7%).
  Likely cloud-side model rotation or server-side non-determinism.
  Cloud findings should be quoted as ranges, not point estimates.

Raw data: results/2026-05-11-cloud-z690-ex-glacial.csv (v2). The v1 CSV
stays in the tree as historical record of the kimi-k2:1t-cloud failure.
2026-05-11 01:56:38 -03:00
Igor Lins e Silva 977568363a docs(benchmarks): add cloud-tier ceiling measurements
Ran four of five cloud candidates at n=30. Kimi K2 1T returned HTTP
500 on every task — retry pending. Four working models:

| Model | room-closed | room-open | entity F1 | mem cov |
|---|---|---|---|---|
| gpt-oss:20b-cloud | 0.833 | 0.555 | 0.755 | 1.000 |
| gpt-oss:120b-cloud | 0.800 | 0.553 | 0.829 | 1.000 |
| qwen3-coder:480b-cloud | 0.900 | 0.587 | 0.804 | 0.967 |
| deepseek-v3.1:671b-cloud | 0.800 | 0.566 | 0.828 | 0.967 |
| local leader (4B q4) | 0.610 | 0.586 | 0.778 | 0.950 |

Two headline conclusions:

1. Closed-set room classification HAS a real ceiling gap. Cloud at
   0.83-0.90 vs local at 0.61. Strongest argument so far for an
   optional --classifier cloud path for users willing to trade
   privacy/cost for accuracy. Default stays local.

2. Open-set discovery ceiling is REFUTED. Best cloud is 0.587 vs
   best local 0.612. Cloud is slightly WORSE. Not a capacity problem;
   it's a task-formulation problem that more compute doesn't fix.
   Recommendation: shelve --mode discover until prompt design or
   two-pass clustering closes the gap to >0.75.

Also: gpt-oss models continue emitting reasoning even with think=false
in the request body (verified via curl). Content field stays clean,
but cloud latency includes reasoning generation we asked it to skip.
Worth an upstream issue against Ollama Cloud.
2026-05-10 21:09:21 -03:00
Igor Lins e Silva bc5c354ec1 Merge pull request #1442 from MemPalace/chore/sync-3.3.5-back-to-develop
chore(release): sync main back to develop after v3.3.5
2026-05-10 20:43:30 -03:00
Igor Lins e Silva 9f8523a035 docs(benchmarks): update analysis with reproducibility + hallucination findings
Three follow-ups since the first pass:

1. Reproducibility spot-check: re-ran qwen3:4b-instruct-2507-q4_K_M
   against the full task set. Accuracy deltas vs original ≤0.7% on
   every metric. Harness is reliable; single-run numbers can be trusted.

2. Memory-extraction hallucination investigation. Hand-inspected 5
   samples and found qwen3:4b's 0.36 "hallucination rate" is a scoring
   artifact, not a model weakness:
   - ~50% are bundled-truth-split-into-atomic predictions
   - ~30% are ground-truth omissions the model correctly caught
   - <5% are genuine hallucinations
   Meanwhile qwen2.5:3b's "0.00 hallucination" is just under-extraction.
   The mean_hallucination_rate metric over-penalizes thorough models.
   qwen3:4b stays the production recommendation. Follow-up: refine
   scoring to use source-text traceability instead of greedy match.

3. Cloud-tier ceiling measurement attempted, blocked on ollama signin
   (interactive OAuth). Five cloud candidates added to candidates.yaml
   for the next pass.

Also: committed the spot-check CSV as a reproducibility baseline.
2026-05-10 20:40:06 -03:00
Igor Lins e Silva 68bab4fdb4 Merge main back into develop after v3.3.5 release 2026-05-10 20:36:34 -03:00
Igor Lins e Silva 4ee1839c28 feat(benchmarks): add cloud-tier candidates + cherry-pick real-format fixture
Adds five Ollama Cloud reference models for ceiling measurement:
- gpt-oss:20b-cloud (lighter reference)
- gpt-oss:120b-cloud
- qwen3-coder:480b-cloud (code-tuned variant)
- deepseek-v3.1:671b-cloud
- kimi-k2:1t-cloud (trillion-param MoE)

These are not production candidates. They serve as accuracy ceilings
to size the gap between local-best and what 5-250x larger models can
do on the same tasks.

Orchestrator gains two new tier filters:
- `cloud`: returns candidates with cloud:true (5 models)
- `local`: returns candidates without cloud:true (15 models)

Also cherry-picks a single real-format-flavored sample (rc_101) into
the room_classification dataset, sourced from Lumi's hand-written v3
fixture on feat/openclaw-integration. Multi-turn with parallel tool
calls; tests that the harness handles realistic OpenClaw transcript
shape without choking. Source attribution in the sample's 'source'
field for traceability.

Dataset now: 101 room_classification samples (was 100), other tasks
unchanged.
2026-05-10 20:33:01 -03:00
Milla J d0163a7bec Merge pull request #1434 from MemPalace/chore/release-3.3.5-prep v3.3.5 2026-05-10 16:25:38 -07:00
Igor Lins e Silva 8f3111f740 docs(benchmarks): commit first matrix run + analysis report
First full matrix run on z690-ex-glacial (Intel i9-12900KF, RTX 3090,
Ollama 0.23.2). 75/75 runs successful, 1 transient warmup timeout on
gemma3:270m entity extraction.

Headline finding: qwen3:4b-instruct-2507-q4_K_M is the best small model
for MemPalace classification across all four tasks (calibration,
closed-set room, entity extraction, memory extraction). q4_K_M holds
within 0.01-0.02 of the fp16 ceiling at half the VRAM.

Open-set discovery is not viable at this model class. Best similarity
score is 0.612 (gemma3:4b-it family), below the 0.70 ship threshold.
Recommendation: keep closed-set classification as the required path,
explore cloud-tier models if the discover feature is still wanted.

Includes:
- 2026-05-10-z690-ex-glacial.csv: raw 75-row CSV (committed as baseline)
- 2026-05-10-z690-ex-glacial.md: auto-rendered tables from summarize.py
- 2026-05-10-analysis.md: human-written interpretation, recommended
  MODEL_TIERS update, surprises and follow-ups

Tier list update justified: drop the speculative qwen3.5:4b/qwen3:3b
patterns that never matched on Ollama, demote sub-3B Qwen 3 hybrid
tags due to weak entity F1 (0.31-0.48 vs 0.78 for the 4B instruct).
Recommended new MODEL_TIERS in the analysis report.
2026-05-10 13:26:51 -03:00
Igor Lins e Silva b90d27013d fix(benchmarks): always pass think=False, drop reasoning candidate
MemPalace classification tasks (room, entity, memory) never benefit
from extended reasoning. The thinking variant of qwen3:4b ran ~40-170x
slower than instruct in initial testing without any accuracy gain,
which matches the project principle that classification uses pure
instruct models or hybrid models with thinking disabled.

Two changes:

1. runner.py: forward think=False on every provider.classify call
   (sample loop and warmup). Pure-instruct models ignore it; hybrid
   Qwen 3 family models stay in fast-classification mode.

2. candidates.yaml: drop qwen3:4b-thinking-2507-q4_K_M. The "test
   whether reasoning helps" question is settled; running it again
   wastes compute. Replaced the entry with a comment documenting
   the policy.

Net candidate count: 15 (Tier 1 = 9). Still covers the full
size/family/quantization grid for production tier-list selection.
2026-05-10 12:22:05 -03:00
Igor Lins e Silva 608d4e6c0a feat(llm_client): add think parameter to classify()
Hybrid-reasoning models (Qwen 3 family, DeepSeek-R1 style) accept a
`think: false` request flag in Ollama 0.7+ to suppress reasoning
emission. Pure-instruct models ignore the flag.

Adds an optional `think` parameter to `LLMProvider.classify` and its
three implementations:
- OllamaProvider: forwards to the wire format only when explicitly set,
  keeping the request body minimal for the default case.
- OpenAICompatProvider, AnthropicProvider: accept the kwarg for
  interface compat but no-op (those providers don't have a per-request
  thinking toggle).

Verified against Ollama 0.23.2 with qwen3:1.7b-q4_K_M:
  think=False: 294ms, no thinking field
  think=True:  523ms (1.8x slower), 516 chars of thinking content
Both return the same answer; only the reasoning overhead differs.

Useful for any caller that wants fast classification without the
<think> token cost — see benchmarks/model_eval for the first user.
2026-05-10 12:21:58 -03:00
Igor Lins e Silva 5f2c554426 feat(benchmarks): add CSV-to-markdown report renderer
summarize.py reads orchestrator output and renders a readable report:
- Production picks: models meeting min accuracy AND max latency thresholds
- Open-set discovery viability: ship/skip recommendation based on
  mean cosine similarity threshold
- Instruct vs reasoning comparison for the qwen3:4b pair
- Per-task accuracy rankings with task-specific extras (F1, coverage,
  similarity stats)
- Speed table from calibration (smallest task, most stable timing)
- VRAM table with resident/peak/delta per model

Usage:
    python -m benchmarks.model_eval.summarize \\
      --csv results/2026-05-10-host.csv \\
      --output reports/2026-05-10-host.md
2026-05-10 11:03:53 -03:00
Igor Lins e Silva 120e240acc fix(benchmarks): use Ollama HTTP /api/ps for vram_resident_mb
`ollama ps --format json` is missing on Ollama 0.23.2 (the --format
flag doesn't exist on older versions). Switch to the HTTP API at
/api/ps which returns clean JSON with size_vram per loaded model.

Verified on RTX 3090 + Ollama 0.23.2: vram_resident_mb now returns
7481 MB for qwen3:4b-instruct-2507-q4_K_M with full 32K context KV
cache loaded.
2026-05-10 11:02:21 -03:00
Igor Lins e Silva 1c1e725b4c feat(benchmarks): add synthetic dataset for model evaluation harness
210 hand-quality samples across four tasks. Synthetic only — no
real-person info. Five fictional personae (Aria, Solas, Fenra,
Bramble, Thresh) with distinct domains, room taxonomies, and
relationship contexts.

- room_classification: 100 samples (20 per agent), 14 marked with
  realistic noise features, 18 'other' / 6 'general' / 76 specific-
  room labels. 5-10 rooms per sample with 'general' always present.
- entity_extraction: 50 samples, 247 entities (114 person, 74 org,
  32 project, 27 place). All entity strings verified to appear in
  source text. Includes deliberate same-surname-different-people
  cases as disambiguation tests.
- memory_extraction: 40 samples, 55 memories across types
  (15 fact, 12 decision, 12 commitment, 9 preference, 7 opinion).
- calibration: 20 samples, 4 per class (question, command, statement,
  exclamation, greeting). Sanity check that the harness measures
  what we think.

All datasets validated: ID alignment across dataset/labels pairs,
closed-set labels are members of their room lists (or 'other'),
calibration labels are members of their classes lists, no leak
terms (igor, milla, domi, lumi, anthropic, openclaw, etc.) anywhere.

Generated 2026-05-10 via subagent + hand-validation. README in the
datasets directory documents the personae, distribution stats, and
labeling conventions.
2026-05-10 09:45:03 -03:00
Igor Lins e Silva c0b23b0d8f feat(benchmarks): add small-model evaluation harness
Skeleton for benchmarking ≤4B-parameter Ollama models on MemPalace
classification and extraction tasks. Outputs per-(model, task, mode)
metrics: accuracy, latency (TTFT, TPS, e2e p50/p95), VRAM (resident
and peak).

Layout under benchmarks/model_eval/:
- candidates.yaml: 16 models across 3 tiers, with family/size/variant
  metadata pulled from ollama.com/library/<family>/tags
- metrics.py: timing extraction from Ollama response, VRAMPoller for
  peak-memory tracking via nvidia-smi, embedding-similarity scoring,
  host-info introspection, percentile aggregation
- runner.py: runs one (model, task, mode) triple, dispatches per-task
  prompt building and scoring
- orchestrator.py: iterates candidates × tasks, writes CSV
- tasks/{calibration,room_classification,entity_extraction,
  memory_extraction}/{prompts.py,score.py}: per-task prompt builders
  and scorers

Uses mempalace.llm_client.get_provider directly so the benchmark runs
the same code path as production. strip_thinking_tokens lives locally
in metrics.py for now; will switch to mempalace.local_model once that
module lands on develop.

Datasets land in a follow-up commit.
2026-05-10 09:44:49 -03:00
Igor Lins e Silva a5ec32561c docs(changelog): correct KG date validator entry for 3.3.5
Copilot review on PR #1434 caught that the existing 3.3.5 entry
described the validator as it was authored under #1167 — accepting
``YYYY``/``YYYY-MM``/``YYYY-MM-DD`` and rejecting ISO datetimes — but
PR #1417 (closes #1374) merged into develop on 2026-05-10 and
inverted that: ``sanitize_iso_temporal()`` now rejects partial dates
and accepts canonical UTC datetimes (``YYYY-MM-DDTHH:MM:SSZ`` /
``+00:00``). ``sanitize_iso_date()`` is kept as a backwards-compat
wrapper.

Update the bullet to describe the *shipped* behavior, name both
functions, list both accepted and rejected forms, and call out the
3.3.4 → 3.3.5 behavior change for partial-date inputs that now error.
Reference both #1167 (original) and #1374/#1417 (the expansion).
2026-05-10 01:35:21 -03:00
Igor Lins e Silva 40524d5b04 Merge develop into main for v3.3.5 release
# Conflicts:
#	.claude-plugin/marketplace.json
#	.claude-plugin/plugin.json
#	.codex-plugin/plugin.json
#	CHANGELOG.md
#	README.md
#	mempalace/version.py
#	pyproject.toml
#	uv.lock
2026-05-10 01:05:36 -03:00
Igor Lins e Silva 1247e175b5 Merge pull request #1432 from MemPalace/chore/release-3.3.5
chore(release): 3.3.5
2026-05-09 22:54:32 -03:00
Igor Lins e Silva 6e9d057a42 docs(changelog): backfill 3.3.4 release date
The 3.3.4 release shipped 2026-05-01 (per GitHub release v3.3.4) but
its CHANGELOG header was never flipped from ``unreleased`` to the
release date. Backfill while we're already touching CHANGELOG for
the 3.3.5 cut.
2026-05-09 21:30:31 -03:00
Igor Lins e Silva 36100f820e Merge pull request #1417 from fatkobra/fix/1374-kg-iso-datetimes
fix(kg): accept ISO datetimes for temporal inputs
2026-05-09 21:27:37 -03:00
Igor Lins e Silva b2bab382f3 Merge remote-tracking branch 'origin/develop' into pr-1417-fix-kg-datetimes
# Conflicts:
#	tests/test_palace_locks.py
2026-05-09 21:12:40 -03:00
Igor Lins e Silva fa9b7e0525 chore(release): 3.3.5
Bumps version 3.3.4 → 3.3.5 across pyproject.toml, version.py, plugin
manifests, README badge, and uv.lock. Flips CHANGELOG.md from
``[3.3.5] — unreleased`` to ``[3.3.5] — 2026-05-09`` and adds entries
for the four PRs that landed after the bug-fix block was authored:

- Bug Fixes: #1396 (tool_search retry on transient HNSW flush)
- Documentation: #1385 (CONTRIBUTING git-identity guidance, closes #1317)
- Internal: #1431 (test multiprocessing fork → spawn)
- Internal: #1430 (test sqlite connection lifecycle via contextlib.closing)

The four open issues remaining on the v3.3.5 milestone (#1266, #1253,
#1092, #1082) have been moved to v3.4 — they form the concurrent-writer
/ HNSW corruption cluster that needs deeper work than this cycle could
absorb.
2026-05-09 21:11:13 -03:00
Igor Lins e Silva b9cec32528 Merge pull request #1430 from MemPalace/chore/test-sqlite-cleanup
chore(tests): wrap sqlite3 connections in contextlib.closing
2026-05-09 20:58:42 -03:00
Igor Lins e Silva df5ca114b2 chore(tests): wrap sqlite3 connections in contextlib.closing
Several tests opened sqlite3 connections without try/finally or
context-manager cleanup, relying on a flat conn.close() after the
work. Any assertion failure or exception between connect and close
leaked the connection until GC, producing
``ResourceWarning: unclosed database`` in CI logs.

On Python 3.13 / macOS the ResourceWarning isn't just noise: a
leaked connection can hold a SQLite advisory lock long enough for
later test setup to block on it, which appears to be the cause of
recent intermittent CI hangs on those two runners.

Wrap each affected ``conn = sqlite3.connect(...)`` block in
``contextlib.closing(...)`` so cleanup runs on the failure path too.
Mirrors the try/finally pattern already used in production code
(searcher.py, repair.py, backends/chroma.py).

No behavior change — same operations, same assertions, just
deterministic cleanup. All 162 affected tests pass locally.
2026-05-09 20:25:47 -03:00
Igor Lins e Silva 7d5e6c4e85 Merge pull request #1431 from MemPalace/fix/test-multiprocessing-spawn
fix(tests): use spawn instead of fork for lock-test subprocesses
2026-05-09 20:25:26 -03:00
Igor Lins e Silva 11d0a64202 fix(tests): use spawn instead of fork for lock-test subprocesses
test_palace_locks.py and test_chroma_collection_lock.py spawned child
processes with the ``fork`` start method on POSIX. Under Python 3.13
this deadlocks reliably enough to hang the Linux 3.13 and macOS CI jobs
indefinitely while Linux 3.9 / 3.11 / Windows complete normally.

Root cause: by the time these tests run, the pytest parent process is
multi-threaded — chromadb and onnxruntime both spawn background threads
on import. ``fork`` snapshots the parent's address space into the
child without those threads, so any lock another thread held at fork
time stays locked in the child forever. Python 3.13 widened the window
where Python's own internal threads can be holding locks (hence the new
DeprecationWarning that fired ten times in our local 3.13 run).

macOS hits a related but distinct issue: Apple's CoreFoundation
explicitly forbids fork-without-exec; once anything in the parent has
loaded a CF-using library (ONNX, anything via Objective-C bridges) a
forked child will silently hang the moment it touches the same
library.

Switching to ``spawn`` re-imports modules in the child (~0.5s overhead
per Process — measurable but bounded), which is the standard fix for
both classes of bug. Lock-file semantics are unchanged: ``spawn``
inherits ``os.environ`` (including monkeypatched ``HOME``), which is
all these tests need from the parent.

Locally on Python 3.13: all 14 lock tests pass in 6.58s.
2026-05-09 20:11:08 -03:00
Igor Lins e Silva d9e60d84ad Merge pull request #1396 from JPdeB61/fix/1315-mcp-search-retry-on-hnsw-flush
fix(mcp): retry tool_search once on Chroma "Error finding   id" transient (#1315)
2026-05-09 19:32:20 -03:00
Igor Lins e Silva b2ce45d301 style: ruff format tests/test_mcp_server.py for ruff <0.5 2026-05-09 18:09:51 -03:00
fatkobra 03f9a73d44 fix(tests): format palace lock test 2026-05-09 12:16:26 +02:00
fatkobra 710fa529c0 style: format test_palace_locks as part of fix(tests): isolate palace lock subprocess state 2026-05-09 11:45:51 +02:00
fatkobra 00041ca736 Update test_palace_locks.py as part of fix(tests): isolate palace lock subprocess state 2026-05-09 11:30:45 +02:00
fatkobra f34b78c8b1 Update conftest.py as part of fix(tests): isolate palace lock subprocess state 2026-05-09 11:20:33 +02:00
Igor Lins e Silva 2fc47a52fc Merge pull request #1004 from coogie/coogie/fix/miner-routing
fix(miner): use token-boundary matching in detect_room
2026-05-09 01:36:51 -03:00
Igor Lins e Silva 1c26ecef0a Merge pull request #1385 from sjhddh/docs/contributor-git-identity-1317
docs: clarify contributor git identity setup
2026-05-09 01:31:05 -03:00
Igor Lins e Silva 5453ad701a Merge pull request #1421 from mvalentsev/feat/sync-gitignore-prune
feat(sync): add gitignore-aware drawer prune (#1252)
2026-05-09 00:12:29 -03:00
mvalentsev 1822756afe fix(test_sync): use tmp_dir for elsewhere path so it stays absolute on Windows
Windows runs treated `/tmp/elsewhere/x.md` as relative because Windows
absolute paths require a drive letter, so `_classify_drawer` routed
`drawer_out_of_scope` to `no_source` instead of `out_of_scope` and
`test_dry_run_classifies_correctly` failed on test-windows.

`Path(tmp_dir) / "elsewhere" / "x.md"` is absolute on every platform
and still lives outside the project root that the synced_world fixture
exposes via `repo_path`, so the bucket assertions hold cross-platform.
2026-05-09 04:12:18 +05:00
mvalentsev 18f877869b style(test_sync): match CI ruff 0.4.x format 2026-05-09 04:04:56 +05:00
mvalentsev 0ff4121404 fix(sync): symmetric source_file resolve + perf optimizations
CI fix: `_classify_drawer` now resolves `source_file` symmetric to
`project_roots` (which `_normalize_project_dirs` and
`_auto_detect_project_roots` already `.resolve()`). Without this, on
platforms where the temp directory is a symlink (macOS `/var/folders` ->
`/private/var/folders`, Windows 8.3 short-name normalization), every
drawer mis-bucketed as `out_of_scope` and survived prune.

Perf:
- `_resolve_project_root`: early-return on first match (sorted-desc
  precondition).
- `_normalize_project_dirs`: sort `(-len(str(p)), str(p))` desc for
  early-return + deterministic tie-break on equal-length paths.
- `_auto_detect_project_roots`: `seen_sources` dedupe so a 200-chunk
  file costs one disk walk, not 200.
- `sync_palace` main loop: per-file classification cache; registry
  sentinels (`_reg_*`, `room=_registry`, `ingest_mode=registry`) routed
  to "kept" before cache lookup so a sentinel sharing a `source_file`
  with a pruned drawer cannot inherit a stale "gitignored" verdict.
- Closet purge: collapse O(N) per-file purge into one
  `where={"source_file": {"$in": [...]}}` get + one bulk delete.

Tests (5 new in `TestSyncPalace`, 38 total):
- `test_symlinked_project_root_resolves`: pins symmetric resolve via
  real `os.symlink` (skipped on Windows).
- `test_classification_cache_avoids_redundant_disk_hits`: monkeypatch
  counter on `_classify_drawer` asserts `call_count == 1` for 5 chunks
  sharing one source_file.
- `test_closet_batch_purge_single_call`: wraps closets collection with
  `CallCountingCol` (forwards `.get`/`.delete`); asserts
  `delete_calls == 1` and `get_calls == 1`; expected `removed_closets`
  derived from `report["by_source"]` to stay robust to fixture changes.
- `test_registry_check_runs_before_cache_lookup`: a regular drawer
  caches "gitignored" first; a sentinel with the same source_file must
  still be kept.
- `test_normalize_project_dirs_sort_stable_on_equal_length`: pins the
  alphabetical secondary key when paths share length.
2026-05-09 04:01:34 +05:00
mvalentsev 1d3eecbf9d feat(sync): add gitignore-aware drawer prune (#1252)
Add `mempalace sync` CLI command and `mempalace_sync` MCP tool that
prune drawers whose source files are gitignored, deleted, or moved
out of the project. Reuses the existing GitignoreMatcher
infrastructure in mempalace/miner.py so the same gitignore rules
that block ingest also drive the corresponding cleanup.

Closes #1252.
2026-05-09 03:16:03 +05:00
Igor Lins e Silva 02dd6dc19e Merge pull request #1292 from MemPalace/dependabot/github_actions/actions/configure-pages-6
chore(deps): bump actions/configure-pages from 5 to 6
2026-05-08 16:15:41 -03:00