Commit Graph
1588 Commits
Author SHA1 Message Date
mvalentsev 4dc3ccf26a fix(mcp): thread captured KG path through _get_kg to lock insert/evict keys
_get_kg now accepts an optional canonical_path argument and uses it
verbatim when given. _call_kg passes its top-of-function captured key
through, so the dict insertion that _get_kg performs and the eviction
lookup _call_kg performs always agree on the key -- even if env state
(MEMPALACE_PALACE_PATH rotation) or FS state (symlink remap) mutates
between attempts. Without the pass-through, _get_kg's internal call
to _canonicalize_kg_path(_resolve_kg_path()) could land on a different
key than the captured path, leaving a closed handle in the cache under
a stale key the eviction lookup no longer matches.

Tests:
  test_call_kg_passes_captured_path_through_resolve_drift -- patches
  _resolve_kg_path with a shifting iterator + identity canonicalize,
  pre-seeds the closed handle under the captured key; asserts both
  _get_kg calls receive the captured key and the closed handle is
  evicted by retry.

  test_call_kg_oserror_at_top_propagates_unmasked -- patches
  _canonicalize_kg_path to raise OSError; asserts it propagates without
  conversion (the rationale for capturing at the top before the retry
  loop -- prevents an OSError mid-handler from masking a
  sqlite3.ProgrammingError it can't retry through).

_patch_mcp_server test helper updated to a varargs lambda so existing
tests that patch _get_kg as a no-arg fixture stay compatible with the
new optional-arg signature.
2026-05-15 22:51:14 +05:00
mvalentsev 51eb279348 fix(mcp): capture canonical KG path once at top of _call_kg
Move _canonicalize_kg_path call from inside the except branch to the
top of _call_kg so OSError from realpath (e.g. transient Windows
junction loss) propagates cleanly instead of masking the original
sqlite3.ProgrammingError mid-handler. Also keeps the eviction key
identical to the insertion key if FS state mutates between attempts,
preventing a closed handle from leaking into the cache under a stale
key the lookup no longer matches.
2026-05-15 22:51:14 +05:00
mvalentsev 3ee158ecdd test(mcp): tighten _canonicalize_kg_path normcase wiring assertion
Patch both realpath and normcase with sentinel wrappers and assert the
result composes them as normcase(realpath(p)). Swapping the order would
leave Windows symlinks under the original case, defeating the dedup.
The previous assertion (result == seen[-1].lower()) was tautological --
the fake normcase already returned p.lower(), so any implementation
that lowercased anything would pass.
2026-05-15 22:51:14 +05:00
mvalentsev 6f9fe8055a fix(mcp): canonicalize KG cache key via realpath + normcase (#1372)
Switch _kg_by_path lookups in _get_kg and _call_kg from os.path.abspath
to a new _canonicalize_kg_path helper applying realpath (collapses
symlink aliases) + normcase (Windows drive-letter casing). Two tenants
pointing at /srv/A and /srv/link-to-A no longer open duplicate
sqlite3.Connection handles over the same file.
2026-05-15 22:51:14 +05:00
Igor Lins e Silva bb313961f3 Merge pull request #1519 from MemPalace/fix/1024-convo-min-chunk-validation
fix(convo_miner): validate explicit min_chunk_size instead of raw config reach
2026-05-15 12:23:06 -03:00
Igor Lins e Silva 342cfa629b fix(config): catch OverflowError in coercion; correct "stricter" wording
Addresses review-bot feedback on #1519:

- gemini: JSON `Infinity`/`1e1000` round-trips to float('inf'); int(inf)
  raises OverflowError, which _try_coerce_int did not catch — bad config
  could still crash ingest. Catch it alongside TypeError/ValueError and
  fall back like any other garbage value. Regression test added.
- Copilot: the 30-char convo floor was described as "stricter" than the
  50-char project default. It is the opposite — lower, i.e. more
  permissive (keeps short exchanges the project default would drop).
  Reworded the docstring and comments to state this accurately.
2026-05-15 11:01:48 -03:00
Igor Lins e Silva a10f65e094 Merge pull request #1520 from MemPalace/fix/duplicate-pytest-import-lint
fix(test): remove duplicate pytest import breaking lint on develop
2026-05-15 10:58:12 -03:00
Igor Lins e Silva 78d57ddf64 fix(test): remove duplicate pytest import breaking lint on develop
The #1024 merge added `import pytest` to tests/test_convo_miner_unit.py
while a pytest import already existed from another merged change. The
collision was textually clean so git didn't flag it, but CI ruff
reports F811 (redefinition of unused `pytest`), turning lint red on
develop and every branch cut from it.

Drop the duplicate import. `ruff check .` (0.4.x, CI pin) clean;
test_convo_miner_unit 23 passed.
2026-05-15 10:46:41 -03:00
Igor Lins e Silva bea7cb1646 fix(convo_miner): validate explicit min_chunk_size instead of raw config reach
#1024 routed min_chunk_size for the project miner through the repairing
MempalaceConfig validation, but the convo path read the raw, unvalidated
_file_config["min_chunk_size"]. A garbage/negative/bool config value
would TypeError the convo length gate or ValueError out of
chunk_exchanges and abort the whole convo mine — breaking the PR's own
"a bad config.json key never takes ingest down" guarantee for one path
only.

Extract _try_coerce_int helper (shared with _coerce_config_int) and add
a public MempalaceConfig.min_chunk_size_explicit accessor that returns
the validated value when the user set it, else None. convo_miner uses
the None sentinel to keep its stricter 30-char floor for untuned users
while honoring a valid override, with no raw _file_config reach.

Adds accessor + convo-fallback regression tests.
2026-05-15 10:42:21 -03:00
Igor Lins e Silva 29d686e5d4 Merge pull request #1024 from techempower-org/pr/configurable-chunking
feat: configurable chunk_size, chunk_overlap, min_chunk_size
2026-05-15 10:39:10 -03:00
Igor Lins e Silva 6ba75851df Merge pull request #1470 from mvalentsev/fix/plugin-hooks-bounded-timeout
fix(hooks): add bounded timeout to plugin Stop/PreCompact (#1465)
2026-05-15 10:28:57 -03:00
Igor Lins e Silva b348483e21 Merge pull request #1466 from mvalentsev/fix/mine-log-skipped-symlinks
fix(miner): log skipped symlinks in mine paths (#1462)
2026-05-15 10:08:04 -03:00
Igor Lins e Silva dce538fc0c Merge pull request #1513 from anastasiiaanfimova/fix/mcp-unknown-param-name
fix(mcp): structured -32602 for unknown parameter name (#1512)
2026-05-15 09:58:43 -03:00
Anastasiia AnfimovaandClaude Opus 4.7 51c03a0c5c fix(mcp): structured -32602 for unknown parameter name (#1512)
Unknown kwargs (wrong parameter name, e.g. text= instead of content=)
were silently dropped by the schema-filter, surfacing only indirectly
as a later "Missing required 'X'". Emit -32602 naming the offending
kwarg instead — symmetric with #1500's missing-required path. Gated by
the existing accepts_var_keyword check; wait_for_previous excluded.

Closes #1512.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-15 13:48:29 +02:00
Igor Lins e Silva 9893100ca4 Merge pull request #1439 from mvalentsev/fix/init-strip-leaked-pythonpath
fix: drop leaked PYTHONPATH on package init (#1423)
2026-05-15 08:05:47 -03:00
Igor Lins e Silva 73d0fe31ec Merge pull request #1424 from arnoldwender/fix/wing-truncation-hyphenated-projects
fix(hooks): correct wing extraction for hyphenated project names (#1410)
2026-05-15 07:54:56 -03:00
Igor Lins e Silva fc2d1dad5f Merge pull request #1507 from workblac/fix/1461-ruff-format-test-hnsw-payload
style: ruff format test_hnsw_payload_health.py (#1461 follow-up)
2026-05-14 19:22:14 -03:00
Bruno Leal 6bd164e4b1 style: ruff format test_hnsw_payload_health.py (#1461 follow-up)
Collapse the multi-line assert startswith() added in #1461 into a
single line as required by ruff format. No logic change.
2026-05-14 19:17:26 -03:00
Igor Lins e Silva a1bf6dea35 Merge pull request #1461 from fatkobra/patch-2
Fix: Reject zero-byte HNSW link lists during quarantine (#1457)
2026-05-14 18:57:01 -03:00
Igor Lins e Silva 2b3af2b2fa Merge pull request #1500 from mvalentsev/fix/mcp-typed-error-1351
fix(mcp): structured error for param-shape mismatch (#1351)
2026-05-14 17:25:53 -03:00
Igor Lins e Silva 8a6537d2fb feat(onboarding): multilingual embedder by default for new installs
Onboarding now asks the user once, on first run, whether to use the
multilingual embedding model. The default answer is yes — defaulting to
English-only made the recall promise effectively unreachable for any
non-English content (cross-lingual cos ~0.35 vs ~0.88 for the multilingual
model). The choice is written to config.json so subsequent runs pick the
right EF without re-prompting; existing installs that never set the env
var or ran onboarding stay on minilm for back-compat. MEMPALACE_EMBEDDING_MODEL
still overrides both.

Multilingual deps (huggingface_hub, tokenizers, numpy) move from the
[multilingual] extra into core. The extra is kept as a no-op alias so
existing install scripts keep working. The 300 MB ONNX model is still
lazy-downloaded on first use, not at install time.

`quick_setup` (the programmatic non-interactive path) grows an optional
`embedding_model` arg so tests and benchmark scripts can pick a model
without writing config.json by accident.

EmbeddinggemmaONNX's "missing deps" error now points at the right
recovery path (reinstall mempalace, since the deps are core) rather
than the obsolete pip install mempalace[multilingual] hint.

Tests: 9 new (3 _ask_embedding_model variants + 2 run_onboarding
persistence + 2 quick_setup + 2 set_embedding_model round-trips). The
existing 2 run_onboarding tests now patch _ask_embedding_model so they
don't print to stdout.
2026-05-14 06:40:27 -03:00
Igor Lins e Silva cef1c62fe7 feat(embedding): EF-mismatch error helper, offline tests, migration docs
Three follow-ups bundled for the embeddinggemma EF added in 51702e9:

1. Offline tests for EmbeddinggemmaONNX (10 tests, 0.08s, no network).
   Mocks huggingface_hub.hf_hub_download, tokenizers.Tokenizer.from_file,
   and onnxruntime.InferenceSession so CI never pulls the 300 MB model.
   Guarded with pytest.importorskip so the file is skipped when the
   multilingual extra isn't installed. Covers: stable name(), lazy-load
   runs exactly once, output shape (n, 384) after MRL truncation, L2
   normalization, sim prefix applied, dispatch from
   get_embedding_function(model="embeddinggemma"), cache key separates
   models, helpful ImportError when deps missing, env override.

2. Friendlier ChromaDB EF-name-mismatch error. Switching
   MEMPALACE_EMBEDDING_MODEL on an existing palace previously surfaced
   ChromaDB's bare "Embedding function conflict: new: X vs persisted: Y"
   ValueError. Now ChromaBackend.get_collection() wraps that error and
   points users at the two recovery paths: revert the env var, or run
   `mempalace repair rebuild-index --palace <path>`. New
   _explain_ef_mismatch helper + 3 tests (unit + end-to-end).

3. Docs: CHANGELOG [Unreleased] entry covers both the new EF and the
   error wrapper. README Requirements section mentions the multilingual
   extra and points at the embedding.py docstring for the migration note.
2026-05-14 05:00:48 -03:00
Igor Lins e Silva 51702e9f0f feat(embedding): add embeddinggemma-300m ONNX as opt-in multilingual embedder
MemPalace's default embedder (all-MiniLM-L6-v2) is English-only-trained.
Cross-lingual cosine similarity on parallel-translated text averages 0.35
across DE/FR/HI/IT/KO/RU — vs 0.88 for embeddinggemma-300m ONNX (q8) with
the semantic-similarity prefix. RU is the worst at 0.17, meaning a Russian
memory and its identical English translation embed to nearly orthogonal
vectors. Multilingual users effectively cannot retrieve their own memories.

This commit adds embeddinggemma-300m as an opt-in alternative:

* New EmbeddinggemmaONNX class implementing ChromaDB's EF protocol.
  Lazy-downloads model_quantized.onnx (~300 MB) via huggingface_hub on
  first use; cached under ~/.cache/huggingface/. Applies the sim prefix,
  runs onnxruntime inference, truncates to 384 dims via Matryoshka
  (MRL), L2-normalizes.

* MRL truncation to 384d is intentional: matches MiniLM's vector width
  so collection schemas don't change, and validation showed 384d MRL
  actually outperforms full 768d on these similarity tasks (0.893 vs
  0.881 avg) — known property of MRL training.

* MEMPALACE_EMBEDDING_MODEL env (default "minilm" for back-compat).
  Switching models on an existing palace requires re-embedding —
  ChromaDB rejects reads with a mismatched EF name. Run
  `mempalace repair rebuild-index` after changing the value.

* New optional dep group: pip install mempalace[multilingual]
  Adds huggingface_hub + tokenizers + numpy. Core deps unchanged.

ONNX q8 validated lossless vs the Ollama gguf benchmarked previously
(max delta 0.002 cos across 240 parallel pairs).
2026-05-14 04:41:00 -03:00
Igor Lins e Silva 8591d133da fix(benchmarks): default --num-ctx to 4096 for apples-to-apples comparison
Without an explicit num_ctx, each candidate ran at its Modelfile default
(32k for the Gemma4 variants, larger for qwen3), so VRAM and latency
weren't comparable across families — a 32k-default model pre-allocates
KV cache a 4k-default model doesn't. The flag's own docstring promised
"apples-to-apples" but defaulted to None, defeating the intent.

All current benchmark prompts fit comfortably under 4k tokens
(memory_extraction is the longest at ~500). Users with longer prompts
can still pass --num-ctx <larger>.

Adds a methodology note to the 2026-05-13 multilingual report so its
VRAM/latency numbers aren't conflated with future runs at the new default.
2026-05-14 01:27:53 -03:00
Igor Lins e Silva 6873426fe6 Merge pull request #1503 from workblac/feat/benchmark-multilingual-extended-languages
feat(benchmarks): add DE/FR/HI/IT/KO/RU datasets + --output-dir + translated labels
2026-05-14 00:21:31 -03:00
Bruno Leal 0b7799cfa5 fix(benchmarks): address PR #1503 review — file-handle bug, untranslated samples, KO labels
Addresses the review feedback from igorls, gemini-code-assist, and Copilot.

HIGH:
- orchestrator: --output single-file mode now shares ONE (fh, writer) across
  all languages instead of opening N handles to the same path. The old code
  caused interleaved buffer corruption: first language opened "w", subsequent
  ones opened "a", and writes from independent file offsets could overwrite
  each other. Verified with a multi-language --output smoke test (4 rows
  written, all distinct).
- 19 untranslated/empty samples re-translated:
  - dataset.de.jsonl: cal_017
  - dataset.hi.jsonl entity_extraction: ent_020, ent_025, ent_032, ent_038
  - dataset.hi.jsonl room_classification: rc_017, rc_026, rc_028, rc_040,
    rc_064, rc_089, rc_091
  - dataset.ko.jsonl room_classification: rc_027, rc_067
  - dataset.it.jsonl room_classification: rc_029, rc_030, rc_031, rc_032,
    rc_053 (previously empty strings)
- labels.ko.jsonl: restored all proper nouns to English (Doreth, Saela, Ivora,
  Ren Solanke, Pol Krisat, Pell Halloran, Bramble, Hollowmounts Institute,
  Wendelsea, Bridgewater Community Garden, Wends, Drukar, Aerwyn cycle,
  Jaccard, Mason bee, Markdown). Also fixed mistranslation 유전자 사과
  (genetic apple) → 재래종 사과 (heirloom apple).

MEDIUM:
- runner.py: refactored label-resolution one-liner into 3 readable lines
  and added an info log when falling back to English ground truth, so
  readers don't misread "score collapse" as model failure.

LOW:
- orchestrator: moved `import socket` to module top (PEP 8); removed
  unused `out_path` from the unpacking tuple.
- translate_datasets.py: renamed loop variable `l` → `code` (ruff E741);
  made the _translate_one fallback return path explicit instead of relying
  on for-loop fall-through; added a privacy warning in the docstring
  flagging that the default `kimi-k2.6:cloud` sends prose to a remote
  endpoint and should not be used over real palace data.
- 2026-05-13-multilingual.md: converted analytical paragraphs from
  Portuguese to English to match the existing repo convention.
2026-05-14 00:11:50 -03:00
Bruno Leal 9e0df1c29f feat(benchmarks): add DE/FR/HI/IT/KO/RU datasets + --output-dir + translated labels
Adds 6 new language datasets (German, French, Hindi, Italian, Korean, Russian)
across all 4 benchmark tasks (calibration, entity_extraction, memory_extraction,
room_classification) — 630 samples total, same conventions as the existing
pt-BR/es/zh datasets: inputs translated, labels/ground-truth stay English
except where noted.

Changes:
- 24 new dataset.{de,fr,hi,it,ko,ru}.jsonl files across all 4 tasks
- labels.ko.jsonl for memory_extraction: Korean ground-truth so the scorer
  compares Korean model output against Korean expected content instead of
  English (fixes ~20pp score gap identified during testing — see report)
- runner.py: loads labels.{lang}.jsonl when present, falls back to labels.jsonl
- orchestrator.py: adds --output-dir (writes <dir>/<lang>/YYYY-MM-DD-<host>.csv
  per language); --output single-file mode unchanged
- candidates.yaml: adds community tier (igorls classifier variants, heretic)
  and local tier (gemma4:e4b)
- translate_datasets.py: script used to generate the translations via Ollama;
  included so contributors can extend to new languages without manual work
- reports/2026-05-13-multilingual.md: 210-run benchmark report across
  6 models × 7 languages × 5 tasks on RTX 3080 Laptop 8 GB
2026-05-13 20:27:48 -03:00
mvalentsevandanastasiiaanfimova f28d0e3511 refactor(mcp): extract _internal_tool_error helper (#1351)
Address gemini-code-assist review on PR #1500: dedup the two -32000
return branches in handle_request tools/call. Behavior unchanged;
TestParamShapeDiagnostics 6/6 green.

Co-authored-by: anastasiiaanfimova <271004602+anastasiiaanfimova@users.noreply.github.com>
2026-05-14 01:28:23 +05:00
mvalentsevandanastasiiaanfimova 982adcf09b fix(mcp): structured error for param-shape mismatch (#1351)
Split the bare except in handle_request tools/call: TypeError matching
Python's signature-mismatch shape (gated on handler __qualname__) returns
-32602 with parsed parameter name(s); handler-internal TypeError and
non-TypeError exceptions stay on the existing generic -32000 path so
internals never leak. Adds TestParamShapeDiagnostics with 6 cases.

Closes #1351

Co-authored-by: anastasiiaanfimova <271004602+anastasiiaanfimova@users.noreply.github.com>
2026-05-14 01:03:31 +05:00
mvalentsev cab41c8a75 test(hooks): use hook.get("type") for graceful assert message
A direct ``hook["type"]`` subscript raised ``KeyError`` before the
assert evaluated when the ``"type"`` key was missing, so the
diagnostic message naming the event and the unexpected value was
never shown. ``hook.get("type")`` lets the assert fail gracefully
with the intended feedback.
2026-05-13 22:53:48 +05:00
mvalentsev 706237476c test(hooks): pin single-entry cardinality in plugin config schema
A duplicate hook entry inside `entries[0]["hooks"]` (or a second
entry under `Stop`/`PreCompact`) silently passes the per-hook
bounds check and would double-fire the hook. Pin `len(entries) == 1`
and `len(sub_hooks) == 1` so cardinality drift fails loudly with a
diagnostic message naming the event and the unexpected count.
2026-05-13 22:53:48 +05:00
mvalentsev d17d14d8d9 fix(hooks): add bounded timeout to plugin Stop/PreCompact (#1465)
The `.claude-plugin/hooks/hooks.json` Stop and PreCompact entries
shipped without an explicit `timeout`, so a hung `mempalace hook
run` (reported on Windows under CLI/plugin version drift) blocks
the Claude Code UI for the full 600s `"type": "command"` default.

Set explicit bounds: Stop = 30s (fire-and-forget save path with
synchronous chromadb diary write, detached mine via _spawn_mine),
PreCompact = 90s (60s inner _mine_sync subprocess timeout in
mempalace/hooks_cli.py plus a 30s slack so legitimate mines for
users with MEMPAL_DIR are not truncated mid-flight).

Schema test in tests/test_claude_plugin_hook_config.py pins both
per-event (floor, ceiling) bounds and catches future events added
to the plugin config without a registered bounds entry.
2026-05-13 22:53:48 +05:00
mvalentsev d7d9604887 fix(miner): polish symlink skip diagnostic (#1462)
* Move SKIP log from stdout to sys.stderr, matching the existing
  warning-channel pattern (the "No mempalace.yaml" notice and
  topic-tunnel warnings). Stdout stays clean for "Files: N" /
  "Drawers filed: N" markers that downstream callers parse.

* Show the path relative to the scan root via
  Path.relative_to(...).as_posix() instead of the leaf filename, so
  a nested symlink in a deep subdirectory is unambiguous and the log
  renders with forward slashes on every platform.

* Wrap the print in try/except OSError, mirroring the existing guard
  around filepath.stat() below; a broken pipe or closed stderr no
  longer aborts the scan mid-walk.

* Document the stderr side effect in both scan_convos and
  scan_project docstrings.

* Gate the new tests with @pytest.mark.skipif(sys.platform == "win32",
  ...) because Path.symlink_to requires SeCreateSymbolicLinkPrivilege
  on Windows.

* Add dangling-symlink and nested-subdirectory tests for both miners,
  locking the two-space "  SKIP:" prefix, the "(symlink)" reason
  marker, and the full relative path format.
2026-05-13 22:53:43 +05:00
mvalentsev becf56172a fix(miner): log skipped symlinks in mine paths (#1462)
Both scan_convos() and scan_project() silently drop symlinked files,
leaving callers staging a temp directory of symlinks with no diagnostic
for the "Files: 0" outcome. Keep the existing skip-symlinks behavior
and add a "  SKIP: {name} (symlink)" log per skipped file, matching
the convention already used by split_mega_files.py for the
oversized-file skip case.
2026-05-13 22:53:43 +05:00
mvalentsev 068930b710 test(init): probe sys.path filter in CLI/MCP env strip tests; docstring side effect
Three-way assertions in test_cli and test_mcp_server now also assert
SENTINEL_IN_PATH is False at import, so a regression in the package
sys.path filter is caught by the same test that catches the env pop
regression. SystemExit catch is narrowed to exit codes (0, None) so a
future argparse change surfaces as a failure instead of being swallowed.

cli.py:main and mcp_server.py:main docstrings document the PYTHONPATH
pop side effect for callers that invoke main() programmatically.

Refs #1423.
2026-05-13 22:53:36 +05:00
mvalentsev 4b1ab50750 fix(init): keep PYTHONPATH env for embedded callers, strip only in CLI entry points
Split the leaked-PYTHONPATH workaround: __init__.py now only filters
sys.path (fixes the ABI crash from #1423 at import time), while
os.environ.pop("PYTHONPATH") moves to cli:main() and mcp_server:main()
where the strip affects only subprocesses we spawn ourselves.

Host applications that import mempalace as a library and rely on
PYTHONPATH for their own child subprocesses are no longer affected.

Refs #1423.
2026-05-13 22:53:36 +05:00
mvalentsev d09cd9dab1 fix(init): preserve cwd marker; decouple test (#1423)
When PYTHONPATH is `.`, both the leaked entry and the implicit
empty-string CWD marker on sys.path normalize to `.`. The previous
filter would remove the CWD marker too, silently changing import
resolution. The sys.path filter now preserves any empty-string entry
unconditionally.

Also drop the asymmetric `.strip()` on PYTHONPATH tokens. CPython
does not strip whitespace from PYTHONPATH before populating sys.path,
so stripping on one side only could let whitespace-padded leaks
through.

Test extended with a `dot` parametrize case and a dedicated
test_init_preserves_cwd_marker_when_pythonpath_collides. Test child
code now asserts on the sentinel-prefix substring directly instead of
re-implementing the production normalization, so future changes to
_norm cannot silently mask test regressions.
2026-05-13 22:53:36 +05:00
mvalentsev c4fdc1ed8c fix(init): normalize path comparison for sys.path strip (#1423)
Previous commit compared sys.path entries to PYTHONPATH tokens by
byte-exact string equality, which let leaks slip through on Windows
(case-insensitive paths) and across trailing-separator or whitespace
quirks (a leading or trailing space in PYTHONPATH would not match the
trimmed sys.path entry the interpreter actually populated).

Both sides of the comparison now run through os.path.normcase +
os.path.normpath so case differences and trailing separators no longer
defeat the filter, and PYTHONPATH tokens are trimmed before the
match.

Test extended to 6 parametrized cases (single, multi-path, trailing
separator, leading pathsep, empty, unset) with explicit ids and
stderr included in assertion diagnostics. Synthetic leak placeholder
replaces /tmp so the test is portable across CI runners.
2026-05-13 22:53:36 +05:00
mvalentsev 765478d37a fix(init): also strip leaked sys.path entries (#1423)
Popping PYTHONPATH from os.environ alone is insufficient: the interpreter
populates sys.path from PYTHONPATH at startup, before package init runs.
Transitive imports of compiled extensions (pydantic_core,
chromadb_rust_bindings) would still resolve from the leaked path even
after the env var is cleared.

Now also remove the matching entries from sys.path so the import search
path no longer contains the wrong Python's site-packages.

Test extended with a multi-path PYTHONPATH case and assertions on both
os.environ and sys.path after import.
2026-05-13 22:53:36 +05:00
mvalentsev af7b6dfebc fix: drop leaked PYTHONPATH on package init (#1423)
Venvs do not strip PYTHONPATH from inheritance. On multi-Python systems
where PYTHONPATH points at a different Python's site-packages, transitive
imports of compiled extensions (pydantic_core, chromadb_rust_bindings)
load the wrong ABI and crash with ModuleNotFoundError on
pydantic_core._pydantic_core.

Place the pop in mempalace/__init__.py (instead of cli.py as the reporter
proposed) so all entry points are covered: the mempalace shell command,
python -m mempalace, and import mempalace from a library.

Tests: tests/test_init.py runs the strip in a subprocess, parametrized
over set / unset PYTHONPATH.

Closes #1423.
2026-05-13 22:53:36 +05:00
jp 7dff734756 docs(recovery): runbook for chromadb dimensionality=None metadata corruption
Documents the recovery procedure for the chromadb index-metadata corruption
shape filed at chroma-core/chroma#6949 and reproduced by mempalace's
rebuild_index code path (#1492).

Symptom: mempalace integrity gate quarantines a segment dir with
"labels present but dimensionality is missing or invalid (None)" at
startup, vector search drops to BM25-only fallback, recall gap appears.

Recovery: patch the dimensionality field back into the index metadata
file (the rest of the segment state is intact). ~90 seconds end-to-end
on a 183k-drawer palace; restored 99.97% of recall.

The "delete the metadata file entirely" workaround from chroma-core/chroma#6949
loses the id_to_label and label_to_id mappings; the patch approach
documented here preserves them.

Companion content:
- docs/recovery/index-metadata-recovery.md (this file)
- Related issues #1492 (producer-side fix) and #1493 (auto-recover
  proposal for the integrity gate)
- External: jphein/palace-daemon docs/recovery/chromadb-metadata-dict-patch.md
  has the same procedure from a palace-daemon HTTP operator's
  perspective, plus tests/test_chromadb_metadata_recovery.py with a
  regression test that builds a real palace + corrupts + recovers.
2026-05-13 06:41:59 -07:00
Igor Lins e Silva 475810ed43 Merge pull request #1487 from jphein/upstream/feat-rebuild-index-progress-callback
feat(repair): rebuild_index accepts progress callback; default prints ETA
2026-05-13 03:11:10 -03:00
jp 24e88437c4 style: ruff format repair.py (fix #1487 CI lint) 2026-05-12 20:50:44 -07:00
jp 251c5a0314 feat(repair): rebuild_index accepts progress callback; default prints ETA
Previously rebuild_index() printed "Staged N/M" / "Re-filed N/M" lines
with no time information, leaving operators to do mental math against
wall-clock to estimate completion on a large palace. On a 183k-drawer
rebuild, it's the difference between "this is a 20-min job" and "this
is an 8-hour job" — non-trivial scheduling information that the
function already has and could easily share.

This patch:

1. Adds a `progress: Optional[Callable[[str], None]] = None` parameter
   to `rebuild_index()`. Defaults to a new `_DefaultProgress` class
   when omitted (backward-compatible default behavior — same content
   plus ETA decoration).

2. `_DefaultProgress.__call__` recognizes `Staged N/M` and `Re-filed N/M`
   lines via regex, computes elapsed/rate/ETA, and appends them:

       Staged 5000/182953 drawers... (elapsed 7m, rate 11.3/s, ETA 4h)

   Non-progress lines (e.g. "Backing up chroma.sqlite3...") pass
   through unchanged.

3. The clock + baseline counter reset at the stage→refile transition
   so refile-phase rate isn't muddied by the slower stage phase
   (refile re-embeds the same drawers and may run at different
   throughput).

4. Replaces the in-function `print(...)` status calls with `progress(...)`
   so a custom callable receives the full status stream — not just the
   batch progress lines.

5. Passes `progress=progress` (instead of hardcoded `progress=print`)
   into `_rebuild_collection_via_temp`.

Use cases for a custom `progress` callable:
- Daemon-side capture for HTTP `/repair/status` (this is the motivating
  case — palace-daemon was tailing the output to surface progress to
  operators via the status endpoint; a callback avoids stdout capture
  trickery)
- Test silence: `progress=lambda *_: None`
- Custom formatting: structured logging, JSON event stream, etc.

The default behavior is unchanged for callers that omit the parameter;
they just get a nicer message format.

Closes #1485.
2026-05-12 20:20:35 -07:00
Igor Lins e Silva 2d6c0bf31a Merge pull request #1474 from jphein/upstream/perf-mine-convos-bulk-prefetch
perf(convo_miner): bulk pre-fetch already-mined set instead of N WHERE queries
2026-05-12 21:43:47 -03:00
Igor Lins e Silva 9e1ecf2ee7 Merge pull request #1459 from jphein/fix/repair-empty-metadata
fix(repair): coerce empty metadata to sentinel during rebuild
2026-05-12 21:42:42 -03:00
Igor Lins e Silva e2273fad9c fix(benchmarks): validate --language input + correct --embed-endpoint defaulting
Addresses Copilot + gemini-code-assist review on #1483.

1. Path-traversal guard for --language. The value is interpolated into
   the dataset filename (`dataset.{language}.jsonl`), so unvalidated
   input could escape `task_dir`. Now:
   - regex `^[A-Za-z][A-Za-z0-9]*(?:[_-][A-Za-z0-9]+)?$` accepts en,
     pt-BR, zh-CN, fr_CA, etc. and rejects anything with path separators
     or `..`
   - belt-and-suspenders `Path.resolve().is_relative_to(task_dir)` check
     before opening the file

2. --embed-endpoint now defaults to None and is resolved after parsing:
   uses --endpoint when --llm-provider=ollama (so remote benchmark
   runs score against the same host), else http://localhost:11434.
   Help text now matches behavior. runner.py's CLI was also missing the
   flag entirely — added so single-task runs honor remote endpoints.
2026-05-12 19:19:43 -03:00
Igor Lins e Silva 8e34ffa343 feat(benchmarks): multilingual datasets + parity controls (embed model, num_ctx, language)
Enables shipping decisions for non-English users and fair comparison across
candidates whose Modelfile defaults disagree.

- --language / --languages: load dataset.{lang}.jsonl alongside the base
  dataset.jsonl. CSV gains a language column. Synthesized candidate
  entries let ad-hoc model tags run without editing candidates.yaml.
- --num-ctx: force Ollama options.num_ctx per request, overriding the
  model's Modelfile default. Required for apples-to-apples VRAM/TPS
  (qwen3:4b-q8 defaults to 32k = 9.7 GB resident; at 8k it's 5.6 GB).
- --embed-model: thread the semantic-similarity embedding model through
  scoring. Default flips to embeddinggemma (was nomic-embed-text v1).
  Reason: v1 cosine on EN<->PT-BR same-meaning pairs sits at ~0.607
  (right at the 0.6 match threshold), so any phrasing drift collapses
  to false-negative. embeddinggemma lands ~0.766 with 2.7x the
  signal/noise spread. PT-BR memory_extraction recovered 0.15 -> 0.85
  on the same outputs after the swap.

Datasets: 12 new files (pt-BR/es/zh x 4 tasks, 633 samples). Input text
translated; proper nouns and labels stay English so cross-lingual
scoring against the existing labels.jsonl works without re-translation.
2026-05-12 19:01:02 -03:00
Igor Lins e Silva 35a0633463 Merge pull request #1482 from MemPalace/feat/docs-ga-wiring
feat(docs): wire Google Analytics into the published docs site
2026-05-12 18:53:31 -03:00
Igor Lins e Silva 8cc1c7cd71 feat(docs): wire MEMPALACE_DOCS_GA_ID into deploy and harden gtag snippet
Pass the GA Measurement ID from the GitHub Actions repo variable into the
docs build so the published site at mempalaceofficial.com actually emits
the gtag tags. Also escape the ID via encodeURIComponent / JSON.stringify
so a malformed value can't break the page.
2026-05-12 18:51:47 -03:00