Commit Graph
3 Commits
Author SHA1 Message Date
Igor Lins e Silva 1d8a800cca Merge origin/develop into feat/benchmark-multilingual
Resolve conflicts:
- backends/chroma.py: keep both new except handlers in get_collection
  (CollectionNotInitializedError from develop + EF-mismatch helper from
  this branch), ordered _ChromaNotFoundError before ValueError to match
  the create-branch handler order.
- uv.lock: regenerated from merged pyproject.toml.

Fix lint: ruff format mempalace/embedding.py + tests/test_embeddinggemma.py
(CI now pins ruff==0.15.9 via develop's workflow update).

Full suite: 1923 passed, 1 skipped.
2026-05-18 17:58:05 -03:00
Igor Lins e Silva 8a6537d2fb feat(onboarding): multilingual embedder by default for new installs
Onboarding now asks the user once, on first run, whether to use the
multilingual embedding model. The default answer is yes — defaulting to
English-only made the recall promise effectively unreachable for any
non-English content (cross-lingual cos ~0.35 vs ~0.88 for the multilingual
model). The choice is written to config.json so subsequent runs pick the
right EF without re-prompting; existing installs that never set the env
var or ran onboarding stay on minilm for back-compat. MEMPALACE_EMBEDDING_MODEL
still overrides both.

Multilingual deps (huggingface_hub, tokenizers, numpy) move from the
[multilingual] extra into core. The extra is kept as a no-op alias so
existing install scripts keep working. The 300 MB ONNX model is still
lazy-downloaded on first use, not at install time.

`quick_setup` (the programmatic non-interactive path) grows an optional
`embedding_model` arg so tests and benchmark scripts can pick a model
without writing config.json by accident.

EmbeddinggemmaONNX's "missing deps" error now points at the right
recovery path (reinstall mempalace, since the deps are core) rather
than the obsolete pip install mempalace[multilingual] hint.

Tests: 9 new (3 _ask_embedding_model variants + 2 run_onboarding
persistence + 2 quick_setup + 2 set_embedding_model round-trips). The
existing 2 run_onboarding tests now patch _ask_embedding_model so they
don't print to stdout.
2026-05-14 06:40:27 -03:00
Igor Lins e Silva cef1c62fe7 feat(embedding): EF-mismatch error helper, offline tests, migration docs
Three follow-ups bundled for the embeddinggemma EF added in 51702e9:

1. Offline tests for EmbeddinggemmaONNX (10 tests, 0.08s, no network).
   Mocks huggingface_hub.hf_hub_download, tokenizers.Tokenizer.from_file,
   and onnxruntime.InferenceSession so CI never pulls the 300 MB model.
   Guarded with pytest.importorskip so the file is skipped when the
   multilingual extra isn't installed. Covers: stable name(), lazy-load
   runs exactly once, output shape (n, 384) after MRL truncation, L2
   normalization, sim prefix applied, dispatch from
   get_embedding_function(model="embeddinggemma"), cache key separates
   models, helpful ImportError when deps missing, env override.

2. Friendlier ChromaDB EF-name-mismatch error. Switching
   MEMPALACE_EMBEDDING_MODEL on an existing palace previously surfaced
   ChromaDB's bare "Embedding function conflict: new: X vs persisted: Y"
   ValueError. Now ChromaBackend.get_collection() wraps that error and
   points users at the two recovery paths: revert the env var, or run
   `mempalace repair rebuild-index --palace <path>`. New
   _explain_ef_mismatch helper + 3 tests (unit + end-to-end).

3. Docs: CHANGELOG [Unreleased] entry covers both the new EF and the
   error wrapper. README Requirements section mentions the multilingual
   extra and points at the embedding.py docstring for the migration note.
2026-05-14 05:00:48 -03:00