Per Gemini review on #1673: chromadb accepts OneOrMany[Document], so a bare
str document was iterated character-by-character by the list comprehension,
splitting it into per-character documents (the silent corruption this method
exists to prevent). Handle isinstance(str) explicitly; add a regression test.
#1235 sanitised lone UTF-16 surrogates for the MCP write tools, but the bulk
ingest paths (miner, convo_miner, sweeper, diary_ingest) build documents
without routing through sanitize_content() and reach ChromaCollection directly.
A single lone surrogate in document text raises UnicodeEncodeError inside
chromadb and aborts the whole add/upsert batch with a -32000 Internal Error,
silently dropping every other row in the same batch.
Complete the chokepoint: add _sanitize_documents_for_chromadb (mirror of
_sanitize_metadatas_for_chromadb) and apply it in add/upsert/update so the
backend guarantees UTF-8-safe documents regardless of caller. IDs and dedup are
unaffected (IDs are computed upstream); only illegal lone surrogates become
U+FFFD, matching the errors="replace" behaviour used elsewhere.
Push the surrogate-cleaning behaviour from per-call-site _clean() helpers
into sanitize_content, sanitize_kg_value, and sanitize_query so every
caller (existing and future) gets the fix automatically. New MCP tools
no longer need to remember to call a separate helper.
- Add strip_lone_surrogates() in mempalace/config.py as the single
regex-based implementation (one U+FFFD per surrogate, not three).
- Wire it into sanitize_content and sanitize_kg_value.
- Wire it into sanitize_query so embedding lookups can't crash either.
- Drop the _clean() helper from mcp_server.py and the per-site calls;
retain a direct strip_lone_surrogates() for source_file/added_by
metadata which doesn't route through any sanitizer.
- Move the ONNX model-cache patch out of the test module and into
conftest.py so it's session-scoped instead of duplicated locally.
- Update tests to assert one U+FFFD per surrogate and exercise the
sanitizer-level entry points directly.
Add tests/test_clean_lone_surrogates.py covering:
Unit tests (TestCleanLoneSurrogates, 11 cases):
- _clean() passes normal ASCII and CJK strings unchanged
- lone surrogates (high/low, single/multiple) are replaced with U+FFFD
- real emoji (\U0001f600 astral code points) pass through unchanged
- empty string, all-surrogate string, SHA-256-hash-after-clean
- the specific \udcad surrogate observed in WorkBuddy production logs
Integration tests (TestLoneSurrogateCleaning, 6 cases):
- tool_add_drawer: surrogate in content and in metadata fields
- tool_check_duplicate: surrogate in query
- tool_search: surrogate in search query
- tool_update_drawer: surrogate in updated content
- tool_diary_write: surrogate in diary entry
Fix test environment issue:
- conftest.py redirects HOME to a temp dir, causing chromadb's
ONNXMiniLM_L6_V2 to look for its ONNX model in the wrong location
and trigger a 79 MB network download on every run.
- Fix: at module import time, recover the real USERPROFILE from
conftest._original_env and patch ONNXMiniLM_L6_V2.DOWNLOAD_PATH
before any ChromaDB collection fixture is invoked.
Mario Gutierrez
Igor Lins e Silva
shushu