Four sites passed content to `collection.upsert(documents=[...])` or
`collection.add(documents=[...])` without per-drawer size cap, hitting
the same `RuntimeError: Invalid buffer size` crash class from the
embedding model's attention buffer:
- `general_extractor.extract_memories`: post-classification slicer
that propagates `memory_type` to every slice. New `chunk_size`
parameter defaulting to `DEFAULT_CHUNK_SIZE` and resolved from
`MempalaceConfig.chunk_size` by the caller.
- `diary_ingest.ingest_diaries`: per-entry drawers via existing
`_split_entries`, with character-chunk fallback for any single
entry larger than `chunk_size`. New `_diary_drawer_id_entry`
helper carries (entry_idx, entry_chunk_idx). `chunk_index` in
metadata is a global counter across the file so
`searcher._expand_with_neighbors` stitches sibling chunks
regardless of entry boundary. Upsert is batched atomic per file
(one call carrying every entry/chunk) so a mid-pass embedding
failure cannot half-write the day. Auto-purge on full rebuild
deletes prior-pass drawers via `where={"source_file": ...}`,
which also migrates pre-#1539 legacy `drawer_diary_` IDs as a
side effect of normal use.
- `mcp_server.tool_diary_write`: split oversized entries into
bounded per-chunk drawers via a single batched `col.add` (atomic,
no half-write on embedding failure). `col.add` is intentional:
`entry_id` is timestamp-based with microsecond precision, so a
duplicate is a same-microsecond clash that should surface rather
than silently overwrite.
- `mcp_server.tool_add_drawer`: same crash class on the more common
add-drawer surface (100 KB sanitize cap, 125x `CHUNK_SIZE`).
Chunked path mirrors Site 3 with batched atomic `col.upsert`,
per-chunk `parent_drawer_id` + `chunk_index` metadata, and a
dual-id idempotency probe (last chunk for atomicity, legacy
`drawer_id` for pre-#1539 single-row backwards-compat). Return
shape additive: `chunks` always present, `chunk_ids` on the
chunked path. `tool_get_drawer` / `tool_delete_drawer` against
the logical handle report "not found" on the chunked path;
callers iterate `chunk_ids` or query `parent_drawer_id`.
Chunk id width is `:06d` so even a single-digit `chunk_size`
config cannot lex-sort chunks out of order.
KeilerHirsch
mvalentsev
Tal Muskal