Files
mempalace/tests
mvalentsevandDavid Glidden 5decc0c83b fix(miner): configurable + raised MAX_CHUNKS_PER_FILE (#1455)
Default raised from 500 to 50_000 so legitimate long-form content
(novels, scholarly editions) is not silently dropped on a typical
literary corpus. Cap configurable via MEMPALACE_MAX_CHUNKS_PER_FILE
env var or --max-chunks-per-file CLI flag; sentinel 0 disables.
Negative or non-int values from either source emit a stderr warning
and fall back to the default.

Separate counter (files_skipped_chunk_cap) and summary line surface
chunk-cap drops independently of the residual already-filed bucket;
counter fires under --dry-run too so a corpus audit shows the same
signal without writing. Skip notice routed to stderr alongside the
existing symlink-skip line.

ONNX bad_alloc protection preserved architecturally:
DRAWER_UPSERT_BATCH_SIZE=1000 bounds per-ONNX-call exposure
regardless of per-file cap. A 50_000-chunk file produces 50 forward
passes of 1_000 chunks each, identical to what 500-chunk files
already produced.

process_file return extended to a 3-tuple (drawers, room,
skip_reason); skip_reason is None on non-chunk-cap paths and
"chunk_cap" when the per-file cap triggered. Internal callers
(_mine_impl, benchmarks/mine_bench.py, tests) updated.

Closes #1455

Co-Authored-By: David Glidden <d@davidglidden.eu>
2026-05-20 02:20:29 +05:00
..