PR #1555 (format coverage + virtual line numbering) merged with twelve
inline polish comments from Copilot + gemini-code-assist that weren't
load-bearing enough to block the original ship but are real cleanups.
This PR addresses them.
Twelve items in scope; one item (drawer ID delimiter — Copilot #13) is
deferred to its own dedicated PR because it's a breaking schema change
that requires migration design beyond the scope of a polish PR.
## Behavioral fixes (5 items, RED-tested first)
1. **FileNotFoundError vs broken symlink (Copilot #8).** ``extract_text``
previously mapped every ``FileNotFoundError`` from ``stat()`` to
``SKIP_BROKEN_SYMLINK``. That's misleading for the common case of a
regular file deleted between scan and extract. Now distinguishes:
``SKIP_BROKEN_SYMLINK`` only when ``p.is_symlink()`` is true;
``SKIP_UNREADABLE`` otherwise.
2. **``file_already_mined`` extract_mode scoping (Copilot #11, #12).**
Both call sites in ``mine_formats`` and ``_file_chunks_locked`` now
pass ``extract_mode="format"``. Previously the format miner could
falsely treat drawers from project / convo miner on the same source
file as "already mined" (and vice versa). Scopes idempotency to the
correct drawer subset.
3. **Sentinel skip for transient missing-dep statuses (Copilot #14).**
New ``_TRANSIENT_MISSING_DEP_STATUSES`` set + ``_register_skip_sentinel_if_appropriate``
helper. Skip variants like ``SKIP_NO_MARKITDOWN`` /
``SKIP_NO_STRIPRTF`` / ``SKIP_MISSING_FORMAT_DEPS`` /
``SKIP_NETWORK_TIMEOUT`` no longer write the "already-mined" sentinel.
Otherwise installing the missing extra later wouldn't trigger a re-mine.
4. **Outer ``except Exception`` in ``mine_formats`` (Gemini #5).** The
outer try around the loop previously caught only ``KeyboardInterrupt``,
leaving any setup-time error (e.g., ``scan_formats`` raising) to
propagate as a bare traceback. Now catches ``Exception`` defensively,
logs it, prints a partial-progress summary, and lets the ``finally``
PID-cleanup run. Mirrors miner.py's belt-and-suspenders pattern.
5. **Thread user's ``chunk_size`` / ``chunk_overlap`` / ``min_chunk_size``
through to ``chunk_text`` (Gemini #3).** ``MempalaceConfig`` was loaded
only to validate readability; users who tuned their config saw no
effect in format-mode mining. Now properly threaded.
## Trivial cleanups (5 items)
6. **Path expanduser in ``extract_text`` (Copilot #7).** ``Path(path)`` →
``Path(path).expanduser()`` so CLI inputs like ``~/docs/file.pdf``
resolve correctly.
7. **Path expanduser+resolve in ``scan_formats`` (Copilot #9).** Same
fix; ``~/docs`` and relative paths now work consistently.
8. **Use resolved ``format_path`` in ``mine_formats`` (Copilot #10).**
``scan_formats(format_dir)`` → ``scan_formats(format_path)`` so the
already-resolved path is used.
9. **``render_with_line_numbers`` type annotation (Copilot #15).**
``text: "str | None"`` reflects the documented + tested ``None``
handling.
10. **Test + docs claims (Copilot #16, #17, #18).** Stale framings
removed:
- ``docs/format-coverage.md`` — 14 fringe cases + "see the file for
the current test inventory" (no more frozen test count).
- ``tests/test_line_numbers.py`` — drops "proposed for mempalace
3.3.6" + "run from the proposal directory" references.
- ``tests/test_format_miner.py`` — drops "MarkItDown is mocked
throughout" (live integration tests exist) + proposal-directory
framing.
## Module-level hoists (enables clean test patching)
- ``MempalaceConfig`` (from ``.config``) hoisted from lazy local import
to module-level so tests can patch ``mempalace.format_miner.MempalaceConfig``.
- ``chunk_text`` (from ``.miner``) hoisted similarly.
Both follow the pattern PR #1565 used for ``compute_hallways_for_wing``.
## Complexity refactor
Extracted ``_print_mine_summary`` from ``mine_formats`` so the orchestrator
stays under the project's ``max-complexity = 25`` ceiling (per
``pyproject.toml [tool.ruff.lint.mccabe]``). Behavior unchanged; pure
extraction.
## Out of scope (intentionally deferred)
- **Drawer ID delimiter collision (Copilot #13)** — ``f"{source_file}{chunk_index}"``
can theoretically collide (``"/path/a1" + "23"`` == ``"/path/a" + "123"``).
Fixing this is a breaking schema change to drawer IDs and requires a
migration plan; will land as its own PR after design.
- The four bot comments that were ALREADY addressed by amendment #3
before the PR #1555 merge (``_SKIP_DIRS`` dedup, ``scan_formats``
symlink skip, ``source_mtime`` tracking, hall+entities metadata) —
no action needed; verified during audit.
## Tests (RED-first)
Six new RED-first tests in ``tests/test_format_miner.py``:
test_extract_text_nonexistent_regular_file_returns_unreadable_not_broken_symlink
test_mine_formats_passes_extract_mode_format_to_file_already_mined
test_mine_formats_does_not_write_sentinel_for_skip_no_markitdown
test_mine_formats_does_not_write_sentinel_for_skip_missing_format_deps
test_mine_formats_catches_unexpected_exception_and_prints_summary
test_mine_formats_threads_chunk_size_from_user_config
All six RED before this commit (failures correctly identified the bugs
they're targeting), all six GREEN after.
One existing test (``test_mine_formats_continues_after_per_file_error``)
updated to patch the new module-level binding
``mempalace.format_miner.chunk_text`` instead of the old
``mempalace.miner.chunk_text`` source location, and to accept the
``**kwargs`` the call now passes through. Behavior unchanged.
## Verification
pytest -q (full mempalace suite)
→ 2065 passed, 3 skipped, 0 regressions
ruff check mempalace/format_miner.py mempalace/searcher.py tests/
→ All checks passed!
ruff format --check ...
→ 4 files already formatted (pinned 0.15.9)
mine_formats complexity
→ ≤ 25 (under the project ceiling)
Two additive features, both following the same read-time-transform pattern:
1. Virtual line numbering — new render_with_line_numbers() and
extract_line_range() in mempalace/searcher.py. Closet pointers like
2026-01-18:L55-L72 resolve to drawer slices rendered with [55] through
[72] line prefixes without modifying any stored content. Lines already
prefixed with [<digits>] pass through unchanged. Pure functions, no I/O.
2. Format coverage (mempalace mine --mode extract) — new mempalace/format_miner.py
reads binary office formats and files drawers via the lock + purge +
upsert pattern convo_miner uses. Source files never modified.
Per-format transformer routing: MarkItDown 0.1.5 does not actually
convert .rtf (returns raw control codes unchanged, verified live), so
.rtf is routed to striprtf which does convert. Other formats stay on
MarkItDown:
.pdf .docx .pptx .xlsx .epub -> MarkItDown
.rtf -> striprtf
13 fringe cases handled with dedicated ExtractionStatus codes:
SKIP_NO_MARKITDOWN, SKIP_NO_STRIPRTF, SKIP_TOO_LARGE, SKIP_CLOUD_ONLY,
SKIP_ENCRYPTED, SKIP_EMPTY, SKIP_PERMISSION, SKIP_BROKEN_SYMLINK,
SKIP_UNRECOGNIZED, SKIP_EXTRACTION_ERROR, SKIP_NETWORK_TIMEOUT,
SKIP_UNREADABLE, plus the encoding fallback handled internally.
Drawers carry ingest_mode=extract + extract_mode=format so they are
distinguishable from project / convo drawers in the palace.
Both transformers are optional extras: pip install mempalace[extract].
MarkItDown requires Python >= 3.10 (env marker ensures it only installs
where supported; Python 3.9 users still get RTF coverage via striprtf).
Tests: 94 new (21 line-numbering + 73 format-miner, including 9
mine_formats orchestrator tests). Full mempalace suite still green
(1992 passed locally). Coverage 86% on mempalace/format_miner.py.
ruff check + ruff format clean on the pinned 0.15.9.
Live verification: mempalace mine --mode extract on a directory with 2
RTFs + 1 PDF produced 90 drawers correctly; mempalace search found
content from both file types in the resulting wing.
Documented limits (out of scope for 3.3.6): custom PDF parsers, OCR on
scanned PDFs, DRM-locked files, pathological corrupt files. These get
reported via skip codes and skipped.
Refs: docs/format-coverage.md, docs/virtual-line-numbering.md
Documents the recovery procedure for the chromadb index-metadata corruption
shape filed at chroma-core/chroma#6949 and reproduced by mempalace's
rebuild_index code path (#1492).
Symptom: mempalace integrity gate quarantines a segment dir with
"labels present but dimensionality is missing or invalid (None)" at
startup, vector search drops to BM25-only fallback, recall gap appears.
Recovery: patch the dimensionality field back into the index metadata
file (the rest of the segment state is intact). ~90 seconds end-to-end
on a 183k-drawer palace; restored 99.97% of recall.
The "delete the metadata file entirely" workaround from chroma-core/chroma#6949
loses the id_to_label and label_to_id mappings; the patch approach
documented here preserves them.
Companion content:
- docs/recovery/index-metadata-recovery.md (this file)
- Related issues #1492 (producer-side fix) and #1493 (auto-recover
proposal for the integrity gate)
- External: jphein/palace-daemon docs/recovery/chromadb-metadata-dict-patch.md
has the same procedure from a palace-daemon HTTP operator's
perspective, plus tests/test_chromadb_metadata_recovery.py with a
regression test that builds a real palace + corrupts + recovers.
Two fixes from Copilot's 2026-04-23 inline review:
1. Drop `-n` from the grep command. Hard-coded line numbers in the
"Expected" block would drift as files evolve, making the
checklist misleading. The check is about presence, not location —
line numbers add noise without helping pass/fail.
2. Reword "`console_script` entry point declared in pyproject.toml"
→ "console script declared under `[project.scripts]` in
pyproject.toml". PEP 621's `[project.scripts]` is the canonical
name for this repo's config form; the old wording conflated it
with setuptools' `console_scripts` entry-point group name.
Expected output block updated to match new grep (no colons before
line numbers).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fulfills the "Optional: release-checklist addition" proposal at the
bottom of #1093 (the v3.3.2 release defect where plugin.json referenced
a mempalace-mcp binary that pyproject.toml never declared, so fresh
`pip install` was broken for everyone until messelink's #340 was
re-cut as v3.3.3).
New file at docs/RELEASING.md (no existing doc at that path) with a
single pre-release grep:
grep -rn mempalace-mcp pyproject.toml .claude-plugin .codex-plugin
The original #1093 proposal specified `pyproject.toml
.claude-plugin/plugin.json` (2 files). This expands via -rn directory
recursion to also cover `.claude-plugin/.mcp.json` and
`.codex-plugin/plugin.json`, which reference `mempalace-mcp` by name
too — same class of regression through a different surface. Happy to
trim to the narrower 2-file form if preferred; one-line edit.
Shows the concrete expected output so a maintainer running this under
release pressure can eyeball "pass" without mental translation, and
points at #340 as the historical fix anchor so "investigate why the
entry is missing" has a diagnostic starting point rather than a dead
end.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Draft plugin specification for source adapters, mirroring RFC 001's
role for storage backends. Formalizes the contract six community
ingester PRs (#274, #23, #169, #232, #567, #98, #702) plus #981's
metadata-only mode have been reinventing ad-hoc, so adapter authors
can build to a stable surface.
Key decisions:
- Single ingest() method; lazy adapters yield SourceItemMetadata
ahead of drawers, eager adapters interleave
- Declared-transformation model (§1.4) replaces informal verbatim
promise with a verifiable one; byte_preserving adapters declare
the empty set, declared_lossy adapters enumerate. Existing
miner.py and the convo_miner+normalize pipeline map cleanly
- Palace is the incremental cursor via is_current(item, metadata);
no sidecar persistence
- Routing is adapter-owned; detect_room/detect_hall move into the
filesystem adapter
- Flat metadata per ChromaDB (RFC 001 §1.4) — entity hints as
json_string field, KG triples route to SQLite knowledge graph
- Closets stay core-built as a post-step; adapters may emit flat
closet_hints. Closes existing gap where convo drawers get no
closets
- No per-drawer field renames: source_file, filed_at, source_mtime,
added_by, normalize_version, entities, ingest_mode all preserved.
Spec adds adapter_name, adapter_version, privacy_class
§9 enumerates the cleanup PR prerequisites (mempalace/sources/
module, PalaceContext facade, KnowledgeGraph.add_triple gaining
backwards-compatible source_drawer_id + adapter_name params).
Tracking issue: #989
Addresses #875. The previous README was 755 lines mixing six purposes
(scam alert, hero, two mea-culpa notes, install guide, architecture
explainer, API reference, file map). Rework it as a pure entry point:
what MemPalace is, how to install, honest benchmark numbers, links to
the website for concept/architecture documentation.
Key content changes:
- Drop the "highest-scoring AI memory system ever benchmarked" framing.
- New tagline: "Local-first AI memory. Verbatim storage, pluggable
backend, 96.6% R@5 raw on LongMemEval — zero API calls." Avoids
naming a specific vector-store implementation since the backend is
pluggable (see mempalace/backends/base.py).
- Remove the cross-system comparison table. Retrieval recall (R@5)
and end-to-end QA accuracy are different metrics and are not
comparable; placing MemPalace's R@5 next to competitor QA accuracy
under a single column header was a category error.
- The "100%" LongMemEval headline is no longer the lead. The honest
held-out figure is 98.4% R@5 on 450 unseen questions. The rerank
pipeline reaches >=99% with any capable LLM (reproduced with
Claude Haiku, Sonnet, and minimax-m2.7 via Ollama) — pipeline-level,
not model-specific.
- Benchmark reproduction commands now reference the correct repo
(MemPalace/mempalace, not the defunct aya-thekeeper/mempal branch).
New file: docs/HISTORY.md as the canonical home for post-launch
corrections, public notices, and retractions. Contains verbatim:
- 2026-04-14 note on this rewrite (links to #875)
- 2026-04-11 impostor-domain notice (moved from README header)
- 2026-04-07 "A Note from Milla & Ben" (moved from README body)
README keeps a one-line scam-alert callout that links to
docs/HISTORY.md for the full timeline.
Merges develop (#820 version sync, #785 strip_noise + NORMALIZE_VERSION,
#784 file locking) and addresses six concerns surfaced during PR review
of the closet feature:
1. Closet append-on-rebuild bug — upsert_closet_lines used to APPEND to
existing closets (mismatched the doc's "fully replaced" promise). With
NORMALIZE_VERSION rebuilds on develop, this would have stacked stale
v1 topics on top of fresh v2 content forever. Fix:
- Drop the read-and-append branch from upsert_closet_lines (now a pure
numbered-id overwrite).
- Add purge_file_closets(closets_col, source_file) helper that wipes
every closet for a source file by where-filter.
- process_file calls purge_file_closets before upsert on every mine,
mirroring the existing drawer purge.
2. Searcher returned whole-file blobs from the closet path while the
direct path returned chunk-level drawers. Refactored:
- _extract_drawer_ids_from_closet parses the `→drawer_a,drawer_b`
pointers out of closet documents.
- _closet_first_hits hydrates exactly those drawer IDs (chunk-level),
not collection.get(where=source_file) (which returned everything).
- Same hit shape as direct-search path; both now carry matched_via.
3. max_distance was bypassed on the closet path. Now applied per-hit;
when every closet candidate gets filtered, _closet_first_hits returns
None and the caller falls through to direct drawer search.
4. Entity extraction caught sentence-starters like "When", "The",
"After" as proper nouns. Added _ENTITY_STOPLIST (~40 common false
positives + day/month names + role words). Real names like Igor /
Milla still survive — covered by tests.
5. CLOSETS.md drifted from the code (claimed "replaced via upsert" but
code appended; claimed BM25 hybrid that doesn't exist; claimed a
10K char hydration cap that wasn't enforced). Rewritten to describe
what actually ships, with explicit notes on the BM25 / convo-closet
follow-ups.
6. Zero tests for ~250 lines. Added tests/test_closets.py with 17 cases:
- build_closet_lines: pointer shape, header extraction, stoplist
filtering (with regression case for "When/After/The"), real-name
survival, fallback-line guarantee, drawer-ref slicing.
- upsert_closet_lines: pure overwrite semantics (regression for the
append bug), char-limit packing without splitting lines.
- purge_file_closets: scoped to source_file, doesn't touch others.
- End-to-end miner rebuild: re-mining a file with fewer topics fully
purges leftover numbered closets from the larger first run.
- _extract_drawer_ids_from_closet: parsing + dedup edge cases.
- search_memories closet-first: fallback when empty, chunk-level
hits with matched_via, no whole-file glue, max_distance enforced.
Merge resolutions: miner.py imports combined NORMALIZE_VERSION/mine_lock
from develop with the closet helpers from this branch. process_file
auto-merged cleanly (closet block sits inside develop's lock body).
724/724 tests pass. ruff + format clean under CI-pinned 0.4.x.
Cherry-picked the docs portion of 67e4ac6 to accompany the closet
feature. Test coverage for closets is omnibus with tests for entity
metadata and BM25 (see PR targeting those features) and will land
together in a follow-up.
Co-Authored-By: MSL <232237854+milla-jovovich@users.noreply.github.com>
Igor Lins e Silva
Milla J
jp
bensig
bensig