PR #1555 (format coverage + virtual line numbering) merged with twelve inline polish comments from Copilot + gemini-code-assist that weren't load-bearing enough to block the original ship but are real cleanups. This PR addresses them. Twelve items in scope; one item (drawer ID delimiter — Copilot #13) is deferred to its own dedicated PR because it's a breaking schema change that requires migration design beyond the scope of a polish PR. ## Behavioral fixes (5 items, RED-tested first) 1. **FileNotFoundError vs broken symlink (Copilot #8).** ``extract_text`` previously mapped every ``FileNotFoundError`` from ``stat()`` to ``SKIP_BROKEN_SYMLINK``. That's misleading for the common case of a regular file deleted between scan and extract. Now distinguishes: ``SKIP_BROKEN_SYMLINK`` only when ``p.is_symlink()`` is true; ``SKIP_UNREADABLE`` otherwise. 2. **``file_already_mined`` extract_mode scoping (Copilot #11, #12).** Both call sites in ``mine_formats`` and ``_file_chunks_locked`` now pass ``extract_mode="format"``. Previously the format miner could falsely treat drawers from project / convo miner on the same source file as "already mined" (and vice versa). Scopes idempotency to the correct drawer subset. 3. **Sentinel skip for transient missing-dep statuses (Copilot #14).** New ``_TRANSIENT_MISSING_DEP_STATUSES`` set + ``_register_skip_sentinel_if_appropriate`` helper. Skip variants like ``SKIP_NO_MARKITDOWN`` / ``SKIP_NO_STRIPRTF`` / ``SKIP_MISSING_FORMAT_DEPS`` / ``SKIP_NETWORK_TIMEOUT`` no longer write the "already-mined" sentinel. Otherwise installing the missing extra later wouldn't trigger a re-mine. 4. **Outer ``except Exception`` in ``mine_formats`` (Gemini #5).** The outer try around the loop previously caught only ``KeyboardInterrupt``, leaving any setup-time error (e.g., ``scan_formats`` raising) to propagate as a bare traceback. Now catches ``Exception`` defensively, logs it, prints a partial-progress summary, and lets the ``finally`` PID-cleanup run. Mirrors miner.py's belt-and-suspenders pattern. 5. **Thread user's ``chunk_size`` / ``chunk_overlap`` / ``min_chunk_size`` through to ``chunk_text`` (Gemini #3).** ``MempalaceConfig`` was loaded only to validate readability; users who tuned their config saw no effect in format-mode mining. Now properly threaded. ## Trivial cleanups (5 items) 6. **Path expanduser in ``extract_text`` (Copilot #7).** ``Path(path)`` → ``Path(path).expanduser()`` so CLI inputs like ``~/docs/file.pdf`` resolve correctly. 7. **Path expanduser+resolve in ``scan_formats`` (Copilot #9).** Same fix; ``~/docs`` and relative paths now work consistently. 8. **Use resolved ``format_path`` in ``mine_formats`` (Copilot #10).** ``scan_formats(format_dir)`` → ``scan_formats(format_path)`` so the already-resolved path is used. 9. **``render_with_line_numbers`` type annotation (Copilot #15).** ``text: "str | None"`` reflects the documented + tested ``None`` handling. 10. **Test + docs claims (Copilot #16, #17, #18).** Stale framings removed: - ``docs/format-coverage.md`` — 14 fringe cases + "see the file for the current test inventory" (no more frozen test count). - ``tests/test_line_numbers.py`` — drops "proposed for mempalace 3.3.6" + "run from the proposal directory" references. - ``tests/test_format_miner.py`` — drops "MarkItDown is mocked throughout" (live integration tests exist) + proposal-directory framing. ## Module-level hoists (enables clean test patching) - ``MempalaceConfig`` (from ``.config``) hoisted from lazy local import to module-level so tests can patch ``mempalace.format_miner.MempalaceConfig``. - ``chunk_text`` (from ``.miner``) hoisted similarly. Both follow the pattern PR #1565 used for ``compute_hallways_for_wing``. ## Complexity refactor Extracted ``_print_mine_summary`` from ``mine_formats`` so the orchestrator stays under the project's ``max-complexity = 25`` ceiling (per ``pyproject.toml [tool.ruff.lint.mccabe]``). Behavior unchanged; pure extraction. ## Out of scope (intentionally deferred) - **Drawer ID delimiter collision (Copilot #13)** — ``f"{source_file}{chunk_index}"`` can theoretically collide (``"/path/a1" + "23"`` == ``"/path/a" + "123"``). Fixing this is a breaking schema change to drawer IDs and requires a migration plan; will land as its own PR after design. - The four bot comments that were ALREADY addressed by amendment #3 before the PR #1555 merge (``_SKIP_DIRS`` dedup, ``scan_formats`` symlink skip, ``source_mtime`` tracking, hall+entities metadata) — no action needed; verified during audit. ## Tests (RED-first) Six new RED-first tests in ``tests/test_format_miner.py``: test_extract_text_nonexistent_regular_file_returns_unreadable_not_broken_symlink test_mine_formats_passes_extract_mode_format_to_file_already_mined test_mine_formats_does_not_write_sentinel_for_skip_no_markitdown test_mine_formats_does_not_write_sentinel_for_skip_missing_format_deps test_mine_formats_catches_unexpected_exception_and_prints_summary test_mine_formats_threads_chunk_size_from_user_config All six RED before this commit (failures correctly identified the bugs they're targeting), all six GREEN after. One existing test (``test_mine_formats_continues_after_per_file_error``) updated to patch the new module-level binding ``mempalace.format_miner.chunk_text`` instead of the old ``mempalace.miner.chunk_text`` source location, and to accept the ``**kwargs`` the call now passes through. Behavior unchanged. ## Verification pytest -q (full mempalace suite) → 2065 passed, 3 skipped, 0 regressions ruff check mempalace/format_miner.py mempalace/searcher.py tests/ → All checks passed! ruff format --check ... → 4 files already formatted (pinned 0.15.9) mine_formats complexity → ≤ 25 (under the project ceiling)
159 lines
5.6 KiB
Python
159 lines
5.6 KiB
Python
"""Tests for virtual line numbering (mempalace 3.3.6, integrated in PR #1555).
|
|
|
|
Run with:
|
|
pytest tests/test_line_numbers.py -v
|
|
"""
|
|
|
|
from mempalace.searcher import ( # noqa: E402
|
|
extract_line_range,
|
|
render_with_line_numbers,
|
|
)
|
|
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# render_with_line_numbers
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
|
|
|
|
def test_render_empty_string():
|
|
assert render_with_line_numbers("") == ""
|
|
|
|
|
|
def test_render_single_line():
|
|
assert render_with_line_numbers("hello") == "[1] hello"
|
|
|
|
|
|
def test_render_multi_line():
|
|
text = "alpha\nbeta\ngamma"
|
|
expected = "[1] alpha\n[2] beta\n[3] gamma"
|
|
assert render_with_line_numbers(text) == expected
|
|
|
|
|
|
def test_render_custom_start_line():
|
|
text = "first\nsecond"
|
|
expected = "[5] first\n[6] second"
|
|
assert render_with_line_numbers(text, start_line=5) == expected
|
|
|
|
|
|
def test_render_preserves_already_numbered_lines():
|
|
"""Lines that already start with [N] must pass through unchanged."""
|
|
text = "[1] already numbered\n[2] also numbered"
|
|
assert render_with_line_numbers(text) == text
|
|
|
|
|
|
def test_render_preserves_already_numbered_with_offset():
|
|
"""Already-numbered lines pass through verbatim regardless of start_line."""
|
|
text = "[42] keep this number\n[43] and this"
|
|
assert render_with_line_numbers(text, start_line=100) == text
|
|
|
|
|
|
def test_render_mixed_numbered_and_plain():
|
|
"""Counter advances on every line; numbered lines pass through, plain lines get the running counter."""
|
|
text = "[10] kept\nplain line\n[12] also kept"
|
|
expected = "[10] kept\n[2] plain line\n[12] also kept"
|
|
assert render_with_line_numbers(text) == expected
|
|
|
|
|
|
def test_render_preserves_blank_lines():
|
|
"""Blank lines must still get a number — they're real positions in the drawer."""
|
|
text = "first\n\nthird"
|
|
expected = "[1] first\n[2] \n[3] third"
|
|
assert render_with_line_numbers(text) == expected
|
|
|
|
|
|
def test_render_preserves_trailing_newline_semantics():
|
|
"""Splitting on \\n and rejoining preserves the original boundary count."""
|
|
text = "a\nb\n"
|
|
result = render_with_line_numbers(text)
|
|
# "a\nb\n".split("\n") → ["a", "b", ""] — three positions
|
|
assert result == "[1] a\n[2] b\n[3] "
|
|
|
|
|
|
def test_render_none_input_returns_empty():
|
|
"""Defensive: None must not crash."""
|
|
assert render_with_line_numbers(None) == ""
|
|
|
|
|
|
def test_render_does_not_modify_original_text():
|
|
"""Function is pure — no mutation of input."""
|
|
original = "line one\nline two"
|
|
snapshot = original
|
|
render_with_line_numbers(original)
|
|
assert original == snapshot
|
|
|
|
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
# extract_line_range
|
|
# ─────────────────────────────────────────────────────────────────────────────
|
|
|
|
|
|
def test_extract_single_line():
|
|
text = "a\nb\nc\nd\ne"
|
|
assert extract_line_range(text, 3, 3) == "[3] c"
|
|
|
|
|
|
def test_extract_inclusive_range():
|
|
text = "a\nb\nc\nd\ne"
|
|
expected = "[2] b\n[3] c\n[4] d"
|
|
assert extract_line_range(text, 2, 4) == expected
|
|
|
|
|
|
def test_extract_full_range():
|
|
text = "first\nsecond\nthird"
|
|
expected = "[1] first\n[2] second\n[3] third"
|
|
assert extract_line_range(text, 1, 3) == expected
|
|
|
|
|
|
def test_extract_end_beyond_length_clips():
|
|
"""If line_end exceeds the document, return what's available — don't error."""
|
|
text = "a\nb\nc"
|
|
expected = "[2] b\n[3] c"
|
|
assert extract_line_range(text, 2, 99) == expected
|
|
|
|
|
|
def test_extract_start_below_one_clamps():
|
|
"""start_line < 1 clamps to 1; numbering starts from where extraction starts."""
|
|
text = "a\nb\nc"
|
|
# Clamping to 1 means we extract from line 1; numbering starts at 1.
|
|
expected = "[1] a\n[2] b"
|
|
assert extract_line_range(text, 0, 2) == expected
|
|
|
|
|
|
def test_extract_start_after_end_returns_empty():
|
|
text = "a\nb\nc"
|
|
assert extract_line_range(text, 5, 2) == ""
|
|
|
|
|
|
def test_extract_empty_text():
|
|
assert extract_line_range("", 1, 5) == ""
|
|
|
|
|
|
def test_extract_honors_already_numbered_lines():
|
|
"""If the slice contains already-numbered lines, they pass through."""
|
|
text = "plain\n[42] numbered\nplain again"
|
|
expected = "[1] plain\n[42] numbered\n[3] plain again"
|
|
assert extract_line_range(text, 1, 3) == expected
|
|
|
|
|
|
def test_extract_uses_drawer_line_numbers_not_relative():
|
|
"""When extracting lines 5-7, the rendered numbers must be [5][6][7], not [1][2][3].
|
|
|
|
This is the closet-pointer contract: a pointer →2026-01-18:L55-L72 must show
|
|
[55] through [72] in the rendered output, so the user sees which drawer
|
|
positions they're reading.
|
|
"""
|
|
text = "\n".join(f"line{i}" for i in range(1, 11)) # line1..line10
|
|
result = extract_line_range(text, 5, 7)
|
|
assert result.startswith("[5] line5")
|
|
assert "[6] line6" in result
|
|
assert result.endswith("[7] line7")
|
|
assert "[1]" not in result
|
|
assert "[8]" not in result
|
|
|
|
|
|
def test_extract_does_not_modify_original_text():
|
|
original = "x\ny\nz"
|
|
snapshot = original
|
|
extract_line_range(original, 1, 2)
|
|
assert original == snapshot
|