Files
mempalace/tests/test_line_numbers.py
T
Milla J b6dc122acb fix(extract): polish PR — address bot review feedback on PR #1555
PR #1555 (format coverage + virtual line numbering) merged with twelve
inline polish comments from Copilot + gemini-code-assist that weren't
load-bearing enough to block the original ship but are real cleanups.
This PR addresses them.

Twelve items in scope; one item (drawer ID delimiter — Copilot #13) is
deferred to its own dedicated PR because it's a breaking schema change
that requires migration design beyond the scope of a polish PR.

## Behavioral fixes (5 items, RED-tested first)

1. **FileNotFoundError vs broken symlink (Copilot #8).** ``extract_text``
   previously mapped every ``FileNotFoundError`` from ``stat()`` to
   ``SKIP_BROKEN_SYMLINK``. That's misleading for the common case of a
   regular file deleted between scan and extract. Now distinguishes:
   ``SKIP_BROKEN_SYMLINK`` only when ``p.is_symlink()`` is true;
   ``SKIP_UNREADABLE`` otherwise.

2. **``file_already_mined`` extract_mode scoping (Copilot #11, #12).**
   Both call sites in ``mine_formats`` and ``_file_chunks_locked`` now
   pass ``extract_mode="format"``. Previously the format miner could
   falsely treat drawers from project / convo miner on the same source
   file as "already mined" (and vice versa). Scopes idempotency to the
   correct drawer subset.

3. **Sentinel skip for transient missing-dep statuses (Copilot #14).**
   New ``_TRANSIENT_MISSING_DEP_STATUSES`` set + ``_register_skip_sentinel_if_appropriate``
   helper. Skip variants like ``SKIP_NO_MARKITDOWN`` /
   ``SKIP_NO_STRIPRTF`` / ``SKIP_MISSING_FORMAT_DEPS`` /
   ``SKIP_NETWORK_TIMEOUT`` no longer write the "already-mined" sentinel.
   Otherwise installing the missing extra later wouldn't trigger a re-mine.

4. **Outer ``except Exception`` in ``mine_formats`` (Gemini #5).** The
   outer try around the loop previously caught only ``KeyboardInterrupt``,
   leaving any setup-time error (e.g., ``scan_formats`` raising) to
   propagate as a bare traceback. Now catches ``Exception`` defensively,
   logs it, prints a partial-progress summary, and lets the ``finally``
   PID-cleanup run. Mirrors miner.py's belt-and-suspenders pattern.

5. **Thread user's ``chunk_size`` / ``chunk_overlap`` / ``min_chunk_size``
   through to ``chunk_text`` (Gemini #3).** ``MempalaceConfig`` was loaded
   only to validate readability; users who tuned their config saw no
   effect in format-mode mining. Now properly threaded.

## Trivial cleanups (5 items)

6. **Path expanduser in ``extract_text`` (Copilot #7).** ``Path(path)`` →
   ``Path(path).expanduser()`` so CLI inputs like ``~/docs/file.pdf``
   resolve correctly.

7. **Path expanduser+resolve in ``scan_formats`` (Copilot #9).** Same
   fix; ``~/docs`` and relative paths now work consistently.

8. **Use resolved ``format_path`` in ``mine_formats`` (Copilot #10).**
   ``scan_formats(format_dir)`` → ``scan_formats(format_path)`` so the
   already-resolved path is used.

9. **``render_with_line_numbers`` type annotation (Copilot #15).**
   ``text: "str | None"`` reflects the documented + tested ``None``
   handling.

10. **Test + docs claims (Copilot #16, #17, #18).** Stale framings
    removed:
    - ``docs/format-coverage.md`` — 14 fringe cases + "see the file for
      the current test inventory" (no more frozen test count).
    - ``tests/test_line_numbers.py`` — drops "proposed for mempalace
      3.3.6" + "run from the proposal directory" references.
    - ``tests/test_format_miner.py`` — drops "MarkItDown is mocked
      throughout" (live integration tests exist) + proposal-directory
      framing.

## Module-level hoists (enables clean test patching)

- ``MempalaceConfig`` (from ``.config``) hoisted from lazy local import
  to module-level so tests can patch ``mempalace.format_miner.MempalaceConfig``.
- ``chunk_text`` (from ``.miner``) hoisted similarly.

Both follow the pattern PR #1565 used for ``compute_hallways_for_wing``.

## Complexity refactor

Extracted ``_print_mine_summary`` from ``mine_formats`` so the orchestrator
stays under the project's ``max-complexity = 25`` ceiling (per
``pyproject.toml [tool.ruff.lint.mccabe]``). Behavior unchanged; pure
extraction.

## Out of scope (intentionally deferred)

- **Drawer ID delimiter collision (Copilot #13)** — ``f"{source_file}{chunk_index}"``
  can theoretically collide (``"/path/a1" + "23"`` == ``"/path/a" + "123"``).
  Fixing this is a breaking schema change to drawer IDs and requires a
  migration plan; will land as its own PR after design.

- The four bot comments that were ALREADY addressed by amendment #3
  before the PR #1555 merge (``_SKIP_DIRS`` dedup, ``scan_formats``
  symlink skip, ``source_mtime`` tracking, hall+entities metadata) —
  no action needed; verified during audit.

## Tests (RED-first)

Six new RED-first tests in ``tests/test_format_miner.py``:

  test_extract_text_nonexistent_regular_file_returns_unreadable_not_broken_symlink
  test_mine_formats_passes_extract_mode_format_to_file_already_mined
  test_mine_formats_does_not_write_sentinel_for_skip_no_markitdown
  test_mine_formats_does_not_write_sentinel_for_skip_missing_format_deps
  test_mine_formats_catches_unexpected_exception_and_prints_summary
  test_mine_formats_threads_chunk_size_from_user_config

All six RED before this commit (failures correctly identified the bugs
they're targeting), all six GREEN after.

One existing test (``test_mine_formats_continues_after_per_file_error``)
updated to patch the new module-level binding
``mempalace.format_miner.chunk_text`` instead of the old
``mempalace.miner.chunk_text`` source location, and to accept the
``**kwargs`` the call now passes through. Behavior unchanged.

## Verification

  pytest -q (full mempalace suite)
    → 2065 passed, 3 skipped, 0 regressions
  ruff check mempalace/format_miner.py mempalace/searcher.py tests/
    → All checks passed!
  ruff format --check ...
    → 4 files already formatted (pinned 0.15.9)
  mine_formats complexity
    → ≤ 25 (under the project ceiling)
2026-05-21 08:41:44 -07:00

159 lines
5.6 KiB
Python

"""Tests for virtual line numbering (mempalace 3.3.6, integrated in PR #1555).
Run with:
pytest tests/test_line_numbers.py -v
"""
from mempalace.searcher import ( # noqa: E402
extract_line_range,
render_with_line_numbers,
)
# ─────────────────────────────────────────────────────────────────────────────
# render_with_line_numbers
# ─────────────────────────────────────────────────────────────────────────────
def test_render_empty_string():
assert render_with_line_numbers("") == ""
def test_render_single_line():
assert render_with_line_numbers("hello") == "[1] hello"
def test_render_multi_line():
text = "alpha\nbeta\ngamma"
expected = "[1] alpha\n[2] beta\n[3] gamma"
assert render_with_line_numbers(text) == expected
def test_render_custom_start_line():
text = "first\nsecond"
expected = "[5] first\n[6] second"
assert render_with_line_numbers(text, start_line=5) == expected
def test_render_preserves_already_numbered_lines():
"""Lines that already start with [N] must pass through unchanged."""
text = "[1] already numbered\n[2] also numbered"
assert render_with_line_numbers(text) == text
def test_render_preserves_already_numbered_with_offset():
"""Already-numbered lines pass through verbatim regardless of start_line."""
text = "[42] keep this number\n[43] and this"
assert render_with_line_numbers(text, start_line=100) == text
def test_render_mixed_numbered_and_plain():
"""Counter advances on every line; numbered lines pass through, plain lines get the running counter."""
text = "[10] kept\nplain line\n[12] also kept"
expected = "[10] kept\n[2] plain line\n[12] also kept"
assert render_with_line_numbers(text) == expected
def test_render_preserves_blank_lines():
"""Blank lines must still get a number — they're real positions in the drawer."""
text = "first\n\nthird"
expected = "[1] first\n[2] \n[3] third"
assert render_with_line_numbers(text) == expected
def test_render_preserves_trailing_newline_semantics():
"""Splitting on \\n and rejoining preserves the original boundary count."""
text = "a\nb\n"
result = render_with_line_numbers(text)
# "a\nb\n".split("\n") → ["a", "b", ""] — three positions
assert result == "[1] a\n[2] b\n[3] "
def test_render_none_input_returns_empty():
"""Defensive: None must not crash."""
assert render_with_line_numbers(None) == ""
def test_render_does_not_modify_original_text():
"""Function is pure — no mutation of input."""
original = "line one\nline two"
snapshot = original
render_with_line_numbers(original)
assert original == snapshot
# ─────────────────────────────────────────────────────────────────────────────
# extract_line_range
# ─────────────────────────────────────────────────────────────────────────────
def test_extract_single_line():
text = "a\nb\nc\nd\ne"
assert extract_line_range(text, 3, 3) == "[3] c"
def test_extract_inclusive_range():
text = "a\nb\nc\nd\ne"
expected = "[2] b\n[3] c\n[4] d"
assert extract_line_range(text, 2, 4) == expected
def test_extract_full_range():
text = "first\nsecond\nthird"
expected = "[1] first\n[2] second\n[3] third"
assert extract_line_range(text, 1, 3) == expected
def test_extract_end_beyond_length_clips():
"""If line_end exceeds the document, return what's available — don't error."""
text = "a\nb\nc"
expected = "[2] b\n[3] c"
assert extract_line_range(text, 2, 99) == expected
def test_extract_start_below_one_clamps():
"""start_line < 1 clamps to 1; numbering starts from where extraction starts."""
text = "a\nb\nc"
# Clamping to 1 means we extract from line 1; numbering starts at 1.
expected = "[1] a\n[2] b"
assert extract_line_range(text, 0, 2) == expected
def test_extract_start_after_end_returns_empty():
text = "a\nb\nc"
assert extract_line_range(text, 5, 2) == ""
def test_extract_empty_text():
assert extract_line_range("", 1, 5) == ""
def test_extract_honors_already_numbered_lines():
"""If the slice contains already-numbered lines, they pass through."""
text = "plain\n[42] numbered\nplain again"
expected = "[1] plain\n[42] numbered\n[3] plain again"
assert extract_line_range(text, 1, 3) == expected
def test_extract_uses_drawer_line_numbers_not_relative():
"""When extracting lines 5-7, the rendered numbers must be [5][6][7], not [1][2][3].
This is the closet-pointer contract: a pointer →2026-01-18:L55-L72 must show
[55] through [72] in the rendered output, so the user sees which drawer
positions they're reading.
"""
text = "\n".join(f"line{i}" for i in range(1, 11)) # line1..line10
result = extract_line_range(text, 5, 7)
assert result.startswith("[5] line5")
assert "[6] line6" in result
assert result.endswith("[7] line7")
assert "[1]" not in result
assert "[8]" not in result
def test_extract_does_not_modify_original_text():
original = "x\ny\nz"
snapshot = original
extract_line_range(original, 1, 2)
assert original == snapshot