Commit Graph
520 Commits
Author SHA1 Message Date
Igor Lins e Silva 2dad4bbd7b Merge pull request #1422 from YuanYiZheXue/fix/lone-surrogate-clean
fix(mcp): sanitize lone surrogates before ChromaDB ops (#1235)
2026-05-24 13:56:18 -03:00
Igor Lins e Silva 382409bd6b Merge origin/develop into feat/benchmark-multilingual
Resolves conflicts in CHANGELOG.md and pyproject.toml by combining
the multilingual-embedder additions (huggingface_hub/tokenizers/numpy
core deps, [multilingual] alias, Features section) with develop's
additions (python-dateutil core dep, [extract] extra, tunnel Bug
Fixes and Internal sections).

Prepares PR #1483 for merge into v3.3.6.
2026-05-24 13:18:08 -03:00
Igor Lins e Silva 3e001921eb refactor(mcp): move lone-surrogate strip into shared sanitizers (#1235)
Push the surrogate-cleaning behaviour from per-call-site _clean() helpers
into sanitize_content, sanitize_kg_value, and sanitize_query so every
caller (existing and future) gets the fix automatically. New MCP tools
no longer need to remember to call a separate helper.

- Add strip_lone_surrogates() in mempalace/config.py as the single
  regex-based implementation (one U+FFFD per surrogate, not three).
- Wire it into sanitize_content and sanitize_kg_value.
- Wire it into sanitize_query so embedding lookups can't crash either.
- Drop the _clean() helper from mcp_server.py and the per-site calls;
  retain a direct strip_lone_surrogates() for source_file/added_by
  metadata which doesn't route through any sanitizer.
- Move the ONNX model-cache patch out of the test module and into
  conftest.py so it's session-scoped instead of duplicated locally.
- Update tests to assert one U+FFFD per surrogate and exercise the
  sanitizer-level entry points directly.
2026-05-24 12:12:52 -03:00
shushu 997ec1b22b test(mcp): add lone-surrogate sanitisation tests (issue #1235)
Add tests/test_clean_lone_surrogates.py covering:

Unit tests (TestCleanLoneSurrogates, 11 cases):
- _clean() passes normal ASCII and CJK strings unchanged
- lone surrogates (high/low, single/multiple) are replaced with U+FFFD
- real emoji (\U0001f600 astral code points) pass through unchanged
- empty string, all-surrogate string, SHA-256-hash-after-clean
- the specific \udcad surrogate observed in WorkBuddy production logs

Integration tests (TestLoneSurrogateCleaning, 6 cases):
- tool_add_drawer: surrogate in content and in metadata fields
- tool_check_duplicate: surrogate in query
- tool_search: surrogate in search query
- tool_update_drawer: surrogate in updated content
- tool_diary_write: surrogate in diary entry

Fix test environment issue:
- conftest.py redirects HOME to a temp dir, causing chromadb's
  ONNXMiniLM_L6_V2 to look for its ONNX model in the wrong location
  and trigger a 79 MB network download on every run.
- Fix: at module import time, recover the real USERPROFILE from
  conftest._original_env and patch ONNXMiniLM_L6_V2.DOWNLOAD_PATH
  before any ChromaDB collection fixture is invoked.
2026-05-24 12:12:14 -03:00
Igor Lins e Silva ce1ee26e11 Merge pull request #1548 from mvalentsev/fix/1537-validate-fts5-after-mine
fix(mine): validate FTS5 at end of mine (#1537)
2026-05-24 12:03:02 -03:00
Igor Lins e Silva 8a813d2807 Merge pull request #711 from sha2fiddy/feature/494-silent-mode-auto-save-toggle
feat: add hooks.auto_save config toggle (silent mode)
2026-05-24 11:58:58 -03:00
mvalentsevandCaleb Wells 0ecf0e5f06 fix(mine): validate FTS5 at end of mine (#1537)
Wires _validate_palace_fts5_after_mine into all three mine entry
points so corrupted-FTS5 palaces cannot silently exit 0 from any
of them:

- _mine_impl (mempalace/miner.py) — project file miner
- mine_convos (mempalace/convo_miner.py) — conversation exports
- mine_formats (mempalace/format_miner.py) — binary office documents
  via --mode extract, introduced on develop by #1555 (3.3.6 release)
  between this PR's open date and its rebase

cmd_mine surfaces MineValidationError as exit 1 + the same
print_sqlite_integrity_abort banner cmd_repair already prints,
appended with a mine-specific stderr note that hedges attribution
(quick_check cannot tell pre-existing corruption from corruption
this mine produced). 17 tests in tests/test_miner_fts5_validation.py
cover the helper, the three call sites, dry-run / KeyboardInterrupt
skip semantics, and the MineValidationError constructor invariants.

Closes #1537.

Co-authored-by: Caleb Wells <15988028+calebcwells@users.noreply.github.com>
2026-05-24 18:26:39 +05:00
Milla J 318940797f feat(entity): filter common English content words from entity detection
Common English content words ("Code", "Brutal", "Phase", "Line", "Note",
"Planning", "Chat", ...) frequently appear capitalized at sentence
start, in headings, or in markdown — and the existing regex-based entity
detector treats every capitalized word appearing 3+ times as a proper
noun candidate. This produces false positives that pollute
known_entities.json, the per-drawer entities metadata, and closet
pointer entity lists.

Adds a curated list of ~1000 common English content words (nouns,
adjectives, verbs — no proper nouns, no known product names) at
mempalace/data/coca_content_words.json, derived from the COCA top-2000
frequency list and from observed false positives in real palaces.
The list is loaded once at module import via _get_coca_filter() (cached
frozenset) and consulted at three sites:

  - entity_detector.extract_candidates — init-time entity detection
  - palace.build_closet_lines           — closet pointer construction
  - miner._extract_entities_for_metadata — per-drawer entity tagging

Matching is case-insensitive: the candidate is lowercased before
membership check, so "Code", "CODE", and "code" are all blocked.
Only single-word candidates are filtered; multi-word phrases (e.g.
"Claude Code") are NOT filtered, so legitimate compound names continue
to be detected. A future PR will add a known-systems lexicon that
explicitly protects compound names — for now, the multi-word pass
provides the implicit protection.

Adds 8 new tests in tests/test_entity_detector.py covering:

  - "Code" appearing 5x is filtered
  - All known false positives from real palaces are filtered
  - Case-insensitive matching works
  - Real names ("Aya", "Riley") are NOT filtered
  - Multi-word phrases like "Claude Code" are NOT filtered
  - The data file ships in the wheel with the expected schema
  - The data file contains every known false positive

Note: this filter prevents NEW false positives at detection time. It
does NOT retroactively scrub existing entries from a user's
~/.mempalace/known_entities.json. Stale entries from previous runs
must be removed via a future entity-management CLI (planned).

Verification: pytest 2191 passed / 3 skipped / 0 failed on macOS.
ruff check + format --check both clean. Cross-platform via
pip install -e ".[dev]" on python:3.9-slim / 3.11-slim / 3.13-slim
containers, each running 65 entity-detector tests including the new
data-file-presence test that imports from the installed package.
End-to-end pipeline test mining a deterministic corpus into an
isolated palace confirmed entity metadata is empty (no false
positives surface in real chromadb-backed storage).
2026-05-24 04:16:42 -07:00
sha2fiddy 4a3b48c56a style: ruff format tests/test_hooks_cli.py with CI-pinned ruff 0.4.x 2026-05-23 12:05:34 -04:00
sha2fiddy c2a61d85ed fix(hooks): retarget auto_save toggle at silent-save path after #1021 rebase 2026-05-23 12:05:34 -04:00
sha2fiddy ada25872a1 fix: correct precompact test assertion + ruff format tests
test_precompact_hook_enabled_by_default asserted
result["decision"] == "block", but hook_precompact has never emitted
decision — it mines synchronously and returns {}. Assertion was
copy-pasted from the stop-hook test. Fix to assert result == {} with
_mine_sync mocked so the test verifies the real contract (enabled →
mine + pass through) without actually mining.

Plus ruff format on 6 test files the CI pin flagged.
2026-05-23 12:05:34 -04:00
sha2fiddy 8fc06b96e2 fix: address review — enrich block reasons, clean up tests
- Add parenthetical hints to block reasons so AI knows what each tool
  saves (session summary, quotes/decisions/code)
- Remove dead config file creation from test_stop_hook_disabled_by_config
- Add missing test for MEMPALACE_HOOKS_AUTO_SAVE=no env var
- Replace bare except: with except Exception: in shell scripts
2026-05-23 12:05:34 -04:00
sha2fiddy 49a8c868fa feat: add hooks.auto_save config toggle and shorten block reasons
Add a clean opt-out for auto-save hook blocking (closes #494).

- New `hooks.auto_save` config option (default: true) in
  ~/.mempalace/config.json and MEMPALACE_HOOKS_AUTO_SAVE env var
- When disabled, stop and precompact hooks pass through without blocking
- Shorten block reason text from 6-line instructions to single-line
  prompts — reduces UI noise while keeping tool names explicit
- Both Python (hooks_cli.py) and standalone shell scripts respect the
  toggle via config file or env var
2026-05-23 12:05:34 -04:00
Igor Lins e Silva f6edb7d1ab Merge pull request #1584 from MemPalace/feat/closet-date-line-pointer-emission
feat(closets): Tier 6a — date+line locators with content-date hierarchy
2026-05-23 12:01:06 -03:00
Milla J 65f24b3bdd fix(closets): address PR #1584 Igor review — wire + harden Tier 6a
Igor's review on PR #1584 (2026-05-22) flagged four issues:

  1. The feature wasn't wired into any production caller — the new
     ``drawer_metas`` kwarg on ``build_closet_lines`` had no real
     consumer in ``miner.py`` / ``diary_ingest.py``, so the 4-segment
     pointer form only existed in tests. Real palaces kept emitting
     the legacy 3-segment shape.
  2. ``_extract_content_date`` hallucinated dates on benign inputs.
     ``dateutil.parser.parse(fuzzy=True)`` would accept anything with
     digits and return a plausible-looking but wrong date —
     ``Version 3.3.6`` → ``2006-03-03``, ``Tested with 1000 drawers``
     → ``1000-05-22``, ``tmp_random_file_5`` → ``2026-05-05``, etc.
     Mtime almost never got reached because fuzzy returned *something*
     from filename or body first. Bad dates were silently persisted
     to ChromaDB.
  3. ``python-dateutil`` was an undeclared dependency, available only
     transitively via ``chromadb → kubernetes → python-dateutil``. Not
     a contract — upstream kubernetes has been trending toward
     stdlib-only.
  4. Two-digit-year disambiguation (70 → 19xx / 00-69 → 20xx) had no
     test pinning the boundary.

This commit addresses all four.

## Changes

### Issue 2 — kill the hallucination (the load-bearing fix)

``mempalace/miner.py``:

- New ``_VALID_DATE_RE`` gate. Three accepted shapes (all require a
  4-digit year explicitly):

    1. Numeric YYYY-MM-DD with ``[-/.\\s]`` separators
       (covers ISO and space-normalized filenames)
    2. Month-name + day + year ("November 8 2024", "Nov 8 2024")
    3. Day + month-name + year ("8 November 2024")

  Partial dates ("2024-06", "April 6", "notes.2024") are
  DELIBERATELY rejected — without all three components we'd pad from
  today's date, which is hallucination not extraction.

- ``_try_filename_date`` and ``_try_content_body_date`` now run the
  gate BEFORE invoking dateutil, and pass ``fuzzy=True`` is REMOVED.
  Dateutil only runs in strict mode on a substring the gate already
  validated.

### Issue 1 — wire the feature into production

``mempalace/miner.py`` batched-upsert path:

- Accumulate ``batch_metas`` across all batches into ``all_metas``
- Pass ``drawer_metas=all_metas`` to ``build_closet_lines``

End-to-end integration test added that mines a real file with a
filename-derived content date and asserts the produced closet
documents contain the 4-segment pointer with that date.

``diary_ingest.py`` is left as-is for this PR. Diary entries are
entry-keyed, not chunk-keyed — they carry no natural
``line_start`` / ``line_end``, so the 4-segment form would return
None for them regardless. Wiring the diary path can land cleanly in
a follow-up once Tier 6a gains an "approximate line range for diary
entries" story.

### Issue 3 — declare the dateutil dependency

``pyproject.toml``: add ``python-dateutil>=2.8`` to
``[project].dependencies``. One-line change; cheaper than the
stdlib-only refactor alternative and keeps the natural-language
recall surface.

### Issue 4 — pin the two-digit-year boundary

Four new tests cover the 1969/1970/1999/2000 corner cases of the
slash-date locale heuristic.

## Tests added (RED-first then GREEN)

  tests/test_miner.py::TestExtractContentDate (11 new):
    Hallucination cases verbatim from Igor's review:
    - test_no_hallucination_junk_filename_with_trailing_digit
    - test_no_hallucination_untitled_with_index
    - test_no_hallucination_filename_year_only
    - test_no_hallucination_filename_year_and_month_only
    - test_no_hallucination_content_with_issue_number
    - test_no_hallucination_content_with_count
    - test_no_hallucination_content_with_version_number
    Two-digit-year boundary cases:
    - test_two_digit_year_69_is_2069
    - test_two_digit_year_70_is_1970
    - test_two_digit_year_99_is_1999
    - test_two_digit_year_00_is_2000

  tests/test_closets.py::TestMinerClosetRebuild (1 new):
    - test_production_miner_emits_4_segment_pointers_with_content_date
      (regression for Issue #1 — real ``mine()`` end-to-end produces
      4-segment closet pointers via the new ``drawer_metas`` wiring)

## Verification

  pytest tests/test_miner.py tests/test_closets.py
         tests/test_format_miner.py tests/test_palace.py
    → 242 passed, 2 skipped, 0 regressions

  pytest tests/test_miner.py::TestExtractContentDate
    → 26 passed (15 prior + 11 new)

  pytest tests/test_closets.py::TestMinerClosetRebuild
    → end-to-end wiring test GREEN

  Sanity (Igor's exact repros):
    "tmp_random_file_5"           → None (was: 2026-05-05)
    "untitled-1"                  → None (was: 2026-05-01)
    "notes.2024.md"               → None (was: 2024-05-22)
    "2024-06.md"                  → None (was: 2024-06-22)
    "Bug fix for issue 42 in module 7" → None (was: 2042-07-22)
    "Tested with 1000 drawers"    → None (was: 1000-05-22)
    "Version 3.3.6 released"      → None (was: 2006-03-03)

  Real dates still extract correctly:
    "2024-11-08.md"               → "2024-11-08"
    "April-6th-2011-notes.md"     → "2011-04-06"
    "Nov-8-2024.md"               → "2024-11-08"

  ruff check + ruff format --check (pinned 0.15.9)
    → All checks passed

  OrbStack triple-Linux verify (Py 3.9 / 3.11 / 3.13)
    → all targeted tests pass; python-dateutil installs explicitly
       via the new declared dependency.
2026-05-22 10:45:43 -07:00
Igor Lins e Silva c89c077d7e Merge pull request #1578 from MemPalace/feat/hallway-tunnel-decay-dynamics
feat(dynamics): Hebbian potentiation + Ebbinghaus decay for halls + tunnels
2026-05-22 13:15:39 -03:00
Igor Lins e Silva 5633406e5e Merge pull request #1585 from milla-jovovich/fix/hooks-cli-shallow-path-guard
fix(hooks_cli): guard parents[3] access against shallow filesystem paths
2026-05-22 11:18:08 -03:00
Igor Lins e SilvaandClaude Opus 4.7 4b7edd9aea Merge branch 'develop' into docs/codex-discoverability
Bring docs PR up to date with develop so CI re-runs against the
current pin set.

Co-Authored-By: Igor Lins e Silva <4753812+igorls@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-22 09:54:27 -03:00
Milla J 749d43359a style(hooks_cli): address PR #1580 gemini review — LBYL + side_effect
Two medium-priority gemini-code-assist comments on PR #1580 both
recommend more-idiomatic Python:

1. **Production code (``mempalace/hooks_cli.py::_mempalace_python``)** —
   replace ``try/except IndexError`` with ``if len(parents) > N:``
   look-before-you-leap checks. Exception handling for bounded-integer
   index lookups is a code smell in Python; LBYL makes the depth check
   explicit and removes exception overhead. Same behavior, clearer
   intent.

   Before (EAFP, ~12 lines + comment):
       try:
           venv_bin = resolved.parents[3] / "bin" / "python"
           if venv_bin.is_file():
               return str(venv_bin)
       except IndexError:
           pass

   After (LBYL, ~5 lines):
       if len(parents) > 3:
           venv_bin = parents[3] / "bin" / "python"
           if venv_bin.is_file():
               return str(venv_bin)

2. **Test code (``tests/test_hooks_cli.py``)** — replace the lambda +
   generator-throw hack with ``MagicMock.side_effect = get_item``,
   where ``get_item`` is a normal function that returns the
   editable-install path for index 1 and raises ``IndexError`` for
   any other index (defensive against a future regression that drops
   the LBYL length check). Standard ``side_effect`` mocking pattern.

   Before:
       fake_parents.__getitem__ = lambda self, idx: (
           RealPath("/work/mempalace")
           if idx == 1
           else (_ for _ in ()).throw(IndexError(idx))
       )

   After:
       def get_item(idx):
           if idx == 1:
               return RealPath("/work/mempalace")
           raise IndexError(idx)

       fake_parents.__len__.return_value = 3
       fake_parents.__getitem__.side_effect = get_item

   Also added ``__len__`` mock so the LBYL length check in production
   sees the simulated shallow path correctly.

## Verification

  pytest tests/test_hooks_cli.py
    → 110 passed, 1 skipped (same as PR #1580 baseline; regression
       test for shallow-path crash still GREEN)

  ruff check + ruff format --check (pinned 0.15.9)
    → All checks passed; 2 files already formatted
2026-05-22 05:12:01 -07:00
Milla J 5482262749 fix(hooks_cli): guard parents[3] access against shallow filesystem paths
``_mempalace_python()`` in ``mempalace/hooks_cli.py`` uses
``Path(__file__).resolve().parents[3]`` to locate the venv Python
interpreter in the standard install layout
``<venv>/lib/pythonX.Y/site-packages/mempalace/hooks_cli.py``. When the
package lives at a shallow filesystem path — Docker containers
mounting at ``/work``, ``/opt/app``, minimal-prefix production
installs — ``parents`` has fewer than 4 elements and the index raises
``IndexError`` instead of falling through to the editable-install
branch.

The crash was caught by OrbStack-based triple-Python CI verification
on PR #1579: 16 tests in ``test_hooks_cli.py`` failed identically on
Linux 3.9 / 3.11 / 3.13 with the same ``IndexError: 3`` from
``pathlib._PathBase.parents.__getitem__`` — and verified pre-existing
on develop tip in the same container. The bug never surfaces in
GitHub Actions CI runners (their workdir at
``/home/runner/work/mempalace/mempalace`` has plenty of parent
directories) but it surfaces immediately for anyone:

  - running mempalace in editable mode inside a Docker dev container
  - shipping mempalace as part of an OCI image where the install
    prefix is ``/app`` or ``/opt/<name>``
  - using OrbStack / Colima / podman-machine for cross-version
    verification

## The fix

Wrap each ``parents[N]`` access in ``try/except IndexError`` so the
helper falls through to the next strategy (editable-install →
``sys.executable``) instead of crashing the hook. Both ``parents[3]``
AND ``parents[1]`` are guarded — the latter is defensive against
extreme cases like a file at root (``/file.py``, parents=[/]) — same
class of bug.

## Test added (RED-first, then GREEN)

  tests/test_hooks_cli.py::test_mempalace_python_handles_shallow_path_without_crashing

Mocks ``Path(__file__).resolve()`` so ``parents[3]`` raises
``IndexError`` and ``parents[1]`` returns a real shallow path
(``/work/mempalace``). Pre-commit: function raises ``IndexError: 3``.
Post-commit: function returns a valid Python interpreter path
(either editable-venv if present, otherwise ``sys.executable``).

## Verification

  pytest tests/test_hooks_cli.py
    → 110 passed, 1 skipped on macOS (the existing run)
    → 110 passed, 1 skipped on Linux 3.9 / 3.11 / 3.13 (OrbStack)
       — was 16 failed, 94 passed before this commit

  ruff check + ruff format --check (pinned 0.15.9)
    → All checks passed; 2 files already formatted
2026-05-22 05:00:04 -07:00
Milla J 67c6ab59f2 style: apply ruff format with pinned 0.15.9 (CI lint) 2026-05-22 04:24:07 -07:00
Milla J 05da803f71 feat(closets): Tier 6a — date+line locators with content-date hierarchy
Closet pointer lines gain a 4th pipe-separated segment of shape
``YYYY-MM-DD:Lstart-Lend`` so retrieval can jump straight to the right
span in the source file instead of opening the whole drawer. The date
is selected via a content-aware hierarchy so legacy ingests (where
``filed_at`` is misleading) still produce honest pointers when the
content itself carries a date.

The legacy 3-segment format remains the fallback for drawers without
the new metadata, so existing palaces keep working without a re-mine.

## Pointer shape

Before:
  topic|entities|→drawer_ids

After (when drawer metadata carries line range + a date):
  topic|entities|2024-11-08:L42-L78|→drawer_ids

Backward compat: when ``drawer_metas`` is not passed, or the first
meta lacks ``line_start`` / ``line_end``, ``build_closet_lines`` emits
the legacy 3-segment form unchanged.

## Date-source hierarchy (NEW)

A new helper ``mempalace.miner._extract_content_date(source_file, content)``
selects a date once per file and stores it as ``content_date`` on every
drawer mined from that file. Priority order, first match wins:

  1. **Filename** — ISO regex on stem, then dateutil fuzzy parse for
     natural-language formats. Handles ``2024-11-08.md``,
     ``April-6th-2011-notes.md``, ``Nov-8-2024.md``, etc.

  2. **YAML frontmatter** — looks at ``date`` / ``created`` /
     ``published`` fields. yaml.safe_load already parses ISO dates as
     ``datetime.date`` objects; dateutil handles string values.

  3. **Content body, first ~10 lines** —
       a. ISO regex (``YYYY-MM-DD`` / ``YYYY/MM/DD`` / ``YYYY.MM.DD``)
       b. Slash dates with locale auto-disambiguation: if any
          first-number > 12 appears anywhere in the head, lock the
          file's locale to DD/MM (the only consistent reading);
          otherwise default to US MM/DD for two-digit-year compactness.
       c. dateutil fuzzy parse for natural-language
          (``November 8, 2024``, ``8th November 2024``, etc.)

  4. **Filesystem mtime** — ``os.path.getmtime`` formatted as
     ``YYYY-MM-DD``.

  5. **None** — caller (``_build_drawer_metadata``) leaves
     ``content_date`` absent; ``build_closet_lines`` falls back to the
     ``filed_at`` ingestion timestamp's date portion.

``_build_date_line_segment`` in ``palace.py`` now prefers ``content_date``
when present, otherwise uses ``filed_at[:10]``. This is the load-bearing
change that makes Tier 6a meaningful for legacy content — a Nov 8 2024
transcript mined today now emits ``2024-11-08:L42-L78`` instead of
``2026-05-22:L42-L78``.

## Changes

1. ``mempalace/miner.py::chunk_text`` — each chunk dict now also carries
   ``line_start`` / ``line_end`` (1-indexed line numbers in the stripped
   source). Computed via ``content[:offset].count("\n") + 1`` —
   approximate locator accurate to ±1 line at chunk boundaries.

2. ``mempalace/miner.py::_extract_content_date`` (NEW) + helpers
   (``_try_filename_date``, ``_try_frontmatter_date``,
   ``_try_content_body_date``, ``_try_mtime_date``). Pure functions, no
   I/O beyond what's named. Uses lazy imports of ``dateutil`` and
   ``yaml`` to keep top-level import surface unchanged.

3. ``mempalace/miner.py::_build_drawer_metadata`` — gains
   ``line_start`` / ``line_end`` / ``content_date`` optional kwargs.
   When provided, stored in drawer metadata; when omitted, keys are
   absent from the dict — backward compatible.

4. ``mempalace/miner.py`` batched-upsert path — calls
   ``_extract_content_date(source_file, content)`` ONCE per file (not
   once per chunk) and passes the result through to every chunk's
   ``_build_drawer_metadata`` call.

5. ``mempalace/format_miner.py::_file_chunks_locked`` — gains a
   ``content`` param so the format-miner path can also run
   ``_extract_content_date`` on the full extracted text. Plumbs
   ``content_date`` into every batched drawer's metadata. Caller passes
   ``text`` (the full extracted markdown).

6. ``mempalace/palace.py::build_closet_lines`` — accepts optional
   ``drawer_metas`` (parallel to ``drawer_ids``). When present and the
   first meta has both ``line_start`` and ``line_end``, the new
   segment is spliced in; otherwise legacy 3-segment form. Helper
   ``_build_date_line_segment`` prefers ``content_date`` over
   ``filed_at`` for the date portion.

## Tests added (RED-first, then GREEN)

  tests/test_miner.py::TestChunkTextLineRanges          (4 tests)
  tests/test_miner.py::TestBuildDrawerMetadataLineRange (2 tests)
  tests/test_miner.py::TestExtractContentDate           (15 tests)
    - filename ISO / natural-language / compact
    - YAML frontmatter date / created
    - Claude session preamble
    - ISO + natural-language content body
    - Slash-date locale disambiguation (DD/MM lock vs default MM/DD)
    - Priority order (filename > frontmatter > content > mtime)
    - mtime fallback
    - Graceful None returns for missing file / empty content

  tests/test_closets.py::TestBuildClosetLines           (5 new tests)
    - includes_date_line_segment_when_metas_provided
    - falls_back_to_3_segment_format_when_metas_missing  (compat)
    - falls_back_to_3_segment_format_when_metas_lack_line_keys (compat)
    - date_segment_uses_filed_at_date_portion_only
    - content_date_preferred_over_filed_at               (new — Tier 6a hierarchy)

26 new tests total. All RED before this commit, all GREEN after.

## Verification

  pytest tests/test_miner.py::TestChunkTextLineRanges
         tests/test_miner.py::TestBuildDrawerMetadataLineRange
         tests/test_miner.py::TestExtractContentDate
         tests/test_closets.py::TestBuildClosetLines
    → 32 passed

  pytest tests/test_miner.py tests/test_closets.py
         tests/test_format_miner.py tests/test_palace.py
    → 230 passed, 2 skipped, 0 regressions in directly-affected files

  ruff check + ruff format --check
    → All checks passed, 5 files already formatted

  End-to-end smoke test on a YAML-frontmatter file (date: 2024-11-08,
  mined "today"):
    content_date extracted: 2024-11-08
    chunks produced: 1
    drawer meta: content_date=2024-11-08, filed_at='2026-05-22...', lines=1-15
    closet pointer:
      "conversation with lumi about brands of dog food|Filler;Lumi;Aya|2024-11-08:L1-L15|→drawer_a,drawer_b"
    → content_date correctly preferred over filed_at end-to-end.

  OrbStack triple-Linux verify (Py 3.9 / 3.11 / 3.13)
    → 32 targeted Tier 6a + content-date tests pass on each.

## Out of scope (deliberate)

- No schema migration of existing drawers. Pre-Tier-6a drawers lack
  ``line_start`` / ``content_date`` and silently fall back to the
  legacy 3-segment pointer shape.
- No re-mine required. Users keep existing palaces; new files mined
  after this lands carry the new fields.
- No drawer ID change (Task #80, separate concern).
- No exact-quote line positioning within a chunk (future tier).
- No NLP-based date extraction (only regex + dateutil fuzzy).
- Linguistic-similarity / concept-recognition retrieval (L5 zettal
  layer) is the natural next step but lives in its own PR family.
2026-05-22 04:02:03 -07:00
Milla J f1aa409a94 fix(dynamics): address PR #1578 gemini review — canonicalize lookup key + idiomatic dict update
Addresses both inline comments left by gemini-code-assist on PR #1578.

## What this commit does

1. **HIGH PRIORITY (gemini, hallways.py:260) — canonicalize dynamics
   preservation lookup key.** The previous code built the lookup key as
   ``(h.get("entity_a"), h.get("entity_b"))``, but the
   ``pair_counts`` dict and ``_hallway_id`` both use sorted entity pairs
   for symmetric IDs. A persisted record with reversed entity order
   (Lumi as entity_a, Aya as entity_b) would silently miss the lookup
   and lose its accumulated dynamics on recompute. Now the lookup
   canonicalizes the same way ID generation does:

       key = tuple(sorted([h.get("entity_a"), h.get("entity_b")]))

   Failure mode is rare in practice (the algorithm normally writes
   sorted records), but any legacy data, manual JSON edit, or future
   import path that violates the ordering would have silently broken
   L7 preservation. Belt-and-suspenders fix.

2. **MEDIUM PRIORITY (gemini, palace_graph.py:512) — idiomatic field
   preservation in tunnel dedup path.** Refactored the manual for-loop
   into a dict-update with a comprehension. Field list lives in one
   place; future schema expansion can't accidentally drop a field. No
   behavior change.

## Test added (RED-first)

  test_recompute_preserves_dynamics_when_existing_record_has_reversed_entity_order

Pre-populates the hallway store with a record whose ``entity_a`` /
``entity_b`` are in REVERSED (non-sorted) order, bumps its strength /
stability / access_count to non-default values, then runs
``compute_hallways_for_wing``. Asserts the recomputation preserves the
bumped values — proving the canonical lookup matches against
arbitrarily-ordered persisted entity pairs.

Reproduces the bug class gemini flagged: RED before this commit
(strength=1.0 instead of 4.2), GREEN after.

## Verification

  pytest tests/test_hallways.py::TestHallwayDynamicsIntegration
        tests/test_palace_graph_tunnels.py::TestTunnelDynamicsIntegration
        tests/test_dynamics.py
    → 32 passed (was 31; added 1 new for the reverse-order case)
  pytest -q (full mempalace suite)
    → 2112 passed, 3 skipped, 0 regressions
  ruff check mempalace/dynamics.py mempalace/hallways.py mempalace/palace_graph.py tests/
    → All checks passed!
  ruff format --check ...
    → 3 files already formatted (pinned 0.15.9)
2026-05-21 21:32:19 -07:00
Igor Lins e SilvaandClaude Opus 4.7 3b6c7986c9 Merge branch 'develop' into fix/tunnel-file-config-and-endpoint-validation
Resolves conflicts from 60-commit divergence:

- tests/test_closets.py: assertion reformat — kept develop's ruff-format-
  preferred multi-line shape (functionally identical).
- tests/test_palace_graph_tunnels.py: both branches added a new test
  class at end-of-file (this PR's TestTunnelFileFollowsConfig + develop's
  TestEntityTunnels from #1564). Kept both, no overlap.

Local: ruff check / format pass; full pytest suite passes
(2113 passed, 3 skipped).

Co-Authored-By: Igor Lins e Silva <4753812+igorls@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 22:31:41 -03:00
Igor Lins e SilvaandClaude Opus 4.7 e97b3621ca Merge branch 'develop' into fix/convos-mine-concurrency
Resolves conflicts from 128-commit divergence:

- mempalace/convo_miner.py imports: kept both `mine_palace_lock` (this PR)
  and `prefetch_mined_set` (develop).
- mempalace/convo_miner.py docstring: kept this PR's lock-wrapping
  description, added a one-line pointer to the chunking-config section
  whose body now lives in `_mine_convos_impl`.
- mempalace/convo_miner.py body: develop placed `cfg_chunk_size` /
  `cfg_min_chunk_size` setup inline in `mine_convos`. This PR factored
  the body into `_mine_convos_impl`, so the inline setup would have
  left `cfg_chunk_size` referenced-but-undefined inside the impl.
  Moved the `MempalaceConfig()` setup into `_mine_convos_impl` so the
  variables are in scope where they're used.
- tests/test_convo_miner.py: kept both additive test sets (lock
  concurrency from this PR + wing_api auto-routing from develop).

Local: ruff check / format pass; full pytest suite passes
(2103 passed, 3 skipped).

Co-Authored-By: Igor Lins e Silva <4753812+igorls@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 22:07:46 -03:00
Milla J a2ba1ccb05 feat(dynamics): Hebbian potentiation + Ebbinghaus decay for halls + tunnels
Adds the L7 living-connection dynamics layer for the hallway primitive
(PR #1558) and the tunnel primitive (PR #1565). Connections now carry
strength, stability, last_activated, and access_count fields; pure-math
helpers in a new ``mempalace.dynamics`` module update these via Hebbian
potentiation (Hebb 1949) on co-access and Ebbinghaus exponential decay
(Ebbinghaus 1885) on time-since-use, with the Cepeda spacing effect
(Cepeda et al. 2006) growing stability when reinforcement is distributed
rather than massed.

Existing palaces continue to work unchanged — the new fields populate
via safe defaults (strength=1.0, stability=1.0, last_activated=created_at,
access_count=0) lazily on any potentiate/decay call. No migration step
required. No schema break. Old records work; new records ship with full
L7 state from creation.

## What this commit does

1. New module ``mempalace/dynamics.py`` — pure math, no I/O, no chromadb.
   Three public helpers:

     - ``initialize_dynamics_fields(connection, *, now=None)``
       Backfill helper. Populates missing strength / stability /
       last_activated / access_count fields. Existing fields are NOT
       overwritten. Safe for pre-L7 records.

     - ``potentiate(connection, *, increment=POTENTIATION_INCREMENT, now=None)``
       Hebbian strengthening on a co-access event. Increments strength
       (capped at MAX_STRENGTH), updates last_activated, increments
       access_count. Grows stability by STABILITY_INCREMENT ONLY when
       the gap since the prior activation is at least SPACED_INTERVAL_HOURS
       — implements the Cepeda spacing effect (rapid bursts don't
       build durability; distributed practice does).

     - ``apply_decay(connection, *, now=None)``
       Ebbinghaus exponential decay. New strength = old * exp(-days_since /
       stability), floored at STRENGTH_FLOOR (0.05) so connections never
       reach zero — only become dim. Idempotent at the same instant.
       Higher stability = slower decay.

2. Integration in ``mempalace/hallways.py:compute_hallways_for_wing`` —
   before persisting the new hallway list for a wing, looks up the
   existing wing's hallways and builds a (entity_a, entity_b) → dynamics
   lookup. Each new record gets the preserved dynamics if the entity
   pair existed before, OR default dynamics via
   ``initialize_dynamics_fields`` if it's a brand-new pair. Without
   this preservation, every mine would wipe accumulated connection
   weights — defeating the whole L7 layer.

3. Integration in ``mempalace/palace_graph.py:create_tunnel`` — the
   existing dedup-on-canonical-id path now also preserves the four
   dynamics fields from the existing record onto the recreated tunnel
   (in addition to the already-preserved ``created_at``). Brand-new
   tunnels and legacy records (created before L7) get default dynamics
   via ``initialize_dynamics_fields``.

## Tests (RED-first)

Twenty-five new tests in ``tests/test_dynamics.py`` covering the pure
math: field initialization defaults, potentiation strength increment +
cap, last_activated update, access_count increment, spaced vs. rapid
reinforcement (Cepeda spacing effect), decay rate (exp(-days/stability)),
floor clamping, idempotency at same instant, higher-stability slower-decay
behavior, no-time-passed no-op, backfill on missing fields, mutate-and-
return-same-dict chaining. Plus integration scenarios: potentiation
restoring decayed strength, repeated spaced reinforcement growing
stability monotonically, burst reinforcement not growing stability.

Six new integration tests:

  test_new_hallway_record_carries_all_dynamics_fields
  test_recompute_preserves_accumulated_strength
  test_recompute_initializes_dynamics_for_brand_new_pairs
  test_new_tunnel_carries_all_dynamics_fields
  test_recreate_tunnel_preserves_accumulated_dynamics
  test_recreate_tunnel_initializes_dynamics_for_legacy_records

All RED-first (KeyError: 'strength' before the integration commits).
All GREEN after.

## Backward compatibility

Pre-existing palaces work unchanged:

- Old hallway records with no dynamics fields are still readable; the
  next ``compute_hallways_for_wing`` recompute populates the missing
  fields via ``initialize_dynamics_fields``.
- Old tunnel records similarly; the next ``create_tunnel`` recreate (or
  any future call passing the record through ``initialize_dynamics_fields``)
  populates the missing fields.
- A ``test_recreate_tunnel_initializes_dynamics_for_legacy_records`` test
  pins this contract explicitly.

No forced migration. No schema break. Old records work; new behavior
kicks in on first touch.

## What this does NOT do (intentionally scoped out)

- Spreading activation at search time — that's a search-layer change;
  bigger PR; separate work.
- Lazy decay-on-read in ``list_hallways`` / ``list_tunnels`` — the math
  is here; wiring it into the read path is a follow-up.
- Config exposure of the tunable constants — STRENGTH_FLOOR, MAX_STRENGTH,
  POTENTIATION_INCREMENT etc. are hardcoded module-level for v1. If
  empirical palace data shows the defaults need tuning, expose via
  MempalaceConfig in a follow-up.
- Decay-aware ranking in search — the strength field is now there; the
  search ranker doesn't yet use it. Follow-up.

## Verification

  pytest tests/test_dynamics.py
    → 25 passed (RED before this commit; GREEN after)
  pytest tests/test_hallways.py::TestHallwayDynamicsIntegration
    → 3 passed (RED before; GREEN after)
  pytest tests/test_palace_graph_tunnels.py::TestTunnelDynamicsIntegration
    → 3 passed (RED before; GREEN after)
  pytest -q (full mempalace suite)
    → 2111 passed, 3 skipped, 0 regressions
  ruff check mempalace/dynamics.py mempalace/hallways.py mempalace/palace_graph.py tests/
    → All checks passed!
  ruff format --check ...
    → 6 files already formatted (pinned 0.15.9)

## Research grounding

  - Hebb, D. O. (1949). The Organization of Behavior. Wiley.
    → "Neurons that fire together, wire together" → potentiate()
  - Ebbinghaus, H. (1885). Über das Gedächtnis.
    → Exponential forgetting curve → apply_decay()
  - Cepeda, N. J., Pashler, H., Vul, E., Wixted, J. T., & Rohrer, D.
    (2006). Distributed practice in verbal recall tasks: A review and
    quantitative synthesis. Psychological Bulletin, 132(3), 354-380.
    → Spacing effect → stability growth on spaced reinforcement
2026-05-21 14:35:20 -07:00
mvalentsev 6658a4d11f fix(audit): chunk and batch content before embedding upsert (#1539)
Four sites passed content to `collection.upsert(documents=[...])` or
`collection.add(documents=[...])` without per-drawer size cap, hitting
the same `RuntimeError: Invalid buffer size` crash class from the
embedding model's attention buffer:

- `general_extractor.extract_memories`: post-classification slicer
  that propagates `memory_type` to every slice. New `chunk_size`
  parameter defaulting to `DEFAULT_CHUNK_SIZE` and resolved from
  `MempalaceConfig.chunk_size` by the caller.
- `diary_ingest.ingest_diaries`: per-entry drawers via existing
  `_split_entries`, with character-chunk fallback for any single
  entry larger than `chunk_size`. New `_diary_drawer_id_entry`
  helper carries (entry_idx, entry_chunk_idx). `chunk_index` in
  metadata is a global counter across the file so
  `searcher._expand_with_neighbors` stitches sibling chunks
  regardless of entry boundary. Upsert is batched atomic per file
  (one call carrying every entry/chunk) so a mid-pass embedding
  failure cannot half-write the day. Auto-purge on full rebuild
  deletes prior-pass drawers via `where={"source_file": ...}`,
  which also migrates pre-#1539 legacy `drawer_diary_` IDs as a
  side effect of normal use.
- `mcp_server.tool_diary_write`: split oversized entries into
  bounded per-chunk drawers via a single batched `col.add` (atomic,
  no half-write on embedding failure). `col.add` is intentional:
  `entry_id` is timestamp-based with microsecond precision, so a
  duplicate is a same-microsecond clash that should surface rather
  than silently overwrite.
- `mcp_server.tool_add_drawer`: same crash class on the more common
  add-drawer surface (100 KB sanitize cap, 125x `CHUNK_SIZE`).
  Chunked path mirrors Site 3 with batched atomic `col.upsert`,
  per-chunk `parent_drawer_id` + `chunk_index` metadata, and a
  dual-id idempotency probe (last chunk for atomicity, legacy
  `drawer_id` for pre-#1539 single-row backwards-compat). Return
  shape additive: `chunks` always present, `chunk_ids` on the
  chunked path. `tool_get_drawer` / `tool_delete_drawer` against
  the logical handle report "not found" on the chunked path;
  callers iterate `chunk_ids` or query `parent_drawer_id`.

Chunk id width is `:06d` so even a single-digit `chunk_size`
config cannot lex-sort chunks out of order.
2026-05-22 00:21:25 +05:00
Milla J be64371611 Merge pull request #1572 from MemPalace/fix/pr-1555-polish-bot-feedback
fix(extract): polish PR — address bot review feedback on PR #1555
2026-05-21 11:40:47 -07:00
Milla J 60d460b395 Merge pull request #1236 from MemPalace/feat/convo-miner-wing-api-auto-route
feat(convo_miner): auto-route AI tool sessions to wing_api
2026-05-21 09:45:37 -07:00
Milla J b6dc122acb fix(extract): polish PR — address bot review feedback on PR #1555
PR #1555 (format coverage + virtual line numbering) merged with twelve
inline polish comments from Copilot + gemini-code-assist that weren't
load-bearing enough to block the original ship but are real cleanups.
This PR addresses them.

Twelve items in scope; one item (drawer ID delimiter — Copilot #13) is
deferred to its own dedicated PR because it's a breaking schema change
that requires migration design beyond the scope of a polish PR.

## Behavioral fixes (5 items, RED-tested first)

1. **FileNotFoundError vs broken symlink (Copilot #8).** ``extract_text``
   previously mapped every ``FileNotFoundError`` from ``stat()`` to
   ``SKIP_BROKEN_SYMLINK``. That's misleading for the common case of a
   regular file deleted between scan and extract. Now distinguishes:
   ``SKIP_BROKEN_SYMLINK`` only when ``p.is_symlink()`` is true;
   ``SKIP_UNREADABLE`` otherwise.

2. **``file_already_mined`` extract_mode scoping (Copilot #11, #12).**
   Both call sites in ``mine_formats`` and ``_file_chunks_locked`` now
   pass ``extract_mode="format"``. Previously the format miner could
   falsely treat drawers from project / convo miner on the same source
   file as "already mined" (and vice versa). Scopes idempotency to the
   correct drawer subset.

3. **Sentinel skip for transient missing-dep statuses (Copilot #14).**
   New ``_TRANSIENT_MISSING_DEP_STATUSES`` set + ``_register_skip_sentinel_if_appropriate``
   helper. Skip variants like ``SKIP_NO_MARKITDOWN`` /
   ``SKIP_NO_STRIPRTF`` / ``SKIP_MISSING_FORMAT_DEPS`` /
   ``SKIP_NETWORK_TIMEOUT`` no longer write the "already-mined" sentinel.
   Otherwise installing the missing extra later wouldn't trigger a re-mine.

4. **Outer ``except Exception`` in ``mine_formats`` (Gemini #5).** The
   outer try around the loop previously caught only ``KeyboardInterrupt``,
   leaving any setup-time error (e.g., ``scan_formats`` raising) to
   propagate as a bare traceback. Now catches ``Exception`` defensively,
   logs it, prints a partial-progress summary, and lets the ``finally``
   PID-cleanup run. Mirrors miner.py's belt-and-suspenders pattern.

5. **Thread user's ``chunk_size`` / ``chunk_overlap`` / ``min_chunk_size``
   through to ``chunk_text`` (Gemini #3).** ``MempalaceConfig`` was loaded
   only to validate readability; users who tuned their config saw no
   effect in format-mode mining. Now properly threaded.

## Trivial cleanups (5 items)

6. **Path expanduser in ``extract_text`` (Copilot #7).** ``Path(path)`` →
   ``Path(path).expanduser()`` so CLI inputs like ``~/docs/file.pdf``
   resolve correctly.

7. **Path expanduser+resolve in ``scan_formats`` (Copilot #9).** Same
   fix; ``~/docs`` and relative paths now work consistently.

8. **Use resolved ``format_path`` in ``mine_formats`` (Copilot #10).**
   ``scan_formats(format_dir)`` → ``scan_formats(format_path)`` so the
   already-resolved path is used.

9. **``render_with_line_numbers`` type annotation (Copilot #15).**
   ``text: "str | None"`` reflects the documented + tested ``None``
   handling.

10. **Test + docs claims (Copilot #16, #17, #18).** Stale framings
    removed:
    - ``docs/format-coverage.md`` — 14 fringe cases + "see the file for
      the current test inventory" (no more frozen test count).
    - ``tests/test_line_numbers.py`` — drops "proposed for mempalace
      3.3.6" + "run from the proposal directory" references.
    - ``tests/test_format_miner.py`` — drops "MarkItDown is mocked
      throughout" (live integration tests exist) + proposal-directory
      framing.

## Module-level hoists (enables clean test patching)

- ``MempalaceConfig`` (from ``.config``) hoisted from lazy local import
  to module-level so tests can patch ``mempalace.format_miner.MempalaceConfig``.
- ``chunk_text`` (from ``.miner``) hoisted similarly.

Both follow the pattern PR #1565 used for ``compute_hallways_for_wing``.

## Complexity refactor

Extracted ``_print_mine_summary`` from ``mine_formats`` so the orchestrator
stays under the project's ``max-complexity = 25`` ceiling (per
``pyproject.toml [tool.ruff.lint.mccabe]``). Behavior unchanged; pure
extraction.

## Out of scope (intentionally deferred)

- **Drawer ID delimiter collision (Copilot #13)** — ``f"{source_file}{chunk_index}"``
  can theoretically collide (``"/path/a1" + "23"`` == ``"/path/a" + "123"``).
  Fixing this is a breaking schema change to drawer IDs and requires a
  migration plan; will land as its own PR after design.

- The four bot comments that were ALREADY addressed by amendment #3
  before the PR #1555 merge (``_SKIP_DIRS`` dedup, ``scan_formats``
  symlink skip, ``source_mtime`` tracking, hall+entities metadata) —
  no action needed; verified during audit.

## Tests (RED-first)

Six new RED-first tests in ``tests/test_format_miner.py``:

  test_extract_text_nonexistent_regular_file_returns_unreadable_not_broken_symlink
  test_mine_formats_passes_extract_mode_format_to_file_already_mined
  test_mine_formats_does_not_write_sentinel_for_skip_no_markitdown
  test_mine_formats_does_not_write_sentinel_for_skip_missing_format_deps
  test_mine_formats_catches_unexpected_exception_and_prints_summary
  test_mine_formats_threads_chunk_size_from_user_config

All six RED before this commit (failures correctly identified the bugs
they're targeting), all six GREEN after.

One existing test (``test_mine_formats_continues_after_per_file_error``)
updated to patch the new module-level binding
``mempalace.format_miner.chunk_text`` instead of the old
``mempalace.miner.chunk_text`` source location, and to accept the
``**kwargs`` the call now passes through. Behavior unchanged.

## Verification

  pytest -q (full mempalace suite)
    → 2065 passed, 3 skipped, 0 regressions
  ruff check mempalace/format_miner.py mempalace/searcher.py tests/
    → All checks passed!
  ruff format --check ...
    → 4 files already formatted (pinned 0.15.9)
  mine_formats complexity
    → ≤ 25 (under the project ceiling)
2026-05-21 08:41:44 -07:00
Igor Lins e Silva 95caf80fe3 Merge pull request #1555 from MemPalace/feat/3.3.6-line-numbers-and-format-coverage
feat(3.3.6): virtual line numbering + format coverage via --mode extract
2026-05-21 00:39:26 -03:00
Igor Lins e Silva ed80f80cf9 Merge pull request #1565 from MemPalace/feat/entity-tunnels-from-hallways
feat(tunnels): cross-wing entity tunnels derived from hallways
2026-05-20 23:47:58 -03:00
Igor Lins e Silva ad875a8658 Merge pull request #1558 from MemPalace/feat/hallways-within-wing-connectors
feat(hallways): within-wing entity-to-entity connector primitive
2026-05-20 23:46:40 -03:00
Milla J ffc2ae4ae1 fix(extract): surface MissingDependencyException + pull format sub-extras + skip symlink tests on Windows
Addresses PR #1555 review (Igor) — two bugs that block this PR from
merging cleanly:

1. ``mempalace[extract]`` extras did not pull MarkItDown's per-format
   sub-dependencies. A real PDF after ``pip install mempalace[extract]``
   would raise ``MissingDependencyException`` asking for ``markitdown[pdf]``
   — the code then routed that exception through the generic
   ``except Exception`` and surfaced it as ``SKIP_EXTRACTION_ERROR``,
   stripping the actionable signal from the user.

2. ``test_fringe_broken_symlink`` and ``test_scan_formats_skips_symlinks``
   called ``Path.symlink_to()`` unguarded, which raises ``OSError``
   (``WinError 1314``) on Windows test environments without
   ``SeCreateSymbolicLinkPrivilege`` — surfacing as hard test failures
   before any product code ran.

## What this commit does

1. ``mempalace/format_miner.py`` — adds a new
   ``ExtractionStatus.SKIP_MISSING_FORMAT_DEPS`` enum member and a
   matching catch in ``extract_text`` that fires BEFORE the generic
   ``except Exception`` block. The catch matches by exception type name
   (``type(exc).__name__ == "MissingDependencyException"``) so the static
   import surface doesn't change — MarkItDown stays an optional
   dependency.

2. ``pyproject.toml`` — changes ``[extract]`` to include MarkItDown's
   per-format sub-extras:

     "markitdown[docx,pdf,pptx,xlsx]>=0.1.5; python_version >= '3.10'"

   These pull ``pdfminer-six``, ``pdfplumber``, ``mammoth``, ``lxml``,
   ``python-pptx``, ``openpyxl``, ``pandas`` — the deps each per-format
   converter actually needs at runtime. Verified against MarkItDown
   0.1.5's PyPI metadata (Provides-Extra includes ``pdf``, ``docx``,
   ``pptx``, ``xlsx`` among others). ``.epub`` is handled by base
   markitdown (``EpubConverter`` uses ``beautifulsoup4``, a base
   requirement, so no ``[epub]`` extra is needed — and none exists in
   0.1.5). ``.rtf`` is covered by the existing ``striprtf`` entry.
   ``markitdown[all]`` was NOT used because it pulls audio
   (``pydub``, ``speechrecognition``), YouTube, and Azure deps that
   mempalace does not claim support for and that would bloat the
   install for users.

3. ``tests/test_format_miner.py`` — adds a ``_make_symlink_or_skip``
   helper near the top of the file that wraps ``Path.symlink_to()`` in
   try/except ``OSError`` and calls ``pytest.skip(...)`` on failure.
   Refactors both existing symlink tests
   (``test_fringe_broken_symlink``, ``test_scan_formats_skips_symlinks``)
   to use the helper. Behavior is unchanged on macOS/Linux; on Windows
   without symlink privileges the tests skip cleanly instead of
   spuriously failing.

## Tests (RED-first)

Two new RED-first tests:

  test_extract_text_missing_format_dep_returns_distinct_status
      Patches ``_extract_via_markitdown`` to raise a fake exception
      with ``__name__ == "MissingDependencyException"``. Asserts the
      dispatcher returns ``SKIP_MISSING_FORMAT_DEPS``, not
      ``SKIP_EXTRACTION_ERROR``.

  test_pyproject_extract_extra_pulls_markitdown_format_subdeps
      Parses ``pyproject.toml`` (via ``tomllib`` 3.11+ or ``tomli`` on
      3.9/3.10 — the latter is already a base dependency under the
      ``python_version < '3.11'`` marker). Asserts ``[extract]``
      includes ``pdf``, ``docx``, ``pptx``, ``xlsx`` inside SOME
      ``markitdown[...]`` bracketed group.

Both RED before this commit. Both GREEN after.

Also updates ``test_extraction_status_enum_has_all_documented_codes`` to
include the new ``SKIP_MISSING_FORMAT_DEPS`` code in the documented set,
so the enum-completeness doc-test stays honest.

## Verification

  pytest tests/test_format_miner.py::test_extract_text_missing_format_dep_returns_distinct_status
        tests/test_format_miner.py::test_pyproject_extract_extra_pulls_markitdown_format_subdeps
    → 2 passed (RED before, GREEN after)
  pytest tests/test_format_miner.py::test_fringe_broken_symlink
        tests/test_format_miner.py::test_scan_formats_skips_symlinks
    → 2 passed on macOS (Windows behaviour validated by CI test-windows)
  pytest tests/test_format_miner.py
    → 64 passed, 2 skipped, 0 regressions
  pytest -q (full mempalace suite)
    → 2002 passed, 3 skipped, 0 regressions
  ruff check mempalace/format_miner.py tests/test_format_miner.py
    → All checks passed!
  ruff format --check ...
    → 2 files already formatted (pinned 0.15.9)

## Why no OrbStack run this time

OrbStack runs Linux containers, which have unprivileged symlinks. It
cannot reproduce the Windows ``WinError 1314`` failure mode this
amendment fixes. The only authoritative gate for the symlink fix is
GitHub CI's ``test-windows`` job. For the enum + extras changes, both
are pure-Python / install-time concerns the CI test-linux matrix covers
identically to OrbStack. The ``tomli`` fallback in the new pyproject
test was verified against the base dependency declaration in
``pyproject.toml:32``.
2026-05-20 15:26:41 -07:00
Milla J c18879b5bc feat(tunnels): cross-wing entity tunnels derived from hallways
Adds the architectural counterpart to ``compute_topic_tunnels`` that
materializes cross-wing tunnels from the within-wing hallway records
introduced in PR #1558. When an entity (person, project, concept,
interest) has hallways in two wings, an entity tunnel bridges them —
anchored on the entity. This completes the v4 sequence: Wing →
Drawer-entities → Hallway → Tunnel.

Topic tunnels are NOT replaced. Both systems coexist for one release
cycle so existing palaces don't lose tunnels between mines. Deprecation
of topic tunnels is a separate follow-up PR after entity tunnels prove
out in real use.

## What this commit does

1. Adds ``entity_tunnels_for_wing(wing, hallways, label_prefix)`` to
   ``mempalace/palace_graph.py``. Pure function: groups hallway records
   by entity-and-wing, finds entities present in ``wing`` AND ≥1 other
   wing, and emits one ``create_tunnel`` call per (entity, other_wing)
   pair. Uses ``kind="entity"`` and synthetic endpoint room
   ``entity:<name>`` so the new tunnels are distinguishable from
   explicit/topic tunnels at read time but interchangeable with them
   via the standard ``list_tunnels`` / ``follow_tunnels`` API.

2. Adds ``_compute_entity_tunnels_for_wing(wing)`` wrapper to
   ``mempalace/miner.py``. Loads hallway records via
   ``hallways.list_hallways()`` and calls the algorithm. Module-level
   so tests can patch it as ``mempalace.miner._compute_entity_tunnels_for_wing``.

3. Wires the wrapper into ``_mine_impl`` immediately after the existing
   hallway-compute block. Same try/except fault-tolerance pattern as
   the topic-tunnel and hallway blocks — entity-tunnel computation is
   a derived analytic and must never fail a mine.

## Tests (RED-first)

Nine algorithm tests in ``tests/test_palace_graph_tunnels.py``
(new ``TestEntityTunnels`` class):

  test_entity_tunnels_creates_cross_wing_tunnel_for_shared_entity
  test_entity_tunnels_skips_entities_in_only_one_wing
  test_entity_tunnels_counts_entity_in_either_pair_position
  test_entity_tunnels_three_wings_pairwise_from_focus_wing
  test_entity_tunnels_idempotent_on_rerun
  test_entity_tunnels_retrievable_via_list_tunnels
  test_entity_tunnels_empty_hallways_is_noop
  test_entity_tunnels_unknown_wing_is_noop
  test_entity_tunnel_room_does_not_collide_with_literal_room

Two integration tests in ``tests/test_miner.py``:

  test_mine_computes_entity_tunnels_for_wing_post_mine
  test_mine_entity_tunnel_failure_does_not_crash_mine

All 11 RED before this commit (AttributeError on the missing names).
All 11 GREEN after.

## Out of scope (deferred to follow-up PRs)

- ``format_miner.py`` and ``convo_miner.py`` integration: separate PRs
  per the scope discipline used for #1560.
- Deprecating ``_compute_topic_tunnels_for_wing``: separate PR after
  entity tunnels prove out in real use.
- Surfacing ``kind="entity"`` in MCP / search-result UI: not yet
  required by any reader; behaviorally interchangeable with the other
  tunnel kinds today.

## Stacking

This PR stacks on PR #1558 (which introduces the hallway primitive and
its miner integration). Base branch is
``feat/hallways-within-wing-connectors``. When #1558 merges to develop,
GitHub auto-updates this PR's base to ``develop`` and the diff reduces
to just the entity-tunnel additions.

## Verification

  pytest tests/test_palace_graph_tunnels.py::TestEntityTunnels
    → 9 passed (RED before, GREEN after)
  pytest tests/test_miner.py::test_mine_computes_entity_tunnels_for_wing_post_mine
        tests/test_miner.py::test_mine_entity_tunnel_failure_does_not_crash_mine
    → 2 passed (RED before, GREEN after)
  pytest tests/test_palace_graph_tunnels.py
    → 39 passed (no regressions)
  pytest tests/test_miner.py
    → 49 passed (no regressions)
  pytest -q (full mempalace suite)
    → 1949 passed, 1 skipped, 0 regressions
  ruff check mempalace/palace_graph.py mempalace/miner.py tests/
    → All checks passed!
  ruff format --check ...
    → 4 files already formatted (pinned 0.15.9)
2026-05-20 13:55:52 -07:00
Igor Lins e Silva 498b22ffed Merge pull request #1554 from mvalentsev/fix/1455-max-chunks-configurable
fix(miner): configurable + raised MAX_CHUNKS_PER_FILE (#1455)
2026-05-20 17:21:56 -03:00
Igor Lins e Silva 2e29014a28 Merge pull request #1562 from MemPalace/igorls/fix-subprocess-hook-chromadb-deadlock-wi
fix: stale-PID timeout, MCP idle exit, structured errors (#1552)
2026-05-20 13:19:46 -03:00
Igor Lins e SilvaandCopilot 5f0a214016 style: format hook tests
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-20 13:09:15 -03:00
Igor Lins e SilvaandCopilot 1c818d5cdd fix: address stale hook review feedback
- Disable mine and MCP idle timeouts when env values are invalid.
- Use bare-PID slot file mtime as the compatibility timestamp instead of treating old slots as infinitely stale.
- Treat malformed PID-slot timestamps as stale without crashing hook execution.
- Use process-level idle watchdog termination so stale MCP servers actually release file handles.
- Restore tenant-isolation assertions accidentally removed from the KG cache test.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-20 15:59:19 -03:00
Igor Lins e SilvaandCopilot 7bddb65cb4 fix: stale-PID timeout, MCP idle exit, structured errors (#1552)
- Add MEMPALACE_MINE_TIMEOUT_HOURS (default 2h): PID files now record
  '{pid} {unix_timestamp}'; _mine_already_running() treats alive-but-old
  processes as stale, unblocking queued mines after a ChromaDB hang.
  Backward-compatible: bare-PID files (old format) treated as stale.
- Add MEMPALACE_MCP_IDLE_HOURS (default 8h): daemon watchdog thread in
  mcp_server calls sys.exit(0) after the configured idle period, preventing
  accumulation of stale server processes holding ChromaDB/HNSW file handles.
  Set to 0 to disable.
- Enrich _internal_tool_error() with optional exc parameter: adds
  data: {error_class, message} to JSON-RPC error body so callers can
  distinguish lock contention, ChromaDB transients, and segfaults without
  scraping the message string. MineAlreadyRunning handler in tool_sync()
  adds error_class: 'LockHeldByOtherProcess' to the result dict.
- Update miner._cleanup_mine_pid_file() to parse first whitespace token
  as PID (handles both old and new PID file formats).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-05-20 15:31:57 -03:00
Milla J fc13be568a feat(miner): integrate compute_hallways_for_wing into post-mine flow
Wires the hallway primitive (from PR #1558) into the project miner so
that every mine that touches a wing also materializes within-wing
entity hallways for that wing. Without this integration, the hallway
module is dead code — no miner triggers it, no hallways ever land in
~/.mempalace/hallways.json.

## What this commit does

1. Adds a module-level import of ``compute_hallways_for_wing`` from
   ``.hallways`` near the top of ``miner.py``. Module-level (not lazy)
   so tests can patch it as ``mempalace.miner.compute_hallways_for_wing``
   — lazy imports inside a function wouldn't expose the seam.

2. In ``_mine_impl``, immediately after the existing
   ``_compute_topic_tunnels_for_wing(wing)`` post-mine block, adds a
   parallel hallway block. The block:

   - calls ``compute_hallways_for_wing(wing, col=collection)``
   - prints the count if any hallways were materialized
   - wraps the whole thing in try/except so a hallway-compute failure
     is logged + degraded, never propagated. Mirrors the tunnel block's
     fault-tolerance pattern exactly. Hallway computation is a derived
     analytic, not load-bearing for the drawer write that already
     committed above.

## Stacking

This PR stacks on PR #1558 (which introduces the hallway primitive
module). PR #1558 is the prerequisite — without it, the import
``from .hallways import compute_hallways_for_wing`` doesn't resolve.

Base branch for this PR is ``feat/hallways-within-wing-connectors``
(PR #1558's branch). When PR #1558 merges to develop, GitHub will
auto-update this PR's base to develop and the diff will reduce to
just the miner.py and tests/test_miner.py changes.

## Out of scope (deferred to follow-up PRs)

- ``format_miner.py`` integration: lives on a different branch (PR #1555)
  so the parallel call there goes in as a follow-up amendment to that
  branch (or a separate post-merge PR).
- ``convo_miner.py`` integration: convo_miner currently doesn't call
  ``_compute_topic_tunnels_for_wing`` either. Adding both calls is a
  separate concerned PR about convo_miner parity, not just hallways.
- Refactoring ``_compute_topic_tunnels_for_wing`` to BUILD ON hallways
  (rather than computing from raw topic words): the architecturally
  meaningful follow-up that completes the Wing → Drawer-entities →
  Hallway → Tunnel sequence. Real refactor, separate PR.

## Tests

Two new RED-first tests in ``tests/test_miner.py``:

  test_mine_computes_hallways_for_wing_post_mine
      Stubs ``mempalace.miner.compute_hallways_for_wing`` via monkeypatch.
      Runs a real mine into a tmp palace. Asserts the stub was called
      exactly once, with the wing name from mempalace.yaml and a live
      ChromaDB collection (not None).

  test_mine_hallway_failure_does_not_crash_mine
      Stubs the hallway function to raise. Runs a real mine. Asserts
      mine() doesn't propagate, and that the drawer write (which
      happens BEFORE the hallway block) still committed.

Both RED before this commit (AttributeError — module had no attribute
``compute_hallways_for_wing``). Both GREEN after.

## Verification

  ruff check mempalace/miner.py tests/test_miner.py
    → All checks passed!
  ruff format --check ...
    → 2 files already formatted (pinned 0.15.9)
  pytest tests/test_miner.py
    → 47 passed (the 2 new + 45 pre-existing)
  pytest -q (full mempalace suite)
    → 1938 passed, 1 skipped, 0 regressions

Linux CI parity replicated locally via OrbStack containers
(Python 3.9, 3.11, 3.13): 2/2 new integration tests pass, ruff clean
on all three.
2026-05-20 04:08:02 -07:00
Milla J 07807ab8ab feat(hallways): within-wing entity-to-entity connector primitive
Introduces the missing primitive in mempalace's palace architecture.

## The architectural gap

Mempalace's data model today is Wing → Rooms → Tunnels. Tunnels connect
rooms ACROSS wings (via shared topic words). What's missing — and what
this commit adds — is the connector between ENTITIES (people, projects,
concepts, interests) inside one wing.

The intended full architecture is:

    WING → has DRAWERS (each tagged with entities)
            entities → connected to other entities by HALLWAYS
                       (within-wing, built from drawer co-occurrence)
                       hallways → are the primitive
                                   tunnels → use hallways to spawn
                                             cross-wing connections

A hallway is the structural fact of "these two entities travel together
inside this wing." If Aya and Lumi are mentioned together in 47 drawers
across diary, letters, and ideas rooms, that's a hallway between Aya
and Lumi. If Aya and "consciousness" co-occur in 19 drawers, that's
another hallway. Entities aren't only people — projects, concepts, and
interests share the same entity-tag substrate, so a hallway can connect
a person to a concept (Aya ↔ consciousness) just as easily as person to
person (Aya ↔ Lumi).

`palace_graph.py:253`'s docstring currently calls cross-wing tunnels
"hallways" — that conflation is fixed in a separate naming-disambiguation
follow-up. This commit just introduces the primitive.

## Public API

New module `mempalace/hallways.py`:

  compute_hallways_for_wing(wing, col=None, min_count=2) -> list[dict]
      Query drawers for wing. For each drawer with >=2 entities, every
      unordered pair of entities is one co-occurrence; increment a counter
      and track the room. For each entity pair with count >= min_count,
      materialize a hallway record. Persists to ~/.mempalace/hallways.json.
      Returns records created for this wing (records for other wings
      already on disk are preserved).

  list_hallways(wing=None) -> list[dict]
      Return all hallways, optionally filtered by wing.

  delete_hallway(hallway_id) -> bool
      Remove one record by id. Returns True on success.

  _load_hallways() / _save_hallways(state)
      JSON persistence at ~/.mempalace/hallways.json with 0600 perms on
      POSIX and atomic os.replace writes. Mirrors palace_graph's tunnel
      persistence pattern exactly.

## Data model (hallways.json)

```jsonc
{
  "schema_version": 1,
  "hallways": [
    {
      "id": "hallway_<wing>_<entity_a>_<entity_b>_<sha8>",
      "wing": "wing_aya",
      "entity_a": "Aya",
      "entity_b": "Lumi",
      "co_occurrence_count": 47,
      "rooms": ["diary", "letters", "ideas"],
      "label": "Aya ↔ Lumi (co-occur in 47 drawers across 3 rooms: diary, letters, ideas)",
      "created_at": "2026-05-20T03:30:00Z",
      "created_by": "auto"
    }
  ]
}
```

Hallway IDs are deterministic — `sha256(wing + sorted(entity_a, entity_b))`
truncated. Sorted-pair canonicalization means (Aya, Lumi) and (Lumi, Aya)
produce the same id, so re-runs upsert idempotently and the same drawer
appearing as "Aya;Lumi" or "Lumi;Aya" counts as one co-occurrence (not two).

## Tests

17 RED-first tests in `tests/test_hallways.py`:

Storage:
  - _load_hallways returns [] for missing file
  - _load_hallways returns [] for corrupt JSON
  - _save_hallways + _load_hallways round-trips

Algorithm:
  - returns [] for unknown wing
  - returns [] when no drawer has >=2 entities
  - creates hallway for entity pair when threshold met (Aya↔Lumi in 3 drawers)
  - connects person to concept (Aya↔consciousness — entities aren't only people)
  - respects min_count threshold (rejects 2 co-occurrences when min_count=3)
  - creates deterministic id per entity pair (idempotent re-runs)
  - entity pair is symmetric (Aya;Lumi and Lumi;Aya are the same pair)
  - persists to JSON file
  - tracks rooms across co-occurrences (rooms set vs. count distinction)
  - skips sentinel drawers (file_already_mined bookkeeping noise)

Query API:
  - list_hallways returns all when no filter
  - list_hallways filters by wing
  - delete_hallway removes record and persists
  - delete_hallway unknown id returns False

## Verification

  ruff check mempalace/hallways.py tests/test_hallways.py
    → All checks passed!
  ruff format --check ...
    → 2 files already formatted (pinned 0.15.9)
  pytest tests/test_hallways.py
    → 17 passed
  pytest -q (full mempalace suite)
    → 1936 passed, 1 skipped, 0 regressions caused by this PR

Linux CI parity replicated locally via OrbStack containers
(Python 3.9, 3.11, 3.13): 17/17 hallway tests pass, ruff clean on all
three.

## Not in this commit (deferred)

This PR introduces the primitive — the module, the entity-centric API,
the storage, the tests. It does NOT yet:

  - integrate compute_hallways_for_wing() into miner.py / convo_miner.py /
    format_miner.py post-mine flow (next PR — small one-line addition
    per miner alongside the existing _compute_topic_tunnels call)
  - refactor _compute_topic_tunnels_for_wing to BUILD ON hallways
    (the architecturally meaningful follow-up — own PR)
  - rename palace_graph.py:253's misleading docstring that calls
    cross-wing tunnels "hallways" (cleanup PR)
  - expose hallways through the MCP server (when there's a clear UX
    need for it)

Each follow-up is small enough to review in isolation. Shipping the
primitive first lets reviewers think about the API surface separately
from the integration choices.

Per the design sequence: Wing → Drawer-entities → Hallway → Tunnel.
Hallways come first. This is them.
2026-05-20 03:44:05 -07:00
Milla J 0b9c5629b7 fix(miner): mirror init's case-insensitive entity matching in per-drawer tagger
mempalace's init-time entity scanner (`entity_detector.py:276`) already
matches names case-insensitively against corpus content:

    name_line_indices = [i for i, line in enumerate(lines)
                         if name_lower in line.lower()]

That's how a corpus that mentions "aya" all in lowercase still surfaces
"Aya" as a confirmed entity during `mempalace init`.

But the per-drawer tagger in `miner.py:_extract_entities_for_metadata`
was never updated to use the same flag — it walks the same
`known_entities.json` seed list and matches case-sensitively:

    for name in known:
        if re.search(r"(?<!\w)" + re.escape(name) + r"(?!\w)", content):
                                                              # ^ no IGNORECASE
            matched.add(name)

So "Aya" in `known_entities.json` matches "Aya" in drawer content but
silently misses every "aya" / "AYA" mention. Chat transcripts,
voice-typed journals, and any lowercase-style corpus get their drawers
under-tagged in ways that don't show up in init's "we found Aya/Lumi/..."
confirmation list.

## The fix

One regex flag, line 788:

    -if re.search(r"(?<!\w)" + re.escape(name) + r"(?!\w)", content):
    +if re.search(r"(?<!\w)" + re.escape(name) + r"(?!\w)", content,
    +             re.IGNORECASE):

Plus an explanatory comment so the parity with `entity_detector.py:276`
is documented at the call site.

## Empirical proof of the bug (and the fix)

Before this commit, with the actual installed `~/.mempalace/known_entities.json`
containing "Aya", "Lumi":

    Content: "Aya talked to Lumi. Lumi answered."      → tagged 'Aya;Lumi'  ✓
    Content: "aya talked to lumi. lumi answered."      → tagged ''           ✗ MISS
    Content: "AYA mentioned LuMi to ben."              → tagged ''           ✗ MISS

After this commit (same content):

    Content: "Aya talked to Lumi. Lumi answered."      → tagged 'Aya;Lumi'  ✓
    Content: "aya talked to lumi. lumi answered."      → tagged 'Aya;Lumi'  ✓
    Content: "AYA mentioned LuMi to ben."              → tagged 'Aya;Ben;Lumi'  ✓

## Tests

New RED-first test in `tests/test_miner.py`:

    test_entity_metadata_matches_known_names_case_insensitively

Stubs `_load_known_entities` to a controlled `{"Aya", "Lumi"}` seed
and asserts that lowercase + mixed-case mentions both produce the
correctly capitalized tag. Failed on the pre-fix regex (confirmed RED
in CI parity runs across Linux 3.9 / 3.11 / 3.13); passes after the
one-line fix.

## No regressions

Full mempalace test suite: 1920 passed, 1 skipped, 0 failed.
Targeted dependent suites (`test_known_entities_registry`, `test_closets`,
`test_readme_claims`): 123 passed.
ruff check + ruff format clean on pinned 0.15.9.

CI parity replicated locally via Linux containers (Python 3.9, 3.11,
3.13): all three pass the new test and the full `test_miner.py`
sweep (46 tests).

## Why this matters

This is a quiet bug — there's no error message, no failed assertion,
just an empty `entities` field on drawers whose content actually
mentions people. The init phase looks fine ("we detected Aya, Lumi,
Ben") because init does its own case-insensitive scan; mining silently
drops the matches that init's scanner already proved are real.

Personal / journal / chat-style corpora (which use lowercase
conversational style) are the most affected. Code / docs corpora
(which use proper capitalization for names) work either way and saw
no behavior change.
2026-05-20 03:15:45 -07:00
Milla J 5c40cc3d70 feat(format_miner): detect_room + topic tunnel computation (miner.py parity)
Third amendment to PR #1555. Closes the remaining parity gaps between
format_miner and the existing project miner that Aya's palace audit
surfaced after amendment #2.

## What was broken

A smoke test against the just-mined v6 palace confirmed:

  WING: wing_aya
    ROOM: documents             2904 drawers   ← ALL in one room
  Tunnels for wing_aya:         0              ← no cross-wing links

Cause: format_miner hardcoded `room = "documents"` for every drawer and
never called `_compute_topic_tunnels_for_wing()` post-mine. miner.py
calls `detect_room()` per drawer and computes tunnels after the loop.
format_miner did neither.

## What this commit changes

1. Module-level imports of `detect_room`, `load_config`, and
   `_compute_topic_tunnels_for_wing` from `.miner`. Module-level (not
   lazy) so test seams `patch("mempalace.format_miner.detect_room", ...)`
   work — lazy imports inside a function don't expose attributes on the
   module object.

2. `mine_formats()` loads the project's `mempalace.yaml` via
   `load_config()` to get the rooms list. Falls back to a single
   "documents" room if no config exists.

3. The hardcoded `room = "documents"` line is replaced with:
       room = detect_room(filepath, text, rooms, format_path)
   Mirrors miner.py:904 exactly — folder-match → filename-match →
   content-keyword scoring → fallback "general".

4. After the per-file loop completes (in an `else` branch on the outer
   try, so it does NOT run on KeyboardInterrupt), call
   `_compute_topic_tunnels_for_wing(wing)` in try/except. Exact mirror
   of miner.py:1241-1249. Tunnel-compute failures must never fail a mine.

5. Import `sys` at module level (used by the tunnel-compute error path
   for `print(..., file=sys.stderr)`).

## Tests

5 new RED-first tests in `tests/test_format_miner.py`:

  - test_mine_formats_calls_load_config_for_rooms
  - test_mine_formats_calls_detect_room_per_file
  - test_mine_formats_uses_detected_room_in_drawer_metadata
  - test_mine_formats_calls_compute_topic_tunnels_after_loop
  - test_mine_formats_tunnel_failure_does_not_crash_mine

All 5 failed against pre-commit code with AttributeError on missing
module attributes (proof the bug existed). All 5 pass after the
implementation. Total format_miner test count: 62. Total proposal test
count (format_miner + line_numbers): 83. Full mempalace test suite:
2002 passed, 1 skipped (no regressions).

## Verification

  ruff check mempalace/format_miner.py tests/test_format_miner.py
    → All checks passed!
  ruff format --check ...
    → already formatted (pinned ruff 0.15.9)
  pytest tests/test_format_miner.py tests/test_line_numbers.py
    → 83 passed
  pytest -q  (full suite)
    → 2002 passed, 1 skipped

## Not in this commit (deferred)

Within-wing hallway primitives are a separate architectural addition
(separate PR — being designed). The current `_compute_topic_tunnels_for_wing`
matches miner.py exactly but inherits miner.py's gap: it computes
tunnels from raw topic words rather than from hallway primitives. That
refactor lands in its own PR after the hallway primitive ships.
2026-05-20 03:02:39 -07:00
Milla J e362dfa0af feat(format_miner): address bot review — parity fixes with miner.py
Six follow-up fixes addressing gemini-code-assist review on PR #1555.
All medium-priority parity gaps between format_miner.py and the
existing miner.py / convo_miner.py patterns.

1. Reuse palace.SKIP_DIRS instead of duplicating it locally.
   Replaces the local _SKIP_DIRS set with the shared constant so the
   format miner's directory-skip set stays in sync automatically.

2. scan_formats now explicitly skips symlinks. Prevents circular
   links, following symlinks to /dev/urandom, and processing the same
   file via multiple paths. Mirrors miner.py:1068 exactly.

3. mine_formats loads MempalaceConfig at start. The current chunker
   (miner.chunk_text) uses module-level CHUNK_SIZE constants and
   doesn't accept overrides, so the loaded config values aren't yet
   threaded into chunking; the load is wired so future chunk_text
   refactors that accept per-call sizing pick this up automatically.
   Honest scope note: this gives format_miner the same config-load
   stance as miner.py without overreaching what chunk_text supports.

4. source_mtime tracking + check_mtime=True. Each drawer's metadata
   now carries source_mtime, and file_already_mined is called with
   check_mtime=True so an edited PDF/DOCX is correctly re-mined on
   the next pass. Mirrors miner.py semantics exactly. Previously,
   updated documents would have been silently skipped.

5. Per-file try/except wrap + KeyboardInterrupt handling + PID
   cleanup. One malformed file in a mine of thousands no longer
   crashes the whole orchestrator — the error is logged, a counter
   bumps, and the loop continues with the next file. KeyboardInterrupt
   still produces a clean summary on Ctrl-C. _cleanup_mine_pid_file
   is called in a finally block so hook-spawned mines don't leak a
   stale PID file on exit.

6. hall + entities metadata on every drawer. Format-mined drawers
   now carry the same hall (via miner.detect_hall) and entities (via
   miner._extract_entities_for_metadata) tags as project-mined
   drawers. Matches the search-result quality the other miners ship.

Tests: 5 new tests covering the new behaviors:
  - test_scan_formats_skips_symlinks
  - test_mine_formats_uses_check_mtime_true
  - test_mine_formats_records_source_mtime_in_drawer_metadata
  - test_mine_formats_records_hall_in_drawer_metadata
  - test_mine_formats_continues_after_per_file_error

Total: 78 tests passing (was 73). Full mempalace test suite green
modulo one pre-existing flaky test (test_chroma_close_palace_..., a
sqlite-lock-timing flake unrelated to this change — passes 4/4 in
isolation). ruff check + ruff format clean on pinned 0.15.9.
Coverage on mempalace/format_miner.py: 84% (clears the 80% gate).

Refs: PR #1555 review comments from gemini-code-assist[bot]
2026-05-19 19:51:39 -07:00
Milla J 5e3d3dde2e feat(3.3.6): virtual line numbering + format coverage via --mode extract
Two additive features, both following the same read-time-transform pattern:

1. Virtual line numbering — new render_with_line_numbers() and
   extract_line_range() in mempalace/searcher.py. Closet pointers like
   2026-01-18:L55-L72 resolve to drawer slices rendered with [55] through
   [72] line prefixes without modifying any stored content. Lines already
   prefixed with [<digits>] pass through unchanged. Pure functions, no I/O.

2. Format coverage (mempalace mine --mode extract) — new mempalace/format_miner.py
   reads binary office formats and files drawers via the lock + purge +
   upsert pattern convo_miner uses. Source files never modified.

   Per-format transformer routing: MarkItDown 0.1.5 does not actually
   convert .rtf (returns raw control codes unchanged, verified live), so
   .rtf is routed to striprtf which does convert. Other formats stay on
   MarkItDown:

     .pdf .docx .pptx .xlsx .epub  -> MarkItDown
     .rtf                          -> striprtf

   13 fringe cases handled with dedicated ExtractionStatus codes:
   SKIP_NO_MARKITDOWN, SKIP_NO_STRIPRTF, SKIP_TOO_LARGE, SKIP_CLOUD_ONLY,
   SKIP_ENCRYPTED, SKIP_EMPTY, SKIP_PERMISSION, SKIP_BROKEN_SYMLINK,
   SKIP_UNRECOGNIZED, SKIP_EXTRACTION_ERROR, SKIP_NETWORK_TIMEOUT,
   SKIP_UNREADABLE, plus the encoding fallback handled internally.

   Drawers carry ingest_mode=extract + extract_mode=format so they are
   distinguishable from project / convo drawers in the palace.

Both transformers are optional extras: pip install mempalace[extract].
MarkItDown requires Python >= 3.10 (env marker ensures it only installs
where supported; Python 3.9 users still get RTF coverage via striprtf).

Tests: 94 new (21 line-numbering + 73 format-miner, including 9
mine_formats orchestrator tests). Full mempalace suite still green
(1992 passed locally). Coverage 86% on mempalace/format_miner.py.
ruff check + ruff format clean on the pinned 0.15.9.

Live verification: mempalace mine --mode extract on a directory with 2
RTFs + 1 PDF produced 90 drawers correctly; mempalace search found
content from both file types in the resulting wing.

Documented limits (out of scope for 3.3.6): custom PDF parsers, OCR on
scanned PDFs, DRM-locked files, pathological corrupt files. These get
reported via skip codes and skipped.

Refs: docs/format-coverage.md, docs/virtual-line-numbering.md
2026-05-19 19:15:40 -07:00
mvalentsevandDavid Glidden 5decc0c83b fix(miner): configurable + raised MAX_CHUNKS_PER_FILE (#1455)
Default raised from 500 to 50_000 so legitimate long-form content
(novels, scholarly editions) is not silently dropped on a typical
literary corpus. Cap configurable via MEMPALACE_MAX_CHUNKS_PER_FILE
env var or --max-chunks-per-file CLI flag; sentinel 0 disables.
Negative or non-int values from either source emit a stderr warning
and fall back to the default.

Separate counter (files_skipped_chunk_cap) and summary line surface
chunk-cap drops independently of the residual already-filed bucket;
counter fires under --dry-run too so a corpus audit shows the same
signal without writing. Skip notice routed to stderr alongside the
existing symlink-skip line.

ONNX bad_alloc protection preserved architecturally:
DRAWER_UPSERT_BATCH_SIZE=1000 bounds per-ONNX-call exposure
regardless of per-file cap. A 50_000-chunk file produces 50 forward
passes of 1_000 chunks each, identical to what 500-chunk files
already produced.

process_file return extended to a 3-tuple (drawers, room,
skip_reason); skip_reason is None on non-chunk-cap paths and
"chunk_cap" when the per-file cap triggered. Internal callers
(_mine_impl, benchmarks/mine_bench.py, tests) updated.

Closes #1455

Co-Authored-By: David Glidden <d@davidglidden.eu>
2026-05-20 02:20:29 +05:00
yonefive71 7550de27a0 fix(graph): repo-format tests, %s for tunnel paths, single config per call
Three changes addressing #1469 CI red + Gemini perf review:

1. ruff format (0.4.x) on tests/test_closets.py and
   tests/test_palace_graph_tunnels.py — the lint job pins
   ruff>=0.4.0,<0.5 and was flagging format drift.

2. Replace %r with '%s' in legacy / corrupt tunnel-file warnings.
   On Windows %r escapes backslashes in repr, so
   test_load_tunnels_warns_on_orphaned_legacy_file's
   'str(legacy) in caplog.text' assertion was failing on
   test-windows even though the warning was firing.

3. Address gemini-code-assist review on #1469: pass a single
   MempalaceConfig() through _get_tunnel_file / _load_tunnels /
   _save_tunnels per create_tunnel call instead of each helper
   re-instantiating its own (which re-reads mempalace.yaml from
   disk). Helpers keep their config=None defaults so external
   callers and existing tests are unaffected.
2026-05-19 05:12:48 +00:00
Igor Lins e Silva 1b94f4efb4 Merge pull request #1538 from mvalentsev/fix/1534-paragraph-chunk-size-enforcement
fix(convo_miner): enforce CHUNK_SIZE in paragraph chunker (#1534)
2026-05-18 21:32:40 -03:00
Igor Lins e Silva 100539d2ca Merge pull request #1543 from fatkobra/fix/1443-mine-slot-placeholder
Fix: Write placeholder PID when claiming mine slot
2026-05-18 21:32:21 -03:00