2dad4bbd7b3596d9df3b9c5217ef08b6fa052d72
520
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
2dad4bbd7b |
Merge pull request #1422 from YuanYiZheXue/fix/lone-surrogate-clean
fix(mcp): sanitize lone surrogates before ChromaDB ops (#1235) |
||
|
|
382409bd6b |
Merge origin/develop into feat/benchmark-multilingual
Resolves conflicts in CHANGELOG.md and pyproject.toml by combining the multilingual-embedder additions (huggingface_hub/tokenizers/numpy core deps, [multilingual] alias, Features section) with develop's additions (python-dateutil core dep, [extract] extra, tunnel Bug Fixes and Internal sections). Prepares PR #1483 for merge into v3.3.6. |
||
|
|
3e001921eb |
refactor(mcp): move lone-surrogate strip into shared sanitizers (#1235)
Push the surrogate-cleaning behaviour from per-call-site _clean() helpers into sanitize_content, sanitize_kg_value, and sanitize_query so every caller (existing and future) gets the fix automatically. New MCP tools no longer need to remember to call a separate helper. - Add strip_lone_surrogates() in mempalace/config.py as the single regex-based implementation (one U+FFFD per surrogate, not three). - Wire it into sanitize_content and sanitize_kg_value. - Wire it into sanitize_query so embedding lookups can't crash either. - Drop the _clean() helper from mcp_server.py and the per-site calls; retain a direct strip_lone_surrogates() for source_file/added_by metadata which doesn't route through any sanitizer. - Move the ONNX model-cache patch out of the test module and into conftest.py so it's session-scoped instead of duplicated locally. - Update tests to assert one U+FFFD per surrogate and exercise the sanitizer-level entry points directly. |
||
|
|
997ec1b22b |
test(mcp): add lone-surrogate sanitisation tests (issue #1235)
Add tests/test_clean_lone_surrogates.py covering: Unit tests (TestCleanLoneSurrogates, 11 cases): - _clean() passes normal ASCII and CJK strings unchanged - lone surrogates (high/low, single/multiple) are replaced with U+FFFD - real emoji (\U0001f600 astral code points) pass through unchanged - empty string, all-surrogate string, SHA-256-hash-after-clean - the specific \udcad surrogate observed in WorkBuddy production logs Integration tests (TestLoneSurrogateCleaning, 6 cases): - tool_add_drawer: surrogate in content and in metadata fields - tool_check_duplicate: surrogate in query - tool_search: surrogate in search query - tool_update_drawer: surrogate in updated content - tool_diary_write: surrogate in diary entry Fix test environment issue: - conftest.py redirects HOME to a temp dir, causing chromadb's ONNXMiniLM_L6_V2 to look for its ONNX model in the wrong location and trigger a 79 MB network download on every run. - Fix: at module import time, recover the real USERPROFILE from conftest._original_env and patch ONNXMiniLM_L6_V2.DOWNLOAD_PATH before any ChromaDB collection fixture is invoked. |
||
|
|
ce1ee26e11 |
Merge pull request #1548 from mvalentsev/fix/1537-validate-fts5-after-mine
fix(mine): validate FTS5 at end of mine (#1537) |
||
|
|
8a813d2807 |
Merge pull request #711 from sha2fiddy/feature/494-silent-mode-auto-save-toggle
feat: add hooks.auto_save config toggle (silent mode) |
||
|
|
0ecf0e5f06 |
fix(mine): validate FTS5 at end of mine (#1537)
Wires _validate_palace_fts5_after_mine into all three mine entry points so corrupted-FTS5 palaces cannot silently exit 0 from any of them: - _mine_impl (mempalace/miner.py) — project file miner - mine_convos (mempalace/convo_miner.py) — conversation exports - mine_formats (mempalace/format_miner.py) — binary office documents via --mode extract, introduced on develop by #1555 (3.3.6 release) between this PR's open date and its rebase cmd_mine surfaces MineValidationError as exit 1 + the same print_sqlite_integrity_abort banner cmd_repair already prints, appended with a mine-specific stderr note that hedges attribution (quick_check cannot tell pre-existing corruption from corruption this mine produced). 17 tests in tests/test_miner_fts5_validation.py cover the helper, the three call sites, dry-run / KeyboardInterrupt skip semantics, and the MineValidationError constructor invariants. Closes #1537. Co-authored-by: Caleb Wells <15988028+calebcwells@users.noreply.github.com> |
||
|
|
318940797f |
feat(entity): filter common English content words from entity detection
Common English content words ("Code", "Brutal", "Phase", "Line", "Note",
"Planning", "Chat", ...) frequently appear capitalized at sentence
start, in headings, or in markdown — and the existing regex-based entity
detector treats every capitalized word appearing 3+ times as a proper
noun candidate. This produces false positives that pollute
known_entities.json, the per-drawer entities metadata, and closet
pointer entity lists.
Adds a curated list of ~1000 common English content words (nouns,
adjectives, verbs — no proper nouns, no known product names) at
mempalace/data/coca_content_words.json, derived from the COCA top-2000
frequency list and from observed false positives in real palaces.
The list is loaded once at module import via _get_coca_filter() (cached
frozenset) and consulted at three sites:
- entity_detector.extract_candidates — init-time entity detection
- palace.build_closet_lines — closet pointer construction
- miner._extract_entities_for_metadata — per-drawer entity tagging
Matching is case-insensitive: the candidate is lowercased before
membership check, so "Code", "CODE", and "code" are all blocked.
Only single-word candidates are filtered; multi-word phrases (e.g.
"Claude Code") are NOT filtered, so legitimate compound names continue
to be detected. A future PR will add a known-systems lexicon that
explicitly protects compound names — for now, the multi-word pass
provides the implicit protection.
Adds 8 new tests in tests/test_entity_detector.py covering:
- "Code" appearing 5x is filtered
- All known false positives from real palaces are filtered
- Case-insensitive matching works
- Real names ("Aya", "Riley") are NOT filtered
- Multi-word phrases like "Claude Code" are NOT filtered
- The data file ships in the wheel with the expected schema
- The data file contains every known false positive
Note: this filter prevents NEW false positives at detection time. It
does NOT retroactively scrub existing entries from a user's
~/.mempalace/known_entities.json. Stale entries from previous runs
must be removed via a future entity-management CLI (planned).
Verification: pytest 2191 passed / 3 skipped / 0 failed on macOS.
ruff check + format --check both clean. Cross-platform via
pip install -e ".[dev]" on python:3.9-slim / 3.11-slim / 3.13-slim
containers, each running 65 entity-detector tests including the new
data-file-presence test that imports from the installed package.
End-to-end pipeline test mining a deterministic corpus into an
isolated palace confirmed entity metadata is empty (no false
positives surface in real chromadb-backed storage).
|
||
|
|
4a3b48c56a | style: ruff format tests/test_hooks_cli.py with CI-pinned ruff 0.4.x | ||
|
|
c2a61d85ed | fix(hooks): retarget auto_save toggle at silent-save path after #1021 rebase | ||
|
|
ada25872a1 |
fix: correct precompact test assertion + ruff format tests
test_precompact_hook_enabled_by_default asserted
result["decision"] == "block", but hook_precompact has never emitted
decision — it mines synchronously and returns {}. Assertion was
copy-pasted from the stop-hook test. Fix to assert result == {} with
_mine_sync mocked so the test verifies the real contract (enabled →
mine + pass through) without actually mining.
Plus ruff format on 6 test files the CI pin flagged.
|
||
|
|
8fc06b96e2 |
fix: address review — enrich block reasons, clean up tests
- Add parenthetical hints to block reasons so AI knows what each tool saves (session summary, quotes/decisions/code) - Remove dead config file creation from test_stop_hook_disabled_by_config - Add missing test for MEMPALACE_HOOKS_AUTO_SAVE=no env var - Replace bare except: with except Exception: in shell scripts |
||
|
|
49a8c868fa |
feat: add hooks.auto_save config toggle and shorten block reasons
Add a clean opt-out for auto-save hook blocking (closes #494). - New `hooks.auto_save` config option (default: true) in ~/.mempalace/config.json and MEMPALACE_HOOKS_AUTO_SAVE env var - When disabled, stop and precompact hooks pass through without blocking - Shorten block reason text from 6-line instructions to single-line prompts — reduces UI noise while keeping tool names explicit - Both Python (hooks_cli.py) and standalone shell scripts respect the toggle via config file or env var |
||
|
|
f6edb7d1ab |
Merge pull request #1584 from MemPalace/feat/closet-date-line-pointer-emission
feat(closets): Tier 6a — date+line locators with content-date hierarchy |
||
|
|
65f24b3bdd |
fix(closets): address PR #1584 Igor review — wire + harden Tier 6a
Igor's review on PR #1584 (2026-05-22) flagged four issues: 1. The feature wasn't wired into any production caller — the new ``drawer_metas`` kwarg on ``build_closet_lines`` had no real consumer in ``miner.py`` / ``diary_ingest.py``, so the 4-segment pointer form only existed in tests. Real palaces kept emitting the legacy 3-segment shape. 2. ``_extract_content_date`` hallucinated dates on benign inputs. ``dateutil.parser.parse(fuzzy=True)`` would accept anything with digits and return a plausible-looking but wrong date — ``Version 3.3.6`` → ``2006-03-03``, ``Tested with 1000 drawers`` → ``1000-05-22``, ``tmp_random_file_5`` → ``2026-05-05``, etc. Mtime almost never got reached because fuzzy returned *something* from filename or body first. Bad dates were silently persisted to ChromaDB. 3. ``python-dateutil`` was an undeclared dependency, available only transitively via ``chromadb → kubernetes → python-dateutil``. Not a contract — upstream kubernetes has been trending toward stdlib-only. 4. Two-digit-year disambiguation (70 → 19xx / 00-69 → 20xx) had no test pinning the boundary. This commit addresses all four. ## Changes ### Issue 2 — kill the hallucination (the load-bearing fix) ``mempalace/miner.py``: - New ``_VALID_DATE_RE`` gate. Three accepted shapes (all require a 4-digit year explicitly): 1. Numeric YYYY-MM-DD with ``[-/.\\s]`` separators (covers ISO and space-normalized filenames) 2. Month-name + day + year ("November 8 2024", "Nov 8 2024") 3. Day + month-name + year ("8 November 2024") Partial dates ("2024-06", "April 6", "notes.2024") are DELIBERATELY rejected — without all three components we'd pad from today's date, which is hallucination not extraction. - ``_try_filename_date`` and ``_try_content_body_date`` now run the gate BEFORE invoking dateutil, and pass ``fuzzy=True`` is REMOVED. Dateutil only runs in strict mode on a substring the gate already validated. ### Issue 1 — wire the feature into production ``mempalace/miner.py`` batched-upsert path: - Accumulate ``batch_metas`` across all batches into ``all_metas`` - Pass ``drawer_metas=all_metas`` to ``build_closet_lines`` End-to-end integration test added that mines a real file with a filename-derived content date and asserts the produced closet documents contain the 4-segment pointer with that date. ``diary_ingest.py`` is left as-is for this PR. Diary entries are entry-keyed, not chunk-keyed — they carry no natural ``line_start`` / ``line_end``, so the 4-segment form would return None for them regardless. Wiring the diary path can land cleanly in a follow-up once Tier 6a gains an "approximate line range for diary entries" story. ### Issue 3 — declare the dateutil dependency ``pyproject.toml``: add ``python-dateutil>=2.8`` to ``[project].dependencies``. One-line change; cheaper than the stdlib-only refactor alternative and keeps the natural-language recall surface. ### Issue 4 — pin the two-digit-year boundary Four new tests cover the 1969/1970/1999/2000 corner cases of the slash-date locale heuristic. ## Tests added (RED-first then GREEN) tests/test_miner.py::TestExtractContentDate (11 new): Hallucination cases verbatim from Igor's review: - test_no_hallucination_junk_filename_with_trailing_digit - test_no_hallucination_untitled_with_index - test_no_hallucination_filename_year_only - test_no_hallucination_filename_year_and_month_only - test_no_hallucination_content_with_issue_number - test_no_hallucination_content_with_count - test_no_hallucination_content_with_version_number Two-digit-year boundary cases: - test_two_digit_year_69_is_2069 - test_two_digit_year_70_is_1970 - test_two_digit_year_99_is_1999 - test_two_digit_year_00_is_2000 tests/test_closets.py::TestMinerClosetRebuild (1 new): - test_production_miner_emits_4_segment_pointers_with_content_date (regression for Issue #1 — real ``mine()`` end-to-end produces 4-segment closet pointers via the new ``drawer_metas`` wiring) ## Verification pytest tests/test_miner.py tests/test_closets.py tests/test_format_miner.py tests/test_palace.py → 242 passed, 2 skipped, 0 regressions pytest tests/test_miner.py::TestExtractContentDate → 26 passed (15 prior + 11 new) pytest tests/test_closets.py::TestMinerClosetRebuild → end-to-end wiring test GREEN Sanity (Igor's exact repros): "tmp_random_file_5" → None (was: 2026-05-05) "untitled-1" → None (was: 2026-05-01) "notes.2024.md" → None (was: 2024-05-22) "2024-06.md" → None (was: 2024-06-22) "Bug fix for issue 42 in module 7" → None (was: 2042-07-22) "Tested with 1000 drawers" → None (was: 1000-05-22) "Version 3.3.6 released" → None (was: 2006-03-03) Real dates still extract correctly: "2024-11-08.md" → "2024-11-08" "April-6th-2011-notes.md" → "2011-04-06" "Nov-8-2024.md" → "2024-11-08" ruff check + ruff format --check (pinned 0.15.9) → All checks passed OrbStack triple-Linux verify (Py 3.9 / 3.11 / 3.13) → all targeted tests pass; python-dateutil installs explicitly via the new declared dependency. |
||
|
|
c89c077d7e |
Merge pull request #1578 from MemPalace/feat/hallway-tunnel-decay-dynamics
feat(dynamics): Hebbian potentiation + Ebbinghaus decay for halls + tunnels |
||
|
|
5633406e5e |
Merge pull request #1585 from milla-jovovich/fix/hooks-cli-shallow-path-guard
fix(hooks_cli): guard parents[3] access against shallow filesystem paths |
||
|
|
4b7edd9aea |
Merge branch 'develop' into docs/codex-discoverability
Bring docs PR up to date with develop so CI re-runs against the current pin set. Co-Authored-By: Igor Lins e Silva <4753812+igorls@users.noreply.github.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
749d43359a |
style(hooks_cli): address PR #1580 gemini review — LBYL + side_effect
Two medium-priority gemini-code-assist comments on PR #1580 both recommend more-idiomatic Python: 1. **Production code (``mempalace/hooks_cli.py::_mempalace_python``)** — replace ``try/except IndexError`` with ``if len(parents) > N:`` look-before-you-leap checks. Exception handling for bounded-integer index lookups is a code smell in Python; LBYL makes the depth check explicit and removes exception overhead. Same behavior, clearer intent. Before (EAFP, ~12 lines + comment): try: venv_bin = resolved.parents[3] / "bin" / "python" if venv_bin.is_file(): return str(venv_bin) except IndexError: pass After (LBYL, ~5 lines): if len(parents) > 3: venv_bin = parents[3] / "bin" / "python" if venv_bin.is_file(): return str(venv_bin) 2. **Test code (``tests/test_hooks_cli.py``)** — replace the lambda + generator-throw hack with ``MagicMock.side_effect = get_item``, where ``get_item`` is a normal function that returns the editable-install path for index 1 and raises ``IndexError`` for any other index (defensive against a future regression that drops the LBYL length check). Standard ``side_effect`` mocking pattern. Before: fake_parents.__getitem__ = lambda self, idx: ( RealPath("/work/mempalace") if idx == 1 else (_ for _ in ()).throw(IndexError(idx)) ) After: def get_item(idx): if idx == 1: return RealPath("/work/mempalace") raise IndexError(idx) fake_parents.__len__.return_value = 3 fake_parents.__getitem__.side_effect = get_item Also added ``__len__`` mock so the LBYL length check in production sees the simulated shallow path correctly. ## Verification pytest tests/test_hooks_cli.py → 110 passed, 1 skipped (same as PR #1580 baseline; regression test for shallow-path crash still GREEN) ruff check + ruff format --check (pinned 0.15.9) → All checks passed; 2 files already formatted |
||
|
|
5482262749 |
fix(hooks_cli): guard parents[3] access against shallow filesystem paths
``_mempalace_python()`` in ``mempalace/hooks_cli.py`` uses ``Path(__file__).resolve().parents[3]`` to locate the venv Python interpreter in the standard install layout ``<venv>/lib/pythonX.Y/site-packages/mempalace/hooks_cli.py``. When the package lives at a shallow filesystem path — Docker containers mounting at ``/work``, ``/opt/app``, minimal-prefix production installs — ``parents`` has fewer than 4 elements and the index raises ``IndexError`` instead of falling through to the editable-install branch. The crash was caught by OrbStack-based triple-Python CI verification on PR #1579: 16 tests in ``test_hooks_cli.py`` failed identically on Linux 3.9 / 3.11 / 3.13 with the same ``IndexError: 3`` from ``pathlib._PathBase.parents.__getitem__`` — and verified pre-existing on develop tip in the same container. The bug never surfaces in GitHub Actions CI runners (their workdir at ``/home/runner/work/mempalace/mempalace`` has plenty of parent directories) but it surfaces immediately for anyone: - running mempalace in editable mode inside a Docker dev container - shipping mempalace as part of an OCI image where the install prefix is ``/app`` or ``/opt/<name>`` - using OrbStack / Colima / podman-machine for cross-version verification ## The fix Wrap each ``parents[N]`` access in ``try/except IndexError`` so the helper falls through to the next strategy (editable-install → ``sys.executable``) instead of crashing the hook. Both ``parents[3]`` AND ``parents[1]`` are guarded — the latter is defensive against extreme cases like a file at root (``/file.py``, parents=[/]) — same class of bug. ## Test added (RED-first, then GREEN) tests/test_hooks_cli.py::test_mempalace_python_handles_shallow_path_without_crashing Mocks ``Path(__file__).resolve()`` so ``parents[3]`` raises ``IndexError`` and ``parents[1]`` returns a real shallow path (``/work/mempalace``). Pre-commit: function raises ``IndexError: 3``. Post-commit: function returns a valid Python interpreter path (either editable-venv if present, otherwise ``sys.executable``). ## Verification pytest tests/test_hooks_cli.py → 110 passed, 1 skipped on macOS (the existing run) → 110 passed, 1 skipped on Linux 3.9 / 3.11 / 3.13 (OrbStack) — was 16 failed, 94 passed before this commit ruff check + ruff format --check (pinned 0.15.9) → All checks passed; 2 files already formatted |
||
|
|
67c6ab59f2 | style: apply ruff format with pinned 0.15.9 (CI lint) | ||
|
|
05da803f71 |
feat(closets): Tier 6a — date+line locators with content-date hierarchy
Closet pointer lines gain a 4th pipe-separated segment of shape
``YYYY-MM-DD:Lstart-Lend`` so retrieval can jump straight to the right
span in the source file instead of opening the whole drawer. The date
is selected via a content-aware hierarchy so legacy ingests (where
``filed_at`` is misleading) still produce honest pointers when the
content itself carries a date.
The legacy 3-segment format remains the fallback for drawers without
the new metadata, so existing palaces keep working without a re-mine.
## Pointer shape
Before:
topic|entities|→drawer_ids
After (when drawer metadata carries line range + a date):
topic|entities|2024-11-08:L42-L78|→drawer_ids
Backward compat: when ``drawer_metas`` is not passed, or the first
meta lacks ``line_start`` / ``line_end``, ``build_closet_lines`` emits
the legacy 3-segment form unchanged.
## Date-source hierarchy (NEW)
A new helper ``mempalace.miner._extract_content_date(source_file, content)``
selects a date once per file and stores it as ``content_date`` on every
drawer mined from that file. Priority order, first match wins:
1. **Filename** — ISO regex on stem, then dateutil fuzzy parse for
natural-language formats. Handles ``2024-11-08.md``,
``April-6th-2011-notes.md``, ``Nov-8-2024.md``, etc.
2. **YAML frontmatter** — looks at ``date`` / ``created`` /
``published`` fields. yaml.safe_load already parses ISO dates as
``datetime.date`` objects; dateutil handles string values.
3. **Content body, first ~10 lines** —
a. ISO regex (``YYYY-MM-DD`` / ``YYYY/MM/DD`` / ``YYYY.MM.DD``)
b. Slash dates with locale auto-disambiguation: if any
first-number > 12 appears anywhere in the head, lock the
file's locale to DD/MM (the only consistent reading);
otherwise default to US MM/DD for two-digit-year compactness.
c. dateutil fuzzy parse for natural-language
(``November 8, 2024``, ``8th November 2024``, etc.)
4. **Filesystem mtime** — ``os.path.getmtime`` formatted as
``YYYY-MM-DD``.
5. **None** — caller (``_build_drawer_metadata``) leaves
``content_date`` absent; ``build_closet_lines`` falls back to the
``filed_at`` ingestion timestamp's date portion.
``_build_date_line_segment`` in ``palace.py`` now prefers ``content_date``
when present, otherwise uses ``filed_at[:10]``. This is the load-bearing
change that makes Tier 6a meaningful for legacy content — a Nov 8 2024
transcript mined today now emits ``2024-11-08:L42-L78`` instead of
``2026-05-22:L42-L78``.
## Changes
1. ``mempalace/miner.py::chunk_text`` — each chunk dict now also carries
``line_start`` / ``line_end`` (1-indexed line numbers in the stripped
source). Computed via ``content[:offset].count("\n") + 1`` —
approximate locator accurate to ±1 line at chunk boundaries.
2. ``mempalace/miner.py::_extract_content_date`` (NEW) + helpers
(``_try_filename_date``, ``_try_frontmatter_date``,
``_try_content_body_date``, ``_try_mtime_date``). Pure functions, no
I/O beyond what's named. Uses lazy imports of ``dateutil`` and
``yaml`` to keep top-level import surface unchanged.
3. ``mempalace/miner.py::_build_drawer_metadata`` — gains
``line_start`` / ``line_end`` / ``content_date`` optional kwargs.
When provided, stored in drawer metadata; when omitted, keys are
absent from the dict — backward compatible.
4. ``mempalace/miner.py`` batched-upsert path — calls
``_extract_content_date(source_file, content)`` ONCE per file (not
once per chunk) and passes the result through to every chunk's
``_build_drawer_metadata`` call.
5. ``mempalace/format_miner.py::_file_chunks_locked`` — gains a
``content`` param so the format-miner path can also run
``_extract_content_date`` on the full extracted text. Plumbs
``content_date`` into every batched drawer's metadata. Caller passes
``text`` (the full extracted markdown).
6. ``mempalace/palace.py::build_closet_lines`` — accepts optional
``drawer_metas`` (parallel to ``drawer_ids``). When present and the
first meta has both ``line_start`` and ``line_end``, the new
segment is spliced in; otherwise legacy 3-segment form. Helper
``_build_date_line_segment`` prefers ``content_date`` over
``filed_at`` for the date portion.
## Tests added (RED-first, then GREEN)
tests/test_miner.py::TestChunkTextLineRanges (4 tests)
tests/test_miner.py::TestBuildDrawerMetadataLineRange (2 tests)
tests/test_miner.py::TestExtractContentDate (15 tests)
- filename ISO / natural-language / compact
- YAML frontmatter date / created
- Claude session preamble
- ISO + natural-language content body
- Slash-date locale disambiguation (DD/MM lock vs default MM/DD)
- Priority order (filename > frontmatter > content > mtime)
- mtime fallback
- Graceful None returns for missing file / empty content
tests/test_closets.py::TestBuildClosetLines (5 new tests)
- includes_date_line_segment_when_metas_provided
- falls_back_to_3_segment_format_when_metas_missing (compat)
- falls_back_to_3_segment_format_when_metas_lack_line_keys (compat)
- date_segment_uses_filed_at_date_portion_only
- content_date_preferred_over_filed_at (new — Tier 6a hierarchy)
26 new tests total. All RED before this commit, all GREEN after.
## Verification
pytest tests/test_miner.py::TestChunkTextLineRanges
tests/test_miner.py::TestBuildDrawerMetadataLineRange
tests/test_miner.py::TestExtractContentDate
tests/test_closets.py::TestBuildClosetLines
→ 32 passed
pytest tests/test_miner.py tests/test_closets.py
tests/test_format_miner.py tests/test_palace.py
→ 230 passed, 2 skipped, 0 regressions in directly-affected files
ruff check + ruff format --check
→ All checks passed, 5 files already formatted
End-to-end smoke test on a YAML-frontmatter file (date: 2024-11-08,
mined "today"):
content_date extracted: 2024-11-08
chunks produced: 1
drawer meta: content_date=2024-11-08, filed_at='2026-05-22...', lines=1-15
closet pointer:
"conversation with lumi about brands of dog food|Filler;Lumi;Aya|2024-11-08:L1-L15|→drawer_a,drawer_b"
→ content_date correctly preferred over filed_at end-to-end.
OrbStack triple-Linux verify (Py 3.9 / 3.11 / 3.13)
→ 32 targeted Tier 6a + content-date tests pass on each.
## Out of scope (deliberate)
- No schema migration of existing drawers. Pre-Tier-6a drawers lack
``line_start`` / ``content_date`` and silently fall back to the
legacy 3-segment pointer shape.
- No re-mine required. Users keep existing palaces; new files mined
after this lands carry the new fields.
- No drawer ID change (Task #80, separate concern).
- No exact-quote line positioning within a chunk (future tier).
- No NLP-based date extraction (only regex + dateutil fuzzy).
- Linguistic-similarity / concept-recognition retrieval (L5 zettal
layer) is the natural next step but lives in its own PR family.
|
||
|
|
f1aa409a94 |
fix(dynamics): address PR #1578 gemini review — canonicalize lookup key + idiomatic dict update
Addresses both inline comments left by gemini-code-assist on PR #1578. ## What this commit does 1. **HIGH PRIORITY (gemini, hallways.py:260) — canonicalize dynamics preservation lookup key.** The previous code built the lookup key as ``(h.get("entity_a"), h.get("entity_b"))``, but the ``pair_counts`` dict and ``_hallway_id`` both use sorted entity pairs for symmetric IDs. A persisted record with reversed entity order (Lumi as entity_a, Aya as entity_b) would silently miss the lookup and lose its accumulated dynamics on recompute. Now the lookup canonicalizes the same way ID generation does: key = tuple(sorted([h.get("entity_a"), h.get("entity_b")])) Failure mode is rare in practice (the algorithm normally writes sorted records), but any legacy data, manual JSON edit, or future import path that violates the ordering would have silently broken L7 preservation. Belt-and-suspenders fix. 2. **MEDIUM PRIORITY (gemini, palace_graph.py:512) — idiomatic field preservation in tunnel dedup path.** Refactored the manual for-loop into a dict-update with a comprehension. Field list lives in one place; future schema expansion can't accidentally drop a field. No behavior change. ## Test added (RED-first) test_recompute_preserves_dynamics_when_existing_record_has_reversed_entity_order Pre-populates the hallway store with a record whose ``entity_a`` / ``entity_b`` are in REVERSED (non-sorted) order, bumps its strength / stability / access_count to non-default values, then runs ``compute_hallways_for_wing``. Asserts the recomputation preserves the bumped values — proving the canonical lookup matches against arbitrarily-ordered persisted entity pairs. Reproduces the bug class gemini flagged: RED before this commit (strength=1.0 instead of 4.2), GREEN after. ## Verification pytest tests/test_hallways.py::TestHallwayDynamicsIntegration tests/test_palace_graph_tunnels.py::TestTunnelDynamicsIntegration tests/test_dynamics.py → 32 passed (was 31; added 1 new for the reverse-order case) pytest -q (full mempalace suite) → 2112 passed, 3 skipped, 0 regressions ruff check mempalace/dynamics.py mempalace/hallways.py mempalace/palace_graph.py tests/ → All checks passed! ruff format --check ... → 3 files already formatted (pinned 0.15.9) |
||
|
|
3b6c7986c9 |
Merge branch 'develop' into fix/tunnel-file-config-and-endpoint-validation
Resolves conflicts from 60-commit divergence: - tests/test_closets.py: assertion reformat — kept develop's ruff-format- preferred multi-line shape (functionally identical). - tests/test_palace_graph_tunnels.py: both branches added a new test class at end-of-file (this PR's TestTunnelFileFollowsConfig + develop's TestEntityTunnels from #1564). Kept both, no overlap. Local: ruff check / format pass; full pytest suite passes (2113 passed, 3 skipped). Co-Authored-By: Igor Lins e Silva <4753812+igorls@users.noreply.github.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
e97b3621ca |
Merge branch 'develop' into fix/convos-mine-concurrency
Resolves conflicts from 128-commit divergence: - mempalace/convo_miner.py imports: kept both `mine_palace_lock` (this PR) and `prefetch_mined_set` (develop). - mempalace/convo_miner.py docstring: kept this PR's lock-wrapping description, added a one-line pointer to the chunking-config section whose body now lives in `_mine_convos_impl`. - mempalace/convo_miner.py body: develop placed `cfg_chunk_size` / `cfg_min_chunk_size` setup inline in `mine_convos`. This PR factored the body into `_mine_convos_impl`, so the inline setup would have left `cfg_chunk_size` referenced-but-undefined inside the impl. Moved the `MempalaceConfig()` setup into `_mine_convos_impl` so the variables are in scope where they're used. - tests/test_convo_miner.py: kept both additive test sets (lock concurrency from this PR + wing_api auto-routing from develop). Local: ruff check / format pass; full pytest suite passes (2103 passed, 3 skipped). Co-Authored-By: Igor Lins e Silva <4753812+igorls@users.noreply.github.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
a2ba1ccb05 |
feat(dynamics): Hebbian potentiation + Ebbinghaus decay for halls + tunnels
Adds the L7 living-connection dynamics layer for the hallway primitive (PR #1558) and the tunnel primitive (PR #1565). Connections now carry strength, stability, last_activated, and access_count fields; pure-math helpers in a new ``mempalace.dynamics`` module update these via Hebbian potentiation (Hebb 1949) on co-access and Ebbinghaus exponential decay (Ebbinghaus 1885) on time-since-use, with the Cepeda spacing effect (Cepeda et al. 2006) growing stability when reinforcement is distributed rather than massed. Existing palaces continue to work unchanged — the new fields populate via safe defaults (strength=1.0, stability=1.0, last_activated=created_at, access_count=0) lazily on any potentiate/decay call. No migration step required. No schema break. Old records work; new records ship with full L7 state from creation. ## What this commit does 1. New module ``mempalace/dynamics.py`` — pure math, no I/O, no chromadb. Three public helpers: - ``initialize_dynamics_fields(connection, *, now=None)`` Backfill helper. Populates missing strength / stability / last_activated / access_count fields. Existing fields are NOT overwritten. Safe for pre-L7 records. - ``potentiate(connection, *, increment=POTENTIATION_INCREMENT, now=None)`` Hebbian strengthening on a co-access event. Increments strength (capped at MAX_STRENGTH), updates last_activated, increments access_count. Grows stability by STABILITY_INCREMENT ONLY when the gap since the prior activation is at least SPACED_INTERVAL_HOURS — implements the Cepeda spacing effect (rapid bursts don't build durability; distributed practice does). - ``apply_decay(connection, *, now=None)`` Ebbinghaus exponential decay. New strength = old * exp(-days_since / stability), floored at STRENGTH_FLOOR (0.05) so connections never reach zero — only become dim. Idempotent at the same instant. Higher stability = slower decay. 2. Integration in ``mempalace/hallways.py:compute_hallways_for_wing`` — before persisting the new hallway list for a wing, looks up the existing wing's hallways and builds a (entity_a, entity_b) → dynamics lookup. Each new record gets the preserved dynamics if the entity pair existed before, OR default dynamics via ``initialize_dynamics_fields`` if it's a brand-new pair. Without this preservation, every mine would wipe accumulated connection weights — defeating the whole L7 layer. 3. Integration in ``mempalace/palace_graph.py:create_tunnel`` — the existing dedup-on-canonical-id path now also preserves the four dynamics fields from the existing record onto the recreated tunnel (in addition to the already-preserved ``created_at``). Brand-new tunnels and legacy records (created before L7) get default dynamics via ``initialize_dynamics_fields``. ## Tests (RED-first) Twenty-five new tests in ``tests/test_dynamics.py`` covering the pure math: field initialization defaults, potentiation strength increment + cap, last_activated update, access_count increment, spaced vs. rapid reinforcement (Cepeda spacing effect), decay rate (exp(-days/stability)), floor clamping, idempotency at same instant, higher-stability slower-decay behavior, no-time-passed no-op, backfill on missing fields, mutate-and- return-same-dict chaining. Plus integration scenarios: potentiation restoring decayed strength, repeated spaced reinforcement growing stability monotonically, burst reinforcement not growing stability. Six new integration tests: test_new_hallway_record_carries_all_dynamics_fields test_recompute_preserves_accumulated_strength test_recompute_initializes_dynamics_for_brand_new_pairs test_new_tunnel_carries_all_dynamics_fields test_recreate_tunnel_preserves_accumulated_dynamics test_recreate_tunnel_initializes_dynamics_for_legacy_records All RED-first (KeyError: 'strength' before the integration commits). All GREEN after. ## Backward compatibility Pre-existing palaces work unchanged: - Old hallway records with no dynamics fields are still readable; the next ``compute_hallways_for_wing`` recompute populates the missing fields via ``initialize_dynamics_fields``. - Old tunnel records similarly; the next ``create_tunnel`` recreate (or any future call passing the record through ``initialize_dynamics_fields``) populates the missing fields. - A ``test_recreate_tunnel_initializes_dynamics_for_legacy_records`` test pins this contract explicitly. No forced migration. No schema break. Old records work; new behavior kicks in on first touch. ## What this does NOT do (intentionally scoped out) - Spreading activation at search time — that's a search-layer change; bigger PR; separate work. - Lazy decay-on-read in ``list_hallways`` / ``list_tunnels`` — the math is here; wiring it into the read path is a follow-up. - Config exposure of the tunable constants — STRENGTH_FLOOR, MAX_STRENGTH, POTENTIATION_INCREMENT etc. are hardcoded module-level for v1. If empirical palace data shows the defaults need tuning, expose via MempalaceConfig in a follow-up. - Decay-aware ranking in search — the strength field is now there; the search ranker doesn't yet use it. Follow-up. ## Verification pytest tests/test_dynamics.py → 25 passed (RED before this commit; GREEN after) pytest tests/test_hallways.py::TestHallwayDynamicsIntegration → 3 passed (RED before; GREEN after) pytest tests/test_palace_graph_tunnels.py::TestTunnelDynamicsIntegration → 3 passed (RED before; GREEN after) pytest -q (full mempalace suite) → 2111 passed, 3 skipped, 0 regressions ruff check mempalace/dynamics.py mempalace/hallways.py mempalace/palace_graph.py tests/ → All checks passed! ruff format --check ... → 6 files already formatted (pinned 0.15.9) ## Research grounding - Hebb, D. O. (1949). The Organization of Behavior. Wiley. → "Neurons that fire together, wire together" → potentiate() - Ebbinghaus, H. (1885). Über das Gedächtnis. → Exponential forgetting curve → apply_decay() - Cepeda, N. J., Pashler, H., Vul, E., Wixted, J. T., & Rohrer, D. (2006). Distributed practice in verbal recall tasks: A review and quantitative synthesis. Psychological Bulletin, 132(3), 354-380. → Spacing effect → stability growth on spaced reinforcement |
||
|
|
6658a4d11f |
fix(audit): chunk and batch content before embedding upsert (#1539)
Four sites passed content to `collection.upsert(documents=[...])` or
`collection.add(documents=[...])` without per-drawer size cap, hitting
the same `RuntimeError: Invalid buffer size` crash class from the
embedding model's attention buffer:
- `general_extractor.extract_memories`: post-classification slicer
that propagates `memory_type` to every slice. New `chunk_size`
parameter defaulting to `DEFAULT_CHUNK_SIZE` and resolved from
`MempalaceConfig.chunk_size` by the caller.
- `diary_ingest.ingest_diaries`: per-entry drawers via existing
`_split_entries`, with character-chunk fallback for any single
entry larger than `chunk_size`. New `_diary_drawer_id_entry`
helper carries (entry_idx, entry_chunk_idx). `chunk_index` in
metadata is a global counter across the file so
`searcher._expand_with_neighbors` stitches sibling chunks
regardless of entry boundary. Upsert is batched atomic per file
(one call carrying every entry/chunk) so a mid-pass embedding
failure cannot half-write the day. Auto-purge on full rebuild
deletes prior-pass drawers via `where={"source_file": ...}`,
which also migrates pre-#1539 legacy `drawer_diary_` IDs as a
side effect of normal use.
- `mcp_server.tool_diary_write`: split oversized entries into
bounded per-chunk drawers via a single batched `col.add` (atomic,
no half-write on embedding failure). `col.add` is intentional:
`entry_id` is timestamp-based with microsecond precision, so a
duplicate is a same-microsecond clash that should surface rather
than silently overwrite.
- `mcp_server.tool_add_drawer`: same crash class on the more common
add-drawer surface (100 KB sanitize cap, 125x `CHUNK_SIZE`).
Chunked path mirrors Site 3 with batched atomic `col.upsert`,
per-chunk `parent_drawer_id` + `chunk_index` metadata, and a
dual-id idempotency probe (last chunk for atomicity, legacy
`drawer_id` for pre-#1539 single-row backwards-compat). Return
shape additive: `chunks` always present, `chunk_ids` on the
chunked path. `tool_get_drawer` / `tool_delete_drawer` against
the logical handle report "not found" on the chunked path;
callers iterate `chunk_ids` or query `parent_drawer_id`.
Chunk id width is `:06d` so even a single-digit `chunk_size`
config cannot lex-sort chunks out of order.
|
||
|
|
be64371611 |
Merge pull request #1572 from MemPalace/fix/pr-1555-polish-bot-feedback
fix(extract): polish PR — address bot review feedback on PR #1555 |
||
|
|
60d460b395 |
Merge pull request #1236 from MemPalace/feat/convo-miner-wing-api-auto-route
feat(convo_miner): auto-route AI tool sessions to wing_api |
||
|
|
b6dc122acb |
fix(extract): polish PR — address bot review feedback on PR #1555
PR #1555 (format coverage + virtual line numbering) merged with twelve inline polish comments from Copilot + gemini-code-assist that weren't load-bearing enough to block the original ship but are real cleanups. This PR addresses them. Twelve items in scope; one item (drawer ID delimiter — Copilot #13) is deferred to its own dedicated PR because it's a breaking schema change that requires migration design beyond the scope of a polish PR. ## Behavioral fixes (5 items, RED-tested first) 1. **FileNotFoundError vs broken symlink (Copilot #8).** ``extract_text`` previously mapped every ``FileNotFoundError`` from ``stat()`` to ``SKIP_BROKEN_SYMLINK``. That's misleading for the common case of a regular file deleted between scan and extract. Now distinguishes: ``SKIP_BROKEN_SYMLINK`` only when ``p.is_symlink()`` is true; ``SKIP_UNREADABLE`` otherwise. 2. **``file_already_mined`` extract_mode scoping (Copilot #11, #12).** Both call sites in ``mine_formats`` and ``_file_chunks_locked`` now pass ``extract_mode="format"``. Previously the format miner could falsely treat drawers from project / convo miner on the same source file as "already mined" (and vice versa). Scopes idempotency to the correct drawer subset. 3. **Sentinel skip for transient missing-dep statuses (Copilot #14).** New ``_TRANSIENT_MISSING_DEP_STATUSES`` set + ``_register_skip_sentinel_if_appropriate`` helper. Skip variants like ``SKIP_NO_MARKITDOWN`` / ``SKIP_NO_STRIPRTF`` / ``SKIP_MISSING_FORMAT_DEPS`` / ``SKIP_NETWORK_TIMEOUT`` no longer write the "already-mined" sentinel. Otherwise installing the missing extra later wouldn't trigger a re-mine. 4. **Outer ``except Exception`` in ``mine_formats`` (Gemini #5).** The outer try around the loop previously caught only ``KeyboardInterrupt``, leaving any setup-time error (e.g., ``scan_formats`` raising) to propagate as a bare traceback. Now catches ``Exception`` defensively, logs it, prints a partial-progress summary, and lets the ``finally`` PID-cleanup run. Mirrors miner.py's belt-and-suspenders pattern. 5. **Thread user's ``chunk_size`` / ``chunk_overlap`` / ``min_chunk_size`` through to ``chunk_text`` (Gemini #3).** ``MempalaceConfig`` was loaded only to validate readability; users who tuned their config saw no effect in format-mode mining. Now properly threaded. ## Trivial cleanups (5 items) 6. **Path expanduser in ``extract_text`` (Copilot #7).** ``Path(path)`` → ``Path(path).expanduser()`` so CLI inputs like ``~/docs/file.pdf`` resolve correctly. 7. **Path expanduser+resolve in ``scan_formats`` (Copilot #9).** Same fix; ``~/docs`` and relative paths now work consistently. 8. **Use resolved ``format_path`` in ``mine_formats`` (Copilot #10).** ``scan_formats(format_dir)`` → ``scan_formats(format_path)`` so the already-resolved path is used. 9. **``render_with_line_numbers`` type annotation (Copilot #15).** ``text: "str | None"`` reflects the documented + tested ``None`` handling. 10. **Test + docs claims (Copilot #16, #17, #18).** Stale framings removed: - ``docs/format-coverage.md`` — 14 fringe cases + "see the file for the current test inventory" (no more frozen test count). - ``tests/test_line_numbers.py`` — drops "proposed for mempalace 3.3.6" + "run from the proposal directory" references. - ``tests/test_format_miner.py`` — drops "MarkItDown is mocked throughout" (live integration tests exist) + proposal-directory framing. ## Module-level hoists (enables clean test patching) - ``MempalaceConfig`` (from ``.config``) hoisted from lazy local import to module-level so tests can patch ``mempalace.format_miner.MempalaceConfig``. - ``chunk_text`` (from ``.miner``) hoisted similarly. Both follow the pattern PR #1565 used for ``compute_hallways_for_wing``. ## Complexity refactor Extracted ``_print_mine_summary`` from ``mine_formats`` so the orchestrator stays under the project's ``max-complexity = 25`` ceiling (per ``pyproject.toml [tool.ruff.lint.mccabe]``). Behavior unchanged; pure extraction. ## Out of scope (intentionally deferred) - **Drawer ID delimiter collision (Copilot #13)** — ``f"{source_file}{chunk_index}"`` can theoretically collide (``"/path/a1" + "23"`` == ``"/path/a" + "123"``). Fixing this is a breaking schema change to drawer IDs and requires a migration plan; will land as its own PR after design. - The four bot comments that were ALREADY addressed by amendment #3 before the PR #1555 merge (``_SKIP_DIRS`` dedup, ``scan_formats`` symlink skip, ``source_mtime`` tracking, hall+entities metadata) — no action needed; verified during audit. ## Tests (RED-first) Six new RED-first tests in ``tests/test_format_miner.py``: test_extract_text_nonexistent_regular_file_returns_unreadable_not_broken_symlink test_mine_formats_passes_extract_mode_format_to_file_already_mined test_mine_formats_does_not_write_sentinel_for_skip_no_markitdown test_mine_formats_does_not_write_sentinel_for_skip_missing_format_deps test_mine_formats_catches_unexpected_exception_and_prints_summary test_mine_formats_threads_chunk_size_from_user_config All six RED before this commit (failures correctly identified the bugs they're targeting), all six GREEN after. One existing test (``test_mine_formats_continues_after_per_file_error``) updated to patch the new module-level binding ``mempalace.format_miner.chunk_text`` instead of the old ``mempalace.miner.chunk_text`` source location, and to accept the ``**kwargs`` the call now passes through. Behavior unchanged. ## Verification pytest -q (full mempalace suite) → 2065 passed, 3 skipped, 0 regressions ruff check mempalace/format_miner.py mempalace/searcher.py tests/ → All checks passed! ruff format --check ... → 4 files already formatted (pinned 0.15.9) mine_formats complexity → ≤ 25 (under the project ceiling) |
||
|
|
95caf80fe3 |
Merge pull request #1555 from MemPalace/feat/3.3.6-line-numbers-and-format-coverage
feat(3.3.6): virtual line numbering + format coverage via --mode extract |
||
|
|
ed80f80cf9 |
Merge pull request #1565 from MemPalace/feat/entity-tunnels-from-hallways
feat(tunnels): cross-wing entity tunnels derived from hallways |
||
|
|
ad875a8658 |
Merge pull request #1558 from MemPalace/feat/hallways-within-wing-connectors
feat(hallways): within-wing entity-to-entity connector primitive |
||
|
|
ffc2ae4ae1 |
fix(extract): surface MissingDependencyException + pull format sub-extras + skip symlink tests on Windows
Addresses PR #1555 review (Igor) — two bugs that block this PR from merging cleanly: 1. ``mempalace[extract]`` extras did not pull MarkItDown's per-format sub-dependencies. A real PDF after ``pip install mempalace[extract]`` would raise ``MissingDependencyException`` asking for ``markitdown[pdf]`` — the code then routed that exception through the generic ``except Exception`` and surfaced it as ``SKIP_EXTRACTION_ERROR``, stripping the actionable signal from the user. 2. ``test_fringe_broken_symlink`` and ``test_scan_formats_skips_symlinks`` called ``Path.symlink_to()`` unguarded, which raises ``OSError`` (``WinError 1314``) on Windows test environments without ``SeCreateSymbolicLinkPrivilege`` — surfacing as hard test failures before any product code ran. ## What this commit does 1. ``mempalace/format_miner.py`` — adds a new ``ExtractionStatus.SKIP_MISSING_FORMAT_DEPS`` enum member and a matching catch in ``extract_text`` that fires BEFORE the generic ``except Exception`` block. The catch matches by exception type name (``type(exc).__name__ == "MissingDependencyException"``) so the static import surface doesn't change — MarkItDown stays an optional dependency. 2. ``pyproject.toml`` — changes ``[extract]`` to include MarkItDown's per-format sub-extras: "markitdown[docx,pdf,pptx,xlsx]>=0.1.5; python_version >= '3.10'" These pull ``pdfminer-six``, ``pdfplumber``, ``mammoth``, ``lxml``, ``python-pptx``, ``openpyxl``, ``pandas`` — the deps each per-format converter actually needs at runtime. Verified against MarkItDown 0.1.5's PyPI metadata (Provides-Extra includes ``pdf``, ``docx``, ``pptx``, ``xlsx`` among others). ``.epub`` is handled by base markitdown (``EpubConverter`` uses ``beautifulsoup4``, a base requirement, so no ``[epub]`` extra is needed — and none exists in 0.1.5). ``.rtf`` is covered by the existing ``striprtf`` entry. ``markitdown[all]`` was NOT used because it pulls audio (``pydub``, ``speechrecognition``), YouTube, and Azure deps that mempalace does not claim support for and that would bloat the install for users. 3. ``tests/test_format_miner.py`` — adds a ``_make_symlink_or_skip`` helper near the top of the file that wraps ``Path.symlink_to()`` in try/except ``OSError`` and calls ``pytest.skip(...)`` on failure. Refactors both existing symlink tests (``test_fringe_broken_symlink``, ``test_scan_formats_skips_symlinks``) to use the helper. Behavior is unchanged on macOS/Linux; on Windows without symlink privileges the tests skip cleanly instead of spuriously failing. ## Tests (RED-first) Two new RED-first tests: test_extract_text_missing_format_dep_returns_distinct_status Patches ``_extract_via_markitdown`` to raise a fake exception with ``__name__ == "MissingDependencyException"``. Asserts the dispatcher returns ``SKIP_MISSING_FORMAT_DEPS``, not ``SKIP_EXTRACTION_ERROR``. test_pyproject_extract_extra_pulls_markitdown_format_subdeps Parses ``pyproject.toml`` (via ``tomllib`` 3.11+ or ``tomli`` on 3.9/3.10 — the latter is already a base dependency under the ``python_version < '3.11'`` marker). Asserts ``[extract]`` includes ``pdf``, ``docx``, ``pptx``, ``xlsx`` inside SOME ``markitdown[...]`` bracketed group. Both RED before this commit. Both GREEN after. Also updates ``test_extraction_status_enum_has_all_documented_codes`` to include the new ``SKIP_MISSING_FORMAT_DEPS`` code in the documented set, so the enum-completeness doc-test stays honest. ## Verification pytest tests/test_format_miner.py::test_extract_text_missing_format_dep_returns_distinct_status tests/test_format_miner.py::test_pyproject_extract_extra_pulls_markitdown_format_subdeps → 2 passed (RED before, GREEN after) pytest tests/test_format_miner.py::test_fringe_broken_symlink tests/test_format_miner.py::test_scan_formats_skips_symlinks → 2 passed on macOS (Windows behaviour validated by CI test-windows) pytest tests/test_format_miner.py → 64 passed, 2 skipped, 0 regressions pytest -q (full mempalace suite) → 2002 passed, 3 skipped, 0 regressions ruff check mempalace/format_miner.py tests/test_format_miner.py → All checks passed! ruff format --check ... → 2 files already formatted (pinned 0.15.9) ## Why no OrbStack run this time OrbStack runs Linux containers, which have unprivileged symlinks. It cannot reproduce the Windows ``WinError 1314`` failure mode this amendment fixes. The only authoritative gate for the symlink fix is GitHub CI's ``test-windows`` job. For the enum + extras changes, both are pure-Python / install-time concerns the CI test-linux matrix covers identically to OrbStack. The ``tomli`` fallback in the new pyproject test was verified against the base dependency declaration in ``pyproject.toml:32``. |
||
|
|
c18879b5bc |
feat(tunnels): cross-wing entity tunnels derived from hallways
Adds the architectural counterpart to ``compute_topic_tunnels`` that materializes cross-wing tunnels from the within-wing hallway records introduced in PR #1558. When an entity (person, project, concept, interest) has hallways in two wings, an entity tunnel bridges them — anchored on the entity. This completes the v4 sequence: Wing → Drawer-entities → Hallway → Tunnel. Topic tunnels are NOT replaced. Both systems coexist for one release cycle so existing palaces don't lose tunnels between mines. Deprecation of topic tunnels is a separate follow-up PR after entity tunnels prove out in real use. ## What this commit does 1. Adds ``entity_tunnels_for_wing(wing, hallways, label_prefix)`` to ``mempalace/palace_graph.py``. Pure function: groups hallway records by entity-and-wing, finds entities present in ``wing`` AND ≥1 other wing, and emits one ``create_tunnel`` call per (entity, other_wing) pair. Uses ``kind="entity"`` and synthetic endpoint room ``entity:<name>`` so the new tunnels are distinguishable from explicit/topic tunnels at read time but interchangeable with them via the standard ``list_tunnels`` / ``follow_tunnels`` API. 2. Adds ``_compute_entity_tunnels_for_wing(wing)`` wrapper to ``mempalace/miner.py``. Loads hallway records via ``hallways.list_hallways()`` and calls the algorithm. Module-level so tests can patch it as ``mempalace.miner._compute_entity_tunnels_for_wing``. 3. Wires the wrapper into ``_mine_impl`` immediately after the existing hallway-compute block. Same try/except fault-tolerance pattern as the topic-tunnel and hallway blocks — entity-tunnel computation is a derived analytic and must never fail a mine. ## Tests (RED-first) Nine algorithm tests in ``tests/test_palace_graph_tunnels.py`` (new ``TestEntityTunnels`` class): test_entity_tunnels_creates_cross_wing_tunnel_for_shared_entity test_entity_tunnels_skips_entities_in_only_one_wing test_entity_tunnels_counts_entity_in_either_pair_position test_entity_tunnels_three_wings_pairwise_from_focus_wing test_entity_tunnels_idempotent_on_rerun test_entity_tunnels_retrievable_via_list_tunnels test_entity_tunnels_empty_hallways_is_noop test_entity_tunnels_unknown_wing_is_noop test_entity_tunnel_room_does_not_collide_with_literal_room Two integration tests in ``tests/test_miner.py``: test_mine_computes_entity_tunnels_for_wing_post_mine test_mine_entity_tunnel_failure_does_not_crash_mine All 11 RED before this commit (AttributeError on the missing names). All 11 GREEN after. ## Out of scope (deferred to follow-up PRs) - ``format_miner.py`` and ``convo_miner.py`` integration: separate PRs per the scope discipline used for #1560. - Deprecating ``_compute_topic_tunnels_for_wing``: separate PR after entity tunnels prove out in real use. - Surfacing ``kind="entity"`` in MCP / search-result UI: not yet required by any reader; behaviorally interchangeable with the other tunnel kinds today. ## Stacking This PR stacks on PR #1558 (which introduces the hallway primitive and its miner integration). Base branch is ``feat/hallways-within-wing-connectors``. When #1558 merges to develop, GitHub auto-updates this PR's base to ``develop`` and the diff reduces to just the entity-tunnel additions. ## Verification pytest tests/test_palace_graph_tunnels.py::TestEntityTunnels → 9 passed (RED before, GREEN after) pytest tests/test_miner.py::test_mine_computes_entity_tunnels_for_wing_post_mine tests/test_miner.py::test_mine_entity_tunnel_failure_does_not_crash_mine → 2 passed (RED before, GREEN after) pytest tests/test_palace_graph_tunnels.py → 39 passed (no regressions) pytest tests/test_miner.py → 49 passed (no regressions) pytest -q (full mempalace suite) → 1949 passed, 1 skipped, 0 regressions ruff check mempalace/palace_graph.py mempalace/miner.py tests/ → All checks passed! ruff format --check ... → 4 files already formatted (pinned 0.15.9) |
||
|
|
498b22ffed |
Merge pull request #1554 from mvalentsev/fix/1455-max-chunks-configurable
fix(miner): configurable + raised MAX_CHUNKS_PER_FILE (#1455) |
||
|
|
2e29014a28 |
Merge pull request #1562 from MemPalace/igorls/fix-subprocess-hook-chromadb-deadlock-wi
fix: stale-PID timeout, MCP idle exit, structured errors (#1552) |
||
|
|
5f0a214016 |
style: format hook tests
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
|
1c818d5cdd |
fix: address stale hook review feedback
- Disable mine and MCP idle timeouts when env values are invalid. - Use bare-PID slot file mtime as the compatibility timestamp instead of treating old slots as infinitely stale. - Treat malformed PID-slot timestamps as stale without crashing hook execution. - Use process-level idle watchdog termination so stale MCP servers actually release file handles. - Restore tenant-isolation assertions accidentally removed from the KG cache test. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
|
7bddb65cb4 |
fix: stale-PID timeout, MCP idle exit, structured errors (#1552)
- Add MEMPALACE_MINE_TIMEOUT_HOURS (default 2h): PID files now record
'{pid} {unix_timestamp}'; _mine_already_running() treats alive-but-old
processes as stale, unblocking queued mines after a ChromaDB hang.
Backward-compatible: bare-PID files (old format) treated as stale.
- Add MEMPALACE_MCP_IDLE_HOURS (default 8h): daemon watchdog thread in
mcp_server calls sys.exit(0) after the configured idle period, preventing
accumulation of stale server processes holding ChromaDB/HNSW file handles.
Set to 0 to disable.
- Enrich _internal_tool_error() with optional exc parameter: adds
data: {error_class, message} to JSON-RPC error body so callers can
distinguish lock contention, ChromaDB transients, and segfaults without
scraping the message string. MineAlreadyRunning handler in tool_sync()
adds error_class: 'LockHeldByOtherProcess' to the result dict.
- Update miner._cleanup_mine_pid_file() to parse first whitespace token
as PID (handles both old and new PID file formats).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
||
|
|
fc13be568a |
feat(miner): integrate compute_hallways_for_wing into post-mine flow
Wires the hallway primitive (from PR #1558) into the project miner so that every mine that touches a wing also materializes within-wing entity hallways for that wing. Without this integration, the hallway module is dead code — no miner triggers it, no hallways ever land in ~/.mempalace/hallways.json. ## What this commit does 1. Adds a module-level import of ``compute_hallways_for_wing`` from ``.hallways`` near the top of ``miner.py``. Module-level (not lazy) so tests can patch it as ``mempalace.miner.compute_hallways_for_wing`` — lazy imports inside a function wouldn't expose the seam. 2. In ``_mine_impl``, immediately after the existing ``_compute_topic_tunnels_for_wing(wing)`` post-mine block, adds a parallel hallway block. The block: - calls ``compute_hallways_for_wing(wing, col=collection)`` - prints the count if any hallways were materialized - wraps the whole thing in try/except so a hallway-compute failure is logged + degraded, never propagated. Mirrors the tunnel block's fault-tolerance pattern exactly. Hallway computation is a derived analytic, not load-bearing for the drawer write that already committed above. ## Stacking This PR stacks on PR #1558 (which introduces the hallway primitive module). PR #1558 is the prerequisite — without it, the import ``from .hallways import compute_hallways_for_wing`` doesn't resolve. Base branch for this PR is ``feat/hallways-within-wing-connectors`` (PR #1558's branch). When PR #1558 merges to develop, GitHub will auto-update this PR's base to develop and the diff will reduce to just the miner.py and tests/test_miner.py changes. ## Out of scope (deferred to follow-up PRs) - ``format_miner.py`` integration: lives on a different branch (PR #1555) so the parallel call there goes in as a follow-up amendment to that branch (or a separate post-merge PR). - ``convo_miner.py`` integration: convo_miner currently doesn't call ``_compute_topic_tunnels_for_wing`` either. Adding both calls is a separate concerned PR about convo_miner parity, not just hallways. - Refactoring ``_compute_topic_tunnels_for_wing`` to BUILD ON hallways (rather than computing from raw topic words): the architecturally meaningful follow-up that completes the Wing → Drawer-entities → Hallway → Tunnel sequence. Real refactor, separate PR. ## Tests Two new RED-first tests in ``tests/test_miner.py``: test_mine_computes_hallways_for_wing_post_mine Stubs ``mempalace.miner.compute_hallways_for_wing`` via monkeypatch. Runs a real mine into a tmp palace. Asserts the stub was called exactly once, with the wing name from mempalace.yaml and a live ChromaDB collection (not None). test_mine_hallway_failure_does_not_crash_mine Stubs the hallway function to raise. Runs a real mine. Asserts mine() doesn't propagate, and that the drawer write (which happens BEFORE the hallway block) still committed. Both RED before this commit (AttributeError — module had no attribute ``compute_hallways_for_wing``). Both GREEN after. ## Verification ruff check mempalace/miner.py tests/test_miner.py → All checks passed! ruff format --check ... → 2 files already formatted (pinned 0.15.9) pytest tests/test_miner.py → 47 passed (the 2 new + 45 pre-existing) pytest -q (full mempalace suite) → 1938 passed, 1 skipped, 0 regressions Linux CI parity replicated locally via OrbStack containers (Python 3.9, 3.11, 3.13): 2/2 new integration tests pass, ruff clean on all three. |
||
|
|
07807ab8ab |
feat(hallways): within-wing entity-to-entity connector primitive
Introduces the missing primitive in mempalace's palace architecture.
## The architectural gap
Mempalace's data model today is Wing → Rooms → Tunnels. Tunnels connect
rooms ACROSS wings (via shared topic words). What's missing — and what
this commit adds — is the connector between ENTITIES (people, projects,
concepts, interests) inside one wing.
The intended full architecture is:
WING → has DRAWERS (each tagged with entities)
entities → connected to other entities by HALLWAYS
(within-wing, built from drawer co-occurrence)
hallways → are the primitive
tunnels → use hallways to spawn
cross-wing connections
A hallway is the structural fact of "these two entities travel together
inside this wing." If Aya and Lumi are mentioned together in 47 drawers
across diary, letters, and ideas rooms, that's a hallway between Aya
and Lumi. If Aya and "consciousness" co-occur in 19 drawers, that's
another hallway. Entities aren't only people — projects, concepts, and
interests share the same entity-tag substrate, so a hallway can connect
a person to a concept (Aya ↔ consciousness) just as easily as person to
person (Aya ↔ Lumi).
`palace_graph.py:253`'s docstring currently calls cross-wing tunnels
"hallways" — that conflation is fixed in a separate naming-disambiguation
follow-up. This commit just introduces the primitive.
## Public API
New module `mempalace/hallways.py`:
compute_hallways_for_wing(wing, col=None, min_count=2) -> list[dict]
Query drawers for wing. For each drawer with >=2 entities, every
unordered pair of entities is one co-occurrence; increment a counter
and track the room. For each entity pair with count >= min_count,
materialize a hallway record. Persists to ~/.mempalace/hallways.json.
Returns records created for this wing (records for other wings
already on disk are preserved).
list_hallways(wing=None) -> list[dict]
Return all hallways, optionally filtered by wing.
delete_hallway(hallway_id) -> bool
Remove one record by id. Returns True on success.
_load_hallways() / _save_hallways(state)
JSON persistence at ~/.mempalace/hallways.json with 0600 perms on
POSIX and atomic os.replace writes. Mirrors palace_graph's tunnel
persistence pattern exactly.
## Data model (hallways.json)
```jsonc
{
"schema_version": 1,
"hallways": [
{
"id": "hallway_<wing>_<entity_a>_<entity_b>_<sha8>",
"wing": "wing_aya",
"entity_a": "Aya",
"entity_b": "Lumi",
"co_occurrence_count": 47,
"rooms": ["diary", "letters", "ideas"],
"label": "Aya ↔ Lumi (co-occur in 47 drawers across 3 rooms: diary, letters, ideas)",
"created_at": "2026-05-20T03:30:00Z",
"created_by": "auto"
}
]
}
```
Hallway IDs are deterministic — `sha256(wing + sorted(entity_a, entity_b))`
truncated. Sorted-pair canonicalization means (Aya, Lumi) and (Lumi, Aya)
produce the same id, so re-runs upsert idempotently and the same drawer
appearing as "Aya;Lumi" or "Lumi;Aya" counts as one co-occurrence (not two).
## Tests
17 RED-first tests in `tests/test_hallways.py`:
Storage:
- _load_hallways returns [] for missing file
- _load_hallways returns [] for corrupt JSON
- _save_hallways + _load_hallways round-trips
Algorithm:
- returns [] for unknown wing
- returns [] when no drawer has >=2 entities
- creates hallway for entity pair when threshold met (Aya↔Lumi in 3 drawers)
- connects person to concept (Aya↔consciousness — entities aren't only people)
- respects min_count threshold (rejects 2 co-occurrences when min_count=3)
- creates deterministic id per entity pair (idempotent re-runs)
- entity pair is symmetric (Aya;Lumi and Lumi;Aya are the same pair)
- persists to JSON file
- tracks rooms across co-occurrences (rooms set vs. count distinction)
- skips sentinel drawers (file_already_mined bookkeeping noise)
Query API:
- list_hallways returns all when no filter
- list_hallways filters by wing
- delete_hallway removes record and persists
- delete_hallway unknown id returns False
## Verification
ruff check mempalace/hallways.py tests/test_hallways.py
→ All checks passed!
ruff format --check ...
→ 2 files already formatted (pinned 0.15.9)
pytest tests/test_hallways.py
→ 17 passed
pytest -q (full mempalace suite)
→ 1936 passed, 1 skipped, 0 regressions caused by this PR
Linux CI parity replicated locally via OrbStack containers
(Python 3.9, 3.11, 3.13): 17/17 hallway tests pass, ruff clean on all
three.
## Not in this commit (deferred)
This PR introduces the primitive — the module, the entity-centric API,
the storage, the tests. It does NOT yet:
- integrate compute_hallways_for_wing() into miner.py / convo_miner.py /
format_miner.py post-mine flow (next PR — small one-line addition
per miner alongside the existing _compute_topic_tunnels call)
- refactor _compute_topic_tunnels_for_wing to BUILD ON hallways
(the architecturally meaningful follow-up — own PR)
- rename palace_graph.py:253's misleading docstring that calls
cross-wing tunnels "hallways" (cleanup PR)
- expose hallways through the MCP server (when there's a clear UX
need for it)
Each follow-up is small enough to review in isolation. Shipping the
primitive first lets reviewers think about the API surface separately
from the integration choices.
Per the design sequence: Wing → Drawer-entities → Hallway → Tunnel.
Hallways come first. This is them.
|
||
|
|
0b9c5629b7 |
fix(miner): mirror init's case-insensitive entity matching in per-drawer tagger
mempalace's init-time entity scanner (`entity_detector.py:276`) already
matches names case-insensitively against corpus content:
name_line_indices = [i for i, line in enumerate(lines)
if name_lower in line.lower()]
That's how a corpus that mentions "aya" all in lowercase still surfaces
"Aya" as a confirmed entity during `mempalace init`.
But the per-drawer tagger in `miner.py:_extract_entities_for_metadata`
was never updated to use the same flag — it walks the same
`known_entities.json` seed list and matches case-sensitively:
for name in known:
if re.search(r"(?<!\w)" + re.escape(name) + r"(?!\w)", content):
# ^ no IGNORECASE
matched.add(name)
So "Aya" in `known_entities.json` matches "Aya" in drawer content but
silently misses every "aya" / "AYA" mention. Chat transcripts,
voice-typed journals, and any lowercase-style corpus get their drawers
under-tagged in ways that don't show up in init's "we found Aya/Lumi/..."
confirmation list.
## The fix
One regex flag, line 788:
-if re.search(r"(?<!\w)" + re.escape(name) + r"(?!\w)", content):
+if re.search(r"(?<!\w)" + re.escape(name) + r"(?!\w)", content,
+ re.IGNORECASE):
Plus an explanatory comment so the parity with `entity_detector.py:276`
is documented at the call site.
## Empirical proof of the bug (and the fix)
Before this commit, with the actual installed `~/.mempalace/known_entities.json`
containing "Aya", "Lumi":
Content: "Aya talked to Lumi. Lumi answered." → tagged 'Aya;Lumi' ✓
Content: "aya talked to lumi. lumi answered." → tagged '' ✗ MISS
Content: "AYA mentioned LuMi to ben." → tagged '' ✗ MISS
After this commit (same content):
Content: "Aya talked to Lumi. Lumi answered." → tagged 'Aya;Lumi' ✓
Content: "aya talked to lumi. lumi answered." → tagged 'Aya;Lumi' ✓
Content: "AYA mentioned LuMi to ben." → tagged 'Aya;Ben;Lumi' ✓
## Tests
New RED-first test in `tests/test_miner.py`:
test_entity_metadata_matches_known_names_case_insensitively
Stubs `_load_known_entities` to a controlled `{"Aya", "Lumi"}` seed
and asserts that lowercase + mixed-case mentions both produce the
correctly capitalized tag. Failed on the pre-fix regex (confirmed RED
in CI parity runs across Linux 3.9 / 3.11 / 3.13); passes after the
one-line fix.
## No regressions
Full mempalace test suite: 1920 passed, 1 skipped, 0 failed.
Targeted dependent suites (`test_known_entities_registry`, `test_closets`,
`test_readme_claims`): 123 passed.
ruff check + ruff format clean on pinned 0.15.9.
CI parity replicated locally via Linux containers (Python 3.9, 3.11,
3.13): all three pass the new test and the full `test_miner.py`
sweep (46 tests).
## Why this matters
This is a quiet bug — there's no error message, no failed assertion,
just an empty `entities` field on drawers whose content actually
mentions people. The init phase looks fine ("we detected Aya, Lumi,
Ben") because init does its own case-insensitive scan; mining silently
drops the matches that init's scanner already proved are real.
Personal / journal / chat-style corpora (which use lowercase
conversational style) are the most affected. Code / docs corpora
(which use proper capitalization for names) work either way and saw
no behavior change.
|
||
|
|
5c40cc3d70 |
feat(format_miner): detect_room + topic tunnel computation (miner.py parity)
Third amendment to PR #1555. Closes the remaining parity gaps between format_miner and the existing project miner that Aya's palace audit surfaced after amendment #2. ## What was broken A smoke test against the just-mined v6 palace confirmed: WING: wing_aya ROOM: documents 2904 drawers ← ALL in one room Tunnels for wing_aya: 0 ← no cross-wing links Cause: format_miner hardcoded `room = "documents"` for every drawer and never called `_compute_topic_tunnels_for_wing()` post-mine. miner.py calls `detect_room()` per drawer and computes tunnels after the loop. format_miner did neither. ## What this commit changes 1. Module-level imports of `detect_room`, `load_config`, and `_compute_topic_tunnels_for_wing` from `.miner`. Module-level (not lazy) so test seams `patch("mempalace.format_miner.detect_room", ...)` work — lazy imports inside a function don't expose attributes on the module object. 2. `mine_formats()` loads the project's `mempalace.yaml` via `load_config()` to get the rooms list. Falls back to a single "documents" room if no config exists. 3. The hardcoded `room = "documents"` line is replaced with: room = detect_room(filepath, text, rooms, format_path) Mirrors miner.py:904 exactly — folder-match → filename-match → content-keyword scoring → fallback "general". 4. After the per-file loop completes (in an `else` branch on the outer try, so it does NOT run on KeyboardInterrupt), call `_compute_topic_tunnels_for_wing(wing)` in try/except. Exact mirror of miner.py:1241-1249. Tunnel-compute failures must never fail a mine. 5. Import `sys` at module level (used by the tunnel-compute error path for `print(..., file=sys.stderr)`). ## Tests 5 new RED-first tests in `tests/test_format_miner.py`: - test_mine_formats_calls_load_config_for_rooms - test_mine_formats_calls_detect_room_per_file - test_mine_formats_uses_detected_room_in_drawer_metadata - test_mine_formats_calls_compute_topic_tunnels_after_loop - test_mine_formats_tunnel_failure_does_not_crash_mine All 5 failed against pre-commit code with AttributeError on missing module attributes (proof the bug existed). All 5 pass after the implementation. Total format_miner test count: 62. Total proposal test count (format_miner + line_numbers): 83. Full mempalace test suite: 2002 passed, 1 skipped (no regressions). ## Verification ruff check mempalace/format_miner.py tests/test_format_miner.py → All checks passed! ruff format --check ... → already formatted (pinned ruff 0.15.9) pytest tests/test_format_miner.py tests/test_line_numbers.py → 83 passed pytest -q (full suite) → 2002 passed, 1 skipped ## Not in this commit (deferred) Within-wing hallway primitives are a separate architectural addition (separate PR — being designed). The current `_compute_topic_tunnels_for_wing` matches miner.py exactly but inherits miner.py's gap: it computes tunnels from raw topic words rather than from hallway primitives. That refactor lands in its own PR after the hallway primitive ships. |
||
|
|
e362dfa0af |
feat(format_miner): address bot review — parity fixes with miner.py
Six follow-up fixes addressing gemini-code-assist review on PR #1555. All medium-priority parity gaps between format_miner.py and the existing miner.py / convo_miner.py patterns. 1. Reuse palace.SKIP_DIRS instead of duplicating it locally. Replaces the local _SKIP_DIRS set with the shared constant so the format miner's directory-skip set stays in sync automatically. 2. scan_formats now explicitly skips symlinks. Prevents circular links, following symlinks to /dev/urandom, and processing the same file via multiple paths. Mirrors miner.py:1068 exactly. 3. mine_formats loads MempalaceConfig at start. The current chunker (miner.chunk_text) uses module-level CHUNK_SIZE constants and doesn't accept overrides, so the loaded config values aren't yet threaded into chunking; the load is wired so future chunk_text refactors that accept per-call sizing pick this up automatically. Honest scope note: this gives format_miner the same config-load stance as miner.py without overreaching what chunk_text supports. 4. source_mtime tracking + check_mtime=True. Each drawer's metadata now carries source_mtime, and file_already_mined is called with check_mtime=True so an edited PDF/DOCX is correctly re-mined on the next pass. Mirrors miner.py semantics exactly. Previously, updated documents would have been silently skipped. 5. Per-file try/except wrap + KeyboardInterrupt handling + PID cleanup. One malformed file in a mine of thousands no longer crashes the whole orchestrator — the error is logged, a counter bumps, and the loop continues with the next file. KeyboardInterrupt still produces a clean summary on Ctrl-C. _cleanup_mine_pid_file is called in a finally block so hook-spawned mines don't leak a stale PID file on exit. 6. hall + entities metadata on every drawer. Format-mined drawers now carry the same hall (via miner.detect_hall) and entities (via miner._extract_entities_for_metadata) tags as project-mined drawers. Matches the search-result quality the other miners ship. Tests: 5 new tests covering the new behaviors: - test_scan_formats_skips_symlinks - test_mine_formats_uses_check_mtime_true - test_mine_formats_records_source_mtime_in_drawer_metadata - test_mine_formats_records_hall_in_drawer_metadata - test_mine_formats_continues_after_per_file_error Total: 78 tests passing (was 73). Full mempalace test suite green modulo one pre-existing flaky test (test_chroma_close_palace_..., a sqlite-lock-timing flake unrelated to this change — passes 4/4 in isolation). ruff check + ruff format clean on pinned 0.15.9. Coverage on mempalace/format_miner.py: 84% (clears the 80% gate). Refs: PR #1555 review comments from gemini-code-assist[bot] |
||
|
|
5e3d3dde2e |
feat(3.3.6): virtual line numbering + format coverage via --mode extract
Two additive features, both following the same read-time-transform pattern:
1. Virtual line numbering — new render_with_line_numbers() and
extract_line_range() in mempalace/searcher.py. Closet pointers like
2026-01-18:L55-L72 resolve to drawer slices rendered with [55] through
[72] line prefixes without modifying any stored content. Lines already
prefixed with [<digits>] pass through unchanged. Pure functions, no I/O.
2. Format coverage (mempalace mine --mode extract) — new mempalace/format_miner.py
reads binary office formats and files drawers via the lock + purge +
upsert pattern convo_miner uses. Source files never modified.
Per-format transformer routing: MarkItDown 0.1.5 does not actually
convert .rtf (returns raw control codes unchanged, verified live), so
.rtf is routed to striprtf which does convert. Other formats stay on
MarkItDown:
.pdf .docx .pptx .xlsx .epub -> MarkItDown
.rtf -> striprtf
13 fringe cases handled with dedicated ExtractionStatus codes:
SKIP_NO_MARKITDOWN, SKIP_NO_STRIPRTF, SKIP_TOO_LARGE, SKIP_CLOUD_ONLY,
SKIP_ENCRYPTED, SKIP_EMPTY, SKIP_PERMISSION, SKIP_BROKEN_SYMLINK,
SKIP_UNRECOGNIZED, SKIP_EXTRACTION_ERROR, SKIP_NETWORK_TIMEOUT,
SKIP_UNREADABLE, plus the encoding fallback handled internally.
Drawers carry ingest_mode=extract + extract_mode=format so they are
distinguishable from project / convo drawers in the palace.
Both transformers are optional extras: pip install mempalace[extract].
MarkItDown requires Python >= 3.10 (env marker ensures it only installs
where supported; Python 3.9 users still get RTF coverage via striprtf).
Tests: 94 new (21 line-numbering + 73 format-miner, including 9
mine_formats orchestrator tests). Full mempalace suite still green
(1992 passed locally). Coverage 86% on mempalace/format_miner.py.
ruff check + ruff format clean on the pinned 0.15.9.
Live verification: mempalace mine --mode extract on a directory with 2
RTFs + 1 PDF produced 90 drawers correctly; mempalace search found
content from both file types in the resulting wing.
Documented limits (out of scope for 3.3.6): custom PDF parsers, OCR on
scanned PDFs, DRM-locked files, pathological corrupt files. These get
reported via skip codes and skipped.
Refs: docs/format-coverage.md, docs/virtual-line-numbering.md
|
||
|
|
5decc0c83b |
fix(miner): configurable + raised MAX_CHUNKS_PER_FILE (#1455)
Default raised from 500 to 50_000 so legitimate long-form content (novels, scholarly editions) is not silently dropped on a typical literary corpus. Cap configurable via MEMPALACE_MAX_CHUNKS_PER_FILE env var or --max-chunks-per-file CLI flag; sentinel 0 disables. Negative or non-int values from either source emit a stderr warning and fall back to the default. Separate counter (files_skipped_chunk_cap) and summary line surface chunk-cap drops independently of the residual already-filed bucket; counter fires under --dry-run too so a corpus audit shows the same signal without writing. Skip notice routed to stderr alongside the existing symlink-skip line. ONNX bad_alloc protection preserved architecturally: DRAWER_UPSERT_BATCH_SIZE=1000 bounds per-ONNX-call exposure regardless of per-file cap. A 50_000-chunk file produces 50 forward passes of 1_000 chunks each, identical to what 500-chunk files already produced. process_file return extended to a 3-tuple (drawers, room, skip_reason); skip_reason is None on non-chunk-cap paths and "chunk_cap" when the per-file cap triggered. Internal callers (_mine_impl, benchmarks/mine_bench.py, tests) updated. Closes #1455 Co-Authored-By: David Glidden <d@davidglidden.eu> |
||
|
|
7550de27a0 |
fix(graph): repo-format tests, %s for tunnel paths, single config per call
Three changes addressing #1469 CI red + Gemini perf review: 1. ruff format (0.4.x) on tests/test_closets.py and tests/test_palace_graph_tunnels.py — the lint job pins ruff>=0.4.0,<0.5 and was flagging format drift. 2. Replace %r with '%s' in legacy / corrupt tunnel-file warnings. On Windows %r escapes backslashes in repr, so test_load_tunnels_warns_on_orphaned_legacy_file's 'str(legacy) in caplog.text' assertion was failing on test-windows even though the warning was firing. 3. Address gemini-code-assist review on #1469: pass a single MempalaceConfig() through _get_tunnel_file / _load_tunnels / _save_tunnels per create_tunnel call instead of each helper re-instantiating its own (which re-reads mempalace.yaml from disk). Helpers keep their config=None defaults so external callers and existing tests are unaffected. |
||
|
|
1b94f4efb4 |
Merge pull request #1538 from mvalentsev/fix/1534-paragraph-chunk-size-enforcement
fix(convo_miner): enforce CHUNK_SIZE in paragraph chunker (#1534) |
||
|
|
100539d2ca |
Merge pull request #1543 from fatkobra/fix/1443-mine-slot-placeholder
Fix: Write placeholder PID when claiming mine slot |