The original sweep in this PR was scoped against develop as of April. Since
then upstream added console output carrying the same characters, so the fix
had drifted into covering roughly half of the affected surface.
Re-measured against current develop and replaced the remaining occurrences of
the same five characters this PR already targets -- U+2014 em-dash, U+2500 box
drawing, U+2192 arrow, U+25CF/U+25CB circles -- on every print(), input() and
logger call reachable from a terminal:
repair.py (10), cli.py (7), format_miner.py (7), mcp_server.py (5),
miner.py (2), onboarding.py (2), migrate.py (2), searcher.py (2),
service.py (1), embedding.py (1), dialect.py (1)
Substitutions match the ones already used in this branch: em-dash -> "--",
box drawing -> "-", arrow -> "->", filled/empty circles -> "#"/".".
Four assertions matched the previous literals and were updated alongside the
strings they cover (test_cli, test_miner x2, test_repair).
Docstrings, comments and stored content are deliberately untouched -- only
what reaches a terminal, where GBK/CP1252 consoles raise UnicodeEncodeError.
hnsw_capacity_status() (chroma.py) exists precisely to preflight the
#1222 SIGSEGV/pyo3-panic class before anything touches the HNSW
segment, but repo-wide it was wired into only 4 call sites while raw
count()/collection.count() is called at 20+ others -- a bare
except Exception around count() cannot catch a native crash, since
the process dies regardless of any Python try/except. This wires the
existing, already-tested probe into the 7 remaining call sites the
audit identified as CRITICAL:
- #89 palace.py::_enforce_embedder_identity -- the universal
get_collection() chokepoint every tool passes through, previously
guarded only by except Exception. Highest leverage: skips this
bookkeeping-only check on divergence instead of risking count().
- #90 migrate.py::migrate -- routes straight to the same
SQLite-extraction fallback the except branch already used, instead
of ever reaching col.count() when diverged.
- #91 repair.py::scan_palace / prune_corrupt -- both abort with the
existing from-sqlite recovery guidance instead of opening the
collection.
- #10 repair.py::rebuild_index -- preflights divergence alongside its
existing sqlite-integrity and poisoned-max-seq-id preflights, before
opening the collection.
- #13 repair.py::rebuild_index never rebuilt or reported on the
closets collection -- now warns when closets is still diverged
after a drawers-only rebuild, pointing at --mode from-sqlite instead
of letting 'Repair complete' stand unqualified.
- #92 dedup.py::get_source_groups -- takes an optional palace_path
(threaded from both callers) to preflight before count(); omitted by
existing tests, which keep their pre-existing behavior.
- #93 miner.py::status -- preflights before the ChromaDB-client
fallback path (used when the direct sqlite read is unavailable).
7 new regression tests, each confirmed failing against the pre-fix
code (via git stash of the source files only) and passing after the
fix. One existing dedup.py test updated for the new palace_path kwarg
in its call-signature assertion. Full suite: 3154 passed, 1 unrelated
pre-existing flake (test_mcp_server.py peer-writer-lock module-global
state leaking across test files in full-suite ordering -- this diff
never touches mcp_server.py).
chunk_text recomputed line_start/line_end with a full-prefix
content.count("\n", 0, pos) per chunk (O(N*K) overall), so large files
ground for days. Keep the emitted values byte-identical but tally
newlines incrementally over the newly-scanned span (O(N) total), with a
from-scratch fallback that keeps every value exact.
Allow projects to specify .gitignore-style patterns that the miner should
skip, without relying on .gitignore for mining control.
A new optional exclude_patterns list in mempalace.yaml is parsed by the
existing GitignoreMatcher class via a new from_patterns() classmethod —
same syntax, same semantics as .gitignore, no new dependency.
exclude_patterns:
- '*.md'
- '*.yaml'
- 'docs/' # dir-only: prunes entire tree without descending
- 'dist/'
- 'coverage/'
Key behaviour:
- Patterns follow .gitignore rules: anchoring (/pattern), dir-only
- dirs[:] pruning via GitignoreMatcher.matches(..., is_dir=True) so
excluded subtrees are never walked
- Checked after .gitignore filtering; force_include (--include-ignored)
bypasses exclude_patterns
- Pre-scanned files lists (init double-scan optimisation) are filtered too
- Backwards compatible: omitting exclude_patterns changes nothing
Changes:
- GitignoreMatcher.from_patterns(): new classmethod, same rule parser as
from_dir(), reads from a list instead of a file on disk
- scan_project(): builds one exclude_matcher before os.walk; used for
both dirs[:] pruning and per-file filtering
- _mine_impl(): applies the exclude matcher to pre-scanned files lists
when provided by the caller
- tests/test_miner.py: three new tests
test_scan_project_exclude_patterns_skips_matching_files
test_scan_project_exclude_patterns_prunes_entire_directory
test_scan_project_exclude_patterns_include_ignored_bypasses_exclusion
LaTeX source files and BibTeX bibliographies are prose-rich content that
benefits from both palace mining and entity detection. Adds the two
extensions to the two extension lists most relevant to them, each with a
matching test.
- ``mempalace/miner.py:READABLE_EXTENSIONS`` — ``.tex`` / ``.bib`` join the
mining allowlist (parallel to the Swift/Kotlin PR #1368 and the PHP
ecosystem PR #1819).
- ``mempalace/entity_detector.py:PROSE_EXTENSIONS`` — ``.tex`` / ``.bib``
also join the *preferred* entity-detection bucket alongside ``.md`` /
``.rst`` / ``.csv``, NOT the broader code-file fallback. The reason
``PROSE_EXTENSIONS`` exists separately is documented in-code:
programming-language files have lots of capitalized identifiers (class
names, function names) that produce false-positive person matches.
LaTeX/BibTeX don't have that problem — they're typesetting languages
for prose documents. ``.bib`` in particular is almost entirely author
names, one of the highest real-entity densities of any file type the
detector scans.
Tests follow the patterns established by the prior extension PRs:
``tests/test_miner.py::test_scan_project_includes_latex_files`` mirrors
the Swift/Kotlin scan tests, and
``tests/test_entity_detector.py::test_scan_for_detection_includes_latex_prose``
mirrors ``test_scan_for_detection_finds_prose``. The existing
``test_prose_extensions`` was extended to assert the two new entries.
Full env-cleared suite: 3216 passed, 20 skipped. ``ruff check .`` and
``ruff format --check .`` both clean.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KC5Qsknh2zFRtRvVyjXiTA
The original commit printed SKIP for oversized files to stdout but the
sibling SKIP for symlinks in the same scan_project / scan_convos already
went to stderr. Align the new line with that convention.
Also adds a SKIP-with-error log for the except OSError arm right below the
size check. Files whose stat() raises (permission denied, racing delete,
broken symlink that survived the earlier is_symlink check) were the same
bug class as the silent oversize drop.
Tests switched from captured.out to .err and tightened to the full
template; new test covers the OSError arm via a selective Path.stat
monkeypatch with a follow_symlinks gate for Python 3.10+.
Both miner.py and convo_miner.py silently skip files larger than the
10 MB limit with a bare continue. This is especially painful for
conversation mining where long Claude/ChatGPT exports routinely
exceed 10 MB and vanish with no trace.
Print a SKIP warning per oversized file, matching the existing format
in split_mega_files.py.
test_load_config_uses_defaults_when_yaml_missing asserted the derived
wing equals project_root.name. That only held when the random tempfile
name had no separators; tempfile's alphabet includes '_', so once
normalize_wing_name strips leading/trailing '_' (this PR), a name like
'tmpXXXX_' makes the derived wing diverge from the raw name. Compare
against normalize_wing_name(project_root.name) — the actual contract —
which is deterministic across platforms. (Surfaced as a test-windows
failure on this PR, but it was cross-platform flaky.)
`mempalace status` opened the ChromaDB collection purely to tally drawers by
wing/room — and opening it cold-loads the HNSW vector index. On a 398k-drawer
palace that load costs ~60s of CPU on every invocation, even though the counts
live in chroma.sqlite3's relational tables (`repair-status` already reads them
in <1s; `status` was the outlier).
Read the wing/room histogram directly from chroma.sqlite3 via a new
`_sqlite_wing_room_counts` helper, falling back to the existing ChromaDB-client
path when the sqlite read is unavailable (missing DB, un-bootstrapped
collection, sustained writer lock, or an unexpected schema) — preserving the
state-specific guidance for absent/empty palaces.
Measured on a 398,315-drawer / 3.5GB palace: status CPU ~60s -> ~1s.
Review hardening:
- PRAGMA busy_timeout so a transient checkpoint lock is waited out rather than
instantly demoted to the slow path; a sustained lock still falls back.
- COALESCE over string/int/float so a numeric wing/room matches the ChromaDB
path instead of dropping to "?".
- Explicit `s.scope = 'METADATA'` so the segment join can't silently
double-count on a future ChromaDB layout.
Tests: exact-tally (anti fan-out), no-cold-load regression (proven failable by
reverting the fix), numeric-metadata, partial-metadata "?" bucketing,
locked-DB fallback, and collection-absent None routing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Under the additive-mining model (drawer history preserved across re-mines),
a single source_file can have multiple parent_drawer_id groups in the
palace, one per mining pass. Each group carries its own stored
source_mtime and normalize_version.
`file_already_mined` previously used `collection.get(where={"source_file": X},
limit=1)` in the `extract_mode is None` branch and only checked the single
returned row's metadata. ChromaDB's `get(..., limit=1)` has no ordering
guarantee across multiple matching rows, so the returned row was effectively
arbitrary. When ChromaDB happened to return a stale group (older mining
pass with a different stored mtime), the function returned False, the
additive miner concluded the file had changed, and wrote yet another
duplicate group of drawers for a file that had not actually changed since
the last successful mine.
Steady-state failure: for any source_file that has ever been edited (so
that the palace contains groups with different stored mtimes), each re-mine
has a probability of spuriously concluding "file changed" and writing
another duplicate group. Each spurious group compounds the problem because
its stored mtime can also trigger the next spurious re-mine. Storage grows
without bound; search results, hallway counts, and entity-frequency stats
become inflated proportionally.
The fix mirrors the paginated-iteration pattern already used in the
`extract_mode is not None` branch — iterate every drawer for the
source_file in 1000-row pages, short-circuit on the first matching group.
A correct group is one that passes the existing checks: normalize_version
not stale; if check_mtime, stored source_mtime within 0.001 seconds of the
current file mtime. The two branches (extract_mode is None vs set) collapse
into one loop that skips the extract_mode check when no extract_mode was
specified.
Trade-off: average-case cost rises from O(1) limit=1 query to O(N/1000)
paginated scan. For typical sources (1-3 groups) the cost is unchanged
because the loop short-circuits on the first matching group within the
first page. For pathological sources with thousands of groups, the cost
is O(number-of-pages-until-match) — still bounded, no longer flaky.
RED test pins the failure space deterministically
`test_file_already_mined_handles_multiple_groups_under_one_source_file`
uses a MockCollection that simulates two parent_drawer_id groups under one
source_file: a STALE group (older mtime) and a CURRENT group (matching
mtime). The mock returns the STALE group when called with limit=1
(worst-case ChromaDB ordering) and returns BOTH groups when called with
limit=1000 (what a correctly-iterating implementation must do). Test asserts
file_already_mined returns True even when limit=1 picks the stale group.
- Against pre-fix code: test FAILS (function returns False because
limit=1 picks stale group, mtime mismatch returns False)
- Against post-fix code: test PASSES (iteration finds the current group,
short-circuits to True)
Verified RED-then-GREEN locally. Existing 4 file_already_mined tests in
tests/test_miner.py continue to pass:
- test_file_already_mined_check_mtime
- test_file_already_mined_scopes_convo_extract_mode
- test_file_already_mined_extract_mode_paginates_large_sources
- test_file_already_mined_returns_false_for_stale_normalize_version
Verification
- macOS Python 3.12 (local) full pytest : 2268 passed, 0 failed
- Linux Python 3.9.25 (OrbStack) : 2260 passed, 0 failed
- Linux Python 3.11.15 (OrbStack) : 2261 passed, 0 failed
- Linux Python 3.13.13 (OrbStack) : 2261 passed, 0 failed
- ruff check + ruff format --check : all clean
Provenance
Surfaced during the per-query audit on the PR #1628 amendment cycle (the
search for every bare `where={"source_file": ...}` query in the repo).
One of six sites identified. The other five are legitimately file-global
in intent (closet purges, full-rebuild deletes, paginated mode-filtered
scans). This site is the one whose failure mode mirrors the cross-group
stitching pattern PR #1628 fixed at the searcher layer.
Igor's review on PR #1584 (2026-05-22) flagged four issues:
1. The feature wasn't wired into any production caller — the new
``drawer_metas`` kwarg on ``build_closet_lines`` had no real
consumer in ``miner.py`` / ``diary_ingest.py``, so the 4-segment
pointer form only existed in tests. Real palaces kept emitting
the legacy 3-segment shape.
2. ``_extract_content_date`` hallucinated dates on benign inputs.
``dateutil.parser.parse(fuzzy=True)`` would accept anything with
digits and return a plausible-looking but wrong date —
``Version 3.3.6`` → ``2006-03-03``, ``Tested with 1000 drawers``
→ ``1000-05-22``, ``tmp_random_file_5`` → ``2026-05-05``, etc.
Mtime almost never got reached because fuzzy returned *something*
from filename or body first. Bad dates were silently persisted
to ChromaDB.
3. ``python-dateutil`` was an undeclared dependency, available only
transitively via ``chromadb → kubernetes → python-dateutil``. Not
a contract — upstream kubernetes has been trending toward
stdlib-only.
4. Two-digit-year disambiguation (70 → 19xx / 00-69 → 20xx) had no
test pinning the boundary.
This commit addresses all four.
## Changes
### Issue 2 — kill the hallucination (the load-bearing fix)
``mempalace/miner.py``:
- New ``_VALID_DATE_RE`` gate. Three accepted shapes (all require a
4-digit year explicitly):
1. Numeric YYYY-MM-DD with ``[-/.\\s]`` separators
(covers ISO and space-normalized filenames)
2. Month-name + day + year ("November 8 2024", "Nov 8 2024")
3. Day + month-name + year ("8 November 2024")
Partial dates ("2024-06", "April 6", "notes.2024") are
DELIBERATELY rejected — without all three components we'd pad from
today's date, which is hallucination not extraction.
- ``_try_filename_date`` and ``_try_content_body_date`` now run the
gate BEFORE invoking dateutil, and pass ``fuzzy=True`` is REMOVED.
Dateutil only runs in strict mode on a substring the gate already
validated.
### Issue 1 — wire the feature into production
``mempalace/miner.py`` batched-upsert path:
- Accumulate ``batch_metas`` across all batches into ``all_metas``
- Pass ``drawer_metas=all_metas`` to ``build_closet_lines``
End-to-end integration test added that mines a real file with a
filename-derived content date and asserts the produced closet
documents contain the 4-segment pointer with that date.
``diary_ingest.py`` is left as-is for this PR. Diary entries are
entry-keyed, not chunk-keyed — they carry no natural
``line_start`` / ``line_end``, so the 4-segment form would return
None for them regardless. Wiring the diary path can land cleanly in
a follow-up once Tier 6a gains an "approximate line range for diary
entries" story.
### Issue 3 — declare the dateutil dependency
``pyproject.toml``: add ``python-dateutil>=2.8`` to
``[project].dependencies``. One-line change; cheaper than the
stdlib-only refactor alternative and keeps the natural-language
recall surface.
### Issue 4 — pin the two-digit-year boundary
Four new tests cover the 1969/1970/1999/2000 corner cases of the
slash-date locale heuristic.
## Tests added (RED-first then GREEN)
tests/test_miner.py::TestExtractContentDate (11 new):
Hallucination cases verbatim from Igor's review:
- test_no_hallucination_junk_filename_with_trailing_digit
- test_no_hallucination_untitled_with_index
- test_no_hallucination_filename_year_only
- test_no_hallucination_filename_year_and_month_only
- test_no_hallucination_content_with_issue_number
- test_no_hallucination_content_with_count
- test_no_hallucination_content_with_version_number
Two-digit-year boundary cases:
- test_two_digit_year_69_is_2069
- test_two_digit_year_70_is_1970
- test_two_digit_year_99_is_1999
- test_two_digit_year_00_is_2000
tests/test_closets.py::TestMinerClosetRebuild (1 new):
- test_production_miner_emits_4_segment_pointers_with_content_date
(regression for Issue #1 — real ``mine()`` end-to-end produces
4-segment closet pointers via the new ``drawer_metas`` wiring)
## Verification
pytest tests/test_miner.py tests/test_closets.py
tests/test_format_miner.py tests/test_palace.py
→ 242 passed, 2 skipped, 0 regressions
pytest tests/test_miner.py::TestExtractContentDate
→ 26 passed (15 prior + 11 new)
pytest tests/test_closets.py::TestMinerClosetRebuild
→ end-to-end wiring test GREEN
Sanity (Igor's exact repros):
"tmp_random_file_5" → None (was: 2026-05-05)
"untitled-1" → None (was: 2026-05-01)
"notes.2024.md" → None (was: 2024-05-22)
"2024-06.md" → None (was: 2024-06-22)
"Bug fix for issue 42 in module 7" → None (was: 2042-07-22)
"Tested with 1000 drawers" → None (was: 1000-05-22)
"Version 3.3.6 released" → None (was: 2006-03-03)
Real dates still extract correctly:
"2024-11-08.md" → "2024-11-08"
"April-6th-2011-notes.md" → "2011-04-06"
"Nov-8-2024.md" → "2024-11-08"
ruff check + ruff format --check (pinned 0.15.9)
→ All checks passed
OrbStack triple-Linux verify (Py 3.9 / 3.11 / 3.13)
→ all targeted tests pass; python-dateutil installs explicitly
via the new declared dependency.
Closet pointer lines gain a 4th pipe-separated segment of shape
``YYYY-MM-DD:Lstart-Lend`` so retrieval can jump straight to the right
span in the source file instead of opening the whole drawer. The date
is selected via a content-aware hierarchy so legacy ingests (where
``filed_at`` is misleading) still produce honest pointers when the
content itself carries a date.
The legacy 3-segment format remains the fallback for drawers without
the new metadata, so existing palaces keep working without a re-mine.
## Pointer shape
Before:
topic|entities|→drawer_ids
After (when drawer metadata carries line range + a date):
topic|entities|2024-11-08:L42-L78|→drawer_ids
Backward compat: when ``drawer_metas`` is not passed, or the first
meta lacks ``line_start`` / ``line_end``, ``build_closet_lines`` emits
the legacy 3-segment form unchanged.
## Date-source hierarchy (NEW)
A new helper ``mempalace.miner._extract_content_date(source_file, content)``
selects a date once per file and stores it as ``content_date`` on every
drawer mined from that file. Priority order, first match wins:
1. **Filename** — ISO regex on stem, then dateutil fuzzy parse for
natural-language formats. Handles ``2024-11-08.md``,
``April-6th-2011-notes.md``, ``Nov-8-2024.md``, etc.
2. **YAML frontmatter** — looks at ``date`` / ``created`` /
``published`` fields. yaml.safe_load already parses ISO dates as
``datetime.date`` objects; dateutil handles string values.
3. **Content body, first ~10 lines** —
a. ISO regex (``YYYY-MM-DD`` / ``YYYY/MM/DD`` / ``YYYY.MM.DD``)
b. Slash dates with locale auto-disambiguation: if any
first-number > 12 appears anywhere in the head, lock the
file's locale to DD/MM (the only consistent reading);
otherwise default to US MM/DD for two-digit-year compactness.
c. dateutil fuzzy parse for natural-language
(``November 8, 2024``, ``8th November 2024``, etc.)
4. **Filesystem mtime** — ``os.path.getmtime`` formatted as
``YYYY-MM-DD``.
5. **None** — caller (``_build_drawer_metadata``) leaves
``content_date`` absent; ``build_closet_lines`` falls back to the
``filed_at`` ingestion timestamp's date portion.
``_build_date_line_segment`` in ``palace.py`` now prefers ``content_date``
when present, otherwise uses ``filed_at[:10]``. This is the load-bearing
change that makes Tier 6a meaningful for legacy content — a Nov 8 2024
transcript mined today now emits ``2024-11-08:L42-L78`` instead of
``2026-05-22:L42-L78``.
## Changes
1. ``mempalace/miner.py::chunk_text`` — each chunk dict now also carries
``line_start`` / ``line_end`` (1-indexed line numbers in the stripped
source). Computed via ``content[:offset].count("\n") + 1`` —
approximate locator accurate to ±1 line at chunk boundaries.
2. ``mempalace/miner.py::_extract_content_date`` (NEW) + helpers
(``_try_filename_date``, ``_try_frontmatter_date``,
``_try_content_body_date``, ``_try_mtime_date``). Pure functions, no
I/O beyond what's named. Uses lazy imports of ``dateutil`` and
``yaml`` to keep top-level import surface unchanged.
3. ``mempalace/miner.py::_build_drawer_metadata`` — gains
``line_start`` / ``line_end`` / ``content_date`` optional kwargs.
When provided, stored in drawer metadata; when omitted, keys are
absent from the dict — backward compatible.
4. ``mempalace/miner.py`` batched-upsert path — calls
``_extract_content_date(source_file, content)`` ONCE per file (not
once per chunk) and passes the result through to every chunk's
``_build_drawer_metadata`` call.
5. ``mempalace/format_miner.py::_file_chunks_locked`` — gains a
``content`` param so the format-miner path can also run
``_extract_content_date`` on the full extracted text. Plumbs
``content_date`` into every batched drawer's metadata. Caller passes
``text`` (the full extracted markdown).
6. ``mempalace/palace.py::build_closet_lines`` — accepts optional
``drawer_metas`` (parallel to ``drawer_ids``). When present and the
first meta has both ``line_start`` and ``line_end``, the new
segment is spliced in; otherwise legacy 3-segment form. Helper
``_build_date_line_segment`` prefers ``content_date`` over
``filed_at`` for the date portion.
## Tests added (RED-first, then GREEN)
tests/test_miner.py::TestChunkTextLineRanges (4 tests)
tests/test_miner.py::TestBuildDrawerMetadataLineRange (2 tests)
tests/test_miner.py::TestExtractContentDate (15 tests)
- filename ISO / natural-language / compact
- YAML frontmatter date / created
- Claude session preamble
- ISO + natural-language content body
- Slash-date locale disambiguation (DD/MM lock vs default MM/DD)
- Priority order (filename > frontmatter > content > mtime)
- mtime fallback
- Graceful None returns for missing file / empty content
tests/test_closets.py::TestBuildClosetLines (5 new tests)
- includes_date_line_segment_when_metas_provided
- falls_back_to_3_segment_format_when_metas_missing (compat)
- falls_back_to_3_segment_format_when_metas_lack_line_keys (compat)
- date_segment_uses_filed_at_date_portion_only
- content_date_preferred_over_filed_at (new — Tier 6a hierarchy)
26 new tests total. All RED before this commit, all GREEN after.
## Verification
pytest tests/test_miner.py::TestChunkTextLineRanges
tests/test_miner.py::TestBuildDrawerMetadataLineRange
tests/test_miner.py::TestExtractContentDate
tests/test_closets.py::TestBuildClosetLines
→ 32 passed
pytest tests/test_miner.py tests/test_closets.py
tests/test_format_miner.py tests/test_palace.py
→ 230 passed, 2 skipped, 0 regressions in directly-affected files
ruff check + ruff format --check
→ All checks passed, 5 files already formatted
End-to-end smoke test on a YAML-frontmatter file (date: 2024-11-08,
mined "today"):
content_date extracted: 2024-11-08
chunks produced: 1
drawer meta: content_date=2024-11-08, filed_at='2026-05-22...', lines=1-15
closet pointer:
"conversation with lumi about brands of dog food|Filler;Lumi;Aya|2024-11-08:L1-L15|→drawer_a,drawer_b"
→ content_date correctly preferred over filed_at end-to-end.
OrbStack triple-Linux verify (Py 3.9 / 3.11 / 3.13)
→ 32 targeted Tier 6a + content-date tests pass on each.
## Out of scope (deliberate)
- No schema migration of existing drawers. Pre-Tier-6a drawers lack
``line_start`` / ``content_date`` and silently fall back to the
legacy 3-segment pointer shape.
- No re-mine required. Users keep existing palaces; new files mined
after this lands carry the new fields.
- No drawer ID change (Task #80, separate concern).
- No exact-quote line positioning within a chunk (future tier).
- No NLP-based date extraction (only regex + dateutil fuzzy).
- Linguistic-similarity / concept-recognition retrieval (L5 zettal
layer) is the natural next step but lives in its own PR family.
Resolves conflicts from 60-commit divergence:
- tests/test_closets.py: assertion reformat — kept develop's ruff-format-
preferred multi-line shape (functionally identical).
- tests/test_palace_graph_tunnels.py: both branches added a new test
class at end-of-file (this PR's TestTunnelFileFollowsConfig + develop's
TestEntityTunnels from #1564). Kept both, no overlap.
Local: ruff check / format pass; full pytest suite passes
(2113 passed, 3 skipped).
Co-Authored-By: Igor Lins e Silva <4753812+igorls@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the architectural counterpart to ``compute_topic_tunnels`` that
materializes cross-wing tunnels from the within-wing hallway records
introduced in PR #1558. When an entity (person, project, concept,
interest) has hallways in two wings, an entity tunnel bridges them —
anchored on the entity. This completes the v4 sequence: Wing →
Drawer-entities → Hallway → Tunnel.
Topic tunnels are NOT replaced. Both systems coexist for one release
cycle so existing palaces don't lose tunnels between mines. Deprecation
of topic tunnels is a separate follow-up PR after entity tunnels prove
out in real use.
## What this commit does
1. Adds ``entity_tunnels_for_wing(wing, hallways, label_prefix)`` to
``mempalace/palace_graph.py``. Pure function: groups hallway records
by entity-and-wing, finds entities present in ``wing`` AND ≥1 other
wing, and emits one ``create_tunnel`` call per (entity, other_wing)
pair. Uses ``kind="entity"`` and synthetic endpoint room
``entity:<name>`` so the new tunnels are distinguishable from
explicit/topic tunnels at read time but interchangeable with them
via the standard ``list_tunnels`` / ``follow_tunnels`` API.
2. Adds ``_compute_entity_tunnels_for_wing(wing)`` wrapper to
``mempalace/miner.py``. Loads hallway records via
``hallways.list_hallways()`` and calls the algorithm. Module-level
so tests can patch it as ``mempalace.miner._compute_entity_tunnels_for_wing``.
3. Wires the wrapper into ``_mine_impl`` immediately after the existing
hallway-compute block. Same try/except fault-tolerance pattern as
the topic-tunnel and hallway blocks — entity-tunnel computation is
a derived analytic and must never fail a mine.
## Tests (RED-first)
Nine algorithm tests in ``tests/test_palace_graph_tunnels.py``
(new ``TestEntityTunnels`` class):
test_entity_tunnels_creates_cross_wing_tunnel_for_shared_entity
test_entity_tunnels_skips_entities_in_only_one_wing
test_entity_tunnels_counts_entity_in_either_pair_position
test_entity_tunnels_three_wings_pairwise_from_focus_wing
test_entity_tunnels_idempotent_on_rerun
test_entity_tunnels_retrievable_via_list_tunnels
test_entity_tunnels_empty_hallways_is_noop
test_entity_tunnels_unknown_wing_is_noop
test_entity_tunnel_room_does_not_collide_with_literal_room
Two integration tests in ``tests/test_miner.py``:
test_mine_computes_entity_tunnels_for_wing_post_mine
test_mine_entity_tunnel_failure_does_not_crash_mine
All 11 RED before this commit (AttributeError on the missing names).
All 11 GREEN after.
## Out of scope (deferred to follow-up PRs)
- ``format_miner.py`` and ``convo_miner.py`` integration: separate PRs
per the scope discipline used for #1560.
- Deprecating ``_compute_topic_tunnels_for_wing``: separate PR after
entity tunnels prove out in real use.
- Surfacing ``kind="entity"`` in MCP / search-result UI: not yet
required by any reader; behaviorally interchangeable with the other
tunnel kinds today.
## Stacking
This PR stacks on PR #1558 (which introduces the hallway primitive and
its miner integration). Base branch is
``feat/hallways-within-wing-connectors``. When #1558 merges to develop,
GitHub auto-updates this PR's base to ``develop`` and the diff reduces
to just the entity-tunnel additions.
## Verification
pytest tests/test_palace_graph_tunnels.py::TestEntityTunnels
→ 9 passed (RED before, GREEN after)
pytest tests/test_miner.py::test_mine_computes_entity_tunnels_for_wing_post_mine
tests/test_miner.py::test_mine_entity_tunnel_failure_does_not_crash_mine
→ 2 passed (RED before, GREEN after)
pytest tests/test_palace_graph_tunnels.py
→ 39 passed (no regressions)
pytest tests/test_miner.py
→ 49 passed (no regressions)
pytest -q (full mempalace suite)
→ 1949 passed, 1 skipped, 0 regressions
ruff check mempalace/palace_graph.py mempalace/miner.py tests/
→ All checks passed!
ruff format --check ...
→ 4 files already formatted (pinned 0.15.9)
Wires the hallway primitive (from PR #1558) into the project miner so
that every mine that touches a wing also materializes within-wing
entity hallways for that wing. Without this integration, the hallway
module is dead code — no miner triggers it, no hallways ever land in
~/.mempalace/hallways.json.
## What this commit does
1. Adds a module-level import of ``compute_hallways_for_wing`` from
``.hallways`` near the top of ``miner.py``. Module-level (not lazy)
so tests can patch it as ``mempalace.miner.compute_hallways_for_wing``
— lazy imports inside a function wouldn't expose the seam.
2. In ``_mine_impl``, immediately after the existing
``_compute_topic_tunnels_for_wing(wing)`` post-mine block, adds a
parallel hallway block. The block:
- calls ``compute_hallways_for_wing(wing, col=collection)``
- prints the count if any hallways were materialized
- wraps the whole thing in try/except so a hallway-compute failure
is logged + degraded, never propagated. Mirrors the tunnel block's
fault-tolerance pattern exactly. Hallway computation is a derived
analytic, not load-bearing for the drawer write that already
committed above.
## Stacking
This PR stacks on PR #1558 (which introduces the hallway primitive
module). PR #1558 is the prerequisite — without it, the import
``from .hallways import compute_hallways_for_wing`` doesn't resolve.
Base branch for this PR is ``feat/hallways-within-wing-connectors``
(PR #1558's branch). When PR #1558 merges to develop, GitHub will
auto-update this PR's base to develop and the diff will reduce to
just the miner.py and tests/test_miner.py changes.
## Out of scope (deferred to follow-up PRs)
- ``format_miner.py`` integration: lives on a different branch (PR #1555)
so the parallel call there goes in as a follow-up amendment to that
branch (or a separate post-merge PR).
- ``convo_miner.py`` integration: convo_miner currently doesn't call
``_compute_topic_tunnels_for_wing`` either. Adding both calls is a
separate concerned PR about convo_miner parity, not just hallways.
- Refactoring ``_compute_topic_tunnels_for_wing`` to BUILD ON hallways
(rather than computing from raw topic words): the architecturally
meaningful follow-up that completes the Wing → Drawer-entities →
Hallway → Tunnel sequence. Real refactor, separate PR.
## Tests
Two new RED-first tests in ``tests/test_miner.py``:
test_mine_computes_hallways_for_wing_post_mine
Stubs ``mempalace.miner.compute_hallways_for_wing`` via monkeypatch.
Runs a real mine into a tmp palace. Asserts the stub was called
exactly once, with the wing name from mempalace.yaml and a live
ChromaDB collection (not None).
test_mine_hallway_failure_does_not_crash_mine
Stubs the hallway function to raise. Runs a real mine. Asserts
mine() doesn't propagate, and that the drawer write (which
happens BEFORE the hallway block) still committed.
Both RED before this commit (AttributeError — module had no attribute
``compute_hallways_for_wing``). Both GREEN after.
## Verification
ruff check mempalace/miner.py tests/test_miner.py
→ All checks passed!
ruff format --check ...
→ 2 files already formatted (pinned 0.15.9)
pytest tests/test_miner.py
→ 47 passed (the 2 new + 45 pre-existing)
pytest -q (full mempalace suite)
→ 1938 passed, 1 skipped, 0 regressions
Linux CI parity replicated locally via OrbStack containers
(Python 3.9, 3.11, 3.13): 2/2 new integration tests pass, ruff clean
on all three.
mempalace's init-time entity scanner (`entity_detector.py:276`) already
matches names case-insensitively against corpus content:
name_line_indices = [i for i, line in enumerate(lines)
if name_lower in line.lower()]
That's how a corpus that mentions "aya" all in lowercase still surfaces
"Aya" as a confirmed entity during `mempalace init`.
But the per-drawer tagger in `miner.py:_extract_entities_for_metadata`
was never updated to use the same flag — it walks the same
`known_entities.json` seed list and matches case-sensitively:
for name in known:
if re.search(r"(?<!\w)" + re.escape(name) + r"(?!\w)", content):
# ^ no IGNORECASE
matched.add(name)
So "Aya" in `known_entities.json` matches "Aya" in drawer content but
silently misses every "aya" / "AYA" mention. Chat transcripts,
voice-typed journals, and any lowercase-style corpus get their drawers
under-tagged in ways that don't show up in init's "we found Aya/Lumi/..."
confirmation list.
## The fix
One regex flag, line 788:
-if re.search(r"(?<!\w)" + re.escape(name) + r"(?!\w)", content):
+if re.search(r"(?<!\w)" + re.escape(name) + r"(?!\w)", content,
+ re.IGNORECASE):
Plus an explanatory comment so the parity with `entity_detector.py:276`
is documented at the call site.
## Empirical proof of the bug (and the fix)
Before this commit, with the actual installed `~/.mempalace/known_entities.json`
containing "Aya", "Lumi":
Content: "Aya talked to Lumi. Lumi answered." → tagged 'Aya;Lumi' ✓
Content: "aya talked to lumi. lumi answered." → tagged '' ✗ MISS
Content: "AYA mentioned LuMi to ben." → tagged '' ✗ MISS
After this commit (same content):
Content: "Aya talked to Lumi. Lumi answered." → tagged 'Aya;Lumi' ✓
Content: "aya talked to lumi. lumi answered." → tagged 'Aya;Lumi' ✓
Content: "AYA mentioned LuMi to ben." → tagged 'Aya;Ben;Lumi' ✓
## Tests
New RED-first test in `tests/test_miner.py`:
test_entity_metadata_matches_known_names_case_insensitively
Stubs `_load_known_entities` to a controlled `{"Aya", "Lumi"}` seed
and asserts that lowercase + mixed-case mentions both produce the
correctly capitalized tag. Failed on the pre-fix regex (confirmed RED
in CI parity runs across Linux 3.9 / 3.11 / 3.13); passes after the
one-line fix.
## No regressions
Full mempalace test suite: 1920 passed, 1 skipped, 0 failed.
Targeted dependent suites (`test_known_entities_registry`, `test_closets`,
`test_readme_claims`): 123 passed.
ruff check + ruff format clean on pinned 0.15.9.
CI parity replicated locally via Linux containers (Python 3.9, 3.11,
3.13): all three pass the new test and the full `test_miner.py`
sweep (46 tests).
## Why this matters
This is a quiet bug — there's no error message, no failed assertion,
just an empty `entities` field on drawers whose content actually
mentions people. The init phase looks fine ("we detected Aya, Lumi,
Ben") because init does its own case-insensitive scan; mining silently
drops the matches that init's scanner already proved are real.
Personal / journal / chat-style corpora (which use lowercase
conversational style) are the most affected. Code / docs corpora
(which use proper capitalization for names) work either way and saw
no behavior change.
Default raised from 500 to 50_000 so legitimate long-form content
(novels, scholarly editions) is not silently dropped on a typical
literary corpus. Cap configurable via MEMPALACE_MAX_CHUNKS_PER_FILE
env var or --max-chunks-per-file CLI flag; sentinel 0 disables.
Negative or non-int values from either source emit a stderr warning
and fall back to the default.
Separate counter (files_skipped_chunk_cap) and summary line surface
chunk-cap drops independently of the residual already-filed bucket;
counter fires under --dry-run too so a corpus audit shows the same
signal without writing. Skip notice routed to stderr alongside the
existing symlink-skip line.
ONNX bad_alloc protection preserved architecturally:
DRAWER_UPSERT_BATCH_SIZE=1000 bounds per-ONNX-call exposure
regardless of per-file cap. A 50_000-chunk file produces 50 forward
passes of 1_000 chunks each, identical to what 500-chunk files
already produced.
process_file return extended to a 3-tuple (drawers, room,
skip_reason); skip_reason is None on non-chunk-cap paths and
"chunk_cap" when the per-file cap triggered. Internal callers
(_mine_impl, benchmarks/mine_bench.py, tests) updated.
Closes#1455
Co-Authored-By: David Glidden <d@davidglidden.eu>
palace_graph._TUNNEL_FILE was a module-level constant initialised from
os.path.expanduser("~") + "/.mempalace/tunnels.json", ignoring the
MempalaceConfig.palace_path config (and MEMPALACE_PALACE_PATH env var)
that drawers, KG, and every other piece of palace state honour. Under
any setup where $HOME and the configured palace diverge — subagent
profiles, sandboxes, multi-tenant hosts, container mounts moving the
palace to /srv/ — drawers landed in the configured palace while
tunnels silently landed in a different file invisible to other
processes touching the same palace.
Replace the constant with _get_tunnel_file(config=None) deriving the
path from a new MempalaceConfig.tunnel_file property (sibling of
palace_path). Default install unchanged because default palace_path
is ~/.mempalace/palace whose sibling tunnels.json is the legacy path.
Add a _legacy_tunnel_file() helper and a one-line WARNING in
_load_tunnels for the case where the configured tunnel file is missing
but the pre-fix hardcoded path has one. No auto-migration — silently
merging tunnel state across two locations risks clobbering newer data.
fix(graph): validate explicit-tunnel endpoints exist (#1468)
create_tunnel previously only validated that wing/room names were
non-empty strings; nothing queried chroma to confirm at least one
drawer carried matching {wing, room} metadata. Pointing an explicit
tunnel at a phantom room silently succeeded. Combined with #1467's
read-bubble, an agent could create_tunnel → list_tunnels and have both
calls return its own bogus write, self-confirming a tunnel that didn't
exist in the shared palace.
create_tunnel now calls _check_room_exists(wing, room, col) for both
endpoints before persisting an explicit tunnel; zero rows raises
ValueError naming the endpoint. Three deliberate carve-outs:
- kind != "explicit" skips validation because topic tunnels use
synthetic topic:<name> room ids that don't correspond to real rooms
- _get_collection returning None (palace not yet created, transient
failure, tests without backend) skips validation rather than
fail-closed — matches tolerance pattern used throughout palace_graph
- Query exceptions are logged and treated as 'can't verify, allow' so
a flaky index doesn't block legitimate writes
Behaviour change: callers that previously created tunnels pointing at
empty rooms (scaffolding before mining) will now raise. File the
drawer first, then create the tunnel.
Tests:
- _use_tmp_tunnel_file helper now also neutralises _get_collection so
existing tests don't accidentally trip the new validation path when
test-order pollution leaves a real chroma backend bound
- test_closets.py::TestTunnels setup/teardown updated to monkeypatch
resolver functions instead of the removed constant; also neutralises
_get_collection for the same reason
- Three tests in test_miner.py exercising compute_topic_tunnels are
unchanged in intent — they monkeypatch the new resolvers and pass
without stubbing _get_collection because kind=topic skips validation
- New TestTunnelFileFollowsConfig and TestCreateTunnelEndpointValidation
classes cover the regression surface for both fixes
`mempalace status`, `search`, and `compress` printed the same misleading
"No palace found / Run: mempalace init" output for three distinct
palace states (no dir / no DB / no collection). Most user-visible on
the first-run state where `init` had run but `mine` had not: the hint
to re-run `init` is a no-op and wastes the user's time.
Backend gets a new typed exception `CollectionNotInitializedError`
(subclass of `PalaceNotFoundError`, transitively `FileNotFoundError`,
so legacy callers keep working). `ChromaBackend.get_collection(
create=False)` wraps chromadb's bare `NotFoundError` as the new typed
exception instead of leaking the chromadb-specific class to callers.
A new internal helper `_open_collection_or_explain` in `palace.py`
runs filesystem-first state checks before the backend call to avoid
chromadb's lazy `chroma.sqlite3` creation as a side-effect of a
read-only inspection, then catches the typed exceptions and prints a
state-specific actionable message. `BackendClosedError` is explicitly
re-raised so a programmer error is not masked as a UX hint.
Two CLI bug sites route through the helper: `miner.status` and
`cli.cmd_compress`. `searcher.search` catches the typed exceptions
directly so it can preserve the cause chain in `SearchError(...) from
e` for programmatic search-API consumers. `cli.cmd_sync` gained an
inline filesystem distinction (no helper needed: it does not use the
collection handle). `repair.status` (capacity check, which by design
must work on corrupted palaces without opening a chromadb client) got
the same distinction via `sqlite_drawer_count`-based empty detection.
The MCP `tool_status` is intentionally left alone: PR #831 already
fixed it there with a `create=True` bootstrap strategy appropriate
for programmatic clients.
Mining a transcript with --extract general was silently skipped when the
same file had already been mined with --extract exchange (or vice versa)
because file_already_mined() and prefetch_mined_set() only looked at
source_file. The two extraction modes produce different drawer content
and rooms, so they should coexist for the same source.
Changes:
- file_already_mined() and prefetch_mined_set() take an optional
extract_mode arg and only return True when stored drawer metadata
matches. Legacy drawers without extract_mode are treated as
exchange-mode for back-compat.
- _file_chunks_locked() purges only same-mode drawers when rebuilding
on a normalize-version bump, so a schema bump on one mode does not
drop drawers filed under the other.
- Drawer ids and sentinel ids include extract_mode so the two modes
cannot collide on hash.
- Pagination on the direct skip-check path so large transcripts (>1k
drawers) are classified correctly when the bulk prefetch is skipped.
Adds regression coverage for the extract-mode-aware helper, the
pagination path, and an end-to-end mine_convos run that files
exchange then general for the same transcript without skipping.
* Move SKIP log from stdout to sys.stderr, matching the existing
warning-channel pattern (the "No mempalace.yaml" notice and
topic-tunnel warnings). Stdout stays clean for "Files: N" /
"Drawers filed: N" markers that downstream callers parse.
* Show the path relative to the scan root via
Path.relative_to(...).as_posix() instead of the leaf filename, so
a nested symlink in a deep subdirectory is unambiguous and the log
renders with forward slashes on every platform.
* Wrap the print in try/except OSError, mirroring the existing guard
around filepath.stat() below; a broken pipe or closed stderr no
longer aborts the scan mid-walk.
* Document the stderr side effect in both scan_convos and
scan_project docstrings.
* Gate the new tests with @pytest.mark.skipif(sys.platform == "win32",
...) because Path.symlink_to requires SeCreateSymbolicLinkPrivilege
on Windows.
* Add dangling-symlink and nested-subdirectory tests for both miners,
locking the two-space " SKIP:" prefix, the "(symlink)" reason
marker, and the full relative path format.
Both scan_convos() and scan_project() silently drop symlinked files,
leaving callers staging a temp directory of symlinks with no diagnostic
for the "Files: 0" outcome. Keep the existing skip-symlinks behavior
and add a " SKIP: {name} (symlink)" log per skipped file, matching
the convention already used by split_mega_files.py for the
oversized-file skip case.
Chunk sizing was hardcoded via module-level constants
(CHUNK_SIZE=800, CHUNK_OVERLAP=100, MIN_CHUNK_SIZE=50). One size
does not fit all — source material varies (dense code vs prose
transcripts vs sparse logs) and so do users' context-window
budgets.
This makes all three values overridable via ~/.mempalace/config.json:
{
"chunk_size": 1200,
"chunk_overlap": 150,
"min_chunk_size": 40
}
Values are exposed as MempalaceConfig properties, threaded
through mine() -> process_file() -> chunk_text() as optional
keyword arguments. Defaults (800/100/50) are preserved when the
config keys are absent, so existing palaces behave identically.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The hook PID guard used a single global ``~/.mempalace/hook_state/mine.pid``
file, which failed two ways:
1. ``_mine_already_running`` read-then-spawn was a TOCTOU race. Two
near-simultaneous Stop hook fires both passed the existence/liveness
check before either wrote — so both ended up calling
``_spawn_mine``.
2. ``_spawn_mine`` unconditionally overwrote the global PID file with
the new child's PID. The first PID was lost, orphaning the first
child. The user-visible result in #1212 was two concurrent
``mempalace mine`` processes running against the same source, both
driving HNSW inserts in parallel — exactly the corruption pattern
the guard was meant to prevent. #1206 reported the same shape from
the perspective of the user (two mines hung on a 350MB folder).
Replace the global file with per-target slots under
``~/.mempalace/hook_state/mine_pids/``, keyed by sha256 of the mine
sub-arguments (everything after ``mine``). The slot is claimed via
``O_CREAT | O_EXCL`` so the claim is atomic — two simultaneous fires
can never both pass. Stale slots (PID exists but is dead) are
reclaimed transparently. Different targets (e.g. project mine vs
transcript ingest, or two different MEMPAL_DIRs) get independent
slots and run in parallel.
The mine subprocess receives its slot path via
``MEMPALACE_MINE_PID_FILE`` env var; ``miner._cleanup_mine_pid_file``
reads that var on exit and removes the slot if it points at our PID,
so orphaned slots from crashed mines don't accumulate.
Also routes ``_ingest_transcript`` through ``_spawn_mine`` so the
transcript ingest path now participates in the same dedup — repeated
Stop fires for the same transcript no longer stack parallel mines.
Closes#1212Closes#1206
Substring checks in path/filename routing caused systemic misrouting
in large monorepos — e.g., "views" ⊂ "interviews" sent every file
under views/ to the interviews room. Switch to separator-bounded
token matching (-, _, ., /) via a _name_matches helper, applied to
priority 1 (path parts) and priority 2 (filename).
Three small changes that together address the failure modes in #1296:
1. Add pnpm-lock.yaml and yarn.lock to SKIP_FILENAMES, mirroring the
existing package-lock.json rule. A 24K-line pnpm-lock.yaml produced
~1124 chunks in one batch and tripped onnxruntime bad_alloc on
Windows; pnpm/yarn lockfiles are no more useful to mine than npm's.
2. Skip any file that produces more than MAX_CHUNKS_PER_FILE (500)
chunks, with a clear log line. Catches the broader class — generated
CSV/JSON, build artifacts, etc. — that the named-file SKIP list will
never fully cover. The cap is conservative (500 chunks * 800 chars ≈
400 KB of source) so legitimate hand-written content still mines.
3. Print a partial-progress summary on any exception in _mine_impl, not
just KeyboardInterrupt, then re-raise. Without this, an arbitrary
exception (ONNX bad_alloc, chromadb HNSW error, OS fault) propagates
silently — the operator sees only the last progress line and assumes
the mine succeeded. The new path mirrors the KeyboardInterrupt
summary (files_processed, drawers_filed, last_file) plus the
exception type and message, then re-raises so the original traceback
surfaces and the exit code is non-zero.
Tests cover: SKIP_FILENAMES contents, the chunk-cap path returning
(0, room) with no upserts, and the new mine-aborted summary surfacing
both the partial counters and the exception class.
- Updated READABLE_EXTENSIONS in miner.py to include ".swift", ".kt", and ".kts".
- Added tests in test_miner.py to ensure scanning includes Swift and Kotlin files.
Two follow-ups against the review on this PR:
1. ``miner.load_config`` no-yaml fallback was returning the raw dirname
as the wing, while ``cmd_init`` writes ``topics_by_wing`` under the
normalized slug. A hyphenated project mined without a ``mempalace.yaml``
file silently lost every topic tunnel — same key-miss class as #1194,
just down the no-yaml branch (raised by Qodo on this PR).
2. ``convo_miner`` was applying the lower/replace rule inline at one
call site. Now folded through ``normalize_wing_name`` so all wing-slug
producers — ``cmd_init``, ``room_detector_local``, ``miner.load_config``
fallback, ``convo_miner`` — share a single source of truth. No
behavior change for any input; pure consolidation.
Added ``test_load_config_no_yaml_normalizes_hyphenated_wing`` to lock
the fallback path to the normalized slug — fails on develop without
the miner change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The pre-existing test_maybe_run_mine_prompt_declined_prints_hint
asserted the bare unquoted form `mempalace mine {tmp_path}`. After
the production code switched to shlex.quote on the resume hint, this
passed on Linux/macOS (POSIX paths have no characters that trigger
quoting) but failed on Windows where backslashes always get wrapped
in single quotes.
Mirror the production code in the assertion via shlex.quote so it's
portable across platforms; do the same for the two new
spaces-in-path tests for consistency.
The "Skipped. Run mempalace mine <dir>" hint after declining the init
prompt and the "Re-run mempalace mine <dir> to resume" hint after a
Ctrl-C interruption both interpolated project_dir without shell-quoting.
A path containing spaces or metacharacters produced a copy-paste-broken
command.
Both spots now use shlex.quote(project_dir). Adds regression tests
covering each hint with a path that contains a space.
`mempalace init` now ends with a `Mine this directory now? [Y/n]`
prompt and runs `mine()` in-process when accepted; `--yes` skips the
prompt and auto-mines for non-interactive callers. Declining prints
the resume command. Removes the "remember to type the next command"
friction since rooms + entities just got set up.
`mempalace mine` now wraps its main loop in `try / except
KeyboardInterrupt` and prints `files_processed`, `drawers_filed`, and
`last_file` before exiting with code 130 on Ctrl-C. Re-mining is safe
because deterministic drawer IDs make the upsert idempotent. The
hooks PID lock at `~/.mempalace/hook_state/mine.pid` is now actively
removed in a `finally` when its entry points at us, on clean exit,
error, or interrupt — preventing the next hook fire from briefly
waiting on a stale PID.
Closes#1181, #1182.
Previously a cross-wing topic tunnel for "Angular" stored the room as
"Angular" — colliding with a wing's literal folder-derived "Angular" room
at follow_tunnels/list_tunnels read time, and exposing raw topic strings
(which may contain characters rejected by sanitize_name) to the MCP
surface.
Topic tunnels now store their room as "topic:<original-casing>" and carry
kind="topic" on the stored dict. Explicit tunnels get kind="explicit"
(default). follow_tunnels("wing", "Angular") on a literal Angular room
no longer surfaces topic connections for the same name, and any LLM
scanning list_tunnels has a visible discriminator.
When two wings have one or more confirmed TOPIC labels in common, the
miner now drops a symmetric tunnel between them at mine time so the
palace graph reflects shared themes (frameworks, vendors, recurring
concepts).
- llm_refine: TOPIC label routes to a dedicated `topics` bucket so the
signal survives confirmation instead of getting collapsed into
`uncertain` and dropped.
- entity_detector / project_scanner: bucket plumbed through the
detection pipeline; `confirm_entities` returns confirmed topics
alongside people/projects.
- miner.add_to_known_entities: optional `wing` parameter records the
confirmed topics under `topics_by_wing` in
`~/.mempalace/known_entities.json`. Wing names do NOT leak into the
flat known-name set used by drawer-tagging.
- palace_graph: `compute_topic_tunnels` and `topic_tunnels_for_wing`
create symmetric tunnels via the existing `create_tunnel` API so they
share dedup and persistence with explicit tunnels.
- miner.mine: post-file-loop pass calls `topic_tunnels_for_wing` for
the freshly-mined wing. Failures are logged but never abort the mine.
- config: `topic_tunnel_min_count` knob (env
`MEMPALACE_TOPIC_TUNNEL_MIN_COUNT` or `~/.mempalace/config.json`),
default 1.
Tests cover topic persistence through init->mine, tunnel creation when
wings share a topic, no tunnel below threshold, cross-wing tunnel
retrieval via `list_tunnels`, dedup on recompute, case-insensitive
overlap, and the end-to-end mine-time wiring.
Out of scope for this PR (called out in the PR body): manifest-
dependency overlap, per-topic allow/deny lists, search-result surfacing.
`status()` walks `col.get(include=["metadatas"])` and buckets each drawer
into a `wing_rooms[wing][room]` histogram. The same ChromaDB return shape
fixed in the search print path — `None` entries in the `metadatas` list
for drawers with no stored metadata — crashes the status command with:
AttributeError: 'NoneType' object has no attribute 'get'
Applies the matching ``m = m or {}`` guard so None-metadata drawers roll
up under the existing `?/?` fallback bucket instead of killing the
command mid-tally. Reproduced on a 135K-drawer palace where two drawers
had `metadata=None`; both now show under `WING: ? / ROOM: ?` in the
tally while the command prints the full histogram as designed.
Adds a regression test that feeds `status()` a fake collection whose
`get()` returns a `None` in the middle of the metadatas list and asserts
both the fallback bucket and the real wing render.
entity_detector.py was refactored in #911 to load candidate patterns
from i18n locale JSON files, supporting non-Latin scripts (Cyrillic,
accented Latin, etc.). But three other code paths still hardcoded the
ASCII-only regex [A-Z][a-z]{2,}, silently missing non-Latin entity
names in metadata tagging, closet indexing, and registry lookups.
Replace the hardcoded regex with a shared _candidate_entity_words()
helper that reuses the same i18n candidate_patterns as entity_detector.
Arnold Wender
KeilerHirsch
mvalentsev
Igor Lins e Silva
Lochness
Pim Messelink
ManuelReschke
Brian potter
Milla J
yonefive71
Rahul
jp
Stephen Coogan
git
copilot-swe-agent[bot]