normalize() joined every conversation in a Claude.ai privacy export
bundle into one string before it was hashed, so re-exporting the bundle
with one new conversation added changed the whole-file hash and the
existing conversations were re-mined as duplicates. Split normalization
into normalize_conversations() so each conversation can be hashed and
deduped on its own.
Also scope the content-hash map by wing — mining the same transcript
into a second wing is a deliberate re-file, not a duplicate, and was
leaving that wing with only the registry sentinel.
Repeated exports from Claude/ChatGPT land the same conversation under a
new filename each run (timestamped bundle, regenerated slug, etc.), so
the existing source_file/mtime skip never recognized it as already
mined and re-filed a duplicate set of drawers. Now content is hashed
after normalization and checked against previously filed hashes, so
the same conversation under a new path is skipped instead of
duplicated.
- _scan_metadatas: fetch cap+1 and compare len > cap, so a collection
holding exactly STATUS_SCAN_LIMIT rows is no longer reported as
truncated (the view is complete). Callers still get at most cap rows.
- status/list_wings/list_rooms: tolerate None metadata entries from
legacy palaces / raw writers instead of failing the tool call.
- _match_wing_by_keywords: skip non-string keywords so a hand-edited
wing_config.json can't break live turn filing.
- file_conversation_exchange: extra_metadata can no longer overwrite
canonical keys (matches the documented append-only contract), and
wing/room are validated with sanitize_name — invalid names fall back
to wing_general / conversations rather than dropping the turn, per
the verbatim-first mandate.
- Fix two stale docstrings left from the pre-split layout.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Conversation transcripts were assumed immutable once mined: the bulk
skip-check (prefetch_mined_set) only tracked "have we seen this
source_file before at the current normalize_version", with no mtime
comparison at all. That's wrong for how Claude Code sessions actually
work -- a session keeps appending to its own JSONL file while active,
and /compact or /clear can rewrite one in place. Once a session file
was mined, any content appended after that point would silently never
get mined, with no error or warning -- the file just looked
"already filed" forever.
palace.py:
- prefetch_mined_set() now returns dict[source_file, stored_mtime]
instead of a bare set[source_file]. `if src in mined_set` still works
identically (dict `in` checks keys), so this is a source-compatible
change for that access pattern; a caller that wants staleness
detection reads mined_set[src] and compares against the file's
current mtime itself. None means no mtime was ever stored (or
getmtime failed when the drawer was written) and must be treated as
stale, not "unknown, assume unchanged".
- Removed bulk_check_mined(): it already existed for exactly this
purpose (bulk mtime prefetch) but had zero callers anywhere in the
codebase and was missing the normalize_version/extract_mode filtering
prefetch_mined_set has -- folded its intent into prefetch_mined_set
instead of maintaining two subtly-different, overlapping bulk scans
over the same underlying data.
- file_already_mined()'s docstring corrected: it previously claimed
"transcripts are assumed immutable" for convo mining. That's no
longer true; corrected to describe the actual current split (convo
miner's bulk skip-check uses prefetch_mined_set's stored mtimes; this
function's check_mtime=True path is now only its per-file,
lock-held race-condition recheck).
convo_miner.py:
- New _is_unchanged_since_last_mine() helper (extracted to keep
_mine_convos_impl under the repo's cyclomatic-complexity gate):
false whenever the file isn't in the prefetched map, its stored mtime
is None, getmtime fails, or the mtimes don't match -- true only when
genuinely unchanged.
- _file_chunks_locked's metadata now stamps source_mtime on every real
drawer (mirroring miner.py's existing pattern), and its in-lock
recheck now passes check_mtime=True.
- _register_file's 0-chunk sentinel also stamps source_mtime, so a
short file that later grows past MIN_CHUNK_SIZE is detected as
changed instead of being skipped forever by the sentinel.
One-time cost worth flagging: no existing convo drawer has source_mtime
stored (this field never existed for convo mining before now), so the
first `mempalace mine --mode convos` after this ships will see every
already-mined file as stale and fully re-mine it. Not a bug --
_file_chunks_locked's existing purge-before-insert means no
duplication results -- just a real, one-time cost across a large
corpus.
tests/test_convo_miner.py: 7 new tests -- grown-file re-mine picks up
new content, unchanged file still skipped (the mtime check must not
regress the existing optimization), grown-file re-mine purges stale
drawers rather than accumulating duplicates (checked via unique content
markers, not raw counts -- ChromaDB collections can carry unrelated
bookkeeping rows), prefetch_mined_set's returned mtime matches the real
file, None handling for a drawer with no stored mtime, a legacy
drawer (no source_mtime field, simulating pre-this-change data) is
correctly re-mined rather than skipped forever, and the sentinel path
stamps source_mtime too.
Full suite: 3327 passed, 20 skipped, 0 failed. ruff check / ruff
format -- clean.
Resolves conflicts from 128-commit divergence:
- mempalace/convo_miner.py imports: kept both `mine_palace_lock` (this PR)
and `prefetch_mined_set` (develop).
- mempalace/convo_miner.py docstring: kept this PR's lock-wrapping
description, added a one-line pointer to the chunking-config section
whose body now lives in `_mine_convos_impl`.
- mempalace/convo_miner.py body: develop placed `cfg_chunk_size` /
`cfg_min_chunk_size` setup inline in `mine_convos`. This PR factored
the body into `_mine_convos_impl`, so the inline setup would have
left `cfg_chunk_size` referenced-but-undefined inside the impl.
Moved the `MempalaceConfig()` setup into `_mine_convos_impl` so the
variables are in scope where they're used.
- tests/test_convo_miner.py: kept both additive test sets (lock
concurrency from this PR + wing_api auto-routing from develop).
Local: ruff check / format pass; full pytest suite passes
(2103 passed, 3 skipped).
Co-Authored-By: Igor Lins e Silva <4753812+igorls@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
One-time mechanical reformat so `ruff format --check .` passes under the
newly pinned ruff. Layout only (assert-message parenthesization etc.),
no behavior change. 29 files: 28 under tests/ + 1 tools helper, no core
mempalace/ modules. Produced by `ruff format .`.
CI pins ruff>=0.4.0,<0.5 (resolves 0.4.10); one new assert block was
laid out in ruff-0.5+ style, failing `ruff format --check .`.
Reformat with the exact CI ruff version (assertion layout only, no
semantic change). `ruff check .` + `ruff format --check .` both pass
under 0.4.10.
Mining a transcript with --extract general was silently skipped when the
same file had already been mined with --extract exchange (or vice versa)
because file_already_mined() and prefetch_mined_set() only looked at
source_file. The two extraction modes produce different drawer content
and rooms, so they should coexist for the same source.
Changes:
- file_already_mined() and prefetch_mined_set() take an optional
extract_mode arg and only return True when stored drawer metadata
matches. Legacy drawers without extract_mode are treated as
exchange-mode for back-compat.
- _file_chunks_locked() purges only same-mode drawers when rebuilding
on a normalize-version bump, so a schema bump on one mode does not
drop drawers filed under the other.
- Drawer ids and sentinel ids include extract_mode so the two modes
cannot collide on hash.
- Pagination on the direct skip-check path so large transcripts (>1k
drawers) are classified correctly when the bulk prefetch is skipped.
Adds regression coverage for the extract-mode-aware helper, the
pagination path, and an end-to-end mine_convos run that files
exchange then general for the same transcript without skipping.
The project-files mine path (miner.mine) has wrapped _mine_impl in
mine_palace_lock since #1264 — a non-blocking flock that raises
MineAlreadyRunning so the second runner exits cleanly instead of
queueing as a waiter that drives parallel HNSW inserts. The convos
mine path (convo_miner.mine_convos) was missing the same guard.
In practice this meant any caller that spawned `mempalace mine
--mode convos` repeatedly against the same palace — most notably
the Stop-hook transcript ingest before the per-target PID slot
landed — could stack up arbitrarily many concurrent mines, each
holding a ChromaDB client open, each writing to the same HNSW
index. Recently observed: 28 stuck convos mines on one machine
consuming ~18 GB of RAM and contributing to a load spike.
Fix: refactor mine_convos into a thin wrapper that holds the
per-palace flock around _mine_convos_impl, mirroring miner.mine
exactly. Dry-run skips the lock since it never writes.
Tests: two cross-process tests in tests/test_convo_miner.py —
one asserts MineAlreadyRunning when a child process holds the
lock, one asserts dry-run is unaffected. Same spawn-context
pattern as test_palace_locks.py (fork-with-chromadb deadlocks
on Python 3.13).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When mempalace mine --mode convos is invoked against a directory inside
a known AI-tool storage path (Claude Code, Codex CLI, Gemini CLI), the
destination wing now auto-defaults to wing_api rather than the directory
basename. Conversations from external API-keyed tools land grouped under
a single dedicated wing for visibility.
Detected paths (exact-segment match — substrings like .gemini-backup or
.codex-archive do NOT match):
- any segment .codex (Codex CLI sessions / archives)
- any segment .gemini (Gemini CLI sessions under ~/.gemini/tmp/...)
- the consecutive segment pair .claude/projects (Claude Code).
.claude alone is NOT matched - that is the settings/config dir,
not a conversation source.
Wing-resolution precedence (first match wins):
1. Explicit --wing argument from the user - always wins
2. AI-tool path detection -> wing_api
3. Basename fallback (existing behavior, unchanged)
Two new helpers split out of mine_convos for unit-test coverage:
- _is_ai_tool_path(path: Path) -> bool
- _resolve_wing(convo_path: Path, wing: Optional[str]) -> str
mine_convos now calls _resolve_wing in place of its inline basename
logic. No other call sites or downstream consumers change.
Test coverage:
- 15 unit tests covering positive matches (Claude Code subdir + root,
Codex root + sessions, Gemini root + chats), negative cases
(.claude alone is settings dir, unrelated paths, substring no-match
on .gemini-backup / .codex-archive), explicit --wing override,
auto-route trio, basename fallback, empty-string-as-no-wing.
- End-to-end smoke test (manual): real-shape Claude Code JSONL fixture
mined via the actual CLI; sqlite read-back of /tmp palace confirms
drawers landed with wing='wing_api' and verbatim content preserved;
mempalace search --wing wing_api returns expected content ranked.
- Full pytest sweep: 1388 baseline + 15 new = 1403 passed, zero
regressions.
Design context:
This change reflects Aya's product call that conversations from
API-keyed AI tools should land in a structural wing_api rather than be
scattered across topical wings derived from directory basenames. Igor's
ADR-0017 in mempalace-ts proposes the alternative of source-prefix
metadata (source LIKE 'api/%') with topical wing assignment instead;
that approach has architectural merit (wings stay topical) but does not
deliver the single-wing visibility users get here. Open for review
discussion - explicit --wing flag and basename fallback both unchanged,
so this is additive and reversible.
Closes part of #59 for the auto-routing UX.
Without this, the strip_noise improvement only helps new mines. Every
user who had already mined Claude Code JSONL sessions would keep their
noise-polluted drawers forever, because convo_miner's file_already_mined
skip short-circuits before re-processing.
Adds a versioned schema gate so upgrades propagate silently:
- palace.NORMALIZE_VERSION=2 — bumped when the normalization pipeline
changes shape (this PR's strip_noise is the v1→v2 bump).
- file_already_mined now returns False if the stored normalize_version
is missing or less than current, triggering a rebuild on next mine.
- Both miners stamp drawers with the current normalize_version.
- convo_miner now purges stale drawers before inserting fresh chunks
(mirrors miner.py's existing delete+insert), extracted into
_file_convo_chunks helper to keep mine_convos under ruff's C901 limit.
User experience: upgrade mempalace, run `mempalace mine` as usual, old
noisy drawers get silently replaced with clean ones. No erase needed,
no "you need to rebuild" changelog footgun.
Tests:
- test_file_already_mined_returns_false_for_stale_normalize_version —
pins the version gate contract for missing/v1/current.
- test_add_drawer_stamps_normalize_version — fresh project-miner drawers
carry the field.
- test_mine_convos_rebuilds_stale_drawers_after_schema_bump — end-to-end
proof that a pre-v2 palace gets silently cleaned on next mine, with
orphan drawers purged and NOT skipped.
Existing test_file_already_mined_check_mtime updated to include the
new field; all other tests unaffected.
* fix: register 0-chunk files to prevent re-processing on every mine (#654)
mine_convos() has three early-exit paths (OSError, content too short,
zero chunks) that skip writing anything to ChromaDB. Since
file_already_mined() checks for the presence of a document with a
matching source_file, these files are re-read and re-processed on
every subsequent run.
Add _register_file() that upserts a lightweight sentinel document
(room="_registry", ingest_mode="registry") so file_already_mined()
returns True on future runs.
Note: Bug 2 from the issue (drawers_added counter always 0) was
already resolved upstream via the switch from collection.add() to
collection.upsert().
* fix: resolve macOS path symlink in test + remove unused variable
- Add tests for config, convo_miner, spellcheck, knowledge_graph
- Fix Windows PermissionError in test cleanup (chromadb file locks)
- Add UTF-8 encoding to split_mega_files, entity_registry, hooks_cli
- Fix mcp_server parse_known_args logging for unknown args
- Set coverage threshold to 85 in pyproject.toml and CI
- Reset all version files to 3.0.11
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
raman325
Igor Lins e Silva
Sandro da Silva
rohan richard
J.R. Murray
mvalentsev
Milla J
Rahul
Fergus Ching
Tal Muskal
bensig