Third test now genuinely fails pre-fix (raw UTF-8 skip_name read), fixing the reviewer-flagged passes-either-way case. skip_names are lowercased on load, so assert on .lower().
dialect.py: all 11 text-mode open() calls (6 read, 5 write) omitted encoding=, so on a non-UTF-8-locale process (e.g. German Windows / cp1252) UTF-8-written JSON and AAAK text is decoded via the OS codepage, corrupting umlauts. config.py: 4 text opens lacked encoding= (config.json + people_map read paths, two writes); the other json.dump write paths already pinned UTF-8.
Audit findings #51 (dialect.py:360) and #84 (config.py:377). Regression: tests/test_encoding_hardening.py forces cp1252 default open and asserts umlaut round-trips through from_config / config.json read / raw-UTF-8 skip_name.
Independent review of the preceding commit found that dedup_palace's own
'Drawers: N' count() print runs a few lines before it calls
get_source_groups -- #92's palace_path guard on get_source_groups only
covered that function's internal count(), leaving this earlier, separate
call site in the same function fully exposed to the same #1222
SIGSEGV/panic class. Same fix pattern: preflight hnsw_capacity_status
before this print and abort the whole dedup_palace run on divergence.
show_stats has no equivalent exposure -- it goes straight to
get_source_groups with no separate count() print.
hnsw_capacity_status() (chroma.py) exists precisely to preflight the
#1222 SIGSEGV/pyo3-panic class before anything touches the HNSW
segment, but repo-wide it was wired into only 4 call sites while raw
count()/collection.count() is called at 20+ others -- a bare
except Exception around count() cannot catch a native crash, since
the process dies regardless of any Python try/except. This wires the
existing, already-tested probe into the 7 remaining call sites the
audit identified as CRITICAL:
- #89 palace.py::_enforce_embedder_identity -- the universal
get_collection() chokepoint every tool passes through, previously
guarded only by except Exception. Highest leverage: skips this
bookkeeping-only check on divergence instead of risking count().
- #90 migrate.py::migrate -- routes straight to the same
SQLite-extraction fallback the except branch already used, instead
of ever reaching col.count() when diverged.
- #91 repair.py::scan_palace / prune_corrupt -- both abort with the
existing from-sqlite recovery guidance instead of opening the
collection.
- #10 repair.py::rebuild_index -- preflights divergence alongside its
existing sqlite-integrity and poisoned-max-seq-id preflights, before
opening the collection.
- #13 repair.py::rebuild_index never rebuilt or reported on the
closets collection -- now warns when closets is still diverged
after a drawers-only rebuild, pointing at --mode from-sqlite instead
of letting 'Repair complete' stand unqualified.
- #92 dedup.py::get_source_groups -- takes an optional palace_path
(threaded from both callers) to preflight before count(); omitted by
existing tests, which keep their pre-existing behavior.
- #93 miner.py::status -- preflights before the ChromaDB-client
fallback path (used when the direct sqlite read is unavailable).
7 new regression tests, each confirmed failing against the pre-fix
code (via git stash of the source files only) and passing after the
fix. One existing dedup.py test updated for the new palace_path kwarg
in its call-signature assertion. Full suite: 3154 passed, 1 unrelated
pre-existing flake (test_mcp_server.py peer-writer-lock module-global
state leaking across test files in full-suite ordering -- this diff
never touches mcp_server.py).
#104 (CRITICAL, data-loss): the sweeper writes drawers with no
extract_mode at all (ingest_mode="sweep"). _metadata_matches_extract_mode's
legacy-compat rule -- "no extract_mode means treat as a legacy exchange
row" -- couldn't tell that apart from a genuine pre-schema convo_miner
row, so mempalace mine --mode convos (default extract=exchange) swept
every sweeper drawer for a shared transcript into its purge scope and
deleted them on the very next re-mine. The legacy-compat fallback now
only applies when the drawer is otherwise convo_miner's own (no
ingest_mode at all, or convo_miner's own "convos" tag) -- a drawer
positively identified as another producer's (sweep, or any other
foreign ingest_mode) never matches, using the same ingest_mode
discriminator sync.py already relies on for its own registry-row check.
#105 (MEDIUM, silent-failure): convo_miner's own instance of the
purge-failure swallow already fixed for miner.py at #23 -- a failed
purge in _file_chunks_locked was logged at debug level and mining
proceeded anyway, silently producing duplicate/stale drawers under
mixed schema versions. Now aborts (returns skipped=True, leaving the
old drawers' stored mtime untouched so the next mine retries) and
prints a visible warning.
428 tests pass across test_palace.py/test_convo_miner*.py/test_miner.py/
test_sweeper.py/test_hallways.py/test_format_miner.py/test_repair.py, no
regressions.
extract_via_sqlite drove its extraction with an INNER JOIN starting at
embedding_metadata. Any embedding with zero rows there (a sparse
historical write with no chroma:document and no other key -- the same
condition _extract_drawers already sanitizes for the collection-layer
rebuild path, see #1458) never appeared in the join result and was
silently dropped by mempalace repair --mode from-sqlite, with no count
mismatch and no warning.
Drive the query from embeddings (LEFT JOIN embedding_metadata) instead,
defaulting to an empty metadata dict when a row has none. Order by
e.id rather than em.id since em.id is NULL for unmatched rows.
Regression test seeds a real chromadb collection, then strips one
drawer's metadata rows directly via SQLite (current chromadb validates
against empty metadata on write, so this only reproduces on rows
already sitting in an older palace) and asserts the drawer still
comes back through the SQLite bypass.
cmd_repair() has a separate, inlined implementation of the same
live-swap-failure recovery that _rebuild_collection_via_temp's shared
fix (preserve the verified temp copy) already covers for rebuild_index().
Before this, cmd_repair's own except-block still did shutil.rmtree()
on the whole palace directory (destroying that same temp copy) and
restored a pre-repair full-directory backup -- printing recovery
guidance that pointed at data the same code path had just deleted.
Now both callers promote from the verified temp copy consistently.
Bare (#8)/#12: markers were internal finding IDs from a local audit
report, not GitHub references -- but they collide with real closed
issues in this repo (#8, #12) and would misdirect reviewers. Replaced
with plain prose.
Two independent bugs from the 2026-07-28 full-repo audit:
1. _rebuild_collection_via_temp deleted its own verified temp collection
in the except-handler even when the live collection had already been
replaced (live_replaced=True) -- at that point the temp copy is the
ONLY intact data left. Now preserved, and the error message points the
operator at it.
rebuild_index's own recovery path used to 'restore' by copying back a
pre-rebuild chroma.sqlite3 file backup and claim the palace was back
to pre-repair state -- but the live collection's on-disk HNSW segment
directories were already destroyed by the delete, so the restored
sqlite3 referenced segment UUIDs that no longer existed. Replaced with
a new _promote_temp_collection() that recovers by re-extracting from
the verified temp copy and re-uploading into a fresh live collection,
with an honest fallback message if that promotion also fails.
2. _paginate_ids silently returned a truncated ID list when offset-based
pagination failed and the no-offset fallback got structurally stuck
at the first page (1000 results). Callers (scan_palace, rebuild) treat
the result as the complete palace, so this could act on incomplete
data with no warning. Now cross-checks against the collection's own
count() to disambiguate a genuinely-complete page-sized collection
from a truncated larger one, and raises instead of silently truncating
when it can't tell.
Both fixes verified against an independent adversarial review pass
(mutation-tested: confirmed the new tests catch a wrong-source-collection
read and a swapped ids/documents payload in the recovery path, which the
original test set did not).
Deliberately out of scope: none of the rebuild paths hold mine_palace_lock
for their full duration, so a concurrent writer can still interleave with
a rebuild. That's a separate, larger architectural gap (queue-and-replay
vs. block-on-write), tracked for its own follow-up.
A stdio mcp_server orphaned by a dropped SSH session used to swallow
pipe errors in the generic exception handler and sleep forever on the
dead channel, holding the mine_palace flock (2026-07-10 write-path
outage, PID 23392). The loop now logs and shuts down on stdin EOF,
OSError from stdin reads (EIO/EBADF from orphaned ptys), and
BrokenPipeError/OSError on stdout writes. Shutdown returns from the
loop for a normal interpreter exit 0 — no os._exit — so held flocks
release via process teardown; a broken stdout is re-pointed at devnull
first so the shutdown flush cannot turn the exit into status 120.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DpEdb9SNWF42BpupHWBhC3
The English entity-candidate pattern in i18n/en.json backtracks
catastrophically on a long unbroken run of printable ASCII (base64,
minified JS, hashes, data URIs), pinning `mempalace mine` on a single
~5000-char window for hours. Collapse such runs to a space before
single-word candidate matching, in both consumers that apply the pattern
(palace._candidate_entity_words and entity_detector.extract_candidates).
Scoped to ASCII ([!-~]) so non-ASCII scripts stay untouched — a CJK
paragraph is one unbroken run with no ASCII whitespace, and zh-CN/zh-TW
have no multi-word fallback, so a blanket collapse would erase their
detection. Threshold 24 sits above the 20-char cap of the simple-name
pattern, so no real name is dropped.
Fixes#2063.
Co-authored-by: Ryan Wei <9876551+RyanWei@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A daemon job refused the palace write lock was marked terminal `failed` and
never retried. `mine_palace_lock` guards the palace write itself, so a refusal
means no drawer was filed, but the queue recorded it exactly like a crash
mid-execution, whose outcome is unknown. The work was dropped: it only ran
again if a hook happened to re-submit equivalent work, and a job kind no hook
re-emits was lost outright.
Refusals now defer. The job goes back to `queued` with `started_at` cleared,
the claim's attempt increment undone, and the reason recorded. The update is
scoped to the claim that was refused (started_at match), so a defer racing a
recovery re-claim cannot re-queue work it does not own. `claim_next` clears
the recorded reason so it never outlives the claim it describes. Every other
failure stays terminal: a crashed job's outcome is unknown, and blindly
re-running a non-idempotent kind would re-file verbatim content, which is what
MAX_ATTEMPTS guards.
The worker cools a refused job off in memory and moves on rather than sleeping
in line. It is the only worker and the holder keeps the lock until its write
finishes, which can be a long mine, so blocking would stall every unrelated job
behind a lock that has nothing to do with them, and a job merely queued behind
the refused one would never be claimed. `claim_next(exclude=...)` skips a
cooling job, so the oldest-first ordering cannot hand the same refused job back
forever while newer work waits. The filter runs in Python rather than an
`id NOT IN (?, ?, ...)` list, which would bind one host parameter per cooling
job against a cap that defaults to 999 before SQLite 3.32. The cooldown
lives in the worker, not the schema, which has no migrations; a restart just
retries at once, costing one refusal, never work.
`tool_diary_write` swallowed `MineAlreadyRunning` in its bare `except
Exception`, so the refusal reached the daemon with no `error_class` and
`diary_write` would still have been dead-lettered. It now uses a typed handler
ahead of the bare `Exception`, the way `tool_mine` and `tool_sync` already do.
Deferral makes a refused job non-terminal, and `DaemonClient.wait` only returns
on a terminal state, so callers that wait on purpose would have waited for a
state a parked job cannot reach: a foreground `mine --daemon` for the one-hour
default, and the `hooks_cli` pre-compaction mine and Stop-hook diary paths for
their whole timeout on every fire, each then reporting a failure that never
happened. `wait(stop_on_lock_deferral=True)` hands the parked job back instead.
The CLI echoes the global `--palace` back into the command it suggests, so the
suggestion does not silently list the default palace's queue instead of the
one the job is parked in. A job that is genuinely running is still waited out.
Co-Authored-By: mjvmsteixeira <185609735+mjvmsteixeira@users.noreply.github.com>
chunk_text's windowing loop (start = end - chunk_overlap) stops advancing
when chunk_overlap exceeds chunk_size // 2 on short-line content, looping
forever at 100% CPU. A paragraph or line boundary pull only moves end past
start + chunk_size // 2, so a pulled chunk spans more than half the chunk
size; the step then advances only while chunk_overlap <= chunk_size // 2.
Tighten the existing guard from chunk_overlap >= chunk_size to
chunk_overlap > chunk_size // 2 in both chunk_text (raise, with the numeric
bound in the message) and MempalaceConfig._validated_chunk_config (repair to
min(DEFAULT_CHUNK_OVERLAP, chunk_size // 2)). The windowing loop is
unchanged, so every already-valid config produces byte-identical chunks; 50%
overlap (chunk_overlap == chunk_size // 2) stays valid.
Fixes#2056.
chunk_text recomputed line_start/line_end with a full-prefix
content.count("\n", 0, pos) per chunk (O(N*K) overall), so large files
ground for days. Keep the emitted values byte-identical but tally
newlines incrementally over the newly-scanned span (O(N) total), with a
from-scratch fallback that keeps every value exact.
hnsw_capacity_status() runs on every search, duplicate check and status
call (MCP and CLI). Each call cost a COUNT(*) over the embeddings table
and a full unpickle of the segment metadata, both scaling with palace
size. Cache the verdict per (palace_path, collection_name) and reuse it
while an (inode, mtime_ns, size) signature over chroma.sqlite3, its -wal
sidecar and index_metadata.pickle is unchanged, so an external write
invalidates it at once rather than after a fixed window. The #1222
divergence guard stays fresh: a mid-probe write or a locked read is
returned but never cached, and the signature brackets the whole probe so
a pickle rewrite during it cannot be pinned as fresh.
normalize() joined every conversation in a Claude.ai privacy export
bundle into one string before it was hashed, so re-exporting the bundle
with one new conversation added changed the whole-file hash and the
existing conversations were re-mined as duplicates. Split normalization
into normalize_conversations() so each conversation can be hashed and
deduped on its own.
Also scope the content-hash map by wing — mining the same transcript
into a second wing is a deliberate re-file, not a duplicate, and was
leaving that wing with only the registry sentinel.
Repeated exports from Claude/ChatGPT land the same conversation under a
new filename each run (timestamped bundle, regenerated slug, etc.), so
the existing source_file/mtime skip never recognized it as already
mined and re-filed a duplicate set of drawers. Now content is hashed
after normalization and checked against previously filed hashes, so
the same conversation under a new path is skipped instead of
duplicated.
Implements the pgvector server-side implementation of BaseCollection.facet_counts(field, where, limit), completing the pgvector fast path for the aggregation contract merged in #1868 and specified in #1835.
The new real-Chroma checkpoint test dropped its create-time client with a
bare del, leaving the per-path SharedSystemClient's SQLite/HNSW handles
open on Windows (#1128). Close both clients so the temp palace is released,
matching the conftest fixture's close-not-del pattern.
mempalace_checkpoint hard-coded added_by="checkpoint" for every drawer,
dropping the filing agent's identity even though it arrives in the same
call via diary.agent_name. Add an optional top-level added_by parameter
and resolve attribution as explicit > diary agent_name > "checkpoint";
blank/whitespace/non-string values defer to the next source. The value
is declared in the tool schema so tools/call admits it on both stdio and
HTTP transports.
Fixes#2023
Co-Authored-By: epinethrone <172391900+epinethrone@users.noreply.github.com>
Hermes' memory tool defaults to target="memory" (the agent's own
notes); filtering on_memory_write to target == "user" silently dropped
the majority of writes. Mirror both targets under distinct subjects —
user→asserted for user facts, hermes→noted for agent notes — so
kg_query("user") never surfaces environment quirks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The provider filed and searched self._palace_path while the mcp_server
passthrough tools resolved mempalace's global config — a custom Hermes
palace_path searched one palace while drawer CRUD, duplicate checks,
and tunnels wrote another. Publish the resolved palace to
MEMPALACE_PALACE_PATH (mcp_server's own --palace mechanism) with
ownership tracking so a stale bridge never outranks edited config and
a user-set env var is never touched. The KG tools become native
handlers: mcp_server resolves its KG from DEFAULT_KG_PATH unless its
own CLI flag was given, which no env bridge can influence.
collection_name and the unset-palace default now defer to
MempalaceConfig — the same single chain the MCP server itself uses.
Also document that hermes backup does not cover ~/.mempalace (no ABC
hook exists for contributing external paths).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
on_session_end and on_pre_compress blind-re-filed the raw message list,
duplicating every turn sync_turn had already stored — filed_at is hashed
into the drawer id, so upserts cannot collapse the copies. Drop the
re-filing (and the pre-compress hint that over-promised persistence);
on_session_end keeps only the wake-up cache refresh.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- _scan_metadatas: fetch cap+1 and compare len > cap, so a collection
holding exactly STATUS_SCAN_LIMIT rows is no longer reported as
truncated (the view is complete). Callers still get at most cap rows.
- status/list_wings/list_rooms: tolerate None metadata entries from
legacy palaces / raw writers instead of failing the tool call.
- _match_wing_by_keywords: skip non-string keywords so a hand-edited
wing_config.json can't break live turn filing.
- file_conversation_exchange: extra_metadata can no longer overwrite
canonical keys (matches the documented append-only contract), and
wing/room are validated with sanitize_name — invalid names fall back
to wing_general / conversations rather than dropping the turn, per
the verbatim-first mandate.
- Fix two stale docstrings left from the pre-split layout.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Hermes provider from feat/hermes-integration, split out per review
on #1684 — provider + tests only; backfill, the hermes install CLI,
and docs follow in a stacked PR.
Changes vs the original branch:
- _file_turn routes through convo_miner.file_conversation_exchange()
instead of a hand-rolled col.upsert, so live turns carry canonical
drawer metadata and the ids.py ID recipe.
- The backfill/live wing-routing parity test moves to the backfill PR.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Live agent integrations and their backfills need to file one
conversation exchange at a time, but hand-rolling the upsert leaves
drawers without hall / entities / filed_at / extract_mode metadata —
silently invisible to hallway traversal, entity search, and the
since/before date filters.
file_conversation_exchange() builds the same metadata the convo miner
writes, and make_exchange_drawer_id() moves the ID construction into
ids.py per its single-source-of-truth contract (full-content hash, no
prefix collisions; filed_at keeps repeated exchanges distinct).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
KeilerHirsch
Sandro da Silva
Igor Lins e Silva
xcarbo
Grace Gettert
mvalentsev
rohan richard
Pim Messelink
raman325