Close the last open items on #743 before merge:
- Conformance: document two isolation arms (cross-id for all backends;
same-id/different-namespace for supports_namespace_isolation advertisers).
- No silent drop: non-advertising backends must raise UnsupportedCapabilityError
when PalaceRef.namespace is set, rather than accept-and-ignore.
- Wire require_namespace_support() into chroma/sqlite_exact; add conformance test.
- Refresh implementation-status banner now that #1727/#1731/#1732/#1734 landed.
#757 landed mtime/inode cache invalidation and mempalace_reconnect
in mcp_server._get_client(). Both are Chroma-specific (stat of
chroma.sqlite3). They should migrate into ChromaBackend.get_collection
and ChromaBackend.close_palace during the §10 cleanup so the freshness
contract lives inside the backend, not in the caller.
Addresses the actual spec defects flagged in #743 review, ignoring
operator-UX asks that are not plugin-contract concerns.
- Goal #3: 'without data loss' → mirrors §8.2's capability-conditional
lossless-vs-reembed framing. No more overpromise.
- §1.5: `server_embedder` is no longer an implicit escape hatch from
identity/dimension rules. Such backends MUST expose an effective
identity via `effective_embedder_identity()` and are bound by the
same three-state check.
- §7.3: adds `maintenance_kinds: ClassVar[frozenset[str]]` advertisement
mechanism. `run_maintenance(kind)` must raise
UnsupportedMaintenanceKindError for unadvertised kinds. Benchmark
harness reads this set rather than guessing kind names. Reserves
`analyze`/`compact`/`reindex` as well-defined names.
- §1.2: adds `update()` as optional method with a default get+merge+
upsert implementation. §2.1: `supports_update` redefined to gate
atomic single-round-trip semantics (not mere capability), since the
default impl already supports partial updates.
Operator asks explicitly NOT adopted (diplomatic shims, not contract
defects): `.to_dict()` compat on typed results, migration progress
reporting, `BaseBackend.repair()` separate from `run_maintenance`,
per-palace capability variance, identity recording on read-only ops.
Incorporates review feedback from skuznetsov (Postgres, #665) and
dekoza (Lance, #574) on issue #737:
- §1.5: split 'accepts embeddings=' (signature compliance) from
'persists embeddings as-is' (correctness). Adds
supports_embeddings_passthrough capability; the former is universal,
the latter is required to label a migration lossless.
- §1.5: model identity check becomes a three-state machine
(known_match / known_mismatch / unknown) so legacy palaces without
recorded identity don't hard-fail on upgrade.
- §1.4: makes explicit that supports_contains_fast is the ONLY
performance floor the spec promises; without it callers MUST assume
O(n). $contains is a correctness requirement, not a performance one.
- §3.3: clarifies auto-detect is an upgrade-compat path only, never
the selection mechanism for new palaces.
- §8.2: migrate CLI refuses to run against a target lacking
supports_embeddings_passthrough unless --accept-re-embed is passed;
migration record now captures lossless status and model identities.
Copilot review flagged back-references in §1.4 and §6 that still used
the pre-skuznetsov-rename names (`$contains_fast`, `sync_capable`,
`change_feed`). Updated to the `supports_*` prefix used in the §2.1
capability table.
Formalizes the BaseCollection/BaseBackend contract introduced as a seam
in #413 into an interchangeability spec that third-party backends can
build to. Driven by six in-flight backend PRs (#574, #643, #665, #697,
#700, #381) each implementing the interface differently.
Key decisions captured: entry-point distribution, typed QueryResult/
GetResult replacing Chroma dict shape, daemon-first multi-palace model
via PalaceRef, required where-clause subset (incl. $contains),
mandatory embedder injection with model-identity validation, capability
tokens, shared pytest conformance suite, and a backend-neutral
migrate/verify CLI.
The website pages arrived from the full RFC 004 branch and documented the
memory op-log, `mempalace oplog`, and automatic bidirectional memory
convergence. None of that is in this slice: the CLI has logstream, artifact,
and replica, and memory moves by one-way `replica pull`.
- cli.md: drop the `mempalace oplog` section (the command does not exist);
note that full convergence is staged.
- replicated-palace.md: split the sync table into shipping (coordination,
multi-master) vs next (memory ops, bidirectional); say plainly that two
machines capturing independently do not merge yet.
- shared-brain.md: mark step 3 a pull you re-run, not a converging loop.
- RFC 004: record which steps landed.
Identity becomes the triple host:agent:project — the granularity at which an
actor actually shares filesystem, local config, and knowledge. Two sessions in
the same project on the same box are the SAME actor with the SAME identity;
session/PID is event metadata, never identity. Fixes the live collision where
two windows-claude sessions interleaved inboxes, claims, and diary under one
flat name.
Two decisions resolved (both live in the backend/hub):
- Matcher: NO change. Route the tuple as an opaque string on RFC 003's exact +
'*' broadcast; colons already pass _sanitize_routing. Hierarchical/glob
routing is deferred — it's a real index/correctness surface and no concrete
need has landed. This ships as a convention + one renderer change, zero risk
to the routing hot path.
- Order: host:agent:project (host-first), matching how mac-*/windows-*/blade-*
names already read; migration is a pure suffix append, not a rename.
Claim safety under a shared identity needs no lease server: natural mutex
(don't claim what your identity already claimed) plus RFC 004 R3's earliest-HLC
tiebreak for the residual race. The durable fix is one place — the renderer of
the mempalace-shared-brain block emits the tuple, so every box re-derives on
next sync instead of being hand-edited. Deferred behind the RFC 004 write-flip;
touches no storage or merge semantics.
(cherry picked from commit b043c29fd6a65ea32f92252b8afab1b62bc95bf5)
Revisions kept forever (no GC path; search surfaces heads only); v4
migration staged on a copy before live-with-backup; step 2 ships as 2a
(drawers+KG ops) then 2b (registry/hallways); mac origin dual-writes
during a shadow period before cutover; remote write-flip (step 3) gated
on the local-capture promotion validating end-to-end.
(cherry picked from commit 34a264e12bdc3ba6f40877972e9435e3f912ee6d)
Driven by a real customer: the Windows machine holds un-mined projects and
conversations. Capture must happen where the data lives (mining reads local
disk), so step-1 replicas may mine locally now — unstamped drawers are
untouchable by reconciliation — and step 2 commits to a one-time promotion
pass turning them into drawer.add ops under the replica's identity.
Content-pure v4 ids double as the cross-machine dedup: identical content
mined on two origins collapses to one drawer at merge.
(cherry picked from commit d18b2a03519a1cfbe9e10bbf12c902c307195c15)
Transport/lifecycle sections (§5), transport-side security (§10.1, §10.3),
and the PalaceMind appendix authored by windows-claude and delivered as a
patch artifact over the logstream (correlation
rfc004_replicated_palace_position, artifact art_20260702T081251_76aef8fb111a,
sha256 verified, applied at ab6f362). §10.2 at-rest encryption drafted by
mac-claude on apply: per-replica data keys that never traverse the mesh,
two-tier design (FDE baseline + SQLCipher-style page encryption over the
canonical stores, OS-keystore wrapped), replica-local rotation, R7
composition with the revocation ceremony.
The seam contract (§5.1) verifies against the Layer 2 design: request/stream
map to anti-entropy pull + SSE push-notify, and ReplicaId doubles as
origin_replica so provenance and authentication are one fact. RFC 004 is now
fully drafted and awaiting Igor's review.
(cherry picked from commit 9f3333998673fb196664325396c104036f771d1f)
One logical palace per human, fully replicated per machine: agents always
local, services converge via provenance-stamped ops over an encrypted
mesh. Judged against offline as the default posture (R0-R8).
Storage sections drafted (mac-claude ownership per the logstream debate on
correlation rfc004_replicated_palace_position): canonical op-log with HLC +
per-origin version vectors, complete merge-semantics table, v4 content-pure
id migration (verified mutable-state inventory with file refs), backends
demoted to fold-and-index consumers, provenance/source-bound maintenance,
4-step sequencing starting with logstream multi-master. Transport (§5),
transport-side security (§10), and the PalaceMind appendix are reserved
stubs for windows-claude, delivered as patch artifacts to keep a single
committer per repo.
(cherry picked from commit ab6f362af7a2ce947c2c37c11910fa8aeb6b983d)
GET /logstream/stream on the hub HTTP transport: bearer-authenticated
Server-Sent Events with the event_list filter set, since_event_id /
Last-Event-ID replay-then-tail resume (live tail only without a cursor,
via new Logstream.latest_event_id), ~15s heartbeat comments, and a
bounded client pool (MEMPALACE_SSE_MAX_CLIENTS, default 8; 503 +
Retry-After on cap). Frames carry the exact event_list JSON envelope —
the compat surface promised to the PalaceMind desktop viewer.
Also fixes a latent hub-starvation hazard the first SSE consumer would
have tripped: do_POST serialized every request under the global HTTP
lock, so one agent's five-minute event_wait long-poll blocked the whole
fleet. Logstream tools (own WAL db, no Chroma/KG state) now dispatch
lock-free via _http_dispatch; everything else keeps the global lock.
Requested by claude-fable-5-windows for the PalaceMind live viewer
(correlation palacemind_desktop_logstream_viewer). Full suite: 3411
passed.
(cherry picked from commit a7c720507974acc2f02957ac24b7045503ad74da)
Patch authored by windows-codex and handed off through the coordination
layer itself: correlation_id task_logstream_e2e_dogfood, patch artifact
art_20260702T041319_cdf92e302efe (sha256 a17012d6...9f64c4, verified),
test log artifact art_20260702T041254_c46525e2d550 (72 passed on
Windows 11, Python 3.12.11). Applied on mac, re-verified (72 passed),
acked applied. Trailing-newline nit in the submitted diff repaired on
apply (producer stripped the final newline when storing the artifact).
(cherry picked from commit 38f3d35804155840cd5512411e9823fb38dc7e95)
Durable append-only event + artifact layer in logstream.sqlite3 inside
the active palace dir, so agents can delegate work, wait for replies,
and exchange exact patches through the shared hub.
Phase 1 (mempalace/logstream.py): events, artifacts, event_artifacts
per the RFC schema; WAL + per-instance lock, no Chroma dependency;
append/list/wait/ack/put_artifact/get_artifact/submit_patch; explicit
size limits (body 256 KiB, artifact 4 MiB, metadata 64 KiB); rowid-backed
exclusive since_event_id cursor exposed as seq; inclusive
since_created_at; to_agent filters match '*' broadcasts; artifact_ids
validated in-transaction so listed events never dangle.
Phase 2 (mcp_server.py): seven MCP tools. Mutating four join
_MUTATING_TOOLS (hidden + refused in --read-only). All seven are exempt
from the Chroma SQLite integrity gate; the mutating four are exempt from
the peer-writer lease via _PEER_WRITER_EXEMPT_TOOLS (logstream owns its
own WAL db, and a running mine must not block coordination).
mempalace_reconnect drains cached Logstream handles like the KG cache.
(cherry picked from commit cdf72a994133242da862b11b21033a4546c5e146)
* feat(convo): preserve authored timestamp from transcripts
Conversation drawers only carried `filed_at` (ingest time), so a bulk
re-mine collapsed every drawer to a single instant and the chronological
signal was lost — even though each Claude Code / Codex JSONL line already
carries an ISO-8601 `timestamp`. The recency-window fallback and any
date-aware consumer then saw ingest order, not when content was written.
- convo_miner: derive `authored_at` (per-file max line `timestamp`) and
store it as drawer metadata; falls back to `filed_at` when absent
- searcher: surface `authored_at` in search results, and break exact
hybrid-score ties toward the more recently authored drawer (ISO strings
sort chronologically; missing dates sort oldest) — benchmark-neutral as
it only reorders exact ties
- tests: cover `_extract_authored_at` (latest wins, skips/tolerates lines
without timestamps, non-jsonl/missing -> None) and the tie-break
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(search): surface authored_at in CLI + backfill for existing data
Completes the authored_at work so the field is visible end-to-end and
existing palaces can adopt it without re-mining.
- layers: CLI `search` output shows an `authored:` date line per result
(peer of the existing date; markdown drawers fall back to filed_at)
- scripts/backfill_authored_at.py: in-place migration that stamps
authored_at on convos drawers from their source transcripts — metadata
only (no re-embedding), idempotent, dry-run by default
- docs/authored-at.md: documents created_at (ingest) vs authored_at
(written) and both backfill paths (in-place / drop-and-recreate)
- tests: backfill integration tests over an ephemeral ChromaDB collection
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(search): address review — non-string timestamp guard + top-level authored_at tiebreak
Two correctness fixes from the PR review:
- _extract_authored_at: only compare when the parsed `timestamp` is a str.
A non-string timestamp on a malformed/foreign JSONL line previously raised
TypeError outside the try and could crash the mine.
- _hybrid_rank: the tie-break read `authored_at` only from nested `metadata`,
but the search_memories path (MCP / Claude Code) carries it at the top level
of each hit — so the tie-break silently no-op'd there. Read both shapes.
- tests: non-string timestamp cases, and a top-level-shape tie-break test
(which fails before this fix).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* style: apply ruff format to authored_at changes
CI ruff format --check flagged 4 files; ruff check already passed.
Formatting only — no behavior change.
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Igor Lins e Silva <4753812+igorls@users.noreply.github.com>
Address review on #1698: committing the version bump straight to main
bypasses branch protection and drifts develop behind. Bump on develop
first; it reaches main via the develop -> main merge.
Publish to PyPI on a published GitHub Release via Trusted Publishing
(OIDC — no stored token), gated by the `pypi` environment's manual
approval. The build job verifies the release tag is reachable from main
and matches mempalace/version.py before building the sdist + wheel; a
separate publish job holds the id-token scope and does the upload.
Documents the one-time setup (PyPI trusted publisher + `pypi`
environment) and the per-release runbook in docs/RELEASING.md.
PR #1555 (format coverage + virtual line numbering) merged with twelve
inline polish comments from Copilot + gemini-code-assist that weren't
load-bearing enough to block the original ship but are real cleanups.
This PR addresses them.
Twelve items in scope; one item (drawer ID delimiter — Copilot #13) is
deferred to its own dedicated PR because it's a breaking schema change
that requires migration design beyond the scope of a polish PR.
## Behavioral fixes (5 items, RED-tested first)
1. **FileNotFoundError vs broken symlink (Copilot #8).** ``extract_text``
previously mapped every ``FileNotFoundError`` from ``stat()`` to
``SKIP_BROKEN_SYMLINK``. That's misleading for the common case of a
regular file deleted between scan and extract. Now distinguishes:
``SKIP_BROKEN_SYMLINK`` only when ``p.is_symlink()`` is true;
``SKIP_UNREADABLE`` otherwise.
2. **``file_already_mined`` extract_mode scoping (Copilot #11, #12).**
Both call sites in ``mine_formats`` and ``_file_chunks_locked`` now
pass ``extract_mode="format"``. Previously the format miner could
falsely treat drawers from project / convo miner on the same source
file as "already mined" (and vice versa). Scopes idempotency to the
correct drawer subset.
3. **Sentinel skip for transient missing-dep statuses (Copilot #14).**
New ``_TRANSIENT_MISSING_DEP_STATUSES`` set + ``_register_skip_sentinel_if_appropriate``
helper. Skip variants like ``SKIP_NO_MARKITDOWN`` /
``SKIP_NO_STRIPRTF`` / ``SKIP_MISSING_FORMAT_DEPS`` /
``SKIP_NETWORK_TIMEOUT`` no longer write the "already-mined" sentinel.
Otherwise installing the missing extra later wouldn't trigger a re-mine.
4. **Outer ``except Exception`` in ``mine_formats`` (Gemini #5).** The
outer try around the loop previously caught only ``KeyboardInterrupt``,
leaving any setup-time error (e.g., ``scan_formats`` raising) to
propagate as a bare traceback. Now catches ``Exception`` defensively,
logs it, prints a partial-progress summary, and lets the ``finally``
PID-cleanup run. Mirrors miner.py's belt-and-suspenders pattern.
5. **Thread user's ``chunk_size`` / ``chunk_overlap`` / ``min_chunk_size``
through to ``chunk_text`` (Gemini #3).** ``MempalaceConfig`` was loaded
only to validate readability; users who tuned their config saw no
effect in format-mode mining. Now properly threaded.
## Trivial cleanups (5 items)
6. **Path expanduser in ``extract_text`` (Copilot #7).** ``Path(path)`` →
``Path(path).expanduser()`` so CLI inputs like ``~/docs/file.pdf``
resolve correctly.
7. **Path expanduser+resolve in ``scan_formats`` (Copilot #9).** Same
fix; ``~/docs`` and relative paths now work consistently.
8. **Use resolved ``format_path`` in ``mine_formats`` (Copilot #10).**
``scan_formats(format_dir)`` → ``scan_formats(format_path)`` so the
already-resolved path is used.
9. **``render_with_line_numbers`` type annotation (Copilot #15).**
``text: "str | None"`` reflects the documented + tested ``None``
handling.
10. **Test + docs claims (Copilot #16, #17, #18).** Stale framings
removed:
- ``docs/format-coverage.md`` — 14 fringe cases + "see the file for
the current test inventory" (no more frozen test count).
- ``tests/test_line_numbers.py`` — drops "proposed for mempalace
3.3.6" + "run from the proposal directory" references.
- ``tests/test_format_miner.py`` — drops "MarkItDown is mocked
throughout" (live integration tests exist) + proposal-directory
framing.
## Module-level hoists (enables clean test patching)
- ``MempalaceConfig`` (from ``.config``) hoisted from lazy local import
to module-level so tests can patch ``mempalace.format_miner.MempalaceConfig``.
- ``chunk_text`` (from ``.miner``) hoisted similarly.
Both follow the pattern PR #1565 used for ``compute_hallways_for_wing``.
## Complexity refactor
Extracted ``_print_mine_summary`` from ``mine_formats`` so the orchestrator
stays under the project's ``max-complexity = 25`` ceiling (per
``pyproject.toml [tool.ruff.lint.mccabe]``). Behavior unchanged; pure
extraction.
## Out of scope (intentionally deferred)
- **Drawer ID delimiter collision (Copilot #13)** — ``f"{source_file}{chunk_index}"``
can theoretically collide (``"/path/a1" + "23"`` == ``"/path/a" + "123"``).
Fixing this is a breaking schema change to drawer IDs and requires a
migration plan; will land as its own PR after design.
- The four bot comments that were ALREADY addressed by amendment #3
before the PR #1555 merge (``_SKIP_DIRS`` dedup, ``scan_formats``
symlink skip, ``source_mtime`` tracking, hall+entities metadata) —
no action needed; verified during audit.
## Tests (RED-first)
Six new RED-first tests in ``tests/test_format_miner.py``:
test_extract_text_nonexistent_regular_file_returns_unreadable_not_broken_symlink
test_mine_formats_passes_extract_mode_format_to_file_already_mined
test_mine_formats_does_not_write_sentinel_for_skip_no_markitdown
test_mine_formats_does_not_write_sentinel_for_skip_missing_format_deps
test_mine_formats_catches_unexpected_exception_and_prints_summary
test_mine_formats_threads_chunk_size_from_user_config
All six RED before this commit (failures correctly identified the bugs
they're targeting), all six GREEN after.
One existing test (``test_mine_formats_continues_after_per_file_error``)
updated to patch the new module-level binding
``mempalace.format_miner.chunk_text`` instead of the old
``mempalace.miner.chunk_text`` source location, and to accept the
``**kwargs`` the call now passes through. Behavior unchanged.
## Verification
pytest -q (full mempalace suite)
→ 2065 passed, 3 skipped, 0 regressions
ruff check mempalace/format_miner.py mempalace/searcher.py tests/
→ All checks passed!
ruff format --check ...
→ 4 files already formatted (pinned 0.15.9)
mine_formats complexity
→ ≤ 25 (under the project ceiling)
Two additive features, both following the same read-time-transform pattern:
1. Virtual line numbering — new render_with_line_numbers() and
extract_line_range() in mempalace/searcher.py. Closet pointers like
2026-01-18:L55-L72 resolve to drawer slices rendered with [55] through
[72] line prefixes without modifying any stored content. Lines already
prefixed with [<digits>] pass through unchanged. Pure functions, no I/O.
2. Format coverage (mempalace mine --mode extract) — new mempalace/format_miner.py
reads binary office formats and files drawers via the lock + purge +
upsert pattern convo_miner uses. Source files never modified.
Per-format transformer routing: MarkItDown 0.1.5 does not actually
convert .rtf (returns raw control codes unchanged, verified live), so
.rtf is routed to striprtf which does convert. Other formats stay on
MarkItDown:
.pdf .docx .pptx .xlsx .epub -> MarkItDown
.rtf -> striprtf
13 fringe cases handled with dedicated ExtractionStatus codes:
SKIP_NO_MARKITDOWN, SKIP_NO_STRIPRTF, SKIP_TOO_LARGE, SKIP_CLOUD_ONLY,
SKIP_ENCRYPTED, SKIP_EMPTY, SKIP_PERMISSION, SKIP_BROKEN_SYMLINK,
SKIP_UNRECOGNIZED, SKIP_EXTRACTION_ERROR, SKIP_NETWORK_TIMEOUT,
SKIP_UNREADABLE, plus the encoding fallback handled internally.
Drawers carry ingest_mode=extract + extract_mode=format so they are
distinguishable from project / convo drawers in the palace.
Both transformers are optional extras: pip install mempalace[extract].
MarkItDown requires Python >= 3.10 (env marker ensures it only installs
where supported; Python 3.9 users still get RTF coverage via striprtf).
Tests: 94 new (21 line-numbering + 73 format-miner, including 9
mine_formats orchestrator tests). Full mempalace suite still green
(1992 passed locally). Coverage 86% on mempalace/format_miner.py.
ruff check + ruff format clean on the pinned 0.15.9.
Live verification: mempalace mine --mode extract on a directory with 2
RTFs + 1 PDF produced 90 drawers correctly; mempalace search found
content from both file types in the resulting wing.
Documented limits (out of scope for 3.3.6): custom PDF parsers, OCR on
scanned PDFs, DRM-locked files, pathological corrupt files. These get
reported via skip codes and skipped.
Refs: docs/format-coverage.md, docs/virtual-line-numbering.md
Documents the recovery procedure for the chromadb index-metadata corruption
shape filed at chroma-core/chroma#6949 and reproduced by mempalace's
rebuild_index code path (#1492).
Symptom: mempalace integrity gate quarantines a segment dir with
"labels present but dimensionality is missing or invalid (None)" at
startup, vector search drops to BM25-only fallback, recall gap appears.
Recovery: patch the dimensionality field back into the index metadata
file (the rest of the segment state is intact). ~90 seconds end-to-end
on a 183k-drawer palace; restored 99.97% of recall.
The "delete the metadata file entirely" workaround from chroma-core/chroma#6949
loses the id_to_label and label_to_id mappings; the patch approach
documented here preserves them.
Companion content:
- docs/recovery/index-metadata-recovery.md (this file)
- Related issues #1492 (producer-side fix) and #1493 (auto-recover
proposal for the integrity gate)
- External: jphein/palace-daemon docs/recovery/chromadb-metadata-dict-patch.md
has the same procedure from a palace-daemon HTTP operator's
perspective, plus tests/test_chromadb_metadata_recovery.py with a
regression test that builds a real palace + corrupts + recovers.
Two fixes from Copilot's 2026-04-23 inline review:
1. Drop `-n` from the grep command. Hard-coded line numbers in the
"Expected" block would drift as files evolve, making the
checklist misleading. The check is about presence, not location —
line numbers add noise without helping pass/fail.
2. Reword "`console_script` entry point declared in pyproject.toml"
→ "console script declared under `[project.scripts]` in
pyproject.toml". PEP 621's `[project.scripts]` is the canonical
name for this repo's config form; the old wording conflated it
with setuptools' `console_scripts` entry-point group name.
Expected output block updated to match new grep (no colons before
line numbers).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fulfills the "Optional: release-checklist addition" proposal at the
bottom of #1093 (the v3.3.2 release defect where plugin.json referenced
a mempalace-mcp binary that pyproject.toml never declared, so fresh
`pip install` was broken for everyone until messelink's #340 was
re-cut as v3.3.3).
New file at docs/RELEASING.md (no existing doc at that path) with a
single pre-release grep:
grep -rn mempalace-mcp pyproject.toml .claude-plugin .codex-plugin
The original #1093 proposal specified `pyproject.toml
.claude-plugin/plugin.json` (2 files). This expands via -rn directory
recursion to also cover `.claude-plugin/.mcp.json` and
`.codex-plugin/plugin.json`, which reference `mempalace-mcp` by name
too — same class of regression through a different surface. Happy to
trim to the narrower 2-file form if preferred; one-line edit.
Shows the concrete expected output so a maintainer running this under
release pressure can eyeball "pass" without mental translation, and
points at #340 as the historical fix anchor so "investigate why the
entry is missing" has a diagnostic starting point rather than a dead
end.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Draft plugin specification for source adapters, mirroring RFC 001's
role for storage backends. Formalizes the contract six community
ingester PRs (#274, #23, #169, #232, #567, #98, #702) plus #981's
metadata-only mode have been reinventing ad-hoc, so adapter authors
can build to a stable surface.
Key decisions:
- Single ingest() method; lazy adapters yield SourceItemMetadata
ahead of drawers, eager adapters interleave
- Declared-transformation model (§1.4) replaces informal verbatim
promise with a verifiable one; byte_preserving adapters declare
the empty set, declared_lossy adapters enumerate. Existing
miner.py and the convo_miner+normalize pipeline map cleanly
- Palace is the incremental cursor via is_current(item, metadata);
no sidecar persistence
- Routing is adapter-owned; detect_room/detect_hall move into the
filesystem adapter
- Flat metadata per ChromaDB (RFC 001 §1.4) — entity hints as
json_string field, KG triples route to SQLite knowledge graph
- Closets stay core-built as a post-step; adapters may emit flat
closet_hints. Closes existing gap where convo drawers get no
closets
- No per-drawer field renames: source_file, filed_at, source_mtime,
added_by, normalize_version, entities, ingest_mode all preserved.
Spec adds adapter_name, adapter_version, privacy_class
§9 enumerates the cleanup PR prerequisites (mempalace/sources/
module, PalaceContext facade, KnowledgeGraph.add_triple gaining
backwards-compatible source_drawer_id + adapter_name params).
Tracking issue: #989
Addresses #875. The previous README was 755 lines mixing six purposes
(scam alert, hero, two mea-culpa notes, install guide, architecture
explainer, API reference, file map). Rework it as a pure entry point:
what MemPalace is, how to install, honest benchmark numbers, links to
the website for concept/architecture documentation.
Key content changes:
- Drop the "highest-scoring AI memory system ever benchmarked" framing.
- New tagline: "Local-first AI memory. Verbatim storage, pluggable
backend, 96.6% R@5 raw on LongMemEval — zero API calls." Avoids
naming a specific vector-store implementation since the backend is
pluggable (see mempalace/backends/base.py).
- Remove the cross-system comparison table. Retrieval recall (R@5)
and end-to-end QA accuracy are different metrics and are not
comparable; placing MemPalace's R@5 next to competitor QA accuracy
under a single column header was a category error.
- The "100%" LongMemEval headline is no longer the lead. The honest
held-out figure is 98.4% R@5 on 450 unseen questions. The rerank
pipeline reaches >=99% with any capable LLM (reproduced with
Claude Haiku, Sonnet, and minimax-m2.7 via Ollama) — pipeline-level,
not model-specific.
- Benchmark reproduction commands now reference the correct repo
(MemPalace/mempalace, not the defunct aya-thekeeper/mempal branch).
New file: docs/HISTORY.md as the canonical home for post-launch
corrections, public notices, and retractions. Contains verbatim:
- 2026-04-14 note on this rewrite (links to #875)
- 2026-04-11 impostor-domain notice (moved from README header)
- 2026-04-07 "A Note from Milla & Ben" (moved from README body)
README keeps a one-line scam-alert callout that links to
docs/HISTORY.md for the full timeline.
Merges develop (#820 version sync, #785 strip_noise + NORMALIZE_VERSION,
#784 file locking) and addresses six concerns surfaced during PR review
of the closet feature:
1. Closet append-on-rebuild bug — upsert_closet_lines used to APPEND to
existing closets (mismatched the doc's "fully replaced" promise). With
NORMALIZE_VERSION rebuilds on develop, this would have stacked stale
v1 topics on top of fresh v2 content forever. Fix:
- Drop the read-and-append branch from upsert_closet_lines (now a pure
numbered-id overwrite).
- Add purge_file_closets(closets_col, source_file) helper that wipes
every closet for a source file by where-filter.
- process_file calls purge_file_closets before upsert on every mine,
mirroring the existing drawer purge.
2. Searcher returned whole-file blobs from the closet path while the
direct path returned chunk-level drawers. Refactored:
- _extract_drawer_ids_from_closet parses the `→drawer_a,drawer_b`
pointers out of closet documents.
- _closet_first_hits hydrates exactly those drawer IDs (chunk-level),
not collection.get(where=source_file) (which returned everything).
- Same hit shape as direct-search path; both now carry matched_via.
3. max_distance was bypassed on the closet path. Now applied per-hit;
when every closet candidate gets filtered, _closet_first_hits returns
None and the caller falls through to direct drawer search.
4. Entity extraction caught sentence-starters like "When", "The",
"After" as proper nouns. Added _ENTITY_STOPLIST (~40 common false
positives + day/month names + role words). Real names like Igor /
Milla still survive — covered by tests.
5. CLOSETS.md drifted from the code (claimed "replaced via upsert" but
code appended; claimed BM25 hybrid that doesn't exist; claimed a
10K char hydration cap that wasn't enforced). Rewritten to describe
what actually ships, with explicit notes on the BM25 / convo-closet
follow-ups.
6. Zero tests for ~250 lines. Added tests/test_closets.py with 17 cases:
- build_closet_lines: pointer shape, header extraction, stoplist
filtering (with regression case for "When/After/The"), real-name
survival, fallback-line guarantee, drawer-ref slicing.
- upsert_closet_lines: pure overwrite semantics (regression for the
append bug), char-limit packing without splitting lines.
- purge_file_closets: scoped to source_file, doesn't touch others.
- End-to-end miner rebuild: re-mining a file with fewer topics fully
purges leftover numbered closets from the larger first run.
- _extract_drawer_ids_from_closet: parsing + dedup edge cases.
- search_memories closet-first: fallback when empty, chunk-level
hits with matched_via, no whole-file glue, max_distance enforced.
Merge resolutions: miner.py imports combined NORMALIZE_VERSION/mine_lock
from develop with the closet helpers from this branch. process_file
auto-merged cleanly (closet block sits inside develop's lock body).
724/724 tests pass. ruff + format clean under CI-pinned 0.4.x.
Cherry-picked the docs portion of 67e4ac6 to accompany the closet
feature. Test coverage for closets is omnibus with tests for entity
metadata and BM25 (see PR targeting those features) and will land
together in a follow-up.
Co-Authored-By: MSL <232237854+milla-jovovich@users.noreply.github.com>
Igor Lins e Silva
Sandro da Silva
Josh
Milla J
jp
bensig
bensig