206cfbb6ca
Drawer/closet/triple IDs built by concatenating strings without a delimiter before hashing can collide: hash((s1 + str(i1))) == hash((s2 + str(i2))) whenever s1+str(i1) == s2+str(i2). Concretely, source_file="foo" + chunk_index=12 produces the same hash input as source_file="foo1" + chunk_index=2 — both yield "foo12". Under ChromaDB's primary-key constraint the second write silently overwrites the first; one chunk's content is lost without surfacing an error (the surrounding broad `except Exception:` blocks swallow any backend complaint). The defect class is "string + string concat fed to a hash without a delimiter." Per the styleguide's partial-scope-key-migration rule, every site sharing the pattern is a candidate and must be triaged — not just the ones the original issue named. FIX — 6 sites - mempalace/miner.py:1253 drawer_id, add_drawer() back-compat - mempalace/miner.py:1386 drawer_id, batched mine loop - mempalace/miner.py:1416 drawer_ids, closet-build re-derivation - mempalace/format_miner.py:643 drawer_id, format_miner batched loop - mempalace/mcp_server.py:1136 drawer_id, tool_add_drawer - mempalace/knowledge_graph.py:305 triple_id, KG triple insertion MIGRATED FOR CONSISTENCY — 2 sites - mempalace/convo_miner.py:87 sentinel_key — was `:`, now `|` - mempalace/convo_miner.py:422 drawer_key — was `:`, now `|` Pre-existing delimiter was `:`. Migrated because (a) source_file can contain literal `:` on Windows paths (`C:\Users\…`) and URL-like sources (`https://host:8080`), weakening `:` as a separator; (b) `|` is reserved in Windows filenames so source_file cannot contain it, making it strictly safer. DELIMITER CHOICE - `|` chosen over `:` (Windows / URL source_file edge cases) - `|` chosen over `\x00` (NUL breaks log greppability + print() debug + downstream string handling for marginal extra safety) - Matches the established precedent in mempalace/diary_ingest.py (lines 52, 76, 91, 98 — all already on `|`) EXEMPT — audited and correct as-is Single-input hashes (nothing to delimit): - mempalace/miner.py:1432 closet_id (source_file only) - mempalace/format_miner.py:559 sentinel_id (source_file only) - mempalace/palace.py:433 lock filename (source_file only) - mempalace/palace.py:629 palace_key (lock_key_source only) - mempalace/diary_ingest.py:158 content_hash (text only) - mempalace/hooks_cli.py:329 pidfile digest (joined cmd only) - mempalace/sources/context.py:141 record digest (source_file only) Already correctly delimited: - mempalace/hallways.py:157 `f"{wing}::{a}::{b}"` (`::`) - mempalace/palace_graph.py:454 `f"{a}↔{b}"` (`↔`) - mempalace/diary_ingest.py:52,76,91,98 (`|` precedent) Protected by composition (uniqueness guaranteed by the ID prefix, not by the hash slice): - mempalace/mcp_server.py:1635 entry_id is `diary_{wing}_{strftime('%Y%m%d_%H%M%S%f')}_{sha[:12]}`. Microsecond-resolution timestamp prefix supplies uniqueness; the trailing hash is a content-discriminator, not the write-time uniqueness guarantor. NEW MODULES - mempalace/ids.py — single source of truth for ID construction. Five helpers (make_drawer_id_from_chunk, make_drawer_id_from_content, make_convo_drawer_id, make_convo_sentinel_id, make_triple_id) plus an ID_RECIPE = "v2" constant. - mempalace/collision_scan.py — pre-mining defense. Runs immediately before each batched ChromaDB upsert; raises CollisionError naming the colliding (source_file, chunk_index) pairs if any proposed drawer_id appears more than once with conflicting metadata across the union of incoming and existing rows. DETECTION ChromaDB's primary-key constraint means past collisions were silently resolved at upsert time (last-write-wins), erasing evidence. A post-hoc audit cannot reconstruct what was lost from palace state alone. Therefore: - Pre-mining risk scan. Before each batched upsert, compute the proposed drawer_ids for the incoming chunk set AND query existing drawer_ids from the collection. If any proposed id appears more than once in the union (incoming-vs-incoming or incoming-vs- existing) with conflicting (source_file, chunk_index), abort the mine with an actionable error naming the colliding pairs. Collision is caught BEFORE it destroys data, which is the only point at which palace state still carries the evidence. - New metadata key: `"id_recipe": "v2"` on every drawer written under the delimited recipe. Audits compare like-for-like; drawers without `id_recipe` are treated as v1 legacy (undelimited or `:`-delimited), not as collisions. - Honest disclosure: palaces mined under any pre-v2 mempalace may carry silent past collisions whose original content is unrecoverable from palace state. Future library tier work will give users a per-drawer audit + opt-in archival path. TESTS - tests/test_ids.py: 17 unit tests covering all 5 helpers, the ID_RECIPE constant, the private `_delimited_sha256` helper, and the four defect-class collision shapes (chunk_index boundary, content boundary, extract_mode boundary, ISO datetime boundary). RED collision tests confirm the v2 recipe breaks the defect class. - tests/test_collision_scan.py: 10 tests covering clean batches, idempotent re-mines, incoming-vs-incoming collisions, incoming-vs- existing collisions, error-message quality, empty batches, metadata without chunk_index, and ChromaDB backend errors propagating cleanly. - tests/test_convo_miner_unit.py: FakeCol gains a stub `get()` so the pre-mining scan can probe an empty in-test collection. BACKWARDS COMPATIBILITY - Existing v1 drawers remain queryable; no rewrite of stored IDs. - New mining passes write v2 drawers alongside v1 via PR #1628's additive-mining model. - No user action required; opt-in cleanup ships separately. VERIFICATION - macOS Python 3.12 (local): 2304 passed, 3 skipped, 0 failed, coverage 85.44% (above 80% threshold). - Linux Python 3.9/3.11/3.13 (OrbStack with full `pip install -e '.[dev]'` flow): see PR body for per-version pass counts. - `ruff check .` + `ruff format --check .`: clean. - pylint score: ids.py 10.00/10, collision_scan.py 10.00/10; all modified files >= 8.92 (no regression). - bandit: clean on all new files; the one MEDIUM finding in knowledge_graph.py is on lines 385/407 (pre-existing SQL string construction in timeline queries), not in code touched by this PR. - pre_push_check.sh: ALL CHECKS PASSED. Refs: deferred from PR #1572 (Copilot finding #13). Styleguide audit documented above: 8 fix/migration + 11 enumerated exempt sites. |
||
|---|---|---|
| .agents/plugins | ||
| .claude-plugin | ||
| .codex-plugin | ||
| .devcontainer | ||
| .github | ||
| assets | ||
| benchmarks | ||
| docs | ||
| examples | ||
| hooks | ||
| integrations/openclaw | ||
| landing | ||
| mempalace | ||
| tests | ||
| tools | ||
| website | ||
| .gitignore | ||
| .pre-commit-config.yaml | ||
| .python-version | ||
| AGENTS.md | ||
| CHANGELOG.md | ||
| CLAUDE.md | ||
| CONTRIBUTING.md | ||
| LICENSE | ||
| MISSION.md | ||
| README.md | ||
| ROADMAP.md | ||
| SECURITY.md | ||
| openarena-claim.txt | ||
| pyproject.toml | ||
| uv.lock | ||
README.md
MemPalace
Local-first AI memory. Verbatim storage, pluggable backend, 96.6% R@5 raw on LongMemEval — zero API calls.
[!CAUTION] Beware of impostor sites. MemPalace has no other official websites. The only official sources are this GitHub repository, the PyPI package, and the docs at mempalaceofficial.com. Any other domain (including
.tech,.net, or other.comvariants) is an impostor and may distribute malware. Details and timeline: docs/HISTORY.md.
[!IMPORTANT] Claude Code sessions expire in 30 days without auto-save hooks wired. Read this →
Need the shortest recovery/setup path? Use the Claude Code retention setup checklist.
What it is
MemPalace stores your conversation history as verbatim text and retrieves it with semantic search. It does not summarize, extract, or paraphrase. The index is structured — people and projects become wings, topics become rooms, and original content lives in drawers — so searches can be scoped rather than run against a flat corpus.
The retrieval layer is pluggable. The current default is ChromaDB; the
interface is defined in mempalace/backends/base.py
and alternative backends can be dropped in without touching the rest of
the system.
Nothing leaves your machine unless you opt in.
Architecture, concepts, and mining flows: mempalaceofficial.com/concepts/the-palace.
Install
MemPalace ships a CLI, so install it in an isolated environment to avoid
PEP 668 errors on Debian/Ubuntu/Homebrew Pythons and to keep mempalace's
deps (chromadb, numpy, grpcio, …) from conflicting with anything
else in your global site-packages.
We recommend uv — uv tool install puts
the mempalace CLI in an isolated environment on your PATH:
uv tool install mempalace
mempalace init ~/projects/myapp
pipx works the same way if you prefer it:
pipx install mempalace.
Prefer plain pip only inside an activated virtualenv where you
explicitly want import mempalace available:
python -m venv .venv && source .venv/bin/activate
pip install mempalace
Quickstart
# Mine content into the palace
mempalace mine ~/projects/myapp # project files
mempalace mine ~/.claude/projects/ --mode convos # Claude Code sessions (scope with --wing per project)
# Search
mempalace search "why did we switch to GraphQL"
# Load context for a new session
mempalace wake-up
For Claude Code, Gemini CLI, MCP-compatible tools, and local models, see mempalaceofficial.com/guide/getting-started.
Benchmarks
All numbers below are reproducible from this repository with the commands
in benchmarks/BENCHMARKS.md. Full
per-question result files are committed under benchmarks/results_*.
LongMemEval — retrieval recall (R@5, 500 questions):
| Mode | R@5 | LLM required |
|---|---|---|
| Raw (semantic search, no heuristics, no LLM) | 96.6% | None |
| Hybrid v4, held-out 450q (tuned on 50 dev, not seen during training) | 98.4% | None |
| Hybrid v4 + LLM rerank (full 500) | ≥99% | Any capable model |
The raw 96.6% requires no API key, no cloud, and no LLM at any stage. The hybrid pipeline adds keyword boosting, temporal-proximity boosting, and preference-pattern extraction; the held-out 98.4% is the honest generalisable figure.
The rerank pipeline promotes the best candidate out of the top-20
retrieved sessions using an LLM reader. It works with any reasonably
capable model — we have reproduced it with Claude Haiku, Claude Sonnet,
and minimax-m2.7 via Ollama Cloud (no Anthropic dependency). The gap
between raw and reranked is model-agnostic; we do not headline a "100%"
number because the last 0.6% was reached by inspecting specific wrong
answers, which benchmarks/BENCHMARKS.md flags as teaching to the test.
Other benchmarks (full results in benchmarks/BENCHMARKS.md):
| Benchmark | Metric | Score | Notes |
|---|---|---|---|
| LoCoMo (session, top-10, no rerank) | R@10 | 60.3% | 1,986 questions |
| LoCoMo (hybrid v5, top-10, no rerank) | R@10 | 88.9% | Same set |
| ConvoMem (all categories, 250 items) | Avg recall | 92.9% | 50 per category |
| MemBench (ACL 2025, 8,500 items) | R@5 | 80.3% | All categories |
We deliberately do not include a side-by-side comparison against Mem0, Mastra, Hindsight, Supermemory, or Zep. Those projects publish different metrics on different splits, and placing retrieval recall next to end-to-end QA accuracy is not an honest comparison. See each project's own research page for their published numbers.
Reproducing every result:
git clone https://github.com/MemPalace/mempalace.git
cd mempalace
uv sync --extra dev # or: pip install -e ".[dev]"
# see benchmarks/README.md for dataset download commands
uv run python benchmarks/longmemeval_bench.py /path/to/longmemeval_s_cleaned.json
Knowledge graph
MemPalace includes a temporal entity-relationship graph with validity windows — add, query, invalidate, timeline — backed by local SQLite. Usage and tool reference: mempalaceofficial.com/concepts/knowledge-graph.
MCP server
29 MCP tools cover palace reads/writes, knowledge-graph operations, cross-wing navigation, drawer management, and agent diaries. Installation and the full tool list: mempalaceofficial.com/reference/mcp-tools.
Agents
Each specialist agent gets its own wing and diary in the palace.
Discoverable at runtime via mempalace_list_agents — no bloat in your
system prompt:
mempalaceofficial.com/concepts/agents.
Auto-save hooks
Two Claude Code hooks save periodically and before context compression: mempalaceofficial.com/guide/hooks.
If you are installing under time pressure, start with the
Claude Code retention setup checklist:
wire the hooks, back up existing JSONL transcripts, and backfill them with
mempalace mine ~/.claude/projects/ --mode convos.
For per-message recall on top of the file-level chunks the hooks produce,
run mempalace sweep <transcript-dir> periodically — it stores one
verbatim drawer per user/assistant message, idempotent and resume-safe.
Requirements
- Python 3.9+
- A vector-store backend (ChromaDB by default)
- ~300 MB disk for the embedding model. Onboarding (
python -m mempalace.onboarding) offersembeddinggemma-300m(multilingual, 100+ languages, recommended) orall-MiniLM-L6-v2(English-only, ~30 MB). See the docstring atmempalace/embedding.pyfor details and migration notes.
No API key is required for the core benchmark path.
Docs
- Getting started → mempalaceofficial.com/guide/getting-started
- CLI reference → mempalaceofficial.com/reference/cli
- Python API → mempalaceofficial.com/reference/python-api
- Full benchmark methodology → benchmarks/BENCHMARKS.md
- Release notes → CHANGELOG.md
- Corrections and public notices → docs/HISTORY.md
Contributing
PRs welcome. See CONTRIBUTING.md.
License
MIT — see LICENSE.