Commit Graph
1588 Commits
Author SHA1 Message Date
Igor Lins e Silva 82af1fff27 Merge pull request #1678 from mvalentsev/fix/1676-mcp-wal-import-side-effect
fix(mcp): defer WAL setup so import no longer recreates ~/.mempalace (#1676)
2026-06-05 23:43:12 -03:00
Igor Lins e Silva 5289bdfedb Merge pull request #1631 from YuRiNjjot/fix-embeddinggemma-chromadb
fix(embedding): add embed_query/embed_documents to EmbeddinggemmaONNX for ChromaDB 1.5.x
2026-06-05 23:42:51 -03:00
Igor Lins e Silva 694289c423 Merge pull request #1698 from MemPalace/ci/pypi-trusted-publishing
ci: add PyPI trusted-publishing workflow
2026-06-05 23:33:37 -03:00
Igor Lins e Silva 723e45bf2b Merge pull request #1679 from MemPalace/codex/pluggable-vector-backends-qdrant
Add pluggable vector backends
2026-06-05 23:33:16 -03:00
Igor Lins e Silva 37e34f6cb5 docs: correct pgvector table_dimension comment after real-Postgres testing
Validated the backend against a real Postgres 18.4 + pgvector 0.8.2 instance
(full live roundtrip incl. close/reopen + same-dim write). The reviewer's
claim that a vector(n) column's atttypmod is dimension+4 does NOT reproduce:
raw atttypmod equals the bare dimension on 0.8.x, so the original direct read
was already correct. Keep format_type() anyway as the canonical, version-proof
way to read the typmod, and correct the comment to reflect reality instead of
asserting a bug that does not exist.
2026-06-05 23:21:20 -03:00
Igor Lins e Silva f88951e8d4 Merge remote-tracking branch 'origin/develop' into codex/pluggable-vector-backends-qdrant
# Conflicts:
#	README.md
2026-06-05 23:18:54 -03:00
Igor Lins e Silva 4bc1ea8078 fix: address Copilot second-pass review on the pluggable-backend PR
Three findings from the Copilot review on ec5d1eb:

- pgvector (real correctness bug): table_dimension() read the raw
  pg_attribute.atttypmod of the vector(n) column, which is not the bare
  dimension, so reopening a stored pgvector palace could raise a false
  DimensionMismatchError on the next same-dimension write. Now rounds through
  format_type(atttypid, atttypmod) (the type's own typmod_out), which yields
  the canonical vector(N) regardless of encoding or pgvector version. The live
  roundtrip test now closes + reopens and writes a same-dim vector to guard it.

- chroma (real correctness bug): _lexical_search_via_sqlite() returned
  LexicalHit.id as the internal embeddings.id rowid instead of the public
  embeddings.embedding_id, so lexical_search -> get(ids=...) did not round-trip
  (broke hybrid-search id lookups). Now selects e.embedding_id and maps rowid
  -> public id. Existing FTS test schema updated to include embedding_id (real
  Chroma schema) and assert the public id; added an end-to-end round-trip test
  through a real ChromaBackend collection.

- sqlite_exact (error-message quality): CollectionNotInitializedError was
  raised with palace_path instead of the collection name in get_collection and
  delete_collection, inconsistent with the other backends and line 287. Now
  names the collection; added a regression test.

Earlier first-pass findings (palace.py unknown-backend KeyError, dedup.py
docstring) were already fixed in ec5d1eb.
2026-06-05 23:14:35 -03:00
Igor Lins e Silva dc7d0be10c docs: clarify publish.yml trigger comment
Address review on #1698: a published GitHub Release may create the v* tag
or reuse an existing one. Reword the header so it no longer implies the tag
is always created at release time, and state the real guarantee — the
in-workflow checks make the pipeline self-contained (tag on main + matches
the version manifest), independent of version-guard.
2026-06-05 23:13:20 -03:00
Igor Lins e Silva 15852edbaf Merge pull request #1696 from sebaplaza/feat/docker-image
feat(docker): add Docker image for MCP server and CLI
2026-06-05 23:07:31 -03:00
Igor Lins e Silva eb3ff2e61a ci(docker): fix latest/main publishing, add arm64 + GPU build check
Review fixes for the Docker packaging PR:

- docker-publish: tie the `latest` tag to pushes on `main` (the release
  branch). Previously it was gated on `is_default_branch`, but the
  default branch is `develop` and the workflow never ran there, so
  `latest` was never produced. main + `v*` tags publish; develop is
  validated via the pull_request trigger but does not publish.
- docker-publish: publish multi-arch amd64+arm64 (Apple Silicon / ARM)
  on real pushes via setup-qemu-action; PRs stay amd64-only for speed.
- docker-publish: only export the GHA cache on in-repo events (fork PRs
  get a read-only cache, which just emits 403 noise).
- docker-publish: add a build-only job that validates Dockerfile.gpu
  compiles so the CUDA variant can't silently rot.
- Dockerfile: correct the persistence comment — the default `minilm`
  model caches under ~/.cache/chroma (ChromaDB S3), not
  ~/.cache/huggingface (that's the optional embeddinggemma model).
- docker-compose: drop the redundant MEMPALACE_PALACE_PATH override (it
  duplicated the HOME=/data default); document overrides as examples.
2026-06-05 22:39:15 -03:00
Igor Lins e Silva 0225eb4a08 docs: bump version on develop, not directly on main
Address review on #1698: committing the version bump straight to main
bypasses branch protection and drifts develop behind. Bump on develop
first; it reaches main via the develop -> main merge.
2026-06-05 22:30:05 -03:00
Igor Lins e Silva 1dadd33f35 ci: add PyPI trusted-publishing workflow
Publish to PyPI on a published GitHub Release via Trusted Publishing
(OIDC — no stored token), gated by the `pypi` environment's manual
approval. The build job verifies the release tag is reachable from main
and matches mempalace/version.py before building the sdist + wheel; a
separate publish job holds the id-token scope and does the upload.

Documents the one-time setup (PyPI trusted publisher + `pypi`
environment) and the per-release runbook in docs/RELEASING.md.
2026-06-05 22:26:41 -03:00
Igor Lins e Silva ec5d1eb1c7 feat: add pgvector backend + namespace-isolation conformance contract
Adds a second external storage backend (Postgres/pgvector) alongside Qdrant
to prove the BaseBackend/BaseCollection contract generalizes across substrates
(SQL + JSONB containment filters + pgvector `<=>` ranking vs Qdrant's REST/dict
model), and addresses the review feedback on PR #1679.

Backend (mempalace/backends/pgvector.py):
- table-per-(namespace, palace, collection) isolation; advertises
  supports_namespace_isolation
- JSONB filter pushdown for the containment subset, local-exact fallback for
  $or/$contains/comparisons/where_document
- BM25 lexical search; marker-based mismatch protection
- optional psycopg dependency (lazy import), in-memory fake for CI, live test
  gated on MEMPALACE_PGVECTOR_LIVE_URL
- registered in registry/__init__/pyproject entry point + [pgvector] extra;
  MEMPALACE_PGVECTOR_DSN / MEMPALACE_PGVECTOR_NAMESPACE config; README docs

Isolation contract (RFC 001):
- PalaceRef/BaseBackend document the per-id MUST and the cross-namespace MUST,
  gated on the new supports_namespace_isolation capability token
- runnable conformance suite (tests/_backend_conformance.py,
  tests/test_backend_conformance.py); qdrant + pgvector run it via their fakes

Marker fail-loud guard:
- qdrant and pgvector now refuse get_collection when local_path is None instead
  of silently opening a remote collection with no mismatch protection

Review fixes:
- palace._open_collection_or_explain handles unknown-backend KeyError as a CLI
  state message instead of an escaping stack trace
- dedup.py docstring no longer claims "No API calls" unconditionally (false for
  remote backends)
2026-06-05 21:41:21 -03:00
margaretjgu 9be2b97b6c fix(backups): add max_backups retention to bound backup disk usage
mempalace migrate (.pre-migrate.* full-palace copies) and mempalace repair
max-seq-id (chroma.sqlite3.max-seq-id-backup-* DB copies) each wrote a fresh,
full-size, timestamped backup every run and never deleted the old ones. On a
machine that mines or repairs on a schedule, those copies could silently
accumulate until they filled the disk.

Add a configurable max_backups setting (default 10; env MEMPALACE_MAX_BACKUPS
or config.json) and a shared prune_backups helper that trims the oldest copies
after each new backup is written. Pruning is keyed by filesystem mtime, scoped
strictly to each backup's own naming pattern so live data is never touched, and
best-effort so a deletion failure can never abort the migrate/repair that just
succeeded. Set max_backups to 0 to keep every backup.
2026-06-05 14:52:06 -04:00
plaza-s 8757ba2650 fix(docker): address gemini review on MR #1696 2026-06-05 13:35:36 +02:00
sebaplaza 717a57ff43 feat(docker): add container image for MCP server and CLI
Add a multi-stage, uv-based Dockerfile producing a CPU image (with the
extract + spellcheck extras), plus a CUDA variant (Dockerfile.gpu) for
onnxruntime-gpu accelerated embeddings.

A single flexible entrypoint dispatches to the MCP stdio server (default)
or the mempalace CLI. All state -- palace, config, and the lazily
downloaded embedding model -- persists under /data via HOME, runs as a
non-root user, and is exposed as a volume.

Also add a docker-compose.yml for convenience, a GHCR publish workflow,
and a Docker section in the README.
2026-06-05 12:32:33 +02:00
mvalentsevandYC-AIUSER 4e0fce010e fix(hooks): file stop-hook diary checkpoints under the harness agent identity (#1693)
Stop-hook checkpoints were saved via _save_diary_direct with a hardcoded
agent_name="session-hook". tool_diary_read filters Chroma metadata by
agent, so mempalace_diary_read(agent_name="claude") never surfaced any
hook-saved checkpoint. Derive the diary identity from the harness
(claude-code -> claude, codex -> codex; an unknown harness keeps its own
name) and thread it through to tool_diary_write. Make agent_name a
required keyword argument so the identity is always explicit.

Co-Authored-By: YC-AIUSER <273917354+YC-AIUSER@users.noreply.github.com>
2026-06-05 12:26:27 +05:00
robertcprice 5ba8399bb2 Fix crash when tool_use input is a list instead of dict
Some Claude Code JSONL transcripts have tool_use blocks where the
`input` field is a list rather than a dict (e.g. multi-content tool
calls). This causes an AttributeError on line 554 when code tries to
call `.get()` on the list.

Guard by normalizing list inputs to an empty dict, allowing mining
to proceed without losing other tool metadata.

Fixes: mining Claude Code conversations crashes with
  AttributeError: 'list' object has no attribute 'get'
  at normalize.py:554
2026-06-03 20:29:48 -04:00
sjhddh f5d2212577 style(tests): apply ruff 0.4.x format to test_normalize
Fixes lint CI: ruff format --check flagged blank-line and long-dict
wrapping in the Continue.dev parser tests.
2026-06-03 18:36:50 +02:00
Brian potterandClaude Opus 4.8 ce46747282 fix(status): count drawers from sqlite instead of cold-loading the HNSW index
`mempalace status` opened the ChromaDB collection purely to tally drawers by
wing/room — and opening it cold-loads the HNSW vector index. On a 398k-drawer
palace that load costs ~60s of CPU on every invocation, even though the counts
live in chroma.sqlite3's relational tables (`repair-status` already reads them
in <1s; `status` was the outlier).

Read the wing/room histogram directly from chroma.sqlite3 via a new
`_sqlite_wing_room_counts` helper, falling back to the existing ChromaDB-client
path when the sqlite read is unavailable (missing DB, un-bootstrapped
collection, sustained writer lock, or an unexpected schema) — preserving the
state-specific guidance for absent/empty palaces.

Measured on a 398,315-drawer / 3.5GB palace: status CPU ~60s -> ~1s.

Review hardening:
- PRAGMA busy_timeout so a transient checkpoint lock is waited out rather than
  instantly demoted to the slow path; a sustained lock still falls back.
- COALESCE over string/int/float so a numeric wing/room matches the ChromaDB
  path instead of dropping to "?".
- Explicit `s.scope = 'METADATA'` so the segment join can't silently
  double-count on a future ChromaDB layout.

Tests: exact-tally (anti fan-out), no-cold-load regression (proven failable by
reverting the fix), numeric-metadata, partial-metadata "?" bucketing,
locked-DB fallback, and collection-absent None routing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-03 02:08:19 -05:00
Milla J 02b8753d97 Merge pull request #1680 from MemPalace/fix/1619-hallway-pagination
fix(hallways): paginate drawer fetch to avoid SQLite variable overflow on large wings (#1619)
2026-06-02 22:18:15 -07:00
Milla J 300e824e94 docs(hallways): correct col contract in compute_hallways_for_wing docstring (#1680)
The docstring still said col "must support .get(where=..., include=...)",
but this PR changed the fetch to count() + paginated
get(limit=, offset=, include=) filtered client-side, precisely to avoid the
get(where=...) path that overflows SQLite's variable limit on large wings.

Update the Args entry to describe the real contract so fake collections and
alternate backends implement the right shape.

Docstring only — no behavior change.
2026-06-02 21:56:29 -07:00
Milla J af2bd0e10a style(tests): format test_hallways_pagination.py with ruff (#1680)
The where-filter update to the pagination regression test left one list
comprehension past the line-length limit, so `ruff format --check .` failed
in CI while every test platform passed. Wrap the comprehension as ruff
format produces it — no logic change.

Restores the lint job to green.
2026-06-02 21:45:14 -07:00
Milla Jandgemini-code-assist[bot] 526d58ae93 Update tests/test_hallways_pagination.py
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-06-02 19:56:16 -07:00
Milla J 4d3daf96d3 fix(hallways): paginate drawer fetch to avoid SQLite variable overflow on large wings (#1619)
compute_hallways_for_wing fetched the whole wing in a single
col.get(where={"wing": wing}). ChromaDB binds one SQL variable per matched id,
so on a wing larger than SQLITE_MAX_VARIABLE_NUMBER (32766) the call raised
"too many SQL variables" inside chromadb. The exception was caught, so the mine
completed — but the wing's hallway graph silently never built, and the
cross-wing tunnels promoted from it were starved, on exactly the large wings
that benefit most from navigation. Confirmed threshold: a 42,062-drawer wing
crashed; a 29,629-drawer wing succeeded.

Replace the single where-get with the established pagination pattern: count()
+ get(limit=5000, offset=...) filtered to the wing client-side — matching
miner.status, palace.regenerate_closets, and palace_graph.build_graph, which
already paginate to dodge the same 32766 limit.

Tests:
- test_hallways_pagination: a collection whose where-get raises (simulating the
  overflow) while count() + paginated get works — RED before, GREEN after.
- test_hallways: _fake_collection updated to the paginated API; existing
  hallway tests are unchanged in behavior.

Closes #1619.
2026-06-02 19:15:10 -07:00
Igor Lins e Silva 30b82ba95f fix: avoid qdrant lexical full scan on empty text hits 2026-06-02 21:50:08 -03:00
Igor Lins e Silva 6aa8e93bc9 feat: add pluggable vector backends 2026-06-02 21:38:53 -03:00
mvalentsevandGrace Gettert 0dd0ade3b9 fix(mcp): defer WAL setup so import no longer recreates ~/.mempalace (#1676)
The write-ahead-log directory was created at module scope in
mempalace/mcp_server.py, so importing the MCP server ran
`_WAL_DIR.mkdir(parents=True, exist_ok=True)` and recreated `~/.mempalace`
even after a user removed it to engage the documented kill-switch
(`hooks_cli._palace_root_exists()`, #1305). On every session start this
re-armed the autosave/mining hooks the user had disabled.

Move the WAL directory setup into a lazy `_ensure_wal()` called from the
write path (`_wal_log`). Importing the module no longer touches disk; the
directory is created on the first real write, when the palace is being
written to anyway. The WAL is intentionally not gated on
`_palace_root_exists()` (the ChromaDB/KG layer recreates the palace
regardless, so gating would only drop audit records); runtime kill-switch
enforcement for MCP writes is tracked in #504.

Add regression tests: a subprocess import asserts `~/.mempalace` is not
created, and a write test asserts the directory is created lazily with the
expected permissions.

Co-Authored-By: Grace Gettert <9805362+ggettert@users.noreply.github.com>
2026-06-03 02:55:19 +05:00
M8SONandClaude Opus 4.8 b5f3915d5b fix(config): strip leading/trailing separators in normalize_wing_name
A path-encoded dirname like `-home-user-proj` produced a leading-underscore
slug (`_home_user_proj`) that sanitize_name — and therefore the MCP write
tools — reject, so the conversation miner filed transcripts into wings the
MCP could never write to. Strip leading/trailing `_` after collapsing
separators so the slug is valid. Adds tests.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-01 21:21:56 -04:00
Mario Gutierrez e54896699b fix(backends): keep single-string documents whole in surrogate sanitiser
Per Gemini review on #1673: chromadb accepts OneOrMany[Document], so a bare
str document was iterated character-by-character by the list comprehension,
splitting it into per-character documents (the silent corruption this method
exists to prevent). Handle isinstance(str) explicitly; add a regression test.
2026-06-01 17:19:10 -05:00
Mario Gutierrez 6dd99ec4b0 fix(backends): strip lone surrogates from documents at the ChromaDB chokepoint
#1235 sanitised lone UTF-16 surrogates for the MCP write tools, but the bulk
ingest paths (miner, convo_miner, sweeper, diary_ingest) build documents
without routing through sanitize_content() and reach ChromaCollection directly.
A single lone surrogate in document text raises UnicodeEncodeError inside
chromadb and aborts the whole add/upsert batch with a -32000 Internal Error,
silently dropping every other row in the same batch.

Complete the chokepoint: add _sanitize_documents_for_chromadb (mirror of
_sanitize_metadatas_for_chromadb) and apply it in add/upsert/update so the
backend guarantees UTF-8-safe documents regardless of caller. IDs and dedup are
unaffected (IDs are computed upstream); only illegal lone surrogates become
U+FFFD, matching the errors="replace" behaviour used elsewhere.
2026-06-01 16:38:40 -05:00
undeadindustriesandCursor a329acf23c fix(antigravity): resolve interpreter from console-script shebang
`uv tool install mempalace` / `pipx install` place the mempalace
console scripts in an isolated environment whose interpreter is not
the system python3. mempal_resolve_python previously resolved
`command -v python3`, landing on a Python that cannot import
mempalace: the `-m mempalace --version` probe failed and mining
silently never fired (hit by a real user on PR #1633).

Resolution now derives the interpreter from the mempalace-mcp /
mempalace console-script shebang on PATH (the same script the MCP
server launches) before falling back to python3. It is pure shebang
parsing + stat — no Python subprocess at source time — so the hook
performance budget is preserved. An env-style `#!/usr/bin/env python`
shebang and a non-executable interpreter are both rejected and fall
through. MEMPAL_PYTHON remains the explicit override.

Adds 6 resolver regression tests, documents resolution + MEMPAL_PYTHON
in the guide and hooks README (fixing the stale `command -v mempalace`
note), and a CHANGELOG entry.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-31 10:41:09 +10:00
Milla J 206cfbb6ca fix(ids): delimit hash inputs to prevent drawer_id collisions (#80)
Drawer/closet/triple IDs built by concatenating strings without a
delimiter before hashing can collide: hash((s1 + str(i1))) ==
hash((s2 + str(i2))) whenever s1+str(i1) == s2+str(i2). Concretely,
source_file="foo" + chunk_index=12 produces the same hash input as
source_file="foo1" + chunk_index=2 — both yield "foo12". Under
ChromaDB's primary-key constraint the second write silently
overwrites the first; one chunk's content is lost without surfacing
an error (the surrounding broad `except Exception:` blocks swallow
any backend complaint).

The defect class is "string + string concat fed to a hash without
a delimiter." Per the styleguide's partial-scope-key-migration rule,
every site sharing the pattern is a candidate and must be triaged —
not just the ones the original issue named.

FIX — 6 sites
- mempalace/miner.py:1253        drawer_id, add_drawer() back-compat
- mempalace/miner.py:1386        drawer_id, batched mine loop
- mempalace/miner.py:1416        drawer_ids, closet-build re-derivation
- mempalace/format_miner.py:643  drawer_id, format_miner batched loop
- mempalace/mcp_server.py:1136   drawer_id, tool_add_drawer
- mempalace/knowledge_graph.py:305  triple_id, KG triple insertion

MIGRATED FOR CONSISTENCY — 2 sites
- mempalace/convo_miner.py:87    sentinel_key — was `:`, now `|`
- mempalace/convo_miner.py:422   drawer_key   — was `:`, now `|`

Pre-existing delimiter was `:`. Migrated because (a) source_file can
contain literal `:` on Windows paths (`C:\Users\…`) and URL-like
sources (`https://host:8080`), weakening `:` as a separator; (b) `|`
is reserved in Windows filenames so source_file cannot contain it,
making it strictly safer.

DELIMITER CHOICE
- `|` chosen over `:` (Windows / URL source_file edge cases)
- `|` chosen over `\x00` (NUL breaks log greppability + print()
  debug + downstream string handling for marginal extra safety)
- Matches the established precedent in mempalace/diary_ingest.py
  (lines 52, 76, 91, 98 — all already on `|`)

EXEMPT — audited and correct as-is
  Single-input hashes (nothing to delimit):
    - mempalace/miner.py:1432         closet_id (source_file only)
    - mempalace/format_miner.py:559   sentinel_id (source_file only)
    - mempalace/palace.py:433         lock filename (source_file only)
    - mempalace/palace.py:629         palace_key (lock_key_source only)
    - mempalace/diary_ingest.py:158   content_hash (text only)
    - mempalace/hooks_cli.py:329      pidfile digest (joined cmd only)
    - mempalace/sources/context.py:141  record digest (source_file only)

  Already correctly delimited:
    - mempalace/hallways.py:157     `f"{wing}::{a}::{b}"`  (`::`)
    - mempalace/palace_graph.py:454 `f"{a}↔{b}"`           (`↔`)
    - mempalace/diary_ingest.py:52,76,91,98                (`|` precedent)

  Protected by composition (uniqueness guaranteed by the ID prefix,
  not by the hash slice):
    - mempalace/mcp_server.py:1635  entry_id is
      `diary_{wing}_{strftime('%Y%m%d_%H%M%S%f')}_{sha[:12]}`.
      Microsecond-resolution timestamp prefix supplies uniqueness;
      the trailing hash is a content-discriminator, not the
      write-time uniqueness guarantor.

NEW MODULES
- mempalace/ids.py — single source of truth for ID construction.
  Five helpers (make_drawer_id_from_chunk, make_drawer_id_from_content,
  make_convo_drawer_id, make_convo_sentinel_id, make_triple_id) plus
  an ID_RECIPE = "v2" constant.
- mempalace/collision_scan.py — pre-mining defense. Runs immediately
  before each batched ChromaDB upsert; raises CollisionError naming
  the colliding (source_file, chunk_index) pairs if any proposed
  drawer_id appears more than once with conflicting metadata across
  the union of incoming and existing rows.

DETECTION
ChromaDB's primary-key constraint means past collisions were
silently resolved at upsert time (last-write-wins), erasing
evidence. A post-hoc audit cannot reconstruct what was lost from
palace state alone. Therefore:

- Pre-mining risk scan. Before each batched upsert, compute the
  proposed drawer_ids for the incoming chunk set AND query existing
  drawer_ids from the collection. If any proposed id appears more
  than once in the union (incoming-vs-incoming or incoming-vs-
  existing) with conflicting (source_file, chunk_index), abort the
  mine with an actionable error naming the colliding pairs.
  Collision is caught BEFORE it destroys data, which is the only
  point at which palace state still carries the evidence.

- New metadata key: `"id_recipe": "v2"` on every drawer written
  under the delimited recipe. Audits compare like-for-like;
  drawers without `id_recipe` are treated as v1 legacy (undelimited
  or `:`-delimited), not as collisions.

- Honest disclosure: palaces mined under any pre-v2 mempalace may
  carry silent past collisions whose original content is
  unrecoverable from palace state. Future library tier work will
  give users a per-drawer audit + opt-in archival path.

TESTS
- tests/test_ids.py: 17 unit tests covering all 5 helpers, the
  ID_RECIPE constant, the private `_delimited_sha256` helper, and
  the four defect-class collision shapes (chunk_index boundary,
  content boundary, extract_mode boundary, ISO datetime boundary).
  RED collision tests confirm the v2 recipe breaks the defect class.
- tests/test_collision_scan.py: 10 tests covering clean batches,
  idempotent re-mines, incoming-vs-incoming collisions, incoming-vs-
  existing collisions, error-message quality, empty batches,
  metadata without chunk_index, and ChromaDB backend errors
  propagating cleanly.
- tests/test_convo_miner_unit.py: FakeCol gains a stub `get()` so
  the pre-mining scan can probe an empty in-test collection.

BACKWARDS COMPATIBILITY
- Existing v1 drawers remain queryable; no rewrite of stored IDs.
- New mining passes write v2 drawers alongside v1 via PR #1628's
  additive-mining model.
- No user action required; opt-in cleanup ships separately.

VERIFICATION
- macOS Python 3.12 (local): 2304 passed, 3 skipped, 0 failed,
  coverage 85.44% (above 80% threshold).
- Linux Python 3.9/3.11/3.13 (OrbStack with full `pip install -e
  '.[dev]'` flow): see PR body for per-version pass counts.
- `ruff check .` + `ruff format --check .`: clean.
- pylint score: ids.py 10.00/10, collision_scan.py 10.00/10; all
  modified files >= 8.92 (no regression).
- bandit: clean on all new files; the one MEDIUM finding in
  knowledge_graph.py is on lines 385/407 (pre-existing SQL string
  construction in timeline queries), not in code touched by this PR.
- pre_push_check.sh: ALL CHECKS PASSED.

Refs: deferred from PR #1572 (Copilot finding #13). Styleguide audit
documented above: 8 fix/migration + 11 enumerated exempt sites.
2026-05-30 12:44:47 -07:00
mvalentsevandC-LaForest 22804b8e08 fix(backends): guard mtime_appeared discard behind _freshness membership
Prevent redundant quarantine re-run when a fresh ChromaBackend instance
opens a palace that was already quarantined by another instance in the
same process.  The mtime_appeared transition (cached 0.0 -> real mtime)
now only triggers a discard if the instance previously tracked the path,
distinguishing genuine file appearance from first-access default.

Addresses gemini-code-assist review on PR #1602.

Co-Authored-By: C-LaForest <C-LaForest@users.noreply.github.com>
2026-05-30 22:40:01 +05:00
mvalentsevandC-LaForest c19d894f68 fix(backends): re-arm HNSW quarantine gate on mtime change and explicit reconnect (#1573)
The _quarantined_paths gate fired once per palace per process and never
re-armed after external in-place writes (closet_llm, mine, compress)
that drift HNSW segments.  The MCP server path (make_client static) had
zero discard logic -- quarantine never re-armed even on inode change.

Extend the discard guard in _client() from inode_changed-only to
inode_changed or mtime_changed or mtime_appeared.  Add a guarded
discard in mcp_server._get_client() before make_client(), and an
unconditional discard in tool_reconnect().

Remove dead _auto_repair / palace-daemon comment (does not exist in
this codebase) and correct misleading _get_collection retry-path
comments that overclaimed quarantine re-runs.

Co-Authored-By: C-LaForest <C-LaForest@users.noreply.github.com>
2026-05-30 22:40:01 +05:00
mvalentsevand0xKingVee9527 7b1f6247d0 fix(backends): detect sub-threshold segments by link_lists state, not data size
chromadb pre-allocates data_level0.bin at index creation (~168 KB for
384-dim embeddings) regardless of record count, so the previous
data-size-vs-floor heuristic in _segment_appears_healthy could not
distinguish a single-record segment (sub-threshold, never persisted)
from an interrupted persist.

Restructure _segment_appears_healthy: when index_metadata.pickle is
absent, check link_lists.bin instead of data_level0.bin size.  Empty or
absent link_lists + absent metadata = sub-threshold (never persisted).
Non-empty link_lists + absent metadata = interrupted persist.

Co-Authored-By: 0xKingVee9527 <0xWinner98@users.noreply.github.com>
2026-05-30 22:35:34 +05:00
mvalentsev b11f41484c style(test): use _HNSW_MISSING_METADATA_DATA_FLOOR constant instead of magic 1024 2026-05-30 22:35:34 +05:00
mvalentsevandTim Harmon 10d1ce6a94 fix(backends): lower HNSW bloat-guard thresholds to fix sub-50k persist (#1579)
_HNSW_BLOAT_GUARD set batch_size and sync_threshold to 50,000 to
prevent link_lists.bin sparse-file bloat in pre-1.5.x Python chromadb
(#344).  chromadb >=1.5.4 Rust bindings do not exhibit that bloat.

The 50k guard meant any mine under 50,000 drawers never triggered
chromadb's _persist(), leaving index_metadata.pickle absent and
link_lists.bin empty.  quarantine_stale_hnsw then renamed the segment
on every cold open after a 300s mtime gap, accumulating .drift-*
directories indefinitely.

Lower both thresholds to 2 (empirical Rust-side minimum; 1 is rejected
with InvalidArgumentError) so any mine of 2+ drawers triggers a natural
persist.  Verified: batch_size=2 with 20k records produces
link_lists.bin at 171 KB with zero sparse-file inflation.

Existing palaces retain the old 50k thresholds in their collection
metadata until the user runs repair --mode from-sqlite.

Co-Authored-By: Tim Harmon <tim-harmon@users.noreply.github.com>
2026-05-30 22:35:34 +05:00
Arnold Wender 038a7e0e15 fix(convo_miner): preserve blank lines and indentation in AI responses
_chunk_by_exchange stripped every line, joined them with single spaces, and
silently dropped blank lines. That violated the verbatim-always principle
stated in CLAUDE.md and contradicted the function's own docstring, which
claimed 'The full AI response is preserved verbatim.'

Concrete consequences before this change:
- paragraph breaks fused: 'para1\n\npara2' → 'para1 para2'
- list items fused: '1. a\n2. b' → '1. a 2. b'
- code fences destroyed: indented code collapsed to a single line
- search quality degraded because tokenization changed at ingest

Fix is surgical: keep each line as-is, join on newline, trim only
trailing newlines produced by the loop stopping at the next '>' turn.

The fallback path _chunk_by_paragraph has a narrower version of the
same bug (it strips each paragraph); that is out of scope here and left
for a follow-up.
2026-05-30 14:47:16 +02:00
Arnold Wender 29db125b9b fix(closet_llm): replace non-ASCII symbols in progress output (#1034)
GBK consoles (Windows PowerShell/CMD default) cannot encode U+2713 (✓),
U+2717 (✗), and U+2014 (—). The same class of UnicodeEncodeError fixed
in miner.py via #681 affects closet_llm.py and cli.py.

Replace with ASCII equivalents: [OK], [FAIL], [!], and hyphen.
2026-05-30 14:47:03 +02:00
Arnold Wender 818e4f3b31 fix(normalize): use utf-8-sig to handle BOM-prefixed transcript files
Windows exports of Claude Code JSONL sessions prepend a UTF-8 BOM
(\xef\xbb\xbf). With encoding='utf-8', json.loads() raises JSONDecodeError
on the first line, _try_claude_code_jsonl silently skips every line, and
the file falls through as raw text — losing all structured message content.

utf-8-sig strips the BOM transparently and is backward-compatible with
BOM-free files on all platforms.
2026-05-30 14:47:01 +02:00
Igor Lins e Silva 48f4eb6dfe test(embedding): expect 3 hf_hub_download calls (model + .onnx_data weights + tokenizer)
The EmbeddinggemmaONNX lazy-load now fetches the ONNX external-weights file
(model.onnx_data) in addition to the model graph and tokenizer, so a single
warm-up issues 3 downloads, not 2. The lazy-load-once invariant is unchanged
(InferenceSession and Tokenizer.from_file are still each built exactly once).
2026-05-30 02:57:09 -03:00
Igor Lins e Silva 9b7cfc9940 Merge pull request #1617 from mvalentsev/fix/1611-config-json-str-type
fix(backends): repair missing _type in collection config (#1611)
2026-05-30 00:37:13 -03:00
undeadindustriesandCursor c420a9f66c fix(cursor): address igorls review on PR #1632
Resolves the maintainer review on the Cursor IDE support PR. Cursor-only
scope; cross-IDE items (wing-naming convention, shared-file merge order)
are coordinated on the separate Antigravity branch.

followup_message default (the one "decide before merge" item):
- Keep the stop-hook followup ON by default. Cursor's transcript format
  is undocumented and mempalace/normalize.py has no Cursor parser, so the
  background `mempalace mine --mode convos` is best-effort only and does
  not yet yield clean verbatim drawers. The followup is therefore the
  load-bearing verbatim-capture path; defaulting it off would leave a
  default Cursor install capturing nothing.
- Add an opt-out (MEMPAL_CURSOR_SILENT=1, or MEMPAL_VERBOSE=false) for
  users who want the Claude-style "zero tokens in chat" behaviour. The
  hook still mines and keeps its counters/markers when silenced.
- Correct the misleading "background mine captures it" comments in the
  save and precompact hooks; update hooks/cursor/README.md and the guide.

Hygiene fixes:
- Drop the hardcoded "version" field from .cursor-plugin/plugin.json and
  marketplace.json (mempalace/version.py is the single source of truth);
  tests now assert the field stays absent.
- Remove the committed .cursor-plugin/{commands,skills} symlinks (they
  break on Windows clones with core.symlinks=false and were redundant
  with the real repo-root components that `source: "."` already serves);
  add a guard test that no symlinks exist under .cursor-plugin/.
- Document the preCompact synchronous-mine timeout tradeoff and that an
  incremental/append-only mine is recoverable if killed (no corruption).
- Add a Cursor-namespaced, daily-throttled TTL sweep (MEMPAL_STATE_TTL_DAYS,
  default 30) to lib/common.sh that GCs stale cursor_*.count/.pending only,
  after the kill-switch check; shared logs and antigravity_* are untouched.

Verification: full suite green (2424 passed, 3 skipped), ruff check +
format clean, bash -n clean on all cursor scripts. +30 Cursor tests
(followup opt-out, state GC, TTL validation, no-symlink/version guards).

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-30 11:13:51 +10:00
undeadindustriesandCursor df295bd743 fix(antigravity): atomic counter write, background --version probe, state-file GC
Addresses igorls' review on PR #1633 (antigravity branch only):

- Atomic counter write: add mempal_write_counter_atomic (same-dir temp +
  mv -f rename) and use it in the save hook, replacing the truncate-then
  -write printf that the comment falsely called "atomic". Concurrent
  readers now always see a complete value.
- Background the expensive probe: `mempalace --version` pays the full
  chromadb/onnx cold-start import (the mine subparser imports
  mempalace.miner before argparse handles --version), so running it in
  the foreground blew the <500ms save budget. Probe + mine + pending
  -marker cleanup now run in one detached subshell; the foreground
  returns immediately. This also retires the kill -0 watcher (the prior
  gemini fix) since cleanup is now sequential within the mine's own
  shell, removing the sibling-PID hazard entirely.
- State-file GC: add mempal_state_ttl_days (default 30, env
  MEMPAL_STATE_TTL_DAYS) and mempal_gc_stale_state, a daily-throttled
  sweep (antigravity_last_sweep marker) that removes stale
  antigravity_save_count_*, antigravity_pending_*, and
  antigravity_woke_* artifacts. Called after the kill-switch check so a
  disabled hook touches nothing; specific name globs leave shared logs
  untouched.

Tests: atomic-counter behavior + no-temp-leftover, single-subshell
structure (no wait/kill -0/MINE_PID), backgrounded-probe timing (3s
stub returns in <2s), async log polling for the missing-module path,
GC sweep/throttle/TTL-validation, and GC gated by the kill switch.
Also hardens the wake-missing test to pin MEMPAL_PYTHON so a shell
-exported interpreter can't defeat the simulation.

bash 3.2.57 safe, fail-open on every path, {}-only Stop output.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-30 10:18:46 +10:00
Igor Lins e Silva 387d50df27 Merge pull request #1647 from MemPalace/dependabot/pip/ruff-0.15.15
build(deps-dev): bump ruff from 0.15.14 to 0.15.15
2026-05-29 19:38:12 -03:00
Igor Lins e Silva 55d4e379a7 Merge pull request #1652 from MemPalace/fix/file-already-mined-multi-mtime-groups
fix(palace): file_already_mined iterates all groups when source_file has multiple parent_drawer_id mining passes
2026-05-29 19:37:56 -03:00
Milla J 13c38ac588 fix(palace): file_already_mined iterates all groups when source_file has multiple parent_drawer_id mining passes
Under the additive-mining model (drawer history preserved across re-mines),
a single source_file can have multiple parent_drawer_id groups in the
palace, one per mining pass. Each group carries its own stored
source_mtime and normalize_version.

`file_already_mined` previously used `collection.get(where={"source_file": X},
limit=1)` in the `extract_mode is None` branch and only checked the single
returned row's metadata. ChromaDB's `get(..., limit=1)` has no ordering
guarantee across multiple matching rows, so the returned row was effectively
arbitrary. When ChromaDB happened to return a stale group (older mining
pass with a different stored mtime), the function returned False, the
additive miner concluded the file had changed, and wrote yet another
duplicate group of drawers for a file that had not actually changed since
the last successful mine.

Steady-state failure: for any source_file that has ever been edited (so
that the palace contains groups with different stored mtimes), each re-mine
has a probability of spuriously concluding "file changed" and writing
another duplicate group. Each spurious group compounds the problem because
its stored mtime can also trigger the next spurious re-mine. Storage grows
without bound; search results, hallway counts, and entity-frequency stats
become inflated proportionally.

The fix mirrors the paginated-iteration pattern already used in the
`extract_mode is not None` branch — iterate every drawer for the
source_file in 1000-row pages, short-circuit on the first matching group.
A correct group is one that passes the existing checks: normalize_version
not stale; if check_mtime, stored source_mtime within 0.001 seconds of the
current file mtime. The two branches (extract_mode is None vs set) collapse
into one loop that skips the extract_mode check when no extract_mode was
specified.

Trade-off: average-case cost rises from O(1) limit=1 query to O(N/1000)
paginated scan. For typical sources (1-3 groups) the cost is unchanged
because the loop short-circuits on the first matching group within the
first page. For pathological sources with thousands of groups, the cost
is O(number-of-pages-until-match) — still bounded, no longer flaky.

RED test pins the failure space deterministically

`test_file_already_mined_handles_multiple_groups_under_one_source_file`
uses a MockCollection that simulates two parent_drawer_id groups under one
source_file: a STALE group (older mtime) and a CURRENT group (matching
mtime). The mock returns the STALE group when called with limit=1
(worst-case ChromaDB ordering) and returns BOTH groups when called with
limit=1000 (what a correctly-iterating implementation must do). Test asserts
file_already_mined returns True even when limit=1 picks the stale group.

  - Against pre-fix code: test FAILS (function returns False because
    limit=1 picks stale group, mtime mismatch returns False)
  - Against post-fix code: test PASSES (iteration finds the current group,
    short-circuits to True)

Verified RED-then-GREEN locally. Existing 4 file_already_mined tests in
tests/test_miner.py continue to pass:
  - test_file_already_mined_check_mtime
  - test_file_already_mined_scopes_convo_extract_mode
  - test_file_already_mined_extract_mode_paginates_large_sources
  - test_file_already_mined_returns_false_for_stale_normalize_version

Verification

  - macOS Python 3.12 (local) full pytest  : 2268 passed, 0 failed
  - Linux Python 3.9.25  (OrbStack)        : 2260 passed, 0 failed
  - Linux Python 3.11.15 (OrbStack)        : 2261 passed, 0 failed
  - Linux Python 3.13.13 (OrbStack)        : 2261 passed, 0 failed
  - ruff check + ruff format --check       : all clean

Provenance

Surfaced during the per-query audit on the PR #1628 amendment cycle (the
search for every bare `where={"source_file": ...}` query in the repo).
One of six sites identified. The other five are legitimately file-global
in intent (closet purges, full-rebuild deletes, paginated mode-filtered
scans). This site is the one whose failure mode mirrors the cross-group
stitching pattern PR #1628 fixed at the searcher layer.
2026-05-29 11:50:44 -07:00
dependabot[bot] a0508cb065 build(deps-dev): bump ruff from 0.15.14 to 0.15.15
Bumps [ruff](https://github.com/astral-sh/ruff) from 0.15.14 to 0.15.15.
- [Release notes](https://github.com/astral-sh/ruff/releases)
- [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md)
- [Commits](https://github.com/astral-sh/ruff/compare/0.15.14...0.15.15)

---
updated-dependencies:
- dependency-name: ruff
  dependency-version: 0.15.15
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-29 07:59:47 +00:00
Tom Boucher 2d4fae1ac3 fix(mcp): retry stale-index transient in tool_search 2026-05-27 12:37:02 -04:00