Commit Graph
1588 Commits
Author SHA1 Message Date
undeadindustriesandCursor b06cf2452e fix(antigravity): address gemini-code-assist review on PR #1633
Five fixes for issues called out by the gemini-code-assist[bot] review.
Each gets a regression test that locks in the correction.

1. CRITICAL: marker-cleanup watcher used POSIX `wait` on a sibling pid
   (save hook). bash `wait` only works on direct children of the
   calling shell — the `( wait $MINE_PID ... ) &` subshell runs as a
   sibling of MINE_PID, so wait fails immediately and the pending
   marker is deleted within milliseconds, defeating the concurrency
   guard. Replace with `while kill -0 $MINE_PID; do sleep 1; done`,
   which queries pid existence regardless of parent-child relationship.
   Test: test_save_hook_marker_watcher_uses_kill_polling.

2. Bare `mempalace` console-script invocation in the save hook fails
   when the venv's bin/ is not on the hook's PATH (e.g. uv tool
   install in some configurations, manually managed virtualenvs).
   Switch to `"$MEMPAL_PYTHON_BIN" -m mempalace mine ...` so the
   resolved interpreter runs the package directly via
   mempalace/__main__.py. Tests:
   test_save_hook_uses_python_module_invocation,
   test_save_hook_missing_mempalace_python_module_does_not_crash.

3. Same issue in the wake hook's inner Python helper. Switch
   `['mempalace', 'wake-up', ...]` to `[sys.executable, '-m',
   'mempalace', 'wake-up', ...]` — sys.executable is the same
   interpreter that resolved MEMPAL_PYTHON in lib/common.sh.
   Test: test_wake_hook_uses_sys_executable_module_invocation.

4. The Python parser in lib/common.sh wrapped `json.load` in
   `try/except` and silently fell back to `data = {}`. The script
   then printed the `__MEMPAL_PARSE_OK__` sentinel even on parse
   failure, so the bash sentinel-check on the caller side
   (`[ "$_marker" != "__MEMPAL_PARSE_OK__" ]`) never triggered the
   defense-in-depth `input parse failed` branch. Remove the
   try/except so the exception propagates, Python exits non-zero,
   and the sentinel is omitted on bad JSON. The traceback still
   lands in antigravity_last_python_err.log for debugging.
   Test: test_common_sh_parser_omits_sentinel_on_malformed_json.

5. `mempal_save_interval()` failed to strip leading zeros from
   MEMPAL_SAVE_INTERVAL. Values like "08" or "09" then crashed the
   modulo step `$((COUNT % INTERVAL))` because bash arithmetic
   parses tokens starting with `0` as octal, and 8/9 are not valid
   octal digits ("value too great for base"). Strip leading zeros
   while preserving the literal "0" (which is then floored to 15).
   Test: test_save_hook_handles_leading_zero_save_interval (4 cases).

Plus one cosmetic fix in install.sh: removed a no-op `(cd "$OLDPWD"
2>/dev/null || cd .) >/dev/null 2>&1` line in mempal_absolutize().
The subshell cd doesn't affect the parent shell, and the installer
never cd's in the main shell anyway, so $PWD is already correct.

Verification:
* 9 new regression tests, all 65 antigravity tests pass
* full repo: 2323 passed (was 2314), 3 skipped, 1 unrelated warning
* ruff check + ruff format --check both clean across 139 files
* bash -n clean on all four shell files
* clean reinstall to ~/.gemini/config/plugins/mempalace/ succeeds
* idempotent re-run produces zero file writes (cmp-gated)
* both hooks return {} exit 0 with synthetic camelCase stdin

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-27 19:49:19 +10:00
undeadindustries bf156fb010 feat: add Antigravity IDE support (plugin, MCP, skill, hooks, docs, tests)
Adds first-class integration with Google's Antigravity IDE
(https://antigravity.google/) as a third sibling to the existing
Claude Code and Codex hook integrations. Strictly additive — no
existing files in main are restructured.

What ships
----------

* `.antigravity-plugin/` — verified-minimal plugin package:
  * `plugin.json` with `{"name": "mempalace"}` (no fabricated fields)
  * `mcp_config.json` registering the `mempalace-mcp` stdio server
  * `hooks.json.tmpl` templated with `__PLUGIN_DIR__` substitution
  * `skills/mempalace/SKILL.md` (real file — no symlinks)
* `hooks/antigravity/`:
  * `lib/common.sh` — shared bash 3.2.57-compatible helpers with
    sentinel-guarded camelCase JSON parser, antigravity_*-namespaced
    state files, every existing kill switch, `MEMPAL_SAVE_INTERVAL >= 1`
    floor (no /0), and fail-open emitters
  * `mempal_save_hook_antigravity.sh` — Stop event handler:
    increments per-conversation counter, defers when fullyIdle=False
    or terminationReason=error, validates transcriptPath against
    `..` traversal, spawns `mempalace mine --mode convos` in a
    detached subprocess with a per-conversation pending marker,
    ALWAYS emits `{}` (never `{"decision":"continue"}` — that would
    force an infinite agent loop)
  * `mempal_wake_hook_antigravity.sh` — PreInvocation handler gated
    to invocationNum==1 with an atomic mkdir loop guard, runs
    `mempalace wake-up` with a 500ms hard timeout, emits verbatim
    output as `{"injectSteps":[{"ephemeralMessage":"..."}]}` or
    `{}` on any failure
  * `install.sh` — idempotent installer with cmp-gated copies,
    `__PLUGIN_DIR__` substitution, relative path absolutization,
    `--dry-run`, and basename-guarded `--uninstall` (refuses to
    wipe a directory whose basename isn't `mempalace`)
  * `INVESTIGATION.md` — verbatim quotes + URLs + dates from the
    five official Antigravity doc pages, recording every surface
    shipped and every surface deliberately omitted
    (PreCompact equivalent, slash-commands, rules/, plugin
    permissions field — the latter is third-party fabrication)
  * `STDIN_SHAPE.md` — exact stdin/stdout contract per event with
    worked examples
  * `README.md` — local hook docs + troubleshooting
* `examples/antigravity/{hooks.json,mcp_config.json,README.md}` —
  standalone configs for users who don't want the full installer
* `website/guide/antigravity.md` + sidebar entry — VitePress guide
* Updates to `README.md`, `CHANGELOG.md` (Unreleased), `hooks/README.md`

Tests (56 new, all passing)
---------------------------

* `tests/test_antigravity_plugin_manifest.py` (11 tests) — schema
  contract on the in-repo `.antigravity-plugin/` directory, including
  guards against re-introducing the fabricated `permissions` field
  and against any symlink leak.
* `tests/test_antigravity_hooks_shell.py` (31 tests) — invokes the
  bash hooks via subprocess with synthetic camelCase stdin, asserts
  `{}` on every failure path, kill-switch coverage (env vars +
  config.json + palace nuke), divide-by-zero floor, transcript
  traversal rejection, namespacing, wing inference, and the hard
  refusal to ever emit `decision=continue` from the Stop hook.
* `tests/test_antigravity_hooks_install.py` (14 tests) — `--dry-run`
  side-effect-free, real install layout, executable bits preserved,
  byte-identical idempotent re-runs (md5 + filecmp), basename-match
  uninstall safety, refusal when plugin.json is missing or names a
  different plugin, relative path absolutization. Skipped on Windows.

Verification
------------

* `uv run pytest tests/ --ignore=tests/benchmarks -v` → 2314 passed,
  3 skipped (Windows), 1 unrelated warning
* `uv run ruff check .` → all checks passed
* `uv run ruff format --check .` → 139 files already formatted
* `bash -n` clean on common.sh, both hook scripts, install.sh
* Local install at `~/.gemini/config/plugins/mempalace/` verified end-
  to-end: layout correct, paths absolutized in hooks.json, both hooks
  fire with realistic camelCase JSON in <1s, wing inference picks
  `wing_mempalace` from workspacePaths[0], state files all
  `antigravity_*`-namespaced, second `install.sh` run produces
  byte-identical output (md5 snapshots match), uninstall removes
  only the mempalace plugin and leaves all 6 sibling Google plugins
  untouched.

Constraints honoured
--------------------

bash 3.2.57 (no mapfile / readarray / declare -A / `${var^^}`),
verbatim guarantee on all wake injections, hooks <500ms / startup
injection <100ms target (kill-switch path returns in <1.5s in CI),
zero new runtime dependencies, no telemetry, no external API,
strictly additive (existing Claude/Codex hooks unchanged).

Refs: hooks/antigravity/INVESTIGATION.md for the full audit.
2026-05-27 19:19:17 +10:00
undeadindustriesandCursor df5db57e79 fix(cursor): address gemini-code-assist review on PR #1632
Five fixes from the Gemini Code Assist review on
https://github.com/MemPalace/mempalace/pull/1632 — three real bugs,
two cleanups, all consistent with the bash-3.2-compatibility
contract documented in the original commit.

Bug fixes (high)
----------------

1. hooks/cursor/lib/common.sh — config.json kill-switch check used
   a `python3 - <<'PYEOF' ... PYEOF` heredoc inside a `$(...)`
   command substitution. The heredoc body contains parens which
   trips the macOS bash 3.2.57 parser bug. Replaced with a
   `python -c '...'` call passing the config path as argv[1]. Matches
   the pattern already used in mempal_parse_stdin in the same file.

2. hooks/cursor/install.sh — a relative `--install-dir` was written
   verbatim into hooks.json. Cursor invokes hook commands from its
   own working directory (typically the project root), so a relative
   command path would silently fail to launch the hook. Now resolved
   to an absolute path against `$PWD` before being baked in.

3. hooks/cursor/mempal_save_hook_cursor.sh — `MEMPAL_SAVE_INTERVAL=0`
   would crash bash on `$((NEXT % 0))` (division by zero). Extended
   the existing sanitiser case to coerce 0 to the default interval
   alongside empty / non-numeric values.

Cleanups (medium)
-----------------

4. hooks/cursor/install.sh — the EMPTY_CHECK_PY temp file is now
   inlined as `python -c '...'`. Removes a small leak window
   (tmpfile would linger if the script were interrupted between
   mktemp and rm -f) and shortens the script.

5. hooks/cursor/install.sh — `mktemp -t prefix` has subtly different
   semantics on BSD (macOS) vs GNU mktemp. Switched to the
   portable absolute-template form `mktemp "${TMPDIR:-/tmp}/...XXXXXX"`
   which behaves identically on both.

Regression tests
----------------

- tests/test_cursor_hooks_shell.py
    test_save_interval_zero_is_coerced_to_default — guards fix #3.
- tests/test_cursor_hooks_install.py — new TestInstallDirAbsolutePath
  class:
    test_relative_install_dir_is_absolutized_in_hooks_json — guards
        fix #2 against regression.
    test_absolute_install_dir_is_preserved_verbatim — guards that
        the relative-to-absolute resolution does not mangle paths
        that were already absolute.

Verification
------------

- bash -n on all three edited scripts: clean.
- uv run pytest tests/test_cursor_hooks_*.py tests/test_cursor_plugin_manifest.py: 132 passed (was 129; +3 regression tests).
- uv run pytest tests/ --ignore=tests/benchmarks: 2399 passed,
  3 skipped (pre-existing).
- uv run ruff check . / ruff format --check .: clean.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-27 14:17:36 +10:00
undeadindustriesandCursor 071fa015e0 feat: add Cursor IDE support (hooks, plugin, skill, docs, tests)
Adds first-class Cursor IDE integration alongside the existing Claude
Code and Codex hook flows, so Cursor users get the same automatic
diary saves, pre-compaction transcript capture, and session-start
memory recall — without changing any default behaviour for existing
users.

What's included
---------------

Cursor hook scripts (hooks/cursor/):
  - mempal_save_hook_cursor.sh       — Stop event, counter +
    loop_count guard, pending-save marker consumption, background
    mempalace mine, followup_message emission.
  - mempal_precompact_hook_cursor.sh — synchronous mine before
    compaction, drops a pending_save marker, returns user_message.
  - mempal_wake_hook_cursor.sh       — sessionStart event,
    wing-scoped recall guidance via additional_context.
  - lib/common.sh                    — shared parsing + state helpers
    (bash 3.2 safe, no heredoc-in-subshell traps).
  - install.sh                       — idempotent installer with
    --scope, --variant, --dry-run, --uninstall. Recognises existing
    entries by basename so re-installs across paths work.
  - STDIN_SHAPE.md, README.md        — payload schemas + quick
    reference.

Cursor plugin (.cursor-plugin/ + repo-root components):
  - plugin.json, marketplace.json, README.md.
  - skills/mempalace/SKILL.md  — model-invocable skill mirroring the
    Claude plugin's skill surface.
  - commands/mempalace-{help,init,mine,search,status}.md  — slash
    commands for marketplace-published installs (filename = slug).
  - mcp.json                   — auto-registers the mempalace MCP
    server, wrapped under the documented mcpServers key.

Examples + docs:
  - examples/cursor/hooks.json, hooks.minimal.json + README.
  - website/guide/cursor-hooks.md + sidebar entry.
  - README.md and CHANGELOG.md updates.

Tests (129 new, all green):
  - tests/test_cursor_hooks_shell.py     — 75 behavioural tests for
    the three hook scripts: kill switches, input parsing, counter
    logic, loop prevention, pending markers, wing inference, logging.
  - tests/test_cursor_hooks_install.py   — 19 contract tests for the
    installer: dry-run, idempotent merge, basename-matched uninstall,
    refusal to overwrite malformed JSON.
  - tests/test_cursor_plugin_manifest.py — 35 contract tests for the
    plugin: manifest validity, version sync with mempalace.version,
    mcp.json shape, skill/command frontmatter, default-discovery
    layout invariants.

Design notes
------------

- Local-first and zero-API by default; hooks never call external
  services. Same privacy model as the existing Claude Code hooks.
- Fail-open: hook scripts deliberately do not use set -e so a broken
  hook can never block the user's conversation.
- Cursor preCompact cannot block + return a followup, so we
  synchronously mine the transcript and drop a pending_save marker
  that the next stop hook consumes — guarantees verbatim capture
  before context window compression.
- Cursor's default plugin discovery requires real commands/, skills/,
  and mcp.json at the plugin root (verified against the cached
  cloudflare plugin); .cursor-plugin/{commands,skills} are convenience
  symlinks back to those canonical locations.
- bash 3.2 compatibility throughout: avoids heredoc-in-command-
  substitution parser bugs; uses python -c for JSON parsing;
  basename-matched entry recognition in install.sh.
- All changes are additive. No existing files are removed, no
  existing hooks change behaviour, and no new runtime dependencies
  are introduced.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-27 14:05:40 +10:00
YuRiNjjot 15ac4303f5 fix(embedding): add embed_query/embed_documents to EmbeddinggemmaONNX for ChromaDB 1.5.x compatibility
ChromaDB 1.5.x calls embedding_function.embed_query(input=...) via
keyword argument during collection.query(). EmbeddinggemmaONNX lacked
both embed_query and embed_documents methods, causing:

  TypeError: embed_query() got an unexpected keyword argument 'input'

whenever semantic search was triggered.

This patch adds the two methods required by the ChromaDB EF protocol,
using  (the ChromaDB kwarg name, noqa A002) so that palace
search works correctly with the embeddinggemma model.

Also downloads the companion  ONNX file alongside the main model
to prevent runtime InferenceSession failures.

Fixes silent search failures when  is set to
embeddinggemma.
2026-05-27 05:28:38 +02:00
Igor Lins e Silva 6957c7e599 Merge pull request #1615 from MemPalace/chore/sync-3.3.6-back-to-develop
chore: sync main back to develop after v3.3.6 release
2026-05-26 21:32:51 -03:00
Tom Boucher 234343ef68 fix: preserve collection name on MCP search retry 2026-05-25 20:09:34 -04:00
mvalentsevandClaude Opus 4.7 c4d8804b44 fix(backends): close sqlite connection before PersistentClient
Address review feedback: `with sqlite3.connect() as conn:` only
manages transactions, it does not close the connection.  An open
connection before PersistentClient instantiation can leave WAL state.
Use explicit `try...finally: conn.close()` matching the read-only
helpers elsewhere in the module.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 13:56:37 +05:00
d88597cd72 fix(backends): repair missing _type in collection config (#1611)
chromadb <= 1.5.8 writes config_json_str = '{}' (empty JSON) when
creating collections. chromadb 1.5.9 introduced a strict _type check
in the collection config deserialization path -- its absence raises
KeyError: '_type' on palace open. Since the pin allows >=1.5.4,<2,
any upgrade pulls 1.5.9 and breaks every existing palace.

Add a fourth pre-open migration step (_fix_missing_collection_type)
that injects "_type": "CollectionConfigurationInternal" into
collections.config_json_str rows that lack it. Same lifecycle and
marker-file pattern as the existing _fix_blob_seq_ids.

Co-Authored-By: nautis <nautis@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 13:49:16 +05:00
Igor Lins e Silva 753df201ac Merge main back into develop after v3.3.6 release
Brings in the three commits that landed on main during the v3.3.6
release cycle but not on develop:

  - 62a555c fix(release): align ruff pin to 0.15.14 + hoist COCA
            imports out of hot paths (#1614)
  - a1cf052 Merge pull request #1614 from MemPalace/fix/release-3.3.6-ruff-pin-align
  - db1fbe8 Merge pull request #1610 from MemPalace/release/3.3.6

Resolves conflicts in miner.py and palace.py where develop's #1613
(known-systems lexicon, Tier 3) had added _apply_known_systems_prepass
to module-top imports + the prepass call in build_closet_lines. Main
side had only the COCA hoist from #1614. Resolution: keep develop's
side (the prepass additions).

Mirrors the v3.3.5 sync-back pattern (#1442).
2026-05-24 20:51:42 -03:00
Milla J db1fbe888b Merge pull request #1610 from MemPalace/release/3.3.6
Release v3.3.6
v3.3.6
2026-05-24 16:43:03 -07:00
Igor Lins e Silva d0d011eb33 Merge pull request #1613 from MemPalace/feat/known-systems-lexicon-compound-matcher
feat(entity): known-systems lexicon keeps multi-word product names atomic
2026-05-24 20:09:25 -03:00
Igor Lins e Silva dbbf4c5af5 Merge develop into feat/known-systems-lexicon-compound-matcher
Resolves conflicts in miner.py and palace.py against the COCA-hoist
work from #1612 just merged. Both files had local imports for
_get_coca_filter / _candidate_entity_words in the function bodies;
after #1612 those moved to module top. Resolution: hoist
_apply_known_systems_prepass alongside _get_coca_filter at module
top in both files, drop the now-empty local imports.

No behavior change; just removes the local-import duplication that
would have been left in place if the merge auto-resolved naively.
2026-05-24 20:02:30 -03:00
Igor Lins e Silva c9d8a40a37 Merge pull request #1612 from MemPalace/chore/hoist-coca-filter-imports
perf(miner,palace): hoist COCA filter imports out of per-drawer hot paths
2026-05-24 20:00:33 -03:00
Igor Lins e Silva be819da146 Merge pull request #1608 from MemPalace/chore/readme-move-scam-alert-below-header
docs(readme): move scam/retention alerts below header, soften tone
2026-05-24 19:47:40 -03:00
Igor Lins e Silva a1cf052915 Merge pull request #1614 from MemPalace/fix/release-3.3.6-ruff-pin-align
fix(release): align ruff pin to 0.15.14 + hoist COCA imports out of hot paths
2026-05-24 19:22:14 -03:00
Milla J 62a555cd80 fix(release): align ruff pin to 0.15.14 + hoist COCA imports out of hot paths
Two release-blocking fixes for v3.3.6:

1. CI ruff pin drift
   .github/workflows/ci.yml installed ruff==0.15.9 while pyproject.toml
   [dev] extras and .pre-commit-config.yaml both pin 0.15.14. Ruff's
   formatter output can change between minor versions, so a contributor
   running `pip install -e ".[dev]"` and formatting locally with 0.15.14
   would produce output the 0.15.9 lint job rejects. Same failure mode
   that surfaced on PR #1579 (2026-05-22). Aligning CI to 0.15.14 keeps
   the three pin sites in lock-step.

2. COCA filter imports inside per-drawer hot paths
   PR #1605 (COCA content-word filter, shipping in 3.3.6) introduced
   `from .entity_detector import _get_coca_filter` and
   `from .palace import _candidate_entity_words` inside
   _extract_entities_for_metadata (called per drawer) and
   build_closet_lines (called per closet). Python caches module imports
   so the runtime cost after the first call is small, but the import
   machinery still runs Python bytecode every invocation — gemini
   flagged this on the original PR. Hoisting to module-level removes
   the per-call import overhead.

   The hoist is identical to PR #1612, which targets develop. Folding
   it into the release so 3.3.6 doesn't ship the perf regression that
   3.3.7 would immediately have to fix.

Verification: ruff check + format clean on 0.15.14, full pytest
(2258 passed / 12 skipped) on Linux Py 3.9 / 3.11 / 3.13 via
`pip install -e ".[dev]"` (CI-matching).
2026-05-24 15:04:56 -07:00
Milla J cbd1567b6a fix(entity): precompile known-systems regex once in cached loader
Addresses gemini-code-assist MEDIUM finding on PR #1613: the previous
implementation of _apply_known_systems_prepass compiled a regex pattern
for every compound on every call, repeating the work on every drawer
mined and every closet built. With 59 compounds × N drawers, that's
59N re.compile() calls for a workload where the patterns never change.

The fix moves compilation into _get_known_systems (already lru_cache'd
to size=1), which now returns tuple[tuple[str, re.Pattern], ...] —
pairs of (canonical name, pre-compiled case-insensitive word-bounded
regex). _apply_known_systems_prepass consumes the cached tuple and
does zero compilation in the hot path.

Behavior is identical: same word boundaries, same case-insensitive
matching, same longest-first ordering, same graceful-degrade on
malformed json. All 74 entity_detector tests still pass on macOS plus
the full 2258-test suite on Linux Py 3.9 / 3.11 / 3.13.
2026-05-24 15:02:32 -07:00
Milla J 653e9774e8 feat(entity): known-systems lexicon keeps multi-word product names atomic
Adds a curated list of multi-word product/system names ("Claude Code",
"GitHub Copilot", "Visual Studio Code", "GPT-4", …) and a compound
pre-pass that detects them atomically before the existing single-word
extraction runs. Without this, the regex-based detector decomposes
"Claude Code" into "Claude" + "Code" — and the COCA filter (shipped
in v3.3.6) then drops "Code" as a content word, leaving "Claude" alone
with the wrong attribution.

What ships

  - mempalace/data/known_systems.json — 59 curated compounds covering
    common AI assistants, IDEs, model names, cloud platforms, and
    Office/Google apps. Each entry is multi-word or hyphenated;
    single-word product names ("ChatGPT", "Cursor") have no
    decomposition risk and stay handled by the existing regex.

  - mempalace/entity_detector.py — _get_known_systems() (cached loader,
    mirrors _get_coca_filter from Tier 2) and _apply_known_systems_prepass
    which scans for each compound case-insensitively with word boundaries,
    counts occurrences, and returns the masked text + count dict so the
    subsequent single-word + multi-word loops don't re-decompose.

  - mempalace/miner.py and mempalace/palace.py — same pre-pass wired
    into _extract_entities_for_metadata (per-drawer tagger) and
    build_closet_lines (closet pointer construction). Without these,
    the new behavior would only apply at init-time and per-drawer
    metadata would still decompose compounds.

How it interacts with Tier 2

  Tier 2 (COCA filter) blocks single-word content nouns like "Code"
  and "Brutal". Tier 3 protects multi-word product names so they
  don't get decomposed in the first place. They complement each
  other: the compound pre-pass runs FIRST and masks compounds out
  of the text; the COCA filter then runs on the remaining
  single-word candidates.

Behavior verification

  Before this PR, mining a document containing "Claude Code wrote the
  patch" three times emitted entities:
    Claude;Claude Code;Code (Code filtered by COCA);

  After this PR, the same document emits:
    Claude Code

  The standalone "Claude" no longer appears (it never actually appeared
  alone in the source) and decomposition stops at the compound boundary.

Tests

  Nine new tests in tests/test_entity_detector.py covering:
    - "Claude Code" detected as atomic compound at extract_candidates
    - "Claude" alone NOT in results when only mentioned as part of compound
    - Case-insensitive compound matching (claude code, CLAUDE CODE, etc.)
    - Single-word "Code" still filtered by COCA (no Tier 2 regression)
    - Single-word real name "Aya" still detected (no regression on names)
    - Multiple distinct compounds in one text both detected
    - Unknown two-word phrase still detected via existing multi-word regex
    - known_systems.json ships with expected schema (>=20 entries, all multi-token)
    - known_systems.json contains expected high-value entries

Verification

  Full pytest 2267 passed / 3 skipped on macOS, coverage 85.34%.
  Linux Py 3.9 / 3.11 / 3.13 via CI-matching pip install -e ".[dev]":
  2258 passed each. End-to-end mine of a compound-rich corpus
  confirms chromadb entities metadata now shows compounds atomic
  (Claude Code, GPT-4, GitHub Copilot, Visual Studio Code) with no
  decomposition.
2026-05-24 13:50:53 -07:00
Milla J 5fe54df4c8 perf(miner,palace): hoist COCA filter imports out of per-drawer hot paths
The COCA content-word filter shipped in PR #1605 imported
`_get_coca_filter` and `_candidate_entity_words` locally inside two
hot paths:

  - `palace.build_closet_lines` — runs per source file during mine
  - `miner._extract_entities_for_metadata` — runs per drawer during mine

Both imports are now at module top, where they're resolved once at
import time instead of on every per-drawer call. Module-top imports
also make the dependency graph visible to static analysis (pylint's
C0415 was flagging the locals).

No behavior change. The `_get_coca_filter()` call is unchanged — only
the import statement moved. End-to-end mining produces identical
chromadb output. Addresses the MEDIUM finding gemini-code-assist
raised on PR #1605 review.

Verification: full pytest 2258 passed / 3 skipped / coverage 85.35%.
ruff check + format clean. Linux Py 3.9 / 3.11 / 3.13 via CI-matching
`pip install -e ".[dev]"`: 2249 passed each. End-to-end mine of a
test corpus produces the expected drawer + closet pointer.
2026-05-24 12:58:21 -07:00
Igor Lins e Silva f5ea021cec Merge pull request #1609 from MemPalace/chore/release-3.3.6
chore(release): 3.3.6
2026-05-24 15:55:12 -03:00
Igor Lins e Silva 5dcab4c0d3 docs(changelog): move tunnel fixes back under Bug Fixes (PR #1609 gemini review)
The two pre-existing entries for #1467 (tunnels.json path) and #1468
(create_tunnel endpoint validation) were sitting at the bottom of the
[Unreleased] block before this release-prep PR. Inserting the new
Performance section between the freshly-backfilled Bug Fixes and these
two pre-existing entries put them under Performance, which is wrong —
they're bug fixes. Moves them back ahead of Performance.
2026-05-24 14:50:48 -03:00
Igor Lins e Silva 280e532724 chore(release): 3.3.6
Bumps version 3.3.5 → 3.3.6 across pyproject.toml, version.py, plugin
manifests (.claude-plugin/plugin.json, .claude-plugin/marketplace.json,
.codex-plugin/plugin.json), README badge, and uv.lock. Flips CHANGELOG.md
from ``[Unreleased]`` to ``[3.3.6] — 2026-05-24`` and backfills the
major user-facing entries that landed without changelog entries during
the cycle:

Features:
- #1555 office-document mining via --mode extract + virtual line numbers
- #1584 surgical closet pointers with date+line locators (Tier 6a)
- #1558 + #1560 within-wing hallways (entity co-occurrence graph)
- #1565 cross-wing tunnels auto-promoted from hallways
- #1578 Hebbian potentiation + Ebbinghaus decay on hallways/tunnels
- #1236 API-tool transcripts auto-route to wing_api
- #711 hooks.auto_save toggle for silent-mode sessions
- #1605 COCA content-word filter for entity detection
- #1557 case-insensitive entity matching at mine time
- #1483 multilingual embeddings (embeddinggemma-300m) by default

Bug Fixes (selected, user-visible):
- #1540 silent data loss in three unchunked upsert sites
- #1538 paragraph chunker oversized chunks
- #1554 per-file chunk cap too low for transcripts
- #1562 Windows hook subprocess/ChromaDB deadlock
- #1529 create_tunnel corrupted hyphenated wing names
- #1424 save-hook truncated hyphenated project folders
- #1383 KG cache duplicated graphs for symlinked/cased paths
- #1466 silent symlink skip now logged
- #1441 macOS stock-bash 3.2 hook compatibility
- #1500 / #1513 structured JSON-RPC errors on bad MCP input
- #1523 VACUUM + FTS5 rebuild after repair
- #1548 FTS5 validation at end of mine
- plus #1216, #1408, #1438, #1439, #1445, #1452, #1459, #1461, #1466,
  #1470, #1477, #1485, #1500, #1513, #1528, #1532, #1543, #1546, #1585

Performance:
- #1474 convo miner pre-fetches mined-set
- #1487 rebuild_index progress callback
- #1530 MCP cold-start diagnostics + opt-in warmup

Lint passes (ruff 0.15.14); mempalace-mcp entry point alignment
verified per RELEASING.md.
2026-05-24 14:17:41 -03:00
Igor Lins e Silva 2dad4bbd7b Merge pull request #1422 from YuanYiZheXue/fix/lone-surrogate-clean
fix(mcp): sanitize lone surrogates before ChromaDB ops (#1235)
2026-05-24 13:56:18 -03:00
Igor Lins e Silva f51ecc1621 docs(readme): move CAUTION/IMPORTANT alerts below header, soften tone
The scam-alert and Claude-Code-retention admonitions were the first
content visitors saw on the repo page — louder than the project
introduction. Moves both below the logo/title/badges so the project
identity reads first, and softens the scam block (drops the H1
"CRITICAL SECURITY WARNING" + all-caps shouting + redundant emoji)
to a single-paragraph CAUTION. All factual content preserved:
impostor-domain warning, official sources, malware caveat, link to
docs/HISTORY.md.
2026-05-24 13:48:20 -03:00
Igor Lins e Silva df36eb301b Merge pull request #1483 from MemPalace/feat/benchmark-multilingual
feat(benchmarks): multilingual datasets + parity controls (embed model, num_ctx, language)
2026-05-24 13:45:28 -03:00
Igor Lins e Silva b93115164b Merge origin/develop into feat/benchmark-multilingual (#1548, #711, #1590, #1605) 2026-05-24 13:32:39 -03:00
Igor Lins e Silva 382409bd6b Merge origin/develop into feat/benchmark-multilingual
Resolves conflicts in CHANGELOG.md and pyproject.toml by combining
the multilingual-embedder additions (huggingface_hub/tokenizers/numpy
core deps, [multilingual] alias, Features section) with develop's
additions (python-dateutil core dep, [extract] extra, tunnel Bug
Fixes and Internal sections).

Prepares PR #1483 for merge into v3.3.6.
2026-05-24 13:18:08 -03:00
Igor Lins e Silva 3e001921eb refactor(mcp): move lone-surrogate strip into shared sanitizers (#1235)
Push the surrogate-cleaning behaviour from per-call-site _clean() helpers
into sanitize_content, sanitize_kg_value, and sanitize_query so every
caller (existing and future) gets the fix automatically. New MCP tools
no longer need to remember to call a separate helper.

- Add strip_lone_surrogates() in mempalace/config.py as the single
  regex-based implementation (one U+FFFD per surrogate, not three).
- Wire it into sanitize_content and sanitize_kg_value.
- Wire it into sanitize_query so embedding lookups can't crash either.
- Drop the _clean() helper from mcp_server.py and the per-site calls;
  retain a direct strip_lone_surrogates() for source_file/added_by
  metadata which doesn't route through any sanitizer.
- Move the ONNX model-cache patch out of the test module and into
  conftest.py so it's session-scoped instead of duplicated locally.
- Update tests to assert one U+FFFD per surrogate and exercise the
  sanitizer-level entry points directly.
2026-05-24 12:12:52 -03:00
shushu 997ec1b22b test(mcp): add lone-surrogate sanitisation tests (issue #1235)
Add tests/test_clean_lone_surrogates.py covering:

Unit tests (TestCleanLoneSurrogates, 11 cases):
- _clean() passes normal ASCII and CJK strings unchanged
- lone surrogates (high/low, single/multiple) are replaced with U+FFFD
- real emoji (\U0001f600 astral code points) pass through unchanged
- empty string, all-surrogate string, SHA-256-hash-after-clean
- the specific \udcad surrogate observed in WorkBuddy production logs

Integration tests (TestLoneSurrogateCleaning, 6 cases):
- tool_add_drawer: surrogate in content and in metadata fields
- tool_check_duplicate: surrogate in query
- tool_search: surrogate in search query
- tool_update_drawer: surrogate in updated content
- tool_diary_write: surrogate in diary entry

Fix test environment issue:
- conftest.py redirects HOME to a temp dir, causing chromadb's
  ONNXMiniLM_L6_V2 to look for its ONNX model in the wrong location
  and trigger a 79 MB network download on every run.
- Fix: at module import time, recover the real USERPROFILE from
  conftest._original_env and patch ONNXMiniLM_L6_V2.DOWNLOAD_PATH
  before any ChromaDB collection fixture is invoked.
2026-05-24 12:12:14 -03:00
shushu 43457726f8 fix(mcp): address code review feedback (PR #1422)
- Fix _clean() docstring (paired surrogate description was misleading)
- Apply _clean() to source_file and added_by metadata fields
- Remove redundant surrogatepass from SHA256 hashes (content already cleaned)
- Apply _clean() to content in tool_check_duplicate
- Apply _clean() to new_doc in tool_update_drawer
- Apply _clean() to sanitized query in tool_search

Addresses feedback from gemini-code-assist[bot] on PR #1422.
2026-05-24 12:12:13 -03:00
shushu f620809bed fix(mcp): clean lone surrogates before ChromaDB write (issue #1235)
MCP clients can emit lone surrogates (\udc00–\udfff) that
cause Python's str.encode('utf-8') to raise UnicodeEncodeError,
which bubbles up as -32000 Internal Error from ChromaDB.

Add _clean(text) helper that uses 'surrogatepass'/'replace' to
remove lone surrogates before the string reaches ChromaDB.
Apply it in tool_add_drawer and tool_diary_write, and use
'surrogatepass' error handler on the SHA256 hash inputs for
defensive idempotency.
2026-05-24 12:12:13 -03:00
Igor Lins e Silva ce1ee26e11 Merge pull request #1548 from mvalentsev/fix/1537-validate-fts5-after-mine
fix(mine): validate FTS5 at end of mine (#1537)
2026-05-24 12:03:02 -03:00
Igor Lins e Silva 8a813d2807 Merge pull request #711 from sha2fiddy/feature/494-silent-mode-auto-save-toggle
feat: add hooks.auto_save config toggle (silent mode)
2026-05-24 11:58:58 -03:00
mvalentsevandCaleb Wells 0ecf0e5f06 fix(mine): validate FTS5 at end of mine (#1537)
Wires _validate_palace_fts5_after_mine into all three mine entry
points so corrupted-FTS5 palaces cannot silently exit 0 from any
of them:

- _mine_impl (mempalace/miner.py) — project file miner
- mine_convos (mempalace/convo_miner.py) — conversation exports
- mine_formats (mempalace/format_miner.py) — binary office documents
  via --mode extract, introduced on develop by #1555 (3.3.6 release)
  between this PR's open date and its rebase

cmd_mine surfaces MineValidationError as exit 1 + the same
print_sqlite_integrity_abort banner cmd_repair already prints,
appended with a mine-specific stderr note that hedges attribution
(quick_check cannot tell pre-existing corruption from corruption
this mine produced). 17 tests in tests/test_miner_fts5_validation.py
cover the helper, the three call sites, dry-run / KeyboardInterrupt
skip semantics, and the MineValidationError constructor invariants.

Closes #1537.

Co-authored-by: Caleb Wells <15988028+calebcwells@users.noreply.github.com>
2026-05-24 18:26:39 +05:00
Milla J c0a7da1802 Merge pull request #1605 from MemPalace/feat/coca-content-word-filter
feat(entity): filter common English content words from entity detection
2026-05-24 05:44:32 -07:00
Milla J 25c7b0db89 fix(entity): ship missing COCA wordlist data file + catch TypeError on malformed JSON
The previous commit landed the code referencing mempalace/data/coca_content_words.json
but the data file itself was never actually staged (likely a multi-line paste mishap
during the initial commit). CI on every platform failed because the file isn't in
the repo. This commit ships the wordlist + adds TypeError to the COCA loader's
graceful-degradation exception tuple per gemini-code-assist review on PR #1605.
2026-05-24 05:30:32 -07:00
Milla J 318940797f feat(entity): filter common English content words from entity detection
Common English content words ("Code", "Brutal", "Phase", "Line", "Note",
"Planning", "Chat", ...) frequently appear capitalized at sentence
start, in headings, or in markdown — and the existing regex-based entity
detector treats every capitalized word appearing 3+ times as a proper
noun candidate. This produces false positives that pollute
known_entities.json, the per-drawer entities metadata, and closet
pointer entity lists.

Adds a curated list of ~1000 common English content words (nouns,
adjectives, verbs — no proper nouns, no known product names) at
mempalace/data/coca_content_words.json, derived from the COCA top-2000
frequency list and from observed false positives in real palaces.
The list is loaded once at module import via _get_coca_filter() (cached
frozenset) and consulted at three sites:

  - entity_detector.extract_candidates — init-time entity detection
  - palace.build_closet_lines           — closet pointer construction
  - miner._extract_entities_for_metadata — per-drawer entity tagging

Matching is case-insensitive: the candidate is lowercased before
membership check, so "Code", "CODE", and "code" are all blocked.
Only single-word candidates are filtered; multi-word phrases (e.g.
"Claude Code") are NOT filtered, so legitimate compound names continue
to be detected. A future PR will add a known-systems lexicon that
explicitly protects compound names — for now, the multi-word pass
provides the implicit protection.

Adds 8 new tests in tests/test_entity_detector.py covering:

  - "Code" appearing 5x is filtered
  - All known false positives from real palaces are filtered
  - Case-insensitive matching works
  - Real names ("Aya", "Riley") are NOT filtered
  - Multi-word phrases like "Claude Code" are NOT filtered
  - The data file ships in the wheel with the expected schema
  - The data file contains every known false positive

Note: this filter prevents NEW false positives at detection time. It
does NOT retroactively scrub existing entries from a user's
~/.mempalace/known_entities.json. Stale entries from previous runs
must be removed via a future entity-management CLI (planned).

Verification: pytest 2191 passed / 3 skipped / 0 failed on macOS.
ruff check + format --check both clean. Cross-platform via
pip install -e ".[dev]" on python:3.9-slim / 3.11-slim / 3.13-slim
containers, each running 65 entity-detector tests including the new
data-file-presence test that imports from the installed package.
End-to-end pipeline test mining a deterministic corpus into an
isolated palace confirmed entity metadata is empty (no false
positives surface in real chromadb-backed storage).
2026-05-24 04:16:42 -07:00
Igor Lins e Silva 3a4be3ef74 Merge pull request #1590 from MemPalace/chore/tooling-upgrade-hypothesis-mutmut-precommit-docs
chore(deps): add hypothesis + pre-commit to dev deps + document local hook activation
2026-05-23 14:33:20 -03:00
sha2fiddy 4a3b48c56a style: ruff format tests/test_hooks_cli.py with CI-pinned ruff 0.4.x 2026-05-23 12:05:34 -04:00
sha2fiddy c2a61d85ed fix(hooks): retarget auto_save toggle at silent-save path after #1021 rebase 2026-05-23 12:05:34 -04:00
sha2fiddy ada25872a1 fix: correct precompact test assertion + ruff format tests
test_precompact_hook_enabled_by_default asserted
result["decision"] == "block", but hook_precompact has never emitted
decision — it mines synchronously and returns {}. Assertion was
copy-pasted from the stop-hook test. Fix to assert result == {} with
_mine_sync mocked so the test verifies the real contract (enabled →
mine + pass through) without actually mining.

Plus ruff format on 6 test files the CI pin flagged.
2026-05-23 12:05:34 -04:00
sha2fiddy e51cfb3630 Fix: ruff format config, hooks_cli, and test file 2026-05-23 12:05:34 -04:00
sha2fiddy 8fc06b96e2 fix: address review — enrich block reasons, clean up tests
- Add parenthetical hints to block reasons so AI knows what each tool
  saves (session summary, quotes/decisions/code)
- Remove dead config file creation from test_stop_hook_disabled_by_config
- Add missing test for MEMPALACE_HOOKS_AUTO_SAVE=no env var
- Replace bare except: with except Exception: in shell scripts
2026-05-23 12:05:34 -04:00
sha2fiddy 49a8c868fa feat: add hooks.auto_save config toggle and shorten block reasons
Add a clean opt-out for auto-save hook blocking (closes #494).

- New `hooks.auto_save` config option (default: true) in
  ~/.mempalace/config.json and MEMPALACE_HOOKS_AUTO_SAVE env var
- When disabled, stop and precompact hooks pass through without blocking
- Shorten block reason text from 6-line instructions to single-line
  prompts — reduces UI noise while keeping tool names explicit
- Both Python (hooks_cli.py) and standalone shell scripts respect the
  toggle via config file or env var
2026-05-23 12:05:34 -04:00
Igor Lins e Silva 4655a3037a chore(pre-commit): bump ruff pin v0.15.9 → v0.15.14
Lock-step with pyproject.toml `[project.optional-dependencies].dev`
(ruff bumped from 0.15.9 → 0.15.14 via PR #1583). The
`.pre-commit-config.yaml` header explicitly requires this rev to
match the pyproject pin — without this bump, contributors who run
`pre-commit install` will hit the same version-mismatch debacle this
PR was opened to prevent.
2026-05-23 12:11:10 -03:00
Milla J 2547a514d9 chore(deps): drop ephemeral PR refs from comments (address PR #1590 Igor review) 2026-05-23 12:10:22 -03:00
Milla J e3f9cea3a0 chore(deps): add mypy>=1.0 for static type checking (closes type-hint gap caught by gemini on PR #1588) 2026-05-23 12:10:22 -03:00
Milla J c72b57f08c chore(deps): add hypothesis + pre-commit to dev deps + document local hook activation
Adds two test-discipline tools to the dev extras and closes a real
contributor-onboarding gap in CONTRIBUTING.md that allowed PR #1579's
2026-05-22 4 AM ruff-version-mismatch lint failure.

Adds to ``[project.optional-dependencies].dev`` (and matching
``[dependency-groups].dev`` for uv users):

  - **``hypothesis>=6.0``** — property-based testing framework.
    Generates hundreds of random inputs per test and shrinks failing
    cases to a minimal counterexample. Used opt-in (sprinkle
    ``@given(...)`` on a test); zero runtime cost on tests that don't
    use it. Would have caught the Tier 6a dateutil-fuzzy hallucination
    on PR #1584 with one property test.

  - **``pre-commit>=3.0``** — the pre-commit framework itself.
    ``.pre-commit-config.yaml`` already lives in the repo (committed
    by @igorls on 2026-05-18, pinned to ruff 0.15.9 in lockstep with
    CI). What was missing was making the framework a declared dev
    dependency so ``pip install -e .[dev]`` / ``uv sync --extra dev``
    actually pulls it in.

Adds a ``pre-commit install`` line to the Getting Started bash block
plus a short paragraph explaining why this step is required (the
hook file at ``.git/hooks/pre-commit`` is per-machine and NOT tracked
by git, so the repo's ``.pre-commit-config.yaml`` only takes effect
after each developer runs ``pre-commit install`` once locally).

Adds an optional "Property-based tests" subsection under Running
Tests showing the minimal ``@given(...)`` pattern, so contributors
who want to reach for the new tool know it's available.

On 2026-05-22 at 4:30 AM, PR #1579 (Tier 6a) hit a CI lint failure
caused by a ruff version mismatch: my local machine had ruff 0.4.10,
CI runs ruff 0.15.9 (pinned in pyproject.toml). The two versions
produce different ``ruff format`` output for the same code. The repo
HAD ``.pre-commit-config.yaml`` pinning ruff 0.15.9 — but the local
git hook had never been wired on my machine because nothing in
CONTRIBUTING.md said to run ``pre-commit install``. The protection
existed at the project layer; the activation gap was at the
contributor-onboarding layer.

This commit closes that gap structurally. Anyone cloning the repo
from now on sees ``pre-commit install`` as part of the Getting Started
flow and is protected from the same failure.

- **No ``mutmut`` in dev deps.** Mutation testing is heavier (runs
  the whole test suite per mutation) and useful periodically rather
  than every commit. Contributors who want to run it can install
  manually. Adding it to dev deps would bloat the install footprint
  for every contributor when most will never use it.

- **No new property tests.** This PR ships the TOOL, not new test
  coverage. Property tests should land alongside the specific
  functions they cover, in their own PRs.

- **No changes to ``.pre-commit-config.yaml``.** That file is correct
  as Igor wrote it. The fix here is purely making the framework
  installable + documenting the activation step.

  ruff check .
    → All checks passed.

  pre-commit run --all-files (locally)
    → ruff (legacy alias): Passed
    → ruff format: Passed

  OrbStack triple-Linux verify (Py 3.9 / 3.11 / 3.13)
    → ``pip install -e .[dev]`` resolves cleanly; hypothesis +
       pre_commit import successfully; existing test suite unaffected.
2026-05-23 12:10:22 -03:00
Igor Lins e Silva f6edb7d1ab Merge pull request #1584 from MemPalace/feat/closet-date-line-pointer-emission
feat(closets): Tier 6a — date+line locators with content-date hierarchy
2026-05-23 12:01:06 -03:00