Commit Graph
1108 Commits
Author SHA1 Message Date
Igor Lins e Silva 2dad4bbd7b Merge pull request #1422 from YuanYiZheXue/fix/lone-surrogate-clean
fix(mcp): sanitize lone surrogates before ChromaDB ops (#1235)
2026-05-24 13:56:18 -03:00
Igor Lins e Silva df36eb301b Merge pull request #1483 from MemPalace/feat/benchmark-multilingual
feat(benchmarks): multilingual datasets + parity controls (embed model, num_ctx, language)
2026-05-24 13:45:28 -03:00
Igor Lins e Silva b93115164b Merge origin/develop into feat/benchmark-multilingual (#1548, #711, #1590, #1605) 2026-05-24 13:32:39 -03:00
Igor Lins e Silva 382409bd6b Merge origin/develop into feat/benchmark-multilingual
Resolves conflicts in CHANGELOG.md and pyproject.toml by combining
the multilingual-embedder additions (huggingface_hub/tokenizers/numpy
core deps, [multilingual] alias, Features section) with develop's
additions (python-dateutil core dep, [extract] extra, tunnel Bug
Fixes and Internal sections).

Prepares PR #1483 for merge into v3.3.6.
2026-05-24 13:18:08 -03:00
Igor Lins e Silva 3e001921eb refactor(mcp): move lone-surrogate strip into shared sanitizers (#1235)
Push the surrogate-cleaning behaviour from per-call-site _clean() helpers
into sanitize_content, sanitize_kg_value, and sanitize_query so every
caller (existing and future) gets the fix automatically. New MCP tools
no longer need to remember to call a separate helper.

- Add strip_lone_surrogates() in mempalace/config.py as the single
  regex-based implementation (one U+FFFD per surrogate, not three).
- Wire it into sanitize_content and sanitize_kg_value.
- Wire it into sanitize_query so embedding lookups can't crash either.
- Drop the _clean() helper from mcp_server.py and the per-site calls;
  retain a direct strip_lone_surrogates() for source_file/added_by
  metadata which doesn't route through any sanitizer.
- Move the ONNX model-cache patch out of the test module and into
  conftest.py so it's session-scoped instead of duplicated locally.
- Update tests to assert one U+FFFD per surrogate and exercise the
  sanitizer-level entry points directly.
2026-05-24 12:12:52 -03:00
shushu 997ec1b22b test(mcp): add lone-surrogate sanitisation tests (issue #1235)
Add tests/test_clean_lone_surrogates.py covering:

Unit tests (TestCleanLoneSurrogates, 11 cases):
- _clean() passes normal ASCII and CJK strings unchanged
- lone surrogates (high/low, single/multiple) are replaced with U+FFFD
- real emoji (\U0001f600 astral code points) pass through unchanged
- empty string, all-surrogate string, SHA-256-hash-after-clean
- the specific \udcad surrogate observed in WorkBuddy production logs

Integration tests (TestLoneSurrogateCleaning, 6 cases):
- tool_add_drawer: surrogate in content and in metadata fields
- tool_check_duplicate: surrogate in query
- tool_search: surrogate in search query
- tool_update_drawer: surrogate in updated content
- tool_diary_write: surrogate in diary entry

Fix test environment issue:
- conftest.py redirects HOME to a temp dir, causing chromadb's
  ONNXMiniLM_L6_V2 to look for its ONNX model in the wrong location
  and trigger a 79 MB network download on every run.
- Fix: at module import time, recover the real USERPROFILE from
  conftest._original_env and patch ONNXMiniLM_L6_V2.DOWNLOAD_PATH
  before any ChromaDB collection fixture is invoked.
2026-05-24 12:12:14 -03:00
shushu 43457726f8 fix(mcp): address code review feedback (PR #1422)
- Fix _clean() docstring (paired surrogate description was misleading)
- Apply _clean() to source_file and added_by metadata fields
- Remove redundant surrogatepass from SHA256 hashes (content already cleaned)
- Apply _clean() to content in tool_check_duplicate
- Apply _clean() to new_doc in tool_update_drawer
- Apply _clean() to sanitized query in tool_search

Addresses feedback from gemini-code-assist[bot] on PR #1422.
2026-05-24 12:12:13 -03:00
shushu f620809bed fix(mcp): clean lone surrogates before ChromaDB write (issue #1235)
MCP clients can emit lone surrogates (\udc00–\udfff) that
cause Python's str.encode('utf-8') to raise UnicodeEncodeError,
which bubbles up as -32000 Internal Error from ChromaDB.

Add _clean(text) helper that uses 'surrogatepass'/'replace' to
remove lone surrogates before the string reaches ChromaDB.
Apply it in tool_add_drawer and tool_diary_write, and use
'surrogatepass' error handler on the SHA256 hash inputs for
defensive idempotency.
2026-05-24 12:12:13 -03:00
Igor Lins e Silva ce1ee26e11 Merge pull request #1548 from mvalentsev/fix/1537-validate-fts5-after-mine
fix(mine): validate FTS5 at end of mine (#1537)
2026-05-24 12:03:02 -03:00
Igor Lins e Silva 8a813d2807 Merge pull request #711 from sha2fiddy/feature/494-silent-mode-auto-save-toggle
feat: add hooks.auto_save config toggle (silent mode)
2026-05-24 11:58:58 -03:00
mvalentsevandCaleb Wells 0ecf0e5f06 fix(mine): validate FTS5 at end of mine (#1537)
Wires _validate_palace_fts5_after_mine into all three mine entry
points so corrupted-FTS5 palaces cannot silently exit 0 from any
of them:

- _mine_impl (mempalace/miner.py) — project file miner
- mine_convos (mempalace/convo_miner.py) — conversation exports
- mine_formats (mempalace/format_miner.py) — binary office documents
  via --mode extract, introduced on develop by #1555 (3.3.6 release)
  between this PR's open date and its rebase

cmd_mine surfaces MineValidationError as exit 1 + the same
print_sqlite_integrity_abort banner cmd_repair already prints,
appended with a mine-specific stderr note that hedges attribution
(quick_check cannot tell pre-existing corruption from corruption
this mine produced). 17 tests in tests/test_miner_fts5_validation.py
cover the helper, the three call sites, dry-run / KeyboardInterrupt
skip semantics, and the MineValidationError constructor invariants.

Closes #1537.

Co-authored-by: Caleb Wells <15988028+calebcwells@users.noreply.github.com>
2026-05-24 18:26:39 +05:00
Milla J c0a7da1802 Merge pull request #1605 from MemPalace/feat/coca-content-word-filter
feat(entity): filter common English content words from entity detection
2026-05-24 05:44:32 -07:00
Milla J 25c7b0db89 fix(entity): ship missing COCA wordlist data file + catch TypeError on malformed JSON
The previous commit landed the code referencing mempalace/data/coca_content_words.json
but the data file itself was never actually staged (likely a multi-line paste mishap
during the initial commit). CI on every platform failed because the file isn't in
the repo. This commit ships the wordlist + adds TypeError to the COCA loader's
graceful-degradation exception tuple per gemini-code-assist review on PR #1605.
2026-05-24 05:30:32 -07:00
Milla J 318940797f feat(entity): filter common English content words from entity detection
Common English content words ("Code", "Brutal", "Phase", "Line", "Note",
"Planning", "Chat", ...) frequently appear capitalized at sentence
start, in headings, or in markdown — and the existing regex-based entity
detector treats every capitalized word appearing 3+ times as a proper
noun candidate. This produces false positives that pollute
known_entities.json, the per-drawer entities metadata, and closet
pointer entity lists.

Adds a curated list of ~1000 common English content words (nouns,
adjectives, verbs — no proper nouns, no known product names) at
mempalace/data/coca_content_words.json, derived from the COCA top-2000
frequency list and from observed false positives in real palaces.
The list is loaded once at module import via _get_coca_filter() (cached
frozenset) and consulted at three sites:

  - entity_detector.extract_candidates — init-time entity detection
  - palace.build_closet_lines           — closet pointer construction
  - miner._extract_entities_for_metadata — per-drawer entity tagging

Matching is case-insensitive: the candidate is lowercased before
membership check, so "Code", "CODE", and "code" are all blocked.
Only single-word candidates are filtered; multi-word phrases (e.g.
"Claude Code") are NOT filtered, so legitimate compound names continue
to be detected. A future PR will add a known-systems lexicon that
explicitly protects compound names — for now, the multi-word pass
provides the implicit protection.

Adds 8 new tests in tests/test_entity_detector.py covering:

  - "Code" appearing 5x is filtered
  - All known false positives from real palaces are filtered
  - Case-insensitive matching works
  - Real names ("Aya", "Riley") are NOT filtered
  - Multi-word phrases like "Claude Code" are NOT filtered
  - The data file ships in the wheel with the expected schema
  - The data file contains every known false positive

Note: this filter prevents NEW false positives at detection time. It
does NOT retroactively scrub existing entries from a user's
~/.mempalace/known_entities.json. Stale entries from previous runs
must be removed via a future entity-management CLI (planned).

Verification: pytest 2191 passed / 3 skipped / 0 failed on macOS.
ruff check + format --check both clean. Cross-platform via
pip install -e ".[dev]" on python:3.9-slim / 3.11-slim / 3.13-slim
containers, each running 65 entity-detector tests including the new
data-file-presence test that imports from the installed package.
End-to-end pipeline test mining a deterministic corpus into an
isolated palace confirmed entity metadata is empty (no false
positives surface in real chromadb-backed storage).
2026-05-24 04:16:42 -07:00
Igor Lins e Silva 3a4be3ef74 Merge pull request #1590 from MemPalace/chore/tooling-upgrade-hypothesis-mutmut-precommit-docs
chore(deps): add hypothesis + pre-commit to dev deps + document local hook activation
2026-05-23 14:33:20 -03:00
sha2fiddy 4a3b48c56a style: ruff format tests/test_hooks_cli.py with CI-pinned ruff 0.4.x 2026-05-23 12:05:34 -04:00
sha2fiddy c2a61d85ed fix(hooks): retarget auto_save toggle at silent-save path after #1021 rebase 2026-05-23 12:05:34 -04:00
sha2fiddy ada25872a1 fix: correct precompact test assertion + ruff format tests
test_precompact_hook_enabled_by_default asserted
result["decision"] == "block", but hook_precompact has never emitted
decision — it mines synchronously and returns {}. Assertion was
copy-pasted from the stop-hook test. Fix to assert result == {} with
_mine_sync mocked so the test verifies the real contract (enabled →
mine + pass through) without actually mining.

Plus ruff format on 6 test files the CI pin flagged.
2026-05-23 12:05:34 -04:00
sha2fiddy e51cfb3630 Fix: ruff format config, hooks_cli, and test file 2026-05-23 12:05:34 -04:00
sha2fiddy 8fc06b96e2 fix: address review — enrich block reasons, clean up tests
- Add parenthetical hints to block reasons so AI knows what each tool
  saves (session summary, quotes/decisions/code)
- Remove dead config file creation from test_stop_hook_disabled_by_config
- Add missing test for MEMPALACE_HOOKS_AUTO_SAVE=no env var
- Replace bare except: with except Exception: in shell scripts
2026-05-23 12:05:34 -04:00
sha2fiddy 49a8c868fa feat: add hooks.auto_save config toggle and shorten block reasons
Add a clean opt-out for auto-save hook blocking (closes #494).

- New `hooks.auto_save` config option (default: true) in
  ~/.mempalace/config.json and MEMPALACE_HOOKS_AUTO_SAVE env var
- When disabled, stop and precompact hooks pass through without blocking
- Shorten block reason text from 6-line instructions to single-line
  prompts — reduces UI noise while keeping tool names explicit
- Both Python (hooks_cli.py) and standalone shell scripts respect the
  toggle via config file or env var
2026-05-23 12:05:34 -04:00
Igor Lins e Silva 4655a3037a chore(pre-commit): bump ruff pin v0.15.9 → v0.15.14
Lock-step with pyproject.toml `[project.optional-dependencies].dev`
(ruff bumped from 0.15.9 → 0.15.14 via PR #1583). The
`.pre-commit-config.yaml` header explicitly requires this rev to
match the pyproject pin — without this bump, contributors who run
`pre-commit install` will hit the same version-mismatch debacle this
PR was opened to prevent.
2026-05-23 12:11:10 -03:00
Milla J 2547a514d9 chore(deps): drop ephemeral PR refs from comments (address PR #1590 Igor review) 2026-05-23 12:10:22 -03:00
Milla J e3f9cea3a0 chore(deps): add mypy>=1.0 for static type checking (closes type-hint gap caught by gemini on PR #1588) 2026-05-23 12:10:22 -03:00
Milla J c72b57f08c chore(deps): add hypothesis + pre-commit to dev deps + document local hook activation
Adds two test-discipline tools to the dev extras and closes a real
contributor-onboarding gap in CONTRIBUTING.md that allowed PR #1579's
2026-05-22 4 AM ruff-version-mismatch lint failure.

Adds to ``[project.optional-dependencies].dev`` (and matching
``[dependency-groups].dev`` for uv users):

  - **``hypothesis>=6.0``** — property-based testing framework.
    Generates hundreds of random inputs per test and shrinks failing
    cases to a minimal counterexample. Used opt-in (sprinkle
    ``@given(...)`` on a test); zero runtime cost on tests that don't
    use it. Would have caught the Tier 6a dateutil-fuzzy hallucination
    on PR #1584 with one property test.

  - **``pre-commit>=3.0``** — the pre-commit framework itself.
    ``.pre-commit-config.yaml`` already lives in the repo (committed
    by @igorls on 2026-05-18, pinned to ruff 0.15.9 in lockstep with
    CI). What was missing was making the framework a declared dev
    dependency so ``pip install -e .[dev]`` / ``uv sync --extra dev``
    actually pulls it in.

Adds a ``pre-commit install`` line to the Getting Started bash block
plus a short paragraph explaining why this step is required (the
hook file at ``.git/hooks/pre-commit`` is per-machine and NOT tracked
by git, so the repo's ``.pre-commit-config.yaml`` only takes effect
after each developer runs ``pre-commit install`` once locally).

Adds an optional "Property-based tests" subsection under Running
Tests showing the minimal ``@given(...)`` pattern, so contributors
who want to reach for the new tool know it's available.

On 2026-05-22 at 4:30 AM, PR #1579 (Tier 6a) hit a CI lint failure
caused by a ruff version mismatch: my local machine had ruff 0.4.10,
CI runs ruff 0.15.9 (pinned in pyproject.toml). The two versions
produce different ``ruff format`` output for the same code. The repo
HAD ``.pre-commit-config.yaml`` pinning ruff 0.15.9 — but the local
git hook had never been wired on my machine because nothing in
CONTRIBUTING.md said to run ``pre-commit install``. The protection
existed at the project layer; the activation gap was at the
contributor-onboarding layer.

This commit closes that gap structurally. Anyone cloning the repo
from now on sees ``pre-commit install`` as part of the Getting Started
flow and is protected from the same failure.

- **No ``mutmut`` in dev deps.** Mutation testing is heavier (runs
  the whole test suite per mutation) and useful periodically rather
  than every commit. Contributors who want to run it can install
  manually. Adding it to dev deps would bloat the install footprint
  for every contributor when most will never use it.

- **No new property tests.** This PR ships the TOOL, not new test
  coverage. Property tests should land alongside the specific
  functions they cover, in their own PRs.

- **No changes to ``.pre-commit-config.yaml``.** That file is correct
  as Igor wrote it. The fix here is purely making the framework
  installable + documenting the activation step.

  ruff check .
    → All checks passed.

  pre-commit run --all-files (locally)
    → ruff (legacy alias): Passed
    → ruff format: Passed

  OrbStack triple-Linux verify (Py 3.9 / 3.11 / 3.13)
    → ``pip install -e .[dev]`` resolves cleanly; hypothesis +
       pre_commit import successfully; existing test suite unaffected.
2026-05-23 12:10:22 -03:00
Igor Lins e Silva f6edb7d1ab Merge pull request #1584 from MemPalace/feat/closet-date-line-pointer-emission
feat(closets): Tier 6a — date+line locators with content-date hierarchy
2026-05-23 12:01:06 -03:00
Igor Lins e Silva e5bbc5f9ef Merge pull request #1583 from MemPalace/dependabot/pip/ruff-0.15.14
build(deps-dev): bump ruff from 0.15.9 to 0.15.14
2026-05-22 19:25:37 -03:00
Milla J 65f24b3bdd fix(closets): address PR #1584 Igor review — wire + harden Tier 6a
Igor's review on PR #1584 (2026-05-22) flagged four issues:

  1. The feature wasn't wired into any production caller — the new
     ``drawer_metas`` kwarg on ``build_closet_lines`` had no real
     consumer in ``miner.py`` / ``diary_ingest.py``, so the 4-segment
     pointer form only existed in tests. Real palaces kept emitting
     the legacy 3-segment shape.
  2. ``_extract_content_date`` hallucinated dates on benign inputs.
     ``dateutil.parser.parse(fuzzy=True)`` would accept anything with
     digits and return a plausible-looking but wrong date —
     ``Version 3.3.6`` → ``2006-03-03``, ``Tested with 1000 drawers``
     → ``1000-05-22``, ``tmp_random_file_5`` → ``2026-05-05``, etc.
     Mtime almost never got reached because fuzzy returned *something*
     from filename or body first. Bad dates were silently persisted
     to ChromaDB.
  3. ``python-dateutil`` was an undeclared dependency, available only
     transitively via ``chromadb → kubernetes → python-dateutil``. Not
     a contract — upstream kubernetes has been trending toward
     stdlib-only.
  4. Two-digit-year disambiguation (70 → 19xx / 00-69 → 20xx) had no
     test pinning the boundary.

This commit addresses all four.

## Changes

### Issue 2 — kill the hallucination (the load-bearing fix)

``mempalace/miner.py``:

- New ``_VALID_DATE_RE`` gate. Three accepted shapes (all require a
  4-digit year explicitly):

    1. Numeric YYYY-MM-DD with ``[-/.\\s]`` separators
       (covers ISO and space-normalized filenames)
    2. Month-name + day + year ("November 8 2024", "Nov 8 2024")
    3. Day + month-name + year ("8 November 2024")

  Partial dates ("2024-06", "April 6", "notes.2024") are
  DELIBERATELY rejected — without all three components we'd pad from
  today's date, which is hallucination not extraction.

- ``_try_filename_date`` and ``_try_content_body_date`` now run the
  gate BEFORE invoking dateutil, and pass ``fuzzy=True`` is REMOVED.
  Dateutil only runs in strict mode on a substring the gate already
  validated.

### Issue 1 — wire the feature into production

``mempalace/miner.py`` batched-upsert path:

- Accumulate ``batch_metas`` across all batches into ``all_metas``
- Pass ``drawer_metas=all_metas`` to ``build_closet_lines``

End-to-end integration test added that mines a real file with a
filename-derived content date and asserts the produced closet
documents contain the 4-segment pointer with that date.

``diary_ingest.py`` is left as-is for this PR. Diary entries are
entry-keyed, not chunk-keyed — they carry no natural
``line_start`` / ``line_end``, so the 4-segment form would return
None for them regardless. Wiring the diary path can land cleanly in
a follow-up once Tier 6a gains an "approximate line range for diary
entries" story.

### Issue 3 — declare the dateutil dependency

``pyproject.toml``: add ``python-dateutil>=2.8`` to
``[project].dependencies``. One-line change; cheaper than the
stdlib-only refactor alternative and keeps the natural-language
recall surface.

### Issue 4 — pin the two-digit-year boundary

Four new tests cover the 1969/1970/1999/2000 corner cases of the
slash-date locale heuristic.

## Tests added (RED-first then GREEN)

  tests/test_miner.py::TestExtractContentDate (11 new):
    Hallucination cases verbatim from Igor's review:
    - test_no_hallucination_junk_filename_with_trailing_digit
    - test_no_hallucination_untitled_with_index
    - test_no_hallucination_filename_year_only
    - test_no_hallucination_filename_year_and_month_only
    - test_no_hallucination_content_with_issue_number
    - test_no_hallucination_content_with_count
    - test_no_hallucination_content_with_version_number
    Two-digit-year boundary cases:
    - test_two_digit_year_69_is_2069
    - test_two_digit_year_70_is_1970
    - test_two_digit_year_99_is_1999
    - test_two_digit_year_00_is_2000

  tests/test_closets.py::TestMinerClosetRebuild (1 new):
    - test_production_miner_emits_4_segment_pointers_with_content_date
      (regression for Issue #1 — real ``mine()`` end-to-end produces
      4-segment closet pointers via the new ``drawer_metas`` wiring)

## Verification

  pytest tests/test_miner.py tests/test_closets.py
         tests/test_format_miner.py tests/test_palace.py
    → 242 passed, 2 skipped, 0 regressions

  pytest tests/test_miner.py::TestExtractContentDate
    → 26 passed (15 prior + 11 new)

  pytest tests/test_closets.py::TestMinerClosetRebuild
    → end-to-end wiring test GREEN

  Sanity (Igor's exact repros):
    "tmp_random_file_5"           → None (was: 2026-05-05)
    "untitled-1"                  → None (was: 2026-05-01)
    "notes.2024.md"               → None (was: 2024-05-22)
    "2024-06.md"                  → None (was: 2024-06-22)
    "Bug fix for issue 42 in module 7" → None (was: 2042-07-22)
    "Tested with 1000 drawers"    → None (was: 1000-05-22)
    "Version 3.3.6 released"      → None (was: 2006-03-03)

  Real dates still extract correctly:
    "2024-11-08.md"               → "2024-11-08"
    "April-6th-2011-notes.md"     → "2011-04-06"
    "Nov-8-2024.md"               → "2024-11-08"

  ruff check + ruff format --check (pinned 0.15.9)
    → All checks passed

  OrbStack triple-Linux verify (Py 3.9 / 3.11 / 3.13)
    → all targeted tests pass; python-dateutil installs explicitly
       via the new declared dependency.
2026-05-22 10:45:43 -07:00
Igor Lins e Silva c89c077d7e Merge pull request #1578 from MemPalace/feat/hallway-tunnel-decay-dynamics
feat(dynamics): Hebbian potentiation + Ebbinghaus decay for halls + tunnels
2026-05-22 13:15:39 -03:00
Igor Lins e Silva 5633406e5e Merge pull request #1585 from milla-jovovich/fix/hooks-cli-shallow-path-guard
fix(hooks_cli): guard parents[3] access against shallow filesystem paths
2026-05-22 11:18:08 -03:00
Igor Lins e Silva 9f9640fa40 Merge pull request #1237 from OmkarKirpan/fix/readme-pipx-install
docs(readme): recommend pipx for install (fixes PEP 668 + global dep conflicts)
2026-05-22 10:33:57 -03:00
Igor Lins e Silva 3e4e1eee28 Merge pull request #1436 from dergachoff/docs/codex-discoverability
docs: make Codex MCP setup discoverable
2026-05-22 10:33:18 -03:00
Igor Lins e Silva ddf489b0bf Merge pull request #1395 from Zk-nd3r/docs/claude-code-retention-setup
docs: add Claude Code retention setup checklist
2026-05-22 10:15:10 -03:00
Igor Lins e Silva 149c1976b3 Merge pull request #1257 from oussamalembarki/fix/security-warning-visibility
docs: improve visibility of phishing/malware warnings
2026-05-22 09:59:13 -03:00
Igor Lins e SilvaandClaude Opus 4.7 28b2f48547 Merge branch 'develop' into docs/claude-code-retention-setup
Bring docs PR up to date with develop so CI re-runs against the
current pin set.

Co-Authored-By: Igor Lins e Silva <4753812+igorls@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-22 09:54:29 -03:00
Igor Lins e SilvaandClaude Opus 4.7 4b7edd9aea Merge branch 'develop' into docs/codex-discoverability
Bring docs PR up to date with develop so CI re-runs against the
current pin set.

Co-Authored-By: Igor Lins e Silva <4753812+igorls@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-22 09:54:27 -03:00
Igor Lins e SilvaandClaude Opus 4.7 00ecd8dcb2 Merge branch 'develop' into fix/readme-pipx-install
Bring docs PR up to date with develop so CI re-runs against the
current pin set.

Co-Authored-By: Igor Lins e Silva <4753812+igorls@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-22 09:54:25 -03:00
Igor Lins e SilvaandClaude Opus 4.7 4d79918724 docs: fold dangling 'Details and timeline' link back into the scam-alert callout
The original sentence ('may distribute malware. Details and timeline:
docs/HISTORY.md') was split mid-sentence by the visibility reformat,
leaving a fragment 'malware. Details and timeline: ...' as an orphaned
blockquote outside the [!CAUTION] callout. Fold the link into the
malware line so the callout stays self-contained.

Co-Authored-By: Igor Lins e Silva <4753812+igorls@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-22 09:52:59 -03:00
Igor Lins e Silva bd3a867f22 Merge pull request #1142 from jphein/docs-releasing
docs: add RELEASING.md with mempalace-mcp pre-release check
2026-05-22 09:39:50 -03:00
Igor Lins e Silva f7f66cb621 Merge pull request #1494 from jphein/docs-recovery-runbook
docs(recovery): runbook for chromadb dimensionality=None metadata corruption
2026-05-22 09:39:46 -03:00
Milla J 749d43359a style(hooks_cli): address PR #1580 gemini review — LBYL + side_effect
Two medium-priority gemini-code-assist comments on PR #1580 both
recommend more-idiomatic Python:

1. **Production code (``mempalace/hooks_cli.py::_mempalace_python``)** —
   replace ``try/except IndexError`` with ``if len(parents) > N:``
   look-before-you-leap checks. Exception handling for bounded-integer
   index lookups is a code smell in Python; LBYL makes the depth check
   explicit and removes exception overhead. Same behavior, clearer
   intent.

   Before (EAFP, ~12 lines + comment):
       try:
           venv_bin = resolved.parents[3] / "bin" / "python"
           if venv_bin.is_file():
               return str(venv_bin)
       except IndexError:
           pass

   After (LBYL, ~5 lines):
       if len(parents) > 3:
           venv_bin = parents[3] / "bin" / "python"
           if venv_bin.is_file():
               return str(venv_bin)

2. **Test code (``tests/test_hooks_cli.py``)** — replace the lambda +
   generator-throw hack with ``MagicMock.side_effect = get_item``,
   where ``get_item`` is a normal function that returns the
   editable-install path for index 1 and raises ``IndexError`` for
   any other index (defensive against a future regression that drops
   the LBYL length check). Standard ``side_effect`` mocking pattern.

   Before:
       fake_parents.__getitem__ = lambda self, idx: (
           RealPath("/work/mempalace")
           if idx == 1
           else (_ for _ in ()).throw(IndexError(idx))
       )

   After:
       def get_item(idx):
           if idx == 1:
               return RealPath("/work/mempalace")
           raise IndexError(idx)

       fake_parents.__len__.return_value = 3
       fake_parents.__getitem__.side_effect = get_item

   Also added ``__len__`` mock so the LBYL length check in production
   sees the simulated shallow path correctly.

## Verification

  pytest tests/test_hooks_cli.py
    → 110 passed, 1 skipped (same as PR #1580 baseline; regression
       test for shallow-path crash still GREEN)

  ruff check + ruff format --check (pinned 0.15.9)
    → All checks passed; 2 files already formatted
2026-05-22 05:12:01 -07:00
Milla J 5482262749 fix(hooks_cli): guard parents[3] access against shallow filesystem paths
``_mempalace_python()`` in ``mempalace/hooks_cli.py`` uses
``Path(__file__).resolve().parents[3]`` to locate the venv Python
interpreter in the standard install layout
``<venv>/lib/pythonX.Y/site-packages/mempalace/hooks_cli.py``. When the
package lives at a shallow filesystem path — Docker containers
mounting at ``/work``, ``/opt/app``, minimal-prefix production
installs — ``parents`` has fewer than 4 elements and the index raises
``IndexError`` instead of falling through to the editable-install
branch.

The crash was caught by OrbStack-based triple-Python CI verification
on PR #1579: 16 tests in ``test_hooks_cli.py`` failed identically on
Linux 3.9 / 3.11 / 3.13 with the same ``IndexError: 3`` from
``pathlib._PathBase.parents.__getitem__`` — and verified pre-existing
on develop tip in the same container. The bug never surfaces in
GitHub Actions CI runners (their workdir at
``/home/runner/work/mempalace/mempalace`` has plenty of parent
directories) but it surfaces immediately for anyone:

  - running mempalace in editable mode inside a Docker dev container
  - shipping mempalace as part of an OCI image where the install
    prefix is ``/app`` or ``/opt/<name>``
  - using OrbStack / Colima / podman-machine for cross-version
    verification

## The fix

Wrap each ``parents[N]`` access in ``try/except IndexError`` so the
helper falls through to the next strategy (editable-install →
``sys.executable``) instead of crashing the hook. Both ``parents[3]``
AND ``parents[1]`` are guarded — the latter is defensive against
extreme cases like a file at root (``/file.py``, parents=[/]) — same
class of bug.

## Test added (RED-first, then GREEN)

  tests/test_hooks_cli.py::test_mempalace_python_handles_shallow_path_without_crashing

Mocks ``Path(__file__).resolve()`` so ``parents[3]`` raises
``IndexError`` and ``parents[1]`` returns a real shallow path
(``/work/mempalace``). Pre-commit: function raises ``IndexError: 3``.
Post-commit: function returns a valid Python interpreter path
(either editable-venv if present, otherwise ``sys.executable``).

## Verification

  pytest tests/test_hooks_cli.py
    → 110 passed, 1 skipped on macOS (the existing run)
    → 110 passed, 1 skipped on Linux 3.9 / 3.11 / 3.13 (OrbStack)
       — was 16 failed, 94 passed before this commit

  ruff check + ruff format --check (pinned 0.15.9)
    → All checks passed; 2 files already formatted
2026-05-22 05:00:04 -07:00
Milla J c3c79e18d6 perf(closets): address PR #1579 gemini review — avoid O(N) slicing in hot paths
Three medium-priority gemini-code-assist comments on PR #1579 all flagged
the same anti-pattern: slicing or splitting the entire ``content`` string
to inspect a small prefix. On a 500MB file with 50K chunks, the original
``content[:offset]`` and ``content.split("\n")`` calls produce O(N²) work
and severe memory churn. None of this was wrong semantically — it was
wrong about *cost*.

This commit replaces all three sites with bounded equivalents that read
the original string in place:

## Changes

1. **``mempalace/miner.py::chunk_text``** — replace
   ``content[:start].count("\n")`` with ``content.count("\n", 0, start)``.
   The ``str.count`` method's bounds form counts on the original string
   without allocating a slice. Same result, O(start) instead of
   O(N) memory per chunk.

2. **``mempalace/miner.py::_try_frontmatter_date``** — replace the
   ``stripped.split("\n")`` + line-by-line scan with ``stripped.find("\n---", 3)``
   to locate the closing delimiter, then slice the frontmatter region
   directly. Frontmatter is always near the start of the file, so the
   find walks at most a few KB even on huge files. The earlier
   implementation produced a list of every line in the file.

3. **``mempalace/miner.py::_try_content_body_date``** — two-part fix:
   (a) skip frontmatter via ``find("\n---", 3)`` (same pattern as #2);
   (b) replace ``stripped.split("\n")[:10]`` with
   ``stripped.split("\n", 10)[:10]`` — the ``maxsplit`` form caps work
   to scanning 10 newlines rather than splitting the entire file.

## Why no behavior change

All three transformations are equivalence-preserving:

- ``s[:n].count(ch) == s.count(ch, 0, n)`` — by definition.
- ``s.split("\n")[i].strip() == "---"`` for some i ≤ N is equivalent to
  ``s.find("\n---") != -1`` (both detect "a line consisting of exactly
  ---") for our use case where the opening fence is already validated by
  ``stripped.startswith("---")``.
- ``s.split("\n")[:10]`` produces the same first-10-elements as
  ``s.split("\n", 10)[:10]``; the difference is the latter stops after
  10 newlines instead of scanning the rest.

## Verification

  pytest tests/test_miner.py::TestChunkTextLineRanges
         tests/test_miner.py::TestBuildDrawerMetadataLineRange
         tests/test_miner.py::TestExtractContentDate
         tests/test_closets.py::TestBuildClosetLines
    → 32 passed (same as PR #1579's baseline; no test changes needed
       because behavior is identical, only memory/time profile changed)

  ruff check + ruff format --check (pinned 0.15.9)
    → All checks passed; 1 file already formatted
2026-05-22 04:35:49 -07:00
Milla J 67c6ab59f2 style: apply ruff format with pinned 0.15.9 (CI lint) 2026-05-22 04:24:07 -07:00
Milla J 05da803f71 feat(closets): Tier 6a — date+line locators with content-date hierarchy
Closet pointer lines gain a 4th pipe-separated segment of shape
``YYYY-MM-DD:Lstart-Lend`` so retrieval can jump straight to the right
span in the source file instead of opening the whole drawer. The date
is selected via a content-aware hierarchy so legacy ingests (where
``filed_at`` is misleading) still produce honest pointers when the
content itself carries a date.

The legacy 3-segment format remains the fallback for drawers without
the new metadata, so existing palaces keep working without a re-mine.

## Pointer shape

Before:
  topic|entities|→drawer_ids

After (when drawer metadata carries line range + a date):
  topic|entities|2024-11-08:L42-L78|→drawer_ids

Backward compat: when ``drawer_metas`` is not passed, or the first
meta lacks ``line_start`` / ``line_end``, ``build_closet_lines`` emits
the legacy 3-segment form unchanged.

## Date-source hierarchy (NEW)

A new helper ``mempalace.miner._extract_content_date(source_file, content)``
selects a date once per file and stores it as ``content_date`` on every
drawer mined from that file. Priority order, first match wins:

  1. **Filename** — ISO regex on stem, then dateutil fuzzy parse for
     natural-language formats. Handles ``2024-11-08.md``,
     ``April-6th-2011-notes.md``, ``Nov-8-2024.md``, etc.

  2. **YAML frontmatter** — looks at ``date`` / ``created`` /
     ``published`` fields. yaml.safe_load already parses ISO dates as
     ``datetime.date`` objects; dateutil handles string values.

  3. **Content body, first ~10 lines** —
       a. ISO regex (``YYYY-MM-DD`` / ``YYYY/MM/DD`` / ``YYYY.MM.DD``)
       b. Slash dates with locale auto-disambiguation: if any
          first-number > 12 appears anywhere in the head, lock the
          file's locale to DD/MM (the only consistent reading);
          otherwise default to US MM/DD for two-digit-year compactness.
       c. dateutil fuzzy parse for natural-language
          (``November 8, 2024``, ``8th November 2024``, etc.)

  4. **Filesystem mtime** — ``os.path.getmtime`` formatted as
     ``YYYY-MM-DD``.

  5. **None** — caller (``_build_drawer_metadata``) leaves
     ``content_date`` absent; ``build_closet_lines`` falls back to the
     ``filed_at`` ingestion timestamp's date portion.

``_build_date_line_segment`` in ``palace.py`` now prefers ``content_date``
when present, otherwise uses ``filed_at[:10]``. This is the load-bearing
change that makes Tier 6a meaningful for legacy content — a Nov 8 2024
transcript mined today now emits ``2024-11-08:L42-L78`` instead of
``2026-05-22:L42-L78``.

## Changes

1. ``mempalace/miner.py::chunk_text`` — each chunk dict now also carries
   ``line_start`` / ``line_end`` (1-indexed line numbers in the stripped
   source). Computed via ``content[:offset].count("\n") + 1`` —
   approximate locator accurate to ±1 line at chunk boundaries.

2. ``mempalace/miner.py::_extract_content_date`` (NEW) + helpers
   (``_try_filename_date``, ``_try_frontmatter_date``,
   ``_try_content_body_date``, ``_try_mtime_date``). Pure functions, no
   I/O beyond what's named. Uses lazy imports of ``dateutil`` and
   ``yaml`` to keep top-level import surface unchanged.

3. ``mempalace/miner.py::_build_drawer_metadata`` — gains
   ``line_start`` / ``line_end`` / ``content_date`` optional kwargs.
   When provided, stored in drawer metadata; when omitted, keys are
   absent from the dict — backward compatible.

4. ``mempalace/miner.py`` batched-upsert path — calls
   ``_extract_content_date(source_file, content)`` ONCE per file (not
   once per chunk) and passes the result through to every chunk's
   ``_build_drawer_metadata`` call.

5. ``mempalace/format_miner.py::_file_chunks_locked`` — gains a
   ``content`` param so the format-miner path can also run
   ``_extract_content_date`` on the full extracted text. Plumbs
   ``content_date`` into every batched drawer's metadata. Caller passes
   ``text`` (the full extracted markdown).

6. ``mempalace/palace.py::build_closet_lines`` — accepts optional
   ``drawer_metas`` (parallel to ``drawer_ids``). When present and the
   first meta has both ``line_start`` and ``line_end``, the new
   segment is spliced in; otherwise legacy 3-segment form. Helper
   ``_build_date_line_segment`` prefers ``content_date`` over
   ``filed_at`` for the date portion.

## Tests added (RED-first, then GREEN)

  tests/test_miner.py::TestChunkTextLineRanges          (4 tests)
  tests/test_miner.py::TestBuildDrawerMetadataLineRange (2 tests)
  tests/test_miner.py::TestExtractContentDate           (15 tests)
    - filename ISO / natural-language / compact
    - YAML frontmatter date / created
    - Claude session preamble
    - ISO + natural-language content body
    - Slash-date locale disambiguation (DD/MM lock vs default MM/DD)
    - Priority order (filename > frontmatter > content > mtime)
    - mtime fallback
    - Graceful None returns for missing file / empty content

  tests/test_closets.py::TestBuildClosetLines           (5 new tests)
    - includes_date_line_segment_when_metas_provided
    - falls_back_to_3_segment_format_when_metas_missing  (compat)
    - falls_back_to_3_segment_format_when_metas_lack_line_keys (compat)
    - date_segment_uses_filed_at_date_portion_only
    - content_date_preferred_over_filed_at               (new — Tier 6a hierarchy)

26 new tests total. All RED before this commit, all GREEN after.

## Verification

  pytest tests/test_miner.py::TestChunkTextLineRanges
         tests/test_miner.py::TestBuildDrawerMetadataLineRange
         tests/test_miner.py::TestExtractContentDate
         tests/test_closets.py::TestBuildClosetLines
    → 32 passed

  pytest tests/test_miner.py tests/test_closets.py
         tests/test_format_miner.py tests/test_palace.py
    → 230 passed, 2 skipped, 0 regressions in directly-affected files

  ruff check + ruff format --check
    → All checks passed, 5 files already formatted

  End-to-end smoke test on a YAML-frontmatter file (date: 2024-11-08,
  mined "today"):
    content_date extracted: 2024-11-08
    chunks produced: 1
    drawer meta: content_date=2024-11-08, filed_at='2026-05-22...', lines=1-15
    closet pointer:
      "conversation with lumi about brands of dog food|Filler;Lumi;Aya|2024-11-08:L1-L15|→drawer_a,drawer_b"
    → content_date correctly preferred over filed_at end-to-end.

  OrbStack triple-Linux verify (Py 3.9 / 3.11 / 3.13)
    → 32 targeted Tier 6a + content-date tests pass on each.

## Out of scope (deliberate)

- No schema migration of existing drawers. Pre-Tier-6a drawers lack
  ``line_start`` / ``content_date`` and silently fall back to the
  legacy 3-segment pointer shape.
- No re-mine required. Users keep existing palaces; new files mined
  after this lands carry the new fields.
- No drawer ID change (Task #80, separate concern).
- No exact-quote line positioning within a chunk (future tier).
- No NLP-based date extraction (only regex + dateutil fuzzy).
- Linguistic-similarity / concept-recognition retrieval (L5 zettal
  layer) is the natural next step but lives in its own PR family.
2026-05-22 04:02:03 -07:00
dependabot[bot] 1ff7aa1eac build(deps-dev): bump ruff from 0.15.9 to 0.15.14
Bumps [ruff](https://github.com/astral-sh/ruff) from 0.15.9 to 0.15.14.
- [Release notes](https://github.com/astral-sh/ruff/releases)
- [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md)
- [Commits](https://github.com/astral-sh/ruff/compare/0.15.9...0.15.14)

---
updated-dependencies:
- dependency-name: ruff
  dependency-version: 0.15.14
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-22 07:57:48 +00:00
Milla J f1aa409a94 fix(dynamics): address PR #1578 gemini review — canonicalize lookup key + idiomatic dict update
Addresses both inline comments left by gemini-code-assist on PR #1578.

## What this commit does

1. **HIGH PRIORITY (gemini, hallways.py:260) — canonicalize dynamics
   preservation lookup key.** The previous code built the lookup key as
   ``(h.get("entity_a"), h.get("entity_b"))``, but the
   ``pair_counts`` dict and ``_hallway_id`` both use sorted entity pairs
   for symmetric IDs. A persisted record with reversed entity order
   (Lumi as entity_a, Aya as entity_b) would silently miss the lookup
   and lose its accumulated dynamics on recompute. Now the lookup
   canonicalizes the same way ID generation does:

       key = tuple(sorted([h.get("entity_a"), h.get("entity_b")]))

   Failure mode is rare in practice (the algorithm normally writes
   sorted records), but any legacy data, manual JSON edit, or future
   import path that violates the ordering would have silently broken
   L7 preservation. Belt-and-suspenders fix.

2. **MEDIUM PRIORITY (gemini, palace_graph.py:512) — idiomatic field
   preservation in tunnel dedup path.** Refactored the manual for-loop
   into a dict-update with a comprehension. Field list lives in one
   place; future schema expansion can't accidentally drop a field. No
   behavior change.

## Test added (RED-first)

  test_recompute_preserves_dynamics_when_existing_record_has_reversed_entity_order

Pre-populates the hallway store with a record whose ``entity_a`` /
``entity_b`` are in REVERSED (non-sorted) order, bumps its strength /
stability / access_count to non-default values, then runs
``compute_hallways_for_wing``. Asserts the recomputation preserves the
bumped values — proving the canonical lookup matches against
arbitrarily-ordered persisted entity pairs.

Reproduces the bug class gemini flagged: RED before this commit
(strength=1.0 instead of 4.2), GREEN after.

## Verification

  pytest tests/test_hallways.py::TestHallwayDynamicsIntegration
        tests/test_palace_graph_tunnels.py::TestTunnelDynamicsIntegration
        tests/test_dynamics.py
    → 32 passed (was 31; added 1 new for the reverse-order case)
  pytest -q (full mempalace suite)
    → 2112 passed, 3 skipped, 0 regressions
  ruff check mempalace/dynamics.py mempalace/hallways.py mempalace/palace_graph.py tests/
    → All checks passed!
  ruff format --check ...
    → 3 files already formatted (pinned 0.15.9)
2026-05-21 21:32:19 -07:00
Igor Lins e Silva ae8074e41a Merge pull request #1469 from yonefive71/fix/tunnel-file-config-and-endpoint-validation
fix(graph): tunnel file follows palace_path; validate explicit-tunnel endpoints exist
2026-05-21 22:38:07 -03:00
Igor Lins e SilvaandClaude Opus 4.7 3b6c7986c9 Merge branch 'develop' into fix/tunnel-file-config-and-endpoint-validation
Resolves conflicts from 60-commit divergence:

- tests/test_closets.py: assertion reformat — kept develop's ruff-format-
  preferred multi-line shape (functionally identical).
- tests/test_palace_graph_tunnels.py: both branches added a new test
  class at end-of-file (this PR's TestTunnelFileFollowsConfig + develop's
  TestEntityTunnels from #1564). Kept both, no overlap.

Local: ruff check / format pass; full pytest suite passes
(2113 passed, 3 skipped).

Co-Authored-By: Igor Lins e Silva <4753812+igorls@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 22:31:41 -03:00
Igor Lins e Silva 257712e818 Merge pull request #1477 from funguf/fix/convos-mine-concurrency
fix(convo_miner): wrap mine_convos in mine_palace_lock
2026-05-21 22:15:12 -03:00