mempalace

xiaoxue_admin/mempalace

Fork 0

Commit Graph

Author	SHA1	Message	Date
Milla J	206cfbb6ca	fix(ids): delimit hash inputs to prevent drawer_id collisions (#80 ) Drawer/closet/triple IDs built by concatenating strings without a delimiter before hashing can collide: hash((s1 + str(i1))) == hash((s2 + str(i2))) whenever s1+str(i1) == s2+str(i2). Concretely, source_file="foo" + chunk_index=12 produces the same hash input as source_file="foo1" + chunk_index=2 — both yield "foo12". Under ChromaDB's primary-key constraint the second write silently overwrites the first; one chunk's content is lost without surfacing an error (the surrounding broad `except Exception:` blocks swallow any backend complaint). The defect class is "string + string concat fed to a hash without a delimiter." Per the styleguide's partial-scope-key-migration rule, every site sharing the pattern is a candidate and must be triaged — not just the ones the original issue named. FIX — 6 sites - mempalace/miner.py:1253 drawer_id, add_drawer() back-compat - mempalace/miner.py:1386 drawer_id, batched mine loop - mempalace/miner.py:1416 drawer_ids, closet-build re-derivation - mempalace/format_miner.py:643 drawer_id, format_miner batched loop - mempalace/mcp_server.py:1136 drawer_id, tool_add_drawer - mempalace/knowledge_graph.py:305 triple_id, KG triple insertion MIGRATED FOR CONSISTENCY — 2 sites - mempalace/convo_miner.py:87 sentinel_key — was `:`, now `\|` - mempalace/convo_miner.py:422 drawer_key — was `:`, now `\|` Pre-existing delimiter was `:`. Migrated because (a) source_file can contain literal `:` on Windows paths (`C:\Users\…`) and URL-like sources (`https://host:8080`), weakening `:` as a separator; (b) `\|` is reserved in Windows filenames so source_file cannot contain it, making it strictly safer. DELIMITER CHOICE - `\|` chosen over `:` (Windows / URL source_file edge cases) - `\|` chosen over `\x00` (NUL breaks log greppability + print() debug + downstream string handling for marginal extra safety) - Matches the established precedent in mempalace/diary_ingest.py (lines 52, 76, 91, 98 — all already on `\|`) EXEMPT — audited and correct as-is Single-input hashes (nothing to delimit): - mempalace/miner.py:1432 closet_id (source_file only) - mempalace/format_miner.py:559 sentinel_id (source_file only) - mempalace/palace.py:433 lock filename (source_file only) - mempalace/palace.py:629 palace_key (lock_key_source only) - mempalace/diary_ingest.py:158 content_hash (text only) - mempalace/hooks_cli.py:329 pidfile digest (joined cmd only) - mempalace/sources/context.py:141 record digest (source_file only) Already correctly delimited: - mempalace/hallways.py:157 `f"{wing}::{a}::{b}"` (`::`) - mempalace/palace_graph.py:454 `f"{a}↔{b}"` (`↔`) - mempalace/diary_ingest.py:52,76,91,98 (`\|` precedent) Protected by composition (uniqueness guaranteed by the ID prefix, not by the hash slice): - mempalace/mcp_server.py:1635 entry_id is `diary_{wing}_{strftime('%Y%m%d_%H%M%S%f')}_{sha[:12]}`. Microsecond-resolution timestamp prefix supplies uniqueness; the trailing hash is a content-discriminator, not the write-time uniqueness guarantor. NEW MODULES - mempalace/ids.py — single source of truth for ID construction. Five helpers (make_drawer_id_from_chunk, make_drawer_id_from_content, make_convo_drawer_id, make_convo_sentinel_id, make_triple_id) plus an ID_RECIPE = "v2" constant. - mempalace/collision_scan.py — pre-mining defense. Runs immediately before each batched ChromaDB upsert; raises CollisionError naming the colliding (source_file, chunk_index) pairs if any proposed drawer_id appears more than once with conflicting metadata across the union of incoming and existing rows. DETECTION ChromaDB's primary-key constraint means past collisions were silently resolved at upsert time (last-write-wins), erasing evidence. A post-hoc audit cannot reconstruct what was lost from palace state alone. Therefore: - Pre-mining risk scan. Before each batched upsert, compute the proposed drawer_ids for the incoming chunk set AND query existing drawer_ids from the collection. If any proposed id appears more than once in the union (incoming-vs-incoming or incoming-vs- existing) with conflicting (source_file, chunk_index), abort the mine with an actionable error naming the colliding pairs. Collision is caught BEFORE it destroys data, which is the only point at which palace state still carries the evidence. - New metadata key: `"id_recipe": "v2"` on every drawer written under the delimited recipe. Audits compare like-for-like; drawers without `id_recipe` are treated as v1 legacy (undelimited or `:`-delimited), not as collisions. - Honest disclosure: palaces mined under any pre-v2 mempalace may carry silent past collisions whose original content is unrecoverable from palace state. Future library tier work will give users a per-drawer audit + opt-in archival path. TESTS - tests/test_ids.py: 17 unit tests covering all 5 helpers, the ID_RECIPE constant, the private `_delimited_sha256` helper, and the four defect-class collision shapes (chunk_index boundary, content boundary, extract_mode boundary, ISO datetime boundary). RED collision tests confirm the v2 recipe breaks the defect class. - tests/test_collision_scan.py: 10 tests covering clean batches, idempotent re-mines, incoming-vs-incoming collisions, incoming-vs- existing collisions, error-message quality, empty batches, metadata without chunk_index, and ChromaDB backend errors propagating cleanly. - tests/test_convo_miner_unit.py: FakeCol gains a stub `get()` so the pre-mining scan can probe an empty in-test collection. BACKWARDS COMPATIBILITY - Existing v1 drawers remain queryable; no rewrite of stored IDs. - New mining passes write v2 drawers alongside v1 via PR #1628's additive-mining model. - No user action required; opt-in cleanup ships separately. VERIFICATION - macOS Python 3.12 (local): 2304 passed, 3 skipped, 0 failed, coverage 85.44% (above 80% threshold). - Linux Python 3.9/3.11/3.13 (OrbStack with full `pip install -e '.[dev]'` flow): see PR body for per-version pass counts. - `ruff check .` + `ruff format --check .`: clean. - pylint score: ids.py 10.00/10, collision_scan.py 10.00/10; all modified files >= 8.92 (no regression). - bandit: clean on all new files; the one MEDIUM finding in knowledge_graph.py is on lines 385/407 (pre-existing SQL string construction in timeline queries), not in code touched by this PR. - pre_push_check.sh: ALL CHECKS PASSED. Refs: deferred from PR #1572 (Copilot finding #13). Styleguide audit documented above: 8 fix/migration + 11 enumerated exempt sites.	2026-05-30 12:44:47 -07:00

Author

SHA1

Message

Date

Milla J

206cfbb6ca

fix(ids): delimit hash inputs to prevent drawer_id collisions (#80 )

Drawer/closet/triple IDs built by concatenating strings without a
delimiter before hashing can collide: hash((s1 + str(i1))) ==
hash((s2 + str(i2))) whenever s1+str(i1) == s2+str(i2). Concretely,
source_file="foo" + chunk_index=12 produces the same hash input as
source_file="foo1" + chunk_index=2 — both yield "foo12". Under
ChromaDB's primary-key constraint the second write silently
overwrites the first; one chunk's content is lost without surfacing
an error (the surrounding broad `except Exception:` blocks swallow
any backend complaint).

The defect class is "string + string concat fed to a hash without
a delimiter." Per the styleguide's partial-scope-key-migration rule,
every site sharing the pattern is a candidate and must be triaged —
not just the ones the original issue named.

FIX — 6 sites
- mempalace/miner.py:1253        drawer_id, add_drawer() back-compat
- mempalace/miner.py:1386        drawer_id, batched mine loop
- mempalace/miner.py:1416        drawer_ids, closet-build re-derivation
- mempalace/format_miner.py:643  drawer_id, format_miner batched loop
- mempalace/mcp_server.py:1136   drawer_id, tool_add_drawer
- mempalace/knowledge_graph.py:305  triple_id, KG triple insertion

MIGRATED FOR CONSISTENCY — 2 sites
- mempalace/convo_miner.py:87    sentinel_key — was `:`, now `|`
- mempalace/convo_miner.py:422   drawer_key   — was `:`, now `|`

Pre-existing delimiter was `:`. Migrated because (a) source_file can
contain literal `:` on Windows paths (`C:\Users\…`) and URL-like
sources (`https://host:8080`), weakening `:` as a separator; (b) `|`
is reserved in Windows filenames so source_file cannot contain it,
making it strictly safer.

DELIMITER CHOICE
- `|` chosen over `:` (Windows / URL source_file edge cases)
- `|` chosen over `\x00` (NUL breaks log greppability + print()
  debug + downstream string handling for marginal extra safety)
- Matches the established precedent in mempalace/diary_ingest.py
  (lines 52, 76, 91, 98 — all already on `|`)

EXEMPT — audited and correct as-is
  Single-input hashes (nothing to delimit):
    - mempalace/miner.py:1432         closet_id (source_file only)
    - mempalace/format_miner.py:559   sentinel_id (source_file only)
    - mempalace/palace.py:433         lock filename (source_file only)
    - mempalace/palace.py:629         palace_key (lock_key_source only)
    - mempalace/diary_ingest.py:158   content_hash (text only)
    - mempalace/hooks_cli.py:329      pidfile digest (joined cmd only)
    - mempalace/sources/context.py:141  record digest (source_file only)

  Already correctly delimited:
    - mempalace/hallways.py:157     `f"{wing}::{a}::{b}"`  (`::`)
    - mempalace/palace_graph.py:454 `f"{a}↔{b}"`           (`↔`)
    - mempalace/diary_ingest.py:52,76,91,98                (`|` precedent)

  Protected by composition (uniqueness guaranteed by the ID prefix,
  not by the hash slice):
    - mempalace/mcp_server.py:1635  entry_id is
      `diary_{wing}_{strftime('%Y%m%d_%H%M%S%f')}_{sha[:12]}`.
      Microsecond-resolution timestamp prefix supplies uniqueness;
      the trailing hash is a content-discriminator, not the
      write-time uniqueness guarantor.

NEW MODULES
- mempalace/ids.py — single source of truth for ID construction.
  Five helpers (make_drawer_id_from_chunk, make_drawer_id_from_content,
  make_convo_drawer_id, make_convo_sentinel_id, make_triple_id) plus
  an ID_RECIPE = "v2" constant.
- mempalace/collision_scan.py — pre-mining defense. Runs immediately
  before each batched ChromaDB upsert; raises CollisionError naming
  the colliding (source_file, chunk_index) pairs if any proposed
  drawer_id appears more than once with conflicting metadata across
  the union of incoming and existing rows.

DETECTION
ChromaDB's primary-key constraint means past collisions were
silently resolved at upsert time (last-write-wins), erasing
evidence. A post-hoc audit cannot reconstruct what was lost from
palace state alone. Therefore:

- Pre-mining risk scan. Before each batched upsert, compute the
  proposed drawer_ids for the incoming chunk set AND query existing
  drawer_ids from the collection. If any proposed id appears more
  than once in the union (incoming-vs-incoming or incoming-vs-
  existing) with conflicting (source_file, chunk_index), abort the
  mine with an actionable error naming the colliding pairs.
  Collision is caught BEFORE it destroys data, which is the only
  point at which palace state still carries the evidence.

- New metadata key: `"id_recipe": "v2"` on every drawer written
  under the delimited recipe. Audits compare like-for-like;
  drawers without `id_recipe` are treated as v1 legacy (undelimited
  or `:`-delimited), not as collisions.

- Honest disclosure: palaces mined under any pre-v2 mempalace may
  carry silent past collisions whose original content is
  unrecoverable from palace state. Future library tier work will
  give users a per-drawer audit + opt-in archival path.

TESTS
- tests/test_ids.py: 17 unit tests covering all 5 helpers, the
  ID_RECIPE constant, the private `_delimited_sha256` helper, and
  the four defect-class collision shapes (chunk_index boundary,
  content boundary, extract_mode boundary, ISO datetime boundary).
  RED collision tests confirm the v2 recipe breaks the defect class.
- tests/test_collision_scan.py: 10 tests covering clean batches,
  idempotent re-mines, incoming-vs-incoming collisions, incoming-vs-
  existing collisions, error-message quality, empty batches,
  metadata without chunk_index, and ChromaDB backend errors
  propagating cleanly.
- tests/test_convo_miner_unit.py: FakeCol gains a stub `get()` so
  the pre-mining scan can probe an empty in-test collection.

BACKWARDS COMPATIBILITY
- Existing v1 drawers remain queryable; no rewrite of stored IDs.
- New mining passes write v2 drawers alongside v1 via PR #1628's
  additive-mining model.
- No user action required; opt-in cleanup ships separately.

VERIFICATION
- macOS Python 3.12 (local): 2304 passed, 3 skipped, 0 failed,
  coverage 85.44% (above 80% threshold).
- Linux Python 3.9/3.11/3.13 (OrbStack with full `pip install -e
  '.[dev]'` flow): see PR body for per-version pass counts.
- `ruff check .` + `ruff format --check .`: clean.
- pylint score: ids.py 10.00/10, collision_scan.py 10.00/10; all
  modified files >= 8.92 (no regression).
- bandit: clean on all new files; the one MEDIUM finding in
  knowledge_graph.py is on lines 385/407 (pre-existing SQL string
  construction in timeline queries), not in code touched by this PR.
- pre_push_check.sh: ALL CHECKS PASSED.

Refs: deferred from PR #1572 (Copilot finding #13). Styleguide audit
documented above: 8 fix/migration + 11 enumerated exempt sites.

2026-05-30 12:44:47 -07:00