Several tests opened sqlite3 connections without try/finally or
context-manager cleanup, relying on a flat conn.close() after the
work. Any assertion failure or exception between connect and close
leaked the connection until GC, producing
``ResourceWarning: unclosed database`` in CI logs.
On Python 3.13 / macOS the ResourceWarning isn't just noise: a
leaked connection can hold a SQLite advisory lock long enough for
later test setup to block on it, which appears to be the cause of
recent intermittent CI hangs on those two runners.
Wrap each affected ``conn = sqlite3.connect(...)`` block in
``contextlib.closing(...)`` so cleanup runs on the failure path too.
Mirrors the try/finally pattern already used in production code
(searcher.py, repair.py, backends/chroma.py).
No behavior change — same operations, same assertions, just
deterministic cleanup. All 162 affected tests pass locally.
Five findings from the automated review, fixed with targeted tests where
behavior changed:
1. Transformation Protocol (transforms.py). The registry mixed a bytes-to-str
transform (utf8_replace_invalid) with str-to-str transforms under a single
Callable[..., str] type, misleading static type checkers and adapter
authors. Introduced a Transformation Protocol with __call__(data: bytes|str)
-> str and retyped the registry + get_transformation return.
2. Drawer-id collision risk (context.py). Switched _build_drawer_id from
sha1[:16]=64 bits to sha256[:24]=96 bits. 64 bits sits uncomfortably
close to the birthday bound for palace-sized corpora; 96 bits keeps the
collision probability negligible while preserving the existing
<prefix>_<chunk> layout adapters rely on.
3. Fresh-schema KG columns (knowledge_graph.py). source_drawer_id and
adapter_name now live in the canonical CREATE TABLE so new palaces don't
take an ALTER round-trip on first open. _migrate_schema stays for legacy
palaces (SQLite has no ADD COLUMN IF NOT EXISTS, so PRAGMA introspection
is still needed there).
4. Identity-shim comment (transforms.py). Comment said the adapter-specific
transforms "raise if invoked without adapter context" but they return
the input unchanged. Updated the comment to match the actual identity-
shim behavior Copilot suggested.
5. Test docstring (test_sources.py). Comment mentioned default_factory=list
but SourceRef.options uses default_factory=dict. Corrected.
Tests: 1020 passed (up from 1018), +2 new tests for the sha256 id shape
and the fresh-schema column presence on new palaces.
Lands the read-side contract so third-party adapter authors (@Perseusxrltd,
@JakobSachs, @adv3nt3, @zendesk-thittesdorf, @mfhens, @roip, @MrDys) have a
stable target matching what RFC 001 §10 landed on the write side in #995.
Scope (this PR):
- mempalace/sources/base.py: BaseSourceAdapter ABC with kwargs-only
ingest() / describe_schema() and default is_current() / source_summary()
/ close() (§1.1–1.2). Typed records: SourceRef, SourceItemMetadata,
DrawerRecord, RouteHint, SourceSummary, AdapterSchema, FieldSpec (§1.3,
§5.2). Error classes: SourceNotFoundError, AuthRequiredError,
AdapterClosedError, TransformationViolationError, SchemaConformanceError
(§2.7). Class-level identity contract: name / adapter_version /
capabilities / supported_modes / declared_transformations /
default_privacy_class (§2.1, §1.4, §1.5, §6).
- mempalace/sources/transforms.py: reference implementations of the 13
reserved transformations (§1.4) — utf8_replace_invalid, newline_normalize,
whitespace_trim, whitespace_collapse_internal, line_trim, line_join_spaces,
blank_line_drop — as pure functions, plus identity shims for the six
adapter-specific ones (strip_tool_chrome, tool_result_truncate,
tool_result_omitted, spellcheck_user, synthesized_marker,
speaker_role_assignment) that the conversations adapter will override
when migrated. get_transformation(name) resolves by reserved name.
- mempalace/sources/registry.py: entry-point discovery via
importlib.metadata.entry_points(group="mempalace.sources") + explicit
register()/unregister() surface (§3.1–3.2). resolve_adapter_for_source()
implements the §3.3 priority order; crucially, no auto-detection on the
read side (§3.3 is explicit about that — user intent never inferred from
on-disk artifacts).
- mempalace/sources/context.py: PalaceContext facade (§9) bundling the
drawer/closet collections, knowledge graph, palace path, adapter identity,
and progress hooks core passes into adapter.ingest(). upsert_drawer()
applies the spec-mandated adapter_name/adapter_version stamps from §5.1.
skip_current_item() signals laziness; emit() dispatches to hooks and
swallows hook exceptions.
- mempalace/knowledge_graph.py: add_triple() gains optional source_drawer_id
and adapter_name kwargs (§5.5). Backwards-compatible column migration
auto-adds the new columns on open of a pre-RFC 002 palace (PRAGMA
table_info then ALTER TABLE ADD COLUMN), matching the pattern used for
any new palace-side provenance fields.
- pyproject.toml: mempalace.sources entry-point group declared. Empty on
the first-party side for now — miners migrate in a follow-up; the group
being present means third-party packages can begin registering today.
Out of scope (explicit follow-ups):
- miner.py → mempalace/sources/filesystem.py. Behavior-preserving rename
that also moves READABLE_EXTENSIONS, detect_room(), detect_hall() into
the adapter (§9). Larger refactor; lands separately.
- convo_miner.py + normalize.py → mempalace/sources/conversations.py. The
format-detection if-chain in normalize.py becomes per-format plugins;
declared_transformations enumerates what the current pipeline already
does to source bytes (§1.4 existing-code mapping).
- Closet post-step wired into the conversations adapter (§1.7).
- CLI --source flag + --mode deprecation alias (§3.3).
- MCP mempalace_mine tool source parameter.
- AbstractSourceAdapterContractSuite (§7.1–7.3): byte-preservation round-
trip and declared-transformation round-trip tests.
- Privacy-class floor enforcement (§6.2); depends on #389 for
secrets_possible scanning.
Tests: 1018 passed (up from ~990 on develop), +27 targeted tests covering
the ABC instantiation rules, typed records, all reserved transformations,
the registry register/get/unregister surface, PalaceContext upsert + skip +
emit semantics, and both the new KG provenance kwargs and backwards-
compatible legacy-schema migration.
Refs: #989 (RFC 002 tracking), #990 (RFC 002 spec), #995 (RFC 001 §10
cleanup — sibling PR on the write side).
Igor Lins e Silva