On Windows, Path.read_text() and open(path, 'a') use locale encoding
(GBK on Chinese-locale systems) before PEP 686 / Python 3.15. A valid
UTF-8 .gitignore with non-ASCII comments crashes
_ensure_mempalace_files_gitignored() with UnicodeDecodeError, which
aborts 'mempalace init' on Windows for any user whose .gitignore
contains non-ASCII text.
Force encoding='utf-8' on both read and append, with errors='replace'
on read as a defensive fallback for legacy mixed-encoding files.
Co-authored-by: ALaDingAhmad <16530935@qq.com>
Co-authored-by: Igor Lins e Silva <4753812+igorls@users.noreply.github.com>
L1's generate() scored drawers by importance/emotional_weight/weight, and
the docstring promised "prefer high importance, recent filing". But no
ingest path (miner, convo_miner, diary, add_drawer) writes any of those
fields, so the sort collapsed to insertion order (oldest first) and
recency was never consulted. A scoped `wake-up --wing X` therefore
surfaced the *oldest* moments: the opposite of useful.
Add filed_at (present on every drawer, ISO-8601, lexically chronological)
as the secondary sort key. Importance stays primary for the day a scoring
pass populates it; filed_at is the effective signal today, making the
"recent filing" half of the promise true with data already present.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Igor Lins e Silva <4753812+igorls@users.noreply.github.com>
* fix(mcp): stop clobbering host app root logger at import (#1860)
_init_logging() ran at import and called logging.basicConfig(force=True),
resetting the root logger's level, format, and handlers unconditionally. An
app that configured logging before importing mempalace.mcp_server lost its
setup: a host on DEBUG dropped to INFO, custom formatters and handlers were
replaced.
force=True existed (#1495) only to keep MEMPALACE_LOG_FILE working when root
already had handlers. This keeps that contract without the reset: configure
root only when it is unconfigured (standalone); otherwise attach a
mempalace-filtered file handler additively and leave the host's config alone.
Adds _MempalaceLogFilter so the file captures every mempalace logger (the
dotted mempalace.* family plus the flat mempalace_* names) and nothing else.
* fix(mcp): survive importlib.reload and pin file log format (#1860)
Addresses review on #1885.
Restore _logging_configured from globals() so the idempotency guard survives
importlib.reload: a reload re-executes the module body, and a plain reset would
let _init_logging() stack a duplicate file handler on root.
Set an explicit "%(message)s" formatter on the file handler so the embedded
path does not depend on logging's default formatter (which already renders the
same, but is now pinned and identical to the standalone path).
Adds a reload regression test and a format-pin assertion.
Concurrent killed-mid-write mines can leave embedding_fulltext_search in a
malformed-inverted-index state that fails PRAGMA quick_check while the
underlying rows stay intact (integrity_check ok). The repair preflight then
hard-aborts before reaching the FTS5 rebuild step, so `mempalace repair`
refuses to run and full-text search stays broken — the exact loop #1596
reports. The MineValidationError banner even promises "repair --yes rebuilds
the FTS5 virtual table automatically," which the preflight abort made false.
Add maybe_autoheal_fts5_index(): when every quick_check error is an isolated
"malformed inverted index for FTS5 table" failure, rebuild the index in place
from the intact embedding_fulltext_search_content table
(INSERT ... VALUES('rebuild')) under mine_palace_lock, then re-run quick_check.
The rebuild touches no drawer rows. Wired into both repair preflights
(rebuild_index and cli cmd_repair). Any non-FTS5 error in the set, a lock held
by a live mine, or a rebuild that does not clear quick_check leaves the errors
unchanged so the caller still aborts with the recovery banner — broader
corruption is never silently rebuilt over.
An empty link_lists.bin is not corruption on its own: hnswlib stores the
layer-0 graph inside data_level0.bin and only writes link_lists.bin for
elements promoted to level > 0. A small/low-fanout index where every
element stays on layer 0 serializes an empty link_lists.bin and loads
fine. Flagging that shape as corrupt produced a self-perpetuating
quarantine loop — repair rebuilt the byte-identical all-layer-0 segment,
the next cold start re-quarantined it, accumulating drift dirs (221 MB in
the reported case) with no ingestion involved.
Use the persist-completion marker as the discriminator instead. ChromaDB
writes index_metadata.pickle last, so an intact pickle envelope proves the
flush finished and the empty link_lists.bin is the legitimate all-layer-0
shape. Only treat an empty link_lists.bin as a partial flush when there is
real payload AND no completion marker (absent or truncated pickle). The
#1457 partial-flush protection (real payload, no/truncated marker) is
preserved; the byte-sniff is factored into _hnsw_metadata_marker_intact
and reused by _segment_appears_healthy.
Also fixes the related single-writer stale-quarantine false positive
(#1564), which shares this all-layer-0 root cause.
The non-daemon synchronous mine fallback in _mine_sync() spawned the
mine subprocess without CREATE_NO_WINDOW, flashing a visible console
window on every PreCompact fire on Windows. The async paths
(_spawn_mine, _desktop_toast) already pass it via _detached_popen_kwargs();
this sync path was missed. getattr(..., 0) is a no-op off-Windows.
Fixes#1862
Co-authored-by: David Finkelstein <david@finkelstein.us>
* fix: sanitize wing slug for project dirs with special characters
Project folders containing characters outside sanitize_name's set
(e.g. a leading '+') leaked into the derived wing name, producing names
like 'wing_+project' that config.sanitize_name rejects, silently
breaking diary auto-save for that project.
Add _safe_wing_slug(): collapse non-word runs to '_', trim, and fall
back to 'sessions' when a name reduces to nothing. Route the three
wing-derivation sites through it.
Tests: unit cases for the helper plus a hypothesis property test
asserting wing_<slug> always passes sanitize_name for any input.
* fix: preserve dots and apostrophes in wing slug for backward compatibility
The first pass collapsed every non-word character (including dot and
apostrophe) to underscore, renaming existing valid wings — e.g. my.app
became wing_my_app — which would orphan diary entries already filed under
the old name. Keep dot and apostrophe (both accepted by sanitize_name),
collapse consecutive dots to avoid the path-traversal rejection, and trim
edge separators.
Add backward-compatibility tests for previously-valid names plus a
double-dot collapse test.
* fix: cap wing slug length to stay within sanitize_name's limit
sanitize_name rejects names over 128 characters, so a very long project
directory name would produce a wing name that fails validation,
re-triggering the silent auto-save break this PR fixes. Truncate the slug
to 120 chars (the wing_ prefix keeps the total under 128).
Widen the hypothesis property test to max_size=300 so it exercises the
length path, and add an explicit truncation test.
Addresses gemini-code-assist review feedback on PR #1852.
---------
Co-authored-by: Ivan Antsimonau <ivan.antsimonau@katim.com>
daemon.py:_detached_kwargs was the last production spawn site still using DETACHED_PROCESS. Swap it to CREATE_NO_WINDOW, matching the hook miner's _detached_popen_kwargs fixed in #1848 — the dedicated follow-up the review bot asked for. `grep -rn DETACHED_PROCESS mempalace/` now returns zero production hits.
Survivability is unchanged: CREATE_BREAKAWAY_FROM_JOB (escapes the parent Job Object's kill-on-close) plus the daemon never being attached to the launching console carry survive-terminal-close; CREATE_NEW_PROCESS_GROUP (also kept) isolates Ctrl-C/Break. CREATE_NO_WINDOW is ignored when OR'd with DETACHED_PROCESS, so this replaces the flag rather than adding it. The daemon already redirects stdout/stderr to daemon.log and reads no stdin, so it needs no console.
Adds the first tests for _detached_kwargs (posix + windows, cross-platform monkeypatch of the Windows-only flag constants, mirroring the #1848 hooks_cli tests).
The original commit printed SKIP for oversized files to stdout but the
sibling SKIP for symlinks in the same scan_project / scan_convos already
went to stderr. Align the new line with that convention.
Also adds a SKIP-with-error log for the except OSError arm right below the
size check. Files whose stat() raises (permission denied, racing delete,
broken symlink that survived the earlier is_symlink check) were the same
bug class as the silent oversize drop.
Tests switched from captured.out to .err and tightened to the full
template; new test covers the OSError arm via a selective Path.stat
monkeypatch with a follow_symlinks gate for Python 3.10+.
Both miner.py and convo_miner.py silently skip files larger than the
10 MB limit with a bare continue. This is especially painful for
conversation mining where long Claude/ChatGPT exports routinely
exceed 10 MB and vanish with no trace.
Print a SKIP warning per oversized file, matching the existing format
in split_mega_files.py.
The write-ahead log gained its own module in v3.5.0 but sat at 82% coverage;
the uncovered lines were exactly the failure/guard branches that uphold its
contracts: the cache-hit early return, the restricted-FS chmod/mkdir swallow
paths, and the promise that a WAL write failure is logged and never crashes
the calling tool. Add five tests covering those branches plus the non-string
redaction marker, bringing mempalace/wal.py to 100% and locking the
crash-safety guarantees against regression. Test-only; no production change.
- ruff format llm_client.py and miner.py (lint job)
- _copy_file_no_follow: close src fd if the dst open fails (no leak), and
route the rebuild restore through it so backup + restore share one
no-follow/regular-file path
- update repair tests to assert the unified hardened copy instead of the
removed shutil.copy2 calls; backup paths are now timestamped
- update normalize large-file test to stub fstat (size is checked on the
open fd, not via a pre-open os.path.getsize)
Bump version to 3.5.0 across version.py, pyproject.toml, the Claude/Codex
plugin manifests, the README badge, and uv.lock. Refresh the "N MCP tools"
prose from 34 to 35 (delete_by_source #1729 and checkpoint #1851 each added a
tool). Add the 3.5.0 CHANGELOG entry.
ChromaDB's rust HNSW core intermittently fails compaction on Windows with
"Failed to apply logs to the hnsw segment writer" during add/update — a
long-standing, non-reproducible-on-Linux/macOS flake that hits different tests
(test_migrate_wings, test_closets) across unrelated commits and has been
turning otherwise-green release/CI runs red at random.
Add pytest-rerunfailures and wire `--reruns 2 --only-rerun "Failed to apply
logs to the hnsw segment writer"` into the test-windows job only. The
--only-rerun scope means a real, deterministic failure still fails on the first
run; only this specific transient native-dependency error is retried. The
Linux and macOS jobs deliberately keep zero reruns so genuine regressions
surface there loudly.
The opt-in HTTP transport reuses the stdio dispatcher and binds loopback by
default, but /mcp was unauthenticated with no protection against a malicious
web page reaching a DNS-rebound localhost server, and its tests reached for
Starlette/uvicorn (not project deps) so they were silently skipped in CI —
the production _serve_http handler had zero coverage.
Hardening:
- Pin the Host header to loopback literals + the bound host on a loopback bind
(DNS-rebinding defense); relaxed for a deliberately non-loopback bind, which
is the operator's call and may sit behind a Host-rewriting proxy.
- Reject any browser Origin that isn't a loopback origin (rebinding/SSRF guard);
non-browser MCP clients omit Origin and are unaffected.
- Optional bearer token via MEMPALACE_MCP_HTTP_TOKEN (constant-time compare);
required on /mcp, never on /healthz so liveness probes work credential-free.
- Warn loudly when bound to a non-loopback host (palace reachable from network).
Testability:
- Split _build_http_server() out of _serve_http() so tests bind 127.0.0.1:0 and
drive the real handler over a loopback socket via stdlib http.client.
- Replace the skipped Starlette reimplementation with 12 tests covering dispatch,
initialize, /healthz, 404, parse-error, the 16 MiB cap, notification 202, and
the Host/Origin/token rejections — no third-party deps.