b70f06a69e
Allow projects to specify .gitignore-style patterns that the miner should
skip, without relying on .gitignore for mining control.
A new optional exclude_patterns list in mempalace.yaml is parsed by the
existing GitignoreMatcher class via a new from_patterns() classmethod —
same syntax, same semantics as .gitignore, no new dependency.
exclude_patterns:
- '*.md'
- '*.yaml'
- 'docs/' # dir-only: prunes entire tree without descending
- 'dist/'
- 'coverage/'
Key behaviour:
- Patterns follow .gitignore rules: anchoring (/pattern), dir-only
- dirs[:] pruning via GitignoreMatcher.matches(..., is_dir=True) so
excluded subtrees are never walked
- Checked after .gitignore filtering; force_include (--include-ignored)
bypasses exclude_patterns
- Pre-scanned files lists (init double-scan optimisation) are filtered too
- Backwards compatible: omitting exclude_patterns changes nothing
Changes:
- GitignoreMatcher.from_patterns(): new classmethod, same rule parser as
from_dir(), reads from a list instead of a file on disk
- scan_project(): builds one exclude_matcher before os.walk; used for
both dirs[:] pruning and per-file filtering
- _mine_impl(): applies the exclude matcher to pre-scanned files lists
when provided by the caller
- tests/test_miner.py: three new tests
test_scan_project_exclude_patterns_skips_matching_files
test_scan_project_exclude_patterns_prunes_entire_directory
test_scan_project_exclude_patterns_include_ignored_bypasses_exclusion
|
||
|---|---|---|
| .. | ||
| backends | ||
| data | ||
| i18n | ||
| instructions | ||
| sources | ||
| README.md | ||
| __init__.py | ||
| __main__.py | ||
| _stdio.py | ||
| backups.py | ||
| cli.py | ||
| closet_llm.py | ||
| collision_scan.py | ||
| config.py | ||
| convo_miner.py | ||
| convo_scanner.py | ||
| corpus_origin.py | ||
| daemon.py | ||
| dedup.py | ||
| dialect.py | ||
| diary_ingest.py | ||
| dynamics.py | ||
| embedding.py | ||
| entities.py | ||
| entity_detector.py | ||
| entity_registry.py | ||
| exporter.py | ||
| fact_checker.py | ||
| format_miner.py | ||
| general_extractor.py | ||
| hallways.py | ||
| hook_shell.py | ||
| hooks_cli.py | ||
| ids.py | ||
| instructions_cli.py | ||
| knowledge_graph.py | ||
| layers.py | ||
| llm_client.py | ||
| llm_refine.py | ||
| mcp_server.py | ||
| migrate.py | ||
| miner.py | ||
| normalize.py | ||
| onboarding.py | ||
| palace.py | ||
| palace_graph.py | ||
| project_scanner.py | ||
| py.typed | ||
| query_sanitizer.py | ||
| repair.py | ||
| room_detector_local.py | ||
| searcher.py | ||
| service.py | ||
| spellcheck.py | ||
| split_mega_files.py | ||
| sweeper.py | ||
| sync.py | ||
| version.py | ||
| wal.py | ||
README.md
mempalace/ — Core Package
The Python package that powers MemPalace. All modules, all logic.
Modules
| Module | What it does |
|---|---|
cli.py |
CLI entry point — routes to mine, search, init, compress, wake-up |
config.py |
Configuration loading — ~/.mempalace/config.json, env vars, defaults |
normalize.py |
Converts 5 chat formats (Claude Code JSONL, Claude.ai JSON, ChatGPT JSON, Slack JSON, plain text) to standard transcript format |
miner.py |
Project file ingest — scans directories, chunks by paragraph, stores to ChromaDB |
convo_miner.py |
Conversation ingest — chunks by exchange pair (Q+A), detects rooms from content |
searcher.py |
Semantic search via ChromaDB vectors — filters by wing/room, returns verbatim + scores |
layers.py |
4-layer memory stack: L0 (identity), L1 (critical facts), L2 (room recall), L3 (deep search) |
dialect.py |
AAAK compression — entity codes, emotion markers, 30x lossless ratio |
knowledge_graph.py |
Temporal entity-relationship graph — SQLite, time-filtered queries, fact invalidation |
palace_graph.py |
Room-based navigation graph — BFS traversal, tunnel detection across wings |
mcp_server.py |
MCP server — 34 tools, AAAK auto-teach, Palace Protocol, agent diary |
onboarding.py |
Guided first-run setup — asks about people/projects, generates AAAK bootstrap + wing config |
entity_registry.py |
Entity code registry — maps names to AAAK codes, handles ambiguous names |
entity_detector.py |
Auto-detect people and projects from file content |
general_extractor.py |
Classifies text into 5 memory types (decision, preference, milestone, problem, emotional) |
room_detector_local.py |
Maps folders to room names using 70+ patterns — no API |
spellcheck.py |
Name-aware spellcheck — won't "correct" proper nouns in your entity registry |
split_mega_files.py |
Splits concatenated transcript files into per-session files |
Architecture
User → CLI → miner/convo_miner → ChromaDB (palace)
↕
knowledge_graph (SQLite)
↕
User → MCP Server → searcher → results
→ kg_query → entity facts
→ diary → agent journal
The palace (ChromaDB) stores verbatim content. The knowledge graph (SQLite) stores structured relationships. The MCP server exposes both to any AI tool.