30 KiB

Raw Blame History

name

Systematic Debugging

Overview

Random fixes waste time and create new bugs. Quick patches mask underlying issues.

Core principle: ALWAYS find root cause before attempting fixes. Symptom fixes are failure.

Violating the letter of this process is violating the spirit of debugging.

The Iron Law

NO FIXES WITHOUT ROOT CAUSE INVESTIGATION FIRST

If you haven't completed Phase 1, you cannot propose fixes.

When to Use

Use for ANY technical issue:

Test failures
Bugs in production
Unexpected behavior
Performance problems
Build failures
Integration issues

Use this ESPECIALLY when:

Under time pressure (emergencies make guessing tempting)
"Just one quick fix" seems obvious
You've already tried multiple fixes
Previous fix didn't work
You don't fully understand the issue

Don't skip when:

Issue seems simple (simple bugs have root causes too)
You're in a hurry (rushing guarantees rework)
Someone wants it fixed NOW (systematic is faster than thrashing)

The Four Phases

You MUST complete each phase before proceeding to the next.

readiness_status: available

Phase 1: Root Cause Investigation

BEFORE attempting ANY fix:

1. Read Error Messages Carefully

Don't skip past errors or warnings
They often contain the exact solution
Read stack traces completely
Note line numbers, file paths, error codes

Action: Use read_file on the relevant source files. Use search_files to find the error string in the codebase.

2. Reproduce Consistently

Can you trigger it reliably?
What are the exact steps?
Does it happen every time?
If not reproducible → gather more data, don't guess

Action: Use the terminal tool to run the failing test or trigger the bug:

# Run specific failing test
pytest tests/test_module.py::test_name -v

# Run with verbose output
pytest tests/test_module.py -v --tb=long

3. Check Recent Changes

What changed that could cause this?
Git diff, recent commits
New dependencies, config changes

Action:

# Recent commits
git log --oneline -10

# Uncommitted changes
git diff

# Changes in specific file
git log -p --follow src/problematic_file.py | head -100

4. Gather Evidence in Multi-Component Systems

WHEN system has multiple components (API → service → database, CI → build → deploy):

BEFORE proposing fixes, add diagnostic instrumentation:

For EACH component boundary:

Log what data enters the component
Log what data exits the component
Verify environment/config propagation
Check state at each layer

Run once to gather evidence showing WHERE it breaks. THEN analyze evidence to identify the failing component. THEN investigate that specific component.

5. Trace Data Flow

WHEN error is deep in the call stack:

Where does the bad value originate?
What called this function with the bad value?
Keep tracing upstream until you find the source
Fix at the source, not at the symptom

Action: Use search_files to trace references:

# Find where the function is called
search_files("function_name(", path="src/", file_glob="*.py")

# Find where the variable is set
search_files("variable_name\\s*=", path="src/", file_glob="*.py")

Phase 1 Completion Checklist

Error messages fully read and understood
Issue reproduced consistently
Recent changes identified and reviewed
Evidence gathered (logs, state, data flow)
Problem isolated to specific component/code
Root cause hypothesis formed

STOP: Do not proceed to Phase 2 until you understand WHY it's happening.

readiness_status: available

Phase 2: Pattern Analysis

Find the pattern before fixing:

1. Find Working Examples

Locate similar working code in the same codebase
What works that's similar to what's broken?

Action: Use search_files to find comparable patterns:

search_files("similar_pattern", path="src/", file_glob="*.py")

2. Compare Against References

If implementing a pattern, read the reference implementation COMPLETELY
Don't skim — read every line
Understand the pattern fully before applying

3. Identify Differences

What's different between working and broken?
List every difference, however small
Don't assume "that can't matter"

4. Understand Dependencies

What other components does this need?
What settings, config, environment?
What assumptions does it make?

readiness_status: available

Phase 3: Hypothesis and Testing

Scientific method:

1. Form a Single Hypothesis

State clearly: "I think X is the root cause because Y"
Write it down
Be specific, not vague

2. Test Minimally

Make the SMALLEST possible change to test the hypothesis
One variable at a time
Don't fix multiple things at once

3. Verify Before Continuing

Did it work? → Phase 4
Didn't work? → Form NEW hypothesis
DON'T add more fixes on top

4. When You Don't Know

Say "I don't understand X"
Don't pretend to know
Ask the user for help
Research more

readiness_status: available

Phase 4: Implementation

Fix the root cause, not the symptom:

1. Create Failing Test Case

Simplest possible reproduction
Automated test if possible
MUST have before fixing
Use the test-driven-development skill

2. Implement Single Fix

Address the root cause identified
ONE change at a time
No "while I'm here" improvements
No bundled refactoring

3. Verify Fix

# Run the specific regression test
pytest tests/test_module.py::test_regression -v

# Run full suite — no regressions
pytest tests/ -q

4. If Fix Doesn't Work — The Rule of Three

STOP.
Count: How many fixes have you tried?
If < 3: Return to Phase 1, re-analyze with new information
If ≥ 3: STOP and question the architecture (step 5 below)
DON'T attempt Fix #4 without architectural discussion

5. If 3+ Fixes Failed: Question Architecture

Pattern indicating an architectural problem:

Each fix reveals new shared state/coupling in a different place
Fixes require "massive refactoring" to implement
Each fix creates new symptoms elsewhere

STOP and question fundamentals:

Is this pattern fundamentally sound?
Are we "sticking with it through sheer inertia"?
Should we refactor the architecture vs. continue fixing symptoms?

Discuss with the user before attempting more fixes.

This is NOT a failed hypothesis — this is a wrong architecture.

readiness_status: available

Red Flags — STOP and Follow Process

If you catch yourself thinking:

"Quick fix for now, investigate later"
"Just try changing X and see if it works"
"Add multiple changes, run tests"
"Skip the test, I'll manually verify"
"It's probably X, let me fix that"
"I don't fully understand but this might work"
"Pattern says X but I'll adapt it differently"
"Here are the main problems: [lists fixes without investigation]"
Proposing solutions before tracing data flow
"One more fix attempt" (when already tried 2+)
Each fix reveals a new problem in a different place

ALL of these mean: STOP. Return to Phase 1.

If 3+ fixes failed: Question the architecture (Phase 4 step 5).

Common Rationalizations

Excuse	Reality
"Issue is simple, don't need process"	Simple issues have root causes too. Process is fast for simple bugs.
"Emergency, no time for process"	Systematic debugging is FASTER than guess-and-check thrashing.
"Just try this first, then investigate"	First fix sets the pattern. Do it right from the start.
"I'll write test after confirming fix works"	Untested fixes don't stick. Test first proves it.
"Multiple fixes at once saves time"	Can't isolate what worked. Causes new bugs.
"Reference too long, I'll adapt the pattern"	Partial understanding guarantees bugs. Read it completely.
"Add `.ok()` / `// ignore error` to suppress the symptom"	Fix at the source. Suppressing means the bug still exists — it just stopped reporting. Example: ALTER TABLE failing with "column missing" → not "add `.ok()`", but "make CREATE TABLE include the column + migrate existing DBs".
"Skip updating design docs when fixing schema issues"	DESIGN.md and implementation are a contract. If CREATE TABLE SQL is missing a column that DESIGN.md specifies, the fix is BOTH to add the column to the SQL AND to align DESIGN.md, not to hide the gap at runtime.
"Propose a fix before reading all relevant files"	Read the full picture first. Today's session: graph_prune.rs error "pagerank column missing" → the fix required reading graph_sqlite.go CREATE TABLE + DESIGN.md schema spec + understanding the Go→Rust IPC architecture. Jumping to the Rust side masks the Go-side schema gap.
"One more fix attempt" (after 2+ failures)	3+ failures = architectural problem. Question the pattern, don't fix again.

Quick Reference

Phase	Key Activities	Success Criteria
1. Root Cause	Read errors, reproduce, check changes, gather evidence, trace data flow	Understand WHAT and WHY
2. Pattern	Find working examples, compare, identify differences	Know what's different
3. Hypothesis	Form theory, test minimally, one variable at a time	Confirmed or new hypothesis
"Reference too long, I'll adapt the pattern"	Partial understanding guarantees bugs. Read it completely.
"Function logic looks right, deployed"	Test the actual output, not just the code path. A function can have correct logic but broken implementation (wrong regex, wrong operator, wrong import) and still return silently wrong results. Always validate with a concrete test case.

Critical Pitfalls in Regex and Text Processing

`\b` Word Boundary Fails in Non-ASCII Text

Problem: Python's \b word boundary assertion only matches at ASCII word boundaries. In Chinese, Japanese, Korean, or any text using non-Latin characters surrounded by no spaces, \b silently matches nothing — no error, no warning, just zero matches.

import re
text = "海南热带海洋学院分数=120035名"

# WRONG: \b in Chinese text always returns empty
re.findall(r'\b(\d{5,6})\b', text)  # → []

# CORRECT: negative lookahead/lookbehind
re.findall(r'(?<!\d)(\d{5,6})(?!\d)', text)  # → ['120035']

Why it's dangerous: The function using \b compiles fine, runs without error, and returns [] — which can be silently ignored if the caller doesn't check for empty results. The bug looks like "the regex pattern is right but nothing matches" and can be misdiagnosed as "the text doesn't contain the pattern."

How to detect: Write a unit test with Chinese text containing the target pattern and assert that matches are found. If the test returns empty, you have the bug.

Rule: When processing text in any language other than English, use (?<!\d) and (?!d) instead of \b. Or always use them as a matter of habit — they work correctly in both ASCII and non-ASCII contexts.

readiness_status: available

Over-Constrainted Format Assumptions Cause Coordinated Validation Failure

Problem: Multiple independent validation layers silently fail together because they all depend on the same hidden text format assumption.

Real example from gaokao-site (2026-06-19): The AI could hallucinate entirely fabricated schools (e.g., "福建师范大学" at 510分, though the DB minimum is 548分). Two independent validation functions were supposed to catch this:

_verify_schools_in_database: checked if school names existed in DB
_verify_rank_score_consistency: cross-checked score→rank consistency via yiyi table

Both returned [] (no errors) even for completely fabricated schools. The root cause: both functions used the same regex anchor — matching school names only when preceded by ●. But the AI output used | or ： separators, so neither function matched anything.

Why it's dangerous: Defense-in-depth feels robust (two layers!) but both fail simultaneously. When both return [], you believe the content is valid when it is entirely hallucinated.

How to detect it:

Both validators return [] for the same input
The content clearly violates domain knowledge (e.g., a school scoring 510 when DB minimum is 548)
A pattern used as an anchor appears in one function's code and is assumed to be consistent across all validators

The fix: Validators must use format-agnostic extraction — match school names regardless of surrounding punctuation (●, |, ：, \n, etc.). A good pattern for Chinese school names:

# Matches school names BEFORE their score/rank data, regardless of separator
school_pattern = re.findall(
    r'([^\s　,，、。.●▪▸\-|：:\n（）\(\)]{2,12}?(?:大学|学院|学校|职院|院)[^\s　,，、。.●▪▸\-|：:\n（）\(\)]*)',
    reply_text
)

Also add a second check that doesn't rely on text format: if an AI claims school X has score Y, query the DB for X's actual minimum score. If Y is far below the DB minimum, it's a hallucination regardless of whether the text format matches.

Rule: When adding a new validation layer, explicitly ask: "Does this depend on any text format assumptions that could be violated by LLM output variation?" Test with adversarial formats that differ from what the LLM typically produces.

readiness_status: available

Single-Quote Conflict in Inline JS onclick Handlers

Problem: When building HTML strings in JavaScript that contain onclick handlers with arguments, single quotes inside the outer single-quoted string cause a syntax error. The inner ' is parsed as closing the string, not as a literal character, causing the entire script block to fail to parse.

// BROKEN: inner '' in onclick breaks the string
return '<div class="similar-school-card" onclick="showSchoolDetail(''+s.school+'', ''+cat+'')">' +

// FIXED: escape inner single quotes with \'
return '<div class="similar-school-card" onclick="showSchoolDetail(\''+s.school+'\', \''+cat+'\')">' +

Why it's dangerous: The entire <script> block fails to parse — ALL functions become undefined. Server-side logs show 200 OK, API works fine, but no JavaScript on the page runs. The symptom is "click does nothing" which gets misdiagnosed as an API or network issue.

How to detect: Extract inline scripts from the HTML and validate with Node.js:

python3 -c "
import re
with open('page.html','r') as f: html = f.read()
scripts = re.findall(r'<script[^>]*>(.*?)</script>', html, re.DOTALL)
with open('/tmp/page_script.js','w') as f: f.write(scripts[0].strip())
"
node --check /tmp/page_script.js

Rule: After editing inline JS that concatenates HTML with onclick handlers, always run node --check to verify the script block is syntactically valid before deploying. When building HTML onclick attributes with variables, use \''+var+'\' instead of ''+var+'', or use double-quoted outer strings if HTML uses single quotes.

readiness_status: available

Silent `.catch()` Error Swallowing in Promise Chains

Problem: When a Promise chain has a .catch() handler, errors thrown inside .then() blocks (including ReferenceError from undefined libraries) are caught silently. The user sees the catch handler's fallback message, not the actual error, making "a chart failed" look like "the API is down."

Real example from gaokao-site (2026-06-20): renderPlanRadar() threw ReferenceError: echarts is not defined inside a .then() block. The .catch() caught it and overwrote valid API results with "获取数据失败" — leading to 2+ hours of misdiagnosis.

How to detect:

Look for multiple independent operations inside a single .then() block with a .catch()
If the error message doesn't match the expected failure, check if any inner function could throw

Fix: Isolate risky code with try/catch inside .then():

.then(function(data) {
    renderResults(data);  // required — keep in main path
    try { renderChart(data); } catch(e) { console.warn('Chart failed:', e); }
})

Rule: When a .catch() handler exists, isolate any library-dependent or risky sub-operation with try/catch so one sub-feature failure doesn't take down the entire result.

readiness_status: available

Data Collected But Not Sent Across Layer Boundaries

Problem: Form data / user preferences are collected and stored at one layer (frontend JS) but the subsequent API call doesn't include them in the request payload. The backend receives default values and the feature appears to silently do nothing.

Real example from gaokao-site (2026-06-20): formData.major and formData.region were collected on line 375-376 of plan.html, stored correctly, and displayed in the summary. But the fetch('/api/prob', {body: JSON.stringify({score, subject})}) on line 427-431 only sent score and subject — major and region were omitted. Backend always got major='不限', so every combination of preferences returned the same results.

// BROKEN: preferences collected but not sent
formData.major = getSelectedMajors();  // line 375 — collected ✓
formData.region = document.getElementById('region').value;  // line 376 — collected ✓
fetch('/api/prob', {
    body: JSON.stringify({score: formData.score, subject: formData.subject})  // line 430 — major/region MISSING ✗
});

// FIXED: include all collected fields
fetch('/api/prob', {
    body: JSON.stringify({score: formData.score, subject: formData.subject, major: formData.major, region: formData.region})
});

Why it's dangerous: Multiple indicators suggest it works:

The user sees their selection highlighted on screen
The summary text displays the selected values
The API returns 200 OK with valid-looking data
Only a cross-layer comparison reveals the gap

How to detect:

Trace the complete data flow: input → stored in variable → included in API call → received by backend → used in logic
Add a log/print at each boundary: "data collected: X", "data sent: X", "data received: X", "data used: X"
The layer where data is collected != the layer where data is sent is the gap

Root cause pattern: The developer added a new field to the form/settings but forgot to update the serialization code that builds the API request body. This is especially common when the collection and sending are in different functions or different files.

Rule: Every time you add a form field or setting, trace it through ALL layers: collection → serialization → API request → backend handler → query. Verify at each layer with a debug log or test. If the field appears in formData/state but not in the API payload, you have this bug.

readiness_status: available

Data Extracted But Not Threaded Through Backend Call Chain

Problem: Data is extracted from user input at the entry layer of the backend (e.g., profile/settings handler, message parser), but the parameter is not passed through the full function call chain to where it's actually used. The intermediate function uses a default/derived value instead, silently making the user input meaningless for the actual logic.

Real example from gaokao-site (2026-06-25): User typed "历史类544分位次26887". The extract_score_subject() function successfully extracted user_rank=26887. But:

build_reply() extracted it and displayed it in the AI prompt
BUT it called recommend(score, subject, ...) without passing user_rank
recommend() called _calc_prob_data(score, subject, ...) without user_rank
_calc_prob_data() used score_to_rank(score) → got database default 30042

Result: User's rank appeared in the AI prompt text, the AI acknowledged it ("您提供的位次26887与数据库默认位次30042不同"), but ALL school matching used rank=30042. User input was cosmetic only.

Detection pattern:

The AI/response mentions the user's value correctly (extraction works)
But the computed/derived values in the output don't match (school lists, ranks, scores)
Trace the data flow: extract → pass? → use. If any link breaks, the value is lost

# TRACE: Follow the parameter through the call chain
extract_score_subject(msg) → returns user_rank
       ↓
build_reply() → has user_rank, calls recommend(score, subject, **MISSING user_rank**)
       ↓
recommend(score, subject) → calls _calc_prob_data(score, subject, **MISSING user_rank**)
       ↓
_calc_prob_data(score, subject) → uses score_to_rank(score) → **wrong result**

Fix: The parameter must be explicitly threaded through EVERY function in the chain:

# Before: rank extracted but dropped
build_reply():
    _, _, _, user_rank = extract_score_subject(msg)
    r = recommend(score, subject)  # user_rank not passed!

# After: rank threaded through
build_reply():
    _, _, _, user_rank = extract_score_subject(msg)
    r = recommend(score, subject, user_rank=user_rank)

recommend(score, subject, user_rank=None):
    prob = _calc_prob_data(score, subject, user_rank=user_rank)

_calc_prob_data(score, subject, user_rank=None):
    if user_rank:
        student_rank = user_rank
    else:
        student_rank = score_to_rank(conn, subject, score)

Why it's dangerous: The value appears to be "used" because the entry layer sends it to logs, displays it to the user, etc. Only a full call-chain trace reveals it was dropped before reaching the logic layer.

How to detect systematically:

List every function that has parameters that could affect output
Add a debug log at the entry layer showing all extracted values
Add a debug log at the logic layer showing all received values
Compare — if entry has a non-None value that logic doesn't receive, break

Root cause pattern: The function call chain grew incrementally (new features added layer by layer). When a new parameter was added, it was only added to the top-level function signature, not threaded through all intermediate functions. The extractor was updated (new parameter added), but the signature of intermediate functions was not.

Rule: Every time you add a new user-input parameter that affects business logic:

Trace the entire call chain from entry point to where it's used
Add the parameter to EVERY function in the chain
Add a default value (e.g., None) at each intermediate function so existing callers don't break
Verify with a test: pass a known value and confirm it reaches the logic layer

See also: references/rank-matching-gaokao-case.md — full session walkthrough with rank_to_score inverse lookup.

Missing CDN Dependency for Library-Powered Features

Problem: Adding a frontend feature that calls a third-party library (echarts, html2canvas, etc.) without adding the corresponding <script src="..."> tag. The library is undefined at runtime, causing errors that get swallowed by Promise .catch().

Real example from gaokao-site (2026-06-20): Added ECharts radar chart to plan.html but forgot to add the ECharts CDN <script> tag. School.html had it; plan.html didn't. Result: silent failure of entire plan page.

How to detect: Search the script for library calls (echarts, html2canvas), then check for corresponding <script src> tags in the HTML.

Fix: Add CDN script with defer plus a polling wrapper for deferred-load race conditions:

<script defer src="https://cdn.jsdelivr.net/npm/echarts@5.4.3/dist/echarts.min.js"></script>

function waitEcharts(cb){
    if(typeof echarts !== "undefined"){ cb(); return; }
    var check = setInterval(function(){
        if(typeof echarts !== "undefined"){ clearInterval(check); cb(); }
    }, 200);
    setTimeout(function(){ clearInterval(check); }, 15000);
}

Rule: When adding a library-powered feature, ALWAYS: (1) add CDN <script> tag, (2) use defer, (3) wrap calls with a waitLibrary() polling guard, (4) verify with node --check.

readiness_status: available

Investigation Tools

Use these Hermes tools during Phase 1:

search_files — Find error strings, trace function calls, locate patterns
read_file — Read source code with line numbers for precise analysis
terminal — Run tests, check git history, reproduce bugs
web_search/web_extract — Research error messages, library docs

6. LLM Context Data Flow — Debugging "AI Ignores Provided Data"

WHEN an LLM receives structured data in its context but claims it doesn't have it (e.g. "暂无具体专业录取分数据" when scores were provided):

Three-layer check — do in order:

Layer 1 — Data Fetch: Is the data actually being retrieved from the source?

Test the query/API call directly (bypass the AI entirely)
Use raw SQL or curl to confirm data exists
Don't assume "the code looks correct" — run it and verify output

Action:

# Direct SQL test on the actual database
conn.execute("SELECT major, min_score FROM majors WHERE ...").fetchall()
# Or curl the API endpoint
curl http://127.0.0.1:8080/api/endpoint

Layer 2 — Data Format: Is the data formatted correctly for the LLM?

Check for escape bugs: "\\\\n".join() produces literal \n chars, NOT newlines. The LLM sees one long paragraph it can't parse.
Check for truncation: is the context so long the data section gets cut off?
Check encoding: are non-ASCII characters (Chinese, special symbols) rendering correctly?

Common bug patterns:

# WRONG — produces literal \n characters, not newlines
context = "\\\\n".join(lines)
# AI sees: "line1\\nline2\\nline3" — unparseable blob

# CORRECT — produces real newlines
context = "\n".join(lines)
# AI sees:
# line1
# line2
# line3

Detection: Print/repr the context just before it's sent to the API:

with open('/tmp/context_debug.txt', 'w') as f:
    f.write(repr(context[:2000]))  # Check for \\n vs \n

Layer 3 — Prompt Instruction: Does the prompt tell the AI to use the data?

Look for instructions that give the AI permission to say "no data"
"If data is missing, say so" → AI may say "missing" even when data exists
"Use data if available" → weaker than "MUST extract from data"
The instruction must be specific and forceful

Prompt wording matters:

# WEAK — AI will default to "no data" when uncertain
"If data is provided, use it; if not, say so."

# STRONG — AI forced to use what's there
"You MUST extract from the data below. Only say 'no data' if the data explicitly states 'no data'."

Root cause distribution (from real session):

~40%: Escape/format bugs (\\n, encoding, truncation)
~35%: Prompt gives AI permission to ignore data
~25%: Data not actually being fetched (query error, missing table, wrong year)

Action: Trace all three layers before concluding the AI is "being stupid." Most often, the bug is in the pipeline delivering data to it.

See also: references/llm-context-data-gaokao-case.md — real session walkthrough of this exact pattern (31万条 per-major scores ignored by AI due to \\\\n + prompt gap).

readiness_status: available

7. Batch Diagnostic Instrumentation (for multi-instance problems)

WHEN debugging issues across 50+ items (317 pages, 150+ API calls, 1000+ records):

Don't inspect items one by one. Instrument the pipeline to produce a structured diagnostic log, then aggregate.

# Pattern: diagnose.py — runs the full pipeline on every item,
# outputs structured JSON for every item, then aggregates
def diagnose_one(item):
    result = run_pipeline(item)
    return {
        'item_id': item.id,
        'status': 'balanced' if is_balanced(result) else 'unbalanced',
        'expected': extract_expected(result),
        'actual': extract_actual(result),
        'bounds': extract_bounds(result),  # structural metadata
        'raw_data': extract_raw_ocr(result),
    }

# Run all → output/diagnose.json
# Analyze with Python aggregation:
categories = Counter()
for page in report['pages']:
    if not page['balanced']:
        if page['total_debit'] > 0 and page['total_credit'] == 0:
            categories['missing_credits'] += 1
        elif page['total_credit'] > 0 and page['total_debit'] == 0:
            categories['missing_debits'] += 1
        # ... more patterns

Key principles:

Never manually page through 100+ items — write a script that aggregates first
Output structured JSON, not text — enables jq / Python pattern analysis
Include metadata (headers found, raw counts per zone) — not just pass/fail
Classify failures automatically — present category counts, not raw data
When user asks "what's happening?": run diagnostic, present summarized categories with example items, never dump raw data

Action:

# Write diagnose.py → run → analyze with Python
# Present to user: "3 failure categories identified: [A] N items, [B] M items, [C] K items. Root cause of [A]: ..."

For complex multi-component debugging, dispatch investigation subagents:

delegate_task(
    goal="Investigate why [specific test/behavior] fails",
    context="""
    Follow systematic-debugging skill:
    1. Read the error message carefully
    2. Reproduce the issue
    3. Trace the data flow to find root cause
    4. Report findings — do NOT fix yet

    Error: [paste full error]
    File: [path to failing code]
    Test command: [exact command]
    """,
    toolsets=['terminal', 'file']
)

With test-driven-development

When fixing bugs:

Write a test that reproduces the bug (RED)
Debug systematically to find root cause
Fix the root cause (GREEN)
The test proves the fix and prevents regression

Real-World Impact

From debugging sessions:

Systematic approach: 15-30 minutes to fix
Random fixes approach: 2-3 hours of thrashing
First-time fix rate: 95% vs 40%
New bugs introduced: Near zero vs common

No shortcuts. No guessing. Systematic always wins.

30 KiB Raw Blame History Unescape Escape

Systematic Debugging

Overview

The Iron Law

When to Use

The Four Phases

Phase 1: Root Cause Investigation

1. Read Error Messages Carefully

2. Reproduce Consistently

3. Check Recent Changes

4. Gather Evidence in Multi-Component Systems

5. Trace Data Flow

Phase 1 Completion Checklist

Phase 2: Pattern Analysis

1. Find Working Examples

2. Compare Against References

3. Identify Differences

4. Understand Dependencies

Phase 3: Hypothesis and Testing

1. Form a Single Hypothesis

2. Test Minimally

3. Verify Before Continuing

4. When You Don't Know

Phase 4: Implementation

1. Create Failing Test Case

2. Implement Single Fix

3. Verify Fix

4. If Fix Doesn't Work — The Rule of Three

5. If 3+ Fixes Failed: Question Architecture

Red Flags — STOP and Follow Process

Common Rationalizations

Quick Reference

Critical Pitfalls in Regex and Text Processing

\b Word Boundary Fails in Non-ASCII Text

Over-Constrainted Format Assumptions Cause Coordinated Validation Failure

Single-Quote Conflict in Inline JS onclick Handlers

Silent .catch() Error Swallowing in Promise Chains

Data Collected But Not Sent Across Layer Boundaries

Data Extracted But Not Threaded Through Backend Call Chain

Missing CDN Dependency for Library-Powered Features

Investigation Tools

6. LLM Context Data Flow — Debugging "AI Ignores Provided Data"

7. Batch Diagnostic Instrumentation (for multi-instance problems)

With test-driven-development

Real-World Impact

30 KiB

Raw Blame History

`\b` Word Boundary Fails in Non-ASCII Text

Silent `.catch()` Error Swallowing in Promise Chains