Chapter 20 - Fixing A Bug

Part: V - AI-Assisted Development Workflows

Thesis

A bug fix from an AI model must include a test that would have caught the bug before the fix. Without that test, the fix is a guess with temporary success.

Key Line

Never let the agent fix a bug it cannot reproduce.

Reproduce First

Before asking the model for anything, write a test that reproduces the bug.

The test should fail on the current code. If you cannot write a failing test, you do not understand the bug well enough to fix it. This is not a process gate - it is a diagnostic. The inability to write a failing test means the bug is not yet defined. A model prompt written against an undefined bug produces a plausible fix for the most common interpretation of the symptoms.

The reproduction test does not have to be elegant. It has to be specific: given this input in this state, the system does X when it should do Y. That sentence is the test. Write it as code.

Run the test on the current codebase without any fix applied. It must fail. If it passes, either the bug is not reproducible in the test environment or the test is wrong. Do not proceed to fix generation until the test fails for the right reason.

The Plausibility Trap For Bugs

The model reads an error message, recognizes a pattern from its training data, and produces the explanation that fits the most common cause.

The explanation is authoritative. It names a root cause. It proposes a fix. The fix addresses the named cause. When the named cause is the actual cause, this is fast and correct. When the actual bug is less common - an interaction between two libraries, a race condition, a state assumption that fails only on specific input - the confident diagnosis is harder to question than the original error message.

The defense is the reproduction test. If the model’s diagnosis is correct, applying its fix will make the reproduction test pass. If the reproduction test still fails after the fix, the diagnosis was wrong. The test does not care how authoritative the explanation sounded.

This is covered in depth in Chapter 31. The core rule here is: never accept a diagnosis. Accept a fix that makes the failing test pass.

Root Cause vs. Symptom

A fix that addresses the symptom leaves the root cause in place.

Ask explicitly for root cause analysis before the fix. The prompt form that works: “What is the underlying reason this error occurs, not just how to suppress it?” This forces the model past the first-order answer - add a null check, catch the exception, return early - to the structural reason the null arrives, the exception is possible, or the early return is needed.

Symptom fixes produce cascading bugs. The null check hides the fact that the upstream function can return null when it should not. The exception catch hides the fact that the operation is being called in an invalid state. The early return hides the fact that the caller should not have reached that code path. Each fix creates a new constraint that future changes must silently respect.

One question to ask before accepting any fix: “If this fix is correct, why did the bug exist? What structural condition made the bug possible?” A fix that cannot answer this question is probably addressing a symptom.

Classify The Bug By Where The Structure Was Missing

“Why did the bug exist” has a small number of real answers. Naming which one applies determines what the permanent fix looks like.

Model capability. The task was past what this model can reliably do at this size or context length. No prompt fixes this; the fix is a smaller task, a different model, or moving the decision out of the model entirely.
Specification. The rule existed only as an assumption in someone’s head. The prompt, schema, or docs never told the model - or the developer - what was actually required.
Parser/interface. The shape the code expects and the shape actually produced do not match, and nothing caught the mismatch before it caused damage.
Validator. A check exists but has a gap: an edge case it does not cover, a rule it enforces too loosely, or a condition it was never written to catch.
State. The bug is not in any function. It is an assumption about the data - a null that should not be possible, a duplicate that should not exist, a stale value nothing refreshed.
Authorization. The action was performed by something that should not have been allowed to perform it, regardless of whether the action itself was correct.
Commit policy. The proposal was fine right up until it crossed into permanent, public, or executable state without the check that should have gated that crossing.

This taxonomy matters most in an AI-native system, where “the bug” is as likely to be a gap in a validator or a schema as a defect in ordinary application code. A chatbot that occasionally proposes a refund for the wrong order is not a code bug in the usual sense - it is a validator gap: the amount check exists, the order-ownership check does not. Its reproduction test is not a unit test on a function. It is a fixture: a conversation transcript that should have been rejected and was not.

The category also decides whether reprompting is a real fix or a postponement. If the category is specification, and the model genuinely lacked information it now has, a prompt or schema addition can close the gap completely. If the category is validator, state, authorization, or commit policy, a prompt patch closes nothing - it only lowers the odds the model reproduces the same bad proposal on the next similar input, which leaves a bug one unlucky sample away from happening again under different phrasing or a different model version. Chapter 7’s rule applies here without modification: a better prompt is not a substitute for a safer system. The permanent fix moves the missing rule out of the prompt and into whichever reservoir actually enforces it - a schema that rejects the malformed shape, a validator that checks the condition structurally, a state constraint the database enforces, an authorization check that runs regardless of what the model intended.

Reading The Rejection And Escape Logs

Two logs are underused in most debugging workflows, and both already exist if Chapter 6’s and Chapter 10’s controls are in place.

The rejection log records every proposal a validator refused. Check it before writing a fix. A bug that “just started happening” often has weeks of rejection log entries first - a validator catching the bad shape every time, quietly, until a code path changed and started routing around the validator instead of through it.

The escape log - or its absence - is the more urgent read. An escape is a proposal that should have been rejected and was not: it crossed the commit boundary anyway. If the bug reached a user, something escaped. Find where. The rejection log proves the validator was working. The escape shows where it was not consulted, was bypassed, or did not cover the case that got through.

Add both logs to the standard bug-report intake alongside the stack trace. “What did the validator see, and did it fire?” is often a faster route to root cause than the model’s own explanation of what went wrong.

The Fix Plus Test Pattern

The sequence is fixed. Do not skip steps.

Write a failing test that reproduces the bug. Run it. Confirm it fails.
Ask the model to fix the bug, providing the reproduction test as context.
Apply the fix.
Run the reproduction test. It must pass.
Run the full test suite. No other test may break.
Commit. The test is part of the commit.

The test is not optional and it is not separate from the fix. A fix without its regression test is a promise that the bug is gone. A fix with its regression test is a proof that survives future changes.

Step 5 is non-negotiable. The model’s fix will sometimes address the specific failing case by introducing a new failure elsewhere. A fix that breaks a different test is not a fix. Do not merge until the full suite is green.

For a bug whose category is validator, state, authorization, or commit policy, the regression test may not be enough on its own, because the same failure can recur through a code path the unit test never exercises. Add a runtime invariant as well: a check that runs in production and refuses the same failure shape regardless of how the request arrived. The unit test proves the fix once, at commit time. The runtime invariant proves it on every request, indefinitely.

When The Fix Is Wrong

The test will tell you.

If the failing test still fails after applying the model’s fix, the fix is wrong. This is the engineering control working exactly as designed. It prevents a confident but incorrect fix from reaching the commit boundary.

When this happens, do not ask the model to adjust the fix until you understand why the fix failed. Read the test output. Understand what the fix changed and why that change was not sufficient. Then provide that analysis back to the model as context for a second attempt.

The failure loop - fix applied, test still fails, analyze, retry - is not a sign of a broken process. It is a sign of a working one. The loop terminates when the test passes. Not when the model says it should.

Post-Fix Review

After the fix is in and the tests pass, review the surrounding code.

One bug is often a symptom of a class of bugs. The null pointer in this function may be possible in three adjacent functions. The race condition in this handler may be possible in every handler that accesses the same resource. The incorrect assumption in this calculation may exist wherever the same formula appears.

The post-fix review is not a code audit. It is a targeted question: where else in this codebase could the same underlying condition exist? Spend ten minutes reading the code around the fixed location. Check for the same pattern. If you find it, fix it now while the context is loaded, or open a tracking issue immediately.

One more step: add a comment to the fix commit or PR explaining the root cause in a sentence. Not what the fix does - what the bug was. Future engineers working in this area will find it. It is the cheapest form of institutional memory.

Practical Artifact - Bug Fix Checklist

Gate	Action	What It Protects
Reproduction test written	Write a test that fails on current code	Proves you understand the bug
Test fails before fix	Run the test; confirm it fails for the right reason	Prevents testing the wrong thing
Bug classified	Name where structure was missing: model capability, specification, parser/interface, validator, state, authorization, or commit policy	Determines whether the fix must move a rule into a durable control
Rejection/escape logs checked	Review the validator rejection log and escape log for related entries	Surfaces whether the gap was already visible before this incident
Root cause documented	Write one sentence describing why the bug exists	Catches symptom-only fixes
Model fix applied	Apply the model’s proposed fix	-
Reproduction test passes	Run the failing test; confirm it now passes	Proves the fix addresses the bug
Full suite passes	Run all tests; confirm no regressions	Prevents the fix from breaking other behavior
Runtime invariant added	For validator, state, authorization, or commit-policy bugs, add a check that runs in production	Blocks recurrence through code paths the unit test does not exercise
Test committed with fix	The regression test is in the same commit or PR	Creates permanent protection
Surrounding code reviewed	Check adjacent code for the same class of bug	Prevents sibling bugs from surviving

Export

Copy this block into your CLAUDE.md, agent instructions, or project checklist.

Never let the agent fix a bug it cannot reproduce.

vcb_chapter: 20
title: "Fixing A Bug"
key_line: "Never let the agent fix a bug it cannot reproduce."
thesis: "A bug fix from an AI model must include a test that would have caught the bug before the fix. Without that test, the fix is a guess with temporary success."
checklist:
 - item: "Write a failing test before asking for a fix"
 protects: "Proves you understand the bug before generating a fix"
 - item: "Confirm the test fails before any fix is applied"
 protects: "Prevents testing the wrong condition"
 - item: "Classify the bug: model capability, specification, parser/interface, validator, state, authorization, or commit policy"
 protects: "Determines whether the fix must move a rule into a durable control"
 - item: "Check the rejection log and escape log for related entries"
 protects: "Surfaces whether the gap was already visible before this incident"
 - item: "Ask for root cause, not symptom suppression"
 protects: "Prevents cascading bugs from hidden structural failures"
 - item: "Verify the reproduction test passes after the fix"
 protects: "Proves the fix addresses the actual bug"
 - item: "Run the full test suite after the fix"
 protects: "Catches regressions introduced by the fix"
 - item: "Add a runtime invariant for validator, state, authorization, or commit-policy bugs"
 protects: "Blocks recurrence through code paths the unit test does not exercise"
 - item: "Commit the regression test with the fix"
 protects: "Creates permanent protection against the same bug"
 - item: "Review adjacent code for the same class of bug"
 protects: "Prevents sibling bugs from surviving in nearby code"