The Test Passed With the Code Deleted

I was about to let an AI decompose the core module of a service I own. Before that, I did the responsible thing and checked the test suite. It was green. Around 650 tests, all passing. That is the safety net you are supposed to have before a big refactor.

Then I looked closer at three of those test files. They reimplemented the production logic locally, inside the test, and never imported the module they claimed to test. They would have passed if I deleted the production code entirely. Coverage counted them. The module they were supposed to guard looked tested, and it was not.

That is the moment I stopped trusting green and started mutation-testing the code I was about to hand to an AI.

Why this bites harder with an AI

When I refactor by hand, the diff is small and I wrote every line of it. I have a mental model of what changed and what could break. The test suite is a backstop, not my only line of defense.

When an AI refactors, the shape is different. It touches many files at once and moves faster than I read. I go through every diff, but I cannot hold a whole module’s worth of changes in my head the way I hold thirty lines I wrote myself. The suite stops being a backstop and becomes the primary check. Green has to actually mean something, because green is most of what I have.

There is a second problem that is specific to AI. When a test fails during a refactor, the AI is very good at making it pass. Sometimes it makes it pass by fixing the code. Sometimes it makes it pass by weakening the test: loosening an assertion, mocking the thing under test, or reimplementing the logic inside the test so the test and the code always agree. An AI optimizing for a green suite can produce tests that are green by construction. Those tests test nothing, and they look exactly like the real ones in the CI output.

So I have a suite I did not fully write, guarding a refactor I cannot fully eyeball, in a loop where the same agent can lower the bar to pass. Green is not enough. I need to know the tests would fail if the code were wrong.

Why coverage does not answer that

The instinct is to reach for coverage. Coverage was already high on the module I wanted to refactor. It did not help, because coverage answers a weaker question than the one I care about.

Coverage measures which lines executed while the tests ran. It does not measure whether any assertion would fail if those lines did the wrong thing. A test that calls a function, ignores the result, and asserts nothing gives you 100% coverage of that function and zero protection. The three fake tests I found were covered. Every line they “tested” showed up green in the coverage report, because the coverage report only knows the lines ran, not that the lines were checked.

Coverage tells you the code was visited. It does not tell you the code was verified. For a refactor, verified is the only property that matters.

A one-paragraph primer

Mutation testing is an old idea, from the 1970s, with a plain premise: your tests check your code, so what checks your tests? The answer is to break the code on purpose and see if the tests complain. If you can introduce a bug and every test still passes, the tests were not really testing that part. Normal testing runs correct code and expects green. Mutation testing runs deliberately broken code and expects red. When the red does not come, you have found a gap.

What mutation testing actually asks

Mutation testing asks the question coverage cannot: if the production code were wrong, would a test notice?

The tool takes your production code and introduces small changes, one at a time. It flips a > to a >=, a + to a -, a True to a False, deletes a line, swaps an and for an or. Each changed version is a mutant. For each mutant, it runs your test suite.

• If some test fails, the mutant is killed. Good. A test noticed the code was wrong.
• If every test still passes, the mutant survived. That is a change to your production logic that no test caught.

A surviving mutant is the exact thing you are afraid of during a refactor: a behavior change that ships green. The count of survivors is a direct measure of how much your suite is pinning down, as opposed to how many lines it happens to run.

In Python I used mutmut. The loop is simple:

mutmut run --paths-to-mutate src/core_module.py
mutmut results        # list survivors
mutmut show <id>      # show the exact mutation that survived

A survivor looks like this. The tool changed a boundary and nothing broke:

--- src/core_module.py
+++ src/core_module.py
@@ retry logic
-    if attempts >= max_attempts:
+    if attempts > max_attempts:
         raise RetryExhausted()

The suite passed with > instead of >=. That means no test exercises the exact boundary where attempts == max_attempts. In the current code it might not matter. The moment an AI rewrites this block, that off-by-one is unguarded, and it will ship green.

The workflow I used before refactoring

I did not run mutation testing on the whole codebase. It is too slow for that, and most of the codebase was not what I was about to change. I ran it on the one module I was about to hand to the AI, and I ran it before the refactor started.

1. Pick the blast radius. The core module and its direct collaborators. The code the refactor would rewrite.
2. Run mutmut on that module as it exists today. This is the baseline. It tells me how real the current tests are before anything changes.
3. Read the survivors. Each survivor is either a missing test or a mutation that does not change behavior (more on those below). The missing tests are the important ones.
4. Kill the survivors by adding tests, not by changing the code. This is the step that makes the suite a real safety net. Every test I add here pins one more behavior that the refactor now has to preserve.
5. Only then let the AI refactor. Now green means something. If the AI changes behavior, a test that used to kill a mutant now fails on the real change.

The order matters. If you mutation-test after the refactor, you are measuring the new code’s tests, which the AI may have written to pass. If you mutation-test before, you harden a suite you trust against code the AI has not touched yet, and you carry that hardened suite into the refactor as the thing that has to stay green.

Killing a survivor is usually small. For the off-by-one above:

def test_retry_stops_exactly_at_max():
    handler = RetryHandler(max_attempts=3)
    handler.attempts = 3
    with pytest.raises(RetryExhausted):
        handler.check()

That test fails against the > mutant and passes against the real >=. The boundary is now pinned. The refactor cannot move it without CI going red.

What doesn’t work

• It is slow. Mutation testing runs your suite once per mutant, and there are a lot of mutants. On a single core module it is minutes. On a whole codebase it is hours or worse. Scope it to the code you are about to change, not everything you own.
• Equivalent mutants are noise. Some mutations do not change behavior at all, so no test can kill them, and they show up as survivors forever. A classic case is a mutation inside a log message or a change that another line makes irrelevant. You read them, confirm they are harmless, and move on. There is no fully automatic way to tell an equivalent mutant from a real gap; a human has to look.
• It does not write good tests for you. It tells you a behavior is unguarded. It does not tell you what the behavior should be. If you do not understand the code, a survivor just tells you where you are ignorant, not how to fix it.
• A high kill rate is not proof of correctness. Mutation testing checks that your tests pin the current behavior. If the current behavior is wrong, mutation testing will help you pin the bug just as tightly. It protects a refactor from changing behavior; it does not tell you the behavior was right to begin with.
• It is not a gate you run on every commit. It is too slow and too noisy for that. I treat it as a pre-refactor audit and an occasional health check on the critical modules, not a CI step that blocks merges.

The coding agent is what makes this practical

Read that list again. Almost every limitation is human work, and human work is exactly what a coding agent replaces. Mutation testing on its own is a good idea that most teams never adopt, because the cost is a person reading hundreds of survivors and hand-writing a test for each one. Pair it with a coding agent and that cost mostly goes away.

Go limitation by limitation. It is slow: the agent runs it in the background and waits, so the wall-clock cost was never really CPU, it was my attention, and the agent has attention to spare. Equivalent mutants are noise: the agent triages each survivor and tells me which ones are real gaps and which are harmless, and I review its calls instead of doing the first pass myself. It does not write tests for you: a coding agent does, and it does it well, because a survivor is a precise instruction. “Write a test that kills this mutant” is a far better prompt than “write good tests.” The mutant names the exact behavior, the boundary, the line. The agent writes the test, reruns mutmut, and confirms the mutant is now dead.

The one limitation the agent does not remove is judgment. Mutation testing pins current behavior; it cannot tell you the behavior was right to begin with. Neither can the agent. That call stays with me. So the division of labor is clean: the tool finds the gaps, the agent does the toil, and I spend my remaining attention on the one question that actually needs a human, which is whether the behavior being pinned is the behavior I want.

There is a deeper reason the combination works, and it closes the loop I opened earlier. The risk with letting an agent write tests is that it games a fuzzy target. “Make the suite green” is gameable: weaken an assertion, mock the thing under test, reimplement the logic inside the test. That is exactly how a test ends up passing with the code deleted. “Kill this mutant” is not gameable in the same way. The mutant is generated from the production code, not from the test, so a fake test that asserts nothing will not kill it. The survivor stays a survivor. Mutation testing gives the agent a target it cannot fake, which is the only kind of target you can safely point an agent at. The tool makes the agent trustworthy; the agent makes the tool practical. Neither half is the point. The loop is.

The reframe

Coverage tells you a line of code ran while the tests were watching. Mutation testing tells you the tests would have failed if that line were wrong. Those are different claims, and only the second one is a safety net.

The distinction was easy to ignore when I wrote every refactor by hand, because I was my own second check. Handing the work to an AI removed that check. The diffs got too big to fully hold in my head, and the same agent that refactored the code could also rewrite the tests that were supposed to catch it. The suite became the whole safety net, so the suite had to be real.

A test that passes when you delete the code it tests is not a test. It is a green light wired to nothing. Mutation testing is how I find those, and a target the agent cannot fake is how I fix them at the scale an AI refactor needs. I run both before I let an AI near anything I cannot afford to break.