October 29, 2026

90% coverage and the bug still shipped: fixing your signal-to-noise

Coverage counts line executions, not correctness. A 90% score built on weak assertions is noise. Here's how to measure signal instead.

Short answer: coverage counts line executions, not correctness. A suite can touch 90% of your code and verify almost none of it — the number measures which lines ran, not which behaviors are pinned. Fix the assertions, then the score will mean something.

A team I know spent six months pushing their coverage from 65% to 92%. QA signed off. Two days after the next release a discount calculation started giving customers double the expected reduction on edge-case cart sizes. The line in question was covered. The test even called it. There was just no assertion that the result was correct — the test called the function, threw away the return value, and checked that no exception was raised.

Ninety-two percent. Shipped. Caught by a customer.

Why does high coverage still let bugs through?

Three patterns account for most of the gap between coverage and actual correctness.

Pattern 1: the assertion that asserts nothing. This is the most common. The test invokes the code path, but the check is so weak it can’t catch a bug:

def test_apply_discount():
    result = apply_discount(cart, code="SAVE20")
    assert result is not None        # will pass if function returns 0
    assert isinstance(result, dict)  # will pass if the dict is empty

The line coverage tool sees both lines inside apply_discount as executed. From its perspective: covered. From a correctness perspective: nothing was checked.

Pattern 2: the covered path that never tests the hard branch. Statement and line coverage aggregate. They hide which branch ran.

def apply_discount(cart, code):
    if code in VALID_CODES:
        discount = cart.total * DISCOUNT_RATES[code]
        return {**cart.__dict__, "total": cart.total - discount}
    return cart.__dict__  # no discount applied

If your tests only call this with invalid codes, the discount logic is “covered” because the function ran — but the branch that computes discount never executed. Line coverage: 100%. The discount math: never tested.

Branch coverage catches this one, but most teams don’t enforce it. And even branch coverage doesn’t help with Pattern 1 — you can have 100% branch coverage with zero meaningful assertions.

Pattern 3: the mirror assertion. The test derives its expected value from the same computation it’s testing:

def test_cart_total():
    items = [Item(price=10), Item(price=20)]
    cart = Cart(items)
    # asserting the function equals itself
    assert cart.total == sum(i.price for i in cart.items)

This is a tautology. It can never fail, because the assertion re-runs the same arithmetic. Your coverage counts it as executed. A bug in the price accumulation logic would pass this test forever.

What does signal actually look like?

Signal is an assertion that would fail if the behavior changed. The cleanest way to measure it: mutation score.

A mutation tool makes tiny, realistic changes to your source code — flips >= to >, changes + to -, replaces a return value with None — then runs your test suite against each mutant. If the suite stays green, the mutant survived. A survived mutant is a bug your tests would have missed.

# Python: pip install mutmut
$ mutmut run --paths-to-mutate=discount.py

$ mutmut results
Killed: 31  Survived: 12  Timeout: 0
Mutation score: 72%

$ mutmut show 4
--- discount.py
+++ discount.py
@@ -7,7 +7,7 @@
-    discount = cart.total * DISCOUNT_RATES[code]
+    discount = cart.total / DISCOUNT_RATES[code]

Twelve surviving mutants means twelve bugs your suite would not catch. Each one is a specific line and a specific kind of mistake your assertions are blind to. That’s the to-do list — not “write more tests” but “these twelve behaviors are unprotected.”

The cart discount bug above would appear as a survived mutant on the arithmetic line. There’s no way to hide it from mutation testing. Coverage hides it by design.

How do you raise mutation score without drowning in false work?

Don’t run mutation testing across your whole repo at once. The runtime is proportional to (lines of code × mutation operators), and if you do it wrong you’ll turn it off after day one.

The practical workflow:

1. Identify your three riskiest modules. Payments, auth, anything that costs money or leaks data when wrong.
2. Run mutation testing on those modules only. A focused 90-second run beats a 45-minute full-repo run you won’t repeat.
3. Work the survivor list, not the score. Every surviving mutant points at a specific assertion hole. Write the assertion; the score improves as a side effect.
4. Gate CI on mutation score for risk modules. Not 100% — equivalent mutants exist (changes that don’t actually alter behavior). 70% is a reasonable floor to start; raise it as you fix things.

# Add to your CI config, scoped to the module that matters
$ mutmut run --paths-to-mutate=src/billing/
$ mutmut results  # fail the build if mutation score < 70%

For JavaScript and TypeScript, Stryker is the equivalent:

$ npx stryker run --files src/billing/**/*.ts
# stryker.conf.json: set "thresholds": { "break": 70 }

Both tools integrate with most CI systems in under an hour. The configuration cost is front-loaded; once it’s in the pipeline it runs automatically.

What does this solve that branch coverage doesn’t?

Branch coverage tells you a branch ran. Mutation score tells you a branch is tested. The distinction:

You can have 100% branch coverage and a 40% mutation score. It happens constantly with generated test suites, because the generator sees branches and writes tests that reach them, not tests that verify them.

What this doesn’t fix

Mutation testing measures how well your assertions pin current behavior. It says nothing about whether current behavior is correct. If the spec was wrong and the code faithfully implements the wrong spec, mutation score will be high and bugs will still ship — just different bugs.

It also doesn’t help with non-deterministic AI outputs. Testing whether an LLM’s answer is faithful to source documents is a different class of problem; eval suites and property checks handle that, and the same signal-to-noise framing applies there too.

The coverage number can stay. Just stop using it as a proxy for confidence. Mutation score is the thing worth gatekeeping.

If this gap — the distance between “lines ran” and “behavior verified” — is something your team is hitting, the free test lab has a working example: a green 94%-coverage suite sitting on four real bugs, and the mutation run that surfaces each one. AI Software Testing in Practice goes deeper on the full quality pyramid and how to make coverage numbers actually mean something.

FAQ

Why does 90% coverage still let bugs through? Coverage tracks whether a line ran, not whether any assertion checked its output. You can execute 90% of your code and assert almost nothing — every line touched, zero behavior verified.

What should I measure instead of line coverage? Mutation score. Run a mutation tool (mutmut for Python, Stryker for JS/Java/C#), and check how many injected bugs your suite kills. That number tells you how much of your code’s behavior is actually pinned down by assertions.

How do I fix the signal-to-noise ratio in an existing test suite? Run mutation testing on your riskiest module first. Every surviving mutant is a missing assertion. Fix those specific gaps, not your coverage number. Coverage follows from real assertions; real assertions don’t follow from coverage.

← All guides