Open the last pull request where the assistant wrote the tests. In the code they cover, find any comparison — a >, a >=, an == — and flip it. Don't touch the tests. Run the suite.
If it stays green, you now know what your coverage measures.
The experiment costs ten minutes, needs nothing installed, and has a property almost no quality metric has: it admits no argument about methodology, because everyone watched it happen on the same screen. It is also the hand-rolled version of a technique with forty years of literature behind it — mutation testing — which is what this piece is about.
The point is not that generated tests are bad. It is that they are being read as answers to a question they do not answer. We have already looked at what happens to review when the author stops existing as someone you can ask, and at the dependencies that arrive disguised as a single line. Tests are the third thing riding in on the same pull request, and the one wearing the best disguise — because it arrives with a number that goes up.
Four conditions, and coverage checks only the first
For a fault in a line of code to end up showing as a red test, four things have to happen in sequence. The formulation is classic in the testing literature, and Parsai and Demeyer summarise it this way in their industrial study: the faulty statement must be executed; it must infect the program state; that infection must propagate to the output; and the test must reveal it by observing that output and deciding it is wrong.
Four links. Coverage measures the first. Only the first.
| Condition | What has to happen | Where a test written from the code fails |
|---|---|---|
| Reachability | The faulty statement is executed during the test | Almost never. This is exactly what the model is good at: reaching every branch. This is where the number goes up. |
| Infection | Executing it leaves the program state at a value other than the correct one | Sometimes. It depends on whether the chosen inputs touch the boundary where the defect changes anything, and generated values tend towards the central case. |
| Propagation | That incorrect state reaches an observable output | Sometimes. Heavily unit-scoped tests with doubles everywhere cut propagation off before anything surfaces. |
| Reveal | The test observes the output and decides it is wrong | This is where it breaks. The assertion was written by looking at the output the code produces today. If that output is wrong, the assertion certifies it. |
The four conditions, in the formulation recorded by Parsai and Demeyer. The third column is our own reading of how a test derived from the code typically behaves.
Put that way, it is clear why the number can rise without the net improving. A test that executes a line and checks nothing meaningful about what that line produced counts the same towards coverage as one that does. The coverage tool does not distinguish between them: it is not designed to. PIT's own documentation puts it bluntly — line coverage "measures only which code is executed by your tests. It does not check that your tests are actually able to detect faults in the executed code".
An oracle inherits whatever it was born from
The part of a test that decides whether something is right or wrong has its own name in the literature: the oracle. It is the assertion, and it is the only part doing verification work; everything else is scaffolding. So the useful question is not who wrote the test, but where its oracle came from.
Here is the finding that organises everything else. Bodicoat, Jahangirova and Terragni open their January 2026 study by noting that existing test generation techniques "primarily generate regression oracles that predicate on the implemented behavior of the class under test" and that they therefore "do not address the oracle problem: the challenge of distinguishing correct from incorrect program behavior".
A regression oracle is a statement about the current state. It says: this does this today. That is enormously useful — it is what warns you a refactor changed something by accident — but it is a photograph, not a judgement. If the code had a fault the day the test was generated, the test photographs it along with everything else and protects it from then on. The suite stops being a net and becomes a mirror: it confirms the code is still the code it was.
There is a much-repeated frustration that fits here and is almost always told in another context. In Stack Overflow's 2025 survey, with more than 49,000 responses, the number one complaint about AI tools — 66%, well ahead of the second — is "AI solutions that are almost right, but not quite". In a fragment of logic, "almost right" shows: it fails, or does something odd, and somebody looks at it. In an assertion, "almost right" is green. A wrong oracle does not behave like an error. It behaves like a confirmation.
Meta published the whole funnel, and it is the most honest figure available
It is worth looking at the best documented case before forming a view on the average one. In 2024 Meta described TestGen-LLM, a tool that does not generate tests from scratch but improves existing human-written ones and — this is the relevant part — only accepts those that clear a chain of filters guaranteeing a measurable improvement over the original suite. The stated motivation for those filters is to eliminate the problems caused by model hallucination.
In the evaluation on Instagram's Reels and Stories products, the funnel came out like this: 75% of generated test cases built correctly, 57% passed reliably and 25% increased coverage. Across the Instagram and Facebook test-a-thons they improved 11.5% of all classes it was applied to, and Meta's engineers accepted 73% of its recommendations for production.
One in four. With a team that knows what it is doing, an automatic filter in front and Meta's scale behind it. That number is the realistic floor of this conversation, and it is good news: it means that with a filter, this works — and the 73% human acceptance rate confirms it.
But the three filters are worth reading for what they ask. Does it build? Does it pass reliably? Does it increase coverage? Three reasonable questions, and not one of them is would this test catch a fault?. The strictest filter published by industry stops, by design, at the first of the four links. That is not an oversight on Meta's part — coverage is measurable, automatic and cheap, and it is infinitely better than accepting whatever the model says — but it marks where the ceiling of the approach currently sits.
Why coverage was never going to answer this, and we have known since 2014
The reference work is by Laura Inozemtseva and Reid Holmes, presented at the International Conference on Software Engineering in 2014. They generated 31,000 test suites across five real Java systems — Apache POI, Closure Compiler, HSQLDB, JFreeChart and Joda Time, up to 724,089 lines of source code — measured their statement, decision and modified condition coverage, and evaluated their real effectiveness by injecting faults through mutation testing.
Three conclusions, all three still uncomfortable twelve years on. The correlation between coverage and effectiveness is low to moderate once the number of test cases is controlled for. The strength of that relationship varies between systems, so it is not safe to assume it holds in yours. And stronger forms of coverage provide no greater insight than the simple one, despite costing more to measure and to satisfy.
The sentence they close on is the one that belongs pinned to any quality dashboard: coverage, while useful for identifying under-tested parts of a program, should not be used as a quality target, because it is not a good indicator of test suite effectiveness. The systems they studied ranged from 27% statement coverage in HSQLDB to 91% in Joda Time, and the number alone did not rank the suites by detection capability.
None of this is new and none of it is about AI. What has changed is volume. When writing tests cost human time, coverage grew slowly and roughly in proportion to the thought invested. That accidental coupling between the number and the effort was what made the metric appear to work. Generating tests at scale breaks the coupling, and leaves the metric exposed for what it is.
The measure that does move
Mutation testing answers precisely the question coverage does not. The procedure, in PIT's own formulation: faults are automatically seeded into your code and your tests are run; if the tests fail, the mutation is killed; if the tests pass, the mutation lived. The proportion of killed mutants is the mutation score, and it measures detection capability rather than surface covered.
Stryker's documentation illustrates it with the case closest to the one that opened this article: an age check. If flipping the condition turns no test red, there is probably a test missing that explicitly covers the boundary. A surviving mutant is not an abstract metric: it is a missing test, named and pointed at.
The important property is that this number cannot be inflated by tests that execute code without checking anything. A test with no useful assertions raises coverage and kills not a single mutant. It is, by construction, immune to the failure mode that mass generation brings.
| Coverage | Mutation score | |
|---|---|---|
| What it asks | Which code do my tests execute? | Would my tests catch a change in that code? |
| Links covered | The first | All four, by construction |
| Can be inflated by | Tests with no useful assertions | Nothing equivalent |
| Cost to measure | Seconds; already in your CI | High: a full pass is weekend work |
| Its own noise | Low | Equivalent mutants: 0.4% to 35% depending on the project |
| What to use it for | A map of gaps | The exam mark, on the code that matters |
The equivalent-mutant range comes from the five systems measured by Inozemtseva and Holmes: 0.4% in HSQLDB, 35% in Apache POI.
What it actually costs, without the varnish
Mutation testing has not been widely adopted for a solid reason: it is expensive. Parsai and Demeyer set out to measure exactly that across four open source systems and one industrial one, investigating the three standard objections: that it is difficult to integrate into the build system, that branch coverage is already "good enough", and that the time overhead is unaffordable.
The industrial numbers: 12,825 mutants for 38,000 lines of code. They defined an acceptable job as one that fits between Friday 6pm and Monday 8am — 62 hours — so results are ready before the sprint meeting. When the full pass does not fit, they apply mutant sampling down to 34.7%, citing prior work showing that samples as low as 50% still give reliable results. Their conclusion is that mutation coverage reveals additional weaknesses that branch coverage does not see, at an acceptable overhead.
Two honest warnings come out of that. The first is that this is not free and does not belong on every commit: the way to fit it into a short cycle is to mutate only the classes the diff touches. The second comes from the authors themselves and cuts against the comfortable intuition: they did not find sufficient evidence to confirm that branch coverage is a good estimator of mutation coverage in complicated systems. In other words, you cannot use coverage as a cheap proxy for the number that does matter either.
And the technique's own noise is real: equivalent mutants — syntactic changes that do not alter behaviour and that therefore no test can detect — ranged from 0.4% to 35% depending on the project in Inozemtseva and Holmes's study. Telling them apart requires manual inspection. Anyone selling mutation score as a clean, automatic metric is selling something else.
The lever is not the model. It is the input
If an oracle inherits whatever it was born from, there is an obvious intervention before touching anything else: change what it is born from. Instead of handing the model the implementation and asking for tests, hand it what the code was supposed to do.
Ma and Eisty tested this in July 2026 in the cleanest way available: generating oracles purely from natural-language business requirements, with no access to source code or example inputs, and putting them up against ten real bugs from the Defects4J Lang project across five different models. The result that matters is directional: the generated oracles align more closely with the requirement than with the system's implemented behaviour.
It is a small study — ten bugs, five models, one project — and should be treated as a signal rather than a closed demonstration. But it points the same way practice does: the model has no way of knowing an output is wrong if the only thing you have shown it is that output. Give it the criterion and it can check against it; give it the result and all it can do is copy.
This is exactly the Spec-Driven Development argument, applied to the side almost nobody looks at. When a written, versioned, agent-readable specification exists, it does not only serve to generate the code: it is the only material a test that judges rather than describes can come from. What context engineering solves for generation, it solves for verification too — and for the same reason, because the problem was never the model's ability but what it had in front of it.
It is the same logic we already apply to evaluating AI systems: a golden set is worth something because someone with judgement fixed the correct answers before seeing what the system produced, which is why it can say whether something works better rather than merely whether it changed. A test is a golden set for one function. Generate it by looking at the output and you have built the evaluation set out of the candidate's own answers.
The order in which this gets fixed
Three decisions, and the order matters more than any of them individually.
First, measure once. One class, the experiment from the top of this article, ten minutes. Not to get a representative figure, but to turn an argument about opinions into a shared observation. As long as the conversation is "I think our tests are weak", nothing moves; the moment it becomes "I flipped this sign and the suite passed", it moves on its own.
Second, separate the two jobs. The generated suite stays where it is and does what it is good at: catching unintended changes, which is precisely what refactoring with AI requires. On top of that, a small set of tests written from judgement — the boundaries, the business rules, what the specification says must happen — and that set is the one that judges. They are not in competition: one is the mirror and the other is the net, and you need both.
Third, change the target. If there is a coverage threshold on the dashboard, take it off the code where the business logic lives and replace it with mutation score on the diff. Coverage goes back to being what Inozemtseva and Holmes said it was: a map for finding what nobody touches. A coverage threshold over a generated suite is the one combination guaranteed to produce the worst possible outcome — the number rises by construction and tells you nothing.
None of this requires a new tool in week one. It requires no longer reading a number as though it answered a question it never answered, and that alone changes what gets asked in the next conversation about quality and what goes on the team's metrics dashboard. An assistant that writes tests is good news: it multiplies the scaffolding work, which was the boring part. You just have to stop asking it to be the judge as well, because for that it needs something only you have — knowing what the code was asked to do before it existed.
Code quality in the AI coding era has mostly been debated by looking at the code. It is worth looking at what watches over it too: if the suite grew at the same rate as the code, and neither of them was written by someone who knew what was supposed to happen, that green light every morning stopped meaning what you think it means a while ago.
Frequently asked questions
So coverage is useless?
It is useful for what it was always useful for: finding the code no test touches. That is real information and it is cheap to get. What it does not do is tell you whether the tests that do touch the code would catch a fault in it, because it measures only the first of the four conditions that must hold for a fault to be revealed. Inozemtseva and Holmes wrote this down in 2014 after generating 31,000 suites across five Java systems of up to 724,089 lines: the correlation between coverage and effectiveness is low to moderate once the number of test cases is controlled for, and their conclusion is that coverage is useful for identifying under-tested parts of a program but should not be used as a quality target. The practical difference: use it as a map of gaps, not as the exam mark.
Isn't mutation testing far too slow for CI?
To run in full on every commit, yes. Parsai and Demeyer measured it on a real industrial system: 12,825 mutants for 38,000 lines of code, with a time budget defined as the window between Friday 6pm and Monday 8am. It is weekend work, not pull request work. The way to fit it into a short cycle is not to run it in full: mutate only the classes the diff touches, and use mutant sampling when even that does not fit — the same authors cut the sample to 34.7% while preserving the information, and they cite work where samples of 50% still give reliable results. The ten-minute version is doing it by hand on a single class, which is how it is worth starting before setting anything up.
If the model writes the test from the specification, doesn't it end up just as biased?
Biased towards something else, and that is the whole difference. A test derived from the code can only assert what the code does; if the code is wrong, the test enshrines the error. A test derived from the specification asserts what was asked for, and when the two disagree the fault surfaces. Ma and Eisty measured this against ten real Defects4J bugs, generating oracles purely from natural-language requirements with no code in sight: the oracles align more closely with the requirement than with the implemented behaviour. It is a small study — ten bugs, five models — and it is worth treating as a direction rather than a closed proof. But the direction is what matters: an oracle inherits whatever it was born from.
Is it worth letting AI write tests at all, then?
Yes, with a filter in front of it, and without expecting that filter to be coverage. The honest figure was published by Meta with TestGen-LLM: on Instagram's Reels and Stories, 75% of generated test cases built correctly, 57% passed reliably and 25% increased coverage. One in four made it to the end of a funnel designed by people who know what they are doing — and even so, the three filters ask whether the test builds, whether it is stable and whether it executes new code; none of them asks whether it would catch a fault. With that caveat in place, the field result is good: they improved 11.5% of the classes it was applied to and Meta's engineers accepted 73% of its recommendations for production.
What do we do with the generated suite we already have?
Don't throw it away: measure it before deciding. A generated suite almost always contributes something real — it covers paths nobody had touched and it documents current behaviour, which is worth having when refactoring — and its problem is not that it is fake, but that it is being read as something else. The order that works is to measure the mutation score of two or three classes with genuine business logic, see which mutants survive, and hand-write only the tests that kill those survivors. There are usually few of them and they are usually the boundary conditions. What you should not do is set a coverage target for the generated suite: it raises the number without moving detection capability, which is precisely the failure you were trying to correct.
Where do I start this week without setting up a programme?
With the ten-minute experiment on a single class. Pick the class with the most business logic from the last quarter, change one comparison in the code — a sign, a boundary condition — without touching the tests, and run the suite. If it stays green, you have the conversation: no tool and no budget are needed to have it, and the result admits no methodological argument because everyone watched it on the same screen. From there, the decision about whether installing PIT or Stryker is worth it stops being theoretical. The mistake is starting by buying the tool: the figure that moves a team is the mutant that survived in their own code.
Sources cited
- Inozemtseva, L. and Holmes, R. — Coverage Is Not Strongly Correlated with Test Suite Effectiveness (ICSE 2014, University of Waterloo; 31,000 suites across five Java systems of up to 724,089 lines; low-to-moderate correlation controlling for size; equivalent mutants from 0.4% to 35%)
- Alshahwan, N., Chheda, J., Finegenova, A., Gokkaya, B., Harman, M., Harper, I., Marginean, A., Sengupta, S. and Wang, E. — Automated Unit Test Improvement using Large Language Models at Meta (February 2024; Instagram Reels and Stories: 75% built, 57% passed reliably, 25% increased coverage; 11.5% of classes improved and 73% of recommendations accepted)
- Bodicoat, A., Jahangirova, G. and Terragni, V. — Understanding LLM-Driven Test Oracle Generation (9 January 2026; existing techniques generate regression oracles over implemented behaviour and do not address the oracle problem)
- Parsai, A. and Demeyer, S. — Comparing Mutation Coverage Against Branch Coverage in an Industrial Setting (Flanders Make and Universiteit Antwerpen; four open source systems and one industrial; 12,825 mutants for 38,000 lines, 62-hour window, sampling to 34.7%)
- Ma, T. and Eisty, N. U. — From Business Requirements to Test Assertions: Evaluating LLM-Generated Oracles on Real Bugs (11 July 2026; ten bugs from the Defects4J Lang project and five models; oracles generated from requirements align more closely with the requirement than with implemented behaviour)
- PIT — Real world mutation testing (official documentation; line coverage "measures only which code is executed by your tests. It does not check that your tests are actually able to detect faults in the executed code")
- Stryker — Stryker Mutator documentation (StrykerJS, Stryker.NET and Stryker4s; the definition of killed and survived mutants and the boundary-condition example)
- Stack Overflow — 2025 Developer Survey · AI (more than 49,000 responses; "AI solutions that are almost right, but not quite" is the top frustration, at 66%)
Jordi García is Tech Lead at onext. He works on bringing AI into governed production across development and product teams —with Spec-Driven Development, context engineering and human verification at every step— and authors onext's technical insights on the method, quality and cost of applied AI.
LinkedIn →