Mutation Testing
A mature test adequacy verification technique that automatically injects small semantic changes (mutants) into source code, then runs the test suite to measure the proportion of mutants that are "killed." The approach identifies coverage blind spots in a test suite and traces its origins to the 1970s.
Core Facts
Timeline (last 90 days)
In an informal report, 20 AI tests written for a Laravel controller claimed to cover all endpoints, but among 95 injected mutants only 68 were killed, with 27 mutants surviving.
The author maintains a free experiment repository; running `node run-lab.js weak` generates AI tests for deliberately wrong code and they all pass, while running `node run-lab.js verify-candidate` fails and catches all 4 pre-planted bugs.
The way to identify a "check that is structurally impossible to fail" is to ask: if the thing I care about did not happen, would this check still pass?
Mutation testing is stricter than line coverage but has higher computational overhead, because each mutant requires running the tests again.
Mutation Score is the proportion of killed mutants out of the total number injected.
On a test set of 100 tickets, SupportOps-AI had an escalation error rate of 18%, a hallucination rate of 6%, an infinite-loop rate of 3%, and a complete failure/unresolved rate of 4%.
The mutation score in mutation testing measures the proportion of mutants a test suite can kill, and is the direct source of QTestBench's test quality score.
The 'must-fail test case' proposed in the article is in the same vein as mutation testing, but foregoes the complexity of automatically generating mutants, making it the lightest and lowest-cost form of that approach.
Mutation testing works by automatically injecting small semantic changes into source code (mutants, such as changing > to >= or + to -), running the test suite, and counting how many mutants are 'killed'; surviving mutants indicate blind spots in the test suite's coverage.
Mutation Testing is a test adequacy verification technique that dates back to the 1970s.
1 more timeline events
All Facts (12)
Mutation Testing checks whether test cases are truly effective by deliberately introducing defects into the code.
65%UnverifiedThe way to identify a "check that is structurally impossible to fail" is to ask: if the thing I care about did not happen, would this check still pass?
50%UnverifiedMutation testing is stricter than line coverage but has higher computational overhead, because each mutant requires running the tests again.
50%UnverifiedMutation Score is the proportion of killed mutants out of the total number injected.
50%UnverifiedIn an informal report, 20 AI tests written for a Laravel controller claimed to cover all endpoints, but among 95 injected mutants only 68 were killed, with 27 mutants surviving.
50%UnverifiedOn a test set of 100 tickets, SupportOps-AI had an escalation error rate of 18%, a hallucination rate of 6%, an infinite-loop rate of 3%, and a complete failure/unresolved rate of 4%.
50%UnverifiedThe mutation score in mutation testing measures the proportion of mutants a test suite can kill, and is the direct source of QTestBench's test quality score.
50%UnverifiedThe 'must-fail test case' proposed in the article is in the same vein as mutation testing, but foregoes the complexity of automatically generating mutants, making it the lightest and lowest-cost form of that approach.
50%UnverifiedMutation testing works by automatically injecting small semantic changes into source code (mutants, such as changing > to >= or + to -), running the test suite, and counting how many mutants are 'killed'; surviving mutants indicate blind spots in the test suite's coverage.
50%UnverifiedMutation Testing is a test adequacy verification technique that dates back to the 1970s.
50%UnverifiedThe theory of mutation testing can be traced back to a 1971 student paper by Richard Lipton.
50%UnverifiedThe author maintains a free experiment repository; running `node run-lab.js weak` generates AI tests for deliberately wrong code and they all pass, while running `node run-lab.js verify-candidate` fails and catches all 4 pre-planted bugs.
50%