Are your unit tests effective against code mutations? No, you don't need to worry about your code being bombarded by gamma rays or being bitten by a radioactive spider. The question is will your tests catch minuscule changes in the code base that could have disastrous affects. Mutation testing has been around since 1971 but the majority of people in the software world haven't heard of it. So congratulations, you have just learned something that can wow and amaze your peers at the lunch table.
Breaking code is easy
This may be earth shattering news but software engineers are not perfect. We make mistakes but that is why we test. The code written during a 2am coding bender all hyped up on Jolt Cola may be the stuff of legend that creates the next paradigm shifting application but one typo can result in catastrophe. Have you ever spent hours upon hours debugging a critical defect that resulted in something silly like the usage of the wrong operator or worse a missing semicolon?
100% code coverage != effective tests
Despite our best intentions, test code is not treated with the same respect as the rest of your code base. One hundred percent code coverage is a unicorn, really really rare, in complex applications but even that is not enough to ensure your code is rock solid. Even if your unit tests result in a perfect code coverage report, are you positive all your assert statements are effectively verifying the results? Mutant testing is essentially tests for your tests.
And you know what the worst part is?
I NEVER LEARNED TO READ TEST.
If you are taking the time to read this, then there is likely no need to convince you of how vital testing is. However, effective test writing is a skill/art/science but it is one of the many critical skills in the software engineering world that they likely did not teach you in college. Formal training on unit test writing is virtually unheard of. During code reviews, the test code is not given the same type of scrutiny as the production code. Yet the tests are just as critical to the long term success of your project.
So what is mutation testing?
Your tests ensure your application works as you expect it to but what ensures that your tests sufficiently protect your application. Mutation testing is effectively making minor changes to your source code and ensuring your tests catch it and none of those mutants survive. A mutant that survives could one day grow up to be a defect.
mutant survival score = number of mutants killed / total number of mutants
The creation of a mutant army
As previously discussed, mutants are brought into existence by sightly modifying your source code with something as simple as changing an equals into a not equals. Doing this for every conditional, arithmetic, relational, and assignment operator is just the beginning but you can see how the number of mutations even for a fairly simple piece of code can grow quickly.
public int additionOrSubtraction(boolean isAddition, int x, int y) {
if (isAddition) { return x + y; } else {
return x - y; }
}
The code above could result in 50 or more mutants which your unit tests would then be executed against. If any of those mutants do not cause a test failure, it "survives."
Creating a mutant army is not something you want to be doing by hand so look for tools available in your language of choice. Mutation testing is done almost exclusively with unit tests in mind. That is not to say that throwing a few mutants randomly against your integration tests would not be valuable too but throwing a fully mutant army against your functional tests would be absurdly costly.
Are you scared of the mutants yet?
Do not be alarmed. These mutants are here to save you. Send in the mutants! Save your tests. Your project could be one fat fingered key stroke away from disaster.
Find out more: Give your tests mutant powers with PIT [Part 1]
Post with my own permission as the original author. (Source)
