Mutation testing is amazing. It still shocks me that it has not caught on despite being around for decades. This post is in regards to the value add that comes from doing mutation testing.
Don't know what mutation testing is? Check out these posts:
Mutation Testing 101
A mutation is a single syntactic change to the code. Each mutant is different from the original by a single mutation. These small changes can have big results; such as changing an operator from addition to subtraction. Then your unit tests execute against the mutant code and in theory our tests should fail because the code is wrong.
public int additionOrSubtraction(boolean isAddition, int x, int y) {
if (isAddition) {
return x + y;
} else {
return x - y;
}
}
Lesson #1: Mutation Testing Improves Code Reviews
Do you do code reviews? I bet they focus 95% on the code that will go to production and maybe a quick peek at the tests. Despite the fact that a huge portion of the code base are unit tests, we really don't review them often due to there being a finite amount of time. Mutation testing came to my attention when searching for a method to improve the code review process. With mutation testing, the framework adds to the metrics to measure the quality of the tests suite. Metrics provide an unbiased way of illustrating the tests that add value and the tests that just provide a false sense of security.
Lesson #2: Not All Mutations Add Value
Some mutations are just silly and getting 100% mutation test coverage can be more trouble than it is worth. The most common example of this I have seen are mutations that remove log statements. Verifying logging statements add little to no value in 99.9% of cases. The PIT mutation engine does not differentiate between the different types of void method calls so logger calls are fair game. These failures can be ignored and maybe the case can be made to try and exclude them from the remove method call mutator.
Lesson #3: Start Small
Mutation testing is very CPU and time intensive. For example, in a
previous post, I mentioned how a build that normally takes 30 seconds to compile and run all the unit tests took over 9 minutes with mutation testing. The first place I start is actually in Eclipse and run a smaller test suite using
PitClipse. This way, you can see if your tests even can withstand a mutant onslaught. Also the
mutationUnitSize configuration point of PIT allows for the slower ramp up of the number of mutations that are right for your project:
> mvn org.pitest:pitest-maven:mutationCoverage -DmutationUnitSize=2
Lesson #4: Laziness Creates Surviving Mutants
This lesson was learned from the same mutator that caused the example in lesson #2. Not verifying the results of void method calls causes more mutations to fail than almost anything else. In many cases, this is just due to pure laziness. I just found a project where all builder method calls to populate POJOs only had an assert to make sure the result was not null. So the tests did not care what was populated in those beans and just focused on them not being null. This is a perfect example of having 100% code coverage but the tests themselves being worse than if they did not exist at all. Do yourself a favor and verify every element in the objects returned. I know it takes time and many asserts but that extra effort may save you from a production bug born out of laziness.
Lesson #5: People Won't See the Value of Mutation Testing at First and That is Okay
Mutation testing sounds interesting to people as a theoretical exercise but the benefits of it are often hard to see. It takes a mature organization that is already dedicated to a strong unit test base to even consider adopting it. To get it adopted you, yes you, need to be the zealous advocate for mutation testing. Just do it. Update your builds and put a mutation testing job in your continuous integration tool like Jenkins. If your team hopes to do continuous delivery, you need a rock solid test base. Start pointing out mutations that result from recent commits just like you would if the commits result in bad code coverage. Do not try to convince people of the value of mutation testing but instead show them the lack of defects that could result from weak tests.
Lesson #6: Mutation Testing is Not Just for Java
Although the mutation testing tool I am outspoken about is for Java, that doesn't mean that there are no frameworks for other languages.
(I have not used any of the following tools and am therefore unable to attest to their value.)
Lesson #7: Be Relentless in Advocating for Mutation Testing
Mutation testing has been around since 1971. The days have long passed when the computer power is not available to make mutation testing a viable tool. If you don't push for it to be adopted, it likely won't because we haven't reached that critical mass to give it a life of its own. So do it and email out the reports. Do a lightning talk or brown bag session on mutation testing to get the word out. Honestly, I don't care if you even need to borrow heavily from or steal the Mutation Testing Slideshow that I recently created. Get the word out.
Lesson #8: 100% Mutation Kill Score is Possible But Expensive
As previously stated, some mutations are silly, not really useful but there are times when there is no margin for error and your code needs to work. Say for example, you are working on code for a medical device. There is no room for error when someone's life is on the line. You need mutation testing to do everything you can to be defect free in cases such as that. However some mutants just aren't worth the time needed to kill them. It is similar to writing unit tests to test all getters and setters in POJOs. Is it really worth your time to write tests for basic getter and setter methods. I don't know that is a personal decision but squashing some mutants might add even less value than those unit tests.
Lesson #9: Mutation Testing Makes You A Better Programmer
Knowing the ways your code could break and how good your tests need to be to protect against that is not a skill that is developed overnight. Mutation testing ensures that you don't cut corners and take the easy route. Write solid code and bombproof unit tests.
Links to Other Posts Related to Unit Testing and Mutation Testing
