Mutation Testing Series
PIT has the ability to run mutation test with ant, maven, gradle, and more so it can easily be run via the command line and in your favorite build environment. The cost of entry into the mutation testing world is low for Java with PIT.
Sample Project
I created a little sample project to try mutation out which you are welcome to clone. It is nothing special but I wanted to play with it somewhere other than on an enterprise application. Mutation testing can take a while to run on large projects so I would recommend experimenting with the concepts on a smaller scale.
How to Add PIT to a Maven Project
Step 1: Update that pom to include the following (replacing the project base package of course):
<build>
<plugins>
<plugin>
<groupId>org.pitest</groupId>
<artifactId>pitest-maven</artifactId>
<version>1.1.5</version>
<configuration>
<targetclasses>
<param>com.bobbylough.MutationTestingDemo.*</param>
</targetclasses>
</configuration>
</plugin>
</plugins>
</build>
Step 2: Pat yourself on the back because your project is not setup for mutation testing.
How to Run PIT Mutation Tests
Running mutations tests are super simple from the command line.
> mvn org.pitest:pitest-maven:mutationCoverage
* One thing to note is the command above line assumes it is already compiled locally.
Depending on your project size, this can take a while to run. PIT does display an ASCII spinner so you know the mutation (slave) haven't stalled so you remain patient. In the sample project, I was explicit about where the mutation report should go by but by default they are saved in target/pit-reports/YYYYMMDDHHMI.
There are a lot of command line options but you can see them on the PIT website. One of the more interesting ones to me was the "mutators" option where you can specify which mutators to run against your code. There are some which are not used by default like "remove conditionals mutator" (REMOVE_CONDITIONALS).
> mvn org.pitest:pitest-maven:mutationCoverage -Dmutators=REMOVE_CONDITIONALS,MATH
Although the most useful initially may be the "mutationUnitSize" option. It will help to ensure your mutant army doesn't grow too large by specifying the maximum number of mutations created per analysis.
> mvn org.pitest:pitest-maven:mutationCoverage -DmutationUnitSize=2
My Experience On A Real Project
The small, but real enterprise project I started using this on had about 800 unit tests and ended up generating around 2,500 mutations. The test build which normally takes about 30 seconds now took almost 9 minutes with mutation testing. So, the cost of doing mutation testing is nothing to sneeze at but it is worth it. I would not recommend mutation testing to be done as part of a continuous integration but rather as part of a nightly build to generate the reports. The project used for this initial test has 98% line and 100% conditional coverage. (Yes, as previously admitted, I am a bit obsessive about unit testing.) However, much to my embarrassment, the mutation coverage was only 88%.
The mutation testing results showed the value of mutation testing so clearly. Silly stuff was missed like a mutant survived where a line of code which added an element to a list was removed. (How could we have missed an assert to verify an element was not added to the list!) Also, there were mutants that survived where lines where properties were set were removed and negated conditional mutants also survived. It is the simple stuff, the seemingly trivial stuff, that gets overlooked until you spend hours debugging a defect which ends with you banging your head on your desk saying how could I have missed that.
The project does unit test at a class level with mock objects powered by Mockito. The thing that could have helped kill these mutations more than anything else was Mockito.verifyNoMoreInteractions(someMock, anotherMock). It is best practice to have it in there but it often is forgotten about once you finish verifying the results.
