Mutation testing Java example

While JUnit tests helps in validating business logic, Mutation testing further improves the quality of code by checking if tests cover all the scenarios by introducing mutants in the code. Although JUnit tests can be used to measure the code coverage, mutation testing goes one more level to validate logic by altering the code. For example if you are using '>' to compare numbers, mutation testing will introduce mutant in the code by changing '>' to '<' and then check if JUnit covers this condition.

We can do mutation testing using PIT library which creates mutants at bytecode level without alterting the code. In this tutorial, we will show how to configure mutation testing using PIT and improve code coverage.

Configuring and running mutation tests

Step 1) Add maven dependencies

To run and generate mutation test coverage report, add below dependencies & plugins. Notice that we can choose the Java package which will be target for mutation tests using <param><package name>.*</param>

<dependency>
    <groupId>org.pitest</groupId>
    <artifactId>pitest-parent</artifactId>
    <version>1.1.10</version>
    <type>pom</type>
</dependency>

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <version>3.0.0-M1</version>
        </plugin>
        <plugin>
            <groupId>org.pitest</groupId>
            <artifactId>pitest-maven</artifactId>
            <version>1.4.3</version>

            <executions>
                <execution>
                    <id>pit-report</id>
                    <phase>test</phase>
                    <goals>
                        <goal>mutationCoverage</goal>
                    </goals>
                </execution>
            </executions>
            <dependencies>
                <dependency>
                    <groupId>org.pitest</groupId>
                    <artifactId>pitest-junit5-plugin</artifactId>
                    <version>0.8</version>
                </dependency>
            </dependencies>
            <configuration>
                <targetClasses>
                    <param>mutation.*</param>
                    </targetClasses>
                    <targetTests>
                        <param>mutation.*</param>
                </targetTests>
            </configuration>
        </plugin>
    </plugins>
</build>

dependencies {
    classpath 'info.solidsoft.gradle.pitest:gradle-pitest-plugin:1.4.6'
    pitest 'org.pitest:pitest-junit5-plugin:0.12'
}
   
apply plugin: 'info.solidsoft.pitest'
 
pitest {
    testPlugin = 'junit5'
}   

Step 2) Write Numbers class which compares two numbers

In the example below, we will first write a Numbers class which compares two numbers.

package mutation;

public int compare(Integer number1, Integer number2) {
        if (number1.intValue() > number2.intValue()) {
            return 1;
        }
        return -1;
} 

Step 3) Write NumbersTest class which tests compare() method of Numbers Class

package mutation;
import static org.junit.jupiter.api.Assertions.assertEquals;
import org.junit.jupiter.api.Test;

public class NumbersTest {

    @Test
    public void testComapare() {
        Numbers numbers = new Numbers();
        assertEquals(false, numbers.compare(13, 12));
    }

}    

Step 4) Run mutation tests

Run mutation tests using maven clean install command or manually using "mvn clean org.pitest:pitest-maven:mutationCoverage" command. Mutation test report will be generated under ".\target\pit-reports" directory

You will see below report generated. Observer that even though line coverage is 75%, mutation coverage is only 50%. This is because mutation tests will also test the code ny changing "%gt;" to "%lt;" for which tests are missing.

Now we will add more junit tests to validate missed out scenarios and kill the mutants.

    @Test
    public void testComapare() {
        Numbers numbers = new Numbers();
        assertEquals(false, numbers.compare(13, 12));
        assertEquals(-1, numbers.compare(11, 12));
        assertEquals(-1, numbers.compare(11, 11));
    }    

Now run mutation tests and observer report below. We have convered all the scenarios.

By default below mutators are configured but you can also configure the mutators

• INCREMENTS_MUTATOR
• VOID_METHOD_CALL_MUTATOR
• RETURN_VALS_MUTATOR
• MATH_MUTATOR
• NEGATE_CONDITIONALS_MUTATOR
• INVERT_NEGS_MUTATOR
• CONDITIONALS_BOUNDARY_MUTATOR

    <configuration>
        <mutators>
            <mutator>CONSTRUCTOR_CALLS</mutator>
            <mutator>NON_VOID_METHOD_CALLS</mutator>
        </mutators>
    </configuration>