Coding Chica

We have a unit test that works some of our application’s code, but how much is really exercised during our build? Are we confident in our unit testing based upon how much of our code is covered?

Enter the jacoco-maven-plugin. It will allow us to both:

• Report upon how much of our code is exercised during unit testing, and
• Break the Maven build if not enough of our code is covered by unit tests.

pom.xml Properties

We jacoco-maven-plugin will either define or update a property called argLine when the prepare goal runs. However, since Maven doesn’t know about that logic, let’s define it, too, so we don’t show any errors in the pom.xml syntax. Inside of the properties section add:

<!--
argLine will be extended by jacoco-maven-plugin.
Additional argLine values for maven-surefire-plugin can be set here.
-->
<argLine></argLine>

pom.xml Plugin Management Changes

The pluginManagement section is more geared toward multi-module Maven build and our build is just one module. If our Maven build had any sub-modules, they can inherit the configuration in this section or choose to override it. Therefore, even though we don’t need it right now, let’s go ahead and define the majority of our configuration in the pluginManagement section, so we can start getting familiar with it.

maven-surefire-plugin argLine

In order to allow the jacoco-maven-plugin to what what happens while we are unit testing, we need to update how the maven-surefire-plugin does its testing.

Add the following to the existing maven-surefire-plugin definition, just below the version line. In XML, a comment is defined by start (<!–) and end (–>) tags. Everything between those two tags is ignored by Maven.

<configuration>
    <!-- argLine will be set by jacoco-maven-plugin -->
    <argLine>${argLine}</argLine>
</configuration>

Add jacoco-maven-plugin

After the closing plugin tag (AKA </plugin>) for the maven-surefire-plugin, let’s add the following to configure the jacoco-maven-plugin.

Double check which version to use by reviewing the JacoCo – Change History page. As of this writing, the latest release is 0.8.10 – I do not recommend using any SNAPSHOT versions.

<plugin>
  <!--
  https://www.eclemma.org/jacoco/trunk/doc/maven.html
  https://www.eclemma.org/jacoco/trunk/doc/integrations.html
  -->
  <groupId>org.jacoco</groupId>
  <artifactId>jacoco-maven-plugin</artifactId>
  <version>0.8.10</version>
  <executions>
    <execution>
      <id>prepare-ut</id>
      <goals>
        <goal>prepare-agent</goal>
      </goals>
    </execution>

    <execution>
      <id>report-ut</id>
      <goals>
        <goal>report</goal>
      </goals>
      <configuration>
        <outputDirectory>${project.build.directory}/ut-code-coverage</outputDirectory>
      </configuration>
    </execution>

    <execution>
      <id>check-ut</id>
      <goals>
        <goal>check</goal>
      </goals>
      <configuration>
        <rules>
          <rule>
            <!-- In each declared class. -->
            <element>CLASS</element>
            <limits>
              <limit>
                <counter>INSTRUCTION</counter>
                <value>COVEREDRATIO</value>
                <minimum>0.80</minimum>
              </limit>
              <limit>
                <counter>BRANCH</counter>
                <value>COVEREDRATIO</value>
                <minimum>0.80</minimum>
              </limit>
            </limits>
          </rule>
        </rules>
      </configuration>
    </execution>
  </executions>
</plugin>

Let’s examine each execution within the plugin’s configuration:

prepare-ut

    <execution>
      <id>prepare-ut</id>
      <goals>
        <goal>prepare-agent</goal>
      </goals>
    </execution>

The prepare-agent goal runs before unit testing and creates a Java agent that can watch the unit testing occur and record what code was covered. It runs during the initialize phase of the Maven default lifecycle unless we specify otherwise. It also updates the argLine property to reflect that agent’s location.

report-ut

    <execution>
      <id>report-ut</id>
      <goals>
        <goal>report</goal>
      </goals>
      <configuration>
        <outputDirectory>${project.build.directory}/ut-code-coverage</outputDirectory>
      </configuration>
    </execution>

The report goal runs after unit testing and creates an HTML report for the whole project at target/ut-code-coverage/index.html. In IntelliJ: Right click on that file -> Open In -> Browser -> <Choose your Browser Name> to view the file.

This file is helpful when trying to fill in any gaps in code coverage, especially if the build fails due to the JaCoCo build breaker.

check-ut

    <execution>
      <id>check-ut</id>
      <goals>
        <goal>check</goal>
      </goals>
      <configuration>
        <rules>
          <rule>
            <element>CLASS</element>
            <limits>
              <limit>
                <counter>INSTRUCTION</counter>
                <value>COVEREDRATIO</value>
                <minimum>0.80</minimum>
              </limit>
              <limit>
                <counter>BRANCH</counter>
                <value>COVEREDRATIO</value>
                <minimum>0.80</minimum>
              </limit>
            </limits>
          </rule>
        </rules>
      </configuration>
    </execution>

This is the build breaker. It runs after unit testing and the check goal enforces that a successful build will have at least the minimum coverage specified (apart from any exclusions provided in the configuration).

The element supports the following values:

• BUNDLE,
• PACKAGE,
• CLASS,
• SOURCEFILE, and
• METHOD

The counter supports the following values:

• INSTRUCTION,
• LINE,
• BRANCH,
• COMPLEXITY,
• METHOD, and
• CLASS

The value supports the following values, where ratios are values with decimal places between 0 and 1:

• TOTALCOUNT,
• COVEREDCOUNT,
• MISSEDCOUNT,
• COVEREDRATIO, and
• MISSEDRATIO

Maven Build – Failing Build Breaker

At this point, we should run the Maven build to see a failure scenario and ensure our build breaker is working correctly.

[WARNING] Rule violated for class codingchica.java101.App: instructions covered ratio is 0.35, but expected minimum is 0.80
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
...
------------------------------------------------------------------------
[ERROR] Failed to execute goal org.jacoco:jacoco-maven-plugin:0.8.10:check (check-ut) on project java101: Coverage checks have not been met. See log for details. -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

At the top of the snippet, there is a warning with the details about which file failed the build-breaker check:

[WARNING] Rule violated for class codingchica.java101.App: instructions covered ratio is 0.35, but expected minimum is 0.80

Further down the output, we see that the JaCoCo build breaker is what failed the build:

[ERROR] Failed to execute goal org.jacoco:jacoco-maven-plugin:0.8.10:check (check-ut) on project java101: Coverage checks have not been met. See log for details. -> [Help 1]

New Unit Test for Main

Let’s add the following to our codingchica.java101.AppTest.java to address this build failure. We still aren’t ready to do all of the validation, so rather than validations, we’ll add a special comment that starts with TODO – indicating that a task remains.

Many IDEs will highlight these types of comments, so they stand out as you review the code.

    /**
     * Unit tests for the main method.
     *
     * @see App#main(String[])
     */
    @Nested
    class MainTest {

        @Test
        void main_whenInvoked_thenNoException() {
            // Setup

            // Execution
            App.main(null);

            // Validation
            // TODO Only tests that the application runs successfully, not the actual behavior.
        }
    }

In the code snippet above, we also have some new logic to examine. When invoking a static method on a class, we specify the class name and do not need an instance of the class, so we say: App.main. In this case, we also do not have argument list to provide, so we say null in the parameter list instead of a value for args.

Rerun the Maven Build

In the maven build output, we should now see:

1. prepare-ut
2. default-test – which includes our new AppTest$MainTest and the Hello gorgeous! output
3. report-ut
4. check-ut, which now shows that all coverage checks have been met.

...
[INFO] --- jacoco:0.8.10:prepare-agent (prepare-ut) @ java101 ---
...
[INFO] --- surefire:3.1.2:test (default-test) @ java101 ---
...
[INFO] Running codingchica.java101.AppTest
[INFO] Running codingchica.java101.AppTest$MainTest
Hello, gorgeous!
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.041 s -- in codingchica.java101.AppTest$MainTest
...
[INFO] --- jacoco:0.8.10:report (report-ut) @ java101 ---
...
[INFO] --- jacoco:0.8.10:check (check-ut) @ java101 ---
...
[INFO] Analyzed bundle 'java101' with 1 classes
[INFO] All coverage checks have been met.
...
------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------

Remember to Commit

Now is a good time to commit.