Maven Surefire Testing Matrix

The standard way to test code in Java ecosystem is JUnit. The standard way to harness JUnit by Maven is the surefire plugin. Usually one doesn’t need to do anything to turn the plugin on - the standard Maven jar packaging does it automatically. However, it is possible to do magic with the plugin configuration and that is what this post is about.

JUnit Browser Intermezzo

However, before we dwell into the plugin configuration, let us set the context by introducing the DukeScript JUnit extension - it allows one to write the code once and then test it in different DukeScript environments. Where DukeScript environment is a basically a JVM configured to render HTML in some way. With a simple @RunWith annotation your code can be executed multiple times, in multiple different setups.

Where’s the problem? By default the JUnit Browser extension runs all the tests in a single Java virtual machine. Yet, some of the environments don’t go along well - for example the webkit presenter is using different GTK version than the default Apache JavaFX presenter. When used together - they crash the JVM.

Matrix of Runs

We need to run the surefire plugin multiple times. How can one do that? One needs to configure multiple test executions:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.10</version>
    <executions>
        <execution>
            <goals>
                <goal>test</goal>
            </goals>
            <id>default-test</id>
            <phase>test</phase>
            <configuration>
                <classpathDependencyExcludes>
                    <exclude>com.dukescript.presenters:webkit</exclude>
                    <exclude>org.apidesign.bck2brwsr:launcher.http</exclude>
                </classpathDependencyExcludes>
            </configuration>
        </execution>
        <execution>
            <goals>
                <goal>test</goal>
            </goals>
            <id>webkit-test</id>
            <phase>test</phase>
            <configuration>
                <classpathDependencyExcludes>
                    <exclude>org.netbeans.html:net.java.html.boot.fx</exclude>
                    <exclude>org.apidesign.bck2brwsr:launcher.http</exclude>
                </classpathDependencyExcludes>
            </configuration>
        </execution>
        <execution>
            <goals>
                <goal>test</goal>
            </goals>
            <id>bck2brwsr-test</id>
            <phase>verify</phase>
            <configuration>
                <classpathDependencyExcludes>
                    <exclude>org.netbeans.html:net.java.html.boot.fx</exclude>
                    <exclude>com.dukescript.presenters:webkit</exclude>
                </classpathDependencyExcludes>
            </configuration>
        </execution>
    </executions>
</plugin>

There are three execution sections. One overriding the default-test run (as added by the jar packaging), one adding another test run at the same test phase verifying the behavior on webkit presenter. In addition to that there is one more (kind of integration) test run which executes the same code transpiled into a JavaScript. As this step takes longer, it is scheduled for the verify phase. As a result one can:

$ mvn test
# or
$ mvn package

and the code is compiled and tested twice quickly. While one can schedule full verification with

$ mvn verify

Configuring the Matrix

The configuration section of each test run needs to alter the setup somehow - each of the runs is supposed to test slightly different environment, right? One can do that by setting different JVM properties, but our example has chosen a different route: it uses different classpath for each execution!

By default we add all three presenters to the project test classpath:

<dependencies>
    <!-- run tests in JavaFX WebView -->
    <dependency>
        <groupId>org.netbeans.html</groupId>
        <artifactId>net.java.html.boot.fx</artifactId>
        <version>${net.java.html.version}</version>
        <scope>test</scope>
    </dependency>

    <!-- run tests in a webkit presenter -->
    <dependency>
        <groupId>com.dukescript.presenters</groupId>
        <artifactId>webkit</artifactId>
        <version>${presenters.version}</version>
        <scope>test</scope>
    </dependency>

    <!-- run tests in bck2brwsr -->
    <dependency>
        <groupId>org.apidesign.bck2brwsr</groupId>
        <artifactId>launcher.http</artifactId>
        <version>${bck2brwsr.version}</version>
        <scope>test</scope>
    </dependency>
</dependencies>

This makes sure the Maven downloads the necessary artifacts and places them into the local repository. Then, each test execution masks all but one of the presenters using builtin surefire plugin classpathDependencyExcludes directive:

<configuration>
    <classpathDependencyExcludes>
        <exclude>org.netbeans.html:net.java.html.boot.fx</exclude>
        <exclude>org.apidesign.bck2brwsr:launcher.http</exclude>
    </classpathDependencyExcludes>
</configuration>

By doing that there is always only one DukeScript presenter available during individual test execution and the JUnit Browser extension picks that one up.

Test runs are isolated in their own JVM and combination of environments no longer negatively influence each other.

Where's the Matrix?

Nice, but where is the matrix? may be your next question! Well, the above example is taken from the DukeScript definitelytyped project which aims at providing Java API to important JavaScript libraries. There are hundreds of libraries ready and all of them share the same testing setup. As such the configuration of the surefire plugin is done in the master pom.xml in its pluginManagement section.

<build>
    <pluginManagement>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.10</version>
                <configuration>
<!-- .... -->

By extending the configuration of the surefire plugin in the master pom.xml we can enlarge the axis of additional DukeScript environments. By including more submodules one expands the axis of JavaScript libraries to test. The surefire testing matrix grows with every new commit. Try it, contribute too:

Convert your own library! It is easy, read more, fork and pull request.