Spock-Reports - great test reports that you actually want to read

Post date: Sep 26, 2017 6:34:37 PM

Spock-Reports introduced in version 1.3.0 the possibility to embed source code in reports, a feature we call vivid reports.

This may sound like a crazy idea if you are not used to reading and writing Spock specifications! But trust me, the reports created when vivid report is enabled look awesome!

Spock specifications (or tests, if you prefer) tend to be highly readable. Many people have said that the code is normally clear enough that they'd prefer to skip the description block you would see in traditional Spock specifications and just show the code in reports. That's how vivid reports were born.

Just as importantly, Spock error messages (thanks mostly to Groovy power-assertions) are as good as they get.

  A Spock Report for a failed test

When combined, awesome error messages and embedding the source code make these reports extremely useful (unlike reports from most other testing frameworks)!

For example, take this little test that is used to test spock-reports itself:

@Unroll

def "An @Unrolled spec with x=#x and y=#y"() {

    setup:

    "nothing"

    expect:

    x == 0

    and:

    "An error if y is 5"

    if ( y == 5 ) {

        throw new RuntimeException( 'y is 5' )

    }

    where:

    x | y

    0 | 1

    2 | 3

    0 | 5

}

It is obviously going to fail in the second (because x != 0) and third (because y == 5) iterations.

Here's what the report for the failed iterations look like:

Notice that when a test is Unrolled, spock-reports shows each iteration as a separate feature, unlike on the first report shown at the top of this blog post, where the test is example-based but not unrolled.

 Example of a failed vivid report

This is what the reports look like in version 1.3.0 and 1.3.1.

In version 1.3.2, which is just about to come out, we take things even further and highlight the line of code that caused the test to fail!

Below, you can the same section shown in the report above, but with the line that caused the test to fail highlighted:

 Vivid report with line of code that caused the test to fail highlighted

This is really nice. Notice that even the line number is shown as a comment at the end of the guilty line.

Real-world project example

To give a more realistic example, let's look at a Specification for another side project of mine, LogFX. As a log viewer, LogFX needs to watch files for changes, and the FileChangeWatcherSpec tests the code that does it.

Below, we see what the default spock-report for the first feature test looks like:

Real-world project report snippet example

This is good and nice for non-coders... but from experience, most actual consumers of this information are technical people, even if not only programmers... most of them would appreciate the further details provided by source code embedded in the report, as shown below (top summary omitted for brevity):

Real-world project report snippet with source code example

A lot more information is conveyed in this report, but it still looks like a report (as opposed to just printed code, for example)... and in my modest opinion, this is much more useful.

It allows teams to see what's tested, discuss not only the test plan but the implementation... as a side-effect, it might remind the developers writing the test to try to make the tests just a bit more readable as well, as this increases the visibility of the test code. Finally, when the test fails, you get right to the heart of the information you need:

Failed test real-world project report with source code

Even the stack-trace could be displayed if desired by setting the com.athaydes.spockframework.report.internal.HtmlReportCreator.printThrowableStackTrace property to true.

This failed test report is quite useful! The cause of the test failure is much easier to find if you can see exactly how the test is implemented. The fact that Groovy code can be quite compact and easy to read helps a lot... knowing exactly where the error happened without digging through build logs and loads of system output is really valuable as well.

Enabling vivid Spock reports in your tests

First of all, assuming you already use Spock, to start generating reports with spock-reports, just add a test dependency on spock-reports as shown below.

If you're not using Spock yet, head to the Spock documentation and do it now! You won't regret it.

Exclude transitive dependencies to avoid affecting your version of Spock and Groovy.

Notice that spock-reports 1.3.2 requires Spock version 1.1 at least.

Maven

<dependency>

    <groupId>com.athaydes</groupId>

    <artifactId>spock-reports</artifactId>

    <version>1.3.2</version>

    <scope>test</scope>

    <exclusions>

        <exclusion>

            <groupId>*</groupId>

            <artifactId>*</artifactId>

        </exclusion>

    </exclusions>

</dependency>

Gradle

testCompile 'com.athaydes:spock-reports:1.3.2', { transitive = false } 

After you run your tests now, you should find your reports under ${project.buildDir}/spock-reports/. The main report is at index.html.

As vivid reports are not enabled by default, to enable it, set the com.athaydes.spockframework.report.showCodeBlocks system property to true.

In Gradle, that's done by adding these lines to your build file:

test {

    systemProperty 'com.athaydes.spockframework.report.showCodeBlocks', true

}

You can also do that via a config file if you prefer, which allows you to configure just about anything about the reports. See the spock-reports README page for more information.

Comments on Reddit!