Standalone test report fails if no data

Old Forum Archive
1

We are having an aggregated test report task specified along this lines:

task aggregatedTestReport(type: TestReport) {
    testResultDirs = subprojects*.tasks*.withType(Test)*.flatten()*.binResultsDir
    destinationDir = file("$buildDir/reports/tests")
    allprojects.tasks*.withType(Test).flatten()*.finalizedBy it
}

Typically it works fine, except one very specific case: 1. gradle clean 2. gradle aggregatedTestReport // passes 3. gradle aggregatedTestReport // fails!

What happens is that the first time the binary test results is empty and the constructor in TestOutputStore.Reader#Reader() creates an empty ‘output.bin’ file. The second time arround, it sees that the outputs file exists and takes a different branch, throwing exception because the index file is missing.

If we snip the irrelevant code, it looks like this:

public Reader() {
  File indexFile = getIndexFile();
  File outputsFile = getOutputsFile();
    if (outputsFile.exists()) {
    if (!indexFile.exists()) {
        throw new IllegalStateException(String.format("Test outputs data file '{}' exists but the index file '{}' does not", outputsFile, indexFile));
    }
    // snipped
  } else { // no outputs file
    if (indexFile.exists()) {
        throw new IllegalStateException(String.format("Test outputs data file '{}' does not exist but the index file '{}' does", outputsFile, indexFile));
    }
      GFileUtils.touch(getOutputsFile());
    index = new Index();
  }
    // snipped
}

The root cause is that we are running the report without having run the tests, but that seems to be intentionally supported behavior, so I would deem this as a bug.

2

Which Gradle version are you using? I think this may have been fixed already.

3

This is with 1.8 - haven’t tested with others

4

It seems that the history is that the exception on missing data/index files was added in changeset f2998f1 with commit message "GRADLEREV-47 - stricter handling of output store / index file existence. "

Then as part of a refactoring in f9965a6 with commit message “GRADLEREV-48 - Use a RandomAccessFile to read from the test output”, the Reader code was changed to create a test output file if bith the test output and test input are missing.

Here is the code in the current head (b4f924b): http://code-review.gradle.org/browse/Gradle/subprojects/plugins/src/main/groovy/org/gradle/api/internal/tasks/testing/junit/result/TestOutputStore.java?hb=true#to229

In particular, line 274 creates an output file but no index file, while line 235 throws exception if it sees output file without index file.

5

This has been exported as GRADLE-2915.

I’m curious as to your use case here though. Why do you want to generate reports when the tests haven’t been run? That is, why aren’t you using ‘reportsOn()’?

http://www.gradle.org/docs/current/dsl/org.gradle.api.tasks.testing.TestReport.html#org.gradle.api.tasks.testing.TestReport:reportOn(java.lang.Object...)

6

I am not using reportsOn() because I was not aware of it, but even if I did it would not change much (setTestResultDirs delegates to reportOn) - i.e. my task would look like:

task aggregatedTestReport(type: TestReport) {
    reportOn subprojects*.tasks*.withType(Test)*.flatten()
    destinationDir = file("$buildDir/reports/tests")
    allprojects.tasks*.withType(Test).flatten()*.finalizedBy it
}

Our use case is that given a multi-module build, we want to have a single test report which would show us all failing tests. This is convenient to keep open in a browser and just reload, rather than continuously copy-pasting URLs from the console.

When we run the build for a single component, Gradle only runs the test for a single component, so the other component’s test-result directories are empty.

As each test task is finalized by the aggregation report, Gradle will run the aggregated report. The aggregated report collates all test reports, and that is the way we like it (as an aside, is there way to inspect only the tasks that are scheduled to run during the current execution)?

If each test-result directory contains stale reports it will pick them up and that is fine - we know we didn’t run a full build.

If we run the tests for a single build after clean, we get the issue above. The workaround is to always run all the tests first time after clean.

7

I thought ‘reportsOn()’ also added the finalized by, my mistake.

8

‘reportsOn()’ is not quite what you want. It sets up a dependency relationship between the report task and the test tasks.

The issue you reported has been fixed for 1.9.

9

is there any tentative timeline for 1.9?

10

We release every 8-10 weeks.

11

another use case is when Test tasks are skipped with onlyIf closure. TestReport task will still ReportOn this test task even though it should not. Can we make sure TestReport checks if task is enabled before reporting on it?