Can I produce a single aggregated report for all of my tests, while supporting up-to-date checking?

I have a simple, four file multi-project build using the java plugin in M5. If I set the sub projects testReportDir to a location in the root project, the projects are never up to date. Details below.

You can download a zip of the sample project from here.

Files are as follows:

structure:

  .  ├── build.gradle  ├── one  │   └── src  │

├── main │

│ └── java │

└── org │

└── gradle │

└── bug │

└── SampleClass.java │

└── test │

└── java │

└── org │

└── gradle │

└── bug │

└── TestSampleClass.java └── settings.gradle

settings.gradle

include ':one'

build.gradle

allprojects {
    apply plugin: 'java'
          repositories {
      mavenCentral()
    }
          dependencies {
      testCompile "junit:junit:4.8.2"
    }
          test {
      testResultsDir = rootProject.testResultsDir
       //testReportDir = rootProject.testReportDir
     }
  }

The issue here is that I need aggregated test reports for the multi-project build and if I comment in the testReportDir setting above, the tasks [test, check, build] are never up to date.

Execution sample:

nadurra:uptodatetest mbjarland$ gradle --daemon clean
   Note: the Gradle build daemon is an experimental feature.
  As such, you may experience unexpected build failures. You may need to occasionally stop the daemon.
  :clean
  :one:clean
      BUILD SUCCESSFUL
      Total time: 0.851 secs
nadurra:uptodatetest mbjarland$ gradle --daemon build
  Note: the Gradle build daemon is an experimental feature.
  As such, you may experience unexpected build failures. You may need to occasionally stop the daemon.
  :compileJava UP-TO-DATE
  :processResources UP-TO-DATE
  :classes UP-TO-DATE
  :jar
  :assemble
  :compileTestJava UP-TO-DATE
  :processTestResources UP-TO-DATE
  :testClasses UP-TO-DATE
  :test
  :check
  :build
  :one:compileJava
  :one:processResources UP-TO-DATE
  :one:classes
  :one:jar
  :one:assemble
  :one:compileTestJava
  :one:processTestResources UP-TO-DATE
  :one:testClasses
  :one:test
  :one:check
  :one:build
      BUILD SUCCESSFUL
      Total time: 2.264 secs
nadurra:uptodatetest mbjarland$ touch stamp && gradle --daemon build
  Note: the Gradle build daemon is an experimental feature.
  As such, you may experience unexpected build failures. You may need to occasionally stop the daemon.
  :compileJava UP-TO-DATE
  :processResources UP-TO-DATE
  :classes UP-TO-DATE
  :jar UP-TO-DATE
  :assemble UP-TO-DATE
  :compileTestJava UP-TO-DATE
  :processTestResources UP-TO-DATE
  :testClasses UP-TO-DATE
  :test
  :check
  :build
  :one:compileJava UP-TO-DATE
  :one:processResources UP-TO-DATE
  :one:classes UP-TO-DATE
  :one:jar UP-TO-DATE
  :one:assemble UP-TO-DATE
  :one:compileTestJava UP-TO-DATE
  :one:processTestResources UP-TO-DATE
  :one:testClasses UP-TO-DATE
  :one:test
  :one:check
  :one:build
      BUILD SUCCESSFUL
      Total time: 2.135 secs
nadurra:uptodatetest mbjarland$ find . -newer stamp
  ./.gradle/1.0-milestone-5/fileHashes/cache.bin
  ./.gradle/1.0-milestone-5/fileHashes/cache.properties.lock
  ./.gradle/1.0-milestone-5/fileSnapshots/cache.bin
  ./.gradle/1.0-milestone-5/fileSnapshots/cache.properties.lock
  ./.gradle/1.0-milestone-5/outputFileStates/cache.properties.lock
  ./.gradle/1.0-milestone-5/taskArtifacts/cache.bin
  ./.gradle/1.0-milestone-5/taskArtifacts/cache.properties.lock
  ./build/reports/tests/com.sample.html
  ./build/reports/tests/com.sample.TestSampleClass.html
  ./build/reports/tests/index.html
  ./build/test-results/TEST-com.sample.TestSampleClass.xml
  ./build/tmp/jar/MANIFEST.MF
  ./one/build/tmp/jar/MANIFEST.MF

we can see from the output that even though we didn’t change anything between the calls to ‘build’, the ‘test’ task and subsequent check and build tasks are not marked as up to date. This remains the case for an arbitrary number of executions.

The last find command shows the files which were changed during the last build (I did a touch on file ‘stamp’ right before the previous gradle execution). The MANIFEST.MF files are created due to an issue I described in a previous post on these forums so we can ignore those for now. What is interesting is that the test results and reports are regenerated every execution.

Excerpt from the debug log:


...  14:13:48.207 [INFO] [org.gradle.api.internal.changedetection.DefaultTaskArtifactStateRepository] Executing task ':one:test' due to:  Output file /Users/mbjarland/tmp/uptodatetest/build/reports/tests/com.sample.TestSampleClass.html for task ':one:test' has changed.  Output file /Users/mbjarland/tmp/uptodatetest/build/reports/tests/index.html for task ':one:test' has changed.  Output file /Users/mbjarland/tmp/uptodatetest/build/reports/tests/com.sample.html for task ':one:test' has changed.  14:13:48.208 [DEBUG] [org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter] task ':one:test' is not up-to-date  ...  

so the log tells us that the files have changed and we need to re-run the test task. Well the files have changed because…err…the test task itself created the files during the previous run?

I did a quick attempt at trying to figure out why we get this behavior but couldn’t really come to any conclusive explanation. Yet. I’ll keep digging.

Perhaps this is the root project stepping on the sub-projects toes in some way? In either case, it would be nice to be able to combine aggregated test results/reports with a working up to date management. Re-running the tests every time is not an option as test executions can potentially take a long time.

Or perhaps there is a better recommended way of aggregating test results?

Any help much appreciated.

Hi - thanks for the detailed report.

I’ve downloaded and run the sample project using the M5 release, and I’m not seeing the behaviour you’re describing. On the second and all subsequent calls to ‘build’, everything is reported as up to date, and no test execution occurs.

The first thing you might try is ensuring that any old daemon processes are stopped. Another thing to try is to delete the .gradle directory from your project.

Can you please try those things out and let me know if it makes a difference?

Hi Daz and thanks for the fast reply.

Yes, I have tried a few things. Removing the local .gradle for the project, removing the global .gradle (including artifact caches) from my user_home/.gradle, killing the gradle daemon, etc, etc.

Did you comment in the testReport dir? In other words did you change build.gradle from:

...
  //here things work
  test {
    testResultsDir = rootProject.testResultsDir
     //testReportDir = rootProject.testReportDir
   }
  ...

to:

...
  //and here they don't
  test {
    testResultsDir = rootProject.testResultsDir
     testReportDir = rootProject.testReportDir
   }
  ...

before running the test?

I was probably not quite as clear as I should have been in the above, but with that line commented out things works as you would expect. When the line is not commented out (and is active) I get the not-up-to-date behavior described above.

I also changed the linked zip above to exhibit the symptoms without needing any edits to the build file.

Daz, thanks for the example! This was very helpful and solved my problem. I now have the following snippet (in addition to the task class you provided) in my root project:

task aggregateTestReports(type: TestReportAggregator, dependsOn: test) {
    testReportDir = file("${reportsDir}/tests")
    testResultsDir = file("${buildDir}/test-results")
    projects = subprojects
}
build.dependsOn(aggregateTestReports)

this is in a real multi-project build with a large number of sub-projects. The “dependsOn: test” is mostly there for symmetry and to make the aggregateTestReports task appear after all the test tasks in the build order, including the test task for the root project.

I now get an up-to-date from the entire project chain on sequential builds without changes. We can mark this issue as answered.

Thanks for the useful code, but I’m having trouble calling this effectively. When running the build in Jenkins, I want all the tests to run for every submodule and then the aggregation to run. When I execute:

gradle aggregateTestReports --continue

The “continue” ensures that all the tests run for every module, but if one test fails, then aggregateTestReports doesn’t run at all.

The aggregation doesn’t run because dependent tasks do not execute if a dependency failed: http://gradle.org/docs/current/userguide/tutorial_gradle_command_line.html#sec:continue_build_on_failure

One solution would be make the test tasks ignore failures in continue mode:

tasks.withType(Test) {
  ignoreFailures = gradle.startParameter.continueOnFailure
}

I am new to gradle as a maven escapee, but this looks like it should be the default for the test task. Maven has a difference between test failures (which allow processing to continue) and test errors (which stop the build).

Is it worth filing a task for this to be the default behaviour or is there already work being done on gradle exception handling?

What is the intersection between the command line “–continue” and the above preference? Obviously “–continue” applies to all tasks and the preference is more specific, but one seems to be “stronger” than the other in its effect.

but this looks like it should be the default for the test task

It’s unlikely that we’d ignore errors by default for test tasks. Most people want to stop the world when this happens.

Is it worth filing a task for this to be the default behaviour or is there already work being done on gradle exception handling?

There are already some issues on this and it’s something we are always working on.

What is the intersection between the command line “–continue” and the above preference?

‘Test’ implements ‘VerificationTask’, which has the semantics of allowing you to be able to specify whether a verification failure should be a build failure. The intersection of the two features is that if you ignore failures in a ‘VerificationTask’ and it fails, subsequent dependent tasks will still execute because there wasn’t a real task failure.

Thanks for this info Luke. But to be clear, I wasn’t suggesting that test errors should be ignored. They should definitely stop the tests. However test failures are a different thing. Mostly we want tests to continue to run so that we can:

  1. collect all the test failures from all the modules and not just the first failure 2. aggregate the test results (so we can display it in our CI)

The default the we have, of stopping on test failure works for most people and changing it now would be a breaking change.

You can enable the behaviour you want with:

tasks.withType(Test) {
  ignoreFailures = true
}

Any ideas how to make it today? As Daz DeBoer warned the API change and this doesn’t work anymore(

tried this and one of the sub projects just has output jars and no tests but it’s failing with

Could not determine the dependencies of task ‘aggregateTestReports’. > Could not find property ‘testResultsDir’ on task ‘myjar:test’.

although myjar folder has a build folder but no build.gradle. it is referenced in settings.gradle. do i need to add a build.gradle to just tell it that it has no test functionality (its a holdover from the old maven build where we took all the subprojects under one and built them into a jar to put in local repo. one

- one-services

- one-models

- one-controllers

if i remove that project from the settings it still errors on the first project. any suggestions?

OK now it makes sense! The reason that you’re seeing this behaviour is because you effectively have 2 projects generating test reports into the same directory: the ‘root’ project and the ‘:one’ subproject.

Ignoring the fact that maybe gradle shouldn’t generate a test report for the root project that has no tests, it makes sense that the ‘:one:test’ task checks if it’s outputs have changed, and re-executes if they have. Although in this case this could be fixed by using ‘subprojects’ instead of ‘allprojects’, you’re likely to have multiple subprojects in a real multiproject build, so this isn’t really a solution.

So until gradle supports this sort of test aggregation in the wild, you’ll need to code up a task to do this yourself.

We have something that does this in the gradle build:

task aggregateTestReports(type: TestReportAggregator) {
    testReportDir = file("${reportsDir}/tests")
    testResultsDir = file("${buildDir}/test-results")
    projects = subprojects
}
  class TestReportAggregator extends Copy {
    def projects
      File testResultsDir
      @OutputDirectory
    File testReportDir
      def TestReportAggregator() {
        dependsOn { testTasks }
        from { inputTestResultDirs }
        into { testResultsDir }
    }
      @TaskAction
    def aggregate() {
        def report = new org.gradle.api.internal.tasks.testing.junit.report.DefaultTestReport(testReportDir: testReportDir, testResultsDir: testResultsDir)
        report.generateReport()
    }
      def getTestTasks() {
        projects.collect { it.tasks.withType(Test) }.flatten()
    }
      def getInputTestResultDirs() {
        testTasks*.testResultsDir
    }
}

Note that this example uses an internal class ‘DefaultTestReport’, so the api may change in the future.