Avoid test output from being cached

Help/Discuss
1

I am working on setting up some CI where a mainline branch runs all the tests and writes to a build cache. Later PR builds will read from that cache and only rebuild/test files from affected modules.

One issue I am facing is that the test output (JUnit XML reports) are restored from the cache. In my CI workflow when I process the test results, it looks like all tests have run when in fact only some (or none) have.

Is there a way to avoid caching certain task outputs?

2 
tasks.withType<Test>().configureEach {
    outputs.cacheIf { false }
}
1 Like
3

Won’t outputs.cacheIf { false } just disable task caching completely?

I want the test tasks to be cached so that subsequent CI runs will have FROM-CACHE for unchanged modules. But I don’t want the test XML results to be restored from the cache results.

Here’s a more detailed example of what I’m trying to achieve

Let’s say I have three modules each with some tests

  • common has 20 tests
  • client has 50 tests, depends on common
  • server has 30 tests, depends on client and common

In my mainline build job, I want to run all the tests and cache the results. In my PR builds I will restore from this cache.

  • PR A: changes to “server”. Runs only server tests 30 tests. Others are FROM-CACHE
  • PR B: changes to “client”. Runs client and server tests, 80 total. “common” is still FROM-CACHE
  • PR C: changes to “common”. Runs all tests

This part seems to be working.

The issue is that by restoring the cache entry for the “:test” tasks, the JUnit output files are restored to the file system. My subsequent JUnit Reporter script, which scrapes all the matching XML files for results, is giving misleading results.

From the above example, all three PRs will seemingly have run 100 tests since the test results for FROM-CACHE tests are restored to the file system.

Maybe there’s some other way I should go about this?

Edit: Here is a build scan of run which re-ran only a portion of our full test suite Build Scan® | Develocity. The build scan report correctly shows how many tests were run. However, my custom XML parsing script thinks all tests were run because all of the FROM-CACHE “:test” tasks restored the XML results from previous runs.

4

Yes, you are right, it would make all tests not taken from cache, but all tests run each time. When taking the results from cache it is expected that it is treated like the tests are run, because the task was expected to run, just was using the cache result instead of really running the tests.

If you really want the custom parsing not report the cached result, you probably need to use an operation completion listener to record which tasks were cached and then not report on their results.

5

Thanks, including the test results in the cache does actually make sense, though works against my use case.

I discovered a solution for my workflow. If I disable the HTML and XML reports in my job that populates the cache, it will prevent them from being restored in my PR jobs. Then in the PR job, I would let the reports be enabled. This would result in only XML reports for the tests were actually run in the job.

Here’s the relevant snippet

userDisableJUnitReports = project.hasProperty('disableJUnitReports') && disableJUnitReports.toBoolean()

test {
    // ... other test configs
    reports {
        junitXml.required = !userDisableJUnitReports
        html.required = !userDisableJUnitReports
    }
}

Edit: marking this as Solution since it answers my original question

1 Like