Build cache does not cache test runs

[Sebastian_Schuberth]

I’m trying to accelerate CI builds by not running tests for non-code changes, like a change to only README.md. I was hoping to be able to leverage the build cache for that, but unfortunately it behaves differently that I would expected. Here’s an example (with org.gradle.caching=true in gradle.properties):

First run in a clean tree:

$ git clean -fdx
$ ./gradlew scanner:funTest --tests com.here.ort.scanner.HttpCacheTest
...
:scanner:funTest
com.here.ort.scanner.HttpCacheTest > HttpCacheTest.HTTP GET returns what was PUT STARTED
com.here.ort.scanner.HttpCacheTest > HttpCacheTest.HTTP GET returns what was PUT PASSED
BUILD SUCCESSFUL in 9s
14 actionable tasks: 7 executed, 7 from cache

The tests runs, as expected. Then when executing the task again

$ ./gradlew scanner:funTest --tests com.here.ort.scanner.HttpCacheTest
...
:scanner:compileFunTestKotlin UP-TO-DATE
...
:scanner:funTest UP-TO-DATE
BUILD SUCCESSFUL in 1s
14 actionable tasks: 14 up-to-date

the :scanner:funTest task is UP-TO-DATE, also as expected. However, if I clean the tree now and again run the task

$ git clean -fdx
$ ./gradlew scanner:funTest --tests com.here.ort.scanner.HttpCacheTest
...
:scanner:compileFunTestKotlin FROM-CACHE
...
:scanner:funTest
com.here.ort.scanner.HttpCacheTest > HttpCacheTest.HTTP GET returns what was PUT STARTED
com.here.ort.scanner.HttpCacheTest > HttpCacheTest.HTTP GET returns what was PUT PASSED
BUILD SUCCESSFUL in 8s
14 actionable tasks: 7 executed, 7 from cache

The :scanner:funTest task is run again, although for various other tasks before that I see FROM-CACHE. Why is there no FROM-CACHE from :scanner:funTest, to be consistent with the behavior of running the task two time in a row?

[Stefan_Wolf]

Hi Sebastian,

I suppose :scanner:funTest is of type Test. Then the task is cacheable. So it may be that this test task does not have stable task inputs. Do you maybe add some inputs via the runtime API? Please have a look at https://guides.gradle.org/using-build-cache/#debugging_and_diagnosing_cache_misses and see if you can diagnose why you had the cache miss.

Cheers,
Stefan

[Sebastian_Schuberth]

I suppose :scanner:funTest is of type Test.

Yes, you can see the definition here:

https://github.com/heremaps/oss-review-toolkit/blob/master/build.gradle#L104-L124

[Stefan_Wolf]

The task definition looks good. Do you maybe have some generated resources (property files?) which contain a timestamp?

[Sebastian_Schuberth]

No, nothing like that.

[Sebastian_Schuberth]

And I just realized there’s the same problem with built-in (unit) test tasks, no need for custom funTest tasks in order to reproduce the issue.

[Stefan_Wolf]

What version of Gradle and the Kotlin plugin are you using? Is all the compilation FROM-CACHE?

[Sebastian_Schuberth]

I’m using Gradle 4.5, see https://github.com/heremaps/oss-review-toolkit/blob/master/gradle/wrapper/gradle-wrapper.properties.

Here’s the full task output:

$ ./gradlew scanner:funTest --tests com.here.ort.scanner.HttpCacheTest
Build cache is an incubating feature.
Parallel execution with configuration on demand is an incubating feature.
:utils:compileKotlin
:model:processResources
:downloader:processResources
:scanner:processResources
:utils-test:processResources
:downloader:processResources NO-SOURCE
:scanner:processResources NO-SOURCE
:model:processResources NO-SOURCE
:utils-test:processResources NO-SOURCE
:scanner:processTestResources
:scanner:processFunTestResources NO-SOURCE
:utils:compileKotlin FROM-CACHE
:utils:compileJava
:utils-test:compileKotlin
:utils:compileJava NO-SOURCE
:utils:processResources
:utils:classes
:utils:jar
:utils-test:compileKotlin FROM-CACHE
:model:compileKotlin
:utils-test:compileJava NO-SOURCE
:utils-test:classes UP-TO-DATE
:utils-test:jar
:model:compileKotlin FROM-CACHE
:model:compileJava NO-SOURCE
:model:classes UP-TO-DATE
:model:jar
:downloader:compileKotlin FROM-CACHE
:downloader:compileJava NO-SOURCE
:downloader:classes UP-TO-DATE
:downloader:jar
:scanner:compileKotlin FROM-CACHE
:scanner:compileJava NO-SOURCE
:scanner:classes UP-TO-DATE
:scanner:compileTestKotlin FROM-CACHE
:scanner:compileTestJava NO-SOURCE
:scanner:testClasses
:scanner:compileFunTestKotlin FROM-CACHE
:scanner:compileFunTestJava NO-SOURCE
:scanner:funTestClasses UP-TO-DATE
:scanner:funTest
...

Can it be related to being a Kotlin project?

[Sebastian_Schuberth]

@Stefan_Wolf, any more hints what to try?

[Stefan_Wolf]

Did you follow the steps in the guide?

If you don’t have Gradle Enterprise you would need to compare the input property hashes by hand. These are shown on the info log for the test task.

If you have Gradle Enterprise (we have free trials, too), you can use task input comparison to see what has changed between the two invocations.

[Sebastian_Schuberth]

As far as I can see the property hashes match. This console output with --info is the same in first and second runs:

Appending outputPropertyName to build cache key: binResultsDir
Appending outputPropertyName to build cache key: jacoco.destinationFile
Appending outputPropertyName to build cache key: reports.enabledDirectoryReportDestinations.html
Appending outputPropertyName to build cache key: reports.enabledDirectoryReportDestinations.junitXml
Build cache key for task ':scanner:funTest' is 4b571eaf8c9001c727ab7519044030f0

But I saw

Caching disabled for task ':scanner:funTest': 'JaCoCo agent configured with `append = true`' satisfied

Luckily, @sterling has proposed a work-around for this already!

[trejkaz (Trejkaz (pen name))]

Nuts.

We hit exactly the same issue here, where the cache IDs were the same, and we’re using JaCoCo too, but we’re not happy to turn off coverage to get caching, and we do want to run tests in parallel.

I’m guessing there is no solution then?

[Lance (uklance)]

Just chiming in here without fully understanding the problem. I see mention in the linked post regarding JaCoCo and append=true

Is it possible to

1. Have a separate JaCoCo exec file for each test task
2. Set append=false for each
3. Use a JacocoMerge task to merge the multiple exec files into one
4. Generate a JacocoReport from the merged exec file

[trejkaz (Trejkaz (pen name))]

I’ve done a bit of investigation and it seems like JaCoCo’s tasks already create a separate test report for each task, so the broken thing must be that when you run a single test task with the tests in parallel, it doesn’t cope with that parallelism.

[Lance (uklance)]

Ah, I’m guessing the solution is to enhance the JaCoCo plugin to create an exec file per fork/test worker. These files would be merged together once all the tests are finished. That should yield a repeatable/cacheable result.

Another solution (that doesn’t require a JaCoCo plugin fix) is to use a single test worker (forkEvery 0) and append=false but I’m guessing that tests run too slowly in this scenario? In that case you could separate tests into separate projects which would allow gradle to run multiple test tasks in parallel (each with one fork/worker)

[Lance (uklance)]

I put together a proof of concept which creates 10 “test-shard” projects, each has it’s own Test task which runs a subset of tests from the “all-tests” project. This allows each Test task to use append=false so that each is cacheable. Each Test task runs a single fork/testWorker but the test projects can be run in parallel. I’ve used a JacocoMerge task to merge the shards into a single report.

Interesting code at

uklance/gradle-cacheable-jacoco/blob/master/all-tests/build.gradle#L27-L44

I’ve tested this with the build cache and I can see the test results are cached correctly