Understanding the cause of remote build cache misses

Help/Discuss
1

Apologies in advance if this has been answered elsewhere already, or if I am putting this in the wrong spot, but my various searches haven’t yielded any real answers to the question.

I nearly added this to Req. Beginners guide to Remote Build Cache but I think this really is a bit of a different question.

We recently established a remote build cache, and it is already paying dividends for our CI builds. Because we have a strategy that produces versions for each build, and our versioning is based on the git commit, we needed to ensure that our artifacts in our multi-module project (JAR files in our case) did not include the version number at build time so that the build cache would recognize the artifacts as equivalent. As soon as we changed that, we could see that our CI builds were successfully making use of the build cache to skip test execution where we had no upstream changes.

We would like the build cache to work for our developer workstations, too, so that we can share the benefits. Right now, it isn’t working.

Running the build scan, we can see:

The task was not up-to-date because of the following reasons:	
Input property 'stableClasspath' file module-two\build\libs\module-two.jar has changed.	
Input property 'stableClasspath' file module-one\build\libs\module-one.jar has changed.

When we run the build with -Dorg.gradle.caching.debug=true, I can see entries like this:

Appending input file fingerprints for 'stableClasspath' to build cache key: 5f58fdf21875d0f47e11ab80c5998e4d -
CLASSPATH{C:\Some\Path\project\module-one\build\classes\java\test\some\package\SomeTest.class='some/package/SomeTest.class' / da3607191430f9fb52f4be23122dc3bb, 
...
 C:\Some\Path\project\module-one\build\libs\module-one-test-fixtures.jar=IGNORED / d0b4fcba5f13ad4d9b5a486d09b834c8, 
...

I suspect that our cache misses stem from a path sensitivity issue, and I understand that Gradle’s default path sensitivity is ‘ABSOLUTE’, which if not properly specified in a task (or in a plugin) would cause a cache miss if the input resources come from a different path. We’re in fact on an entirely different platform (Windows vs Linux) so this will alwasy be the case for our builds.

Our build doesn’t use many custom tasks, and is largely a composition of standard plugins and our own convention plugins.

Do you have any advice as to how we might go about identifying the root cause of our issues? Could the problem come from one of the plugins that we are using (such as the java-test-fixtures or jacaco plugins )?

2

Actually, the jar name should not disturb. As long as it is in a @Classpath or @CompileClasspath property. There the file names should be irrelevant I think.

How I usually investigate such non-hits without Develocity at hand where you can compare scans, is by using the property you mentioned once on the build producing the cache entry and once on the build that does not reuse it, then use a diff tool on that debug output.

3

That’s a bit tricky because the cache entry producer runs on Linux and will have a different path for each entry than the consumer, which is running on Windows (and has a different code checkout path anyway). Is the path ignored? If it is, I can try stripping it out before using the diff tool.

4

You should not throw the paths out, it could be important information.
For each file part of the cache the path can be relevant or not, depending on the annotations on the respective property or runtime calls. But you see it in the debug output.
For example in the love you shared you see “…\module-one-test-fixtures.jar=IGNORED …”. So for that file the path and also file name is irrelevant and if the checksum in that line is different, it is not because of the path, but because of the content.