Hi there I’m looking for some help / ideas in debugging my cache misses while using remote build cache on local builds.
The current state is that the vast majority of tasks are cache hits, but a few of them are misses. It’s reproducible each time. The first one of them is task named compile<Redacted>DebugJavaWithJavac.
To debug the issue, I’ve run this task on both CI and locally with -Dorg.gradle.caching.debug=true flag and compare the outputs. From there I can only see 2 differences:
Key difference
I confirmed that the build cache key is different
CI
Local
Appending input file fingerprints for ‘classpath’ to build cache key: 86d4d26077705d61f3ee310d0c7135fe
Appending input file fingerprints for ‘classpath’ to build cache key: 4f4dc77917220498a2e32769b22eb27a
Then, I compared kotlin-stdlib-2.1.10.jar locally and from the CI. The don’t differ except timestamps.
Questions
Can these 2 differences, marked as IGNORED influence the calculation of the build cache key?
If yes, is it possible to “normalize” jars created by a transformation? This is the transformation that results with the above jars: BuildToolsApiClasspathEntrySnapshotTransform. Nevertheless, it’d be strange that only these two jars have different keys, and the rest jars are precisely the same
Is there anything more I can do to address my problem?
Can these 2 differences, marked as IGNORED influence the calculation of the build cache key?
Yes, the IGNORED is the path sensitivity.
That does not change that the content of the file is considered input.
If yes, is it possible to “normalize” jars created by a transformation?
As it is a cacheable transform, I’d guess that the transform should have been taken from cache too. Maybe the inputs for the transform were different?
How the jar file is normalized depends on where the jar is used.
If the input it is used for for example is @Classpath or @CompileClasspath, things like timestamps or file order in the jar is normalized. If it is used as @InputFile, then the file’s actual bytes are the input. In the latter case, the transform would have to make sure to generate binary-identical results, or you would need to chain another transform in, that does the normalization after the actual transform did run.
Nevertheless, it’d be strange that only these two jars have different keys, and the rest jars are precisely the same
Could for example be that - as it is a cacheable transform - you had those two already built locally and put to your cache and the others were retrieved from the remote cache or something like that.
Maybe the inputs for the transform were different?
I think they should be the same: both Kotlin and Okio are external dependencies fetched from Maven.
If the input it is used for for example is @Classpath or @CompileClasspath , things like timestamps or file order in the jar is normalized
I see - this makes sense. The BuildToolsApiClasspathEntrySnapshotTransform input is annotated with @Classpath and it seems to ignore things like timestamp in all other tasks.
Could for example be that - as it is a cacheable transform - you had those two already built locally and put to your cache and the others were retrieved from the remote cache or something like that.
Thank you: this was a good lead! For lack of better ideas, I removed the whole ~/.gradle/caches directory and wrapper files and confirmed this transformation was executed. Then the task started to be fetched from remote cache
Unfortunately, a different task stopped. But I guess the problem is similar, I’ll be debugging it and follow up if it’ll be related.
I’m looking for some help / ideas in debugging my cache misses while using remote build cache on local builds.
Key difference
