Gradle build cache key


(Bob Bell) #1

Is it possible to get access to the value of the Gradle build cache key? In short, I need to utilize a unique version marker – a build fingerprint of sorts. The Gradle build cache would already be turned on. It would be extremely convenient if I could just report out the Gradle build cache key, rather than generating my own value based on basically the same information…


(Stefan Wolf) #2

Hi Bob,

it is currently not possible to access the build cache key for a task. What would you do with the version marker? Put it in the jar file? Choose the file name of the Jar file based on the version marker?

Cheers,
Stefan


(Bob Bell) #3

Hi, @Stefan_Wolf,

We aren’t using JAR files, but basically the latter, I suppose.

Essentially, I’ve been tasked with using Gradle to create a build system to tie together repos that are utilized more or less like a single repo. I had a plan about automatically computing a hash based on the source code, the dependencies, and properties of the build, saving the results to a server, pulling them down instead of building if a version can be found matching all the same inputs, etc. … and then I discovered the Gradle build cache, which is essentially already doing all of that. Win! We can let Gradle handle all of that, leaving less for use to maintain.

However, the Gradle build cache is good for transparent caching of these inputs and outputs. That works great for things that are a “means to an end”. However, for that end product (and maybe a few selected “along the way” artifacts), there is independent value. That is, some subset of the artifacts are things someone may want to be able to independently download. Therefore, they should be explicitly published and retrievable, not just saved in the more ethereal build cache.

So for those artifacts, I’m back to needing an automatic version to assign to them when publishing them. The filenames used as inputs and outputs don’t necessarily need to change (I don’t want to break caching), but I need a unique version under which they can be published, so that they are can retrieved later. I didn’t think the build cache key was currently accessible, but it would do the job, hence my asking.

Thanks,
Bob


(Stefan Wolf) #4

Hi Bob,

the build cache key only is in place for a single task, so not for the whole build. So you would need to somehow create a task which takes all the inputs of the build as an input. That does not sound very desirable. How about you use the git commit id as version for the published artifact?

Did you look at composite builds? Using composite builds together with the build cache may do what you want. There is also some work going on around source dependencies (for an example see here). Though this feature is not polished yet and may still undergo some changes.

Cheers,
Stefan


(Bob Bell) #5

Indeed, I’ve looked at composite builds, and they understandably have some appeal, particularly because there’s a desire to not always have all the source code present, and they deal well with that case.

However, composite builds seem best when there’s an explicit “hand-off” between repositories. That is, when (in general) one repository periodically produces a “release” to be consumed. Of course, composite builds do an excellent job when you want to check out both the producing and consuming repos and build the current version of each. But the “normal” case when you have just the consuming repo is to use a set version of the producing repo, usually referenced in the Gradle source code on that consuming repo. Please correct me if I’ve misunderstood something!

Unfortunately, that’s not the case for us. Our normal case is really to work from the heads of matching branches from each repository. In order to really fit into the composite build model well, we’d have to automatically set a unique version, and then automatically determine which version to use … and then we are right back into calculating all the build inputs, hashing them, etc., something I was hoping to just let Gradle handle, since I don’t want to have more unique code than is necessary. And for the most part, I think we can let Gradle handle that.

It’s not actually a challenge for us to have a task that consumes all of the inputs. In our use case, Gradle is just being used as the “glue” that connects together these repositories. They (unfortunately) do not use Gradle themselves. So for each repo there is really just one (common) task for “build the repo”, with all of the inputs and outputs collected up. Gradle doesn’t really have insight into how that might happy, which could be highly variable between repos.

It occurs to me that in my use case, when I want to publish those “selected” artifacts, I might be better off with a version based on the CI build number. I know I don’t want to use inputs/outputs with the build number in the filenames, because I would never get the Gradle cache hits that I want. However, I believe that I could produce a foo.tar.gz output, and then – if it’s one of the “select few” important artifacts – publish it as version 123 of foo.

Again, I appreciate your response, and please let me know if I am off-target with this.

Thanks again,
Bob


(Bob Bell) #6

FYI, this has come up again. A senior engineer was looking to create a unique hash value to use on every Docker image. He created his own value, when the build cache key would have been as good, or better.