Hi,
I’m looking for a way to change the value for cacheMissingModulesAndArtifactsFor.
It’s possible to change cacheChangingModulesFor and cacheDynamicVersionsFor, but it doesn’t seem to be possible to change cacheMissingModulesAndArtifactsFor.
Would this require a change in Gradle or might there be a way to work around this?
Thanks!
I ended up doing this:
‘’’ configurations.all {
resolutionStrategy.getCachePolicy().eachModule({ module ->
if (module.getCachedResult() == null) {
module.cacheFor(50, ‘years’);
}
} as Action)
resolutionStrategy.getCachePolicy().eachArtifact({ artifact ->
if (artifact.getCachedResult() == null) {
artifact.cacheFor(50, ‘years’);
}
} as Action) } ‘’’
Not sure if it worked yet, I guess I’ll know in 24 hours… 
Unfortunately, that did not seem to do the trick. 24 hours later, I got a bunch of “Resource missing.” again… 
So our situation is that we have a number of repositories (say 5) and for a few of them we have a number of Ivy patterns to try and for each one it takes a while to get the 404. So once per day we take a 30 minute hit when building our projects.
As far as I can understand, what’s happening is that it rechecks the repos that didn’t have the dependency, although we have one repo (maybe further down the list) that actually does have the dependency downloaded already. It would be great if we could check for this fact before attempting to refresh. To be clear, these are all release versions and not snapshots or dynamic versions.
Any ideas?
Thanks!
“I like hearing myself talk. It is one of my greatest pleasures. I often have long conversations all by myself, and I am so clever that I sometimes don’t understand a single word of what I am saying.”
― Oscar Wilde, The Remarkable Rocket
I figured out that adding the same block of code into the buildscript block removed the last "Resource missing"s.
Two questions remain: 1. Would it be an easy fix to make this configurable? 2. Would it make sense to first check if an artifact exists (cached) in any of the repositories before re-checking repositories where the artifact is missing?
Thanks!
Sounds like you’ve got a pretty good understanding of the way Gradle checks for missing modules and artifacts.
The problem comes down to the following: 1. The order of declared repositories is significant, so early repositories should be used in preference to later. 2. Gradle wants to provide consistent results for dependency resolution. This should not be affected by the state of the cache, within reasonable timeout periods.
So to answer your specific questions: - Yep, I think it makes sense to make this value configurable at the same level as ‘cacheChangingModulesFor’. - Possibly, we should always use a module we found previously in preference to checking for one where we previously detected it’s absence. But this would require us to re-examine the assumption mentioned in 3 above.
I’ve raised GRADLE-3107 for the first point. Would you be interested in working with us to provide a pull request?
Thanks for your response!
I’d be happy to create a pull request!
In regards to checking for a locally cached dependency first, I actually think this makes the “repeatable builds” promise stronger. If build x used a dependency from repo b (because it didn’t exist in a) and then when we run build x+1, it’s probably better to use the already cached dependency in repo b rather than downloading it from repo a if it now suddenly exists there.
To be clear, we can (and should) still consider the absence of a module as transient, but if we have already downloaded that module (albeit from a different repo), we should treat this exact version as the source of truth (at least for this particular build). Does this make sense?
I’ve pushed a fix here: https://github.com/viktornordling/gradle/commit/59481bb20ab73ea101b3df8b9943267d6c0789df, happy to create a pull request, but maybe I need to fix up a few more things first?
Hi @yorkish,
Pull requests can be changed once they have been submitted. I’d recommend going ahead and submitting the PR, then we can review it and work with you on getting it in. It’s easier for us to do this once it’s a PR.
Cool, I’m going ahead then. Thanks!
When we refer to “repeatable builds”, we are thinking not about “repeatable on my machine” but “repeatable on any machine”. So what’s in your cache should in general have no effect on the build outcome over some performance improvements.
While preferring a locally cached dependency might make a build repeatable on a particular machine, it increases the chance that somebody else will have a different experience.
Yay!
I’m very happy about that, and also happy to take a look at how to implement this, but I haven’t dug into that part of the code yet, so it might take me a while to get up to speed.
In terms of release management, could it make sense to accept my pull request for a “short term fix” if it might take some time before we’ve implemented and released this new behaviour?
No, we’re pretty strongly against adding extra configuration options in this area. So you’ll need to stick with the ‘resolutionStrategy.getCachePolicy().eachModule’ way of doing this in the meantime.
Hi again. I’ve been discussing this issue with Adam, in particular whether we should always use a module we found previously (in any repository) in preference to checking periodically checking the repositories in order.
Through this discussion we’ve come to the conclusion that the current behaviour is not really beneficial or necessary, so we’re happy to switch to only re-checking missing modules when the module has not been found in another repository. This will effectively set the cache expiry for missing modules and artifacts to “forever”.
If you’re interested in helping out with this that would be great. Thanks again for your thorough analysis.