Dependency resolution improvements


#1

Over the past couple of weeks we’ve made some significant improvements to the way dependency resolution works in Gradle. There are some things that are just refactorings, some performance improvements, and a couple of new features. These changes should make it into M6, which is due out soon.

Caching Improvements: -------------------- A big change is that Gradle now takes control of the caching of Dynamic Versions (eg 1.+) and Changing Modules (eg SNAPSHOT). A new metadata file in the cache, “dynamic-versions.bin”, caches information about module resolution on a per-resolver basis.

Dynamic Versions:

If a resolver tells us that 1.+ is actually 1.5, we remember that for next time. That way, we can look directly in the module cache to get the descriptor for 1.+, rather than first having to resolve 1.+ to 1.5.

Dynamic Version cache entries expire by default in 24 hours. This value can be modified using

‘configuration.resolutionStrategy.cacheDynamicVersionsFor 2, “minutes”’

Changing Modules: Similarly, if a dependency is flagged as changing, or a resolver tells us the module is changing (eg SNAPSHOT), we remember when we last fetched the module meta-data and artifacts. If we find a cache entry indicating it’s ok to use the cached version, we use them. Otherwise, we bypass the cache and download the descriptor and artifacts again.

Changing Module cache entries expire by default in 24 hours. This value can be modified using ‘configuration.resolutionStrategy.cacheChangingModulesFor 2, “minutes”’

Maven Snapshots: By implementing these resolution caches ourselves, we have been able to remove the previous snapshot ttl that was provided by the WharfIbiblioResolver. This has greatly simplified things internally.

Offline Mode: Although nothing official has been added, it should be possible to add an offline switch to your build that sets these cache values very high. In theory, no requests should be required when everything has be previously retrieved. An official way to run your build offline is planned for 1.0.

You can read more about the support for Dynamic Versions and Changing Modules in the docs.

Performance Improvements: ------------------------ Although the new caching improvements help a lot with performance when things have been resolved before, we’ve also done some work to improve the performance when resolving dependencies against maven repositories. Namely:

  • Don’t request javadoc and source jars when resolving the POM - Don’t request .sha1 and .md5 files when not required - Don’t use HTTP HEAD to check existence of file prior to downloading them.

Together, these improvements have reduced the number of HTTP requests required to resolve a static module from 11 down to 2, and a simple SNAPSHOT module from 25 requests down to 4!

Other Refactoring: ------------- We spent a bunch of time replacing the core Ivy ResolveEngine with our own implementation. This has already proven beneficial in making the other improvements simpler.

We’ve also started using our own from-scratch implementation of org.apache.ivy.plugins.resolver.DependencyResolver, in place of extending AbstractResolver, BasicResolver, ChainResolver etc. Our own implementations need only implement 3 key methods on the interface, and removing a whole bunch of unnecessary (and misunderstood) complexity in the process.

Another improvement has been to remove some of the power of the low-level ivy resolvers that perform the actual resolution. Previously, each resolver could look at what had been resolved earlier in the chain, and determine if it do anything or accept the earlier resolution. Now each resolver behaves independently, with a higher level controller doing the work of choosing the best resolved version, short-circuiting resolution when looking up static versions.


(Liyu Wang) #2

I have problem with the new release, it resolved the depenencies different with any old version, and it cause some trouble, I have create the ticket GRADLE-1944


#3

Are there any plans to implement a scheme whereby dynamic version such as 1+ would always result in a valid configuration, even if this result did not yield the latest version of all components?

Typical example would be if there were multiple versions of the artifact in a binary repository that satisfied the dynamic version selector, but the latest could not be selected because it in turn selects versions which, if selected, would yield a conflicting result due to other dependents selecting a different version.


(Gábor Guta) #4

Hi, It would be nice to allow users by default to have direct control over dependency resolution. i.e. fix the set of the resolved dependency combination and stay in offline mode until the the developer explicitly requests dependency update (similar way as he/she updates the source from the SCM). Typically developers don’t want to update the dynamic dependency at each compile.


(Szczepan Faber) #5

Typically developers don’t want to update the dynamic dependency at each compile.

Current Gradle’s default for snapshot dependencies / dynamic versions is 24 hours. It is also configurable. Hope that helps!