Excessive downloads during resolution

Benjamin_Manes · June 20, 2015, 4:36am

The following script unnecessarily results in Gradle downloading irrelevant snapshot version metadata when resolving the latest version. This appears to be triggered only when the jcenter() repository is included, as if excluded the expected behavior is exhibited. I cannot find any reason in the repository metadata that triggers this bug.

This is a minimalistic snippet from the gradle-versions-plugin, which uses the resolution results to report potential version upgrades. The Gradle cache effectively makes this a one time cost, but the excessive downloads and the sequential http requests (GRADLE-2582) results in a poor initial experience.

Running gradle resolve with JCenter takes 2m9s and without 6s.

import static org.gradle.api.specs.Specs.SATISFIES_ALL

repositories {
  jcenter()
  maven { url "https://repository.apache.org/content/repositories/snapshots/" }
  maven { url "https://repository.apache.org/content/repositories/releases/" }
}

task resolve() {
  doFirst {
    def deps = [
      dependencies.create("org.apache.camel:camel-core:latest.milestone") {
        transitive = false
      }
    ] as Dependency[]
    def conf = project.configurations.detachedConfiguration(deps)
    def resolved = conf.resolvedConfiguration.lenientConfiguration
    resolved.getFirstLevelModuleDependencies(SATISFIES_ALL)
  }
}

mark_vieira · June 23, 2015, 7:19am

Thanks for the report, I’ve raised GRADLE-3309 to track this.

daz · June 23, 2015, 6:16pm

This is an interesting case, and exposes an inefficiency in the way we determine the ‘status’ for a Maven repository. However, the issue is not really related to the presence of the jcenter repository.

The underlying problem is that the POM files for camel-core reference parent POM files that are not included in the maven repositories you declare (the missing grandparent is org.optaplanner:optaplanner-bom). You can see the failure if you don’t use a LenientConfiguration to get the result.

So this is the sequence:

Gradle lists all of the versions in a repository
For `latest.milestone’, Gradle needs to check each of these versions to see if it has ‘milestone’ status, which means resolving the module metadata.
When jcenter is missing, Gradle fails to resolve the module metadata (due to missing parent POM), and aborts the process. This is why only a single version is downloaded.
When jcenter is present, Gradle resolves the module metadata of each version, only to find that it is not a ‘milestone’ release, and moves on to try the next available version.

Ideally, Gradle wouldn’t download the POM to determine the status of a Maven module, since it can be determined from the version. But at the point we make this decision we are treating all modules the same way, and not referring to the backing repository type.

daz · June 23, 2015, 6:26pm

On more point. The reason that this seems very inefficient in this case is the separation of SNAPSHOT and release versions into separate repositories. Gradle iterates overs all versions in one repository before moving onto the next. If all versions were in the same repository, Gradle would stop at the first version with non-SNAPSHOT version.

The way we’d probably fix this is to change resolution so that we first list all versions from all repositories, then sort the combined list of all versions (newest first), and finally iterate until we find the first one that satisfies the dependency requirement. If we did that, we wouldn’t download POM files for SNAPSHOTs that are older than the newest release version.

Benjamin_Manes · June 23, 2015, 7:33pm

The fix is what most of my users imagine that the plugin is doing, not realizing that its delegating all of the hard work to Gradle. They think that “release”, “milestone”, and “integration” are labels for filters that I designed, and often complain that versions including the string “beta”, “rc”, “release”, etc. are not properly sorted / excluded. Of course Gradle is doing the right thing though, since those version strings do not follow the Maven/Ivy conventions.

Is there any way to coerce Gradle to reject versions that follow a naming pattern and continue the search until it finds an acceptable version? I have tried ComponentSelectionRules but rejecting a candidate does not lead to the next version being evaluated, but instead an unresolved dependency. Alternatively if there was a way to receive the version history, e.g. through the ArtifactResolutionQuery, then I could sort it out directly as an extension to the internal VersionComparator.

daz · June 23, 2015, 9:39pm

ComponentSelectionRules won’t let you do this directly: if you specify ‘latest.milestone’ as your selector, then the rule will only be evaluated for versions that have been determined to match that version. So you can do further filtering to reject versions after Gradle has accepted them, but you can’t access versions that do not match the selector.

If you use a selector like ‘latest.integration’ combined with a ComponentSelectionRule you would have access to every version, but not the metadata to determine the component status. If your rule has ‘ComponentModuleMetadata’ as an input parameter you’ll be able to see the status, but you’ll be back at downloading the POM for every version in order (until you accept one).

Benjamin_Manes · June 23, 2015, 11:50pm

It appears that component rules and version statuses do no work together. This explains why I was so puzzled by rules when I originally experimented with them, as this quirk causes only one candidate to be evaluated.

In the following example we restrict ourselves to only release candidates versions (X.Y-rc), to make it easy to run this on real data. The latest release version is 18.0 and the last candidate was 18.0-rc2. We should expect the resolution to reject 18.0 and try the next candidate. You can run the example using gradle dependencies -i --refresh-dependencies.

When the dependency’s version is declared as latest.integration only a single candidate is evaluated, rejected, and the resolution fails. However if we change the version to + then Gradle tries the next candidate, 18.0-rc2, and successfully resolves the dependency.

Is this a bug? Should I switch the versions plugin to use + instead of “latest.${status}”?

repositories {
  jcenter()
}

configurations {
  release_candidate_only {
    resolutionStrategy {
      componentSelection {
        all { ComponentSelection selection ->
          if (!selection.candidate.version.contains('rc')) {
            selection.reject("Expected 'rc' in version")
          }
        }
      }
    }
  }
}

dependencies {
  release_candidate_only 'com.google.guava:guava:latest.integration'
}

Benjamin_Manes · June 24, 2015, 4:19am

I converted over to using + and a selection rule to reject based on the metadata status. This works beautifully, thanks!

daz · June 24, 2015, 7:11pm

This isn’t a bug, but is instead a subtle difference between declaring ‘1.+’ and ‘latest.integration’.

If you state that you depend on ‘1.+’, you are effectively stating that you are compatible with any version in the 1.x stream. So any of [1.0, 1.1, …] will satisfy that dependency.

If you state that you depend on ‘latest.integration’, there is only 1 version that satisfies that requirement. Only 1 version can be the ‘latest’. This is why Gradle doesn’t present other versions to the ComponentSelectionRule: the other versions don’t meet the dependency requirement.

Hope that helps make sense of it.

Benjamin_Manes · June 24, 2015, 7:15pm

That makes perfect sense, thanks. It would be helpful to put that explanation in either the documentation or the exception that UnresolvedDependency carries.