Artifact resolution failure with multiple Maven resolvers

varju · November 10, 2016, 11:08pm

Hi there,

We’ve been seeing some sporadic failures that we believe to be a bug, but we’re not 100% confident that we aren’t doing something wrong.

Gradle Version: 3.1
Operating System and JVM version: macOS 10.12
Is this a regression? Yes. Worked on Gradle 2.9.

Our setup involves multiple external maven repositories. In this case, we’ve got an artifact that only exists in the second registered repository (‘public’), but Gradle is sometimes skipping that repository, instead returning the cached ‘missing artifact’ response from the first repository. We think the failure is sporadic because of timing issues; we can see a timestamp check in the caching code.

Sample debug output for a case where it failed:

[DEBUG] [org.gradle.api.internal.artifacts.ivyservice.resolveengine.graph.DependencyGraphBuilder] Visiting dependency learn.install:birt:unspecified(birt) -> dependency: com.lowagie.text:text:2.1.7.v201004222200 from-conf: birt to-conf: default
[DEBUG] [org.gradle.api.internal.artifacts.ivyservice.resolveengine.graph.DependencyGraphBuilder] Selecting new module version com.lowagie.text:text:2.1.7.v201004222200
[DEBUG] [org.gradle.api.internal.artifacts.ivyservice.ivyresolve.RepositoryChainComponentMetaDataResolver] Attempting to resolve component for com.lowagie.text:text:2.1.7.v201004222200 using repositories [integrations, public, MavenRepo]
[DEBUG] [org.gradle.api.internal.artifacts.repositories.resolver.DefaultExternalResourceArtifactResolver] Loading http://maven.bbpd.io/content/repositories/integrations/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.pom
[DEBUG] [org.gradle.internal.resource.transfer.DefaultCacheAwareExternalResourceAccessor] Constructing external resource: http://maven.bbpd.io/content/repositories/integrations/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.pom
[DEBUG] [org.gradle.internal.resource.transport.http.HttpResourceAccessor] Constructing external resource metadata: http://maven.bbpd.io/content/repositories/integrations/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.pom
[DEBUG] [org.gradle.internal.resource.transport.http.HttpClientHelper] Performing HTTP HEAD: http://maven.bbpd.io/content/repositories/integrations/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.pom
[DEBUG] [org.apache.http.impl.execchain.MainClientExec] Executing request HEAD /content/repositories/integrations/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.pom HTTP/1.1
[INFO] [org.gradle.internal.resource.transport.http.HttpClientHelper] Resource missing. [HTTP HEAD: http://maven.bbpd.io/content/repositories/integrations/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.pom]
[DEBUG] [org.gradle.api.internal.artifacts.repositories.resolver.DefaultExternalResourceArtifactResolver] Loading http://maven.bbpd.io/content/repositories/integrations/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.jar
[DEBUG] [org.gradle.internal.resource.transport.http.HttpResourceAccessor] Constructing external resource metadata: http://maven.bbpd.io/content/repositories/integrations/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.jar
[DEBUG] [org.gradle.internal.resource.transport.http.HttpClientHelper] Performing HTTP HEAD: http://maven.bbpd.io/content/repositories/integrations/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.jar
[DEBUG] [org.apache.http.impl.execchain.MainClientExec] Executing request HEAD /content/repositories/integrations/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.jar HTTP/1.1
[INFO] [org.gradle.internal.resource.transport.http.HttpClientHelper] Resource missing. [HTTP HEAD: http://maven.bbpd.io/content/repositories/integrations/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.jar]
[DEBUG] [org.gradle.api.internal.artifacts.repositories.resolver.ExternalResourceResolver] No meta-data file or artifact found for module 'com.lowagie.text:text:2.1.7.v201004222200' in repository 'integrations'.

// This time, it didn’t know that it wasn’t in ‘integrations’ so it checked and recorded that it didn’t exist
[DEBUG] [org.gradle.api.internal.artifacts.ivyservice.modulecache.DefaultModuleMetaDataCache] Recording absence of module descriptor in cache: com.lowagie.text:text:2.1.7.v201004222200 [changing = false]

// But then decided that it didn’t exist in public either.
[DEBUG] [org.gradle.api.internal.artifacts.ivyservice.ivyresolve.CachingModuleComponentRepository] Detected non-existence of module 'com.lowagie.text:text:2.1.7.v201004222200' in resolver cache 'public'

[DEBUG] [org.gradle.api.internal.artifacts.ivyservice.ivyresolve.CachingModuleComponentRepository] Detected non-existence of module 'com.lowagie.text:text:2.1.7.v201004222200' in resolver cache 'MavenRepo'
[DEBUG] [org.gradle.api.internal.artifacts.repositories.resolver.DefaultExternalResourceArtifactResolver] Loading https://repo1.maven.org/maven2/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.pom
[DEBUG] [org.gradle.internal.resource.transfer.DefaultCacheAwareExternalResourceAccessor] Constructing external resource: https://repo1.maven.org/maven2/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.pom
[DEBUG] [org.gradle.internal.resource.transport.http.HttpResourceAccessor] Constructing external resource metadata: https://repo1.maven.org/maven2/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.pom
[DEBUG] [org.gradle.internal.resource.transport.http.HttpClientHelper] Performing HTTP HEAD: https://repo1.maven.org/maven2/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.pom
[DEBUG] [org.apache.http.impl.execchain.MainClientExec] Executing request HEAD /maven2/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.pom HTTP/1.1
[INFO] [org.gradle.internal.resource.transport.http.HttpClientHelper] Resource missing. [HTTP HEAD: https://repo1.maven.org/maven2/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.pom]
[DEBUG] [org.gradle.api.internal.artifacts.repositories.resolver.DefaultExternalResourceArtifactResolver] Loading https://repo1.maven.org/maven2/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.jar
[DEBUG] [org.gradle.internal.resource.transport.http.HttpResourceAccessor] Constructing external resource metadata: https://repo1.maven.org/maven2/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.jar
[DEBUG] [org.gradle.internal.resource.transport.http.HttpClientHelper] Performing HTTP HEAD: https://repo1.maven.org/maven2/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.jar
[DEBUG] [org.apache.http.impl.execchain.MainClientExec] Executing request HEAD /maven2/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.jar HTTP/1.1
[INFO] [org.gradle.internal.resource.transport.http.HttpClientHelper] Resource missing. [HTTP HEAD: https://repo1.maven.org/maven2/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.jar]
[DEBUG] [org.gradle.api.internal.artifacts.repositories.resolver.ExternalResourceResolver] No meta-data file or artifact found for module 'com.lowagie.text:text:2.1.7.v201004222200' in repository 'MavenRepo'.
[DEBUG] [org.gradle.api.internal.artifacts.ivyservice.modulecache.DefaultModuleMetaDataCache] Recording absence of module descriptor in cache: com.lowagie.text:text:2.1.7.v201004222200 [changing = false]
[ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] > Could not find com.lowagie.text:text:2.1.7.v201004222200.
[ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]       http://maven.bbpd.io/content/repositories/integrations/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.pom
[ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]       http://maven.bbpd.io/content/repositories/integrations/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.jar
[ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]       https://repo1.maven.org/maven2/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.pom
[ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]       https://repo1.maven.org/maven2/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.jar
[ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Caused by: org.gradle.internal.resolve.ModuleVersionNotFoundException: Could not find com.lowagie.text:text:2.1.7.v201004222200.
[ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]     http://maven.bbpd.io/content/repositories/integrations/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.pom
[ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]     http://maven.bbpd.io/content/repositories/integrations/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.jar
[ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]     https://repo1.maven.org/maven2/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.pom
[ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]     https://repo1.maven.org/maven2/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.jar

We’ve been experimenting with the following change locally, which we appears to fix the problem, although we don’t actually think this change should be required:

diff --git a/subprojects/dependency-management/src/main/java/org/gradle/api/internal/artifacts/repositories/resolver/ExternalResourceResolver.java b/subprojects/dependency-management/src/main/java/org/gradle/api/internal/artifacts/repositories/resolver/ExternalResourceResolver.java
index e13300d..89ad8c1 100644
--- a/subprojects/dependency-management/src/main/java/org/gradle/api/internal/artifacts/repositories/resolver/ExternalResourceResolver.java
+++ b/subprojects/dependency-management/src/main/java/org/gradle/api/internal/artifacts/repositories/resolver/ExternalResourceResolver.java
@@ -487,6 +487,8 @@ public abstract class ExternalResourceResolver<T extends ModuleComponentResolveM
     private static String generateId(ExternalResourceResolver resolver) {
         StringBuilder sb = new StringBuilder(resolver.getClass().getName());
         sb.append("::");
+        sb.append(resolver.name);
+        sb.append("::");
         joinPatterns(sb, resolver.ivyPatterns);

We’re pretty sure that there is a cache index corruption involved. With a breakpoint in BTreePersistentIndexedCache.doOpen(), a call to this.verify() failed with “DataBlock 18497 overlaps with DataBlock 18497”. If the index itself is useful, it is available here: https://dl.dropboxusercontent.com/u/21013992/module-metadata.bin

We’ve tried deleting the cache indexes, but eventually this problem appears to re-occur. Ours is a fairly large multi-project build, and we’re suspicious that parallel compilation is leading to index corruptions.

Topic		Replies	Views
After failing to locate an artifact in remote Ivy repo, Gradle never finds it on subsequent builds Old Forum Archive	13	3756	October 18, 2012
Gradle seems to be slow to resolve dependencies from Maven Nexus Repository Running with --debug shows NullPointerException Old Forum Archive	2	4021	January 8, 2015
Gradle 2.3: ivy configuration mapping makes dependencies resolution to fail (*,!test) Old Forum Archive	3	2083	February 19, 2015
Welcome to the new Gradle Dependency Cache Old Forum Archive	15	6791	June 20, 2012
NPE in DependencyGraphBuilder$DependencyEdge.getFailure Old Forum Archive	11	1053	June 20, 2013

Artifact resolution failure with multiple Maven resolvers

Related Topics