Artifact resolution failure with multiple Maven resolvers


(Alex Varju) #1

Hi there,

We’ve been seeing some sporadic failures that we believe to be a bug, but we’re not 100% confident that we aren’t doing something wrong.

Gradle Version: 3.1
Operating System and JVM version: macOS 10.12
Is this a regression? Yes. Worked on Gradle 2.9.

Our setup involves multiple external maven repositories. In this case, we’ve got an artifact that only exists in the second registered repository (‘public’), but Gradle is sometimes skipping that repository, instead returning the cached ‘missing artifact’ response from the first repository. We think the failure is sporadic because of timing issues; we can see a timestamp check in the caching code.

Sample debug output for a case where it failed:

[DEBUG] [org.gradle.api.internal.artifacts.ivyservice.resolveengine.graph.DependencyGraphBuilder] Visiting dependency learn.install:birt:unspecified(birt) -> dependency: com.lowagie.text:text:2.1.7.v201004222200 from-conf: birt to-conf: default
[DEBUG] [org.gradle.api.internal.artifacts.ivyservice.resolveengine.graph.DependencyGraphBuilder] Selecting new module version com.lowagie.text:text:2.1.7.v201004222200
[DEBUG] [org.gradle.api.internal.artifacts.ivyservice.ivyresolve.RepositoryChainComponentMetaDataResolver] Attempting to resolve component for com.lowagie.text:text:2.1.7.v201004222200 using repositories [integrations, public, MavenRepo]
[DEBUG] [org.gradle.api.internal.artifacts.repositories.resolver.DefaultExternalResourceArtifactResolver] Loading http://maven.bbpd.io/content/repositories/integrations/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.pom
[DEBUG] [org.gradle.internal.resource.transfer.DefaultCacheAwareExternalResourceAccessor] Constructing external resource: http://maven.bbpd.io/content/repositories/integrations/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.pom
[DEBUG] [org.gradle.internal.resource.transport.http.HttpResourceAccessor] Constructing external resource metadata: http://maven.bbpd.io/content/repositories/integrations/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.pom
[DEBUG] [org.gradle.internal.resource.transport.http.HttpClientHelper] Performing HTTP HEAD: http://maven.bbpd.io/content/repositories/integrations/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.pom
[DEBUG] [org.apache.http.impl.execchain.MainClientExec] Executing request HEAD /content/repositories/integrations/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.pom HTTP/1.1
[INFO] [org.gradle.internal.resource.transport.http.HttpClientHelper] Resource missing. [HTTP HEAD: http://maven.bbpd.io/content/repositories/integrations/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.pom]
[DEBUG] [org.gradle.api.internal.artifacts.repositories.resolver.DefaultExternalResourceArtifactResolver] Loading http://maven.bbpd.io/content/repositories/integrations/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.jar
[DEBUG] [org.gradle.internal.resource.transport.http.HttpResourceAccessor] Constructing external resource metadata: http://maven.bbpd.io/content/repositories/integrations/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.jar
[DEBUG] [org.gradle.internal.resource.transport.http.HttpClientHelper] Performing HTTP HEAD: http://maven.bbpd.io/content/repositories/integrations/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.jar
[DEBUG] [org.apache.http.impl.execchain.MainClientExec] Executing request HEAD /content/repositories/integrations/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.jar HTTP/1.1
[INFO] [org.gradle.internal.resource.transport.http.HttpClientHelper] Resource missing. [HTTP HEAD: http://maven.bbpd.io/content/repositories/integrations/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.jar]
[DEBUG] [org.gradle.api.internal.artifacts.repositories.resolver.ExternalResourceResolver] No meta-data file or artifact found for module 'com.lowagie.text:text:2.1.7.v201004222200' in repository 'integrations'.

// This time, it didn’t know that it wasn’t in ‘integrations’ so it checked and recorded that it didn’t exist
[DEBUG] [org.gradle.api.internal.artifacts.ivyservice.modulecache.DefaultModuleMetaDataCache] Recording absence of module descriptor in cache: com.lowagie.text:text:2.1.7.v201004222200 [changing = false]

// But then decided that it didn’t exist in public either.
[DEBUG] [org.gradle.api.internal.artifacts.ivyservice.ivyresolve.CachingModuleComponentRepository] Detected non-existence of module 'com.lowagie.text:text:2.1.7.v201004222200' in resolver cache 'public'

[DEBUG] [org.gradle.api.internal.artifacts.ivyservice.ivyresolve.CachingModuleComponentRepository] Detected non-existence of module 'com.lowagie.text:text:2.1.7.v201004222200' in resolver cache 'MavenRepo'
[DEBUG] [org.gradle.api.internal.artifacts.repositories.resolver.DefaultExternalResourceArtifactResolver] Loading https://repo1.maven.org/maven2/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.pom
[DEBUG] [org.gradle.internal.resource.transfer.DefaultCacheAwareExternalResourceAccessor] Constructing external resource: https://repo1.maven.org/maven2/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.pom
[DEBUG] [org.gradle.internal.resource.transport.http.HttpResourceAccessor] Constructing external resource metadata: https://repo1.maven.org/maven2/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.pom
[DEBUG] [org.gradle.internal.resource.transport.http.HttpClientHelper] Performing HTTP HEAD: https://repo1.maven.org/maven2/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.pom
[DEBUG] [org.apache.http.impl.execchain.MainClientExec] Executing request HEAD /maven2/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.pom HTTP/1.1
[INFO] [org.gradle.internal.resource.transport.http.HttpClientHelper] Resource missing. [HTTP HEAD: https://repo1.maven.org/maven2/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.pom]
[DEBUG] [org.gradle.api.internal.artifacts.repositories.resolver.DefaultExternalResourceArtifactResolver] Loading https://repo1.maven.org/maven2/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.jar
[DEBUG] [org.gradle.internal.resource.transport.http.HttpResourceAccessor] Constructing external resource metadata: https://repo1.maven.org/maven2/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.jar
[DEBUG] [org.gradle.internal.resource.transport.http.HttpClientHelper] Performing HTTP HEAD: https://repo1.maven.org/maven2/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.jar
[DEBUG] [org.apache.http.impl.execchain.MainClientExec] Executing request HEAD /maven2/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.jar HTTP/1.1
[INFO] [org.gradle.internal.resource.transport.http.HttpClientHelper] Resource missing. [HTTP HEAD: https://repo1.maven.org/maven2/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.jar]
[DEBUG] [org.gradle.api.internal.artifacts.repositories.resolver.ExternalResourceResolver] No meta-data file or artifact found for module 'com.lowagie.text:text:2.1.7.v201004222200' in repository 'MavenRepo'.
[DEBUG] [org.gradle.api.internal.artifacts.ivyservice.modulecache.DefaultModuleMetaDataCache] Recording absence of module descriptor in cache: com.lowagie.text:text:2.1.7.v201004222200 [changing = false]
[ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] > Could not find com.lowagie.text:text:2.1.7.v201004222200.
[ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]       http://maven.bbpd.io/content/repositories/integrations/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.pom
[ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]       http://maven.bbpd.io/content/repositories/integrations/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.jar
[ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]       https://repo1.maven.org/maven2/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.pom
[ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]       https://repo1.maven.org/maven2/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.jar
[ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Caused by: org.gradle.internal.resolve.ModuleVersionNotFoundException: Could not find com.lowagie.text:text:2.1.7.v201004222200.
[ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]     http://maven.bbpd.io/content/repositories/integrations/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.pom
[ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]     http://maven.bbpd.io/content/repositories/integrations/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.jar
[ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]     https://repo1.maven.org/maven2/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.pom
[ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]     https://repo1.maven.org/maven2/com/lowagie/text/text/2.1.7.v201004222200/text-2.1.7.v201004222200.jar

We’ve been experimenting with the following change locally, which we appears to fix the problem, although we don’t actually think this change should be required:

diff --git a/subprojects/dependency-management/src/main/java/org/gradle/api/internal/artifacts/repositories/resolver/ExternalResourceResolver.java b/subprojects/dependency-management/src/main/java/org/gradle/api/internal/artifacts/repositories/resolver/ExternalResourceResolver.java
index e13300d..89ad8c1 100644
--- a/subprojects/dependency-management/src/main/java/org/gradle/api/internal/artifacts/repositories/resolver/ExternalResourceResolver.java
+++ b/subprojects/dependency-management/src/main/java/org/gradle/api/internal/artifacts/repositories/resolver/ExternalResourceResolver.java
@@ -487,6 +487,8 @@ public abstract class ExternalResourceResolver<T extends ModuleComponentResolveM
     private static String generateId(ExternalResourceResolver resolver) {
         StringBuilder sb = new StringBuilder(resolver.getClass().getName());
         sb.append("::");
+        sb.append(resolver.name);
+        sb.append("::");
         joinPatterns(sb, resolver.ivyPatterns);

We’re pretty sure that there is a cache index corruption involved. With a breakpoint in BTreePersistentIndexedCache.doOpen(), a call to this.verify() failed with “DataBlock 18497 overlaps with DataBlock 18497”. If the index itself is useful, it is available here: https://dl.dropboxusercontent.com/u/21013992/module-metadata.bin

We’ve tried deleting the cache indexes, but eventually this problem appears to re-occur. Ours is a fairly large multi-project build, and we’re suspicious that parallel compilation is leading to index corruptions.