Build Hangs on "Stale Connection Check" On Mac


(gilbode) #1

We’re having issues running Gradle on Mac (Mavericks) where the build seems to take forever. When running gradle with -debug the symptoms are we see the “Stale connection check” message in the log and the build just hangs. It eventually continue but the hang is for something like 5 minutes.

  18:27:08.915 [DEBUG] [org.gradle.api.internal.externalresource.transport.http.HttpResourceAccessor] Constructing external resource metadata: http://MYHOST/myrepo/net/java/jvnet-parent/3/jvnet-parent-3.pom  18:27:08.915 [DEBUG] [org.gradle.api.internal.externalresource.transport.http.HttpClientHelper] Performing HTTP HEAD: http://MYHOST/myrepo/net/java/jvnet-parent/3/jvnet-parent-3.pom  18:27:08.916 [DEBUG] [org.apache.http.impl.conn.PoolingClientConnectionManager] Connection request: [route: {}->http://MYHOST][total kept alive: 1; route allocated: 1 of 5; total allocated: 1 of 10]  18:27:08.916 [DEBUG] [org.apache.http.impl.conn.PoolingClientConnectionManager] Connection leased: [id: 11][route: {}->http://MYHOST][total kept alive: 0; route allocated: 1 of 5; total allocated: 1 of 10]  18:27:08.917 [DEBUG] [org.apache.http.impl.client.SystemDefaultHttpClient] Stale connection check  

This seems to happen especially when traversing POMs. We may have mulitple issues going on but I should note we have a lot of projects following the BOM POM pattern and there are a huge # of artifacts if you follow them recursively. I’ve definitely seen where Gradle recursively traverses the import scoped POMs and attempts to retrieve each POM first as a JAR and have also seen it make HEAD requests for artifacts listed in the dependencyManagement section of those POMs that are never actually referenced. See below for a snippet that occurs just before we get to the stale connection check where Gradle first attempts to retrieve a POM as a JAR:

  18:21:00.490 [DEBUG] [org.gradle.api.internal.externalresource.transfer.DefaultCacheAwareExternalResourceAccessor] Constructing external resource: http://MYHOST/myrepo/org/apache/apache/7/apache-7.pom  18:21:00.491 [DEBUG] [org.gradle.api.internal.artifacts.repositories.resolver.ExternalResourceResolver] Downloading http://MYHOST/myrepo/org/apache/apache/7/apache-7.pom to /private/var/folders/bv/mckp6zk52q1d80bgfs6lc4cc0000gn/T/gradle_download7815650427985613578bin  18:21:00.498 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.ivyresolve.CachingModuleVersionRepository] Downloaded artifact 'org.apache:apache:7:apache.pom' from resolver: maven  18:21:00.517 [DEBUG] [org.gradle.api.internal.artifacts.repositories.resolver.ExternalResourceResolver] Loading http://MYHOST/myrepo/org/apache/commons/commons-parent/17/commons-parent-17.jar  18:21:00.518 [DEBUG] [org.gradle.api.internal.externalresource.transport.http.HttpResourceAccessor] Constructing external resource metadata: http://MYHOST/myrepo/org/apache/commons/commons-parent/17/commons-parent-17.jar  18:21:00.518 [DEBUG] [org.gradle.api.internal.externalresource.transport.http.HttpClientHelper] Performing HTTP HEAD: http://MYHOST/myrepo/org/apache/commons/commons-parent/17/commons-parent-17.jar  18:23:00.992 [INFO] [org.gradle.api.internal.externalresource.transport.http.HttpClientHelper] Resource missing. [HTTP HEAD: http://MYHOST/myrepo/org/apache/commons/commons-parent/17/commons-parent-17.jar]  18:23:00.993 [DEBUG] [org.gradle.api.internal.artifacts.repositories.resolver.ExternalResourceResolver] Resource not reachable for org.apache.commons#commons-parent;17!commons-parent.jar: res=MissingResource: http://MYHOST/myrepo/org/apache/commons/commons-parent/17/commons-parent-17.jar  18:23:00.993 [DEBUG] [org.gradle.api.internal.artifacts.repositories.resolver.ExternalResourceResolver] Ivy file found for module 'org.apache.commons#commons-parent;17' in repository 'maven'.  18:23:00.994 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.modulecache.DefaultModuleMetaDataCache] Recording module descriptor in cache: org.apache.commons#commons-parent;17 [changing = false]  18:23:01.013 [DEBUG] [org.gradle.api.internal.file.copy.DeleteActionImpl] Deleting /Users/mgilbode/.gradle/caches/modules-2/metadata-2.2/descriptors/org.apache.commons/commons-parent/17/c728efc846f4ba1515016aca5acb6e67/ivy.xml.fslck  18:23:01.014 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.ivyresolve.UserResolverChain] Using module 'org.apache.commons:commons-parent:17' from repository 'maven'  18:23:01.015 [DEBUG] [org.gradle.api.internal.artifacts.repositories.resolver.ExternalResourceResolver] Loading http://MYHOST/myrepo/org/apache/commons/commons-parent/17/commons-parent-17.pom  18:23:01.015 [DEBUG] [org.gradle.api.internal.externalresource.transfer.DefaultCacheAwareExternalResourceAccessor] Constructing external resource: http://MYHOST/myrepo/org/apache/commons/commons-parent/17/commons-parent-17.pom  18:23:01.017 [DEBUG] [org.gradle.api.internal.artifacts.repositories.resolver.ExternalResourceResolver] Downloading http://MYHOST/myrepo/org/apache/commons/commons-parent/17/commons-parent-17.pom to /private/var/folders/bv/mckp6zk52q1d80bgfs6lc4cc0000gn/T/gradle_download7200640887084156138bin  18:23:01.030 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.ivyresolve.CachingModuleVersionRepository] Downloaded artifact 'org.apache.commons:commons-parent:17:commons-parent.pom' from resolver: maven  

The only thing that has helped is to add @jar to all of our transitive dependencies + to configure a repository to ignore poms like this, but it’s not ideal:

add(new org.apache.ivy.plugins.resolver.URLResolver()) {
             name = "jarOnlyMavenCentral" // For non standard resolvers the name is required
             m2compatible = true // Tells Ivy to split the organization into subdirs.
            addArtifactPattern('http://MYHOST/myrepo/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]')
         }

This behavior seems to be worst on Mac where builds take hours that should take minutes. Any ideas?


(gilbode) #2

FYI - tracked this down to a JDK bug which exists on both JDK7 & JDK8:

https://bugs.openjdk.java.net/browse/JDK-8036554