Downloading from jcenter can hang in certain circumstances

I have been tracking down this issue for over a week.Ther have been reports from some people on Twitter that Gradle + JCenter is hanging.It seems that requesting non-existant artifacts from Jcenter is the casue. I have also experienced this, but only on one network. Testing on another network was fine. (If at this point, you think it is a network issue, you may be right, but please read on).

I have also been in discussion with the Bintray folks about this, who has been quite helpful to try and track the problem down.

Let’s start with my testscript

buildscript {  
repositories {    
  jcenter()    
  // mavenCentral()
  maven {      url 'https://plugins.gradle.org/m2'   } 
}
 dependencies {   
  // Use something that does not exist on Bintray   
  classpath 'com.gradle.publish:plugin-publish-plugin:0.9.0'    
  }
}

apply plugin : 'com.gradle.plugin-publish'

The test was specific with Gradle 2.4 as that was what would be used on the site where I was. Testing was by running

gradle -d --gradle-user-home $(pwd)/CACHE tasks

cleaning CACHE in between every run.

If jcenter was first then a hang will occur at what the last line of the debug log shows - the HEAD verb.

08:56:55.904 [DEBUG] [org.apache.http.impl.conn.DefaultClientConnection] Sending request: HEAD /com/gradle/publish/plugin-publish-plugin/0.9.0/plugin-publish-plugin-0.9.0.jar HTTP/1.1
08:56:56.053 [DEBUG] [org.apache.http.impl.conn.DefaultClientConnection] Receiving response: HTTP/1.1 302 
08:56:56.055 [DEBUG] [org.apache.http.impl.client.SystemDefaultHttpClient] Connection can be kept alive indefinitely
08:56:56.056 [DEBUG] [org.apache.http.impl.client.DefaultRedirectStrategy] Redirect requested to location 'http://repo.jfrog.org/artifactory/libs-release-bintray/com/gradle/publish/plugin-publish-plugin/0.9.0/plugin-publish-plugin-0.9.0.jar'
08:56:56.057 [DEBUG] [org.apache.http.impl.client.SystemDefaultHttpClient] Resetting target auth state
08:56:56.057 [DEBUG] [org.apache.http.impl.client.SystemDefaultHttpClient] Redirecting to 'http://repo.jfrog.org/artifactory/libs-release-bintray/com/gradle/publish/plugin-publish-plugin/0.9.0/plugin-publish-plugin-0.9.0.jar' via {}->http://repo.jfrog.org
08:56:56.058 [DEBUG] [org.apache.http.impl.conn.PoolingClientConnectionManager] Connection [id: 0][route: {s}->https://jcenter.bintray.com] can be kept alive indefinitely
08:56:56.059 [DEBUG] [org.apache.http.impl.conn.PoolingClientConnectionManager] Connection released: [id: 0][route: {s}->https://jcenter.bintray.com][total kept alive: 2; route allocated: 1 of 5; total allocated: 2 of 10]
08:56:56.059 [DEBUG] [org.apache.http.impl.conn.PoolingClientConnectionManager] Connection request: [route: {}->http://repo.jfrog.org][total kept alive: 2; route allocated: 1 of 5; total allocated: 2 of 10]
08:56:56.060 [DEBUG] [org.apache.http.impl.conn.PoolingClientConnectionManager] Connection leased: [id: 1][route: {}->http://repo.jfrog.org][total kept alive: 1; route allocated: 1 of 5; total allocated: 2 of 10]
08:56:56.060 [DEBUG] [org.apache.http.impl.client.SystemDefaultHttpClient] Stale connection check
08:56:56.063 [DEBUG] [org.apache.http.client.protocol.RequestAddCookies] CookieSpec selected: best-match
08:56:56.064 [DEBUG] [org.apache.http.client.protocol.RequestAuthCache] Auth cache not set in the context
08:56:56.065 [DEBUG] [org.apache.http.client.protocol.RequestTargetAuthentication] Target auth state: UNCHALLENGED
08:56:56.066 [DEBUG] [org.apache.http.client.protocol.RequestProxyAuthentication] Proxy auth state: UNCHALLENGED
08:56:56.066 [DEBUG] [org.apache.http.impl.client.SystemDefaultHttpClient] Attempt 2 to execute request
08:56:56.067 [DEBUG] [org.apache.http.impl.conn.DefaultClientConnection] Sending request: HEAD /artifactory/libs-release-bintray/com/gradle/publish/plugin-publish-plugin/0.9.0/plugin-publish-plugin-0.9.0.jar HTTP/1.1

If one runs curl --head http://repo.jfrog.org/artifactory/libs-release-bintray/com/gradle/publish/plugin-publish-plugin/0.9.0/plugin-publish-plugin-0.9.0.jar it does work.

If one now disables jcenter() in the script and enables mavenCentral() everything works fine. To a great extent this eliminates most trivial cases against it being a network problem, but not quite.

The most interesting part of this is when looking at the network trace

***.***.***.191.32885-054.086.126.152.00080: GET /artifactory/libs-release-bintray/com/gradle/publish/plugin-publish-plugin/0.9.0/plugin-publish-plugin-0.9.0.pom HTTP/1.1
 Accept-Encoding: gzip,deflate
 Host: repo.jfrog.org
 Connection: Keep-Alive
 User-Agent: Gradle/2.4 (Linux;3.17.2-desktop-3.mga5;amd64) (Oracle
 Corporation;1.8.0_40;25.40-b25)


 write error to stdout

 054.086.126.152.00080-***.***.***.191.32885: HTTP/1.1 404 Not Found
 Content-Type: application/json;charset=ISO-8859-1
 Date: Fri, 22 May 2015 09:04:36 GMT
 Server: nginx
 X-Artifactory-Id: repo.jfrog.org
 transfer-encoding: chunked
 Connection: keep-alive

 58
 {
   "errors" : [ {
       "status" : 404,
           "message" : "Could not find resource"
      } ]
 }

 write error to stdout

 054.086.126.152.00080-***.***.***.32885: 0

 write error to stdout

***.***.***.191.32885-054.086.126.152.00080: HEAD
 /artifactory/libs-release-bintray/com/gradle/publish/plugin-publish-plugin/0.9.0/plugin-publish-plugin-0.9.0.jar
 HTTP/1.1
 Accept-Encoding: gzip,deflate
 Host: repo.jfrog.org
 Connection: Keep-Alive
 User-Agent: Gradle/2.4 (Linux;3.17.2-desktop-3.mga5;amd64)
 (Oracle Corporation;1.8.0_40;25.40-b25)


 write error to stdout

What is the write error to stdout text doing in the body of the request going out from Gradle? It looks like a bug to me. Especially given that it is a HEAD veerb, there should be no body AFAIK.

It could actually be that the text is causing an upstream network web traffic device to hiccup,

To add to the above, this issue can be reproduced by the below. In the first case, I am adding the write error to stdout, which causes a BAD REQUEST and immediate disconnect.

icidi:tmp schalkc$ telnet repo.jfrog.org 80
Trying 54.86.126.152...
Connected to jfrog-org-587243889.us-east-1.elb.amazonaws.com.
Escape character is '^]'.
HEAD /artifactory/libs-release-bintray/com/gradle/publish/plugin-publish-plugin/0.9.0/plugin-publish-plugin-0.9.0.jar HTTP/1.1
Accept-Encoding: gzip,deflate
Host: repo.jfrog.org
Connection: Keep-Alive
User-Agent: Gradle/2.4 (Linux;3.17.2-desktop-3.mga5;amd64) (Oracle Corporation;1.8.0_40;25.40-b25)

write error to stdout
HTTP/1.1 404 Not Found
Content-Type: application/json;charset=ISO-8859-1
Date: Fri, 22 May 2015 10:05:03 GMT
Server: nginx
X-Artifactory-Id: repo.jfrog.org
Connection: keep-alive

HTTP/1.1 400 BAD_REQUEST
Content-Length: 0
Connection: Close

Connection closed by foreign host.

When I just do the HEAD request, then I get the 404 as expected, plus my keep-alive is working.

icidi:tmp schalkc$ telnet repo.jfrog.org 80
Trying 54.86.126.152...
Connected to jfrog-org-587243889.us-east-1.elb.amazonaws.com.
Escape character is '^]'.
HEAD /artifactory/libs-release-bintray/com/gradle/publish/plugin-publish-plugin/0.9.0/plugin-publish-plugin-0.9.0.jar HTTP/1.1
Accept-Encoding: gzip,deflate
Host: repo.jfrog.org
Connection: Keep-Alive
User-Agent: Gradle/2.4 (Linux;3.17.2-desktop-3.mga5;amd64) (Oracle Corporation;1.8.0_40;25.40-b25)

HTTP/1.1 404 Not Found
Content-Type: application/json;charset=ISO-8859-1
Date: Fri, 22 May 2015 10:06:03 GMT
Server: nginx
X-Artifactory-Id: repo.jfrog.org
Connection: keep-alive

I would thus definitely consider this a bug in Gradle networking core itself - it is just case of certain networks amplifying the problem.

Thanks to the Bintray folks who pointed me at the following two issues on write error to stdout.

It seems that the message " is printed when the expected length and the actually written length are different ". Still does not explain the issue completely though. However, since Bintray folks have tweaked their setup, this issue seem to have disappeared.

Maybe not a bug (for now).