Gradle cache client socket timeout

Help/Discuss
1

Gradle test task consistently fails with Socket Timeout error when attempting to UPLOAD a new cache artefact for any task that takes more than 4 minutes (eg. a subset of integration tests in my case).

Enabling debug on the gradle client shows the Upload is attempted but times-out:

The CLIENT (gradle build) side output is as follows:

16:52:33.088 [DEBUG] [org.gradle.caching.internal.controller.service.OpFiringBuildCacheServiceHandle] Store entry 550e0accbfe8e3ca5cead3249cba5788 in remote build cache
16:52:33.088 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationExecutor] Build operation 'Store entry 550e0accbfe8e3ca5cead3249cba5788 in remote build cache' started
16:52:33.088 [DEBUG] [org.gradle.internal.resource.transport.http.HttpClientHelper] Performing HTTP PUT: http://gradle-cache.westeurope.cloudapp.azure.com:5071/cache/550e0accbfe8e3ca5cead3249cba5788
16:52:33.089 [DEBUG] [org.apache.http.client.protocol.RequestAddCookies] CookieSpec selected: default
16:52:33.089 [DEBUG] [org.apache.http.client.protocol.RequestAuthCache] Auth cache not set in the context
16:52:33.089 [DEBUG] [org.apache.http.impl.conn.PoolingHttpClientConnectionManager] Connection request: [route: {}->http://gradle-cache.westeurope.cloudapp.azure.com:5071][total kept alive: 1; route allocated: 1 of 20; total allocated: 1 of 20]
16:52:33.090 [DEBUG] [org.apache.http.impl.conn.PoolingHttpClientConnectionManager] Connection leased: [id: 0][route: {}->http://gradle-cache.westeurope.cloudapp.azure.com:5071][total kept alive: 0; route allocated: 1 of 20; total allocated: 1 of 20]
16:52:33.090 [DEBUG] [org.apache.http.impl.conn.DefaultManagedHttpClientConnection] http-outgoing-0: set socket timeout to 30000
16:52:33.090 [DEBUG] [org.apache.http.impl.conn.DefaultManagedHttpClientConnection] http-outgoing-0: set socket timeout to 30000
16:53:03.114 [DEBUG] [org.apache.http.impl.conn.DefaultManagedHttpClientConnection] http-outgoing-0: Close connection
16:53:03.114 [DEBUG] [org.apache.http.impl.conn.DefaultManagedHttpClientConnection] http-outgoing-0: Shutdown connection
16:53:03.115 [DEBUG] [org.apache.http.impl.execchain.MainClientExec] Connection discarded
16:53:03.115 [DEBUG] [org.apache.http.impl.conn.PoolingHttpClientConnectionManager] Connection released: [id: 0][route: {}->http://gradle-cache.westeurope.cloudapp.azure.com:5071][total kept alive: 0; route allocated: 0 of 20; total allocated: 0 of 20]
16:53:03.115 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationExecutor] Completing Build operation 'Store entry 550e0accbfe8e3ca5cead3249cba5788 in remote build cache'
16:53:03.115 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationExecutor] Build operation 'Store entry 550e0accbfe8e3ca5cead3249cba5788 in remote build cache' completed
16:53:03.116 [WARN] [org.gradle.caching.internal.controller.service.OpFiringBuildCacheServiceHandle] Could not store entry 550e0accbfe8e3ca5cead3249cba5788 for task ':node:integrationTest' in remote build cache
org.gradle.caching.BuildCacheException: Read timed out

Note above the HTTP PUT is issued, a connection is requested and obtained, a socket timeout of 30s is set, and then nothing until 30s later when the connection is shutdown.

Connectivity between client and server machines (Azure VM’s running Ubuntu Linux 16.04.10) is confirmed to be working, and it is possible to manually upload the cached artifact via Apache AB after the above reported failure:

ab -u /home/colljos/.gradle/caches/build-cache-1/550e0accbfe8e3ca5cead3249cba5788 -T 'application/octet-stream' http://gradle-cache.westeurope.cloudapp.azure.com:5071/cache/
This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking gradle-cache.westeurope.cloudapp.azure.com (be patient).....done

Server Software:        
Server Hostname:        gradle-cache.westeurope.cloudapp.azure.com
Server Port:            5071

Document Path:          /cache/
Document Length:        0 bytes

Concurrency Level:      1
Time taken for tests:   0.010 seconds
Complete requests:      1
Failed requests:        0
Non-2xx responses:      1
Total transferred:      136 bytes
Total body sent:        403305
HTML transferred:       0 bytes
Requests per second:    95.47 [#/sec] (mean)
Time per request:       10.474 [ms] (mean)
Time per request:       10.474 [ms] (mean, across all concurrent requests)
Transfer rate:          12.68 [Kbytes/sec] received
                        37602.88 kb/s sent
                        37615.56 kb/s total

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        3    3   0.0      3       3
Processing:     8    8   0.0      8       8
Waiting:        0    0   0.0      0       0
Total:         10   10   0.0     10      10

Please advise.

2

Turns out that on Microsoft Azure Cloud VM’s outbound socket connections have a 4-minute idle timeout.
“This timeout is not adjustable. However, you can use transport (for example, TCP keepalives) or application-layer keepalives to refresh an idle flow and reset this idle timeout if necessary.”

Is there any configuration mechanism to enable keepalives within the Gradle Cache Client ?

I’ve had a look at the available JavaHttpTimeoutSettings properties that can be passed into the HttpClientHelper (used to initiate the PUT) but there doesn’t appear to be anything available for keepalive.
https://docs.oracle.com/javase/8/docs/api/java/net/SocketOptions.html#SO_KEEPALIVE

3

I have raised https://github.com/gradle/gradle/issues/6461 to just turn keep-alive on for all HTTP connections.