So, after some digging we found out what we think is happening.
We have a gradle build that does the following
a) compile java (reaches out to our artifactory and download dependencies)
b) runs owasp dependency check (takes around 10 ~ 20 minutes)
c) compile test (reaches out to artifactory again)
in step c)… we see the dreaded
2020-06-30T18:50:51.441+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] > Could not HEAD 'https://artylab/public-maven-virtual/org/junit/jupiter/junit-jupiter-engine/5.6.2/junit-jupiter-engine-5.6.2.jar'.
2020-06-30T18:50:51.441+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] > Connection reset
After digging more into the debug mode we saw
2020-06-30T18:50:51.107+0000 [DEBUG] [org.gradle.internal.resource.transport.http.HttpClientHelper] Performing HTTP HEAD: https://artylab.expedia.biz/public-maven-virtual/org/junit/jupiter/junit-jupiter-engine/5.6.2/junit-jupiter-engine-5.6.2.jar
2020-06-30T18:50:51.107+0000 [DEBUG] [org.apache.http.client.protocol.RequestAddCookies] CookieSpec selected: default
2020-06-30T18:50:51.107+0000 [DEBUG] [org.apache.http.impl.conn.PoolingHttpClientConnectionManager] Connection request: [route: {s}->https://artylab.expedia.biz:443][total kept alive: 4; route allocated: 4 of 20; total allocated: 4 of 20]
2020-06-30T18:50:51.108+0000 [DEBUG] [org.apache.http.impl.conn.PoolingHttpClientConnectionManager] Connection leased: [id: 35][route: {s}->https://artylab.expedia.biz:443][total kept alive: 3; route allocated: 4 of 20; total allocated: 4 of 20]
2020-06-30T18:50:51.108+0000 [DEBUG] [org.apache.http.impl.conn.DefaultManagedHttpClientConnection] http-outgoing-35: set socket timeout to 30000
2020-06-30T18:50:51.108+0000 [DEBUG] [org.apache.http.impl.conn.DefaultManagedHttpClientConnection] http-outgoing-35: set socket timeout to 30000
2020-06-30T18:50:51.108+0000 [DEBUG] [org.apache.http.impl.execchain.MainClientExec] Executing request HEAD /public-maven-virtual/org/junit/jupiter/junit-jupiter-engine/5.6.2/junit-jupiter-engine-5.6.2.jar HTTP/1.1
2020-06-30T18:50:51.109+0000 [DEBUG] [org.apache.http.impl.execchain.MainClientExec] Target auth state: SUCCESS
2020-06-30T18:50:51.109+0000 [DEBUG] [org.apache.http.impl.execchain.MainClientExec] Proxy auth state: UNCHALLENGED
2020-06-30T18:50:51.110+0000 [DEBUG] [org.apache.http.impl.conn.DefaultManagedHttpClientConnection] http-outgoing-35: Close connection
2020-06-30T18:50:51.110+0000 [DEBUG] [org.apache.http.impl.conn.DefaultManagedHttpClientConnection] http-outgoing-35: Shutdown connection
2020-06-30T18:50:51.110+0000 [DEBUG] [org.apache.http.impl.execchain.MainClientExec] Connection discarded
2020-06-30T18:50:51.110+0000 [DEBUG] [org.apache.http.impl.conn.PoolingHttpClientConnectionManager] Connection released: [id: 35][route: {s}->https://artylab.expedia.biz:443][total kept alive: 3; route allocated: 3 of 20; total allocated: 3 of 20]
and before that we saw
20-06-30T18:26:36.059+0000 [DEBUG] [org.apache.http.impl.conn.PoolingHttpClientConnectionManager] Connection released: [id: 35][route: {s}->https://artylab.expedia.biz:443][total kept alive: 4; route allocated: 4 of 20; total allocated: 4 of 20]
which means that is been more than 20 minutes after the connection was returned to the pool…
Also connection 35 says
2020-06-30T18:26:29.555+0000 [DEBUG] [org.apache.http.impl.conn.PoolingHttpClientConnectionManager] Connection [id: 35][route: {s}->https://artylab.expedia.biz:443] can be kept alive indefinitely
so we presume keepalives are being sent… and we know that we might have a problem with our Load balancer not forwarding the keepalives
so my questions are
a) is there a way to specify to gradle to evict an idle connection after a little while? (say idle for 5 minutes)
b) we presume the connection pool is sending keepalives (https://github.com/gradle/gradle/issues/6461) but maybe these are created differently and not doing so?
c) even with the connection reset, I would have expected that is a transient (socketexception) exception and would’ve been taken care of by (https://github.com/gradle/gradle/issues/8264) but it didn’t. Is this to be expected?
Any help would be appreciated. Thank you!