We recently rolled out the latest release of application and are now maintaining three gradle releases for each of about 300 components and therefore running about 1000 jobs in our Jenkins setup. Since we added the Jenkins jobs for the new release, we are seeing many of the following errors and jobs are failing. This at a critical time in our delivery process.
We do not use dynamic versions (or transitive dependencies) , so all cached jar files are individually versioned. Consequently, I am not understanding these locks that suddenly began causing issues as the referenced jar contents should be the same for all jobs (regardless of SCM root, branch, release, etc.)
How do we combat these issues?
Timeout waiting to lock jars (/tmpfs/jenkins/.gradle/caches/jars-2). It is currently in use by another Gradle instance.
Owner PID: 11388
Our PID: 29369
Owner Operation:
Our operation:
Lock file: /tmpfs/jenkins/.gradle/caches/jars-2/jars-2.lock
How many builds do you have running on each of these Jenkins agents? Also, are you sharing this directory amongst many agents? We’ve seen issues like this in the past where many Jenkins slaves were sharing a single Gradle user home directory.
Eight concurrent builds might be hitting a threshold of contention which is triggering this wait timeouts. You might try splitting these up, perhaps among 2 different Gradle user home directories. You can do this one of two ways:
Use separate system accounts for the executors, by default this directory is ~/.gradle (i.e. jenkins1 and jenkins2)
Use the -g command line argument to explicitly specify a home directory. Although as far as I can tell you’d have to do this per job, and there would be no good way to ensure a maximum number per directory, so this is probably not ideal.
This appears to have been caused (3.2 has been running in our Jenkins environment for months) from adding build scan support last week. I had added the following to our build script
plugins {
id “com.gradle.build-scan” version “1.3”
}
which caused the dependency to be resolved from the external plugins site for about 300 of our Jenkins jobs. My QA lead pointed it out when running on a server without a proxy defined (from behind our firewall). Our policy is to have no external network access during the build. I placed the jar in a local directory as for other thirdyparty dependencies and the issue has resolved.
Does your build otherwise not resolve remote dependencies, not even from a local Nexus or Artifactory? If you use a local artifact repository you can place the build scan plugin there if you want and configure your builds to fetch plugins from that repository.
I placed the jar in a local directory as for other thirdyparty dependencies
If I’m interpreting this right you are saying all your dependencies are checked-in as JARs in VCS. If so then yes, putting the build-scan plugin in there as well will be fine.