parallel gradle builds running on jenkins causing issue with gradle global cache

Help/Discuss
1

We run our CI on Jenkins. The below image is a decent explanation of the whole setup (non-relevant steps are removed for the sake of simplicity).

831×841 73.2 KB

To summarize, our overall process:

  1. We have a master and a beefier agent machine (both VMs running on the cloud) running all our CI.
  2. Currently, we have 26 git repos that are managed by this system.
  3. All our CI jobs are multi-branch Jenkins project.
  4. All repos are Gradle projects.

Jenkins details: 1. All CI jobs run on Jenkins agent. 2. The Jenkins agent is configured with 4 executors.

The problem:

Multiple Gradle builds running in parallel step on each other. All Gradle builds use the same grade global cache ( /var/lib/jenkins/.gradle/caches ) and while one executor is trying to compile, the other executor runs clean which does :

rm -rf /var/lib/jenkins/.gradle/caches

This results in problems like:

  1. Compilation failed
  2. Class not found
  3. Could not resolve all artifacts for configuration ‘:classpath’.

etc.

To bypass this, I have disabled build caching. This has definitely stabilized the builds, but now it is taking around 40% longer to complete a CI job. So, I have to enable back caching as soon as possible.

Question:

What is the optimal way out of it? I’m considering configuring separate global cache per project (in the root of the project) but have not tested it yet. Do you have any other suggestions?

Any pointer is highly appreciated. Thanks in advance.

2

Hi,

This is deleting the local build cache but also all kinds of other Gradle caches which are used during the build:

Why do you run this as part of clean? It should not be necessary and will break you build for sure. What happens if you don’t remove /var/lib/jenkins/.gradle/caches.

Cheers,
Stefan

3

I have commented out this very line and have seen situation improve. Basically, this line

rm -rf /var/lib/jenkins/.gradle/caches

existed for historical reasons, which I’m unaware of and nobody in the company knows. A guess would be as follows:

stages {
stage(‘clean’) {
steps {
script {
try {
echo ‘Current Branch…’ + env.BRANCH_NAME
//sh ‘rm -rf /var/lib/jenkins/.gradle/caches’
sh ‘./gradlew clean --stacktrace’
} catch (e) {
currentBuild.result = ‘FAILURE’
slackHelper.buildGenericJobFailureNotificationMessage()
throw e
}
}
}
}

AFAIK,

gradlew clean

only clears the build directory. Maybe the original author felt the strong need for clearing the global cache because some of the jars (libraries) in use are updated internally quite frequently and not clearing the global cache would result in not having right dependency version in the classpath. For various reasons, including but not limited to :
1/ poor versioning strategy
2/ unclear release model
we add these dependencies in build.gradle as below:

> dependencies {
>   compile "com.xyz:model:+"
>   compile "com.xyz:apiConnector:+"
>   compile "com.xyz:service:+"
> }

I also have disabled use global cache for now.

The plan now is to :slight_smile:
a. Enable global cache.
b. (if a fails) configure project specifc cache.

@Stefan_Wolf my questions for you:

  • Do you see any issue with #b?
  • When and how does gradle decides to refresh global cache? [I could not conclusively decide on this looking at the documentation. If I missed something, please point me to the right link]
4

I don’t know what you mean by this. What is the global “cache”? Gradle has many caches (e.g. dependency cache, file hashes cache, metadata cache, build cache, etc.).

What cache are you talking about here? Do you mean you want to use a project specific Gradle user home per build? Or do you want to configure the local directory build cache to use a directory in the current project for the build cache.

If you are talking about the build cache, using a local cache per project should be fine, as long as the cache is shared by multiple builds on the same project. If you run into problems with concurrent access, use a remote build cache.

I suppose this is about the dependency cache? There is the --refresh-dependencies command line switch. You also may be interested in how to handle changing and dynamic versions.

Cheers,
Stefan

5

I also bumped into this when we wanted to do efficient Gradle builds with ArgoCD.

It does not seem possible to share the Dependency Cache between parallel Gradle builds running on a CI server in a way so that potentially both can write into it. (read-only mode is now supported but that is worthless for CI servers because it needs a manual populating effort)

Also, the Gradle plugin of Jenkins doesn’t support building Gradle Dependency Caches per Executor.
(another area in which the tooling support for Gradle is much weaker compared to Maven).

This is a mess.

Gradle should support sharing the Dependency Cache between parallel Gradle builds in a multi-write way.

This is the only way efficient CI builds could work without very specific Gradle-tooling support (which is not present in most of the CI solutions)

I do not really understand why this is not supported.

It could be activated with a specific flag so it doesn’t slow down single, non-parallel builds.

Even when active, it would only need to incur a performance penalty when downloading a new dependency and placing it into the Dependency Cache. (which should be rare)