Hello everyone,
we are using Tekton pipelines on OpenShift Container Platform.
The Tekton pipelines build Kotlin/Java services using Gradle (7.5.1).
Using a central volume, all pipelines share a central Gradle-Home directory.
Within the pipelines, individual Gradle tasks are executed as follows (here using the example of “assemble”):
When we work on the same service in parallel as a team, we often end up with parallel pipeline executions for the same service and thus also parallel executions of Gradle tasks.
This then often results in failed Gradle builds. Here is an error for a failed “assemble” task:
FAILURE: Build failed with an exception.
* What went wrong:
Gradle could not start your build.
> Cannot create service of type BuildSessionActionExecutor using method LauncherServices$ToolingBuildSessionScopeServices.createActionExecutor() as there is a problem with parameter #21 of type FileSystemWatchingInformation.
> Cannot create service of type BuildLifecycleAwareVirtualFileSystem using method VirtualFileSystemServices$GradleUserHomeServices.createVirtualFileSystem() as there is a problem with parameter #7 of type GlobalCacheLocations.
> Cannot create service of type GlobalCacheLocations using method GradleUserHomeScopeServices.createGlobalCacheLocations() as there is a problem with parameter #1 of type List<GlobalCache>.
> Could not create service of type FileAccessTimeJournal using GradleUserHomeScopeServices.createFileAccessTimeJournal().
> Timeout waiting to lock journal cache (/workspace/output/.gradle/caches/journal-1). It is currently in use by another Gradle instance.
Owner PID: 80
Our PID: 80
Owner Operation:
Our operation:
Lock file: /workspace/output/.gradle/caches/journal-1/journal-1.lock
How can we avoid such faulty Gradle builds in our described setup?
I wonder that the journal is locked that long that you run into a timeout.
And I don’t know if it is related to the filesystem watching you enable.
Actually the filesystem watching is enabled automatically anyway for systems where it is supported.
But either way using it is pretty useless if you use --no-daemon and probably ephemeral build agents, because the filesystem watching watches the filesystem between builds on a daemon.
So without a surviving daemon filesystem watching is pretty pointless.
I’d suggest you try if removing the explicit filesystem watching helps.
Hello Björn,
Thanks for your reply and your suggestion about filesystem watching.
I added --no-watch-fs to our gradlew command and started a few parallel pipeline runs. Unfortunatly the problem remains.
I started 6 pipeline runs, one after the other, with approx. 5 seconds delay in between. Here are the results of the pipeline runs: pipeline-commons-9dg2v cd /workspace/output/pipeline-commons-9dg2v ./gradlew --gradle-user-home /workspace/output/.gradle --no-daemon --build-cache --no-watch-fs --parallel '-Dorg.gradle.jvmargs=-Xmx1g -XX:MaxMetaspaceSize=512m' -PprojectVersion=1.5.1.1673592867-SNAPSHOT spotlessCheck FAILURE: Build failed with an exception. Timeout waiting to lock journal cache (/workspace/output/.gradle/caches/journal-1). It is currently in use by another Gradle instance. pipeline-commons-5r66k cd /workspace/output/pipeline-commons-5r66k ./gradlew --gradle-user-home /workspace/output/.gradle --no-daemon --build-cache --no-watch-fs --parallel '-Dorg.gradle.jvmargs=-Xmx1g -XX:MaxMetaspaceSize=512m' -PprojectVersion=1.5.1.1673592903-SNAPSHOT spotlessCheck FAILURE: Build failed with an exception. Timeout waiting to lock journal cache (/workspace/output/.gradle/caches/journal-1). It is currently in use by another Gradle instance. pipeline-commons-tn9ws cd /workspace/output/pipeline-commons-tn9ws ./gradlew --gradle-user-home /workspace/output/.gradle --no-daemon --build-cache --no-watch-fs --parallel '-Dorg.gradle.jvmargs=-Xmx1g -XX:MaxMetaspaceSize=512m' -PprojectVersion=1.5.1.1673592902-SNAPSHOT spotlessCheck FAILURE: Build failed with an exception. Timeout waiting to lock journal cache (/workspace/output/.gradle/caches/journal-1). It is currently in use by another Gradle instance. pipeline-app-7s2mj cd /workspace/output/pipeline-app-7s2mj ./gradlew --gradle-user-home /workspace/output/.gradle --no-daemon --build-cache --no-watch-fs --parallel '-Dorg.gradle.jvmargs=-Xmx1g -XX:MaxMetaspaceSize=512m' -PprojectVersion=1.0.13.1673592918-SNAPSHOT spotlessCheck BUILD SUCCESSFUL in 24s 5 actionable tasks: 1 executed, 2 from cache, 2 up-to-date pipeline-app-nm97f cd /workspace/output/pipeline-app-nm97f ./gradlew --gradle-user-home /workspace/output/.gradle --no-daemon --build-cache --no-watch-fs --parallel '-Dorg.gradle.jvmargs=-Xmx1g -XX:MaxMetaspaceSize=512m' -PprojectVersion=1.0.13.1673592918-SNAPSHOT spotlessCheck FAILURE: Build failed with an exception. Timeout waiting to lock journal cache (/workspace/output/.gradle/caches/journal-1). It is currently in use by another Gradle instance. pipeline-app-c4psp cd /workspace/output/pipeline-app-c4psp ./gradlew --gradle-user-home /workspace/output/.gradle --no-daemon --build-cache --no-watch-fs --parallel '-Dorg.gradle.jvmargs=-Xmx1g -XX:MaxMetaspaceSize=512m' -PprojectVersion=1.0.13.1673592911-SNAPSHOT spotlessCheck FAILURE: Build failed with an exception. Timeout waiting to lock journal cache (/workspace/output/.gradle/caches/journal-1). It is currently in use by another Gradle instance.
Every pipeline runs in its own container, but they share a persistent volume (/workspace/output/.gradle) as their Gradle-Home.
Another approach I’m currently investigating is to avoid the central Gradle-Home directory. In this case, every pipeline has its own local Gradle-Home directory, that will not be shared with other processes. In this case I start the gradlew command using --no-build-cache and --no-watch-fs, because it seems to me, that build cache and filesystem watching do not apply for this scenario. Running several parallel pipelines with these configurations works just fine. However, the pipelines take a lot longer, because Gradle cannot speed up builds using a central build and dep cache.
I came across GRADLE_RO_DEP_CACHE here: Understanding dependency resolution. I tried that approach in combination with pipeline/container-local Gradle-Home directories. This reduces the durations of the pipeline runs a little bit (5-10%), but all together the pipeline runs still take a lot longer compared to using a central Gradle-Home directory as a central build and dep cache.
In this case I start the gradlew command using --no-build-cache and --no-watch-fs , because it seems to me, that build cache and filesystem watching do not apply for this scenario
Why do you think so? (regarding build cache)
If the Gradle user home directories are per pipeline but persistent, you still benefit from build cache unless you change each cacheable tasks input until the next run on the same pipeline.
You could also consider adding a remote build cache so that all the pipelines can share cache entries in a proper way.
However, the pipelines take a lot longer, because Gradle cannot speed up builds using a central build and dep cache.
Naturally. But as described above, you could consider adding a remote build cache to continue having a shared build cache.
The dependencies indeed would not be shared, but you could actually share a bootstrapped read-only one as documented. Have a look at Understanding dependency resolution to see how to do it.
I came across GRADLE_RO_DEP_CACHE here
Ah, yeah, you found it already
This reduces the durations of the pipeline runs a little bit (5-10%), but all together the pipeline runs still take a lot longer compared to using a central Gradle-Home directory as a central build and dep cache.
Reenable the build cache, and also give the remote cache a try. I’d guess you get good enough results then.
Yes, the Gradle user home directories are per pipeline, but the directory has a unique name in each pipeline run and the directory will be deleted at the end of the pipeline run. Each project/repo can have several pipeline runs in parallel and that is why every pipeline run has a unique name, that is used for the Gradle user home directory.
Good idea, thank you. I will give it a try after I configured the Gradle read-only dependency cache.
Ah, yeah, if each pipeline run has a unique non-persisted Gradle User Home, the local build cache does not help of course.
That’s then exactly what the “ephemeral build” section is talking about.
So yeah, the read-only dependency cache in combination with the remote build cache will hopefully help you get up to speed again.
