Concurrency issues with composite builds

We are currently experimenting with composite builds and we seem to stumble into some concurrency issues (using gradle 6.3), for regular java tasks when the same project is included in more than one included build.

The docs Composing builds says:

“Multiple composite builds may conflict when run in parallel, if more than one includes the same build. Gradle does not share the project lock of a shared composite build to between Gradle invocation to prevent concurrent execution.”

But we get random build failures for single composite builds which might have for example two included builds each including some overlapping sub projects. Where for example a compileJava task for a multiply included project sometimes fails due to missing dependencies (it looks like the tasks fail due to interleaved execution)

What can be the cause of this?

The cause is - if I got you right - that you should never ever ever include one project in more than one build.
One project should belong to exactly one build and if that project is needed in another build, include the whole build to get that project.

Including one project in more than one build can definitely cause such behavior and even if the builds including that project are separate builds not combined in a composite build, it is highly discouraged and can lead to more or less obvious strange things.

OK, thank you Björn - I appreciate the feedback. So we might very well do something we should not be doing…

Here’s a very brief, high level description of our setup:

  • We have a few thousand reusable software modules each mapped 1:1 into a unique gradle project.
  • We have a few hundred gradle projects which combines several software modules above into a build which most often result in a software package to be installed later on. Most software modules are included many times in multiple of these “packaging” projects.

But we actually often script the package builds using shell scripts to be able to easily massage the build structure into something that can easily be packaged. We are now experimenting with replacing the shell scripts with gradle exclusively, mainly to simplify but we also hope for build speed improvements. But a few of these packaging projects seem to require composite builds to be able to end up in a successful package (since they rely on several settings files).

Is it discouraged to do like we do and have each software module as a separate gradle project which can be included in several “higher level” projects? What is the alternative? Should we treat everything as a big composite build where each software module is a separate build itself (using includeBuild instead of include)?

From what I read that is what I’d say is appropriate.
Each of these standalone modules should be an own build that you then can compose into bigger entities.
You can even compose composite builds which you will then do in your setup.

Great, we will work on adapting our build to this then. Again, thanks.