Composite build hangs when there is legal cyclic dependency between sub-builds

Help/Discuss
1

Composite build hangs when there is cyclic dependency between sub-builds. Here’re our projects:

  1. “ligradle-core” is the LinkedIn Gradle Plugins API. It is a java project so it needs “ligradle-jvm” on buildscript classpath.
  2. “ligradle-jvm” is an implementation of LinkedIn Gradle Plugins API and contains plugins for Jvm platform (Java, War, Scala, Groovy, etc.). “ligradle-jvm” itself is a java project so it needs “ligradle-jvm” on buildscript classpath. Technically, “ligradle-jvm” depends on itself, at a previously released version.

Here’s the diagram:

In this scenario, the composite build hangs. More details (with Gradle 3.2):

  1. Debug output: https://gist.github.com/szczepiq/e06adb719a6883914d90bbbeeef443c5
  2. GradleDaemon process jstack: https://gist.github.com/szczepiq/ceccb8206008373d16e86d1c1941143f

This use case is a daily bread in our org - it would be great if composite build supported it!

2

For scenarios like this we allow a composite build to use declared substitutions , so that only particular modules
or versions will be substituted within the composite.

For a plugin that has a buildscript dependency on an older version of itself, you would include the build such that the older version isn’t substituted, but the newer (under-development) version is.

I’ve put together a GH repository that replicates the structure you have described above, including a composite build that works to combine the separate plugin builds. https://github.com/bigdaz/ligradle-composite. Take particular note of the includeBuild statement that uses declared substitution.

Let’s use this functional repository as a starting point to explore what’s not working in your specific case.

3

Thank you Daz for setting up comprehensive the GitHub repo for this use case!!! I plan to test it out with LinkedIn Multiproducts today.

When I complete my tests I’ll get back with feedback. I feel that Gradle should behave more gracefully for this scenario (currently, it is hanging build).

4

Hey Daz! I encountered an issue trying to tweak your sample projects so that they are more similar to LinkedIn projects. I’ve submitted a ticket to your repo with the details: https://github.com/bigdaz/ligradle-composite/issues/1

5

Thanks: I’ve updated the demo to more closely match your scenario, and included some feedback on the issue.

This demo assumes that you will be using a test build that consumes the ligradle-jvm plugin in order to develop and test the plugin. Is this the way you usually develop these plugins? Or do you rely on integration tests (using Gradle TestKit) to develop and test?

I’m trying to think of a more convenient way to allow the substitutions to be declared, so that the old versions are resolved from the repository and new versions are substituted with project dependencies. Right now, the demo hard-codes the version to substitute. Excluding buildscript dependencies from substitution won’t work in this example, since that’s how the composite build consumes the ligradle-jvm plugin project. An alternative might be to hard-code the old version, applying substitution to any other version.

I’m curious to understand what would work best in your case.