How to order a Composite Build with dependsOn?

Help/Discuss
1

I have multiple separate Git repos to build individual Docker images, ex:

I’ve created a Composite Project, to make it easy to automate the parallelized building of all the images:

I have a collection of simple language Docker images, ex: java, node.js, python, ruby, etc. and then a build for child images, ex: Java -> Elasticsearch. How do I create a task in a Composite Project for an image like Elasticsearch to dependsOn the build of the Java image?

gradle.build

task("buildOpenjdk") {
    dependsOn gradle.includedBuild("packer-openjdk").task("build:rhel7.6")
    dependsOn gradle.includedBuild("packer-openjdk").task("build:centos7.6")
    dependsOn gradle.includedBuild("packer-openjdk").task("build:ubuntu18.04")
    dependsOn gradle.includedBuild("packer-openjdk").task("build:ubuntu16.04")
}
task("buildElasticsearch") {
    dependsOn "buildOpenjdk"
    mustRunAfter "buildOpenjdk"
    dependsOn gradle.includedBuild("packer-elasticsearch").task("build:rhel7.6")
    dependsOn gradle.includedBuild("packer-elasticsearch").task("build:centos7.6")
    dependsOn gradle.includedBuild("packer-elasticsearch").task("build:ubuntu18.04")
    dependsOn gradle.includedBuild("packer-elasticsearch").task("build:ubuntu16.04")
}
task("build") {
    dependsOn "buildElasticsearch"
}

taskTree

gradle build taskTree

:build
+--- :buildElasticsearch
|    +--- :buildOpenjdk
|    |    +--- :gradle-build:centos7.6:build
|    |    +--- :gradle-build:rhel7.6:build
|    |    +--- :gradle-build:ubuntu16.04:build
|    |    \--- :gradle-build:ubuntu18.04:build
|    +--- :gradle-build:centos7.6:build
|    +--- :gradle-build:rhel7.6:build
|    +--- :gradle-build:ubuntu16.04:build
|    \--- :gradle-build:ubuntu18.04:build

command

gradle build --parallel --rerun-tasks

When I run that specific gradle command, the 8+ parallel task runners will start trying to build the Elasticsearch image, before the Openjdk image has been built. How do I fix that?

2

It seems like a little bug to me… possibly best for the Gradle team to comment here

I’m guessing you could workaround by creating task wrappers/task dependencies in the current build

Eg:

def osList = ["rhel7.6", "centos7.6", "ubuntu18.04", "ubuntu16.04"]
task("buildOpenjdk") {} 
task("buildElasticsearch") {} 
osList.each { os ->
   tasks.create("buildOpenjdk-$os") {
      dependsOn gradle.includedBuild("packer-openjdk").task("build:$os")
   } 
   tasks.create("buildElasticsearch-$os") {
      dependsOn "buildOpenjdk-$os" 
      dependsOn gradle.includedBuild("packer-elasticsearch").task("build:$os")
   } 
   buildOpenjdk.dependsOn "buildOpenjdk-$os"
   buildElasticsearch.dependsOn "buildElasticsearch-$os"
}
3

Yup, I’ve been breaking it out into 2 separate tasks: buildBaseImages and then buildChildImages

4

I’d probably create a separate submodule in a multi module build for each item in osList so gradle can build each os in parallel

5

Yup, already did that:

6

I’m not sure we’re on the same page… I meant something like

settings.gradle

ext {
   osList = ["rhel7.6", "centos7.6", "ubuntu18.04", "ubuntu16.04"]
} 
osList.each { os -> include ":$os" }

build.gradle

gradle.settings.osList.each { os ->
   project(":$os") {
      apply plugin: 'base' 
      tasks.create("buildOpenjdk") {
         dependsOn gradle.includedBuild("packer-openjdk").task("build:$os")
      }
      tasks.create("buildElasticsearch") {
         dependsOn "buildOpenjdk" 
         dependsOn gradle.includedBuild("packer-elasticsearch").task("build:$os")
      } 
      build.dependsOn buildElasticsearch
   } 
}
7

Yeah, I’m already using a closure to iter the osList to tasks:

That doesn’t fix it. I get the impression that the Composite Build doesn’t take into account the ordering of the dependsOn, and also doesn’t correctly implement mustRunAfter, but wanted to check.

I want to keep the Composite Build structure, since it lets me segment out each of the image builds into separate repos, but I can’t figure out how to include the build tasks for “child Image builds” into a dedicated task, since the Composite Build uses dependsOn to include the sub-project tasks.

8

I get the impression that the Composite Build doesn’t take into account the ordering of the dependsOn

Yes, I understand this might be a bug. Which is why my solution creates a task in the “master” build for each composite task. My theory being that Gradle will adhere to the dependsOn rules in the “local” task graph

I want to keep the Composite Build structure

Of course, and my solution does not change that

but I can’t figure out how to include the build tasks for “child Image builds” into a dedicated task

My theory is to create extra tasks in the “master” build which wrap tasks in the composite. And only ever depend on tasks in the “master” build, never on tasks in composite builds as a workaround for the (potential) bug

9

I don’t think it’s so much a “bug” as a “failure to meet expectations” :thinking:

Directed Acyclic Graphs get me every time. Idempotency is surprisingly rare. I’ll give your suggestion a try, and see if that helps Gradle interpret the “local” DAG better than importing the includeBuild DAG.

10

Gradle task dependencies within a non-composite build should be bullet proof. If you specify a “dependsOn” relationship then the tasks will always run in the correct order.

Gradle will always run tasks within a single module in series (never in parallel). In a non-composite build, the only way to benefit from parallel execution is to use multiple modules.

I’ve read somewhere on these forums that Gradle always assumes that composite builds can be run in parallel. I get the feeling you’ve got a use case where this assumption falls apart

My first code snippet (above) put all the composite task wrappers in a single module which would mean that all execution would be in series (never in parallel). My second code snippet (above) put each operating system’s tasks in separate modules to benefit from Gradle’s parallel task execution