Is task execution unordered?

Help/Discuss
1

I’m having an issue with a plugin I’m developing – sometimes tasks are executed in an order that causes the build to succeed, and sometimes it’s executed in a seemingly different order that causes the build to fail. All I have to do is run ./gradlew clean build over and over, and sometimes it works, sometimes it doesn’t.

gradle.properties has org.gradle.parallel=true. Could this be related?

How can I troubleshoot this? Are there tips on how to ensure that a task runs after different lifecycle tasks have run on its dependencies?

2

Within each project, the tasks have a deterministic ordering (based on task dependencies) and will always execute in the same order.

When org.gradle.parallel=true this allows tasks in different projects to execute in parallel provided they don’t depend on one another.

So it’s possible that you have tasks in separate projects that logically depend on one another but that dependency hasn’t been configured in your gradle build. If you want to debug the task ordering you could put printlns in a TaskExecutionListener or perhaps a build scan.

Another cause of these sorts of problems is tests where one test “bleeds” state (eg mutable static) into another so TestA will succeed when it runs before TestB but will fail if it runs after. To debug test ordering there’s beforeTest{} and afterTest{} handlers on the Test task

3

Good to know. In this case, it’s not test cases that are causing the issue. I need to perform validation and codegen from models that depend on other models from within the same project. The way models are discovered is through the classpath and SPI.

So let’s say I have a “producer” model and a “consumer” model. The consumer model uses the model created by the producer model’s JAR. In order to build the consumer model’s JAR, I need the producer model’s JAR. It seems like sometimes producer’s JAR is available before consumer runs, and sometimes it isn’t.

Here’s the first attempt to wire this up, but it doesn’t work reliably.

Then I tried this to clean it up, but still it doesn’t work reliably.

4

registerSmithyBuildTask has a bit of a code smell to me. Task dependencies are usually set in stone and don’t change based on other tasks being disabled

Consider the following

task copy1(type: Copy) {
   from 'dir1' 
   into 'dir2' 
} 
task copy2(type: Copy) {
   from tasks.copy1
   into 'dir3' 
}

Gradle will automatically make copy2 depend on copy1 in the example above. Under the hood from tasks.copy1 does two things.

  1. Creates a FileCollection which delegates to copy1’s task outputs.
  2. Uses the FileCollection as TaskInput which causes Gradle to also create a task dependency

I prefer to use this type of configuration where possible, it’s more readable and simpler than manually configuring task dependencies

5

Totally agree. I was tried to clean that up in the second link: https://github.com/awslabs/smithy-gradle-plugin/blob/fix-build-graph-2/src/main/java/software/amazon/smithy/gradle/SmithyPlugin.java#L46-L68. I still feel like I have no idea how to make this do what I want though.

6

Can you share the relevant producer / consumer task declaration snippets?

7

Sure. Here’s the plugin code I’m using (publishing to mavenLocal, and updated based on your feedback):

Here’s a minimal integration test where I’ve tried to reproduce the failure. However, this always works:

But when used in this much larger project, the build will often fail due to dependencies not being available when smithy-aws-protocol-tests is built:

Is there something going on that’s funky with my main build.gradle.kts for the larger project?

8

Unfortunately none of these snippets show the configuration of something being produced by one task (a file of some form) and then being consumed by another task

9

I ended up making these changes, and it seems to fix my build ordering issue: https://github.com/awslabs/smithy-gradle-plugin/pull/20

I ran into a lot of other problems that are unrelated to this original post (like how to dynamically add a project dependency, and that Gradle’s interception of stdout/stderr doesn’t seem to be thread-safe), but those can be raised elsewhere.

10

Here’s two ways that I know of to add project dependencies.

void addProjectDeps(Project project, Configuration config) {
   Project p1 = project.project(':project1');
   Project p2 = project.project(':project2');
   DependencyHandler dh = project.getDependencies();
   dh.add(config.getName(), p1); // only works when project.configurations.contains(config) 
   Dependency d2 = dh.create(p2);
   config.getDependencies().add(p2);
}