mustRunAfter not working in Composite Build

Help/Discuss
1

from `mustRunAfter` not working in Composite Build · Issue #32758 · gradle/gradle · GitHub

Current Behavior

I’m having a composite build with a mustRunAfter declared task using:

tasks.register("checkDist") {
    dependsOn(gradle.includedBuild("build-tests").task(":test"))
    mustRunAfter("assemble")
}

tasks.check {
    dependsOn("checkDist")
}

but when running ./gradlew clean assemble check tests from the included build are executed before the assemble task is finished.

When I try with ---dry-run all looks ok:

composite-build on  main via 🅶 v8.10 via ☕ v21.0.5 via 🅺 
❯ ./gradlew assemble check --dry-run                                              
:dist SKIPPED
:distTar SKIPPED
:distZip SKIPPED
:assemble SKIPPED
:checkDist SKIPPED
:check SKIPPED

> Task :build-tests:test

DistTest > testDistFile() PASSED

BUILD SUCCESSFUL in 898ms
9 actionable tasks: 1 executed, 8 up-to-date

but when executing the checkDist task is run before the assemble task.

composite-build on  main via 🅶 v8.10 via ☕ v21.0.5 via 🅺 
❯ ./gradlew assemble check          

> Task :build-tests:test

DistTest > testDistFile() PASSED

> Task :dist
Writing dist file: /Users/workspace/composite-build/build/data/dist.txt
Writing dist file finished

BUILD SUCCESSFUL in 5s
10 actionable tasks: 2 executed, 8 up-to-date

Expected Behavior

Expected behavior is that the checkDist task is executed after the assemble task is finished.

Self-contained Reproducer Project

https://github.com/marceloverdijk/composite-build-must-run-after

Gradle version

8.10

2

So from feedback:

There is a data dependency between your test task that consumes a file and dist task that produces a file. However, your build logic doesn’t express this dependency, and, therefore, Gradle is free to execute these tasks in arbitrary order.

The ordering between checkDist and assemble introduced by mustRunAfter doesn’t imply any ordering between their dependencies.

There is a weak ordering based on the order of tasks in the command line but it doesn’t help there (it only affects clean).

You should introduce an explicit ordering constraint between test and dist.

But how to introduce such an explicit ordering constraint between test and dist?

3

Expected behavior is that the checkDist task is executed after the assemble task is finished.

That is the case.
But you are not complaining about checkDist running before assemble, you are complaining that :build-tests:test is running before assemble and there is no defined order for that in your build. Just that you say that checkDist has to run after assemble say nothing about the dependencies of checkDist.

I don’t think that you can or should somehow make test from the included build depend on the dist task of the including build.

If you want to test the logic of tasks in build-logic, you should most probably use tests (e.g. functional tests using TestKit) within the build-logic project and not try to have a separate test build that tests the plugin on the main build, that sounds like a quite awkward setup. You can of course have your tests in a separate build, but still it should most probably not try to test the build logic using the main build, but still there use TestKit tests to test the plugin logic.

You could probably use the Tooling API (which in the Gradle context is available directly anyway) in the execution phase of checkDist to run the build-tests build and not have it as included build, that would then guarantee the order you want as then the dependencies of checkDist would have been executed and at the execution phase of checkDist you then use the tooling API to drive that other build that executes the tests.

But again, if this is intended to test the build logic in build-logic, that would be an awkward setup. :slight_smile:

4

Thanks @Vampire for your help.

I probably I have to explain my setup a bit more in detail.
The example provided is just a minimal base project to reproduce the behaviour I noticed, and is causing me issues now.

In my actual project I have currently:

  • A main project which applies the distribution plugin
    • It has no Java code, nor any Java related output artifact
    • It has in src/data a bunch of data file which act as the source
  • A buildSrc build logic module which defines a a custom plugin to generate various output formats from sources in src/data
    • This buildSrc project applies itself the java-gradle-plugin to provide the plugin to the main project
    • The buildSrc project also applies the org.jsonschema2pojo to convert a json schema to classes
    • There is generateDist task that uses the generated classes from the org.jsonschema2pojo plugin to read the raw data from src/data, but also to enriches the raw data, and finally writes the enriched data to various output formats including json, sql etc.

The actual code can be checked here btw: f1db/buildSrc at main · f1db/f1db · GitHub

This worked quite well, but now I wanted to add some check task to verify the outputs.
E.g. to verify if the generated json complies to the schema, or that the generated sql dumps can be actually imported without errors using testcontainers.

I initially started with tests in buildSrc but I faced the same problem, I was not able to run the tests after the dist/assemble was finialized.
I tried many things like mustRunAfter, finalizedBy etc, but without success.
To move the tests the main project was also not a really option, as in the tests I would like to use the generated classes…

I then decided try a composite build as that was suggested at several places if the build logic in buildSrc becomes to complicates, or to share code.

So that’s why I moved my plugin code to a build-logic module, which basically provides the plugin to my main project.
To share the generated code between this build-logic module and a test module, I moved the shared code to build-logic-commons.
That worked, I could generate the distributions successfully.

But again I’m struggling with the tests…
If possible I don’t want to add them to the main project itself, to not clutter it up, so introduced build-tests and added them there. Maybe the name is debatable, as it does not contain unit tests for the build logic itself, but rather functional tests to verify the outputs, and e.. running database testcontainers.
Note I’ve set this up successfully, but I can’t seem be able to run these tests after the distributions is finished.

I hope this makes a bit more clear, and maybe someone can provide some new pointers which direction I should go.

5

E.g. to verify if the generated json complies to the schema, or that the generated sql dumps can be actually imported without errors using testcontainers.

And why can you not simply have your plugin add such verification tasks to the main build, e. g. as finalizedBy tasks of the generation task?

I initially started with tests in buildSrc but I faced the same problem, I was not able to run the tests after the dist/assemble was finialized.

Of course, tests in buildSrc are for testing the code in buildSrc.
To great lengths buildSrc is more or less the same as an included build except for some nuances that are different.
In this case the applies, the included build is not (and should not be) aware of the including build and not have tasks that require that some task of the including build was run before.

I then decided try a composite build as that was suggested at several places

I would always prefer an included build if possible, but as stated above for the points here they are more or less the same effectively.

Note I’ve set this up successfully, but I can’t seem be able to run these tests after the distributions is finished.

That (the “successfully” is debatable. :smiley:
The tests require files of the other project and just access them by hard-coded path and you try to manually add task dependencies or ordering.
That is inherently broken and bad practice even within one build or even within one project.

If your “test build” wants to consume things of your “main build”, then the “test build” should include the “main build”, not the other way around.
And you should also then not try to use direct dependencies from includedBuilds and hard-coded paths, but properly consume things from the one build in the other by normal dependencies, e.g. similar to what is shown on How to share outputs between projects

6

Thanks again for your feedback!

And why can you not simply have your plugin add such verification tasks to the main build, e. g. as finalilzedBy tasks of the generation task?

Maybe that’s the simplest solution and I would be absolutely fine with that…
E.g. the plugin could provide next to the generateDist an additional verifyDist task.

But can that verifyDist task then run junit tests (and where should the be located), and still use the generated Java code in buildSrc?

Thx,
Marcel

7

No JUnit tests would there probably not be the right thing.
Validating against a schema probably would be.
If you want JUnit tests then the other options are probably more viable.

8

I see. Currently I had some schema validation as part of the generateDist task.
But I want to extend that no to also validate sql dumps using testcontainers, and junit with testcontainer annotations seemed a nice solution, als for the reporting of failed tests…

9

Quite possible.
As I said, you should then probably have a test build that includes the main build an depends on the main build artifacts and then has the tests.