Depending on specific subprojects tasks in a lazy way

Help/Discuss
1

I’m on big project that was started way before configuration cache was a thing. Or lazy API, or Kotlin DSL.

I just noticed this task

  task("instrumentationReport") {
    dependsOn(project.getAllTasks(true).values().flatten().findAll { it.name.endsWith("generateReport") })
    finalizedBy(tasks.named('mergeReports'))
  }

So getAllTasks do not appear lazy especially being recursive, it’s sibling getTasksByName don’t look better. And I was thinking about lazy alternatives.

  1. This looks interesting but it requires to this is enough subprojects.tasks.collect { it.named("generateReport") }
  2. dependsOn can take a Provider , which could be used to compute the tasks using getAllTasks. Or the alternative in (1), inspired by link below
  3. Apply the task dependency to parentProject.tasks.named(mergeReports) when generateReport is registered.

Out of curiosity what would be the best approach to be lazy when writing task rule (tasks::addRule), e.g. to search among tasks names. To be honest I barely know tasks rules, but I think I discovered some interesting use cases, like handling our own task pattern, for example handling numbers in task names.

2

There is no direct way.

All alternatives you showed are bad, because they either do cross-project configuration, or at least cross-project querying which is almost as bad.

The only “proper” way is using string-y dependencies like subprojects.forEach { dependsOn("${it.path}:generateReport") } which of course requires that each and every project has a task with that name, whether it does any work or not.

The other way is to request a specific variant of all projects and then in the projects make sure that variant is depending on the intended task. This would not require the variant being present on all tasks if an artifact view is used.

3

Note, that getting a “global” merged report for all subprojects is what is wanted here. So the cost when running that task looks acceptable, ie even if it configures or query every suprojects.

I understand your point though. Going into variants seems a bit early in this project.

By the way there are indeed some “empty” subprojects without that task, which sadly prevents the string-y apprach.

I wonder why such capability is missing it seems like a gap in the API that could be useful for plug-ins or build script authors without going to other concepts.

4

So the cost when running that task looks acceptable

It’s not only about cost, it is highly discouraged and bad practice.
It will work against more sophisticated Gradle features and optimizations.
For example the build will probably never become compatible with isolated projects that way.

Also, it is not only about running, but also about configuring.
The runtime is the same, the configuration is what makes a difference here.
And also as shown this task does not leverage task-configuration avoidance so will be configured on each build invocation, no matter which task or if any at all is invoked.

Going into variants seems a bit early in this project.

You practically always work with variants, even if you are not aware. And using it here is really the best option that comes to mind unless you make sure every project has that task in question.

Also, as you wrote “Note, that getting a “global” merged report for all subprojects is what is wanted here.”, you anyway also have to get the data for the merged report over from the subprojects to the merger project. also there you must not reach into the other project or just use the files from disk. You really should do it like the JaCoCo report aggregation and Test report aggregation plugins.

I wonder why such capability is missing it seems like a gap in the API that could be useful for plug-ins or build script authors without going to other concepts.

Which capability?

5

Yeah by cost I was also including configuring. But I confer this task is only registered and run in CI.

Which capability

I meant the capability of “making” a task that depends on other sub-projects that have such task(s). I.e. having the ability to write something as simple as that subprojects.forEach { dependsOn("${it.path}:generateReport", TaskExistance.IGNORED) } that would ignores when a task do not exists.

And also as shown this task does not leverage task-configuration avoidance so will be configured on each build invocation, no matter which task or if any at all is invoked.

Even by wrapping its dependsOn in a regular provider ? If the invoked task is not invoked or depended upon shouldn’t it be ignored and as such its dependsOn provider not invoked ?

You really should do it like the JaCoCo report aggregation and Test report aggregation plugins.

Thanks I”ll take a look.

Also, I believe it’s quite fine to not have to deal with variants if you don’t have to. Regardless Gradle use variants anyway. Maybe that’s something that is actually needed in this case as it is indeed looking a lot like jacoco / test report, but for tasks dependencies I believe there the small gap in the API as I mentionned above.

And on a close topic, I’m not sure how one could use tasks.addRule in a lazy way. Yet thinks looks interesting. I don’t need that, yet having to deal with number in tasks this could be useful.

6

I meant the capability of “making” a task that depends on other sub-projects that have such task(s). I.e. having the ability to write something as simple as that subprojects.forEach { dependsOn("${it.path}:generateReport", TaskExistance.IGNORED) } that would ignores when a task do not exists.

Maybe just noone needed it so far?
Well, the functionality is there, it is getTasksByName("generateReport", true), it is just not a good idea as this is also cross-project accessing stuff that you shouldn’t do. :slight_smile:
For a lazy / safe / IP-conform way, you would probably need to open a feature request.

Even by wrapping its dependsOn in a regular provider ? If the invoked task is not invoked or depended upon shouldn’t it be ignored and as such its dependsOn provider not invoked ?

Well, the task as shown yes, as it is not registered but created by using task("...") as that predates even the ability to use task-configuration avoidance.
How it is with the heavy work wrapped in a provider, that might work nevertheless, not sure just try it by adding a println or a breakpoint inside.
But taken all the time we talk here, you probably could have solved it the proper way already. :smiley:

but for tasks dependencies I believe there the small gap in the API as I mentionned above.

But you don’t only need task dependencies, do you?
You also need the information to aggregate just like the JaCoCo and Test report aggregation plugins.

And on a close topic, I’m not sure how one could use tasks.addRule in a lazy way. Yet thinks looks interesting. I don’t need that, yet having to deal with number in tasks this could be useful.

What do you mean by “in a lazy way”?
Actually I never felt the urge to use a task rule so far.
One I know that is handy is the one that adds a clean... task for all existing tasks automatically that cleans away its outputs.
But for most things I’d probably prefer an @Option unless you want to run multiple of such tasks in one invocation with different parameters.

7

But taken all the time we talk here, you probably could have solved it the proper way already. :smiley:

I have an acceptable workaround for now, but I have need to tackle bigger issues than this one, so it’s not yet prioritized to have the “best” solution there. Thanks for the discussion and the food for thought by the way !

Indeed. That’s a good point!

Well, the task as shown yes, as it is not registered but created by using task("...") as that predates even the ability to use task-configuration avoidance.

Yeah sorry, it’s code I noticed, I fixed that right away with a task registration.

For a lazy / safe / IP-conform way, you would probably need to open a feature request.

OK, fair :slight_smile:

For the addRule I was thinking of things like handling tasks like :foo:allTests, where the foo project has different test suites, and one want to run all of them.

Other use case unrelated to the original discussion could be expanding a number to mean something, unsure if that’s doable tough e.g. :foo:allTests21 where 21 could be expended to some property, something equivalent to :foo:allTests -Pjvm=21.

Clearly there’s no urge, and I’m not sure if it’s a good idea at this point either. But for the allTests idea this could require to ask projects if they have tasks that match a criteria.

8

For the addRule I was thinking of things like handling tasks like :foo:allTests, where the foo project has different test suites, and one want to run all of them.

That’s not a use-case for a task rule.
That should simply be

val allTests by tasks.registering {
    dependsOn(tasks.withType<Test>())
}

I think.

Other use case unrelated to the original discussion could be expanding a number to mean something, unsure if that’s doable tough e.g. :foo:allTests21 where 21 could be expended to some property, something equivalent to :foo:allTests -Pjvm=21 .

As I said, unless you want to run multiple such tasks with different input in the same invocation, I’d just use an @Option so that you can do :foo:allTests --jvm 21 .

But for the allTests idea this could require to ask projects if they have tasks that match a criteria.

Well, …, if you conclude that it is a good, idea, …, did I mention artifact/variant-aware resolution? :slight_smile:

9

OK I get the message :sweat_smile:

Also, I forgot to mention but I also saw some folks passing glob style paths, e.g. :foo:b*:compile*Java. The add rule discovers projects and tasks that match the path glob. And I found this an interesting cli ux.

10

Found the place that suggested the above usage for addRule: How to run a task for a subset of multiproject-build

11

Fancy.
Except that it does not work for the project part