Extra side-effects of Project.getTasksByName()?

Help/Discuss
#4

The plugin is already working much as you describe

No, it’s not. The fact that it’s using Set<String> and Set<Task> shows the problem.

Please read here for related topic

#5

This Set<String> contains the task names read from Gradle’s command line parameters:

Set<String> requestedTaskNames = new HashSet<>(project.getGradle().getStartParameter().getTaskNames());

I am certain that this Set<String> is “static” and not “live”, and is therefore safe to use in a matching() lambda.

However, I do agree that then using findByPath() to convert the Set<String> to Set<Task> is a Bad Idea, and have removed this code.

BTW, would it be fair to say that Project.findTasksByName() will eagerly configure every single task in the entire project tree? (And is therefore toxic to any efforts to use task configuration avoidance?)

Cheers,
Chris

#6

I am certain that this Set<String> is “static” and not “live”

I agree, this is fine

However, I do agree that then using findByPath() to convert the Set<String> to Set<Task> is a Bad Idea, and have removed this code.

Ok, great. I assume you have replaced with tasks.matching { ... }.all { ...}

would it be fair to say that Project.findTasksByName() will eagerly configure every single task in the entire project tree?

I’m assuming you are referring to Project.getTasksByName(...) and I wouldn’t jump to this conclusion. All of the register(...) methods require a task name. So I think it’s fair to assume Gradle would only configure the tasks with the provided name. But, as we’ve discussed, this method is not “live” so it’s best to avoid so that your plugin is agnostic to the order in which it is applied.

#7

Here’s a few comments on your original post.

This note is a disclaimer about the impact it will have on the --configure-on-demand feature, which attempts to limit the number of projects configured to only those that are relevant to the build. If you’re not using --configure-on-demand, or your projects depend on each other, they will all be configured regardless, so this specific NOTE isn’t impactful.

It’s not necessarily more. Project.findTasksByName() and Project.evaluationDependsOnChildren() are doing completely different things. Project.evaluationDependsOnChildren() can absolutely break things horribly depending on your project.

Basically, Project.evaluationDependsOnChildren() pauses the execution of the code in the build.gradle where it is called and causes the build.gradle of the child projects to be evaluated before continuing. This might be necessary if you’re configuring something in a parent project that you need to reference, but it is created in the subproject’s build.gradle.

A contrived example could be that you want your subprojects to have:

plugins {
    id 'java'
}

but want to put this configuration in the root project (compileJava exists only after applying the Java plugin):

subprojects {
    compileJava {
        encoding = 'UTF-8'
    }
}

Yes, there’s much better ways to handle this, but it’s generally this kind of not quite following best practices that ends up introducing ordering dependencies anyway.

If you have the opposite (or your plugins do) where the subprojects are actually expecting something to be accessible from or inherited from the root, calling this will definitely break things. This really shouldn’t be used in a plugin because it makes little sense for the plugin to force a different order for what’s in the build.gradle files (i.e. not in the plugin).

#8

Hi, thanks for the explanation about Project.evaluationDependsOnChildren().

This particular plugin can only be applied by the root project, and is invoking evaluationDependsOnChildren() as the first step in the root’s Project.afterEvaluate() handler. The intention is that whatever the handler does next can assume that absolutely everything else in the entire project tree has been evaluated, and so exists to be found. I can’t think of any other way of achieving this either, short of maybe applying this plugin on the very last line of the root project’s build.gradle file :face_vomiting:…

#9

I consider afterEvaluate {...} and evaluationDependsOnX() as hacks that should be avoided. Think of the case where two plugins want to perform some logic in an afterEvaluate closure. You can often achieve the desired behaviour using the “live” APIs.

When using the “live” APIs, a plugin is (usually) agnostic to evaluation order

Creating MavenPublication lazily - How?
#10

Oh, I agree. I wrote my first Gradle plugin back in the Gradle 3.x days, and was burnt by afterEvaluate() when I needed to coordinate with another (third party) plugin, but couldn’t because there is no way to control whose afterEvaluate() handler runs first. So naturally I’ve since ported our plugins to use lazy properties, and more recently to use “task configuration avoidance” too.

I am only using Project.afterEvaluate() in this particular case because I really cannot think of any other way of achieving what this plugin needs to do.

#11

Please start another thread with your actual problem, NOT your current solution

#12

Thanks, but my actual problem was solved by removing Project.getTasksByName(...) from the plugin. I am still using

project.evaluationDependsOnChildren()

in my root project’s afterEvaluate() handler, but I would only describe this as “vulgar and distasteful” :wink:.

#13

As I said, afterEvaluate and evaluationDependsOnX are both hacks. Perhaps if you describe the problem they can be eliminated

#14

Thanks, I may take you up on that offer at some point. However, for now, the plugin is working sufficiently well that I need to focus more on what it is supposed to be doing rather than how it is using Gradle. I may have had several dark and miserable experiences with Project.afterEvaluate() in the past, but it is still a documented and supported API that everyone learns about from Gradle’s introductory guide. Using both it and evaluationDependsOnChildren() is far better that invoking

project.getTasksByName("doesNotExist");

in order to configure the entire project tree by force, which is what it was doing before. Now at least its dirty little secret is exposed. The plugin has duly signed the “Gradle Offenders” register, and it will no longer be allowed to frighten small children at parties.

That will have to do for now.
Thanks again,
Chris

#15

project.getTasksByName(“doesNotExist”)

Just so you know there’s

gradle.taskGraph.whenReady { ... }

Which is an event where you can inspect the TaskExecutionGraph and perform actions based on the tasks that are, or are not, in the task graph.

There’s also Task Rules where you can create dynamic/missing tasks based on name patterns.

1 Like
#16

The problem with using TaskExecutionGraph is that once the graph has been built, it is by definition too late to create any more tasks or add any new task dependencies. I have looked at using ProjectEvaluationListener, but this just appears to be Project.afterEvaluate() again in a not very good disguise. However, it would allow me to coordinate afterEvaluate() handling across the entire project tree.

I haven’t considered using TaskRules before, and will need to think about this.

Cheers,
Chris

1 Like
#17

We have had to remove the

project.afterEvaluate(Project::evaluationDependsOnChildren);

line from our plugin in the end because we have discovered that evaluationDependsOnChildren() only forces the evaluation of the immediate child projects and not of all sub-projects. We then toyed briefly with

project.subprojects { sub ->
    project.evaluationDependsOn(sub);
}

but I think that way would have lead to madness. We have finally settled on replacing afterEvaluate() with

project.getGradle().projectsEvaluated(new OurHandler(project));

class OurHandler implements Action<Gradle> {
    ...
}

which does what we had been trying to do all along, i.e. execute a block of code after every project has been evaluated but before the task graph has been finalized.

#18

Just one last try at an event based solution, have you consider using the live methods on PluginContainer

Eg:

allprojects {
   plugins.withType(JavaPlugin) {
      // do stuff only if java plugin is applied 
      tasks.withType(JavaCompile) { ... } 
   } 
   plugins.withType(FooPlugin) {
      // you get the idea 
   } 
}
#19

Thanks, but it no longer matters. The plugin already has an appointment scheduled with The Wicker Man, and there is no more time, money or appetite for anything other than this simple fix before we all hold hands and sing songs while it burns.

The plugin might have escaped this fate if I’d realised a couple of months ago that Project.evaluationDependsOnChildren() only affects the project’s immediate children and doesn’t traverse all sub-projects(*), but it’s too late for that now. I have at least checked the rest of my code to be sure that I’m not using it anywhere else (which thankfully I am not), because there are no circumstances where “immediate children” would be the behaviour that I would want.

Judging by your reaction, it sounds like the projectsEvaluated() handler it yet another of those “documented and un-deprecated but still should never be used under any circumstances” APIs that make Gradle’s programming model so ridiculously complicated. Duly noted, but it still does exactly what we were wanting all along.

At least I won’t have to update this plugin to support “configuration caching” now :relieved:.

Cheers,
Chris

(*) Possible, but not guaranteed - I was trying to salvage its “donkey” at the time even then.

#20

The problem with Project.afterEvaluate and anything similar is that you want to run “after everything else”. What happens when another plugin comes along that also wants to run last? You’re just kicking the can down the road.

Unfortunately, you never actually described your problem. In reality, you probably don’t need to run “after everything else”. You probably actually need to run “after some specific type of interaction”. It’s much better to target the exact event rather the blanket “run after everything” approach.

This reminds me of web designers and z-index where everyone wants their element to “be on top of everything else on the page”. So the first developer sets z-index=999 on their element, then the next sets z-index=9999 on their element etc, etc, etc. Before you know it you’re up to 999999999

#21

I am aware of the limitations of afterEvaluate. The plugin was trying to divide up instances of a large project’s Test tasks into jobs which could be executed in parallel on a Kubernetes cluster. Some of these tasks were created by the Java plugin, but others were created by hand, and only when all projects had been evaluated could the “sub-dividing” process begin.

It seem highly unlikely that someone would need to create a Test task at the very end of the build, but even if they did then we were also in a position to tell them not to instead.

And I am quite sure that this plugin could have been written completely differently from the outset, but that is not where I was starting from. Sometimes evolution presents you with something like a duck-billed platypus and you just have to do what you can in the time available.

Cheers,
Chris

#22

The plugin was trying to divide up instances of a large project’s Test tasks into jobs which could be executed in parallel on a Kubernetes cluster

Ah, finally the actual problem statement!

You could

  1. Add a subdividing task to the model
  2. use tasks.withType(Test){...} to listen to the task creation event.
  3. Make each test task “dependsOn” the subdividing task
  4. Configure each test task with a Provider that points to the kubernetes node/ingress/load balancer /whatever. This Provider is not initialized at configuration time
  5. When the subdividing task runs during the execution phase, it calculates/populates all the providers. Since all test tasks depend on the subdividing task its guaranteed to run before the tests
#23

Yes, a lot of that sounds familiar. We also have to generate a Docker image, of course, and choose which kind of Test task (unit, integration or other) we’re targeting. And generate a set of appropriate Gradle command lines.

And then there are the added complications of the “task configuration avoidance” because we don’t want to configure the tests unless we’re actually going to run them.

I can pass your suggestions on to Lord Summerisle, but I strongly suspect he will be far more concerned about next year’s apple harvest.