Way to use task parallelism on Custom Tasks - Via Worker API only?


(Franz Krainer) #1

Hi Gradle folks!

Was playing a bit with parallel task execution recently. When I look at documentation it reads like when having two different custom tasks in two different projects and following certain rules (no doLast, no doFirst, etc.) that Gradle shall realize that these are parallelizable. In my case with Gradle 4.5.1 they are clearly not parallelized, using gradle clean build --parallel; they are executed one after another on one executor.

This is my simple test code:

class GreetingTask extends DefaultTask {
private File greets_input;
private File greets_output;

@InputFiles
public File getGreetsInput() {
return this.greets_input;
}

@InputFiles
public setGreetsInput(File input) {
this.greets_input=input;
}

@OutputDirectory
public File getGreetsOutput() {
return this.greets_output;
}

@OutputDirectory
public File setGreetsOutput(File output) {
this.greets_output=output;
}

@TaskAction
def greet() {
int count=0;
1.upto(10000000, {
count++;
})
}
}

project(":postProcessorProject3") {
task postProcessor3 (type : GreetingTask) {
setGreetsInput(file("$rootDir/emptyInput/test.txt"))
setGreetsOutput(file("$rootDir/emptyOutput3"))
}
}

project(":postProcessorProject2") {
task postProcessor2 (type : GreetingTask) {
setGreetsInput(file("$rootDir/emptyInput/test.txt"))
setGreetsOutput(file("$rootDir/emptyOutput2"))
}
}

task postProcessor1 () {
dependsOn tasks.getByPath(’:postProcessorProject2:postProcessor2’)
dependsOn tasks.getByPath(’:postProcessorProject3:postProcessor3’)

doLast{
	1.upto(10, {
		println "Number ${it}"
	})
}

}

So, my question is eventually:
Is the Worker API the only way now to make Custom Tasks being parallelized? If yes, how does this work for your own plug-ins? All Worker-API-ized already :-)? In your docu it seemed as if using different projects could be sufficient for Custom Tasks already (the example in docu is generic).

Thanks a lot in advance!
BR,
Franz Krainer


(Stefan Oehme) #2

They should run in parallel if they are in different projects, don’t depend on each other and you pass the ‘–parallel’ option. If this doesn’t work for you, please create a reproducible example.

That being said, the worker Api is the way forward as it allows safe paralellization without the ‘–parallel’ flag and even within the same project.


(Franz Krainer) #3

Hi @st_oehme!

I think I might have possibly found the issue. It is as you said but what I didn’t tell here in the example is that I actually kicked off the central postProcessor1 via ‘build.finalizedBy(tasks.getByPath(’:postProcessor1’))’ from a previously happening native build.
Is maybe finalizedBy killing the parallelism? If I remove this finslizedBy calling structure, then all works: with Worker API and with Multi-Project, whatever I choose, works fine.
Is this limitation on finalizedBy intentional? If this is the wrong thing to use what would be fine (just working with ‘dependsOn’ possibly)?

BR,
Franz


(Stefan Oehme) #4

It’s really hard to tell without seeing an example repo and hearing the use case behind it.

In any case, finalizedBy shouldn’t limit parallelism here, so having an example would help us track down what’s going on.


(Franz Krainer) #5

Hi @st_oehme!

Sry for my very late reply - I am on on a somehow-vacation ;-).
Pls. see attached my examples:

  • The Good One leading to parallel execution of two post processing tasks (NOT using finalizedBy in the beginning of triggering post processing)
    – Example code attached
    – Build scan: https://scans.gradle.com/s/7quc5rugzrxu4

  • The Bad One leading somehow to sequential execution (using finalizedBy in the beginning of triggering post processing)
    – Example code attached
    – Build scan: https://scans.gradle.com/s/2mpc37b2fbqiw

Both builds were started via ‘gradle clean build --parallel’

What do you think about these measurements? Following your stmts. before, this is not as expected, right?

Thanks a lot in advance!
BR,
Franz

TheBadNonParallelOne.zip (6.2 KB)
TheGoodParallelOne.zip (6.1 KB)


(Franz Krainer) #6

Hi @st_oehme!

Did not hear back from you so far!
Any update on our story here? Did you have a bit of time to look into my examples?

Thanks a lot in advance!
BR,
Franz


(Stefan Oehme) #7

Sorry I did not have time to look into this yet, but thank you very much for the examples, that should make investigation much easier!


(Franz Krainer) #8

Hi,

ok; np.
This is a central point for our business application. That’s why I am pushing a bit here. Am intending to use a bit artificial multi-project style for certain check tasks which could run in parallel. If these are not parallelized because of the problem/case I described, this would be an issue.

Looking fwd. to your analysis results and feeling!
Thanks a lot in advance!!
BR,
Franz


(Franz Krainer) #9

Hi @st_oehme, @george-moberly,

was there any update on analysis, etc. recently?
We are still stuck wrt. this story.
Do you have an idea when you could have a look into this?

Thanks a lot in advance!
BR,
Franz


(Stefan Oehme) #10

Hey Franz,

sorry for the long wait, I was very busy with all the improvements for Gradle 4.8. I’m going to get back to you on this early next week.

Cheers,
Stefan


(Franz Krainer) #11

Hi Stefan,

fine :), clear. Just wanted to remind and push here a bit to make sure we do not fully forget. Will become very interesting for us in the next optimization step.

Pls @redsquirrel, keep in mind our discussion here :).

Looking fwd. to your first analysis/answer!
Thanks a lot in advance!
BR,
Franz


(Stefan Oehme) #12

It turned out to be easier than I anticipated, there should be a nightly with a fix soon.


(Franz Krainer) #13

more than cool. Am eager to try this out when it’s there :).
Thanks a lot!