What is the status and future of parallel task execution?

Old Forum Archive
1

I have a project currently using a custom Groovy DSL that is really an ad-hoc build/automation tool. From what I have read in the Gradle examples and documentation, it seems like it would work very well as a Gradle plugin, leveraging Gradle’s infrastructure for many things that I am currently doing myself.

However, one of my major needs is for parallel operations. I’ve seen that parallel task execution is on the horizon for Gradle, but doesn’t seem to be on the 1.6. roadmap.

Is parallel task execution something that is planned for inclusion soon or at least actively being worked on, or something that’s nice to have but will not be available in the near future?

2

It’s already there, but in incubating form (read: has some caveats to be aware of). Lots of users are using it already though. We are actively working on improve it in every release.

http://www.gradle.org/docs/current/userguide/multi_project_builds.html#sec:parallel_execution

1 Like
3

From what I have read, that seems to only apply to building multiple projects in a multi-project build in parallel. Is there any support for parallel builds of tasks within the same project?

4

Not yet.

5

From what I understand, the parallel-threads parameter allows me top specify the number of threads I want to use, but limited to the number of cores of my server.

I use distributed functional tests controlled by my Gradle script and I want to allow as many threads as possible to spawn functional tests towards other hosts.

Why limit the number of threads when I try to specify more? It appears that Gradle is trying to be too clever by telling me that my specified parameters are probably wrong and it will calculate a better value for me.

6

From what I understand, the parallel-threads parameter allows me top specify the number of threads I want to use, but limited to the number of cores of my server.

What makes you think this? Gradle doesn’t put an upper bound on.

7

So what could be the reason for me not getting more than 4 threads? The subprojects does NOT have any dependencies to each other.

8

Only having 4 subprojects?

9

That would qualify as a correct answer to my question, but not the one I was looking for.

I made a very simple project to investigate if I could possibly have some unseen dependencies in my project. The project was defined as having 6 subprojects range a-f:

subprojects {

project.task(‘theTask’) {

doFirst{

println project.name

Thread.sleep(5000)

}

} }

Running the project gave me the same result, printing a-d then a wait 5 seconds before printing the other 2.

me@host /tmp > gradle theTask --parallel-threads 6 Parallel project execution is an incubating feature. Enjoy it and let us know how it works for you. :b:theTask :a:theTask :d:theTask :c:theTask d b a c :e:theTask e :f:theTask f

BUILD SUCCESSFUL

Total time: 14.911 secs

:Same result. I am using Gradle 1.5 on JVM: 1.7.0_07. Should I file a bug report?

10

The plot thickens, there is an upper bound of 4.

https://github.com/gradle/gradle/blob/master/subprojects/core/src/main/groovy/org/gradle/execution/taskgraph/ParallelTaskPlanExecutor.java#L69

I missed when this was added and don’t know why it’s there. Apologies for the bad info.

I’m asking amongst the developers why this was added and will report back.

11

Raised http://issues.gradle.org/browse/GRADLE-2760 and removed. Now the number of projects is the upper bound.

This will be available in the next nightly: http://www.gradle.org/nightly.

12

Great! I will pick it up when released. Thanks.

13

I removed the star again as I misunderstood the meaning of it.

What I meant was, thanks for the reply!

14

Hi, is there already a ticket I can watch?

15

Parallel task executions are said to be only for multi-project builds. My single-project build contains many tasks which can be executed in parallel and it is used often. Too bad, it won’t benefit from the 6 cores of my CPU. Is there any plan in the future to add parallelism in single-project builds?