[Proposal] Task parallelization group

Help/Discuss
1

I have searched on the internet to find out if it is possible to run tasks from a single project build in parallel in Gradle. The answer appears to be “no”. Apparently there is some way to run some work in parallel using WorkerExecutor, but this seems to be overly difficult and in any way not what I want.

I have come up with a simple proposal. Let Gradle tasks have parallelization groups. Those could be e.g. simple strings, though that’s up to Gradle developers to decide. As long as two (or more) tasks have the same explicily set group, they can be run in parallel. Since it is fully up to the user to assign tasks to groups, any resulting failure can be blaimed onto the user. In other words, Gradle itself doesn’t have to provide tasks that are “safe” in this respect: it is the responsibility of the user to decide if in his build given two tasks are independent from each other or not.

For example, given a build with tasks minifyJs and minifyCss, I would likely assign the two tasks to the same parallelization group like e.g. this:

task ('minifyJs', type: ...) {
    parallelizationGroup = 'minifying'
}
task ('minifyCss', type: ...) {
    parallelizationGroup = 'minifying'
}

But in any case, Gradle wouldn’t impose anything on me: if I see the tasks are really independent, I do this, but if I don’t, by default Gradle simply always runs the tasks sequentially, as now.

Proposed Gradle behavior when deciding which task to run:

  • build a set of tasks with dependencies (dependsOn()) already satisfied;
  • if at least two tasks from the set have the same user-assigned parallelization group, run all (or at least some, subject to available CPUs, for example) those tasks in parallel;
  • otherwise, i.e. if all tasks belong to different groups or have no groups set at all, behave as now.

Advantages:

  • more control for the users: they know better what the tasks are supposed to do and thus can determine if they are independent or not;
  • user is fully responsible for assigning parallelization groups, meaning that Gradle itself is not to blame if something is not correct (just like if a user forgets to add dependsOn() where needed);
  • applicable to any standard task, tasks with doLast() and what not; doesn’t require custom tasks;
  • requires minimal changes to build.gradle; likewise, requires minimal changes to disable this.
2

I cannot speak for the Gradle guys, and if you really want to suggest this, I’d recommend opening a feature request issue on GitHub instead, this is a community forum.
But I don’t think this will be implemented, especially as with the upcoming configuration cache feature all tasks can be run in parallel if they don’t have an interdependency.

Btw. regarding

just like if a user forgets to add dependsOn() where needed

whenever you add a manual dependsOn() (except when the left-hand side is a lifecycle task) you are most probably adding a code smell.
You should usually wire task outputs to task inputs or other places where implicit task dependencies are understood and by this automatically get the necessary task dependency for all required cases.

Just a small example, if you have a task that generates some code and then register the path manually as source path and adding a manual dependency from the compile task, to the generate task, you immediately have an invalid build. Any other place that needs sources will either miss the generated sources if the generate task didn’t happen to run before, or get stale sources if the generate task run before but not in the same invocation but previously. If you instead properly register the task output (or if all outputs are fine the task itself) as source directory, then it is automatically a dependency for all consumers of the source files like JavaDoc task, sources jar task, …