[Proposal] Task parallelization group

I have searched on the internet to find out if it is possible to run tasks from a single project build in parallel in Gradle. The answer appears to be “no”. Apparently there is some way to run some work in parallel using WorkerExecutor, but this seems to be overly difficult and in any way not what I want.

I have come up with a simple proposal. Let Gradle tasks have parallelization groups. Those could be e.g. simple strings, though that’s up to Gradle developers to decide. As long as two (or more) tasks have the same explicily set group, they can be run in parallel. Since it is fully up to the user to assign tasks to groups, any resulting failure can be blaimed onto the user. In other words, Gradle itself doesn’t have to provide tasks that are “safe” in this respect: it is the responsibility of the user to decide if in his build given two tasks are independent from each other or not.

For example, given a build with tasks minifyJs and minifyCss, I would likely assign the two tasks to the same parallelization group like e.g. this:

task ('minifyJs', type: ...) {
    parallelizationGroup = 'minifying'
}
task ('minifyCss', type: ...) {
    parallelizationGroup = 'minifying'
}

But in any case, Gradle wouldn’t impose anything on me: if I see the tasks are really independent, I do this, but if I don’t, by default Gradle simply always runs the tasks sequentially, as now.

Proposed Gradle behavior when deciding which task to run:

  • build a set of tasks with dependencies (dependsOn()) already satisfied;
  • if at least two tasks from the set have the same user-assigned parallelization group, run all (or at least some, subject to available CPUs, for example) those tasks in parallel;
  • otherwise, i.e. if all tasks belong to different groups or have no groups set at all, behave as now.

Advantages:

  • more control for the users: they know better what the tasks are supposed to do and thus can determine if they are independent or not;
  • user is fully responsible for assigning parallelization groups, meaning that Gradle itself is not to blame if something is not correct (just like if a user forgets to add dependsOn() where needed);
  • applicable to any standard task, tasks with doLast() and what not; doesn’t require custom tasks;
  • requires minimal changes to build.gradle; likewise, requires minimal changes to disable this.