Mutex for Parallel execution


(Corneil du Plessis) #1

I would suggest a simple mechanism where a task can be linked to named mutex to ensure that the task does not run at same time as other tasks with named mutex. The task execution mechanism will lock using named mutex before execution starts and unlock when execution is completed. If no mutex is defined the task executes as normal. Sample of definition could be:

task MyTask {
    mutex = 'MyTaskMutex'
}

In a project where the same task is executed for many sub-projects the mutex will ensure serial execution for the task.


(Attila Kelemen) #2

In general, I like the idea of allowing a form explicit synchronization. I’d adjust the idea by:

  • Allow multiple mutexes. - Any type, not just strings (based on ‘equals’ and ‘hashCode’). Although this might require some special care. For example, call ‘toString’ on ‘GString’ instances. - There is a better way to implement this than explicit locks.

(Corneil du Plessis) #3

Can you elaborate on ‘There is a better way to implement this than explicit locks.’ The reason I was thinking of a mutex is that I have a large project with over 300 sub projects and when I use parallel I get a nice improvement but failures with uploadArchives to local maven repo. A mutex on uploadArchives will resolve this problem.


(Attila Kelemen) #4

Lock and wait is very dead-lock prone in this situation since there could be many locks and preventing the lock reordering issue is not trivial.

However, it can be implemented without using locks (like ReentrantLock), at least not for this purpose. For example, when Gradle executes a task it effectively pulls the task (I don’t know how it is actually implemented but should be something like that) to be executed from a set of tasks allowed to be executed (their dependencies has already been run). It should be possible to prevent pulling a task if a task is currently being executed which needs a “mutex” if the “mutex” is being held, simply by storing the held “mutexes” in a set.

So, it is effectively the same as locking without risking a dead-lock.


(Corneil du Plessis) #5

This sounds promising. I’ll take a look at the code.


(Mark Koops) #6

Did anyone of you already made progress on this?

Setting a mutex on my integration tests will allow me to run my build in parallel mode while the integration tests that depend on a single external server can performed sequentially.


Parallelizable JAXB/XJC Plugin/Task
(Attila Kelemen) #7

I didn’t do anything. I’m currently working on better NetBeans support, exploiting the changes expected in Gradle 1.8. So, I probably won’t work on this issue in the near future.


(Mark Koops) #8

I have just submitted my implementation for it on Github: https://github.com/gradle/gradle/pull/206

–Mark