Can we run tasks in parallel in single build project

manoj7shekhawat · July 2, 2016, 3:18pm

I wanted to run task parallely but I’m not able to do so. I read multiple threads and it looks like it’s NOT supported by Gradle at least for single build project.

But I also found GPars and one Gradle hidden property -Dorg.gradle.parallel.intra=true. Tried both but it doesn’t work.

Approach 1:
group 'com.oracle.parallel’
version ‘1.0-SNAPSHOT’

apply plugin: 'java'

sourceCompatibility = 1.8

@ParallelizableTask
class ParallelTask extends Test {}

task testsA(type: ParallelTask) {
    1.upto(100) {
        Thread.sleep(500)
        println "Test HACK ${it}"
    }
}

task testsB(type: ParallelTask) {
    1.upto(100) {
        Thread.sleep(500)
        println "Test hack ${it}"
    }
}

Approach 1 Output:
8:25:36 PM: Executing external tasks ‘testsA testsB --parallel -Dorg.gradle.parallel.intra=true’…
Parallel execution is an incubating feature.
Test HACK 1
Test HACK 2
Test HACK 3
…
Test HACK 99
Test HACK 100
Test hack 1
Test hack 2
Test hack 3
…
Test hack 99
Test hack 100
:compileJava UP-TO-DATE
:processResources UP-TO-DATE
:classes UP-TO-DATE
:compileTestJava UP-TO-DATE
:processTestResources UP-TO-DATE
:testClasses UP-TO-DATE
:testsA UP-TO-DATE
:testsB UP-TO-DATE

BUILD SUCCESSFUL

Total time: 1 mins 45.319 secs
8:27:22 PM: External tasks execution finished 'testsA testsB --parallel  -Dorg.gradle.parallel.intra=true'.

Approach 2:
buildscript {
repositories {
mavenCentral()
}
dependencies {
classpath “org.codehaus.gpars:gpars:1.2.1”
}
}

group 'com.oracle.parallel'
version '1.0-SNAPSHOT'

apply plugin: 'java'

sourceCompatibility = 1.8

import java.util.concurrent.*
import groovyx.gpars.GParsExecutorsPool

task testsA {
    1.upto(100) {
        Thread.sleep(500)
        println "Test HACK ${it}"
    }
}

task testsB {
    1.upto(100) {
        Thread.sleep(500)
        println "Test hack ${it}"
    }
}

task parallelTests << {

    def tasksToRun = []
    tasksToRun << 'testsA'
    tasksToRun << 'testsB'

    GParsExecutorsPool.withPool(5) { ExecutorService exService ->
        tasksToRun.eachParallel { taskToRun ->
            exService.submit({ tasks[taskToRun].execute() } as Runnable)
        }
    }
}

Approach 2 Output:
8:31:58 PM: Executing external task ‘parallelTests’…
Test HACK 1
Test HACK 2
Test HACK 3
…
Test HACK 99
Test HACK 100
Test hack 1
Test hack 2
Test hack 3
…
Test hack 99
Test hack 100
:parallelTests

BUILD SUCCESSFUL

Total time: 1 mins 49.709 secs
8:33:48 PM: External task execution finished 'parallelTests'.

Can somebody help to run these two tasks (testsA & testsB) parallely, If this is at all possible?

mark_vieira · July 2, 2016, 3:49pm

In the first example your issue is that the logic is running at configuration time (not parallelized) instead of execution time. You’ll need to put that logic in a task action (i.e. method annotation with @TaskAction). Be aware also that tasks with custom actions (i.e. doLast or doFirst) will never be run in parallel.

 Custom actions
 
 Any task that has custom actions (i.e. ones added via {@link org.gradle.api.Task#doLast(org.gradle.api.Action)} or {@link org.gradle.api.Task#doFirst(org.gradle.api.Action)})
 is not considered parallelizable even if its type carries this annotation.
 This is because it cannot be known whether the added action is parallel safe or not.

manoj7shekhawat · July 2, 2016, 6:07pm

Hi @mark_vieira, Thanks for quick response.

I tried below but still it’s NOT working. It would be really helpful if you could give me a small example that works for you.

task testsA (type: TestTaskA)

class TestTaskA extends DefaultTask {
    @TaskAction
    def test() {
        1.upto(100) {
            Thread.sleep(500)
            println "Test HACK ${it}"
        }
    }
}

task testsB (type: TestTaskB)

class TestTaskB extends DefaultTask {
    @TaskAction
    def test() {
        1.upto(100) {
            Thread.sleep(500)
            println "Test hack ${it}"
        }
    }
}

Output:
11:32:14 PM: Executing external tasks ‘testsA testsB --parallel -Dorg.gradle.parallel.intra=true’…
Parallel execution is an incubating feature.
:testsA
Test HACK 1
Test HACK 2
Test HACK 3
…
Test HACK 99
Test HACK 100
:testsB
Test hack 1
Test hack 2
Test hack 3
…
Test hack 99
Test hack 100

BUILD SUCCESSFUL

Total time: 1 mins 45.933 secs
11:34:00 PM: External tasks execution finished 'testsA testsB --parallel -Dorg.gradle.parallel.intra=true'.

mark_vieira · July 2, 2016, 8:17pm

You still need to annotate with @ParallelizableTask.

manoj7shekhawat · July 3, 2016, 5:33am

Thanks you soo much @mark_vieira, it works now. This is what I did:
group ‘com.oracle.parallel’
version ‘1.0-SNAPSHOT’

apply plugin: 'java'

sourceCompatibility = 1.8

task testsA (type: TestTaskA)

@ParallelizableTask
class TestTaskA extends DefaultTask {
    @TaskAction
    def test() {
        1.upto(100) {
            Thread.sleep(500)
            println "Test HACK ${it}"
        }
    }
}

task testsB (type: TestTaskB)

@ParallelizableTask
class TestTaskB extends DefaultTask {
    @TaskAction
    def test() {
        1.upto(100) {
            Thread.sleep(500)
            println "Test hack ${it}"
        }
    }
}

Output:
10:59:29 AM: Executing external tasks ‘testsA testsB --parallel -Dorg.gradle.parallel.intra=true’…
Parallel execution is an incubating feature.
:testsA
:testsB
Test HACK 1
Test hack 1
Test HACK 2
Test hack 2
Test HACK 3
Test hack 3
…
Test HACK 99
Test hack 99
Test HACK 100
Test hack 100

BUILD SUCCESSFUL

Total time: 54.656 secs
11:00:24 AM: External tasks execution finished 'testsA testsB --parallel -Dorg.gradle.parallel.intra=true'.

manoj7shekhawat · July 3, 2016, 2:23pm

Hi @mark_vieira,

Can we apply same parallel concept to Test type tasks? Like I wanted to run testsA & testsB in parallel:

task testsA(type: Test) {
    options.suites("src/test/resources/testng.xml") 
    reports {
        junitXml.outputPerTestCase = true
    }
    reports.junitXml.destination = "$buildDir/results/A"
}

task testsACopy(type: Copy) {
    from "$buildDir/results/A"
    into "$buildDir/results"
    rename 'TEST-(.+).xml' , 'A-$1.xml'
}
testsA.finalizedBy testsACopy

task testsB(type: Test) {
    options.suites("src/test/resources/testng.xml") 
    reports {
        junitXml.outputPerTestCase = true
    }
    reports.junitXml.destination = "$buildDir/results/B"
}

task testsBCopy(type: Copy) {
    from "$buildDir/results/B"
    into "$buildDir/results"
    rename 'TEST-(.+).xml' , 'B-$1.xml'
}
testsB.finalizedBy testsBCopy

mark_vieira · July 8, 2016, 8:35pm

This should work, however typically tests are parallelized via a single task with maxParallelForks set to something greater than 1.

radhakrishnan10192 · February 17, 2017, 10:24am

Hi Mark,

Based on this discussion i assumed Gradle supports parallel build for single projects too and replicated the same in my project to reduce the build time.

I am not able to run more than one parallel tasks collection in a proper order. Please check like Parallel task execution breaks the order when parallel ,normal tasks mixed together for more detail and let me know why its behaves like this.

mark_vieira · April 2, 2018, 2:57pm

The @ParallelizableTask annotation has been removed in Gradle 4.0. I suggest you use the Gradle Worker API if you want to leverage intra-project parallelization.

grv87 · November 12, 2018, 5:31pm

@mark_vieira, I correctly understand that Worker API allows to parallelize actions of a single task only? What if I want to run several units of work in parallel, but each unit of work has its own inputs and outputs, task dependencies and tasks depending on it? And requires its own up-to-date checking.

I see that with @ParallelizableTask I could just create several tasks. But is it feasible with Worker API?
I have to create subproject for each task right now.

mark_vieira · November 13, 2018, 5:06am

This will work as you describe with the worker API. If I have two tasks which each schedule their own work items, those tasks can be run in parallel, even when belonging to the same project.

alphaguy4 · July 19, 2019, 6:31am

Hi @mark_vieira
I have two gradle tasks, taskA and taskB. Both tasks can be worked independently. The purpose of each task to to get a jar from mavenRepo and perform some operation using javaexec. However, i saw examples of worker api but it involves injecting worker executor and stuff. I dont want to do that. I have just two simple gradle tasks and i want to run them in parallel to reduce build time. How can i use worker api here ?
Any link/resources would help.

Thanks

mark_vieira · July 19, 2019, 3:01pm

Unfortunately, the only way to get intra-project parallelism is use the worker api. There is a bit of boilerplate required to get going but it’s pretty simple.

https://guides.gradle.org/using-the-worker-api/

Lance · July 19, 2019, 6:33pm

Last I checked, the worker api is very limited since you don’t have access to the Project instance so can’t use files(…) fileTree(…) copy(…) and exec(…) etc

alphaguy4 · July 22, 2019, 10:51am

@mark_vieira can you answer this https://stackoverflow.com/questions/57143637/using-gradles-worker-api-to-reduce-build-time-in-gradle-build-task

alphaguy4 · July 25, 2019, 5:49am

@Lance If so, then worker api is not very useful then. If we dont have project instance then how are we suppose to do any operations related to project? one of the task i have is to download a JAR form maven repo and use javaexec. if i cannot use copy, exec() how can i leverage WorkerAPI in this case?
@mark_vieira can you help answer this ?

Lance · July 25, 2019, 6:43am

If we dont have project instance then how are we suppose to do any operations related to project?

You can pass read only information from the main thread to the worker thread via WorkerConfiguration.setParams(Object...) and @Inject on the worker class. Keep in mind that workers are running on a separate jvm to the main thread so I think params need to be serializable (so you can’t pass the project instance)

if i cannot use copy, exec() how can i leverage WorkerAPI in this case?

It’s my opinion that Gradle needs to make file operations, javaexec etc available to workers somehow without actually sharing the mutable project with workers. Until then workers only have access to core java APIs so they are only useful in limited scenarios

alphaguy4 · July 26, 2019, 10:45am

@parallelizableTask was more useful it seems as i can directly use it over my tasks written in build.gradle where i have the instance for project.This enabled parallelizing task of single project. After Gradle 4.0 or later this was deprecated but introduced worker api doesn’t have this support when project object is required.

tricketg · July 30, 2019, 2:22pm

Agree with alphaguy4. Have several differing long running tasks in a project that could all run in parallel, however worker API does not support this… The @parallelizableTask seemed ideal.,if apache ant can do this with the parallel task then surely gradle should.

Topic		Replies	Views
How we can execute multiple tasks in parallel using gradle 4.3.1 Plugin Portal	0	473	July 18, 2019
Parallel execution of tasks in gradle 6.5 Help/Discuss	3	402	October 18, 2023
Faster Builds please Old Forum Archive	4	565	July 18, 2013
Sample needed please on parallel build Old Forum Archive	3	663	October 26, 2013
Parallel task execution supported in gradle? Old Forum Archive	3	2835	November 10, 2011

Can we run tasks in parallel in single build project

Related topics