Can we run tasks in parallel in single build project

I wanted to run task parallely but I’m not able to do so. I read multiple threads and it looks like it’s NOT supported by Gradle at least for single build project.

But I also found GPars and one Gradle hidden property -Dorg.gradle.parallel.intra=true. Tried both but it doesn’t work.

Approach 1:
group 'com.oracle.parallel’
version ‘1.0-SNAPSHOT’

apply plugin: 'java'

sourceCompatibility = 1.8

@ParallelizableTask
class ParallelTask extends Test {}

task testsA(type: ParallelTask) {
    1.upto(100) {
        Thread.sleep(500)
        println "Test HACK ${it}"
    }
}

task testsB(type: ParallelTask) {
    1.upto(100) {
        Thread.sleep(500)
        println "Test hack ${it}"
    }
}

Approach 1 Output:
8:25:36 PM: Executing external tasks ‘testsA testsB --parallel -Dorg.gradle.parallel.intra=true’…
Parallel execution is an incubating feature.
Test HACK 1
Test HACK 2
Test HACK 3

Test HACK 99
Test HACK 100
Test hack 1
Test hack 2
Test hack 3

Test hack 99
Test hack 100
:compileJava UP-TO-DATE
:processResources UP-TO-DATE
:classes UP-TO-DATE
:compileTestJava UP-TO-DATE
:processTestResources UP-TO-DATE
:testClasses UP-TO-DATE
:testsA UP-TO-DATE
:testsB UP-TO-DATE

BUILD SUCCESSFUL

Total time: 1 mins 45.319 secs
8:27:22 PM: External tasks execution finished 'testsA testsB --parallel  -Dorg.gradle.parallel.intra=true'.

Approach 2:
buildscript {
repositories {
mavenCentral()
}
dependencies {
classpath “org.codehaus.gpars:gpars:1.2.1”
}
}

group 'com.oracle.parallel'
version '1.0-SNAPSHOT'

apply plugin: 'java'

sourceCompatibility = 1.8

import java.util.concurrent.*
import groovyx.gpars.GParsExecutorsPool

task testsA {
    1.upto(100) {
        Thread.sleep(500)
        println "Test HACK ${it}"
    }
}

task testsB {
    1.upto(100) {
        Thread.sleep(500)
        println "Test hack ${it}"
    }
}

task parallelTests << {

    def tasksToRun = []
    tasksToRun << 'testsA'
    tasksToRun << 'testsB'

    GParsExecutorsPool.withPool(5) { ExecutorService exService ->
        tasksToRun.eachParallel { taskToRun ->
            exService.submit({ tasks[taskToRun].execute() } as Runnable)
        }
    }
}

Approach 2 Output:
8:31:58 PM: Executing external task ‘parallelTests’…
Test HACK 1
Test HACK 2
Test HACK 3

Test HACK 99
Test HACK 100
Test hack 1
Test hack 2
Test hack 3

Test hack 99
Test hack 100
:parallelTests

BUILD SUCCESSFUL

Total time: 1 mins 49.709 secs
8:33:48 PM: External task execution finished 'parallelTests'.

Can somebody help to run these two tasks (testsA & testsB) parallely, If this is at all possible?

1 Like

In the first example your issue is that the logic is running at configuration time (not parallelized) instead of execution time. You’ll need to put that logic in a task action (i.e. method annotation with @TaskAction). Be aware also that tasks with custom actions (i.e. doLast or doFirst) will never be run in parallel.

 Custom actions
 
 Any task that has custom actions (i.e. ones added via {@link org.gradle.api.Task#doLast(org.gradle.api.Action)} or {@link org.gradle.api.Task#doFirst(org.gradle.api.Action)})
 is not considered parallelizable even if its type carries this annotation.
 This is because it cannot be known whether the added action is parallel safe or not.
2 Likes

Hi @mark_vieira, Thanks for quick response.

I tried below but still it’s NOT working. It would be really helpful if you could give me a small example that works for you.

task testsA (type: TestTaskA)

class TestTaskA extends DefaultTask {
    @TaskAction
    def test() {
        1.upto(100) {
            Thread.sleep(500)
            println "Test HACK ${it}"
        }
    }
}

task testsB (type: TestTaskB)

class TestTaskB extends DefaultTask {
    @TaskAction
    def test() {
        1.upto(100) {
            Thread.sleep(500)
            println "Test hack ${it}"
        }
    }
} 

Output:
11:32:14 PM: Executing external tasks ‘testsA testsB --parallel -Dorg.gradle.parallel.intra=true’…
Parallel execution is an incubating feature.
:testsA
Test HACK 1
Test HACK 2
Test HACK 3

Test HACK 99
Test HACK 100
:testsB
Test hack 1
Test hack 2
Test hack 3

Test hack 99
Test hack 100

BUILD SUCCESSFUL

Total time: 1 mins 45.933 secs
11:34:00 PM: External tasks execution finished 'testsA testsB --parallel -Dorg.gradle.parallel.intra=true'.

You still need to annotate with @ParallelizableTask.

1 Like

Thanks you soo much @mark_vieira, it works now. This is what I did:
group ‘com.oracle.parallel’
version ‘1.0-SNAPSHOT’

apply plugin: 'java'

sourceCompatibility = 1.8

task testsA (type: TestTaskA)

@ParallelizableTask
class TestTaskA extends DefaultTask {
    @TaskAction
    def test() {
        1.upto(100) {
            Thread.sleep(500)
            println "Test HACK ${it}"
        }
    }
}

task testsB (type: TestTaskB)

@ParallelizableTask
class TestTaskB extends DefaultTask {
    @TaskAction
    def test() {
        1.upto(100) {
            Thread.sleep(500)
            println "Test hack ${it}"
        }
    }
}

Output:
10:59:29 AM: Executing external tasks ‘testsA testsB --parallel -Dorg.gradle.parallel.intra=true’…
Parallel execution is an incubating feature.
:testsA
:testsB
Test HACK 1
Test hack 1
Test HACK 2
Test hack 2
Test HACK 3
Test hack 3

Test HACK 99
Test hack 99
Test HACK 100
Test hack 100

BUILD SUCCESSFUL

Total time: 54.656 secs
11:00:24 AM: External tasks execution finished 'testsA testsB --parallel -Dorg.gradle.parallel.intra=true'.
1 Like

Hi @mark_vieira,

Can we apply same parallel concept to Test type tasks? Like I wanted to run testsA & testsB in parallel:

task testsA(type: Test) {
    options.suites("src/test/resources/testng.xml") 
    reports {
        junitXml.outputPerTestCase = true
    }
    reports.junitXml.destination = "$buildDir/results/A"
}

task testsACopy(type: Copy) {
    from "$buildDir/results/A"
    into "$buildDir/results"
    rename 'TEST-(.+).xml' , 'A-$1.xml'
}
testsA.finalizedBy testsACopy

task testsB(type: Test) {
    options.suites("src/test/resources/testng.xml") 
    reports {
        junitXml.outputPerTestCase = true
    }
    reports.junitXml.destination = "$buildDir/results/B"
}

task testsBCopy(type: Copy) {
    from "$buildDir/results/B"
    into "$buildDir/results"
    rename 'TEST-(.+).xml' , 'B-$1.xml'
}
testsB.finalizedBy testsBCopy

This should work, however typically tests are parallelized via a single task with maxParallelForks set to something greater than 1.

1 Like

Hi Mark,

Based on this discussion i assumed Gradle supports parallel build for single projects too and replicated the same in my project to reduce the build time.

I am not able to run more than one parallel tasks collection in a proper order. Please check like Parallel task execution breaks the order when parallel ,normal tasks mixed together for more detail and let me know why its behaves like this.

The @ParallelizableTask annotation has been removed in Gradle 4.0. I suggest you use the Gradle Worker API if you want to leverage intra-project parallelization.

@mark_vieira, I correctly understand that Worker API allows to parallelize actions of a single task only? What if I want to run several units of work in parallel, but each unit of work has its own inputs and outputs, task dependencies and tasks depending on it? And requires its own up-to-date checking.

I see that with @ParallelizableTask I could just create several tasks. But is it feasible with Worker API?
I have to create subproject for each task right now.

This will work as you describe with the worker API. If I have two tasks which each schedule their own work items, those tasks can be run in parallel, even when belonging to the same project.

1 Like

Hi @mark_vieira
I have two gradle tasks, taskA and taskB. Both tasks can be worked independently. The purpose of each task to to get a jar from mavenRepo and perform some operation using javaexec. However, i saw examples of worker api but it involves injecting worker executor and stuff. I dont want to do that. I have just two simple gradle tasks and i want to run them in parallel to reduce build time. How can i use worker api here ?
Any link/resources would help.

Thanks

1 Like

Unfortunately, the only way to get intra-project parallelism is use the worker api. There is a bit of boilerplate required to get going but it’s pretty simple.

https://guides.gradle.org/using-the-worker-api/

Last I checked, the worker api is very limited since you don’t have access to the Project instance so can’t use files(…) fileTree(…) copy(…) and exec(…) etc

@mark_vieira can you answer this https://stackoverflow.com/questions/57143637/using-gradles-worker-api-to-reduce-build-time-in-gradle-build-task

@Lance If so, then worker api is not very useful then. If we dont have project instance then how are we suppose to do any operations related to project? one of the task i have is to download a JAR form maven repo and use javaexec. if i cannot use copy, exec() how can i leverage WorkerAPI in this case?
@mark_vieira can you help answer this ?

If we dont have project instance then how are we suppose to do any operations related to project?

You can pass read only information from the main thread to the worker thread via WorkerConfiguration.setParams(Object...) and @Inject on the worker class. Keep in mind that workers are running on a separate jvm to the main thread so I think params need to be serializable (so you can’t pass the project instance)

if i cannot use copy, exec() how can i leverage WorkerAPI in this case?

It’s my opinion that Gradle needs to make file operations, javaexec etc available to workers somehow without actually sharing the mutable project with workers. Until then workers only have access to core java APIs so they are only useful in limited scenarios

@parallelizableTask was more useful it seems as i can directly use it over my tasks written in build.gradle where i have the instance for project.This enabled parallelizing task of single project. After Gradle 4.0 or later this was deprecated but introduced worker api doesn’t have this support when project object is required.

Agree with alphaguy4. Have several differing long running tasks in a project that could all run in parallel, however worker API does not support this… The @parallelizableTask seemed ideal.,if apache ant can do this with the parallel task then surely gradle should.