Can we run tasks in parallel in single build project

(Manoj Shekhawat) #1

I wanted to run task parallely but I’m not able to do so. I read multiple threads and it looks like it’s NOT supported by Gradle at least for single build project.

But I also found GPars and one Gradle hidden property -Dorg.gradle.parallel.intra=true. Tried both but it doesn’t work.

Approach 1:
group 'com.oracle.parallel’
version ‘1.0-SNAPSHOT’

apply plugin: 'java'

sourceCompatibility = 1.8

@ParallelizableTask
class ParallelTask extends Test {}

task testsA(type: ParallelTask) {
    1.upto(100) {
        Thread.sleep(500)
        println "Test HACK ${it}"
    }
}

task testsB(type: ParallelTask) {
    1.upto(100) {
        Thread.sleep(500)
        println "Test hack ${it}"
    }
}

Approach 1 Output:
8:25:36 PM: Executing external tasks ‘testsA testsB --parallel -Dorg.gradle.parallel.intra=true’…
Parallel execution is an incubating feature.
Test HACK 1
Test HACK 2
Test HACK 3

Test HACK 99
Test HACK 100
Test hack 1
Test hack 2
Test hack 3

Test hack 99
Test hack 100
:compileJava UP-TO-DATE
:processResources UP-TO-DATE
:classes UP-TO-DATE
:compileTestJava UP-TO-DATE
:processTestResources UP-TO-DATE
:testClasses UP-TO-DATE
:testsA UP-TO-DATE
:testsB UP-TO-DATE

BUILD SUCCESSFUL

Total time: 1 mins 45.319 secs
8:27:22 PM: External tasks execution finished 'testsA testsB --parallel  -Dorg.gradle.parallel.intra=true'.

Approach 2:
buildscript {
repositories {
mavenCentral()
}
dependencies {
classpath “org.codehaus.gpars:gpars:1.2.1”
}
}

group 'com.oracle.parallel'
version '1.0-SNAPSHOT'

apply plugin: 'java'

sourceCompatibility = 1.8

import java.util.concurrent.*
import groovyx.gpars.GParsExecutorsPool

task testsA {
    1.upto(100) {
        Thread.sleep(500)
        println "Test HACK ${it}"
    }
}

task testsB {
    1.upto(100) {
        Thread.sleep(500)
        println "Test hack ${it}"
    }
}

task parallelTests << {

    def tasksToRun = []
    tasksToRun << 'testsA'
    tasksToRun << 'testsB'

    GParsExecutorsPool.withPool(5) { ExecutorService exService ->
        tasksToRun.eachParallel { taskToRun ->
            exService.submit({ tasks[taskToRun].execute() } as Runnable)
        }
    }
}

Approach 2 Output:
8:31:58 PM: Executing external task ‘parallelTests’…
Test HACK 1
Test HACK 2
Test HACK 3

Test HACK 99
Test HACK 100
Test hack 1
Test hack 2
Test hack 3

Test hack 99
Test hack 100
:parallelTests

BUILD SUCCESSFUL

Total time: 1 mins 49.709 secs
8:33:48 PM: External task execution finished 'parallelTests'.

Can somebody help to run these two tasks (testsA & testsB) parallely, If this is at all possible?

1 Like
Running integration tests in parallel for each test class, with multiple sub-projects
(Mark Vieira) #2

In the first example your issue is that the logic is running at configuration time (not parallelized) instead of execution time. You’ll need to put that logic in a task action (i.e. method annotation with @TaskAction). Be aware also that tasks with custom actions (i.e. doLast or doFirst) will never be run in parallel.

 Custom actions
 
 Any task that has custom actions (i.e. ones added via {@link org.gradle.api.Task#doLast(org.gradle.api.Action)} or {@link org.gradle.api.Task#doFirst(org.gradle.api.Action)})
 is not considered parallelizable even if its type carries this annotation.
 This is because it cannot be known whether the added action is parallel safe or not.
2 Likes
(Manoj Shekhawat) #3

Hi @mark_vieira, Thanks for quick response.

I tried below but still it’s NOT working. It would be really helpful if you could give me a small example that works for you.

task testsA (type: TestTaskA)

class TestTaskA extends DefaultTask {
    @TaskAction
    def test() {
        1.upto(100) {
            Thread.sleep(500)
            println "Test HACK ${it}"
        }
    }
}

task testsB (type: TestTaskB)

class TestTaskB extends DefaultTask {
    @TaskAction
    def test() {
        1.upto(100) {
            Thread.sleep(500)
            println "Test hack ${it}"
        }
    }
} 

Output:
11:32:14 PM: Executing external tasks ‘testsA testsB --parallel -Dorg.gradle.parallel.intra=true’…
Parallel execution is an incubating feature.
:testsA
Test HACK 1
Test HACK 2
Test HACK 3

Test HACK 99
Test HACK 100
:testsB
Test hack 1
Test hack 2
Test hack 3

Test hack 99
Test hack 100

BUILD SUCCESSFUL

Total time: 1 mins 45.933 secs
11:34:00 PM: External tasks execution finished 'testsA testsB --parallel -Dorg.gradle.parallel.intra=true'.
(Mark Vieira) #4

You still need to annotate with @ParallelizableTask.

1 Like
(Manoj Shekhawat) #5

Thanks you soo much @mark_vieira, it works now. This is what I did:
group 'com.oracle.parallel’
version ‘1.0-SNAPSHOT’

apply plugin: 'java'

sourceCompatibility = 1.8

task testsA (type: TestTaskA)

@ParallelizableTask
class TestTaskA extends DefaultTask {
    @TaskAction
    def test() {
        1.upto(100) {
            Thread.sleep(500)
            println "Test HACK ${it}"
        }
    }
}

task testsB (type: TestTaskB)

@ParallelizableTask
class TestTaskB extends DefaultTask {
    @TaskAction
    def test() {
        1.upto(100) {
            Thread.sleep(500)
            println "Test hack ${it}"
        }
    }
}

Output:
10:59:29 AM: Executing external tasks ‘testsA testsB --parallel -Dorg.gradle.parallel.intra=true’…
Parallel execution is an incubating feature.
:testsA
:testsB
Test HACK 1
Test hack 1
Test HACK 2
Test hack 2
Test HACK 3
Test hack 3

Test HACK 99
Test hack 99
Test HACK 100
Test hack 100

BUILD SUCCESSFUL

Total time: 54.656 secs
11:00:24 AM: External tasks execution finished 'testsA testsB --parallel -Dorg.gradle.parallel.intra=true'.
1 Like
(Manoj Shekhawat) #6

Hi @mark_vieira,

Can we apply same parallel concept to Test type tasks? Like I wanted to run testsA & testsB in parallel:

task testsA(type: Test) {
    options.suites("src/test/resources/testng.xml") 
    reports {
        junitXml.outputPerTestCase = true
    }
    reports.junitXml.destination = "$buildDir/results/A"
}

task testsACopy(type: Copy) {
    from "$buildDir/results/A"
    into "$buildDir/results"
    rename 'TEST-(.+).xml' , 'A-$1.xml'
}
testsA.finalizedBy testsACopy

task testsB(type: Test) {
    options.suites("src/test/resources/testng.xml") 
    reports {
        junitXml.outputPerTestCase = true
    }
    reports.junitXml.destination = "$buildDir/results/B"
}

task testsBCopy(type: Copy) {
    from "$buildDir/results/B"
    into "$buildDir/results"
    rename 'TEST-(.+).xml' , 'B-$1.xml'
}
testsB.finalizedBy testsBCopy
(Mark Vieira) #7

This should work, however typically tests are parallelized via a single task with maxParallelForks set to something greater than 1.

1 Like
(Radha Krishnan) #8

Hi Mark,

Based on this discussion i assumed Gradle supports parallel build for single projects too and replicated the same in my project to reduce the build time.

I am not able to run more than one parallel tasks collection in a proper order. Please check like Parallel task execution breaks the order when parallel ,normal tasks mixed together for more detail and let me know why its behaves like this.

(Mark Vieira) #10

The @ParallelizableTask annotation has been removed in Gradle 4.0. I suggest you use the Gradle Worker API if you want to leverage intra-project parallelization.

(Basil Peace) #11

@mark_vieira, I correctly understand that Worker API allows to parallelize actions of a single task only? What if I want to run several units of work in parallel, but each unit of work has its own inputs and outputs, task dependencies and tasks depending on it? And requires its own up-to-date checking.

I see that with @ParallelizableTask I could just create several tasks. But is it feasible with Worker API?
I have to create subproject for each task right now.

(Mark Vieira) #12

This will work as you describe with the worker API. If I have two tasks which each schedule their own work items, those tasks can be run in parallel, even when belonging to the same project.

1 Like