There is test run duplication when using a TestNG suite XML file with the maxParallelForks gradle test setting

Sud_Ramasamy · June 28, 2014, 4:48pm

There is a bug in either gradle or TestNG when using a TestNG suite XML file and the maxParallelForks option where if you have a second TestNG test class the tests get executed twice. The sample project is located here - https://github.com/sudr/gradle-testng-fork-bug.

Here are the steps to reproduce the issue.

Create a simple project with one TestNG class with a testng.xml file with the TestNG test class in the suite. See here for an example:

github.com

sudr/gradle-testng-fork-bug/blob/master/src/test/resources/testng.xml

<?xml version="1.0" encoding="UTF-8"?>
<suite name="Suite1">
  <test name="test" >
    <classes>
       <class name="org.gradle.PersonTest" />
    </classes>
  </test>
</suite>

github.com

sudr/gradle-testng-fork-bug/blob/master/src/test/java/org/gradle/PersonTest.java

package org.gradle;

import java.lang.management.ManagementFactory;

import org.testng.Assert;
import org.testng.annotations.Test;

public class PersonTest {
    
    @Test
    public void canConstructAPersonWithAName() throws InterruptedException {
        System.out.println("person started: " +pid());
        Person person = new Person("Larry");
        Assert.assertEquals("Larry", person.getName());
        Thread.sleep(2000);
        System.out.println("person passed: " + pid());
    }
    
    String pid() {
        return ManagementFactory.getRuntimeMXBean().getName();

This file has been truncated. show original

Create a build.gradle that uses TestNG and points to the TestNG suite XML file and contains the maxParallelForks option. See here for an example:

github.com

sudr/gradle-testng-fork-bug/blob/master/build.gradle

apply plugin: 'java'
apply plugin: 'eclipse'

repositories {
    mavenCentral()
}

dependencies {
    testCompile group: 'org.testng', name: 'testng', version:'6.8.8'
}

test {
    useTestNG() {
        suites 'src/test/resources/testng.xml'
    }
    
    maxParallelForks=5
    
    onOutput { descriptor, event ->
        logger.lifecycle(event.message )

This file has been truncated. show original

Run the gradle build and you should see that single test (PersonTest) be forked and execute.
Now add a second TestNG class to the project. There is no need to add it to the TestNG suite XML. The mere fact that there is another TestNG class in the path triggers the bug.

https://github.com/sudr/gradle-testng-fork-bug/blob/master/src/test/java/org/gradle/AnimalTest.java (you’ll need to uncomment the @Test annotation)

Now re-run the build and you will see that the PersonTest is now executed twice.