Unit tests seem to run multiple times when using TestNG with 'suites' XML and maxParallelForks > 1

Old Forum
1

As mentioned in the title, I’m having issues running unit tests in Gradle using TestNG when maxParallelForks is > 1 if specifying tests in an XML file. This happens with both Gradle and Gradlew, version 1.9, 1.10 and 1.12 (I haven’t tested other versions).

It’s really a weird problem to explain with text so let’s go straight to the examples: Example code here : https://gist.github.com/FredDeschenes/4663772327926e9b0928

  • Create an empty project directory - ‘gradle init --type java-library’ - Modify your build.gradle file to use TestNG (see gist) - Port ‘LibraryTest.java’ to TestNG (see gist) - Create 2 other test classes (see ‘OtherTest’ and ‘OtherOtherTest’ from gist, but any other class with @Test or a method with @Test works). - ‘gradle(w) test’ - Open ./build/reports/tests/index.html

Since only “LibraryTest.java” is declared in “testng.xml”, only this test is ran, but the output looks like this :

Now delete ‘OtherOtherTest.java’ and run ‘gradle(w) clean test’, output now looks like this :

Now delete ‘OtherTest.java’ and run ‘gradle(w) clean test’ again, output now looks like this :

Setting “scanForTestClasses = false” in build.gradle doesn’t change anything. It also doesn’t look like it’s simply a display/report error since the tests actually take more time to run.

This might also very well be a bug in TestNG, but I have no idea where to start looking for it!

Thanks for your help!

EDIT: Added image I forgot to link. EDIT2: Tested with Gradle 1.12, same issue.

2

Not all TestNG and Gradle features can be freely combined, and I don’t think that TestNG suites and Gradle parallel execution are compatible. You may want to look into using TestNG’s parallel execution features rather than Gradle’s here.

3

Thanks for the quick reply!

Is there a list somewhere of which TestNG features might not work properly in Gradle? Should I expect the same kind of issues with JUnit?

4

There is no such list. All limitations I’m aware of are related to combining certain test framework features with Gradle test parallelization. To my knowledge, the suites feature does work properly. It just can’t be parallelized by Gradle, since that would require Gradle to be in control of which tests to execute. Same holds for JUnit suites, although there I think Gradle’s behavior is more appropriate in that it doesn’t even try to parallelize execution of the suite.

5

So basically if we want to fork the tests, we shouldn’t even have any suites and just let Gradle find the test classes?

6

Correct. Alternatively, you can use TestNG’s own parallelization features, although this will mean running all tests in the same (test) JVM.

7

Alright thanks for your help!