TestNG results overwritten when running in parallel

Old Forum
1

Hello,

I have run into a problem when running TestNG tests with maxparallelforks > 1

The tests I am running can be found here

https://github.com/iainrose/vanq-java

If I run the tests with maxParallelForks = 1, I see the results from all tests in my results. This is good.

However if I run the tests with maxParallelForks = 5, I only see the results of what I’m guessing is the last forked thread to complete.

If I run the tests enough times, I can get them to run in a different order which I assume causes a different test to complete last and overwrite the other results.

I can see that all tests are running as for both cases, the console output is …

$ ./gradlew test
:compileJava
:processResources UP-TO-DATE
:classes
:compileTestJava
:processTestResources UP-TO-DATE
:testClasses
:test
Running test: test method april262012PresenterTest(org.vanq.tests.MeetingsPageTests)
Running test: test method april262012PresenterTest(org.vanq.tests.MeetingsPageTest)
Running test: test method clickHomeTabTest(org.vanq.tests.NavigationBarTest)
Running test: test method clickMeetingsTabTest(org.vanq.tests.NavigationBarTest)
Running test: test method isLogoDisplayedTest(org.vanq.tests.HeaderTest)
Running test: test method clickMissionTabTest(org.vanq.tests.NavigationBarTest)
Running test: test method clickSponsorTabTest(org.vanq.tests.NavigationBarTest)
  BUILD SUCCESSFUL
  Total time: 29.392 secs

I’m using Gradle 1.0 RC3 & TestNG 6.3.1

Am I doing something wrong?

2

This may be a limitation of our TestNG support. For TestNG we leverage TestNG’s own reports, which probably means that a separate report is generated per fork.

An alternative to ‘maxParallelForks’ is to use TestNG’s own parallel testing support, in which case there should be just one report created.

3

Thanks again for the quick reply Peter.

I’ve done some work to try and isolate the problem. I ripped out the ReportNG listeners but get the same results.

My updated build.gradle is here https://github.com/iainrose/vanq-java/blob/master/build.gradle

I did find one interesting caveat though which is that with maxParallelForks > 1, TestNG is creating jUnit reports properly and I have a separate TEST*.xml for each test class in my parallel run. I just don’t have an aggregated report I can use to view my results.

The html report, emailable html report and testng-results.xml output all only contain the results from the last test class to run.

I’ve also tried removing the maxParallelForks setting and using the following TestNGOptions but the tests are only running in one thread. I do get a single report that contains all the results though.

tasks.withType(Test) {
    useTestNG {
        parallel="tests"
        threadCount=5
    }
}

Are either / both of these bugs? Would you like me to log them?

4

TestNG creates its reports directly, not from the XML results. That’s why it doesn’t help that the XML results don’t overwrite each other. The same holds for ReportNG.

Looking at our code, I found that ‘parallel’ and ‘threadCount’ aren’t passed on to TestNG. This definitely looks like a bug, and I’ll fix it for 1.1. Meanwhile, you might be able to set these properties from a suite (see ‘suites()’ and ‘suiteXmlBuilder()’).

5

I’ll give suiteXmlBuilder a try and let you know if I find a usable workaround.

I think that the first issue is a fairly serious bug, any chance you might consider that one for the 1.1 release too? I think there should be a way to create aggregated TestNG reports when using maxParallelForks > 1

In the meantime, thanks again for the support. Much appreciated

6

Doesn’t look like the suiteXmlBuilder approach is going to work either

Here’s the snippet from build.gradle

tasks.withType(Test) {
    useTestNG() {
        suiteXmlBuilder().suite(name: 'testing-testng', parallel: 'classes', 'thread-count': '5') {
            test (name : 'testing-testng', annotations : 'JDK', verbose:'1') {
                packages {
                    'package' (name: 'org.vanq.tests')
                }
            }
        }
    }
}

This creates an xml file that looks correct however the tests still only get run in a single thread

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name='testing-testng' parallel='classes' thread-count='5'>
  <test name='testing-testng' annotations='JDK' verbose='1'>
    <packages>
      <package name='org.vanq.tests' />
    </packages>
  </test>
</suite>

http://testng.org/doc/documentation-main.html#parallel-tests

7

Just to add my two cents, I tried to help Iain configure this using TestNGOptions specified to the useTestNG() block, but looking at the implementation of TestNGTestClassProcessor where it appears to be consumed, there’s no mention made of the ‘parallel’ or ‘threadCount’ configuration parameters when creating the TestNG() object. I might be looking in the wrong place, but I am referring to the TestNGTestClassProcessor.stop() method on the present master branch. I would expect to see something similar to this within that method(if it is indeed the correct place), but there’s nothing there:

if(options.getParallel() != null) {
            testNg.setParallel(options.getParallel());
        }
        if(options.getThreadCount() > 0){
            testNg.setThreadCount(options.getThreadCount());
        }
8

Yeah, that’s exactly what I found (and said).

9

Sorry - missed this, didn’t refresh the page :frowning:

10

Going back to the original issue, I think that the reports only containing the test results from one class is a downstream effect of separate testng.xml files being created and run for each class.

Example:

The testng.xml file for this looks like …

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite verbose="0" name="Gradle suite">
  <test name="Gradle test" preserve-order="false">
    <classes>
      <class name="org.vanq.tests.MeetingsPageTest"/>
    </classes>
  </test>
</suite>

I assume that others were created / overwritten for each class too.

That is probably the root cause of the issue although I can’t think what the solution would be, especially as when the correct (to my eyes) TestNG.xml file is created (as above with suiteXmlBuilder example) the tests still only run in series.

11

There should be one TestNG invocation (and therefore report) per fork. How often a new JVM is forked depends on the values of ‘forkEvery’ and ‘maxParallelForks’. What we could do is to replace TestNG’s HTML report with our own, like we do for JUnit. This would allow us to produce just a single report.

When you tried with your own parallel suite, did you make sure that ‘forkEvery=0’ and ‘maxParallelForks=1’? These are the defaults, but you might have overwritten them earlier. Did you try with enough test classes, and how did you verify that they were all run in the same thread?

12

“There should be one TestNG invocation (and therefore report) per fork” – that looks to be exactly what is happening.

That solution you described sounds like a good idea. If you need any help testing it out when you have something ready I’d be happy to assist.

To give you some more context about how I ran my tests … when I used suiteXmlBuilder, I added a 20s static pause to one method in each test class then watched the number of browser sessions that got opened (by viewing the number of firefox instances in my dock) and compared the run time.

I had 4 test classes in this test, I never changed the forkEvery setting from default, maxParallelForks was configured as below.

Test 1:

Setup:

tasks.withType(Test) {
    useTestNG{
        suiteXmlBuilder().suite(name: 'testing-testng', parallel: 'classes', 'thread-count': '5') {
            test (name : 'testing-testng', verbose:'1') {
                packages {
                    'package' (name: 'org.vanq.tests')
                }
            }
        }
    }
    maxParallelForks = 1
}

Results: 1 browser session Total time: 1 mins 50.597 secs Reports contains results from all test classes

Test 2: Setup:

tasks.withType(Test) {
    useTestNG()
    maxParallelForks = 1

Results: 1 browser session Total time: 1 mins 50.743 secs Reports contain results from all test classes

Test 3: Setup:

tasks.withType(Test) {
    useTestNG()
    maxParallelForks = 5
}

Results: Can see 4 browser sessions open at once Total time: 49.219 secs Reports contain results from 1 test class only

13

Has there been any progress on this issue? I’m experiencing the same problem. I’d like to be able to use the TestNG reporting if at all possible as it is much more clear than generating a report from the JUnit XML “pile” post-build.

14

Unfortunately there has not been any progress. Would you be interested in contributing a fix?

15

Thanks for the update, Luke. I’m quite busy at this time with work but I’d be more than happy to dig around and see if I can isolate and fix the issue when time allows.

Cheers,

Darwin

16

Note that Gradle’s own test report is by now supported for TestNG as well. Hence the problem should only occur if you wish to generate the original TestNG report.