TestNG failures not (always) being reported

Old Forum
1

Hooray for Gradle 1.0!

I have a bunch of test classes in my Java project, all using TestNG. When I run Gradle, it builds build/reports/tests/index.html, but that only contains a summary of the last test class to run. So if class #5 out of 10 had a failure, it won’t show up in that summary. Gradle does, however, notice that there were failing tests; and it fails the build as it should. This is how I’m using TestNG:

dependencies {
        testCompile('org.testng:testng:6+')
    }
    test.useTestNG()
2

Do you have ‘test.maxParallelForks’ set, i.e. do you use Gradle’s parallel testing support?

3

Nope. I haven’t changed that setting from the default.

4

I’ve figured out how to reproduce this problem:

test {
    useTestNG()
    forkEvery = 1
}
5

Does it also happen with ‘forkEvery=0’ and ‘maxParallelForks=1’? Otherwise I’m afraid it’s a known limitation. In contrary to the JUnit case, Gradle doesn’t have control over TestNG report generation. This means that if tests are run in multiple JVMs, multiple reports will be generated, possibly overwriting each other.

6

You’re right – that appears to be exactly what’s going on. Thanks for the reply.

Would it be reasonable for Gradle to use JUnit to build reports of TestNG test results? It’s a trick I’ve used with my Ant-based build systems. All the TestNG test reports are written in JUnit format, stored in build/reports/tests/junitreports.

7

Do you have any TestNG suites configured, for example with ‘test.options.suites()’? If ‘test.useTestNG()’ is your whole test configuration (no ‘forkEvery’ etc.), then I don’t understand why you are running into this problem.

8

I’m using forkEvery = 1 for this project. I’m also changing the workingDir, and I have a doFirst that sets up the workingDir.

I suppose I could put in my own doLast to build JUnit reports, but I thought it might be useful for more people than just me.

9

forkEvery=1? Why is that?

I’ve tried to hook up our TestNG integration with Gradle’s JUnit test report, and it seems to work fine. I should be able to get this into one of the next releases.

1 Like
10

Thanks!

I’m using forkEvery = 1 because of artifacts in the project I’m working on. Many of the test classes depend on test setup being done exactly once. It’s one of those things I’d like to fix…