TestNG failures not (always) being reported


(Chris Jones) #1

Hooray for Gradle 1.0!

I have a bunch of test classes in my Java project, all using TestNG. When I run Gradle, it builds build/reports/tests/index.html, but that only contains a summary of the last test class to run. So if class #5 out of 10 had a failure, it won’t show up in that summary. Gradle does, however, notice that there were failing tests; and it fails the build as it should. This is how I’m using TestNG:

dependencies {
        testCompile('org.testng:testng:6+')
    }
    test.useTestNG()

(Peter Niederwieser) #2

Do you have ‘test.maxParallelForks’ set, i.e. do you use Gradle’s parallel testing support?


(Chris Jones) #3

Nope. I haven’t changed that setting from the default.


(Chris Jones) #4

I’ve figured out how to reproduce this problem:

test {
    useTestNG()
    forkEvery = 1
}

(Peter Niederwieser) #5

Does it also happen with ‘forkEvery=0’ and ‘maxParallelForks=1’? Otherwise I’m afraid it’s a known limitation. In contrary to the JUnit case, Gradle doesn’t have control over TestNG report generation. This means that if tests are run in multiple JVMs, multiple reports will be generated, possibly overwriting each other.


(Chris Jones) #6

You’re right – that appears to be exactly what’s going on. Thanks for the reply.

Would it be reasonable for Gradle to use JUnit to build reports of TestNG test results? It’s a trick I’ve used with my Ant-based build systems. All the TestNG test reports are written in JUnit format, stored in build/reports/tests/junitreports.


(Peter Niederwieser) #7

Do you have any TestNG suites configured, for example with ‘test.options.suites()’? If ‘test.useTestNG()’ is your whole test configuration (no ‘forkEvery’ etc.), then I don’t understand why you are running into this problem.


(Chris Jones) #8

I’m using forkEvery = 1 for this project. I’m also changing the workingDir, and I have a doFirst that sets up the workingDir.

I suppose I could put in my own doLast to build JUnit reports, but I thought it might be useful for more people than just me.


(Peter Niederwieser) #9

forkEvery=1? Why is that?

I’ve tried to hook up our TestNG integration with Gradle’s JUnit test report, and it seems to work fine. I should be able to get this into one of the next releases.


(Chris Jones) #10

Thanks!

I’m using forkEvery = 1 because of artifacts in the project I’m working on. Many of the test classes depend on test setup being done exactly once. It’s one of those things I’d like to fix…