Skipped tests in html test report


(Marek Gerhart) #1

Html test report generated by gradle is really poor. For example - there are no skipped tests shown and passed tests are also not listed (only classes but not the tests themselves). I’m using gradle 1.6 with testng 6.8.1. Is there some plan to improve reports?


(Attila Kelemen) #2

I don’t know if the report for TestNG is different or not but for JUnit, tests (including ignored) are listed. You just need to click on the test class name.


(Marek Gerhart) #3

It doesn’t work with TestNG, please check build.gradle: https://gist.github.com/mawek/6000643 and test class: https://gist.github.com/mawek/6000649

Full source code: https://www.dropbox.com/s/km64wr5gmjv2xfx/gradle_logging_testng.zip

Generated html report contains no information about skipped tests.


(Attila Kelemen) #4

The example project you have linked, seems to generate a good html report for me (even showing ignored). Just like for JUnit.

Also, I have checked that throwing ‘SkipException’ will be reported as ignored in the html report as well. However ‘@Test(enabled=false)’ is not shown in the html report.


(Marek Gerhart) #5

Maybe we see something different:) This is what I get when I run ‘gradle test’:

Am I doing something wrong?


(Attila Kelemen) #6

The classname in “Classes” (i.e., ‘com.mawek.test.Test01’) is a link to more detailed information. There you should see test methods of that particular class.


(Marek Gerhart) #7

Oh I can see that now. It’s really burried down :slight_smile: There is no way I can see that on main(index) page? I’d like to see on main page something like ‘2 tests completed, 1 failed, 1 skipped’.

The thing is that I’m trying to show this report on jenkins job main page, and it’s not clear that some test were skipped at all. I have to dig really deep to discover skipped tests.


(Attila Kelemen) #8

Probably it would be nice if the number of skipped tests would be listed just like failed and succeeded.

I think, what you can do know is to manually parse the generated xml and count the ignored tests in a Gradle task.

EDIT: Although the number of skipped tests can be calculated from the “Tests” and “Failures” column.


(Marek Gerhart) #9

Thanks for your time Attila. Can I somehow create feature request in gradle jira for this? I’d really like to see this on main page.

Calculating skipped tests does work but it’s not visible at the first sight, and parsing generated xml files or hooking into TestCountLogger seems like temporary hack to me.


(Attila Kelemen) #10

As far as I know, you can no longer open jira tickets yourself. Instead, you should post RFE as an idea on the forum.


(Marek Gerhart) #11

Ok thanks. By the way, I just found out that reportng reports do contain skipped tests in main page (but you can’t use it with ‘maxParallelForks’ switch in test task because it causes reports to overwrite themselves)

dependencies {

testCompile “org.testng:testng:6.8.1”

testCompile(“org.uncommons:reportng:1.1.2”) {

exclude module: ‘testng’

}

}

test {

useTestNG() {

useDefaultListeners = true // generate html report by TestNG

}

testReport = false // don’t generate test report by gradle

}


(Luke Daley) #12

So you would like number of skipped tests on the front page?


(Luke Daley) #13

TestNG’s HTML report stuff is incompatible with maxParallelForks. This is something we can’t fix.


(Marek Gerhart) #14

Yes, that would be cool.

Also percentage successful is kind of misleading. If I have 10 tests, one of them passes, one fails and 8 (dependent) are skipped, I get 90% success rate.

I understand that skipped tests are not necessary failures, but when I see 90% success rate, I expect that most of the tests passed OK which is not really true :slight_smile:


(Darwin Allen) #15

I, too, am in this same situation and because TestNG does not play well with parallel testing, I’m forced to use Gradle’s built-in reporting. Not having the skipped tests is tough to digest. I’d love to see this feature implemented.

Any plans to do so?


(Luke Daley) #16

Raised as GRADLE-2976.


(Darwin Allen) #17

Awesome, I’m looking forward to this when you get a chance. Both me and my team will be quite pleased if and when this makes it in.


(Jean-Baptiste Nizet) #18

I just submitted a pull request to implement this improvement: https://github.com/gradle/gradle/pull/230


(Perryn Fowler) #19

GRADLE-2976 has been resolved with jnizet’s pull request. It should be available from 1.11


(dinio.dinev) #20

And If I can add done more thing for the report. If you follow this example http://mrhaki.blogspot.com/2010/11/gradle-goodness-running-tests-in.html For parallel thread using and start it with 13 threads fore example one of the execution for my machine was:

BUILD SUCCESSFUL
  Total time: 13.463 secs

But in the report was said the total time only

341 tests 0 failures 1m8.91s duration

Duration of sum time is over 1 minute, but not the whole execution. So must be also added information about the real time spend for tests run. Or I am wrong?