I believe the time displayed is a sum of all of the test times - at least that’s how it appears for me running multiple Selenium nodes in parallel. I just go by the time on the Jenkins build log. I actually like the way that the time is reported in the Gradle report as it gives me a baseline to determine how much time I’m saving (and the diminishing return when adding more) by running multiple threads. But yeah, I agree it should probably at least specify how the time is calculated as it does throw you off a bit.
Very interesting. Thanks for that post. Glad I didn’t have to discover the problem on my own. Can surefire do this via the TestNG plugin?
Just thought I’d mention that this had been addressed in the latest release candidate
Hey,
I just tried 1.11-rc-1 and can confirm skipped tests are now taken into account in the html test report. Thanks !
It would be awesome to have a “Skipped tests” tab to browse them. We have thousands of tests in a big multimodule build, this would be very useful to track down skipped tests.
Cheers
/Paul
@Paul, would you be interested in contributing this feature?
Sure why not. Point me into the right direction (how to run a modified gradle version and offending files/modules) and I’ll try to come up with something.
Great. There some info on development here: https://github.com/gradle/gradle/blob/master/README.md
The DefaultTestReport class is the place to start.
After experimenting with this I must say that when using TestNG features in Gradle, you would have to be crazy to not prefer to use the TestNG report over the Gradle report (this means not using maxParallelForks in Gradle) because with the TestNG report you can add a listener that takes a ITestResult object and then you can basically customize the report to look however you want (with a little difficulty), which includes presenting skipped tests in whatever fashion you like. I am experimenting with this in this project (which isn’t a gradle project but it outlines the idea): https://github.com/djangofan/selenium-framework-concept2
Thanks, I’ll start this next week. Cheers
Great, if you get stuck at all then just holler on the dev mailing list or on this thread.
My pull-request is ready for review: https://github.com/gradle/gradle/pull/248 Things went pretty well. Tell me what you think. I sent the signed CLA already. Cheers
I agree that the TestNG report is nicer but the way the framework is built that I’m currently using does not play well when running it in parallel via TestNG as it was not built with this in mind when it was first started. Now it’s a very large framework and without each test class in its own JVM there are multithread-related issues galore. Sure I could go rewrite everything but that’s way more effort than it’s worth for a report.
In a perfect world the TestNG reporting system would be able to aggregate several reports into one as a post-build step or something. You can also still do cool stuff with custom listeners even when forking from the Gradle core (nabbing screenshots, dumping stack traces into logs, etc). The only catch is similar to the TestNG report problem where the only [truly] safe way to capture any of those things are in different files due to possible timing conflicts.