Gradle reports contain incorrect values for test case times

Old Forum
1

I just upgraded to Gradle 1.4. I was excited to try out the new XML report output for CI as well as the new HTML report output for TestNG tests (big improvement!). But while testing it out I discovered that Gradle is reporting incorrect values for test case execution times when my tests are executed. I’ve tried running just the tests (gradle test) and with cobertura (gradle cobertura). Regardless, both the HTML files and the CI test output (TEST-<classname>) report execution times are all negative and of the incorrect magnitude. And on top of that the execution times differ for the same tests between each type of report.

Here are the times reported for three tests in build/test-results/TEST-com.xyz.MemoryAwareECStreamTemplateTest.xml:

<testsuite name="com.xyz.MemoryAwareECStreamTemplateTest" tests="7" failures="0" errors="0" timestamp="2013-02-06T01:19:52" hostname="myhost" time="0.0">
  <properties/>
  <testcase name="testShouldAddEstimatedObjectSizeToTotalMemoryBeforeInjection" classname="com.xyz.MemoryAwareECStreamTemplateTest" time="-1.360113592028E9"/>
  <testcase name="testShouldBlockWhenTotalMemoryExceedsMaxMemory" classname="com.xyz.MemoryAwareECStreamTemplateTest" time="-1.360113592087E9"/>
  <testcase name="testShouldMarkFutureWithFailureWhenExceptionOccurs" classname="com.xyz.MemoryAwareECStreamTemplateTest" time="-1.360113592412E9"/>
  ...
  </testsuite>

Here are snapshots of the HTML output for the same tests for comparison. Notice how the execution times of the same tests differ from the CI XML:

Oddly enough, the individual test case times reported in build/reports/tests/Gradle suite/Gradle test.xml appear to be correct. Though the total time seems high (Gradle reports the total build time of 29.975 secs on the command-line). Here’s a snippet:

<testsuite name="Gradle test" failures="0" tests="263" time="41.872" errors="0">
  <properties/>
  ...
  <testcase name="testShouldAddEstimatedObjectSizeToTotalMemoryBeforeInjection" time="0.058" classname="com.xyz.MemoryAwareECStreamTemplateTest"/>
  <testcase name="testShouldBlockWhenTotalMemoryExceedsMaxMemory" time="0.324" classname="com.xyz.MemoryAwareECStreamTemplateTest"/>
  <testcase name="testShouldMarkFutureWithFailureWhenExceptionOccurs" time="0.0070" classname="com.xyz.MemoryAwareECStreamTemplateTest"/>
  ...
  </testsuite>

Environment info:

------------------------------------------------------------
 Gradle 1.4
 ------------------------------------------------------------
 Gradle build time: Monday, January 28, 2013 3:42:46 AM UTC
 Groovy: 1.8.6
 Ant: Apache Ant(TM) version 1.8.4 compiled on May 22 2012
 Ivy: 2.2.0
 JVM: 1.6.0_33 (Sun Microsystems Inc. 20.8-b03)
 OS: Linux 3.1.10-gentoo-r1 i386
2

Are you sure you just don’t have really fast tests? :slight_smile:

Is there any chance you could provide a reproducible sample that we can use to verify our fix? That would be very helpful. If you’re willing you can attach it to GRADLE-2672.

3

I think Gradle is off the hook for this issue. While building my reproducible sample I discovered that my project was pulling in an older version of TestNG (5.13.1). So I tried the latest version and it worked as expected. Further investigation revealed that the issue affects TestNG 5.13.x. Versions less than or greater than those work as expected. I’ve attached my sample in case it’s useful in any way.