TestNG dynamically changes test status; not properly reported

Old Forum
1

In TestNG it can be useful to change the test status after its been executed. This used to work moderately OK in Gradle v1.3, but no longer reports correctly in v1.4. It would be reported to Gradle’s build status as the original result, but the reports would show the updated status. That report could be consumed by Jenkins to fail the build, so it was still reported adequately. I’ve used this technique in Maven with a lot of success in the past, which reports the updated status to the console.

There are two use-cases where I’ve found this helpful: 1) Downgrade from a skip to a failure to due a configuration or data provider bugs (causes the dependent tests to be skipped) 2) As a validation aspect (using an after method) to perform common and complex state checking, e.g. walk a concurrent structure for corruption

In Gradle v1.4

<ignored-testcase name="foobar" classname="FooBarTest" time="0.0010"/>

In Gradle v1.3

<testcase name="foobar" time="0.577" classname="FooBarTest">
<failure type="org.testng.TestNGException" message=" Method foobar requires 1 parameters but 0 were supplied in the @Test annotation.">
<![CDATA[
org.testng.TestNGException: Method foobar requires 1 parameters but 0 were supplied in the @Test annotation. at org.testng.internal.Parameters.checkParameterTypes(Parameters.java:198) at org.testng.internal.Parameters.createParameters(Parameters.java:134) at org.testng.internal.Parameters.createParameters(Parameters.java:370) at org.testng.internal.Parameters.handleParameters(Parameters.java:447) at org.testng.internal.Invoker.handleParameters(Invoker.java:1389) at org.testng.internal.Invoker.createParameters(Invoker.java:1081) at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1186) at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127) at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111) at org.testng.TestRunner.privateRun(TestRunner.java:767) at org.testng.TestRunner.run(TestRunner.java:617) at org.testng.SuiteRunner.runTest(SuiteRunner.java:334) at org.testng.SuiteRunner.access$000(SuiteRunner.java:37) at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368) at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662)
]]>
</failure>
</testcase>

public final class ConfigurationSkipAsFailureListener extends TestListenerAdapter {
    @Override
  public void onConfigurationSkip(ITestResult result) {
    result.setStatus(ITestResult.FAILURE);
  }
    @Override
  public void onConfigurationFailure(ITestResult result) {
    result.setStatus(ITestResult.FAILURE);
  }
    @Override
  public void onTestSkipped(ITestResult result) {
    if (result.getThrowable() instanceof TestNGException) {
      result.setStatus(ITestResult.FAILURE);
    }
  }
}
  public final FooBarTest {
  @Test
  public void foobar(Object o) {}
    @DataProvider
  public Object[][] data() {
    assertThat(true, is(false));
    return new Object[][] {{ new Object() }};
  }
}