Gradle JUnit Test Result Output


(Ahsan Rabbani) #1

Is there an easy way to customize the JUnit test output so that it does not output ignored-testcase elements? Our CI server (Bamboo 2.7) does not seem to be able to parse any files with this element and as a result all the other test results in the file are lost.


(Peter Niederwieser) #2

Are you talking about the XML output? That’s a de-facto standard (first introduced by Ant), and as far as I know, Gradle doesn’t offer any hooks to customize it. Do you think Gradle’s output is wrong? Have you submitted an issue for Bamboo?


(Ahsan Rabbani) #3

Yes, I’m referring to the XML output. The weird thing is when I run the tests using our Ant scripts the test output files don’t have those elements. But if it’s a de-facto standard then I guess I’ll submit an issue for Bamboo.

Thanks


(Peter Niederwieser) #4

Can you post an example (Ant vs. Gradle)? Please use HTML code tags. Are there actually any test methods being ignored?


(Ahsan Rabbani) #5

Yes, there are tests that are annotated with @Ignore. Below is a snippet of the XML output from Ant and Gradle. The only difference really is that Ant doesn’t seem to include the ignored test in the output.

Ant

<testcase classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" name="billInvoice_givenFMAClientWithFinancialAccountPrimary_andBillingSucceeds" time="0.162" />
  <testcase classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" name="billInvoice_givenFMAClientWithFinancialAccountPrimary_andTaxes_andBillingSucceeds" time="0.066" />
  <testcase classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" name="billInvoice_givenFMAClientWithFinancialAccountPrimary_andInsufficientFunds" time="0.053" />
  <testcase classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" name="billInvoice_givenPrepayClientWithFinancialAccountPrimary_andBillingSucceeds" time="0.048" />
  <testcase classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" name="billInvoice_givenPrepayClientWithFinancialAccountPrimary_andInsufficientFunds" time="0.054" />
  <testcase classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" name="billInvoice_givenPrepayClientSomehowWithFinancialAccountPrimaryFromOtherBillable_andPermissionDenied" time="0.046" />
  <testcase classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" name="billInvoice_givenOEMClientWithFinancialAccountPrimary_andBillingSucceeds" time="0.048" />
  <testcase classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" name="billInvoice_givenOEMClientWithFinancialAccountPrimary_andInsufficientFunds" time="0.047" />
  <testcase classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" name="billInvoice_givenFMAClientWithFinancialAccountPrimary_andPreviouslyInsufficientFunds_andBillingSucceeds" time="0.057" />
  <testcase classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" name="billInvoice_givenNonFMAClient_andSomehowWithFMAFinancialAccountPrimary_PermissionDenied" time="0.052" />
  <testcase classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" name="billInvoice_givenNonOEMClient_andSomehowWithOEMFinancialAccountPrimary_PermissionDenied" time="0.049" />
  <testcase classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" name="billInvoice_givenFMAClientWithNoPrimaryPaymentMethod_andBillingFails" time="0.042" />
  <testcase classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" name="billInvoice_givenFMAClientWithNonPrimaryPaymentMethodOnInvoice_andBillingSucceeds" time="0.047" />
  <testcase classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" name="billInvoice_givenFMAClientWithFinancialAccountPrimary_withUpsellInvoice_andBillingSucceeds" time="0.055" />

Gradle

<testcase name="billInvoice_givenFMAClientWithFinancialAccountPrimary_andBillingSucceeds" classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" time="0.375"/>
  <testcase name="billInvoice_givenFMAClientWithFinancialAccountPrimary_andTaxes_andBillingSucceeds" classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" time="0.075"/>
  <testcase name="billInvoice_givenFMAClientWithFinancialAccountPrimary_andInsufficientFunds" classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" time="0.08"/>
  <testcase name="billInvoice_givenPrepayClientWithFinancialAccountPrimary_andBillingSucceeds" classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" time="0.062"/>
  <testcase name="billInvoice_givenPrepayClientWithFinancialAccountPrimary_andInsufficientFunds" classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" time="0.052"/>
  <testcase name="billInvoice_givenPrepayClientSomehowWithFinancialAccountPrimaryFromOtherBillable_andPermissionDenied" classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" time="0.088"/>
  <ignored-testcase name="billInvoice_givenDisplayFMAClientWithFinancialAccountPrimary_andBillingSucceeds" classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" time="0.0"/>
  <testcase name="billInvoice_givenOEMClientWithFinancialAccountPrimary_andBillingSucceeds" classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" time="0.048"/>
  <testcase name="billInvoice_givenOEMClientWithFinancialAccountPrimary_andInsufficientFunds" classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" time="0.047"/>
  <testcase name="billInvoice_givenFMAClientWithFinancialAccountPrimary_andPreviouslyInsufficientFunds_andBillingSucceeds" classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" time="0.052"/>
  <testcase name="billInvoice_givenNonFMAClient_andSomehowWithFMAFinancialAccountPrimary_PermissionDenied" classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" time="0.053"/>
  <testcase name="billInvoice_givenNonOEMClient_andSomehowWithOEMFinancialAccountPrimary_PermissionDenied" classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" time="0.077"/>
  <testcase name="billInvoice_givenFMAClientWithNoPrimaryPaymentMethod_andBillingFails" classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" time="0.047"/>
  <testcase name="billInvoice_givenFMAClientWithNonPrimaryPaymentMethodOnInvoice_andBillingSucceeds" classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" time="0.057"/>
  <testcase name="billInvoice_givenFMAClientWithFinancialAccountPrimary_withUpsellInvoice_andBillingSucceeds" classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" time="0.054"/>

(Ahsan Rabbani) #6

FYI, the issue does not occur anymore after updating to a newer version of Bamboo (4.4) so they must have fixed it somewhere in between.


(Peter Niederwieser) #7

Thanks for letting us know!


(Greg) #8

Just FYI it’s filed here: http://issues.gradle.org/browse/GRADLE-2731 Does Bamboo show those tests as skipped? AFAIK no tools support these “ignored-testcase” tags, Bamboo should be just ignoring them.