Is there an easy way to customize the JUnit test output so that it does not output ignored-testcase elements? Our CI server (Bamboo 2.7) does not seem to be able to parse any files with this element and as a result all the other test results in the file are lost.
Are you talking about the XML output? That’s a de-facto standard (first introduced by Ant), and as far as I know, Gradle doesn’t offer any hooks to customize it. Do you think Gradle’s output is wrong? Have you submitted an issue for Bamboo?
Yes, I’m referring to the XML output. The weird thing is when I run the tests using our Ant scripts the test output files don’t have those elements. But if it’s a de-facto standard then I guess I’ll submit an issue for Bamboo.
Thanks
Can you post an example (Ant vs. Gradle)? Please use HTML code tags. Are there actually any test methods being ignored?
Yes, there are tests that are annotated with @Ignore. Below is a snippet of the XML output from Ant and Gradle. The only difference really is that Ant doesn’t seem to include the ignored test in the output.
Ant
<testcase classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" name="billInvoice_givenFMAClientWithFinancialAccountPrimary_andBillingSucceeds" time="0.162" />
<testcase classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" name="billInvoice_givenFMAClientWithFinancialAccountPrimary_andTaxes_andBillingSucceeds" time="0.066" />
<testcase classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" name="billInvoice_givenFMAClientWithFinancialAccountPrimary_andInsufficientFunds" time="0.053" />
<testcase classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" name="billInvoice_givenPrepayClientWithFinancialAccountPrimary_andBillingSucceeds" time="0.048" />
<testcase classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" name="billInvoice_givenPrepayClientWithFinancialAccountPrimary_andInsufficientFunds" time="0.054" />
<testcase classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" name="billInvoice_givenPrepayClientSomehowWithFinancialAccountPrimaryFromOtherBillable_andPermissionDenied" time="0.046" />
<testcase classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" name="billInvoice_givenOEMClientWithFinancialAccountPrimary_andBillingSucceeds" time="0.048" />
<testcase classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" name="billInvoice_givenOEMClientWithFinancialAccountPrimary_andInsufficientFunds" time="0.047" />
<testcase classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" name="billInvoice_givenFMAClientWithFinancialAccountPrimary_andPreviouslyInsufficientFunds_andBillingSucceeds" time="0.057" />
<testcase classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" name="billInvoice_givenNonFMAClient_andSomehowWithFMAFinancialAccountPrimary_PermissionDenied" time="0.052" />
<testcase classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" name="billInvoice_givenNonOEMClient_andSomehowWithOEMFinancialAccountPrimary_PermissionDenied" time="0.049" />
<testcase classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" name="billInvoice_givenFMAClientWithNoPrimaryPaymentMethod_andBillingFails" time="0.042" />
<testcase classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" name="billInvoice_givenFMAClientWithNonPrimaryPaymentMethodOnInvoice_andBillingSucceeds" time="0.047" />
<testcase classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" name="billInvoice_givenFMAClientWithFinancialAccountPrimary_withUpsellInvoice_andBillingSucceeds" time="0.055" />
Gradle
<testcase name="billInvoice_givenFMAClientWithFinancialAccountPrimary_andBillingSucceeds" classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" time="0.375"/>
<testcase name="billInvoice_givenFMAClientWithFinancialAccountPrimary_andTaxes_andBillingSucceeds" classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" time="0.075"/>
<testcase name="billInvoice_givenFMAClientWithFinancialAccountPrimary_andInsufficientFunds" classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" time="0.08"/>
<testcase name="billInvoice_givenPrepayClientWithFinancialAccountPrimary_andBillingSucceeds" classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" time="0.062"/>
<testcase name="billInvoice_givenPrepayClientWithFinancialAccountPrimary_andInsufficientFunds" classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" time="0.052"/>
<testcase name="billInvoice_givenPrepayClientSomehowWithFinancialAccountPrimaryFromOtherBillable_andPermissionDenied" classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" time="0.088"/>
<ignored-testcase name="billInvoice_givenDisplayFMAClientWithFinancialAccountPrimary_andBillingSucceeds" classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" time="0.0"/>
<testcase name="billInvoice_givenOEMClientWithFinancialAccountPrimary_andBillingSucceeds" classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" time="0.048"/>
<testcase name="billInvoice_givenOEMClientWithFinancialAccountPrimary_andInsufficientFunds" classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" time="0.047"/>
<testcase name="billInvoice_givenFMAClientWithFinancialAccountPrimary_andPreviouslyInsufficientFunds_andBillingSucceeds" classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" time="0.052"/>
<testcase name="billInvoice_givenNonFMAClient_andSomehowWithFMAFinancialAccountPrimary_PermissionDenied" classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" time="0.053"/>
<testcase name="billInvoice_givenNonOEMClient_andSomehowWithOEMFinancialAccountPrimary_PermissionDenied" classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" time="0.077"/>
<testcase name="billInvoice_givenFMAClientWithNoPrimaryPaymentMethod_andBillingFails" classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" time="0.047"/>
<testcase name="billInvoice_givenFMAClientWithNonPrimaryPaymentMethodOnInvoice_andBillingSucceeds" classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" time="0.057"/>
<testcase name="billInvoice_givenFMAClientWithFinancialAccountPrimary_withUpsellInvoice_andBillingSucceeds" classname="com.yodle.payment.invoice.processor.pinning.PaymentProcessing_Client_FinancialAccountPM_PinningTests" time="0.054"/>
FYI, the issue does not occur anymore after updating to a newer version of Bamboo (4.4) so they must have fixed it somewhere in between.
Thanks for letting us know!
Just FYI it’s filed here: http://issues.gradle.org/browse/GRADLE-2731 Does Bamboo show those tests as skipped? AFAIK no tools support these “ignored-testcase” tags, Bamboo should be just ignoring them.