NPE in SimpleMarkupWriter when generating report for [Thucydides/Jbehave/Junit] tests

When running Thucydides/Jbehave/Junit tests Gradle fails with NPE error in SimpleMarkupWriter class.

It looks like the problem is somehow related to binary results of the tests because * With Gradle 1.3 all works fine; * With Gradle 1.4+ report generation fails with NPE;

Here are snippet of stacktrace for Gradle 1.5: Caused by: org.gradle.api.GradleException: Could not write XML test results for pdf.story to file C:\D\work\thucydides\build\test-results\TEST-pdf.story.xml.

at org.gradle.api.internal.tasks.testing.junit.result.Binary2JUnitXmlReportGenerator$1.execute(Binary2JUnitXmlReportGenerator.java:57)

at org.gradle.api.internal.tasks.testing.junit.result.Binary2JUnitXmlReportGenerator$1.execute(Binary2JUnitXmlReportGenerator.java:48)

at org.gradle.api.internal.tasks.testing.junit.result.TestReportDataCollector.visitClasses(TestReportDataCollector.java:95)

at org.gradle.api.internal.tasks.testing.junit.result.Binary2JUnitXmlReportGenerator.generate(Binary2JUnitXmlReportGenerator.java:48)

at org.gradle.api.tasks.testing.Test.executeTests(Test.java:455)

at org.gradle.api.internal.BeanDynamicObject$MetaClassAdapter.invokeMethod(BeanDynamicObject.java:216)

at org.gradle.api.internal.BeanDynamicObject.invokeMethod(BeanDynamicObject.java:122)

at org.gradle.api.internal.CompositeDynamicObject.invokeMethod(CompositeDynamicObject.java:147)

at org.gradle.api.tasks.testing.Test_Decorated.invokeMethod(Unknown Source)

at org.gradle.util.ReflectionUtil.invoke(ReflectionUtil.groovy:23)

at org.gradle.api.internal.project.taskfactory.AnnotationProcessingTaskFactory$4.execute(AnnotationProcessingTaskFactory.java:161)

at org.gradle.api.internal.project.taskfactory.AnnotationProcessingTaskFactory$4.execute(AnnotationProcessingTaskFactory.java:156)

at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:510)

at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:499)

at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:64)

… 60 more Caused by: java.lang.NullPointerException

at org.gradle.api.internal.xml.SimpleMarkupWriter.writeXmlAttributeEncoded(SimpleMarkupWriter.java:372)

at org.gradle.api.internal.xml.SimpleMarkupWriter.attribute(SimpleMarkupWriter.java:241)

at org.gradle.api.internal.tasks.testing.junit.result.JUnitXmlResultWriter.writeTests(JUnitXmlResultWriter.java:83)

at org.gradle.api.internal.tasks.testing.junit.result.JUnitXmlResultWriter.write(JUnitXmlResultWriter.java:60)

at org.gradle.api.internal.tasks.testing.junit.result.Binary2JUnitXmlReportGenerator$1.execute(Binary2JUnitXmlReportGenerator.java:54)

Looks like an unexpected value (e.g. null method name) is reported back to Gradle. Can you provide a reproducible example?

Sure, I have already prepared a simple gradle project. How I can upload zip archive with example?

The forums don’t support uploads. Can you turn it into a GitHub project?

I have created a new GitHub project (https://github.com/soberaionima/gradle-thucydides-integration). Please let me know if you need more information.

Forgot to mention - run the example project from root folder with the following command: gradlew build -s

Thanks for the sample. It’s another one of those cases where a custom JUnit runner intentionally violates the JUnit ‘RunNotifier’ contract to implement hierarchical tests, and Gradle can’t handle it. Specifically, ‘de.codecentric.jbehave.juni.monitoring.JUnitScenarioReporter.beforeScenario’ calls ‘org.junit.runner.notification.RunNotifier.fireTestStarted’ with a ‘Description’ that is non-atomic (that’s the violation) and that returns ‘null’ for ‘getMethodName()’. Gradle successfully persists the ‘null’ into the binary report, but later can’t handle it when generating the XML report.

In other words, JUnit is built around the assumption that there are two levels - test classes and test methods. JBehave tries to turn that into three levels - story, scenario, and step. Gradle’s reporting can’t handle this. It’s also not clear to me how to produce a sensible JUnit XML report for this. It’s a similar problem as Gradle+Cucumber (via JUnit). In order for our generic testing infrastructure to support these tools, we’d probably have to enhance the XML/HTML reporting to handle more than two levels.

Are you even interested in the XML/HTML reports? If not, then we might get away with an option to turn off the XML report. (There’s already an option to turn off the HTML report.)

You are right, I don’t need Gradle/Junit XML/HTML reports for acceptance tests because Thucydides has its own reporting engine which relys on XML file generated by Thucydides . So if there is an option to disable XML report generation, then it would be good. I assume, that in such case I can create additional task of type Test which will run only acceptance tests and in this task I can disable Gradle/Junit XML/HTML reports generation.

As a first step, I’ve raised GRADLE-2765.

In other words, JUnit is built around the assumption that there are two levels - test classes and test methods.

is it? JUnit’s Description is just a node in a tree (as per http://junit.sourceforge.net/javadoc/org/junit/runner/Description.html#addChild(org.junit.runner.Description) with some convenience methods to support junit’s normal usage of a 2 level suite/test. BDD tools commonly use this for 3 levels and other tools that talk in junit terms (e.g. IDEs) understand this.

Gradle relies on the JUnit RunNotifier/RunListener contract which mandates that ‘testStarted’ etc. is only called for atomic tests (see Javadoc). We can probably enhance the HTML report to support multiple levels; not sure what to do about the XML report.

OK yes I see. I had missed the definition of an atomic test as a Description with no children.

Possibly related to

http://forums.gradle.org/gradle/topics/cucumber_test_report_file_formation_is_failing_in_gradle_1_5

and

GRADLE-2739

I’d like to see an option to disable the XML reports. I don’t need them, and this issue is preventing me from migrating from gradle 1.3 to gradle latest.

Inside the Test-task: reports.junitXml.enabled = false … and possibly for good measure: reports.html.enabled = false

@Peter: Now I get my JBehave/Thucydides tests to run, however it does not fail when the tests fail. Do you have any idea of how to accomplish this?

And given the extent of this problem, would it not be a good idea to look into some support for this situation? At this point, it seems like I have to “integrate” Maven to get this stuff crackin’