NPE in SimpleMarkupWriter when generating report for [Thucydides/Jbehave/Junit] tests


(Dzmitry Katsiuba) #1

When running Thucydides/Jbehave/Junit tests Gradle fails with NPE error in SimpleMarkupWriter class.

It looks like the problem is somehow related to binary results of the tests because * With Gradle 1.3 all works fine; * With Gradle 1.4+ report generation fails with NPE;

Here are snippet of stacktrace for Gradle 1.5: Caused by: org.gradle.api.GradleException: Could not write XML test results for pdf.story to file C:\D\work\thucydides\build\test-results\TEST-pdf.story.xml.

at org.gradle.api.internal.tasks.testing.junit.result.Binary2JUnitXmlReportGenerator$1.execute(Binary2JUnitXmlReportGenerator.java:57)

at org.gradle.api.internal.tasks.testing.junit.result.Binary2JUnitXmlReportGenerator$1.execute(Binary2JUnitXmlReportGenerator.java:48)

at org.gradle.api.internal.tasks.testing.junit.result.TestReportDataCollector.visitClasses(TestReportDataCollector.java:95)

at org.gradle.api.internal.tasks.testing.junit.result.Binary2JUnitXmlReportGenerator.generate(Binary2JUnitXmlReportGenerator.java:48)

at org.gradle.api.tasks.testing.Test.executeTests(Test.java:455)

at org.gradle.api.internal.BeanDynamicObject$MetaClassAdapter.invokeMethod(BeanDynamicObject.java:216)

at org.gradle.api.internal.BeanDynamicObject.invokeMethod(BeanDynamicObject.java:122)

at org.gradle.api.internal.CompositeDynamicObject.invokeMethod(CompositeDynamicObject.java:147)

at org.gradle.api.tasks.testing.Test_Decorated.invokeMethod(Unknown Source)

at org.gradle.util.ReflectionUtil.invoke(ReflectionUtil.groovy:23)

at org.gradle.api.internal.project.taskfactory.AnnotationProcessingTaskFactory$4.execute(AnnotationProcessingTaskFactory.java:161)

at org.gradle.api.internal.project.taskfactory.AnnotationProcessingTaskFactory$4.execute(AnnotationProcessingTaskFactory.java:156)

at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:510)

at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:499)

at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:64)

… 60 more Caused by: java.lang.NullPointerException

at org.gradle.api.internal.xml.SimpleMarkupWriter.writeXmlAttributeEncoded(SimpleMarkupWriter.java:372)

at org.gradle.api.internal.xml.SimpleMarkupWriter.attribute(SimpleMarkupWriter.java:241)

at org.gradle.api.internal.tasks.testing.junit.result.JUnitXmlResultWriter.writeTests(JUnitXmlResultWriter.java:83)

at org.gradle.api.internal.tasks.testing.junit.result.JUnitXmlResultWriter.write(JUnitXmlResultWriter.java:60)

at org.gradle.api.internal.tasks.testing.junit.result.Binary2JUnitXmlReportGenerator$1.execute(Binary2JUnitXmlReportGenerator.java:54)


(Peter Niederwieser) #2

Looks like an unexpected value (e.g. null method name) is reported back to Gradle. Can you provide a reproducible example?


(Dzmitry Katsiuba) #3

Sure, I have already prepared a simple gradle project. How I can upload zip archive with example?


(Peter Niederwieser) #4

The forums don’t support uploads. Can you turn it into a GitHub project?


(Dzmitry Katsiuba) #5

I have created a new GitHub project (https://github.com/soberaionima/gradle-thucydides-integration). Please let me know if you need more information.


(Dzmitry Katsiuba) #6

Forgot to mention - run the example project from root folder with the following command: gradlew build -s


(Peter Niederwieser) #7

Thanks for the sample. It’s another one of those cases where a custom JUnit runner intentionally violates the JUnit ‘RunNotifier’ contract to implement hierarchical tests, and Gradle can’t handle it. Specifically, ‘de.codecentric.jbehave.juni.monitoring.JUnitScenarioReporter.beforeScenario’ calls ‘org.junit.runner.notification.RunNotifier.fireTestStarted’ with a ‘Description’ that is non-atomic (that’s the violation) and that returns ‘null’ for ‘getMethodName()’. Gradle successfully persists the ‘null’ into the binary report, but later can’t handle it when generating the XML report.

In other words, JUnit is built around the assumption that there are two levels - test classes and test methods. JBehave tries to turn that into three levels - story, scenario, and step. Gradle’s reporting can’t handle this. It’s also not clear to me how to produce a sensible JUnit XML report for this. It’s a similar problem as Gradle+Cucumber (via JUnit). In order for our generic testing infrastructure to support these tools, we’d probably have to enhance the XML/HTML reporting to handle more than two levels.

Are you even interested in the XML/HTML reports? If not, then we might get away with an option to turn off the XML report. (There’s already an option to turn off the HTML report.)


(Dzmitry Katsiuba) #8

You are right, I don’t need Gradle/Junit XML/HTML reports for acceptance tests because Thucydides has its own reporting engine which relys on XML file generated by Thucydides . So if there is an option to disable XML report generation, then it would be good. I assume, that in such case I can create additional task of type Test which will run only acceptance tests and in this task I can disable Gradle/Junit XML/HTML reports generation.


(Peter Niederwieser) #9

As a first step, I’ve raised GRADLE-2765.


(Matt Khan) #10

In other words, JUnit is built around the assumption that there are two levels - test classes and test methods.

is it? JUnit’s Description is just a node in a tree (as per http://junit.sourceforge.net/javadoc/org/junit/runner/Description.html#addChild(org.junit.runner.Description) with some convenience methods to support junit’s normal usage of a 2 level suite/test. BDD tools commonly use this for 3 levels and other tools that talk in junit terms (e.g. IDEs) understand this.


(Peter Niederwieser) #11

Gradle relies on the JUnit RunNotifier/RunListener contract which mandates that ‘testStarted’ etc. is only called for atomic tests (see Javadoc). We can probably enhance the HTML report to support multiple levels; not sure what to do about the XML report.


(Matt Khan) #12

OK yes I see. I had missed the definition of an atomic test as a Description with no children.


(Mark Petrovic) #13

Possibly related to

http://forums.gradle.org/gradle/topics/cucumber_test_report_file_formation_is_failing_in_gradle_1_5

and

GRADLE-2739


(Mark Petrovic) #14

I’d like to see an option to disable the XML reports. I don’t need them, and this issue is preventing me from migrating from gradle 1.3 to gradle latest.


(Endre Stølsvik) #15

Inside the Test-task: reports.junitXml.enabled = false … and possibly for good measure: reports.html.enabled = false


(Endre Stølsvik) #16

@Peter: Now I get my JBehave/Thucydides tests to run, however it does not fail when the tests fail. Do you have any idea of how to accomplish this?

And given the extent of this problem, would it not be a good idea to look into some support for this situation? At this point, it seems like I have to “integrate” Maven to get this stuff crackin’