With custom JUnit runner, failures are ignored

(Anne Stellingwerf) #1

I’ve created a custom JUnit runner, where each runner has 0 or more children, and has some assertions on it’s own too. Any runner might fail or succeed, independent of it’s descendants or ancestors. A very simplified version is on display https://gist.github.com/astellingwerf/cc70696adf2244165317 When running this within Eclipse (Kepler), in the report it tells me it ran all 16 tests, out of which 8 failed. When running this in Gradle, it happily tells me that all 9 tests out of 9 succeeded. No trace is to be found of the failed tests.

(Peter Niederwieser) #2

Gradle definitely works with custom JUnit runners in general (we use custom runners all the time). However, it expects that runners adhere closely to JUnit’s ‘RunNotifier’ contract. For example, ‘fireTestStarted’/‘fireTestFailure’/‘fireTestFinished’ must only be called for atomic tests. Also, ‘fireTestStarted’ and ‘fireTestFinished’ must always be called in pairs, no matter if there were errors or not. Chances are that your runner deviates in some of these respects. You might get more insight by running with ‘–debug’.

(Anne Stellingwerf) #3

Thanks Peter, for pointing that out. I didn’t know that I should also call ‘finished’ when the test failed. I corrected that, and now it runs much better. However, Gradle cannot serialize the test results: ‘org.junit.runners.ParentRunner.getDescription()’ creates a ‘SuiteDescription’ based on the result of ‘this.getName()’. That method should return “a name to describe this Runner”. This is a very broad interface: it should simply return a String with some text to describe the runner. But in Gradle, ‘org.gradle.api.internal.tasks.testing.junit.JUnitTestEventAdapter.methodName(Description)’ returns null when the name does not match the pattern ‘"(.)\((.)\)"’, which results in a nullpointer during serialization: https://gist.github.com/astellingwerf/cc70696adf2244165317#file-stacktrace-in-case-of-not-matching-pattern

I also modified the code example in the gist to demonstrate the issue, delete the parentheses on line 23 to have the same failure.

(Peter Niederwieser) #4

Can you post the full stack trace (seems to be cut off)? If necessary, try ‘–full-stacktrace’. Which Gradle version are you using?

(Anne Stellingwerf) #5

Updated, but doesn’t make too much of a difference I’m afraid. I’m using Gradle 1.6, and also added a gist for Gradle 1.7

(Peter Niederwieser) #6

There seem to be lines missing after ‘Caused by: java.lang.NullPointerException’.

(Anne Stellingwerf) #7

I know, but they really aren’t there.

(Anne Stellingwerf) #8

Is there any way I can help you to get that the rest of that stacktrace?

(Peter Niederwieser) #9

If ‘–full-stacktrace’ doesn’t help, then I don’t know what’s going in.

(Anne Stellingwerf) #10

Managed to get it still. I updated the 1.7 stack trace of the Gist.

(Peter Niederwieser) #11

Doesn’t appear to be a 1.7 stack trace. Line numbers are way off.

(Anne Stellingwerf) #12

Would it be of significance that I’m using the gradle wrapper?

(Peter Niederwieser) #13

Maybe I’m wrong and master has moved beyond what’s in 1.7. Will have to investigate later.

(Anne Stellingwerf) #14

No, it turned out to be 1.6… I’m creating anew wrapper and running again.

(Anne Stellingwerf) #15

I updated the 1.7 stack trace of the Gist.