I’ve created a custom JUnit runner, where each runner has 0 or more children, and has some assertions on it’s own too. Any runner might fail or succeed, independent of it’s descendants or ancestors. A very simplified version is on display https://gist.github.com/astellingwerf/cc70696adf2244165317 When running this within Eclipse (Kepler), in the report it tells me it ran all 16 tests, out of which 8 failed. When running this in Gradle, it happily tells me that all 9 tests out of 9 succeeded. No trace is to be found of the failed tests.
Gradle definitely works with custom JUnit runners in general (we use custom runners all the time). However, it expects that runners adhere closely to JUnit’s ‘RunNotifier’ contract. For example, ‘fireTestStarted’/‘fireTestFailure’/‘fireTestFinished’ must only be called for atomic tests. Also, ‘fireTestStarted’ and ‘fireTestFinished’ must always be called in pairs, no matter if there were errors or not. Chances are that your runner deviates in some of these respects. You might get more insight by running with ‘–debug’.
Thanks Peter, for pointing that out. I didn’t know that I should also call ‘finished’ when the test failed. I corrected that, and now it runs much better. However, Gradle cannot serialize the test results: ‘org.junit.runners.ParentRunner.getDescription()’ creates a ‘SuiteDescription’ based on the result of ‘this.getName()’. That method should return “a name to describe this Runner”. This is a very broad interface: it should simply return a String with some text to describe the runner. But in Gradle, ‘org.gradle.api.internal.tasks.testing.junit.JUnitTestEventAdapter.methodName(Description)’ returns null when the name does not match the pattern ‘"(.)\((.)\)"’, which results in a nullpointer during serialization: https://gist.github.com/astellingwerf/cc70696adf2244165317#file-stacktrace-in-case-of-not-matching-pattern
I also modified the code example in the gist to demonstrate the issue, delete the parentheses on line 23 to have the same failure.
Can you post the full stack trace (seems to be cut off)? If necessary, try ‘–full-stacktrace’. Which Gradle version are you using?
Updated, but doesn’t make too much of a difference I’m afraid. I’m using Gradle 1.6, and also added a gist for Gradle 1.7
There seem to be lines missing after ‘Caused by: java.lang.NullPointerException’.
I know, but they really aren’t there.
Is there any way I can help you to get that the rest of that stacktrace?
If ‘–full-stacktrace’ doesn’t help, then I don’t know what’s going in.
Managed to get it still. I updated the 1.7 stack trace of the Gist.
Doesn’t appear to be a 1.7 stack trace. Line numbers are way off.
Would it be of significance that I’m using the gradle wrapper?
Maybe I’m wrong and master has moved beyond what’s in 1.7. Will have to investigate later.
No, it turned out to be 1.6… I’m creating anew wrapper and running again.
I updated the 1.7 stack trace of the Gist.