How to build a custom TestFramework

Help/Discuss
1

Hi,
I’m trying to build a custom TestFramework, which basically does the same as the default JUnitTestFramework, but additionally adds a JUnit RunListener.

Why?
For my masters thesis I want to record test coverage with JaCoCo on a per-test-method basis. To do so I need to get notified synchronously when the execution of a test method starts and finishes. Gradle’s TestListener unfortunately is called asynchronously (AFAIK), so I cannot use this approach.
I also tried to use a custom runner implementation and this approach works, but its a real pain to annotate every test class with @RunWith annotations and additionally wrap all already existing custom runners.

So I was looking for a less invasive way to setup a listener similar to Maven’s listener configuration. I guess gradle won’t support this by default for the foreseeable future.

How?
From my understanding it should somehow be possible to create a custom TestFramework that does set a listener before executing the tests, so that it could be configured like this:

test {
    useListeningJUnit("com.test.MyRunListener")
}

I started by copying and renaming JUnitTestFramework class to ListeningJUnitTestFramework with the following configuration:

test {
    useTestFramework(new ListeningJUnitTestFramework(it, it.filter))
}

But when executing the test task I get the following exception immediately after the test JVM is started:

org.gradle.internal.UncheckedException: java.lang.ClassNotFoundException: org.gradle.api.internal.tasks.testing.junit.ListeningJUnitTestFramework$TestClassProcessorFactoryImpl
        at org.gradle.internal.UncheckedException.throwAsUncheckedException(UncheckedException.java:63)
        at org.gradle.internal.UncheckedException.throwAsUncheckedException(UncheckedException.java:40)
        at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:101)
        at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:60)
        at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:62)
        at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:67)
Caused by: java.lang.ClassNotFoundException: org.gradle.api.internal.tasks.testing.junit.ListeningJUnitTestFramework$TestClassProcessorFactoryImpl
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:677)
        at org.gradle.internal.io.ClassLoaderObjectInputStream.resolveClass(ClassLoaderObjectInputStream.java:40)
        at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1826)
        at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
        at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:99)
        ... 3 more

To me this seems like gradle is serializing the TestClassProcessorFactoryImpl returned by ListeningJUnitTestFramework#getProcessorFactory() and passes it somehow to the test JVM, but then fails to deserialize it, because it’s missing on the classpath. But I couldn’t figure out how to get it there.

I would be very thankful for any answer on how to resolve this error or on the approach in general!

Thanks in advance

2

There is no way to provide a custom test framework at this point.

Junit 5 supports loading run listeners from the classpath automatically, so you might just want to give it a try instead.

3

Thank you!
Then I’ll experiment with JUnit 5 :wink: