We have a custom Ant Task that runs JUnit 4 tests such that: * tests can be run in parallel * tests can be filtered by ‘@Category’ annotations
I understand that parallel test execution will be built into Gradle. What’s the ETA for this?
Currently, we can run the tests with, say, ‘-Dtest.categories.excludes=’/Manual.class,/DevMachine.class’ -Dtest.categories.includes=’/Small.class,/Flaky.class’’ such that only tests categorized as Small and Flaky but not Manual or DevMachine will be run. The categories can be applied on test classes and test methods. What’s the best way to add the filtering mechanism to Gradle?
It looks like hooks will need to be added into JUnitTestClassExecuter.runTestClass(String testClassName) such that we can use a custom Runner and/or Request.
Gradle is able to run tests in parallel since long ago; you just need to set ‘test.maxParallelForks’. It doesn’t currently support filtering by ‘@Category’, although that would be useful to add. Would you be interested in contributing such a feature?
Yes, I’m definitely willing to contribute. I’ve been looking through the source already, but I’m not sure where exactly to put the hook. Eventually, JUnitTestClassExecuter.runTestClass will need to be able to call a RequestFactory. What’s the best way to set the RequestFactory through the layers of JUnitTestFramework and JUnitTestClassProcessor?
On second thought, a RequestFactory may not be needed. If a filter isn’t set, just default it to filter out @Ignore’d tests. That’ll make things work like normal.
Lemme code up something and send it in for code review.
Gradle JVM scans candidate class files (with ASM) to find JUnit test classes. * Gradle JVM dispatches requests to run test classes to Gradle test JVM(s). * Gradle test JVM(s) execute test classes using ‘JunitTestClassExecuter’ and report back results (start/end events, std out/err, exceptions, etc.).
I haven’t looked deeply into this, but probably at least the class filtering should happen in the Gradle JVM, for example while scanning for test classes (which is done with ASM, not reflection). Filtering could take ‘@Category’ and/or arbitrary user-defined annotations into account. The method filtering could happen on either side.
Given that this is not a trivial change, we’d probably have to collaborate on a spec (see ‘/design-docs’ for examples), with implementation and tests to follow. This will take some effort, but then it also tackles a popular feature request, as indicated by several open JIRA issues. If you are still interested in tackling this, let me know.
public class FilteredRequestFactory {
private final ClassLoader classLoader;
private final String[] filters;
private final String[] methods;
public FilteredRequestFactory(final ClassLoader classLoader, final String[] filters, final String[] methods) {
this.classLoader = classLoader;
this.filters = filters;
this.methods = methods;
}
public Request createRequest(final String testName) throws Exception {
final Class<?> testClass = getClass(testName);
final Filter filter = createFilter();
final RunnerBuilder runnerBuilder = new ConfigurableRunnerBuilder(ImmutableList.of(
new FilteredBuilder(filter),
new AllDefaultPossibilitiesBuilder(true)));
final Runner runner = runnerBuilder.safeRunnerForClass(testClass);
return Request.runner(runner).filterWith(filter);
}
private Filter createFilter() throws Exception {
Filter result = (methods == null)
? new PassThroughFilter()
: new MethodFilter(methods);
try {
for (String filter : filters) {
final Class<?> filterClass = getClass(filter);
final Filter filterInstance = (Filter) filterClass.getConstructor().newInstance();
result = result.intersect(filterInstance);
}
} catch (ClassNotFoundException e) {
throw new FilterNotFoundException(e.getMessage());
}
return result;
}
private Class<?> getClass(final String className) throws ClassNotFoundException {
return (classLoader == null)
? Class.forName(className)
: Class.forName(className, true, classLoader);
}
public class FilterNotFoundException extends ClassNotFoundException {
public FilterNotFoundException(final String message) {
super(message);
}
}
}
I’m also trying to work with the JUnit project to see which, if any, of the code belongs in that project (eg specific Filter and Runner implementations).
One thing that’s important to us is that Gradle not perform the filtering itself. Rather, it ought to delegate to the support already provided by JUnit. This is important because we report on tests that are skipped (either due to Category values or to @Ignore attributes – we handle @Ignore using JUnit’s Filters, too).
Anyway, yes, I’m still very much interested in working on this. Please let me know how we can proceed.
I’d start out by only making the categories/annotations to include/exclude configurable. For this I’d probably add two ‘Set<String>’ properties to ‘JUnitOptions’, and enhance the remoting protocol to send that information across. (Have a look at ‘TestNGOptions’/‘TestNGSpec’ and their usages.) As mentioned before, class filtering would probably have to be done earlier, based on scanning the .class files. Method filtering can only be done by/near ‘JunitTestClassExecuter’.
I see how that sets the max but how do you actually execute multiple threads? I am very interested in how I can run unit tests threaded (without needing to use testNG or surefire). The closest example I could find was this: https://gist.github.com/djangofan/4946958 .
Can anyone point me to where there is more info on this or fork my Gist and show me how it should work? Otherwise, I will follow this thread and hope for more hints.
You just set ‘test.maxParallelForks’, and Gradle does the rest. It won’t execute a single test in multiple threads, but it will run multiple test classes in parallel (in separate JVMs).
Oh, I see what you mean. So, all I need to do is separate out tests into their separate classes and run with a suite task? I think I get what you mean.