How to extend JUnit support?


(Noel Yap) #1

We have a custom Ant Task that runs JUnit 4 tests such that: * tests can be run in parallel * tests can be filtered by ‘@Category’ annotations

I understand that parallel test execution will be built into Gradle. What’s the ETA for this?

Currently, we can run the tests with, say, ‘-Dtest.categories.excludes=’/Manual.class,/DevMachine.class’ -Dtest.categories.includes=’/Small.class,/Flaky.class’’ such that only tests categorized as Small and Flaky but not Manual or DevMachine will be run. The categories can be applied on test classes and test methods. What’s the best way to add the filtering mechanism to Gradle?


(Noel Yap) #2

It looks like hooks will need to be added into JUnitTestClassExecuter.runTestClass(String testClassName) such that we can use a custom Runner and/or Request.


(Peter Niederwieser) #3

Gradle is able to run tests in parallel since long ago; you just need to set ‘test.maxParallelForks’. It doesn’t currently support filtering by ‘@Category’, although that would be useful to add. Would you be interested in contributing such a feature?


(Noel Yap) #4

Yes, I’m definitely willing to contribute. I’ve been looking through the source already, but I’m not sure where exactly to put the hook. Eventually, JUnitTestClassExecuter.runTestClass will need to be able to call a RequestFactory. What’s the best way to set the RequestFactory through the layers of JUnitTestFramework and JUnitTestClassProcessor?

On second thought, a RequestFactory may not be needed. If a filter isn’t set, just default it to filter out @Ignore’d tests. That’ll make things work like normal.

Lemme code up something and send it in for code review.


(Peter Niederwieser) #5

What ‘RequestFactory’ are you referring to?

The basic workflow is like this:

  • Gradle JVM scans candidate class files (with ASM) to find JUnit test classes. * Gradle JVM dispatches requests to run test classes to Gradle test JVM(s). * Gradle test JVM(s) execute test classes using ‘JunitTestClassExecuter’ and report back results (start/end events, std out/err, exceptions, etc.).

I haven’t looked deeply into this, but probably at least the class filtering should happen in the Gradle JVM, for example while scanning for test classes (which is done with ASM, not reflection). Filtering could take ‘@Category’ and/or arbitrary user-defined annotations into account. The method filtering could happen on either side.

Given that this is not a trivial change, we’d probably have to collaborate on a spec (see ‘/design-docs’ for examples), with implementation and tests to follow. This will take some effort, but then it also tackles a popular feature request, as indicated by several open JIRA issues. If you are still interested in tackling this, let me know.


(Noel Yap) #6

RequestFactory doesn’t exist; I was speaking in the abstract.

The current JunitTestClassExecuter code looks like:

private void runTestClass(String testClassName) throws ClassNotFoundException {
        Class<?> testClass = Class.forName(testClassName, true, applicationClassLoader);
        Runner runner = Request.aClass(testClass).getRunner();
        RunNotifier notifier = new RunNotifier();
        notifier.addListener(listener);
        runner.run(notifier);
    }

it would have to be changed to something like:

private void runTestClass(String testClassName) throws ClassNotFoundException {
        Runner runner = requestFactory.createRequest().getRunner();
        RunNotifier notifier = new RunNotifier();
        notifier.addListener(listener);
        runner.run(notifier);
    }

My current code is:

public class FilteredRequestFactory {
    private final ClassLoader classLoader;
    private final String[] filters;
    private final String[] methods;
      public FilteredRequestFactory(final ClassLoader classLoader, final String[] filters, final String[] methods) {
        this.classLoader = classLoader;
        this.filters = filters;
        this.methods = methods;
    }
      public Request createRequest(final String testName) throws Exception {
        final Class<?> testClass = getClass(testName);
          final Filter filter = createFilter();
        final RunnerBuilder runnerBuilder = new ConfigurableRunnerBuilder(ImmutableList.of(
                new FilteredBuilder(filter),
                new AllDefaultPossibilitiesBuilder(true)));
          final Runner runner = runnerBuilder.safeRunnerForClass(testClass);
          return Request.runner(runner).filterWith(filter);
    }
      private Filter createFilter() throws Exception {
        Filter result = (methods == null)
                ? new PassThroughFilter()
                : new MethodFilter(methods);
          try {
            for (String filter : filters) {
                final Class<?> filterClass = getClass(filter);
                final Filter filterInstance = (Filter) filterClass.getConstructor().newInstance();
                  result = result.intersect(filterInstance);
            }
        } catch (ClassNotFoundException e) {
            throw new FilterNotFoundException(e.getMessage());
        }
          return result;
    }
      private Class<?> getClass(final String className) throws ClassNotFoundException {
        return (classLoader == null)
                ? Class.forName(className)
                : Class.forName(className, true, classLoader);
    }
      public class FilterNotFoundException extends ClassNotFoundException {
        public FilterNotFoundException(final String message) {
            super(message);
        }
    }
}

I’m also trying to work with the JUnit project to see which, if any, of the code belongs in that project (eg specific Filter and Runner implementations).

One thing that’s important to us is that Gradle not perform the filtering itself. Rather, it ought to delegate to the support already provided by JUnit. This is important because we report on tests that are skipped (either due to Category values or to @Ignore attributes – we handle @Ignore using JUnit’s Filters, too).

Anyway, yes, I’m still very much interested in working on this. Please let me know how we can proceed.


(Noel Yap) #7

It looks like JUnitOptions might be a good place to put the RequestFactory:

public class JUnitOptions extends TestFrameworkOptions {
    private RequestFactory requestFactory = new ClassRequestFactory();
      public RequestFactory requestFactory() {
        return requestFactory;
    }
      public JUnitOptions withRequestFactory(final RequestFactory requestFactory) {
        this.requestFactory = requestFactory;
          return this;
    }
}

where:

public class ClassRequestFactory implements RequestFactory {
    @Override
    public Request createRequest(final ClassLoader classLoader, final String testName) throws ClassNotFoundException {
        Class<?> testClass = Class.forName(testName, true, classLoader);
          return Request.aClass(testClass);
    }
}

But how can that be set from within a ‘build.gradle’ file? Would we do something like:

test {
  useJUnit
  getOptions.withRequestFactory(new FilteredRequestFactory())
}

(Peter Niederwieser) #8

I’d start out by only making the categories/annotations to include/exclude configurable. For this I’d probably add two ‘Set<String>’ properties to ‘JUnitOptions’, and enhance the remoting protocol to send that information across. (Have a look at ‘TestNGOptions’/‘TestNGSpec’ and their usages.) As mentioned before, class filtering would probably have to be done earlier, based on scanning the .class files. Method filtering can only be done by/near ‘JunitTestClassExecuter’.


(Jon Austen) #9

I see how that sets the max but how do you actually execute multiple threads? I am very interested in how I can run unit tests threaded (without needing to use testNG or surefire). The closest example I could find was this: https://gist.github.com/djangofan/4946958 .

Can anyone point me to where there is more info on this or fork my Gist and show me how it should work? Otherwise, I will follow this thread and hope for more hints.


(Peter Niederwieser) #10

You just set ‘test.maxParallelForks’, and Gradle does the rest. It won’t execute a single test in multiple threads, but it will run multiple test classes in parallel (in separate JVMs).


(Jon Austen) #11

Oh, I see what you mean. So, all I need to do is separate out tests into their separate classes and run with a suite task? I think I get what you mean.

I also found another intriguing link: https://gist.github.com/djangofan/4947053

. I will be experimenting around with this stuff. Sounds like it could take me a month or more to figure it all out.


(Peter Niederwieser) #12

To repeat, you just need to set ‘test.maxParallelForks’ (to a value greater than ‘1’). Of course it only helps if you have more than one test class.