JUnit Category Filtering, forkEvery, TestRunners

Old Forum Archive Bugs
1

Hi All,

I have what I consider to be a bug, relating to the invocation of JUnit tests.

I am attempting to use the JVM @Catgegory annotation to filter the tests that should be executed by my build. This allows me to annotate a test class with @Category(IntegrationTest.class) and separate those classes into a different build phase from the vanilla JUnit tests.

I have this successfully working with the following syntax:

    test {
      useJUnit {
        excludeCategories "com.acme.IntegrationTest"
      }
    }

So far so good :slight_smile:, I am able to filter the tests that I would like to be able to run. However, it’s about to go very wrong very quickly :frowning2:

I have discovered through remotely debugging the gradle process that this filtering is taking place in the external process that is forked by the ForkingTestClassProcessor. This is the first major problem that I have. I am in a really unfortunate situation where I need a new JVM for each test case, and thus have set forkEvery 1.

From what I have observed, this means that gradle ends up forking a new JVM for every test class (even if it should be filtered out by the excludeCategories setting). Given there is a significant cost of starting up a new JVM, this is really unfortunate. It means that if I have 100 integration tests, 100 JVMs are forked that subsequently exit when they realize there are no tests to be run due to the filter :frowning2:

This is having a DRAMATIC impact on my build times. Many of my integration tests are relatively quick to run, and the startup time for a new JVM far outweighs the amount of time it’d actual test code. As a result my unit-only build takes almost as much time as my unit+integration build.

There’s worse still. I’ve also found that the externally launched JVM is also instantiating the JUnit runner (as specified with the @RunWith annotation) for each test before applying the category filter.

This means that any test runners – including those that unfortunately have expensive logic in their constructors – have their constructors invoked, despite the test subsequently being excluded by the excludeCategories filter.

As a result, my “quick” complile+unit build is actually remarkably slow :frowning2:

I’d be interested to hear if other people agree that this is a bug? I know it’s a pretty niche setup, but it’s causing me real day-to-day problems.

2 Likes
2

A shameless bump. Hoping this topic won’t remain overlooked.