Not using all test executors when executing JUnit tests filtered by the Category annotation

Help/Discuss
1

I’m trying to get Gradle to run our JUnit/Selenium tests in parallel. It’s been working awesome for a smaller project that we have with ~20 tests. I set maxParallelThreads to 10 (the number of nodes we have available for Selenium), and it will use up to 10 threads to run the tests.

The problem is when I try to filter tests by category using the includeCategories option for useJUnit. This is in a different task that only has 10 tests to run, but there are ~1000 tests in the project.
When running with the -i flag, I can see all 10 workers being started up, but some of them finish almost immedately. I can also see that only 5 to 7 of the Selenium nodes we have are in use at any time.

Here’s the task in my build.gradle:

task production(type: Test) {
    maxParallelForks = 10
    useJUnit{
        includeCategories = ['Production']
    }
}

and here’s what I’m seeing when running gradle -i production using the categories:

16:29:32 Successfully started process 'Gradle Test Executor 10'
16:29:32 Successfully started process 'Gradle Test Executor 9'
16:29:32 Successfully started process 'Gradle Test Executor 1'
16:29:33 Successfully started process 'Gradle Test Executor 2'
16:29:33 Successfully started process 'Gradle Test Executor 6'
16:29:33 Successfully started process 'Gradle Test Executor 3'
16:29:33 Successfully started process 'Gradle Test Executor 7'
16:29:33 Successfully started process 'Gradle Test Executor 4'
16:29:33 Successfully started process 'Gradle Test Executor 8'
16:29:33 Successfully started process 'Gradle Test Executor 5'
16:29:34 Gradle Test Executor 1 started executing tests.
16:29:34 Gradle Test Executor 10 started executing tests.
16:29:34 Gradle Test Executor 9 started executing tests.
16:29:34 Gradle Test Executor 2 started executing tests.
16:29:34 Gradle Test Executor 5 started executing tests.
16:29:34 Gradle Test Executor 3 started executing tests.
16:29:34 Gradle Test Executor 8 started executing tests.
16:29:34 Gradle Test Executor 6 started executing tests.
16:29:34 Gradle Test Executor 7 started executing tests.
16:29:34 Gradle Test Executor 4 started executing tests.
16:29:35 Gradle Test Executor 9 finished executing tests.
16:29:35 Gradle Test Executor 3 finished executing tests.
16:29:35 Gradle Test Executor 5 finished executing tests.
16:29:35 Gradle Test Executor 4 finished executing tests.
16:29:35 Gradle Test Executor 7 finished executing tests.
...
16:41:21 :production (Thread[main,5,main]) completed. Took 11 mins 49.149 secs.

If I change the task to just include the 10 tests by name, the threading works fine and all 10 run simultaneously. Here’s the stdout running gradle production -i with each test explicitly included:

13:16:43 Successfully started process 'Gradle Test Executor 7'
13:16:43 Successfully started process 'Gradle Test Executor 1'
13:16:43 Successfully started process 'Gradle Test Executor 8'
13:16:43 Successfully started process 'Gradle Test Executor 9'
13:16:43 Successfully started process 'Gradle Test Executor 2'
13:16:43 Successfully started process 'Gradle Test Executor 5'
13:16:43 Successfully started process 'Gradle Test Executor 4'
13:16:43 Successfully started process 'Gradle Test Executor 6'
13:16:43 Successfully started process 'Gradle Test Executor 3'
13:16:44 Gradle Test Executor 8 started executing tests.
13:16:44 Gradle Test Executor 5 started executing tests.
13:16:44 Gradle Test Executor 2 started executing tests.
13:16:44 Gradle Test Executor 9 started executing tests.
13:16:44 Gradle Test Executor 4 started executing tests.
13:16:44 Gradle Test Executor 7 started executing tests.
13:16:44 Gradle Test Executor 1 started executing tests.
13:16:44 Gradle Test Executor 3 started executing tests.
13:16:44 Gradle Test Executor 6 started executing tests.
13:17:18 ...
13:20:26 :production (Thread[main,5,main]) completed. Took 3 mins 43.421 secs.

I’ve also tried running with maxParallelForks set to 20, but it still ends up only using 5 to 7 threads for the tests

Filtering by filename isn’t really an option for our large 1000 test project. We want to have multiple categories on each test.

It looks like when the category based including is happening, all the workers load up and run each test class, but then stop immediately if the category is not found. That was all the information I was able to find though. I can’t figure out why they’re finishing when there are still more tests that need to be run.

Is there some option I’m missing, or some way I can get more info about what is actually happening when I run this task?

2

I figured this one out, just going to provide an update in case anyone else has a similar question.

It looks like gradle is batching up the tests ahead of time for the worker threads. I made a change in my task to get some more log info:

testLogging {
    info {
        events "started", "passed", "skipped", "failed"
        displayGranularity 1
        minGranularity 1
    }
}

From this I was able to see all the executors that were finishing early:

jason@link ~/selenium> cat log.txt  | grep "finished executing tests"
Gradle Test Executor 4 finished executing tests.
Gradle Test Executor 18 finished executing tests.
Gradle Test Executor 19 finished executing tests.
Gradle Test Executor 1 finished executing tests.
Gradle Test Executor 7 finished executing tests.
Gradle Test Executor 9 finished executing tests.
Gradle Test Executor 16 finished executing tests.
Gradle Test Executor 10 finished executing tests.
Gradle Test Executor 17 finished executing tests.
Gradle Test Executor 13 finished executing tests.
Gradle Test Executor 15 finished executing tests.
Gradle Test Executor 5 finished executing tests.
Gradle Test Executor 14 finished executing tests.

All of these executors (I had maxParallelThreads = 20 for this run) have executed 51 or 52 tests, which is 1/20 of our total 1021 tests.

jason@link ~/selenium> cat log.txt  | grep -e " 14 >.*STARTED" | wc -l
51

I’m not sure if this is a bug, a feature, bad design, or good design. Either way, that’s what causes this behaviour.

1 Like
3

I too face same issue. Can anyone from Gradle team clarify what is happening?