I am seeing something happen when running tests on the Solr project, which uses the gradle wrapper for a build system.
The solr-core part of the build has over four thousand tests. I have noticed that as the tests get close to the end of the sequence that there are a whole bunch of gradle threads that are IDLE.
I suspect what happens is that it takes those four thousand tests, creates queues for each available thread, loads tests into those queues, and then sets it loose to run the tests.
A better option would be to create a single queue and have idle threads pull a test off that queue, so that every thread is busy until there are no more tests in the queue.
I’m a committer on the Solr project, but I didn’t create the build setup. Is there anything we can do in the setup to change how tests are queued, or would that require a change to gradle?
This screenshot shows all the idle threads as the task nears completion. There are only two threads executing tests and 11 idle threads. If all the threads were busy then the tests would complete quite a bit faster.
Iirc this would need a significant change in Gradle itself.
The maxParallelForks you use afair assigns test classes to the threads on a simple round-robin base.
So if you for example have 5 classes where the first 4 have 1 test method and the last has 1000 test methods, the threads will execute 1, 1, 1, 1, and 1000 tests.
Or if one test class needs significant longer as it has slower tests, it would also care for “unequal distribution”.
You might get better results if you would not use the maxParallelForks of Gradle, but the built-in parallel test execution support of the test framework you use if it has one. Spock and Jupiter for example have built-in parallel test execution that would better leverage the available threads as it can work more fine-grained.
