The plugin can be used to print beautiful logs on the console while tests are being executed. Typically the console output would look something like (colours stripped for simplicity):
com.example.test.FirstSpec Test abc PASSED Test def SKIPPED Test ghi PASSED com.example.test.SecondSpec Test jkl PASSED Test mno SKIPPED Test pqr PASSED
As you can see, the tests are grouped by suites. This is all fine and dandy as long as
maxParallelForks is set to
1 (Gradle default). But when more than one fork is executing the tests, grouping isn’t deterministic anymore as the tests are executed in parallel and the output can become mixed up. So we may see
FirstSpec executing tests
def followed by
mno followed by
FirstSpec executing tests
ghi followed by
pqr. Classic problem with parallel processing and grouping.
com.example.test.FirstSpec Test abc PASSED Test def SKIPPED com.example.test.SecondSpec Test jkl PASSED Test mno SKIPPED com.example.test.FirstSpec Test ghi PASSED com.example.test.SecondSpec Test pqr PASSED
I find that an output of the above form makes the grouping useless. To resolve this problem, I can think of two options but both have their limitations.
Trap all output in a collection (a map keyed by suite name) and render after executing the last test. While this would resolve the grouping issue, visually it’s not very elegant as no output would appear on the console until the last test is executed and suddenly a large volume of output may be produced. Looks like this approach is used by mocha-parallel-tests.
Use a different theme (the plugin has a concept of themes which changes the style of the output) that doesn’t involve grouping when
maxParallelForks > 1. For instance the new theme could render the output as follows:
com.example.test.FirstSpec: Test abc PASSED com.example.test.FirstSpec: Test def SKIPPED com.example.test.SecondSpec: Test jkl PASSED com.example.test.SecondSpec: Test mno SKIPPED com.example.test.FirstSpec: Test ghi PASSED com.example.test.SecondSpec: Test pqr PASSED
The downsides of this approach are that there may be too much repetition of the suite name thereby reducing the readability of the output and that the plugin will have to have separate themes for parallel and sequential modes.
I would like to get an opinion of the Gradle community regarding this problem and would also welcome any other suggestions. For example, if there is a way to get more information during the execution of forked tests in the Gradle API, that might prove invaluable too.
Thank you and appreciate your time.