Basically, I have a bunch of JUnit-based Selenium tests, which talk to a web server started via a Gradle shared build service. So far, so good… it’s been working pretty well, pretty reliably for about six months now.
The catch is that it’s slow, because Selenium tests are generally slow, we have a lot of them, and running them sequentially can take half an hour on some of the larger projects. That has me looking at running with maxParallelForks > 1 — but as the docs say, “make sure your tests are properly isolated from one another”. Our tests definitely aren’t safe to run concurrently against the same database, so the “random failure” rate is high.
With a previous incarnation of these tests (homebrew, non-JUnit based), we’ve solved the problem simply by deploying multiple copies of the application in parallel… so one worker thread gets http://localhost:8080/app1, a second worker gets http://localhost:8080/app2, etc. It works pretty well for those older tests, so I’m trying to apply the same concept on top of the Gradle JUnit runner.
What I’m missing is the ability to assign a URL to each test worker. As far as I can see, there’s no way to set worker-specific properties at the Gradle end — I can get the values from the build service, but I can’t provide different values per worker. In the tests themselves, I can read org.gradle.test.worker — but that’s effectively a random value (a thread id?), not something that can be used to derive a URL from a pattern.
Do you have any suggestions on this? A per-worker system property would be ideal, but I’m open to any ideas that will help.
How about not using parallelism on Gradle level, but on JUnit level, assuming you use JUnit 5 which supports parallel test execution and in a better way than on the Gradle level can due to technical reasons.
Then you could for example configure the parallelism to 5, provide 5 instances and then have some way to manage those instances, so that a test can get one of those instances exclusively during its execution.
Yes, that’s a possibility. We currently use JUnit 4 — but now that we’re on Gradle instead of our old Ant build, upgrading just the Selenium tests to JUnit 5 is more feasible than it would have been a year ago. And they’re a separate source set / configuration / test set from the unit tests, so tweaking their configuration to enable parallelism can be done in isolation.
I’ve used JUnit 5 a little in other projects, but I don’t have any experience with its parallel execution. We have more static state in those tests than I’m comfortable with, so do you happen to know what kind of isolation (if any) they provide between concurrently-running tests? From the docs, I’m guessing that the answer is “none” — the only mention of isolation is in the context of forcing some tests into single-threaded mode — does that sound correct to you?
Well, if you switch anyway, switch to Spock, then testing suddenly starts to be fun.
But besides that, I didn’t use JUnit Jupiter parallelism, but as the main logic is in the JUnit Platform it should probably be very similar to the Spock parallelism. So yes, no isolation in typical scenarios this would probably just cost unnecessary overhead. If you use resources from multiple tests, you can assign resources so that those specific tests are not run in parallel, or also mark tests as isolated, so that they run without any other test running.
Well, I’ll take a look at it, try some experiments. The isolation thing shouldn’t be a blocker as such… it’s just a matter of how much technical debt needs to be paid off to make it work. Upgrading JUnit is one of those things I’ve been meaning to do for a while anyway, so even the parallelism doesn’t pan out, it’s not a bad place to start.
Spock, I’ve heard of but never looked closely at. At first glance, it’s probably too much of a leap for my current project, but I’ll add it to my list of reading material…
Ok, I think you’re right that using JUnit parallelism rather than Gradle test parallelism is the way to go. I can mostly automate the upgrade to JUnit5, and I’ve got a working prototype of the approach I’m looking at for untangling statics and allocating URLs to concurrent tests. It’ll take a while to get everything into place — I’m working this as background activity, no budget — but I think it’s going to work well.
For the benefit of anyone else stumbling upon this question, a brief outline of the proposed solution.
A custom Gradle plugin (developed previously) provides a build service which starts a web server containing multiple copies of the application code. That build service also supplies methods for obtaining a set of valid URLs to other parts of the Gradle build.
The relevant Gradle test task is configured to obtain the URLs from the build service, and pass them as Java system properties. This is likewise existing code which needs only to be extended to support multiple URLs - but from memory, the original had to do some nasty tricks to pass dynamic values for system properties. The same code will use that list to configure the JUnit parallelisation parameters to match the number of URLs available.
A custom JUnit extension reads the various system properties, creates a pool from the URLs, allocating them to a test via a before-all handler, injecting them via a property lookup, and returning the URLs to the pool via an after-all handler.
It’s not implemented in its entirety yet, but the first two parts are just extending existing solutions developed for the simpler single-thread case, and a working prototype of the last part only took a couple of hours.
