The very initial background was, that once in a while a nasty bug found its way into either production code or JUnit tests leading to a hanging gradle test and thus hanging gradle build. I must admit that we were quite surprised that there did not seem to be a way to specify test timeouts in gradle. However, in case this occurs just sporadically – especially if it is more a kind of JUnit based integration test – just a termination would have not been satisfying either, as additional means like a thread dump might have been needed to fix the bug in any case.
Recently I therefore started to post some ideas/thoughts on the JUnit project:
- JUnit 772 Optional global timeout via system property
Ideas behind this initial commit for discussion/review on global timeouts are: > (1) I want to make sure that the set of unit tests really only consists of unit tests, i.e. they need to be very fast (to be executable in each incremental build), so I would like to set a global timeout (certainly not on each single test class or test case) of e.g. 50ms (in fact it should be of course even smaller, but one should scare all devs off right away…). > (2) To avoid very long builds or even infinitely stuck builds due to very slow or even infinitely stuck tests by just setting a tolerant maximum timeout: again I only want to do that once (globally) for all the tests in a build. > > Note that different global timeout strategy variants can be useful: > (a) should the global timeout conditionally override specific test timeouts that are higher then the global timeout? to make sure a maximum test duration is not exceeded; also allows to test that something really must be very quick > (b) should a specific timeout always override the global timeout? e.g. to avoid the need to increase the global timeout, just because a single test takes longer > …
– kcooney commented on that: > … > We’ve generally not liked approaches where globals (like system properties) affect the behavior of tests. The concern have been 1) it isn’t clear looking at the test code that there is a timeout (or other global behavior) being applied, and 2) the behavior of the test would be different in an IDE than in a build (unless, of course, your IDE users specify the system property). >
Note that we recently added support in JUnitCore for parsing command line flags (currently used for filtering tests, but could be used for other things). So if having global settings for timeouts made sense, using command line flags would be more consistent than using system properties. > …
— I responded: > … > While I see and share your concerns, I must admit that my position is kind of pragmatic and paranoid at the same time: (a) I want to avoid that devs need to consider anything that should not bother them (like having to use a special test runner, or adding timeouts) for normal plain unit tests, (b) especially if this is very easily just forgotten and would at least only pop up very late, if at all (e.g. if being detected by manual checks), or would require redundant/duplicate dedicated additional automation checks, (c) of course local integration at a devs machine must always mean (incremental) building of the project with the same build system (infrastructure) than on the build server, and that of course includes the execution of all unit tests. >
After spending a little bit of time investigating the sources of our build system (gradle), I came to the conclusion that there is just no (elegant and/or maintainable and/or non-duplicating) way to using command line flags (yet). To be honest I would also not know how to nicely implement the multiple handing over of these timeouts through the “layers” of the JUnit system (core, request, runner) so that each runner is supporting it. > … > Maybe the following provides an additional thought on why I am thinking of this approach and I still think this is a pragmatic one for the real world: Let’s say we are not starting a greenfield project and so there is quite some code base (including tests based on JUnit: please note that I don’t call them unit tests, as they IMHO consist of real unit tests, but also of kind of big/slow/… integration tests) already there. So at first we should not be too harsh to the devs by setting a too stringent timeout per test case, as otherwise they get frustrated or even unmotivated to write any unit tests at all anmyore. There is also hope that due to getting rid or refactor or improve these big/slow/… non-unit tests, the devs actually are getting happier, because their local building and integration (including of course running all the unit tests) is significantly faster and makes them more productive. And so they actually see the benefit of real unit tests. When this has happened, one would hope that decreasing the timeout for unit tests again (to converge to acceptable real unit test time needs of a few milliseconds, so maybe this should be a several decrease step procedure) is even embraced and welcome by the devs… >
Of course the introduction of big/slow/… non-unit integration tests can and typically really will also always happen in real world projects, I think.
- JUnit 770 Full thread stack dump on timeout with custom handler
So – since I am not familiar at all with the gradle source and especially its coding with respect to JUnit execution – my questions are:
-
Have I overlooked something completely?
-
Or are there already any plans in trying to use this JUnit (v4.12?) core parsing of args (which may or may not address the complaining comments in the gradle JUnit support code)? (I must repeat that even then “I would also not know how to nicely implement the multiple handing over of these timeouts through the “layers” of the JUnit system (core, request, runner) so that each runner is supporting it”.)
Thanks, Reinhold