Unit testing: (re)fork and retry on failure, monitor resource use per test

Learner · November 26, 2018, 6:14pm

Hi!

We have many thousands of unit tests in a project recently migrated from Maven to Gradle. They all pass individually (i.e. when run using forkEvery 1). However, things start to fail if “forkEvery” isn’t set to 1. Setting it to 1 introduces a very severe performance penalty for obvious reasons.

While we tried to play with various values of forkEvery we eventually decided that such effort was pointless, at least for us, as with whatever value greater than 1 (e.g. even 2), there is a risk of running a “polite” test with a “naughty” one and have it fail because of the state it encountered at its start. We’ve observed this play out exactly like that at “forkEvery 2”.

Of course, we know that the best thing to do is to make the tests not leaky. The problem is that there are just too many of them and they are very hard to identify at this stage. I’ve tried fishing them out by severely limiting the max heap size for them and flagging the ones that fail even at “forkEvery 1” and that seemed to help… until it did not. We also don’t know what kinds of resources are leaked and/or how they are (not) cleaned up after test execution as not all failures are due to OutOfMemoryErrors (but all succeed with sufficient memory - say 64M - and forkEvery 1).

At this point we have two questions and some recommendations:

How can we customize our Gradle scripts to fork on failure and retry the failed test(s)? This way forkEvery would no longer be needed as fork would occur automatically when needed and would be almost perfectly optimal. True, this could hide some errors (such as tests that expect unqualified exceptions that now occur because of a previously run leaky test), but it would get us going and we would know what to expect. I would consider this to be a good feature in the base product too but we wouldn’t wait for it, if we don’t have to.
Is there any way to monitor the resource (memory and other) use of test workers between tests? This way we could, perhaps, see what is going on there and help us identify the causes. Note that we do not know what is going on. It may be memory or, perhaps, failure to clean up mocking set up by previous tests (we’ve seen class loading issues seemingly not caused by OutOfMemoryErrors).

Please help!

Learner · November 26, 2018, 10:10pm

I tried to see how can I get Java memory utilization before/after tests using RunListeners and/or Gradle TestListeners. Not sure (yet) how to configure these and whether these run within the same JVM that the tests run in or not. Can anyone help point me in the right direction w.r.t. that?

Learner · November 27, 2018, 2:35pm

Somewhat related, best I could find, but still no info there either:

Learner · November 27, 2018, 3:55pm

Also related to this - that should not have been closed, as it isn’t always the failing test that causes the issue:

Learner · November 29, 2018, 6:29pm

Still nothing?

In the interim I added custom runners for each of our tests using @RunWith, as I have no way of setting a global one. This uncovered at least two issues beyond our direct control, both causing tests to not run well with other tests:

and:

The only hope to gain some performance is to find a way to automatically group tests into forks so that we don’t have to forkEvery 1.

Anything?

sbabcoc · March 10, 2019, 7:19am

I don’t know if this will help or not, but the JUnit Foundation library includes an automatic retry feature.

github.com

Nordstrom/JUnit-Foundation/blob/master/README.md#automatic-retry-of-failed-tests

[![Maven Central](https://img.shields.io/maven-central/v/com.nordstrom.tools/junit-foundation.svg)](https://mvnrepository.com/artifact/com.nordstrom.tools/junit-foundation)

# INTRODUCTION

**JUnit Foundation** is a lightweight collection of JUnit watchers, interfaces, and static utility classes that supplement and augment the functionality provided by the JUnit API. The facilities provided by **JUnit Foundation** include method invocation hooks, test method timeout management, automatic retry of failed tests, shutdown hook installation, and test artifact capture.

## Test Lifecycle Notifications

The standard **RunListener** feature of JUnit provides a basic facility for implementing setup, cleanup, and monitoring procedures. However, the granularity of notifications offered by this feature is relatively coarse, firing before the first **`@Before`** method and after the last **`@After`** method - a unit of functionality known as an `atomic test`. Notifications are available for the start, finish, and failure of atomic tests, but not for the `particle methods` of which they're composed - individual **`@Test`** and configuration methods (**`@Before`**, **`@After`**, **`@BeforeClass`**, and **`@AfterClass`**).

With **JUnit Foundation**, you can get notifications for the invocation of every configuration and test method. This method interception feature is analogous to the **IInvokedMethodListener** feature of TestNG. You can also get notifications for the creation of test class instances, the creation and invocation of JUnit runners (both test classes and suites), and the completion of test runs. **JUnit Foundation** also provides notifications for the start, finish, and failure of `atomic tests`, with all of the details and context that are omitted by the standard JUnit **RunListener**.

### Notification Context and Test Run Hierarchy

The notifications provided by **JUnit Foundation** include the context that owns them - the JUnit runner. With this context and associated mapping methods, you're able to explore the entire hierarchy of the test run. For example, you can get the class runner that owns an invoked method or the suite runner that owns a class runner:

#### Walking the Object Hierarchy

The objects passed to your service provider implementation are members of a hierarchy that **JUnit** builds to represent the test collection being executed. **JUnit Foundation** provides a set of static methods that enable you walk this object hierarchy.

This file has been truncated. show original

Topic		Replies	Views
Want something like forkEvery but for every method Help/Discuss	0	456	December 1, 2015
Migration help - Running individual unit tests on a unique JVM Help/Discuss	1	442	October 15, 2018
TestNG not forking test processes Help/Discuss	1	1625	October 5, 2015
Is there a way to launch the VM only with tests that will actually be run? Old Forum Archive	1	521	March 28, 2014
Hung worker/OOME running TestNG Help/Discuss	20	2149	February 14, 2017

Unit testing: (re)fork and retry on failure, monitor resource use per test

Related topics