I have a few tests with TestFX that occasionally fails.
It is allways the same TestFX tests that fails.
A RuntimeException of either
ConcurrentModificationException
NullPointerException
IllegalStateException
It is always the same tests that fails.
Firstly it was only one test that occasionally failed. The more TestFX tests I created then others also began to fail. Always the same, but not all every time.
Test > someTest() FAILED
java.lang.RuntimeException
Caused by: java.util.ConcurrentModificationException
DifferentTest > someTest() FAILED
java.lang.RuntimeException
Caused by: java.lang.NullPointerException
AnotherTest > someTest() FAILED
java.lang.RuntimeException
Caused by: java.lang.IllegalStateException
Two TestFX tests I had in two test classes used to occasionally fail, but not too often.
Then recently I added one new TestFX test. Then several other different TestFX test failed, This time now always the same and on every build. I can disable this test class, and it then goes back to occasionally fail on two-three TestFX tests. I find this very strange, I cannot find anything wrong with my tests, they worked before I added this other totally non-related test class.
This latest test I have added that causes other tests to fail, is a class that takes in a result and then displays an Alert.
testResultError Displays an Alert
testResultTimeout Displays an Alert
testResultSuccess Does not display an Alert
Does anyone have any idea what I should check for? What could be wrong in my tests or in my code? Running my application works just fine, and does not get any errors for the components of those tests that are now failing.
I am using Monocle with TestFX and JUnit 5 to run headless on Jenkins. While building on my own computer it works fine and I almost never get these test errors. Though I have gotten them, but extremely rarely.
testCompile group: 'org.testfx', name: 'testfx-core', version : '4.0.13-alpha'
testCompile group: 'org.testfx', name: 'testfx-junit5', version : '4.0.13-alpha'
testCompile group: 'org.testfx', name: 'openjfx-monocle', version: '8u76-b04'
I annotate my tests with:
@ExtendWith(ApplicationExtension.class)
@TestInstance(Lifecycle.PER_CLASS)
Most of my onStart methods contain very little to almost nothing:
@Start
public void onStart(Stage stage) {
}
This is from one test of my custom control which fails every time now, that previously ran just fine.
@Start
public void onStart(Stage stage) {
contactList = new ContactList(createContacts());
StackPane root = new StackPane();
root.getChildren().add(contactList);
Scene scene = new Scene(root, 600, 300);
Path path = Paths.get("src/main/resources/com/company/application/fx/light.css");
scene.getStylesheets().add("file://" + path.toAbsolutePath().toString());
stage.setScene(scene);
stage.show();
}
It is very strange that just adding one or more TestFX test classes that some other TestFX tests would fail.
This is the first one that failed. Failed only occasionally. It is small so I can include it here.
A StageHelper to create a new Stage with our company logo icons.
public class StageHelper {
/**
* Get a new {@link Stage} with the Company Logos.
* @return new Stage window
*/
public static Stage newStage() {
final Stage stage = new Stage();
StageHelper.setStageIcons(stage);
StageHelper.setPrepareShow(stage);
return stage;
}
/**
* Get a new {@link Stage} with the Company Logos and a given {@link Scene}.
* @return new Stage window
*/
public static Stage newStage(final Scene scene) {
final Stage stage = new Stage();
stage.setScene(scene);
StageHelper.setStageIcons(stage);
StageHelper.setPrepareShow(stage);
return stage;
}
/**
* Set the {@link Stage} window with the Company Logos.
* @param stage the stage window
*/
public static void setStageIcons(Stage stage) {
ImageLocator.obj().setDefaultIcons(stage);
}
/**
* Prepare the {@link Stage} to show with center on screen, request focus and to front.
* @param stage the stage window
*/
public static void setPrepareShow(Stage stage) {
stage.centerOnScreen();
stage.requestFocus();
stage.toFront();
}
}
This is its test class with TestFX
@ExtendWith(ApplicationExtension.class)
@TestInstance(Lifecycle.PER_CLASS)
public class StageHelperTest {
private Stage stage;
@Start
public void onStart(Stage stage) {
final StackPane root = new StackPane();
final Scene scene = new Scene(root, 200, 200);
stage = StageHelper.newStage(scene);
this.stage = stage;
stage.setScene(scene);
stage.show();
}
@Test
public void succeedNewStage() throws InterruptedException, ExecutionException {
Task<Stage> task = new Task<Stage>() {
private Stage stage;
@Override
protected Stage call() throws Exception {
Platform.runLater(() -> {
stage = StageHelper.newStage();
});
return stage;
}
@Override
protected void succeeded() {
stage = StageHelper.newStage();
}
};
final ExecutorService executor = Executors.newSingleThreadExecutor();
executor.submit(task);
}
@Test
public void stageHasIcons(FxRobot robot) {
ObservableList<Image> images = stage.getIcons();
Assertions.assertArrayEquals(ImageLocator.obj().getLogoIconList().toArray(), images.toArray());
}
@Test
public void failNewStage() throws InterruptedException, ExecutionException {
final Executable closureContainingCodeToTest = () -> {
StageHelper.newStage();
};
assertThrows(IllegalStateException.class, closureContainingCodeToTest);
}
}
There was one test here that always failed when this test class failed and that was failNewStage.
com.company.application.utils.StageHelperTest > failNewStage() FAILED
java.lang.RuntimeException
Caused by: java.util.ConcurrentModificationException
Running on the command line on the server manually it almost never fails. But on my Jenkins CI now it Always fails on the same three tests. If I disable my latest test (which does not fail), then these three tests goes through fine.
gradle --no-daemon --refresh-dependencies clean build -Pheadless
test {
useJUnitPlatform()
if (project.hasProperty('headless')) {
systemProperty 'java.awt.headless', 'true'
systemProperty 'testfx.robot', 'glass'
systemProperty 'testfx.headless', 'true'
systemProperty 'prism.order', 'sw'
systemProperty 'prism.text', 't2k'
systemProperty 'testfx.setup.timeout', '8500'
}
}