Cucumber tests not discovered using new TestSuite

Help/Discuss
1

Hi,

We’re running a Gradle 7.6.1 multi-project Kotlin build with manually created additional source sets and test tasks to run Cucumber integration tests. This works fine.

While trying the Gradle 8.1 upgrade, I read about the new test suite feature and decided to give it a try. I created a suite for the Cucumber tests, and used useJUnitJupiter() on it.

I also defined the dependencies and everything compiles.

But when I run the generated test task, it completes very quickly without any errors, but no tests are run. If I run the task in IntelliJ it says that no test events were received.

We use the JUnit @Suite annotation to identify the Cucumber configuration:

import io.cucumber.core.options.Constants.PLUGIN_PROPERTY_NAME
import org.junit.platform.suite.api.ConfigurationParameter
import org.junit.platform.suite.api.IncludeEngines
import org.junit.platform.suite.api.SelectClasspathResource
import org.junit.platform.suite.api.Suite

@Suite
@IncludeEngines("cucumber")
@SelectClasspathResource("features")
@ConfigurationParameter(
    key = PLUGIN_PROPERTY_NAME,
    value = "pretty, summary, html:build/cucumber/report.html,json:build/cucumber/cucumber.json,junit:build/cucumber/cucumber.xml"
)
class CucumberConfiguration

The default test task, also created from a test suite, runs just fine, and runs all the unit tests, but these are annotated with the @Test annotation.

I’m just wondering, should I expect these Cucumber tests to run, or is the @Suite annotation maybe not supported yet by the test suites feature?

Thanks,

Paul

2

I also defined the dependencies and everything compiles.

Define “the dependencies”.
I have no idea about Cucumber, but from a very short look at their docs, you also need to add io.cucumber:cucumber-junit-platform-engine as runtimeOnly dependency to your test suite so that the test get executed on JUnit Platform.

Be aware, JUnit Platform != JUnit Jupiter.
JUnit Jupiter is on JUnit Platform engine that allows to run JUnit Jupiter (the so to say JUnit 5) tests.
Spock for example is a differend JUnit Platform engine.
And from Cucumber their docs / source it seems you need that dependency which also is a JUnit Platform engine for the tests to be discovered and executed.

3

Thanks, from reading the docs I think I have it setup correctly, but I might be missing something. I have this in build.gradle.kts:

        register<JvmTestSuite>("serviceTest") {
            useJUnit()

            dependencies {
                val cucumberVersion = "7.3.4"

                implementation(project())
                implementation("org.assertj:assertj-core:3.24.2")
                implementation("org.springframework.boot:spring-boot-starter-test")
                implementation("org.springframework.kafka:spring-kafka")
                implementation("org.apache.kafka:kafka-streams")
                implementation("org.testcontainers:testcontainers")
                implementation("org.testcontainers:postgresql")
                implementation("org.testcontainers:junit-jupiter")
                implementation("org.testcontainers:kafka")
                implementation("org.testcontainers:gcloud:1.17.5")
                implementation("org.springframework.cloud:spring-cloud-contract-wiremock:2.2.5.RELEASE")
                implementation("org.awaitility:awaitility:4.0.2")
                implementation("org.springframework.cloud:spring-cloud-gcp-pubsub:1.2.8.RELEASE")
                implementation("io.cucumber:cucumber-java8:$cucumberVersion")
                implementation("io.cucumber:cucumber-junit-platform-engine:$cucumberVersion")
                implementation("org.junit.platform:junit-platform-suite")
                implementation("io.cucumber:cucumber-spring:$cucumberVersion")
            }
            group = "test"
            targets {
                all {
                    testTask.configure {
                        shouldRunAfter(test)
                    }
                }
            }
            sources {
                kotlin {
                    setSrcDirs(listOf("src/service-test/kotlin"))
                }
            }
        }

If I run the service tests, nothing happens, though

$ ./gradlew --console=plain --no-build-cache :services:flowfx:serviceTest                                                                                                                          
Configuration on demand is an incubating feature.
> Task :services:flowfx:processResources UP-TO-DATE
> Task :services:flowfx:processServiceTestResources NO-SOURCE
> Task :libs:logger:compileKotlin UP-TO-DATE
> Task :libs:logger:compileJava NO-SOURCE
> Task :libs:logger:processResources UP-TO-DATE
> Task :libs:logger:classes UP-TO-DATE
> Task :libs:logger:jar UP-TO-DATE
> Task :services:flowfx:compileKotlin UP-TO-DATE
> Task :services:flowfx:compileJava UP-TO-DATE
> Task :services:flowfx:classes UP-TO-DATE
> Task :services:flowfx:jar UP-TO-DATE
> Task :services:flowfx:compileServiceTestKotlin UP-TO-DATE
> Task :services:flowfx:compileServiceTestJava NO-SOURCE
> Task :services:flowfx:serviceTestClasses UP-TO-DATE
> Task :services:flowfx:serviceTest UP-TO-DATE

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/8.1.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 5s
9 actionable tasks: 9 up-to-date

The unit tests are also a test suite, and they run just fine, using JUnit Jupiter. The obvious difference is of course JUnit/Cucumber vs JUnit/Jupiter, where the Cucumber tests should get picked up via the @Suite annotation. This is configured so:

package com.kambr.eddy.flowfx.configuration

import io.cucumber.core.options.Constants.PLUGIN_PROPERTY_NAME
import org.junit.platform.suite.api.ConfigurationParameter
import org.junit.platform.suite.api.IncludeEngines
import org.junit.platform.suite.api.SelectClasspathResource
import org.junit.platform.suite.api.Suite

@Suite
@IncludeEngines("cucumber")
@SelectClasspathResource("features")
@ConfigurationParameter(
    key = PLUGIN_PROPERTY_NAME,
    value = "pretty, summary, html:build/cucumber/report.html,json:build/cucumber/cucumber.json,junit:build/cucumber/cucumber.xml"
)
class CucumberConfiguration

So I was wondering if that’s maybe not supported (yet) using the test suite config. Note that they run just fine when the source sets and configurations are setup manually. I’m trying to convert a running setup into a test suite, and the cucumber test just stop working.

4

Well, I had indeed missed something. I hadn’t configured the test resources so there were no features to run at all. It works now, the final test suite configuration is this:

        register<JvmTestSuite>("serviceTest") {
            useJUnitJupiter()

            dependencies {
                val cucumberVersion = "7.11.2"

                implementation(project())
                implementation("org.assertj:assertj-core:3.24.2")
                implementation("org.springframework.boot:spring-boot-starter-test")
                implementation("org.springframework.kafka:spring-kafka")
                implementation("org.apache.kafka:kafka-streams")
                implementation("org.testcontainers:testcontainers")
                implementation("org.testcontainers:postgresql")
                implementation("org.testcontainers:junit-jupiter")
                implementation("org.testcontainers:kafka")
                implementation("org.testcontainers:gcloud:1.17.5")
                implementation("org.springframework.cloud:spring-cloud-contract-wiremock:2.2.5.RELEASE")
                implementation("org.awaitility:awaitility:4.0.2")
                implementation("org.springframework.cloud:spring-cloud-gcp-pubsub:1.2.8.RELEASE")
                implementation("io.cucumber:cucumber-java8:$cucumberVersion")
                implementation("io.cucumber:cucumber-junit-platform-engine:$cucumberVersion")
                implementation("org.junit.platform:junit-platform-suite")
                implementation("io.cucumber:cucumber-spring:$cucumberVersion")
            }
            group = "test"
            targets {
                all {
                    testTask.configure {
                        shouldRunAfter(test)
                    }
                }
            }
            sources {
                kotlin {
                    srcDirs("src/service-test/kotlin")
                }
                resources {
                    srcDirs("src/service-test/resources")
                }
            }
        }
5

Your main problem though was, that you used useJUnit(), which means JUnit 4.
There is unfortunately no useJUnitPlatform() yet for the test suites dsl, only directly on test tasks.
So you either need to do the useJUnitPlatform() on the test task, or use something like useSpock() or useJUnitPlatform() like you did so that it actually uses JUnit Platform and your Cucumber engine can do its work.

6

You can thumbs-up and follow that issue at `useJUnitPlatform()` for JVM Test Suite plugin test suites · Issue #21299 · gradle/gradle · GitHub