How do 'jvm-test-suite' and 'java-test-fixtures' relate to each other and how can they be used cross project

We’re currently using the GitHub - unbroken-dome/gradle-testsets-plugin: A plugin for the Gradle build system that allows specifying test sets (like integration or acceptance tests). to configure integrationTests.
These sets have 2 purposes:

  • main reason is of cause containing tests for the component that require special setup and need to be executed separately, therefor they also need to be packaged and be consumable by an ear project within the same build
  • in some instances however they serve the purpose of containing testFixtures that will be consumed by other projects, either in the same multiproject build of even by other standalone builds

With the rise of the jvm-test-suite plugin I thought any test-set related use case should now be possible with this plugin. However there are some questions I could not find a good answer for in the documentation:

  • the java-test-fixtures provided a special dependency feature to be consumed by other projects in the same build, but I can’t seem to find out how this should look like for jvm-test-suite suites
dependencies {
    implementation(project(":lib"))

    testImplementation 'junit:junit:4.13'
    testImplementation(testFixtures(project(":lib"))) // how to declare this for a test-suite?
}
  • test-fixtures-plugin also automatically added a software variant that was distributed and could easily be consumed by other builds, I can’t seem to find anything like that for jvm-test-suites or did I miss something?
  • is it even possible to define a (unit)testFixture with jvm-test-suites (seems to be the question here: Jvm-test-suite - create common test sources for use in multiple test suites)?

So overall I am kind of confused about the relationship or the future of the java-test-fixtures vs jvm-test-suites plugins.
And the thing that matters the most to me how should we set up proper dependencies on jvm-test-suites?

Maybe it’s not the time yet to apply the test suites in my scenario (requiring unitTestFixtures, unitTests, integrationTestFixtures and integrationTests) but I was actually hoping to be able to achieve that already without depending on external plugins (or having to deal with a whole lot of configuration).

kind regards
Daniel

Reading through the docs I found various articles all stating how additional test suites may be configured all going into different kind of details and having another pro and cons.

Testing in Java & JVM projects seems to be there for the longest time. But also looks rather rough compared to the other options.

I found that to be helpful even though it’s just missing the distribution part:
https://docs.gradle.org/7.6/samples/sample_jvm_multi_project_with_additional_test_types.html

Looking into testFixtures I could find everything I was looking for (simple declaration, configuration and distribution of artifacts), just that it’s limited to that specific use-case and can not be extrapolated to be useable for integrationTests

The newest addition is obviously the jvm-test-suite:
https://docs.gradle.org/7.6/userguide/jvm_test_suite_plugin.html#jvm_test_suite_plugin
but the concept is unclear to me (having multiple targets and arguing to not use the global dependencies block anymore is obviously preparing smth. but I don’t understand what and it got no significant updates since it’s introduction, right?) and most important for me i- t’s missing the artifact building and distribution feature.

From the textFixture plugin I got the idea of using feature variants:
https://docs.gradle.org/7.6/userguide/feature_variants.html

def integrationTest = sourceSets.create('integrationTest')
java {
    registerFeature('integrationTest') {
        usingSourceSet(integrationTest)
    }
}

this works fine in tandem with the “Using additional test types Sample” approach and I can easily depend on the code and artifacts:

dependencies {
    testImplementation(project(':feature-base')) {
        capabilities {
            requireCapability("com.example:feature-base-integration-test")
        }
    }
}

It was a bit difficult to match that with the jvm test-suite as I could not define the SourceSet for the feature registration, but had to wait after the definition of the testSuite for it to be there…
This breaks:

def integrationTest = sourceSets.create('integrationTest')
testing {
    suites {
        integrationTest(JvmTestSuite) {}
    }
}
// ends up throwing:
/*
Could not find method call() for arguments [interface org.gradle.api.plugins.jvm.JvmTestSuite, build_bp1rs6g1bwg6xxynjkqddzgl9$_run_closure1$_closure4$_closure5@2b2b926c] on source set 'integration test' of type org.gradle.api.internal.tasks.DefaultSourceSet.
*/
Stacktrace
  • Exception is:
    org.gradle.api.GradleScriptException: A problem occurred evaluating project ‘:suite-base’.
    at org.gradle.groovy.scripts.internal.DefaultScriptRunnerFactory$ScriptRunnerImpl.run(DefaultScriptRunnerFactory.java:93)
    at org.gradle.configuration.DefaultScriptPluginFactory$ScriptPluginImpl.lambda$apply$0(DefaultScriptPluginFactory.java:133)
    at org.gradle.configuration.ProjectScriptTarget.addConfiguration(ProjectScriptTarget.java:79)
    at org.gradle.configuration.DefaultScriptPluginFactory$ScriptPluginImpl.apply(DefaultScriptPluginFactory.java:136)
    at org.gradle.configuration.BuildOperationScriptPlugin$1.run(BuildOperationScriptPlugin.java:65)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:29)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:26)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.run(DefaultBuildOperationRunner.java:47)
    at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:68)
    at org.gradle.configuration.BuildOperationScriptPlugin.lambda$apply$0(BuildOperationScriptPlugin.java:62)
    at org.gradle.configuration.internal.DefaultUserCodeApplicationContext.apply(DefaultUserCodeApplicationContext.java:44)
    at org.gradle.configuration.BuildOperationScriptPlugin.apply(BuildOperationScriptPlugin.java:62)
    at org.gradle.api.internal.project.DefaultProjectStateRegistry$ProjectStateImpl.lambda$applyToMutableState$0(DefaultProjectStateRegistry.java:360)
    at org.gradle.api.internal.project.DefaultProjectStateRegistry$ProjectStateImpl.fromMutableState(DefaultProjectStateRegistry.java:378)
    at org.gradle.api.internal.project.DefaultProjectStateRegistry$ProjectStateImpl.applyToMutableState(DefaultProjectStateRegistry.java:359)
    at org.gradle.configuration.project.BuildScriptProcessor.execute(BuildScriptProcessor.java:42)
    at org.gradle.configuration.project.BuildScriptProcessor.execute(BuildScriptProcessor.java:26)
    at org.gradle.configuration.project.ConfigureActionsProjectEvaluator.evaluate(ConfigureActionsProjectEvaluator.java:35)
    at org.gradle.configuration.project.LifecycleProjectEvaluator$EvaluateProject.lambda$run$0(LifecycleProjectEvaluator.java:109)
    at org.gradle.api.internal.project.DefaultProjectStateRegistry$ProjectStateImpl.lambda$applyToMutableState$0(DefaultProjectStateRegistry.java:360)
    at org.gradle.api.internal.project.DefaultProjectStateRegistry$ProjectStateImpl.lambda$fromMutableState$1(DefaultProjectStateRegistry.java:383)
    at org.gradle.internal.work.DefaultWorkerLeaseService.withReplacedLocks(DefaultWorkerLeaseService.java:345)
    at org.gradle.api.internal.project.DefaultProjectStateRegistry$ProjectStateImpl.fromMutableState(DefaultProjectStateRegistry.java:383)
    at org.gradle.api.internal.project.DefaultProjectStateRegistry$ProjectStateImpl.applyToMutableState(DefaultProjectStateRegistry.java:359)
    at org.gradle.configuration.project.LifecycleProjectEvaluator$EvaluateProject.run(LifecycleProjectEvaluator.java:100)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:29)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:26)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.run(DefaultBuildOperationRunner.java:47)
    at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:68)
    at org.gradle.configuration.project.LifecycleProjectEvaluator.evaluate(LifecycleProjectEvaluator.java:72)
    at org.gradle.api.internal.project.DefaultProject.evaluate(DefaultProject.java:762)
    at org.gradle.api.internal.project.DefaultProject.evaluate(DefaultProject.java:153)
    at org.gradle.api.internal.project.ProjectLifecycleController.lambda$ensureSelfConfigured$1(ProjectLifecycleController.java:63)
    at org.gradle.internal.model.StateTransitionController.lambda$doTransition$12(StateTransitionController.java:236)
    at org.gradle.internal.model.StateTransitionController.doTransition(StateTransitionController.java:247)
    at org.gradle.internal.model.StateTransitionController.doTransition(StateTransitionController.java:235)
    at org.gradle.internal.model.StateTransitionController.lambda$maybeTransitionIfNotCurrentlyTransitioning$9(StateTransitionController.java:196)
    at org.gradle.internal.work.DefaultSynchronizer.withLock(DefaultSynchronizer.java:34)
    at org.gradle.internal.model.StateTransitionController.maybeTransitionIfNotCurrentlyTransitioning(StateTransitionController.java:192)
    at org.gradle.api.internal.project.ProjectLifecycleController.ensureSelfConfigured(ProjectLifecycleController.java:63)
    at org.gradle.api.internal.project.DefaultProjectStateRegistry$ProjectStateImpl.ensureConfigured(DefaultProjectStateRegistry.java:334)
    at org.gradle.execution.TaskPathProjectEvaluator.configure(TaskPathProjectEvaluator.java:33)
    at org.gradle.execution.TaskPathProjectEvaluator.configureHierarchy(TaskPathProjectEvaluator.java:49)
    at org.gradle.configuration.DefaultProjectsPreparer.prepareProjects(DefaultProjectsPreparer.java:50)
    at org.gradle.configuration.BuildTreePreparingProjectsPreparer.prepareProjects(BuildTreePreparingProjectsPreparer.java:64)
    at org.gradle.configuration.BuildOperationFiringProjectsPreparer$ConfigureBuild.run(BuildOperationFiringProjectsPreparer.java:52)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:29)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:26)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.run(DefaultBuildOperationRunner.java:47)
    at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:68)
    at org.gradle.configuration.BuildOperationFiringProjectsPreparer.prepareProjects(BuildOperationFiringProjectsPreparer.java:40)
    at org.gradle.initialization.VintageBuildModelController.lambda$prepareProjects$2(VintageBuildModelController.java:84)
    at org.gradle.internal.model.StateTransitionController.lambda$doTransition$12(StateTransitionController.java:236)
    at org.gradle.internal.model.StateTransitionController.doTransition(StateTransitionController.java:247)
    at org.gradle.internal.model.StateTransitionController.doTransition(StateTransitionController.java:235)
    at org.gradle.internal.model.StateTransitionController.lambda$transitionIfNotPreviously$10(StateTransitionController.java:210)
    at org.gradle.internal.work.DefaultSynchronizer.withLock(DefaultSynchronizer.java:34)
    at org.gradle.internal.model.StateTransitionController.transitionIfNotPreviously(StateTransitionController.java:206)
    at org.gradle.initialization.VintageBuildModelController.prepareProjects(VintageBuildModelController.java:84)
    at org.gradle.initialization.VintageBuildModelController.getConfiguredModel(VintageBuildModelController.java:64)
    at org.gradle.internal.build.DefaultBuildLifecycleController.lambda$withProjectsConfigured$1(DefaultBuildLifecycleController.java:116)
    at org.gradle.internal.model.StateTransitionController.lambda$notInState$3(StateTransitionController.java:143)
    at org.gradle.internal.work.DefaultSynchronizer.withLock(DefaultSynchronizer.java:44)
    at org.gradle.internal.model.StateTransitionController.notInState(StateTransitionController.java:139)
    at org.gradle.internal.build.DefaultBuildLifecycleController.withProjectsConfigured(DefaultBuildLifecycleController.java:116)
    at org.gradle.internal.build.DefaultBuildToolingModelController.locateBuilderForTarget(DefaultBuildToolingModelController.java:57)
    at org.gradle.internal.buildtree.DefaultBuildTreeModelCreator$DefaultBuildTreeModelController.lambda$locateBuilderForTarget$0(DefaultBuildTreeModelCreator.java:73)
    at org.gradle.internal.build.DefaultBuildLifecycleController.withToolingModels(DefaultBuildLifecycleController.java:180)
    at org.gradle.internal.build.AbstractBuildState.withToolingModels(AbstractBuildState.java:123)
    at org.gradle.internal.buildtree.DefaultBuildTreeModelCreator$DefaultBuildTreeModelController.locateBuilderForTarget(DefaultBuildTreeModelCreator.java:73)
    at org.gradle.internal.buildtree.DefaultBuildTreeModelCreator$DefaultBuildTreeModelController.locateBuilderForDefaultTarget(DefaultBuildTreeModelCreator.java:68)
    at org.gradle.tooling.internal.provider.runner.DefaultBuildController.getTarget(DefaultBuildController.java:157)
    at org.gradle.tooling.internal.provider.runner.DefaultBuildController.getModel(DefaultBuildController.java:101)
    at org.gradle.tooling.internal.consumer.connection.ParameterAwareBuildControllerAdapter.getModel(ParameterAwareBuildControllerAdapter.java:39)
    at org.gradle.tooling.internal.consumer.connection.UnparameterizedBuildController.getModel(UnparameterizedBuildController.java:113)
    at org.gradle.tooling.internal.consumer.connection.NestedActionAwareBuildControllerAdapter.getModel(NestedActionAwareBuildControllerAdapter.java:31)
    at org.gradle.tooling.internal.consumer.connection.UnparameterizedBuildController.findModel(UnparameterizedBuildController.java:97)
    at org.gradle.tooling.internal.consumer.connection.NestedActionAwareBuildControllerAdapter.findModel(NestedActionAwareBuildControllerAdapter.java:31)
    at org.gradle.tooling.internal.consumer.connection.UnparameterizedBuildController.findModel(UnparameterizedBuildController.java:81)
    at org.gradle.tooling.internal.consumer.connection.NestedActionAwareBuildControllerAdapter.findModel(NestedActionAwareBuildControllerAdapter.java:31)
    at org.gradle.tooling.internal.consumer.connection.UnparameterizedBuildController.findModel(UnparameterizedBuildController.java:66)
    at org.gradle.tooling.internal.consumer.connection.NestedActionAwareBuildControllerAdapter.findModel(NestedActionAwareBuildControllerAdapter.java:31)
    at org.jetbrains.plugins.gradle.model.ProjectImportAction.execute(ProjectImportAction.java:125)
    at org.jetbrains.plugins.gradle.model.ProjectImportAction.execute(ProjectImportAction.java:42)
    at org.gradle.tooling.internal.consumer.connection.InternalBuildActionAdapter.execute(InternalBuildActionAdapter.java:64)
    at org.gradle.tooling.internal.provider.runner.AbstractClientProvidedBuildActionRunner$ActionAdapter.runAction(AbstractClientProvidedBuildActionRunner.java:131)
    at org.gradle.tooling.internal.provider.runner.AbstractClientProvidedBuildActionRunner$ActionAdapter.beforeTasks(AbstractClientProvidedBuildActionRunner.java:99)
    at org.gradle.internal.buildtree.DefaultBuildTreeModelCreator.beforeTasks(DefaultBuildTreeModelCreator.java:52)
    at org.gradle.internal.buildtree.DefaultBuildTreeLifecycleController.lambda$fromBuildModel$2(DefaultBuildTreeLifecycleController.java:82)
    at org.gradle.internal.buildtree.DefaultBuildTreeLifecycleController.lambda$runBuild$5(DefaultBuildTreeLifecycleController.java:113)
    at org.gradle.internal.model.StateTransitionController.lambda$transition$5(StateTransitionController.java:166)
    at org.gradle.internal.model.StateTransitionController.doTransition(StateTransitionController.java:247)
    at org.gradle.internal.model.StateTransitionController.lambda$transition$6(StateTransitionController.java:166)
    at org.gradle.internal.work.DefaultSynchronizer.withLock(DefaultSynchronizer.java:44)
    at org.gradle.internal.model.StateTransitionController.transition(StateTransitionController.java:166)
    at org.gradle.internal.buildtree.DefaultBuildTreeLifecycleController.runBuild(DefaultBuildTreeLifecycleController.java:110)
    at org.gradle.internal.buildtree.DefaultBuildTreeLifecycleController.fromBuildModel(DefaultBuildTreeLifecycleController.java:81)
    at org.gradle.tooling.internal.provider.runner.AbstractClientProvidedBuildActionRunner.runClientAction(AbstractClientProvidedBuildActionRunner.java:43)
    at org.gradle.tooling.internal.provider.runner.ClientProvidedPhasedActionRunner.run(ClientProvidedPhasedActionRunner.java:53)
    at org.gradle.launcher.exec.ChainingBuildActionRunner.run(ChainingBuildActionRunner.java:35)
    at org.gradle.internal.buildtree.ProblemReportingBuildActionRunner.run(ProblemReportingBuildActionRunner.java:49)
    at org.gradle.launcher.exec.BuildOutcomeReportingBuildActionRunner.run(BuildOutcomeReportingBuildActionRunner.java:65)
    at org.gradle.tooling.internal.provider.FileSystemWatchingBuildActionRunner.run(FileSystemWatchingBuildActionRunner.java:136)
    at org.gradle.launcher.exec.BuildCompletionNotifyingBuildActionRunner.run(BuildCompletionNotifyingBuildActionRunner.java:41)
    at org.gradle.launcher.exec.RootBuildLifecycleBuildActionExecutor.lambda$execute$0(RootBuildLifecycleBuildActionExecutor.java:40)
    at org.gradle.composite.internal.DefaultRootBuildState.run(DefaultRootBuildState.java:122)
    at org.gradle.launcher.exec.RootBuildLifecycleBuildActionExecutor.execute(RootBuildLifecycleBuildActionExecutor.java:40)
    at org.gradle.internal.buildtree.DefaultBuildTreeContext.execute(DefaultBuildTreeContext.java:40)
    at org.gradle.launcher.exec.BuildTreeLifecycleBuildActionExecutor.lambda$execute$0(BuildTreeLifecycleBuildActionExecutor.java:65)
    at org.gradle.internal.buildtree.BuildTreeState.run(BuildTreeState.java:53)
    at org.gradle.launcher.exec.BuildTreeLifecycleBuildActionExecutor.execute(BuildTreeLifecycleBuildActionExecutor.java:65)
    at org.gradle.launcher.exec.RunAsBuildOperationBuildActionExecutor$3.call(RunAsBuildOperationBuildActionExecutor.java:61)
    at org.gradle.launcher.exec.RunAsBuildOperationBuildActionExecutor$3.call(RunAsBuildOperationBuildActionExecutor.java:57)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)
    at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73)
    at org.gradle.launcher.exec.RunAsBuildOperationBuildActionExecutor.execute(RunAsBuildOperationBuildActionExecutor.java:57)
    at org.gradle.launcher.exec.RunAsWorkerThreadBuildActionExecutor.lambda$execute$0(RunAsWorkerThreadBuildActionExecutor.java:36)
    at org.gradle.internal.work.DefaultWorkerLeaseService.withLocks(DefaultWorkerLeaseService.java:249)
    at org.gradle.internal.work.DefaultWorkerLeaseService.runAsWorkerThread(DefaultWorkerLeaseService.java:109)
    at org.gradle.launcher.exec.RunAsWorkerThreadBuildActionExecutor.execute(RunAsWorkerThreadBuildActionExecutor.java:36)
    at org.gradle.tooling.internal.provider.continuous.ContinuousBuildActionExecutor.execute(ContinuousBuildActionExecutor.java:110)
    at org.gradle.tooling.internal.provider.SubscribableBuildActionExecutor.execute(SubscribableBuildActionExecutor.java:64)
    at org.gradle.internal.session.DefaultBuildSessionContext.execute(DefaultBuildSessionContext.java:46)
    at org.gradle.tooling.internal.provider.BuildSessionLifecycleBuildActionExecuter$ActionImpl.apply(BuildSessionLifecycleBuildActionExecuter.java:100)
    at org.gradle.tooling.internal.provider.BuildSessionLifecycleBuildActionExecuter$ActionImpl.apply(BuildSessionLifecycleBuildActionExecuter.java:88)
    at org.gradle.internal.session.BuildSessionState.run(BuildSessionState.java:69)
    at org.gradle.tooling.internal.provider.BuildSessionLifecycleBuildActionExecuter.execute(BuildSessionLifecycleBuildActionExecuter.java:62)
    at org.gradle.tooling.internal.provider.BuildSessionLifecycleBuildActionExecuter.execute(BuildSessionLifecycleBuildActionExecuter.java:41)
    at org.gradle.tooling.internal.provider.StartParamsValidatingActionExecuter.execute(StartParamsValidatingActionExecuter.java:63)
    at org.gradle.tooling.internal.provider.StartParamsValidatingActionExecuter.execute(StartParamsValidatingActionExecuter.java:31)
    at org.gradle.tooling.internal.provider.SessionFailureReportingActionExecuter.execute(SessionFailureReportingActionExecuter.java:52)
    at org.gradle.tooling.internal.provider.SessionFailureReportingActionExecuter.execute(SessionFailureReportingActionExecuter.java:40)
    at org.gradle.tooling.internal.provider.SetupLoggingActionExecuter.execute(SetupLoggingActionExecuter.java:47)
    at org.gradle.tooling.internal.provider.SetupLoggingActionExecuter.execute(SetupLoggingActionExecuter.java:31)
    at org.gradle.launcher.daemon.server.exec.ExecuteBuild.doBuild(ExecuteBuild.java:65)
    at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:37)
    at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
    at org.gradle.launcher.daemon.server.exec.WatchForDisconnection.execute(WatchForDisconnection.java:39)
    at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
    at org.gradle.launcher.daemon.server.exec.ResetDeprecationLogger.execute(ResetDeprecationLogger.java:29)
    at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
    at org.gradle.launcher.daemon.server.exec.RequestStopIfSingleUsedDaemon.execute(RequestStopIfSingleUsedDaemon.java:35)
    at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
    at org.gradle.launcher.daemon.server.exec.ForwardClientInput$2.create(ForwardClientInput.java:78)
    at org.gradle.launcher.daemon.server.exec.ForwardClientInput$2.create(ForwardClientInput.java:75)
    at org.gradle.util.internal.Swapper.swap(Swapper.java:38)
    at org.gradle.launcher.daemon.server.exec.ForwardClientInput.execute(ForwardClientInput.java:75)
    at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
    at org.gradle.launcher.daemon.server.exec.LogAndCheckHealth.execute(LogAndCheckHealth.java:55)
    at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
    at org.gradle.launcher.daemon.server.exec.LogToClient.doBuild(LogToClient.java:63)
    at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:37)
    at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
    at org.gradle.launcher.daemon.server.exec.EstablishBuildEnvironment.doBuild(EstablishBuildEnvironment.java:84)
    at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:37)
    at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
    at org.gradle.launcher.daemon.server.exec.StartBuildOrRespondWithBusy$1.run(StartBuildOrRespondWithBusy.java:52)
    at org.gradle.launcher.daemon.server.DaemonStateCoordinator$1.run(DaemonStateCoordinator.java:297)
    at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
    at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
    Caused by: org.gradle.internal.metaobject.AbstractDynamicObject$CustomMessageMissingMethodException: Could not find method call() for arguments [interface org.gradle.api.plugins.jvm.JvmTestSuite, build_bp1rs6g1bwg6xxynjkqddzgl9$_run_closure1$_closure4$_closure5@2b2b926c] on source set ‘integration test’ of type org.gradle.api.internal.tasks.DefaultSourceSet.
    at org.gradle.internal.metaobject.AbstractDynamicObject$CustomMissingMethodExecutionFailed.(AbstractDynamicObject.java:190)
    at org.gradle.internal.metaobject.AbstractDynamicObject.methodMissingException(AbstractDynamicObject.java:184)
    at org.gradle.internal.metaobject.AbstractDynamicObject.invokeMethod(AbstractDynamicObject.java:167)
    at org.gradle.api.internal.tasks.DefaultSourceSet_Decorated.invokeMethod(Unknown Source)
    at build_bp1rs6g1bwg6xxynjkqddzgl9$_run_closure1$_closure4.doCall(C:\EDF\sources\Temp\reproducer-it\suite-base\build.gradle:10)
    at org.gradle.util.internal.ClosureBackedAction.execute(ClosureBackedAction.java:73)
    at org.gradle.util.internal.ConfigureUtil.configureTarget(ConfigureUtil.java:155)
    at org.gradle.util.internal.ConfigureUtil.configureSelf(ConfigureUtil.java:143)
    at org.gradle.api.internal.AbstractNamedDomainObjectContainer.configure(AbstractNamedDomainObjectContainer.java:91)
    at org.gradle.api.internal.AbstractNamedDomainObjectContainer.configure(AbstractNamedDomainObjectContainer.java:38)
    at org.gradle.internal.extensibility.MixInClosurePropertiesAsMethodsDynamicObject.tryInvokeMethod(MixInClosurePropertiesAsMethodsDynamicObject.java:55)
    at org.gradle.internal.metaobject.ConfigureDelegate.invokeMethod(ConfigureDelegate.java:57)
    at build_bp1rs6g1bwg6xxynjkqddzgl9$_run_closure1.doCall(C:\EDF\sources\Temp\reproducer-it\suite-base\build.gradle:9)
    at org.gradle.util.internal.ClosureBackedAction.execute(ClosureBackedAction.java:73)
    at org.gradle.util.internal.ConfigureUtil.configureTarget(ConfigureUtil.java:155)
    at org.gradle.util.internal.ConfigureUtil.configure(ConfigureUtil.java:106)
    at org.gradle.util.internal.ConfigureUtil$WrappedConfigureAction.execute(ConfigureUtil.java:167)
    at org.gradle.internal.extensibility.ExtensionsStorage$ExtensionHolder.configure(ExtensionsStorage.java:173)
    at org.gradle.internal.extensibility.ExtensionsStorage.configureExtension(ExtensionsStorage.java:64)
    at org.gradle.internal.extensibility.DefaultConvention.configureExtension(DefaultConvention.java:364)
    at org.gradle.internal.extensibility.DefaultConvention.access$500(DefaultConvention.java:45)
    at org.gradle.internal.extensibility.DefaultConvention$ExtensionsDynamicObject.tryInvokeMethod(DefaultConvention.java:301)
    at org.gradle.internal.metaobject.CompositeDynamicObject.tryInvokeMethod(CompositeDynamicObject.java:98)
    at org.gradle.internal.extensibility.MixInClosurePropertiesAsMethodsDynamicObject.tryInvokeMethod(MixInClosurePropertiesAsMethodsDynamicObject.java:36)
    at org.gradle.groovy.scripts.BasicScript$ScriptDynamicObject.tryInvokeMethod(BasicScript.java:135)
    at org.gradle.internal.metaobject.AbstractDynamicObject.invokeMethod(AbstractDynamicObject.java:163)
    at org.gradle.groovy.scripts.BasicScript.invokeMethod(BasicScript.java:84)
    at build_bp1rs6g1bwg6xxynjkqddzgl9.run(C:\EDF\sources\Temp\reproducer-it\suite-base\build.gradle:8)
    at org.gradle.groovy.scripts.internal.DefaultScriptRunnerFactory$ScriptRunnerImpl.run(DefaultScriptRunnerFactory.java:91)
    … 177 more

If one does it like this it’s working but having to declare them in order does not math the lazy configuration goals, where a order is not guaranteed, right?

testing {
    suites {
        integrationTest(JvmTestSuite) { /*...*/ }
    }
}
java {
    registerFeature('integrationTest') {
        usingSourceSet(sourceSets.named('integrationTest').get())
    }
}

I’m affraid this ends up breaking at some point.

So the question is, what’s the gradle way of doing it and do you maybe want to update the documentation to provide a crystal clear view on the approaches that will be future proof?

kind regards
Daniel

Not everything is order independent.
Plugins should indeed be written order-independently, so that they either react to another plugin being applied in the past or future, or applying the plugin directly if it is always needed.

But the statements within one build script always were and always will be order sensitive.
For example you first have to create a configuration, before you can declare dependencies on it.
Or you first have to create a source set before you can declare a feature from it.
And so on.

  • the java-test-fixtures provided a special dependency feature to be consumed by other projects in the same build

Not only in the same build, test fixtures by default are also published, so you can also depend on them from outside the current build if you don’t turn it off.

but I can’t seem to find out how this should look like for jvm-test-suite suites

One has nothing to do with the other.
The test fixtures feature provides a mean to provide some classes - typically test fixtures or test utilities - that can be consumed by the same project, or by other projects inside or outside the current build within their tests.
The jvm-test-suite plugin is to have multiple test suites like unit tests, integ tests, functional tests, load tests, that are clearly separated.
They are by default not meant to be depended upon, but to contain tests that are executed.
But if you insist, you can of course define feature variants for those source sets as you already discovered and then depend on them.

So overall I am kind of confused about the relationship or the future of the java-test-fixtures vs jvm-test-suites plugins.

As I said, they are two completely different things with completely different purposes that can nicely work together though, and that are most probably both are there to stay.

Maybe it’s not the time yet to apply the test suites in my scenario (requiring unitTestFixtures, unitTests, integrationTestFixtures and integrationTests) but I was actually hoping to be able to achieve that already without depending on external plugins (or having to deal with a whole lot of configuration).

The test fixtures plugin only provides one test fixture per project.
But it is actually just some convenience sugar for defining a feature variant, so you can pretty easily do it yourself.
Just create a source set for your additional fixtures (I wouldn’t use a test suite for that, as test suites are meant to contain executable tests and get tasks for executing them and so on, not to provide test fixtures).
Then declare a feature variant from that source set.
And on the consumer side, depend on the capability of that feature when needing the fixtures.

So the question is, what’s the gradle way of doing it

As I described above, you practically found the Gradle way already, except maybe with a separate source set, not the test suite. Except you also have tests in there that you want to execute in your project while still depending on them from somewhere else, but that feels a bit like a mix of responsibilities. (Hence by default the separate plugins for test fixtures and test suites)

and do you maybe want to update the documentation to provide a crystal clear view on the approaches that will be future proof?

If you feel it is not clear enough, you should open an issue or pull request over on GitHub, so that the Gradle folks can consider it. This is a community forum where mostly users like me are helping other users.

It might also be worth opening a feature request so that the test fixtures plugin could create one fixture feature per test suite and not only one for the default test suite, either always or configurable.

having multiple targets

Unfortunately, that is actually not yet possible but something to come. Currently you can only have one target per suite.

and arguing to not use the global dependencies block anymore

I wouldn’t call it arguing, you can still use the global dependencies block.
It is mainly syntax sugar and help for the Kotlin DSL.
Because in the test suite dependencies block, you can use implementation, runtimeOnly, and so on.
In the global dependencies block you would need to use integrationTestImplementation, integrationTestRuntimeOnly, and so on.
And as those configurations are not created by applying a plugin, but by additional configuration of the plugin in the build script, there are for example no type-safe accessors for those generated for Kotlin DSL, so you would first to get them by name for Kotlin DSL or use the string-y version "integrationTestImplementation"("...") which both is too cumbersome when you can just do it in the test suite dependencies block.

and it got no significant updates since it’s introduction, right?

Depends on what you call significant :smiley:

and most important for me i- t’s missing the artifact building and distribution feature.

Well, as described above, that is not “missing” but just not the right responsibility by default, which you can realtively easily change. :slight_smile:


Btw., this:

Could not find method call() for arguments [interface org.gradle.api.plugins.jvm.JvmTestSuite, build_bp1rs6g1bwg6xxynjkqddzgl9$_run_closure1$_closure4$_closure5@2b2b926c] on source set ‘integration test’ of type org.gradle.api.internal.tasks.DefaultSourceSet.

is one of the major reasons I recommend using the Kotlin DSL - which now also is the official default - instead of the Groovy DSL. You instantly get type-safe build scripts, actually helpful error messages if you mess up the syntax instead of unhelpul errors like the one you showed, and amazingly better IDE support if you use a proper IDE like IntelliJ.

Thank you so much for the extensive answer.

I get that my requirements may be a bit off due to a very antique testing framework, where the integrationTests must be packaged and deployed in a testing application. That’s why the publishing is so important for me, but I get it may not be like that for everyone.

Not having to test the testFixtures was also a good point thanks for bringing this to my attention. Let’s see how much attention the feature request to support more than one fixture will get.

About the error message, yeah kotlin is on my todo list for way too long now, but in the end I’m using the groovy syntax just for prototyping. Switching from Gradle 7.6 to 8.3 gave the right message:

A problem occurred evaluating project ':suite-base'.
> Could not create an instance of type org.gradle.api.plugins.jvm.internal.DefaultJvmTestSuite.
   > Cannot add a SourceSet with name 'integrationTest' as a SourceSet with that name already exists.

I’ll just try to open up a issue that this should not fail, that would also dissolve my concern about the sequential dependency.

Suites and fixture are heavily related to one another, but from the build perspective they look rather different, it’s just too convenient to handle them in one place like the test-sets plugin did.

kind regards
Daniel

Switching from Gradle 7.6 to 8.3 gave the right message

Yeah, in some situations, these error messages were improved, but they still come too often when using Groovy DSL.

I’ll just try to open up a issue that this should not fail, that would also dissolve my concern about the sequential dependency.

But it has to fail, that is not a bug.
You try to create the source set twice, one time directly, one time indirectly, and that should fail properly, which it does.
Feel free to open an issue if you have a different opinion, but don’t expect this to be changed. :slight_smile:

Reading this thread already helped me a lot, but I’m still not able to create a build.gradle.kts, where the integrationTest suite is able to access the testFixtures.
If I understood correctly I should remove java-test-fixtures and create an explicit testFixtures source set, and add it to the integrationTest suite? Could somebody give a complete example? That would help me a lot.

Here’s what I got so far:

plugins {
  java
  `java-test-fixtures`
  `jvm-test-suite`
  idea
  id("org.springframework.boot") version "3.1.5"
  id("io.spring.dependency-management") version "1.1.4"
}

java { sourceCompatibility = JavaVersion.VERSION_17 }

repositories { mavenCentral() }

dependencies {
  implementation("org.springframework.boot:spring-boot-starter-webflux")
  implementation("org.springframework.boot:spring-boot-starter-validation")

  testFixturesImplementation("org.apache.commons:commons-lang3:3.13.0")
  testFixturesImplementation("org.apache.commons:commons-rng-simple:1.5")
}

testing {
  suites {
    withType(JvmTestSuite::class).configureEach {
      useJUnitJupiter()
      dependencies {
        implementation("org.junit.jupiter:junit-jupiter-api")
        implementation("org.assertj:assertj-core")
        runtimeOnly("org.junit.platform:junit-platform-launcher")
        runtimeOnly("org.junit.jupiter:junit-jupiter-engine")
      }
    }
    val test by
      getting(JvmTestSuite::class) {
        useJUnitJupiter()
        dependencies {
          implementation(project())
          implementation("org.junit.jupiter:junit-jupiter-params")
        }
      }

    val integrationTest by
      register<JvmTestSuite>("integrationTest") {
        useJUnitJupiter()
        testType.set(TestSuiteType.INTEGRATION_TEST)
        dependencies {
          implementation(project())
          implementation("org.springframework.boot:spring-boot-starter-test")
          implementation("org.springframework.boot:spring-boot-starter-webflux")
          runtimeOnly("io.netty:netty-resolver-dns-native-macos:4.1.101.Final:osx-aarch_64")
        }
        targets { all { testTask.configure { shouldRunAfter(test) } } }
      }
  }
}

idea { module { testSources.from(sourceSets["integrationTest"].java.sourceDirectories) } }

tasks.check { dependsOn(testing.suites.named("integrationTest")) }

Besides that you should stop using the Spring Dependency Management plugin (it is an obsolete relict from times when Gradle did not have integrated BOM support, by now does more harm than good, and even its maintainer recommends not to use it anymore), what you want should be as simple as implementation(testFixtures(project())) in the dependencies of the integrationTest suite.