How do 'jvm-test-suite' and 'java-test-fixtures' relate to each other and how can they be used cross project

We’re currently using the GitHub - unbroken-dome/gradle-testsets-plugin: A plugin for the Gradle build system that allows specifying test sets (like integration or acceptance tests). to configure integrationTests.
These sets have 2 purposes:

  • main reason is of cause containing tests for the component that require special setup and need to be executed separately, therefor they also need to be packaged and be consumable by an ear project within the same build
  • in some instances however they serve the purpose of containing testFixtures that will be consumed by other projects, either in the same multiproject build of even by other standalone builds

With the rise of the jvm-test-suite plugin I thought any test-set related use case should now be possible with this plugin. However there are some questions I could not find a good answer for in the documentation:

  • the java-test-fixtures provided a special dependency feature to be consumed by other projects in the same build, but I can’t seem to find out how this should look like for jvm-test-suite suites
dependencies {
    implementation(project(":lib"))

    testImplementation 'junit:junit:4.13'
    testImplementation(testFixtures(project(":lib"))) // how to declare this for a test-suite?
}
  • test-fixtures-plugin also automatically added a software variant that was distributed and could easily be consumed by other builds, I can’t seem to find anything like that for jvm-test-suites or did I miss something?
  • is it even possible to define a (unit)testFixture with jvm-test-suites (seems to be the question here: Jvm-test-suite - create common test sources for use in multiple test suites)?

So overall I am kind of confused about the relationship or the future of the java-test-fixtures vs jvm-test-suites plugins.
And the thing that matters the most to me how should we set up proper dependencies on jvm-test-suites?

Maybe it’s not the time yet to apply the test suites in my scenario (requiring unitTestFixtures, unitTests, integrationTestFixtures and integrationTests) but I was actually hoping to be able to achieve that already without depending on external plugins (or having to deal with a whole lot of configuration).

kind regards
Daniel

Reading through the docs I found various articles all stating how additional test suites may be configured all going into different kind of details and having another pro and cons.

Testing in Java & JVM projects seems to be there for the longest time. But also looks rather rough compared to the other options.

I found that to be helpful even though it’s just missing the distribution part:
https://docs.gradle.org/7.6/samples/sample_jvm_multi_project_with_additional_test_types.html

Looking into testFixtures I could find everything I was looking for (simple declaration, configuration and distribution of artifacts), just that it’s limited to that specific use-case and can not be extrapolated to be useable for integrationTests

The newest addition is obviously the jvm-test-suite:
https://docs.gradle.org/7.6/userguide/jvm_test_suite_plugin.html#jvm_test_suite_plugin
but the concept is unclear to me (having multiple targets and arguing to not use the global dependencies block anymore is obviously preparing smth. but I don’t understand what and it got no significant updates since it’s introduction, right?) and most important for me i- t’s missing the artifact building and distribution feature.

From the textFixture plugin I got the idea of using feature variants:
https://docs.gradle.org/7.6/userguide/feature_variants.html

def integrationTest = sourceSets.create('integrationTest')
java {
    registerFeature('integrationTest') {
        usingSourceSet(integrationTest)
    }
}

this works fine in tandem with the “Using additional test types Sample” approach and I can easily depend on the code and artifacts:

dependencies {
    testImplementation(project(':feature-base')) {
        capabilities {
            requireCapability("com.example:feature-base-integration-test")
        }
    }
}

It was a bit difficult to match that with the jvm test-suite as I could not define the SourceSet for the feature registration, but had to wait after the definition of the testSuite for it to be there…
This breaks:

def integrationTest = sourceSets.create('integrationTest')
testing {
    suites {
        integrationTest(JvmTestSuite) {}
    }
}
// ends up throwing:
/*
Could not find method call() for arguments [interface org.gradle.api.plugins.jvm.JvmTestSuite, build_bp1rs6g1bwg6xxynjkqddzgl9$_run_closure1$_closure4$_closure5@2b2b926c] on source set 'integration test' of type org.gradle.api.internal.tasks.DefaultSourceSet.
*/
Stacktrace
  • Exception is:
    org.gradle.api.GradleScriptException: A problem occurred evaluating project ‘:suite-base’.
    at org.gradle.groovy.scripts.internal.DefaultScriptRunnerFactory$ScriptRunnerImpl.run(DefaultScriptRunnerFactory.java:93)
    at org.gradle.configuration.DefaultScriptPluginFactory$ScriptPluginImpl.lambda$apply$0(DefaultScriptPluginFactory.java:133)
    at org.gradle.configuration.ProjectScriptTarget.addConfiguration(ProjectScriptTarget.java:79)
    at org.gradle.configuration.DefaultScriptPluginFactory$ScriptPluginImpl.apply(DefaultScriptPluginFactory.java:136)
    at org.gradle.configuration.BuildOperationScriptPlugin$1.run(BuildOperationScriptPlugin.java:65)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:29)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:26)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.run(DefaultBuildOperationRunner.java:47)
    at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:68)
    at org.gradle.configuration.BuildOperationScriptPlugin.lambda$apply$0(BuildOperationScriptPlugin.java:62)
    at org.gradle.configuration.internal.DefaultUserCodeApplicationContext.apply(DefaultUserCodeApplicationContext.java:44)
    at org.gradle.configuration.BuildOperationScriptPlugin.apply(BuildOperationScriptPlugin.java:62)
    at org.gradle.api.internal.project.DefaultProjectStateRegistry$ProjectStateImpl.lambda$applyToMutableState$0(DefaultProjectStateRegistry.java:360)
    at org.gradle.api.internal.project.DefaultProjectStateRegistry$ProjectStateImpl.fromMutableState(DefaultProjectStateRegistry.java:378)
    at org.gradle.api.internal.project.DefaultProjectStateRegistry$ProjectStateImpl.applyToMutableState(DefaultProjectStateRegistry.java:359)
    at org.gradle.configuration.project.BuildScriptProcessor.execute(BuildScriptProcessor.java:42)
    at org.gradle.configuration.project.BuildScriptProcessor.execute(BuildScriptProcessor.java:26)
    at org.gradle.configuration.project.ConfigureActionsProjectEvaluator.evaluate(ConfigureActionsProjectEvaluator.java:35)
    at org.gradle.configuration.project.LifecycleProjectEvaluator$EvaluateProject.lambda$run$0(LifecycleProjectEvaluator.java:109)
    at org.gradle.api.internal.project.DefaultProjectStateRegistry$ProjectStateImpl.lambda$applyToMutableState$0(DefaultProjectStateRegistry.java:360)
    at org.gradle.api.internal.project.DefaultProjectStateRegistry$ProjectStateImpl.lambda$fromMutableState$1(DefaultProjectStateRegistry.java:383)
    at org.gradle.internal.work.DefaultWorkerLeaseService.withReplacedLocks(DefaultWorkerLeaseService.java:345)
    at org.gradle.api.internal.project.DefaultProjectStateRegistry$ProjectStateImpl.fromMutableState(DefaultProjectStateRegistry.java:383)
    at org.gradle.api.internal.project.DefaultProjectStateRegistry$ProjectStateImpl.applyToMutableState(DefaultProjectStateRegistry.java:359)
    at org.gradle.configuration.project.LifecycleProjectEvaluator$EvaluateProject.run(LifecycleProjectEvaluator.java:100)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:29)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:26)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.run(DefaultBuildOperationRunner.java:47)
    at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:68)
    at org.gradle.configuration.project.LifecycleProjectEvaluator.evaluate(LifecycleProjectEvaluator.java:72)
    at org.gradle.api.internal.project.DefaultProject.evaluate(DefaultProject.java:762)
    at org.gradle.api.internal.project.DefaultProject.evaluate(DefaultProject.java:153)
    at org.gradle.api.internal.project.ProjectLifecycleController.lambda$ensureSelfConfigured$1(ProjectLifecycleController.java:63)
    at org.gradle.internal.model.StateTransitionController.lambda$doTransition$12(StateTransitionController.java:236)
    at org.gradle.internal.model.StateTransitionController.doTransition(StateTransitionController.java:247)
    at org.gradle.internal.model.StateTransitionController.doTransition(StateTransitionController.java:235)
    at org.gradle.internal.model.StateTransitionController.lambda$maybeTransitionIfNotCurrentlyTransitioning$9(StateTransitionController.java:196)
    at org.gradle.internal.work.DefaultSynchronizer.withLock(DefaultSynchronizer.java:34)
    at org.gradle.internal.model.StateTransitionController.maybeTransitionIfNotCurrentlyTransitioning(StateTransitionController.java:192)
    at org.gradle.api.internal.project.ProjectLifecycleController.ensureSelfConfigured(ProjectLifecycleController.java:63)
    at org.gradle.api.internal.project.DefaultProjectStateRegistry$ProjectStateImpl.ensureConfigured(DefaultProjectStateRegistry.java:334)
    at org.gradle.execution.TaskPathProjectEvaluator.configure(TaskPathProjectEvaluator.java:33)
    at org.gradle.execution.TaskPathProjectEvaluator.configureHierarchy(TaskPathProjectEvaluator.java:49)
    at org.gradle.configuration.DefaultProjectsPreparer.prepareProjects(DefaultProjectsPreparer.java:50)
    at org.gradle.configuration.BuildTreePreparingProjectsPreparer.prepareProjects(BuildTreePreparingProjectsPreparer.java:64)
    at org.gradle.configuration.BuildOperationFiringProjectsPreparer$ConfigureBuild.run(BuildOperationFiringProjectsPreparer.java:52)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:29)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:26)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.run(DefaultBuildOperationRunner.java:47)
    at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:68)
    at org.gradle.configuration.BuildOperationFiringProjectsPreparer.prepareProjects(BuildOperationFiringProjectsPreparer.java:40)
    at org.gradle.initialization.VintageBuildModelController.lambda$prepareProjects$2(VintageBuildModelController.java:84)
    at org.gradle.internal.model.StateTransitionController.lambda$doTransition$12(StateTransitionController.java:236)
    at org.gradle.internal.model.StateTransitionController.doTransition(StateTransitionController.java:247)
    at org.gradle.internal.model.StateTransitionController.doTransition(StateTransitionController.java:235)
    at org.gradle.internal.model.StateTransitionController.lambda$transitionIfNotPreviously$10(StateTransitionController.java:210)
    at org.gradle.internal.work.DefaultSynchronizer.withLock(DefaultSynchronizer.java:34)
    at org.gradle.internal.model.StateTransitionController.transitionIfNotPreviously(StateTransitionController.java:206)
    at org.gradle.initialization.VintageBuildModelController.prepareProjects(VintageBuildModelController.java:84)
    at org.gradle.initialization.VintageBuildModelController.getConfiguredModel(VintageBuildModelController.java:64)
    at org.gradle.internal.build.DefaultBuildLifecycleController.lambda$withProjectsConfigured$1(DefaultBuildLifecycleController.java:116)
    at org.gradle.internal.model.StateTransitionController.lambda$notInState$3(StateTransitionController.java:143)
    at org.gradle.internal.work.DefaultSynchronizer.withLock(DefaultSynchronizer.java:44)
    at org.gradle.internal.model.StateTransitionController.notInState(StateTransitionController.java:139)
    at org.gradle.internal.build.DefaultBuildLifecycleController.withProjectsConfigured(DefaultBuildLifecycleController.java:116)
    at org.gradle.internal.build.DefaultBuildToolingModelController.locateBuilderForTarget(DefaultBuildToolingModelController.java:57)
    at org.gradle.internal.buildtree.DefaultBuildTreeModelCreator$DefaultBuildTreeModelController.lambda$locateBuilderForTarget$0(DefaultBuildTreeModelCreator.java:73)
    at org.gradle.internal.build.DefaultBuildLifecycleController.withToolingModels(DefaultBuildLifecycleController.java:180)
    at org.gradle.internal.build.AbstractBuildState.withToolingModels(AbstractBuildState.java:123)
    at org.gradle.internal.buildtree.DefaultBuildTreeModelCreator$DefaultBuildTreeModelController.locateBuilderForTarget(DefaultBuildTreeModelCreator.java:73)
    at org.gradle.internal.buildtree.DefaultBuildTreeModelCreator$DefaultBuildTreeModelController.locateBuilderForDefaultTarget(DefaultBuildTreeModelCreator.java:68)
    at org.gradle.tooling.internal.provider.runner.DefaultBuildController.getTarget(DefaultBuildController.java:157)
    at org.gradle.tooling.internal.provider.runner.DefaultBuildController.getModel(DefaultBuildController.java:101)
    at org.gradle.tooling.internal.consumer.connection.ParameterAwareBuildControllerAdapter.getModel(ParameterAwareBuildControllerAdapter.java:39)
    at org.gradle.tooling.internal.consumer.connection.UnparameterizedBuildController.getModel(UnparameterizedBuildController.java:113)
    at org.gradle.tooling.internal.consumer.connection.NestedActionAwareBuildControllerAdapter.getModel(NestedActionAwareBuildControllerAdapter.java:31)
    at org.gradle.tooling.internal.consumer.connection.UnparameterizedBuildController.findModel(UnparameterizedBuildController.java:97)
    at org.gradle.tooling.internal.consumer.connection.NestedActionAwareBuildControllerAdapter.findModel(NestedActionAwareBuildControllerAdapter.java:31)
    at org.gradle.tooling.internal.consumer.connection.UnparameterizedBuildController.findModel(UnparameterizedBuildController.java:81)
    at org.gradle.tooling.internal.consumer.connection.NestedActionAwareBuildControllerAdapter.findModel(NestedActionAwareBuildControllerAdapter.java:31)
    at org.gradle.tooling.internal.consumer.connection.UnparameterizedBuildController.findModel(UnparameterizedBuildController.java:66)
    at org.gradle.tooling.internal.consumer.connection.NestedActionAwareBuildControllerAdapter.findModel(NestedActionAwareBuildControllerAdapter.java:31)
    at org.jetbrains.plugins.gradle.model.ProjectImportAction.execute(ProjectImportAction.java:125)
    at org.jetbrains.plugins.gradle.model.ProjectImportAction.execute(ProjectImportAction.java:42)
    at org.gradle.tooling.internal.consumer.connection.InternalBuildActionAdapter.execute(InternalBuildActionAdapter.java:64)
    at org.gradle.tooling.internal.provider.runner.AbstractClientProvidedBuildActionRunner$ActionAdapter.runAction(AbstractClientProvidedBuildActionRunner.java:131)
    at org.gradle.tooling.internal.provider.runner.AbstractClientProvidedBuildActionRunner$ActionAdapter.beforeTasks(AbstractClientProvidedBuildActionRunner.java:99)
    at org.gradle.internal.buildtree.DefaultBuildTreeModelCreator.beforeTasks(DefaultBuildTreeModelCreator.java:52)
    at org.gradle.internal.buildtree.DefaultBuildTreeLifecycleController.lambda$fromBuildModel$2(DefaultBuildTreeLifecycleController.java:82)
    at org.gradle.internal.buildtree.DefaultBuildTreeLifecycleController.lambda$runBuild$5(DefaultBuildTreeLifecycleController.java:113)
    at org.gradle.internal.model.StateTransitionController.lambda$transition$5(StateTransitionController.java:166)
    at org.gradle.internal.model.StateTransitionController.doTransition(StateTransitionController.java:247)
    at org.gradle.internal.model.StateTransitionController.lambda$transition$6(StateTransitionController.java:166)
    at org.gradle.internal.work.DefaultSynchronizer.withLock(DefaultSynchronizer.java:44)
    at org.gradle.internal.model.StateTransitionController.transition(StateTransitionController.java:166)
    at org.gradle.internal.buildtree.DefaultBuildTreeLifecycleController.runBuild(DefaultBuildTreeLifecycleController.java:110)
    at org.gradle.internal.buildtree.DefaultBuildTreeLifecycleController.fromBuildModel(DefaultBuildTreeLifecycleController.java:81)
    at org.gradle.tooling.internal.provider.runner.AbstractClientProvidedBuildActionRunner.runClientAction(AbstractClientProvidedBuildActionRunner.java:43)
    at org.gradle.tooling.internal.provider.runner.ClientProvidedPhasedActionRunner.run(ClientProvidedPhasedActionRunner.java:53)
    at org.gradle.launcher.exec.ChainingBuildActionRunner.run(ChainingBuildActionRunner.java:35)
    at org.gradle.internal.buildtree.ProblemReportingBuildActionRunner.run(ProblemReportingBuildActionRunner.java:49)
    at org.gradle.launcher.exec.BuildOutcomeReportingBuildActionRunner.run(BuildOutcomeReportingBuildActionRunner.java:65)
    at org.gradle.tooling.internal.provider.FileSystemWatchingBuildActionRunner.run(FileSystemWatchingBuildActionRunner.java:136)
    at org.gradle.launcher.exec.BuildCompletionNotifyingBuildActionRunner.run(BuildCompletionNotifyingBuildActionRunner.java:41)
    at org.gradle.launcher.exec.RootBuildLifecycleBuildActionExecutor.lambda$execute$0(RootBuildLifecycleBuildActionExecutor.java:40)
    at org.gradle.composite.internal.DefaultRootBuildState.run(DefaultRootBuildState.java:122)
    at org.gradle.launcher.exec.RootBuildLifecycleBuildActionExecutor.execute(RootBuildLifecycleBuildActionExecutor.java:40)
    at org.gradle.internal.buildtree.DefaultBuildTreeContext.execute(DefaultBuildTreeContext.java:40)
    at org.gradle.launcher.exec.BuildTreeLifecycleBuildActionExecutor.lambda$execute$0(BuildTreeLifecycleBuildActionExecutor.java:65)
    at org.gradle.internal.buildtree.BuildTreeState.run(BuildTreeState.java:53)
    at org.gradle.launcher.exec.BuildTreeLifecycleBuildActionExecutor.execute(BuildTreeLifecycleBuildActionExecutor.java:65)
    at org.gradle.launcher.exec.RunAsBuildOperationBuildActionExecutor$3.call(RunAsBuildOperationBuildActionExecutor.java:61)
    at org.gradle.launcher.exec.RunAsBuildOperationBuildActionExecutor$3.call(RunAsBuildOperationBuildActionExecutor.java:57)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)
    at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73)
    at org.gradle.launcher.exec.RunAsBuildOperationBuildActionExecutor.execute(RunAsBuildOperationBuildActionExecutor.java:57)
    at org.gradle.launcher.exec.RunAsWorkerThreadBuildActionExecutor.lambda$execute$0(RunAsWorkerThreadBuildActionExecutor.java:36)
    at org.gradle.internal.work.DefaultWorkerLeaseService.withLocks(DefaultWorkerLeaseService.java:249)
    at org.gradle.internal.work.DefaultWorkerLeaseService.runAsWorkerThread(DefaultWorkerLeaseService.java:109)
    at org.gradle.launcher.exec.RunAsWorkerThreadBuildActionExecutor.execute(RunAsWorkerThreadBuildActionExecutor.java:36)
    at org.gradle.tooling.internal.provider.continuous.ContinuousBuildActionExecutor.execute(ContinuousBuildActionExecutor.java:110)
    at org.gradle.tooling.internal.provider.SubscribableBuildActionExecutor.execute(SubscribableBuildActionExecutor.java:64)
    at org.gradle.internal.session.DefaultBuildSessionContext.execute(DefaultBuildSessionContext.java:46)
    at org.gradle.tooling.internal.provider.BuildSessionLifecycleBuildActionExecuter$ActionImpl.apply(BuildSessionLifecycleBuildActionExecuter.java:100)
    at org.gradle.tooling.internal.provider.BuildSessionLifecycleBuildActionExecuter$ActionImpl.apply(BuildSessionLifecycleBuildActionExecuter.java:88)
    at org.gradle.internal.session.BuildSessionState.run(BuildSessionState.java:69)
    at org.gradle.tooling.internal.provider.BuildSessionLifecycleBuildActionExecuter.execute(BuildSessionLifecycleBuildActionExecuter.java:62)
    at org.gradle.tooling.internal.provider.BuildSessionLifecycleBuildActionExecuter.execute(BuildSessionLifecycleBuildActionExecuter.java:41)
    at org.gradle.tooling.internal.provider.StartParamsValidatingActionExecuter.execute(StartParamsValidatingActionExecuter.java:63)
    at org.gradle.tooling.internal.provider.StartParamsValidatingActionExecuter.execute(StartParamsValidatingActionExecuter.java:31)
    at org.gradle.tooling.internal.provider.SessionFailureReportingActionExecuter.execute(SessionFailureReportingActionExecuter.java:52)
    at org.gradle.tooling.internal.provider.SessionFailureReportingActionExecuter.execute(SessionFailureReportingActionExecuter.java:40)
    at org.gradle.tooling.internal.provider.SetupLoggingActionExecuter.execute(SetupLoggingActionExecuter.java:47)
    at org.gradle.tooling.internal.provider.SetupLoggingActionExecuter.execute(SetupLoggingActionExecuter.java:31)
    at org.gradle.launcher.daemon.server.exec.ExecuteBuild.doBuild(ExecuteBuild.java:65)
    at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:37)
    at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
    at org.gradle.launcher.daemon.server.exec.WatchForDisconnection.execute(WatchForDisconnection.java:39)
    at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
    at org.gradle.launcher.daemon.server.exec.ResetDeprecationLogger.execute(ResetDeprecationLogger.java:29)
    at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
    at org.gradle.launcher.daemon.server.exec.RequestStopIfSingleUsedDaemon.execute(RequestStopIfSingleUsedDaemon.java:35)
    at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
    at org.gradle.launcher.daemon.server.exec.ForwardClientInput$2.create(ForwardClientInput.java:78)
    at org.gradle.launcher.daemon.server.exec.ForwardClientInput$2.create(ForwardClientInput.java:75)
    at org.gradle.util.internal.Swapper.swap(Swapper.java:38)
    at org.gradle.launcher.daemon.server.exec.ForwardClientInput.execute(ForwardClientInput.java:75)
    at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
    at org.gradle.launcher.daemon.server.exec.LogAndCheckHealth.execute(LogAndCheckHealth.java:55)
    at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
    at org.gradle.launcher.daemon.server.exec.LogToClient.doBuild(LogToClient.java:63)
    at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:37)
    at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
    at org.gradle.launcher.daemon.server.exec.EstablishBuildEnvironment.doBuild(EstablishBuildEnvironment.java:84)
    at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:37)
    at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
    at org.gradle.launcher.daemon.server.exec.StartBuildOrRespondWithBusy$1.run(StartBuildOrRespondWithBusy.java:52)
    at org.gradle.launcher.daemon.server.DaemonStateCoordinator$1.run(DaemonStateCoordinator.java:297)
    at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
    at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
    Caused by: org.gradle.internal.metaobject.AbstractDynamicObject$CustomMessageMissingMethodException: Could not find method call() for arguments [interface org.gradle.api.plugins.jvm.JvmTestSuite, build_bp1rs6g1bwg6xxynjkqddzgl9$_run_closure1$_closure4$_closure5@2b2b926c] on source set ‘integration test’ of type org.gradle.api.internal.tasks.DefaultSourceSet.
    at org.gradle.internal.metaobject.AbstractDynamicObject$CustomMissingMethodExecutionFailed.(AbstractDynamicObject.java:190)
    at org.gradle.internal.metaobject.AbstractDynamicObject.methodMissingException(AbstractDynamicObject.java:184)
    at org.gradle.internal.metaobject.AbstractDynamicObject.invokeMethod(AbstractDynamicObject.java:167)
    at org.gradle.api.internal.tasks.DefaultSourceSet_Decorated.invokeMethod(Unknown Source)
    at build_bp1rs6g1bwg6xxynjkqddzgl9$_run_closure1$_closure4.doCall(C:\EDF\sources\Temp\reproducer-it\suite-base\build.gradle:10)
    at org.gradle.util.internal.ClosureBackedAction.execute(ClosureBackedAction.java:73)
    at org.gradle.util.internal.ConfigureUtil.configureTarget(ConfigureUtil.java:155)
    at org.gradle.util.internal.ConfigureUtil.configureSelf(ConfigureUtil.java:143)
    at org.gradle.api.internal.AbstractNamedDomainObjectContainer.configure(AbstractNamedDomainObjectContainer.java:91)
    at org.gradle.api.internal.AbstractNamedDomainObjectContainer.configure(AbstractNamedDomainObjectContainer.java:38)
    at org.gradle.internal.extensibility.MixInClosurePropertiesAsMethodsDynamicObject.tryInvokeMethod(MixInClosurePropertiesAsMethodsDynamicObject.java:55)
    at org.gradle.internal.metaobject.ConfigureDelegate.invokeMethod(ConfigureDelegate.java:57)
    at build_bp1rs6g1bwg6xxynjkqddzgl9$_run_closure1.doCall(C:\EDF\sources\Temp\reproducer-it\suite-base\build.gradle:9)
    at org.gradle.util.internal.ClosureBackedAction.execute(ClosureBackedAction.java:73)
    at org.gradle.util.internal.ConfigureUtil.configureTarget(ConfigureUtil.java:155)
    at org.gradle.util.internal.ConfigureUtil.configure(ConfigureUtil.java:106)
    at org.gradle.util.internal.ConfigureUtil$WrappedConfigureAction.execute(ConfigureUtil.java:167)
    at org.gradle.internal.extensibility.ExtensionsStorage$ExtensionHolder.configure(ExtensionsStorage.java:173)
    at org.gradle.internal.extensibility.ExtensionsStorage.configureExtension(ExtensionsStorage.java:64)
    at org.gradle.internal.extensibility.DefaultConvention.configureExtension(DefaultConvention.java:364)
    at org.gradle.internal.extensibility.DefaultConvention.access$500(DefaultConvention.java:45)
    at org.gradle.internal.extensibility.DefaultConvention$ExtensionsDynamicObject.tryInvokeMethod(DefaultConvention.java:301)
    at org.gradle.internal.metaobject.CompositeDynamicObject.tryInvokeMethod(CompositeDynamicObject.java:98)
    at org.gradle.internal.extensibility.MixInClosurePropertiesAsMethodsDynamicObject.tryInvokeMethod(MixInClosurePropertiesAsMethodsDynamicObject.java:36)
    at org.gradle.groovy.scripts.BasicScript$ScriptDynamicObject.tryInvokeMethod(BasicScript.java:135)
    at org.gradle.internal.metaobject.AbstractDynamicObject.invokeMethod(AbstractDynamicObject.java:163)
    at org.gradle.groovy.scripts.BasicScript.invokeMethod(BasicScript.java:84)
    at build_bp1rs6g1bwg6xxynjkqddzgl9.run(C:\EDF\sources\Temp\reproducer-it\suite-base\build.gradle:8)
    at org.gradle.groovy.scripts.internal.DefaultScriptRunnerFactory$ScriptRunnerImpl.run(DefaultScriptRunnerFactory.java:91)
    … 177 more

If one does it like this it’s working but having to declare them in order does not math the lazy configuration goals, where a order is not guaranteed, right?

testing {
    suites {
        integrationTest(JvmTestSuite) { /*...*/ }
    }
}
java {
    registerFeature('integrationTest') {
        usingSourceSet(sourceSets.named('integrationTest').get())
    }
}

I’m affraid this ends up breaking at some point.

So the question is, what’s the gradle way of doing it and do you maybe want to update the documentation to provide a crystal clear view on the approaches that will be future proof?

kind regards
Daniel

Not everything is order independent.
Plugins should indeed be written order-independently, so that they either react to another plugin being applied in the past or future, or applying the plugin directly if it is always needed.

But the statements within one build script always were and always will be order sensitive.
For example you first have to create a configuration, before you can declare dependencies on it.
Or you first have to create a source set before you can declare a feature from it.
And so on.

  • the java-test-fixtures provided a special dependency feature to be consumed by other projects in the same build

Not only in the same build, test fixtures by default are also published, so you can also depend on them from outside the current build if you don’t turn it off.

but I can’t seem to find out how this should look like for jvm-test-suite suites

One has nothing to do with the other.
The test fixtures feature provides a mean to provide some classes - typically test fixtures or test utilities - that can be consumed by the same project, or by other projects inside or outside the current build within their tests.
The jvm-test-suite plugin is to have multiple test suites like unit tests, integ tests, functional tests, load tests, that are clearly separated.
They are by default not meant to be depended upon, but to contain tests that are executed.
But if you insist, you can of course define feature variants for those source sets as you already discovered and then depend on them.

So overall I am kind of confused about the relationship or the future of the java-test-fixtures vs jvm-test-suites plugins.

As I said, they are two completely different things with completely different purposes that can nicely work together though, and that are most probably both are there to stay.

Maybe it’s not the time yet to apply the test suites in my scenario (requiring unitTestFixtures, unitTests, integrationTestFixtures and integrationTests) but I was actually hoping to be able to achieve that already without depending on external plugins (or having to deal with a whole lot of configuration).

The test fixtures plugin only provides one test fixture per project.
But it is actually just some convenience sugar for defining a feature variant, so you can pretty easily do it yourself.
Just create a source set for your additional fixtures (I wouldn’t use a test suite for that, as test suites are meant to contain executable tests and get tasks for executing them and so on, not to provide test fixtures).
Then declare a feature variant from that source set.
And on the consumer side, depend on the capability of that feature when needing the fixtures.

So the question is, what’s the gradle way of doing it

As I described above, you practically found the Gradle way already, except maybe with a separate source set, not the test suite. Except you also have tests in there that you want to execute in your project while still depending on them from somewhere else, but that feels a bit like a mix of responsibilities. (Hence by default the separate plugins for test fixtures and test suites)

and do you maybe want to update the documentation to provide a crystal clear view on the approaches that will be future proof?

If you feel it is not clear enough, you should open an issue or pull request over on GitHub, so that the Gradle folks can consider it. This is a community forum where mostly users like me are helping other users.

It might also be worth opening a feature request so that the test fixtures plugin could create one fixture feature per test suite and not only one for the default test suite, either always or configurable.

having multiple targets

Unfortunately, that is actually not yet possible but something to come. Currently you can only have one target per suite.

and arguing to not use the global dependencies block anymore

I wouldn’t call it arguing, you can still use the global dependencies block.
It is mainly syntax sugar and help for the Kotlin DSL.
Because in the test suite dependencies block, you can use implementation, runtimeOnly, and so on.
In the global dependencies block you would need to use integrationTestImplementation, integrationTestRuntimeOnly, and so on.
And as those configurations are not created by applying a plugin, but by additional configuration of the plugin in the build script, there are for example no type-safe accessors for those generated for Kotlin DSL, so you would first to get them by name for Kotlin DSL or use the string-y version "integrationTestImplementation"("...") which both is too cumbersome when you can just do it in the test suite dependencies block.

and it got no significant updates since it’s introduction, right?

Depends on what you call significant :smiley:

and most important for me i- t’s missing the artifact building and distribution feature.

Well, as described above, that is not “missing” but just not the right responsibility by default, which you can realtively easily change. :slight_smile:


Btw., this:

Could not find method call() for arguments [interface org.gradle.api.plugins.jvm.JvmTestSuite, build_bp1rs6g1bwg6xxynjkqddzgl9$_run_closure1$_closure4$_closure5@2b2b926c] on source set ‘integration test’ of type org.gradle.api.internal.tasks.DefaultSourceSet.

is one of the major reasons I recommend using the Kotlin DSL - which now also is the official default - instead of the Groovy DSL. You instantly get type-safe build scripts, actually helpful error messages if you mess up the syntax instead of unhelpul errors like the one you showed, and amazingly better IDE support if you use a proper IDE like IntelliJ.

1 Like

Thank you so much for the extensive answer.

I get that my requirements may be a bit off due to a very antique testing framework, where the integrationTests must be packaged and deployed in a testing application. That’s why the publishing is so important for me, but I get it may not be like that for everyone.

Not having to test the testFixtures was also a good point thanks for bringing this to my attention. Let’s see how much attention the feature request to support more than one fixture will get.

About the error message, yeah kotlin is on my todo list for way too long now, but in the end I’m using the groovy syntax just for prototyping. Switching from Gradle 7.6 to 8.3 gave the right message:

A problem occurred evaluating project ':suite-base'.
> Could not create an instance of type org.gradle.api.plugins.jvm.internal.DefaultJvmTestSuite.
   > Cannot add a SourceSet with name 'integrationTest' as a SourceSet with that name already exists.

I’ll just try to open up a issue that this should not fail, that would also dissolve my concern about the sequential dependency.

Suites and fixture are heavily related to one another, but from the build perspective they look rather different, it’s just too convenient to handle them in one place like the test-sets plugin did.

kind regards
Daniel

Switching from Gradle 7.6 to 8.3 gave the right message

Yeah, in some situations, these error messages were improved, but they still come too often when using Groovy DSL.

I’ll just try to open up a issue that this should not fail, that would also dissolve my concern about the sequential dependency.

But it has to fail, that is not a bug.
You try to create the source set twice, one time directly, one time indirectly, and that should fail properly, which it does.
Feel free to open an issue if you have a different opinion, but don’t expect this to be changed. :slight_smile:

Reading this thread already helped me a lot, but I’m still not able to create a build.gradle.kts, where the integrationTest suite is able to access the testFixtures.
If I understood correctly I should remove java-test-fixtures and create an explicit testFixtures source set, and add it to the integrationTest suite? Could somebody give a complete example? That would help me a lot.

Here’s what I got so far:

plugins {
  java
  `java-test-fixtures`
  `jvm-test-suite`
  idea
  id("org.springframework.boot") version "3.1.5"
  id("io.spring.dependency-management") version "1.1.4"
}

java { sourceCompatibility = JavaVersion.VERSION_17 }

repositories { mavenCentral() }

dependencies {
  implementation("org.springframework.boot:spring-boot-starter-webflux")
  implementation("org.springframework.boot:spring-boot-starter-validation")

  testFixturesImplementation("org.apache.commons:commons-lang3:3.13.0")
  testFixturesImplementation("org.apache.commons:commons-rng-simple:1.5")
}

testing {
  suites {
    withType(JvmTestSuite::class).configureEach {
      useJUnitJupiter()
      dependencies {
        implementation("org.junit.jupiter:junit-jupiter-api")
        implementation("org.assertj:assertj-core")
        runtimeOnly("org.junit.platform:junit-platform-launcher")
        runtimeOnly("org.junit.jupiter:junit-jupiter-engine")
      }
    }
    val test by
      getting(JvmTestSuite::class) {
        useJUnitJupiter()
        dependencies {
          implementation(project())
          implementation("org.junit.jupiter:junit-jupiter-params")
        }
      }

    val integrationTest by
      register<JvmTestSuite>("integrationTest") {
        useJUnitJupiter()
        testType.set(TestSuiteType.INTEGRATION_TEST)
        dependencies {
          implementation(project())
          implementation("org.springframework.boot:spring-boot-starter-test")
          implementation("org.springframework.boot:spring-boot-starter-webflux")
          runtimeOnly("io.netty:netty-resolver-dns-native-macos:4.1.101.Final:osx-aarch_64")
        }
        targets { all { testTask.configure { shouldRunAfter(test) } } }
      }
  }
}

idea { module { testSources.from(sourceSets["integrationTest"].java.sourceDirectories) } }

tasks.check { dependsOn(testing.suites.named("integrationTest")) }

Besides that you should stop using the Spring Dependency Management plugin (it is an obsolete relict from times when Gradle did not have integrated BOM support, by now does more harm than good, and even its maintainer recommends not to use it anymore), what you want should be as simple as implementation(testFixtures(project())) in the dependencies of the integrationTest suite.

I got stuck on a slightly simpler issue than the one the OP has. I’m just trying to get test suites to access test fixtures (without the extra twist of artifacts, multiple test fixtures, etc). Since even this discussion still leaves me scratching my head, I at least created a documentation issue in hopes of others not hitting the same unexpected speed bump: Explain how test suites can access test fixtures · Issue #29357 · gradle/gradle (github.com)

@Vampire when you said “what you want should be as simple as…” were you saying that the short snippet of code you gave there actually works in Gradle today? Or just saying that gradle really should be enhanced so that snippet would work?

I tried: (sorry, I’m using groovy)

dependencies {
    testImplementation testFixtures(project) // only applies to "test" task
}

testing {
  suites {
    configureEach {
      dependencies {
        implementation testFixtures(project) // I wish this worked for test suites
      }
    }
  }
}

but that resulted in:

Could not find method testFixtures() for arguments [project ':my.project'] on property 'dependencies' of type java.lang.Object.

sorry, I’m using groovy

Don’t apologize, you are just hurting yourself with that, that’s ok for me. :smiley:

Besides that it by now is the default DSL, you immediately get type-safe build scripts, and actually helpful error messages when you mess up the syntax, if you use a good IDE like IntelliJ IDEA or Android Studio you get an amazingly better IDE support that would probably have allowed you to simply resolve your problem already. :slight_smile:

were you saying that the short snippet of code you gave there actually works in Gradle today? Or just saying that gradle really should be enhanced so that snippet would work?

The former, this works perfectly fine.

I tried

Not really, you did not try what I gave you, but left out various characters, and too many of them. :wink:

Thanks for the reply @Vampire.

This is working for me:

dependencies {
    testImplementation testFixtures(project)
}

But in the test suites configuration this isn’t: (seems to do nothing at all)

      dependencies {
        implementation testFixtures(project) // I wish this worked for test suites
      }

These look pretty similar to me, so I’m confused as to which characters I left out. I’ve tried several variations of that code with different parens, etc, without success. I must need “dummy” level help.

(Side note: You’ve convinced me to work on switching to kotlin, but I’ve got >2500 lines of build script code for a complex project with a dozen modules, etc, and I haven’t even learned kotlin yet. I was hoping to get test suites working (with test fixtures) before porting everything to kotlin.)

(By the way, I single-handlely developed those 2500 lines of groovy. So the fact that I’m stuck on this issue should be noteworthy… I’m not a total newbie just expecting someone to hold my hand and write the code for me.)

implementation(testFixtures(project())) <== what I said
implementation testFixtures(project) <== what you do

@Vampire This is why it’s annoying that Gradle supports both groovy and kotlin… it creates confusion and chaos everywhere. I thought your example was in kotlin, since you’re advocating for that… so why would I copy/paste your code as-is when I’m using groovy? Nevertheless, I’ll go try that exact code and see what happens…

It is Kotlin.
And it is Groovy.

The problem is, that you use project which is the project instance, not project() which is a method call in the context of that dependencies block of the JVM test suites and gives a project dependency.

You can in Groovy do implementation testFixtures(project()) as it allows to omit the parentheses which makes it more DSLy in Groovy.
But you cannot omit just all method call parentheses.

@Vampire Ok I think I get it now. Thanks for your patience.

I realized what happened here… I was “onion peeling” through multiple issues when I was struggling with this originally, and I accidentally misinterpreted some error messages as being related to this code (when I first tried it using your snippet as-is), when actually they were related to a different thing.

Now I’m stuck on that different thing, which is related to using flatDir dependencies for test suites. The same syntax that I’m using for “testImplementation” is not working as “implementation” in the suite dependencies.

All I’m trying to do is create separate test tasks that run different combinations of JUnit5 tags. I was surprised that suites don’t just borrow their dependencies from the normal test dependencies by default. If there were some code that just allowed me to configure “use all the normal test dependencies” for a test suite, that would be very helpful.

Anyways… I’ll go see if I can solve the flatDir dependencies for test suites on my own…

All I’m trying to do is create separate test tasks that run different combinations of JUnit5 tags.

Then I’d say separate test suites are not what you want.
Just make additional test tasks configured from the main test suite and just with different tags.

I was surprised that suites don’t just borrow their dependencies from the normal test dependencies by default.

What are “normal test dependencies”?
The test test suite is just a normal test suite that is added by default.
There is not much special to it.
And you surely do not want to “borrow the dependencies” in most cases.
If you want to, just do it, e.g. by making the respective configurations of the additional test suites extend from the configurations of the test test suite.
But in most cases additional test suites are for example for functional tests, integration tests, system tests, and so on and those usually need much different dependencies.

If there were some code that just allowed me to configure “use all the normal test dependencies” for a test suite, that would be very helpful.

As mentioned above, make the configurations extend the ones from the test test suite.

Now I’m stuck on that different thing, which is related to using flatDir dependencies for test suites. The same syntax that I’m using for “testImplementation” is not working as “implementation” in the suite dependencies.

Do you really mean flatDir (which would be good), or files / fileTree (which would be bad)?
Because flatDir is in repositories section, so have nothing to do with any dependencies block and the dependencies you declare are just the same syntax as with remote repositories, so I’d wonder what syntax problem you could have.

@Vampire Yes I really mean flatDir. My old working code was using things like:

implementation name: 'javafx.base'

But for test suite dependencies, I needed this (which is probably better anyway):

implementation ':javafx.base'

Oh wow, I really made this harder than it needed to be then! Which is good, because after getting the dependencies working, I got stuck on accessing test resources from suites. I’m sure there’s some simple way to fix that, but I think I’ll try your suggestion instead, since it’s hugely simpler.

1 Like

implementation name: ‘javafx.base’

Oh, you were using the map-style, ok, that explains it. :smiley:
I didn’t use that syntax for many years, but always the string-style.

If you would really be in favor of the map-style, you could probably do

implementation(project.dependencies.create(name: 'javafx.base'))

but I agree that just using :javafx.base is simpler.


Btw., you know that the JavaFX libraries are published to Maven Central? And since version 0.1.0 of the JavaFX Gradle plugin they are also useable in a sensible way, even to do cross-platform builds, using Gradle feature variants. (And there is also a draft PR ongoing to natively have the variants published without the need for a separate plugin)

@Vampire Yup you’re right of course. I wish it were as easy as just using everything straight from maven central for our project, but we have the uncommon requirement that one must be able to build and run completely offline starting with only our git repo. I use maven’s “copy dependencies” plugin to grab most of the dependencies into a standard maven directory structure inside our git repo. For some reason my older code was using flatDir for JavaFX.
Anyways, thanks again for the awesome help here!

1 Like