Configure JVM Test Suite to build/run several suite depending on different artefacts (Multi-Project)

Hi,

I have two projects, A and B.
A has three sourceSets. The default one contains an API and the two additional sourceSets contain two different implementations of this API. It is declared as follow using builde.gradle :

plugins {
	id 'java'
}

sourceSets {
    impl1 {
        java { 
			srcDir "src/impl1/java/"
			compileClasspath += sourceSets.main.output
			runtimeClasspath += sourceSets.main.output
		}
    }
	impl2 {
		java {
			srcDir "src/impl2/java/"
			compileClasspath += sourceSets.main.output
			runtimeClasspath += sourceSets.main.output
		}
	}
}

task impl1(type: Jar) {
	from sourceSets.impl1.output
	archiveFileName = "A-impl1.jar"
}

task impl2(type: Jar) {
	from sourceSets.impl2.output
	archiveFileName = "A-impl2.jar"
}

jar.dependsOn impl1, impl2

It produces three jars (A.jar , A-impl1.jar and A-impl2.jar).

Enters the second project, B.
This project is meant to implement (and run) the tests. The tests relies on the API defined in the project A. I need to compile/run the tests for the two implementations defined in A, impl1 and impl2.

The idea is to use JVM Test Suite to define two suites : one using the implementation provided in A-impl1.jarand the other using the one provided byA-impl2.jar.

So far, I’ve defined the following build.gradle for project B:

plugins {
	id 'java'
	id 'jvm-test-suite'
}

testing {
	suites {
		configureEach {
			useJUnitJupiter()
			dependencies {
				implementation 'org.junit.jupiter:junit-jupiter:5.10.2'
				implementation 'org.junit.jupiter:junit-jupiter-api:5.10.2'
				implementation project(':A')
				runtimeOnly 'org.junit.jupiter:junit-jupiter-engine:5.10.2'
			}
		}

		impl1Test(JvmTestSuite) {
			dependencies {
				// How to manage runtimeOnly and/or implementation
			}
		}

		impl2Test(JvmTestSuite) {
			dependencies {
				// How to manage runtimeOnly and/or implementation
			}
		}
	}
}

// ... other convenience declarations (logging, task dependencies)

The thing is that I have hard time to specify the dependencies to the artifacts produces in project A (A-impl1.jar and A-impl2.jar).

The plugin dependencies section is a bit different than the ā€œregularā€ one. For example, I could add project-wise dependency in B as :

dependencies {
	implementation project(':A').sourceSets.impl2.output
}

In that case, I don’t even have to care about the produced artifacts. Everything compile and run BUT only for one implementation only.

If I try to set the same dependency in the test suites, I get errors as sourceSets is not recognized. Same goes for artifacts and such.

So, where am i wrong here ? Is there a way to refer to artifacts and/or sourceSets from another project ?

Thx.

There are ways.
And you should not even consider thinking about any of them, as it is an extremely bad idea.
If you want to use something from another project, you need to do it properly as shown on How to share outputs between projects

So for example create feature variants from your source sets, or at least outgoing configurations.

Or use separate projects instead of source sets. One for the API and one for each of the impls.

Thank you for your answer. I’ve dug the feature variants aspect, but I’m still stuck (skill issues).
Sorry for the long post to come.
I’ve made some changes in my projects gradle configurations.

The Project A still have three sourceSets. The main, where the API belongs, then impl1 and impl2 where the two differents implementations of the API reside.

The change in the build.gradle of Project A is that I define feature variants as follow:

// [... plugins and others ...]

group = 'org.xander.test'

// [... additional sourceSets impl1 and impl2 ...]

java {
	registerFeature('var1Impl') {
		usingSourceSet(sourceSets.impl1)
	}
	registerFeature('var2Impl') {
		usingSourceSet(sourceSets.impl2)
	}
}

So now, Project A produces and exposes two variants, ā€˜var1Impl’ and ā€˜var2Impl’.
The catch here is that implementations are in the same namespace (or package). Let say the API defines the interface org.xander.test.Main , then each implementation will define its own version of org.xander.test.MainImplimplementing Main.

No change since my first post, Project B is still dedicated to tests. It contains only one source sets and relies on the API defined in Project A and implementations. For example, at one point in the tests, we will have that kind of instanciation : Main toTest = new MainImpl();

It’s pretty easy to choose one of the implementations by using the following gradle configuration for Project B:

plugins {
	id 'java'
}

group='org.xander.test'

def selectedImpl = project.hasProperty("impl") ? project.impl : "var1"

dependencies {
	// .. classic JUnit dependencies
	implementation project(':ProjectA') // could be testImplementation
	testImplementation(project(":ProjectA")) {
		capabilities {
			requireCapability("org.xander.test:ProjectA-${selectedImpl}-impl")
		}
	}
}

I can choose one of the implementation by running the command line : ./gradlew build -Pimpl=varX

Unfortunately, it’s not what I want. I’d like to be able to run the test for the two variants, var1 and var2 in a single gradle invocation.

If I simply define two test tasks (one for each variant), I obviously end with a classpath containing both the implementations (as it resolves the dependencies), and as my implementations occurs in the same package names, the ā€˜first’ implementation will be choosed.

I’ve tried to really separate the dependencies, but i’ve stubbled on the ā€˜main’ case that required something to be done, so i’ve done something utterly wrong that doesn’t work anyway ( compileTestJava.onlyIf{ false }).

So here is my last attempt : two test tasks, with two separate dependencies. But it seems to resolve the dependencies by merging those of the to tasks …

task var2Test(type: Test) {
	dependencies {
		testImplementation(project(":ProjectA"))
		testImplementation(project(":ProjectA")) {
			capabilities {
				requireCapability("org.xander.test:ProjectA-var2-impl")
			}
		}
	}
}

task var1Test(type: Test) {
	dependencies {
		testImplementation(project(":ProjectA"))
		testImplementation(project(":ProjectA")) {
			capabilities {
				requireCapability("org.xander.test:ProjectA-var1-impl")
			}
		}
	}
}

I think I’m close to find the solution … but i’m out of options.

compileTestJava.onlyIf{ false }

That would be ā€œbetterā€ written as compileTestJava.enabled = false, but yes, this is most probably a very bad idea anyway. :slight_smile:

I think I’m close to find the solution

yes :slight_smile:

but i’m out of options.

no :slight_smile:

But it seems to resolve the dependencies by merging those of the to tasks …

Tasks do not have dependencies.

This

task var2Test(type: Test) {
	dependencies {
		testImplementation(project(":ProjectA"))
		testImplementation(project(":ProjectA")) {
			capabilities {
				requireCapability("org.xander.test:ProjectA-var2-impl")
			}
		}
	}
}

task var1Test(type: Test) {
	dependencies {
		testImplementation(project(":ProjectA"))
		testImplementation(project(":ProjectA")) {
			capabilities {
				requireCapability("org.xander.test:ProjectA-var1-impl")
			}
		}
	}
}

is exactly the same as

task var2Test(type: Test)
dependencies {
	testImplementation(project(":ProjectA"))
	testImplementation(project(":ProjectA")) {
		capabilities {
			requireCapability("org.xander.test:ProjectA-var2-impl")
		}
	}
}

task var1Test(type: Test)
dependencies {
	testImplementation(project(":ProjectA"))
	testImplementation(project(":ProjectA")) {
		capabilities {
			requireCapability("org.xander.test:ProjectA-var1-impl")
		}
	}
}

and thus as

task var2Test(type: Test)
task var1Test(type: Test)
dependencies {
	testImplementation(project(":ProjectA"))
	testImplementation(project(":ProjectA")) {
		capabilities {
			requireCapability("org.xander.test:ProjectA-var2-impl")
		}
	}
	testImplementation(project(":ProjectA"))
	testImplementation(project(":ProjectA")) {
		capabilities {
			requireCapability("org.xander.test:ProjectA-var1-impl")
		}
	}
}

Having the dependencies block within the task configuration is just bad.
It is mainly visual clutter, with the side-effect that it would ā€œworkā€ somehow if you only execute one of the two tasks (and not otherwise break task-configuration avoidance) as you add the dependencies if those tasks get configured.

As you need separate classpaths because of the classes being in the same package, you need to properly separate them.
For example define separate JVM test suites and declare the dependencies on those.

1 Like

Thank you so much for your answer and explanations. Things are more clear now.

I’m now using JVMTestSuite and feature variants. In project B, the test code is located at the default path src/test/java. Btw, i’ve change the test implementation:

// From ...
Main toTest = new MainImpl();
// To ...
Main toTest = Class.forName("org.xander.test.MainImpl").newInstance();

I’m not very happy with that, but it allows me to go throught the compileTestJava task, even tho i’m suspecting i could have sort it out with a correct gradle configuration …

So, for now, in Project B, I have the following:

// [...] plugin declarations, etc. [...]

testing {
    suites {
        configureEach [
            useJUnitJupiter()
            dependencies {
                // [...] some JUnit dependencies
                implementation project(':ProjectA') // Adding the API code only.
            }
        }

        impl1Test(JvmTestSuite) {
            dependencies {
                runtimeOnly(project(':ProjectA')) {
                    capabilities {
                        requireCapability("org.xander.test:ProjectA-var1-impl")
                    }
                }
            }
        }

        // [...] Same goes for test suite impl2Test relying on var2Impl feature variant
}

// Let's not forget this part as indicated in the JVMTestSuite documentation
task.named('check') {
    dependsOn testing.suites.impl1Test, testing.suites.impl2Test
}

Bad news, it doesn’t work … I’ve launched the build script with:
./gradlew clean build --info
and sadly, got the following messages:

Resolve mutations for :ProjectB:compileImpl1TestJava (Thread[Execution worker,5,main]) started.
:ProjectB:compileImpl1TestJava (Thread[Execution worker,5,main]) started.

> Task :ProjectB:compileImpl1TestJava NO-SOURCE
Skipping task ':ProjectB:compileImpl1TestJava' as it has no source files and no previous output files.
Resolve mutations for :ProjectB:processImpl1TestResources (Thread[Execution worker,5,main]) started.
:ProjectB:processImpl1TestResources (Thread[Execution worker Thread 15,5,main]) started.

> Task :ProjectB:processImpl1TestResources NO-SOURCE
Skipping task ':ProjectB:processImpl1TestResources' as it has no source files and no previous output files.
Resolve mutations for :ProjectB:impl1TestClasses (Thread[Execution worker Thread 15,5,main]) started.
:ProjectB:impl1TestClasses (Thread[Execution worker Thread 8,5,main]) started.

> Task :ProjectB:impl1TestClasses UP-TO-DATE
Skipping task ':ProjectB:impl1TestClasses' as it has no actions.
Resolve mutations for :ProjectB:impl1Test (Thread[Execution worker,5,main]) started.
:ProjectB:impl1Test (Thread[Execution worker,5,main]) started.

> Task :ProjectB:impl1Test NO-SOURCE
Skipping task ':ProjectB:impl1Test' as it has no source files and no previous output files.

The truth is out there, at reach.

I could have skipped the test task with -x test here as I only want the test suites … which are not run even tho the tasks are executed BUT skipped (as it has no actions, cascaded from previous tasks stating Skipping ... as it has no source files and no previous output). Which is kinda true as there’s no specific source for the different test suites. We only link the common test with JARs containing the contextual implementation. The solution might be to actually produce an artifact out of the source in Project B src/test/javaand the referenced JAR.

As far as i’ve understood/observed, the tests don’t have any implementation (which is obvious when you think about it). Here, I want to share the test implementation in src/test/java with all the test suite, and run the SAME tests with different dependencies.

By default, using JVMTestSuite, the sourceSets impl1Test and impl2Test will be created. I could put specific test in src/impl1Test/java and src/impl2Test/java respectively. But copying the common tests stored in src/test/java in each of this file tree is not an option (or not a clean one).

I’ve tried to include the generic sourceSet by adding implementation sourceSets.test.output into each suites dependencies, but that doesn’t seem to work.

So, it leaves me with at least two other leads :

  1. Trying to adding test sourceSets output to impl1Test and impl2Test output.
  2. Diving into TextFixtures.

I’ve finally made it ! And without Test Fixtures.

As a reminder : I have tests in Project B, which code don’t change, but that need to be executed according to different implementations of a same API, exported as jars from a Project (called A).

After using the feature variants in Project A, i need to refer each variant using test suites. The catch is that the test suites must include the source of the common tests (located in default sourceSets test) as implementation. I’ve finally came to the following (working) solution:

// [...] build.gradle preamble

testing {
    suites {
        configureEach [
            useJUnitJupiter()
            dependencies {
                // [...] some JUnit dependencies
                implementation project(':ProjectA') // Adding the API code only.
            }
        }

        impl1Test(JvmTestSuite) {
            sources {
                java {
                   srcDir "src/impl1Test/java" // This one might not be needed (only if specific tests)
                   srcDirs += sourceSets.test.allJava.srcDirs
               }
            }
            dependencies {
                runtimeOnly(project(':ProjectA')) {
                    capabilities {
                        requireCapability("org.xander.test:ProjectA-var1-impl")
                    }
                }
            }
        }

        // [...] Same goes for impl2Test

    } // end of suites
}

Is that the ā€œmost correctā€ way to proceed ? At least, it works for my use case.

I’m not very happy with that, but it allows me to go throught the compileTestJava task, even tho i’m suspecting i could have sort it out with a correct gradle configuration …

Well yeah, you do not depend on the impl from the test source set, so you cannot compile against it.
Add one as compileOnly then you should be able to compile.

and sadly, got the following messages:

Well, what do you not like with it?
You declared new test suites and did not add any code to them, so there is nothing to compile or test.
… as you concluded already :slight_smile:

Is that the ā€œmost correctā€ way to proceed ?

I don’t think so.
For example you compile the sources twice, once for each test suite.
Probably test fixtures is indeed the better way to do it.
Or as a poor-mans solution, maybe adding the source set output to the runtime classpath of the other test tasks or something like that. :man_shrugging:

Never had such an uncommon setup yet myself.
It would probably better if each implementation has its own namespace / package.
Then you could also just test everything in the test test suite.

1 Like

For example you compile the sources twice, once for each test suite.
Probably test fixtures is indeed the better way to do it.

That’s indeed my main concern here. I hate to have to compile test twice (or more if there’s more variants), especially because it adds unecessary compilation time overhead.
So, I’ll give a try on fixtures. But I have to keep in mind that I’ll have to deploy this solution on an existing voluminous project with way more intricacies than this simple mock-up :rofl:

Thank you again for your guidance. I’ll come back with another (hopefully better) attempt !

1 Like