How to make Gradle testkit depend on output jar rather than just classes?

Ned_Twigg · August 15, 2016, 1:38pm

When you apply java-gradle-plugin it automatically creates a plugin-under-test-metadata.properties file with the appropriate classpath for use with testkit.

The problem is, that classpath doesn’t include build/libs/myplugin.jar, instead it includes build/classes/main and build/resources/main. This means that it isn’t a true functional test - if there’s a problem in the “jar” step, it doesn’t get caught. It also makes it hard to test a plugin which can’t run without special packaging (e.g. a plugin which is also an OSGi bundle).

I guess the reason for pluginUnderTestMetadata to depend on classes rather than assemble is largely a performance optimization. But for me, the time to assemble the jar is much faster than the startup time for testkit. It would be great if there was an easy way to test the final build output, rather than just the folder of class files.

bmuschko · August 15, 2016, 9:14pm

This means that it isn’t a true functional test - if there’s a problem in the “jar” step, it doesn’t get caught.

I’d probably agree that the deliverable artifact (in this case the JAR) has a stronger resemblance to what the end user would use than a pointer to its runtime classpath. What specific problem could you potentially foresee with the JAR creation? That required files are not included that are needed at runtime and therefore the plugin fails at runtime? Can you provide an example?

It also makes it hard to test a plugin which can’t run without special packaging (e.g. a plugin which is also an OSGi bundle).

Can you explain your use case? Why does a Gradle plugin have to be an OSGi bundle at the same time?

It should be fairly simple to require the task pluginUnderTestMetadata to use a different classpath. Something like this should do:

pluginUnderTestMetadata { 
    pluginClasspath = files(jar.archivePath) + pluginUnderTestMetadata.pluginClasspath - files(extensions.gradlePlugin.pluginSourceSet.output.classesDir, extensions.gradlePlugin.pluginSourceSet.output.resourcesDir)
}

At the moment I can only think of two good reasons for depending on the source set output instead of the JAR.

Less tasks to execute and therefore slightly faster to run.
Easier to construct internally in the plugin implementation.

Ned_Twigg · August 15, 2016, 9:44pm

I’ve gotten burned by this in two different ways, in two different plugins:

Fat jars

I have a formatting plugin which uses the Eclipse formatter, the latest and best of which isn’t available via maven, only eclipse p2 repositories, so I have to embed the classfiles “fat jar” style.

github.com

diffplug/spotless/blob/master/build.gradle#L85-L97


buildscript {
	repositories { maven { url 'https://plugins.gradle.org/m2/' } }
	dependencies {
		classpath "com.diffplug.gradle:goomph:${VER_GOOMPH}"
		classpath "com.jfrog.bintray.gradle:gradle-bintray-plugin:${VER_BINTRAY}"
		classpath "com.gradle.publish:plugin-publish-plugin:${VER_PLUGIN_PUBLISH}"
		classpath "org.ajoberstar:gradle-git:${VER_GRADLE_GIT}"
		classpath "ch.raffael.pegdown-doclet:pegdown-doclet:${VER_PEGDOWN_DOCLET}"
	}
}


// root eclipse project
apply plugin: 'com.diffplug.gradle.eclipse.resourcefilters'
eclipseResourceFilters {
	exclude().folders().name('.git')
	exclude().folders().name('build').recursive()


	exclude().folders().name('lib')
	exclude().folders().name('lib-extra')
	exclude().folders().name('plugin-gradle')

This file has been truncated. show original

I lost several hours when a change broke this part of my build, and I didn’t realize that was the case, partially because I thought I had integration tests when I actually didn’t.

OSGi / jars with specific layout

I have a gradle plugin which will setup an Eclipse IDE for your project. To do so, it starts eclipse, inserts itself into that running eclipse, and manipulates the workspace using things which you set in your gradle script.

github.com

diffplug/goomph/blob/master/src/test/java/com/diffplug/gradle/oomph/OomphIdePluginTest.java#L26-L57


	// This test fails because of how gradle's testkit
	// constructs its classpath.  It uses a separate folder for
	// class files and resource files, rather than the jar'ed result,
	// which means that OSGi can't find the MANIFEST.
	//
	// You can work around this manually by changing the
	// plugin-under-test-metadata.properties file, but it's
	// a hassle to automate.
	//
	// Asked for a workaround / change from Gradle here, we'll see how it goes.
	// https://discuss.gradle.org/t/how-to-make-gradle-testkit-depend-on-output-jar-rather-than-just-classes/18940
	@Ignore
	@Test
	public void assertReadmeExampleWorks() throws IOException {
		write("build.gradle",
				"plugins {",
				"    id 'com.diffplug.gradle.oomph.ide'",
				"}",
				"",
				"oomphIde {",

This file has been truncated. show original

I lost a bunch of hours here before I realized that testkit didn’t actually test my plugin, just the plugin classes.

It seems like premature optimization to me to skip the assemble step on something as slow as an integration test.

bmuschko · August 18, 2016, 7:18pm

Thanks for providing the examples. Providing the JAR as input for pluginUnderTestMetadata sounds useful. If you are interested in providing a pull request, we’d be happy to review and usher it through the process. We are looking forward to your contributions!

I have a formatting plugin which uses the Eclipse formatter, the latest and best of which isn’t available via maven, only eclipse p2 repositories, so I have to embed the classfiles “fat jar” style.

The fat JAR I would have probably tested separately in a test case that executes the jar task verifies that the generated JAR file contains the right files.

Ned_Twigg · August 18, 2016, 7:50pm

The fat JAR I would have probably tested separately

If you build that test and it fails, it’s great cuz you know right where to look.

The point of a functional test, imo, is to shake out all the stuff you didn’t think to look at. That’s why I think this default is broken.

If you are interested in providing a pull request

It’d be cool to land something in Gradle, but I’ve already got my workarounds in place so I don’t think this issue is gonna make it to the top of my priority queue. You guys are busy too, no worries if it doesn’t make it to the top of yours either.

Topic		Replies	Views
Gradle Custom Plugin - Functional Test Help/Discuss plugins	0	614	December 14, 2015
Java-gradle-plugin and idea plugin incompatibilities Bugs not-a-bug	1	1224	October 11, 2016
Exclude file dependencies of compile from testCompile Old Forum Archive	3	10416	October 21, 2014
GradleRunner: missing resources files in .gradle-test-kit/.../main.jar Plugin Portal	4	324	December 13, 2023
TestKit for plugin that depends on another module in a multi-module build Help/Discuss	7	986	May 14, 2022

How to make Gradle testkit depend on output jar rather than just classes?

Fat jars

OSGi / jars with specific layout

Related topics