How to properly append to the JavaExec and Test task classpaths?

Help/Discuss
1

What is the correct (i.e. lazy, unnecessary-task-configuration-avoiding, up-to-date-checking, configuration-cache-compatible) API to use for adding the artifacts of a custom configuration (it’s a resolvable configuration) to the JavaExec and Test task classpaths?

The JavaExec task has a classpath() method that accepts and seems to work with a NamedDomainObjectProvider<Configuration> from configurations.existing. This is somewhat working for me—except not with the run task of the Application plugin if I apply it in the plugins { } block ordered before my precompiled script plugin with this configuration—so I want to be sure I’m truly using the API correctly.

The Test task seems to lack that method, and just has a classpath property. I can’t seem to find the right incantation for appending to the property that resolves the artifacts for the direct and transitive dependencies of my custom configuration, while at the same time not triggering a configuration cache violation by referencing my custom Configuration from the task action.

Here is my latest attempt (no configuration cache warnings, but the custom configuration artifacts aren’t being added to the test classpath):

tasks {
    // The variant-aware custom resources resolvable configuration.
    val customClasspath by configurations.existing

    withType<JavaExec>().configureEach {

        // For the `JavaExec` tasks created by the Gradle Application Plugin,
        // the Spring Boot Gradle Plugin, or generated by IntelliJ...
        if (name in listOf("run", "bootRun", "bootTestRun") ||
            name.endsWith("main()")) {

            // Add the custom resources to the runtime classpath.
            classpath(customClasspath)
        }
    }

    withType<Test>().configureEach {

        val customClasspathFileCollection =
            customClasspath.get().fileCollection()

        doFirst {
            // Add the custom resources to the classpath for all `Test` tasks.
            classpath = classpath.plus(customClasspathFileCollection)
        }
    }
}

Full reproducer: GitHub - sdkotlin/sd-kotlin-spring-talks at ea616f42d50e41bc6a22873a9f7d56fa07181a54

2

You really shouldn’t change the configuration of the task at execution phase.
I actually wonder that this does not fail with configuration cache, afair changing configuration should be disallowed with CC on, so you should maybe report it as bug if it does not fail.

What would want is

withType<Test>().configureEach {
	classpath += files(customClasspath)
}

The problem is, that classpath gets overwritten after you did that, probably because the Test task is registered after calling that and thus this configureEach is executed first and then it is set again to another value.

If you wrap it in an afterEvaluate ( :‘’'-( ) it works and makes the test green.

It is probably better to target specific test tasks and apply the plugins that register them first or something like that.

3

Thanks for the reply.

Oh boy, my working understanding has generally been that if the DSL/APIs are used correctly, plugin application order shouldn’t matter with Gradle. I’m a bit surprised it doesn’t resolve all the task registrations and corresponding task type configurations up front, and sequence them in dependency order.

Either way, the afterEvaluate wrapping and Test task configuration approach you suggested seems to address both of my issues. My tests are green, and I seem to be able to apply my custom classpath convention plugin in any order relative to the Application plugin and the run task still works.

Result here: sd-kotlin-spring-talks/build-logic/src/main/kotlin/org.sdkotlin.buildlogic.custom-resources-convention.gradle.kts at 81097ef7a071474302daf4188dd9f3bda4bee69c · sdkotlin/sd-kotlin-spring-talks · GitHub

Using afterEvaluate is new to me, so I am wondering what if any downsides it might have on the performance or maintainability of my main project (380+ subprojects). I’m applying an equivalent to this custom classpath convention plugin to most of them via my main convention plugin. That way any project readily has the option of contributing artifacts to my consumable customElements configuration, or declaring corresponding dependencies via my customScope configuration.

4

Oh boy, my working understanding has generally been that if the DSL/APIs are used correctly, plugin application order shouldn’t matter with Gradle.

That’s correct and the desired outcome.
So as a conclusion, do not use tasks.withType... in a plugin but target specific tasks of plugins you applied from within the same plugin. :slight_smile:

I’m a bit surprised it doesn’t resolve all the task registrations and corresponding task type configurations up front, and sequence them in dependency order.

Not sure what you mean by that.
Gradle executes configuration actions in the order they were registered.
How should Gradle otherwise know which configuration action depends on which other?

If you for example have:

tasks.configureEach { println("before") }
tasks.register("foo") { println("within") }
tasks.configureEach { println("after") }

you get first “before” then “within” then “after”.
You can imagine tasks.register("foo") { println("within") } as a short form of

tasks.register("foo")
tasks.named("foo") { println("within") }

if that helps grasping it, then you would have

tasks.configureEach { println("before") }
tasks.register("foo")
tasks.named("foo") { println("within") }
tasks.configureEach { println("after") }

Now in your case you most probably set classpath in “before” and then it gets re-set in “within” or “after”, as there is no additive classpath(...) function and even if there were, “within” or “after” could use the setter to overwrite the whole thing.

Using afterEvaluate is new to me, so I am wondering what if any downsides it might have on the performance or maintainability

It does not have problems on the performance side, it just schedules an action to be done later after the evaluation is done and all so-far registered afterEvaluate actions are done.

The problem with afterEvaluate and why it is highly discouraged is, that its main effect is to introduce ordering problems, timing problems, and race conditions. If you for example use afterEvaluate so that something is done already (like the classpath property set already) and you can act upon it, then someone might have the same idea and register an afterEvaluate action that also sets the classpath property and if that happens to run after your afterEvaluate action, then your configuration is lost again, …

Using afterEvaluate to fix an issue is pretty much like using invokeLater in Swing or runLater in JavaFX to “fix” a GUI problem. It usually is just symptom treatment, moving the actual issues to a later, more hidden, harder to reproduce, harder to find, and harder to fix, point in time.

Due to all these shenenigans the Property / Provider APIs were introduced to hopefully remove the need to use afterEvaluate and instead being able to wire things together at configuration time that are then only evaluated at execution time. With Gradle 9 hopefully the remaining core APIs will finally be migrated to the new types and remove much more situations where you might be tempted to use afterEvaluate. Whether it would change your situation I don’t know.

Maybe a much better idea would be to not change the classpath of the test tasks, but instead to make the testRuntimeClasspath configuration extendsFrom your customClasspath, then you should not need to change the test task at all. Unless of course not all those tasks use the testRuntimeClasspath or other tasks use it that should not get those dependencies.

5

Thank you for the detailed explanation.

I’ll try to consider other ways to factor my approach. What I was after was a single cohesive mix-in for my “custom classpath build feature”. The idea being, if it’s applied to any project, that project can:

  • Optionally have custom resources in a ‘src/main/custom’ folder that become an artifact (with distinct variant) of the project.
    • No resource processing or building is required for my use case, so I’m skipping source sets and custom build tasks and all that.
  • Optionally declare variant-aware customScope dependencies on other projects with such artifacts.
  • Have the custom resources—its own, and those from any customScoped dependencies it declares—on the runtime classpath of any JavaExec tasks it registers.
    • This would be regardless of whether it registers the tasks directly, or they come from applying the Application Plugin, the Spring Boot Plugin, some other plugin, or are generated by IntelliJ.
  • Have the custom resources on the runtime classpath of any Test tasks it registers.
    • Again, be that the default test suite, or any suite from applying one of my JVM Test Suite Plugin convention plugins.

If I start targeting specific JavaExec and Test tasks by name from a single convention plugin, I couple it to specific other plugins more than I’d like. If I split up the configuration into multiple convention plugins that must be applied when corresponding other plugins are applied (e.g. “custom-resources” plus “custom-resources-application” when the Application plugin is applied), my feature is not as cohesive as I’d like, and I have a bunch of convention plugins that need to be selectively applied to projects in the right combinations.

I meant that regardless of plugin application order, declared tasks of a given type are all registered by Gradle before any configuration actions that apply to their type are invoked. I’m not particularly concerned with classpath ordering, just that all my “customClasspath” and “runtimeClasspath” artifacts are on it. As such, I’m not concerned with configuration action order in this case, so long as none are missed or overwrite the classpath additions of any other.

I made a quick example project to test this plugin application, task registration, and configuration action ordering question: GitHub - ianbrandt/gradle-task-configuration-ordering at 981f014069958bb61f4013ef440760c7b1bb4474

It does seem that Gradle works out the task registration vs. configuration action invocation dependency. For example, if I apply the task registration plugin last, it does still appear to be configured by the two preceding configuration convention plugins:

plugins {
    id("GreetWorldConvention")
    id("GreetGradleDevsConvention")
    id("GreetingPlugin")
}

> Task :sayHello
Hello World, and Gradle Devs, and Myself!

Exactly as you explained, the register and withType configuration actions seem to be invoked in the order their declaring plugins are applied, but again that’s fine for my use case so long as they don’t stomp on each other.

I probably should have described my use case better up front. I have some resources that I want on the classpath for portable loading when running and testing in the IDE, on my CI server, or when my application is installed. I need to keep those resources out of my JARs/lib folder though, so I can locate them in appropriate user-writable locations when my application is installed. My installer generator (Install4J) can accept multiple distinct file sets as classpath strings. By feeding it customClasspath.asPath and runtimeClasspath.asPath separately, I can easily locate my user-editable files and JARs where I need them, while still including both in the application’s runtime classpath.

So as you said, it seems my classpath additions are getting stomped on somewhere, somehow.

I’ve observed in the initial project I linked that the Spring Boot Plugin and its bootRun task don’t seem to care about plugin vs. configuration convention plugin application order, whereas the Application plugin and its run task do.

Unless someone can see something I’m still doing wrong in my withType<JavaExec> and withType<Test> configuration actions that’s driving the need for afterEvaluate, I’m thinking I should try to get proper reproducers together, and file these as one or more bugs/issues.

6
  • Optionally declare variant-aware customScope dependencies on other projects with such artifacts.

But why on a separate configuration and applying to each and every run and test task?
That might not be appropriate, a consumer can for example also register some custom run task that invokes some utility that has nothing at all to do with those dependencies.
Wouldn’t it maybe make more sense to provide some helper like testFixtures(...) that sets the attributes directly on the dependency, then the consumer can add the variant dependency exactly where he needs it, for example on implementation or runtimeOnly or whatever.
Then it would also automatically be on the relevant run and test tasks.

  • Have the custom resources—its own, and those from any customScoped dependencies it declares—on the runtime classpath of any JavaExec tasks it registers.
    • This would be regardless of whether it registers the tasks directly, or they come from applying the Application Plugin, the Spring Boot Plugin, some other plugin, or are generated by IntelliJ.
  • Have the custom resources on the runtime classpath of any Test tasks it registers.
    • Again, be that the default test suite, or any suite from applying one of my JVM Test Suite Plugin convention plugins.

As noted above, doing it on each and every task might not be the right thing anyway.
So it might make more sense to make runtimeClasspath extend from customScope if not following the above suggestion to set the attributes on the dependency directly.
Then it should be on all run and test tasks that use the production runtime classpath too just like with the attributes on the dependency directly.

I meant that regardless of plugin application order, declared tasks of a given type are all registered by Gradle before any configuration actions that apply to their type are invoked.

What would that change, besides that it probably is technically impossible as you don’t know when maybe other tasks of the same type would be added. But even if it would be determinable, that wouldn’t change anything.
It would most probably still execute the registered configure actions in registration order, so you would have the exact same behaviour and situation even if you would wait for all tasks of the same type to be registered.

As such, I’m not concerned with configuration action order in this case, so long as none are missed or overwrite the classpath additions of any other.

But that’s exactly the point. You set up some classpath and some later configuration action sets it to a different value. That’s like you setting “aBoolean” to “true” and a later configuration action is setting it to “false”. So yes, configuration action order is exactly the topic here. I was not referring to the classpath order that might make additional problems as classpaths optimally should not have duplicate classes and thus ordering not be relevant. :slight_smile:

By feeding it customClasspath.asPath and runtimeClasspath.asPath separately, I can easily locate my user-editable files and JARs where I need them, while still including both in the application’s runtime classpath.

Well, you could still do that. At that time you can either subtract customClasspath from runtimeClasspath again, assuming there is no lib in both, and thus supplying them separate, or when using attributes on the dependencies instead of a custom configuration as suggested above, you can use artifact views to separate the runtimeClasspath by attributes before you supply it.
This is not a theoretical statement.
We have an own application starter, similar to Java WebStart, and it has separation for classpath/modulepath and for windows/linux/mac/cross-platform.
And I use artifact transforms and artifact views to segregate the libs how I need them, even depending on some consumer-set property like “put everything on the classpath”, “put everything on the modulepath”, “put full and automatic modules on the modulepath”, “put only full modules on the module path” and so on.

So for your case where the artifacts that have a custom attribute anyway need to be spearated, it should pretty simple be possible to separate them using an artifact view, or checking the resolved artifacts for the attribute and so on.

Unless someone can see something I’m still doing wrong in my withType<JavaExec> and withType<Test> configuration actions that’s driving the need for afterEvaluate , I’m thinking I should try to get proper reproducers together, and file these as one or more bugs/issues.

As I said, it is nothing you do there that “breaks it”, neither is it something to report as bug. It is just like you setting a boolean to false while a later configuration action is setting it to true.

Additionally, who knows how it will look like in Gradle 9 where the big lazification is done, using providers, properties, and so on everywhere hopefully. Maybe also this api changes then. :man_shrugging:

7

Those are good points, and I’m certainly open to a better way of realizing my use case.

We started out trying subtractive approaches with Configuration.minus() and Configuration.filter(). I forget the details, but we weren’t able to get that to work, and pivoted to the separate configurations approach. The latter has worked for us for a while, up until some plugin declarations got reordered.

I recall rereading the artifact transforms documentation multiple times when we were trying to figure out an approach for this, but I didn’t put two and two together as to how that could be used to subset configurations. Knowing that it’s working for that purpose for someone else helps, and the testFixtures()-like helper for setting attributes on a dependency tip helps too. I’ll revisit the docs with all that in mind, and see if I can’t come up with a working prototype.

I’m not sure a boolean property is really a comparable example to a classpath property. I don’t have any expectation of being able to append to a boolean. Strings, collections, these types I generally expect to be able to concatenate, append, or plus-assign to. The JavaExec.classpath() documentation reads, “Adds elements to the classpath for executing the main class.” With Test.classpath it’s a little less explicit, but classpath += files(configuration) compiles and runs without warning, and seems perfectly reasonable.

Knowing now that it seems to not be a withType before register issue, but rather a collection property resetting issue, I can try to track down exactly what is stomping on my classpath additions in each case. If not “bugs” per se, I’d call them design integrity issues that should be addressed.

If Gradle 9 and subsequent “Declarative Gradle” efforts reign in the number of partially flushed out, inconsistent, and wrong ways to do things that are readily available in the DSL/API, and execute without failing fast or logging clear warnings, I will very much welcome those efforts. Even if they entail aggressive DSL and API deprecation and removal, I’d see the short-term pain as being worth it. Of course migration tooling to ease the process would be welcomed too. :sweat_smile:

8

I’m not sure a boolean property is really a comparable example to a classpath property. I don’t have any expectation of being able to append to a boolean.

Sure:
myBoolean = myBoolean || myOtherBoolean
vs.
myBoolean = myOtherBoolean
:slight_smile:
The classpath property is not any different.
You can calculate a new value with adding to what is already there, or you can overwrite what is already there with a completely own value.

The JavaExec.classpath() documentation reads, “Adds elements to the classpath for executing the main class.”

Yes, if you use classpath(...) you add to it, just like you would with classpath += ....
And in both cases if someone later does classpath = ... what you configured is potentially gone.
It is exactly the same as for Test, just there is an additional helper method that replaces the += usage and exactly the same as with myBoolean.

If not “bugs” per se, I’d call them design integrity issues that should be addressed.

Sure, feel free to find out and complain to the maintainer of whatever does it.
Just wanted you to be prepared to hear a “works as designed” eventually. :slight_smile:
One of the issues you’ll stumble upon and that indeed gets fixed hopefully is https://youtrack.jetbrains.com/issue/IDEA-343338 :slight_smile:

9

It seems we’ll have to agree to disagree about property handling. It depends on the specific context, but I generally believe a better design for single value types—such as boolean debug and enableAssertions flags—would be that they can only be assigned to once (TornadoFX had a nice singleAssign() delegate for this), and for collection types—such as classpath and environment—that they can only be appended to. Otherwise the system is more amenable to one part unintentionally and quietly undermining another. I realize that’s not the Gradle we have today, so this is drifting a bit off topic.

I’ve filed issues with reproducers for JavaExec.classpath and Test.classpath:

We’ll see what happens with them.

I voted for and will be following IDEA-343338.

10

It seems we’ll have to agree to disagree about property handling

Seems so, yes.
For the “one-time assign” you can do that with Property.
But for “collection types” it is extremely important that you are able to reset them to a completely different value.
There are situations where you need to filter something out from what is set by convention or similar.
Whether the built-in plugins should use a setter or an adder, that’s a different topic, but it has to be possible latest for the build script, but actually also for some convention plugins, or you make some use-cases extremely harder to realize and taking away flexibility.

In Gradle you can even clear() the actions of a task and thus completely replace what it would have been doing with something else, including the built-in action, and in some cases this also is necessary. Would be really sad if Gradle would loose that level of flexibility that you sometimes need. :slight_smile:

11

I don’t see any reason why you couldn’t have both. I’m straining to remember specific examples, but a pattern I’ve seen used to good effect in the past is akin to BasicApi with a getAdvancedApi(): AdvancedApi, or similar. Particularly when it comes to internal DSLs, where scope pollution and management can be a challenge, this can be a very effective idiom. Typical users are guided straight to the this is what you should probably be using API, while advanced users still have ready access to the here be dragons API. It doesn’t stop someone from reaching for and misusing the advanced API, but at least it’s clear which is which, when the advanced API is being used, and that there may be consequences to using it without proper understanding.

12

I’ve got it working where I set the attributes for a runtimeOnly dependency on a project that has a corresponding custom resources artifact:

Now I’m trying to DRY that up into a testFixtures()-like helper function, but I’m not seeing exactly how to to do that.

I tried making a plain old Kotlin extension function in my included ‘build-logic’ build that I’d import, but I get a “Mutation of attributes is not allowed” error with that:

fun DependencyHandler.customResources(
    objectFactory: ObjectFactory,
    notation: Any,
): Dependency {

    val dependency = create(notation)

    require(dependency is ProjectDependency) {
        "Dependency type ${dependency::class.qualifiedName} unknown!"
    }

    dependency.attributes.applyResourceAttributes(
        objectFactory = objectFactory,
        resourceAttributeValue = CUSTOM_RESOURCE
    )

    return dependency
}

I’ve looked at adding an extension with dependencies.extensions.add("customResources", CustomResourcesDependencyHandlerExtension::class), but that seems to add a dynamic property with optional configure action to the Kotlin DSL, not a new DependencyHandler.customResources(notation: Any) helper method.

Is there a way for a binary or precompiled script plugin to add a new Kotlin DSL helper function to the DependencyHandler scope?

13

You would need to add an extension that has the method and then call it qualified.
You cannot easily and portably add a method directly.
But even if you could or if you would use the extension way, I guess the error would remain the same.
I’m not sure, maybe you cannot change the attributes at that time on that object?
The testFixtures(...) is setting a capability, not an attribute.
Not sure, maybe needs some trying out to find the “right” way.

14

Bummer. I found this seemingly-related request:

In the meantime, I realized I can use a functional interface for the extension public type that overrides invoke to fake out an unqualified method call:

It turned out I just needed to change from eager to lazy API. It works now. :man_facepalming:

15

Ah, right. I fell into that trap myself a few times already. Should really start to remember this. :smiley:

16

I’m on trying to use an ArtifactView to filter the runtimeClasspath for only my “custom resources” directory artifacts.

I’m not sure what I’m doing wrong, but when I print the filtered view it seems to include my custom resources directory, but also all my subproject resources directories, and all the third-party dependency JARs. For whatever reason it does not include any of the subproject classes directories or JARs.

> Task :subprojects:app:dumpCustomClasspath
customResourcesAsPath: 
C:\Dev\Repos\SDKotlin\sd-kotlin-spring-talks\subprojects\time-logger\build\resources\main;
C:\Dev\Repos\SDKotlin\sd-kotlin-spring-talks\subprojects\time-service\build\resources\main;
...
C:\Dev\.gradle\caches\modules-2\files-2.1\org.springframework.boot\spring-boot-starter\3.2.4\842cf7f0ed2ecfef3011f3191fc53c59ceed752\spring-boot-starter-3.2.4.jar;
...
C:\Dev\Repos\SDKotlin\sd-kotlin-spring-talks\subprojects\custom-resources\src\main\custom;
...
17

Yeah, the whole attribute and variant stuff is not too trivial.
If you request your attribute, then all that do not have the attribute are considered compatible.
You either need to set the attribute for all other variants to some different value or do a manual filtering to filter out the dependencies that do not have a value for the attribute.

From the artifact view you would go through .artifacts.resolvedArtifacts, and then filter those on variant.attributes, checking whether your attribute is set and to a non-null value for example.

18

I’m trying the manual filtering approach, but not having much luck with the API:

register("printCustomClasspath") {

    group = "custom"

    val customResourcesFiles: List<File> =
        configurations.runtimeClasspath.get().incoming.artifactView {

            lenient(false)

            @Suppress("UnstableApiUsage")
            (withVariantReselection())

            attributes {
                attribute(RESOURCE_ATTRIBUTE, objects.named(CUSTOM_RESOURCE))
            }
        }.artifacts.resolvedArtifacts.get()
            .filter { it: ResolvedArtifactResult? ->
                val variant = it?.variant
                val attributes = variant?.attributes
                println("*** attributes: $attributes")
                val resourceAttributeValue = attributes?.getAttribute(RESOURCE_ATTRIBUTE)
                println("*** resourceAttributeValue: $resourceAttributeValue")
                resourceAttributeValue == objects.named(CUSTOM_RESOURCE)
            }.map { it.file }

    val customResourceFileCollection = files(customResourcesFiles)

    doLast {

        val customResourcesAsPath = customResourceFileCollection.asPath

        println("customResourcesAsPath: $customResourcesAsPath")
    }
}

I am getting ResolvedArtifactResults, one of which I can see has my attribute, but .getAttribute(RESOURCE_ATTRIBUTE) still comes back null. I tried debugging into it, and it seems the attribute is found, but then at some point is deemed not org.gradle.internal.isolation.Isolatable, and isn’t returned. I assume I’m using the API wrong, but I’m not sure what I need to be doing differently.

In addition, I thought it wasn’t good to resolve a Configuration during the configuration phase, and yet closuring one in a doLast isn’t compatible with the configuration cache. I’m also not clear on the purpose of continuing to use an ArtifactView if we’re just filtering through all the resolved artifacts manually. Perhaps I’ve just outright misunderstood your suggested approach.

Short of any further guidance, my plan is to start looking into the option of setting the attribute for all other variants.

19

I cannot give much active assistance right now, as I’m without computer next two weeks. One point to keep in mind is, that when manually comparing the attribute values, they could be the named object or a pure string depending on where the value is coming from (Gradle Module Metadata or added at execution time of the build).

Maybe something similar was with the attribute itself, as it is not specifically designed for such manual filtering. I think I iterated over the attributes and compared their name to get the right one.

The artifact view is handy of an attribute for example can have multiple values. Or you get as different variant that has the attribute of you request it. The manual filtering iirc is then mainly helpful to filter out the ones without the attribute.

Regarding resolving configuration at configuration time, yeah better not do it explicitly, so do not call get but use map on the resolved artifacts, so that the resolution and so on is not done to early but only when the configuration cache is persisted.

20

I recently had luck getting subsets of the runtimeClasspath using ArtifactView.componentFilter { ... } for separating out library jars and project jars for ProGuard. That inspired me to take another stab at getting resource subsets by attribute.

I made numerous varying attempts at getting manual filtering to work, but everything I tried failed on one error or another. I gave up on that approach, and dug my heels in on the attribute matching approach instead. I finally got that to work.

The trick was to not add a variant with a new attribute, but instead add a variant with a new value for the existing org.gradle.libraryelements attribute. That seems to not fight the variant matching algorithm and existing compatibility and disambiguation rules at play for the runtimeClasspath configuration.

I generalized it into a plugin. I haven’t extracted it from my test project and published it, but here’s the source in case anyone else ever finds this and has the same need:

Example of use: