Multi jdk distribution plugin

Help/Discuss
1

Hello everyone,
I have problem and I hope some more experience gradle plugin developer can help me find what exactly I’m doing wrong. Problem I’m trying to solve by plugin is following. I would like to compile my Java source code with multiple JDK. To be able to produce all LTS jdks artifact in single build. I took inspiration originally from plugin generating multi jdk jars. However I’m facing issue with dependencies in multi-project build.
To start from beginning, I defined plugin and plugin extension classes. In plugin I register extension collect some objects for extension and that is about it. Extension class definite method to consume requested JDK configuration and in this method I’m doing most of configuration. Fist step is creating SourceSet specific to extension and extending it by main source set to allow specific compiler discover classes defined in main source set. Setting up some filtering for modules classes in case when JDK8 and define used compiler in compile task including target platform.
As far as I can say this is working if project has no dependencies. However when one project module deepens on other project module I’m facing problem how to configure build. After week or so experimenting and debugging my best success was adding compile/classes task configuration when I was able to locate configured dependencies and propagate them to new feature configuration.
Unfortunately this approach even to me feels like hack and not properly working. Dependencies are propagated unfortunately when gradle try to resolve variant for specific JDK I’m getting following variant error:

  • Variant ‘jdk8RuntimeElements’ declares a library for use during runtime, compatible with Java 8, packaged as a jar, and its dependencies declared externally:
    • Other compatible attribute:
      • Doesn’t say anything about its target Java environment (preferred optimized for standard JVMs)

This part of message is for variant I expect to be used as dependency in sub-module compiled by JDK8 registered as feature. Can someone help me to identify problem?
BR,
Thank you,

2

At least me, I cannot really follow your description, and an MCVE might help.

Just regarding the error message, if you would provide the full error, optimally as a build --scan URL, maybe something could be said about. Like that most of the error is missing as far as I can tell.

3

Thank you for answer. I can do better. I plan to release it under opensource license. Source code is for sure messy, but I will try provide some more details here in message.

In this folder is implementation state I currently have. As I mention is far from clean, current state is result of around of week of experimentation.
complete output from build:

> Could not determine the dependencies of task ':collection:bean:compileJdk8Java'.
> Could not resolve all dependencies for configuration ':collection:bean:jdk8CompileClasspath'.
   > Could not resolve project :collection:core.
     Required by:
         project :collection:bean
      > No matching variant of project :collection:core was found. The consumer was configured to find a library for use during compile-time, compatible with Java 8, packaged as a jar, preferably optimized for standard JVMs, and its dependencies declared externally but:
          - Variant 'apiElements' declares a library for use during compile-time, packaged as a jar, and its dependencies declared externally:
              - Incompatible because this component declares a component, compatible with Java 23 and the consumer needed a component, compatible with Java 8
              - Other compatible attribute:
                  - Doesn't say anything about its target Java environment (preferred optimized for standard JVMs)
          - Variant 'javadocElements' declares a component for use during runtime, and its dependencies declared externally:
              - Incompatible because this component declares documentation and the consumer needed a library
              - Other compatible attributes:
                  - Doesn't say anything about its elements (required them packaged as a jar)
                  - Doesn't say anything about its target Java environment (preferred optimized for standard JVMs)
                  - Doesn't say anything about its target Java version (required compatibility with Java 8)
          - Variant 'jdk8ApiElements' declares a library for use during compile-time, compatible with Java 8, packaged as a jar, and its dependencies declared externally:
              - Other compatible attribute:
                  - Doesn't say anything about its target Java environment (preferred optimized for standard JVMs)
          - Variant 'jdk8RuntimeElements' declares a library for use during runtime, compatible with Java 8, packaged as a jar, and its dependencies declared externally:
              - Other compatible attribute:
                  - Doesn't say anything about its target Java environment (preferred optimized for standard JVMs)
          - Variant 'mainSourceElements' declares a component, and its dependencies declared externally:
              - Incompatible because this component declares a component of category 'verification' and the consumer needed a library
              - Other compatible attributes:
                  - Doesn't say anything about its elements (required them packaged as a jar)
                  - Doesn't say anything about its target Java environment (preferred optimized for standard JVMs)
                  - Doesn't say anything about its target Java version (required compatibility with Java 8)
                  - Doesn't say anything about its usage (required compile-time)
          - Variant 'runtimeElements' declares a library for use during runtime, packaged as a jar, and its dependencies declared externally:
              - Incompatible because this component declares a component, compatible with Java 23 and the consumer needed a component, compatible with Java 8
              - Other compatible attribute:
                  - Doesn't say anything about its target Java environment (preferred optimized for standard JVMs)
          - Variant 'sourcesElements' declares a component for use during runtime, and its dependencies declared externally:
              - Incompatible because this component declares documentation and the consumer needed a library
              - Other compatible attributes:
                  - Doesn't say anything about its elements (required them packaged as a jar)
                  - Doesn't say anything about its target Java environment (preferred optimized for standard JVMs)
                  - Doesn't say anything about its target Java version (required compatibility with Java 8)
          - Variant 'testResultsElementsForTest':
              - Incompatible because this component declares a component of category 'verification' and the consumer needed a library
              - Other compatible attributes:
                  - Doesn't say anything about how its dependencies are found (required its dependencies declared externally)
                  - Doesn't say anything about its elements (required them packaged as a jar)
                  - Doesn't say anything about its target Java environment (preferred optimized for standard JVMs)
                  - Doesn't say anything about its target Java version (required compatibility with Java 8)
                  - Doesn't say anything about its usage (required compile-time)

* Try:
> No matching variant errors are explained in more detail at https://docs.gradle.org/8.10.2/userguide/variant_model.html#sub:variant-no-match.
> Review the variant matching algorithm at https://docs.gradle.org/8.10.2/userguide/variant_attributes.html#sec:abm_algorithm.
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
BUILD FAILED in 39s
8 actionable tasks: 8 up-to-date
Publishing a build scan to scans.gradle.com requires accepting the Gradle Terms of Use defined at https://gradle.com/help/legal-terms-of-use. Do you accept these terms? [yes, no] 
Gradle Terms of Use accepted.
Publishing build scan...
https://gradle.com/s/dllbupk42dmco

Short description MultiJdkPlugin class is main plugin class which apply java-lib plugin to project and create extension MultiJdkExtension.

In extension class main entry method is target where value from build.gradle configuration should be conusumed and as result of this consumtion new feature is registred into java compoment.

Method configurePlatform in same class represent steps done to register feature into java component. Current state represent my last experiment of separation dependencies definition (originally probably wrongly propagated as part of compile task) into Classes task.

4

Hm, the error looks quite strange, as it says it cannot find a matching variant, but then lists the variant that would have been compatible.

Sounds to me from a gut feeling that at the time the dependency is resolved the variant is not yet there, and then when the error message is constructed the variant was added in the meantime somehow.

5

Thank you, I was not sure if I understand it correctly and was on path to start searching how to fix that node about other compatibilities attributes.
I would say your gut feeling is more than correct. It is was my suspicion from beginning the way how I try to handle dependency propagation is not correct and it is done to late in process.
In my current implementation this is done as part of this code in MultiJdkExtension class:

        projectInternal.tasks.named(name+ "Classes") {
            JvmFeatureInternal mainFeature = mainComponent.getMainFeature();
            JvmFeatureInternal newFeature = mainComponent.getFeatures().findByName(name)
            configureDependencies(newFeature, mainFeature)
        }

It seems obvious compile or classes tasks are to late for changing anything in dependencies. So probably question of the day is where or how to propagate dependencies between main feature and newly created one. In essence new feature should have same dependencies and just requirement for different target JDK version.
I’m sorry if this sounds like stupid question but as I mention before I’m getting out of ideas how to get it working.

6

In my current implementation this is done as part of this code in MultiJdkExtension class:

That is an extremely bad idea.
Besides that all those “internal” things strongly hint at you doing something very bad,
additionally you really should not configure anything from a configure action of a task, except for that task.
For example in that case, you are doing this configuration when the name+ "Classes" task is configured and only if it is configured.
So if you for example execute ./gradlew compileJava, the name+ "Classes" is not going to be configured and so your configuration is not done.
Unless of course you break task-configuration avoidance somewhere, causing each and every task to be configured, which then is a great waste of time on almost any build execution.

Also how it works - if it would work - depends on order of things, so maybe it works today, then you do a change that makes the name+ "Classes" task be configured earlier and suddenly it does not work anymore.

If you want to last-minute add some dependencies to a configuration, use withDependencies like

configurations.implementation {
    withDependencies {
        //...
    }
}

These actions are executed right before the configuration in question first participates in dependency resolution and allows for example to add peer dependencies for others that got declared and similar.

Depending on what is present in another configuration there is also a bit problematic, because if you for example resolve configuration A at configuration time, adjust its dependencies depending on content of configuration B, and then add something to B, you will miss that and so on. But it might at least work slightly better than what you have currently.

But actually, you should probably do it differently.
Usually you do not propagate any dependencies manually.
You make one configuration extend another configuration.
By that you inherit all the declared dependencies like if they were declared explicitly.

For example runtimeClasspath extends implementation and runtimeOnly, by that all dependencies you declare on implementation or runtimeOnly are part of runtimeClasspath too.
So if you for example make jdk8CompileClasspath extend from compileClasspath, then all dependenices that are part of the latter are automatically also part of the former.

7

Hello sorry for later reaction. Everything you have said is correct, to be honest I was surprised I have missed that method for extending configuration from other instance. Change code to extend configuration same way like is in your example. This feel clear solution. However I was still not able to make it work. Error message seems to be still same.

Could not determine the dependencies of task ':collection:bean:compileJdk8Java'.
> Could not resolve all dependencies for configuration ':collection:bean:jdk8CompileClasspath'.
   > Could not resolve project :collection:core.
     Required by:
         project :collection:bean
      > No matching variant of project :collection:core was found. The consumer was configured to find a library for use during compile-time, compatible with Java 8, packaged as a jar, preferably optimized for standard JVMs, and its dependencies declared externally but:
          - Variant 'apiElements' declares a library for use during compile-time, packaged as a jar, and its dependencies declared externally:
              - Incompatible because this component declares a component, compatible with Java 23 and the consumer needed a component, compatible with Java 8
              - Other compatible attribute:
                  - Doesn't say anything about its target Java environment (preferred optimized for standard JVMs)
          - Variant 'javadocElements' declares a component for use during runtime, and its dependencies declared externally:
              - Incompatible because this component declares documentation and the consumer needed a library
              - Other compatible attributes:
                  - Doesn't say anything about its elements (required them packaged as a jar)
                  - Doesn't say anything about its target Java environment (preferred optimized for standard JVMs)
                  - Doesn't say anything about its target Java version (required compatibility with Java 8)
          - Variant 'jdk8ApiElements' declares a library for use during compile-time, compatible with Java 8, packaged as a jar, and its dependencies declared externally:
              - Other compatible attribute:
                  - Doesn't say anything about its target Java environment (preferred optimized for standard JVMs)
          - Variant 'jdk8RuntimeElements' declares a library for use during runtime, compatible with Java 8, packaged as a jar, and its dependencies declared externally:
              - Other compatible attribute:
                  - Doesn't say anything about its target Java environment (preferred optimized for standard JVMs)
          - Variant 'mainSourceElements' declares a component, and its dependencies declared externally:
              - Incompatible because this component declares a component of category 'verification' and the consumer needed a library
              - Other compatible attributes:
                  - Doesn't say anything about its elements (required them packaged as a jar)
                  - Doesn't say anything about its target Java environment (preferred optimized for standard JVMs)
                  - Doesn't say anything about its target Java version (required compatibility with Java 8)
                  - Doesn't say anything about its usage (required compile-time)
          - Variant 'runtimeElements' declares a library for use during runtime, packaged as a jar, and its dependencies declared externally:
              - Incompatible because this component declares a component, compatible with Java 23 and the consumer needed a component, compatible with Java 8
              - Other compatible attribute:
                  - Doesn't say anything about its target Java environment (preferred optimized for standard JVMs)
          - Variant 'sourcesElements' declares a component for use during runtime, and its dependencies declared externally:
              - Incompatible because this component declares documentation and the consumer needed a library
              - Other compatible attributes:
                  - Doesn't say anything about its elements (required them packaged as a jar)
                  - Doesn't say anything about its target Java environment (preferred optimized for standard JVMs)
                  - Doesn't say anything about its target Java version (required compatibility with Java 8)
          - Variant 'testResultsElementsForTest':
              - Incompatible because this component declares a component of category 'verification' and the consumer needed a library
              - Other compatible attributes:
                  - Doesn't say anything about how its dependencies are found (required its dependencies declared externally)
                  - Doesn't say anything about its elements (required them packaged as a jar)
                  - Doesn't say anything about its target Java environment (preferred optimized for standard JVMs)
                  - Doesn't say anything about its target Java version (required compatibility with Java 8)
                  - Doesn't say anything about its usage (required compile-time)

* Try:
> No matching variant errors are explained in more detail at https://docs.gradle.org/8.10.2/userguide/variant_model.html#sub:variant-no-match.
> Review the variant matching algorithm at https://docs.gradle.org/8.10.2/userguide/variant_attributes.html#sec:abm_algorithm.
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
BUILD FAILED in 761ms
8 actionable tasks: 8 up-to-date
Publishing a build scan to scans.gradle.com requires accepting the Gradle Terms of Use defined at https://gradle.com/help/legal-terms-of-use. Do you accept these terms? [yes, no] 
Gradle Terms of Use accepted.
Publishing build scan...
https://gradle.com/s/74l2tplv4tjh4

I implemented propagation after new source set for platform is created target method of my extension so it should be triggered as soon as information about jdk version is loaded from config file.

        def dependenciesSource = project.configurations.findByName(mainSourceSet.compileClasspathConfigurationName);
        def variantDependencies = project.configurations.findByName(created.compileClasspathConfigurationName);
        variantDependencies.extendsFrom(dependenciesSource)

To be hones I’m still stuck, I only think I can think of is I’m still missing some propagation of supported JDK because it is now configured on compile task level. Or do you think there is something else missing.
Thank you for your help.
I have also clean up my code and pushed to repository for more easy orientation :slight_smile: all internal object has been removed to :slight_smile: cz.a-d.java.tools/buildSrc/src/main/groovy/cz/ad/gradle/plugin/jdk at main · adLuk/cz.a-d.java.tools · GitHub

8

Your problem is the capability.

Whether it is a bug in the resolution, or a bug in the error message not mentioning the capability which it usually does I do not know, but you should anyway open an issue about it, as it is very confusing and not obvious.

But if you invoke the :collection:core:outgoingVariants task you see that “Variant apiElements” has capability cz.a-d.java.tools.collection.core:core:1.0-SNAPSHOT while “Variant jdk8ApiElements” has capability tools.collection:core:unspecified.

If I change it.capability(this.project.getGroup().toString(), this.project.getName(), this.project.getVersion().toString()) to it.capability("cz.a-d.java.tools.collection.core", this.project.getName(), "1.0-SNAPSHOT") resolution works as expected.

The problem is, that when you read the group and version, they are not set yet and currently they are not yet lazy.
If you move the group and version to gradle.properties files, they are available in time so any plugin reading those values (not just your own) will get the proper values.
So if you add collection/core/gradle.properties with content

group = cz.a-d.java.tools.collection.core

and gradle.properties with content

group = cz.a-d.java.tools
version = 1.0-SNAPSHOT

you can remove the group and version values and the resolution works as intended.

9

Thank you, solution with property files is working, I noticed version is unspecified, I even find comments in code about these properties will be lazy loaded in future. However I have not connected dots and for sure putting these values to property file as workaround for current implementation state had really not cross my mind.
I really appreciate your help , thank you. I have improved code of plugin by better configuration and added some java doc to plugin implementation to make it more readable and for now I will consider prototype finished since it is finally doing everything required by this project.
In future I plan to extract plugin into separate repository and address additional problems like changing target configuration values to properties to have same syntax, performing configuration actions when all configurations are loaded and some more features I have in my mind.
I hope it will go smoothly but if not I hope I can borrow your brain again to overcome problems I will encounter.
I have also created issue as you have suggested: Related issue
One more time thank you you saved me :slight_smile:

1 Like
10

putting these values to property file as workaround for current implementation state

As long as the values are hard-coded static values, I do not see this as work-around but as the proper way to do it actually. :slight_smile: