Why does my consumer prefer java 8 variant to java 11 variant while building with Java 11?

Pierre1 · March 3, 2021, 4:46pm

Hello,

I would like to add a java 11 variant to a java 8 library and I probably not use the correct Gradle API to do that but I cannot pinpoint where is my error.

The goals are:

to be able to publish a java library that can be executed with Java 8;
- hard constraint: that library consumes APIs that were removed in the Java 11 API (such as JAXB in javax.xml package for instance),
- soft constraint: the build should work using a Java 8 or a Java 11 toolchain a-like.
the consumers of this library may be Gradle or Maven builds;
- hard constraint: it is expected that their build tools are up-to-date.
the consumers may build their projects using a Java 8 or a Java 11 toolchain;
- hard constraint: a consumer using a Java 8 toolchain must not use dependencies required for Java 11 in its compile/runtime classpath.

I start with modeling a multi projects build with a producer and a consumer:

settings.gradle

rootProject.name = 'test'
include 'producer', 'consumer'

producer/build.gradle

plugins {
    id 'java-library'
}

java {
    registerFeature('java11') {
        usingSourceSet(sourceSets.main)
    }
    sourceCompatibility = JavaVersion.VERSION_1_8
    targetCompatibility = JavaVersion.VERSION_1_8
}

configurations {
    java11ApiElements {
        attributes {
            attribute TargetJvmVersion.TARGET_JVM_VERSION_ATTRIBUTE, 11
        }
    }
    java11RuntimeElements {
        attributes {
            attribute TargetJvmVersion.TARGET_JVM_VERSION_ATTRIBUTE, 11
        }
    }
}

The outgoing variants seem to be well configured:

~/test2$ ./gradlew :producer:outgoingVariants

> Task :producer:outgoingVariants
--------------------------------------------------
Variant apiElements
--------------------------------------------------
Description = API elements for main.

Capabilities
    - test:producer:unspecified (default capability)
Attributes
    - org.gradle.category            = library
    - org.gradle.dependency.bundling = external
    - org.gradle.jvm.version         = 8
    - org.gradle.libraryelements     = jar
    - org.gradle.usage               = java-api

Artifacts
    - build/libs/producer.jar (artifactType = jar)

Secondary variants (*)
    - Variant : classes
       - Attributes
          - org.gradle.category            = library
          - org.gradle.dependency.bundling = external
          - org.gradle.jvm.version         = 8
          - org.gradle.libraryelements     = classes
          - org.gradle.usage               = java-api
       - Artifacts
          - build/classes/java/main (artifactType = java-classes-directory)

--------------------------------------------------
Variant java11ApiElements
--------------------------------------------------
Description = API elements for feature java11

Capabilities
    - test:producer-java11:unspecified

Attributes
    - org.gradle.category            = library
    - org.gradle.dependency.bundling = external
    - org.gradle.jvm.version         = 11
    - org.gradle.libraryelements     = jar
    - org.gradle.usage               = java-api

Artifacts
    - build/libs/producer.jar (artifactType = jar)

Secondary variants (*)
    - Variant : classes
       - Attributes
          - org.gradle.category            = library
          - org.gradle.dependency.bundling = external
          - org.gradle.jvm.version         = 11
          - org.gradle.libraryelements     = classes
          - org.gradle.usage               = java-api
       - Artifacts
          - build/classes/java/main (artifactType = java-classes-directory)

--------------------------------------------------
Variant java11RuntimeElements
--------------------------------------------------
Description = Runtime elements for feature java11

Capabilities
    - test:producer-java11:unspecified

Attributes
    - org.gradle.category            = library
    - org.gradle.dependency.bundling = external
    - org.gradle.jvm.version         = 11
    - org.gradle.libraryelements     = jar
    - org.gradle.usage               = java-runtime

Artifacts
    - build/libs/producer.jar (artifactType = jar)

--------------------------------------------------
Variant runtimeElements
--------------------------------------------------
Description = Elements of runtime for main.

Capabilities
    - test:producer:unspecified (default capability)
Attributes
    - org.gradle.category            = library
    - org.gradle.dependency.bundling = external
    - org.gradle.jvm.version         = 8
    - org.gradle.libraryelements     = jar
    - org.gradle.usage               = java-runtime

Artifacts
    - build/libs/producer.jar (artifactType = jar)

Secondary variants (*)
    - Variant : classes
       - Attributes
          - org.gradle.category            = library
          - org.gradle.dependency.bundling = external
          - org.gradle.jvm.version         = 8
          - org.gradle.libraryelements     = classes
          - org.gradle.usage               = java-runtime
       - Artifacts
          - build/classes/java/main (artifactType = java-classes-directory)
    - Variant : resources
       - Attributes
          - org.gradle.category            = library
          - org.gradle.dependency.bundling = external
          - org.gradle.jvm.version         = 8
          - org.gradle.libraryelements     = resources
          - org.gradle.usage               = java-runtime
       - Artifacts
          - build/resources/main (artifactType = java-resources-directory)


(*) Secondary variants are variants created via the Configuration#getOutgoing(): ConfigurationPublications API which also participate in selection, in addition to the configuration itself.

BUILD SUCCESSFUL in 869ms
1 actionable task: 1 executed

The consumer now:

consumer/build.gradle

plugins {
    id 'java-library'
}

repositories {
    mavenCentral()
}

dependencies {
    implementation project(':producer')
}

According to the documentation Working with Variant Attributes, I expect that the consumer uses the java 11 variant because the compatibility and disambiguation rules state that:

Attribute named org.gradle.jvm.version “Defaults to the JVM version used by Gradle, lower is compatible with higher, prefers highest compatible.”

And the current VM is Java 11:

~/test2$ ./gradlew --version

------------------------------------------------------------
Gradle 6.8.2
------------------------------------------------------------

Build time:   2021-02-05 12:53:00 UTC
Revision:     b9bd4a5c6026ac52f690eaf2829ee26563cad426

Kotlin:       1.4.20
Groovy:       2.5.12
Ant:          Apache Ant(TM) version 1.10.9 compiled on September 27 2020
JVM:          11.0.10 (Oracle Corporation 11.0.10+9)
OS:           Linux 5.10.19-1-lts amd64

Yet, it picks the normal variant:

~/test2$ ./gradlew :consumer:dependencyInsight --dependency=producer

> Task :consumer:dependencyInsight
project :producer
   variant "apiElements" [
      org.gradle.category            = library
      org.gradle.dependency.bundling = external
      org.gradle.usage               = java-api
      org.gradle.libraryelements     = jar (compatible with: classes)
      org.gradle.jvm.version         = 8 (compatible with: 11)
   ]

project :producer
\--- compileClasspath

A web-based, searchable dependency report is available by adding the --scan option.

BUILD SUCCESSFUL in 843ms
1 actionable task: 1 executed

If I change the consumer.gradle to:

dependencies {
    implementation(project(':producer')) {
        capabilities {
            requireCapability("${project.group}:producer-java11:${project.version}")
        }
    }
}

Then, Gradle picks the correct variant:

~/test2$ ./gradlew :consumer:dependencyInsight --dependency=producer

> Task :consumer:dependencyInsight
project :producer
   variant "java11ApiElements" [
      org.gradle.category            = library
      org.gradle.dependency.bundling = external
      org.gradle.usage               = java-api
      org.gradle.libraryelements     = jar (compatible with: classes)
      org.gradle.jvm.version         = 11
   ]

project :producer
\--- compileClasspath

A web-based, searchable dependency report is available by adding the --scan option.

BUILD SUCCESSFUL in 1s
1 actionable task: 1 executed

I’d like to avoid adding capabilities to my dependencies as this would mean that every consumer of my lib should do the same. I would like to leverage the target vm attribute set by Gradle if possible.

The next step would be to add the missing API removed from Java 11 to the java11Implementation configurations and consume them only if the java 11 variant is picked.

I surely must have overlooked something important regarding variant definition and/or consumption but I cannot pinpoint it. A bit of help would be welcome.

johnybravo · March 9, 2021, 3:08pm

Hello, did you manage to come up with something? I’m facing the exact same problem and I’m quite surprised how hard it is to get some answer on this forum…

Pierre1 · March 9, 2021, 7:16pm

Hi,

Unfortunately I had not much time last week to dig into that issue. But the more I look at it, the more it looks like an actual bug to me.

There are few issues opened recently tagged with variants on github:

a case looking like quite similar (IMHO) to the one discussed here: Variant aware dependency resolution fails to select a secondary variant with the correct attribute · Issue #16368 · gradle/gradle · GitHub
another case, maybe a bit further but still with an apparent gap between the documentation and the actual result : Grails Views - Cannot choose between the following variants of project: productionRuntimeClasspath, runtimeElements · Issue #15721 · gradle/gradle · GitHub

So either a lot of people have misread the documentation or do not apply the concept as they should, or there is maybe a tiny bit bug in the variant resolver that produce quite a bunch of issues. To be honest I haven’t been able to pinpoint something in the source code and debugging the resolution is not easy because we can add compatibility/disambiguation rules to the resolver, but we cannot remove the initial ones sets by the java plugin, sadly.

Chris_Dore · March 10, 2021, 6:04am

I think what you both might be having trouble with is that registering a feature variant creates a new capability (essentially a new sub-GAV). This is not the same thing as creating a new outgoing variant of the original capability (the GAV of the project). @Pierre1 this is why things work when using requireCapability. I’m not an expert on this either, so I could be completely mistaken. I would agree with you if you thought the term “variant” has been overloaded a bit.

Unfortunately I do not have a direct solution on hand for you. However, I think you should not use “feature variants” and instead do something similar to the instrumented-jars example here, but without actually creating a new library elements value.

You could try attaching additional capabilities that match the project’s capabilities to the feature variant’s outgoing configs. However, I’m not sure if that lands you in hack/abuse territory.

Best of luck!

Pierre1 · March 10, 2021, 1:52pm

Thanks a lot @Chris_Dore,

Your message helped me locate the misunderstanding.

Basically if I change the producer/build.gradle to:

plugins {
    id 'java-library'
}

java {
    sourceCompatibility = JavaVersion.VERSION_1_8
    targetCompatibility = JavaVersion.VERSION_1_8
}

configurations {
    java11ApiElements {
        canBeConsumed = true
        canBeResolved = false
        attributes {
            attribute Bundling.BUNDLING_ATTRIBUTE, objects.named(Bundling, Bundling.EXTERNAL)
            attribute Category.CATEGORY_ATTRIBUTE, objects.named(Category, Category.LIBRARY)
            attribute LibraryElements.LIBRARY_ELEMENTS_ATTRIBUTE, objects.named(LibraryElements, LibraryElements.JAR)
            attribute Usage.USAGE_ATTRIBUTE, objects.named(Usage, Usage.JAVA_API)
            attribute TargetJvmVersion.TARGET_JVM_VERSION_ATTRIBUTE, 11
        }
        outgoing {
            variants {
                classes {
                    artifact(sourceSets.main.java.classesDirectory) {
                        attributes {
                            attribute LibraryElements.LIBRARY_ELEMENTS_ATTRIBUTE, objects.named(LibraryElements, LibraryElements.CLASSES)
                        }
                        type = 'java-classes-directory'
                    }
                }
            }
        }
        extendsFrom(configurations.apiElements)
    }
    java11RuntimeElements {
        canBeConsumed = true
        canBeResolved = false
        attributes {
            attribute Bundling.BUNDLING_ATTRIBUTE, objects.named(Bundling, Bundling.EXTERNAL)
            attribute Category.CATEGORY_ATTRIBUTE, objects.named(Category, Category.LIBRARY)
            attribute LibraryElements.LIBRARY_ELEMENTS_ATTRIBUTE, objects.named(LibraryElements, LibraryElements.JAR)
            attribute Usage.USAGE_ATTRIBUTE, objects.named(Usage, Usage.JAVA_RUNTIME)
            attribute TargetJvmVersion.TARGET_JVM_VERSION_ATTRIBUTE, 11
        }
        outgoing {
            variants {
                classes {
                    artifact(sourceSets.main.java.classesDirectory) {
                        attributes {
                            attribute LibraryElements.LIBRARY_ELEMENTS_ATTRIBUTE, objects.named(LibraryElements, LibraryElements.CLASSES)
                        }
                        type = 'java-classes-directory'
                    }
                }
                resources {
                    artifact(sourceSets.main.output.resourcesDir) {
                        attributes {
                            attribute LibraryElements.LIBRARY_ELEMENTS_ATTRIBUTE, objects.named(LibraryElements, LibraryElements.CLASSES)
                        }
                        type = 'java-resources-directory'
                    }
                }
            }
        }
        extendsFrom(configurations.runtimeElements)
    }
}

Then I get the expected outgoing variants, and the consumer selects the most appropriate variant based on the version of the JVM running the build. The capability is no longer required in the consumer/build.gradle:

plugins {
    id 'java-library'
}

repositories {
    mavenCentral()
}

dependencies {
    implementation project(':producer')
}

~/test2$ JAVA_HOME=~/lib/jdk1.8.0_202/ ./gradlew :consumer:dependencyInsight --dependency=producer

> Task :consumer:dependencyInsight
project :producer
   variant "apiElements" [
      org.gradle.category            = library
      org.gradle.dependency.bundling = external
      org.gradle.usage               = java-api
      org.gradle.libraryelements     = jar (compatible with: classes)
      org.gradle.jvm.version         = 8
   ]

project :producer
\--- compileClasspath

A web-based, searchable dependency report is available by adding the --scan option.

BUILD SUCCESSFUL in 998ms
1 actionable task: 1 executed
~/test2$ ./gradlew :consumer:dependencyInsight --dependency=producer

> Task :consumer:dependencyInsight
project :producer
   variant "java11ApiElements" [
      org.gradle.dependency.bundling = external
      org.gradle.category            = library
      org.gradle.libraryelements     = jar (compatible with: classes)
      org.gradle.usage               = java-api
      org.gradle.jvm.version         = 11
   ]

project :producer
\--- compileClasspath

A web-based, searchable dependency report is available by adding the --scan option.

BUILD SUCCESSFUL in 1s
1 actionable task: 1 executed

I am quite baffled by the amount of configuration needed to model that use case. I would have expected a shorter way to express this need. Still that gives me a good starting way to add new configurations receiving dependencies required for java 11 only.

Thank you very much for the help

johnybravo · March 10, 2021, 5:48pm

Only works when the consumer project declares the producer as implementation project(':producer'). When I publish my producer to maven local and try to consume it by implementation 'org.example:producer:1.0-SNAPSHOT' it always goes for apiElements variant… Should I somehow register those variants it in my publishing task or something?

Pierre1 · March 10, 2021, 7:27pm

Hi @johnybravo ,

There is a method, mapToMavenScope that may be called to add outgoing variants to a maven scope.

Something like this should do the trick (untested):

AdhocComponentWithVariants javaComponent = (AdhocComponentWithVariants) project.components.findByName("java")
javaComponent.withVariantsFromConfiguration(configurations.java11ApiElements) {
	it.mapToMavenScope('runtime') // or compile if you want these to leak to the consumer compile classpath
}
javaComponent.withVariantsFromConfiguration(configurations.java11RuntimeElements) {
	it.mapToMavenScope('runtime')
}

As far as I am concerned, I am fine with the default behavior. I just need to tweak the pom generated by the maven-publish plugin to add java11 only dependencies in a Maven profile activated by jdk >= 11.

That’s quite close to plumbing, but should be doable with few adjustments.

johnybravo · March 10, 2021, 10:21pm

It’s javaComponent.addVariantsFromConfiguration(...) and it finally works! Thank you very much!

ljacomet · April 2, 2021, 3:07pm

Hey,

Something that might not be clear enough from the documentation: when adding a feature variant, it is given a default capability only when no explicit capability is given.

This means that the expected result should be obtained with doing:

    registerFeature('java11') {
        usingSourceSet(sourceSets.main)
        capability(<group>, <project>, <version>)
    }

and still leverage the registerFeature API.

That example in the documentation illustrates this.

PhoenixFlash · November 24, 2021, 4:29pm

Does this hold true for vice-versa as well? Meaning a project built on Java 11 can produce a target for Java 8 as well? @Pierre1 @Chris_Dore Please suggest?

Vampire · November 25, 2021, 3:35pm

Sure, you can use release or even better use the Java toolchains feature.