Dependency resolution with variant-aware logic

Help/Discuss
1

The what:

I am trying to make a Gradle plugin that manages an extra source between multiple projects. For simplicity’s sake, I made a producer and a consumer plugin, though they really could have been the same plugin class.

I am trying to make it so the dependencies of my java project already declared for compilation are used for the inputs to the consumer

The why:

The big goal with this plugin is to avoid as much manual configuration as possible for developers. I would rather the plugin be doing more heavy lifting, even if the plugin becomes extra complex as a result.

Details:

The producer plugins’ job is to create a Zip containing the generated code and publish the zip file. I make use of the distribution plugin to do that. I set an attribute to distinguish the zip file as a specific variant. The producer seems to be working well, I produce a zip, it is published, and attached to the Java software component. The Gradle module metadata file shows my zip file has the correct variant information.

The consumer’s job is to extract the zip file from the producer, and then run a code generation task. I have a dependsOn declaration between the code generation task and the extraction task. I managed to select the artifact dependencies I needed by using a lenient artifact view from the configuration I want to pull from. This works great in a repo containing a single project. It unfortunately falls apart when a consumer wants to use the output of a project that applied the producer plugin. The extraction task in the consumer tries to run before the distZip task runs in the producer, and so the consumer fails with an error saying the zip file doesn’t exist.

I know it’s bad practice (and maybe illegal in the version of Gradle I am using, 8.10) to depend on tasks from another project, but that’s effectively what I want to happen.

I did see the section in the docs saying to make a configuration in the producer and have the consumer depend on that project for that configuration.

I couldn’t get that to work programmatically and again, I don’t want developers to have to duplicate their dependencies.

Any help would be appreciated, and I’m open to feedback on the approach as well as just specific how-to info. Thanks in advance!

2

The producer plugins’ job is to create a Zip containing the generated code and publish the zip file.

Why should you publish source code as zip file?
That sounds like you then compile it in the consumers and include it in the consumers.
Which means that every consumer has an own version of those classes.
This is usually a pretty bad idea and latest will get you into trouble when those consumers are going to be used in the same classpath. Even worse if multiple consumers are landing in the same modulepath, because then different modules are not even allowed to have different classes in the same package, let alone the same classes.

Usually you would generate the code in the producer, compile it in the producer, and provide the compiled classes in the jar of the producer that consumers then simply depend on, then also no complex setup is necessary.

If you actually do use the class files from the jar of the producer and just need the sources of the producer in the consumer to do the mentioned code-generation in the consumer build, also no custom artifact should be necessary, you could simply use withSourcesJar() which packages up the sources as jar and publishes them as a proper feature variant on which you then can depend on in the consumer if needed.

I have a dependsOn declaration between the code generation task and the extraction task.

Any manual dependsOn where on the left-hand side is not a lifecycle task is a code smell and usually a sign of you doing something incorrectly. Usually it means, you are not properly wiring task outputs to task inputs. If this is done correctly, the task dependency is automatically there implicitly. So if for example your extraction task properly declares its outputs and the generation task is properly declaring its inputs from the outputs of the extraction task, the task dependency should automatically be there.

Besides that, for such an extraction I would usually not use a task, but an artifact transform. An artifact transform that does unzip the dependency artifacts is also the use-case used in the documentation for artifact transforms.

by using a lenient artifact view

Be cautious with that! “lenient artifact view” means that each and every failure is ignored, including download errors, resolution errors, everything. If you intend to do “give me this variant for all dependencies that have it”, this leniency is the default behavior of an artifact view and you should not set isLenient to true for that.

It unfortunately falls apart when a consumer wants to use the output of a project that applied the producer plugin. The extraction task in the consumer tries to run before the distZip task runs in the producer, and so the consumer fails with an error saying the zip file doesn’t exist.

For this we are at a point where an abstract description is not enough. If the setup would be proper, it would work. But without seeing an MCVE it is impossible to tell where you might have slipped in a bug that causes this behavior.

I know it’s bad practice (and maybe illegal in the version of Gradle I am using, 8.10) to depend on tasks from another project, but that’s effectively what I want to happen.

It is highly discouraged and bad practice to do so explicitly, yes.
But if you for example depend from one project on another project, you also implicitly depend on the jar task that produces that jar and the tasks it depends on and that is fine.
If the setup would be done properly - as I said - it would just work as expected, but without seeing the code it is impossible to hint at where the problem might be.

I did see the section in the docs saying to make a configuration in the producer and have the consumer depend on that project for that configuration.

Actually, depending on a certain configuration in another project is the small brother of declaring a proper variant, and it is discouraged by the Gradle folks to use that facility nowadays, but to use proper variant-aware resolution like you tried to, there is probably just some error in your setup.

3

Thanks for all this, it’s a ton of info. I am going to play with it a bit more and then get a MCVE back to you.

I was considering making use of an artifact transform; I will give that shot.

One note: our code generation task doesn’t just involve Java. We generate one or more files based on the contents of a project, and then use that to generate java source code (within the same project). The first file is what I need to package up for later consumption, as consuming projects need the file to build up references. I can’t change that behavior, unfortunately.

There’s value in having the source separate from this file, so using the source jar likely isn’t appropriate.

1 Like
4

Hello, sorry for the long delay. I’m trying to swap to your recommendations, and still not having much luck. So, I implemented the UnZip transform. I more or less follow the example from the docs. It doesn’t seem to get triggered, which again, from the docs, seems to indicate I have no input artifacts.

I’m coming this time with an MCVE for the consumer plugin. As I said before, the producer plugin appears to be working as intended.

producer plugin:

public void apply(final Project project) {
    project.getPlugins().apply(JavaPlugin.class);

    //Create a task that extracts a tarball (task details omitted, likely should just be an artifact transform)
    project.getTasks().register("extractTarball", Copy.class, copySepc -> {...});

    Provider<Directory> unzipLocation = project.getLayout().getBuildDirectory().dir("unzip-location");

    //Create the variant objects
    LibraryElements producedVariant = project.getObjects().named(LibraryElements.class, "produced");
    LibraryElements unzippedProducedVariant = project.getObjects().named(LibraryElements.class, "produced-unzipped");

    //Register Artifact Transform
    project.getDependencies().registerTransform(UnzipTransform.class, transformSpec -> {
        transformSpec.getFrom().attribute(LibraryElements.LIBRARY_ELEMENTS_ATTRIBUTE, producedVariant);
        transformSpec.getTo().attribute(LibraryElements.LIBRARY_ELEMENTS_ATTRIBUTE, unzippedProducedVariant);
        transformSpec.getParameters().setOutputDirectory(unzipLocation.get().getAsFile());
    });

    //Register code generation task
    project.getTasks().register("generateCode", Exec.class, taskSpec -> {
        //Needs to select the unzipped produced variant of all dependencies in implementation and api configurations of the Java project we're being applied to.
        ArtifactView view = project.getConfigurations().named("compileClasspath").get().getIncoming().artifactView(viewConfiguration -> {
            viewConfig.attributes(attrContainer -> attrContainer.attribute(LibraryElements.LIBRARY_ELEMENTS_ATTRIBUTE, unzippedProducedVariant));
        });
        //With my understanding of how this works, something like below should cause Gradle to trigger my artifact transform.
        taskSpec.getInputs().files(view.getFiles().getFiles());

        // Do something with the unzippedLocation directory (details omitted)
    });
}

So in java projects where I apply this, I want to select the produced variant for any dependencies in the api or implementation configurations (as per Java plugin). Note, there are no explicit dependencies on the variant in question:

java consumer project’s build.gradle:

dependencies {
    //Select produced variant artifact
    api project(':LocalApiProjectThatHasProducedVariant')
    implementation project(':LocalImplementationProjectThatHasProducedVariant')

    //Nothing to select
    api project(':LocalApiProjectWithoutProducedVariant')
    implementation project(':LocalImplementationProjectWithoutProducedVariant')

    //Select produced variant artifact
    api 'group:ArtifactApiDependencyWithAvailableProducedVariant:version'
    implementation 'group:ArtifactImplementationDependencyWithAvailableProducedVariant:version'

    //Nothing to select
    api 'group:ArtifactApiDependencyWithoutAvailableProducedVariant:version'
    implementation 'group:ArtifactImplementationDependencyWithoutAvailableProducedVariant:version'
}

Hopefully this is enough information to get a bit of help - I’m pretty stumped!

5

I’m coming this time with an MCVE for the consumer plugin.

Where is it?
C means complete, a small excerpt from a code file is not really C. :wink:

producer plugin:

An artifact transform is a consumer thing.
But you register it at the producer, so you probably should follow the artifact transform docs more closely before trying to adapt it to your case, so that you learn how to use them. :slight_smile:

Also, never get() a Property at configuration time or you again introduce race conditions as the Property could be changed after you have read it. If you would want to give a directory to the transform parameters, have a DirectoryProperty there and set it without get-ing another Property.

Besides that it does not make sense at all to give an output directory to an artifact transform.
Again, if you want to use artifact transforms, better first follow the docs for them more closely without trying to apply it to your situation, because it seems you really majorly misunderstood how to use them.

And also getting a Set<File> from a FileCollection (the second getFiles()) at configuration time is similar bad as get()ing a Provider, especially when it is totally unnecessary.

6

Hello again.

I apologize; I mislabelled the plugin code. That is indeed the consumer plugin code. I appreciate the comments surrounding the use of the .get() for the providers, and have swapped to try and avoid that. The fact that I felt the need to do that is probably caused by the root cause of why this isn’t working.

I’ve isolated the problem to a bit narrower in scope. In the end, I need to take all of the extracted zip files, and copy the contents to a single directory (which was the reason I was misusing the Artifact Transform).

public void apply(final Project project) {
    Provider<Directory> unzipLocation = project.getLayout().getBuildDirectory().dir("unzip-location");

    //Create the variant objects
    LibraryElements producedVariant = project.getObjects().named(LibraryElements.class, "produced");
    LibraryElements unzippedProducedVariant = project.getObjects().named(LibraryElements.class, "produced-unzipped");

    //Register Artifact Transform
    project.getDependencies().registerTransform(UnzipTransform.class, transformSpec -> {
        transformSpec.getFrom().attribute(LibraryElements.LIBRARY_ELEMENTS_ATTRIBUTE, producedVariant);
        transformSpec.getTo().attribute(LibraryElements.LIBRARY_ELEMENTS_ATTRIBUTE, unzippedProducedVariant);
    });

    //Needs to select the unzipped produced variant of all dependencies in implementation and api configurations of the Java project we're being applied to.
    ArtifactView view = project.getConfigurations().named("compileClasspath").get().getIncoming().artifactView(viewConfiguration -> {
        viewConfig.attributes(attrContainer -> attrContainer.attribute(LibraryElements.LIBRARY_ELEMENTS_ATTRIBUTE, unzippedProducedVariant));
    });

    project.getTasks("collectProducedCode", Copy.class, copySpec -> {
        copySpec.from(view)
        copySpec.into(unzipLocation)
    });
}

So, when I apply this plugin to a project, and run the collectProducedCode task, it doesn’t do anything, and says there is no source. This is despite the fact that the project depends on other artifacts and projects.

From what I can tell, depending on a project doesn’t cause this copy task and/or artifact view to be able to resolve the non-java variant.

7

Besides that you seldomly want to use Copy or copy but usually Sync or sync to remove stale files, still an MCVE would be helpful to show your current complete state. It is hard to just read a small excerpt and guess what the problem might be especially with a complex topic like variants, artifacts, and artifact transforms. Even if I have the time and mood to look at your problem in-depth, I would not try to rebuild the situation from scratch, because it would most probably not be the exact same situation.

8

I’ve attached what should be working builds of both the plugin code AND projects that should implement them. For posterity, I’ll include it all here. The zips also contain build.gradle and setting.gradle files I don’t mention here.

producer plugin:

import org.gradle.api.Plugin;
import org.gradle.api.Project;
import org.gradle.api.artifacts.Configuration;
import org.gradle.api.attributes.LibraryElements;
import org.gradle.api.component.AdhocComponentWithVariants;
import org.gradle.api.distribution.Distribution;
import org.gradle.api.distribution.DistributionContainer;
import org.gradle.api.distribution.plugins.DistributionPlugin;
import org.gradle.api.plugins.BasePlugin;
import org.gradle.api.tasks.TaskProvider;
import org.gradle.api.tasks.bundling.AbstractArchiveTask;

public class ProducerPlugin implements Plugin<Project> {
    public void apply(Project project) {
        project.getPlugins().apply(BasePlugin.class);
        project.getPlugins().apply(DistributionPlugin.class);
        
        Distribution dist = project.getExtensions().getByType(DistributionContainer.class).maybeCreate("producedVariant");
        dist.getContents().from("myGeneratedCodePath", copySpec -> copySpec.include("**/*.generatedCode"));
        dist.getContents().into("/");
        dist.getContents().setIncludeEmptyDirs(false);
        
        Configuration distConfiguration = project.getConfigurations().maybeCreate("producedVariantDistribution");
        distConfiguration.setCanBeResolved(true);
        distConfiguration.setCanBeConsumed(true);
        
        //Label this distributions' configuration as a "produced" variant
        LibraryElements producedElements = project.getObjects().named(LibraryElements.class, "produced");
        distConfiguration.attributes(attrContainer -> {
            attrContainer.attribute(LibraryElements.LIBRARY_ELEMENTS_ATTRIBUTE, producedElements);
        });
        
        AdhocComponentWithVariants javaComponent = (AdhocComponentWithVariants) project.getComponents().getByName("java");
        
        //Wire everything together
        TaskProvider<AbstractArchiveTask> zipTask = project.getTasks().named(dist.getName() + "DistZip");
        project.getArtifacts().add(distConfiguration.getName(), zipTask.get());
        project.getTasks().named("assemble").configure(assemble -> assemble.dependsOn(zipTask));
        
        javaComponent.addVariantsFromConfiguration(distConfiguration, details -> {
            details.mapToMavenScope("compile");
            details.mapToOptional();
        });
    }
}

Consumer plugin code:

import org.gradle.api.Plugin;
import org.gradle.api.Project;
import org.gradle.api.artifacts.ArtifactView;
import org.gradle.api.attributes.LibraryElements;
import org.gradle.api.file.Directory;
import org.gradle.api.provider.Provider;
import org.gradle.api.tasks.Sync;

public class ConsumerPlugin implements Plugin<Project> {
    public void apply(Project project) {
        Provider<Directory> unzipLocation = project.getLayout().getBuildDirectory().dir("unzip-location");
        
        LibraryElements producedVariant = project.getObjects().named(LibraryElements.class, "produced");
        LibraryElements unzippedProducedVariant = project.getObjects().named(LibraryElements.class, "produced-unzipped");
        
        project.getDependencies().registerTransform(UnzipTransform.class, transform -> {
            transform.getFrom().attribute(LibraryElements.LIBRARY_ELEMENTS_ATTRIBUTE, producedVariant);
            transform.getTo().attribute(LibraryElements.LIBRARY_ELEMENTS_ATTRIBUTE, unzippedProducedVariant);
        });
        
        ArtifactView view = project.getConfigurations().named("compileClasspath").get().getIncoming().artifactView(viewConfiguration -> {
            viewConfiguration.attributes(attrContainer -> attrContainer.attribute(LibraryElements.LIBRARY_ELEMENTS_ATTRIBUTE, unzippedProducedVariant));
        });
        
        project.getTasks().register("collectProducedCode", Sync.class, sync -> {
            sync.from(view.getFiles());
            sync.into(unzipLocation);
        });
    }
}

A sample producer project’s usage of the plugin:

plugins {
   id 'java'
   id 'org.example.producer' version '1.0-SNAPSHOT'
}

group = 'org.example'
version = '1.0-SNAPSHOT'

dependencies {}

A sample consumer project’s usage of the plugin:

plugins {
   id 'java'
   id 'org.example.consumer' version '1.0-SNAPSHOT'
}

group = 'org.example'
version = '1.0-SNAPSHOT'

dependencies {
    implementation project(':TestProject1')
}

In my case, I’ve got a sample layout in the producer’s project:

myGeneratedCodePath/test1/test.generatedCode

myGeneratedCodePath/test2/badfile.txt

After running gradle TestProject1:assemble, I see the distributions get made, and the have the correct contents. I checked gradle TestProject1:outgoingVariants, which I was relatively surprised to see had the tar and zip for the same distribution (I’ll have to remember to go disable the tar task, I suppose), but also the normal jar for the project.

So, I think the producer plugin is working because of this.

Next, to test the consumer, I ran gradle collectProducedCode. It does not run the task as the task is marked with NO SOURCE.

The NO SOURCE is what is confusing me. Appreciate any and all help; I hope you can see that I’m quite interested in feedback.

PluginsMCVE.zip (5.7 KB)

ProjectMCVE.zip (4.0 KB)

9

I spent quite a bit of time yesterday scouring as much info as I could from the docs. I found two things that I’m doing wrong.

First, if I want a different resolution result for the artifact, I think I need to use .withVariantReselection().

And second, having a configuration both consumable and resolvable are not acceptable. Indeed, I got different results when marking the distConfiguration from my MCVE as only consumable.

With both of these in place, things are a bit different. Now, I get a proper variant when running gradle TestProject1:outgoingVariants. I had mistaken the listing of the artifact I wanted as the variant somehow (perhaps too little coffee).

Unfortunately, success ends here, as for some reason my artifact view is returning the source code. I am not certain why this is happening.

10

I’ve attached what should be working builds

Not at all unfortunately.
They don’t even include the Gradle wrapper files, so the Gradle version is not defined.
Using 8.10 as you mentioned above still does not really help, because the code does not compile due to syntax errors.

11

I have run the precise code in the provided ZIPs. Also generated the gradle wrapper, which isn’t something I use typically. I confirmed the output is the same as my previous reply.

I end up with the source code for the Java project in the location I want to unzip the generated code.

PluginsMCVE.zip (66.5 KB)

ProjectMCVE.zip (58.1 KB)

12

I have run the precise code in the provided ZIPs.

In the original ones?
That is impossible unless you use a Java compiler that supports Syntax that is not valid in the Java language.
There was at least the UnzipTransform.java file which did contain invalid Java syntax.
In the now attached files this is fixed.

Also generated the gradle wrapper, which isn’t something I use typically.

I consider each and every build and be it tiny to have a build bug if the 4 wrapper files are not included.
They control the exact Gradle version that is used to execute the build. Without it, the build can work or not depending on which Gradle version is used, or also fail loudly or just silently misbehave.

I end up with the source code for the Java project in the location I want to unzip the generated code.

At Variant Selection and Attribute Matching you find the matching algorithm.

You configured the artifact view to do variant reselection, so the selection algorithm kicks in.
Be aware, that during variant selection artifact transforms do not kick in, as they are artifact transforms, not variant transforms (which do not exist yet, you can find feature requests about it).

Following the selection algorithm closely with the information you got from outgoingVariants, the mainSourcesElements variant is matched as it is the closest match with only providing attributes that were not requested and not providing attributes that were requested. So even if artifact transforms would potentially kick in, they would not as a matching variant is already found. If you want to use the libraryelements attribute, you should probably also set the “parent” attribute “category” to “library”, then the sources would no langer match in the matching algorithm.

What you for example could do is instead a configuration that extends compileClasspath and defines that it wants the produced variant and on that an artifact view that then leverages the artifact transform, but that would then probably fail if you have in the compileClasspath dependencies that do not have the produced variant.

Alternatively, you should probably instead transform the artifactType like shown in the exact example I referred you to in the Gralde docs just with ZIP_TYPE instead of JAR_TYPE to DIRECTORY_TYPE, because that is what your transform does, it does not change the contents, just the artifact appearance. Because that attribute is an artifact-level attribute so anyway does not participate in variant selection you could then use an artifact view with variant reselection that requests the produced variant with artifact type DIRECTORY_TYPE. The variant reselection will then find the produced variant and after that the artifact transform can do the ZIP_TYPE => DIRECTORY_TYPE transformation and it will work no matter which dependencies you have as an artifact view ignores dependencies where it cannot be fulfilled.

13

As you say, I meant I ran the code I had newly uploaded in the same reply.

That’s fair, and probably appropriate in forums like the one we find ourselves in currently. I happen to work in an environment where getting the wrong version of Gradle simply isn’t possible.

Yeah, I see what you’re saying here. This definitely violates what I would like to achieve.

I’ll be honest, I’m not sure I’ve processed the implementation details here fully (that’s fine, not really your problem). This at least seems to satisfy what I want. I guess, I need to decide if its worth waiting for variant transforms, which I would think would be cleaner (maybe?), or use this.

14

As you say, I meant I ran the code I had newly uploaded in the same reply.

Ah, ok, then I misinterpreted, sorry.

I happen to work in an environment where getting the wrong version of Gradle simply isn’t possible.

Well, it is a strong recommendation by the Gradle folks.
If it is not necessary for your environment: :man_shrugging:
But if you share something, better add it latest then. :slight_smile:

I guess, I need to decide if its worth waiting for variant transforms

No. As far as I know there is no work in that area and it is also not said that it will ever come.

which I would think would be cleaner (maybe?)

I don’t think so.
Without variant transfer, a valid variant must be found for each dependency without any transformations to resolve a configuration.
An artifact view can then select a sibling variant or also do artifact transformations.
But the source artifact for the transform must be found by resolving the configuration or by reselecting a variant.

From what you showed so far, I think using reselction to select the produced variant and then a transform to unzip it is indeed the way to go.
Because your transform does not do a variant transform, it does not transform the semantic content of the artifact, it just transforms the physical representation of the artifact - the artifact type.

1 Like