Advise regarding publishing dependent artifacts

Hi,
I’d like to ask for some advise regarding the following problem I’m trying to solve in Gradle.

We have a proprietary Gradle plugin which adds certain native compiliation functionality to a single project. Such projects, which have this plugin applied, have a configuration for the produced binaries as well as another configuration for the binaries’ compilation output. Other projects can depend on these native projects in turn and consume their native binary artifacts.

When publishing a project which has other native projects as dependencies, I’d like to be able to somehow publish the compilation output artifacts for all the projects on which this project depends on.
Ideally though, I’d like to collect all these relevant compilation output artifacts into a single artifact and publish that. The benefit to me with this kind of solution is that I can create a package per application containing all compilation output relevant for that applciation (as opposed to let’s say publishing every native artifact’s compilation output separately which would be difficult to keep track of).

I’m thinking about creating a BuildService into which the plugin which adds native compilation functionality registers its compilation output artifacts and then creating a new plugin which is appied to the leaf project which gets all the registered artifacts from the BuildService and creates a task to package them all in a single artifact as well as a publication to publish them.

When trying naively to register configurations with such a BuildService and then to read them in a different context, Gradle is reporting a “Resolution of the configuration x was attempted from a context different than the project context” warning. So I’m not sure I’m on the right track with this.

This seems a bit involved. Any ideas, suggestions and such would be much appreciated.
Thanks a lot.

I’m not doing native, but what you describe sounds like in the Java world creating a bad-practice fat JAR and publishing that, which is practically never a good idea.

Why would it be difficult if each project publishes its compilation output?

As long as you publish them as proper variants with identifiable attribute set, it should not be a problem to consume them in bulk.

If those variants also have the necessary dependencies declared, you can use a configuration that requests the according attributes and get the according variant of the projects.

Or if not having the dependencies declared on those variants (like how it is for the sources variant in Java land), you can still use an artifact view to get the proper variant for all those projects.

I know of and understand only some of the concepts you mention, so it sounds like I need to do some more reading.

Perhaps my compilation outputs should be variants and I should then be using an artifact view. I’ll look into it.

Thanks @Vampire !

1 Like

I’ve migrated my plugins to use variants which was very interesting and I’ve upgraded my project to use Gradle 8.14 which includes artifact views.
Now, I’m trying the following:

tasks.register<ResolveFiles>("resolveListings") {
    files.from(packageRemote.incoming.artifactView {
        withVariantReselection()
        attributes {
            attribute(Usage.USAGE_ATTRIBUTE, objects.named(Usage::class.java, "zos-native"))
            attribute(Category.CATEGORY_ATTRIBUTE, objects.named(Category::class.java, Category.LIBRARY))
            //attribute(LibraryElements.LIBRARY_ELEMENTS_ATTRIBUTE, objects.named(LibraryElements::class.java, "objectsListings"))
        }
    }.files)
}

As is, the resolveListings task fails due to:

   > The consumer was configured to find attribute 'org.gradle.category' with value 'library', attribute 'org.gradle.usage' with value 'zos-native'. There are several available matching variants of project :model9-zos:model9-zos-jvm-launcher
     The only attribute distinguishing these variants is 'org.gradle.libraryelements'. Add this attribute to the consumer's configuration to resolve the ambiguity:
       - Value: 'loadModules' selects variant: 'loadModules'
       - Value: 'loadModulesListings' selects variant: 'loadModulesListings'
       - Value: 'macros' selects variant: 'macros'
       - Value: 'objects' selects variant: 'objects'
       - Value: 'objectsListings' selects variant: 'objectsListings'

This is expected.
But when I uncomment the last attribute which should distinguish the required variant, the task fails due to:

Unable to find a variant with the requested capabilities: []:

I’m creating the configuration as:

val packageRemote = configurations.create("packageRemote")

Any idea what I’m doing wrong?
Thanks

Just doing configurations.create actually (for backwards compatibility reasons) creates a configuration with legacy setup, you should either set isResolvable to false, or right away use the convenience helper configurations.consumable(...).

But that’s just a side-note and should not cause the problem.

From what you provided I don’t see what the problem might be.
What is the full error message?
And what does :theProducerProject:outgoingVariants task display?

Thanks for the side-note, good to know.

I have something like this in the dependencies:

packageRemote(project(":model9-zos:model9-zos-below-the-bar-storage", "dlls"))

And this producer project’s outgoingVariants are:

> Task :model9-zos:model9-zos-below-the-bar-storage:outgoingVariants
--------------------------------------------------
Variant apiElements
--------------------------------------------------
API elements for the 'main' feature.

Capabilities
    - zosbackupserver:model9-zos-below-the-bar-storage:4.2.0-SNAPSHOT (default capability)
Attributes
    - org.gradle.category            = library
    - org.gradle.dependency.bundling = external
    - org.gradle.jvm.version         = 8
    - org.gradle.libraryelements     = jar
    - org.gradle.usage               = java-api
Artifacts
    - build/libs/model9-zos-below-the-bar-storage-4.2.0-SNAPSHOT.jar (artifactType = jar)

Secondary Variants (*)

    --------------------------------------------------
    Secondary Variant classes
    --------------------------------------------------
    Directories containing compiled class files for main.

    Attributes
        - org.gradle.category            = library
        - org.gradle.dependency.bundling = external
        - org.gradle.jvm.version         = 8
        - org.gradle.libraryelements     = classes
        - org.gradle.usage               = java-api
    Artifacts
        - build/classes/java/main (artifactType = java-classes-directory)

--------------------------------------------------
Variant archives
--------------------------------------------------
Configuration for archive artifacts.

Capabilities
    - zosbackupserver:model9-zos-below-the-bar-storage:4.2.0-SNAPSHOT (default capability)
Artifacts
    - build/dist/model9-zos-below-the-bar-storage-dlls-headers.tar.gz
    - build/dist/model9-zos-below-the-bar-storage-dlls-listings.tar.gz
    - build/dist/model9-zos-below-the-bar-storage-dlls-sources.tar.gz
    - build/dist/model9-zos-below-the-bar-storage-dlls.tar.gz
    - build/dist/model9-zos-below-the-bar-storage-dlls.tar.gz (artifactType = gz)
    - build/dist/model9-zos-below-the-bar-storage-main-hlasm-load-modules-listings.tar.gz
    - build/dist/model9-zos-below-the-bar-storage-main-hlasm-load-modules.tar.gz
    - build/dist/model9-zos-below-the-bar-storage-main-hlasm-macros.tar.gz
    - build/dist/model9-zos-below-the-bar-storage-main-hlasm-objects-listings.tar.gz
    - build/dist/model9-zos-below-the-bar-storage-main-hlasm-objects.tar.gz
    - build/dist/model9-zos-below-the-bar-storage-main-hlasm-sources.tar.gz
    - build/dist/model9-zos-below-the-bar-storage-test-hlasm-load-modules-listings.tar.gz
    - build/dist/model9-zos-below-the-bar-storage-test-hlasm-load-modules.tar.gz
    - build/dist/model9-zos-below-the-bar-storage-test-hlasm-macros.tar.gz
    - build/dist/model9-zos-below-the-bar-storage-test-hlasm-objects-listings.tar.gz
    - build/dist/model9-zos-below-the-bar-storage-test-hlasm-objects.tar.gz
    - build/dist/model9-zos-below-the-bar-storage-test-hlasm-sources.tar.gz
    - build/libs/model9-zos-below-the-bar-storage-4.2.0-SNAPSHOT.jar (artifactType = jar)

--------------------------------------------------
Variant coverageDataElementsForTest (i)
--------------------------------------------------
Binary results containing Jacoco test coverage for all targets in the 'test' Test Suite.

Capabilities
    - zosbackupserver:model9-zos-below-the-bar-storage:4.2.0-SNAPSHOT (default capability)
Attributes
    - org.gradle.category         = verification
    - org.gradle.testsuite.name   = test
    - org.gradle.verificationtype = jacoco-coverage
Artifacts
    - build/jacoco/test.exec (artifactType = binary)

--------------------------------------------------
Variant default
--------------------------------------------------
Configuration for default artifacts.

Capabilities
    - zosbackupserver:model9-zos-below-the-bar-storage:4.2.0-SNAPSHOT (default capability)
Artifacts
    - build/libs/model9-zos-below-the-bar-storage-4.2.0-SNAPSHOT.jar (artifactType = jar)

--------------------------------------------------
Variant dllHeaders
--------------------------------------------------

Capabilities
    - zosbackupserver:model9-zos-below-the-bar-storage:4.2.0-SNAPSHOT (default capability)
Attributes
    - org.gradle.category        = library
    - org.gradle.libraryelements = dllHeaders
    - org.gradle.usage           = zos-native
Artifacts
    - build/dist/model9-zos-below-the-bar-storage-dlls-headers.tar.gz

--------------------------------------------------
Variant dllSources
--------------------------------------------------

Capabilities
    - zosbackupserver:model9-zos-below-the-bar-storage:4.2.0-SNAPSHOT (default capability)
Attributes
    - org.gradle.category = library
    - org.gradle.docstype = sources
    - org.gradle.usage    = zos-native
Artifacts
    - build/dist/model9-zos-below-the-bar-storage-dlls-sources.tar.gz

--------------------------------------------------
Variant dlls
--------------------------------------------------

Capabilities
    - zosbackupserver:model9-zos-below-the-bar-storage:4.2.0-SNAPSHOT (default capability)
Attributes
    - org.gradle.category        = library
    - org.gradle.libraryelements = dlls
    - org.gradle.usage           = zos-native
Artifacts
    - build/dist/model9-zos-below-the-bar-storage-dlls.tar.gz
    - build/dist/model9-zos-below-the-bar-storage-dlls.tar.gz (artifactType = gz)

--------------------------------------------------
Variant loadModulesListings
--------------------------------------------------

Capabilities
    - zosbackupserver:model9-zos-below-the-bar-storage:4.2.0-SNAPSHOT (default capability)
Attributes
    - org.gradle.category        = library
    - org.gradle.docstype        = listings
    - org.gradle.libraryelements = loadModulesListings
    - org.gradle.usage           = zos-native
Artifacts
    - build/dist/model9-zos-below-the-bar-storage-main-hlasm-load-modules-listings.tar.gz

--------------------------------------------------
Variant macros
--------------------------------------------------

Capabilities
    - zosbackupserver:model9-zos-below-the-bar-storage:4.2.0-SNAPSHOT (default capability)
Attributes
    - org.gradle.category        = library
    - org.gradle.libraryelements = macros
    - org.gradle.usage           = zos-native
Artifacts
    - build/dist/model9-zos-below-the-bar-storage-main-hlasm-macros.tar.gz

--------------------------------------------------
Variant mainSourceElements (i)
--------------------------------------------------
List of source directories contained in the Main SourceSet.

Capabilities
    - zosbackupserver:model9-zos-below-the-bar-storage:4.2.0-SNAPSHOT (default capability)
Attributes
    - org.gradle.category            = verification
    - org.gradle.dependency.bundling = external
    - org.gradle.verificationtype    = main-sources
Artifacts
    - src/main/java (artifactType = directory)
    - src/main/resources (artifactType = directory)

--------------------------------------------------
Variant objects
--------------------------------------------------

Capabilities
    - zosbackupserver:model9-zos-below-the-bar-storage:4.2.0-SNAPSHOT (default capability)
Attributes
    - org.gradle.category        = library
    - org.gradle.libraryelements = objects
    - org.gradle.usage           = zos-native
Artifacts
    - build/dist/model9-zos-below-the-bar-storage-main-hlasm-objects.tar.gz

--------------------------------------------------
Variant objectsListings
--------------------------------------------------

Capabilities
    - zosbackupserver:model9-zos-below-the-bar-storage:4.2.0-SNAPSHOT (default capability)
Attributes
    - org.gradle.category        = library
    - org.gradle.docstype        = listings
    - org.gradle.libraryelements = objectsListings
    - org.gradle.usage           = zos-native
Artifacts
    - build/dist/model9-zos-below-the-bar-storage-main-hlasm-objects-listings.tar.gz

--------------------------------------------------
Variant runtimeElements
--------------------------------------------------
Runtime elements for the 'main' feature.

Capabilities
    - zosbackupserver:model9-zos-below-the-bar-storage:4.2.0-SNAPSHOT (default capability)
Attributes
    - org.gradle.category            = library
    - org.gradle.dependency.bundling = external
    - org.gradle.jvm.version         = 8
    - org.gradle.libraryelements     = jar
    - org.gradle.usage               = java-runtime
Artifacts
    - build/libs/model9-zos-below-the-bar-storage-4.2.0-SNAPSHOT.jar (artifactType = jar)

Secondary Variants (*)

    --------------------------------------------------
    Secondary Variant classes
    --------------------------------------------------
    Directories containing compiled class files for main.

    Attributes
        - org.gradle.category            = library
        - org.gradle.dependency.bundling = external
        - org.gradle.jvm.version         = 8
        - org.gradle.libraryelements     = classes
        - org.gradle.usage               = java-runtime
    Artifacts
        - build/classes/java/main (artifactType = java-classes-directory)

    --------------------------------------------------
    Secondary Variant resources
    --------------------------------------------------
    Directories containing assembled resource files for main.

    Attributes
        - org.gradle.category            = library
        - org.gradle.dependency.bundling = external
        - org.gradle.jvm.version         = 8
        - org.gradle.libraryelements     = resources
        - org.gradle.usage               = java-runtime
    Artifacts
        - build/resources/main (artifactType = java-resources-directory)

--------------------------------------------------
Variant testResultsElementsForTest (i)
--------------------------------------------------
Binary results obtained from running all targets in the 'test' Test Suite.

Capabilities
    - zosbackupserver:model9-zos-below-the-bar-storage:4.2.0-SNAPSHOT (default capability)
Attributes
    - org.gradle.category         = verification
    - org.gradle.testsuite.name   = test
    - org.gradle.verificationtype = test-results
Artifacts
    - build/test-results/test/binary (artifactType = directory)

And I’m afraid that’s most of the error message. Here it is with part of the stacktrace:

Could not determine the dependencies of task ':model9-acd:model9-agent-package:resolveSources'.
> Could not resolve all dependencies for configuration ':model9-acd:model9-agent-package:packageRemote'.
   > Unable to find a variant with the requested capabilities: []:

* Try:
> Review the variant matching algorithm at https://docs.gradle.org/8.14/userguide/variant_attributes.html#sec:abm_algorithm.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.

* Exception is:
org.gradle.api.internal.tasks.TaskDependencyResolveException: Could not determine the dependencies of task ':model9-acd:model9-agent-package:resolveSources'.
	at org.gradle.api.internal.tasks.CachingTaskDependencyResolveContext.getDependencies(CachingTaskDependencyResolveContext.java:70)
	at org.gradle.execution.plan.TaskDependencyResolver.resolveDependenciesFor(TaskDependencyResolver.java:49)
	at org.gradle.execution.plan.LocalTaskNode.getDependencies(LocalTaskNode.java:151)
	at org.gradle.execution.plan.LocalTaskNode.resolveDependencies(LocalTaskNode.java:125)
	at org.gradle.execution.plan.DefaultExecutionPlan.discoverNodeRelationships(DefaultExecutionPlan.java:182)
	at org.gradle.execution.plan.DefaultExecutionPlan.doAddEntryNodes(DefaultExecutionPlan.java:154)
	at org.gradle.execution.plan.DefaultExecutionPlan.addEntryTasks(DefaultExecutionPlan.java:129)
	at org.gradle.execution.plan.DefaultExecutionPlan.addEntryTasks(DefaultExecutionPlan.java:121)
	at org.gradle.execution.TaskNameResolvingBuildTaskScheduler.scheduleRequestedTasks(TaskNameResolvingBuildTaskScheduler.java:61)
	at org.gradle.execution.DefaultTasksBuildTaskScheduler.scheduleRequestedTasks(DefaultTasksBuildTaskScheduler.java:72)
	at org.gradle.initialization.DefaultTaskExecutionPreparer.lambda$scheduleRequestedTasks$0(DefaultTaskExecutionPreparer.java:48)