Collecting platform information dynamically

I have a Gradle build that includes a platform/BOM. There is also a project that has a bunch of sub projects under it. It is these projects that I want to include in the platform. To complicate things, the sub projects contains artifacts with classifiers. These artifacts should also be included in the platform. Not all sub projects have the same artifacts with classifiers and some have none.

The problem is that I want to avoid specifying these projects and their artifacts manually. The first step is easy. I can simply loop through the subprojects via project(":parent").subprojects.forEach{} and add them to the dependency constraints. The real problem comes when I need to add the artifacts with classifiers. Since the Java platform plugin do not support artifacts with classifiers, I need to add them manually with pom {withXML{ }}. This is problematic since in the platform project, I do not know which artifacts with classifiers that each sub project has.

To solve this I have created a precompiled convention plugin that all sub projects have. In this plugin, a task is created that produces a text file with a list of artifacts and their classifiers. This artifact is then added to a custom consumable configuration. The platform has the same configuration but it is resolvable instead. The platform can then add all sub projects as dependencies in this configuration. The next problem is that I cannot simply resolve this configuration straight away. I need to resolve it at execution time to avoid getting missing files exceptions.

To solve this, I moved the XML post processing to the POM generation task in a doLast. This resolves the configuration at execution time while still adding the necessary XML to the POM. My question is about whether this is a good solution? Can this be done with less steps and/or does it contain any bad practises? I would think that this problem is not super uncommon and that there is a best practice way of doing it. Maybe the best practice is not to automate it, but then I would have to manually specify the artifacts both in the sub project and the platform.

Here is the code:

Binary Plugin applied to sub projects (kotlin):

// Register task during plugin apply
project.tasks.register("generateMetadata")

// The task that generates the metadata declared later when the project has supplied the required info
project.tasks.named("generateMetadata") {
	val outputFile = project.layout.buildDirectory.file("generated/metadata.txt")
	val text = "The list of artifacts and their classifiers"

	outputs.file(outputFile)

	doLast {
		val file = outputFile.get().asFile
		file.parentFile.mkdirs()
		file.writeText(text)
	}
}

Precompiled plugin applied to sub projects (kotlin):

// Get metadata task
val metadataTask = tasks.named("generateMetadata")

// Custom metadata configuration
configurations.register("metadata"){
    isCanBeResolved = false
    isCanBeConsumed = true
    attributes {
        attribute(Usage.USAGE_ATTRIBUTE, objects.named(Usage::class.java, "metadata"))
    }
    outgoing.artifact(metadataTask){
        classifier = "metadata"
        extension = "txt"
        builtBy(metadataTask)
    }
}

Precompiled plugin applied to platform (kotlin):

// Get a list of the sub projects modules
val modules = mutableListOf<String>()
project(":modules").subprojects.forEach { subProject ->
    modules.add(subProject.name)
}

// Custom metadata configuration
val metadataConfiguration = configurations.create("metadata"){
    isCanBeResolved = true
    isCanBeConsumed = false
    attributes {
        attribute(Usage.USAGE_ATTRIBUTE, objects.named(Usage::class.java, "metadata"))
    }
}

// Add sub projects as dependencies with metadata configuration
dependencies {
    modules.forEach { module ->
        add(metadataConfiguration.name, project(":modules:${module}"))
    }
}

// Data container for each artifact
class ClassifierArtifact(
    val module: String,
    val classifier: String,
)

tasks.withType<GenerateMavenPom>().configureEach {
    doLast {
        val classifierArtifacts = mutableListOf<ClassifierArtifact>()

        metadataConfiguration.resolve().forEach { file ->
            file.readLines().forEach { line ->
                // lines are of the form group:name:version:classifier
                val parts = line.split(":")
                val id = parts[1]
                val classifier = parts[3]
                classifierArtifacts.add(ClassifierArtifact(id, classifier))
            }
        }

        pom.withXml {
            // Do XML post processing with classifierArtifacts list
        }
    }
}

dependencies {
    constraints {
        modules.forEach { module ->
            api(project(":modules:${module}"))
        }
    }
}

The code has been cut out of context but I hope it shows enough to give some thoughts.

The main question I have is, why do you need the classifiers in the platform?
I don’t think they are of any use in there?
The platform plugin not providing a way to add them is also a hint in that direction.
If that is a wrong assumption, maybe a good idea would be a feature request or pull request to add this capability.

Maven seem to need the version to be specified for both the main artifact and any classified artifacts while Gradle do not. The XML post processing then becomes necessary due to the platform plugin not supporting classified artifacts. I have found a few relevant issues regarding this:

It would seem that this is a known problem and that Gradle devs have no interest in adding support for classified artifacts. At least not when the first issue was posted. The suggested solution is to post process the XML as I have done. I could post a new feature request if you believe the situation has changed since then, but I do not believe it has.

I have also confirmed that Maven need the version for classified artifacts in the BOM for my self. In the Maven setup below, the version has to be specified for the native if it is not part of the BOM. I have commented out the version in both the consumer and BOM POM files. One of them has to be uncommented in order to successfully build the app. Having to specify the version more than once defeats the purpose of having a BOM in the first place. The version for the classified jars should therefore be specified in the BOM. I could be wrong of course, since I have not used Maven before.

The error message:

'dependencies.dependency.version' for org.lwjgl:lwjgl:jar:natives-windows is missing

Maven project POM (consumer):

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	
	<modelVersion>4.0.0</modelVersion>
	
	<groupId>com.dwarfley</groupId>
	<artifactId>lwjgl</artifactId>
	<version>1.0</version>
	
	<properties>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
		<maven.compiler.release>17</maven.compiler.release>
	</properties>
	
	<dependencyManagement>
		<dependencies>
			<dependency>
				<groupId>org.lwjgl</groupId>
				<artifactId>lwjgl-bom</artifactId>
				<version>3.4.0</version>
				<type>pom</type>
				<scope>import</scope>
			</dependency>
		</dependencies>
	</dependencyManagement>
	
	<dependencies>
		<dependency>
			<groupId>org.lwjgl</groupId>
			<artifactId>lwjgl</artifactId>
		</dependency>
		<dependency>
			<groupId>org.lwjgl</groupId>
			<artifactId>lwjgl</artifactId>
			<!--<version>3.4.0</version>-->
			<classifier>natives-windows</classifier>
		</dependency>
	</dependencies>
	
	<build>
		<pluginManagement>
			<plugins>
				<plugin>
					<artifactId>maven-compiler-plugin</artifactId>
					<version>3.13.0</version>
					<configuration>
						<source>17</source>
						<target>17</target>
					</configuration>
				</plugin>
				<plugin>
					<groupId>org.codehaus.mojo</groupId>
					<artifactId>exec-maven-plugin</artifactId>
					<version>3.1.0</version>
					<configuration>
						<mainClass>com.dwarfley.App</mainClass>
					</configuration>
				</plugin>
			</plugins>
		</pluginManagement>
	</build>
	
</project>

Maven BOM POM (lwjgl-bom):

<?xml version="1.0" encoding="UTF-8"?>
<project xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd" xmlns="http://maven.apache.org/POM/4.0.0"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
	
	<modelVersion>4.0.0</modelVersion>
	
	<groupId>org.lwjgl</groupId>
	<artifactId>lwjgl-bom</artifactId>
	<version>3.4.0</version>
	<packaging>pom</packaging>
	
	<dependencyManagement>
		<dependencies>
			
			<dependency>
				<groupId>org.lwjgl</groupId>
				<artifactId>lwjgl</artifactId>
				<version>3.4.0</version>
			</dependency>
			
			<!--
			<dependency>
				<groupId>org.lwjgl</groupId>
				<artifactId>lwjgl</artifactId>
				<version>3.4.0</version>
				<classifier>natives-windows</classifier>
			</dependency>
			-->
			
		</dependencies>
	</dependencyManagement>

</project>

As a side note, I have also discovered that my current solution is not compatible with the configuration cache. It would appear that the platform side of the solution is causing the following error:

- Task `:lwjgl-bom:generatePomFileForLwjglBomPublication` of type `org.gradle.api.publish.maven.tasks.GenerateMavenPom`:
cannot serialize Gradle script object references as these are not supported with the configuration cache.
See https://docs.gradle.org/9.0.0/userguide/configuration_cache_requirements.html#config_cache:requirements:disallowed_types

After looking at the linked page, there seem to be some rules that I must follow when using the configuration cache. The error message is not super helpful in narrowing down exactly what the issue is. I suspect I need to change the way I use the project and the metadata configuration objects. I did however remove those to see if it worked and it would seem that the XML part is also a part of the problem as I get the following error:

Execution failed for task ':lwjgl-bom:generatePomFileForLwjglBomPublication'.
> The value of this property has been discarded during serialization.

The code that was used to produce this was:

tasks.withType<GenerateMavenPom>().configureEach {
    doLast {
        pom.withXml {
			
        }
    }
}

I can provide more context to the things I have mentioned so far if you want.

Oh, that’s sad. ;-(
I wonder that they don’t consider this if Maven consumers do need these entries. :frowning:

The error message is not super helpful in narrowing down exactly what the issue is.

The message means, that you somewhere reference something that is declared within the build script, so to access it at execution time you would need an instance of the build script, which is not possible when using configuration cache. If I would see the actual code where this happens I might hint at what the problem is and how to fix it. For example problematic is:

val foo = "foo"
val bar by tasks.registering {
    doLast {
        println(foo)
    }
}

and this would fix it as the execution phase action no longer accesses the property of the build script object, but the local variable copy:

val foo = "foo"
val bar by tasks.registering {
    val foo = foo
    doLast {
        println(foo)
    }
}

I did however remove those to see if it worked and it would seem that the XML part is also a part of the problem as I get the following error

It is generally a bad idea to change task configuration at execution time if possible at all.
It seems specifically this pom property is not available anymore at execution time.
If you look at the implementation of GenerateMavenPom you see that getPom uses a Transient.Var which means after serialization and deserialization from configuration cache this property is not available anymore, but was transformed into the private internal mavenPomSpec.

So if you really need to follow that strategy, you probably need to mark the task with notCompatibleWithConfigurationCache("...") which will then disable configuration cache for executions that contain this task. But usually marking a task as not-CC-compatible should only be a temporary measure until it is made properly CC compatible. If this is not somehow possible, it might be a candidate for a feature request. But actually the pom property is marked with ToBeReplacedByLazyProperty, so hopefully the API will evolve in a later major version to better support your use-case, as unfortunately they did not finish the lazification of built-in stuff for Gradle 9 as was planned.

Maybe you can somehow follow a different approach like defining in a central place like a shared build service which modules have which classified artifacts, and then use that in the individual projects to configure those and use it at configuration time to configure the platform instead of doing it at execution phase?

I agree, it complicates things when I want to support Maven. :frowning:

Well, I do not need to follow this strategy exactly. The only requirement I have is that the platform project need to know which classified artifacts each member of the platform produces. This information will then be used to post process the XML. I can not resolve the metadata configuration in the configuration phase as it will create problems of its own with missing artifacts.

I actually discovered the lazy property efforts while working on this project. I also saw that they were unable to complete it in time for 9.0.0. Hopefully this will be completed soon as it would give us more options in situations like this.

This was my initial thought, but I could not figure out how to do it without using functions like “evaluationDependsOn”. If I create a plugin that the modules can register their artifacts, how do I make sure that the platform only performs the post processing when all artifacts have been registered? Or if the platform register a callback that each module calls, how can I make sure the callback is registered before any artifacts?

I also looked into shared build services. They seem to be task oriented and are primarily used at execution time. The mention in the docs that the usage of shared build services should be avoided during the configuration phase.

It would seem that there is no good options and I have to choose the option which is the least bad. If you could expand on what kind of solution you had in mind when mentioned a central place to store the list of artifacts, it would be appreciated. Perhaps I missed something.

This was my initial thought, but I could not figure out how to do it without using functions like “evaluationDependsOn”. If I create a plugin that the modules can register their artifacts, how do I make sure that the platform only performs the post processing when all artifacts have been registered?

You maybe misread what I said.
Not the projects register themselves, but the information which project has which classified artifacts is defined centrally, for example in a shared build service, that can then used by the projects to see which artifacts they should create and the platform to configure the XML accordingly.

I also looked into shared build services. They seem to be task oriented and are primarily used at execution time.

That’s only one of their use-cases.
Sharing information or services between projects - also at configuration time - or from settings script to project scripts are other user-cases where they can help.

If you could expand on what kind of solution you had in mind when mentioned a central place to store the list of artifacts, it would be appreciated.

I did tell already twice, once in my last comment and once in this comment. :slight_smile:

Another possibility would be to not try to reconfigure the pom property in a doFirst action, but instead add a doLast action that reads the produced file, enhances it with the additional information and writes it out again. A doLast action is part of the task execution so there it is fine to manipulate output files in-place. Things like up-to-date checks and similar work with the end result.

I see, so you mean that I should extract all metadata from the project level and store this in one big collection that is accessible to both the modules and the platform? This does mean that to configure a project, I have to also find this central place and edit the collection. This seems slightly awkward and counter intuitive. Would it not be better to keep a project’s configuration in the project itself? I do however understand that I might need to make sacrifices if no better alternatives exists.

The documentation seem to disagree. Unless it only refers to the task use case. I could not find any documentation on the other use cases. It might just be that they are not documented. Regardless, even if I create a shared build service, I would still need to coordinate the usage of it so that no artifacts are missed. Unless I use a single collection like I mentioned above.

I did think of another possible solution. What if I make a plugin that all modules and the platform applies. The plugin would make it possible to register artifacts which would be stored in a global list and not per project. There would also be a function for the platform that registers an action that is executed for each artifact. I would need to keep both the registered artifacts and actions to make sure that no artifacts are missed. When I add an action, it is executed on all previously registered artifacts. When I register an artifact, all previously registered actions are executed with it. This make it so that the order of registration is not important and no artifacts are missed.

The only question is if keeping state across projects within a plugin is possible and/or is a good idea. Does projects share the same instance of a plugin and do I need to make it thread safe? I guess I could use the shared build service and implement this logic within. However like I said before, the documentation makes it seem like using the shared build service during configuration is a bad idea.

Would it not be better to keep a project’s configuration in the project itself?

Definitely, if you get it managed somehow.
Just sharing ideas how you might maybe make things working.
Also, maybe Jendrik will have some ideas for it when he is back from holidays.

The documentation seem to disagree.

No, it is just the most typical use-case, and if the shared build service is used to represent expensive state like booting up some database or something like that, it might be a bad idea to do that at configuration time. But the linked paragraph also says “However, sometimes, using the service at configuration time can make sense” which are for example in the cases I mentioned.

Regardless, even if I create a shared build service, I would still need to coordinate the usage of it so that no artifacts are missed.

Again, the suggestion was for the shared build service to be the central place where you define the information the projects and the platform get their information from without anyone writing to it so no coordination would be necessary.

I did think of another possible solution.

Feel free to give it a try, but I think it will become problematic eventually.
From a gut feeling it sounds like effectively doing cross-project configuration even if done with an indirection and latest when isolated projects are coming this might get problematic. :man_shrugging:

The only question is if keeping state across projects within a plugin is possible and/or is a good idea.

Did I mention that shared build services are the way to go for that? :slight_smile:

Does projects share the same instance of a plugin and do I need to make it thread safe?

No.
Configuration is done single-threaded anyway iirc, and each project should have a separate instance of the plugin.

However like I said before, the documentation makes it seem like using the shared build service during configuration is a bad idea.

Do you trust an expert using Gradle since pre-1.0 and is working daily on build logic, or documentation which naturally is outdated or missing information the moment it is written like any documentation out there? :slight_smile:

And again, it says that they are “generally intended for that”, that “if they represent expensive state it is a bad idea to access them at configuration time”, and that “it can make sense to use them at configuration time”. The further use-cases are just not (or no longer) documented it seems. :slight_smile:

Maybe your best bet is indeed the doLast manipulation I suggested which should be compatible with your current approach that just fails due to pom not being available anymore.

I appreciate all ideas. If you have not guessed it already, this is related to the previous discussion about LWJGL. I will soon publish the PR and will then await Jendrik’s feedback.

Thanks for the clarification.

I read up on isolated projects. I agree that projects should in an ideal world not have to communicate. The only real solution that then still works with the configuration cache and isolated projects would appear to be to use the metadata setup in my original post. This still leaves the problem of resolving the metadata artifacts.

I meant no disrespect, I simply discovered a contradiction between your response and the documentation and wanted a clarification on what was correct. I of course trust the expert more than the documentation but I believed a clarification was needed. :slight_smile:

Yes, if this works then the pom.withXml issue would be solved. The remaining problem is how/when to extract the artifacts files from the metadata configuration. I will have to try to get some kind of property collection from configuration without actually resolving it. Then in the execution phase (doLast) I can call get on it/them. I will consult the api/docs to see if I can find something like that.

Anyhow, thanks you for your ideas and feedback so far.

The remaining problem is how/when to extract the artifacts files from the metadata configuration.

I don’t think I get what you mean.
You have a resolvable configuration and that you retrieve at execution time.
So what is the actual problem with that?

Well, I am having problems with the configuration cache. It does not allow me to access the metadata configuration during execution. I somehow need to please it by getting some intermediate object? The docs suggest a FileCollection. I am not sure how to go about getting it. This is what I have:

val metadataConfiguration: Configuration = configurations.create("metadata") {
    isCanBeResolved = true
    isCanBeConsumed = false
    attributes {
        attribute(Usage.USAGE_ATTRIBUTE, objects.named(Usage::class.java, "metadata"))
    }
}

tasks.withType<GenerateMavenPom>().configureEach {
    inputs.files(metadataConfiguration)
    doLast {
        val test = metadataConfiguration
    }
}

The error message is again:

1 problem was found storing the configuration cache.
- Task `:lwjgl-bom:generatePomFileForLwjglBomPublication` of type `org.gradle.api.publish.maven.tasks.GenerateMavenPom`:
  cannot serialize Gradle script object references as these are not supported with the configuration cache.
  See https://docs.gradle.org/9.0.0/userguide/configuration_cache_requirements.html#config_cache:requirements:disallowed_types

As a side question, what does the error message refer to? Is gradle script references a general term for references to gradle object (project, configuration etc) or is it refering to references to a specific type of gradle object?

Also does the restriction of which types are used also apply to classes and functions declared alongside the task in the same .gradle.kts file? I cannot seem to get it to work with functions. I guess classes is ok because they are not part of the script class that wraps the kotlin script.

The error message is again:

And it is exactly what I explained above.
metadataConfiguration is a property of the script object that you try to access in the execution time code so it would need to be serialized which is not possible.
Move the val test = metadataConfiguration outside the doLast but inside the configureEach like I showed above.

Additionally you will need to explicitly type test as FileCollection.
Configuration extends FileCollection so you can just assign it and it will work properly if the type of the local variable is FileCollection. Just if the type is implied to Configuration CC will complain.

Is gradle script references a general term for references to gradle object (project , configuration etc) or is it refering to references to a specific type of gradle object?

It is referring to the instance of the script object.
Imaging that the script content is the body of a class.
Things you think are local variables are properties of that class, code is in a function of that class.
That’s overly simplified what is happening.

I cannot seem to get it to work with functions.

Same here, they are functions of that script class instance and thus you cannot use them.

I guess classes is ok because they are not part of the script class that wraps the kotlin script.

Not “are” but “can be”.
If they are “static” classes, that is they do not refer to anything of the script object like metadataConfguration it will work.
If that class would access something like metadataConfguration it again is not static but bound to the script object instance and it will complain again.

But anyway, if you have functions or classes in your build scripts, it is a strong sign that you should refactor your stuff and move things to proper classes in an included build or buildSrc if you prefer. Build scripts should optimally be as declarative as possible and only contain minimal or optimally no procedural code.

I now have:

tasks.withType<GenerateMavenPom>().configureEach {
    val metadataFiles: FileCollection = metadataConfiguration
    inputs.files(metadataFiles)
    doLast {
        val test = metadataFiles
    }
}

It now seem to work correctly.

Thank you for the help :slightly_smiling_face: . I will now mark this discussion as completed.

1 Like

Dunno whether you follow the #8561 issue you linked to earlier, it was at least reopened now for team-reconsideration. :slight_smile:

Thanks for letting me know! :slightly_smiling_face:

1 Like