Handling LWJGL natives with custom dependency configurations

Background

I am using a multi-project gradle build to develop a java application. The build contains both library and application sub-projects. The applications depend on the libraries and these can in turn depend on other libraries. Each sub-project is a java module that requires other modules in a module-info.java This must must be kept in sync with the dependencies in the gradle build script.

Problem

Some of the libraries depend on LWJGL which contains natives. The natives are separate dependencies that need to be handled. The way that I initially solved this was to make the module depend on the .native version of the LWJGL modules. This also requires the build script to put the natives on the implementation configuration. This is less ideal as the java module should not care about natives.

The build script generated by the LWJGL customize tool on their website recommends putting the natives on the runtimeOnly configuration. The java module should then require the non-native modules. This does however result in the problem I am trying to solve. Due to the native dependencies (which are included in the runtimeOnly configuration) ending up on the module path, they are only loaded if a module requires them. This means that I have to include them in the classpath instead. The natives would then need to be extracted into a folder which can be added to the classpath.

Only the applications need the natives to run and should therefore be the one to collect the natives. It is important that all natives from all libraries that the application depends on are collected. This includes libraries that depend on other libraries.

Partial Solution

After a bit of searching and looking at other projects, I came up with the solution of adding the native dependencies to a custom dependency configuration. This has the benefit of clear seperation between natives and non-natives. It would also allow the application sub-projects to collect all the natives through gradles existing dependency system.

Looking at the documentation for dependency configurations and the java library plugin, this would mean adding the configurations natives, nativesClasspath and nativeElements. This would mimic the standard configurations created by the java plugins. The following is the build script for creating the configurations.

configurations.register("natives") {
	canBeConsumed = false
	canBeResolved = false
	transitive = true
}

configurations.register("nativesPath") {
	canBeConsumed = false
	canBeResolved = true
	transitive = true
	extendsFrom(configurations.natives)
	attributes {
		attribute(Usage.USAGE_ATTRIBUTE, objects.named(Usage, 'java-natives'))
	}
}

configurations.register("nativesElements") {
	canBeConsumed = true
	canBeResolved = false
	transitive = false
	extendsFrom(configurations.natives)
	attributes {
		attribute(Usage.USAGE_ATTRIBUTE, objects.named(Usage, 'java-natives'))
	}
}

From my understanding, I should then be able to add native dependencies like this.

natives 'org.lwjgl:lwjgl-glfw::natives-windows'
natives project(':lib')

The nativesClasspath configuration on the application sub-projects should then contain all the natives. These could then be extracted into a folder using a copy task such as this.

tasks.register('extractNatives', Copy) {
	
	group = "natives"
	description = "Extracts native libraries from all subprojects"
	
	inputs.files configurations.nativesClasspath
	outputs.dir "$buildDir/natives"
	
	duplicatesStrategy = DuplicatesStrategy.EXCLUDE
	
	from {
		configurations.nativesClasspath.files
	}
	
	into "$buildDir/natives"
	
}

Finally, the folder can be added to the classpath of the run task and the eclipse project files. It can also be included in releases using the distribution plugin.

The problem with this is that the LWJGL platform/BOM that supplies the versions for all libraries and natives seem to require that the configuration uses the usage attribute java-runtime instead of a custom one. I do not know what effect it would have to use java-runtime for natives as well. Also, the three configurations seem to need some artifact setup as the natives are not collected properly. Only the projects end up in the dependency graph and not their natives.

Alternatives

I do not know if using the three configurations is overkill and fewer would be better. The java configurations seem to suggest it would be best to separate the configuration into these three. I could also just list the natives in the application sub-projects but that would defeat the purpose of having a separate library. Also, if using custom configurations is not the right way to do things, then please point me in a better direction. I am not necessarily looking for a specific solution, just a good one.

From a cursory look I’d say lwjgl should optimally publish proper feature variants for the native variants. If it does not, you can request it as feature and in the meantime use a component metadata rule to dynamically add these feature variants.

Then you can use an artifact view with variant reselection on the runtime classpath configurationto get the native variants of all dependencies that have one.

I have now done some research into feature variants, variants & attributes, variant selection and artifact views.

If I understand you correctly, each module (maven publication) should contain a gradle module metadata file alongside the pom file. This tells gradle what variants the module contain and what attributes they have. There would then be a variant with only the base jar artifact and one with both the base and native jar artifacts. There could also be other variants that only have the native artifact or one with natives artifacts for all platforms.

Lwjgl does not currently publish this gradle module metadata which means that I have to add it manually. This can be done using component metadata rules which adds metadata to lwjgl modules. The docs for component metadata rules interestingly use lwjgl as a use case.

Also to clarify how lwjgl currently handles natives, they have a single module for both the base library jar artifact and the corresponding native jar artifacts. To choose a native jar, classifiers are used.

So far only regular variants have been discussed. You mentioned feature variants, but those seem to add optional code from source sets. In this scenario the variants only need to have different artifacts that are already built. Regular variants seem to fit this need better.

When the variants have been created, I can then add attributes to the runtime configuration to specify which platform to use. I can also create an extract natives task that creates an artifact view that selects the variant which only includes the natives.

This solves the problem of finding and selecting natives automatically. Then there is the original problem of actually making sure that the natives get used during the execution of the application. My suggestion was to extract them to a folder and then add this folder to the classpath. This would make sure that they are loaded since the module path is not used. If I understand you correctly, this is the way to go. An alternative would be if I could have gradle use both the module and classpath at the same time but I do not know if this is possible.

This is the complete build.gradle script that I have come up with so far.


plugins {
	id 'application'
}

repositories {
	mavenCentral()
}

@CacheableRule
abstract class LwjglRule implements ComponentMetadataRule {
	
	private def nativeVariants = [
		[os: OperatingSystemFamily.LINUX,   arch: "arm32",  classifier: "natives-linux-arm32"],
		[os: OperatingSystemFamily.LINUX,   arch: "arm64",  classifier: "natives-linux-arm64"],
		[os: OperatingSystemFamily.WINDOWS, arch: "x86",    classifier: "natives-windows-x86"],
		[os: OperatingSystemFamily.WINDOWS, arch: "x86-64", classifier: "natives-windows"],
		[os: OperatingSystemFamily.MACOS,   arch: "x86-64", classifier: "natives-macos"]
	]
	
	@Inject abstract ObjectFactory getObjects()
	
	void execute(ComponentMetadataContext context) {
		
		def natives = nativeVariants
		
		context.details.withVariant("runtime") {
			attributes {
				attributes.attribute(OperatingSystemFamily.OPERATING_SYSTEM_ATTRIBUTE, objects.named(OperatingSystemFamily, "none"))
				attributes.attribute(MachineArchitecture.ARCHITECTURE_ATTRIBUTE, objects.named(MachineArchitecture, "none"))
			}
		}
		
		natives.each { variantDefinition ->
			
			context.details.addVariant("${variantDefinition.classifier}-runtime", "runtime") {
				
				attributes {
					attributes.attribute(OperatingSystemFamily.OPERATING_SYSTEM_ATTRIBUTE, objects.named(OperatingSystemFamily, variantDefinition.os))
					attributes.attribute(MachineArchitecture.ARCHITECTURE_ATTRIBUTE, objects.named(MachineArchitecture, variantDefinition.arch))
				}
				
				withFiles {
					addFile("${context.details.id.name}-${context.details.id.version}-${variantDefinition.classifier}.jar")
				}
				
			}
			
			context.details.addVariant("${variantDefinition.classifier}") {
				
				attributes {
					attributes.attribute(Usage.USAGE_ATTRIBUTE, objects.named(Usage, "none"))
					attributes.attribute(OperatingSystemFamily.OPERATING_SYSTEM_ATTRIBUTE, objects.named(OperatingSystemFamily, variantDefinition.os))
					attributes.attribute(MachineArchitecture.ARCHITECTURE_ATTRIBUTE, objects.named(MachineArchitecture, variantDefinition.arch))
				}
				
				withFiles {
					addFile("${context.details.id.name}-${context.details.id.version}-${variantDefinition.classifier}.jar")
				}
				
			}
			
		}
		
		context.details.addVariant("natives-all-runtime", "runtime") {
			
			attributes {
				attributes.attribute(OperatingSystemFamily.OPERATING_SYSTEM_ATTRIBUTE, objects.named(OperatingSystemFamily, "all"))
				attributes.attribute(MachineArchitecture.ARCHITECTURE_ATTRIBUTE, objects.named(MachineArchitecture, "all"))
			}
			
			withFiles {
				natives.each { variantDefinition ->
					addFile("${context.details.id.name}-${context.details.id.version}-${variantDefinition.classifier}.jar")
				}
			}
			
		}
		
		context.details.addVariant("natives-all") {
			
			attributes {
				attributes.attribute(Usage.USAGE_ATTRIBUTE, objects.named(Usage, "none"))
				attributes.attribute(OperatingSystemFamily.OPERATING_SYSTEM_ATTRIBUTE, objects.named(OperatingSystemFamily, "all"))
				attributes.attribute(MachineArchitecture.ARCHITECTURE_ATTRIBUTE, objects.named(MachineArchitecture, "all"))
			}
			
			withFiles {
				natives.each { variantDefinition ->
					addFile("${context.details.id.name}-${context.details.id.version}-${variantDefinition.classifier}.jar")
				}
			}
			
		}
		
	}
	
}

def nativeConfigurations = [
    configurations.runtimeClasspath,
    configurations.testRuntimeClasspath
]

nativeConfigurations.each{ config ->
	config.attributes {
		attribute(OperatingSystemFamily.OPERATING_SYSTEM_ATTRIBUTE, objects.named(OperatingSystemFamily, OperatingSystemFamily.WINDOWS))
		attribute(MachineArchitecture.ARCHITECTURE_ATTRIBUTE, objects.named(MachineArchitecture, "x86-64"))
	}
}

project.ext.lwjglVersion = "3.3.6"

dependencies {
	
	components {
        withModule("org.lwjgl:lwjgl", LwjglRule)
		withModule("org.lwjgl:lwjgl-glfw", LwjglRule)
    }
	
	implementation platform("org.lwjgl:lwjgl-bom:$lwjglVersion")
	
	implementation "org.lwjgl:lwjgl"
	implementation "org.lwjgl:lwjgl-glfw"
	
	testImplementation libs.junit
	testRuntimeOnly 'org.junit.platform:junit-platform-launcher'
	
}

application {
	mainModule = 'com.dwarfley.app'
	mainClass = 'com.dwarfley.app.App'
}

java {
	toolchain {
		languageVersion = JavaLanguageVersion.of(21)
	}
}

tasks.named('test') {
	useJUnitPlatform()
}

tasks.register('extractNatives', Copy) {
	
	group = "natives"
	description = "Extracts natives from all runtime dependencies"
	
	def artifactView = configurations.runtimeClasspath.incoming.artifactView {
		
		withVariantReselection()
		
		attributes {
			attribute(Usage.USAGE_ATTRIBUTE, objects.named(Usage, "none"))
			attribute(OperatingSystemFamily.OPERATING_SYSTEM_ATTRIBUTE, objects.named(OperatingSystemFamily, "all"))
			attribute(MachineArchitecture.ARCHITECTURE_ATTRIBUTE, objects.named(MachineArchitecture, "all"))
		}
		
	}
	
	inputs.files artifactView.files
	outputs.dir "$buildDir/natives"
	
	duplicatesStrategy = DuplicatesStrategy.EXCLUDE
	
	from artifactView.files
	into "$buildDir/natives"
	
}

tasks.named('run') {
	
	jvmArgs += ["--class-path", "$buildDir\\natives\\*"]
	
    dependsOn tasks.named('extractNatives')
	
}

Is this what you had in mind? I think I got it all correct including configuration avoidance.

This adds the following variants with their attributes:

  • runtime
    • os: none, arch: none, usage: java-runtime
  • natives-(platform)-runtime
    • os: (platform), arch: (platform), usage: java-runtime
  • natives-(platform)
    • os: (platform), arch: (platform), usage: none
  • natives-all-runtime
    • os: all, arch: all, usage: java-runtime
  • natives-all
    • os: all, arch: all, usage: none

If not using java modules, the attributes for the configurations should be the target platform. If using modules, the attributes should be “none” as the natives should be extracted instead of being included in the runtime configuration. The extract task then uses the natives-all variant which avoids the base jar artifact from being extracted. It also extracts all natives so the app becomes platofrm independant. Finally, the run task is configured to depend on the extract task and its classpath is modified to inculde the natives. It would have been better if the same build script could be used for both modular and non modular java projects. However since the natives need to be extracted in modular projects, this would be impossible.

This works quite nicely. The next thing to do is adding the native folder to the eclipse project files and distributions. After that, the only thing remaining is to make lwjgl publish the metadata. I discovered a github issue related to this. There does not seem to be any active development that I can see, so I could provide a PR to speed things up. I only ask for a final ok from you that I have understood everything correctly and that the plan/system is a good one before spending time on a PR.

The way that lwjgl handles building and publising is also a bit awkward since they use ant to build the artifacts and gradle to publish them (maven-publish plugin). This means that there is no real software component and gradle can not generate the metadata. One solution could be to manually create custom components in an ad-hoc way. I will have to investigate this. Here is a link to their build script for reference.

If I understand you correctly, each module (maven publication) should contain a gradle module metadata file alongside the pom file. This tells gradle what variants the module contain and what attributes they have.

Yes, the GMM is like the POM just way richer and not so limited, it can bear more information like various variants or even just additionally attributes for the main variant like the Java version compatibility information that Gradle consumers can use.

There would then be a variant with only the base jar artifact and one with both the base and native jar artifacts.

Whether the additional variant contains only the native artifact or both depends. If you typically need the natives separate maybe only the native artifact would be in. Or if always need both together in the same classpath maybe you right away have both artifacts in the variant. Or you could also have both variants if it makes sense. If two variants have different capabilities, you could also depend on both in the same configuration.

There could also be other variants that only have the native artifact or one with natives artifacts for all platforms.

Theoretically yes, if it makes sense and you figure out proper attribute constructions that make sense and work properly. Figuring out the proper attributes and attribute values is one of the hardest parts in defining variants.

Also to clarify how lwjgl currently handles natives, they have a single module for both the base library jar artifact and the corresponding native jar artifacts. To choose a native jar, classifiers are used.

Yes, that’s the typical Maven way of representing variants as the POM cannot express them. That’s one of the reasons for GMM. Almost all “classified” artifacts would with GMM be modeled as variants, sometimes only with different attributes, sometimes also with different capability. With GMM different variants can also have different dependencies, …

The “sources” jar and the “javadoc” jar are also variants in GMM if done properly and can be resolved using attribute-aware resolution.

So far only regular variants have been discussed. You mentioned feature variants, but those seem to add optional code from source sets. In this scenario the variants only need to have different artifacts that are already built. Regular variants seem to fit this need better.

I didn’t there is a difference between “feature variant” and “variant”. I think I usually use these terms interchangeably. There are just variants that have different attributes and same or different capabilities.

When the variants have been created, I can then add attributes to the runtime configuration to specify which platform to use.

That’s the idea, yeah.
Either on a dependency directly, or on a resolvable configuration, or on an artifact view, depending on concrete situation and needs.

My suggestion was to extract them to a folder and then add this folder to the classpath. This would make sure that they are loaded since the module path is not used. If I understand you correctly, this is the way to go.

I cannot really tell you, I don’t know how lwjgl works and what it needs where and how it behaves with things on the module or classpath, that is probably not so much a Gradle topic? :man_shrugging:

Usually I’d now probably ask for an MCVE, but I’m on vacation without computer at hand, so I could not have a look anyway.

An alternative would be if I could have gradle use both the module and classpath at the same time but I do not know if this is possible.

Should be, yes

Is this what you had in mind? I think I got it all correct including configuration avoidance.

As I said, no box right now, so cannot really properly look at, so just a few points from a very cursory look:

  • why don’t you use the standard values for the architecture attribute?
  • ext / extra properties are practically always a work-around for not doing something properly and you should always feel directly when using them. Here due example either store version in a version catalog TOML file, or at least use just a local variable instead of an ext property.
  • Copy or copy { ... } is very seldomly what you want, usually you instead intended to use Sync or sync { ... }.
  • Usage “none” does not sound like it should be used for anything anytime, my gut feeling says that this is never an appropriate value for that attribute.
  • Don’t use buildDir, that’s deprecated, use layout.buildDirectory instead.
  • almost always when you need to specify a duplicates strategy, it is more a smell pointing at a quite different problem. Usually you want to find out why files are duplicate actually and fix that instead.
  • I don’t think the explicit inputs and outputs configuration for the extract task is necessary, the from and into should already take care of that
  • maybe reconsider the task name, as it does not extract anything but just copies some files together
  • reconsider whether you need that task at all or if it is just wasted time, when you could also just directly use the files where they are for the classpath
  • practically any explicit dependsOn that does not have a lifecycle task on the left-hand side is a code smell and usually a sign that you do not properly write task outputs to task inputs. If you really need the extract task and really need the manual jvm arg, instead use a jvmargprovider that properly declares the inputs so that the run task inherites that information, but probably better configure the classpath property and use the files where they are without copying
  • more a personal recommendation, switch to Kotlin DSL. By now it is the default DSL, you immediately get type-safe build scripts, actually helpful error messages if you mess up the syntax, and amazingly better IDE support if you use a good IDE like IntelliJ IDEA or Android Studio.

I only ask for a final ok from you that I have understood everything correctly and that the plan/system is a good one before spending time on a PR.

No idea, I have no idea about lwjgl and what makes or does not make sense, besides general things I said already above.

Thank you for your response. I too have been away, but have now returned home.

It would seem like we need some concrete input from LWJGL to continue this discussion in a more meaningfull way. I have created a question issue over at their github repo. Hopefully they can answer as to what the intended way of consuming their natives is in a modular project.

In the meantime, I will think about your list of suggestions. Also to get my project running, I will simply require the natives in the module-info.java and add them using the implementation configuration. This will allow me to continue developing the application while we wait for a response.

It could also be useful to investigate how both the class-path and module-path can be used simultaneously. This could be the best solution if it requires little setup.

The issue I created also contains a minimal example if you want to test it for yourself. The last part of the example shows that the run task fails due to LWJGL not finding the natives. This example uses a condensed version of the files generated by the init taks and the generated build script provided by the customiser on LWJGL’s web page.

The discussion over at LWJGL is now finished.

The discussion started with an explanation of the design when it comes to modules and dependencies. To provide a quick summary, LWJGL publishes a core module and several binding modules. The binding modules depends on the core module. Each module including the core module can have any number of platform specific natives packaged in jars, one for each platform. These jars are published together with the base jar for each module and are differentiated with classifiers. Even though the native jars do not contain any Java code, they do contain a module definition where the name is the base module’s name followed by .natives. Some bindings require natives only for some platforms and some bindings are not supported on certain platforms.

The current intended way of consuming LWJGL bindings is to add both the base and the correct native module as compile (implementation) dependencies. In the module-info, only the .natives module is required since the .natives module internally requires the base module. This is the way I have been doing it all along. I thought that I was doing something wrong, but it turns out I was not. The customiser on their website that generates build script automatically was also changed to make both the base and native dependencies use the implementation configuration which helps in avoiding the confusion.

The discussion then turned to Gradle. They currently use ant to build all the artifacts and only use Gradle to publish the artifacts to Maven. This is awkward when trying to add Gradle module metadata. There have been attempts at converting the ant build to use Gradle but they have not been successful. In the discussion, they mentioned that this migration is not a priority since it would not provide any major benefits over the current setup that already works. I fully understand that the benefits might not be worth the substantial amount of work required for such a migration.

I instead suggested that ad hoc software components could be an option. This would result in Gradle module metadata being published without having to convert the entire repo to use Gradle. The discussion ended with them welcoming contributions for this and also updating the wiki to help explain things better.

I have now created a fork of their repo and finished work on adding the ad hoc software components with the variants. I welcome any feedback that you might have. One point of discussion is the variants I choose to include as well as the attributes that I choose for each variant. Are they sensible? I reused the native platform attributes (os and arch) for the platform specific variants. Instead of having the variant with only the native for that platform use the usage of none, I set it to native runtime. Here is the list of variants and their attributes:

  • {module}ApiElements
    • Usage: JAVA_API
    • Category: LIBRARY
    • Bundling: EXTERNAL
    • LibraryElements: JAR
    • TargetJvmVersion: 8
    • “org.lwjgl.module”: “{module}”
  • {module}RuntimeElements
    • Usage: JAVA_RUNTIME
    • Category: LIBRARY
    • Bundling: EXTERNAL
    • LibraryElements: JAR
    • TargetJvmVersion: 8
    • “org.lwjgl.module”: “{module}”
  • {module}{platform}ApiElements
    • Usage: JAVA_API
    • Category: LIBRARY
    • Bundling: EXTERNAL
    • LibraryElements: JAR
    • OperatingSystemFamily: “{platform.os}”
    • MachineArchitecture: “{platform.arch}”
    • TargetJvmVersion = 8
    • “org.lwjgl.module”: “{module}”
  • {module}{platform}RuntimeElements
    • Usage: JAVA_RUNTIME
    • Category: LIBRARY
    • Bundling: EXTERNAL
    • LibraryElements: JAR
    • OperatingSystemFamily: “{platform.os}”
    • MachineArchitecture: “{platform.arch}”
    • TargetJvmVersion: 8
    • “org.lwjgl.module”: “{module}”
  • {module}{platform}NativeElements
    • Usage: NATIVE_RUNTIME
    • Category: LIBRARY
    • Bundling: EXTERNAL
    • LibraryElements: JAR
    • OperatingSystemFamily: “{platform.os}”
    • MachineArchitecture: “{platform.arch}”
    • “org.lwjgl.module”: “{module}”
  • {module}JavadocElements
    • Usage: JAVA_RUNTIME
    • Category: DOCUMENTATION
    • Bundling: EXTERNAL
    • DocsType: JAVADOC
    • “org.lwjgl.module”: “{module}”
  • {module}SourcesElements
    • Usage: JAVA_RUNTIME
    • Category: DOCUMENTATION
    • Bundling: EXTERNAL
    • DocsType: SOURCES
    • “org.lwjgl.module”: “{module}”

The {module}ApiElements and {module}RuntimeElements variants only contain the base artifact. The {module}{platform}ApiElements and {module}{platform}RuntimeElements variants contains the base and the native artifact. The {module}{platform}NativeElements variant contains only the native artifact and could be useful if someone wants to extract the natives when releasing the application. This could be done with artifact views.

The reason for having a custom attribute containing the module name is that Gradle complains if I create more than one variant with the same attributes in a single project. I needed to add some attribute or capability to differentiate them. It was also important that the solution does not affect other build tools by modifying existing artifacts. From what I can tell I managed to achieve this.

This setup allows for this:

val lwjglVersion = "3.3.6"
val lwjglNatives = "natives-windows"

dependencies {
	
	implementation(platform("org.lwjgl:lwjgl-bom:$lwjglVersion "))

	implementation("org.lwjgl:lwjgl")
	implementation("org.lwjgl:lwjgl-glfw")
	implementation("org.lwjgl:lwjgl-opengl")
	...
	
	implementation("org.lwjgl:lwjgl::$lwjglNatives")
	implementation("org.lwjgl:lwjgl-glfw::$lwjglNatives")
	implementation("org.lwjgl:lwjgl-opengl::$lwjglNatives")
	...
	
}

to be turned into this:

val lwjglVersion = "3.3.6"

configurations.matching(Configuration::isCanBeResolved).configureEach{
	attributes {
		attribute(OperatingSystemFamily.OPERATING_SYSTEM_ATTRIBUTE, objects.named("windows"))
		attribute(MachineArchitecture.ARCHITECTURE_ATTRIBUTE, objects.named("x64"))
	}
}

dependencies {
	
	implementation(platform("org.lwjgl:lwjgl-bom:$lwjglVersion "))

	implementation("org.lwjgl:lwjgl")
	implementation("org.lwjgl:lwjgl-glfw")
	implementation("org.lwjgl:lwjgl-opengl")
	...
	
}

Finally, I have a question regarding multi project builds. If I have a lib project that uses LWJGL and sets the platform configuration attributes. Must any consumer of this lib project also set the same attributes?

I could not get it to work if I just set them in the lib project. It would seem that the selected variants and its artifacts is not propagated to any consumers. This means that the the final application must set the attributes on the runtime (runtimeOnly) configurations and the lib project must set them on the compile (implementation) configurations. Note that the native module is required at compile time due to the Java module design mentioned earlier. The code that sets the attributes I showed earlier sets them for both and could be included in both the lib and the final application project.

I will wait for your response before publishing the pull request. If you do not have anything further to discuss, you can consider this discussion finished after your response.

I’m neither an expert in designing variants, nor in using LWJGL, so I will not be able to meaningfully “approve” your solution or not.
Maybe @jendrik can have a look and helping hand.

Regarding the custom attribute, I don’t see how that should be necessary or help. From what you showed all variants have distinct attributes, and as the custom attribute has the same value on all variants it would not help anyway.

Regarding setting the attributes, neither setting them on runtimeOnly, nor on implementation should have any effect anywhere. If you set attributes on configurations, then on resolvable configurations like compileClasspath or runtimeClasspath. But if you consume a project in another project, you consume the consumable configurations like apiElements and runtimeElements and I think setting it on those would also not help. You would probably need to set the attributes that should propagate on the dependency directly, that might with.

Gradle does not complain about the variants inside each module, but rather about the same type of variant for each module. I create a software component for each module and attach custom configurations to each one. When I for example create the software components for lwjgl-glfw and lwjgl-opengl, I create the variants lwjglGlfwRuntimeElements and lwjglOpenglRuntimeElements respectively. Gradle then complains that these two variants have the same attributes/capabilities. Normally each software component is part of its own project. In this setup all software components are created dynamically in a single project, hence the need for the custom attribute.

My mistake, I meant to say apiElements and runtimeElements in the text quoted below.

In this setup all software components are created dynamically in a single project, hence the need for the custom attribute.

Urgh, but why should you do that? Use different projects and you don’t need a useless pseudo-attribute. As this whole construct is pretty convoluted anyway, you can probably also add some bad practice and do cross-project configuration if the point is to have one script for all modules or a loop that configures the modules. You can (but in normal situations shouldn’t) use project(":lwjgl-glfw") { ... } to configure that other project.

My mistake, I meant to say apiElements and runtimeElements in the text quoted below.

Does that work? As far as I remember this should have no effect, but I might be wrong.

Well, where you set attributes also depends on actual needs. If you for example set it on the lib, so that it is automatically set for the consumer, you probably cannot do cross-platform builds. That might be fine if for some reason you can only build for the current platform anyway, but otherwise maybe not.

An example. Project lib says “I need glfw” but does not specify os/arch attributes. Then app says “I need lib” and can request it’s dependencies with different sets of attributes to build distributions for different platforms. If lib fixes the attributes, app always gets the dependency for a specific platform that might not be the one it intends to build.

I know, but since they do not use Gradle for anything but publishing, they only have a single project. I too would prefer that they used separate Gradle projects, this would have made things easier. My solution was the shortcut to gain Gradle module metadata without having to migrate the whole repo to Gradle. I guess it could be possible to dynamically create Gradle projects but this seems like a bad idea.

My mistake again, it was late and I was not thinking clearly :upside_down_face:. It should be compileClasspath and runtimeClasspath as these are resolvable. I set the attributes for all resolvable configurations in the example. To break down which ones are actually needed, I mentioned that the app would have to set them on runtime (runtimeClasspath) and the lib would have to set them on compile (compileClasspath). The attributes should also be set on the equivalent test configurations. To simplify this, the example just set them on all resolvable configurations.

I guess the ideal setup would only require app to set the attributes. The lib project should be able to provide any platform to the app. One way to make this work could be to make the lib project also come in variants. It could more or less mimic the LWJGL variants and internally select the correct LWJGL variant for each of its own variants. This hides the LWJGL logic away from the app project while still allowing for cross-platform builds of the app.

I know, but since they do not use Gradle for anything but publishing, they only have a single project.

No reason not to change that in your PR, especially if the alternative is a sense-free attribute. As I said, you can still have the logic in one build script if they prefer, the physical build script of a project is optional, what projects exist is purely defined by the settings script.

I did not know that you could have completely empty projects. So if I create all projects in the settings file, I can then use subprojects{ } or better yet project(":root:${module}"){ } in the build file to configure my empty projects? Or did you have something else in mind? Also will a build directory be generated for my empty projects? If so, then I would have to store them somewhere. I would only publish artifacts (using the maven publish plugin) within each project and no build should be required?

This setup does also create problems of its own since the list of modules to publish is in an enum in the build file. I do not have access to it in the settings file, but I could scan the binary folder that contains a folder for each module. These folders then contains the artifacts, but the folder name should be enough to create all the projects. I could then use the enum to configure each empty sub project. Then I would have to integrate this into the current setup but that should not be a problem. This would eliminate the need for the custom attribute. A side effect of this would be that instead of having all publications in one project, they would each have their own. I do not see any downside to this.

So if I create all projects in the settings file, I can then use subprojects{ } or better yet project(":root:${module}"){ } in the build file to configure my empty projects

Exactly. (well, :${module} most probably, not :root:${module})
In 99.9 % of normal build scripts this is a big no-go.
But in your specific situation it might be appropriate.

Also will a build directory be generated for my empty projects?

If you run any task that creates it, yes, if not, then not.
And even if you do, the build directory is configurable, unless you use some bad plugins that have the build directory hard-coded instead of using the configured one, but that would be a bug in that plugin.

I would only publish artifacts (using the maven publish plugin) within each project and no build should be required?

I think it will create the POM and GMM in the build directory, so you might need to set a common build directory for the projects if you want to prevent the separate ones.

I do not have access to it in the settings file

Move the enum to the settings script, and just get the projects from allprojects or subprojects in the build script, or store the list of projects in a separate file that both scripts can read, or scan the file-system for subfolders / artifacts if that is suitable.

A side effect of this would be that instead of having all publications in one project, they would each have their own. I do not see any downside to this.

No, more the opposite :slight_smile:

A few thoughts from my side:

Yes, the final consumer has to say what it wants to select on its classpath. That’s the beauty of the system. A drawback for “native” Jars right now is that there is no default built into Gradle for this. Which means a consumer always needs to make some selection. I would like to write an issue at Gradle to discuss what could be done about that. As it not only concerns LWJGL but all libraries with native Jars.
We have a plugin that you can apply that configures some defaults. (But also adds tasks for jpackage.) I am thinking about extracting the variant selection part into a separate plugin. But first I want to start the general discussion if something like that could eventually become part of Gradle core.

For the variant design, there are certainly different options to design it. The following is what I would do after also looking at LWJGL as an example for some time (Here is an example using it and patching variants: GitHub - jjohannes/javarcade: Example used in presentations about Modularization and Dependency Management in Java projects)

  • I would not make a difference between Module System and not Module System. I am not sure I understand why there should be a distinction. It is a way of how Java is loading the Jars, but in the end the same Jars should be selected no matter if the Module System is used or not. Otherwise, things get unnecessarily complicated.
  • I think there is no need for compile time “native” variants. The Jars are only needed at runtime. At compile time, you only need the “main” Jar. So the standard “apiElements” variant should suffice.
  • Reiterating over this, I would no longer do what the docs do in the Component Metadata example (having variants with two Jars). Instead, I would add variants only to the native component and, in addition to the attributes, add a capability to them (aka feature variant). As I did here. The capability allows Gradle to select two different variants of the same component. Thus, I would add a rule that adds a dependency that points at the “native” variant via capability of the same component to the “main” component so that it automatically get pulled in. Unfortunately, component metadata rules do not support this yet. But in the published Gradle metdata you can do that. In my example, I add the dependencies to the “native” feature directly as I can’t do it in a rule. If I would be able to do it in the rule, the complete rule would look somehow like this:
      module("org.lwjgl:lwjgl$module") {
        // Add dependency to "natives" feature (mockup, as this is not yet available in the 
        // jvm-conflict-resolution plugin due to github.com/gradle/gradle/issues/30088)
        addRuntimeOnlyDependencyWithFeature("org.lwjgl:lwjgl$module", "natives")

        // Add serveral "natives" feature variants with different attributes
        addTargetPlatformVariant("natives",
          "natives-linux", LINUX, X86_64)
        addTargetPlatformVariant("natives",
          "natives-linux-arm64", LINUX, ARM64)
        addTargetPlatformVariant("natives",
          "natives-macos", MACOS, X86_64)
        addTargetPlatformVariant("natives",
          "natives-macos-arm64", MACOS, ARM64)
        addTargetPlatformVariant("natives",
          "natives-windows", WINDOWS, X86_64)
        addTargetPlatformVariant("natives",
          "natives-windows-arm64", WINDOWS, ARM64)
      }

I am almost gone for holidays, but I am very interested in getting this right for LWJGL as I created the issue you linked. If you continue on this, I would be happy to review a (Draft) PR. I think it is easier to discuss this on a PR we can all try out instead of here.
I will only be able to give more feedback in September though, when I am back.

1 Like

The natives are required both at runtime and compile time. This due to how LWJGL designed the java modules. The natives contain a java module and the intended way of consuming LWJGL is to require that native module in your own module. This means that the java module system will complain if the native module is not present at compile time.

This is not ideal, but no better solution was found. I gave it a shot by suggesting that the module path is scanned manually to find the natives. This would eliminate the need for requiring the native module directly. It would then be truly runtime only. Here is the response:

LWJGL having to scan the module path and manually use ModuleFinder feels off. In any case though, it only solves part of the problem (running the program). You still have to list all native modules explicitly in the jlink command. If you can do that, you can also do it at runtime. I don’t think the win here justifies the complexity.

Anyhow, I have now completed the PR and would love your feedback when you get back. I used the capability setup you described. I had to include empty variants when a native was not required. I assumed that if a capability is not found Gradle will complain.