Handling LWJGL natives with custom dependency configurations

Background

I am using a multi-project gradle build to develop a java application. The build contains both library and application sub-projects. The applications depend on the libraries and these can in turn depend on other libraries. Each sub-project is a java module that requires other modules in a module-info.java This must must be kept in sync with the dependencies in the gradle build script.

Problem

Some of the libraries depend on LWJGL which contains natives. The natives are separate dependencies that need to be handled. The way that I initially solved this was to make the module depend on the .native version of the LWJGL modules. This also requires the build script to put the natives on the implementation configuration. This is less ideal as the java module should not care about natives.

The build script generated by the LWJGL customize tool on their website recommends putting the natives on the runtimeOnly configuration. The java module should then require the non-native modules. This does however result in the problem I am trying to solve. Due to the native dependencies (which are included in the runtimeOnly configuration) ending up on the module path, they are only loaded if a module requires them. This means that I have to include them in the classpath instead. The natives would then need to be extracted into a folder which can be added to the classpath.

Only the applications need the natives to run and should therefore be the one to collect the natives. It is important that all natives from all libraries that the application depends on are collected. This includes libraries that depend on other libraries.

Partial Solution

After a bit of searching and looking at other projects, I came up with the solution of adding the native dependencies to a custom dependency configuration. This has the benefit of clear seperation between natives and non-natives. It would also allow the application sub-projects to collect all the natives through gradles existing dependency system.

Looking at the documentation for dependency configurations and the java library plugin, this would mean adding the configurations natives, nativesClasspath and nativeElements. This would mimic the standard configurations created by the java plugins. The following is the build script for creating the configurations.

configurations.register("natives") {
	canBeConsumed = false
	canBeResolved = false
	transitive = true
}

configurations.register("nativesPath") {
	canBeConsumed = false
	canBeResolved = true
	transitive = true
	extendsFrom(configurations.natives)
	attributes {
		attribute(Usage.USAGE_ATTRIBUTE, objects.named(Usage, 'java-natives'))
	}
}

configurations.register("nativesElements") {
	canBeConsumed = true
	canBeResolved = false
	transitive = false
	extendsFrom(configurations.natives)
	attributes {
		attribute(Usage.USAGE_ATTRIBUTE, objects.named(Usage, 'java-natives'))
	}
}

From my understanding, I should then be able to add native dependencies like this.

natives 'org.lwjgl:lwjgl-glfw::natives-windows'
natives project(':lib')

The nativesClasspath configuration on the application sub-projects should then contain all the natives. These could then be extracted into a folder using a copy task such as this.

tasks.register('extractNatives', Copy) {
	
	group = "natives"
	description = "Extracts native libraries from all subprojects"
	
	inputs.files configurations.nativesClasspath
	outputs.dir "$buildDir/natives"
	
	duplicatesStrategy = DuplicatesStrategy.EXCLUDE
	
	from {
		configurations.nativesClasspath.files
	}
	
	into "$buildDir/natives"
	
}

Finally, the folder can be added to the classpath of the run task and the eclipse project files. It can also be included in releases using the distribution plugin.

The problem with this is that the LWJGL platform/BOM that supplies the versions for all libraries and natives seem to require that the configuration uses the usage attribute java-runtime instead of a custom one. I do not know what effect it would have to use java-runtime for natives as well. Also, the three configurations seem to need some artifact setup as the natives are not collected properly. Only the projects end up in the dependency graph and not their natives.

Alternatives

I do not know if using the three configurations is overkill and fewer would be better. The java configurations seem to suggest it would be best to separate the configuration into these three. I could also just list the natives in the application sub-projects but that would defeat the purpose of having a separate library. Also, if using custom configurations is not the right way to do things, then please point me in a better direction. I am not necessarily looking for a specific solution, just a good one.

From a cursory look I’d say lwjgl should optimally publish proper feature variants for the native variants. If it does not, you can request it as feature and in the meantime use a component metadata rule to dynamically add these feature variants.

Then you can use an artifact view with variant reselection on the runtime classpath configurationto get the native variants of all dependencies that have one.

I have now done some research into feature variants, variants & attributes, variant selection and artifact views.

If I understand you correctly, each module (maven publication) should contain a gradle module metadata file alongside the pom file. This tells gradle what variants the module contain and what attributes they have. There would then be a variant with only the base jar artifact and one with both the base and native jar artifacts. There could also be other variants that only have the native artifact or one with natives artifacts for all platforms.

Lwjgl does not currently publish this gradle module metadata which means that I have to add it manually. This can be done using component metadata rules which adds metadata to lwjgl modules. The docs for component metadata rules interestingly use lwjgl as a use case.

Also to clarify how lwjgl currently handles natives, they have a single module for both the base library jar artifact and the corresponding native jar artifacts. To choose a native jar, classifiers are used.

So far only regular variants have been discussed. You mentioned feature variants, but those seem to add optional code from source sets. In this scenario the variants only need to have different artifacts that are already built. Regular variants seem to fit this need better.

When the variants have been created, I can then add attributes to the runtime configuration to specify which platform to use. I can also create an extract natives task that creates an artifact view that selects the variant which only includes the natives.

This solves the problem of finding and selecting natives automatically. Then there is the original problem of actually making sure that the natives get used during the execution of the application. My suggestion was to extract them to a folder and then add this folder to the classpath. This would make sure that they are loaded since the module path is not used. If I understand you correctly, this is the way to go. An alternative would be if I could have gradle use both the module and classpath at the same time but I do not know if this is possible.

This is the complete build.gradle script that I have come up with so far.


plugins {
	id 'application'
}

repositories {
	mavenCentral()
}

@CacheableRule
abstract class LwjglRule implements ComponentMetadataRule {
	
	private def nativeVariants = [
		[os: OperatingSystemFamily.LINUX,   arch: "arm32",  classifier: "natives-linux-arm32"],
		[os: OperatingSystemFamily.LINUX,   arch: "arm64",  classifier: "natives-linux-arm64"],
		[os: OperatingSystemFamily.WINDOWS, arch: "x86",    classifier: "natives-windows-x86"],
		[os: OperatingSystemFamily.WINDOWS, arch: "x86-64", classifier: "natives-windows"],
		[os: OperatingSystemFamily.MACOS,   arch: "x86-64", classifier: "natives-macos"]
	]
	
	@Inject abstract ObjectFactory getObjects()
	
	void execute(ComponentMetadataContext context) {
		
		def natives = nativeVariants
		
		context.details.withVariant("runtime") {
			attributes {
				attributes.attribute(OperatingSystemFamily.OPERATING_SYSTEM_ATTRIBUTE, objects.named(OperatingSystemFamily, "none"))
				attributes.attribute(MachineArchitecture.ARCHITECTURE_ATTRIBUTE, objects.named(MachineArchitecture, "none"))
			}
		}
		
		natives.each { variantDefinition ->
			
			context.details.addVariant("${variantDefinition.classifier}-runtime", "runtime") {
				
				attributes {
					attributes.attribute(OperatingSystemFamily.OPERATING_SYSTEM_ATTRIBUTE, objects.named(OperatingSystemFamily, variantDefinition.os))
					attributes.attribute(MachineArchitecture.ARCHITECTURE_ATTRIBUTE, objects.named(MachineArchitecture, variantDefinition.arch))
				}
				
				withFiles {
					addFile("${context.details.id.name}-${context.details.id.version}-${variantDefinition.classifier}.jar")
				}
				
			}
			
			context.details.addVariant("${variantDefinition.classifier}") {
				
				attributes {
					attributes.attribute(Usage.USAGE_ATTRIBUTE, objects.named(Usage, "none"))
					attributes.attribute(OperatingSystemFamily.OPERATING_SYSTEM_ATTRIBUTE, objects.named(OperatingSystemFamily, variantDefinition.os))
					attributes.attribute(MachineArchitecture.ARCHITECTURE_ATTRIBUTE, objects.named(MachineArchitecture, variantDefinition.arch))
				}
				
				withFiles {
					addFile("${context.details.id.name}-${context.details.id.version}-${variantDefinition.classifier}.jar")
				}
				
			}
			
		}
		
		context.details.addVariant("natives-all-runtime", "runtime") {
			
			attributes {
				attributes.attribute(OperatingSystemFamily.OPERATING_SYSTEM_ATTRIBUTE, objects.named(OperatingSystemFamily, "all"))
				attributes.attribute(MachineArchitecture.ARCHITECTURE_ATTRIBUTE, objects.named(MachineArchitecture, "all"))
			}
			
			withFiles {
				natives.each { variantDefinition ->
					addFile("${context.details.id.name}-${context.details.id.version}-${variantDefinition.classifier}.jar")
				}
			}
			
		}
		
		context.details.addVariant("natives-all") {
			
			attributes {
				attributes.attribute(Usage.USAGE_ATTRIBUTE, objects.named(Usage, "none"))
				attributes.attribute(OperatingSystemFamily.OPERATING_SYSTEM_ATTRIBUTE, objects.named(OperatingSystemFamily, "all"))
				attributes.attribute(MachineArchitecture.ARCHITECTURE_ATTRIBUTE, objects.named(MachineArchitecture, "all"))
			}
			
			withFiles {
				natives.each { variantDefinition ->
					addFile("${context.details.id.name}-${context.details.id.version}-${variantDefinition.classifier}.jar")
				}
			}
			
		}
		
	}
	
}

def nativeConfigurations = [
    configurations.runtimeClasspath,
    configurations.testRuntimeClasspath
]

nativeConfigurations.each{ config ->
	config.attributes {
		attribute(OperatingSystemFamily.OPERATING_SYSTEM_ATTRIBUTE, objects.named(OperatingSystemFamily, OperatingSystemFamily.WINDOWS))
		attribute(MachineArchitecture.ARCHITECTURE_ATTRIBUTE, objects.named(MachineArchitecture, "x86-64"))
	}
}

project.ext.lwjglVersion = "3.3.6"

dependencies {
	
	components {
        withModule("org.lwjgl:lwjgl", LwjglRule)
		withModule("org.lwjgl:lwjgl-glfw", LwjglRule)
    }
	
	implementation platform("org.lwjgl:lwjgl-bom:$lwjglVersion")
	
	implementation "org.lwjgl:lwjgl"
	implementation "org.lwjgl:lwjgl-glfw"
	
	testImplementation libs.junit
	testRuntimeOnly 'org.junit.platform:junit-platform-launcher'
	
}

application {
	mainModule = 'com.dwarfley.app'
	mainClass = 'com.dwarfley.app.App'
}

java {
	toolchain {
		languageVersion = JavaLanguageVersion.of(21)
	}
}

tasks.named('test') {
	useJUnitPlatform()
}

tasks.register('extractNatives', Copy) {
	
	group = "natives"
	description = "Extracts natives from all runtime dependencies"
	
	def artifactView = configurations.runtimeClasspath.incoming.artifactView {
		
		withVariantReselection()
		
		attributes {
			attribute(Usage.USAGE_ATTRIBUTE, objects.named(Usage, "none"))
			attribute(OperatingSystemFamily.OPERATING_SYSTEM_ATTRIBUTE, objects.named(OperatingSystemFamily, "all"))
			attribute(MachineArchitecture.ARCHITECTURE_ATTRIBUTE, objects.named(MachineArchitecture, "all"))
		}
		
	}
	
	inputs.files artifactView.files
	outputs.dir "$buildDir/natives"
	
	duplicatesStrategy = DuplicatesStrategy.EXCLUDE
	
	from artifactView.files
	into "$buildDir/natives"
	
}

tasks.named('run') {
	
	jvmArgs += ["--class-path", "$buildDir\\natives\\*"]
	
    dependsOn tasks.named('extractNatives')
	
}

Is this what you had in mind? I think I got it all correct including configuration avoidance.

This adds the following variants with their attributes:

  • runtime
    • os: none, arch: none, usage: java-runtime
  • natives-(platform)-runtime
    • os: (platform), arch: (platform), usage: java-runtime
  • natives-(platform)
    • os: (platform), arch: (platform), usage: none
  • natives-all-runtime
    • os: all, arch: all, usage: java-runtime
  • natives-all
    • os: all, arch: all, usage: none

If not using java modules, the attributes for the configurations should be the target platform. If using modules, the attributes should be “none” as the natives should be extracted instead of being included in the runtime configuration. The extract task then uses the natives-all variant which avoids the base jar artifact from being extracted. It also extracts all natives so the app becomes platofrm independant. Finally, the run task is configured to depend on the extract task and its classpath is modified to inculde the natives. It would have been better if the same build script could be used for both modular and non modular java projects. However since the natives need to be extracted in modular projects, this would be impossible.

This works quite nicely. The next thing to do is adding the native folder to the eclipse project files and distributions. After that, the only thing remaining is to make lwjgl publish the metadata. I discovered a github issue related to this. There does not seem to be any active development that I can see, so I could provide a PR to speed things up. I only ask for a final ok from you that I have understood everything correctly and that the plan/system is a good one before spending time on a PR.

The way that lwjgl handles building and publising is also a bit awkward since they use ant to build the artifacts and gradle to publish them (maven-publish plugin). This means that there is no real software component and gradle can not generate the metadata. One solution could be to manually create custom components in an ad-hoc way. I will have to investigate this. Here is a link to their build script for reference.

If I understand you correctly, each module (maven publication) should contain a gradle module metadata file alongside the pom file. This tells gradle what variants the module contain and what attributes they have.

Yes, the GMM is like the POM just way richer and not so limited, it can bear more information like various variants or even just additionally attributes for the main variant like the Java version compatibility information that Gradle consumers can use.

There would then be a variant with only the base jar artifact and one with both the base and native jar artifacts.

Whether the additional variant contains only the native artifact or both depends. If you typically need the natives separate maybe only the native artifact would be in. Or if always need both together in the same classpath maybe you right away have both artifacts in the variant. Or you could also have both variants if it makes sense. If two variants have different capabilities, you could also depend on both in the same configuration.

There could also be other variants that only have the native artifact or one with natives artifacts for all platforms.

Theoretically yes, if it makes sense and you figure out proper attribute constructions that make sense and work properly. Figuring out the proper attributes and attribute values is one of the hardest parts in defining variants.

Also to clarify how lwjgl currently handles natives, they have a single module for both the base library jar artifact and the corresponding native jar artifacts. To choose a native jar, classifiers are used.

Yes, that’s the typical Maven way of representing variants as the POM cannot express them. That’s one of the reasons for GMM. Almost all “classified” artifacts would with GMM be modeled as variants, sometimes only with different attributes, sometimes also with different capability. With GMM different variants can also have different dependencies, …

The “sources” jar and the “javadoc” jar are also variants in GMM if done properly and can be resolved using attribute-aware resolution.

So far only regular variants have been discussed. You mentioned feature variants, but those seem to add optional code from source sets. In this scenario the variants only need to have different artifacts that are already built. Regular variants seem to fit this need better.

I didn’t there is a difference between “feature variant” and “variant”. I think I usually use these terms interchangeably. There are just variants that have different attributes and same or different capabilities.

When the variants have been created, I can then add attributes to the runtime configuration to specify which platform to use.

That’s the idea, yeah.
Either on a dependency directly, or on a resolvable configuration, or on an artifact view, depending on concrete situation and needs.

My suggestion was to extract them to a folder and then add this folder to the classpath. This would make sure that they are loaded since the module path is not used. If I understand you correctly, this is the way to go.

I cannot really tell you, I don’t know how lwjgl works and what it needs where and how it behaves with things on the module or classpath, that is probably not so much a Gradle topic? :man_shrugging:

Usually I’d now probably ask for an MCVE, but I’m on vacation without computer at hand, so I could not have a look anyway.

An alternative would be if I could have gradle use both the module and classpath at the same time but I do not know if this is possible.

Should be, yes

Is this what you had in mind? I think I got it all correct including configuration avoidance.

As I said, no box right now, so cannot really properly look at, so just a few points from a very cursory look:

  • why don’t you use the standard values for the architecture attribute?
  • ext / extra properties are practically always a work-around for not doing something properly and you should always feel directly when using them. Here due example either store version in a version catalog TOML file, or at least use just a local variable instead of an ext property.
  • Copy or copy { ... } is very seldomly what you want, usually you instead intended to use Sync or sync { ... }.
  • Usage “none” does not sound like it should be used for anything anytime, my gut feeling says that this is never an appropriate value for that attribute.
  • Don’t use buildDir, that’s deprecated, use layout.buildDirectory instead.
  • almost always when you need to specify a duplicates strategy, it is more a smell pointing at a quite different problem. Usually you want to find out why files are duplicate actually and fix that instead.
  • I don’t think the explicit inputs and outputs configuration for the extract task is necessary, the from and into should already take care of that
  • maybe reconsider the task name, as it does not extract anything but just copies some files together
  • reconsider whether you need that task at all or if it is just wasted time, when you could also just directly use the files where they are for the classpath
  • practically any explicit dependsOn that does not have a lifecycle task on the left-hand side is a code smell and usually a sign that you do not properly write task outputs to task inputs. If you really need the extract task and really need the manual jvm arg, instead use a jvmargprovider that properly declares the inputs so that the run task inherites that information, but probably better configure the classpath property and use the files where they are without copying
  • more a personal recommendation, switch to Kotlin DSL. By now it is the default DSL, you immediately get type-safe build scripts, actually helpful error messages if you mess up the syntax, and amazingly better IDE support if you use a good IDE like IntelliJ IDEA or Android Studio.

I only ask for a final ok from you that I have understood everything correctly and that the plan/system is a good one before spending time on a PR.

No idea, I have no idea about lwjgl and what makes or does not make sense, besides general things I said already above.