How to use LWJGL? Or: how to use native libraries?

[kiras (kirasnarenz)]

For the last few days I have been trying to find a good way to solve this problem: I want to use LWJGL (http://lwjgl.org) in a project, but LWJGL uses native libraries (C/C++ shared libraries), which seems to make things a little less straightforward than usual. This page illustrates how to use LWJGL in Maven: http://lwjgl.org/wiki/index.php?title=LWJGL_use_in_Maven It looks like there is a Maven plugin to automatically handle this sort of thing, but even without such a plugin, it seems like it shouldn’t be too hard to accomplish the same.

It appears that they are checking for classifiers that look like “natives-${OS}” and then unpacking the jars into a specific directory that is added to java.library.path.

It appears the plugin (http://code.google.com/p/mavennatives/) offers a certain amount of customization of where the dependencies are unpacked to. By default it seems to unpack the native dependencies to the “target/natives” directory, but “when enabling separateDirs the plugin will unpack each native dependency to a subdir of the nativesTargetDir named like its classifier (for example: natives-windows will go to target/natives/windows)”. I’m not sure if there is a standard location preferred in Java projects, although the separate directories approach seems like it could be better to me.

Could anyone here show me the best way to achieve similar results with Gradle?

I should probably mention that I am completely new to both Java and Gradle, so please have patience with me.

[Szczepan_Faber]

I don’t have deep knowledge how mentioned maven plugins work. If you’re asking how to unpack dependencies based on their classifier / name, you can do it using the Copy task. Take a look at documentation/user guide around unpacking/unzipping. As far as dependencies go, you’re interested in http://gradle.org/docs/current/dsl/org.gradle.api.artifacts.Configuration.html object. It gives you ways to query/iterate through declared or resolved dependencies.

Good luck!

[kiras (kirasnarenz)]

Thanks for the reply.

I pretty much am asking how to unpack dependencies based on their classifier. As recommended, I have been looking at the Copy task and the Configuration object. Unfortunately, I have not had much luck with figuring out how to adapt this information to my needs.

Let’s say I have a simple build.gradle that looks like this:

apply plugin: 'java'
  repositories {
    mavenCentral()
}
  dependencies {
    compile 'org.lwjgl.lwjgl:lwjgl:2.8.2'
}

I can access all the dependencies through configurations.compile.dependencies and I can create a Copy task that will copy all the jars to a specified directory, for example something like this:

task copyNativesWindows(type: Copy) {
    from configurations.compile.files
    into "$buildDir/libs/natives/windows"
}

Unfortunately, this is not quite what I want. It seems like the information needed to filter the files should all be available from configurations.compile, but I’m not sure how to get at it. I’m probably missing something really obvious. If I could just get the files belonging to the dependencies that had the proper classifier, I could unpack those jars into the desired directory, but I have not been able to discover the right way to do this yet. The files seem to be File objects, that have no notion of a classifier. I have found this page which seems to offer information regarding Dependency objects http://gradle.org/docs/current/groovydoc/org/gradle/api/artifacts/Dependency.html however this doesn’t seem to help me much either, since configurations.compile.dependencies only seems to have the lwjgl dependency. I have a feeling the combination of being tired and unfamiliar with the various technologies and terminologies at play here is leading me to miss something simple.

I will continue to try to find a solution, but I’d definitely appreciate more advice.

[Szczepan_Faber]

Ok I see. Provided that you are interested in direct dependencies only, you can do:

from configurations.someConfig.files {
   it instanceof ExternalModuleDependency && it.artifacts.every { it.classifier == 'foo' }
}
 }

Cheers!

[kiras (kirasnarenz)]

Well, I tried modifying the code you posted, but I haven’t been able to make it do what I thought it should.

task copyNativesWindows(type: Copy) {
    from configurations.compile.files {
         it instanceof ExternalModuleDependency && it.artifacts.every { it.classifier == 'natives-windows' }
    }
     into "$buildDir/libs/natives/windows"
}

I would have expected this code to check each of the files in the compile configuration and if a file was both an instance of an ExternalModuleDependency and all of the artifacts it produced shared a classifier of ‘natives-windows’ to copy the jar to the specified directory. Unfortunately, it seems to just copy all of the files in configurations.compile.files. Changing the string the classifier is being matched against seems to have no effect. Did I misinterpret what you suggested?

Thanks again.

[Szczepan_Faber]

Hmmm, that definitely works. Please try to minimize the example, experiment with the closure, remove some surroundings that may affect the output (some other task putting the files in the output folder? jars were already there?, etc.)

Hope that helps!

[kiras (kirasnarenz)]

Still no luck… Here is my build.gradle:

apply plugin: 'java'
  repositories {
    mavenCentral()
}
  dependencies {
    compile 'org.lwjgl.lwjgl:lwjgl:2.8.2'
}
  task copyNativesWindows(type: Copy) {
    from configurations.compile.files {
         it instanceof ExternalModuleDependency && it.artifacts.every { it.classifier == 'natives-windows' }
    }
     into "$buildDir/natives/windows"
}

If I run “gradle clean” followed by “gradle copyNativesWindows”, $buildDir/natives/windows looks like this:

jinput-2.0.5.jar jinput-platform-2.0.5-natives-linux.jar jinput-platform-2.0.5-natives-osx.jar jinput-platform-2.0.5-natives-windows.jar jutils-1.0.0.jar lwjgl-2.8.2.jar lwjgl-platform-2.8.2-natives-linux.jar lwjgl-platform-2.8.2-natives-osx.jar lwjgl-platform-2.8.2-natives-windows.jar

I would have expected it to look more like: jinput-platform-2.0.5-natives-windows.jar lwjgl-platform-2.8.2-natives-windows.jar

Really not sure what I am doing wrong.

I have tried getting more information about what exactly is happening in copyNativesWindows. For example, I have tried something like this:

task copyNativesWindows(type: Copy) {
    from configurations.compile.files {
         println("it: " + it)
        println("it.artifacts: " + it.artifacts)
        it instanceof ExternalModuleDependency && it.artifacts.every { it.classifier == 'natives-windows' }
    }
     into "$buildDir/natives/windows"
}

My output ends up looking like this:

it: DefaultExternalModuleDependency{group=‘org.lwjgl.lwjgl’, name=‘lwjgl’, version=‘2.8.2’, configuration=‘default’} it.artifacts: [] it: DefaultExternalModuleDependency{group=‘org.lwjgl.lwjgl’, name=‘lwjgl’, version=‘2.8.2’, configuration=‘default’} it.artifacts: [] :copyNativesWindows

BUILD SUCCESSFUL

My guess as to why all these files are being copied is that every one of the artifacts pass the classifier comparison, since there seem to be no artifacts to run the comparison against. I’m not sure why it seems as though the closure is being called twice. Clearly I’m doing something wrong, but I’m not sure what. Could do me the favor of posting your build.gradle for me to examine? Any additional advice would be welcome.

[Szczepan_Faber]

I’m not sure why artifacts are always empty but the classified artifacts you’re interested in are not 1st level dependencies (take a look at the pom).

It’s getting hairy so let’s tackle it differently. How about an explicit configuration for the windows natives?

configurations {
  winNatives
}
  dependencies {
  compile 'org.lwjgl.lwjgl:lwjgl:2.8.2'
 winNatives 'org.lwjgl.lwjgl:lwjgl-platform:2.8.2:natives-windows'
}
   task copyNativesWindows(type: Copy) {
  from configurations.winNatives
 into "out"
 }

[kiras (kirasnarenz)]

I had taken a look at the pom and more or less suspected transitive dependencies to be the cause of my problems. However, being unfamiliar with Maven, I wasn’t sure if I should mention it or not.

The alternative method you suggest does solve my immediate problem, but I was hoping for a solution that didn’t rely on my specifying things so much. I did a few searches regarding transitive dependencies and I found this thread: http://gradle.1045684.n5.nabble.com/transitive-dependency-list-td4793535.html After various attempts at using the information contained in the thread, I came up with a build.gradle that looks like this:

apply plugin: 'java'
  repositories {
    mavenCentral()
}
  dependencies {
    compile 'org.lwjgl.lwjgl:lwjgl:2.8.2'
}
  task natives {
    resolvedArtifacts = configurations.compile.resolvedConfiguration.resolvedArtifacts
    osNatives('windows', resolvedArtifacts)
    osNatives('linux', resolvedArtifacts)
    osNatives('osx', resolvedArtifacts)
}
   def osNatives(os, artifacts) {
    copy {
        classifier = 'natives-' + os
        matches = artifacts.findAll { it.classifier == classifier }
        from matches.collect { it.file }.collect { zipTree(it) }
        into "$buildDir/natives/" + os
    }
}

This does what I want it to do, but I’m not sure if it is the best way to accomplish this or not. I hadn’t come across the various properties mentioned in the thread when I was looking through the documentation, so hopefully it is ok to use these properties. Considering the source of the information, I’m going to assume this is safe.

Other than that, I’m mostly wondering if the way I am handling the copying could be improved in some way. Previously I had been using a Copy Task, now I just have a normal task that calls a function that uses the copy method. What are the pros and cons of using the Copy Task vs the copy function? Do you see a clean way of changing this to use a Copy Task?

Thanks again for all your help, I really appreciate it.

[Szczepan_Faber]

Some changes I’d do:

1. copy operation is performed at the evaluation phase which is not really a good pattern in general. 2. I’d generally favor copy tasks instead of copy operation because it takes full advantage of incremental build (and solves problem #1).

Otherwise, good stuff

[kiras (kirasnarenz)]

Good points.

Taking your advice into consideration, I’ve come up with this:

apply plugin: 'java'
  repositories {
    mavenCentral()
}
  dependencies {
    compile 'org.lwjgl.lwjgl:lwjgl:2.8.2'
}
  resolvedArtifacts = configurations.compile.resolvedConfiguration.resolvedArtifacts
platforms = ['windows', 'linux', 'osx']
platforms.each {
    platform = it
    tasks.add(name: "$platform" + 'Natives', type: Copy) {
        from platformNatives(platform, resolvedArtifacts).toArray()
        into "$buildDir/natives-" + platform
    }
}
  task natives {
    description "Copies native libraries to an appropriate directory."
    dependsOn tasks.matching { it.name.contains('Natives') }
}
  def platformNatives(platform, artifacts) {
    classifier = 'natives-' + platform
    matches = artifacts.findAll { it.classifier == classifier }
    matches.collect { it.file }.collect { zipTree(it) }
}

I have a few questions about this though. First, does this solve the issues you mentioned? Second, is there a better way to do this? I feel like my previous version was both shorter and cleaner than this version, but unfortunately didn’t make use of copy tasks and suffered from the shortcomings you mentioned as a result. Can you think of a better way to use copy tasks for this than what I have so far? Any other improvements you can recommend would be appreciated as well.

Thanks again.

[Szczepan_Faber]

Hmmm, I guess you cannot really use Copy task because accessing ‘resolvedConfiguration’ will trigger resolve (an that would happen at evaluation time).

BTW. In your last version, you still perform dependency resolution at evaluation time which should be always avoided. Here’s what I came up with. Take a look at the task inputs/outputs API - this gives you ways to add ‘incrementality’ to your builds. Also the dependency resolution is at execution time.

apply plugin: ‘java’

repositories {

mavenCentral() } dependencies {

compile ‘org.lwjgl.lwjgl:lwjgl:2.8.2’ }

[‘windows’, ‘linux’, ‘osx’].each { platform ->

task “${platform}Natives” {

outputDir = “$buildDir/natives-$platform”

inputs.files(configurations.compile)

outputs.dir(outputDir)

doLast {

copy {

def artifacts = configurations.compile.resolvedConfiguration.resolvedArtifacts

.findAll { it.classifier == “natives-$platform” }

artifacts.each {

from zipTree(it.file)

}

into outputDir

}

}

} }

[kiras (kirasnarenz)]

Well, that definitely seems to work and having done some research into the different phases of the build lifecycle, I think I have a decent understanding of what the problem with my earlier versions was. I hadn’t really understood the different phases and how to specify what should happen during each phase until I spent some time looking into it, so thanks for pointing the issue out to me. Just in case it helps someone else in the future, here are a few links that I found informative:

http://gradle.1045684.n5.nabble.com/task-run-order-lt-lt-syntax-doLast-doFirst-etc-td3337481.html http://gradle.markmail.org/search/?q=#query:list:org.codehaus.gradle.user%20date:201101%20+page:1+mid:ke54bg7gn3jsnbsg+state:results

Also, this is my build.gradle as it currently stands. It’s almost identical to what you posted last, with the main difference being that I defined an additional task called natives that depends on the dynamically created platform-specific tasks. If you get a chance, maybe you could suggest a better way if my method of handling this is less than optimal.

apply plugin: 'java'
  repositories {
    mavenCentral()
}
  dependencies {
    compile 'org.lwjgl.lwjgl:lwjgl:2.8.2'
}
  platforms = ['windows', 'linux', 'osx']
platforms.each { platform ->
    task "${platform}Natives" {
        outputDir = "$buildDir/natives/$platform"
        inputs.files(configurations.compile)
        outputs.dir(outputDir)
        doLast {
            copy {
                def artifacts = configurations.compile.resolvedConfiguration.resolvedArtifacts
                    .findAll { it.classifier == "natives-$platform" }
                artifacts.each {
                     from zipTree(it.file)
                }
                            into outputDir
            }
        }
    }
}
  task natives {
    description "Copies native libraries to an appropriate directory."
    dependsOn platforms.collect { "${it}Natives" }.findAll { tasks[it] }
}

Thanks again for all your help.

[Szczepan_Faber]

Looks good. The method collecting the tasks is fine. An alternative solution might be declaring natives task first and then inside of your main ‘platform’ loop:

def task = task "${platform}Natives" { ...
natives.dependsOn task

Hope that helps!

[Jesus_Zazueta (Jesús Zazueta)]

Sorry for waking up an old question, but I think that what’s really missing with Gradle (or at least something I have yet to discover) is the ability to extract platform specific information and then choose the appropriate jars based on that.

Essentially, when we’re dealing with libraries which rely on native components we need to know:

• OS name (‘Windows, Linux, OSX, Solaris’, etc.). - OS architecture (‘x86, x86_64, sparc’, etc.). - OS architecture bit width (32, 64 bits).

That should be sufficient to know which jars (if any available) of a particular library are required on the class path. Now I think think this could be done by asking the ‘java.lang.System’ class directly in the Groovy scripts.

Is Gradle able to do this already?

Thanks!

JZM