"has more than one client module definitions" when trying to use dependencies.module more than once with different classifiers

Help/Discuss
1

This one is probably just user error.

I started out with some dependencies like this:

dependencies {
    implementation('org.nd4j:nd4j-native:0.9.1') {
        exclude group: 'org.projectlombok', module: 'lombok'
        exclude group: 'com.google.code.findbugs', module: 'annotations'
        exclude group: 'com.github.stephenc.findbugs', module: 'findbugs-annotations'
        exclude group: 'junit', module: 'junit'
    }
    implementation('org.nd4j:nd4j-native:0.9.1:windows-x86_64') {
        exclude group: 'org.projectlombok', module: 'lombok'
        exclude group: 'com.google.code.findbugs', module: 'annotations'
        exclude group: 'com.github.stephenc.findbugs', module: 'findbugs-annotations'
        exclude group: 'junit', module: 'junit'
    }
}

That works fine.

For reasons beyond the scope of this particular issue, I am trying to refactor this to:

dependencies {
    implementation(module('org.nd4j:nd4j-native:0.9.1') {
        exclude group: 'org.projectlombok', module: 'lombok'
        exclude group: 'com.google.code.findbugs', module: 'annotations'
        exclude group: 'com.github.stephenc.findbugs', module: 'findbugs-annotations'
        exclude group: 'junit', module: 'junit'
    })
    implementation(module('org.nd4j:nd4j-native:0.9.1:windows-x86_64') {
        exclude group: 'org.projectlombok', module: 'lombok'
        exclude group: 'com.google.code.findbugs', module: 'annotations'
        exclude group: 'com.github.stephenc.findbugs', module: 'findbugs-annotations'
        exclude group: 'junit', module: 'junit'
    })
}

But now I get an error:

> Could not resolve all dependencies for configuration ':integration-bom:testCompileClasspath'.
   > org.nd4j:nd4j-native:0.9.1 has more than one client module definitions.

Which is bad grammar, but beyond that, it seems like I have stumbled upon a limitation where I’m not allowed to create two ClientModule instances for the same (groupId, artifactId, version) triplet, even though the classifiers are different. I wasn’t able to dig up any documentation explaining why this is a limitation.

Is there a way to avoid this while still defining these two as client modules?

2

The only documentation about client modules I was able to dig up was DependencyHandler - Gradle DSL Version 8.4 and that indeed says:

The module notation is the same as the dependency notations described above, except that the classifier property is not available.

But despite you saying it is out of scope, I’d still like to ask why you want to use client modules.
As far as I remember they are for modeling a dependency for which no descriptor (POM, Ivy file, Gradle Module Metadata) is available to properly model its dependencies and artifacts in a semantically correct way.

To do what you want, you would need something like

implementation(module("org.nd4j:nd4j-native:0.9.1") {
    artifact {
        name = "nd4j-native"
        type = DependencyArtifact.DEFAULT_TYPE
    }
    artifact {
        name = "nd4j-native"
        type = DependencyArtifact.DEFAULT_TYPE
        classifier = "windows-x86_64"
    }
})

where you also don’t need the excludes, as you model the dependencies manually anyway and if you don’t add any, there are none.
To just get the jars, you could also simply do

implementation("org.nd4j:nd4j-native:0.9.1@jar")
implementation("org.nd4j:nd4j-native:0.9.1:windows-x86_64@jar")

If this actually is about fixing wrong metadata like dependencies in the published pom, you might instead consider using component metadata rules: Gradle User Manual: Version 8.4

3

Right, I guess the main takeaway here is that excludes aren’t necessary because when using a client module it comes with no dependencies in the first place. I had thought that it might have dragged in the declared dependencies for the module and that adding dependency lines was adding additional ones, but that was wrong.

So I have two options now then:

(a) Keep using client modules and declare the dependencies we do want. Benefit being that transitive dependencies will never drag in unexpected libraries (a real problem we have faced since switching to Gradle for dependency resolution.)

(b) Use dependencies.create() instead, which I discovered by digging through the docs, and which lets us declare the field as Dependency (giving type safety ready for throwing away Groovy for our build scripts).

We actually have a lot of problems, but it seems like the main one is figuring out how to rewrite our build to use Kotlin DSL without having to change every build script in one shot. Whereas Gradle does let us apply a Kotllin DSL script to a Groovy build and vice versa, the slight difference in how the two are structures makes this a fairly large undertaking the way things are currently structured for us.

4

That is not a benefit, it is a major drawback.
Why do you use dependencies that declare what they depend on if you don’t trust what they declare?
Gradle should give you the same transitive dependencies other tools like Maven or Ivy give you if you resolve the same libraries.
If that is not the case you maybe should better file a but about that (like recently where I discovered that optional dependencies are included if there are spaces in the tag which is pretty uncommon).

And as I said, to fix potentially wrong metadata like dependencies you should use component metadata rules instead as that is what they are for. Client modules are for compensating missing module descriptors, not for fixing dependency declarations.

And if you really just want the jars without dependencies - which I really would not recommend - then just use the artifact syntax (the one with @jar in the end in the example).

5

Hence the desire to use client modules, so that we can stop needing to trust dependencies libraries are putting in, and remove the need to do this sort of thing:

    implementation('org.openimaj:faces:1.3.10') {
        exclude group: 'colt', module: 'colt'
        exclude group: 'com.aetrion.flickr', module: 'flickrapi'
        exclude group: 'com.caffeineowl.graphics', module: 'BezierUtils'
        exclude group: 'com.esotericsoftware.kryo', module: 'kryo'
        exclude group: 'com.flickr4java', module: 'flickr4java'
        exclude group: 'com.googlecode.matrix-toolkits-java', module: 'mtj'
        exclude group: 'com.googlecode.netlib-java', module: 'netlib-java'
        exclude group: 'com.sun.media', module: 'jai-codec'
        exclude group: 'javax.media', module: 'jai-core'
        exclude group: 'com.thoughtworks.xstream', module: 'xstream'
        exclude group: 'gov.sandia.foundry', module: 'gov-sandia-cognition-common-core'
        exclude group: 'gov.sandia.foundry', module: 'gov-sandia-cognition-learning-core'
        exclude group: 'jama', module: 'jama'
        exclude group: 'jgrapht', module: 'jgrapht'
        exclude group: 'net.billylieurance.azuresearch', module: 'azure-bing-search-java'
        exclude group: 'net.sf.jafama', module: 'JaFaMa'
        exclude group: 'net.sourceforge.jeuclid', module: 'jeuclid-core'
        exclude group: 'net.sourceforge.jmatio', module: 'jmatio'
        exclude group: 'org.objenesis', module: 'objenesis'
        exclude group: 'org.apache.ant', module: 'ant'
        exclude group: 'org.apache.commons', module: 'commons-math3'
        exclude group: 'org.apache.commons', module: 'commons-vfs2'
        exclude group: 'org.jbibtex', module: 'jbibtex'
        exclude group: 'org.la4j', module: 'la4j'
        exclude group: 'org.openimaj', module: 'core-aop-support'
        exclude group: 'org.openimaj', module: 'core-experiment'
        exclude group: 'org.openimaj', module: 'core-feature'
        exclude group: 'org.openimaj', module: 'core-video'
        exclude group: 'org.openimaj', module: 'FaceTracker'
        exclude group: 'org.openimaj', module: 'image-feature-extraction'
        exclude group: 'org.openimaj', module: 'image-local-features'
        exclude group: 'org.openimaj', module: 'JTransforms'
        exclude group: 'org.openimaj', module: 'klt-tracker'
        exclude group: 'org.openimaj', module: 'machine-learning'
        exclude group: 'org.openimaj', module: 'MatrixLib'
        exclude group: 'org.openimaj', module: 'video-processing'
        exclude group: 'uk.ac.ed.ph.snuggletex', module: 'snuggletex-core'
        exclude group: 'uk.ac.ed.ph.snuggletex', module: 'snuggletex-jeuclid'
        exclude group: 'uk.ac.ed.ph.snuggletex', module: 'snuggletex-upconversion'
        exclude group: 'vigna.dsi.unimi.it', module: 'jal'
        exclude group: 'xmlpull', module: 'xmlpull'
        exclude group: 'xpp3', module: 'xpp3_min'
    }

Removing stray dependencies which you don’t actually use decreases your installer size. It is a major benefit, and anyone who can’t see that is probably not shipping their software to anyone who cares about the size of what they are downloading. (e.g., maybe you only develop web applications, in which case you don’t have to care about installer size.)

6

Or maybe what you’re asking me is, why did I choose a library with so many bad dependencies? And the answer to that is, I didn’t. Someone else on another team did, and now it’s hard to get rid of it. And this applies to many, many libraries we depend on. :frowning:

7

Well, most of the libs out there define their dependencies properly.
Granted, there are of course black sheeps that define non-sense, like depending on junit in their production dependencies and similar things.
But just because of that discarding all dependency declarations and just declaring all dependencies yourself feels more than wrong to me.
But hey, whatever works for you. :slight_smile:

8

I think it’s a bit of a mix. Forcing devs to figure out which dependencies of the library they’re adding are actually used has benefits beyond the obvious - it discourages people from using the bad libraries that depend on a dozen things to do something simple, and encourages people to pay attention to what they are causing us to ship.

To me, automatic dependency resolution necessarily causes the opposite effect:
(1) It breeds developers who forget that adding more dependencies adds to the size of what is shipped.
(2) It increases the number of bad libraries in the system.

Don’t get me wrong though, if I’m setting up a new project as a prototype, I do want all the work of figuring out the dependencies to be taken out of my hands. I just believe that it has no place in production software, where you ultimately have to ship everything you depend on.

9

But still, a component metadata rule like shown here would be more appropriate semantically. :slight_smile:

10

Yeah, that does still seem better than client modules for sure. My memory with client modules too was that it broke the BOM files we were exporting (although we have one usage of them still in our build and nobody has complained about the BOM being wrong for over a year now, so it might be fine.)

It’s hard to keep on top of all the new things being added by Gradle each version. :slight_smile:

Even using the @jar specifier might be appropriate for our case if we group together arrays of them for each dependency. At the moment I’m just trying to figure out all the options and guess how long it’s going to take to do the real work.

1 Like