Gradle picks wrong jars in multi-module project with diamond dependencies

We have a multi-module project with five modules. One of the modules depends on two other modules, and depends on an external project that depends on jars compiled from those same two modules, which is a diamond problem.

The expected behavior is that Gradle would look first in build/libs for jars compiled locally, and use those with highest priority over all other jars, but it doesn’t do that. Instead, it puts the jars from the diamond problem on the classpath, and completely skips the jars in build/libs. This makes classes that are recently added to the modules unavailable to the module that depends on the modules, and our tests fail.

The solution is to exclude the diamond-problem jars, but we don’t think we should have to do this. It’s something Gradle should do automatically. It’s hard to see an argument for using jars from a remote Maven repo in preference to local jars that were literally built a few seconds ago. Use the newest jars!

Alternatively, Gradle could set up inter-module dependencies not based on jars but based on build/classes.

Here’s what the classpath looks like when the diamond-problem jars aren’t excluded:

/home/jenkins/.gradle/caches/5.6.1/workerMain/gradle-worker.jar
/var/lib/jenkins/workspace/our-project_our_branch/module-that-depends-on-other-modules/build/tmp/expandedArchives/org.jacoco.agent-0.8.4.jar_982888894296538c98d7324f3ca78d8f/jacocoagent.jar
/var/lib/jenkins/workspace/our-project_our_branch/module-that-depends-on-other-modules/build/classes/java/test/
/var/lib/jenkins/workspace/our-project_our_branch/module-that-depends-on-other-modules/build/resources/test/
/var/lib/jenkins/workspace/our-project_our_branch/module-that-depends-on-other-modules/build/classes/java/main/
/var/lib/jenkins/workspace/our-project_our_branch/module-that-depends-on-other-modules/build/resources/main/
/home/jenkins/.gradle/caches/modules-2/files-2.1/com.our.company/other-project-diamond1/10.0.0-RELEASE/a1ab46f68d634411329ab7982527e5140caa5ddd/other-project-diamond1-10.0.0-RELEASE.jar
/home/jenkins/.gradle/caches/modules-2/files-2.1/com.our.company/other-project-diamond2/5.0.3-RELEASE/6b864e8a62cdf5f3426c3193038903e54156c953/other-project-diamond2-5.0.3-RELEASE.jar
/home/jenkins/.gradle/caches/modules-2/files-2.1/com.our.company/first-depended-upon-module/5.5.9-RELEASE/adb6494ba9c1a16868cbb0eaab56667269eb200c/first-depended-upon-module-5.5.9-RELEASE.jar THIS IS WRONG
/home/jenkins/.gradle/caches/modules-2/files-2.1/com.our.company/second-depended-upon-module/3.5.8-RELEASE/4b548005f70694b1c51a5f558eefb7c73f46b2a9/second-depended-upon-module-3.5.8-RELEASE.jar THIS IS WRONG

And here’s what the classpath looks like when the diamond-problem jars are excluded:

/home/jenkins/.gradle/caches/5.6.1/workerMain/gradle-worker.jar
/var/lib/jenkins/workspace/our-project_our_branch/module-that-depends-on-other-modules/build/tmp/expandedArchives/org.jacoco.agent-0.8.4.jar_982888894296538c98d7324f3ca78d8f/jacocoagent.jar
/var/lib/jenkins/workspace/our-project_our_branch/module-that-depends-on-other-modules/build/classes/java/test/
/var/lib/jenkins/workspace/our-project_our_branch/module-that-depends-on-other-modules/build/resources/test/
/var/lib/jenkins/workspace/our-project_our_branch/module-that-depends-on-other-modules/build/classes/java/main/
/var/lib/jenkins/workspace/our-project_our_branch/module-that-depends-on-other-modules/build/resources/main/
/var/lib/jenkins/workspace/our-project_our_branch/first-depended-upon-module/build/libs/first-depended-upon-module-21.1.0-dev.7.uncommitted+tracing.poc.36d23df.jar
/var/lib/jenkins/workspace/our-project_our_branch/second-depended-upon-module/build/libs/second-depended-upon-module-21.1.0-dev.7.uncommitted+tracing.poc.36d23df.jar
/home/jenkins/.gradle/caches/modules-2/files-2.1/com.our.company/other-project-diamond1/10.0.0-RELEASE/a1ab46f68d634411329ab7982527e5140caa5ddd/other-project-diamond1-10.0.0-RELEASE.jar
/home/jenkins/.gradle/caches/modules-2/files-2.1/com.our.company/other-project-diamond2/5.0.3-RELEASE/6b864e8a62cdf5f3426c3193038903e54156c953/other-project-diamond2-5.0.3-RELEASE.jar

I have to assume a number of things, based upon the classpath examples you posted. It will help if you maybe add the output of gradle dependencies --configuration runtime.

Also:

  • By default Gradle will always select the higher version of a dependency when more than one vesin is found. If this is not satisfactory then, resolutionStrategy or some simplification thereof can be utilised.
  • If the dependency is found in a linked project it will be preferred over the remoe version (unless you have configured it not to).
  • If the depedency is build in a sibliing project, but NOT linked to the current subproject via project(':NAME') it will use the remote dependency instead.
  • If you specifiy both a project dependency and a remote dependency I suspect the higher version of the artifact will still be used.

That’s what isn’t happening. Despite being linked via project(“:local-project”) it’s pulling in the remote.

It’s possibly some interaction with the Nebula plugin, which creates mangled jar names (“first-depended-upon-module-21.1.0-dev.7.uncommitted+tracing.poc.36d23df.jar”), but even so, there’s no reason Gradle should prefer remote over local in a multi-module build.

I’ll dump out the configuration tomorrow and post it here.

That produces no output. Maybe it doesn’t work on multi-module projects?

I found How can I get a list of all configurations for a Gradle project? - Stack Overflow, and doing what it says produces the following output. Is that what you were looking for?

annotationProcessor - Annotation processors and their dependencies for source set ‘main’.
api - API dependencies for source set ‘main’. (n)
apiElements - API elements for main. (n)
archives - Configuration for archive artifacts.
compile - Dependencies for source set ‘main’ (deprecated, use ‘implementation’ instead).
compileClasspath - Compile classpath for source set ‘main’.
compileOnly - Compile only dependencies for source set ‘main’.
default - Configuration for default artifacts.
implementation - Implementation only dependencies for source set ‘main’. (n)
jacocoAgent - The Jacoco agent to use to get coverage data.
jacocoAnt - The Jacoco ant tasks to use to get execute Gradle tasks.
runtime - Runtime dependencies for source set ‘main’ (deprecated, use ‘runtimeOnly’ instead).
runtimeClasspath - Runtime classpath of source set ‘main’.
runtimeElements - Elements of runtime for main. (n)
runtimeOnly - Runtime only dependencies for source set ‘main’. (n)
testAnnotationProcessor - Annotation processors and their dependencies for source set ‘test’.
testCompile - Dependencies for source set ‘test’ (deprecated, use ‘testImplementation’ instead).
testCompileClasspath - Compile classpath for source set ‘test’.
testCompileOnly - Compile only dependencies for source set ‘test’.
testImplementation - Implementation only dependencies for source set ‘test’. (n)
testRuntime - Runtime dependencies for source set ‘test’ (deprecated, use ‘testRuntimeOnly’ instead).
testRuntimeClasspath - Runtime classpath of source set ‘test’.
testRuntimeOnly - Runtime only dependencies for source set ‘test’. (n)

You need to run the dependencies task of the subproject whose dependencies you want to dump. As you are not going into detail of your setup, that’s as precise I can get.

On a tangent: from this and your other posts, it looks that you like Maven and are annoyed by Gradle. It also looks like you would rather throw a solution back with “it doesn’t work” (not the most useful feedback) than take a hint and try to learn how to make it work…

If your project is a good fit for Maven, and you are not prepared to spend the time learning Gradle, why not stick to what you know?

We’re a Gradle shop. The problem isn’t disliking Gradle, it’s that we keep hitting problems in Gradle.

In this particular case, the problem can be worked around by excluding the older jar, but that goes counter to the expected behavior of Gradle, which is to pick the newest jar unless told to do otherwise.

You yourself said, quote: “By default Gradle will always select the higher version of a dependency when more than one version is found.”, but I assure you, in the case described here, that is not the actual behavior.

Try it yourself?

Unfortunately newer platform features (i.e BOMs) can cause a version to be selected that is lower. Even with your own resolution strategies within the build file, things might still nto work. Ordering suddenly becomes important.