How to force a dependency version while also substituting a transitive dependency?

@CedricChampeau Moving https://github.com/gradle/gradle/issues/5190 to forums.

My gradle project depends on third-party packages that transitively depend on various implementations of JSR-305 annotations:

  • com.google.code.findbugs:jsr305
  • net.jcip:jcip-annotations
  • com.github.stephenc.jcip:jcip-annotations
  • com.google.code.findbugs:annotations

I’d like to avoid duplicate classes at runtime, so I’m keeping just the last one and substituting the others.

I also need to make sure that certain dependencies or groups of dependencies use some exact versions:

  • io.grpc:grpc-core, io.grpc:grpc-netty, io.grpc:grpc-auth, etc all need to be 1.11.0
  • com.fasterxml.jackson.core:jackson-databind, com.fasterxml.jackson.datatype:jackson-datatype-joda, com.fasterxml.jackson.core:jackson-core, etc all need to be 2.8.10
  • other groups of packages
  • junit:junit needs to be 4.12, javax.ws.rs:javax.ws.rs-api needs to be 2.0.1, etc

To ensure I compile against those versions and end up with them at runtime as well, I’m using force directives.

Here’s what my resolutionStrategy ends up looking like:

allprojects {
  configurations.all {
    resolutionStrategy {
      preferProjectModules()
      dependencySubstitution {
        // clashes with com.google.code.findbugs:annotations which includes jsr305 and more
        substitute(module("com.google.code.findbugs:jsr305")).with(module("com.google.code.findbugs:annotations:3.0.1"))
        substitute(module("net.jcip:jcip-annotations")).with(module("com.google.code.findbugs:annotations:3.0.1"))
        substitute(module("com.github.stephenc.jcip:jcip-annotations")).with(module("com.google.code.findbugs:annotations:3.0.1"))
       ...
     }

     force("io.grpc:grpc-core:1.11.0")
     force("io.grpc:grpc-netty:1.11.0")
     ... // and so on for a bunch of packages

Expected Behavior

I’d expect packages that I asked to be substituted like “com.google.code.findbugs:jsr305” to actually end up substituted.

Current Behavior

The dependency graph shows them sticking around. For example:

|    |    |    |    +--- com.twitter:util-cache_2.12:18.3.0
|    |    |    |    |    +--- org.scala-lang:scala-library:2.12.4 -> 2.12.5
|    |    |    |    |    +--- com.twitter:util-core_2.12:18.3.0 (*)
|    |    |    |    |    +--- com.github.ben-manes.caffeine:caffeine:2.3.4 -> 2.6.2
|    |    |    |    |    \--- com.google.code.findbugs:jsr305:2.0.1 -> 3.0.2

Context

Consequently I’m having a difficult time locking down my dependencies while at the same time applying necessary substitutions.

Is there a way to apply both force and substitute without one disabling the other?

1 Like

Sorry, it looks like this was a mistake in when I applied the resolution strategy block.

I had a conditional around it that only applied the resolution strategy block when the configuration name was one of “compile”, “testCompile”, “implementation”, “testImplementation”, or “runtimeClasspath”. I was whitelisting those configurations to avoid forcing wrong library versions for the “zinc” configuration used by the Scala plugin. Consequently I missed a number of other configurations.

I’ved changed that filtering now to exclude “zinc” and apply the rules to all others.

Avoiding the “zinc” configuration was an attempt to work around https://github.com/typesafehub/zinc/issues/87

Hi @dsilvasc,

This question actually mixes different problems, and it’s a great opportunity for me to introduce some recent changes in the dependency management engine (available in 4.7) that we’re working on. Most of them are not publicized yet because we need to gather feedback, so we’d appreciate if you could try it out.

Capabilities

The first problem you have is that you have several implementations of JSR-305 on the dependency graph. This is a very common problem, and a painful one. Loggers and their plethora of bindings is another one. For this, we separate conceptually things in 2 categories:

  1. realizing that you have a problem
  2. fixing the problem

For 1., we introduced the concept of “capability”. com.google.code.findbugs:jsr305, net.jcip:jcip-annotations, com.github.stephenc.jcip:jcip-annotations and com.google.code.findbugs:annotations are all things that provide the same capability (jsr305). This is something that should be declared, and, ideally, published by the authors of those libraries. Because we live in an unperfect world, and because Gradle metadata is not yet ready for prime time, the horrible truth is that they can’t explain this. Worry no more, we’re going to fix this. Gradle now provides the ability to modify published metadata on the consumer side. We call this metadata rules. Those rules can be used to “patch” external module metadata with knowledge which wasn’t available at publication time. So the idea here will be to declare a capability on all those modules.

A capability is similar to a module. It has coordinates (group, artifact, version) and by default, any module provides a capability corresponding to its GAV (so, the module com:foo:1.0 provides the capability com:foo:1.0). Capabilities participate in conflict resolution. So, what we want to say is that com.google.code.findbugs:jsr305, net.jcip:jcip-annotations, com.github.stephenc.jcip:jcip-annotations and com.google.code.findbugs:annotations all provide the jsr305 capability. Obviously, there’s no such thing yet, so we’re going to declare an arbitrary one: jsr:jsr305:1.0. We can do this using metadata rules:

dependencies {
    components {
        ['com.google.code.findbugs:jsr305', 'net.jcip:jcip-annotations', 'com.github.stephenc.jcip:jcip-annotations', 'com.google.code.findbugs:annotations'].each { provider ->
            withModule(provider) { details ->
                allVariants {
                    withCapabilities {
                        addCapability('jsr', 'jsr305', '1.0')
                    }
                }
            }
        }
    }
}

Now if you run your build again, you will notice that Gradle fails as soon as it finds 2 of those modules on the classpath. Take this sample build script:

plugins {
    id 'java-library'
}

repositories {
    jcenter()
}

dependencies {
    components {
        ['com.google.code.findbugs:jsr305', 'net.jcip:jcip-annotations', 'com.github.stephenc.jcip:jcip-annotations', 'com.google.code.findbugs:annotations'].each { provider ->
            withModule(provider) { details ->
                allVariants {
                    withCapabilities {
                        addCapability('jsr', 'jsr305', '1.0')
                    }
                }
            }
        }
    }

    implementation "com.google.code.findbugs:jsr305:2.0.1"
    implementation "net.jcip:jcip-annotations:1.0"
}

And let’s run gradle dependencies. The build will now fail with an error saying that both libraries provide the same capability and that you have to choose.

So at this point, you might wonder why we did this, because in the end Gradle fails, and it’s not nice. Well, we did this precisely for that. Imagine that you didn’t have to write the rules above, because all of the providers live in the Gradle world, which publishes the capabilities. Then, as soon as you add a dependency that introduces a conflict, we find it!. So you can take this as a pre-emptive failure. The nice thing is that the error message tells you what happens.

Ok so now, we need to fix the problem. There are several ways to do it. The first one is the one you have done. I think it’s a good solution, and we also do this internally at Gradle. But another option is to rely on capabilities again. By default, capabilities are upgraded to the latest version. So if one of the providers has a higher version, it’s going to be selected. So you could change the rule to put 1.1 on com.google.code.findbugs:jsr305, and leave 1.0 for the others. This way your preferred implementation is going to be selected automatically by Gradle! (Soon the build scan will also tell you that it was selected because it provides a higher version of the capability).

Another option would be to change the metadata of all components to remove the dependencies on the other implementations. The same dependencies.components.all { ... } API can be used to do this, but let’s move on to the next problem.

Strict constraints

You are saying:

I also need to make sure that certain dependencies or groups of dependencies use some exact versions

Again, this is a requirement that we want you to be able to express. For this, we have richer version constraints. Your build script would typically say:

dependencies {
   implementation("io.grpc:grpc-core") {
      version {
         strictly "1.11.0"
      }
      because "whatever reason why it doesn't work on other versions"
   }
}

If you write this, you’re explaining to Gradle that if it ever finds, at any point in time, a transitive dependency which disagrees with this 1.11.0, then the build is going to fail. Again, this is just pre-emptive, meaning that you declare your requirements, but as soon as the build fails, you have to fix it.

For that, you have used force, but a better solution would be to use, now the component metadata rules again. Because you know that your project only works with 1.11.0 and that you thoughtfully want to ignore whatever version other components want (you take the responsibility), you can “fix” the metadata of those components:

dependencies {
    components.all {
        allVariants {
            withDependencies { deps ->
                deps.each { dep ->
                    if (dep.group == 'io.grpc' && dep.name =='grpc-core') {
                        dep.version {
                            prefer "1.11.0"
                        }
                        dep.because "We only work with 1.11.0"
                    }
                }
            }
        }
    }
}

Note that component metadata rules work independently of the configuration. So if you need a different behavior depending on the configuration you resolve, they are not the right answer. If that’s the case, let us know.

2 Likes

Hi @CedricChampeau,

Thank you for the explanations here.

Capabilities look interesting, although I don’t understand why they would be versioned instead of the the build script explicitly choosing a provider to use. What would it mean for the authors of com.google.code.findbugs:jsr305 to claim they provide jsr305 version 1.123 and the authors of net.jcip:jcip-annotations to claim they provide jsr305 version 1.456? As a consumer I’d have to override their declared capability versions in order to choose one myself.

I can see that it’s helpful to ask Gradle to fail whenever there’s a version conflict, but it seems painful to do this for every dependency instead of specifying a failOnVersionConflict resolution strategy once for the entire multi-project.

I’m not sure that solves the issue of needing all members of the io.grpc group to be locked to the same version though (or all members of the com.google.appengine group, the org.glassfish.jersey.* groups, the com.fasterxml.jackson.* groups, etc). If I compile and run against grpc-core and grpc-netty both at version 1.11.0 then I know I’m running against a combination that the grpc team tested together; if one is at 1.11.0 and the other at 1.9.0 then all bets are off. I want to specify this in a single place (not at each sub-project that includes one of those dependencies directly or transitively), and for all configurations (not just implementation). This would be similar in concept to a strict Maven BOM, an npm package lockfile, a ruby bundler lockfile, or an Amazon version set.

What’s the difference between saying force("io.grpc:grpc-core:1.11.0") (in a declarative manner) and mutating the version of any dependency on io.grpc:grpc-core (in an imperative manner)? Isn’t that what Gradle ends up doing when it processes force constraints?

I don’t understand why they would be versioned instead of the the build script explicitly choosing a provider to use

You’re right and that’s why it’s not necessarily the right solution to fix this problem, and that module replacement rules are probably the best answer. Capabilities are versioned typically because a module which provides the 1.1 version of an API is not the same as a module which provides the 1.0 version of the same API.

it seems painful to do this for every dependency instead of specifying a failOnVersionConflict resolution strategy once for the entire multi-project.

Strict dependencies would typically be used by platform providers. When you run on, say, Glassfish 3.4, then you know exactly what versions of a library are provided, and it’s a non-sense to upgrade them. But as I explained they can also be used to discover future conflicts. Our goal is to model the wide variety of reasons why you specify a version in a build file, and what is actually the meaning of it.

if one is at 1.11.0 and the other at 1.9.0 then all bets are off. I want to specify this in a single place (not at each sub-project that includes one of those dependencies directly or transitively), and for all configurations (not just implementation). This would be similar in concept to a strict Maven BOM, an npm package lockfile, a ruby bundler lockfile, or an Amazon version set.

We have different solutions to this. First of all, this is not implemented yet, but we will support alignment out of the box. Alignment is a constraint which basically says that all modules belonging to a specific set must have the same version. So it allows upgrades, as long as all modules in the set have the same version.

Gradle 4.8 will also ship with dependency locking, but it serves a different purpose, which is mainly to make sure that you have reproducible builds when using dynamic versions (version ranges, typically).

Thanks, @CedricChampeau – that makes sense.

I’m looking forward to alignment. Do you think that will land in 4.8 as well?

The other form of alignment I was hoping for is across subprojects. For example, I might have a subproject :common:auth for authentication code, :common:concurrent for thread pools, and :foo:service for a service using both :common:auth and :common:concurrent. Now suppose they all depend on com.google.guava:guava, either directly or through transitive dependencies. If any of them use certain APIs from Guava version 20, then we have to make sure we use that version for everyone at compile time, test compile time, test runtime, and runtime (because v19 and v21 are both incompatible in different ways).

So I end up with a file buildSrc/src/main/kotlin/thirdParty/thirdParty.kt that looks like

val guavaVersion = "20.0"
val guava = "com.google.guava:guava:$guavaVersion"

and build.gradle.kts files in a subset of those projects with some variation of

dependencies {
  compile(thirdParty.guava)
}

where compile might actually be implementation, api, testCompile, runtimeClasspath, etc.

If all subprojects mention the thirdParty.guava dependency explicitly, then we know that’s the version everyone’s getting. If they don’t, we might still have a project compiling against a wrong version. So we alter that thirdParty.kt file to say:

val allDeps = mutableListOf<String>()

fun dep(s: String): String {
  allDeps.add(s)
  return s
}

val guavaVersion = "20.0"
val guava = dep("com.google.guava:guava:$guavaVersion")

// other versioned third-party definitions also using the dep function...

and add this to the root project build.gradle.kts:

// Don't manipulate versions for gradle internals: https://github.com/typesafehub/zinc/issues/87
val blacklistConfigs = setOf("zinc")

allprojects {
  configurations.all {
    if (!blacklistConfigs.contains(name)) {
        for (thirdPartyDep in thirdParty.allDeps) {
          force(thirdPartyDep)
        }
      }
    }
  }
}

So far that’s working, but I wonder if there’s a way to say in a central place “if any subproject uses Guava, directly or indirectly, in any configuration other than internal Gradle plugins, it’s version 20.0” in a simpler way.

I’m looking forward to alignment. Do you think that will land in 4.8 as well?

Technically speaking you can already do it, thanks to dependency constraints. What we will likely provide later is a convenience to define a module set, but it’s unclear if and when we’re going to do it. Which leads me to the 2d part of your question. I think what you need is precisely dependency constraints, which you can already use.

Basically, you can write:

dependencies {
    constraints {
        implementation("com.google.guava:guava") {
            version {
                prefer(guavaVersion) // if you want to allow upgrades
                strictly(guavaVersion) // if you want to fail the build in case of conflict
            }
        }
    }
}

Constraints are applied transitively, you don’t need direct dependencies for them to be used.

Does implementation in that example mean the constraint is only applied to the implementation configuration or can we apply it to all configurations (other than Gradle-internal ones like zinc)?

Do the constraints need to be defined per project (within an allprojects block) or just once in the root?

Should dependencies on that module be expressed by subprojects as the string "com.google.guava:guava" without a version?

Finally, what’s the difference between a constraint like this one and force?

Does implementation in that example mean the constraint is only applied to the implementation configuration or can we apply it to all configurations (other than Gradle-internal ones like zinc)?

It applies to specific configurations. So say you resolve compileClasspath which extends from implementation, the constraint will apply. But if you resolve another independent configuration, it wouldn’t.

Do the constraints need to be defined per project (within an allprojects block) or just once in the root?

Per project, and they will be published in Gradle metadata, meaning that they would be applied transitively. Consider them as any other dependency, but not a “hard one”: they only constraint what is actually found in the graph.

Should dependencies on that module be expressed by subprojects as the string “com.google.guava:guava” without a version?

They can, but it’s not mandatory. A specific project may want a different version, in which case conflict resolution would kick in like usual.

Finally, what’s the difference between a constraint like this one and force?

Constraints are not dependencies. Force imposes that you have a first level dependency, and is not enforced transitively once published.

Thank you, @CedricChampeau – this is very helpful.

The idea of strict locking used together with component metadata rules to express my intent to use particular version of dependencies looks very interesting. My use case is build for MapReduce jobs. There is a set of dependencies starting with org.apache.hadoop:hadoop-* that transitively brings various commons libs, guava, logging, zookeeper, and more to my projects. I can get the real set of those dependencies when I run hadoop classpath on a cluster and want to tweak my Gradle build to use them.
So far so good but I am afraid that component metadata rules affecting every configuration are risk for my build. It means configuration like checkstyle, whole group of Scala related configurations (zinc, incrementalScala*), handfull of Python related configurations from PyGradle plugin can be affected.
Does it mean that I should go with strict version + force?