Managing dependencies in a big project

Help/Discuss
1

I am modernizing a Gradle 2.1 project to Gradle 4.10 in preparation for migrating to 5.x when it is available in the company.

As part of that, one of the problems the developers face is to manage dependencies in a consistent and visible way. Currently they have a map int hr rootProject with dependencies for various artifacts, but it is being used inconsistently and many dependencies have explicit versions declared at the reference site.

What we want is to have the dependencies declared without versions and have all the versions declared in an easy to understand central location. I was thinking of using the Nebula dependency recommender, but then I read somewhere that Gradle now satisfies this use case with its BOM support.

So I wrote BOMs, in fact 3 of them, for different layers of the application (the APIs need to use older versions of certain libraries, etc). The BOMs cannot be published for administrative reasons, so I checked them in under rootProject.file('gradle/bom/')

So far so good, but when I tried to add the BOMs as file(...) dependency, they didn’t do anything (because they were added to the classpath, not interpreted as metadata).

My next attempt was to lay them out as a file:...Maven repo and add them as artifacts. This is slightly better - I can see the BOMs present in every configuration, but no versions are resolved. I looked through the debug logs, but could not find details of why the BOMs were not used for the dependency resolution.

By BOM I mean a *.pom file, with pom packaging, no dependency section and with properties and dependencyManagement section. The dependencies under dependencyManagement have GAV coordinates and nothing else. Versions often reference properties.

What we really need is to make sure that any versionless dependencies are always listed in the BOM and the version is always what we see in the bom (even if transitive deps will fetch a higher version).

Nice-to-have 1: be able to see which explicit dependencies have versions not coming from the BOM.

Nice-to-have 2: fail the build if a transitive dep requires higher version than the BOM.

I realize that may be too much, so I would like to know which of these requirements can be satisfied with BOMs, for which do I need to use the constraint DSL, resolution strategies (or any other new mechanisms added in the 4.x releases).

Also any tips of why the local BOM didn’t work or where is the code that adds the constraints for BOM-ed versions would be welcome.

2

Fwiw, I’ve played with several approaches.

First, before dependency constraints was a thing, I’ve been using resolutionStrategy for force(…) certain versions of discrete dependencies, and using some scripting with eachDependency to select versions for entire groups of dependencies (based on groupId, groupId prefix, or groupId+artifactId prefix). I can share some snippet if want (I’ve already shared some time ago but don’t remember where; possibly here, or maybe on the issue tracker).

Now, with dependency constraints, I apply the base module to the root project (assuming a project that doesn’t contain any code here), add my dependency constraints there, then loop over all subprojects to have them extend (depend on) the root project’s default configuration to inherit the constraints. Lately, I’ve been adding a separate configuration for the annotation processors.

It goes like this (using Gradle 4.x):

plugins {
  base
}
dependencies {
  default("com.google.inject:guice-bom:4.2.0")
  default("org.jooq:jooq-parent:3.10.5")
  default("org.eclipse.jetty:jetty-bom:9.4.8.v20171121")
  default("org.jboss.resteasy:resteasy-bom:3.5.0.Final")
  default("com.fasterxml.jackson:jackson-bom:2.9.5")
  constraints {
    // https://github.com/gradle/kotlin-dsl/issues/710
    add("default", "org.slf4j:slf4j-api:1.7.25")
    add("default", "org.apache.logging.log4j:log4j-slf4j-impl:2.10.0")
  }
}
subprojects {
  plugins.withType<JavaBasePlugin> {
    configure<JavaPluginConvention> {
      sourceSets.all {
        // All other (source set) configurations extend those 4
        arrayOf(
          compileConfigurationName,
          compileOnlyConfigurationName,
          annotationProcessorConfigurationName,
          runtimeOnlyConfigurationName
        ).forEach {
          dependencies.add(it, rootProject)
        }
      }
    }
  }
}

Nice-to-have 1: be able to see which explicit dependencies have versions not coming from the BOM.

You can probably do that using resolutionStrategy’s eachDependency, assuming it triggers before constraints are applied.

Nice-to-have 2: fail the build if a transitive dep requires higher version than the BOM.

Using defining constraints as ad("default", "…notation…") { version { strictly("version") } } will do that. I don’t think you can do that for BOMs (not until Gradle 5 and its enforcedPlatform).

1 Like
3

The answer to this question is going to be focused on answering what you tried, rather than coming up with a different strategy. Note that providing recommendation on such issue is one of the goals of the dependency management team.

  • Gradle 4.6 to 4.10: In order to consume a BOM, you will need to enableFeaturePreview("IMPROVED_POM_SUPPORT") in settings.gradle. And the BOMs must be, as you said they were, POM with pom packaging. Gradle will then import the dependencyManagement entries as constraints. The BOM must be added as a dependency in the configurations that require its input.
    If you add such a dependency, then you should be able to declare other dependencies without version. If a simple example works, you know the wiring is happening properly.
  • Gradle 5.0 and above: See the BOM documentation
  • Upcoming feature(s): The Gradle team is currently working on adding support for publishing platforms, which, once the Gradle metadata format goes out of feature preview, will enable even more options than what you can do with a BOM. For example, you will be able to use rich version declarations, such as strictly mentioned in another answer. With this, you will be able to better tailor your platform by indicating which versions are strict, which have more leniency, potentially using ranges, etc …

The 4.x solution does not offer a solution for the above. In 5.0, this is achieved with enforcedPlatform. With platform support and Gradle metadata, this can be refined further.

This is something you will have to roll on your own, there is no built in support for it, nor is it planned.

You can do that by doing resolutionStrategy.failOnVersionConflict() which effectively fail your build the moment a version conflict needs to be resolved. Note that using a force effectively makes the version conflicts disappear and so hides them. For example, enabling this and using enforcedPlatform will end up with most conflicts hidden.

Given your requirements, I would recommend feature preview of BOM support in 4.x with resolutionStrategy.failOnVersionConflict, migrate to platform in 5.0 and try out the upcoming feature the moment they appear in an RC.

2 Likes
4

BTW, what do you think of the inheritance strategy I outlined? Particularly, I this something you would advise against, or is it fine?

5

Well, your usage of the inheritance word had me troubled. But the snippet you posted is really about adding a dependency to the root project in all subprojects that qualify. I think that approach is fine and shows that having a dedicated plugin for this, the Java platform one, will answer a need.

However your description of using configuration inheritance across project boundaries scared me :slight_smile: so I would not recommend that as there is no guarantee Gradle would keep supporting it, or even does today.

6

But the snippet you posted is really about adding a dependency to the root project in all subprojects that qualify. I think that approach is fine

Yes, it’s technically adding a dependency to the root project, but with the only goal to import its dependency constraints, as the rootProject doesn’t produce any output/artifact.

However your description of using configuration inheritance across project boundaries scared me

It’s no different actually, simply using project(path: ":", configuration: "foobar") rather than rootProject or project(":") when adding the dependency to the root project. (Gradle indeed doesn’t let you add a configuration from another project: dependencies { implementation rootProject.configurations.foobar } or subprojects { dependencies { implementation foobar } }will raise an error)

7

For me, configuration inheritance means configuration.extendsFrom and I initially thought you were trying to do something like:
project(":foo").configurations.implementation.extendsFrom(project(":").configurations.default)

8

Thanks for the background. It turned out my problem was trivial, yet difficult to catch.

As I moved the BOMs to a Maven repo layout, I renamed our synthetic bom to my-app-bom while the XML was still saying <artifactId>bom</artifactId>. Somehow this caused Gradle to not create constraints for this particular POM, but it didn’t log any errors either (didn’t look too much, but the breakpoint didn’t hit).

Does that make sense? Is there any use-case for POM resolved from certain GAV coordinates, but internally having different GAV coordinates? How about making this an error, or a warning?