What is the 'proper' way to make dependencies consistent in a multi-project setup?

Help/Discuss
1

We’ve lived, somehow, without addressing this directly but it has now become a problem that requires Gradle expertise beyond my own. I recognize that out project setup is NOT ideal but I don’t know how to change it.

I searched online and found some mentions of similar issues, some answered briefly, others not at all, such as:

https://discuss.gradle.org/t/how-can-i-get-gradle-dependency-order-to-be-consistent-in-intellijs-modules/3011
https://discuss.gradle.org/t/using-one-the-latest-version-of-the-dependency-across-all-subprojects/4213

Our current setup is based on what I read about somewhere long time ago and feared a bit, but we lived… so far. We have one *.gradle file that declares our third party dependencies as essentially string forms of dependenices or arrays of those. It is essentially a file of constants. we declare actual project dependencies from those projects’ build.gradle files by referencing those constants.

Since there is some inevitable version disagreement between our transitive dependencies we do rely on Gradle “deciding” to use the newest of the versions. In some cases we had to force the versions but this, we realize, is unmaintainable. However, as different (sub)projects have different dependencies, they see those conflicts resolved differently and we end up with one project using one version of some jar and another project uses another version of the same jar.

This causes problems when we try to package up our product as we end up with all jar versions competing for a spot in the resulting distribution bundle, unless we do something about these. Problem is also in Eclipse, where we try to use consistent versions and have attempted to have any one jar be a part of as few classpaths as possible - i.e. if it is inherited from a dependency project, it should not be re-declared as a dependency. Problem is that they may differ by version only. This causes problems if the classpath order happens to be the wrong one (older version before the newer version) and affects product startup that involves classpath scanning for various reasons.

I thought of creating a project that will be there ONLY to declare all dependencies. It would essentially replace our constants gradle file I mentioned above, having separate configuration for each and every dependency that any of the ‘real’ projects need. Real projects would then only be allowed to declare dependencies onto these configurations of this special ‘third party dependencies holder project’.

However, I am not sure that is the correct way either or that it would work. For one, it forces further centralization of all dependencies, instead of encapsulating them within the projects that need them. Perhaps more problematically, I am not sure this would solve anything depending on when the transitive dependencies are resolved.

We are presently still on gradle 2.4 but would upgrade if there is some help in newer versions.

Please help!

2

Actually, this question extends to single-project builds as well. I just created a simple test:

configurations {
  one {
    transitive = false;
  }
  
  two {
    transitive = false;
  }
  
  everything {
    extendsFrom(one, two)
  }
}

dependencies {
  one 'org.slf4j:slf4j-api:1.7.2'
  two 'org.slf4j:slf4j-api:1.7.9'
}

The output of asking for dependencies on this one is:

:dependencies

------------------------------------------------------------
Root project
------------------------------------------------------------

everything
+--- org.slf4j:slf4j-api:1.7.2 -> 1.7.9
\--- org.slf4j:slf4j-api:1.7.9

one
\--- org.slf4j:slf4j-api:1.7.2

two
\--- org.slf4j:slf4j-api:1.7.9

BUILD SUCCESSFUL

Total time: 2.44 secs

I specified two different versions and disabled transtive dependencies to simplify the example only. Read on. While ‘everything’ configuration is resolved to the newer version, ‘one’ keeps pointing to the older one. This seems to indicate, to have consistency without forcing versions everywhere, that we either do something like this:

ext {
  slf4j = 'org.slf4j:slf4j-api:1.7.9'
}

configurations {
  one {
    transitive = false;
  }
  
  two {
    transitive = false;
  }
  
  everything {
    extendsFrom(one, two)
  }
}

dependencies {
  one slf4j
  two slf4j
}

Or every singular dependency must have its own configuration, i.e. as follows:

configurations {
  slf4j {
    transitive = false;
  }
  
  one {
    transitive = false;
    extendsFrom(slf4j)
  }
  
  two {
    transitive = false;
    extendsFrom(slf4j)
  }
  
  everything {
    extendsFrom(one, two)
  }
}

dependencies {
  slf4j 'org.slf4j:slf4j-api:1.7.9'
}

However, neither solution addresses inconsinstency brought by transitive dependencies (changed one and two to transitive =true), e.g:

configurations {
  one {
    transitive = true;
  }
  
  two {
    transitive = true;
  }
  
  everything {
    extendsFrom(one, two)
  }
}

dependencies {
  one 'org.scala-lang.modules:scala-xml_2.11:1.0.3'
  two 'org.scala-lang:scala-reflect:2.11.6'
}

Results in (note the transitive scala-library dependency):

:dependencies

------------------------------------------------------------
Root project
------------------------------------------------------------

everything
+--- org.scala-lang.modules:scala-xml_2.11:1.0.3
|    \--- org.scala-lang:scala-library:2.11.4 -> 2.11.6
\--- org.scala-lang:scala-reflect:2.11.6
     \--- org.scala-lang:scala-library:2.11.6

one
\--- org.scala-lang.modules:scala-xml_2.11:1.0.3
     \--- org.scala-lang:scala-library:2.11.4

two
\--- org.scala-lang:scala-reflect:2.11.6
     \--- org.scala-lang:scala-library:2.11.6

BUILD SUCCESSFUL

Total time: 2.893 secs

The method with constants does not change anything here as we aren’t directly aware of the transitive dependencies. We would need to be aware of each and every transitivedependency in every configuration and keep monitoring for new ones appearing at any time due to version changes who-knows-where.

So, how do we force version consistency across multiple configurations in a single project AND across multiple projects?

3

Version resolution seems to be restricted to be working within a single configuration, separately for each configuration. The only way consistent, yet automatic, resolution seems to be by putting ALL dependencies in a single configuration of a single project that all other projects depend on. All these other project then must have a way to ‘extract’ relevant parts of that single configuration some way. Is there anything that does this already?

Aside from the above, manually specified, forced versions are required. And the only way to make sure none are missed when they occur is to use failOnVersionConflict().

Any help, from anyone?

4

If you already have a file that lists all dependencies and the versions you want, why don’t you use a ResolutionStrategy that forces those versions. You can even make it fail if someone tries to use a dependency that you haven’t listed.

I don’t fully understand the problem though. The Gradle eclipse plugin will do proper transitive dependency resolution since Gradle 2.5, so you won’t get any duplicates.

When you package your application, the packaged dependencies should come from a configuration. That configuration would also give you conflict resolution, so you should never end up with multiple versions of the same artifact. That sounds like something is broken in how you build your distribution.

5

I know about and described forcing. We have many dependencies, many of them transient and lots of people working on many projects. Versions are changing all the time. Requiring everyone to keep making adjustments to forced versions is something we tried but are now trying to avoid.

The problem is not specific to Eclipse, although it is comforting to know that SOME duplication will be removed (Eclipse will continue doing its bit of this whatever Gradle does).

When we package dependencies they do come from a configuration. But there are multiple packs (this is a complex project), requiring (multiple) separate configurations. Each can get its own version resolution and diverge. These bits interact at runtime, where an effective classpath is composed from all these packs, depending on runtime conditions.

6

Let me illustrate a bit more. Our projects are deployable modules. They depend on one another. Not all need to be deployed (dependencies do). Because of this our jars are not in one place - each module brings and houses jars that are their own or are needed in addition to jars already brought by modules they depend on. If a dependency already brings a logger, for example, then the dependent module does not bring the same logger again.

Say there are only two projects, A and B.

  • Project A depends on “a:a:1”
  • Project B depends on project A, “a:a:2” (different version) and “b:b:1”

Obviously they both depend on “a:a”. Ideally we would want both to depend on “a:a:2”. Trouble is, project A will bring “a:a:1”. Project B will bring “a:a:2”. These may or not look like the same thing. When we bundle the jars we actually subtract project A’s configuration from project B’s configuration but this fails to result in desired outcome if these two jars are only different in version. Both will end up on classpath.

7

I don’t get this part. A is an upstream project. It cannot know that some client called B uses a different transitive dependency. Also, what if A doesn’t work with a:a:2? I think you meant it the other way round.

You could ignore the versions and filter by artifact name only. A better solution would be to have a configuration for upstream modules and then always picking the versions from that configuration while resolving your own transitive dependencies.

8

It won’t because Gradle does not reexport any dependencies. That is since 2.5

9

I did mean what I wrote. Yes, A is an upstream project and, yes, it presently cannot know what downstream project needs. The point is that both are in the same build (i.e. they are not built in independent calls to gradle). I am looking for a way to make sure that they end up using common dependencies. This means that they would all have to, somehow, be linked to a common “dependency resolution scope”. All configurations (from all projects) in a single such scope have to be resolved before any one of them can be ‘materialized’.

That does not always work as names sometimes also change with versions. Forgot where we saw that. Furthermore, it doesn’t solve the cause of the problem but only attempts to fix a symptom and a separate change like this would be needed for each such symptom.

OK, this is something I was thinking of and mentioned it above. However, I don’t know how to do that so that it works. The only way to make consistent dependencies would be to put them ALL in a single configuration. Then I would have to pick and chose from those when making other, partial, configurations. The only thing that comes to mind is to play with exclude:

https://docs.gradle.org/current/javadoc/org/gradle/api/artifacts/ModuleDependency.html#exclude(java.util.Map)

(there is no ‘include’) … but that is absolutely nasty.

If there are two otherwise independent dependency projects bringing and exporting the same jar and both of these projects participate in the launch classpath, Eclipse will have the jar twice. There is nothing Gradle can do about this as the jars are needed in the dependency projects. The only way is to only bring projects without their dependencies and redefine the entire list of all runtime dependencies.

10

Just because two projects are in the same build doesn’t mean that they should all select the same dependency versions. That would basically couple all your projects together, creating one big monolith.

You create an upstream configuration (or whatever you would like to call it). You add the upstream modules to that configuration. Now, in the resolution strategy of the compile configuration, whenever you are resolving a dependency that is present in upstream, you force the version that was selected in upstream. Then, during packaging you are guaranteed that compile - upstream will get rid of all dependencies that are already provided by an upstream module.

Again, Gradle 2.5 sets export=false on all dependencies. Please upgrade :slight_smile:

11

I agree 100%. But you implied something I didn’t mean. Specifically, I didn’t mean that this has to be the case for everyone and all gradle builds out there. I mean this as an available place to put the coordination bits in when this is required. If builds were entirely separate, this would be theoretically impossible - once something is built we cannot go back to it and say “sorry, you have to change this”. However, if multiple projects participate in the same build, there is room for coordination, on time, before they actually produce anything.

Well, no. That is not what we seek or have… or what would work (as we need to separate out modules). Furthermore, there is a hierarchy of projects. A parent project could serve as a dependency resolution coordinator (and specifier) for the subprojects that may (or not) be able to opt in or not in this.

We need to select the newest version demanded, not the one selected in the upstream.

Will do, will do :). That is about Eclipse though. My main problem here is beyond/outside Eclipse.

12

Here’s what I am thinking of, maybe, having to do… Create (or modify our existing) gradle plugin that takes dependency declaration over. I.e. projects would no longer express dependencies the way they normally would in gradle but by invoking constructs of this new plugin.

That plugin would create a dedicated singleton configuration somewhere. Each dependency specified anywhere, in any relevant project would first be added to that joint/monolithic configuration. It would also store what was declared for the project internally - not declare it yet.

It would also expose a separate way of obtaining resolved configuration. This would cause that magic monolithic singleton configuration to be resolved and then “walked” to extract relevant bits from it.

Problem is, this is a TON of work and I am not sure how I can tie this back into grade’s own project configurations and execution stages. Hence I am looking for an existing solution.

For Gradle itself I can see two ways going forward, if there truly isn’t a way to do this today:

a) Be able to specify that configuration needs to be “coordinated” (somewhat similar to extendsFrom) with or by another configuration. All such configurations would need to be jointly resolved before any one is considered resolved on its own (it actually wouldn’t be, it would be just extracting the resolved bits from the coordination one - similar to what I described for the plugin above)

b) Introduce a concept of subconfigurations or component/part configuration sthat are resolved together with the root one. In this case I could convert each module’s configuration into this subconfiguration and have the root be the the coordinator.

(a) and (b) are basically the same thing. It is just that (a) adds consistent resolution outside existing configurations and (b) does this internally.

13

@learner This thread was tl:dr for me but it looks we have similar issues dealing with an very large build with a tangled web of dependencies.

If you’re still dealing with these problems take a look at the Netflix Dependency Recommender plugin. You can configure it to strategy OverrideTransitives to avoid dependency version conflicts, but the question is where do you get a consistent set of depenencies from?

The key to our solution is publishing BuildInfo to Artifactory from jenkins. There are many jenkins pipelines involved, however one special job assembles all of the applications so that it contains every dependency in the system. It always consumes lastest of all the dependencies (which is our case is the head of the integration branch). What it produces is canonical list of all the versions it used during the build which can be used for any other component build in the system. It will of course perform some end to end testing of the servers so that it will only publish a “working” list of dependencies.

No component in the system knows the version number of any of it’s dependencies without this file.

Hope that gives you some ideas!

Edit:
Oh I should have mentioned that we had to abandon the multi-project build. It was a major pain to maintain, extremely slow and many plugins that we needed didn’t support multi-project. New builds are much faster without it because it’s significantly reduced the amount of compiling!