Excludes are tedious for several reasons

jimshowalter · June 11, 2019, 5:19pm

We want to strictly converge our dependencies, so there aren’t any foo v1 -> foo v2 types of entries in the output from ./gradlew dependencies.

This is turning out to be tedious.

Excluding at global scope excludes even root dependencies specifically listed in the dependencies{} section.

Excluding under a root dependency specifically listed in the dependencies{} section doesn’t cascade to the children (https://stackoverflow.com/a/27044481).

Excluding all references in a closure requires making each dependency a root dependency, in some cases way down in the details of the closure (these have no business being exposed). This is also fragile if any of the root dependencies change.

We wind up with something hideous like this:

api('com.fasterxml.jackson.core:jackson-databind:2.9.9') {
    exclude group: 'com.fasterxml.jackson.core', module: 'jackson-annotations'
}
api('com.fasterxml.jackson.core:jackson-annotations:2.9.9') {
    exclude group: 'com.fasterxml.jackson.core', module: 'jackson-databind'
}
api('tv.cntt:slf4s-api_2.12:1.7.25') {
    exclude group: 'org.scala-lang', module: 'scala-library'
    exclude group: 'org.scala-lang', module: 'scala-reflect'
    exclude group: 'org.slf4j', module: 'slf4j-api'
}
api('org.json4s:json4s-jackson_2.12:3.6.6') {
    exclude group: 'org.scala-lang', module: 'scala-library'
    exclude group: 'com.fasterxml.jackson.core', module: 'jackson-databind'
}
api('org.json4s:json4s-core_2.12:3.6.6') {
    exclude group: 'org.scala-lang', module: 'scala-library'
}
api('org.json4s:json4s-ast_2.12:3.6.6') {
    exclude group: 'org.scala-lang', module: 'scala-library'
}
api('org.json4s:json4s-scalap_2.12:3.6.6') {
    exclude group: 'org.scala-lang', module: 'scala-library'
}
api('org.scala-lang.modules:scala-xml_2.12:1.2.0') {
    exclude group: 'org.scala-lang', module: 'scala-library'
}
api group: 'org.slf4j', name: 'slf4j-api', version: '1.7.26'
api('ch.qos.logback:logback-classic:1.2.3') {
    exclude group: 'org.slf4j', module: 'slf4j-api'
}
api('org.apache.commons:commons-configuration2:2.5') {
    exclude group: 'org.apache.commons', module: 'commons-lang3'
}
api('org.apache.commons:commons-text:1.6') {
    exclude group: 'org.apache.commons', module: 'commons-lang3'
}
api group: 'org.apache.commons', name: 'commons-lang3', version: '3.9'
api group: 'org.scala-lang', name: 'scala-library', version: '2.12.8'
api group: 'org.scala-lang', name: 'scala-reflect', version: '2.12.8'

Are we doing something incorrectly?

lingocoder · June 11, 2019, 8:36pm

Would something like this work for you?

...
configurations {
    api {
        exclude group: 'org.scala-lang', module: 'scala-library'
        exclude group: 'com.fasterxml.jackson.core', module: 'jackson-databind'        
        exclude group: 'org.apache.commons', module: 'commons-lang3'        
        exclude group: 'org.slf4j', module: 'slf4j-api'        
        exclude group: 'com.fasterxml.jackson.core', module: 'jackson-annotations'        
        exclude group: 'org.scala-lang', module: 'scala-reflect'        
    }
}
...
dependencies {
    api 'com.fasterxml.jackson.core:jackson-databind:2.9.9' 
    api 'com.fasterxml.jackson.core:jackson-annotations:2.9.9'
    api 'tv.cntt:slf4s-api_2.12:1.7.25'
    api 'org.json4s:json4s-jackson_2.12:3.6.6'
    api 'org.json4s:json4s-core_2.12:3.6.6'
    api 'org.json4s:json4s-ast_2.12:3.6.6'
    api 'org.json4s:json4s-scalap_2.12:3.6.6'
    api 'org.scala-lang.modules:scala-xml_2.12:1.2.0'
    api group: 'org.slf4j', name: 'slf4j-api', version: '1.7.26'
    api 'ch.qos.logback:logback-classic:1.2.3'
    api 'org.apache.commons:commons-configuration2:2.5'
    api 'org.apache.commons:commons-text:1.6'
    api group: 'org.apache.commons', name: 'commons-lang3', version: '3.9'
    api group: 'org.scala-lang', name: 'scala-library', version: '2.12.8'
    api group: 'org.scala-lang', name: 'scala-reflect', version: '2.12.8'
}
...

In that’s not what you’re after, then maybe that together with (or replaced by) dependency constraints could make your script more concise?

Also, would you mind me asking? Why are you assigning all of those things to the api configuration? Do you need to explicitly place absolutely everything there on your consumers’ compilation class path?

Of course, you know the needs of your consumers better than I would. But just as a casual observer, things like slf4j for example? My first thought seeing that was I would not want to force those kinds of things to be on my consumers’ class path if I could help it.

lingocoder · June 12, 2019, 8:36pm

Hey there I’m really curious to learn whatever your solution for this ends up being.

Please share whenever you can? Thanks.

jimshowalter · June 12, 2019, 11:16pm

Thanks! I’ll try that syntax. It’s definitely cleaner.

I don’t mind you asking. It’s just that with > 200 jars on a classpath, chaos reigns if dependencies aren’t locked down. The above is just one of the steps to getting to a shared plugin that contains nothing but dependencies (the Gradle equivalent of a BOM), so all teams see the same build and runtime classpaths, and nobody gets a rude shock at runtime in prod. Also, having the dependencies defined in a central location and locked down makes it easier to perform OWASP and other security audits, check source licenses for compliance, etc.

I worked on a monolith with over a thousand jars of various vintages and many diamond problems, and making changes was so painful it became an obstacle to progress. Don’t want that to happen here.

lingocoder · June 13, 2019, 4:30am

I hear ya I figured you were making the kind of effort you’re making, to achieve something along those lines.

Can I ask you something? With the following as context for my question…

…and this one…

In the context of those guidelines for api vs implementation, I’m curious to learn. What is your take on the above approach?

jimshowalter · June 13, 2019, 3:04pm

This was basically first stab at the problem.

I understand the idea of separating api from implementation, but wasn’t sure what the effect would be if some dependencies were marked implementation (which turns into Maven’s runtime when publishing a POM).

Would projects that depend on this project inherit only the api dependencies? If so, what would happen with their classpaths in turn? Would they wind up having to redo the exclusions in order to re-converge their dependencies? If so, that would defeat the purpose of the shared dependencies specification, and therefore I declared everything api.

Ideally we would be able to define a base set of root (a.k.a. primary) dependencies in the shared specification, along with whatever excludes are necessary to strictly converge, and the excludes would apply from the roots down, so low-level internal details don’t have to be turned into root dependencies just to have something to exclude from, but Gradle seems to lack support for that, which forces exposing things that shouldn’t have to be exposed. Unless we’re missing something.

jimshowalter · June 13, 2019, 8:17pm

By the way, we have this in our build.gradle:

configurations.all {
resolutionStrategy {
failOnVersionConflict()
}
}

It fails the build unless dependencies are strictly converged.

lingocoder · June 13, 2019, 9:07pm

With the caveat that I might myself be included in your „we“ but based on my current understanding:

Yes. I’m almost 100% certain that’s the whole point of the api configuration.

Well, it says in the documentation I linked to, that things assigned to api…

…and for implementation…

…which I’m almost 100% certain means that things a library assigns to the implementation configuration, will be on consumers’ runtime class path.

Again, the docs say „Dependencies found in the implementation configuration will…not be exposed to consumers…“. So I interpret that to mean that there wouldn’t be anything to exclude.

Are you getting a different interpretation of that particular part of the docs?

That’s one of the somethings that I’m missing. Sorry. I’m not familiar with the usage of converge/re-converge in this context.

Can you reword what you mean by it? But in a very simple ELI₅ way for me? Please?

That might be another one of the somethings that I might be missing.

I think I understand what your purpose is. But, of course, I could also have misinterpreted everything you’ve shared so far. I’ve been known to do that sometimes

So to make sure that I’m not making any incorrect assumptions, can you ELI₅ your purpose again for me? Please? Thanks.

A more honed-in explanation might encourage others with more expertise than me, who might be lurking in this post, to chime in.

lingocoder · June 14, 2019, 1:27am

I see this a lot! In project after project. And I always wonder to myself, „Is that really necessary?“

So, I ran a little experiment. Where, first, I declared everything api. And then I declared mostly everything implementation. I made some, hopefully useful, observations.

I encourage you to reproduce my experiment with the following steps:

Using gradle init, create a simple java-library project
Copy to the generated build script, the configurations{} and the dependencies{} blocks from my comment above
Using the Maven Publish Plugin, publish the library to your local Maven repository
- set the group and version properties of the build script to something appropriate
Using gradle init, create a simple java-application project; this app will be the consumer for your library
Add the module coordinates (group:name:version) of the library created in 1 to the implementation configuration of the dependencies{} block in the consumer’s build script
- the consumer’s repositories{} block should include mavenLocal(); or whatever local repo you published the library to
From the root directory of the consumer project, run gradle dependencies > {some_file_1.txt}; you’ll compare these to the one below
Back in the library project, except for only one of the dependencies (your choice), change all of the others to be assigned the implementation configuration.
- that’ll leave you with only one dependency with api preceding it
- edit the configurations{} block accordingly
Repeat step 3, but this time change the version property to something different than the first published module
Back in the consumer project, edit the version part of the implementation dependency to the newly-published module from setp 8
Repeat step 6, but write the output to a file with a different name.
- compare the two dependency trees of the two different library modules

What I observed (and you can independently verify) is that:

When there was only one api dependency, the consumer’s compilation classpath was w-a-a-a-y smaller than the one where everything was assigned to api
- the one dependency that I made api was api 'org.apache.commons:commons-lang3:3.9' (chosen arbitrarily)
The consumer’s runtime classpath was more or less the same for both api-heavy and api-light versions of the library
In both cases, none of the consumer’s classpaths (compilation, runtime, etc) ever showed any „foo v1 → foo v2 types of entries in the output from ./gradlew dependencies“
- I’m assuming that you consider this to mean that the dependencies in both cases are, therefore, „converged“
Calling gradle run on the consumer project succeeded using both api-heavy and api-light versions of the library.
- the only difference I observed was that for the api-heavy run, Gradle displayed all the artifacts’ resolution/downloading status; which it didn’t display on the api-light run.
The outcomes I observed are exactly what the Gradle docs claimed they would be

If you’re interested in seeing the dependency trees produced in my experiment, I’ve shared them here. Please let me know if that download doesn’t work for you? Then I will reshare them some other way.

jimshowalter · June 14, 2019, 1:47am

You’ll get no argument from me that if there are fewer things listed as api dependencies, there will be fewer things in the compile classpath, but all that does is reduce compile time marginally.

The real concern here is what jars wind up on the runtime classpath.

You say the consumer’s runtime classpath was more or less the same, which is fine.

But you also say: " In both cases, none of the consumer’s classpaths ( compilation, runtime, etc ) ever showed any „ foo v1 -> foo v2 types of entries in the output from ./gradlew dependencies “, and I can show examples (not here, I have to get them from work, and clean them up before posting) where there are waaaaay more -> entries when not strictly converged.

It’s possible you’re not working with a classpath that’s complex enough to manifest the problem known as JAR Hell (or, on Windows, as DLL Hell), but because those terms exist, there must be some reason they exist.

Here’s an experiment you could try. Use Tomcat, Hibernate, Redis, and N other big projects in your experimental project. Then dump ./gradlew dependencies, and look for ->'s. Then try to get past failOnVersionConflict.

lingocoder · June 14, 2019, 3:14am

You’ll get no argument from me about JAR Hell being a thing.

I assumed that you shared the example build script you shared because it results in a sufficiently-complex classpath. One that represents a particular real-world instance of a JAR Hell that you’re experiencing. Or have I incorrectly assumed that?

Based on your example dependencies, I thought of one option you might have available to you (the syntax change). I’m sufficiently convinced that it could mitigate the impact of JAR Hell on the system you described in your original post. I assumed that the particular mitigation you were after, was the absense of „foo v1 → foo v2 types of entries in the output from ./gradlew dependencies“.

What convinced me of that was, working with your representational classpath in a simplified experiment, and having made the proposed edits to the syntax of the build script, I observed that those edits did indeed result in the desired „converged“ dependencies. At least they do in a simplified representation of the problem that uses the same complex classpath you shared.

Could the difference between those examples and the results from my experiment, be that the former probably does not implement the „syntax“ edits that proved to be successful in the latter?

I’ll look into that. But wouldn’t it be simpler for you — in the meantime — to try the proposed build script syntax edit that I talked about above on your already-existing complex system, than it would be for me to recreate such a complex system from scratch?

lingocoder · June 28, 2019, 1:09pm

Hi @jimshowalter,

How are you?

Base on your stackoverflow answer I stumbled across, I presume you’ve also solved the problem you asked about here several days ago.

I was hoping you felt like sharing the excludes solution you settled on with the forum here. Please?

A lot of us here could benefit from whatever you’ve learned from the solution you found.

Thanks in advance.

jimshowalter · June 28, 2019, 2:38pm

The stackoverflow question I posted about slf4j isn’t related to dependency convergence.

I haven’t tried your suggestion because the whole process was becoming way too manual. I’m currently researching what tools are available.

lingocoder · June 28, 2019, 4:42pm

Do you mean tools to manage dependency convergence? As in dependency management tools?