Any advice/tools to determine why a dependency is _not_ present?

I’m familiar with basic dependency debugging using the dependencies and dependencyInsight tasks. Those tools are very good for finding out how a dependency makes its way into a project, why a specific version was selected, etc.

However I’m in a situation now where runtimeClasspath is missing a dependency that I expect to be present. With the tools I know, there doesn’t seem to be any way to distinguish between a dependency that was never part of a configuration vs. one that was removed by some dependency resolution mechanism (exclusions, component metatdata rules etc.).

I managed to reproduce something like the issue I’m seeing in a sample project. See explanation below. My question has 2 parts:

  1. Can someone help me figure out why protobuf-java is omitted from runtimeClasspath in this sample project?
  2. Are there any good tools for discovering if/how any dependencies were removed from a given configuration? While the problem is pretty noticeable in the small sample project, in my real project which is much larger it took a very long time to find the source of the removal (and even now I don’t understand why it was removed).

Here is the build.gradle for the sample project:

plugins {
  id 'java-library'
}

repositories {
  mavenCentral()
}

dependencies {
  implementation 'mysql:mysql-connector-java'
  // uncomment below to make protobuf-java vanish
  // implementation 'com.webauthn4j:webauthn4j-device-check'

  constraints {
    api 'mysql:mysql-connector-java:8.0.21!!'
    api 'com.webauthn4j:webauthn4j-device-check:0.17.2.RELEASE!!'
  }
}

Normally, mysql-connector-java transitively brings in protobuf-java, which can be seen by printing the dependency tree:

$ ./gradlew dependencies --configuration='runtimeClasspath'

> Task :dependencies

------------------------------------------------------------
Root project 'mysql-connector-test'
------------------------------------------------------------

runtimeClasspath - Runtime classpath of source set 'main'.
+--- mysql:mysql-connector-java -> 8.0.21
|    \--- com.google.protobuf:protobuf-java:3.11.4
\--- mysql:mysql-connector-java:{strictly 8.0.21} -> 8.0.21 (c)

(c) - dependency constraint
A web-based, searchable dependency report is available by adding the --scan option.

BUILD SUCCESSFUL in 519ms
1 actionable task: 1 executed

However, when I uncomment the dependency on webauthn4j, protobuf-java mysteriously gets removed. Here is the same task being run with the line uncommented:

$ ./gradlew dependencies --configuration='runtimeClasspath'

> Task :dependencies

------------------------------------------------------------
Root project 'mysql-connector-test'
------------------------------------------------------------

runtimeClasspath - Runtime classpath of source set 'main'.
+--- mysql:mysql-connector-java -> 8.0.21
+--- com.webauthn4j:webauthn4j-device-check -> 0.17.2.RELEASE
|    +--- org.springframework.boot:spring-boot-dependencies:2.5.5
|    |    +--- com.fasterxml.jackson.core:jackson-databind:2.12.5 (c)
|    |    +--- com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:2.12.5 (c)
|    |    +--- mysql:mysql-connector-java:8.0.26 -> 8.0.21 (c)
|    |    +--- org.slf4j:slf4j-api:1.7.32 (c)
|    |    +--- com.fasterxml.jackson.core:jackson-annotations:2.12.5 (c)
|    |    \--- com.fasterxml.jackson.core:jackson-core:2.12.5 (c)
|    +--- org.slf4j:slf4j-api:1.7.32
|    +--- org.apache.kerby:kerby-asn1:2.0.1
|    +--- com.fasterxml.jackson.core:jackson-databind:2.12.5
|    |    +--- com.fasterxml.jackson.core:jackson-annotations:2.12.5
|    |    |    \--- com.fasterxml.jackson:jackson-bom:2.12.5
|    |    |         +--- com.fasterxml.jackson.core:jackson-annotations:2.12.5 (c)
|    |    |         +--- com.fasterxml.jackson.core:jackson-core:2.12.5 (c)
|    |    |         +--- com.fasterxml.jackson.core:jackson-databind:2.12.5 (c)
|    |    |         \--- com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:2.12.5 (c)
|    |    +--- com.fasterxml.jackson.core:jackson-core:2.12.5
|    |    |    \--- com.fasterxml.jackson:jackson-bom:2.12.5 (*)
|    |    \--- com.fasterxml.jackson:jackson-bom:2.12.5 (*)
|    +--- com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:2.12.5
|    |    +--- com.fasterxml.jackson.core:jackson-databind:2.12.5 (*)
|    |    +--- com.fasterxml.jackson.core:jackson-core:2.12.5 (*)
|    |    \--- com.fasterxml.jackson:jackson-bom:2.12.5 (*)
|    +--- org.checkerframework:checker-qual:3.18.0
|    +--- com.webauthn4j:webauthn4j-core:0.17.2.RELEASE
|    |    +--- org.springframework.boot:spring-boot-dependencies:2.5.5 (*)
|    |    +--- org.slf4j:slf4j-api:1.7.32
|    |    +--- org.apache.kerby:kerby-asn1:2.0.1
|    |    +--- com.fasterxml.jackson.core:jackson-databind:2.12.5 (*)
|    |    +--- com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:2.12.5 (*)
|    |    +--- org.checkerframework:checker-qual:3.18.0
|    |    +--- com.webauthn4j:webauthn4j-util:0.17.2.RELEASE
|    |    |    +--- org.springframework.boot:spring-boot-dependencies:2.5.5 (*)
|    |    |    +--- com.fasterxml.jackson.core:jackson-databind:2.12.5 (*)
|    |    |    +--- com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:2.12.5 (*)
|    |    |    +--- org.checkerframework:checker-qual:3.18.0
|    |    |    +--- com.fasterxml.jackson.core:jackson-databind:2.12.5 (c)
|    |    |    +--- com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:2.12.5 (c)
|    |    |    +--- org.apache.kerby:kerby-asn1:2.0.1 (c)
|    |    |    +--- org.slf4j:slf4j-api:1.7.32 (c)
|    |    |    \--- org.checkerframework:checker-qual:3.18.0 (c)
|    |    +--- com.fasterxml.jackson.core:jackson-databind:2.12.5 (c)
|    |    +--- com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:2.12.5 (c)
|    |    +--- org.apache.kerby:kerby-asn1:2.0.1 (c)
|    |    +--- org.slf4j:slf4j-api:1.7.32 (c)
|    |    \--- org.checkerframework:checker-qual:3.18.0 (c)
|    +--- com.fasterxml.jackson.core:jackson-databind:2.12.5 (c)
|    +--- com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:2.12.5 (c)
|    +--- org.apache.kerby:kerby-asn1:2.0.1 (c)
|    +--- org.slf4j:slf4j-api:1.7.32 (c)
|    \--- org.checkerframework:checker-qual:3.18.0 (c)
+--- mysql:mysql-connector-java:{strictly 8.0.21} -> 8.0.21 (c)
\--- com.webauthn4j:webauthn4j-device-check:{strictly 0.17.2.RELEASE} -> 0.17.2.RELEASE (c)

(c) - dependency constraint
(*) - dependencies omitted (listed previously)

A web-based, searchable dependency report is available by adding the --scan option.

BUILD SUCCESSFUL in 567ms
1 actionable task: 1 executed

Further, it’s only removed from the runtimeClasspath. compileClasspath contains protobuf-java as expected:

$ ./gradlew dependencies --configuration='compileClasspath'

> Task :dependencies

------------------------------------------------------------
Root project 'mysql-connector-test'
------------------------------------------------------------

compileClasspath - Compile classpath for source set 'main'.
+--- mysql:mysql-connector-java -> 8.0.21
|    \--- com.google.protobuf:protobuf-java:3.11.4
+--- com.webauthn4j:webauthn4j-device-check -> 0.17.2.RELEASE
|    \--- com.webauthn4j:webauthn4j-core:0.17.2.RELEASE
|         \--- com.webauthn4j:webauthn4j-util:0.17.2.RELEASE
+--- mysql:mysql-connector-java:{strictly 8.0.21} -> 8.0.21 (c)
\--- com.webauthn4j:webauthn4j-device-check:{strictly 0.17.2.RELEASE} -> 0.17.2.RELEASE (c)

(c) - dependency constraint
A web-based, searchable dependency report is available by adding the --scan option.

BUILD SUCCESSFUL in 495ms
1 actionable task: 1 executed

Thanks in advance for the help!

I forgot to mention that this is with the gradle wrapper using Gradle 7.4. Here is the content of gradle-wrapper.properties:

distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
distributionUrl=https\://services.gradle.org/distributions/gradle-7.4-bin.zip
zipStoreBase=GRADLE_USER_HOME
zipStorePath=wrapper/dists

Actually you used the right tool already, just didn’t look further in the right direction.

If you look closer at the dependencies output, you see that com.webauthn4j:webauthn4j-device-check depends on org.springframework.boot:spring-boot-dependencies:2.5.5 which introduces a constraint on mysql:mysql-connector-java:8.0.26 -> 8.0.21 (c).
If you then look at the spring-boot-dependencies BOM, you see the details of the constraint:

      <dependency>
        <groupId>mysql</groupId>
        <artifactId>mysql-connector-java</artifactId>
        <version>${mysql.version}</version>
        <exclusions>
          <exclusion>
            <groupId>com.google.protobuf</groupId>
            <artifactId>protobuf-java</artifactId>
          </exclusion>
        </exclusions>
      </dependency>

Thanks for the response! That explains why protobuf is excluded from the webauthn4j subtree, but I still don’t understand why it’s excluded from my project’s direct dependency on mysql-connector. My understanding is that Gradle should be taking the whole dependency graph into account. Here’s a quote from the Gradle docs: Downgrading versions and excluding dependencies

Gradle’s exclude handling is, in contrast to Maven, taking the whole dependency graph into account. So if there are multiple dependencies on a library, excludes are only exercised if all dependencies agree on them. For example, if we add opencsv as another dependency to our project above, which also depends on commons-beanutils , commons-collection is no longer excluded as opencsv itself does not exclude it.

I’m not familiar with BOMs (as opposed to POMs), do they work differently in this regard?

In any case, this seems like an unsafe way for webauthn4j to alter the dependency graph: other parts of my project may very well use mysql-connector features that exercise protobuf even if webauthn4j doesn’t. Is this just a problem with the way that webauthn4j was authored?

The section you quote only talks about dependencies.
So if you have

compileClasspath - Compile classpath for source set 'main'.
+--- mysql:mysql-connector-java -> 8.0.21
|    \--- com.google.protobuf:protobuf-java:3.11.4
+--- com.webauthn4j:webauthn4j-device-check -> 0.17.2.RELEASE
     \--- mysql:mysql-connector-java -> 8.0.21
          \--- exclude com.google.protobuf:protobuf-java:3.11.4

you will have protobuf as the upper dependency does not exclude it.

But what you have there is a constraint.
A constraint that dictates a version lets you declare the dependency without a version like you also did manually.
A constraint that excludes a transitive dependency will exclude it where you include the actual dependency, so it is like if you excluded it manually where including mysql connector.
A POM declares dependencies.
A BOM declares constraints.

Maybe the webauthn4j-device-check should not publish its dependency on the Spring Boot BOM but instead publish resolved versions or similar if it is not exclusively to be used in downstream projects that also use the Spring Boot BOM anyway.

Thanks that explains it very well. I’m actually migrating from a Maven project and this was one of the differences. I’m assuming webauthn4j was authored with Maven’s dependency management in mind. I guess the takaways from this are:

  1. Be wary of libraries that expose a BOM dependency with excludes
  2. Pay attention to the ‘(c)’ in the dependencies output as it may be doing more than the version change that is shown.