Efficient way to find the origin of transitive dependencies?

Help/Discuss
1

Excuse me if the question is super-basic. I’m sorta new to Gradle.

I have a simple Gradle/Grails project I am developing for work. We use Artifactory for libraries and it was pretty well populated when I started using it. We are restricted to only use that system for our dependencies. Recently they started to use a security scanning feature that identifies known issues in libraries then blocks them if they have issues.

This is a good thing, but it causes some issues when it blocks dependencies that a transitive several levels down. I find it difficult to identify what is pulling in this dependency when I can’t run a dependency report because it can’t download the transitive dependency I’m trying to find.

I have an old dependency report that I have used to successfully identify some of the top level dependencies so I could make the necessary changes in the build file.

The problem is that sometimes this isn’t enough to identify what is triggering the attempted download of a blocked dependency.

I have used grep on the command line to identify this and sometimes it works:

“grep “[insert artifact ID here]” -R ~/.gradle/“

This gives me a list of poms that have the desired artifact ID. This helps, but sometimes this list can be huge and also contains parent dependencies from other projects.

Is there a way to

  • Run a dependency report despite not being able to download all dependencies.
  • Run some sort of report to identify which top level dependencies are requiring a specified transitive dependency?
  • Some other more efficient way to resolve dependency blocked issues?

Thanks.

2

I slapped this task together that dumps out paths to unresolvable dependencies on the compileClasspath configuration.

Hopefully if you’re using Groovy DSL that you can translate it easy enough.

repositories {
    mavenCentral()
}
dependencies {
    implementation("junit:junit:4.+")
    implementation("com.nothing:dne:1") // a first-level unresolvable
}
configurations.compileClasspath {
    resolutionStrategy {
        dependencySubstitution {
            // simulate a transitive unresolvable
            substitute(module("org.hamcrest:hamcrest-core")).using(module("com.nothing:dne:1"))
        }
    }
}
tasks.register("unresolvable") {
    doLast {
        configurations.compileClasspath.get().incoming.resolutionResult.allDependencies {
            if (this is UnresolvedDependencyResult) {
                val a = attempted
                if (a is ModuleComponentSelector) {
                    // Could filter
                    //if (a.group == "com.nothing" && a.module == "dne") {
                    logger.quiet(a.toString())
                    val toProcess = ArrayList<Pair<Int, ResolvedComponentResult>>()
                    toProcess.add(Pair(1, from))
                    while (toProcess.isNotEmpty()) {
                        val curr = toProcess.pop()
                        logger.quiet("{}{}", "    ".repeat(curr.first), curr.second)
                        toProcess.addAll(curr.second.dependents.map { Pair(curr.first + 1, it.from) })
                    }
                    //}
                }
            }
        }
    }
}

Results:

> ./gradlew -q unresolvable
com.nothing:dne:1
    junit:junit:4.13.2
        project :
com.nothing:dne:1
    project :
3

Actually, both dependencies task and dependencyInsight task should work fine with dependencies failing to resolve.
Also with the example @Chris_Dore made the tasks are working perfectly fine and do not need an additional custom task.

If any of them does not work, you might need to provide an MCVE to reproduce the problem.
Maybe it depends on how the Artifactory blocks those dependencies.
At least if it delivers a 404 response it should work fine.
It could be problematic if it returns a different error but in that case the custom task of Chris will most probably also not work as intended.

4

I wrote the task because dependencyInsight wasn’t helping:

Executing 'dependencyInsight --configuration compileClasspath --dependency com.nothing:dne'...
> Task :dependencyInsight
com.nothing:dne:1 FAILED
   Failures:
      - Could not find com.nothing:dne:1.
        Searched in the following locations:
          - https://repo.maven.apache.org/maven2/com/nothing/dne/1/dne-1.pom
        If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration.

com.nothing:dne:1 FAILED
\--- compileClasspath

Removing the first-level dependency gives even less info:

Executing 'dependencyInsight --configuration compileClasspath --dependency com.nothing:dne'...
> Task :dependencyInsight
No dependencies matching given input were found in configuration ':compileClasspath'
5

The dependencies task output and then searching for FAILED is comparable to your task, isn’t it?

compileClasspath - Compile classpath for source set 'main'.
+--- junit:junit:4.+ -> 4.13.2
|    \--- org.hamcrest:hamcrest-core:1.3 -> com.nothing:dne:1 FAILED
\--- com.nothing:dne:1 FAILED

That the dependencyInsight task does not help without the first-level dependency is probably just an artifact of the dependency substitution or could be considered a bug. If you call dependencyInsight with org.hamcrest:hamcrest-core it shows the expected output.

But anyway, my main point was, that the tasks work fine and as expected if the repository delivers 404 responses and that with responses that disturb those tasks, your custom task might be affected as well. But this is just a guess and hard to verify without being able to reproduce OPs situation. :slight_smile:

1 Like