Gather transitive dependencies and associated JAR files programmaticaly

Help/Discuss
1

Hi there,

We do Eclipse development and currently, all our third-party modules are stored in X projects inside the Git. We have already converted all non-Eclipse project to Gradle so for those, third-party projects are not required to be in source control.

As a first step of conversion for the rest of the projects, we would like to reduce the X projects into a single one, let’s call it vendors. This vendors project will contain all third-party dependencies JAR files we need.

Of course, we can build this by hand but I would like to automate this a bit by using Gradle knowledge. What I envisioned to accomplish that is the following.

I would create a vendors.gradle script for the project. This script would depend on all other projects. Then, I would create a single task, that would “trigger” dependencies resolution (transitively) and then, I could query Gradle API to retrieve the list of dependencies and the location of the JAR files. Finally, I would simply copy those JAR to the right location and commit this (or not, not sure yet), into the vendors project.

Currently, I have this build script:

configurations {
    vendors {
      transitive = true
    }
}

dependencies {
    vendors project(path: ':api-services', configuration: 'runtime')
    vendors project(path: ':api-services', configuration: 'testRuntime')
}

task gatherDependencies << {
    configurations.vendors.allDependencies.each { dep -> println dep.name }
}

But this only lists api-services as a dependencies and nothing else. I’m not sure how can I query Gradle API to get the full dependency tree and this is what I would like to know. What would be correct APIs to call to achieve what I’m looking. Only pointers in the right directions are fine.

You can take a look of the whole thing on my test repository in the branch single-vendor-automated.

Regards,
Matt

2

To get transitive dependencies you have to resolve the configuration. Simply iterating over the dependency set will only give you the direct dependencies. You can access the resolved graph via configurations.vendors.incoming.resolutionResult.

https://docs.gradle.org/current/javadoc/org/gradle/api/artifacts/ResolvableDependencies.html#getResolutionResult()

3

Hi Mark,

Excellent, exactly what I needed. Thank you for the answer.

Will reply if I have further questions about API but all seems good right now.

Regards,
Matt

4

So, I was able to retrieve the list of artifacts files of the configuration. However, I had to ditch configurations.vendors.incoming.resolutionResult as I was not able to find a good way to retrieve file associated with a DependencyResult. Instead, I used for now configurations.vendors.resolvedArtifacts with a filter on artifact’s group to exclude those generated by project. Anyway, that’s another story.

Now, I would like to also retrieve the associated sources artifact. Do you know how I could do this? I guess I need to call some repository API to retrieve it (hopefully, a cached copy).

As a side note, the old forums and old mailing lists (gradle-user) already contained such information, so sorry for the post, I should have searched more harder. Here some links for posterity:

Regards,
Matt

5

So I was able to do this using dependencies.createArtifactResolutionQuery() to make a query to retrieve SourcesArtifact. Final code is (tested with Gradle 2.2):

def collectModuleComponents = {
  def dependencies = configurations.vendors.incoming.resolutionResult.allDependencies
  def moduleDependencies = dependencies.findAll {
    it instanceof ResolvedDependencyResult && it.requested instanceof ModuleComponentSelector
  }

  moduleDependencies*.selected.toSet()
}

def collectBinaryArtifacts = {
  def moduleComponents = collectModuleComponents()

  configurations.vendors.resolvedConfiguration.resolvedArtifacts.findAll { resolvedArtifact ->
    moduleComponents.any {
      resolvedArtifact.moduleVersion.id.equals(it.moduleVersion)
    }
  }
}

def collectSourceArtifacts = {
  def moduleComponents = collectModuleComponents()
  def queryResult = dependencies.createArtifactResolutionQuery()
                                .forComponents(moduleComponents*.id)
                                .withArtifacts(JvmLibrary, SourcesArtifact)
                                .execute()

  queryResult.resolvedComponents.collectMany {
    it.getArtifacts(SourcesArtifact)
  }.toSet()
}

task copyBinaryArtifacts(type: Copy) {
    from(collectBinaryArtifacts()*.getFile())
    into('lib')
}

task copySourceArtifacts(type: Copy) {
    from(collectSourceArtifacts()*.getFile())
    into('lib')
}

task copyArtifacts(dependsOn: ['copyBinaryArtifacts', 'copySourceArtifacts'])