Best practice for distributing dependencies (how to build a maven repo containing dependencies?)

TL;DR: We want/need to package all libraries used by our build into the final distribution.zip as a repo including metadata (poms). Below is why, and the question is “how?”

We are building an application framework for use by our own consultants in various projects. The application framework has many dependencies (currently 311 jar files across all our components), most of which can be found at various public maven repositories. For the ones that don’t have maven coordinates, we have created private pom-files which fulfill the job, so that gradle can use those to track dependencies.

Our requirements specify that we must ship the dependencies along with the framework. This has both practical reasons (several client sites are not connected to the internet) and legal reasons (contracts and escrow regulations stipulate that the deliverable must include “all necessary libraries and tools”). Also, we have been bitten by public repositories going offline and disappearing, which leads to nothing good.

Traditionally (back from ant-times), we have manually downloaded jar files and stored them in a directory structure in subversion, then used manual dependency management to make sure we got all the proper inclusions. We have, of course, messed this up more than once, and the applications built on this have a tendency to always include all the available jar files, no matter how many were needed.

We can solve the legal requirements by using the mechanism used by Spring to build their “dist-with-deps” version, as shown in https://github.com/SpringSource/spring-framework/blob/master/build.gradle#L948. This lets us collect all the jar-files in a directory, but of course we loose all the transient dependencies, and our projects will have to deal with dependency management manually. We don’t want that.

Instead, what we would like to do is to have our build process build a maven repo with all dependencies, including pom files and signatures, and have that included in the distribution, somewhat like below (directory structure is simplified to save vertical space). :

\--- distribution.zip
      +--- docs
      +--- repo
           +--- com -> fasterxml -> jackson -> core -> jackson-core -> 2.2.2
                +--- jackson-core-2.2.2.pom
                \--- jackson-core-2.2.2.jar
           +--- org -> springframework -> jdbc -> spring-jdbc -> 3.2.3.RELEASE
                +--- spring-jdbc-3.2.3.RELEASE.pom
                \--- spring-jdbc-3.2.3.RELEASE.jar
      \--- samples

With this, our projects would be able to use a single file-based repository that contained everything required by the framework. Should they want other (or newer) stuff, and have the required access, they can easily replace this with mavenCentral() or even a private repository manager.

So, these are the questions:

  • Is this possible to do with Gradle? * Are there any examples out there? * Or, at least, can anyone give me any pointers or hints to useful mechanisms inside gradle to make this happen?

Eirik

Gradle has no built-in feature to create a Maven repository. If you need a Maven repository containing all libraries used by your build, you could maintain such a repository in your binary repository manager (Artifactory, Nexus). You would use this repository to build your software and include it in your distribution. Export and download of the repository should be automatable via the repository manager’s (REST) API.

We were hoping to not go down that route, partly of simplicity, and partly because the repository manager quickly grows to contain many things we don’t want to export (as in if we start with spring 3.2.2 and then switch to 3.2.3 before shipping, the repository manager suddenly contains both, dependencies we used during development but decided to not use in the shipping version, etc)

Given that there is nothing automatic, we might have to build something (hey, flexibility is a desirable quality, right?). A few questions, then:

  • I know that i can access the jar-files (the same way that spring does). Is there a way that my build can access the corresponding pom-files? * Or, is there an api to access the gradle cache for the required files? * Or, would there be code from an IDE-plugin that would help, given that they (the IDE-plugins) download both source, javadoc and jar-files – but what about the pom?

None of this is easily accessible at this time. I think going the repository way is simpler and will lead to better results. You just have to curate the repository’s contents, rather than proxying other repositories. You can also run some acceptance tests on the final distribution.

Yeah, no… I don’t like it. This mechanism suddenly has me doing manual dependency management again, in particular to get rid of old stuff.

Googling and searching and looking around, I found a mechanism used by Andreas G (http://forums.gradle.org/gradle/topics/is_this_the_best_way_to_access_the_pom_file_for_a_dependency#reply_11977659), which I then expanded a bit.

configurations {
        repo
        poms
    }
      // http://forums.gradle.org/gradle/topics/is_this_the_best_way_to_access_the_pom_file_for_a_dependency
    task listAllDependencies << {
        def projects = subprojects.findAll { it.plugins.hasPlugin('java') }
        def projectNames = projects*.name
        def artifacts = projects*.configurations*.runtime*.resolvedConfiguration*.resolvedArtifacts
                                                                                                                                                         .flatten()
                        .findAll { !projectNames.contains(it.moduleVersion.id.name) }
                                                                                                                                                       artifacts.each { artifact ->
            dependencies {
                                                                                                                                                                                                                          repo (
                                                                                                                                                                                                                                  group: artifact.moduleVersion.id.group,
                                                                                                                                                                                             name: artifact.moduleVersion.id.name,
                                                                                                                                                                                               version: artifact.moduleVersion.id.version,
                                                                                                                                                                                         ext: artifact.extension
                                                                                                                                                                                                         )
                                                                                                                                                                                                                                   repo (
                                                                                                                                                                                                                                  group: artifact.moduleVersion.id.group,
                                                                                                                                                                                             name: artifact.moduleVersion.id.name,
                                                                                                                                                                                               version: artifact.moduleVersion.id.version,
                                                                                                                                                                                         ext: 'pom'
                                                                                                                                                                                                                      )
                                                                                                                                                                                                                               }
                                                                                                                                                                                                                               }
        configurations.repo.resolvedConfiguration.resolvedArtifacts.each { println "Repo: ${it.moduleVersion.id.name}-${it.moduleVersion.id.version}.${it.extension}" }
    }

Does this look like a reasonable way to get hold of POMs and JARs?

Looks reasonable, although this won’t do any conflict resolution. Instead of resolving subproject configurations separately, an alternative is to create a new configuration, declare all subproject configurations as dependencies thereof, and resolve the configuration.

The harder part is to get at the sources Jars, Javadoc Jars, parent POMs, checksums, etc. If you want to go down this route, have a look at the source code of Gradle’s IDE plugins, which at least covers sources and Javadoc Jars. To get at parent POMs, you may have to keep parsing and downloading POMs until there are no more parents.