Managing Evolving Dependencies

Old Forum Archive
1

This is a broad “hail mary” question , but I’m hoping there are some of you who work on large scale development efforts that could share your experience with me.

My job currently has around 100 Mercurial repositories and we’re on pace to have around 1000 by the time we’re done. We’re just starting the migration from Ant/Ivy to Gradle. As part of this, I’ve been trying to define a strategy for handling repos and 3rd party (open source) dependencies that are constantly evolving. We know the Gradle wrapper & semantic versioning of our custom plugins, production, and test code are key to this strategy, but we’re not sure what the right architecture & policies are.

Gradle gives us several choices: * Each repo could have a build.gradle with its own declaring its own dependencies. This offers the ultimate independence, but makes it very hard to roll out large scale changes, e.g., logging, the JDK, etc.

  • I’ve prototyped a DSL that could be used as a wrapper around the Gradle dependencies DSL. My DSL allows developers to express dependencies at a higher more abstract level. For example,
dependenciesSimplified {
  junit
}

would result in our custom plugins creating a dependency on our project’s default version of JUnit. If the repo wanted something other than the default, additional information could be passed, e.g. “lastVersionOf junit”. I like this because ** It prevents people from knowing Gradle/Ivy dependency syntax. ** It ensures the same dependencies are expressed correctly across repos (we’ve found we already have a lot of inconsistencies due to wide variety of Gradle/Ivy configurations used by ourselves but especially 3rd parties). ** It allows us to change the defaults and have them picked up by repos the next time they are built OR en masse via something like the Jenkins Ivy plugin (supporting automatic downstream builds) or Gradle multi-project builds (which doesn’t seem realistic on a project of our size).

  • The last option is a bit ambitious: Our custom plugins could have a package-to-dependency map which could be used to automatically declare dependencies based on introspecting the source code. This would have all the features of the previous DSL option with the obvious added benefit of minimizing or eliminating dependency declarations in each repo.

Finally, there are some dependencies we don’t manage well: * Dependencies on the JDK, OS, etc. * Actual runtime dependencies - the runtime configuration we use is really a “unit test” configuration. Real runtime is where the transitive dependencies are deployed, installed, and activated in a compatible JVM.

We’re currently using Chef and OBR (OSGi) for this now with some success, but we have big gaps in what this supports. But it is unclear how to merge the deploy/install/activate time dependency management with the compile/test/package time dependencies Gradle excels at.

Sorry for such a long, rambling post. Hopefully these are problems of interest to a wide audience …

2

You are indeed touching on a lot of topics. Here are some thoughts:

Plugins are a good way to standardize dependencies across builds. In the simplest case, a plugin could just provide a map that gives names to dependency declarations:

ext.libs = [
  junit: "junit:junit:4.10",
  spring: ["org.springframework:spring-core:3.1.0.RELEASE", "org.springframework:spring-web:3.1.0.RELEASE"],
  other: dependencies.create("other:other:1.0") { exclude module: "foo" }
]

As shown above, the map can contain any dependency notation supported in a ‘dependencies’ block. Builds can then use the map like so:

apply plugin: "commonLibs"
  dependencies {
  compile libs.spring
  testCompile libs.junit
}

I’m not sure if I’d introduce a separate DSL for common dependencies. Users will probably have to be able to add regular dependencies anyway, and hence they’ll also have to know the Gradle-provided notation(s). One potential advantage of a separate DSL is that the plugin could warn a user when he adds a regular dependency that should instead be applied from common libs. As you already said, a separate DSL also gives you more control over how dependencies get resolved, although you could probably do something similar with ‘ext.libs’ (see https://github.com/pniederw/elastic-deps for inspiration).

If you want to offer something like ‘lastVersionOf junit’, make sure to implement it in such a way that it is reproducible when checking out an old version of the project.

I think it’s valuable to state compile dependencies explicitly. I’m unsure whether deriving them from the source code is desirable, how well it would work in practice, and how the user could stay in control if he needed to.

Why is it that you don’t use configurations.runtime for the actual runtime dependencies? Does this mean that you don’t use transitive dependency management for your own artifacts?

3

Thanks for the feedback.

One of my early prototypes was the libs map you described. My implementation used methods for each product: * A no-arg method used the default version, e.g., “libs.junit()” * A method that could accept the version, e.g., “libs.junit(1)”, where “1” was some internal one-up number we create to identify the versions we use.

I like the concept of defaults because it makes it easy to globally control versions. However, the first time an incompatible version is introduced, you’d have to have the individual build.gradle’s pass in a version number to choose to use the new incompatible version. From then on, you’ve lost your default version.

I agree that I’d like to avoid reinventing the wheel by creating our own dependency DSL. One of the reasons I started leaning towards a DSL is because in order to use some dependencies, we have to declare dependencies on multiple configurations, e.g.,

dependencies {
   compile
     configuration: "api", group: "org", name: "name", version: "version"
   testCompile configuration: "runtime", group: "org", name: "name", version: "version"
   osgi
          configuration: "osgi", group: "org", name: "name", version: "version"
}

I feel like this is exposing build implementation to each repo, i.e., the custom plugins have an implementation that chooses to create compile, testCompile, and osgi configruations. Exposing this implementation to each build.gradle in each repo prevents me from being able centrally change the types of configurations we use. For example, I don’t like having compile or testCompile (in this case) depend on concrete classes (via the runtime configuration. I’d like the ability to centrally enforce this policy. I’m also not convinced having a separate osgi configuration is necessary.

We do use the “runtime” configuration for the actual runtime environment. We also use an “osgi” configuration that contains the output of the runtime configuration manipulated with IPojo.

My original point is that one can have each repo building with a distinct version of Gradle, Gradle plugins, JDK, groovy, etc., but in the end, these repos get deployed into a single runtime environment running a single JDK, groovy, etc. Just because unit tests on each repo pass in the unit test environment defined by each repo doesn’t mean the integration tests will pass in the actual runtime environment. Centralized dependency management needs to keep this in mind.

I like the idea of deriving dependencies from source code for couple of reasons: * It trivializes the content of build.gradle. * It prevents extraneous dependencies, e.g., removing a real dependency without changing the build.gradle.

I’m nervous about this approach because of all the reasons you stated. My thought is to give user’s the best of both worlds: auto discovery based on source AND the libs map/methods/DSL described above which overrides anything automatically discovered.

Thanks again for the feedback!