Setting up build environments with a multi-project build + maven

[Dan]

Hello all! We’re switching over to Gradle and reorganizing some code, and I was hoping to get some feedback on best practices.

We have a Jenkins CI server, and we’re aiming to reduce build times. Our project is primarily Java, with other components such as Grunt/Gulp Javascript builds and native projects.

Consider the following simple multi-project:

base-project/
|- widget-one/
|- widget-two/
|- container-app/
|- settings.gradle

Each sub-project would be its own git repository and have necessary build instructions. I would like each component to be separately buildable by Jenkins, which would then publish to our Maven repository. In the Jenkins context, it would have no knowldge of the multi-project.

However, on our local environments it can be expected each component would be under the ‘base-project’ directory, and inherit from settings.gradle, etc. In this environment, when I build ‘container-app’, which is dependent on the widgets, it would then build all dependencies as necessary.

I was thinking each component by default would publish to Maven, and in the development multi-project environment, it would override the repositories to MavenLocal(). Then, sub-projects could conditionally depend on other sub-projects if they exist, which would trigger builds on demand and publish to MavenLocal(). Then, since artifacts are updated and available in local maven cache, the sub-projects would pull down the recently-built artifacts without much build code duplication.

My question is, is this a sane configuration? Is there a more idiomatic way to accomplish what I’m working toward? My goal is that developers can pull down all the code and not worry about component artifacts being out of sync locally, while Jenkins builds remaining fast and atomic, decoupled from the large multi-project.

Also, as mentioned, some projects are not Java, but I don’t feel this is an issue conflicting with Maven or anything.

Again, I’m new to Gradle so I could be missing an essential idiom of some kind… thanks so much!

[mark_vieira]

If I understand this correctly, you are effectively creating a multi-project build from a bunch of disparate Git repositories (perhaps as Git submodules)? The issue I see with this is that you will run into problems if any of those submodules is in turn a Gradle multi-project. Even without that being the case, you now are creating an environment in which sometimes these builds are part of a multi-project and sometimes they are not. This could result in build inconsistencies where your development build does not properly match your CI build.

If your intent is to leverage your local maven repo to “distribute” your code updates between modules then there isn’t much benefit in creating an overarching project other than being able to specify dependencies between subprojects.

This is actually a use case that I think is pretty common. Especially when developing on any kind of services stack in which each service is likely it’s own Git repo and project (or multi-project). Personally, (I would love to hear other folks’ opinions) I would probably implement the developer build as either a) another project/repo or b) add it as additional tasks to an existing project (in this case probably the ‘container-app’ project). As far as keeping everything in sync, as mentioned above, you’ll have to rebuild/republish the submodules for developers to see changes. You could accomplish this by using the ‘GradleBuild’ task to call into the other project builds. You should still be able to leverage incremental build support and define a single point for developers to kick off their build producing a runnable development environment.

This is a scenario I find near and dear to my heart. One of the consequences of things like SOA is the increased complexity. Getting developers up to speed quickly with a development environment consisting of lots of interworking pieces and parts can be difficult. A lot of times creating a simple “composite” developer project like you’ve described is the best solution.

[Dan]

Thanks for your reply! Yes, we would be doing something similar to submodules.

Thinking about it more, this causes an additional problem of needing to bump versions of each component in order to properly maintain it in the artifact repository. This would be a hassle for some components that are not really needing versions, as they’re only dependencies of one parent.

The issue with calling into other builds is that I would rather the CI machine not need to pull down all the source. However, only pulling down a component seems to require it to maintain a proper version state. Maintaining that version, and all references to the version across the project tree sounds like it could become unmaintainable.

There’s an option of automating all version revving for components as a task of the multi-project, so no sub-project references a version number touched by a human outside of the multi-project root.

Just some ideas… really would love to iron out a good system that isn’t over-engineered across these environments!

[mark_vieira]

As far as my suggestion for the dev build calling into the other builds, that wouldn’t affect the CI build. That would be done only on the development builds. In that scenario the existing projects, structured as they are to build individually on the CI machines would remain the same and another project would be created that would reference the other. This additional project would only ever be used during development.

As far as versioning, yes, if you intend to have projects depend on each other in the form of a repository artifact then they will have to be versioned. Depending on the number of submodules that could become burdensome but that could always be automated.

However, that brings up an interesting point. If these project indeed depend on eachother in such a way as they will always be concurrently versioned together, then perhaps they should all fall under a single multi-project build. I understand the desire to shred your project out to improve CI build times but if they are tightly coupled it might make more sense for them to live together in a single repo under a single build.

So the way I see it is we are evaluating a trade off between simplicity and performance/modularity.