Incremental builds in a multi-project repository with Travis


(Rolv Seehuus) #1

Hi all,

We are using Gradle to build a repository that sets up a handful of services. We are using Travis to do the build, and the build publishes pr. service artefacts to Artifactory and then trigger a deploy (to a test-environment) through our CD system (Spinnaker.)

I’d like to set up our build, so that only changed and dependent parts are deployed into the new environment. Say if there’s three services A, B and C, and A is the frontend service for services B and C - a change to B should only trigger build and deployment cycle for A and B. Doing this in Travis is somewhat cumbersome, because Travis builds are immutable (There’s mechanisms to provide caching and so on, but it doesn’t seem to work for this problem.)

We are all fairly new to Gradle, so it might be that there’s some magic to apply somewhere to make this work satisfactory, but we have not been able to find anything yet.

I’ve had a glance on how this is solved by Bazel, and it is done by listing files changed in a commit-range (using the env-var TRAVIS_COMMIT_RANGE) and setting up an execution plan based on the files changed by the commit (e.g. see https://github.com/bazelbuild/bazel/blob/master/scripts/ci/ci.sh) I would very much like to see a plugin solving the same problem for Gradle - first and foremost for java/scala builds, but I would suppose it’s relevant for more language environments.

What I’ve figured so far, through some exploration (read: making random statments at random places in our build-scripts) is that it should be possible to list the files touched in a commit-range, filter these towards task inputs and then execute the remaining set of tasks - the task-dependency tree should then be able to do the rest.

My questions then, before I continue any work on this, is:

  • if this is a problem someone else have solved, or
  • if this is a problem someone else is working on solving, or
  • if there are other ways to solve this

Thank y’all in advance,
-r-


(Stefan Oehme) #2

You could build this in a three-step script which goes somewhat like this:

  • find out which files changed using git
  • run a Gradle task which iterates over all tasks and inspects their TaskInputs to see whether they contain those files
  • run another Gradle build with all the tasks found in the second step

I don’t know if someone has already built something like that. TeamCity does a simpler version of this where it detects which subprojects have changes (using the assumption that projects hierarchically contain all their files) and then only builds those subprojects.

In the near future, Gradle will provide a task output cache which will make this much simpler: You would then just share the cache from one build to another and tasks whose outputs can be found in the cache would just be skipped.


(Rolv Seehuus) #3

In the near future, Gradle will provide a task output cache which will make this much simpler: You would then just share the cache from one build to another and tasks whose outputs can be found in the cache would just be skipped.

This sounds excellent. When is this near future? :smiley:

Cheers,
-r-


(Stefan Oehme) #4

We’re targeting early next year for a first release. That version will not yet have a built-in cache implementation though, so you’d need to spin up your own (maybe a Hazelcast cache running in the container that runs your build).