I read through Remote and Distributed Build Patterns but didn’t get much useful information out of it.
Problem: The build chain I maintain is massive and there are a large number of projects that have very long build times (~20min - 1hr per project). Luckily, a fair number of these projects can be built in parallel. I’m already making using of incremental builds with the build cache and it’s great. I want to speed things up even more but introducing distributed builds.
From what I can tell Gradle does not support this out of the box. Gradle Enterprise seems to support distributed test execution (would this work for builds too? I’d assume so.)
I’m looking for advice on how to approach this problem.
My current plan is to:
- Create a RemoteExec task, which is dirt simple - it would copy the entire working directory to a remote machine, execute a single command (such as running gcc or running a makefile) then copy all files back to the current working directory. (In addition to output files, there are debugging files from the build process that are valuable to have).
- This task would likely actually talk to some proxy that is a broker, which would then decide which worker machine should be used to run the given command.
- Run gradle with the –parallel flag, and this should automatically do a distributed parallel build.
Agnostic of specific details about the build process, does this approach seem reasonable? Are there other approaches I could take to solve this problem using gradle (such as abusing test distribution from the gradle enterprise feature)?