Using Gradle & Jenkins on large multi projects


(Omri Spector) #1

When building “multi-module” projects with Jenkins CI, Jenkins, Nexus/Artifactory & Maven cooperate so that only the minimal set of modules is built. For large projects this is crucial, and makes the difference between fast several minute CI and slow several hours build.

I am seeking to reach the same build performance with gradle and multi-project.
(1) Is there a way to build only a subset of the sub-projects and their dependent sub-projects (similar to maven flags -pl and -am)
(2) Is there any jenkins integration so that indeed only these sub-projects will be invoked?
(3) If not - has anyone found a different way to replicate this behaviour based on a gradle plugin / task?


(Mark Vieira) #2

You can build a subproject (and its dependencies) by simply supplying the path to the subproject to the gradle CLI.

gradle :projectA:build :projectB:build

In Jenkins, if you are using the Gradle plugin you will simply provide these arguments as the “tasks” to execute.


(Omri Spector) #3

the challenge is not to build a sub-project - it is to integrate it into a CI so that:
(1) only subprojects that changed are triggered, automatically.
(2) dependant subprojects are also built

The way I understand it, your suggestion does not achieve either.
Am I missing something?


(Mark Vieira) #4

I think you are just talking about incremental build. Simply ensure you don’t do a clean or delete the Jenkins workspace between builds and you will get the behavior you want. Gradle will only rerun tasks whose inputs have changed. You can test this locally by doing gradle build changing a file in one of your subprojects and building again. The behavior you should see is that only the project that changed (and those that depend on it) should be built.


(Omri Spector) #5

No, there is actually a big difference.
With incremental build you do not clean up your workspace, which may lead to various issues (different discussion).
With maven multi module CI using jenkins, you always clean up (“mvn clean deploy”) - but since jenkins passes the list of modules that changed you get a minimal build. And since maven identifies inter-module dependencies - you get a consistent build.

It really is a very important feature for large projects, and if you would like further explanations I will gladly have a 1x1 discussion and show you how it works.
I like Gradle, but this is a major drawback when working on large projects (think hundreds of sub projects).


(Mark Vieira) #6

How does Jenkins determine which modules have changed, is it just based on VCS paths? Can you point me at some documentation as to how this works? If a Jenkins plugin is simply passing an argument list to the build tool I see no reason why this type of integration could not be possible with Gradle. It would just be a matter of implementing it.


(Omri Spector) #7

(1) Jenkins analyzes POM to determine what modules are part of the build
(2) It then checks which files have changed in the last commit, and maps
these to modules
(3) It them passes maven the list of changed modules (-pl m1,m2,m3,…) and
(4) requests “also make dependents” (-amd)

(1) , (2) & (3) can be implemented as part of a jenkins gradle plugin.
Maybe the current plugin
https://wiki.jenkins-ci.org/display/JENKINS/Gradle+Plugin is already
there?
(4) requires gradle support - and that is what I am asking about


(Mark Vieira) #8

I don’t believe items 1-3 are supported by the Gradle Jenkins plugin currently. Item 4 however is supported by Gradle. You simply run the buildDependents task.

gradle :changedModule:buildDependents

(Omri Spector) #9

Excellent.
Will look into the gradle jenkins plugin to see what is supported.

Thank you!


(Dilip Ladhani) #10

Hey Ormi,
Did you figure this out. I am looking for the exact same thing. I want Jenkins to only build changed modules in a multi-project gradle build


(Omri Spector) #11

At the time it was not supported. Moved on, so maybe today it is.


(Dilip Ladhani) #12

Thank you for getting back, Omri


(Natarajan) #13

I want Jenkins to only build changed modules in a multi-project gradle build. Any one figured out this?


(sugun999) #14

Hi,

Anyone has solution for this issue.