I’ve been working on a proof of concept at my current job to migrate a rather massive (10 million LOC) product to a single from-source Gradle build, using Gradle 3.5 with build cache.
gradlew assemble (when up-to-date) only takes 6-7 seconds.
gradlew clean assemble only takes 41 seconds(!). The Build Cache makes a massive difference here.
Even with changed source files to a variety of projects, the worst case for gradlew assemble is 2 to 3 minutes.
Just for some perspective, the current build process for this codebase takes 4 hours to run through, end to end, excluding test time.
Kudos to the Gradle team for their tireless work on performance. It makes a world of difference.
Thanks @rhencke. We’re really excited about it ourselves.
Could you describe a little bit more about your project? Is it all Java? Have you made any tasks cacheable that aren’t cacheable out of the box? Are you running with --parallel?
Are you just using the local cache? Have you tried using a remote build cache?
@sterling It’s a mix of a few languages, but the bulk of it is pure Java (currently compiling/targeting Java 8). There’s some Groovy used on the testing side, but nothing really on the production side. No annotation processing to worry about (whew). Java’s used for everything from the back-end server processes that run (JBoss/Tomcat) to this product’s previous front-end (applets, which are still supported, but the new front-end is HTML-based).
Currently, I’m using parallel, configure on demand, and the daemon. This is using the local cache - I’ve experimented with the remote cache enough to know it works, but I haven’t stressed it any.
I’ve done some preliminary testing with the remote build cache, using a plain Artifactory repo, but that’s not a terribly ideal solution - I’m trying to push the ball forward on something like Gradle Enterprise, something that has cache expiration policy, etc. But even with just the local build cache, there’s still some really nice wins. One example is branch switching. There are often multiple versions that require support (think service packs), and it’s not unusual to switch back and forth between versions using Git branching to accomplish this. With a local build cache, the recompilation costs around this are extremely minimal.
I haven’t experimented with creating custom cacheable tasks yet. For my use case, thankfully, everything on the path of ‘gradlew build’ so far is a case where you all already did the hard work.
@rhencke FWIW, I’d encourage you to start playing with remote cache as soon as you can. I got local caching up and working in a manner of minutes, but once you go cross-platform or try to push task outputs from a build/CI machine and then read them locally, lots of stuff can go haywire!
It’s totally worth the investment though to keep investigating; it’s flushing out a lot of bugs in my code. You can read more about my experience here.
Here’s an example of a test I built last week to compare the hash of a task when run from different directories. I have to obfuscate a few of the details, and technically it doesn’t use build caching at all, but rather compares the content of the resulting info-level log message that starts with Cache key for task when build caching is enabled. Enjoy
class MyPluginCacheabilityTest extends ... {
/**
* For this test only, we override GradleRunner's TestKitDir property with a randomly generated folder.
* This ensures the local cache does not linger for subsequent test runs.
*/
@Rule
public final TemporaryFolder testKitDir = new TemporaryFolder()
@Rule
public final TemporaryFolder testProjectDir = new TemporaryFolder()
@Rule
public final TemporaryFolder additionalTestProjectDir = new TemporaryFolder()
/**
* Creates two identical project folders with the same resources and buildscript
*/
@Before
void beforeMethod() {
[testProjectDir, additionalTestProjectDir].each { target ->
//copy or create your identical build sources here
}
buildScript << '''
apply plugin: 'your plugin'
repositories {
...
}
dependencies {
...
}
//other setup specific to your plugin
}
'''
//make a copy of the buildscript in the additional testkit dir
additionalTestProjectDir.newFile('build.gradle') << buildScript.getText(StandardCharsets.UTF_8.toString())
}
@Test
void 'Run same build from two different directories; assert task hash is equal'() {
GradleRunner runner = GradleRunner.create()
.withProjectDir(testProjectDir.root)
.withTestKitDir(testKitDir.root)
.withPluginClasspath()
.withGradleVersion('3.5')
.withArguments('myTask', '--build-cache', '-i')
BuildResult result = runner.build()
String firstExecutionCacheKey = getCacheKeyFromBuildResult(result)
runner = GradleRunner.create()
.withProjectDir(additionalTestProjectDir.root) // this line is critical. If your plugin/task is path-sensitive, running it this way should shake out cache misses
.withTestKitDir(testKitDir.root)
.withPluginClasspath()
.withGradleVersion('3.5')
.withArguments('myTask', '--build-cache', '-i')
result = runner.build()
String secondExecutionCacheKey = getCacheKeyFromBuildResult(result)
assertThat(firstExecutionCacheKey)
.as('Expected cache keys to be equal if path sensitivity is set correctly')
.isEqualTo(secondExecutionCacheKey)
}
private static String getCacheKeyFromBuildResult(BuildResult result) {
List<String> matchingLines = result.output.readLines().findAll { it.startsWith('Cache key for task \':genXmlSources\' is ') }
assertThat(matchingLines)
.as('there should only be one cache key calculated for this task in this build invocation')
.hasSize(1)
return matchingLines.first()
}
}
Thank you for the interesting statistics after migration to Gradle.
Did you use Maven previously? I am curios because I am also planning to migrate for some of the projects so I am interested about performance benefits.
