OK, Build Cache is pretty awesome


(Robert Hencke) #1

I’ve been working on a proof of concept at my current job to migrate a rather massive (10 million LOC) product to a single from-source Gradle build, using Gradle 3.5 with build cache.

  • gradlew assemble (when up-to-date) only takes 6-7 seconds.
  • gradlew clean assemble only takes 41 seconds(!). The Build Cache makes a massive difference here.
  • Even with changed source files to a variety of projects, the worst case for gradlew assemble is 2 to 3 minutes.

Just for some perspective, the current build process for this codebase takes 4 hours to run through, end to end, excluding test time.

Kudos to the Gradle team for their tireless work on performance. It makes a world of difference.


Remote Build Cache gotchas
(Sterling Greene) #2

Thanks @rhencke. We’re really excited about it ourselves.

Could you describe a little bit more about your project? Is it all Java? Have you made any tasks cacheable that aren’t cacheable out of the box? Are you running with --parallel?

Are you just using the local cache? Have you tried using a remote build cache?


(Robert Hencke) #4

@sterling It’s a mix of a few languages, but the bulk of it is pure Java (currently compiling/targeting Java 8). There’s some Groovy used on the testing side, but nothing really on the production side. No annotation processing to worry about (whew). Java’s used for everything from the back-end server processes that run (JBoss/Tomcat) to this product’s previous front-end (applets, which are still supported, but the new front-end is HTML-based).

Currently, I’m using parallel, configure on demand, and the daemon. This is using the local cache - I’ve experimented with the remote cache enough to know it works, but I haven’t stressed it any.

I’ve done some preliminary testing with the remote build cache, using a plain Artifactory repo, but that’s not a terribly ideal solution - I’m trying to push the ball forward on something like Gradle Enterprise, something that has cache expiration policy, etc. But even with just the local build cache, there’s still some really nice wins. One example is branch switching. There are often multiple versions that require support (think service packs), and it’s not unusual to switch back and forth between versions using Git branching to accomplish this. With a local build cache, the recompilation costs around this are extremely minimal.

I haven’t experimented with creating custom cacheable tasks yet. For my use case, thankfully, everything on the path of ‘gradlew build’ so far is a case where you all already did the hard work. :wink:


(Kyle Moore) #5

@rhencke FWIW, I’d encourage you to start playing with remote cache as soon as you can. I got local caching up and working in a manner of minutes, but once you go cross-platform or try to push task outputs from a build/CI machine and then read them locally, lots of stuff can go haywire!

It’s totally worth the investment though to keep investigating; it’s flushing out a lot of bugs in my code. You can read more about my experience here.


(Sterling Greene) #6

That’s a good suggestion, @DPUkyle.

You can also get a lot out of just trying to build from two different directories using the local cache. That’ll chase out any absolute-path gremlins.


(Kyle Moore) #7

Here’s an example of a test I built last week to compare the hash of a task when run from different directories. I have to obfuscate a few of the details, and technically it doesn’t use build caching at all, but rather compares the content of the resulting info-level log message that starts with Cache key for task when build caching is enabled. Enjoy :slight_smile:

class MyPluginCacheabilityTest extends ... {

    /**
     * For this test only, we override GradleRunner's TestKitDir property with a randomly generated folder.
     * This ensures the local cache does not linger for subsequent test runs.
     */
    @Rule
    public final TemporaryFolder testKitDir = new TemporaryFolder()

    @Rule
    public final TemporaryFolder testProjectDir = new TemporaryFolder()

    @Rule
    public final TemporaryFolder additionalTestProjectDir = new TemporaryFolder()

    /**
     * Creates two identical project folders with the same resources and buildscript
     */
    @Before
    void beforeMethod() {
      [testProjectDir, additionalTestProjectDir].each { target -> 
          //copy or create your identical build sources here
      }

      buildScript << '''
        apply plugin: 'your plugin'
        
        repositories {
          ...
        }
        
        dependencies {
          ...
        }
        
      //other setup specific to your plugin
      }
      '''
      
      //make a copy of the buildscript in the additional testkit dir
      additionalTestProjectDir.newFile('build.gradle') << buildScript.getText(StandardCharsets.UTF_8.toString())
    }
    
    @Test
    void 'Run same build from two different directories; assert task hash is equal'() {
      GradleRunner runner = GradleRunner.create()
        .withProjectDir(testProjectDir.root)
        .withTestKitDir(testKitDir.root)
        .withPluginClasspath()
        .withGradleVersion('3.5')
        .withArguments('myTask', '--build-cache', '-i')

      BuildResult result = runner.build()

      String firstExecutionCacheKey = getCacheKeyFromBuildResult(result)
        
      runner = GradleRunner.create()
        .withProjectDir(additionalTestProjectDir.root) // this line is critical. If your plugin/task is path-sensitive, running it this way should shake out cache misses
        .withTestKitDir(testKitDir.root)
        .withPluginClasspath()
        .withGradleVersion('3.5')
        .withArguments('myTask', '--build-cache', '-i')
        
      result = runner.build()

      String secondExecutionCacheKey = getCacheKeyFromBuildResult(result)

      assertThat(firstExecutionCacheKey)
        .as('Expected cache keys to be equal if path sensitivity is set correctly')
        .isEqualTo(secondExecutionCacheKey)
       
    }
    
    private static String getCacheKeyFromBuildResult(BuildResult result) {
        List<String> matchingLines = result.output.readLines().findAll { it.startsWith('Cache key for task \':genXmlSources\' is ') }
        assertThat(matchingLines)
          .as('there should only be one cache key calculated for this task in this build invocation')
          .hasSize(1)
        return matchingLines.first()
    }
}

(sergey.morenets) #8

Hi @sterling

Thank you for the interesting statistics after migration to Gradle.
Did you use Maven previously? I am curios because I am also planning to migrate for some of the projects so I am interested about performance benefits.

Thanks,
Sergey