Very slow incremental compile with empty project-cache-dir on build server

Help/Discuss
1

I spent some time today debugging very slow Java compiles (20+ seconds instead of 1 second) on our build server.

The culprit turned out to be a combination of the use of clean checkouts and the incremental java compiler. The clean checkout blows away the .gradle directory inside the checkout, which seems to trigger some very expensive re-analysis of compile inputs if incremental compilation is disabled, which are not cached anywhere else:

build 20-Feb-2018 18:54:10 18:54:10.053 [INFO] [org.gradle.api.internal.tasks.compile.incremental.jar.JarClasspathSnapshotMaker] Created jar classpath snapshot for incremental compilation in 24.082 secs. 1010 duplicate classes found in classpath (see all with --debug).
build 20-Feb-2018 18:54:10 18:54:10.220 [INFO] [org.gradle.execution.taskgraph.DefaultTaskPlanExecutor] :nz.co.tvnz.play.api.impl:compileJava (Thread[Task worker for ‘:’ Thread 4,5,main]) completed. Took 28.665 secs.

The fix was easy : disable incremental java compilation on the build server. However, a few questions…

  • That’s a pretty painful cold start. Should it actually be this slow with a project-cache-dir directory mising ? Builds on clean checkouts happen all the time, especially on build servers. If the slowdown had only been tenfold, I might not have spend the time to debug, and given incremental compiles are enabled by default, there may well be a large number of continuous builds out there wasting a lot of time in JarClasspathSnapshotMaker.

  • Given that it’s such a massive hit, should this be highlighted more obviously in the documentation? Or in build scans? The build scan presents a very opaque view of a block of time just spent in compileJava, it wasn’t until debug was enabled things became more obvious. The Java Plugin mentions jar analysis is cached and can be slower on cold start, but I read this initially as the only cache I was aware of, the shared gradle cache.

  • And should/can something this expensive be cached in that shared gradle cache? Until today I wasn’t actually aware of the project-cache-dir, and had concentrated attention on the local and remote caches, which are well documented and have hit/misses well reported. Though it’s interesting to now have to consider a third cache concept (fourth if you count ~/.gradle, then also the daemon) - eg should we move our project-cache-dir cache somewhere else when running on the build server? etc - it’s ultimately another moving part.

2

The short answer is that it shouldn’t be that slow. The snapshot for each individual JAR is cached in the Gradle user home, not in the project directory. The project directory cache only contains a thin layer aggregating those snapshots into a classpath snapshot per compile task. This should be fast and would be a performance bug if it isn’t.

Are you cleaning the Gradle user home (a.k.a. ~/.gradle) between builds?

3

That’s interesting - no, we’re not clearing the home dir that I know of.

Also, I was able to reproduce the slow compile on my local machine just by deleting the project .gradle dir and running a compile.

4

Thanks! I’ve created this issue to track the problem. The workaround is to deactivate incremental compilation on CI as you correctly pointed out. The Android plugin switches incremental compile on by default and we want to do so for all Gradle build soon. So this will be prioritized soon.

5

Great, good to know it’s not working as intended, if you get my meaning.