I spent some time today debugging very slow Java compiles (20+ seconds instead of 1 second) on our build server.
The culprit turned out to be a combination of the use of clean checkouts and the incremental java compiler. The clean checkout blows away the
.gradle directory inside the checkout, which seems to trigger some very expensive re-analysis of compile inputs if incremental compilation is disabled, which are not cached anywhere else:
build 20-Feb-2018 18:54:10 18:54:10.053 [INFO] [org.gradle.api.internal.tasks.compile.incremental.jar.JarClasspathSnapshotMaker] Created jar classpath snapshot for incremental compilation in 24.082 secs. 1010 duplicate classes found in classpath (see all with --debug).
build 20-Feb-2018 18:54:10 18:54:10.220 [INFO] [org.gradle.execution.taskgraph.DefaultTaskPlanExecutor] :nz.co.tvnz.play.api.impl:compileJava (Thread[Task worker for ‘:’ Thread 4,5,main]) completed. Took 28.665 secs.
The fix was easy : disable incremental java compilation on the build server. However, a few questions…
That’s a pretty painful cold start. Should it actually be this slow with a project-cache-dir directory mising ? Builds on clean checkouts happen all the time, especially on build servers. If the slowdown had only been tenfold, I might not have spend the time to debug, and given incremental compiles are enabled by default, there may well be a large number of continuous builds out there wasting a lot of time in JarClasspathSnapshotMaker.
Given that it’s such a massive hit, should this be highlighted more obviously in the documentation? Or in build scans? The build scan presents a very opaque view of a block of time just spent in compileJava, it wasn’t until debug was enabled things became more obvious. https://docs.gradle.org/current/userguide/java_plugin.html mentions jar analysis is cached and can be slower on cold start, but I read this initially as the only cache I was aware of, the shared gradle cache.
And should/can something this expensive be cached in that shared gradle cache? Until today I wasn’t actually aware of the project-cache-dir, and had concentrated attention on the local and remote caches, which are well documented and have hit/misses well reported. Though it’s interesting to now have to consider a third cache concept (fourth if you count
~/.gradle, then also the daemon) - eg should we move our project-cache-dir cache somewhere else when running on the build server? etc - it’s ultimately another moving part.