Faster Incremental Builds

As I have been converting our existing make / ant build system over to Gradle, I’ve been running various build-time comparison tests. For a full clean & build, the new gradle build is much faster than our old make / ant build system (likely in large part because the old system had grown so large and cumbersome that it was doing a lot of work it didn’t need to be doing). However for small changes to files, the build times are much worse under gradle. According to our analysis, this is because Gradle is “doing it right” and Make & Ant are “doing it wrong”.

Specifically, if I change a single Java source file (say, add an empty line) then in Ant only that one file is generated. By default make behaves the same way. The result is, it is really fast to build! And, the resulting build is completely unreliable. If you changed a file in a way that requires other files to be rebuilt, you’re hosed. So you just have to know what you’re doing to decide whether you need to do a clean & build or just a build.

Gradle on the other hand recognizes that any file changed related to a task may require the task to be re-executed (maybe the end result of re-executing the task is an identical output, but Gradle doesn’t know that until the task runs so it has to re-execute the task whenever anything changed that may impact the final result). This produces reliable builds, but also longer build times. In my case adding a single empty line to rebuilds the entire “graphics” sub-project which itself takes 30 seconds and then builds a half-dozen other dependent projects which takes over a minute in total (and a full clean & build takes 2 minutes).

However, Intellij IDEA and Eclipse both seem to have the best of both worlds – reliable builds and exceptionally fast build times. I think the difference is that these IDEs both keep extensive metadata information about the classes in a Java project, and so they can minimize which files actually are recompiled.

For example, suppose that following a full build you were to analyze the resulting class files. Each class file indicates the various imports that it relies on. Coupled with the information Gradle already has regarding the class path, it would be possible to determine exactly which class files any given class file depends on.

Suppose I have,, and

public class A { } public class B extends A { } public class C { }

In this case B is dependent on A, but C is independent of both. In such a case, I should be able to determine statically that if A changes I only have to recompile A and B. If B changes I only have to recompile B. And if C changes I only have to recompile C. All of this holds true regardless of which projects or tasks depends on which other projects or tasks.

This level of optimization could be added to further refine the javac tasks (that is, determining which Java source files need to be recompiled) without affecting the other existing logic for tasks. For example, the tasks could still process resources in the same way as they do now, and task dependencies are calculated and executed exactly as they are now. The only difference is that if a meta-data database exists for the class files in question, then we know what needs to be recomputed. So it would work (perhaps) something like this:

Task buildA determines that A has changed. A set of all source files that rely on A.class is constructed based on pre-existing meta-data that was computed during the last full build cycle. As each remaining task executes, any javac execution will only build those files contained in this set. Note that this set contains the transitive closure of all files that are affected by a change in A. So it includes not only B, but suppose also D where public class D { B b = new B(“Some Constructor”); }.

It is not a simple thing to get right and testing such a system seems like quite a task, but if done right I expect there to be an incredible increase in build time performance for large-project incremental builds when a change to a single file or small number of files is performed.

A friend pointed out that when using ant to compile, the useDepend option allows for something close to this (very close in fact):

There are limitations in Ant’s implementation. Also I am warned that:

“The CompileOptions.useAnt property has been deprecated and is scheduled to be removed in Gradle 2.0. There is no replacement for this property.”

And therefore some replacement for this functionality must be found in Gradle if the ant support is in fact removed in 2.0.

The limitation here is that there is no available incremental Java compiler for us to use. The folks from Typesafe implemented an incremental compiler for Scala that we integrated, but alas there is no Java equivalent at this time.

We are hopeful that someone will implement such a compiler that we can leverage.

Luke, see “sjavac” related to JEP in OpenJDK. I haven’t tried it out myself but I hear it works well. You can see it is funded by Oracle (meaning, we’re working on it) and was targeted for Java 8 milestone 6 (which shipped in January: also

I think you’re got your incremental compiler for Java now :slight_smile:

@Bair: thanks for the link. We are aware of this effort and are very keen to leverage it. We haven’t been talking about it publicly as it is still a little early.

If it works out, you can expect to see Gradle support for this.

@Luke, now that the “sjavac” suggested by @Bair has been completed: Could you update us the status of incremental Java compilation in Gradle?

Incremental Java compilation support was added in Gradle 2.1.

Awesome! Thanks @Mark. Does this work for Android projects as well?

I see no reason why not. The Android plugin uses standard Gradle ‘JavaCompile’ tasks. Take a look at their documentation on how to modify the build tasks.

Thanks. Hopefully I configured it correctly, but I got a “is not incremental, Unable to infer the source directories.” error.

Executing task ':app:compileDebugJava' (up-to-date check took 0.02 secs) due to:
  Input file /AndroidStudioProjects/CompileJavaTest/app/src/main/java/compilejavatest/ has changed.
:app:compileDebugJava - is not incremental. Unable to infer the source directories.
Compiling with JDK Java compiler API.
Class dependency analysis for incremental compilation took 0.031 secs.
Created jar classpath snapshot for incremental compilation in 0.032 secs.
Written jar classpath snapshot for incremental compilation in 0.005 secs.
:app:compileDebugJava (Thread[main,5,main]) completed. Took 1.664 secs.

And this is my configuration in build.gradle:

afterEvaluate {
    android.applicationVariants.each { variant ->
        variant.javaCompile.options.incremental = true

This error is thrown when any of the configured sources does not implement ‘SourceDirectorySet’. This must be due to the way the Android plugin is configuring the compilation tasks.

Hi Linton,

There’s been no changes since the release of 2.1. We are far from finished in this area.

As for ‘sjavac’, it’s too early to tell how practical this is going to be to use for builds outside of the JDK builds.