Incremental compile rebuilds more files than expected

I am trying to get incremental builds working in a legacy code base converted over from ant. One module is particularly challenging and has 15K java files to compile (trying to split it up but not there yet) which also makes it the most important since it is the most likely to have changes made to it and takes the longest time to build. In addition to being very large the build also uses the log4j annotation processor which was disabling incremental builds. To bypass this, I set ‘options.annotationProcessorPath’ to an empty file set for this module which did seem to work, now incremental compile is trying to determine which files need to be recompiled instead of just recompiling everything.

To test incremental compile I picked our top level java file which is imported into 31 other java classes in the same module, added a comment, then ran ‘gradlew compileJava’. To my surprise I found that this one change triggered a recompile of nearly 12K files of the 15k files. Further ad hoc testing seems to indicate that changing any of the ~12K recompiled files will trigger the others to also be rebuilt.

Running a similar test using the IDEA IDE build does not trigger the same scale of rebuild. If I change only this one top level file in IDEA and do a ‘build project’, IDEA rebuilds only the file changed. The IDEA project and its dependencies were built from the idea gradle plugin using the same gradle project tested above.

Is there some way to determine the trigger why the gradle incremental compiler is picking files for recompile in order to change the code to remove these dependencies (it is not obvious to me by inspection)? I tried using ‘–debug’ for the gradle build and did see the following (used gradle 4.10).

16:16:16.939 [INFO] [org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter] Task ‘:modules:mediation:compileJava’ is not up-to-date because:
Input property ‘source’ file /home/tdennler/ws/fspv/modules/mediation/src/main/java/com/adva/nlms/mediation/ has changed.

16:16:17.012 [INFO] [org.gradle.api.internal.tasks.compile.incremental.classpath.ClasspathSnapshotMaker] Created classpath snapshot for incremental compilation in 0.071 secs. 1590 duplicate classes found in classpath

16:18:07.455 [INFO] [org.gradle.api.internal.tasks.compile.incremental.SelectiveCompiler] Incremental compilation of 11736 classes completed in 1 mins 50.514 secs.

This is most likely because we don’t yet do api/impl separation within a single sourceSet. So the change will ripple along any reference even if that reference is only inside a method body. As a result we recompile too much, especially in more tangled code bases.

We definitely want to fix this, but until then modularization is your best option.