How to do cross-project compile dependencies for incremental java compiles?

I’ve been trying unsuccessfully to accomplish incremental compile of java files in a large multi-project build with “project to project” compile dependencies. After a number of iterations, I can’t get it to recompile a ‘referencing’ java file when the ‘referenced’ java file in another project changes. I have a relatively simple test bed that illustrates what I’m seeing. I don’t see this problem when the ‘referencer’ is in the same gradle project. I’ve edited my original post to show in bold what appears to be the problem (Gradle sees a class has changed but does not recompile a class that references it unless that class is in the same project).
I’m hoping to get some advice on what I am doing wrong in my build scripts. I’ve tried different ways of defining my source/inputs and my compile dependencies and haven’t gotten this to work yet.
What I see is everything compiles fine the 1st time. Then if I alter the Person.java file in the src directory, that file is recompiled and so is the Person2.java file that references it in the same project. However, the DeptInSrc2.java (that also references it) in the src2 project is not recompiled.
When I use the --info flag on the gradle command line, the detail shows that the src2 project’s inputs sees that Person.class changed. But then, it says there is nothing to compile. (I pasted a snippet of that info at the bottom of this post).
I’m using Gradle 2.7 and my simple test bed for this currently has the following structure:

  • gradle-test (project’s root directory)
    build.gradle ( ‘:’ project’s build script to compile src java files into output)
    settings.gradle
    gradle.properties
    • buildSrc (directory containing some groovy utilites for path resolution etc)
    • output (directory where resulting class files are being put)
    • src (dir simulates common source component for other components to reference)
      com.my.test.Person.java (referenced in Person2.java & in “src2” project’s DeptInSrc2.java)
      com.my.test.Person2.java (java source that references Person’s getName() method
    • src2 (dir simulates a source component that references a common component)
      build.gradle (dependent gradle project’s build script compiles src2 files into output)
      com.my.test.DeptInSrc2.java (references Person’s getName() method from src)

The build.gradle in root project (that compiles the src) looks like this:
apply plugin: 'java’
apply plugin: 'eclipse’
ext { // add these to project to use for constants
propBasedJavaFlags = ["-encoding",“UTF-8”,"-g","-deprecation","-implicit:none","-proc:none",
   “-target”,“1.5”,"-source",“1.5”]
}
sourceSets {
   main {
     java {
        srcDirs = [‘src’]
        include ‘**/*.java’
     }
   }
}
compileJava {
destinationDir=file(‘output/src’)
options.incremental = true
options.compilerArgs.addAll(project.propBasedJavaFlags)
source = sourceSets.main.java
}

The build.gradle in the src2 (simulated dependant component) looks like this:

apply plugin: ‘java’
apply plugin: ‘eclipse’
sourceSets {
main {
   java {
      srcDirs = [’…/src2’]
      include ‘**/*.java’
   }
}
}
compileJava {
   destinationDir=file(’…/output/src2’)
   options.incremental = true
   options.compilerArgs.addAll(project.propBasedJavaFlags)
   source = sourceSets.main.java
   dependencies {
      compile files(’…/output/src’)
      // next line no better for incremental than above
      //compile project(’:’)
   }
}

Here’s what a snippet of the relevant --info shows for a compile where I’ve altered a method in Person.java that is referenced in Person2.java of src and DeptInSrc2.java of src2.
Note: Even if I delete a method from Person.java that is referenced in the DeptInSrc2.java file, that file is not recompiled. Notice though that it does recognize an input for consideration in src2:compileJava has changed. Just does not recompile anything. What am I doing wrong here that is causing this behaviour?
:compileJava (Thread[Daemon worker,5,main]) started.
:compileJava
Executing task ‘:compileJava’ (up-to-date check took 0.023 secs) due to:
Input file C:\hmc_workspaces\trunk_ws\gradle-test\src\com\my\test\Person.java has changed.
Created jar classpath snapshot for incremental compilation in 0.0 secs.
Compiling with JDK Java compiler API.
Incremental compilation of 2 classes completed in 0.457 secs.
Class dependency analysis for incremental compilation took 0.009 secs.
Written jar classpath snapshot for incremental compilation in 0.003 secs.
:compileJava (Thread[Daemon worker,5,main]) completed. Took 0.535 secs.
:src2:compileJava (Thread[Daemon worker,5,main]) started.
:src2:compileJava
Executing task ‘:src2:compileJava’ (up-to-date check took 0.018 secs) due to:
Input file C:\hmc_workspaces\trunk_ws\gradle-test\output\src\com\my\test\Person.class has changed.
Created jar classpath snapshot for incremental compilation in 0.001 secs.
None of the classes needs to be compiled! Analysis took 0.007 secs.
Written jar classpath snapshot for incremental compilation in 0.003 secs.
:src2:compileJava (Thread[Daemon worker,5,main]) completed. Took 0.049 secs.
BUILD SUCCESSFUL

Wow, I am surprised there have been no replies to this post…is this a known issue or is this how things are supposed to work. We are having the same issue with our move to Gradle and not having the type of incremental java compile support described in this post is a show stopper for us. Thanks!

Here’s what I think happened:

The initial build script was something like (I used two subprojects, common and impl):

subprojects {
   apply plugin: 'java'
}

project(":impl") {
   dependencies {
      compile project(":common")
   }
}

This worked. Then later the location of the class files changed and incremental compilation was enabled:

subprojects {
   apply plugin: 'java'
   compileJava { 
      options.incremental = true 
      destinationDir = file("../output/${project.name}")
   }
}
project(":impl") {
   dependencies {
      compile project(":common")
   }
}

This appears to sort of work, but the downstream project (impl) doesn’t seem to rebuild when common changes.

Try something other than a project dependency:


subprojects {
   apply plugin: 'java'
   compileJava { 
      options.incremental = true 
      destinationDir = file("../output/${project.name}")
   }
}
project(":impl") {
   dependencies {
      compile project(":common")
      compile files("../output/common")
   }
}

This seems to work better than just the project dependency (build order looks right, impl detects changes to classes from common), but incremental compile doesn’t work. That’s as close as I could get to reproducing the same kind of problem.

The issue is that a project dependency works by sharing jar files. So when the impl project needs to use the common project, Gradle builds a jar that contains the output of the main Java source set (i.e., sourceSets.main.java.output). By changing the output directory of the compile task directly, the jar no longer sees those classes as an input. Things keep working because the default path to the classes is still an input to the jar and those class files won’t be deleted (except for a clean).

Assuming there’s not something missing from the example and you were using only project dependencies, if you were to clean the build directory and try to rebuild, downstream projects would fail to build because they would be building against mostly empty jars.

Adding both the new class output directory and the project dependency almost fixes it because the jar-with-old-classes has the same class files as the output directory, so the analyzer sees those class files first and says nothing has changed.

To change the output directory for the classes, you want to change the source set’s output:

   sourceSets {
      main {
         java {
            output.classesDir = file("../output/${project.name}")
         }
      }
   }

The project’s compile task and jar should pick up the new path automatically.

To follow up on Sterling’s post, the example in the original post has a number of errors which may have contributed to the behavior. For instance, the include patterns on the source sets are wrong (i.e. ‘*/.java’ would not match any files) and the redirection of the compile output prevents the jar task from creating a proper jar for inter-project dependencies (as well as the associated task dependencies). In any event, when I cleaned things up, the incremental compile appears to work as expected.

Having said that, it’s entirely possible that there could still be an issue with incremental compilation. If you could create a simple example that demonstrates the behavior (and ideally publish in a Github repo or something similar), I’m sure someone will look into it.

@Gary_Hale I thought the same thing too, but that’s just the forum eating some of the build script.

Sorry. I never noticed that the forum entry lost characters like the double asterisk in the include paths etc… I grabbed all that directly from scripts that are working except for the issue I am describing. These 2 scripts were ones I made-up just to try to demonstrate what I was seeing with my much more complex production scripts.
The commented-out project based dependency like //compile project(’:’) was not my original script. I did that after experiencing the problems just as one of many attempts at doing things with different syntax to try to find a flavor that worked. When you describe me “moving” the class files, you lose me. I’m new at this, and am probably missing something important in that statement you made. I’ve always tried to put the class files in an output/src directory from the main project’s compilation and in an output/src2 directory for the subproject’s. Regardless - thanks for the responses and I will now try the different syntax you show for where to define the output directory. I’ll update this with what I find after that attempt. Thanks again for the response. I really appreciate it. We’ve got a huge and complex hierarchical structure of Java projects in addition to a lot of native code. We are trying to change to Gradle and this type incremental compile issue in the Java portion has been a big problem for us.

No problem.

Why do you need the different location for the class files?

I’m trying to replace an existing build system without impacting downstream processes that use the results that the old system created. There are web interfaces and downstream builders etc… that are expecting certain structure of the output directory. Also, (and I’m not saying this is a good idea, but…) there are some different versions of classes (with the same path/name) in the various Java projects. So, if we used the same output location we’d be overwriting them etc for these.
BTW, I tried the change you suggested and I’ve gotten the same result of it not compiling the subproject’s java file that references the common one. The file that makes the same exact kind of reference to it within the same project is always recompiled when the referenced one changes.
Basically both now use the sourcesets output.classesDir like you show… i.e. following for the subproject’s script:

import java.nio.file.Files;

apply plugin: 'java’
apply plugin: ‘eclipse’

sourceSets {
main {
java {
srcDirs = [’…/src2’]
include ‘**/*.java’
output.classesDir = file(’…/output/src2’)
}
}
}

compileJava {
    options.incremental = true
    options.compilerArgs.addAll(project.propBasedJavaFlags)
    source = sourceSets.main.java
    dependencies {
         compile files('../output/src')
    }
}

maybe does the dependencies part of the compileJava need different syntax as well?

When you said “the redirection of the compile output” was a problem, are you talking about the lines in my build script like: “output.classesDir = file(‘output/src’)” ? If so, does that mean there is no way for Gradle incremental builds to work properly when the class files from one project are desired to be in a different location than the class files from another project? We have a need to do that somehow. I thought it was kinda comparable to Eclipse Java projects potentially placing their class files in a bin directory that is different than the bin directory of other projects - and yet - that incremental compiler seems to be able to handle cross project references.
I’m hoping if output class location is the issue that its just my syntax/methodology for doing it is wrong? Finally, I took your suggestion and put the example I’ve been working with on GitHub here: https://github.com/grejones/gradle-test
I hope that gets rid of some of the confusion that was caused by characters like my asterisks getting “scrogged”.

Does your question “Why do you need different location for the class files” imply that it can’t be done with Gradle? i.e. Is it true that there isn’t a way to make incremental Java compiles work correctly with multiple projects (each placing class files in different sub directories of a common output directory) ? That’s what I’m trying to do. I have my sample code in GitHub here: https://github.com/grejones/gradle-test
If it’s not possible, then what is the actual best practices way to even do the multi-project Java incremental compiles when the classes are going to the same output directory? Just curious as I haven’t quite got that to work yet with my first few attempts either.

You should have a project dependency here, not depending on class files manually:

dependencies {
  compile project(':yourOtherProject')
}

Thank you ! I had tried that before and must have had something else
wrong at the time. I really appreciate that you replied to this. I’ll put
something in the forum after I try some more testing.
This illustrates something I’ve had a lot of difficultly with (as someone
completely new to Gradle). I may be missing something, but the
documentation seems (in some cases) to indicate multiple different ways to
accomplish a thing. But (I at least) missed the fact that some of those
optional ways to define stuff are not compatible with things like
’incremental’ builds. For instance how one can define source, output
location, dependencies etc… But some of those approaches don’t work with
things like “incremental” even though they do work when just doing full
builds.

I’ve been doing full builds for awhile and hadn’t tried incremental until
more recently. All 54 of my java project builders are working. But then I
realized I hadn’t tested/tried incremental. All my builders set
properties on the CompileJava tasks like “destinationDir” and “.source”. I
never used sourceSets type syntax like shown below because I have some
properties files that I read things from for each project and then I build
dynamically where each project’s source is, where its destination
directory is, where other compiled classes that it needs for classpath are
located, what compiler options it uses, etc…
When I first started trying to get incremental to work, it seemed like I
had to change all my builders to define those sourceSets more explicitly
(to even get close to getting incremental) to work. Now similarly, I’ve
been defining dependencies all along referencing class file locations with
"compile files(’…" type syntax. But it appears only “compile
project(’…” maybe will work. Oh well. I will have to see what impact
changing all the compile dependencies to be project based has on my code.
But I really appreciate your answer. It was the first one clear enough
for a newbie like me to understand.

sourceSets {
main {
java {

Greg Jones
Software Engineer - System z Firmware Development
Dept. A0LG Div 7T
Bld. 256-3rd floor (office CC04)
1701 North Street
Endicott, New York 13760
Phone (607)429-4485; Fax (607) 429-5460Tie(620)
Internet: grejones@us.ibm.com

Thank you for sharing more details on your use case. I understand now why you were using classes directories directly. In that case, you’ll want to remove the incremental option until you have migrated your projects to a more explicit structure.

I’ll bring this up internally, we could at least warn the user that incremental compilation will probably not work when he has class folders on the classpath.

Thank you for pointing this out :slight_smile: