Prevent repeated compilation of generated code

Help/Discuss
1

Hi Team,

I am seeking help for an issue pretty similar to this one: Cannot access compiled protobuf java class from one project in src/test of another project.

Since code says more than a thousand words, I prepared the public repository multi-project to discuss details.

Actually, I am struggeling a bit to distinguish between a plain multi-project-approach and really share outputs between projects. I am not fully sure, if my generated code counts as “(sub-) project”, since there is no main class or similar, just generated java code which is used elsewhere and shall not repeatedly compiled.

But first things first.

Composite Build

The branch composite-build shows a producer and consumer (sub-) project (package?), where producer generates the java code from a sample proto file and the consumer tries to use it in FooService.java. When compiling the consumer, I receive

$ ./gradlew :consumer:build -Duser.language=en

> Task :consumer:compileJava FAILED
***\multi-project\consumer\src\main\java\consumer\foo\FooService.java:17: error: cannot access GeneratedFile
        var request = Foo.FooRequest.newBuilder().setBar("Raise the Bar!").build();
                         ^
  class file for com.google.protobuf.GeneratedFile not found
1 error

[Incubating] Problems report is available at: file:///***/multi-project/build/reports/problems/problems-report.html

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':consumer:compileJava'.
> Compilation failed; see the compiler output below.
  ***\multi-project\consumer\src\main\java\consumer\foo\FooService.java:17: error: cannot access GeneratedFile
          var request = Foo.FooRequest.newBuilder().setBar("Raise the Bar!").build();
                           ^
    class file for com.google.protobuf.GeneratedFile not found
  1 error

* Try:
> Check your code and dependencies to fix the compilation error(s)
> Run with --scan to get full insights from a Build Scan (powered by Develocity).

BUILD FAILED in 1s
8 actionable tasks: 1 executed, 7 up-to-date

Shared Artifacts

This is similar to the issue in the linked ticket above, i.e., I try to follow the share outputs between projects on branch shared-artifacts. When trying to build the consumer, it seems even worse. Maybe my settings.gradle.kts is not setup properly?

$ ./gradlew :consumer:build -Duser.language=en

> Task :consumer:compileJava FAILED
***\multi-project\consumer\src\main\java\consumer\foo\FooService.java:3: error: package foo does not exist
import foo.Foo;
          ^
***\multi-project\consumer\src\main\java\consumer\foo\FooService.java:4: error: package foo does not exist
import foo.FooServiceGrpc;
          ^
***\multi-project\consumer\src\main\java\consumer\foo\FooService.java:10: error: package FooServiceGrpc does not exist
    private final FooServiceGrpc.FooServiceBlockingStub blockingStub;
                                ^
***\multi-project\consumer\src\main\java\consumer\foo\FooService.java:12: error: package FooServiceGrpc does not exist
    public FooService(FooServiceGrpc.FooServiceBlockingStub blockingStub) {
                                    ^
***\multi-project\consumer\src\main\java\consumer\grpc\GrpcClientFactory.java:3: error: package foo does not exist
import foo.FooServiceGrpc;
          ^
***\multi-project\consumer\src\main\java\consumer\grpc\GrpcClientFactory.java:12: error: package FooServiceGrpc does not exist
    public FooServiceGrpc.FooServiceBlockingStub fooServiceBlockingStub(GrpcChannelFactory factory){
                         ^
***\multi-project\consumer\src\main\java\consumer\foo\FooService.java:17: error: package Foo does not exist
        var request = Foo.FooRequest.newBuilder().setBar("Raise the Bar!").build();
                         ^
***\multi-project\consumer\src\main\java\consumer\grpc\GrpcClientFactory.java:13: error: cannot find symbol
        return FooServiceGrpc.newBlockingStub(factory.createChannel("foo"));
               ^
  symbol:   variable FooServiceGrpc
  location: class GrpcClientFactory
8 errors

[Incubating] Problems report is available at: file:///***/multi-project/build/reports/problems/problems-report.html

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':consumer:compileJava'.
> Compilation failed; see the compiler output below.
  ***\multi-project\consumer\src\main\java\consumer\grpc\GrpcClientFactory.java:13: error: cannot find symbol
          return FooServiceGrpc.newBlockingStub(factory.createChannel("foo"));
                 ^
    symbol:   variable FooServiceGrpc
    location: class GrpcClientFactory
  ***\multi-project\consumer\src\main\java\consumer\foo\FooService.java:3: error: package foo does not exist
  import foo.Foo;
            ^
  ***\multi-project\consumer\src\main\java\consumer\foo\FooService.java:4: error: package foo does not exist
  import foo.FooServiceGrpc;
            ^
  ***\multi-project\consumer\src\main\java\consumer\foo\FooService.java:10: error: package FooServiceGrpc does not exist
      private final FooServiceGrpc.FooServiceBlockingStub blockingStub;
                                  ^
  ***\multi-project\consumer\src\main\java\consumer\foo\FooService.java:12: error: package FooServiceGrpc does not exist
      public FooService(FooServiceGrpc.FooServiceBlockingStub blockingStub) {
                                      ^
  ***\multi-project\consumer\src\main\java\consumer\grpc\GrpcClientFactory.java:3: error: package foo does not exist
  import foo.FooServiceGrpc;
            ^
  ***\multi-project\consumer\src\main\java\consumer\grpc\GrpcClientFactory.java:12: error: package FooServiceGrpc does not exist
      public FooServiceGrpc.FooServiceBlockingStub fooServiceBlockingStub(GrpcChannelFactory factory){
                           ^
  ***\multi-project\consumer\src\main\java\consumer\foo\FooService.java:17: error: package Foo does not exist
          var request = Foo.FooRequest.newBuilder().setBar("Raise the Bar!").build();
                           ^
  8 errors

* Try:
> Check your code and dependencies to fix the compilation error(s)
> Run with --scan to get full insights from a Build Scan (powered by Develocity).

BUILD FAILED in 1s
1 actionable task: 1 executed

Custom Compile Task aka Incremental Build

Last but not least, I fully agree with this comment: Shouldn’t it be possible to simply add a custom task which compiles only the generated code? To my understanding, gradle requires for a custom task means to decide if the task is “UP-TO-DATE”, i.e., if the input and output is still the same?

For that approach, I aggregated all code in a single src/main folder-structure on the branch task. I tried to get support for this approach on Stackoverflow: Gradle Incremental Build Generated gRPC Java Code, but probably the question there is too generic without more precise context.

Summarized, how to make sure only parts of the code are re-compiled which have been changed. Any help is appreciated. Please, feel free to ask for details if I did not explain something detailed enough.

Regards, pd

2

Disclaimer: did not fully read your text here, following your “Since code says more than a thousand words” but mainly looked at your code

Your question is not really about generated code at all actually, it would just be exactly the same with manually written code.

On your branch composite-build you don’t have a “composite build”.
A “composite build” is a build that includes another build, so if you have includeBuild in a settings script, it is a composite build, if not, then not.
The settings scripts in consumer and producer are not used and are not used when running the top build, and actually they are very bad practice, as it means consumer and producer are part of multiple build, the one in the top directory, but also the builds you define in the subdirectories.

The setup you have in the top directory is a multi-project build with project dependencies.
A composite build you would for example have if your consumer build would includeBuild the producer build and then declare a normal dependency with coordinates, but it is unlikely that you want that, but instead want to remove the settings scripts in the subdirectories.

The compile error you get is, because you apply the wrong plugin and use the wrong scope for the protobuf dependency in producer.
As producer is providing a library, you should apply java-library, not java.
This - among other things - adds an api configuration besides the implementation configuration.
In api you should declare all dependencies that you use in your public API, that is superclasses, parameter types, return types, … (but not annotations) of public or protected methods.

This will make sure that these dependencies are in the compile-classpath of consumers.

The compile error says:
You try to compile FooService in consumer against Foo in producer.
You have proto only as implementation dependency in producer.
Foo extends com.google.protobuf.GeneratedFile.
com.google.protobuf.GeneratedFile is not part of the compile classpath of consumer.

As a bad-practice work-around consumer could also declare a compileOnly dependency on proto to manually add it to the compile classpath, but if the producer would be configured correctly as described above, then it would just work.

I also greatly recommend using the GitHub - autonomousapps/dependency-analysis-gradle-plugin: Gradle plugin for JVM projects written in Java, Kotlin, Groovy, or Scala; and Android projects written in Java or Kotlin. Provides advice for managing dependencies and other applied plugins · GitHub plugin which - once your code compiles - ensures that you declare your properties in the correct scopes (for the most part at least).

On shared-artifact you have the same bad-practice with the projects being part of multiple builds as described above, one project should always only be part of exactly one build and never be part of multiple builds like you have it.

The setup you have there is extremely unnecessary (except for applying the correct plugin in producer now), as the protobuf plugin already configures everything properly.
You add the generated sources again to the main source set (and in a bad way, never configure paths or you miss task dependencies) where they already are.
You create a second jar with the same content as the normal jar.
You create a variant with most important attributes missing to share that unnecessary additional jar and then only add it as classpath to a custom JavaExec task.
So yeah, of course your consumer compilation does not find anything from producer anymore, as you now do not only miss the proto dependency, but also the dependency to producer completely in your consumer compile classpath.
See again above for the proper solution.

Shouldn’t it be possible to simply add a custom task which compiles only the generated code?

You could do that, but it does not make any sense at all, why should you do that?
Any consumer of source code like sources jar tasks, static code analysers, and so on and so forth will miss the generated code and also the task dependency.
You usually have a task that generates the code and register that tasks output as source dir like in the comment you linked to and like is already done by the proto plugin.
Java Compilation tasks already are incremental, so only necessary things are recompiled anyway if possible even without having a separate compile task.

To my understanding, gradle requires for a custom task means to decide if the task is “UP-TO-DATE”, i.e., if the input and output is still the same?

For each and every task, whether custom or not, you must configure the proper inputs and outputs for various reasons. up-to-date checks is one, cacheability another, wiring task outputs to task inputs to get proper implicit task wiring yet another, …

Summarized, how to make sure only parts of the code are re-compiled which have been changed.

Whether you use the multi-project approach for some reason, or the task approach, Java compilation is incremental in Gradle, so you anyway only recompile what was changed or depends on something that was changed, so the answer is that you don’t need to do anything special for that.
If that was the only reason to split the code generation into a separate subproject, then this was quite unnecessary too.

Btw. you should stop using the io.spring.dependency-management plugin.
It is a relict from times when Gradle did not support consuming BOMs itself, by now does more harm than good, and even its maintainer recommends not to use it anymore.
Instead just consume the Spring BOM as platform dependency like also documented in the Spring docs.

3

Hi @Vampire ,

thank you very much for your thorough explanation.

Yes, probably my presented approaches are really not necessary, but the discussion leads to further points.

Btw. you should stop using the io.spring.dependency-management plugin.

That’s interesting. Because, when I got to https://start.spring.io and simply hit the Explore-button, I immediately see

plugins {
  java
  id("org.springframework.boot") version "4.1.0"
  id("io.spring.dependency-management") version "1.1.7"
}

at the top of the build.gradle.ktsfile. So, which documentation are you refering to, since the managing-dependencies documentation also uses it - although setting apply=false.

I also greatly recommend using the GitHub - autonomousapps/dependency-analysis-gradle-plugin: Gradle plugin for JVM projects written in Java, Kotlin, Groovy, or Scala; and Android projects written in Java or Kotlin. Provides advice for managing dependencies and other applied plugins · GitHub plugin which - once your code compiles - ensures that you declare your properties in the correct scopes (for the most part at least).

Also very interesting. I tried it with my latest main-branch, but when running $ ./gradlew buildHealth, I receive

Advice for root project
Unused dependencies which should be removed:
  implementation(libs.grpc.all)
  implementation(libs.spring.boot.starter)
  implementation(libs.spring.boot.starter.web)
  implementation(libs.spring.framework.grpc)

Doesn’t this plugin get along with version catalogs?

Last but not least - coming back to my original topic - I guess you are right, that I can simply relay on the incremental build feature. I looked deeper into Improve the Performance of Gradle Builds.

I updated the main branch to mostly reflect my current project such that the proto-files are even not part of the source-code, but imported. In my “real” source-code, the imported proto-files are rather huge, leading to approx. 47MB of generated code. Thus, I thought this code is re-compiled all the time, but I will inspect with --info.

4

That’s interesting. Because, when I got to https://start.spring.io and simply hit the Explore-button, I immediately see

That might be, but doesn’t change anything in what I said. :slight_smile:
Feel free to report a bug to them if there is none. :smiley:

which documentation are you refering to, since the managing-dependencies documentation also uses it

Scroll down :wink:

Doesn’t this plugin get along with version catalogs?

That’s not a question of version catalogs.
That’s because those “Starter dependencies and so on are actually slightly bad-practice”.
Best practice is, that you declare the dependencies you need where you need them.
Those “catch-all” dependencies are intended to declare one dependencies to add a bunch of other dependencies transitively and use those transitive dependencies.
This has quite some drawbacks like making compile-classpaths bigger than necessary and thus making compilation slower than necessary and decrease likelihood of task up-to-dateness or potential cache-hits.

That plugin recommends the proper way to declare your dependencies, but you can tell it by configuration about those “catch-all” dependencies and which are the transitive dependencies you actually use through adding it which you don’t declare explicitly something with bundle in dependencyAnalysis { structure { ... } }: Customizing plugin behavior · autonomousapps/dependency-analysis-gradle-plugin Wiki · GitHub

5

@Vampire Okay, then let’s do it that way. Thanks for your thorough explanations, I guess I am nore better suited to ask ‘proper’ questions … which will follow. :wink: