Incremental Java compilation is at least 4 times slower than normal compilation


(Trejkaz (pen name)) #1

I’m used to our whole build taking about 20 minutes to run check -x test.

I thought I’d try to turn on incremental compilation to speed things up - now it has been 25 minutes, but it’s only 25% complete. This is on Gradle 4.5, but I’m grabbing 4.6 next to see whether it’s any better.

The output is full of:

Expiring Daemon because JVM Tenured space is exhausted
Expiring Daemon because JVM Tenured space is exhausted
Expiring Daemon because JVM Tenured space is exhausted
Expiring Daemon because JVM Tenured space is exhausted
Expiring Daemon because JVM Tenured space is exhausted
Expiring Daemon because JVM Tenured space is exhausted
Expiring Daemon because JVM Tenured space is exhausted
Expiring Daemon because JVM Tenured space is exhausted
Expiring Daemon because JVM Tenured space is exhausted
Expiring Daemon because JVM Tenured space is exhausted
<===----------> 24% EXECUTING [27m 53s]
> :storage-core:compileJava

(James Justinic) #2

Your build is starved for memory, so you’re losing a lot of time to garbage collection. This is bad for performance in any case, but keep in mind that incremental compilation also isn’t free. To avoid recompiling code that doesn’t need to be recompiled, a model has to be built to determine what does need to be recompiled. Turning on incremental compilation isn’t going to help you unless you can provide enough resources, which is going to be more than without it. It’s far worse if you can get very little real work done between each GC.


(Trejkaz (pen name)) #3

I’m not the one who chose the amount of memory for the daemon though - Gradle did that itself.


(James Justinic) #4

Gradle doesn’t choose the amount of memory. If you don’t choose, you get the JVM default, just like any JVM application. The JVM has no intelligence about what you’re trying to run, or what its memory requirements might be.


(Trejkaz (pen name)) #5

Gradle most certainly did choose it.

tester 44270 322.3  1.6  6942496 266240   ??  Rs   12:21pm   0:05.30 /Users/tester/DevEnv/Java/jdk1.8.0_92.jdk/Contents/Home/bin/java -XX:+HeapDumpOnOutOfMemoryError -Xmx1024m -Dfile.encoding=UTF-8 -Duser.country=AU -Duser.language=en -Duser.variant -cp /Users/tester/.gradle/wrapper/dists/gradle-4.5-all/cg9lyzfg3iwv6fa00os9gcgj4/gradle-4.5/lib/gradle-launcher-4.5.jar org.gradle.launcher.daemon.bootstrap.GradleDaemon 4.5

(James Justinic) #6

This message suggests that you believe that Gradle chose a memory value, and that the value should be sufficient because the tool chose it. Chose suggests there was a choice. There’s no choice (unless you choose), only a default.

The default memory for a JVM application is typically the smaller of 1/4th system memory or 1024m. On modern hardware, this should usually be 1024m, but there are also cases where 1/4th might be smaller, like in a container / virtualized environment. The Gradle daemon defaults to 1024m, matching the usual effective JVM default, but avoiding situations where 1/4th could be too small to work in any case.

The point is that you need to choose a memory value that is sufficient for your build, and any enabled options, if the default is not. You should not expect the tool to “choose” a sufficient value for you.


(Trejkaz (pen name)) #7

Okay, so from some testing, Java, if you don’t tell it what to use, uses 1/4 of the system memory. For my machine (the oldest dev box, I think), that comes to what looks like about 4GiB. This is on Java 8, at least, Oracle JVM, providing no other flags.

But instead of letting it use the default, Gradle chose to reduce it to 1024MiB. You can complain about semantics all you want, but the fact of the matter is that some developer at Gradle made that choice, and I suffered for it.

So after digging around a bit, I found the place to raise the memory back to 4GB.

Now the incremental build takes 35 minutes, so it’s still 75% slower than not doing it. So I guess Gradle’s incremental compilation is just slow then? Or does it need more than 4GB of RAM? How can I, as the end user, determine how much memory I’m supposed to set for this tool? Given that the only people who would know how much memory a tool uses are the people who made it, Gradle should be responsible for setting this correctly in the first place, not me.


(James Justinic) #8

You’re setting memory for your build, not specifically the tool itself. If you can’t determine acceptable settings through basic observation, profile your build just as you would when tuning memory settings for any JVM application. It shouldn’t be unreasonable for a Java developer to do this for the application and development tools used.

This is an unreasonable expectation. Memory requirements fluctuate with what you’re building and how you’re building it. Your build can have dramatically different memory requirements compared to other builds. Plugins used, code built, and options enabled all impact this. You’re the one in the position to know if the defaults are not working for you, and observe how various settings impact your build.

Defaults change over time, vary based on resources available, and are impacted by things unrelated to the actual work that needs to be done. If you want to use the defaults, no matter what you’re trying to do, your experience isn’t going to be good.


(Trejkaz (pen name)) #9

I think I’ll just add the memory problem to the large pile of “things that make Gradle not really a declarative build system”. It’s not like it’s the only problem with Gradle, it’s like the tip of a rather large shit heap that extends to some distance underground.

The real question here was about incremental build being this much slower than the normal build. If it’s going to take more memory and still run slower, I guess it’s just something we can’t turn on and will wait until the feature is ready for public use.


(stdoubleu) #10

The way you arrogantly talk,
the way you ignore James’ patient answers,
the way you don’t understand the TOOL gradle,
the way you don’t WANT to understand how the tool gradle works,
leads me to the conclusion that you are not a developer.
And as such you shouldn’t use development tools.
Because usually development tools are complex and not “one-button”-applications.

But I will bring it back to a real-life example for you. If you use a tool (lets call it “chainsaw”) most likely you will think about how to hold it before you turn it on.


(Trejkaz (pen name)) #11

I hardly ignored the answers, I replied to them. I didn’t actually get an answer about the original question at all, though.


(James Justinic) #12

The original question stated that you were observing slow build times while seeing many occurrences of Expiring Daemon because JVM Tenured space is exhausted. Seeing this message, especially repeatedly, indicates memory trouble. You need to adjust memory settings to at least avoid this message, although you might not discover the best possible settings without some profiling. I understand that you don’t think you should have to do this, but that’s the nature of optimizing the JVM memory settings for your specific case.

Incremental compilation is not a silver bullet. It adds up front overhead with the goal that you can do much less work on subsequent runs. There are projects where it is easy to see the benefit of incremental compilation, but there are also projects where the overhead will always outweigh the benefit.

I don’t know what you’re building, but it seems that it is either quite massive or there’s some other problem that needs to be addressed as well. If it’s a huge legacy app, there might be enough tangle that you can only lose time from incremental compilation. If it’s something else and you have a way to reproduce the performance you’re seeing or describe some items that seem to make the incremental compilation much worse in your case, then there’s a chance someone can help. If not, there’s not much more that can be said other than adjust your memory settings and don’t enable the feature if it doesn’t help.


(Mark Anderson) #13

Ok let ,e put my 2 bobs worth in here. 63yo with 43 years of IBM Mainframe and top end IT, for what it’s worth.

My experience to date is the Gradle is buggy and frankly a mess. It would be great if it was not a “thing” and could me bypassed completely. It causes massive pain between versions, apps suddenly stop working and so on. Ridiculous.

And @stdoubleu You are way off with that remark. @trejkaz was spot on, succinct and quite justified in his points. In fact it is James who us splitting hairs and weaving around the place.

Bottom line, Gradle needs to mature as a product and team, produce a product that does not regularly fall over with nonsensical messages, and not operate like Google with their ongoing deprecated code versions.


(Eric Wendelin) #14

@trejkaz One can configure the Gradle client memory and Gradle Daemon memory separately. Here is some docs that should show you how to configure it. Gradle cannot predict how much memory a build will consume, but we give you some tools to show you how much memory is used. Incremental compilation requires a lot more caching, thus more memory.

@jjustinic Thank you for helping to answer this and many other questions, sir.

@stdoubleu Attacking someone personally as you have violates our Code of Conduct. Here we shall focus on attacking only technical problems, and we happily support anyone with honest Gradle questions. You have been warned.

I’m shutting this topic down so everyone can cool off.

Cheers,
Eric


(Eric Wendelin) #15