Could not start compiler daemon - could not reserve space for object heap

Hi. I get this problem all the time:

:core:compileGroovy Executing task ‘:core:compileGroovy’ due to: No history is available for task ‘:core:compileGroovy’. Starting Gradle compiler daemon. Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Error occurred during initialization of VM Could not reserve enough space for object heap

How much does this compiler daemon actually require? That’s the output right before invoking gradle:

$ free -m

total

used

free

shared

buffers

cached Mem:

8192

2229

5962

0

0

0 -/+ buffers/cache:

2229

5962 Swap:

0

0

0

which means there is almost 6gb of free ram.

My build has org.gradle.daemon=true set in gradle.properties. The compilation seems to start yet another daemon. When I turn off the gradle daemon, it still fails. Here is a part of the log right before it dies (I leave out a log of info, mainly classpaths):

22:51:54.355 [INFO] [org.gradle.api.internal.tasks.compile.daemon.CompilerDaemonManager] Starting Gradle compiler daemon. 22:51:54.367 [DEBUG] [org.gradle.api.internal.tasks.compile.daemon.CompilerDaemonManager] DaemonForkOptions{minHeapSize=null, maxHeapSize=null, jvmArgs=[], classpath=…

My Java: $ java -version java version “1.7.0_05” Java™ SE Runtime Environment (build 1.7.0_05-b05) Java HotSpot™ 64-Bit Server VM (build 23.1-b03, mixed mode)

Does anybody have any tips?

wujek

Compiler daemon is not the same as Gradle daemon. Anyway, if you don’t explicitly specify heap options for compilation, the JVM’s defaults will be used, which are typically fairly low.

Does this make a difference?

compileGroovy.groovyOptions.forkOptions.memoryInitialSize = “64m”

compileGroovy.groovyOptions.forkOptions.memoryMaximumSize = “64m”

Which OS?

It’s a 64bit debian. Well, default Java options are not low at all… At this machine, every new JVM swallows over 2GB:

$ java -XX:+PrintFlagsFinal -version | grep ‘(Initial|Max)HeapSize’

uintx InitialHeapSize

:= 134217728

{product}

uintx MaxHeapSize

:= 2147483648

{product}

Even jps: $ jps -J-XX:+PrintFlagsFinal | grep ‘(Initial|Max)HeapSize’

uintx InitialHeapSize

:= 8388608

{product}

uintx MaxHeapSize

:= 2147483648

{product}

And yes, changing the -Xmx settings do make a difference. This would mean, that at the time my groovy code gets compiled (after java), there are at least 3 JVMs? One for gradle, one for compiling java, and one for groovy (which fails to start). Is that right? Are the daemons reused for compiling code in other projects as well, or does, say, every compileJava / compileJava start a new daemon?

Is it necessary for gradle to fork? How big would you suggest Xmx is for the daemon, and for the compilers?

=====

Read below is you are interested in my findings.

The ‘problem’ is the following: this machine is what this document: http://docs.oracle.com/javase/7/docs/technotes/guides/vm/server-class.html calls a ‘server class machine’, that’s why it starts ith with the -server option on. Also, here: http://docs.oracle.com/javase/7/docs/technotes/guides/vm/gc-ergonomics.html it says, that the Xmx is set to smaller of 1/4 or 1gb of ram. At this particular machine, as shown above, it somehow uses the 1/4 (~2gb), which means: every new JVM uses -Xmx2g, no wonder that it fails at the third JVM (I also have tomcat 7 running, which also takes its share… And there are no JAVA_OPTS or whatever used. That’s very strange.

What is also strange (IMHO), is that even though initial heap is about 134m in the above case, the JVM will not start when there is no Xmx mem available. It seems it just allocated everything it can, instead of doing it only when needed.

During my searches, I found something interesting: the _JAVA_OPTIONS env variable. Check this out: $ export _JAVA_OPTIONS="-Xmx512m" $ jps -J-Xmx4g -J-XX:+PrintFlagsFinal | grep ‘(Initial|Max)HeapSize’ Picked up _JAVA_OPTIONS: -Xmx512m

uintx InitialHeapSize

:= 8388608

{product}

uintx MaxHeapSize

:= 536870912

{product}

I export it to set Xmx to 512m, and then, even though I start jps with -Xmx4g, it still uses 512. Note also the ‘Picked up _JAVA_OPTIONS’ part. This is a highly obscure, undocumented feature of HotSpot, there is very little in the net about it, but this setting forces every JVM to use no more than what it says. I wonder what other JVMs do.

It gets even more interesting: $ export JAVA_TOOL_OPTIONS="-Xmx256m" jps -J-Xmx4g -J-XX:+PrintFlagsFinal | grep ‘(Initial|Max)HeapSize’ Picked up JAVA_TOOL_OPTIONS: -Xmx256m Picked up _JAVA_OPTIONS: -Xmx512m

uintx InitialHeapSize

:= 8388608

{product}

uintx MaxHeapSize

:= 536870912

{product} Although JAVA_TOOL_OPTIONS is documented in JTVMI documentation. But _JAVA_OPTIONS still has priority.

wujek

Above, it should be

_JAVA_OPTIONS

, the software ate the leading _.

OK, I looked into it, and its a problem in the virtual environment that the platform uses. It tries to allocate the whole Xmx, instead of just a small initial fraction. Anyways, I now got it all to work.

To answer your other question, the compiler daemon currently only stays around for one build, and will be reused as much as possible (between tasks and also between Java and Groovy compilation). If a new compiler daemon needs to be started (e.g. due to some task having a higher ‘memoryMaximumSize’ setting), any existing compiler daemon will first be shut down.

My observations show:

  1. compileJava doesn’t start any daemon at all, it uses Java6 compiler API directly from the same process, I see that in debug log (stripped down to the bare minimum):

20:00:42.708 [DEBUG] [org.gradle.api.internal.tasks.compile.NormalizingJavaCompiler] Compiler arguments: -d /var/tomcat/jenkins_data/workspace/gradletest/api/build/classes/main -deprecation -g:source,lines,vars -encoding UTF-8 -classpath 20:00:42.708 [INFO] [org.gradle.api.internal.tasks.compile.jdk6.Jdk6JavaCompiler] Compiling with JDK 6 Java compiler API. 20:00:43.170 [DEBUG] [org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter] Finished executing task ‘:api:compileJava’

  1. compileGroovy starts a daemon, and reuses it for tasks in the same project, and for multiple projects (very nice):

20:00:44.530 [INFO] [org.gradle.api.internal.tasks.compile.daemon.CompilerDaemonManager] Starting Gradle compiler daemon. 20:00:44.546 [DEBUG] [org.gradle.api.internal.tasks.compile.daemon.CompilerDaemonManager] DaemonForkOptions{minHeapSize=null, maxHeapSize=null, jvmArgs=[], classpath=]}

  1. tests also start a daemon, and it seems to also be reused for multiple projects (but I have tests in a single module yet, so I might be wrong):

20:00:47.399 [DEBUG] [org.gradle.process.internal.ProcessBuilderFactory] creating process builder for Gradle Worker 2 20:00:47.399 [DEBUG] [org.gradle.process.internal.ProcessBuilderFactory] in directory /var/tomcat/jenkins_data/workspace/gradletest/core 20:00:47.400 [DEBUG] [org.gradle.process.internal.ProcessBuilderFactory] with argument#0 = -Djava.security.manager=org.gradle.process.internal.child.BootstrapSecurityManager 20:00:47.400 [DEBUG] [org.gradle.process.internal.ProcessBuilderFactory] with argument#1 = -Xmx128m 20:00:47.400 [DEBUG] [org.gradle.process.internal.ProcessBuilderFactory] with argument#2 = -Dfile.encoding=US-ASCII 20:00:47.400 [DEBUG] [org.gradle.process.internal.ProcessBuilderFactory] with argument#3 = -ea 20:00:47.400 [DEBUG] [org.gradle.process.internal.ProcessBuilderFactory] with argument#4 = -cp 20:00:47.400 [DEBUG] [org.gradle.process.internal.ProcessBuilderFactory] with argument#5 = /var/tomcat/.gradle/caches/1.0/workerMain/classes 20:00:47.400 [DEBUG] [org.gradle.process.internal.ProcessBuilderFactory] with argument#6 = org.gradle.process.internal.launcher.GradleWorkerMain 20:00:47.405 [DEBUG] [org.gradle.process.internal.DefaultExecHandle] Started Gradle Worker 2.

(the args, -Xmx128m are actually configured by me)

Can you confirm this?

wujek

ad 1/2. Both ‘Compile’ and ‘GroovyCompile’ tasks will use (and share) a compiler daemon if run in fork mode. The difference is that ‘Compile.options.fork’ defaults to ‘false’, whereas ‘GroovyCompile.groovyOptions.fork’ defaults to ‘true’.

ad 3. Each ‘Test’ task uses one or more worker processes for executing tests.

Yes, I see it in the documentation now. But thanks for your confirmation of my observations.

Regarding, tests, I use the default, which is a single worker. Does it get reused, or each test task gets its own worker(s)?

wujek

Why not just try? Each test tasks uses its own processes.