Gradle hangs and CPU at 100%


(Ganesh) #1

I have a huge project say around 28 maven modules. We are evaluating Gradle as we have huge build time with maven. I did gralde init and then gradle build even after couple of hours it kept running and when I run in debug mode (–debug) I see the following message in the logs file repeatedly

16:32:15.388 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired.
16:32:15.390 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.
16:32:15.391 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon

Is there a way I can troubleshoot this further and get more details on what is hapeening ?. I am on windows 7; Running through command line with Java 8.


(Ganesh) #2

Around 280 maven modules and not 28.

Any though would be appreciated.


(Eric Wendelin) #3

Hi Ganush,

Hm, it looks like there’s deadlock occurring. Sorry to hear this.

Some initial questions for you:
What Gradle version are you using?
Are you using any Gradle plugins? Which ones?
What Gradle options are you using? For example, are you using --parallel, --daemon, or others?
What do the --debug logs say just before they repeat the lock acquired/released messages?

Any other details you can share about your build or environment would be helpful.

Cheers,
Eric


(Ganesh) #4

What Gradle version are you using?
I am using gradle 3.1.
Are you using any Gradle plugins? Which ones?
I am not using any special plugin mine was a maven project and trying to converting to gradle by default the below are added before the repositories, may be maven plugin could be culprit ?.
allprojects {
apply plugin: ‘maven’

group = 'com.sabre.apd.crewmanager’
version = ‘1.0-SNAPSHOT’
}
apply plugin: 'java’
compileJava {
options.fork = true // Fork your compilation into a child process
options.forkOptions.setMemoryMaximumSize(“4g”) // Set maximum memory to 4g
}

What Gradle options are you using? For example, are you using --parallel, --daemon, or others?
Not using any special options.

What do the --debug logs say just before they repeat the lock acquired/released messages?
13:00:05.013 [DEBUG] [org.gradle.model.internal.registry.DefaultModelRegistry] Project :cmx-rv:cmx-rv-openpairings - Transitioning model element ‘tasks.check’ to state Initialized.
13:00:05.013 [DEBUG] [org.gradle.model.internal.registry.DefaultModelRegistry] Project :cmx-rv:cmx-rv-openpairings - Registering model element ‘tasks.jar’ (hidden = false)
13:00:05.014 [DEBUG] [org.gradle.model.internal.registry.DefaultModelRegistry] Project :cmx-rv:cmx-rv-openpairings - Registering model element ‘tasks.install’ (hidden = false)
13:00:12.371 [DEBUG] [org.gradle.launcher.daemon.server.Daemon] DaemonExpirationPeriodicCheck running

Any other details you can share about your build or environment would be helpful.

Java 1.8; Originally a maven project build with Maven 3.0.5.

Are there any other way to debug/trace what is going on ?. May be to the level during what stage it is hanging ?


(Ganesh) #5

Build is on windows platform.


(Eric Wendelin) #6

Thus far I don’t see anything amiss from a few DEBUG log lines.

Could you share which task is hanging?

Do you really mean Gradle 3.1? That’s not released yet. If that is so, you should consider using the latest stable version, 3.0.


(Lari Hotari) #7

If it’s simply stalled and you don’t get log output, I’d recommend using the jstack command line tool to get a thread dump of the Gradle daemon process since that’s a simple thing to do.

You can also use other standard JVM tools like VisualVM (jvisualvm) to get some information about what’s going on. The Oracle JDK comes with Java Mission Control / Java Flight Recorder bundled in it since 1.7.0_40 . The jmc command opens a GUI where you can choose the Java process to profile. You have to accept the license terms to use JMC/JFR. It’s a commercial product that needs a paid license for production use. Recording the execution for some time (1-2 minutes) will give a very accurate view of what is going on under the covers.

For large builds, you have to tune the memory settings and give Gradle enough memory. This is usual JVM GC tuning. Giving too much memory might lead to other problems. JMC/JFR shows extensive GC metrics


(Lari Hotari) #8

Is there a particular reason for using forked compilation? Currently Gradle isn’t very resource efficient when --parallel is used together with forked compilation. The number of forked compiler processes cannot be controlled separately.


(Ganesh) #9

There is no specific reason to fork, when I searched for similar problem in internet someone tried and worked for them. In my case the CPU is going 100% which is a major concern. I will try with JVIsual VM. Also, when I did the other smaller projects gradle init/gradle build it went well. Only this bigger project has this problem.

Thanks for all the guidance I will get some more metrics and see what is going on.


(Ganesh) #10

I tried with out fork and also gave more heap size still the issue persists. I think there is some problem with the graldle daemon as it was showing the spike (100% CPU and also memory usage). I tried to put the screen shot of Jvisual VM when the build is happening, I am not upload more than one images because I am a new user :slight_smile: )

Any other options ?.I also tried on all the version it remains the same (2.x to 3.x)


(Lari Hotari) #11

@itsganu Are you familiar with jstack? I found Redhat’s instructions for using jstack on Windows. Since you are using Windows those instructions might be helpful, for others on unixes man jstack and man jps contain useful information about using jstack from the command line.
The output of 3 subsequent jstack calls made a few seconds apart would be helpful.
Instead of uploading/pasting the files to this forum, it’s preferred to upload the content to https://gist.github.com and just add a link to the forum.

The jstack thread dump reveals where the JVM threads are executing at the moment the dump was created. Make sure you pick the correct GradleDaemon process id from the jps listing.

To get much better understanding, I’d recommend creating a Java Flight Recorder recording for at least 1-2 minutes of execution. Please be aware that the JFR log file reveals some environment details (process list and environment variables) by default so don’t share it publicly. It’s possible to disable the process list and environment variable listing by using certain settings. The JFR profile with less environment details can be imported to JMC/JFR in the “Template Manager” dialog when starting a new JFR recording in the JMC UI. If you want to share a JFR log with me, use these settings and share the zipped JFR log file with me by sending it by email to lari _a_t gradle com .