Composite build hangs after upgrade to Gradle 4.0

Hi,

I’m using a composite build, where the local version of a library is used when available (i.e. in the development environment), but the published version of the library is used when running the build from the CI server. This was working fine when using Gradle 3.x, but after upgrading to Gradle 4.0 the build hangs with the following output:

<========-----> 61% EXECUTING [1m 33s]
> :compileJava > Resolve files of :compileClasspath
> :Colorize Java Commons:compileJava > Resolve files of :Colorize Java Commons:compileClasspath

The included build is a library that can be found on GitHub, the build.gradle file can be found here: https://github.com/colorizenl/colorize-java-commons/blob/master/build.gradle

The project that includes the library is not on GitHub, but the structure of the build.gradle file is similar. The contents of settings.gradle, including the setup of the composite build, is as follows:

rootProject.name = 'Dashboard'

if (file('../Colorize Java Commons').exists()) {
    includeBuild('../Colorize Java Commons') {
	    dependencySubstitution {
		    substitute module('nl.colorize:colorize-java-commons') with project(':')
	    }
    }
}

From looking at the release notes and the topics on this forum it doesn’t seem that anyone else is encountering this problem, so we’re probably doing something unusual here. However, based on the documentation this shouldn’t be considered as “wrong”, so I do think the hanging build is a bug, and the build should either succeed or fail with an error.

Same behaviour here using Gradle 4.0.1. Works fine in 3.5.

@colorizenl Could you collect a stacktrace from the Gradle daemon while the build is hanging?

If you’re on linux, you can use kill -QUIT <pid>.

Environment:
------------------------------------------------------------
Gradle 4.0.1
------------------------------------------------------------

Build time:   2017-07-07 14:02:41 UTC
Revision:     38e5dc0f772daecca1d2681885d3d85414eb6826

Groovy:       2.4.11
Ant:          Apache Ant(TM) version 1.9.6 compiled on June 29 2015
JVM:          1.8.0_131 (Oracle Corporation 25.131-b11)
OS:           Linux 4.11.9-1-ARCH amd64

Command Line

./gradlew --include-build /home/sray/src/idexx/git/gradle-plugins/ReleaseVersions

Stack Trace

Starting a Gradle Daemon, 1 busy Daemon could not be reused, use --status for details
[buildinfo] Not using buildInfo properties file for this build.
Build scan data will not be captured due to this build being a Composite Build.
:ReleaseVersions:compileJava NO-SOURCE
:ReleaseVersions:compileGroovy2017-07-12 08:36:46
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.131-b11 mixed mode):

"forward input" #17 prio=5 os_prio=0 tid=0x00007fc6a8e94800 nid=0x744e waiting on condition [0x00007fc68b3eb000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000076d0df9a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at org.gradle.util.DisconnectableInputStream.read(DisconnectableInputStream.java:138)
	at org.gradle.launcher.daemon.client.InputForwarder$1.run(InputForwarder.java:77)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
	at org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:46)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
	at java.lang.Thread.run(Thread.java:748)

"DisconnectableInputStream source reader" #16 daemon prio=5 os_prio=0 tid=0x00007fc6a8e94000 nid=0x744d runnable [0x00007fc68b626000]
   java.lang.Thread.State: RUNNABLE
	at java.io.FileInputStream.readBytes(Native Method)
	at java.io.FileInputStream.read(FileInputStream.java:255)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
	at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
	- locked <0x0000000770afe180> (a java.io.BufferedInputStream)
	at org.gradle.util.DisconnectableInputStream$1.run(DisconnectableInputStream.java:96)
	at java.lang.Thread.run(Thread.java:748)

"process reaper" #12 daemon prio=10 os_prio=0 tid=0x00007fc628006800 nid=0x742e runnable [0x00007fc68b525000]
   java.lang.Thread.State: RUNNABLE
	at java.lang.UNIXProcess.waitForProcessExit(Native Method)
	at java.lang.UNIXProcess.lambda$initStreams$3(UNIXProcess.java:289)
	at java.lang.UNIXProcess$$Lambda$9/210830465.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)

"Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007fc6a80d3000 nid=0x73f6 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007fc6a80c7800 nid=0x73f5 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007fc6a80c5800 nid=0x73f4 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fc6a80c4000 nid=0x73f3 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fc6a80c1000 nid=0x73f2 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fc6a80bd000 nid=0x73f1 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fc6a808c000 nid=0x73f0 in Object.wait() [0x00007fc6915b2000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x0000000770a80d08> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
	- locked <0x0000000770a80d08> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fc6a8087800 nid=0x73ef in Object.wait() [0x00007fc6916b3000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x0000000770a88690> (a java.lang.ref.Reference$Lock)
	at java.lang.Object.wait(Object.java:502)
	at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
	- locked <0x0000000770a88690> (a java.lang.ref.Reference$Lock)
	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

"main" #1 prio=5 os_prio=0 tid=0x00007fc6a8009800 nid=0x73e5 runnable [0x00007fc6afeb4000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	- locked <0x000000076cf7a3d8> (a sun.nio.ch.Util$3)
	- locked <0x000000076cf7a350> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000076cf79f88> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
	at org.gradle.internal.remote.internal.inet.SocketConnection$SocketInputStream.read(SocketConnection.java:178)
	at com.esotericsoftware.kryo.io.Input.fill(Input.java:139)
	at com.esotericsoftware.kryo.io.Input.require(Input.java:159)
	at com.esotericsoftware.kryo.io.Input.readInt(Input.java:308)
	at org.gradle.internal.serialize.kryo.KryoBackedDecoder.readSmallInt(KryoBackedDecoder.java:120)
	at org.gradle.internal.serialize.DefaultSerializerRegistry$TaggedTypeSerializer.read(DefaultSerializerRegistry.java:139)
	at org.gradle.internal.serialize.Serializers$StatefulSerializerAdapter$1.read(Serializers.java:36)
	at org.gradle.internal.remote.internal.inet.SocketConnection.receive(SocketConnection.java:79)
	at org.gradle.launcher.daemon.client.DaemonClientConnection.receive(DaemonClientConnection.java:75)
	at org.gradle.launcher.daemon.client.DaemonClientConnection.receive(DaemonClientConnection.java:35)
	at org.gradle.launcher.daemon.client.DaemonClient.monitorBuild(DaemonClient.java:210)
	at org.gradle.launcher.daemon.client.DaemonClient.executeBuild(DaemonClient.java:178)
	at org.gradle.launcher.daemon.client.DaemonClient.execute(DaemonClient.java:141)
	at org.gradle.launcher.daemon.client.DaemonClient.execute(DaemonClient.java:92)
	at org.gradle.launcher.cli.RunBuildAction.run(RunBuildAction.java:51)
	at org.gradle.internal.Actions$RunnableActionAdapter.execute(Actions.java:173)
	at org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.execute(CommandLineActionFactory.java:287)
	at org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.execute(CommandLineActionFactory.java:260)
	at org.gradle.launcher.cli.JavaRuntimeValidationAction.execute(JavaRuntimeValidationAction.java:33)
	at org.gradle.launcher.cli.JavaRuntimeValidationAction.execute(JavaRuntimeValidationAction.java:24)
	at org.gradle.launcher.cli.ExceptionReportingAction.execute(ExceptionReportingAction.java:33)
	at org.gradle.launcher.cli.ExceptionReportingAction.execute(ExceptionReportingAction.java:22)
	at org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:253)
	at org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:182)
	at org.gradle.launcher.Main.doAction(Main.java:33)
	at org.gradle.launcher.bootstrap.EntryPoint.run(EntryPoint.java:45)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.gradle.launcher.bootstrap.ProcessBootstrap.runNoExit(ProcessBootstrap.java:60)
	at org.gradle.launcher.bootstrap.ProcessBootstrap.run(ProcessBootstrap.java:37)
	at org.gradle.launcher.GradleMain.main(GradleMain.java:23)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.gradle.wrapper.BootstrapMainStarter.start(BootstrapMainStarter.java:31)
	at org.gradle.wrapper.WrapperExecutor.execute(WrapperExecutor.java:108)
	at org.gradle.wrapper.GradleWrapperMain.main(GradleWrapperMain.java:61)

"VM Thread" os_prio=0 tid=0x00007fc6a807f800 nid=0x73ee runnable 

"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fc6a801f000 nid=0x73e6 runnable 

"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fc6a8020800 nid=0x73e7 runnable 

"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007fc6a8022800 nid=0x73e8 runnable 

"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007fc6a8024000 nid=0x73e9 runnable 

"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007fc6a8026000 nid=0x73ea runnable 

"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007fc6a8027800 nid=0x73eb runnable 

"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007fc6a8029800 nid=0x73ec runnable 

"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007fc6a802b000 nid=0x73ed runnable 

"VM Periodic Task Thread" os_prio=0 tid=0x00007fc6a80d5800 nid=0x73f7 waiting on condition 

JNI global references: 291

Heap
 PSYoungGen      total 74752K, used 15110K [0x000000076cb80000, 0x0000000771e80000, 0x00000007c0000000)
  eden space 64512K, 10% used [0x000000076cb80000,0x000000076d20cfb0,0x0000000770a80000)
  from space 10240K, 82% used [0x0000000770a80000,0x00000007712b4b60,0x0000000771480000)
  to   space 10240K, 0% used [0x0000000771480000,0x0000000771480000,0x0000000771e80000)
 ParOldGen       total 171008K, used 16K [0x00000006c6200000, 0x00000006d0900000, 0x000000076cb80000)
  object space 171008K, 0% used [0x00000006c6200000,0x00000006c6204000,0x00000006d0900000)
 Metaspace       used 12275K, capacity 12648K, committed 12928K, reserved 1060864K
  class space    used 1667K, capacity 1828K, committed 1920K, reserved 1048576K

Thanks @Seoras, but that’s the thread dump from the Gradle client JVM that talks to the daemon.

If you run jps, you should see something like:

92064 GradleDaemon
92371 Jps
92354 GradleWrapperMain
91190 Launcher

The thread dump from the process called GradleDaemon is the important one.

Sure. I had to put it online due to this forum having a limit on how much text you can post: http://www.colorize.nl/dev/stacktrace.txt

I experienced the same issue today. Only slightly different. I had a running composite build and then suddenly it stopped working with the exact same picture: the build was stuck at Resolve files of :...:compileClasspath from the included build project. I am using Gradle 4.0.1.

The only real difference was that I had added one additional dependency to my build:

compile 'org.apache.commons:commons-lang3:3.6'

When I removed the dependency, everything worked again. Funny enough, after I moved this dependency declaration after the included / replaced project dependency things also worked again!

Sounds crazy, I know. Maybe this helps.

I’m pretty sure there’s something racy going on with parallel dependency resolution and composite build execution.

If you catch it again, please try to collect a thread dump.

We’ve revamped a lot of this in 4.1, so I’m trying to reproduce it in both 4.0 and the latest code.

Hey Sterling. Luckily I can reproduce the behaviour pretty easy. :slight_smile:

Please find the requested stack trace of the GradleDaemon here:

Thanks @lreimer.

I’ve not been able to reproduce this with my dummy project, but this code is very different in the next release. Could you give a nightly a try?

Update your wrapper with:
./gradlew wrapper --gradle-version=4.1-20170714213039+0000

Or use distribution:
https://services.gradle.org/distributions-snapshots/gradle-4.1-20170714213039+0000-bin.zip

Quick info: Looks good. I was unable to reproduce the problem with the nightly wrapper version you supplied.

Thanks for checking it out!

For me the issue is also fixed when using the nightly build. Thanks a lot guys!

Just ran into this issue with composite builds in Intellij IDEA 2017.3 EAP with gradle 4.0.2. The build of the main project with composite dependencies enabled was consistently hanging resolving files of the compileClasspath. I thought it was an Intellij bug at first since they have made many changes to that part of the IDE recently.

I upgraded to gradle v4.2.1 after reading this thread and the problem is gone. Thanks for reporting this issue. I wasted a good bit of time troubleshooting and hope others with the same problem are as fortunate and find this thread.