Gradle hangs indefinitely in testClasses (CI)


(steve) #1

On our CI environment, running our Gradle build hangs indefinitely on the testClasses task. The job is at http://ci.hibernate.org/job/hibernate-orm-master-h2

The job is configured to run ‘./gradlew clean test check :documentation:aggregateJavadocs publish’

I have tried a number of different Oracle JDK (7) versions to rule out JDK problems, including 7u21, 7u45, 7u51. No difference.

For one run I enabled strace to try to better see where it hangs. Unfortunately the yielded a 3.4G log file that is not very useful due to its sheer size.

I have also tried running Gradle with --debug and --info options. The problem is that using either of those causes the hang to not happen.

At this point I am completely out of ideas, but still the Gradle build hangs.

NOTE : I say it hangs in testClasses because that is the last line I see in the Jenkins build console:

:hibernate-core:runTestSourceGenerators UP-TO-DATE
:hibernate-core:compileTestJavaNote: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
  :hibernate-core:processTestResources
:hibernate-core:testClasses

Maybe it actually hangs trying to transition to the next task.


(Peter Niederwieser) #2

Can you provide a thread dump?


(steve) #3

How would I get a thread dump from a running Jenkins job?


(Peter Niederwieser) #4

In TeamCity you’d just press a button. Perhaps Jenkins offers something similar? An alternative would be to log in to the slave machine and run ‘jstack’.


(steve) #5

Also, which process? I generally see GradleWrapper and GradleWorker


(Peter Niederwieser) #6

Should be ‘GradleWrapperMain’.


(steve) #7

Is there a particular process/thread you want me to paste here? Or should I paste the whole dump?


(steve) #8

Right I had to use jstack. Jenkins does have a thread dumpl plugin but we have not installed it as of yet


(Peter Niederwieser) #9

GradleWrapperMain process, all threads. Please make it a GitHub Gist or something, and paste the link.


(steve) #10

That’s just asking for broken links later when people find these threads. But ok, whatever


(steve) #11

https://gist.github.com/sebersole/9979921


(Peter Niederwieser) #12

Thanks. Looks like the build is stuck during test execution. Can you also provide a dump of any ‘GradleWorkerMain’ process? Meanwhile, you could try to vary ‘tasks.withType(Test) { maxParallelForks = …; forkEvery = … }’ and see if this changes anything. Which Gradle version are you using?


(steve) #13

MMM, then why is there not a single reference to “org.hibernate” anywhere in any of these traces? Both dumps are there, scroll down


(Peter Niederwieser) #14

Looks like it’s stuck trying to shut down one of the test worker processes. Not sure why…


(steve) #15

What do I put for the “…”? I am using 1.9


(Peter Niederwieser) #16

e.g. ‘1’ and ‘100’ (in that order). Aren’t you setting these properties already?


(steve) #17

no, i dont touch these settings at all


(steve) #18

Is there anyway to specify these from command line so I can try that out on CI directly first?


(Peter Niederwieser) #19

Not without touching the build script.


(steve) #20

So setting maxParallelForks=1 and forkEvery=100 this now hangs locally on my laptop…