Hi,
we’re using Gradle 1.0-rc-2 with Jenkins (Linux 2.6.18-308.11.1.el5, JDK 1.6.0_31, all 64b).
The projects contains ~600 unit tests and when running them Gradle hangs. It’s not always the same test that hangs, 9 out of 10 builds hang when executing tests. --debug doesn’t give any errors, there are no exceptions, the system load ist not too high, memory is sufficient (looking at top) - so I don’t have any clue what’s wrong with this.
The same build skript runs fine on my local box (Win7), but that doesn’t help…
Has anybody experienced something like this and can give me a hint what to do?
Best regards Mark
I did the Gradle-Update yesterday, no change…
Now trying to sort out the tests …
hmmm, not that easy to “partition” tests with gradle… is there a way to do this using packages?
when I try it with -Dtest.single=de/my/package/ the sub-packages are not tested
What do you mean by “by including only a subset of tests in the test task”
I propably can use include/exclude in “test” in build.gradle, but that means I have to constantly check in versions of the build skript for Jenkins to use it…
Regards Mark.
Yes, I mean using ‘test.include’ and/or ‘test.exclude’. ‘-Dtest.single=de/my/package/**’ might work as well, but I haven’t checked.
ah, it’s
-Dtest.single=de/my/package/**/ (trailing slash!)
no reproducable behaviour - stripped it down to test runs with 20 - 30 tests each.
Some parts fail, then succeed when re-running them. Some succeed, then fail when re-running them.
Well…
Can you provide a sample project that allows to reproduce the problem, or at least some thread dumps?
Sample project is difficult, but for now I can provide a thread dump of the GradleWorker:
http://pastebin.com/3vgh35eq
Thanks for your help!
Mark.
What I can see is that Gradle is currently running ‘PhvTarifControllerTest.testNewContract’, and it doesn’t appear to be stuck. Please make sure you run with Gradle 1.3. Can you provide two thread dumps taken at the same time for worker and Gradle JVM?
Upgraded to Gradle 1.3, here are the dumps:
GradleWorker:
http://pastebin.com/EBXFwtEf
GradleMain:
http://pastebin.com/2GWDMEdx
Best Regards Mark.
Looks like the wrong dump for the Gradle side. Are you running with the Gradle daemon? In that case, use ‘–no-daemon’.
Used --no-daemon, new dumps:
though, the following Java-processes are running:
0 S jenkins
6899
1 11
76
0 - 904737 futex_ Nov19 ?
11:34:25 /usr/java/current_local/bin/java -Djava.awt.headless=true -Xmx2048m -XX:MaxPermSize=1024m -DJENKINS_HOME=/data/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cache/jenkins/war --httpPort=8080 --ajp13Port=8009 --debug=5 --handlerCountMax=100 --handlerCountMaxIdle=20
0 S jenkins
26659
6899
2
85
0 - 55180 futex_ 13:40 ?
00:00:07 /usr/java/current_prod/bin/java -Dorg.gradle.appname=gradle -classpath /usr/share/gradle/lib/gradle-launcher-1.3.jar org.gradle.launcher.GradleMain --info --stacktrace --no-daemon clean forTest build
0 S jenkins
26767 26659 13
85
0 - 309280 futex_ 13:40 ?
00:00:48 /usr/java/jdk1.6.0/bin/java -Xmx1024m -Dfile.encoding=ISO-8859-15 -cp /usr/share/gradle/lib/gradle-launcher-1.3.jar org.gradle.launcher.daemon.bootstrap.GradleDaemon 1.3 /var/lib/jenkins/.gradle/daemon 120000 a8f72a38-c88b-479e-bbd6-3730c8273797 -Xmx1024m -Dfile.encoding=ISO-8859-15
0 S jenkins
27250 26767 15
85
0 - 294345 futex_ 13:41 ?
00:00:52 /usr/java/jdk1.6.0/bin/java -Demma.coverage.out.file=/data/jenkins/workspace/SbS/build/tmp/emma/metadata.emma -Demma.coverage.out.merge=true -Djava.security.manager=jarjar.org.gradle.process.internal.child.BootstrapSecurityManager -XX:MaxPermSize=512m -XX:-UseSplitVerifier -Xmx512m -Dfile.encoding=ISO-8859-15 -ea -cp /var/lib/jenkins/.gradle/caches/1.3/workerMain/gradle-worker.jar jarjar.org.gradle.process.internal.launcher.GradleWorkerMain
GradleMain: http://pastebin.com/p8evAiuc
GradleWorker: http://pastebin.com/H2kJUC7R
GradleDaemon: (wouldn’t produce any dump… kill -3 didn’t give any result)
Gradle
We need the dump from the GradleDaemon process. Try with ‘jstack’.
jstack -F produces a Java crash report 
Attaching to process ID 2628, please wait...
#
# A fatal error has been detected by the Java Runtime Environment:
#
#
SIGSEGV (0xb) at pc=0x00002aaab1cd1e97, pid=5311, tid=1102846272
#
# JRE version: 6.0_31-b04
# Java VM: Java HotSpot(TM) 64-Bit Server VM (20.6-b01 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C
[libsaproc.so+0x1e97]
long double restrict+0x1f
#
# An error report file with more information is saved as:
# /root/hs_err_pid5311.log
#
# If you would like to submit a bug report, please visit:
#
http://java.sun.com/webapps/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
Abgebrochen
Not sure what to do then. Maybe upgrade your JDK, or don’t use compressed oops. All I can say is that judging from the ‘GradleWorker’ log, the test is running normally. And that we never encountered such an issue ourselves. Does the ‘GradleWorker’ process hang too? Can you provide a thread dump of that process at a time when all tests should have finished long ago?
It might be a problem with your test(s). This test thread is waiting to be notified:
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x94784fb0> (a com.dumbster.smtp.SimpleSmtpServer)
at java.lang.Object.wait(Object.java:485)
at com.dumbster.smtp.SimpleSmtpServer.start(Unknown Source)
- locked <0x94784fb0> (a com.dumbster.smtp.SimpleSmtpServer)
at de.aspect.sbs.phv.PhvServiceTest$1.before(PhvServiceTest.java:353)
Maybe ‘SimpleSmtpServer’ isn’t working correctly when run on the Jenkins box.
thanks a lot for your time, I will look into this today.
Regards Mark.