Why is gradle dependency resolution so slow?

Old Forum
1

Running the task “foo” in the file below takes in excess of 90 seconds. I intend to be writing a task which uses this information to further perform some processing, but the amount of time this takes is making it virtually impossible to use gradle at the moment.

Note: I thought I was running gradle m6, but instead find myself running m5. (I am a ubuntu user, but for the moment am stuck at a mac and don’t quite know how to change versions using brew :frowning: )

$ gradle --version

------------------------------------------------------------ Gradle 1.0-milestone-5 ------------------------------------------------------------

Gradle build time: Tuesday, October 25, 2011 9:26:08 AM GMT+05:30 Groovy: 1.7.10 Ant: Apache Ant™ version 1.8.2 compiled on December 20 2010 Ivy: 2.2.0 JVM: 1.6.0_29 (Apple Inc. 20.4-b02-402) OS: Mac OS X 10.7.2 x86_64

build.gradle

apply plugin: ‘java’ apply plugin: ‘scala’ apply plugin: ‘osgi’

repositories {

mavenLocal()

mavenCentral()

maven {

url = ‘http://scala-tools.org/repo-releases’

}

maven {

url = ‘http://repo.typesafe.com/typesafe/releases’

}

maven {

url = ‘http://repository.codehaus.org/org/codehaus’

} }

dependencies {

scalaTools ‘org.scala-lang:scala-compiler:2.9.1’

scalaTools ‘org.scala-lang:scala-library:2.9.1’

compile ‘org.scala-lang:scala-library:2.9.1’

compile ‘org.scala-lang:scala-compiler:2.9.1’

compile ‘se.scalablesolutions.akka:akka-actor:1.3-RC2’

compile ‘se.scalablesolutions.akka:akka-remote:1.3-RC2’

compile ‘com.weiglewilczek.slf4s:slf4s_2.9.1:1.0.7’

compile ‘ch.qos.logback:logback-classic:1.0.0’

compile ‘org.osgi:org.osgi.core:4.1.0’

compile ‘org.osgi:org.osgi.compendium:4.2.0’

testCompile ‘junit:junit:4.5’

testCompile ‘org.specs2:specs2_2.9.1:1.6.1’

testCompile ‘org.specs2:specs2-scalaz-core_2.9.1:6.0.1’

testCompile ‘se.scalablesolutions.akka:akka-testkit:1.3-RC2’ }

jar {

manifest {

// deleted

} }

task foo << {

configurations.compile.each { println it } }

2

You should definitely switch to milestone-6. The performance of dependency resolution is the major improvement in M6. We noticed the slow performance in M5 too but now we’re happy with M6

3

I’m testing this with m6 now.

Be aware that this build isn’t portable because of the use of ‘mavenLocal()’. You need the following repo for the akka deps:

maven {
 url = "http://akka.io/repository/"
}
4

You also need:

maven {
  url = "http://guiceyfruit.googlecode.com/svn/repo/releases/"
}

It’s generally a good idea to not use ‘mavenLocal()’.

5

After adding these repositories, the ‘gradle clean foo’ is about 700ms for me. There is also very little difference between m5 and m6 for me.

I’m unsure what it might be in your environment. The first thing to do would be to try m6.

6

Note there is a similar forum topic where at least 3 different people are experiencing dependency resolution problems to the point that Gradle milestone-6 is unusable. Found here: http://forums.gradle.org/gradle/topics/milestone_6_dependency_resolution_keeps_on_going

Luke, you commented there that your team would take a look at some debug logs to figure out what’s going on. Is this investigation still taking place? Any status updates?

7

I can confirm on a different machine starting with a clean maven repo and for the same project, for gradle m6, the time taken is between 4.7 to 4.8 seconds.

Will investigate the slowness on my other machine further today. Anything in particular I should be looking out for ? My experience with gradle is a bit limited (about 8 hrs or so), so any tips would be helpful. I’ll of course try out the debug mode to see what extra might be happening or might be causing the delay.

8

Ok, mystery solved at least partially, here are the observations till now

  1. Mac OSX, m5, jars partially spread across .gradle and .maven repos - 90 seconds 2. Ubuntu, m6, all jars in .gradle repo => 4.7 seconds 3. Mac OSX, m5, all jars in local maven repo - 100 seconds 4. Mac OSX, m5, all jars in gradle repo (m2 repo deleted) - 5.5 seconds

So not accounting for differences if any between m5 and m6, it seems a large amount of time is required if the jars are in a local maven repo and not the local gradle repo.

9

That does raise an interesting point. If one is transitioning from maven to gradle (if there are a large number of projects), using a local maven repo is absolutely critical (since maven projects can’t see the gradle repo). So is there a way to not pay this performance penalty?

10

I think ‘unusable’ is a bit strong. There’s a regression in milestone-6, where it doesn’t handle deep dependency graphs. There will be some fixes for this next week. They probably won’t make it into milestone-7, but will be available soon after that, and certainly in milestone-8.

11

Any chance you can test m6 with all jars in local maven repo?

12

Will do at night. Its morning here at the moment.

13

I can’t speak for andrew, but I used the very same term yesterday. https://twitter.com/#!/dnene/status/144780267184340992 The reason I did that was because I was exploring migration from maven to gradle, and there was simply no way I could actually go back and recommend to my team that we should start using gradle unless people felt comfortable that the performance was reasonable enough to support frequent recompile scenarios. (That was before the performance issue being discussed here was partially resolved).

14

Sorry to use strong words, but I’m simply stating the facts. Milestone 6 is unable to resolve dependencies for at least one of my projects within reasonable limits e.g. simply running ‘gradle dependencies’ takes several minutes and then crashes because it runs out of heap space. Even if I increase the memory allocation, it is still unusable due to the amount of time it takes. And apparently I’m not alone on this island.

Note that I’m not stating that Gradle as a whole is unusable, just milestone 6. Milestone 3 does not have this problem, so that’s what my team will use until this issue is resolved.

I should mention that we are not using ‘mavenLocal()’ on my project, so the problem is not isolated to that use case.

Also note that I have posted debug logs from running ‘gradle dependencies’ for my project here: https://github.com/javaSpock/problem-gradle-1m6-dep-res/raw/master/gradle-logs.zip

Any suggestions you might have would be appreciated.

15

@andrewbgoode, the ‘mavenLocal()’ problem is different to the problem you’re seeing. There should be a fix available next week. If you would be willing to test it out, that would be appreciated. In the meantime, there’s not much you can do other than stick with milestone-3.

We’re still digging into the ‘mavenLocal()’ problem.

16

Could you expand on what you want to use the maven local repo for? To avoid downloading artifacts that have already been downloaded by maven? To share locally built artifacts between projects?

17

As requested,

  1. Ubuntu, m6, all jars in .m2 repo => 4.2 to 4.5 seconds.

I haven’t yet run m6 on the mac (and it likely will take me some time before I get to it).

18

To share locally. I imagined we would slowly transition one project at a time.

19

So does using m6 fix the problem for you, then?

20

Just did an upgrade to m6 on the Mac. The timings are about 5.6 seconds. The problem is solved.

Thank you for the enthusiastic support.