Dependency resolution became unbearably slow after upgrading to Gradle 1.0


(Urs Reupke) #1

I’m using gradle (via the gradle wrapper) for a medium-sized project. My standard time for a build from scratch is about 7 minutes, including downloading gradle, downloading all maven dependencies and building my project.

Yesterday, after reading about your 1.0 release, I tried upgrading the build from 1.0M5 to 1.0 final.

A first run told me about the change from ‘‘properties’’ to ‘‘ext.properties’’, so I made the change and started a full build. Surprised when 10 minutes later, the tasks still were not done, I went away and let gradle work. Two hours later, it was still resolving and downloading some dependencies. I imagined that maybe there was some hickup with the files I already had, and deleted the gradle folder from my home directory, restarted - to no avail. Gradle ran for an entire day (>8 hours), and still wasn’t finished resolving/downloading all dependencies.

Just to make sure, I reverted to Gradle 1.0M5, removed the cache from my user home once again, and had a build within 6.5 minutes.

I’ve looked through the migration guides, but didn’t notice anything that points to this problem. All issues regarding resolution speed (starting with M6) appear to be resolved. I’m feeling stupid, because after the long run-up, the issue certainly is on my side, yet it is not obvious to me.

Can you help me out?


#2

There were significant changes to dependency resolution between M5 and the 1.0. We can help out if you can give us a bit more information.

First, can you provide the ‘repositories{}’ section of your build file? Are you using an ivy DependencyResolver, or are your using the new repository DSL?

The most important info you can provide in this instance is the logging coming out of Gradle 1.0. Please try running your build with the ‘–info’ flag so we can see what is going on with dependency resolution during your build. Since the ‘–info’ output doesn’t display timestamps, a section of log run with ‘–debug’ would be useful as well. (It’s often useful to supply these using a service like https://gist.github.com/).

Once we have this information, it should be possible to work out what’s going on with your build.

So your build will run with M5 in ~7 minutes with a clean cache? So that includes downloading all dependencies?

Can you provide the output of running your build with Gradle 1.0 using the ‘–info’ flag? Obviously we don’t need the entire log output


(Urs Reupke) #3

As I wrote above, that fast build under M5 includes the full download.

Regarding the log output, I can provide, but I am not sure what I can produce in reasonable time. I just let it run with ‘’–info --debug’’ for about an hour and send over the results.

Finally, here’s the ‘‘repositories{}’’ block. I’m not sure what kind of DependencyResolver I am using - I am using a DSL, but it is the only one I know, so I can’t say whether it is the old one or the new one. Note that it is in the main build file’s ‘‘subproject{}’’ section, could that cause trouble?

EDIT: I can’t get the forum to not tamper with the build file’s code. Either see the original above or the gisted excerpt here.

… subprojects {

repositories {

maven {

url ‘https://github.com/anathema/anathema-thirdparty/raw/master/snapshots/

}

maven {

url ‘https://github.com/anathema/anathema-thirdparty/raw/master/releases/

}

mavenCentral()

}

… } …


#4

I’m sure we don’t need even an hour or logs: my assumption is that the same thing is happening repeatedly. We’ve had problems with github-based repositories reported before : GRADLE-2083.


(Urs Reupke) #5

Thanks. I’ll produce the logs tomorrow.


(Urs Reupke) #6

I have gisted the log files.

First, I called ‘‘gradle wrapper’’ from an installed gradle 1.0 to create the wrapper. Then, I deleted gradle’s cache in the user home. Then, I called ‘‘gradlew clean buildZip --debug >log.txt’’, where ‘‘buildZip’’ is one of my custom buildTasks. The build failed after ~7 minutes due to a timed out connection to one of the maven repos. I saved the file, it is log1.txt in the gist. Then, I called the same command again, and had it running for about 30 minutes.

It is log2.txt in the gist.

I noticed that calls to the maven repo hosted at github frequently have “garbage” in their responses, and that gradle waits for about 90 seconds afterwards until it closes the connection. With the amount of requests involved, it seems very likely to be the cause for my problem.

To me, this poses two questions:

  1. How can I avoid the garbage response and the long wait to close? 2) Is the change in behaviour from gradle 1.0M5 to in gradle 1.0 final intentional?

(Urs Reupke) #7

They are up, see my reply.


#8

Repositories hosted on GitHub and GoogleCode can be problematic, due to their unreliable handling of standard HTTP requests. We are currently looking at ways to address these problems: GRADLE-2083 and GRADLE-2124. We also want to be better behaved with repositories that timeout: GRADLE-2345.

To me, this poses two questions:

  1. How can I avoid the garbage response and the long wait to close?

One thing to try would be to move your mavenCentral() declaration above the github repositories in your build script. That way, only things that aren’t found in mavenCentral will be searched for in GitHub.

  1. Is the change in behaviour from gradle 1.0M5 to in gradle 1.0 final intentional?

In a sense, yes. We have done a bunch of work to make resolving dependencies against HTTP servers more efficient. But this has involved quite a number of changes, including depending on a reliable response to HTTP HEAD requests and upgrading to HttpClient v4.0.

However, it looks like you’ve uncovered an issue where we never recover from the “garbage” returned by GitHub as reported by HttpClient: I was unaware of this issue, and I think it warrants further investigation.


(Urs Reupke) #9

Making an educated guess about the repositories and changing their order around brings me down to ~6 Minutes including the full download. That’s perfect, thank you very much!

As for the rest of issues, I’ll keep an eye on them. Would it be of help if I entered a new issue for the garbage problem?