Dependencies specified with version ranges cause slow builds

Old Forum
1

Can anyone explain why gralde builds take so long? I do not mean startup time that can be tolerated or worked around with “daemon” setting.

For example, the project I am working on now contains 15 java files so simple recompilation in Eclipse takes about 2 seconds.

But gradle builds same project for more than 70 seconds every time. In my case “gradle -profile” says:

Total Build Time 1:10.713, Configuration 2.034 Startup 0.599 Settings and BuildSrc 0.393 Loading Projects 0.095 Configuring Projects 2.111 Total Task Execution 1:07.288

Task Execution : 1:07.288 (total) :compileJava 39.808 UP-TO-DATE :war 27.172 UP-TO-DATE :deployToJetty 0.291

:processResources 0.017 UP-TO-DATE :classes 0.000 UP-TO-DATE

(:deployToJetty is custom task with simple file copy)

Even more puzzling is that even java sources were not changed it still takes 40 seconds to tell that they are up-to date. Summing that up, it looks to me that at least in this case build time is about 10 times longer than it should be.

So the questions are 1. can I speed up build process? 2. If Gradle in fact does a lot more than it appears on the surface (maybe it thoroughly analyzes dependencies while checking java sources or whatever), are there any plans to speed that up?

update: In case that matters, I use: Gradle 1.0-milestone-3 Gradle build time: Thursday, September 8, 2011 4:06:52 PM UTC Groovy: 1.7.10 Ant: Apache Ant™ version 1.8.2 compiled on August 19 2011 Ivy: non official version JVM: 1.6.0_23 (Sun Microsystems Inc. 20.0-b11) OS: Linux 3.0.0-12-generic i386

2

OK. It seems I found main time consumer. I used ‘+’ sign in several places instead of concrete version numbers and that seems caused dependency resolution work that apparently were not cached.

I re-ran it on the different computer with

Gradle 1.0-milestone-5 Gradle build time: Tuesday, October 25, 2011 10:56:08 AM NOVT Groovy: 1.7.10 Ant: Apache Ant™ version 1.8.2 compiled on December 20 2010 Ivy: 2.2.0 JVM: 1.6.0_23 (Sun Microsystems Inc. 20.0-b11) OS: Linux 3.0.0-12-generic-pae i386

And here it is reasonably fast.

However would not it be useful to have some sanity checks and issue warning in this case?

3

We have a project with deep jboss dependencies and about 15 or 20 subprojects. The builds are taking about 18 min with M5, and aren’t any faster than with M3. But with M5, it runs out of memory. We need to crack max heap to 2gig to complete the build. And we aren’t using any version ranges, nor any snapshots (went through and eliminated all snapshot dependencies to speed things up).

I’m puzzled why this is so low. The build pauses on every dependency resolution, even when it’s resolve the same dependencies already resolved by other subprojects.

I’ve only got a vague idea what’s going on under the covers with dependency caching, but it sort of looks like for every configuration of every subproject, gradle calls out to Ivy to do a complete dependency tree, then caches that entire unique tree. Not quite sure how that cache is then ever leveraged.

Seems like the naive way to do this would be to store each resolved dependency (not the entire tree, just an individual library) in a map by a hash of the unique coordinates of the dependency (group, name, version, classifier/configuration, etc) plus anything else that makes that library unique (like exclusion rules). Any sub-dependencies would be pointers back to other keys in that map.

Dependencies would essentially be fly-weight: no unique dependency would exist more than once. gradle would only need to call out to Ivy for resolution when looking for a dependency not already found in the map. And hopefully, you can ask Ivy to only resolve one level deep, so before diving further you can consult the cache again for hits. The cache would be global, across all subprojects in the build. Might also be serializable and written to disk for reuse across builds.

That would be super fast and memory efficient. Is it not doable for some reason? Insufficient Ivy apis? Dependencies must be modifiable so won’t fit the flyweight pattern?

4

We’ve fixed performance for version ranges (eg +) in the master branch. This will be available in the next release. We are continuing to work on dependency resolution performance over the next few releases.

In milestone-5 we added information about dependency resolution time to the profiling report, as this is a common time-consumer. It reports on total resolution at this stage, but even this is a hint about where to look. We will flesh this out over time, so that, in your case, it would be clear that Gradle is spending it’s time resolving the + dependencies.

Also in milestone-5, we added a ‘resolving’ message to the status bar at the bottom of the console output, so you can (subjectively) see that Gradle is spending time resolving dependencies.

5

@russellegan, this is exactly what we’re working towards, over the next few releases. The ivy APIs do not make this easy, but we are making progress. Unfortunately, there will almost certainly be regressions due to the complexity of this code (but we’re also making progress on reducing the complexity, too), such as the memory problem you’re running into.

6

Thanks, that’s good to hear. I don’t know the details, but it seems like that general design would cut build times to…well, a lot. Looking at my 18min build, probably 15 min of that is redundant dependency resolution.