How does `pluginRepositories` handle SNAPSHOT versions?


(Ned Twigg) #1

Thrilled to see the new pluginRepositories block in 2.14-rc-1.

pluginRepositories {
    maven {
        url 'https://private.mycompany.com/m2'
    }
    gradlePluginPortal()
    ivy {
        url 'https://repo.partner.com/m2'
    }
}

What is the SNAPSHOT caching policy? Will we need to put

configurations.all {
    resolutionStrategy.cacheChangingModulesFor 0, 'seconds'
}

To ensure we’ve always got the latest snapshot of a WIP plugin?


(Mark Vieira) #2

These are still resolved as buildscript dependencies so you’ll have to set the resolution strategy on the appropriate configuration. Of course things like --refresh-dependencies will also continue to work.

buildscript {
    configurations.classpath {
        resolutionStrategy.cacheChangingModulesFor 0, 'seconds'
    }
}

(Ned Twigg) #3

Understood. I’ve always felt this default was decidedly backwards. In case there’s any chance of a change in 3.0, here’s a pitch:

               | Cache by default                 | Don't cache by default          |
---------------+----------------------------------+---------------------------------+
Build time     | Instantaneous after first load   | +100ms to check for stale POM   |
---------------+----------------------------------+---------------------------------+
Developer time | Anytime someone tests a SNAPSHOT | +100ms per build so long as     | 
               | and has a problem, you have to   | the dev is testing a SNAPSHOT.  |
               | ask "is it old?" and describe    |                                 |
               | the various workarounds.         |                                 |
---------------+----------------------------------+---------------------------------+
Difficulty of  | To disable caching, add 5 lines  | What is the usecase for wanting |
switching to   | to buildscript, using concepts   | a stale SNAPSHOT?  Whose        |
other behavior | that users wouldn't otherwise    | bandwidth is too expensive to   |
               | ever have to know about.         | check for a stale POM?          |
---------------+----------------------------------+---------------------------------+

Also, I feel like you guys have built all the hard parts of this fantastic plugin system, but you keep getting the plugin workflow wrong in a fundamental way. Here’s a user story that’s been broken since the beginning of the plugins{} block, and is still broken with the introduction of the pluginRepositories{} block.

I maintain a Gradle plugin, and my users ask me for an improvement / bugfix. I publish a SNAPSHOT, and they want to test it.

First off - I can’t upload SNAPSHOTs to the plugin portal, so I have to make sure they add the OSS snapshots repo to their buildscript. Which now they can, hooray!! Not quite as good as if the portal handled SNAPSHOTs, but at least we can now use the plugins{} block.

pluginRepositories {
    maven {
        url 'https://oss.sonatype.org/content/repositories/snapshots/'
    }
}

They test it, and find a problem. I fix it, and publish a second SNAPSHOT. But they say it’s still broken. So now we spend some cycles making sure they add this:

pluginRepositories {
    maven {
        url 'https://oss.sonatype.org/content/repositories/snapshots/'
    }
}
buildscript {
    configurations.all {
        resolutionStrategy.cacheChangingModulesFor 0, 'seconds'
    }
}

And even that might be wrong because maybe the buildscript block has to go first? I don’t know. That’s TEN lines of boilerplate that has to go on the top of every single buildscript before users can be an active, SNAPSHOT-testing member of the community. 5 lines are because the portal doesn’t support SNAPSHOTs, and the other 5 are because the default is backwards.

There’s a couple minor problems stacked on top of each other that make the overall process quite hairy, and make it impossible to take advantage of your otherwise beautiful design. But maybe I’m missing a shortcut?


(Stefan Oehme) #4

Re-checking every snapshot on every build would really degrade the user experience, so this is out of question from my point of view. Plus, the user doesn’t have to add the buildscript block, you can tell them to do --refresh-dependencies when you have pushed another attempt.

I personally don’t like the concept of snapshots at all, you never know what you get :slight_smile: Why not do a little bugfix release once you have fixed the bug?


(Ned Twigg) #5

Re-checking every snapshot on every build would really degrade the user experience

I disagree strongly. It only affects performance if you’re using snapshots. If a user told you they wanted a snapshot, which is more important to their experience: < 100ms per build, or that they have the correct snapshot?

Here are 2,200 buildscripts full of boilerplate who don’t like the default

Here are 1,600 CI scripts calling --refresh-dependencies, which is less efficient than only refreshing changing dependencies, which is probably what they actually care about

Google “gradle snapshot refresh”.

Here’s what the Maven docs have to say:

The SNAPSHOT value refers to the ‘latest’ code along a development branch, and provides no guarantee the code is stable or unchanging.

Snapshot is extremely handy for doing integration testing between interdependent libraries. But only if you’ve remembered to add --refresh-deps to all your CI servers, which is unnecessarily slow, or a bunch of boilerplate to your build scripts.

Not using snapshots is a reasonable approach. But supporting snapshot, and then not actually checking for new versions is sort of like saying “You shouldn’t drive a gas car because electric is better. But since you made us build a gas car, we’re going to make the gas cap really tiny so its hard to fill and burn too much gas. If you want to make the gas cap bigger, here’s a saw.”

Why not do a little bugfix release once you have fixed the bug?

If I was 100% sure I had fixed the bug, I would! But sometimes a user has a problem that’s hard to reproduce. It’s great to be able to easily iterate with a user without involving the more conservative community who is waiting on a fully-vetted release.

A quick way to answer this would be a call of hands at Gradle Summit: “Raise your hand if you like and use snapshot dependencies, and keep it up. Now, lower your hand if you think Gradle should recheck for new SNAPSHOT artifacts on every build by default”. I’m not sure what would happen, but I am curious :slight_smile:


(Mark Vieira) #6

FYI, our default behavior is the same as Maven’s with regards to caching SNAPSHOT artifacts. Both have a default caching strategy of 24 hours unless otherwise set in your settings.xml. In Gradle you can get a similar result with an init script and avoid the boilerplate you describe.


(Ned Twigg) #7

our default behavior is the same as Maven’s

Apache Maven started in 2004. I was still on dial-up, and it was a year before YouTube and two years before Spotify. Bandwidth conservation techniques which were reasonable compromises in 2004 are now legacy cruft. Gradle 3.0 is a chance to shed legacy cruft. I hope you take the opportunity to examine your assumptions, but I’ll be using and evangelizing Gradle regardless :slight_smile:


(Mark Vieira) #8

I’d be curious what the general consensus is here. I expect this is a bit of a “squeeky wheel” situation though, as those that are content with the default are unlikely to publicly express that opinion unless pressed. I’d be open to a quick poll at the summit.

In general we try to optimize defaults to be as performant as possible. Dependency resolution is by far the most expensive, non-execution related task builds do. For large multiprojects with expansive dependency graphs this is especially the case. As is the nature of defaults, they tend to alienate certain use cases. We are unlikely to change this default behavior, however there are ways to make this particular use case less painful. Mainly, doing cross-project development with snapshots is a pain, and for other reasons besides artifact caching. Composite builds will make this a much better experience. We plan on demoing this capability at the summit :slight_smile:


(Ned Twigg) #9

Composite builds will make this a much better experience.

I look forward to it!

Dependency resolution is by far the most expensive, non-execution related task builds do.

My build has ~20 subprojects, we use SNAPSHOT heavily during development, and I don’t notice the performance hit unless there’s actually a new SNAPSHOT available (in which case we 100% want that SNAPSHOT). POM’s are usually less than 1k!

I’d be open to a quick poll at the summit.

I am indeed a squeaky wheel :slight_smile: Amongst my squeaky customers, some have helped me find new opportunities and some have wasted my time. I’m not sure what kind of squeak I am for you, but I don’t think I’ve ever gotten a survey from Gradle…


(Mark Vieira) #10

No offense, but 20 projects does not a large multiproject make :wink:


(Mark Vieira) #11

We could probably do this more. Typically feedback on the forums is pretty good and we can gauge off that well enough. The problem with surveys is the lack context, which limits their usefulness. We also have a lot of close contact with big Gradle “enterprise” users which provides feedback as well. The trick is finding a balance between the big complex users of Gradle and the “casual” Gradle users. They are very different use cases and I fear that any kind of poll or survey would skew towards the latter. The attendees at the summit tend to be more of the former.


(Ned Twigg) #12

As always, the Gradle team shows great judgement :slight_smile:

I would imagine your enterprise customers have dedicated build engineers who are familiar with init.gradle and cacheChangingDeps and override lots of defaults anyway. And enterprise adoption is driven to some extent by engineers having great experiences with gradle on their hobby projects.

I would optimize defaults for the people who aren’t going to set them, rather than for people who have a team dedicated to overriding them.


(Mark Vieira) #13

This is a probably a good practice. I would argue that defaulting for performance is in the best interest of this demographic. Folks don’t want to have to fiddle with things to get the fastest experience. Many folks also don’t want snapshots to continually update. It’s often disruptive to the development process. They would then end up using something like --offline to get around this.

So let’s circle back to your use case, which is really what we want to solve. It’s not a matter of coming to agreement on what the best default caching strategy is, it’s about making cross-project development easier. So I contest that composite builds is the solution to this use case, rather than changing the default caching strategy for snapshots. So in general we want to find good solutions to use cases, rather than stress over default behaviors aligning with the most common development practices (which change, dramatically, over time). Not to mention changing defaults like this is not a backwards compatible change, pissing off yet another group of folks that rely on the exiting behavior :wink: