Custom gradle distribution updates

Old Forum Archive
1

In my company I am creating a custom gradle distribution that I am hosting on an internal HTTP location. The users projects are configured via the gradle wrapper to point to this location.

What I’m seeing though is if I make updates to the distribution and push it out to the HTTP location, the users desktops don’t see that the distribution has changed and therefore do not re-download it.

Basically what I need to accomplish is to have 1 location where the users desktops point to so they always have the latest version of the distribution. I don’t want to (& really can’t) go out to every single project (close to 1,000 projects) and have them modify their gradle-wrapper.properties to point to a new “version” of the custom distribution file out on the HTTP location. I want do set up the wrapper once on each project and then have the projects always see a new version of the distribution if I should choose to push a new one out.

Any thoughts/ideas?

2

The distribution itself is not the right mechanism for push updates. It’s better to have your custom logic manage this. Either by applying from a URL (always latest) or by applying a binary plugin from a repo with a dynamic version number.

The main issue with auto updating distributions is that you’re forcing a 40mb download on users when only a very small portion has changed.

3

I didn’t realize you could apply from a URL. So that would look something like this?

apply from: 'http://somewhere/inside/my/company/some-gradle-file.gradle'

Then that gradle file would have something like this in it?

buildscript {
    dependencies {
        classpath files(fileTree(dir: 'lib/build', includes: ['*.jar']))
// Wire in the custom plugins/tasks into Gradle's bootstrapping
    }
}

We don’t (& won’t) use Maven/ivy repositories to host dependencies, so how would I keep my custom plugins jar file up to date? Would I have to bake in logic that knows how to go and fetch updates to itself? If I did, then if I found updates I would need to restart the current instance of gradle, since i would already be past the bootstrapping part of where my custom plugins jar file is wired into gradle correct?

What I’m really after is a way to dynamically be able to push updates to my plugins without the users having to do anything - and doing that without using other tools such as maven or ivy.

4

Then that gradle file would have something like this in it?

You can’t modify the host build script from an applied script. The application happens after the classpath needs to be finalised.

What kind of things does your custom distribution do? Is it applying plugins? Or is all the logic in the init script?

5

There is very little logic in the build script. It wires in a jar file into the build script and then applies a “base” plugin that we provide, which sets up some default properties/tasks for every build in the company. The jar file contains many different plugins which any given project can apply to get different functionality. Our custom plugins apply gradle’s default plugins and then do other things specific to our builds here. Our goal is to provide as much functionality via the plugins so that the projects build scripts are very minimal and purely declarative and provide just configuration.

That being said, when we build our plugins we have to “certify” our plugins against a specific version of gradle. If either we update our plugins or we update the version of gradle we want to use then we need to “'re-certify” our plugins and then get them to the users. We would llove a solution where the users did not have to do anything when we upgrade our plugins or upgrade the version of gradle - and the gradle wrapper/custom distribution seemed like the best way to go. If there’s a better way then I’m all ears :slight_smile:

6

Pushing out so much implicitly and automatically seems risky to me. You’re running the risk of having someone’s build work one day and then fail the next for reasons they don’t understand because they haven’t done anything. The other problem is when you check out a project from 12 months ago and it no longer works with your latest plugins/logic and there’s no easy way to get back to what it did work with.

You’re trading off stability for upgrade convenience. In an scenario with this many builds, this can cause big problems. But obviously you are in a far better situation to comment on what would work for your workplace.

If it were me, I’d consider developing the build platform just like you would a shared library. That is, you version the platform and people upgrade as they can/when necessary. Once you release a stable version, it’s around indefinitely. Your development versions can use changing versions (e.g. SNAPSHOT) of your binary plugins to stay on the edge, then your stable versions use pinned versions of these plugins.

7

We can’t support build versions indefinitely. If a project which hasn’t done anything in 2 years comes along and needs to build, then they need to make necessary changes to be able to upgrade. in our current ant based infrastructure every build ALWAYS uses the latest bversion of the master ant build script. This has not caused us any issues in the years it has been working (5+). Our requirement remains - we need to control which version of gradle to use, matched with the version of our custom plugins, and the projects themselves shouldn’t have to do anything to always keep up to date with lhe latest versions of each.

8

There’s no way to automatically upgrade the version of Gradle used right now. That is, there is no change checking between the local version and the original location. Distributions are considered to be stable.

The best that I can think of off the top of my head would be to check at runtime if the version numbers match. Your plugin/logic (applied via URL or changing version plugin) should now what the target Gradle version is. If there’s a discrepancy (you can get the runtime version by ‘GradleVersion.current()’) you can fail the build and tell the user to run a task to update the gradle wrapper (which your plugin installs). The user then runs this task, and on the next run the wrapper mechanism downloads the new distro.

9

That makes sense for keeping the gradle version up to date. What about keeping our plugins up to date, assuming we aren’t changing the version of the underlying gradle distribution?

I guess we would have to create a versioning scheme which encapsulated both the gradle version & our plugin version (something like custom-gradle-1.1-1.0-bin.zip - gradle 1.1 & plugin version 1.0).

10

You can use a changing dependency:

buildscript {
  dependencies {
    classpath "foo:bar:1.0", { changing = true }
  }
}

Changing dependencies are cached for 24 hours by default. You can tune this via ‘buildscript.configurations.classpath.resolutionStrategy’ (see ‘resolutionStrategy’ of ‘Configuration’ in the DSL ref).

You might also want to check out this post: http://forums.gradle.org/gradle/topics/how_can_you_use_an_init_script_to_specify_a_repo_dependency_for_a_plugin_jar_but_allow_version_to

11

Great I’ll take a look at that. I appreciate all of the input you’ve given me. It looks like that, no matter what approach I want to take, that I will have to do something on my side to get it all to work.

It would be nice if the gradle wrapper had something like a “latest distribution,” where it would check via a timestamp or something from a location of a distribution - if that location is newer than what is on disk, then download it and use it - otherwise use what is on disk. Seems like the gradle wrapper already has probably 75% of this functionality already baked into it - would be nice to have the other 25% :slight_smile:

12

We are exploring some options for this, but it’s worth noting that it’s a rare use case. Users (that we have spoken to) to date have been far more conservative in managing updates.

I also just want to point out that it’s not a trivial problem. We are very conscious of the weight of the wrapper as people have to check it in to their version control repo. Therefore, we have to design this very carefully. This isn’t to say we aren’t going to do anything, just that it’s not simple enough to get through quickly.

In the next week or two we’ll have a better idea about what we will do here. At that point we’ll open an “idea” post on these forums that you can watch/track for updates.

13

That would be great. Thanks for the follow up.

14

I’ve done some investigating around this.

I guess I’m having trouble seeing the value in what the gradle wrapper provides from a corporate standpoint. We want to control which version of the core gradle distribution is used across the enterprise by 1,000+ projects. If the version of gradle changes, then that means the wrapper task provided to each & every project has to be re-executed, so that the gradlew.bat, gradlew, gradle-wrapper.jar, & gradle-wrapper.properties get re-generated based on the correct version of gradle. How do we manage this so that when we push a new version of gradle (say gradle-1.2-bin.zip) out to an http server, we don’t have to go into each & every project and re-run the wrapper task manually?

I assume its not safe to re-use the gradlew.bat, gradlew, gradle-wrapper.jar, & gradle-wrapper.properties files from one version of gradle to another - they need to be re-generated by executing a wrapper task from the appropriate distribution.

If my goal is to always make sure that projects stay up to date with the version of gradle (without having to go into each project & do anything) then how am I to accomplish this?

15

How do we manage this so that when we push a new version of gradle (say gradle-1.2-bin.zip) out to an http server, we don’t have to go into each & every project and re-run the wrapper task manually?

There’s nothing in Gradle to do push out new distributions in this manner. You’ll need to implement your own solution.

How did you handle pushing out new versions of Ant with your previous infrastructure?

I assume its not safe to re-use the gradlew.bat, gradlew, gradle-wrapper.jar, & gradle-wrapper.properties files from one version of gradle to another - they need to be re-generated by executing a wrapper task from the appropriate distribution.

That is what is recommended yes. Though, the interface between the wrapper and Gradle is very light and has not broken since the invention of the wrapper. Many people do now update the wrapper scripts, they just modify the ‘gradle/wrapper/gradle-wrapper.properties’… but I would not recommend this.

If my goal is to always make sure that projects stay up to date with the version of gradle (without having to go into each project & do anything) then how am I to accomplish this?

It will depend on your infrastructure, and you’re going to have to write some code. Off the top of my head, I’d probably investigate writing a plugin that injects a check to the start of each build that compares the version of Gradle running to what the “latest” is, by hitting some web service. If there’s a mismatch, fail the build and tell the user to run the task ‘latestWrapper’ which is an instance of task ‘Wrapper’ that your plugin adds and configures (using the web service) to grab the latest. We do something along these lines for managing the wrapper in the Gradle build itself: https://github.com/gradle/gradle/blob/master/gradle/wrapper.gradle