Could we provide a more concise way of centralizing repository configurations?

To prephrase this suggestion a bit, let’s paint up some context. In most enterprise situations I would suspect that there is one centralized maven/ivy/artifactory/nexus server which serves all the artifacts for the project. I believe creating one single ‘virtual repository’ which mirrors all repositories needed for a project has even been mentioned as a best practice for gradle projects. This makes the life of the project maintainer simpler and probably makes the gradle build executions faster as there is a single point of truth when it comes to artifacts.

So far all is well. In most scenarios like this I would further suspect that the repository is password protected since it is probably also hosting the output of proprietary in-house projects. In most larger long-lived projects the enterprise would probably also be using a custom plugin to encapsulate and hide some of the build logic and leave the interface to the guy writing the code as clean as possible.

So where does this leave us with regards to configuration for repositories and dependencies? Well here is an example from my current project settings.gradle file:

buildscript {
      repositories {
    maven {
      credentials.username artifactoryReader
      credentials.password artifactoryReaderPwd
      url "$artifactoryReaderUrl/repo"
    }
  }
      configurations.classpath {
    resolutionStrategy.cacheDynamicVersionsFor 5, 'minutes'
    resolutionStrategy.cacheChangingModulesFor 5, 'minutes'
  }
    dependencies {
    classpath('my.custom.build:plugin:1.0') { changing=true }
  }
       //ugly hack, but 10x faster than doing the gradle.allprojects { buildscript {} } below
   //configurations.classpath.each { file -> settings.classLoader.addURL(file.toURI().toURL()) }
}
  gradle.allprojects {
   repositories {
    maven {
      credentials.username artifactoryReader
      credentials.password artifactoryReaderPwd
      url "$artifactoryReaderUrl/repo"
    }
  }
        buildscript {
     repositories {
      maven {
        credentials.username artifactoryReader
        credentials.password artifactoryReaderPwd
        url "$artifactoryReaderUrl/repo"
      }
    }
        configurations.classpath {
      resolutionStrategy.cacheDynamicVersionsFor 5, 'minutes'
      resolutionStrategy.cacheChangingModulesFor 5, 'minutes'
    }
          dependencies {
      classpath('my.custom.build:plugin:1.0') { changing=true }
    }
      }
}

In this specific case I need the custom build logic plugin classes to also be available in the settings.gradle file. So we need the root level buildscript block.

I then further want the plugin code to be available in all project build files. For this there is a non-supported hack which is commented out in the code above. I get the feeling that gradle.allprojects loops through the projects and we end up with a build log which flickers through the project list for a number of seconds. The hack runs more or less instantaneously, but I was told that those APIs might change and it is therefore not a recommended practice.

As an additional and final requirement for this particular case I want to centralize the artifactory username/password usage to this one file. For this reason I need the gradle.allprojects { repositories { } } block.

Now all the individual pieces of the DSL in the snippet above make sense when taken individually. But taken together it makes me wish for a more concise way of expressing this. The code above has a lot of duplication and we all hate duplication.

Perhaps something along the lines of:

repositories.default {
  maven {
    credentials.username artifactoryReader
    credentials.password artifactoryReaderPwd
    url "$artifactoryReaderUrl/repo"
  }
}
  buildscripts {
             //with an S at the end
  configurations.classpath {
    resolutionStrategy.cacheDynamicVersionsFor 5, 'minutes'
    resolutionStrategy.cacheChangingModulesFor 5, 'minutes'
  }
        dependencies {
    classpath('my.custom.build:plugin:1.0') { changing=true }
  }
}

could be an alternative?

repositories.default would be global for both buildscript and normal dependencies. i.e. if you don’t specify a repositories block in your project, we use the .default setting.

The ‘buildscripts’ closure would configure the classpath for all Script files in the project, both the settings.gradle file and the normal build files.

If there is an existing clean way of doing this I would very much appreciate the pointer. If not, perhaps it might be worth considering some additions to the DSL?

The problem of repetition between buildScript and project parts is really annoying, I believe the devs are aware of it. Considering all the Artifactory configuration - JFrog provides Gradle plugin that tackles it with a straightforward DSL (thanks for Gradle’s DSL extension points). Here’s the reference: http://wiki.jfrog.org/confluence/display/RTF/Gradle+Artifactory+Plugin

yes, we are actually already using the gradle artifactory plugin. Thanks to your reply I did some extra digging. For the following requirements:

  • every build file needs to be able to resolve the artifactory plugin * every build file needs to be able to resolve normal project artifacts via the artifactory plugin * every build file needs to be able to resolve buildscript dependencies via the artifactory plugin * the settings.gradle file needs to be able to resolve buildscript dependencies

we get an interesting set of buildfile configurations that do not work.

A few non-working examples:

1 . adding an allprojects block to the root project:

allprojects {
   buildscript {
     repositories { ... }
     dependencies {
      classpath 'org.jfrog.buildinfo:build-info-extractor-gradle:2.0.11'
    }
  }
}

breaks as we are trying to alter a ‘resolved’ configuration ‘classpath’. I.e. the buildscript block does not seem to belong in the allprojects clause.

2 . For another non-working setup, adding the following block to settings.gradle:

gradle.allprojects {
   buildscript {
     repositories {
      maven {
        credentials.username artifactoryReader
        credentials.password artifactoryReaderPwd
        url "$artifactoryReaderUrl/myrepo"
      }
    }
          dependencies {
      classpath 'org.jfrog.buildinfo:build-info-extractor-gradle:2.0.11'
    }
  }
          apply plugin: 'artifactory'
  apply plugin: 'maven'
  apply plugin: 'java'
    artifactory {
       resolve {
      contextUrl = artifactoryReaderUrl
      repository {
        repoKey = 'myrepo'
        username = artifactoryReader
        password = artifactoryReaderPwd
        maven = true
       }
    }
    }
}

breaks with a “plugin ‘artifactory’ not found” message. I.e. the ‘buildscript’ block does seem to belong in the gradle.allprojects block but the “apply: plugin ‘x’” does not.

Ok so to make things work we split up the configuration between the two allprojects calls and end up with:

//settings.gradle
gradle.allprojects {
   buildscript {
     repositories {
      maven {
        credentials.username artifactoryReader
        credentials.password artifactoryReaderPwd
        url "$artifactoryReaderUrl/myrepo"
      }
    }
          dependencies {
      classpath 'org.jfrog.buildinfo:build-info-extractor-gradle:2.0.11'
    }
  }
      }

and

//root project build.gradle
allprojects {
       apply plugin: 'artifactory'
  apply plugin: 'maven'
  apply plugin: 'java'
    artifactory {
       resolve {
      contextUrl = artifactoryReaderUrl
      repository {
        repoKey = 'myrepo'
        username = artifactoryReader
        password = artifactoryReaderPwd
        maven = true
       }
    }
    }
}

This actually works and also resolves subproject buildscript dependencies via the artifactory plugin (or it seems that way…but who knows…it’s certainly not crystal clear at this point they could as well be resolved by the settings.gradle repository setup). It however still does not solve the problem of injecting buildscript dependencies for the settings.gradle file. To solve this we need to add another buildscript { repositories {} dependencies {} } bock to settings.gradle file thus adding further redundancy.

Confusing? I would be inclined to think so. The artifactory plugin does provide a nice DSL but as I need to supply repository credentials for every build script anyway in order to resolve the artifactory plugin itself and then duplicate that information again in the artifactory plugin configuration DSL, the duplication is definitely not eliminated. Adding to this the further redundancy of credentials for the settings file buildscript block seems to leave us more or less where we started.

I uploaded a small zip with a sample project illustrating this at the following link. The project requires you to set up properties artifactoryReaderUrl, artifactoryReader, and artifactoryReaderPwd in your gradle.properties to point at some authenticated internal repository capable of resolving http-builder and the artifactory plugin. Running ‘gradle clean’ or any other java plugin task will illustrate that the resolution works.

If there is a less convoluted, simpler way of fulfilling the requirements I would be most interested in finding out.

edit 1: Also, for a large multi-project build (50+ projects), the gradle.allprojects block in settings.gradle becomes quite slow. Using the hack in the original post solves this. This would be another argument for providing a DSL which does not force gradle to loop through all 50+ projects, but injects certain dependencies to all build scripts with one call.

edit 2: Naturally, if we need the resolutionStrategy blocks in all these places we get a lot more code. Also the artifactory plugin does not currently seem aware of the new settings so we end up having to mix the two DSLs anyway.

This kind of global configuration is best done by an init script. Though, this is not entirely practical right now because of GRADLE-2067 and there being no conventional location for a project scope init script. Just sharing for completeness.

We were discussing this issue some time ago, and the main point missing for the perfect init script injecting these configuration was: We need a way to place a listener on every loading of a gradle script file. Not project or gradle event, but pure onScript events.

My 2cts from memory, Fred.

Been a few months since I initially posted this question.

Any new thoughts or plans on a solution for this? Have the init script mechanics perhaps improved to a point where this would be best done from an init script?

In enterprise I’ve seen this use case solved by an init script.

The problem I have with init scripts is that they pollute the command line interface of the end user.

With that being said, could an init script solve both the settings gradle buildscript classpath problem and the corresponding subproject buildscript classpaths problem? I.e. could we extrernalize the repository definitions, artifact handling directives, and buildscript dependency definitions from the initial code sample above to just live in the init script without any references to those things in settings.gradle or any subproject build.gradle files?

If not, I still think that having some mechanism for defining a “default repository” and “default resolution strategy” for all project buildscripts and settings.gradle would be very useful. If configured to do so, such a setting could also be used for the actual project (i.e. non buildscript) repository and resolution strategy definitions. This would move a triple (settings buildscript, other buildscript, project) duplication of all these settings into just one place. If a dsl change such as the one I mentioned initially is too disruptive, I would even prefer an extra file such as “resolution.gradle” to the current state of duplication.

PS Would hacking a gradlew wrapper script to auto-include an init script be against gradle best practices? This would to some extent (you still can’t build sub-projects without checking in a gradlew script in every subproject dir) remove the command line interface pollution DS

(sorry for the late response here)

This is indeed a bit of awkwardness in the way that things currently work. The reason that we haven’t solved this problem is that it’s quite tricky. We’ve kicked around ideas a few times on how to smooth this out but haven’t been happy with the compromises.

At the base level, we absolutely cannot assume that build dependencies (e.g. gradle plugins) will come from the same place as application dependencies. So to provide any kind of one liner, it needs to be syntactic sugar.

The ‘buildscript {}’ mechanism is also a bit tricky. This actually works by using a two-pass strategy. We use a Groovy compile time transform to effectively split each build script into two scripts: One with all of the ‘buildscript {}’ statements and then another with everything else. The buildscript one is executed first. At the end of this we know how to create the classpath for the “main” script. So, the sugar would need to be in a ‘buildscript {}’ block.

We’ll work out a good solution to this ordering problem at some point.

Right now, the best you can do is to use an enterprise init script that pre populates the build script and project repo containers. Enterprise init scripts were covered towards the end of the webinar I did on “Standardizing and administering your enterprise build environment with Gradle”.

Hi Luke. I would like to start off with a huge thank you for a most excellent webinar (though for posterity, it might be good to fix the link above). I thought I had the basics of gradle figured out, but the webinar (and gradle) kept on revealing new nuggets of wisdom all through the video. DomainObjectCollection was an eye opener, I had ran into the classes, just hadn’t made the connection that the class was that pervasive and general. +1 for more webinars like this. Very valuable.

I was however still left with a question. I have a fairly complex custom object plugin deployed to an artifactory server. Some of the classes in the plugin need to be available in the settings.gradle script as the plugin code is used to build up a multi project build structure.

I can see how I could create a custom gradle distribution with an embedded init script and how I should be able to eliminate all the:

buildscript {
   repository {}
   dependencies {}
   configurations {}
}

blocks from my project build files this way. I have still not been able to figure out how I would externalize the buildscript block for the settings script itself. The only settings related hook I could find is:

//init.gradle
  afterSettingsEvaluated { settings ->
     }

which as far as I understand things is way too late in the game to be meaningful. As you point out, we would have to hook in very early to affect the settings buildscript block as it is preprocessed before even the actual script code is. It seems that the:

//settings.gradle
  buildscript {
     apply from: 'something.gradle', to: this.buildscript
  }

construct you briefly mentioned for project buildscript blocks in the webinar actually works for the settings one also, but this would make it impossible to define a version you would like to depend on as the involved Script etc instances do not seem to be extensions aware (or rather, this is my understanding of things).

Otherwise we could have done something like:

//settings.gradle
  buildscript {
     apply from: 'something.gradle', to: this.buildscript
    corporate.useBuildPlugin "1.0"
  }
    new CustomPluginClass(this).with {
    setupMultiProjectStructure "some/search/path/**"
  }

Any suggestions? Or is there some other place I should move my multi project building logic into?

In general, using plugin code in the settings script still seems a tad less elegant than is the case for the rest of gradle. For reference, and now that I dug through the forums an extra time, this is not the first time we’ve talked about this, previous threads:

look for rootdir init gradle and apply as an init script if it exists while we are waiting for plugins in settings gradle

and related issues:

GRADLE-2059 open GRADLE-2066 open GRADLE-2065 open GRADLE-2067 open

Hi Matias,

Thanks for the feedback on the webinar. We put a lot of effort into them so it’s great to know that they are helpful.

This is indeed a tricky problem, because of the deficiencies identified in the issues that you linked to. We are starting the planning for 1.4. I’ll raise this problem in that discussion and post back with what we plan for 1.4 (if anything; I can’t promise that this will make it).

It’s extremely hackish, but what you could try in the meantime is attaching your extension to ‘buildscript.dependencies’ or ‘buildscript.repositoryHandler’. Not nice, but it should work.

If you distribute your settings plugin as a script plugin, inside your custom Gradle distribution you should be able to load it relative to ‘gradle.gradleHomeDir’.

As 1.3 is out and the 1.4 feature set seems to have crystallised, any news on the scheduling for this issue?

No updates at this time.

With the risk of getting a tad repetitive, 1.6 is now out and there have been a few tangential improvements (such as plugins in init scripts and settings files)…any new ways or thoughts on centralizing repository configurations or news on the scheduling for this?

Same situation.

This time with the certainty of getting repetitive, 1.8 is out. Any news or tricks to nullify this issue?

Personally, as workaround I have created file ~/login/.gradle/init.gradle which will be loaded by gradle every time. This file declares all commonly used repos, it looks like below:

allprojects {
 ext.RepoConfigurator = {
  mavenLocal()
  maven { url = uri('internal NEXUS URL here') }
 }
   buildscript.repositories RepoConfigurator
 repositories RepoConfigurator
}

I have a more logic that takes into account release/snapshot versions, but the idea is clear from this snippet.

Creating of this file is a step of initial setup for build environment, and it’s close to what one do for setting up maven’s settings.xml.

Nice! I was just thinking along similar lines of having a configuration closure. Using an init script is however somewhat invalidated in our scenario as having a portable source repo which can be built without special configuration on the client machine is vital to the project in question.

This could have been accomplished via a project-level init script…if we had them. The idea has been rasied about a year ago. The following forum thread:

http://forums.gradle.org/gradle/topics/while_we_are_waiting_for_plugins_in_settings_gradle

resulted in the following idea:

http://forums.gradle.org/gradle/topics/look_for_rootdir_init_gradle_and_apply_as_an_init_script_if_it_exists

but AFAIK this has not been implemented yet.

Matias, you are right and pointed links are explain useful scenarios. But this thread was started about centralizing repository configurations, as I understood. And I have shared how I solved this issue on my work.

In detail, we having locally installed NEXUS and it’s bad approach to copy same data into each project (we have many). So having one place where this is declared is reasonable for us. So the init script I have includes snapshot/release repo declarations, credentials (we won’t commit it with sources in any case) with publishing rights. So init.gradle located in homedir is useful only as a safe place for credentials.

Hi GradleWare. Long time since this thread was last updated.

Any update on centralizing repository/dependency blocks now that we are on gradle 2.x?