How to control/understand the execution order of build listeners?

Old Forum Archive
1

My plugins apply a variety of conventions to enable a standard, but configurable, way to build any given project. It does this at three points in time;

  1. when the plugin is applied 2. when ‘Gradle#projectsEvaluated’ is fired 3. when ‘TaskExecutionGraph#whenReady’ is fired

This works perfectly well but I have 2 main problems;

  1. it is not v easy to navigate, particularly for those new to this code 2. the order in which the callbacks fire appears to be undefined (in api terms) and callback registration order (in practice)

The former can be worked around.

The latter feels brittle when there are multiple “independent” plugins involved along with n subprojects, each of which are doing their own thing. It becomes v difficult to know what order things will happen in as you have to work through the order things are listed in the build script itself and then go through each project to see what each one does.

My Q is whether there are common strategies that people use to ameliorate this effect?

Alternatively are there any plans to extend this in gradle core? It feels like it would be nice to have a (simpler) task style dependency graph that fires on each callback rather than a bunch of callbacks that fire in insertion order (though I realise this may be overkill).

Cheers Matt

2

I don’t think we want to clutter the api with elaborate ways of specifying the order / precedence of the callbacks. Instead, we want to solve the use cases so that callbacks like projectsEvaluated, afterEvaluate are rarely needed.

You need to tell us a little bit more why do you use projectsEvaluated and whenReady in order to give you an alternative solution.

Do use the withType and other live filters features? For example, let’s say your plugin adds something fancy to the java projects. You don’t know if the java plugin is applied before your plugin. You can do things like:

project.plugins.withType(JavaPlugin) {
  //this will be triggered for existing plugin or for JavaPlugin applied later
}
  project.tasks.withType(Test) {
  //this is triggered for existing tasks of type Test, and any Test tasks added later.
}
  project.tasks.matching { it.name.startsWith('foo') }.all {
  //...
}
3

Our plugins intend to provide all our internal conventions such that the build.gradle looks a lot more like an ivy.xml, i.e. it’s just a bunch of project specific dependencies. However it still lets you tweak all the things we provide to change precise behaviour for a given project. This leads us using projectsEvaluated as the callback to determine how to respond to changes in the things exposed through the DSL as advised [in this thread ] (http://forums.gradle.org/gradle/topics/how_to_create_a_plugin_that_can_be_configured_through_the_build_gradle). Most of this logic is assorted configuration of the maven & idea plugins as well as setting up different test integrations (we use junit, testng and cucumber-jvm at different times). Generally I think the point is that we have behaviour that depends on more than 1 DSL construct so it has to go into the projectsEvaluated section.

The main use of whenReady is to twiddle certain attributes of our release plugin and to ensure different binary repositories get added in a specific order (we have some staging repositories for certain situations) to ensure we resolve the “right” version without needing to try and intercept the resolution process (which I don’t think is possible as you don’t expose the lateststrategy).

4

From the thread that you linked to:

The general approach is to use lazy collections or lazy APIs if available, and convention mapping otherwise. However be aware that convention mapping isn’t currently part of Gradle’s public API. If you want to play it safe, you’ll have to resort to aforementioned hooks (gradle.projectsEvaluated {} etc.) instead of convention mapping.

If ‘gradle.projectsEvaluated {}’ doesn’t suffice in your case, convention mapping is probably your best bet.

5

my point is that projectsEvaluated works perfectly well but when you start using it a lot, exactly what code is executing becomes more difficult to reason about so then it becomes a question of what are the best ways to deal with that.

6

when you start using it a lot, exactly what code is executing becomes more difficult to reason about so then it becomes a question of what are the best ways to deal with that.

The only answer I can give is figure out an alternative way than projectsEvaluated :slight_smile: That’s how we deal with this problem.

Can you provide a small example for what you use the projectsEvaluated (for example stuff you mentioned for idea plugin)? I think we can provide an alternative solution to projectsEvaluated if we know a little bit more.

7

One of the simple (aka self contained) examples is configuration of the remote binary repository. In here we have;

  • nexus = something we expose to the build script to let you configure the nexus repo we use - bundle = an extension that lets you customise how a “far jar” is created (basically lets you specify 1-n group/artefact regexes against which the dependencies are matched and if you pass, you get put in the far jar and removed from the pom)

Generally the defaults on these objects (except for username/password) are fine but it’s nice to be able to override them.

Most of our use of these callbacks is in this category, it’s basically a simple view on something more complex under the covers. We don’t need to expose everything to individual developers after all.

project.gradle.projectsEvaluated {
  project.tasks.uploadArchives {
   repositories.mavenDeployer {
    name = 'httpDeployer'
    uniqueVersion = false
    configuration = project.configurations.deployerJars
    repository(url: nexus.getEspritUploadUrl(project.rootProject.maturity)) {
     authentication(userName: nexus.username,
           password: nexus.password)
    }
    snapshotRepository(url: nexus.espritSnapshotsUrl) {
     authentication(userName: nexus.username,
           password: nexus.password)
    }
    customisePublishedArtefacts(project, it as PomFilterContainer)
   }
  }
    void customisePublishedArtefacts(Project project,
         PomFilterContainer pomFilter) {
 if (project.extensions.bundle.bundled) {
   pomFilter.addFilter('normal') { artifact, file ->
   artifact.id.artifactId.name != "${project.name}-complete"
  }
  pomFilter.addFilter('bundle') { artifact, file ->
   artifact.id.artifactId.name == "${project.name}-complete"
  }
  doPomCustomisation(pomFilter.pom('normal'))
  doPomCustomisation(pomFilter.pom('bundle'), project.extensions.bundle.includes)
  pomFilter.pom('bundle').whenConfigured { generatedPom ->
   generatedPom.project { name = "${project.name}-complete" }
  }
 } else {
  doPomCustomisation(pomFilter.pom)
 }
}
   doPomCustomisation(def pom) {
    // assorted configuration on the pom including translating dynamic dependencies into concrete versions
 }
8

What’s the concrete problem here? That you can’t override the values?

9

You could avoid projectsEvaluated hook by using the execution time configuration idiom. E.g. add a task that configures uploadArchives task and make the uploadArchives depend on it.

10

@Peter - there is no problem with this individual piece of code. It’s more a question of comprehending what is going on when there are n other blocks of such code in play and someone needs to make a change to the behaviour. It can be difficult to see the dependencies (or to see that they are independent) because it’s just a bunch of callbacks. I realise this could just be my inexperience with groovy/gradle showing of course.

11

That’s just the nature of such global callbacks. Convention mapping callbacks are much more fine-granular (per property) and therefore easier to reason about.

12

OK but this a bit of a chicken and egg problem then given that convention mapping is not part of the public api and that some things one wants to configure involve configuring the task graph itself (which can’t be fiddled with once it has been created). Convention mapping is fine grained but this means it is naturally limited to injection of specific values (and is quite hard to discover that that is going on) unless I’m missing something about convention mapping?

13

OK but this a bit of a chicken and egg problem then given that convention mapping is not part of the public api

I agree, but it’s the best that Gradle has to offer at this point. You’ll either have to live with certain limitations, or use an internal API.

Convention mapping is fine grained but this means it is naturally limited to injection of specific values

The purpose of convention mapping is to define default values for model or task properties, and to make it possible for them to depend on the values of other model/task properties.

and is quite hard to discover that that is going on

Not sure what you mean by that. Users will be fine as long as the defaults are documented properly. At some point, Gradle will help you with generating such documentation. There is hardly ever a need to reason about when convention mappings kick in - they will do so exactly at the right time (namely when a value is requested) and in the right order (in case one value depends on another). The fine grained nature of convention mappings also helps keeping the code clean.

Even convention mapping doesn’t solve all problems, and that’s why there’s still hesitation to make them public. It’s a big decision because we are extremely serious about guaranteeing long-term backwards compatibility for public features. Nevertheless, when used for its defined purpose (see above), convention mapping is the best known solution that’s available today.

14

The purpose of convention mapping is to define default values for model or task properties, > and to make it possible for them to depend on the values of other model/task properties.

OK. This leaves in place the time when you have things you want to configure that are based on some set of disparate values which then comes back to the global callbacks.

Not sure what you mean by that.

on 2 dimensions; exactly how and when it works & that it is being used. I am now referring to the user community here, I am referring to developers extending these custom plugins. One reason I’ve been uncomfortable using it (as opposed to other internal apis which I am happy to use) is because it’s not obvious to me how it works & exactly what you can or can’t do. I would have to try and infer that from cases where it is used internally which is a risky business. Is there some design doc that explains convention mapping?

15

This leaves in place the time when you have things you want to configure that are based on some set of disparate values which then comes back to the global callbacks.

You can use convention mapping for this as well. For each property you configure, add one convention mapping. The convention mapping’s closure can refer to an arbitrary number of values.

Is there some design doc that explains convention mapping?

I’m not aware of any design docs. The convention mapping’s closure is used to compute the property’s value whenever someone gets the property, until the property has been set once. After that, convention mapping has no more effect. When accessing a property from within the class it’s declared in (assuming the class is written in Groovy), it’s important to use ‘getFoo()’ rather than ‘foo’, because otherwise Groovy will access the field directly (rather than calling the getter), and the convention mapping doesn’t get a chance to kick in.

As I said, that’s all we have today. Good luck!

16

How does one declare something as being aware of conventions? or is this mixed in automatically to some set of things (extensions/things added to containers perhaps?)

17

It’s mixed in automatically for tasks, important domain objects, extensions, and nested extensions. It isn’t available for any object that’s not instantiated by Gradle.