Multi-project configuration order differs based on where gradle is invoked


(Sam Snyder) #1

Authors of plugins and anyone else who might understand the nuances of their application to multi-project builds perhaps you can help.

The Problem, Briefly

A root project applies a plugin to its subprojects. This works when I invoke gradle from the root project. This breaks when I invoke gradle from a subproject directory, complaining that it does not understand the DSL.

Do I really need to go into the build.gradle for each subproject and add evaluationDependsOn ':parent' to get :parent to be evaluated before its children regardless of where gradle is invoked from?

Edit further experimentation shows this problem is specific to the presence of the --configure-on-demand flag. For performance reasons removing this flag is really not an option for me.

##Some Context

  • Gradle 3.3
  • Multi-project build with ~270 projects, most of them Java
  • Size of build necessitates always using --configure-on-demand together with --parallel
  • Aside from parent project configuring child projects, there is no other cross-project-configuration (projects are decoupled)

##The Problem, not so briefly

I’m attempting to create a plugin which aggregates build and dependency artifacts from sub-projects. Because reasons currently outside of my control there is a many-to-many mapping from project artifacts to where those artifacts need to be copied to

Simplified structure of the buildstructure:

  : (root project)
    :parent
      :child1
      :child2

:parent's build.gradle defines a plugin which creates a DSL on the subprojects of :parent:

class ExampleExtension {
    String foo
}
class ExamplePlugin implements Plugin<Project> {
    void apply(Project project) {
        project.subprojects { subproject ->
            subproject.extensions.create('example', ExampleExtension)
            subproject.afterEvaluate {
                println subproject.example.foo
            }
        }
        
    }
}
apply plugin: ExamplePlugin

Allowing the build.gradle in :parent:child1 and :parent:child2 to have a DSL like so:

example {
     foo = 'bar'
}

So now if I’m :parent's directory and I invoke gradle everything works just like I expect it to! I see 'bar' printed once during configuration time for each of :parent's subprojects.

But if I cd into the directory of :parent:child1 or :parent:child2 configuration of those projects will fail with a message indicating that ExampleExtension has not yet been applied, demonstrating that I don’t understand project evaluation order as well as I thought I did:

* What went wrong:
A problem occurred evaluating project ':parent:child1'.
> Could not find method example() for arguments [build_7z4mibcpx3v4soamzppwy1fnu$_run_closure2@6281add3] on project ':parent:child1' of type org.gradle.api.Project.

Again, if I go into each child project and add an evaluationDependsOn ':parent' it works.
But I thought that parent projects were already evaluated before their children? There are examples of parent projects configuring their child projects throughout the documentation. What am I missing about project evaluation order/plugin application?


(Sam Snyder) #2

Additional information:
gradlew :parent:child1:taskname --configure-on-demand --parallel fails
gradlew :parent:child1:taskname --configure-on-demand fails
gradlew :parent:child1:taskname --parallel passes
gradlew :parent:child1:taskname passes
gradlew taskname passes regardless of any combination of --configure-on-demand or --parallel

So it looks like this evaluation ordering problem is specific to the presence of --configure-on-demand and not affected by --parallel


(Stefan Oehme) #3

--configure-on-demand only evaulates the project you call and anything it declares as a dependency. But putting evaluationDependsOn everywhere is probably not what you want, see below.

This is counter to the idea of configure-on-demand. If this root project is evaluated, then it will apply the plugin to all subprojects, no matter if they will be needed or not. Using the subprojects or allprojects block defeats the performance benefit of configure-on-demand.

The configure-on-demand-friendly solution for your specific problem would be splitting this into two plugins: One plugin for the “aggregator” (the parent in your case) and one plugin for the “provider” (the children in your case). The children would the apply the provider plugin where applicable and the parent would use the aggregator plugin to collect all the results from the children that have the provider plugin.

Another question is: How slow is your configuration that it makes you want --configure-on-demand? Usually there is a lot to improve. Even for large projects configuration should not exceed a few seconds.


(Sam Snyder) #4

Stefan, thank you for taking a look at my post and providing insight into the guiding philosophy behind --configure-on-demand

Regarding project coupling
The example I provided was simplified. In the case where I’m using subprojects{} it configures 164 out of a total 283 projects. So there us still be some benefit to using --configure-on-demand even if its efficacy is diminished by subprojects{}.

Is there a way to enable a some kind of “strict warnings” mode that would complain when something performed --conifgure-on-demand incompatible cross-project coupling? Sometimes I miss code reviews where that sneaks in.

Regarding project configuration duration

Configuring the 283 projects in our multi-project build takes 15.3s on my dev box according to single run I just did with --profile. I’ve produced this little histogram showing how many projects can be bucketed into each configuration duration. There are a couple bad apples off the right edge of the chart that take 0.5s and 1.0s respectively to configure. But I’m aware of those and will sort them out.

So the biggest bucket shows 92 projects which each take ~0.04s to configure, or cumulatively 3.68s.
0.04s per typical project doesn’t seem terribly long so I don’t know if this shows much opportunity to optimize except by configuring/building fewer projects. A typical project will have the java, eclipse-wtp, idea, checkstyle, and dependency-check plugins applied, along with a couple script plugin which configures defaults we like.

I’d be happy for you to tell me that there was some way to make configuring java projects much faster!
It’s our intention to split this big build into at least 2 or 3 smaller builds using Composite Builds but haven’t really experimented with that yet.

Regarding your suggestions on plugin architecture
I like your suggestion of splitting the plugin into a producer and a consumer.

I was getting clever with the “producer” part adding methods to the metaclass of the “consumer”'s DSL extension, allowing there to be a single source of truth about the domain and a very terse DSL syntax.
If the “producer” and “consumer” parts are decoupled into separate plugins there won’t be any way for the “producer” to inform the “consumer” about what data it should accept.
But I can just have the “consumer” part accept a map whose semantics it knows nothing about and do all the validation in the “producer”

I’ll try that out, thanks!