Taking advantage of configuration on demand?


(Adrian Abraham) #1

I have a multiproject build with over 100 subprojects. As much as I’ve tried to optimize the build, the configure stage still takes about 30 seconds. Given that no individual project depends on more than a dozen other projects, I’m hoping that configuration-on-demand will help me out.

Unfortunately, my build is organized with all the build logic in an allprojects{} block in master, leaving each subproject’s build.gradle to declare its dependencies. From the documentation and my own testing in a dummy project, this couples all the projects together in such a way that configuration-on-demand doesn’t do anything.

According to the documentation, “This means that using any form of shared build script logic or configuration injection (allprojects, subprojects, etc.) will cause your projects to be coupled. As we extend the concept of project decoupling and provide features that take advantage of decoupled projects, we will also introduce new features to help you to solve common use cases (like configuration injection) without causing your projects to be coupled.”

Even without these extra features, could I get some guidance on the best way to proceed? The only thing I can come up with is to create master/allprojects.gradle, then use “apply from” in each subproject. I believe this will mimic having an allprojects {} in master, but should effectively decouple the projects from each other. Of course, this is a pain, and it also “smells.”

Is there another strategy I can use to get decoupled shared configuration?


(Adrian Abraham) #2

I just implemented what I described in my initial post. The actual conversion didn’t take too much effort, but its worth is questionable. It seems that now, instead of compiling allprojects{} once, it’s compiling it for every subproject (the initial run took 5 minutes just to compile all the build scripts).

Technically, it is allowing configure-on-demand to work - but it’s a net loss. Given how long each project takes to configure now, if a project has more than a handful of other projects it depends on, configuration takes longer than it used to to configure everything.

I had been thinking about trying to hack something up in settings.gradle to include only the projects I need - unless there’s something I’m missing about configuration-on-demand, I think I’m going to have to revisit that idea.


(Peter Niederwieser) #3

I’m not sure whether a shared build script gets compiled multiple times, but compiled build scripts are cached. Hence there should only be a compilation cost after editing the script.

Configuration time can certainly become a problem for huge builds, but 30 seconds for 100+ projects sounds like a lot, and maybe you are doing unnecessary work in the configuration phase. Typical candidates would be resolving configurations at configuration time, unnecessarily declaring configurations for each project, or setting the same build script class path for each project (rather than just for the root project). If you haven’t already, have a look at the report generated by ‘–profile’, and check if ‘–offline’ makes a difference.

Scripting ‘settings.gradle’ to dynamically include a subset of projects is certainly an option, and you wouldn’t be the first to do this. Here is a proof of concept: https://github.com/pniederw/elastic-deps


(Szczepan Faber) #4

In “Configuration on demand” mode the root project is always configured. So, the easiest way to proceed would be putting your allprojects {}, subprojects {} or other general configuration injection in the root project.

Let me know if it helps.


(Adrian Abraham) #5

As I said, both from my own experience and from the documentation, using allprojects{} or subprojects{} ends up coupling all the projects together. I’ve created a simple project to demonstrate this:

https://github.com/Nazrax/gradle-configuration-on-demand-sample (commit b9acceb)

Obviously, this is a sample and not something I’d do in a real project. I’ve created a file, main.gradle, containing the actual build logic. When run with “gradle-1.4 :projA:hi”, master/build.gradle uses an allproject block to apply main.gradle. When run with “gradle-1.4 :projA:hi -Pinclude=true”, each project individually applies main.gradle.

main.gradle has a statement that’s printed when the project is configured. With the default allprojects{} version, master, projA, and projB are all configured. With the “include” version, only master and projA are configured.

If I’m doing something wrong, please let me know!


(Szczepan Faber) #6

Using allprojects/subprojects in a root project does couple the root with everything else. However, this is the most natural way users implement common configuration. So configuration on demand supports that.

At some point we will make allprojects {} subprojects {} smarter (or move them to settings.gradle, there are a couple of options). Then, the configuration on demand mode will no longer automatically configure the root.

Hope that helps! I’ll try to take a look at your sample some time later.


(Adrian Abraham) #7

Well, I spoke too soon about the efficiency of allprojects{} vs “apply from.” I’ve spent quite a bit of time making sure to properly clear the caches and run things over and over, and it turns out that, while “apply from” seems slightly less efficient, is a small fraction of the total time.

Dependency resolution doesn’t seem to be the problem - according to the profile, the total dependency resolution time is around 0.05 seconds (–offline doesn’t seem to make a difference - probably because I don’t have any SNAPSHOT dependencies).

Most of my projects take a third of a second or less - but across over 100 projects, that time adds up.

I’ve done everything I can think of to reduce configuration time. I’ve created a “lazy configuration” taskGraph.beforeTask block that I use for anything that might cause a lookup or other calculation. i’ve gone over and over my scripts looking for something I’m missing, and I just can’t find anything. Of course, I could still be missing something.


(Luke Daley) #8

@Szcepan should Adrian try using dsl:org.gradle.api.invocation.Gradle:beforeProject(groovy.lang.Closure) ?

@Adrian the difference between this and ‘allprojects {}’ is that it doesn’t actually execute the code then and there. It will only execute it before the project evaluates, which with configure on demand will be a subset.


(Adrian Abraham) #9

@Szcepan

You say that configuration on demand supports allprojects, but I’m not sure what that actually means.

From the example I have on GitHub:

$ gradle-1.4 :projA:hiThanks for using the incubating configuration-on-demand mode. Enjoy it and let us know how it works for you.
Configuring master
Configuring projA
Configuring projB
:projA:hi
projA says hi
  BUILD SUCCESSFUL

You can see it configuring projB, even though projB has nothing to do with :projA:hi. When run using apply from instead of allprojects:

$ gradle-1.4 :projA:hi -Pinclude=true
Thanks for using the incubating configuration-on-demand mode. Enjoy it and let us know how it works for you.
Configuring master
Configuring projA
:projA:hi
projA says hi
  BUILD SUCCESSFUL

Here you can see it skips configuring projB.

This isn’t just a theoretical issue. With my real project, each subproject takes roughly a third of a second to configure. Across 100+ subprojects, that’s a significant amount of time. When I enabled configuration on demand for my unaltered project, nothing changed. It still configures every project. When I cut the allprojects{} block, pasted it into a separate file, and applied it from each subproject, then I configuration on demand actually started to do something, and my configuration times dropped significantly when I call tasks lower down in the project graph.

@Luke beforeProject seems to work nicely. Much better than putting the same “apply from” line in every build.gradle!


(Luke Daley) #10

@Adrian: what impact has beforeProject() had on performance vs ‘allprojects {}’ and spreading ‘apply’ across all the scripts?


(Adrian Abraham) #11

@Luke After running these things over and over, I think all three methods are within the margin of error for taking the same amount of time.


(Adrian Abraham) #12

Ok, so now I have a new problem.

The compile tasks work fine, but the Idea task is broken. I’m assuming it’s not actually the Idea task as such, but some strange thing happening with projects not getting coupled properly.

I’ve updated my example (https://github.com/Nazrax/gradle-configuration-on-demand-sample , commit d55ef47) so that projB depends on projA and both projects have the Java and Idea plugins.

Running the “hi” task from projB does exactly what is expected: it configures master and projB, then says Hi. Running the “compileJava” task from projB does exactly what is expected: it configures master and projB, then projA, then compiles projA and projB.

Running the “idea” task from projB blows up. It configures master and projB, tries to run :projB:ideaModule, and fails:

projB $ gradle-1.4 idea
Thanks for using the incubating configuration-on-demand mode. Enjoy it and let us know how it works for you.
BeforeProject: projB
Main: projB
:projB:ideaModule FAILED
  FAILURE: Build failed with an exception.
  * What went wrong:
Execution failed for task ':projB:ideaModule'.
> Could not resolve all dependencies for configuration ':projB:compile'.
   > Module version group:master, module:projB, version:unspecified, configuration:compile declares a dependency on configuration 'default' which is not declared in the module descriptor for group:master, module:projA, version:unspecified
  * Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
  BUILD FAILED

(Luke Daley) #13

This is expected. ‘allprojects {}’ applies the configuration right there and then at that point. configure-on-demand doesn’t change this. What does change is that the projB’s build.gradle will not be executed.


(Adrian Abraham) #14

Oh. Given that ALL of my build logic is in allprojects {}, and the only thing in the individual subprojects’ build.gradles is a dependencies{} block, that really isn’t very helpful …


(Luke Daley) #15

Right, which is why you should use beforeProject {} which has different semantics which are more compatible with configure on demand.


(Szczepan Faber) #16

Can you try out with latest nightly? http://www.gradle.org/nightly (don’t worry that the nightly is 1.6 - we are in the process of releasing the 1.5 candidate). I suspect that you’re hitting a problem that we have already fixed and it will be available in 1.5.


(Szczepan Faber) #17

Very interesting idea with beforeProject. It should work as advertised.


(Adrian Abraham) #18

Using the nightly seems to have fixed the problem with Idea. However, I can no longer get configuration on demand working from gradle.properties - I’m having to use commandline options. I know that the name of the property changed, and I updated the properties file accordingly. Here’s my (overkill) master/gradle.properties:

systemProp.org.gradle.configuration.ondemand=true
systemProp.org.gradle.configureondemand=true
org.gradle.configuration.ondemand=true
org.gradle.configureondemand=true

Running “gradle-1.6n -Dorg.gradle.configureondemand=true hi” works, though.


(Luke Daley) #19

Try moving gradle.properties to the top level dir, out of master.


(Adrian Abraham) #20

Still doesn’t work.

I added code to master’s build.gradle to dump both rootProject’s properties and the System properties. With gradle.properties in master, all the properties are set properly (4 Gradle properties and 2 Java properties). With gradle.properties in the top level directory, nothing is set.