Lazy build script evaluation

Recently I have been thinking about how can be Gradle more performant when doing partial builds. By partial builds, I mean only building a single project (and its dependencies) of a multi-project build. As is currently, Gradle eagerly executes each project’s script before attempting to run a task. Although it is a more safe approach (since any Gradle project might adjust the configuration of any other project), I believe that assuming that an unrelated project might adjust another project does not add more power to build scripts but hinders optimzations (especially in case of partial builds). That is, it should be sufficient for a build script to rely on itself; and no build script may assume that some other script configures it (code can still be separated in multiple files using “apply from”).

A more performant approach

The following proposal is likely to break lots of builds out there, so if implemented, must be opt-in.

To my understanding, as of currently, Gradle works in two phases: configure, run tasks. This clear separation is not necessary. Here is how I believe, Gradle could work (after reading settings.gradle):

  1. Execute the build script of the currently selected project (the one whose directory is the current working directory).

  2. Execute the build script of the child projects of the selected project recursively.

  3. Start executing the requested tasks.

In the above code, whenever an ‘org.gradle.api.Project’ instance is retrieved (e.g.: ‘project(":parent-project:sub-project")’), Gradle should ensure that before the Project instance is returned to the caller, the script of the project to be returned is executed (except when prevention of infinite loop in the initialization is required). That is, in a very similar manner as the static initializers in Java are executed.

Benefits

Currently, I often find myself not adding a project to a multi-project build because the project is rarely needed and adding projects considerably slows down unrelated builds. The proposed changes would make Gradle scale better with projects containing lots of subprojects; even in the case of partial builds. The cost of adding a project to a multi-project build would be negligible.

With IDEs it would be possible not to load each project if you are not interested in each of them at the moment.

Does this address what you are looking for? Gradle’s project configuration model design doc.

I think this attempts to address the same or similar issue but the document you pointed to chooses a different kind of solution. It completely disallows accessing other projects. This seems very restrictive to me because at the very least I want to access child projects. For example, create merged javadoc. I believe that my proposal is more compatible with existing scripts. Obviously, the disadvantage is that my proposal allows for somewhat less optimization.

Our proposal allows access to any outputs of other projects, just not to their Gradle object model. I’t not just about optimization, but also about partial checkouts, distributed builds, etc.

What do you mean on “output”? The files generated? That is not enough for a merged javadoc. I actually need a reference to the ‘org.gradle.api.Project’ instance because I need to gather the source paths of those projects (and the dependencies) to pass it as an input for the javadoc command.

I mean everything that it produces, in one way or another. This could include information necessary to produce a merged Javadoc.

We have discussed solutions similar to the one you proposed, and came to the conclusion that to make the next big step, we have to make more drastic changes. To continue your example, we would want merging of Javadoc to work even if individual projects are built on different machines at different times, with their outputs published to a binary repository.

I see. When do you expect these changes to be actually implemented in Gradle?

I expect us to make incremental steps towards that goal as we continue to work on parallel, and later distributed, execution. I don’t have an ETA.

Hey Attila, thanks for feedback and suggestions! We care very much about scalability and we have a plan for incremental improvements.