Composite Build Workflow

#1

When trying to include a project into another project via composite build, there is a restriction that states:

“must not itself be a composite build.”

Please confirm that I am understanding correctly:

“Core Library” has a number of core classes for our organization.

“General Accounting Library” project is configured to be a composite build, and includes “Core Library”.

“Accounting-EU Library” project is also configured to be a composite build, but cannot include “General Accounting Library” because “General Accounting Library” is a composite build itself.

Perhaps I am missing something, please clarify if so.

If not, then I think this is a common dependency scenario that I would vote to be supported in a future release.

(Stefan Oehme) #2

Composite builds can’t be nested at the moment, it’s on our list of improvements though. For now you could include “core”, “general” and “accounting-eu” in one big composite and they will be able to reference each other.

#3

Ok, thank you for confirming, it does seem like a pretty big limitation after all then.

Glad to hear it’s on the roadmap. It was a little concerning because it’s not mentioned in the section called:

“future plans for composite builds”

I would definitely suggest you add it there.

I assume it will be a significant challenge to add support for this scenario (otherwise it would probably have been supported out of the box). Can you confirm if we should expect to see this soon, or a ways down the road?

(Stefan Oehme) #4

We added it in the latest nightly.

It would help us prioritize if you could share more about your use case, why it needs to be hierarchical etc. I wouldn’t count on seeing hierarchical composites this year.

#5

Thanks for the prompt and candid feedback!

Here is a good example:

This github repo contains 4 projects that are related, we’ll talk about 3:

  1. client-runtime . This is the equivalent of “core” in my example. It is the foundation of a number of other Microsoft libraries (any service that has a REST API, and has moved to Swagger), and thus it is a widely used dependency. It provides a bit of a “preconfigured” REST Client. Despite being in the same github project as the others, it is an independent project, and needs to be compilable and testable on it’s own.

  2. azure-client-runtime. This is one of the libraries that depends directly on client-runtime. It has a number of even more specific customizations for Azure. Again, it should be an independent project with an independent build.

  3. Countless libraries generated from swagger specs which depend on these.
    https://github.com/Azure/azure-rest-api-specs
    The current process for customers for working with Azure API is to generate the source files from these swagger specs, create a project for them, and build. All of these have dependencies on #2 above (azure-client-runtime). Microsoft staff needs to be able to generate source from the swagger specs, and test them alongside the other two libraries and develop the three of them together, This is the perfect use case for composite build feature.

I hope that helps.

I guess one thing that confuses me is the suggestion to make them part of one big composite build. I thought the point was to stitch separate independent builds together without needing an over-arching project or “build” that encompasses all the transitive dependency projects manually. I feel like this suggestion is closer to a multi-project build than it is to the “idea behind” composite builds.

Again, it’s likely that I don’t understand things clearly, but please let me know your thoughts.

(Stefan Oehme) #6

Thank you for the detailed example! I think this is a very good case for nested composites. It’s the “spitting a monolith” use case that we have in mind for nested composites.

It’s not the perfect solution for your use case for sure. But it’s a workaround that might be acceptable for now. You could also use a for-loop to iterate over builds to make your life a little easier.

(Ryan Gustafson) #7

I can add a bit to what advanced composite building looks like for our usage…

We have an entire DAG of projects (think dozens) which reference each other, and can initiate a build from within any one of them. The selection of which project (and it’s upstream dependencies) one wants to build is not constrained. There is not single natural starting point, any project can be a starting point depending on what you are trying to do at any given moment.

The ‘includeBuild’ does not seem to have an option to support the absence of the directory on the file system, and devolve into standard dependency resolution. This is useful, because for the DAG above, you may have an arbitrary subset of the dependencies available locally. It would be preferred if the settings.gradle could be setup one time, permanently, to support whatever dynamic subset of projects you may have locally in a canonical file system arrangement. There is no need to re-setup Gradle again to build whatever subset you have local this minute, this would be rather onerous.

The customization we currently use in the settings.gradle, which we call ‘includeFlatIfLocal’, and predates the new composite feature, supports:

  • Every projects being “composite” or not.
  • The including project overrides the “composite” settings from the included project.
  • Compositing is conditional on the project directory being present. If absent, standard dependency resolution is used.

Finally, while not an issue with compositing or our customization, what I would truly prefer is to not have to configure compositing at all (per project), and merely have told Gradle once, “please resolve whatever project dependencies you can find in these directories, I’ve got exactly the stuff I want wired together sitting there. And here are the exceptions, if any, …”. Unfortunately, the blocker I’ve had for such a feature is that settings.gradle doesn’t know what dependencies it’s going to need to use in the build.gradle, and thus it is not possible to dynamically determine what to “includeFlat”.

#8

Indeed this is a fantastic example, and your insight from engineering your own solution could be valuable if somehow the Gradle team got a look into it.

(Stefan Oehme) #9

Hi Ryan,

includeBuild is just a plain old Java method, so you can surround it with for-loops, if-conditions etc. Have a look at the sample I showed in my blog post.

You can do that too, by extracting that logic into a plugin which you apply to all your settings.gradle files.

Cheers,
Stefan

(Ryan Gustafson) #10

Hi Stefan,

Thanks for the response!

includeBuild is just a plain old Java method, so you can surround it with for-loops, if-conditions etc. Have a look at the sample I showed in my blog post.

As is includeFlat, which we already conditionally wrap in our customization. Until includeBuild has transitivity and defaults to conditional inclusion, I don’t see why we would use it, as we have both already. I did look at the blog post, nice examples for the new feature, but stuff we’ve been doing for a few years now with our customization.

Dropping customizations and adopting the built-in gradle solutions is what we’d prefer to do.

You can do that too, by extracting that logic into a plugin which you apply to all your settings.gradle files.

Really? I’d love to know what magic lets me reliably read the dependencies from the build.gradle from within the settings.gradle. If we could do this, then we would not have to list the transitive closure of project dependencies in each project’s settings.gradle (you basically do this too in the “Splitting Monoliths” example of your blog post). Would love to not have to do that! But this eludes me and runs counter to the Gradle lifecycle as I understand it.

If you’re implying to blindly include the dozens of unrelated projects in a directory, that doesn’t really work. That would be very slow build configure and quite likely may not even work (e.g. thanks to currently unrelated broken project Fubar). Only the related projects should be configured in the build (e.g local transitive dependencies).

As far as a “plugin”, we currently use the apply from: construct to grab script from a well known internal repository location. Is there support for dynamic plugin resolution in settings.gradle now? Maybe we’re not thinking the same thing.

All of this is really just looking for built-in support to default to dynamically resolving to local projects instead of external dependencies. Whether this is resolved from a directory when using command line, or from a collection of projects in an IDE, it is the same problem. There are reoccurring patterns of general build composition which should just work with practically no effort. Currently this requires maintaining customizations/kludges to make this work in Gradle and the IDEs. When transitivity is added to compositing, it may reduce some of the work, but it is still anything less than automatic.

You’re close when you can do: ‘git clone foo; git clone bar; git clone baz; git clone boz; cd boz; gradle build’ with standard Gradle, and without a pile of Gradle customizations, repetitious settings.gradle files, and command line configuration (e.g. no --include-build).

(nathan.niesen) #11

This is a stripped down version of our composite plugin and boilerplate scripts that Ryan mentioned. I also included a skeleton project structure containing some of the projects with stripped down settings and build.gradle files showing the DAG. We normally pull the boilerplate scripts directly from our repo with URL but I changed those to relative files to simplify it.

composite-projects.zip (8.9 KB)

Briefly:

  • In settings.gradle, the includeFlatIfLocal only adds the includeFlat project dependency if the project is on the local file system.
  • In build.gradle dependencies block, the composite.resolveDependency is used dynamically create a project dependency or just return the normal dependency.

The DAG looks like this:
myco-services ----> core-persistence-repository -> core-persistence-model --> core -> test
…\–> model-transform ------------------------------------------------/
…\-> expression-language -----------------------------------------/

Note: I only showed one “top-level” application (myco-services) but we have multiple and the DAGs widely vary.

Organization help!
(Stefan Oehme) #12

There is currently no plan to expand composite builds to do this kind of automatic detection. You can however build it yourself by combining composite builds ( to compose projects) with the tooling api ( to find out their dependencies so you know what to compose).

(nathan.niesen) #13

Then you might want to update the documentation to reflect that:
Section 10.1: “By default, Gradle will attempt to determine the dependencies that can be substituted by an included build.”