Hiearchical multi-project support

[rlf (Razorax)]

This is a repost of (http://forums.gradle.org/gradle/topics/multi-projects-in-gradle-lack-hiearchical-properties) that seems to have disappeared.

The current multi-project-support in Gradle is lacking all the properties that makes gradle awesome.

It imposes the following on the structuring of a multi-project build:
a) ONE settings.gradle file in the ROOT of the monolith
b) All inter-project-dependencies declared with fully-qualified-path
c) All invocations of gradle performed at root-level

These three constraints makes managing more than tutorial-sized projects almost impossible (without the help of the pride project).

Some features that would make the multi-project managable:
a) Multiple settings.gradle files (i.e. the posibility of using ‘:superprojectA’ in another settings.gradle file, and having ‘:superprojectA:A-Z’ defined only locally).
b) Project resolving more flexible (i.e. for referencing ‘superprojectA:C’ within superprojectA, ‘:C’ should suffice.)

I’ve searched high and low for some “gradley-way” to solve this, but pride (https://github.com/prezi/pride) has so far been the only remotely viable option.

It pains me, since I’m trying to be the “gradle-ambassador”, but the multi-project support is hard to defend, against any of the alternatives.

[mark_vieira]

@rlf

b) All inter-project-dependencies declared with fully-qualified-path

Relative paths can be used to find subprojects. I can reference project ‘projectA:C’ from ‘projectA’ by simply omitting the the path prefix (ex. project('C')). As stated in the Prezi Pride notes however, relative paths to ancestor projects isn’t possible.

@rlf

c) All invocations of gradle performed at root-level

You can invoke a build at any level of the project hierarchy. Gradle will search upwards for a settings.gradle file and configure the multi-project build. This can be tricky however when using the Gradle wrapper, although there are third-party scripts available that make this more transparent.

[hansd (Hans Dockter)]

@rlf

This is a repost of (http://forums.gradle.org/gradle/topics/multi-projects-in-gradle-lack-hiearchical-properties) that seems to have disappeared.

Sorry for the inconvenience. We had some loss of content during the migration for various reasons. We are still waiting for a data dump with the remaining content from our old forum provider.

@rlf

c) All invocations of gradle performed at root-level

If you are using the wrapper, yes. Otherwise not. We want to fix that.

@rlf

These three constraints makes managing more than tutorial-sized projects almost impossible (without the help of the pride project).

I would not agree with that statement. There are some very powerful and flexible multi-module builds out there with hundreds or even thousands of submodules. But to get to this power you have to do more than what should be required. You have a lot of flexibility in what you can do dynamically in the settings.gradle.

@rlf

a) Multiple settings.gradle files (i.e. the posibility of using ‘:superprojectA’ in another settings.gradle file, and having ‘:superprojectA:A-Z’ defined only locally).

This is I guess the biggest flaw of the current approach. We will eventually get rid of the singularity of a settings.gradle and fully allow for composites.

@rlf

I’ve searched high and low for some “gradley-way” to solve this, but pride (GitHub - prezi/pride: Manages a pride of Gradle modules) has so far been the only remotely viable option.

We are working together with Prezi on making Pride a core concept. What you will see in 2.5 (most of the code is already in master) is that external dependencies and project dependencies are interchangeable.

@rlf

It pains me, since I’m trying to be the “gradle-ambassador”, but the multi-project support is hard to defend, against any of the alternatives.

We will make a lot of progress this year on improving this. I would be very interested in learning more about your concrete organizational use cases behind the features you mentioned that you would like to have.

[rlf (Razorax)]

Wow, thanks for all the fast feedback.

First, let me confess, my Groovy/Gradle knowledge is somewhat limited (still a newb).

Reg. c) Yes, it was a blunder on my part (multiple settings.gradle files, making the location of invocation quite important to success).

Integrating some of the pride-features into gradle sounds like a fantastic idea, and if it will already be available in v2.5 - I can easily defend the somewhat cumbersome multi-project structure until then (though, the critics will most likely just argue to postpone the move to gradle).

Reg. the “absoluteness”, “relative” and “flexible multi-module” projects, I’d love to see how that flexibility is realised (do you have a good open-source example?).

What we have, is a 12 year old “software-asset” based on a multitude of Ant, Make, Bash and Velocity scripts.
Naturally, the modularization of these is somewhat heterogeneous, but I’ll try to exemplify my conundrum.

BusinessModuleA
  gwtModules
    gwtA [dependsOn: BusinessModuleA/model, BusinessModuleA/j2eeModules/servletModule]
  j2eeModules
    j2eeA
    servletModule
  model [dependsOn: Core/model, Framework/model]
BusinessModuleB
  gwtModules
    gwtB [dependsOn: gwtC]
    gwtC
Framework
  j2eeModules
    commonServlets
    abstractj2ee
  model [dependsOn: Core/model]
Core
  model

I know, this is a mess, but, with the “current-gradle-multi-project-model” (and my limited knowledge of Gradle), the most “straigtforward” way to get this to “work”, is to place a settings.gradle file in the root of it all.

It would contain stuff like:

include ":BusinessModuleB:gwtModules:gwtB", :BusinessModuleB:gwtModules:gwtC"

Which is ugly, but managable (pride can actually help here, at least for generation of the files), so we can have a “pure gradle build environment”, and only use pride when adding/removing modules.

The real problem arise, when sub-modules need to refer to each other.

Since there is no “resolving of nearest module”, I can’t do stuff like:

dependencies {
  compile project(":gwtC")
}

in the gwtB module, but need to include the full path of the gwtC module (which is relative to the settings.gradle file).

This makes maintaining all the build.gradle files hard, verbose and dependent on parent-structure (i.e. refactoring/moving a module would not be easy, I’d have to manually search-and-replace all the build.gradle files).

Perhaps all this lies on some misconception on my part - which is why, a “good example” of mid-sized multi-project with multi-module dependency resolving would be most appreciated.

[hansd (Hans Dockter)]

@rlf

I know, this is a mess, but, with the “current-gradle-multi-project-model” (and my limited knowledge of Gradle), the most “straigtforward” way to get this to “work”, is to place a settings.gradle file in the root of it all.

Does all the code lives in one source repository?

@rlf

in the gwtB module, but need to include the full path of the gwtC module (which is relative to the settings.gradle file).

Not sure if this will help in your case, but I would like to mention that In a Gradle build the multi-project structure can be fully separated from the physical structure. For example

BusinessModuleA
  gwtModules
    gwtA [dependsOn: BusinessModuleA/model, BusinessModuleA/j2eeModules/servletModule]
  j2eeModules
    j2eeA
    servletModule
  model [dependsOn: Core/model, Framework/model]

Can be mapped to:

:gwtA
:j2eeA
:servletModule
:j2ee-model
:core-model
:framework-model

Check out https://github.com/gradle/gradle/blob/master/settings.gradle for how you can assign a project dir to a Gradle subproject you have declared.

[rlf (Razorax)]

Yes, we’ve got all the inter-dependent code in the same git repository atm.

I knew about the multi-project structure being separated from the physical structure, but I hadn’t connected the dots like you just did

At least that way, we can avoid some of the frustrations, when referencing other projects deep inside our structure.

I’ll give it a spin, and hopefully it will be manageable for the time being.