Best strategy for large (unbounded) multi-project builds

Help/Discuss
1

Hello, we have an existing build system based on Ant, where the total number of tasks is unbounded in size, in that they will continue to grow for as long as the production department is working.

Consider the following layout:

.
├── common
└── courses
├── a1
│ ├── b11
│ │ ├── c111
│ │ ├── c112
│ │ └── c113
│ └── b12
│ ├── c121
│ ├── c122
│ └── c123
└── a2
├── b21
│ ├── c211
│ ├── c212
│ └── c213
└── b22
├── c221
├── c222
└── c223

A, B, and C represent a project types that are similar in nature, but will continue to grow in size. Currently we are using the SubAnt task to filter out and include any set of the above into a final deployable package.

The problem is that I have hit a wall scaling this up any further (than is practical) due to restrictions in java PermGenSpace size. To solve this problem, I have been looking into using Gradle to run parallel build process (all the A’s, then all the A-B’s, etc.), but I’m having trouble figuring out the best way to set this up.

It appears to me the “Gralde way” is to use the steps described in the multi_project_builds docs, with a gradle.settings at the content root. However, it would not be practical to define all the includes without being able to generate them somehow.

I think what I need perhaps is a composite build, but it was unclear to me how to run the added participants in parallel.

Any ideas to get me started would be greatly appreciated.

Cheers and thanks for your time!

2

Depending on the nature of the tasks, you may be able to use task-rules (don’t confuse with model-rules)

The task-rules create on-the-fly tasks, depending on what you requested on the command line. You have full flexibility how you define these tasks - you may just parameterize using parts of the requested task name, or you can parse XML files and call out to service registries (which is obviously a bad idea, still - you can).

3

Ah, that looks like a good place to start, thanks! I started down the model-rules path at first, but it didn’t seem sustainable for O(n^3) sized models.

The C type projects are the most computationally expensive, while the A’s are primarily config settings (dependency selection, etc.), and the B’s spend most of their time waiting for the C’s. There are the basic parallel options for ant, but I was hoping to use this opportunity to improve the overall build architecture, and set up some kind of fly-weight executer system for running jobs (we have CI, but needs to work offline as well).

I would still love to hear if anyone else has some insights they would like to share.

Cheers and thank you!