Gradle composite builds and setup for breaking changes?

We have an automated build system that does the following that we documented here

Now, for a mono-repo, it’s very important to not build the world and only build what needs to be tested from the changes. (This is done at twitter so they can confirm they are not breaking master for any other teams). sooooooo, I am wondering how to tie this into

  • Composite builds
  • Git changed files

in a way that if I change files in library A, then library A build is run, AND then project C and D are run with the changed library. I do not need library B nor project E and F compiled or run here since they do not depend on library A.

How is this done with composite builds?

referencing @st_oehme in case you know this one too?

It may be nice if we can run the git command ourselves AND pass gradle the changed files such that gradle will determine what needs to be build. This is what pants build system does(except it queries git) so that it can build only part of the monorepo each auto build that goes off.

Ok, so I found this https://github.com/zladovan/monorepo WHICH honestly, I ‘think’ gradle should be doing and in fact, he talks about doing a gradle plugin BUT I think it needs to be a full feature. Where to do a feature request.

I think twitter screwed up their mono-repo in one way that gradle SHOULD fix if they wanted. For every project B that ‘includes’ a library A, force the developer(or break the build) to have library A declare that project B depends on it. This is MAJOR in the fact the the team owning library A must approve. If you don’t do this, what happened at twitter is things got out of control and everyone would start depending on a library without even telling the team that owned that library. This solves TWO things

  • notification of the team owning library A they have a new customer and they give it a ship it
  • when the CI system calls build, AND files were changes in library A, it can build library A, then walk up the chain and build project B.

thanks,
Dean

This is WAY WAY more on the topic of monobuilds from having been deep into this at twitter.

  1. If you are a core library developer, you need a MUCH crazier build system that can kick OFF MANY remote builds at the same time. local just won’t suffice if you are a local dev
  2. If you are service developer in a department with a ‘few’ libraries just for your department, you need a local runMonoBuild.sh that detects file changes -> maps file changes to projects(gradle should do this!!!) -> maps projects changed to projects need to build/validate -> build all projects locally

IN ALL CASES, there is a step of mapping projects changed to projects that depend on that project. For this, imagine a repo where you end up with 3000 directories. you DO NOT WANT to scan the whole repo to see who depends on library X. Instead, library X in a composite build ‘should’ be required (only if some gradle setting is set) to declare ‘who’ depends on him (and the library that depends on him perhaps does the same. While this is a bit of extra work for the developer, it does 2 things

  1. high performance on traversing who depends on library X directly or indirectly as in library X -> library Y -> service B is known right away
  2. CRITICAL -> now companies have a way of slowing down just anyone depending on my teams library so that we can discuss or refer to the library they should use. People reach in all the time and this is not good
  3. VERY VERY well documented who depends on my library in my mono-repo

I want to get gradle to a point where it ‘could’ replace the twitter mono-repo build system but it is not quite there yet.