Hey.
We’ve just migrated our Android team’s git repos to a monorepo with the Gradle build system at its root. It consists of 3 large apps and multiple small libraries (150k lines of code in total). The issue is that devs are getting lots of performance issues when opening the monorepo in the Android studio regarding build time and indexing (find usages, navigating in the project, …). What have we potentially done wrong in the process of migration and what are the solutions for this type of isses?
Having a monorepo does not mean you need to have a monobuild.
If all code is one build, then of course the IDE needs to sync it all and index it all and so on.
I guess it would be better if all the libraries are standalone builds that you then compose using composite builds.
That way if you open project A, only those projects that it uses are synced to the IDE, or you could also open a library project directly, only syncing that and what it uses.
So you mean there should be multiple separated composite builds for each app which connect together inside a monorepo and separated IDE instances are opened for each app?
We’re currently doing exactly this approach; but there are some constraints regarding shared libs and some build scripts.
I’ve seen large monorepos like Gradle and Kotlin and they use composite build for some of the few subprojects and projects are mostly included as non-composite. Opening the seperated projects in those massive monorepos result in gradle build error. Is there any other tool that people use in those repos? Or what?
Plus, I’ve read about performance advantages for non-composite builds vs composite builds.
So you mean there should be multiple separated composite builds for each app which connect together inside a monorepo and separated IDE instances are opened for each app?
Not the right wording, but I think you got what I meant. 
We’re currently doing exactly this approach; but there are some constraints regarding shared libs and some build scripts.
If you don’t name them, I can hardly say anything about it.
I’ve seen large monorepos like Gradle and Kotlin and they use composite build for some of the few subprojects and projects are mostly included as non-composite.
Don’t know about Kotlin, but where is Gradle a “monorepo”?
A monorepo for me means that multiple “independent” apps are in one VCS repository, which is fine if they are versioned, branched, and released uniformly together so that you can also have clean tags and so on, but is imho bad practice if that point does not hold.
The Gradle repo is one build from which one product is built.
It could be split into multiple builds that are combined using a composite build, but it would probably not help too much there, as you anyway want all together in the IDE.
Opening the seperated projects in those massive monorepos result in gradle build error.
I don’t know what you mean, I never had a problem to open the Gradle or Kotlin projects in my IDE.
You might have given the IDE too little max heap for such big projects, but other than that I never had substantial problems opening them.
Is there any other tool that people use in those repos? Or what?
Other tool than what?
I don’t know what “people” do, I just know what I do, and I just open those projects in my IDE.
Plus, I’ve read about performance advantages for non-composite builds vs composite builds.
Like always, either solution has its pros and cons.
And also such a statement cannot be done generically.
In some situations one might perform better, in some situations the other.
This is highly project-specific and you just have to find the way that works best for your actual use-case.
I just provided another alternative to what you described and from what you said is the problem.
You said the problem is, that the whole all-in-one-mega build needs ages to sync and index in the IDE and that this is your current main concern.
And if you would have separate builds that are combined in some composite builds, you could easily open individual projects in the IDE where much less has to be synced and indexed.
Whether that is the right solution for you or not, you have to find out.
Thanks for your detailed answer.
Let me rephrase and clarify the problems once more:
-
The whole purpose of the monorepo migration for our case was the ability to share code without having the concern to maintain different versions of shared libraries. When using composite build for each app, the shared library inclusion is not so natural (using Gradle’s
dependencySubstitutionand such). -
150k lines of code in our monorepo causes IDE to perform poorly. Directory exclusion reduces the issue however it’s not that helpful. Opening each project’s directory separately inside new IDE instance fixes the issue.
How do large repos like Gradle’s developers use their IDE without getting any performance loss with that much lines of code inside those repos?
As you said, Gradle is a one build project too, so I assume Directory exclusion and other “Decreasing time for indexing” methods or opening a new IDE instance at a certain directory could not work in that case too! So what am I missing here?
Additional data: Our team mostly use M1 & M2 pro Macbooks and 16 gb of ram with IDE at its full heap size. Also, we have 3 Android apps and ~15 shared libraries (which currently are used using regular repository publishing and with artifact ID and version and all of those things!) in our monorepo.
When using composite build for each app, the shared library inclusion is not so natural (using Gradle’s
dependencySubstitutionand such).
If you need to use dependencySubstitution, you did something wrongly. If you did it properly, you just declare a dependency using the coordinates instead of project("...") and it just works. The manual dependencySubstitution is mainly for the case where you need to include other projects that did not do it properly and thus require you to do a manual substitution.
150k lines of code in our monorepo causes IDE to perform poorly.
Maybe you should give it more max heap as I suggested?
Our main project at work has over 2m lines of code and the IDE performs quite well with it without significant problems.
How do large repos like Gradle’s developers use their IDE without getting any performance loss with that much lines of code inside those repos?
I have no idea what the Gradle folks do, I’m just a user like you, this is mainly a users-help-users community.
And as I said, when I occasionally open the Gradle project in the IDE I also do not have significant performance problems.
Of course syncing and indexing needs a moment, but then it performs fine and you don’t continously sync and reindex.
with IDE at its full heap size
That’s most probably the problem as I said.
If it is as full heap size, that means it needs more and could quite well perform bad.
Increase the max heap size.