In my team we’ve been noticing we sometimes get really slow builds when switching between branches and we think we’ve tracked it down to our build cache getting invalidated because a library has been added to or removed from the version catalog.
Android Studio’s build analyser tells us that tasks are re-running because the classpath has changed. I assume because the version catalog’s generated code has changed. Feels like quite a harsh punishment.
Keeping the build cache valid for as long as possible is pretty important to us. Is there a workaround for such a thing? We’re using build.gradle.kts and I’m now wondering if there’s the same issue for groovy or not.
We also share the version catalog with an includeBuild where we keep a custom build plugin. We’re considering isolating that with it’s own version catalog too.
Yes, if you add or remove an entry in the version catalog, the generated accessors change and thus the build classpath changes.
Only changing versions in the version catalog does not change the build classpath, which is part of the design.
Maybe you should embrace the build cache to mitigate this problem, so that worthy build results can be reused when switching branches.
Or you could instead work with different worktrees instead, so that within the same worktree you do not hit that “problem”.
Thanks for the reply! We are indeed using the build cache, but it feels like a wasted effort when it’s so fragile that adding a single dependency invalidates it and triggers a full rebuild. I think we would consider going back to a type unsafe solution for the amount of time we lose.
If you mean changing from Kotlin DSL to Groovy DSL to not need to generate those accessors - while I’m not sure whether that would help - imho it is a bad idea. The pros of the Kotlin DSL are imho much greater than the cons including that. But do whatever works best for you.
I agree, we wouldn’t want to switch back to Groovy. I was more thinking about a build flag to turn off the accessor generation. I guess we would then be left with something a bit more like a platform BOM where you would specify dependencies as module notation strings without versions.
For my team, we’re talking about the difference between a 2-3 minute rebuild and a 20-40 minute rebuild, all because a dependency accessor gets added. Perhaps I’m missing something and we should expect the build cache to remain valid through this? Changing a dependency version can also add a new transitive dependency so I don’t really see why this is different. It strikes we that version catalog is basically a plugin that gets applied to every module. If there’s a cost to pay it should only be paid by the module that is adding a new dependency, not all of them.
I was also thinking about programatically creating a version catalog with every possible dependency we might ever want and find some way to fill in the correct version for the ones we actually use, but that feels a bit silly.
I was more thinking about a build flag to turn off the accessor generation.
I’m not aware how you would do that.
I guess we would then be left with something a bit more like a platform BOM where you would specify dependencies as module notation strings without versions.
platform and version catalog are sibling concepts that can also be used together, they are not alternatives
For my team, we’re talking about the difference between a 2-3 minute rebuild and a 20-40 minute rebuild, all because a dependency accessor gets added.
That’s really strange, you should probably check why stuff really is out of date.
I just tested in my play project.
I added new version catalog entries, removed version catalog entries, even added and removed their usage in the build script.
The tasks stayed up-to-date, or when also changing other task inputs forth and back were taken from the build cache.
So while I thought things would get out-of-date, it is indeed not the case and you probably should investigate what your real issue is.
If you still think it is due to version catalog entries being added or removed, please provide an MCVE that demonstrates the problem.
I am able to repro in a fresh kotlin project created from IntelliJ template and a fresh Android project created from template in Android Studio. I add org.gradle.caching=true in the gradle.properties of both to enable the build cache.
With the kotlin project I can see that
after building once, adding a library to libs.versions.toml means that I get > Task :compileKotlin which indicates a re-run
cleaning and running again I get > Task :compileKotlin FROM-CACHE which indicates the build cache working
clicking build again I get > Task :compileKotlin UP-TO-DATE which indicates no change
With the Android Studio Project I get a bit more info in the Build Analyser tab when I make the version catalog change which says
Reason task ran
Class path of task ':app:compileDebugKotlin' has changed from 8375e050831e772526972c49fbb91885 to a74ddb503978bd746c4e62c543fe6b91.
I assume this is a hash of the class path and it reproducibly changes back and forth as I add and remove a library item to and from libs.versions.toml
What you provided unfortunately is not an MCVE.
And even if I try to build one from the description you provided, it doesn’t match, because the generated Kotlin project does not even have a version catalog.
Here is a link to a zip file of the project with a libs.versions.toml added. I cannot upload it here as I’m a new user.
It can be reproduced by either importing to Intellij and clicking “Build project” in the “Build” menu to repeatedly build with changes to the libs.versions.toml, or from the command line it’s probably best to run ./gradlew assemble --console=plain to get the plain output and see if the :compileKotlin task is re-running or not.
Ah, right, sorry.
The difference is, that I tested with compileJava which is a built-in task and thus is unaffected of these build script classpath changes.
I had a quick closer look.
The accessors generated from the version catalog are put to the buildSrc classloader.
So doing changes that cause different things to be generated, is like doing any other change in buildSrc, making all 3rd party tasks out-of-date as the runtime classpath changes due to that.
If you think there could be some systemic improvement be done, you should open a feature request on GitHub Gradle project. But I guess there cannot be done much about that.
If you change back and forth between two branches, time-intensive things should still be taken from the cache though, so I wonder about that time-increase.
Actually, there is also a work-around, though maybe not too nice, depending on taste.
The buildSrc classloader is a child classloader of the settings classloader.
So plugins present on the settings classloader are not directly affected by buildSrc (and thus version catalog entry) changes.
so if you add kotlin("jvm") version "2.0.0" apply false to the plugins block of the settings script, the compileKotlin task stays up-to-date, even if you change version catalog entries, just like in my test with compileJava.
I’ve verified that this workaround works for us and I’ve tried to see if there was a nice way to programatically add all the plugin dependencies from our libs.versions.toml from a custom Settings plugin, but I can’t see how you can add a Settings plugin from within a Settings plugin. Perhaps that is deliberate restriction. I think I would have to write a lint check to keep these plugins up to date, which could work, but not ideal.
I haven’t checked, but I assume there are downsides to this. I expect that an update to a plugin that we add to settings.gradle will have a bigger affect on the build cache than before.
I think I will. This feels a bit unexpected. Perhaps there should be some better class path isolation. I imagine that the version catalog generated code should be applied like any other plugin instead of contaminating all third party plugins. Not sure on the details, but feels like something should be possible.
but I can’t see how you can add a Settings plugin from within a Settings plugin
If you mean put it to the classpath, you need to add it as dependency. Dynamically this is not possible - at least not without significant dark magic - afaik.
I haven’t checked, but I assume there are downsides to this. I expect that an update to a plugin that we add to settings.gradle will have a bigger affect on the build cache than before.
Definitely, it is always a question of tradeoffs.
I mentioned it already, but maybe the better solution would be if people would not switch between significant branches in the same worktree but use multiple worktrees.
Having the plugins all in the settings classloader is basically like having them as buildSrc dependencies or buildSrc code, so I think any change in those plugins will change the runtime classpath of those tasks and thus again produce a different cache key.
I imagine that the version catalog generated code should be applied like any other plugin instead of contaminating all third party plugins.
It kind of is.
It has the same effect than having a dependency or code in buildSrc or in the settings classpath.
Not sure on the details, but feels like something should be possible.
I guess the version catalog accessors need to not be put to the buildSrc class loader, but to some class loader that is a child of the actual build script classpath class loader, so that it is available to the build script, but does not “pollute” the plugin runtime classpaths.
But I have no idea whether that would then have different drawbacks.
Just wanted to follow up here that we use about 20 third party plugins in our build so I’ve written some code in our setting.gradle.kts to do some very rudimentary parsing of our libs.versions.toml and add all the plugins defined there as described above. You don’t appear to be able to do this from a Plugin<Settings> and you don’t have access to much at this point in the build so I couldn’t even use Regex for example. It doesn’t handle everything, but does the job for us for now. Now when I run ./gradlew buildenvironment there are no libraries on the classpath where there were plenty before and we don’t get lengthy rebuilds when adding new libraries anymore.
