Splitting & aggregating Gradle cache over multiple Github workflows

Help/Discuss
1

Hi,

I’ve got the idea to split our CI workflows on GitHub using the Gradle GitHub action into multiple workflows, but still want to leverage the Gradle build cache in all workflows.

Currently we have one GitHub workflow, which roughly consists of three “stages”:

  • compilation, checkstyle, spotless, assemble/Javadoc
  • test target
  • intTest target

The Gradle cache(s) are properly loaded and restored - and that’s fine.

Since the code base grew and (depending on the change) the whole workflow now takes a while (70 minutes at max at the moment), but I wanna bring it back to ~20 minutes.

So I had the idea to split the single workflow into different workflows running in parallel.

+---------------------+
| initial CI workflow |
+---------------------+
      |
      +--------------+-----------------+-----------------------+
      |              |                 |                       |
+-------------+ +----------------+ +------------------+ +----------------+
| checkstyle, | | ./gradlew test | | ./gradle intTest | | more workflows |
| etc         | |                | |                  | |                |
+-------------+ +----------------+ +------------------+ +----------------+
      |              |                 |                       |
      +--------------+-----------------+-----------------------+
      |
+---------------------+
| final CI workflow   |
| (maybe)             |
+---------------------+

The problem at hand is that the Gradle cache results from multiple workflows diverge: for example, one workflow has the test results, another has the intTest results, and so on.

My question is, whether there’s a way to aggregate the “final state” of the Gradle caches from the parallel workflows. Or could I “just”:tm: take the caches from all parallel workflows and just “copy those together”? Or is there already something available to do that?

Robert

2

I personally would recommend using the gradle-cache-action instead.
The official action is younger, the gradle-cache-action is more sophisticated.
And it provides a remote build cache that is backed by GitHub Actions cache on a cache-entry by cache-entry basis, so it will probably just work properly.

3

Yea, that one looks like a better alternative!

4

Well, it looks good, but it fails in practice, because it puts too many requests against GitHub’s cache backend, which responds with HTTP/429 (too many requests), effectively making it use no cache.

5

Oh, never had that problem so far, but was only building relatively small projects.
You should report it, maybe he can do something clever about it.

6

I doubt, it’s a hard limit on the GH side.

7

Doesn’t mean he cannot come up with something.
Maybe some partitioning to need less requests, or whatever else.

8

The only thing you know if you do not report it is, that for sure nothing can happen for it to be improved.