Gradle remote build cache collisions

After increasing the Gradle remote build cache size to 1 TB and TTL to 3 days, we started experiencing issues that can only be explained by Gradle remote cache collisions.

We have a large project with dozens of Android applications that we regularly build, and recently, some builds have started failing with errors stating that a class or function was not found in a module. This issue consistently occurred multiple times until we purged the remote cache, after which the build proceeded smoothly.

This has happened twice already. We are using a relatively old version of Gradle (7.6), but I have not noticed any significant changes related to the build cache in newer versions.

So, my questions today are:

  1. How can this issue occur?
  2. How can we diagnose the root cause of the problem?
  3. How can we prevent it from happening again?