Why is dependency cache located under project root on Jenkins?

Old Forum Archive
1

Using gradle 1.5. I have created a multi gradle project with several subproject each having a bunch of 3rdparty dependencies. In the root of the multi project folder there is a “caches/artifacts-23/filestore” that takes up 4 GB. Does each gradle project really contain all 3rdparty libs locally?

Why not put them in the .gradle folder for the user (eg. like .m2 in the user folder for maven projects)?

2

By default, the Gradle cache is located below the user home. Sounds like you reconfigured it, perhaps by setting the ‘GRADLE_USER_HOME’ environment variable.

3

No I did not change that variable. I have looked into the content of the “caches/artifacts-23/filestore” and what takes up so much space is external dependencies (not module dependencies) to our own projects located in:

/var/jenkins/workspace/my_project/caches/artifacts-23/filestore/com.mycompany

Inside this folder 21 of external dependencies to our own projects are stored which makes sense (besides that they are stored inside the multi-project).

If I go to the “jar” folder of one of the dependencies:

/var/jenkins/workspace/my_project/caches/artifacts-23/filestore/com.mycompany/my_external_dep/1.0-SNAPSHOT/jar/

there is a list of 18 folders named with something like a UUID. Each folder containing the actual "my_external_dep-1.0-SNAPSHOT.jar.

And this is the case for all the external dependencies which explains the disk consumption.

I build each project with “–refresh-dependencies” since I always need to get the latest external dependencies from artifactory.

But it still bothers me that all dependencies are stored INSIDE each of my gradle projects.

4

But it still bothers me that all dependencies are stored INSIDE each of my gradle projects.

Again, this won’t happen unless ‘GRADLE_USER_HOME’ is explicitly set to the project directory (default is ‘~/.gradle’). My best guess is that either your CI machines or CI jobs are responsible for this.

Since the Gradle dependency cache is multi-process safe, you can share it across all build jobs on a machine. If you publish/consume a lot of snapshots, the cache can grow very large over time. There are plans to offer a feature to purge the cache, but for now I’d schedule a cron job that deletes the cache every once in a while.

5

I have checked the system info for master and all slaves and there is no GRADLE_USER_HOME variable.

I have tried to add this:

configurations.all {
    resolutionStrategy {
      cacheDynamicVersionsFor 0, 'seconds'
      cacheChangingModulesFor 0, 'seconds'
    }
  }

to the root build.gradle file but it does not seem to have any effect.

Any suggestions to where else the GRADLE_USER_HOME could be modified?

all slaves also have a ~/.gradle/caches folder but the “caches” folder is still created in the root of the workspace for a specific job on the slave when checking out and build a gradle project with git.

6

I’d print out the following at the top of the build script:

  • ‘GRADLE_USER_HOME’ environment variable * ‘gradle.user.home’ system property * ‘gradle.startParameter.gradleUserHomeDir’ property (corresponds to the ‘–gradle-user-home’ command line parameter, except that if falls back to a default if the latter is not set)

Hopefully this will help to find the offender. You can also try to set up Gradle as a shell script job, rather than using the Jenkins Gradle plugin.

7

when I build it after adding :

//
println "$gradle.user.home" does not exits
  println "DEBUGGIN: $gradle.startParameter.gradleUserHomeDir"
//
println "$GRADLE_USER_HOME" does not exits

I get:

DEBUGGIN: /var/jenkins/workspace/my_project

which is clearly wrong.

Could be a “bug” in the Jenkins gradle plugin that it for some reason uses the workspace root.

I will try to see what happens if set gradle.user.home globally like is necessary with maven:

8

Apparently, the problem is indeed caused by the Jenkins Gradle plugin: https://github.com/jenkinsci/gradle-plugin/pull/12

9

Thanks. They actually pose a potential problem using a global cache. So maybe its best to use a local to the workspace and then configure:

configurations.all {
    resolutionStrategy {
      cacheDynamicVersionsFor 0, 'seconds'
      cacheChangingModulesFor 0, 'seconds'
    }
  }

this should clean up the caches folder right?

10

Again, the Gradle dependency cache is designed to be shared between builds and agents. Unless you have thousands of Jenkins jobs like the author of that patch, I would recommend to share the cache. If you encounter any downsides (e.g. diminished performance), we would like to hear about them.

this should clean up the caches folder right?

This will disable caching of dynamic versions and changing modules, and will likely slow down your builds. It won’t affect the size of the cache. Again, right now the only way to purge the cache is to periodically delete it.

11

You write: “This will disable caching of dynamic versions and changing modules” but would that not exactly disable caching of our own projects which looks like this myproject-1.0.0-SNAPHOT.jar or are dynamic and changing modules something else?

12

Try for yourself. You can read up on these concepts in the Gradle User Guide.

13

It is a “bug” in the most recent version of the plugin. The ~/.m2/repository is a repo and cache. A repo might need to be a shared, but a cache should not need to be shared. In Gradle, it’s a pure cache, and like any cache you can’t make guarantees about what’s currently in it.