Dependency cache issues on build-agents

Help/Discuss
1

I am seeing the following error on our jenkins build-agents intermittently. Any ideas what might be causing this?

14:21:07 FAILURE: Build failed with an exception.
14:21:07 
14:21:07 * What went wrong:
14:21:07 Failed to capture snapshot of output files for task 'compileJava' during up-to-date check.
14:21:07 > Could not remove entry '/mnt/mesos/sandbox/workspace/xxx/xx/xxx/xxx/build/dependency-cache' from cache outputFileStates.bin (/gradlecache/2.11-20151210230032+0000/taskArtifacts/outputFileStates.bin).
14:21:07 
14:21:07 * Try:
14:21:07 Run with --info or --debug option to get more log output.
14:21:07 
14:21:07 * Exception is:
14:21:07 org.gradle.api.UncheckedIOException: Failed to capture snapshot of output files for task 'compileJava' during up-to-date check.
14:21:07 	at org.gradle.api.internal.changedetection.rules.TaskUpToDateState.<init>(TaskUpToDateState.java:51)
14:21:07 	at org.gradle.api.internal.changedetection.changes.DefaultTaskArtifactStateRepository$TaskArtifactStateImpl.getStates(DefaultTaskArtifactStateRepository.java:132)
14:21:07 	at org.gradle.api.internal.changedetection.changes.DefaultTaskArtifactStateRepository$TaskArtifactStateImpl.isUpToDate(DefaultTaskArtifactStateRepository.java:70)
14:21:07 	at org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:52)
14:21:07 	at org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:58)
14:21:07 	at org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:52)
14:21:07 	at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:52)
14:21:07 	at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:53)
14:21:07 	at org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:43)
14:21:07 	at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter$EventFiringTaskWorker.execute(DefaultTaskGraphExecuter.java:203)
14:21:07 	at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter$EventFiringTaskWorker.execute(DefaultTaskGraphExecuter.java:185)
14:21:07 	at org.gradle.execution.taskgraph.AbstractTaskPlanExecutor$TaskExecutorWorker.processTask(AbstractTaskPlanExecutor.java:66)
14:21:07 	at org.gradle.execution.taskgraph.AbstractTaskPlanExecutor$TaskExecutorWorker.run(AbstractTaskPlanExecutor.java:50)
14:21:07 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)
14:21:07 	at org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40)
14:21:07 Caused by: org.gradle.api.UncheckedIOException: Could not remove entry '/mnt/mesos/sandbox/workspace/xxx/xxx/xx/xx/xx//build/dependency-cache' from cache outputFileStates.bin (/gradlecache/2.11-20151210230032+0000/taskArtifacts/outputFileStates.bin).
14:21:07 	at org.gradle.cache.internal.btree.BTreePersistentIndexedCache.remove(BTreePersistentIndexedCache.java:170)
14:21:07 	at org.gradle.cache.internal.DefaultMultiProcessSafePersistentIndexedCache$3.run(DefaultMultiProcessSafePersistentIndexedCache.java:62)
14:21:07 	at org.gradle.cache.internal.DefaultFileLockManager$DefaultFileLock.doWriteAction(DefaultFileLockManager.java:173)
14:21:07 	at org.gradle.cache.internal.DefaultFileLockManager$DefaultFileLock.writeFile(DefaultFileLockManager.java:163)
14:21:07 	at org.gradle.cache.internal.DefaultCacheAccess$UnitOfWorkFileAccess.writeFile(DefaultCacheAccess.java:404)
14:21:07 	at org.gradle.cache.internal.DefaultMultiProcessSafePersistentIndexedCache.remove(DefaultMultiProcessSafePersistentIndexedCache.java:60)
14:21:07 	at org.gradle.api.internal.changedetection.state.InMemoryTaskArtifactCache$1.remove(InMemoryTaskArtifactCache.java:104)
14:21:07 	at org.gradle.api.internal.changedetection.state.OutputFilesCollectionSnapshotter$1.run(OutputFilesCollectionSnapshotter.java:85)
14:21:07 	at org.gradle.internal.Factories$1.create(Factories.java:22)
14:21:07 	at org.gradle.cache.internal.DefaultCacheAccess.useCache(DefaultCacheAccess.java:192)
14:21:07 	at org.gradle.cache.internal.DefaultCacheAccess.useCache(DefaultCacheAccess.java:175)
14:21:07 	at org.gradle.cache.internal.DefaultPersistentDirectoryStore.useCache(DefaultPersistentDirectoryStore.java:106)
14:21:07 	at org.gradle.cache.internal.DefaultCacheFactory$ReferenceTrackingCache.useCache(DefaultCacheFactory.java:187)
14:21:07 	at org.gradle.api.internal.changedetection.state.DefaultTaskArtifactStateCacheAccess.useCache(DefaultTaskArtifactStateCacheAccess.java:60)
14:21:07 	at org.gradle.api.internal.changedetection.state.OutputFilesCollectionSnapshotter.snapshot(OutputFilesCollectionSnapshotter.java:73)
14:21:07 	at org.gradle.api.internal.changedetection.state.OutputFilesCollectionSnapshotter.snapshot(OutputFilesCollectionSnapshotter.java:44)
14:21:07 	at org.gradle.api.internal.changedetection.rules.OutputFilesStateChangeRule.create(OutputFilesStateChangeRule.java:34)
14:21:07 	at org.gradle.api.internal.changedetection.rules.TaskUpToDateState.<init>(TaskUpToDateState.java:49)
14:21:07 	... 14 more
14:21:07 Caused by: org.gradle.cache.internal.btree.CorruptedCacheException: Corrupted DataBlock 119908 found in cache '/gradlecache/2.11-20151210230032+0000/taskArtifacts/outputFileStates.bin'.
14:21:07 	at org.gradle.cache.internal.btree.FileBackedBlockStore$BlockImpl.blockCorruptedException(FileBackedBlockStore.java:247)
14:21:07 	at org.gradle.cache.internal.btree.FileBackedBlockStore$BlockImpl.read(FileBackedBlockStore.java:223)
14:21:07 	at org.gradle.cache.internal.btree.FileBackedBlockStore.read(FileBackedBlockStore.java:98)
14:21:07 	at org.gradle.cache.internal.btree.CachingBlockStore.read(CachingBlockStore.java:84)
14:21:07 	at org.gradle.cache.internal.btree.FreeListBlockStore.read(FreeListBlockStore.java:78)
14:21:07 	at org.gradle.cache.internal.btree.StateCheckBlockStore.read(StateCheckBlockStore.java:61)
14:21:07 	at org.gradle.cache.internal.btree.BTreePersistentIndexedCache.remove(BTreePersistentIndexedCache.java:166)
14:21:07 	... 31 more

I had another question regrading the local gradle cache. Our build-agents are ephemeral. So we don’t want to download 3rd-party dependencies for a build from artifactory every time a build-agents starts up. We can get around it by using the GRADLE_USER_HOME of a jenkins job that populates the gradle cache and copying it when the build-agent starts-up. This way, we don’t have to download all the 3rd-party artifacts again. Is there a better solution for this?

We used to do the same thing for maven and it worked well for us and reduced build times significantly.

2

What are your arguments to Gradle? How are you setting the Gradle user home? Are you also changing the project cache directory?

Are you using any GradleBuild tasks?

WRT your second question, you could setup a filesystem repository (like a Maven local) and share that. We don’t cache dependencies that come from a “local” repository (one that has a file:// URL).

3

I am actually following what I described the question earlier. We have a separate job that populates the GRADLE_USER_HOME. Before we start a build on a new build agent, we copy the GRADLE_USER_HOME into the build-agent:

GRADLE_CACHE_FROM='/git-cache/gradle-xxx/'
GRADLE_CACHE_TO='/gradlecache/'
GRADLE_LOCAL_REPO="${WORKSPACE}/aaa/.gradle"
if [[ ! -d ${GRADLE_CACHE_TO} ]] && [[ ! -e ${GRADLE_LOCAL_REPO} ]]; then
  echo "Local gradle cache under ${GRADLE_CACHE_TO} does not exist, creating it"
  time rsync -a ${GRADLE_CACHE_FROM} ${GRADLE_CACHE_TO}
  ln -s ${GRADLE_CACHE_TO} ${GRADLE_LOCAL_REPO}
elif [[ ! -e ${GRADLE_LOCAL_REPO} ]]; then
  echo "Local gradle cache under ${GRADLE_CACHE_TO} already exist, linking ${GRADLE_LOCAL_REPO} to it"
  ln -s ${GRADLE_CACHE_TO} ${GRADLE_LOCAL_REPO}
else
  echo "${GRADLE_LOCAL_REPO} folder already exist, not doing anything"
fi
find /gradlecache -type f -name "*.lock" -exec rm -rf '{}' +

I am just running ./gradlew clean check.

I will look into setting up a “local” repository.

Thanks for your prompt reply!

4

Ah, you’re not changing the GRADLE_USER_HOME, you’re changing the project cache directory.

GRADLE_USER_HOME has the global cache for external dependencies and Gradle wrapper distributions (among other internal things).

The project cache directory (the .gradle directory in the project root) is for project specific information (mostly up-to-date information per task). You shouldn’t share that between projects.

If you’re putting the Gradle cache in /gradlecache, then you can export GRADLE_USER_HOME set to that or run Gradle with -g /gradlecache.

5

I forgot to mention that before I start the build I run:

export GRADLE_USER_HOME=${WORKSPACE}/aaa/.gradle
./gradlew clean check

I will look into changing the build to use “local” repo.

Thanks!

6

It would also be great if you could describe your use case a bit more.

Are the build agents “far” from the source of the third party dependencies (so syncing the cache from elsewhere is faster)?