Issues with Gradle 5.1.x and 5.2.x - :processResources task is deleting build\resources\main

Help/Discuss
1

Hi all,

I’m using a plugin to generate a file in build\resources\main directory. But I have a problem when running the plugin with Gradle 5.1.x and 5.2.x (but not on other Gradle versions, older and newer)

For some reason, on these gradle versions, :processResources task (specifically CleanupStaleOutputsExecuter) will always delete build\resources\main directory and my generated file. It will show some message like

Deleting stale output file: build\resources\main

Does anyone know the reason for this strange behavior for these specific Gradle versions?

More info: https://github.com/n0mer/gradle-git-properties/issues/128

Thank you in advance!

2

If two tasks writes to the same output folder, things can get messy. And so, to avoid any potential problems, Gradle will clean the folder if it finds any files which should not be there.

So to fix your problem, you should generate your file in a new folder unique to that task, instead of using the build/resources/main folder which is used by “processResources” task.

If you need the generated file on the classpath, you can register the new folder as a resource directory to the “main” source set.

3

Thanks Bjorn_Vester.

I know it is the right solution for the issue. However, I still hope to find some way continue to use the “build/resources/main” folder for the generated file because I don’t want to make the plugin users to be surprised by the changes (and the issue seems to be specific to gradle 5.1 and 5.2 only).

Current plugin configuration:
gitPropertiesDir = “$(project.buildDir}/resources/main/com/mygroup”

New plugin configuration (If I use another classpath folder):

gitPropertiesParentDir = “$(project.buildDir}/generated-resources” // root of new classpath folder
gitPropertiesSubDir = “com/mygroup” // relative location inside the new classpath folder

If I cannot find some workaround way, I might have to change to a new directory. Thanks.

-Thai

4

@tha2015 Did you try to declare the file you are generating as an output of the task which generates it? That should prevent it from being cleaned up. Gradle will only clean up files in the outputs of tasks when it doesn’t know how they got there (i.e. not an output of any task). We call those files “stale” files, since they are leftover files from external tools or previous Gradle versions.

5

Yes, I used @OutputFile. But it looks like on gradle 5.1 and 5.2, the whole directory (build/resources/main) is deleted without checking if any other plugin is sharing the same directory for output files.

6

Can you share a reproducer? This sounds like a bug.

7

Thanks Stefan_Wolf.

This is the example project https://github.com/n0mer/gradle-git-properties/files/3384489/example.zip
(uploaded for GitHub issue https://github.com/n0mer/gradle-git-properties/issues/128)

Unzip and run “gradlew jar -i”
The message “Deleting stale output file” will be displayed for :processResources task every time running the command (no need for deleting .gradle folder)

Plugin source code can be found at https://github.com/n0mer/gradle-git-properties/blob/master/src/main/groovy/com/gorylenko/GenerateGitPropertiesTask.groovy

Thanks,
-Thai

8

@tha2015 I tried using your reproducer project, though I didn’t reproduce what you described. This is what I get:

> Task :project1:generateGitProperties
Deleting stale output file: <project-root>/project1/build/resources/main/com/git.properties
dotGitDirectory = [<project-root>/.git]
Build cache key for task ':project1:generateGitProperties' is 3d9e4a8c1c628578ae44552af2e57db4
Task ':project1:generateGitProperties' is not up-to-date because:
  Task.upToDateWhen is false.
git.properties location = [<project-root>/project1/build/resources/main/com/git.properties]
Written properties to [<project-root>/project1/build/resources/main/com/git.properties]...
Packing task ':project1:generateGitProperties'
:project1:generateGitProperties (Thread[Execution worker for ':',5,main]) completed. Took 0.015 secs.
:project1:processResources (Thread[Execution worker for ':',5,main]) started.

> Task :project1:processResources
Build cache key for task ':project1:processResources' is 53d5ff7c1d55f857a328cf73afa29a5c
Caching disabled for task ':project1:processResources': Caching has not been enabled for the task
Task ':project1:processResources' is not up-to-date because:
  No history is available.
:project1:processResources (Thread[Execution worker for ':',5,main]) completed. Took 0.003 secs.
:project1:classes (Thread[Execution worker for ':',5,main]) started.

So it seems to work correctly: generateGitProperties runs first, detects that there is a leftover file, removes the leftover file and the output directory is registered.

I tried with Gradle 5.2 and Gradle 5.6.2, deleting .gradle and not deleting .gradle. Can you give me some step by step instructions how to reproduce what you are seeing?

Maybe it would be better for the generateGitProperties to generate the file into a new directory (like build/gitProperties/main) and then add the directories as a resource directory to the source set. This way, the two tasks won’t have overlapping outputs.

Cheers,
Stefan

9

Hi @Stefan_Wolf,

Thanks for looking into this issue. Please see below for the commands I used and the log.

You can see that after running "./gradlew jar -i " command, the git.properties is not there (the command “ls project1/build/resources/” didn’t show anything.)

And you can see the problem is because “processResources” task (running after “generateGitProperties” task) deleted the generated file:

Task :project1:processResources NO-SOURCE
Deleting stale output file: /home/tha/test/example/project1/build/resources/main

This is the full log

tha@tha:~/test$ wget https://github.com/n0mer/gradle-git-properties/files/3384489/example.zip
..
2019-09-16 17:50:41 (413 KB/s) - ‘example.zip’ saved [68341/68341]

tha@tha:~/test$ unzip example.zip 
..

tha@tha:~/test/example$ chmod u+x gradlew

tha@tha:~/test/example$ ./gradlew jar -i 

Downloading https://services.gradle.org/distributions/gradle-5.1.1-all.zip
............................................................................................................................
Unzipping /home/tha/.gradle/wrapper/dists/gradle-5.1.1-all/97z1ksx6lirer3kbvdnh7jtjg/gradle-5.1.1-all.zip to /home/tha/.gradle/wrapper/dists/gradle-5.1.1-all/97z1ksx6lirer3kbvdnh7jtjg
Set executable permissions for: /home/tha/.gradle/wrapper/dists/gradle-5.1.1-all/97z1ksx6lirer3kbvdnh7jtjg/gradle-5.1.1/bin/gradle
Initialized native services in: /home/tha/.gradle/native

Welcome to Gradle 5.1.1!

Here are the highlights of this release:
 - Control which dependencies can be retrieved from which repositories
 - Production-ready configuration avoidance APIs

For more details see https://docs.gradle.org/5.1.1/release-notes.html

Using 4 worker leases.
Starting Build
<-------------> 0% INITIALIZING [0s]
> Evaluating settings
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.codehaus.groovy.vmplugin.v7.Java7$1 (file:/home/tha/.gradle/wrapper/dists/gradle-5.1.1-all/97z1ksx6lirer3kbvdnh7jtjg/gradle-5.1.1/lib/groovy-all-1.0-2.5.4.jar) to constructor java.lang.invoke.MethodHandles$Lookup(java.lang.Class,int)
WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.vmplugin.v7.Java7$1
Compiling settings file '/home/tha/test/example/settings.gradle' using SubsetScriptTransformer.ations
Compiling settings file '/home/tha/test/example/settings.gradle' using BuildScriptTransformer.
Settings evaluated using settings file '/home/tha/test/example/settings.gradle'.
Projects loaded. Root project using build file '/home/tha/test/example/build.gradle'.
Included projects: [root project 'project1-root', project ':project1']

> Configure project :
Evaluating root project 'project1-root' using build file '/home/tha/test/example/build.gradle'.
Compiling build file '/home/tha/test/example/build.gradle' using SubsetScriptTransformer.
Found locally available resource with matching checksum: [https://plugins.gradle.org/m2/gradle/plugin/com/gorylenko/gradle-git-properties/gradle-git-properties/2.0.0/gradle-git-properties-2.0.0.pom, /home/tha/.gradle/caches/modules-2/files-2.1/gradle.plugin.com.gorylenko.gradle-git-properties/gradle-git-properties/2.0.0/16d72a6f767d2df87a8ac513c9434228e2f9e59d/gradle-git-properties-2.0.0.pom]
Found locally available resource with matching checksum: [https://plugins.gradle.org/m2/org/ajoberstar/grgit/grgit-core/3.0.0/grgit-core-3.0.0.pom, /home/tha/.gradle/caches/modules-2/files-2.1/org.ajoberstar.grgit/grgit-core/3.0.0/216e0970c0465ea6bb6fa9c45e0b58e5cdaf1b85/grgit-core-3.0.0.pom]
Found locally available resource with matching checksum: [https://plugins.gradle.org/m2/org/eclipse/jgit/org.eclipse.jgit/5.1.3.201810200350-r/org.eclipse.jgit-5.1.3.201810200350-r.pom, /home/tha/.gradle/caches/modules-2/files-2.1/org.eclipse.jgit/org.eclipse.jgit/5.1.3.201810200350-r/64fb79859c8d126206a6bc79634739e097ffad8d/org.eclipse.jgit-5.1.3.201810200350-r.pom]
Found locally available resource with matching checksum: [https://plugins.gradle.org/m2/org/eclipse/jgit/org.eclipse.jgit-parent/5.1.3.201810200350-r/org.eclipse.jgit-parent-5.1.3.201810200350-r.pom, /home/tha/.gradle/caches/modules-2/files-2.1/org.eclipse.jgit/org.eclipse.jgit-parent/5.1.3.201810200350-r/68fb1fa149b97a735ad82ad3dd8816e89178e1ae/org.eclipse.jgit-parent-5.1.3.201810200350-r.pom]
Found locally available resource with matching checksum: [https://plugins.gradle.org/m2/com/jcraft/jsch/0.1.54/jsch-0.1.54.pom, /home/tha/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/8e82e6a2cb5664170faa2a5abdc297e589a37df1/jsch-0.1.54.pom]
Found locally available resource with matching checksum: [https://plugins.gradle.org/m2/org/slf4j/slf4j-api/1.7.2/slf4j-api-1.7.2.pom, /home/tha/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.2/3fceb45ce8f7a6f87f3f2077a24a3833d1ecb4c6/slf4j-api-1.7.2.pom]
Found locally available resource with matching checksum: [https://plugins.gradle.org/m2/com/googlecode/javaewah/JavaEWAH/1.1.6/JavaEWAH-1.1.6.pom, /home/tha/.gradle/caches/modules-2/files-2.1/com.googlecode.javaewah/JavaEWAH/1.1.6/759dda489d69ebd237d9ee353e67a460274b14f0/JavaEWAH-1.1.6.pom]
Found locally available resource with matching checksum: [https://plugins.gradle.org/m2/com/jcraft/jzlib/1.1.1/jzlib-1.1.1.pom, /home/tha/.gradle/caches/modules-2/files-2.1/com.jcraft/jzlib/1.1.1/d990b68017884e9615990c39ef81cf2b5884d464/jzlib-1.1.1.pom]
Found locally available resource with matching checksum: [https://plugins.gradle.org/m2/org/slf4j/slf4j-parent/1.7.2/slf4j-parent-1.7.2.pom, /home/tha/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-parent/1.7.2/26c68730e36b29249e0c3bb70eba4a5dbaa4f76c/slf4j-parent-1.7.2.pom]
Found locally available resource with matching checksum: [https://plugins.gradle.org/m2/org/sonatype/oss/oss-parent/5/oss-parent-5.pom, /home/tha/.gradle/caches/modules-2/files-2.1/org.sonatype.oss/oss-parent/5/3ae20880ad3d5da6b1caec19e3de7e70dd2dd762/oss-parent-5.pom]
Found locally available resource with matching checksum: [https://plugins.gradle.org/m2/org/sonatype/oss/oss-parent/6/oss-parent-6.pom, /home/tha/.gradle/caches/modules-2/files-2.1/org.sonatype.oss/oss-parent/6/765c355ec09ad070065d9d12a9245bba5c689d96/oss-parent-6.pom]
Found locally available resource with matching checksum: [https://plugins.gradle.org/m2/com/jcraft/jsch/0.1.54/jsch-0.1.54.jar, /home/tha/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar]
Found locally available resource with matching checksum: [https://plugins.gradle.org/m2/com/googlecode/javaewah/JavaEWAH/1.1.6/JavaEWAH-1.1.6.jar, /home/tha/.gradle/caches/modules-2/files-2.1/com.googlecode.javaewah/JavaEWAH/1.1.6/94ad16d728b374d65bd897625f3fbb3da223a2b6/JavaEWAH-1.1.6.jar]
Found locally available resource with matching checksum: [https://plugins.gradle.org/m2/gradle/plugin/com/gorylenko/gradle-git-properties/gradle-git-properties/2.0.0/gradle-git-properties-2.0.0.jar, /home/tha/.gradle/caches/modules-2/files-2.1/gradle.plugin.com.gorylenko.gradle-git-properties/gradle-git-properties/2.0.0/106853534412510693fcc24982b6046583dfa472/gradle-git-properties-2.0.0.jar]
Found locally available resource with matching checksum: [https://plugins.gradle.org/m2/com/jcraft/jzlib/1.1.1/jzlib-1.1.1.jar, /home/tha/.gradle/caches/modules-2/files-2.1/com.jcraft/jzlib/1.1.1/a1551373315ffc2f96130a0e5704f74e151777ba/jzlib-1.1.1.jar]
Found locally available resource with matching checksum: [https://plugins.gradle.org/m2/org/slf4j/slf4j-api/1.7.2/slf4j-api-1.7.2.jar, /home/tha/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.2/81d61b7f33ebeab314e07de0cc596f8e858d97/slf4j-api-1.7.2.jar]
Found locally available resource with matching checksum: [https://plugins.gradle.org/m2/org/ajoberstar/grgit/grgit-core/3.0.0/grgit-core-3.0.0.jar, /home/tha/.gradle/caches/modules-2/files-2.1/org.ajoberstar.grgit/grgit-core/3.0.0/741436d5e7321148a284096629b5b9bcd6e94792/grgit-core-3.0.0.jar]
Found locally available resource with matching checksum: [https://plugins.gradle.org/m2/org/eclipse/jgit/org.eclipse.jgit/5.1.3.201810200350-r/org.eclipse.jgit-5.1.3.201810200350-r.jar, /home/tha/.gradle/caches/modules-2/files-2.1/org.eclipse.jgit/org.eclipse.jgit/5.1.3.201810200350-r/f270dbd1d792d5ad06074abe018a18644c90b60e/org.eclipse.jgit-5.1.3.201810200350-r.jar]
Compiling build file '/home/tha/test/example/build.gradle' using BuildScriptTransformer.

> Configure project :project1
Evaluating project ':project1' using build file '/home/tha/test/example/project1/build.gradle'.
All projects evaluated.
Selected primary task 'jar' from project :
Tasks to be executed: [task ':project1:compileJava', task ':project1:generateGitProperties', task ':project1:processResources', task ':project1:classes', task ':project1:jar']
:project1:compileJava (Thread[Execution worker for ':',5,main]) started.

> Task :project1:compileJava
Task ':project1:compileJava' is not up-to-date because:
  No history is available.
All input files are considered out-of-date for incremental task ':project1:compileJava'.
Full recompilation is required because no incremental change information is available. This is usually caused by clean builds or changing compiler arguments.
Compiling with JDK Java compiler API.
Created classpath snapshot for incremental compilation in 0.002 secs.
:project1:compileJava (Thread[Execution worker for ':',5,main]) completed. Took 1.325 secs.
:project1:generateGitProperties (Thread[Execution worker for ':' Thread 3,5,main]) started.

> Task :project1:generateGitProperties
dotGitDirectory = [/home/tha/test/example/.git]
Task ':project1:generateGitProperties' is not up-to-date because:
  Task.upToDateWhen is false.
git.properties location = [/home/tha/test/example/project1/build/resources/main/com/git.properties]
Written properties to [/home/tha/test/example/project1/build/resources/main/com/git.properties]...
:project1:generateGitProperties (Thread[Execution worker for ':' Thread 3,5,main]) completed. Took 0.553 secs.
:project1:processResources (Thread[Execution worker for ':' Thread 3,5,main]) started.

> Task :project1:processResources NO-SOURCE
Deleting stale output file: /home/tha/test/example/project1/build/resources/main
Skipping task ':project1:processResources' as it has no source files and no previous output files.
:project1:processResources (Thread[Execution worker for ':' Thread 3,5,main]) completed. Took 0.02 secs.
:project1:classes (Thread[Execution worker for ':' Thread 3,5,main]) started.

> Task :project1:classes
Skipping task ':project1:classes' as it has no actions.
:project1:classes (Thread[Execution worker for ':' Thread 3,5,main]) completed. Took 0.003 secs.
:project1:jar (Thread[Execution worker for ':' Thread 3,5,main]) started.

> Task :project1:jar
Task ':project1:jar' is not up-to-date because:
  No history is available.
:project1:jar (Thread[Execution worker for ':' Thread 3,5,main]) completed. Took 0.231 secs.

BUILD SUCCESSFUL in 39s
3 actionable tasks: 3 executed
jars (/home/tha/.gradle/caches/jars-3) removing files not accessed on or after Mon Sep 09 17:51:48 PDT 2019.
jars (/home/tha/.gradle/caches/jars-3) cleaned up in 0.086 secs.
artifact cache (/home/tha/.gradle/caches/modules-2) [subdir: /home/tha/.gradle/caches/modules-2/resources-2.1] removing files not accessed on or after Sat Aug 17 17:51:49 PDT 2019.
artifact cache (/home/tha/.gradle/caches/modules-2) [subdir: /home/tha/.gradle/caches/modules-2/files-2.1] removing files not accessed on or after Sat Aug 17 17:51:49 PDT 2019.
artifact cache (/home/tha/.gradle/caches/modules-2) cleaned up in 0.118 secs.
Artifact transforms cache (/home/tha/.gradle/caches/transforms-2) [subdir: /home/tha/.gradle/caches/transforms-2/files-2.1] removing files not accessed on or after Mon Sep 09 17:51:53 PDT 2019.
Artifact transforms cache (/home/tha/.gradle/caches/transforms-2) cleaned up in 0.0 secs.


tha@tha:~/test/example$ ls project1/build/resources/

tha@tha:~/test/example$ 

tha@tha:~/test/example$ ./gradlew --version

------------------------------------------------------------
Gradle 5.1.1
------------------------------------------------------------

Build time:   2019-01-10 23:05:02 UTC
Revision:     3c9abb645fb83932c44e8610642393ad62116807

Kotlin DSL:   1.1.1
Kotlin:       1.3.11
Groovy:       2.5.4
Ant:          Apache Ant(TM) version 1.9.13 compiled on July 10 2018
JVM:          11.0.4 (Ubuntu 11.0.4+11-post-Ubuntu-1ubuntu219.04)
OS:           Linux 5.0.0-27-generic amd64
10

I followed your instruction, and I still get:

> Task :project1:processResources NO-SOURCE
Skipping task ':project1:processResources' as it has no source files and no previous output files.
:project1:processResources (Thread[Execution worker for ':',5,main]) completed. Took 0.0 secs.
:project1:classes (Thread[Execution worker for ':',5,main]) started.

The only difference I can see is that I am on macOS and you are on Linux.

I’ll try out on Linux and see if I can reproduce the problem there.

11

Thanks @Stefan_Wolf. It also happens with JDK8/Linux.

12

So I tried on Linux with JDK8. I can reproduce what you are seeing with Gradle 5.1.1. Though the problem seems to fixed with Gradle 5.6.2. Can you confirm this?

13

Yes. The issue happened on Gradle 5.1.x and 5.2.x only (not happening on older or newer versions).