I recently upgraded a project to Gradle 7.0 and got a bunch of warnings about implicit dependencies. I asked about this on StackOverflow and created a repository to demonstrate the problem without any extension steps.
It appears that Gradle 7.0 treats any task that reads any file from a particular directory as implicitly dependent on every task that writes any file to that directory. Is that correct? Are there any mechanisms that I can use to control that behavior?
I have lots of build scripts that write intermediate files to build/ and then process them with subsequent (explicitly dependent) tasks. It’s going to be extremely annoying if I have to rewrite every build script so that it uses separate subdirectories for every kind of intermediate file.
Having tasks with overlapping outputs is bad, as this works against up-to-date checks and cacheability and as you have seen also makes problems with other optimizations. Each task shoud have a dedicated output that he owns exclusively. In case of a Copy task, the output is the outputDirectory property and so any files in there at the end of the task execution are considered an output, even if they originate from some other task that was accidentally or incidentally run before. That’s also why for example task output caching for cacheable tasks is disabled automatically if overlapping outputs are found.
Due to this you can for example also not use a task of type Copy to copy something into the project directory like for example an updated readme file from a template, as then the whole directory, including VCS metadata, build artifacts and everything would count as output of the task and that would not only be incorrect but even fail to calculate the hashes. In such a case you could for example instead make an ad-hoc task where you define the actual output files using outputs.file and then use project.copy { ... } in a doLast { ... } to do the actual copying. The spec for that copy is the same as for the task of type Copy, but as it is not a task of type Copy anymore that defines the destination directory as output directory, but you specify the outputs of the task manually as specific files, problems like the whole project directory being an output or overlapping outputs are gone.
The other thing to mention is, that hard-coding paths and using manual task dependencies is bad practice.
Instead your tasks should properly define their outputs. The tasks using these outputs can then simply refer to the outputs of the task and the task can also be defined as input. If the tasks using these outputs define file collections as inputs or the new lazy properties, you can also simply configure the producing task as the file collection to use, or wire the output property to the input property of the consuming task and you get implicit task dependencies. This then also works if files are generated to other paths or if you don’t need some file anymore and thus also not the task dependency and so on.
With your example project, a more idiomatic version that also fixes the deprecation warning while preserving paths would for example be this:
Thank you. That is a very useful explanation. There are definitely things I still need to learn about Gradle. I’ll try to apply these ideas to the actual project and see how far I can get.
If pointless_tar produced several output files and I wanted to refer to each of them in some subsequent task (like pointless_gzip), how would I disambiguate them from files(pointless_tar)?
Actually that’s a point where you should consider writing a proper custom task either in buildSrc or what I prefer not in an included build, for example within gradle/build-logic/.
There you can then have multiple properties annotated as output and in the downstream tasks use only the output property you need.
If those output properties use the property / provider API, they should also carry that implicit task dependency automatically even if used “directly” individually.
If you don’t want that, but continue to use ad-hoc tasks, you indeed need to use the path.
But for that some additional thoughts:
if possible define the file path in a variable, a simple def outputfileoftaskX = ... will do, then you can use it in the producer and consumer task and if you change the path, both get the change
you need to specify the task dependency manually again, but better not with dependsOn, but with builtBy, which at least carries the semantic why you have that dependency like in Example 85 on Gradle User Manual: Version 7.0
I’ve struggled to find examples that I understand in the Gradle web resources. Can you point me to an example of a custom task that works the way you describe? I’m perfectly happy to write custom tasks where that’s the “right way”.
I’m not aware of an example with multiple outputs, but take any example with one output and make two of them.
It is also hard to give an example here, as it depends on the language you are going to use (Groovy, Java, Kotlin, any other JVM language, …).
But basically said you just declare two abstract read-only properties of type RegularFileProperty that you annotate with @OutputFile.
Thank you! I will take a look and do some experimenting. I use Gradle all over the place and I really would like to up my game. I expect I’m doing some pretty ham fisted things in some of my repos!
If you’re curious, my first experiment with fixing the issues I was having is checked into the XProc 3.0 test suite repository. It works, so it’s a success at least in that regard. I had two copy tasks that needed the rewrite you described, so I attempted to encapsulate that in an extension task type. It’s a bit clunky. But that got me thinking about using extensions and I think I improved a bunch of Docker-related stuff with a Docker extension.
Odds are, I’ll be able to improve this over time, but for now, at least it works!
@Vampire , would you please clarify why .copy is better than Copy? I see how it can help to workaround the warning, however, it looks like replacing Copy with copy would introduce bugs and the distinction is moot.
I didn’t say it is better, I said it is different.
And using it is not a work-around for the warning, but a proper fix of the problem that was present before just was not detected and warned about by Gradle.
You can for example instead also define a unique destination directory for the task of type Copy as I said.
If you have tasks with overlapping outputs => very bad for various things like up-to-date checks, task output caching, …
If you have Copy task “copy A into folder X” and Copy task “copy B into folder X”, then both of these tasks have the contents of X as output and thus are overlapping and depending on which tasks were run in which order recorded outputs can be different and so on.
If you have an ad-hoc task where you use the copy method, then you can define properly that only that one copied file (or whatever files you copied) is the output of that task.
If I have the choice, I would usually prefer to use a Copy task with a distinct output directory. Or actually then not a Copy task but a Sync task to also make sure there are no stale files in the destination. But if the destination cannot be changed, for example by request of Norm or because you want to copy a readme template into the root project directory with updated version numbers, then using the ad-hoc task with copy method is an acceptable implementation.
More idiomatic would of course be to write a full custom task in buildSrc or an included build and there use the copy method in the task action or use any other means of copying the file.
hi @Vampire good explanation…but i am stuck with another strange issue…
i am trying to create release build apk of a react-native project… and when i try it, BUT, i got a lot of gradle problem-warning …here is one sample…
Task :app:processReleaseManifestForPackage Execution optimizations have been disabled for task ‘:app:processReleaseManifestForPackage’ to ensure correctness due to the following reasons:
Gradle detected a problem with the following location: ‘D:\testapp\android\app\build\intermediates\merged_manifests\release’. Reason: Task ‘:app:processReleaseManifestForPackage’ uses this output of task ‘:app:copyReleaseBundledJs’ without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed. Please refer to Dealing with validation problems for more details about this problem.
this same problem-warning is repeated for many other tasks!!.. NB i havent added any packages or changed any source code… its the barebone app coming with react-native!.. so how do i solve this issue?.. how to fix my settings so that this warning can be removed… (i copied one of these tasks-names and did a text-search inside all files in my project directory to find out where its mentioned…but shockingly, its not mentioned anywhere!!!..so where are all these problem-warnings coming from? and how can we get a neat gradle build?) this same warning is repeated in lots of lines in gradle output for other tasks too, such as:
I have no idea about Android or react-native. What the problem is, is described in the docs and here. What your concrete problem is or how to fix it I can hardly guess. But as you say “its the barebone app coming with react-native”, maybe you should complain to react-native.
thanks for trying to help @Vampire … in your excellent reply above you have described how a person can implicitly set order of execution of tasks… but if i am not mistaken its about a task in same package…
but if a task is in another package, how can i access it?..
I’m not sure what you mean.
Which of my comments are you referring to, that was more than 10 months ago.
But whatever you mean, packages should not have any influence.
