Implicit task dependency on directory

Help/Discuss
1

Hello,

I’m trying to understand implicit dependencies between tasks. Basically, I want a producer task to write something into a directory and a consumer task that reads something from there. As far as I understood implicit dependencies it should be possible that the consumer has an implicit dependency on the producer using the shared directory.
I don’t want a dependency on the task itself, because I will have more than one task writing into said directory and the consumer should be dependent on all of them.

What I’ve tried is:

abstract class ProducerTask: DefaultTask() {
    @get:OutputDirectory
    abstract val outDir : DirectoryProperty
    @TaskAction
    fun action() {
        println("Producer: ${outDir.get()}")
        outDir.get().file("foo.bar").asFile.writeText("foobar")
    }
}

abstract class ConsumerTask : DefaultTask() {
    @get:InputDirectory
    abstract val dir : DirectoryProperty
    @TaskAction
    fun test() {
        println("Consumer: ${dir.get()}, Content: ${dir.file("foo.bar").get().asFile.readText()}")
    }
}

and in the build.gradle.kts:

val myDir = layout.buildDirectory.dir("mytest")

task<ProducerTask>("producer") {
    outDir.set(myDir)
}

task<ConsumerTask>("consumer") {
    dir.set(myDir)
}

I expected that when I call the ConsumerTask the ProducerTask would be automatically triggered. But this does not work, instead the Consumer throws an error because the directory does not exist.

Can you please shed some light on how this should work?

2

You got it exactly the wrong way around.
You should not configure paths manually like that.
You should wire task output properties to task input properties.
By wiring the dependencies, you get the intended implicit task dependency that you actually should use. Practically any use of dependsOn without a lifecycle task on the left-hand side is a code smell.

Regarding depending on all tasks that write to that directory,
the point is that it is extremely bad if multiple tasks have overlapping outputs.
So having multiple tasks with the same output directory is a very bad practice anyway and should be avoided whereever possible (and it should always be possible).
With overlapping task outputs, the task up-to-date checks cannot work properly, task output caching cannot work properly, wrong path usage cannot be calculated properly, …

3

Thank you for your answer, Björn.

I know that using the same output directory is considered a code smell.
In our use case we have a custom task (TransformXml) which is executed multiple times with different parameters. After all of them are executed I want a task to pack them all into 1 zip file. My hope was, that the zip-File task could trigger all of these tasks at once (like mentioned here: Gradle 7.0 seems to take an overzealous approach to inter-task dependencies)

Can you help me with another similar problem:
I have a Task to extract an archive. Now I want another task to work with the extracted files and have an implicit dependency between them. How do I specify the output-input-relation correctly? I tried:

val unzip = task<Copy>("unzip")  {
    from(zipTree(myConfiguration.singleFile).matching {
        include("de/foobar/**")
    }.files)
    into("build/foobar")
}

task<ZipConsumer>("zipConsumer")
{
    inDir.set(unzip.destinationDir)
}

abstract class ZipConsumer: DefaultTask() {
    @get:InputDirectory
    abstract val inDir : DirectoryProperty

    @TaskAction
    fun action() {
        println("ZipConsumer: ${inDir.asFile.get().absolutePath}")
    }
}

But again, the unzip-Task is not executed automatically when I call zipConsumer. I really don’t want to use dependsOn because I can totally understand that this is not the intended way.

4

know that using the same output directory is considered a code smell.

It’s not only a code smell, it is a serious problem for build correctness.
Things like that are the reason you always have to use clean for Maven builds. :smiley:

In our use case we have a custom task (TransformXml) which is executed multiple times with different parameters. After all of them are executed I want a task to pack them all into 1 zip file. My hope was, that the zip-File task could trigger all of these tasks at once

Shouldn’t be a problem here, just use all of them as input to your zip task.
Actually if your TransformXml task has the single files it outputs as outputfiles, you can also generate them to the same output directory without problem, as then there is no overlapping outputs.
Just if you for example define the same @OutputDirectory for the tasks, then you have a problem.

But even then, just configure the TransformXml tasks to generate into separate directories. And then use the TransformXml tasks as input to your zip task and all works properly and cleanly.

If you want all existing TransformXml tasks, you can do something like

val zipit by tasks.registering(Zip::class) {
    from(tasks.withType<TransformXml>())
}

If you only want specific ones, well, configure them.

val foo by tasks.registering(TransformXml::class) { ... }
val bar by tasks.registering(TransformXml::class) { ... }
val zipit by tasks.registering(Zip::class) {
    from(foo)
    from(bar)
}

val unzip = task<Copy>("unzip")  {

Not your question, but don’t use this method, it is the old API that is not doing task configuration avoidance, so wastes your precious time when executing the build. See Task Configuration Avoidance for more information.

But again, the unzip-Task is not executed automatically when I call zipConsumer

unzip.destinationDir is a File.
The automatic task dependencies only work if you wire tasks, providers, file collections, or similar Gradle types to the inputs, as those can actually carry the information about the task dependency. The standard Java File class can not.
In this case it indeed gets a bit ugly to get the proper implicit dependency:

val unzip by tasks.registering(Copy::class) {
    from(zipTree(myConfiguration.singleFile).matching {
        include("de/foobar/**")
    }.files)
    into("build/foobar")
}

val zipConsumer by tasks.registering(ZipConsumer::class) {
    inDir.set(unzip.map { layout.dir(provider { it.destinationDir }).get() })
}

Unless there is an easier way I forgot right now as it is almost 4 AM. :smiley:

1 Like
5

Thank you again for taking the time to explain this to me (especially in the middle of the night).

I guess one of my main problems getting started with all of this are the many different ways to do something - and lot’s of deprecated ways, too. Depending on the time an example you look at was created, each one doing the same thing looks completely different.

For example the code to implicitly depend on the unzip: I would have never found this out on my own. My thinking was: this should be really easy, because doing something with the result of an unzip should be so common, there has to be an easy way. It will be difficult to convince my team to use something like “unzip.map { layout.dir(provider { it.destinationDir }).get() }” if a simple dependsOn just works.

Again, thank you!

6

Well, to work with the unzipped files themselves would be pretty easy.

The problem is, that you need the directory and that the Copy class do not provide that as Directory or a provider thereof so have to transform the File to a Provider<Directory> with the task dependency.

Maybe you like this version better:

val unzipOutput = layout.buildDirectory.dir("foobar")

val unzip by tasks.registering(Copy::class) {
    from(zipTree(myConfiguration.singleFile).matching {
        include("de/foobar/**")
    }.files)
    into(unzipOutput)
}

val zipConsumer by tasks.registering(ZipConsumer::class) {
    inDir.set(unzip.map { unzipOutput.get() })
}

Another option would be to not use a DirectoryProperty but a ConfigurableFileTree:

val unzip by tasks.registering(Copy::class) {
    from(zipTree(myConfiguration.singleFile).matching {
        include("de/foobar/**")
    }.files)
    into(layout.buildDirectory.dir("foobar"))
}

val zipConsumer by tasks.registering(ZipConsumer::class) {
    inFiles.from(unzip.map { it.destinationDir })
}

abstract class ZipConsumer : DefaultTask() {
    @get:InputFiles
    abstract val inFiles: ConfigurableFileTree

    @TaskAction
    fun action() {
        println("ZipConsumer: ${inFiles.dir.absolutePath}")
        println("ZipConsumer files: ${inFiles.files}")
    }
}
7

Sorry, I’ve been away a week.

YES, your first option looks pretty close to what I wanted. Now I just have to comment why I use unzip.map […] instead of just the directory, but it should be a lot more understandable now.

Thank you again for your help!

8

Why does Gradle assume that task outputs in the same directory == overlapping task outputs? I came to realize that 2 tasks outputting to the same dir is indeed a problem for Gradle, but I don’t understand why it should be. If a task has a very well defined output file path /a/b/output-1.txt, and another task has a very well defined output file path /a/b/output-2.txt, how is the /a/b prefix relevant? These are not overlapping file paths, and only the files are the outputs, not the dir. We can check those outputs creation time, hash them, delete them, etc. independently.

9

The prefix should not be relevant.
But as soon as one of the two uses /a/b as @OutputDir, you have overlapping outputs.
If those two tasks have only the concrete files as single output files, there is no problem.
If you see a problem with such a setup, please provide an MCVE :slight_smile: