Short-circuit dependency chain if task outputs exist

Help/Discuss
1

I have a linear chain of dependent tasks: consume depends on generate depends on download. The download task is slow and unreliable, but changes rarely, so we intend to produce generate’s single output file once and store it in our source repository. We’d like to consider the generate output as up-to-date if it exists at all. But if generate’s output file is missing, then the download should happen followed by generate.

These three tasks are configured as follows, making full use of lazy configuration and task configuration avoidance:

final download = tasks.register('download') {
	outputs.file layout.buildDirectory.file(name)
	doFirst {
		println "$it: run"
		outputs.files.singleFile.text = ''
	}
}

final generate = tasks.register('generate') {
	inputs.files download
	outputs.file layout.buildDirectory.file(name)
	doFirst {
		println "$it: run"
		outputs.files.singleFile.text = ''
	}
	outputs.upToDateWhen { outputs.files.singleFile.exists() }
}

tasks.register('consume') {
	inputs.files generate
	outputs.file layout.buildDirectory.file(name)
	doFirst {
		println "$it: run"
		outputs.files.singleFile.text = ''
	}
}

Unfortunately, the outputs.upToDateWhen { outputs.files.singleFile.exists() } directive in the generate task is insufficient. If I remove build/download, then running the consume task causes the download task to run as well, even though generate’s own outputs are claimed to be up-to-date already. I have also tried adding onlyIf { !outputs.files.singleFile.exists() } to generate’s configuration, but that is likewise ineffective: removing build/download still causes download to rerun even if generate’s output file already exists.

The only way I can think of to prevent unwanted download task runs is to configure consume in two different ways depending on whether generate’s output file exists:

tasks.register('consume') {
	final alreadyGenerated = generate.get().outputs.files.singleFile
	inputs.files (alreadyGenerated.exists() ? alreadyGenerated : generate)
	...
}

That will work, but having the task dependency graph change depending on build-time conditions feels sketchy. Is there a more elegant way to manage my situation?

2

Hi Ben,

I don’t see what you try to achieve by adding calls to upToDateWhen. All the tasks will be up-to-date when you declare the outputs and inputs and nothing changes. You cannot prevent Gradle from checking the contents of the inputs and outputs, so adding the call to upToDateWhen doesn’t help.

So what you want to do is have up-to-date checks for the download task? The task will fail if it can’t download the file and it will be up-to-date if the file is already downloaded. So you should be good already, right?

Cheers,
Stefan

3

If you declare a file in inputs.files or outputs.files then gradle will ALWAYS consider the file(s) in the UP-TO-DATE checks.

Adding an outputs.upToDateWhen {} closure does not stop the normal input/output file checking. It simply gives you as a developer another chance to stop the task from being UP-TO-DATE. Returning true from this closure does not override the usual input/output file hash checking.

I get the feeling that you should only use a custom upToDateWhen {} closure and not use inputs.files This will mean that your custom logic is the only thing that’s considered in the UP-TO-DATE check.

4

I don’t see what you try to achieve by adding calls to upToDateWhen.

What I “try to achieve” is to consider the generate output as up-to-date if it exists at all. Surely I stated that clearly in my original post. Your deeper understanding of Gradle may make it obvious to you that my attempts to achieve this were misguided, but you can’t honestly say that you don’t know what I am trying to achieve.

You cannot prevent Gradle from checking the contents of the inputs and outputs, so adding the call to upToDateWhen doesn’t help.

This is an interesting limitation of upToDateWhen that I did not find in the documentation. That limitation may be obvious to you given your clearly superior knowledge of Gradle, but was certainly not obvious to me. Perhaps the upToDateWhen documentation should point this out for the benefit of ignorant folks like me.

So what you want to do is have up-to-date checks for the download task? The task will fail if it can’t download the file and it will be up-to-date if the file is already downloaded.

We want to retain (in version control) the output files produced by the generate task. You seem to be proposing that we instead retain the file retrieved by the download task. Unfortunately, that is not an option for intellectual-property reasons. We’re not allowed to retain the file that download retrieves, even though we are allowed to retain the file that generate produces from it.

5

Adding an outputs.upToDateWhen {} closure does not stop the normal input/output file checking. It simply gives you as a developer another chance to stop the task from being UP-TO-DATE. Returning true from this closure does not override the usual input/output file hash checking.

OK, that goes a long way toward explaining what I was seeing. Thank you for explaining this so clearly, without making me feel ignorant for making an honest mistake.

This may be a common mistake as well. I have come across similar questions from other Gradle users who could not get upToDateWhen to do what they wanted. I think they had the same misconception I did: they didn’t realize that upToDateWhen can only augment the standard checks, not replace them. Perhaps the upToDateWhen documentation should point out this limitation for the benefit of other Gradle users.

I get the feeling that you should only use a custom upToDateWhen {} closure and not use inputs.files This will mean that your custom logic is the only thing that’s considered in the UP-TO-DATE check.

Hmm, interesting option. I was already tweaking inputs.files for the consume task depending on whether the generated file already existed. I’m not sure whether you’re suggesting keeping my hack there or instead leaving inputs.files empty in the generate task. Either way, I see what you mean as an overall strategy: by leaving inputs.files empty I retain more complete control over the UP-TO-DATE check, whatever I want that logic to be. That is a useful way of looking at things. Nice.

I think I will keep my inputs.files (alreadyGenerated.exists() ? alreadyGenerated : generate) approach for now. It does the right thing, with no use of upToDateWhen. The more I think about it, the more I think it is an accurate reflection of the dependency semantics I want:

  1. If the generated file already exists, then the file itself is the input to consume, and we don’t know or care how that file came to be.

  2. But if the generated file does not already exist, then the file as produced by the generate task is the input to consume.

That’s a pretty straightforward interpretation of what inputs.files (alreadyGenerated.exists() ? alreadyGenerated : generate) is doing. Given my nonstandard requirements, that may be as elegant as I can expect things to be.

Thanks again for improving my mental model, @Lance! I expect it will be useful in the future too.

6

I think I understand now what you want your build to do:
skip the generate tasks when the output file of the generate task already exists.

I am not yet sure if you need the output of the download task as well, even when you have the output of the generate task. My suspicion is that you don’t, so you may want to skip it as well when the output of the generate task is present.

Gradle has a way to skip tasks (i.e. don’t execute them depending on a condition): Task.onlyIf(). See the user manual for more information.

In your case, you can add the onlyIf condition to the generate and download tasks:

download.onlyIf { !layout.buildDirectory.file(name).exists() }
generate.onlyIf { !layout.buildDirectory.file(name).exists() }

Cheers,
Stefan