How to make a task input depend on another task output which is computed during the task execution?

Help/Discuss
1

I want to write two tasks, first of which generates a set of files which are keyed by a string identifier (so this is basically a Map<String, File>), and another, separate task which would accept this map of files and a command-line option from the user identifying a file from this map, something like this:

# create a set of files, e.g. build/packaging/something-{A, B, C}-1.0.0.zip
$ ./gradlew packageGeneric 

# select the above build/packaging/something-A-1.0.0.zip and package it in a different way
# naturally, should invoke packageGeneric first
$ ./gradlew packageAlternative --kind A

The important thing here is that the actual set of identifiers, namely, A, B and C above, is not known until the packageGeneric task action is executed (which queries the file system, among everything else, to collect these identifiers), and it is not and should not be set by the user.

My initial approach was to define the tasks like this:

open class PackageGenericTask : DefaultTask() {
    // Used by the task action code, indirectly determines the set of output keys
    val configDirectory: DirectoryProperty = newInputDirectory()
        @InputDirectory get

    val packagedArtifacts: Provider<Map<String, File>>
        @OutputFiles get() = mutablePackagedArtifacts

    private val mutablePackagedArtifacts = project.objects.property<Map<String, File>>()

    @TaskAction
    fun run() {
        // do some computations and file operations and call mutablePackagedArtifacts.set(map)
    }
}

open class PackageAlternativeTask : DefaultTask() {
    val packagedArtifacts: Property<Map<String, File>> = project.objects.property()
        @InputFiles get

    var kind: String? = null
        @Input get
        @Option(option = "kind", description = "Kind") set
}

// Inside the MyPlugin::apply method

val packageGeneric = project.tasks.create("packageGeneric", PackageGenericClass::class.java) {
    // copy settings from extension
}

project.tasks.create("packageAlternative", PackageAlternativeClass::class.java) {
    packagedArtifacts.set(packageGeneric.packagedArtifacts)
    dependsOn(packageGeneric)
}

This approach failed with weird stacktraces about unset property values.

I have already understood that it would not work anyway; @Output* properties are also supposed to be set by the user too and are actually input properties for the task from the configuration point of view; the only difference from @Input* properties is semantics of what actually is stored in these configured properties, but from the configuration standpoint they all should be set before the task is run. Therefore, because packagedArtifacts.set in the second task configuration is run at the configuration time, nothing really would be set here (I had hoped that “lazy” properties would do the trick, but apparently they did not, even after I removed the @Output* annotations to avoid validations).

Surprisingly, I was not able to find anything on how to do what I want in Gradle. Most of the information on the Internet is about how to make a task depend on another or how to handle simple cases when one task depends on configuration of another task (e.g. using jar.archivePath in some packaging task), which does not work for my use case because I don’t have any concrete values until the task is actually executed.

To add a bit of context, I come from the SBT world, and this problem does not exist there: an SBT task always has an output value, and if some task depends on another task, it can read this value similarly to a function invocation:

packageGeneric := {
  // do computations
  map  // return value has type Map[String, File]
}

packageAlternative := {
  // the .value method both establishes a dependency on the packageGeneric task and returns its value
  val packagedArtifacts: Map[String, File] = packageGeneric.value
  val kind: String = argParser.parsed  // obtain the kind argument from the user
  packagedArtifacts(kind)  // access the map
}

My only remaining thought on how to properly do this is to store this map as a file (e.g. serialize the map as JSON) and pick it up in the dependent task, but this feels like a really, really huge crutch, way too huge for a really simple problem. I understand that I’m probably trying to do something which does not really map well to the Gradle model, and I suspect that there is an idiomatic way to do what I want in Gradle, but unfortunately I couldn’t find one.

1 Like
2

The best way to persist state between gradle invocations is the file system

You could just write two files in the packageGeneric task, the zip file and also a properties (json, xml, yaml) file containing the calculated variables.

The packageAlternative task could first read/parse the property file before looking up the zip etc

3

I don’t really need to persist data between gradle invocations - I would like to invoke one task which depends on another task, and pass the result of the second task to the invoked one.

That being said, I think I see now why this does not fit the gradle runtime model. Since tasks may not even be invoked because of caching, it is not guaranteed that the task would be able to expose the output values for another task to consume if its execution is cached. So I guess storing the output to filesystem is indeed the way to go for me. Thanks!

4

That’s a fair assessment

5

Note there’s also this pattern

task taskOne {
   doLast {
       FileCollection fromFiles = someExpensiveOperation() 
       tasks['task2'].configure {
          from fromFiles
       }
   } 
} 
task task2(type:Copy) {
    dependsOn 'task1' 
    into 'xyz'
}

But as you’ve discovered this will mean task1 will always need to execute. So someExpensiveOperation() can never be skipped hence file system is probably best

2 Likes