How to create a task with outputs that aren't known during configuration phase?

Help/Discuss
1

My main goal is to create a task that takes in a list of files as TaskInputs, performs some filtering, then returns the result using TaskOutputs so other tasks can easily reference the filtered jar list. Other tasks need to be able to reference the filtered list during the configuration phase without actually resolving the file collection.

My task’s inputs come from a custom configuration called allProjectMainJars; declaring that looks like this:
inputs.files configurations.allProjectMainJars

This works as expected and at task execution time the inputs resolve into the expected list of jar files.

The problem I’m running into is that I can’t actually update the task outputs at task execution time due to this error: Cannot call TaskOutputs.file(Object) on task 'xxx' after task has started execution. Is there are workaround for this? Is there an alternate way to product a lazily-resolved list of files that can be consumed by other tasks during the config phase?

I need to be able to reference the filtered list during the configuration phase because tasks like Copy will just be skipped if no valid input files are registered during the config phase, but I want to avoid triggering any actual resolution during the config phase since that makes the build script become sensitive to the ordering of project declarations which is definitely undesirable.

2

Why not use TaskOutputs.dir(...) and put all the outputs in the same directory (nested subdirectories allowed)?

1 Like
3

Definitely a reasonable workaround, but I was hoping to avoid having to copy files around disk just to work around Gradle restrictions (and also it’s just more convenient to be able to expose an actual list of Files to consumers rather than a dir that they might have to manually list out).

I eventually worked out a solution that seems to behave how I want it to based on info from https://docs.gradle.org/current/userguide/lazy_configuration.html :

project('filter-test') {
    task makeFullFileList {
        outputs.files files(["f1", "f2"])
    }

    task makeFilteredFileList {
        inputs.files makeFullFileList.outputs
        
        ext.filteredFileList = files() // Create an empty ConfigurableFileCollection property on this task
        filteredFileList.builtBy it // Mark this task as the builder of the ConfigurableFileCollection
        
        doLast {
            for (File f in inputs.files) {
                if (f.name == 'f2') {
                    filteredFileList.from f
                }
            }
        }
    }

    task printFilteredFileList {
        inputs.files makeFilteredFileList.filteredFileList
        
        println "Config phase inputs: "+inputs.files.files
        
        doLast {
            println "Exec phase inputs: "+inputs.files.files
        }
    }
}

./gradlew filter-test:printFilteredFileList 
Config phase inputs: []
Exec phase inputs: [~~~\filter-test\f2]
4

There’s FileTree.matching(…) which might be what you’re looking for. Eg:

task makeFullFileList { ... }
task makeFilteredFileList {
   // better to declare the task as an input than just wire the outputs to the inpus. This will setup 
   // a 'dependsOn' relationship between the two tasks as well wiring the outputs to the inputs
   inputs.files makeFullFileList
   outputs.files = makeFullFileList.outputs.files.asFileTree.matching {
      include '**/f2.txt'
      include '**/*.xml'
   }
   ...
}
5

Hi,

Gradle expects that the outputs are actually created by the task. If I understand correctly, your task does not create any outputs but instead selects some outputs from the input files.

You can create a FileTree from the allProjectMainJars with an adequate matching clause and then use that throughout without creating a task, right?

Another option is to write an artifact transform which filters out the undesired artifacts.

Would one of those two options work for you?

Cheers,
Stefan

1 Like
6

Hmm, the fileTree approach looks promising, but I can’t actually get it to work for some reason. Even in a simple test case where I don’t perform any filtering, I get a strange error about being unable to create a directory when I try to convert the configuration to a fileTree and declare it as an output.

After some more playing around I think the best approach might be to just set up a method (rather than a task) that produces a filtered file collection based on the configuration. @Stefan_Wolf is right that gradle is wired up to expect that task output files are actually created by the task; violating that might cause other weird problems down the road so I’ll avoid it.

Here’s a basic example showing the filtered FileCollection approach:

configurations {
	myConfig
}

def makeFilteredFileList() {
	return configurations.myConfig.filter {
		it.name.contains('area')
	}
}

task printFilteredFileList {
	inputs.files makeFilteredFileList()
	
	doLast {
		println "Unfiltered files: "+configurations.myConfig.files
		println "Filtered files: "+inputs.files.files
	}
}

dependencies {
	myConfig 'com.fasterxml.jackson.core:jackson-databind:2.9.9'
	myConfig 'com.fifesoft:rsyntaxtextarea:2.6.0'
}

As expected / desired, the “Filtered files:” printout only includes the rsyntaxtextarea jar and the configuration does not actually get resolved until task execution time.

7

The code above is forcing configurations.myConfig to be resolved in the configuration phase rather than the execution phase because you are using Collection.filter(...)

8

Based on my testing, calling filter does not actually trigger resolution of the configuration. Note that I intentionally put the dependencies block at the bottom of the file in that example to verify / demonstrate this. If it was actually triggering resolution, the dependencies block would blow up during the config phase with an error about not being able to modify a config after it has been resolved.

FYI I’m running this under Gradle 6.6 which may or may not be relevant.