Task file inputs vs task file dependencies

Help/Discuss
1

I have a task that takes .js files as input for processing. I have setup an incremental task for processing these files. In additional, I have a file collection where if the files or contents change, all input files should be reprocessed. The files in the second collection should not be input files that processed by the task itself, they only cause the task to process all input files as opposed to attempting incremental processing. Thoughts on how to accomplish this?

2

Hi Michael,

Achieving “these inputs invalidate these other inputs” might be tricky with an incremental task. We’re curious if you are using an incremental task because processing files one-at-a-time is slow or because your solution requires incremental changes.

It would be helpful if you would share more specifically the type of JS transformation you’re working on? This sounds like a bundling task like RequireJS.

3

Like @eriwen mentioned, there’s not a good built in way to say “these inputs invalid everything”.

You can fake it by inspecting which files have changed and rebuilding everything yourself.

Here’s an example:

class ExampleTask extends SourceTask {
    @InputFiles
    FileCollection rebuildAll

    @OutputDirectory
    File outputDir = new File(project.buildDir, "output")

    @TaskAction
    void generate(IncrementalTaskInputs inputs) {
        if (inputs.isIncremental()) {
            boolean mustRebuild = false
            List<File> rebuildOnly = []
            inputs.outOfDate { change ->
                if (rebuildAll.contains(change.file)) {
                    mustRebuild = true
                } else {
                    rebuildOnly << change.file
                }
            }
            inputs.removed { change ->
                if (rebuildAll.contains(change.file)) {
                    mustRebuild = true
                } else {
                    mapToOutput(sourceFile).delete()
                }
            }
            if (mustRebuild) {
                rebuildEverything()
            } else {
                processFiles(rebuildOnly, "incremental\n")
            }
        } else {
            rebuildEverything()
        }
    }

    File mapToOutput(File input) {
        new File(outputDir, input.name)
    }

    void processFiles(Iterable<File> source, String message) {
        for (File sourceFile : source) {
            mapToOutput(sourceFile).text = message
        }
    }

    void rebuildEverything() {
        project.delete(outputDir)
        outputDir.mkdirs()
        processFiles(source, "rebuildEverything\n")
    }
}

task mkSrc {
    doLast {
        file("src").mkdirs()
        file("src/srcA.txt").text = "A"
        file("src/srcB.txt").text = "B"
        file("src/srcC.txt").text = "C"

        file("config").mkdirs()
        file("config/configX.txt").text = "X"
    }
}

task example(type: ExampleTask) {
    source fileTree("src")
    rebuildAll = fileTree("config")
}

Run gradle mkSrc to create the src and config directories. Then run gradle example (all of the output files should have a “rebuildEverything” message). Then change a single file in src and the corresponding output file should change to “incremental”. Then change a file in config and all the outputs should have “rebuildEverything” again.

4

sterling, this was my first thought on how to handle this situation. But, was hoping that there was built-in support for this scenario. In my case, the additional “rebuild all” files are similar to a classpath used during the processing.

What I have in place that I am trying out now is an input property that contains a combined checksum of the set of non-processed files. This property value will change with any change to the file collection or the individual file contents (in fact, I believe that your suggestion would not be able to tell which list a deleted file came from, where the checksum difference will allow the full rebuild to be triggered). The processing does seem to read much better than checking each change/removed file for which list it is in and processing accordingly.

5

We are going to improve incremental task inputs soon, so that each change will also contain the information from which input property it came. Then you can more easily make your decision without doing contains() checks or other workarounds.