Is it possible (or even desirable) to combine @Incremental and a WorkerQueue?

Help/Discuss
1

There is an XSL task which processes many files from one directory into another. This process takes some time per file but could in theory be highly parallelized as all files are independent. As I was wondering if this work could be in any way parallelized I tried to combine @Incremental with a WorkerQueue, but I can’t get it to work. Not sure what I’m doing wrong.

interface XSLTWorkParameters extends WorkParameters {
    RegularFileProperty getStylesheet()
    Property<File> getInputFile()
    Property<File> getTargetFile()
}

abstract class TransformWorkAction implements WorkAction<XSLTWorkParameters> { 
    @Override
    public void execute() {
        try {
            File stylesheet = getParameters().getStylesheet().get().asFile
            File inputFile = getParameters().getInputFile().get()
            File targetFile = getParameters().getTargetFile().get()

            def transformer = new Transformer(stylesheet)
            println "Processing: " + inputFile
            transformer.transform(inputFile, targetFile)
        } catch (Exception e) {
            throw new RuntimeException(e)
        }
    }
}

abstract class TransformMultiTask extends DefaultTask {
    @Incremental
    @PathSensitive(PathSensitivity.NAME_ONLY)
    @InputDirectory
    abstract DirectoryProperty getInput()

    @PathSensitive(PathSensitivity.NAME_ONLY)
    @InputFile
    abstract RegularFileProperty getStylesheet()

    @OutputDirectory
    abstract DirectoryProperty getOutput()

    @Inject
    abstract public WorkerExecutor getWorkerExecutor()

    @TaskAction
    void processFiles(InputChanges inputChanges) {
        def cleanOutputDir = !inputChanges.incremental
        if(cleanOutputDir) {
            logger.info('Cleaning: {}', output.get().asFile)
            project.delete(output)
            output.get().asFile.mkdirs()
        }

        WorkQueue workQueue = getWorkerExecutor().noIsolation()

        logger.info('Stylesheet: {}', stylesheet.get().asFile)

        inputChanges.getFileChanges(input).each { change ->
            if (change.fileType == FileType.DIRECTORY) return

            def targetFile = output.file(change.normalizedPath).get().asFile

            if (change.changeType == ChangeType.REMOVED) {
                logger.info('Removing: {}', targetFile)
                targetFile.delete()
            } else {
                workQueue.submit(
                    TransformWorkAction.class,
                    parameters -> {
                        parameters.getStylesheet().set(getStylesheet())
                        parameters.getInputFile().set(change.file)
                        parameters.getTargetFile().set(targetFile)
                    }
                )
            }
        }
    }
}

Using Gradle 7.6.2 it results in

* What went wrong:
Execution failed for task ':my.project:generateFiles'.
> A failure occurred while executing TransformWorkAction
   > Could not isolate value XSLTWorkParameters_Decorated@6f261e34 of type XSLTWorkParameters
      > java.lang.StackOverflowError (no error message)

The stack-overflow comes from

Caused by: org.gradle.internal.snapshot.impl.IsolationException: Could not isolate value XSLTWorkParameters_Decorated@6f261e34 of type XSLTWorkParameters
        at org.gradle.internal.snapshot.impl.DefaultIsolatableFactory.isolate(DefaultIsolatableFactory.java:51)
        at org.gradle.workers.internal.DefaultActionExecutionSpecFactory.newIsolatedSpec(DefaultActionExecutionSpecFactory.java:49)
        at org.gradle.workers.internal.DefaultWorkerExecutor.submitWork(DefaultWorkerExecutor.java:198)
        ... 120 more
Caused by: java.lang.StackOverflowError
        at org.gradle.internal.snapshot.impl.AbstractValueProcessor.processValue(AbstractValueProcessor.java:130)
        at org.gradle.internal.snapshot.impl.AbstractValueProcessor.processValue(AbstractValueProcessor.java:134)
        at org.gradle.internal.snapshot.impl.AbstractValueProcessor.processValue(AbstractValueProcessor.java:134)
       ....
2

From a quick look I’d say parameters.getStylesheet().set(getStylesheet()) is not doing what you assume it does, but sets parameters.stylesheet to parameters.stylesheet resulting in the StackOverflowError due to this self-reference of the property.

1 Like
3

@Vampire You are fantastic. That was exactly the issue. I looked at it for hours and just couldn’t see it.
Many thanks!

1 Like