ParallelizableTask is removed in gradle 4.0

Help/Discuss
1

Hi there,

Apparently, ParalleizableTask is removed in gradle 4.0. Is it still possible for me to enable parallel tasks in a single project?

Much appreciated.
Cheers,
Tony

2

We replaced @ParallelizableTask (which was inherently unsafe) with the new WorkerExecutor API. This API allows tasks to kick off some work and allows other tasks to start executing while the first one is waiting for that work to finish. The API is much safer than @Parallelizable task because it strongly encapsulates the work from any mutable state. The API is not yet documented well, something we will fix in 4.1. For now, this integration test is the best hint I can give you.

3

thanks Stefan,

I played around with workerExecutor a little bit and I wonder if you could help me convert from @ParalleliableTask syntax to workerExecutor syntax

Originally I have something like this:

@ParallelizableTask
class CustomExec extends DefaultTask {
    @TaskAction
    public exec() {
       project.exec {
           commandLine cmdArgs
       }
    }
}

now with WorkerExecutor, how do I pass the project variable to the Runnable?

class CustomExec extends DefaultTask {
        public class CustomExecRunnable implements Runnable {
             @Inject  
             public CustomExecRunnable(cmdArgs) {
                  this.cmdArgs = cmdArgs
             }
        
        public void run() {
            // I tried to pass this project through workerExecutor.submit 
            // and it complains that project_decorated is not serializable
            project.exec {
            }
        }
    }
    
    @TaskAction
    public exec() {
        workerExecutor.submit(CustomExecRunnable.class) {
            params = [cmdArgs] // project can't be passed to the thread through params
        }
    }
}

or maybe, I should word the question as is there any other way to pass the exec closure to the CustomExecRunnable? or should I implement my customExec in different way so it’s easier to work with workerExecutor?

thanks
much appreciated
Tony

4

Passing the project through is not allowed since that would leak all the mutable state, making the API unsafe. We will allow injecting some important services in future releases. One of those services would be the ability to use exec(). Right now you will have to use Java’s ProcessBuilder API.

5

thanks Stefan,

It seems like I’m facing 2 options here:

  1. using Java’s ProcessBuilder API, I have done this in the past but it didn’t work out quite well for me due to the fact that I need to get the process builder to work nicely with gradle api (like logging, and interpreting the exit code). Do you have any recommended references (url, blog, wiki, etc…) that I can use to use ProcessBuilder with gradle customTask object?

  2. delay upgrading gradle until you allow inject “exec()” service into workerExecutor? do you have jira ticket on that so I can watch, at the moment, can you have rough rough guess-estimate on when would be that “future” release? is it soon? or we’re looking at like 1 year from now?

Thank you very much for your prompt response Stefan.

Cheers,
Tony

6

There shouldn’t be much to do there. Gradle already forwards output if you just use System.out/System.err and if you get a bad exit code you just throw an exception.

We’re very likely to fix that this year, because we’ll need it ourselves to make our own tasks fully parallelizable. Feel free to open issues on GitHub for any missing pieces.

7

When you were using @ParallelizableTask, did you also set org.gradle.parallel.intra=true? If you didn’t, @ParallelizableTask had no effect and you don’t need to change anything.

8

I did set the org.gradle.parallel.intra=true. I have tried that and it doesn’t work.

According to this commit (https://github.com/gradle/gradle/commit/06bc2b88551bad2eef8004e08ec22636bfdbc74e) that org.gradle.parallel.intra flag check has been removed, so I think that org.gradle.parallel.intra doesn’t have any effect anymore.

9

This really sucks.

One of the huge performance benefits we were getting from gradle was the ability to run tasks within a project in parallel. Just as an example, I have several long-running javascript build tasks (some for angular, some for dojo) as part of the same webapp that have been running in parallel quite happily for some time and suddenly – without even a comment in the release notes – that’s all dead if I want to upgrade.

I jumped through all the hoops to make sure every task was correctly running after any other task it needed to and spent days tweaking all the mustRunAfter settings to make sure that we could use this feature and now I can’t upgrade. If I were to upgrade to 4.0 my build would slow down by 10 minutes or more.

Were tasks that were previously marked with @ParallelizableTask (like Copy, Jar, JavaCompile, etc…) upgraded to use whatever replacement mechanism you’re going with? If not, is that planned or will it be left to the developer?

I really like gradle, and I appreciate how quick you all generally are to answer questions and help out but doing this without even a mention in the release notes just sucks. Incubating or not, this was a feature that had existed for nearly 3 years and you just killed it, somewhat blocking the upgrade path for anyone using it.

1 Like
10

Hey Mike,

the parallel.intra flag was never an official feature and we’ve rarely seen it used in the wild, because it required such extreme caution to get right. Even the somewhat safer --parallel flag is only used by a small fraction of our user base, because it is inherently unsafe. That being said, a mention in the release notes would have been worth it.

The Worker API (see my responses above) replaces @ParallelizableTask, providing a safe way to run work in the background.

The existing tasks have not yet been migrated to use the worker API, but this is absolutely planned. You can already migrate your own tasks. If those are dominating your build, you should get similar performance to what you had before.

We’ll add proper documentation for the worker API soon, in the meantime feel free to reach out to me on this topic.

Cheers,
Stefan

1 Like
12

Hey,
Do we have this feature now in the Worker API ? I want to use project instance in my tasks heavily but couldn’t use as worker API doesnt support it. Can we inject now functions like copy(), exec(), fileTree() etc ?

13

Can we uses project.javaexec in worker api ?