Extending Copy task to support parallel execution

Help/Discuss
1

Hello!

I want to optimize my build by running certain things in parallel. Using the combinations of mustRunAfter and others, I have seemed to optimize the build quite significantly, but there is still one spot where I can’t seem to force gradle to run things in parallel.

The last three tasks of my build are all Copy type tasks. All three of them simply need to wait until “assemble” task is complete for every single module (I’m working with a multi-project architecture) and then all three tasks should fire at the same time. But they only run one after another.

Looks something like this:

task parallelTask(dependsOn: ['assemble', 'copyOne', 
'copyTwo', 'copyThree']) {
    copyOne.mustRunAfter(assemble)
    copyTwo.mustRunAfter(assemble)
    copyThree.mustRunAfter(assemble)
}

First, I have found a solution to try and use @ParallelizableTask, but as I understand, this annotations has been removed in gradle 4. So I am trying to use WorkerExecutor API to extend Copy task so it can run in parallel, but so far I have found no way of actually doing it. Is it even possible to do? Maybe I am looking in the wrong direction altogether? I am still fairly new to gradle, so please excuse me for I am still very much a noob.

Thank you!

2

Ultimately, there should be a core gradle task which can copy in parallel. Until this time, it’s possibly easier to write a limited ParallelCopy task from scratch rather extending the Copy task.

Eg:

class ParallelCopy extends DefaultTask {
   private final WorkerExecutor workerExecutor

   @Inject
   ParallelCopy(WorkerExecutor workerExecutor) { this.workerExecutor = workerExecutor; }

   @InputFiles
   private List<FileCollection> froms = []

   @OutputDirectory
   private File into

   public void from(Object from) {
      this.froms << project.files(from)
   }
   public void into(Object into) {
      this.into = project.file(into)
   }
   @TaskAction
   public void parallelCopy() {
      for (FileCollection fc : froms) {
         if (fc instanceof FileTree) {
            ((FileTree) fc).visit { FileVisitDetails fvd ->
               if (!fvd.directory) {
                  File destination = new File(into, fvd.path)
                  submitWorker(fvd.file, destination)
               }
            }
         } else {
            for (File file : fc.files) {
               File destination = new File(into, file.name)
               submitWorker(file, destination)
            }
         }
      }
   }
   private void submitWorker(File source, File destination) {
            workerExecutor.submit(ParallelCopyWorker.class, new Action<WorkerConfiguration>() { 
                @Override
                public void execute(WorkerConfiguration config) {
                    config.setIsolationMode(IsolationMode.NONE); 
                    config.params(source, destination); 
                }
            });
   }

   public static class ParalellCopyWorker implements Runnable {
      private final File source;
      private final File destination;
      @Inject
      public ParalellCopyWorker(File source, File destination) {
         this.source = source;
         this.destination = destination;
      }
      public void run() {
         destination.parentFile.mkdirs()
         destination.bytes = source.bytes // TODO use buffer/stream instead
      }
   }
}

Usage

task copy1(type:ParallelCopy) {
   from fileTree('somePath').matching { 
      include '**/*.txt'
   }
   from 'another/path'
   into 'some/destination'
}
1 Like
3

Thank you very much! This seems to work amazingly well.

Also, do you know how to make sure that this task doesn’t copy unchanged files? Yesterday stubled upon the fact that default “Copy” task sufferes from this issue.

4

I’m just asking maybe there is a better way than simply using

FileUtils.contentEquals

for every single file

5

Also, do you know how to make sure that this task doesn’t copy unchanged files?

I think you’re looking for the behaviour of Gradle’s Sync task. I suggest that you look at the source code

6

Hmm, from what I understand, Sync task simply deletes everything from the source folder before copying. I don’t think if performs any sort of check whether the fils is unchanged

7

I don’t think if performs any sort of check whether the fils is unchanged

Ah, that’s possibly the behaviour. You could change my implementation to an incremental task by accepting IncrementalTaskInputs argument in parallelCopy() method and then driving the worker creation and deleting logic based on IncrementalTaskInputs

8

Thank you, I will try that

9

I’ve just noticed that InputFileDetails does not have a getPath() method which might make it hard to do this via an incremental task if you are copying a FileTree. How annoying, it would be nice if the Gradle team could add this (under the hood it would be the value of FileVisitDetails.getPath() for input files within a FileTree)

10

I created a github issue to support nested files in incremental tasks

11

Since Gradle 5.4 we have a new API for incremental tasks that supports getting the normalized path of a change. See https://docs.gradle.org/5.6/userguide/custom_tasks.html#sec:implementing_an_incremental_task.

1 Like