Incremental build of copy tasks

Help/Discuss
1

Hello,

I have an issue with the task Copy. When I have for example a simple task like the one below, then I thought, that gradle supports the incremental execution of this task out of the box.

task copyMyStuff(type: Copy) << {
        from 'src'
        into 'build'
}

Lets say the folder ‘src’ contains two files. After executing the task above, both file are copied by gradle into ‘build’. If I then delete one of the two files from the folder ‘src’ and execute the task again, then gradle assumes that everything is UP-TO-DATE, but the folder ‘build’ contains still both files.

Did I something wrong or is this a bug?

Markus

2

Remove the left shift operator (<<) from your task definition and it will work properly. Currently, you are defining the configuration for your task as action code. Action code (doFirst, doLast) is executed during the execution phase of Gradle’s build lifecycle which is too late for the task to be configured. Please see the user guide for more information.

3

Thanks for the fast reply. If I remove the left shift operator, then I have another issue: After deleting one of the src-files, the task is executed by gradle, but the folder ‘build’ still contains the deleted file. I am looking for a task, that copies the new and modified files from the src and also removes deleted files.

4

The Sync task is probably what you are looking for.

5

Thanks. You’re right. I was looking for the Sync task. I am wondering me that the java plugin uses also the Copy task and it is working like the Sync task.

6

None of the tasks provided with the Java plugin are of type Sync. You can have a look at their type on the documentation page. You are probably running the clean task so it might appear that previous build outputs are removed.

7

Hi @bmuschko,

I’m stuck on getting an incremental copy. I plan to do something more sophisticated, but at the moment, I can’t get the following to work incrementally:

task copyJsps(type: Copy) {
    from 'src/main/webapp'
    into 'build/webapp'
    include '**/*.jsp'
}

Whenever I change a single jsp file under src/main/webapp, the task copies all jsp files, not just the one I changed. I’m running the following command, with Gradle version 3.4.1:

./gradlew -t copyJsps

After which, I see the following output:

Continuous build is an incubating feature.
Parallel execution is an incubating feature.
copyJsps

BUILD SUCCESSFUL

Total time: 7.56 secs

Waiting for changes to input files of tasks...

Upon modifying a single jsp file, I see the following (with parts of the file path redacted):

new file: ...\src\main\webapp\...\...jsp
Change detected, executing build...

copyJsps

BUILD SUCCESSFUL

Total time: 4.094 secs

Waiting for changes to input files of tasks...

I thought it was odd that it took several seconds to copy a single file (even longer when I didn’t limit the copy to jsp files only). It also appears odd that the output includes new file: ... when I’m modifying an existing file.

Observing the output directory, I see that every time I change only 1 file, all jsp files are copied to the output directory, as evidenced by the fact that the timestamps on all files in the output directory are updated.

I thought this should be an incremental operation, meaning that only changed files would be copied. What am I missing?

Thanks,
Chuck

8

Nothing. The build is incremental in the sense that the copy task only executes when things have changed but it isn’t incremental itself, in that it only copies inputs that have been modified. This could be improved but right now if you want more fine grained incrementalism, your only option is to split the task up into pieces.

9

It wouldn’t be difficult to write an IncrementalCopy task if you’d like to only copy / delete files which have changed. See here

10

@mark_vieira Thanks for clarifying. I thought I was doing something wrong. Would you be able to provide some detail regarding your suggestion of splitting the task into pieces?

11

@Lance Thanks for the pointer. I’ll take this approach. I had presumed that the out-of-box Copy task was actually doing what you’re suggesting I must implement myself. In fact, it seems strange that it doesn’t. Any idea why it does not behave as I thought it would? Offhand, I can’t imagine why its non-incremental output would be useful.

12

Whilst it’s often used for simple copy from A to B the Copy task supports hierarchies of with(CopySpec) and also renaming files and transforming content via custom closures etc.

Eg: you could put the current date/time in the copied file. What should an incremental copy do in this instance?

Incremental task support is a recently new addition to gradle but the Copy task has been there since day obe.

I imagine that incremental support whilst also keeping the current features is not trivial. A custom simplified incremental Copy task on the other hand would be easy to implement

13

Thanks for the details. Regarding your example of performing a transformation on an input file, as far incremental behavior is concerned, my answer to your question would be that the transformation should occur only on the changed input file(s).

At least from my perspective, the point of “incremental” is to perform whatever actions/transformations are desired only on the incrementally changed inputs, producing only incrementally changed outputs. That is, the purpose of making a task “incremental” is precisely to avoid taking unnecessary action in order to significantly shorten build time, isn’t it?

If you’re doing something like inserting the current date/time (or whatever the action is) in a changed input file, why would you want to also do so on all of the unchanged input files?

Of course, as you said, given that the existing Copy task has been around since the beginning, and support for incremental tasks is quite recent, it make sense that the existing task does not behave as I was hoping, so creating a simplified custom incremental copy task is a perfectly viable solution for me.

14

The Copy task could be incremental this way. I believe this is more historical than anything else, and in most cases the operation is quick, unless you are shipping around really big files. There are likely edge cases that make this trickier than it initially seems as well, transforms definitely complicate things.

15

Did someone implemented it ? I tried to look at gradle source code for Copy task, but it’s not easy to understand.
(And I can’t do a naive implementation as my input is a CopySpec and I have no idea how to read/handle information from it)