Iterate over a CopySpec

Help/Discuss
1

Is there a simple solution to iterate over a CopySpec without copy the source files to a temporay location. The method eachFile is only invoked on execute. I need the file and the relativePath of the FileCopyDetails.

What I want to do is to convert some binary files. Samples:

  • The signing of a batch of jar files. Only the signed jar files should be copy in the target.
  • Transforming a batch of XML files.
  • Add a watermark to some images.
  • Calculating a checksum files.
  • Any type of compiling.
2

There is no public API for iterating over a CopySpec. However, these examples are all transformative in nature. As in, I have some input files, and I create output files. This should really be implemented in a custom task rather than a filter { } or eachFile { } as part of a copy. Realistically this would be modeled as a FileCollection or FileTree which is an input to a task whose action does whatever modifications to the input files as necessary.

3

I want to do this in a custom task. The task has a CopySpec as input. A FileCollection or FileTree are to simple because it does not support features like rename. A CopySpec as input is normal for many other tasks like Copy. Zip, Jar, 
 Why can I does not use in my own tasks?

4

CopySpec was simply never intended to be a generic input to custom tasks. It is closely tied to its implementation, which we provide. In your example of ‘rename’, CopySpec doesn’t really provide that feature, it simply provides the API. Your custom class would have to provide the implementation, as well as the implementation for all the other parts of the CopySpec API. In general the recommended approach is to define inputs using FileTree and FileCollection and then any other transforms you might potentially want to do as additional properties of the task. I would be wiling to guess in reality it will be a subset of that which is available on CopySpec.

5

Stepping back and thinking about this more I think it boils down to separating the “transform” bit of your build into a separate step. Realistically, those files that need transformation applied to them would be handled by a single task that does just that work. The output of that task would then likely be the input to a CopySpec. This is just good idiomatic Gradle, meaning that tasks should be highly cohesive. In a way we go down a slippery slope where the compileJava task is a Copy task with a highly complex transform applied to it. In general I think we want our copy tasks to be a dumb as possible and localize complex logic in separate tasks.

6

@Horcrux7 I think this conversation somewhat overlaps with this discussion on the mailing list. Perhaps you could provide some insight and elaborate on your use case there.

7

Here are some examples how it is done in plugins:

In the asciidoctor plugin, it is simply manipulating the CopySpec into a FileCollection.

In the vfs plugin, it is more complex where we actually extend and then implement the CopySourceSpec interface

8

Our use case is that all files come from a file repository. In our case it is ivy. The nature of the repositories is that the file names are not stable. The version change every day on every build. But our distributions required stable file names. There are many causes because file names must be stable. For example hard coding in existing configuration files of customers. Replacement on update of the software, etc.

We need also a special hierarchy of files. Plugins in the plugin directory, Libs in the lib directory, Templates in the template directory. etc. The repository is flat. The result is plain list.

The CopySpec is the only option to create a match between the files in the repository and the final file names. FileTree and FileCollection can not handle this in the declaration phase.

Of course we can copy the files to a temp location. This is what we currently do. But this sounds us an overkill only to receive the final file names.

9

@Horcrux7, thanks for the clarification. In the example above, it seems the the copy operation is primarily being used to establish a hierarchical file structure from a flat one, in addition to performing a rename. I agree, this seems like a good for for a CopySpec. Where do the file transformations come in? Of these files, is it just a subset that needs some sort of further processing?

10

It can be that I have not understand Gradles best practice. What we want to do repeatedly is to write a task that do some file processing. Like a copy task, zip task, etc. It has some input files define in a CopySpec and an output directory. Then the task can simple be used as input for the next task.

A possible BinaryFilter of the CopySpec can be a solution but this sounds not a good solution for me.

11

This is something we could potentially make easier. In a way this is exactly what a @LanguageTransform is in the software model.

I don’t think tasks whose primary role is to transform files should have a CopySpec as an input. The main issue being that a CopySpec can contain transform definitions (rename, filter, expand, eachFile, etc). It would then be upon each task’s implementation as to what to do with those (ignore, do copyspec transform first/last, etc).

I still believe the best thing to do is declare inputs via FileTree or FileCollection when doing the transform and use CopySpec for doing things like creating complex layout structures, filters, etc. It may be the the results of the copy task are an input to the transform task or vice versa.

1 Like