How to create a S3 File Collection


I’ve been working on extending and adding to one of the AWS plugins that allows S3 downloads/uploads. Most of the plugins in the repository that deal with AWS S3 do so by adding Tasks that specialize in handling S3 content. However, I think that there is a better way to integrate S3 into gradle than at the task level. FileCollection and FileTree are very handy interfaces for working with collections of files and zipTree() vs fileTree() are interchangeable for most tasks. That’s why if you want to unzip a zip file CopyTask can handle it because the extraction process is already handled in the zipTree.

This feels like the right place to integrate S3 and other remote file based technologies (NFS, Samba, etc). However, FileCollection assumes you are dealing with a object always, and that is a bit cumbersome since with remote file systems you don’t have a physical file on the disk yet in all situations like the local versions of that interface do.

Has there been any discussion or thought to adding some indirection to these interfaces to allow for the concrete collections to implement a common interface possibly? That way each concrete implementation could provide it’s own implementation of a RemoteFile or LocalFile so they could be treated uniformly.

I have another plugin that uses S3 tasks to download/upload collections of files, and once I started going down this path if I could treat S3 as just another FileCollection then using something like S3FileCollection would dramatically simplify my plugin design while making it more powerful since I don’t assume S3 is always the source or destination of things.

Is there another interface that I could use to create a similar place for integration?