Unzip files that are part of a configuration

[mwkohout (Michael Kohout)]

Hi-

I’m trying to write a custom (java) plugin that will extract files for processing by other tasks that are coming from a custom configuration. In this case, they are filled with avro protocol files.
from build.gradle:

dependencies {
    implementation "org.apache.avro:avro:1.9.2"
    avroDefinitions group: 'com.bigco.project', name: 'schemas', version: '1.4', ext: 'zip'
    avroDefinitions group: 'com.bigco.project2', name: 'moreSchemas', version: '1.4', ext: 'zip'
}

I’m having trouble figuring out how to extract these files using the Copy task:
from custom java plugin:

 public class SchemaDirectorPlugin implements Plugin<Project> { 
    public void apply(Project project) {
         Configuration avroDefinitions = project.getConfigurations().create("avroDefinitions");
         var unzipAvro = project.getTasks().register("unzipAvro");

         avroDefinitions.getAll().forEach(configuration -> {
             //todo:name it better
             var c = project.getTasks().register("unzipAvro" + UUID.randomUUID().toString(), Copy.class, copy -
                     copy.doLast(t -> {
                         copy.from((project.zipTree(configuration.getFiles())));
                         copy.into(new File(project.getBuildDir(), extension.getIprExtractDir().get()));
                     }));
             unzipAvro.get().getDependsOn().add(c.get());
             project.getDefaultTasks().add(c.getName());
         });
 ....

when I run the task to extract the files for this definition, the copy tasks don’t seem to see the zipfiles:

Task :unzipAvrobe1ff033-e6c2-4d09-b517-bbdb0685b05e NO-SOURCE
Task :unzipAvrobfcb9b75-eafe-4cce-9f06-4097f50a059a NO-SOURCE
Task :unzipAvroc0924af2-b008-4e29-9982-d25412952741 NO-SOURCE
Task :unzipAvrodf08bbf7-b610-41f4-8571-21ce32a1d3be NO-SOURCE
Task :unzipAvrof2fa5ec5-e079-4e28-bca0-29868ea12d76 NO-SOURCE
Task :unzipAvrofeff4bd7-2c64-485c-913e-a258f931dd7c NO-SOURCE

and don’t extract anything.

Is there something obvious I’m missing?

thanks

[Chris_Dore (Chris Doré)]

@mwkohout

         Configuration avroDefinitions = project.getConfigurations().create("avroDefinitions");
         var unzipAvro = project.getTasks().register("unzipAvro");

(1)      avroDefinitions.getAll().forEach(configuration -> {
             //todo:name it better
             var c = project.getTasks().register("unzipAvro" + UUID.randomUUID().toString(), Copy.class, copy -
(2)                  copy.doLast(t -> {
                         copy.from((project.zipTree(configuration.getFiles())));
(3)                      copy.into(new File(project.getBuildDir(), extension.getIprExtractDir().get()));
                     }));
(4)          unzipAvro.get().getDependsOn().add(c.get());
(5)          project.getDefaultTasks().add(c.getName());
         });

1. Calling Configuration.getAll() is probably not what you want since it will return all of the configurations in the project.
2. The Action passed to doLast is executed during the Gradle task execution phase. Calling from and into needs to happen during Gradle’s configuration phase (sometime also called evaluation). Some more info on the build lifecycle can be viewed here. This is root problem causing the behaviour you’re seeing because the tasks are not being configured with anything to copy and therefore never execute because there’s “NO-SOURCE”.
3. You do not want to call get() on providers during Gradle’s configuration phase. The intent of the property/provider model is to defer accessing the true values until the last possible moment (during task execution). Some more info here.
4. Calling TaskProvider.get() negates the use of TaskContainer.register() because it causes the tasks to be prematurely realized. Here’s some more info on the task configuration avoidance API.
5. My personal opinion; plugins should not manipulate the default tasks of a project. That should be a project specific concern left up to the build script.

An example of how you could re-work things:

     Configuration avroDefinitions = project.getConfigurations().create("avroDefinitions");
     var unzipAvro = project.getTasks().register("unzipAvro");

     var c = project.getTasks().register("unzipAvroDefinitions", Copy.class, copy -> {
         copy.from(project.getProviders().provider(        // (1)
             () -> avroDefinitions.getFiles().stream()
                 .map( zip -> project.zipTree(zip) )
                 .collect(Collectors.toSet)));
         copy.into(project.layout.getBuildDirectory().dir(extension.getIprExtractDir()));
     });
     unzipAvro.configure(t -> t.dependsOn(c));

1. The provider is used to defer the resolution of the avroDefinitions configuration so that it doesn’t happen during Gradle configuration phase (sorry, the term “configuration” is overloaded in Gradle).

NOTE; I wrote the code in the forum editor, so I’m sure there’s some errors in it

[mwkohout (Michael Kohout)]

I didn’t see your really helpful suggestion until now;

I ended up solving what I needed by pretty much doing what you suggested:

        project.getTasks().create("unzipAvro", Copy.class, unzip->{
        unzip.from(avroDefinitions.getFiles()
                .stream()
                .map(project::zipTree)
                .collect(Collectors.toList()));
        unzip.eachFile(cfd -> {
            cfd.setRelativePath(new RelativePath(true, cfd.getRelativePath().getLastName()));
            Integer.toString(1);
        });
        unzip.setIncludeEmptyDirs(false);
        Provider<Directory> output = project.getLayout().getBuildDirectory().dir(extension.getIprExtractDir().forUseAtConfigurationTime());
        unzip.into(output);
    });

It’s working out really well. Thank you!