Consuming artifacts from a custom task between projects

Given this example of a minimal gradle java multiproject (using gradle 6.3 and explicitly using compile instead of api/implementation):

  • build.gradle
   allprojects {
       apply plugin: 'java'
   }
  • settings.gradle
   rootProject.name = 'gradle-fruits'
   include ':apple'
   include ':banana'
  • apple/src/main/java/test/fruit/stonefruits/Apple.java
   package test.fruit.stonefruits;
   public class Apple {
       public static String NAME = "Apple";
   }
  • banana/build.gradle
   dependencies {
       compile project(':apple')
   }
  • banana/src/main/java/test/fruit/herbs/Banana.java
   package test.fruit.herbs;
   import test.fruit.stonefruits.Apple;
   public class Banana {
       public static void main(String args[]) {
           System.out.println(Apple.NAME);
       }
   }

We would like to use gradle to generate text files for each project from the compile dependencies and project specific input files (template.in). Such that:

  • apple/template/template.in
    Stonefruits

  • banana/template/template.in
    Herbs

results in:

  • apple/template/template-apple
    Stonefruits

  • banana/template/template-banana

   Stonefruits
   Herbs
  • template/template-gradle-fruits
   Stonefruits
   Herbs

where

:apple :compileTemplate reads apple/template/template.in and
writes apple/template/template-apple

:banana :compileTemplate reads banana/template/template.in
apple/template/template-apple and concatenates the files into
banana/template/template-banana

:gradle-fruits:compileTemplate reads template/template.in
apple/template/template-apple banana/template/template-banana and
concatenates the files into template/template-gradle-fruits

The task compileTemplate below is supposed to generate a
template/template-${project.name} as an artifact that may be used as
further input for dependant compileTemplate tasks, where the dependency
info comes from the compile configuration:

  • build.gradle
   void concat(File target, Iterable<File> files) {
       target.bytes = []
       files.each { f ->
           target.append(f.getBytes())
       }
   }
   
   allprojects {
       apply plugin: 'java'
   
       task compileTemplate {
           File templ_in = file "template/template.in"
           File templ_out = file "template/template-${project.name}"
           inputs.files templ_in
           outputs.file templ_out
   
           doLast {
               List<File> deps = configurations.template.incoming.dependencies.collectMany { d ->
                   d.artifacts.url
               }
               List<File> all = [ templ_in ] + deps
               println "deps: ${configurations.template.incoming.dependencies.collect { d-> d.name }}"
               println "dep artifacts: $deps"
               println "all artifacts: $all"
               concat(templ_out, all)
           }
       }
   
       task cleanTemplate {
           File templ_out = file "template/template-${project.name}"
           doLast {
               delete templ_out
           }
       }
   
       configurations {
           template {
               canBeConsumed = true
               canBeResolved = true
               extendsFrom compile
           }
       }
   
       artifacts {
           template(subprojects.compileTemplate.templ_out) {
               builtBy(compileTemplate)
           }
       }
   
       tasks.clean.dependsOn(cleanTemplate)
   }
   
   task template {
       dependsOn allprojects.compileTemplate
   }

however this results in:

   $ gradle template
   
   FAILURE: Build failed with an exception.
   
   * Where:
   Build file 'build.gradle' line: 46
   
   * What went wrong:
   A problem occurred evaluating root project 'gradle-fruits'.
   > Could not get unknown property 'templ_out' for task ':compileTemplate of type org.gradle.api.DefaultTask.

Is this the way to do this with gradle and is there any solution for this error?

Thanks for any help.

OK, so the error can be fixed with something like this:

  • build.gradle
class TemplateTask extends DefaultTask {
    @InputFiles File templ_in
    @OutputFile File templ_out
    ConfigurationContainer configurations

    void concat(File target, Iterable<File> files) {
        target.bytes = []
        files.each { f ->
            target.append(f.text)
        }
    }

    @TaskAction
    def compile() {
        List<File> deps = configurations.template.incoming.dependencies.collectMany { d -> d.artifacts.url }
        List<File> all = [ templ_in ] + deps
        println "deps: ${configurations.template.incoming.dependencies.collect { d-> d.name }}"
        println "dep artifacts: $deps"
        println "all artifacts: $all"
        concat(templ_out, all)
    }
}

and use TemplateTask like this:

allprojects {
    apply plugin: 'java'

    task compileTemplate(type:TemplateTask) {
        templ_in = file "template/template.in"
        templ_out = file "template/template-${project.name}"
        configurations = project.configurations
    }

Also remove extendsFrom compile + add explicit dependency in banana/build.gradle to make it more explicit:

    task cleanTemplate {
        File templ_out = file "template/template-${project.name}"
        doLast {
            delete templ_out
        }
    }

    configurations {
        template {
            canBeConsumed = true
            canBeResolved = true
            //extendsFrom compile
        }
    }

    artifacts {
        template(compileTemplate.templ_out) {
            builtBy(compileTemplate)
        }
    }

    tasks.clean.dependsOn(cleanTemplate)
}

task template {
    dependsOn allprojects.compileTemplate
}
  • banana/build.gradle
dependencies {
    compile project(':apple')
    template project(':apple')
}

But the artifact lists from the debug prints are empty:

$ gradle clean template

> Task :compileTemplate
deps: []
dep artifacts: []
all artifacts: [gradle-fruits/template/template.in]

> Task :apple:compileTemplate
deps: []
dep artifacts: []
all artifacts: [gradle-fruits/apple/template/template.in]

> Task :banana:compileTemplate
deps: [apple]
dep artifacts: []
all artifacts: [gradle-fruits/banana/template/template.in]

I want this instead:

> Task :apple :compileTemplate
deps: []
dep artifacts: []
all artifacts: [gradle-fruits/apple/template/template.in]

> Task :banana :compileTemplate
deps: [apple]
dep artifacts: [gradle-fruits/apple/template/template-apple]
all artifacts: [gradle-fruits/banana/template/template.in, gradle-fruits/apple/template/template-apple]

> Task :compileTemplate
deps: [apple, banana]
dep artifacts: [gradle-fruits/apple/template/template-apple, gradle-fruits/banana/template/template-banana]
all artifacts: [gradle-fruits/template/template.in, gradle-fruits/apple/template/template-apple, gradle-fruits/banana/template/template-banana]

How do I write compileTemplate to use the output from other dependant compileTemplate tasks as its input?

… and I’ve also experimented, unsuccessfully with a more explicit dependency just for the sake of seeing if it works (although had this worked it would probably not be an OK solution within our real gradle project):

$ git diff
diff --git a/banana/build.gradle b/banana/build.gradle
index e9746d4..e297fba 100644
--- a/banana/build.gradle
+++ b/banana/build.gradle
@@ -1,4 +1,4 @@
 dependencies {
     compile project(':apple')
-    template project(':apple')
+    template files(project(':apple').tasks.compileTemplate.templ_out)
 }

but it also fails:

$ gradle clean template

> Task :compileTemplate
deps: []
dep artifacts: []
all artifacts: [gradle-fruits/template/template.in]

> Task :apple :compileTemplate
deps: []
dep artifacts: []
all artifacts: [gradle-fruits/apple/template/template.in]

> Task :banana :compileTemplate FAILED

FAILURE: Build failed with an exception.

* Where:
Build file 'gradle-fruits/build.gradle' line: 15

* What went wrong:
Execution failed for task ':banana :compileTemplate'.
> Could not get unknown property 'artifacts' for object of type org.gradle.api.internal.artifacts.dependencies.DefaultSelfResolvingDependency.

(Getting closer) with this change I’m able to pipe the results between dependant tasks:

diff --git a/build.gradle b/build.gradle
index 2d33ab9..7f507fd 100644
--- a/build.gradle
+++ b/build.gradle
@@ -12,7 +12,7 @@ abstract class TemplateTask extends DefaultTask {
 
     @TaskAction
     def compile() {
-        List<File> deps = configurations.template.incoming.dependencies.collectMany { d -> d.artifacts.url }
+        Collection<Project> deps = configurations.template.incoming.dependencies.collect { d -> d.dependencyProject.tasks.compileTemplate.templ_out }
         List<File> all = [ templ_in ] + deps
         println "deps: ${configurations.template.incoming.dependencies.collect { d-> d.name }}"
         println "dep artifacts: $deps"

the build output:

$ gradle clean template

> Task :compileTemplate
deps: []
dep artifacts: []
all artifacts: [gradle-fruits/template/template.in]

> Task :apple :compileTemplate
deps: []
dep artifacts: []
all artifacts: [gradle-fruits/apple/template/template.in]

> Task :banana :compileTemplate
deps: [apple]
dep artifacts: [gradle-fruits/apple/template/template-apple]
all artifacts: [gradle-fruits/banana/template/template.in, gradle-fruits/apple/template/template-apple]

Now I only need to figure out how to reduce the results into the root project template/template-gradle-fruits:

$ cat apple/template/template-apple 
Stonefruit
$ cat banana/template/template-banana 
Herbs
Stonefruit

/Rubber Duck

sorry, apparently this does not rebuild properly if a dependant template is changed (e.g. apple/template/template.in). So I have to add an outputs.upToDateWhen { false } to the task:

diff --git a/build.gradle b/build.gradle
index 7f507fd..ad4a7d0 100644
--- a/build.gradle
+++ b/build.gradle
@@ -29,6 +29,7 @@ allprojects {
         templ_in = file "template/template.in"
         templ_out = file "template/template-${project.name}"
         configurations = project.configurations
+        outputs.upToDateWhen { false }
     }
 
     task cleanTemplate {

can this be avoided?

here is the final version which works ok for us, any feedback on a more idiomatic way of doing this or any pitfalls is highly appreciated.

abstract class TemplateTask extends DefaultTask {
    abstract @InputFiles File templ_in
    abstract @OutputFile File templ_out

    void concat(File target, Iterable<File> files) {
        target.bytes = []
        files.each { f ->
            target.append(f.text)
        }
    }

    def compile(Collection<Project> depProjects) {
        Collection<File> deps = depProjects.collect { p -> p.tasks.compileTemplate.templ_out }
        List<File> all = [ templ_in ] + deps
        println "deps: ${depProjects.collect { d-> d.name }}"
        println "dep artifacts: $deps"
        println "all artifacts: $all"
        concat(templ_out, all)
    }
}

subprojects {
    apply plugin: 'java'

    task compileTemplate(type:TemplateTask) {
        templ_in = file "template/template.in"
        templ_out = file "template/template-${project.name}"
        outputs.upToDateWhen { false }

        doLast {
            compile project.configurations.template.incoming.dependencies.collect { d -> d.dependencyProject }
        }
    }

    task cleanTemplate {
        File templ_out = file "template/template-${project.name}"
        doLast {
            delete templ_out
        }
    }

    configurations {
        template {
            canBeConsumed = true
            canBeResolved = true
            extendsFrom compile
        }
    }

    artifacts {
        template(compileTemplate.templ_out) {
            builtBy(compileTemplate)
        }
    }

    tasks.clean.dependsOn(cleanTemplate)
}

task collectTemplates(type:TemplateTask) {
    dependsOn subprojects.compileTemplate
    templ_in = file "template/template.in"
    templ_out = file "template/template-${project.name}"
    outputs.upToDateWhen { false }

    doLast {
        compile subprojects
    }
}

task template {
    dependsOn collectTemplates
}

apply plugin: 'java'

configurations {
    template {
        canBeConsumed = true
        canBeResolved = true
        extendsFrom compile
    }
}

artifacts {
    template(collectTemplates.templ_out) {
        builtBy(collectTemplates)
    }
}

thanks
/Per

I did not read all code of all posts, but as you explicitly ask for points, I had a quick look at your final version. It has many bad practices, legacy things, and similar. Just some in no specific order:

  • do not use subprojects { ... } but convention plugins
  • do not use the legacy apply but use plugins { ... }
  • use task configuration avoidance by using tasks.register and all other points documented in the according doc chapter
  • do not use File or other primitives for task properties but Property and friends, like RegularFileProperty (Lazy Configuration)
  • define all inputs, outputs, destroyables, a.s.o. of your tasks (e. g. that cleanTemplate deletes that file) Authoring Tasks
  • do not access other projects tasks directly, that is unsafe, use proper cross-project publishing instead as documented at Sharing outputs between projects
  • do not use compile configuration, it is deprecated since years and in Gradle 7 finally got removed
  • do not create legacy configurations; legacy configurations are those that can be consumed and resolved; this is actually the default value but only due to backwards compatibility; configurations should now only have one of them set to true and the other set to false or both set to false depending on intended usage
  • if you directly define the compileTemplate task as artifact you do not need an explicit builtBy as the defined task output is then considered the artifact and the task dependency is implicit; if you would have used a Property like said above it would even work if you use the property as artifact, as the implicit task dependency automatically propagates to Propertys that are declared as output of the task

I’m sure I missed some more, that was just from a quick skim-over. :slight_smile:

Thank you for your feedback :slight_smile: . We will certainly look into several of the suggestions to improve our gradle build configuration in time. I think properties + lazy configuration is part of the solution we have to use to make it work as we’d like.

But I will try to minimise the problem even further to make it clear what our main problem is.

Below is a build.gradle looking like this and all the other files is as previously described. Simply put we have a gradle java multi-project build with two projects :banana and :apple where :banana has a compile dependency on :apple. We want to create a custom gradle task that uses the compile dependencies to “compile” input files into outputfiles and pipe the output between the tasks, such that :banana :compileTemplate uses :apple :compileTemplate.outFile as its input.

abstract class TemplateTask extends DefaultTask {
    abstract @InputFiles ListProperty<RegularFile> getInFiles()
    abstract @OutputFile RegularFileProperty getOutFile()
}

subprojects {
    apply plugin: 'java'

    configurations {
        template {
            extendsFrom compile
        }
    }

    def compileTemplate = tasks.register('compileTemplate', TemplateTask)

    compileTemplate.configure {

        inFiles.add(layout.projectDirectory.file("template/template.in"))
        // wrong: this adds the output from its own task as the input
        // to itself, we want to add the output from a depending
        // compileTask as the input for the other compileTask
        // such that :banana:compileTemplate inFiles contains
        // gradle-fruits/apple/template/template-apple:
        inFiles.add(outFile)
        outFile = layout.projectDirectory.file("template/template-${project.name}")

        doLast {
            println "project=${project}\n\tinFiles=${inFiles.get()}\n\toutFile=${outFile.get()}"
        }
    }
}

task template {
    dependsOn subprojects.compileTemplate
}

We want this output:

$ gradle template
project=project ':apple'
    inFiles=[gradle-fruits/apple/template/template.in]
    outFile=gradle-fruits/apple/template/template-apple
project=project ':banana'
    inFiles=[gradle-fruits/banana/template/template.in, gradle-fruits/apple/template/template-apple]
    outFile=gradle-fruits/banana/template/template-banana

If you want to use the output of a task in apple within banana, you should follow this chapter and use attribute-aware resolution mechanism: Sharing outputs between projects