Compile avoidance in custom tasks

Help/Discuss
1

Hi there. I am in the process of migrating a large, multi-project build to Gradle. This build includes a compiler (C to JVM classfiles) module as well as modules that are compiled by this compiler via a JavaExec task.

In the various tasks, we use the TaskInputs and TaskOutputs to insure that Gradle can properly check whether a compilation task needs to be run again.

It looks something like this:

configurations {
    tools
}

dependencies {
    tools project("tools:compiler")
    compile project("libraryA")
}

sourceSets {
    main {
        output.dir("$buildDir/c", builtBy: 'compileC')
    }
}

task compileC(type: JavaExec) {
    classpath configurations.tools
    classpath configurations.compile

    main = 'org.renjin.MyCompiler'

    inputs.dir 'src/main/c'
    outputs.dir "$buildDir/c'
}

This works pretty well, but the compileC task still runs a bit too often: if I make any change to libraryA, a Java project, even a minor change to a method body, then compileC is marked as out of date. In principle, compileC only needs to run again if any method signatures in libraryA change.

I read the blog post on Incremental Compilation for Java with great interest here:

Is there any way to apply this kind of logic to my compileC task? To be clear, the compileC task should be considered up to date if:

  1. the source dir ‘src/main/c’ is unchanged
  2. the output of project(“tools:compiler”) is completely unchanged
  3. the ABI of Java project(“libraryA”) is unchanged

Thanks for any pointers.

2

Hi Alex,

I think what you want is possible. First I would suggest you define a custom task class for compileC instead of using JavaExec. You can still call project.exec in your task action.

Then you can declare the different inputs and outputs via annotations with new properties. IIUC, then the task class would look something like this:

class CompileC extends DefaultTask {
    @Classpath
    final ConfigurableFileCollection tools = project.objects.files()

    @CompileClasspath
    final ConfigurableFileCollection dependentProject = project.objects.files()

    @PathSensitive(PathSensitivity.RELATIVE)
    @InputDirectory
    final DirectoryProperty sourceDirectory = project.objects.directoryProperty()

    @OutputDirectory
    final DirectoryProperty destinationDir = project.objects.directoryProperty()

    @TaskAction
    void compile() {
        project.javaexec { ... }
    }
}
1 Like
3

Thank you! I think @CompileClasspath is exactly what I’m looking for.

Is it possible to achieve the same thing by declaring the task inline in build.gradle, or do I need to move my task to `buildSrc/’ ?

4

It is. You can use inputs.file(...).withNormalizer(CompileClasspathNormalizer). I would strongly suggest you introduce your custom task type though. You can add the class directly to build.gradle, no need to move it to buildSrc.

Is there any reason why you don’t want to use a custom task class?

5

Thanks! That’s perfect.

Is it possible to share custom tasks between modules in a project without putting them in buildSrc?

These tasks concern building and bootstrapping the compiler itself, so I feel like this kind of one-off logic belongs in the build script. The next step will be to create a proper gradle plugin for building R packages for Renjin, but right now these are just a bunch of tasks executing Main classes in other modules. That’s thinking at least!

6

It seems like they are not one-off logic if you want to share them between projects. You can do that, though it is much better to add the types to buildSrc. Regarding plugins: did you know about precompiled script plugins? That might be better than keeping the ad-hoc configuration in the root build file.

7

Alright you’ve convinced me. The build logic started out as fragments, but they have grown quite complex. I will move them to buildSrc.

But now my next question is whether I can reuse these tasks outside of the project? That is, I would also like to create a Gradle plugin that is useable outside of the project. Can that plugin depend on buildSrc?

For example:

+ Root project
  - buildSrc: includes CompileTask.groovy which invokes compiler via javaExec
  - compiler
  - libraryA: uses CompileTask to compile sources
  - libraryB: uses CompileTask to compile sources
  - plugin: gradle plugin that allows other projects to use CompileTask

Maybe just brute force copy the output from the buildSrc into the main.output of plugin?

8

This is the great thing about gradle build logic. Here’s a pretty normal lifecycle.

  1. Logic starts off as a snippet inside build.gradle
  2. Things start to get a bit hairy so you create some custom tasks under buildSrc
  3. It gets even more hairy so you bundle the tasks with an extension object inside a plugin (still under buildSrc)
  4. The plugin is quite useful so you extract it to a separate project, publish it to nexus and use it as a dependency in multiple projects.

For step 4 you can build and publish the plugin project similar to normal java libraries. To include your plugin you’ll need to specify your repository in the buildscript {...} section of the project which depends on the plugin. Note: it’s not a requirement to implement a plugin, it’s perfectly valid to have a task or two without a plugin

9

Ok, I’m almost there, but getting the evaluation order with these task/plugin classes is a nightmare.

I need this task to also depend on the compile configuration, ideally using the @CompileClasspath annotation.

I have declared a field like this:

@CompileClasspath
final ConfigurableFileCollection compileClasspath = project.objects.fileCollection()

But now how and when do I initialize this to the value of configurations.compile ??

I have tried this:

CompileGimpleTask() {
    compileClasspath.setFrom(project.configurations.compile)
}

But that seems to runs before the dependencies { } block is evaluated, so compileClasspath is empty.

I have also tried:

   def compileGimpleTask = project.tasks.register('compileGimple', CompileGimpleTask)
    compileGimpleTask.configure {
        compileClasspath.from(project.configurations.compile)
    }

But that’s also too early.