Replacing deprecated `Project#exec` in `doFirst`/`doLast`

Help/Discuss
1

I’ve got a custom Gradle task like this:

tasks.register("myTask") {
  doLast {
    exec { commandLine("sh", "-c", "myscript.sh") }
    exec { commandLine("sh", "-c", "myscript2.sh") }
  }
}

When I run it, I get this warning:

The Project.exec(Action) method has been deprecated. This is scheduled to be removed in Gradle 9.0. Use ExecOperations.exec(Action) or ProviderFactory.exec(Action) instead. Consult the upgrading guide for further information: Upgrading your build from Gradle 8.x to the latest

The guide there is super unhelpful.

At execution time, for example in @TaskAction or doFirst/doLast callbacks, the use of Project instance is not allowed when the configuration cache is enabled. To run external processes, tasks should use an injected ExecOperation service, which has the same API and can act as a drop-in replacement. The standard Java/Groovy/Kotlin process APIs, like java.lang.ProcessBuilder can be used as well.

I have no idea what any of that means.

The ExecOperation page gives this, uh, rather verbose code example.

With some ceremony, it is possible to use ExecOperations in an ad-hoc task defined in a build script:

interface InjectedExecOps {
    @get:Inject val execOps: ExecOperations
}

tasks.register("myAdHocExecOperationsTask") {
    val injected = project.objects.newInstance<InjectedExecOps>()

    doLast {
        injected.execOps.exec {
            commandLine("ls", "-la")
        }
    }
}

Is that what the Gradle team recommends that I do to run a quickie one-line shell script?

It seems like not, because right underneath the code sample, there’s a “tip.”

:light_bulb: Tip

This is a good time to consider extracting the ad-hoc task into a proper class.

Would using a “proper class” be better, somehow? The example in the docs looks like this:

abstract class MyExecOperationsTask
@Inject constructor(private var execOperations: ExecOperations) : DefaultTask() {

    @TaskAction
    fun doTaskAction() {
        execOperations.exec {
            commandLine("ls", "-la")
        }
    }
}

tasks.register("myInjectedExecOperationsTask", MyExecOperationsTask::class) {}

It seems like Gradle is saying that it was wrong of me to write exec { commandLine("myscript.sh") }, but that instead I ought to have defined a “proper class” to do it instead.

But, on the other hand, the docs seem to be claiming that the new thing is a “drop-in replacement.” This is not a drop-in replacement; this is replacing a one-liner with an injected AbstractSingletonProxyFactoryBean.

Is there a drop-in replacement available to me? Do I have replace all of my exec calls with these weird injections?

2

Is that what the Gradle team recommends that I do to run a quickie one-line shell script?

I doubt that they do.
To execute a one-line shell script, use a task of type Exec instead:

tasks.register<Exec>("myTask") {
    commandLine("sh", "-c", "myscript.sh")
}

But you are not doing “a quickie one-line shell script”, but you do “complex” logic, doing two external invocations.

Would using a “proper class” be better, somehow?

If you write it inline inside the build script, no.
But what it meant to tell is, to have the task class in an included “build-logic” build or in buildSrc.
There custom task classes are more maintainable and also testable by unit and integration tests latest when they become more complex.

Writing a task class inline in the build script is seldomly helpful and only uglifies things and makes them less idiomatic.

But, on the other hand, the docs seem to be claiming that the new thing is a “drop-in replacement.” This is not a drop-in replacement

It is a drop-in replacement.
ExecOperations#exec is a drop-in replacement for Project#exec.
They have the same API and the same behavior.
How you get hold of an instance of ExecOperations or Project is a totally separate topic.

Is there a drop-in replacement available to me? Do I have replace all of my exec calls with these weird injections?

You have to replace it by something, with that is up to you.
Options you have include:

  • Using task of type Exec for a single call, that’s the cleanest anyway in that situation
  • Getting hold of an ExecOperations somehow and using exec on that (for example by using an interface and objects.newInstance or a dedicated task class
  • Using standard Java or Kotlin or Groovy ways to execute something like for example ProcessBuilder.
  • There is in the meantime also providers.exec wich you can use, but beware that this is not a drop-in replacement. The confguration has the same API, but the whole thing is lazy, intended to only be executed if necessary and as soon as necessary. So doing providers.exec { ... } will not actually execut anything yet, you either have to get stdout, stderr, or the result from it like for example providers.exec { ... }.result.get()
1 Like
3

Thanks, I’ve learned a lot here. I’ve filed an issue that this should be easier:

4

I absolutely do not wonder that LLMs cannot fix it for you, because LLMs are not AIs even if they are called like it these days. They are mainly next-word-guessers and I’m yet to see a “decent LLM” that can produce correct code out-of-the box. They are nice to help coding faster in small scale but they very seldomly produce production-ready or even compilable code.

Btw., why do you not like providers.exec “which seems right at first but is subtly the wrong thing to use at execution time”?
It’s perfectly fine to use it at execution time, you just have to remember that it is lazy and not actually executed if you don’t trigger it.

1 Like
5

Maybe I’m misreading the documentation here (it’s very confusing), but my understanding is that providers.exec isn’t designed to run during the execution phase (though you can use it (abuse it?) to do that); it’s designed to run during the configuration phase, allowing you to invalidate the cached configuration model (the task graph). https://docs.gradle.org/current/kotlin-dsl/gradle/org.gradle.api.provider/-provider-factory/exec.html

Allows lazy access to the output of the external process.

When the process output is read at configuration time it is considered as an input to the configuration model. Consequent builds will re-execute the process to obtain the output and check if the cached model is still up-to-date.

The process input and output streams cannot be configured.

I don’t want to invalidate the configuration cache. I just want to run a process at execution time.

Another issue with providers.exec is that stdout and stderr aren’t auto-piped to the console and “cannot be configured.”

tasks.register("myTask") {
    doLast {
        exec {
            commandLine("sh", "-c", "echo OK myTask")
        }
    }
}

tasks.register("myTask2") {
    doLast {
        providers.exec {
            commandLine("sh", "-c", "echo OK myTask2")
        }.result.get()
    }
}

myTask logs OK on the Gradle console; myTask2 logs nothing. To get the stdout and stderr off of providers.exec, I had to write it like this:

tasks.register("myTask2ManualOutput") {
    doLast {
        val execOutput = providers.exec {
            commandLine("sh", "-c", "echo OK myTask2; echo ERR myTask2 >&2; exit 1")
            isIgnoreExitValue = true
        }
        
        val exitCode = execOutput.result.get().exitValue
        val stdout = execOutput.standardOutput.asText.get()
        val stderr = execOutput.standardError.asText.get()
        
        println("STDOUT: $stdout")
        println("STDERR: $stderr")
        println("Exit Code: $exitCode")

        if (exitCode != 0) {
            throw GradleException("Command failed with exit code $exitCode")
        }
    }
}

The docs said with some embarrassment that using ExecOperations.exec requires “some ceremony”, but this is way more ceremonious. It’s not even close to a drop-in replacement for project.exec. (But then, it was never really supposed to be.)

BTW, you know who generated this sample? An LLM! I didn’t know the difference between an ExecResult and an ExecOutput. (And why would I? None of this is documented in any user guide or tutorial.) The LLM generated it in one shot.

Before this week, I’d never heard of a Gradle “provider” or a Gradle “service injection” in Gradle scripts. Gradle tutorials don’t cover this material, because ordinary users of Gradle (even developers of basic custom tasks) didn’t normally need to learn such things.

https://docs.gradle.org/current/kotlin-dsl/gradle/org.gradle.api.provider/-provider-factory/exec.html doesn’t even say how to use it. I only learned that I could call providers.exec when an LLM told me about it.

That’s why I filed https://github.com/gradle/gradle/issues/34483. I had to read the entire Gradle book to even understand how to use ExecOperations.exec and providers.exec. I’m still not 100% sure I understand them well enough to use them correctly.

And I’m not the only one struggling!

I strongly agree with @johanE’s take on the situation. Gradle never required me to learn this much about the internals of Gradle (providers, service injections, ProcessOutputValueSource, ExecOutput vs ExecResult).

I’m sure it all made sense at the time, but executing a process in Gradle now requires an AbstractSingletonProxyFactoryBean, the sort of architecture for architecture’s sake that people have been teasing us JVM folks about for decades.

Having said all that, the configuration cache seems like a good, important thing to have, and I do see why having build scripts monkeying around with the Project at execution time would make optimizations impossible.

What I’m asking for in my filed issue is a one-liner transformation from project.exec to another thing. In #34483 I suggest execOps.exec instead, and that execOps would be automatically available in scope at execution time, or, barring that, having a one-liner at the top of my build script that will give me an execOps instance that I can use at execution time.

The closest thing we have right now is to add this at the top of my build.gradle.kts file:

interface InjectedExecOps {
    @get:Inject val execOps: ExecOperations
}

val execOps = project.objects.newInstance<InjectedExecOps>().execOps

And then I can replace project.exec (or just plain-old exec) with execOps.exec and get the exact behavior I want.

1 Like
6

I’ve written up an answer for this on StackOverflow in the hopes that it eases someone’s pain. This is the information I would have wanted in Gradle’s own documentation.

1 Like
7

Maybe I’m misreading the documentation here (it’s very confusing), but my understanding is that providers.exec isn’t designed to run during the execution phase (though you can use it (abuse it?) to do that); it’s designed to run during the configuration phase, allowing you to invalidate the cached configuration model (the task graph).

That’s a slight misinterpretation on your side, yes.

If you try to run an external process by not explicitly CC-supported mechanisms with configuration cache enabled during configuration phase, you get a CC problem and CC cannot be used.
This includes things like project.exec, ExecOperations#exec or ProcessBuilder.

If you use one of the means that is explicitly supported by CC, this is not the case. These are for example using a ValueSource in which you can do anything including using ExecOperations#exec or ProcessBuilder. For a ValueSource that is queried during configuration phase, it goes like this:

  • if a CC entry is about to be reused, all the configuration-phase queried ValueSources are executed
  • if all their result is the same as in the CC entry, the CC entry is reused
  • if one of the ValueSources produces a different result, the CC entry is discarded and a new CC entry is built, including executing the ValueSources a second time

A providers.exec that you get at configuration time is a simplified way to define a ValueSource that does exactly execute one external process with fixed arguments, where stdout, stderr, and the result code are the outputs of the ValueSource. It behaves exactly like a ValueSource described above, so if a CC entry is about to be reused, the command is executed and if stdout, stderr, and result code are the same, the CC entry is reused, otherwise it is built freshly including executing the command a second time.

For all ValueSources, including the ones built using providers.exec, if you only get them at execution time which is perfectly valid, they do not cause the CC entry to be discarded, they are also not executed twice or at all if the execution time getting does not happen because it is guarded by some if or similar.

If you for example do something like this:

abstract class MyTask : DefaultTask() {
    @get:Internal
    abstract val input: Property<String>

    @TaskAction
    fun execute() {
        println("Hello from ${this::class.simpleName} with input ${input.get()}")
    }
}

val foo by tasks.registering(MyTask::class) {
    input = providers.exec { commandLine("bash", "-c", "date") }.standardOutput.asText
}

This only executes at execution time and only if that line is reached, and that is perfectly fine and as intended.

If you want to execute some task as part of a task implementation and you have a proper task class, you could also do it like

abstract class MyTask : DefaultTask() {
    @get:Inject
    abstract val providerFactory: ProviderFactory

    @TaskAction
    fun execute() {
        println("Hello from ${this::class.simpleName}: ${providerFactory.exec { commandLine("bash", "-c", "date") }.standardOutput.asText.get()}")
    }
}

val foo by tasks.registering(MyTask::class)

Especially if you want to get the output of the command.

But if you just want to execute something and don’t want to further process the output but only have it displayed as part of the Gradle output, it just does not make too much sense and is more succinct to there use ExecOperations like:

abstract class MyTask : DefaultTask() {
    @get:Inject
    abstract val execOperations: ExecOperations

    @TaskAction
    fun execute() {
        execOperations.exec { commandLine("bash", "-c", "date") }
    }
}

val foo by tasks.registering(MyTask::class)

So it is more a question of the exact use-case what you need and thus what to use,
but all are fine to be used at execution time.

And if you do not have a full task class but just an ad-hoc doLast task,
you can of course get the ExecOperations via some ad-hoc producer interface,
but it is just more convenient and succinct to use providers.exec there.

You just have to remember that it is lazy and not executed if you don’t trigger it,
and that you have to log the output yourself if you want to have it displayed as part of the build output.

What I’m asking for in my filed issue is a one-liner transformation from project.exec to another thing. In #34483 I suggest execOps.exec instead, and that execOps would be automatically available in scope at execution time, or, barring that, having a one-liner at the top of my build script that will give me an execOps instance that I can use at execution time.

I did not question your request to make ExecOperations more easily usable in ad-hoc tasks, that I let the Gradle folks think about. I just questioned your statement that providers.exec is not suitable or not designed for execution phase usage, which is just plainly wrong. :slight_smile:

1 Like
8

I think the problem here is that for projects that suddenly break when Gradle 9 disappears the project.exec function, a reasonable conclusion that a person might reach from searching these forums would be to just substitute it with providers.exec. And since the API is the same, the script will compile with no warnings and then … just not work.

Yes, you can eagerly call .standardOutput.asText.get() and .standardError.asText.get() and manually write them to the console and call .result.get().exitValue and manually fail the task when the process has a non-zero exit code, but this isn’t anywhere close to how the project.exec function used to behave.

Further research then leads to a learning/lecturing journey about laziness and the configuration cache and all the wonderful things that it enables. But for someone who just wants to run a process the same way as they have been for years, it is an unwelcome surprise to learn that what you actually need to do is to create a new type with an injected “ExecOperations” and then instantiate it and then get the instance’s execOps and then run exec on that instance, and then things will be back to normal.

For someone not fluent in the API and nuances of Gradle, this is an unwelcome side-quest that ought to have been unnecessary. exec { } could just have done that for you, the same way it always did.

1 Like
9

I think the problem here is that for projects that suddenly break when Gradle 9 disappears the project.exec function

That’s why you should for a smooth experience always:

  • Update to latest patch version within same major version
  • Update all plugins to latest possible version
  • Fix all deprecation warnings
  • Update to latest patch version in the directly following major version
  • Go to second step and repeat until reaching the target version

Gradle is pretty strict with their backwards compatibility so breaking changes are usually only done on major version bump and after a deprecation cycle.

a reasonable conclusion that a person might reach from searching these forums would be to just substitute it with providers.exec .

Well, I don’t think so, and noone ever said that would be a drop-in replacement.
And latest when they test changes they do (hopefully they do test) they should recognize that something is not working as expected. :man_shrugging:

That was also not the point.
The point was, that providers.exec is perfectly fine for usage at execution time and also designed for that use-case, despite the statement that it is not which is plainly wrong.

Btw. “And since the API is the same” is also not correct, you can for example not set stdout or stdin like you can with Project#exec or ExecOperations.Exec, and also the return value is different. The APIs are merely similar, but by far not identical, not only behaviour-wise.

Yes, you can eagerly call […] but this isn’t anywhere close to how the project.exec function used to behave.

And noone said that it is.
Using providers.exec is one way that can be used as replacement, not necessarily the way.
It depends on the use-case and intentions.
If you want a drop-in replacement that behaves the same, then somehow get hold of an ExecOperations instance and use that, because it has the same API and behaves the same.
For example when you want to get the output of the call, it is most often much easier to replace

val stdout = ByteArrayOutputStream()
exec {
    executable = "foo"
    standardOutput = stdout
}
val output = stdout.toString(Charsets.UTF_8)

by

val output2 = providers.exec {
    executable = "foo"
}.standardOutput.asText.get()

But for someone who just wants to run a process the same way as they have been for years, it is an unwelcome surprise to learn that what you actually need to do is to create a new type with an injected “ExecOperations” and then instantiate it and then get the instance’s execOps and then run exec on that instance, and then things will be back to normal.

Again, I’m not questioning that request.
I just questioned the non-sense statement that providers.exec is not suitable for execution phase usage which is just plainly wrong and that’s a fact.

Also, if you are not afraid of using internal API and are on Kotlin DSL, you can just use serviceOf<ExecOperations>().exec { ... } which I also wished were part of the public API a couple of times already.

exec { } could just have done that for you, the same way it always did.

Not really, because you actually call project.exec there even if it is hidden, and you must no longer access project at execution phase. So to keep it, it would have been necessary to add exec to the Task API, so that you can use it from any task and there it has absolutely nothing to do imho. So while it is more effort for the users, it results in cleaner API and better future for us all, instead of polluting the Task API with totally unrelated methods. :slight_smile:

10

FYI, the Gradle team has decided that It's too verbose to `ExecOperations.exec` in `doFirst`/`doLast` · Issue #34483 · gradle/gradle · GitHub is a duplicate of Reduce ceremony to use common public services in Gradle scripts · Issue #13121 · gradle/gradle · GitHub which has been open for five years and will probably never be fixed.

I’ll just copy and paste my comment here for posterity. The Gradle dev team has never posted anything in response to my feedback, so I’m cross-posting it in the hopes that maybe, just maybe, someone will read it and listen.

My issue #34483 was just marked as a duplicate of this very old issue; I hope that means that the dev team is at least considering addressing this issue here.

IMO this issue got a lot more urgent in Gradle 9, because now even doLast { exec } requires excessive ceremony. But, it’s not just that running exec and fs.delete in doLast requires too many unnecessary lines of code, but the lines of code are very confusing to Gradle newbies.

If you read the Gradle tutorial from start to finish, you will never have heard of services or service injection. Services and Service Injection

That Service Injection documentation page appears only in the “Gradle Types” section of "Authoring Builds,’ after the “Advanced Tutorial,” and, honestly, that’s where it belongs. Nobody who just wants to write a doLast { exec } task should need to learn about service injections.

But, since Gradle 9 removed Project#exec (for good performance reasons!) it’s now no longer possible to write doLast { exec } without understanding a lot of Gradle’s internals.

Service injections are hard to use in Gradle partly because there are so many different ways to do it. Constructor injection, property injection. You can do it in an abstract task class, or you can define an interface and use an ObjectFactory, which you can access from project.objects, but only at configuration time, not at execution time.

Newbies don’t even know the difference between configuration time and execution time. They don’t know what a “service” is, or what “injection” is, or how to choose which way to inject a service. They don’t know what Gradle’s ObjectFactory is, or why they would want to call ObjectFactory#newInstance. They don’t know that you can get an ObjectFactory off of project.objects, and they don’t understand that you must do that at configuration time, inside the registration block, but not at the top level of your build.gradle.kts file.

In order to understand this code:

interface InjectedExecOps {
    @get:Inject val execOps: ExecOperations
}

tasks.register("myAdHocExecOperationsTask") {
    val injected = project.objects.newInstance<InjectedExecOps>()

    doLast {
        injected.execOps.exec {
            commandLine("ls", "-la")
        }
    }
}

You have to read and understand Services and Service Injection fully and thoroughly, and also Build Lifecycle (in the “Beyond the Basics” section), so you know what the configuration phase is. You’ll probably need to read Configuration Cache in order to understand why you have to put val injected = ... inside the register section but not at the top level of build.gradle.kts.

Once you’ve read and digested all of that material, you can finally write a basic doLast { exec }, something that was absolutely trivial in Gradle 8.

The fact that it’s four lines of code isn’t that important. The fact that it’s four lines of code that are utterly incomprehensible to newbies is the fundamental problem.

And did I mention that those lines of code aren’t mentioned at all in Upgrading to Gradle 9.0.0 ? It does link to Upgrading within Gradle 8.x which links to Services and Service Injection but, the documentation and code there is completely incomprehensible out of context.

1 Like
11

Actually, I think your “newbie” argument is not that relevant.

A newbie will simply not try to use doLast { exec } as they are not aware of it, but use a task of type Exec which for simple cases is preferable anyway. And if they want to do multiple commands within one task action or more complex logic, then it simply is not a newbie topic anymore. :man_shrugging:

12

I disagree strongly with this. Any newbie will be looking at the last decade of documented examples on how to do this and wonder why they don’t work.

1 Like
13

Well, following bad and / or outdated examples can always happen and is always bad. :man_shrugging: