Test cases with GradleRunner reusing workspace?

I’m implementing a Gradle Plugin that executes a 3rd party application. In my functional tests, I decided to simply mock that application to obtain its expected output for further procession (parser etc).
But I ran into a trouble that makes me question my own understanding of the GradleRunner.

I managed to get the minimum reproducible setup for my issue below (the duplicate code is deliberate).
To reproduce it, just create a new Gradle plugin from init and paste this:

class ReproCommandTestIssuePluginFunctionalTest {
    @TempDir
    File projectDir;

    @Test void canRunTask() throws IOException {
        System.out.println("\n\n##### RUNNING: " + new Object(){}.getClass().getEnclosingMethod().getName());
        writeString(new File(projectDir, "settings.gradle"), "");
        writeString(new File(projectDir, "build.gradle"),
            "task myToolRun {\n" +
            "   doLast {\n" +
            "       exec {\n" +
            "           commandLine 'printenv'\n" +
            "       }\n" +
            "       exec {\n" + 
            "           commandLine 'whereis', 'myTool'\n" +
            "       }\n" +
            "       exec {\n" +
            "           commandLine 'myTool'\n" +
            "       }\n" +
            "   }\n" +
            "}\n");

        // Create Tools folder
        File toolsFolder = new File(projectDir, "tools");
        Files.createDirectories(toolsFolder.toPath());
        // Create Tool
        File mockTool = new File(toolsFolder, "myTool");
        writeString(mockTool, "echo Hello World!");
        mockTool.setExecutable(true);

        // Run the build
        GradleRunner runner = GradleRunner.create();
        runner.forwardOutput();
        runner.withPluginClasspath();
        runner.withEnvironment(Map.of("PATH", toolsFolder.getAbsolutePath() + System.getProperty("path.separator") + "/usr/bin"));
        runner.withArguments("myToolRun");
        runner.withProjectDir(projectDir);
        BuildResult result = runner.build();

        // Verify the result
        assertTrue(result.getOutput().contains("Hello World!"));
    }

    @Test void canRunTask2() throws IOException {
        System.out.println("\n\n##### RUNNING: " + new Object(){}.getClass().getEnclosingMethod().getName());
        writeString(new File(projectDir, "settings.gradle"), "");
        writeString(new File(projectDir, "build.gradle"),
            "task myToolRun {\n" +
            "   doLast {\n" +
            "       exec {\n" +
            "           commandLine 'printenv'\n" +
            "       }\n" +
            "       exec {\n" + 
            "           commandLine 'whereis', 'myTool'\n" +
            "       }\n" +
            "       exec {\n" +
            "           commandLine 'myTool'\n" +
            "       }\n" +
            "   }\n" +
            "}\n");

        // Create Tools folder
        File toolsFolder = new File(projectDir, "tools");
        Files.createDirectories(toolsFolder.toPath());
        // Create Tool
        File mockTool = new File(toolsFolder, "myTool");
        writeString(mockTool, "echo Hello World!");
        mockTool.setExecutable(true);

        // Run the build
        GradleRunner runner = GradleRunner.create();
        runner.forwardOutput();
        runner.withPluginClasspath();
        runner.withEnvironment(Map.of("PATH", toolsFolder.getAbsolutePath() + System.getProperty("path.separator") + "/usr/bin"));
        runner.withArguments("myToolRun");
        runner.withProjectDir(projectDir);
        BuildResult result = runner.build();

        // Verify the result
        assertTrue(result.getOutput().contains("Hello World!"));
    }

    private void writeString(File file, String string) throws IOException {
        try (Writer writer = new FileWriter(file)) {
            writer.write(string);
        }
    }
}

Both methods canRunTask() and canRunTask2() are identical.
They create a simple build.gradle with a printenv and a whereis myTool just for debug, and at last the commandLine that executes myTool.
Next, it creates a dummy tools\myTool under the projectDir.

Finally, I’ve setup a withEnvironment(...) that overwrites the PATH.

Here’s the output of this test:

##### RUNNING: canRunTask

> Task :myToolRun
PATH=/tmp/junit3891473089882920638/tools:/usr/bin
myTool: /tmp/junit3891473089882920638/tools/myTool
Hello World!

BUILD SUCCESSFUL in 4s
1 actionable task: 1 executed


##### RUNNING: canRunTask2

> Task :myToolRun FAILED
PATH=/tmp/junit17090999217294325759/tools:/usr/bin
myTool: /tmp/junit17090999217294325759/tools/myTool

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':myToolRun'.
> A problem occurred starting process 'command 'myTool''

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 208ms
1 actionable task: 1 executed

Notice that each test got its own projectDir as expected, and even the whereis myTool gave the correct location in each respective test. But the execution only works for the first test executed, while all the remaining ones fail as if the myTool couldn’t be found (in my actual project I have tens of tests).
This also happens across test classes.

What’s more, if I annotate the projectDir with @TempDir(cleanup = CleanupMode.NEVER) then all tests succeed! It’s as if all subsequent tests were looking at the executable from the first test.


My expectation was that each GradleRunner.create().build() would be isolated, but it seems I’m wrong.
Why is that? Any idea how to solve this issue? Thanks!

Try doing a hash -r inbetween.
At least on plain bash usage for example if you change the PATH or just the files found in the PATH I often see it remember the old tool unless I use hash -r.
Maybe that helps in your case too.

1 Like

It didn’t help, but for a strange reason.
I tried to put hash -r in the test’s build.gradle itself, (after all, I’m running all tests in sequence and I needed to inject the hash in between, somewhere, somehow)

        writeString(new File(projectDir, "build.gradle"),
            "task myToolRun {\n" +
            "   doLast {\n" +
            "       exec {\n" +
            "           commandLine 'printenv'\n" +
            "       }\n" +
            "       exec {\n" + 
            "           commandLine 'whereis', 'myTool'\n" +
            "       }\n" +
            "       exec {\n" +
            "           commandLine 'hash', '-r'\n" +
            "       }\n" +
            "       exec {\n" +
            "           commandLine 'myTool'\n" +
            "       }\n" +
            "   }\n" +
            "}\n");

Interesting is that Gradle didn’t find it, even though I do have the hash command in my environment:

Caused by: java.io.IOException: Cannot run program "hash" (in directory "/tmp/junit14447412925337895586"): error=2, No such file or directory
	at net.rubygrapefruit.platform.internal.DefaultProcessLauncher.start(DefaultProcessLauncher.java:25)

Btw, I did that after setting runner.withEnvironment(null); to make that the test environment inherits my own path (which printenv confirmed also). So, right now I have no idea why hash couldn’t be found.


I’ll probably end up creating the mock application at test time under a ‘static’ folder within /tmp and move on with that. There is no harm, I think. I just hoped to not leave traces behind.

Ah, hash is not an actual executable, it is a bash built-in command.

Gradle daemons are reused if possible.
There are only a handfull of system properties or JVM properties that are treated as unmodifiable and cause a new daemon to be spawned.
Obviously the PATH environment variable is not one of those, because if you output the process ID in both tests, you see that both tests use the same Gradle daemon.

The actual command invoking is done by a plain old Java ProcessBuilder.
This ultimately ends up in native code that does the actual command invocation.
I didn’t read the native code, but I guess that there is a similar lookup cache like in bash for which hash -r helps.

So you might actually have found a Gradle bug here that you should report as issue ticket.
It seems that a different PATH environment variable should maybe also be treated as unmodifiable and cause a new daemon to be started eventually. You should imho definitely report it if it is not reporte yet.

As a work-around you could for example use a “static” mock like suggested, or you could simply not set PATH but instead inject the full path to the generated mock tool into the exec so that it is called by absolute path which might be the cleaner solution eventually.

There’s an open issue for this: Pass-through Environment Variables to Gradle Daemon Processes · Issue #10483 · gradle/gradle · GitHub

1 Like