Gradle Daemon in CI


(Sebastian Peters) #1

According to this article, enabling the daemon in a CI environment (such as Jenkins, I presume) isn’t recommended. There is no explanation why, however, and I don’t seem to find any. I’m mostly curious why I shouldn’t use it all the time, especially now that it’s enabled by default in 3.0+.

On that behalf:

[…] Gradle will not emit the message if the CI environment variable is present.

I cannot seem to find any reference what variable that is and where/how to set it.


(Stefan Oehme) #2

I think this advise is much less relevant now than it was back when that chapter was written. The daemon is rock solid by now and could speed up your CI builds if you reuse the same machines. @eriwen do you think this chapter should be updated now that the daemon is on by default?


(Eric Wendelin) #3

In containerized environments (like Travis CI), the choice is obvious to not use the Daemon because presumably the Daemon will never be used more than once.

We’ve worked hard to ensure parity between --no-daemon and --daemon builds, so you should feel confident that you can use the Daemon on CI to get a significant performance boost.

That said, some engineering shops strongly prefer that builds are 100% isolated (that is, no cached state can affect build result, including speed). You have to answer for yourself if you have this opinion.

Regarding the “CI environment variable”, it is meant literally: export CI=true will disable the Daemon. We kept this “feature” for backward compatibility for those who use it (hint: use --no-daemon please).

Cheers,
Eric


(Danny Althoff) #4

Maybe the daemon is, but not all used gradle-plugins or other stuff that happens while gradle is working.
While developing some workaround for the javafx-gradle-plugin cause of a nasty bug inside the JDK, I needed to “mess” with the classloader of the daemon, as you can’t stop me from doing such things (muhaha ;D), I don’t think that this should be removed from the documentation.
Especially when some libs are used which do not expect the same JVM to be running on multiple executions, this might lead to more problems and false-positives on CI-environment.

Following scenarios are coming up my mind which could lead to problems:

  • bugs inside other libs/plugins
  • unclosed/dangling file-handler inside the JVM (annoying on windows-systems!)
  • messed classloaders on multiple executions.

Greetings,
Danny


(Stefan Oehme) #5

The solution to these problems is not “Don’t use the daemon”, but “Run these broken tools in a forked VM instead of inside the daemon”.


(Danny Althoff) #6

I don’t understand, when grade-daemon is used, and the plugin is plugged into that daemon-classloader, how does this help to use a forked JVM? Who has to fork this and how?


(Stefan Oehme) #7

The plugin should not depend directly on the tool and execute it in-process. It should resolve the tool at runtime and fork it instead. This minimizes classpath pollution, memory leaks etc. It also allows to decouple the plugin version from the version of the tool being used.

See this task (and its plugin) which executes the jOOQ code generator for example.


(Danny Althoff) #8

I’m not talking about external tools, I’m talking about including externals libs as part of the plugin.

These were just my few cents about “can we remove that line inside the documentation”.


(Stefan Oehme) #9

Your example was the JavaFX plugin and the problem was that JavaFX tooling wasn’t working quite right. This can be avoided by running that tooling in a forked process instead of invoking JavaFX code in the daemon itself.


(Danny Althoff) #10

Yes, I was talking about the JavaFX-Gradle-plugin, but it is just one example of plugin, which requires 100% isolation between builds. You just can’t know, what companies are using for their internal projects, thats why that “CI should not use daemons” should stay.

To repeat a bit: you just don’t know what the plugins are doing, especially plugins written for gradle 2.x which did not have daemons per default.

It is just impossible to “execute” some library, because not everything is accessible that way, so “forking some process instead of using daemon-thread” does just not work. Compared with OSGi it would be possible to have that level of isolation between builds, but as this is not part of Gradle, you just cant bet for sure that users will use gradle as expected :wink: