Clean way to start up / shut down some resource around test tasks

Because Elasticsearch took away our ability to use Elasticsearch in embedded mode for tests (a really convenient feature, it’s just a shame that Elastic don’t care about developer experience anymore), we have experimented with various ways to automatically start a real cluster before tests, tearing it down after tests.

Initially, we did it somewhat like this:

val startEsCluster by tasks.registering { ... }
val stopEsCluster by tasks.registering { ... }

tasks.test {
    dependsOn(startEsCluster)
    finalizedBy(stopEsCluster)
}

If multiple tests ran, though, startEsCluster only ran once, so we eventually changed this to:

fun startEsCluster() { ... }
fun stopEsCluster() { ... }

tasks.test {
    doFirst { startEsCluster() }
    doLast { stopEsCluster() }
}

The remaining issue is that two tests which do this could be run at the same time - the first test which finishes will stop the cluster and break the other tests. There’s a third integration test suite being written which will do the same, so the odds of this happening are about to increase.

So we’re looking for a cleaner way to do this.

For CI, we could potentially spin it up from Jenkins, but that doesn’t help normal devs who expect to be able to run the tests locally.

So I’m wondering if there is some way to have a shared build resource which is started on first usage and shut down only on last usage.

I considered just spinning it up once and shutting it down at JVM shutdown, but when running inside the daemon, that effectively means it never shuts down. We do want it to shut down so that people have a decent chance of being able to run the clean task.

This is vaguely similar to the use case of starting/stopping a database, which might be a fairly common use case in some applications, so I figure someone has some idea of how this could work?

Your second example also has the drawback that the cluster will never stop if any test failed unless you configure the test task to not fail on failed tests.

But you actually already said yourself what the proper way is to spin up and tear down such resources:

So I’m wondering if there is some way to have a shared build resource which is started on first usage and shut down only on last usage.

That’s exactly the way to go.
A shared build service is shut down somewhen between the last task that needs it is finished and the build is finished and it is started on first usage.

I considered just spinning it up once and shutting it down at JVM shutdown, but when running inside the daemon, that effectively means it never shuts down.

Even if the daemon would shut down or you would disable the daemon, shutdown hooks are never guaranteed to run, so that would be flaky by design too. If the process is killed for example they are not run, or if any shutdown hook needs too much time, all later shutdown hooks also do not run.

Then it would be better to do it on build finish, again using a build service, but then registered as operation completion listener and implementing AutoCloseable.
But as said above, your suggested solution is already the way to go.

So, at the time, I had the wording for what I wanted, but had no idea there was such a thing in Gradle.

So today we have been trying to use the build service approach.

So far, based on the example in the docs:

abstract class ElasticsearchCluster implements BuildService<ElasticsearchCluster.Params>, AutoCloseable {
    public ElasticsearchCluster() {
        // Start the server somehow here - problem is this requires running a task
    }

    @Override
    public void close() {
        shutdownES()
    }

    interface Params extends BuildServiceParameters {
        // nothing right now
    }
}

Then in each project which needs it…

tasks.test {
    usesService(project.extra["elasticsearchCluster"] as Provider<*>)
}

The docs tell us how to structure the service class at the outer layer, but when it comes to “how to actually run the service”, it’s a “draw the rest of the owl” situation.

To start the elasticsearch cluster, we must run at a minimum some task which builds the directory with the configured cluster in it. But build services are per-build, so I can’t put task dependencies on the service itself. And before you say “just do all the setup in the service constructor” - one of the plugins we have to install in the cluster is built by one of our own subprojects, so no matter what we do, some task needs to be run.

So now I get the choice between these alternatives:

  1. Any task planning to use the service still needs to declare a dependency on some other task which sets up the dependencies for the service. I could maybe bundle the two together into a utility method to reduce the pain.
  2. The service runs a second instance of Gradle to run the task to prepare the service. Seems fairly clean other than maybe risking some kind of deadlock.

I haven’t yet scoured GitHub to see how other people are using this feature, but I suppose I might find some examples out there. It seems like anyone who has to start a web server as part of a build is going to have to do something to prepare the web server, and in many cases that’s going to depend on some of their own code, otherwise what endpoints are they planning to talk to?

It turns out even harder than I thought to get this feature to work.

Two of the subprojects I’m using it on work fine. When I test a third subproject, the service is never instantiated.

I put println logging in, and I can see that usesService is called - the configuration block when registering the service is called - and then the constructor for the service is not called, and the tests then get started without the service having been started.

So now I am digging around Gradle internals trying to figure out why it doesn’t get instantiated. I guess I’ll have to breakpoint in there during one of the tasks which works, to figure out which Gradle class is supposed to be doing it.

I have also taken a good look around GitHub, and it looks like the vast majority of the usages out there are merely abusing the feature to do parallel task throttling and don’t even have any code in their service class. Hardly anyone is actually starting a service, and the one case I found which does was just starting postgres and not anything which would have depended on the results of the build. :frowning:

A test project I made to test the feature out in isolation exhibits the same behaviour.

At no point do I see the service being instantiated. :thinking:

Starting Gradle Daemon...
Connected to the target VM, address: '127.0.0.1:20572', transport: 'socket'
Gradle Daemon started in 2 s 99 ms
> Task :buildSrc:generateExternalPluginSpecBuilders UP-TO-DATE
> Task :buildSrc:extractPrecompiledScriptPluginPlugins UP-TO-DATE
> Task :buildSrc:compilePluginsBlocks UP-TO-DATE
> Task :buildSrc:generatePrecompiledScriptPluginAccessors UP-TO-DATE
> Task :buildSrc:generateScriptPluginAdapters UP-TO-DATE
> Task :buildSrc:compileKotlin UP-TO-DATE
> Task :buildSrc:compileJava NO-SOURCE
> Task :buildSrc:compileGroovy NO-SOURCE
> Task :buildSrc:pluginDescriptors UP-TO-DATE
> Task :buildSrc:processResources UP-TO-DATE
> Task :buildSrc:classes UP-TO-DATE
> Task :buildSrc:inspectClassesForKotlinIC UP-TO-DATE
> Task :buildSrc:jar UP-TO-DATE
> Task :buildSrc:assemble UP-TO-DATE
> Task :buildSrc:compileTestKotlin NO-SOURCE
> Task :buildSrc:pluginUnderTestMetadata UP-TO-DATE
> Task :buildSrc:compileTestJava NO-SOURCE
> Task :buildSrc:compileTestGroovy NO-SOURCE
> Task :buildSrc:processTestResources NO-SOURCE
> Task :buildSrc:testClasses UP-TO-DATE
> Task :buildSrc:test NO-SOURCE
> Task :buildSrc:validatePlugins UP-TO-DATE
> Task :buildSrc:check UP-TO-DATE
> Task :buildSrc:build UP-TO-DATE
> Configure project :project-a
*** my-service configuration block called
*** usesService called for task: task ':project-a:test'
> Task :compileKotlin NO-SOURCE
> Task :compileJava NO-SOURCE
> Task :processResources NO-SOURCE
> Task :classes UP-TO-DATE
> Task :jar UP-TO-DATE
> Task :inspectClassesForKotlinIC UP-TO-DATE
> Task :compileTestKotlin NO-SOURCE
> Task :compileTestJava NO-SOURCE
> Task :processTestResources NO-SOURCE
> Task :testClasses UP-TO-DATE
> Task :test NO-SOURCE
> Task :project-a:compileJava NO-SOURCE
> Task :project-a:processResources NO-SOURCE
> Task :project-a:classes UP-TO-DATE
Disconnected from the target VM, address: '127.0.0.1:20572', transport: 'socket'
> Task :project-a:compileTestJava UP-TO-DATE
> Task :project-a:processTestResources NO-SOURCE
> Task :project-a:testClasses UP-TO-DATE
Connected to the target VM, address: 'localhost:20599', transport: 'socket'
Disconnected from the target VM, address: 'localhost:20599', transport: 'socket'
Connected to the target VM, address: '127.0.0.1:20572', transport: 'socket'
> Task :project-a:test
*** test task being run: task ':project-a:test'
BUILD SUCCESSFUL in 14s
16 actionable tasks: 1 executed, 15 up-to-date
10:38:50: Execution finished 'test'.

Then in each project which needs it…

tasks.test {
   usesService(project.extra["elasticsearchCluster"] as Provider<*>)
}

This is not enough, to get the service started you do not only need to declare the task is using it, you also have to use it.
So add a doFirst { ... } that get()s the service from the provider and it will be started and stopped accordingly.

The docs tell us how to structure the service class at the outer layer, but when it comes to “how to actually run the service”, it’s a “draw the rest of the owl” situation.

Yeah, sure, each service is differently, the Gradle docs can hardly give any reasonable advice here.

To start the elasticsearch cluster, we must run at a minimum some task which builds the directory with the configured cluster in it.

I don’t think you can run a task. in the context of a build service.
Maybe if you use the tooling api to run the build again in parallel to execute the task in question.
But not within the same build run.

is built by one of our own subprojects

Maybe it would also somehow work when using composite build instead of subproject in that case, but I don’t know.

So now I get the choice between these alternatives:

Yeah, sound reasonable.

I put println logging in, and I can see that usesService is called - the configuration block when registering the service is called - and then the constructor for the service is not called, and the tests then get started without the service having been started.

Yeah, as described above. usesService is not enough, you have to actually use the service. :slight_smile:

are merely abusing the feature to do parallel task throttling and don’t even have any code in their service class

That’s actually not an abuse, it is also a valid use case for shared build services, to represent a shared resource that may only be used X times in parallel, including the CPU or RAM being the resource and X being 1.