Build lifecylce question / task ordering with org.gradle.workers.max > 1

[tkrah (Torsten K.)]

Hi,

coming from non parallel task execution, before configuration cache was a thing, I got some tasks which need some external stuff running (which I do spawn and shutdown via dependsOn + finalizers), that worked fine so far and it does so with org.gradle.workers.max=1, which may be a lucky coincidence though.

Trying to make use of the parallel nature for the rest of my build, I thought using a shared build service with maximum parallel usage would be sufficient for those who need it which does work - however, my build prerequisites and finalizers are executed in a different order now, which is obviously fatal for the build itself, because that external stuff does need that order (compose container setup with conflicting services and ports which I stubbed out for that example).

As an example GitHub - tkrah/spring-sec-14911 at buildservice · GitHub you can run:

./gradlew integrationTest

With 1 worker it will have that order:

> Task :composeUp
composeUp

> Task :docker1:runTest
runTest

> Task :composeDown
composeDown

> Task :e2eComposeUp
e2eComposeUp

> Task :docker2:e2eRunTest
e2eRunTest

> Task :e2eComposeDown
e2eComposeDown

With more than one worker however (change the worker property in the project prop file) the order does change to:

> Task :composeUp
composeUp

> Task :e2eComposeUp
e2eComposeUp

> Task :docker1:runTest
runTest

> Task :docker2:e2eRunTest
e2eRunTest

> Task :composeDown
composeDown

> Task :e2eComposeDown
e2eComposeDown

I am wrapping my head around the lifecycle, if I use the –task-graph argument I get:

> ./gradlew integrationTest --task-graph
Reusing configuration cache.
Task graph printing is an incubating feature.
Tasks graph for: integrationTest
+--- :docker1:integrationTest (org.gradle.api.DefaultTask)
|    +--- :composeUp (org.gradle.api.tasks.Exec)
|    |    \--- :composeDown (org.gradle.api.tasks.Exec, finalizer)
|    \--- :docker1:runTest (org.gradle.api.tasks.Exec)
\--- :docker2:integrationTest (org.gradle.api.DefaultTask)
     +--- :e2eComposeUp (org.gradle.api.tasks.Exec)
     |    \--- :e2eComposeDown (org.gradle.api.tasks.Exec, finalizer)
     \--- :docker2:e2eRunTest (org.gradle.api.tasks.Exec)

So far so good but even with a build service with max usage = 1 that will not run in my desired order anymore.

So - how would I solve that problem so that I can make sure

a) some tasks are not run in parallel at all and

b) the pre-requisites are only run just before the actual task which needs it and finalizers are executed immediately and not deferred?

From an isolated view I can see why Gradle does that because it does not know that those tasks may conflict externally and just spawn those one after another although serially - while they do not run in parallel anymore, they still will fail because of external conflicts (although I am wondering why I get my expected order with workers = 1).

Suggestions how to fix the current structure and / or rewrite it with some other feature of Gradle to get that running with workers > 1?

Thanks.

[Vampire (Björn Kautler)]

(which I do spawn and shutdown via dependsOn + finalizers)

Not the best idea.
That is one of the main use-cases of shared build services.
For example, if the runTest task will be up-to-date or coming from cache and thus not being executed, spinning up and tearing down will still be done for no reason while with a shared build service properly done, it will only invest the time when actually necessary.

however, my build prerequisites and finalizers are executed in a different order now, which is obviously fatal for the build itself

If it worked before, that was just a coincidence.
If you do not define an order (e.g. by dependsOn, mustRunAfter, shouldRunAfter, finalizedBy, implicit task dependencies by properly wiring task outputs to task inputs, …) then there is no guaranteed order.
If it works with 1 max worker (or any other special circumstances), that is just sheer luck and could change with any patch update of Gradle.
If you need a certain order between two tasks, you have to ensure that order by configuring it one way or another.

some tasks are not run in parallel at all and

build service with max usage 1 is the correct way

the pre-requisites are only run just before the actual task which needs it and finalizers are executed immediately and not deferred?

With tasks?
You cannot.
Not without for example configuring that e2eComposeUp mustRunAfter composeDown, so that if these two tasks will run in the same build run, the up must run after the down and the other task dependencies will then result in your intended order.

But even then, it is not really a clean way to set it up, and there is and was also no guarantee whatsoever, that a task is run right before its dependent task, or that a finalizer task is run directly after the task it finalizes. It is only guaranteed, that (only using the first triple) runTest runs only after composeUp is finished at some point in time, composeDown runs only after runTest is finished at some point in time, and that composeUp and composeDown are added to the task graph automatically if runTest is part of the task graph (unless excluded like with -x).

although I am wondering why I get my expected order with workers = 1

Not guaranteed coincidence that could fail on you any time.

Suggestions how to fix the current structure and / or rewrite it with some other feature of Gradle to get that running with workers > 1?

Use a shared build service to spin up and tear down the external dependency, that’s one of the main use-cases of a shared build service, and will also have the benefit of not doing unnecessary work.

Actually, the tear down via AutoCloseable will also not be exactly what you want and need, because it runs “somewhen between the last task needing the service finished and the end of the build” which usually means it is run after all tasks finished (but that’s also just implementation detail which could change).

But you could for example make your build service an operation completion listener and do the tear down after you were notified that the task needing the service finished, that way your teardown will be done immediately after the task and also is the only way to do it I’m aware of, besides adding the logic to the task actions of the task in question directly instead of having separate tasks.

[tkrah (Torsten K.)]

I thought you would say something like that

But using a shared buildservice to spin up those containers would not be possible, at least I don’t see how that would work.

Each of that composeUp tasks of every subproject can configure the environment needed to spin up those containers (env var substitution, different compose file, different services to start) - I don’t know how I would transport that to the build service.

With the listener I could tear it down, but how would I spin it up again after it was brought down - there is no “task submitted” callback, right?

Receiving information about task execution - Using Shared Build Services

There is only a finishedEvent to tearDown my resources used - but where is the “task submitted” event to start the resources again once brought down?

Additionally reading the docs it has:

Currently, build services are scoped to a build, rather than a project, and these services are available to be shared by the tasks of all projects.

My resource is scoped to a (sub)project - each project has its own resource stack I need to start / stop (which I do with tasks at the moment) - a build service which has a build scope does not fit to that requirement, does it?

So if i can’t force dependsOn / finalizers to run in time - I could try switching the thing to be a

doFirst() / doLast()

of my test task itself - that way I can be sure it runs first and is shutdown at the task itself, right - although if it fails doLast() won’t run … see below, that’s why I choose finalizers?

And more of a general question - what is the Gradle doc approach here to handle such scenarios where you would need to spin up / tear down resources (in time) to run your integration tests?

Looking at the docs e.g. here:

Finalizer tasks - Controlling Task Execution

It has - quote:

Finalizer tasks are useful when the build creates a resource that must be cleaned up, regardless of whether the build fails or succeeds. An example of such a resource is a web container that is started before an integration test task and must be shut down, even if some tests fail.

So that’s why I choose tasks / finalizers tasks in the first place, my compose stack is such a web container, database etc. for my test task - your quote “Not the best idea.” makes me wonder why not, because the docs suggest such a thing - it even has “is started before an integration task” - so how would it be started when not with a task itself (so it can be finalized).

You’re telling my that it is not possible to force gradles lifecycle here in time - so to me that doc suggestion is imho wrong, because you can’t tear down things at the time it is needed (and I can’t let the run to the end of the build, the RAM / memory would be exhausted as soon as I start a second stack in some environments) - shared build services don’t fit in the picture because I can’t configure them adhoc like I need to - so how would you handle that with Gradle?

[Vampire (Björn Kautler)]

Each of that composeUp tasks of every subproject

Well, what you showed was one compose up task in root project for a test task in a subproject.
But anyway,

But using a shared buildservice to spin up those containers would not be possible, at least I don’t see how that would work.

Why?
Just do it.
Like you have tasks in every subproject, you can have a shared build service for each of those configurations.
It does not have to be the same service with which you ensure tasks are not running in parallel.
You can either have different services for the different configurations, or just different instances of the service which you give different parameters.

With the listener I could tear it down, but how would I spin it up again after it was brought down

As I said, one service or one service instance per spin-up tear-down, that you can for example give via parameters how to spin it up and also as parameter the task after which it should be teared down.

Or instead of doing the spin-up in the service constructor, you can also instead have a method which you give the parameters for the spin-up and do it with one service instance, then you can also use that for the max-use and can even have a boolean where you track whether the tear-down was already done, in case you forget to add a task after which the teardown should be done. The method you would then call in a doFirst action of the relevant task.

there is no “task submitted” callback, right?

That would not make any sense at all.
Tasks are not submitted, the task graph is calculated at the end of the configuration phase and not changeable afterwards, besides that it would nt help as you want to do the spin up before the task is executed, not when it is “submitted”, and also only if the task is actually really executed and not for example up-to-date or coming from cache.

of my test task itself - that way I can be sure it runs first and is shutdown at the task itself, right - although if it fails doLast() won’t run … see below, that’s why I choose finalizers?

Yes, that’s why I recommended the service and listener approach instead.
You actually can do it with doFirst / doLast alone, by getting the actions of the task, clearing the actions, and executing them in a new doLast action with a try-finally block to do the teardown in the finally.
But this is slightly fragile, as you might miss actions added to the task after you stored and cleared the action.

With the build service and listener approach you still do the spin-up in the doLast of the task as you there either realize the service so that the constructor runs or call the spin-up method.

While thinking about it, you would anyway not do the spin-up in the constructor of the service, because by registering the service as listener you already realize it and you don’t want to do the spin-up there. So you would anyway have a separate spin-up method no matter whether you use one service instance or one per configuration which you then call in a doFirst action to do the spin up and have the tear-down via the listener right after the task finished no matter with which status.

And more of a general question - what is the Gradle doc approach here to handle such scenarios where you would need to spin up / tear down resources (in time) to run your integration tests?

No idea whether they have a concrete example for that, but a shared build service is the intended approach.

It has - quote:

That largely pre-dates the existence of shared build services, so is more the legacy approach.
But you can of course still do it, as I said, you just have to ensure the ordering of those spin-up and tear-down tasks if they otherwise conflict without correct order.

You’re telling my that it is not possible to force gradles lifecycle here in time - so to me that doc suggestion is imho wrong, because you can’t tear down things at the time it is neede

It is not wrong, it is perfectly fine per-se.
And if you declare the order in which the tasks must run it also works as expected.

It is just a big waste of time as you spin-up and tear-down when you don’t actually need it, as the tests are not actually executed.

shared build services don’t fit in the picture because I can’t configure them adhoc like I need to

Sure you can, in various ways as just described.

[tkrah (Torsten K.)]

@Vampire

Each of that composeUp tasks of every subproject

Well, what you showed was one compose up task in root project for a test task in a subproject.
But anyway,

Of cause I trimmed it down for the example itself, but I’ll try to enrich the example a little bit again if new question come up, which I am sure they do

The whole thing has many subprojects, where each of that project does run those tasks and configure them with their own environment.

@Vampire

But using a shared buildservice to spin up those containers would not be possible, at least I don’t see how that would work.

Why?
Just do it.
Like you have tasks in every subproject, you can have a shared build service for each of those configurations.

That’s going to be many build services than but okay … but if I have different build services, I can’t tell them that they have to be only one running in parallel at all … if I define a second one I can tell that one it has to be a maximumUsage of 1, but that does not effect the other services - so the stuff is still running in parallel now which I don’t want - or how can I tell a group of services that the max usage must be one for all of them at once?

@Vampire

It does not have to be the same service with which you ensure tasks are not running in parallel.
You can either have different services for the different configurations, or just different instances of the service which you give different parameters.

With the listener I could tear it down, but how would I spin it up again after it was brought down

As I said, one service or one service instance per spin-up tear-down, that you can for example give via parameters how to spin it up and also as parameter the task after which it should be teared down.

Or instead of doing the spin-up in the service constructor, you can also instead have a method which you give the parameters for the spin-up and do it with one service instance, then you can also use that for the max-use and can even have a boolean where you track whether the tear-down was already done, in case you forget to add a task after which the teardown should be done. The method you would then call in a doFirst action of the relevant task.

Hm - the build services examples in the docs are not really helpful about such a usage / pattern to be honest - I try to come up with something in the demo, but I am sure I will ask for help again here while coming up with a solution for what you suggest.

@Vampire

there is no “task submitted” callback, right?

That would not make any sense at all.
Tasks are not submitted, the task graph is calculated at the end of the configuration phase and not changeable afterwards, besides that it would nt help as you want to do the spin up before the task is executed, not when it is “submitted”, and also only if the task is actually really executed and not for example up-to-date or coming from cache.

“submitted” here was of cause meant to be executed - I am interested in the execution of the task like the completion one, so yeah we can agree to that here, meant exactly that - but there is no such execution callback.

@Vampire

of my test task itself - that way I can be sure it runs first and is shutdown at the task itself, right - although if it fails doLast() won’t run … see below, that’s why I choose finalizers?

Yes, that’s why I recommended the service and listener approach instead.
You actually can do it with doFirst / doLast alone, by getting the actions of the task, clearing the actions, and executing them in a new doLast action with a try-finally block to do the teardown in the finally.
But this is slightly fragile, as you might miss actions added to the task after you stored and cleared the action.

Nah - I am here asking for help because I want to find something non fragile which is somehow Gradle approved I would rather not do that but find a better solution like it is done now.

@Vampire

With the build service and listener approach you still do the spin-up in the doLast of the task as you there either realize the service so that the constructor runs or call the spin-up method.

While thinking about it, you would anyway not do the spin-up in the constructor of the service, because by registering the service as listener you already realize it and you don’t want to do the spin-up there. So you would anyway have a separate spin-up method no matter whether you use one service instance or one per configuration which you then call in a doFirst action to do the spin up and have the tear-down via the listener right after the task finished no matter with which status.

I try to understand those suggestion and come up with something (the Gradle docs could really need some best practice example about this - I don’t think my usecase is such uncommon to have some resources spin up + down for an integration test).

@Vampire

And more of a general question - what is the Gradle doc approach here to handle such scenarios where you would need to spin up / tear down resources (in time) to run your integration tests?

No idea whether they have a concrete example for that, but a shared build service is the intended approach.

Fair enough, I’ll give it a try.

@Vampire

It has - quote:

That largely pre-dates the existence of shared build services, so is more the legacy approach.
But you can of course still do it, as I said, you just have to ensure the ordering of those spin-up and tear-down tasks if they otherwise conflict without correct order.

Hm that’s like the "“when” documentation about parallel usage and configuration cache stuff - I love to trip about such pre-dates docs and legacy things although that setup is so old, I don’t know if shared build services where a thing at that time.

Anyway, I am here to learn

@Vampire

You’re telling my that it is not possible to force gradles lifecycle here in time - so to me that doc suggestion is imho wrong, because you can’t tear down things at the time it is neede

It is not wrong, it is perfectly fine per-se.
And if you declare the order in which the tasks must run it also works as expected.

But that won’t work - I can’t declare the order of those tasks (only in one project) but not across all subprojects which I would need to, because with project isolation to become a thing soon that will not help - so I think better not try that at all - I’ll take that service approach.

@Vampire

It is just a big waste of time as you spin-up and tear-down when you don’t actually need it, as the tests are not actually executed.

I know I waste some time here if it is not needed but for the moment I would rather take that hit if it would work at all, but as I does not work anyway with workers > 1 at the moment I still need to figure out how to drive those resources and the lifecycle of Gradle here.

@Vampire

shared build services don’t fit in the picture because I can’t configure them adhoc like I need to

Sure you can, in various ways as just described.

Hm - which various ones - basically only buildServices are viable, right, after looking at the other suggestions and considering the impact above?

And I am still not sure how I can make different build Services e.g. one for {db, web} => A and another one with {db} => B tell, not to run in parallel at all - can I?

Or I would need to have one build service with different spinup / shutdown methods (or methods with a parameter) where I tell the service to run A or B (where the spinup would be made in a doFirst { } call and the tear down in the completion listener callback) - that should work though, if I did get your message right, right?

Btw, your answers / help / input is very much appreciated - so please bear with me if I fail to understand the message or need more time than you may expect here, I can assure you that’s not on purpose.

PS: I am going to update the demo repo (may take some time to learn and process the input) and will ask for feedback if I hit a obstacle on the way .

[Vampire (Björn Kautler)]

but if I have different build services, I can’t tell them that they have to be only one running in parallel at all […] or how can I tell a group of services that the max usage must be one for all of them at once?

As I said, you can either have the multiple services for spin-up and tear-down, and one additional one for doing the max-usage guard, or you can have one service where you give the configuration via method parameters which you need anyway to not spin-up through adding as listener.

which is somehow Gradle approved

I’m not Gradle, I’m just a user like you, this is a user community.

But that won’t work

It “can” work even with IP, but it will probably fast become ugly, like publishing some pseudo-variant of the projects that has a task dependency on the to-be-depended-on task and then in the other project have a dependency on that variant that you make an input for the to-depend-on task. But you probably don’t want to go that route either.

shared build services don’t fit in the picture because I can’t configure them adhoc like I need to

Sure you can, in various ways as just described.

Hm - which various ones - basically only buildServices are viable, right, after looking at the other suggestions and considering the impact above?

The question here was about being able to configure the build service, not about build service alternatives.
You can use build service parameters and multiple instances of the build service, or you can use setters on the build service to configure the parameters and call them before the spin-up method call, or you can give the parameters to the spin-up method call as parameters, …

Or I would need to have one build service with different spinup / shutdown methods (or methods with a parameter) where I tell the service to run A or B (where the spinup would be made in a doFirst { } call and the tear down in the completion listener callback) - that should work though, if I did get your message right, right?

That was one of the probably most viable options, yes, now you might have gotten it.

Btw, your answers / help / input is very much appreciated - so please bear with me if I fail to understand the message or need more time than you may expect here, I can assure you that’s not on purpose.

Don’t worry, I’m just allergic against asking the same question multiple times, ignoring the answer I already gave, not about asking for more details / help to understand. Giving answers is also always a gamble between how detailed do you answer fitting the knowledge of the other person and boring them with stuff they already know .

I am going to update the demo repo

Sure, didn’t actually look at it yet actually, your description seemed clear enough to me for answering so far, but if there are more fine-grained problems / questions, it might come in very handy

[tkrah (Torsten K.)]

@Vampire

I’m not Gradle, I’m just a user like you, this is a user community.

I know, but lets say, I appreciate your input and consider your opinion / answers on Gradle very seriously

Back to topic - I updated the repository trying to implement your suggestion and I am facing a problem.

It does start my backend command:

> Task :docker1:integrationTest
Starting up A with env: {PORT=5434}
Compose Output: 5434

I implemented Using Shared Build Services , but the listener callback is never called - if you run:

./gradlew integrationTest

You will get:

> Task :docker1:integrationTest
Starting up A with env: {PORT=5434}
Compose Output: 5434


> Task :docker2:integrationTest FAILED
TearDown A after null?.displayName

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':docker2:integrationTest' (registered in build file 'docker2/build.gradle.kts').
> tearDown not called!

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights from a Build Scan (powered by Develocity).
> Get more help at https://help.gradle.org.

BUILD FAILED in 3s
7 actionable tasks: 7 executed

The AutoClosable kicks in as can be seen on the output, but the listener does not although the task did clearly run - ideas?

Is usesService(taskSemaphore)not sufficient here?

[tkrah (Torsten K.)]

I found in a Gradle bug ticket this internal API thing:

gradle.taskGraph.whenReady {
    gradle.serviceOf<BuildEventsListenerRegistry>().onTaskCompletion(taskSemaphore)
}

If I add that to both subprojects it seems to work BUT this is not documented on the buildServices usage page / docs at all.

Output:

> Task :buildSrc:jar
:jar: No valid plugin descriptors were found in META-INF/gradle-plugins
Handle event: Task :docker1:compileJava skipped
Handle event: Task :docker1:processResources skipped
Handle event: Task :docker1:classes UP-TO-DATE
Handle event: Task :docker1:jar SUCCESS
Handle event: Task :docker1:compileIntegrationTestJava SUCCESS
Handle event: Task :docker1:processIntegrationTestResources skipped
Handle event: Task :docker1:integrationTestClasses SUCCESS

> Task :docker1:integrationTest
Starting up A with env: {PORT=5434}
Compose Output: 5434

Handle event: Task :docker1:integrationTest SUCCESS
TearDown A after Task :docker1:integrationTest SUCCESS?.displayName

> Task :docker2:integrationTest
Starting up B with env: {PORT=5435}
Compose Output: 5435

B
Handle event: Task :docker2:integrationTest SUCCESS
TearDown B after Task :docker2:integrationTest SUCCESS?.displayName

BUILD SUCCESSFUL in 2s
7 actionable tasks: 7 executed

But now it seems all tasks are handled through my build service somehow as you can see on the buildSrc:jar part, which I do not want actually - feels like I am doing something wrong here

[Vampire (Björn Kautler)]

I implemented Using Shared Build Services , but the listener callback is never called - if you run:

Of course not, if you don’t register the listener as shown in the page you linked in the last example, it does not somehow magically self-register anywhere.

If I add that to both subprojects it seems to work BUT this is not documented on the buildServices usage page / docs at all.

Of course, why should the doc document using internal API and bad practice?
And the correct way is documented on the page you linked in the last example.
It does neither make sense to use the internal serviceOf, nor doing the registration in taskGraph.whenReady.
Just inject the BuildEventsListenerRegistry service into your plugin like with any other Gradle service and then use it to register your listener after creating it.

But now it seems all tasks are handled through my build service somehow as you can see on the buildSrc:jar part, which I do not want actually - feels like I am doing something wrong here

Of course you get all event, you registered it as listener.
You cannot pre-select which events to react to.
Like with most listeners in most event-based systems, your listener has to filter which events it is interested in.
That’s why I above also said you give the task path to the service so that you can tear down after the appropriate event.

[tkrah (Torsten K.)]

@Vampire

Of course, why should the doc document using internal API and bad practice?
And the correct way is documented on the page you linked in the last example.
It does neither make sense to use the internal serviceOf, nor doing the registration in taskGraph.whenReady.
Just inject the BuildEventsListenerRegistry service into your plugin like with any other Gradle service and then use it to register your listener after creating it.

My bad, you’re right - should have read that last paragraph too.

@Vampire

But now it seems all tasks are handled through my build service somehow as you can see on the buildSrc:jar part, which I do not want actually - feels like I am doing something wrong here

Of course you get all event, you registered it as listener.
You cannot pre-select which events to react to.
Like with most listeners in most event-based systems, your listener has to filter which events it is interested in.
That’s why I above also said you give the task path to the service so that you can tear down after the appropriate event.

Ah I thought that while using useService(..)it would register that for exactly that task only - but fair enough, I can filter that on my own (still need to figure that out, but I am sure I can do that ).

But I got another problem which I can solve, I put the registration to a Plugin like seen in Using Shared Build Services in the TaskEventPlugin.

However I am fail to acquire a Provider<..> of that in my build.gradle.kts file.

If I try

interface Injected {
    @get:ServiceReference("taskSemaphore")
    val taskSemaphore: Property<TaskSemaphore>
}
val injected = project.objects.newInstance<Injected>()

like seen in the link above where it is a Property, Gradle complains about:

   > class org.gradle.api.internal.provider.DefaultProperty cannot be cast to class org.gradle.api.services.internal.RegisteredBuildServiceProvider (org.gradle.api.internal.provider.DefaultProperty and org.gradle.api.services.internal.RegisteredBuildServiceProvider are in unnamed module of loader org.gradle.internal.classloader.VisitableURLClassLoader @5010be6)

And it would not help my because the usesService(..) code needs a Provider<BuildService> as an argument.

So I am stuck here - how would I inject / acquire the Provider<BuildService> I registered via a Plugin in a build.gradle.kts file to be used with usesService(..) ?

I tried to inject a Provider<..> but this will end in:

Could not create an instance of type Build_gradle$Injected.
> Could not generate a decorated class for type Build_gradle.Injected.
   > Cannot have abstract method Injected.getTaskSemaphore(): Provider<TaskSemaphore>

and I found ServiceReference should support Provider · Issue #33291 · gradle/gradle · GitHub - so this is not supported yet, right - but there must be a way to get a reference to that, right?

[Vampire (Björn Kautler)]

Ah I thought that while using useService(..) it would register that for exactly that task only

No, that just declares that the task uses the service, independent from any listening.
If you inject the service to a custom task using @SerivceReference that usesService call is done automatically.
The usesService is for example necessary that the AutoCloseable part is not invoked when the service is still needed, and so that the max parallel usage works like intended.

like seen in the link above where it is a Property, Gradle complains about

As far as I remember, you can only use @ServiceReference for a task property.
To get the provider for the service, just do the registerIfAbsent call where you need the service and you get the already registered service provider that you can also use for the usesService call.

If you need or want to avoid doing registerIfAbsent multiple times, register an extension on Project in your plugin that does the registration and provide the service provider from a property of that extension.

[tkrah (Torsten K.)]

@Vampire

As far as I remember, you can only use @ServiceReference for a task property.
To get the provider for the service, just do the registerIfAbsent call where you need the service and you get the already registered service provider that you can also use for the usesService call.

Yeah had that, but wanted to avoid that multiple calls.

@Vampire

If you need or want to avoid doing registerIfAbsent multiple times, register an extension on Project in your plugin that does the registration and provide the service provider from a property of that extension.

I used a

 val taskSemaphore: Property<Provider<TaskSemaphore>> =
            objects.property(Provider::class.java) as Property<Provider<TaskSemaphore>>

to transport my provider (works) - can this be simplified somehow to not use the Property?

And another question about the task path.

The reference I can take from the Event via:

event.descriptor.taskPath

On the task itself where I would need to provide it - do have to do that in a way that I know the path would be e.g. “:docker1:integrationTest” or is there a method on the task - this - to build that, is so any hints appreciated?

doFirst {
            ts.get().startUp("A", mapOf("PORT" to "5434"), ":docker1:integrationTest")
        }

[Vampire (Björn Kautler)]

to transport my provider (works) - can this be simplified somehow to not use the Property?

A Property only makes sense if it should be configurable, which does not make any sense here.
Just expose it as property in the extension directly.

or is there a method on the task - this - to build that, is so any hints appreciated?

… well … getPath()?

[tkrah (Torsten K.)]

@Vampire

to transport my provider (works) - can this be simplified somehow to not use the Property?

A Property only makes sense if it should be configurable, which does not make any sense here.
Just expose it as property in the extension directly.

Hm - I should not use a Property but expose it as a property - now I am lost, do you have a simple example of which property do you speak of?

The example Part 2: Add an Extension in the docs does only have Property ones and I only used those yet, or do you mean I should put it as a member of the class just like:

@Inject
    constructor(
        objects: ObjectFactory,
        val taskSemaphore: Provider<TaskSemaphore>,
    )

@Vampire

or is there a method on the task - this - to build that, is so any hints appreciated?

… well … getPath()?

Hm … I guess it was late or I was blind, that works well. Although it is annotated as @Internal - objections on that?

[tkrah (Torsten K.)]

And while for small tasks this works - I found that the event emitted when the task is finished, is emitted / received later than the incoming start task of the next execution - async delivery which obviously is a problem here.

I implemented a waiting primitive in my build service to “sleep” until that other event is delivered and processed hopefully in time (with some timeout).

Seems like a lot of work and things to consider for something as common like, I need to start / stop external resource(s) for my test(s).

[Vampire (Björn Kautler)]

or do you mean I should put it as a member of the class just like

Exactly.
If it is not configurable, it does not make much sense to have it be a Property.

Although it is annotated as @Internal - objections on that?

That does not mean Gradle-internal don’t use it.
It is part of the public API.
That annotation is part of the input/output annotations which each and every task property has to carry and defines that it is a task-internal value that is not part of the formal task input or outputs which are used for up-to-date checks or cache key fingerprinting.

async delivery which obviously is a problem here.

Ah, was not aware the listener is called async.
But even then, this is not too much of a problem if you have one service you use for all starting and stopping.
Just have a Semaphore with one permit in the service that you acquire in the beginning of spin-up and release as last action in tear-down, then the spin-up for the next task waits for the tear-down.

for something as common like, I need to start / stop external resource(s) for my test(s).

Something that you think is common, but is not that common, considering all Gradle builds out there.

[tkrah (Torsten K.)]

@Vampire

Ah, was not aware the listener is called async.
But even then, this is not too much of a problem if you have one service you use for all starting and stopping.
Just have a Semaphore with one permit in the service that you acquire in the beginning of spin-up and release as last action in tear-down, then the spin-up for the next task waits for the tear-down.

Ack, I used a semaphore to manage that - already thinking about an extension to have that service manage different sets of compose stacks to start / tear down in parallel (as long as the service key used is different) - that way I could run different stacks in parallel.

Subprojects which request the same key are still serialized on the semaphore tracked for that key.

@Vampire

for something as common like, I need to start / stop external resource(s) for my test(s).

Something that you think is common, but is not that common, considering all Gradle builds out there.

Fair enough - that was just my assumption here. Lets say, it would be nice if the Gradle samples or best practices docs chapter would have some guidance / help about such a thing.

Nevertheless, thanks to your valuable input and feedback I got it running with parallel builds and workers > 1, thank you very much.

Edit: Got it working - having 4 workers with 3 subprojects, where subproject 2 and 3 will have the same service key Gradle will run subproject 1 + 2 in parallel while 3 is waiting for 2 to finish and will start after that one.

And the best thing is - it will not start / teardown any build service if it can take results from the cache - nice (I know you said that, just mentioning it).

[Vampire (Björn Kautler)]

@tkrah

already thinking about an extension to have that service manage different sets of compose stacks to start / tear down in parallel (as long as the service key used is different) - that way I could run different stacks in parallel.

Subprojects which request the same key are still serialized on the semaphore tracked for that key.

Maybe for that you should have one instance of the service for each, so that then the max parallel usage also works as expected?

Fair enough - that was just my assumption here. Lets say, it would be nice if the Gradle samples or best practices docs chapter would have some guidance / help about such a thing.

Feel free to open a FR ticket to suggest it to them.

Edit: Got it working - having 4 workers with 3 subprojects, where subproject 2 and 3 will have the same service key Gradle will run subproject 1 + 2 in parallel while 3 is waiting for 2 to finish and will start after that one.

And the best thing is - it will not start / teardown any build service if it can take results from the cache - nice (I know you said that, just mentioning it).

[tkrah (Torsten K.)]

@Vampire

Maybe for that you should have one instance of the service for each, so that then the max parallel usage also works as expected?

Yeah good suggestion