Disabling parallel for certain tasks only

Hi,

We use the org.gradle.parallel=true pretty heavily in our project. There are certain tasks however that usually fail when run in parallel mode. Is there any way to specify that some tasks are never to be run in parallel? I know I can set -Dorg.gradle.parallel=false when executing my task by itself but there are times when I’d like to run several tasks together and for the system to just know not to execute certain ones in parallel.

This usually has to do when installing or deleting multiple APKs.

Thanks, Derek

2 Likes

I think we check if tasks have overlapping output directories and won’t run them in parallel if they do. So you could try having a dummy output directory that’s shared by all tasks that shouldn’t run in parallel. We also won’t run tasks that are in the same project in parallel (this will eventually change).

Setting org.gradle.parallel=false after task execution has started won’t turn off parallel execution.

You could also look at the mustRunAfter methods on Task. You could then setup the tasks to run in a particular order, but not require all of them to run (like dependsOn would).

Awesome, the common output directory works. Thanks for the idea.

This is an example of a “cool”, non-standard, trick that ultimately reads “hack” and makes builds hard to understand and maintain over time. It would be far preferable to have some dynamic equivalent of parallel=false that. Please consider that.

2 Likes

Yeah, this is in “hack” territory. ‘mustRunAfter’ would still be hacky, but at least it would make it clear there’s an ordering that could be documented (vs an output that’s never used).

Just having a “don’t run this task in parallel” isn’t quite what we’d want long term. We’d like to be able to say there’s some common resource and that’s an input/output of the task and Gradle understands that some tasks need to wait for that dependency to become available.

This is a perfect example of the overall philosophy of gradle that I’d like to see tempered a bit in some cases. The usual gradle model is one where it infers & computes the proper build behavior on the basis of task dependencies, inputs, outputs, etc. This is a very good thing, especially given the complexity of large builds, where no one person has the knowledge or ability to comprehend intricate project & build inter-dependencies. However, there are times where the developer working in a small part of the build simply knows that there are build constraints and needs a straightforward and clear way aside from the normal artifact/task-based model of constraining the build in some specific way, such as parallel=false, which overrides the model-based behavior.

3 Likes

I found a use case where parallel=false in a task makes sense to me.

I want to list all the class paths of all my sub-projects.

Right now because I want to build in parallel I get my class list intertwined. How do I make this work properly (i.e. the way I want it to :wink: )?

subproject {
    task printJavaPaths << {
      println "${project} mainClasspath:"
      sourceSets.main.runtimeClasspath.each { File file ->
        println "      ${file}"
      }
}

(or at least the ability to switch parallel off in the command line, it seems i can switch parallel on from the command line, but not off)

1 Like

I tried mustRunAfter in a multi module project for the build tasks of the modules, however I am not able to use that. My usecase is that I want all of my modules to build parallelly except for one which I want to build after all the other modules have been built parallely.

It would be helpful if you could suggest something for this.

Thanks.

I was looking for something like above. On running some tasks in parallel and some serially. I have tests that binds to a port and if two of those tests runs at the same time one of them will fail. Any ideas?

1 Like

My guess would be to either explicitly say that whatever tasks contend for that port need to “.mustRunAfter” each other, or set those tasks outputs to an overlapping location so that Gradle won’t run them in-parallel.

We did the first option when we found that a third party app that claimed parallelism support actually didn’t, so we programmatically found those tasks by name and set them to run after each other in a line.

I have another use case for disabling parallel execution. We have a tool which simply refuses to work if you start two instances of it, but the order in which two tasks are run is otherwise irrelevant.

A related reason is that the tool in question will use all available cores on a machine for a single process, so running it in parallel wouldn’t make help, even if it worked.

It’s also a composite build, where having to configure .mustRunAfter on arbitrary sub-tasks becomes a bit awkward.

So in my case, I’d like to say that a task of a certain type must not run at the same time as another task of that type, or subtype.

And in gradle 4.5 using --no-parallel doesn’t seem to disable parallel execution for a composite build either.

--no-parallel didn’t seem to disable parallel composite builds in Gradle 4.8.1.

Is Gradle currently supposed to be able to be forced to build composite builds serially instead of in parallel?

  • If so, then what is the proper way to do so?
    • Is that methodology currently broken by a bug? If so, is there a workaround?
  • If not, is there a GitHub issue to add this functionality?

This is now possible with Build Services. Just define a service with limited parallelism and tell the tasks in question to use that service.

I have the same issue - and was trying to figure out how to take advantage of this. I’m relatively new to gradle.

We have a large multi-module project (100s of projects) and most run well in parallel. However we have some ‘heavy’ functional tests at the end of the build that involve

  • the usual set of tasks that result in building the test cases etc… and then
  • Starting up a server jvm process running from one of our assemblies created in the build
  • launching a script to run tests against the server (depends on above – probably should use mustrun*)
  • Shutting down the server process (finalizedby currently at the end)

Then these 3 stages appear across perhaps 8 modules. Each is correctly written with unique output directories, so on a ‘good machine’ (16GB+) they can run in parallel. But they get CPU constrained, increasing the overall time. Go down to a 4GB machine and the daemon will collapse with out of memory.

Clearly we can run the overall build with --no-parallel , but this would severely impact the other several 100 modules .

Are there any examples of just protecting these expensive projects (composed of 3 tasks) using the BuildServices. With My limited gradle skills a) I understand the broad idea b) I’m not sure how I could wrap the protection around multiple tasks (since I want those 3 steps to execute close together) c) I’m not sure how to actually codify an example for synchronization using that construct

I seem to remember that if two tasks share the same input file they won’t run in parallel. So you could create a dummy file for each group of tasks you don’t want to run in parallel then set it as a task input to all the tasks in the group

Eg:

['task1', 'task2','task3'].each { name ->
   tasks.named(name).configure { 
      inputs.file "$rootDir/dummy.mutex"
   } 
} 

It should be noted that this does not limit parallelism across multiple included builds, (composite build), since each included build gets its own Gradle instance with its own sharedServices registry. Thus it is not global in the sense of a composite build gradle assemble run.
Github comment

I had a similar problem: a large JUnit test suite across many subprojects, out of which some tests were too slow or resource-intense to be parallelised. I managed to solve it with task dependencies and JUnit Tags (in the case you’re using JUnit - maybe something similar is available on other platforms).

  1. Mark the slow tests with @Tag(“SlowTest”)
  2. Create test tasks in your subproject(s) like so:
task slowTest (type: Test) {
    group = 'verification'
    description = "Runs the slow tests that can't be ran in parallel."

    useJUnitPlatform {
		includeTags("SlowTest")
    }
}
// the following line is needed to make sure the slowTest tasks don't get parallelised across subprojects
slowTest.mustRunAfter(':otherSubProject:slowTest')
  1. Create a global slowTests task depending on all the subprojects’ tasks.
  2. Configure the global test task to depend on slowTests, to exclude the ‘SlowTests’ Tag and - optionally - to run its tests in parallel threads (on top of the gradle --parallel flags):
 test {
    	dependsOn serialTests
    	maxParallelForks = Runtime.runtime.availableProcessors()

        useJUnitPlatform {
			excludeTags("SlowTest")
		}
    }

this way, running a gradle build will execute the slow tests serially, then all the other tests in parallel.

Hope this helps!

@Lance
According to Sterling above, it is output, not input.
And either way, it is a dirty intransparent hack and a shared build service with limited parallelism is the better way to go. :wink:

@planetf1
What problem do you have with the shared build service approach?
Actually optimally you should not have three tasks anyway, but one of the intended use-cases for a shared build service is exactly that, to start a server when needed and shut it down when not needed anymore. And if you need three different servers, then just create four build services. Three that start up and shutdown the correct test server and one that all three tasks use and has a max parallelity of 1 to serialize the tasks.

@TdrFLp
Some notes to your solution if you allow:

  • I’d probably check whether the additional test task could better be modelled using the new JVM Testsuites plugin
  • For serialization, I’d still recommend using the intended tool, which is a shared build service, or you also have to add arbitrary ordering constraints each time a new subproject has slow tests while they are semantically not really correct.
  • What do you mean with “global slowTests” task? If you mean in the root project, then why? If you do ./gradlew slowTests Gradle is already executing all slowTests tasks in all projects that have one. And the fast tests also do not really depend on the slow tests, so I’d make check depend on the slow tests and make the fast tests mustRunAfter the slow tests which is semantically more what you want I think and thus increases maintainability and transparency.

Hi @Vampire, thank you for your input (no pun intended).

  • The Testsuites plugin is the cleaner solution (i.e. creating separate src/ and resources/ folders), but in my case, we were talking about 60-70 tests in a suite of more than 5000, so I opted for the tags for simplicity.
  • What do you mean exactly by shared build service as a solution for serialization?
  • As of “global” task I did mean “a task defined in the root project”. I defined the root project task slowTests depending on the 6 subproject tasks, each called slowTest (no “s”), in order to define the dependency test dependsOn slowTests in the subProjects section of the root project. Omitting any of these gave me a compilation error. Note that most project don’t have a slowTest task, and we want it that way - the idea, in our company, is to have less and less of these.
  • I found out about the better suitability of mustRunAfter check myself a few hours after posting this, Definitely the correct solution.