Finalizer tasks can take a very long time

Old Forum Archive Bugs
,
1

Hi. Would it be possible to take a look at the performance of DefaultTaskExecutionPlan.enforceFinalizerTasks()? If a finalizer task has dependencies of its own, this call can take a very long time. Sample script to reproduce:

defaultTasks "actualTask"

50.times { index ->
    def testTask = task "testTask$index" << {}
    index.times { testTask.dependsOn tasks.findByName("testTask$it") }
}
println "Done setting up"

task finalizerTask
finalizerTask.dependsOn testTask33

task actualTask
actualTask.finalizedBy finalizerTask

Note: This is using gradle version 2.2.1; sorry for the false alarm if not an issue in later versions.

Thank you for your help!

2

Thanks. We appreciate self contained examples.

I’ve raised GRADLE-3283, it’s still a problem in the latest build.

3

I was able to run the sample script and hook it up with JConsole to get the following stack trace:

Name: main
State: RUNNABLE
Total blocked: 3  Total waited: 0
Stack trace: 
java.util.TreeMap.getFirstEntry(TreeMap.java:2124)
java.util.TreeMap.keyIterator(TreeMap.java:1106)
java.util.TreeMap$KeySet.iterator(TreeMap.java:1119)
java.util.TreeSet.iterator(TreeSet.java:181)
org.gradle.execution.taskgraph.DefaultTaskExecutionPlan.enforceWithDependencies(DefaultTaskExecutionPlan.java:663)
org.gradle.execution.taskgraph.DefaultTaskExecutionPlan.enforceWithDependencies(DefaultTaskExecutionPlan.java:664)
org.gradle.execution.taskgraph.DefaultTaskExecutionPlan.enforceWithDependencies(DefaultTaskExecutionPlan.java:664)
org.gradle.execution.taskgraph.DefaultTaskExecutionPlan.enforceWithDependencies(DefaultTaskExecutionPlan.java:664)
org.gradle.execution.taskgraph.DefaultTaskExecutionPlan.enforceWithDependencies(DefaultTaskExecutionPlan.java:664)
org.gradle.execution.taskgraph.DefaultTaskExecutionPlan.enforceWithDependencies(DefaultTaskExecutionPlan.java:664)
org.gradle.execution.taskgraph.DefaultTaskExecutionPlan.enforceWithDependencies(DefaultTaskExecutionPlan.java:664)
org.gradle.execution.taskgraph.DefaultTaskExecutionPlan.enforceWithDependencies(DefaultTaskExecutionPlan.java:664)
org.gradle.execution.taskgraph.DefaultTaskExecutionPlan.enforceWithDependencies(DefaultTaskExecutionPlan.java:664)
org.gradle.execution.taskgraph.DefaultTaskExecutionPlan.enforceWithDependencies(DefaultTaskExecutionPlan.java:664)
org.gradle.execution.taskgraph.DefaultTaskExecutionPlan.enforceWithDependencies(DefaultTaskExecutionPlan.java:664)
org.gradle.execution.taskgraph.DefaultTaskExecutionPlan.enforceWithDependencies(DefaultTaskExecutionPlan.java:664)
org.gradle.execution.taskgraph.DefaultTaskExecutionPlan.enforceWithDependencies(DefaultTaskExecutionPlan.java:664)
org.gradle.execution.taskgraph.DefaultTaskExecutionPlan.enforceWithDependencies(DefaultTaskExecutionPlan.java:664)
org.gradle.execution.taskgraph.DefaultTaskExecutionPlan.enforceWithDependencies(DefaultTaskExecutionPlan.java:664)
org.gradle.execution.taskgraph.DefaultTaskExecutionPlan.enforceWithDependencies(DefaultTaskExecutionPlan.java:664)
org.gradle.execution.taskgraph.DefaultTaskExecutionPlan.enforceWithDependencies(DefaultTaskExecutionPlan.java:664)
org.gradle.execution.taskgraph.DefaultTaskExecutionPlan.enforceFinalizerTasks(DefaultTaskExecutionPlan.java:657)
org.gradle.execution.taskgraph.DefaultTaskExecutionPlan.taskComplete(DefaultTaskExecutionPlan.java:641)
org.gradle.execution.taskgraph.AbstractTaskPlanExecutor$TaskExecutorWorker.processTask(AbstractTaskPlanExecutor.java:66)
org.gradle.execution.taskgraph.AbstractTaskPlanExecutor$TaskExecutorWorker.run(AbstractTaskPlanExecutor.java:50)
org.gradle.execution.taskgraph.DefaultTaskPlanExecutor.process(DefaultTaskPlanExecutor.java:25)
org.gradle.execution.taskgraph.DefaultTaskGraphExecuter.execute(DefaultTaskGraphExecuter.java:110)
org.gradle.execution.SelectedTaskExecutionAction.execute(SelectedTaskExecutionAction.java:37)
org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:37)
org.gradle.execution.DefaultBuildExecuter.access$000(DefaultBuildExecuter.java:23)
org.gradle.execution.DefaultBuildExecuter$1.proceed(DefaultBuildExecuter.java:43)
org.gradle.execution.DryRunBuildExecutionAction.execute(DryRunBuildExecutionAction.java:32)
org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:37)
org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:30)
org.gradle.initialization.DefaultGradleLauncher$4.run(DefaultGradleLauncher.java:155)
org.gradle.internal.Factories$1.create(Factories.java:22)
org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:90)
org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:52)
org.gradle.initialization.DefaultGradleLauncher.doBuildStages(DefaultGradleLauncher.java:152)
org.gradle.initialization.DefaultGradleLauncher.access$200(DefaultGradleLauncher.java:33)
org.gradle.initialization.DefaultGradleLauncher$1.create(DefaultGradleLauncher.java:100)
org.gradle.initialization.DefaultGradleLauncher$1.create(DefaultGradleLauncher.java:94)
org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:90)
org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:62)
org.gradle.initialization.DefaultGradleLauncher.doBuild(DefaultGradleLauncher.java:94)
org.gradle.initialization.DefaultGradleLauncher.run(DefaultGradleLauncher.java:83)
org.gradle.launcher.exec.InProcessBuildActionExecuter$DefaultBuildController.run(InProcessBuildActionExecuter.java:94)
org.gradle.tooling.internal.provider.ExecuteBuildActionRunner.run(ExecuteBuildActionRunner.java:28)
org.gradle.launcher.exec.ChainingBuildActionRunner.run(ChainingBuildActionRunner.java:35)
org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:43)
org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:28)
org.gradle.launcher.exec.ContinuousBuildActionExecuter.execute(ContinuousBuildActionExecuter.java:77)
org.gradle.launcher.exec.ContinuousBuildActionExecuter.execute(ContinuousBuildActionExecuter.java:47)
org.gradle.launcher.exec.DaemonUsageSuggestingBuildActionExecuter.execute(DaemonUsageSuggestingBuildActionExecuter.java:51)
org.gradle.launcher.exec.DaemonUsageSuggestingBuildActionExecuter.execute(DaemonUsageSuggestingBuildActionExecuter.java:28)
org.gradle.launcher.cli.RunBuildAction.run(RunBuildAction.java:43)
org.gradle.internal.Actions$RunnableActionAdapter.execute(Actions.java:170)
org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.execute(CommandLineActionFactory.java:237)
org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.execute(CommandLineActionFactory.java:210)
org.gradle.launcher.cli.JavaRuntimeValidationAction.execute(JavaRuntimeValidationAction.java:35)
org.gradle.launcher.cli.JavaRuntimeValidationAction.execute(JavaRuntimeValidationAction.java:24)
org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:206)
org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:169)
org.gradle.launcher.cli.ExceptionReportingAction.execute(ExceptionReportingAction.java:33)
org.gradle.launcher.cli.ExceptionReportingAction.execute(ExceptionReportingAction.java:22)
org.gradle.launcher.Main.doAction(Main.java:33)
org.gradle.launcher.bootstrap.EntryPoint.run(EntryPoint.java:45)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:483)
org.gradle.launcher.bootstrap.ProcessBootstrap.runNoExit(ProcessBootstrap.java:54)
org.gradle.launcher.bootstrap.ProcessBootstrap.run(ProcessBootstrap.java:35)
org.gradle.launcher.GradleMain.main(GradleMain.java:23)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:483)
org.gradle.wrapper.BootstrapMainStarter.start(BootstrapMainStarter.java:30)
org.gradle.wrapper.WrapperExecutor.execute(WrapperExecutor.java:129)
org.gradle.wrapper.GradleWrapperMain.main(GradleWrapperMain.java:61)

At the point I connected, I was at a 2% build. If you like, I can investigate further and see what’s going on.

4

Hey Chris,

We always appreciate PRs. although we’re pretty busy (and hiring).

My suspicion is that we work out an execution plan for “regular” dependencies pretty efficiently (see this), but we just tack on the finalizer tasks without considering their dependencies at that point in time. There may be a good reason for that, as it seems unusual to have dependencies for a finalizer (at least as many as are in the OP), but it could just be oversight as well. I would take a look at the history of that method to see if we ever considered finalizer dependencies.

Either way, since the finalizer’s dependencies aren’t in the original execution plan, we end up traversing all the finalizer dependencies using a Shlemiel the painter’s algorithm. There are a lot of tests in DefaultTaskExecutionPlanTest to check for the correct ordering of things, so there are some things to think through.

5

Sorry to ask, but what’s a PR? :stuck_out_tongue:

Looking at the code, I think it could be optimized to not expand a task if
it has already been expanded. I’ll take a look at the link you sent me and
see if I can apply it to the finalizer tasks.

You mentioned being busy, are there any tasks you’d recommend for somebody
just getting started?

Thanks,

Chris

6

PR is a pull request on GitHub.

I’d say this particular problem is of the type that it would sit around awhile (unusual use case, really sensitive area of code, hard to test).

Documentation fixes are usually merged quickly. Just a quick look around the Bugs topics here and some issues on our JIRA:

This is mostly removing stuff:
scalaConsole problem

This looks kind of straightforward, I think (check out FilterChain):
https://issues.gradle.org/browse/GRADLE-3122

This is a little more involved (Lorant had a PR last week for it, but it broke other things):
https://issues.gradle.org/browse/GRADLE-3329

Honestly, answering some questions on the forums is really helpful too.

Would you be working on Linux/Windows/Mac? Would you be interested in fixing things related to native (C/C++) builds?

7

To answer your questions:
I’m building my source on Linux. I’ve been mostly looking at the Java/Groovy stuff, but can take a look at C/C++ builds as well.

Is there any documentation on what dependencies are needed to compile C/C++?

8

We don’t have any C/C++ code in the Gradle tree that we build (everything is Java/Groovy), but we have some integration tests that build other languages (Assembler/C/C++/Obj-C/Obj-C++). Any GCC4.x should work, I think. I’d try building one of the C++ samples that come in the distribution, if that works, you have everything you need.

The biggest thing that helps with PR is test coverage. For everything except trivial changes, we need to have some sort of test coverage. Sometimes that’s just unit tests, sometimes that’s integration tests. Basically, if there’s a bug, we want an integration test that demonstrates it and then changes to fix it, so that it doesn’t happen again. If it’s a new feature, we want integration tests that exercise it (and samples/docs if that makes sense).

So for setting up, I’d say:

  • Use IntelliJ, if you’re not already
  • Make sure daemon/parallel flags are turned on in ~/.gradle/gradle.properties
  • Run ./gradlew sanityCheck or ./gradlew $project:check for your changes to $project
  • Keep in mind that we target JDK6 (so don’t use JDK7 or JDK8 features).

Is there anything in particular you’re interested in helping out with?

9

Truth be told, I’m trying to learn as much as I can about Gradle so that I can educate the people I work with. I’m very strong with Java and C/C++ and would like to get more experience with Groovy. I’ve only really used JUnit and TestNG for testing, so I’d love to learn more about the different test frameworks that are out there.