Low processor utilization for native builds since 3.3

Jan · September 19, 2017, 12:55pm

I recently tried newer versions of gradle for a 200+ component native build. I´ve noticed that processor utilization seriously decreases for this build, when using gradle versions 3.3 or higher. When using gradle 3.1 or 3.2 all of the processros eight cores have a near 100% utilization. Using 3.3 or newer processor utilitation drops to around 10 to 30 percent on all cores, which leads to an increase in build time.

It seems to me, that the forked compile processes gain lower prioriority since 3.3., or maybe i am completely missing something? I would like to enable my build to use as much of the machine ressources as possible.

Daniel_L · September 19, 2017, 7:33pm

Hi @Jan, thanks for reporting this. This is unexpected. Could you give us a bit more information about your project (Ratio of C/C++/Assembly, custom tasks used, generated source set used, etc.)? If you could submit a build scan, it would also be helpful.

Could you also try 4.2 RC2? Around 4.0, ParallelizableTask was removed in favor of the worker API. The native tasks were refactored to use the worker API in 4.2 and you should have the same performance and maybe a bit more as reported by some of our partners.

Thanks,

Daniel

Jan · September 20, 2017, 12:56pm

Hi Daniel,

thanks for replying to my problem, As you have requested i run some build scans with the project. Allthough i was not able to produce a complete build with gradle 3.3+, at least the performance lose is visibile. I will try to fix the build for 4.1+ in the next days.

Buildscan with 3.2.1 (23 min 31 sec, completed successfully)
https://scans.gradle.com/s/bh6qqonkftwoi

Buildscan with 3.3 (42 min 42 sec, failed at around 95% due to out of heap space)
https://scans.gradle.com/s/4ehx6nkw54me2

Buildscan with 4.1 (33 min 31 sec, failed at around 40% due to a link error)
https://scans.gradle.com/s/cqm3wdg3zrcxy

About the build:

The build uses several custom inhouse gradle plugins to expand gradles native functionalities.

We recently migrated the scm and the build system (omake -> gradle, clearcase -> git). The source is spread over around 80 git repositories and is 99% C++. Each repository contains a configuration file which describes the gradle projects in this repository, as well as other repositories which may contain further dependencies. In a pre step prior to the build we clone/checkout all required repositories based on the configurations and generate a settings.gradle file which defines a project hierarchy containing all projects from all repositories.

Inside the projects we use a custom dsl to define include paths, dependencies to other project libraries AND dependencies to external dependencies, which are retrieved from an ivy repository. From the information of the dsl we use a custom plugin and rule sources to add include paths, and new library objects to the native gradle model.

Compiler and linker arguments have been parsed form the old omake build logs and currently reside inside a large csv file (which we call wallpaper). With a another plugin and more rule sources these arguments are also added to the native gradle model.

The whole configuration phase takes around 20 to 30 seconds, which we are pretty happy with, considerung the vast amount of manipulation.

As these custom expansions happen prior to the build or within the configuration phase, i don´t see a reason for impacting the execution performance.

We hope to be able to remove most of our custom plugins, when recently anounced changes on the gradle blog have been implemented:

4.2 RC 2

The release candidate produced following stack trace

R:\>gradlew clean assemble --parallel --scan -s

FAILURE: Build failed with an exception.

* What went wrong:
Could not create an instance of type org.gradle.initialization.DefaultSettings_Decorated.
> String index out of range: 0

* Try:
Run with --info or --debug option to get more log output.

* Exception is:
org.gradle.api.reflect.ObjectInstantiationException: Could not create an instance of type org.gradle.initialization.DefaultSettings_Decorated.
    at org.gradle.internal.reflect.DirectInstantiator.newInstance(DirectInstantiator.java:53)
    at org.gradle.api.internal.ClassGeneratorBackedInstantiator.newInstance(ClassGeneratorBackedInstantiator.java:36)
    at org.gradle.initialization.SettingsFactory.createSettings(SettingsFactory.java:46)
    at org.gradle.initialization.ScriptEvaluatingSettingsProcessor.process(ScriptEvaluatingSettingsProcessor.java:61)
    at org.gradle.initialization.PropertiesLoadingSettingsProcessor.process(PropertiesLoadingSettingsProcessor.java:37)
    at org.gradle.initialization.NotifyingSettingsProcessor$1.call(NotifyingSettingsProcessor.java:45)
    at org.gradle.initialization.NotifyingSettingsProcessor$1.call(NotifyingSettingsProcessor.java:42)
    at org.gradle.internal.progress.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:350)
    at org.gradle.internal.progress.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:340)
    at org.gradle.internal.progress.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:199)
    at org.gradle.internal.progress.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:120)
    at org.gradle.initialization.NotifyingSettingsProcessor.process(NotifyingSettingsProcessor.java:42)
    at org.gradle.initialization.DefaultSettingsLoader.findSettingsAndLoadIfAppropriate(DefaultSettingsLoader.java:90)
    at org.gradle.initialization.DefaultSettingsLoader.findAndLoadSettings(DefaultSettingsLoader.java:47)
    at org.gradle.internal.composite.CompositeBuildSettingsLoader.findAndLoadSettings(CompositeBuildSettingsLoader.java:49)
    at org.gradle.initialization.NotifyingSettingsLoader.findAndLoadSettings(NotifyingSettingsLoader.java:31)
    at org.gradle.initialization.DefaultGradleLauncher$LoadBuild.run(DefaultGradleLauncher.java:235)
    at org.gradle.internal.progress.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:336)
    at org.gradle.internal.progress.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:328)
    at org.gradle.internal.progress.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:199)
    at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:110)
    at org.gradle.initialization.DefaultGradleLauncher.loadSettings(DefaultGradleLauncher.java:159)
    at org.gradle.initialization.DefaultGradleLauncher.doBuildStages(DefaultGradleLauncher.java:122)
    at org.gradle.initialization.DefaultGradleLauncher.executeTasks(DefaultGradleLauncher.java:109)
    at org.gradle.internal.invocation.GradleBuildController$1.call(GradleBuildController.java:78)
    at org.gradle.internal.invocation.GradleBuildController$1.call(GradleBuildController.java:75)
    at org.gradle.internal.work.DefaultWorkerLeaseService.withLocks(DefaultWorkerLeaseService.java:152)
    at org.gradle.internal.invocation.GradleBuildController.doBuild(GradleBuildController.java:100)
    at org.gradle.internal.invocation.GradleBuildController.run(GradleBuildController.java:75)
    at org.gradle.tooling.internal.provider.ExecuteBuildActionRunner.run(ExecuteBuildActionRunner.java:28)
    at org.gradle.launcher.exec.ChainingBuildActionRunner.run(ChainingBuildActionRunner.java:35)
    at org.gradle.tooling.internal.provider.ValidatingBuildActionRunner.run(ValidatingBuildActionRunner.java:32)
    at org.gradle.launcher.exec.RunAsBuildOperationBuildActionRunner$1.run(RunAsBuildOperationBuildActionRunner.java:43)
    at org.gradle.internal.progress.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:336)
    at org.gradle.internal.progress.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:328)
    at org.gradle.internal.progress.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:199)
    at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:110)
    at org.gradle.launcher.exec.RunAsBuildOperationBuildActionRunner.run(RunAsBuildOperationBuildActionRunner.java:40)
    at org.gradle.tooling.internal.provider.SubscribableBuildActionRunner.run(SubscribableBuildActionRunner.java:51)
    at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:47)
    at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:30)
    at org.gradle.launcher.exec.BuildTreeScopeBuildActionExecuter.execute(BuildTreeScopeBuildActionExecuter.java:39)
    at org.gradle.launcher.exec.BuildTreeScopeBuildActionExecuter.execute(BuildTreeScopeBuildActionExecuter.java:25)
    at org.gradle.tooling.internal.provider.ContinuousBuildActionExecuter.execute(ContinuousBuildActionExecuter.java:80)
    at org.gradle.tooling.internal.provider.ContinuousBuildActionExecuter.execute(ContinuousBuildActionExecuter.java:53)
    at org.gradle.tooling.internal.provider.ServicesSetupBuildActionExecuter.execute(ServicesSetupBuildActionExecuter.java:57)
    at org.gradle.tooling.internal.provider.ServicesSetupBuildActionExecuter.execute(ServicesSetupBuildActionExecuter.java:32)
    at org.gradle.tooling.internal.provider.GradleThreadBuildActionExecuter.execute(GradleThreadBuildActionExecuter.java:36)
    at org.gradle.tooling.internal.provider.GradleThreadBuildActionExecuter.execute(GradleThreadBuildActionExecuter.java:25)
    at org.gradle.tooling.internal.provider.ParallelismConfigurationBuildActionExecuter.execute(ParallelismConfigurationBuildActionExecuter.java:43)
    at org.gradle.tooling.internal.provider.ParallelismConfigurationBuildActionExecuter.execute(ParallelismConfigurationBuildActionExecuter.java:29)
    at org.gradle.tooling.internal.provider.StartParamsValidatingActionExecuter.execute(StartParamsValidatingActionExecuter.java:64)
    at org.gradle.tooling.internal.provider.StartParamsValidatingActionExecuter.execute(StartParamsValidatingActionExecuter.java:29)
    at org.gradle.tooling.internal.provider.SessionFailureReportingActionExecuter.execute(SessionFailureReportingActionExecuter.java:59)
    at org.gradle.tooling.internal.provider.SessionFailureReportingActionExecuter.execute(SessionFailureReportingActionExecuter.java:44)
    at org.gradle.tooling.internal.provider.SetupLoggingActionExecuter.execute(SetupLoggingActionExecuter.java:45)
    at org.gradle.tooling.internal.provider.SetupLoggingActionExecuter.execute(SetupLoggingActionExecuter.java:30)
    at org.gradle.launcher.daemon.server.exec.ExecuteBuild.doBuild(ExecuteBuild.java:67)
    at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:36)
    at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
    at org.gradle.launcher.daemon.server.exec.WatchForDisconnection.execute(WatchForDisconnection.java:37)
    at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
    at org.gradle.launcher.daemon.server.exec.ResetDeprecationLogger.execute(ResetDeprecationLogger.java:26)
    at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
    at org.gradle.launcher.daemon.server.exec.RequestStopIfSingleUsedDaemon.execute(RequestStopIfSingleUsedDaemon.java:34)
    at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
    at org.gradle.launcher.daemon.server.exec.ForwardClientInput$2.call(ForwardClientInput.java:74)
    at org.gradle.launcher.daemon.server.exec.ForwardClientInput$2.call(ForwardClientInput.java:72)
    at org.gradle.util.Swapper.swap(Swapper.java:38)
    at org.gradle.launcher.daemon.server.exec.ForwardClientInput.execute(ForwardClientInput.java:72)
    at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
    at org.gradle.launcher.daemon.server.exec.LogAndCheckHealth.execute(LogAndCheckHealth.java:55)
    at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
    at org.gradle.launcher.daemon.server.exec.LogToClient.doBuild(LogToClient.java:62)
    at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:36)
    at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
    at org.gradle.launcher.daemon.server.exec.EstablishBuildEnvironment.doBuild(EstablishBuildEnvironment.java:82)
    at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:36)
    at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
    at org.gradle.launcher.daemon.server.exec.StartBuildOrRespondWithBusy$1.run(StartBuildOrRespondWithBusy.java:50)
    at org.gradle.launcher.daemon.server.DaemonStateCoordinator$1.run(DaemonStateCoordinator.java:295)
    at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:63)
    at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:46)
    at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:55)
Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: 0
    at org.gradle.util.NameValidator.validate(NameValidator.java:37)
    at org.gradle.initialization.DefaultProjectDescriptor.<init>(DefaultProjectDescriptor.java:56)
    at org.gradle.initialization.DefaultSettings.createProjectDescriptor(DefaultSettings.java:99)
    at org.gradle.initialization.DefaultSettings.<init>(DefaultSettings.java:82)
    at org.gradle.initialization.DefaultSettings_Decorated.<init>(Unknown Source)
    at org.gradle.internal.reflect.DirectInstantiator.newInstance(DirectInstantiator.java:51)
    ... 83 more

Jan · September 26, 2017, 9:41am

I have done further research. It looks like the worker threads have different thread priorities.

Daemon worker thread priority:
Gradle 3.2.1 --> 7
Gradle 3.3 --> 2

I think this is the reason for the decrease in performance i witness.

I would like to increase the priority of the daemon worker threads manually, but i have not yet found a way to accomplish this, nor have i found any helpful documentation on this topic.

Jan · September 26, 2017, 2:36pm

I was finally able to fix the build to work with 4.2.
The processor utilization and performance is at similiar levels to 3.2.1.

Thanks for encouraging me to try 4.2.