I am trying to limit the number of cores used for my gradle build. I am trying to setup a build machine that has predictable behavior (i.e., each job is allocated 2 cores and therefore should take x amount of minutes to complete… assuming I have plenty of memory and disk io is not a problem). If I have a couple parallel gradle jobs running, they seem to steal cpu resources from each other and the completion time is variable (one job kicked off at a later stage can stretch the first job considerably).
I was looking at the –max-workers command line arg. At first, this appeared to be what I was looking for since the default value is set to the number of processors on your machine. In practice, however, I set this value to 2 and my cpu was over 300% (on a 4 core box) during a single job run. I am using gradle 2.14.
Is anything like this supported natively in gradle… or perhaps this should be more on the unix scripting side. Thanks.
One potential reason for this is that --max-workers is only taken into account for parallel task execution, and parallel native compilation in Gradle 2.14 and lower. That means that if you have others tasks that do parallel work, like test execution (using maxParallelForks), then you could end up with more workers then that specified by --max-workers. This has been addressed in Gradle 3.0 so that text executors respect the --max-workers value.
I would say 2 workers consuming 300% CPU is about “par-for-the-course” due to the JVM overheads of Garbage Collection & JIT-compiling the code, which run in parallel to the build/test threads.
one job kicked off at a later stage can stretch the first job considerably
One other thing to watch out for is hitting any memory pressure (whether JVM’s internal heap, or system-wide). For internal heap sizing, Java likes to have a lot of empty space. After each GC you generally want to see the heap usage drop well below 50% of the Max that you’ve allocated it (the so-called low-water mark) otherwise performance will start to suffer, possibly dramatically.
But if you’re ruled that out, then you might be able to modify the build scripts to include a few shouldRunAfter statements between tasks (which are just hints, in contrast to mustRunAfter which affect the build logic) to coax the tasks to run in a more predictable/efficient order. (Having said that, I’ve never gone there myself as it would be too much work to check & maintain such fine-grained tweaks - as they say, such “performance optimisations often have a short shelf-life”.)