--profile is significantly less useful than before

Help/Discuss
1

So I’ve been pushing this build migration to Gradle for quite some time now. Between third party bugs and other tasks and the monstrously huge size of our build (which is why we needed something that performs in the first place), it’s taken at least a year and a half. I think we waited 9 months to see if IntelliJ would fix their issues but this was in vain and we had to adapt our build so that both Eclipse and IntelliJ would work. Now I’ve got it running and I’d like to show everyone how much faster it is than Maven… except I can’t. It isn’t faster. It’s about the same. I’d like to do some investigating but I can’t do that either since the --profile option seems to have been castrated. I know Gradle has a cloud widget which we can’t use in accordance with our corporate security and intellectual property policies, nor would I ever recommend an unnecessary third party exposure to anyone else. I’d be more likely to recommend an open source fork of Gradle or at least a plugin that does what is needed and does it locally (and securely).

Soooo… what do I do next? I suppose what I’ll have to do, for now, is downgrade long enough to figure out the bottleneck unless someone has a better idea.

2

Hi Jim, I work for Gradle Inc. Lets have a chat to see how we can help. It should certainly be faster. I don’t have your email. can you email me? rooz [at] gradle [dot] com

3

Hi Jim,

We haven’t made any changes to --profile. Its functionality hasn’t been reduced in any way. Can you please elaborate on what you are missing? It may be down to the configuration of your build.

4

Well first off, I stand corrected. I eventually found it all. Why does it seem like it was so much easier to find before? As I recall wasn’t everything linked in to the root or something? These are the results I’m seeing for various tests. Some of them do seem anomalous.

gradle clean build --profile
Total time: 1 hrs 36 mins 17.953 secs

gradle clean build
Total time: 1 hrs 14 mins 24.001 secs

OK it makes sense that profiling should take a little longer.

gradle clean build --parallel
Total time: 1 hrs 29 mins 11.395 secs

OK It DOESN’T make sense that parallel exectuion is slower than serial.

gradle clean build --parallel --offline
Total time: 1 hrs 30 mins 17.105 secs

The same. Should be even faster since we aren’t contacting a remote repo.

mvn clean install
Total time: 01:50 h

5

One thing that was very useful to us that I can’t find was a measure of which tests took the longest to run. We found like 2 or 3 tests that were taking 4-5 minutes apiece and re-examined them. What I’m talking about is the answer to this question: In a build with hundreds of sub-components all with tests, which were the 10 slowest and how long were they taking?

6

Re: Now I’ve got it running and I’d like to show everyone how much faster it is than Maven… except I can’t. It isn’t faster. It’s about the same.

… just a ‘side note’ …

We’ve switched to Gradle from Maven and are running on Gradle for roughly 1.5 year. I would say that in terms of full ‘clean’ build they are indeed ‘about the same’ (note: I see you did not try executing MAVEN in multi-threaded mode – something like mvn clean install -T 8 - in our case it reduced build time by 60-70%).

The killer feature of Gradle is its flexibility in terms of configuration – we have roughly 400 modules in main part of the project - most of them are of 3-4 typical types. Gradle can inject external configuration scripts via ‘apply from’ and by using this feature overall size of build configuration reduced 100 times – type of the module is now encoded in the module folder name, e.g. via suffix ‘-api’, ‘-impl’, ‘-web’ and 99% of configuration is performed from root build.gradle via invocation of ‘api.gradle’ and so on … this is a bit oversimplified, but just to give an idea. (Note: configuring Gradle is also not a piece of cake – on large-scale projects. It took from us roughly 1 FTE-month to make initial build with Gradle so that all features of previous MAVEN build were still in place – missing intellisense for DSL, missing plugins, glitches with integration of 3rd party staff and so on).

In terms of performance I would say you can get significant performance difference vs. MAVEN only on incremental builds.

  • Unlike maven Gradle does check for non-changed code and builds only required parts (same can be done in MAVEN via some hacks, but – they are hacks)

  • Gradle has ‘configureondemand’ feature – this way build of individual sub-module can be significantly faster, i.e. out of 400 only 40 required modules will be initialized. Though this feature has few problems:

  • Build configuration should be thoroughly crafted - i.e. never use ‘projects.all’ or ‘subprojects.all’ – this triggers configuration of ‘all’ modules. Instead – use in root project gradle.beforeProject or ‘gradle.afterProject’

  • ‘parallel’ and ‘configureondemand’ are ‘incubating’ features – as long as I use Gradle :slight_smile: – various unexpected things may happen (and some-time build fails somewhere in Gradle’s internals - some time it takes days to identify the root cause). For example Gradle itself fails to build when ‘configureondemand’ is turned ‘on’ (see Cannot build official Gradle repo).

7

OK, I can provide our root build.gradle but I think that’s the best we can do in a project it would otherwise take hours to redact.

plugins {

    id 'nebula.dependency-recommender' version '3.3.0'
    id 'idea'
    id "org.sonarqube" version "2.0.1"
    id "jacoco"
}

group = 'com.xyz'

task wrapper(type: Wrapper) {
    description = "This task allows you to change the Gradle version used by this build."
    gradleVersion = '2.14.1'
}

apply from: "${project.rootDir}/buildSrc/config/dbpropertiesLoad.gradle"

ext {

    printComponentDependencies = [ 'org.slf4j:slf4j-api',
                                   'org.apache.httpcomponents:httpclient',
                                   'org.apache.commons:commons-lang3',
                                   'com.google.code.gson:gson',
                                   'com.fasterxml.jackson.core:jackson-core',
                                   'com.lowagie:itext',
                                   'com.fasterxml.jackson.core:jackson-databind',
                                   'com.jscape:sftp' ]

   batchTemplateDirectory =  "${rootProject.rootDir}/build-utils/batch_templates"
   batchdeployDir = "/development/batch/jobs"
}

jacoco {

    toolVersion = "0.7.6.201602180812"
}

project(":commons:databaseutils") {
    sonarqube {
        skipProject = true
    }
}

project(":commons:schemas") {
    sonarqube {
        skipProject = true
    }
}

sonarqube {//This merges the -D properties to the sonarqube plugin.
    System.properties.findAll {
        it.key.toString().startsWith("sonar")
    }.each { k, v ->
        properties {
            property k, v
        }
    }
}

subprojects {

    apply plugin: 'java'
    apply plugin: 'idea'
    apply plugin: 'jacoco'
    apply plugin: 'eclipse'

    apply from: "${project.rootDir}/buildSrc/config/dependencyRecommendations.gradle"

    project.group = "${easGroup}"

    sourceCompatibility = System.getProperty('java.version')
    targetCompatibility = System.getProperty('java.version')

    test {
        jacoco {
            excludes = [ "stuff.*",
                         "stuff.v1.*",
                         "stuff.*",
                         "stuff.*",
                         "**/generated-src/**",
                         "javolution.*" ]
        }

        exclude '**/*ITCase*'
        include '**/*Test*'
    }

    jacocoTestReport {
        dependsOn tasks.withType(Test)
        reports {
            xml.enabled true
            csv.enabled false
            html.destination "${buildDir}/jacocoHtml"        }
    }

    //Using new repo URL's...requires datapipe connection
    repositories {
        maven { url 'https://[redacted]/artifactory/libs-release' }
        maven { url 'https://[redacted]/artifactory/libs-snapshot' }
        mavenLocal()
    }

    rootProject.tasks["sonarqube"].dependsOn jacocoTestReport

    //Applies only for Eclipse users. Cleans up BOM dependencies from classpath entries.
    apply from: "${rootProject.rootDir}/buildSrc/config/eclipseMisc.gradle"
}

idea {
    project {
        vcs = 'Git'
    }
}
8

Hey Jim,

generally the “clean build” performance of large projects is dominated by test execution. This is mostly out of the build system’s control, the only thing Gradle can do for you is tell you which tests are taking long, so you can make a decision what to do with those tests. Have a look at the TestListener API.

The most important performance improvement is simply: Don’t call “clean”. Gradle has great support for incremental builds, making them fast and correct. This is probably the biggest change for users coming from Maven, where “clean” is the norm, because incremental build support in Maven is broken. When you use incremental builds, Gradle will only reexecute tasks that actually need to run, which means that (probably) many of your test tasks can be skipped.

It does make sense if your machine or the JVM running Gradle is memory constrained. Running things in parallel will use more memory at the same time.

Gradle will only contact the remote repo once. Unless you have dynamic/changing versions, in which case it will contact the repo again after the timeout (default 24h) expires.

Cheers,
Stefan