Jacoco global default thresholds / local modifications setup

Hi,
comparing the palantir jacoco plugin (https://github.com/palantir/gradle-jacoco-coverage) with the gradle plugin (https://docs.gradle.org/current/userguide/jacoco_plugin.html), it seems configuring custom local thresholds is much more of a pain.

What I mean is in a gradle multi-project build, I would like to define a global policy (like 90% line coverage), but for a file inside submodule X, I want to be more lenient like (allowing 50% line for class X.java).

With palantir, this is trivial, globally I set

jacocoCoverage.fileThreshold 0.9, LINE

and in the module X, I add

jacocoCoverage.fileThreshold 0.5, LINE, "X.java"

However with the gradle plugin, I need to define a rule, so if I have a global rule

        rule {
            limit {
                minimum = 0.8
                counter = 'LINE'
            }
        }

Tnen I cannot override this rule in the submodule for class X. Redefining the rule for class X globally however breaks module encapsulation. So I find myself having to either copy/paste global policies, or to write some ugly gradle function that computes policies for each submodule.

Do I miss anything obvious? Is there any clean solution?

Ah, I guess what happened is that Gradle copied the configuration approach from Maven, instead of copying from the palantir plugin. Would it be too late to throw away the Maven aproach and use the palantir approach instead?

The problem is not just for multi-module builds. Whenever a team wants to have some default global thresholds, and some exceptions for specific files and packages, the Maven-like approach with rules is extremely difficult to use, because multiple rules have to be defined with mutually exclusive includes and excludes.

Also see https://jivimberg.io/blog/2018/04/26/gradle-verify-coverage-with-exclusions/ for more confusion with the current plugin.

Just to show an example:

Consider enforcing global line coverage of 90%, except using 80% for files Foo.java. Also global branch coverage of 80%, except using 25% for package com.example.dto

With palantir, that is:

# from a shared global_defaults.gradle
fileThreshold 0.9, LINE
fileThreshold 0.8, BRANCH
 
# (sub-)project-specific coverage.gradle
fileThreshold 0.8, LINE, "Foo.java"
fileThreshold 0.25, BRANCH, "com.example.dto"

As you can see the DSL is pretty close to my natural language explanation, and it is easy and natural to maintain global defaults in one place, and local overrides in another place.
Now with the standard gradle plugin (following Maven), that would be

jacocoTestCoverageVerification {
violationRules {
rule {
    element = 'REPORT'
    excludes = ['Foo.java'] // need to exclude here

    limit {
        counter = 'LINE'
        minimum = 0.9
    }
}
rule {
    element = 'REPORT'
    includes = ['Foo.java']

    limit {
        counter = 'LINE'
        minimum = 0.8
    }
}
rule {
    element = 'REPORT'
    excludes = ['com.example.dto'] // need to exclude here

    limit {
        counter = 'BRANCH'
        minimum = 0.8
    }
}
rule {
    element = 'REPORT'
    includes = ['com.example.dto']

    limit {
        counter = 'BRANCH'
        minimum = 0.25
    }
}
}
}

The more non-defaults to be used, the worse it gets, for any new threshold value in the same category, exclusions have to be correctly set in all similar rules. It’s horrible spaghetti-references that emerge.

Hey Thibault,

I’m aware that this is a old topic, but we’re currently facing the same problem and it seems that there is nothin new “build in”, so we build our own solution that I’d like to share.

First of all we build a jacoco report in the root project containing the executions of all submodules:

tasks {
    // ...
    jacocoTestReport {
        executionData.setFrom(fileTree(project.rootDir.absolutePath).include("**/build/jacoco/*.exec"))

        subprojects
                .filter { it.name !in listOf("integrationtests") }
                .forEach {
                    this@jacocoTestReport.sourceSets(it.sourceSets.main.get())
                    this@jacocoTestReport.dependsOn(it.tasks.test)
                }

        reports {
            xml.isEnabled = true
            xml.destination = file("$buildDir/reports/jacoco/report.xml")
        }
    }
   // ...
}    

Then we have a helper method to parse the XML report:

import org.w3c.dom.NodeList
import org.xml.sax.InputSource
import java.io.StringReader
import javax.xml.parsers.DocumentBuilderFactory
import javax.xml.xpath.XPathConstants
import javax.xml.xpath.XPathFactory

fun calculateCoverage(): Map<String, Float> {
    val xmlFile = File("$buildDir/reports/jacoco/report.xml").readText()

    val dbFactory = DocumentBuilderFactory.newInstance().apply {
        setFeature("http://apache.org/xml/features/nonvalidating/load-external-dtd", false)
    }

    val dBuilder = dbFactory.newDocumentBuilder()
    val doc = dBuilder.parse(InputSource(StringReader(xmlFile)))

    val xpFactory = XPathFactory.newInstance()
    val xPath = xpFactory.newXPath()

    (xPath.evaluate("/report/counter", doc, XPathConstants.NODESET) as NodeList).let { nodes ->
        return (0 until nodes.length).map { i ->
            val node = nodes.item(i)
            val type = node.attributes.getNamedItem("type").nodeValue
            val missed = node.attributes.getNamedItem("missed").nodeValue.toInt()
            val covered = node.attributes.getNamedItem("covered").nodeValue.toInt()
            val total = missed + covered
            val coverage = covered.toFloat() / total.toFloat()
            return@map (type to coverage)
        }.toMap()
    }
}

I assume that it will be less code with groovy and XmlSlurper

and finally a custom coverage verification task project level:

tasks {
    // ...
    val verifyCoverage by registering {
        dependsOn(jacocoTestReport)
        doLast {
            val expectedCoverage = 0.85f
            val coverage = calculateCoverage()
            if ((coverage["LINE"] ?: 0f) < expectedCoverage) {
                throw GradleException("Expecting $expectedCoverage LINE coverage but was ${coverage["LINE"]}")
            }
        }
    }
    check.get().dependsOn(verifyCoverage)
    // ...
}

I know that this is no very nice solution, but this way you can have a global verification in root project and more fine grained verifications using jacocoTestCoverageVerification task in every submodule.

1 Like

thanks for sharing.

viele Gruesse

We implemented this in a slightly simpler way.

Shared coverage.gradle:

jacocoTestCoverageVerification {
    violationRules {
        rule {
            element = 'CLASS'

            limit {
                minimum = 1
            }
        }
    }
}

Project specific build.gradle:

jacocoTestCoverageVerification {
    violationRules {
        // To exclude a specific class from the global rule
        rules.findAll { it.element == 'CLASS' }.each {
            it.excludes = [
                'com.example.Foo'
            ]
        }
        
        // Rule specific to the Foo class 
        rule {
            element = 'CLASS'
            includes = [
                'com.example.Foo'
            ]

            limit {
                counter = 'INSTRUCTION'
                minimum = 0.16
            }

            limit {
                counter = 'LINE'
                minimum = 0.3
            }

            limit {
                counter = 'COMPLEXITY'
                minimum = 0.5
            }

            limit {
                counter = 'METHOD'
                minimum = 0.5
            }
        }
    }
}