I have some functional tests with Gradle’s test kit, for a custom plugin. The tests work fine and IntelliJ reports 100% code coverage on it. When I put Jacoco into Gradle and run the same report it claims 0% code coverage on every @TaskAction I have. I have looked into forum posts and attempted to use the the plugin Jacoco TestKit Plugin, to no avail, I think I’m missing something. If anyone has used that plugin with Java and building a custom plugin any help would be great.
If you also know how to change the build.gradle.kts to make it work as well that would be great.
Look at the build scripts for the asciidoctor-gradle-plugin how Jacoco is being used. test reports are being generated for each subproject on unit, integration and compatibility test level. The toplevel Jacoco task combines reports from every subproject. Hopefully you’ll find some inspiration in there.
Looks like my first issue about the code coverage was that an unit test was covering up the real issue. The bug/issue is https://github.com/koral--/jacoco-gradle-testkit-plugin/issues/17. I was using Gradle 6.5, down grading to 6.4.1 for now until the bug is fixed. Thanks for the link to the build script your using.
I’m still at a loss about how to test default dependencies and make them work for a Gradle plugin. I followed the steps but it always seems to use the default dependency version.
I looked at that plugin and remembered there was a bug in the kotlin DSL with testkit. Normally if you want code coverage with testkit you just have to run GradleRunner using withDebug(true). It fails for Kotlin DSL. I’m not sure if it has been resolved in the later version of Gradle. That is the reason why GradleTest also turns off debug mode when it sees a build,gradle.kts file in the test set.
BTW there is a cheap way of testing dependencies without resorting to TestKit. Set up a normal unit tests using ProjectBuilder instead. Call project.evaluate() and then do your assertions.
Yeah I was using ProjectBuilder as well, doing that caused the error message for jacoco-gradle-testkit-plugin to be hidden. I figured out the default dependencies issue, it is very important you use " in your dependencies when overriding the default version.
Sorry to bring this old thread up. I just wanted to know if this is still the case today.
My issue is that I do want to validate coverage with testKit, but I need to inject environment variables to my build and withDebug cannot be used with withEnvironment
Error message:
org.gradle.testkit.runner.InvalidRunnerConfigurationException: Debug mode is not allowed when environment variables are specified. Debug mode runs 'in process' but we need to fork a separate process to pass environment variables. To run with debug mode, please remove environment variables.
It is still true that using withDebug(true) would work to get coverage as the tests are run in-process, but it is also true that you can then for example not use withEnvironment.
I actually set this up some days ago to work properly without needing to do withDebug.
I basically do something very similar to what the plugin linked in OP here is doing, I just didn’t like too much how it is done there and had needed infrastructure already anyway, that is a base class for the functional tests that prepares a base build I extend in the individual tests, so was able to just write the gradle.properties there as needed.
I
create a resolvable configuration jacocoAgentJar
add as dependency org.jacoco:org.jacoco.agent:...:runtime
set jacocoAgentJar.singleFile.absolutePath as the value of a system property for the test task
set the<JacocoTaskExtension>().destinationFile!!.absolutePath as the value of another system property for the test task
add a doLast { ... } action to the test task that gets the read lock on the JaCoCo result file to wait for the test having written to it like
doLast {
// wait for the read lock on the file, otherwise the daemon might still
// have the write lock and Gradle complains that it cannot snapshot
// the file as output of this task and as input of the report task
FileChannel.open(jacocoDestfile.toPath(), READ).use {
it.lock(0, Long.MAX_VALUE, true).release()
}
}
and in the base class for the tests I prepare the gradle.properties file with
Actually, it just failed naturally.
If your tests started 5 daemons, those 5 daemons try to get the write lock, and the task doLast { ... } tries to get the read lock.
Nothing guarantees that the write locks are granted first, so while reduced, it still happens that the file is not accessible as it is currently written to by one of the daemons.
So I now changed it a bit.
I still request the read lock on the file to be extra sure, but I changed the agent string to
And then in the settings script of the projects under test I always add
import java.lang.management.ManagementFactory
import javax.management.ObjectName
abstract class JacocoDumper : BuildService<BuildServiceParameters.None>, AutoCloseable {
override fun close() {
val mBeanServer = ManagementFactory.getPlatformMBeanServer()
val jacocoObjectName = ObjectName.getInstance("org.jacoco:type=Runtime")
if (mBeanServer.isRegistered(jacocoObjectName)) {
mBeanServer.invoke(jacocoObjectName, "dump", arrayOf(true), arrayOf("boolean"))
}
}
}
val jacocoDumper = gradle.sharedServices.registerIfAbsent("jacocoDumper", JacocoDumper::class) {}
jacocoDumper.get()
gradle.allprojects {
tasks.configureEach {
usesService(jacocoDumper)
}
}
Now the writing of the JaCoCo data is done at the end of the build still within the SUT build run, so definitely before the file is then accessed by the outer build.
I think this should workout better.
For me, it seems to work perfectly so far, even though I’m testing my plugin with all minor Gradle versions beginning with 6.0, combined with each of Java 8, 11 and 17… so the setup is quite complicated.
What was an important catch for me is that doLast { ... } actions don’t get executed when any of the tests fail. (That was a long debugging session. ) What does work even in that case is a test listener (AbstractTestTask.addTestListener) that implements TestListener.afterSuite.
As I said, with only that it is less flaky, but still flaky, especially if multiple Gradle daemons were started, for example when running tests in parallel or on different Gradle versions.
So not dumping on exit, but doing it from within the inner test as part of the test is better as then you know it is done before the task finishes and tries to read the file.
What was an important catch for me is that doLast { ... } actions don’t get executed when any of the tests fail.
That’s correct. For me this is irrelevant, as I also do
to have an implicit task dependency from the jacoco report task to the test task.
That means the jacoco report task is anyway not executed if the test task failed.
But with the dumping as part of the test as I wrote in my later comments, this is also not a problem, as the build agent close code will also be run on a failed build and will always run as part of the test, so doesn’t care whether there are test failures or not, so if your jacoco report task does not depend on the test task, that should also work better for you then. Using a test listener should have the same flakiness my doLast still has without the inline dumping.
I read your earlier comment saying that, but I don’t understand how it can still be flaky. Do you perhaps have multiple Test tasks, and jacocoTestReport doesn’t depend on all of them?
The fact that your code is in a doLast block, or in my case, in an afterSuite callback, should in fact guarantee that the write lock is granted first, because that is requested during the main test task, no?
Or are you saying that the TestKit JVM could start up without getting any lock on the file, then the test cases run and finish, then the Gradle test task ends, so Gradle executes the callback and gets the read lock (that it releases right away), and only after all of this does the TestKit JVM even start shutting down, running its own termination hooks, that cause the write lock to be requested?
What you described last could be one of the cases.
Another case - and no, just talking about one test task - is, that multiple Gradle daemons are started during the test.
This can for example be the case when you enable parallel test execution for Spock or Jupiter and thus for example run 7 tests in parallel so 7 daemons running.
Or if you run tests against different Gradle versions, also different daemons need to be started.
Now let’s assume the last 7 tests finished and those 7 daemons are idling.
Let’s also assume the daemon shutdowns are initiated immediately when the tests finished (although I don’t know, but any other case would just be worse, so assume the best possible here).
Let’s also assume that it is not waited for the daemons to completely shut down (which is a safe assumption or the whole problem here wouldn’t exist).
So now we have 7 daemons requesting and waiting for the write lock on the JaCoCo data file.
The first daemon gets it and writes its stuff.
In the meantime the doLast is run and now also waits for the write lock.
When the first daemon is finished with writing, someone else gets the lock which could also be the doLast which immediately releases it again and the next daemon takes the lock, making the file again unreadable for Gradle just as in your last described case.
And this is not a theoretical discussion. I still had this problem appearing even with the doLast, so this is a real problem happening and could theoretically even happen with only one daemon in play if it is not fast enough to acquire the write lock.
The tactic with the inline dumping immediately after each test indeed is the safest option and with that I never had problems again so far.
I believe Gradle TestKit daemons are shut down by the DefaultGradleConnector.close() call, which is executed by a JVM shutdown hook that gets registered in ToolingApiGradleExecutor.maybeRegisterCleanup.
Closes the tooling API, releasing all resources. Blocks until completed.
May attempt to expire some or all daemons started by this tooling API client. The exact behaviour here is implementation-specific and not guaranteed.
The expiration is best effort only. This method may return before the daemons have stopped.
Note: this is not yet part of the public tooling API yet.
Not sure whether this helps anyone, but added it for the record.
For anyone struggling with the Jacoco on the Gradle 8.7 release, you can use the Björns Kautler patch that makes the MCVE work on the 8.7 branch. The code can be found here GitHub - tomkoptel/jacoco-gradle-testkit
) What does work even in that case is a test listener (