I notice that if I set a property in my test configuration as below, it is not available to my TestNG @BeforeSuite method. Is this expected behavior?
(re-edit - changed test to task the extends Test; which is where i am really seeing this)
task integrationTest(type: Test) {
useTestNG() {
includeGroups 'integration'
}
systemProperty "foo", "bar"
}
public abstract class Base
{
@BeforeSuite
protected void suiteSetup()
{
System.out.println(System.getProperty("foo", "MISSING")); // Will print "MISSING"
}
}
public class Sub extends Base
{
@Test public void dummy_test() { assertTrue(true); }
}
changed test to task the extends Test
I don’t understand what this means.
sorry, I re-edited this post to make my task of type “Test”. My original post, was just a basic
test {
useTestNG()
systemProperty "foo", "bar"
}
Just ignore that comment i guess.
The following works just fine for me:
build.gradle:
apply plugin: "java"
repositories {
mavenCentral()
}
dependencies {
testCompile "org.testng:testng:6.8"
}
test {
useTestNG()
systemProperty "foo", "bar"
testLogging.showStandardStreams = true
}
src/test/java/Base.java:
import org.testng.annotations.*;
public abstract class Base
{
@BeforeSuite
protected void suiteSetup()
{
System.out.println(System.getProperty("foo", "MISSING"));
}
}
src/test/java/Sub.java:
import org.testng.annotations.*;
import static org.testng.Assert.*;
public class Sub extends Base
{
@Test public void dummy_test() { assertTrue(true); }
}
# prints 'bar'
$gradle clean test
What if you do this instead. I am able to reproduce this behavior consistently. When i use the regular “test” task, the property is defined, but when I extend the Test task, it is not getting the property.
task integrationTest(type: Test, dependsOn: test) {
useTestNG() {
includeGroups 'integration'
}
systemProperty "foo", "bar"
}
@Test(groups = {"integration"})
public class Sub extends Base ...
$ gradle clean integrationTest
Not sure what you mean by “extend the Test task”, but when you declare your own ‘Test’ task, you’ll have to explicitly configure ‘classpath’, ‘testClassesDir’, etc. See the Gradle Build Language Reference and the ‘java/withIntegrationTests’ sample in the full Gradle distribution. As your task declaration stands, it won’t run any tests.
I based this strategy off the following blog post
We have been using the method outlined in that blog for specifying integration tests for a while and it has worked fine.
I put together a small example on github at: example project
What i noticed is that system properties defined in “test” are available to our both our regular tests and integration tests. However, no system property defined in the integrationTest task is available to our integration tests.
What is odd is that
- my integrationTest task is executing (it works) 2) I can set some settings like includeGroups for testng 3) But I cannot configure other aspectes of my integration test task (e.g. systemProperty)
So you use the same source set and dependencies for unit tests and integration tests, and just separate them via includes? In that case, your configuration might work.
What i noticed is that system properties defined in “test” are available to our both our regular tests and integration tests. However, no system property defined in the integrationTest task is available to our integration tests.
This is a strong indication that your integration tests are run by the ‘test’ task, not the ‘integrationTest’ task. I can’t think of any other explanation. Have you configured ‘excludeGroups’ for the ‘test’ task?
Yes, my test task is defined as
test {
useTestNG() { excludeGroups 'integration' }
systemProperty "t", "test"
}
task integrationTest(type: Test, dependsOn: test) {
useTestNG() {
includeGroups 'integration'
}
systemProperty "i", "integration"
testLogging.showStandardStreams = true
}
As you can see, I am excluding integration tests from running when the normal ‘test’ task executes.
It seems like there is some sort of bug with respect to they system properties when you create a task of type ‘Test’. the ‘useTestNG()’ configuration is being honored. And so is the ‘testLogging.showStandardStreams’ settings.
It seems like there is some bug in the handling of system properties in this case. Would you agree? I feel that what I am doing is standard Gradle - I am simply configuring a task of a particular type.
If I had a task like
task mySync(type: Sync) {
caseSensitive = false
// other configuration
}
And then it did not honor the ‘caseSensitive’ property, wouldn’t this be considered a bug. How is what I am doing different?
As I already said, the only explanation I have is that your ‘test’ task also runs the integration tests. This should be easy for you to verify.
I updated my example project to verify that the ‘test’ task does not also run the integration tests. I added a ‘doFirst’ closure to print if the task is executed.
test {
doFirst {
println "test task executing"
}
useTestNG() { excludeGroups 'integration' }
systemProperty "t", "test"
}
task integrationTest(type: Test, dependsOn: test) {
doFirst {
println "integration test task executing"
}
useTestNG() {
includeGroups 'integration'
}
systemProperty "i", "integration"
testLogging.showStandardStreams = true
}
The ‘test’ output is
$ gradle clean test
:clean
:compileJava UP-TO-DATE
:processResources UP-TO-DATE
:classes UP-TO-DATE
:compileTestJava
:processTestResources UP-TO-DATE
:testClasses
:test
test task executing
t = test
i = no i
BUILD SUCCESSFUL
Total time: 1.995 secs
The ‘integrationTest’ output is
$ gradle clean integration
:clean
:compileJava UP-TO-DATE
:processResources UP-TO-DATE
:classes UP-TO-DATE
:compileTestJava
:processTestResources UP-TO-DATE
:testClasses
:test
test task executing
t = test
i = no i
:integrationTest
integration test task executing
Gradle test > Sub.integration_test FAILED
java.lang.AssertionError at Sub.java:16
2 tests completed, 1 failed
:integrationTest FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':integrationTest'.
> There were failing tests. See the report at: file:///home/mgiannini/dev/java/integrationTestExample/build/reports/tests/index.html
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
BUILD FAILED
Total time: 3.032 secs
I can’t reproduce your problems. Your GitHub project works just fine for me. The way ‘integration_test’ is written, it is bound to fail, because it tests for both system properties. When it is changed to only test for the ‘i’ system property, it passes as expected.
Other things I noticed:
- The ‘suiteSetup()’ method is only run by the ‘test’ task (I guess because it doesn’t have a group set). * The order of arguments in ‘assertEquals’ is wrong. Maybe that’s what leads you to the wrong conclusion. * Since ‘integrationTest.testReportDir’ isn’t configured, the HTML reports of the two test tasks will overwrite each other. * I recommend to remove the task dependency from ‘integrationTest’ on ‘test’ because there is no semantic dependency between them. This will also make it easier to observe what’s happening.
First, let me thank you for sticking with me on this. I think my ultimate problem is due to your first observation:
The suiteSetup() method is only run by the test task (I guess because it doesn’t have a group set).
I wrongly assumed that testng would always run annotated methods that did not explicitly give the set of groups. I think i need to re-think how we group our tests.
The order of arguments in assertEquals is wrong. Maybe that’s what leads you to the wrong conclusion.
You are right that my assertEquals arguments are backwards, but I don’t think that matters in this case.
Since integrationTest.testReportDir isn’t configured, the HTML reports of the two test tasks will overwrite each other.
We actually have a task for generating a test report that seems to properly join the outputs of the two tests together:
task testReport(type: TestReport, description: 'Generate test report for all subprojects') {
destinationDir = file("$buildDir/reports/allTests")
reportOn javaProjects*.test
reportOn javaProjects*.integrationTest
}