Add support for integration tests


(Adam Murdoch) #1

Integration tests, functional tests, acceptance tests are all common features of a build. It would be nice to capture this in an ‘integration-test’ plugin and start establishing some conventions around this.

Implementation ideas

Such a plugin might do something like this:

  • Add an ‘intTest’ source set for the test source to live. * Wire up the compile and runtime dependency configurations for this source set. * Add an ‘intTest’ task and wire up its inputs and outputs.

(Andrew Oberstar) #2

You mention integration, functional, and acceptance tests, but then describe a plugin that only seems to address one. Would this plugin merely add one more set of tests for integration tests? It seems like it would be nice to have a plugin that could add additional tests of whatever name makes sense. An example of a tests DSL below.

tests {
  intTest extendsFrom test
  accTest extendsFrom test
  otherTest extendsFrom intTest, accTest
}

(Adam Murdoch) #3

The goal would be to end up with something where you could easily stamp out different types of test suites. Something like your example, above.

These ideas tagged ‘contribute’ are intended to be (relatively) simple to implement for new contributors to Gradle, where someone could jump in and quickly put together something useful.

In that spirit, I’d be happy with just capturing the idea of integration tests to start with. Even better if the plugin captures all the different flavors of tests.


(Paul Jimenez) #4

Isn’t it just a new task with different includes/excludes?

test {
    exclude "tests/**/integration/**"
}
  task intTest(type: Test) {
    include "tests/**/integration/**"
}

?


(Adam Murdoch) #5

It’s about more than this.

One part is about establishing some conventions around integration tests (eg if you put your integration tests under src/test/java in an ‘integration’ package, then they will be treated as integration tests rather than unit tests). And about defining where in the standard lifecycle these will run. Eg are they run when you do ‘gradle check’? Or is there another lifecycle task that runs the integration tests and unit tests?

Another part of this is to add some way to declare ‘this project has integration tests’, so that other plugins can do useful stuff. For example, perhaps the IDE tasks map the integration tests to a separate IDE project. Or the jetty plugin automatically builds and deploys the web app before running the int tests, injecting the URL to the web app as a system property. Or perhaps the application plugin could build and install the application ready for integration testing.

As far as implementation goes, it might be better to model integration tests as a separate source set, with separate compile and runtime dependencies, and separate source directory, as it is often the case that unit test and integration tests are fairly independent.


(Paul Jimenez) #6

I’m actually trying to do this now, though I’m not aiming so much for ‘unit’ vs ‘integration’ as just plain ‘fast’ vs ‘slow’ tests. One batch of our tests runs in about half an hour. Another batch runs in about 3 hours. Yet another batch takes about 8 hours. Just plain ‘test’ doesn’t really cover it. I’m currently considering the (admittedly hacky) method of having testFast, testSlow, and testVerySlow all set variables and then each test{} block in the subprojects have conditionals on those variables, but I’d love some kind of HOWTO or best practices covering this, since I’m new to gradle.


(aispina) #7

I’ve created a very immature plugin that does the minimum necessary. I needed to add integration tests to my project so I figured I’d try to make something a little reusable. It is hosted publicly here:

https://bitbucket.org/aispina/gradle-integration

Contributions and feedback welcome!


(Robert Ryals) #8

I agree with pjimenez3 on simply specifying different include/excludes. For our usage we have different permutations of TestNG groups and java packages that determine what integration tests we run. Fixing this bug - http://issues.gradle.org/browse/GRADLE-2307 would allow us to configure this external to our gradle script and drive it from our CI system. I do not like the idea of a build tool forcing you to use a specific source folder hiearchy or package name.


(aispina) #9

I think that we’re talking about two different build problems here. Separating tests into groups based on speed is important because developers don’t want to wait on a build. Separating tests by function is also important because different functionality often requires a different environment (for example my integration tests need to run against my WAR in a web container, while my unit tests have no external dependencies).

The problems are related because one way of implementing fast/slow separation is by making separate a source set for each speed. This is the natural approach to splitting by function. However, I agree with Robert and PJ that separate source sets aren’t desirable when splitting tests by speed. When splitting by speed, you want to easily be able to mark a test as one or another speed without having to move it on the file system. You also want to be able to support as many speeds as you’d like without having to have an arbitrary number of top level directories.

This doesn’t work well for a division of tests by functionality though. When dividing by functionality, I would expect a user to want to be able to group supporting files with the tests. For example, database files that are only used by integration tests.

In conclusion, I think there’s two problems here that deserve separate treatment. In functional splitting (which my plugin attempts to start addressing) you want separate sourcesets for your tests. In speed splitting, you want a series of include/exclude clauses to define groups of tests.

I’d like to suggest that we branch this discussion so that the two separate use cases can get the attention they deserve without interfering with each other.


(thomas.glaeser) #10

What is the status on a naming convention for integration test ‘sourceSets’/‘configurations’ vs. having a Java package ‘integration’ under the ‘test’ source set. It would be great if we could agree on those namings early on. As integration tests often need to be assembled as Java ARchives first, the separate source set approach seems to be clearer. Also, while camelCase notation works fine when in the gradle script, it should not matter at the filesystem level, in fact it would be better to use lower case only there.

So what will be the name for the integration test source set … ‘integrationTest’, ‘integrationtest’, ‘integTest’, ‘integtest’, ‘intTest’, ‘inttest’, ‘integration’, or something else? Looking at the Gradle 1.1 ‘org.gradle.build.integtest.IntegTestConvention’ class it seems a mix of some of those options.

Can we please agree on the naming conventions first? Thanks.


(Luke Daley) #11

If we had to pick a name for the source set, it would be ‘integTest’. For no other reason than that is what we use for the Gradle build.


(thomas.glaeser) #12

Thanks. What about the fs? Do you agree that cc notation is unusual for folder names?


(Luke Daley) #13

We haven’t had any issues with it in the Gradle project. It should definitely be the same as the name of the sourceSet.


(aispina) #14

In hindsight, my integration test plugin is a pretty poor start. Feel free to ignore it. Alas.


(Danny Yates) #15

When someone picks this up, they need to consider this:

http://stackoverflow.com/questions/12675080/how-to-cleanly-separate-two-instances-of-the-test-task-in-a-gradle-build


(Stig Kleppe-Jørgensen) #16

Re camel case on integration test folder names: Just remember that in some OSes there cannot be files that just differs in casing.


(Stig Kleppe-Jørgensen) #17

Re camel case on integration test folder names: Just remember that in some OSes there cannot be files that just differs in casing.


(Wujek Srujek) #18

Hi all. I have been messing around with Gradle recently, and did some initial work on this feature. The repository branch can be found here: https://github.com/wujek-srujek/gradle/commits/testing-plugin. The first commit contains the core classes that constitute the plugin. The main class, TestingPlugin, contains some ideas that I implemented this with, and also a few questions. This is targeted at Adam especially. The second commit contains an example project that uses the plugin and introduces some tests and configures them slightly. Feedback welcome.


(Adam Murdoch) #19

Here’s a rough spec of how I imagined we’d tackle this: https://github.com/gradle/gradle/blob/master/design-docs/test-suites.md

I think an important place to start is to introduce the concept of a test suite. Then, we can add a bunch of different ways of defining and constructing these test suites.


(Joachim Nilsson) #20

Adam. I have some suggestions for more use cases.

  • I want to measure code coverage for my integration tests * I want int test code coverage to be included in sonarRunner

  • In order to shorten the feedback loop, I want to have the possibility to control setup and test invocation so that i can distribute my integration tests towards a number of servers in my machine pool