I’m working on conversion of a large, legacy codebase from ant to gradle. One characterstic of this code base is that it as a “test project” that contains test code for other projects. Eventually we would like to be able to move test code out of the test project into the projects that are being tested, but this is not practical due to the fact that some of the tests are integration tests with lots of interdependencies and also just because of the size and complexity of the codebase. There is even code that knows how to find resources relative to where they are in the source tree, which sometimes means code changes are required to move two classes to separate directories. Basically it’s the usually kind of stuff you’d expect for a large, legacy codebase.
Our ant build has a concept of test collections that are defined using ant filesets and selectors. Typical collections do stuff like start with a directory, include some files based on file name patterns, subtract out files included by some other collection, do stuff based on a lexical search for certain tokens in the test files, etc. For example, here’s a simplified version of the kind of thing we do.
- collection
quarantinecontains a list of specific test classes that are known to be fragile in a specific order - collection
xyzcontains all classes matchingtest/src/**/*XYZTest.javaexcept those included in quarantine - collection
abccontains all classes matchingtest/src/**Test.javathat import something fromcom.example.abcas determined by some filter except those inxyzand those inquarantine - collection
otherfirst contains the test classtest/src/g/h/i/BreaksUnlessFirstTest.java, then contains everything in test/src/**Test.java except for all the other things previously defined
So it’s pretty ugly and has lots of workarounds for things like tests that are order-dependent, tests that are known to be slow but aren’t marked with any particular category, and stuff like that, and it is also full of inversions where we say everything here except stuff we already included somewhere else, and its hundreds of thousands of tests. The result is that it is quite complicated to unravel exactly why anything is where it is, but it is possible to make sure everything is in exactly one place without very much duplication of logic. This is achieved mainly through the use of inversions: subtracting out contents of another test collection.
We are trying to come up with some reasonable way of doing this in gradle. The general approach we are taking toward incrementally gradleizing this build is to define the conventions we want people to follow and then to create ways to override those conventions for legacy code, which is pretty much true to what gradle is all about. For lots of reasons, we’re not fixing the legacy code to force it into our conventions as part of the conversion process but are handling that as priorities allow once things are in gradle. If we do it right, then that process usually looks like refactoring the product code and correspondingly simplifying the gradle code.
In the case of our test collections, what we think we’re most likely aiming for is something where unit test code should ideally live with the project it’s testing, and our discrete collections of integration test code should come from separate sourceSets. But we can’t get there all at once and will likely never get all the way there for the legacy code.
I’ve looked at test filtering, but it’s very rudimentary right now. In particular, I can’t seem to find any way of excludng stuff. Even something like having a collection called “small” for all tests with the x.y.z.Small category and then another framework called “other” that contains all tests without a category seems hard to specify. I can definitely use test filtering to separate things by package, and I can use file inclusion and exclusion to probably hand-craft collections that look pretty much like our existing ones (with some extra stuff to try to preserve ordering where necessary), but I’m not seeing a good way to do this without duplicating logic. There’s not a way that I can see to define a test task that tests everything not tested by some other test task without having to explicitly create a pattern that matches everything not matched by the other tasks.
I’m looking for suggestions on how to proceed. These could include anything from some different way of using subprojects or sourceSets than I’m currently doing to hooking into or even extending the test filtering mechanism to make it possible to do what we want to some other idea I haven’t even considered. If there are things in gradle that could be modified to do what we want, given a little guidance, we would be open to possibly working on those changes and contributing them, depending on what level of resources we are able to commit to this problem. For example, being able to create test filters with similar flexibility to how you can create configurations would be great. That way we could do something like
x = somethingRepresentingAllTestsInTheTestConfiguration
y = tests with category small
z = x - y
or however something like that would look. Right now I’m not even sure how to retrieve the list of tests that a particular test task would run or even whether this is known in advance.
How much of what I want is already there and I haven’t found it yet? How much is not there but could be added without too much trouble? Or am I just going about this the wrong way? Thanks for any input.