Test groups for JUnit and specs2

Old Forum Archive
1

I’m trying to introduce test groups into an existing code base in order to organize promote testing. The goal is for developers to not focus on what type of test they’re writing, but on when the test is executed. I’d like to use the convention,

  • Fast: tests that are run during the build process (default group)

  • Slow: tests that are typically run only on Jenkins after a commit, e.g. integration testing between a node.js front-end and Java backend servers

  • Daily: tests that are typically run only on Jenkins once per day, e.g. alpha testing against Chrome Dev Channel for detecting incompatibilities

  • ???: other groups may be required in the future, e.g. Flaky, but I’d prefer avoiding that. Scenarios like performance regression tests tend to belong outside of a typical xUnit framework.

Using the build-per-branch Jenkins plugin and the Github Commit Status API, each time a developer pushes to their branch a job is triggered. That promotes testing by making it easy to get feedback asynchronously during development and reviewers of a pull request can see the build status before approving.

Currently we have a mix of testing frameworks, each of which have similar groups mechanisms:

  • JUnit for legacy integration tests - categories

  • TestNG for new Java - groups

  • specs2 for Scala - tags

  • qunit tests for node.js - modules (uses the grunt build tool)

  • maybe ScalaTest when helpful (e.g. for the WebDriver DSL) - tags

Naively this seems possible to do using classpath scanning and a group configuration, like the TestNG DSL. However I’m running into a few problems that I haven’t figured out how to resolve: - JUnit doesn’t have a classpath suite runner that supports categories. It looks like the prototype was never released. (should I try to resurrect it?) - Gradle’s JUnit options don’t support categories (Maven’s Surefire plugin does), so not sure how to integrate w/o dedicated suites. - Long term specs2 / ScalaTest support would be nice so that tags are used instead of relying on the JUnit adapters.

I’d prefer not having to separate test groups into different directories or use a naming convention as a work around. Both seem to focus developers on deciding the type of test instead of when their run, which causes more refactoring churn as slow tests become fast (e.g. replacing MySQL with H2) or vice versa.

Is there an integration trick that I’m missing? If not, what is the preferred approach?

Thanks!

2

A common approach is to use different source sets and test tasks for different kinds of tests (unit, integration, etc.). The test tasks will often differ in their configuration (external dependencies, memory settings, forkEvery, maxParallelForks, etc.).

As far as I know, the only way to tell JUnit which categories to run is with JUnit suites. But maybe Maven implemented their own filtering mechanism for categories.

3

Thanks Peter.

I wasn’t able to get JUnit categories to work with the ClasspathSuite. Luckily since this is only needed by legacy tests a naming convention is a reasonably concession.

It turns out that specs2 can use system properties to pass arguments through the JUnit runner to control test execution. This allowed me to get tags and sections to work, so my fear that specs2 would be limited by JUnit was unfounded.

I have some prototype test tasks for junit / testng / specs2 that all work with my grouping scheme, so I think my concerns are pretty much resolved.