Behavior of includeTestsMatching with custom JUnit test engine

Help/Discuss
1

Hi,

I notice a weird behavior of the includeTestsMatching method used to filter tests, when I use a custom JUnit test engine. It seems to not support wildcard correctly in this case (it works fine with the JUnit Jupiter test engine).
I reproduced with an almost empty test engine which only traces the requested tests in the discover method:

    @Override
    public TestDescriptor discover(EngineDiscoveryRequest request, UniqueId uniqueId) {
        System.out.println("Discovering tests for My JUnit test engine");

        EngineDescriptor engineDescriptor = new EngineDescriptor(uniqueId, "My TestSuite");

        request.getSelectorsByType(ClasspathRootSelector.class).forEach(selector -> {
            System.out.println("Discovered ClasspathRootSelector " + selector.getClasspathRoot());
            // Register test...
        });

        request.getSelectorsByType(PackageSelector.class).forEach(selector -> {
            System.out.println("Discovered PackageSelector " + selector.getPackageName());
            // Register test...
        });

        // Gradle provides ClassSelector selectors only
        request.getSelectorsByType(ClassSelector.class).forEach(selector -> {
            System.out.println("Discovered ClassSelector " + selector.getClassName());
            // Register test...
        });

        request.getSelectorsByType(MethodSelector.class).forEach(selector -> {
            System.out.println("Discovered MethodSelector " + selector.getMethodName());
            // Register test...
        });

        return engineDescriptor;
    }

I have 2 test classes in my project: org.example.MyFirstTest and org.example.MySecondTest. With the following test configuration:

testing {
    suites {
        val test by getting(JvmTestSuite::class) {
            useJUnitJupiter()

            targets {
                all {
                    testTask.configure {
                        useJUnitPlatform {
                            includeEngines("my-test-engine")
                        }
                        filter {
                            includeTestsMatching("*MySecondTest*")
                        }
                    }
                }
            }
        }
    }
}

The discover method receives the 2 test classes:

Discovered ClassSelector org.example.MyFirstTest
Discovered ClassSelector org.example.MySecondTest

whereas if I change the pattern with includeTestsMatching("*MySecondTest*"), it works fine:

Discovered ClassSelector org.example.MySecondTest

I attached the sample project (64.9 KB) which reproduces the issue. It can be tested by executing the test task (the build fails because no test is registered but the problem is before).

I am not sure where does the issue come from, does Gradle send the wrong test classes? Is there something to add in the custom test engine?

Regards.

2

I’m not sure I can follow.
In your description the posted code and the “change the pattern with” are identical.
And in the sample project if either of the three filter lines is present, the discovery is not done at all, only if all three lines are commented out both tests are discovered.

3

Ah, I remember what probably is the problem and how you probably meant your question.
When using the concrete class without wildcard, Gradle can directly match the class name and give that to the discovery request.
With the wildcard it first lets the engines discover the tests and then uses a post discovery filter to filter out the unwanted tests.
So if you would actually execute the tests, it would probably behave as expected.

4

In your description the posted code and the “change the pattern with” are identical.

My bad, the second pattern should have been includeTestsMatching("MySecondTest").

When using the concrete class without wildcard, Gradle can directly match the class name and give that to the discovery request.
With the wildcard it first lets the engines discover the tests and then uses a post discovery filter to filter out the unwanted tests.
So if you would actually execute the tests, it would probably behave as expected.

That’s not exactly what I observe but that’s interesting. When I use includeTestsMatching("*MySecondTest*"), the 2 tests are discovered and executed. But only the class MySecondTest is displayed as executed in the JUnit HTML report. That tends to confirm that there is post discover filter to implement in the discover method of our test engine. I will look that way, and also have a look at the Jupiter test engine to check how it is done in this engine.

Thanks!

5

iirc you should not need to do this manually.
Are you sure both tests are executed?
In your MCVE you only logged the discovery.
That the discovery finds both is expected.
The post discovery filter should be taken care of by the JUnit Platform, not by your engine.

6

I enriched a bit my example (65.3 KB) to list the executed tests and I noticed that the 2 test classes are discovered and registered, but the test method of the MyFirstTest class is not executed when using includeTestsMatching("*MySecondTest*"):

    Discovering tests for My JUnit test engine
    Discovered ClassSelector org.example.MyFirstTest
    Discovered ClassSelector org.example.MySecondTest
    Executing tests for My JUnit test engine
    Execute test class org.example.MyFirstTest
    Execute test class org.example.MySecondTest
      - Execute test method test1

whereas it is when I remove the includeTestsMatching:

    Discovering tests for My JUnit test engine
    Discovered ClassSelector org.example.MyFirstTest
    Discovered ClassSelector org.example.MySecondTest
    Executing tests for My JUnit test engine
    Execute test class org.example.MyFirstTest
      - Execute test method test1
    Execute test class org.example.MySecondTest
      - Execute test method test1

So there seems to be a filter in the registered TestDescriptor before the call of the execute method.
In my custom test engine, I rely only on the classes, that’s probably why I didn’t see this post-filter. I will check that.

7

Well, the result is ok, isn’t it?
The descriptor for MyFirstTest is kept, but the descriptor for MyFirstTest#test1 is filtered out by the Gradle post discovery filter.
Due to that you still get the output for “Execute test class”, but not the output for “Execute test method”, as the test is not executed.

Btw. you might want to set ClassSource.from(javaClass) as third parameter in the super call in JunitClassTestDescriptor.

8

That was not ok in my case because I relied on the classes, not the methods. We have a specific use case, we have a pre-processor which wraps test classes in ‘main’ classes, then the test engine executes each ‘main’ class (the main classes are transpiled and compiled to native binary to be executed on embedded devices).
Intuitively I thought that the discover method would receive only class TestDescriptor if it contains at least one test. Now that I know how it works I am able to adapt my test engine to fix the issue.

Thanks for your help!

9

On JUnit Platform, there is no such thing as “class” and “test” like you envision.
There are only TestDescriptor which can be “container”, “test”, or “container and test”.
There can be any arbitrary nesting level of containers in containers in containers with in the end having tests or also being tests.
And the engine can even dynamically at runtime add further test childs.

Having only one level of pure containers that are classes with one level of childs that are tests,
is a pure decision of the test engine … or not.

With JUnit Platform, you can even generate test containers and tests out of thin air, or by reading some CSV values, or by requesting the user to enter input, or by reading from a database, or whatever the engine likes to do to make up how the test hierarchy looks like.

10

Of course such test engines are most probably not really compatible with using test filters or --tests on Gradle or other integrations, and the whole topic is more a question for some JUnit community, than for the Gradle community. :slight_smile: