Unit test fail when I run "gradle test" but run in Intellij successfully

Help/Discuss
1

I wrote a project, which I use testng to test it. It can be built successfully.
When I open this project in Intellij, and switch to each single unit test file. All the test can be passed.
But when I was trying to run “gradle test --info” in the command line, all the unit tests failed.

I don’t why they can pass in the intellij, but failed in command line. I was trying to check the error message. There is nothing special here. Just some message said that the test failed because the assert statement.

Could someone tell me how to fix this? Pretty weird.

> Task :parseq-lambda-names:testClasses
Skipping task ':parseq-lambda-names:testClasses' as it has no actions.
:parseq-lambda-names:testClasses (Thread[Task worker for ':',5,main]) completed. Took 0.0 secs.
:parseq-lambda-names:test (Thread[Task worker for ':',5,main]) started.
Gradle Test Executor 19 started executing tests.
Gradle Test Executor 19 finished executing tests.

> Task :parseq-lambda-names:test FAILED
Task ':parseq-lambda-names:test' is not up-to-date because:
  Task has failed previously.
Starting process 'Gradle Test Executor 19'. Working directory: /Users/jguan/Desktop/fixbug/parseq_trunk/parseq/subprojects/parseq-lambda-names Command: /Library/Java/JavaVirtualMachines/jdk1.8.0_181.jdk/Contents/Home/bin/java -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager -Dorg.gradle.native=false -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant -ea -cp /Users/jguan/.gradle/caches/4.8/workerMain/gradle-worker.jar worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Test Executor 19'
Successfully started process 'Gradle Test Executor 19'

Gradle suite > Gradle test > com.linkedin.parseq.lambda.TestInterface.testAbstractFunctionInvocationOnInterface FAILED
    java.lang.AssertionError: expected [true] but found [false]
        at org.testng.Assert.fail(Assert.java:94)
        at org.testng.Assert.failNotEquals(Assert.java:496)
        at org.testng.Assert.assertTrue(Assert.java:42)
        at org.testng.Assert.assertTrue(Assert.java:52)
        at com.linkedin.parseq.lambda.TestInterface.testAbstractFunctionInvocationOnInterface(TestInterface.java:87)
1 Like
2

@jjustinic Could you take a look at this? Many thanks.

3

Possibly caused by state from a previous test bleeding into another (eg mutable static variables). Have you tried running the single test from gradle command line? Or are you always running all tests?

Another possibility is a classpath difference. Perhaps you can get the same failure in intellij by ticking “separate module per source set”

4

@Lance
Thank you so much for your reply. When I run the test, I always do the clean first. And I tried running the single test from gradle command line as you said. It still failed.

Then I tried do “separate module per source set”, somehow, I can’t build the whole project by running “build project” in Intellij(but I can build the whole project by running “gradle build” in command line). Since this is a multiproject, my submodule A depends on a test utility submodule B. I failed to run the unit test in submodule A.

Do you have any idea about this? After setting the project to “separate module per source set”, I can’t build now…

5

99% cases of such behavior is mutable classes state and execution order, because IDE and gradle execute tests in different order. Have you checked what exactly causes testAbstractFunctionInvocationOnInterface() assert fail and in which other test classes that variable is changed?

6

I was initially thinking the same (one test bleeding mutable state into another). I think we eliminated this possibility by running just a single test from gradle command line.

The single test fails from gradle command line and passes in intellij

1 Like
7

I am facing the same issue. Tests via Intellij are passing, but failing via gradle command and even on Jenkins(which runs same gradle command, but clean build). By any change, did you get it resolved? Any help is really appreciated

8

Ran into this issue as well. Works fine from IntelliJ running a single test or all tests in a class but fails when I try to build with gradle (e.g. gradlew clean build).

Found that setting forkEvery=1 allows the build to complete successfully. This is a work around to having some leaky tests. (https://docs.gradle.org/current/dsl/org.gradle.api.tasks.testing.Test.html#org.gradle.api.tasks.testing.Test:forkEvery)

test {
    forkEvery = 1
}

For debugging why it’s happening, I’ve setup a junit test suite to run multiple test classes in a row from within IntelliJ. https://github.com/junit-team/junit4/wiki/Aggregating-tests-in-suites

1 Like