Hi,
I just ran into an issue that I think should be handled differently by gradle.
In one project I have multiple tests divided into differet classes (they use Spock).
Some of there require certain objetcs to be mocked whereas others need the real objects.
As far as I’ve been able to tell since (in my case) a Mocked test is started first all instances of the Object is mocked.
Using 1 JVM per test would be to expensive and slow down testing dramatically. JUnit and TestNG are designed to be able to run multiple tests in a single JVM.
I bet your tests classes are doing something funky and imposing the issue your seeing. If you are using Mock(Foo) to create your mock in some of the test classes then you should be able to use real classes when needed.
Hi,
What I have is in test class 1:
def foo = Mock(Foo) <-- Foo is a singleton
-> All pass
in test class 2 (in separate file and namespace but same project)
def bar = bar.function_using_foo() -> Passes if run independently but fails when all tests are being executed
This is where foo is a Mock object.
All tests are using Spock as the test framework.
I think this might be more of a StackOverflow question. Without knowing anything about your project I would make sure you don’t have any static state in any of your classes.
Hi,
Still think that the behaviour of running each project in the same JVM is not the expected behaviour.
But your input pointed me in the right direction.
The singleton in question has an instance variable. By combining MockDetector and setup() I can make sure that the object is of the type I expect in each test.
/Cheers
Your tests should use whatever you want them to use. The build tool gives default settings. If you don’t like them, change them to something that suits you better. The major build tools in the Java ecosystem have different defaults, but I don’t think any of them default to what you’re wanting, so for purposes of comparison:
Ant has fork (on/off, default: off) and forkmode (perTest/perBatch/once, default: perTest). You would have to turn fork on and leave forkmode as perTest for 1 JVM per test class.
Maven SureFire has forkCount (#, default: 1) and reuseForks (true/false, default: true). You would have to set reuseForks to false for 1 JVM per test class.
Finally, Gradle always forks tests, but has a forkEvery setting. If you want a new JVM for every test class in a project, set forkEvery to 1.
As far as what Gradle should do, the default settings run fast and if you have an issue that requires doing something slower for test isolation problems or memory settings, you can make that change. Defaulting to settings that would cause poor build times with no advantage in many scenarios would not be a sensible default.
As @ethankhall mentions, it is a dramatic slow down to use a new JVM for each test. I’ve been involved with projects where the execution times of tests could be increased from 3 minutes to 2 hours 12 minutes by enabling 1 JVM per test. However, even with the ability to do this, fixing the few test isolation issues was a much better option than waiting an extra two hours for every build to complete.