from time to time we got problems running gradle on our CI builds:
When a test class is removed, the compileTestJava is triggered. However, the test target still has this class in the classpath, so it is either up-to-date or it will run the deleted test under build/classes/test.
When deleting the whole src/test folder, the compileTestJava is up-to-date due to
Skipping task ':project....:compileTestJava' as it has no source files.
. In this case the test runner would run all the deleted tests.
The obvious workaround would be to do a clean first, but this would kill the incremental speed-up.