Test task fails since Gradle v1.11: resources are not in test classpath after method under test throws an exception

Old Forum
1

Hi,

There seems to be a bug at least since Gradle v1.11.

I have to use a file (i.e. load it if it was not already loaded before) in getMessage() of a custom-exception. This exception is thrown in the method under test. With Gradle v1.10 it worked fine. Since v1.11 the file is not found in the classpath when I try to load it in getMessage().

My build.gradle-file looks as simple as that:

apply plugin: 'java'
  sourceCompatibility = 1.7
version = '1.0'
  repositories {
    mavenCentral()
}
  dependencies {
    testCompile group: 'junit', name: 'junit', version: '4.11'
}
  test{
    ignoreFailures = true
    testLogging {
//
      exceptionFormat "full"
    }
}

I just executed:

gradle test

(tested with v1.9, v1.10, v1.11, v1.12, v2.2, v2.2.1)

I have provided a Java-testcase here: http://gist.github.com/fortySixAnd2/a032da56acc2d5c7e116

MyClassTest.java is the test-class and the resource/file is loaded in MyLogWrapper.java which is invoked due to an exception in MyClass. Of course this is simplified example and it looks a little bit strange, but you can reproduce the issue easily.

First I thought there is only 1 issue, but now I think there are 2 issues with Gradle since v1.11:

1st issue - as described before)

when I use ignoreFailure=true and testlogging.exceptionFormat ‘short’, I see in the stacktrace of the test-report that my file wasn’t found. => gradle output: BUILD SUCCESSFUL (since ignoreFailure = true)

2nd issue)

when I use ignoreFailure=true and testlogging.exceptionFormat ‘full’, an exception occurs in the gradle code, no test-report will be generated and gradle output is BUILD FAILED

----- with Gradle v1.10 and also in IntelliJ/Eclipse (i.e. with the IDE JUnit-Runner) I get the expected result: my file was found and MyException is in the stacktrace with exception-message ‘someMessage’ and BUILD SUCCESSFUL (since ignoreFailure = true)

Are there any workarounds for these issues in Gradle v1.11 (or newer)?

Thanks & best regards, Manuel

2

Can you please explain why you are using thread context classloader to load that resource?

3

The resource as well as all those classes are in a jar which is packaged in a war. In that war there are also other jars which are using this MyException.class.

The war is deployed on an application server (in my case JBoss7).

So loading this resource via classloader is the only way which works on AS - as far as I know…

Regarding the 2nd issue I mentioned: Is this is an expected behaviour? - I mean I wouldn’t expect that setting testlogging.exceptionFormat ‘full’ (in combination with ignoreFailure=true) leads to a failing build since in my case MyException is still in the context of the class under test.

Thanks, Manuel

4

No, what you describe as the 2nd issue is not the expected behaviour. Can you please provide a reproducible sample so that I can have a look?

5

Sure:

https://github.com/fortySixAnd2/gradletestcase

With shortExceptionFormat.zip you can reproduce the 1st issue, with fullExceptionFormat.zip the 2nd one.

Just execute build-task

As already mentioned in both cases I would expect a sucessful build with 1 failing test with exception-message ‘someMessage’ in the stacktrace.

Thanks, Manuel

6

So the problem is that you are essentially throwing an exception not from the method under test but from ‘MyException.getMessage()’ which is not called anywhere in your test. When you set ‘exceptionFormat’ to be ‘“short”’ then ‘MyException.getMessage()’ never gets called. If you set it to ‘“full”’ though, gradle will call ‘getMessage()’ on the exception when showing the exception details as you’ve asked it for. This is not in context of test anymore but in context of the build. I would suggest moving the message initialisation from ‘getMessage()’ to the constructor of ‘MyException’. You will then get a report like this and a successful build:

com.example.MyClassTest > testMyMethod FAILED

com.example.MyException: someMessage

at com.example.MyClass.methodToTest(MyClass.java:8)

at com.example.MyClassTest.testMyMethod(MyClassTest.java:9)

7

Thanks again for looking into this. Oh yes, you are right that the getMessage()-method won’t be called directly in the method under test and I also agree with you that loadint this resource should be moved instead of directly loading it in the getMessage-method. This (adapted) code is from a legacy application we are migrating from Ant to Gradle. And it was done in order to support lazy loading of the resource. But I still have some questions related to your explanation:

1.) You say that with ‘exceptionFormat short’ the getMessage() won’t be called. But why do a I see the IllegalStateException with message “Resource file abc.txt not found!” (instead of MyException with ‘someMessage’) in the test-report (see my testcase ‘shortExceptionFormat.zip’ above). So it seems with ‘short’ Gradle also calls getMessage. Sorry for asking again, but I still don’t understand it: when the getMessage get’s invoked by Gradle why are my classes in this seperate (?) classloader, but no resources anymore - just a optimization by Gradle?

2.) regarding ‘exceptionFormat full’: I would still expect that Gradle catches such situations (without setting the build to failed :wink: ) and it seems that this was the case until version 1.10. I mean in getMessage also just a NPE or other Exceptions may occur.

3.) As already mentioned I agree with you that the initialization should be moved e.g. to the constructor of MyException. But for me it sounds a little bit contradictory to slogans like ‘Gradle adapts to your needs’. Because now I have to change my implementation. Otherwise with Gradle the tests will always fail. Or I switch to an older Gradle version - but I’ve to know which version.

Furthermore let’s consider the situation where I’m using a 3rd party library where I’m not able to change the code…

Don’t get me wrong, I like Gradle very much, but I just want you to maybe think about this.

Thanks, Manuel

8

Yeah, after thinking about it we should be more consistent about not failing for short format and failing for full, I’ll raise an issue for this. We will have to figure out what the exact behaviour will be, but the key is that it should be consistent.

With the classloading it’s not an optimisation. It’s a bit complex because there are two VMs involved here. One is the build VM and one is the test VM (the one in which the tests are executed). You are throwing the exception in the test VM and it then gets sent over the wire to the build VM in which ‘getMessage()’ is called for which the context class loader is different. If you followed Joshua Bloch’s advice from Effective Java, Item 63 and initialised the message in the constructor then it would work, even with loading the resource off the context classloader.

9

https://issues.gradle.org/browse/GRADLE-3237