Is there a way to configure build to fail on skipped tests when using testng?


(dougy-gradle) #1

I’m working on porting a maven build I have, and I’m running into problems in the test phase when running testng tests.

As a background, my TestNG tests all extend spring-test’s AbstractTransactionalTestNGSpringContextTests, which means there are some inherited @Before tags which are responsible for reading my spring application context from bean files and dynamically injecting needed beans into my Test class before executing any of the tests.

Turns out I misconfigured my test resources to point to the wrong directory, so spring couldn’t find my application context files. As a result, an exception was thrown, but rather than marking the tests as failed, they were marked as ‘skipped’. Since they were not marked as ‘failed’, gradle thought everything was good and continued on with the build.

Is there away to configure gradle or testng to fail the build in this case? This is the behavior I have see in maven. Maven still marks the tests as skipped, but the build fails.

Hopefully if I can get this resolved, it will help persuade folks to consider officially supporting gradle in our org.

Thanks.

Doug


Current status of handling skipped tests when using gradle/TestNG
(dougy-gradle) #2

I’m thinking this is a bug, as I looked in the gradle source code, and it appears that in the TestNGTestResultProcessorAdapter class, which listens for testng configuration failures, there is code that looks like its essentially creating a fake test and marking it as failed, presumably so that the test tasks will fail. However, this doesn’t seem to be working.

I tried to sync with git and get the latest from the trunk, but that was my first time doing it, and I ran into problems related to the new local wharf cache and gave up. So I’m not sure if the problem still exists on the trunk. I thought it would be a good exercise for my first time attempting to look at and potentially patch the code if it is indeed a problem, but then I have to figure out what the local cache errors I was getting were caused by, so I’m giving up and that and in the meantime will submit a defect report.


(Adam Murdoch) #3

Try again with the latest from git. I’ve just pushed a fix for the cache problem.


(dougy-gradle) #4

Thanks, Adam. I resynced after getting your message and that resolved the problem I was having with the local cache.

I have a little more information on this and confirmed this is a bug. Turns out it that it depends on what version of testng you are using.

If you use testng version 6.2 and higher ( as of 11/10/11, only 6.2.1 is the highest ), then the test tasks will NOT fail if your testng classes throw an exception in any your methods annotated with @BeforeMethod,@BeforeClass,@BeforeTest

However, if you use an earlier version it works fine and the test task WILL fail appropriately.

I looked through the Gradle source code and it turns out that there is already an integration test that checks this behavior:

./subprojects/integ-test/src/integTest/resources/org/gradle/integtests/testng/TestNGIntegrationTest/javaJdk15Failing/src/test/java/org/gradle/TestWithBrokenSetup.java

I noticed the build.gradle file used to drive that test is using org.testng:testng:5.14.10. If you change that version to 6.2, then the 2 tests org.gradle.TestWithBrokenSetup and org.gradle.BrokenAfterSuite will NOT fail.

So, there is something in the 6.2 version of testng that appears to NOT be backwards compatible in whatever API calls gradle is integrated in.

I’ll try to dig around in the testng release notes for 6.2 and see if there is anything obvious.

But, in the meantime, I’m happy as I can just back up to an earlier version of testng and that solves my problem.


(Doug Lethin) #5

Just wanted to comment on this issue. I updated the Jira (http://issues.gradle.org/browse/GRADLE-1822) I reported for this issue.

I found some time to try to work through the gradle source code in the hopes of maybe submitting a patch to this issue. I think I did indeed validate that my original guess of the problem is correct. It seem that starting with testng 6.2, gradles testng IConfigurationListener implementation is not getting notified of ConfigurationFailures, therefore it does not know to fail the build. It seem that in testng 6.2, that interface finally became “public”. That is, it was some kind of hidden unofficial interface I guess prior. But it seems that in the move to create the public version of that interface, the original one is not getting triggered.

At this point, its a bit beyond me to figure out how to patch and resolve the problem.

I consider this a pretty severe issue, but I don’t quite know if it a matter of testng patching there code to support that API again, or gradle adapting somehow to the correct methodology depending on the version.

As I mentioned in an above comment, at first I could live with just dropping down to testng 6.1.1. However, I’m starting to integrate jmockit library into my codebase, and it seem that the best versions to use for compatibily are testng 6.2 and jmockit 0.999-11. So, I’m really hoping to see this fixed.

Please let me know if there is any other additional troubleshooting I can provide.

Thanks.

Doug


(Doug Lethin) #6

By the way, this topic was originally filed in the “questions” forum since at the time I didn’t know it was actually a bug (or regression with testng).

I don’t want to create senseless noise on the forums, but

I’m curious to know if its worth closing this and creating a new topic in the “Problems” forum?


(Adam Murdoch) #7

I’ve changed it to a problem.