We mostly have task-caching disabled for our test-tasks, due to issues with test flakiness. We would very much like to make use of it, and are wondering if it is possible to prevent results getting saved to the cache, if the test-task produced test-failures (without, however, having to make that task actually fail / fail the build).
The way we run a lot of tests in CI (in a larger multi-project build) is to run a given test-task across all projects in a given job, and we generally want all those tasks to complete (whether there were some test-failures or not). So making one of those tasks fail the build in case of test-failures is not really an option for us, which is why we have ignoreFailures = true set for the tasks.
Is there any way this could be achieved? This would be a very helpful feature for us - it would allow us to enable caching, but still be sure that only “good” results actually get to be stored in the cache. Otherwise, with caching we frequently see that some flaky tests appear as “persistently-failing” across builds, because their failures got cached along - which can be quite confusing and which we want to avoid.
Any help appreciated.
Context: We’re running Gradle 5.4.1 (so not quite the most recent version), in case that’s of relevance. Updates pending
Maybe not what you want to hear, but remove ignoreFailures = true
If you set this to true you tell the task to be successful even if there were test failures, so there is nothing preventing the task to be cached.
There are cacheIf and doNotCacheIf with which you can control whether the task output is cached or not, but I’m not sure whether those are evaluated after the task completed so that you can look at the result, that would have been to be tried out but even then as long as that is not documented behavior it might be unreliable to depend on.
But to your rescue comes an alternative suggestion to ignoreFailures, as ignoreFailures is not what you actually want. ignoreFailures says “let the task be successful even if there are failing tests”, so the result can also be cached. What you actually want to say is "continue running the build even if there were failing test tasks. And that is was --continue does for you. It continues running the build even if there are failing tasks, for all tasks that do not depend on that failing task. So if a compileJava task fails, the according jar task would not be run, but other compileJava tasks that do not depend on the failed one will.
I was thinking about this too first - however, for all I understand those (cacheIf etc.) generally specify both of whether tasks should a) read from the cache (i.e. skip task execution, if we’re getting a cache hit), as well as b) if we should fill the cache with task outputs, if we actually had to execute the task. So I assume this all has to be evaluated already before we start task execution (since for a), we may actually skip task execution because of it).
For some use-cases like this one, it would actually be a nice feature if we were able to control those two sides of caching separately (reading from cache vs. writing to it), instead of “always” having it mixed together.
Nice to know, I didn’t know about --continue yet. As you mentioned this may really be a good alternative for our use-case. Guess we’ll check that out to see if it fits into our pipeline, thanks for that hint!
