Ok, the whole project was trying to test the plugin with ProjectBuilder (which I have used in the past to test a plugin was applied to the test project with no issues).
When I switched to TestKit, the problem disappeared. The downside is that tests are much slower now and some hacking of the build file is needed to test certain situations (like adding tasks that produce certain output that then need to be parsed to assert things)