Hi Andy,
Thank you for raising this. I am deeply sorry that we have allowed this leaky abstraction to persist to the point it has repeatedly screwed over plugin authors (of which I am one). It should not require a prominent contributor raising this in this manner to enact change. If you write a custom task that is not binary compatible with a newer version of Gradle, that is a Gradle bug. We need to ensure that users do not run into those situations anymore in the future.
Let me restate your goal to make sure we’re clear. You wish to refactor your plugins (Spring Boot and others perhaps) such that you are not forced to maintain multiple versions because a future version of Gradle has broken compatibility. Please clarify this goal if you see it any differently - it’s crucial that we understand each other on this.
As you said, trying to implement a custom Task on top of only public APIs would require rewriting a significant amount of gradle/gradle. I hope we can avoid that. Here are the short, medium, and long term steps I propose we take to systematically address this problem:
Immediate Changes
- (X) Fix ProjectBuilder binary compatibility with plugins compiled with Gradle 2.x. This will be done for 3.2. The issue raised here is only relevant to testing a plugin with
ProjectBuilder(though I’m sure you have more). - (X) Add the Spring Boot Plugin to our Dependent/Smoke Test builds (we test a bunch of 3rd party plugins on every commit, but until now Spring Boot was not included)
- ( ) Collaborate to fix any other immediate outstanding issues that prevent supporting 3.0. What other problems are blocking you?
Medium Term Avoidance of Breakage
Utilize a binary compatibility report (JDiff) to ensure we avoid breakage while we address the root problem directly (see below). We recently adopted this and missed this problem I’m afraid 
We may identify some quick wins in places where classes or interfaces are clearly incorrectly marked and fix them.
We continue to add compatibility tests for other plugins to be as proactive about catching breakages as much as possible. We will need community members to help us with examples that we haven’t thought of.
Long Term Solution
Discussion on this topic has long churned internally, and we will need to plan how to incrementally remove the “internal” designation on certain classes or provide an alternative public API.
I believe we should start this process by opening up a much more public and transparent discussion between Gradle and the community on this topic. This can continue here or perhaps on JIRA or GitHub as we get into technical implementation.
This is not going to be easy, but we cannot continue this pattern of quick, one-off fixes. I hope we have not burned you so badly that you do not want any involvement with this.
Cheers,
Eric