Implementing a custom Task without depending on Gradle's internals

Help/Discuss
1

I’ve recently been bitten by a breaking API change in Gradle 3.0 (the relocation of DynamicObject). My plugin is exposed to it by extending org.gradle.api.DefaultTask which extends org.gradle.api.internal.AbstractTask. The getAsDynamicObject method on AbstractTask returns a DynamicObject. As a result, all of my custom tasks are tightly coupled to Gradle’s internals.

I’m now looking at breaking that coupling and implementing org.gradle.api.Task directly would appear to be the only way to do so. Unfortunately, this looks like a huge undertaking. There are 49 methods on Task alone. Futhermore, looking at the return types I believe I’d also have to implement several other interfaces and abstract classes that do not appear to have standard implementations in the public API:

  • AntBuilder
  • Convention
  • TaskDependency
  • TaskInputs
  • TaskOutputs
  • TaskState

What is the recommended way for plugin developers to implement custom tasks without being exposed to breaking changes that are made to Gradle’s internals?

2

Hi Andy,

Thank you for raising this. I am deeply sorry that we have allowed this leaky abstraction to persist to the point it has repeatedly screwed over plugin authors (of which I am one). It should not require a prominent contributor raising this in this manner to enact change. If you write a custom task that is not binary compatible with a newer version of Gradle, that is a Gradle bug. We need to ensure that users do not run into those situations anymore in the future.

Let me restate your goal to make sure we’re clear. You wish to refactor your plugins (Spring Boot and others perhaps) such that you are not forced to maintain multiple versions because a future version of Gradle has broken compatibility. Please clarify this goal if you see it any differently - it’s crucial that we understand each other on this.

As you said, trying to implement a custom Task on top of only public APIs would require rewriting a significant amount of gradle/gradle. I hope we can avoid that. Here are the short, medium, and long term steps I propose we take to systematically address this problem:

Immediate Changes

Medium Term Avoidance of Breakage
Utilize a binary compatibility report (JDiff) to ensure we avoid breakage while we address the root problem directly (see below). We recently adopted this and missed this problem I’m afraid :frowning:

We may identify some quick wins in places where classes or interfaces are clearly incorrectly marked and fix them.

We continue to add compatibility tests for other plugins to be as proactive about catching breakages as much as possible. We will need community members to help us with examples that we haven’t thought of.

Long Term Solution
Discussion on this topic has long churned internally, and we will need to plan how to incrementally remove the “internal” designation on certain classes or provide an alternative public API.

I believe we should start this process by opening up a much more public and transparent discussion between Gradle and the community on this topic. This can continue here or perhaps on JIRA or GitHub as we get into technical implementation.

This is not going to be easy, but we cannot continue this pattern of quick, one-off fixes. I hope we have not burned you so badly that you do not want any involvement with this.

Cheers,
Eric

3

Thank you, Eric.

You wish to refactor your plugins (Spring Boot and others perhaps) such that you are not forced to maintain multiple versions because a future version of Gradle has broken compatibility.

There are two plugins that are particularly important to us. The dependency management plugin and the Spring Boot plugin. The former is used by the latter.

Our goal is to be able to support multiple versions of Gradle with the same plugin binaries. As things stand, we’ve lost confidence in Gradle’s binary compatibility across versions and don’t think that we can do so. As a result, we’ve recently updated the documentation for both plugins to state that Gradle 3 isn’t supported. I’d like to be able to reverse that.

Add the Spring Boot Plugin to our Dependent/Smoke Test builds

This is great to see. Thank you. And I’ve just noticed that you already have a smoke test for the dependency management plugin. Thanks for that too.

Collaborate to fix any other immediate outstanding issues that prevent supporting 3.0. What other problems are blocking you?

When we were trying to support Gradle 3.0 we hit another problem with an IllegalAccessError in the dependency management plugin. I had a very helpful discussion with @lhotari and @CedricChampeau about it where Cedric confirmed that trying to support Gradle 1 (Groovy 1.8) and Gradle 2 (Groovy 2.3 and 2.4) was rather ambitious due to differences in the byte code that’s produced by the different versions of Groovy. I plan to port as much of the dependency management plugin to Java as possible but it makes use of some of Groovy’s DSL goodness in places so some Groovy code will have to remain. This makes me nervous.

I’d been lulled into a false sense of security by Java’s excellent backwards compatibility and had naively assumed that the same would apply for Groovy. Now that I know that it doesn’t, I’d welcome some general guidance for plugin developers on what they should due to make their plugins as broadly compatible as possible. Where possible, using Java seems like the most robust choice but when you want to provide a DSL some use of Groovy is required as far as I can tell. If there’s a safe or lower risk way to do so, I think it’d be very helpful to document it somewhere.

I believe we should start this process by opening up a much more public and transparent discussion between Gradle and the community on this topic. This can continue here or perhaps on JIRA or GitHub as we get into technical implementation.

This sounds excellent and I would like to contribute to that discussion if I can. Something that I have observed in the past is that some of Gradle’s own plugins make use of internal API. One such area was the Java plugin setting up its publication. I wanted to do something similar in Spring Boot’s plugin but it uses ArchivePublishArtifact which is internal API. I realise this is considerably easier said that done, but if all of Gradle’s own plugins were written purely against public API, it would make it much easier for plugin developers to offer similar functionality and would also help to give users a more consistent experience across plugins if the underlying machinery was reusable.

I hope we have not burned you so badly that you do not want any involvement with this

I was feeling pretty burned originally, as I showed in my grumpy and unconstructive tweet. I try to hold myself to higher standards than that but didn’t manage to here. Sorry once again. I’m heartened by how the situation has turned around and things seem to be heading in a very constructive direction now. I’d definitely like to be involved in fixing the problems. We have a considerable number of users whose preferred build system is Gradle and we want to keep them happy.

1 Like
4

In my experience Groovy has generally maintained good backwards compatibility and I don’t think this is a Groovy problem.

Every framework has upgrade challenges, I’ve lead Grails for a long time now and rarely were the upgrade issues Groovy related.

The various threads out there seemed to push it towards that way, but unfortunately Groovy has become an easy target for all the problems in Gradle recently, which is a shame. As @lhotari indicated this is almost certainly an issue with the byte code instrumentation and/or classloading in Gradle and hence is a Gradle issue.

5

My belief that it’s, in part at least, a Groovy problem was based on what @CedricChampeau said:

I am unsure what could cause this, but I agree that compiling with 1.8 and running with 2.4 is likely to cause problems. Doing this is very ambitious, yes :slight_smile: Given that the Groovy runtime in Gradle 1, 2 and 3 are very different, if you want to do this, I would rather suggest to use “good” old Java.

That’s reason enough for me to move as much code as possible to Java so I can be certain that it won’t cause backwards compatibility problems.

6

Sorry to jump in here, but I have to second the point quoted above. I agree with all that’s been said, but I think this point is fundamental. Having Gradle dogfood its plugin API will help with improving it substantially. Most plugins in the Gradle repository I’ve seen have at least some form of internal API usage. That’s terrible for one simple reason: every plugin developer will look at these plugins as examples of what to do, but we don’t have the luxury of including the plugin in the Gradle binary to ensure compatibility.

My suggestion would be to separate Gradle-core functionality (plugins, DSL, generic model-space management, dependency management, etc.) from what are user visible plugins. And when I say user-visible, I mean visible to the Gradle user and to plugin developers (e.g., some software model plugins that other plugins use). I know this is a long-term request (Gradle 6?), but I think it will really improve Gradle’s prospects in the long road.

As a small digression, I found it very weird that in order to create a custom archive task, I sub-classed AbstractArchiveTask, which is not an internal class, only to find out I had to implement the abstract method createCopyAction which must return an object that implements the internal interface CopyAction. It’s the same situation as not being able to create a custom generic task, but this illustrates that even custom not-so generic tasks have these inconsistencies.

7

@Andy_Wilkinson @graeme.rocher I believe Groovy deserves nearly zero blame for backward compatibility issues. There are 3 major contributing factors in my mind across the different major versions of Gradle:

  • Shading different classes in Gradle JARs across different versions
  • How we use ClassLoaders
  • Changing Groovy versions over time

Groovy deserves zero blame for the last point. We had to know adopting a new major version of Groovy could break compatibility between Gradle versions in some ways. We took that risk for the great new features Groovy 2.x gave us.

@Andy_Wilkinson

Where possible, using Java seems like the most robust choice but when you want to provide a DSL some use of Groovy is required as far as I can tell.

For Gradle 2.13 and earlier, Groovy is more or less required for a DSL. We have had to eliminate this requirement so that we can enable build scripts written in other languages, starting with Kotlin.

If there’s a safe or lower risk way to do so, I think it’d be very helpful to document it somewhere.

Acknowledged. I cannot prematurely announce what we’re working on in order to help here, but I will solicit your feedback directly when we do. Soon.

Perhaps we can help initially by reviewing code changes in the above mentioned plugins that you have doubts about. If you’d rather have our input earlier, it would be best to create a new post in this forum stating the technical goal you’re trying to achieve and the Gradle team can advise.

@melix @lhotari I’m trying to support Gradle 1, 2, and 3 with the same binary. Is that overly ambitious? I’m currently building the plugin with Gradle 1.12. I also build and test with 2.x and 3.0.

As you and @CedricChampeau agreed, this is ambitious. It will become even more ambitious as we make progress in fixing this by moving stuff out of “internal” or providing new public APIs. I think you may end up limiting your plugin capabilites very much or alternatively have to provide a lot of code of your own to compensate. More on this below.

This sounds excellent and I would like to contribute to that discussion if I can.

Great. This very discussion can is a start.

I understand why you want to be able to support Gradle 1.x and forward with 1 binary. I wish I had a silver bullet here for you, but I don’t. I don’t think it is reasonable for you to live the amount of pain that would cause you and I further don’t expect to convince you to revert Drop support for Gradle 3.0 · spring-gradle-plugins/dependency-management-plugin@6f1c3d9 · GitHub immediately.

The way I see it, there are 2 ways forward:
We try to ensure backward compatibility between 2.x and 3.x from your needs in the future
We convince you and everyone that Gradle 3.x is worth supporting. I see so many compelling features in Gradle 3.1 - 3.3 that if you still remain unconvinced… then I guess Gradle needs a significant pivot in strategy.

Other than plugin docs and compatibility avoidance, is there anything else that would help? Would it require releasing a Gradle 1.13 that made select classes available across all versions?

@jvff @Andy_Wilkinson

if all of Gradle’s own plugins were written purely against public API, it would things make it much easier for plugin developers to offer similar functionality and would also help to give users a more consistent experience across plugins if the underlying machinery was reusable.

You nailed it. I believe this is at the heart of the problem, but this is going to be very arduous to untangle.

My suggestion would be to separate Gradle-core functionality (plugins, DSL, generic model-space management, dependency management, etc.) from what are user visible plugins. And when I say user-visible, I mean visible to the Gradle user and to plugin developers (e.g., some software model plugins that other plugins use). I know this is a long-term request (Gradle 6?), but I think it will really improve Gradle’s prospects in the long road.

Nailed it again. I cannot make a single promise for when we do something like this, but I want you to know we have been talking about doing this exact thing for a few months at least. Furthermore, we have identified some quick wins here and I expect you won’t have to wait until Gradle 6 to see them.

Wow that is a long response already. If I missed addressing something, could you please reiterate it so we can make sure it didn’t get missed? We will continue work on the short-term items above.

Cheers,
Eric

8

Thankfully, that’s only a relatively short term requirement.

The Spring Boot 1.4.x line will be the last that supports Gradle 1.x. Spring Boot 1.5 (due towards the end of this year) will support Gradle 2.x and, if we can manage it Gradle 3.x. For Spring Boot 2.0 (due in the middle of next year) we may require Gradle 3.0, and almost certainly will if we’ve been unable to support Gradle 2.x and 3.x at the same time.

Thanks, but I don’t think so. My main concern is how much we’re asking of Groovy by compiling Groovy code with 1.8 and running it with 2.3 and 2.4. We’ll have to live with that for now, mitigating it somewhat by reducing the amount of Groovy in our plugins, and then avoid it completely by dropping Gradle 1 support.

I’m heartened to here that you’re aware of the problem and that there are some quick wins that we can look forward to soon. Thanks again for all your input and providing some insight into the Gradle team’s thinking.

9

This is very helpful to know. Please reach out if we can provide any support that would help you get Spring Boot 1.5 on Gradle 3.x. We’re eager to help make sure this happens.

10

@eriwen With an eye on Gradle 3 compatibility, could you please offer some advice on what version of Gradle 2 we should require?

At the time of writing, Gradle projects generated using http://start.spring.io default to 2.13 but in all likelihood we’ll want to support earlier versions of Gradle 2 as well. Is there a sweet spot in terms of features, adoption, and compatibility with Gradle 3 that would make a particular version of Gradle 2 an obvious candidate? As Spring Boot is built with Maven, we’re in the odd situation of building its Gradle plugin with Maven as well and we’d like to be able to continue to do so. Unfortunately, that means that features like Testkit probably shouldn’t influence our minimum version of Gradle as I suspect it can’t be used from a Maven-based build.

11

Based on relative adoption of Gradle versions and known high-value features, I believe your sweet spot is supporting at least Gradle 2.9+ and later.

We are discussing ways to make it easier for Gradle plugin authors to develop and publish with Maven and will follow up with you once we have news.

12

@Andy_Wilkinson I have been looking into the compatibility problem that concerns using ProjectBuilder for plugin testing:

https://discuss.gradle.org/t/cannot-apply-spring-boot-gradle-plugin-in-own-plugin-with-gradle-3-0

[#GRADLE-3558] Applying a plugin built with Gradle 2.x through ProjectBuilder can't find internal classes

It is now fixed for Gradle 3.2. I am not sure from the conversation if that is the major concern for you or only one of many. But if you think this might solve some of your issue already, I would encourage you to try it out by running the corresponding tests with our latest nightly. If you have feedback, we can put further effort into improving this for 3.2. And if you have other concrete issues with reproducible examples, which are not covered by this, please share them.

The test that is part of the fix also contains a workaround to make the project builder tests work with 3.0 / 3.1 if that is a concern for you:

13

@Andy_Wilkinson I want to update you briefly on another concrete step we’re taking to directly address the issue you bring up.

We are prioritizing externalizing highly-used internal functionality in plugins like the ability to create polymorphic containers for Gradle 3.3. Changes like this will be ongoing, but we will start with internal classes that we see frequently used from GitHub Archive data and votes from the community (more on that soon).

Looking forward to your input here, and I’ll share more with you and everyone as we make these changes.

Cheers,
Eric

1 Like
14

I have built plugins using a variety of Gradle versions and tested them all from Gradle 2.0 - Gradle 3.x (whichever the latest was at the time). I’ve used both TestKit asis or th GradleTest plugin. I know that all of those plugisn are compatible across the range of versions even with many of them relying on internal APIs. The latetr cases have been delat with acordingly when breakages occur.

I would also say that it depends which feastures you require. If you want new model support then the earliest version to build against Gradle 2.12 as a minimum. For simplistic plugins it does not really matter.

15

More good news. Thanks, @eriwen.

16

Thank you, @jendrik.

One area that’s still a concern to me is the problem that was linked to in an earlier post of mine in this thread. I don’t think we ever got to the bottom of it. Right now I can’t describe when things will and will not work, and certainly can’t do so succinctly. I also suspect it means that the underlying problems won’t be addressed as no one seems to know exactly what they all are yet.