Release buildship build system?

Help/Discuss Buildship
1

Hi there,

I looked at how you are building Buildship with Gradle directly calling PDE Build automation tasks and I would like to know if you have plan to externalize the code you have in buildSrc directory in a dedicated plugin that could be leveraged by other teams.

This seems like a good base of a Eclipse PDE build plugin. Would accept contributions to externalize this as a gradle plugin that can be re-used?

Regards,
Matt

2 Likes
2

Hey Matt,

I would say go ahead and feel free to extract things out of the buildship project into a common plugin. But you might want to have a look at an existing project for building eclipse stuff using gradle: https://github.com/akhikhl/wuff
It might be better to join forces and contribute to this plugin instead of creating another similar one from scratch.

1 Like
3

Hi RenƩ,

Thanks for the heads-up. Yep I already explored Wuff a bit. From my preliminary analysis, both have took a quite different approach at building Eclipse Bundles (RCP, Plugin, etc.). Iā€™m still looking at the pros and cons of each way of doing things and Iā€™m not yet sure which one will better suite my workflow.

For example, it still unclear to me as how to continue develop my plugins within Eclipse IDE once Wuff is used to build it. While clearly, this is ā€œsolvedā€ in buildship with current approach. Moreover, Iā€™m still unsure if both approach would be viable in the same Gradle plugin.

Anyway, be assure that I wonā€™t start off a complete new project unless it is really necessary due to fundamental problems.

Any insights from Buildship buildSrc developer(s) on potentials problems and pain points about current way of doing things in Buildship is appreciated. Will try to keep you posted on my findings and progress.

Regards,
Matt

4

You are not the only one interested in the build scripts used to build Buildship. Looks like there is a common interest to export this functionality into a separate plug-in.

From the beginning we had the plan extract a separate plug-in and make it available for contributions. We were (and are) simply overwhelmed by our other duties to properly set up everything.

IMHO The biggest issue we had to solve in the scripts is to handle the dependency resolution and compilation of Eclipse plug-ins. The current implementation is the same what Wuff does: it ā€˜mavenizesā€™ the references p2 repositories and compiles the plugins against that one as it was an ordinary Java library. The implementation is taken from Wuff too.

Iā€™d like to invite you for a discussion, if you have any plans to reuse/fork our solution letā€™s discuss it. At least not to do the same job twice.

5

Yes, exporting what you did for Builship seems like the reasonable thing to do.

I understand extracting it it is not a priority and that you guys have little free time for this task. Thatā€™s where I can help because I would have some interests in re-using what you did and exporting it.

Iā€™m glad to here that Wuff is ā€œcompatibleā€ with what you have currently. I feared that it was a fundamental different solution then Wuff.

Wuff seems like a really nice plugin and worked great the first time I tried. But I need to play more with it. We have a fairly complex projects layout where I work. Iā€™m currently building a trimmed replicate of our setup so I can play with Wuff and buildSrc Buildship directory with a more complex example.

Iā€™m clearly not in favor of doing another project from scratch. My current goal would be to port what you have into Wuff directly so re-use is optimal.

It would be a good thing to discuss with you about what you did in Buildship, your needs and objectives, build pipeline and work flow. I will try to play a bit with Buildship stuff prior this so I have at least a basic knowledge of the current state.

Will keep you posted.

And thanks for the heads-up, I appreciate it.

Regards,
Matt

6

Hi Matt

Thanks for your involvement.

We started with Maven Tycho, then went to Wuff, and then decided to write our own Gradle plugin classes to build Buildship. Tycho and Wuff did not really satisfy our needs.

What you see in builSrc is quite mature. Its implementation had started at the beginning of the year and a lot of work went into it. By now, we hardly touch the buildSrc anymore since it works very reliably for us - both when building locally and when building on TeamCity.

The reason we did not extract buildSrc into a ā€˜third-partyā€™ plugin dependency is that so far, it was always quick and easy to change the build. But this is less relevant now that our build is stable.

Feel free to extract things into a full set of Gradle plugins that can be shared across builds. If you do so, I suggest to at least first do it very mechanical and not to change any logic / configuration. Once done and our Buildship build still works exactly the same, steps can be taken to make more things configurable (if needed).

7

Hi Etienne,

Glad to hear you tried Tycho and Wuff. Wuff, for now, is not satisfying my needs neither. Knowing others felt the same reinforce my own feeling about current state of Eclipse plug-in building via Gradle.

Iā€™m following the Buildship repository since a couple months now and I was able to see the evolution of the buildSrc. Iā€™m glad that you now see it as mature.

From your own knowledge, do you think integrating what you did in Wuff directly is feasable? Is it a good idea?

And be assure that once I start extracting what you did, the initial version will be a direct port of what you have.

Regards,
Matt

8

Not sure. I do not know Wuff well enough. Our primary goal was to have a build whose behavior we fully understand and can control. We now have this - and it is about as much as we can currently invest in building Eclipse plugins with Gradle.

Sounds good.

9

Hi @etiennestuder and @Matthieu_Vachon,
I would like to join this discussion as Iā€™m going to migrate my company build system to Gradle. I created a post with a description of our development environment here: https://discuss.gradle.org/t/newbie-needs-advices-and-best-practises-to-move-to-gradle/11589.

Iā€™ve also taken a small look at Wuff, and it seems pretty complex. Iā€™ve not yet had a deep look at Buildship, but I understand from this discussion that Buildship uses PDE, which is definitely a great help. Iā€™m going to look at it closer because I wonder how guys you solved the issue with the versioned PDE components (a pretty stupid thing for everyone who want to build headlessly imho).

Iā€™m new to Gradle, but I wrote our build system in 2008 to build headlessly around 100 projects exclusively made of Eclipse plugins, features and RCP components. So I can share my knowledge about such things, Iā€™m far from being an Eclipse expert though.

Happy to find people with the same problems, Iā€™m a bit closer to the solution!

10

Hi Romu,

You are not alone :slight_smile: In our case, the big difference is that everything was normalized. What I mean is that we had the same third-parties (checked-in source control) and same Eclipse target platform (also checked-in) inside a single big monolithic repository. Our old build system was custom but did not have Eclipse headless building feature like you.

Now, the main goal I had was to extract a subset of projects (related to same end product), gradlelized them and still be able to develop in Eclipse (and build command line all of them except those having direct Eclipse dependencies). The next milestone after extraction will be to make Eclipse headless build work from command line.

Extraction is now complete and all goals have been fulfilled for this part. To make it work, my current approach was to have a vendor project, build semi-automatically that contains all third-parties jar. Target platform has been extracted to a distinct repository but still needs to be added manually to Eclipse workspace.

I did my experiments on this part in a repository. You can see a WIP branch (real work is however finish in the enterprise repository). Iā€™m hoping to finish it soon as well with the associated documentation. You can check it out here.

Now, my next goal is to make the Eclipse headless build come true within my test repository above. Here some possible solutions I will try:

  • Wuff
    Wuff seems rather complex but itā€™s not. It simply downloads an Eclipse SDK (so it has access to Eclipse jars), This acts as the target platform. Then, it mavenize the Eclise jars (because PDE has no notion of group, only artifact name and version) by tranforming for say the bundle org.eclipse.core.runtime into a dependency usable by Gradle of the (here fake) form eclipse37:org.eclipse.core.runtime:3.7.0.

    My current concerns with it are: how to use a target platform instead of downloading an Eclipse SDK? how to continue development within Eclipse when using Wuff? I started a thread on Wuff repository to gather feedbacks on those two questions.

  • Buildship build system
    I did not investigate it a lot yet but it would better fit my needs. As Builship is using them, I have a real-life working example of you I could leverage it. Just need to extract it. This is probably my next experimentation step for my Eclipse headless build task.

  • Bndtools
    This is a tool used to work with OSGI at large based on bnd (donā€™t be afraid from there old site). I just encountered it recently but it has a Gradle support as well as an Eclipse plugin support. But this is a different beast as it completely ditched Eclipse PDE. What it means is that you must use some bndtool metadata file. An inside Eclipse, bundle, features and plugins are build without relying on PDE at all. Itā€™s the bndtools plugin that do the hard work of resolving everything and ensuring everything conforms to OSGI spec, completely replacing PDE. This is a hard step because itā€™s completely different. Not sure where I could go with it. But itā€™s wort mentioning.

Iā€™m hoping to start again new experiments for Eclipse headless build in the upcoming weeks. Feel free to share any knowledge, findings, tips & tricks, problems with me. We could even chat about it sometimes.

Hope this helped you a little bit. Donā€™t hesitate to ask more information if needed.

Regards,
Matt

1 Like
11

At least, finding one guy working on the same problem is really great Matt!!

Of course, we call chat, you can join me on skype (rtisserand). Just be aware Iā€™m in France, so we donā€™t have the same hours, but letā€™s connect, that would be a pleasure.

Iā€™ll try to shorter things. I wonā€™t go bndtools. To me, the Buildship source code or Wuff would be enough. Iā€™ve read your thread on the Wuff GitHub, but Iā€™ve to say I donā€™t really catch your problem. My developers use Eclipse and are able to compile their projects from their workstations without any trouble. But only the components built from our build server are the ones distributed to our customers.

For the moment, Iā€™m learning Gradle (by reading documentation and watching some videos on YouTube).

Iā€™ve some specific requirements:

  • As I wrote in my thread above, I canā€™t normalized the dependencies chain
  • Developers have asked to be able to build locally as they do remotely even if they are outside our offices

Those requests mean Iā€™ve to carefully manage the dependencies, probably with an internal Ivy server.

Looking forward to continue this discussion.

1 Like
12

As the Buildship team has limited resources, it is unlikely that weā€™ll release the build system as a separate project on our own in the near future.

But this is an open source project on GitHub with a standard EPL license. Nothing prohibits you to fork it, or send pull requests :wink:

If you have a specific questions about the build system, weā€™re happy to answer.

13

It looks like Wuff is dead. However https://github.com/diffplug/goomph seems to be an alternative.