Partially importing projects


(Stefan Oehme) #1

Currently Buildship can only import a project fully. However, for bigger projects I often only want to work on a small portion of the code base. The line between a project dependency and an external dependency should be fluent.

For comparison, you can do this with both PDE and m2e:

PDE: Have the last nightly of your projects in the target platform. When you close one of the projects in your workspace, the others will link against the one in the target platform instead.

m2e: Have the last successful build installed in your local Maven repo. When you close one of the projects in your workspace, the others will link against the local repo.


(Etienne Studer) #2

Thanks for your input, Stefan.

In the scenarios you have described above, what happens if the closed project exports project/external dependencies and another project depends on the exported dependencies of the closed project?

Gradle 2.5 will most likely contain the ‘dependency substitution’ feature:
https://github.com/gradle/gradle/blob/master/design-docs/dependency-substitution.md

This provides the functionality you have described on the level of Gradle. Whether/how to solve this specifically in Buildship needs some more discussions/thinking.


(Stefan Oehme) #3

The transitive dependencies should stay exactly the same. So if I have 3
projects (webapp, core and lib) in the workspace and I close ‘core’, then
’webapp’ should still link against ‘lib’ in the workspace.


(Etienne Studer) #4

If lib was a transitive dependency of webapp through core, then web app is not directly linked to lib. But when core is closed there should now be a direct link from webapp to lib?

We plan to make some changes to the project dependency setup soon (project & external dependencies). We should revisit this topic once it is in as well as dependency substitution.


(Stefan Oehme) #5

Yes, what you said is what I would expect.

But let me propose an even simpler mechanism:

Always include all transitive dependencies in the Gradle container and
don’t depend on reexports. This is good in my opinion, since it allows me
to switch off the Gradle dependency container for some projects without
breaking downstream projects.

A typical use case would be a mix of Eclipse Plugin Projects and normal
Java projects. Eclipse Plugin projects are very picky about their
classpath, so it is best to leave them alone (== only Plugin dependency
container, no Gradle classpath container). But you might have downstream
Java projects that consume those Eclipse plugins projects. And you want
them to have proper dependencies, even though the Eclipse Plugin Projects
don’t reexport.

I can zip up a small example of the problem if you want =)


(Etienne Studer) #6

We plan to make some changes to how the dependencies are mapped to the Gradle Eclipse project model. There is a good likelihood that the need to re-export disappears.

I’m not sure why a Java project would depend on an Eclipse Plugin Project (is this a real use case?). But, I think I understand what you are saying in general.

A small example would be nice to illustrate your use case. Code can say more than a thousand words :wink:


(Stefan Oehme) #7

It is a use case, although not the most common. Xtext has it for instance:

org.eclipse.xtext (an Eclipse bundle, which is also published to Maven
Central)
org.eclipse.xtext.web (a normal Java project that depends on
org.eclipse.xtext)

Inside Eclipse, we want to keep ‘org.eclipse.xtext’ as close to the normal
PDE build as possible. The main reasons are that we are used to it and that
the largest part of our clients are Eclipse plugins too. So the closer we
stay to their world, the sooner we see problems.

‘org.eclipse.xtext’ does not reexport anything to ‘org.eclipse.xtext.web
(the Plugin dependencies are only visible to other Plugin projects). So
xtext.web needs to take care of the transitive dependencies itself.


(Etienne Studer) #8

Ok. Thanks for the example. I think we are on the same page now.

One more question: do you build these Eclipse plugins with Gradle, too, I assume? i.e. you have a multi-project build with Eclipse Plugin Projects plus Plain Java Projects in it? And when you import to Buildship, you want the Eclipse Plugin Projects to be left alone in terms of the dependency container, but the the Java Projects to each contain all their transitive dependencies?

Etienne


(Stefan Oehme) #9

Yes, it is a multi project build, but we don’t compile the Eclipse Plugins
with Gradle. We call out to Maven Tycho instead, as it is the only mature
solution at the moment. We just tell Gradle how to determine the
dependencies from the manifest and how to assemble a Jar from Tycho’s
output.

And yes, inside Eclipse, we want the Eclipse Plugins to be left alone, but
the plain Java projects should have all transitive dependencies.


(Etienne Studer) #10

Got it. As I understand your description, you create a Jar from Tycho’s output such that you can have it as a dependency of the org.eclipse.xtext.web project. You also use Gradle to get the external dependencies from the org.eclipse.xtext project (defined in its manifest file) and then add them to the org.eclipse.xtext.web such that org.eclipse.xtext.web contains all the (transitive) dependencies and org.eclipse.xtext does not need to re-export anything. Correct?

Buildship currently does not modify (the classpath, etc.) of projects that do not have the Java plugin applied. Thus, in your concrete scenario, I would say that even if we re-export the libraries for Java projects (as we today but probably not tomorrow), it would not affect org.eclipse.xtext.

I believe, during import, Buildship should leave Eclipse Plugin Projects alone for the most part, and all Java projects should be set up such that each project contains all transitive external dependencies and none of them being marked as ‘exported’.


(Stefan Oehme) #11

I think you got it, but your wording was slightly different from what we
do, so I’ll explain more details just to be sure :slight_smile:

The dependencies from the manifest are not added to the downstream
projects. They are added to org.eclipse.xtext itself. So on the command
line, everything works as if it was a plain java project. The downstream
projects couldn’t tell the difference. See
https://github.com/eclipse/xtext/blob/master/gradle/eclipse-plugin-project.gradle

But inside Eclipse we don’t want to have these dependencies reexported, as
it can easily mess with pde. So instead we flatten the dependency tree on
the downstream projects, so they have all transitive dependencies directly.
See
https://github.com/eclipse/xtext/blob/master/gradle/unroll-dependencies.gradle

So if this flattening was the default in Buildship (as you suggest for the
future), we could get rid of the hack above :slight_smile:

One minor thing: You said “all transitive external dependencies”. It should
be all dependencies, including other projects.


(Etienne Studer) #12

I can confirm that it is fully clear now. Thanks for the code references.


(Mark Worger) #13

I’m currently evaluating Gradle for use in a project with around 300 sub-projects, so importing the whole thing is really not an option.

Has this feature implementation been planned at all, when is it likely to be available?

Additionally, our fan out of project dependencies is pretty high, so as discussed before, it would be good if you were able to import the single sub-project only - I think the old gradle plugin for eclipse forced you to import sub projects for dependencies as well.

The managed solution that you’ve described would be great, but even an override that allows for manual configuration would be welcome.


(Etienne Studer) #14

Stefan, for completeness of this thread, the things discussed above have been implemented and are available in the latest milestone (except the partial import).


(Etienne Studer) #15

Hi Mark

We will certainly tackle this in the future, but I do not know when and how, yet. Sorry for having to be so vague at this point in time. We want to be sure that the user experience for full projects is solid before we tackle partial imports.

Etienne


(Felix Mayer) #16

Is there any news on the partial project import? I also need this for a project I am working on. This project consists of multiple Eclipse plugins and a web site where you can download the plugins. For the Eclipse plugins, I want to use the PDE instead of Gradle when in Eclipse. But my Gradle build does build everything so the web site has up-to-date plugins.


(Stefan Oehme) #17

You don’t need partial import for that, you can just use PDE. Configure your Gradle project so that it includes the PDE classpath container and PDE nature and remove all external dependencies from the classpath. Then import it as a Gradle project and it will configure it as you want.


(Luke U) #18

I’ve been happily building with Gradle on the CL for some months. The last obstacle stopping me permanently switching Eclipse to Buildship is precisely the m2e feature that @st_oehme described here:

m2e: Have the last successful build installed in your local Maven repo. When you close one of the projects in your workspace, the others will link against the local repo.

I’ve tried to look into the docs about dependency-substitution, but unfortunately I’m not that proficient in the Gradle DSL (I didn’t develop the build scripts) and I can’t figure out if this is supported now, and if so how I would go about defining it.

So far I’ve got apply plugin: 'maven-publish' in place, so that I can publish this one module to my local repo (which works nicely).

Could anyone give me a pointer on how to request (my) Buildship to load it back from there? (Ideally, just when I close this project in Eclipse like I do with m2e, but other options might be viable.) Note that other team members do not need this. (They use IntelliJ, vim and/or command line.)

I can elaborate more on the specific use-case if that would be useful. (Also let me know if this request is better spun off as a new thread.)

Many thanks in advance for any tips.


(Luke U) #19

Just to wrap-up my question: I’ve now got to a workable solution as a result of this discussion).

Also just as a reference (for anyone else stumbling upon this discussion) I spotted that a similar request for some “opering / closing project magic” was also logged recently: