"java-library" plugin... why?


(Douglas Kaminsky) #1

So I understand the splitting of dependencies that should be propagated (api) from those that shouldn’t (implementation) but the example is extremely non-illustrative of how this could be useful. Namely, if the code referenced by an implementation dependency is accessible via any public code-path, all you’re doing is creating a strong risk of class/method not found issues down the road. Taking the example given, any call to the public API method in which “ExceptionUtils.rethrow(e)” is called potentially exposes the downstream caller to an unexpected runtime exception that using the “java” plugin would have avoided.

How could this work better?

First, it seems to me that the most useful case for this particular mechanism is for “optional” dependencies, where there is a sample implementation or specific integration available alongside a core library that a downstream user may not want to be exposed to. Could the example be updated to something that actually makes sense, such as this?

Second, this only covers 1/2 of the challenge of writing an API – hiding dependencies that shouldn’t be used at all transitively. It does not solve the other 1/2 of the challenge which is preventing dependencies that NEED to be visible transitively from interfering with conflicting downstream dependencies. Could we potentially promote the “shadow” plugin to a first-class citizen by merging it with this plugin? This would make a lot of sense – hiding internal dependencies via namespace mangling (perhaps via a third configuration) would give a cohesive framework for developing APIs whose dependencies don’t interfere with downstream implementations.

(Stefan Oehme) #2

api/impl separation is about compile time, not runtime. Impl dependencies must not be exposed via any public code path, otherwise it’s part of your api. The missing piece is defining your api packages, we’ll add that to the java-library plugin and then start validating your dependency declarations.

There won’t be runtime errors, since implementation dependencies are all there at runtime. Its just about properly defining your api, i.e. what people can or cannot rely on.

This is completely different from optional dependencies. Implementation dependencies are not optional. They are just not part of your public signatures and you may remove them at any time. E.g. you may use Guava for data structures today and switch to fastutil tomorrow. If your clients need Guava they should declare so themselves. That’s what api/impl separation ensures.

(Douglas Kaminsky) #3

This wasn’t clear to me from the description/example— so let me see if I understand.

You’re saying the “implementation” classes will be transitively present in downstream projects as runtime dependencies but not compile time dependencies?

Isn’t this still of limited usefulness without the native ability to obfuscate namespaces? Resolving conflicts by forcing is fine but if semantic versioning isn’t enforced consistently across all dependencies you’re still inviting large amounts of risk.

Imagine for instance that you’re part of a large organization with literally hundreds of dev teams. Which do you think is a bigger operational problem: each project “accidentally” having the correct class path because I rely on an upstream project transitively specifying something I actually need? Or making every project define its own dependencies at their own version and forcing them to resolve conflicts or end up with runtime exceptions because of versioning conflicts?

The former actually breaks more reliably. In the latter, even using java-library, my resolution of dependencies may allow me to compile but not run properly in select scenarios without failing fast.

(Stefan Oehme) #4

That’s a different problem from api/implementation separation though. The same happens within a single project when you add runtime-only dependencies and it bumps the version of something you compiled against. Or if your test framework brings in a newer version of a library you use and thus your tests don’t match your production setup. This is what module metadata rules are there to fix. You could even go as far as adding a resolution hook that checks whether the versions in your compile classpath match the versions in your runtime classpath. We might add a convenience for this in the future, since it certainly makes sense for most common cases.

Shading is a very expensive solution to conflicts. If everyone shaded all their dependencies we would all end up with huge libraries and executables and slower programs because of more work for the JIT compiler. Sometimes it can’t be avoided, but generally resolving conflicts at the dependency level is preferable.

(Douglas Kaminsky) #5

My point wasn’t that there’s only one way to solve dependency issues, or that one way is better than another. As to your point about dependency checking hooks: EXACTLY! Solving the problem of inadvertent transitive dependencies at compile time is nice but it’s only one of the many challenges of dependency management that’s particular to libraries.

Notwithstanding above, there are two things that I’m suggesting to be enhanced: (1) in my view the example on the plugin documentation is unclear (as mentioned I read it as more of an “optional” mechanic, which I now understand it isn’t) and (2) if we are taking the trouble to have a separate plugin for making it easier to produce sensible library dependencies, there are other features we ought to expose so that we don’t need an ecosystem of <hyperbole>17 plugins </hyperbole> and custom dependency resolution to get to the 99% desired behavior