The mrJar Plugin v0.0.13 has been released — Modular MRJAR Files Made Easy

• … (see above)
@bigmuffer
@IrfanSyed-PSC
@huehnerlady
@davebrown211
@andreasb70
@SebastianHerzog
@jimshowalter
@vorburger
@ddelponte
@cloversai
TIA :+1:

@lingocoder

The directory structure I proposed is probably better for everyone.

In my initial post on the directory, I tried to explain why this is probably better, but maybe I didn’t effectively communicate my reasoning.

My comparison below is based on my guess that you implemented your plugin in a certain way. Maybe I’m wrong, but I assume that you create a JavaCompile for each target Java version, use the corresponding source directory (e.g., src/main/java9 for Java 9), configure a version-specific output directory, place the main & older Java versioned code on the classpath, and set the correct --release option (e.g., --release 9). If you do something different, please let me know.

Creating an extra source should automatically perform much of this wiring for you. This becomes more important when you try to support more than just the most basic Java-only scenarios.

Normally, each SourceSet has an AbstractCompile task for each language plugin that has been applied to a project (e.g., Java, Kotlin, Scala, etc).

Imagine if someone has Kotlin code that they want to include in the mrjar, with certain Kotlin as the baseline (main), other Kotlin for Java 9+ (java9), etc. How would you support distributing classes compiled from Kotlin into different version directories in the mrjar?

In your setup, you shouldn’t put the Kotlin code under src/main/java9 because it is Kotlin, not Java. Would you use src/main/kotlin9? What about other languages? Do you know how to associate every language’s directory name with its corresponding AbstractCompile task?

What if the default source directory name for a language ends with numbers? How could you disambiguate between a hypothetical language named Plan 9 with default directory name plan9 & a language named Plan with a default directory name plan which is to be built for java 9+, so it would be plan9 with the suffix?

Even if you allow people to manually configure this in mrjar {} in build.gradle.kts, that’s extra work, plus the directory name is still ambiguous without checking the build.gradle.kts.

With my setup, wiring source directories to AbstractCompiles would automatically occur for all languages, since each language will be stored under its default directory. e.g., Java & Kotlin for 9 would be under src/java9/java & src/java9/kotlin, respectively.

Thus,. the easiest way to separate code from many languages into separate target JVM versions is probably to have the directories within, e.g., src/main (e.g., java, kotlin, groovy, etc.) just segment based on language, and to have the segmentation for target JVM (java9, java11, etc.) be at the source set level, e.g., src/main (i.e. baseline), src/java9, etc.

This plugin is only for Gradle. If you’re in the Gradle world, you should follow the Gradle conventions. It will make it easier for you to integrate your plugin into the Gradle APIs & ecosystem, and it will make it easier for other Gradle users to understand & work with your plugin.

e.g., Java & C# use different method naming conventions. Even if there’s tons more Java code than C# in the world, you shouldn’t use Java naming conventions in C# code.

Most of the examples didn’t appear to be for Gradle. A few random people using a certain directory structure for their Gradle projects doesn’t mean that it is optimal.

Also, these projects are all standalone projects, not plugins for a whole ecosystem built around specific conventions & interoperability. None of these few projects had non-Java JVM code (like Kotlin, Scala, Groovy, etc.), so they didn’t need to figure out how to deal with including version-specific class files from languages other than Java within a mrjar. If you want your plugin to be used by many people, you need to support all of their needs, not just your own limited use cases.

If anyone has any criticisms of my proposal, I’d like to hear them, as I haven’t really used multiple source sets within a project (other than just for testing that plugins work with them), so maybe there are some issues that I’ve missed.

Your opinions are sincerely appreciated, @rgoldberg. Honest. But have you actually tried using the plugin? The reason I ask that is because comments of yours like this…

…and this…

…suggest that your opinions are based on something other than having observed the plugin’s actual functionality.

If you „haven’t yet used the plugin“, then you sharing why you haven’t would be appreciated too. TIA :+1:

Such a capability would just be gold-plating at this stage of the plugin’s life. At best. At worst, implementing the myriad of capabilities you propose — in what is really a beta — would be putting the cart in front of the horse. Wouldn’t it be?

If mrJar’s aim was to be all things to everyone, then all that gold-plating would have a place. But mrJar has no illusions of being all things to everyone. So all of your suggestions, as clever as they indeed are, are unnecessary. At this particular point in time, anyway.

If any of your suggestions ever do become necessary, then mrJar will cross that road when it gets to it. If that’s OK with you, that is :wink:

So although your proposed features are excellent ideas that a future version of mrJar might provide, the point of this post was not to solicit the myriad of every conceivable feature a mrJar -type plugin could support.

So please forgive me if I don’t address every single one of your suggestions right now. That would be because I’m trying to bring the discussion back around to the actual point of the thread: To request reviews and critiques of the actual functionality that the plugin implements today

Have you actually tried using mrJar, @rgoldberg?

I don’t want to use a plugin that isn’t open source. If you’ve already released the source, or if you release it in the future, please provide a link.

Even if I use the plugin, that won’t tell me how it’s implemented, unless I can backtrack from potential logging statements (which might not be there), or decompile the code & look at it, which is probably too much of s hassle.

I won’t know how to use the plugin for other languages (if even supported) unless it’s documented / I search through code. You’ve released a few different versions & I haven’t had time to check the documentation again. Maybe some of the details were available earlier, and I forgot or missed them, but I just replied off the top of my head from the subway, so, instead of researching, I gave you my assumptions and asked if they were correct.

I don’t have to use a plugin to know what features it might need or be advisable to have. I won’t necessarily be able to determine the exact implementation without source. So, I offered some ideas based on the limited information I have, and I don’t think that running the plugin would likely inform me much on these specific issues.

Could you either point me to docs and/or code that answers my questions / lets me know if my assumptions are correct or incorrect? If none is available, could you just reply with answers? It would help me, and presumably others, help you with your project. It might also teach me and/or others techniques that I and/or they don’t know.

It would be useful if you would provide rationales for your design / implementation decisions other than “Why not?” or “Some other people did this in simple, non-Gradle and/or non-plugin, Java-only related circumstances” (obviously that’s paraphrased, not an actual quote). Then we could all have a more informed discussion, and people reading this could better understand all the issues.

Then you’ve never published a plugin to Gradle’s Plugin Portal using their Plugin Publishing Plugin I take it? Because that’s not an open source plugin either.

I could point you to a ton of other examples of other software that you very likely use everyday, that is also not open source. But I’m sure that single example is sufficient to drive home my point. Which is: Regardless of how entitled us devs feel we are to have access to an application’s source code, having access to source code is not necessary.

I disagree that it’s putting the cart before the horse. Well-designed software anticipates future enhancements and is structured so that they can easily be added without refactoring. The more that you get into the habit of designing systems with change management in mind, the easier it becomes. It gets to the point where, even if you didn’t anticipate a feature, when you do think of it, your architecture will have a natural slot in which to add it with minimal or no refactoring, because your system is well modeled.

Not getting things right the first time, before adding in additional features, requires more refactoring later, more developer time later to enact the refactoring, compatibility issues for legacy uses & technical debt for users of the older versions.

It’s also good to discuss these issues on forums like this so that everyone can learn from, & teach, others about Gradle best practices.

You asked for suggestions in your initial post. I didn’t see anything requesting limiting discussion to only existing features until now. Either way, how you wire various parts of your plugin is an existing feature, since that limits how build code and/or other plugins can interact with your plugin. e.g., you set the main class of the application plugin. If that plugin didn’t allow you to easily programmatically set that, then it would have limited your interactions with it.

Let me be more precise: I don’t want to use non-open source projects, but I will use them when I need to. I don’t need to use an mrjar now, so I don’t need your plugin. I also don’t need to use your plugin to discuss its functionality.

You keep avoiding answering my questions. That’s not a way to encourage people to help you.

It now seems, anyway, that you don’t want my help since you’ve decided to only discuss a select subset of your plugin’s current functionality.

This is exactly what I suggested in my initial post:

Notice my intentional emphasis on the word usage? Several times? What definition of that word are you applying to interpret it as a request for a source code review?

I have projects that use languages besides Java. Thus, usages of your plugin for me would necessitate handling Java-version-specific classes compiled from other languages. You have now made it clear that your plugin doesn’t support that functionality, so I suggested that you support it, and offered advice about how to provide such support.

This is making suggestions about usage of your plugin, which you asked for. I don’t have to run your plugin to know that a usage would not produce my desired results. You now seem to be asking solely for bug reports, which is different than suggestions about usage.

You seem remarkably truculent for someone who solicited my (and others’) help. Especially when the only comments by others about my suggestions were in wholehearted agreement.

You won’t even share basic simple implementation details. It’s not like I’m asking for proprietary cryptographic algorithms. And those implementation details are part of your plugin’s API, because they’re exposed via globally accessible Gradle objects like Tasks, SourceSets, Extensions, etc.

Even if I had asked about completely internal implementation details, it would be courteous to answer my questions if they aren’t exceedingly time consuming and/or prying into proprietary trade secrets. I can’t imagine that it would have taken long to answer my questions, or that there is anything appropriately secret about the answers.

Nice way to pivot :+1:

Ah! I think I see what the problem is now. The difference in context of our usage (and interpretation) of the word: usage.

I presumed you already used a plugin that already does everything you’re proposing mine do. So based on that presumption, I wasn’t requesting that you use the beta mrJar in an everyday usage scenario. The way you already use your own gradle-java-modules plugin.

I was using the word in more the try using it to see if it works context. I hope that’s clearer now?

Yeah, we were talking about different things.

My fork of the Gradle Jigsaw plugin doesn’t create mrjars. I never got around to adding the support, as I haven’t yet needed them.

Oh! Wow! So are you referring to the same Jigsaw Plugin that @CedricChampeau is demonstrating in this awesome JavaOne 2016 webinar? OK! Cool!

The first time I’d ever heard of the Jigsaw Plugin was when I stumbled across melix’s little goldmine of a webinar a couple days ago. And I wondered why I’d never seen it ever mentioned anywhere in the whole of the Gradle User Manual.

So have you’ve taken over maintaining that plugin now, @rgoldberg? Awesome!

Interestingly, at around the 00:45 mark, you will see Champeau using the src/main/java7, src/main/java8 and src/main/java9 directory hiearchies that are apparently so anathema in Gradle world.

So if that directory structure is good enough for my favorite Gradle Core Dev, it’s good enough for mrJar :+1:

The plugin that you mentioned above is my fork (rgoldberg.experimental-jigsaw) of the Gradle team’s plugin (org.gradle.java.experimental-jigsaw).

I am not maintaining the original plugin, I’m maintaining my fork.

I will happily submit PRs to the original plugin, if the Gradle team ever gets around to accepting my first PR for it. I didn’t submit tons of PRs at once because I don’t want to have to maintain them all in case I have to change / rebase any of the existing commits due to feedback.

I don’t know if that video references the plugin from which I forked my plugin. Maybe I’ll watch the video later to see.

I wouldn’t use an appeal to authority as an argument; I use technical arguments. If you’re going to argue by appeal to authority, (at least some, but likely all, on) the Gradle team are against using mrjars at all, so then you shouldn’t even make this plugin.

Moreover, I imagine that the video was just demo code, not a full-fledged plugin to support tons of use cases. The former need not be as robust as the latter.

If the demo didn’t include other languages being targeted to multiple Java versions, then there was no need for my alternate directory structure, but there would be a case for it for a mrjar plugin that handles multiple languages. I probably would have coded a quick demo using the same directory structure that you propose, because it is sufficient for that simple use case. Different contexts can require different solutions.

That’s pretty much precisely the point I’ve been trying to bring home all along when I said this…

…and this…

Maybe I should have explicitly used the word demo instead of beta in those comments? But in any case, that’s exactly the stage mrJar is at right now. I haven’t positioned it as anything other than a demo at this point. That’s why I don’t think it needs to „support tons of use cases“ right this instant. Nor does it necessarily need to adhere to strict conventions in its nascent demo stage of development.

You practically took the words right out of my mouth. So cheers :+1:

Or, how about: Minimally Viable Product?

Hi folks,

Sorry for not commenting earlier, the dependency management team I work in is very busy with the upcoming 6.0 release, which is of great interest to what you’re doing. First, I’d recommend reading my point of view regarding mrjars, which hasn’t changed much. In a lot of cases, mrjars are just a poor man’s technical response to bad dependency management tools.

However, there are valid use cases for mrjars, in particular when you don’t know what the target runtime is. To me, however, mrjar or not is a packaging problem. Whatever solution is designed (and I think eventually this should be part of Gradle itself), it’s the library designer decision to publish a mrjar or not.
However, it is technically possible to delay that decision to the end of the process. In other words, there’s nothing which prevents from having a project directory structure which allows both individual jars for each target platform (foo-jdk9, foo-jdk10, foo-jdk11, …) and a mrjar (foo-multiarch). In the end, those are all variants of the same thing that Gradle can handle nicely.

So, it’s all about finding the right structure. The blog post I hinted at demonstrates such a structure, but it doesn’t mean it’s final. I also think this is orthogonal to using modules and I don’t think a mrjar plugin should be tied to modular Java. I’d also be careful with the directory layout shown in various Java documentations, because they are not particularly keen on following conventions.

Said differently, I think having a mrjar plugin, or, mrjar support in Gradle itself, is a natural candidate for the advanced variant-aware dependency management Gradle 6.0 is providing. There are some docs available here if you’re interested: https://docs.gradle.org/nightly/userguide/modeling_features.html

3 Likes

Hey thanks for replying @CedricChampeau!

Yeah, that very blog post is why I said you are my favorite Gradle Core Dev :wink: It’s like I was telling @tlinkowski the other day, it was your blog post that got me hooked on MRJARs several months ago.

To handle those very specific valid uses cases is the primary reason mrJar was created. The secondary reason, was because I wanted to learn whether it was even technically possible to do at all. The reason I wanted to learn if an MRJAR plugin was technically possible, was because I recall reading somewhere in the forums where some Gradle core dev or another said something to the effect that adding an MRJARing capability to Gradle was not technically possible. And I thought to myself: That doesn’t sound right. Surely, it must be possible! So…

Right now you mean? Or in the upcoming version 6? Either way, I think that’s awesome! I will definitely be reading that nightly documentation link you shared. Thanks, CedricChampeau :+1: What is the ETA for 6, by the way?

I hear ya CedricChampeau. But to be fair, although mrJar does provide those two capabilities in the one plugin, those two capabilities are not „tied“ to one another in the strict sense of the word. In fact, the JPMS-modularizing capability was tacked on as more or less an afterthought — after I’d already completely implemented the MRJARing capability.

The relationship between the two capabilities in mrJar is exactly the same relationship of the compilation and archiving capabilities provided by Gradle’s built-in Java Plugin.

In both plugins, compiling the source code of a project and jarring the artifact of the project, are two distinct capabilities. They just so happen to be provided by the one plugin. I mean there is not a distinct core Gradle Jar Plugin and a separate distinct Compile Java Plugin. Right?

So just like with Gradle’s own Java Plugin, with mrJar you can use one capability without ever going anywhere near the other. The thing that makes that decoupling of the two capabilities possible in mrJar, is exactly the thing you mentioned earlier: it’s all about the directory structure.

Thanks for creating the opportunity for me to clarify that, CedricChampeau :+1:

That sounds like such a cool feature! Again, like I said to somebody the other day: if I could do what I did in mrJar, Gradle’s devs could put my efforts to shame in a matter of hours :laughing: I can’t wait for version 6 to hit! :+1:

I haven’t read the entire Modeling features and their dependencies page from your nightly link yet, @CedricChampeau. So please forgive me if my understanding is incomplete at the moment.

But regarding just this particular caveat…

Capabilities are published to Gradle Module Metadata. However, they have no equivalent in POM or Ivy metadata files. As a consequence, when publishing such a component, Gradle will warn you that this feature is only for Gradle consumers

For the scenario you talk about here…

…the „they have no equivalent in POM…this feature is only for Gradle consumers“ consequence is not relevant to the above foo-jdkNfoo-multiarch packaging situation right? Because those are just JARs. And my expectation is that there shouldn’t be any reason why Maven/Ivy consumers would not be able to use standard JARs as normal — regardless of how they were built. Am I understanding that part correctly?