Possible to build Spark fat Jar with Gradle?

Help/Discuss
1

I am trying to build a fat Jar for Spark Scala app. Databricks recommends using SBT but I am more comfortable with Gradle and would like to use Gradle. Gradle built-in JAR support works fine up to a point when I need to merge files in merge META-INF/services directories. Here, I am instructed to use Gradle shadow plugin.

I almost got this plugin to work but it builds a JAR which is corrupt in a subtle way and doesn’t start. Here is the issue I opened and StackOverflow question on the same topic. Neither got much attention, it seems like I need to go at it myself, which will require at least few days of work based on my experience with this plugin so far.

So I need to decide whether I invest more time or switch to SBT. Wanted to ask the Gradle community: what is your take, what would you do?

Thanks for any feedback.
– Sasha

2

Imho always worth to use Gradle over any other build tool, but what answer did you expect when asking the Gradle community? :smiley: If you ask the SBT community you most likely get the opposite answer. :slight_smile:

But regarding your actual question, if you use a proper tool, you see that you pack some classes multiple times:

$ zipinfo build/libs/shadow-jar.jar  | grep shadow_test
drwxr-xr-x  2.0 unx        0 b- defN 22-Feb-17 15:43 shadow_test/
-rw-r--r--  2.0 unx     1129 b- defN 22-Feb-17 15:43 shadow_test/Main$$anonfun$1.class
-rw-r--r--  2.0 unx     1129 b- defN 22-Feb-17 15:43 shadow_test/Main$$anonfun$2.class
-rw-r--r--  2.0 unx     2560 b- defN 22-Feb-17 15:43 shadow_test/Main$.class
-rw-r--r--  2.0 unx      589 b- defN 22-Feb-17 15:43 shadow_test/Main.class
-rw-r--r--  2.0 unx     1129 b- defN 22-Feb-17 15:43 shadow_test/Main$$anonfun$1.class
-rw-r--r--  2.0 unx     1129 b- defN 22-Feb-17 15:43 shadow_test/Main$$anonfun$2.class
-rw-r--r--  2.0 unx     2560 b- defN 22-Feb-17 15:43 shadow_test/Main$.class
-rw-r--r--  2.0 unx      589 b- defN 22-Feb-17 15:43 shadow_test/Main.class

Once using from sourceSets.main.output and a second time using with jar.

But that is not your actual problem.
Your actual problem is, that you also shadow a signed JAR and copy over its signature unchanged.

$ zipinfo build/libs/shadow-jar.jar | grep META-INF | grep 'SF\|DSA'
-rw----     2.0 fat    53906 b- defN 18-Aug-23 09:16 META-INF/DUMMY.SF
-rw----     2.0 fat     1043 b- defN 18-Aug-23 09:16 META-INF/DUMMY.DSA

When then the JRE tries to read that JAR, it verifies the signature and sees it being not valid.

One of the many reasons, why imho fat jars are a very bad practice and an abuse of Java functionality and should be avoided at almost any cost. Spring Boot JARs are actually the exception, because as far as I have seen, they do it the proper way.

When I had seen this error being made in the past at least the JRE showed an a bit more meaningful error message, I don’t know why not in this case.
To solve your issue, just exclude the signature files using

exclude 'META-INF/*.DSA'
exclude 'META-INF/*.SF'

and it works as expected.

1 Like
3

Fantastic, thanks so much Bjorn! I had these exclusions in the jar version but somehow assumed the Shadow plugin would do them automatically.

Guess it was a right question to ask the Gradle community :wink:

Thanks,
– Sasha

1 Like