Possible to build Spark fat Jar with Gradle?

Help/Discuss
2

Imho always worth to use Gradle over any other build tool, but what answer did you expect when asking the Gradle community? :smiley: If you ask the SBT community you most likely get the opposite answer. :slight_smile:

But regarding your actual question, if you use a proper tool, you see that you pack some classes multiple times:

$ zipinfo build/libs/shadow-jar.jar  | grep shadow_test
drwxr-xr-x  2.0 unx        0 b- defN 22-Feb-17 15:43 shadow_test/
-rw-r--r--  2.0 unx     1129 b- defN 22-Feb-17 15:43 shadow_test/Main$$anonfun$1.class
-rw-r--r--  2.0 unx     1129 b- defN 22-Feb-17 15:43 shadow_test/Main$$anonfun$2.class
-rw-r--r--  2.0 unx     2560 b- defN 22-Feb-17 15:43 shadow_test/Main$.class
-rw-r--r--  2.0 unx      589 b- defN 22-Feb-17 15:43 shadow_test/Main.class
-rw-r--r--  2.0 unx     1129 b- defN 22-Feb-17 15:43 shadow_test/Main$$anonfun$1.class
-rw-r--r--  2.0 unx     1129 b- defN 22-Feb-17 15:43 shadow_test/Main$$anonfun$2.class
-rw-r--r--  2.0 unx     2560 b- defN 22-Feb-17 15:43 shadow_test/Main$.class
-rw-r--r--  2.0 unx      589 b- defN 22-Feb-17 15:43 shadow_test/Main.class

Once using from sourceSets.main.output and a second time using with jar.

But that is not your actual problem.
Your actual problem is, that you also shadow a signed JAR and copy over its signature unchanged.

$ zipinfo build/libs/shadow-jar.jar | grep META-INF | grep 'SF\|DSA'
-rw----     2.0 fat    53906 b- defN 18-Aug-23 09:16 META-INF/DUMMY.SF
-rw----     2.0 fat     1043 b- defN 18-Aug-23 09:16 META-INF/DUMMY.DSA

When then the JRE tries to read that JAR, it verifies the signature and sees it being not valid.

One of the many reasons, why imho fat jars are a very bad practice and an abuse of Java functionality and should be avoided at almost any cost. Spring Boot JARs are actually the exception, because as far as I have seen, they do it the proper way.

When I had seen this error being made in the past at least the JRE showed an a bit more meaningful error message, I don’t know why not in this case.
To solve your issue, just exclude the signature files using

exclude 'META-INF/*.DSA'
exclude 'META-INF/*.SF'

and it works as expected.