shadowJar leaves dependencies out

Help/Discuss
,
1

When trying to create a fat jar, shadowJar is leaving out a lot of dependencies. I created a build.gradle file from a working pom.xml. Maven builds a fat jar without issue. Reading through the Gradle’s information about building a fat jar, the shadowJar plugin appears to be the recommended method. However, my Gradle fat jar was giving the following runtime error:

Jul 19, 2018 2:35:10 PM restclients.RestStatus2Streams writeToStream
SEVERE: Error occurred while instantiating com.me.streams.impl.producer.MarlinProducerV10.
==> java.lang.NoClassDefFoundError: org/apache/hadoop/fs/PathId.

Obviously, my far jar was missing something. Sure enough, org/apache/hadoop/fs/PathId and entire directories of dependencies were missing. Although the pom.xml does not explicitly include the ‘org.apache.hadoop:hadoop-common’ dependency, I explicitly added the artifact as a dependency to my build.gradle file and still no joy.

To answer a few question:

  • Is the hadoop-common artifact in the repository? Yes.
  • Is the named version of the artifact in the repository? Yes
  • Is the class PathId is the artifact that’s in the repository? A ‘jar xf’ of the artifact indicates, “Yes.”

Since the Maven fat jar works perfectly and unjaring the Maven fat jar shows the PathId class (and others missing from the Gradle fat jar), then there’s something wrong with my build.gradle file and/or the shadowJar plugin and/or Gradle.

build.gradle file:

apply plugin: 'java'
apply plugin: 'maven'
apply plugin: 'com.github.johnrengelman.shadow'

group = 'com.me.ps'
version = me.version

sourceCompatibility = 1.8
targetCompatibility = 1.8

defaultTasks 'shadowJar'

buildscript {
    repositories {
        jcenter()
    }

    dependencies {
        classpath 'com.github.jengelman.gradle.plugins:shadow:2.0.3'
    }
}

repositories {
     [...]
}

dependencies {
    // Explicityly added to in hopes of being included in the fat jar
    compile group: 'org.apache.hadoop', name: 'hadoop-common', version: '2.7.0-me-1803'

    // dependencies listed in the pom.xml
    compile group: 'com.me.ojai', name: 'me-ojai-driver', version: '6.0.1-me'
    compile group: 'com.fasterxml.jackson.core', name: 'jackson-databind', version: '2.6.5'
    compile group: 'org.apache.kafka', name: 'connect-json', version: '1.0.1-me-1803-streams-6.0.1'
    compile group: 'org.apache.spark', name: 'spark-core_2.11', version: '2.2.1'
    compile group: 'org.apache.spark', name: 'spark-streaming_2.11', version: '2.2.1-me-1803'
    compile group: 'org.apache.spark', name: 'spark-streaming-kafka-0-10_2.11', version: '2.2.1-me-1803'
    compile group: 'org.apache.kafka', name: 'kafka-clients', version: '1.0.1-me-1803-streams-6.0.1'

    // dependencies not in the pom.xml, but added to make the gradle build work
    compile group: 'xerces', name: 'xercesImpl', version: '2.11.0'
    compile group: 'org.json', name: 'json', version: '20171018'
    compile group: 'com.google.guava', name: 'guava', version: '19.0'

}

shadowJar {
    zip64     true
    baseName 'me-edge'
}

What is my build.gradle file doing wrong?

Gradle 4.9
OS: Linux 3.10.0-514.el7.x86_64 amd64
JVM: 1.8.0_151 (Oracle Corporation 25.151-b12)