Fat jar cannot fails with "Failed to copy" errors on Windows when META-INF\license and META-INF\LICENSE collide


(Bill Darbie) #1

I am trying to build a fat jar using gradle on windows. The same gradle project builds fine on unix. The problem in that 2 jar files that the fat jar includes have the following files: META-INF\LICENSE (from jettison1.1.jar file) META-INF\license\someOtherFile (from several other jar files)

On unix things work because it is case sensitive. On Windows there are errors because the file LICENSE and the directory license are identically named (because it is not case sensitve).

I get messages like this: [ant:copy] Failed to copy build\tmp\expandedArchives\netty-3.6.6.Final.jar_39m6h0e7fj2p01jrbtakogpl9l\META-INF\license\LICENSE.base64.txt to build\tmp\fatjar-stage\META-INF\license\LICENSE.base64.txt due to failed to create the parent directory for build\tmp\fatjar-stage\META-INF\license\LICENSE.base64.txt

I have not been able to find a way to tell gradle to exclude or ignore duplicates when it is copying things into the fatjar-stage directory. All the options I have found seem to control things when you are building your own jar. In this case it is the preparation of the fat jar files that is failing.

I am very new to gradle, so please include specific code that I would need to use to get around this.

Thanks.


(Peter Niederwieser) #2

Please show how you are creating the fat Jar.


(Bill Darbie) #3

Here is my build.gradle file:

defaultTasks ‘build’

apply plugin: ‘java’ apply plugin: ‘eclipse’ apply plugin: ‘application’ apply plugin: ‘maven’ apply plugin: ‘jacoco’

eclipse {

// project { name = “${projectName}” }

classpath {

downloadSources = true

downloadJavadoc = true

} }

task wrapper(type: Wrapper) {

gradleVersion = ‘1.8’

jarFile = ‘tools/wrapper.jar’ }

buildscript {

repositories { mavenCentral() }

dependencies {

classpath ‘org.apache.maven.wagon:wagon-http:2.2’

classpath ‘eu.appsatori:gradle-fatjar-plugin:0.2’

classpath ‘org.codehaus.groovy.modules.http-builder:http-builder:0.5.2’

classpath ‘info.solidsoft.gradle.pitest:gradle-pitest-plugin:0.30.1’

classpath ‘org.pitest:pitest:0.30’

} }

apply plugin: ‘fatjar’ apply plugin: ‘pitest’ apply from: ‘tools/pmd.gradle’ apply from: ‘tools/findbugs.gradle’ apply from: ‘tools/checkstyle.gradle’ apply from: ‘tools/wizardrunner.gradle’ apply from: ‘tools/pelco-tasks.gradle’

version = currentVersion status = buildStatus mainClassName = “com.pelco.aspen.application.AspenService” sourceCompatibility = 1.7 description ‘core aspen’ group ‘com.pelco.aspen’

configurations {

specs { resolutionStrategy { cacheDynamicVersionsFor 300, ‘seconds’ } }

all*.exclude group: ‘org.mortbay.jetty’, module: ‘jetty-util’

all*.exclude group: ‘org.mortbay.jetty’, module: ‘jetty’

all*.exclude group: ‘org.mortbay.jetty’, module: ‘servlet-api’

all*.exclude group: ‘org.hectorclient’, module: ‘hector-core’

bndConfiguration }

artifacts { archives sourceJar, javadocJar, testJar, reportJar, fatJar }

uploadArchives {

repositories.mavenDeployer {

repository(url: “${artifactory}/${pubRepo}”) {

authentication(userName: “${pubUser}”, password: “${pubPassword}”)

}

} }

repositories {

//mavenLocal()

mavenCentral()

maven { url ‘https://oss.sonatype.org/content/groups/public’ }

maven { url “${artifactory}/${pubRepo}” } }

configurations.all {

resolutionStrategy.eachDependency { DependencyResolveDetails details ->

if (details.requested.name == ‘jaxb-impl’) {

// prefer version provided with Jersey 1.17

details.useTarget “com.sun.xml.bind:jaxb-impl:2.2.3-1”

}

if (details.requested.name == ‘libthrift’ && details.requested.version == ‘0.7.0’) {

// incompatible with Jersey - attempts to bring in servelt api 2.5

details.useTarget “org.apache.thrift:libthrift:0.9.0”

}

if (details.requested.name == ‘slf4j-log4j12’) {

// substitute ‘slf4j-log4j12’ with ‘log4j-over-slf4j’

details.useTarget “org.slf4j:log4j-over-slf4j:1.+”

}

if (details.requested.name == ‘log4j’) {

// substitute ‘log4j’ with ‘log4j-over-slf4j’

details.useTarget “org.slf4j:log4j-over-slf4j:1.+”

}

if (details.requested.group == ‘io.netty’ && details.requested.name == ‘netty’) {

// prefer version provided with AsyncHttpClient 1.7.20

details.useVersion ‘3.6.6.Final’

}

if (details.requested.group == ‘javax.servlet’ && details.requested.name == ‘servlet-api’ && details.requested.version == ‘2.5’) {

// prefer version supplied by Jersey 1.17

details.useVersion ‘3.0.1’

}

} }

dependencies { // When branching, we will statically fix the version numbers to help guarantee repeatable builds // of the branch in the future // using the command ‘gradle dependencies | grep “.+”’ will show you the unpinned dependencies // and what they are currently resolving to so you can supply the fixed version.

// mongodb

compile “org.mongodb:mongo-java-driver:2.11.3”

// DataStax Cassandra client

//compile ‘com.datastax.cassandra:cassandra-driver-core:1.+’

// compile ‘com.datastax.cassandra:cassandra-driver-core:1.0.4-dse’

// spec tests

// To pick up only the spec tests that are built from a branch,

// you would probably match the version Major.Minor.+ to the Aspen branch Major.Minor

specs “com.pelco.aspen:aspen-specs:0.+”

specs “com.pelco.aspen:aspen-specs:0.+@yml”

specs “com.pelco.aspen:aspen-specs:0.+@cfg”

// util Guava

// https://issues.apache.org/jira/browse/CASSANDRA-6007 (Guava 15)

compile ‘com.google.guava:guava:14.+’

// dropwizard

compile ‘com.yammer.dropwizard:dropwizard-core:0.6.2’

compile ‘com.yammer.dropwizard:dropwizard-auth:0.6.2’

// compile ‘com.yammer.dropwizard:dropwizard-client:0.6.2’

// crypto support

// compile ‘org.jasypt:jasypt:1.9.0:lite’ // small version

// compile ‘org.jasypt:jasypt:1.9.0’

// fat version

// compile ‘org.apache.shiro:shiro-core:1.2.1’ // http://shiro.apache.org/download.html

// Aspen client support

compile ‘com.ning:async-http-client:1.7.20’

compile ‘com.sun.jersey:jersey-client:1.17’

compile ‘org.jfarcand:jersey-ahc-client-1-0-4-pelco:1.0.24’

// compile ‘com.sun.jersey.contribs:jersey-apache-client4:1.15’

// compile ‘org.apache.httpcomponents:httpclient:4.2.2’

// compile ‘com.yammer.metrics:metrics-httpclient:2.2.0’

// log4j-over-slf4j

// logback-core

// logback-parent

// logback-classic

// log4j-over-slf4j

// jul-to-slf4j

// hibernate-validator

// hibernate-validator-parent

// validation-api

// Dependency Injection

compile ‘com.google.inject:guice:3.0’

// compile ‘com.google.inject.extensions:guice-servlet:3.0’

// Logger

// compile ‘org.slf4j:slf4j-simple:1.6.1’

// util Jackson

compile ‘com.fasterxml.jackson.core:jackson-annotations:2.2.+’

compile ‘com.fasterxml.jackson.core:jackson-core:2.2.+’

compile ‘com.fasterxml.jackson.core:jackson-databind:2.2.+’

compile ‘com.fasterxml.jackson.datatype:jackson-datatype-joda:2.2.+’

compile ‘com.fasterxml.jackson.datatype:jackson-datatype-guava:2.2.+’

// Servlet support

// providedCompile ‘javax.servlet:servlet-api:2.5’ /* provided by jersey-bundle */

// Jersey servlet support

// compile ‘com.sun.jersey:jersey-bundle:1.17’ /* provided by dropwizard */

// compile ‘com.sun.jersey:jersey-project:1.17’

// compile ‘com.sun.jersey:jersey-core:1.17’ /* provided by jersey-bundle */

// compile ‘com.sun.jersey:jersey-server:1.17’ /* provided by jersey-bundle */

// compile ‘com.sun.jersey:jersey-servlet:1.17’ /* provided by jersey-bundle */

// compile ‘com.sun.jersey:jersey-json:1.17’ /* provided by jersey-bundle */

// Jersey extensions

// compile ‘com.sun.jersey.contribs:jersey-guice:1.17’

// compile ‘com.sun.jersey.contribs:jersey-multipart:1.17’

// Atmosphere websockets

compile ‘org.atmosphere:atmosphere-runtime:2.0.+’

compile ‘org.atmosphere:atmosphere-annotations:2.0.+’

// Jetty websockets

compile ‘org.eclipse.jetty:jetty-websocket:8.1.+’

// Apache commons lang3

compile ‘org.apache.commons:commons-lang3:3.+’

// test

testCompile ‘junit:junit:4.+’

testCompile ‘org.mockito:mockito-all:1.9.+’

testCompile ‘org.powermock:powermock-api-mockito:1.5.+’

testCompile ‘org.powermock:powermock-module-junit4:1.5.+’

testCompile ‘org.easytesting:fest-assert-core:2.+’

testCompile ‘com.sun.jersey.jersey-test-framework:jersey-test-framework-core:1.17’

testCompile ‘com.sun.jersey.jersey-test-framework:jersey-test-framework-inmemory:1.17’

testCompile(‘com.sun.jersey.jersey-test-framework:jersey-test-framework-grizzly:1.17’) {

exclude group: ‘javax.servlet’, module: ‘servlet-api’ /* Provided by dropwizard transitive dependencies */

}

testCompile ‘de.flapdoodle.embed:de.flapdoodle.embed.mongo:1.+’

testCompile ‘org.glassfish.tyrus:tyrus-client:1.+’

testCompile ‘org.glassfish.tyrus:tyrus-container-grizzly-client:1.+’

testCompile ‘com.yammer.dropwizard:dropwizard-testing:0.6.2’

// OSGI runtime

compile ‘org.osgi:osgi_R4_core:1.0’, {

ext {

// don’t include this dependency in fatJar, because it’s already part of OSGI

// container runtime

fatJarExclude = true

}

}

compile ‘org.osgi:osgi_R4_compendium:1.0’, {

ext {

// don’t include this dependency in fatJar, because it’s already part of OSGI

// container runtime

fatJarExclude = true

}

}

// Add the bnd tool dependency so we can make fatjar OSGI compliant

bndConfiguration ‘biz.aQute.bnd:bnd:2.2.0’ }

pitest {

targetClasses = [‘com.pelco.*’] }

task copyAspenSiteConfig << {

if ( aspenConfig.equals(“specs”) ) {

println ‘using aspen-specs.cfg’

copy {

from configurations.specs.filter { it.toString().endsWith(".cfg") }.getSingleFile()

into ‘config/’

rename { String fileName -> fileName = “aspen-specs.cfg” }

}

} }

run {

dependsOn copyAspenSiteConfig

def config = aspenConfig.equals(“specs”) ? configurations.specs.filter { it.toString().endsWith(".yml") }.getSingleFile() : aspenConfig

args = [“server”, config ] }

startScripts {

applicationName = ‘aspen’

optsEnvironmentVar= ‘OPTS’

exitEnvironmentVar = ‘EXIT’ }

distZip {

//including the config folder into distZip folder

into ("${project.name}-${version}") {

from ‘.’

include ‘config/*’

} }

fatJar {

classifier ‘fat’

exclude ‘META-INF/INDEX.LIST’

exclude ‘META-INF/DEPENDENCIES’

exclude ‘META-INF/LICENSE’

exclude ‘META-INF/LICENSE.txt’

exclude ‘META-INF/NOTICE’

exclude ‘META-INF/NOTICE.txt’

exclude ‘META-INF/ASL2.0’

exclude ‘META-INF/*.SF’

exclude ‘META-INF/*.MF’

exclude ‘META-INF/*.DSA’

exclude ‘META-INF/*.RSA’

exclude ‘about.html’

exclude ‘jetty-dir.css’

exclude ‘LICENSE’

exclude ‘NOTICE’

exclude ‘about_files’

exclude ‘plugin.properties’

manifest {

attributes ‘Main-Class’: ‘com.pelco.aspen.application.AspenService’,

‘Implementation-Title’: “Aspen Service”,

‘Implementation-Version’: “${currentVersion}”,

‘Implementation-Vendor’: “Pelco”,

‘Bundle-Activator’: ‘com.pelco.aspen.application.AspenBundleActivator’

} }

// Makes an osgi compatible fatjar, currently outputing to repos/aspen/trunk/aspen task osgiFatJar(type: Exec, dependsOn: fatJar) {

// Download bndtool and store path

String bndToolName;

configurations.bndConfiguration.resolve().each {

if (it.getAbsolutePath().contains(“bnd”) && it.getAbsolutePath().endsWith(".jar")) {

bndToolName = it.getAbsolutePath()

}

}

String bndProperties = “config/aspen-trunk-dev-fat.bnd”

// run bnd command line tool to add correct import/export statements to the fatJar manifest

commandLine ‘java’, ‘-jar’, “${bndToolName}”, ‘wrap’, ‘–properties’, “${bndProperties}”, “${project.fatJar.archivePath}” }

// Copy the bnd generated fatJar from aspen root to it’s proper place in build/libs, // then delete the copy in aspen root. osgiFatJar.doLast {

copy {

String archivePath = “${project.fatJar.archivePath}”

int indexLastSlash = archivePath.lastIndexOf("/");

String copyPath = archivePath.substring(0, indexLastSlash);

from “${project.fatJar.archiveName}”

into “${copyPath}”

}

delete “${project.fatJar.archiveName}” }

task runFatJar(type:JavaExec, dependsOn: [fatJar, copyAspenSiteConfig] ) {

main = ‘-jar’

def config = aspenConfig.equals(“specs”) ? configurations.specs.filter { it.toString().endsWith(".yml") }.getSingleFile() : aspenConfig

args = ["${project.fatJar.archivePath}", “server”, “${config}” ]

classpath configurations.runtime }

// allow remote debugging on port 8001 task debug(type:JavaExec, dependsOn: [fatJar, copyAspenSiteConfig] ) {

main = ‘-jar’

def config = aspenConfig.equals(“specs”) ? configurations.specs.filter { it.toString().endsWith(".yml") }.getSingleFile() : aspenConfig

args = ["${project.fatJar.archivePath}", “server”, “${config}” ]

classpath configurations.runtime

jvmArgs = ["-Xdebug", “-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8001”] }

task httpcheck << {

def endpoint = “http://${specs_host}:${specs_admin_port}/ping”

def exitEx, count = 120

for (ii in 0…count) {

try { return HttpUtil.getClient(endpoint).request(HttpUtil.GET, HttpUtil.TEXT){} }

catch (Exception ex) { exitEx=ex; sleep 1000; }

}

throw exitEx }

task aspenExit << {

def endpoint = “http://${specs_host}:${specs_admin_port}/tasks/exit”

HttpUtil.getClient(endpoint).request(HttpUtil.POST){} }

task specs(type:JavaExec, dependsOn: httpcheck) {

def argList = []

argList.addAll(specs_args.split())

argList.addAll(specs_tags.split())

args = argList

main = “cucumber.api.cli.Main”

classpath configurations.specs }

class HttpUtil {

static def GET = groovyx.net.http.Method.GET

static def POST = groovyx.net.http.Method.POST

static def TEXT = groovyx.net.http.ContentType.TEXT

static groovyx.net.http.HTTPBuilder getClient(path) {

return new groovyx.net.http.HTTPBuilder(path)

} }

// display which test is being run on the console output test { // doFirst { //

println ‘test.doFirst2\n’ // } // // doLast { //

println ‘test.doLast2\n’ // }

beforeTest { desc ->

println “\nStarting test ${desc.className}.${desc.name}”

}

afterTest { desc, result ->

println “Finished test ${desc.className}.${desc.name} with result: ${result.resultType}”

}

// // have console output go to the console when running tests // // NOTE: uncommenting these lines prevents debugging from working in Netbeans // testLogging.showStandardStreams = true // logging.captureStandardOutput LogLevel.INFO

// always rerun all tests

outputs.upToDateWhen{false} }

//task startMongodb << { // println ‘starting up mongodb\n’ //} // //task stopMongodb << { // println ‘stopping mongodb\n’ //}

// make mongodb start and stop at the appropriate times //test.dependsOn startMongodb //stopMongodb.mustRunAfter test

//task startServer (type: Exec) { //

workingDir ‘tomcat/bin’ //

// using START hopefully forks the process //

commandLine ‘START’, ‘start.bat’ //

standardOutput = new ByteArrayOutputStream() //

ext.output = { //

return standardOutput.toString() //

} //

// loop through output stream for finished flag //

// or just put a timeout here //} // //task testIt (type: Test) { //

description “To test it.” //

include ‘org/foo/Test*.*’ //} // hints: http://docs.codehaus.org/display/GRADLE/Cookbook http://gradle.codehaus.org/Cookbook

/* Examples task getSpecsApp(type:Copy) {

configurations.specs.filter { it.toString().endsWith(".zip") }.each {

ext.currentSpecsDir = it.name.minus(’.zip’)

from zipTree(it)

into ‘specs’

} }

task specsApp(type:Exec, dependsOn:getSpecsApp) {

def argList = []

argList.addAll(specs_args.split())

argList.addAll(specs_tags.split())

args = argList

executable = “bin/aspen-specs”

workingDir = file(“specs/${getSpecs.currentSpecsDir}”) }

task runner(type:JavaExec, dependsOn: assemble) {

args = [“server”, aspenConfig ]

main mainClassName

classpath sourceSets.main.runtimeClasspath } */


(Peter Niederwieser) #4

Please only post the relevant parts, and wrap it with an HTML code block.


(Bill Darbie) #5

I was worried I might leave something out that was important. But here is a stripped down version:

buildscript {
    repositories { mavenCentral() }
    dependencies {
      classpath 'org.apache.maven.wagon:wagon-http:2.2'
      classpath 'eu.appsatori:gradle-fatjar-plugin:0.2'
      classpath 'org.codehaus.groovy.modules.http-builder:http-builder:0.5.2'
      classpath 'info.solidsoft.gradle.pitest:gradle-pitest-plugin:0.30.1'
      classpath 'org.pitest:pitest:0.30'
    }
}
  apply plugin: 'fatjar'
  fatJar {
  classifier 'fat'
  exclude 'META-INF/INDEX.LIST'
  exclude 'META-INF/DEPENDENCIES'
  exclude 'META-INF/LICENSE'
  exclude 'META-INF/LICENSE.txt'
  exclude 'META-INF/NOTICE'
  exclude 'META-INF/NOTICE.txt'
  exclude 'META-INF/ASL2.0'
  exclude 'META-INF/*.SF'
  exclude 'META-INF/*.MF'
  exclude 'META-INF/*.DSA'
  exclude 'META-INF/*.RSA'
  exclude 'about.html'
  exclude 'jetty-dir.css'
  exclude 'LICENSE'
  exclude 'NOTICE'
  exclude 'about_files'
  exclude 'plugin.properties'
  manifest {
    attributes 'Main-Class': 'com.pelco.aspen.application.AspenService',
               'Implementation-Title': "Aspen Service",
               'Implementation-Version': "${currentVersion}",
               'Implementation-Vendor': "Pelco",
               'Bundle-Activator': 'com.pelco.aspen.application.AspenBundleActivator'
  }
}

(Peter Niederwieser) #6

Thanks. Apparently you are using a third-party fat Jar plugin. I don’t know if that plugin is capable of dealing with conflicts, and if so how. You might want to consult the plugin’s documentation or contact the author (unless he is reading along here).


(Bill Darbie) #7

Is there a built in far jar that I can use that would give me the ability to get around this problem? If so I would be interested in trying that.


(Peter Niederwieser) #8

There is no built-in fat Jar plugin. Depending on your exact needs, you could use a conventional ‘Jar’ task (which allows to choose a duplicates strategy), or, if the goal is to create an executable Jar (i.e. an application rather than a library), the third-party gradle-one-jar plugin.