Jar file adding duplicate files in services folder

Old Forum
1

Recently I’ve had a problem where my jar file created by gradle ran differently than my configuration in eclipse, it didn’t take my very long to determine that problem, but rather I don’t know how to fix the problem in gradle.

Problem:

I’m running an embedded jetty in my desktop app and using resteasy as a restful framework. When I jar up my program it says that a JSON provider could not be found. The reason why it couldn’t find it is because I have more than one javax.ws.rs.ext.Providers file in my META-INF/services folder.(It looked like it did read the first one it found but my jettison provider wasn’t in the top file).

I verified that this was the problem by taking all the contents of the 3 files, combining them into one file, then deleting the 3 files and then adding my one combined file back to the jar, I ran the jar and it worked as expected.

So I understand the problem but I don’t have the experience in gradle to fix the problem, I was wondering if any of you had an idea for the solution to take any duplicates in the META-INF/services folder and combine the results for those files… I would think that it would be safe to merge them in all cases with the exception of the duplicate files maybe having duplciate services declared and in that case you would also have to root out duplicates again and just have the service declared once.

Here is my jar code in gradle

jar {
 from { configurations.compile.collect { it.isDirectory() ? it : zipTree(it) } }
 manifest {
  attributes("Main-Class": project.mainClass, "Implementation-Version": project.version)
 }
}

Here is what the contents of one of the Providers file looks like

org.jboss.resteasy.plugins.providers.DataSourceProvider
org.jboss.resteasy.plugins.providers.DocumentProvider
org.jboss.resteasy.plugins.providers.DefaultTextPlain
org.jboss.resteasy.plugins.providers.StringTextStar
org.jboss.resteasy.plugins.providers.InputStreamProvider
org.jboss.resteasy.plugins.providers.ByteArrayProvider
org.jboss.resteasy.plugins.providers.FormUrlEncodedProvider
org.jboss.resteasy.plugins.providers.FileProvider
org.jboss.resteasy.plugins.providers.StreamingOutputProvider
org.jboss.resteasy.plugins.providers.IIOImageProvider
org.jboss.resteasy.plugins.interceptors.CacheControlInterceptor
org.jboss.resteasy.plugins.interceptors.encoding.AcceptEncodingGZIPInterceptor
org.jboss.resteasy.plugins.interceptors.encoding.ClientContentEncodingHeaderInterceptor
org.jboss.resteasy.plugins.interceptors.encoding.GZIPDecodingInterceptor
org.jboss.resteasy.plugins.interceptors.encoding.GZIPEncodingInterceptor
org.jboss.resteasy.plugins.interceptors.encoding.ServerContentEncodingHeaderInterceptor

and another file

org.jboss.resteasy.plugins.providers.jaxb.json.JsonCollectionProvider
org.jboss.resteasy.plugins.providers.jaxb.json.JsonMapProvider
org.jboss.resteasy.plugins.providers.jaxb.json.JsonJAXBContextFinder
org.jboss.resteasy.plugins.providers.jaxb.json.JettisonElementProvider
org.jboss.resteasy.plugins.providers.jaxb.json.JettisonXmlRootElementProvider
org.jboss.resteasy.plugins.providers.jaxb.json.JettisonXmlSeeAlsoProvider
org.jboss.resteasy.plugins.providers.jaxb.json.JettisonXmlTypeProvider

so those would combine to look like this

org.jboss.resteasy.plugins.providers.DataSourceProvider
org.jboss.resteasy.plugins.providers.DocumentProvider
org.jboss.resteasy.plugins.providers.DefaultTextPlain
org.jboss.resteasy.plugins.providers.StringTextStar
org.jboss.resteasy.plugins.providers.InputStreamProvider
org.jboss.resteasy.plugins.providers.ByteArrayProvider
org.jboss.resteasy.plugins.providers.FormUrlEncodedProvider
org.jboss.resteasy.plugins.providers.FileProvider
org.jboss.resteasy.plugins.providers.StreamingOutputProvider
org.jboss.resteasy.plugins.providers.IIOImageProvider
org.jboss.resteasy.plugins.interceptors.CacheControlInterceptor
org.jboss.resteasy.plugins.interceptors.encoding.AcceptEncodingGZIPInterceptor
org.jboss.resteasy.plugins.interceptors.encoding.ClientContentEncodingHeaderInterceptor
org.jboss.resteasy.plugins.interceptors.encoding.GZIPDecodingInterceptor
org.jboss.resteasy.plugins.interceptors.encoding.GZIPEncodingInterceptor
org.jboss.resteasy.plugins.interceptors.encoding.ServerContentEncodingHeaderInterceptor
org.jboss.resteasy.plugins.providers.jaxb.json.JsonCollectionProvider
org.jboss.resteasy.plugins.providers.jaxb.json.JsonMapProvider
org.jboss.resteasy.plugins.providers.jaxb.json.JsonJAXBContextFinder
org.jboss.resteasy.plugins.providers.jaxb.json.JettisonElementProvider
org.jboss.resteasy.plugins.providers.jaxb.json.JettisonXmlRootElementProvider
org.jboss.resteasy.plugins.providers.jaxb.json.JettisonXmlSeeAlsoProvider
org.jboss.resteasy.plugins.providers.jaxb.json.JettisonXmlTypeProvider

Thank you for taking the time to look at and consider this problem.

2

It shouldn’t be necessary to merge these descriptors. Eclipse won’t do it either. Why does your Jar contain multiple descriptors in the first place, i.e. where do they originally come from?

Generally speaking, it’s perfectly normal to have multiple service descriptors with the same path name. I’m not completely sure if service/class loaders can handle the case where they are all contained in the same Jar, although I would think so.

3

They come from the jar files

resteasy-jaxrs-2.3.4.Final.jar resteasy-jettison-provider-2.3.4.Final.jar resteasy-jaxb-provider-2.3.4.Final.jar

Yes I also exported to a jar in Eclipse and had the same problem with that jar.

4

Why do you include these files in your Jar?

5

Are you asking why I include the javax.ws.rs.ext.Providers files in my jar??

These files are neccesary for the java Service Provider Interface, RestEasy keeps it’s data providers in seperate jars and it chooses to grab these data providers using service provider interface.

Although from above it sounds like you knew that already so I didn’t understand the question I guess.

6

Yes, that’s exactly my question. Why don’t you use the original Jars? Are you trying to create an uber Jar?

7

Oh I see what you are asking now, yes I am going for an uber jar.

8

I could give more details, I’m creating a jaxrs client using resteasy-client and sharing interfaces so that I can call resteasy services. I’m trying to make it a single jar that you can run against a configuration file and my jar will access the server, call some services and download an application specified in the configuration and launch it for you.

9

How about using Gradle’s application plugin instead? This plugin will automatically create a distribution zip containing all the necessary Jars along with a startup script.

I can’t tell offhand whether a class loader can load multiple resource files with the same path name from the same Jar. If not and you absolutely must use a single Jar, then you might have to merge the resource files. But first I’d evaluate other options.

10

Thanks for your help, My need for a single jar is not absolute but I could see it simplifying process for users. I’ll take a look at the application plugin.

If I were to give one last effort towards this do you know if there is a way that inside of the eachFile action http://gradle.org/docs/current/dsl/org.gradle.api.tasks.bundling.Jar.html if I could access files that are already in the jar or if I would be able to see files that are about to be placed in the jar? And if so I could then just exclude future files from being added.

If not that’s ok,

Thanks again for your help.

11

You could use ‘eachFile’ to remember the files that are being placed in the Jar and call ‘FileCopyDetails.exclude()’ for any duplicate you encounter. Though that alone won’t solve the problem.

In the future, we will provide an out-of-the-box option to prevent duplicate files in Jars if desired.

12

I think you were right before when you said that you think the class loader should be able to handle them all being in the same jar file. Having to merge them does seem kind of like a dirty solution.

I’m not saying that does work I’m just saying it does seem like it should work.

13

It looks like this plugin fixes the problem that I mention.

https://github.com/musketyr/gradle-fatjar-plugin

14

I’ve been struggling with this one my self and just finalized a solution. This allows multiple flavors and merges META-INF/services from the project and dependency libraries.

Just add the snipped below to your build.gradle file. It will merge META-INF/services files from dependency libraries and anything under your {sourceSet}/resources/META-INF/services.

android.applicationVariants.all { variant ->
    def t = task "merge${variant.name.capitalize()}ServiceFiles" {
        ext.depJars = variant.javaCompile.classpath
        ext.servicesDir = new File(variant.processJavaResources.destinationDir, "/META-INF/services/")
        inputs.files depJars
        doLast {
            servicesDir.mkdirs()
            depJars.each { jarFile ->
                // Find all META-INF/services/* files and merge their content
                // into product apk
                FileTree filtered = zipTree(jarFile).matching {
                    include "META-INF/services/*"
                }
                filtered.each { serviceFile ->
                    // Add service file into global packagingOptions exclusion list
                    // FIXME: any way to add variant specific packagingOptions?
                    android.packagingOptions.exclude("META-INF/services/" + serviceFile.name)
                    File target = new File(servicesDir, serviceFile.name)
                    serviceFile.eachLine {
                        // Add only lines that are valid implementation names and are not included already
                        // This is not really fool proof but works pretty well
                        if (!target.exists() || (it ==~ /^\s*\w.*/ && !target.getText("UTF-8").contains(it))) {
                            // println "Add " + it + " to META-INF/services/" + target.name
                            target << it
                            // If line did not end with new line, add one.
                            // Mainly to handle situation when missing new line from end of the file
                            if (!it.endsWith("\n")) {
                                target << "\n"
                            }
                        }
                    }
                }
            }
        }
    }
    // Run this task before createing application package
    // and after processJavaResources task to make sure
    //
    // processJavaResources will copy content from
    // {sourceSet}/resources/META-INF/services to intermediates
    // and destroys any existing content so we must merge
    // services files from dependency libraries only after
    // this step and also we need to run this task always
    // before packageApplication to populate
    // android.packagingOptions.excludes
    variant.packageApplication.dependsOn t
    t.dependsOn variant.processJavaResources
}