When Gradle makes very large JAR files, they're corrupt


(Tom Anderson) #1

This build script generates a load of dummy resources, packs them into a jar, and verifies the jar using the command-line zip tool:

apply plugin: 'java'
  def generatedResourcesDir = new File(project.buildDir, "generated-resources")
  task generateResources << {
    def rnd = new Random()
    def buf = new byte[128 * 1024];
    for (x in 0..250) {
        def dir = new File(generatedResourcesDir, x.toString())
        dir.mkdirs()
        for (y in 0..250) {
            def file = new File(dir, y.toString())
            rnd.nextBytes(buf)
            file.bytes = buf
        }
    }
}
  sourceSets {
    main {
        output.dir(generatedResourcesDir, builtBy: generateResources)
    }
}
  task verifyJar(dependsOn: jar) << {
    def command = 'zip -T ' + jar.archivePath
    def process = command.execute()
    def status = process.waitFor()
    assert status == 0, "command ${command} failed: \"${process.text.trim()}\""
}
check.dependsOn verifyJar

When i run this with gradle 1.9, it fails with:

FAILURE: Build failed with an exception.
  * Where:
Build file '/home/tanderson/workspace/GradleZipBug/build.gradle' line: 37
  * What went wrong:
Execution failed for task ':verifyJar'.
> java.lang.AssertionError: command zip -T /home/tanderson/workspace/GradleZipBug/build/libs/GradleZipBug.jar failed: "zip warning: missing end signature--probably not a zip file (did you
   zip warning: remember to use binary mode when you transferred it?)
   zip warning: (if you are trying to read a damaged archive try -F)
      zip error: Zip file structure invalid (/home/tanderson/workspace/GradleZipBug/build/libs/GradleZipBug.jar)". Expression: (status == 0). Values: status = 3

I get the same error with 1.7, and a slightly different one with 1.8.

This is a pretty stupid build, but it’s a distillation of a genuine case i have where i am building an uberjar containing an application and all its dependencies that ends up being absolutely gigantic - about 90 MB. That is also a stupid build, but it’s the build i am currently chained to, and it’s not likely to get substantially less stupid any time soon. However, with that build, it works fine under 1.7, and fails under 1.8 ( i have yet to try it with 1.9). When i examine the jar files produced by that build using 1.7 and 1.8, i see that the corrupt file produced by 1.8 has a truncated end of central directory record at the end; rather than being the 22-byte structure described in the specification, it is only 8 bytes, suggesting a buffering issue somewhere.

If you play with the parameters of the dummy resource generation to make more, smaller, files, you can get different results: 1.7 produces jars with valid central directories, whereas 1.8 does not. Sadly, this test still fails, because the command-line zip program cannot handle archives with more than 65535 entries, even though the various bits of jar support in Java can.

My Java is:

java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)

My lsb_release -a is:

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 12.04.3 LTS
Release: 12.04
Codename: precise

Is this a known bug? Is there a workaround? Is this something to do with gradle, or a problem with the JDK?


(Luke Daley) #2

I’m not sure what this would be. We’ll have to dig deeper into it.

Raised as GRADLE-2964.


(gradleuser123) #3

Has any digging been performed? What’s the status on this?


(Szczepan Faber) #4

This may be related to the limitations of the ZipOutputStream that we use internally for handling zips. Large zips (>4g, >65k entries) need extra handling (Zip64). Typically this is handled automatically by ZipOutputStream, but not always (see org.apache.tools.zip.ZipOutputStream#setUseZip64).

The example project produces about 62k files that weight 8g, hence it might be related. If my theory stands, a change in Gradle is needed to support those cases.


(Luke Daley) #5

Fixed in 1.12.

You can set ‘useZip64 = true’ on all Zip tasks.


(Tom Anderson) #6

Awesome! Many thanks to you and MisterTea for sorting this out. I look forward to 1.12 going final so we can roll this out. Although my colleague GradleUser123 (not his real name) is already threatening to roll out the release candidate …