CachingPatternSpecFactory does occupy to much heap space

Bugs
,
1

The following issue occurred when updating gradle from 2.5 to 2.9 java.lang.OutOfMemoryError: GC overhead limit exceeded. I traced it back to the CachingPatternSpecFactory which is used by following code in my application (see copy task)

    FileTree zip = project.zipTree(it) // a very large zip file 2GB with about 300k+ files
    // only some folder should be extracted from the zip file
    config.requiredFolders.each { folder ->
        if (!project.file(config.getTargetDir(folder)).exists()) {
            logger.lifecycle("Copying additionally required folder ${folder}")
            project.copy {
                from zip
                into "."
                include "${folder}/**";
            }
        }
    }

Where config is a class which only provides some information about which folders to extract. This code is executed for several large zip files (2GB with about 300k+ files). It is possible to raise the Xmx value to 8GB then it will work but this should not be the solution.

The following screenshots outline the problem.

jvisualvm_classes_xmx_2G.png1612×206 39.8 KB

jvisualvm_gc_heap_xmx_2G.png1613×469 66.7 KB

jvisualvm_gc_heap_xmx_4G.png1633×575 108 KB

I would appreciate if you can provide a temporary solution till this problem is solved.

2

Have you tried using a single copy? This behaves differently if there are no requiredFolders (copies everything), but that is fixable.

project.copy {
    from project.zipTree(it) // a very large zip file 2GB with about 300k+ files
    into "."
    config.requiredFolders.each { folder ->
        if (!project.file(config.getTargetDir(folder)).exists()) {
            logger.lifecycle("Copying additionally required folder ${folder}")
            include "${folder}/**"
        }
    }
}
3

Thank you for the suggestion to use a single copy. This works fine.

4

Thanks, we’ll still look at the memory issue. Gradle shouldn’t die like that.

5

This seems to be a problem for zipTree/tarTree. Besides that, this is a problem on JDK6 for all directory tree operations. I’m working on a fix for 2.10 . The memory usage problem is caused by duplicate String instances in all of the RelativePath instances. The fix is to use the StringInterner based on Guava’s WeakInterner to re-duplicate the String instances (path segments) that are referenced by the RelativePath instances.