PermGen leak in ScalaCompile using the in-process ant builder

[afitz (Adam Fitzgerald)]

I am seeing a PermGen leak when building a primarily scala project with the in-process ant builder, since at least 2.7 and as recently as master @ 6b0067a.

I have pasted a simple reproduction below (as a patch, because I can’t attach any files).

The Scalac ant task is constructing 2 classloaders for the classpath and bootclasspath, and it looks like those classes are leaking into the system classloader (related to: http://melix.github.io/blog/2015/08/permgenleak.html?). As a test of this I hacked together a version of LeakyOnJava7GroovySystemLoader and DefaultIsolatedAntBuilder to explicitly clean scala.* metadata from the classloaders. This resulted in more space being reclaimed when under memory pressure, but presumably it is still missing some other things that continue to leak because in the below reproduction we still end up OOM.

The scala version we are using is 2.10.4, and java is:
Java™ SE Runtime Environment (build 1.7.0_21-b11)
Java HotSpot™ 64-Bit Server VM (build 23.21-b01, mixed mode)

Thanks,
Adam

Reproduction

diff --git a/build.gradle b/build.gradle
new file mode 100644
index 0000000..56b78f1
--- /dev/null
+++ b/build.gradle
@@ -0,0 +1,27 @@
+apply plugin: 'scala'
+
+configurations {
+       scalaTools
+}
+
+dependencies {
+       scalaTools /* scala dependency */
+       compile /* scala dependency */
+}
+
+compileScala.scalaClasspath = configurations.scalaTools
+
+for(int i = 0; i < 1000; i++) {
+       sourceSets.create('test'+i) {
+               scala {
+                       srcDirs = ['src/main/scala']
+                       compileClasspath = configurations.compile
+               }
+       }
+       compileScala.dependsOn('compileTest'+i+'Scala')
+}
+
+tasks.withType(ScalaCompile).all {
+       scalaClasspath = configurations.scalaTools
+}
+
diff --git a/gradle.properties b/gradle.properties
new file mode 100644
index 0000000..4dfae82
--- /dev/null
+++ b/gradle.properties
@@ -0,0 +1 @@
+org.gradle.jvmargs=-Xms1g -Xmx1g -XX:PermSize=256m -XX:MaxPermSize=256m
diff --git a/src/main/scala/toplevel.scala b/src/main/scala/toplevel.scala
new file mode 100644
index 0000000..e69de29

experimental changes

a/subprojects/core/src/main/groovy/org/gradle/api/internal/classloading/LeakyOnJava7GroovySystemLoader.java
+++ b/subprojects/core/src/main/groovy/org/gradle/api/internal/classloading/LeakyOnJava7GroovySystemLoader.java
@@ -108,6 +108,27 @@ public class LeakyOnJava7GroovySystemLoader implements GroovySystemLoader {
         }
     }

+    public void discardTypesFromPackage(String prefix) {
+        // Remove cached value for every class seen by this ClassLoader that was loaded by the given ClassLoader
+        try {
+            Iterator<?> it = globalClassSetIterator();
+            while (it.hasNext()) {
+                Object classInfo = it.next();
+                if (classInfo != null) {
+                    Class clazz = (Class) clazzField.get(classInfo);
+                    if (clazz.getName().startsWith(prefix)) {
+                        removeFromGlobalClassValue.invoke(globalClassValue, clazz);
+                        if (LOG.isDebugEnabled()) {
+                            LOG.debug(String.format("Removed ClassInfo from %s loaded by %s", clazz.getName(), clazz.getClassLoader()));
+                        }
+                    }
+                }
+            }
+        } catch (Exception e) {
+            throw new GradleException("Could not remove types for package prefix '" + prefix + "' from the Groovy system " + leakingLoader, e);
+        }
+    }
+
     private Iterator<?> globalClassSetIterator() throws IllegalAccessException, InvocationTargetException {
         return (Iterator) globalClassSetIteratorMethod.invoke(globalClassSetItems);
     }

--- a/subprojects/core/src/main/groovy/org/gradle/api/internal/project/antbuilder/DefaultIsolatedAntBuilder.java
+++ b/subprojects/core/src/main/groovy/org/gradle/api/internal/project/antbuilder/DefaultIsolatedAntBuilder.java
@@ -21,6 +21,7 @@ import org.gradle.api.Action;
 import org.gradle.api.internal.ClassPathRegistry;
 import org.gradle.api.internal.classloading.GroovySystemLoader;
 import org.gradle.api.internal.classloading.GroovySystemLoaderFactory;
+import org.gradle.api.internal.classloading.LeakyOnJava7GroovySystemLoader;
 import org.gradle.api.internal.project.IsolatedAntBuilder;
 import org.gradle.api.logging.LogLevel;
 import org.gradle.api.logging.Logger;
@@ -205,6 +206,9 @@ public class DefaultIsolatedAntBuilder implements IsolatedAntBuilder, Stoppable
         // Remove classes from core Gradle API
         gradleApiGroovyLoader.discardTypesFrom(antAdapterLoader);
         gradleApiGroovyLoader.discardTypesFrom(antLoader);
+        if(gradleApiGroovyLoader instanceof LeakyOnJava7GroovySystemLoader) {
+            ((LeakyOnJava7GroovySystemLoader)gradleApiGroovyLoader).discardTypesFromPackage("scala.");
+        }

         // Shutdown the adapter Groovy system
         antAdapterGroovyLoader.shutdown();

[CedricChampeau (Cédric Champeau)]

Hi Adam,

I am surprised to see an actual difference with your patch. What the leaking prevention strategy does is actually “simple” : it gets all classes that are known to Groovy and potentially leaked, and cleans them, independently of where they come from. So Scala classes should be cleaned too. One possibility here is that the scala task creates a classloader which is not “discoverable” by Gradle, hence not recoverable. That is to say, the lifecycle of this classloader would be within the Ant task, and the Groovy runtime would have access to it. I’m not familiar enough with the scala compiler to tell if this could happen, but what you are saying suggests it is the case.

If so, then the strategy should be applied on the whole classloader, not limiting it to scala classes.

Last but not least, we might want to consider upgrading to Groovy 2.4.5, which temporarily disabled usage of ClassValue, until the JDK team fixes the bug.

[afitz (Adam Fitzgerald)]

Hi Cedric,

Thanks for the suggestions!

So Scala classes should be cleaned too. One possibility here is that the scala task creates a classloader which is not “discoverable” by Gradle, hence not recoverable.

I think what is happening is that the classloaders created by the scala have a lifetime bound by the ant task execution at most, so they may not exist at the time we’re doing the cleanup. I don’t have hard evidence of the ordering, but the presence of the classloaders in heap dumps taken during execution, and absence in dumps taken in-between/after lend some support to that idea. It’s not clear to me that this would directly cause the classes to not be ‘discoverable’ by the cleanup mechanism - is that plausible?

Some more data points:
I’ve rerun the above reproduction case with groovy rebuilt against 2.4.5, and there is still a permgen leak - whether its the same or different cause I’ve not had a chance to verify.

I saw a comment from you on [GROOVY-7591] Use of ClassValue causes major memory leak - ASF JIRA indicating you were able to find no leak with 2.3.11 and so gave a downgrade to 2.3.11 a try. hHen building the above reproducing project I still observe a PermGen leak and eventual OOM, though a bit more slowly.

Thanks,
Adam

[CedricChampeau (Cédric Champeau)]

I think what is happening is that the classloaders created by the scala have a lifetime bound by the ant task execution at most, so they may not exist at the time we’re doing the cleanup. I don’t have hard evidence of the ordering, but the presence of the classloaders in heap dumps taken during execution, and absence in dumps taken in-between/after lend some support to that idea. It’s not clear to me that this would directly cause the classes to not be ‘discoverable’ by the cleanup mechanism - is that plausible?

Yes, it is plausible. However if it was the same memory leak, I would suspect that the classloaders would stay around, and be visible in the dump after the task execution. One thing that could help is if you could provide us with a dump before and after execution. I could then take a look and check if I see something that looks familiar.

[afitz (Adam Fitzgerald)]

Apologies for the delay following up. It turns out that this was due to the use of (leaking) ThreadLocals in the scala compiler under scala 2.10.4. Scala 2.11.6, another version we were able to test, does not leak in this way.

To workaround for scala 2.10.4 we are resorting to creating a new thread per AntScalaCompiler (not the cleanest implementation but it gets the point across):

diff --git a/subprojects/scala/src/main/groovy/org/gradle/api/internal/tasks/scala/AntScalaCompiler.groovy b/subprojects/scala/src/main/groovy/org/graindex 0f5a397..ab44ae7 100644
--- a/subprojects/scala/src/main/groovy/org/gradle/api/internal/tasks/scala/AntScalaCompiler.groovy
+++ b/subprojects/scala/src/main/groovy/org/gradle/api/internal/tasks/scala/AntScalaCompiler.groovy
@@ -27,6 +27,7 @@ import org.slf4j.LoggerFactory

 class AntScalaCompiler implements Compiler<ScalaCompileSpec> {
     private static final Logger LOGGER = LoggerFactory.getLogger(AntScalaCompiler)
+    private final Thread.UncaughtExceptionHandler uncaughtHandler = { t, ex -> t.failure = ex } as Thread.UncaughtExceptionHandler

     private final IsolatedAntBuilder antBuilder
     private final Iterable<File> bootclasspathFiles
@@ -56,25 +57,42 @@ class AntScalaCompiler implements Compiler<ScalaCompileSpec> {
         LOGGER.debug("Ant scalac task options: {}", options)

         antBuilder.withClasspath(scalaClasspath).execute { ant ->
-            taskdef(resource: 'scala/tools/ant/antlib.xml')
+            final Runnable antAction = {
+                taskdef(resource: 'scala/tools/ant/antlib.xml')

-            "${taskName}"(options) {
-                spec.source.addToAntBuilder(ant, 'src', FileCollection.AntType.MatchingTask)
-                bootclasspathFiles.each {file ->
-                    bootclasspath(location: file)
-                }
-                extensionDirs.each {dir ->
-                    extdirs(location: dir)
-                }
-                compileClasspath.each {file ->
-                    classpath(location: file)
+                "${taskName}"(options) {
+                    spec.source.addToAntBuilder(ant, 'src', FileCollection.AntType.MatchingTask)
+                    bootclasspathFiles.each { file ->
+                        bootclasspath(location: file)
+                    }
+                    extensionDirs.each { dir ->
+                        extdirs(location: dir)
+                    }
+                    compileClasspath.each { file ->
+                        classpath(location: file)
+                    }
                 }
+            } as Runnable
+            final Thread antScalaCompilerThread = createAntScalaCompilerThread(antAction)
+            antScalaCompilerThread.start()
+            antScalaCompilerThread.join()
+            antScalaCompilerThread.with {
+                if (failure) { throw failure }
             }
         }

         return { true } as WorkResult
     }

+    private Thread createAntScalaCompilerThread(Runnable antAction) {
+        Thread t = new Thread(antAction, "AntScalaCompilerThread")
+        t.metaClass.failure = null
+        t.uncaughtExceptionHandler = uncaughtHandler
+
+        t
+    }
+
+
     private VersionNumber sniffScalaVersion(Iterable<File> classpath) {
         def classLoader = new URLClassLoader(classpath*.toURI()*.toURL() as URL[], (ClassLoader) null)
         try {

However, with both versions of scala we’re seeing a massive increase in compilation execution time (with and without this patch under scala 2.11) - but I’ll start a separate thread for that issue.