I am seeing a PermGen leak when building a primarily scala project with the in-process ant builder, since at least 2.7 and as recently as master @ 6b0067a.
I have pasted a simple reproduction below (as a patch, because I can’t attach any files).
The Scalac ant task is constructing 2 classloaders for the classpath and bootclasspath, and it looks like those classes are leaking into the system classloader (related to: http://melix.github.io/blog/2015/08/permgenleak.html?). As a test of this I hacked together a version of LeakyOnJava7GroovySystemLoader and DefaultIsolatedAntBuilder to explicitly clean scala.* metadata from the classloaders. This resulted in more space being reclaimed when under memory pressure, but presumably it is still missing some other things that continue to leak because in the below reproduction we still end up OOM.
The scala version we are using is 2.10.4, and java is:
Java™ SE Runtime Environment (build 1.7.0_21-b11)
Java HotSpot™ 64-Bit Server VM (build 23.21-b01, mixed mode)
I am surprised to see an actual difference with your patch. What the leaking prevention strategy does is actually “simple” : it gets all classes that are known to Groovy and potentially leaked, and cleans them, independently of where they come from. So Scala classes should be cleaned too. One possibility here is that the scala task creates a classloader which is not “discoverable” by Gradle, hence not recoverable. That is to say, the lifecycle of this classloader would be within the Ant task, and the Groovy runtime would have access to it. I’m not familiar enough with the scala compiler to tell if this could happen, but what you are saying suggests it is the case.
If so, then the strategy should be applied on the whole classloader, not limiting it to scala classes.
Last but not least, we might want to consider upgrading to Groovy 2.4.5, which temporarily disabled usage of ClassValue, until the JDK team fixes the bug.
So Scala classes should be cleaned too. One possibility here is that the scala task creates a classloader which is not “discoverable” by Gradle, hence not recoverable.
I think what is happening is that the classloaders created by the scala have a lifetime bound by the ant task execution at most, so they may not exist at the time we’re doing the cleanup. I don’t have hard evidence of the ordering, but the presence of the classloaders in heap dumps taken during execution, and absence in dumps taken in-between/after lend some support to that idea. It’s not clear to me that this would directly cause the classes to not be ‘discoverable’ by the cleanup mechanism - is that plausible?
Some more data points:
I’ve rerun the above reproduction case with groovy rebuilt against 2.4.5, and there is still a permgen leak - whether its the same or different cause I’ve not had a chance to verify.
I saw a comment from you on [GROOVY-7591] Use of ClassValue causes major memory leak - ASF JIRA indicating you were able to find no leak with 2.3.11 and so gave a downgrade to 2.3.11 a try. hHen building the above reproducing project I still observe a PermGen leak and eventual OOM, though a bit more slowly.
I think what is happening is that the classloaders created by the scala have a lifetime bound by the ant task execution at most, so they may not exist at the time we’re doing the cleanup. I don’t have hard evidence of the ordering, but the presence of the classloaders in heap dumps taken during execution, and absence in dumps taken in-between/after lend some support to that idea. It’s not clear to me that this would directly cause the classes to not be ‘discoverable’ by the cleanup mechanism - is that plausible?
Yes, it is plausible. However if it was the same memory leak, I would suspect that the classloaders would stay around, and be visible in the dump after the task execution. One thing that could help is if you could provide us with a dump before and after execution. I could then take a look and check if I see something that looks familiar.
Apologies for the delay following up. It turns out that this was due to the use of (leaking) ThreadLocals in the scala compiler under scala 2.10.4. Scala 2.11.6, another version we were able to test, does not leak in this way.
To workaround for scala 2.10.4 we are resorting to creating a new thread per AntScalaCompiler (not the cleanest implementation but it gets the point across):
However, with both versions of scala we’re seeing a massive increase in compilation execution time (with and without this patch under scala 2.11) - but I’ll start a separate thread for that issue.