Memory leaks

[jglick]

When running a flow repeatedly, the WorkflowScript (a dynamic CpsScript subclass) is not promptly collected, nor is its GroovyClassLoader.InnerLoader and thus an assortment of derivative classes and objects. Demo:

jmap -histo $pid | fgrep WorkflowScript | wc -l

prints the number of builds you have just run. (Note that each line is a class called WorkflowScript from its own class loader.)

I think I have fixed all the hard-reference leaks, so Jenkins will collect the remaining SoftReferences under heavy memory pressure rather than throwing an OutOfMemoryError; to demo, using the current patch run in /script

for (i = 1; ; i *= 2) {
  System.gc();
  new Object[i];
}

and the WorkflowScript count will drop to 0. Sometimes this also happens after merely

System.gc();

But in the meantime the heap (or in Java 6/7, permanent generation) slowly grows.

@reviewbybees (ref. ZD-24759)

• reproduce hard leaks in functional test
• fix CpsFlowExecution.shell leak
• fix Caller.Info leak
• for above leak, integrate Memory leak cloudbees/groovy-cps#2 once merged and released
• fix SerializableClassRegistry leak (arguably a bug in JBoss Marshalling, though JDK-6389107 makes it hard to fix)
• fix java.beans.Introspector leak (also related to JDK-6389107)
• fix org.codehaus.groovy.reflection.ClassInfo$ClassInfoSet$Segment.table leak, or hope that groovy/groovy-core@97d78e9 in 2.4.0-beta-2 solves this (cf. [JENKINS-21249] Updated Groovy to 2.4.5 jenkins#1085)
• changelog

[stephenc]

(checkpoint) changes so far look ok to me, but I lack the context to be certain

[jglick]

Actually not looking too bad at the moment in Java 8. If I set the job to rebuild itself repeatedly (once per second), and monitor the class count, it climbs to around 50–60 before spontaneously clearing.

The ClassInfo leak in detail, from MAT:

Class Name                                                                                             | Ref. Objects | Shallow Heap | Ref. Shallow Heap | Retained Heap
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
java.lang.Thread @ 0x708730d28  Computer.threadPoolForRemoting [#5] Native Stack, Thread               |           16 |          120 |               768 |         3,504
'- contextClassLoader org.jenkinsci.maven.plugins.hpi.RunMojo$2 @ 0x7075d4208  /jenkins                |           16 |           96 |               768 |     2,002,944
   '- classes java.util.Vector @ 0x7075e15a0                                                           |           16 |           32 |               768 |     1,701,504
      '- elementData java.lang.Object[5120] @ 0x707878848                                              |           16 |       20,496 |               768 |     1,701,472
         '- [3552] class org.codehaus.groovy.reflection.ClassInfo @ 0x7079139a0                        |           16 |           24 |               768 |           232
            '- globalClassSet org.codehaus.groovy.reflection.ClassInfo$ClassInfoSet @ 0x707913a38      |           16 |           32 |               768 |           112
               '- segments org.codehaus.groovy.util.AbstractConcurrentMapBase$Segment[16] @ 0x707913a58|           16 |           80 |               768 |            80
                  |- [15] org.codehaus.groovy.reflection.ClassInfo$ClassInfoSet$Segment @ 0x707913aa8  |            1 |           48 |                48 |       828,488
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------

or from VisualVM:

this     - value: groovy.lang.GroovyClassLoader$InnerLoader #18
 <- <classLoader>     - class: WorkflowScript, value: groovy.lang.GroovyClassLoader$InnerLoader #18
  <- referent     - class: org.codehaus.groovy.util.ReferenceType$SoftRef, value: WorkflowScript class WorkflowScript
   <- ref     - class: org.codehaus.groovy.reflection.ClassInfo, value: org.codehaus.groovy.util.ReferenceType$SoftRef #2309
    <- [30]     - class: java.lang.Object[], value: org.codehaus.groovy.reflection.ClassInfo #462
     <- table     - class: org.codehaus.groovy.reflection.ClassInfo$ClassInfoSet$Segment, value: java.lang.Object[] #46259
      <- segment     - class: org.codehaus.groovy.reflection.ClassInfo, value: org.codehaus.groovy.reflection.ClassInfo$ClassInfoSet$Segment #13
       <- classInfo     - class: org.codehaus.groovy.reflection.stdclasses.ArrayCachedClass, value: org.codehaus.groovy.reflection.ClassInfo #151
        <- ARR_CLASS     - class: org.codehaus.groovy.runtime.dgmimpl.arrays.IntegerArrayGetAtMetaMethod, value: org.codehaus.groovy.reflection.stdclasses.ArrayCachedClass #11

[jglick]

Java 7 shows the same behavior of WorkflowScript count resetting to 0. However the number of InnerLoaders continues to increase and the permanent generation steadily grows until I force an OutOfMemoryError.

[jglick]

On a lark I tried

org.codehaus.groovy.util.ReferenceBundle.softBundle.type = org.codehaus.groovy.util.ReferenceBundle.weakBundle.type

but it does not seem to help.

[jglick]

Well even on Java 7 (u67) the permanent generation does apparently get collected if you are just patient enough; takes around 1000 InnerLoaders before that happens, but over time I get a sawtooth curve on the PermGen graph, with cycles around every 15 minutes on my machine. So I think I am declaring victory for now.

[tfennelly]

So the main thing so far has been the upgrade of groovy-cpsand the use of org.jboss.marshalling.reflect.SerializableClassRegistry and java.beans.Introspector to clean out the classes loaded by the script classloader?

[jglick]

And clearing the shell field after the build finishes.

[jglick]

@reviewbybees ready to merge from my perspective.

[tfennelly]

👍