<html><head>
      <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
   <title>Preventing Memory Leaks</title><link rel="stylesheet" type="text/css" href="css/docbook.css"><meta name="generator" content="DocBook XSL Stylesheets V1.79.1"><meta name="keywords" content="jetty, servlet, servlet-api, cometd, http, websocket, eclipse, maven, java, server, software"><link rel="home" href="index.html" title="Jetty"><link rel="up" href="troubleshooting.html" title="Chapter&nbsp;33.&nbsp;Troubleshooting"><link rel="prev" href="troubleshooting-locked-files-on-windows.html" title="Troubleshooting Locked Files on Windows"><link rel="next" href="troubleshooting-slow-deployment.html" title="Troubleshooting Slow Deployment"><link xmlns:jfetch="java:org.eclipse.jetty.xslt.tools.JavaSourceFetchExtension" xmlns:fetch="java:org.eclipse.jetty.xslt.tools.SourceFetchExtension" xmlns:d="http://docbook.org/ns/docbook" xmlns:l="http://docbook.sourceforge.net/xmlns/l10n/1.0" xmlns:xslthl="http://xslthl.sf.net" xmlns:gcse="http://www.google.com" xmlns:date="http://exslt.org/dates-and-times" rel="shortcut icon" href="images/favicon.ico"><link rel="stylesheet" href="css/highlighter/foundation.css"><script src="js/highlight.pack.js"></script><script>
      hljs.initHighlightingOnLoad();
    </script><link type="text/css" rel="stylesheet" href="css/font-awesome/font-awesome.min.css"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><table xmlns:jfetch="java:org.eclipse.jetty.xslt.tools.JavaSourceFetchExtension" xmlns:fetch="java:org.eclipse.jetty.xslt.tools.SourceFetchExtension" xmlns:d="http://docbook.org/ns/docbook" xmlns:l="http://docbook.sourceforge.net/xmlns/l10n/1.0" xmlns:xslthl="http://xslthl.sf.net" xmlns:gcse="http://www.google.com" xmlns:date="http://exslt.org/dates-and-times"><tr><td style="width: 25%"><a href="http://www.eclipse.org/jetty"><img src="images/jetty-header-logo.png" alt="Jetty Logo"></a><br><span style="font-size: small">
            Version: 9.4.28-SNAPSHOT</span></td><td style="width: 50%"></td></tr></table><div xmlns:jfetch="java:org.eclipse.jetty.xslt.tools.JavaSourceFetchExtension" xmlns:fetch="java:org.eclipse.jetty.xslt.tools.SourceFetchExtension" xmlns:d="http://docbook.org/ns/docbook" xmlns:l="http://docbook.sourceforge.net/xmlns/l10n/1.0" xmlns:xslthl="http://xslthl.sf.net" xmlns:gcse="http://www.google.com" xmlns:date="http://exslt.org/dates-and-times" class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">Preventing Memory Leaks</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="troubleshooting-locked-files-on-windows.html"><i class="fa fa-chevron-left" aria-hidden="true"></i> Previous</a>&nbsp;</td><th width="60%" align="center">Chapter&nbsp;33.&nbsp;Troubleshooting<br><a accesskey="p" href="index.html"><i class="fa fa-home" aria-hidden="true"></i> Home</a></th><td width="20%" align="right">&nbsp;<a accesskey="n" href="troubleshooting-slow-deployment.html">Next <i class="fa fa-chevron-right" aria-hidden="true"></i></a></td></tr></table><hr></div><div xmlns:jfetch="java:org.eclipse.jetty.xslt.tools.JavaSourceFetchExtension" xmlns:fetch="java:org.eclipse.jetty.xslt.tools.SourceFetchExtension" xmlns:d="http://docbook.org/ns/docbook" xmlns:l="http://docbook.sourceforge.net/xmlns/l10n/1.0" xmlns:xslthl="http://xslthl.sf.net" xmlns:gcse="http://www.google.com" xmlns:date="http://exslt.org/dates-and-times" class="jetty-callout"><h5 class="callout"><a href="http://www.webtide.com/">Contact the core Jetty developers at
          <span class="website">www.webtide.com</span></a></h5><p>
 private support for your internal/customer projects ... custom extensions and distributions ... versioned snapshots for indefinite support ...
 scalability guidance for your apps and Ajax/Comet projects ... development services for sponsored feature development
      </p></div><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="preventing-memory-leaks"></a>Preventing Memory Leaks</h2></div></div></div><div class="toc"><dl class="toc"><dt><span class="section"><a href="preventing-memory-leaks.html#preventing-webapp-classloader-pinning">Preventing WebApp Classloader Pinning</a></span></dt><dt><span class="section"><a href="preventing-memory-leaks.html#jsp-bugs">JSP Bugs: Permgen Problems</a></span></dt><dt><span class="section"><a href="preventing-memory-leaks.html#jvm-bugs">JVM Bugs</a></span></dt></dl></div><p>If you have memory leaks, and you have thoroughly investigated tools like jconsole, yourkit, jprofiler, jvisualvm or any of the other profiling and analysis tools, and you can eliminate your code as the source of the problem, read the following sections about how to prevent memory leaks in your application.</p><div class="section"><div class="titlepage"><div><div><h3 class="title"><a name="preventing-webapp-classloader-pinning"></a>Preventing WebApp Classloader Pinning</h3></div></div></div><div class="blockquote"><blockquote class="blockquote"><div xmlns:jfetch="java:org.eclipse.jetty.xslt.tools.JavaSourceFetchExtension" xmlns:fetch="java:org.eclipse.jetty.xslt.tools.SourceFetchExtension" xmlns:d="http://docbook.org/ns/docbook" xmlns:l="http://docbook.sourceforge.net/xmlns/l10n/1.0" xmlns:xslthl="http://xslthl.sf.net" xmlns:gcse="http://www.google.com" xmlns:date="http://exslt.org/dates-and-times" class="note" style="margin-left: 0.5in; margin-right: 0.5in;"><h3 class="title"><i class="fa fa-asterisk" aria-hidden="true"></i> Note</h3><p>This feature is available for Jetty 7.6.6 and later.</p></div></blockquote></div><p>Code that keeps references to a webapp classloader can cause memory leaks.
These leaks fall generally into two categories: static fields and daemon threads.</p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem">A static field is initialized with the value of the classloader, which happens to be a webapp classloader; as Jetty undeploys and redeploys the webapp, the static reference lives on, meaning garbage collecting cannot occur for the webapp classloader.</li><li class="listitem">When Jetty starts as a daemon thread and is outside the lifecycle of the webapp, threads have references to the context classloader that created them, leading to a memory leak if that classloader belongs to a webapp.
For a good discussion of the issue see <a class="link" href="http://cdivilly.wordpress.com/tag/sun-awt-appcontext/" target="_top">Anatomy of a PermGen Memory Leak.</a></li></ul></div><p>We provide a number of <a class="link" href="http://www.eclipse.org/jetty/javadoc/9.4.28-SNAPSHOT//org/eclipse/jetty/util/preventers/package-summary.html" target="_top">workaround classes</a> that preemptively invoke the problematic code with the Jetty classloader, thereby ensuring the webapp classloader is not pinned.
Be aware that since some of the problematic code creates threads, you should be selective about which preventers you enable, and use only those that are specific to your application.</p><div class="section"><div class="titlepage"><div><div><h4 class="title"><a name="preventers-table"></a>Preventers</h4></div></div></div><p>Jetty includes the following preventers.</p><div class="informaltable"><table class="informaltable" border="1"><colgroup><col class="col_1"><col class="col_2"></colgroup><thead><tr><th align="left" valign="top">Preventer Name</th><th align="left" valign="top">Problem Addressed</th></tr></thead><tbody><tr><td align="left" valign="top"><p>AppContextLeakPreventer</p></td><td align="left" valign="top"><p>The call to <code class="literal">AppContext.getAppContext()</code> keeps a static reference to the context classloader. The JRE can invoke AppContext in many different places.</p></td></tr><tr><td align="left" valign="top"><p>AWTLeakPreventer</p></td><td align="left" valign="top"><p>The <code class="literal">java.awt.Toolkit</code> class has a static field that is the default toolkit.
Creating the default toolkit causes the creation of an <code class="literal">EventQueue</code>, which has a classloader field initialized with the thread context class loader.
See <a class="link" href="https://issues.jboss.org/browse/AS7-3733" target="_top">JBoss bug AS7-3733.</a></p></td></tr><tr><td align="left" valign="top"><p>DOMLeakPreventer</p></td><td align="left" valign="top"><p>DOM parsing can cause the webapp classloader to be pinned, due to the static field ` RuntimeException` of <code class="literal">com.sun.org.apache.xerces.internal.parsers.AbstractDOMParser.</code> <a class="link" href="http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6916498" target="_top">Oracle bug 6916498</a> specifically mentions that a heap dump might not identify the GCRoot as the uncollected loader, making it difficult to identify the cause of the leak.</p></td></tr><tr><td align="left" valign="top"><p>DriverManagerLeakPreventer</p></td><td align="left" valign="top"><p>The number of threads dedicated to accepting incoming connections.</p></td></tr><tr><td align="left" valign="top"><p>GCThreadLeakPreventer</p></td><td align="left" valign="top"><p>Calls to <code class="literal">sun.misc.GC.requestLatency</code> create a daemon thread that keeps a reference to the context classloader.
A known caller of this method is the RMI impl. See <a class="link" href="http://stackoverflow.com/questions/6626680/does-java-garbage-collection-log-entry-full-gc-system-mean-some-class-called" target="_top">Stackoverflow: Does java garbage collection log entry <span class="emphasis"><em>Full GC system</em></span> mean some class
called System.gc()?</a></p></td></tr><tr><td align="left" valign="top"><p>Java2DLeakPreventer</p></td><td align="left" valign="top"><p><code class="literal">sun.java2d.Disposer</code> keeps a reference to the classloader.
See <a class="link" href="https://issues.apache.org/bugzilla/show_bug.cgi?id=51687" target="_top">ASF bug 51687.</a></p></td></tr><tr><td align="left" valign="top"><p>LDAPLeakPreventer</p></td><td align="left" valign="top"><p>If <code class="literal">com.sun.jndi.LdapPoolManager</code> class is loaded and the system property `   com.sun.jndi.ldap.connect.pool.timeout` is set to a nonzero value, a daemon thread starts and keeps a reference to the context classloader.</p></td></tr><tr><td align="left" valign="top"><p>LoginConfigurationLeakPreventer</p></td><td align="left" valign="top"><p>The <code class="literal">javax.security.auth.login.Configuration</code> class keeps a static reference to the thread context classloader.</p></td></tr><tr><td align="left" valign="top"><p>SecurityProviderLeakPreventer</p></td><td align="left" valign="top"><p>Some security providers, such as <code class="literal">sun.security.pkcs11.SunPKCS11</code> start a deamon thread that traps the thread context classloader.</p></td></tr></tbody></table></div></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a name="configuring-preventers"></a>Configuring Preventers</h4></div></div></div><p>You can individually enable each preventer by adding an instance to a Server with the ` addBean(Object)` call. Here&#8217;s an example of how to do it in code with the <code class="literal">org.eclipse.jetty.util.preventers.AppContextLeakPreventer</code>:</p><pre xmlns:jfetch="java:org.eclipse.jetty.xslt.tools.JavaSourceFetchExtension" xmlns:fetch="java:org.eclipse.jetty.xslt.tools.SourceFetchExtension" xmlns:d="http://docbook.org/ns/docbook" xmlns:l="http://docbook.sourceforge.net/xmlns/l10n/1.0" xmlns:xslthl="http://xslthl.sf.net" xmlns:gcse="http://www.google.com" xmlns:date="http://exslt.org/dates-and-times"><code>Server server = new Server();
server.addBean(new AppContextLeakPreventer());</code></pre><p>You can add the equivalent in code to the <code class="literal">$JETTY_HOME/etc/jetty.xml</code> file or any jetty xml file that is configuring a Server instance.
Be aware that if you have more than one Server instance in your JVM, you should configure these preventers on just <span class="emphasis"><em>one</em></span> of them.
Here&#8217;s the example from code put into xml:</p><pre xmlns:jfetch="java:org.eclipse.jetty.xslt.tools.JavaSourceFetchExtension" xmlns:fetch="java:org.eclipse.jetty.xslt.tools.SourceFetchExtension" xmlns:d="http://docbook.org/ns/docbook" xmlns:l="http://docbook.sourceforge.net/xmlns/l10n/1.0" xmlns:xslthl="http://xslthl.sf.net" xmlns:gcse="http://www.google.com" xmlns:date="http://exslt.org/dates-and-times"><code>&lt;Configure id="Server" class="org.eclipse.jetty.server.Server"&gt;

   &lt;Call name="addBean"&gt;
    &lt;Arg&gt;
      &lt;New class="org.eclipse.jetty.util.preventers.AppContextLeakPreventer"/&gt;
    &lt;/Arg&gt;
   &lt;/Call&gt;

&lt;/Configure&gt;</code></pre></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a name="jsp-bugs"></a>JSP Bugs: Permgen Problems</h3></div></div></div><p>The JSP engine in Jetty is Jasper.
This was originally developed under the Apache Tomcat project, but over time many different project have forked it.
All Jetty versions up to 6 used Apache-based Jasper exclusively, with Jetty 6 using Apache Jasper only for JSP 2.0.
With the advent of JSP 2.1, Jetty 6 switched to using Jasper from Sun&#8217;s <a class="link" href="https://glassfish.java.net/" target="_top">Glassfish</a> project, which is now the reference implementation.</p><p>All forks of Jasper suffer from a problem whereby using JSP tag files puts the permgen space under pressure.
This is because of the classloading architecture of the JSP implementation.
Each JSP file is effectively compiled and its class loaded in its own classloader to allow for hot replacement.
Each JSP that contains references to a tag file compiles the tag if necessary and then loads it using its own classloader.
If you have many JSPs that refer to the same tag file, the tag&#8217;s class is loaded over and over again into permgen space, once for each JSP.
See <a class="link" href="http://java.net/jira/browse/GLASSFISH-3963" target="_top">Glassfish bug 3963</a> and <a class="link" href="https://issues.apache.org/bugzilla/show_bug.cgi?id=43878" target="_top">Apache bug 43878.</a>
The Apache Tomcat project has already closed this bug with status WON&#8217;T FIX, however the Glassfish folks still have the bug open and have scheduled it to be fixed.
When the fix becomes available, the Jetty project will pick it up and incorporate into our release program.</p></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a name="jvm-bugs"></a>JVM Bugs</h3></div></div></div><p>This section describes garbage collection and direct ByteBuffer problems.</p><div class="section"><div class="titlepage"><div><div><h4 class="title"><a name="jvm-garbage-collection-problems"></a>Garbage Collection Problems</h4></div></div></div><p>One symptom of a cluster of JVM related memory issues is the OOM exception accompanied by a message such as <code class="literal">java.lang.OutOfMemoryError: requested xxxx bytes for xxx.
Out of swap space?</code></p><p><a class="link" href="http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4697804" target="_top">Oracle bug 4697804</a> describes how this can happen in the scenario when the garbage collector needs to allocate a bit more space during its run and tries to resize the heap, but fails because the machine is out of swap space.
One suggested work around is to ensure that the JVM never tries to resize the heap, by setting min heap size to max heap size:</p><pre xmlns:jfetch="java:org.eclipse.jetty.xslt.tools.JavaSourceFetchExtension" xmlns:fetch="java:org.eclipse.jetty.xslt.tools.SourceFetchExtension" xmlns:d="http://docbook.org/ns/docbook" xmlns:l="http://docbook.sourceforge.net/xmlns/l10n/1.0" xmlns:xslthl="http://xslthl.sf.net" xmlns:gcse="http://www.google.com" xmlns:date="http://exslt.org/dates-and-times"><code>java -Xmx1024m -Xms1024m</code></pre><p>Another workaround is to ensure you have configured sufficient swap space on your device to accommodate all programs you are running concurrently.</p></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a name="direct-byte-buffers"></a>Direct ByteBuffers</h4></div></div></div><p>Exhausting native memory is another issue related to JVM bugs.
The symptoms to look out for are the process size growing, but heap use remaining relatively constant.
Both the JIT compiler and nio ByteBuffers can consume native memory.
<a class="link" href="http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6210541" target="_top">Oracle bug 6210541</a> discusses a still-unsolved problem whereby the JVM itself allocates a direct ByteBuffer in some circumstances while the system never garbage collects, effectively eating native memory.
Guy Korland&#8217;s blog discusses this problem <a class="link" href="http://www.jroller.com/gkorland/entry/java_s_memory_isn_t" target="_top">here</a> and <a class="link" href="http://www.jroller.com/gkorland/entry/java_s_memory_managment_is" target="_top">here.</a>
As the JIT compiler consumes native memory, the lack of available memory may manifest itself in the JIT as OutOfMemory exceptions such as <code class="literal">Exception in thread "CompilerThread0" java.lang.OutOfMemoryError: requested xxx bytes for ChunkPool::allocate. Out of swap
      space?</code></p><p>By default, Jetty allocates and manages its own pool of direct ByteBuffers for io if you configure the nio SelectChannelConnector.
It also allocates MappedByteBuffers to memory-map static files via the DefaultServlet settings.
However, you could be vulnerable to this JVM ByteBuffer allocation problem if you have disabled either of these options.
For example, if you&#8217;re on Windows, you may have disabled the use of memory-mapped buffers for the static file cache on the DefaultServlet to avoid the file-locking problem.</p></div></div></div><script type="text/javascript">
      SyntaxHighlighter.all()
    </script><div class="navfooter"><hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="troubleshooting-locked-files-on-windows.html"><i class="fa fa-chevron-left" aria-hidden="true"></i> Previous</a>&nbsp;</td><td width="20%" align="center"><a accesskey="u" href="troubleshooting.html"><i class="fa fa-chevron-up" aria-hidden="true"></i> Top</a></td><td width="40%" align="right">&nbsp;<a accesskey="n" href="troubleshooting-slow-deployment.html">Next <i class="fa fa-chevron-right" aria-hidden="true"></i></a></td></tr><tr><td width="40%" align="left" valign="top">Troubleshooting Locked Files on Windows&nbsp;</td><td width="20%" align="center"><a accesskey="h" href="index.html"><i class="fa fa-home" aria-hidden="true"></i> Home</a></td><td width="40%" align="right" valign="top">&nbsp;Troubleshooting Slow Deployment</td></tr></table></div><p xmlns:jfetch="java:org.eclipse.jetty.xslt.tools.JavaSourceFetchExtension" xmlns:fetch="java:org.eclipse.jetty.xslt.tools.SourceFetchExtension" xmlns:d="http://docbook.org/ns/docbook" xmlns:l="http://docbook.sourceforge.net/xmlns/l10n/1.0" xmlns:xslthl="http://xslthl.sf.net" xmlns:gcse="http://www.google.com" xmlns:date="http://exslt.org/dates-and-times"><div class="jetty-callout">
            See an error or something missing?
            <span class="callout"><a href="http://github.com/eclipse/jetty.project">Contribute to this documentation at
                <span class="website"><i class="fa fa-github" aria-hidden="true"></i> Github!</span></a></span><span style="float: right"><i>(Generated: 2020-03-10)</i></span></div></p></body></html>