<html><head>
      <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
   <title>High Load</title><link rel="stylesheet" type="text/css" href="css/docbook.css"><meta name="generator" content="DocBook XSL Stylesheets V1.79.1"><meta name="keywords" content="jetty, servlet, servlet-api, cometd, http, websocket, eclipse, maven, java, server, software"><link rel="home" href="index.html" title="Jetty"><link rel="up" href="optimizing.html" title="Chapter&nbsp;20.&nbsp;Optimizing Jetty"><link rel="prev" href="optimizing.html" title="Chapter&nbsp;20.&nbsp;Optimizing Jetty"><link rel="next" href="limit-load.html" title="Limiting Load"><link xmlns:jfetch="java:org.eclipse.jetty.xslt.tools.JavaSourceFetchExtension" xmlns:fetch="java:org.eclipse.jetty.xslt.tools.SourceFetchExtension" xmlns:d="http://docbook.org/ns/docbook" xmlns:l="http://docbook.sourceforge.net/xmlns/l10n/1.0" xmlns:xslthl="http://xslthl.sf.net" xmlns:gcse="http://www.google.com" xmlns:date="http://exslt.org/dates-and-times" rel="shortcut icon" href="images/favicon.ico"><link rel="stylesheet" href="css/highlighter/foundation.css"><script src="js/highlight.pack.js"></script><script>
      hljs.initHighlightingOnLoad();
    </script><link type="text/css" rel="stylesheet" href="css/font-awesome/font-awesome.min.css"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><table xmlns:jfetch="java:org.eclipse.jetty.xslt.tools.JavaSourceFetchExtension" xmlns:fetch="java:org.eclipse.jetty.xslt.tools.SourceFetchExtension" xmlns:d="http://docbook.org/ns/docbook" xmlns:l="http://docbook.sourceforge.net/xmlns/l10n/1.0" xmlns:xslthl="http://xslthl.sf.net" xmlns:gcse="http://www.google.com" xmlns:date="http://exslt.org/dates-and-times"><tr><td style="width: 25%"><a href="http://www.eclipse.org/jetty"><img src="images/jetty-header-logo.png" alt="Jetty Logo"></a><br><span style="font-size: small">
            Version: 9.3.28.v20191105</span></td><td style="width: 50%"></td></tr></table><div xmlns:jfetch="java:org.eclipse.jetty.xslt.tools.JavaSourceFetchExtension" xmlns:fetch="java:org.eclipse.jetty.xslt.tools.SourceFetchExtension" xmlns:d="http://docbook.org/ns/docbook" xmlns:l="http://docbook.sourceforge.net/xmlns/l10n/1.0" xmlns:xslthl="http://xslthl.sf.net" xmlns:gcse="http://www.google.com" xmlns:date="http://exslt.org/dates-and-times" class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">High Load</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="optimizing.html"><i class="fa fa-chevron-left" aria-hidden="true"></i> Previous</a>&nbsp;</td><th width="60%" align="center">Chapter&nbsp;20.&nbsp;Optimizing Jetty<br><a accesskey="p" href="index.html"><i class="fa fa-home" aria-hidden="true"></i> Home</a></th><td width="20%" align="right">&nbsp;<a accesskey="n" href="limit-load.html">Next <i class="fa fa-chevron-right" aria-hidden="true"></i></a></td></tr></table><hr></div><div xmlns:jfetch="java:org.eclipse.jetty.xslt.tools.JavaSourceFetchExtension" xmlns:fetch="java:org.eclipse.jetty.xslt.tools.SourceFetchExtension" xmlns:d="http://docbook.org/ns/docbook" xmlns:l="http://docbook.sourceforge.net/xmlns/l10n/1.0" xmlns:xslthl="http://xslthl.sf.net" xmlns:gcse="http://www.google.com" xmlns:date="http://exslt.org/dates-and-times" class="jetty-callout"><h5 class="callout"><a href="http://www.webtide.com/">Contact the core Jetty developers at
          <span class="website">www.webtide.com</span></a></h5><p>
 private support for your internal/customer projects ... custom extensions and distributions ... versioned snapshots for indefinite support ...
 scalability guidance for your apps and Ajax/Comet projects ... development services for sponsored feature development
      </p></div><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="high-load"></a>High Load</h2></div></div></div><div class="toc"><dl class="toc"><dt><span class="section"><a href="high-load.html#_load_generation_for_load_testing">Load Generation for Load Testing</a></span></dt><dt><span class="section"><a href="high-load.html#_operating_system_tuning">Operating System Tuning</a></span></dt></dl></div><p>Configuring Jetty for high load, whether for load testing or for production, requires that the operating system, the JVM, Jetty, the application, the network and the load generation all be tuned.</p><div class="section"><div class="titlepage"><div><div><h3 class="title"><a name="_load_generation_for_load_testing"></a>Load Generation for Load Testing</h3></div></div></div><p>Machines handling load generation must have their OS, JVM, etc., tuned just as much as the server machines.</p><p>The load generation should not be over the local network on the server machine, as this has unrealistic performance and latency as well as different packet sizes and transport characteristics.</p><p>The load generator should generate a realistic load.
Avoid the following pitfalls:</p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem">A common mistake is that load generators often open relatively few connections that are extremely busy sending as many requests as possible over each connection.
This causes the measured throughput to be limited by request latency (see <a class="link" href="http://blogs.webtide.com/gregw/entry/lies_damned_lies_and_benchmarks" target="_top">Lies, Damned Lies and Benchmarks</a> for an analysis of such an issue).</li><li class="listitem">Another common mistake is to use TCP/IP for a single request, and to open many, many short-lived connections.
This often results in accept queues filling and limitations due to file descriptor and/or port starvation.</li><li class="listitem">A load generator should model the traffic profile from the normal clients of the server.
For browsers, this is often between two and six connections that are mostly idle and that are used in sporadic bursts with read times in between.
The connections are typically long held HTTP/1.1 connections.</li><li class="listitem">Load generators should be written in asynchronously so that a limited number of threads does not restrict the maximum number of users that can be simulated.
If the generator is not asynchronous, a thread pool of 2000 may only be able to simulate 500 or fewer users.
The Jetty <code class="literal">HttpClient</code> is an ideal choice for building a load generator as it is asynchronous and can simulate many thousands of connections (see the CometD Load Tester for a good example of a realistic load generator).</li></ul></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a name="_operating_system_tuning"></a>Operating System Tuning</h3></div></div></div><p>Both the server machine and any load generating machines need to be tuned to support many TCP/IP connections and high throughput.</p><div class="section"><div class="titlepage"><div><div><h4 class="title"><a name="_linux"></a>Linux</h4></div></div></div><p>Linux does a reasonable job of self-configuring TCP/IP, but there are a few limits and defaults that you should increase.
You can configure most of these in <code class="literal">/etc/security/limits.conf</code> or via <code class="literal">sysctl</code>.</p><div class="section"><div class="titlepage"><div><div><h5 class="title"><a name="_tcp_buffer_sizes"></a>TCP Buffer Sizes</h5></div></div></div><p>You should increase TCP buffer sizes to at least 16MB for 10G paths and tune the auto-tuning (keep in mind that you need to consider buffer bloat).</p><div class="screenexample"><pre class="screen">$ sysctl -w net.core.rmem_max=16777216
$ sysctl -w net.core.wmem_max=16777216
$ sysctl -w net.ipv4.tcp_rmem="4096 87380 16777216"
$ sysctl -w net.ipv4.tcp_wmem="4096 16384 16777216"</pre></div></div><div class="section"><div class="titlepage"><div><div><h5 class="title"><a name="_queue_sizes"></a>Queue Sizes</h5></div></div></div><p><code class="literal">net.core.somaxconn</code> controls the size of the connection listening queue.
The default value is 128.
If you are running a high-volume server and connections are getting refused at a TCP level, you need to increase this value.
This setting can take a bit of finesse to get correct: if you set it too high, resource problems occur as it tries to notify a server of a large number of connections, and many remain pending, but if you set it too low, refused connections occur.</p><div class="screenexample"><pre class="screen"> $ sysctl -w net.core.somaxconn=4096</pre></div><p>The <code class="literal">net.core.netdev_max_backlog</code> controls the size of the incoming packet queue for upper-layer (Java) processing.
The default (2048) may be increased and other related parameters adjusted with:</p><div class="screenexample"><pre class="screen">$ sysctl -w net.core.netdev_max_backlog=16384
$ sysctl -w net.ipv4.tcp_max_syn_backlog=8192
$ sysctl -w net.ipv4.tcp_syncookies=1</pre></div></div><div class="section"><div class="titlepage"><div><div><h5 class="title"><a name="_ports"></a>Ports</h5></div></div></div><p>If many outgoing connections are made (for example, on load generators), the operating system might run low on ports.
Thus it is best to increase the port range, and allow reuse of sockets in <code class="literal">TIME_WAIT</code>:</p><div class="screenexample"><pre class="screen">$ sysctl -w net.ipv4.ip_local_port_range="1024 65535"
$ sysctl -w net.ipv4.tcp_tw_recycle=1</pre></div></div><div class="section"><div class="titlepage"><div><div><h5 class="title"><a name="_file_descriptors"></a>File Descriptors</h5></div></div></div><p>Busy servers and load generators may run out of file descriptors as the system defaults are normally low.
These can be increased for a specific user in <code class="literal">/etc/security/limits.conf</code>:</p><pre class="literallayout">theusername            hard nofile     40000
theusername            soft nofile     40000</pre></div><div class="section"><div class="titlepage"><div><div><h5 class="title"><a name="_congestion_control"></a>Congestion Control</h5></div></div></div><p>Linux supports pluggable congestion control algorithms.
To get a list of congestion control algorithms that are available in your kernel run:</p><div class="screenexample"><pre class="screen">$ sysctl net.ipv4.tcp_available_congestion_control</pre></div><p>If cubic and/or htcp are not listed, you need to research the control algorithms for your kernel.
You can try setting the control to cubic with:</p><div class="screenexample"><pre class="screen">$ sysctl -w net.ipv4.tcp_congestion_control=cubic</pre></div></div><div class="section"><div class="titlepage"><div><div><h5 class="title"><a name="_mac_os"></a>Mac OS</h5></div></div></div><p>Tips welcome.</p></div><div class="section"><div class="titlepage"><div><div><h5 class="title"><a name="_windows"></a>Windows</h5></div></div></div><p>Tips welcome.</p></div><div class="section"><div class="titlepage"><div><div><h5 class="title"><a name="_network_tuning"></a>Network Tuning</h5></div></div></div><p>Intermediaries such as nginx can use a non-persistent HTTP/1.0 connection.
Make sure to use persistent HTTP/1.1 connections.</p></div><div class="section"><div class="titlepage"><div><div><h5 class="title"><a name="_jvm_tuning"></a>JVM Tuning</h5></div></div></div><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem">Tune the <a class="link" href="optimizing.html#tuning-examples" title="Tuning Examples">Garbage Collection</a></li><li class="listitem">Allocate sufficient memory</li><li class="listitem">Use the -server option</li><li class="listitem">Jetty Tuning</li></ul></div></div><div class="section"><div class="titlepage"><div><div><h5 class="title"><a name="_acceptors"></a>Acceptors</h5></div></div></div><p>The standard rule of thumb for the number of Accepters to configure is one per CPU on a given machine.</p></div><div class="section"><div class="titlepage"><div><div><h5 class="title"><a name="_low_resource_limits"></a>Low Resource Limits</h5></div></div></div><p>Must not be configured for less than the number of expected connections.</p></div><div class="section"><div class="titlepage"><div><div><h5 class="title"><a name="_thread_pool"></a>Thread Pool</h5></div></div></div><p>Configure with goal of limiting memory usage maximum available.
Typically this is &gt;50 and &lt;500</p></div></div></div></div><script type="text/javascript">
      SyntaxHighlighter.all()
    </script><div class="navfooter"><hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="optimizing.html"><i class="fa fa-chevron-left" aria-hidden="true"></i> Previous</a>&nbsp;</td><td width="20%" align="center"><a accesskey="u" href="optimizing.html"><i class="fa fa-chevron-up" aria-hidden="true"></i> Top</a></td><td width="40%" align="right">&nbsp;<a accesskey="n" href="limit-load.html">Next <i class="fa fa-chevron-right" aria-hidden="true"></i></a></td></tr><tr><td width="40%" align="left" valign="top">Chapter&nbsp;20.&nbsp;Optimizing Jetty&nbsp;</td><td width="20%" align="center"><a accesskey="h" href="index.html"><i class="fa fa-home" aria-hidden="true"></i> Home</a></td><td width="40%" align="right" valign="top">&nbsp;Limiting Load</td></tr></table></div><p xmlns:jfetch="java:org.eclipse.jetty.xslt.tools.JavaSourceFetchExtension" xmlns:fetch="java:org.eclipse.jetty.xslt.tools.SourceFetchExtension" xmlns:d="http://docbook.org/ns/docbook" xmlns:l="http://docbook.sourceforge.net/xmlns/l10n/1.0" xmlns:xslthl="http://xslthl.sf.net" xmlns:gcse="http://www.google.com" xmlns:date="http://exslt.org/dates-and-times"><div class="jetty-callout">
            See an error or something missing?
            <span class="callout"><a href="http://github.com/eclipse/jetty.project">Contribute to this documentation at
                <span class="website"><i class="fa fa-github" aria-hidden="true"></i> Github!</span></a></span><span style="float: right"><i>(Generated: 2019-11-05)</i></span></div></p></body></html>