<html><head>
      <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
   <title>Chapter&nbsp;28.&nbsp;Continuations</title><link rel="stylesheet" type="text/css" href="css/docbook.css"><meta name="generator" content="DocBook XSL Stylesheets V1.79.1"><meta name="keywords" content="jetty, servlet, servlet-api, cometd, http, websocket, eclipse, maven, java, server, software"><link rel="home" href="index.html" title="Jetty"><link rel="up" href="jetty-dev-guide.html" title="Part&nbsp;IV.&nbsp;Jetty Development Guide"><link rel="prev" href="jetty-websocket-client-api.html" title="Jetty WebSocket Client API"><link rel="next" href="continuations-using.html" title="Using Continuations"><link xmlns:jfetch="java:org.eclipse.jetty.xslt.tools.JavaSourceFetchExtension" xmlns:fetch="java:org.eclipse.jetty.xslt.tools.SourceFetchExtension" xmlns:d="http://docbook.org/ns/docbook" xmlns:l="http://docbook.sourceforge.net/xmlns/l10n/1.0" xmlns:xslthl="http://xslthl.sf.net" xmlns:gcse="http://www.google.com" xmlns:date="http://exslt.org/dates-and-times" rel="shortcut icon" href="images/favicon.ico"><link rel="stylesheet" href="css/highlighter/foundation.css"><script src="js/highlight.pack.js"></script><script>
      hljs.initHighlightingOnLoad();
    </script><link type="text/css" rel="stylesheet" href="css/font-awesome/font-awesome.min.css"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><table xmlns:jfetch="java:org.eclipse.jetty.xslt.tools.JavaSourceFetchExtension" xmlns:fetch="java:org.eclipse.jetty.xslt.tools.SourceFetchExtension" xmlns:d="http://docbook.org/ns/docbook" xmlns:l="http://docbook.sourceforge.net/xmlns/l10n/1.0" xmlns:xslthl="http://xslthl.sf.net" xmlns:gcse="http://www.google.com" xmlns:date="http://exslt.org/dates-and-times"><tr><td style="width: 25%"><a href="http://www.eclipse.org/jetty"><img src="images/jetty-header-logo.png" alt="Jetty Logo"></a><br><span style="font-size: small">
            Version: 9.3.27.v20190418</span></td><td style="width: 50%"></td></tr></table><div xmlns:jfetch="java:org.eclipse.jetty.xslt.tools.JavaSourceFetchExtension" xmlns:fetch="java:org.eclipse.jetty.xslt.tools.SourceFetchExtension" xmlns:d="http://docbook.org/ns/docbook" xmlns:l="http://docbook.sourceforge.net/xmlns/l10n/1.0" xmlns:xslthl="http://xslthl.sf.net" xmlns:gcse="http://www.google.com" xmlns:date="http://exslt.org/dates-and-times" class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">Chapter&nbsp;28.&nbsp;Continuations</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="jetty-websocket-client-api.html"><i class="fa fa-chevron-left" aria-hidden="true"></i> Previous</a>&nbsp;</td><th width="60%" align="center">Part&nbsp;IV.&nbsp;Jetty Development Guide<br><a accesskey="p" href="index.html"><i class="fa fa-home" aria-hidden="true"></i> Home</a></th><td width="20%" align="right">&nbsp;<a accesskey="n" href="continuations-using.html">Next <i class="fa fa-chevron-right" aria-hidden="true"></i></a></td></tr></table><hr></div><div xmlns:jfetch="java:org.eclipse.jetty.xslt.tools.JavaSourceFetchExtension" xmlns:fetch="java:org.eclipse.jetty.xslt.tools.SourceFetchExtension" xmlns:d="http://docbook.org/ns/docbook" xmlns:l="http://docbook.sourceforge.net/xmlns/l10n/1.0" xmlns:xslthl="http://xslthl.sf.net" xmlns:gcse="http://www.google.com" xmlns:date="http://exslt.org/dates-and-times" class="jetty-callout"><h5 class="callout"><a href="http://www.webtide.com/">Contact the core Jetty developers at
          <span class="website">www.webtide.com</span></a></h5><p>
 private support for your internal/customer projects ... custom extensions and distributions ... versioned snapshots for indefinite support ...
 scalability guidance for your apps and Ajax/Comet projects ... development services for sponsored feature development
      </p></div><div class="chapter"><div class="titlepage"><div><div><h2 class="title"><a name="continuations"></a>Chapter&nbsp;28.&nbsp;Continuations</h2></div></div></div><div class="toc"><p><b>Table of Contents</b></p><dl class="toc"><dt><span class="section"><a href="continuations.html#continuations-intro">Introduction</a></span></dt><dt><span class="section"><a href="continuations-using.html">Using Continuations</a></span></dt><dt><span class="section"><a href="continuations-patterns.html">Common Continuation Patterns</a></span></dt></dl></div><div class="section"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="continuations-intro"></a>Introduction</h2></div></div></div><div class="toc"><dl class="toc"><dt><span class="section"><a href="continuations.html#_why_asynchronous_servlets">Why Asynchronous Servlets ?</a></span></dt><dt><span class="section"><a href="continuations.html#_asynchronous_servlet_examples">Asynchronous Servlet Examples</a></span></dt><dt><span class="section"><a href="continuations.html#_servlet_threading_model">Servlet Threading Model</a></span></dt></dl></div><p>Continuations are a mechanism to implement Asynchronous servlets similar to asynchronous features in Servlet 3.0, but provides a simpler and portable interface.</p><div class="section"><div class="titlepage"><div><div><h3 class="title"><a name="_why_asynchronous_servlets"></a>Why Asynchronous Servlets ?</h3></div></div></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a name="_not_asynchronous_io"></a>Not Asynchronous IO</h4></div></div></div><p>The concept of Asynchronous Servlets is often confused with Asynchronous IO or the use of NIO.
However, Asynchronous Servlets are not primarily motivated by asynchronous IO, since:</p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem">HTTP Requests are mostly small and arrive in a single packet. Servlets rarely block on requests.</li><li class="listitem">Many responses are small and fit within the server buffers, so servlets often do not block writing responses.</li><li class="listitem">Even if we could expose asynchronous IO in a servlet, it is a hard paradigm to program. For example what would an application do if it read 2 bytes of a 3 byte UTF-8 character?
It would have to buffer and wait for more bytes.
This is best done by the container rather than the application.</li></ul></div></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a name="_asynchronous_waiting"></a>Asynchronous Waiting</h4></div></div></div><p>The main use-case for asynchronous servlets is waiting for non-IO events or resources.
Many web applications need to wait at some stage during the processing of a HTTP request, for example:</p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem">Waiting for a resource to be available before processing the request (e.g., thread, JDBC Connection).</li><li class="listitem">Waiting for an application event in an AJAX Comet application (e.g., chat message, price change).</li><li class="listitem">Waiting for a response from a remote service (e.g., RESTful or SOAP call to a web service).</li></ul></div><p>The servlet API (pre 2.5) supports only a synchronous call style, so that any waiting that a servlet needs to do must be with blocking.
Unfortunately this means that the thread allocated to the request must be held during that wait along with all its resources: kernel thread, stack memory and often pooled buffers, character converters, EE authentication context, etc.
It is wasteful of system resources to hold these resources while waiting. Significantly better scalability and quality of service can be achieved if waiting is done asynchronously.</p></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a name="_asynchronous_servlet_examples"></a>Asynchronous Servlet Examples</h3></div></div></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a name="_ajax_comet_server_push"></a>AJAX Comet Server Push</h4></div></div></div><p>Web 2.0 applications can use the <a class="link" href="http://en.wikipedia.org/wiki/Comet_(programming)" target="_top">comet</a> technique (aka AJAX Push, Server Push, Long Polling) to dynamically update a web page without refreshing the entire page.</p><p>Consider a stock portfolio web application. Each browser will send a long poll request to the server asking for any of the user&#8217;s stock prices that have changed. The server will receive the long poll requests from all its clients, but will not immediately respond.
Instead the server waits until a stock price changes, at which time it will send a response to each of the clients with that stock in their portfolio.
The clients that receive the long poll response will immediately send another long poll request so they may obtain future price changes.</p><p>Thus the server will typically hold a long poll request for every connected user, so if the servlet is not asynchronous, there would need more than 1000 threads available to handle 1000 simultaneous users.
1000 threads can consume over 256MB of memory; that would be better used for the application rather than idly waiting for a price to change.</p><p>If the servlet is asynchronous, then the number of threads needed is governed by the time to generate each response and the frequency of price changes.
If every user receives a price every 10 seconds and the response takes 10ms to generate, then 1000 users can be serviced with just 1 thread, and the 256MB of stack be freed for other purposes.</p><p>For more on comet see the <a class="link" href="http://cometd.org/" target="_top">cometd</a> project that works asynchronously with Jetty.</p></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a name="_asynchronous_restful_web_service"></a>Asynchronous RESTful Web Service</h4></div></div></div><p>Consider a web application that accesses a remote web service (e.g., SOAP service or RESTful service).
Typically a remote web service can take hundreds of milliseconds to produce a response&#8201;&#8212;&#8201;eBay&#8217;s RESTful web service frequently takes 350ms to respond with a list of auctions matching a given keyword&#8201;&#8212;&#8201;while only a few 10s of milliseconds of CPU time are needed to locally process a request and generate a response.</p><p>To handle 1000 requests per second, which each perform a 200ms web service call, a webapp would needs 1000*(200+20)/1000 = 220 threads and 110MB of stack memory.
It would also be vulnerable to thread starvation if bursts occurred or the web service became slower. If handled asynchronously, the web application would not need to hold a thread while waiting for web service response.
Even if the asynchronous mechanism cost 10ms (which it doesn&#8217;t), then this webapp would need 1000*(20+10)/1000 = 30 threads and 15MB of stack memory.
This is a 86% reduction in the resources required and 95MB more memory would be available for the application.
Furthermore, if multiple web services request are required, the asynchronous approach allows these to be made in parallel rather than serially, without allocating additional threads.</p><p>For an example of Jetty&#8217;s solution, see the <a class="link" href="https://webtide.com/async-rest-jetty-9/" target="_top">Asynchronous REST example</a></p></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a name="_quality_of_service_e_g_jdbc_connection_pool"></a>Quality of Service (e.g., JDBC Connection Pool)</h4></div></div></div><p>Consider a web application handling on average 400 requests per second, with each request interacting with the database for 50ms.
To handle this load, 400*50/1000 = 20 JDBC connections are need on average.
However, requests do not come at an even rate and there are often bursts and pauses.
To protect a database from bursts, often a JDBC connection pool is applied to limit the simultaneous requests made on the database.
So for this application, it would be reasonable to apply a JDBC pool of 30 connections, to provide for a 50% margin.</p><p>If momentarily the request rate doubled, then the 30 connections would only be able to handle 600 requests per second, and 200 requests per second would join those waiting on the JDBC Connection pool.
Then if the servlet container had a thread pool with 200 threads, that would be entirely consumed by threads waiting for JDBC connections in 1 second of this request rate.
After 1s, the web application would be unable to process any requests at all because no threads would be available.
Even requests that do not use the database would be blocked due to thread starvation.
To double the thread pool would require an additional 100MB of stack memory and would only give the application another 1s of grace under load!</p><p>This thread starvation situation can also occur if the database runs slowly or is momentarily unavailable.
Thread starvation is a very frequently reported problem, and causes the entire web service to lock up and become unresponsive.
If the web container was able to suspend the requests waiting for a JDBC connection without threads, then thread starvation would not occur, as only 30 threads would be consumed by requests accessing the database and the other 470 threads would be available to process the request that do not access the database.</p><p>For an example of Jetty&#8217;s solution, see the Quality of Service Filter.</p></div></div><div class="section"><div class="titlepage"><div><div><h3 class="title"><a name="_servlet_threading_model"></a>Servlet Threading Model</h3></div></div></div><p>The scalability issues of Java servlets are caused mainly by the server threading model:</p><div class="section"><div class="titlepage"><div><div><h4 class="title"><a name="_thread_per_connection"></a>Thread per connection</h4></div></div></div><p>The traditional IO model of Java associated a thread with every TCP/IP connection.
If you have a few very active threads, this model can scale to a very high number of requests per second.</p><p>However, the traffic profile typical of many web applications is many persistent HTTP connections that are mostly idle while users read pages or search for the next link to click. With such profiles, the thread-per-connection model can have problems scaling to the thousands of threads required to support thousands of users on large scale deployments.</p></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a name="_thread_per_request"></a>Thread per request</h4></div></div></div><p>The Java NIO libraries support asynchronous IO, so that threads no longer need to be allocated to every connection.
When the connection is idle (between requests), then the connection is added to an NIO select set, which allows one thread to scan many connections for activity.
Only when IO is detected on a connection is a thread allocated to it.
However, the servlet 2.5 API model still requires a thread to be allocated for the duration of the request handling.</p><p>This thread-per-request model allows much greater scaling of connections (users) at the expense of a small reduction to maximum requests per second due to extra scheduling latency.</p></div><div class="section"><div class="titlepage"><div><div><h4 class="title"><a name="_asynchronous_request_handling"></a>Asynchronous Request handling</h4></div></div></div><p>The Jetty Continuation (and the servlet 3.0 asynchronous) API introduce a change in the servlet API that allows a request to be dispatched multiple times to a servlet.
If the servlet does not have the resources required on a dispatch, then the request is suspended (or put into asynchronous mode), so that the servlet may return from the dispatch without a response being sent.
When the waited-for resources become available, the request is re-dispatched to the servlet, with a new thread, and a response is generated.</p></div></div></div></div><script type="text/javascript">
      SyntaxHighlighter.all()
    </script><div class="navfooter"><hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="jetty-websocket-client-api.html"><i class="fa fa-chevron-left" aria-hidden="true"></i> Previous</a>&nbsp;</td><td width="20%" align="center"><a accesskey="u" href="jetty-dev-guide.html"><i class="fa fa-chevron-up" aria-hidden="true"></i> Top</a></td><td width="40%" align="right">&nbsp;<a accesskey="n" href="continuations-using.html">Next <i class="fa fa-chevron-right" aria-hidden="true"></i></a></td></tr><tr><td width="40%" align="left" valign="top">Jetty WebSocket Client API&nbsp;</td><td width="20%" align="center"><a accesskey="h" href="index.html"><i class="fa fa-home" aria-hidden="true"></i> Home</a></td><td width="40%" align="right" valign="top">&nbsp;Using Continuations</td></tr></table></div><p xmlns:jfetch="java:org.eclipse.jetty.xslt.tools.JavaSourceFetchExtension" xmlns:fetch="java:org.eclipse.jetty.xslt.tools.SourceFetchExtension" xmlns:d="http://docbook.org/ns/docbook" xmlns:l="http://docbook.sourceforge.net/xmlns/l10n/1.0" xmlns:xslthl="http://xslthl.sf.net" xmlns:gcse="http://www.google.com" xmlns:date="http://exslt.org/dates-and-times"><div class="jetty-callout">
            See an error or something missing?
            <span class="callout"><a href="http://github.com/eclipse/jetty.project">Contribute to this documentation at
                <span class="website"><i class="fa fa-github" aria-hidden="true"></i> Github!</span></a></span><span style="float: right"><i>(Generated: 2019-04-18)</i></span></div></p></body></html>