Generated from commit: fc92f5d04a3b16a7d581042133cecad805dcd296
diff --git a/dump_javadump/index.html b/dump_javadump/index.html
index 4cc1e99..fb8e9ee 100644
--- a/dump_javadump/index.html
+++ b/dump_javadump/index.html
@@ -617,6 +617,13 @@
</li>
+ <li class="md-nav__item">
+ <a href="#hang" title="Hang" class="md-nav__link">
+ Hang
+ </a>
+
+</li>
+
</ul>
</nav>
@@ -3117,6 +3124,13 @@
</li>
+ <li class="md-nav__item">
+ <a href="#hang" title="Hang" class="md-nav__link">
+ Hang
+ </a>
+
+</li>
+
</ul>
</nav>
@@ -3190,6 +3204,7 @@
<li><a href="#java-outofmemoryerror">A Java heap OutOfMemoryError (OOM)</a></li>
<li><a href="#native-outofmemoryerror">A native OutOfMemoryError (OOM)</a></li>
<li><a href="#deadlock">A deadlock situation</a></li>
+<li><a href="#hang">A hang</a></li>
</ul>
<h2 id="java-dump-contents">Java dump contents</h2>
<p>Java dumps summarize the state of the VM when the event occurs, with most of the information relating to components of the VM.
@@ -4264,6 +4279,68 @@
</pre></div>
+<h3 id="hang">Hang</h3>
+<p>An application can hang for a number of reasons but the most common cause is excessive global garbage collection (GC) activity, where your application is repeatedly paused because your Java heap has <em>almost</em> run out of memory. You can identify this problem by looking at verbose GC output. Collect this output by specifying the <code>-verbose:gc</code> option.</p>
+<p>Deadlock situations can also manifest themselves as hangs. For more information on diagnosing this type of problem from a Java dump, see the <a href="#deadlock">deadlock</a> scenario.</p>
+<p>If you have eliminated verbose GC activity and deadlocks, another common hang scenario involves threads that compete and wait for Java object locks. This type of problem can usually be diagnosed by examining a Java dump. The simplest hang scenario involving Java object locks is where a thread acquires a lock that other threads are waiting for, but it doesn't release the lock for some reason.</p>
+<p>The first place to look in the Java dump output is the <strong>LOCKS</strong> section. This section lists all the monitors and shows which threads have acquired a lock and which threads are waiting. If the hang is caused by a thread not releasing a lock that other threads need, you can see a list of waiting threads in the output.</p>
+<p>In this example scenario, the Java dump <strong>LOCKS</strong> section shows that <code>Worker Thread 0</code> (<code>3LKMONOBJECT</code>) has acquired a lock and there are 19 other worker threads waiting to obtain the lock.</p>
+<div class="codehilite"><pre><span></span><span class="nb">NULL</span> <span class="o">------------------------------------------------------------------------</span>
+<span class="mi">0</span><span class="n">SECTION</span> <span class="n">LOCKS</span> <span class="n">subcomponent</span> <span class="n">dump</span> <span class="n">routine</span>
+<span class="nb">NULL</span> <span class="o">===============================</span>
+<span class="nb">NULL</span>
+<span class="mi">1L</span><span class="n">KPOOLINFO</span> <span class="n">Monitor</span> <span class="n">pool</span> <span class="nl">info</span><span class="p">:</span>
+<span class="mi">2L</span><span class="n">KPOOLTOTAL</span> <span class="n">Current</span> <span class="n">total</span> <span class="n">number</span> <span class="n">of</span> <span class="nl">monitors</span><span class="p">:</span> <span class="mi">1</span>
+<span class="nb">NULL</span>
+<span class="mi">1L</span><span class="n">KMONPOOLDUMP</span> <span class="n">Monitor</span> <span class="n">Pool</span> <span class="n">Dump</span> <span class="p">(</span><span class="n">flat</span> <span class="o">&</span> <span class="n">inflated</span> <span class="n">object</span><span class="o">-</span><span class="n">monitors</span><span class="p">)</span><span class="o">:</span>
+<span class="mi">2L</span><span class="n">KMONINUSE</span> <span class="nl">sys_mon_t</span><span class="p">:</span><span class="mh">0x92711200</span> <span class="nl">infl_mon_t</span><span class="p">:</span> <span class="mh">0x92711240</span><span class="o">:</span>
+<span class="mi">3L</span><span class="n">KMONOBJECT</span> <span class="n">java</span><span class="o">/</span><span class="n">lang</span><span class="o">/</span><span class="n">Object</span><span class="mh">@0xB56658D8</span><span class="o">:</span> <span class="n">Flat</span> <span class="n">locked</span> <span class="n">by</span> <span class="s">"Worker Thread 0"</span> <span class="p">(</span><span class="nl">J9VMThread</span><span class="p">:</span><span class="mh">0x92A3EC00</span><span class="p">),</span> <span class="n">entry</span> <span class="n">count</span> <span class="mi">1</span>
+<span class="mi">3L</span><span class="n">KWAITERQ</span> <span class="n">Waiting</span> <span class="n">to</span> <span class="nl">enter</span><span class="p">:</span>
+<span class="mi">3L</span><span class="n">KWAITER</span> <span class="s">"Worker Thread 1"</span> <span class="p">(</span><span class="nl">J9VMThread</span><span class="p">:</span><span class="mh">0x92703F00</span><span class="p">)</span>
+<span class="mi">3L</span><span class="n">KWAITER</span> <span class="s">"Worker Thread 2"</span> <span class="p">(</span><span class="nl">J9VMThread</span><span class="p">:</span><span class="mh">0x92709C00</span><span class="p">)</span>
+<span class="mi">3L</span><span class="n">KWAITER</span> <span class="s">"Worker Thread 3"</span> <span class="p">(</span><span class="nl">J9VMThread</span><span class="p">:</span><span class="mh">0x92710A00</span><span class="p">)</span>
+<span class="mi">3L</span><span class="n">KWAITER</span> <span class="s">"Worker Thread 4"</span> <span class="p">(</span><span class="nl">J9VMThread</span><span class="p">:</span><span class="mh">0x92717F00</span><span class="p">)</span>
+<span class="mi">3L</span><span class="n">KWAITER</span> <span class="s">"Worker Thread 5"</span> <span class="p">(</span><span class="nl">J9VMThread</span><span class="p">:</span><span class="mh">0x9271DC00</span><span class="p">)</span>
+<span class="mi">3L</span><span class="n">KWAITER</span> <span class="s">"Worker Thread 6"</span> <span class="p">(</span><span class="nl">J9VMThread</span><span class="p">:</span><span class="mh">0x92723A00</span><span class="p">)</span>
+<span class="mi">3L</span><span class="n">KWAITER</span> <span class="s">"Worker Thread 7"</span> <span class="p">(</span><span class="nl">J9VMThread</span><span class="p">:</span><span class="mh">0x92729800</span><span class="p">)</span>
+<span class="mi">3L</span><span class="n">KWAITER</span> <span class="s">"Worker Thread 8"</span> <span class="p">(</span><span class="nl">J9VMThread</span><span class="p">:</span><span class="mh">0x92733700</span><span class="p">)</span>
+<span class="mi">3L</span><span class="n">KWAITER</span> <span class="s">"Worker Thread 9"</span> <span class="p">(</span><span class="nl">J9VMThread</span><span class="p">:</span><span class="mh">0x92739400</span><span class="p">)</span>
+<span class="mi">3L</span><span class="n">KWAITER</span> <span class="s">"Worker Thread 10"</span> <span class="p">(</span><span class="nl">J9VMThread</span><span class="p">:</span><span class="mh">0x92740200</span><span class="p">)</span>
+<span class="mi">3L</span><span class="n">KWAITER</span> <span class="s">"Worker Thread 11"</span> <span class="p">(</span><span class="nl">J9VMThread</span><span class="p">:</span><span class="mh">0x92748100</span><span class="p">)</span>
+<span class="mi">3L</span><span class="n">KWAITER</span> <span class="s">"Worker Thread 12"</span> <span class="p">(</span><span class="nl">J9VMThread</span><span class="p">:</span><span class="mh">0x9274DF00</span><span class="p">)</span>
+<span class="mi">3L</span><span class="n">KWAITER</span> <span class="s">"Worker Thread 13"</span> <span class="p">(</span><span class="nl">J9VMThread</span><span class="p">:</span><span class="mh">0x92754D00</span><span class="p">)</span>
+<span class="mi">3L</span><span class="n">KWAITER</span> <span class="s">"Worker Thread 14"</span> <span class="p">(</span><span class="nl">J9VMThread</span><span class="p">:</span><span class="mh">0x9275AA00</span><span class="p">)</span>
+<span class="mi">3L</span><span class="n">KWAITER</span> <span class="s">"Worker Thread 15"</span> <span class="p">(</span><span class="nl">J9VMThread</span><span class="p">:</span><span class="mh">0x92760800</span><span class="p">)</span>
+<span class="mi">3L</span><span class="n">KWAITER</span> <span class="s">"Worker Thread 16"</span> <span class="p">(</span><span class="nl">J9VMThread</span><span class="p">:</span><span class="mh">0x92766600</span><span class="p">)</span>
+<span class="mi">3L</span><span class="n">KWAITER</span> <span class="s">"Worker Thread 17"</span> <span class="p">(</span><span class="nl">J9VMThread</span><span class="p">:</span><span class="mh">0x9276C300</span><span class="p">)</span>
+<span class="mi">3L</span><span class="n">KWAITER</span> <span class="s">"Worker Thread 18"</span> <span class="p">(</span><span class="nl">J9VMThread</span><span class="p">:</span><span class="mh">0x92773100</span><span class="p">)</span>
+<span class="mi">3L</span><span class="n">KWAITER</span> <span class="s">"Worker Thread 19"</span> <span class="p">(</span><span class="nl">J9VMThread</span><span class="p">:</span><span class="mh">0x92778F00</span><span class="p">)</span>
+<span class="nb">NULL</span>
+</pre></div>
+
+
+<p>The next step is to determine why <code>Worker Thread 0</code> is not releasing the lock. The best place to start is the stack trace for this thread, which you can find by searching on the thread name or J9VMThread ID in the <strong>THREADS</strong> section.</p>
+<p>The following extract shows the details for <code>"Worker Thread 0" (J9VMThread:0x92A3EC00)</code>:</p>
+<div class="codehilite"><pre><span></span><span class="nb">NULL</span>
+<span class="mi">3</span><span class="n">XMTHREADINFO</span> <span class="s">"Worker Thread 0"</span> <span class="nl">J9VMThread</span><span class="p">:</span><span class="mh">0x92A3EC00</span><span class="p">,</span> <span class="nl">omrthread_t</span><span class="p">:</span><span class="mh">0x92A3C280</span><span class="p">,</span> <span class="n">java</span><span class="o">/</span><span class="n">lang</span><span class="o">/</span><span class="nl">Thread</span><span class="p">:</span><span class="mh">0xB56668B8</span><span class="p">,</span> <span class="nl">state</span><span class="p">:</span><span class="n">CW</span><span class="p">,</span> <span class="n">prio</span><span class="o">=</span><span class="mi">5</span>
+<span class="mi">3</span><span class="n">XMJAVALTHREAD</span> <span class="p">(</span><span class="n">java</span><span class="o">/</span><span class="n">lang</span><span class="o">/</span><span class="n">Thread</span> <span class="nl">getId</span><span class="p">:</span><span class="mh">0x13</span><span class="p">,</span> <span class="nl">isDaemon</span><span class="p">:</span><span class="nb">false</span><span class="p">)</span>
+<span class="mi">3</span><span class="n">XMTHREADINFO1</span> <span class="p">(</span><span class="n">native</span> <span class="kr">thread</span> <span class="nl">ID</span><span class="p">:</span><span class="mh">0x511F</span><span class="p">,</span> <span class="n">native</span> <span class="nl">priority</span><span class="p">:</span><span class="mh">0x5</span><span class="p">,</span> <span class="n">native</span> <span class="nl">policy</span><span class="p">:</span><span class="n">UNKNOWN</span><span class="p">,</span> <span class="nl">vmstate</span><span class="p">:</span><span class="n">CW</span><span class="p">,</span> <span class="n">vm</span> <span class="kr">thread</span> <span class="nl">flags</span><span class="p">:</span><span class="mh">0x00000401</span><span class="p">)</span>
+<span class="mi">3</span><span class="n">XMTHREADINFO2</span> <span class="p">(</span><span class="n">native</span> <span class="n">stack</span> <span class="n">address</span> <span class="n">range</span> <span class="nl">from</span><span class="p">:</span><span class="mh">0x9297E000</span><span class="p">,</span> <span class="nl">to</span><span class="p">:</span><span class="mh">0x929BF000</span><span class="p">,</span> <span class="nl">size</span><span class="p">:</span><span class="mh">0x41000</span><span class="p">)</span>
+<span class="mi">3</span><span class="n">XMCPUTIME</span> <span class="n">CPU</span> <span class="n">usage</span> <span class="nl">total</span><span class="p">:</span> <span class="mf">0.000211878</span> <span class="n">secs</span><span class="p">,</span> <span class="n">current</span> <span class="n">category</span><span class="o">=</span><span class="s">"Application"</span>
+<span class="mi">3</span><span class="n">XMHEAPALLOC</span> <span class="n">Heap</span> <span class="n">bytes</span> <span class="n">allocated</span> <span class="n">since</span> <span class="n">last</span> <span class="n">GC</span> <span class="n">cycle</span><span class="o">=</span><span class="mi">0</span> <span class="p">(</span><span class="mh">0x0</span><span class="p">)</span>
+<span class="mi">3</span><span class="n">XMTHREADINFO3</span> <span class="n">Java</span> <span class="nl">callstack</span><span class="p">:</span>
+<span class="mi">4</span><span class="n">XESTACKTRACE</span> <span class="n">at</span> <span class="n">java</span><span class="o">/</span><span class="n">lang</span><span class="o">/</span><span class="n">Thread</span><span class="p">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">Native</span> <span class="n">Method</span><span class="p">)</span>
+<span class="mi">4</span><span class="n">XESTACKTRACE</span> <span class="n">at</span> <span class="n">java</span><span class="o">/</span><span class="n">lang</span><span class="o">/</span><span class="n">Thread</span><span class="p">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">Thread</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">941</span><span class="p">)</span>
+<span class="mi">4</span><span class="n">XESTACKTRACE</span> <span class="n">at</span> <span class="n">WorkerThread</span><span class="p">.</span><span class="n">doWork</span><span class="p">(</span><span class="n">HangTest</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">37</span><span class="p">)</span>
+<span class="mi">4</span><span class="n">XESTACKTRACE</span> <span class="n">at</span> <span class="n">WorkerThread</span><span class="p">.</span><span class="n">run</span><span class="p">(</span><span class="n">HangTest</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">31</span><span class="p">)</span>
+<span class="mi">5</span><span class="n">XESTACKTRACE</span> <span class="p">(</span><span class="n">entered</span> <span class="nl">lock</span><span class="p">:</span> <span class="n">java</span><span class="o">/</span><span class="n">lang</span><span class="o">/</span><span class="n">Object</span><span class="mh">@0xB56658D8</span><span class="p">,</span> <span class="n">entry</span> <span class="nl">count</span><span class="p">:</span> <span class="mi">1</span><span class="p">)</span>
+</pre></div>
+
+
+<p>In the last line of this output you can see where the thread acquired the lock. Working up from this line, you can see that <code>WorkerThread.run</code> was called, which in turn called <code>WorkerThread.doWork</code>. The stack shows that the thread then entered a call to <code>java/lang/Thread.sleep</code> in HangTest.java on line 37, which is preventing the thread from completing its work and releasing the lock. In this example the <code>sleep</code> call was added to induce a hang, but in real-world scenarios the cause could be any blocking operation, such as reading from an input stream or socket. Another possibility is that the thread is waiting for <em>another</em> lock owned by yet another thread.</p>
+<p>It is important to remember that each Java dump represents a single snapshot in time. You should generate at least three Java dumps separated by a short pause, for example 30 seconds, and compare the output. This comparison tells you whether the threads involved are stuck in a fixed state or whether they are moving.</p>
+<p>In this example, the threads do not move and the investigation needs to focus on the logic in <code>WorkerThread.doWork</code> to understand why <code>Worker Thread 0</code> entered the <code>java/lang/Thread.sleep</code> call.</p>
+<p>Another common scenario is where each Java dump shows a number of threads waiting for a lock owned by another thread, but the list of waiting threads and the lock-owning thread change over time. In this case the cause is likely to be a bottleneck caused by thread contention, where the threads are continually competing for the same lock. In severe cases, the lock is held only for a small amount of time but there are lots of threads trying to obtain it. Because more time is spent handling the lock and scheduling the thread than executing application code, the degradation in performance is manifested as a hang. Thread contention is usually caused by an application design problem. You can use a similar approach to the one used in this scenario to determime which lines of code are responsible for the contention.</p>
<!-- ==== END OF TOPIC ==== dump_javadump.md ==== -->
diff --git a/search/search_index.json b/search/search_index.json
index a13a1f1..bc82331 100644
--- a/search/search_index.json
+++ b/search/search_index.json
@@ -402,12 +402,12 @@
},
{
"location": "/dump_javadump/",
- "text": "Java dump\n\n\nJava dumps, sometimes referred to as \nJava cores\n, are produced when the VM ends unexpectedly because of an operating system signal, \nOutOfMemoryError\n, or a user-initiated keystroke combination. You can also generate a Java dump by calling the Dump API programmatically from your application or specifying the \n-Xdump:java\n option on the command line.\n\n\nIf your Java application crashes or hangs, Java dumps can provide useful information to help you diagnose the root cause.\n\n\n\n\nIf your application crashes, Javadumps are generated automatically for the following types of failure:\n\n\nthe VM receives an unexpected signal or an assertion failure\n\n\nthe VM runs out of memory\n\n\n\n\n\n\n\n\nIf your application hangs, you can trigger the generation of a Java dump by sending a SIGQUIT signal (\nkill -3\n) to the VM.\n\n\n \nNote:\n On Windows, if you started the VM in a console window you can force the VM to produce a Java dump in response to a SIGBREAK signal\n(Ctrl-Break keyboard combination). If you didn't start in a console window there is no equivalent to a Linux \nkill\n command on Windows for sending signals.\nThe only option here is to trigger a full system dump by finding the VM process in the \nProcesses\n tab of the Windows Task Manager and clicking\n\nCreate dump file\n.\n\n\n\n\n\n\nTo help you understand how a Java dump can help you with problem diagnosis, this topic includes a few scenarios to help you interpret the data:\n\n\n\n\nA crash caused by a general protection fault (gpf)\n\n\nA Java heap OutOfMemoryError (OOM)\n\n\nA native OutOfMemoryError (OOM)\n\n\nA deadlock situation\n\n\n\n\nJava dump contents\n\n\nJava dumps summarize the state of the VM when the event occurs, with most of the information relating to components of the VM.\nThe file is made up of a number of sections that provide different types of information.\n\n\nTITLE\n\n\nThe first section of the Java dump file provides information about the event that triggered the production of the dump.\nIn the following example you can see that a \nvmstop\n event triggered the dump at a specified date and time.\n\n\n0SECTION TITLE subcomponent dump routine\nNULL ===============================\n1TICHARSET UTF-8\n1TISIGINFO Dump Event \"vmstop\" (00000002) Detail \"#0000000000000000\" received\n1TIDATETIME Date: 2018/08/30 at 21:55:47:607\n1TINANOTIME System nanotime: 22012355276134\n1TIFILENAME Javacore filename: /home/doc-javacore/javacore.20180830.215547.30285.0001.txt\n1TIREQFLAGS Request Flags: 0x81 (exclusive+preempt)\n1TIPREPSTATE Prep State: 0x106 (vm_access+exclusive_vm_access+trace_disabled)\n\n\n\n\n\nGPINFO\n\n\nThe GPINFO section provides general information about the system that the VM is running on. The following example is taken\nfrom a Java dump that was generated on a Linux system.\n\n\nNULL ------------------------------------------------------------------------\n0SECTION GPINFO subcomponent dump routine\nNULL ================================\n2XHOSLEVEL OS Level : Linux 3.10.0-862.11.6.el7.x86_64\n2XHCPUS Processors -\n3XHCPUARCH Architecture : amd64\n3XHNUMCPUS How Many : 4\n3XHNUMASUP NUMA is either not supported or has been disabled by user\nNULL \n1XHERROR2 Register dump section only produced for SIGSEGV, SIGILL or SIGFPE.\nNULL \n\n\n\n\n\nThe content of this section can vary, depending on the cause of the dump. For example, if the dump was caused by a\ngeneral protection fault (gpf), the library in which the crash occurred is also recorded, together with a value shown\nas \nVM flags\n. This value can provide some clues about which component of the VM might have been involved. Look for the\nfollowing line in the output:\n\n\n1XHFLAGS VM flags:0000000000000000\n\n\n\n\n\nThe hexadecimal number recorded for \nVM flags\n ends in MSSSS, where M is the VM component and SSSS is component-specific code as shown in the following table:\n\n\n\n\n\n\n\n\nComponent\n\n\nCode value\n\n\n\n\n\n\n\n\n\n\nINTERPRETER\n\n\n0x10000\n\n\n\n\n\n\nGC\n\n\n0x20000\n\n\n\n\n\n\nGROW_STACK\n\n\n0x30000\n\n\n\n\n\n\nJNI\n\n\n0x40000\n\n\n\n\n\n\nJIT_CODEGEN\n\n\n0x50000\n\n\n\n\n\n\nBCVERIFY\n\n\n0x60000\n\n\n\n\n\n\nRTVERIFY\n\n\n0x70000\n\n\n\n\n\n\nSHAREDCLASSES\n\n\n0x80000\n\n\n\n\n\n\n\n\nA value of \n0000000000000000\n (0x00000) indicates that a crash occurred outside of the VM.\n\n\nENVINFO\n\n\nThis section contains useful information about the environment in which the crash took place, including the following data:\n\n\n\n\nJava version (\n1CIJAVAVERSION\n)\n\n\nOpenJ9 VM and subcomponent version information (\n1CIVMVERSION\n, \n1CIJ9VMVERSION\n, \n1CIJITVERSION\n, \n1CIOMRVERSION\n, \n1CIJCLVERSION\n)\n\n\nVM start time (\n1CISTARTTIME\n) and process information (\n1CIPROCESSID\n)\n\n\nJava home (\n1CIJAVAHOMEDIR\n) and DLL (\n1CIJAVADLLDIR\n) directories\n\n\nUser arguments passed on the command line (\n1CIUSERARG\n)\n\n\nUser limits imposed by the system (\n1CIUSERLIMITS\n)\n\n\nEnvironment variables in place (\n1CIENVVARS\n)\n\n\nSystem information (\n1CISYSINFO\n)\n\n\nCPU information (\n1CICPUINFO\n)\n\n\n\n\n\nFor clarity, the following example shows a shortened version of this section, where \n...\n indicates that lines are removed:\n\n\nNULL\n \n------------------------------------------------------------------------\n\n\n0\nSECTION\n \nENVINFO\n \nsubcomponent\n \ndump\n \nroutine\n\n\nNULL\n \n=================================\n\n\n1\nCIJAVAVERSION\n \nJRE\n \n9\n \nLinux\n \namd64\n-\n64\n \n(\nbuild\n \n9.0.4\n-\ninternal\n+\n0\n-\nadhoc\n..\nopenj9\n-\nopenjdk\n-\njdk9\n)\n\n\n1\nCIVMVERSION\n \n20180830\n_000000\n\n\n1\nCIJ9VMVERSION\n \n8e7\nc6ec\n\n\n1\nCIJITVERSION\n \n8e7\nc6ec\n\n\n1\nCIOMRVERSION\n \n553811\nb_CMPRSS\n\n\n1\nCIJCLVERSION\n \nec1d223\n \nbased\n \non\n \njdk\n-\n9.0.4\n+\n12\n\n\n1\nCIJITMODES\n \nJIT\n \nenabled\n,\n \nAOT\n \nenabled\n,\n \nFSD\n \ndisabled\n,\n \nHCR\n \nenabled\n\n\n1\nCIRUNNINGAS\n \nRunning\n \nas\n \na\n \nstandalone\n \nJVM\n\n\n1\nCIVMIDLESTATE\n \nVM\n \nIdle\n \nState\n:\n \nACTIVE\n\n\n1\nCISTARTTIME\n \nJVM\n \nstart\n \ntime\n:\n \n2018\n/\n08\n/\n30\n \nat\n \n21\n:\n55\n:\n47\n:\n387\n\n\n1\nCISTARTNANO\n \nJVM\n \nstart\n \nnanotime\n:\n \n22012135233549\n\n\n1\nCIPROCESSID\n \nProcess\n \nID\n:\n \n30285\n \n(\n0x764D\n)\n\n\n1\nCICMDLINE\n \n[\nnot\n \navailable\n]\n\n\n1\nCIJAVAHOMEDIR\n \nJava\n \nHome\n \nDir\n:\n \n/\nhome\n/\nme\n/\nopenj9\n-\nopenjdk\n-\njdk9\n/\nbuild\n/\nlinux\n-\nx86_64\n-\nnormal\n-\nserver\n-\nrelease\n/\nimages\n/\njdk\n\n\n1\nCIJAVADLLDIR\n \nJava\n \nDLL\n \nDir\n:\n \n/\nhome\n/\nme\n/\nopenj9\n-\nopenjdk\n-\njdk9\n/\nbuild\n/\nlinux\n-\nx86_64\n-\nnormal\n-\nserver\n-\nrelease\n/\nimages\n/\njdk\n/\nbin\n\n\n1\nCISYSCP\n \nSys\n \nClasspath\n:\n \n\n1\nCIUSERARGS\n \nUserArgs\n:\n\n\n2\nCIUSERARG\n \n-\nXoptionsfile\n=/\nhome\n/\nme\n/\nopenj9\n-\nopenjdk\n-\njdk9\n/\nbuild\n/\nlinux\n-\nx86_64\n-\nnormal\n-\nserver\n-\nrelease\n/\nimages\n/\njdk\n/\nlib\n/\noptions\n.\ndefault\n\n\n...\n\n\nNULL\n\n\n1\nCIUSERLIMITS\n \nUser\n \nLimits\n \n(\nin\n \nbytes\n \nexcept\n \nfor\n \nNOFILE\n \nand\n \nNPROC\n)\n\n\nNULL\n \n------------------------------------------------------------------------\n\n\nNULL\n \ntype\n \nsoft\n \nlimit\n \nhard\n \nlimit\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_AS\n \nunlimited\n \nunlimited\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_CORE\n \n0\n \nunlimited\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_CPU\n \nunlimited\n \nunlimited\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_DATA\n \nunlimited\n \nunlimited\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_FSIZE\n \nunlimited\n \nunlimited\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_LOCKS\n \nunlimited\n \nunlimited\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_MEMLOCK\n \n65536\n \n65536\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_NOFILE\n \n4096\n \n4096\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_NPROC\n \n4096\n \n30592\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_RSS\n \nunlimited\n \nunlimited\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_STACK\n \n8388608\n \nunlimited\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_MSGQUEUE\n \n819200\n \n819200\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_NICE\n \n0\n \n0\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_RTPRIO\n \n0\n \n0\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_SIGPENDING\n \n30592\n \n30592\n\n\nNULL\n\n\n1\nCIENVVARS\n \nEnvironment\n \nVariables\n\n\nNULL\n \n------------------------------------------------------------------------\n\n\n2\nCIENVVAR\n \nXDG_VTNR\n=\n1\n\n\n2\nCIENVVAR\n \nSSH_AGENT_PID\n=\n2653\n\n\n...\n\n\nNULL\n \n\n1\nCISYSINFO\n \nSystem\n \nInformation\n\n\nNULL\n \n------------------------------------------------------------------------\n\n\n2\nCISYSINFO\n \n/\nproc\n/\nsys\n/\nkernel\n/\ncore_pattern\n \n=\n \ncore\n\n\n2\nCISYSINFO\n \n/\nproc\n/\nsys\n/\nkernel\n/\ncore_uses_pid\n \n=\n \n1\n\n\nNULL\n \n\n1\nCICPUINFO\n \nCPU\n \nInformation\n\n\nNULL\n \n------------------------------------------------------------------------\n\n\n2\nCIPHYSCPU\n \nPhysical\n \nCPUs\n:\n \n4\n\n\n2\nCIONLNCPU\n \nOnline\n \nCPUs\n:\n \n4\n\n\n2\nCIBOUNDCPU\n \nBound\n \nCPUs\n:\n \n4\n\n\n2\nCIACTIVECPU\n \nActive\n \nCPUs\n:\n \n0\n\n\n2\nCITARGETCPU\n \nTarget\n \nCPUs\n:\n \n4\n\n\n\n\n\n\n\n\n\nNATIVEMEMINFO\n\n\nThis section records information about native memory that is requested by using library functions such as \nmalloc()\n and \nmmap()\n.\nValues are provided as a breakdown, per component, indicating the total number of bytes allocated and the number of native memory allocations.\nIn the following example, 4,682,840 bytes of native memory are allocated (but not yet freed) to VM Classes, which corresponds to 141 allocations.\n\n\nNULL ------------------------------------------------------------------------\n0SECTION NATIVEMEMINFO subcomponent dump routine\nNULL =================================\n0MEMUSER\n1MEMUSER JRE: 2,569,088,312 bytes / 4653 allocations\n1MEMUSER |\n2MEMUSER +--VM: 2,280,088,336 bytes / 2423 allocations\n2MEMUSER | |\n3MEMUSER | +--Classes: 4,682,840 bytes / 141 allocations\n2MEMUSER | |\n3MEMUSER | +--Memory Manager (GC): 2,054,966,784 bytes / 433 allocations\n3MEMUSER | | |\n4MEMUSER | | +--Java Heap: 2,014,113,792 bytes / 1 allocation\n3MEMUSER | | |\n4MEMUSER | | +--Other: 40,852,992 bytes / 432 allocations\n2MEMUSER | |\n3MEMUSER | +--Threads: 10,970,016 bytes / 156 allocations\n3MEMUSER | | |\n4MEMUSER | | +--Java Stack: 197,760 bytes / 16 allocations\n3MEMUSER | | |\n4MEMUSER | | +--Native Stack: 10,616,832 bytes / 17 allocations\n3MEMUSER | | |\n4MEMUSER | | +--Other: 155,424 bytes / 123 allocations\n2MEMUSER | |\n3MEMUSER | +--Trace: 180,056 bytes / 263 allocations\n2MEMUSER | |\n3MEMUSER | +--JVMTI: 17,776 bytes / 13 allocations\n2MEMUSER | |\n3MEMUSER | +--JNI: 36,184 bytes / 52 allocations\n2MEMUSER | |\n3MEMUSER | +--Port Library: 208,179,632 bytes / 72 allocations\n3MEMUSER | | |\n4MEMUSER | | +--Unused <32bit allocation regions: 208,168,752 bytes / 1 allocation\n3MEMUSER | | |\n4MEMUSER | | +--Other: 10,880 bytes / 71 allocations\n2MEMUSER | |\n3MEMUSER | +--Other: 1,055,048 bytes / 1293 allocations\n1MEMUSER |\n2MEMUSER +--JIT: 288,472,816 bytes / 140 allocations\n2MEMUSER | |\n3MEMUSER | +--JIT Code Cache: 268,435,456 bytes / 1 allocation\n2MEMUSER | |\n3MEMUSER | +--JIT Data Cache: 2,097,216 bytes / 1 allocation\n2MEMUSER | |\n3MEMUSER | +--Other: 17,940,144 bytes / 138 allocations\n1MEMUSER |\n2MEMUSER +--Class Libraries: 13,432 bytes / 25 allocations\n2MEMUSER | |\n3MEMUSER | +--VM Class Libraries: 13,432 bytes / 25 allocations\n3MEMUSER | | |\n4MEMUSER | | +--sun.misc.Unsafe: 3,184 bytes / 13 allocations\n4MEMUSER | | | |\n5MEMUSER | | | +--Direct Byte Buffers: 1,056 bytes / 12 allocations\n4MEMUSER | | | |\n5MEMUSER | | | +--Other: 2,128 bytes / 1 allocation\n3MEMUSER | | |\n4MEMUSER | | +--Other: 10,248 bytes / 12 allocations\n1MEMUSER |\n2MEMUSER +--Unknown: 513,728 bytes / 2065 allocations\nNULL \n\n\n\n\n\nThis section does not record memory that is allocated by application or JNI code and is typically a little less than the\nvalue recorded by operating system tools.\n\n\nMEMINFO\n\n\nThis section relates to memory management, providing a breakdown of memory usage in the VM for the object heap,\ninternal memory, memory used for classes, the JIT code cache, and JIT data cache in decimal and hexadecimal format.\nYou can also find out which garbage collection policy is in use when the dump is produced. \n\n\nThe object memory area (\n1STHEAPTYPE\n) records each memory region in use, its start and end address, and region size.\nFurther information is recorded about the memory segments used for internal memory, class memory, the JIT code cache and JIT data cache (\n1STSEGMENT\n).\nThis information includes the address of the segment control data structure, the start and end address of the native memory segment, as well as\nthe segment size.\n\n\nFor clarity, the following example shows a shortened version of this section, where \n...\n indicates that lines are removed:\n\n\nNULL ------------------------------------------------------------------------\n0SECTION MEMINFO subcomponent dump routine\nNULL =================================\nNULL \n1STHEAPTYPE Object Memory\nNULL id start end size space/region\n1STHEAPSPACE 0x00007FF4F00744A0 -- -- -- Generational\n1STHEAPREGION 0x00007FF4F0074CE0 0x0000000087F40000 0x0000000088540000 0x0000000000600000 Generational/Tenured Region\n1STHEAPREGION 0x00007FF4F0074930 0x00000000FFE00000 0x00000000FFF00000 0x0000000000100000 Generational/Nursery Region\n1STHEAPREGION 0x00007FF4F0074580 0x00000000FFF00000 0x0000000100000000 0x0000000000100000 Generational/Nursery Region\nNULL\n1STHEAPTOTAL Total memory: 8388608 (0x0000000000800000)\n1STHEAPINUSE Total memory in use: 2030408 (0x00000000001EFB48)\n1STHEAPFREE Total memory free: 6358200 (0x00000000006104B8)\nNULL\n1STSEGTYPE Internal Memory\nNULL segment start alloc end type size\n1STSEGMENT 0x00007FF4F004CBC8 0x00007FF4CD33C000 0x00007FF4CD33C000 0x00007FF4CE33C000 0x01000440 0x0000000001000000\n1STSEGMENT 0x00007FF4F004CB08 0x00007FF4DE43D030 0x00007FF4DE517770 0x00007FF4DE53D030 0x00800040 0x0000000000100000\nNULL\n1STSEGTOTAL Total memory: 17825792 (0x0000000001100000)\n1STSEGINUSE Total memory in use: 894784 (0x00000000000DA740)\n1STSEGFREE Total memory free: 16931008 (0x00000000010258C0)\nNULL \n1STSEGTYPE Class Memory\nNULL segment start alloc end type size\n1STSEGMENT 0x00007FF4F03B5638 0x0000000001053D98 0x000000000105BD98 0x000000000105BD98 0x00010040 0x0000000000008000\n1STSEGMENT 0x00007FF4F03B5578 0x0000000001048188 0x0000000001050188 0x0000000001050188 0x00010040 0x0000000000008000\n...\nNULL\n1STSEGTOTAL Total memory: 3512520 (0x00000000003598C8)\n1STSEGINUSE Total memory in use: 3433944 (0x00000000003465D8)\n1STSEGFREE Total memory free: 78576 (0x00000000000132F0)\nNULL \n1STSEGTYPE JIT Code Cache\nNULL segment start alloc end type size\n1STSEGMENT 0x00007FF4F00961F8 0x00007FF4CE43D000 0x00007FF4CE445790 0x00007FF4DE43D000 0x00000068 0x0000000010000000\nNULL\n1STSEGTOTAL Total memory: 268435456 (0x0000000010000000)\n1STSEGINUSE Total memory in use: 34704 (0x0000000000008790)\n1STSEGFREE Total memory free: 268400752 (0x000000000FFF7870)\n1STSEGLIMIT Allocation limit: 268435456 (0x0000000010000000)\nNULL \n1STSEGTYPE JIT Data Cache\nNULL segment start alloc end type size\n1STSEGMENT 0x00007FF4F0096668 0x00007FF4CC553030 0x00007FF4CC753030 0x00007FF4CC753030 0x00000048 0x0000000000200000\nNULL\n1STSEGTOTAL Total memory: 2097152 (0x0000000000200000)\n1STSEGINUSE Total memory in use: 2097152 (0x0000000000200000)\n1STSEGFREE Total memory free: 0 (0x0000000000000000)\n1STSEGLIMIT Allocation limit: 402653184 (0x0000000018000000)\nNULL \n1STGCHTYPE GC History \nNULL \n\n\n\n\n\nIn the example, the GC History (\n1STGCHTYPE\n) section is blank. This section is populated if a garbage collection cycle occurred in\na VM that is being diagnosed with the trace facility.\n\n\nLOCKS\n\n\nThis section of the Java dump provides information about locks, which protect shared resources from being accessed by more than one entity at a time. The information is essential in a deadlock situation, where two threads attempt to synchronize on an object and lock an instance of a class. Precise information is recorded about the threads that are causing the problem, which enables you to identify the root cause.\n\n\nThe following example shows a typical LOCKS section, where no deadlocks existed at the time the dump was triggered. For clarity, the following example shows a shortened version of this section, where \n...\n indicates that lines are removed:\n\n\nNULL\n \n------------------------------------------------------------------------\n\n\n0\nSECTION\n \nLOCKS\n \nsubcomponent\n \ndump\n \nroutine\n\n\nNULL\n \n===============================\n\n\nNULL\n \n\n1L\nKPOOLINFO\n \nMonitor\n \npool\n \ninfo\n:\n\n\n2L\nKPOOLTOTAL\n \nCurrent\n \ntotal\n \nnumber\n \nof\n \nmonitors\n:\n \n3\n\n\nNULL\n \n\n1L\nKMONPOOLDUMP\n \nMonitor\n \nPool\n \nDump\n \n(\nflat\n \n&\n \ninflated\n \nobject\n-\nmonitors\n)\n:\n\n\n2L\nKMONINUSE\n \nsys_mon_t\n:\n0x00007FF4B0001D78\n \ninfl_mon_t\n:\n \n0x00007FF4B0001DF8\n:\n\n\n3L\nKMONOBJECT\n \njava\n/\nlang\n/\nref\n/\nReferenceQueue\n@0x00000000FFE26A10\n:\n \n<\nunowned\n>\n\n\n3L\nKNOTIFYQ\n \nWaiting\n \nto\n \nbe\n \nnotified\n:\n\n\n3L\nKWAITNOTIFY\n \n\"Common-Cleaner\"\n \n(\nJ9VMThread\n:\n0x0000000000FD0100\n)\n\n\nNULL\n \n\n1L\nKREGMONDUMP\n \nJVM\n \nSystem\n \nMonitor\n \nDump\n \n(\nregistered\n \nmonitors\n)\n:\n\n\n2L\nKREGMON\n \nThread\n \nglobal\n \nlock\n \n(\n0x00007FF4F0004FE8\n)\n:\n \n<\nunowned\n>\n\n\n2L\nKREGMON\n \n&\n(\nPPG_mem_mem32_subAllocHeapMem32\n.\nmonitor\n)\n \nlock\n \n(\n0x00007FF4F0005098\n)\n:\n \n<\nunowned\n>\n\n\n2L\nKREGMON\n \nNLS\n \nhash\n \ntable\n \nlock\n \n(\n0x00007FF4F0005148\n)\n:\n \n<\nunowned\n>\n\n\n...\n\n\nNULL\n \n\n\n\n\n\nTHREADS\n\n\nThe THREADS section of a Java dump file provides summary information about the VM thread pool and detailed information about Java threads, native threads, and stack traces. Understanding the content of this section can help you diagnose problems that are caused by blocked or waiting threads.\n\n\nA Java thread runs on a native thread. Several lines are recorded for each Java thread in the \nThread Details\n subsection, which include the following key pieces of information:\n\n\n\n\n3XMTHREADINFO\n: The thread name, address information for the VM thread structures and Java thread object, the thread state, and thread priority.\n\n\n3XMJAVALTHREAD\n: The Java thread ID and daemon status from the thread object.\n\n\n3XMTHREADINFO1\n: The native operating system thread ID, priority, scheduling policy, internal VM thread state, and VM thread flags.\n\n\n3XMTHREADINFO2\n: The native stack address range.\n\n\n3XMTHREADINFO3\n: Java callstack information (\n4XESTACKTRACE\n) or Native call stack information (\n4XENATIVESTACK\n).\n\n\n5XESTACKTRACE\n: This line indicates whether locks were taken by a specific method.\n\n\n\n\nJava thread priorities are mapped to operating system priority values. Thread states are shown in the following table:\n\n\n\n\n\n\n\n\nThread state value\n\n\nStatus\n\n\nDescription\n\n\n\n\n\n\n\n\n\n\nR\n\n\nRunnable\n\n\nThe thread is able to run\n\n\n\n\n\n\nCW\n\n\nCondition Wait\n\n\nThe thread is waiting\n\n\n\n\n\n\nS\n\n\nSuspended\n\n\nThe thread is suspended by another thread\n\n\n\n\n\n\nZ\n\n\nZombie\n\n\nThe thread is destroyed\n\n\n\n\n\n\nP\n\n\nParked\n\n\nThe thread is parked by \njava.util.concurrent\n\n\n\n\n\n\nB\n\n\nBlocked\n\n\nThe thread is waiting to obtain a lock\n\n\n\n\n\n\n\n\nFor threads that are parked (P), blocked (B), or waiting (CW), an additional line (\n3XMTHREADBLOCK\n) is included in the output that shows what the thread is parked on, blocked on, or waiting for.\n\n\nFor clarity, the following example shows a shortened version of a typical THREADS section, where \n...\n indicates that lines are removed:\n\n\nNULL\n \n------------------------------------------------------------------------\n\n\n0\nSECTION\n \nTHREADS\n \nsubcomponent\n \ndump\n \nroutine\n\n\nNULL\n \n=================================\n\n\nNULL\n\n\n1\nXMPOOLINFO\n \nJVM\n \nThread\n \npool\n \ninfo\n:\n\n\n2\nXMPOOLTOTAL\n \nCurrent\n \ntotal\n \nnumber\n \nof\n \npooled\n \nthreads\n:\n \n18\n\n\n2\nXMPOOLLIVE\n \nCurrent\n \ntotal\n \nnumber\n \nof\n \nlive\n \nthreads\n:\n \n16\n\n\n2\nXMPOOLDAEMON\n \nCurrent\n \ntotal\n \nnumber\n \nof\n \nlive\n \ndaemon\n \nthreads\n:\n \n15\n\n\nNULL\n \n\n1\nXMTHDINFO\n \nThread\n \nDetails\n\n\nNULL\n \n\n3\nXMTHREADINFO\n \n\"JIT Diagnostic Compilation Thread-7 Suspended\"\n \nJ9VMThread\n:\n0x0000000000EFC500\n,\n \nomrthread_t:\n0x00007FF4F00A77E8\n,\n \njava\n/\nlang\n/\nThread\n:\n0x00000000FFE97480\n,\n \nstate\n:\nR\n,\n \nprio\n=\n10\n\n\n3\nXMJAVALTHREAD\n \n(\njava\n/\nlang\n/\nThread\n \ngetId:\n0xA\n,\n \nisDaemon:true\n)\n\n\n3\nXMTHREADINFO1\n \n(\nnative\n \nthread\n \nID\n:\n0x7657\n,\n \nnative\n \npriority\n:\n0xB\n,\n \nnative\n \npolicy\n:\nUNKNOWN\n,\n \nvmstate\n:\nCW\n,\n \nvm\n \nthread\n \nflags\n:\n0x00000081\n)\n\n\n3\nXMTHREADINFO2\n \n(\nnative\n \nstack\n \naddress\n \nrange\n \nfrom\n:\n0x00007FF4CCC36000\n,\n \nto\n:\n0x00007FF4CCD36000\n,\n \nsize\n:\n0x100000\n)\n\n\n3\nXMCPUTIME\n \nCPU\n \nusage\n \ntotal\n:\n \n0.000037663\n \nsecs\n,\n \ncurrent\n \ncategory=\n\"JIT\"\n\n\n3\nXMHEAPALLOC\n \nHeap\n \nbytes\n \nallocated\n \nsince\n \nlast\n \nGC\n \ncycle\n=\n0\n \n(\n0x0\n)\n\n\n3\nXMTHREADINFO3\n \nNo\n \nJava\n \ncallstack\n \nassociated\n \nwith\n \nthis\n \nthread\n\n\n3\nXMTHREADINFO3\n \nNo\n \nnative\n \ncallstack\n \navailable\n \nfor\n \nthis\n \nthread\n\n\nNULL\n\n\n...\n\n\n3\nXMTHREADINFO\n \n\"Common-Cleaner\"\n \nJ9VMThread\n:\n0x0000000000FD0100\n,\n \nomrthread_t:\n0x00007FF4F022A520\n,\n \njava\n/\nlang\n/\nThread\n:\n0x00000000FFE26F40\n,\n \nstate\n:\nCW\n,\n \nprio\n=\n8\n\n\n3\nXMJAVALTHREAD\n \n(\njava\n/\nlang\n/\nThread\n \ngetId:\n0x2\n,\n \nisDaemon:true\n)\n\n\n3\nXMTHREADINFO1\n \n(\nnative\n \nthread\n \nID\n:\n0x765A\n,\n \nnative\n \npriority\n:\n0x8\n,\n \nnative\n \npolicy\n:\nUNKNOWN\n,\n \nvmstate\n:\nCW\n,\n \nvm\n \nthread\n \nflags\n:\n0x00080181\n)\n\n\n3\nXMTHREADINFO2\n \n(\nnative\n \nstack\n \naddress\n \nrange\n \nfrom\n:\n0x00007FF4CC0B8000\n,\n \nto\n:\n0x00007FF4CC0F8000\n,\n \nsize\n:\n0x40000\n)\n\n\n3\nXMCPUTIME\n \nCPU\n \nusage\n \ntotal\n:\n \n0.000150926\n \nsecs\n,\n \ncurrent\n \ncategory=\n\"Application\"\n\n\n3\nXMTHREADBLOCK\n \nWaiting\n \non\n:\n \njava\n/\nlang\n/\nref\n/\nReferenceQueue\n@\n0x00000000FFE26A10\n \nOwned\n \nby\n:\n \n<\nunowned\n>\n\n\n3\nXMHEAPALLOC\n \nHeap\n \nbytes\n \nallocated\n \nsince\n \nlast\n \nGC\n \ncycle\n=\n0\n \n(\n0x0\n)\n\n\n3\nXMTHREADINFO3\n \nJava\n \ncallstack\n:\n\n\n4\nXESTACKTRACE\n \nat\n \njava\n/\nlang\n/\nObject\n.\nwait\n(\nNative\n \nMethod\n)\n\n\n4\nXESTACKTRACE\n \nat\n \njava\n/\nlang\n/\nObject\n.\nwait\n(\nObject\n.\njava\n:\n221\n)\n\n\n4\nXESTACKTRACE\n \nat\n \njava\n/\nlang\n/\nref\n/\nReferenceQueue\n.\nremove\n(\nReferenceQueue\n.\njava\n:\n138\n)\n\n\n5\nXESTACKTRACE\n \n(\nentered\n \nlock\n:\n \njava\n/\nlang\n/\nref\n/\nReferenceQueue\n@\n0x00000000FFE26A10\n,\n \nentry\n \ncount\n:\n \n1\n)\n\n\n4\nXESTACKTRACE\n \nat\n \njdk\n/\ninternal\n/\nref\n/\nCleanerImpl\n.\nrun\n(\nCleanerImpl\n.\njava\n:\n148\n)\n\n\n4\nXESTACKTRACE\n \nat\n \njava\n/\nlang\n/\nThread\n.\nrun\n(\nThread\n.\njava\n:\n835\n)\n\n\n4\nXESTACKTRACE\n \nat\n \njdk\n/\ninternal\n/\nmisc\n/\nInnocuousThread\n.\nrun\n(\nInnocuousThread\n.\njava\n:\n122\n)\n\n\n3\nXMTHREADINFO3\n \nNo\n \nnative\n \ncallstack\n \navailable\n \nfor\n \nthis\n \nthread\n\n\nNULL\n\n\nNULL\n\n\n3\nXMTHREADINFO\n \n\"IProfiler\"\n \nJ9VMThread\n:\n0x0000000000F03D00\n,\n \nomrthread_t:\n0x00007FF4F00B06F8\n,\n \njava\n/\nlang\n/\nThread\n:\n0x00000000FFE97B60\n,\n \nstate\n:\nR\n,\n \nprio\n=\n5\n\n\n3\nXMJAVALTHREAD\n \n(\njava\n/\nlang\n/\nThread\n \ngetId:\n0xC\n,\n \nisDaemon:true\n)\n\n\n3\nXMTHREADINFO1\n \n(\nnative\n \nthread\n \nID\n:\n0x7659\n,\n \nnative\n \npriority\n:\n0x5\n,\n \nnative\n \npolicy\n:\nUNKNOWN\n,\n \nvmstate\n:\nCW\n,\n \nvm\n \nthread\n \nflags\n:\n0x00000081\n)\n\n\n3\nXMTHREADINFO2\n \n(\nnative\n \nstack\n \naddress\n \nrange\n \nfrom\n:\n0x00007FF4F8940000\n,\n \nto\n:\n0x00007FF4F8960000\n,\n \nsize\n:\n0x20000\n)\n\n\n3\nXMCPUTIME\n \nCPU\n \nusage\n \ntotal\n:\n \n0.004753103\n \nsecs\n,\n \ncurrent\n \ncategory=\n\"JIT\"\n\n\n3\nXMHEAPALLOC\n \nHeap\n \nbytes\n \nallocated\n \nsince\n \nlast\n \nGC\n \ncycle\n=\n0\n \n(\n0x0\n)\n\n\n3\nXMTHREADINFO3\n \nNo\n \nJava\n \ncallstack\n \nassociated\n \nwith\n \nthis\n \nthread\n\n\n3\nXMTHREADINFO3\n \nNo\n \nnative\n \ncallstack\n \navailable\n \nfor\n \nthis\n \nthread\n\n\nNULL\n\n\n...\n\n\n1\nXMWLKTHDERR\n \nThe\n \nfollowing\n \nwas\n \nreported\n \nwhile\n \ncollecting\n \nnative\n \nstacks\n:\n\n\n2\nXMWLKTHDERR\n \nunable\n \nto\n \ncount\n \nthreads\n(\n3\n,\n \n-\n2\n)\n\n\nNULL\n\n\n1\nXMTHDSUMMARY\n \nThreads\n \nCPU\n \nUsage\n \nSummary\n\n\nNULL\n \n=========================\n\n\nNULL\n\n\n1\nXMTHDCATINFO\n \nWarning\n:\n \nto\n \nget\n \nmore\n \naccurate\n \nCPU\n \ntimes\n \nfor\n \nthe\n \nGC\n,\n \nthe\n \noption\n \n-\nXX\n:-\nReduceCPUMonitorOverhead\n \ncan\n \nbe\n \nused\n.\n \nSee\n \nthe\n \nuser\n \nguide\n \nfor\n \nmore\n \ninformation\n.\n\n\nNULL\n\n\n1\nXMTHDCATEGORY\n \nAll\n \nJVM\n \nattached\n \nthreads\n:\n \n0.280083000\n \nsecs\n\n\n1\nXMTHDCATEGORY\n \n|\n\n\n2\nXMTHDCATEGORY\n \n+--\nSystem\n-\nJVM\n:\n \n0.270814000\n \nsecs\n\n\n2\nXMTHDCATEGORY\n \n|\n \n|\n\n\n3\nXMTHDCATEGORY\n \n|\n \n+--\nGC\n:\n \n0.000599000\n \nsecs\n\n\n2\nXMTHDCATEGORY\n \n|\n \n|\n\n\n3\nXMTHDCATEGORY\n \n|\n \n+--\nJIT\n:\n \n0.071904000\n \nsecs\n\n\n1\nXMTHDCATEGORY\n \n|\n\n\n2\nXMTHDCATEGORY\n \n+--\nApplication\n:\n \n0.009269000\n \nsecs\n\n\nNULL\n\n\n\n\n\n\nHOOKS\n\n\nThis section shows internal VM event callbacks, which are used for diagnosing performance problems in the VM. Multiple hook interfaces are listed, which include their individual hook events. The following example shows data for the \nJ9VMHookInterface\n, including the call site location (source file:line number), start time, and duration of the last callback and the longest callback.\n\n\nNULL ------------------------------------------------------------------------\n0SECTION HOOK subcomponent dump routine\nNULL ==============================\n1HKINTERFACE MM_OMRHookInterface\nNULL ------------------------------------------------------------------------\n1HKINTERFACE MM_PrivateHookInterface\nNULL ------------------------------------------------------------------------\n1HKINTERFACE MM_HookInterface\nNULL ------------------------------------------------------------------------\n1HKINTERFACE J9VMHookInterface\nNULL ------------------------------------------------------------------------\n2HKEVENTID 1\n3HKCALLCOUNT 18\n3HKLAST Last Callback\n4HKCALLSITE trcengine.c:392\n4HKSTARTTIME Start Time: 2018-08-30T21:55:47.601\n4HKDURATION DurationMs: 0\n3HKLONGST Longest Callback\n4HKCALLSITE trcengine.c:392\n4HKSTARTTIME Start Time: 2018-08-30T21:55:47.460\n4HKDURATION DurationMs: 1\nNULL\n...\n1HKINTERFACE J9VMZipCachePoolHookInterface\nNULL ------------------------------------------------------------------------\n1HKINTERFACE J9JITHookInterface\nNULL ------------------------------------------------------------------------\n2HKEVENTID 3\n3HKCALLCOUNT 65\n3HKLAST Last Callback\n4HKCALLSITE ../common/mgmtinit.c:191\n4HKSTARTTIME Start Time: 2018-08-30T21:55:47.601\n4HKDURATION DurationMs: 0\n3HKLONGST Longest Callback\n4HKCALLSITE ../common/mgmtinit.c:191\n4HKSTARTTIME Start Time: 2018-08-30T21:55:47.486\n4HKDURATION DurationMs: 0\n...\nNULL\n\n\n\n\n\nSHARED CLASSES\n\n\nIf the shared classes cache is enabled at run time, the information provided in a Java dump file describes settings that were used when creating the cache, together with summary information about the size and content of the cache.\n\n\nIn the following example, the shared classes cache was created with a Class Debug Area (\n-Xnolinenumbers=false\n). Byte code instrumentation (BCI) is enabled, which is the default, and VMs sharing the cache are allowed to store classpaths, which is also the default.\n\n\nThe \nCache Summary\n shows a cache size (\n2SCLTEXTCSZ\n) of 16776608 bytes, with a soft maximum size (\n2SCLTEXTSMB\n) also of 16776608 bytes, which leaves 12691668 bytes of free space (\n2SCLTEXTFRB\n). The size of the Class Debug Area (\n2SCLTEXTDAS\n) is 1331200 bytes and only 11% of this space is used.\n\n\nIn the \nCache Memory Status\n subsection, the line \n2SCLTEXTCMDT\n indicates the name and location of the shared cache and \ncr\n indicates that the cache is a 64-bit compressed references cache.\n\n\nNULL ------------------------------------------------------------------------\n0SECTION SHARED CLASSES subcomponent dump routine\nNULL ========================================\nNULL\n1SCLTEXTCRTW Cache Created With\nNULL ------------------\nNULL\n2SCLTEXTXNL -Xnolinenumbers = false\n2SCLTEXTBCI BCI Enabled = true\n2SCLTEXTBCI Restrict Classpaths = false\nNULL\n1SCLTEXTCSUM Cache Summary\nNULL ------------------\nNULL\n2SCLTEXTNLC No line number content = false\n2SCLTEXTLNC Line number content = true\nNULL\n2SCLTEXTRCS ROMClass start address = 0x00007F423061C000\n2SCLTEXTRCE ROMClass end address = 0x00007F42307B9A28\n2SCLTEXTMSA Metadata start address = 0x00007F42313D42FC\n2SCLTEXTCEA Cache end address = 0x00007F4231600000\n2SCLTEXTRTF Runtime flags = 0x00102001ECA6028B\n2SCLTEXTCGN Cache generation = 35\nNULL\n2SCLTEXTCSZ Cache size = 16776608\n2SCLTEXTSMB Softmx bytes = 16776608\n2SCLTEXTFRB Free bytes = 12691668\n2SCLTEXTRCB ROMClass bytes = 1694248\n2SCLTEXTAOB AOT code bytes = 0\n2SCLTEXTADB AOT data bytes = 0\n2SCLTEXTAHB AOT class hierarchy bytes = 32\n2SCLTEXTATB AOT thunk bytes = 0\n2SCLTEXTARB Reserved space for AOT bytes = -1\n2SCLTEXTAMB Maximum space for AOT bytes = -1\n2SCLTEXTJHB JIT hint bytes = 308\n2SCLTEXTJPB JIT profile bytes = 2296\n2SCLTEXTJRB Reserved space for JIT data bytes = -1\n2SCLTEXTJMB Maximum space for JIT data bytes = -1\n2SCLTEXTNOB Java Object bytes = 0\n2SCLTEXTZCB Zip cache bytes = 919328\n2SCLTEXTRWB ReadWrite bytes = 114080\n2SCLTEXTJCB JCL data bytes = 0\n2SCLTEXTBDA Byte data bytes = 0\n2SCLTEXTMDA Metadata bytes = 23448\n2SCLTEXTDAS Class debug area size = 1331200\n2SCLTEXTDAU Class debug area % used = 11%\n2SCLTEXTDAN Class LineNumberTable bytes = 156240\n2SCLTEXTDAV Class LocalVariableTable bytes = 0\nNULL\n2SCLTEXTNRC Number ROMClasses = 595\n2SCLTEXTNAM Number AOT Methods = 0\n2SCLTEXTNAD Number AOT Data Entries = 0\n2SCLTEXTNAH Number AOT Class Hierarchy = 1\n2SCLTEXTNAT Number AOT Thunks = 0\n2SCLTEXTNJH Number JIT Hints = 14\n2SCLTEXTNJP Number JIT Profiles = 20\n2SCLTEXTNCP Number Classpaths = 1\n2SCLTEXTNUR Number URLs = 0\n2SCLTEXTNTK Number Tokens = 0\n2SCLTEXTNOJ Number Java Objects = 0\n2SCLTEXTNZC Number Zip Caches = 5\n2SCLTEXTNJC Number JCL Entries = 0\n2SCLTEXTNST Number Stale classes = 0\n2SCLTEXTPST Percent Stale classes = 0%\nNULL\n2SCLTEXTCPF Cache is 24% full\nNULL\n1SCLTEXTCMST Cache Memory Status\nNULL ------------------\n1SCLTEXTCNTD Cache Name Feature Memory type Cache path\nNULL\n2SCLTEXTCMDT sharedcc_doc-javacore CR Memory mapped file /tmp/javasharedresources/C290M4F1A64P_sharedcc_doc-javacore_G35\nNULL\n1SCLTEXTCMST Cache Lock Status\nNULL ------------------\n1SCLTEXTCNTD Lock Name Lock type TID owning lock\nNULL\n2SCLTEXTCWRL Cache write lock File lock Unowned\n2SCLTEXTCRWL Cache read/write lock File lock Unowned\nNULL\n\n\n\n\n\nCLASSES\n\n\nThe classes section shows information about class loaders. The first part is a summary that records each available class loader (\n2CLTEXTCLLOADER\n) followed by the number of libraries and classes that it loaded. This information is followed by a more detailed list of libraries (\n1CLTEXTCLLIB\n) and classes (\n1CLTEXTCLLO\n) that are loaded.\n\n\nIn the example you can see that the \njava/lang/InternalAnonymousClassLoader\n loaded 2 classes, \njdk/internal/loader/BuiltinClassLoader$$Lambda$2/00000000F03876A0(0x0000000001030F00)\n and \njdk/internal/loader/BuiltinClassLoader$$Lambda$1/00000000F00D2460(0x0000000001018A00)\n.\n\n\nNULL ------------------------------------------------------------------------\n0SECTION CLASSES subcomponent dump routine\nNULL =================================\n1CLTEXTCLLOS Classloader summaries\n1CLTEXTCLLSS 12345678: 1=primordial,2=extension,3=shareable,4=middleware,5=system,6=trusted,7=application,8=delegating\n2CLTEXTCLLOADER p---st-- Loader *System*(0x00000000FFE1D258)\n3CLNMBRLOADEDLIB Number of loaded libraries 5\n3CLNMBRLOADEDCL Number of loaded classes 638\n2CLTEXTCLLOADER -x--st-- Loader jdk/internal/loader/ClassLoaders$PlatformClassLoader(0x00000000FFE1D4F0), Parent *none*(0x0000000000000000)\n3CLNMBRLOADEDLIB Number of loaded libraries 0\n3CLNMBRLOADEDCL Number of loaded classes 0\n2CLTEXTCLLOADER ----st-- Loader java/lang/InternalAnonymousClassLoader(0x00000000FFE1DFD0), Parent *none*(0x0000000000000000)\n3CLNMBRLOADEDLIB Number of loaded libraries 0\n3CLNMBRLOADEDCL Number of loaded classes 2\n2CLTEXTCLLOADER -----ta- Loader jdk/internal/loader/ClassLoaders$AppClassLoader(0x00000000FFE1DAD0), Parent jdk/internal/loader/ClassLoaders$PlatformClassLoader(0x00000000FFE1D4F0)\n3CLNMBRLOADEDLIB Number of loaded libraries 0\n3CLNMBRLOADEDCL Number of loaded classes 0\n1CLTEXTCLLIB ClassLoader loaded libraries\n2CLTEXTCLLIB Loader *System*(0x00000000FFE1D258)\n3CLTEXTLIB /home/me/openj9-openjdk-jdk9/build/linux-x86_64-normal-server-release/images/jdk/lib/compressedrefs/jclse9_29\n3CLTEXTLIB /home/me/openj9-openjdk-jdk9/build/linux-x86_64-normal-server-release/images/jdk/lib/java\n3CLTEXTLIB /home/me/openj9-openjdk-jdk9/build/linux-x86_64-normal-server-release/images/jdk/lib/compressedrefs/j9jit29\n3CLTEXTLIB /home/me/openj9-openjdk-jdk9/build/linux-x86_64-normal-server-release/images/jdk/lib/zip\n3CLTEXTLIB /home/me/openj9-openjdk-jdk9/build/linux-x86_64-normal-server-release/images/jdk/lib/nio\n1CLTEXTCLLOD ClassLoader loaded classes\n2CLTEXTCLLOAD Loader *System*(0x00000000FFE1D258)\n3CLTEXTCLASS [Ljava/lang/Thread$State;(0x0000000001056400)\n...\n2CLTEXTCLLOAD Loader jdk/internal/loader/ClassLoaders$PlatformClassLoader(0x00000000FFE1D4F0)\n2CLTEXTCLLOAD Loader java/lang/InternalAnonymousClassLoader(0x00000000FFE1DFD0)\n3CLTEXTCLASS jdk/internal/loader/BuiltinClassLoader$$Lambda$2/00000000F03876A0(0x0000000001030F00)\n3CLTEXTCLASS jdk/internal/loader/BuiltinClassLoader$$Lambda$1/00000000F00D2460(0x0000000001018A00)\n2CLTEXTCLLOAD Loader jdk/internal/loader/ClassLoaders$AppClassLoader(0x00000000FFE1DAD0)\n\n\n\n\n\nScenarios\n\n\nGeneral Protection Fault\n\n\nIn this scenario, a Java application has crashed due to a General Protection Fault (GPF), automatically generating a Java dump file.\n\n\nThe first section of the file (TITLE) tells you that the GPF triggered the Java dump.\n\n\n0SECTION TITLE subcomponent dump routine\nNULL ===============================\n1TICHARSET UTF-8\n1TISIGINFO Dump Event \"gpf\" (00002000) received\n1TIDATETIME Date: 2018/09/24 at 15:18:03:115\n1TINANOTIME System nanotime: 4498949283020796\n1TIFILENAME Javacore filename: /home/test/JNICrasher/javacore.20180924.151801.29399.0002.txt\n1TIREQFLAGS Request Flags: 0x81 (exclusive+preempt)\n1TIPREPSTATE Prep State: 0x100 (trace_disabled)\n1TIPREPINFO Exclusive VM access not taken: data may not be consistent across javacore sections\n\n\n\n\n\nTo troubleshoot this problem, you need to know which thread caused the GPF to occur. The thread that was running at the time of the crash is reported as the \ncurrent thread\n\nin the THREADS section of the Java dump. Here is an extract from the THREADS section:\n\n\nNULL ------------------------------------------------------------------------\n0SECTION THREADS subcomponent dump routine\nNULL =================================\nNULL\n1XMPOOLINFO JVM Thread pool info:\n2XMPOOLTOTAL Current total number of pooled threads: 16\n2XMPOOLLIVE Current total number of live threads: 15\n2XMPOOLDAEMON Current total number of live daemon threads: 14\nNULL \n1XMCURTHDINFO Current thread\n3XMTHREADINFO \"main\" J9VMThread:0xB6B60E00, omrthread_t:0xB6B049D8, java/lang/Thread:0xB55444D0, state:R, prio=5\n3XMJAVALTHREAD (java/lang/Thread getId:0x1, isDaemon:false)\n3XMTHREADINFO1 (native thread ID:0x72D8, native priority:0x5, native policy:UNKNOWN, vmstate:R, vm thread flags:0x00000000)\n3XMTHREADINFO2 (native stack address range from:0xB6CE3000, to:0xB74E4000, size:0x801000)\n3XMCPUTIME CPU usage total: 0.319865924 secs, current category=\"Application\"\n3XMHEAPALLOC Heap bytes allocated since last GC cycle=778008 (0xBDF18)\n3XMTHREADINFO3 Java callstack:\n4XESTACKTRACE at JNICrasher.doSomethingThatCrashes(Native Method)\n4XESTACKTRACE at JNICrasher.main(JNICrasher.java:7)\n3XMTHREADINFO3 Native callstack:\n4XENATIVESTACK (0xB6C6F663 [libj9prt29.so+0x3b663])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB6C6F1CE [libj9prt29.so+0x3b1ce])\n4XENATIVESTACK (0xB6C6F2C6 [libj9prt29.so+0x3b2c6])\n4XENATIVESTACK (0xB6C6ED93 [libj9prt29.so+0x3ad93])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB6C6ED07 [libj9prt29.so+0x3ad07])\n4XENATIVESTACK (0xB6C6AA3D [libj9prt29.so+0x36a3d])\n4XENATIVESTACK (0xB6C6C3A4 [libj9prt29.so+0x383a4])\n4XENATIVESTACK (0xB667FA19 [libj9dmp29.so+0xfa19])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB66878CF [libj9dmp29.so+0x178cf])\n4XENATIVESTACK (0xB6688083 [libj9dmp29.so+0x18083])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB6680C0D [libj9dmp29.so+0x10c0d])\n4XENATIVESTACK (0xB667F9D7 [libj9dmp29.so+0xf9d7])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB668B02F [libj9dmp29.so+0x1b02f])\n4XENATIVESTACK (0xB668B4D3 [libj9dmp29.so+0x1b4d3])\n4XENATIVESTACK (0xB66740F1 [libj9dmp29.so+0x40f1])\n4XENATIVESTACK (0xB66726FA [libj9dmp29.so+0x26fa])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB66726A9 [libj9dmp29.so+0x26a9])\n4XENATIVESTACK (0xB6676AE4 [libj9dmp29.so+0x6ae4])\n4XENATIVESTACK (0xB668D75A [libj9dmp29.so+0x1d75a])\n4XENATIVESTACK (0xB6A28DD4 [libj9vm29.so+0x81dd4])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB6A289EE [libj9vm29.so+0x819ee])\n4XENATIVESTACK (0xB6A29A40 [libj9vm29.so+0x82a40])\n4XENATIVESTACK (0xB6C52B6A [libj9prt29.so+0x1eb6a])\n4XENATIVESTACK __kernel_rt_sigreturn+0x0 (0xB7747410)\n4XENATIVESTACK (0xB75330B6 [libffi29.so+0x50b6])\n4XENATIVESTACK ffi_raw_call+0xad (0xB7531C53 [libffi29.so+0x3c53])\n4XENATIVESTACK (0xB69BE4AB [libj9vm29.so+0x174ab])\n4XENATIVESTACK (0xB6A665BC [libj9vm29.so+0xbf5bc])\n4XENATIVESTACK (0xB6A15552 [libj9vm29.so+0x6e552])\n4XENATIVESTACK (0xB6A30894 [libj9vm29.so+0x89894])\n4XENATIVESTACK (0xB6A6F169 [libj9vm29.so+0xc8169])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB6A6F1FA [libj9vm29.so+0xc81fa])\n4XENATIVESTACK (0xB6A30994 [libj9vm29.so+0x89994])\n4XENATIVESTACK (0xB6A2CE4C [libj9vm29.so+0x85e4c])\n4XENATIVESTACK (0xB770487D [libjli.so+0x787d])\n4XENATIVESTACK (0xB7719F72 [libpthread.so.0+0x6f72])\n4XENATIVESTACK clone+0x5e (0xB763543E [libc.so.6+0xee43e])\n\n\n\n\n\nThe extract tells you that the current thread was \njava/lang/Thread\n, and information is provided about the Java callstack and native callstack\n(\n3XMTHREADINFO3\n) at the point at which the crash occurred. To simulate a crash caused by a bug in an application, this example\ncalls a JNI method whose native implementation causes a crash. The Java callstack shows the call to the JNI native method (\nJNIcrasher\n), and the\nnative callstack shows the point of failure. In this example, the native call stack does not include any function names to help you isolate the error\nin the native code. You can get this information from a system dump, which is usually produced alongside the Java dump. Open the\nsystem dump with the \nDump viewer\n and use the \ninfo thread\n command to print the Java and native stack for the current thread.\n\n\nJava OutOfMemoryError\n\n\nIn this scenario, the Java heap runs out of memory, causing an \nOutOfMemoryError\n, which automatically generates a Java dump file.\n\n\nThe first section of the file (TITLE) tells you that a \nsysthrow\n event triggered the Java dump as a result of an OOM (\njava/lang/OutOfMemoryError\n) for\nJava heap space.\n\n\n0SECTION TITLE subcomponent dump routine\nNULL ===============================\n1TICHARSET UTF-8\n1TISIGINFO Dump Event \"systhrow\" (00040000) Detail \"java/lang/OutOfMemoryError\" \"Java heap space\" received\n1TIDATETIME Date: 2018/09/14 at 15:29:42:709\n1TINANOTIME System nanotime: 3635648876608448\n1TIFILENAME Javacore filename: /home/cheesemp/test/javacore.20180914.152929.18885.0003.txt\n1TIREQFLAGS Request Flags: 0x81 (exclusive+preempt)\n1TIPREPSTATE Prep State: 0x104 (exclusive_vm_access+trace_disabled)\n\n\n\n\n\nThe MEMINFO section records how much memory is allocated to the Java heap (\n1STHEAPTYPE Object Memory\n), how much is in use, and how much is free. Solving\nyour problem might be as simple as setting a larger heap size when you start your application.\n\n\nIf you don't know what size the Java heap was set to, you might find that information in the ENVINFO section, which records the command line options that\nwere used when the application started. Look or search for the \n1CIUSERARGS UserArgs:\n string and review the entries recorded for all lines that\nstart \n2CIUSERARG\n. The Java heap size is set by the \n-Xmx\n option. If the size has not been set on the command line by \n-Xmx\n, the default value applies, which\nyou can find in \nDefault Settings\n.\n\n\nIn this scenario the solution to the problem is not an adjustment to the Java heap size. Here is the MEMINFO section:\n\n\n0SECTION MEMINFO subcomponent dump routine\nNULL =================================\nNULL\n1STHEAPTYPE Object Memory\nNULL id start end size space/region\n1STHEAPSPACE 0xB6B49D20 -- -- -- Generational\n1STHEAPREGION 0xB6B4A078 0x95750000 0xB5470000 0x1FD20000 Generational/Tenured Region\n1STHEAPREGION 0xB6B49F10 0xB5470000 0xB54C0000 0x00050000 Generational/Nursery Region\n1STHEAPREGION 0xB6B49DA8 0xB54C0000 0xB5750000 0x00290000 Generational/Nursery Region\nNULL\n1STHEAPTOTAL Total memory: 536870912 (0x20000000)\n1STHEAPINUSE Total memory in use: 302603160 (0x12095B98)\n1STHEAPFREE Total memory free: 234267752 (0x0DF6A468)\n\n\n\n\n\nThe output shows that only 56% of the Java heap is in use, so this suggests that the application is trying to do something sub-optimal. To\ninvestigate further you need to work out which thread was the current thread when the OOM occurred to see what it was trying to do. As in the previous scenario, you can find the\n\ncurrent thread\n in the THREADS section. Here is an extract from the output:\n\n\n0\nSECTION\n \nTHREADS\n \nsubcomponent\n \ndump\n \nroutine\n\n\nNULL\n \n=================================\n\n\nNULL\n\n\n1\nXMPOOLINFO\n \nJVM\n \nThread\n \npool\n \ninfo\n:\n\n\n2\nXMPOOLTOTAL\n \nCurrent\n \ntotal\n \nnumber\n \nof\n \npooled\n \nthreads\n:\n \n16\n\n\n2\nXMPOOLLIVE\n \nCurrent\n \ntotal\n \nnumber\n \nof\n \nlive\n \nthreads\n:\n \n16\n\n\n2\nXMPOOLDAEMON\n \nCurrent\n \ntotal\n \nnumber\n \nof\n \nlive\n \ndaemon\n \nthreads\n:\n \n15\n\n\nNULL\n\n\n1\nXMCURTHDINFO\n \nCurrent\n \nthread\n\n\n3\nXMTHREADINFO\n \n\"main\"\n \nJ9VMThread\n:\n0xB6B60C00\n,\n \nomrthread_t\n:\n0xB6B049D8\n,\n \njava\n/\nlang\n/\nThread\n:\n0x95764520\n,\n \nstate\n:\nR\n,\n \nprio\n=\n5\n\n\n3\nXMJAVALTHREAD\n \n(\njava\n/\nlang\n/\nThread\n \ngetId\n:\n0x1\n,\n \nisDaemon\n:\nfalse\n)\n\n\n3\nXMTHREADINFO1\n \n(\nnative\n \nthread\n \nID\n:\n0x49C6\n,\n \nnative\n \npriority\n:\n0x5\n,\n \nnative\n \npolicy\n:\nUNKNOWN\n,\n \nvmstate\n:\nR\n,\n \nvm\n \nthread\n \nflags\n:\n0x00001020\n)\n\n\n3\nXMTHREADINFO2\n \n(\nnative\n \nstack\n \naddress\n \nrange\n \nfrom\n:\n0xB6CB5000\n,\n \nto\n:\n0xB74B6000\n,\n \nsize\n:\n0x801000\n)\n\n\n3\nXMCPUTIME\n \nCPU\n \nusage\n \ntotal\n:\n \n8.537823831\n \nsecs\n,\n \ncurrent\n \ncategory\n=\n\"Application\"\n\n\n3\nXMHEAPALLOC\n \nHeap\n \nbytes\n \nallocated\n \nsince\n \nlast\n \nGC\n \ncycle\n=\n0\n \n(\n0x0\n)\n\n\n3\nXMTHREADINFO3\n \nJava\n \ncallstack\n:\n\n\n4\nXESTACKTRACE\n \nat\n \njava\n/\nlang\n/\nStringBuffer\n.\nensureCapacityImpl\n(\nStringBuffer\n.\njava\n:\n696\n)\n\n\n4\nXESTACKTRACE\n \nat\n \njava\n/\nlang\n/\nStringBuffer\n.\nappend\n(\nStringBuffer\n.\njava\n:\n486\n(\nCompiled\n \nCode\n))\n\n\n5\nXESTACKTRACE\n \n(\nentered\n \nlock\n:\n \njava\n/\nlang\n/\nStringBuffer\n@0x957645B8\n,\n \nentry\n \ncount\n:\n \n1\n)\n\n\n4\nXESTACKTRACE\n \nat\n \njava\n/\nlang\n/\nStringBuffer\n.\nappend\n(\nStringBuffer\n.\njava\n:\n428\n(\nCompiled\n \nCode\n))\n\n\n4\nXESTACKTRACE\n \nat\n \nHeapBreaker\n.\nmain\n(\nHeapBreaker\n.\njava\n:\n34\n(\nCompiled\n \nCode\n))\n\n\n3\nXMTHREADINFO3\n \nNative\n \ncallstack\n:\n\n\n4\nXENATIVESTACK\n \n(\n0xB6C535B3\n \n[\nlibj9prt29\n.\nso\n+\n0x3b5b3\n])\n\n\n4\nXENATIVESTACK\n \n(\n0xB6C36F3E\n \n[\nlibj9prt29\n.\nso\n+\n0x1ef3e\n])\n\n\n4\nXENATIVESTACK\n \n(\n0xB6C5311E\n \n[\nlibj9prt29\n.\nso\n+\n0x3b11e\n])\n\n\n4\nXENATIVESTACK\n \n(\n0xB6C53216\n \n[\nlibj9prt29\n.\nso\n+\n0x3b216\n])\n\n\n4\nXENATIVESTACK\n \n(\n0xB6C52CE3\n \n[\nlibj9prt29\n.\nso\n+\n0x3ace3\n])\n\n\n4\nXENATIVESTACK\n \n(\n0xB6C36F3E\n \n[\nlibj9prt29\n.\nso\n+\n0x1ef3e\n])\n\n\n4\nXENATIVESTACK\n \n(\n0xB6C52C57\n \n[\nlibj9prt29\n.\nso\n+\n0x3ac57\n])\n\n\n4\nXENATIVESTACK\n \n(\n0xB6C4E9CD\n \n[\nlibj9prt29\n.\nso\n+\n0x369cd\n])\n\n\n4\nXENATIVESTACK\n \n(\n0xB6C502FA\n \n[\nlibj9prt29\n.\nso\n+\n0x382fa\n])\n\n\n\n\n\n\nTo simulate a Java \nOutOfMemoryError\n, this example application repeatedly appends characters to a \nStringBuffer\n object in an infinite loop. The Java callstack shows the \nHeapBreaker.main\n method appending characters (\njava/lang/StringGuffer.append\n) until the method \njava/lang/StringBuffer.ensureCapacityImpl()\n throws the \nOutOfMemoryError\n.\n\n\nStringBuffer objects are wrappers for character arrays (\nchar[]\n) and when the capacity of the underlying array is reached, the contents are automatically copied into a new, larger array. The new array is created in the \nStringBuffer.ensureCapacity()\n method, which more or less doubles the size of the old array. In this scenario, the array takes up all the remaining space in the Java heap.\n\n\nThe MEMINFO section of the Java dump file can also tell you when an unexpectedly large allocation request causes an OOM. Look for the GC History (\n1STGCHTYPE\n) section, which details allocation requests that trigger GC activity. In the sample output you can see that a large allocation request (\nrequestedbytes=603979784\n) triggered a global GC. When the GC could not free up sufficient space in the heap to satisfy the request, the allocation failure generated the OOM.\n\n\n1STGCHTYPE GC History \n3STHSTTYPE 14:29:29:580239000 GMT j9mm.101 - J9AllocateIndexableObject() returning NULL! 0 bytes requested for object of class B6BBC300 from memory space 'Generational' id=B6B49D20\n3STHSTTYPE 14:29:29:579916000 GMT j9mm.134 - Allocation failure end: newspace=2686912/3014656 oldspace=231597224/533856256 loa=5338112/5338112\n3STHSTTYPE 14:29:29:579905000 GMT j9mm.470 - Allocation failure cycle end: newspace=2686912/3014656 oldspace=231597224/533856256 loa=5338112/5338112\n3STHSTTYPE 14:29:29:579859000 GMT j9mm.475 - GlobalGC end: workstackoverflow=0 overflowcount=0 memory=234284136/536870912\n3STHSTTYPE 14:29:29:579807000 GMT j9mm.90 - GlobalGC collect complete\n3STHSTTYPE 14:29:29:579776000 GMT j9mm.137 - Compact end: bytesmoved=301989896\n3STHSTTYPE 14:29:29:313899000 GMT j9mm.136 - Compact start: reason=compact to meet allocation\n3STHSTTYPE 14:29:29:313555000 GMT j9mm.57 - Sweep end\n3STHSTTYPE 14:29:29:310772000 GMT j9mm.56 - Sweep start\n3STHSTTYPE 14:29:29:310765000 GMT j9mm.94 - Class unloading end: classloadersunloaded=0 classesunloaded=0\n3STHSTTYPE 14:29:29:310753000 GMT j9mm.60 - Class unloading start\n3STHSTTYPE 14:29:29:310750000 GMT j9mm.55 - Mark end\n3STHSTTYPE 14:29:29:306013000 GMT j9mm.54 - Mark start\n3STHSTTYPE 14:29:29:305957000 GMT j9mm.474 - GlobalGC start: globalcount=9\n3STHSTTYPE 14:29:29:305888000 GMT j9mm.475 - GlobalGC end: workstackoverflow=0 overflowcount=0 memory=234284136/536870912\n3STHSTTYPE 14:29:29:305837000 GMT j9mm.90 - GlobalGC collect complete\n3STHSTTYPE 14:29:29:305808000 GMT j9mm.137 - Compact end: bytesmoved=189784\n3STHSTTYPE 14:29:29:298042000 GMT j9mm.136 - Compact start: reason=compact to meet allocation\n3STHSTTYPE 14:29:29:297695000 GMT j9mm.57 - Sweep end\n3STHSTTYPE 14:29:29:291696000 GMT j9mm.56 - Sweep start\n3STHSTTYPE 14:29:29:291692000 GMT j9mm.55 - Mark end\n3STHSTTYPE 14:29:29:284994000 GMT j9mm.54 - Mark start\n3STHSTTYPE 14:29:29:284941000 GMT j9mm.474 - GlobalGC start: globalcount=8\n3STHSTTYPE 14:29:29:284916000 GMT j9mm.135 - Exclusive access: exclusiveaccessms=0.016 meanexclusiveaccessms=0.016 threads=0 lastthreadtid=0xB6B61100 beatenbyotherthread=0\n3STHSTTYPE 14:29:29:284914000 GMT j9mm.469 - Allocation failure cycle start: newspace=2678784/3014656 oldspace=80601248/533856256 loa=5338112/5338112 requestedbytes=603979784\n3STHSTTYPE 14:29:29:284893000 GMT j9mm.470 - Allocation failure cycle end: newspace=2678784/3014656 oldspace=80601248/533856256 loa=5338112/5338112\n3STHSTTYPE 14:29:29:284858000 GMT j9mm.560 - LocalGC end: rememberedsetoverflow=0 causedrememberedsetoverflow=0 scancacheoverflow=0 failedflipcount=0 failedflipbytes=0 failedtenurecount=0 failedtenurebytes=0 flipcount=2 flipbytes=64 newspace=2678784/3014656 oldspace=80601248/533856256 loa=5338112/5338112 tenureage=0\n3STHSTTYPE 14:29:29:284140000 GMT j9mm.140 - Tilt ratio: 89\n3STHSTTYPE 14:29:29:283160000 GMT j9mm.64 - LocalGC start: globalcount=8 scavengecount=335 weakrefs=0 soft=0 phantom=0 finalizers=0\n3STHSTTYPE 14:29:29:283123000 GMT j9mm.135 - Exclusive access: exclusiveaccessms=0.016 meanexclusiveaccessms=0.016 threads=0 lastthreadtid=0xB6B61100 beatenbyotherthread=0\n3STHSTTYPE 14:29:29:283120000 GMT j9mm.469 - Allocation failure cycle start: newspace=753616/3014656 oldspace=80601248/533856256 loa=5338112/5338112 requestedbytes=603979784\n3STHSTTYPE 14:29:29:283117000 GMT j9mm.133 - Allocation failure start: newspace=753616/3014656 oldspace=80601248/533856256 loa=5338112/5338112 requestedbytes=603979784\n3STHSTTYPE 14:29:29:269762000 GMT j9mm.134 - Allocation failure end: newspace=2686928/3014656 oldspace=80601248/533856256 loa=5338112/5338112\n3STHSTTYPE 14:29:29:269751000 GMT j9mm.470 - Allocation failure cycle end: newspace=2686976/3014656 oldspace=80601248/533856256 loa=5338112/5338112\n3STHSTTYPE 14:29:29:269718000 GMT j9mm.560 - LocalGC end: rememberedsetoverflow=0 causedrememberedsetoverflow=0 scancacheoverflow=0 failedflipcount=0 failedflipbytes=0 failedtenurecount=0 failedtenurebytes=0 flipcount=0 flipbytes=0 newspace=2686976/3014656 oldspace=80601248/533856256 loa=5338112/5338112 tenureage=0\n3STHSTTYPE 14:29:29:268981000 GMT j9mm.140 - Tilt ratio: 89\n3STHSTTYPE 14:29:29:268007000 GMT j9mm.64 - LocalGC start: globalcount=8 scavengecount=334 weakrefs=0 soft=0 phantom=0 finalizers=0\n3STHSTTYPE 14:29:29:267969000 GMT j9mm.135 - Exclusive access: exclusiveaccessms=0.016 meanexclusiveaccessms=0.016 threads=0 lastthreadtid=0xB6B61100 beatenbyotherthread=0\n3STHSTTYPE 14:29:29:267966000 GMT j9mm.469 - Allocation failure cycle start: newspace=0/3014656 oldspace=80601248/533856256 loa=5338112/5338112 requestedbytes=48\n3STHSTTYPE 14:29:29:267963000 GMT j9mm.133 - Allocation failure start: newspace=0/3014656 oldspace=80601248/533856256 loa=5338112/5338112 requestedbytes=48\n3STHSTTYPE 14:29:29:249015000 GMT j9mm.134 - Allocation failure end: newspace=2686928/3014656 oldspace=80601248/533856256 loa=5338112/5338112\n3STHSTTYPE 14:29:29:249003000 GMT j9mm.470 - Allocation failure cycle end: newspace=2686976/3014656 oldspace=80601248/533856256 loa=5338112/5338112\n3STHSTTYPE 14:29:29:248971000 GMT j9mm.560 - LocalGC end: rememberedsetoverflow=0 causedrememberedsetoverflow=0 scancacheoverflow=0 failedflipcount=0 failedflipbytes=0 failedtenurecount=0 failedtenurebytes=0 flipcount=0 flipbytes=0 newspace=2686976/3014656 oldspace=80601248/533856256 loa=5338112/5338112 tenureage=0\n\n\n\n\n\nAlthough the Java code that was used in this scenario deliberately triggered an \nOutOfMemoryError\n in a pronounced way, similar allocation issues can and do occur when dealing with large data sets such as XML files.\n\n\nThe next step in diagnosing the problem is to open the system dump that gets generated automatically when an \nOutOfMemoryError\n occurs. Open\nthe dump with the \nEclipse Memory Analyzer tool (MAT)\n and search for the \nStringBuffer\n object, which should provide further clues about what went wrong. A common example is seeing the same \nString\n duplicated over and over again, which might indicate that code is stuck in a loop.\n\n\n \nNote:\n If you want to use MAT to analyze your system dump, you must install the Diagnostic Tool Framework for Java (DTFJ) plugin in the Eclipse IDE. Select the following menu items:\n\n\nHelp > Install New Software > Work with \"IBM Diagnostic Tool Framework for Java\" > \n\n\n\n\n\nIf, unlike the previous scenario, you receive an \nOutOfMemoryError\n and the MEMINFO section shows that there is very little space left\non the Java heap, the current thread information is typically not important. The current thread is simply the thread that happened to be current when the space ran out. In this situation you might want to increase your Java heap size. For help with this task, see \nHow to do heap sizing\n.\n\n\nNative OutOfMemoryError\n\n\nIn this scenario, the VM runs out of native memory. Native memory is memory that is used by the VM for storing all virtualized resources and data that it needs for VM operations. Native memory that is available to the VM process is limited by the operating system. The native memory available to the VM might also be subject to additional limits imposed by the operating system, for example Unix \nulimits\n. \n\n\nWhen a \nNativeOutOfMemoryError\n occurs, a Java dump is generated by default. The first section of the file (TITLE) tells you that a systhrow event triggered the Java dump as a result of an OOM (\njava/lang/OutOfMemoryError\n) for native memory.\n\n\n0SECTION TITLE subcomponent dump routine\nNULL ===============================\n1TICHARSET UTF-8\n1TISIGINFO Dump Event \"systhrow\" (00040000) Detail \"java/lang/OutOfMemoryError\" \"native memory exhausted\" received\n1TIDATETIME Date: 2018/09/14 at 15:49:55:887\n1TINANOTIME System nanotime: 3636862054495675\n1TIFILENAME Javacore filename: /home/cheesemp/test/javacore.20180914.154814.19708.0003.txt\n1TIREQFLAGS Request Flags: 0x81 (exclusive+preempt)\n1TIPREPSTATE Prep State: 0x104 (exclusive_vm_access+trace_disabled)\n\n\n\n\n\nSometimes, the current thread is responsible for causing the \nNativeOutOfMemoryError\n. Information about the current thread can be found in the THREADS section, as shown in the following output.\n\n\n0SECTION THREADS subcomponent dump routine\nNULL =================================\nNULL\n1XMPOOLINFO JVM Thread pool info:\n2XMPOOLTOTAL Current total number of pooled threads: 16\n2XMPOOLLIVE Current total number of live threads: 16\n2XMPOOLDAEMON Current total number of live daemon threads: 15\nNULL \n1XMCURTHDINFO Current thread\n3XMTHREADINFO \"main\" J9VMThread:0xB6C60C00, omrthread_t:0xB6C049D8, java/lang/Thread:0xB55E3C10, state:R, prio=5\n3XMJAVALTHREAD (java/lang/Thread getId:0x1, isDaemon:false)\n3XMTHREADINFO1 (native thread ID:0x4CFD, native priority:0x5, native policy:UNKNOWN, vmstate:R, vm thread flags:0x00001020)\n3XMTHREADINFO2 (native stack address range from:0xB6D4E000, to:0xB754F000, size:0x801000)\n3XMCPUTIME CPU usage total: 3.654896026 secs, current category=\"Application\"\n3XMHEAPALLOC Heap bytes allocated since last GC cycle=0 (0x0)\n3XMTHREADINFO3 Java callstack:\n4XESTACKTRACE at sun/misc/Unsafe.allocateDBBMemory(Native Method)\n4XESTACKTRACE at java/nio/DirectByteBuffer.<init>(DirectByteBuffer.java:127(Compiled Code))\n4XESTACKTRACE at java/nio/ByteBuffer.allocateDirect(ByteBuffer.java:311)\n4XESTACKTRACE at NativeHeapBreaker.main(NativeHeapBreaker.java:9)\n3XMTHREADINFO3 Native callstack:\n4XENATIVESTACK (0xB6A9F5B3 [libj9prt29.so+0x3b5b3])\n...\n4XENATIVESTACK (0xB582CC9C [libjclse7b_29.so+0x40c9c])\n4XENATIVESTACK Java_sun_misc_Unsafe_allocateDBBMemory+0x88 (0xB5827F6B [libjclse7b_29.so+0x3bf6b])\n4XENATIVESTACK (0x94A2084A [<unknown>+0x0])\n4XENATIVESTACK (0xB6B2538B [libj9vm29.so+0x6c38b])\n4XENATIVESTACK (0xB6B4074C [libj9vm29.so+0x8774c])\n4XENATIVESTACK (0xB6B7F299 [libj9vm29.so+0xc6299])\n4XENATIVESTACK (0xB6A82F3E [libj9prt29.so+0x1ef3e])\n4XENATIVESTACK (0xB6B7F32A [libj9vm29.so+0xc632a])\n4XENATIVESTACK (0xB6B4084C [libj9vm29.so+0x8784c])\n4XENATIVESTACK (0xB6B3CD0C [libj9vm29.so+0x83d0c])\n4XENATIVESTACK (0xB776F87D [libjli.so+0x787d])\n4XENATIVESTACK (0xB7784F72 [libpthread.so.0+0x6f72])\n4XENATIVESTACK clone+0x5e (0xB76A043E [libc.so.6+0xee43e])\n\n\n\n\n\nFor clarity in the \nNative callstack\n output, \n...\n indicates that some lines are removed.\n\n\nThe Java callstack shows the transition from Java to native code (\nsun/misc/Unsafe.allocateDBBMemory(Native Method)\n), indicating a request for Direct Byte Buffer (DBB) storage. DBB storage is backed by native memory, with the Java heap containing only a reference to the native heap buffer. In this scenario, DBB storage is the likely culprit for this \nNativeOutOfMemoryError\n.\n\n\nThe next step is to investigate the NATIVEMEMINFO section of the Java dump file, which reports the amount of memory used by the JRE process, broken down into component areas.\n\n\n0SECTION NATIVEMEMINFO subcomponent dump routine\nNULL =================================\n0MEMUSER\n1MEMUSER JRE: 3,166,386,688 bytes / 4388 allocations\n1MEMUSER |\n2MEMUSER +--VM: 563,176,824 bytes / 1518 allocations\n2MEMUSER | |\n3MEMUSER | +--Classes: 3,104,416 bytes / 120 allocations\n2MEMUSER | |\n3MEMUSER | +--Memory Manager (GC): 548,181,888 bytes / 398 allocations\n3MEMUSER | | |\n4MEMUSER | | +--Java Heap: 536,932,352 bytes / 1 allocation\n3MEMUSER | | |\n4MEMUSER | | +--Other: 11,249,536 bytes / 397 allocations\n2MEMUSER | |\n3MEMUSER | +--Threads: 10,817,120 bytes / 147 allocations\n3MEMUSER | | |\n4MEMUSER | | +--Java Stack: 115,584 bytes / 16 allocations\n3MEMUSER | | |\n4MEMUSER | | +--Native Stack: 10,616,832 bytes / 17 allocations\n3MEMUSER | | |\n4MEMUSER | | +--Other: 84,704 bytes / 114 allocations\n2MEMUSER | |\n3MEMUSER | +--Trace: 163,688 bytes / 268 allocations\n2MEMUSER | |\n3MEMUSER | +--JVMTI: 17,320 bytes / 13 allocations\n2MEMUSER | |\n3MEMUSER | +--JNI: 23,296 bytes / 55 allocations\n2MEMUSER | |\n3MEMUSER | +--Port Library: 8,576 bytes / 74 allocations\n2MEMUSER | |\n3MEMUSER | +--Other: 860,520 bytes / 443 allocations\n1MEMUSER |\n2MEMUSER +--JIT: 3,744,728 bytes / 122 allocations\n2MEMUSER | |\n3MEMUSER | +--JIT Code Cache: 2,097,152 bytes / 1 allocation\n2MEMUSER | |\n3MEMUSER | +--JIT Data Cache: 524,336 bytes / 1 allocation\n2MEMUSER | |\n3MEMUSER | +--Other: 1,123,240 bytes / 120 allocations\n1MEMUSER |\n2MEMUSER +--Class Libraries: 2,599,463,024 bytes / 2732 allocations\n2MEMUSER | |\n3MEMUSER | +--Harmony Class Libraries: 1,024 bytes / 1 allocation\n2MEMUSER | |\n3MEMUSER | +--VM Class Libraries: 2,599,462,000 bytes / 2731 allocations\n3MEMUSER | | |\n4MEMUSER | | +--sun.misc.Unsafe: 2,598,510,480 bytes / 2484 allocations\n4MEMUSER | | | |\n5MEMUSER | | | +--Direct Byte Buffers: 2,598,510,480 bytes / 2484 allocations\n3MEMUSER | | |\n4MEMUSER | | +--Other: 951,520 bytes / 247 allocations\n1MEMUSER |\n2MEMUSER +--Unknown: 2,112 bytes / 16 allocations\nNULL \n\n\n\n\n\nIn the \nVM Class Libraries\n section, the amount of memory allocated for \nDirect Byte Buffers\n is shown. Because the \nNativeOutOfMemoryError\n was received on a small 32-bit system, a value of \n2,598,510,480 bytes\n indicates that the operating system has run out of memory. On a larger UNIX system, the process might have run out of memory because of the \nulimit\n setting. Increasing the value for \nulimit\n might avoid the error, which you can do temporarily by setting \nulimit -f unlimited\n in your current session.\n\n\nThe theoretical maximum size for a 32-bit process is the size of the 32-bit address space, which is 4 GB. On most operating systems a portion of the address space for each process is used by the kernel, such that the real limit for 32-bit processes is actually significantly less than 4GB. As a result, running out of native memory with a 32-bit VM is quite common.\n\n\nThe same 4 GB limit is also important if you are using a 64-bit VM with compressed references. In compressed references mode, all references to objects, classes, threads, and monitors are represented by 32-bit values for performance reasons, so these structures can be allocated only at 32-bit addresses. However, the operating system might place other allocations within this 4 GB of address space, and if this area becomes sufficiently full or fragmented, the VM throws a native \nNativeOutOfMemoryError\n error. These errors typically occur when the VM tries to create a new thread or load a class. The \nCurrent Thread History\n section should contain more information about what the thread was doing at the VM level when the \nNativeOutOfMemoryError\n error occurred.\n\n\nYou can usually avoid this type of problem by using the \n-Xmcrs\n option to reserve a contiguous area of memory within the lowest 4GB of memory at VM startup.\n\n\nAnother common cause of a \nNativeOutOfMemoryError\n is when an application loads duplicate classes. Classes are allocated outside of the Java heap in native memory. If the value reported for \nClasses\n in the NATIVEMEMINFO section is very large, duplicate classes might be the cause of your problem. The \nEclipse Memory Analyzer Tool (MAT)\n can tell you if you have duplicate classes by using the \nClass Loader Explorer\n feature. Because a system dump is automatically generated as well as a Java dump in response to a \nNativeOutOfMemoryError\n, simply open the system dump in MAT to continue your diagnosis. \n\n\nDeadlock\n\n\nDeadlocks occur when two threads attempt to synchronize on an object and lock an instance of a class. When this happens, your application stops responding and hangs. Generating a Java dump file will quickly tell you whether you have a deadlock situation. Trigger the Java dump by sending a SIGQUIT signal (\nkill -3\n) to the VM.\n\n\nThe VM can detect the most common types of deadlock scenario involving Java monitors. If this type of deadlock is detected, information is provided in the LOCKS section. More complex deadlocks, including those that involve a mixture of native mutexes and Java monitors, are not detected.\n\n\nHere is the output from the code that was used to cause a common deadlock scenario:\n\n\nNULL\n \n\n1L\nKDEADLOCK\n \nDeadlock\n \ndetected\n \n!!!\n\n\nNULL\n \n---------------------\n\n\nNULL\n \n\n2L\nKDEADLOCKTHR\n \nThread\n \n\"Worker Thread 2\"\n \n(\n0x94501D00\n)\n\n\n3L\nKDEADLOCKWTR\n \nis\n \nwaiting\n \nfor\n:\n\n\n4L\nKDEADLOCKMON\n \nsys_mon_t\n:\n0x08C2B344\n \ninfl_mon_t\n:\n \n0x08C2B384\n:\n\n\n4L\nKDEADLOCKOBJ\n \njava\n/\nlang\n/\nObject\n@0xB5666698\n\n\n3L\nKDEADLOCKOWN\n \nwhich\n \nis\n \nowned\n \nby\n:\n\n\n2L\nKDEADLOCKTHR\n \nThread\n \n\"Worker Thread 3\"\n \n(\n0x94507500\n)\n\n\n3L\nKDEADLOCKWTR\n \nwhich\n \nis\n \nwaiting\n \nfor\n:\n\n\n4L\nKDEADLOCKMON\n \nsys_mon_t\n:\n0x08C2B3A0\n \ninfl_mon_t\n:\n \n0x08C2B3E0\n:\n\n\n4L\nKDEADLOCKOBJ\n \njava\n/\nlang\n/\nObject\n@0xB5666678\n\n\n3L\nKDEADLOCKOWN\n \nwhich\n \nis\n \nowned\n \nby\n:\n\n\n2L\nKDEADLOCKTHR\n \nThread\n \n\"Worker Thread 1\"\n \n(\n0x92A3EC00\n)\n\n\n3L\nKDEADLOCKWTR\n \nwhich\n \nis\n \nwaiting\n \nfor\n:\n\n\n4L\nKDEADLOCKMON\n \nsys_mon_t\n:\n0x08C2B2E8\n \ninfl_mon_t\n:\n \n0x08C2B328\n:\n\n\n4L\nKDEADLOCKOBJ\n \njava\n/\nlang\n/\nObject\n@0xB5666688\n\n\n3L\nKDEADLOCKOWN\n \nwhich\n \nis\n \nowned\n \nby\n:\n\n\n2L\nKDEADLOCKTHR\n \nThread\n \n\"Worker Thread 2\"\n \n(\n0x94501D00\n)\n\n\n\n\n\n\nThis output tells you that \nWorker Thread 2\n is waiting for \nWorker Thread 3\n, which is waiting for \nWorker Thread 1\n. Because \nWorker Thread 1\n is also waiting for \nWorker Thread 2\n, there is a deadlock. The next place to look is the output for Java and native stacks, in the THREADS section. By looking at the stack for each of these worker threads you can trace the problem back to specific lines in your application code.\n\n\nIn this example, you can see from the following output that for all worker threads, the stack traces (\n4XESTACKTRACE\n/\n5XESTACKTRACE\n) indicate a problem in line 35 of the application \nDeadLockTest.java\n:\n\n\n3\nXMTHREADINFO\n \n\"Worker Thread 1\"\n \nJ9VMThread\n:\n0x92A3EC00\n,\n \nomrthread_t\n:\n0x92A3C2B0\n,\n \njava\n/\nlang\n/\nThread\n:\n0xB5666778\n,\n \nstate\n:\nB\n,\n \nprio\n=\n5\n\n\n3\nXMJAVALTHREAD\n \n(\njava\n/\nlang\n/\nThread\n \ngetId\n:\n0x13\n,\n \nisDaemon\n:\nfalse\n)\n\n\n3\nXMTHREADINFO1\n \n(\nnative\n \nthread\n \nID\n:\n0x52CF\n,\n \nnative\n \npriority\n:\n0x5\n,\n \nnative\n \npolicy\n:\nUNKNOWN\n,\n \nvmstate\n:\nB\n,\n \nvm\n \nthread\n \nflags\n:\n0x00000201\n)\n\n\n3\nXMTHREADINFO2\n \n(\nnative\n \nstack\n \naddress\n \nrange\n \nfrom\n:\n0x9297E000\n,\n \nto\n:\n0x929BF000\n,\n \nsize\n:\n0x41000\n)\n\n\n3\nXMCPUTIME\n \nCPU\n \nusage\n \ntotal\n:\n \n0.004365543\n \nsecs\n,\n \ncurrent\n \ncategory\n=\n\"Application\"\n\n\n3\nXMTHREADBLOCK\n \nBlocked\n \non\n:\n \njava\n/\nlang\n/\nObject\n@0xB5666688\n \nOwned\n \nby\n:\n \n\"Worker Thread 2\"\n \n(\nJ9VMThread\n:\n0x94501D00\n,\n \njava\n/\nlang\n/\nThread\n:\n0xB56668D0\n)\n\n\n3\nXMHEAPALLOC\n \nHeap\n \nbytes\n \nallocated\n \nsince\n \nlast\n \nGC\n \ncycle\n=\n0\n \n(\n0x0\n)\n\n\n3\nXMTHREADINFO3\n \nJava\n \ncallstack\n:\n\n\n4\nXESTACKTRACE\n \nat\n \nWorkerThread\n.\nrun\n(\nDeadLockTest\n.\njava\n:\n35\n)\n\n\n5\nXESTACKTRACE\n \n(\nentered\n \nlock\n:\n \njava\n/\nlang\n/\nObject\n@0xB5666678\n,\n \nentry\n \ncount\n:\n \n1\n)\n\n\n...\n\n\n3\nXMTHREADINFO\n \n\"Worker Thread 2\"\n \nJ9VMThread\n:\n0x94501D00\n,\n \nomrthread_t\n:\n0x92A3C8F0\n,\n \njava\n/\nlang\n/\nThread\n:\n0xB56668D0\n,\n \nstate\n:\nB\n,\n \nprio\n=\n5\n\n\n3\nXMJAVALTHREAD\n \n(\njava\n/\nlang\n/\nThread\n \ngetId\n:\n0x14\n,\n \nisDaemon\n:\nfalse\n)\n\n\n3\nXMTHREADINFO1\n \n(\nnative\n \nthread\n \nID\n:\n0x52D0\n,\n \nnative\n \npriority\n:\n0x5\n,\n \nnative\n \npolicy\n:\nUNKNOWN\n,\n \nvmstate\n:\nB\n,\n \nvm\n \nthread\n \nflags\n:\n0x00000201\n)\n\n\n3\nXMTHREADINFO2\n \n(\nnative\n \nstack\n \naddress\n \nrange\n \nfrom\n:\n0x946BF000\n,\n \nto\n:\n0x94700000\n,\n \nsize\n:\n0x41000\n)\n\n\n3\nXMCPUTIME\n \nCPU\n \nusage\n \ntotal\n:\n \n0.004555580\n \nsecs\n,\n \ncurrent\n \ncategory\n=\n\"Application\"\n\n\n3\nXMTHREADBLOCK\n \nBlocked\n \non\n:\n \njava\n/\nlang\n/\nObject\n@0xB5666698\n \nOwned\n \nby\n:\n \n\"Worker Thread 3\"\n \n(\nJ9VMThread\n:\n0x94507500\n,\n \njava\n/\nlang\n/\nThread\n:\n0xB5666A18\n)\n\n\n3\nXMHEAPALLOC\n \nHeap\n \nbytes\n \nallocated\n \nsince\n \nlast\n \nGC\n \ncycle\n=\n0\n \n(\n0x0\n)\n\n\n3\nXMTHREADINFO3\n \nJava\n \ncallstack\n:\n\n\n4\nXESTACKTRACE\n \nat\n \nWorkerThread\n.\nrun\n(\nDeadLockTest\n.\njava\n:\n35\n)\n\n\n5\nXESTACKTRACE\n \n(\nentered\n \nlock\n:\n \njava\n/\nlang\n/\nObject\n@0xB5666688\n,\n \nentry\n \ncount\n:\n \n1\n)\n\n\n...\n\n\n3\nXMTHREADINFO\n \n\"Worker Thread 3\"\n \nJ9VMThread\n:\n0x94507500\n,\n \nomrthread_t\n:\n0x92A3CC10\n,\n \njava\n/\nlang\n/\nThread\n:\n0xB5666A18\n,\n \nstate\n:\nB\n,\n \nprio\n=\n5\n\n\n3\nXMJAVALTHREAD\n \n(\njava\n/\nlang\n/\nThread\n \ngetId\n:\n0x15\n,\n \nisDaemon\n:\nfalse\n)\n\n\n3\nXMTHREADINFO1\n \n(\nnative\n \nthread\n \nID\n:\n0x52D1\n,\n \nnative\n \npriority\n:\n0x5\n,\n \nnative\n \npolicy\n:\nUNKNOWN\n,\n \nvmstate\n:\nB\n,\n \nvm\n \nthread\n \nflags\n:\n0x00000201\n)\n\n\n3\nXMTHREADINFO2\n \n(\nnative\n \nstack\n \naddress\n \nrange\n \nfrom\n:\n0x9467E000\n,\n \nto\n:\n0x946BF000\n,\n \nsize\n:\n0x41000\n)\n\n\n3\nXMCPUTIME\n \nCPU\n \nusage\n \ntotal\n:\n \n0.003657010\n \nsecs\n,\n \ncurrent\n \ncategory\n=\n\"Application\"\n\n\n3\nXMTHREADBLOCK\n \nBlocked\n \non\n:\n \njava\n/\nlang\n/\nObject\n@0xB5666678\n \nOwned\n \nby\n:\n \n\"Worker Thread 1\"\n \n(\nJ9VMThread\n:\n0x92A3EC00\n,\n \njava\n/\nlang\n/\nThread\n:\n0xB5666778\n)\n\n\n3\nXMHEAPALLOC\n \nHeap\n \nbytes\n \nallocated\n \nsince\n \nlast\n \nGC\n \ncycle\n=\n0\n \n(\n0x0\n)\n\n\n3\nXMTHREADINFO3\n \nJava\n \ncallstack\n:\n\n\n4\nXESTACKTRACE\n \nat\n \nWorkerThread\n.\nrun\n(\nDeadLockTest\n.\njava\n:\n35\n)\n\n\n5\nXESTACKTRACE\n \n(\nentered\n \nlock\n:\n \njava\n/\nlang\n/\nObject\n@0xB5666698\n,\n \nentry\n \ncount\n:\n \n1\n)",
+ "text": "Java dump\n\n\nJava dumps, sometimes referred to as \nJava cores\n, are produced when the VM ends unexpectedly because of an operating system signal, \nOutOfMemoryError\n, or a user-initiated keystroke combination. You can also generate a Java dump by calling the Dump API programmatically from your application or specifying the \n-Xdump:java\n option on the command line.\n\n\nIf your Java application crashes or hangs, Java dumps can provide useful information to help you diagnose the root cause.\n\n\n\n\nIf your application crashes, Javadumps are generated automatically for the following types of failure:\n\n\nthe VM receives an unexpected signal or an assertion failure\n\n\nthe VM runs out of memory\n\n\n\n\n\n\n\n\nIf your application hangs, you can trigger the generation of a Java dump by sending a SIGQUIT signal (\nkill -3\n) to the VM.\n\n\n \nNote:\n On Windows, if you started the VM in a console window you can force the VM to produce a Java dump in response to a SIGBREAK signal\n(Ctrl-Break keyboard combination). If you didn't start in a console window there is no equivalent to a Linux \nkill\n command on Windows for sending signals.\nThe only option here is to trigger a full system dump by finding the VM process in the \nProcesses\n tab of the Windows Task Manager and clicking\n\nCreate dump file\n.\n\n\n\n\n\n\nTo help you understand how a Java dump can help you with problem diagnosis, this topic includes a few scenarios to help you interpret the data:\n\n\n\n\nA crash caused by a general protection fault (gpf)\n\n\nA Java heap OutOfMemoryError (OOM)\n\n\nA native OutOfMemoryError (OOM)\n\n\nA deadlock situation\n\n\nA hang\n\n\n\n\nJava dump contents\n\n\nJava dumps summarize the state of the VM when the event occurs, with most of the information relating to components of the VM.\nThe file is made up of a number of sections that provide different types of information.\n\n\nTITLE\n\n\nThe first section of the Java dump file provides information about the event that triggered the production of the dump.\nIn the following example you can see that a \nvmstop\n event triggered the dump at a specified date and time.\n\n\n0SECTION TITLE subcomponent dump routine\nNULL ===============================\n1TICHARSET UTF-8\n1TISIGINFO Dump Event \"vmstop\" (00000002) Detail \"#0000000000000000\" received\n1TIDATETIME Date: 2018/08/30 at 21:55:47:607\n1TINANOTIME System nanotime: 22012355276134\n1TIFILENAME Javacore filename: /home/doc-javacore/javacore.20180830.215547.30285.0001.txt\n1TIREQFLAGS Request Flags: 0x81 (exclusive+preempt)\n1TIPREPSTATE Prep State: 0x106 (vm_access+exclusive_vm_access+trace_disabled)\n\n\n\n\n\nGPINFO\n\n\nThe GPINFO section provides general information about the system that the VM is running on. The following example is taken\nfrom a Java dump that was generated on a Linux system.\n\n\nNULL ------------------------------------------------------------------------\n0SECTION GPINFO subcomponent dump routine\nNULL ================================\n2XHOSLEVEL OS Level : Linux 3.10.0-862.11.6.el7.x86_64\n2XHCPUS Processors -\n3XHCPUARCH Architecture : amd64\n3XHNUMCPUS How Many : 4\n3XHNUMASUP NUMA is either not supported or has been disabled by user\nNULL \n1XHERROR2 Register dump section only produced for SIGSEGV, SIGILL or SIGFPE.\nNULL \n\n\n\n\n\nThe content of this section can vary, depending on the cause of the dump. For example, if the dump was caused by a\ngeneral protection fault (gpf), the library in which the crash occurred is also recorded, together with a value shown\nas \nVM flags\n. This value can provide some clues about which component of the VM might have been involved. Look for the\nfollowing line in the output:\n\n\n1XHFLAGS VM flags:0000000000000000\n\n\n\n\n\nThe hexadecimal number recorded for \nVM flags\n ends in MSSSS, where M is the VM component and SSSS is component-specific code as shown in the following table:\n\n\n\n\n\n\n\n\nComponent\n\n\nCode value\n\n\n\n\n\n\n\n\n\n\nINTERPRETER\n\n\n0x10000\n\n\n\n\n\n\nGC\n\n\n0x20000\n\n\n\n\n\n\nGROW_STACK\n\n\n0x30000\n\n\n\n\n\n\nJNI\n\n\n0x40000\n\n\n\n\n\n\nJIT_CODEGEN\n\n\n0x50000\n\n\n\n\n\n\nBCVERIFY\n\n\n0x60000\n\n\n\n\n\n\nRTVERIFY\n\n\n0x70000\n\n\n\n\n\n\nSHAREDCLASSES\n\n\n0x80000\n\n\n\n\n\n\n\n\nA value of \n0000000000000000\n (0x00000) indicates that a crash occurred outside of the VM.\n\n\nENVINFO\n\n\nThis section contains useful information about the environment in which the crash took place, including the following data:\n\n\n\n\nJava version (\n1CIJAVAVERSION\n)\n\n\nOpenJ9 VM and subcomponent version information (\n1CIVMVERSION\n, \n1CIJ9VMVERSION\n, \n1CIJITVERSION\n, \n1CIOMRVERSION\n, \n1CIJCLVERSION\n)\n\n\nVM start time (\n1CISTARTTIME\n) and process information (\n1CIPROCESSID\n)\n\n\nJava home (\n1CIJAVAHOMEDIR\n) and DLL (\n1CIJAVADLLDIR\n) directories\n\n\nUser arguments passed on the command line (\n1CIUSERARG\n)\n\n\nUser limits imposed by the system (\n1CIUSERLIMITS\n)\n\n\nEnvironment variables in place (\n1CIENVVARS\n)\n\n\nSystem information (\n1CISYSINFO\n)\n\n\nCPU information (\n1CICPUINFO\n)\n\n\n\n\n\nFor clarity, the following example shows a shortened version of this section, where \n...\n indicates that lines are removed:\n\n\nNULL\n \n------------------------------------------------------------------------\n\n\n0\nSECTION\n \nENVINFO\n \nsubcomponent\n \ndump\n \nroutine\n\n\nNULL\n \n=================================\n\n\n1\nCIJAVAVERSION\n \nJRE\n \n9\n \nLinux\n \namd64\n-\n64\n \n(\nbuild\n \n9.0.4\n-\ninternal\n+\n0\n-\nadhoc\n..\nopenj9\n-\nopenjdk\n-\njdk9\n)\n\n\n1\nCIVMVERSION\n \n20180830\n_000000\n\n\n1\nCIJ9VMVERSION\n \n8e7\nc6ec\n\n\n1\nCIJITVERSION\n \n8e7\nc6ec\n\n\n1\nCIOMRVERSION\n \n553811\nb_CMPRSS\n\n\n1\nCIJCLVERSION\n \nec1d223\n \nbased\n \non\n \njdk\n-\n9.0.4\n+\n12\n\n\n1\nCIJITMODES\n \nJIT\n \nenabled\n,\n \nAOT\n \nenabled\n,\n \nFSD\n \ndisabled\n,\n \nHCR\n \nenabled\n\n\n1\nCIRUNNINGAS\n \nRunning\n \nas\n \na\n \nstandalone\n \nJVM\n\n\n1\nCIVMIDLESTATE\n \nVM\n \nIdle\n \nState\n:\n \nACTIVE\n\n\n1\nCISTARTTIME\n \nJVM\n \nstart\n \ntime\n:\n \n2018\n/\n08\n/\n30\n \nat\n \n21\n:\n55\n:\n47\n:\n387\n\n\n1\nCISTARTNANO\n \nJVM\n \nstart\n \nnanotime\n:\n \n22012135233549\n\n\n1\nCIPROCESSID\n \nProcess\n \nID\n:\n \n30285\n \n(\n0x764D\n)\n\n\n1\nCICMDLINE\n \n[\nnot\n \navailable\n]\n\n\n1\nCIJAVAHOMEDIR\n \nJava\n \nHome\n \nDir\n:\n \n/\nhome\n/\nme\n/\nopenj9\n-\nopenjdk\n-\njdk9\n/\nbuild\n/\nlinux\n-\nx86_64\n-\nnormal\n-\nserver\n-\nrelease\n/\nimages\n/\njdk\n\n\n1\nCIJAVADLLDIR\n \nJava\n \nDLL\n \nDir\n:\n \n/\nhome\n/\nme\n/\nopenj9\n-\nopenjdk\n-\njdk9\n/\nbuild\n/\nlinux\n-\nx86_64\n-\nnormal\n-\nserver\n-\nrelease\n/\nimages\n/\njdk\n/\nbin\n\n\n1\nCISYSCP\n \nSys\n \nClasspath\n:\n \n\n1\nCIUSERARGS\n \nUserArgs\n:\n\n\n2\nCIUSERARG\n \n-\nXoptionsfile\n=/\nhome\n/\nme\n/\nopenj9\n-\nopenjdk\n-\njdk9\n/\nbuild\n/\nlinux\n-\nx86_64\n-\nnormal\n-\nserver\n-\nrelease\n/\nimages\n/\njdk\n/\nlib\n/\noptions\n.\ndefault\n\n\n...\n\n\nNULL\n\n\n1\nCIUSERLIMITS\n \nUser\n \nLimits\n \n(\nin\n \nbytes\n \nexcept\n \nfor\n \nNOFILE\n \nand\n \nNPROC\n)\n\n\nNULL\n \n------------------------------------------------------------------------\n\n\nNULL\n \ntype\n \nsoft\n \nlimit\n \nhard\n \nlimit\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_AS\n \nunlimited\n \nunlimited\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_CORE\n \n0\n \nunlimited\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_CPU\n \nunlimited\n \nunlimited\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_DATA\n \nunlimited\n \nunlimited\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_FSIZE\n \nunlimited\n \nunlimited\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_LOCKS\n \nunlimited\n \nunlimited\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_MEMLOCK\n \n65536\n \n65536\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_NOFILE\n \n4096\n \n4096\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_NPROC\n \n4096\n \n30592\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_RSS\n \nunlimited\n \nunlimited\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_STACK\n \n8388608\n \nunlimited\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_MSGQUEUE\n \n819200\n \n819200\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_NICE\n \n0\n \n0\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_RTPRIO\n \n0\n \n0\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_SIGPENDING\n \n30592\n \n30592\n\n\nNULL\n\n\n1\nCIENVVARS\n \nEnvironment\n \nVariables\n\n\nNULL\n \n------------------------------------------------------------------------\n\n\n2\nCIENVVAR\n \nXDG_VTNR\n=\n1\n\n\n2\nCIENVVAR\n \nSSH_AGENT_PID\n=\n2653\n\n\n...\n\n\nNULL\n \n\n1\nCISYSINFO\n \nSystem\n \nInformation\n\n\nNULL\n \n------------------------------------------------------------------------\n\n\n2\nCISYSINFO\n \n/\nproc\n/\nsys\n/\nkernel\n/\ncore_pattern\n \n=\n \ncore\n\n\n2\nCISYSINFO\n \n/\nproc\n/\nsys\n/\nkernel\n/\ncore_uses_pid\n \n=\n \n1\n\n\nNULL\n \n\n1\nCICPUINFO\n \nCPU\n \nInformation\n\n\nNULL\n \n------------------------------------------------------------------------\n\n\n2\nCIPHYSCPU\n \nPhysical\n \nCPUs\n:\n \n4\n\n\n2\nCIONLNCPU\n \nOnline\n \nCPUs\n:\n \n4\n\n\n2\nCIBOUNDCPU\n \nBound\n \nCPUs\n:\n \n4\n\n\n2\nCIACTIVECPU\n \nActive\n \nCPUs\n:\n \n0\n\n\n2\nCITARGETCPU\n \nTarget\n \nCPUs\n:\n \n4\n\n\n\n\n\n\n\n\n\nNATIVEMEMINFO\n\n\nThis section records information about native memory that is requested by using library functions such as \nmalloc()\n and \nmmap()\n.\nValues are provided as a breakdown, per component, indicating the total number of bytes allocated and the number of native memory allocations.\nIn the following example, 4,682,840 bytes of native memory are allocated (but not yet freed) to VM Classes, which corresponds to 141 allocations.\n\n\nNULL ------------------------------------------------------------------------\n0SECTION NATIVEMEMINFO subcomponent dump routine\nNULL =================================\n0MEMUSER\n1MEMUSER JRE: 2,569,088,312 bytes / 4653 allocations\n1MEMUSER |\n2MEMUSER +--VM: 2,280,088,336 bytes / 2423 allocations\n2MEMUSER | |\n3MEMUSER | +--Classes: 4,682,840 bytes / 141 allocations\n2MEMUSER | |\n3MEMUSER | +--Memory Manager (GC): 2,054,966,784 bytes / 433 allocations\n3MEMUSER | | |\n4MEMUSER | | +--Java Heap: 2,014,113,792 bytes / 1 allocation\n3MEMUSER | | |\n4MEMUSER | | +--Other: 40,852,992 bytes / 432 allocations\n2MEMUSER | |\n3MEMUSER | +--Threads: 10,970,016 bytes / 156 allocations\n3MEMUSER | | |\n4MEMUSER | | +--Java Stack: 197,760 bytes / 16 allocations\n3MEMUSER | | |\n4MEMUSER | | +--Native Stack: 10,616,832 bytes / 17 allocations\n3MEMUSER | | |\n4MEMUSER | | +--Other: 155,424 bytes / 123 allocations\n2MEMUSER | |\n3MEMUSER | +--Trace: 180,056 bytes / 263 allocations\n2MEMUSER | |\n3MEMUSER | +--JVMTI: 17,776 bytes / 13 allocations\n2MEMUSER | |\n3MEMUSER | +--JNI: 36,184 bytes / 52 allocations\n2MEMUSER | |\n3MEMUSER | +--Port Library: 208,179,632 bytes / 72 allocations\n3MEMUSER | | |\n4MEMUSER | | +--Unused <32bit allocation regions: 208,168,752 bytes / 1 allocation\n3MEMUSER | | |\n4MEMUSER | | +--Other: 10,880 bytes / 71 allocations\n2MEMUSER | |\n3MEMUSER | +--Other: 1,055,048 bytes / 1293 allocations\n1MEMUSER |\n2MEMUSER +--JIT: 288,472,816 bytes / 140 allocations\n2MEMUSER | |\n3MEMUSER | +--JIT Code Cache: 268,435,456 bytes / 1 allocation\n2MEMUSER | |\n3MEMUSER | +--JIT Data Cache: 2,097,216 bytes / 1 allocation\n2MEMUSER | |\n3MEMUSER | +--Other: 17,940,144 bytes / 138 allocations\n1MEMUSER |\n2MEMUSER +--Class Libraries: 13,432 bytes / 25 allocations\n2MEMUSER | |\n3MEMUSER | +--VM Class Libraries: 13,432 bytes / 25 allocations\n3MEMUSER | | |\n4MEMUSER | | +--sun.misc.Unsafe: 3,184 bytes / 13 allocations\n4MEMUSER | | | |\n5MEMUSER | | | +--Direct Byte Buffers: 1,056 bytes / 12 allocations\n4MEMUSER | | | |\n5MEMUSER | | | +--Other: 2,128 bytes / 1 allocation\n3MEMUSER | | |\n4MEMUSER | | +--Other: 10,248 bytes / 12 allocations\n1MEMUSER |\n2MEMUSER +--Unknown: 513,728 bytes / 2065 allocations\nNULL \n\n\n\n\n\nThis section does not record memory that is allocated by application or JNI code and is typically a little less than the\nvalue recorded by operating system tools.\n\n\nMEMINFO\n\n\nThis section relates to memory management, providing a breakdown of memory usage in the VM for the object heap,\ninternal memory, memory used for classes, the JIT code cache, and JIT data cache in decimal and hexadecimal format.\nYou can also find out which garbage collection policy is in use when the dump is produced. \n\n\nThe object memory area (\n1STHEAPTYPE\n) records each memory region in use, its start and end address, and region size.\nFurther information is recorded about the memory segments used for internal memory, class memory, the JIT code cache and JIT data cache (\n1STSEGMENT\n).\nThis information includes the address of the segment control data structure, the start and end address of the native memory segment, as well as\nthe segment size.\n\n\nFor clarity, the following example shows a shortened version of this section, where \n...\n indicates that lines are removed:\n\n\nNULL ------------------------------------------------------------------------\n0SECTION MEMINFO subcomponent dump routine\nNULL =================================\nNULL \n1STHEAPTYPE Object Memory\nNULL id start end size space/region\n1STHEAPSPACE 0x00007FF4F00744A0 -- -- -- Generational\n1STHEAPREGION 0x00007FF4F0074CE0 0x0000000087F40000 0x0000000088540000 0x0000000000600000 Generational/Tenured Region\n1STHEAPREGION 0x00007FF4F0074930 0x00000000FFE00000 0x00000000FFF00000 0x0000000000100000 Generational/Nursery Region\n1STHEAPREGION 0x00007FF4F0074580 0x00000000FFF00000 0x0000000100000000 0x0000000000100000 Generational/Nursery Region\nNULL\n1STHEAPTOTAL Total memory: 8388608 (0x0000000000800000)\n1STHEAPINUSE Total memory in use: 2030408 (0x00000000001EFB48)\n1STHEAPFREE Total memory free: 6358200 (0x00000000006104B8)\nNULL\n1STSEGTYPE Internal Memory\nNULL segment start alloc end type size\n1STSEGMENT 0x00007FF4F004CBC8 0x00007FF4CD33C000 0x00007FF4CD33C000 0x00007FF4CE33C000 0x01000440 0x0000000001000000\n1STSEGMENT 0x00007FF4F004CB08 0x00007FF4DE43D030 0x00007FF4DE517770 0x00007FF4DE53D030 0x00800040 0x0000000000100000\nNULL\n1STSEGTOTAL Total memory: 17825792 (0x0000000001100000)\n1STSEGINUSE Total memory in use: 894784 (0x00000000000DA740)\n1STSEGFREE Total memory free: 16931008 (0x00000000010258C0)\nNULL \n1STSEGTYPE Class Memory\nNULL segment start alloc end type size\n1STSEGMENT 0x00007FF4F03B5638 0x0000000001053D98 0x000000000105BD98 0x000000000105BD98 0x00010040 0x0000000000008000\n1STSEGMENT 0x00007FF4F03B5578 0x0000000001048188 0x0000000001050188 0x0000000001050188 0x00010040 0x0000000000008000\n...\nNULL\n1STSEGTOTAL Total memory: 3512520 (0x00000000003598C8)\n1STSEGINUSE Total memory in use: 3433944 (0x00000000003465D8)\n1STSEGFREE Total memory free: 78576 (0x00000000000132F0)\nNULL \n1STSEGTYPE JIT Code Cache\nNULL segment start alloc end type size\n1STSEGMENT 0x00007FF4F00961F8 0x00007FF4CE43D000 0x00007FF4CE445790 0x00007FF4DE43D000 0x00000068 0x0000000010000000\nNULL\n1STSEGTOTAL Total memory: 268435456 (0x0000000010000000)\n1STSEGINUSE Total memory in use: 34704 (0x0000000000008790)\n1STSEGFREE Total memory free: 268400752 (0x000000000FFF7870)\n1STSEGLIMIT Allocation limit: 268435456 (0x0000000010000000)\nNULL \n1STSEGTYPE JIT Data Cache\nNULL segment start alloc end type size\n1STSEGMENT 0x00007FF4F0096668 0x00007FF4CC553030 0x00007FF4CC753030 0x00007FF4CC753030 0x00000048 0x0000000000200000\nNULL\n1STSEGTOTAL Total memory: 2097152 (0x0000000000200000)\n1STSEGINUSE Total memory in use: 2097152 (0x0000000000200000)\n1STSEGFREE Total memory free: 0 (0x0000000000000000)\n1STSEGLIMIT Allocation limit: 402653184 (0x0000000018000000)\nNULL \n1STGCHTYPE GC History \nNULL \n\n\n\n\n\nIn the example, the GC History (\n1STGCHTYPE\n) section is blank. This section is populated if a garbage collection cycle occurred in\na VM that is being diagnosed with the trace facility.\n\n\nLOCKS\n\n\nThis section of the Java dump provides information about locks, which protect shared resources from being accessed by more than one entity at a time. The information is essential in a deadlock situation, where two threads attempt to synchronize on an object and lock an instance of a class. Precise information is recorded about the threads that are causing the problem, which enables you to identify the root cause.\n\n\nThe following example shows a typical LOCKS section, where no deadlocks existed at the time the dump was triggered. For clarity, the following example shows a shortened version of this section, where \n...\n indicates that lines are removed:\n\n\nNULL\n \n------------------------------------------------------------------------\n\n\n0\nSECTION\n \nLOCKS\n \nsubcomponent\n \ndump\n \nroutine\n\n\nNULL\n \n===============================\n\n\nNULL\n \n\n1L\nKPOOLINFO\n \nMonitor\n \npool\n \ninfo\n:\n\n\n2L\nKPOOLTOTAL\n \nCurrent\n \ntotal\n \nnumber\n \nof\n \nmonitors\n:\n \n3\n\n\nNULL\n \n\n1L\nKMONPOOLDUMP\n \nMonitor\n \nPool\n \nDump\n \n(\nflat\n \n&\n \ninflated\n \nobject\n-\nmonitors\n)\n:\n\n\n2L\nKMONINUSE\n \nsys_mon_t\n:\n0x00007FF4B0001D78\n \ninfl_mon_t\n:\n \n0x00007FF4B0001DF8\n:\n\n\n3L\nKMONOBJECT\n \njava\n/\nlang\n/\nref\n/\nReferenceQueue\n@0x00000000FFE26A10\n:\n \n<\nunowned\n>\n\n\n3L\nKNOTIFYQ\n \nWaiting\n \nto\n \nbe\n \nnotified\n:\n\n\n3L\nKWAITNOTIFY\n \n\"Common-Cleaner\"\n \n(\nJ9VMThread\n:\n0x0000000000FD0100\n)\n\n\nNULL\n \n\n1L\nKREGMONDUMP\n \nJVM\n \nSystem\n \nMonitor\n \nDump\n \n(\nregistered\n \nmonitors\n)\n:\n\n\n2L\nKREGMON\n \nThread\n \nglobal\n \nlock\n \n(\n0x00007FF4F0004FE8\n)\n:\n \n<\nunowned\n>\n\n\n2L\nKREGMON\n \n&\n(\nPPG_mem_mem32_subAllocHeapMem32\n.\nmonitor\n)\n \nlock\n \n(\n0x00007FF4F0005098\n)\n:\n \n<\nunowned\n>\n\n\n2L\nKREGMON\n \nNLS\n \nhash\n \ntable\n \nlock\n \n(\n0x00007FF4F0005148\n)\n:\n \n<\nunowned\n>\n\n\n...\n\n\nNULL\n \n\n\n\n\n\nTHREADS\n\n\nThe THREADS section of a Java dump file provides summary information about the VM thread pool and detailed information about Java threads, native threads, and stack traces. Understanding the content of this section can help you diagnose problems that are caused by blocked or waiting threads.\n\n\nA Java thread runs on a native thread. Several lines are recorded for each Java thread in the \nThread Details\n subsection, which include the following key pieces of information:\n\n\n\n\n3XMTHREADINFO\n: The thread name, address information for the VM thread structures and Java thread object, the thread state, and thread priority.\n\n\n3XMJAVALTHREAD\n: The Java thread ID and daemon status from the thread object.\n\n\n3XMTHREADINFO1\n: The native operating system thread ID, priority, scheduling policy, internal VM thread state, and VM thread flags.\n\n\n3XMTHREADINFO2\n: The native stack address range.\n\n\n3XMTHREADINFO3\n: Java callstack information (\n4XESTACKTRACE\n) or Native call stack information (\n4XENATIVESTACK\n).\n\n\n5XESTACKTRACE\n: This line indicates whether locks were taken by a specific method.\n\n\n\n\nJava thread priorities are mapped to operating system priority values. Thread states are shown in the following table:\n\n\n\n\n\n\n\n\nThread state value\n\n\nStatus\n\n\nDescription\n\n\n\n\n\n\n\n\n\n\nR\n\n\nRunnable\n\n\nThe thread is able to run\n\n\n\n\n\n\nCW\n\n\nCondition Wait\n\n\nThe thread is waiting\n\n\n\n\n\n\nS\n\n\nSuspended\n\n\nThe thread is suspended by another thread\n\n\n\n\n\n\nZ\n\n\nZombie\n\n\nThe thread is destroyed\n\n\n\n\n\n\nP\n\n\nParked\n\n\nThe thread is parked by \njava.util.concurrent\n\n\n\n\n\n\nB\n\n\nBlocked\n\n\nThe thread is waiting to obtain a lock\n\n\n\n\n\n\n\n\nFor threads that are parked (P), blocked (B), or waiting (CW), an additional line (\n3XMTHREADBLOCK\n) is included in the output that shows what the thread is parked on, blocked on, or waiting for.\n\n\nFor clarity, the following example shows a shortened version of a typical THREADS section, where \n...\n indicates that lines are removed:\n\n\nNULL\n \n------------------------------------------------------------------------\n\n\n0\nSECTION\n \nTHREADS\n \nsubcomponent\n \ndump\n \nroutine\n\n\nNULL\n \n=================================\n\n\nNULL\n\n\n1\nXMPOOLINFO\n \nJVM\n \nThread\n \npool\n \ninfo\n:\n\n\n2\nXMPOOLTOTAL\n \nCurrent\n \ntotal\n \nnumber\n \nof\n \npooled\n \nthreads\n:\n \n18\n\n\n2\nXMPOOLLIVE\n \nCurrent\n \ntotal\n \nnumber\n \nof\n \nlive\n \nthreads\n:\n \n16\n\n\n2\nXMPOOLDAEMON\n \nCurrent\n \ntotal\n \nnumber\n \nof\n \nlive\n \ndaemon\n \nthreads\n:\n \n15\n\n\nNULL\n \n\n1\nXMTHDINFO\n \nThread\n \nDetails\n\n\nNULL\n \n\n3\nXMTHREADINFO\n \n\"JIT Diagnostic Compilation Thread-7 Suspended\"\n \nJ9VMThread\n:\n0x0000000000EFC500\n,\n \nomrthread_t:\n0x00007FF4F00A77E8\n,\n \njava\n/\nlang\n/\nThread\n:\n0x00000000FFE97480\n,\n \nstate\n:\nR\n,\n \nprio\n=\n10\n\n\n3\nXMJAVALTHREAD\n \n(\njava\n/\nlang\n/\nThread\n \ngetId:\n0xA\n,\n \nisDaemon:true\n)\n\n\n3\nXMTHREADINFO1\n \n(\nnative\n \nthread\n \nID\n:\n0x7657\n,\n \nnative\n \npriority\n:\n0xB\n,\n \nnative\n \npolicy\n:\nUNKNOWN\n,\n \nvmstate\n:\nCW\n,\n \nvm\n \nthread\n \nflags\n:\n0x00000081\n)\n\n\n3\nXMTHREADINFO2\n \n(\nnative\n \nstack\n \naddress\n \nrange\n \nfrom\n:\n0x00007FF4CCC36000\n,\n \nto\n:\n0x00007FF4CCD36000\n,\n \nsize\n:\n0x100000\n)\n\n\n3\nXMCPUTIME\n \nCPU\n \nusage\n \ntotal\n:\n \n0.000037663\n \nsecs\n,\n \ncurrent\n \ncategory=\n\"JIT\"\n\n\n3\nXMHEAPALLOC\n \nHeap\n \nbytes\n \nallocated\n \nsince\n \nlast\n \nGC\n \ncycle\n=\n0\n \n(\n0x0\n)\n\n\n3\nXMTHREADINFO3\n \nNo\n \nJava\n \ncallstack\n \nassociated\n \nwith\n \nthis\n \nthread\n\n\n3\nXMTHREADINFO3\n \nNo\n \nnative\n \ncallstack\n \navailable\n \nfor\n \nthis\n \nthread\n\n\nNULL\n\n\n...\n\n\n3\nXMTHREADINFO\n \n\"Common-Cleaner\"\n \nJ9VMThread\n:\n0x0000000000FD0100\n,\n \nomrthread_t:\n0x00007FF4F022A520\n,\n \njava\n/\nlang\n/\nThread\n:\n0x00000000FFE26F40\n,\n \nstate\n:\nCW\n,\n \nprio\n=\n8\n\n\n3\nXMJAVALTHREAD\n \n(\njava\n/\nlang\n/\nThread\n \ngetId:\n0x2\n,\n \nisDaemon:true\n)\n\n\n3\nXMTHREADINFO1\n \n(\nnative\n \nthread\n \nID\n:\n0x765A\n,\n \nnative\n \npriority\n:\n0x8\n,\n \nnative\n \npolicy\n:\nUNKNOWN\n,\n \nvmstate\n:\nCW\n,\n \nvm\n \nthread\n \nflags\n:\n0x00080181\n)\n\n\n3\nXMTHREADINFO2\n \n(\nnative\n \nstack\n \naddress\n \nrange\n \nfrom\n:\n0x00007FF4CC0B8000\n,\n \nto\n:\n0x00007FF4CC0F8000\n,\n \nsize\n:\n0x40000\n)\n\n\n3\nXMCPUTIME\n \nCPU\n \nusage\n \ntotal\n:\n \n0.000150926\n \nsecs\n,\n \ncurrent\n \ncategory=\n\"Application\"\n\n\n3\nXMTHREADBLOCK\n \nWaiting\n \non\n:\n \njava\n/\nlang\n/\nref\n/\nReferenceQueue\n@\n0x00000000FFE26A10\n \nOwned\n \nby\n:\n \n<\nunowned\n>\n\n\n3\nXMHEAPALLOC\n \nHeap\n \nbytes\n \nallocated\n \nsince\n \nlast\n \nGC\n \ncycle\n=\n0\n \n(\n0x0\n)\n\n\n3\nXMTHREADINFO3\n \nJava\n \ncallstack\n:\n\n\n4\nXESTACKTRACE\n \nat\n \njava\n/\nlang\n/\nObject\n.\nwait\n(\nNative\n \nMethod\n)\n\n\n4\nXESTACKTRACE\n \nat\n \njava\n/\nlang\n/\nObject\n.\nwait\n(\nObject\n.\njava\n:\n221\n)\n\n\n4\nXESTACKTRACE\n \nat\n \njava\n/\nlang\n/\nref\n/\nReferenceQueue\n.\nremove\n(\nReferenceQueue\n.\njava\n:\n138\n)\n\n\n5\nXESTACKTRACE\n \n(\nentered\n \nlock\n:\n \njava\n/\nlang\n/\nref\n/\nReferenceQueue\n@\n0x00000000FFE26A10\n,\n \nentry\n \ncount\n:\n \n1\n)\n\n\n4\nXESTACKTRACE\n \nat\n \njdk\n/\ninternal\n/\nref\n/\nCleanerImpl\n.\nrun\n(\nCleanerImpl\n.\njava\n:\n148\n)\n\n\n4\nXESTACKTRACE\n \nat\n \njava\n/\nlang\n/\nThread\n.\nrun\n(\nThread\n.\njava\n:\n835\n)\n\n\n4\nXESTACKTRACE\n \nat\n \njdk\n/\ninternal\n/\nmisc\n/\nInnocuousThread\n.\nrun\n(\nInnocuousThread\n.\njava\n:\n122\n)\n\n\n3\nXMTHREADINFO3\n \nNo\n \nnative\n \ncallstack\n \navailable\n \nfor\n \nthis\n \nthread\n\n\nNULL\n\n\nNULL\n\n\n3\nXMTHREADINFO\n \n\"IProfiler\"\n \nJ9VMThread\n:\n0x0000000000F03D00\n,\n \nomrthread_t:\n0x00007FF4F00B06F8\n,\n \njava\n/\nlang\n/\nThread\n:\n0x00000000FFE97B60\n,\n \nstate\n:\nR\n,\n \nprio\n=\n5\n\n\n3\nXMJAVALTHREAD\n \n(\njava\n/\nlang\n/\nThread\n \ngetId:\n0xC\n,\n \nisDaemon:true\n)\n\n\n3\nXMTHREADINFO1\n \n(\nnative\n \nthread\n \nID\n:\n0x7659\n,\n \nnative\n \npriority\n:\n0x5\n,\n \nnative\n \npolicy\n:\nUNKNOWN\n,\n \nvmstate\n:\nCW\n,\n \nvm\n \nthread\n \nflags\n:\n0x00000081\n)\n\n\n3\nXMTHREADINFO2\n \n(\nnative\n \nstack\n \naddress\n \nrange\n \nfrom\n:\n0x00007FF4F8940000\n,\n \nto\n:\n0x00007FF4F8960000\n,\n \nsize\n:\n0x20000\n)\n\n\n3\nXMCPUTIME\n \nCPU\n \nusage\n \ntotal\n:\n \n0.004753103\n \nsecs\n,\n \ncurrent\n \ncategory=\n\"JIT\"\n\n\n3\nXMHEAPALLOC\n \nHeap\n \nbytes\n \nallocated\n \nsince\n \nlast\n \nGC\n \ncycle\n=\n0\n \n(\n0x0\n)\n\n\n3\nXMTHREADINFO3\n \nNo\n \nJava\n \ncallstack\n \nassociated\n \nwith\n \nthis\n \nthread\n\n\n3\nXMTHREADINFO3\n \nNo\n \nnative\n \ncallstack\n \navailable\n \nfor\n \nthis\n \nthread\n\n\nNULL\n\n\n...\n\n\n1\nXMWLKTHDERR\n \nThe\n \nfollowing\n \nwas\n \nreported\n \nwhile\n \ncollecting\n \nnative\n \nstacks\n:\n\n\n2\nXMWLKTHDERR\n \nunable\n \nto\n \ncount\n \nthreads\n(\n3\n,\n \n-\n2\n)\n\n\nNULL\n\n\n1\nXMTHDSUMMARY\n \nThreads\n \nCPU\n \nUsage\n \nSummary\n\n\nNULL\n \n=========================\n\n\nNULL\n\n\n1\nXMTHDCATINFO\n \nWarning\n:\n \nto\n \nget\n \nmore\n \naccurate\n \nCPU\n \ntimes\n \nfor\n \nthe\n \nGC\n,\n \nthe\n \noption\n \n-\nXX\n:-\nReduceCPUMonitorOverhead\n \ncan\n \nbe\n \nused\n.\n \nSee\n \nthe\n \nuser\n \nguide\n \nfor\n \nmore\n \ninformation\n.\n\n\nNULL\n\n\n1\nXMTHDCATEGORY\n \nAll\n \nJVM\n \nattached\n \nthreads\n:\n \n0.280083000\n \nsecs\n\n\n1\nXMTHDCATEGORY\n \n|\n\n\n2\nXMTHDCATEGORY\n \n+--\nSystem\n-\nJVM\n:\n \n0.270814000\n \nsecs\n\n\n2\nXMTHDCATEGORY\n \n|\n \n|\n\n\n3\nXMTHDCATEGORY\n \n|\n \n+--\nGC\n:\n \n0.000599000\n \nsecs\n\n\n2\nXMTHDCATEGORY\n \n|\n \n|\n\n\n3\nXMTHDCATEGORY\n \n|\n \n+--\nJIT\n:\n \n0.071904000\n \nsecs\n\n\n1\nXMTHDCATEGORY\n \n|\n\n\n2\nXMTHDCATEGORY\n \n+--\nApplication\n:\n \n0.009269000\n \nsecs\n\n\nNULL\n\n\n\n\n\n\nHOOKS\n\n\nThis section shows internal VM event callbacks, which are used for diagnosing performance problems in the VM. Multiple hook interfaces are listed, which include their individual hook events. The following example shows data for the \nJ9VMHookInterface\n, including the call site location (source file:line number), start time, and duration of the last callback and the longest callback.\n\n\nNULL ------------------------------------------------------------------------\n0SECTION HOOK subcomponent dump routine\nNULL ==============================\n1HKINTERFACE MM_OMRHookInterface\nNULL ------------------------------------------------------------------------\n1HKINTERFACE MM_PrivateHookInterface\nNULL ------------------------------------------------------------------------\n1HKINTERFACE MM_HookInterface\nNULL ------------------------------------------------------------------------\n1HKINTERFACE J9VMHookInterface\nNULL ------------------------------------------------------------------------\n2HKEVENTID 1\n3HKCALLCOUNT 18\n3HKLAST Last Callback\n4HKCALLSITE trcengine.c:392\n4HKSTARTTIME Start Time: 2018-08-30T21:55:47.601\n4HKDURATION DurationMs: 0\n3HKLONGST Longest Callback\n4HKCALLSITE trcengine.c:392\n4HKSTARTTIME Start Time: 2018-08-30T21:55:47.460\n4HKDURATION DurationMs: 1\nNULL\n...\n1HKINTERFACE J9VMZipCachePoolHookInterface\nNULL ------------------------------------------------------------------------\n1HKINTERFACE J9JITHookInterface\nNULL ------------------------------------------------------------------------\n2HKEVENTID 3\n3HKCALLCOUNT 65\n3HKLAST Last Callback\n4HKCALLSITE ../common/mgmtinit.c:191\n4HKSTARTTIME Start Time: 2018-08-30T21:55:47.601\n4HKDURATION DurationMs: 0\n3HKLONGST Longest Callback\n4HKCALLSITE ../common/mgmtinit.c:191\n4HKSTARTTIME Start Time: 2018-08-30T21:55:47.486\n4HKDURATION DurationMs: 0\n...\nNULL\n\n\n\n\n\nSHARED CLASSES\n\n\nIf the shared classes cache is enabled at run time, the information provided in a Java dump file describes settings that were used when creating the cache, together with summary information about the size and content of the cache.\n\n\nIn the following example, the shared classes cache was created with a Class Debug Area (\n-Xnolinenumbers=false\n). Byte code instrumentation (BCI) is enabled, which is the default, and VMs sharing the cache are allowed to store classpaths, which is also the default.\n\n\nThe \nCache Summary\n shows a cache size (\n2SCLTEXTCSZ\n) of 16776608 bytes, with a soft maximum size (\n2SCLTEXTSMB\n) also of 16776608 bytes, which leaves 12691668 bytes of free space (\n2SCLTEXTFRB\n). The size of the Class Debug Area (\n2SCLTEXTDAS\n) is 1331200 bytes and only 11% of this space is used.\n\n\nIn the \nCache Memory Status\n subsection, the line \n2SCLTEXTCMDT\n indicates the name and location of the shared cache and \ncr\n indicates that the cache is a 64-bit compressed references cache.\n\n\nNULL ------------------------------------------------------------------------\n0SECTION SHARED CLASSES subcomponent dump routine\nNULL ========================================\nNULL\n1SCLTEXTCRTW Cache Created With\nNULL ------------------\nNULL\n2SCLTEXTXNL -Xnolinenumbers = false\n2SCLTEXTBCI BCI Enabled = true\n2SCLTEXTBCI Restrict Classpaths = false\nNULL\n1SCLTEXTCSUM Cache Summary\nNULL ------------------\nNULL\n2SCLTEXTNLC No line number content = false\n2SCLTEXTLNC Line number content = true\nNULL\n2SCLTEXTRCS ROMClass start address = 0x00007F423061C000\n2SCLTEXTRCE ROMClass end address = 0x00007F42307B9A28\n2SCLTEXTMSA Metadata start address = 0x00007F42313D42FC\n2SCLTEXTCEA Cache end address = 0x00007F4231600000\n2SCLTEXTRTF Runtime flags = 0x00102001ECA6028B\n2SCLTEXTCGN Cache generation = 35\nNULL\n2SCLTEXTCSZ Cache size = 16776608\n2SCLTEXTSMB Softmx bytes = 16776608\n2SCLTEXTFRB Free bytes = 12691668\n2SCLTEXTRCB ROMClass bytes = 1694248\n2SCLTEXTAOB AOT code bytes = 0\n2SCLTEXTADB AOT data bytes = 0\n2SCLTEXTAHB AOT class hierarchy bytes = 32\n2SCLTEXTATB AOT thunk bytes = 0\n2SCLTEXTARB Reserved space for AOT bytes = -1\n2SCLTEXTAMB Maximum space for AOT bytes = -1\n2SCLTEXTJHB JIT hint bytes = 308\n2SCLTEXTJPB JIT profile bytes = 2296\n2SCLTEXTJRB Reserved space for JIT data bytes = -1\n2SCLTEXTJMB Maximum space for JIT data bytes = -1\n2SCLTEXTNOB Java Object bytes = 0\n2SCLTEXTZCB Zip cache bytes = 919328\n2SCLTEXTRWB ReadWrite bytes = 114080\n2SCLTEXTJCB JCL data bytes = 0\n2SCLTEXTBDA Byte data bytes = 0\n2SCLTEXTMDA Metadata bytes = 23448\n2SCLTEXTDAS Class debug area size = 1331200\n2SCLTEXTDAU Class debug area % used = 11%\n2SCLTEXTDAN Class LineNumberTable bytes = 156240\n2SCLTEXTDAV Class LocalVariableTable bytes = 0\nNULL\n2SCLTEXTNRC Number ROMClasses = 595\n2SCLTEXTNAM Number AOT Methods = 0\n2SCLTEXTNAD Number AOT Data Entries = 0\n2SCLTEXTNAH Number AOT Class Hierarchy = 1\n2SCLTEXTNAT Number AOT Thunks = 0\n2SCLTEXTNJH Number JIT Hints = 14\n2SCLTEXTNJP Number JIT Profiles = 20\n2SCLTEXTNCP Number Classpaths = 1\n2SCLTEXTNUR Number URLs = 0\n2SCLTEXTNTK Number Tokens = 0\n2SCLTEXTNOJ Number Java Objects = 0\n2SCLTEXTNZC Number Zip Caches = 5\n2SCLTEXTNJC Number JCL Entries = 0\n2SCLTEXTNST Number Stale classes = 0\n2SCLTEXTPST Percent Stale classes = 0%\nNULL\n2SCLTEXTCPF Cache is 24% full\nNULL\n1SCLTEXTCMST Cache Memory Status\nNULL ------------------\n1SCLTEXTCNTD Cache Name Feature Memory type Cache path\nNULL\n2SCLTEXTCMDT sharedcc_doc-javacore CR Memory mapped file /tmp/javasharedresources/C290M4F1A64P_sharedcc_doc-javacore_G35\nNULL\n1SCLTEXTCMST Cache Lock Status\nNULL ------------------\n1SCLTEXTCNTD Lock Name Lock type TID owning lock\nNULL\n2SCLTEXTCWRL Cache write lock File lock Unowned\n2SCLTEXTCRWL Cache read/write lock File lock Unowned\nNULL\n\n\n\n\n\nCLASSES\n\n\nThe classes section shows information about class loaders. The first part is a summary that records each available class loader (\n2CLTEXTCLLOADER\n) followed by the number of libraries and classes that it loaded. This information is followed by a more detailed list of libraries (\n1CLTEXTCLLIB\n) and classes (\n1CLTEXTCLLO\n) that are loaded.\n\n\nIn the example you can see that the \njava/lang/InternalAnonymousClassLoader\n loaded 2 classes, \njdk/internal/loader/BuiltinClassLoader$$Lambda$2/00000000F03876A0(0x0000000001030F00)\n and \njdk/internal/loader/BuiltinClassLoader$$Lambda$1/00000000F00D2460(0x0000000001018A00)\n.\n\n\nNULL ------------------------------------------------------------------------\n0SECTION CLASSES subcomponent dump routine\nNULL =================================\n1CLTEXTCLLOS Classloader summaries\n1CLTEXTCLLSS 12345678: 1=primordial,2=extension,3=shareable,4=middleware,5=system,6=trusted,7=application,8=delegating\n2CLTEXTCLLOADER p---st-- Loader *System*(0x00000000FFE1D258)\n3CLNMBRLOADEDLIB Number of loaded libraries 5\n3CLNMBRLOADEDCL Number of loaded classes 638\n2CLTEXTCLLOADER -x--st-- Loader jdk/internal/loader/ClassLoaders$PlatformClassLoader(0x00000000FFE1D4F0), Parent *none*(0x0000000000000000)\n3CLNMBRLOADEDLIB Number of loaded libraries 0\n3CLNMBRLOADEDCL Number of loaded classes 0\n2CLTEXTCLLOADER ----st-- Loader java/lang/InternalAnonymousClassLoader(0x00000000FFE1DFD0), Parent *none*(0x0000000000000000)\n3CLNMBRLOADEDLIB Number of loaded libraries 0\n3CLNMBRLOADEDCL Number of loaded classes 2\n2CLTEXTCLLOADER -----ta- Loader jdk/internal/loader/ClassLoaders$AppClassLoader(0x00000000FFE1DAD0), Parent jdk/internal/loader/ClassLoaders$PlatformClassLoader(0x00000000FFE1D4F0)\n3CLNMBRLOADEDLIB Number of loaded libraries 0\n3CLNMBRLOADEDCL Number of loaded classes 0\n1CLTEXTCLLIB ClassLoader loaded libraries\n2CLTEXTCLLIB Loader *System*(0x00000000FFE1D258)\n3CLTEXTLIB /home/me/openj9-openjdk-jdk9/build/linux-x86_64-normal-server-release/images/jdk/lib/compressedrefs/jclse9_29\n3CLTEXTLIB /home/me/openj9-openjdk-jdk9/build/linux-x86_64-normal-server-release/images/jdk/lib/java\n3CLTEXTLIB /home/me/openj9-openjdk-jdk9/build/linux-x86_64-normal-server-release/images/jdk/lib/compressedrefs/j9jit29\n3CLTEXTLIB /home/me/openj9-openjdk-jdk9/build/linux-x86_64-normal-server-release/images/jdk/lib/zip\n3CLTEXTLIB /home/me/openj9-openjdk-jdk9/build/linux-x86_64-normal-server-release/images/jdk/lib/nio\n1CLTEXTCLLOD ClassLoader loaded classes\n2CLTEXTCLLOAD Loader *System*(0x00000000FFE1D258)\n3CLTEXTCLASS [Ljava/lang/Thread$State;(0x0000000001056400)\n...\n2CLTEXTCLLOAD Loader jdk/internal/loader/ClassLoaders$PlatformClassLoader(0x00000000FFE1D4F0)\n2CLTEXTCLLOAD Loader java/lang/InternalAnonymousClassLoader(0x00000000FFE1DFD0)\n3CLTEXTCLASS jdk/internal/loader/BuiltinClassLoader$$Lambda$2/00000000F03876A0(0x0000000001030F00)\n3CLTEXTCLASS jdk/internal/loader/BuiltinClassLoader$$Lambda$1/00000000F00D2460(0x0000000001018A00)\n2CLTEXTCLLOAD Loader jdk/internal/loader/ClassLoaders$AppClassLoader(0x00000000FFE1DAD0)\n\n\n\n\n\nScenarios\n\n\nGeneral Protection Fault\n\n\nIn this scenario, a Java application has crashed due to a General Protection Fault (GPF), automatically generating a Java dump file.\n\n\nThe first section of the file (TITLE) tells you that the GPF triggered the Java dump.\n\n\n0SECTION TITLE subcomponent dump routine\nNULL ===============================\n1TICHARSET UTF-8\n1TISIGINFO Dump Event \"gpf\" (00002000) received\n1TIDATETIME Date: 2018/09/24 at 15:18:03:115\n1TINANOTIME System nanotime: 4498949283020796\n1TIFILENAME Javacore filename: /home/test/JNICrasher/javacore.20180924.151801.29399.0002.txt\n1TIREQFLAGS Request Flags: 0x81 (exclusive+preempt)\n1TIPREPSTATE Prep State: 0x100 (trace_disabled)\n1TIPREPINFO Exclusive VM access not taken: data may not be consistent across javacore sections\n\n\n\n\n\nTo troubleshoot this problem, you need to know which thread caused the GPF to occur. The thread that was running at the time of the crash is reported as the \ncurrent thread\n\nin the THREADS section of the Java dump. Here is an extract from the THREADS section:\n\n\nNULL ------------------------------------------------------------------------\n0SECTION THREADS subcomponent dump routine\nNULL =================================\nNULL\n1XMPOOLINFO JVM Thread pool info:\n2XMPOOLTOTAL Current total number of pooled threads: 16\n2XMPOOLLIVE Current total number of live threads: 15\n2XMPOOLDAEMON Current total number of live daemon threads: 14\nNULL \n1XMCURTHDINFO Current thread\n3XMTHREADINFO \"main\" J9VMThread:0xB6B60E00, omrthread_t:0xB6B049D8, java/lang/Thread:0xB55444D0, state:R, prio=5\n3XMJAVALTHREAD (java/lang/Thread getId:0x1, isDaemon:false)\n3XMTHREADINFO1 (native thread ID:0x72D8, native priority:0x5, native policy:UNKNOWN, vmstate:R, vm thread flags:0x00000000)\n3XMTHREADINFO2 (native stack address range from:0xB6CE3000, to:0xB74E4000, size:0x801000)\n3XMCPUTIME CPU usage total: 0.319865924 secs, current category=\"Application\"\n3XMHEAPALLOC Heap bytes allocated since last GC cycle=778008 (0xBDF18)\n3XMTHREADINFO3 Java callstack:\n4XESTACKTRACE at JNICrasher.doSomethingThatCrashes(Native Method)\n4XESTACKTRACE at JNICrasher.main(JNICrasher.java:7)\n3XMTHREADINFO3 Native callstack:\n4XENATIVESTACK (0xB6C6F663 [libj9prt29.so+0x3b663])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB6C6F1CE [libj9prt29.so+0x3b1ce])\n4XENATIVESTACK (0xB6C6F2C6 [libj9prt29.so+0x3b2c6])\n4XENATIVESTACK (0xB6C6ED93 [libj9prt29.so+0x3ad93])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB6C6ED07 [libj9prt29.so+0x3ad07])\n4XENATIVESTACK (0xB6C6AA3D [libj9prt29.so+0x36a3d])\n4XENATIVESTACK (0xB6C6C3A4 [libj9prt29.so+0x383a4])\n4XENATIVESTACK (0xB667FA19 [libj9dmp29.so+0xfa19])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB66878CF [libj9dmp29.so+0x178cf])\n4XENATIVESTACK (0xB6688083 [libj9dmp29.so+0x18083])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB6680C0D [libj9dmp29.so+0x10c0d])\n4XENATIVESTACK (0xB667F9D7 [libj9dmp29.so+0xf9d7])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB668B02F [libj9dmp29.so+0x1b02f])\n4XENATIVESTACK (0xB668B4D3 [libj9dmp29.so+0x1b4d3])\n4XENATIVESTACK (0xB66740F1 [libj9dmp29.so+0x40f1])\n4XENATIVESTACK (0xB66726FA [libj9dmp29.so+0x26fa])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB66726A9 [libj9dmp29.so+0x26a9])\n4XENATIVESTACK (0xB6676AE4 [libj9dmp29.so+0x6ae4])\n4XENATIVESTACK (0xB668D75A [libj9dmp29.so+0x1d75a])\n4XENATIVESTACK (0xB6A28DD4 [libj9vm29.so+0x81dd4])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB6A289EE [libj9vm29.so+0x819ee])\n4XENATIVESTACK (0xB6A29A40 [libj9vm29.so+0x82a40])\n4XENATIVESTACK (0xB6C52B6A [libj9prt29.so+0x1eb6a])\n4XENATIVESTACK __kernel_rt_sigreturn+0x0 (0xB7747410)\n4XENATIVESTACK (0xB75330B6 [libffi29.so+0x50b6])\n4XENATIVESTACK ffi_raw_call+0xad (0xB7531C53 [libffi29.so+0x3c53])\n4XENATIVESTACK (0xB69BE4AB [libj9vm29.so+0x174ab])\n4XENATIVESTACK (0xB6A665BC [libj9vm29.so+0xbf5bc])\n4XENATIVESTACK (0xB6A15552 [libj9vm29.so+0x6e552])\n4XENATIVESTACK (0xB6A30894 [libj9vm29.so+0x89894])\n4XENATIVESTACK (0xB6A6F169 [libj9vm29.so+0xc8169])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB6A6F1FA [libj9vm29.so+0xc81fa])\n4XENATIVESTACK (0xB6A30994 [libj9vm29.so+0x89994])\n4XENATIVESTACK (0xB6A2CE4C [libj9vm29.so+0x85e4c])\n4XENATIVESTACK (0xB770487D [libjli.so+0x787d])\n4XENATIVESTACK (0xB7719F72 [libpthread.so.0+0x6f72])\n4XENATIVESTACK clone+0x5e (0xB763543E [libc.so.6+0xee43e])\n\n\n\n\n\nThe extract tells you that the current thread was \njava/lang/Thread\n, and information is provided about the Java callstack and native callstack\n(\n3XMTHREADINFO3\n) at the point at which the crash occurred. To simulate a crash caused by a bug in an application, this example\ncalls a JNI method whose native implementation causes a crash. The Java callstack shows the call to the JNI native method (\nJNIcrasher\n), and the\nnative callstack shows the point of failure. In this example, the native call stack does not include any function names to help you isolate the error\nin the native code. You can get this information from a system dump, which is usually produced alongside the Java dump. Open the\nsystem dump with the \nDump viewer\n and use the \ninfo thread\n command to print the Java and native stack for the current thread.\n\n\nJava OutOfMemoryError\n\n\nIn this scenario, the Java heap runs out of memory, causing an \nOutOfMemoryError\n, which automatically generates a Java dump file.\n\n\nThe first section of the file (TITLE) tells you that a \nsysthrow\n event triggered the Java dump as a result of an OOM (\njava/lang/OutOfMemoryError\n) for\nJava heap space.\n\n\n0SECTION TITLE subcomponent dump routine\nNULL ===============================\n1TICHARSET UTF-8\n1TISIGINFO Dump Event \"systhrow\" (00040000) Detail \"java/lang/OutOfMemoryError\" \"Java heap space\" received\n1TIDATETIME Date: 2018/09/14 at 15:29:42:709\n1TINANOTIME System nanotime: 3635648876608448\n1TIFILENAME Javacore filename: /home/cheesemp/test/javacore.20180914.152929.18885.0003.txt\n1TIREQFLAGS Request Flags: 0x81 (exclusive+preempt)\n1TIPREPSTATE Prep State: 0x104 (exclusive_vm_access+trace_disabled)\n\n\n\n\n\nThe MEMINFO section records how much memory is allocated to the Java heap (\n1STHEAPTYPE Object Memory\n), how much is in use, and how much is free. Solving\nyour problem might be as simple as setting a larger heap size when you start your application.\n\n\nIf you don't know what size the Java heap was set to, you might find that information in the ENVINFO section, which records the command line options that\nwere used when the application started. Look or search for the \n1CIUSERARGS UserArgs:\n string and review the entries recorded for all lines that\nstart \n2CIUSERARG\n. The Java heap size is set by the \n-Xmx\n option. If the size has not been set on the command line by \n-Xmx\n, the default value applies, which\nyou can find in \nDefault Settings\n.\n\n\nIn this scenario the solution to the problem is not an adjustment to the Java heap size. Here is the MEMINFO section:\n\n\n0SECTION MEMINFO subcomponent dump routine\nNULL =================================\nNULL\n1STHEAPTYPE Object Memory\nNULL id start end size space/region\n1STHEAPSPACE 0xB6B49D20 -- -- -- Generational\n1STHEAPREGION 0xB6B4A078 0x95750000 0xB5470000 0x1FD20000 Generational/Tenured Region\n1STHEAPREGION 0xB6B49F10 0xB5470000 0xB54C0000 0x00050000 Generational/Nursery Region\n1STHEAPREGION 0xB6B49DA8 0xB54C0000 0xB5750000 0x00290000 Generational/Nursery Region\nNULL\n1STHEAPTOTAL Total memory: 536870912 (0x20000000)\n1STHEAPINUSE Total memory in use: 302603160 (0x12095B98)\n1STHEAPFREE Total memory free: 234267752 (0x0DF6A468)\n\n\n\n\n\nThe output shows that only 56% of the Java heap is in use, so this suggests that the application is trying to do something sub-optimal. To\ninvestigate further you need to work out which thread was the current thread when the OOM occurred to see what it was trying to do. As in the previous scenario, you can find the\n\ncurrent thread\n in the THREADS section. Here is an extract from the output:\n\n\n0\nSECTION\n \nTHREADS\n \nsubcomponent\n \ndump\n \nroutine\n\n\nNULL\n \n=================================\n\n\nNULL\n\n\n1\nXMPOOLINFO\n \nJVM\n \nThread\n \npool\n \ninfo\n:\n\n\n2\nXMPOOLTOTAL\n \nCurrent\n \ntotal\n \nnumber\n \nof\n \npooled\n \nthreads\n:\n \n16\n\n\n2\nXMPOOLLIVE\n \nCurrent\n \ntotal\n \nnumber\n \nof\n \nlive\n \nthreads\n:\n \n16\n\n\n2\nXMPOOLDAEMON\n \nCurrent\n \ntotal\n \nnumber\n \nof\n \nlive\n \ndaemon\n \nthreads\n:\n \n15\n\n\nNULL\n\n\n1\nXMCURTHDINFO\n \nCurrent\n \nthread\n\n\n3\nXMTHREADINFO\n \n\"main\"\n \nJ9VMThread\n:\n0xB6B60C00\n,\n \nomrthread_t\n:\n0xB6B049D8\n,\n \njava\n/\nlang\n/\nThread\n:\n0x95764520\n,\n \nstate\n:\nR\n,\n \nprio\n=\n5\n\n\n3\nXMJAVALTHREAD\n \n(\njava\n/\nlang\n/\nThread\n \ngetId\n:\n0x1\n,\n \nisDaemon\n:\nfalse\n)\n\n\n3\nXMTHREADINFO1\n \n(\nnative\n \nthread\n \nID\n:\n0x49C6\n,\n \nnative\n \npriority\n:\n0x5\n,\n \nnative\n \npolicy\n:\nUNKNOWN\n,\n \nvmstate\n:\nR\n,\n \nvm\n \nthread\n \nflags\n:\n0x00001020\n)\n\n\n3\nXMTHREADINFO2\n \n(\nnative\n \nstack\n \naddress\n \nrange\n \nfrom\n:\n0xB6CB5000\n,\n \nto\n:\n0xB74B6000\n,\n \nsize\n:\n0x801000\n)\n\n\n3\nXMCPUTIME\n \nCPU\n \nusage\n \ntotal\n:\n \n8.537823831\n \nsecs\n,\n \ncurrent\n \ncategory\n=\n\"Application\"\n\n\n3\nXMHEAPALLOC\n \nHeap\n \nbytes\n \nallocated\n \nsince\n \nlast\n \nGC\n \ncycle\n=\n0\n \n(\n0x0\n)\n\n\n3\nXMTHREADINFO3\n \nJava\n \ncallstack\n:\n\n\n4\nXESTACKTRACE\n \nat\n \njava\n/\nlang\n/\nStringBuffer\n.\nensureCapacityImpl\n(\nStringBuffer\n.\njava\n:\n696\n)\n\n\n4\nXESTACKTRACE\n \nat\n \njava\n/\nlang\n/\nStringBuffer\n.\nappend\n(\nStringBuffer\n.\njava\n:\n486\n(\nCompiled\n \nCode\n))\n\n\n5\nXESTACKTRACE\n \n(\nentered\n \nlock\n:\n \njava\n/\nlang\n/\nStringBuffer\n@0x957645B8\n,\n \nentry\n \ncount\n:\n \n1\n)\n\n\n4\nXESTACKTRACE\n \nat\n \njava\n/\nlang\n/\nStringBuffer\n.\nappend\n(\nStringBuffer\n.\njava\n:\n428\n(\nCompiled\n \nCode\n))\n\n\n4\nXESTACKTRACE\n \nat\n \nHeapBreaker\n.\nmain\n(\nHeapBreaker\n.\njava\n:\n34\n(\nCompiled\n \nCode\n))\n\n\n3\nXMTHREADINFO3\n \nNative\n \ncallstack\n:\n\n\n4\nXENATIVESTACK\n \n(\n0xB6C535B3\n \n[\nlibj9prt29\n.\nso\n+\n0x3b5b3\n])\n\n\n4\nXENATIVESTACK\n \n(\n0xB6C36F3E\n \n[\nlibj9prt29\n.\nso\n+\n0x1ef3e\n])\n\n\n4\nXENATIVESTACK\n \n(\n0xB6C5311E\n \n[\nlibj9prt29\n.\nso\n+\n0x3b11e\n])\n\n\n4\nXENATIVESTACK\n \n(\n0xB6C53216\n \n[\nlibj9prt29\n.\nso\n+\n0x3b216\n])\n\n\n4\nXENATIVESTACK\n \n(\n0xB6C52CE3\n \n[\nlibj9prt29\n.\nso\n+\n0x3ace3\n])\n\n\n4\nXENATIVESTACK\n \n(\n0xB6C36F3E\n \n[\nlibj9prt29\n.\nso\n+\n0x1ef3e\n])\n\n\n4\nXENATIVESTACK\n \n(\n0xB6C52C57\n \n[\nlibj9prt29\n.\nso\n+\n0x3ac57\n])\n\n\n4\nXENATIVESTACK\n \n(\n0xB6C4E9CD\n \n[\nlibj9prt29\n.\nso\n+\n0x369cd\n])\n\n\n4\nXENATIVESTACK\n \n(\n0xB6C502FA\n \n[\nlibj9prt29\n.\nso\n+\n0x382fa\n])\n\n\n\n\n\n\nTo simulate a Java \nOutOfMemoryError\n, this example application repeatedly appends characters to a \nStringBuffer\n object in an infinite loop. The Java callstack shows the \nHeapBreaker.main\n method appending characters (\njava/lang/StringGuffer.append\n) until the method \njava/lang/StringBuffer.ensureCapacityImpl()\n throws the \nOutOfMemoryError\n.\n\n\nStringBuffer objects are wrappers for character arrays (\nchar[]\n) and when the capacity of the underlying array is reached, the contents are automatically copied into a new, larger array. The new array is created in the \nStringBuffer.ensureCapacity()\n method, which more or less doubles the size of the old array. In this scenario, the array takes up all the remaining space in the Java heap.\n\n\nThe MEMINFO section of the Java dump file can also tell you when an unexpectedly large allocation request causes an OOM. Look for the GC History (\n1STGCHTYPE\n) section, which details allocation requests that trigger GC activity. In the sample output you can see that a large allocation request (\nrequestedbytes=603979784\n) triggered a global GC. When the GC could not free up sufficient space in the heap to satisfy the request, the allocation failure generated the OOM.\n\n\n1STGCHTYPE GC History \n3STHSTTYPE 14:29:29:580239000 GMT j9mm.101 - J9AllocateIndexableObject() returning NULL! 0 bytes requested for object of class B6BBC300 from memory space 'Generational' id=B6B49D20\n3STHSTTYPE 14:29:29:579916000 GMT j9mm.134 - Allocation failure end: newspace=2686912/3014656 oldspace=231597224/533856256 loa=5338112/5338112\n3STHSTTYPE 14:29:29:579905000 GMT j9mm.470 - Allocation failure cycle end: newspace=2686912/3014656 oldspace=231597224/533856256 loa=5338112/5338112\n3STHSTTYPE 14:29:29:579859000 GMT j9mm.475 - GlobalGC end: workstackoverflow=0 overflowcount=0 memory=234284136/536870912\n3STHSTTYPE 14:29:29:579807000 GMT j9mm.90 - GlobalGC collect complete\n3STHSTTYPE 14:29:29:579776000 GMT j9mm.137 - Compact end: bytesmoved=301989896\n3STHSTTYPE 14:29:29:313899000 GMT j9mm.136 - Compact start: reason=compact to meet allocation\n3STHSTTYPE 14:29:29:313555000 GMT j9mm.57 - Sweep end\n3STHSTTYPE 14:29:29:310772000 GMT j9mm.56 - Sweep start\n3STHSTTYPE 14:29:29:310765000 GMT j9mm.94 - Class unloading end: classloadersunloaded=0 classesunloaded=0\n3STHSTTYPE 14:29:29:310753000 GMT j9mm.60 - Class unloading start\n3STHSTTYPE 14:29:29:310750000 GMT j9mm.55 - Mark end\n3STHSTTYPE 14:29:29:306013000 GMT j9mm.54 - Mark start\n3STHSTTYPE 14:29:29:305957000 GMT j9mm.474 - GlobalGC start: globalcount=9\n3STHSTTYPE 14:29:29:305888000 GMT j9mm.475 - GlobalGC end: workstackoverflow=0 overflowcount=0 memory=234284136/536870912\n3STHSTTYPE 14:29:29:305837000 GMT j9mm.90 - GlobalGC collect complete\n3STHSTTYPE 14:29:29:305808000 GMT j9mm.137 - Compact end: bytesmoved=189784\n3STHSTTYPE 14:29:29:298042000 GMT j9mm.136 - Compact start: reason=compact to meet allocation\n3STHSTTYPE 14:29:29:297695000 GMT j9mm.57 - Sweep end\n3STHSTTYPE 14:29:29:291696000 GMT j9mm.56 - Sweep start\n3STHSTTYPE 14:29:29:291692000 GMT j9mm.55 - Mark end\n3STHSTTYPE 14:29:29:284994000 GMT j9mm.54 - Mark start\n3STHSTTYPE 14:29:29:284941000 GMT j9mm.474 - GlobalGC start: globalcount=8\n3STHSTTYPE 14:29:29:284916000 GMT j9mm.135 - Exclusive access: exclusiveaccessms=0.016 meanexclusiveaccessms=0.016 threads=0 lastthreadtid=0xB6B61100 beatenbyotherthread=0\n3STHSTTYPE 14:29:29:284914000 GMT j9mm.469 - Allocation failure cycle start: newspace=2678784/3014656 oldspace=80601248/533856256 loa=5338112/5338112 requestedbytes=603979784\n3STHSTTYPE 14:29:29:284893000 GMT j9mm.470 - Allocation failure cycle end: newspace=2678784/3014656 oldspace=80601248/533856256 loa=5338112/5338112\n3STHSTTYPE 14:29:29:284858000 GMT j9mm.560 - LocalGC end: rememberedsetoverflow=0 causedrememberedsetoverflow=0 scancacheoverflow=0 failedflipcount=0 failedflipbytes=0 failedtenurecount=0 failedtenurebytes=0 flipcount=2 flipbytes=64 newspace=2678784/3014656 oldspace=80601248/533856256 loa=5338112/5338112 tenureage=0\n3STHSTTYPE 14:29:29:284140000 GMT j9mm.140 - Tilt ratio: 89\n3STHSTTYPE 14:29:29:283160000 GMT j9mm.64 - LocalGC start: globalcount=8 scavengecount=335 weakrefs=0 soft=0 phantom=0 finalizers=0\n3STHSTTYPE 14:29:29:283123000 GMT j9mm.135 - Exclusive access: exclusiveaccessms=0.016 meanexclusiveaccessms=0.016 threads=0 lastthreadtid=0xB6B61100 beatenbyotherthread=0\n3STHSTTYPE 14:29:29:283120000 GMT j9mm.469 - Allocation failure cycle start: newspace=753616/3014656 oldspace=80601248/533856256 loa=5338112/5338112 requestedbytes=603979784\n3STHSTTYPE 14:29:29:283117000 GMT j9mm.133 - Allocation failure start: newspace=753616/3014656 oldspace=80601248/533856256 loa=5338112/5338112 requestedbytes=603979784\n3STHSTTYPE 14:29:29:269762000 GMT j9mm.134 - Allocation failure end: newspace=2686928/3014656 oldspace=80601248/533856256 loa=5338112/5338112\n3STHSTTYPE 14:29:29:269751000 GMT j9mm.470 - Allocation failure cycle end: newspace=2686976/3014656 oldspace=80601248/533856256 loa=5338112/5338112\n3STHSTTYPE 14:29:29:269718000 GMT j9mm.560 - LocalGC end: rememberedsetoverflow=0 causedrememberedsetoverflow=0 scancacheoverflow=0 failedflipcount=0 failedflipbytes=0 failedtenurecount=0 failedtenurebytes=0 flipcount=0 flipbytes=0 newspace=2686976/3014656 oldspace=80601248/533856256 loa=5338112/5338112 tenureage=0\n3STHSTTYPE 14:29:29:268981000 GMT j9mm.140 - Tilt ratio: 89\n3STHSTTYPE 14:29:29:268007000 GMT j9mm.64 - LocalGC start: globalcount=8 scavengecount=334 weakrefs=0 soft=0 phantom=0 finalizers=0\n3STHSTTYPE 14:29:29:267969000 GMT j9mm.135 - Exclusive access: exclusiveaccessms=0.016 meanexclusiveaccessms=0.016 threads=0 lastthreadtid=0xB6B61100 beatenbyotherthread=0\n3STHSTTYPE 14:29:29:267966000 GMT j9mm.469 - Allocation failure cycle start: newspace=0/3014656 oldspace=80601248/533856256 loa=5338112/5338112 requestedbytes=48\n3STHSTTYPE 14:29:29:267963000 GMT j9mm.133 - Allocation failure start: newspace=0/3014656 oldspace=80601248/533856256 loa=5338112/5338112 requestedbytes=48\n3STHSTTYPE 14:29:29:249015000 GMT j9mm.134 - Allocation failure end: newspace=2686928/3014656 oldspace=80601248/533856256 loa=5338112/5338112\n3STHSTTYPE 14:29:29:249003000 GMT j9mm.470 - Allocation failure cycle end: newspace=2686976/3014656 oldspace=80601248/533856256 loa=5338112/5338112\n3STHSTTYPE 14:29:29:248971000 GMT j9mm.560 - LocalGC end: rememberedsetoverflow=0 causedrememberedsetoverflow=0 scancacheoverflow=0 failedflipcount=0 failedflipbytes=0 failedtenurecount=0 failedtenurebytes=0 flipcount=0 flipbytes=0 newspace=2686976/3014656 oldspace=80601248/533856256 loa=5338112/5338112 tenureage=0\n\n\n\n\n\nAlthough the Java code that was used in this scenario deliberately triggered an \nOutOfMemoryError\n in a pronounced way, similar allocation issues can and do occur when dealing with large data sets such as XML files.\n\n\nThe next step in diagnosing the problem is to open the system dump that gets generated automatically when an \nOutOfMemoryError\n occurs. Open\nthe dump with the \nEclipse Memory Analyzer tool (MAT)\n and search for the \nStringBuffer\n object, which should provide further clues about what went wrong. A common example is seeing the same \nString\n duplicated over and over again, which might indicate that code is stuck in a loop.\n\n\n \nNote:\n If you want to use MAT to analyze your system dump, you must install the Diagnostic Tool Framework for Java (DTFJ) plugin in the Eclipse IDE. Select the following menu items:\n\n\nHelp > Install New Software > Work with \"IBM Diagnostic Tool Framework for Java\" > \n\n\n\n\n\nIf, unlike the previous scenario, you receive an \nOutOfMemoryError\n and the MEMINFO section shows that there is very little space left\non the Java heap, the current thread information is typically not important. The current thread is simply the thread that happened to be current when the space ran out. In this situation you might want to increase your Java heap size. For help with this task, see \nHow to do heap sizing\n.\n\n\nNative OutOfMemoryError\n\n\nIn this scenario, the VM runs out of native memory. Native memory is memory that is used by the VM for storing all virtualized resources and data that it needs for VM operations. Native memory that is available to the VM process is limited by the operating system. The native memory available to the VM might also be subject to additional limits imposed by the operating system, for example Unix \nulimits\n. \n\n\nWhen a \nNativeOutOfMemoryError\n occurs, a Java dump is generated by default. The first section of the file (TITLE) tells you that a systhrow event triggered the Java dump as a result of an OOM (\njava/lang/OutOfMemoryError\n) for native memory.\n\n\n0SECTION TITLE subcomponent dump routine\nNULL ===============================\n1TICHARSET UTF-8\n1TISIGINFO Dump Event \"systhrow\" (00040000) Detail \"java/lang/OutOfMemoryError\" \"native memory exhausted\" received\n1TIDATETIME Date: 2018/09/14 at 15:49:55:887\n1TINANOTIME System nanotime: 3636862054495675\n1TIFILENAME Javacore filename: /home/cheesemp/test/javacore.20180914.154814.19708.0003.txt\n1TIREQFLAGS Request Flags: 0x81 (exclusive+preempt)\n1TIPREPSTATE Prep State: 0x104 (exclusive_vm_access+trace_disabled)\n\n\n\n\n\nSometimes, the current thread is responsible for causing the \nNativeOutOfMemoryError\n. Information about the current thread can be found in the THREADS section, as shown in the following output.\n\n\n0SECTION THREADS subcomponent dump routine\nNULL =================================\nNULL\n1XMPOOLINFO JVM Thread pool info:\n2XMPOOLTOTAL Current total number of pooled threads: 16\n2XMPOOLLIVE Current total number of live threads: 16\n2XMPOOLDAEMON Current total number of live daemon threads: 15\nNULL \n1XMCURTHDINFO Current thread\n3XMTHREADINFO \"main\" J9VMThread:0xB6C60C00, omrthread_t:0xB6C049D8, java/lang/Thread:0xB55E3C10, state:R, prio=5\n3XMJAVALTHREAD (java/lang/Thread getId:0x1, isDaemon:false)\n3XMTHREADINFO1 (native thread ID:0x4CFD, native priority:0x5, native policy:UNKNOWN, vmstate:R, vm thread flags:0x00001020)\n3XMTHREADINFO2 (native stack address range from:0xB6D4E000, to:0xB754F000, size:0x801000)\n3XMCPUTIME CPU usage total: 3.654896026 secs, current category=\"Application\"\n3XMHEAPALLOC Heap bytes allocated since last GC cycle=0 (0x0)\n3XMTHREADINFO3 Java callstack:\n4XESTACKTRACE at sun/misc/Unsafe.allocateDBBMemory(Native Method)\n4XESTACKTRACE at java/nio/DirectByteBuffer.<init>(DirectByteBuffer.java:127(Compiled Code))\n4XESTACKTRACE at java/nio/ByteBuffer.allocateDirect(ByteBuffer.java:311)\n4XESTACKTRACE at NativeHeapBreaker.main(NativeHeapBreaker.java:9)\n3XMTHREADINFO3 Native callstack:\n4XENATIVESTACK (0xB6A9F5B3 [libj9prt29.so+0x3b5b3])\n...\n4XENATIVESTACK (0xB582CC9C [libjclse7b_29.so+0x40c9c])\n4XENATIVESTACK Java_sun_misc_Unsafe_allocateDBBMemory+0x88 (0xB5827F6B [libjclse7b_29.so+0x3bf6b])\n4XENATIVESTACK (0x94A2084A [<unknown>+0x0])\n4XENATIVESTACK (0xB6B2538B [libj9vm29.so+0x6c38b])\n4XENATIVESTACK (0xB6B4074C [libj9vm29.so+0x8774c])\n4XENATIVESTACK (0xB6B7F299 [libj9vm29.so+0xc6299])\n4XENATIVESTACK (0xB6A82F3E [libj9prt29.so+0x1ef3e])\n4XENATIVESTACK (0xB6B7F32A [libj9vm29.so+0xc632a])\n4XENATIVESTACK (0xB6B4084C [libj9vm29.so+0x8784c])\n4XENATIVESTACK (0xB6B3CD0C [libj9vm29.so+0x83d0c])\n4XENATIVESTACK (0xB776F87D [libjli.so+0x787d])\n4XENATIVESTACK (0xB7784F72 [libpthread.so.0+0x6f72])\n4XENATIVESTACK clone+0x5e (0xB76A043E [libc.so.6+0xee43e])\n\n\n\n\n\nFor clarity in the \nNative callstack\n output, \n...\n indicates that some lines are removed.\n\n\nThe Java callstack shows the transition from Java to native code (\nsun/misc/Unsafe.allocateDBBMemory(Native Method)\n), indicating a request for Direct Byte Buffer (DBB) storage. DBB storage is backed by native memory, with the Java heap containing only a reference to the native heap buffer. In this scenario, DBB storage is the likely culprit for this \nNativeOutOfMemoryError\n.\n\n\nThe next step is to investigate the NATIVEMEMINFO section of the Java dump file, which reports the amount of memory used by the JRE process, broken down into component areas.\n\n\n0SECTION NATIVEMEMINFO subcomponent dump routine\nNULL =================================\n0MEMUSER\n1MEMUSER JRE: 3,166,386,688 bytes / 4388 allocations\n1MEMUSER |\n2MEMUSER +--VM: 563,176,824 bytes / 1518 allocations\n2MEMUSER | |\n3MEMUSER | +--Classes: 3,104,416 bytes / 120 allocations\n2MEMUSER | |\n3MEMUSER | +--Memory Manager (GC): 548,181,888 bytes / 398 allocations\n3MEMUSER | | |\n4MEMUSER | | +--Java Heap: 536,932,352 bytes / 1 allocation\n3MEMUSER | | |\n4MEMUSER | | +--Other: 11,249,536 bytes / 397 allocations\n2MEMUSER | |\n3MEMUSER | +--Threads: 10,817,120 bytes / 147 allocations\n3MEMUSER | | |\n4MEMUSER | | +--Java Stack: 115,584 bytes / 16 allocations\n3MEMUSER | | |\n4MEMUSER | | +--Native Stack: 10,616,832 bytes / 17 allocations\n3MEMUSER | | |\n4MEMUSER | | +--Other: 84,704 bytes / 114 allocations\n2MEMUSER | |\n3MEMUSER | +--Trace: 163,688 bytes / 268 allocations\n2MEMUSER | |\n3MEMUSER | +--JVMTI: 17,320 bytes / 13 allocations\n2MEMUSER | |\n3MEMUSER | +--JNI: 23,296 bytes / 55 allocations\n2MEMUSER | |\n3MEMUSER | +--Port Library: 8,576 bytes / 74 allocations\n2MEMUSER | |\n3MEMUSER | +--Other: 860,520 bytes / 443 allocations\n1MEMUSER |\n2MEMUSER +--JIT: 3,744,728 bytes / 122 allocations\n2MEMUSER | |\n3MEMUSER | +--JIT Code Cache: 2,097,152 bytes / 1 allocation\n2MEMUSER | |\n3MEMUSER | +--JIT Data Cache: 524,336 bytes / 1 allocation\n2MEMUSER | |\n3MEMUSER | +--Other: 1,123,240 bytes / 120 allocations\n1MEMUSER |\n2MEMUSER +--Class Libraries: 2,599,463,024 bytes / 2732 allocations\n2MEMUSER | |\n3MEMUSER | +--Harmony Class Libraries: 1,024 bytes / 1 allocation\n2MEMUSER | |\n3MEMUSER | +--VM Class Libraries: 2,599,462,000 bytes / 2731 allocations\n3MEMUSER | | |\n4MEMUSER | | +--sun.misc.Unsafe: 2,598,510,480 bytes / 2484 allocations\n4MEMUSER | | | |\n5MEMUSER | | | +--Direct Byte Buffers: 2,598,510,480 bytes / 2484 allocations\n3MEMUSER | | |\n4MEMUSER | | +--Other: 951,520 bytes / 247 allocations\n1MEMUSER |\n2MEMUSER +--Unknown: 2,112 bytes / 16 allocations\nNULL \n\n\n\n\n\nIn the \nVM Class Libraries\n section, the amount of memory allocated for \nDirect Byte Buffers\n is shown. Because the \nNativeOutOfMemoryError\n was received on a small 32-bit system, a value of \n2,598,510,480 bytes\n indicates that the operating system has run out of memory. On a larger UNIX system, the process might have run out of memory because of the \nulimit\n setting. Increasing the value for \nulimit\n might avoid the error, which you can do temporarily by setting \nulimit -f unlimited\n in your current session.\n\n\nThe theoretical maximum size for a 32-bit process is the size of the 32-bit address space, which is 4 GB. On most operating systems a portion of the address space for each process is used by the kernel, such that the real limit for 32-bit processes is actually significantly less than 4GB. As a result, running out of native memory with a 32-bit VM is quite common.\n\n\nThe same 4 GB limit is also important if you are using a 64-bit VM with compressed references. In compressed references mode, all references to objects, classes, threads, and monitors are represented by 32-bit values for performance reasons, so these structures can be allocated only at 32-bit addresses. However, the operating system might place other allocations within this 4 GB of address space, and if this area becomes sufficiently full or fragmented, the VM throws a native \nNativeOutOfMemoryError\n error. These errors typically occur when the VM tries to create a new thread or load a class. The \nCurrent Thread History\n section should contain more information about what the thread was doing at the VM level when the \nNativeOutOfMemoryError\n error occurred.\n\n\nYou can usually avoid this type of problem by using the \n-Xmcrs\n option to reserve a contiguous area of memory within the lowest 4GB of memory at VM startup.\n\n\nAnother common cause of a \nNativeOutOfMemoryError\n is when an application loads duplicate classes. Classes are allocated outside of the Java heap in native memory. If the value reported for \nClasses\n in the NATIVEMEMINFO section is very large, duplicate classes might be the cause of your problem. The \nEclipse Memory Analyzer Tool (MAT)\n can tell you if you have duplicate classes by using the \nClass Loader Explorer\n feature. Because a system dump is automatically generated as well as a Java dump in response to a \nNativeOutOfMemoryError\n, simply open the system dump in MAT to continue your diagnosis. \n\n\nDeadlock\n\n\nDeadlocks occur when two threads attempt to synchronize on an object and lock an instance of a class. When this happens, your application stops responding and hangs. Generating a Java dump file will quickly tell you whether you have a deadlock situation. Trigger the Java dump by sending a SIGQUIT signal (\nkill -3\n) to the VM.\n\n\nThe VM can detect the most common types of deadlock scenario involving Java monitors. If this type of deadlock is detected, information is provided in the LOCKS section. More complex deadlocks, including those that involve a mixture of native mutexes and Java monitors, are not detected.\n\n\nHere is the output from the code that was used to cause a common deadlock scenario:\n\n\nNULL\n \n\n1L\nKDEADLOCK\n \nDeadlock\n \ndetected\n \n!!!\n\n\nNULL\n \n---------------------\n\n\nNULL\n \n\n2L\nKDEADLOCKTHR\n \nThread\n \n\"Worker Thread 2\"\n \n(\n0x94501D00\n)\n\n\n3L\nKDEADLOCKWTR\n \nis\n \nwaiting\n \nfor\n:\n\n\n4L\nKDEADLOCKMON\n \nsys_mon_t\n:\n0x08C2B344\n \ninfl_mon_t\n:\n \n0x08C2B384\n:\n\n\n4L\nKDEADLOCKOBJ\n \njava\n/\nlang\n/\nObject\n@0xB5666698\n\n\n3L\nKDEADLOCKOWN\n \nwhich\n \nis\n \nowned\n \nby\n:\n\n\n2L\nKDEADLOCKTHR\n \nThread\n \n\"Worker Thread 3\"\n \n(\n0x94507500\n)\n\n\n3L\nKDEADLOCKWTR\n \nwhich\n \nis\n \nwaiting\n \nfor\n:\n\n\n4L\nKDEADLOCKMON\n \nsys_mon_t\n:\n0x08C2B3A0\n \ninfl_mon_t\n:\n \n0x08C2B3E0\n:\n\n\n4L\nKDEADLOCKOBJ\n \njava\n/\nlang\n/\nObject\n@0xB5666678\n\n\n3L\nKDEADLOCKOWN\n \nwhich\n \nis\n \nowned\n \nby\n:\n\n\n2L\nKDEADLOCKTHR\n \nThread\n \n\"Worker Thread 1\"\n \n(\n0x92A3EC00\n)\n\n\n3L\nKDEADLOCKWTR\n \nwhich\n \nis\n \nwaiting\n \nfor\n:\n\n\n4L\nKDEADLOCKMON\n \nsys_mon_t\n:\n0x08C2B2E8\n \ninfl_mon_t\n:\n \n0x08C2B328\n:\n\n\n4L\nKDEADLOCKOBJ\n \njava\n/\nlang\n/\nObject\n@0xB5666688\n\n\n3L\nKDEADLOCKOWN\n \nwhich\n \nis\n \nowned\n \nby\n:\n\n\n2L\nKDEADLOCKTHR\n \nThread\n \n\"Worker Thread 2\"\n \n(\n0x94501D00\n)\n\n\n\n\n\n\nThis output tells you that \nWorker Thread 2\n is waiting for \nWorker Thread 3\n, which is waiting for \nWorker Thread 1\n. Because \nWorker Thread 1\n is also waiting for \nWorker Thread 2\n, there is a deadlock. The next place to look is the output for Java and native stacks, in the THREADS section. By looking at the stack for each of these worker threads you can trace the problem back to specific lines in your application code.\n\n\nIn this example, you can see from the following output that for all worker threads, the stack traces (\n4XESTACKTRACE\n/\n5XESTACKTRACE\n) indicate a problem in line 35 of the application \nDeadLockTest.java\n:\n\n\n3\nXMTHREADINFO\n \n\"Worker Thread 1\"\n \nJ9VMThread\n:\n0x92A3EC00\n,\n \nomrthread_t\n:\n0x92A3C2B0\n,\n \njava\n/\nlang\n/\nThread\n:\n0xB5666778\n,\n \nstate\n:\nB\n,\n \nprio\n=\n5\n\n\n3\nXMJAVALTHREAD\n \n(\njava\n/\nlang\n/\nThread\n \ngetId\n:\n0x13\n,\n \nisDaemon\n:\nfalse\n)\n\n\n3\nXMTHREADINFO1\n \n(\nnative\n \nthread\n \nID\n:\n0x52CF\n,\n \nnative\n \npriority\n:\n0x5\n,\n \nnative\n \npolicy\n:\nUNKNOWN\n,\n \nvmstate\n:\nB\n,\n \nvm\n \nthread\n \nflags\n:\n0x00000201\n)\n\n\n3\nXMTHREADINFO2\n \n(\nnative\n \nstack\n \naddress\n \nrange\n \nfrom\n:\n0x9297E000\n,\n \nto\n:\n0x929BF000\n,\n \nsize\n:\n0x41000\n)\n\n\n3\nXMCPUTIME\n \nCPU\n \nusage\n \ntotal\n:\n \n0.004365543\n \nsecs\n,\n \ncurrent\n \ncategory\n=\n\"Application\"\n\n\n3\nXMTHREADBLOCK\n \nBlocked\n \non\n:\n \njava\n/\nlang\n/\nObject\n@0xB5666688\n \nOwned\n \nby\n:\n \n\"Worker Thread 2\"\n \n(\nJ9VMThread\n:\n0x94501D00\n,\n \njava\n/\nlang\n/\nThread\n:\n0xB56668D0\n)\n\n\n3\nXMHEAPALLOC\n \nHeap\n \nbytes\n \nallocated\n \nsince\n \nlast\n \nGC\n \ncycle\n=\n0\n \n(\n0x0\n)\n\n\n3\nXMTHREADINFO3\n \nJava\n \ncallstack\n:\n\n\n4\nXESTACKTRACE\n \nat\n \nWorkerThread\n.\nrun\n(\nDeadLockTest\n.\njava\n:\n35\n)\n\n\n5\nXESTACKTRACE\n \n(\nentered\n \nlock\n:\n \njava\n/\nlang\n/\nObject\n@0xB5666678\n,\n \nentry\n \ncount\n:\n \n1\n)\n\n\n...\n\n\n3\nXMTHREADINFO\n \n\"Worker Thread 2\"\n \nJ9VMThread\n:\n0x94501D00\n,\n \nomrthread_t\n:\n0x92A3C8F0\n,\n \njava\n/\nlang\n/\nThread\n:\n0xB56668D0\n,\n \nstate\n:\nB\n,\n \nprio\n=\n5\n\n\n3\nXMJAVALTHREAD\n \n(\njava\n/\nlang\n/\nThread\n \ngetId\n:\n0x14\n,\n \nisDaemon\n:\nfalse\n)\n\n\n3\nXMTHREADINFO1\n \n(\nnative\n \nthread\n \nID\n:\n0x52D0\n,\n \nnative\n \npriority\n:\n0x5\n,\n \nnative\n \npolicy\n:\nUNKNOWN\n,\n \nvmstate\n:\nB\n,\n \nvm\n \nthread\n \nflags\n:\n0x00000201\n)\n\n\n3\nXMTHREADINFO2\n \n(\nnative\n \nstack\n \naddress\n \nrange\n \nfrom\n:\n0x946BF000\n,\n \nto\n:\n0x94700000\n,\n \nsize\n:\n0x41000\n)\n\n\n3\nXMCPUTIME\n \nCPU\n \nusage\n \ntotal\n:\n \n0.004555580\n \nsecs\n,\n \ncurrent\n \ncategory\n=\n\"Application\"\n\n\n3\nXMTHREADBLOCK\n \nBlocked\n \non\n:\n \njava\n/\nlang\n/\nObject\n@0xB5666698\n \nOwned\n \nby\n:\n \n\"Worker Thread 3\"\n \n(\nJ9VMThread\n:\n0x94507500\n,\n \njava\n/\nlang\n/\nThread\n:\n0xB5666A18\n)\n\n\n3\nXMHEAPALLOC\n \nHeap\n \nbytes\n \nallocated\n \nsince\n \nlast\n \nGC\n \ncycle\n=\n0\n \n(\n0x0\n)\n\n\n3\nXMTHREADINFO3\n \nJava\n \ncallstack\n:\n\n\n4\nXESTACKTRACE\n \nat\n \nWorkerThread\n.\nrun\n(\nDeadLockTest\n.\njava\n:\n35\n)\n\n\n5\nXESTACKTRACE\n \n(\nentered\n \nlock\n:\n \njava\n/\nlang\n/\nObject\n@0xB5666688\n,\n \nentry\n \ncount\n:\n \n1\n)\n\n\n...\n\n\n3\nXMTHREADINFO\n \n\"Worker Thread 3\"\n \nJ9VMThread\n:\n0x94507500\n,\n \nomrthread_t\n:\n0x92A3CC10\n,\n \njava\n/\nlang\n/\nThread\n:\n0xB5666A18\n,\n \nstate\n:\nB\n,\n \nprio\n=\n5\n\n\n3\nXMJAVALTHREAD\n \n(\njava\n/\nlang\n/\nThread\n \ngetId\n:\n0x15\n,\n \nisDaemon\n:\nfalse\n)\n\n\n3\nXMTHREADINFO1\n \n(\nnative\n \nthread\n \nID\n:\n0x52D1\n,\n \nnative\n \npriority\n:\n0x5\n,\n \nnative\n \npolicy\n:\nUNKNOWN\n,\n \nvmstate\n:\nB\n,\n \nvm\n \nthread\n \nflags\n:\n0x00000201\n)\n\n\n3\nXMTHREADINFO2\n \n(\nnative\n \nstack\n \naddress\n \nrange\n \nfrom\n:\n0x9467E000\n,\n \nto\n:\n0x946BF000\n,\n \nsize\n:\n0x41000\n)\n\n\n3\nXMCPUTIME\n \nCPU\n \nusage\n \ntotal\n:\n \n0.003657010\n \nsecs\n,\n \ncurrent\n \ncategory\n=\n\"Application\"\n\n\n3\nXMTHREADBLOCK\n \nBlocked\n \non\n:\n \njava\n/\nlang\n/\nObject\n@0xB5666678\n \nOwned\n \nby\n:\n \n\"Worker Thread 1\"\n \n(\nJ9VMThread\n:\n0x92A3EC00\n,\n \njava\n/\nlang\n/\nThread\n:\n0xB5666778\n)\n\n\n3\nXMHEAPALLOC\n \nHeap\n \nbytes\n \nallocated\n \nsince\n \nlast\n \nGC\n \ncycle\n=\n0\n \n(\n0x0\n)\n\n\n3\nXMTHREADINFO3\n \nJava\n \ncallstack\n:\n\n\n4\nXESTACKTRACE\n \nat\n \nWorkerThread\n.\nrun\n(\nDeadLockTest\n.\njava\n:\n35\n)\n\n\n5\nXESTACKTRACE\n \n(\nentered\n \nlock\n:\n \njava\n/\nlang\n/\nObject\n@0xB5666698\n,\n \nentry\n \ncount\n:\n \n1\n)\n\n\n\n\n\n\nHang\n\n\nAn application can hang for a number of reasons but the most common cause is excessive global garbage collection (GC) activity, where your application is repeatedly paused because your Java heap has \nalmost\n run out of memory. You can identify this problem by looking at verbose GC output. Collect this output by specifying the \n-verbose:gc\n option.\n\n\nDeadlock situations can also manifest themselves as hangs. For more information on diagnosing this type of problem from a Java dump, see the \ndeadlock\n scenario.\n\n\nIf you have eliminated verbose GC activity and deadlocks, another common hang scenario involves threads that compete and wait for Java object locks. This type of problem can usually be diagnosed by examining a Java dump. The simplest hang scenario involving Java object locks is where a thread acquires a lock that other threads are waiting for, but it doesn't release the lock for some reason.\n\n\nThe first place to look in the Java dump output is the \nLOCKS\n section. This section lists all the monitors and shows which threads have acquired a lock and which threads are waiting. If the hang is caused by a thread not releasing a lock that other threads need, you can see a list of waiting threads in the output.\n\n\nIn this example scenario, the Java dump \nLOCKS\n section shows that \nWorker Thread 0\n (\n3LKMONOBJECT\n) has acquired a lock and there are 19 other worker threads waiting to obtain the lock.\n\n\nNULL\n \n------------------------------------------------------------------------\n\n\n0\nSECTION\n \nLOCKS\n \nsubcomponent\n \ndump\n \nroutine\n\n\nNULL\n \n===============================\n\n\nNULL\n \n\n1L\nKPOOLINFO\n \nMonitor\n \npool\n \ninfo\n:\n\n\n2L\nKPOOLTOTAL\n \nCurrent\n \ntotal\n \nnumber\n \nof\n \nmonitors\n:\n \n1\n\n\nNULL\n \n\n1L\nKMONPOOLDUMP\n \nMonitor\n \nPool\n \nDump\n \n(\nflat\n \n&\n \ninflated\n \nobject\n-\nmonitors\n)\n:\n\n\n2L\nKMONINUSE\n \nsys_mon_t\n:\n0x92711200\n \ninfl_mon_t\n:\n \n0x92711240\n:\n\n\n3L\nKMONOBJECT\n \njava\n/\nlang\n/\nObject\n@0xB56658D8\n:\n \nFlat\n \nlocked\n \nby\n \n\"Worker Thread 0\"\n \n(\nJ9VMThread\n:\n0x92A3EC00\n),\n \nentry\n \ncount\n \n1\n\n\n3L\nKWAITERQ\n \nWaiting\n \nto\n \nenter\n:\n\n\n3L\nKWAITER\n \n\"Worker Thread 1\"\n \n(\nJ9VMThread\n:\n0x92703F00\n)\n\n\n3L\nKWAITER\n \n\"Worker Thread 2\"\n \n(\nJ9VMThread\n:\n0x92709C00\n)\n\n\n3L\nKWAITER\n \n\"Worker Thread 3\"\n \n(\nJ9VMThread\n:\n0x92710A00\n)\n\n\n3L\nKWAITER\n \n\"Worker Thread 4\"\n \n(\nJ9VMThread\n:\n0x92717F00\n)\n\n\n3L\nKWAITER\n \n\"Worker Thread 5\"\n \n(\nJ9VMThread\n:\n0x9271DC00\n)\n\n\n3L\nKWAITER\n \n\"Worker Thread 6\"\n \n(\nJ9VMThread\n:\n0x92723A00\n)\n\n\n3L\nKWAITER\n \n\"Worker Thread 7\"\n \n(\nJ9VMThread\n:\n0x92729800\n)\n\n\n3L\nKWAITER\n \n\"Worker Thread 8\"\n \n(\nJ9VMThread\n:\n0x92733700\n)\n\n\n3L\nKWAITER\n \n\"Worker Thread 9\"\n \n(\nJ9VMThread\n:\n0x92739400\n)\n\n\n3L\nKWAITER\n \n\"Worker Thread 10\"\n \n(\nJ9VMThread\n:\n0x92740200\n)\n\n\n3L\nKWAITER\n \n\"Worker Thread 11\"\n \n(\nJ9VMThread\n:\n0x92748100\n)\n\n\n3L\nKWAITER\n \n\"Worker Thread 12\"\n \n(\nJ9VMThread\n:\n0x9274DF00\n)\n\n\n3L\nKWAITER\n \n\"Worker Thread 13\"\n \n(\nJ9VMThread\n:\n0x92754D00\n)\n\n\n3L\nKWAITER\n \n\"Worker Thread 14\"\n \n(\nJ9VMThread\n:\n0x9275AA00\n)\n\n\n3L\nKWAITER\n \n\"Worker Thread 15\"\n \n(\nJ9VMThread\n:\n0x92760800\n)\n\n\n3L\nKWAITER\n \n\"Worker Thread 16\"\n \n(\nJ9VMThread\n:\n0x92766600\n)\n\n\n3L\nKWAITER\n \n\"Worker Thread 17\"\n \n(\nJ9VMThread\n:\n0x9276C300\n)\n\n\n3L\nKWAITER\n \n\"Worker Thread 18\"\n \n(\nJ9VMThread\n:\n0x92773100\n)\n\n\n3L\nKWAITER\n \n\"Worker Thread 19\"\n \n(\nJ9VMThread\n:\n0x92778F00\n)\n\n\nNULL\n \n\n\n\n\n\nThe next step is to determine why \nWorker Thread 0\n is not releasing the lock. The best place to start is the stack trace for this thread, which you can find by searching on the thread name or J9VMThread ID in the \nTHREADS\n section.\n\n\nThe following extract shows the details for \n\"Worker Thread 0\" (J9VMThread:0x92A3EC00)\n:\n\n\nNULL\n\n\n3\nXMTHREADINFO\n \n\"Worker Thread 0\"\n \nJ9VMThread\n:\n0x92A3EC00\n,\n \nomrthread_t\n:\n0x92A3C280\n,\n \njava\n/\nlang\n/\nThread\n:\n0xB56668B8\n,\n \nstate\n:\nCW\n,\n \nprio\n=\n5\n\n\n3\nXMJAVALTHREAD\n \n(\njava\n/\nlang\n/\nThread\n \ngetId\n:\n0x13\n,\n \nisDaemon\n:\nfalse\n)\n\n\n3\nXMTHREADINFO1\n \n(\nnative\n \nthread\n \nID\n:\n0x511F\n,\n \nnative\n \npriority\n:\n0x5\n,\n \nnative\n \npolicy\n:\nUNKNOWN\n,\n \nvmstate\n:\nCW\n,\n \nvm\n \nthread\n \nflags\n:\n0x00000401\n)\n\n\n3\nXMTHREADINFO2\n \n(\nnative\n \nstack\n \naddress\n \nrange\n \nfrom\n:\n0x9297E000\n,\n \nto\n:\n0x929BF000\n,\n \nsize\n:\n0x41000\n)\n\n\n3\nXMCPUTIME\n \nCPU\n \nusage\n \ntotal\n:\n \n0.000211878\n \nsecs\n,\n \ncurrent\n \ncategory\n=\n\"Application\"\n\n\n3\nXMHEAPALLOC\n \nHeap\n \nbytes\n \nallocated\n \nsince\n \nlast\n \nGC\n \ncycle\n=\n0\n \n(\n0x0\n)\n\n\n3\nXMTHREADINFO3\n \nJava\n \ncallstack\n:\n\n\n4\nXESTACKTRACE\n \nat\n \njava\n/\nlang\n/\nThread\n.\nsleep\n(\nNative\n \nMethod\n)\n\n\n4\nXESTACKTRACE\n \nat\n \njava\n/\nlang\n/\nThread\n.\nsleep\n(\nThread\n.\njava\n:\n941\n)\n\n\n4\nXESTACKTRACE\n \nat\n \nWorkerThread\n.\ndoWork\n(\nHangTest\n.\njava\n:\n37\n)\n\n\n4\nXESTACKTRACE\n \nat\n \nWorkerThread\n.\nrun\n(\nHangTest\n.\njava\n:\n31\n)\n\n\n5\nXESTACKTRACE\n \n(\nentered\n \nlock\n:\n \njava\n/\nlang\n/\nObject\n@0xB56658D8\n,\n \nentry\n \ncount\n:\n \n1\n)\n\n\n\n\n\n\nIn the last line of this output you can see where the thread acquired the lock. Working up from this line, you can see that \nWorkerThread.run\n was called, which in turn called \nWorkerThread.doWork\n. The stack shows that the thread then entered a call to \njava/lang/Thread.sleep\n in HangTest.java on line 37, which is preventing the thread from completing its work and releasing the lock. In this example the \nsleep\n call was added to induce a hang, but in real-world scenarios the cause could be any blocking operation, such as reading from an input stream or socket. Another possibility is that the thread is waiting for \nanother\n lock owned by yet another thread.\n\n\nIt is important to remember that each Java dump represents a single snapshot in time. You should generate at least three Java dumps separated by a short pause, for example 30 seconds, and compare the output. This comparison tells you whether the threads involved are stuck in a fixed state or whether they are moving.\n\n\nIn this example, the threads do not move and the investigation needs to focus on the logic in \nWorkerThread.doWork\n to understand why \nWorker Thread 0\n entered the \njava/lang/Thread.sleep\n call.\n\n\nAnother common scenario is where each Java dump shows a number of threads waiting for a lock owned by another thread, but the list of waiting threads and the lock-owning thread change over time. In this case the cause is likely to be a bottleneck caused by thread contention, where the threads are continually competing for the same lock. In severe cases, the lock is held only for a small amount of time but there are lots of threads trying to obtain it. Because more time is spent handling the lock and scheduling the thread than executing application code, the degradation in performance is manifested as a hang. Thread contention is usually caused by an application design problem. You can use a similar approach to the one used in this scenario to determime which lines of code are responsible for the contention.",
"title": "Java dump"
},
{
"location": "/dump_javadump/#java-dump",
- "text": "Java dumps, sometimes referred to as Java cores , are produced when the VM ends unexpectedly because of an operating system signal, OutOfMemoryError , or a user-initiated keystroke combination. You can also generate a Java dump by calling the Dump API programmatically from your application or specifying the -Xdump:java option on the command line. If your Java application crashes or hangs, Java dumps can provide useful information to help you diagnose the root cause. If your application crashes, Javadumps are generated automatically for the following types of failure: the VM receives an unexpected signal or an assertion failure the VM runs out of memory If your application hangs, you can trigger the generation of a Java dump by sending a SIGQUIT signal ( kill -3 ) to the VM. Note: On Windows, if you started the VM in a console window you can force the VM to produce a Java dump in response to a SIGBREAK signal\n(Ctrl-Break keyboard combination). If you didn't start in a console window there is no equivalent to a Linux kill command on Windows for sending signals.\nThe only option here is to trigger a full system dump by finding the VM process in the Processes tab of the Windows Task Manager and clicking Create dump file . To help you understand how a Java dump can help you with problem diagnosis, this topic includes a few scenarios to help you interpret the data: A crash caused by a general protection fault (gpf) A Java heap OutOfMemoryError (OOM) A native OutOfMemoryError (OOM) A deadlock situation",
+ "text": "Java dumps, sometimes referred to as Java cores , are produced when the VM ends unexpectedly because of an operating system signal, OutOfMemoryError , or a user-initiated keystroke combination. You can also generate a Java dump by calling the Dump API programmatically from your application or specifying the -Xdump:java option on the command line. If your Java application crashes or hangs, Java dumps can provide useful information to help you diagnose the root cause. If your application crashes, Javadumps are generated automatically for the following types of failure: the VM receives an unexpected signal or an assertion failure the VM runs out of memory If your application hangs, you can trigger the generation of a Java dump by sending a SIGQUIT signal ( kill -3 ) to the VM. Note: On Windows, if you started the VM in a console window you can force the VM to produce a Java dump in response to a SIGBREAK signal\n(Ctrl-Break keyboard combination). If you didn't start in a console window there is no equivalent to a Linux kill command on Windows for sending signals.\nThe only option here is to trigger a full system dump by finding the VM process in the Processes tab of the Windows Task Manager and clicking Create dump file . To help you understand how a Java dump can help you with problem diagnosis, this topic includes a few scenarios to help you interpret the data: A crash caused by a general protection fault (gpf) A Java heap OutOfMemoryError (OOM) A native OutOfMemoryError (OOM) A deadlock situation A hang",
"title": "Java dump"
},
{
@@ -491,6 +491,11 @@
"title": "Deadlock"
},
{
+ "location": "/dump_javadump/#hang",
+ "text": "An application can hang for a number of reasons but the most common cause is excessive global garbage collection (GC) activity, where your application is repeatedly paused because your Java heap has almost run out of memory. You can identify this problem by looking at verbose GC output. Collect this output by specifying the -verbose:gc option. Deadlock situations can also manifest themselves as hangs. For more information on diagnosing this type of problem from a Java dump, see the deadlock scenario. If you have eliminated verbose GC activity and deadlocks, another common hang scenario involves threads that compete and wait for Java object locks. This type of problem can usually be diagnosed by examining a Java dump. The simplest hang scenario involving Java object locks is where a thread acquires a lock that other threads are waiting for, but it doesn't release the lock for some reason. The first place to look in the Java dump output is the LOCKS section. This section lists all the monitors and shows which threads have acquired a lock and which threads are waiting. If the hang is caused by a thread not releasing a lock that other threads need, you can see a list of waiting threads in the output. In this example scenario, the Java dump LOCKS section shows that Worker Thread 0 ( 3LKMONOBJECT ) has acquired a lock and there are 19 other worker threads waiting to obtain the lock. NULL ------------------------------------------------------------------------ 0 SECTION LOCKS subcomponent dump routine NULL =============================== NULL 1L KPOOLINFO Monitor pool info : 2L KPOOLTOTAL Current total number of monitors : 1 NULL 1L KMONPOOLDUMP Monitor Pool Dump ( flat & inflated object - monitors ) : 2L KMONINUSE sys_mon_t : 0x92711200 infl_mon_t : 0x92711240 : 3L KMONOBJECT java / lang / Object @0xB56658D8 : Flat locked by \"Worker Thread 0\" ( J9VMThread : 0x92A3EC00 ), entry count 1 3L KWAITERQ Waiting to enter : 3L KWAITER \"Worker Thread 1\" ( J9VMThread : 0x92703F00 ) 3L KWAITER \"Worker Thread 2\" ( J9VMThread : 0x92709C00 ) 3L KWAITER \"Worker Thread 3\" ( J9VMThread : 0x92710A00 ) 3L KWAITER \"Worker Thread 4\" ( J9VMThread : 0x92717F00 ) 3L KWAITER \"Worker Thread 5\" ( J9VMThread : 0x9271DC00 ) 3L KWAITER \"Worker Thread 6\" ( J9VMThread : 0x92723A00 ) 3L KWAITER \"Worker Thread 7\" ( J9VMThread : 0x92729800 ) 3L KWAITER \"Worker Thread 8\" ( J9VMThread : 0x92733700 ) 3L KWAITER \"Worker Thread 9\" ( J9VMThread : 0x92739400 ) 3L KWAITER \"Worker Thread 10\" ( J9VMThread : 0x92740200 ) 3L KWAITER \"Worker Thread 11\" ( J9VMThread : 0x92748100 ) 3L KWAITER \"Worker Thread 12\" ( J9VMThread : 0x9274DF00 ) 3L KWAITER \"Worker Thread 13\" ( J9VMThread : 0x92754D00 ) 3L KWAITER \"Worker Thread 14\" ( J9VMThread : 0x9275AA00 ) 3L KWAITER \"Worker Thread 15\" ( J9VMThread : 0x92760800 ) 3L KWAITER \"Worker Thread 16\" ( J9VMThread : 0x92766600 ) 3L KWAITER \"Worker Thread 17\" ( J9VMThread : 0x9276C300 ) 3L KWAITER \"Worker Thread 18\" ( J9VMThread : 0x92773100 ) 3L KWAITER \"Worker Thread 19\" ( J9VMThread : 0x92778F00 ) NULL The next step is to determine why Worker Thread 0 is not releasing the lock. The best place to start is the stack trace for this thread, which you can find by searching on the thread name or J9VMThread ID in the THREADS section. The following extract shows the details for \"Worker Thread 0\" (J9VMThread:0x92A3EC00) : NULL 3 XMTHREADINFO \"Worker Thread 0\" J9VMThread : 0x92A3EC00 , omrthread_t : 0x92A3C280 , java / lang / Thread : 0xB56668B8 , state : CW , prio = 5 3 XMJAVALTHREAD ( java / lang / Thread getId : 0x13 , isDaemon : false ) 3 XMTHREADINFO1 ( native thread ID : 0x511F , native priority : 0x5 , native policy : UNKNOWN , vmstate : CW , vm thread flags : 0x00000401 ) 3 XMTHREADINFO2 ( native stack address range from : 0x9297E000 , to : 0x929BF000 , size : 0x41000 ) 3 XMCPUTIME CPU usage total : 0.000211878 secs , current category = \"Application\" 3 XMHEAPALLOC Heap bytes allocated since last GC cycle = 0 ( 0x0 ) 3 XMTHREADINFO3 Java callstack : 4 XESTACKTRACE at java / lang / Thread . sleep ( Native Method ) 4 XESTACKTRACE at java / lang / Thread . sleep ( Thread . java : 941 ) 4 XESTACKTRACE at WorkerThread . doWork ( HangTest . java : 37 ) 4 XESTACKTRACE at WorkerThread . run ( HangTest . java : 31 ) 5 XESTACKTRACE ( entered lock : java / lang / Object @0xB56658D8 , entry count : 1 ) In the last line of this output you can see where the thread acquired the lock. Working up from this line, you can see that WorkerThread.run was called, which in turn called WorkerThread.doWork . The stack shows that the thread then entered a call to java/lang/Thread.sleep in HangTest.java on line 37, which is preventing the thread from completing its work and releasing the lock. In this example the sleep call was added to induce a hang, but in real-world scenarios the cause could be any blocking operation, such as reading from an input stream or socket. Another possibility is that the thread is waiting for another lock owned by yet another thread. It is important to remember that each Java dump represents a single snapshot in time. You should generate at least three Java dumps separated by a short pause, for example 30 seconds, and compare the output. This comparison tells you whether the threads involved are stuck in a fixed state or whether they are moving. In this example, the threads do not move and the investigation needs to focus on the logic in WorkerThread.doWork to understand why Worker Thread 0 entered the java/lang/Thread.sleep call. Another common scenario is where each Java dump shows a number of threads waiting for a lock owned by another thread, but the list of waiting threads and the lock-owning thread change over time. In this case the cause is likely to be a bottleneck caused by thread contention, where the threads are continually competing for the same lock. In severe cases, the lock is held only for a small amount of time but there are lots of threads trying to obtain it. Because more time is spent handling the lock and scheduling the thread than executing application code, the degradation in performance is manifested as a hang. Thread contention is usually caused by an application design problem. You can use a similar approach to the one used in this scenario to determime which lines of code are responsible for the contention.",
+ "title": "Hang"
+ },
+ {
"location": "/dump_heapdump/",
"text": "Heap dump\n\n\nHeap dumps contain a snapshot of all the live objects that are being used by a running Java application on the Java heap. There are two formats for heap dumps; the classic format, which is ascii text and the PHD format, which is compressed and not readable.\n\n\nA heap dump contains a list of all object instances. For each object instance you can find the following additional data:\n\n\n\n\nThe object address\n\n\nThe type or class name\n\n\nThe size\n\n\nAny references to other objects\n\n\n\n\nFor a visual analysis of a heap dump as an aid to problem determination, use the \nEclipse Memory Analyzer tool (MAT)\n or the \nIBM Memory Analyzer tool\n. Both tools require the Diagnostic Tool Framework for Java (DTFJ) plugin. To install the DTFJ plugin in the Eclipse IDE, select the following menu items:\n\n\nHelp > Install New Software > Work with \"IBM Diagnostic Tool Framework for Java\" > IBM Monitoring and Diagnostic Tools > Diagnostic Tool Framework for Java \n\n\n\n\n\nFor more information about using the Heapdump feature, see \nUsing Heapdump\n.\n\n\nSee also\n\n\n\n\nUsing DTFJ",
"title": "Heap dump"
diff --git a/sitemap.xml b/sitemap.xml
index 532cd53..a5cb101 100644
--- a/sitemap.xml
+++ b/sitemap.xml
@@ -4,7 +4,7 @@
<url>
<loc>https://www.eclipse.org/openj9/docs/</loc>
- <lastmod>2018-11-16</lastmod>
+ <lastmod>2018-11-26</lastmod>
<changefreq>daily</changefreq>
</url>
@@ -12,7 +12,7 @@
<url>
<loc>https://www.eclipse.org/openj9/docs/introduction/</loc>
- <lastmod>2018-11-16</lastmod>
+ <lastmod>2018-11-26</lastmod>
<changefreq>daily</changefreq>
</url>
@@ -21,31 +21,31 @@
<url>
<loc>https://www.eclipse.org/openj9/docs/version0.12/</loc>
- <lastmod>2018-11-16</lastmod>
+ <lastmod>2018-11-26</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>https://www.eclipse.org/openj9/docs/version0.11/</loc>
- <lastmod>2018-11-16</lastmod>
+ <lastmod>2018-11-26</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>https://www.eclipse.org/openj9/docs/version0.10/</loc>
- <lastmod>2018-11-16</lastmod>
+ <lastmod>2018-11-26</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>https://www.eclipse.org/openj9/docs/version0.9/</loc>
- <lastmod>2018-11-16</lastmod>
+ <lastmod>2018-11-26</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>https://www.eclipse.org/openj9/docs/version0.8/</loc>
- <lastmod>2018-11-16</lastmod>
+ <lastmod>2018-11-26</lastmod>
<changefreq>daily</changefreq>
</url>
@@ -54,7 +54,7 @@
<url>
<loc>https://www.eclipse.org/openj9/docs/gc/</loc>
- <lastmod>2018-11-16</lastmod>
+ <lastmod>2018-11-26</lastmod>
<changefreq>daily</changefreq>
</url>
@@ -62,7 +62,7 @@
<url>
<loc>https://www.eclipse.org/openj9/docs/jit/</loc>
- <lastmod>2018-11-16</lastmod>
+ <lastmod>2018-11-26</lastmod>
<changefreq>daily</changefreq>
</url>
@@ -70,7 +70,7 @@
<url>
<loc>https://www.eclipse.org/openj9/docs/aot/</loc>
- <lastmod>2018-11-16</lastmod>
+ <lastmod>2018-11-26</lastmod>
<changefreq>daily</changefreq>
</url>
@@ -78,7 +78,7 @@
<url>
<loc>https://www.eclipse.org/openj9/docs/shrc/</loc>
- <lastmod>2018-11-16</lastmod>
+ <lastmod>2018-11-26</lastmod>
<changefreq>daily</changefreq>
</url>
@@ -87,25 +87,25 @@
<url>
<loc>https://www.eclipse.org/openj9/docs/diag_overview/</loc>
- <lastmod>2018-11-16</lastmod>
+ <lastmod>2018-11-26</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>https://www.eclipse.org/openj9/docs/dump_javadump/</loc>
- <lastmod>2018-11-16</lastmod>
+ <lastmod>2018-11-26</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>https://www.eclipse.org/openj9/docs/dump_heapdump/</loc>
- <lastmod>2018-11-16</lastmod>
+ <lastmod>2018-11-26</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>https://www.eclipse.org/openj9/docs/dump_systemdump/</loc>
- <lastmod>2018-11-16</lastmod>
+ <lastmod>2018-11-26</lastmod>
<changefreq>daily</changefreq>
</url>
@@ -115,19 +115,19 @@
<url>
<loc>https://www.eclipse.org/openj9/docs/cmdline_specifying/</loc>
- <lastmod>2018-11-16</lastmod>
+ <lastmod>2018-11-26</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>https://www.eclipse.org/openj9/docs/cmdline_general/</loc>
- <lastmod>2018-11-16</lastmod>
+ <lastmod>2018-11-26</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>https://www.eclipse.org/openj9/docs/cmdline_migration/</loc>
- <lastmod>2018-11-16</lastmod>
+ <lastmod>2018-11-26</lastmod>
<changefreq>daily</changefreq>
</url>
@@ -155,19 +155,19 @@
<url>
<loc>https://www.eclipse.org/openj9/docs/tool_jdmpview/</loc>
- <lastmod>2018-11-16</lastmod>
+ <lastmod>2018-11-26</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>https://www.eclipse.org/openj9/docs/tool_traceformat/</loc>
- <lastmod>2018-11-16</lastmod>
+ <lastmod>2018-11-26</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>https://www.eclipse.org/openj9/docs/tool_builder/</loc>
- <lastmod>2018-11-16</lastmod>
+ <lastmod>2018-11-26</lastmod>
<changefreq>daily</changefreq>
</url>
@@ -177,31 +177,31 @@
<url>
<loc>https://www.eclipse.org/openj9/docs/openj9_support/</loc>
- <lastmod>2018-11-16</lastmod>
+ <lastmod>2018-11-26</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>https://www.eclipse.org/openj9/docs/openj9_defaults/</loc>
- <lastmod>2018-11-16</lastmod>
+ <lastmod>2018-11-26</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>https://www.eclipse.org/openj9/docs/openj9_directories/</loc>
- <lastmod>2018-11-16</lastmod>
+ <lastmod>2018-11-26</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>https://www.eclipse.org/openj9/docs/messages_intro/</loc>
- <lastmod>2018-11-16</lastmod>
+ <lastmod>2018-11-26</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>https://www.eclipse.org/openj9/docs/env_var/</loc>
- <lastmod>2018-11-16</lastmod>
+ <lastmod>2018-11-26</lastmod>
<changefreq>daily</changefreq>
</url>
@@ -210,7 +210,7 @@
<url>
<loc>https://www.eclipse.org/openj9/docs/legal/</loc>
- <lastmod>2018-11-16</lastmod>
+ <lastmod>2018-11-26</lastmod>
<changefreq>daily</changefreq>
</url>
