Generated from commit: 746026d0bc3343e1ab0050bbc6b60fb9dd3e99c3
diff --git a/dump_javadump/index.html b/dump_javadump/index.html
index a46872a..7700a3f 100644
--- a/dump_javadump/index.html
+++ b/dump_javadump/index.html
@@ -591,6 +591,20 @@
</li>
+ <li class="md-nav__item">
+ <a href="#native-outofmemoryerror" title="Native OutOfMemoryError" class="md-nav__link">
+ Native OutOfMemoryError
+ </a>
+
+</li>
+
+ <li class="md-nav__item">
+ <a href="#deadlock" title="Deadlock" class="md-nav__link">
+ Deadlock
+ </a>
+
+</li>
+
</ul>
</nav>
@@ -3077,6 +3091,20 @@
</li>
+ <li class="md-nav__item">
+ <a href="#native-outofmemoryerror" title="Native OutOfMemoryError" class="md-nav__link">
+ Native OutOfMemoryError
+ </a>
+
+</li>
+
+ <li class="md-nav__item">
+ <a href="#deadlock" title="Deadlock" class="md-nav__link">
+ Deadlock
+ </a>
+
+</li>
+
</ul>
</nav>
@@ -3144,10 +3172,12 @@
<strong>Create dump file</strong>.</p>
</li>
</ul>
-<p>To help you understand how a Java dump can help you with problem diagnosis, we've included a few scenarios at the end of the topic that might help you interpret the data:</p>
+<p>To help you understand how a Java dump can help you with problem diagnosis, this topic includes a few scenarios to help you interpret the data:</p>
<ul>
<li><a href="#general-protection-fault">A crash caused by a general protection fault (gpf)</a></li>
<li><a href="#java-outofmemoryerror">A Java heap OutOfMemoryError (OOM)</a></li>
+<li><a href="#native-outofmemoryerror">A native OutOfMemoryError (OOM)</a></li>
+<li><a href="#deadlock">A deadlock situation</a></li>
</ul>
<h2 id="java-dump-contents">Java dump contents</h2>
<p>Java dumps summarize the state of the VM when the event occurs, with most of the information relating to components of the VM.
@@ -3168,7 +3198,7 @@
<h3 id="gpinfo">GPINFO</h3>
-<p>The GPINFO section provides general information about the system that the JVM is running on. The following example is taken
+<p>The GPINFO section provides general information about the system that the VM is running on. The following example is taken
from a Java dump that was generated on a Linux system.</p>
<div class="codehilite"><pre><span></span>NULL ------------------------------------------------------------------------
0SECTION GPINFO subcomponent dump routine
@@ -3771,7 +3801,7 @@
<h3 id="classes">CLASSES</h3>
<p>The classes section shows information about class loaders. The first part is a summary that records each available class loader (<code>2CLTEXTCLLOADER</code>) followed by the number of libraries and classes that it loaded. This information is followed by a more detailed list of libraries (<code>1CLTEXTCLLIB</code>) and classes (<code>1CLTEXTCLLO</code>) that are loaded.</p>
-<p>In the example we can see that the <code>java/lang/InternalAnonymousClassLoader</code> loaded 2 classes, <code>jdk/internal/loader/BuiltinClassLoader$$Lambda$2/00000000F03876A0(0x0000000001030F00)</code> and <code>jdk/internal/loader/BuiltinClassLoader$$Lambda$1/00000000F00D2460(0x0000000001018A00)</code>.</p>
+<p>In the example you can see that the <code>java/lang/InternalAnonymousClassLoader</code> loaded 2 classes, <code>jdk/internal/loader/BuiltinClassLoader$$Lambda$2/00000000F03876A0(0x0000000001030F00)</code> and <code>jdk/internal/loader/BuiltinClassLoader$$Lambda$1/00000000F00D2460(0x0000000001018A00)</code>.</p>
<div class="codehilite"><pre><span></span>NULL ------------------------------------------------------------------------
0SECTION CLASSES subcomponent dump routine
NULL =================================
@@ -3895,7 +3925,7 @@
</pre></div>
-<p>The extract tells you that the current thread was <code>java/lang/Thread</code>, and information is provided about the Java callstack and Native callstack
+<p>The extract tells you that the current thread was <code>java/lang/Thread</code>, and information is provided about the Java callstack and native callstack
(<code>3XMTHREADINFO3</code>) at the point at which the crash occurred. To simulate a crash caused by a bug in an application, this example
calls a JNI method whose native implementation causes a crash. The Java callstack shows the call to the JNI native method (<code>JNIcrasher</code>), and the
native callstack shows the point of failure. In this example, the native call stack does not include any function names to help you isolate the error
@@ -3923,7 +3953,7 @@
were used when the application started. Look or search for the <code>1CIUSERARGS UserArgs:</code> string and review the entries recorded for all lines that
start <code>2CIUSERARG</code>. The Java heap size is set by the <code>-Xmx</code> option. If the size has not been set on the command line by <code>-Xmx</code>, the default value applies, which
you can find in <a href="../openj9_defaults/">Default Settings</a>.</p>
-<p>In our scenario the solution to the problem is not an adjustment to the Java heap size. Here is the MEMINFO section:</p>
+<p>In this scenario the solution to the problem is not an adjustment to the Java heap size. Here is the MEMINFO section:</p>
<div class="codehilite"><pre><span></span>0SECTION MEMINFO subcomponent dump routine
NULL =================================
NULL
@@ -3941,7 +3971,7 @@
<p>The output shows that only 56% of the Java heap is in use, so this suggests that the application is trying to do something sub-optimal. To
-investigate further we need to work out which thread was the current thread when the OOM occurred to see what it was trying to do. As in our previous scenario, we can find the
+investigate further you need to work out which thread was the current thread when the OOM occurred to see what it was trying to do. As in the previous scenario, you can find the
<strong>current thread</strong> in the THREADS section. Here is an extract from the output:</p>
<div class="codehilite"><pre><span></span><span class="mi">0</span><span class="n">SECTION</span> <span class="n">THREADS</span> <span class="n">subcomponent</span> <span class="n">dump</span> <span class="n">routine</span>
<span class="nb">NULL</span> <span class="o">=================================</span>
@@ -3978,7 +4008,7 @@
<p>To simulate a Java <code>OutOfMemoryError</code>, this example application repeatedly appends characters to a <code>StringBuffer</code> object in an infinite loop. The Java callstack shows the <code>HeapBreaker.main</code> method appending characters (<code>java/lang/StringGuffer.append</code>) until the method <code>java/lang/StringBuffer.ensureCapacityImpl()</code> throws the <code>OutOfMemoryError</code>.</p>
-<p>StringBuffer objects are wrappers for character arrays (<code>char[]</code>) and when the capacity of the underlying array is reached, the contents are automatically copied into a new, larger array. The new array is created in the <code>StringBuffer.ensureCapacity()</code> method, which more or less doubles the size of the old array. In our scenario, the array takes up all the remaining space in the Java heap.</p>
+<p>StringBuffer objects are wrappers for character arrays (<code>char[]</code>) and when the capacity of the underlying array is reached, the contents are automatically copied into a new, larger array. The new array is created in the <code>StringBuffer.ensureCapacity()</code> method, which more or less doubles the size of the old array. In this scenario, the array takes up all the remaining space in the Java heap.</p>
<p>The MEMINFO section of the Java dump file can also tell you when an unexpectedly large allocation request causes an OOM. Look for the GC History (<code>1STGCHTYPE</code>) section, which details allocation requests that trigger GC activity. In the sample output you can see that a large allocation request (<code>requestedbytes=603979784</code>) triggered a global GC. When the GC could not free up sufficient space in the heap to satisfy the request, the allocation failure generated the OOM.</p>
<div class="codehilite"><pre><span></span>1STGCHTYPE GC History
3STHSTTYPE 14:29:29:580239000 GMT j9mm.101 - J9AllocateIndexableObject() returning NULL! 0 bytes requested for object of class B6BBC300 from memory space 'Generational' id=B6B49D20
@@ -4027,7 +4057,7 @@
</pre></div>
-<p>Although the Java code we used in this scenario deliberately triggered an <code>OutOfMemoryError</code> in a pronounced way, similar allocation issues can and do occur when dealing with large data sets such as XML files.</p>
+<p>Although the Java code that was used in this scenario deliberately triggered an <code>OutOfMemoryError</code> in a pronounced way, similar allocation issues can and do occur when dealing with large data sets such as XML files.</p>
<p>The next step in diagnosing the problem is to open the system dump that gets generated automatically when an <code>OutOfMemoryError</code> occurs. Open
the dump with the <a href="https://www.eclipse.org/mat/">Eclipse Memory Analyzer tool (MAT)</a> and search for the <code>StringBuffer</code> object, which should provide further clues about what went wrong. A common example is seeing the same <code>String</code> duplicated over and over again, which might indicate that code is stuck in a loop.</p>
<p><i class="fa fa-pencil-square-o" aria-hidden="true"></i> <strong>Note:</strong> If you want to use MAT to analyze your system dump, you must install the Diagnostic Tool Framework for Java (DTFJ) plugin in the Eclipse IDE. Select the following menu items:</p>
@@ -4036,7 +4066,192 @@
<p>If, unlike the previous scenario, you receive an <code>OutOfMemoryError</code> and the MEMINFO section shows that there is very little space left
-on the Java heap, the current thread information is not important. The current thread is simply the thread that happened to be current when the space ran out. In this situation you might want to increase your Java heap size. For help with this task, see <a href="https://www.ibm.com/support/knowledgecenter/SSYKE2_8.0.0/com.ibm.java.vm.80.doc/docs/mm_heapsizing.html">How to do heap sizing</a>.</p>
+on the Java heap, the current thread information is typically not important. The current thread is simply the thread that happened to be current when the space ran out. In this situation you might want to increase your Java heap size. For help with this task, see <a href="https://www.ibm.com/support/knowledgecenter/SSYKE2_8.0.0/com.ibm.java.vm.80.doc/docs/mm_heapsizing.html">How to do heap sizing</a>.</p>
+<h3 id="native-outofmemoryerror">Native OutOfMemoryError</h3>
+<p>In this scenario, the VM runs out of native memory. Native memory is memory that is used by the VM for storing all virtualized resources and data that it needs for VM operations. Native memory that is available to the VM process is limited by the operating system. The native memory available to the VM might also be subject to additional limits imposed by the operating system, for example Unix <code>ulimits</code>. </p>
+<p>When a <code>NativeOutOfMemoryError</code> occurs, a Java dump is generated by default. The first section of the file (TITLE) tells you that a systhrow event triggered the Java dump as a result of an OOM (<code>java/lang/OutOfMemoryError</code>) for native memory.</p>
+<div class="codehilite"><pre><span></span>0SECTION TITLE subcomponent dump routine
+NULL ===============================
+1TICHARSET UTF-8
+1TISIGINFO Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" "native memory exhausted" received
+1TIDATETIME Date: 2018/09/14 at 15:49:55:887
+1TINANOTIME System nanotime: 3636862054495675
+1TIFILENAME Javacore filename: /home/cheesemp/test/javacore.20180914.154814.19708.0003.txt
+1TIREQFLAGS Request Flags: 0x81 (exclusive+preempt)
+1TIPREPSTATE Prep State: 0x104 (exclusive_vm_access+trace_disabled)
+</pre></div>
+
+
+<p>Sometimes, the current thread is responsible for causing the <code>NativeOutOfMemoryError</code>. Information about the current thread can be found in the THREADS section, as shown in the following output.</p>
+<div class="codehilite"><pre><span></span>0SECTION THREADS subcomponent dump routine
+NULL =================================
+NULL
+1XMPOOLINFO JVM Thread pool info:
+2XMPOOLTOTAL Current total number of pooled threads: 16
+2XMPOOLLIVE Current total number of live threads: 16
+2XMPOOLDAEMON Current total number of live daemon threads: 15
+NULL
+1XMCURTHDINFO Current thread
+3XMTHREADINFO "main" J9VMThread:0xB6C60C00, omrthread_t:0xB6C049D8, java/lang/Thread:0xB55E3C10, state:R, prio=5
+3XMJAVALTHREAD (java/lang/Thread getId:0x1, isDaemon:false)
+3XMTHREADINFO1 (native thread ID:0x4CFD, native priority:0x5, native policy:UNKNOWN, vmstate:R, vm thread flags:0x00001020)
+3XMTHREADINFO2 (native stack address range from:0xB6D4E000, to:0xB754F000, size:0x801000)
+3XMCPUTIME CPU usage total: 3.654896026 secs, current category="Application"
+3XMHEAPALLOC Heap bytes allocated since last GC cycle=0 (0x0)
+3XMTHREADINFO3 Java callstack:
+4XESTACKTRACE at sun/misc/Unsafe.allocateDBBMemory(Native Method)
+4XESTACKTRACE at java/nio/DirectByteBuffer.<init>(DirectByteBuffer.java:127(Compiled Code))
+4XESTACKTRACE at java/nio/ByteBuffer.allocateDirect(ByteBuffer.java:311)
+4XESTACKTRACE at NativeHeapBreaker.main(NativeHeapBreaker.java:9)
+3XMTHREADINFO3 Native callstack:
+4XENATIVESTACK (0xB6A9F5B3 [libj9prt29.so+0x3b5b3])
+...
+4XENATIVESTACK (0xB582CC9C [libjclse7b_29.so+0x40c9c])
+4XENATIVESTACK Java_sun_misc_Unsafe_allocateDBBMemory+0x88 (0xB5827F6B [libjclse7b_29.so+0x3bf6b])
+4XENATIVESTACK (0x94A2084A [<unknown>+0x0])
+4XENATIVESTACK (0xB6B2538B [libj9vm29.so+0x6c38b])
+4XENATIVESTACK (0xB6B4074C [libj9vm29.so+0x8774c])
+4XENATIVESTACK (0xB6B7F299 [libj9vm29.so+0xc6299])
+4XENATIVESTACK (0xB6A82F3E [libj9prt29.so+0x1ef3e])
+4XENATIVESTACK (0xB6B7F32A [libj9vm29.so+0xc632a])
+4XENATIVESTACK (0xB6B4084C [libj9vm29.so+0x8784c])
+4XENATIVESTACK (0xB6B3CD0C [libj9vm29.so+0x83d0c])
+4XENATIVESTACK (0xB776F87D [libjli.so+0x787d])
+4XENATIVESTACK (0xB7784F72 [libpthread.so.0+0x6f72])
+4XENATIVESTACK clone+0x5e (0xB76A043E [libc.so.6+0xee43e])
+</pre></div>
+
+
+<p>For clarity in the <code>Native callstack</code> output, <code>...</code> indicates that some lines are removed.</p>
+<p>The Java callstack shows the transition from Java to native code (<code>sun/misc/Unsafe.allocateDBBMemory(Native Method)</code>), indicating a request for Direct Byte Buffer (DBB) storage. DBB storage is backed by native memory, with the Java heap containing only a reference to the native heap buffer. In this scenario, DBB storage is the likely culprit for this <code>NativeOutOfMemoryError</code>.</p>
+<p>The next step is to investigate the NATIVEMEMINFO section of the Java dump file, which reports the amount of memory used by the JRE process, broken down into component areas.</p>
+<div class="codehilite"><pre><span></span>0SECTION NATIVEMEMINFO subcomponent dump routine
+NULL =================================
+0MEMUSER
+1MEMUSER JRE: 3,166,386,688 bytes / 4388 allocations
+1MEMUSER |
+2MEMUSER +--VM: 563,176,824 bytes / 1518 allocations
+2MEMUSER | |
+3MEMUSER | +--Classes: 3,104,416 bytes / 120 allocations
+2MEMUSER | |
+3MEMUSER | +--Memory Manager (GC): 548,181,888 bytes / 398 allocations
+3MEMUSER | | |
+4MEMUSER | | +--Java Heap: 536,932,352 bytes / 1 allocation
+3MEMUSER | | |
+4MEMUSER | | +--Other: 11,249,536 bytes / 397 allocations
+2MEMUSER | |
+3MEMUSER | +--Threads: 10,817,120 bytes / 147 allocations
+3MEMUSER | | |
+4MEMUSER | | +--Java Stack: 115,584 bytes / 16 allocations
+3MEMUSER | | |
+4MEMUSER | | +--Native Stack: 10,616,832 bytes / 17 allocations
+3MEMUSER | | |
+4MEMUSER | | +--Other: 84,704 bytes / 114 allocations
+2MEMUSER | |
+3MEMUSER | +--Trace: 163,688 bytes / 268 allocations
+2MEMUSER | |
+3MEMUSER | +--JVMTI: 17,320 bytes / 13 allocations
+2MEMUSER | |
+3MEMUSER | +--JNI: 23,296 bytes / 55 allocations
+2MEMUSER | |
+3MEMUSER | +--Port Library: 8,576 bytes / 74 allocations
+2MEMUSER | |
+3MEMUSER | +--Other: 860,520 bytes / 443 allocations
+1MEMUSER |
+2MEMUSER +--JIT: 3,744,728 bytes / 122 allocations
+2MEMUSER | |
+3MEMUSER | +--JIT Code Cache: 2,097,152 bytes / 1 allocation
+2MEMUSER | |
+3MEMUSER | +--JIT Data Cache: 524,336 bytes / 1 allocation
+2MEMUSER | |
+3MEMUSER | +--Other: 1,123,240 bytes / 120 allocations
+1MEMUSER |
+2MEMUSER +--Class Libraries: 2,599,463,024 bytes / 2732 allocations
+2MEMUSER | |
+3MEMUSER | +--Harmony Class Libraries: 1,024 bytes / 1 allocation
+2MEMUSER | |
+3MEMUSER | +--VM Class Libraries: 2,599,462,000 bytes / 2731 allocations
+3MEMUSER | | |
+4MEMUSER | | +--sun.misc.Unsafe: 2,598,510,480 bytes / 2484 allocations
+4MEMUSER | | | |
+5MEMUSER | | | +--Direct Byte Buffers: 2,598,510,480 bytes / 2484 allocations
+3MEMUSER | | |
+4MEMUSER | | +--Other: 951,520 bytes / 247 allocations
+1MEMUSER |
+2MEMUSER +--Unknown: 2,112 bytes / 16 allocations
+NULL
+</pre></div>
+
+
+<p>In the <code>VM Class Libraries</code> section, the amount of memory allocated for <code>Direct Byte Buffers</code> is shown. Because the <code>NativeOutOfMemoryError</code> was received on a small 32-bit system, a value of <code>2,598,510,480 bytes</code> indicates that the operating system has run out of memory. On a larger UNIX system, the process might have run out of memory because of the <code>ulimit</code> setting. Increasing the value for <code>ulimit</code> might avoid the error, which you can do temporarily by setting <code>ulimit -f unlimited</code> in your current session.</p>
+<p>The theoretical maximum size for a 32-bit process is the size of the 32-bit address space, which is 4 GB. On most operating systems a portion of the address space for each process is used by the kernel, such that the real limit for 32-bit processes is actually significantly less than 4GB. As a result, running out of native memory with a 32-bit VM is quite common.</p>
+<p>The same 4 GB limit is also important if you are using a 64-bit VM with compressed references. In compressed references mode, all references to objects, classes, threads, and monitors are represented by 32-bit values for performance reasons, so these structures can be allocated only at 32-bit addresses. However, the operating system might place other allocations within this 4 GB of address space, and if this area becomes sufficiently full or fragmented, the VM throws a native <code>NativeOutOfMemoryError</code> error. These errors typically occur when the VM tries to create a new thread or load a class. The <strong>Current Thread History</strong> section should contain more information about what the thread was doing at the VM level when the <code>NativeOutOfMemoryError</code> error occurred.</p>
+<p>You can usually avoid this type of problem by using the <code>-Xmcrs</code> option to reserve a contiguous area of memory within the lowest 4GB of memory at VM startup.</p>
+<p>Another common cause of a <code>NativeOutOfMemoryError</code> is when an application loads duplicate classes. Classes are allocated outside of the Java heap in native memory. If the value reported for <code>Classes</code> in the NATIVEMEMINFO section is very large, duplicate classes might be the cause of your problem. The <a href="https://www.eclipse.org/mat/">Eclipse Memory Analyzer Tool (MAT)</a> can tell you if you have duplicate classes by using the <em>Class Loader Explorer</em> feature. Because a system dump is automatically generated as well as a Java dump in response to a <code>NativeOutOfMemoryError</code>, simply open the system dump in MAT to continue your diagnosis. </p>
+<h3 id="deadlock">Deadlock</h3>
+<p>Deadlocks occur when two threads attempt to synchronize on an object and lock an instance of a class. When this happens, your application stops responding and hangs. Generating a Java dump file will quickly tell you whether you have a deadlock situation. Trigger the Java dump by sending a SIGQUIT signal (<code>kill -3</code>) to the VM.</p>
+<p>The VM can detect the most common types of deadlock scenario involving Java monitors. If this type of deadlock is detected, information is provided in the LOCKS section. More complex deadlocks, including those that involve a mixture of native mutexes and Java monitors, are not detected.</p>
+<p>Here is the output from the code that was used to cause a common deadlock scenario:</p>
+<div class="codehilite"><pre><span></span><span class="nb">NULL</span>
+<span class="mi">1L</span><span class="n">KDEADLOCK</span> <span class="n">Deadlock</span> <span class="n">detected</span> <span class="o">!!!</span>
+<span class="nb">NULL</span> <span class="o">---------------------</span>
+<span class="nb">NULL</span>
+<span class="mi">2L</span><span class="n">KDEADLOCKTHR</span> <span class="n">Thread</span> <span class="s">"Worker Thread 2"</span> <span class="p">(</span><span class="mh">0x94501D00</span><span class="p">)</span>
+<span class="mi">3L</span><span class="n">KDEADLOCKWTR</span> <span class="n">is</span> <span class="n">waiting</span> <span class="k">for</span><span class="o">:</span>
+<span class="mi">4L</span><span class="n">KDEADLOCKMON</span> <span class="nl">sys_mon_t</span><span class="p">:</span><span class="mh">0x08C2B344</span> <span class="nl">infl_mon_t</span><span class="p">:</span> <span class="mh">0x08C2B384</span><span class="o">:</span>
+<span class="mi">4L</span><span class="n">KDEADLOCKOBJ</span> <span class="n">java</span><span class="o">/</span><span class="n">lang</span><span class="o">/</span><span class="n">Object</span><span class="mh">@0xB5666698</span>
+<span class="mi">3L</span><span class="n">KDEADLOCKOWN</span> <span class="n">which</span> <span class="n">is</span> <span class="n">owned</span> <span class="nl">by</span><span class="p">:</span>
+<span class="mi">2L</span><span class="n">KDEADLOCKTHR</span> <span class="n">Thread</span> <span class="s">"Worker Thread 3"</span> <span class="p">(</span><span class="mh">0x94507500</span><span class="p">)</span>
+<span class="mi">3L</span><span class="n">KDEADLOCKWTR</span> <span class="n">which</span> <span class="n">is</span> <span class="n">waiting</span> <span class="k">for</span><span class="o">:</span>
+<span class="mi">4L</span><span class="n">KDEADLOCKMON</span> <span class="nl">sys_mon_t</span><span class="p">:</span><span class="mh">0x08C2B3A0</span> <span class="nl">infl_mon_t</span><span class="p">:</span> <span class="mh">0x08C2B3E0</span><span class="o">:</span>
+<span class="mi">4L</span><span class="n">KDEADLOCKOBJ</span> <span class="n">java</span><span class="o">/</span><span class="n">lang</span><span class="o">/</span><span class="n">Object</span><span class="mh">@0xB5666678</span>
+<span class="mi">3L</span><span class="n">KDEADLOCKOWN</span> <span class="n">which</span> <span class="n">is</span> <span class="n">owned</span> <span class="nl">by</span><span class="p">:</span>
+<span class="mi">2L</span><span class="n">KDEADLOCKTHR</span> <span class="n">Thread</span> <span class="s">"Worker Thread 1"</span> <span class="p">(</span><span class="mh">0x92A3EC00</span><span class="p">)</span>
+<span class="mi">3L</span><span class="n">KDEADLOCKWTR</span> <span class="n">which</span> <span class="n">is</span> <span class="n">waiting</span> <span class="k">for</span><span class="o">:</span>
+<span class="mi">4L</span><span class="n">KDEADLOCKMON</span> <span class="nl">sys_mon_t</span><span class="p">:</span><span class="mh">0x08C2B2E8</span> <span class="nl">infl_mon_t</span><span class="p">:</span> <span class="mh">0x08C2B328</span><span class="o">:</span>
+<span class="mi">4L</span><span class="n">KDEADLOCKOBJ</span> <span class="n">java</span><span class="o">/</span><span class="n">lang</span><span class="o">/</span><span class="n">Object</span><span class="mh">@0xB5666688</span>
+<span class="mi">3L</span><span class="n">KDEADLOCKOWN</span> <span class="n">which</span> <span class="n">is</span> <span class="n">owned</span> <span class="nl">by</span><span class="p">:</span>
+<span class="mi">2L</span><span class="n">KDEADLOCKTHR</span> <span class="n">Thread</span> <span class="s">"Worker Thread 2"</span> <span class="p">(</span><span class="mh">0x94501D00</span><span class="p">)</span>
+</pre></div>
+
+
+<p>This output tells you that <code>Worker Thread 2</code> is waiting for <code>Worker Thread 3</code>, which is waiting for <code>Worker Thread 1</code>. Because <code>Worker Thread 1</code> is also waiting for <code>Worker Thread 2</code>, there is a deadlock. The next place to look is the output for Java and native stacks, in the THREADS section. By looking at the stack for each of these worker threads you can trace the problem back to specific lines in your application code.</p>
+<p>In this example, you can see from the following output that for all worker threads, the stack traces (<code>4XESTACKTRACE</code>/<code>5XESTACKTRACE</code>) indicate a problem in line 35 of the application <code>DeadLockTest.java</code>:</p>
+<div class="codehilite"><pre><span></span><span class="mi">3</span><span class="n">XMTHREADINFO</span> <span class="s">"Worker Thread 1"</span> <span class="nl">J9VMThread</span><span class="p">:</span><span class="mh">0x92A3EC00</span><span class="p">,</span> <span class="nl">omrthread_t</span><span class="p">:</span><span class="mh">0x92A3C2B0</span><span class="p">,</span> <span class="n">java</span><span class="o">/</span><span class="n">lang</span><span class="o">/</span><span class="nl">Thread</span><span class="p">:</span><span class="mh">0xB5666778</span><span class="p">,</span> <span class="nl">state</span><span class="p">:</span><span class="n">B</span><span class="p">,</span> <span class="n">prio</span><span class="o">=</span><span class="mi">5</span>
+<span class="mi">3</span><span class="n">XMJAVALTHREAD</span> <span class="p">(</span><span class="n">java</span><span class="o">/</span><span class="n">lang</span><span class="o">/</span><span class="n">Thread</span> <span class="nl">getId</span><span class="p">:</span><span class="mh">0x13</span><span class="p">,</span> <span class="nl">isDaemon</span><span class="p">:</span><span class="nb">false</span><span class="p">)</span>
+<span class="mi">3</span><span class="n">XMTHREADINFO1</span> <span class="p">(</span><span class="n">native</span> <span class="kr">thread</span> <span class="nl">ID</span><span class="p">:</span><span class="mh">0x52CF</span><span class="p">,</span> <span class="n">native</span> <span class="nl">priority</span><span class="p">:</span><span class="mh">0x5</span><span class="p">,</span> <span class="n">native</span> <span class="nl">policy</span><span class="p">:</span><span class="n">UNKNOWN</span><span class="p">,</span> <span class="nl">vmstate</span><span class="p">:</span><span class="n">B</span><span class="p">,</span> <span class="n">vm</span> <span class="kr">thread</span> <span class="nl">flags</span><span class="p">:</span><span class="mh">0x00000201</span><span class="p">)</span>
+<span class="mi">3</span><span class="n">XMTHREADINFO2</span> <span class="p">(</span><span class="n">native</span> <span class="n">stack</span> <span class="n">address</span> <span class="n">range</span> <span class="nl">from</span><span class="p">:</span><span class="mh">0x9297E000</span><span class="p">,</span> <span class="nl">to</span><span class="p">:</span><span class="mh">0x929BF000</span><span class="p">,</span> <span class="nl">size</span><span class="p">:</span><span class="mh">0x41000</span><span class="p">)</span>
+<span class="mi">3</span><span class="n">XMCPUTIME</span> <span class="n">CPU</span> <span class="n">usage</span> <span class="nl">total</span><span class="p">:</span> <span class="mf">0.004365543</span> <span class="n">secs</span><span class="p">,</span> <span class="n">current</span> <span class="n">category</span><span class="o">=</span><span class="s">"Application"</span>
+<span class="mi">3</span><span class="n">XMTHREADBLOCK</span> <span class="n">Blocked</span> <span class="nl">on</span><span class="p">:</span> <span class="n">java</span><span class="o">/</span><span class="n">lang</span><span class="o">/</span><span class="n">Object</span><span class="mh">@0xB5666688</span> <span class="n">Owned</span> <span class="nl">by</span><span class="p">:</span> <span class="s">"Worker Thread 2"</span> <span class="p">(</span><span class="nl">J9VMThread</span><span class="p">:</span><span class="mh">0x94501D00</span><span class="p">,</span> <span class="n">java</span><span class="o">/</span><span class="n">lang</span><span class="o">/</span><span class="nl">Thread</span><span class="p">:</span><span class="mh">0xB56668D0</span><span class="p">)</span>
+<span class="mi">3</span><span class="n">XMHEAPALLOC</span> <span class="n">Heap</span> <span class="n">bytes</span> <span class="n">allocated</span> <span class="n">since</span> <span class="n">last</span> <span class="n">GC</span> <span class="n">cycle</span><span class="o">=</span><span class="mi">0</span> <span class="p">(</span><span class="mh">0x0</span><span class="p">)</span>
+<span class="mi">3</span><span class="n">XMTHREADINFO3</span> <span class="n">Java</span> <span class="nl">callstack</span><span class="p">:</span>
+<span class="mi">4</span><span class="n">XESTACKTRACE</span> <span class="n">at</span> <span class="n">WorkerThread</span><span class="p">.</span><span class="n">run</span><span class="p">(</span><span class="n">DeadLockTest</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">35</span><span class="p">)</span>
+<span class="mi">5</span><span class="n">XESTACKTRACE</span> <span class="p">(</span><span class="n">entered</span> <span class="nl">lock</span><span class="p">:</span> <span class="n">java</span><span class="o">/</span><span class="n">lang</span><span class="o">/</span><span class="n">Object</span><span class="mh">@0xB5666678</span><span class="p">,</span> <span class="n">entry</span> <span class="nl">count</span><span class="p">:</span> <span class="mi">1</span><span class="p">)</span>
+<span class="p">...</span>
+<span class="mi">3</span><span class="n">XMTHREADINFO</span> <span class="s">"Worker Thread 2"</span> <span class="nl">J9VMThread</span><span class="p">:</span><span class="mh">0x94501D00</span><span class="p">,</span> <span class="nl">omrthread_t</span><span class="p">:</span><span class="mh">0x92A3C8F0</span><span class="p">,</span> <span class="n">java</span><span class="o">/</span><span class="n">lang</span><span class="o">/</span><span class="nl">Thread</span><span class="p">:</span><span class="mh">0xB56668D0</span><span class="p">,</span> <span class="nl">state</span><span class="p">:</span><span class="n">B</span><span class="p">,</span> <span class="n">prio</span><span class="o">=</span><span class="mi">5</span>
+<span class="mi">3</span><span class="n">XMJAVALTHREAD</span> <span class="p">(</span><span class="n">java</span><span class="o">/</span><span class="n">lang</span><span class="o">/</span><span class="n">Thread</span> <span class="nl">getId</span><span class="p">:</span><span class="mh">0x14</span><span class="p">,</span> <span class="nl">isDaemon</span><span class="p">:</span><span class="nb">false</span><span class="p">)</span>
+<span class="mi">3</span><span class="n">XMTHREADINFO1</span> <span class="p">(</span><span class="n">native</span> <span class="kr">thread</span> <span class="nl">ID</span><span class="p">:</span><span class="mh">0x52D0</span><span class="p">,</span> <span class="n">native</span> <span class="nl">priority</span><span class="p">:</span><span class="mh">0x5</span><span class="p">,</span> <span class="n">native</span> <span class="nl">policy</span><span class="p">:</span><span class="n">UNKNOWN</span><span class="p">,</span> <span class="nl">vmstate</span><span class="p">:</span><span class="n">B</span><span class="p">,</span> <span class="n">vm</span> <span class="kr">thread</span> <span class="nl">flags</span><span class="p">:</span><span class="mh">0x00000201</span><span class="p">)</span>
+<span class="mi">3</span><span class="n">XMTHREADINFO2</span> <span class="p">(</span><span class="n">native</span> <span class="n">stack</span> <span class="n">address</span> <span class="n">range</span> <span class="nl">from</span><span class="p">:</span><span class="mh">0x946BF000</span><span class="p">,</span> <span class="nl">to</span><span class="p">:</span><span class="mh">0x94700000</span><span class="p">,</span> <span class="nl">size</span><span class="p">:</span><span class="mh">0x41000</span><span class="p">)</span>
+<span class="mi">3</span><span class="n">XMCPUTIME</span> <span class="n">CPU</span> <span class="n">usage</span> <span class="nl">total</span><span class="p">:</span> <span class="mf">0.004555580</span> <span class="n">secs</span><span class="p">,</span> <span class="n">current</span> <span class="n">category</span><span class="o">=</span><span class="s">"Application"</span>
+<span class="mi">3</span><span class="n">XMTHREADBLOCK</span> <span class="n">Blocked</span> <span class="nl">on</span><span class="p">:</span> <span class="n">java</span><span class="o">/</span><span class="n">lang</span><span class="o">/</span><span class="n">Object</span><span class="mh">@0xB5666698</span> <span class="n">Owned</span> <span class="nl">by</span><span class="p">:</span> <span class="s">"Worker Thread 3"</span> <span class="p">(</span><span class="nl">J9VMThread</span><span class="p">:</span><span class="mh">0x94507500</span><span class="p">,</span> <span class="n">java</span><span class="o">/</span><span class="n">lang</span><span class="o">/</span><span class="nl">Thread</span><span class="p">:</span><span class="mh">0xB5666A18</span><span class="p">)</span>
+<span class="mi">3</span><span class="n">XMHEAPALLOC</span> <span class="n">Heap</span> <span class="n">bytes</span> <span class="n">allocated</span> <span class="n">since</span> <span class="n">last</span> <span class="n">GC</span> <span class="n">cycle</span><span class="o">=</span><span class="mi">0</span> <span class="p">(</span><span class="mh">0x0</span><span class="p">)</span>
+<span class="mi">3</span><span class="n">XMTHREADINFO3</span> <span class="n">Java</span> <span class="nl">callstack</span><span class="p">:</span>
+<span class="mi">4</span><span class="n">XESTACKTRACE</span> <span class="n">at</span> <span class="n">WorkerThread</span><span class="p">.</span><span class="n">run</span><span class="p">(</span><span class="n">DeadLockTest</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">35</span><span class="p">)</span>
+<span class="mi">5</span><span class="n">XESTACKTRACE</span> <span class="p">(</span><span class="n">entered</span> <span class="nl">lock</span><span class="p">:</span> <span class="n">java</span><span class="o">/</span><span class="n">lang</span><span class="o">/</span><span class="n">Object</span><span class="mh">@0xB5666688</span><span class="p">,</span> <span class="n">entry</span> <span class="nl">count</span><span class="p">:</span> <span class="mi">1</span><span class="p">)</span>
+<span class="p">...</span>
+<span class="mi">3</span><span class="n">XMTHREADINFO</span> <span class="s">"Worker Thread 3"</span> <span class="nl">J9VMThread</span><span class="p">:</span><span class="mh">0x94507500</span><span class="p">,</span> <span class="nl">omrthread_t</span><span class="p">:</span><span class="mh">0x92A3CC10</span><span class="p">,</span> <span class="n">java</span><span class="o">/</span><span class="n">lang</span><span class="o">/</span><span class="nl">Thread</span><span class="p">:</span><span class="mh">0xB5666A18</span><span class="p">,</span> <span class="nl">state</span><span class="p">:</span><span class="n">B</span><span class="p">,</span> <span class="n">prio</span><span class="o">=</span><span class="mi">5</span>
+<span class="mi">3</span><span class="n">XMJAVALTHREAD</span> <span class="p">(</span><span class="n">java</span><span class="o">/</span><span class="n">lang</span><span class="o">/</span><span class="n">Thread</span> <span class="nl">getId</span><span class="p">:</span><span class="mh">0x15</span><span class="p">,</span> <span class="nl">isDaemon</span><span class="p">:</span><span class="nb">false</span><span class="p">)</span>
+<span class="mi">3</span><span class="n">XMTHREADINFO1</span> <span class="p">(</span><span class="n">native</span> <span class="kr">thread</span> <span class="nl">ID</span><span class="p">:</span><span class="mh">0x52D1</span><span class="p">,</span> <span class="n">native</span> <span class="nl">priority</span><span class="p">:</span><span class="mh">0x5</span><span class="p">,</span> <span class="n">native</span> <span class="nl">policy</span><span class="p">:</span><span class="n">UNKNOWN</span><span class="p">,</span> <span class="nl">vmstate</span><span class="p">:</span><span class="n">B</span><span class="p">,</span> <span class="n">vm</span> <span class="kr">thread</span> <span class="nl">flags</span><span class="p">:</span><span class="mh">0x00000201</span><span class="p">)</span>
+<span class="mi">3</span><span class="n">XMTHREADINFO2</span> <span class="p">(</span><span class="n">native</span> <span class="n">stack</span> <span class="n">address</span> <span class="n">range</span> <span class="nl">from</span><span class="p">:</span><span class="mh">0x9467E000</span><span class="p">,</span> <span class="nl">to</span><span class="p">:</span><span class="mh">0x946BF000</span><span class="p">,</span> <span class="nl">size</span><span class="p">:</span><span class="mh">0x41000</span><span class="p">)</span>
+<span class="mi">3</span><span class="n">XMCPUTIME</span> <span class="n">CPU</span> <span class="n">usage</span> <span class="nl">total</span><span class="p">:</span> <span class="mf">0.003657010</span> <span class="n">secs</span><span class="p">,</span> <span class="n">current</span> <span class="n">category</span><span class="o">=</span><span class="s">"Application"</span>
+<span class="mi">3</span><span class="n">XMTHREADBLOCK</span> <span class="n">Blocked</span> <span class="nl">on</span><span class="p">:</span> <span class="n">java</span><span class="o">/</span><span class="n">lang</span><span class="o">/</span><span class="n">Object</span><span class="mh">@0xB5666678</span> <span class="n">Owned</span> <span class="nl">by</span><span class="p">:</span> <span class="s">"Worker Thread 1"</span> <span class="p">(</span><span class="nl">J9VMThread</span><span class="p">:</span><span class="mh">0x92A3EC00</span><span class="p">,</span> <span class="n">java</span><span class="o">/</span><span class="n">lang</span><span class="o">/</span><span class="nl">Thread</span><span class="p">:</span><span class="mh">0xB5666778</span><span class="p">)</span>
+<span class="mi">3</span><span class="n">XMHEAPALLOC</span> <span class="n">Heap</span> <span class="n">bytes</span> <span class="n">allocated</span> <span class="n">since</span> <span class="n">last</span> <span class="n">GC</span> <span class="n">cycle</span><span class="o">=</span><span class="mi">0</span> <span class="p">(</span><span class="mh">0x0</span><span class="p">)</span>
+<span class="mi">3</span><span class="n">XMTHREADINFO3</span> <span class="n">Java</span> <span class="nl">callstack</span><span class="p">:</span>
+<span class="mi">4</span><span class="n">XESTACKTRACE</span> <span class="n">at</span> <span class="n">WorkerThread</span><span class="p">.</span><span class="n">run</span><span class="p">(</span><span class="n">DeadLockTest</span><span class="p">.</span><span class="nl">java</span><span class="p">:</span><span class="mi">35</span><span class="p">)</span>
+<span class="mi">5</span><span class="n">XESTACKTRACE</span> <span class="p">(</span><span class="n">entered</span> <span class="nl">lock</span><span class="p">:</span> <span class="n">java</span><span class="o">/</span><span class="n">lang</span><span class="o">/</span><span class="n">Object</span><span class="mh">@0xB5666698</span><span class="p">,</span> <span class="n">entry</span> <span class="nl">count</span><span class="p">:</span> <span class="mi">1</span><span class="p">)</span>
+</pre></div>
+
+
<!-- ==== END OF TOPIC ==== dump_javadump.md ==== -->
diff --git a/search/search_index.json b/search/search_index.json
index e60ab6a..e6b24cb 100644
--- a/search/search_index.json
+++ b/search/search_index.json
@@ -387,12 +387,12 @@
},
{
"location": "/dump_javadump/",
- "text": "Java dump\n\n\nJava dumps, sometimes referred to as \nJava cores\n, are produced when the VM ends unexpectedly because of an operating system signal, \nOutOfMemoryError\n, or a user-initiated keystroke combination. You can also generate a Java dump by calling the Dump API programmatically from your application or specifying the \n-Xdump:java\n option on the command line.\n\n\nIf your Java application crashes or hangs, Java dumps can provide useful information to help you diagnose the root cause.\n\n\n\n\nIf your application crashes, Javadumps are generated automatically for the following types of failure:\n\n\nthe VM receives an unexpected signal or an assertion failure\n\n\nthe VM runs out of memory\n\n\n\n\n\n\n\n\nIf your application hangs, you can trigger the generation of a Java dump by sending a SIGQUIT signal (\nkill -3\n) to the VM.\n\n\n \nNote:\n On Windows, if you started the VM in a console window you can force the VM to produce a Java dump in response to a SIGBREAK signal\n(Ctrl-Break keyboard combination). If you didn't start in a console window there is no equivalent to a Linux \nkill\n command on Windows for sending signals.\nThe only option here is to trigger a full system dump by finding the VM process in the \nProcesses\n tab of the Windows Task Manager and clicking\n\nCreate dump file\n.\n\n\n\n\n\n\nTo help you understand how a Java dump can help you with problem diagnosis, we've included a few scenarios at the end of the topic that might help you interpret the data:\n\n\n\n\nA crash caused by a general protection fault (gpf)\n\n\nA Java heap OutOfMemoryError (OOM)\n\n\n\n\nJava dump contents\n\n\nJava dumps summarize the state of the VM when the event occurs, with most of the information relating to components of the VM.\nThe file is made up of a number of sections that provide different types of information.\n\n\nTITLE\n\n\nThe first section of the Java dump file provides information about the event that triggered the production of the dump.\nIn the following example you can see that a \nvmstop\n event triggered the dump at a specified date and time.\n\n\n0SECTION TITLE subcomponent dump routine\nNULL ===============================\n1TICHARSET UTF-8\n1TISIGINFO Dump Event \"vmstop\" (00000002) Detail \"#0000000000000000\" received\n1TIDATETIME Date: 2018/08/30 at 21:55:47:607\n1TINANOTIME System nanotime: 22012355276134\n1TIFILENAME Javacore filename: /home/doc-javacore/javacore.20180830.215547.30285.0001.txt\n1TIREQFLAGS Request Flags: 0x81 (exclusive+preempt)\n1TIPREPSTATE Prep State: 0x106 (vm_access+exclusive_vm_access+trace_disabled)\n\n\n\n\n\nGPINFO\n\n\nThe GPINFO section provides general information about the system that the JVM is running on. The following example is taken\nfrom a Java dump that was generated on a Linux system.\n\n\nNULL ------------------------------------------------------------------------\n0SECTION GPINFO subcomponent dump routine\nNULL ================================\n2XHOSLEVEL OS Level : Linux 3.10.0-862.11.6.el7.x86_64\n2XHCPUS Processors -\n3XHCPUARCH Architecture : amd64\n3XHNUMCPUS How Many : 4\n3XHNUMASUP NUMA is either not supported or has been disabled by user\nNULL \n1XHERROR2 Register dump section only produced for SIGSEGV, SIGILL or SIGFPE.\nNULL \n\n\n\n\n\nThe content of this section can vary, depending on the cause of the dump. For example, if the dump was caused by a\ngeneral protection fault (gpf), the library in which the crash occurred is also recorded, together with a value shown\nas \nVM flags\n. This value can provide some clues about which component of the VM might have been involved. Look for the\nfollowing line in the output:\n\n\n1XHFLAGS VM flags:0000000000000000\n\n\n\n\n\nThe hexadecimal number recorded for \nVM flags\n ends in MSSSS, where M is the VM component and SSSS is component-specific code as shown in the following table:\n\n\n\n\n\n\n\n\nComponent\n\n\nCode value\n\n\n\n\n\n\n\n\n\n\nINTERPRETER\n\n\n0x10000\n\n\n\n\n\n\nGC\n\n\n0x20000\n\n\n\n\n\n\nGROW_STACK\n\n\n0x30000\n\n\n\n\n\n\nJNI\n\n\n0x40000\n\n\n\n\n\n\nJIT_CODEGEN\n\n\n0x50000\n\n\n\n\n\n\nBCVERIFY\n\n\n0x60000\n\n\n\n\n\n\nRTVERIFY\n\n\n0x70000\n\n\n\n\n\n\nSHAREDCLASSES\n\n\n0x80000\n\n\n\n\n\n\n\n\nA value of \n0000000000000000\n (0x00000) indicates that a crash occurred outside of the VM.\n\n\nENVINFO\n\n\nThis section contains useful information about the environment in which the crash took place, including the following data:\n\n\n\n\nJava version (\n1CIJAVAVERSION\n)\n\n\nOpenJ9 VM and subcomponent version information (\n1CIVMVERSION\n, \n1CIJ9VMVERSION\n, \n1CIJITVERSION\n, \n1CIOMRVERSION\n, \n1CIJCLVERSION\n)\n\n\nVM start time (\n1CISTARTTIME\n) and process information (\n1CIPROCESSID\n)\n\n\nJava home (\n1CIJAVAHOMEDIR\n) and DLL (\n1CIJAVADLLDIR\n) directories\n\n\nUser arguments passed on the command line (\n1CIUSERARG\n)\n\n\nUser limits imposed by the system (\n1CIUSERLIMITS\n)\n\n\nEnvironment variables in place (\n1CIENVVARS\n)\n\n\nSystem information (\n1CISYSINFO\n)\n\n\nCPU information (\n1CICPUINFO\n)\n\n\n\n\n\nFor clarity, the following example shows a shortened version of this section, where \n...\n indicates that lines are removed:\n\n\nNULL\n \n------------------------------------------------------------------------\n\n\n0\nSECTION\n \nENVINFO\n \nsubcomponent\n \ndump\n \nroutine\n\n\nNULL\n \n=================================\n\n\n1\nCIJAVAVERSION\n \nJRE\n \n9\n \nLinux\n \namd64\n-\n64\n \n(\nbuild\n \n9.0.4\n-\ninternal\n+\n0\n-\nadhoc\n..\nopenj9\n-\nopenjdk\n-\njdk9\n)\n\n\n1\nCIVMVERSION\n \n20180830\n_000000\n\n\n1\nCIJ9VMVERSION\n \n8e7\nc6ec\n\n\n1\nCIJITVERSION\n \n8e7\nc6ec\n\n\n1\nCIOMRVERSION\n \n553811\nb_CMPRSS\n\n\n1\nCIJCLVERSION\n \nec1d223\n \nbased\n \non\n \njdk\n-\n9.0.4\n+\n12\n\n\n1\nCIJITMODES\n \nJIT\n \nenabled\n,\n \nAOT\n \nenabled\n,\n \nFSD\n \ndisabled\n,\n \nHCR\n \nenabled\n\n\n1\nCIRUNNINGAS\n \nRunning\n \nas\n \na\n \nstandalone\n \nJVM\n\n\n1\nCIVMIDLESTATE\n \nVM\n \nIdle\n \nState\n:\n \nACTIVE\n\n\n1\nCISTARTTIME\n \nJVM\n \nstart\n \ntime\n:\n \n2018\n/\n08\n/\n30\n \nat\n \n21\n:\n55\n:\n47\n:\n387\n\n\n1\nCISTARTNANO\n \nJVM\n \nstart\n \nnanotime\n:\n \n22012135233549\n\n\n1\nCIPROCESSID\n \nProcess\n \nID\n:\n \n30285\n \n(\n0x764D\n)\n\n\n1\nCICMDLINE\n \n[\nnot\n \navailable\n]\n\n\n1\nCIJAVAHOMEDIR\n \nJava\n \nHome\n \nDir\n:\n \n/\nhome\n/\nme\n/\nopenj9\n-\nopenjdk\n-\njdk9\n/\nbuild\n/\nlinux\n-\nx86_64\n-\nnormal\n-\nserver\n-\nrelease\n/\nimages\n/\njdk\n\n\n1\nCIJAVADLLDIR\n \nJava\n \nDLL\n \nDir\n:\n \n/\nhome\n/\nme\n/\nopenj9\n-\nopenjdk\n-\njdk9\n/\nbuild\n/\nlinux\n-\nx86_64\n-\nnormal\n-\nserver\n-\nrelease\n/\nimages\n/\njdk\n/\nbin\n\n\n1\nCISYSCP\n \nSys\n \nClasspath\n:\n \n\n1\nCIUSERARGS\n \nUserArgs\n:\n\n\n2\nCIUSERARG\n \n-\nXoptionsfile\n=/\nhome\n/\nme\n/\nopenj9\n-\nopenjdk\n-\njdk9\n/\nbuild\n/\nlinux\n-\nx86_64\n-\nnormal\n-\nserver\n-\nrelease\n/\nimages\n/\njdk\n/\nlib\n/\noptions\n.\ndefault\n\n\n...\n\n\nNULL\n\n\n1\nCIUSERLIMITS\n \nUser\n \nLimits\n \n(\nin\n \nbytes\n \nexcept\n \nfor\n \nNOFILE\n \nand\n \nNPROC\n)\n\n\nNULL\n \n------------------------------------------------------------------------\n\n\nNULL\n \ntype\n \nsoft\n \nlimit\n \nhard\n \nlimit\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_AS\n \nunlimited\n \nunlimited\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_CORE\n \n0\n \nunlimited\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_CPU\n \nunlimited\n \nunlimited\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_DATA\n \nunlimited\n \nunlimited\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_FSIZE\n \nunlimited\n \nunlimited\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_LOCKS\n \nunlimited\n \nunlimited\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_MEMLOCK\n \n65536\n \n65536\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_NOFILE\n \n4096\n \n4096\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_NPROC\n \n4096\n \n30592\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_RSS\n \nunlimited\n \nunlimited\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_STACK\n \n8388608\n \nunlimited\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_MSGQUEUE\n \n819200\n \n819200\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_NICE\n \n0\n \n0\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_RTPRIO\n \n0\n \n0\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_SIGPENDING\n \n30592\n \n30592\n\n\nNULL\n\n\n1\nCIENVVARS\n \nEnvironment\n \nVariables\n\n\nNULL\n \n------------------------------------------------------------------------\n\n\n2\nCIENVVAR\n \nXDG_VTNR\n=\n1\n\n\n2\nCIENVVAR\n \nSSH_AGENT_PID\n=\n2653\n\n\n...\n\n\nNULL\n \n\n1\nCISYSINFO\n \nSystem\n \nInformation\n\n\nNULL\n \n------------------------------------------------------------------------\n\n\n2\nCISYSINFO\n \n/\nproc\n/\nsys\n/\nkernel\n/\ncore_pattern\n \n=\n \ncore\n\n\n2\nCISYSINFO\n \n/\nproc\n/\nsys\n/\nkernel\n/\ncore_uses_pid\n \n=\n \n1\n\n\nNULL\n \n\n1\nCICPUINFO\n \nCPU\n \nInformation\n\n\nNULL\n \n------------------------------------------------------------------------\n\n\n2\nCIPHYSCPU\n \nPhysical\n \nCPUs\n:\n \n4\n\n\n2\nCIONLNCPU\n \nOnline\n \nCPUs\n:\n \n4\n\n\n2\nCIBOUNDCPU\n \nBound\n \nCPUs\n:\n \n4\n\n\n2\nCIACTIVECPU\n \nActive\n \nCPUs\n:\n \n0\n\n\n2\nCITARGETCPU\n \nTarget\n \nCPUs\n:\n \n4\n\n\n\n\n\n\n\n\n\nNATIVEMEMINFO\n\n\nThis section records information about native memory that is requested by using library functions such as \nmalloc()\n and \nmmap()\n.\nValues are provided as a breakdown, per component, indicating the total number of bytes allocated and the number of native memory allocations.\nIn the following example, 4,682,840 bytes of native memory are allocated (but not yet freed) to VM Classes, which corresponds to 141 allocations.\n\n\nNULL ------------------------------------------------------------------------\n0SECTION NATIVEMEMINFO subcomponent dump routine\nNULL =================================\n0MEMUSER\n1MEMUSER JRE: 2,569,088,312 bytes / 4653 allocations\n1MEMUSER |\n2MEMUSER +--VM: 2,280,088,336 bytes / 2423 allocations\n2MEMUSER | |\n3MEMUSER | +--Classes: 4,682,840 bytes / 141 allocations\n2MEMUSER | |\n3MEMUSER | +--Memory Manager (GC): 2,054,966,784 bytes / 433 allocations\n3MEMUSER | | |\n4MEMUSER | | +--Java Heap: 2,014,113,792 bytes / 1 allocation\n3MEMUSER | | |\n4MEMUSER | | +--Other: 40,852,992 bytes / 432 allocations\n2MEMUSER | |\n3MEMUSER | +--Threads: 10,970,016 bytes / 156 allocations\n3MEMUSER | | |\n4MEMUSER | | +--Java Stack: 197,760 bytes / 16 allocations\n3MEMUSER | | |\n4MEMUSER | | +--Native Stack: 10,616,832 bytes / 17 allocations\n3MEMUSER | | |\n4MEMUSER | | +--Other: 155,424 bytes / 123 allocations\n2MEMUSER | |\n3MEMUSER | +--Trace: 180,056 bytes / 263 allocations\n2MEMUSER | |\n3MEMUSER | +--JVMTI: 17,776 bytes / 13 allocations\n2MEMUSER | |\n3MEMUSER | +--JNI: 36,184 bytes / 52 allocations\n2MEMUSER | |\n3MEMUSER | +--Port Library: 208,179,632 bytes / 72 allocations\n3MEMUSER | | |\n4MEMUSER | | +--Unused <32bit allocation regions: 208,168,752 bytes / 1 allocation\n3MEMUSER | | |\n4MEMUSER | | +--Other: 10,880 bytes / 71 allocations\n2MEMUSER | |\n3MEMUSER | +--Other: 1,055,048 bytes / 1293 allocations\n1MEMUSER |\n2MEMUSER +--JIT: 288,472,816 bytes / 140 allocations\n2MEMUSER | |\n3MEMUSER | +--JIT Code Cache: 268,435,456 bytes / 1 allocation\n2MEMUSER | |\n3MEMUSER | +--JIT Data Cache: 2,097,216 bytes / 1 allocation\n2MEMUSER | |\n3MEMUSER | +--Other: 17,940,144 bytes / 138 allocations\n1MEMUSER |\n2MEMUSER +--Class Libraries: 13,432 bytes / 25 allocations\n2MEMUSER | |\n3MEMUSER | +--VM Class Libraries: 13,432 bytes / 25 allocations\n3MEMUSER | | |\n4MEMUSER | | +--sun.misc.Unsafe: 3,184 bytes / 13 allocations\n4MEMUSER | | | |\n5MEMUSER | | | +--Direct Byte Buffers: 1,056 bytes / 12 allocations\n4MEMUSER | | | |\n5MEMUSER | | | +--Other: 2,128 bytes / 1 allocation\n3MEMUSER | | |\n4MEMUSER | | +--Other: 10,248 bytes / 12 allocations\n1MEMUSER |\n2MEMUSER +--Unknown: 513,728 bytes / 2065 allocations\nNULL \n\n\n\n\n\nThis section does not record memory that is allocated by application or JNI code and is typically a little less than the\nvalue recorded by operating system tools.\n\n\nMEMINFO\n\n\nThis section relates to memory management, providing a breakdown of memory usage in the VM for the object heap,\ninternal memory, memory used for classes, the JIT code cache, and JIT data cache in decimal and hexadecimal format.\nYou can also find out which garbage collection policy is in use when the dump is produced. \n\n\nThe object memory area (\n1STHEAPTYPE\n) records each memory region in use, its start and end address, and region size.\nFurther information is recorded about the memory segments used for internal memory, class memory, the JIT code cache and JIT data cache (\n1STSEGMENT\n).\nThis information includes the address of the segment control data structure, the start and end address of the native memory segment, as well as\nthe segment size.\n\n\nFor clarity, the following example shows a shortened version of this section, where \n...\n indicates that lines are removed:\n\n\nNULL ------------------------------------------------------------------------\n0SECTION MEMINFO subcomponent dump routine\nNULL =================================\nNULL \n1STHEAPTYPE Object Memory\nNULL id start end size space/region\n1STHEAPSPACE 0x00007FF4F00744A0 -- -- -- Generational\n1STHEAPREGION 0x00007FF4F0074CE0 0x0000000087F40000 0x0000000088540000 0x0000000000600000 Generational/Tenured Region\n1STHEAPREGION 0x00007FF4F0074930 0x00000000FFE00000 0x00000000FFF00000 0x0000000000100000 Generational/Nursery Region\n1STHEAPREGION 0x00007FF4F0074580 0x00000000FFF00000 0x0000000100000000 0x0000000000100000 Generational/Nursery Region\nNULL\n1STHEAPTOTAL Total memory: 8388608 (0x0000000000800000)\n1STHEAPINUSE Total memory in use: 2030408 (0x00000000001EFB48)\n1STHEAPFREE Total memory free: 6358200 (0x00000000006104B8)\nNULL\n1STSEGTYPE Internal Memory\nNULL segment start alloc end type size\n1STSEGMENT 0x00007FF4F004CBC8 0x00007FF4CD33C000 0x00007FF4CD33C000 0x00007FF4CE33C000 0x01000440 0x0000000001000000\n1STSEGMENT 0x00007FF4F004CB08 0x00007FF4DE43D030 0x00007FF4DE517770 0x00007FF4DE53D030 0x00800040 0x0000000000100000\nNULL\n1STSEGTOTAL Total memory: 17825792 (0x0000000001100000)\n1STSEGINUSE Total memory in use: 894784 (0x00000000000DA740)\n1STSEGFREE Total memory free: 16931008 (0x00000000010258C0)\nNULL \n1STSEGTYPE Class Memory\nNULL segment start alloc end type size\n1STSEGMENT 0x00007FF4F03B5638 0x0000000001053D98 0x000000000105BD98 0x000000000105BD98 0x00010040 0x0000000000008000\n1STSEGMENT 0x00007FF4F03B5578 0x0000000001048188 0x0000000001050188 0x0000000001050188 0x00010040 0x0000000000008000\n...\nNULL\n1STSEGTOTAL Total memory: 3512520 (0x00000000003598C8)\n1STSEGINUSE Total memory in use: 3433944 (0x00000000003465D8)\n1STSEGFREE Total memory free: 78576 (0x00000000000132F0)\nNULL \n1STSEGTYPE JIT Code Cache\nNULL segment start alloc end type size\n1STSEGMENT 0x00007FF4F00961F8 0x00007FF4CE43D000 0x00007FF4CE445790 0x00007FF4DE43D000 0x00000068 0x0000000010000000\nNULL\n1STSEGTOTAL Total memory: 268435456 (0x0000000010000000)\n1STSEGINUSE Total memory in use: 34704 (0x0000000000008790)\n1STSEGFREE Total memory free: 268400752 (0x000000000FFF7870)\n1STSEGLIMIT Allocation limit: 268435456 (0x0000000010000000)\nNULL \n1STSEGTYPE JIT Data Cache\nNULL segment start alloc end type size\n1STSEGMENT 0x00007FF4F0096668 0x00007FF4CC553030 0x00007FF4CC753030 0x00007FF4CC753030 0x00000048 0x0000000000200000\nNULL\n1STSEGTOTAL Total memory: 2097152 (0x0000000000200000)\n1STSEGINUSE Total memory in use: 2097152 (0x0000000000200000)\n1STSEGFREE Total memory free: 0 (0x0000000000000000)\n1STSEGLIMIT Allocation limit: 402653184 (0x0000000018000000)\nNULL \n1STGCHTYPE GC History \nNULL \n\n\n\n\n\nIn the example, the GC History (\n1STGCHTYPE\n) section is blank. This section is populated if a garbage collection cycle occurred in\na VM that is being diagnosed with the trace facility.\n\n\nLOCKS\n\n\nThis section of the Java dump provides information about locks, which protect shared resources from being accessed by more than one entity at a time. The information is essential in a deadlock situation, where two threads attempt to synchronize on an object and lock an instance of a class. Precise information is recorded about the threads that are causing the problem, which enables you to identify the root cause.\n\n\nThe following example shows a typical LOCKS section, where no deadlocks existed at the time the dump was triggered. For clarity, the following example shows a shortened version of this section, where \n...\n indicates that lines are removed:\n\n\nNULL\n \n------------------------------------------------------------------------\n\n\n0\nSECTION\n \nLOCKS\n \nsubcomponent\n \ndump\n \nroutine\n\n\nNULL\n \n===============================\n\n\nNULL\n \n\n1L\nKPOOLINFO\n \nMonitor\n \npool\n \ninfo\n:\n\n\n2L\nKPOOLTOTAL\n \nCurrent\n \ntotal\n \nnumber\n \nof\n \nmonitors\n:\n \n3\n\n\nNULL\n \n\n1L\nKMONPOOLDUMP\n \nMonitor\n \nPool\n \nDump\n \n(\nflat\n \n&\n \ninflated\n \nobject\n-\nmonitors\n)\n:\n\n\n2L\nKMONINUSE\n \nsys_mon_t\n:\n0x00007FF4B0001D78\n \ninfl_mon_t\n:\n \n0x00007FF4B0001DF8\n:\n\n\n3L\nKMONOBJECT\n \njava\n/\nlang\n/\nref\n/\nReferenceQueue\n@0x00000000FFE26A10\n:\n \n<\nunowned\n>\n\n\n3L\nKNOTIFYQ\n \nWaiting\n \nto\n \nbe\n \nnotified\n:\n\n\n3L\nKWAITNOTIFY\n \n\"Common-Cleaner\"\n \n(\nJ9VMThread\n:\n0x0000000000FD0100\n)\n\n\nNULL\n \n\n1L\nKREGMONDUMP\n \nJVM\n \nSystem\n \nMonitor\n \nDump\n \n(\nregistered\n \nmonitors\n)\n:\n\n\n2L\nKREGMON\n \nThread\n \nglobal\n \nlock\n \n(\n0x00007FF4F0004FE8\n)\n:\n \n<\nunowned\n>\n\n\n2L\nKREGMON\n \n&\n(\nPPG_mem_mem32_subAllocHeapMem32\n.\nmonitor\n)\n \nlock\n \n(\n0x00007FF4F0005098\n)\n:\n \n<\nunowned\n>\n\n\n2L\nKREGMON\n \nNLS\n \nhash\n \ntable\n \nlock\n \n(\n0x00007FF4F0005148\n)\n:\n \n<\nunowned\n>\n\n\n...\n\n\nNULL\n \n\n\n\n\n\nTHREADS\n\n\nThe THREADS section of a Java dump file provides summary information about the VM thread pool and detailed information about Java threads, native threads, and stack traces. Understanding the content of this section can help you diagnose problems that are caused by blocked or waiting threads.\n\n\nA Java thread runs on a native thread. Several lines are recorded for each Java thread in the \nThread Details\n subsection, which include the following key pieces of information:\n\n\n\n\n3XMTHREADINFO\n: The thread name, address information for the VM thread structures and Java thread object, the thread state, and thread priority.\n\n\n3XMJAVALTHREAD\n: The Java thread ID and daemon status from the thread object.\n\n\n3XMTHREADINFO1\n: The native operating system thread ID, priority, scheduling policy, internal VM thread state, and VM thread flags.\n\n\n3XMTHREADINFO2\n: The native stack address range.\n\n\n3XMTHREADINFO3\n: Java callstack information (\n4XESTACKTRACE\n) or Native call stack information (\n4XENATIVESTACK\n).\n\n\n5XESTACKTRACE\n: This line indicates whether locks were taken by a specific method.\n\n\n\n\nJava thread priorities are mapped to operating system priority values. Thread states are shown in the following table:\n\n\n\n\n\n\n\n\nThread state value\n\n\nStatus\n\n\nDescription\n\n\n\n\n\n\n\n\n\n\nR\n\n\nRunnable\n\n\nThe thread is able to run\n\n\n\n\n\n\nCW\n\n\nCondition Wait\n\n\nThe thread is waiting\n\n\n\n\n\n\nS\n\n\nSuspended\n\n\nThe thread is suspended by another thread\n\n\n\n\n\n\nZ\n\n\nZombie\n\n\nThe thread is destroyed\n\n\n\n\n\n\nP\n\n\nParked\n\n\nThe thread is parked by \njava.util.concurrent\n\n\n\n\n\n\nB\n\n\nBlocked\n\n\nThe thread is waiting to obtain a lock\n\n\n\n\n\n\n\n\nFor threads that are parked (P), blocked (B), or waiting (CW), an additional line (\n3XMTHREADBLOCK\n) is included in the output that shows what the thread is parked on, blocked on, or waiting for.\n\n\nFor clarity, the following example shows a shortened version of a typical THREADS section, where \n...\n indicates that lines are removed:\n\n\nNULL\n \n------------------------------------------------------------------------\n\n\n0\nSECTION\n \nTHREADS\n \nsubcomponent\n \ndump\n \nroutine\n\n\nNULL\n \n=================================\n\n\nNULL\n\n\n1\nXMPOOLINFO\n \nJVM\n \nThread\n \npool\n \ninfo\n:\n\n\n2\nXMPOOLTOTAL\n \nCurrent\n \ntotal\n \nnumber\n \nof\n \npooled\n \nthreads\n:\n \n18\n\n\n2\nXMPOOLLIVE\n \nCurrent\n \ntotal\n \nnumber\n \nof\n \nlive\n \nthreads\n:\n \n16\n\n\n2\nXMPOOLDAEMON\n \nCurrent\n \ntotal\n \nnumber\n \nof\n \nlive\n \ndaemon\n \nthreads\n:\n \n15\n\n\nNULL\n \n\n1\nXMTHDINFO\n \nThread\n \nDetails\n\n\nNULL\n \n\n3\nXMTHREADINFO\n \n\"JIT Diagnostic Compilation Thread-7 Suspended\"\n \nJ9VMThread\n:\n0x0000000000EFC500\n,\n \nomrthread_t:\n0x00007FF4F00A77E8\n,\n \njava\n/\nlang\n/\nThread\n:\n0x00000000FFE97480\n,\n \nstate\n:\nR\n,\n \nprio\n=\n10\n\n\n3\nXMJAVALTHREAD\n \n(\njava\n/\nlang\n/\nThread\n \ngetId:\n0xA\n,\n \nisDaemon:true\n)\n\n\n3\nXMTHREADINFO1\n \n(\nnative\n \nthread\n \nID\n:\n0x7657\n,\n \nnative\n \npriority\n:\n0xB\n,\n \nnative\n \npolicy\n:\nUNKNOWN\n,\n \nvmstate\n:\nCW\n,\n \nvm\n \nthread\n \nflags\n:\n0x00000081\n)\n\n\n3\nXMTHREADINFO2\n \n(\nnative\n \nstack\n \naddress\n \nrange\n \nfrom\n:\n0x00007FF4CCC36000\n,\n \nto\n:\n0x00007FF4CCD36000\n,\n \nsize\n:\n0x100000\n)\n\n\n3\nXMCPUTIME\n \nCPU\n \nusage\n \ntotal\n:\n \n0.000037663\n \nsecs\n,\n \ncurrent\n \ncategory=\n\"JIT\"\n\n\n3\nXMHEAPALLOC\n \nHeap\n \nbytes\n \nallocated\n \nsince\n \nlast\n \nGC\n \ncycle\n=\n0\n \n(\n0x0\n)\n\n\n3\nXMTHREADINFO3\n \nNo\n \nJava\n \ncallstack\n \nassociated\n \nwith\n \nthis\n \nthread\n\n\n3\nXMTHREADINFO3\n \nNo\n \nnative\n \ncallstack\n \navailable\n \nfor\n \nthis\n \nthread\n\n\nNULL\n\n\n...\n\n\n3\nXMTHREADINFO\n \n\"Common-Cleaner\"\n \nJ9VMThread\n:\n0x0000000000FD0100\n,\n \nomrthread_t:\n0x00007FF4F022A520\n,\n \njava\n/\nlang\n/\nThread\n:\n0x00000000FFE26F40\n,\n \nstate\n:\nCW\n,\n \nprio\n=\n8\n\n\n3\nXMJAVALTHREAD\n \n(\njava\n/\nlang\n/\nThread\n \ngetId:\n0x2\n,\n \nisDaemon:true\n)\n\n\n3\nXMTHREADINFO1\n \n(\nnative\n \nthread\n \nID\n:\n0x765A\n,\n \nnative\n \npriority\n:\n0x8\n,\n \nnative\n \npolicy\n:\nUNKNOWN\n,\n \nvmstate\n:\nCW\n,\n \nvm\n \nthread\n \nflags\n:\n0x00080181\n)\n\n\n3\nXMTHREADINFO2\n \n(\nnative\n \nstack\n \naddress\n \nrange\n \nfrom\n:\n0x00007FF4CC0B8000\n,\n \nto\n:\n0x00007FF4CC0F8000\n,\n \nsize\n:\n0x40000\n)\n\n\n3\nXMCPUTIME\n \nCPU\n \nusage\n \ntotal\n:\n \n0.000150926\n \nsecs\n,\n \ncurrent\n \ncategory=\n\"Application\"\n\n\n3\nXMTHREADBLOCK\n \nWaiting\n \non\n:\n \njava\n/\nlang\n/\nref\n/\nReferenceQueue\n@\n0x00000000FFE26A10\n \nOwned\n \nby\n:\n \n<\nunowned\n>\n\n\n3\nXMHEAPALLOC\n \nHeap\n \nbytes\n \nallocated\n \nsince\n \nlast\n \nGC\n \ncycle\n=\n0\n \n(\n0x0\n)\n\n\n3\nXMTHREADINFO3\n \nJava\n \ncallstack\n:\n\n\n4\nXESTACKTRACE\n \nat\n \njava\n/\nlang\n/\nObject\n.\nwait\n(\nNative\n \nMethod\n)\n\n\n4\nXESTACKTRACE\n \nat\n \njava\n/\nlang\n/\nObject\n.\nwait\n(\nObject\n.\njava\n:\n221\n)\n\n\n4\nXESTACKTRACE\n \nat\n \njava\n/\nlang\n/\nref\n/\nReferenceQueue\n.\nremove\n(\nReferenceQueue\n.\njava\n:\n138\n)\n\n\n5\nXESTACKTRACE\n \n(\nentered\n \nlock\n:\n \njava\n/\nlang\n/\nref\n/\nReferenceQueue\n@\n0x00000000FFE26A10\n,\n \nentry\n \ncount\n:\n \n1\n)\n\n\n4\nXESTACKTRACE\n \nat\n \njdk\n/\ninternal\n/\nref\n/\nCleanerImpl\n.\nrun\n(\nCleanerImpl\n.\njava\n:\n148\n)\n\n\n4\nXESTACKTRACE\n \nat\n \njava\n/\nlang\n/\nThread\n.\nrun\n(\nThread\n.\njava\n:\n835\n)\n\n\n4\nXESTACKTRACE\n \nat\n \njdk\n/\ninternal\n/\nmisc\n/\nInnocuousThread\n.\nrun\n(\nInnocuousThread\n.\njava\n:\n122\n)\n\n\n3\nXMTHREADINFO3\n \nNo\n \nnative\n \ncallstack\n \navailable\n \nfor\n \nthis\n \nthread\n\n\nNULL\n\n\nNULL\n\n\n3\nXMTHREADINFO\n \n\"IProfiler\"\n \nJ9VMThread\n:\n0x0000000000F03D00\n,\n \nomrthread_t:\n0x00007FF4F00B06F8\n,\n \njava\n/\nlang\n/\nThread\n:\n0x00000000FFE97B60\n,\n \nstate\n:\nR\n,\n \nprio\n=\n5\n\n\n3\nXMJAVALTHREAD\n \n(\njava\n/\nlang\n/\nThread\n \ngetId:\n0xC\n,\n \nisDaemon:true\n)\n\n\n3\nXMTHREADINFO1\n \n(\nnative\n \nthread\n \nID\n:\n0x7659\n,\n \nnative\n \npriority\n:\n0x5\n,\n \nnative\n \npolicy\n:\nUNKNOWN\n,\n \nvmstate\n:\nCW\n,\n \nvm\n \nthread\n \nflags\n:\n0x00000081\n)\n\n\n3\nXMTHREADINFO2\n \n(\nnative\n \nstack\n \naddress\n \nrange\n \nfrom\n:\n0x00007FF4F8940000\n,\n \nto\n:\n0x00007FF4F8960000\n,\n \nsize\n:\n0x20000\n)\n\n\n3\nXMCPUTIME\n \nCPU\n \nusage\n \ntotal\n:\n \n0.004753103\n \nsecs\n,\n \ncurrent\n \ncategory=\n\"JIT\"\n\n\n3\nXMHEAPALLOC\n \nHeap\n \nbytes\n \nallocated\n \nsince\n \nlast\n \nGC\n \ncycle\n=\n0\n \n(\n0x0\n)\n\n\n3\nXMTHREADINFO3\n \nNo\n \nJava\n \ncallstack\n \nassociated\n \nwith\n \nthis\n \nthread\n\n\n3\nXMTHREADINFO3\n \nNo\n \nnative\n \ncallstack\n \navailable\n \nfor\n \nthis\n \nthread\n\n\nNULL\n\n\n...\n\n\n1\nXMWLKTHDERR\n \nThe\n \nfollowing\n \nwas\n \nreported\n \nwhile\n \ncollecting\n \nnative\n \nstacks\n:\n\n\n2\nXMWLKTHDERR\n \nunable\n \nto\n \ncount\n \nthreads\n(\n3\n,\n \n-\n2\n)\n\n\nNULL\n\n\n1\nXMTHDSUMMARY\n \nThreads\n \nCPU\n \nUsage\n \nSummary\n\n\nNULL\n \n=========================\n\n\nNULL\n\n\n1\nXMTHDCATINFO\n \nWarning\n:\n \nto\n \nget\n \nmore\n \naccurate\n \nCPU\n \ntimes\n \nfor\n \nthe\n \nGC\n,\n \nthe\n \noption\n \n-\nXX\n:-\nReduceCPUMonitorOverhead\n \ncan\n \nbe\n \nused\n.\n \nSee\n \nthe\n \nuser\n \nguide\n \nfor\n \nmore\n \ninformation\n.\n\n\nNULL\n\n\n1\nXMTHDCATEGORY\n \nAll\n \nJVM\n \nattached\n \nthreads\n:\n \n0.280083000\n \nsecs\n\n\n1\nXMTHDCATEGORY\n \n|\n\n\n2\nXMTHDCATEGORY\n \n+--\nSystem\n-\nJVM\n:\n \n0.270814000\n \nsecs\n\n\n2\nXMTHDCATEGORY\n \n|\n \n|\n\n\n3\nXMTHDCATEGORY\n \n|\n \n+--\nGC\n:\n \n0.000599000\n \nsecs\n\n\n2\nXMTHDCATEGORY\n \n|\n \n|\n\n\n3\nXMTHDCATEGORY\n \n|\n \n+--\nJIT\n:\n \n0.071904000\n \nsecs\n\n\n1\nXMTHDCATEGORY\n \n|\n\n\n2\nXMTHDCATEGORY\n \n+--\nApplication\n:\n \n0.009269000\n \nsecs\n\n\nNULL\n\n\n\n\n\n\nHOOKS\n\n\nThis section shows internal VM event callbacks, which are used for diagnosing performance problems in the VM. Multiple hook interfaces are listed, which include their individual hook events. The following example shows data for the \nJ9VMHookInterface\n, including the call site location (source file:line number), start time, and duration of the last callback and the longest callback.\n\n\nNULL ------------------------------------------------------------------------\n0SECTION HOOK subcomponent dump routine\nNULL ==============================\n1HKINTERFACE MM_OMRHookInterface\nNULL ------------------------------------------------------------------------\n1HKINTERFACE MM_PrivateHookInterface\nNULL ------------------------------------------------------------------------\n1HKINTERFACE MM_HookInterface\nNULL ------------------------------------------------------------------------\n1HKINTERFACE J9VMHookInterface\nNULL ------------------------------------------------------------------------\n2HKEVENTID 1\n3HKCALLCOUNT 18\n3HKLAST Last Callback\n4HKCALLSITE trcengine.c:392\n4HKSTARTTIME Start Time: 2018-08-30T21:55:47.601\n4HKDURATION DurationMs: 0\n3HKLONGST Longest Callback\n4HKCALLSITE trcengine.c:392\n4HKSTARTTIME Start Time: 2018-08-30T21:55:47.460\n4HKDURATION DurationMs: 1\nNULL\n...\n1HKINTERFACE J9VMZipCachePoolHookInterface\nNULL ------------------------------------------------------------------------\n1HKINTERFACE J9JITHookInterface\nNULL ------------------------------------------------------------------------\n2HKEVENTID 3\n3HKCALLCOUNT 65\n3HKLAST Last Callback\n4HKCALLSITE ../common/mgmtinit.c:191\n4HKSTARTTIME Start Time: 2018-08-30T21:55:47.601\n4HKDURATION DurationMs: 0\n3HKLONGST Longest Callback\n4HKCALLSITE ../common/mgmtinit.c:191\n4HKSTARTTIME Start Time: 2018-08-30T21:55:47.486\n4HKDURATION DurationMs: 0\n...\nNULL\n\n\n\n\n\nSHARED CLASSES\n\n\nIf the shared classes cache is enabled at run time, the information provided in a Java dump file describes settings that were used when creating the cache, together with summary information about the size and content of the cache.\n\n\nIn the following example, the shared classes cache was created with a Class Debug Area (\n-Xnolinenumbers=false\n). Byte code instrumentation (BCI) is enabled, which is the default, and VMs sharing the cache are allowed to store classpaths, which is also the default.\n\n\nThe \nCache Summary\n shows a cache size (\n2SCLTEXTCSZ\n) of 16776608 bytes, with a soft maximum size (\n2SCLTEXTSMB\n) also of 16776608 bytes, which leaves 12691668 bytes of free space (\n2SCLTEXTFRB\n). The size of the Class Debug Area (\n2SCLTEXTDAS\n) is 1331200 bytes and only 11% of this space is used.\n\n\nIn the \nCache Memory Status\n subsection, the line \n2SCLTEXTCMDT\n indicates the name and location of the shared cache and \ncr\n indicates that the cache is a 64-bit compressed references cache.\n\n\nNULL ------------------------------------------------------------------------\n0SECTION SHARED CLASSES subcomponent dump routine\nNULL ========================================\nNULL\n1SCLTEXTCRTW Cache Created With\nNULL ------------------\nNULL\n2SCLTEXTXNL -Xnolinenumbers = false\n2SCLTEXTBCI BCI Enabled = true\n2SCLTEXTBCI Restrict Classpaths = false\nNULL\n1SCLTEXTCSUM Cache Summary\nNULL ------------------\nNULL\n2SCLTEXTNLC No line number content = false\n2SCLTEXTLNC Line number content = true\nNULL\n2SCLTEXTRCS ROMClass start address = 0x00007F423061C000\n2SCLTEXTRCE ROMClass end address = 0x00007F42307B9A28\n2SCLTEXTMSA Metadata start address = 0x00007F42313D42FC\n2SCLTEXTCEA Cache end address = 0x00007F4231600000\n2SCLTEXTRTF Runtime flags = 0x00102001ECA6028B\n2SCLTEXTCGN Cache generation = 35\nNULL\n2SCLTEXTCSZ Cache size = 16776608\n2SCLTEXTSMB Softmx bytes = 16776608\n2SCLTEXTFRB Free bytes = 12691668\n2SCLTEXTRCB ROMClass bytes = 1694248\n2SCLTEXTAOB AOT code bytes = 0\n2SCLTEXTADB AOT data bytes = 0\n2SCLTEXTAHB AOT class hierarchy bytes = 32\n2SCLTEXTATB AOT thunk bytes = 0\n2SCLTEXTARB Reserved space for AOT bytes = -1\n2SCLTEXTAMB Maximum space for AOT bytes = -1\n2SCLTEXTJHB JIT hint bytes = 308\n2SCLTEXTJPB JIT profile bytes = 2296\n2SCLTEXTJRB Reserved space for JIT data bytes = -1\n2SCLTEXTJMB Maximum space for JIT data bytes = -1\n2SCLTEXTNOB Java Object bytes = 0\n2SCLTEXTZCB Zip cache bytes = 919328\n2SCLTEXTRWB ReadWrite bytes = 114080\n2SCLTEXTJCB JCL data bytes = 0\n2SCLTEXTBDA Byte data bytes = 0\n2SCLTEXTMDA Metadata bytes = 23448\n2SCLTEXTDAS Class debug area size = 1331200\n2SCLTEXTDAU Class debug area % used = 11%\n2SCLTEXTDAN Class LineNumberTable bytes = 156240\n2SCLTEXTDAV Class LocalVariableTable bytes = 0\nNULL\n2SCLTEXTNRC Number ROMClasses = 595\n2SCLTEXTNAM Number AOT Methods = 0\n2SCLTEXTNAD Number AOT Data Entries = 0\n2SCLTEXTNAH Number AOT Class Hierarchy = 1\n2SCLTEXTNAT Number AOT Thunks = 0\n2SCLTEXTNJH Number JIT Hints = 14\n2SCLTEXTNJP Number JIT Profiles = 20\n2SCLTEXTNCP Number Classpaths = 1\n2SCLTEXTNUR Number URLs = 0\n2SCLTEXTNTK Number Tokens = 0\n2SCLTEXTNOJ Number Java Objects = 0\n2SCLTEXTNZC Number Zip Caches = 5\n2SCLTEXTNJC Number JCL Entries = 0\n2SCLTEXTNST Number Stale classes = 0\n2SCLTEXTPST Percent Stale classes = 0%\nNULL\n2SCLTEXTCPF Cache is 24% full\nNULL\n1SCLTEXTCMST Cache Memory Status\nNULL ------------------\n1SCLTEXTCNTD Cache Name Feature Memory type Cache path\nNULL\n2SCLTEXTCMDT sharedcc_doc-javacore CR Memory mapped file /tmp/javasharedresources/C290M4F1A64P_sharedcc_doc-javacore_G35\nNULL\n1SCLTEXTCMST Cache Lock Status\nNULL ------------------\n1SCLTEXTCNTD Lock Name Lock type TID owning lock\nNULL\n2SCLTEXTCWRL Cache write lock File lock Unowned\n2SCLTEXTCRWL Cache read/write lock File lock Unowned\nNULL\n\n\n\n\n\nCLASSES\n\n\nThe classes section shows information about class loaders. The first part is a summary that records each available class loader (\n2CLTEXTCLLOADER\n) followed by the number of libraries and classes that it loaded. This information is followed by a more detailed list of libraries (\n1CLTEXTCLLIB\n) and classes (\n1CLTEXTCLLO\n) that are loaded.\n\n\nIn the example we can see that the \njava/lang/InternalAnonymousClassLoader\n loaded 2 classes, \njdk/internal/loader/BuiltinClassLoader$$Lambda$2/00000000F03876A0(0x0000000001030F00)\n and \njdk/internal/loader/BuiltinClassLoader$$Lambda$1/00000000F00D2460(0x0000000001018A00)\n.\n\n\nNULL ------------------------------------------------------------------------\n0SECTION CLASSES subcomponent dump routine\nNULL =================================\n1CLTEXTCLLOS Classloader summaries\n1CLTEXTCLLSS 12345678: 1=primordial,2=extension,3=shareable,4=middleware,5=system,6=trusted,7=application,8=delegating\n2CLTEXTCLLOADER p---st-- Loader *System*(0x00000000FFE1D258)\n3CLNMBRLOADEDLIB Number of loaded libraries 5\n3CLNMBRLOADEDCL Number of loaded classes 638\n2CLTEXTCLLOADER -x--st-- Loader jdk/internal/loader/ClassLoaders$PlatformClassLoader(0x00000000FFE1D4F0), Parent *none*(0x0000000000000000)\n3CLNMBRLOADEDLIB Number of loaded libraries 0\n3CLNMBRLOADEDCL Number of loaded classes 0\n2CLTEXTCLLOADER ----st-- Loader java/lang/InternalAnonymousClassLoader(0x00000000FFE1DFD0), Parent *none*(0x0000000000000000)\n3CLNMBRLOADEDLIB Number of loaded libraries 0\n3CLNMBRLOADEDCL Number of loaded classes 2\n2CLTEXTCLLOADER -----ta- Loader jdk/internal/loader/ClassLoaders$AppClassLoader(0x00000000FFE1DAD0), Parent jdk/internal/loader/ClassLoaders$PlatformClassLoader(0x00000000FFE1D4F0)\n3CLNMBRLOADEDLIB Number of loaded libraries 0\n3CLNMBRLOADEDCL Number of loaded classes 0\n1CLTEXTCLLIB ClassLoader loaded libraries\n2CLTEXTCLLIB Loader *System*(0x00000000FFE1D258)\n3CLTEXTLIB /home/me/openj9-openjdk-jdk9/build/linux-x86_64-normal-server-release/images/jdk/lib/compressedrefs/jclse9_29\n3CLTEXTLIB /home/me/openj9-openjdk-jdk9/build/linux-x86_64-normal-server-release/images/jdk/lib/java\n3CLTEXTLIB /home/me/openj9-openjdk-jdk9/build/linux-x86_64-normal-server-release/images/jdk/lib/compressedrefs/j9jit29\n3CLTEXTLIB /home/me/openj9-openjdk-jdk9/build/linux-x86_64-normal-server-release/images/jdk/lib/zip\n3CLTEXTLIB /home/me/openj9-openjdk-jdk9/build/linux-x86_64-normal-server-release/images/jdk/lib/nio\n1CLTEXTCLLOD ClassLoader loaded classes\n2CLTEXTCLLOAD Loader *System*(0x00000000FFE1D258)\n3CLTEXTCLASS [Ljava/lang/Thread$State;(0x0000000001056400)\n...\n2CLTEXTCLLOAD Loader jdk/internal/loader/ClassLoaders$PlatformClassLoader(0x00000000FFE1D4F0)\n2CLTEXTCLLOAD Loader java/lang/InternalAnonymousClassLoader(0x00000000FFE1DFD0)\n3CLTEXTCLASS jdk/internal/loader/BuiltinClassLoader$$Lambda$2/00000000F03876A0(0x0000000001030F00)\n3CLTEXTCLASS jdk/internal/loader/BuiltinClassLoader$$Lambda$1/00000000F00D2460(0x0000000001018A00)\n2CLTEXTCLLOAD Loader jdk/internal/loader/ClassLoaders$AppClassLoader(0x00000000FFE1DAD0)\n\n\n\n\n\nScenarios\n\n\nGeneral Protection Fault\n\n\nIn this scenario, a Java application has crashed due to a General Protection Fault (GPF), automatically generating a Java dump file.\n\n\nThe first section of the file (TITLE) tells you that the GPF triggered the Java dump.\n\n\n0SECTION TITLE subcomponent dump routine\nNULL ===============================\n1TICHARSET UTF-8\n1TISIGINFO Dump Event \"gpf\" (00002000) received\n1TIDATETIME Date: 2018/09/24 at 15:18:03:115\n1TINANOTIME System nanotime: 4498949283020796\n1TIFILENAME Javacore filename: /home/test/JNICrasher/javacore.20180924.151801.29399.0002.txt\n1TIREQFLAGS Request Flags: 0x81 (exclusive+preempt)\n1TIPREPSTATE Prep State: 0x100 (trace_disabled)\n1TIPREPINFO Exclusive VM access not taken: data may not be consistent across javacore sections\n\n\n\n\n\nTo troubleshoot this problem, you need to know which thread caused the GPF to occur. The thread that was running at the time of the crash is reported as the \ncurrent thread\n\nin the THREADS section of the Java dump. Here is an extract from the THREADS section:\n\n\nNULL ------------------------------------------------------------------------\n0SECTION THREADS subcomponent dump routine\nNULL =================================\nNULL\n1XMPOOLINFO JVM Thread pool info:\n2XMPOOLTOTAL Current total number of pooled threads: 16\n2XMPOOLLIVE Current total number of live threads: 15\n2XMPOOLDAEMON Current total number of live daemon threads: 14\nNULL \n1XMCURTHDINFO Current thread\n3XMTHREADINFO \"main\" J9VMThread:0xB6B60E00, omrthread_t:0xB6B049D8, java/lang/Thread:0xB55444D0, state:R, prio=5\n3XMJAVALTHREAD (java/lang/Thread getId:0x1, isDaemon:false)\n3XMTHREADINFO1 (native thread ID:0x72D8, native priority:0x5, native policy:UNKNOWN, vmstate:R, vm thread flags:0x00000000)\n3XMTHREADINFO2 (native stack address range from:0xB6CE3000, to:0xB74E4000, size:0x801000)\n3XMCPUTIME CPU usage total: 0.319865924 secs, current category=\"Application\"\n3XMHEAPALLOC Heap bytes allocated since last GC cycle=778008 (0xBDF18)\n3XMTHREADINFO3 Java callstack:\n4XESTACKTRACE at JNICrasher.doSomethingThatCrashes(Native Method)\n4XESTACKTRACE at JNICrasher.main(JNICrasher.java:7)\n3XMTHREADINFO3 Native callstack:\n4XENATIVESTACK (0xB6C6F663 [libj9prt29.so+0x3b663])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB6C6F1CE [libj9prt29.so+0x3b1ce])\n4XENATIVESTACK (0xB6C6F2C6 [libj9prt29.so+0x3b2c6])\n4XENATIVESTACK (0xB6C6ED93 [libj9prt29.so+0x3ad93])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB6C6ED07 [libj9prt29.so+0x3ad07])\n4XENATIVESTACK (0xB6C6AA3D [libj9prt29.so+0x36a3d])\n4XENATIVESTACK (0xB6C6C3A4 [libj9prt29.so+0x383a4])\n4XENATIVESTACK (0xB667FA19 [libj9dmp29.so+0xfa19])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB66878CF [libj9dmp29.so+0x178cf])\n4XENATIVESTACK (0xB6688083 [libj9dmp29.so+0x18083])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB6680C0D [libj9dmp29.so+0x10c0d])\n4XENATIVESTACK (0xB667F9D7 [libj9dmp29.so+0xf9d7])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB668B02F [libj9dmp29.so+0x1b02f])\n4XENATIVESTACK (0xB668B4D3 [libj9dmp29.so+0x1b4d3])\n4XENATIVESTACK (0xB66740F1 [libj9dmp29.so+0x40f1])\n4XENATIVESTACK (0xB66726FA [libj9dmp29.so+0x26fa])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB66726A9 [libj9dmp29.so+0x26a9])\n4XENATIVESTACK (0xB6676AE4 [libj9dmp29.so+0x6ae4])\n4XENATIVESTACK (0xB668D75A [libj9dmp29.so+0x1d75a])\n4XENATIVESTACK (0xB6A28DD4 [libj9vm29.so+0x81dd4])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB6A289EE [libj9vm29.so+0x819ee])\n4XENATIVESTACK (0xB6A29A40 [libj9vm29.so+0x82a40])\n4XENATIVESTACK (0xB6C52B6A [libj9prt29.so+0x1eb6a])\n4XENATIVESTACK __kernel_rt_sigreturn+0x0 (0xB7747410)\n4XENATIVESTACK (0xB75330B6 [libffi29.so+0x50b6])\n4XENATIVESTACK ffi_raw_call+0xad (0xB7531C53 [libffi29.so+0x3c53])\n4XENATIVESTACK (0xB69BE4AB [libj9vm29.so+0x174ab])\n4XENATIVESTACK (0xB6A665BC [libj9vm29.so+0xbf5bc])\n4XENATIVESTACK (0xB6A15552 [libj9vm29.so+0x6e552])\n4XENATIVESTACK (0xB6A30894 [libj9vm29.so+0x89894])\n4XENATIVESTACK (0xB6A6F169 [libj9vm29.so+0xc8169])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB6A6F1FA [libj9vm29.so+0xc81fa])\n4XENATIVESTACK (0xB6A30994 [libj9vm29.so+0x89994])\n4XENATIVESTACK (0xB6A2CE4C [libj9vm29.so+0x85e4c])\n4XENATIVESTACK (0xB770487D [libjli.so+0x787d])\n4XENATIVESTACK (0xB7719F72 [libpthread.so.0+0x6f72])\n4XENATIVESTACK clone+0x5e (0xB763543E [libc.so.6+0xee43e])\n\n\n\n\n\nThe extract tells you that the current thread was \njava/lang/Thread\n, and information is provided about the Java callstack and Native callstack\n(\n3XMTHREADINFO3\n) at the point at which the crash occurred. To simulate a crash caused by a bug in an application, this example\ncalls a JNI method whose native implementation causes a crash. The Java callstack shows the call to the JNI native method (\nJNIcrasher\n), and the\nnative callstack shows the point of failure. In this example, the native call stack does not include any function names to help you isolate the error\nin the native code. You can get this information from a system dump, which is usually produced alongside the Java dump. Open the\nsystem dump with the \nDump viewer\n and use the \ninfo thread\n command to print the Java and native stack for the current thread.\n\n\nJava OutOfMemoryError\n\n\nIn this scenario, the Java heap runs out of memory, causing an \nOutOfMemoryError\n, which automatically generates a Java dump file.\n\n\nThe first section of the file (TITLE) tells you that a \nsysthrow\n event triggered the Java dump as a result of an OOM (\njava/lang/OutOfMemoryError\n) for\nJava heap space.\n\n\n0SECTION TITLE subcomponent dump routine\nNULL ===============================\n1TICHARSET UTF-8\n1TISIGINFO Dump Event \"systhrow\" (00040000) Detail \"java/lang/OutOfMemoryError\" \"Java heap space\" received\n1TIDATETIME Date: 2018/09/14 at 15:29:42:709\n1TINANOTIME System nanotime: 3635648876608448\n1TIFILENAME Javacore filename: /home/cheesemp/test/javacore.20180914.152929.18885.0003.txt\n1TIREQFLAGS Request Flags: 0x81 (exclusive+preempt)\n1TIPREPSTATE Prep State: 0x104 (exclusive_vm_access+trace_disabled)\n\n\n\n\n\nThe MEMINFO section records how much memory is allocated to the Java heap (\n1STHEAPTYPE Object Memory\n), how much is in use, and how much is free. Solving\nyour problem might be as simple as setting a larger heap size when you start your application.\n\n\nIf you don't know what size the Java heap was set to, you might find that information in the ENVINFO section, which records the command line options that\nwere used when the application started. Look or search for the \n1CIUSERARGS UserArgs:\n string and review the entries recorded for all lines that\nstart \n2CIUSERARG\n. The Java heap size is set by the \n-Xmx\n option. If the size has not been set on the command line by \n-Xmx\n, the default value applies, which\nyou can find in \nDefault Settings\n.\n\n\nIn our scenario the solution to the problem is not an adjustment to the Java heap size. Here is the MEMINFO section:\n\n\n0SECTION MEMINFO subcomponent dump routine\nNULL =================================\nNULL\n1STHEAPTYPE Object Memory\nNULL id start end size space/region\n1STHEAPSPACE 0xB6B49D20 -- -- -- Generational\n1STHEAPREGION 0xB6B4A078 0x95750000 0xB5470000 0x1FD20000 Generational/Tenured Region\n1STHEAPREGION 0xB6B49F10 0xB5470000 0xB54C0000 0x00050000 Generational/Nursery Region\n1STHEAPREGION 0xB6B49DA8 0xB54C0000 0xB5750000 0x00290000 Generational/Nursery Region\nNULL\n1STHEAPTOTAL Total memory: 536870912 (0x20000000)\n1STHEAPINUSE Total memory in use: 302603160 (0x12095B98)\n1STHEAPFREE Total memory free: 234267752 (0x0DF6A468)\n\n\n\n\n\nThe output shows that only 56% of the Java heap is in use, so this suggests that the application is trying to do something sub-optimal. To\ninvestigate further we need to work out which thread was the current thread when the OOM occurred to see what it was trying to do. As in our previous scenario, we can find the\n\ncurrent thread\n in the THREADS section. Here is an extract from the output:\n\n\n0\nSECTION\n \nTHREADS\n \nsubcomponent\n \ndump\n \nroutine\n\n\nNULL\n \n=================================\n\n\nNULL\n\n\n1\nXMPOOLINFO\n \nJVM\n \nThread\n \npool\n \ninfo\n:\n\n\n2\nXMPOOLTOTAL\n \nCurrent\n \ntotal\n \nnumber\n \nof\n \npooled\n \nthreads\n:\n \n16\n\n\n2\nXMPOOLLIVE\n \nCurrent\n \ntotal\n \nnumber\n \nof\n \nlive\n \nthreads\n:\n \n16\n\n\n2\nXMPOOLDAEMON\n \nCurrent\n \ntotal\n \nnumber\n \nof\n \nlive\n \ndaemon\n \nthreads\n:\n \n15\n\n\nNULL\n\n\n1\nXMCURTHDINFO\n \nCurrent\n \nthread\n\n\n3\nXMTHREADINFO\n \n\"main\"\n \nJ9VMThread\n:\n0xB6B60C00\n,\n \nomrthread_t\n:\n0xB6B049D8\n,\n \njava\n/\nlang\n/\nThread\n:\n0x95764520\n,\n \nstate\n:\nR\n,\n \nprio\n=\n5\n\n\n3\nXMJAVALTHREAD\n \n(\njava\n/\nlang\n/\nThread\n \ngetId\n:\n0x1\n,\n \nisDaemon\n:\nfalse\n)\n\n\n3\nXMTHREADINFO1\n \n(\nnative\n \nthread\n \nID\n:\n0x49C6\n,\n \nnative\n \npriority\n:\n0x5\n,\n \nnative\n \npolicy\n:\nUNKNOWN\n,\n \nvmstate\n:\nR\n,\n \nvm\n \nthread\n \nflags\n:\n0x00001020\n)\n\n\n3\nXMTHREADINFO2\n \n(\nnative\n \nstack\n \naddress\n \nrange\n \nfrom\n:\n0xB6CB5000\n,\n \nto\n:\n0xB74B6000\n,\n \nsize\n:\n0x801000\n)\n\n\n3\nXMCPUTIME\n \nCPU\n \nusage\n \ntotal\n:\n \n8.537823831\n \nsecs\n,\n \ncurrent\n \ncategory\n=\n\"Application\"\n\n\n3\nXMHEAPALLOC\n \nHeap\n \nbytes\n \nallocated\n \nsince\n \nlast\n \nGC\n \ncycle\n=\n0\n \n(\n0x0\n)\n\n\n3\nXMTHREADINFO3\n \nJava\n \ncallstack\n:\n\n\n4\nXESTACKTRACE\n \nat\n \njava\n/\nlang\n/\nStringBuffer\n.\nensureCapacityImpl\n(\nStringBuffer\n.\njava\n:\n696\n)\n\n\n4\nXESTACKTRACE\n \nat\n \njava\n/\nlang\n/\nStringBuffer\n.\nappend\n(\nStringBuffer\n.\njava\n:\n486\n(\nCompiled\n \nCode\n))\n\n\n5\nXESTACKTRACE\n \n(\nentered\n \nlock\n:\n \njava\n/\nlang\n/\nStringBuffer\n@0x957645B8\n,\n \nentry\n \ncount\n:\n \n1\n)\n\n\n4\nXESTACKTRACE\n \nat\n \njava\n/\nlang\n/\nStringBuffer\n.\nappend\n(\nStringBuffer\n.\njava\n:\n428\n(\nCompiled\n \nCode\n))\n\n\n4\nXESTACKTRACE\n \nat\n \nHeapBreaker\n.\nmain\n(\nHeapBreaker\n.\njava\n:\n34\n(\nCompiled\n \nCode\n))\n\n\n3\nXMTHREADINFO3\n \nNative\n \ncallstack\n:\n\n\n4\nXENATIVESTACK\n \n(\n0xB6C535B3\n \n[\nlibj9prt29\n.\nso\n+\n0x3b5b3\n])\n\n\n4\nXENATIVESTACK\n \n(\n0xB6C36F3E\n \n[\nlibj9prt29\n.\nso\n+\n0x1ef3e\n])\n\n\n4\nXENATIVESTACK\n \n(\n0xB6C5311E\n \n[\nlibj9prt29\n.\nso\n+\n0x3b11e\n])\n\n\n4\nXENATIVESTACK\n \n(\n0xB6C53216\n \n[\nlibj9prt29\n.\nso\n+\n0x3b216\n])\n\n\n4\nXENATIVESTACK\n \n(\n0xB6C52CE3\n \n[\nlibj9prt29\n.\nso\n+\n0x3ace3\n])\n\n\n4\nXENATIVESTACK\n \n(\n0xB6C36F3E\n \n[\nlibj9prt29\n.\nso\n+\n0x1ef3e\n])\n\n\n4\nXENATIVESTACK\n \n(\n0xB6C52C57\n \n[\nlibj9prt29\n.\nso\n+\n0x3ac57\n])\n\n\n4\nXENATIVESTACK\n \n(\n0xB6C4E9CD\n \n[\nlibj9prt29\n.\nso\n+\n0x369cd\n])\n\n\n4\nXENATIVESTACK\n \n(\n0xB6C502FA\n \n[\nlibj9prt29\n.\nso\n+\n0x382fa\n])\n\n\n\n\n\n\nTo simulate a Java \nOutOfMemoryError\n, this example application repeatedly appends characters to a \nStringBuffer\n object in an infinite loop. The Java callstack shows the \nHeapBreaker.main\n method appending characters (\njava/lang/StringGuffer.append\n) until the method \njava/lang/StringBuffer.ensureCapacityImpl()\n throws the \nOutOfMemoryError\n.\n\n\nStringBuffer objects are wrappers for character arrays (\nchar[]\n) and when the capacity of the underlying array is reached, the contents are automatically copied into a new, larger array. The new array is created in the \nStringBuffer.ensureCapacity()\n method, which more or less doubles the size of the old array. In our scenario, the array takes up all the remaining space in the Java heap.\n\n\nThe MEMINFO section of the Java dump file can also tell you when an unexpectedly large allocation request causes an OOM. Look for the GC History (\n1STGCHTYPE\n) section, which details allocation requests that trigger GC activity. In the sample output you can see that a large allocation request (\nrequestedbytes=603979784\n) triggered a global GC. When the GC could not free up sufficient space in the heap to satisfy the request, the allocation failure generated the OOM.\n\n\n1STGCHTYPE GC History \n3STHSTTYPE 14:29:29:580239000 GMT j9mm.101 - J9AllocateIndexableObject() returning NULL! 0 bytes requested for object of class B6BBC300 from memory space 'Generational' id=B6B49D20\n3STHSTTYPE 14:29:29:579916000 GMT j9mm.134 - Allocation failure end: newspace=2686912/3014656 oldspace=231597224/533856256 loa=5338112/5338112\n3STHSTTYPE 14:29:29:579905000 GMT j9mm.470 - Allocation failure cycle end: newspace=2686912/3014656 oldspace=231597224/533856256 loa=5338112/5338112\n3STHSTTYPE 14:29:29:579859000 GMT j9mm.475 - GlobalGC end: workstackoverflow=0 overflowcount=0 memory=234284136/536870912\n3STHSTTYPE 14:29:29:579807000 GMT j9mm.90 - GlobalGC collect complete\n3STHSTTYPE 14:29:29:579776000 GMT j9mm.137 - Compact end: bytesmoved=301989896\n3STHSTTYPE 14:29:29:313899000 GMT j9mm.136 - Compact start: reason=compact to meet allocation\n3STHSTTYPE 14:29:29:313555000 GMT j9mm.57 - Sweep end\n3STHSTTYPE 14:29:29:310772000 GMT j9mm.56 - Sweep start\n3STHSTTYPE 14:29:29:310765000 GMT j9mm.94 - Class unloading end: classloadersunloaded=0 classesunloaded=0\n3STHSTTYPE 14:29:29:310753000 GMT j9mm.60 - Class unloading start\n3STHSTTYPE 14:29:29:310750000 GMT j9mm.55 - Mark end\n3STHSTTYPE 14:29:29:306013000 GMT j9mm.54 - Mark start\n3STHSTTYPE 14:29:29:305957000 GMT j9mm.474 - GlobalGC start: globalcount=9\n3STHSTTYPE 14:29:29:305888000 GMT j9mm.475 - GlobalGC end: workstackoverflow=0 overflowcount=0 memory=234284136/536870912\n3STHSTTYPE 14:29:29:305837000 GMT j9mm.90 - GlobalGC collect complete\n3STHSTTYPE 14:29:29:305808000 GMT j9mm.137 - Compact end: bytesmoved=189784\n3STHSTTYPE 14:29:29:298042000 GMT j9mm.136 - Compact start: reason=compact to meet allocation\n3STHSTTYPE 14:29:29:297695000 GMT j9mm.57 - Sweep end\n3STHSTTYPE 14:29:29:291696000 GMT j9mm.56 - Sweep start\n3STHSTTYPE 14:29:29:291692000 GMT j9mm.55 - Mark end\n3STHSTTYPE 14:29:29:284994000 GMT j9mm.54 - Mark start\n3STHSTTYPE 14:29:29:284941000 GMT j9mm.474 - GlobalGC start: globalcount=8\n3STHSTTYPE 14:29:29:284916000 GMT j9mm.135 - Exclusive access: exclusiveaccessms=0.016 meanexclusiveaccessms=0.016 threads=0 lastthreadtid=0xB6B61100 beatenbyotherthread=0\n3STHSTTYPE 14:29:29:284914000 GMT j9mm.469 - Allocation failure cycle start: newspace=2678784/3014656 oldspace=80601248/533856256 loa=5338112/5338112 requestedbytes=603979784\n3STHSTTYPE 14:29:29:284893000 GMT j9mm.470 - Allocation failure cycle end: newspace=2678784/3014656 oldspace=80601248/533856256 loa=5338112/5338112\n3STHSTTYPE 14:29:29:284858000 GMT j9mm.560 - LocalGC end: rememberedsetoverflow=0 causedrememberedsetoverflow=0 scancacheoverflow=0 failedflipcount=0 failedflipbytes=0 failedtenurecount=0 failedtenurebytes=0 flipcount=2 flipbytes=64 newspace=2678784/3014656 oldspace=80601248/533856256 loa=5338112/5338112 tenureage=0\n3STHSTTYPE 14:29:29:284140000 GMT j9mm.140 - Tilt ratio: 89\n3STHSTTYPE 14:29:29:283160000 GMT j9mm.64 - LocalGC start: globalcount=8 scavengecount=335 weakrefs=0 soft=0 phantom=0 finalizers=0\n3STHSTTYPE 14:29:29:283123000 GMT j9mm.135 - Exclusive access: exclusiveaccessms=0.016 meanexclusiveaccessms=0.016 threads=0 lastthreadtid=0xB6B61100 beatenbyotherthread=0\n3STHSTTYPE 14:29:29:283120000 GMT j9mm.469 - Allocation failure cycle start: newspace=753616/3014656 oldspace=80601248/533856256 loa=5338112/5338112 requestedbytes=603979784\n3STHSTTYPE 14:29:29:283117000 GMT j9mm.133 - Allocation failure start: newspace=753616/3014656 oldspace=80601248/533856256 loa=5338112/5338112 requestedbytes=603979784\n3STHSTTYPE 14:29:29:269762000 GMT j9mm.134 - Allocation failure end: newspace=2686928/3014656 oldspace=80601248/533856256 loa=5338112/5338112\n3STHSTTYPE 14:29:29:269751000 GMT j9mm.470 - Allocation failure cycle end: newspace=2686976/3014656 oldspace=80601248/533856256 loa=5338112/5338112\n3STHSTTYPE 14:29:29:269718000 GMT j9mm.560 - LocalGC end: rememberedsetoverflow=0 causedrememberedsetoverflow=0 scancacheoverflow=0 failedflipcount=0 failedflipbytes=0 failedtenurecount=0 failedtenurebytes=0 flipcount=0 flipbytes=0 newspace=2686976/3014656 oldspace=80601248/533856256 loa=5338112/5338112 tenureage=0\n3STHSTTYPE 14:29:29:268981000 GMT j9mm.140 - Tilt ratio: 89\n3STHSTTYPE 14:29:29:268007000 GMT j9mm.64 - LocalGC start: globalcount=8 scavengecount=334 weakrefs=0 soft=0 phantom=0 finalizers=0\n3STHSTTYPE 14:29:29:267969000 GMT j9mm.135 - Exclusive access: exclusiveaccessms=0.016 meanexclusiveaccessms=0.016 threads=0 lastthreadtid=0xB6B61100 beatenbyotherthread=0\n3STHSTTYPE 14:29:29:267966000 GMT j9mm.469 - Allocation failure cycle start: newspace=0/3014656 oldspace=80601248/533856256 loa=5338112/5338112 requestedbytes=48\n3STHSTTYPE 14:29:29:267963000 GMT j9mm.133 - Allocation failure start: newspace=0/3014656 oldspace=80601248/533856256 loa=5338112/5338112 requestedbytes=48\n3STHSTTYPE 14:29:29:249015000 GMT j9mm.134 - Allocation failure end: newspace=2686928/3014656 oldspace=80601248/533856256 loa=5338112/5338112\n3STHSTTYPE 14:29:29:249003000 GMT j9mm.470 - Allocation failure cycle end: newspace=2686976/3014656 oldspace=80601248/533856256 loa=5338112/5338112\n3STHSTTYPE 14:29:29:248971000 GMT j9mm.560 - LocalGC end: rememberedsetoverflow=0 causedrememberedsetoverflow=0 scancacheoverflow=0 failedflipcount=0 failedflipbytes=0 failedtenurecount=0 failedtenurebytes=0 flipcount=0 flipbytes=0 newspace=2686976/3014656 oldspace=80601248/533856256 loa=5338112/5338112 tenureage=0\n\n\n\n\n\nAlthough the Java code we used in this scenario deliberately triggered an \nOutOfMemoryError\n in a pronounced way, similar allocation issues can and do occur when dealing with large data sets such as XML files.\n\n\nThe next step in diagnosing the problem is to open the system dump that gets generated automatically when an \nOutOfMemoryError\n occurs. Open\nthe dump with the \nEclipse Memory Analyzer tool (MAT)\n and search for the \nStringBuffer\n object, which should provide further clues about what went wrong. A common example is seeing the same \nString\n duplicated over and over again, which might indicate that code is stuck in a loop.\n\n\n \nNote:\n If you want to use MAT to analyze your system dump, you must install the Diagnostic Tool Framework for Java (DTFJ) plugin in the Eclipse IDE. Select the following menu items:\n\n\nHelp > Install New Software > Work with \"IBM Diagnostic Tool Framework for Java\" > \n\n\n\n\n\nIf, unlike the previous scenario, you receive an \nOutOfMemoryError\n and the MEMINFO section shows that there is very little space left\non the Java heap, the current thread information is not important. The current thread is simply the thread that happened to be current when the space ran out. In this situation you might want to increase your Java heap size. For help with this task, see \nHow to do heap sizing\n.",
+ "text": "Java dump\n\n\nJava dumps, sometimes referred to as \nJava cores\n, are produced when the VM ends unexpectedly because of an operating system signal, \nOutOfMemoryError\n, or a user-initiated keystroke combination. You can also generate a Java dump by calling the Dump API programmatically from your application or specifying the \n-Xdump:java\n option on the command line.\n\n\nIf your Java application crashes or hangs, Java dumps can provide useful information to help you diagnose the root cause.\n\n\n\n\nIf your application crashes, Javadumps are generated automatically for the following types of failure:\n\n\nthe VM receives an unexpected signal or an assertion failure\n\n\nthe VM runs out of memory\n\n\n\n\n\n\n\n\nIf your application hangs, you can trigger the generation of a Java dump by sending a SIGQUIT signal (\nkill -3\n) to the VM.\n\n\n \nNote:\n On Windows, if you started the VM in a console window you can force the VM to produce a Java dump in response to a SIGBREAK signal\n(Ctrl-Break keyboard combination). If you didn't start in a console window there is no equivalent to a Linux \nkill\n command on Windows for sending signals.\nThe only option here is to trigger a full system dump by finding the VM process in the \nProcesses\n tab of the Windows Task Manager and clicking\n\nCreate dump file\n.\n\n\n\n\n\n\nTo help you understand how a Java dump can help you with problem diagnosis, this topic includes a few scenarios to help you interpret the data:\n\n\n\n\nA crash caused by a general protection fault (gpf)\n\n\nA Java heap OutOfMemoryError (OOM)\n\n\nA native OutOfMemoryError (OOM)\n\n\nA deadlock situation\n\n\n\n\nJava dump contents\n\n\nJava dumps summarize the state of the VM when the event occurs, with most of the information relating to components of the VM.\nThe file is made up of a number of sections that provide different types of information.\n\n\nTITLE\n\n\nThe first section of the Java dump file provides information about the event that triggered the production of the dump.\nIn the following example you can see that a \nvmstop\n event triggered the dump at a specified date and time.\n\n\n0SECTION TITLE subcomponent dump routine\nNULL ===============================\n1TICHARSET UTF-8\n1TISIGINFO Dump Event \"vmstop\" (00000002) Detail \"#0000000000000000\" received\n1TIDATETIME Date: 2018/08/30 at 21:55:47:607\n1TINANOTIME System nanotime: 22012355276134\n1TIFILENAME Javacore filename: /home/doc-javacore/javacore.20180830.215547.30285.0001.txt\n1TIREQFLAGS Request Flags: 0x81 (exclusive+preempt)\n1TIPREPSTATE Prep State: 0x106 (vm_access+exclusive_vm_access+trace_disabled)\n\n\n\n\n\nGPINFO\n\n\nThe GPINFO section provides general information about the system that the VM is running on. The following example is taken\nfrom a Java dump that was generated on a Linux system.\n\n\nNULL ------------------------------------------------------------------------\n0SECTION GPINFO subcomponent dump routine\nNULL ================================\n2XHOSLEVEL OS Level : Linux 3.10.0-862.11.6.el7.x86_64\n2XHCPUS Processors -\n3XHCPUARCH Architecture : amd64\n3XHNUMCPUS How Many : 4\n3XHNUMASUP NUMA is either not supported or has been disabled by user\nNULL \n1XHERROR2 Register dump section only produced for SIGSEGV, SIGILL or SIGFPE.\nNULL \n\n\n\n\n\nThe content of this section can vary, depending on the cause of the dump. For example, if the dump was caused by a\ngeneral protection fault (gpf), the library in which the crash occurred is also recorded, together with a value shown\nas \nVM flags\n. This value can provide some clues about which component of the VM might have been involved. Look for the\nfollowing line in the output:\n\n\n1XHFLAGS VM flags:0000000000000000\n\n\n\n\n\nThe hexadecimal number recorded for \nVM flags\n ends in MSSSS, where M is the VM component and SSSS is component-specific code as shown in the following table:\n\n\n\n\n\n\n\n\nComponent\n\n\nCode value\n\n\n\n\n\n\n\n\n\n\nINTERPRETER\n\n\n0x10000\n\n\n\n\n\n\nGC\n\n\n0x20000\n\n\n\n\n\n\nGROW_STACK\n\n\n0x30000\n\n\n\n\n\n\nJNI\n\n\n0x40000\n\n\n\n\n\n\nJIT_CODEGEN\n\n\n0x50000\n\n\n\n\n\n\nBCVERIFY\n\n\n0x60000\n\n\n\n\n\n\nRTVERIFY\n\n\n0x70000\n\n\n\n\n\n\nSHAREDCLASSES\n\n\n0x80000\n\n\n\n\n\n\n\n\nA value of \n0000000000000000\n (0x00000) indicates that a crash occurred outside of the VM.\n\n\nENVINFO\n\n\nThis section contains useful information about the environment in which the crash took place, including the following data:\n\n\n\n\nJava version (\n1CIJAVAVERSION\n)\n\n\nOpenJ9 VM and subcomponent version information (\n1CIVMVERSION\n, \n1CIJ9VMVERSION\n, \n1CIJITVERSION\n, \n1CIOMRVERSION\n, \n1CIJCLVERSION\n)\n\n\nVM start time (\n1CISTARTTIME\n) and process information (\n1CIPROCESSID\n)\n\n\nJava home (\n1CIJAVAHOMEDIR\n) and DLL (\n1CIJAVADLLDIR\n) directories\n\n\nUser arguments passed on the command line (\n1CIUSERARG\n)\n\n\nUser limits imposed by the system (\n1CIUSERLIMITS\n)\n\n\nEnvironment variables in place (\n1CIENVVARS\n)\n\n\nSystem information (\n1CISYSINFO\n)\n\n\nCPU information (\n1CICPUINFO\n)\n\n\n\n\n\nFor clarity, the following example shows a shortened version of this section, where \n...\n indicates that lines are removed:\n\n\nNULL\n \n------------------------------------------------------------------------\n\n\n0\nSECTION\n \nENVINFO\n \nsubcomponent\n \ndump\n \nroutine\n\n\nNULL\n \n=================================\n\n\n1\nCIJAVAVERSION\n \nJRE\n \n9\n \nLinux\n \namd64\n-\n64\n \n(\nbuild\n \n9.0.4\n-\ninternal\n+\n0\n-\nadhoc\n..\nopenj9\n-\nopenjdk\n-\njdk9\n)\n\n\n1\nCIVMVERSION\n \n20180830\n_000000\n\n\n1\nCIJ9VMVERSION\n \n8e7\nc6ec\n\n\n1\nCIJITVERSION\n \n8e7\nc6ec\n\n\n1\nCIOMRVERSION\n \n553811\nb_CMPRSS\n\n\n1\nCIJCLVERSION\n \nec1d223\n \nbased\n \non\n \njdk\n-\n9.0.4\n+\n12\n\n\n1\nCIJITMODES\n \nJIT\n \nenabled\n,\n \nAOT\n \nenabled\n,\n \nFSD\n \ndisabled\n,\n \nHCR\n \nenabled\n\n\n1\nCIRUNNINGAS\n \nRunning\n \nas\n \na\n \nstandalone\n \nJVM\n\n\n1\nCIVMIDLESTATE\n \nVM\n \nIdle\n \nState\n:\n \nACTIVE\n\n\n1\nCISTARTTIME\n \nJVM\n \nstart\n \ntime\n:\n \n2018\n/\n08\n/\n30\n \nat\n \n21\n:\n55\n:\n47\n:\n387\n\n\n1\nCISTARTNANO\n \nJVM\n \nstart\n \nnanotime\n:\n \n22012135233549\n\n\n1\nCIPROCESSID\n \nProcess\n \nID\n:\n \n30285\n \n(\n0x764D\n)\n\n\n1\nCICMDLINE\n \n[\nnot\n \navailable\n]\n\n\n1\nCIJAVAHOMEDIR\n \nJava\n \nHome\n \nDir\n:\n \n/\nhome\n/\nme\n/\nopenj9\n-\nopenjdk\n-\njdk9\n/\nbuild\n/\nlinux\n-\nx86_64\n-\nnormal\n-\nserver\n-\nrelease\n/\nimages\n/\njdk\n\n\n1\nCIJAVADLLDIR\n \nJava\n \nDLL\n \nDir\n:\n \n/\nhome\n/\nme\n/\nopenj9\n-\nopenjdk\n-\njdk9\n/\nbuild\n/\nlinux\n-\nx86_64\n-\nnormal\n-\nserver\n-\nrelease\n/\nimages\n/\njdk\n/\nbin\n\n\n1\nCISYSCP\n \nSys\n \nClasspath\n:\n \n\n1\nCIUSERARGS\n \nUserArgs\n:\n\n\n2\nCIUSERARG\n \n-\nXoptionsfile\n=/\nhome\n/\nme\n/\nopenj9\n-\nopenjdk\n-\njdk9\n/\nbuild\n/\nlinux\n-\nx86_64\n-\nnormal\n-\nserver\n-\nrelease\n/\nimages\n/\njdk\n/\nlib\n/\noptions\n.\ndefault\n\n\n...\n\n\nNULL\n\n\n1\nCIUSERLIMITS\n \nUser\n \nLimits\n \n(\nin\n \nbytes\n \nexcept\n \nfor\n \nNOFILE\n \nand\n \nNPROC\n)\n\n\nNULL\n \n------------------------------------------------------------------------\n\n\nNULL\n \ntype\n \nsoft\n \nlimit\n \nhard\n \nlimit\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_AS\n \nunlimited\n \nunlimited\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_CORE\n \n0\n \nunlimited\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_CPU\n \nunlimited\n \nunlimited\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_DATA\n \nunlimited\n \nunlimited\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_FSIZE\n \nunlimited\n \nunlimited\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_LOCKS\n \nunlimited\n \nunlimited\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_MEMLOCK\n \n65536\n \n65536\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_NOFILE\n \n4096\n \n4096\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_NPROC\n \n4096\n \n30592\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_RSS\n \nunlimited\n \nunlimited\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_STACK\n \n8388608\n \nunlimited\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_MSGQUEUE\n \n819200\n \n819200\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_NICE\n \n0\n \n0\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_RTPRIO\n \n0\n \n0\n\n\n2\nCIUSERLIMIT\n \nRLIMIT_SIGPENDING\n \n30592\n \n30592\n\n\nNULL\n\n\n1\nCIENVVARS\n \nEnvironment\n \nVariables\n\n\nNULL\n \n------------------------------------------------------------------------\n\n\n2\nCIENVVAR\n \nXDG_VTNR\n=\n1\n\n\n2\nCIENVVAR\n \nSSH_AGENT_PID\n=\n2653\n\n\n...\n\n\nNULL\n \n\n1\nCISYSINFO\n \nSystem\n \nInformation\n\n\nNULL\n \n------------------------------------------------------------------------\n\n\n2\nCISYSINFO\n \n/\nproc\n/\nsys\n/\nkernel\n/\ncore_pattern\n \n=\n \ncore\n\n\n2\nCISYSINFO\n \n/\nproc\n/\nsys\n/\nkernel\n/\ncore_uses_pid\n \n=\n \n1\n\n\nNULL\n \n\n1\nCICPUINFO\n \nCPU\n \nInformation\n\n\nNULL\n \n------------------------------------------------------------------------\n\n\n2\nCIPHYSCPU\n \nPhysical\n \nCPUs\n:\n \n4\n\n\n2\nCIONLNCPU\n \nOnline\n \nCPUs\n:\n \n4\n\n\n2\nCIBOUNDCPU\n \nBound\n \nCPUs\n:\n \n4\n\n\n2\nCIACTIVECPU\n \nActive\n \nCPUs\n:\n \n0\n\n\n2\nCITARGETCPU\n \nTarget\n \nCPUs\n:\n \n4\n\n\n\n\n\n\n\n\n\nNATIVEMEMINFO\n\n\nThis section records information about native memory that is requested by using library functions such as \nmalloc()\n and \nmmap()\n.\nValues are provided as a breakdown, per component, indicating the total number of bytes allocated and the number of native memory allocations.\nIn the following example, 4,682,840 bytes of native memory are allocated (but not yet freed) to VM Classes, which corresponds to 141 allocations.\n\n\nNULL ------------------------------------------------------------------------\n0SECTION NATIVEMEMINFO subcomponent dump routine\nNULL =================================\n0MEMUSER\n1MEMUSER JRE: 2,569,088,312 bytes / 4653 allocations\n1MEMUSER |\n2MEMUSER +--VM: 2,280,088,336 bytes / 2423 allocations\n2MEMUSER | |\n3MEMUSER | +--Classes: 4,682,840 bytes / 141 allocations\n2MEMUSER | |\n3MEMUSER | +--Memory Manager (GC): 2,054,966,784 bytes / 433 allocations\n3MEMUSER | | |\n4MEMUSER | | +--Java Heap: 2,014,113,792 bytes / 1 allocation\n3MEMUSER | | |\n4MEMUSER | | +--Other: 40,852,992 bytes / 432 allocations\n2MEMUSER | |\n3MEMUSER | +--Threads: 10,970,016 bytes / 156 allocations\n3MEMUSER | | |\n4MEMUSER | | +--Java Stack: 197,760 bytes / 16 allocations\n3MEMUSER | | |\n4MEMUSER | | +--Native Stack: 10,616,832 bytes / 17 allocations\n3MEMUSER | | |\n4MEMUSER | | +--Other: 155,424 bytes / 123 allocations\n2MEMUSER | |\n3MEMUSER | +--Trace: 180,056 bytes / 263 allocations\n2MEMUSER | |\n3MEMUSER | +--JVMTI: 17,776 bytes / 13 allocations\n2MEMUSER | |\n3MEMUSER | +--JNI: 36,184 bytes / 52 allocations\n2MEMUSER | |\n3MEMUSER | +--Port Library: 208,179,632 bytes / 72 allocations\n3MEMUSER | | |\n4MEMUSER | | +--Unused <32bit allocation regions: 208,168,752 bytes / 1 allocation\n3MEMUSER | | |\n4MEMUSER | | +--Other: 10,880 bytes / 71 allocations\n2MEMUSER | |\n3MEMUSER | +--Other: 1,055,048 bytes / 1293 allocations\n1MEMUSER |\n2MEMUSER +--JIT: 288,472,816 bytes / 140 allocations\n2MEMUSER | |\n3MEMUSER | +--JIT Code Cache: 268,435,456 bytes / 1 allocation\n2MEMUSER | |\n3MEMUSER | +--JIT Data Cache: 2,097,216 bytes / 1 allocation\n2MEMUSER | |\n3MEMUSER | +--Other: 17,940,144 bytes / 138 allocations\n1MEMUSER |\n2MEMUSER +--Class Libraries: 13,432 bytes / 25 allocations\n2MEMUSER | |\n3MEMUSER | +--VM Class Libraries: 13,432 bytes / 25 allocations\n3MEMUSER | | |\n4MEMUSER | | +--sun.misc.Unsafe: 3,184 bytes / 13 allocations\n4MEMUSER | | | |\n5MEMUSER | | | +--Direct Byte Buffers: 1,056 bytes / 12 allocations\n4MEMUSER | | | |\n5MEMUSER | | | +--Other: 2,128 bytes / 1 allocation\n3MEMUSER | | |\n4MEMUSER | | +--Other: 10,248 bytes / 12 allocations\n1MEMUSER |\n2MEMUSER +--Unknown: 513,728 bytes / 2065 allocations\nNULL \n\n\n\n\n\nThis section does not record memory that is allocated by application or JNI code and is typically a little less than the\nvalue recorded by operating system tools.\n\n\nMEMINFO\n\n\nThis section relates to memory management, providing a breakdown of memory usage in the VM for the object heap,\ninternal memory, memory used for classes, the JIT code cache, and JIT data cache in decimal and hexadecimal format.\nYou can also find out which garbage collection policy is in use when the dump is produced. \n\n\nThe object memory area (\n1STHEAPTYPE\n) records each memory region in use, its start and end address, and region size.\nFurther information is recorded about the memory segments used for internal memory, class memory, the JIT code cache and JIT data cache (\n1STSEGMENT\n).\nThis information includes the address of the segment control data structure, the start and end address of the native memory segment, as well as\nthe segment size.\n\n\nFor clarity, the following example shows a shortened version of this section, where \n...\n indicates that lines are removed:\n\n\nNULL ------------------------------------------------------------------------\n0SECTION MEMINFO subcomponent dump routine\nNULL =================================\nNULL \n1STHEAPTYPE Object Memory\nNULL id start end size space/region\n1STHEAPSPACE 0x00007FF4F00744A0 -- -- -- Generational\n1STHEAPREGION 0x00007FF4F0074CE0 0x0000000087F40000 0x0000000088540000 0x0000000000600000 Generational/Tenured Region\n1STHEAPREGION 0x00007FF4F0074930 0x00000000FFE00000 0x00000000FFF00000 0x0000000000100000 Generational/Nursery Region\n1STHEAPREGION 0x00007FF4F0074580 0x00000000FFF00000 0x0000000100000000 0x0000000000100000 Generational/Nursery Region\nNULL\n1STHEAPTOTAL Total memory: 8388608 (0x0000000000800000)\n1STHEAPINUSE Total memory in use: 2030408 (0x00000000001EFB48)\n1STHEAPFREE Total memory free: 6358200 (0x00000000006104B8)\nNULL\n1STSEGTYPE Internal Memory\nNULL segment start alloc end type size\n1STSEGMENT 0x00007FF4F004CBC8 0x00007FF4CD33C000 0x00007FF4CD33C000 0x00007FF4CE33C000 0x01000440 0x0000000001000000\n1STSEGMENT 0x00007FF4F004CB08 0x00007FF4DE43D030 0x00007FF4DE517770 0x00007FF4DE53D030 0x00800040 0x0000000000100000\nNULL\n1STSEGTOTAL Total memory: 17825792 (0x0000000001100000)\n1STSEGINUSE Total memory in use: 894784 (0x00000000000DA740)\n1STSEGFREE Total memory free: 16931008 (0x00000000010258C0)\nNULL \n1STSEGTYPE Class Memory\nNULL segment start alloc end type size\n1STSEGMENT 0x00007FF4F03B5638 0x0000000001053D98 0x000000000105BD98 0x000000000105BD98 0x00010040 0x0000000000008000\n1STSEGMENT 0x00007FF4F03B5578 0x0000000001048188 0x0000000001050188 0x0000000001050188 0x00010040 0x0000000000008000\n...\nNULL\n1STSEGTOTAL Total memory: 3512520 (0x00000000003598C8)\n1STSEGINUSE Total memory in use: 3433944 (0x00000000003465D8)\n1STSEGFREE Total memory free: 78576 (0x00000000000132F0)\nNULL \n1STSEGTYPE JIT Code Cache\nNULL segment start alloc end type size\n1STSEGMENT 0x00007FF4F00961F8 0x00007FF4CE43D000 0x00007FF4CE445790 0x00007FF4DE43D000 0x00000068 0x0000000010000000\nNULL\n1STSEGTOTAL Total memory: 268435456 (0x0000000010000000)\n1STSEGINUSE Total memory in use: 34704 (0x0000000000008790)\n1STSEGFREE Total memory free: 268400752 (0x000000000FFF7870)\n1STSEGLIMIT Allocation limit: 268435456 (0x0000000010000000)\nNULL \n1STSEGTYPE JIT Data Cache\nNULL segment start alloc end type size\n1STSEGMENT 0x00007FF4F0096668 0x00007FF4CC553030 0x00007FF4CC753030 0x00007FF4CC753030 0x00000048 0x0000000000200000\nNULL\n1STSEGTOTAL Total memory: 2097152 (0x0000000000200000)\n1STSEGINUSE Total memory in use: 2097152 (0x0000000000200000)\n1STSEGFREE Total memory free: 0 (0x0000000000000000)\n1STSEGLIMIT Allocation limit: 402653184 (0x0000000018000000)\nNULL \n1STGCHTYPE GC History \nNULL \n\n\n\n\n\nIn the example, the GC History (\n1STGCHTYPE\n) section is blank. This section is populated if a garbage collection cycle occurred in\na VM that is being diagnosed with the trace facility.\n\n\nLOCKS\n\n\nThis section of the Java dump provides information about locks, which protect shared resources from being accessed by more than one entity at a time. The information is essential in a deadlock situation, where two threads attempt to synchronize on an object and lock an instance of a class. Precise information is recorded about the threads that are causing the problem, which enables you to identify the root cause.\n\n\nThe following example shows a typical LOCKS section, where no deadlocks existed at the time the dump was triggered. For clarity, the following example shows a shortened version of this section, where \n...\n indicates that lines are removed:\n\n\nNULL\n \n------------------------------------------------------------------------\n\n\n0\nSECTION\n \nLOCKS\n \nsubcomponent\n \ndump\n \nroutine\n\n\nNULL\n \n===============================\n\n\nNULL\n \n\n1L\nKPOOLINFO\n \nMonitor\n \npool\n \ninfo\n:\n\n\n2L\nKPOOLTOTAL\n \nCurrent\n \ntotal\n \nnumber\n \nof\n \nmonitors\n:\n \n3\n\n\nNULL\n \n\n1L\nKMONPOOLDUMP\n \nMonitor\n \nPool\n \nDump\n \n(\nflat\n \n&\n \ninflated\n \nobject\n-\nmonitors\n)\n:\n\n\n2L\nKMONINUSE\n \nsys_mon_t\n:\n0x00007FF4B0001D78\n \ninfl_mon_t\n:\n \n0x00007FF4B0001DF8\n:\n\n\n3L\nKMONOBJECT\n \njava\n/\nlang\n/\nref\n/\nReferenceQueue\n@0x00000000FFE26A10\n:\n \n<\nunowned\n>\n\n\n3L\nKNOTIFYQ\n \nWaiting\n \nto\n \nbe\n \nnotified\n:\n\n\n3L\nKWAITNOTIFY\n \n\"Common-Cleaner\"\n \n(\nJ9VMThread\n:\n0x0000000000FD0100\n)\n\n\nNULL\n \n\n1L\nKREGMONDUMP\n \nJVM\n \nSystem\n \nMonitor\n \nDump\n \n(\nregistered\n \nmonitors\n)\n:\n\n\n2L\nKREGMON\n \nThread\n \nglobal\n \nlock\n \n(\n0x00007FF4F0004FE8\n)\n:\n \n<\nunowned\n>\n\n\n2L\nKREGMON\n \n&\n(\nPPG_mem_mem32_subAllocHeapMem32\n.\nmonitor\n)\n \nlock\n \n(\n0x00007FF4F0005098\n)\n:\n \n<\nunowned\n>\n\n\n2L\nKREGMON\n \nNLS\n \nhash\n \ntable\n \nlock\n \n(\n0x00007FF4F0005148\n)\n:\n \n<\nunowned\n>\n\n\n...\n\n\nNULL\n \n\n\n\n\n\nTHREADS\n\n\nThe THREADS section of a Java dump file provides summary information about the VM thread pool and detailed information about Java threads, native threads, and stack traces. Understanding the content of this section can help you diagnose problems that are caused by blocked or waiting threads.\n\n\nA Java thread runs on a native thread. Several lines are recorded for each Java thread in the \nThread Details\n subsection, which include the following key pieces of information:\n\n\n\n\n3XMTHREADINFO\n: The thread name, address information for the VM thread structures and Java thread object, the thread state, and thread priority.\n\n\n3XMJAVALTHREAD\n: The Java thread ID and daemon status from the thread object.\n\n\n3XMTHREADINFO1\n: The native operating system thread ID, priority, scheduling policy, internal VM thread state, and VM thread flags.\n\n\n3XMTHREADINFO2\n: The native stack address range.\n\n\n3XMTHREADINFO3\n: Java callstack information (\n4XESTACKTRACE\n) or Native call stack information (\n4XENATIVESTACK\n).\n\n\n5XESTACKTRACE\n: This line indicates whether locks were taken by a specific method.\n\n\n\n\nJava thread priorities are mapped to operating system priority values. Thread states are shown in the following table:\n\n\n\n\n\n\n\n\nThread state value\n\n\nStatus\n\n\nDescription\n\n\n\n\n\n\n\n\n\n\nR\n\n\nRunnable\n\n\nThe thread is able to run\n\n\n\n\n\n\nCW\n\n\nCondition Wait\n\n\nThe thread is waiting\n\n\n\n\n\n\nS\n\n\nSuspended\n\n\nThe thread is suspended by another thread\n\n\n\n\n\n\nZ\n\n\nZombie\n\n\nThe thread is destroyed\n\n\n\n\n\n\nP\n\n\nParked\n\n\nThe thread is parked by \njava.util.concurrent\n\n\n\n\n\n\nB\n\n\nBlocked\n\n\nThe thread is waiting to obtain a lock\n\n\n\n\n\n\n\n\nFor threads that are parked (P), blocked (B), or waiting (CW), an additional line (\n3XMTHREADBLOCK\n) is included in the output that shows what the thread is parked on, blocked on, or waiting for.\n\n\nFor clarity, the following example shows a shortened version of a typical THREADS section, where \n...\n indicates that lines are removed:\n\n\nNULL\n \n------------------------------------------------------------------------\n\n\n0\nSECTION\n \nTHREADS\n \nsubcomponent\n \ndump\n \nroutine\n\n\nNULL\n \n=================================\n\n\nNULL\n\n\n1\nXMPOOLINFO\n \nJVM\n \nThread\n \npool\n \ninfo\n:\n\n\n2\nXMPOOLTOTAL\n \nCurrent\n \ntotal\n \nnumber\n \nof\n \npooled\n \nthreads\n:\n \n18\n\n\n2\nXMPOOLLIVE\n \nCurrent\n \ntotal\n \nnumber\n \nof\n \nlive\n \nthreads\n:\n \n16\n\n\n2\nXMPOOLDAEMON\n \nCurrent\n \ntotal\n \nnumber\n \nof\n \nlive\n \ndaemon\n \nthreads\n:\n \n15\n\n\nNULL\n \n\n1\nXMTHDINFO\n \nThread\n \nDetails\n\n\nNULL\n \n\n3\nXMTHREADINFO\n \n\"JIT Diagnostic Compilation Thread-7 Suspended\"\n \nJ9VMThread\n:\n0x0000000000EFC500\n,\n \nomrthread_t:\n0x00007FF4F00A77E8\n,\n \njava\n/\nlang\n/\nThread\n:\n0x00000000FFE97480\n,\n \nstate\n:\nR\n,\n \nprio\n=\n10\n\n\n3\nXMJAVALTHREAD\n \n(\njava\n/\nlang\n/\nThread\n \ngetId:\n0xA\n,\n \nisDaemon:true\n)\n\n\n3\nXMTHREADINFO1\n \n(\nnative\n \nthread\n \nID\n:\n0x7657\n,\n \nnative\n \npriority\n:\n0xB\n,\n \nnative\n \npolicy\n:\nUNKNOWN\n,\n \nvmstate\n:\nCW\n,\n \nvm\n \nthread\n \nflags\n:\n0x00000081\n)\n\n\n3\nXMTHREADINFO2\n \n(\nnative\n \nstack\n \naddress\n \nrange\n \nfrom\n:\n0x00007FF4CCC36000\n,\n \nto\n:\n0x00007FF4CCD36000\n,\n \nsize\n:\n0x100000\n)\n\n\n3\nXMCPUTIME\n \nCPU\n \nusage\n \ntotal\n:\n \n0.000037663\n \nsecs\n,\n \ncurrent\n \ncategory=\n\"JIT\"\n\n\n3\nXMHEAPALLOC\n \nHeap\n \nbytes\n \nallocated\n \nsince\n \nlast\n \nGC\n \ncycle\n=\n0\n \n(\n0x0\n)\n\n\n3\nXMTHREADINFO3\n \nNo\n \nJava\n \ncallstack\n \nassociated\n \nwith\n \nthis\n \nthread\n\n\n3\nXMTHREADINFO3\n \nNo\n \nnative\n \ncallstack\n \navailable\n \nfor\n \nthis\n \nthread\n\n\nNULL\n\n\n...\n\n\n3\nXMTHREADINFO\n \n\"Common-Cleaner\"\n \nJ9VMThread\n:\n0x0000000000FD0100\n,\n \nomrthread_t:\n0x00007FF4F022A520\n,\n \njava\n/\nlang\n/\nThread\n:\n0x00000000FFE26F40\n,\n \nstate\n:\nCW\n,\n \nprio\n=\n8\n\n\n3\nXMJAVALTHREAD\n \n(\njava\n/\nlang\n/\nThread\n \ngetId:\n0x2\n,\n \nisDaemon:true\n)\n\n\n3\nXMTHREADINFO1\n \n(\nnative\n \nthread\n \nID\n:\n0x765A\n,\n \nnative\n \npriority\n:\n0x8\n,\n \nnative\n \npolicy\n:\nUNKNOWN\n,\n \nvmstate\n:\nCW\n,\n \nvm\n \nthread\n \nflags\n:\n0x00080181\n)\n\n\n3\nXMTHREADINFO2\n \n(\nnative\n \nstack\n \naddress\n \nrange\n \nfrom\n:\n0x00007FF4CC0B8000\n,\n \nto\n:\n0x00007FF4CC0F8000\n,\n \nsize\n:\n0x40000\n)\n\n\n3\nXMCPUTIME\n \nCPU\n \nusage\n \ntotal\n:\n \n0.000150926\n \nsecs\n,\n \ncurrent\n \ncategory=\n\"Application\"\n\n\n3\nXMTHREADBLOCK\n \nWaiting\n \non\n:\n \njava\n/\nlang\n/\nref\n/\nReferenceQueue\n@\n0x00000000FFE26A10\n \nOwned\n \nby\n:\n \n<\nunowned\n>\n\n\n3\nXMHEAPALLOC\n \nHeap\n \nbytes\n \nallocated\n \nsince\n \nlast\n \nGC\n \ncycle\n=\n0\n \n(\n0x0\n)\n\n\n3\nXMTHREADINFO3\n \nJava\n \ncallstack\n:\n\n\n4\nXESTACKTRACE\n \nat\n \njava\n/\nlang\n/\nObject\n.\nwait\n(\nNative\n \nMethod\n)\n\n\n4\nXESTACKTRACE\n \nat\n \njava\n/\nlang\n/\nObject\n.\nwait\n(\nObject\n.\njava\n:\n221\n)\n\n\n4\nXESTACKTRACE\n \nat\n \njava\n/\nlang\n/\nref\n/\nReferenceQueue\n.\nremove\n(\nReferenceQueue\n.\njava\n:\n138\n)\n\n\n5\nXESTACKTRACE\n \n(\nentered\n \nlock\n:\n \njava\n/\nlang\n/\nref\n/\nReferenceQueue\n@\n0x00000000FFE26A10\n,\n \nentry\n \ncount\n:\n \n1\n)\n\n\n4\nXESTACKTRACE\n \nat\n \njdk\n/\ninternal\n/\nref\n/\nCleanerImpl\n.\nrun\n(\nCleanerImpl\n.\njava\n:\n148\n)\n\n\n4\nXESTACKTRACE\n \nat\n \njava\n/\nlang\n/\nThread\n.\nrun\n(\nThread\n.\njava\n:\n835\n)\n\n\n4\nXESTACKTRACE\n \nat\n \njdk\n/\ninternal\n/\nmisc\n/\nInnocuousThread\n.\nrun\n(\nInnocuousThread\n.\njava\n:\n122\n)\n\n\n3\nXMTHREADINFO3\n \nNo\n \nnative\n \ncallstack\n \navailable\n \nfor\n \nthis\n \nthread\n\n\nNULL\n\n\nNULL\n\n\n3\nXMTHREADINFO\n \n\"IProfiler\"\n \nJ9VMThread\n:\n0x0000000000F03D00\n,\n \nomrthread_t:\n0x00007FF4F00B06F8\n,\n \njava\n/\nlang\n/\nThread\n:\n0x00000000FFE97B60\n,\n \nstate\n:\nR\n,\n \nprio\n=\n5\n\n\n3\nXMJAVALTHREAD\n \n(\njava\n/\nlang\n/\nThread\n \ngetId:\n0xC\n,\n \nisDaemon:true\n)\n\n\n3\nXMTHREADINFO1\n \n(\nnative\n \nthread\n \nID\n:\n0x7659\n,\n \nnative\n \npriority\n:\n0x5\n,\n \nnative\n \npolicy\n:\nUNKNOWN\n,\n \nvmstate\n:\nCW\n,\n \nvm\n \nthread\n \nflags\n:\n0x00000081\n)\n\n\n3\nXMTHREADINFO2\n \n(\nnative\n \nstack\n \naddress\n \nrange\n \nfrom\n:\n0x00007FF4F8940000\n,\n \nto\n:\n0x00007FF4F8960000\n,\n \nsize\n:\n0x20000\n)\n\n\n3\nXMCPUTIME\n \nCPU\n \nusage\n \ntotal\n:\n \n0.004753103\n \nsecs\n,\n \ncurrent\n \ncategory=\n\"JIT\"\n\n\n3\nXMHEAPALLOC\n \nHeap\n \nbytes\n \nallocated\n \nsince\n \nlast\n \nGC\n \ncycle\n=\n0\n \n(\n0x0\n)\n\n\n3\nXMTHREADINFO3\n \nNo\n \nJava\n \ncallstack\n \nassociated\n \nwith\n \nthis\n \nthread\n\n\n3\nXMTHREADINFO3\n \nNo\n \nnative\n \ncallstack\n \navailable\n \nfor\n \nthis\n \nthread\n\n\nNULL\n\n\n...\n\n\n1\nXMWLKTHDERR\n \nThe\n \nfollowing\n \nwas\n \nreported\n \nwhile\n \ncollecting\n \nnative\n \nstacks\n:\n\n\n2\nXMWLKTHDERR\n \nunable\n \nto\n \ncount\n \nthreads\n(\n3\n,\n \n-\n2\n)\n\n\nNULL\n\n\n1\nXMTHDSUMMARY\n \nThreads\n \nCPU\n \nUsage\n \nSummary\n\n\nNULL\n \n=========================\n\n\nNULL\n\n\n1\nXMTHDCATINFO\n \nWarning\n:\n \nto\n \nget\n \nmore\n \naccurate\n \nCPU\n \ntimes\n \nfor\n \nthe\n \nGC\n,\n \nthe\n \noption\n \n-\nXX\n:-\nReduceCPUMonitorOverhead\n \ncan\n \nbe\n \nused\n.\n \nSee\n \nthe\n \nuser\n \nguide\n \nfor\n \nmore\n \ninformation\n.\n\n\nNULL\n\n\n1\nXMTHDCATEGORY\n \nAll\n \nJVM\n \nattached\n \nthreads\n:\n \n0.280083000\n \nsecs\n\n\n1\nXMTHDCATEGORY\n \n|\n\n\n2\nXMTHDCATEGORY\n \n+--\nSystem\n-\nJVM\n:\n \n0.270814000\n \nsecs\n\n\n2\nXMTHDCATEGORY\n \n|\n \n|\n\n\n3\nXMTHDCATEGORY\n \n|\n \n+--\nGC\n:\n \n0.000599000\n \nsecs\n\n\n2\nXMTHDCATEGORY\n \n|\n \n|\n\n\n3\nXMTHDCATEGORY\n \n|\n \n+--\nJIT\n:\n \n0.071904000\n \nsecs\n\n\n1\nXMTHDCATEGORY\n \n|\n\n\n2\nXMTHDCATEGORY\n \n+--\nApplication\n:\n \n0.009269000\n \nsecs\n\n\nNULL\n\n\n\n\n\n\nHOOKS\n\n\nThis section shows internal VM event callbacks, which are used for diagnosing performance problems in the VM. Multiple hook interfaces are listed, which include their individual hook events. The following example shows data for the \nJ9VMHookInterface\n, including the call site location (source file:line number), start time, and duration of the last callback and the longest callback.\n\n\nNULL ------------------------------------------------------------------------\n0SECTION HOOK subcomponent dump routine\nNULL ==============================\n1HKINTERFACE MM_OMRHookInterface\nNULL ------------------------------------------------------------------------\n1HKINTERFACE MM_PrivateHookInterface\nNULL ------------------------------------------------------------------------\n1HKINTERFACE MM_HookInterface\nNULL ------------------------------------------------------------------------\n1HKINTERFACE J9VMHookInterface\nNULL ------------------------------------------------------------------------\n2HKEVENTID 1\n3HKCALLCOUNT 18\n3HKLAST Last Callback\n4HKCALLSITE trcengine.c:392\n4HKSTARTTIME Start Time: 2018-08-30T21:55:47.601\n4HKDURATION DurationMs: 0\n3HKLONGST Longest Callback\n4HKCALLSITE trcengine.c:392\n4HKSTARTTIME Start Time: 2018-08-30T21:55:47.460\n4HKDURATION DurationMs: 1\nNULL\n...\n1HKINTERFACE J9VMZipCachePoolHookInterface\nNULL ------------------------------------------------------------------------\n1HKINTERFACE J9JITHookInterface\nNULL ------------------------------------------------------------------------\n2HKEVENTID 3\n3HKCALLCOUNT 65\n3HKLAST Last Callback\n4HKCALLSITE ../common/mgmtinit.c:191\n4HKSTARTTIME Start Time: 2018-08-30T21:55:47.601\n4HKDURATION DurationMs: 0\n3HKLONGST Longest Callback\n4HKCALLSITE ../common/mgmtinit.c:191\n4HKSTARTTIME Start Time: 2018-08-30T21:55:47.486\n4HKDURATION DurationMs: 0\n...\nNULL\n\n\n\n\n\nSHARED CLASSES\n\n\nIf the shared classes cache is enabled at run time, the information provided in a Java dump file describes settings that were used when creating the cache, together with summary information about the size and content of the cache.\n\n\nIn the following example, the shared classes cache was created with a Class Debug Area (\n-Xnolinenumbers=false\n). Byte code instrumentation (BCI) is enabled, which is the default, and VMs sharing the cache are allowed to store classpaths, which is also the default.\n\n\nThe \nCache Summary\n shows a cache size (\n2SCLTEXTCSZ\n) of 16776608 bytes, with a soft maximum size (\n2SCLTEXTSMB\n) also of 16776608 bytes, which leaves 12691668 bytes of free space (\n2SCLTEXTFRB\n). The size of the Class Debug Area (\n2SCLTEXTDAS\n) is 1331200 bytes and only 11% of this space is used.\n\n\nIn the \nCache Memory Status\n subsection, the line \n2SCLTEXTCMDT\n indicates the name and location of the shared cache and \ncr\n indicates that the cache is a 64-bit compressed references cache.\n\n\nNULL ------------------------------------------------------------------------\n0SECTION SHARED CLASSES subcomponent dump routine\nNULL ========================================\nNULL\n1SCLTEXTCRTW Cache Created With\nNULL ------------------\nNULL\n2SCLTEXTXNL -Xnolinenumbers = false\n2SCLTEXTBCI BCI Enabled = true\n2SCLTEXTBCI Restrict Classpaths = false\nNULL\n1SCLTEXTCSUM Cache Summary\nNULL ------------------\nNULL\n2SCLTEXTNLC No line number content = false\n2SCLTEXTLNC Line number content = true\nNULL\n2SCLTEXTRCS ROMClass start address = 0x00007F423061C000\n2SCLTEXTRCE ROMClass end address = 0x00007F42307B9A28\n2SCLTEXTMSA Metadata start address = 0x00007F42313D42FC\n2SCLTEXTCEA Cache end address = 0x00007F4231600000\n2SCLTEXTRTF Runtime flags = 0x00102001ECA6028B\n2SCLTEXTCGN Cache generation = 35\nNULL\n2SCLTEXTCSZ Cache size = 16776608\n2SCLTEXTSMB Softmx bytes = 16776608\n2SCLTEXTFRB Free bytes = 12691668\n2SCLTEXTRCB ROMClass bytes = 1694248\n2SCLTEXTAOB AOT code bytes = 0\n2SCLTEXTADB AOT data bytes = 0\n2SCLTEXTAHB AOT class hierarchy bytes = 32\n2SCLTEXTATB AOT thunk bytes = 0\n2SCLTEXTARB Reserved space for AOT bytes = -1\n2SCLTEXTAMB Maximum space for AOT bytes = -1\n2SCLTEXTJHB JIT hint bytes = 308\n2SCLTEXTJPB JIT profile bytes = 2296\n2SCLTEXTJRB Reserved space for JIT data bytes = -1\n2SCLTEXTJMB Maximum space for JIT data bytes = -1\n2SCLTEXTNOB Java Object bytes = 0\n2SCLTEXTZCB Zip cache bytes = 919328\n2SCLTEXTRWB ReadWrite bytes = 114080\n2SCLTEXTJCB JCL data bytes = 0\n2SCLTEXTBDA Byte data bytes = 0\n2SCLTEXTMDA Metadata bytes = 23448\n2SCLTEXTDAS Class debug area size = 1331200\n2SCLTEXTDAU Class debug area % used = 11%\n2SCLTEXTDAN Class LineNumberTable bytes = 156240\n2SCLTEXTDAV Class LocalVariableTable bytes = 0\nNULL\n2SCLTEXTNRC Number ROMClasses = 595\n2SCLTEXTNAM Number AOT Methods = 0\n2SCLTEXTNAD Number AOT Data Entries = 0\n2SCLTEXTNAH Number AOT Class Hierarchy = 1\n2SCLTEXTNAT Number AOT Thunks = 0\n2SCLTEXTNJH Number JIT Hints = 14\n2SCLTEXTNJP Number JIT Profiles = 20\n2SCLTEXTNCP Number Classpaths = 1\n2SCLTEXTNUR Number URLs = 0\n2SCLTEXTNTK Number Tokens = 0\n2SCLTEXTNOJ Number Java Objects = 0\n2SCLTEXTNZC Number Zip Caches = 5\n2SCLTEXTNJC Number JCL Entries = 0\n2SCLTEXTNST Number Stale classes = 0\n2SCLTEXTPST Percent Stale classes = 0%\nNULL\n2SCLTEXTCPF Cache is 24% full\nNULL\n1SCLTEXTCMST Cache Memory Status\nNULL ------------------\n1SCLTEXTCNTD Cache Name Feature Memory type Cache path\nNULL\n2SCLTEXTCMDT sharedcc_doc-javacore CR Memory mapped file /tmp/javasharedresources/C290M4F1A64P_sharedcc_doc-javacore_G35\nNULL\n1SCLTEXTCMST Cache Lock Status\nNULL ------------------\n1SCLTEXTCNTD Lock Name Lock type TID owning lock\nNULL\n2SCLTEXTCWRL Cache write lock File lock Unowned\n2SCLTEXTCRWL Cache read/write lock File lock Unowned\nNULL\n\n\n\n\n\nCLASSES\n\n\nThe classes section shows information about class loaders. The first part is a summary that records each available class loader (\n2CLTEXTCLLOADER\n) followed by the number of libraries and classes that it loaded. This information is followed by a more detailed list of libraries (\n1CLTEXTCLLIB\n) and classes (\n1CLTEXTCLLO\n) that are loaded.\n\n\nIn the example you can see that the \njava/lang/InternalAnonymousClassLoader\n loaded 2 classes, \njdk/internal/loader/BuiltinClassLoader$$Lambda$2/00000000F03876A0(0x0000000001030F00)\n and \njdk/internal/loader/BuiltinClassLoader$$Lambda$1/00000000F00D2460(0x0000000001018A00)\n.\n\n\nNULL ------------------------------------------------------------------------\n0SECTION CLASSES subcomponent dump routine\nNULL =================================\n1CLTEXTCLLOS Classloader summaries\n1CLTEXTCLLSS 12345678: 1=primordial,2=extension,3=shareable,4=middleware,5=system,6=trusted,7=application,8=delegating\n2CLTEXTCLLOADER p---st-- Loader *System*(0x00000000FFE1D258)\n3CLNMBRLOADEDLIB Number of loaded libraries 5\n3CLNMBRLOADEDCL Number of loaded classes 638\n2CLTEXTCLLOADER -x--st-- Loader jdk/internal/loader/ClassLoaders$PlatformClassLoader(0x00000000FFE1D4F0), Parent *none*(0x0000000000000000)\n3CLNMBRLOADEDLIB Number of loaded libraries 0\n3CLNMBRLOADEDCL Number of loaded classes 0\n2CLTEXTCLLOADER ----st-- Loader java/lang/InternalAnonymousClassLoader(0x00000000FFE1DFD0), Parent *none*(0x0000000000000000)\n3CLNMBRLOADEDLIB Number of loaded libraries 0\n3CLNMBRLOADEDCL Number of loaded classes 2\n2CLTEXTCLLOADER -----ta- Loader jdk/internal/loader/ClassLoaders$AppClassLoader(0x00000000FFE1DAD0), Parent jdk/internal/loader/ClassLoaders$PlatformClassLoader(0x00000000FFE1D4F0)\n3CLNMBRLOADEDLIB Number of loaded libraries 0\n3CLNMBRLOADEDCL Number of loaded classes 0\n1CLTEXTCLLIB ClassLoader loaded libraries\n2CLTEXTCLLIB Loader *System*(0x00000000FFE1D258)\n3CLTEXTLIB /home/me/openj9-openjdk-jdk9/build/linux-x86_64-normal-server-release/images/jdk/lib/compressedrefs/jclse9_29\n3CLTEXTLIB /home/me/openj9-openjdk-jdk9/build/linux-x86_64-normal-server-release/images/jdk/lib/java\n3CLTEXTLIB /home/me/openj9-openjdk-jdk9/build/linux-x86_64-normal-server-release/images/jdk/lib/compressedrefs/j9jit29\n3CLTEXTLIB /home/me/openj9-openjdk-jdk9/build/linux-x86_64-normal-server-release/images/jdk/lib/zip\n3CLTEXTLIB /home/me/openj9-openjdk-jdk9/build/linux-x86_64-normal-server-release/images/jdk/lib/nio\n1CLTEXTCLLOD ClassLoader loaded classes\n2CLTEXTCLLOAD Loader *System*(0x00000000FFE1D258)\n3CLTEXTCLASS [Ljava/lang/Thread$State;(0x0000000001056400)\n...\n2CLTEXTCLLOAD Loader jdk/internal/loader/ClassLoaders$PlatformClassLoader(0x00000000FFE1D4F0)\n2CLTEXTCLLOAD Loader java/lang/InternalAnonymousClassLoader(0x00000000FFE1DFD0)\n3CLTEXTCLASS jdk/internal/loader/BuiltinClassLoader$$Lambda$2/00000000F03876A0(0x0000000001030F00)\n3CLTEXTCLASS jdk/internal/loader/BuiltinClassLoader$$Lambda$1/00000000F00D2460(0x0000000001018A00)\n2CLTEXTCLLOAD Loader jdk/internal/loader/ClassLoaders$AppClassLoader(0x00000000FFE1DAD0)\n\n\n\n\n\nScenarios\n\n\nGeneral Protection Fault\n\n\nIn this scenario, a Java application has crashed due to a General Protection Fault (GPF), automatically generating a Java dump file.\n\n\nThe first section of the file (TITLE) tells you that the GPF triggered the Java dump.\n\n\n0SECTION TITLE subcomponent dump routine\nNULL ===============================\n1TICHARSET UTF-8\n1TISIGINFO Dump Event \"gpf\" (00002000) received\n1TIDATETIME Date: 2018/09/24 at 15:18:03:115\n1TINANOTIME System nanotime: 4498949283020796\n1TIFILENAME Javacore filename: /home/test/JNICrasher/javacore.20180924.151801.29399.0002.txt\n1TIREQFLAGS Request Flags: 0x81 (exclusive+preempt)\n1TIPREPSTATE Prep State: 0x100 (trace_disabled)\n1TIPREPINFO Exclusive VM access not taken: data may not be consistent across javacore sections\n\n\n\n\n\nTo troubleshoot this problem, you need to know which thread caused the GPF to occur. The thread that was running at the time of the crash is reported as the \ncurrent thread\n\nin the THREADS section of the Java dump. Here is an extract from the THREADS section:\n\n\nNULL ------------------------------------------------------------------------\n0SECTION THREADS subcomponent dump routine\nNULL =================================\nNULL\n1XMPOOLINFO JVM Thread pool info:\n2XMPOOLTOTAL Current total number of pooled threads: 16\n2XMPOOLLIVE Current total number of live threads: 15\n2XMPOOLDAEMON Current total number of live daemon threads: 14\nNULL \n1XMCURTHDINFO Current thread\n3XMTHREADINFO \"main\" J9VMThread:0xB6B60E00, omrthread_t:0xB6B049D8, java/lang/Thread:0xB55444D0, state:R, prio=5\n3XMJAVALTHREAD (java/lang/Thread getId:0x1, isDaemon:false)\n3XMTHREADINFO1 (native thread ID:0x72D8, native priority:0x5, native policy:UNKNOWN, vmstate:R, vm thread flags:0x00000000)\n3XMTHREADINFO2 (native stack address range from:0xB6CE3000, to:0xB74E4000, size:0x801000)\n3XMCPUTIME CPU usage total: 0.319865924 secs, current category=\"Application\"\n3XMHEAPALLOC Heap bytes allocated since last GC cycle=778008 (0xBDF18)\n3XMTHREADINFO3 Java callstack:\n4XESTACKTRACE at JNICrasher.doSomethingThatCrashes(Native Method)\n4XESTACKTRACE at JNICrasher.main(JNICrasher.java:7)\n3XMTHREADINFO3 Native callstack:\n4XENATIVESTACK (0xB6C6F663 [libj9prt29.so+0x3b663])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB6C6F1CE [libj9prt29.so+0x3b1ce])\n4XENATIVESTACK (0xB6C6F2C6 [libj9prt29.so+0x3b2c6])\n4XENATIVESTACK (0xB6C6ED93 [libj9prt29.so+0x3ad93])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB6C6ED07 [libj9prt29.so+0x3ad07])\n4XENATIVESTACK (0xB6C6AA3D [libj9prt29.so+0x36a3d])\n4XENATIVESTACK (0xB6C6C3A4 [libj9prt29.so+0x383a4])\n4XENATIVESTACK (0xB667FA19 [libj9dmp29.so+0xfa19])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB66878CF [libj9dmp29.so+0x178cf])\n4XENATIVESTACK (0xB6688083 [libj9dmp29.so+0x18083])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB6680C0D [libj9dmp29.so+0x10c0d])\n4XENATIVESTACK (0xB667F9D7 [libj9dmp29.so+0xf9d7])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB668B02F [libj9dmp29.so+0x1b02f])\n4XENATIVESTACK (0xB668B4D3 [libj9dmp29.so+0x1b4d3])\n4XENATIVESTACK (0xB66740F1 [libj9dmp29.so+0x40f1])\n4XENATIVESTACK (0xB66726FA [libj9dmp29.so+0x26fa])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB66726A9 [libj9dmp29.so+0x26a9])\n4XENATIVESTACK (0xB6676AE4 [libj9dmp29.so+0x6ae4])\n4XENATIVESTACK (0xB668D75A [libj9dmp29.so+0x1d75a])\n4XENATIVESTACK (0xB6A28DD4 [libj9vm29.so+0x81dd4])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB6A289EE [libj9vm29.so+0x819ee])\n4XENATIVESTACK (0xB6A29A40 [libj9vm29.so+0x82a40])\n4XENATIVESTACK (0xB6C52B6A [libj9prt29.so+0x1eb6a])\n4XENATIVESTACK __kernel_rt_sigreturn+0x0 (0xB7747410)\n4XENATIVESTACK (0xB75330B6 [libffi29.so+0x50b6])\n4XENATIVESTACK ffi_raw_call+0xad (0xB7531C53 [libffi29.so+0x3c53])\n4XENATIVESTACK (0xB69BE4AB [libj9vm29.so+0x174ab])\n4XENATIVESTACK (0xB6A665BC [libj9vm29.so+0xbf5bc])\n4XENATIVESTACK (0xB6A15552 [libj9vm29.so+0x6e552])\n4XENATIVESTACK (0xB6A30894 [libj9vm29.so+0x89894])\n4XENATIVESTACK (0xB6A6F169 [libj9vm29.so+0xc8169])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB6A6F1FA [libj9vm29.so+0xc81fa])\n4XENATIVESTACK (0xB6A30994 [libj9vm29.so+0x89994])\n4XENATIVESTACK (0xB6A2CE4C [libj9vm29.so+0x85e4c])\n4XENATIVESTACK (0xB770487D [libjli.so+0x787d])\n4XENATIVESTACK (0xB7719F72 [libpthread.so.0+0x6f72])\n4XENATIVESTACK clone+0x5e (0xB763543E [libc.so.6+0xee43e])\n\n\n\n\n\nThe extract tells you that the current thread was \njava/lang/Thread\n, and information is provided about the Java callstack and native callstack\n(\n3XMTHREADINFO3\n) at the point at which the crash occurred. To simulate a crash caused by a bug in an application, this example\ncalls a JNI method whose native implementation causes a crash. The Java callstack shows the call to the JNI native method (\nJNIcrasher\n), and the\nnative callstack shows the point of failure. In this example, the native call stack does not include any function names to help you isolate the error\nin the native code. You can get this information from a system dump, which is usually produced alongside the Java dump. Open the\nsystem dump with the \nDump viewer\n and use the \ninfo thread\n command to print the Java and native stack for the current thread.\n\n\nJava OutOfMemoryError\n\n\nIn this scenario, the Java heap runs out of memory, causing an \nOutOfMemoryError\n, which automatically generates a Java dump file.\n\n\nThe first section of the file (TITLE) tells you that a \nsysthrow\n event triggered the Java dump as a result of an OOM (\njava/lang/OutOfMemoryError\n) for\nJava heap space.\n\n\n0SECTION TITLE subcomponent dump routine\nNULL ===============================\n1TICHARSET UTF-8\n1TISIGINFO Dump Event \"systhrow\" (00040000) Detail \"java/lang/OutOfMemoryError\" \"Java heap space\" received\n1TIDATETIME Date: 2018/09/14 at 15:29:42:709\n1TINANOTIME System nanotime: 3635648876608448\n1TIFILENAME Javacore filename: /home/cheesemp/test/javacore.20180914.152929.18885.0003.txt\n1TIREQFLAGS Request Flags: 0x81 (exclusive+preempt)\n1TIPREPSTATE Prep State: 0x104 (exclusive_vm_access+trace_disabled)\n\n\n\n\n\nThe MEMINFO section records how much memory is allocated to the Java heap (\n1STHEAPTYPE Object Memory\n), how much is in use, and how much is free. Solving\nyour problem might be as simple as setting a larger heap size when you start your application.\n\n\nIf you don't know what size the Java heap was set to, you might find that information in the ENVINFO section, which records the command line options that\nwere used when the application started. Look or search for the \n1CIUSERARGS UserArgs:\n string and review the entries recorded for all lines that\nstart \n2CIUSERARG\n. The Java heap size is set by the \n-Xmx\n option. If the size has not been set on the command line by \n-Xmx\n, the default value applies, which\nyou can find in \nDefault Settings\n.\n\n\nIn this scenario the solution to the problem is not an adjustment to the Java heap size. Here is the MEMINFO section:\n\n\n0SECTION MEMINFO subcomponent dump routine\nNULL =================================\nNULL\n1STHEAPTYPE Object Memory\nNULL id start end size space/region\n1STHEAPSPACE 0xB6B49D20 -- -- -- Generational\n1STHEAPREGION 0xB6B4A078 0x95750000 0xB5470000 0x1FD20000 Generational/Tenured Region\n1STHEAPREGION 0xB6B49F10 0xB5470000 0xB54C0000 0x00050000 Generational/Nursery Region\n1STHEAPREGION 0xB6B49DA8 0xB54C0000 0xB5750000 0x00290000 Generational/Nursery Region\nNULL\n1STHEAPTOTAL Total memory: 536870912 (0x20000000)\n1STHEAPINUSE Total memory in use: 302603160 (0x12095B98)\n1STHEAPFREE Total memory free: 234267752 (0x0DF6A468)\n\n\n\n\n\nThe output shows that only 56% of the Java heap is in use, so this suggests that the application is trying to do something sub-optimal. To\ninvestigate further you need to work out which thread was the current thread when the OOM occurred to see what it was trying to do. As in the previous scenario, you can find the\n\ncurrent thread\n in the THREADS section. Here is an extract from the output:\n\n\n0\nSECTION\n \nTHREADS\n \nsubcomponent\n \ndump\n \nroutine\n\n\nNULL\n \n=================================\n\n\nNULL\n\n\n1\nXMPOOLINFO\n \nJVM\n \nThread\n \npool\n \ninfo\n:\n\n\n2\nXMPOOLTOTAL\n \nCurrent\n \ntotal\n \nnumber\n \nof\n \npooled\n \nthreads\n:\n \n16\n\n\n2\nXMPOOLLIVE\n \nCurrent\n \ntotal\n \nnumber\n \nof\n \nlive\n \nthreads\n:\n \n16\n\n\n2\nXMPOOLDAEMON\n \nCurrent\n \ntotal\n \nnumber\n \nof\n \nlive\n \ndaemon\n \nthreads\n:\n \n15\n\n\nNULL\n\n\n1\nXMCURTHDINFO\n \nCurrent\n \nthread\n\n\n3\nXMTHREADINFO\n \n\"main\"\n \nJ9VMThread\n:\n0xB6B60C00\n,\n \nomrthread_t\n:\n0xB6B049D8\n,\n \njava\n/\nlang\n/\nThread\n:\n0x95764520\n,\n \nstate\n:\nR\n,\n \nprio\n=\n5\n\n\n3\nXMJAVALTHREAD\n \n(\njava\n/\nlang\n/\nThread\n \ngetId\n:\n0x1\n,\n \nisDaemon\n:\nfalse\n)\n\n\n3\nXMTHREADINFO1\n \n(\nnative\n \nthread\n \nID\n:\n0x49C6\n,\n \nnative\n \npriority\n:\n0x5\n,\n \nnative\n \npolicy\n:\nUNKNOWN\n,\n \nvmstate\n:\nR\n,\n \nvm\n \nthread\n \nflags\n:\n0x00001020\n)\n\n\n3\nXMTHREADINFO2\n \n(\nnative\n \nstack\n \naddress\n \nrange\n \nfrom\n:\n0xB6CB5000\n,\n \nto\n:\n0xB74B6000\n,\n \nsize\n:\n0x801000\n)\n\n\n3\nXMCPUTIME\n \nCPU\n \nusage\n \ntotal\n:\n \n8.537823831\n \nsecs\n,\n \ncurrent\n \ncategory\n=\n\"Application\"\n\n\n3\nXMHEAPALLOC\n \nHeap\n \nbytes\n \nallocated\n \nsince\n \nlast\n \nGC\n \ncycle\n=\n0\n \n(\n0x0\n)\n\n\n3\nXMTHREADINFO3\n \nJava\n \ncallstack\n:\n\n\n4\nXESTACKTRACE\n \nat\n \njava\n/\nlang\n/\nStringBuffer\n.\nensureCapacityImpl\n(\nStringBuffer\n.\njava\n:\n696\n)\n\n\n4\nXESTACKTRACE\n \nat\n \njava\n/\nlang\n/\nStringBuffer\n.\nappend\n(\nStringBuffer\n.\njava\n:\n486\n(\nCompiled\n \nCode\n))\n\n\n5\nXESTACKTRACE\n \n(\nentered\n \nlock\n:\n \njava\n/\nlang\n/\nStringBuffer\n@0x957645B8\n,\n \nentry\n \ncount\n:\n \n1\n)\n\n\n4\nXESTACKTRACE\n \nat\n \njava\n/\nlang\n/\nStringBuffer\n.\nappend\n(\nStringBuffer\n.\njava\n:\n428\n(\nCompiled\n \nCode\n))\n\n\n4\nXESTACKTRACE\n \nat\n \nHeapBreaker\n.\nmain\n(\nHeapBreaker\n.\njava\n:\n34\n(\nCompiled\n \nCode\n))\n\n\n3\nXMTHREADINFO3\n \nNative\n \ncallstack\n:\n\n\n4\nXENATIVESTACK\n \n(\n0xB6C535B3\n \n[\nlibj9prt29\n.\nso\n+\n0x3b5b3\n])\n\n\n4\nXENATIVESTACK\n \n(\n0xB6C36F3E\n \n[\nlibj9prt29\n.\nso\n+\n0x1ef3e\n])\n\n\n4\nXENATIVESTACK\n \n(\n0xB6C5311E\n \n[\nlibj9prt29\n.\nso\n+\n0x3b11e\n])\n\n\n4\nXENATIVESTACK\n \n(\n0xB6C53216\n \n[\nlibj9prt29\n.\nso\n+\n0x3b216\n])\n\n\n4\nXENATIVESTACK\n \n(\n0xB6C52CE3\n \n[\nlibj9prt29\n.\nso\n+\n0x3ace3\n])\n\n\n4\nXENATIVESTACK\n \n(\n0xB6C36F3E\n \n[\nlibj9prt29\n.\nso\n+\n0x1ef3e\n])\n\n\n4\nXENATIVESTACK\n \n(\n0xB6C52C57\n \n[\nlibj9prt29\n.\nso\n+\n0x3ac57\n])\n\n\n4\nXENATIVESTACK\n \n(\n0xB6C4E9CD\n \n[\nlibj9prt29\n.\nso\n+\n0x369cd\n])\n\n\n4\nXENATIVESTACK\n \n(\n0xB6C502FA\n \n[\nlibj9prt29\n.\nso\n+\n0x382fa\n])\n\n\n\n\n\n\nTo simulate a Java \nOutOfMemoryError\n, this example application repeatedly appends characters to a \nStringBuffer\n object in an infinite loop. The Java callstack shows the \nHeapBreaker.main\n method appending characters (\njava/lang/StringGuffer.append\n) until the method \njava/lang/StringBuffer.ensureCapacityImpl()\n throws the \nOutOfMemoryError\n.\n\n\nStringBuffer objects are wrappers for character arrays (\nchar[]\n) and when the capacity of the underlying array is reached, the contents are automatically copied into a new, larger array. The new array is created in the \nStringBuffer.ensureCapacity()\n method, which more or less doubles the size of the old array. In this scenario, the array takes up all the remaining space in the Java heap.\n\n\nThe MEMINFO section of the Java dump file can also tell you when an unexpectedly large allocation request causes an OOM. Look for the GC History (\n1STGCHTYPE\n) section, which details allocation requests that trigger GC activity. In the sample output you can see that a large allocation request (\nrequestedbytes=603979784\n) triggered a global GC. When the GC could not free up sufficient space in the heap to satisfy the request, the allocation failure generated the OOM.\n\n\n1STGCHTYPE GC History \n3STHSTTYPE 14:29:29:580239000 GMT j9mm.101 - J9AllocateIndexableObject() returning NULL! 0 bytes requested for object of class B6BBC300 from memory space 'Generational' id=B6B49D20\n3STHSTTYPE 14:29:29:579916000 GMT j9mm.134 - Allocation failure end: newspace=2686912/3014656 oldspace=231597224/533856256 loa=5338112/5338112\n3STHSTTYPE 14:29:29:579905000 GMT j9mm.470 - Allocation failure cycle end: newspace=2686912/3014656 oldspace=231597224/533856256 loa=5338112/5338112\n3STHSTTYPE 14:29:29:579859000 GMT j9mm.475 - GlobalGC end: workstackoverflow=0 overflowcount=0 memory=234284136/536870912\n3STHSTTYPE 14:29:29:579807000 GMT j9mm.90 - GlobalGC collect complete\n3STHSTTYPE 14:29:29:579776000 GMT j9mm.137 - Compact end: bytesmoved=301989896\n3STHSTTYPE 14:29:29:313899000 GMT j9mm.136 - Compact start: reason=compact to meet allocation\n3STHSTTYPE 14:29:29:313555000 GMT j9mm.57 - Sweep end\n3STHSTTYPE 14:29:29:310772000 GMT j9mm.56 - Sweep start\n3STHSTTYPE 14:29:29:310765000 GMT j9mm.94 - Class unloading end: classloadersunloaded=0 classesunloaded=0\n3STHSTTYPE 14:29:29:310753000 GMT j9mm.60 - Class unloading start\n3STHSTTYPE 14:29:29:310750000 GMT j9mm.55 - Mark end\n3STHSTTYPE 14:29:29:306013000 GMT j9mm.54 - Mark start\n3STHSTTYPE 14:29:29:305957000 GMT j9mm.474 - GlobalGC start: globalcount=9\n3STHSTTYPE 14:29:29:305888000 GMT j9mm.475 - GlobalGC end: workstackoverflow=0 overflowcount=0 memory=234284136/536870912\n3STHSTTYPE 14:29:29:305837000 GMT j9mm.90 - GlobalGC collect complete\n3STHSTTYPE 14:29:29:305808000 GMT j9mm.137 - Compact end: bytesmoved=189784\n3STHSTTYPE 14:29:29:298042000 GMT j9mm.136 - Compact start: reason=compact to meet allocation\n3STHSTTYPE 14:29:29:297695000 GMT j9mm.57 - Sweep end\n3STHSTTYPE 14:29:29:291696000 GMT j9mm.56 - Sweep start\n3STHSTTYPE 14:29:29:291692000 GMT j9mm.55 - Mark end\n3STHSTTYPE 14:29:29:284994000 GMT j9mm.54 - Mark start\n3STHSTTYPE 14:29:29:284941000 GMT j9mm.474 - GlobalGC start: globalcount=8\n3STHSTTYPE 14:29:29:284916000 GMT j9mm.135 - Exclusive access: exclusiveaccessms=0.016 meanexclusiveaccessms=0.016 threads=0 lastthreadtid=0xB6B61100 beatenbyotherthread=0\n3STHSTTYPE 14:29:29:284914000 GMT j9mm.469 - Allocation failure cycle start: newspace=2678784/3014656 oldspace=80601248/533856256 loa=5338112/5338112 requestedbytes=603979784\n3STHSTTYPE 14:29:29:284893000 GMT j9mm.470 - Allocation failure cycle end: newspace=2678784/3014656 oldspace=80601248/533856256 loa=5338112/5338112\n3STHSTTYPE 14:29:29:284858000 GMT j9mm.560 - LocalGC end: rememberedsetoverflow=0 causedrememberedsetoverflow=0 scancacheoverflow=0 failedflipcount=0 failedflipbytes=0 failedtenurecount=0 failedtenurebytes=0 flipcount=2 flipbytes=64 newspace=2678784/3014656 oldspace=80601248/533856256 loa=5338112/5338112 tenureage=0\n3STHSTTYPE 14:29:29:284140000 GMT j9mm.140 - Tilt ratio: 89\n3STHSTTYPE 14:29:29:283160000 GMT j9mm.64 - LocalGC start: globalcount=8 scavengecount=335 weakrefs=0 soft=0 phantom=0 finalizers=0\n3STHSTTYPE 14:29:29:283123000 GMT j9mm.135 - Exclusive access: exclusiveaccessms=0.016 meanexclusiveaccessms=0.016 threads=0 lastthreadtid=0xB6B61100 beatenbyotherthread=0\n3STHSTTYPE 14:29:29:283120000 GMT j9mm.469 - Allocation failure cycle start: newspace=753616/3014656 oldspace=80601248/533856256 loa=5338112/5338112 requestedbytes=603979784\n3STHSTTYPE 14:29:29:283117000 GMT j9mm.133 - Allocation failure start: newspace=753616/3014656 oldspace=80601248/533856256 loa=5338112/5338112 requestedbytes=603979784\n3STHSTTYPE 14:29:29:269762000 GMT j9mm.134 - Allocation failure end: newspace=2686928/3014656 oldspace=80601248/533856256 loa=5338112/5338112\n3STHSTTYPE 14:29:29:269751000 GMT j9mm.470 - Allocation failure cycle end: newspace=2686976/3014656 oldspace=80601248/533856256 loa=5338112/5338112\n3STHSTTYPE 14:29:29:269718000 GMT j9mm.560 - LocalGC end: rememberedsetoverflow=0 causedrememberedsetoverflow=0 scancacheoverflow=0 failedflipcount=0 failedflipbytes=0 failedtenurecount=0 failedtenurebytes=0 flipcount=0 flipbytes=0 newspace=2686976/3014656 oldspace=80601248/533856256 loa=5338112/5338112 tenureage=0\n3STHSTTYPE 14:29:29:268981000 GMT j9mm.140 - Tilt ratio: 89\n3STHSTTYPE 14:29:29:268007000 GMT j9mm.64 - LocalGC start: globalcount=8 scavengecount=334 weakrefs=0 soft=0 phantom=0 finalizers=0\n3STHSTTYPE 14:29:29:267969000 GMT j9mm.135 - Exclusive access: exclusiveaccessms=0.016 meanexclusiveaccessms=0.016 threads=0 lastthreadtid=0xB6B61100 beatenbyotherthread=0\n3STHSTTYPE 14:29:29:267966000 GMT j9mm.469 - Allocation failure cycle start: newspace=0/3014656 oldspace=80601248/533856256 loa=5338112/5338112 requestedbytes=48\n3STHSTTYPE 14:29:29:267963000 GMT j9mm.133 - Allocation failure start: newspace=0/3014656 oldspace=80601248/533856256 loa=5338112/5338112 requestedbytes=48\n3STHSTTYPE 14:29:29:249015000 GMT j9mm.134 - Allocation failure end: newspace=2686928/3014656 oldspace=80601248/533856256 loa=5338112/5338112\n3STHSTTYPE 14:29:29:249003000 GMT j9mm.470 - Allocation failure cycle end: newspace=2686976/3014656 oldspace=80601248/533856256 loa=5338112/5338112\n3STHSTTYPE 14:29:29:248971000 GMT j9mm.560 - LocalGC end: rememberedsetoverflow=0 causedrememberedsetoverflow=0 scancacheoverflow=0 failedflipcount=0 failedflipbytes=0 failedtenurecount=0 failedtenurebytes=0 flipcount=0 flipbytes=0 newspace=2686976/3014656 oldspace=80601248/533856256 loa=5338112/5338112 tenureage=0\n\n\n\n\n\nAlthough the Java code that was used in this scenario deliberately triggered an \nOutOfMemoryError\n in a pronounced way, similar allocation issues can and do occur when dealing with large data sets such as XML files.\n\n\nThe next step in diagnosing the problem is to open the system dump that gets generated automatically when an \nOutOfMemoryError\n occurs. Open\nthe dump with the \nEclipse Memory Analyzer tool (MAT)\n and search for the \nStringBuffer\n object, which should provide further clues about what went wrong. A common example is seeing the same \nString\n duplicated over and over again, which might indicate that code is stuck in a loop.\n\n\n \nNote:\n If you want to use MAT to analyze your system dump, you must install the Diagnostic Tool Framework for Java (DTFJ) plugin in the Eclipse IDE. Select the following menu items:\n\n\nHelp > Install New Software > Work with \"IBM Diagnostic Tool Framework for Java\" > \n\n\n\n\n\nIf, unlike the previous scenario, you receive an \nOutOfMemoryError\n and the MEMINFO section shows that there is very little space left\non the Java heap, the current thread information is typically not important. The current thread is simply the thread that happened to be current when the space ran out. In this situation you might want to increase your Java heap size. For help with this task, see \nHow to do heap sizing\n.\n\n\nNative OutOfMemoryError\n\n\nIn this scenario, the VM runs out of native memory. Native memory is memory that is used by the VM for storing all virtualized resources and data that it needs for VM operations. Native memory that is available to the VM process is limited by the operating system. The native memory available to the VM might also be subject to additional limits imposed by the operating system, for example Unix \nulimits\n. \n\n\nWhen a \nNativeOutOfMemoryError\n occurs, a Java dump is generated by default. The first section of the file (TITLE) tells you that a systhrow event triggered the Java dump as a result of an OOM (\njava/lang/OutOfMemoryError\n) for native memory.\n\n\n0SECTION TITLE subcomponent dump routine\nNULL ===============================\n1TICHARSET UTF-8\n1TISIGINFO Dump Event \"systhrow\" (00040000) Detail \"java/lang/OutOfMemoryError\" \"native memory exhausted\" received\n1TIDATETIME Date: 2018/09/14 at 15:49:55:887\n1TINANOTIME System nanotime: 3636862054495675\n1TIFILENAME Javacore filename: /home/cheesemp/test/javacore.20180914.154814.19708.0003.txt\n1TIREQFLAGS Request Flags: 0x81 (exclusive+preempt)\n1TIPREPSTATE Prep State: 0x104 (exclusive_vm_access+trace_disabled)\n\n\n\n\n\nSometimes, the current thread is responsible for causing the \nNativeOutOfMemoryError\n. Information about the current thread can be found in the THREADS section, as shown in the following output.\n\n\n0SECTION THREADS subcomponent dump routine\nNULL =================================\nNULL\n1XMPOOLINFO JVM Thread pool info:\n2XMPOOLTOTAL Current total number of pooled threads: 16\n2XMPOOLLIVE Current total number of live threads: 16\n2XMPOOLDAEMON Current total number of live daemon threads: 15\nNULL \n1XMCURTHDINFO Current thread\n3XMTHREADINFO \"main\" J9VMThread:0xB6C60C00, omrthread_t:0xB6C049D8, java/lang/Thread:0xB55E3C10, state:R, prio=5\n3XMJAVALTHREAD (java/lang/Thread getId:0x1, isDaemon:false)\n3XMTHREADINFO1 (native thread ID:0x4CFD, native priority:0x5, native policy:UNKNOWN, vmstate:R, vm thread flags:0x00001020)\n3XMTHREADINFO2 (native stack address range from:0xB6D4E000, to:0xB754F000, size:0x801000)\n3XMCPUTIME CPU usage total: 3.654896026 secs, current category=\"Application\"\n3XMHEAPALLOC Heap bytes allocated since last GC cycle=0 (0x0)\n3XMTHREADINFO3 Java callstack:\n4XESTACKTRACE at sun/misc/Unsafe.allocateDBBMemory(Native Method)\n4XESTACKTRACE at java/nio/DirectByteBuffer.<init>(DirectByteBuffer.java:127(Compiled Code))\n4XESTACKTRACE at java/nio/ByteBuffer.allocateDirect(ByteBuffer.java:311)\n4XESTACKTRACE at NativeHeapBreaker.main(NativeHeapBreaker.java:9)\n3XMTHREADINFO3 Native callstack:\n4XENATIVESTACK (0xB6A9F5B3 [libj9prt29.so+0x3b5b3])\n...\n4XENATIVESTACK (0xB582CC9C [libjclse7b_29.so+0x40c9c])\n4XENATIVESTACK Java_sun_misc_Unsafe_allocateDBBMemory+0x88 (0xB5827F6B [libjclse7b_29.so+0x3bf6b])\n4XENATIVESTACK (0x94A2084A [<unknown>+0x0])\n4XENATIVESTACK (0xB6B2538B [libj9vm29.so+0x6c38b])\n4XENATIVESTACK (0xB6B4074C [libj9vm29.so+0x8774c])\n4XENATIVESTACK (0xB6B7F299 [libj9vm29.so+0xc6299])\n4XENATIVESTACK (0xB6A82F3E [libj9prt29.so+0x1ef3e])\n4XENATIVESTACK (0xB6B7F32A [libj9vm29.so+0xc632a])\n4XENATIVESTACK (0xB6B4084C [libj9vm29.so+0x8784c])\n4XENATIVESTACK (0xB6B3CD0C [libj9vm29.so+0x83d0c])\n4XENATIVESTACK (0xB776F87D [libjli.so+0x787d])\n4XENATIVESTACK (0xB7784F72 [libpthread.so.0+0x6f72])\n4XENATIVESTACK clone+0x5e (0xB76A043E [libc.so.6+0xee43e])\n\n\n\n\n\nFor clarity in the \nNative callstack\n output, \n...\n indicates that some lines are removed.\n\n\nThe Java callstack shows the transition from Java to native code (\nsun/misc/Unsafe.allocateDBBMemory(Native Method)\n), indicating a request for Direct Byte Buffer (DBB) storage. DBB storage is backed by native memory, with the Java heap containing only a reference to the native heap buffer. In this scenario, DBB storage is the likely culprit for this \nNativeOutOfMemoryError\n.\n\n\nThe next step is to investigate the NATIVEMEMINFO section of the Java dump file, which reports the amount of memory used by the JRE process, broken down into component areas.\n\n\n0SECTION NATIVEMEMINFO subcomponent dump routine\nNULL =================================\n0MEMUSER\n1MEMUSER JRE: 3,166,386,688 bytes / 4388 allocations\n1MEMUSER |\n2MEMUSER +--VM: 563,176,824 bytes / 1518 allocations\n2MEMUSER | |\n3MEMUSER | +--Classes: 3,104,416 bytes / 120 allocations\n2MEMUSER | |\n3MEMUSER | +--Memory Manager (GC): 548,181,888 bytes / 398 allocations\n3MEMUSER | | |\n4MEMUSER | | +--Java Heap: 536,932,352 bytes / 1 allocation\n3MEMUSER | | |\n4MEMUSER | | +--Other: 11,249,536 bytes / 397 allocations\n2MEMUSER | |\n3MEMUSER | +--Threads: 10,817,120 bytes / 147 allocations\n3MEMUSER | | |\n4MEMUSER | | +--Java Stack: 115,584 bytes / 16 allocations\n3MEMUSER | | |\n4MEMUSER | | +--Native Stack: 10,616,832 bytes / 17 allocations\n3MEMUSER | | |\n4MEMUSER | | +--Other: 84,704 bytes / 114 allocations\n2MEMUSER | |\n3MEMUSER | +--Trace: 163,688 bytes / 268 allocations\n2MEMUSER | |\n3MEMUSER | +--JVMTI: 17,320 bytes / 13 allocations\n2MEMUSER | |\n3MEMUSER | +--JNI: 23,296 bytes / 55 allocations\n2MEMUSER | |\n3MEMUSER | +--Port Library: 8,576 bytes / 74 allocations\n2MEMUSER | |\n3MEMUSER | +--Other: 860,520 bytes / 443 allocations\n1MEMUSER |\n2MEMUSER +--JIT: 3,744,728 bytes / 122 allocations\n2MEMUSER | |\n3MEMUSER | +--JIT Code Cache: 2,097,152 bytes / 1 allocation\n2MEMUSER | |\n3MEMUSER | +--JIT Data Cache: 524,336 bytes / 1 allocation\n2MEMUSER | |\n3MEMUSER | +--Other: 1,123,240 bytes / 120 allocations\n1MEMUSER |\n2MEMUSER +--Class Libraries: 2,599,463,024 bytes / 2732 allocations\n2MEMUSER | |\n3MEMUSER | +--Harmony Class Libraries: 1,024 bytes / 1 allocation\n2MEMUSER | |\n3MEMUSER | +--VM Class Libraries: 2,599,462,000 bytes / 2731 allocations\n3MEMUSER | | |\n4MEMUSER | | +--sun.misc.Unsafe: 2,598,510,480 bytes / 2484 allocations\n4MEMUSER | | | |\n5MEMUSER | | | +--Direct Byte Buffers: 2,598,510,480 bytes / 2484 allocations\n3MEMUSER | | |\n4MEMUSER | | +--Other: 951,520 bytes / 247 allocations\n1MEMUSER |\n2MEMUSER +--Unknown: 2,112 bytes / 16 allocations\nNULL \n\n\n\n\n\nIn the \nVM Class Libraries\n section, the amount of memory allocated for \nDirect Byte Buffers\n is shown. Because the \nNativeOutOfMemoryError\n was received on a small 32-bit system, a value of \n2,598,510,480 bytes\n indicates that the operating system has run out of memory. On a larger UNIX system, the process might have run out of memory because of the \nulimit\n setting. Increasing the value for \nulimit\n might avoid the error, which you can do temporarily by setting \nulimit -f unlimited\n in your current session.\n\n\nThe theoretical maximum size for a 32-bit process is the size of the 32-bit address space, which is 4 GB. On most operating systems a portion of the address space for each process is used by the kernel, such that the real limit for 32-bit processes is actually significantly less than 4GB. As a result, running out of native memory with a 32-bit VM is quite common.\n\n\nThe same 4 GB limit is also important if you are using a 64-bit VM with compressed references. In compressed references mode, all references to objects, classes, threads, and monitors are represented by 32-bit values for performance reasons, so these structures can be allocated only at 32-bit addresses. However, the operating system might place other allocations within this 4 GB of address space, and if this area becomes sufficiently full or fragmented, the VM throws a native \nNativeOutOfMemoryError\n error. These errors typically occur when the VM tries to create a new thread or load a class. The \nCurrent Thread History\n section should contain more information about what the thread was doing at the VM level when the \nNativeOutOfMemoryError\n error occurred.\n\n\nYou can usually avoid this type of problem by using the \n-Xmcrs\n option to reserve a contiguous area of memory within the lowest 4GB of memory at VM startup.\n\n\nAnother common cause of a \nNativeOutOfMemoryError\n is when an application loads duplicate classes. Classes are allocated outside of the Java heap in native memory. If the value reported for \nClasses\n in the NATIVEMEMINFO section is very large, duplicate classes might be the cause of your problem. The \nEclipse Memory Analyzer Tool (MAT)\n can tell you if you have duplicate classes by using the \nClass Loader Explorer\n feature. Because a system dump is automatically generated as well as a Java dump in response to a \nNativeOutOfMemoryError\n, simply open the system dump in MAT to continue your diagnosis. \n\n\nDeadlock\n\n\nDeadlocks occur when two threads attempt to synchronize on an object and lock an instance of a class. When this happens, your application stops responding and hangs. Generating a Java dump file will quickly tell you whether you have a deadlock situation. Trigger the Java dump by sending a SIGQUIT signal (\nkill -3\n) to the VM.\n\n\nThe VM can detect the most common types of deadlock scenario involving Java monitors. If this type of deadlock is detected, information is provided in the LOCKS section. More complex deadlocks, including those that involve a mixture of native mutexes and Java monitors, are not detected.\n\n\nHere is the output from the code that was used to cause a common deadlock scenario:\n\n\nNULL\n \n\n1L\nKDEADLOCK\n \nDeadlock\n \ndetected\n \n!!!\n\n\nNULL\n \n---------------------\n\n\nNULL\n \n\n2L\nKDEADLOCKTHR\n \nThread\n \n\"Worker Thread 2\"\n \n(\n0x94501D00\n)\n\n\n3L\nKDEADLOCKWTR\n \nis\n \nwaiting\n \nfor\n:\n\n\n4L\nKDEADLOCKMON\n \nsys_mon_t\n:\n0x08C2B344\n \ninfl_mon_t\n:\n \n0x08C2B384\n:\n\n\n4L\nKDEADLOCKOBJ\n \njava\n/\nlang\n/\nObject\n@0xB5666698\n\n\n3L\nKDEADLOCKOWN\n \nwhich\n \nis\n \nowned\n \nby\n:\n\n\n2L\nKDEADLOCKTHR\n \nThread\n \n\"Worker Thread 3\"\n \n(\n0x94507500\n)\n\n\n3L\nKDEADLOCKWTR\n \nwhich\n \nis\n \nwaiting\n \nfor\n:\n\n\n4L\nKDEADLOCKMON\n \nsys_mon_t\n:\n0x08C2B3A0\n \ninfl_mon_t\n:\n \n0x08C2B3E0\n:\n\n\n4L\nKDEADLOCKOBJ\n \njava\n/\nlang\n/\nObject\n@0xB5666678\n\n\n3L\nKDEADLOCKOWN\n \nwhich\n \nis\n \nowned\n \nby\n:\n\n\n2L\nKDEADLOCKTHR\n \nThread\n \n\"Worker Thread 1\"\n \n(\n0x92A3EC00\n)\n\n\n3L\nKDEADLOCKWTR\n \nwhich\n \nis\n \nwaiting\n \nfor\n:\n\n\n4L\nKDEADLOCKMON\n \nsys_mon_t\n:\n0x08C2B2E8\n \ninfl_mon_t\n:\n \n0x08C2B328\n:\n\n\n4L\nKDEADLOCKOBJ\n \njava\n/\nlang\n/\nObject\n@0xB5666688\n\n\n3L\nKDEADLOCKOWN\n \nwhich\n \nis\n \nowned\n \nby\n:\n\n\n2L\nKDEADLOCKTHR\n \nThread\n \n\"Worker Thread 2\"\n \n(\n0x94501D00\n)\n\n\n\n\n\n\nThis output tells you that \nWorker Thread 2\n is waiting for \nWorker Thread 3\n, which is waiting for \nWorker Thread 1\n. Because \nWorker Thread 1\n is also waiting for \nWorker Thread 2\n, there is a deadlock. The next place to look is the output for Java and native stacks, in the THREADS section. By looking at the stack for each of these worker threads you can trace the problem back to specific lines in your application code.\n\n\nIn this example, you can see from the following output that for all worker threads, the stack traces (\n4XESTACKTRACE\n/\n5XESTACKTRACE\n) indicate a problem in line 35 of the application \nDeadLockTest.java\n:\n\n\n3\nXMTHREADINFO\n \n\"Worker Thread 1\"\n \nJ9VMThread\n:\n0x92A3EC00\n,\n \nomrthread_t\n:\n0x92A3C2B0\n,\n \njava\n/\nlang\n/\nThread\n:\n0xB5666778\n,\n \nstate\n:\nB\n,\n \nprio\n=\n5\n\n\n3\nXMJAVALTHREAD\n \n(\njava\n/\nlang\n/\nThread\n \ngetId\n:\n0x13\n,\n \nisDaemon\n:\nfalse\n)\n\n\n3\nXMTHREADINFO1\n \n(\nnative\n \nthread\n \nID\n:\n0x52CF\n,\n \nnative\n \npriority\n:\n0x5\n,\n \nnative\n \npolicy\n:\nUNKNOWN\n,\n \nvmstate\n:\nB\n,\n \nvm\n \nthread\n \nflags\n:\n0x00000201\n)\n\n\n3\nXMTHREADINFO2\n \n(\nnative\n \nstack\n \naddress\n \nrange\n \nfrom\n:\n0x9297E000\n,\n \nto\n:\n0x929BF000\n,\n \nsize\n:\n0x41000\n)\n\n\n3\nXMCPUTIME\n \nCPU\n \nusage\n \ntotal\n:\n \n0.004365543\n \nsecs\n,\n \ncurrent\n \ncategory\n=\n\"Application\"\n\n\n3\nXMTHREADBLOCK\n \nBlocked\n \non\n:\n \njava\n/\nlang\n/\nObject\n@0xB5666688\n \nOwned\n \nby\n:\n \n\"Worker Thread 2\"\n \n(\nJ9VMThread\n:\n0x94501D00\n,\n \njava\n/\nlang\n/\nThread\n:\n0xB56668D0\n)\n\n\n3\nXMHEAPALLOC\n \nHeap\n \nbytes\n \nallocated\n \nsince\n \nlast\n \nGC\n \ncycle\n=\n0\n \n(\n0x0\n)\n\n\n3\nXMTHREADINFO3\n \nJava\n \ncallstack\n:\n\n\n4\nXESTACKTRACE\n \nat\n \nWorkerThread\n.\nrun\n(\nDeadLockTest\n.\njava\n:\n35\n)\n\n\n5\nXESTACKTRACE\n \n(\nentered\n \nlock\n:\n \njava\n/\nlang\n/\nObject\n@0xB5666678\n,\n \nentry\n \ncount\n:\n \n1\n)\n\n\n...\n\n\n3\nXMTHREADINFO\n \n\"Worker Thread 2\"\n \nJ9VMThread\n:\n0x94501D00\n,\n \nomrthread_t\n:\n0x92A3C8F0\n,\n \njava\n/\nlang\n/\nThread\n:\n0xB56668D0\n,\n \nstate\n:\nB\n,\n \nprio\n=\n5\n\n\n3\nXMJAVALTHREAD\n \n(\njava\n/\nlang\n/\nThread\n \ngetId\n:\n0x14\n,\n \nisDaemon\n:\nfalse\n)\n\n\n3\nXMTHREADINFO1\n \n(\nnative\n \nthread\n \nID\n:\n0x52D0\n,\n \nnative\n \npriority\n:\n0x5\n,\n \nnative\n \npolicy\n:\nUNKNOWN\n,\n \nvmstate\n:\nB\n,\n \nvm\n \nthread\n \nflags\n:\n0x00000201\n)\n\n\n3\nXMTHREADINFO2\n \n(\nnative\n \nstack\n \naddress\n \nrange\n \nfrom\n:\n0x946BF000\n,\n \nto\n:\n0x94700000\n,\n \nsize\n:\n0x41000\n)\n\n\n3\nXMCPUTIME\n \nCPU\n \nusage\n \ntotal\n:\n \n0.004555580\n \nsecs\n,\n \ncurrent\n \ncategory\n=\n\"Application\"\n\n\n3\nXMTHREADBLOCK\n \nBlocked\n \non\n:\n \njava\n/\nlang\n/\nObject\n@0xB5666698\n \nOwned\n \nby\n:\n \n\"Worker Thread 3\"\n \n(\nJ9VMThread\n:\n0x94507500\n,\n \njava\n/\nlang\n/\nThread\n:\n0xB5666A18\n)\n\n\n3\nXMHEAPALLOC\n \nHeap\n \nbytes\n \nallocated\n \nsince\n \nlast\n \nGC\n \ncycle\n=\n0\n \n(\n0x0\n)\n\n\n3\nXMTHREADINFO3\n \nJava\n \ncallstack\n:\n\n\n4\nXESTACKTRACE\n \nat\n \nWorkerThread\n.\nrun\n(\nDeadLockTest\n.\njava\n:\n35\n)\n\n\n5\nXESTACKTRACE\n \n(\nentered\n \nlock\n:\n \njava\n/\nlang\n/\nObject\n@0xB5666688\n,\n \nentry\n \ncount\n:\n \n1\n)\n\n\n...\n\n\n3\nXMTHREADINFO\n \n\"Worker Thread 3\"\n \nJ9VMThread\n:\n0x94507500\n,\n \nomrthread_t\n:\n0x92A3CC10\n,\n \njava\n/\nlang\n/\nThread\n:\n0xB5666A18\n,\n \nstate\n:\nB\n,\n \nprio\n=\n5\n\n\n3\nXMJAVALTHREAD\n \n(\njava\n/\nlang\n/\nThread\n \ngetId\n:\n0x15\n,\n \nisDaemon\n:\nfalse\n)\n\n\n3\nXMTHREADINFO1\n \n(\nnative\n \nthread\n \nID\n:\n0x52D1\n,\n \nnative\n \npriority\n:\n0x5\n,\n \nnative\n \npolicy\n:\nUNKNOWN\n,\n \nvmstate\n:\nB\n,\n \nvm\n \nthread\n \nflags\n:\n0x00000201\n)\n\n\n3\nXMTHREADINFO2\n \n(\nnative\n \nstack\n \naddress\n \nrange\n \nfrom\n:\n0x9467E000\n,\n \nto\n:\n0x946BF000\n,\n \nsize\n:\n0x41000\n)\n\n\n3\nXMCPUTIME\n \nCPU\n \nusage\n \ntotal\n:\n \n0.003657010\n \nsecs\n,\n \ncurrent\n \ncategory\n=\n\"Application\"\n\n\n3\nXMTHREADBLOCK\n \nBlocked\n \non\n:\n \njava\n/\nlang\n/\nObject\n@0xB5666678\n \nOwned\n \nby\n:\n \n\"Worker Thread 1\"\n \n(\nJ9VMThread\n:\n0x92A3EC00\n,\n \njava\n/\nlang\n/\nThread\n:\n0xB5666778\n)\n\n\n3\nXMHEAPALLOC\n \nHeap\n \nbytes\n \nallocated\n \nsince\n \nlast\n \nGC\n \ncycle\n=\n0\n \n(\n0x0\n)\n\n\n3\nXMTHREADINFO3\n \nJava\n \ncallstack\n:\n\n\n4\nXESTACKTRACE\n \nat\n \nWorkerThread\n.\nrun\n(\nDeadLockTest\n.\njava\n:\n35\n)\n\n\n5\nXESTACKTRACE\n \n(\nentered\n \nlock\n:\n \njava\n/\nlang\n/\nObject\n@0xB5666698\n,\n \nentry\n \ncount\n:\n \n1\n)",
"title": "Java dump"
},
{
"location": "/dump_javadump/#java-dump",
- "text": "Java dumps, sometimes referred to as Java cores , are produced when the VM ends unexpectedly because of an operating system signal, OutOfMemoryError , or a user-initiated keystroke combination. You can also generate a Java dump by calling the Dump API programmatically from your application or specifying the -Xdump:java option on the command line. If your Java application crashes or hangs, Java dumps can provide useful information to help you diagnose the root cause. If your application crashes, Javadumps are generated automatically for the following types of failure: the VM receives an unexpected signal or an assertion failure the VM runs out of memory If your application hangs, you can trigger the generation of a Java dump by sending a SIGQUIT signal ( kill -3 ) to the VM. Note: On Windows, if you started the VM in a console window you can force the VM to produce a Java dump in response to a SIGBREAK signal\n(Ctrl-Break keyboard combination). If you didn't start in a console window there is no equivalent to a Linux kill command on Windows for sending signals.\nThe only option here is to trigger a full system dump by finding the VM process in the Processes tab of the Windows Task Manager and clicking Create dump file . To help you understand how a Java dump can help you with problem diagnosis, we've included a few scenarios at the end of the topic that might help you interpret the data: A crash caused by a general protection fault (gpf) A Java heap OutOfMemoryError (OOM)",
+ "text": "Java dumps, sometimes referred to as Java cores , are produced when the VM ends unexpectedly because of an operating system signal, OutOfMemoryError , or a user-initiated keystroke combination. You can also generate a Java dump by calling the Dump API programmatically from your application or specifying the -Xdump:java option on the command line. If your Java application crashes or hangs, Java dumps can provide useful information to help you diagnose the root cause. If your application crashes, Javadumps are generated automatically for the following types of failure: the VM receives an unexpected signal or an assertion failure the VM runs out of memory If your application hangs, you can trigger the generation of a Java dump by sending a SIGQUIT signal ( kill -3 ) to the VM. Note: On Windows, if you started the VM in a console window you can force the VM to produce a Java dump in response to a SIGBREAK signal\n(Ctrl-Break keyboard combination). If you didn't start in a console window there is no equivalent to a Linux kill command on Windows for sending signals.\nThe only option here is to trigger a full system dump by finding the VM process in the Processes tab of the Windows Task Manager and clicking Create dump file . To help you understand how a Java dump can help you with problem diagnosis, this topic includes a few scenarios to help you interpret the data: A crash caused by a general protection fault (gpf) A Java heap OutOfMemoryError (OOM) A native OutOfMemoryError (OOM) A deadlock situation",
"title": "Java dump"
},
{
@@ -407,7 +407,7 @@
},
{
"location": "/dump_javadump/#gpinfo",
- "text": "The GPINFO section provides general information about the system that the JVM is running on. The following example is taken\nfrom a Java dump that was generated on a Linux system. NULL ------------------------------------------------------------------------\n0SECTION GPINFO subcomponent dump routine\nNULL ================================\n2XHOSLEVEL OS Level : Linux 3.10.0-862.11.6.el7.x86_64\n2XHCPUS Processors -\n3XHCPUARCH Architecture : amd64\n3XHNUMCPUS How Many : 4\n3XHNUMASUP NUMA is either not supported or has been disabled by user\nNULL \n1XHERROR2 Register dump section only produced for SIGSEGV, SIGILL or SIGFPE.\nNULL The content of this section can vary, depending on the cause of the dump. For example, if the dump was caused by a\ngeneral protection fault (gpf), the library in which the crash occurred is also recorded, together with a value shown\nas VM flags . This value can provide some clues about which component of the VM might have been involved. Look for the\nfollowing line in the output: 1XHFLAGS VM flags:0000000000000000 The hexadecimal number recorded for VM flags ends in MSSSS, where M is the VM component and SSSS is component-specific code as shown in the following table: Component Code value INTERPRETER 0x10000 GC 0x20000 GROW_STACK 0x30000 JNI 0x40000 JIT_CODEGEN 0x50000 BCVERIFY 0x60000 RTVERIFY 0x70000 SHAREDCLASSES 0x80000 A value of 0000000000000000 (0x00000) indicates that a crash occurred outside of the VM.",
+ "text": "The GPINFO section provides general information about the system that the VM is running on. The following example is taken\nfrom a Java dump that was generated on a Linux system. NULL ------------------------------------------------------------------------\n0SECTION GPINFO subcomponent dump routine\nNULL ================================\n2XHOSLEVEL OS Level : Linux 3.10.0-862.11.6.el7.x86_64\n2XHCPUS Processors -\n3XHCPUARCH Architecture : amd64\n3XHNUMCPUS How Many : 4\n3XHNUMASUP NUMA is either not supported or has been disabled by user\nNULL \n1XHERROR2 Register dump section only produced for SIGSEGV, SIGILL or SIGFPE.\nNULL The content of this section can vary, depending on the cause of the dump. For example, if the dump was caused by a\ngeneral protection fault (gpf), the library in which the crash occurred is also recorded, together with a value shown\nas VM flags . This value can provide some clues about which component of the VM might have been involved. Look for the\nfollowing line in the output: 1XHFLAGS VM flags:0000000000000000 The hexadecimal number recorded for VM flags ends in MSSSS, where M is the VM component and SSSS is component-specific code as shown in the following table: Component Code value INTERPRETER 0x10000 GC 0x20000 GROW_STACK 0x30000 JNI 0x40000 JIT_CODEGEN 0x50000 BCVERIFY 0x60000 RTVERIFY 0x70000 SHAREDCLASSES 0x80000 A value of 0000000000000000 (0x00000) indicates that a crash occurred outside of the VM.",
"title": "GPINFO"
},
{
@@ -447,7 +447,7 @@
},
{
"location": "/dump_javadump/#classes",
- "text": "The classes section shows information about class loaders. The first part is a summary that records each available class loader ( 2CLTEXTCLLOADER ) followed by the number of libraries and classes that it loaded. This information is followed by a more detailed list of libraries ( 1CLTEXTCLLIB ) and classes ( 1CLTEXTCLLO ) that are loaded. In the example we can see that the java/lang/InternalAnonymousClassLoader loaded 2 classes, jdk/internal/loader/BuiltinClassLoader$$Lambda$2/00000000F03876A0(0x0000000001030F00) and jdk/internal/loader/BuiltinClassLoader$$Lambda$1/00000000F00D2460(0x0000000001018A00) . NULL ------------------------------------------------------------------------\n0SECTION CLASSES subcomponent dump routine\nNULL =================================\n1CLTEXTCLLOS Classloader summaries\n1CLTEXTCLLSS 12345678: 1=primordial,2=extension,3=shareable,4=middleware,5=system,6=trusted,7=application,8=delegating\n2CLTEXTCLLOADER p---st-- Loader *System*(0x00000000FFE1D258)\n3CLNMBRLOADEDLIB Number of loaded libraries 5\n3CLNMBRLOADEDCL Number of loaded classes 638\n2CLTEXTCLLOADER -x--st-- Loader jdk/internal/loader/ClassLoaders$PlatformClassLoader(0x00000000FFE1D4F0), Parent *none*(0x0000000000000000)\n3CLNMBRLOADEDLIB Number of loaded libraries 0\n3CLNMBRLOADEDCL Number of loaded classes 0\n2CLTEXTCLLOADER ----st-- Loader java/lang/InternalAnonymousClassLoader(0x00000000FFE1DFD0), Parent *none*(0x0000000000000000)\n3CLNMBRLOADEDLIB Number of loaded libraries 0\n3CLNMBRLOADEDCL Number of loaded classes 2\n2CLTEXTCLLOADER -----ta- Loader jdk/internal/loader/ClassLoaders$AppClassLoader(0x00000000FFE1DAD0), Parent jdk/internal/loader/ClassLoaders$PlatformClassLoader(0x00000000FFE1D4F0)\n3CLNMBRLOADEDLIB Number of loaded libraries 0\n3CLNMBRLOADEDCL Number of loaded classes 0\n1CLTEXTCLLIB ClassLoader loaded libraries\n2CLTEXTCLLIB Loader *System*(0x00000000FFE1D258)\n3CLTEXTLIB /home/me/openj9-openjdk-jdk9/build/linux-x86_64-normal-server-release/images/jdk/lib/compressedrefs/jclse9_29\n3CLTEXTLIB /home/me/openj9-openjdk-jdk9/build/linux-x86_64-normal-server-release/images/jdk/lib/java\n3CLTEXTLIB /home/me/openj9-openjdk-jdk9/build/linux-x86_64-normal-server-release/images/jdk/lib/compressedrefs/j9jit29\n3CLTEXTLIB /home/me/openj9-openjdk-jdk9/build/linux-x86_64-normal-server-release/images/jdk/lib/zip\n3CLTEXTLIB /home/me/openj9-openjdk-jdk9/build/linux-x86_64-normal-server-release/images/jdk/lib/nio\n1CLTEXTCLLOD ClassLoader loaded classes\n2CLTEXTCLLOAD Loader *System*(0x00000000FFE1D258)\n3CLTEXTCLASS [Ljava/lang/Thread$State;(0x0000000001056400)\n...\n2CLTEXTCLLOAD Loader jdk/internal/loader/ClassLoaders$PlatformClassLoader(0x00000000FFE1D4F0)\n2CLTEXTCLLOAD Loader java/lang/InternalAnonymousClassLoader(0x00000000FFE1DFD0)\n3CLTEXTCLASS jdk/internal/loader/BuiltinClassLoader$$Lambda$2/00000000F03876A0(0x0000000001030F00)\n3CLTEXTCLASS jdk/internal/loader/BuiltinClassLoader$$Lambda$1/00000000F00D2460(0x0000000001018A00)\n2CLTEXTCLLOAD Loader jdk/internal/loader/ClassLoaders$AppClassLoader(0x00000000FFE1DAD0)",
+ "text": "The classes section shows information about class loaders. The first part is a summary that records each available class loader ( 2CLTEXTCLLOADER ) followed by the number of libraries and classes that it loaded. This information is followed by a more detailed list of libraries ( 1CLTEXTCLLIB ) and classes ( 1CLTEXTCLLO ) that are loaded. In the example you can see that the java/lang/InternalAnonymousClassLoader loaded 2 classes, jdk/internal/loader/BuiltinClassLoader$$Lambda$2/00000000F03876A0(0x0000000001030F00) and jdk/internal/loader/BuiltinClassLoader$$Lambda$1/00000000F00D2460(0x0000000001018A00) . NULL ------------------------------------------------------------------------\n0SECTION CLASSES subcomponent dump routine\nNULL =================================\n1CLTEXTCLLOS Classloader summaries\n1CLTEXTCLLSS 12345678: 1=primordial,2=extension,3=shareable,4=middleware,5=system,6=trusted,7=application,8=delegating\n2CLTEXTCLLOADER p---st-- Loader *System*(0x00000000FFE1D258)\n3CLNMBRLOADEDLIB Number of loaded libraries 5\n3CLNMBRLOADEDCL Number of loaded classes 638\n2CLTEXTCLLOADER -x--st-- Loader jdk/internal/loader/ClassLoaders$PlatformClassLoader(0x00000000FFE1D4F0), Parent *none*(0x0000000000000000)\n3CLNMBRLOADEDLIB Number of loaded libraries 0\n3CLNMBRLOADEDCL Number of loaded classes 0\n2CLTEXTCLLOADER ----st-- Loader java/lang/InternalAnonymousClassLoader(0x00000000FFE1DFD0), Parent *none*(0x0000000000000000)\n3CLNMBRLOADEDLIB Number of loaded libraries 0\n3CLNMBRLOADEDCL Number of loaded classes 2\n2CLTEXTCLLOADER -----ta- Loader jdk/internal/loader/ClassLoaders$AppClassLoader(0x00000000FFE1DAD0), Parent jdk/internal/loader/ClassLoaders$PlatformClassLoader(0x00000000FFE1D4F0)\n3CLNMBRLOADEDLIB Number of loaded libraries 0\n3CLNMBRLOADEDCL Number of loaded classes 0\n1CLTEXTCLLIB ClassLoader loaded libraries\n2CLTEXTCLLIB Loader *System*(0x00000000FFE1D258)\n3CLTEXTLIB /home/me/openj9-openjdk-jdk9/build/linux-x86_64-normal-server-release/images/jdk/lib/compressedrefs/jclse9_29\n3CLTEXTLIB /home/me/openj9-openjdk-jdk9/build/linux-x86_64-normal-server-release/images/jdk/lib/java\n3CLTEXTLIB /home/me/openj9-openjdk-jdk9/build/linux-x86_64-normal-server-release/images/jdk/lib/compressedrefs/j9jit29\n3CLTEXTLIB /home/me/openj9-openjdk-jdk9/build/linux-x86_64-normal-server-release/images/jdk/lib/zip\n3CLTEXTLIB /home/me/openj9-openjdk-jdk9/build/linux-x86_64-normal-server-release/images/jdk/lib/nio\n1CLTEXTCLLOD ClassLoader loaded classes\n2CLTEXTCLLOAD Loader *System*(0x00000000FFE1D258)\n3CLTEXTCLASS [Ljava/lang/Thread$State;(0x0000000001056400)\n...\n2CLTEXTCLLOAD Loader jdk/internal/loader/ClassLoaders$PlatformClassLoader(0x00000000FFE1D4F0)\n2CLTEXTCLLOAD Loader java/lang/InternalAnonymousClassLoader(0x00000000FFE1DFD0)\n3CLTEXTCLASS jdk/internal/loader/BuiltinClassLoader$$Lambda$2/00000000F03876A0(0x0000000001030F00)\n3CLTEXTCLASS jdk/internal/loader/BuiltinClassLoader$$Lambda$1/00000000F00D2460(0x0000000001018A00)\n2CLTEXTCLLOAD Loader jdk/internal/loader/ClassLoaders$AppClassLoader(0x00000000FFE1DAD0)",
"title": "CLASSES"
},
{
@@ -457,15 +457,25 @@
},
{
"location": "/dump_javadump/#general-protection-fault",
- "text": "In this scenario, a Java application has crashed due to a General Protection Fault (GPF), automatically generating a Java dump file. The first section of the file (TITLE) tells you that the GPF triggered the Java dump. 0SECTION TITLE subcomponent dump routine\nNULL ===============================\n1TICHARSET UTF-8\n1TISIGINFO Dump Event \"gpf\" (00002000) received\n1TIDATETIME Date: 2018/09/24 at 15:18:03:115\n1TINANOTIME System nanotime: 4498949283020796\n1TIFILENAME Javacore filename: /home/test/JNICrasher/javacore.20180924.151801.29399.0002.txt\n1TIREQFLAGS Request Flags: 0x81 (exclusive+preempt)\n1TIPREPSTATE Prep State: 0x100 (trace_disabled)\n1TIPREPINFO Exclusive VM access not taken: data may not be consistent across javacore sections To troubleshoot this problem, you need to know which thread caused the GPF to occur. The thread that was running at the time of the crash is reported as the current thread \nin the THREADS section of the Java dump. Here is an extract from the THREADS section: NULL ------------------------------------------------------------------------\n0SECTION THREADS subcomponent dump routine\nNULL =================================\nNULL\n1XMPOOLINFO JVM Thread pool info:\n2XMPOOLTOTAL Current total number of pooled threads: 16\n2XMPOOLLIVE Current total number of live threads: 15\n2XMPOOLDAEMON Current total number of live daemon threads: 14\nNULL \n1XMCURTHDINFO Current thread\n3XMTHREADINFO \"main\" J9VMThread:0xB6B60E00, omrthread_t:0xB6B049D8, java/lang/Thread:0xB55444D0, state:R, prio=5\n3XMJAVALTHREAD (java/lang/Thread getId:0x1, isDaemon:false)\n3XMTHREADINFO1 (native thread ID:0x72D8, native priority:0x5, native policy:UNKNOWN, vmstate:R, vm thread flags:0x00000000)\n3XMTHREADINFO2 (native stack address range from:0xB6CE3000, to:0xB74E4000, size:0x801000)\n3XMCPUTIME CPU usage total: 0.319865924 secs, current category=\"Application\"\n3XMHEAPALLOC Heap bytes allocated since last GC cycle=778008 (0xBDF18)\n3XMTHREADINFO3 Java callstack:\n4XESTACKTRACE at JNICrasher.doSomethingThatCrashes(Native Method)\n4XESTACKTRACE at JNICrasher.main(JNICrasher.java:7)\n3XMTHREADINFO3 Native callstack:\n4XENATIVESTACK (0xB6C6F663 [libj9prt29.so+0x3b663])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB6C6F1CE [libj9prt29.so+0x3b1ce])\n4XENATIVESTACK (0xB6C6F2C6 [libj9prt29.so+0x3b2c6])\n4XENATIVESTACK (0xB6C6ED93 [libj9prt29.so+0x3ad93])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB6C6ED07 [libj9prt29.so+0x3ad07])\n4XENATIVESTACK (0xB6C6AA3D [libj9prt29.so+0x36a3d])\n4XENATIVESTACK (0xB6C6C3A4 [libj9prt29.so+0x383a4])\n4XENATIVESTACK (0xB667FA19 [libj9dmp29.so+0xfa19])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB66878CF [libj9dmp29.so+0x178cf])\n4XENATIVESTACK (0xB6688083 [libj9dmp29.so+0x18083])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB6680C0D [libj9dmp29.so+0x10c0d])\n4XENATIVESTACK (0xB667F9D7 [libj9dmp29.so+0xf9d7])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB668B02F [libj9dmp29.so+0x1b02f])\n4XENATIVESTACK (0xB668B4D3 [libj9dmp29.so+0x1b4d3])\n4XENATIVESTACK (0xB66740F1 [libj9dmp29.so+0x40f1])\n4XENATIVESTACK (0xB66726FA [libj9dmp29.so+0x26fa])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB66726A9 [libj9dmp29.so+0x26a9])\n4XENATIVESTACK (0xB6676AE4 [libj9dmp29.so+0x6ae4])\n4XENATIVESTACK (0xB668D75A [libj9dmp29.so+0x1d75a])\n4XENATIVESTACK (0xB6A28DD4 [libj9vm29.so+0x81dd4])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB6A289EE [libj9vm29.so+0x819ee])\n4XENATIVESTACK (0xB6A29A40 [libj9vm29.so+0x82a40])\n4XENATIVESTACK (0xB6C52B6A [libj9prt29.so+0x1eb6a])\n4XENATIVESTACK __kernel_rt_sigreturn+0x0 (0xB7747410)\n4XENATIVESTACK (0xB75330B6 [libffi29.so+0x50b6])\n4XENATIVESTACK ffi_raw_call+0xad (0xB7531C53 [libffi29.so+0x3c53])\n4XENATIVESTACK (0xB69BE4AB [libj9vm29.so+0x174ab])\n4XENATIVESTACK (0xB6A665BC [libj9vm29.so+0xbf5bc])\n4XENATIVESTACK (0xB6A15552 [libj9vm29.so+0x6e552])\n4XENATIVESTACK (0xB6A30894 [libj9vm29.so+0x89894])\n4XENATIVESTACK (0xB6A6F169 [libj9vm29.so+0xc8169])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB6A6F1FA [libj9vm29.so+0xc81fa])\n4XENATIVESTACK (0xB6A30994 [libj9vm29.so+0x89994])\n4XENATIVESTACK (0xB6A2CE4C [libj9vm29.so+0x85e4c])\n4XENATIVESTACK (0xB770487D [libjli.so+0x787d])\n4XENATIVESTACK (0xB7719F72 [libpthread.so.0+0x6f72])\n4XENATIVESTACK clone+0x5e (0xB763543E [libc.so.6+0xee43e]) The extract tells you that the current thread was java/lang/Thread , and information is provided about the Java callstack and Native callstack\n( 3XMTHREADINFO3 ) at the point at which the crash occurred. To simulate a crash caused by a bug in an application, this example\ncalls a JNI method whose native implementation causes a crash. The Java callstack shows the call to the JNI native method ( JNIcrasher ), and the\nnative callstack shows the point of failure. In this example, the native call stack does not include any function names to help you isolate the error\nin the native code. You can get this information from a system dump, which is usually produced alongside the Java dump. Open the\nsystem dump with the Dump viewer and use the info thread command to print the Java and native stack for the current thread.",
+ "text": "In this scenario, a Java application has crashed due to a General Protection Fault (GPF), automatically generating a Java dump file. The first section of the file (TITLE) tells you that the GPF triggered the Java dump. 0SECTION TITLE subcomponent dump routine\nNULL ===============================\n1TICHARSET UTF-8\n1TISIGINFO Dump Event \"gpf\" (00002000) received\n1TIDATETIME Date: 2018/09/24 at 15:18:03:115\n1TINANOTIME System nanotime: 4498949283020796\n1TIFILENAME Javacore filename: /home/test/JNICrasher/javacore.20180924.151801.29399.0002.txt\n1TIREQFLAGS Request Flags: 0x81 (exclusive+preempt)\n1TIPREPSTATE Prep State: 0x100 (trace_disabled)\n1TIPREPINFO Exclusive VM access not taken: data may not be consistent across javacore sections To troubleshoot this problem, you need to know which thread caused the GPF to occur. The thread that was running at the time of the crash is reported as the current thread \nin the THREADS section of the Java dump. Here is an extract from the THREADS section: NULL ------------------------------------------------------------------------\n0SECTION THREADS subcomponent dump routine\nNULL =================================\nNULL\n1XMPOOLINFO JVM Thread pool info:\n2XMPOOLTOTAL Current total number of pooled threads: 16\n2XMPOOLLIVE Current total number of live threads: 15\n2XMPOOLDAEMON Current total number of live daemon threads: 14\nNULL \n1XMCURTHDINFO Current thread\n3XMTHREADINFO \"main\" J9VMThread:0xB6B60E00, omrthread_t:0xB6B049D8, java/lang/Thread:0xB55444D0, state:R, prio=5\n3XMJAVALTHREAD (java/lang/Thread getId:0x1, isDaemon:false)\n3XMTHREADINFO1 (native thread ID:0x72D8, native priority:0x5, native policy:UNKNOWN, vmstate:R, vm thread flags:0x00000000)\n3XMTHREADINFO2 (native stack address range from:0xB6CE3000, to:0xB74E4000, size:0x801000)\n3XMCPUTIME CPU usage total: 0.319865924 secs, current category=\"Application\"\n3XMHEAPALLOC Heap bytes allocated since last GC cycle=778008 (0xBDF18)\n3XMTHREADINFO3 Java callstack:\n4XESTACKTRACE at JNICrasher.doSomethingThatCrashes(Native Method)\n4XESTACKTRACE at JNICrasher.main(JNICrasher.java:7)\n3XMTHREADINFO3 Native callstack:\n4XENATIVESTACK (0xB6C6F663 [libj9prt29.so+0x3b663])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB6C6F1CE [libj9prt29.so+0x3b1ce])\n4XENATIVESTACK (0xB6C6F2C6 [libj9prt29.so+0x3b2c6])\n4XENATIVESTACK (0xB6C6ED93 [libj9prt29.so+0x3ad93])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB6C6ED07 [libj9prt29.so+0x3ad07])\n4XENATIVESTACK (0xB6C6AA3D [libj9prt29.so+0x36a3d])\n4XENATIVESTACK (0xB6C6C3A4 [libj9prt29.so+0x383a4])\n4XENATIVESTACK (0xB667FA19 [libj9dmp29.so+0xfa19])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB66878CF [libj9dmp29.so+0x178cf])\n4XENATIVESTACK (0xB6688083 [libj9dmp29.so+0x18083])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB6680C0D [libj9dmp29.so+0x10c0d])\n4XENATIVESTACK (0xB667F9D7 [libj9dmp29.so+0xf9d7])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB668B02F [libj9dmp29.so+0x1b02f])\n4XENATIVESTACK (0xB668B4D3 [libj9dmp29.so+0x1b4d3])\n4XENATIVESTACK (0xB66740F1 [libj9dmp29.so+0x40f1])\n4XENATIVESTACK (0xB66726FA [libj9dmp29.so+0x26fa])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB66726A9 [libj9dmp29.so+0x26a9])\n4XENATIVESTACK (0xB6676AE4 [libj9dmp29.so+0x6ae4])\n4XENATIVESTACK (0xB668D75A [libj9dmp29.so+0x1d75a])\n4XENATIVESTACK (0xB6A28DD4 [libj9vm29.so+0x81dd4])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB6A289EE [libj9vm29.so+0x819ee])\n4XENATIVESTACK (0xB6A29A40 [libj9vm29.so+0x82a40])\n4XENATIVESTACK (0xB6C52B6A [libj9prt29.so+0x1eb6a])\n4XENATIVESTACK __kernel_rt_sigreturn+0x0 (0xB7747410)\n4XENATIVESTACK (0xB75330B6 [libffi29.so+0x50b6])\n4XENATIVESTACK ffi_raw_call+0xad (0xB7531C53 [libffi29.so+0x3c53])\n4XENATIVESTACK (0xB69BE4AB [libj9vm29.so+0x174ab])\n4XENATIVESTACK (0xB6A665BC [libj9vm29.so+0xbf5bc])\n4XENATIVESTACK (0xB6A15552 [libj9vm29.so+0x6e552])\n4XENATIVESTACK (0xB6A30894 [libj9vm29.so+0x89894])\n4XENATIVESTACK (0xB6A6F169 [libj9vm29.so+0xc8169])\n4XENATIVESTACK (0xB6C52F6E [libj9prt29.so+0x1ef6e])\n4XENATIVESTACK (0xB6A6F1FA [libj9vm29.so+0xc81fa])\n4XENATIVESTACK (0xB6A30994 [libj9vm29.so+0x89994])\n4XENATIVESTACK (0xB6A2CE4C [libj9vm29.so+0x85e4c])\n4XENATIVESTACK (0xB770487D [libjli.so+0x787d])\n4XENATIVESTACK (0xB7719F72 [libpthread.so.0+0x6f72])\n4XENATIVESTACK clone+0x5e (0xB763543E [libc.so.6+0xee43e]) The extract tells you that the current thread was java/lang/Thread , and information is provided about the Java callstack and native callstack\n( 3XMTHREADINFO3 ) at the point at which the crash occurred. To simulate a crash caused by a bug in an application, this example\ncalls a JNI method whose native implementation causes a crash. The Java callstack shows the call to the JNI native method ( JNIcrasher ), and the\nnative callstack shows the point of failure. In this example, the native call stack does not include any function names to help you isolate the error\nin the native code. You can get this information from a system dump, which is usually produced alongside the Java dump. Open the\nsystem dump with the Dump viewer and use the info thread command to print the Java and native stack for the current thread.",
"title": "General Protection Fault"
},
{
"location": "/dump_javadump/#java-outofmemoryerror",
- "text": "In this scenario, the Java heap runs out of memory, causing an OutOfMemoryError , which automatically generates a Java dump file. The first section of the file (TITLE) tells you that a systhrow event triggered the Java dump as a result of an OOM ( java/lang/OutOfMemoryError ) for\nJava heap space. 0SECTION TITLE subcomponent dump routine\nNULL ===============================\n1TICHARSET UTF-8\n1TISIGINFO Dump Event \"systhrow\" (00040000) Detail \"java/lang/OutOfMemoryError\" \"Java heap space\" received\n1TIDATETIME Date: 2018/09/14 at 15:29:42:709\n1TINANOTIME System nanotime: 3635648876608448\n1TIFILENAME Javacore filename: /home/cheesemp/test/javacore.20180914.152929.18885.0003.txt\n1TIREQFLAGS Request Flags: 0x81 (exclusive+preempt)\n1TIPREPSTATE Prep State: 0x104 (exclusive_vm_access+trace_disabled) The MEMINFO section records how much memory is allocated to the Java heap ( 1STHEAPTYPE Object Memory ), how much is in use, and how much is free. Solving\nyour problem might be as simple as setting a larger heap size when you start your application. If you don't know what size the Java heap was set to, you might find that information in the ENVINFO section, which records the command line options that\nwere used when the application started. Look or search for the 1CIUSERARGS UserArgs: string and review the entries recorded for all lines that\nstart 2CIUSERARG . The Java heap size is set by the -Xmx option. If the size has not been set on the command line by -Xmx , the default value applies, which\nyou can find in Default Settings . In our scenario the solution to the problem is not an adjustment to the Java heap size. Here is the MEMINFO section: 0SECTION MEMINFO subcomponent dump routine\nNULL =================================\nNULL\n1STHEAPTYPE Object Memory\nNULL id start end size space/region\n1STHEAPSPACE 0xB6B49D20 -- -- -- Generational\n1STHEAPREGION 0xB6B4A078 0x95750000 0xB5470000 0x1FD20000 Generational/Tenured Region\n1STHEAPREGION 0xB6B49F10 0xB5470000 0xB54C0000 0x00050000 Generational/Nursery Region\n1STHEAPREGION 0xB6B49DA8 0xB54C0000 0xB5750000 0x00290000 Generational/Nursery Region\nNULL\n1STHEAPTOTAL Total memory: 536870912 (0x20000000)\n1STHEAPINUSE Total memory in use: 302603160 (0x12095B98)\n1STHEAPFREE Total memory free: 234267752 (0x0DF6A468) The output shows that only 56% of the Java heap is in use, so this suggests that the application is trying to do something sub-optimal. To\ninvestigate further we need to work out which thread was the current thread when the OOM occurred to see what it was trying to do. As in our previous scenario, we can find the current thread in the THREADS section. Here is an extract from the output: 0 SECTION THREADS subcomponent dump routine NULL ================================= NULL 1 XMPOOLINFO JVM Thread pool info : 2 XMPOOLTOTAL Current total number of pooled threads : 16 2 XMPOOLLIVE Current total number of live threads : 16 2 XMPOOLDAEMON Current total number of live daemon threads : 15 NULL 1 XMCURTHDINFO Current thread 3 XMTHREADINFO \"main\" J9VMThread : 0xB6B60C00 , omrthread_t : 0xB6B049D8 , java / lang / Thread : 0x95764520 , state : R , prio = 5 3 XMJAVALTHREAD ( java / lang / Thread getId : 0x1 , isDaemon : false ) 3 XMTHREADINFO1 ( native thread ID : 0x49C6 , native priority : 0x5 , native policy : UNKNOWN , vmstate : R , vm thread flags : 0x00001020 ) 3 XMTHREADINFO2 ( native stack address range from : 0xB6CB5000 , to : 0xB74B6000 , size : 0x801000 ) 3 XMCPUTIME CPU usage total : 8.537823831 secs , current category = \"Application\" 3 XMHEAPALLOC Heap bytes allocated since last GC cycle = 0 ( 0x0 ) 3 XMTHREADINFO3 Java callstack : 4 XESTACKTRACE at java / lang / StringBuffer . ensureCapacityImpl ( StringBuffer . java : 696 ) 4 XESTACKTRACE at java / lang / StringBuffer . append ( StringBuffer . java : 486 ( Compiled Code )) 5 XESTACKTRACE ( entered lock : java / lang / StringBuffer @0x957645B8 , entry count : 1 ) 4 XESTACKTRACE at java / lang / StringBuffer . append ( StringBuffer . java : 428 ( Compiled Code )) 4 XESTACKTRACE at HeapBreaker . main ( HeapBreaker . java : 34 ( Compiled Code )) 3 XMTHREADINFO3 Native callstack : 4 XENATIVESTACK ( 0xB6C535B3 [ libj9prt29 . so + 0x3b5b3 ]) 4 XENATIVESTACK ( 0xB6C36F3E [ libj9prt29 . so + 0x1ef3e ]) 4 XENATIVESTACK ( 0xB6C5311E [ libj9prt29 . so + 0x3b11e ]) 4 XENATIVESTACK ( 0xB6C53216 [ libj9prt29 . so + 0x3b216 ]) 4 XENATIVESTACK ( 0xB6C52CE3 [ libj9prt29 . so + 0x3ace3 ]) 4 XENATIVESTACK ( 0xB6C36F3E [ libj9prt29 . so + 0x1ef3e ]) 4 XENATIVESTACK ( 0xB6C52C57 [ libj9prt29 . so + 0x3ac57 ]) 4 XENATIVESTACK ( 0xB6C4E9CD [ libj9prt29 . so + 0x369cd ]) 4 XENATIVESTACK ( 0xB6C502FA [ libj9prt29 . so + 0x382fa ]) To simulate a Java OutOfMemoryError , this example application repeatedly appends characters to a StringBuffer object in an infinite loop. The Java callstack shows the HeapBreaker.main method appending characters ( java/lang/StringGuffer.append ) until the method java/lang/StringBuffer.ensureCapacityImpl() throws the OutOfMemoryError . StringBuffer objects are wrappers for character arrays ( char[] ) and when the capacity of the underlying array is reached, the contents are automatically copied into a new, larger array. The new array is created in the StringBuffer.ensureCapacity() method, which more or less doubles the size of the old array. In our scenario, the array takes up all the remaining space in the Java heap. The MEMINFO section of the Java dump file can also tell you when an unexpectedly large allocation request causes an OOM. Look for the GC History ( 1STGCHTYPE ) section, which details allocation requests that trigger GC activity. In the sample output you can see that a large allocation request ( requestedbytes=603979784 ) triggered a global GC. When the GC could not free up sufficient space in the heap to satisfy the request, the allocation failure generated the OOM. 1STGCHTYPE GC History \n3STHSTTYPE 14:29:29:580239000 GMT j9mm.101 - J9AllocateIndexableObject() returning NULL! 0 bytes requested for object of class B6BBC300 from memory space 'Generational' id=B6B49D20\n3STHSTTYPE 14:29:29:579916000 GMT j9mm.134 - Allocation failure end: newspace=2686912/3014656 oldspace=231597224/533856256 loa=5338112/5338112\n3STHSTTYPE 14:29:29:579905000 GMT j9mm.470 - Allocation failure cycle end: newspace=2686912/3014656 oldspace=231597224/533856256 loa=5338112/5338112\n3STHSTTYPE 14:29:29:579859000 GMT j9mm.475 - GlobalGC end: workstackoverflow=0 overflowcount=0 memory=234284136/536870912\n3STHSTTYPE 14:29:29:579807000 GMT j9mm.90 - GlobalGC collect complete\n3STHSTTYPE 14:29:29:579776000 GMT j9mm.137 - Compact end: bytesmoved=301989896\n3STHSTTYPE 14:29:29:313899000 GMT j9mm.136 - Compact start: reason=compact to meet allocation\n3STHSTTYPE 14:29:29:313555000 GMT j9mm.57 - Sweep end\n3STHSTTYPE 14:29:29:310772000 GMT j9mm.56 - Sweep start\n3STHSTTYPE 14:29:29:310765000 GMT j9mm.94 - Class unloading end: classloadersunloaded=0 classesunloaded=0\n3STHSTTYPE 14:29:29:310753000 GMT j9mm.60 - Class unloading start\n3STHSTTYPE 14:29:29:310750000 GMT j9mm.55 - Mark end\n3STHSTTYPE 14:29:29:306013000 GMT j9mm.54 - Mark start\n3STHSTTYPE 14:29:29:305957000 GMT j9mm.474 - GlobalGC start: globalcount=9\n3STHSTTYPE 14:29:29:305888000 GMT j9mm.475 - GlobalGC end: workstackoverflow=0 overflowcount=0 memory=234284136/536870912\n3STHSTTYPE 14:29:29:305837000 GMT j9mm.90 - GlobalGC collect complete\n3STHSTTYPE 14:29:29:305808000 GMT j9mm.137 - Compact end: bytesmoved=189784\n3STHSTTYPE 14:29:29:298042000 GMT j9mm.136 - Compact start: reason=compact to meet allocation\n3STHSTTYPE 14:29:29:297695000 GMT j9mm.57 - Sweep end\n3STHSTTYPE 14:29:29:291696000 GMT j9mm.56 - Sweep start\n3STHSTTYPE 14:29:29:291692000 GMT j9mm.55 - Mark end\n3STHSTTYPE 14:29:29:284994000 GMT j9mm.54 - Mark start\n3STHSTTYPE 14:29:29:284941000 GMT j9mm.474 - GlobalGC start: globalcount=8\n3STHSTTYPE 14:29:29:284916000 GMT j9mm.135 - Exclusive access: exclusiveaccessms=0.016 meanexclusiveaccessms=0.016 threads=0 lastthreadtid=0xB6B61100 beatenbyotherthread=0\n3STHSTTYPE 14:29:29:284914000 GMT j9mm.469 - Allocation failure cycle start: newspace=2678784/3014656 oldspace=80601248/533856256 loa=5338112/5338112 requestedbytes=603979784\n3STHSTTYPE 14:29:29:284893000 GMT j9mm.470 - Allocation failure cycle end: newspace=2678784/3014656 oldspace=80601248/533856256 loa=5338112/5338112\n3STHSTTYPE 14:29:29:284858000 GMT j9mm.560 - LocalGC end: rememberedsetoverflow=0 causedrememberedsetoverflow=0 scancacheoverflow=0 failedflipcount=0 failedflipbytes=0 failedtenurecount=0 failedtenurebytes=0 flipcount=2 flipbytes=64 newspace=2678784/3014656 oldspace=80601248/533856256 loa=5338112/5338112 tenureage=0\n3STHSTTYPE 14:29:29:284140000 GMT j9mm.140 - Tilt ratio: 89\n3STHSTTYPE 14:29:29:283160000 GMT j9mm.64 - LocalGC start: globalcount=8 scavengecount=335 weakrefs=0 soft=0 phantom=0 finalizers=0\n3STHSTTYPE 14:29:29:283123000 GMT j9mm.135 - Exclusive access: exclusiveaccessms=0.016 meanexclusiveaccessms=0.016 threads=0 lastthreadtid=0xB6B61100 beatenbyotherthread=0\n3STHSTTYPE 14:29:29:283120000 GMT j9mm.469 - Allocation failure cycle start: newspace=753616/3014656 oldspace=80601248/533856256 loa=5338112/5338112 requestedbytes=603979784\n3STHSTTYPE 14:29:29:283117000 GMT j9mm.133 - Allocation failure start: newspace=753616/3014656 oldspace=80601248/533856256 loa=5338112/5338112 requestedbytes=603979784\n3STHSTTYPE 14:29:29:269762000 GMT j9mm.134 - Allocation failure end: newspace=2686928/3014656 oldspace=80601248/533856256 loa=5338112/5338112\n3STHSTTYPE 14:29:29:269751000 GMT j9mm.470 - Allocation failure cycle end: newspace=2686976/3014656 oldspace=80601248/533856256 loa=5338112/5338112\n3STHSTTYPE 14:29:29:269718000 GMT j9mm.560 - LocalGC end: rememberedsetoverflow=0 causedrememberedsetoverflow=0 scancacheoverflow=0 failedflipcount=0 failedflipbytes=0 failedtenurecount=0 failedtenurebytes=0 flipcount=0 flipbytes=0 newspace=2686976/3014656 oldspace=80601248/533856256 loa=5338112/5338112 tenureage=0\n3STHSTTYPE 14:29:29:268981000 GMT j9mm.140 - Tilt ratio: 89\n3STHSTTYPE 14:29:29:268007000 GMT j9mm.64 - LocalGC start: globalcount=8 scavengecount=334 weakrefs=0 soft=0 phantom=0 finalizers=0\n3STHSTTYPE 14:29:29:267969000 GMT j9mm.135 - Exclusive access: exclusiveaccessms=0.016 meanexclusiveaccessms=0.016 threads=0 lastthreadtid=0xB6B61100 beatenbyotherthread=0\n3STHSTTYPE 14:29:29:267966000 GMT j9mm.469 - Allocation failure cycle start: newspace=0/3014656 oldspace=80601248/533856256 loa=5338112/5338112 requestedbytes=48\n3STHSTTYPE 14:29:29:267963000 GMT j9mm.133 - Allocation failure start: newspace=0/3014656 oldspace=80601248/533856256 loa=5338112/5338112 requestedbytes=48\n3STHSTTYPE 14:29:29:249015000 GMT j9mm.134 - Allocation failure end: newspace=2686928/3014656 oldspace=80601248/533856256 loa=5338112/5338112\n3STHSTTYPE 14:29:29:249003000 GMT j9mm.470 - Allocation failure cycle end: newspace=2686976/3014656 oldspace=80601248/533856256 loa=5338112/5338112\n3STHSTTYPE 14:29:29:248971000 GMT j9mm.560 - LocalGC end: rememberedsetoverflow=0 causedrememberedsetoverflow=0 scancacheoverflow=0 failedflipcount=0 failedflipbytes=0 failedtenurecount=0 failedtenurebytes=0 flipcount=0 flipbytes=0 newspace=2686976/3014656 oldspace=80601248/533856256 loa=5338112/5338112 tenureage=0 Although the Java code we used in this scenario deliberately triggered an OutOfMemoryError in a pronounced way, similar allocation issues can and do occur when dealing with large data sets such as XML files. The next step in diagnosing the problem is to open the system dump that gets generated automatically when an OutOfMemoryError occurs. Open\nthe dump with the Eclipse Memory Analyzer tool (MAT) and search for the StringBuffer object, which should provide further clues about what went wrong. A common example is seeing the same String duplicated over and over again, which might indicate that code is stuck in a loop. Note: If you want to use MAT to analyze your system dump, you must install the Diagnostic Tool Framework for Java (DTFJ) plugin in the Eclipse IDE. Select the following menu items: Help > Install New Software > Work with \"IBM Diagnostic Tool Framework for Java\" > If, unlike the previous scenario, you receive an OutOfMemoryError and the MEMINFO section shows that there is very little space left\non the Java heap, the current thread information is not important. The current thread is simply the thread that happened to be current when the space ran out. In this situation you might want to increase your Java heap size. For help with this task, see How to do heap sizing .",
+ "text": "In this scenario, the Java heap runs out of memory, causing an OutOfMemoryError , which automatically generates a Java dump file. The first section of the file (TITLE) tells you that a systhrow event triggered the Java dump as a result of an OOM ( java/lang/OutOfMemoryError ) for\nJava heap space. 0SECTION TITLE subcomponent dump routine\nNULL ===============================\n1TICHARSET UTF-8\n1TISIGINFO Dump Event \"systhrow\" (00040000) Detail \"java/lang/OutOfMemoryError\" \"Java heap space\" received\n1TIDATETIME Date: 2018/09/14 at 15:29:42:709\n1TINANOTIME System nanotime: 3635648876608448\n1TIFILENAME Javacore filename: /home/cheesemp/test/javacore.20180914.152929.18885.0003.txt\n1TIREQFLAGS Request Flags: 0x81 (exclusive+preempt)\n1TIPREPSTATE Prep State: 0x104 (exclusive_vm_access+trace_disabled) The MEMINFO section records how much memory is allocated to the Java heap ( 1STHEAPTYPE Object Memory ), how much is in use, and how much is free. Solving\nyour problem might be as simple as setting a larger heap size when you start your application. If you don't know what size the Java heap was set to, you might find that information in the ENVINFO section, which records the command line options that\nwere used when the application started. Look or search for the 1CIUSERARGS UserArgs: string and review the entries recorded for all lines that\nstart 2CIUSERARG . The Java heap size is set by the -Xmx option. If the size has not been set on the command line by -Xmx , the default value applies, which\nyou can find in Default Settings . In this scenario the solution to the problem is not an adjustment to the Java heap size. Here is the MEMINFO section: 0SECTION MEMINFO subcomponent dump routine\nNULL =================================\nNULL\n1STHEAPTYPE Object Memory\nNULL id start end size space/region\n1STHEAPSPACE 0xB6B49D20 -- -- -- Generational\n1STHEAPREGION 0xB6B4A078 0x95750000 0xB5470000 0x1FD20000 Generational/Tenured Region\n1STHEAPREGION 0xB6B49F10 0xB5470000 0xB54C0000 0x00050000 Generational/Nursery Region\n1STHEAPREGION 0xB6B49DA8 0xB54C0000 0xB5750000 0x00290000 Generational/Nursery Region\nNULL\n1STHEAPTOTAL Total memory: 536870912 (0x20000000)\n1STHEAPINUSE Total memory in use: 302603160 (0x12095B98)\n1STHEAPFREE Total memory free: 234267752 (0x0DF6A468) The output shows that only 56% of the Java heap is in use, so this suggests that the application is trying to do something sub-optimal. To\ninvestigate further you need to work out which thread was the current thread when the OOM occurred to see what it was trying to do. As in the previous scenario, you can find the current thread in the THREADS section. Here is an extract from the output: 0 SECTION THREADS subcomponent dump routine NULL ================================= NULL 1 XMPOOLINFO JVM Thread pool info : 2 XMPOOLTOTAL Current total number of pooled threads : 16 2 XMPOOLLIVE Current total number of live threads : 16 2 XMPOOLDAEMON Current total number of live daemon threads : 15 NULL 1 XMCURTHDINFO Current thread 3 XMTHREADINFO \"main\" J9VMThread : 0xB6B60C00 , omrthread_t : 0xB6B049D8 , java / lang / Thread : 0x95764520 , state : R , prio = 5 3 XMJAVALTHREAD ( java / lang / Thread getId : 0x1 , isDaemon : false ) 3 XMTHREADINFO1 ( native thread ID : 0x49C6 , native priority : 0x5 , native policy : UNKNOWN , vmstate : R , vm thread flags : 0x00001020 ) 3 XMTHREADINFO2 ( native stack address range from : 0xB6CB5000 , to : 0xB74B6000 , size : 0x801000 ) 3 XMCPUTIME CPU usage total : 8.537823831 secs , current category = \"Application\" 3 XMHEAPALLOC Heap bytes allocated since last GC cycle = 0 ( 0x0 ) 3 XMTHREADINFO3 Java callstack : 4 XESTACKTRACE at java / lang / StringBuffer . ensureCapacityImpl ( StringBuffer . java : 696 ) 4 XESTACKTRACE at java / lang / StringBuffer . append ( StringBuffer . java : 486 ( Compiled Code )) 5 XESTACKTRACE ( entered lock : java / lang / StringBuffer @0x957645B8 , entry count : 1 ) 4 XESTACKTRACE at java / lang / StringBuffer . append ( StringBuffer . java : 428 ( Compiled Code )) 4 XESTACKTRACE at HeapBreaker . main ( HeapBreaker . java : 34 ( Compiled Code )) 3 XMTHREADINFO3 Native callstack : 4 XENATIVESTACK ( 0xB6C535B3 [ libj9prt29 . so + 0x3b5b3 ]) 4 XENATIVESTACK ( 0xB6C36F3E [ libj9prt29 . so + 0x1ef3e ]) 4 XENATIVESTACK ( 0xB6C5311E [ libj9prt29 . so + 0x3b11e ]) 4 XENATIVESTACK ( 0xB6C53216 [ libj9prt29 . so + 0x3b216 ]) 4 XENATIVESTACK ( 0xB6C52CE3 [ libj9prt29 . so + 0x3ace3 ]) 4 XENATIVESTACK ( 0xB6C36F3E [ libj9prt29 . so + 0x1ef3e ]) 4 XENATIVESTACK ( 0xB6C52C57 [ libj9prt29 . so + 0x3ac57 ]) 4 XENATIVESTACK ( 0xB6C4E9CD [ libj9prt29 . so + 0x369cd ]) 4 XENATIVESTACK ( 0xB6C502FA [ libj9prt29 . so + 0x382fa ]) To simulate a Java OutOfMemoryError , this example application repeatedly appends characters to a StringBuffer object in an infinite loop. The Java callstack shows the HeapBreaker.main method appending characters ( java/lang/StringGuffer.append ) until the method java/lang/StringBuffer.ensureCapacityImpl() throws the OutOfMemoryError . StringBuffer objects are wrappers for character arrays ( char[] ) and when the capacity of the underlying array is reached, the contents are automatically copied into a new, larger array. The new array is created in the StringBuffer.ensureCapacity() method, which more or less doubles the size of the old array. In this scenario, the array takes up all the remaining space in the Java heap. The MEMINFO section of the Java dump file can also tell you when an unexpectedly large allocation request causes an OOM. Look for the GC History ( 1STGCHTYPE ) section, which details allocation requests that trigger GC activity. In the sample output you can see that a large allocation request ( requestedbytes=603979784 ) triggered a global GC. When the GC could not free up sufficient space in the heap to satisfy the request, the allocation failure generated the OOM. 1STGCHTYPE GC History \n3STHSTTYPE 14:29:29:580239000 GMT j9mm.101 - J9AllocateIndexableObject() returning NULL! 0 bytes requested for object of class B6BBC300 from memory space 'Generational' id=B6B49D20\n3STHSTTYPE 14:29:29:579916000 GMT j9mm.134 - Allocation failure end: newspace=2686912/3014656 oldspace=231597224/533856256 loa=5338112/5338112\n3STHSTTYPE 14:29:29:579905000 GMT j9mm.470 - Allocation failure cycle end: newspace=2686912/3014656 oldspace=231597224/533856256 loa=5338112/5338112\n3STHSTTYPE 14:29:29:579859000 GMT j9mm.475 - GlobalGC end: workstackoverflow=0 overflowcount=0 memory=234284136/536870912\n3STHSTTYPE 14:29:29:579807000 GMT j9mm.90 - GlobalGC collect complete\n3STHSTTYPE 14:29:29:579776000 GMT j9mm.137 - Compact end: bytesmoved=301989896\n3STHSTTYPE 14:29:29:313899000 GMT j9mm.136 - Compact start: reason=compact to meet allocation\n3STHSTTYPE 14:29:29:313555000 GMT j9mm.57 - Sweep end\n3STHSTTYPE 14:29:29:310772000 GMT j9mm.56 - Sweep start\n3STHSTTYPE 14:29:29:310765000 GMT j9mm.94 - Class unloading end: classloadersunloaded=0 classesunloaded=0\n3STHSTTYPE 14:29:29:310753000 GMT j9mm.60 - Class unloading start\n3STHSTTYPE 14:29:29:310750000 GMT j9mm.55 - Mark end\n3STHSTTYPE 14:29:29:306013000 GMT j9mm.54 - Mark start\n3STHSTTYPE 14:29:29:305957000 GMT j9mm.474 - GlobalGC start: globalcount=9\n3STHSTTYPE 14:29:29:305888000 GMT j9mm.475 - GlobalGC end: workstackoverflow=0 overflowcount=0 memory=234284136/536870912\n3STHSTTYPE 14:29:29:305837000 GMT j9mm.90 - GlobalGC collect complete\n3STHSTTYPE 14:29:29:305808000 GMT j9mm.137 - Compact end: bytesmoved=189784\n3STHSTTYPE 14:29:29:298042000 GMT j9mm.136 - Compact start: reason=compact to meet allocation\n3STHSTTYPE 14:29:29:297695000 GMT j9mm.57 - Sweep end\n3STHSTTYPE 14:29:29:291696000 GMT j9mm.56 - Sweep start\n3STHSTTYPE 14:29:29:291692000 GMT j9mm.55 - Mark end\n3STHSTTYPE 14:29:29:284994000 GMT j9mm.54 - Mark start\n3STHSTTYPE 14:29:29:284941000 GMT j9mm.474 - GlobalGC start: globalcount=8\n3STHSTTYPE 14:29:29:284916000 GMT j9mm.135 - Exclusive access: exclusiveaccessms=0.016 meanexclusiveaccessms=0.016 threads=0 lastthreadtid=0xB6B61100 beatenbyotherthread=0\n3STHSTTYPE 14:29:29:284914000 GMT j9mm.469 - Allocation failure cycle start: newspace=2678784/3014656 oldspace=80601248/533856256 loa=5338112/5338112 requestedbytes=603979784\n3STHSTTYPE 14:29:29:284893000 GMT j9mm.470 - Allocation failure cycle end: newspace=2678784/3014656 oldspace=80601248/533856256 loa=5338112/5338112\n3STHSTTYPE 14:29:29:284858000 GMT j9mm.560 - LocalGC end: rememberedsetoverflow=0 causedrememberedsetoverflow=0 scancacheoverflow=0 failedflipcount=0 failedflipbytes=0 failedtenurecount=0 failedtenurebytes=0 flipcount=2 flipbytes=64 newspace=2678784/3014656 oldspace=80601248/533856256 loa=5338112/5338112 tenureage=0\n3STHSTTYPE 14:29:29:284140000 GMT j9mm.140 - Tilt ratio: 89\n3STHSTTYPE 14:29:29:283160000 GMT j9mm.64 - LocalGC start: globalcount=8 scavengecount=335 weakrefs=0 soft=0 phantom=0 finalizers=0\n3STHSTTYPE 14:29:29:283123000 GMT j9mm.135 - Exclusive access: exclusiveaccessms=0.016 meanexclusiveaccessms=0.016 threads=0 lastthreadtid=0xB6B61100 beatenbyotherthread=0\n3STHSTTYPE 14:29:29:283120000 GMT j9mm.469 - Allocation failure cycle start: newspace=753616/3014656 oldspace=80601248/533856256 loa=5338112/5338112 requestedbytes=603979784\n3STHSTTYPE 14:29:29:283117000 GMT j9mm.133 - Allocation failure start: newspace=753616/3014656 oldspace=80601248/533856256 loa=5338112/5338112 requestedbytes=603979784\n3STHSTTYPE 14:29:29:269762000 GMT j9mm.134 - Allocation failure end: newspace=2686928/3014656 oldspace=80601248/533856256 loa=5338112/5338112\n3STHSTTYPE 14:29:29:269751000 GMT j9mm.470 - Allocation failure cycle end: newspace=2686976/3014656 oldspace=80601248/533856256 loa=5338112/5338112\n3STHSTTYPE 14:29:29:269718000 GMT j9mm.560 - LocalGC end: rememberedsetoverflow=0 causedrememberedsetoverflow=0 scancacheoverflow=0 failedflipcount=0 failedflipbytes=0 failedtenurecount=0 failedtenurebytes=0 flipcount=0 flipbytes=0 newspace=2686976/3014656 oldspace=80601248/533856256 loa=5338112/5338112 tenureage=0\n3STHSTTYPE 14:29:29:268981000 GMT j9mm.140 - Tilt ratio: 89\n3STHSTTYPE 14:29:29:268007000 GMT j9mm.64 - LocalGC start: globalcount=8 scavengecount=334 weakrefs=0 soft=0 phantom=0 finalizers=0\n3STHSTTYPE 14:29:29:267969000 GMT j9mm.135 - Exclusive access: exclusiveaccessms=0.016 meanexclusiveaccessms=0.016 threads=0 lastthreadtid=0xB6B61100 beatenbyotherthread=0\n3STHSTTYPE 14:29:29:267966000 GMT j9mm.469 - Allocation failure cycle start: newspace=0/3014656 oldspace=80601248/533856256 loa=5338112/5338112 requestedbytes=48\n3STHSTTYPE 14:29:29:267963000 GMT j9mm.133 - Allocation failure start: newspace=0/3014656 oldspace=80601248/533856256 loa=5338112/5338112 requestedbytes=48\n3STHSTTYPE 14:29:29:249015000 GMT j9mm.134 - Allocation failure end: newspace=2686928/3014656 oldspace=80601248/533856256 loa=5338112/5338112\n3STHSTTYPE 14:29:29:249003000 GMT j9mm.470 - Allocation failure cycle end: newspace=2686976/3014656 oldspace=80601248/533856256 loa=5338112/5338112\n3STHSTTYPE 14:29:29:248971000 GMT j9mm.560 - LocalGC end: rememberedsetoverflow=0 causedrememberedsetoverflow=0 scancacheoverflow=0 failedflipcount=0 failedflipbytes=0 failedtenurecount=0 failedtenurebytes=0 flipcount=0 flipbytes=0 newspace=2686976/3014656 oldspace=80601248/533856256 loa=5338112/5338112 tenureage=0 Although the Java code that was used in this scenario deliberately triggered an OutOfMemoryError in a pronounced way, similar allocation issues can and do occur when dealing with large data sets such as XML files. The next step in diagnosing the problem is to open the system dump that gets generated automatically when an OutOfMemoryError occurs. Open\nthe dump with the Eclipse Memory Analyzer tool (MAT) and search for the StringBuffer object, which should provide further clues about what went wrong. A common example is seeing the same String duplicated over and over again, which might indicate that code is stuck in a loop. Note: If you want to use MAT to analyze your system dump, you must install the Diagnostic Tool Framework for Java (DTFJ) plugin in the Eclipse IDE. Select the following menu items: Help > Install New Software > Work with \"IBM Diagnostic Tool Framework for Java\" > If, unlike the previous scenario, you receive an OutOfMemoryError and the MEMINFO section shows that there is very little space left\non the Java heap, the current thread information is typically not important. The current thread is simply the thread that happened to be current when the space ran out. In this situation you might want to increase your Java heap size. For help with this task, see How to do heap sizing .",
"title": "Java OutOfMemoryError"
},
{
+ "location": "/dump_javadump/#native-outofmemoryerror",
+ "text": "In this scenario, the VM runs out of native memory. Native memory is memory that is used by the VM for storing all virtualized resources and data that it needs for VM operations. Native memory that is available to the VM process is limited by the operating system. The native memory available to the VM might also be subject to additional limits imposed by the operating system, for example Unix ulimits . When a NativeOutOfMemoryError occurs, a Java dump is generated by default. The first section of the file (TITLE) tells you that a systhrow event triggered the Java dump as a result of an OOM ( java/lang/OutOfMemoryError ) for native memory. 0SECTION TITLE subcomponent dump routine\nNULL ===============================\n1TICHARSET UTF-8\n1TISIGINFO Dump Event \"systhrow\" (00040000) Detail \"java/lang/OutOfMemoryError\" \"native memory exhausted\" received\n1TIDATETIME Date: 2018/09/14 at 15:49:55:887\n1TINANOTIME System nanotime: 3636862054495675\n1TIFILENAME Javacore filename: /home/cheesemp/test/javacore.20180914.154814.19708.0003.txt\n1TIREQFLAGS Request Flags: 0x81 (exclusive+preempt)\n1TIPREPSTATE Prep State: 0x104 (exclusive_vm_access+trace_disabled) Sometimes, the current thread is responsible for causing the NativeOutOfMemoryError . Information about the current thread can be found in the THREADS section, as shown in the following output. 0SECTION THREADS subcomponent dump routine\nNULL =================================\nNULL\n1XMPOOLINFO JVM Thread pool info:\n2XMPOOLTOTAL Current total number of pooled threads: 16\n2XMPOOLLIVE Current total number of live threads: 16\n2XMPOOLDAEMON Current total number of live daemon threads: 15\nNULL \n1XMCURTHDINFO Current thread\n3XMTHREADINFO \"main\" J9VMThread:0xB6C60C00, omrthread_t:0xB6C049D8, java/lang/Thread:0xB55E3C10, state:R, prio=5\n3XMJAVALTHREAD (java/lang/Thread getId:0x1, isDaemon:false)\n3XMTHREADINFO1 (native thread ID:0x4CFD, native priority:0x5, native policy:UNKNOWN, vmstate:R, vm thread flags:0x00001020)\n3XMTHREADINFO2 (native stack address range from:0xB6D4E000, to:0xB754F000, size:0x801000)\n3XMCPUTIME CPU usage total: 3.654896026 secs, current category=\"Application\"\n3XMHEAPALLOC Heap bytes allocated since last GC cycle=0 (0x0)\n3XMTHREADINFO3 Java callstack:\n4XESTACKTRACE at sun/misc/Unsafe.allocateDBBMemory(Native Method)\n4XESTACKTRACE at java/nio/DirectByteBuffer.<init>(DirectByteBuffer.java:127(Compiled Code))\n4XESTACKTRACE at java/nio/ByteBuffer.allocateDirect(ByteBuffer.java:311)\n4XESTACKTRACE at NativeHeapBreaker.main(NativeHeapBreaker.java:9)\n3XMTHREADINFO3 Native callstack:\n4XENATIVESTACK (0xB6A9F5B3 [libj9prt29.so+0x3b5b3])\n...\n4XENATIVESTACK (0xB582CC9C [libjclse7b_29.so+0x40c9c])\n4XENATIVESTACK Java_sun_misc_Unsafe_allocateDBBMemory+0x88 (0xB5827F6B [libjclse7b_29.so+0x3bf6b])\n4XENATIVESTACK (0x94A2084A [<unknown>+0x0])\n4XENATIVESTACK (0xB6B2538B [libj9vm29.so+0x6c38b])\n4XENATIVESTACK (0xB6B4074C [libj9vm29.so+0x8774c])\n4XENATIVESTACK (0xB6B7F299 [libj9vm29.so+0xc6299])\n4XENATIVESTACK (0xB6A82F3E [libj9prt29.so+0x1ef3e])\n4XENATIVESTACK (0xB6B7F32A [libj9vm29.so+0xc632a])\n4XENATIVESTACK (0xB6B4084C [libj9vm29.so+0x8784c])\n4XENATIVESTACK (0xB6B3CD0C [libj9vm29.so+0x83d0c])\n4XENATIVESTACK (0xB776F87D [libjli.so+0x787d])\n4XENATIVESTACK (0xB7784F72 [libpthread.so.0+0x6f72])\n4XENATIVESTACK clone+0x5e (0xB76A043E [libc.so.6+0xee43e]) For clarity in the Native callstack output, ... indicates that some lines are removed. The Java callstack shows the transition from Java to native code ( sun/misc/Unsafe.allocateDBBMemory(Native Method) ), indicating a request for Direct Byte Buffer (DBB) storage. DBB storage is backed by native memory, with the Java heap containing only a reference to the native heap buffer. In this scenario, DBB storage is the likely culprit for this NativeOutOfMemoryError . The next step is to investigate the NATIVEMEMINFO section of the Java dump file, which reports the amount of memory used by the JRE process, broken down into component areas. 0SECTION NATIVEMEMINFO subcomponent dump routine\nNULL =================================\n0MEMUSER\n1MEMUSER JRE: 3,166,386,688 bytes / 4388 allocations\n1MEMUSER |\n2MEMUSER +--VM: 563,176,824 bytes / 1518 allocations\n2MEMUSER | |\n3MEMUSER | +--Classes: 3,104,416 bytes / 120 allocations\n2MEMUSER | |\n3MEMUSER | +--Memory Manager (GC): 548,181,888 bytes / 398 allocations\n3MEMUSER | | |\n4MEMUSER | | +--Java Heap: 536,932,352 bytes / 1 allocation\n3MEMUSER | | |\n4MEMUSER | | +--Other: 11,249,536 bytes / 397 allocations\n2MEMUSER | |\n3MEMUSER | +--Threads: 10,817,120 bytes / 147 allocations\n3MEMUSER | | |\n4MEMUSER | | +--Java Stack: 115,584 bytes / 16 allocations\n3MEMUSER | | |\n4MEMUSER | | +--Native Stack: 10,616,832 bytes / 17 allocations\n3MEMUSER | | |\n4MEMUSER | | +--Other: 84,704 bytes / 114 allocations\n2MEMUSER | |\n3MEMUSER | +--Trace: 163,688 bytes / 268 allocations\n2MEMUSER | |\n3MEMUSER | +--JVMTI: 17,320 bytes / 13 allocations\n2MEMUSER | |\n3MEMUSER | +--JNI: 23,296 bytes / 55 allocations\n2MEMUSER | |\n3MEMUSER | +--Port Library: 8,576 bytes / 74 allocations\n2MEMUSER | |\n3MEMUSER | +--Other: 860,520 bytes / 443 allocations\n1MEMUSER |\n2MEMUSER +--JIT: 3,744,728 bytes / 122 allocations\n2MEMUSER | |\n3MEMUSER | +--JIT Code Cache: 2,097,152 bytes / 1 allocation\n2MEMUSER | |\n3MEMUSER | +--JIT Data Cache: 524,336 bytes / 1 allocation\n2MEMUSER | |\n3MEMUSER | +--Other: 1,123,240 bytes / 120 allocations\n1MEMUSER |\n2MEMUSER +--Class Libraries: 2,599,463,024 bytes / 2732 allocations\n2MEMUSER | |\n3MEMUSER | +--Harmony Class Libraries: 1,024 bytes / 1 allocation\n2MEMUSER | |\n3MEMUSER | +--VM Class Libraries: 2,599,462,000 bytes / 2731 allocations\n3MEMUSER | | |\n4MEMUSER | | +--sun.misc.Unsafe: 2,598,510,480 bytes / 2484 allocations\n4MEMUSER | | | |\n5MEMUSER | | | +--Direct Byte Buffers: 2,598,510,480 bytes / 2484 allocations\n3MEMUSER | | |\n4MEMUSER | | +--Other: 951,520 bytes / 247 allocations\n1MEMUSER |\n2MEMUSER +--Unknown: 2,112 bytes / 16 allocations\nNULL In the VM Class Libraries section, the amount of memory allocated for Direct Byte Buffers is shown. Because the NativeOutOfMemoryError was received on a small 32-bit system, a value of 2,598,510,480 bytes indicates that the operating system has run out of memory. On a larger UNIX system, the process might have run out of memory because of the ulimit setting. Increasing the value for ulimit might avoid the error, which you can do temporarily by setting ulimit -f unlimited in your current session. The theoretical maximum size for a 32-bit process is the size of the 32-bit address space, which is 4 GB. On most operating systems a portion of the address space for each process is used by the kernel, such that the real limit for 32-bit processes is actually significantly less than 4GB. As a result, running out of native memory with a 32-bit VM is quite common. The same 4 GB limit is also important if you are using a 64-bit VM with compressed references. In compressed references mode, all references to objects, classes, threads, and monitors are represented by 32-bit values for performance reasons, so these structures can be allocated only at 32-bit addresses. However, the operating system might place other allocations within this 4 GB of address space, and if this area becomes sufficiently full or fragmented, the VM throws a native NativeOutOfMemoryError error. These errors typically occur when the VM tries to create a new thread or load a class. The Current Thread History section should contain more information about what the thread was doing at the VM level when the NativeOutOfMemoryError error occurred. You can usually avoid this type of problem by using the -Xmcrs option to reserve a contiguous area of memory within the lowest 4GB of memory at VM startup. Another common cause of a NativeOutOfMemoryError is when an application loads duplicate classes. Classes are allocated outside of the Java heap in native memory. If the value reported for Classes in the NATIVEMEMINFO section is very large, duplicate classes might be the cause of your problem. The Eclipse Memory Analyzer Tool (MAT) can tell you if you have duplicate classes by using the Class Loader Explorer feature. Because a system dump is automatically generated as well as a Java dump in response to a NativeOutOfMemoryError , simply open the system dump in MAT to continue your diagnosis.",
+ "title": "Native OutOfMemoryError"
+ },
+ {
+ "location": "/dump_javadump/#deadlock",
+ "text": "Deadlocks occur when two threads attempt to synchronize on an object and lock an instance of a class. When this happens, your application stops responding and hangs. Generating a Java dump file will quickly tell you whether you have a deadlock situation. Trigger the Java dump by sending a SIGQUIT signal ( kill -3 ) to the VM. The VM can detect the most common types of deadlock scenario involving Java monitors. If this type of deadlock is detected, information is provided in the LOCKS section. More complex deadlocks, including those that involve a mixture of native mutexes and Java monitors, are not detected. Here is the output from the code that was used to cause a common deadlock scenario: NULL 1L KDEADLOCK Deadlock detected !!! NULL --------------------- NULL 2L KDEADLOCKTHR Thread \"Worker Thread 2\" ( 0x94501D00 ) 3L KDEADLOCKWTR is waiting for : 4L KDEADLOCKMON sys_mon_t : 0x08C2B344 infl_mon_t : 0x08C2B384 : 4L KDEADLOCKOBJ java / lang / Object @0xB5666698 3L KDEADLOCKOWN which is owned by : 2L KDEADLOCKTHR Thread \"Worker Thread 3\" ( 0x94507500 ) 3L KDEADLOCKWTR which is waiting for : 4L KDEADLOCKMON sys_mon_t : 0x08C2B3A0 infl_mon_t : 0x08C2B3E0 : 4L KDEADLOCKOBJ java / lang / Object @0xB5666678 3L KDEADLOCKOWN which is owned by : 2L KDEADLOCKTHR Thread \"Worker Thread 1\" ( 0x92A3EC00 ) 3L KDEADLOCKWTR which is waiting for : 4L KDEADLOCKMON sys_mon_t : 0x08C2B2E8 infl_mon_t : 0x08C2B328 : 4L KDEADLOCKOBJ java / lang / Object @0xB5666688 3L KDEADLOCKOWN which is owned by : 2L KDEADLOCKTHR Thread \"Worker Thread 2\" ( 0x94501D00 ) This output tells you that Worker Thread 2 is waiting for Worker Thread 3 , which is waiting for Worker Thread 1 . Because Worker Thread 1 is also waiting for Worker Thread 2 , there is a deadlock. The next place to look is the output for Java and native stacks, in the THREADS section. By looking at the stack for each of these worker threads you can trace the problem back to specific lines in your application code. In this example, you can see from the following output that for all worker threads, the stack traces ( 4XESTACKTRACE / 5XESTACKTRACE ) indicate a problem in line 35 of the application DeadLockTest.java : 3 XMTHREADINFO \"Worker Thread 1\" J9VMThread : 0x92A3EC00 , omrthread_t : 0x92A3C2B0 , java / lang / Thread : 0xB5666778 , state : B , prio = 5 3 XMJAVALTHREAD ( java / lang / Thread getId : 0x13 , isDaemon : false ) 3 XMTHREADINFO1 ( native thread ID : 0x52CF , native priority : 0x5 , native policy : UNKNOWN , vmstate : B , vm thread flags : 0x00000201 ) 3 XMTHREADINFO2 ( native stack address range from : 0x9297E000 , to : 0x929BF000 , size : 0x41000 ) 3 XMCPUTIME CPU usage total : 0.004365543 secs , current category = \"Application\" 3 XMTHREADBLOCK Blocked on : java / lang / Object @0xB5666688 Owned by : \"Worker Thread 2\" ( J9VMThread : 0x94501D00 , java / lang / Thread : 0xB56668D0 ) 3 XMHEAPALLOC Heap bytes allocated since last GC cycle = 0 ( 0x0 ) 3 XMTHREADINFO3 Java callstack : 4 XESTACKTRACE at WorkerThread . run ( DeadLockTest . java : 35 ) 5 XESTACKTRACE ( entered lock : java / lang / Object @0xB5666678 , entry count : 1 ) ... 3 XMTHREADINFO \"Worker Thread 2\" J9VMThread : 0x94501D00 , omrthread_t : 0x92A3C8F0 , java / lang / Thread : 0xB56668D0 , state : B , prio = 5 3 XMJAVALTHREAD ( java / lang / Thread getId : 0x14 , isDaemon : false ) 3 XMTHREADINFO1 ( native thread ID : 0x52D0 , native priority : 0x5 , native policy : UNKNOWN , vmstate : B , vm thread flags : 0x00000201 ) 3 XMTHREADINFO2 ( native stack address range from : 0x946BF000 , to : 0x94700000 , size : 0x41000 ) 3 XMCPUTIME CPU usage total : 0.004555580 secs , current category = \"Application\" 3 XMTHREADBLOCK Blocked on : java / lang / Object @0xB5666698 Owned by : \"Worker Thread 3\" ( J9VMThread : 0x94507500 , java / lang / Thread : 0xB5666A18 ) 3 XMHEAPALLOC Heap bytes allocated since last GC cycle = 0 ( 0x0 ) 3 XMTHREADINFO3 Java callstack : 4 XESTACKTRACE at WorkerThread . run ( DeadLockTest . java : 35 ) 5 XESTACKTRACE ( entered lock : java / lang / Object @0xB5666688 , entry count : 1 ) ... 3 XMTHREADINFO \"Worker Thread 3\" J9VMThread : 0x94507500 , omrthread_t : 0x92A3CC10 , java / lang / Thread : 0xB5666A18 , state : B , prio = 5 3 XMJAVALTHREAD ( java / lang / Thread getId : 0x15 , isDaemon : false ) 3 XMTHREADINFO1 ( native thread ID : 0x52D1 , native priority : 0x5 , native policy : UNKNOWN , vmstate : B , vm thread flags : 0x00000201 ) 3 XMTHREADINFO2 ( native stack address range from : 0x9467E000 , to : 0x946BF000 , size : 0x41000 ) 3 XMCPUTIME CPU usage total : 0.003657010 secs , current category = \"Application\" 3 XMTHREADBLOCK Blocked on : java / lang / Object @0xB5666678 Owned by : \"Worker Thread 1\" ( J9VMThread : 0x92A3EC00 , java / lang / Thread : 0xB5666778 ) 3 XMHEAPALLOC Heap bytes allocated since last GC cycle = 0 ( 0x0 ) 3 XMTHREADINFO3 Java callstack : 4 XESTACKTRACE at WorkerThread . run ( DeadLockTest . java : 35 ) 5 XESTACKTRACE ( entered lock : java / lang / Object @0xB5666698 , entry count : 1 )",
+ "title": "Deadlock"
+ },
+ {
"location": "/dump_heapdump/",
"text": "Heap dump\n\n\nHeap dumps contain a snapshot of all the live objects that are being used by a running Java application on the Java heap. There are two formats for heap dumps; the classic format, which is ascii text and the PHD format, which is compressed and not readable.\n\n\nA heap dump contains a list of all object instances. For each object instance you can find the following additional data:\n\n\n\n\nThe object address\n\n\nThe type or class name\n\n\nThe size\n\n\nAny references to other objects\n\n\n\n\nFor a visual analysis of a heap dump as an aid to problem determination, use the \nEclipse Memory Analyzer tool (MAT)\n or the \nIBM Memory Analyzer tool\n. Both tools require the Diagnostic Tool Framework for Java (DTFJ) plugin. To install the DTFJ plugin in the Eclipse IDE, select the following menu items:\n\n\nHelp > Install New Software > Work with \"IBM Diagnostic Tool Framework for Java\" > IBM Monitoring and Diagnostic Tools > Diagnostic Tool Framework for Java \n\n\n\n\n\nFor more information about using the Heapdump feature, see \nUsing Heapdump\n.\n\n\nSee also\n\n\n\n\nUsing DTFJ",
"title": "Heap dump"
diff --git a/sitemap.xml b/sitemap.xml
index 18475f6..3736011 100644
--- a/sitemap.xml
+++ b/sitemap.xml
@@ -4,7 +4,7 @@
<url>
<loc>https://www.eclipse.org/openj9/docs/</loc>
- <lastmod>2018-11-09</lastmod>
+ <lastmod>2018-11-12</lastmod>
<changefreq>daily</changefreq>
</url>
@@ -12,7 +12,7 @@
<url>
<loc>https://www.eclipse.org/openj9/docs/introduction/</loc>
- <lastmod>2018-11-09</lastmod>
+ <lastmod>2018-11-12</lastmod>
<changefreq>daily</changefreq>
</url>
@@ -21,25 +21,25 @@
<url>
<loc>https://www.eclipse.org/openj9/docs/version0.11/</loc>
- <lastmod>2018-11-09</lastmod>
+ <lastmod>2018-11-12</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>https://www.eclipse.org/openj9/docs/version0.10/</loc>
- <lastmod>2018-11-09</lastmod>
+ <lastmod>2018-11-12</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>https://www.eclipse.org/openj9/docs/version0.9/</loc>
- <lastmod>2018-11-09</lastmod>
+ <lastmod>2018-11-12</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>https://www.eclipse.org/openj9/docs/version0.8/</loc>
- <lastmod>2018-11-09</lastmod>
+ <lastmod>2018-11-12</lastmod>
<changefreq>daily</changefreq>
</url>
@@ -48,7 +48,7 @@
<url>
<loc>https://www.eclipse.org/openj9/docs/gc/</loc>
- <lastmod>2018-11-09</lastmod>
+ <lastmod>2018-11-12</lastmod>
<changefreq>daily</changefreq>
</url>
@@ -56,7 +56,7 @@
<url>
<loc>https://www.eclipse.org/openj9/docs/jit/</loc>
- <lastmod>2018-11-09</lastmod>
+ <lastmod>2018-11-12</lastmod>
<changefreq>daily</changefreq>
</url>
@@ -64,7 +64,7 @@
<url>
<loc>https://www.eclipse.org/openj9/docs/aot/</loc>
- <lastmod>2018-11-09</lastmod>
+ <lastmod>2018-11-12</lastmod>
<changefreq>daily</changefreq>
</url>
@@ -72,7 +72,7 @@
<url>
<loc>https://www.eclipse.org/openj9/docs/shrc/</loc>
- <lastmod>2018-11-09</lastmod>
+ <lastmod>2018-11-12</lastmod>
<changefreq>daily</changefreq>
</url>
@@ -81,25 +81,25 @@
<url>
<loc>https://www.eclipse.org/openj9/docs/diag_overview/</loc>
- <lastmod>2018-11-09</lastmod>
+ <lastmod>2018-11-12</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>https://www.eclipse.org/openj9/docs/dump_javadump/</loc>
- <lastmod>2018-11-09</lastmod>
+ <lastmod>2018-11-12</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>https://www.eclipse.org/openj9/docs/dump_heapdump/</loc>
- <lastmod>2018-11-09</lastmod>
+ <lastmod>2018-11-12</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>https://www.eclipse.org/openj9/docs/dump_systemdump/</loc>
- <lastmod>2018-11-09</lastmod>
+ <lastmod>2018-11-12</lastmod>
<changefreq>daily</changefreq>
</url>
@@ -109,19 +109,19 @@
<url>
<loc>https://www.eclipse.org/openj9/docs/cmdline_specifying/</loc>
- <lastmod>2018-11-09</lastmod>
+ <lastmod>2018-11-12</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>https://www.eclipse.org/openj9/docs/cmdline_general/</loc>
- <lastmod>2018-11-09</lastmod>
+ <lastmod>2018-11-12</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>https://www.eclipse.org/openj9/docs/cmdline_migration/</loc>
- <lastmod>2018-11-09</lastmod>
+ <lastmod>2018-11-12</lastmod>
<changefreq>daily</changefreq>
</url>
@@ -149,19 +149,19 @@
<url>
<loc>https://www.eclipse.org/openj9/docs/tool_jdmpview/</loc>
- <lastmod>2018-11-09</lastmod>
+ <lastmod>2018-11-12</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>https://www.eclipse.org/openj9/docs/tool_traceformat/</loc>
- <lastmod>2018-11-09</lastmod>
+ <lastmod>2018-11-12</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>https://www.eclipse.org/openj9/docs/tool_builder/</loc>
- <lastmod>2018-11-09</lastmod>
+ <lastmod>2018-11-12</lastmod>
<changefreq>daily</changefreq>
</url>
@@ -171,31 +171,31 @@
<url>
<loc>https://www.eclipse.org/openj9/docs/openj9_support/</loc>
- <lastmod>2018-11-09</lastmod>
+ <lastmod>2018-11-12</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>https://www.eclipse.org/openj9/docs/openj9_defaults/</loc>
- <lastmod>2018-11-09</lastmod>
+ <lastmod>2018-11-12</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>https://www.eclipse.org/openj9/docs/openj9_directories/</loc>
- <lastmod>2018-11-09</lastmod>
+ <lastmod>2018-11-12</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>https://www.eclipse.org/openj9/docs/messages_intro/</loc>
- <lastmod>2018-11-09</lastmod>
+ <lastmod>2018-11-12</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>https://www.eclipse.org/openj9/docs/env_var/</loc>
- <lastmod>2018-11-09</lastmod>
+ <lastmod>2018-11-12</lastmod>
<changefreq>daily</changefreq>
</url>
@@ -204,7 +204,7 @@
<url>
<loc>https://www.eclipse.org/openj9/docs/legal/</loc>
- <lastmod>2018-11-09</lastmod>
+ <lastmod>2018-11-12</lastmod>
<changefreq>daily</changefreq>
</url>
