Get thread dump and heap dump for MID Server

Overview

On rare occasions when a MID Server cannot or is slow to pick up messages from the instance, you would like to get a thread dump to troubleshoot. This article describes how to do so. Note that you need to have login access to the MID Server.

How to get thread dump and heap dump

1. Download and install a JDK compatible with the MID Server JRE to get the jstack.exe executable from the JDK.

   You might run into some unsupported JVM version exceptions if you install the wrong JDK version.

2. Download and unzip PSTools: https://technet.microsoft.com/en-us/sysinternals/bb897553.aspx. You might not have to do this but PsExec is needed to invoke jstack.exe on a process that is run as Local System, which is typical when the MID Server is run as a Windows service.

3.  Open Task Manager. In 'Processes' tab, sort by Image Name then find 'java.exe' processes. Note the process ID under 'PID' column. If 'PID' column is not shown, goto View > Select Column... and select 'PID' column.

   

4. If multiple MID Servers are running, right click each 'java.exe' process then select Properties, to verify correct MID Server process.

    

    

5. Open Command Prompt as Administrator change to directory where PSTools is unzipped in step 2. Run the following command to get the MID Server thread dump

   if MID Server service account is 'SYSTEM':

   psexec -s "<path_to_jdk_install>\bin\jstack" -l PID_HERE >> <path_for_generated_file>\threadDump.txt

   if MID Server service account is a user:

   psexec -u <username> "<path_to_jdk_install>\bin\jstack" -l PID_HERE >> <path_for_generated_file>\threadDump.txt

6. Run following command to get the heap dump.

   if MID Server service account is 'SYSTEM':

   psexec -s "<path_to_jdk_install>\bin\jmap" -dump:file=<path_for_generated_file>\heapDump.bin PID_HERE

   if MID Server service account is a user:

   psexec -u "<path_to_jdk_install>\bin\jmap" -dump:file=<path_for_generated_file>\heapDump.bin PID_HERE

7. Alternately, to dump the heap automatically on out-of-memory exception, modify the wrapper-override.conf file found in the /agent/conf/ directory by adding the following:

        "wrapper.app.additional.2=-XX:+HeapDumpOnOutOfMemoryError" OR "wrapper.java.additional.2=-XX:+HeapDumpOnOutOfMemoryError"

             In the wrapper-override.conf file, the number 2 used above is the next unused number starting from 1. If 2 is being used then you will need to use 3 and so on.

             There should be no gaps in the numbers. For example if 1, 2, and 3 are used, do not use 5. 

             There should be no duplicates in either /agent/conf/wrapper.conf or /agent/conf/wrapper-override.conf files. 

             Here is an example:

***WARNING: This parameter should only be used for short term with active monitoring. As soon as OOM is reproduced and Heapdump is generated, the parameter must be removed. Leaving this parameter in place for extended period can potentially exhaust disk space.

References

jmap - Memory Map – How to use jmap

jstack - Stack Trace – How to use jstack