556 views

How to resolve Java crashes and Java out of memory errors on Edge Encryption proxies 

 

Overview
The following errors can occur related to the Java Virtual Machine (JVM) in either the edgeencryption.log or wrapper_<date>.log in the /logs directory of the Edge Proxy:

2017-06-15 20:58:33,091 ERROR Java heap space
2017-06-15 20:58:44,931 ERROR Java heap space
2017-06-15 21:00:11,359 ERROR Java heap space

OR

2016-11-29 09:18:26,765 [sn221104245-44485] WARN org.eclipse.jetty.util.thread.QueuedThreadPool - Unexpected thread death: org.eclipse.jetty.util.thread.QueuedThreadPool$3@3ad8d4eb in qtp221104245{STARTED,8<=200<=200,i=0,q=72}

2016-11-29 09:18:28,672 [sn221104245-44387] WARN com.snc.edgeencryption.EdgeEncryptionLog - GetRequestTransformer exception java.lang.OutOfMemoryError: GC overhead limit exceeded

2016-11-29 09:18:34,875 [sn221104245-44342] WARN org.eclipse.jetty.servlet.ServletHandler - Error for /amb/connect java.lang.OutOfMemoryError: GC overhead limit exceeded

2016-11-29 09:18:35,813 [sn221104245-44344] WARN org.eclipse.jetty.servlet.ServletHandler - Error for /amb/handshake java.lang.OutOfMemoryError: Java heap space

2016-11-29 09:18:39,703 [sn221104245-44337] WARN org.eclipse.jetty.servlet.ServletHandler - Error for /angular.do java.lang.OutOfMemoryError: Java heap space

2016-11-29 09:18:55,719 [sn221104245-44338] WARN org.eclipse.jetty.servlet.ServletHandler - Error for /amb/connect java.lang.OutOfMemoryError: GC overhead limit exceeded

2016-11-29 09:18:31,094 [sn221104245-44532] ERROR com.snc.edgeencryption.EdgeEncryptionLog - GC overhead limit exceeded

2016-11-29 09:18:55,719 [sn221104245-44484] ERROR com.snc.edgeencryption.EdgeEncryptionLog - GC overhead limit exceeded 

OR

INFO | jvm 64 | 2017/06/01 09:29:38.610 | [CodeBlob (0x0000000001af2290)]
INFO | jvm 64 | 2017/06/01 09:29:38.610 | Framesize: 0
INFO | jvm 64 | 2017/06/01 09:29:38.610 | BufferBlob (0x0000000001af2290) used for StubRoutines (2)
INFO | jvm 64 | 2017/06/01 09:29:40.033 | #
INFO | jvm 64 | 2017/06/01 09:29:40.033 | # A fatal error has been detected by the Java Runtime Environment:
INFO | jvm 64 | 2017/06/01 09:29:40.033 | #
INFO | jvm 64 | 2017/06/01 09:29:40.033 | # Internal Error (sharedRuntime.cpp:834), pid=3856, tid=3824
INFO | jvm 64 | 2017/06/01 09:29:40.033 | # fatal error: exception happened outside interpreter, nmethods and vtable stubs at pc 0x0000000001af4a10
INFO | jvm 64 | 2017/06/01 09:29:40.033 | #
INFO | jvm 64 | 2017/06/01 09:29:40.033 | # JRE version: Java(TM) SE Runtime Environment (8.0_40-b26) (build 1.8.0_40-b26)
INFO | jvm 64 | 2017/06/01 09:29:40.033 | # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode windows-amd64 compressed oops)
INFO | jvm 64 | 2017/06/01 09:29:40.033 | # Core dump written. Default location: C:\EdgeProxy\ServiceNow_443\bin\hs_err_pid3856.mdmp
INFO | jvm 64 | 2017/06/01 09:29:40.033 | #
INFO | jvm 64 | 2017/06/01 09:29:40.033 | # An error report file with more information is saved as:
INFO | jvm 64 | 2017/06/01 09:29:40.033 | # C:\EdgeProxy\ServiceNow_443\bin\hs_err_pid3856.log
INFO | jvm 64 | 2017/06/01 09:29:40.033 | #
INFO | jvm 64 | 2017/06/01 09:29:40.033 | # If you would like to submit a bug report, please visit:
INFO | jvm 64 | 2017/06/01 09:29:40.033 | # http://bugreport.java.com/bugreport/crash.jsp
INFO | jvm 64 | 2017/06/01 09:29:40.033 | #

ERROR | wrapper | 2017/06/01 09:29:40.142 | JVM exited unexpectedly.
STATUS | wrapper | 2017/06/01 09:29:44.739 | Launching a JVM...
INFO | jvm 65 | 2017/06/01 09:29:44.959 | Wrapper (Version 3.2.3.SNC.1) http://wrapper.organization.org
INFO | jvm 65 | 2017/06/01 09:29:44.959 | Copyright 1999-2006 Company, Inc. All Rights Reserved.

INFO | jvm 65 | 2017/06/01 09:29:44.959 |
INFO | jvm 65 | 2017/06/01 09:29:44.959 | WARNING - The Wrapper jar file currently in use is version "3.2.3.SNC.1"
INFO | jvm 65 | 2017/06/01 09:29:44.959 | while the version of the Wrapper which launched this JVM is
INFO | jvm 65 | 2017/06/01 09:29:44.959 | "3.2.3".
INFO | jvm 65 | 2017/06/01 09:29:44.959 | The Wrapper may appear to work correctly but some features may
INFO | jvm 65 | 2017/06/01 09:29:44.959 | not function correctly. This configuration has not been tested
INFO | jvm 65 | 2017/06/01 09:29:44.959 | and is not supported.
INFO | jvm 65 | 2017/06/01 09:29:44.959 |
INFO | jvm 65 | 2017/06/01 09:31:04.519 | >>> Total classes: 1,000 total length: 6,926,185


OR

ERROR  | wrapper  | 2017/06/22 19:14:28.388 | JVM exited unexpectedly.
STATUS | wrapper  | 2017/06/22 19:14:28.396 | JVM exited in response to signal SIGKILL (9).
STATUS | wrapper  | 2017/06/22 19:14:32.540 | Launching a JVM...?
INFO   | jvm 4    | 2017/06/22 19:14:34.543 | Wrapper (Version 3.2.3.SNC.1) http://wrapper.organization.org
INFO   | jvm 4    | 2017/06/22 19:14:34.543 |   Copyright 1999-2006 Company, Inc.  All Rights Reserved.

INFO   | jvm 4    | 2017/06/22 19:14:34.543 | 
INFO   | jvm 4    | 2017/06/22 19:14:35.952 | Jun 22, 2017 7:14:35 PM com.snc.da.cfg.GatewayScanner getTypesXmlResources
INFO   | jvm 4    | 2017/06/22 19:14:35.953 | INFO: Scanning for gateways using xml resource method
INFO   | jvm 4    | 2017/06/22 19:14:36.253 | Jun 22, 2017 7:14:36 PM com.snc.da.cfg.GatewayScanner getTypesMap

 

Symptoms
  • A blank broswer screen appears when trying to reach the instance via the Edge Proxy, or an error message similar to "This site cannot be reached" is displayed on the browser screen
  • Slow browser navigation when interacting with the instance via the Edge Proxy   

  

Cause
The Edge Proxy Java Virtual Machine (JVM) may be running out of memory or crashing.

 

Resolution
In order to resolve with these JVM related issues check and take the following actions.

Checking and upgrading, if necessary, the Java JRE that is used by the Edge Encryption proxy

  1. On the proxy, open the file: $Edge_Install_Directory/conf/wrapper.conf.
  2. Check the line containing "wrapper.java.command." it may look like the following:

    wrapper.java.command=../java/jre/bin/java
    or
    wrapper.java.command=/home/oracle/jre1.8.0_121/bin/java
    or
    wrapper.java.command=C:\Program Files\Java\jre1.8.0_144\bin\java

  3. Navigate to the bin directory as shown in the setting for wrapper.java.command.
  4. Find the version of the JRE by executing java -version.

    Linux:

    ./java -version
    java version "1.8.0_40"
    Java(TM) SE Runtime Environment (build 1.8.0_40-b27)
    Java HotSpot(TM) 64-Bit Server VM (build 25.40-b25, mixed mode)

    Windows:

    C:\istanbul_ee_install\istan_eep_443\java\jre\bin>java -version
    java version "1.8.0_40"
    Java(TM) SE Runtime Environment (build 1.8.0_40-b26)
    Java HotSpot(TM) 64-Bit Server VM (build 25.40-b25, mixed mode)

  5. If steps 3-4 indicate a jre version below 1.8.0_121, update the JRE to at least the 1.8.0_121 version by following the steps in the next section.

 

Upgrading the JRE for Windows

  1. Navigate to the jre download site at http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html.
  2. Accept the licensing agreement and download the appropriate version of the JRE for the operating system being used (for example, Windows x64 Offline -> jre-8u144-windows-x64.exe).
  3. After downloading, execute jre-8u144-windows-x64.exe, install using the GUI, and keep the older versions of the JRE if asked.
  4. Navigate to the installation directory, which is most likely: C:\Program Files\Java\jre1.8.0_xxx.
  5. If using 256 bit encryption keys, copy the the following two jar files from the original java JRE installation directory into the new one.

    From the old one at:

    C:\<Old JRE install directory>\java\jre\lib\security\local_policy.jar
    and
    C:\<Old JRE install directory>\java\jre\lib\security\US_export_policy.jar

    To the new one at:

    C:\Program Files\Java\jre1.8.0_xxx\lib\security\local_policy.jar
    and
    C:\Program Files\Java\jre1.8.0_xxx\lib\security\US_export_policy.jar

  6. Stop the Edge Proxy server.
  7. Modify the C:\<Edge_Install_Directory>\conf\wrapper.conf file to point to the new location of the java installation:

    wrapper.java.command=C:\Program Files\Java\jre1.8.0_xxx\bin\java

  8. Save the wrapper.conf.

The next time the proxy is started, it runs on the new jre version.

 

Upgrading the JRE for Linux

  1. Navigate to the jre download site at http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html.
  2. Accept the licensing agreement and download the appropriate version of the JRE for the operating system being used (for example, Linux x64 -> jre-8u144-linux-x64.tar.gz).
  3. Save the downloaded version to the Edge Proxy machines and open, unzip, and untar. The installation directory structure should look like:

    $directory_path/jre1.8.0_<version>

  4. If using 256 bit encryption keys, copy the following two jar files from the original java JRE installation directory into the new one.

    From the old one at:

    $Old_JRE_install_directory/java/jre/lib/security/local_policy.jar
    and
    $Old_JRE_install_directory/java/jre/lib/security/US_export_policy.jar

    To the new one at:

    $New_JRE_install_directory/java/jre/lib/security/local_policy.jar
    and
    $New_JRE_install_directory/java/jre/lib/security/US_export_policy.jar

  5. Stop the Edge Proxy server.
  6. Modify the $Edge_Install_Directory/conf/wrapper.conf file to point to the new location of the java installation:

    wrapper.java.command=$jre_install_directory/bin/java

    (For example, wrapper.java.command=/home/sn/jre1.8.0_144/bin/java.)

The next time the proxy is started, it runs on the new jre version.

 

Adding the java startup property -XX:-UseAESIntrinsics

Note: Do this step only if you did NOT follow the steps in the earlier section to upgrade the Java JRE that is used by the Edge Encryption proxy, since the startup property is not needed if the JRE is on 1.8.0_121 or later.

 

 

 

Add wrapper.java.additional.4=-XX:-UseAESIntrinsics to the wrapper.conf files for each proxy. (Note that the "4" in the property can be changed to a higher unused number if you are already using "4" for a different java startup property.)

Instructions can be found in:

KB0622015: Edge Encryption proxy goes to 95-100% CPU and fails with Java exception.

After the wrapper.conf files are modified, the proxies must be restarted.

 

Ensuring that the proxies are scaled correctly and have enough JVM heap memory allocated

The recommended requirements for the proxies are explained in the following product documentation pages:

When setting the initial and upper bound memory limits, allocate 6GB (ideally) to the proxy JVMs or at least 4GB (by default, the proxy is run with 2GB in the JVM). To increase the JVM memory follow this documentation:

To accommodate the 6GB of heap memory allocation for the proxy JVM, the machines hosting each proxy must have at least 8GB of RAM available. Leave at least 2GB to the operating system and 6GB to the proxy JVM.

To make the proxies run at 6GB, edit the wrapper.conf file at $Edge_Install_Directory/conf from this:

# Initial Java Heap Size (in MB)
#wrapper.java.initmemory=3
# Maximum Java Heap Size (in MB)
#wrapper.java.maxmemory=64

To this:

# Initial Java Heap Size (in MB)
wrapper.java.initmemory=6144
# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=6144

This change requires proxy restart.

To size the number of proxies needed, determine the number of proxies you need. For more information, see Sizing your Edge Encryption environment.

You must have a good idea of how many users are accessing the instance via the proxies at any peak time during the day. Keep in mind the number of proxies needed for "X," which is the number of simultaneous users. Add an extra proxy server for every X simultaneous users.

 

Checking if the Edge Proxy JVM is killed by the Linux Operating System

If the edge proxy runs on Linux, check if the wrapper_<date>.log shows this “SIGKILL (9)” error:

ERROR  | wrapper  | 2017/06/22 19:14:28.388 | JVM exited unexpectedly.
STATUS | wrapper  | 2017/06/22 19:14:28.396 | JVM exited in response to signal SIGKILL (9).

STATUS | wrapper  | 2017/06/22 19:14:32.540 | Launching a JVM...
INFO   | jvm 4    | 2017/06/22 19:14:34.543 | Wrapper (Version 3.2.3.SNC.1) http://wrapper.tanukisoftware.org
INFO   | jvm 4    | 2017/06/22 19:14:34.543 |   Copyright 1999-2006 Tanuki Software, Inc.  All Rights Reserved.

INFO   | jvm 4    | 2017/06/22 19:14:34.543 | >
INFO   | jvm 4    | 2017/06/22 19:14:35.952 | Jun 22, 2017 7:14:35 PM com.snc.da.cfg.GatewayScanner getTypesXmlResources
INFO   | jvm 4    | 2017/06/22 19:14:35.953 | INFO: Scanning for gateways using xml resource method
INFO   | jvm 4    | 2017/06/22 19:14:36.253 | Jun 22, 2017 7:14:36 PM com.snc.da.cfg.GatewayScanner getTypesMap

If this error is displayed, check the /var/log/messages file for more information. For example you may see:

Jun 22 19:14:28 eo-0ee971a048 kernel: Out of memory: Kill process 8405 (java) score 939 or sacrifice child

Jun 22 19:14:28 eo-0ee971a048 kernel: Killed process 8405 (java) total-vm:9275688kB, anon-rss:7259600kB, file-rss:0kB, shmem-rss:0kB

Jun 22 19:14:28 eo-0ee971a048 wrapper[2121]: JVM exited unexpectedly.

 

 

This may be due to PRB1075054 for more information see KB0656318

 

Article Information

Last Updated:2017-11-14 08:20:02
Published:2017-07-28