Notifications

366 views

Description

Using Oracle Java 8 Update 261 (JRE 1.8.0_261) with a MID Server installation causes the MID Server to go down with a EXCEPTION_ACCESS_VIOLATION  exception.

Potentially large memory dump (.mdmp) files will be left in the agent folder for each crash, that can have an impact on disk space of the whole server as well, eventually bringing the host server down too.

Oracle 1.8.0_251 does not cause the issue, and nor does the bundled OpenJDK 1.8.0_231.

Steps to Reproduce

  1. Install a MID Server
  2. Edit wrapper-override.conf to use a standalone Java install of Oracle 1.8.0_251, or earlier, or leave it using the bundled 1.8.0_231 jre.
  3. Upgrade that java install to 1.8.0_261

The MID Server will go down on startup with an exception relating to sigar-amd64-winnt.dll EXCEPTION_ACCESS_VIOLATION.

The agent log may show:

07/23/20 03:48:50 (075) Collector SEVERE *** ERROR *** Sigar exception when attempting to instantiate Cpu
org.hyperic.sigar.SigarException: Unknown OS Error
 at org.hyperic.sigar.Cpu.gather(Native Method)
 at org.hyperic.sigar.Cpu.fetch(Cpu.java:30)
 at org.hyperic.sigar.Sigar.getCpu(Sigar.java:320)
 at com.service_now.monitor.StatusMonitor$Collector.run(StatusMonitor.java:310)
 at java.util.TimerThread.mainLoop(Unknown Source)
 at java.util.TimerThread.run(Unknown Source)

The wrapper log and hs_err_pidXXXX files in the agect folder will contain this error message:

# A fatal error has been detected by the Java Runtime Environment:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x0000000010014ed4, pid=6116, tid=0x0000000000001b7c
#
# JRE version: Java(TM) SE Runtime Environment (8.0_261-b12) (build 1.8.0_261-b12)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.261-b12 mixed mode windows-amd64 compressed oops)
# Problematic frame:
# C [sigar-amd64-winnt.dll+0x14ed4]
#
# Core dump written. Default location: <MID Server install folder>\agent\hs_err_pid6116.mdmp
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.

The pid and tid values will be dirrect each time. e.g.

#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00000000006d4ed4, pid=492, tid=0x00000000000016ac
# Core dump written. Default location: <MID Server install folder>\agent\hs_err_pid492.mdmp

Once disk space on the server is used up, you may see this error in the wrapper log:

2020/07/24 13:00:47 | java.util.logging.ErrorManager: 2
2020/07/24 13:00:47 | java.io.IOException: There is not enough space on the disk
2020/07/24 13:00:47 |  at java.io.FileOutputStream.writeBytes(Native Method)
2020/07/24 13:00:47 |  at java.io.FileOutputStream.write(Unknown Source)
2020/07/24 13:00:47 |  at java.io.BufferedOutputStream.flushBuffer(Unknown Source)
2020/07/24 13:00:47 |  at java.io.BufferedOutputStream.flush(Unknown Source)
2020/07/24 13:00:47 |  at java.util.logging.FileHandler$MeteredStream.flush(Unknown Source)
2020/07/24 13:00:47 |  at sun.nio.cs.StreamEncoder.implFlush(Unknown Source)
2020/07/24 13:00:47 |  at sun.nio.cs.StreamEncoder.flush(Unknown Source)
2020/07/24 13:00:47 |  at java.io.OutputStreamWriter.flush(Unknown Source)
2020/07/24 13:00:47 |  at java.util.logging.StreamHandler.flush(Unknown Source)
2020/07/24 13:00:47 |  at java.util.logging.FileHandler.publish(Unknown Source)
2020/07/24 13:00:47 |  at java.util.logging.Logger.log(Unknown Source)
2020/07/24 13:00:47 |  at java.util.logging.Logger.doLog(Unknown Source)
2020/07/24 13:00:47 |  at java.util.logging.Logger.log(Unknown Source)
2020/07/24 13:00:47 |  at java.util.logging.Logger.info(Unknown Source)
2020/07/24 13:00:47 |  at com.glide.util.Log.info0(Log.java:155)
2020/07/24 13:00:47 |  at com.glide.util.Log.info(Log.java:134)
2020/07/24 13:00:47 |  at com.snc.core_automation_common.logging.LoggerImpl.info(LoggerImpl.java:45)
2020/07/24 13:00:47 |  at com.service_now.mid.services.Monitors.createMonitor(Monitors.java:228)
2020/07/24 13:00:47 |  at com.service_now.mid.services.Monitors.loadInternalMonitor(Monitors.java:153)
2020/07/24 13:00:47 |  at com.service_now.mid.services.Monitors.startFileSyncer(Monitors.java:141)
2020/07/24 13:00:47 |  at com.service_now.mid.services.StartupSequencer.waitForInitialFileSync(StartupSequencer.java:332)
2020/07/24 13:00:47 |  at com.service_now.mid.services.StartupSequencer.testsSucceeded(StartupSequencer.java:118)
2020/07/24 13:00:47 |  at com.service_now.mid.services.StartupSequencer.access$300(StartupSequencer.java:76)
2020/07/24 13:00:47 |  at com.service_now.mid.services.StartupSequencer$Starter.run(StartupSequencer.java:430)

Workaround

1. If customer desires the security updates from newer versions of java, upgrade your JVM to use Java 11.  It is important to note that this option will disable MID CPU monitoring functionality on Windows.

2. Downgrade your JVM to 8.0_251, or switch to using the bundled 1.8.0_231

You can contact ServiceNow Technical Support or subscribe to this Known Error article by clicking the Subscribe button at the top right of this form to be notified when more information will become available.


Related Problem: PRB1419594

Seen In

SR - IRM - Audit Management - New York 2019 Q3
SR - IRM - GRC Profiles - Madrid 2019 Q2
SR - IRM - GRC Workbench - New York 2019 Q3
SR - IRM - Policy and Compliance - Madrid 2019 Q2
SR - IRM - Risk Management - New York 2019 Q3
SR - IRM - SIG Assessment Legacy - Madrid 2019 Q1
SR - IRM - Vendor Risk Management - Madrid 2019 Q1
SR - ITOM - Discovery and Service Mapping - 201908
SR - ITOM - Discovery and Service Mapping - v1.0.35
SR - Security - Integration Framework - Madrid 2019 Q2
SR - Security - Support Common - Madrid 2019 Q2
SR - Security - Support Orchestration - Madrid 2019 Q2
SR - SIR - Security Incident Response - Madrid 2019 Q2
SR - SIR - Store SecOps Setup Assistant - Madrid 2019 Q2
SR - SIR - Store Threat Core - Madrid 2019 Q2
SR - SIR - Store Trusted Security Circles Client - New York 2019 Q3
SR - VR - Configuration Compliance - New York 2019 Q3
SR - VR - Qualys - New York 2019 Q3
SR - VR - Shodan Exploit - New York 2019 Q3
SR - VR - Solution Management Madrid Q2
SR - VR - Vulnerability Response - New York 2019 Q3
SR - VR - Vulnerability Response PA Content - Madrid 2019 Q2

Intended Fix Version

Quebec

Safe Harbor Statement

This "Intended Fix Version" information is meant to outline ServiceNow's general product direction and should not be relied upon in making a purchasing decision. The information provided here is for information purposes only and may not be incorporated into any contract. It is not a commitment, promise, or legal obligation to deliver any material, code, or functionality. The development, release, and timing of any features or functionality described for our products remains at ServiceNow's sole discretion.

Associated Community Threads

There is no data to report.

Article Information

Last Updated:2020-09-11 01:12:25
Published:2020-09-11