Notifications

2109 views

Description

When upgrading a MID server hosted on a CentOS Linux server, and other Linux distributions, from London to Madrid or Madrid to Madrid, the upgrade fails.

The upgrade files are downloaded, extracted to tmp, and the agent log will show the ServiceNow Platform Distribution Upgrade. However the logs for that upgrade service will show it stopped before finishing, leaving the MID Server in a Down state.

TERM trapped. Shutting down. 

Steps to Reproduce

  1. Install a Madrid MID Server using an older zip file than the instance version, or Pin an existing MID Server to a different version in order to force an upgrade
  2. Install it on the Linux machine as daemon service
    <agent>/bin/sh mid.sh install
  3. Start MID server
    service mid start
  4. Let the MID Server to auto upgrade to match to instance (or Pinned) version
  5. Observe in agent0.log.0 that it finished downloading zip file, extract zip content to temp file and observe last message
    11/05/18 17:47:56 (660) MIDServer MID Server stopping
    11/05/18 17:47:56 (801) MIDServer MID Server stopped
  6. Observe the wrapper.log and last message
    2018/11/05 17:49:55 | WrapperStartStopApp Debug: stopping. Waiting for 1 threads to complete.
    2018/11/05 17:49:55 | Shutdown failed: Timed out waiting for signal from JVM.
    2018/11/05 17:49:56 | JVM did not exit on request, termination requested.
    2018/11/05 17:49:56 | Signal trapped. Details:
    2018/11/05 17:49:56 | signal number=17 (SIGCHLD), source="unknown"
    2018/11/05 17:49:56 | Received SIGCHLD, checking JVM process status.
    2018/11/05 17:49:56 | JVM received a signal SIGKILL (9).
    2018/11/05 17:49:56 | socket read no code (closed?).
    2018/11/05 17:49:56 | Closing backend socket.
    2018/11/05 17:49:56 | JVM process is gone.
    2018/11/05 17:49:56 | JVM process exited with a code of 1, setting the Wrapper exit code to 1.
    2018/11/05 17:49:56 | JVM exited after being requested to terminate.
    2018/11/05 17:49:56 | <-- Wrapper Stopped
  7. Observe the glide-dist-upgrade.log
    INFO | jvm 1 | 2018/11/05 17:49:56.214 | Nov 05, 2018 5:49:56 PM com.snc.dist.mid_upgrade.UpgradeMain missingOrCanDelete
    INFO | jvm 1 | 2018/11/05 17:49:56.214 | INFO: /root/hm/linux_agent2/bin/wrapper-linux-x86-64 cannot be deleted: /root/hm/linux_agent2/bin/wrapper-linux-x86-64 (Text file busy)
    INFO | jvm 1 | 2018/11/05 17:49:56.214 | Retrying in 1000ms...
    STATUS | wrapper | 2018/11/05 17:49:56.657 | TERM trapped. Shutting down.
    INFO | jvm 1 | 2018/11/05 17:49:56.757 | Nov 05, 2018 5:49:56 PM com.snc.dist.mid_upgrade.UpgradeMain$1 stop
    INFO | jvm 1 | 2018/11/05 17:49:56.757 | INFO: WrapperListener.stop()
    INFO | jvm 1 | 2018/11/05 17:49:57.158 | Nov 05, 2018 5:49:57 PM com.snc.dist.mid_upgrade.UpgradeMain missingOrCanDelete
    INFO | jvm 1 | 2018/11/05 17:49:57.158 | INFO: /root/hm/linux_agent2/bin/wrapper-linux-x86-64 is no longer locked after 125 checks.
    INFO | jvm 1 | 2018/11/05 17:49:57.158 | Nov 05, 2018 5:49:57 PM com.snc.dist.mid_upgrade.UpgradeMarker markService
    INFO | jvm 1 | 2018/11/05 17:49:57.158 | INFO: Added service `snc-platform-dist-upgrade--tmp-1541468756837-0-upgrade-wrapper` to upgrade marker file.
    INFO | jvm 1 | 2018/11/05 17:49:57.358 | Nov 05, 2018 5:49:57 PM com.snc.dist.mid_upgrade.UpgradeMain wipeDirs
    INFO | jvm 1 | 2018/11/05 17:49:57.358 | INFO: ** Debug Stale files ** TargetDirectory Path = /root/hm/linux_agent2
    INFO | jvm 1 | 2018/11/05 17:49:57.358 | Nov 05, 2018 5:49:57 PM com.snc.dist.mid_upgrade.UpgradeMain wipeDirs
    INFO | jvm 1 | 2018/11/05 17:49:57.359 | INFO: ** Debug Stale files ** Files in TargetDirectory = wrapper-tsims-addendum-1.3.txt extlib work bin scripts stop.sh installer.bat ServiceNow Open Source Disclosure 062818LMS0.pdf conf lib jre config.xml EULA.pdf .DS_Store properties installer.sh midinstaller start.sh wrapper-development-license-1.3.txt etc README.doc keystore stop.bat package start.bat logs
    INFO | jvm 1 | 2018/11/05 17:49:57.417 | Nov 05, 2018 5:49:57 PM com.snc.dist.mid_upgrade.UpgradeMain wipeDirs
    INFO | jvm 1 | 2018/11/05 17:49:57.417 | INFO: ** Debug Stale files ** Files from Matching pattern = ServiceNow Open Source Disclosure 062818LMS0.pdf EULA.pdf README.doc
    INFO | jvm 1 | 2018/11/05 17:49:57.417 | Nov 05, 2018 5:49:57 PM com.snc.dist.mid_upgrade.UpgradeMain migrateToTarget
    INFO | jvm 1 | 2018/11/05 17:49:57.417 | INFO: Copying files to MID server installation path.
    STATUS | wrapper | 2018/11/05 17:49:57.618 | <-- Wrapper Stopped

The message "TERM trapped. Shutting down" indicate upgrade did not finished and wrapper service shutdown unexpectedly.

Workaround

This problem has been fixed in New York, which will include the Tanuki wrapper version to 3.5.36 to resolve the issue.

There is no workaround to prevent this happening, however once the problem has been detected, you can get the upgrade to finish by manually running the upgrade-wrapper/bin/glide-dist-upgrade.sh script.

  1. Look in the agent log to find the tmp folder. e.g.
    05/18/19 10:32:07 (333) AutoUpgrade.3600 Added marker `/tmp/1558168258322-0` to upgrade marker file.
  2. Within that folder you will find the upgrade-wrapper/bin/glide-dist-upgrade.sh script
  3. Edit that script to turn on "wrapper.ignore_signals"
    # If uncommented, causes the Wrapper to be shutdown using an anchor file.
    # When launched with the 'start' command, it will also ignore all INT and
    # TERM signals.
    IGNORE_SIGNALS=true
  4. Run it as sudo
    sudo ./glide-dist-upgrade.sh start
  5. The following line will be noted on the "glide-dist-upgrade.log", but this doesn't prevent the upgrade completing because we ignore it.
    TERM trapped, but ignored.

If that is not possible, then a manual MID Server upgrade would be required.

After upgrade we need to re-install the service in NY. 


Related Problem: PRB1312206

Seen In

SR - IRM - Audit Management - New York 2019 Q3
SR - IRM - GRC Profiles - Madrid 2019 Q2
SR - IRM - GRC Workbench - New York 2019 Q3
SR - IRM - Policy and Compliance - Madrid 2019 Q2
SR - IRM - Risk Management - New York 2019 Q3
SR - ITOM - CMDB CI Class Models - 201907
SR - ITOM - Discovery and Service Mapping - v1.0.35
SR - Security - Integration Framework - Madrid 2019 Q2
SR - Security - Support Common - Madrid 2019 Q2
SR - Security - Support Orchestration - Madrid 2019 Q2
SR - SIR - Security Incident Response - Madrid 2019 Q2
SR - SIR - Security Incident Response UI Patch - London 2019 Q2 v.6.2.3
SR - SIR - Store SecOps Setup Assistant - Madrid 2019 Q2
SR - SIR - Store Threat Core - Madrid 2019 Q2
SR - SIR - Store Trusted Security Circles Client - New York 2019 Q3
SR - SIR - Threat intelligence - New York 2019 Q3
SR - VR - Qualys - New York 2019 Q3
SR - VR - Rapid7 - London 2019 Q2 v.6.2.1
SR - VR - Shodan Exploit - New York 2019 Q3
SR - VR - Vulnerability Response - New York 2019 Q3
SR - VR - Vulnerability Response PA Content - Madrid 2019 Q2

Associated Community Threads

There is no data to report.

Article Information

Last Updated:2020-10-05 15:41:18
Published:2020-10-05