Notifications

97 views

Description

When upgrading a MID server hosted on a CentOS Linux server, and other Linux distributions, from London to Madrid or Madrid to Madrid, the upgrade fails.

The upgrade files are downloaded, extracted to tmp, and the agent log will show the ServiceNow Platform Distribution Upgrade. However the logs for that upgrade service will show it stopped before finishing, leaving the MID Server in a Down state.

TERM trapped. Shutting down. 

Steps to Reproduce

  1. Install a Madrid MID Server using an older zip file than the instance version, or Pin an existing MID Server to a different version in order to force an upgrade
  2. Install it on the Linux machine as daemon service
    <agent>/bin/sh mid.sh install
  3. Start MID server
    service mid start
  4. Let the MID Server to auto upgrade to match to instance (or Pinned) version
  5. Observe in agent0.log.0 that it finished downloading zip file, extract zip content to temp file and observe last message
    11/05/18 17:47:56 (660) MIDServer MID Server stopping
    11/05/18 17:47:56 (801) MIDServer MID Server stopped
  6. Observe the wrapper.log and last message
    2018/11/05 17:49:55 | WrapperStartStopApp Debug: stopping. Waiting for 1 threads to complete.
    2018/11/05 17:49:55 | Shutdown failed: Timed out waiting for signal from JVM.
    2018/11/05 17:49:56 | JVM did not exit on request, termination requested.
    2018/11/05 17:49:56 | Signal trapped. Details:
    2018/11/05 17:49:56 | signal number=17 (SIGCHLD), source="unknown"
    2018/11/05 17:49:56 | Received SIGCHLD, checking JVM process status.
    2018/11/05 17:49:56 | JVM received a signal SIGKILL (9).
    2018/11/05 17:49:56 | socket read no code (closed?).
    2018/11/05 17:49:56 | Closing backend socket.
    2018/11/05 17:49:56 | JVM process is gone.
    2018/11/05 17:49:56 | JVM process exited with a code of 1, setting the Wrapper exit code to 1.
    2018/11/05 17:49:56 | JVM exited after being requested to terminate.
    2018/11/05 17:49:56 | <-- Wrapper Stopped
  7. Observe the glide-dist-upgrade.log
    INFO | jvm 1 | 2018/11/05 17:49:56.214 | Nov 05, 2018 5:49:56 PM com.snc.dist.mid_upgrade.UpgradeMain missingOrCanDelete
    INFO | jvm 1 | 2018/11/05 17:49:56.214 | INFO: /root/hm/linux_agent2/bin/wrapper-linux-x86-64 cannot be deleted: /root/hm/linux_agent2/bin/wrapper-linux-x86-64 (Text file busy)
    INFO | jvm 1 | 2018/11/05 17:49:56.214 | Retrying in 1000ms...
    STATUS | wrapper | 2018/11/05 17:49:56.657 | TERM trapped. Shutting down.
    INFO | jvm 1 | 2018/11/05 17:49:56.757 | Nov 05, 2018 5:49:56 PM com.snc.dist.mid_upgrade.UpgradeMain$1 stop
    INFO | jvm 1 | 2018/11/05 17:49:56.757 | INFO: WrapperListener.stop()
    INFO | jvm 1 | 2018/11/05 17:49:57.158 | Nov 05, 2018 5:49:57 PM com.snc.dist.mid_upgrade.UpgradeMain missingOrCanDelete
    INFO | jvm 1 | 2018/11/05 17:49:57.158 | INFO: /root/hm/linux_agent2/bin/wrapper-linux-x86-64 is no longer locked after 125 checks.
    INFO | jvm 1 | 2018/11/05 17:49:57.158 | Nov 05, 2018 5:49:57 PM com.snc.dist.mid_upgrade.UpgradeMarker markService
    INFO | jvm 1 | 2018/11/05 17:49:57.158 | INFO: Added service `snc-platform-dist-upgrade--tmp-1541468756837-0-upgrade-wrapper` to upgrade marker file.
    INFO | jvm 1 | 2018/11/05 17:49:57.358 | Nov 05, 2018 5:49:57 PM com.snc.dist.mid_upgrade.UpgradeMain wipeDirs
    INFO | jvm 1 | 2018/11/05 17:49:57.358 | INFO: ** Debug Stale files ** TargetDirectory Path = /root/hm/linux_agent2
    INFO | jvm 1 | 2018/11/05 17:49:57.358 | Nov 05, 2018 5:49:57 PM com.snc.dist.mid_upgrade.UpgradeMain wipeDirs
    INFO | jvm 1 | 2018/11/05 17:49:57.359 | INFO: ** Debug Stale files ** Files in TargetDirectory = wrapper-tsims-addendum-1.3.txt extlib work bin scripts stop.sh installer.bat ServiceNow Open Source Disclosure 062818LMS0.pdf conf lib jre config.xml EULA.pdf .DS_Store properties installer.sh midinstaller start.sh wrapper-development-license-1.3.txt etc README.doc keystore stop.bat package start.bat logs
    INFO | jvm 1 | 2018/11/05 17:49:57.417 | Nov 05, 2018 5:49:57 PM com.snc.dist.mid_upgrade.UpgradeMain wipeDirs
    INFO | jvm 1 | 2018/11/05 17:49:57.417 | INFO: ** Debug Stale files ** Files from Matching pattern = ServiceNow Open Source Disclosure 062818LMS0.pdf EULA.pdf README.doc
    INFO | jvm 1 | 2018/11/05 17:49:57.417 | Nov 05, 2018 5:49:57 PM com.snc.dist.mid_upgrade.UpgradeMain migrateToTarget
    INFO | jvm 1 | 2018/11/05 17:49:57.417 | INFO: Copying files to MID server installation path.
    STATUS | wrapper | 2018/11/05 17:49:57.618 | <-- Wrapper Stopped

The message "TERM trapped. Shutting down" indicate upgrade did not finished and wrapper service shutdown unexpectedly.

Workaround

This problem has been fixed in New York, which will include the Tanuki wrapper version to 3.5.36 to resolve the issue.

There is no workaround to prevent this happening, however once the problem has been detected, you can get the upgrade to finish by manually running the upgrade-wrapper/bin/glide-dist-upgrade.sh script.

  1. Look in the agent log to find the tmp folder. e.g.
    05/18/19 10:32:07 (333) AutoUpgrade.3600 Added marker `/tmp/1558168258322-0` to upgrade marker file.
  2. Within that folder you will find the upgrade-wrapper/bin/glide-dist-upgrade.sh script
  3. Edit that script to turn on "wrapper.ignore_signals"
    # If uncommented, causes the Wrapper to be shutdown using an anchor file.
    # When launched with the 'start' command, it will also ignore all INT and
    # TERM signals.
    IGNORE_SIGNALS=true
  4. Run it as sudo
    sudo ./glide-dist-upgrade.sh start
  5. The following line will be noted on the "glide-dist-upgrade.log", but this doesn't prevent the upgrade completing because we ignore it.
    TERM trapped, but ignored.

If that is not possible, then a manual MID Server upgrade would be required.


Related Problem: PRB1312206

Seen In

There is no data to report.

Intended Fix Version

Madrid Patch 5

Safe Harbor Statement

This "Intended Fix Version" information is meant to outline ServiceNow's general product direction and should not be relied upon in making a purchasing decision. The information provided here is for information purposes only and may not be incorporated into any contract. It is not a commitment, promise, or legal obligation to deliver any material, code, or functionality. The development, release, and timing of any features or functionality described for our products remains at ServiceNow's sole discretion.

Associated Community Threads

There is no data to report.

Article Information

Last Updated:2019-06-17 14:14:42
Published:2019-05-23