There are several known problems that cause a MID Server upgrade to stop while in the middle of running the temporary "ServiceNow Platform Distribution Upgrade" service. These are often due to timeouts or exceptions, often to do with files of the main MID Server service still being locked at the time the Upgrade is trying to delete/overwrite those files.
This method allows the upgrade service to have another go at replacing the files and restarting the main service. This should be tried first, before doing anything destructive like a manual upgrade (see KB0713557), or a re-install.
Symptoms where this can be useful:
- A MID Server that is set as Down shortly after an instance upgrade.
- The agent log of the MID Server shows that it shut itself down shortly after launching the Upgrade, and there are no more entries.
- When inspecting the Windows Services of the host running the MID Server, you see nothing is running, and the ServiceNow Platform Distribution Upgrade service remains in the list
The windows services names may look like this (sorry no screenshot yet):
- ServiceNow MID Server_<MID Server Name>
- ServiceNow Platform Distribution Upgrade (<MID Server Name>) <<== This is the one to Start
- ServiceNow WMI Collector
Release or Environment
Potential Causes where this might be useful, and which might be identified from the logs:
- KB0749292 / PRB1314105 During the MID upgrade 2 minutes time-out for deleting old files is not enough for some customers
- KB0715612 / PRB1307275 MID Server auto-upgrade will fail if Windows has 'Application Experience' disabled
- KB0715244 / PRB1279578 MID server auto-upgrade fails while copying files from temp to agent folder on Windows even with ''Application Experience' enabled
- PRB1322816 Mid server restart during an instance upgrade coincident with auto upgrade causes MID Server Down
- PRB1170628 Autoupgrade fails to restart the MID Server successfully in certain conditions (mostly when there are too many running or hung threads) causing the MID to go down
- KB0746874 / PRB1312206 Linux MID server upgrade fails, and MID Server remains Down - Tanuki wrapper service shutdown as 'TERM Trapped'
- Optional: Ideally, copy logs that may help explain the cause - see below.
- Depending on the operating system:
- Windows: From the Services control panel, 'Start' the 'ServiceNow Platform Distribution Upgrade (...)' Service.
- Linux: The temp folder (see below) will contain a script that can be re-run with "sudo ./glide-dist-upgrade.sh start"
- Wait for a few minutes, and see if the MID Server is Up in the instance.
If you are able to, please use the following steps to capture the upgrade service logs, so you can pass them on to ServiceNow Technical Support in a HI Case. The agent/logs/agent0.log.0 and agent/logs/wrapper.log may only show what the main service was doing up to the upgrade only, and not the upgrade itself.
If this doesn't work then Support will absolutely need this additional log from the temp folder:-
- Search the agent log of the mid server for the string "Added marker". You should find a line like this, although the folder will be different.
AutoUpgrade.3600 Added marker `C:\WINDOWS\TEMP\1569035472492-0` to upgrade marker file.
- Open that folder, and then navigate through further sub-folder to upgrade-wrapper\logs\glide-dist-upgrade.log
e.g. C:\WINDOWS\TEMP\<a long number>\upgrade-wrapper\logs\glide-dist-upgrade.log
That upgrade-wrapper.log may be the only chance to explain why the upgrade service crashed, especially if the crash occurred before the log was copied into the main wrapper.log. Attempting to start the main "ServiceNow MID Server_<MID Server Name>" may lead to the temporary folder being deleted, at which point that is lost.
To understand the MID Server upgrade process, the steps involved, how it could in theory go wrong, and why this method might work for you, please refer to:
KB0696937 MID Server upgrade process - What actually happens when a MID Server upgrades itself?