Notifications

23 views

Description

MID Server service starts regardless of whether a failed Upgrade Service still remains, leading to deletion of the upgrade service, and potentially leaving the MID Server running in a semi-upgraded and corrupt state

If a MID Server upgrade fails during the execution of the ServiceNow Platform Distribution Upgrade service, then the MID Server is not started. We deliberately don't start it when errors occur and the upgrade does not finish, because we know the upgrade is probably incomplete, with old files that may not have been deleted, or new files that have not been copied yet to the mid server folder, which may be to replace missing files that have already been deleted.

We should not be allowing the main MID Server service to be manually started or automatically started by a server restart, if the incomplete upgrade service remains. Or at least create an Issue record so the custom is aware they may be running a MID Server that is now corrupt and not to be trusted.

If the main MID Server service is started in this situation:

  • the incomplete service is deleted
  • the remaining upgrade files that should have been copied to the mid sever folder are deleted
  • the upgrade service files, are deleted from the tmp folder
  • the MID Server then tries to start, and tries to run jobs, which may fail badly due to the installation now being corrupt/incomplete

This prevents workarounds to continue and finish the upgrade being used:
KB0779816 How to continue a MID Server upgrade after it has crashed in the middle of the ServiceNow Platform Distribution Upgrade service, leaving the MID Server Down and the Service not running

The MID Server is likely to now need a full manual upgrade or re-install.

Steps to Reproduce

1/ Encounter a problem like PRB1279578 during a MID Server upgrade
The upgrade service will stop, and the main service will not be started.

You will now have 2 Services listed in the windows services (services.msc):
ServiceNow MID Server_<MID Server Name>
ServiceNow Platform Distribution Upgrade (<MID Server Name>)
(and maybe "ServiceNow WMI Collector" prior to Orlando)

You will also have files in the temp folder e.g.
C:\WINDOWS\TEMP\1578473047528-0\agent\ - files that should have been copied to the mid server install folder
C:\WINDOWS\TEMP\1578473047528-0\upgrade-wrapper\

2/ Start the "ServiceNow MID Server_<MID Server Name>" service.

Agent logs will show something like:
01/08/20 09:45:47 (174) StartupSequencer Deleted path from upgrade marker, `C:\WINDOWS\TEMP\1578473047528-0`.
01/08/20 09:45:47 (332) StartupSequencer Uninstalled previous upgrader Windows service `snc-platform-dist-upgrade-empdpiper-laptop1`.

and the MID Server will then try to continue running. Other errors due to the corrupted install may then be seen, plus other symptoms for the execution of particular probes.

Workaround

This problem is currently under review. You can contact ServiceNow Technical Support or subscribe to this Known Error article by clicking the Subscribe button at the top right of this form to be notified when more information will become available.

If you see from your instance that an upgrade has failed, or that a MID Server goes down around the time of an instance upgrade, then please avoid starting any services again before you have had a chance to investigate the MID Server logs and configuration.

Installing a new clean known good MID Server is a much safer approach if you need a MID Server up again immediately to replace the down one.


Related Problem: PRB1381000

Seen In

There is no data to report.

Associated Community Threads

There is no data to report.

Article Information

Last Updated:2020-03-03 22:02:36
Published:2020-01-20