During an instance upgrade, that may be many hours long, the instance system scheduler is paused for all but upgrade-safe jobs. This means the "MID Server Monitor" sys_trigger scheduled job, which launches MID Server Heartbeat Probes every 5 minutes, doesn't run during upgrades. It won't detect any MID Servers going Down MID during the upgrade.
If a MID Server goes down for any reason during the instance upgrade, the MID Server record will remain in Up status, and no event will be triggered, until the instance upgrade has finished and the scheduler is un-paused.
As the MID Server is still marked as Up, new jobs will still be given to the down mid server, and will be stuck in the queue and not process. Only after the upgrade finishes, and the MID Server is marked down, will these be reassigned to another MID Server if the MID Server is part of a fail-over cluster (by the Fail over MID server script action triggered by the mid_server.down event created by the scheduled job).
Steps to Reproduce
- Upgrade an instance.
- Check the ecc_queue table, and see there are no outputs created with topic=HeartbeatProbe during the upgrade
This problem is currently under review. You can contact ServiceNow Technical Support or subscribe to this Known Error article by clicking the Subscribe button at the top right of this form to be notified when more information will become available.
A workaround is to set the "MID Server Monitor" sys_trigger record to Upgrade Safe=True.
Related Problem: PRB1382208