Notifications

31 views

Description

MID Server remains Down after the auto-Upgrade service times out after 10 minutes, while still waiting for agent\lib\accessors-smart.jar file lock to be freed up. The file remains locked by java.exe, which is the main MID Server service, that has not completely shut down yet.

The upgrade service does not finish, and does not start the MID Server again.

This problem applies only to Madrid and later MID Servers where the timeout is 10m, rather than 2m in earlier versions.

Steps to Reproduce

Specific real-world conditions required to reproduce unknown at this time. This can be forced by deliberately locking a file in lib.

glide-dist-upgrade.log within the temp folder shows:

ERROR | wrapper | 2019/11/04 10:48:22.961 | The ServiceNow Platform Distribution Upgrade (<MID Server Name>) service is not installed - The specified service does not exist as an installed service. (0x424)
STATUS | wrapper | 2019/11/04 10:48:25.195 | ServiceNow Platform Distribution Upgrade (<MID Server name>) service installed.
STATUS | wrapper | 2019/11/04 10:48:32.071 | Starting the ServiceNow Platform Distribution Upgrade (<MID Server name>) service...
STATUS | wrapper | 2019/11/04 10:48:32.102 | --> Wrapper Started as Service
STATUS | wrapper | 2019/11/04 10:48:32.134 | Java Service Wrapper Standard Edition 64-bit 3.5.17
STATUS | wrapper | 2019/11/04 10:48:32.149 | Copyright (C) 1999-2012 Tanuki Software, Ltd. All Rights Reserved.
STATUS | wrapper | 2019/11/04 10:48:32.165 | http://wrapper.tanukisoftware.com
STATUS | wrapper | 2019/11/04 10:48:32.180 | Licensed to ServiceNow, Inc. for MID Upgrade
STATUS | wrapper | 2019/11/04 10:48:32.215 |
STATUS | wrapper | 2019/11/04 10:48:32.587 | Launching a JVM...
INFO | jvm 1 | 2019/11/04 10:48:32.821 | WrapperManager: Initializing...
INFO | jvm 1 | 2019/11/04 10:48:33.055 | nov 04, 2019 10:48:32 AM com.snc.dist.mid_upgrade.UpgradeMain$1 start
INFO | jvm 1 | 2019/11/04 10:48:33.087 | INFO: WrapperListener.start()
INFO | jvm 1 | 2019/11/04 10:48:33.149 | nov 04, 2019 10:48:32 AM com.snc.dist.mid_upgrade.UpgradeMain logArguments
INFO | jvm 1 | 2019/11/04 10:48:33.244 | INFO: Command-line arguments: --dist-file=C:\Windows\Temp\1572860822078-0\mid-upgrade.madrid-12-18-2018__patch7a-10-01-2019_10-03-2019_1530.universal.universal.zip --mode=dist-upgrade --current-dir=<install folder>\agent
INFO | jvm 1 | 2019/11/04 10:48:33.384 | nov 04, 2019 10:48:32 AM com.snc.dist.mid_upgrade.UpgradeMain run
INFO | jvm 1 | 2019/11/04 10:48:33.434 | INFO: MID Server upgrade v2.0 starting...
INFO | jvm 1 | 2019/11/04 10:48:33.446 | nov 04, 2019 10:48:32 AM com.snc.dist.mid_upgrade.UpgradeMain run
INFO | jvm 1 | 2019/11/04 10:48:33.462 | INFO: Upgrading MID `<install folder>\agent` with contents from `C:\Windows\Temp\1572860822078-0\agent`.
INFO | jvm 1 | 2019/11/04 10:48:33.477 | nov 04, 2019 10:48:33 AM com.snc.dist.mid_upgrade.UpgradeMain missingOrCanDelete
INFO | jvm 1 | 2019/11/04 10:48:33.494 | INFO: <install folder>\agent\lib\sigar-amd64-winnt.dll cannot be deleted: <install folder>\agent\lib\sigar-amd64-winnt.dll (The process cannot access the file because it is being used by another process)
INFO | jvm 1 | 2019/11/04 10:48:33.524 | Retrying in 1000ms...
INFO | jvm 1 | 2019/11/04 10:48:34.087 | nov 04, 2019 10:48:34 AM com.snc.dist.mid_upgrade.UpgradeMain missingOrCanDelete
INFO | jvm 1 | 2019/11/04 10:48:34.087 | INFO: <install folder>\agent\lib\sigar-amd64-winnt.dll cannot be deleted: <install folder>\agent\lib\sigar-amd64-winnt.dll (The process cannot access the file because it is being used by another process)
INFO | jvm 1 | 2019/11/04 10:48:34.118 | Retrying in 1000ms...
STATUS | wrapper | 2019/11/04 10:48:35.134 | ServiceNow Platform Distribution Upgrade (<MID Server name>) started.
INFO | jvm 1 | 2019/11/04 10:48:35.118 | nov 04, 2019 10:48:35 AM com.snc.dist.mid_upgrade.UpgradeMain missingOrCanDelete
INFO | jvm 1 | 2019/11/04 10:48:35.149 | INFO: <install folder>\agent\lib\sigar-amd64-winnt.dll cannot be deleted: <install folder>\agent\lib\sigar-amd64-winnt.dll (The process cannot access the file because it is being used by another process)
...
INFO | jvm 1 | 2019/11/04 10:48:47.213 | nov 04, 2019 10:48:47 AM com.snc.dist.mid_upgrade.UpgradeMain missingOrCanDelete
INFO | jvm 1 | 2019/11/04 10:48:47.213 | INFO: <install folder>\agent\lib\sigar-amd64-winnt.dll cannot be deleted: <install folder>\agent\lib\sigar-amd64-winnt.dll (The process cannot access the file because it is being used by another process)
INFO | jvm 1 | 2019/11/04 10:48:47.244 | Retrying in 1000ms...
INFO | jvm 1 | 2019/11/04 10:48:48.229 | nov 04, 2019 10:48:48 AM com.snc.dist.mid_upgrade.UpgradeMain missingOrCanDelete
INFO | jvm 1 | 2019/11/04 10:48:48.244 | INFO: <install folder>\agent\lib\sigar-amd64-winnt.dll is no longer locked after 15 checks.
INFO | jvm 1 | 2019/11/04 10:48:48.244 | nov 04, 2019 10:48:48 AM com.snc.dist.mid_upgrade.UpgradeMarker markService
INFO | jvm 1 | 2019/11/04 10:48:48.260 | INFO: Added service `snc-platform-dist-upgrade-<MID Server name>` to upgrade marker file.
INFO | jvm 1 | 2019/11/04 10:58:51.863 | nov 04, 2019 10:58:51 AM com.snc.dist.mid_upgrade.UpgradeMain timedDelete
INFO | jvm 1 | 2019/11/04 10:58:51.863 | SEVERE: Timeout deleting <install folder>\agent\lib\accessors-smart.jar
INFO | jvm 1 | 2019/11/04 10:58:51.879 | nov 04, 2019 10:58:51 AM com.snc.dist.mid_upgrade.UpgradeMain run
INFO | jvm 1 | 2019/11/04 10:58:51.879 | SEVERE: java.io.IOException: Unable to delete `<install folder>\agent\lib\accessors-smart.jar`, MID Server upgrade cannot continue. If the MID Server will not restart you may need to reinstall the MID server.
INFO | jvm 1 | 2019/11/04 10:58:51.894 | java.io.IOException: Unable to delete `<install folder>\agent\lib\accessors-smart.jar`, MID Server upgrade cannot continue. If the MID Server will not restart you may need to reinstall the MID server.
INFO | jvm 1 | 2019/11/04 10:58:51.925 | at com.snc.dist.mid_upgrade.UpgradeMain.wipeDirectory(UpgradeMain.java:450)
INFO | jvm 1 | 2019/11/04 10:58:51.925 | at com.snc.dist.mid_upgrade.UpgradeMain.wipeDirs(UpgradeMain.java:398)
INFO | jvm 1 | 2019/11/04 10:58:51.941 | at com.snc.dist.mid_upgrade.UpgradeMain.run(UpgradeMain.java:309)
INFO | jvm 1 | 2019/11/04 10:58:51.941 | at java.lang.Thread.run(Thread.java:748)
INFO | jvm 1 | 2019/11/04 10:58:51.957 |
INFO | jvm 1 | 2019/11/04 10:58:51.957 | nov 04, 2019 10:58:51 AM com.snc.dist.mid_upgrade.UpgradeMain appendMidLogs
INFO | jvm 1 | 2019/11/04 10:58:51.972 | INFO: Flushing logs

Workaround

This problem is currently under review. You can contact ServiceNow Technical Support or subscribe to this Known Error article by clicking the Subscribe button at the top right of this form to be notified when more information will become available..

To get the MID Server working again after this has happened:

Although the error suggests "If the MID Server will not restart you may need to reinstall the MID server", that is usually not the best approach. Don't try to start the main MID Server service. Instead try this, and the upgrade will probably successfully complete:

  1. Check the main MID Server service is now stopped, and kill the java.exe process if necessary.
  2. Follow the steps in KB0779816 How to continue a MID Server upgrade after it has crashed in the middle of the ServiceNow Platform Distribution Upgrade service, leaving the MID Server Down and the Service not running

There is no workaround to avoid this issue happening at present.


Related Problem: PRB1373214

Seen In

There is no data to report.

Associated Community Threads

There is no data to report.

Article Information

Last Updated:2019-11-11 01:15:05
Published:2019-11-11