Description
MID Server upgrade fails leaving MID Server Down, due to Cisco Advanced Malware Protection (AMP) for Endpoints preventing the upgrade service deleting the Wrapper executable.
The Upgrade starts, the MID Server launches the temporary upgrade service and shuts itself down, and then due to the wrapper-windows-x86-64.exe file being locked/blocked by Cisco CMP, the upgrade service has a FileNotFoundException and stops, leaving the MID Server down.
The MID Server wrapper.log will show this at the end (assuming no manual attempt was made to start it since):
May 12, 2020 2:57:33 PM com.snc.dist.mid_upgrade.UpgradeMain run
SEVERE: com.snc.dist.mid_upgrade.UpgradeException: java.io.FileNotFoundException: C:\ServiceNow\agent\bin\wrapper-windows-x86-64.exe (Access is denied)
com.snc.dist.mid_upgrade.UpgradeException: java.io.FileNotFoundException: C:\ServiceNow\agent\bin\wrapper-windows-x86-64.exe (Access is denied)
at com.snc.dist.mid_upgrade.UpgradeMain.migrateToTarget(UpgradeMain.java:840)
at com.snc.dist.mid_upgrade.UpgradeMain.run(UpgradeMain.java:313)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.FileNotFoundException: C:\ServiceNow\agent\bin\wrapper-windows-x86-64.exe (Access is denied)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.(FileOutputStream.java:213)
at java.io.FileOutputStream.(FileOutputStream.java:162)
at org.apache.commons.io.FileUtils.doCopyFile(FileUtils.java:1142)
at org.apache.commons.io.FileUtils.doCopyDirectory(FileUtils.java:1446)
at org.apache.commons.io.FileUtils.doCopyDirectory(FileUtils.java:1444)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:1388)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:1317)
at com.snc.dist.mid_upgrade.UpgradeMain.migrateToTarget(UpgradeMain.java:837)
... 2 more
May 12, 2020 2:57:33 PM com.snc.dist.mid_upgrade.UpgradeMain appendMidLogs
INFO: Flushing logs
<< UPGRADE LOG END >>
Note: This PRB is specific to MID Server outages caused by Cisco CMP during upgrades, but this symptom is not always caused by Cisco AMP. The same symptom has also been seem on server not running CMP.
Steps to Reproduce
- Install a MID Server on a Windows host running Cisco Advanced Malware Protection (AMP)
- Cause the MID Server to upgrade
- Some upgrades will fail to upgrade at the point that the old agent\bin\wrapper-windows-x86-64.exe is deleted
Workaround
It may be possible to get the Upgrade to finish cleanly using this process:
KB0779816 How to continue a MID Server upgrade after it has crashed in the middle of the ServiceNow Platform Distribution Upgrade service, leaving the MID Server Down and the Service not running
To prevent the issue re-occurring, exclusions will need adding to Cisco CMP. Details TBC.
Related Problem: PRB1408516