Notifications

533 views

Description

MID Server upgrade fails leaving MID Server Down, due to Cisco Advanced Malware Protection (AMP) for Endpoints preventing the upgrade service deleting the Wrapper executable.

The Upgrade starts, the MID Server launches the temporary upgrade service and shuts itself down, and then due to the wrapper-windows-x86-64.exe file being locked/blocked by Cisco CMP, the upgrade service has a FileNotFoundException and stops, leaving the MID Server down.

The MID Server wrapper.log will show this at the end (assuming no manual attempt was made to start it since):

May 12, 2020 2:57:33 PM com.snc.dist.mid_upgrade.UpgradeMain run
SEVERE: com.snc.dist.mid_upgrade.UpgradeException: java.io.FileNotFoundException: C:\ServiceNow\agent\bin\wrapper-windows-x86-64.exe (Access is denied)
com.snc.dist.mid_upgrade.UpgradeException: java.io.FileNotFoundException: C:\ServiceNow\agent\bin\wrapper-windows-x86-64.exe (Access is denied)
at com.snc.dist.mid_upgrade.UpgradeMain.migrateToTarget(UpgradeMain.java:840)
at com.snc.dist.mid_upgrade.UpgradeMain.run(UpgradeMain.java:313)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.FileNotFoundException: C:\ServiceNow\agent\bin\wrapper-windows-x86-64.exe (Access is denied)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.(FileOutputStream.java:213)
at java.io.FileOutputStream.(FileOutputStream.java:162)
at org.apache.commons.io.FileUtils.doCopyFile(FileUtils.java:1142)
at org.apache.commons.io.FileUtils.doCopyDirectory(FileUtils.java:1446)
at org.apache.commons.io.FileUtils.doCopyDirectory(FileUtils.java:1444)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:1388)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:1317)
at com.snc.dist.mid_upgrade.UpgradeMain.migrateToTarget(UpgradeMain.java:837)
... 2 more

May 12, 2020 2:57:33 PM com.snc.dist.mid_upgrade.UpgradeMain appendMidLogs
INFO: Flushing logs
<< UPGRADE LOG END >>

Note: This PRB is specific to MID Server outages caused by Cisco CMP during upgrades, but this symptom is not always caused by Cisco AMP. The same symptom has also been seem on server not running CMP.

Steps to Reproduce

  1. Install a MID Server on a Windows host running Cisco Advanced Malware Protection (AMP)
  2. Cause the MID Server to upgrade
  3. Some upgrades will fail to upgrade at the point that the old agent\bin\wrapper-windows-x86-64.exe is deleted

Workaround

It may be possible to get the Upgrade to finish cleanly using this process:
KB0779816 How to continue a MID Server upgrade after it has crashed in the middle of the ServiceNow Platform Distribution Upgrade service, leaving the MID Server Down and the Service not running

To prevent the issue re-occurring, exclusions will need adding to Cisco CMP. Details TBC.


Related Problem: PRB1408516

Seen In

SR - IRM - Audit Management - New York 2019 Q3
SR - IRM - GRC Profiles - Madrid 2019 Q2
SR - IRM - GRC Workbench - New York 2019 Q3
SR - IRM - Policy and Compliance - Madrid 2019 Q2
SR - IRM - Risk Management - New York 2019 Q3
SR - IRM - Vendor Risk Management - Madrid 2019 Q1
SR - ITOM - CMDB CI Class Models - 201907
SR - ITOM - Discovery and Service Mapping - v1.0.35
SR - ITOM - Fundamentals Istanbul Jakarta Kingston r1 - v5.99.6
SR - Security - Integration Framework - Madrid 2019 Q2
SR - Security - Support Common - Madrid 2019 Q2
SR - Security - Support Orchestration - Madrid 2019 Q2
SR - VR - Rapid7 - London 2019 Q2 v.6.2.1
SR - VR - Vulnerability Response - New York 2019 Q3

Associated Community Threads

There is no data to report.

Article Information

Last Updated:2020-09-22 09:32:47
Published:2020-06-09