When a MID server does an auto upgrade, the temporary upgrade service can fail to copy wrapper-windows-x86-64.exe, activation.jar or other files from the temp to agent folder. The UPGRADE LOG section of the wrapper log is output from the temporary upgrade service, and suggests something - e.g. Windows and/or Antivirus software - is holding on to files too long, or has them locked.
Important Note: Please discount the following problems first, before considering linking PRB1279578 to an issue:
- Check that only one Windows Service is configured to use this MID Server's install folder. The'other' service could be running and locking the files. The locked file will usually be "mid.bat" in that situation. See:
KB0743123 / PRB1330396 MID Server start.bat fails to check if a Windows Service already exists for the installation folder before creating another service
KB0743043 How to debug and resolve the "A MID Server with a duplicate name or sys_id was prevented from connecting." Issue
- Check that 'Application Experience' is Enabled. This is a requirement for any MID Server installation on Windows. See:
PRB1307275/KB0715612 MID Server auto-upgrade will fail if Windows has 'Application Experience' disabled
- This problem occurs before the overall timeout for the file copies. That timeout is 10 minutes since Madrid, and 2 Minutes before Madrid. If the error occurs at or close to 10m from the upgrade service starting, and mentions "timeout" in the errors, then the issue is likely to be the following or similar:
KB0784943/PRB1373214 MID Server Down after auto-Upgrade service 10 minute file lock timeout: SEVERE: java.io.IOException: Unable to delete agent\lib\accessors-smart.jar
Steps to Reproduce
Steps to reproduce this issue on demand in a controlled test environment are currently not known.
- The issue occurs intermittently. The logs above has been observed on Windows Server 2012 R2, with McAfee antivirus running.
- Configure the MID server to auto upgrade.
- Check the 'Application Experience' is enabled for the MID server auto upgrade to work.
- Check if he MID server service comes back up.
- Inspect the MID agent logs and wrapper logs.
The wrapper.log (or the upgrade-wrapper\logs\glide-dist-upgrade.log file in the temp folder) will include the highlighted lines (although may not be the wrapper-windows-x86-64.exe file).
INFO | jvm 1 | 2018/04/27 17:12:03.035 | WrapperManager: Initializing...
INFO | jvm 1 | 2018/04/27 17:12:03.269 | Apr 27, 2018 5:12:03 PM com.snc.dist.mid_upgrade.UpgradeMain$1 start
INFO | jvm 1 | 2018/04/27 17:12:19.232 | Apr 27, 2018 5:12:19 PM com.snc.dist.mid_upgrade.UpgradeMain run (Note that this is only 16 seconds after starting the Upgrade Service)
INFO | jvm 1 | 2018/04/27 17:12:19.232 | SEVERE: com.snc.dist.mid_upgrade.UpgradeException: java.io.FileNotFoundException: C:\MID_Server\agent\bin\wrapper-windows-x86-64.exe (Access is denied)
INFO | jvm 1 | 2018/04/27 17:12:19.248 | com.snc.dist.mid_upgrade.UpgradeException: java.io.FileNotFoundException: C:\MID_Server\agent\bin\wrapper-windows-x86-64.exe (Access is denied)
This problem is currently under review. You can contact ServiceNow Technical Support or subscribe to this Known Error article by clicking the Subscribe button at the top right of this form to be notified when more information will become available.
Very Important Note: Steps to reproduce this issue on demand in a controlled test environment are currently (June 2020) not fully known so reproducible cases on customer instances are required for ServiceNow to narrow down configurations and environment settings before we can fix this problem.
Cisco CMP antivirus software has been proved to be a cause of this, which is being tracked by PRB1408516.
Record the following information and report it to ServiceNow:
- agent and wrapper logs, plus upgrade-wrapper\logs\glide-dist-upgrade.log
- Windows Version and patch, plus ideally a list of windows updates/hotfixes applied
- List of Services and Running processes at the time of the failed upgrade
- Details of any Anti-virus software and their versions
- Anything else unusual about your environment, such as non-default security policies.
- The number of MID Servers installed on the host, and how many of those were upgrading at the time.
It may be possible to get the Upgrade to finish cleanly using this process:
KB0779816 How to continue a MID Server upgrade after it has crashed in the middle of the ServiceNow Platform Distribution Upgrade service, leaving the MID Server Down and the Service not running
If that fails, then a manual upgrade may be necessary:
KB0713557 How to manually Upgrade and/or Restore a MID server after a failed Upgrade
To try and find out which process is locking the file, you may try this troubleshooting process:
KB0779852 Finding which Process has Locked a File during a MID Server Auto-Upgrade, Using Windows Resource Monitor (resmon.exe)
A workaround to prevent this problem happening in future upgrades is also not know, however please do ensure that all documented MID Server system/environment requirements are met:
- Ensure that Windows 'Application Experience' is enabled on the windows host running MID server and that the MID host was restarted after the change. KB0715612
- Run MID server service with a local or domain account that is a member of the local Administrators group KB0693327
- Ensure that the MID server is not installed inside a user profile such as the desktop of a user.
Related Problem: PRB1279578