Troubleshooting | Table of Contents

  1. Was the instance recently upgraded?
  2. Is the MID Server still up?
  3. Can you read the agent log?
  4. Cannot download upgrade files from instance
  5. Cannot delete or add files on the local host file system
  6. User cannot authenticate
  7. File not found | File access denied

Troubleshooting | Symptoms

  • MID Server status is Down in the MID Server list
  • Discovery scans get stuck
  • MID Server does not keep running
  • MID Server status is Up but is not responding

Troubleshooting | Details

When an instance is upgraded, any MID Server pointing to that instance will attempt to auto-upgrade to the same version. The Quarterly Patching Program (QPP), which started in early 2016, forces instances to upgrade to the next patch. The patches are in the same release family (for example, Geneva Patch 1 to Geneva Patch 12) and will not upgrade outside of the release family (for example, Geneva Patch 1 will not auto-upgrade to Helsinki Patch 4). This means that MID Servers attached to instances will be auto-upgrading at least once per quarter. The increased frequency of upgrades might lead to uncommon and unexpected conditions on the machines where the MID Server is installed.

Troubleshooting | Question and Answer

  • Was the instance recently upgraded either manually or with a QPP patch?
    • I don’t know
        • To find out if your instance has been updated recently:
          1. Navigate to System Diagnostics > Upgrade History.
          2. Search the To column for an entry that indicates that the instance was upgraded to a new patch. It will look something like this:
    • Yes, the instance was recently upgraded
        • See the MID Server logs to determine the state of the upgrade.
      • Is the Mid Server up?
        • I don't know
          1. Navigate to MID server > Servers.
          2. Locate the MID Server in the list. It will report Up with a green dot if it is running.
              1. Navigate to MID server > Servers.
              2. Locate the MID Server in the list.
              3. Open the Mid Server record.
              4. In the Related Links form section, select Grab Mid Logs.
              5. The ECC Queue list will display two log commands: one for agent0.log.0 and one for wrapper.log.
              6. When the requests return, open up the ECC Queue record and download the agent log.
          • No, Mid Server is not up
              • In order to assess the state of the MID Server upgrade with the down MID Server:
                1. Log on to the host machine the Mid Server is running on.
                2. In the file system, navigate to the Agent > Logs directory.
                3. Open the agent log.
          • Yes, the mid server is up
          • Can you read the agent log? [Back to top]
            • Yes, I can read the agent log.
              • Open the agent log file and search for the following phrases:
              • "The MID Server was unable to download" [Back to top]
              • "Unable to delete file" [Back to top]
                • ROOT CAUSE: As part of the upgrade process, the MID Server needs to remove and replace certain files. If for some reason one of those files cannot be deleted, the MID Server upgrade fails. Unfortunately, at the time of the Istanbul release, there is no way to recover and proceed with the upgrade. The MID server must be re-installed once the issue with the file system has been resolved.
                • SOLUTION: Resolve local environment issues of the MID Server host and try again to upgrade the MID Server.
              • "User is unable to authenticate" [Back to top]
                • ROOT CAUSE: All MID Servers are authenticated by the user and password that is entered into the config.xml file on the MID Server host machine. The authentication information is stored in the parameter's mid.instance.username and mid.instance.password. The user in this configuration file must also exist on the instance in the System > Users table. The MID user must have, as a minimum, the mid_server role. Failure to authenticate during an upgrade can hang both the MID and the Upgrade Service itself.
                • SOLUTION: Troubleshoot MID Server credentials and try again to upgrade the MID Server.
              • "SEVERE:com.snc.dist.mid_upgrade.UpgradeExecption:"
                "\bin\mid.bat (Access is denied)" [Back to top]
                • ROOT CAUSE: A complete example containing the two searchable phrases provided is: com.snc.dist.mid_upgrade.UpgradeException: D:\ServiceNow\agent\bin\mid.bat (Access is denied). This message can indicate that the host machine the MID Server is on is a Windows Machine with Application Experience disabled. Per Microsoft's security documentation, this service is internal and cannot actually ever be disabled. Setting this to disabled in the Windows environment only stops the service from being called. It does not stop the service from running. The purpose of the Application Experience, generally speaking, is to evaluate incoming compatibility of application updates to existing installed software. Since the running service does not receive the request, the evaluation never happens and the install of the update fails. This is what is happening to the MID Server.
                • SOLUTION: Resolve local environment issues of the  MID Server host and try again to upgrade the MID Server.
            • No, I cannot read the agent log.
              • If you have not been able to gain access to the agent log following this troubleshooting guide, contact your network or system administrator. The MID Server host machine might be powered off or inadvertently off network. Resolve the network or hardware issues and try again to upgrade the MID Server.


      Article Information

      Last Updated:2019-08-02 21:24:52