When an instance is upgraded, the MID Server needs to upgrade itself to match and needs to Download Upgrade Files from https://install.service-now.com/ and if it does not have access, it fails to upgrade. The automatic Test MID Server connectivity feature will check for and notify you if the MID Server can't.

 Warning: A MID Server on the wrong version can cause code and data mismatches between MID Server and instance, potentially causing the MID Server to fail to process commands sent it by the instance, or the instance to not process data coming back from the MID Server. APIs related to Validation and encryption Keychains may also not match.

Symptom

AutoUpgrade.3600 Performing pre-upgrade validation tests.
AutoUpgrade.3600 Downloading from https://install.service-now.com/glide/distribution/builds/package/mid-upgrade/2019/07/16/mid-upgrade.mid.newyork-06-26-2019__patch0-hotfix2-07-10-2019_07-16-2019_1636.preUpgradeCheck.zip
AutoUpgrade.3600 WARNING *** WARNING *** java.net.SocketTimeoutException: connect timed out when posting to https://install.service-now.com/glide/distribution/builds/package/mid-upgrade/2019/07/16/mid-upgrade.mid.newyork-06-26-2019__patch0-hotfix2-07-10-2019_07-16-2019_1636.preUpgradeCheck.zip
AutoUpgrade.3600 SEVERE *** ERROR *** java.net.SocketTimeoutException: connect timed out when posting to https://install.service-now.com/glide/distribution/builds/package/mid-upgrade/2019/07/16/mid-upgrade.mid.newyork-06-26-2019__patch0-hotfix2-07-10-2019_07-16-2019_1636.preUpgradeCheck.zip
AutoUpgrade.3600 Downloading from http://install.service-now.com/glide/distribution/builds/package/mid-upgrade/2019/07/16/mid-upgrade.mid.newyork-06-26-2019__patch0-hotfix2-07-10-2019_07-16-2019_1636.preUpgradeCheck.zip
AutoUpgrade.3600 WARNING *** WARNING *** org.apache.commons.httpclient.ConnectTimeoutException: The host did not accept the connection within timeout of 10000 ms when posting to http://install.service-now.com/glide/distribution/builds/package/mid-upgrade/2019/07/16/mid-upgrade.mid.newyork-06-26-2019__patch0-hotfix2-07-10-2019_07-16-2019_1636.preUpgradeCheck.zip
AutoUpgrade.3600 SEVERE *** ERROR *** org.apache.commons.httpclient.ConnectTimeoutException: The host did not accept the connection within timeout of 10000 ms when posting to http://install.service-now.com/glide/distribution/builds/package/mid-upgrade/2019/07/16/mid-upgrade.mid.newyork-06-26-2019__patch0-hotfix2-07-10-2019_07-16-2019_1636.preUpgradeCheck.zip
AutoUpgrade.3600 SEVERE *** ERROR *** Aborting MID Server upgrade due to pre-upgrade check failure: Unable to download updates from install server
AutoUpgrade.3600 Setting mid status to Upgrade Failed
AutoUpgrade.3600 Instance.updateAgentRecordState(), OperationalState=UPGRADE_FAILED

The MID Server form and Issues table will also repeat the error:

Aborting MID Server upgrade due to pre-upgrade check failure: Unable to download updates from install server

All versions since Geneva, with Windows or Linux MID Servers.

The fix for PRB1387272 means ZIP files are now signed and have a different URL since Paris, Orlando Patch 3, New York Patch 9, and Madrid Patch 10 Hot Fix 1b.

In the following situations, your MID Server computer has no access to the install server and cannot auto-upgrade itself. You should try to resolve these configuration issues internally first, as these are our documented connectivity requirements:

• The instance is on-premise and installed inside the customer network (with no Internet access), and the MID Server also has no internet access at all.
• The instance is hosted in our datacenter, but although the MID Server does have access to the instance, you have not yet arranged for it to have access to our upgrade server: https://install.service-now.com/

This manual procedure fools the MID Server into thinking it has downloaded the files itself already, allowing it to upgrade itself in the normal way, and if necessary must be done for every MID Server after every upgrade or patch of every instance, without fail. 

1) Find the filenames from the Agent log:

On the MID Server computer, check the latest 'AutoUpgrade' or 'StartupSequencer' thread entries in the Agent Log for the "Missing:" ZIP file names:
<install folder>\agent\logs\agent0.log.0

AutoUpgrade.3600 Current packages:
AutoUpgrade.3600 Installed: [mid-core.kingston-10-17-2017__patch0-11-06-2017_11-11-2017_1422.universal.universal.zip, mid-jre.kingston-10-17-2017__patch0-11-06-2017_11-11-2017_1422.windows.x86-64.zip]
AutoUpgrade.3600 Assigned: [mid-upgrade.newyork-06-26-2019__patch0-hotfix2-07-10-2019_07-16-2019_1636.universal.universal.zip, mid-core.newyork-06-26-2019__patch0-hotfix2-07-10-2019_07-16-2019_1636.universal.universal.zip, mid-jre.newyork-06-26-2019__patch0-hotfix2-07-10-2019_07-16-2019_1636 .windows.x86-64.zip]
AutoUpgrade.3600 Missing: [mid-upgrade.newyork-06-26-2019__patch0-hotfix2-07-10-2019_07-16-2019_1636.universal.universal.zip, mid-core.newyork-06-26-2019__patch0-hotfix2-07-10-2019_07-16-2019_1636.universal.universal.zip, mid-jre.newyork-06-26-2019__patch0-hotfix2-07-10-2019_07-16-2019_1636 .windows.x86-64.zip]
AutoUpgrade.3600 Downloaded: []

In that example, the MID Server is still Kingston Patch 0, even though the instance is already upgraded to New York Patch 0 Hotfix 2, and the three missing files in this particular example are:

• mid-upgrade.newyork-06-26-2019__patch0-hotfix2-07-10-2019_07-16-2019_1636.universal.universal.zip
• mid-jre.newyork-06-26-2019__patch0-hotfix2-07-10-2019_07-16-2019_1636.windows.x86-64.zip
• mid-core.newyork-06-26-2019__patch0-hotfix2-07-10-2019_07-16-2019_1636.universal.universal.zip

Note: The file names and URLs you need will be specific to a particular version of the platform, that is running in your specific instance. 

2) Next figure out the full URL for those files.

The attached Excel Spreadsheet provides the links to the files for a specific version. 

You can find the current MID Buildstamp of your instance, which is the version the MID Server should upgrade to, in the Stats page:
https://<instance_name>.service-now.com/stats.do

Paste it into the attached spreadsheet, and find the URLs for the files identified above. The filenames are very similar, so be careful to select the correct file (e.g. mid-upgrade) and architecture (e.g. windows.x64):

Note: If you are running a version earlier than Paris, Orlando Patch 3, New York Patch 9, and Madrid Patch 10 Hot Fix 1b, then you will need to use the Unsigned ZIP Files lower down the spreadsheet. Otherwise use the Signed ones.

3) Download these files manually on another computer that has internet access and then copy those ZIP files to the MID Server folder:

<install folder>\agent\package\incoming

4) Prevent the Pre-Check being run

If you see the following error in the agent log, disable the MID Server "preUpgradeCheck" by adding the MID Server parameter mid.upgrade.run_precheck=false to each MID Server that does not have access to the install server.

AutoUpgrade.3600 SEVERE *** ERROR *** Aborting MID Server upgrade due to pre-upgrade check failure: Unable to download updates from install server 

See Disabling the pre-upgrade check in the docs.

5) Restart the MID Server

At this point you can wait for the AutoUpgrade thread to run again (it is on a 1 hour interval) or restart the MID Server service to force it to upgrade now.

The next time the AutoUpgrade thread runs, Downloaded: shows the files present in the <install folder>\agent\logs\agent0.log.0 log. It then goes ahead and does the upgrade.

StartupSequencer   Downloaded: [mid-upgrade.newyork-06-26-2019__patch0-hotfix2-07-10-2019_07-16-2019_1636.universal.universal.zip, mid-core.newyork-06-26-2019__patch0-hotfix2-07-10-2019_07-16-2019_1636.universal.universal.zip, mid-jre.newyork-06-26-2019__patch0-hotfix2-07-10-2019_07-16-2019_1636 .windows.x86-64.zip]

You can then re-use the downloaded files for any other MID Servers that are also connecting to the same instance.

For new London or Madrid releases, this would not have been an issue due to defaulting to using the instance as a kind of proxy (mid.download.through.instance=true), meaning direct access to the install server was not necessary. Since London Patch 8, Madrid Patch 3 and New York, the MID Server once again has to have access to the install server (PRB1332088/PRB627019). It is recommended that the property is set to false, and access is provided to the install server, in order to avoid API_INT semaphore exhaustion, MID Servers failing to connect to the instance and shutting down, and extended upgrade times during instance upgrades.