There is some confusion over the certificates and Java Keystores involved with MID Servers, which this KB hopes to clarify.
1. Outbound from MID Server to Instance, Integrations, or Proxy/Firewall in between
1.1 OCSP Check
Since Orlando, the MID Server needs to validate that certificates haven't been revoked for any reason, so the MID Server host needs to be able to connect to the OCSP/CRL servers of the certificate, as well as the instance URL.
As of the Paris release, that was purchased from Entrust, and so the URLs of the Entrust OCSP/CRL servers need to be open in your firewall. Entrust use DNS-based load balancing the IPs that these resolve may be different each time, so the URL always needs opening in the firewall.
You can use this tool on the 3rd party "SSL Labs" WebSite to see what the OCSP/CRL URLs are for the certificate of your instance. Swap the "hi.service-now.com" URL for your instance URL. The "Revocation information" section will give the URLs that will need opening in your firewall for the MID Server hosts. HTTP port 80 is normal for OCSP/CRL servers, and as it is a security protocol, should normally be possible to make an exception for in your company's security policies.
If the certificate truly has been revoked, this tool will also confirm it. In this situation it is right for the MID Server to block the endpoint:
MID Server Agent log will have this error for multiple threads:
WARNING *** WARNING *** OCSP revoke check IOException for *.service-now.com
WARNING *** WARNING *** org.apache.commons.httpclient.HttpException: Connection reset
The fix for PRB1417416 in Quebec (Concurrency performance: JVM locks on OCSPCheckedCertificateCache are not released) also adds the OCSP url to the error output, so that it is clear that it is the authority and not the endpoint that encountered an error.
Knowledge Base articles:
- PRB1385357 MID Server can fail to install or upgrade to Orlando due to new external connectivity requirement to ocsp.entrust.net for OCSP certification revocation verification check
- KB0854165 MID server OSCP requirement details (includes a temporary workaround, however this should not be used as a long term solution)
- PRB1417416 Concurrency performance: JVM locks on OCSPCheckedCertificateCache are not released leading to import schedules taking exponentially longer (WARNING *** Freed stuck JVM lock: OCSPCheckedCertificateCache)
1.2 Certificate Chain Check
Keystore File: .\agent\jre\lib\security\cacerts
Default Password: changeit
These are Public keys for the SSL/TLS (Transport Layer Security) certificates of outgoing HTTPS connections. Outgoing integrations, such as to the Instance, LDAPS servers, or any proxy/firewall in between, will need to set up the secure HTTPS connection using a certificate. HTTP connections are forbidden for the connections to the ServiceNow instance. If the server we make the request to has a self-signed certificate, or one that does not have the normal root certificates in it's chain, then you will require the certificate loading into the cacerts file.
This is done using the agent\jre\bin\keytool.exe (/agent/jre/bin/keytool for linux). The whole chain of certificates may need adding, including intermediate certificates.
All required certificates for all instances and mid servers could be loaded into the same cacerts file, and then copy that file to all MID Servers. It's also a good idea to then keep a backup of that file.
This is not just used for 3rd party integrations. If a Proxy server, firewall, or other transparent network device is intercepting the traffic from MID Server to instance, then the certificate the MID Server needs to trust is the network devices, not the instance's. You can confirm this by opening the instance in a browser on the MID Server host and checking the Certificate. If it is the *.service-now.com one, then nothing needs doing. If it is not, then you will possibly have to export that certificate from the browser, and all the others up the chain, and import those to the cacerts file.
In your browser accessing the instance from the MID Server host, you would expect the certificate chain to look like this (as of Paris timeframe). Click the padlock icon next to the url, and select certificate:
If the chain looks different, then you probably have a transparent network device in-between that is using the normal certificate between it and the instance, but substituting a different certificate for the HTTPS connection between the MID Server and itself. In the below examples, the blacked out parts were the customer's company name, suggesting these are self-signed certificates generated from their own CA server.
Some proxy/firewall devices may be set to regularly change the certificate, and the certificate you manually added to cacerts no longer works. The only solution may be to add an exception to the firewall for mid server hosts connecting to instance URLs.
If a traditional Proxy server is used, the hostname will be shown in the MID Server Configuration Parameters on the MID Server form. If you don't see the mid.proxy.use_proxy=true parameter, there may still be a proxy/firewall intercepting the connection.
For instances with worker nodes, and the MID Servers are set to connect to the special URL for worker nodes, the same certificate is also used, which is also used with the install.service-now.com server that has the install and upgrade files.
Orlando Patch 7a/8, Paris Patch 1 Hot Fix 3a/5a, Paris Patch 2, and Quebec add code to check certificate chains for all outbound requests using TLS certificates (PRB1419895), which may cause errors due to missing certificates in cacerts that had previously gone unchecked, breaking integrations and the connection to the instance. The above workaround for turning the OCSP check off temporarily also turns off this check.
Knowledge Base articles:
- KB0816002 How to obtain SSL certificate from the browser
- KB0863109 MID Server cannot connect to ServiceNow instance, error in agent log: org.apache.commons.httpclient.HttpException: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
- KB0746078 Retiring TLS1.0 and 1.1
- MID Servers and Worker Nodes
- Add SSL certificates for the MID Server
- Oracle Java: keytool
- Wikipedia: HTTPS, Transport Layer Security (TLS/SSL), Java Keystore
- PRB1447511 Hotfix for PRB1419895 Lack of Certificate Validation causes MID Server DOWN.
- PRB1320637 A MID Server upgrade that includes a new JRE version will overwrite the cacerts file
- PRB1419895 Lack of Certificate Validation KB0861038 Orlando Patch 7a / KB0861889 Paris Patch 1 Hotfix 3a Security fix Release Notes
The first 2 examples include "Request not sent to uri", which means the MID Server decided it could not trust the URI and not send the request at all. This is a good clue that it is something about the endpoint certificates.
Connecting to the instance through a proxy/firewall may see this error in the agent logs. This happens to be from the JAR file requests, but all other requests to the instance will have a similar error. Note the "uri" in the error is for the instance, which is the diagnostic error for this cause. In this case the certificate required was for the network device URL, not the instance URL. "Unable to find certificate chain" is the clue here.
File sync worker: ecc_agent_jar WARNING *** WARNING *** Request not sent to uri= https://<INSTANCE_NAME>.service-now.com/MIDFileSyncSnapshot.do?SOAP : javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: java.security.cert.CertPathBuilderException: Unable to find certificate chain.
File sync worker: ecc_agent_jar SEVERE *** ERROR *** SOAP Request: <SOAP-ENV:Envelope xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:tns="http://www.service-now.com/GetMIDInfo" xmlns:m="http://www.service-now.com" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"><SOAP-ENV:Body><m:execute><attachments xsi:type="xsd:string">true</attachments><type xsi:type="xsd:string">ecc_agent_jar</type></m:execute></SOAP-ENV:Body></SOAP-ENV:Envelope>
File sync worker: ecc_agent_jar SEVERE *** ERROR *** SOAP Response: Status code=0, Response body=null
File sync worker: ecc_agent_jar WARNING *** WARNING *** Could not get file sync snapshot because: Request not sent to uri= https://<INSTANCE_NAME>.service-now.com/MIDFileSyncSnapshot.do?SOAP : javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: java.security.cert.CertPathBuilderException: Unable to find certificate chain.
A similar error for the instance certificate, where the Entrust root certificate was not available, and required adding to the cacerts file. This happens to be from the InstanceInfo request, but all other requests to the instance will have a similar error "peer not authenticated" is the clue here.
StartupSequencer WARNING *** WARNING *** Unable to get InstanceInfo: Request not sent to uri= https://<INSTANCE_NAME>.service-now.com/InstanceInfo.do?SOAP : org.apache.commons.httpclient.HttpException: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
(Network Configuration issue) Please check that the MID server can ping the instance: https://c<INSTANCE_NAME>.service-now.com/
You may also need to configure the network that the MID server uses to allow traffic over TCP port 443.
An LDAPProbe import, with a missing certificate for the AD server, as seen from the Test Load records form in the instance, which will also be seen in the MID Server agent log. "No issuer certificate for certificate in certification path found" is the clue here.
MID Server reported error: javax.naming.CommunicationException: RTCDOMPRD03.rt.corp:636 [Root exception is javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: java.security.cert.CertPathBuilderException: No issuer certificate for certificate in certification path found.]
Caused by error in MID Server script include 'CSMIDServerRemoteFileImport' at line 23
2. MID Server to Instance (for validation/encryption)
Keystore File: .\agent\keystore\agent_keystore.jks
This is not user-editable.
This is a private key, automatically created on MID Server startup if it is not there, and contains a certificate to establish the trust between instance and mid server, and for decrypting data such as discovery credentials and other ecc_queue payload data that is encrypted. The MID Server cannot be validated to run any jobs unless that file is correct, or decrypt credentials from the instance to use when running jobs.
It is often recreated automatically on first startup after upgrade, and is also regularly refreshed.
When that refresh works, it looks something like this in the agent log of the MID Server:
StartupSequencer Updated public key, new certificate:
 Version: 3
Start Date: Mon Oct 26 10:11:32 PDT 2020
Final Date: Tue Oct 26 10:11:32 PDT 2021
Public Key: RSA Public Key [30:57:b5:a0:57:98:94:a2:e2:66:dd:79:b3:7f:37:6c:d8:9a:31:ff],[56:66:d1:a4]
public exponent: 10001
Signature Algorithm: SHA512WITHRSA
When that fails, it may look more like:
StartupSequencer SEVERE *** ERROR *** UpdatePublicKey error: Digital signature of new public key must be provided. MID Server encryption keys do not match and are no longer valid. To restore proper functionality, invalidate and re-validate the MID Server.
StartupSequencer WARNING *** WARNING *** Unable to update public key, will try again next MID Server restart
Typically, it will say your signature is bad rather than missing. e.g.
SEVERE *** ERROR *** Unable to load keystore: Unexpected IOException loading KeyStore, caused by: Keystore was tampered with, or password was incorrect SEVERE *** ERROR *** Keystore and config.xml files out of sync. 1) Delete keystore/agent_keystore.jks or restore config.xml to its previous state, 2) ensure MID server has write permissions to config.xml and to keystore directory, 3) restart MID server. WARNING *** WARNING *** Encountered error: [Unable to load keystore] while starting up the service. Retry...
If the MID Server remain UP, then clicking the Re-Key related link on the MID Server form can sometimes resolve this.
If the MID Server is Down, then deleting the file .\agent\keystore\agent_keystore.jks, clearing the keypairs.mid_id parameter value from the config.xml file, and restarting the MID Server service usually resolves this. A new good file should be created automatically on startup. The MID Server might still need validating from the MID Server record in the instance.
3. Inbound to MID Server Web Server extension from Integrations
Keystore File: .\agent\keystore\webserver_keystore.jceks
Password: Whatever you set it to be on manual creation
When the MID Server is acting as a Web Server for receiving push integrations, such as inbound REST/SOAP, Event Management event collection, or for Agent Client Collector installs, the certificate to use for this web server is uploaded to .\agent\keystore\webserver_keystore.jceks
This is the private key for the web server, supplied/generated by the customer.
This is also added using the Keytool utility. Only the certificate of the host needs adding, so the file has just 1 certificate.
This certificate is for the URL that will be used when sending requests to the MID Server's Web server, which may not always be the hostname of the mid server, if DNS aliases or a load balancer are used.
The Event Management feature has the best documentation on this, although other features use the same web server extension:
- PRB1442388 MID Servers Web Server Extension set to use a Certificate, may fail to start after upgrade to Paris (newer jetty version uses deprecated SSL package, breaking certificate chains)
Error: java.lang.IllegalStateException: KeyStores with multiple certificates are not supported on the base class org.eclipse.jetty.util.ssl.SslContextFactory