4045 views

Description

There is some confusion over the certificates and Java Keystores involved with MID Servers, which this KB hopes to clarify.

1. Outbound from MID Server to Instance, Integrations, or Proxy/Firewall in between

1.1 OCSP Check

Since Orlando, the MID Server needs to validate that certificates haven't been revoked for any reason, so the MID Server host needs to be able to connect to the OCSP/CRL servers of the certificate, as well as the instance URL.

As of the Paris release, that was purchased from Entrust, and so the URLs of the Entrust OCSP/CRL servers need to be open in your firewall. Entrust use DNS-based load balancing the IPs that these resolve may be different each time, so the URL always needs opening in the firewall.
CRL: http://crl.entrust.net/level1k.crl
OCSP: http://ocsp.entrust.net

You can use this tool on the 3rd party "SSL Labs" WebSite to see what the OCSP/CRL URLs are for the certificate of your instance. Swap the "hi.service-now.com" URL for your instance URL. The "Revocation information" section will give the URLs that will need opening in your firewall for the MID Server hosts. HTTP port 80 is normal for OCSP/CRL servers, and as it is a security protocol, should normally be possible to make an exception for in your company's security policies.

If the certificate truly has been revoked, this tool will also confirm it. In this situation it is right for the MID Server to block the endpoint:

MID Server Agent log will have this error for multiple threads:

WARNING *** WARNING *** OCSP revoke check IOException for *.service-now.com 
WARNING *** WARNING *** org.apache.commons.httpclient.HttpException: Connection reset

The fix for PRB1417416 in Quebec (Concurrency performance: JVM locks on OCSPCheckedCertificateCache are not released) also adds the OCSP url to the error output, so that it is clear that it is the authority and not the endpoint that encountered an error.

Documentation links:

Knowledge Base articles:

1.2 Certificate Chain Check

Keystore File: .\agent\jre\lib\security\cacerts
Default Password: changeit

These are Public keys for the SSL/TLS  (Transport Layer Security) certificates of outgoing HTTPS connections. Outgoing integrations, such as to the Instance, LDAPS servers, or any proxy/firewall in between, will need to set up the secure HTTPS connection using a certificate. HTTP connections are forbidden for the connections to the ServiceNow instance. If the server we make the request to has a self-signed certificate, or one that does not have the normal root certificates in it's chain, then you will require the certificate loading into the cacerts file. 

This is done using the agent\jre\bin\keytool.exe (/agent/jre/bin/keytool for linux). The whole chain of certificates may need adding, including intermediate certificates.

All required certificates for all instances and mid servers could be loaded into the same cacerts file, and then copy that file to all MID Servers. It's also a good idea to then keep a backup of that file.

This is not just used for 3rd party integrations. If a Proxy server, firewall, or other transparent network device is intercepting the traffic from MID Server to instance, then the certificate the MID Server needs to trust is the network devices, not the instance's. You can confirm this by opening the instance in a browser on the MID Server host and checking the Certificate. If it is the *.service-now.com one, then nothing needs doing. If it is not, then you will possibly have to export that certificate from the browser, and all the others up the chain, and import those to the cacerts file.

In your browser accessing the instance from the MID Server host, you would expect the certificate chain to look like this (as of Paris timeframe). Click the padlock icon next to the url, and select certificate:


If the chain looks different, then you probably have a transparent network device in-between that is using the normal certificate between it and the instance, but substituting a different certificate for the HTTPS connection between the MID Server and itself. In the below examples, the blacked out parts were the customer's company name, suggesting these are self-signed certificates generated from their own CA server.



Some proxy/firewall devices may be set to regularly change the certificate, and the certificate you manually added to cacerts no longer works. The only solution may be to add an exception to the firewall for mid server hosts connecting to instance URLs.

If a traditional Proxy server is used, the hostname will be shown in the MID Server Configuration Parameters on the MID Server form. If you don't see the mid.proxy.use_proxy=true parameter, there may still be a proxy/firewall intercepting the connection.

For instances with worker nodes, and the MID Servers are set to connect to the special URL for worker nodes, the same certificate is also used, which is also used with the install.service-now.com server that has the install and upgrade files.

Orlando Patch 7a/8, Paris Patch 1 Hot Fix 3a/5a, Paris Patch 2, and Quebec add code to check certificate chains for all outbound requests using TLS certificates (PRB1419895), which may cause errors due to missing certificates in cacerts that had previously gone unchecked, breaking integrations and the connection to the instance. The above workaround for turning the OCSP check off temporarily also turns off this check.

Knowledge Base articles:

Documentation links:

Related Problems:

Examples

The first 2 examples include "Request not sent to uri", which means the MID Server decided it could not trust the URI and not send the request at all. This is a good clue that it is something about the endpoint certificates.

Connecting to the instance through a proxy/firewall may see this error in the agent logs. This happens to be from the JAR file requests, but all other requests to the instance will have a similar error. Note the "uri" in the error is for the instance, which is the diagnostic error for this cause. In this case the certificate required was for the network device URL, not the instance URL. "Unable to find certificate chain" is the clue here.

File sync worker: ecc_agent_jar WARNING *** WARNING *** Request not sent to uri= https://<INSTANCE_NAME>.service-now.com/MIDFileSyncSnapshot.do?SOAP : javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: java.security.cert.CertPathBuilderException: Unable to find certificate chain.
File sync worker: ecc_agent_jar SEVERE *** ERROR *** SOAP Request: <SOAP-ENV:Envelope xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:tns="http://www.service-now.com/GetMIDInfo" xmlns:m="http://www.service-now.com" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"><SOAP-ENV:Body><m:execute><attachments xsi:type="xsd:string">true</attachments><type xsi:type="xsd:string">ecc_agent_jar</type></m:execute></SOAP-ENV:Body></SOAP-ENV:Envelope>
File sync worker: ecc_agent_jar SEVERE *** ERROR *** SOAP Response: Status code=0, Response body=null
File sync worker: ecc_agent_jar WARNING *** WARNING *** Could not get file sync snapshot because: Request not sent to uri= https://<INSTANCE_NAME>.service-now.com/MIDFileSyncSnapshot.do?SOAP : javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: java.security.cert.CertPathBuilderException: Unable to find certificate chain.

A similar error for the instance certificate, where the Entrust root certificate was not available, and required adding to the cacerts file. This happens to be from the InstanceInfo request, but all other requests to the instance will have a similar error "peer not authenticated" is the clue here. 

StartupSequencer WARNING *** WARNING *** Unable to get InstanceInfo: Request not sent to uri= https://<INSTANCE_NAME>.service-now.com/InstanceInfo.do?SOAP : org.apache.commons.httpclient.HttpException: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
(Network Configuration issue) Please check that the MID server can ping the instance: https://c<INSTANCE_NAME>.service-now.com/
You may also need to configure the network that the MID server uses to allow traffic over TCP port 443.

An LDAPProbe import, with a missing certificate for the AD server, as seen from the Test Load records form in the instance, which will also be seen in the MID Server agent log. "No issuer certificate for certificate in certification path found" is the clue here.

MID Server reported error: javax.naming.CommunicationException: RTCDOMPRD03.rt.corp:636 [Root exception is javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: java.security.cert.CertPathBuilderException: No issuer certificate for certificate in certification path found.]
at com.sun.jndi.ldap.Connection.<init>(Connection.java:238)
at com.sun.jndi.ldap.LdapClient.<init>(LdapClient.java:137)
at com.sun.jndi.ldap.LdapClient.getInstance(LdapClient.java:1609)
at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2749)
at com.sun.jndi.ldap.LdapCtx.<init>(LdapCtx.java:319)
at com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(LdapCtxFactory.java:192)
at com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(LdapCtxFactory.java:210)
at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(LdapCtxFactory.java:153)
at com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(LdapCtxFactory.java:83)
at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:684)
at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:313)
at javax.naming.InitialContext.init(InitialContext.java:244)
at javax.naming.ldap.InitialLdapContext.<init>(InitialLdapContext.java:154)
at com.service_now.mid.probe.LDAPProbe.verifyConnectivity(LDAPProbe.java:124)
at com.service_now.mid.probe.LDAPProbe.doWork(LDAPProbe.java:99)
at com.service_now.mid.probe.LDAPProbe.probe(LDAPProbe.java:77)
at com.service_now.mid.probe.AProbe.process(AProbe.java:103)

A JavascriptProbe, with missing certificate for an endpoint:

Agent Log:
Worker-Standard:JavascriptProbe-05422d31db9c9050d6f9f9b2f396193f WARNING *** WARNING *** javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: java.security.cert.CertPathBuilderException: Unable to find certificate chain.
Caused by error in MID Server script include 'CSMIDServerRemoteFileImport' at line 23

Wrapper Log:
Worker-Standard:JavascriptProbe-05422d31db9c9050d6f9f9b2f396193f, SEND TLSv1.2 ALERT: fatal, description = certificate_unknown
Worker-Standard:JavascriptProbe-05422d31db9c9050d6f9f9b2f396193f, WRITE: TLSv1.2 Alert, length = 2
Worker-Standard:JavascriptProbe-05422d31db9c9050d6f9f9b2f396193f, called closeSocket()
Worker-Standard:JavascriptProbe-05422d31db9c9050d6f9f9b2f396193f, handling exception: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: java.security.cert.CertPathBuilderException: Unable to find certificate chain.

2. MID Server to Instance (for validation/encryption)

Keystore File: .\agent\keystore\agent_keystore.jks

This is not user-editable.

This is a private key, automatically created on MID Server startup if it is not there, and contains a certificate to establish the trust between instance and mid server, and for decrypting data such as discovery credentials and other ecc_queue payload data that is encrypted. The MID Server cannot be validated to run any jobs unless that file is correct, or decrypt credentials from the instance to use when running jobs.

It is often recreated automatically on first startup after upgrade, and is also regularly refreshed.

When that refresh works, it looks something like this in the agent log of the MID Server:

StartupSequencer Updated public key, new certificate:
[0] Version: 3
SerialNumber: 1603732292
IssuerDN: CN=snc-mid-datacenterdev_dva400_disco_midserver17-52721102f0331010e36955d33f0f9500,DC=service-now,DC=com
Start Date: Mon Oct 26 10:11:32 PDT 2020
Final Date: Tue Oct 26 10:11:32 PDT 2021
SubjectDN: CN=snc-mid-datacenterdev_dva400_disco_midserver17-52721102f0331010e36955d33f0f9500,DC=service-now,DC=com
Public Key: RSA Public Key [30:57:b5:a0:57:98:94:a2:e2:66:dd:79:b3:7f:37:6c:d8:9a:31:ff],[56:66:d1:a4]
modulus: d2cd8c54a36990c57bd4146eea3677913b491fd82541ea462d15cf22a057a010c88bcd154f910134251ceccbbcbc5bbf00315199f557428986fc21b041fe4f8941ddc43a79d0257cc01c48fd681a39a7490255a6b2fb63dbaf01184727d15dcebfc9d1d2e3a532ece1512b418626f8c663d193d9cb896db29bec44fafcea63458b5497964155580d6389653a4613c4ceabd52d4a60228f127e59da4c32d394eab013ea0d3bb236c1ad1d5642135869de2047124164f7ea2f1a692df2e6bf174d896ea4c49c93e979cd905875c528061df8867fd06ac79f17b882ecaedf75bb54535a71d52368dff6a3baecc45e9ae9202289ccda7621015ff2372ac22906e0d1
public exponent: 10001
Signature Algorithm: SHA512WITHRSA
Signature: 6a6202d16462659433f461c5153207a2aa6a63dc
5c4263c1ac2af3aba7da8b2b0208606db958ae5a
577f38df6a2fdfb80d21f5306a1ec0e695a64cf6
aa97b4da715f19d41907df95e4e6a2e4e6157577
9e68407d48a794248a76890e6bfc22cb60e29db2
7100976fd5d2f947aa267cda262e6b2ba52e705d
af47ff7cb972b44b0238b5d0eda24a252499d304
8845e00bb112e762186755ce3079ec45194cfc0f
23b3adca0f7a38137aea35e1bf978f3f65b4cfe2
68aae654469716d54e2fda433b1142fe478fccb0
f1ae683e050ba14062e8593e4cc4e684c68f339e
09e376c3b8f74fc8d54bd4452dbc4a4cfbf9d047
a42cdad2e608eeab1ea16a753bb54923

When that fails, it may look more like:

StartupSequencer SEVERE *** ERROR *** UpdatePublicKey error: Digital signature of new public key must be provided. MID Server encryption keys do not match and are no longer valid. To restore proper functionality, invalidate and re-validate the MID Server.
StartupSequencer WARNING *** WARNING *** Unable to update public key, will try again next MID Server restart

Typically, it will say your signature is bad rather than missing. e.g.

SEVERE *** ERROR *** Unable to load keystore: Unexpected IOException loading KeyStore, caused by: Keystore was tampered with, or password was incorrect
SEVERE *** ERROR *** Keystore and config.xml files out of sync. 1) Delete keystore/agent_keystore.jks or restore config.xml to its previous state, 2) ensure MID server has write permissions to config.xml and to keystore directory, 3) restart MID server.
WARNING *** WARNING *** Encountered error: [Unable to load keystore] while starting up the service. Retry...

If the MID Server remain UP, then clicking the Re-Key related link on the MID Server form can sometimes resolve this.

If the MID Server is Down, then deleting the file .\agent\keystore\agent_keystore.jks, clearing the keypairs.mid_id parameter value from the config.xml file, and restarting the MID Server service usually resolves this. A new good file should be created automatically on startup. The MID Server might still need validating from the MID Server record in the instance.

3. Inbound to MID Server Web Server extension from Integrations

Keystore File: .\agent\keystore\webserver_keystore.jceks
Password: Whatever you set it to be on manual creation

When the MID Server is acting as a Web Server for receiving push integrations, such as inbound REST/SOAP, Event Management event collection, or for Agent Client Collector installs, the certificate to use for this web server is uploaded to .\agent\keystore\webserver_keystore.jceks 

This is the private key for the web server, supplied/generated by the customer.

This is also added using the Keytool utility. Only the certificate of the host needs adding, so the file has just 1 certificate.

This certificate is for the URL that will be used when sending requests to the MID Server's Web server, which may not always be the hostname of the mid server, if DNS aliases or a load balancer are used.

The Event Management feature has the best documentation on this, although other features use the same web server extension:

Related Problem:

Article Information

Last Updated:2020-11-27 07:02:33
Published:2020-11-27