MID Servers may go Down and be unable to connect when WS-Security header verification for all incoming SOAP requests is enabled

MID Servers may go Down and be unable to connect when WS-Security header verification for all incoming SOAP requests is enabled. That SOAP Web Services system Property is documented here:
https://docs.servicenow.com/csh?topicname=t_EnableWS-SecurityVerification.html&version=latest

Once the following properties are set, the instance will enforce WSS on all incoming SOAP Requests.
glide.soap.require_ws_security = true
glide.soap.default_security_policy = <policy_name>

MID Servers use SOAP to connect to the instance, for various MID Server-specific scripted SOAP Services and for the Table API, and so this means MID Servers will no longer be authenticated with the instance.

1/ Create a MID Server user, with the mid_server role, as per the docs
https://docs.servicenow.com/csh?topicname=t_SetupMIDServerRole.html&version=latest
2/ Set up a MID Server using that user, as per
https://docs.servicenow.com/csh?topicname=mid-server-installation.html&version=latest

At this point the MID Server will be shown as Up.

3/ Set up a security policy and enable WS-Security Verification as per
https://docs.servicenow.com/csh?topicname=t_EnableWS-SecurityVerification.html&version=latest

4/ Check the agent log of the MID Server and you will see something like:

09/21/19 06:54:54 (984) ECCQueueMonitor.40 WARNING *** WARNING *** Method failed: (https://<instance>.service-now.com/ecc_queue.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 500 Internal Server Error with code: 500
09/21/19 06:54:54 (984) ECCQueueMonitor.40 SEVERE *** ERROR *** getRecords failed (An error was discovered processing the WS-Security header No certificate(s) found in WS-Security profile)

This will mean although it is still up and running, it cannot pick up any new jobs from the instance. That includes the Heartbeat probe, so the MID Server will be set as DOWN within 5 minutes.

Other errors may include the thread that is supposed to be sending results back to the instance.

ECCSender.1 WARNING *** WARNING *** RemoteGlideRecord failed to send data to https://instancename.service-now.com/ with (An error was discovered processing the WS-Security header)

5/ Restart the MID Server

The MID Server will now be unable to connect to the instance.

09/21/19 14:39:58 (364) StartupSequencer WARNING *** WARNING *** Could not authenticate user 'MID.USER' on the ServiceNow instance
09/21/19 14:39:58 (365) StartupSequencer SEVERE *** ERROR *** test failure
java.lang.IllegalStateException: User cannot be authenticated or is missing the proper roles. If you have deleted or changed the MID server keystore, and config.xml mid.instance.password value is encrypted, you may need to change this value to plain text (during MID startup, password is re-encrypted using current keystore and written back to mid.instance.password).

While that is happening, errors like the following will be seen in the instance node logs for SOAP requests to /ecc_queue.do?, :

05:14:40.246 Info http-44 SYSTEM New transaction 17C7FC51DB88405041CDC344059619F4 #676425 /MIDServerCheck.do
05:14:40.255 Info API_INT-thread-4 SYSTEM txid=5bc7fc51db88 HTTP authorization validated user 'MID.USER'
05:14:40.255 Info API_INT-thread-4 SYSTEM txid=5bc7fc51db88 Session user set to VO_MID_SERVER_PROD
05:14:40.259 Info API_INT-thread-4 17C7FC51DB88405041CDC344059619F4 txid=5bc7fc51db88 *** Start #676425 /MIDServerCheck.do, user: MID.USER 
05:14:40.259 Info API_INT-thread-4 17C7FC51DB88405041CDC344059619F4 txid=5bc7fc51db88 SOAPProcessor: initial session inactivity timeout is 60 seconds
05:14:40.259 Info API_INT-thread-4 17C7FC51DB88405041CDC344059619F4 txid=5bc7fc51db88 SOAPProcessor: initial soap request timeout is 60 seconds
05:14:40.259 Info API_INT-thread-4 17C7FC51DB88405041CDC344059619F4 txid=5bc7fc51db88 SOAPProcessor: session inactivity timeout changed to 60 seconds
05:14:40.259 Info API_INT-thread-4 17C7FC51DB88405041CDC344059619F4 txid=5bc7fc51db88 SOAPProcessor: mid_server,soap,soap_script,soap_query,soap_create,soap_delete,soap_ecc,soap_script,soap_update
05:14:40.261 Info API_INT-thread-4 17C7FC51DB88405041CDC344059619F4 txid=5bc7fc51db88 WebServiceSecurity: Retrieving Server Certificate to Validate
05:14:40.262 Info API_INT-thread-4 17C7FC51DB88405041CDC344059619F4 txid=5bc7fc51db88 WebServiceSecurity: Verifying Signature
05:14:40.263 Warning API_INT-thread-4 17C7FC51DB88405041CDC344059619F4 txid=5bc7fc51db88 WARNING *** WARNING *** WSSecurityEngine did not return any results - no SOAP Header element in incoming request? - Unable to verify signed request
05:14:40.263 Warning API_INT-thread-4 17C7FC51DB88405041CDC344059619F4 txid=5bc7fc51db88 WARNING *** WARNING *** WS-Security request rejected
05:14:40.263 Info API_INT-thread-4 17C7FC51DB88405041CDC344059619F4 txid=5bc7fc51db88 Sending response
05:14:40.263 Warning API_INT-thread-4 17C7FC51DB88405041CDC344059619F4 txid=5bc7fc51db88 WARNING *** WARNING *** SOAP Fault: wsse:InvalidSecurityAn error was discovered processing the WS-Security headerWSSecurityEngine did not return any results - no SOAP Header element in incoming request? - Unable to verify signed request
05:14:40.263 Info API_INT-thread-4 17C7FC51DB88405041CDC344059619F4 txid=5bc7fc51db88 User agent with HTTP/1.1 and no encoding: internal_soap_client
05:14:40.263 Info API_INT-thread-4 17C7FC51DB88405041CDC344059619F4 txid=5bc7fc51db88 Response bytes sent: 447
05:14:40.263 Info API_INT-thread-4 17C7FC51DB88405041CDC344059619F4 txid=5bc7fc51db88 *** End #676425 /MIDServerCheck.do, user: MID.USER , total time: 0:00:00.017, processing time: 0:00:00.017, SQL time: 0:00:00.004 (count: 13), ACL time: 0:00:00.001, source: 185.61.73.80 , type:soap, method:POST, api_name:SOAP APIs, resource:MIDServerCheck.do, user_id:8eb1b666db29334041cdc3440596199d, response_status:500

This problem is currently under review. You can contact ServiceNow Technical Support or subscribe to this Known Error article by clicking the Subscribe button at the top right of this form to be notified when more information will become available.

The solution is to Mark the MID server user as an internal integration user.

1. Navigate to User Administration > Users.
2. Select the user account for the MID Server or ODBC Driver.
3. Configure the form to add the Internal Integration User field, or edit through a list.
4. Select the Internal Integration User checkbox.
5. Click Update.

If you are running London or earlier version, then while updating the MID Server users you will receive the error "A user with the 'Internal Integration User' option set to true cannot be granted the mid_server role". That is because of PRB1206641, and is fixed from Madrid.

To workaround that, please temporarily de-activate these 2 business rules, and then revert them back to the out-of-box version (don't simply tick activate again as that will remain as a customization):

• Incompatible MID Server user role (sys_id = 64923f4bc713220003fa9c569b9763e6)
• User settings incompatible with MID (sys_id = c006104c723220003fa9c569b9763c5 or fc006104c723220003fa9c569b9763c5)

In this situation, deactivate these business rules and then activate them again. To find the business rules:

1. Navigate to System Definition > Business Rules.
2. Search by the sys_id or name.