Default semaphores exhausted with large number of oauth_token.do transactions

Default semaphores exhausted with large number of oauth_token.do transactions that remain active for several minutes. User transaction response time increase due to limited or no available Default Semaphores to process other transactions.

Testing LDAP connectivity by navigating to System LDAP > LDAP Servers and selecting the proper LDAP configuration results in errors or intermittent failures.

Examining the /stats.do page reveals all or most Default Semaphores flooded with long running /oauth_token.do transactions.

0:4475E9321B6640D461DB8663CC4BCB0B #28517236 /oauth_token.do (Default-thread-3) (0:01:24.565)
1:ED75213A1B6640D461DB8663CC4BCB72 #28517331 /oauth_token.do (Default-thread-8) (0:01:19.103)
2:E585653A1B6640D461DB8663CC4BCB1C #28517585 /oauth_token.do (Default-thread-2) (0:01:02.820)

Analyzing the system logs or the localhost log file uncovers several LDAP and OAuth errors and warnings.

• SEVERE *** ERROR *** LDAP: Validation failed for CN=snserviceuser,OU=ServiceNow Service Accounts,OU=Service Accounts
• WARNING *** WARNING *** OAuthTokenProcessor caught a InvalidCredentialProvidedException with message invalid_scope: The provided OAuth token is not valid

All platforms.

Customer's LDAP server was undergoing an unspecified issue and was unable to process and respond to authentication requests from several external application, including the ServiceNow instance. The instance user account 'snserviceuser' could not get an authentication token which caused it to remain active in the Default Semaphores until time out of 300 seconds was reached.

Customer restarted their LDAP server and restored access for all external application, including ServiceNow instance service accounts. All pending oauth_token.do transactions were processed, and clear and new transactions were processed with no delay.