Notifications

427 views

Description

After configuring an OAuth2 token with a short TTL you observe an increase in credentials_reload records in the ECC queue and observe increasing performance impacts on the instance and with activities utilizing the MID server.

Symptoms:

All MID Servers get the same credentials_reload jobs, so all will be running flat-out fetching, processing, and returning the results to the instance via the API_INT semaphores. This will delay or block other features using those semaphores, for example inbound REST and SOAP integrations and to other APIs may fail with status code 429 rejections, including MID Server connections.

The MID Servers's queue of jobs are likely to be delayed by hours, causing many of those jobs to time out. Import sets may run for hours. Discovery may time out at max runtime.

Confirming you have the problem:

To have this problem, you will need to have an active OAuth2 credential in the Discovery Credentials table, set to All MID Servers. When opening the OAuth2 credential records, you are likely to see a warning banner for imminent token expiry. Reloading the form a minute or so later will likely give a different expiry time, showing the token has a very short TTL:
OAuth Access token is available but will expire soon at 25-Jun-20 10:42:47. Verify the OAuth configuration and click the 'Get OAuth Token' link below to request a new token.
OAuth Access token is available but will expire soon at 25-Jun-20 10:43:07. Verify the OAuth configuration and click the 'Get OAuth Token' link below to request a new token.

Look in the ECC Queue table, and filter on source=credentials_reload. The ECC Queue is rotated and only holds 4-5 days worth of data, so if you see thousands of these records then you probably have this problem.
If you filter further on Queue=Output and Status=ready, then the records shown will be the backlog that the MID Servers are struggling to process. 

Instances running versions older than Orlando and New York Patch 8 are likely to also have additional problem, but if you also have a huge number of credentials_reload records in the ecc_queue then this is the problem to link with the case, as this problems workaround will be required for the additional steps for the token TTL change and cleanup.
PRB1342894 / KB0748823 An error occurred while decrypting credentials from instance - Creating OAuth 2.0 credential results in no credentials being retrieved by the MID server and Probes no longer being able to use them

Steps to Reproduce

  1. Configure OAuth2 token with TTL <1 minute
  2. Set OAuth2 discovery_credential record to "All MID Servers", which is the default
  3. Configure significant number of MID servers >100
  4. Let MID servers run

Note: This problem has been seen with considerably fewer MID Servers. It is a combination of the short TTL and number of MID Servers that causes the impact.

Workaround

- Disable "Sync Credentials to MID servers"
- Set problematic OAuth2 credential to "Specific MIDs" and empty MID assignment
- Set "credentials_reload" ecc_queue records in Output queue and Ready state to "Error" either through Database or background script

- Increase OAuth2 token TTL >30 minutes

- After these steps are done then the "Sync Credentials to MID servers" business rule can be reactivated


Related Problem: PRB1411442

Seen In

SR - Finance - Common - NY v1.1
SR - Finance - ERP Integration - NY v1.1
SR - Finance - Financial Close - NY v1.1
SR - Finance - Risk Management - NY v1.1
SR - IRM - Audit Management - New York 2019 Q3
SR - IRM - GRC Profiles - Madrid 2019 Q2
SR - IRM - GRC Workbench - New York 2019 Q3
SR - IRM - PA Premium Integration - New York 2019 Q3
SR - IRM - Policy and Compliance - Madrid 2019 Q2
SR - IRM - Risk Management - New York 2019 Q3
SR - IRM - SIG Assessment Legacy - Madrid 2019 Q1
SR - IRM - SIG Questionnaire - New York 2019 Q3
SR - IRM - Vendor Risk Management - Madrid 2019 Q1
SR - ITOM - Cloud Management Google Cloud Connector - v1.0
SR - ITOM - Cloud Management Terraform Connector - v1.1
SR - ITOM - CMDB CI Class Models - 201908
SR - ITOM - CMDB CI Class Models - 201909
SR - ITOM - Discovery and Service Mapping - 201908
SR - ITOM - Fundamentals Istanbul Jakarta Kingston r1 - v5.99.6
SR - Security - Integration Framework - Madrid 2019 Q2
SR - Security - Support Common - Madrid 2019 Q2
SR - Security - Support Orchestration - Madrid 2019 Q2
SR - SIR - Palo Alto WildFire Integration - New York 2019 Q3
SR - SIR - Security Incident Response - Madrid 2019 Q2
SR - SIR - Security Incident Response PA Content - New York 2019 Q3
SR - SIR - Splunk Sighting Search Integration - Madrid 2019 Q1
SR - SIR - Store SecOps Setup Assistant - Madrid 2019 Q2
SR - SIR - Store Threat Core - Madrid 2019 Q2
SR - SIR - Store Trusted Security Circles Client - New York 2019 Q3
SR - SIR - Tanium Integration - New York 2019 Q3
SR - SIR - Threat intelligence - New York 2019 Q3
SR - SIR - VirusTotal Integration - New York 2019 Q3
SR - SIR - WHOIS Integration - New York 2019 Q3
SR - VR - Qualys - New York 2019 Q3
SR - VR - Shodan Exploit - New York 2019 Q3
SR - VR - Vulnerability Response - New York 2019 Q3
SR - VR - Vulnerability Response PA Content - Madrid 2019 Q2

Intended Fix Version

Quebec

Fixed In

New York Patch 10
Orlando Patch 7
Paris Patch 1

Safe Harbor Statement

This "Intended Fix Version" information is meant to outline ServiceNow's general product direction and should not be relied upon in making a purchasing decision. The information provided here is for information purposes only and may not be incorporated into any contract. It is not a commitment, promise, or legal obligation to deliver any material, code, or functionality. The development, release, and timing of any features or functionality described for our products remains at ServiceNow's sole discretion.

Associated Community Threads

There is no data to report.

Article Information

Last Updated:2020-09-15 06:47:41
Published:2020-07-08