Notifications

2952 views

Description

If there is an OAuth 2.0 type Credential [oauth_2_0_credentials] added to the list of Credentials in the instance, the instance will no longer return any credentials to the MID server when the MID Server tries to re-load the credential list. The MID Server will no longer be able to run any probes that requires credentials. This includes all Discovery, Orchestration, Event Management Connector probes, and others.

The MID Server agent logs will show a "SEVERE *** ERROR *** An error occurred while decrypting credentials from instance" when running each affected probe.

Steps to Reproduce

  1. Create a new OAuth 2.0 credential using an OOB OAuth Entity Profile (e.g. Google default_profile) on an instance with at least one MID running.
  2. Verify that an ECC queue output record has been processed by the MID(s) with the topic credentials_reload. The agent logs will also confirm this ran.
  3. Verify that in the agent log file for the MID(s) there is an exception that looks like the following. This is the exception thrown when no key element is found in the response (i.e., the response is empty)
SEVERE *** ERROR *** An error occurred while decrypting credentials from instance
com.snc.automation_common.integration.exceptions.AutomationIOException: Unable to retrieve data from instance. This MID may not be validated. <<<=== Red Herring - The MID Server is Validated.
at com.glide.util.MIDServerInfoPayloadDecrypter.decryptPayload(MIDServerInfoPayloadDecrypter.java:25)
at com.service_now.mid.creds.provider.standard.StandardCredentialsProvider.loadCredentials(StandardCredentialsProvider.java:289)
at com.service_now.mid.creds.provider.standard.StandardCredentialsProvider.load(StandardCredentialsProvider.java:256)
at com.service_now.mid.creds.provider.standard.StandardCredentialsProvider.init(StandardCredentialsProvider.java:275)
at com.service_now.mid.creds.provider.MIDCredentialsConfigProvider.getCredentialsProvider(MIDCredentialsConfigProvider.java:58)
at com.snc.commons.credentials.CredentialsProviderFactory.getCredentialsProvider(CredentialsProviderFactory.java:30)
...

Lines below this in the stack trace will be specific to the probe running.

Workaround

This problem has been fixed. There is no workaround available. If you are able to upgrade, review the Fixed In section to determine the latest version with a permanent fix your instance can be upgraded to.

The workaround is to ensure the affected MID Servers do not have any OAuth Credentials set up for them:

  • Deactivating all OAuth 2.0 Credentials in the Credentials table will workaround this issue, however they will then not be available to the features that require them.
  • If OAuth Credentials are still needed to be Active for integrations or other features, set the OAuth 2.0 Credential record field "Applies To:" to "Specific MID Servers" and leave the field "MID servers" as empty. This will make the credential inaccessible from any mid server and only available for instance. Note that, even for OAuth 2.0 credentials to be used on instance, the flow should be executed as system user and not session user.
  • Please make sure to restart the mid service manually from the mid host if you see the same issue even after de-activating the OAuth 2.0 credentials.

NOTE: The same symptom/error could also be due to PRB1305469 Excluding table-per-class (TPC) extended tables from a clone can cause orphaned Discovery Credentials with the 'Record not found' error when trying to open them

If you also have thousands of source=credentials_reload jobs backed-up in the ECC Queue, then you probably also experiencing this, which requires additional steps to resolve:
PRB1411442 / KB0829702 OAuth 2.0 credential that is configured with extremely short TTL (<1 minute) causes ECC queue flooding (credentials_reload command) and leads to semaphore exhaustion on instance


Related Problem: PRB1342894

Seen In

Madrid Patch 2
SR - DevOps - DevOps Insights 1.6
SR - Finance - Common - Madrid v2.0.0
SR - Finance - ERP Integration - Madrid v4.0.0
SR - Finance - Financial Close - Madrid v4.1.2
SR - Finance - Risk Management - Madrid v5.1.0
SR - IntegrationHub - Box Integration - Madrid - v2.0.0
SR - IntegrationHub - Docusign Integration r1 v1
SR - IntegrationHub - JIRA Service Desk Integration r2 - v2.0.0
SR - IRM - Audit Management - New York 2019 Q3
SR - IRM - GRC Profiles - Madrid 2019 Q2
SR - IRM - GRC Workbench - New York 2019 Q3
SR - IRM - PA Premium Integration - New York 2019 Q3
SR - IRM - Policy and Compliance - Madrid 2019 Q2
SR - IRM - Risk Management - New York 2019 Q3
SR - IRM - SIG Assessment Legacy - Madrid 2019 Q1
SR - IRM - SIG Questionnaire - New York 2019 Q3
SR - IRM - Vendor Risk Management - Madrid 2019 Q1
SR - ITBM - Agile 2.0 Dashboards v1.0
SR - ITBM - Scrum Dashboards Common v1.0
SR - ITOM - Cloud Management Google Cloud Connector - v1.0
SR - ITOM - Cloud Management Terraform Connector - v1.1
SR - ITOM - CMDB CI Class Models - 201907
SR - ITOM - CMDB CI Class Models - 201908
SR - ITOM - CMDB CI Class Models - 201909
SR - ITOM - Discovery and Service Mapping - 201908
SR - ITOM - Discovery and Service Mapping - v1.0.35
SR - ITOM - Fundamentals Istanbul Jakarta Kingston r1 - v5.99.6
SR - PAR - Performance Analytics Content Pack for Service Portal - v1.0
SR - Security - Integration Framework - Madrid 2019 Q2
SR - Security - Support Common - Madrid 2019 Q2
SR - Security - Support Orchestration - Madrid 2019 Q2
SR - SIR - Have I Been Pwned Integration - New York 2019 Q3
SR - SIR - Security Incident Response - Madrid 2019 Q2
SR - SIR - Security Incident Response PA Content - New York 2019 Q3
SR - SIR - Security Incident Response UI Patch - London 2019 Q2 v.6.2.3
SR - SIR - Splunk Enterprise Integration - New York 2019 Q3
SR - SIR - Splunk Sighting Search Integration - Madrid 2019 Q1
SR - SIR - Store SecOps Setup Assistant - Madrid 2019 Q2
SR - SIR - Store Threat Core - Madrid 2019 Q2
SR - SIR - Store Trusted Security Circles Client - New York 2019 Q3
SR - SIR - Store Trusted Security Circles Client Advanced- Madrid 2019 Q1
SR - SIR - Threat intelligence - New York 2019 Q3
SR - SIR - VirusTotal Integration - New York 2019 Q3
SR - SIR - WHOIS Integration - New York 2019 Q3
SR - VR - Configuration Compliance - New York 2019 Q3
SR - VR - Qualys - New York 2019 Q3
SR - VR - Rapid7 - London 2019 Q2 v.6.2.1
SR - VR - Shodan Exploit - New York 2019 Q3
SR - VR - Vulnerability Response - New York 2019 Q3
SR - VR - Vulnerability Response PA Content - Madrid 2019 Q2
SR Hybrid Analysis Kingston r1 - v5.0.9

Fixed In

New York Patch 8
Orlando

Associated Community Threads

There is no data to report.

Article Information

Last Updated:2020-09-03 07:34:45
Published:2020-07-08