If there is an OAuth 2.0 type Credential [oauth_2_0_credentials] added to the list of Credentials in the instance, the instance will no longer return any credentials to the MID server when the MID Server tries to re-load the credential list. The MID Server will no longer be able to run any probes that requires credentials. This includes all Discovery, Orchestration, Event Management Connector probes, and others.
The MID Server agent logs will show a "SEVERE *** ERROR *** An error occurred while decrypting credentials from instance" when running each affected probe.
Steps to Reproduce
- Create a new OAuth 2.0 credential using an OOB OAuth Entity Profile (e.g. Google default_profile) on an instance with at least one MID running.
- Verify that an ECC queue output record has been processed by the MID(s) with the topic credentials_reload. The agent logs will also confirm this ran.
- Verify that in the agent log file for the MID(s) there is an exception that looks like the following. This is the exception thrown when no key element is found in the response (i.e., the response is empty)
SEVERE *** ERROR *** An error occurred while decrypting credentials from instance
com.snc.automation_common.integration.exceptions.AutomationIOException: Unable to retrieve data from instance. This MID may not be validated. <<<=== Red Herring - The MID Server is Validated.
Lines below this in the stack trace will be specific to the probe running.
This problem has been fixed. There is no workaround available. If you are able to upgrade, review the Fixed In section to determine the latest version with a permanent fix your instance can be upgraded to.
The workaround is to ensure the affected MID Servers do not have any OAuth Credentials set up for them:
- Deactivating all OAuth 2.0 Credentials in the Credentials table will workaround this issue, however they will then not be available to the features that require them.
- If OAuth Credentials are still needed to be Active for integrations or other features, set the OAuth 2.0 Credential record field "Applies To:" to "Specific MID Servers" and leave the field "MID servers" as empty. This will make the credential inaccessible from any mid server and only available for instance. Note that, even for OAuth 2.0 credentials to be used on instance, the flow should be executed as system user and not session user.
- Please make sure to restart the mid service manually from the mid host if you see the same issue even after de-activating the OAuth 2.0 credentials.
NOTE: The same symptom/error could also be due to PRB1305469 Excluding table-per-class (TPC) extended tables from a clone can cause orphaned Discovery Credentials with the 'Record not found' error when trying to open them
If you also have thousands of source=credentials_reload jobs backed-up in the ECC Queue, then you probably also experiencing this, which requires additional steps to resolve:
PRB1411442 / KB0829702 OAuth 2.0 credential that is configured with extremely short TTL (<1 minute) causes ECC queue flooding (credentials_reload command) and leads to semaphore exhaustion on instance
Related Problem: PRB1342894