Notifications

13 views

Symptoms

AWS cloud discovery of member account resources using dynamically acquired credentials fail with error 401 - AuthFailure

Release

All currently supported releases.

Steps to Reproduce

  1. Navigate to "Cloud Management > Service Accounts".
  2. Select a member account.
  3. Click on the related link "Discover Datacenters".
  4. Navigate to "Cloud API > Cloud API Trail".
  5. Review logs for discovery. 

Cause

When discovering member accounts, the AWS discovery credential and master service account are used to generate a temporary token. In order to generate this token, the master account needs to have "AssumeRole" permission.

When the AWS organizations console is used to create a member account, AWS Organizations automatically creates an IAM role in the account. This is the "OrganizationAccountAccessRole". This role contains the necessary "AssumeRole" permission. The "OrganizationAccountAccessRole" is the role used by the MID server in order to generate the token.

If the "OrganizationAccountAccessRole" role is not created and with "AssumeRole" permission, the discovery of the member account will fail with error 401 - AuthFailure.

Reproduce the error, see "Steps to Reproduce" in this KB, and review the MID server log files to confirm this is the root cause for the 401 error. On the MID server log files the following error should be present:

<time_stamp> (523) Worker-Interactive:APIProxyProbe-<ecc_queue_sys_id> SEVERE *** ERROR *** Exception caught while trying to acquireTemporaryCredentialFromAWS()&#13;
com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException: Access denied (Service: AWSSecurityTokenService; Status Code: 403; Error Code: AccessDenied; Request ID: a2205d56-5189-11e9-a38b-9715ef911473)&#13;
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1640)&#13;
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1304)&#13;
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1058)&#13;
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)&#13;
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)&#13;
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)&#13;
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)&#13;
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)&#13;
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)&#13;
at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.doInvoke(AWSSecurityTokenServiceClient.java:1307)&#13;
at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1283)&#13;
at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.executeAssumeRole(AWSSecurityTokenServiceClient.java:466)&#13;
at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.assumeRole(AWSSecurityTokenServiceClient.java:442)&#13;
at com.service_now.mid.util.CloudServiceAccountCredentialUtil.generateFreshTemporaryCredentialForAccount(CloudServiceAccountCredentialUtil.java:606)&#13;

On the above, we see the acquireTemporaryCredentialFromAWS fails. Next, we attempt the discovery without a valid credential and therefore get "Could not complete API call AWS was not able to validate the provided access credentials (Service: AmazonEC2; Status Code: 401" error.

Resolution

Ensure the "OrganizationAccountAccessRole" is available and that has the "AssumeRole" permission.

Additional Information

The following AWS links provide helpful information on the "OrganizationAccountAccessRole" role and concepts, for a member account and invited accounts:

Article Information

Last Updated:2019-04-17 05:44:44
Published:2019-04-17