Notifications

38 views

Description

This is primarily focused on AWS and Azure Cloud Discovery, other Cloud providers have not been explicitly considered but likely have the same behavior as presented below.

During Cloud Discovery you may note errors similar to the following:

Azure:

HTTP/1.1 429 Too Many Requests x-ms-ratelimit-remaining-resource: Microsoft.Compute/HighCostGet3Min;46 x-ms-ratelimit-remaining-resource: Microsoft.Compute/HighCostGet30Min;0 Retry-After: 1200 Content-Type: application/json; charset=utf-8 { "code": "OperationNotAllowed", "message": "The server rejected the request because too many requests have been received for this subscription.", "details": [ { "code": "TooManyRequests", "target": "HighCostGet30Min", "message": "{\"operationGroup\":\"HighCostGet30Min\",\"startTime\":\"2018-06-29T19:54:21.0914017+00:00\",\"endTime\":\"2018-06-29T20:14:21.0914017+00:00\",\"allowedRequestCount\":800,\"measuredRequestCount\":1238}" } ] }

AWS:

<ErrorResponse xmlns="http://cloudformation.amazonaws.com/doc/2010-05-15/">
<Error>
<Type>Sender</Type>
<Code>Throttling</Code>
<Message>Rate exceeded</Message>
</Error>

 

Release or Environment

All

Cause

These errors are caused by the relevant Cloud Provider throttling/denying the Cloud Discovery API request due to the API limits being exceeded.

Resolution

Due to how Cloud Discovery is designed, it would be exceedingly difficult for Cloud Discovery itself breaching the API throttling limits for AWS and Azure.

For Azure their limits are 12,000 API requests per hour (see additional info)

For AWS their limits are 100 API requests, per region, per second (see additional info)

With how Cloud Discovery is designed we iterate every region for an API call before moving onto the next (e.g: We execute DescribeInstances on us-west-1, then us-west-2, etc.), this would make it impossible to breach the 100 API calls per second per region unless there was over 100 Cloud Discovery schedules executing their API calls at the exact same time.

With a conservative estimation (higher number of API calls) we roughly make 100 API calls per region for a given Discovery Schedule, so it would be difficult to breach Azure's 12,000 API call per hour limit as well.

 

In all scenarios in which Throttling/Rate Exceeded exceptions have been observed in Cloud Discovery, the root cause has always been an external software/integration that is inefficiently querying/spamming Azure or AWS' API endpoint.

 

To resolve these issues you will need to identify what outside tools/software are querying your AWS/Azure API and either fine tune them or disable them.

Additional Information

AWS API Throttling documentation:

https://docs.aws.amazon.com/AWSEC2/latest/APIReference/throttling.html#throttling-limits

 

Azure API Throttling documentation:

https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/request-limits-and-throttling

Article Information

Last Updated:2020-05-08 13:13:24
Published:2020-05-08