MID Server down due to error "Too Many Requests with code: 429"
The API_INT is one of the semaphores used by the MID servers when communicating to the instance. If the API_INT semaphores are exhausted, and the queue depth has reached the max queue depth, then the instance will return the error "Too Many Requests with code: 429" to any MID server clients attempting communication. Because there are no available semaphores, the instance will not be able to receive any inputs or heartbeats from the MID server. Therefore the MID server status will be set to down.
When this happens, the following error can be seen in the MID server logs:
ECCQueueMonitor.5 WARNING *** WARNING *** Method failed: (https://YOUR_INSTANCE.service-now.com/ecc_queue.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 429 Too Many Requests with code: 429 ECCQueueMonitor.5 SEVERE *** ERROR *** getRecords failed (Method failed: (https://YOUR_INSTANCE.service-now.com/ecc_queue.do?SOAP&displayvalue=all&redirectSupported=true)HTTP/1.1 429 Too Many Requests with code: 429)
To check whether the API_INT semaphores are exhausted:
In a browser window, navigate to <your_instance>.service-now.com/stats.do.
Search for API_INT.
Check that there are no available semaphores and the queue depth equals the max queue depth.
An exhausted API_INT resembles the following example.
API_INT Available semaphores: Queue depth: 50 Max queue depth: 50 0:F48CE383DB6A760098E53CAF9D961901 #583094 /api/now/table/sys_audit (API_INT-thread-2) (2:05:30.388) 1:C25E23C3DB6A760098E53CAF9D961962 #583187 /api/now/table/sys_audit (API_INT-thread-4) (1:57:30.087) 2:6BCE63C3DB6A760098E53CAF9D9619F4 #583264 /api/now/table/sys_audit (API_INT-thread-1) (1:55:29.678) 3:95157F47DB6A760098E53CAF9D9619D9 #584585 /api/now/table/sys_audit (API_INT-thread-3) (1:28:03.004)
In this example, note that long queries on the sys_audit table are using up all available API_INT semaphores. However, the long-running queries could be on a different table.
The MID server being down and the 429 errors are the result of the API_INT semaphores being exhausted. Therefore, the solution depends on what is keeping the semaphores busy.
For immediate relief:
Navigate to User Administration > All Active Transactions.
Find the long-running transactions keeping the semaphores busy.
Confirm with the team responsible for the transactions that killing the transactions will not cause any issues.
Kill the transactions to free up the semaphores.
For long-term relief:
Contact the team responsible for the long-running transactions and work with the team on improving the transactions.