In times of heavy traffic, a 429 error can occur when a node attempts to process API requests from multiple MID Servers that exceed the default database queue limit of 50. If this happens when a MID Server is loading credentials, Discovery stalls because the MID Server has no queue and cannot retry. 

Some of the contributing factors are:

  • More than 100 transactions per second on an individual node.
  • Heavy cache reload traffic, such as many script include changes in an update set.
  • More large API requests than available threads.
  • F5 load balancing issues caused by the assignment of MID Servers to nodes. See KB0623708 for details.


Steps to Reproduce


Send a node four very slow API requests, followed by 100 credential reloads.
After all the reloads, the MID Server will not have any credentials available, and any subsequent Discoveries will stall.


If a MID Server is stuck in this state, it can be recovered either by restarting it or by issuing a credential reload probe followed by a query.

A credential reload causes the MID Server to download the credentials from the instance again.

  • topic = SystemCommand
  • source = credentials_reload

Reseting the query window tells the MID Server it might have missed some older ecc_queue entries and to search more deeply the next time it looks.

  • topic = SystemCommand
  • source = resetQueryWindow

If either of these methods is successful, ecc_queue outputs for the stalled MID Server change to Processing and then Processed, and corresponding input records begin to appear.

NOTE: Optionally, you can "tune" your system using one of the following long-term solutions, which are advanced procedures that must be performed by ServiceNow Customer Support.

  • Increase the number of semaphores available to the node.
  • Delete the "cookies" the system uses to decide which node the MID Server communicates with. See KB0623708 for details.

Related Problem: PRB750509

Seen In

Geneva Patch 3 Hot Fix 7

Fixed In

Istanbul Patch 9
Jakarta Patch 3

Associated Community Threads

There is no data to report.

Article Information

Last Updated:2018-03-14 07:39:05