- If you have received notifications on an incident created from an alert generated by the ServiceNow Monitoring System with the subject as "AMB Send Semaphore Exhausted" and if your instance has been upgraded to Madrid, this article will apply.
- Please note that the incident may be generated for specific nodes and may not apply to all nodes.
Steps to Reproduce
If you are logged in to the affected node(s), please refer to the following to verify the issue.
- When logged in to your ServiceNow instance, open a new browser tab and access the xmlstats.do page - https://<Your ServiceNow instance name>.service-now.com/xmlstats.do
- After the page is loaded, perform a browser search for "AMB_SEND" to look for the AMB_SEND semaphore section. Please see the following for an example.
<semaphores available="4" max_queue_depth="7" maximum_concurrency="4" name="AMB_SEND" queue_age="0" queue_depth="2" queue_depth_limit="50" rejected_executions="0"/> <queue>
<queued queued_at="Mon Feb 16 06:17:41 PST 2019" queued_duration="215378320">null #369349 send /amb/meta/subscribe</queued>
<queued queued_at="Mon Feb 16 06:17:41 PST 2019" queued_duration="215378317">null #369350 send /amb/meta/subscribe</queued>
- You will also note that:
- the value for "queue_depth" is greater than 0
- the queued AMB_SEND transactions have a queued_at date which is earlier than the current time (it may be a few days/weeks earlier)
- the transactions have a "null" session id (before the transaction number).
- If the queue_depth is low, you may not have noticed any impact as the queue depth (the number of transactions that can be queued) is 50
- However, if the queue_depth is close to 50, there may be rejected transactions (too many prior requests if the same kind are already waiting to run).
- Please note that only the AMB_SEND transactions will be affected, for example, live changes on the form (usually indicated by a blue icon) may not show up and chat may not be working.
- To clear the buildup in the AMB_SEND queue, a node restart can be performed by ServiceNow Support during a change window of your choice. If all nodes are affected, a rolling restart of the instance may be required. As the instance will be restarted in a rolling fashion, there will not be a downtime.
- Please provide your consent and the preferred change window in the incident.
- Please refer to the "Fixed In" section for the fixed releases.
Related Problem: PRB1325392