In Istanbul, a memory leak has been discovered that prevents the AMB session message queue from being flushed correctly.
- Once the session_message_queue_size has reached full size (10MB), AMB messages will not be processed on the node(s) experiencing the max-out queue
- Each node in a cluster has a different AMB session message queue, so 1 or more nodes may experience the issue over time
- Current size of the queue may be seen on xmlstats.do, in <amb><session_message_queue_size> in the max value
- Warning will be present in the log files when the max size has been reached with the content:
AMBClusterSynchronizer Did not process messages because the AMBSessionMessageQueues has reached max capacity.
Clarification on impact of this memory leak:
The impact of this memory leak is limited and not instance-wide. This means that, when an instance is suffering from this problem, some or all messages related to following features may not be delivered:
- Record watcher
There is no other impact to the instance i.e. instance will continue to perform as expected for all other features. It will not slow down, or exhibit any incorrect behavior.
Steps to Reproduce
- Normal usage of ServiceNow, which leverages AMB (Connect, record watcher, etc.) will slowly cause the issue over time.
- Leak may be exacerbated with heavier usage of AMB features, especially Chat.
The node(s) exceeding capacity in session_message_queue_size must be restarted.
This issue is under review. You can also subscribe to this known error article (click Subscribe button at the top of the article) to receive notifications when more information is available about this issue.
Related Problem: PRB935863