1436 views

Description

AMB Messages queued for delivery can push a session's waiting transactions beyond the system allowed maximum. The result is that legitimate UI requests are ignored. 

The concept of "Max Waiters" was built into the platform to protect one user from exhausting the queue of an application node. We only allow 10 transactions per session to wait in the queue at the time of each transaction execution and anything over that threshold is ignored. Sessions use session synchronization so only one transaction can process per user at a time. 

Symptoms

This issue can produce blank pages or pages that seem to be loading, but never do. It can also cause many different kinds of issues for things that rely heavily on requests such as Connect and Service Portal.

Look for the following in the application logs:

"WARNING *** WARNING *** GlideRequestManager:Request ignored: <some_transaction>"

Here is an example log entry:

2017-09-11 12:34:55 (193) http-16 WARNING *** WARNING *** GlideRequestManager:Request ignored: /api/now/live/profiles/e39511210a0a3c6d00ef8620582702f1

Note that for releases IP6 and before, the presence of node log warning messages alone is not enough to confirm that the issue is this problem. If you are on IP6 or before, set the property com.glide.request.max_waiters to 10 before confirming. After this change, if you still see the log messages above, it is highly likely that you are running into this issue. Otherwise, if changing the value to 10 removes the logs messages, then it is highly likely not this problem. If needed, in that case, open an incident with ServiceNow Customer Support.

Steps to Reproduce

  1. Enable Connect.
    You can also do this with any other channel subscription.
  2. In one browser, log in as an ITIL user and go to the navpage.
  3. In a second browser, execute the following in Scripts - Background:

    for (var i = 0; i < 5000, i++) {
    gs.publish("/connect/message/6816f79cc0a8016401c5a33be04be441", {});
    }


Workaround

Although the following workaround will not completely resolve the issue, it can offer relief in many cases.

  • Add a system property named com.glide.request.max_waiters.
    • Type: Integer
    • Value: 20 (we do not recommend setting this value any higher than 20)

This workaround only works if the amount of waiters doesn't go above 20. To check the number of waiters please turn on glide.servlet.debug_logging system property for some time and observe what transactions increment the max waiters counter in the logs. Look for statements like this:

 glide.amb.cluster.synchronizer Increment waiters to 1 for A0E50423E7001300917484FA03F6A9DC 
 
If the number of waiter go above 20 this workaround will not work for you.

Another suggestion is to determine if there is a process or transaction that is slow and is causing the waiters to queue up. If yes, see if the time spent on that transaction can be reduced.


Related Problem: PRB1177878

Seen In

Kingston

Fixed In

Istanbul Patch 11
Istanbul Patch 5 Hot Fix 5
Jakarta Patch 6
Kingston Patch 1
London

Associated Community Threads

There is no data to report.

Article Information

Last Updated:2018-09-19 23:18:49
Published:2018-05-09