General instance slowness accompanied by slowness in chat

Users might experience transaction slowness in ServiceNow, but instance health appears good from most (if not all) key performance indicators.  In addition to the general slowness, there are examples of connect chat errors and/or slowness in conversations.

Any current release of the platform using Connect Chat functionality.

Connect Chat transactions are background REST calls and should usually NOT cause a delay for other types of UI transactions like loading of a form or a list.  There are two specific exceptions to this, however. 

Will will touch on both briefly here as being aware of each is important to know if either is impacting user UI transactions:

1. One way in which connect chat conversation CAN cause slowness to occur is when ServiceDesk users (and others) have a large number of open (or active) chats.  There is a simple background script documented within KB0717049 which will provide a list of users with more than 50 open conversations.  Corrective action should be taken if this is the case. 
2. A second, and less obvious way in which chat transactions CAN cause UI slowness is when these transactions are themselves exceptionally slow due to custom Business Rules.

This article will focus on the second scenario, assuming the reader has already focussed and ruled out there being too many open/active chats per user.

This issue can be initially identified by a ServiceNow admin, but secondary verification of the cause may require ServiceNow performance support involvement to determine WHICH business rules are causing the transaction delay.

To identify chat as being a potential cause of transaction slowness.  Look for the following two indicators.  The first might not always be present, but users should be complaining of chat slowness regardless.  Look for errors like this:

The second and easier to capture indicator of this issue comes from filtering within the transaction logs.

Go to the module link, Transactions (all users) and query for a single day, with the following filters:

- Created on <whatever day>

- URL startswith /api/now/connect/conversations

- Response time greater than 4500

- Session wait time is 0

- Created by is not guest

Be sure that Response time, Business rule time, URL columns on on the list (as shown below).

Assuming the slow business rule time is found to be present for the specific REST "/api/now/connect/conversations" transactions, determining which business rule is causing the delay can be a little more cumbersome.  These are 3 options you can use:

1. Debug Business Rules can be attempted, and then opening and using chat might lead to the identification
2. Download the application node log for the node where a single slow transaction example occurred for the given day.  Using the System ID field on the transaction list will tell you which node to download the log file from.  Download a node log from System Logs--->System Log-->Node Log File Downloader.
3. Open a case with ServiceNow performance support and ask for assistance verifying the which Business Rules are making the /api/now/connect/conversations transactions slow.

NOTE: Option 2 can be a slow process.  Application node logs are VERY verbose and these are typically hundreds or megabytes and take a long time to download.  

Here is an example log file entry for a single chat conversation showing the BR in question:

~~~snip~~~
2019-08-27 07:05:19 (895) Default-thread-2 F941AC5513ABBB00A0F67C395144B06A txid=ba41ec5513ab Slow business rule 'VTB task query' on vtb_task:, time was: 0:00:00.502
2019-08-27 07:05:20 (432) Default-thread-2 F941AC5513ABBB00A0F67C395144B06A txid=ba41ec5513ab Slow business rule 'VTB task query' on vtb_task:, time was: 0:00:00.532
2019-08-27 07:05:20 (979) Default-thread-2 F941AC5513ABBB00A0F67C395144B06A txid=ba41ec5513ab Slow business rule 'VTB task query' on vtb_task:, time was: 0:00:00.541
2019-08-27 07:05:21 (529) Default-thread-2 F941AC5513ABBB00A0F67C395144B06A txid=ba41ec5513ab Slow business rule 'VTB task query' on vtb_task:, time was: 0:00:00.545
2019-08-27 07:05:22 (090) Default-thread-2 F941AC5513ABBB00A0F67C395144B06A txid=ba41ec5513ab Slow business rule 'VTB task query' on vtb_task:, time was: 0:00:00.556
~~~/snip~~~

A careful review of the business rule determined it was custom code and was no longer necessary.  Deactivating the business rule promptly resolved the Connect Chat slowness and as a result, more obvious UI transactions were no longer delayed due to excessive session wait (which can also be seen in the transaction list for other transactions).

KB0717049 should be reviewed to verify that users do NOT have an unnecessary volume of active chats.  If they do, these should be cleaned up first as mentioned in the KB article.  Afterward, checking for BR time slowness for /api/now/connect/conversations will confirm whether this article is still relevant.