Issue
Users across many states are experiencing 5 seconds to 5-minute delays when trying to do anything. The problem is also random in appearance.
Release
Helsinki and above release with Live Feed / Live Profile functionality.
Cause
A very large live profile image for a user profile icon has been uploaded and stored in sys_attachement. This small thumbnail type image isn't reduced in size though even though it is displayed as a small picture. When ANY record with the users live profile icon is displayed, it causes the form or transaction to be slow if the image is not already locally cached in the browser. In this example, User A accesses a cagtalog task updated by User B. User B has a 50mb image uploaded as their live profile avatar. This causes a significant delay.
The outward evidence of this is somewhat hard to find because calls to the profile attachment image (iix file) are not visible from within the "Transaction (all user)" module as it's just a sys_id.iix like the following:
https://<instance_name>.service-now.com/90a152891b13b74028f57661cd4bcb84.iix
This transaction isn't viewable from the platform front end but can be seen from a web browser using the native developer tools to see the network timing:
The iix transaction isn't visible within the transactions module, but it causes very high session wait for many other transactions as shown here and this can be a pointer to this issue as well.
Resolution
The root cause of the random slowness was confirmed to have been user-id XYZ live profile picture from sys_attachement.
The "blocking" background REST transaction call was to the following:
https://instance_name.service-now.com/90a152891b13b74028f57661cd4bcb84.iix
Many transactions had very high session wait due to the above background image being called silently behind the scenes.
Multiple users with transaction slowness could be found to be calling the above image. This was not easy to find because this specific transaction was not visible in the transaction log within the platform, but only from the application node log as and from the browser network developer console.
The sys_attachment record below was uploaded by XYZ and was 50mb in size and took over 2 minutes to download in the background for any users in the platform who accessed a record user XYZ had updated or if they were in a chat group with user XYZ
https://instance_name.service-now.com/sys_attachment_list.do?sysparm_query=sys_idSTARTSWITH90a152891b13b74028f57661cd4bcb84 <<<--- this sys_id is the same as the .iix file name prefix.
Removing this attachment record found all transaction slowness to be resolved.
Related Links
If unable to determine if the live profile phot/avatar image the source of transaction slowness (did you read the above?), consider checking for large images in the sys_attachment table where table_name = ZZ_YYlive_profile. Sort Z-A on Size bytes. It's best to stay away from any large images, especial any over 5mb. In the above exampe the image was 50mb (50000000 byes).
