Notifications

42 views

Description

ServiceNow internal monitoring system alerts when it detects semaphore exhaustion and elevated response times on your instance 
At this time you may also experience increased response times due to longer running transactions 

You can find sys cache flush queries running on the Database at this time

Query 2611863 Writing to net abc appxxxx abc_1 14:28 SELECT sys_cache_flush0.`sequence`, sys_cache_flush0.`sys_id`, sys 
Query 2611829 Writing to net abc appxxxx abc_1 14:05 SELECT sys_cache_flush0.`sequence`, sys_cache_flush0.`sys_id`, sys 
Query 2608814 Writing to net abc appxxxx abc_1 13:19 SELECT sys_cache_flush0.`sequence`, sys_cache_flush0.`sys_id`, sys 
Query 2611848 Writing to net abc appxxxx abc_1 12:06 SELECT sys_cache_flush0.`sequence`, sys_cache_flush0.`sys_id`, sys 
Query 2611934 Writing to net abc appxxxx abc_1 12:06 SELECT sys_cache_flush0.`sequence`, sys_cache_flush0.`sys_id`, sys 
Query 2610291 Writing to net abc appxxxx abc_1 11:04 SELECT sys_cache_flush0.`sequence`, sys_cache_flush0.`sys_id`, sys 
Query 2610304 Writing to net abc appxxxx abc_1 09:50 SELECT sys_cache_flush0.`sequence`, sys_cache_flush0.`sys_id`, sys 
Query 2610298 Writing to net abc appxxxx abc_1 09:43 SELECT sys_cache_flush0.`sequence`, sys_cache_flush0.`sys_id`, sys 
Query 2610293 Writing to net abc appxxxx abc_1 09:29 SELECT sys_cache_flush0.`sequence`, sys_cache_flush0.`sys_id`, sys

Release or Environment

London and all previous releases

Cause

- Multiple update sets have been ran immediately prior to the start of impact 

- Update sets are known to trigger potentially impactful cache flushes which cause increases in instance response times especially if the update sets contain Database View

- This is due to a specific known issue PRB1342593 
This issue is encountered specifically when flushing caches required to support database views 

- A single update set can trigger multiple cache flushes during its processing which cause all active nodes to do the same 
- The result is that each application node has to rebuild these caches which is done by querying the database 

- A cache is only rebuilt when a specific transaction that needs to access it and finds it needed to be update 

- For specific caches this typically happens when over a period of several minutes as the various users access specific features 
However database view data is stored in a common cache that is required by all transactions so requests to rebuild this cache will trigger immediately across all nodes 

- The severity of PRB1342593 is directly related to the total number of application nodes in the instance so sub production instances are unlikely to see similar levels of impact, or even be impacted at all 

- Once the cache for database views is flushed all application node immediately start to query the database to repopulate it as soon as the next transaction is triggered 

- If an update set flushes the cache multiple times then this causes multiple queries all running to keep rebuilding the cache causing database connection that leads to performance issues as seen in this impact 

Resolution

Temporary Relief Measure

Restart the most heavily impacted application nodes 
This terminates all impactful transactions and allows caches to be rebuilt following the restart 

Solution

Please apply the patch for PRB1342593 

PRB1342593 was previously only resolved in future releases of the Madrid version of the ServiceNow platform , however it is now also available for London Patch 8 Hot Fix 4  release

Preventative Measure(s) 

In the mean time ServiceNow recommends that if you need to commit any update sets containing updates set you following the provided workaround and look to schedule the deployments outside of core business hours

Article Information

Last Updated:2019-08-28 16:31:05
Published:2019-08-28