Notifications

11 views

Description

During Nutanix VM event base discovery, we see thousands of "ASYNC: Discovery - Sensors" jobs running in short interval processing results from "Nutanix VM Event" patterns leading to DB CPU spikes and DB contentions.

Release or Environment

New York

Cause

Out of the box, we have "Nutanix-Events-job" scheduled job which runs every 5 minutes and create "NutanixStackDriverGetRest" Javascript Probes to fetch VM events from Nutanix Clusters. For each event we will launch Horizontal Discovery to discover the VM using "Nutanix VM Event" Pattern. This event based discovery is used to complement the standard Nutanix Serverless discovery to get updated VM information near real time.

During patching intervals, thousands of VMs are being recycled. This generates thousands of VM Events which subsequently triggering thousands of Horizontal Discovery. The discovery is fast but due to sheer # of discovery results needed to be processed, this leads to DB contention on ServiceNow leading to performance degradation.

Resolution

Development team is currently looking for options to enhance this feature to reduce # redundant discoveries and handling VM events better. If necessary, Customer can turn off the "Nutanix-Events-job" scheduled job and let the regular Nutanix Serverless discovery schedule discover the VMs.

Article Information

Last Updated:2020-03-19 15:09:30
Published:2020-03-19