Notifications

533 views

Description

When performing a non-query operation (Insert,Update,Delete) the 'cmdb_ci_cloud_service_account' table an OOTB business rule creates a SystemCommand for every MID server. This command keeps the cloud service accounts in synch across all MID servers. These records would normally be updated with every Cloud Discovery.

If there are many accounts (e.g.: 1000+ as we've seen in some instances) and these are updated all at once (as Discovery or Imports would) then all MID servers will be flooded with 'service_account_update' SystemCommands and will be effectively down until those messages are processed. The Interactive thread pool will be blocked for hours, making the MID Server unusable.

Steps to Reproduce

Pre-Requisite: Clean instance with one or more running MID servers

  1. Navigate to sys.scripts.do
  2. Open ecc_queue_list.do in another tab
  3. From sys_scripts.do run the following. The purpose of this is to reproduce what a scheduled Discovery or re-Discovery of all Cloud accounts would do.
var gr = new GlideRecord('cmdb_ci_cloud_service_account');
for(var i = 0; i <= 1000; i++) { gr.initialize();
gr.datacenter_type='cmdb_ci_aws_datacenter';
gr.account_id = '12345678' + (('0000' + i).slice(-4));
gr.is_master_account = false;
gr.name = gr.account_id;
gr.object_id = gr.account_id;
gr.skip_sync = true;
gr.insert();
}
  1. Go to the ecc_queue and take note of the quantity of commands and the time it takes to process them.

Workaround

This problem is currently under review. You can contact ServiceNow Technical Support or subscribe to this Known Error article by clicking the Subscribe button at the top right of this form to be notified when more information will become available..

As a workaround, the following Business Rule can be added to the instance. Import and Commit this update set:

sys_remote_update_set_66873776db76fb0024a61681399619e4.xml

It contains a business rule to abort the insert of any new service_account_update jobs if the MID Server already has one queued in ready state. Each job synchronizes for all accounts, so only 1 job would need to run after the last of the updates are complete. This will prevent the backlog growing too large, without preventing any required jobs running.


Related Problem: PRB1354641

Seen In

SR - IRM - Audit Management - New York 2019 Q3
SR - IRM - GRC Profiles - Madrid 2019 Q2
SR - IRM - GRC Workbench - New York 2019 Q3
SR - IRM - PA Premium Integration - New York 2019 Q3
SR - IRM - Policy and Compliance - Madrid 2019 Q2
SR - IRM - Risk Management - New York 2019 Q3
SR - ITOM - CMDB CI Class Models - 201908
SR - ITOM - Discovery and Service Mapping - 201908
SR - ITOM - Discovery and Service Mapping - v1.0.35
SR - Security - Integration Framework - Madrid 2019 Q2
SR - Security - Support Common - Madrid 2019 Q2
SR - Security - Support Orchestration - Madrid 2019 Q2
SR - SIR - Security Incident Response - Madrid 2019 Q2
SR - SIR - Security Incident Response PA Content - New York 2019 Q3
SR - SIR - Store SecOps Setup Assistant - Madrid 2019 Q2
SR - SIR - Store Threat Core - Madrid 2019 Q2
SR - SIR - Store Trusted Security Circles Client - New York 2019 Q3
SR - SIR - Threat intelligence - New York 2019 Q3
SR - SIR - VirusTotal Integration - New York 2019 Q3
SR - VR - Qualys - New York 2019 Q3
SR - VR - Solution Management Madrid Q2
SR - VR - Vulnerability Response - New York 2019 Q3
SR - VR - Vulnerability Response PA Content - Madrid 2019 Q2

Fixed In

Orlando

Associated Community Threads

There is no data to report.

Article Information

Last Updated:2020-05-15 19:06:18
Published:2019-07-26