Notifications

75 views

Description

When performing a non-query operation (Insert,Update,Delete) the 'cmdb_ci_cloud_service_account' table an OOTB business rule creates a SystemCommand for every MID server. This command keeps the cloud service accounts in synch across all MID servers. These records would normally be updated with every Cloud Discovery.

If there are many accounts (e.g.: 1000+ as we've seen in some instances) and these are updated all at once (as Discovery or Imports would) then all MID servers will be flooded with 'service_account_update' SystemCommands and will be effectively down until those messages are processed. The Interactive thread pool will be blocked for hours, making the MID Server unusable.

Steps to Reproduce

Pre-Requisite: Clean instance with one or more running MID servers

  1. Navigate to sys.scripts.do
  2. Open ecc_queue_list.do in another tab
  3. From sys_scripts.do run the following. The purpose of this is to reproduce what a scheduled Discovery or re-Discovery of all Cloud accounts would do.
var gr = new GlideRecord('cmdb_ci_cloud_service_account');
for(var i = 0; i <= 1000; i++) { gr.initialize();
gr.datacenter_type='cmdb_ci_aws_datacenter';
gr.account_id = '12345678' + (('0000' + i).slice(-4));
gr.is_master_account = false;
gr.name = gr.account_id;
gr.object_id = gr.account_id;
gr.skip_sync = true;
gr.insert();
}
  1. Go to the ecc_queue and take note of the quantity of commands and the time it takes to process them.

Workaround

This problem is currently under review. You can contact ServiceNow Technical Support or subscribe to this Known Error article by clicking the Subscribe button at the top right of this form to be notified when more information will become available..

As a workaround, the following Business Rule can be added to the instance. Import and Commit this update set:

sys_remote_update_set_66873776db76fb0024a61681399619e4.xml

It contains a business rule to abort the insert of any new service_account_update jobs if the MID Server already has one queued in ready state. Each job synchronizes for all accounts, so only 1 job would need to run after the last of the updates are complete. This will prevent the backlog growing too large, without preventing any required jobs running.


Related Problem: PRB1354641

Seen In

There is no data to report.

Intended Fix Version

Orlando

Safe Harbor Statement

This "Intended Fix Version" information is meant to outline ServiceNow's general product direction and should not be relied upon in making a purchasing decision. The information provided here is for information purposes only and may not be incorporated into any contract. It is not a commitment, promise, or legal obligation to deliver any material, code, or functionality. The development, release, and timing of any features or functionality described for our products remains at ServiceNow's sole discretion.

Associated Community Threads

There is no data to report.

Article Information

Last Updated:2019-08-20 18:18:06
Published:2019-07-26