424 views

Description

ServiceNow has identified an issue with the SAM Counter scheduled job that is adversely impacting instance performance.

There was an incorrect query condition in the logic that cleans up the software installs at the end of each software counter execution. This causes the unintentional update of many records in the software installs table (with execution of business rules). The update being performed sets cached field to false, even if it is already false. With this problem, each execution of a software counter results in an update to the majority of the software installs. The higher the number of software counters and number of software installs, the higher the performance impact.

Steps to Reproduce

This known error causes performance degradation in the Software Asset Management license counters. It applies to instances where the following are true:

  • The Software Asset Management plugin is enabled
  • One of the following versions is installed on the instance: Fuji Patch 13, Geneva Patch 6, 7 or 8 (except those on Patch 7 Hotfix 7), or Helsinki before Patch 5
  • There are active software counters and records in the software installs table

The performance issue impacts the following flows:

  • Scheduled job SAM License Counter
  • Manually executing an individual software counter from the Count Licenses UI action on the Software Counter detail form
  • Manually executing software counters from the Check License Compliance menu in the navigator

The extent of the performance impact on the instance depends on the number of records in the software installs table and the number of active software counters. If the instance has a high volume of data, overall performance can be impacted.

There are three methods available to assess the level of risk on your instance.

(1) Check the processing duration of the SAM License Counters scheduled job

  1. Navigate to System Scheduler > Scheduled Jobs > Scheduled Jobs.
  2. In the Name column, search for SAM License Counters.
  3. Check the State column.

    If the job is Running, check the most recent start time. If the job started more than 10 hours ago, the instance may be impacted. The higher the number, the more likely the job is impacting system performance.

    If the job is Ready, the job has completed its last run. Personalize the list to display the Processing Duration column in the scheduled job list view. This number is the processing duration of the last successful job. (Note that if the job is Running, this may be a very small number because it may not have been updated yet.) The processing duration unit of measure is milliseconds. Convert processing duration to hours by dividing the number from the column by 3,600,000. If this is more than 10 hours, the instance may be impacted. The higher the number, the more likely the job is impacting system performance.

(2) Check the number of active software counters and the number of records in the software install table - multiply the 2 numbers

Check the number of active software counters:

  1. Navigate to Software Asset > Reconciliation > Software Counters.
  2. Filter by Active = true.
  3. Note the total number of records in the table.

Check the number of records in the software installations table:

  1. Navigate to Software Asset > Discovery > Software Installations.
  2. Note the total number of records in the table.

Multiply these two results against against each other. The higher the number, the greater the impact on the system.

 

(3) Using the Scripts - Background tool, execute the following script to print the number of active software counters, number of software installs, and their product

Note that if the tables do not exist, the results will be undefined and the instance is not at risk.

  var tSwInstall = new TableUtils('cmdb_sam_sw_install');
  if (tSwInstall.tableExists()) {
    var countCmdbInstallTable = -1;
    var gaCmdbInstall = new GlideAggregate('cmdb_sam_sw_install');
    gaCmdbInstall.addAggregate('COUNT');
    gaCmdbInstall.query();
    if (gaCmdbInstall.next())
      countCmdbInstallTable = gaCmdbInstall.getAggregate('COUNT');
  }

  var tSwCounter = new TableUtils('sam_sw_counter');
  if (tSwCounter.tableExists()) {
    var countSamCounterTableActive = -1;
    var gaSamCounterActive = new GlideAggregate('sam_sw_counter');
    gaSamCounterActive.addQuery('active',true);
    gaSamCounterActive.addAggregate('COUNT');
    gaSamCounterActive.query();
    if (gaSamCounterActive.next())
      countSamCounterTableActive = gaSamCounterActive.getAggregate('COUNT');
  }

  gs.print('Number of Installs: ' + countCmdbInstallTable);
  gs.print('Number of Active Software Counters: ' + countSamCounterTableActive);
  gs.print('Number of Installs * Number of Active Software Counters: ' + countCmdbInstallTable * countSamCounterTableActive);

 

When assessing risk using methods 2 or 3, the higher the number calculated, the higher the risk. For example, an instance with 100 active software counters and 100k installs is less impacted than an instance with 200 active software counters and 100k installs or an instance with 100 active software counters and 1M installs.

Use the list below to assess the risk of adverse performance impact due to the SAM License Counter scheduled job. The exact level of impact to your instance will vary depending on other factors, including instance size and if other database intensive activity is running during the scheduled job.

  • Low: Less than 1M
  • Medium: 1-10M
  • High: 10M-100M
  • Very High: 100M+

 

Workaround

The recommended steps vary depend on how high the risk numbers are from the steps to reproduce.

  • If the SAM License counter job is able to finish in less than 24 hours, a possible workaround is to move the scheduled job to run on a weekly basis and run it over the weekend during off-peak hours.
  • If the SAM License counter job is unable to finish or there are a high number of software installs/counters, we recommend disabling the job until the fix is applied to the instance. To disable, change the scheduled job to On Demand.
  • A third option is to disable software counters by setting them to inactive. If the number of active counters is reduced, the total amount of time required to run the scheduled job is reduced, potentially providing some relief until the instance can be upgraded to a release with the fix. Remember to set the software counters to active after the upgrade is completed.

Related Problem: PRB704167

Seen In

Fuji Patch 10
Fuji Patch 12 Hot Fix 1
Fuji Patch 13 Hot Fix 1
Geneva Patch 4
Geneva Patch 7
Geneva Patch 7 Hot Fix 2
Geneva Patch 8
Geneva Patch 8 Hot Fix 1
Geneva Patch 9
Helsinki Patch 2
Helsinki Patch 3
Helsinki Patch 3 Hot Fix 4
Helsinki Patch 3 Hot Fix 7
Helsinki Patch 4

Fixed In

Geneva Patch 10
Geneva Patch 7 Hot Fix 7
Geneva Patch 9
Helsinki Patch 5
Helsinki Patch 6
Istanbul

Associated Community Threads

There is no data to report.

Article Information

Last Updated:2017-06-13 10:23:46
Published:2016-11-29