Notifications

88 views

Description

The CMDB Baseline tables can grow to huge sizes, which leads to performance issues when querying the tables, as well as database server disk space issues. Table sizes of >100GB, and even 3TB have been seen.

Each cmdb_baseline_entry record can in theory be up to 16MB, if the CI has a lot of relationships and related records.

By default, a baseline will be created for all CIs. It is quite common to see CMDBs with >50M records. Most of that data is pointless, as dscy_.. and other network discovery and Service Mapping CI classes will be included. A default filter that excludes a lot of classes is required. The scheduler worker thread creating the baselines can run for many hours and put a considerable load on the database.

The wiki used to give an example scheduled script for Creating baselines, which is commonly used. This is no longer in the Docs, but it was not replaced with a more sensible script for managing creation and deletion/rotation of baselines.

The platform has no feature for cleaning or rotation old baselines, or any best practice advice or documentation on managing baselines, and the best way to use them, and so this table gets bigger and bigger.

Steps to Reproduce

  1. Implement CMDB Baseline with a scheduled job to regularly create a new baseline, for the whole cmdb
  2. Monitor the Database size growth over the weeks and years
  3. Monitor the database load while the baselines are created

Example of a production instance cmdb_baseline_entry record count:

pbrun snow query XXXXX --count --timer --timeout-seconds=500 "select count(*) from cmdb_baseline_entry"
Result set includes 1 rows
Query time 14 seconds and 282 milliseconds
+------+----------+
| Port | count(*) |
+------+----------+
| 3401 | 86960736 |
+------+----------+

Example thread stack trace of BaselineCMDB.create running for 23 days due to the huge size and slow performance of the cmdb_baseline_entry table:

main,glide.scheduler.worker.1,4,ASYNC: Script Job (1986859542 ms)
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
java.net.SocketInputStream.read(SocketInputStream.java:171)
java.net.SocketInputStream.read(SocketInputStream.java:141)
java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
java.io.BufferedInputStream.read(BufferedInputStream.java:345)
org.mariadb.jdbc.internal.io.input.StandardPacketInputStream.getPacketArray(StandardPacketInputStream.java:236)
org.mariadb.jdbc.internal.io.input.StandardPacketInputStream.getPacket(StandardPacketInputStream.java:207)
org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:1430)
org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:1410)
org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:222)
org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:355)
org.mariadb.jdbc.MariaDbStatement.execute(MariaDbStatement.java:412)
sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
com.glide.db.StatementWrapper.invoke(StatementWrapper.java:44)
com.sun.proxy.$Proxy20.execute(Unknown Source)
com.glide.db.DBI.executeStatement0(DBI.java:1132)
com.glide.db.DBI.executeStatement(DBI.java:1088)
com.glide.db.DBQueryExecutor.execute0(DBQueryExecutor.java:175)
com.glide.db.DBQueryExecutor.execute0(DBQueryExecutor.java:203)
com.glide.db.DBQueryExecutor.execute(DBQueryExecutor.java:49)
com.glide.db.DBQuery.executeAsResultSet0(DBQuery.java:355)
com.glide.db.DBQuery.executeAndReturnTable(DBQuery.java:332)
com.glide.db.DBAction.executeNormal(DBAction.java:256)
com.glide.db.DBAction.executeAndReturnException(DBAction.java:210)
com.glide.db.RDBMSQueryContext.executeQuery(RDBMSQueryContext.java:42)
com.glide.db.DBQuery.execute(DBQuery.java:2705)
com.glide.db.meta.Table.queryBasic(Table.java:338)
com.glide.db.meta.Table.query(Table.java:206)
com.glide.script.GlideRecordITable.query(GlideRecordITable.java:108)
com.snc.cmdb.BaselineCMDB.getRelatedRecords(BaselineCMDB.java:347)
com.snc.cmdb.BaselineCMDB.getRelation(BaselineCMDB.java:581)
com.snc.cmdb.BaselineCMDB.getRelations(BaselineCMDB.java:565)
com.snc.cmdb.BaselineCMDB.addRelations(BaselineCMDB.java:279)
com.snc.cmdb.BaselineCMDB.setBaseLine(BaselineCMDB.java:265)
com.snc.cmdb.BaselineCMDB.processCIs(BaselineCMDB.java:190)
com.snc.cmdb.BaselineCMDB.create(BaselineCMDB.java:152)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.mozilla.javascript.MemberBox.invoke(MemberBox.java:138)
org.mozilla.javascript.NativeJavaMethod.call(NativeJavaMethod.java:292)
org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2585)
org.mozilla.javascript.optimizer.OptRuntime.callProp0(OptRuntime.java:85)
org.mozilla.javascript.gen.sys_trigger_1abe5ebadb091410c1b2a0ae859619c3_2518._c_script_0(sys_trigger.1abe5ebadb091410c1b2a0ae859619c3:1)
...

Workaround

This problem is currently under review. You can contact ServiceNow Technical Support or subscribe to this Known Error article by clicking the Subscribe button at the top right of this form to be notified when more information will become available.

Following best practice would avoid these problems, however that is not documented. The following Tech Support article fills in some of the gaps:
KB0829681 CMDB Baseline Life-cycle, Best practices and Diff Formatter Troubleshooting



Related Problem: PRB1283806

Seen In

Dublin EA 8
Dublin Patch 1
Dublin Patch 6
Eureka Patch 12 Hot Fix 2
Eureka Patch 13 Hot Fix 2
Fuji Patch 7 Hot Fix 5
Geneva Patch 5
Geneva Patch 6 Hot Fix 2
SR - IRM - Audit Management - New York 2019 Q3
SR - IRM - GRC Profiles - Madrid 2019 Q2
SR - IRM - Policy and Compliance - Madrid 2019 Q2
SR - IRM - Risk Management - New York 2019 Q3
SR - ITOM - Fundamentals Istanbul Jakarta Kingston r1 - v5.99.6
SR - Security - Integration Framework - Madrid 2019 Q2
SR - Security - Support Common - Madrid 2019 Q2
SR - Security - Support Orchestration - Madrid 2019 Q2
SR - SIR - Security Incident Response - Madrid 2019 Q2
SR - SIR - Security Incident Response PA Content - New York 2019 Q3
SR - SIR - Security Incident Response UI Patch - London 2019 Q2 v.6.2.3
SR - SIR - Store SecOps Setup Assistant - Madrid 2019 Q2
SR - SIR - Store Threat Core - Madrid 2019 Q2
SR - SIR - Store Trusted Security Circles Client - New York 2019 Q3
SR - VR - Configuration Compliance - New York 2019 Q3
SR - VR - Qualys - New York 2019 Q3
SR - VR - Vulnerability Response - New York 2019 Q3
SR - VR - Vulnerability Response PA Content - Madrid 2019 Q2

Associated Community Threads

There is no data to report.

Article Information

Last Updated:2020-09-08 18:03:57
Published:2020-06-25