Notifications

79 views

Description

Sometimes insertion to discovery_device_duplicate_ips fails due to unique key violation, and once it happens the ECC "input" record stays stuck on "ready". As a side effect, the corresponding discovery status never completes, or gets Cancelled at max runtime.

One of these stack traces is shown in the instance localhost logs when the sensor running for the ecc queue input fails:.

This issue doesn't consistently occur.

You need a couple of input records from the same IP Address in order to reproduce this error:

FAILED TRYING TO EXECUTE ON CONNECTION glide.16 (connpid=119398): INSERT INTO discovery_device_duplicate_ips (`cmdb_ci`,`sys_mod_count`,`source`,`sys_updated_on`,`sys_domain_path`,`sys_id`,`sys_updated_by`,`sys_created_on`,`sys_domain`,`state`,`sys_created_by`,`status`,`ecc_queue`) VALUES('55e6c42bdb8073000bc50726ca96198e',0,'10.196.39.217','2019-03-31 14:56:22','/','53281959dbe833000bc50726ca9619d0','mid_server','2019-03-31 14:56:22','global','skip','mid_server','f8f75555dbe833004ba0df8b4b961916','1a289955dbe833004ba0df8b4b961964') /* popsny1003, gs:glide.scheduler.worker.0, tx:d3281959dbe833000bc50726ca9619ce */ 
Unique Key violation detected by database (Duplicate entry '10.196.39.217-f8f75555dbe833004ba0df8b4b961916' for key 'source')
: java.sql.SQLIntegrityConstraintViolationException: Duplicate entry '10.196.39.217-f8f75555dbe833004ba0df8b4b961916' for key 'source': org.mariadb.jdbc.internal.SQLExceptionMapper.get(SQLExceptionMapper.java:132)
org.mariadb.jdbc.internal.SQLExceptionMapper.throwException(SQLExceptionMapper.java:106)
org.mariadb.jdbc.MySQLStatement.executeQueryEpilog(MySQLStatement.java:274)
org.mariadb.jdbc.MySQLStatement.execute(MySQLStatement.java:302)
org.mariadb.jdbc.MySQLStatement.execute(MySQLStatement.java:393)
sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
com.glide.db.StatementWrapper.invoke(StatementWrapper.java:40)
com.sun.proxy.$Proxy12.execute(Unknown Source)
com.glide.db.DBI.executeStatement0(DBI.java:1102)
com.glide.db.DBI.executeStatement(DBI.java:1058)
com.glide.db.DBI.executeStatement(DBI.java:1011)
com.glide.db.DBAction.executeAsResultSet(DBAction.java:284)
com.glide.db.DBCompositeAction.executeAsResultSet(DBCompositeAction.java:178)
com.glide.db.DBCompositeAction.executeChunk(DBCompositeAction.java:138)
com.glide.db.DBCompositeAction.executeAsResultSet0(DBCompositeAction.java:100)
com.glide.db.DBAction.executeAndReturnTable(DBAction.java:247)
com.glide.db.DBAction.executeNormal(DBAction.java:236)
com.glide.db.DBAction.executeAndReturnException(DBAction.java:190)
com.glide.script.GlideRecordITable.insert(GlideRecordITable.java:144)
com.glide.script.GlideRecord.insert(GlideRecord.java:4847)
com.glide.script.GlideRecord.insert(GlideRecord.java:4764)
com.snc.discovery.logging.DeviceDuplicateIp.createDeviceDupIp(DeviceDuplicateIp.java:360)
com.snc.discovery.logging.DeviceDuplicateIp.isDuplicate(DeviceDuplicateIp.java:179)
com.snc.discovery.SensorProcessor.process(SensorProcessor.java:182)...

Or multiple IP addresses of the same device to be processed simultaneously for this to happen:

2019-05-10 09:20:35 (093) worker.0 worker.0 txid=4f0b2f4adb99 WARNING *** WARNING *** Attempted create of mutex DeviceDuplicateIp_b8ea230a1b55bf4075010dc8cd4bcbf8_343d3a7c1b857f0075010dc8cd4bcbe5 failed : com.glide.db.GlideSQLException: FAILED TRYING TO EXECUTE ON CONNECTION 29: INSERT INTO sys_mutex (`sys_id`,`sys_updated_by`,`system_id`,`sys_created_on`,`name`,`sys_mod_count`,`sys_updated_on`,`sys_created_by`) VALUES('070b2f4adb99ff40b1fdf38139961902','midserver','available','2019-05-10 16:20:35','DeviceDuplicateIp_b8ea230a1b55bf4075010dc8cd4bcbf8_343d3a7c1b857f0075010dc8cd4bcbe5',0,'2019-05-10 16:20:35','midserver') /* amicadev001, gs:glide.scheduler.worker.0, tx:4f0b2f4adb99ff40b1fdf38139961900 */ 
Unique Key violation detected by database (Duplicate entry 'DeviceDuplicateIp_b8ea230a1b55bf4075010dc8cd4bcbf8_343d3a7c1b857' for key 'name')
2019-05-10 09:20:35 (094) worker.0 worker.0 txid=4f0b2f4adb99 Spin waiting for mutex DeviceDuplicateIp_b8ea230a1b55bf4075010dc8cd4bcbf8_343d3a7c1b857f0075010dc8cd4bcbe5 to be freed

 

Steps to Reproduce

As this issue doesn't occur consistently, you may need to synthesize this scenario.
A good chance to encounter this is when you execute the All Applications discovery schedule and let it run.
You can use the following instance for that purpose: popsny1 (feel free to hop in). Assuming that it's still there, you could also execute the scheduled discovery named "Sprint 190".
Also, if you look at the system log errors, you may see a handful of such errors:
https://popsny1.service-now.com/syslog_list.do?sysparm_query=level%3D2%5EGOTOmessageLIKEINSERT%20INTO%20discovery_device_duplicate_ips&sysparm_first_row=1&sysparm_view=

Alternatively, set up a Discovery schedule with an IP range that includes only all the IPs of a large switch. e.g. a switch with >!0 IPs, that end up with some having the SNMP - Classify sensor running at the same time.

Workaround

This problem is currently under review. You can contact ServiceNow Technical Support or subscribe to this Known Error article by clicking the Subscribe button at the top right of this form to be notified when more information will become available.

In the case of Multiple IPs for the same device being probed simultaneously, It would be possible to exclude all but one of the IPs of that specific device from the Discovery Schedule's IP ranges to avoid the problem.


Related Problem: PRB1334206

Seen In

There is no data to report.

Intended Fix Version

Orlando

Safe Harbor Statement

This "Intended Fix Version" information is meant to outline ServiceNow's general product direction and should not be relied upon in making a purchasing decision. The information provided here is for information purposes only and may not be incorporated into any contract. It is not a commitment, promise, or legal obligation to deliver any material, code, or functionality. The development, release, and timing of any features or functionality described for our products remains at ServiceNow's sole discretion.

Associated Community Threads

There is no data to report.

Article Information

Last Updated:2019-07-11 16:50:03
Published:2019-05-24