Overview

---

The identification and reconciliation engine (IRE) performs identification relying on identification rules. Based on the data passed to the IRE and the identification rules, the identification engine should find the correct configuration item (CI) in the configuration management database (CMDB). The IRE should be used when updating or creating CIs. Using the IRE leads to a consistent CMDB with no duplicates. 

Discovery patterns collect data from CIs and update the CMDB. However, it is possible that duplicate records within the same payload can be passed to the IRE. DUPLICATE_PAYLOAD_RECORDS error will be thrown when duplicate records are passed on the same payload to the IRE. DUPLICATE_PAYLOAD_RECORDS can be removed from a payload via steps in the pattern. This KB outlines a simple method of removing such duplicates. 

Example error:

DUPLICATE_LOOKUP_PAYLOAD Found duplicate Lookup items (0 and 1) in the payload index 22 using fields serial_number,serial_number_type: no thrown error 

Remove Duplicates

---

 A simple way to remove duplicates is to use the DuplicateRemover script. The steps would be:

1. Determine the table in the payload with the duplicates. In the example above the table was cmdb_serial_number, the fields mentioned in the error will help in determining the correct table.
2. Add a step to the pattern where operation = "Set Parameter Value".
3. Fill out the Value with the script to remove the duplicates such as:
   

   EVAL(
   javascript:
   // This table will be the new table without the duplicates
   var tableWithoutDuplicates = '';
   
   // Pass the table_to_remove_duplicates_from, and the columns to use for identification, to the DuplicateRemover
   tableWithoutDuplicates =  DuplicateRemover.removeDuplicates(${table_to_remove_duplicates_from},["column_1","column_2"]);
   
   // Replace original table with new table duplicates free
   CTX.setAttribute("table_to_remove_duplicates_from",  tableWithoutDuplicates);
   );

4.  The step should now look as such:
   
5. Save the step and publish the pattern.

Notes:

• In most cases, it is best to add the step to remove duplicates right after the table with the duplicates is created.
• The name in the step image above, $temp_unique_service, can be any name that does not match a variable already in use in the pattern.
• The script will have to be updated to contain the correct table name and columns.
• The columns mentioned in the script, column_1 and column_2, are what the script will use to determine what a duplicate is and should be replaced accordingly. If the rows for an example CI identifiable by only a column, lets say serial_number, then only one column would be necessary as such:
  

  tableWithoutDuplicates =  DuplicateRemover.removeDuplicates(${cmdb_serial_number},["serial_number"]);

Additional Information

---

The following documents provide good information in troubleshooting and understanding issues regarding the IRE:

• Identification and Reconciliation Components and Process
• Identification Engine Error Messages