Overview

 A simple way to remove duplicates is to use the DuplicateRemover script. The steps would be:

1. Determine the table in the payload with the duplicates. In the example above the table was cmdb_serial_number, the fields mentioned in the error will help in determining the correct table.
2. Add a step to the pattern where operation = "Set Parameter Value".
3. Fill out the Value with the script to remove the duplicates such as:
   

   EVAL(
   javascript:
   // This table will be the new table without the duplicates
   var tableWithoutDuplicates = '';
   
   // Pass the table_to_remove_duplicates_from, and the columns to use for identification, to the DuplicateRemover
   tableWithoutDuplicates =  DuplicateRemover.removeDuplicates(${table_to_remove_duplicates_from},["column_1","column_2"]);
   
   // Replace original table with new table duplicates free
   CTX.setAttribute("table_to_remove_duplicates_from",  tableWithoutDuplicates);
   );

4.  The step should now look as such:
   
5. Save the step and publish the pattern.

Notes:

• In most cases, it is best to add the step to remove duplicates right after the table with the duplicates is created.
• The name in the step image above, $temp_unique_service, can be any name that does not match a variable already in use in the pattern.
• The script will have to be updated to contain the correct table name and columns.
• The columns mentioned in the script, column_1, and column_2, are what the script will use to determine what a duplicate is and should be replaced accordingly. If the rows for an example CI identifiable by only a column, let's say serial_number, then only one column would be necessary as such:
  

  tableWithoutDuplicates =  DuplicateRemover.removeDuplicates(${cmdb_serial_number},["serial_number"]);

The following documents provide good information in troubleshooting and understanding issues regarding the IRE:

• Identification and Reconciliation Components and Process
• Identification Engine Error Messages