Issue
This article aims to explain the CI matching process from Qualys Integration.
This article might answer some of these common questions:
1) When there are duplicate CI in the system, the VIT that is being created does not always select the best CI. How to improve it?
2) What is the difference between "Qualys CI" (sn_vul_qualys_ci) table and "UnMatched CI" (sn_sec_cmn_unmatched_ci) table and when are they used?
3) What is the difference between "CI Lookup Rules" (sn_sec_cmn_ci_lookup_rules) and "CI Identifier Rules"(sn_sec_cmn_ci_identifier_rule) ?
Release
London, Madrid
Resolution
To understand the CI matching process, the 'Discovered Item' should be understood at first
Prior to London, the Qualys integration used Qualys fields to match to CMDB records. If it didn’t find a record with the specified Qualys id, it will execute the lookup rules to find existing records without a Qualys ID. If it doesn’t find one, or the host already has a different Qualys ID, a new “Qualys CI” (sn_vul_qualys_ci) record will be inserted.
In London, CI matching method has been improved. Instead of using a field on the CI table, a new table called “Discovered Items” (sn_sec_cmn_src_ci) has been introduced. This allows multiple Qualys Ids to map to a single CI. It persists the Qualys ID in the “Source ID” field and the “Configuration Item” field references the CI.
The first step in the record matching process is that the system searches by Qualys ID in the "Discovered Items" table and if a record exists in the Discovered Items table, it will create a vulnerable item using that CI
If no “Discovered Items” record exists, it will execute the CI Lookup Rules in the table “CI Lookup Rules” (sn_sec_cmn_ci_lookup_rules). If it finds a match using any of these rules, it will create a new “Discovered Items” record. If it doesn’t find a match, it will create a new record in the “Unmatched CIs” table (sn_sec_cmn_unmatched_ci), which extends cmdb_ci table. After creating the 'Unmatched CI' record, a new 'Discovered Item' record is also created associating the unmatched CI with the Qualys ID so that the next time for the same Qualys ID, a match will be found in the first step, in the 'Discovered Items' table, and the CI lookup rules will not be executed.
For the issue related to the "best CI not being matched", this might be the cause. There might be a "Discovered Item" record created (which would have been a duplicate 'Unmatched CIs' record that was created due to improper CI lookup rules in the first place), that gets returned during the first step and the CI lookup rule is never being called.
So to avoid the issue, please remove the Duplicate CIs records that was already inserted, using the CMDB De-duplication wizard.
https://docs.servicenow.com/csh?topicname=de-duplication-tasks.html&version=latest
The one caveat to using that is before trying to de-dup vulnerability-related issues, two additional tables need to be added to the Related Items list: sn_sec_cmn_src_ci and sn_vul_vulnerable_item. That way when duplicates are removed or re-mapped, all their appropriate references in the VR records will get updated as well.
It should also be noted that the "CI Lookup Rules” (sn_sec_cmn_ci_lookup_rules) is used for CI matching starting from London. Previously, the other table "CI Identifier Rule" (sn_sec_cmn_ci_identifier_rule) was being used which is NOT used anymore.
This can be validated by following the code in the system:
During the import, the script include "QualysHostImportReportProcessor" calls another script include named "ImportHost" which would first check if there is a "Discovered Item" record and if NOT, then it would call another script include named "CIIdenitify" where we are querying the "CI Lookup Rules" (sn_sec_cmn_ci_lookup_rule) table to see if there are any CIs that matches the condition defined in the CI Lookup rules.
