Users will receive notifications when they have subscribed to updates to the Configuration Items (CI) that they are interested in.
The way the platform sends the notifications to the subscribed users is as follows:
- when there is an update to a task record (such incident/problem/change) which has the CI of interest as an affected CI or is a related CI of the affected CI, the out of the box business rule - Affected ci notifications will get triggered
- as this business rule is an ASYNC business rule, an ASYNC: Affected ci notifications job will be created and executed.
- this job then creates an event: ci.affected which, when processed, will then send an email based on the Notification: CI affected template
Generally, the ASYNC: Affected ci notifications job runs quite fast (less than a few seconds). However, in situations where the CI of interest has a large number of relations when walked 5 levels down (from child to parent) or if there are cyclic references (for example, a child is a parent of itself), the ASYNC: Affected ci notifications job may utilize a significant amount of memory. This may also cause excessive garbage collection activities. The response time of these jobs may be high as well (in minutes).
When there are a number of such jobs running, users may experience performance degradation. If there are many updates to the task records, the schedulers may also be overloaded with ASYNC: Affected ci notifications jobs which may also cause a delay in processing other jobs.
The performance of the instance should return to normal when the ASYNC: Affected ci notifications job(s) complete.
Steps to Reproduce
This would depend on the complexity and depth of your CI relationships.
If you navigate to System Logs > Transactions (Background) and filter for URL starts with JOB: ASYNC: Affected ci notification job and Created on Today (or the required date) and sort the Response time column in descending order (Sort: z-a), if the following is true, this is an indication that this KB will be applicable.
- the response time of some of the jobs exceed 5 minutes
- there a number of ASYNC: Affected ci notification jobs which are having Response time in minutes and Created at around the same time
- you have encountered slowness around the time when the jobs were running
To mitigate the high memory usage, please consider the following 3 options:
Option 1: Modify the existing business rule and system property
a. Lower the value of the system property: NotifyAffectedCI.max_rel_level. The out-of-box value is 5. Please refer to the documentation for Additional email properties for more information regarding NotifyAffectedCI.max_rel_level.
b. Modify the existing condition of the business rule: Affected ci notifications to the following which will prevent the business rule from being triggered for child task records.
If the schedulers are flooded by ASYNC: Affected ci notifications jobs and are preventing other jobs to run, you can consider modifying Priority value to 200. This will allow other jobs to take priority over these ASYNC: Affected ci notifications jobs.
Option 2: Import the attached xml file for the updated business rule: Affected ci notifications (which is available in a fixed release).
The difference between Option 1 and this option is that the system property: NotifyAffectedCI.max_rel_level (OOB value of 5) need not be modified. The code will perform a 5 level walk of all CIs based on the affected CI.
Option 3: Upgrade to a fixed release
Related Problem: PRB1272847