Description

Symptoms

After creating Incident, the escalation document sends the first notification, then never sends another notification. On-Call Rotation stopped sending more than one notification

Release

London Patch 1 Hot Fix 2

Cause

Reviewing node logs in Debug we can see after BR: 'mk Oncall Notify', BR: mkOnCallEscalation" triggered. These are custom Business Rules.

Reviewed Table "cmn_rota_escalation" and found escalation document.

XML document shows following:

<?xml version="1.0" encoding="UTF-8"?>
<escalations current_rota="778a331b957e61005d67c5f9f488ca38" group="81a6abf16f57590057d79b9eae3ee47b" primary_user_id="8edaef1a6fd01a040b3f687f8e3ee433" primary_user_name="Test User" processed_rota_list="">
<notifications>
<sent delay="0" member="5ff93f346fdd924457d79b9eae3ee4c2" notified="2018-12-06 02:47:36" rota="778a331b957e61005d67c5f9f488ca38" run_at="1544086056540" run_at_user_time="2018-12-06 02:47:36" type="rotation" user="8edaef1a6fd01a040b3f687f8e3ee433" user_name="Test User"/>
<pending delay="30" member="5ff93f346fdd924457d79b9eae3ee4c2" rota="778a331b957e61005d67c5f9f488ca38" run_at="1544086086540" run_at_user_time="2018-12-06 02:48:06" type="rotation" user="8edaef1a6fd01a040b3f687f8e3ee433" user_name="Test User"/>
<pending delay="60" member="5ff93f346fdd924457d79b9eae3ee4c2" rota="778a331b957e61005d67c5f9f488ca38" run_at="1544086116540" run_at_user_time="2018-12-06 02:48:36" type="rotation" user="8edaef1a6fd01a040b3f687f8e3ee433" user_name="Test User"/>
<pending delay="90" member="5ff93f346fdd924457d79b9eae3ee4c2" rota="778a331b957e61005d67c5f9f488ca38" run_at="1544086146540" run_at_user_time="2018-12-06 02:49:06" type="rotation" user="8edaef1a6fd01a040b3f687f8e3ee433" user_name="Test User"/>
</notifications>
</escalations>

The escalation document shows first notification sent after delay=0 successfully
However after delay="30", delay="60", delay="90" all showing Pending.

Resolution

Upgrade to LP2 as it includes PRB1296458 as fix. However, in this case customer was not ready to upgrade to LP2 hence provided following workaround.

NOTE: Notified customer that updates against 3 script includes need to be reverted back on upgrade so that they will continue to get fixes and enhancements.

Workaround Provided by Dev.

OnCallRotation:
===============

_checkForMoreRotas: function() {
this._escalationList.addRotaToList(this._rotaSysId);
var time = new GlideDateTime();
this._firstDelay = this._getFirstDelay(time);
if (this._rotaSysId) {
time.addSeconds(this._firstDelay);
this._getRotaNotificationsByTime(time, true);
}
var moreRotas = !this.isEmpty();
if (this._log.atLevel(GSLog.DEBUG))
this._log.debug("[_checkForMoreRotas] moreRotas: " + moreRotas);
return moreRotas;
},

/**
* Get the set of notifications for the current rota based on the rota's escalation type and rotation members
* gdt [GlideDateTime] the time point to get notifications for
* nullifyOverrideRoster [boolean]
*/
_getRotaNotificationsByTime: function(gdt, nullifyOverrideRoster) {
var gr = new GlideRecord('cmn_rota_roster');
gr.addQuery("rota", this._rotaSysId);
gr.addActiveQuery();
gr.query();
this._getRotaNotifications(gr, gdt, nullifyOverrideRoster);
},

================
OnCallEscalation
================

/**
* Load up the escalation information from the xml string
*/
fromXml: function(escalationXmlStr) {
this.clear();
if (!escalationXmlStr)
return;
var doc = GlideXMLUtil.parse(escalationXmlStr);
if (!doc)
return;
var root = doc.getDocumentElement();
this.groupSysId = root.getAttribute("group");
this.rotaSysId = root.getAttribute("current_rota") + "";
this.primaryUserSysId = root.getAttribute("primary_user_id");
this.primaryUserName = root.getAttribute("primary_user_name");
this.rotaList = [];
var rotaList = root.getAttribute("processed_rota_list").split(",");
for(var k = 0; k < rotaList.length; k++) {
this.rotaList.push(rotaList[k]);
}
var pending = GlideXMLUtil.selectNodes(root, "/escalations/notifications/" + OnCallEscalationEntrySNC.PENDING);
for (var i = 0, length = pending.getLength(); i < length; i++) {
var pendingEntry = new OnCallEscalationEntry();
pendingEntry.fromXml(pending.item(i));
var key = this.createKey(pendingEntry.getDelay());
this.entries[key] = pendingEntry;
this.entryKeys.push(key);
}
var sent = GlideXMLUtil.selectNodes(root, "/escalations/notifications/" + OnCallEscalationEntrySNC.SENT);
var sentLength = sent.getLength();
for (var j = 0; j < sentLength; j++) {
var sentEntry = new OnCallEscalationEntry();
sentEntry.fromXml(sent.item(j));
this.notifiedList.push(sentEntry);
if (j === 0)
this.firstNotifiedGdt = new GlideDateTime(sentEntry.getNotifiedTime());
}
// Start the iterator at the beginning of the list
this.gotoTop();
},

=====================
OnCallEscalationEntry
=====================

toXml: function(doc, parentElement) {
parentElement.appendChild(this._createElement(doc));
},

Additional Information

As a reminder, updates against 3 script includes need to be reverted back on upgrade so that customer will continue to get fixes and enhancements.

Notifications