Symptoms

After creating Incident, the escalation document sends the first notification, then never sends another notification. On-Call Rotation stopped sending more than one notification

Release

London Patch 1 Hot Fix 2

Cause

Reviewing node logs in Debug we can see after BR: 'mk Oncall Notify', BR: mkOnCallEscalation" triggered. These are custom Business Rules.

Reviewed Table "cmn_rota_escalation" and found escalation document.

XML document shows following:

<?xml version="1.0" encoding="UTF-8"?> 
<escalations current_rota="778a331b957e61005d67c5f9f488ca38" group="81a6abf16f57590057d79b9eae3ee47b" primary_user_id="8edaef1a6fd01a040b3f687f8e3ee433" primary_user_name="Test User" processed_rota_list=""> 
<notifications> 
<sent delay="0" member="5ff93f346fdd924457d79b9eae3ee4c2" notified="2018-12-06 02:47:36" rota="778a331b957e61005d67c5f9f488ca38" run_at="1544086056540" run_at_user_time="2018-12-06 02:47:36" type="rotation" user="8edaef1a6fd01a040b3f687f8e3ee433" user_name="Test User"/> 
<pending delay="30" member="5ff93f346fdd924457d79b9eae3ee4c2" rota="778a331b957e61005d67c5f9f488ca38" run_at="1544086086540" run_at_user_time="2018-12-06 02:48:06" type="rotation" user="8edaef1a6fd01a040b3f687f8e3ee433" user_name="Test User"/> 
<pending delay="60" member="5ff93f346fdd924457d79b9eae3ee4c2" rota="778a331b957e61005d67c5f9f488ca38" run_at="1544086116540" run_at_user_time="2018-12-06 02:48:36" type="rotation" user="8edaef1a6fd01a040b3f687f8e3ee433" user_name="Test User"/> 
<pending delay="90" member="5ff93f346fdd924457d79b9eae3ee4c2" rota="778a331b957e61005d67c5f9f488ca38" run_at="1544086146540" run_at_user_time="2018-12-06 02:49:06" type="rotation" user="8edaef1a6fd01a040b3f687f8e3ee433" user_name="Test User"/> 
</notifications> 
</escalations> 

The escalation document shows first notification sent after delay=0 successfully
However after delay="30", delay="60", delay="90" all showing Pending. 

Resolution

Upgrade to LP2 as it includes PRB1296458 as fix. However, in this case customer was not ready to upgrade to LP2 hence provided following workaround.

NOTE: Notified customer that updates against 3 script includes need to be reverted back on upgrade so that they will continue to get fixes and enhancements. 

Workaround Provided by  Dev.

OnCallRotation: 
=============== 

_checkForMoreRotas: function() { 
this._escalationList.addRotaToList(this._rotaSysId); 
var time = new GlideDateTime(); 
this._firstDelay = this._getFirstDelay(time); 
if (this._rotaSysId) { 
time.addSeconds(this._firstDelay); 
this._getRotaNotificationsByTime(time, true); 
} 
var moreRotas = !this.isEmpty(); 
if (this._log.atLevel(GSLog.DEBUG)) 
this._log.debug("[_checkForMoreRotas] moreRotas: " + moreRotas); 
return moreRotas; 
}, 


/** 
* Get the set of notifications for the current rota based on the rota's escalation type and rotation members 
* gdt [GlideDateTime] the time point to get notifications for 
* nullifyOverrideRoster [boolean] 
*/ 
_getRotaNotificationsByTime: function(gdt, nullifyOverrideRoster) { 
var gr = new GlideRecord('cmn_rota_roster'); 
gr.addQuery("rota", this._rotaSysId); 
gr.addActiveQuery(); 
gr.query(); 
this._getRotaNotifications(gr, gdt, nullifyOverrideRoster); 
}, 

================ 
OnCallEscalation 
================ 

/** 
* Load up the escalation information from the xml string 
*/ 
fromXml: function(escalationXmlStr) { 
this.clear(); 
if (!escalationXmlStr) 
return; 
var doc = GlideXMLUtil.parse(escalationXmlStr); 
if (!doc) 
return; 
var root = doc.getDocumentElement(); 
this.groupSysId = root.getAttribute("group"); 
this.rotaSysId = root.getAttribute("current_rota") + ""; 
this.primaryUserSysId = root.getAttribute("primary_user_id"); 
this.primaryUserName = root.getAttribute("primary_user_name"); 
this.rotaList = []; 
var rotaList = root.getAttribute("processed_rota_list").split(","); 
for(var k = 0; k < rotaList.length; k++) { 
this.rotaList.push(rotaList[k]); 
} 
var pending = GlideXMLUtil.selectNodes(root, "/escalations/notifications/" + OnCallEscalationEntrySNC.PENDING); 
for (var i = 0, length = pending.getLength(); i < length; i++) { 
var pendingEntry = new OnCallEscalationEntry(); 
pendingEntry.fromXml(pending.item(i)); 
var key = this.createKey(pendingEntry.getDelay()); 
this.entries[key] = pendingEntry; 
this.entryKeys.push(key); 
} 
var sent = GlideXMLUtil.selectNodes(root, "/escalations/notifications/" + OnCallEscalationEntrySNC.SENT); 
var sentLength = sent.getLength(); 
for (var j = 0; j < sentLength; j++) { 
var sentEntry = new OnCallEscalationEntry(); 
sentEntry.fromXml(sent.item(j)); 
this.notifiedList.push(sentEntry); 
if (j === 0) 
this.firstNotifiedGdt = new GlideDateTime(sentEntry.getNotifiedTime()); 
} 
// Start the iterator at the beginning of the list 
this.gotoTop(); 
}, 


===================== 
OnCallEscalationEntry 
===================== 

toXml: function(doc, parentElement) { 
parentElement.appendChild(this._createElement(doc)); 
},

Additional Information

As a reminder, updates against 3 script includes need to be reverted back on upgrade so that customer will continue to get fixes and enhancements.