Description
Symptoms
Issue: Events stuck in ready state and the scheduled jobs are stuck waiting for a response. The stack trace is as follows waiting for the RestResponse.
main,glide.scheduler.worker.0,4,Event Management - process events - 2 (18760340 ms)
java.lang.Thread.sleep(Native Method)
com.glide.ecc.ECCResponsePoller.poll(ECCResponsePoller.java:50)
com.glide.rest.outbound.ecc.ECCRESTResponse.fetchAndProcessEccResponse(ECCRESTResponse.java:232)
com.glide.rest.outbound.ecc.ECCRESTResponse.getBody(ECCRESTResponse.java:124)
com.glide.rest.outbound.scriptable.ScriptableRESTResponse.jsFunction_getBody(ScriptableRESTResponse.java:82)
sun.reflect.GeneratedMethodAccessor695.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.mozilla.javascript.MemberBox.invoke(MemberBox.java:138)
org.mozilla.javascript.FunctionObject.doInvoke(FunctionObject.java:670)
org.mozilla.javascript.FunctionObject.call(FunctionObject.java:614)
org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2582)
org.mozilla.javascript.optimizer.OptRuntime.callProp0(OptRuntime.java:85)
org.mozilla.javascript.gen.sys_script_2cc85296dbe9b3c087aa34f668961901_script_561._c_executeRule_1(sys_script.2cc85296dbe9b3c087aa34f668961901.script:12)
org.mozilla.javascript.gen.sys_script_2cc85296dbe9b3c087aa34f668961901_script_561.call(sys_script.2cc85296dbe9b3c087aa34f668961901.script)
org.mozilla.javascript.ScriptRuntime.doCall2(ScriptRuntime.java:2650)
org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2590)
org.mozilla.javascript.optimizer.OptRuntime.call2(OptRuntime.java:42)
org.mozilla.javascript.gen.sys_script_2cc85296dbe9b3c087aa34f668961901_script_561._c_script_0(sys_script.2cc85296dbe9b3c087aa34f668961901.script:1)
org.mozilla.javascript.gen.sys_script_2cc85296dbe9b3c087aa34f668961901_script_561.call(sys_script.2cc85296dbe9b3c087aa34f668961901.script)
org.mozilla.javascript.gen.sys_script_2cc85296dbe9b3c087aa34f668961901_script_561.exec(sys_script.2cc85296dbe9b3c087aa34f668961901.script)
com.glide.script.ScriptEvaluator.execute(ScriptEvaluator.java:279)
com.glide.script.ScriptEvaluator.evaluateString(ScriptEvaluator.java:118)
com.glide.script.ScriptEvaluator.evaluateString(ScriptEvaluator.java:82)
com.glide.script.fencing.GlideScopedEvaluator.evaluateScript(GlideScopedEvaluator.java:309)
com.glide.script.fencing.GlideScopedEvaluator.evaluateScript(GlideScopedEvaluator.java:214)
com.glide.script.fencing.GlideScopedEvaluator.evaluateScript(GlideScopedEvaluator.java:201)
com.glide.script.BusinessRule.runScript(BusinessRule.java:403)
com.glide.script.BusinessRule.runSyncBusinessRule(BusinessRule.java:293)
com.glide.script.BusinessRule.runBusinessRule(BusinessRule.java:204)
com.glide.script.BusinessRule.run(BusinessRule.java:180)
com.glide.script.GlideRecordBusinessRules.run(GlideRecordBusinessRules.java:95)
com.glide.script.GlideRecord.runScripts(GlideRecord.java:1348)
com.glide.script.GlideRecord.runScriptsAndEngines(GlideRecord.java:1282)
com.glide.script.GlideRecord.updateWithoutResettingOperation(GlideRecord.java:4437)
com.glide.script.GlideRecord.update(GlideRecord.java:4365)
com.glide.script.GlideRecord.update(GlideRecord.java:4284)
com.glideapp.itom.snac.common.AlertState.updateAlert(AlertState.java:613)
com.glideapp.itom.snac.common.AlertState$2.execute(AlertState.java:116)
com.glideapp.itom.snac.processor.EventTransformationHandler.createOrUpdateAlert(EventTransformationHandler.java:119)
Release
All releases
Cause
- This can be caused if there is an insert/update business rules that sends out a REST call to an endpoint.
- As everything is done as part of a same schedule worker, it will wait until the response for this REST call is returned.
- This will cause backlog of Jobs and add additional delay in processing events.
Resolution
- It is not recommended to have business rule on em_event table as it adds additional delay.
- If you are writing a business rule to send a REST call, make sure it is an ASYNC business rules so that a new sys_trigger record is created.
- Make sure you define a time out on the REST call so that there is no delay and scheduled workers are freed up