OutOfMemory due to the killSwitch() observer objects that would not be deleted. This does not cause an outage or an issue with the instance while it is on Dublin.

NOTE: This issue is similar but totally distinct from PRB602599 "A workflow mutex is obtained but not released... causing instance performance degradations". The underlying commonality in both cases is that wf_command records try to fire against wf_context records that cancelled, finished, or faulted and can cause the same symptoms.

Customers will experience the following:
- Slow performance
- OutOfMemory errors
- Unexpected restarts/logouts/re-authentication resulting in loss of form data

In Eureka we introduced a background process called killSwitch observers. These observers are added to monitor the completion of workflows. If the command queue (wf_command table) contains commands pointed toward completed or canceled workflows (wf_context table) then the system will keep adding additional observers to the killSwitch. The observers persist as objects in Java memory and retain memory. As more observers are created but not destroyed a memory leak situation ensues.

Because the killSwitch observers were only added in Eureka and the "orphaned" commands will no longer be generated in Eureka, the workaround only needs to be applied once. This issue will only affect instances running Eureka versions prior to Eureka Patch 1 Hot Fix 2. For example, the issue will not occur on Eureka Patch 2.

Steps to Reproduce

We were able to replicate the out-of-memory in testing by executing this script in Scripts – Background:

new Workflow().broadcastEvent('<sys_id of a completed context here>', 'some_event');

The workflow engine then leaks a CommandQueue.

To observe this, we had to repeat this process many times and watch memory being consumed (for example, using jMetere). Alternatively, add logging (or breakpoints) to CommandManager.CommandQueue where we store a reference to CommandQueue in killSwitch and where we remove it. When you use CommandManager.CommandQueue and run this test, you will see that we save CommandQueue but never release it.


 Delete the records manually.

  1. Navigate to the following URL (substituting your instance name for <myinstance>):


  2. Scroll to the bottom of the list and click the check box to select all.
  3. From the Actions on selected rows drop-down menu, select Delete.
  4. If a popup appears asking if you want to perform the action against all records matching the filter, as opposed to just the ones currently displaying on the page, click Yes.

Alternatively, you may wish to run this script:

var gr = new GlideRecord('wf_command');

var qc = gr.addQuery('context.state', 'finished');

qc.addOrCondition('context.state', 'faulted');

qc.addOrCondition('context.state', 'cancelled');





while( gr.next() ) {


    " Command Created : " + gr. sys_created_on + 

    " Context Ended : " + gr.context.ended + 

    " Context Last Updated Date : " + gr.context.sys_updated_on + 

    " Created On " + gr.context.sys_created_on + 

    " || Context Sys_id : " + gr.context.sys_id + 

    " || Command : " + gr.command );



Related Problem: PRB601828

Seen In

Calgary Patch 2 Hot Fix 5
Calgary Patch 4
Dublin EA 8
Dublin EA 8 Hot Fix 1
Dublin EA 9
Dublin Patch 1 Hot Fix 4
Dublin Patch 3 Hot Fix 1
Dublin Patch 5
Dublin Patch 7 Hot Fix 2
Dublin Patch 7 Hot Fix 5
Dublin Patch 8 Hot Fix 3
Eureka Patch 1
Eureka Patch 1 Hot Fix 2
Eureka Patch 10
Eureka Patch 2
Eureka Patch 3 Hot Fix 1
Eureka Patch 4
Eureka Patch 4 Hot Fix 1
Eureka Patch 5
Eureka Patch 5 Hot Fix 1
Eureka Patch 6
Eureka Patch 7
Eureka Patch 8
Eureka Patch 9 Hot Fix 1
Eureka Patch 9 Hot Fix 4
Fuji Patch 8

Fixed In

Eureka Patch 1 Hot Fix 2
Eureka Patch 2

Associated Community Threads

Community ThreadTitleResponsesUpvotes
https://community.servicenow.com/message/772199Out of memory exception error in SNOW instance4

Article Information

Last Updated:2016-07-25 12:35:39