8 views

Symptoms


MID Server memory usage trends up over time eventually exhausting the configured heap size and run into OutOfMemory errors. Once memory leak is severe enough it might not be able to respond to heartbeat probes from the Service Now instance and the instance will mark the MID Server as down. 

Release


Jakarta and Newer.

 

Cause


Heapdump analysis shows there are memory leak due to H2 Database connection leak. H2 is an in-memory Database used by Discover Patterns and Operational Intelligence. There is a bug where if an exception occur while executing a SQL query, the connection is not cleaned up and accumulates. Eventually this leads to memory exhaustion. Below is screenshot of the leaked H2 connection objects when performing Heap analysis:

 

Resolution


This is fixed in PRB1293425 for Kingston Patch 8.

Additional Information


In MID Server's agent0.log[0-9], look for the "LogStatusMonitor" log entries. These are MID Server health checks. If you look at 'used' memory, it trends up over time even when it's not processing any probes.

 

09/01/18 23:23:50 (616) LogStatusMonitor.60 stats threads: 83, memory max: 910.0mb, allocated: 496.0mb, used: 89.0mb, standard.queued: 0 probes, standard.processing: 0 probes, expedited.queued: 0 probes, expedited.processing: 0 probes, interactive.queued: 0 probes, interactive.processing: 0 probes

09/02/18 23:23:50 (616) LogStatusMonitor.60 stats threads: 83, memory max: 910.0mb, allocated: 712.0mb, used: 243.0mb, standard.queued: 0 probes, standard.processing: 0 probes, expedited.queued: 0 probes, expedited.processing: 0 probes, interactive.queued: 0 probes, interactive.processing: 0 probes

09/03/18 22:24:50 (327) LogStatusMonitor.60 stats threads: 83, memory max: 910.0mb, allocated: 910.0mb, used: 498.0mb, standard.queued: 0 probes, standard.processing: 0 probes, expedited.queued: 0 probes, expedited.processing: 0 probes, interactive.queued: 0 probes, interactive.processing: 0 probes

09/04/18 23:57:50 (567) LogStatusMonitor.60 stats threads: 87, memory max: 910.0mb, allocated: 910.0mb, used: 732.0mb, standard.queued: 0 probes, standard.processing: 0 probes, expedited.queued: 0 probes, expedited.processing: 0 probes, interactive.queued: 0 probes, interactive.processing: 0 probes

09/05/18 10:28:18 (491) LogStatusMonitor.60 stats threads: 83, memory max: 910.0mb, allocated: 910.0mb, used: 877.0mb, standard.queued: 0 probes, standard.processing: 0 probes, expedited.queued: 0 probes, expedited.processing: 0 probes, interactive.queued: 0 probes, interactive.processing: 0 probes

 

Article Information

Last Updated:2018-09-14 16:59:00
Published:2018-09-14