Notifications

712 views

Description

When the WMI Collector service is running but not responding, the system tries to restart it. In some cases, the process hangs when trying to find the processes that are locking log4net.dll. This may slow down the MID Server to the point that it does not function because all worker threads are hanging.

The following example shows a sample thread dump.

"Worker-Standard:HorizontalDiscoveryProbe" #1729 daemon prio=5 os_prio=0 tid=0x45df3800 nid=0xf80 in Object.wait() [0x49a2e000]
2017/10/15 21:41:26 | java.lang.Thread.State: WAITING (on object monitor)
2017/10/15 21:41:26 | at java.lang.Object.wait(Native Method)
2017/10/15 21:41:26 | at java.lang.Thread.join(Thread.java:1245)
2017/10/15 21:41:26 | - locked <0x1f673070> (a com.snc.sw.context.support.proc.LocalCommandRunner$StreamGobbler)
2017/10/15 21:41:26 | at java.lang.Thread.join(Thread.java:1319)
2017/10/15 21:41:26 | at com.snc.sw.context.support.proc.LocalCommandRunner.executeCommandWithTimeout(LocalCommandRunner.java:129)
2017/10/15 21:41:26 | at com.snc.sw.context.support.proc.LocalCommandRunner.executeCommand(LocalCommandRunner.java:159)
2017/10/15 21:41:26 | at com.snc.sw.providers.wmi.WmiCollectorServiceStarter.killDllLockers(WmiCollectorServiceStarter.java:271)
2017/10/15 21:41:26 | at com.snc.sw.providers.wmi.WmiCollectorServiceStarter.stopService(WmiCollectorServiceStarter.java:315)
2017/10/15 21:41:26 | at com.snc.sw.providers.wmi.WmiCollectorServiceStarter.checkService(WmiCollectorServiceStarter.java:107)
2017/10/15 21:41:26 | at com.snc.sw.providers.wmi.WMIProviderImpl.send(WMIProviderImpl.java:725)
2017/10/15 21:41:26 | at com.snc.sw.providers.wmi.WMIProviderImpl.executeQuery(WMIProviderImpl.java:254)
2017/10/15 21:41:26 | at com.snc.sw.providers.wmi.WMIProviderImpl.executeQuery(WMIProviderImpl.java:1326)
2017/10/15 21:41:26 | at com.snc.sw.commands.wmi.ProcessListWmiCommand.getProcessList(ProcessListWmiCommand.java:61)
2017/10/15 21:41:26 | at com.snc.sw.commands.wmi.ProcessListWmiCommand.execute(ProcessListWmiCommand.java:41)
2017/10/15 21:41:26 | at com.snc.sw.commands.wmi.ProcessListWmiCommand.execute(ProcessListWmiCommand.java:28)
2017/10/15 21:41:26 | at com.snc.sw.commands.CommandsRunner.execute(CommandsRunner.java:83)
2017/10/15 21:41:26 | at com.snc.sw.commands.CommandsRunner.execute(CommandsRunner.java:40)
2017/10/15 21:41:26 | at com.snc.sw.commands.DiscoveryCommandDomainManagerImpl.getProcesses(DiscoveryCommandDomainManagerImpl.java:120)
2017/10/15 21:41:26 | at com.snc.sw.commands.FindProcessByPID.findProcess(FindProcessByPID.java:23)
2017/10/15 21:41:26 | at com.snc.sw.context.ExecutionContextImpl.initializeEntryPointProcess(ExecutionContextImpl.java:689)
2017/10/15 21:41:26 | at com.snc.sw.pattern.HorizontalDiscoveryPatternExecutor.getProcess(HorizontalDiscoveryPatternExecutor.java:92)
2017/10/15 21:41:26 | at com.snc.sw.pattern.HorizontalDiscoveryPatternExecutor.executeIdentifications(HorizontalDiscoveryPatternExecutor.java:116)
2017/10/15 21:41:26 | at com.snc.sw.pattern.HorizontalDiscoveryPatternExecutor.executePattern(HorizontalDiscoveryPatternExecutor.java:62)
2017/10/15 21:41:26 | at com.snc.sw.pattern.DefaultPatternExecutor.executePattern(DefaultPatternExecutor.java:26)
2017/10/15 21:41:26 | at com.service_now.mid.probe.HorizontalDiscoveryProbe.runPattern(HorizontalDiscoveryProbe.java:199)
2017/10/15 21:41:26 | at com.service_now.mid.probe.HorizontalDiscoveryProbe.runProbe(HorizontalDiscoveryProbe.java:88)
2017/10/15 21:41:27 | at com.service_now.mid.probe.ServiceWatchProbe.probe(ServiceWatchProbe.java:36)
2017/10/15 21:41:27 | at com.service_now.mid.probe.AProbe.process(AProbe.java:81)
2017/10/15 21:41:27 | at com.service_now.mid.queue_worker.AWorker.runWorker(AWorker.java:119)
2017/10/15 21:41:27 | at com.service_now.mid.queue_worker.AWorkerThread.run(AWorkerThread.java:20)
2017/10/15 21:41:27 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
2017/10/15 21:41:27 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
2017/10/15 21:41:27 | at java.lang.Thread.run(Thread.java:745)

Steps to Reproduce

 

Watch for the following symptoms:

  • Slow Discovery or Discovery timeout 
  • MID server being unresponsive

Workaround

 

Note – If using the process introduced in Jakarta that doesn't have any corresponding probes, just disable the pattern.

For other releases, disable the patterns and use probes.

  1. Disable the pattern.

    1. Go to Discovery Definition > Processes, and look up the pattern name.

    2. On the process page, scroll down to the Triggers Probes. Set the patterns to False and all of the related probes to True.

  2. Disable the pattern.

    1. Go to the probe/pattern page and uncheck the Active checkbox.

    2. Click Update.

  3. Enable the probe.

    1. Go to the probe/pattern page and check the Active checkbox.

    2. Click the Update button.

      The pattern is disabled and all related probes are enabled.

     


Related Problem: PRB1234649

Seen In

There is no data to report.

Fixed In

Jakarta Patch 6
Kingston Patch 1
London

Associated Community Threads

There is no data to report.

Article Information

Last Updated:2018-03-07 05:21:54
Published:2017-11-07