415 views

Description

When the WMI Collector service is running but not responding, the system tries to restart it. In some cases, the process hangs when trying to find the processes that are locking log4net.dll. This may slow down the MID Server to the point that it does not function because all worker threads are hanging.

The following example shows a sample thread dump.

"Worker-Standard:HorizontalDiscoveryProbe" #1729 daemon prio=5 os_prio=0 tid=0x45df3800 nid=0xf80 in Object.wait() [0x49a2e000]
2017/10/15 21:41:26 | java.lang.Thread.State: WAITING (on object monitor)
2017/10/15 21:41:26 | at java.lang.Object.wait(Native Method)
2017/10/15 21:41:26 | at java.lang.Thread.join(Thread.java:1245)
2017/10/15 21:41:26 | - locked <0x1f673070> (a com.snc.sw.context.support.proc.LocalCommandRunner$StreamGobbler)
2017/10/15 21:41:26 | at java.lang.Thread.join(Thread.java:1319)
2017/10/15 21:41:26 | at com.snc.sw.context.support.proc.LocalCommandRunner.executeCommandWithTimeout(LocalCommandRunner.java:129)
2017/10/15 21:41:26 | at com.snc.sw.context.support.proc.LocalCommandRunner.executeCommand(LocalCommandRunner.java:159)
2017/10/15 21:41:26 | at com.snc.sw.providers.wmi.WmiCollectorServiceStarter.killDllLockers(WmiCollectorServiceStarter.java:271)
2017/10/15 21:41:26 | at com.snc.sw.providers.wmi.WmiCollectorServiceStarter.stopService(WmiCollectorServiceStarter.java:315)
2017/10/15 21:41:26 | at com.snc.sw.providers.wmi.WmiCollectorServiceStarter.checkService(WmiCollectorServiceStarter.java:107)
2017/10/15 21:41:26 | at com.snc.sw.providers.wmi.WMIProviderImpl.send(WMIProviderImpl.java:725)
2017/10/15 21:41:26 | at com.snc.sw.providers.wmi.WMIProviderImpl.executeQuery(WMIProviderImpl.java:254)
2017/10/15 21:41:26 | at com.snc.sw.providers.wmi.WMIProviderImpl.executeQuery(WMIProviderImpl.java:1326)
2017/10/15 21:41:26 | at com.snc.sw.commands.wmi.ProcessListWmiCommand.getProcessList(ProcessListWmiCommand.java:61)
2017/10/15 21:41:26 | at com.snc.sw.commands.wmi.ProcessListWmiCommand.execute(ProcessListWmiCommand.java:41)
2017/10/15 21:41:26 | at com.snc.sw.commands.wmi.ProcessListWmiCommand.execute(ProcessListWmiCommand.java:28)
2017/10/15 21:41:26 | at com.snc.sw.commands.CommandsRunner.execute(CommandsRunner.java:83)
2017/10/15 21:41:26 | at com.snc.sw.commands.CommandsRunner.execute(CommandsRunner.java:40)
2017/10/15 21:41:26 | at com.snc.sw.commands.DiscoveryCommandDomainManagerImpl.getProcesses(DiscoveryCommandDomainManagerImpl.java:120)
2017/10/15 21:41:26 | at com.snc.sw.commands.FindProcessByPID.findProcess(FindProcessByPID.java:23)
2017/10/15 21:41:26 | at com.snc.sw.context.ExecutionContextImpl.initializeEntryPointProcess(ExecutionContextImpl.java:689)
2017/10/15 21:41:26 | at com.snc.sw.pattern.HorizontalDiscoveryPatternExecutor.getProcess(HorizontalDiscoveryPatternExecutor.java:92)
2017/10/15 21:41:26 | at com.snc.sw.pattern.HorizontalDiscoveryPatternExecutor.executeIdentifications(HorizontalDiscoveryPatternExecutor.java:116)
2017/10/15 21:41:26 | at com.snc.sw.pattern.HorizontalDiscoveryPatternExecutor.executePattern(HorizontalDiscoveryPatternExecutor.java:62)
2017/10/15 21:41:26 | at com.snc.sw.pattern.DefaultPatternExecutor.executePattern(DefaultPatternExecutor.java:26)
2017/10/15 21:41:26 | at com.service_now.mid.probe.HorizontalDiscoveryProbe.runPattern(HorizontalDiscoveryProbe.java:199)
2017/10/15 21:41:26 | at com.service_now.mid.probe.HorizontalDiscoveryProbe.runProbe(HorizontalDiscoveryProbe.java:88)
2017/10/15 21:41:27 | at com.service_now.mid.probe.ServiceWatchProbe.probe(ServiceWatchProbe.java:36)
2017/10/15 21:41:27 | at com.service_now.mid.probe.AProbe.process(AProbe.java:81)
2017/10/15 21:41:27 | at com.service_now.mid.queue_worker.AWorker.runWorker(AWorker.java:119)
2017/10/15 21:41:27 | at com.service_now.mid.queue_worker.AWorkerThread.run(AWorkerThread.java:20)
2017/10/15 21:41:27 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
2017/10/15 21:41:27 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
2017/10/15 21:41:27 | at java.lang.Thread.run(Thread.java:745)

Steps to Reproduce

 

Watch for the following symptoms:

  • Slow Discovery or Discovery timeout 
  • MID server being unresponsive

Workaround

 

Note – If using the process introduced in Jakarta that doesn't have any corresponding probes, just disable the pattern.

For other releases, disable the patterns and use probes.

  1. Disable the pattern.

    1. Go to Discovery Definition > Processes, and look up the pattern name.

    2. On the process page, scroll down to the Triggers Probes. Set the patterns to False and all of the related probes to True.

  2. Disable the pattern.

    1. Go to the probe/pattern page and uncheck the Active checkbox.

    2. Click Update.

  3. Enable the probe.

    1. Go to the probe/pattern page and check the Active checkbox.

    2. Click the Update button.

      The pattern is disabled and all related probes are enabled.

     


Related Problem: PRB1234649

Seen In

There is no data to report.

Intended Fix Version

London

Fixed In

Jakarta Patch 6
Kingston Patch 1

Safe Harbor Statement

This "Intended Fix Version" information is meant to outline ServiceNow's general product direction and should not be relied upon in making a purchasing decision. The information provided here is for information purposes only and may not be incorporated into any contract. It is not a commitment, promise, or legal obligation to deliver any material, code, or functionality. The development, release, and timing of any features or functionality described for our products remains at ServiceNow's sole discretion.

Associated Community Threads

There is no data to report.

Article Information

Last Updated:2018-03-07 05:21:54
Published:2017-11-07