Notifications

255 views

Description

When Discovery starts and ShazzamLaunch is run, lists of IPs are compiled from the discovery schedule ranges and listed in the parameters of the Shazzam ecc_queue output. Shazzam creates huge payloads because of the way it handles IP address ranges in XML, which is inefficient for this sort of data. 
 
After the job is run in the MID Server, the parameters are copied in the input ecc queue record, and we include the IP ranges in the result. This ends up with Shazzam inputs that cannot be processed, due to the huge lists of IP addresses.
 
If ShazzamLaunch succeeds, a very large ECC Queue Shazzam output is created. A symptom of a Shazzam Input that can't be processed, coming back from one of those outputs is this Error in the discovery status log:
Sensor error when processing Shazzam/See Payload: No sensors defined
 
If it fails to create the Shazzam output, you may will see this error in the Discovery Log:
"The Discovery range is empty".
 
 

Steps to Reproduce

1 - This issue is all about Scaling, so it requires a perfectly configured Discovery schedule that has many of complex IP Ranges in order to reproduce. 

2 - Run the discovery schedule.
 
3 - Note the runtime and size of the lists of IPs and parameters in the ecc_queue records.
 
An example threads.do stack trace for while running ShazzamLaunch (sys_script_include.37d16e0b0a0a0b8200ebb61333ec7d71) to create the ecc_output. This had been running running for 1h 48m:
 
main,glide.scheduler.worker.5,4,,<DISCOVERY SCHEDULE NAME> (6539737 ms)
java.lang.Object.clone(Native Method)
com.snc.commons.networks.Address32Bit$Notation.values(Address32Bit.java:53)
com.snc.commons.networks.Address32Bit.stringToBytes(Address32Bit.java:323)
com.snc.commons.networks.Address32Bit.<init>(Address32Bit.java:114)
com.snc.commons.networks.Address32Bit.validate(Address32Bit.java:421)
com.snc.commons.networks.IPList$IPListElement.<init>(IPList.java:297)
com.snc.commons.networks.IPList.add(IPList.java:89)
com.snc.commons.networks.IPList.add(IPList.java:73)
com.snc.commons.networks.IPList.getInterval(IPList.java:184)
com.snc.commons.networks.IPList.splitAt(IPList.java:174)
com.snc.commons.networks.IPIncludeExcludeCollection.splitAt(IPIncludeExcludeCollection.java:77)
com.snc.commons.networks.IPMetaCollection.splitAt(IPMetaCollection.java:285)
com.snc.commons.networks.DiscoveryRanges.toBatchesBySize(DiscoveryRanges.java:358)
com.snc.discovery.js.DiscoveryRangesDB.jsFunction_toBatchesBySize(DiscoveryRangesDB.java:196)
sun.reflect.GeneratedMethodAccessor1577.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.mozilla.javascript.MemberBox.invoke(MemberBox.java:138)
org.mozilla.javascript.FunctionObject.doInvoke(FunctionObject.java:670)
org.mozilla.javascript.FunctionObject.call(FunctionObject.java:547)
org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2582)
org.mozilla.javascript.optimizer.OptRuntime.callN(OptRuntime.java:52)
org.mozilla.javascript.gen.sys_script_include_37d16e0b0a0a0b8200ebb61333ec7d71_script_6548._c_anonymous_13(sys_script_include.37d16e0b0a0a0b8200ebb61333ec7d71.script:493)
org.mozilla.javascript.gen.sys_script_include_37d16e0b0a0a0b8200ebb61333ec7d71_script_6548.call(sys_script_include.37d16e0b0a0a0b8200ebb61333ec7d71.script)
org.mozilla.javascript.ScriptRuntime.doCall2(ScriptRuntime.java:2650)
org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2590)
org.mozilla.javascript.optimizer.OptRuntime.callN(OptRuntime.java:52)
org.mozilla.javascript.gen.sys_script_include_37d16e0b0a0a0b8200ebb61333ec7d71_script_6548._c_anonymous_7(sys_script_include.37d16e0b0a0a0b8200ebb61333ec7d71.script:280)
org.mozilla.javascript.gen.sys_script_include_37d16e0b0a0a0b8200ebb61333ec7d71_script_6548.call(sys_script_include.37d16e0b0a0a0b8200ebb61333ec7d71.script)
org.mozilla.javascript.ScriptRuntime.doCall2(ScriptRuntime.java:2650)
org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2590)
org.mozilla.javascript.optimizer.OptRuntime.callN(OptRuntime.java:52)
org.mozilla.javascript.gen.sys_script_include_37d16e0b0a0a0b8200ebb61333ec7d71_script_6548._c_anonymous_5(sys_script_include.37d16e0b0a0a0b8200ebb61333ec7d71.script:165)
org.mozilla.javascript.gen.sys_script_include_37d16e0b0a0a0b8200ebb61333ec7d71_script_6548.call(sys_script_include.37d16e0b0a0a0b8200ebb61333ec7d71.script)
org.mozilla.javascript.ScriptRuntime.doCall2(ScriptRuntime.java:2650)
org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2590)
org.mozilla.javascript.optimizer.OptRuntime.callN(OptRuntime.java:52)
org.mozilla.javascript.gen.sys_script_include_37d16e0b0a0a0b8200ebb61333ec7d71_script_6548._c_anonymous_2(sys_script_include.37d16e0b0a0a0b8200ebb61333ec7d71.script:59)
org.mozilla.javascript.gen.sys_script_include_37d16e0b0a0a0b8200ebb61333ec7d71_script_6548.call(sys_script_include.37d16e0b0a0a0b8200ebb61333ec7d71.script)
org.mozilla.javascript.ScriptRuntime.doCall2(ScriptRuntime.java:2650)
org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2590)
org.mozilla.javascript.optimizer.OptRuntime.callN(OptRuntime.java:52)
org.mozilla.javascript.gen.sys_script_include_092266f60ab30150007b0466d082578c_script_6532._c_anonymous_7(sys_script_include.092266f60ab30150007b0466d082578c.script:132)
org.mozilla.javascript.gen.sys_script_include_092266f60ab30150007b0466d082578c_script_6532.call(sys_script_include.092266f60ab30150007b0466d082578c.script)
org.mozilla.javascript.ScriptRuntime.doCall2(ScriptRuntime.java:2650)
org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2590)
org.mozilla.javascript.optimizer.OptRuntime.callProp0(OptRuntime.java:85)
org.mozilla.javascript.gen.sys_script_include_092266f60ab30150007b0466d082578c_script_6532._c_anonymous_2(sys_script_include.092266f60ab30150007b0466d082578c.script:36)
org.mozilla.javascript.gen.sys_script_include_092266f60ab30150007b0466d082578c_script_6532.call(sys_script_include.092266f60ab30150007b0466d082578c.script)
org.mozilla.javascript.ScriptRuntime.doCall2(ScriptRuntime.java:2650)
org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2590)
org.mozilla.javascript.optimizer.OptRuntime.call2(OptRuntime.java:42)
org.mozilla.javascript.gen.sys_script_include_a6cdaf5bc0a802550004f460b6c04967_script_495._c_anonymous_3(sys_script_include.a6cdaf5bc0a802550004f460b6c04967.script:20)
org.mozilla.javascript.gen.sys_script_include_a6cdaf5bc0a802550004f460b6c04967_script_495.call(sys_script_include.a6cdaf5bc0a802550004f460b6c04967.script)
org.mozilla.javascript.ScriptRuntime.doCall2(ScriptRuntime.java:2650)
org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2590)
org.mozilla.javascript.optimizer.OptRuntime.callProp0(OptRuntime.java:85)
org.mozilla.javascript.gen._refname__205573._c_script_0(<refname>:1)
org.mozilla.javascript.gen._refname__205573.call(<refname>)
org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:563)
org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3428)
org.mozilla.javascript.gen._refname__205573.call(<refname>)
org.mozilla.javascript.gen._refname__205573.exec(<refname>)
com.glide.script.ScriptEvaluator.execute(ScriptEvaluator.java:263)
com.glide.script.ScriptEvaluator.evaluateString(ScriptEvaluator.java:110)
com.glide.script.ScriptEvaluator.evaluateString(ScriptEvaluator.java:76)
com.glide.script.ScriptEvaluator.evaluateString(ScriptEvaluator.java:67)
com.glide.script.Evaluator.evaluateString(Evaluator.java:91)
com.snc.automation.ScriptJob.execute(ScriptJob.java:43)
com.glide.schedule.JobExecutor.executeJob(JobExecutor.java:103)
com.glide.schedule.JobExecutor.execute(JobExecutor.java:89)
com.glide.schedule.GlideScheduleWorker.executeJob(GlideScheduleWorker.java:223)
com.glide.schedule.GlideScheduleWorker.lambda$process$112(GlideScheduleWorker.java:162)
com.glide.schedule.GlideScheduleWorker$$Lambda$41/11453743.run(Unknown Source)
com.glide.worker.TransactionalWorkerThread.executeInTransaction(TransactionalWorkerThread.java:35)
com.glide.schedule.GlideScheduleWorker.process(GlideScheduleWorker.java:162)
com.glide.schedule.GlideScheduleWorker.run(GlideScheduleWorker.java:73)
 
 
 

Workaround

As a workaround, after identifying which Discovery Schedules are failing due to this, split the IP ranges out into separate Discovery Schedules.

Please review the Fixed In section to plan your instance upgrade to a version containing the fixing patch in the core code.
 

Related Problem: PRB1249668

Seen In

There is no data to report.

Fixed In

Jakarta Patch 9
Kingston Patch 6
London

Associated Community Threads

There is no data to report.

Article Information

Last Updated:2018-09-13 12:26:43
Published:2018-07-30