Notifications

18 views

Symptoms


Many Discovery jobs are suddenly created, multiple jobs per Discovery Schedule for some schedules flooding the scheduled job queue (sys_trigger table) and ECC Queue (ecc_queue) table.

Release


Verified on Istanbul, Jakarta, and Kingston.

Cause


- The instance has several 'Run After' discovery schedules with 'Even if canceled' checked. EX: Sched A -> Sched B -> Sched C -> Sched D -> Sched E -> Sched F
- These Schedules are configured to run on a MID Server Cluster. When 2 or more MID Servers in the cluster became 'Invalidated', multiple 'discovery.canceled' events are published, one for each MID Server. 
- For Example, if 2 MID servers are 'Invalidated', 2 'discovery.canceled' are published for Sched A when it's launched. These two events triggered 2 discovery jobs for Sched B and this schedule will be cancelled as will with 2 'discovery.canceled' published for each job for a total of 4 jobs. Subsequently, 4 discovery jobs are started for Sched C, 8 for sched D, 16 for Sched E, and 32 for Sched F. The # of jobs are multiplicative with more MID Servers 'Invalidated'

Resolution


Addressed in PRB1293504 which targets all releases from Jakarta and newer.

 

Article Information

Last Updated:2019-01-14 01:18:52
Published:2019-01-12