Symptoms
Many Discovery jobs are suddenly created, multiple jobs per Discovery Schedule for some schedules flooding the scheduled job queue (sys_trigger table) and ECC Queue (ecc_queue) table.
Release
Verified on Istanbul, Jakarta, and Kingston.
Cause
- The instance has several 'Run After' discovery schedules with 'Even if canceled' checked. EX: Sched A -> Sched B -> Sched C -> Sched D -> Sched E -> Sched F
- These Schedules are configured to run on a MID Server Cluster. When 2 or more MID Servers in the cluster became 'Invalidated', multiple 'discovery.canceled' events are published, one for each MID Server.
- For Example, if 2 MID servers are 'Invalidated', 2 'discovery.canceled' are published for Sched A when it's launched. These two events triggered 2 discovery jobs for Sched B and this schedule will be cancelled as will with 2 'discovery.canceled' published for each job for a total of 4 jobs. Subsequently, 4 discovery jobs are started for Sched C, 8 for sched D, 16 for Sched E, and 32 for Sched F. The # of jobs are multiplicative with more MID Servers 'Invalidated'
Resolution
Addressed in PRB1293504 which targets all releases from Jakarta and newer.