Incoming trap/event records are slowly processed and most of the events are stuck in ready state. There are many reasons that can cause events to be processing slowly or stuck. This knowledge base article handles the troubleshooting of only one case where the traps received are all assigned to the same event processing job.
Release or Environment
All instance versions
In order to make sure that the issue mentioned in this KB is the one you are facing, you should be seeing the following behavior:
- When checking the incoming traps records, notice that all of them are assigned to the same bucket:
Check the bucket field of the event record
- From the scheduler, you'll notice only one or two "Event Management - process events # " jobs running (and that job will correspond to the bucket number assigned to the traps):
System schedule > Scheduled Jobs > Event Management - process events #
- Going back to the trap records, you'll notice that the fields used to assign buckets are not populated. In order to assign a bucket number, the event/trap needs to have "Message Key" populated. If its empty, then "Source, Node, Type, Resource, and Metric Name" fields values will be used in the calculation. If only one of those fields is populated then all the incoming traps will be assigned to the same bucket and therefore, only the event processing job that this bucket is assigned to will be active.
- The number of process events jobs running should exist according to the Number of scheduled jobs processing events (event_processor_job_count) property configured.
- Please refer to our documentation with regards to the fields used in the Message Key population: Event field format for event collection
- For more information regarding bucket assignment and event processing jobs, please refer Event Management: Process events jobs
The Resolution here is to configure the trap source to populate either the "Message Key" field or the other fields mentioned that are used in the bucket calculation.
In NewYork release, a new feature has been introduced that allows the configuration of specific fields to be used in the Message Key population instead of the out of the box fields. This can be implemented through Mid Server Parameter "mid.snmp.event.oid.keys": Example fields that can be configured: node,snmpTrapOID,ifIndex,enterprise
Although this parameter will not resolve the issue described, it will assist in picking the needed fields to be used in the bucket assignment process given that the fields are populated and not received as empty.