Scheduled jobs named "SMTP Sender" would occasionally remain in "queued" state and not advance to "running". Emails would not be processed and back up.
Release or Environment
Behavior observed in on-prem instance but possible in hosted instances on any version.
During the normal processing of the "SMTP Sender" job, the schedule type is set to "interval" with a a time window of one minute. The job is in "Ready" state, it is picked up by an active node and the state is changed to "Queued".
The node that picked up the "SMTP Sender" job was "offline" but running so the job state would change to "Queued" but never processed by the node.
The admin node in "offline" state but running was first shut down. Then the state of the "SMTP Sender" job was changed from "Queued" to "Ready". This allowed another "online" node to pick up the job which then changed the state to "Queued" and then "Running". All backed up emails were processed.