Issue
Symptoms
Workflows are struck in timer. A 10 second timer takes 4 minute or more sometimes to complete.
Release
All releases
Cause
Perspectium Jobs that ran on the same Node
RITM Workflow & Perspectium Job or Any Background Jobs run on that Node by Customer
Resolution
Looking at the transaction Logs (Background) and the timer that runs on the workflow there is a clear indication that timer job was queued up behind perspectium jobs. The time job did not get a worker thread until perspectium jobs were over. Due to that, a 20-second timer turned out to take 4 minutes or higher. Each timer is also of type scheduler and competes with the perspectium jobs for workflow thread and gets queued up till that freed up to attend to the timer in the queue.
The transaction Logs and workflow context logs indicate that the Queue has timer pushed to the end only after the Perspectium jobs have completed.
Look at the Background Transaction Jobs, the start and end times from the workflow context/workflow timer activity which will tell you when the timer executed and for how long.
Additional Information
Best Practices/Ideas to minimize the run times of Perspectium job. (best Practices from performance team)
1) Disable the long running 'Perspectium Jobs' by setting it to 'On-Demand' and then truncate the psp_out_message table if the size of this table >= 5GB
2) Identify any long running queries as part of this job and tune it further with indexes/index hints so that it releases the worker.
3) Move the Business Rules as part of the Perspectium jobs to run asynchronously
What happens when a user action triggers an ASYNC business rule: the platform creates a scheduled job (inserts a record in sys_trigger table) from the business rule after the user submits the form and after any action is taken on the record on the database. From this point of view, the ASYNC business rules are similar to after business rules, but ASYNC rules run in the background, allowing the user to carry out normal transactions without having to wait for the business rule to complete. But this still occupies the worker thread so the solution may not be valid because a timer which runs as a scheduled jobs runs in the background as a scheduler job.