Notifications

508 views

Description

Description

The MID Server and the ECC Queue support Priority. Higher priority jobs can jump the queue, and execute quicker, in higher priority threads.

PriorityThread GroupNo. Threads (default)ParameterInput Queue in MemoryJava Prioritywork\monitors\ECCSender folder Examples of probes
2 (default)Standard 25 threads.max500 (20*Threads)5output_2 JDBC, REST, SOAP, LDAP, Discovery...
Expedited 20threads.expedited.max400 (20*Threads)8output_1 
Interactive 10 threads.interactive.max40 (4*Threads)10output_0HeartbeatProbe
SystemCommand: restartService, grabLog, etc.

When the MID Server retrieves a batch of output records from the ECC Queue, it will prioritise the higher priority jobs first (lower priority field value). The same applies to the ECCSender thread when passing results back to the instance. e.g. If a large Standard priority JDBC Import set is mid-way through, an Expedited result will not have to wait for all the import rows to be sent back to the instance first.

The MID Server will fetch more jobs than it has free threads if they are in the ecc_queue. These remain queued in MID Server memory until a thread is free. The ECC Queue output record will be in Processing state, but it may be some time before it actually starts to be processed in a thread. Note: The Processed timestamp field in the ECC Queue cannot be relied on for this reason, as all it really means the time the MID Server retrieve the record, not when the job started to run.

The higher priority jobs will be given more CPU time in the MID Server application once running. The thread pools map to different Java thread priorities, and Java will assign more resources to the job.

Java is a multi-threaded platform, and will use all the CPUs and Threads available to it. A dedicated host server/VM with many dedicated CPUs/Cores is recommended, or threads may execute considerably slower at busy times than they might when the MID Server is almost idle. That is important if you are expecting "Interactive" jobs to complete quick enough to avoid user transactions waiting too long for forms to update. e.g. A single LDAPListener thread may process user updates a lot slower while Discovery is also running in all other threads.

Making use of this information

Generally everything should run at Standard priority, unless there is a very good reason why it needs to jump the queue.

As of Madrid, very few features deliberately set this to anything other than Standard.

  • Discovery/Service Mapping will use the priory of the Discovery Status record for all probes. This may vary depending on how the Discovery is launched:
    • Interactive: Discover Now and Quick Discovery, and Pattern Designer
    • Expedited: Service Mapping
    • Standard: Scheduled Horizontal Discoveries
  • MID Server:
    • Interactive: All SystemCommands, plus "Test Credentials".
  • Orchestration:
    • Expedited: Test Inputs in Activity Designer, Start workflow from the Workflow Editor
    • Standard: All normal workflow executions.
  • Event Management
    • Interactive:  Pull Connectors, as new events need getting into the instance without delay
  • Change Management
    • Standard: Automatic and Manually triggered Discovery of Affected CIs

There is scope for certain real-time updated User-Interface triggered jobs, or quick integrations running as part of a user transaction, to be allowed to run as Expedited priority. Such as job running at higher priority would improve the user experience, and prevent blocking instance threads while waiting for the response which would increase overall instance performance. However often those integrations would be better running truly asynchronously, with a Sensor dealing with the eventual response from a low priority job in the background, without holding up a user's form or instance thread at all.

Important note: Once all Interactive threads are in use, critical MID Server system commands can be blocked, including HeartbeatProbe which is used by the instance to know if the MID Server is up, so it could be set as Down and become unusable. It's therefore a good idea to never use Interactive, and use Expedited for anything that must jump the Standard queue.

Applicable Versions

Since Geneva, when the Priority feature was added to the MID Server and ECC Queue.

Additional Information

LDAPConnectionTesterProbe for an LDAP Server connecting via a MID Server runs as Standard priority, even though it only waits a short time for a response. If a MID Server has up to 500 other standard jobs queued in memory, such as during a Discovery, this will often timeout and cause the LDAP Server to be set as Down. That behavior may be changed by PRB1331240.

CommandPipeline for "Test Credentials" runs interactive runs as Interactive priority. This has been known to cause all threads to be in use when Powershell jobs using a CyberArk credential that is missing gets stuck, and cause a MID Server to go Down due to being unable to run any HeartbeatProbe or SystemCommands. That behavior may be changed by PRB1330701.

When a Discovery is Cancelled, the message to the MID Server to stop running any more jobs for it needs to jump the queue, and at present simply inherits the priority of the discovery_status. That can lead to some jobs continuing running for up to an hour after cancelling. That behavior may be changed by PRB1332197.

Final Note: "When everything is a priority, nothing is a priority", so don't be silly with this in custom implementations or you will defeat the purpose.

Article Information

Last Updated:2019-08-02 20:49:41
Published:2019-07-01