431 views

Alert: Integration Semaphore (SOAP/API_INT) Exhausted on 'instance'

Problem
Some alerts could appear with subject Integration Semaphore (SOAP/API_INT) Exhausted on 'instance'

Symptoms


You receive this notification indicating our internal monitoring has detected an issue with your instance, which is being investigated. This is just a warning on your queue usage.


Cause

An instance has multiple application nodes. The default settings are for each application node to have a maximum of 50 semaphores, with 4 active threads at the time. If transactions come too fast or take too long, they can queue up. If the queue reaches the maximum setting, new transactions on the queue are refused.

For example, per node, with 4 threads and a queue of 50, if the transactions take 10 seconds, your application node can process 4 transactions in parallel every 10 seconds on the queue. It also means that the fifth transaction will wait up to 10 seconds to start. If 100 transactions come at one time,  46 (50-4) will queue up and the rest will be refused.

Resolution

Ensure that your nodes are not being overloaded with long Web Service calls or too many calls in a short period of time.

Perform the following actions:

  • Review your active nodes.

    For each node, validate the API_INT queue. For example, the following is the list of semaphores:

     

                      AMB  API_INT  Debug  Default  Presence
    --------------  ---  -------  -----  -------  --------
    node________01  0/4  1/4      0/4    2/16     0/8
    node________02  0/4  0/4      0/4    3/16     0/8

    In this example, the problem is no longer occurring.

    If the problem were still occurring, it would show 4/4.

  • Review the stats.do on each node.

    On each node, review stats.do and search for the Semaphore set for API_INT. The information available would appear as follows:

    node________01 - Available semaphores: 4 - Queue depth: 4 - Max queue depth: 50
    node________02 - Available semaphores: 4 - Queue depth: 4 - Max queue depth: 50

    In this example, the queues look normal. Note that some queries are queued up because it is not reaching the maximum set of 4. However, the presence of "Max queue depth: 50" shows the queue had reached its maximum.

    If the problem is still occurring, you will see the Available semaphores equal 0. In that case, check the thread stack trace link on the semaphore queue to find more information.

  • To find the source of the problem, review your transaction logs.

    To find the root cause of the problem, check the transaction logs at the time the problem occurs.

    Look for long transactions or for a large amount of them in a short period of time.

    For example:

    <instance>/syslog_transaction_list.do?sysparm_query=sys_created_onONToday@javascript:gs.daysAgoStart(0)@javascript:gs.daysAgoEnd(0)%5EGROUPBYsys_created_by&sysparm_first_row=1&sysparm_view=

    Modify the example search to match the alert findings, for example, change the timings to the events that disrupted the queue.

    Try grouping the results by 'Created by', then find the user that has more calls and validate the transaction that could be related to the problem.

    Finally, analyze the full picture of the information available.

    The most common solutions are:

    • If the client Web Service application is making too many calls, try to reduce the throughput.
    • If the calls are taking too long, make the Web Service execute faster by making any long processing asynchronous.
    • If there are too many calls, try to group them together and process them in a data source or multiple update calls to Web Services.
    • If a user is abusing the system, make the user active = false and contact them.
    • Check whether the client Web Services is retrying the transactions too quite or too often (for example, because of refused responses)
    • Check whether the client application is making a full crawl on the instance (for example, mirroring or indexing Web Services applications), then time it at off-peak times with fewer parallel queries. 

 

Article Information

Last Updated:2018-01-25 04:58:46
Published:2017-09-19