Notifications

2275 views

Description

When trying to execute an asynchronous Outbound REST/SOAP message through MID Server, the error below is thrown if the response takes longer than 30 seconds (even when setting different timeout with waitForResponse() in your script):

No response for ECC message request with sysid=testtesttesttesttesttesttesttest after waiting for 30 seconds in ECC Queue

Release or Environment

Madrid, Kingston P14, London P7, P8

Cause

These two new properties were added under PRB1305586, released with Madrid and back-ported to KP14, LP7,LP8.

Resolution

To define a timeout longer than 30 seconds, set the glide.http.outbound.max_timeout.enabled system property to false and use the waitForResponse() method to set the timeout.

For examples, see Asynchronous RESTMessageV2 example and Asynchronous SOAPMessageV2 example.

NOTE: If glide.http.outbound.max_timeout.enabled is set to true and a value is passed in the waitForResponse() method, the system uses the smallest value from either the waitForResponse() method or the glide.http.outbound.max_timeout system property.

Additional Information

 

The ECC queue is fundamentally a queue, and a MID Server may need some time before it can pick up new jobs from the ecc queue, and some further time to execute the jobs and return the results back to the ecc queue. That process is completely asynchronous.
 
However the RESTMessageV2/SOAPMessageV2 APIs allow running outbound messages Synchronously, and while waiting for a result will actually sleep() a precious instance thread, blocking that thread from being used by anything else during the waiting time, causing backlogs and even thread exhaustion in Scheduler workers, Event processing, and Semaphores.
 
Setting a timeout is possible with waitForResponse(), but all that does is limit how long you are needlessly blocking a thread. When using a MID Server that timeout would need setting very high to take into account the additional time a MID Server will take, and have an additional margin of error to take into account any unexpected delays with the MID Server, such as temporary AMB channel problems, MID Server down or upgrades, backlogged MID Servers,  etc. which are all common.
 

Until our official documentation and examples are updated to reflect best practice, the following guidelines should give an idea of what needs doing.

If a MID Server is involved, you should not be waiting for the response in the script that makes the request.

If you don't need to wait for a response:

  • Then Don't. Use the executeAsync() method and then just carry on with your script.

If you do need to use the script API and wait for a response, either to confirm the request was successful or to use data in the response:

  1. Use .setEccCorrelator(String) method, which will cause the string value to be set in the agent_correlator value on both the output and input ecc_queue records, so that your sensor can match the response up with the original request.  Provide a unique correlator for each outbound request to associate the correct results in the ECC queue with the request when designing asynchronous automation through a MID Server.  
  2. Use .executeAsync() in your script, then do nothing else with the RESTMessageV2 or SOAPMessageV2 "response". Do not use .execute(), or use any other methods on the 'response' after this. e.g. response.haveError(). If you do, it will in effect wait synchronously.
  3. Create a Sensor business rule for
    • Table: ECC Queue [ecc_queue]
    • Insert=True
    • When: "Async" - This is necessary because you are going to be running this business rule as the MID Server inserts the record. It will be doing that in one of the node's 4 API_INT semaphore threads. Using this for more than a few milliseconds could end up with semaphore exhaustion, and your MID Servers start to go down because they can't communicate with the instance through these semaphores being blocked by your sensor. Running ASYNC will cause a scheduler worker thread to be used instead.
    • Conditions: Queue=Input, Status=Ready, Topic=RESTProbe or SOAPProbe
    • Other conditions specific to your custom job:
      • Agent Correlator STARTSWITH <a integration specific prefix> e.g. "ebonding.inc." - This is exactly the same trick that Orchestration uses to know which are it's inputs ('rba.' in that case).
      • Name CONTAINS <something in the URL to make this specific to a particular endpoint>
      • You also need to make sure no other of your sensors will pick up this job as well.
    • Script: This will be your custom code:
      • You may want to script additional conditions, perhaps based on database lookups before going ahead with the rest of your sensor. 
      • Use current.agent_correlator to match this response with the original job/record/script/log.
        If that value contains a prefix to identify the custom implementation, and data to identify a particular record sys_id, then you can then query for that to continue updating that record.
      • The script also needs to update the Status value of the ecc_queue input to Processed. This is one of the very rare situations where setting current.status='processed' or 'error'; followed by current.update(); is OK. 
  • Propertyglide.http.outbound.max_timeout
  • Description: Specifies the number of seconds that RESTMessageV2 and SOAPMessageV2 APIs wait for a response from a synchronous call. The maximum value is 30 seconds.

Article Information

Last Updated:2019-08-11 19:42:25
Published:2019-08-12