Notifications

76 views

Description

In certain configurations, WebService ImportSets may slow waiting for a mutex to be freed. This may degrade performance of HTTP requests on the API_INT semaphore and ultimately result in rejected executions (HTTP 429). The following configuration may yield this performance degradation:

  • Transform Map has field mapping(s) set to coalesce
  • The coalesced field map(s) have "Coalesce empty fields" enabled
  • Large number of concurrent WebService ImportSet HTTP requests sent, where many of the records have empty values set for the coalesced field(s) with "Coalesce empty fields" enabled

This configuration may be prevalent for scenarios like creating/modifying Incidents, where Incident.some_unique_number is the coalesced value. In the case of a new Incident, some_unique_number will be empty and a business rule will fill it in. In the case of an update to an existing Incident, some_unique_number will have a value.

WebService Import Set Transformation will create a lock (mutex) for the coalesced field(s) to ensure that only 1 record exists in the target table for the same coalesce value(s). In the case where 'Coalesce empty fields' is used, the mutex will be named "ImportSetTransformer.TARGET_TABLE_" for the records with an empty coalesce value(s). With multiple concurrent WebService Import Sets with empty values for the coalesced field(s) (like the case of many new Incidents), they will all share the same mutex name, and thus block each other.

This behavior is actually desired in the case where only 1 empty value may be present in the target table for the coalesced field(s). So the fix for this PRB will NOT change this behavior. Instead, the fix includes a new option on transform maps called 'Create new record on empty coalesce fields' that must be chosen. Please see product documentation for more information on this fix and how to leverage it.

 

 

Steps to Reproduce

1. Make an inbound REST call to a table that extends sys_import_set_row table.
2. Make sure there is a transform map for this table with a target table set to incident for example.
3. Coalesce on a field that maps to the number field on target table.
4. Leave the value in the source coalesce field as empty.
5. Send many requests through Jmeter maybe within a short span of time. (100 requests per minute)

The response times in the transaction logs will show that requests are processed sequentially and are waiting for the previous request to finish processing even though they acquired different API_INT semaphores already.

In the localhost logs, you'll see something like this:
2018-01-01 01:01:01 (001) API_INT-thread-1 111111111111 Spin waiting for mutex ImportSetTransformer.incident_ to be freed
2018-01-01 01:01:01 (001) API_INT-thread-1 111111111111 Mutex ImportSetTransformer.incident_ acquired after spinning 188 times

This means that the mutex was acquired after 188 seconds.
If you check the localhost logs for each request that was processed sequentially, you'll observe that the number of spin times would gradually increase for each request because they are queued up even to acquire the mutex.

Workaround

There is a wide variety of configurations that might expose this PRB. However, there have been successful workarounds. Please consider the following use-case and the steps taken to resolve it:

Problematic scenario:

  • Transform map defined with "coalesce" and "Coalesce empty fields" selected on target column Incident.some_unique_number
  • Insert business rule exists on Incident that generates a value for some_unique_number

Workaround provided:

1. Remove transform map entry for 'some_unique_number' field
2. Add a new transform map entry with coalesce checked where source field is below mentioned script and target field is Sys ID

answer = (function transformEntry(source) {
if(!source.some_unique_number) // if empty return GUID
return GlideGuid.generate(null);

var gr = new GlideRecord('incident');
gr.addQuery('number', source.some_unique_number);
gr.query();
if(gr.next()){
return gr.sys_id; //if a match exists, return the sys_id of the matching Incident record
} else {
return GlideGuid.generate(null);
}
})(source);


Related Problem: PRB1292084

Seen In

There is no data to report.

Intended Fix Version

Madrid

Safe Harbor Statement

This "Intended Fix Version" information is meant to outline ServiceNow's general product direction and should not be relied upon in making a purchasing decision. The information provided here is for information purposes only and may not be incorporated into any contract. It is not a commitment, promise, or legal obligation to deliver any material, code, or functionality. The development, release, and timing of any features or functionality described for our products remains at ServiceNow's sole discretion.

Associated Community Threads

There is no data to report.

Article Information

Last Updated:2019-03-20 07:35:12
Published:2019-03-06