Issue
This article provides a method of identifying "Lookup Select Box" variables that may cause slowness due to excessive lookups. This slowness may be incurred/observed in various places:
- Initial loading of a catalog item
- Displaying a service catalog task (sc_request, sc_req_item, sc_task, etc.) where the form includes the Variable Editor formatter
- Text Indexing of a service catalog task
You may see a stack trace similar to the following, which will indicate that you are hitting this type of issue:
main,Default-thread-109,5,attrs=(session_id=8D3C0B18DB1F3F00D3C39EB5DB9619F3)
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
java.net.SocketInputStream.read(SocketInputStream.java:171)
java.net.SocketInputStream.read(SocketInputStream.java:141)
java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
java.io.BufferedInputStream.read(BufferedInputStream.java:345)
org.mariadb.jdbc.internal.common.packet.buffer.ReadUtil.readFully(ReadUtil.java:82)
org.mariadb.jdbc.internal.common.packet.buffer.ReadUtil.readFully(ReadUtil.java:92)
org.mariadb.jdbc.internal.common.packet.RawPacket.nextPacket(RawPacket.java:77)
org.mariadb.jdbc.internal.common.packet.SyncPacketFetcher.getRawPacket(SyncPacketFetcher.java:67)
org.mariadb.jdbc.internal.mysql.MySQLProtocol.getResult(MySQLProtocol.java:966)
org.mariadb.jdbc.internal.mysql.MySQLProtocol.executeQuery(MySQLProtocol.java:1076)
org.mariadb.jdbc.internal.mysql.MySQLProtocol.executeQuery(MySQLProtocol.java:1055)
org.mariadb.jdbc.MySQLStatement.execute(MySQLStatement.java:295)
org.mariadb.jdbc.MySQLStatement.execute(MySQLStatement.java:393)
sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
com.glide.db.StatementWrapper.invoke(StatementWrapper.java:40)
com.sun.proxy.$Proxy7.execute(Unknown Source)
com.glide.db.DBI.executeStatement0(DBI.java:1053)
com.glide.db.DBI.executeStatement(DBI.java:1009)
com.glide.db.DBI.executeStatement(DBI.java:970)
com.glide.db.DBI.executeStatement(DBI.java:958)
com.glide.db.DBQuery.executeAsResultSet0(DBQuery.java:376)
com.glide.db.DBQuery.executeAsResultSet0(DBQuery.java:351)
com.glide.db.DBQuery.executeAndReturnTable(DBQuery.java:308)
com.glide.db.DBAction.executeNormal(DBAction.java:236)
com.glide.db.DBAction.executeAndReturnException(DBAction.java:190)
com.glide.db.RDBMSQueryContext.executeQuery(RDBMSQueryContext.java:46)
com.glide.db.DBQuery.execute(DBQuery.java:2636)
com.glide.db.meta.Table.queryBasic(Table.java:315)
com.glide.db.meta.Table.query(Table.java:201)
com.glide.script.GlideRecordITable.query(GlideRecordITable.java:101)
com.glide.script.GlideRecord.query0(GlideRecord.java:3256)
com.glide.script.GlideRecord.query(GlideRecord.java:2981)
com.glideapp.questionset.LookupSelectQuestion.loadChoiceList(LookupSelectQuestion.java:102)
com.glideapp.questionset.LookupSelectQuestion.getChoiceList(LookupSelectQuestion.java:46)
com.glideapp.questionset.AbstractChoiceListQuestion.getDisplayValue(AbstractChoiceListQuestion.java:35)
com.glideapp.questionset.AbstractChoiceListQuestion.getDisplayValue(AbstractChoiceListQuestion.java:26)
com.glideapp.questionset.LookupSelectQuestion.getDisplayValue(LookupSelectQuestion.java:58)
com.glide.service_portal.variables.VariableModel.setValue(VariableModel.java:710)
com.glide.service_portal.variables.VariableModel.augmentByType(VariableModel.java:539)
com.glide.service_portal.variables.VariableModel.buildVariable(VariableModel.java:311)
com.glide.service_portal.variables.VariableModel.get(VariableModel.java:140)
com.glide.service_portal.catalog.SPCatalog.getItem(SPCatalog.java:188)
com.glide.service_portal.catalog.SPCatalog.getItem(SPCatalog.java:48)
com.glide.service_portal.catalog.SPCatalog.getItem(SPCatalog.java:44)
com.glide.service_portal.widget.SPScriptable.getCatalogItem(SPScriptable.java:957)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.mozilla.javascript.MemberBox.invoke(MemberBox.java:138)
org.mozilla.javascript.NativeJavaMethod.call(NativeJavaMethod.java:292)
org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2585)
org.mozilla.javascript.optimizer.OptRuntime.call2(OptRuntime.java:42)
org.mozilla.javascript.gen.sp_widget_0fd6a6f247230200ba13a5554ee490b3_script_3292._c_anonymous_2(sp_widget.0fd6a6f247230200ba13a5554ee490b3.script:54)
org.mozilla.javascript.gen.sp_widget_0fd6a6f247230200ba13a5554ee490b3_script_3292.call(sp_widget.0fd6a6f247230200ba13a5554ee490b3.script)
org.mozilla.javascript.ScriptRuntime.doCall2(ScriptRuntime.java:2650)
org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2590)
org.mozilla.javascript.optimizer.OptRuntime.call0(OptRuntime.java:23)
org.mozilla.javascript.gen.sp_widget_0fd6a6f247230200ba13a5554ee490b3_script_3292._c_anonymous_1(sp_widget.0fd6a6f247230200ba13a5554ee490b3.script:2)
org.mozilla.javascript.gen.sp_widget_0fd6a6f247230200ba13a5554ee490b3_script_3292.call(sp_widget.0fd6a6f247230200ba13a5554ee490b3.script)
org.mozilla.javascript.ScriptRuntime.doCall2(ScriptRuntime.java:2650)
org.mozilla.javascript.ScriptRuntime.doCall(ScriptRuntime.java:2590)
org.mozilla.javascript.optimizer.OptRuntime.call0(OptRuntime.java:23)
org.mozilla.javascript.gen.sp_widget_0fd6a6f247230200ba13a5554ee490b3_script_3292._c_script_0(sp_widget.0fd6a6f247230200ba13a5554ee490b3.script:1)
org.mozilla.javascript.gen.sp_widget_0fd6a6f247230200ba13a5554ee490b3_script_3292.call(sp_widget.0fd6a6f247230200ba13a5554ee490b3.script)
org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:563)
org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3428)
org.mozilla.javascript.gen.sp_widget_0fd6a6f247230200ba13a5554ee490b3_script_3292.call(sp_widget.0fd6a6f247230200ba13a5554ee490b3.script)
org.mozilla.javascript.gen.sp_widget_0fd6a6f247230200ba13a5554ee490b3_script_3292.exec(sp_widget.0fd6a6f247230200ba13a5554ee490b3.script)
com.glide.script.ScriptEvaluator.execute(ScriptEvaluator.java:279)
com.glide.script.ScriptEvaluator.evaluateString(ScriptEvaluator.java:118)
com.glide.script.ScriptEvaluator.evaluateString(ScriptEvaluator.java:82)
com.glide.service_portal.SPScriptEvaluator.eval(SPScriptEvaluator.java:37)
com.glide.service_portal.widget.SPScriptRunner.eval(SPScriptRunner.java:221)
com.glide.service_portal.widget.SPScriptRunner.runScript(SPScriptRunner.java:96)
com.glide.service_portal.widget.SPWidget.get(SPWidget.java:123)
com.glide.service_portal.widget.SPWidget.get(SPWidget.java:106)
com.glide.service_portal.page.SPPageLayoutCache.getColumnRectangles(SPPageLayoutCache.java:49)
com.glide.service_portal.page.SPPageLayout.getRowColumns(SPPageLayout.java:115)
com.glide.service_portal.page.SPPageLayout.getContainerRows(SPPageLayout.java:82)
com.glide.service_portal.page.SPPageLayout.getContainers(SPPageLayout.java:56)
com.glide.service_portal.page.SPPageLayoutCache.getContainers(SPPageLayoutCache.java:34)
com.glide.service_portal.page.SPPageCache.getContainers(SPPageCache.java:26)
com.glide.service_portal.page.SPPage.buildPageResponse(SPPage.java:94)
com.glide.service_portal.page.SPPage.getPage(SPPage.java:81)
com.glide.service_portal.page.SPPage.get(SPPage.java:57)
com.glide.service_portal.SPRestService.page(SPRestService.java:80)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
com.glide.rest.handler.impl.ServiceHandlerImpl.invokeService(ServiceHandlerImpl.java:43)
com.glide.rest.processors.RESTAPIProcessor.process(RESTAPIProcessor.java:286)
com.glide.processors.AProcessor.runProcessor(AProcessor.java:532)
com.glide.processors.AProcessor.processTransaction(AProcessor.java:230)
com.glide.processors.ProcessorRegistry.process0(ProcessorRegistry.java:178)
com.glide.processors.ProcessorRegistry.process(ProcessorRegistry.java:167)
com.glide.ui.GlideServletTransaction.process(GlideServletTransaction.java:31)
com.glide.sys.Transaction.run(Transaction.java:2091)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
The "LookupSelectQuestion" keyword indicates that this is a "Lookup Select Box"-type variable.
There are several solutions to speeding up these variables:
- Convert the variable from type "Lookup Select Box" type to a "Reference" type variable. This will change the user experience from using a listbox to using a "Lookup Using List".
- Add a more selective filter to the reference qualifier for the variable.
- Use an appropriate* source reference table.
* An inappropriate source reference table would be task or incident. An appropriate source reference table would be the Choices [sys_choice] table. For example, if you had an incident record producer, and that had a variable named "category", it would be inappropriate to query the entire incident table to get all of the category values. Instead, the reference table should be set to the sys_choice table and a reference qualifier should specify the table and field (eg., name=incident^element=category).
Procedure
Navigate to the System Maintenance > Scripts - Background module, copy the following snippet of javascript into the field labeled, "Run script (JavaScript executed on server)", and click on the "Run Script" button.
var MAX_LOOKUP_RECORDS = 1000;
var gsuri = gs.getProperty('glide.servlet.uri');
var stdout = ['\n'];
// Query the Variables table for
var gr = new GlideRecord('item_option_new');
gr.addQuery('type', '=', 18);
gr.query();
while (gr.next()) {
try {
// Check to make sure the lookup_table is defined
if (gr.lookup_table) {
// Determine how many records the Lookup Select Box will return
var count = new GlideAggregate(gr.lookup_table);
if (count.isValid()) {
if (gr.reference_qual) {
var encoded_query = null;
if (gr.reference_qual.startsWith('javascript')) {
if (gr.reference_qual.includes('current.variables')) {
continue;
}
var evaluator = new GlideScopedEvaluator();
encoded_query = evaluator.evaluateScript(gr, 'reference_qual', null);
} else {
encoded_query = gr.reference_qual;
}
if (!encoded_query) {
continue;
}
count.addEncodedQuery(encoded_query);
}
count.addAggregate('COUNT');
count.query();
var lookupRecords = 0;
if (count.next()) {
lookupRecords = count.getAggregate('COUNT');
}
// Report a warning if the number of matching records exceeds the MAX LOOKUP RECORDS
if (lookupRecords > MAX_LOOKUP_RECORDS) {
stdout.push('Variable: ' + gr.name + '\n' + gsuri + 'item_option_new.do?sys_id=' + gr.sys_id + '\n');
stdout.push('\tLookup Table: ' + gr.lookup_table);
stdout.push('\tReference Qualifier: ' + count.getEncodedQuery() + '\n');
stdout.push('\tThere are ' + lookupRecords + ' matching records\n' + '\tLIST URL = ' + gsuri + gr.lookup_table + '_list.do?sysparm_query=' + count.getEncodedQuery() + '\n');
}
} else {
// not a valid table
stdout.push('Variable - ' + gr.name + '\n' + gsuri + 'item_option_new.do?sys_id=' + gr.sys_id + '\n');
stdout.push('\t*** Invalid Variable: The lookup_table value for this variable ( ' + gr.lookup_table + ' ) does not match not a valid table.\n ');
}
} else {
stdout.push('Variable - ' + gr.name + '\n' + gsuri + 'item_option_new.do?sys_id=' + gr.sys_id + '\n');
stdout.push('\t*** Invalid Variable: The lookup_table value for this variable IS NULL.\n ');
}
} catch (err) {
// An error occurred; report something helpful
var st = err.getStackTrace();
stdout.push('Variable - ' + gr.name + '\n' + gsuri + 'item_option_new.do?sys_id=' + gr.sys_id + '\n' + 'Stack Trace:\n');
for (var i = 0; i < st.length; i++) {
stdout.push(st[i].toString());
}
stdout.push('\n\n');
}
}
gs.print(stdout.join('\n'));
You can adjust the value of the MAX_LOOKUP_RECORDS variable in the script to suit your purposes; however, as a best practice, a Lookup Select Box should never return more than 100 records.
Example Output
The script will provide output such as the following:
*** Script: Variable - department https://myinstance.service-now.com/item_option_new.do?sys_id=e70ec99e53032200eb7c0a1806dc3449
There are 100002 matching records LIST URL = https://myinstance.service-now.com/sys_user_list.do?sysparm_query= Variable - table https://myinstance.service-now.com/item_option_new.do?sys_id=e849da79471102003db6d7527c9a7128
There are 3189 matching records LIST URL = https://myinstance.service-now.com/sys_db_object_list.do?sysparm_query= Variable - broccoli https://myinstance.service-now.com/item_option_new.do?sys_id=e849da79471102003db6d7527c9a7128
Invalid Variable: The lookup_table value for this variable IS NULL. Variable - meowmeow https://myinstance.service-now.com/item_option_new.do?sys_id=e849da79471102003db6d7527c9a7128
Invalid Variable: The lookup_table value for this variable (meownow ) does not match not a valid table.
The output first shows the name of the variable and the link to the variable definition. Next, it shows the number of rows that would be returned by the lookup for this variable. Finally, it provides a link to the list of the data that would be used by the variable. The LIST URL will likely render quickly, because we are usually only displaying the first 1-20 rows; however, keep in mind that the variable lookup would not paginate; it would retrieve all records. The output also shows errors if the lookup table is empty, is invalid, or for some other reason we are unable to get the rowcount of the variable.
Release
All
Related Links
Create a service catalog variable
