Notifications

2631 views

Description

This will explain how to determine if a text index or reindex has completed, and if not, how to check the progress of text indexing events processing.

This can be useful in troubleshooting the following situations as well as others. 

  1. A new or recently updated record is not returned in search results because text indexing events are still running.
  2. A table is not returning results after a clone because text index is still running. 
  3. After enabling text indexing on a table no results are returned

 

Release or Environment

ALL

Resolution

WARNING

Please note that completely rebuilding the text indexes for a table can be very time consuming and also impact performance. The single largest factor to consider is the number of records in the target table being re-indexed. While the index is being rebuilt the "text index events process" job will not be processing newer text index events which are flowing into the sysevent table. This can mean that records that are newly created/updated cannot be found by the terms that should return them until they have been processed.

In most cases regarding unexpected behavior of text searches, completely rebuilding text indexes for a table should not be necessary. It is strongly recommended that it is used as a last resort if a large amount of records are impacted and only after validating that re-indexing a single record does resolve the issue being experienced. Before attempting this in a production environment you should test in a non-production to provide a rough approximation of timing as in some cases this can take multiple days.

If you are going to rebuild the indexes for a table and you are on a London or greater release then you can use the option to reubuild the indexes in place which will preserve the existing indexes as we reprocess each record. This allows records to still be found while the re-indexing is occurring. However, newly inserted and updated records will not be found by those new terms until the sysevents for them are processed after the re-indexing of the table is complete. For more information on indexing in place please see:

https://docs.servicenow.com/bundle/london-platform-administration/page/administer/search-administration/task/real-time-reindexing.html

 

  1. To begin with, if you have just cloned your instance we do not clone text indexes which means there can be a substantial delay for text indexing to work after a clone as all the indexes are rebuilt after the clone completes.
  2. Navigate to System Definition > Text Indexes.
  3. Search for and select the index which is not returning results. 
    • If you can't find a record for the table you are indexing, it may that the table is parented by another table such as sys_metadata or task. Text indexes are build against the highest level parent table.
    • If this is for a new index and not a task or system table, the record may not show until the job is scheduled so continue on to verify the status of the Job. 
    • If the state is Indexing go to step 8 to check the event queue processing.
    • If it is not indexing yet the state will say Ready, so we need to check that the Job is scheduled.
  4. Navigate to System Scheduler > Scheduled Jobs > Scheduled Jobs (sys_trigger table).
  5. Search for and open the record for 'text index events process'.
  6. Verify that the state is not Error and that the Next action shows as being in the very near future. This job is normally set to run every 30 seconds unless it has been changed.
  7. That job should be getting processed at 30 second intervals on one of the available nodes. If it has been running for longer than normal it will show up under the Scheduled Workers on the stats.do page of the node it is running on. If you are not a ServiceNow support person you may not be able to access all the nodes to validate this in which case a support engineer can assist in checking this for you.
  8. Once the job is running you can check its progress in the events table. 
  9. Navigate to System Logs > Events.
  10. Filter by Name = 'text_index' and Processing Duration (is empty) and State is not transferred.
    • This will give you a list of text index events which still need to be processed.
    • These are processed chronologically, so if another index or large update to an already indexed table started before the one you are investigating that may be why the index you are concerned with has not completed.
    • You can filter by the Table column and then check the Created times to see which events will be indexed first.
  11. Once all of the events are processed the index should be complete. 

Article Information

Last Updated:2019-08-16 10:31:45
Published:2019-08-16