Issue
Description
Servicenow always recommend to export data in chunks in case of large data set with millions of records to avoid any performance implications and impact on other services running on the system. The Procedure below will explain on how to Export Bulk Data from Servicenow via REST Web Service Pagination.
Procedure
- By default ServiceNow has the max limit of 10000 records which will be returned in any rest call and is being caused by the omission of the parameter sysparm_limit which default value is 10000. If you specify a higher value in the URL then you can get the desired amount of records.
Example: /api/now/table/incident?sysparm_limit=40000
- The best practice to retrieve large amounts of records is to by using pagination. That is, get sub-sets of records in different calls, for example, first, you get records from 1 to 10000 in a call, then you get from 10001 to 20000, then from 20001 to 30000, etc.
NOTE: Increasing the maximum returned record (10000 by default) may have performance implications.
To do so, you can make use of the parameters
- sysparm_limit: to define how many records you get in one call (10000 by default)
- sysparm_offset: to define the records to exclude from the query
- sysparm_query=ORDERBYsys_created_on: to sort the list of records based on the created date/time
In this case, the first call would be something like
/api/now/table/incident?sysparm_limit=10000&sysparm_offset=0&sysparm_query=ORDERBYsys_created_on
the second one
/api/now/table/incident?sysparm_limit=10000&sysparm_offset=10000&sysparm_query=ORDERBYsys_created_on
the third one
/api/now/table/incident?sysparm_limit=10000&sysparm_offset=20000&sysparm_query=ORDERBYsys_created_on
.
. and so on...
You can put that in a loop to get all the records using pagination untill all the records are read.
Applicable Versions
All Versions
Additional Information
NOTE: It is always a best practice to not to keep the value of sysparm_limit in check as the default Quota Rules for REST API transactions is set to 60 secs per transaction and increasing sysparm_limit drastically can cause the transaction to time out and get canceled.