Issue
Servicenow always recommends exporting data in chunks in case of large data set with millions of records to avoid any performance implications and impact on other services running on the system. The Procedure below will explain how to Export Bulk Data from Servicenow via REST Web Service Pagination.
Procedure
By default, ServiceNow has the max limit of 10000 records which will be returned in any rest call and is being caused by the omission of the parameter sysparm_limit which default value is 10000. If you specify a higher value in the URL then you can get the desired amount of records.
Example: /api/now/table/incident?sysparm_limit=40000
The best practice to retrieve large amounts of records is to by using pagination. That is, get sub-sets of records in different calls, for example, first, you get records from 1 to 10000 in a call, then you get from 10001 to 20000, then from 20001 to 30000, etc.
To do so, you can make use of the parameters
- sysparm_limit: to define how many records you get in one call (10000 by default)
- sysparm_offset: to define the records to exclude from the query
- sysparm_query=ORDERBYsys_created_on: to sort the list of records based on the created date/time
In this case, the first call would be something like
/api/now/table/incident?sysparm_limit=10000&sysparm_offset=0&sysparm_query=ORDERBYsys_created_on
the second one
/api/now/table/incident?sysparm_limit=10000&sysparm_offset=10000&sysparm_query=ORDERBYsys_created_on
the third one
/api/now/table/incident?sysparm_limit=10000&sysparm_offset=20000&sysparm_query=ORDERBYsys_created_on
.
. and so on...
You can put that in a loop to get all the records using pagination until all the records are read.
Resolution
Related Links
Example Script
See the attached script, or below, for an example that uses the Linux/Unix program cURL and demonstrates how to bulk-download data from an instance using REST API with pagination.
#!/bin/bash
## Example of how to use REST API pagination to bulk download data
## Settings
PAGESIZE=10000
PAGESTOFETCH=10
TABLE=tablename # e.g incident, question_answer, task
INSTANCENAME=myinstancename # e.g if instance is acme.service-now.com then put 'acme' here
USERNAME=myuser
PASSWORD=mypassword
OUTPUTFORMAT=json # either xml or json
## Program
echo "Starting at `date`"
i=0
pageoffset=0
while [[ $i -le $PAGESTOFETCH ]];
do
echo "Starting download of $PAGESIZE records from table $TABLE at offset $pageoffset"
curl "https://$INSTANCENAME.service-now.com/api/now/table/$TABLE?sysparm_offset=$pageoffset&sysparm_limit=$PAGESIZE" \
--request GET \
--header "Accept:application/$OUTPUTFORMAT" \
--header "Content-Type:application/json" \
--data "{}" \
--user ${USERNAME}:${PASSWORD} \
--output question_answer$i.xml \
--silent
echo "Saved $PAGESIZE records to question_answer$i.xml"
((pageoffset+=$PAGESIZE))
((i++))
done
echo "Finished at `date`"
Exporting Attachments from a ServiceNow Instance
Because exporting attachments involves downloading the binary data for each attachment the procedure is different. See: