The OOB content of on any ServiceNow instance will be:
User-agent: *
Disallow: /
Some customers need to customise this to allow indexing software (e.g. SortSite) which respects robots.txt to index public pages on the instance

Release or Environment

Madrid and newer



- Once the plugin is installed go to Custom Search Integration->Robots.txt Definitions. At this point there will be no 'Robots file' (Robots.txt Definitions) records, and the robots.txt will be blank
- Add a 'Robots file' record with the required content and Active ticked, for example:
User-agent: SomeSiteIndexingSoftware
Allow: /

Additional Information

Some customers do not want to use the 'Google custom search integration' plugin for search, they just need the ability to customise the Robots.txt file. To achieve this these customers can apply a special Update Set that removes all of the 'Google custom search integration' plugin except the part, 'Robots.txt Definitions', that allows you to customise robots.txt.

This Update Set is available on KB0692532, but this article is not visible to the public because the Update Set has not been tested or verified by development and should be used with caution by customers. To get this Update Set please raise a Case in HI with ServiceNow support requesting the Update Set in KB0692532. Always test carefully on a sub-production instance. 

Article Information

Last Updated:2019-11-28 16:07:42