How to customise the Robots.txt File on a ServiceNow Instance (KB0787281)

Description

The OOB content of https://instancename.service-now.com/robots.txt on any ServiceNow instance will be:

User-agent: *

Disallow: /

Some customers need to customise this to allow indexing software (e.g. SortSite) which respects robots.txt to index public pages on the instance

Release or Environment

Madrid and newer

Instructions

- Request the 'Google custom search integration' plugin using the steps in the documentation: https://docs.servicenow.com/bundle/newyork-platform-administration/page/administer/search-administration/task/t_ActivaGoogCustmSrchIntegr.html

- Once the plugin is installed go to Custom Search Integration->Robots.txt Definitions. At this point there will be no 'Robots file' (Robots.txt Definitions) records, and the robots.txt will be blank

- Add a 'Robots file' record with the required content and Active ticked, for example:

User-agent: SomeSiteIndexingSoftware

Allow: /

- Verify the results at https://instancename.service-now.com/robots.txt

Additional Information

Some customers do not want to use the 'Google custom search integration' plugin for search, they just need the ability to customise the Robots.txt file. To achieve this these customers can apply a special Update Set that removes all of the 'Google custom search integration' plugin except the part, 'Robots.txt Definitions', that allows you to customise robots.txt.

This Update Set is available on KB0692532, but this article is not visible to the public because the Update Set has not been tested or verified by development and should be used with caution by customers. To get this Update Set please raise a Case in HI with ServiceNow support requesting the Update Set in KB0692532. Always test carefully on a sub-production instance.

Notifications

Description

Release or Environment

Instructions

Additional Information

Article Information