X-Robots-Tag
The HTTP X-Robots-Tag header provides instructions to web crawlers concerning how URLs of a website are to be indexed.
Usage
The HTTP X-Robots-Tag header is used to direct web crawlers in terms of indexing URLs on the website. It has several directives, as follows:
Note
The "X-" naming convention for HTTP headers, "X" referring to "experimental", has been deprecated and need to be transitioned to formal naming convention for HTTP headers.
index/noindex
The index and noindex directives instruct the web crawler whether the URL can be indexed, or not. Specifically noindex directive is required to prevent the URL from being displayed in search results.
X-Robots-Tag: noindex
Note
When absent on a URL, within the HTML source code and the HTTP X-Robots-Tag header, the default value for this directive is assumed to be index and the URL may be indexed by search engines.
follow/nofollow
The follow and nofollow directives inform web crawlers as to whether or not to follow links that are on the URL.
X-Robots-Tag: nofollow
Note
When the follow directive is combined with the noindex directive for a longer period of time on a URL, the follow may be interpreted by search engines as the nofollow directive instead due to the nature of the noindex directive.
archive/noarchive/nocache
The archive and noarchive directives direct a crawler concerning whether to show or not show a cached version of a URL during a search. The nocache directive is equivalent of using the noarchive directive, however only Bing is using this directive.
X-Robots-Tag: noarchive
Note
When absent on a URL, within the HTML source code and the HTTP X-Robots-Tag header, the default value for this directive is assumed to be archive and the URL may be archived by search engines.
all/none
Specifying the all directive is the same thing as including the index and follow directives for all of the text and links on the site. Using none is the same thing as specifying noindex and nofollow, effectively blocking all of the URLs on the site from the search results.
X-Robots-Tag: none
Note
When absent on a URL, within the HTML source code and the HTTP X-Robots-Tag header, the default value for this directive is assumed to be all and the URL may be indexed by search engines.
nosnippet
The nosnippet directive instructs the web crawler not to show a snippet or video in the search results.
X-Robots-Tag: nosnippet
nositelinkssearchbox
The nositelinkssearchbox directive instructs the web crawler not to show a sitelinks search box in the search results.
X-Robots-Tag: nositelinkssearchbox
max-snippet
The max-snippet directive sets the maximum snippet size, as measured by the number of characters. In the following example the snippet is limited to 42 characters.
X-Robots-Tag: max-snippet: 42
A value of zero 0 is equivalent to the nosnippet directive, whereas a value of -1 removes any limitations imposed on search engines and lets search engines decide what is most effective.
Note
When absent on a URL, within the HTML source code and the HTTP X-Robots-Tag header, the default value for this directive is assumed to be -1 with no limitations on snippet length for search engines.
max-image-preview
The max-image-preview directive limits the size for images that are shown in search results. A value of none prevents an image preview from being shown in the search results, whereas a setting of standard instructs search engines to show the default image preview and a setting of large allows a larger image preview may be shown, up to the width of the viewport.
X-Robots-Tag: max-image-preview: large
max-video-preview
The max-video-preview directive limits the maximum length of videos that are displayed in the search results. Video length is measured in seconds. In the following example the video preview is limited to 42 seconds.
X-Robots-Tag: max-video-preview: 42
A value of zero 0 allows only static images, whereas a setting of -1 removes length restrictions altogether.
notranslate
When the notranslate directive is present, search engines will not translate the URL in the search results.
X-Robots-Tag: notranslate
noimageindex
Setting the noimageindex directive will prevent images on the URL from being indexed.
X-Robots-Tag: noimageindex
indexifembedded
Setting the indexifembedded directive, when accompanied with noindex, allows search engines to index the content if the URL is embedded in another URL through iframes despite of the noindex directive.
X-Robots-Tag: indexifembedded, noindex
unavailable_after
When the unavailable_after directive is present, it instructs search engines not to show the URL after the specified date.
HTTP/1.1 200 OK
Date: Tue, 1 Dec 2010 11:22:13 UTC
X-Robots-Tag: unavailable_after: 31 Dec 2021 23:59:59 UTC
Note
If no valid date and time are specified or this directive is absent, the URL is assumed to have no expiration date and may be shown in search results indefinitely.
nopagereadaloud
When the nopagereadaloud directive is present, the URL is completely opting out of Google Read Aloud from accessing the content of the URL.
X-Robots-Tag: google: nopagereadaloud
Note
The nopagereadaloud directive is specific to Google Read Aloud only and is initiated by user requests rather than an automated web crawler.
Conflicting directives
In the case of conflicting directives, web crawlers such as Google will apply the more restrictive directives. For example, if noindex and max-snippet: -1 are both specified then the URL will not be indexed, and the snippet will not be shown.
X-Robots-Tag: noindex, max-snippet: -1
Targeting specific web crawlers
Specific web crawlers can be targeted with the HTTP X-Robots-Tag header, and multiple HTTP X-Robots-Tag headers can be present in the HTTP response. In the following example, the first two headers are specific to a web crawler, while the second header is for all other web crawlers.
X-Robots-Tag: bingbot: noindex, follow
X-Robots-Tag: googlebot: nosnippet, notranslate
X-Robots-Tag: index
Takeaway
The HTTP X-Robots-Tag header is used by a server to instruct web crawlers on how the resources on the site must be indexed for search engines.