The HTTP X-Robots-Tag header provides instructions to web crawlers concerning how URLs of a website are to be indexed.
The HTTP X-Robots-Tag header is used to direct web crawlers in terms of indexing URLs on the website. It has several directives, as follows:
The "X-" naming convention for HTTP headers, "X" referring to "experimental", has been deprecated and need to be transitioned to formal naming convention for HTTP headers.
nofollow directives inform web crawlers as to whether or not to follow links that are on the URL.
follow directive is combined with the
noindex directive for a longer period of time on a URL, the
follow may be interpreted by search engines as the
nofollow directive instead due to the nature of the
noarchive directives direct a crawler concerning whether to show or not show a cached version of a URL during a search. The
nocache directive is equivalent of using the
noarchive directive, however only Bing is using this directive.
Specifying the all directive is the same thing as including the
index and follow directives for
all of the text and links on the site. Using
none is the same thing as specifying
nofollow, effectively blocking all of the URLs on the site from the search results.
nosnippet directive instructs the web crawler not to show a snippet or video in the search results.
nositelinkssearchbox directive instructs the web crawler not to show a sitelinks search box in the search results.
max-snippet directive sets the maximum snippet size, as measured by the number of characters. In the following example the snippet is limited to 42 characters.
X-Robots-Tag: max-snippet: 42
A value of zero
0 is equivalent to the
nosnippet directive, whereas a value of
-1 removes any limitations imposed on search engines and lets search engines decide what is most effective.
When absent on a URL, within the HTML source code and the HTTP X-Robots-Tag header, the default value for this directive is assumed to be
-1 with no limitations on snippet length for search engines.
max-image-preview directive limits the size for images that are shown in search results. A value of
none prevents an image preview from being shown in the search results, whereas a setting of
standard instructs search engines to show the default image preview and a setting of
large allows a larger image preview may be shown, up to the width of the viewport.
X-Robots-Tag: max-image-preview: large
max-video-preview directive limits the maximum length of videos that are displayed in the search results. Video length is measured in seconds. In the following example the video preview is limited to 42 seconds.
X-Robots-Tag: max-video-preview: 42
A value of zero
0 allows only static images, whereas a setting of
-1 removes length restrictions altogether.
notranslate directive is present, search engines will not translate the URL in the search results.
noimageindex directive will prevent images on the URL from being indexed.
indexifembedded directive, when accompanied with
noindex, allows search engines to index the content if the URL is embedded in another URL through iframes despite of the
X-Robots-Tag: indexifembedded, noindex
unavailable_after directive is present, it instructs search engines not to show the URL after the specified date.
HTTP/1.1 200 OK Date: Tue, 1 Dec 2010 11:22:13 UTC X-Robots-Tag: unavailable_after: 31 Dec 2021 23:59:59 UTC
If no valid date and time are specified or this directive is absent, the URL is assumed to have no expiration date and may be shown in search results indefinitely.
X-Robots-Tag: google: nopagereadaloud
nopagereadaloud directive is specific to Google Read Aloud only and is initiated by user requests rather than an automated web crawler.
In the case of conflicting directives, web crawlers such as Google will apply the more restrictive directives. For example, if
max-snippet: -1 are both specified then the URL will not be indexed, and the snippet will not be shown.
X-Robots-Tag: noindex, max-snippet: -1
Targeting specific web crawlers
Specific web crawlers can be targeted with the HTTP X-Robots-Tag header, and multiple HTTP X-Robots-Tag headers can be present in the HTTP response. In the following example, the first two headers are specific to a web crawler, while the second header is for all other web crawlers.
X-Robots-Tag: bingbot: noindex, follow X-Robots-Tag: googlebot: nosnippet, notranslate X-Robots-Tag: index
The HTTP X-Robots-Tag header is used by a server to instruct web crawlers on how the resources on the site must be indexed for search engines.