X-Robots-Tag

Non-HTML resources like PDFs and images lack a meta tag for controlling search engine behavior. The X-Robots-Tag unofficial response header delivers indexing and serving directives to web crawlers through HTTP headers instead.

Table of Contents

Usage
Directives
Example
Troubleshooting
See also

Usage

The X-Robots-Tag header controls how search engines index and display URLs in search results. Unlike the HTML <meta name="robots"> tag, the HTTP header applies to any resource type: HTML pages, PDFs, images, videos, and other non-HTML files where a meta tag is not an option.

The header accepts one or more directives as a comma-separated list. When multiple directives conflict, search engines apply the most restrictive directive. For example, combining max-snippet:50 with nosnippet results in no snippet being shown, since nosnippet is the more restrictive rule.

Directives apply to all crawlers by default. To target a specific crawler, prefix the directives with the crawler name followed by a colon. Multiple X-Robots-Tag headers with different crawler targets are allowed in a single response.

The header is supported by major search engines including Google and Bing. Yandex supports a subset of directives.

Directives

noindex

The noindex directive prevents the URL from appearing in search results. Without this directive, the default behavior is index, meaning the URL is eligible for indexing.

nofollow

The nofollow directive instructs crawlers not to follow links on the page. The default behavior is follow.

noarchive

The noarchive directive requests that search engines not show a cached copy of the page. Google retired its cached page feature, so noarchive no longer affects Google results. Bing repurposed noarchive, and the separate nocache directive, to govern Copilot linking and AI-training use of the content.

nosnippet

The nosnippet directive prevents search engines from displaying a text snippet or video preview for the page in search results.

noimageindex

The noimageindex directive prevents images on the page from being indexed by image search.

none

The none directive is equivalent to specifying both noindex and nofollow, blocking the URL from search results and preventing link crawling.

all

The all directive is equivalent to specifying both index and follow. This is the default behavior when no X-Robots-Tag header is present.

max-snippet

The max-snippet directive sets the maximum character length for text snippets in search results. A value of 0 is equivalent to nosnippet. A value of -1 removes length restrictions and lets the search engine determine the snippet length. When absent, search engines determine the snippet length at their discretion.

max-image-preview

The max-image-preview directive controls the maximum size of image previews in search results. Accepted values: none (no image preview), standard (default-sized preview), and large (preview up to viewport width).

max-video-preview

The max-video-preview directive limits video preview duration in seconds. A value of 0 allows only a static image. A value of -1 removes duration restrictions.

notranslate

The notranslate directive prevents search engines from offering a translation of the page in search results.

indexifembedded

The indexifembedded directive allows content to be indexed when embedded via an iframe, even when noindex is also set. This applies only to the embedded URL. The embedding page is unaffected.

unavailable_after

The unavailable_after directive specifies a date and time after which the URL is removed from search results. Google accepts any widely adopted date format, including ISO 8601 and the older email-style date formats.

X-Robots-Tag: unavailable_after: 31 Dec 2026 23:59:59 GMT

Example

A server preventing a PDF from being indexed. The noindex directive works at the HTTP level since PDFs have no HTML meta tag support.

X-Robots-Tag: noindex

Targeting a specific crawler alongside a general rule. The first header restricts Googlebot, while the second header applies to all crawlers, including Googlebot, where it is merged with the scoped directives.

X-Robots-Tag: googlebot: nosnippet, notranslate
X-Robots-Tag: index, follow

A server applying multiple restrictions with a snippet length limit. The noarchive directive prevents cached copies and the max-snippet directive limits snippet length to 100 characters.

X-Robots-Tag: noarchive, max-snippet:100

Troubleshooting

Indexing directives applied through X-Robots-Tag take effect only after a crawler processes the response, and misconfigurations often go unnoticed for weeks.

Page still indexed despite noindex. Search engines must crawl and process the header before removing a URL from results. This takes days to weeks depending on crawl frequency. Verify the header is present with curl -sS -o /dev/null -D- https://example.re/page and check Google Search Console > URL Inspection to confirm the directive was detected. Blocking the URL in robots.txt prevents the crawler from seeing the header at all, which keeps the page indexed.
noindex applied to wrong pages. A broad server configuration rule can attach the header to pages that need indexing. In nginx, a location block like location /staging/ { add_header X-Robots-Tag "noindex"; } must match only the intended paths. In Apache, <Directory> or <Location> blocks with Header set X-Robots-Tag "noindex" need precise path scoping. Audit active pages by crawling the site with a tool like Screaming Frog and filtering for the header.
Conflicting meta robots and X-Robots-Tag. When both the HTML <meta name="robots"> tag and the X-Robots-Tag header are present, search engines combine directives and apply the most restrictive set. A noindex in either source removes the page from results even if the other says index. Remove the conflicting directive from the source that should not restrict indexing.
noindex with follow degrades over time. Setting noindex, follow keeps the page out of results while allowing link crawling. Over extended periods search engines reduce crawl frequency on noindexed pages, making the follow directive unreliable. Move links that need crawling to an indexed page instead of relying on a noindexed page to pass link signals.
Multiple X-Robots-Tag headers are additive. Sending more than one X-Robots-Tag header in a response does not override earlier values. Search engines merge all directives. A response containing both X-Robots-Tag: index and X-Robots-Tag: noindex results in noindex winning. Check for unintended additions from reverse proxies, CDNs, or application frameworks by inspecting the full response with curl -I.
Verifying directives in Google Search Console. URL Inspection shows the rendered page, detected meta tags, and HTTP headers. The Page indexing report flags pages excluded by noindex. Use these tools to confirm the header is reaching Googlebot and producing the expected indexing outcome. Changes take effect after the next crawl and processing cycle.