X-Robots-Tag

Non-HTML resources like PDFs and images lack a meta tag for controlling search engine behavior. The X-Robots-Tag unofficial response header delivers indexing and serving directives to web crawlers through HTTP headers instead.

Table of Contents

Usage

The X-Robots-Tag header controls how search engines index and display URLs in search results. Unlike the HTML <meta name="robots"> tag, the HTTP header applies to any resource type: HTML pages, PDFs, images, videos, and other non-HTML files where a meta tag is not an option.

The header accepts one or more directives as a comma-separated list. When multiple directives conflict, search engines apply the most restrictive directive. For example, combining noindex with max-snippet:-1 results in the page being excluded from search results entirely.

Directives apply to all crawlers by default. To target a specific crawler, prefix the directives with the crawler name followed by a colon. Multiple X-Robots-Tag headers with different crawler targets are allowed in a single response.

The header is supported by major search engines including Google and Bing. Yandex supports a subset of directives.

Note

robots.txt controls crawling, determining whether a search engine fetches a URL at all. X-Robots-Tag controls indexing, determining whether a fetched URL appears in search results. Blocking a URL in robots.txt prevents the crawler from seeing X-Robots-Tag directives, so the page may remain indexed based on external signals. For effective deindexing, allow crawling (remove the robots.txt block) and set X-Robots-Tag: noindex.

Bingbot directive support

Bingbot supports max-snippet, max-image-preview, and max-video-preview directives. The nocache directive is a Bing-specific synonym for noarchive. To prevent Bing image indexing, apply noindex on the image URL itself.

Note

The "X-" naming convention for HTTP headers, "X" referring to "experimental", has been deprecated and needs to be transitioned to the formal naming convention for HTTP headers.

Directives

noindex

The noindex directive prevents the URL from appearing in search results. Without this directive, the default behavior is index, meaning the URL is eligible for indexing.

nofollow

The nofollow directive instructs crawlers not to follow links on the page. The default behavior is follow.

Note

Applying noindex over a prolonged period causes search engines to eventually stop following links on the page, even if follow was never explicitly removed. The follow signal becomes unreliable once a page is consistently excluded from indexing.

noarchive

The noarchive directive prevents search engines from showing a cached copy of the page in search results. Bing also recognizes the equivalent nocache directive.

nosnippet

The nosnippet directive prevents search engines from displaying a text snippet or video preview for the page in search results.

noimageindex

The noimageindex directive prevents images on the page from being indexed by image search.

none

The none directive is equivalent to specifying both noindex and nofollow, blocking the URL from search results and preventing link crawling.

all

The all directive is equivalent to specifying both index and follow. This is the default behavior when no X-Robots-Tag header is present.

max-snippet

The max-snippet directive sets the maximum character length for text snippets in search results. A value of 0 is equivalent to nosnippet. A value of -1 removes length restrictions and lets the search engine determine the snippet length. When absent, search engines determine the snippet length at their discretion.

max-image-preview

The max-image-preview directive controls the maximum size of image previews in search results. Accepted values: none (no image preview), standard (default-sized preview), and large (preview up to viewport width).

max-video-preview

The max-video-preview directive limits video preview duration in seconds. A value of 0 allows only a static image. A value of -1 removes duration restrictions.

notranslate

The notranslate directive prevents search engines from offering a translation of the page in search results.

indexifembedded

The indexifembedded directive allows content to be indexed when embedded via an iframe, even when noindex is also set. This applies only to the embedded URL. The embedding page is unaffected.

unavailable_after

The unavailable_after directive specifies a date and time after which the URL is removed from search results. Google documents the old HTTP date format for this directive.

X-Robots-Tag: unavailable_after: 31 Dec 2026 23:59:59 GMT

Example

A server preventing a PDF from being indexed. The noindex directive works at the HTTP level since PDFs have no HTML meta tag support.

X-Robots-Tag: noindex

Targeting a specific crawler while leaving the default behavior for others. The first header restricts Googlebot, while the second header applies to all other crawlers.

X-Robots-Tag: googlebot: nosnippet, notranslate
X-Robots-Tag: index, follow

A server applying multiple restrictions with a snippet length limit. The noarchive directive prevents cached copies and the max-snippet directive limits snippet length to 100 characters.

X-Robots-Tag: noarchive, max-snippet:100

Troubleshooting

Indexing directives applied through X-Robots-Tag take effect only after a crawler processes the response, and misconfigurations often go unnoticed for weeks.

  1. Page still indexed despite noindex. Search engines must crawl and process the header before removing a URL from results. This takes days to weeks depending on crawl frequency. Verify the header is present with curl -I https://example.re/page and check Google Search Console > URL Inspection to confirm the directive was detected. Blocking the URL in robots.txt prevents the crawler from seeing the header at all, which keeps the page indexed.

  2. noindex applied to wrong pages. A broad server configuration rule can attach the header to pages that need indexing. In nginx, a location block like location /staging/ { add_header X-Robots-Tag "noindex"; } must match only the intended paths. In Apache, <Directory> or <Location> blocks with Header set X-Robots-Tag "noindex" need precise path scoping. Audit active pages by crawling the site with a tool like Screaming Frog and filtering for the header.

  3. Conflicting meta robots and X-Robots-Tag. When both the HTML <meta name="robots"> tag and the X-Robots-Tag header are present, search engines combine directives and apply the most restrictive set. A noindex in either source removes the page from results even if the other says index. Remove the conflicting directive from the source that should not restrict indexing.

  4. noindex with follow degrades over time. Setting noindex, follow keeps the page out of results while allowing link crawling. Over extended periods search engines reduce crawl frequency on noindexed pages, making the follow directive unreliable. Move links that need crawling to an indexed page instead of relying on a noindexed page to pass link signals.

  5. Multiple X-Robots-Tag headers are additive. Sending more than one X-Robots-Tag header in a response does not override earlier values. Search engines merge all directives. A response containing both X-Robots-Tag: index and X-Robots-Tag: noindex results in noindex winning. Check for unintended additions from reverse proxies, CDNs, or application frameworks by inspecting the full response with curl -I.

  6. Verifying directives in Google Search Console. URL Inspection shows the rendered page, detected meta tags, and HTTP headers. The Coverage report flags pages excluded by noindex. Use these tools to confirm the header is reaching Googlebot and producing the expected indexing outcome. Changes take effect after the next crawl and processing cycle.

See also

Last updated: April 4, 2026