X-OpenAI-Host-Hash
OpenAI's crawlers attach the X-OpenAI-Host-Hash unofficial request header when fetching pages. The value is an opaque token derived from the target host, sent alongside the crawler's other identifying headers.
Usage
OpenAI documents its crawlers, GPTBot, OAI-SearchBot, and ChatGPT-User, without documenting this header, so the observed behavior is the available record. Probe data from verified crawler traffic shows GPTBot sending the header on every sampled request and OAI-SearchBot on a majority, paired with a From header carrying an OpenAI contact address.
The name points at the purpose: a hash of the requested host, useful to the crawler's own infrastructure for routing or deduplicating work per site. Nothing in the header asks the server to act, and servers serve requests carrying the header the same way as any other crawler request.
For verifying that a request claiming to be an OpenAI crawler is genuine, the header offers no guarantee, since any client sets arbitrary request headers. Reverse DNS checks against OpenAI's published IP ranges remain the verification path, the same approach User-Agent verification uses for other crawlers.
Example
A GPTBot request carrying the header alongside the crawler's standard identification.
GET /articles/pricing HTTP/1.1
Host: www.example.re
User-Agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot
From: gptbot(at)openai.com
X-OpenAI-Host-Hash: 442243811
Accept-Encoding: gzip, br, deflate
The shown value is illustrative, since OpenAI publishes no format. The name reads as a per-host derivation rather than a per-request identifier, which matches the header's pairing with per-request identifiers like X-Request-ID on sibling OpenAI clients.
Crawlers and X-OpenAI-Host-Hash
Verified crawler probe
Probe data from verified crawler traffic (September 2026 snapshot) shows which bots send X-OpenAI-Host-Hash when requesting pages. Yes marks the header on at least nine of ten sampled requests from the bot, Sometimes marks a smaller share, and No marks absence.
| Crawler | Sends X-OpenAI-Host-Hash |
|---|---|
| Amazonbot | No |
| Applebot | No |
| Baiduspider | No |
| Bingbot | No |
| ClaudeBot | No |
| DuckDuckBot | No |
| GPTBot | Yes |
| Googlebot | No |
| Googlebot (smartphone) | No |
| Googlebot-Image | No |
| Meta-ExternalAgent | No |
| Meta-WebIndexer | No |
| OAI-SearchBot | Sometimes |
| SeznamBot | No |
| YandexBot | No |
| YandexFavicons | No |