Signature-Agent

The HTTP Signature-Agent request header is an unofficial header identifying the key directory that holds the public keys for a signed HTTP request. It is part of the Web Bot Auth protocol, sent alongside Signature and Signature-Input to authenticate automated clients such as crawlers. Cloudflare implements Web Bot Auth across the sites it fronts, including Shopify storefronts.

Table of Contents

Usage

Content delivery networks and bot management layers block or rate-limit unrecognized automated traffic. SEO crawlers, auditing tools like Screaming Frog and Sitebulb, and custom scripts need a way to prove they are authorized before the edge lets requests through. Shopify stores, which sit behind Cloudflare, are one common place this header appears.

Web Bot Auth solves this by attaching cryptographic signatures to every request. The protocol builds on HTTP Message Signatures and requires three HTTP headers: Signature-Agent, Signature, and Signature-Input. Shopify generates all three values in the admin panel under Online Store > Preferences > Crawler access. Merchants copy the values into the custom headers configuration of their crawling tool, and Shopify's edge verifies every incoming request against the registered signature before allowing access without throttling.

The Signature-Agent value is a URL pointing to the signer's public key directory. Each signing authority uses its own URL. Shopify hosts the key directory on behalf of its merchants, so signatures Shopify generates carry "https://shopify.com". The surrounding double quotes are part of the value.

Values

Key directory URL

The value is a quoted URL string referencing the .well-known/http-message-signatures-directory endpoint of the signing authority. The canonical protocol example is "https://signature-agent.test". In Shopify's implementation, the value is "https://shopify.com". The edge resolves this URL to retrieve the public keys needed to verify the accompanying Signature.

Example

A crawler authenticating against a Shopify store sends all three Web Bot Auth headers with every request. The Signature-Agent identifies Shopify as the signing authority, Signature-Input describes the signed components and parameters, and Signature carries the cryptographic proof.

GET /collections/all HTTP/1.1
Host: example.myshopify.com
Signature-Agent: "https://shopify.com"
Signature-Input: sig1=();created=1720000000;expires=1720003600;keyid="abc123def456";tag="web-bot-auth"
Signature: sig1=:BASE64ENCODEDSIGNATUREVALUE:

The Signature-Input line declares a signature label sig1 with a creation timestamp, an expiration window, a key identifier matching the registered crawler key, and the web-bot-auth tag. The Signature line carries the corresponding Ed25519 cryptographic value encoded in Base64.

Tools like Screaming Frog and Sitebulb have built-in fields for these headers. In Screaming Frog, the values are entered under Configuration > HTTP Header. In Sitebulb, the fields appear under Crawler Settings > Authentication > Shopify Settings. The Signature-Agent== value is often pre-filled by the tool, since for Shopify stores it is "https://shopify.com".

Signatures created in the Shopify admin expire after a maximum of three months. Once expired, the edge rejects requests carrying stale signatures, and new values need to be generated and reconfigured in the crawling tool.

See also

Last updated: June 5, 2026