Signature-Agent
The HTTP Signature-Agent request header is an unofficial, Shopify-specific header identifying the key directory responsible for a signed HTTP request. Sent as part of Shopify's Web Bot Auth system alongside the Signature and Signature-Input headers to authenticate crawlers and automated tools accessing a Shopify storefront.
Usage
Shopify stores sit behind Cloudflare's bot management layer, which blocks or rate-limits unrecognized automated traffic. SEO crawlers, auditing tools like Screaming Frog and Sitebulb, and custom scripts need a way to prove they are authorized before the edge lets requests through.
Web Bot Auth solves this by attaching cryptographic signatures to every request. The protocol builds on HTTP Message Signatures and requires three HTTP headers: Signature-Agent, Signature, and Signature-Input. Shopify generates all three values in the admin panel under Online Store > Preferences > Crawler access. Merchants copy the values into the custom headers configuration of their crawling tool, and Shopify's edge verifies every incoming request against the registered signature before allowing access without throttling.
The Signature-Agent value itself is a URL pointing to
the signer's public key directory. For Shopify-generated
signatures, the value is always "https://shopify.com"
because Shopify hosts the key directory on behalf of the
merchant. The surrounding double quotes are required by
the protocol.
Values
Key directory URL
The value is a quoted URL string referencing the
.well-known/http-message-signatures-directory endpoint
of the signing authority. In Shopify's implementation,
this is always "https://shopify.com". The edge resolves
this URL to retrieve the public keys needed to verify the
accompanying Signature.
Example
A crawler authenticating against a Shopify store sends all three Web Bot Auth headers with every request. The Signature-Agent identifies Shopify as the signing authority, Signature-Input describes the signed components and parameters, and Signature carries the cryptographic proof.
GET /collections/all HTTP/1.1
Host: example.myshopify.com
Signature-Agent: "https://shopify.com"
Signature-Input: sig1=();created=1720000000;expires=1720003600;keyid="abc123def456";tag="web-bot-auth"
Signature: sig1=:BASE64ENCODEDSIGNATUREVALUE:
The Signature-Input line declares a signature label
sig1 with a creation timestamp, an expiration window,
a key identifier matching the registered crawler key, and
the web-bot-auth tag. The Signature line carries the
corresponding Ed25519 cryptographic value encoded in
Base64.
Tools like Screaming Frog and Sitebulb have built-in
fields for these headers. In Screaming Frog, the values
are entered under Configuration > HTTP Header. In
Sitebulb, the fields appear under
Crawler Settings > Authentication > Shopify Settings.
The Signature-Agent== value is often pre-filled by the
tool since the value is always "https://shopify.com"
for Shopify stores.
Signatures created in the Shopify admin expire after a maximum of three months. Once expired, the edge rejects requests carrying stale signatures, and new values need to be generated and reconfigured in the crawling tool.
Takeaway
The Signature-Agent header identifies the signing authority in Shopify's Web Bot Auth system. Together with Signature and Signature-Input, the header authenticates crawlers and automated tools so they access Shopify storefronts without being blocked or rate-limited.