HTTP Caching

HTTP caching stores responses so future requests for the same resource are served from a local or intermediate copy instead of the origin server. Caches appear at every layer of the HTTP chain: inside the browser (private cache), at corporate proxies (shared cache), and across CDN edge nodes (reverse cache). The rules governing what gets stored, how long a stored response stays usable, and when revalidation with the origin is required are all part of the HTTP caching specification.

Cache types

Private cache

A private cache belongs to a single client, typically the browser. Stored responses are available only to the user who triggered the original request. Private caches are allowed to store responses marked with the Cache-Control private directive, including responses containing personalized content like account pages or API results with session cookies.

Shared cache

A shared cache sits between clients and the origin, serving stored responses to multiple users. Corporate forward proxies and CDN edge servers are shared caches.

Shared caches face stricter rules than private caches. A response with a Cache-Control: private directive must not be stored by a shared cache. Responses to requests containing an Authorization header require an explicit public, s-maxage, or must-revalidate directive before a shared cache stores them.

The s-maxage directive overrides max-age in shared caches and implies proxy-revalidate.

Freshness

The freshness model determines whether a stored response is usable without contacting the origin. Freshness is defined as a comparison:

response_is_fresh =
  freshness_lifetime > current_age

Freshness lifetime

A cache evaluates freshness lifetime in this order of precedence:

1. s-maxage directive (shared caches only)
2. max-age directive
3. Expires header value minus the Date header value
4. Heuristic freshness (when none of the above exist)

The max-age directive is expressed in seconds:

Cache-Control: max-age=3600

This response stays fresh for one hour after the origin generated the response.

Heuristic freshness

When a response carries no explicit expiration information (no max-age, no s-maxage, no Expires), a cache is permitted to assign a heuristic freshness lifetime. A common approach is using 10% of the interval between the Last-Modified date and the response generation time.

A response with Last-Modified set to 10 days ago and no other cache directives gets a heuristic freshness lifetime of roughly one day.

Current age

The Age header conveys a cache's estimate of how many seconds have passed since the origin generated or validated the response. When a cache forwards a stored response, the Age header tells downstream clients and caches how old the response is.

Age: 120

This response left the origin two minutes ago (or was revalidated two minutes ago). Clients subtract the Age value from the freshness lifetime to determine remaining freshness.

Validation

When a stored response becomes stale, the cache does not discard the response. Instead, the cache sends a conditional request to the origin to check whether the resource changed. This process is called revalidation.

Validators

Two validator mechanisms exist:

• ETag: An opaque identifier representing a specific version of the resource. Strong ETags guarantee byte-for-byte equivalence. Weak ETags (prefixed with W/) signal semantic equivalence.

• Last-Modified: A timestamp indicating when the resource last changed. Has one-second resolution, making ETags the more reliable validator for rapidly changing resources.

Conditional request headers

The cache attaches validators to the revalidation request:

• If-None-Match sends the stored ETag. If the origin has the same ETag, the response has not changed.
• If-Modified-Since sends the stored Last-Modified date. If the resource has not changed since the date, the origin confirms freshness.

When both headers are present, If-None-Match takes precedence.

A successful revalidation returns 304 Not Modified with no body. The cache updates the stored metadata (headers, freshness lifetime) and continues serving the stored response. A failed revalidation returns the new representation with a 200 status.

Invalidation

Unsafe methods like POST, PUT, and DELETE change server state. When a cache receives a non-error response to an unsafe request, the cache must invalidate stored responses for the target URI. The cache also invalidates responses for URIs in the Location and Content-Location headers if they share the same origin.

Invalidation marks stored responses as requiring revalidation. Not every cache in the chain sees the unsafe request, so caches on other paths retain their stored responses until those entries go stale naturally or receive their own invalidation signals.

Cache Groups

Cache Groups provide a mechanism for grouping related cached responses so a single unsafe request invalidates an entire set of resources.

The Cache-Groups response header assigns a response to one or more named groups. The value is a list of case-sensitive strings.

HTTP/1.1 200 OK
Cache-Groups: "product-listings", "homepage"

The Cache-Group-Invalidation response header triggers invalidation of all cached responses belonging to the named groups. The header is processed only on responses to unsafe methods like POST or PUT.

HTTP/1.1 200 OK
Cache-Group-Invalidation: "product-listings"

After receiving this response, a cache invalidates all stored responses tagged with the "product-listings" group from the same origin. The invalidation does not cascade: invalidated responses do not trigger further group-based invalidations.

Cache Groups operate within a single cache and apply only to responses sharing the same origin. Group names are opaque identifiers with no inherent structure.

Cache-Control directives

The Cache-Control header carries directives controlling cache storage, reuse, and revalidation. Directives appear in both requests and responses.

Response directives

Directive	Effect
max-age=N	Response is fresh for N seconds
s-maxage=N	Overrides max-age in shared caches; implies proxy-revalidate
no-cache	Cache stores the response but must revalidate before every reuse
no-store	Cache must not store any part of the response
public	Any cache stores the response, even for authenticated requests
private	Only private caches store the response
must-revalidate	Once stale, the cache must revalidate before reuse; serves 504 on failure
proxy-revalidate	Same as must-revalidate for shared caches only
no-transform	Intermediaries must not alter the response body
must-understand	Cache stores the response only if the status code semantics are understood
immutable	Response body does not change while fresh; skip revalidation on user-initiated reload
stale-while-revalidate=N	Serve stale for up to N seconds while revalidating in the background
stale-if-error=N	Serve stale for up to N seconds when revalidation encounters a [[500

Request directives

Directive	Effect
max-age=N	Accept a response no older than N seconds
max-stale[=N]	Accept a stale response, optionally no more than N seconds past expiry
min-fresh=N	Accept a response fresh for at least N more seconds
no-cache	Do not serve from cache without revalidating first
no-store	Do not store the request or response
no-transform	Intermediaries must not alter the body
only-if-cached	Return a stored response or 504

The Vary header

The Vary header tells caches to store separate variants of the same URL based on specific request header values. A response with Vary: Accept-Encoding instructs the cache to key stored entries on the Accept-Encoding header. A client requesting br encoding and a client requesting gzip each get their own cached copy.

Vary: Accept-Encoding, Accept-Language

Without Vary, a cache serves the same stored response to all clients regardless of request headers. This causes problems when the origin returns different representations based on encoding, language, or other negotiated features.

Vary: * effectively disables caching. Every request is treated as unique.

Stale response extensions

Two Cache-Control extensions exist for handling stale responses gracefully.

stale-while-revalidate

Cache-Control: max-age=600, stale-while-revalidate=30

This directive permits a cache to serve a stale response immediately while revalidating in the background. The response above stays fresh for 600 seconds, then the cache serves stale content for up to 30 additional seconds while fetching a fresh copy. After the 30-second window, the cache blocks on revalidation like normal.

Background revalidation hides latency from clients. The tradeoff is a brief window where clients receive slightly outdated content.

stale-if-error

Cache-Control: max-age=600, stale-if-error=86400

When revalidation fails with a 5xx server error, the cache falls back to the stale response for up to the specified number of seconds. This keeps sites functional during origin outages. The response above allows serving stale content for up to 24 hours during errors.

CDN and edge caching

A CDN (content delivery network) operates a distributed network of shared caches at edge locations close to end users. CDN caches follow the same HTTP caching rules as proxy caches, with some platform-specific extensions.

Cache keys

CDN caches identify stored responses by a cache key, typically the request URL. Many CDNs extend the cache key with additional components: query string parameters, request headers (per Vary), Cookies, or geographic region. A misconfigured cache key is a common source of cache poisoning or unintended content sharing between users.

Cache purging

CDNs provide purging APIs to invalidate cached content before expiry. Purging by URL removes a single resource. Purging by tag (surrogate key) removes all responses tagged with a specific label. Tag-based purging is useful for invalidating all pages referencing a changed asset or data source.

Surrogate-Control

Some CDNs honor the Surrogate-Control header for CDN-specific caching instructions while forwarding a different Cache-Control value to downstream clients. This separates CDN cache policy from browser cache policy.

Googlebot and HTTP caching

Google's crawling infrastructure implements heuristic HTTP caching. Googlebot supports ETag / If-None-Match and Last-Modified / If-Modified-Since for cache validation when re-crawling URLs. When both validators are present, Googlebot uses the ETag value as the HTTP standard requires.

The Cache-Control max-age directive helps Googlebot determine how often to re-crawl a URL. A page with a long max-age is re-fetched less frequently, while a page with a short max-age or no-cache signals the content changes often and warrants more frequent visits.

Common caching patterns

Versioned assets (cache forever)

Static assets with a fingerprint or version string in the URL are safe to cache indefinitely.

Cache-Control: max-age=31536000, immutable

The URL /assets/app.d9f8e7.js changes whenever the file content changes. The immutable directive tells the browser not to revalidate even on a user-initiated reload.

HTML pages (always revalidate)

HTML pages change frequently and benefit from revalidation on every load.

Cache-Control: no-cache

The cache stores the response but checks with the origin before serving. Combined with a strong ETag, this pattern ensures fresh content with minimal transfer cost when nothing changed.

Sensitive content (never cache)

Login pages, banking portals, and other pages with private data must not be cached.

Cache-Control: no-store

Shared cache with revalidation fallback

API responses served through a CDN with graceful degradation during outages.

Cache-Control: s-maxage=300, stale-if-error=3600

The CDN caches the response for five minutes. On origin failure, the CDN serves stale content for up to one hour.

Example

A browser requests a CSS file for the first time. The origin responds with caching headers and validators.

Initial request

GET /styles/main.css HTTP/1.1
Host: www.example.re
Accept-Encoding: gzip

Initial response

HTTP/1.1 200 OK
Content-Type: text/css
Content-Encoding: gzip
Content-Length: 8420
Cache-Control: max-age=86400
ETag: "css-v42"
Last-Modified: Mon, 03 Mar 2025 10:00:00 GMT
Vary: Accept-Encoding

(compressed body)

The browser caches the response for 24 hours. The next day, the cached entry goes stale. The browser sends a conditional request.

Revalidation request

GET /styles/main.css HTTP/1.1
Host: www.example.re
Accept-Encoding: gzip
If-None-Match: "css-v42"
If-Modified-Since: Mon, 03 Mar 2025 10:00:00 GMT

Resource unchanged

HTTP/1.1 304 Not Modified
ETag: "css-v42"
Cache-Control: max-age=86400
Age: 0

The 304 response carries no body. The browser updates the cached entry metadata and resets the freshness timer. No bytes of CSS were transferred.

Resource changed

HTTP/1.1 200 OK
Content-Type: text/css
Content-Encoding: gzip
Content-Length: 9100
Cache-Control: max-age=86400
ETag: "css-v43"
Last-Modified: Tue, 04 Mar 2025 14:30:00 GMT
Vary: Accept-Encoding

(new compressed body)

The browser replaces the stored entry with the new response and begins a fresh 24-hour freshness window.

Takeaway

HTTP caching reduces latency and bandwidth by storing responses at the browser, proxy, or CDN level. Cache-Control directives govern freshness, storage, and revalidation policy. Stale responses are not discarded but revalidated through conditional requests using ETag and Last-Modified validators. The Vary header ensures caches differentiate responses by request characteristics like encoding or language. Extensions like stale-while-revalidate and stale-if-error add resilience by serving stale content during background refreshes and origin outages.

See also

• RFC 9111: HTTP Caching
• Google: HTTP caching for crawlers
• Google Search Blog: Crawling December, HTTP caching
• RFC 5861: Cache-Control Extensions for Stale Content
• RFC 9875: HTTP Cache Groups
• RFC 9110: HTTP Semantics, Section 8.8 (Validators)
• Cache-Control
• Vary
• Age
• ETag
• Last-Modified
• Expires
• If-None-Match
• If-Modified-Since
• Conditional-Requests
• Compression
• 304
• 504
• HTTP headers