The Vary Header

The HTTP Vary header is deceptively simple in concept but profoundly complex in practice. A single misplaced Vary directive can reduce your cache hit ratio by 80%, silently degrading performance across your entire CDN infrastructure. Understanding how to use it correctly is essential for anyone responsible for web performance and CDN optimization.

What is Vary Exactly?

In HTTP caching, the fundamental rule is simple: one URL equals one cached object. But the real world is more complicated. The same URL might return different content depending on factors like client device type, language preferences, geographic location, or accepted compression format.

The Vary header solves this problem by telling caches that a single URL might serve multiple variations of content. When you send a Vary header, you're instructing caches to create separate cache entries not just based on the URL, but based on the combination of the URL plus the HTTP headers specified in the Vary directive.

For example:

Vary: Accept-Encoding, User-Agent

This tells caches that the same URL might return different content depending on both the Accept-Encoding header (usually indicating whether the client supports compression) and the User-Agent header (the browser/device making the request).

This is content negotiation in practice. Your server is saying: "I serve different content based on these headers, so cache each variation separately."

The Classic Problem: Response Compression

The most common and legitimate use of Vary is for HTTP compression. When a client supports gzip or brotli compression, it sends:

Accept-Encoding: gzip, deflate, br

Your origin server needs to decide: should I send the response compressed or uncompressed? The intelligent choice is usually to compress for modern browsers and leave uncompressed for legacy clients.

But here's the cache problem: if you send a compressed response to Client A and an uncompressed response to Client B for the same URL, those are two completely different byte sequences. If your cache stored the compressed version when Client A requested it, and then Client B requests the same URL but doesn't support compression, the cache can't serve the compressed version—it would break the client.

The solution: Vary: Accept-Encoding

This tells caches to maintain separate cache entries for compressed and uncompressed versions of the same content, serving each variant based on the client's capabilities.

Impact on Cache Hit Ratio: The Clone Attack

Now consider a different scenario. Your website has responsive design that adapts to mobile, tablet, and desktop viewports. You might be tempted to use:

Vary: User-Agent

The problem: there are roughly 20,000 different User-Agent strings in the wild. Each unique User-Agent value creates a separate cache entry. A single URL that should have a 90% cache hit ratio suddenly has a 0.005% cache hit ratio because each of 20,000 different clients sees a cache miss.

This is the "clone attack" on cache efficiency. Theoretically you're serving different content for different clients. Practically, you're destroying your cache hit ratio by creating thousands of variants that are actually identical.

The solution: don't use Vary: User-Agent for responsive design. Instead, use a combination of device detection and normalization. Many CDNs, including Transparent Edge, provide X-Device or similar headers that normalize User-Agent into a small set of meaningful variants (mobile, tablet, desktop).

Common Use Cases and Management

Accept-Encoding (Compression)

Legitimate use case. Most web servers and CDNs set this automatically. It's the exception that gets it wrong. Vary: Accept-Encoding is not just acceptable, it's essential for proper cache behavior with compression.

Origin (CORS)

When serving Cross-Origin Resource Sharing responses, different origins might require different Access-Control headers. In this case:

Vary: Origin

This is legitimate, but dangerous. Every unique Origin header value creates a separate cache entry. If you're serving a CORS resource across 100 different client domains, you now have 100 cache variants. This is usually acceptable because Origin values are limited to the specific domains making requests. But it's worth understanding the cache cost.

Accept-Language (Language)

For multilingual websites, you might serve different content based on language preference:

Vary: Accept-Language

This is reasonable if you have a limited number of language variants (5-10), but problematic if you support many languages because Accept-Language header values can be very granular (en-US, en-GB, pt-BR, etc.) and clients can send multiple values with quality indicators.

Accept

Content negotiation based on MIME type acceptance:

Vary: Accept

Less common now with REST APIs, but still relevant for APIs that serve both JSON and XML. Again, the danger is creating too many variants.

Transparent Edge Recommendations

The key principle: minimize Vary directives and normalize the headers you do vary on. Here's the technical approach:

1. Header Normalization

Before applying Vary rules, normalize incoming headers to a manageable set of variants. Instead of allowing 20,000 User-Agent strings, normalize to 3-5 device categories. Instead of all 50 Accept-Language variants, normalize to your supported language list.

2. Use Transparent Edge's X-Vary-TCDN Header

Our CDN provides X-Vary-TCDN for exactly this purpose. Instead of varying on raw client headers, you vary on our normalized header that represents meaningful distinctions:

sub vcl_recv {
    set req.http.X-Vary-TCDN = req.http.X-Vary-TCDN + ";" + req.http.X-Device;
    # Only 3 variants per URL (mobile, tablet, desktop), not 20,000.
}

This VCL code runs during request processing and sets up device normalization. Instead of a Vary header creating thousands of cache entries based on User-Agent, you get exactly 3 device variants, each with proper cache hit ratios.

3. Cache Busting for Intentional Variants

When you genuinely need multiple variants, use cache-busting techniques like URL parameters rather than Vary headers:

https://example.com/page?device=mobile
https://example.com/page?device=desktop

Different URLs = different cache entries. This eliminates the cache efficiency problems that Vary creates while being completely transparent about which variant is being served.

The Performance Impact of Careless Vary

To illustrate the cost of careless Vary usage: a website serving 10 million requests per day with a single careless Vary: User-Agent header might experience:

• Cache hit ratio drops from 85% to 5%
• Origin server load increases 10x (90% fewer requests served from cache)
• Latency increases because most requests require origin fetch
• Bandwidth costs increase dramatically
• User experience degrades noticeably

This isn't theoretical. It's a common mistake that many organizations discover only after performance degrades significantly.

The solution is simple: be intentional about Vary. Use it only when you have genuine content variations, normalize headers to minimize variants, and leverage CDN-provided normalization headers like X-Vary-TCDN to keep your cache hit ratios high while still serving appropriate variants to different clients.