Raise your hands if you’ve ever coded a web application that did any of the following:

  1. Supported more than one language, and automatically chose the language to present based on the contents of the Accept‑Language HTTP header.
  2. Generated different response formats (e.g. JSON vs XML) to REST queries based on the contents of the Accept HTTP header.
  3. Supported a “desktop” and “mobile” version of your user interface, and returned different content to the user based on examining the User‑Agent HTTP header. (Automatically redirecting mobile browsers to a different URL counts, too.)
  4. Attempted to reduce your site’s bandwidth by enabling “gzip” compression for browsers that support it.

I see a reasonable number of hands. Now, how many of you designed your application to tell caches and proxies that you were doing this? Far fewer hands, I see. Those of you with your hands still up can wander off to the buffet while the rest of us discuss the Vary HTTP header.

Let’s recall what caches (and caching proxies) do. Presuming that a cache is sitting between the user’s web browser and your web application, each time a request for a particular URL goes past, the cache will do the following:

  1. Look at the URL to see if it has seen a request/response for this specific URL.
  2. If it has, look at the content it last got back to see if it has expired.
  3. If it has not, return the cached content instead of hitting your web service again.

In some cases, the cache will modify step #3 by sending a conditional request to your web app saying “send me this content only if it’s changed since I last pulled it.” Your web app can then decide whether the content has changed and if it hasn’t, send back a 304 Not Modified response, thus saving you (and them) the bandwidth of re-downloading something that hasn’t changed.

Now look at the potential problem:

  1. John and Juanita are both located behind the same caching proxy.
  2. John hits your language-aware web app with his web browser set to ask for English.
  3. Your app looks at the Accept‑Language header, sees “English”, and returns “Hello world.” Since this page page doesn’t change often, it also tells the cache that it can hold onto the page for, say, an hour.
  4. John gets the page in English, as expected.
  5. Two minutes later, Juanita hits the same page with her web browser set to ask for Spanish. She expects to see “Hola mundo.”
  6. The cache, seeing the same URL and unexpired content, returns the content Rob originally pulled, meaning that Juanita gets the English version of the page instead of the Spanish version.

This may not be insurmountable problem, since perhaps Juanita reads English. But if it had been me instead of John, and if Juanita had gotten there first, I’d be in trouble, since I do not read Spanish. Either way, however, the two of us did not get the experience you wanted. Another name for this is “cache corruption” – the content that came back from the cache wasn’t correct. In this case, the cache didn’t realize that the two different requests should have gotten different content returned, because you didn’t tell it that the same URL could have different presentations based on the Accept‑Language header. No amount of fiddling with the local browser will fix this, since the problem is with the intermediate proxy.

Similar problems could occur if:

  • The cache inadvertently returned gzip-compressed content to a browser that couldn’t handle it. (Fortunately, most caches are pretty smart about this particular case.)
  • The cache inadvertently returned mobile content to a non-mobile browser or vice versa.
  • The cache returned XML when the client wanted JSON, or vice versa.

Enter the Vary header.

From the HTTP 1.1 specification at http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html:

The Vary field value indicates the set of request-header fields that fully determines, while the response is fresh, whether a cache is permitted to use the response to reply to a subsequent request without revalidation. For uncacheable or stale responses, the Vary field value advises the user agent about the criteria that were used to select the representation.

…

An HTTP/1.1 server SHOULD include a Vary header field with any cacheable response that is subject to server-driven negotiation. Doing so allows a cache to properly interpret future requests on that resource and informs the user agent about the presence of negotiation on that resource. A server MAY include a Vary header field with a non-cacheable response that is subject to server-driven negotiation, since this might provide the user agent with useful information about the dimensions over which the response varies at the time of the response.

Quite a mouthful. Translating a few phrases:

  • whether a cache is permitted to use the response to reply to a subsequent request without revalidation means “Whether the cache is allowed to just return cached content to the browser, or whether it has to ask your app whether or not it has changed.”
  • server-driven negotiation means “Any item whose response content can depend on something other than just the URL
  • the dimensions over which the response varies means “what other information you used to decide what content to return”

What this translates down to is:

  1. If you add the HTTP header Vary: Accept‑Language to the response, the cache knows that the content of this response may depend on the setting of the request’s Accept‑Language header.
  2. Having seen this as part of the first response, the cache will only consider returning its cached content if the second request’s Accept‑Language header matches the first request’s.

Problem solved, assuming that the cache conforms to the HTTP 1.1 specification. (We’re not going to consider non-compliant caches – for the purposes of this discussion, that’s Somebody Else’s Problem.)

I can hear the eye-rolling going on out there. Do we really have to care about stuff like this? My answer comes in three parts:

  1. I presume you care about your users’ experiences with your site. You have no way of controlling (or even knowing) if some of your users may be behind a caching proxy:
    1. Some companies implement proxies as an alternative to firewalls as a way of protecting their internal networks.
    2. Some mobile carriers implement proxies as a way of cutting down on the amount of bandwidth in and out of their internal networks.
  2. Browsers themselves implement caches. In your language-specific case, if the user changes the language setting in their browser, you want the browser to invalidate existing cached information, since it may be in the wrong language. (I grant you this is a less likely case, but it’s still valid.)
  3. Perhaps less obviously, there is an 800-pound gorilla you probably don’t want to ignore: Google. You’ve heard of them, right?

Oh, have you forgotten that Google grabs copies of your web pages and stores, er, “caches” them? I’ll bet that you were really hoping that:

  1. When a user comes across your web site in a Google search result, they see the result in the language in which they’re searching. That means that Google has to have some indication that your site chooses the language in its response dynamically.
  2. When user does a mobile Google search, the fact that you have a mobile version of your site gives you a “leg up” over non-mobile sites. That means that Google has to have some indication that your site will return different content to different browsers.

Just as with a caching proxy, you can tell Google things about your site by including Vary headers. As indicated on https://developers.google.com/webmasters/smartphone-sites/details:

The Vary HTTP header … helps Googlebot discover your mobile-optimized content faster, as a valid Vary HTTP header is one of the signals we may use to crawl URLs that serve mobile-optimized content.

This is in the section discussing the case in which you return mobile or non-mobile content based on the content of the User‑Agent header. Including a Vary: User‑Agent HTTP header tells Google that the result may depend on the browser pulling it, which they may interpret as “check with your other search bot.”

If, instead, you detect mobile browsers and redirect them to a different URL, including the same Vary: User‑Agent HTTP header in the redirect helps Google know that the redirect is browser-specific. As indicated at https://developers.google.com/webmasters/smartphone-sites/redirects:

If your site serves content or redirects users depending on the user agent string, i.e. the response varies, we strongly recommend that your server also send the Vary HTTP header on URLs that serve automatic redirects. This helps with ISP caching and is another signal for Googlebot and our algorithms to discover and understand your website’s configuration.

Now, of course, there is always a dark cloud attached to any silver lining and, as experienced web developers know too well, it goes by the initials of MSIE – our old pal Microsoft Internet Explorer.

  • MSIE 6 only understood Accept‑Encoding and User‑Agent as valid values for the Vary header. If any other content was returned, or if Accept‑Encoding was returned for uncompressed content, the browser wouldn’t cache the content at all, and would always completely re-pull it. Thus, adding a Vary header with any other content would completely defeat any attempt on your web app’s part to cache content. Fortunately MSIE 6 is pretty well dead at this point.
  • MSIE 7 exhibited the same behavior, with the exception that it would issue a conditional query for the content the next time it was needed, allowing your server to send back a 304 Not Changed. It only did this, however, if you included an ETag header. If you tried to force caching via an Expires, IE wouldn’t cache it at all. Worse, MSIE wouldn’t write the file to disk at all, which meant that certain kinds of content that depend on being re-read off disk by another program (like HTML help files – .chm or vCard files – .vcf) wouldn’t work.
  • As of IE 9, Microsoft has the following to say (see http://blogs.msdn.com/b/ie/archive/2010/07/14/caching-improvements-in-internet-explorer-9.aspx):

With Internet Explorer 9, we’ve enhanced support for key Vary header scenarios. Specifically, IE9 will no longer require server revalidation for responses which contain Vary: Accept‑Encoding and Vary: Host directives.

We can safely support these two directives because:

  1. All requests implicitly vary by Host, because the host is a component of the request URL.
  2. IE always decompresses HTTP responses in the cache, making Vary: Accept‑Encoding redundant.

Like IE6 and above, IE9 will also ignore the Vary: User‑Agent directive.

If a response contains a Vary directive that specifies a header other than Accept‑Encoding, Host, or User‑Agent (or any combination of these) then Internet Explorer will still cache the response if the response contains an ETAG header. However, that response will be treated as stale and a conditional HTTP request will be made before reuse to determine if the cached copy is valid.

Thus, if you want to do the right thing header-wise and still allow caching, MSIE forces you to use the ETag method of cache verification. This approach has some issues when you have multiple servers behind a load balancer, because some servers generate ETag values in a server-specific manner. The Apache HTTP server, for example, includes a file’s inode in its ETag generation algorithm – presumably so that two different files on the server will always generate two different ETag values without Apache having to look at the contents. This means, however, that it’s extraordinarily unlikely that two different instances of Apache will generate the same ETag value for the same file, which completely defeats the purpose in a load-balanced environment.

All that being said, it may be better to not have MSIE cache information than to potentially have some other browser, cache or caching proxy return the wrong information.

I should point out that you do have other alternatives:

  1. For language-aware sites, you could design your site so that the language is encoded into the URL, either in the host name or the path. In this case, you would only need to include the Vary: Accept‑Language header in the initial “automatically detect the language and redirect to the appropriate URL” response.
  2. Similarly, for mobile sites:
    1. You could use the same “detect and redirect” technique above only on the site’s home page.
    2. You could use a “responsive web design” approach in which the same content is returned in both mobile and non-mobile cases, but the actual presentation changes based on CSS media queries.

Both of these techniques have the advantage that they don’t lock a user into a particular presentation (you can get them between the different portions of the site by standard links), and caching within the sub-sites doesn’t depend on headers. Indeed, you don’t want the initial redirect cached, so the fact that MSIE may not doesn’t hurt you. Further, cross-site links will help ensure that the 800-pound gorilla finds all the sub-sections of your site.

Thus, your mileage may “vary,” just as at least a portion of your site’s should. (Sorry – couldn’t resist the atrocious pun.)