wiki:FAQ/Compression

This document is out of date and will be removed shortly. Please see  https://www.varnish-cache.org/docs/trunk/users-guide/compression.html

Compression support in Varnish

Varnish itself does not compress or decompress objects, although that has been on our wish list for quite a while. However, it will operate correctly with backends that support compression.

Why should I use compression?

Compression reduces the bandwidth required to transmit an object. The compression algorithms used on the web require so little CPU that the extra time your browser spends decompressing data is more than offset by the time saved by transmitting less data. Therefore, compression will not only reduce your bandwidth bill and allow more clients to be served in the same amount of time, but it will also make your site snappier.

How does it work?

Compression is implemented using HTTP's content negotiation mechanism. This is a general mechanism that allows a browser to inform the server of certain preferences, such as a preferred language, a preferred character encoding (UTF-8 rather than ISO-8859-1), a preferred file format (PNG rather than GIF), or a preferred transfer encoding (which is where compression comes in).

The server is free to ignore the client's preferences if it does not have a version of the requested object that matches them. In practice, very few servers actually implement content negotiation, except for compression. The server indicates this with the Vary header, as described below.

The HTTP RFC specifies two compression algorithms: deflate, which is the algorithm used by PKZIP, WinZIP etc., and gzip, which is (obviously) the algorithm used by the gzip compression tool.

The browser indicates its support and / or preference for a particular compression algorithm (gzip, for instance) by adding the following header to its request:

Accept-Encoding: gzip

Or, if it supports more than one algorithm:

Accept-Encoding: gzip, deflate

The server indicates that it cares about the content of the Accept-Encoding header by adding the following header to its response:

Vary: Accept-Encoding

This tells any proxy along the path between the client and the server - including Varnish - that the content of the requested object will vary according to the content of the client's Accept-Encoding header.

How does Varnish handle this?

When Varnish requests an object from the backend, and the response includes a Vary header, Varnish will tag the object so it remembers to pay close attention to the corresponding header in any subsequent request for the same object.

Can't this be done in vcl_hash?

At the point where vcl_hash is called, Varnish does not yet know whether the object has a Vary header, and therefore whether it is meaningful to include Accept-Encoding in the hash string.

If Varnish did not have native Vary support, a poor man's version could be implemented in vcl_hash, but it would greatly reduce cache efficiency.

So what is the correct way to handle it?

The simple way is to do nothing. Varnish will automatically take Vary into account and cache different versions of the same object.

I thought you said this was complicated?

Well, yes. Even though there are few possible values for Accept-Encoding, Varnish treats them literally rather than semantically, so even a small difference which makes no difference to the backend can reduce cache efficiency by making Varnish cache too many different versions of an object.

A specific example of this is that Firefox usually sends the following:

Accept-Encoding: gzip,deflate

while Microsoft Internet Explorer sends the following:

Accept-Encoding: gzip, deflate

This single extra space can make a huge difference for a large site. There is much to gain by adding code to vcl_recv to normalize Accept-Encoding. In addition, IE doesn't support deflate properly, so we don't want to send deflated content to IE.

A good way to do this is as follows:

    if (req.http.Accept-Encoding) {
        if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
            # No point in compressing these
            remove req.http.Accept-Encoding;
        } elsif (req.http.Accept-Encoding ~ "gzip") {
            set req.http.Accept-Encoding = "gzip";
        } elsif (req.http.Accept-Encoding ~ "deflate" && req.http.user-agent !~ "MSIE") {
            set req.http.Accept-Encoding = "deflate";
        } else {
            # unkown algorithm
            remove req.http.Accept-Encoding;
        }
    }

This will reduce the Accept-Encoding header to either gzip or deflate, with a preference for the gzip algorithm, which generally compresses better than deflate. It will also remove the header entirely if it contains neither gzip nor deflate. Thus, there will be at most three versions of each object in the cache: one uncompressed, one compressed with gzip, and one compressed with deflate.

In addition, it will unconditionally remove the Accept-Encoding header for certain file types which are already compressed and therefore shouldn't be compressed again. This may be redundant, as the backend server is likely to be configured to ignore Accept-Encoding (and therefore not send out Vary) for these file types.