wiki:ESIfeatures

ESI Features

Varnish implementes a subset of the  ESI Language 1.0 defined by W3C, this document lays out some of the thoughts and rationale for choices made and advice for usage of these features.

Why not full ESI support ?

A fair bit of what ESI offers, is in the ESI language because Akamais customers cannot configure the Akamai edge proxies any other way than "in-band". While this is perfectly sensible for the Akamai business model, it is of little relevance in a Varnish context, where the content-provider is in control of the Varnish servers.

Roughly speaking, ESI consists of three parts, of which Varnish so far, implements only one:

  • <esi:include>, <esi:remove> and <!--esi ... -->
  • Content substitution based on variables and cookies
  • Caching policy control

Varnish implements caching policy control in the VCL programming language, so the last bullet is already catered for.

Content substitution is mostly a cosmetic feature where messages like "You are logged in as Samuel B. Kennedy" can be added to otherwise unchanged documents, or where, using Akamais geolocation service, content can be manipulated relative to the assumed geographical position of the client. For now we have deemed this feature uninteresting, but adding it is just a matter of programming.

The first bullet is the interesting one because <esi:include> is to web content what #include is to C-language: A handy way to categorize different bits and pieces.

How does <esi:include> work ?

Imagine a hypothetical newsoutlets webpages. Most pages contain one article decorated by ads, a "hot news" box, possibly an info box with the weather and so on.

Without <esi:include> the TTL for each of these articles must be kept fairly low, to keep the decorations, in particular the "hot news" box relevant, since it is inlined in the cached object for this article.

ESI:include allows the TTL for the article to be, in principle infinity, because the article object does not contain the "hot news" box, but instead an instruction to varnish about what and where to insert it: <esi:include src="/news/hot_news.html"/>

Varnish handles the article object separately from the "hot news", and therefore the article object can have a TTL of a year while the "hot news" object has a TTL of one minute, and whenever a client requests the article, the news will be up to date, without the need to retrieve the article text from the CMS database.

An <esi:include> example

/cgi-bin/date.cgi contains:

#!/bin/sh

echo 'Content-type: text/html'
echo ''
date "+%Y-%m-%d %H:%M"

test.html contains:

<HTML>
<BODY>
The time is: <esi:include src="/cgi-bin/date.cgi"/>
at this very moment.
</BODY>
</HTML>

And finally, in your VCL code you have:

sub vcl_fetch {
    if (req.url == "/test.html") {
        esi;  /* Do ESI processing */
        set obj.ttl = 24 h;
    } elseif (req.url == "/cgi-bin/date.cgi") {
        set obj.ttl = 1m;
    }
}

Note that this VCL is for varnish 2.0. The ESI section in the  documentation describes how this is done in 3.0.

Things to remember

  • Do not esi process all your web content, in particular not binary objects like images, since they could become garbled as random byte strings match the <esi:... syntax.
  • Remember the trailing slash in <esi:include src="something.html"/>, it must be a self-closing XML element.
  • Look out for ESI_xmlerror records in the varnishlog output, they will tell you the bytenumber in the file, if trouble is found during the ESI parsing.
  • The target of <esi:include> must not be handled as pipe, the object will be ignored in that case.
  • You can have recursive <esi:include> constructs, but their depth is limited by the parameter max_esi_includes defaulting to 5.
  • You can have any number of <esi:include instructions in one document.
  • At the moment, only the src-Attribute is supported, alt and onerror are silently ignored.