Dynamic backend selection for ESI fragments

[Lukas Rieder]

Hello, 

I am using Varnish as an ESI backend, and therefore I've stumbled upon one constraint.
Varnish backends must be known at configuration time. I have not found a possibility to create a backend "on the fly" or let a DNS server decide about backends.

The problem is the following:
I use Varnish for caching and resolving ESI fragments with absolute URLs.
This works well for all defined backends. But the problem in my project is, that new services are being added or removed all the time.
Now it would be amazing if I could find a way, how to resolve ESI fragments dynamically through a DNS server.

I know that there is a good reason why ESI fragments are bound to the available backends. It would be a security issue to allow ESI fragments point everywhere.
But imagine a setup with a DNS server who manages the reachable services and rejects to resolve requests for unknown hosts.

At the moment I am running the following workaround:

1) The ESI fragment defines a relative path i.e.:

  <esi:include src="/fragments/service.host.com/path/uri" />

2) Varnish forwards requests with urls matching /fragments/ to Nginx (the `proxy` backend in the vcl). Here is the key part of this configuration:

  sub vcl_recv {

  if (req.url ~ "^/fragments/([a-z0-9.-]+)/?(.*)$") {

    set req.http.Host = regsub(req.url, "^/fragments/([a-z0-9.-]+)/?(.*)$", "\1");
    set req.url = regsub(req.url, "^/fragments/([a-z0-9.-]+)/?(.*)$", "/\2");
    set req.backend = proxy;

  } else {

  set req.backend = default;

  }

  }

3) Nginx resolves the request through our own DNS server. The DNS server only resolves known hosts and does not forward DNS requests.


This approach works for now as a proof of concept, but there is a lot of overhead.

Thank you for reading. Please tell me your thoughts.
Or maybe you know how to write a VMOD that can control backend selection. I have never written a VMOD so far, and unfortunately I do not know much about the backend selection internals.
But with some help, I'd be willing to implement a VMOD (and release it as open source for sure).


Lukas Rieder
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20111207/716fc9b6/attachment-0003.html>