wiki:VarnishAndWordpress

For The Impatient: Preparing Varnish/Wordpress? for a Slashdotting in 60 seconds or less…

NOTE: the information on this page is old - new reference material can be found at VCL Examples

  • Drop all cookies when processing non-Wordpress resources. This will (probably) stop any widgets/plugins you've got that rely on cookies from doing whatever they normally do (it shouldn't *break* them though, they just won't have any data available to display custom info for that user -- something you'll care very little about if you start getting 2,000+ requests a second, degrade gracefully and all that).
    # Drop any cookies sent to Wordpress.
    sub vcl_recv {
    	if (!(req.url ~ "wp-(login|admin)")) {
    		unset req.http.cookie;
    	}
    }
    
    # Drop any cookies Wordpress tries to send back to the client.
    sub vcl_fetch {
    	if (!(req.url ~ "wp-(login|admin)")) {
    		unset beresp.http.set-cookie;
    	}
    }
    
  • Deactivate as many plugins as possible. Even the most benign plugins can turn a snappy blog into a sluggish blog (a slog?). Even if you haven't got any plugin widgets on your sidebars, just having a plugin activated will typically result in it spitting out extra HTML on every page load. That HTML almost always includes fetching crap (.js, .pngs, etc) from other sites, which slows down page loads considerably. Plugins all seem to like hijacking <head></head> as well, so you can easily get into the situation where 5+ seconds have passed before the user's browser has even moved onto compositing the page. This is what leads to a sluggish feeling, regardless of cache hit ratio.
  • Install and activate a PHP cache like WP-Cache 2 (trivial to install). If you do use WP-Cache 2, make sure you actually 'enable' it via its options page after you 'activate' it on Wordpress's plugin page. (I had it activated, but not enabled, for at least two or three weeks before realising.)
  • Monitor cache hits via 'varnishstat'. It should be very high, especially if you're getting Slashdotted.
    • If your cache hit is low, your cache miss is even lower, and your incoming requests/connections are going through the roof, then Varnish isn't able to cache the content Wordpress is creating for some reason other than cookies.
    • If your cache hit is low and your cache miss is high, then you're getting lots of diverse hits and your cache hasn't warmed up yet. Two options in that case: time, or a bigger backing store if the one you've got is too small. (Granted, changing that will require a restart, which will leave you with an even colder cache.)

Follow those steps and you'll be as well equipped for extremely high load against your Varnish-accelerated Wordpress website as you can be within 60 seconds. (Make sure to read the general Varnish tuning recommendations when your server stops melting.)

Read on for a more elaborate discussion...

Wordpress the Cookie Monster…

What happens when you stick an out-of-the-box Varnish in front of an out-of-the-box Wordpress site? Not much. Wordpress is a hormonally-imbalanced Cookie Monster -- every request and response firing back and forth between peoples' browsers and your Varnished Wordpress site will be laden with (potentially useless) cookies that prevent Varnish from caching any content being generated. (This page provides greater detail on how cookies affect Varnish's ability to cache output.)

This page looks at the steps you can take to improve both the cache hit of Varnish when placed in front of Wordpress, as well as removing the 'sluggish' feeling that can haunt a site even when your cache hit ratio is excellent.

First Step: Observe varnishstat w/ Default Configuration

varnishstat is extremely useful for monitoring caching performance, amongst other things. In this situation, we run it just to confirm that we're getting woeful cache hit ratios with an out-of-the-box configuration of Varnish and Wordpress.

Restart varnishd and your Wordpress server (i.e. httpd/lighttpd), then load your site in your browser. Hit refresh/reload a few times. You should see a pretty poor cache hit ratio. Here's a snippet of what I was seeing after having an out-of-the-box setup running for a week under very light load:

4+10:37:44                                                     my.server.com
Hitrate ratio:        1        1        1
Hitrate avg:     0.0364   0.0364   0.0364
 
        2324         0.00         0.01 Client connections accepted
        6191         0.00         0.02 Client requests received
          12         0.00         0.00 Cache hits
           7         0.00         0.00 Cache hits for pass
         318         0.00         0.00 Cache misses
        6179         0.00         0.02 Backend connections success
           0         0.00         0.00 Backend connections not attempted
           0         0.00         0.00 Backend connections too many
           0         0.00         0.00 Backend connections failures
        4057         0.00         0.01 Backend connections reuses
        6151         0.00         0.02 Backend connections recycles
        ...

12 cache hits over 4 days and ~6000 requests! (That's bad, if you were wondering.)

The reason the cache hit ratio is so low is that Wordpress isn't spitting anything out that Varnish can cache because there are always cookies set on each request/response.

In my case, not only was the cache hit ratio slow, but the site felt generally sluggish. As it happens, the two aren't as intrinsically linked as you'd think, you can have a near perfect cache hit ratio, yet be stuck with a sluggish Wordpress site.

Second Step: Disable All Cookies

We need to see if Wordpress still feels sluggish with a near-perfect cache hit ratio. The quickest way to coerce Varnish to start caching all Wordpress content is to strip all cookies via the following VCL rules:

# Enter your backend Wordpress site here.
backend default {
    .host = "127.0.0.1"; # XXX CHANGE THIS
    .port = "80";        # (and maybe this)
}   

# Drop any cookies sent to Wordpress.
sub vcl_recv {
    unset req.http.cookie;
}

# Drop any cookies Wordpress tries to send back to the client.
sub vcl_fetch {
    unset beresp.http.set-cookie;
}

Restart varnishd and run the same litmus paper test you ran earlier: load your site in your browser, hit reload a few times, and monitor the cache hit ratio. It should be much, much better. The first reload will obviously trigger lots of cache misses, but after that, you should be getting near-perfect cache hit ratios.

(If you're still not getting vastly improved cache hit ratios, you're not running into the Varnish/Wordpress? cookie/caching issue this page is trying to address, and I have no idea what to suggest.)

Third Step: Blame Game

Varnish will be caching optimally at this point. Now, the big question, does your site still feel sluggish? In my case, it did. It was taking a few seconds to load some initial content; by the time it had finished its page composition, it was easily a good 5+ seconds, which didn't feel right.

Read the narrative at the top of this page for an explanation of what to expect in this situation; perfect cache hit ratios and sluggish pages point to too many plugins being loaded. Disable everything except your PHP caching plugins (like WP-Cache 2) and re-run your litmus paper tests. If the site *still* feels sluggish, even with Varnish running optimally and no plugins, then your problem lies elsewhere.

If it feels snappy with no plugins and an optimal Varnish (i.e. cache hit ratio is still very high), then your job is almost done. Just make sure you alter the VCL such that you allow cookies to be set on the login and admin pages as per the sample at the top of the page.

These notes are based on my experience with preparing  http://www.snakebite.org/ for a potential Slashdotting.

Trent Nelson