<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>cc'ing the varnish dev list for comments...</div><div><br></div><div>On Feb 27, 2009, at 1:33 PM, Cloude Porteus wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>John,<br>Goodto hear from you. You must be slammed at Twitter. I'm happy to<br>hear that ESI is holding up for you. It's been in my backlog since you<br>mentioned it to me pre-Twitter.<br><br>Any performance info would be great.<br><br></div></blockquote><div><br></div><div>Any comments on our setup are welcome. You may also choose to call us crazypants. Many, many thanks to Artur Bergman of Wikia for helping us get this configuration straightened out.</div><div><br></div><div>Right now, we're running varnish (on search) in a bit of a non-standard way. We plan to use it in the normal fashion (varnish to Internet, nothing inbetween) on our API at some point. We're running version 2.0.2, no patches. Cache hit rates range from 10% to 30%, or higher when a real-time event is flooding search.</div><div><br></div><div>2.0.2 is quite stable for us, with the occasional child death here and there when we get massive headers coming in that flood sess_workspace. I hear this is fixed in 2.0.3, but haven't had time to try it yet. </div><div><br></div><div>We have a number of search boxes, and each search box has an apache instance on it, and varnish instance. We plan to merge the varnish instances at some point, but we use very low TTLs (Twitter is the real time web!) and don't see much of a savings by running less of them.</div><div><br></div><div>We do:</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>Apache --> Varnish --> Apache -> Mongrels</div><div><br></div><div>Apaches are using mod_proxy_balancer. The front end apache is there because we've long had a fear that Varnish would crash on us, which it did many times prior to our figuring out the proper parameters for startup. We have two entries in that balancer. Either the request goes to varnish, or, if varnish bombs out, it goes directly to the mongrel. </div><div><br></div><div>We do this, because we need a load balancing algorithm that varnish doesn't support, called bybusiness. Without bybusiness, varnish tries to direct requests to Mongrels that are busy, and requests end up in the listen queue. that adds ~100-150mS to load times, and that's no good for our desired service times of 200-250mS (or less.)</div><div><br></div><div>We'd be so happy if someone put bybusiness into Varnish's backend load balancing, but it's not there yet. </div><div><br></div><div>We also know that taking the extra hop through localhost costs us next to nothing in service time, so it's good to have Apache there incase we need to yank out Varnish. In the future, we might get rid of Apache and use HAProxy (it's load balancing and backend monitoring is much richer than Apache, and, it has a beautiful HTTP interface to look at.) </div><div><br></div><div>Some variables and our decisions:</div><div><div><br></div><div><font class="Apple-style-span" face="Courier"> -p obj_workspace=4096 \</font></div><div><span class="Apple-tab-span" style="white-space: pre; "><font class="Apple-style-span" face="Courier"> </font></span><font class="Apple-style-span" face="Courier"> -p sess_workspace=262144 \</font></div><div><br></div><div>Absolutely vital! Varnish does not allocate enough space by default for headers, regexps on cookies, and otherwise. It was increased in 2.0.3, but really, not increased enough. Without this we were panicing every 20-30 requests and overflowing the sess hash.</div><div><br></div><div><font class="Apple-style-span" face="Courier"> -p listen_depth=8192 \</font></div><div><br></div><div>8192 is probably excessive for now. If we're queuing 8k conns, something is really broke!</div><div><br></div><div><font class="Apple-style-span" face="Courier"> -p log_hashstring=off \</font></div><div><br></div><div>Who cares about this - we don't need it. </div><div><br></div><div><span class="Apple-tab-span" style="white-space: pre; "><font class="Apple-style-span" face="Courier"> </font></span><font class="Apple-style-span" face="Courier"> -p lru_interval=60 \</font></div><div><br></div><div>We have many small objects in the search cache. Run LRU more often.</div><div><br></div><div><font class="Apple-style-span" face="Courier"> -p sess_timeout=10 \</font></div><div><br></div><div>If you keep session data around for too long, you waste memory.</div><div><br></div><div><span class="Apple-tab-span" style="white-space: pre; "><font class="Apple-style-span" face="Courier"> </font></span><font class="Apple-style-span" face="Courier"> -p shm_workspace=32768 \</font></div><div><br></div><div>Give us a bit more room in shm </div><div><font class="Apple-style-span" face="Courier"><br></font></div><div><font class="Apple-style-span" face="Courier"> -p ping_interval=1 \</font></div><div><br></div><div>Frequent pings in case the child dies on us. </div><div><font class="Apple-style-span" face="Courier"><br></font></div><div><font class="Apple-style-span" face="Courier"> -p thread_pools=4 \</font></div><div><font class="Apple-style-span" face="Courier"> -p thread_pool_min=100 \</font></div><div><br></div><div>This must match up with VARNISH_MIN_THREADS. We use four pools, (pools * thread_pool_min == VARNISH_MIN_THREADS)</div><div><br></div><div><span class="Apple-tab-span" style="white-space: pre; "><font class="Apple-style-span" face="Courier"> </font></span><font class="Apple-style-span" face="Courier"> -p srcaddr_ttl=0 \</font></div><div><br></div><div><span class="Apple-style-span" style="font-family: arial; font-size: 13px; ">Disable the (effectively unused) per source-IP statistics</span></div><div><br></div><div><span class="Apple-tab-span" style="white-space: pre; "><font class="Apple-style-span" face="Courier"> </font></span><font class="Apple-style-span" face="Courier"> -p esi_syntax=1 </font></div></div><div><br></div><div>Disable ESI syntax verification so we can use it to process JSON requests. </div><div><br></div><div>If you have more than 2.1M objects, you should also add:</div><div><div><font class="Apple-style-span" face="Courier"># -h classic,250007 = recommeded value for 2.1M objects</font></div><div><font class="Apple-style-span" face="Courier"># number should be 1/10 expected working set.</font></div><div><br></div></div><div><br></div><div>In our VCL, we have a few fancy tricks that we use. We label the cache server and cache hit/miss rate in vcl_deliver with this code:</div><div><div><br></div><div><b>Top of VCL:</b></div><div><font class="Apple-style-span" face="Courier">C{</font></div><div><font class="Apple-style-span" face="Courier">#include <stdio.h></font></div><div><font class="Apple-style-span" face="Courier">#include <unistd.h></font></div><div><font class="Apple-style-span" face="Courier"><br></font></div><div><font class="Apple-style-span" face="Courier">char myhostname[255] = "";</font></div><div><font class="Apple-style-span" face="Courier"> </font></div><div><font class="Apple-style-span" face="Courier">}C</font></div><div><br></div><div><span class="Apple-style-span" style="font-weight: bold; ">vcl_deliver:</span></div></div><div><div><font class="Apple-style-span" face="Courier">C{</font></div><div><font class="Apple-style-span" face="Courier"> VRT_SetHdr(sp, HDR_RESP, "\014X-Cache-Svr:", myhostname, vrt_magic_string_end);</font></div><div><font class="Apple-style-span" face="Courier">}C</font></div><div><font class="Apple-style-span" face="Courier"> /* mark hit/miss on the request */</font></div><div><font class="Apple-style-span" face="Courier"> if (obj.hits > 0) {</font></div><div><font class="Apple-style-span" face="Courier"> set resp.http.X-Cache = "HIT";</font></div><div><font class="Apple-style-span" face="Courier"> set resp.http.X-Cache-Hits = obj.hits;</font></div><div><font class="Apple-style-span" face="Courier"> } else {</font></div><div><font class="Apple-style-span" face="Courier"> set resp.http.X-Cache = "MISS";</font></div><div><font class="Apple-style-span" face="Courier"> }</font></div><div><font class="Apple-style-span" face="Courier"><br></font></div><div><font class="Apple-style-span" face="Courier"> </font></div><div><b>vcl_recv:</b></div><div><b><div><span class="Apple-style-span" style="font-weight: normal; "><font class="Apple-style-span" face="Courier">C{</font></span></div><div><span class="Apple-style-span" style="font-weight: normal;"><font class="Apple-style-span" face="Courier"> if (myhostname[0] == '\0') {</font></span></div><div><span class="Apple-style-span" style="font-weight: normal;"><font class="Apple-style-span" face="Courier"> /* only get hostname once - restart required if hostname changes */</font></span></div><div><span class="Apple-style-span" style="font-weight: normal;"><font class="Apple-style-span" face="Courier"> gethostname(myhostname, 255);</font></span></div><div><span class="Apple-style-span" style="font-weight: normal;"><font class="Apple-style-span" face="Courier"> }</font></span></div><div><span class="Apple-style-span" style="font-weight: normal;"><font class="Apple-style-span" face="Courier">}C</font></span></div><div><span class="Apple-style-span" style="font-weight: normal;"> </span></div><div><span class="Apple-style-span" style="font-weight: normal;"><b><br></b></span></div></b></div></div><div>Portions of /etc/sysconfig/varnish follow...</div><div><font class="Apple-style-span" face="Courier"><br></font></div><div><div><font class="Apple-style-span" face="Courier"># The minimum number of worker threads to start</font></div><div><font class="Apple-style-span" face="Courier">VARNISH_MIN_THREADS=400</font></div><div><font class="Apple-style-span" face="Courier"><br></font></div><div><font class="Apple-style-span" face="Courier"># The Maximum number of worker threads to start</font></div><div><font class="Apple-style-span" face="Courier">VARNISH_MAX_THREADS=1000</font></div><div><font class="Apple-style-span" face="Courier"><br></font></div><div><font class="Apple-style-span" face="Courier"># Idle timeout for worker threads</font></div><div><font class="Apple-style-span" face="Courier">VARNISH_THREAD_TIMEOUT=60</font></div><div><font class="Apple-style-span" face="Courier"><br></font></div><div><font class="Apple-style-span" face="Courier"># Cache file location</font></div><div><font class="Apple-style-span" face="Courier">VARNISH_STORAGE_FILE=/var/lib/varnish/varnish_storage.bin</font></div><div><font class="Apple-style-span" face="Courier"><br></font></div><div><font class="Apple-style-span" face="Courier"># Cache file size: in bytes, optionally using k / M / G / T suffix,</font></div><div><font class="Apple-style-span" face="Courier"># or in percentage of available disk space using the % suffix.</font></div><div><font class="Apple-style-span" face="Courier">VARNISH_STORAGE_SIZE="8G"</font></div><div><font class="Apple-style-span" face="Courier">#</font></div><div><font class="Apple-style-span" face="Courier"># Backend storage specification</font></div><div><font class="Apple-style-span" face="Courier">VARNISH_STORAGE="malloc,${VARNISH_STORAGE_SIZE}"</font></div><div><font class="Apple-style-span" face="Courier"><br></font></div><div><font class="Apple-style-span" face="Courier"># Default TTL used when the backend does not specify one</font></div><div><font class="Apple-style-span" face="Courier">VARNISH_TTL=5</font></div><div><font class="Apple-style-span" face="Courier"><br></font></div><div><font class="Apple-style-span" face="Courier"># the working directory</font></div><div><font class="Apple-style-span" face="Courier"><br></font></div></div><div><div><font class="Apple-style-span" face="Courier">DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \</font></div><div><font class="Apple-style-span" face="Courier"> -f ${VARNISH_VCL_CONF} \</font></div><div><font class="Apple-style-span" face="Courier"> -T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \</font></div><div><font class="Apple-style-span" face="Courier"> -t ${VARNISH_TTL} \</font></div><div><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="Courier"> </font></span><font class="Apple-style-span" face="Courier"> -n ${VARNISH_WORKDIR} \</font></div><div><font class="Apple-style-span" face="Courier"> -w ${VARNISH_MIN_THREADS},${VARNISH_MAX_THREADS},${VARNISH_THREAD_TIMEOUT} \</font></div><div><font class="Apple-style-span" face="Courier"> -u varnish -g varnish \</font></div><div><font class="Apple-style-span" face="Courier"> -p obj_workspace=4096 \</font></div><div><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="Courier"> </font></span><font class="Apple-style-span" face="Courier"> -p sess_workspace=262144 \</font></div><div><font class="Apple-style-span" face="Courier"> -p listen_depth=8192 \</font></div><div><font class="Apple-style-span" face="Courier"> -p log_hashstring=off \</font></div><div><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="Courier"> </font></span><font class="Apple-style-span" face="Courier"> -p lru_interval=60 \</font></div><div><font class="Apple-style-span" face="Courier"> -p sess_timeout=10 \</font></div><div><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="Courier"> </font></span><font class="Apple-style-span" face="Courier"> -p shm_workspace=32768 \</font></div><div><font class="Apple-style-span" face="Courier"> -p ping_interval=1 \</font></div><div><font class="Apple-style-span" face="Courier"> -p thread_pools=4 \</font></div><div><font class="Apple-style-span" face="Courier"> -p thread_pool_min=100 \</font></div><div><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="Courier"> </font></span><font class="Apple-style-span" face="Courier"> -p srcaddr_ttl=0 \</font></div><div><span class="Apple-tab-span" style="white-space:pre"><font class="Apple-style-span" face="Courier"> </font></span><font class="Apple-style-span" face="Courier"> -p esi_syntax=1 \</font></div><div><font class="Apple-style-span" face="Courier"> -s ${VARNISH_STORAGE}"</font></div><div><br></div></div><div><div><br></div>---</div><div apple-content-edited="true"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>John Adams</div><div>Twitter Operations</div><div><a href="mailto:jna@twitter.com">jna@twitter.com</a></div><div><a href="http://twitter.com/netik">http://twitter.com/netik</a></div><div><br></div></div><br class="Apple-interchange-newline"></div><br class="Apple-interchange-newline"> </div><br></body></html>