Table of Contents

  1. Running with many objects

Varnish settings

To see further description of these settings, also check param.show -l in the Varnish management interface.

  • -p thread_pool_min=200 (default: 5)

Idle threads are harmless. This number is multipled by the number of thread pools you have available, and the total should be roughly what you need to run on a normal day.

  • -p thread_pool_max=4000 (default 1000)

The maximum number of threads is in essence limited by available file descriptors, however, setting it too high does not increase performance. Having a number of idle threads is reasonably harmless, but do not increase this number above roughly 5000 or you risk running into file-descriptor related issues, among other things.

  • -p thread_pools=4 (default 2)

To reduce lock contention, you might want to increase this number a little. But just a little. The rule of thumb is to set it to the number of CPU cores you have available.

  • -p thread_pool_add_delay=2 (default: 20ms)

Reducing the add_delay lets you create threads faster which is essential - specially at startup - to avoid filling up the queue and dropping requests.

  • -p listen_depth=4096 (default 1024)

You may want to increase this, but there is little advantage to increasing it too high. Set it to your peak connection/second rate, so that you get a buffer of a full second if the acceptor gets busy. More than that is not going to do anything good.

  • -p session_linger=50/100/150 (default: 0ms in <= 2.0.4 and 50ms in > 2.0.4)

To avoid context switching hell when you starve your CPU (and in general), letting each thread wait for new requests is essential. The value depends on how long it takes you to deliver the typical object. This will also reduce the amount of threads piling up (which is somewhat counter intuitive).

Running with many objects

If you have many objects (more than 100000), you may need to set the following command line options:

  • -p lru_interval=20 (default: 2 seconds)

If your cache servers cache most/all objects for a longer time, it makes sense to increase the period before an object is moved to the LRU list. This reduces the amount of lock operations necessary for LRU list access.

WARNING: Do _not_ set this to, say, 3600 (the old suggestion listed here) if there is ANY chance that you will run out of space, as the results can be catastrophic.

  • -h classic,500009 (default: 16383)

To keep hash lookups fast, you should not have more than 10 objects per hash bucket. If you have 3 million objects, number of objects should be at least 300000. The number should be a prime number. You can generate one on  http://www.prime-numbers.org/.

  • -p obj_workspace=4096 (default: 8192)

For every object, this amount of memory is allocated for HTTP protocol header information. Try to decrease this setting, it will decrease the need for VM space to fit all your objects. Be aware that Varnish currently crashes if there is an object is too big for this limit (see #214)

  • -s malloc,50G

Try running with malloc storage if you experience VM hangs. You do this instead of setting up data files, and might have to increase the amount of swap space needed. You can set a limit for how much to allocate, which should be smaller than available swap space on the machine. Possible benefit of not having any swap space on the OS/system disk.

VCL Setting

Enable grace period (varnish serves stale (but cacheable) objects while retriving object from backend)

in vcl_recv: 

set req.grace = 30s;

in vcl_fetch: 

set obj.grace = 30s;

FreeBSD

  • If using FreeBSD 7.0 or newer, try using SCHED_ULE instead of SCHED_4BSD in your kernel config.
  • Turn off soft-updates on the filesystems where you keep your Varnish data files. It will not help Varnish.

kern.ipc.nmbclusters=65536 kern.ipc.somaxconn=16384 kern.maxfiles=131072 kern.maxfilesperproc=104856 kern.threads.max_threads_per_proc=4096

  • loader.conf settings:

kern.ipc.maxsockets="131072" kern.ipc.maxpipekva="104857600" (only if you get the "kern.ipc.maxpipekva exceeded" messages in your logs, varnish does not use pipes for worker pool synchronization any more)

  • If you run 32-bit FreeBSD, you will need to change set kern.maxdsiz (maximum data size per process in number of bytes) in loader.conf to a larger number if you want to cache more than 512 MB (the default setting) of objects.
  • If you use the malloc storage type, and your system hangs with "swap zone exhausted, increase kern.maxswzone" on the console, try increasing kern.maxswzone (default is 32 MB in FreeBSD 7.0) in loader.conf.

Linux

Edit /etc/sysctl.conf

These are numbers from a highly loaded varnishe serving about 4000-8000 req/s

(details:  http://lists.varnish-cache.org/pipermail/varnish-misc/2008-April/001763.html) 

net.ipv4.ip_local_port_range = 1024 65536
net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.ipv4.tcp_rmem=4096 87380 16777216
net.ipv4.tcp_wmem=4096 65536 16777216
net.ipv4.tcp_fin_timeout = 3
net.core.netdev_max_backlog = 30000
net.ipv4.tcp_no_metrics_save=1
net.core.somaxconn = 262144
net.ipv4.tcp_syncookies = 0
net.ipv4.tcp_max_orphans = 262144
net.ipv4.tcp_max_syn_backlog = 262144
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 2

All UNIX platforms

  • Set the mount option noatime and nodiratime on the filesystems where you keep your Varnish data files. There is no point in keeping track of how often they are accessed, it will waste cycles/give unneccessary disk activity.

Monitoring

  • Monitor the number of Varnish threads. It should never be as high as the Varnish thread_pool_max setting.