I have a bunch of Varnish servers (Exactly same configurations, caching different keys on Intel 2650 series Computer + 16GB RAM + 32GB SSD each). I am enclosing my configurations and settings along with my current varnishstats.<div>
<br></div><div>I am looking for more debug information and how to find the route cause of my varnish getting restarting every 6-8 hours (all varnish servers)</div><div><br></div><div>Please do let me know in case you need any more information. I am loosing all my cache (I tried 20G rather than 32G but still it restarts flushing all the cache and becoming unavailable for split seconds). I am using them behind nginx load balancers (which also distribute traffic among different varnish servers based on some keys). The varnish are fueled by 4 powerful LAMP servers.</div>
<div><br></div><div>I have high miss rate because I keep loosing all my cache</div><div><br></div><div>Daemon Options:</div><div><div>START=yes</div><div>NFILES=131072</div><div>MEMLOCK=82000</div><div>INSTANCE=$(uname -n)</div>
<div>VARNISH_VCL_CONF=/etc/varnish/default.vcl</div><div>VARNISH_LISTEN_ADDRESS=10.56.140.12</div><div>VARNISH_LISTEN_PORT=80</div><div>VARNISH_ADMIN_LISTEN_ADDRESS=10.56.140.12</div><div>VARNISH_ADMIN_LISTEN_PORT=6082</div>
<div>VARNISH_MIN_THREADS=200</div><div>VARNISH_MAX_THREADS=4000</div><div>VARNISH_THREAD_TIMEOUT=60</div><div>VARNISH_STORAGE_SIZE=32G</div><div>VARNISH_STORAGE_MALLOC="malloc"</div><div>VARNISH_USER="varnish"</div>
<div>VARNISH_GROUP="varnish"</div><div><br></div><div>DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \</div><div> -f ${VARNISH_VCL_CONF} \</div><div> -T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \</div>
<div> -w ${VARNISH_MIN_THREADS},${VARNISH_MAX_THREADS},${VARNISH_THREAD_TIMEOUT} \</div><div> -s ${VARNISH_STORAGE_MALLOC},${VARNISH_STORAGE_SIZE} \</div><div> -u ${VARNISH_USER} -g ${VARNISH_GROUP} \</div>
<div> -i ${INSTANCE} \</div><div> -p thread_pool_add_delay=2 \</div><div> -p thread_pools=32 \</div><div> -p thread_pool_min=25 \</div><div> -p thread_pool_max=4000 \</div>
<div> -p sess_timeout=3 \</div><div> -p session_max=1000000 \</div><div> -p session_linger=2000"</div></div><div><br></div><div><br></div><div>Varnishstat -1:</div><div><div>client_conn 49815 26.78 Client connections accepted</div>
<div>client_drop 0 0.00 Connection dropped, no sess/wrk</div><div>client_req 1822206 979.68 Client requests received</div><div>cache_hit 253196 136.13 Cache hits</div>
<div>cache_hitpass 1379 0.74 Cache hits for pass</div><div>cache_miss 1567631 842.81 Cache misses</div><div>backend_conn 16163 8.69 Backend conn. success</div><div>
backend_unhealthy 0 0.00 Backend conn. not attempted</div><div>backend_busy 0 0.00 Backend conn. too many</div><div>backend_fail 1264 0.68 Backend conn. failures</div>
<div>backend_reuse 1552269 834.55 Backend conn. reuses</div><div>backend_toolate 3 0.00 Backend conn. was closed</div><div>backend_recycle 1553574 835.25 Backend conn. recycles</div>
<div>backend_retry 0 0.00 Backend conn. retry</div><div>fetch_head 0 0.00 Fetch head</div><div>fetch_length 1567981 843.00 Fetch with Length</div><div>fetch_chunked 394 0.21 Fetch chunked</div>
<div>fetch_eof 0 0.00 Fetch EOF</div><div>fetch_bad 0 0.00 Fetch had bad headers</div><div>fetch_close 0 0.00 Fetch wanted close</div><div>fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed</div>
<div>fetch_zero 0 0.00 Fetch zero len</div><div>fetch_failed 0 0.00 Fetch failed</div><div>fetch_1xx 0 0.00 Fetch no body (1xx)</div><div>fetch_204 0 0.00 Fetch no body (204)</div>
<div>fetch_304 0 0.00 Fetch no body (304)</div><div>n_sess_mem 968 . N struct sess_mem</div><div>n_sess 209 . N struct sess</div><div>n_object 1565359 . N struct object</div>
<div>n_vampireobject 0 . N unresurrected objects</div><div>n_objectcore 1565551 . N struct objectcore</div><div>n_objecthead 1565798 . N struct objecthead</div>
<div>n_waitinglist 965 . N struct waitinglist</div><div>n_vbc 19 . N struct vbc</div><div>n_wrk 800 . N worker threads</div><div>n_wrk_create 836 0.45 N worker threads created</div>
<div>n_wrk_failed 0 0.00 N worker threads not created</div><div>n_wrk_max 0 0.00 N worker threads limited</div><div>n_wrk_lqueue 0 0.00 work request queue length</div>
<div>n_wrk_queued 210 0.11 N queued work requests</div><div>n_wrk_drop 0 0.00 N dropped work requests</div><div>n_backend 4 . N backends</div>
<div>
n_expired 1637 . N expired objects</div><div>n_lru_nuked 0 . N LRU nuked objects</div><div>n_lru_moved 177419 . N LRU moved objects</div><div>
losthdr 0 0.00 HTTP header overflows</div>
<div>n_objsendfile 0 0.00 Objects sent with sendfile</div><div>n_objwrite 1817561 977.18 Objects sent with write</div><div>n_objoverflow 0 0.00 Objects overflowing workspace</div>
<div>s_sess 49763 26.75 Total Sessions</div><div>s_req 1822206 979.68 Total Requests</div><div>s_pipe 0 0.00 Total pipe</div><div>s_pass 1379 0.74 Total pass</div>
<div>s_fetch 1568375 843.21 Total fetch</div><div>s_hdrbytes 543007555 291939.55 Total header bytes</div><div>s_bodybytes 366746858 197175.73 Total body bytes</div><div>sess_closed 1973 1.06 Session Closed</div>
<div>sess_pipeline 0 0.00 Session Pipeline</div><div>sess_readahead 0 0.00 Session Read Ahead</div><div>sess_linger 1821411 979.25 Session Linger</div><div>sess_herd 50288 27.04 Session herd</div>
<div>shm_records 132731919 71361.25 SHM records</div><div>shm_writes 5168655 2778.85 SHM writes</div><div>shm_flushes 0 0.00 SHM flushes due to overflow</div><div>shm_cont 31135 16.74 SHM MTX contention</div>
<div>shm_cycles 63 0.03 SHM cycles through buffer</div><div>sms_nreq 635 0.34 SMS allocator requests</div><div>sms_nobj 0 . SMS outstanding allocations</div>
<div>sms_nbytes 0 . SMS outstanding bytes</div><div>sms_balloc 265430 . SMS bytes allocated</div><div>sms_bfree 265430 . SMS bytes freed</div>
<div>
backend_req 1569626 843.88 Backend requests made</div><div>n_vcl 2 0.00 N vcl total</div><div>n_vcl_avail 2 0.00 N vcl available</div><div>n_vcl_discard 0 0.00 N vcl discarded</div>
<div>n_ban 48 . N total active bans</div><div>n_ban_add 49 0.03 N new bans added</div><div>n_ban_retire 1 0.00 N old bans deleted</div><div>
n_ban_obj_test 82431 44.32 N objects tested</div><div>n_ban_re_test 211534 113.73 N regexps tested against</div><div>n_ban_dups 35 0.02 N duplicate bans removed</div>
<div>hcb_nolock 1822862 980.03 HCB Lookups without lock</div><div>hcb_lock 1568249 843.14 HCB Lookups with lock</div><div>hcb_insert 1568249 843.14 HCB Inserts</div>
<div>esi_errors 0 0.00 ESI parse errors (unlock)</div><div>esi_warnings 0 0.00 ESI parse warnings (unlock)</div><div>accept_fail 0 0.00 Accept failures</div>
<div>client_drop_late 0 0.00 Connection dropped late</div><div>uptime 1860 1.00 Client uptime</div><div>dir_dns_lookups 0 0.00 DNS director lookups</div>
<div>dir_dns_failed 0 0.00 DNS director failed lookups</div><div>dir_dns_hit 0 0.00 DNS director cached lookups hit</div><div>dir_dns_cache_full 0 0.00 DNS director full dnscache</div>
<div>vmods 0 . Loaded VMODs</div><div>n_gzip 0 0.00 Gzip operations</div><div>n_gunzip 3377737 1815.99 Gunzip operations</div><div>LCK.sms.creat 4 0.00 Created locks</div>
<div>LCK.sms.destroy 0 0.00 Destroyed locks</div><div>LCK.sms.locks 360546 193.84 Lock Operations</div><div>LCK.sms.colls 0 0.00 Collisions</div><div>LCK.smp.creat 0 0.00 Created locks</div>
<div>LCK.smp.destroy 0 0.00 Destroyed locks</div><div>LCK.smp.locks 0 0.00 Lock Operations</div><div>LCK.smp.colls 0 0.00 Collisions</div><div>LCK.sma.creat 8 0.00 Created locks</div>
<div>LCK.sma.destroy 0 0.00 Destroyed locks</div><div>LCK.sma.locks 223989728 120424.58 Lock Operations</div><div>LCK.sma.colls 0 0.00 Collisions</div><div>LCK.smf.creat 0 0.00 Created locks</div>
<div>LCK.smf.destroy 0 0.00 Destroyed locks</div><div>LCK.smf.locks 0 0.00 Lock Operations</div><div>LCK.smf.colls 0 0.00 Collisions</div><div>LCK.hsl.creat 0 0.00 Created locks</div>
<div>LCK.hsl.destroy 0 0.00 Destroyed locks</div><div>LCK.hsl.locks 0 0.00 Lock Operations</div><div>LCK.hsl.colls 0 0.00 Collisions</div><div>LCK.hcb.creat 4 0.00 Created locks</div>
<div>LCK.hcb.destroy 0 0.00 Destroyed locks</div><div>LCK.hcb.locks 74720192 40172.15 Lock Operations</div><div>LCK.hcb.colls 0 0.00 Collisions</div><div>LCK.hcl.creat 0 0.00 Created locks</div>
<div>LCK.hcl.destroy 0 0.00 Destroyed locks</div><div>LCK.hcl.locks 0 0.00 Lock Operations</div><div>LCK.hcl.colls 0 0.00 Collisions</div><div>LCK.vcl.creat 4 0.00 Created locks</div>
<div>LCK.vcl.destroy 0 0.00 Destroyed locks</div><div>LCK.vcl.locks 19211 10.33 Lock Operations</div><div>LCK.vcl.colls 0 0.00 Collisions</div><div>LCK.stat.creat 4 0.00 Created locks</div>
<div>LCK.stat.destroy 0 0.00 Destroyed locks</div><div>LCK.stat.locks 6926 3.72 Lock Operations</div><div>LCK.stat.colls 0 0.00 Collisions</div><div>LCK.sessmem.creat 4 0.00 Created locks</div>
<div>LCK.sessmem.destroy 0 0.00 Destroyed locks</div><div>LCK.sessmem.locks 2970656 1597.13 Lock Operations</div><div>LCK.sessmem.colls 0 0.00 Collisions</div><div>LCK.wstat.creat 4 0.00 Created locks</div>
<div>LCK.wstat.destroy 0 0.00 Destroyed locks</div><div>LCK.wstat.locks 7972177 4286.12 Lock Operations</div><div>LCK.wstat.colls 0 0.00 Collisions</div><div>LCK.herder.creat 4 0.00 Created locks</div>
<div>LCK.herder.destroy 0 0.00 Destroyed locks</div><div>LCK.herder.locks 6214 3.34 Lock Operations</div><div>LCK.herder.colls 0 0.00 Collisions</div><div>LCK.wq.creat 128 0.07 Created locks</div>
<div>LCK.wq.destroy 0 0.00 Destroyed locks</div><div>LCK.wq.locks 9782145 5259.22 Lock Operations</div><div>LCK.wq.colls 0 0.00 Collisions</div><div>LCK.objhdr.creat 74421535 40011.58 Created locks</div>
<div>LCK.objhdr.destroy 271865 146.16 Destroyed locks</div><div>LCK.objhdr.locks 349625261 187970.57 Lock Operations</div><div>LCK.objhdr.colls 0 0.00 Collisions</div><div>LCK.exp.creat 4 0.00 Created locks</div>
<div>LCK.exp.destroy 0 0.00 Destroyed locks</div><div>LCK.exp.locks 74639057 40128.53 Lock Operations</div><div>LCK.exp.colls 0 0.00 Collisions</div><div>LCK.lru.creat 8 0.00 Created locks</div>
<div>LCK.lru.destroy 0 0.00 Destroyed locks</div><div>LCK.lru.locks 74353412 39974.95 Lock Operations</div><div>LCK.lru.colls 0 0.00 Collisions</div><div>LCK.cli.creat 4 0.00 Created locks</div>
<div>LCK.cli.destroy 0 0.00 Destroyed locks</div><div>LCK.cli.locks 32390 17.41 Lock Operations</div><div>LCK.cli.colls 0 0.00 Collisions</div><div>LCK.ban.creat 4 0.00 Created locks</div>
<div>LCK.ban.destroy 0 0.00 Destroyed locks</div><div>LCK.ban.locks 78392714 42146.62 Lock Operations</div><div>LCK.ban.colls 0 0.00 Collisions</div><div>LCK.vbp.creat 4 0.00 Created locks</div>
<div>LCK.vbp.destroy 0 0.00 Destroyed locks</div><div>LCK.vbp.locks 92827 49.91 Lock Operations</div><div>LCK.vbp.colls 0 0.00 Collisions</div><div>LCK.vbe.creat 4 0.00 Created locks</div>
<div>LCK.vbe.destroy 0 0.00 Destroyed locks</div><div>LCK.vbe.locks 4095777 2202.03 Lock Operations</div><div>LCK.vbe.colls 0 0.00 Collisions</div><div>LCK.backend.creat 16 0.01 Created locks</div>
<div>LCK.backend.destroy 0 0.00 Destroyed locks</div><div>LCK.backend.locks 227718873 122429.50 Lock Operations</div><div>LCK.backend.colls 0 0.00 Collisions</div><div>SMA.s0.c_req 3136117 1686.08 Allocator requests</div>
<div>SMA.s0.c_fail 0 0.00 Allocator failures</div><div>SMA.s0.c_bytes 206790032960 111177437.08 Bytes allocated</div><div>SMA.s0.c_freed 205374002652 110416130.46 Bytes freed</div><div>
SMA.s0.g_alloc 3132208 . Allocations outstanding</div>
<div>SMA.s0.g_bytes 1416030308 . Bytes outstanding</div><div>SMA.s0.g_space 32943708060 . Bytes available</div><div>SMA.Transient.c_req 2758 1.48 Allocator requests</div>
<div>SMA.Transient.c_fail 0 0.00 Allocator failures</div><div>SMA.Transient.c_bytes 56648986 30456.44 Bytes allocated</div><div>SMA.Transient.c_freed 56648986 30456.44 Bytes freed</div><div>
SMA.Transient.g_alloc 0 . Allocations outstanding</div><div>SMA.Transient.g_bytes 0 . Bytes outstanding</div><div>SMA.Transient.g_space 0 . Bytes available</div>
<div>VBE.dev1(10.56.140.8,,80).vcls 8 . VCL references</div><div>VBE.dev1(10.56.140.8,,80).happy18446744073709551615 . Happy health probes</div><div>VBE.dev2(10.56.140.2,,80).vcls 8 . VCL references</div>
<div>VBE.dev2(10.56.140.2,,80).happy18446744073709551615 . Happy health probes</div><div>VBE.dev3(10.56.140.4,,80).vcls 8 . VCL references</div><div>VBE.dev3(10.56.140.4,,80).happy18446744073709551615 . Happy health probes</div>
<div>VBE.dev4(10.56.140.6,,80).vcls 8 . VCL references</div><div>VBE.dev4(10.56.140.6,,80).happy18446744073709551615 . Happy health probes</div></div><div><br></div><div>Thanks</div><div>
Sparsh Gupta<br>
</div>