<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif; ">
<div>I am trying to properly size our Varnish cache size after it's been allocated way too much memory. The system is using about 10GB of RAM for Varnish.</div>
<div><br>
</div>
<div>Here is what I am using to calculate the total needed:</div>
<div><br>
</div>
<div>
<div>~ 100MB = Varnish Master Process</div>
<div>~ 80MB = Varnish Shared Memory Log</div>
<div>~ 2KB * n objects = Object storage overhead</div>
<div>~ yKB * n objects = Object storage</div>
</div>
<div><br>
</div>
<div>With this, I am estimating about 4.2GB for Varnish using a 32KB average object size. What I am having trouble with is the grace (set for 1 hour) and how that comes into play with storing objects. Are they stored for 1 hour after expiry in case a backend
goes down and after the 1 hour available to be evicted? When exactly do objects free memory? I've found different information, but it seems the prevailing thought is only when a purge command occurs.</div>
<div><br>
</div>
<div>I am under the impression the expired objects just sit in memory until the cache is full (just before nuking objects) and then LRU eviction occurs. Is this accurate? I mean, even though I need about 5GB for Varnish, since I allocate 16GB, will the entire
16GB be used by expired objects?</div>
<div><br>
</div>
<div>Is there a way to tell exactly how much RAM is being used for VALID objects?</div>
<div><br>
</div>
<div>TIA,</div>
<div>Travis</div>
<div><br>
</div>
<div>
<div>client_conn 7668521 58.59 Client connections accepted</div>
<div>client_drop 0 0.00 Connection dropped, no sess/wrk</div>
<div>client_req 15846658 121.06 Client requests received</div>
<div>cache_hit 10757881 82.19 Cache hits</div>
<div>cache_hitpass 32698 0.25 Cache hits for pass</div>
<div>cache_miss 4011878 30.65 Cache misses</div>
<div>backend_conn 4465314 34.11 Backend conn. success</div>
<div>backend_unhealthy 0 0.00 Backend conn. not attempted</div>
<div>backend_busy 0 0.00 Backend conn. too many</div>
<div>backend_fail 0 0.00 Backend conn. failures</div>
<div>backend_reuse 0 0.00 Backend conn. reuses</div>
<div>backend_toolate 0 0.00 Backend conn. was closed</div>
<div>backend_recycle 0 0.00 Backend conn. recycles</div>
<div>backend_retry 0 0.00 Backend conn. retry</div>
<div>fetch_head 89 0.00 Fetch head</div>
<div>fetch_length 2928665 22.37 Fetch with Length</div>
<div>fetch_chunked 1416614 10.82 Fetch chunked</div>
<div>fetch_eof 0 0.00 Fetch EOF</div>
<div>fetch_bad 0 0.00 Fetch had bad headers</div>
<div>fetch_close 95492 0.73 Fetch wanted close</div>
<div>fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed</div>
<div>fetch_zero 0 0.00 Fetch zero len</div>
<div>fetch_failed 8 0.00 Fetch failed</div>
<div>fetch_1xx 0 0.00 Fetch no body (1xx)</div>
<div>fetch_204 0 0.00 Fetch no body (204)</div>
<div>fetch_304 22537 0.17 Fetch no body (304)</div>
<div>n_sess_mem 682 . N struct sess_mem</div>
<div>n_sess 369 . N struct sess</div>
<div>n_object 119236 . N struct object</div>
<div>n_vampireobject 0 . N unresurrected objects</div>
<div>n_objectcore 119437 . N struct objectcore</div>
<div>n_objecthead 127556 . N struct objecthead</div>
<div>n_waitinglist 6994 . N struct waitinglist</div>
<div>n_vbc 10 . N struct vbc</div>
<div>n_wrk 400 . N worker threads</div>
<div>n_wrk_create 400 0.00 N worker threads created</div>
<div>n_wrk_failed 0 0.00 N worker threads not created</div>
<div>n_wrk_max 0 0.00 N worker threads limited</div>
<div>n_wrk_lqueue 0 0.00 work request queue length</div>
<div>n_wrk_queued 0 0.00 N queued work requests</div>
<div>n_wrk_drop 0 0.00 N dropped work requests</div>
<div>n_backend 11 . N backends</div>
<div>n_expired 3890039 . N expired objects</div>
<div>n_lru_nuked 0 . N LRU nuked objects</div>
<div>n_lru_moved 8128646 . N LRU moved objects</div>
<div>losthdr 91 0.00 HTTP header overflows</div>
<div>n_objsendfile 0 0.00 Objects sent with sendfile</div>
<div>n_objwrite 13494880 103.10 Objects sent with write</div>
<div>n_objoverflow 0 0.00 Objects overflowing workspace</div>
<div>s_sess 7668501 58.59 Total Sessions</div>
<div>s_req 15846658 121.06 Total Requests</div>
<div>s_pipe 1251 0.01 Total pipe</div>
<div>s_pass 453681 3.47 Total pass</div>
<div>s_fetch 4462324 34.09 Total fetch</div>
<div>s_hdrbytes 7307416930 55826.98 Total header bytes</div>
<div>s_bodybytes 435088023618 3323972.25 Total body bytes</div>
<div>sess_closed 2239685 17.11 Session Closed</div>
<div>sess_pipeline 0 0.00 Session Pipeline</div>
<div>sess_readahead 0 0.00 Session Read Ahead</div>
<div>sess_linger 13878128 106.03 Session Linger</div>
<div>sess_herd 12875482 98.37 Session herd</div>
<div>shm_records 916560440 7002.31 SHM records</div>
<div>shm_writes 61086120 466.68 SHM writes</div>
<div>shm_flushes 14 0.00 SHM flushes due to overflow</div>
<div>shm_cont 10334 0.08 SHM MTX contention</div>
<div>shm_cycles 405 0.00 SHM cycles through buffer</div>
<div>sms_nreq 313389 2.39 SMS allocator requests</div>
<div>sms_nobj 0 . SMS outstanding allocations</div>
<div>sms_nbytes 0 . SMS outstanding bytes</div>
<div>sms_balloc 544043304 . SMS bytes allocated</div>
<div>sms_bfree 544043304 . SMS bytes freed</div>
<div>backend_req 4463489 34.10 Backend requests made</div>
<div>n_vcl 3 0.00 N vcl total</div>
<div>n_vcl_avail 3 0.00 N vcl available</div>
<div>n_vcl_discard 0 0.00 N vcl discarded</div>
<div>n_ban 29993 . N total active bans</div>
<div>n_ban_gone 29127 . N total gone bans</div>
<div>n_ban_add 61384 0.47 N new bans added</div>
<div>n_ban_retire 31391 0.24 N old bans deleted</div>
<div>n_ban_obj_test 16197865 123.75 N objects tested</div>
<div>n_ban_re_test 3174198250 24250.14 N regexps tested against</div>
<div>n_ban_dups 33565 0.26 N duplicate bans removed</div>
<div>hcb_nolock 14802058 113.08 HCB Lookups without lock</div>
<div>hcb_lock 3831356 29.27 HCB Lookups with lock</div>
<div>hcb_insert 3831344 29.27 HCB Inserts</div>
<div>esi_errors 0 0.00 ESI parse errors (unlock)</div>
<div>esi_warnings 0 0.00 ESI parse warnings (unlock)</div>
<div>accept_fail 0 0.00 Accept failures</div>
<div>client_drop_late 0 0.00 Connection dropped late</div>
<div>uptime 130894 1.00 Client uptime</div>
<div>dir_dns_lookups 0 0.00 DNS director lookups</div>
<div>dir_dns_failed 0 0.00 DNS director failed lookups</div>
<div>dir_dns_hit 0 0.00 DNS director cached lookups hit</div>
<div>dir_dns_cache_full 0 0.00 DNS director full dnscache</div>
<div>vmods 0 . Loaded VMODs</div>
<div>n_gzip 0 0.00 Gzip operations</div>
<div>n_gunzip 3226586 24.65 Gunzip operations</div>
<div>LCK.sms.creat 1 0.00 Created locks</div>
<div>LCK.sms.destroy 0 0.00 Destroyed locks</div>
<div>LCK.sms.locks 940167 7.18 Lock Operations</div>
<div>LCK.sms.colls 0 0.00 Collisions</div>
<div>LCK.smp.creat 0 0.00 Created locks</div>
<div>LCK.smp.destroy 0 0.00 Destroyed locks</div>
<div>LCK.smp.locks 0 0.00 Lock Operations</div>
<div>LCK.smp.colls 0 0.00 Collisions</div>
<div>LCK.sma.creat 2 0.00 Created locks</div>
<div>LCK.sma.destroy 0 0.00 Destroyed locks</div>
<div>LCK.sma.locks 21106576 161.25 Lock Operations</div>
<div>LCK.sma.colls 0 0.00 Collisions</div>
<div>LCK.smf.creat 0 0.00 Created locks</div>
<div>LCK.smf.destroy 0 0.00 Destroyed locks</div>
<div>LCK.smf.locks 0 0.00 Lock Operations</div>
<div>LCK.smf.colls 0 0.00 Collisions</div>
<div>LCK.hsl.creat 0 0.00 Created locks</div>
<div>LCK.hsl.destroy 0 0.00 Destroyed locks</div>
<div>LCK.hsl.locks 0 0.00 Lock Operations</div>
<div>LCK.hsl.colls 0 0.00 Collisions</div>
<div>LCK.hcb.creat 1 0.00 Created locks</div>
<div>LCK.hcb.destroy 0 0.00 Destroyed locks</div>
<div>LCK.hcb.locks 7544189 57.64 Lock Operations</div>
<div>LCK.hcb.colls 0 0.00 Collisions</div>
<div>LCK.hcl.creat 0 0.00 Created locks</div>
<div>LCK.hcl.destroy 0 0.00 Destroyed locks</div>
<div>LCK.hcl.locks 0 0.00 Lock Operations</div>
<div>LCK.hcl.colls 0 0.00 Collisions</div>
<div>LCK.vcl.creat 1 0.00 Created locks</div>
<div>LCK.vcl.destroy 0 0.00 Destroyed locks</div>
<div>LCK.vcl.locks 17100 0.13 Lock Operations</div>
<div>LCK.vcl.colls 0 0.00 Collisions</div>
<div>LCK.stat.creat 1 0.00 Created locks</div>
<div>LCK.stat.destroy 0 0.00 Destroyed locks</div>
<div>LCK.stat.locks 7668837 58.59 Lock Operations</div>
<div>LCK.stat.colls 0 0.00 Collisions</div>
<div>LCK.sessmem.creat 1 0.00 Created locks</div>
<div>LCK.sessmem.destroy 0 0.00 Destroyed locks</div>
<div>LCK.sessmem.locks 7703601 58.85 Lock Operations</div>
<div>LCK.sessmem.colls 0 0.00 Collisions</div>
<div>LCK.wstat.creat 1 0.00 Created locks</div>
<div>LCK.wstat.destroy 0 0.00 Destroyed locks</div>
<div>LCK.wstat.locks 146130 1.12 Lock Operations</div>
<div>LCK.wstat.colls 0 0.00 Collisions</div>
<div>LCK.herder.creat 1 0.00 Created locks</div>
<div>LCK.herder.destroy 0 0.00 Destroyed locks</div>
<div>LCK.herder.locks 1 0.00 Lock Operations</div>
<div>LCK.herder.colls 0 0.00 Collisions</div>
<div>LCK.wq.creat 2 0.00 Created locks</div>
<div>LCK.wq.destroy 0 0.00 Destroyed locks</div>
<div>LCK.wq.locks 30510977 233.10 Lock Operations</div>
<div>LCK.wq.colls 0 0.00 Collisions</div>
<div>LCK.objhdr.creat 3831455 29.27 Created locks</div>
<div>LCK.objhdr.destroy 3703972 28.30 Destroyed locks</div>
<div>LCK.objhdr.locks 79026186 603.74 Lock Operations</div>
<div>LCK.objhdr.colls 0 0.00 Collisions</div>
<div>LCK.exp.creat 1 0.00 Created locks</div>
<div>LCK.exp.destroy 0 0.00 Destroyed locks</div>
<div>LCK.exp.locks 8038327 61.41 Lock Operations</div>
<div>LCK.exp.colls 0 0.00 Collisions</div>
<div>LCK.lru.creat 2 0.00 Created locks</div>
<div>LCK.lru.destroy 0 0.00 Destroyed locks</div>
<div>LCK.lru.locks 4017535 30.69 Lock Operations</div>
<div>LCK.lru.colls 0 0.00 Collisions</div>
<div>LCK.cli.creat 1 0.00 Created locks</div>
<div>LCK.cli.destroy 0 0.00 Destroyed locks</div>
<div>LCK.cli.locks 1802 0.01 Lock Operations</div>
<div>LCK.cli.colls 0 0.00 Collisions</div>
<div>LCK.ban.creat 1 0.00 Created locks</div>
<div>LCK.ban.destroy 0 0.00 Destroyed locks</div>
<div>LCK.ban.locks 37966814 290.06 Lock Operations</div>
<div>LCK.ban.colls 0 0.00 Collisions</div>
<div>LCK.vbp.creat 1 0.00 Created locks</div>
<div>LCK.vbp.destroy 0 0.00 Destroyed locks</div>
<div>LCK.vbp.locks 286759 2.19 Lock Operations</div>
<div>LCK.vbp.colls 0 0.00 Collisions</div>
<div>LCK.vbe.creat 1 0.00 Created locks</div>
<div>LCK.vbe.destroy 0 0.00 Destroyed locks</div>
<div>LCK.vbe.locks 8930660 68.23 Lock Operations</div>
<div>LCK.vbe.colls 0 0.00 Collisions</div>
<div>LCK.backend.creat 11 0.00 Created locks</div>
<div>LCK.backend.destroy 0 0.00 Destroyed locks</div>
<div>LCK.backend.locks 13425358 102.57 Lock Operations</div>
<div>LCK.backend.colls 0 0.00 Collisions</div>
<div>SMA.s0.c_req 8642204 66.02 Allocator requests</div>
<div>SMA.s0.c_fail 0 0.00 Allocator failures</div>
<div>SMA.s0.c_bytes 413441801561 3158600.10 Bytes allocated</div>
<div>SMA.s0.c_freed 405211048073 3095719.04 Bytes freed</div>
<div>SMA.s0.g_alloc 238498 . Allocations outstanding</div>
<div>SMA.s0.g_bytes 8230753488 . Bytes outstanding</div>
<div>SMA.s0.g_space 8949115696 . Bytes available</div>
<div>SMA.Transient.c_req 884306 6.76 Allocator requests</div>
<div>SMA.Transient.c_fail 0 0.00 Allocator failures</div>
<div>SMA.Transient.c_bytes 31276155463 238942.62 Bytes allocated</div>
<div>SMA.Transient.c_freed 31276023159 238941.61 Bytes freed</div>
<div>SMA.Transient.g_alloc 2 . Allocations outstanding</div>
<div>SMA.Transient.g_bytes 132304 . Bytes outstanding</div>
<div>SMA.Transient.g_space 0 . Bytes available</div>
<div>VBE.web10(10.2.180.74,,80).vcls 2 . VCL references</div>
<div>VBE.web10(10.2.180.74,,80).happy18446744073709551615 . Happy health probes</div>
<div>VBE.web11(10.2.180.75,,80).vcls 3 . VCL references</div>
<div>VBE.web11(10.2.180.75,,80).happy18446744073709551615 . Happy health probes</div>
<div>VBE.web12(10.2.180.185,,80).vcls 3 . VCL references</div>
<div>VBE.web12(10.2.180.185,,80).happy 18446744073709551615 . Happy health probes</div>
<div>VBE.web13(10.2.180.186,,80).vcls 3 . VCL references</div>
<div>VBE.web13(10.2.180.186,,80).happy 18446744073709551615 . Happy health probes</div>
<div>VBE.web14(10.2.180.198,,80).vcls 3 . VCL references</div>
<div>VBE.web14(10.2.180.198,,80).happy 18446744073709551615 . Happy health probes</div>
<div>VBE.web15(10.2.180.199,,80).vcls 3 . VCL references</div>
<div>VBE.web15(10.2.180.199,,80).happy 18446744073709551615 . Happy health probes</div>
<div>VBE.web16(10.2.180.47,,80).vcls 3 . VCL references</div>
<div>VBE.web16(10.2.180.47,,80).happy 18446744073709551615 . Happy health probes</div>
<div>VBE.web17(10.2.180.48,,80).vcls 3 . VCL references</div>
<div>VBE.web17(10.2.180.48,,80).happy 18446744073709551615 . Happy health probes</div>
<div>VBE.web18(10.2.180.49,,80).vcls 3 . VCL references</div>
<div>VBE.web18(10.2.180.49,,80).happy 18446744073709551615 . Happy health probes</div>
<div>VBE.web19(10.2.180.50,,80).vcls 3 . VCL references</div>
<div>VBE.web19(10.2.180.50,,80).happy 18446744073709551615 . Happy health probes</div>
<div>VBE.web20(10.2.180.44,,80).vcls 2 . VCL references</div>
<div>VBE.web20(10.2.180.44,,80).happy 18446744073709551615 . Happy health probes</div>
</div>
</body>
</html>