<div dir="ltr"><div class="gmail_quote"><div dir="ltr"><div class="gmail_quote"><div dir="ltr"><div style="font-family:arial,sans-serif;font-size:13px">Hi, we've been experimenting with varnish 3.0.5 and we seem to be experiencing a memory leak.</div>
<div style="font-family:arial,sans-serif;font-size:13px">
<br></div><div style="font-family:arial,sans-serif;font-size:13px">We're running varnish-3.0.5-r4 (from portage):</div><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">
<a href="http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/www-servers/varnish/varnish-3.0.5-r4.ebuild" target="_blank">http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/www-servers/varnish/varnish-3.0.5-r4.ebuild</a><br>
</div><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">In the graph bellow you can see that the memory kept growing even after the malloc memory (12G) was full. I understand that there's a 1K overhead per object and that Varnish will use more then 12G, but it keeps growing even after the object space is full (now at 43.4G).</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px"><img src="cid:ii_1464d596054a7e8c" alt="Inline image 2" width="544" height="183.15577889447235"><br>
</div><div style="font-family:arial,sans-serif;font-size:13px"></div><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px"><div>box:</div><div><br></div>
<div><div>$ uname -a</div><div>Linux 3.8.13-gentoo #3 SMP Wed Jun 26 11:29:09 EDT 2013 x86_64 Intel(R) Xeon(R) CPU E5-2450 0 @ 2.10GHz GenuineIntel GNU/Linux</div></div><div><br></div></div><div style="font-family:arial,sans-serif;font-size:13px">
configs:</div><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">$ cat /etc/conf.d/varnishd</div><div style="font-family:arial,sans-serif;font-size:13px">
# /etc/conf.d/varnishd</div><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">VARNISHD="/usr/sbin/varnishd"</div><div style="font-family:arial,sans-serif;font-size:13px">
VARNISHADM="/usr/bin/varnishadm"</div><div style="font-family:arial,sans-serif;font-size:13px">CONFIGFILE="/etc/varnish/default.vcl"</div><div style="font-family:arial,sans-serif;font-size:13px"><br></div>
<div style="font-family:arial,sans-serif;font-size:13px">VARNISHD_OPTS="-a :80 -p default_grace=600 -s malloc,12G -p thread_pool_max=2000 -p sess_timeout=120 -p thread_pool_min=100 -f $CONFIGFILE"</div><div style="font-family:arial,sans-serif;font-size:13px">
<br></div><div style="font-family:arial,sans-serif;font-size:13px">rc_ulimit="-n 32786 -l 82000"</div><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">
$ cat /etc/varnish/default.vcl</div><div style="font-family:arial,sans-serif;font-size:13px">sub vcl_fetch {</div><div style="font-family:arial,sans-serif;font-size:13px"> unset beresp.http.access-control-allow-origin;</div>
<div style="font-family:arial,sans-serif;font-size:13px"> unset beresp.http.cache-control;</div><div style="font-family:arial,sans-serif;font-size:13px"> unset beresp.http.content-type;</div><div style="font-family:arial,sans-serif;font-size:13px">
unset beresp.http.p3p;</div><div style="font-family:arial,sans-serif;font-size:13px"> unset beresp.http.pragma;</div><div style="font-family:arial,sans-serif;font-size:13px"> unset beresp.http.server;</div><div style="font-family:arial,sans-serif;font-size:13px">
<br></div><div style="font-family:arial,sans-serif;font-size:13px"> set beresp.do_stream = false;</div><div style="font-family:arial,sans-serif;font-size:13px"> set beresp.http.Content-Type = "application/json";</div>
<div style="font-family:arial,sans-serif;font-size:13px"> set beresp.ttl = 12h;</div><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px"> return (deliver);</div>
<div style="font-family:arial,sans-serif;font-size:13px">}</div><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">sub vcl_recv {</div><div style="font-family:arial,sans-serif;font-size:13px">
if (req.url ~ "^/seg\?") {</div><div style="font-family:arial,sans-serif;font-size:13px"> set req.url = regsub(req.url, "^/seg\?", "/db/client/1/seg.json?adsafe_url=");</div><div style="font-family:arial,sans-serif;font-size:13px">
}</div><div style="font-family:arial,sans-serif;font-size:13px"> else if (req.url ~ "^/raw\?") {</div><div style="font-family:arial,sans-serif;font-size:13px"> set req.url = regsub(req.url, "^/raw\?", "/db/client/1/itgrl.json?adsafe_url=");</div>
<div style="font-family:arial,sans-serif;font-size:13px"> }</div><div style="font-family:arial,sans-serif;font-size:13px"> else {</div><div style="font-family:arial,sans-serif;font-size:13px"> error 404 "Nope";</div>
<div style="font-family:arial,sans-serif;font-size:13px"> }</div><div style="font-family:arial,sans-serif;font-size:13px">}</div><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">
backend default {</div><div style="font-family:arial,sans-serif;font-size:13px"> .host = "localhost";</div><div style="font-family:arial,sans-serif;font-size:13px"> .port = "8080";</div><div style="font-family:arial,sans-serif;font-size:13px">
}</div><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">varnishstat:<br></div><div style="font-family:arial,sans-serif;font-size:13px"><br></div>
<div style="font-family:arial,sans-serif;font-size:13px">
<div>$ varnishstat -1</div><div>client_conn 118007 1.43 Client connections accepted</div><div>client_drop 0 0.00 Connection dropped, no sess/wrk</div><div>client_req <a href="tel:2674303461" value="+12674303461" target="_blank">2674303461</a> 32451.20 Client requests received</div>
<div>cache_hit 2581606982 31326.38 Cache hits</div><div>cache_hitpass 0 0.00 Cache hits for pass</div><div>cache_miss 92696487 1124.82 Cache misses</div><div>backend_conn 143901 1.75 Backend conn. success</div>
<div>backend_unhealthy 0 0.00 Backend conn. not attempted</div><div>backend_busy 0 0.00 Backend conn. too many</div><div>backend_fail 113149 1.37 Backend conn. failures</div>
<div>backend_reuse 92437824 1121.68 Backend conn. reuses</div><div>backend_toolate 143808 1.75 Backend conn. was closed</div><div>backend_recycle 92583388 1123.45 Backend conn. recycles</div>
<div>backend_retry 44 0.00 Backend conn. retry</div><div>fetch_head 0 0.00 Fetch head</div><div>fetch_length 92583297 1123.45 Fetch with Length</div><div>fetch_chunked 0 0.00 Fetch chunked</div>
<div>fetch_eof 0 0.00 Fetch EOF</div><div>fetch_bad 0 0.00 Fetch had bad headers</div><div>fetch_close 0 0.00 Fetch wanted close</div><div>fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed</div>
<div>fetch_zero 0 0.00 Fetch zero len</div><div>fetch_failed 0 0.00 Fetch failed</div><div>fetch_1xx 0 0.00 Fetch no body (1xx)</div><div>fetch_204 0 0.00 Fetch no body (204)</div>
<div>fetch_304 0 0.00 Fetch no body (304)</div><div>n_sess_mem 2193 . N struct sess_mem</div><div>n_sess 1743 . N struct sess</div><div>n_object 17035658 . N struct object</div>
<div>n_vampireobject 0 . N unresurrected objects</div><div>n_objectcore 17036857 . N struct objectcore</div><div>n_objecthead 17287988 . N struct objecthead</div>
<div>n_waitinglist 1224 . N struct waitinglist</div><div>n_vbc 21 . N struct vbc</div><div>n_wrk 1225 . N worker threads</div><div>n_wrk_create 4831 0.06 N worker threads created</div>
<div>n_wrk_failed 0 0.00 N worker threads not created</div><div>n_wrk_max 0 0.00 N worker threads limited</div><div>n_wrk_lqueue 0 0.00 work request queue length</div>
<div>n_wrk_queued 97676 1.19 N queued work requests</div><div>n_wrk_drop 0 0.00 N dropped work requests</div><div>n_backend 1 . N backends</div>
<div>
n_expired 1913697 . N expired objects</div><div>n_lru_nuked 73634035 . N LRU nuked objects</div><div>n_lru_moved 597753138 . N LRU moved objects</div><div>
losthdr 0 0.00 HTTP header overflows</div>
<div>n_objsendfile 0 0.00 Objects sent with sendfile</div><div>n_objwrite 2670567550 32405.87 Objects sent with write</div><div>n_objoverflow 0 0.00 Objects overflowing workspace</div>
<div>s_sess 118079 1.43 Total Sessions</div><div>s_req <a href="tel:2674303508" value="+12674303508" target="_blank">2674303508</a> 32451.20 Total Requests</div><div>s_pipe 0 0.00 Total pipe</div>
<div>s_pass 0 0.00 Total pass</div><div>s_fetch 92583297 1123.45 Total fetch</div><div>s_hdrbytes 582930172361 7073536.86 Total header bytes</div><div>s_bodybytes 447671320223 5432245.12 Total body bytes</div>
<div>sess_closed 113192 1.37 Session Closed</div><div>sess_pipeline 0 0.00 Session Pipeline</div><div>sess_readahead 0 0.00 Session Read Ahead</div><div>sess_linger <a href="tel:2674190310" value="+12674190310" target="_blank">2674190310</a> 32449.83 Session Linger</div>
<div>sess_herd 1005183056 12197.34 Session herd</div><div>shm_records 84088575128 1020368.59 SHM records</div><div>shm_writes <a href="tel:3867468359" value="+13867468359" target="_blank">3867468359</a> 46929.60 SHM writes</div>
<div>shm_flushes 3637 0.04 SHM flushes due to overflow</div><div>shm_cont 47998203 582.43 SHM MTX contention</div><div>shm_cycles 28175 0.34 SHM cycles through buffer</div>
<div>sms_nreq 113190 1.37 SMS allocator requests</div><div>sms_nobj 0 . SMS outstanding allocations</div><div>sms_nbytes 0 . SMS outstanding bytes</div>
<div>sms_balloc 47426610 . SMS bytes allocated</div><div>sms_bfree 47426610 . SMS bytes freed</div><div>backend_req 92566803 1123.25 Backend requests made</div>
<div>
n_vcl 1 0.00 N vcl total</div><div>n_vcl_avail 1 0.00 N vcl available</div><div>n_vcl_discard 0 0.00 N vcl discarded</div><div>n_ban 1 . N total active bans</div>
<div>n_ban_gone 1 . N total gone bans</div><div>n_ban_add 1 0.00 N new bans added</div><div>n_ban_retire 0 0.00 N old bans deleted</div><div>
n_ban_obj_test 0 0.00 N objects tested</div><div>n_ban_re_test 0 0.00 N regexps tested against</div><div>n_ban_dups 0 0.00 N duplicate bans removed</div>
<div>hcb_nolock 2665700150 32346.80 HCB Lookups without lock</div><div>hcb_lock 91663289 1112.28 HCB Lookups with lock</div><div>hcb_insert 91656230 1112.20 HCB Inserts</div>
<div>esi_errors 0 0.00 ESI parse errors (unlock)</div><div>esi_warnings 0 0.00 ESI parse warnings (unlock)</div><div>accept_fail 0 0.00 Accept failures</div>
<div>client_drop_late 0 0.00 Connection dropped late</div><div>uptime 82410 1.00 Client uptime</div><div>dir_dns_lookups 0 0.00 DNS director lookups</div>
<div>dir_dns_failed 0 0.00 DNS director failed lookups</div><div>dir_dns_hit 0 0.00 DNS director cached lookups hit</div><div>dir_dns_cache_full 0 0.00 DNS director full dnscache</div>
<div>vmods 0 . Loaded VMODs</div><div>n_gzip 0 0.00 Gzip operations</div><div>n_gunzip 0 0.00 Gunzip operations</div><div>sess_pipe_overflow 0 . Dropped sessions due to session pipe overflow</div>
<div>LCK.sms.creat 1 0.00 Created locks</div><div>LCK.sms.destroy 0 0.00 Destroyed locks</div><div>LCK.sms.locks 339570 4.12 Lock Operations</div><div>LCK.sms.colls 0 0.00 Collisions</div>
<div>LCK.smp.creat 0 0.00 Created locks</div><div>LCK.smp.destroy 0 0.00 Destroyed locks</div><div>LCK.smp.locks 0 0.00 Lock Operations</div><div>LCK.smp.colls 0 0.00 Collisions</div>
<div>LCK.sma.creat 2 0.00 Created locks</div><div>LCK.sma.destroy 0 0.00 Destroyed locks</div><div>LCK.sma.locks 466325781 5658.61 Lock Operations</div><div>LCK.sma.colls 0 0.00 Collisions</div>
<div>LCK.smf.creat 0 0.00 Created locks</div><div>LCK.smf.destroy 0 0.00 Destroyed locks</div><div>LCK.smf.locks 0 0.00 Lock Operations</div><div>LCK.smf.colls 0 0.00 Collisions</div>
<div>LCK.hsl.creat 0 0.00 Created locks</div><div>LCK.hsl.destroy 0 0.00 Destroyed locks</div><div>LCK.hsl.locks 0 0.00 Lock Operations</div><div>LCK.hsl.colls 0 0.00 Collisions</div>
<div>LCK.hcb.creat 1 0.00 Created locks</div><div>LCK.hcb.destroy 0 0.00 Destroyed locks</div><div>LCK.hcb.locks 166286374 2017.79 Lock Operations</div><div>LCK.hcb.colls 0 0.00 Collisions</div>
<div>LCK.hcl.creat 0 0.00 Created locks</div><div>LCK.hcl.destroy 0 0.00 Destroyed locks</div><div>LCK.hcl.locks 0 0.00 Lock Operations</div><div>LCK.hcl.colls 0 0.00 Collisions</div>
<div>LCK.vcl.creat 1 0.00 Created locks</div><div>LCK.vcl.destroy 0 0.00 Destroyed locks</div><div>LCK.vcl.locks 4238534 51.43 Lock Operations</div><div>LCK.vcl.colls 0 0.00 Collisions</div>
<div>LCK.stat.creat 1 0.00 Created locks</div><div>LCK.stat.destroy 0 0.00 Destroyed locks</div><div>LCK.stat.locks 118472 1.44 Lock Operations</div><div>LCK.stat.colls 0 0.00 Collisions</div>
<div>LCK.sessmem.creat 1 0.00 Created locks</div><div>LCK.sessmem.destroy 0 0.00 Destroyed locks</div><div>LCK.sessmem.locks 118731 1.44 Lock Operations</div><div>LCK.sessmem.colls 0 0.00 Collisions</div>
<div>LCK.wstat.creat 1 0.00 Created locks</div><div>LCK.wstat.destroy 0 0.00 Destroyed locks</div><div>LCK.wstat.locks 35382849 429.35 Lock Operations</div><div>LCK.wstat.colls 0 0.00 Collisions</div>
<div>LCK.herder.creat 1 0.00 Created locks</div><div>LCK.herder.destroy 0 0.00 Destroyed locks</div><div>LCK.herder.locks 24026 0.29 Lock Operations</div><div>LCK.herder.colls 0 0.00 Collisions</div>
<div>LCK.wq.creat 2 0.00 Created locks</div><div>LCK.wq.destroy 0 0.00 Destroyed locks</div><div>LCK.wq.locks <a href="tel:2013141564" value="+12013141564" target="_blank">2013141564</a> 24428.37 Lock Operations</div>
<div>LCK.wq.colls 0 0.00 Collisions</div><div>LCK.objhdr.creat 91644710 1112.06 Created locks</div><div>LCK.objhdr.destroy 74373301 902.48 Destroyed locks</div><div>LCK.objhdr.locks 10786415928 130887.22 Lock Operations</div>
<div>LCK.objhdr.colls 0 0.00 Collisions</div><div>LCK.exp.creat 1 0.00 Created locks</div><div>LCK.exp.destroy 0 0.00 Destroyed locks</div><div>LCK.exp.locks 168213500 2041.18 Lock Operations</div>
<div>LCK.exp.colls 0 0.00 Collisions</div><div>LCK.lru.creat 2 0.00 Created locks</div><div>LCK.lru.destroy 0 0.00 Destroyed locks</div><div>LCK.lru.locks 166217433 2016.96 Lock Operations</div>
<div>LCK.lru.colls 0 0.00 Collisions</div><div>LCK.cli.creat 1 0.00 Created locks</div><div>LCK.cli.destroy 0 0.00 Destroyed locks</div><div>LCK.cli.locks 27464 0.33 Lock Operations</div>
<div>LCK.cli.colls 0 0.00 Collisions</div><div>LCK.ban.creat 1 0.00 Created locks</div><div>LCK.ban.destroy 0 0.00 Destroyed locks</div><div>LCK.ban.locks 168213537 2041.18 Lock Operations</div>
<div>LCK.ban.colls 0 0.00 Collisions</div><div>LCK.vbp.creat 1 0.00 Created locks</div><div>LCK.vbp.destroy 0 0.00 Destroyed locks</div><div>LCK.vbp.locks 0 0.00 Lock Operations</div>
<div>LCK.vbp.colls 0 0.00 Collisions</div><div>LCK.vbe.creat 1 0.00 Created locks</div><div>LCK.vbe.destroy 0 0.00 Destroyed locks</div><div>LCK.vbe.locks 514115 6.24 Lock Operations</div>
<div>LCK.vbe.colls 0 0.00 Collisions</div><div>LCK.backend.creat 1 0.00 Created locks</div><div>LCK.backend.destroy 0 0.00 Destroyed locks</div><div>LCK.backend.locks 185937951 2256.25 Lock Operations</div>
<div>LCK.backend.colls 0 0.00 Collisions</div><div>SMA.s0.c_req 315117125 3823.77 Allocator requests</div><div>SMA.s0.c_fail 129837159 1575.50 Allocator failures</div><div>
SMA.s0.c_bytes 73800148989 895524.20 Bytes allocated</div><div>SMA.s0.c_freed 60915247331 739173.00 Bytes freed</div><div>SMA.s0.g_alloc 34071316 . Allocations outstanding</div><div>
SMA.s0.g_bytes 12884901658 . Bytes outstanding</div>
<div>SMA.s0.g_space 230 . Bytes available</div><div>SMA.Transient.c_req 12 0.00 Allocator requests</div><div>SMA.Transient.c_fail 0 0.00 Allocator failures</div>
<div>SMA.Transient.c_bytes 409344 4.97 Bytes allocated</div><div>SMA.Transient.c_freed 409344 4.97 Bytes freed</div><div>SMA.Transient.g_alloc 0 . Allocations outstanding</div>
<div>SMA.Transient.g_bytes 0 . Bytes outstanding</div><div>SMA.Transient.g_space 0 . Bytes available</div><div>VBE.default(38.74.193.146,,8080).vcls 1 . VCL references</div>
<div>VBE.default(38.74.193.146,,8080).happy 0 . Happy health probes</div></div><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">
Are we doing something stupid? We tried running Varnish 4.0.0-r1 but it was leaking threads so we downgraded...</div><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">
Thanks,</div><div style="font-family:arial,sans-serif;font-size:13px">Louis-Philippe Gauthier</div><div style="font-family:arial,sans-serif;font-size:13px">Team Lead, AdGear Trader</div></div>
</div><br></div>
</div><br></div>