<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;"><span id="OLK_SRC_BODY_SECTION" style="font-family: Calibri, sans-serif;"><div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt"><br></div><blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;"><div><div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px"><br></div><div style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px">The system is currently fully using available physical memory and has burned through about 15% of the swap space. The varnishstat was taken after the physical memory was exhausted
and with just a few Gigs of swap in use. Here is a current snapshot from top and varnishstat –1 if that helps. </div></div></blockquote><div><br></div><div>Could it be that your memory allocator is leaking memory? The varnishstat looks decent, there are no counters that are off the charts. Last time we saw behaviour like that was when we were seeing seeing leaks with malloc on Linux, before we switched to
jemalloc.</div><div><br></div><div>I think Varnish will try to link with jemalloc if it is available on your system. Is it? As you can see there are quite a few allocations happening and if that memory is not reclaimed and reused you'll be leaking quite a bit.</div><div><br></div></div></div></div></div></div></blockquote></span><div style="font-family: Calibri, sans-serif;"><br></div><span id="OLK_SRC_BODY_SECTION" style="font-family: Calibri, sans-serif;"><div dir="ltr"><div class="gmail_extra">My understanding is that the base malloc in freebsd 9.2 is jemalloc — the code was ported back into the base malloc.c. Between FreeBSD 9.2 and FreeBSD 10.0, they changed how it is done, and now is using the pure jemalloc.c code that should be the same as in Linux.</div></div></span><div style="font-family: Calibri, sans-serif;"><br></div><div style="font-family: Calibri, sans-serif;"><a href="https://svnweb.freebsd.org/base/release/9.2.0/lib/libc/stdlib/malloc.c?revision=255898&view=markup">https://svnweb.freebsd.org/base/release/9.2.0/lib/libc/stdlib/malloc.c?revision=255898&view=markup</a> </div><div style="font-family: Calibri, sans-serif;"><br></div><div style="font-family: Calibri, sans-serif;">Vs</div><div style="font-family: Calibri, sans-serif;"><br></div><div style="font-family: Calibri, sans-serif;"><a href="https://svnweb.freebsd.org/base/release/10.0.0/contrib/jemalloc/src/jemalloc.c?revision=260789&view=markup">https://svnweb.freebsd.org/base/release/10.0.0/contrib/jemalloc/src/jemalloc.c?revision=260789&view=markup</a></div><div style="font-family: Calibri, sans-serif;"><br></div><div style="font-family: Calibri, sans-serif;">That said, we have a freebsd 10 server running the same version of varnishd that has the same problem (the reason I had discounted freebsd as the culprit initially). This server’s hardware is much newer and as you can see has quite a bit more physical memory as well. It is running with a few more threads as you can see, because it has more cpu cores and was configured with more thread_pools. Otherwise, it is the same as our 9.2 boxes.</div><div style="font-family: Calibri, sans-serif;"><br></div><div style="font-family: Calibri, sans-serif;">From our FreeBSD 10 box running varnish:</div><div style="font-family: Calibri, sans-serif;"><br></div><div><div><font face="Consolas">CPU: 4.0% user, 0.0% nice, 0.7% system, 0.2% interrupt, 95.1% idle</font></div><div><font face="Consolas">Mem: 4509M Active, 111G Inact, 6229M Wired, 73M Cache, 2757M Free</font></div><div><font face="Consolas">ARC: 1911M Total, 464M MFU, 730M MRU, 400K Anon, 390M Header, 326M Other</font></div><div><font face="Consolas">Swap: 32G Total, 8353M Used, 24G Free, 25% Inuse</font></div><div><font face="Consolas"><br></font></div><div><font face="Consolas"> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND</font></div><div><font face="Consolas"> 1311 www 1291 20 0 354G 116G uwait 8 476.0H 21.19% varnishd</font></div></div><div style="font-family: Calibri, sans-serif;"><br></div><div><div><font face="Consolas">$ uname -v</font></div><div><font face="Consolas">> FreeBSD 10.0-RELEASE-p7 #0: Tue Jul 8 06:37:44 UTC 2014 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC </font></div></div><div><font face="Consolas"><br></font></div><div><font face="Consolas">varnishstat –1:</font></div><div><div><font face="Consolas">client_conn 317914959 133.36 Client connections accepted</font></div><div><font face="Consolas">client_drop 0 0.00 Connection dropped, no sess/wrk</font></div><div><font face="Consolas">client_req 507413109 212.85 Client requests received</font></div><div><font face="Consolas">cache_hit 236222630 99.09 Cache hits</font></div><div><font face="Consolas">cache_hitpass 21505999 9.02 Cache hits for pass</font></div><div><font face="Consolas">cache_miss 213793146 89.68 Cache misses</font></div><div><font face="Consolas">backend_conn 111736980 46.87 Backend conn. success</font></div><div><font face="Consolas">backend_unhealthy 0 0.00 Backend conn. not attempted</font></div><div><font face="Consolas">backend_busy 8690564 3.65 Backend conn. too many</font></div><div><font face="Consolas">backend_fail 516882 0.22 Backend conn. failures</font></div><div><font face="Consolas">backend_reuse 265640872 111.43 Backend conn. reuses</font></div><div><font face="Consolas">backend_toolate 14139104 5.93 Backend conn. was closed</font></div><div><font face="Consolas">backend_recycle 279780619 117.36 Backend conn. recycles</font></div><div><font face="Consolas">backend_retry 25784 0.01 Backend conn. retry</font></div><div><font face="Consolas">fetch_head 1770 0.00 Fetch head</font></div><div><font face="Consolas">fetch_length 121268619 50.87 Fetch with Length</font></div><div><font face="Consolas">fetch_chunked 254268690 106.66 Fetch chunked</font></div><div><font face="Consolas">fetch_eof 0 0.00 Fetch EOF</font></div><div><font face="Consolas">fetch_bad 0 0.00 Fetch had bad headers</font></div><div><font face="Consolas">fetch_close 1590881 0.67 Fetch wanted close</font></div><div><font face="Consolas">fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed</font></div><div><font face="Consolas">fetch_zero 0 0.00 Fetch zero len</font></div><div><font face="Consolas">fetch_failed 167507 0.07 Fetch failed</font></div><div><font face="Consolas">fetch_1xx 0 0.00 Fetch no body (1xx)</font></div><div><font face="Consolas">fetch_204 0 0.00 Fetch no body (204)</font></div><div><font face="Consolas">fetch_304 172230 0.07 Fetch no body (304)</font></div><div><font face="Consolas">n_sess_mem 6946 . N struct sess_mem</font></div><div><font face="Consolas">n_sess 593 . N struct sess</font></div><div><font face="Consolas">n_object 992149 . N struct object</font></div><div><font face="Consolas">n_vampireobject 0 . N unresurrected objects</font></div><div><font face="Consolas">n_objectcore 993155 . N struct objectcore</font></div><div><font face="Consolas">n_objecthead 988241 . N struct objecthead</font></div><div><font face="Consolas">n_waitinglist 1273 . N struct waitinglist</font></div><div><font face="Consolas">n_vbc 71 . N struct vbc</font></div><div><font face="Consolas">n_wrk 1280 . N worker threads</font></div><div><font face="Consolas">n_wrk_create 1595 0.00 N worker threads created</font></div><div><font face="Consolas">n_wrk_failed 0 0.00 N worker threads not created</font></div><div><font face="Consolas">n_wrk_max 0 0.00 N worker threads limited</font></div><div><font face="Consolas">n_wrk_lqueue 0 0.00 work request queue length</font></div><div><font face="Consolas">n_wrk_queued 4530 0.00 N queued work requests</font></div><div><font face="Consolas">n_wrk_drop 0 0.00 N dropped work requests</font></div><div><font face="Consolas">n_backend 7 . N backends</font></div><div><font face="Consolas">n_expired 135419385 . N expired objects</font></div><div><font face="Consolas">n_lru_nuked 76410300 . N LRU nuked objects</font></div><div><font face="Consolas">n_lru_moved 128024378 . N LRU moved objects</font></div><div><font face="Consolas">losthdr 28 0.00 HTTP header overflows</font></div><div><font face="Consolas">n_objsendfile 0 0.00 Objects sent with sendfile</font></div><div><font face="Consolas">n_objwrite 355076718 148.95 Objects sent with write</font></div><div><font face="Consolas">n_objoverflow 0 0.00 Objects overflowing workspace</font></div><div><font face="Consolas">s_sess 317934238 133.37 Total Sessions</font></div><div><font face="Consolas">s_req 507413109 212.85 Total Requests</font></div><div><font face="Consolas">s_pipe 0 0.00 Total pipe</font></div><div><font face="Consolas">s_pass 165619470 69.47 Total pass</font></div><div><font face="Consolas">s_fetch 377135515 158.20 Total fetch</font></div><div><font face="Consolas">s_hdrbytes 189267454297 79393.14 Total header bytes</font></div><div><font face="Consolas">s_bodybytes 3732756334613 1565801.44 Total body bytes</font></div><div><font face="Consolas">sess_closed 35770111 15.00 Session Closed</font></div><div><font face="Consolas">sess_pipeline 24574 0.01 Session Pipeline</font></div><div><font face="Consolas">sess_readahead 7384 0.00 Session Read Ahead</font></div><div><font face="Consolas">sess_linger 476097250 199.71 Session Linger</font></div><div><font face="Consolas">sess_herd 494921706 207.61 Session herd</font></div><div><font face="Consolas">shm_records 45611717569 19133.02 SHM records</font></div><div><font face="Consolas">shm_writes 2842104094 1192.19 SHM writes</font></div><div><font face="Consolas">shm_flushes 2794 0.00 SHM flushes due to overflow</font></div><div><font face="Consolas">shm_cont 5861694 2.46 SHM MTX contention</font></div><div><font face="Consolas">shm_cycles 19426 0.01 SHM cycles through buffer</font></div><div><font face="Consolas">sms_nreq 2859746 1.20 SMS allocator requests</font></div><div><font face="Consolas">sms_nobj 0 . SMS outstanding allocations</font></div><div><font face="Consolas">sms_nbytes 0 . SMS outstanding bytes</font></div><div><font face="Consolas">sms_balloc 8118447295 . SMS bytes allocated</font></div><div><font face="Consolas">sms_bfree 8118447295 . SMS bytes freed</font></div><div><font face="Consolas">backend_req 377315426 158.27 Backend requests made</font></div><div><font face="Consolas">n_vcl 1 0.00 N vcl total</font></div><div><font face="Consolas">n_vcl_avail 1 0.00 N vcl available</font></div><div><font face="Consolas">n_vcl_discard 0 0.00 N vcl discarded</font></div><div><font face="Consolas">n_ban 1 . N total active bans</font></div><div><font face="Consolas">n_ban_gone 1 . N total gone bans</font></div><div><font face="Consolas">n_ban_add 1 0.00 N new bans added</font></div><div><font face="Consolas">n_ban_retire 0 0.00 N old bans deleted</font></div><div><font face="Consolas">n_ban_obj_test 0 0.00 N objects tested</font></div><div><font face="Consolas">n_ban_re_test 0 0.00 N regexps tested against</font></div><div><font face="Consolas">n_ban_dups 0 0.00 N duplicate bans removed</font></div><div><font face="Consolas">hcb_nolock 0 0.00 HCB Lookups without lock</font></div><div><font face="Consolas">hcb_lock 0 0.00 HCB Lookups with lock</font></div><div><font face="Consolas">hcb_insert 0 0.00 HCB Inserts</font></div><div><font face="Consolas">esi_errors 0 0.00 ESI parse errors (unlock)</font></div><div><font face="Consolas">esi_warnings 0 0.00 ESI parse warnings (unlock)</font></div><div><font face="Consolas">accept_fail 459 0.00 Accept failures</font></div><div><font face="Consolas">client_drop_late 0 0.00 Connection dropped late</font></div><div><font face="Consolas">uptime 2383927 1.00 Client uptime</font></div><div><font face="Consolas">dir_dns_lookups 0 0.00 DNS director lookups</font></div><div><font face="Consolas">dir_dns_failed 0 0.00 DNS director failed lookups</font></div><div><font face="Consolas">dir_dns_hit 0 0.00 DNS director cached lookups hit</font></div><div><font face="Consolas">dir_dns_cache_full 0 0.00 DNS director full dnscache</font></div><div><font face="Consolas">vmods 0 . Loaded VMODs</font></div><div><font face="Consolas">n_gzip 257811067 108.15 Gzip operations</font></div><div><font face="Consolas">n_gunzip 276663353 116.05 Gunzip operations</font></div><div><font face="Consolas">sess_pipe_overflow 0 . Dropped sessions due to session pipe overflow</font></div><div><font face="Consolas">LCK.sms.creat 1 0.00 Created locks</font></div><div><font face="Consolas">LCK.sms.destroy 0 0.00 Destroyed locks</font></div><div><font face="Consolas">LCK.sms.locks 8579238 3.60 Lock Operations</font></div><div><font face="Consolas">LCK.sms.colls 0 0.00 Collisions</font></div><div><font face="Consolas">LCK.smp.creat 0 0.00 Created locks</font></div><div><font face="Consolas">LCK.smp.destroy 0 0.00 Destroyed locks</font></div><div><font face="Consolas">LCK.smp.locks 0 0.00 Lock Operations</font></div><div><font face="Consolas">LCK.smp.colls 0 0.00 Collisions</font></div><div><font face="Consolas">LCK.sma.creat 2 0.00 Created locks</font></div><div><font face="Consolas">LCK.sma.destroy 0 0.00 Destroyed locks</font></div><div><font face="Consolas">LCK.sma.locks 2111376064 885.67 Lock Operations</font></div><div><font face="Consolas">LCK.sma.colls 0 0.00 Collisions</font></div><div><font face="Consolas">LCK.smf.creat 0 0.00 Created locks</font></div><div><font face="Consolas">LCK.smf.destroy 0 0.00 Destroyed locks</font></div><div><font face="Consolas">LCK.smf.locks 0 0.00 Lock Operations</font></div><div><font face="Consolas">LCK.smf.colls 0 0.00 Collisions</font></div><div><font face="Consolas">LCK.hsl.creat 0 0.00 Created locks</font></div><div><font face="Consolas">LCK.hsl.destroy 0 0.00 Destroyed locks</font></div><div><font face="Consolas">LCK.hsl.locks 0 0.00 Lock Operations</font></div><div><font face="Consolas">LCK.hsl.colls 0 0.00 Collisions</font></div><div><font face="Consolas">LCK.hcb.creat 0 0.00 Created locks</font></div><div><font face="Consolas">LCK.hcb.destroy 0 0.00 Destroyed locks</font></div><div><font face="Consolas">LCK.hcb.locks 0 0.00 Lock Operations</font></div><div><font face="Consolas">LCK.hcb.colls 0 0.00 Collisions</font></div><div><font face="Consolas">LCK.hcl.creat 250007 0.10 Created locks</font></div><div><font face="Consolas">LCK.hcl.destroy 0 0.00 Destroyed locks</font></div><div><font face="Consolas">LCK.hcl.locks 942040132 395.16 Lock Operations</font></div><div><font face="Consolas">LCK.hcl.colls 0 0.00 Collisions</font></div><div><font face="Consolas">LCK.vcl.creat 1 0.00 Created locks</font></div><div><font face="Consolas">LCK.vcl.destroy 0 0.00 Destroyed locks</font></div><div><font face="Consolas">LCK.vcl.locks 4406502 1.85 Lock Operations</font></div><div><font face="Consolas">LCK.vcl.colls 0 0.00 Collisions</font></div><div><font face="Consolas">LCK.stat.creat 1 0.00 Created locks</font></div><div><font face="Consolas">LCK.stat.destroy 0 0.00 Destroyed locks</font></div><div><font face="Consolas">LCK.stat.locks 317940634 133.37 Lock Operations</font></div><div><font face="Consolas">LCK.stat.colls 0 0.00 Collisions</font></div><div><font face="Consolas">LCK.sessmem.creat 1 0.00 Created locks</font></div><div><font face="Consolas">LCK.sessmem.destroy 0 0.00 Destroyed locks</font></div><div><font face="Consolas">LCK.sessmem.locks 318157037 133.46 Lock Operations</font></div><div><font face="Consolas">LCK.sessmem.colls 0 0.00 Collisions</font></div><div><font face="Consolas">LCK.wstat.creat 1 0.00 Created locks</font></div><div><font face="Consolas">LCK.wstat.destroy 0 0.00 Destroyed locks</font></div><div><font face="Consolas">LCK.wstat.locks 6160352 2.58 Lock Operations</font></div><div><font face="Consolas">LCK.wstat.colls 0 0.00 Collisions</font></div><div><font face="Consolas">LCK.herder.creat 1 0.00 Created locks</font></div><div><font face="Consolas">LCK.herder.destroy 0 0.00 Destroyed locks</font></div><div><font face="Consolas">LCK.herder.locks 1734 0.00 Lock Operations</font></div><div><font face="Consolas">LCK.herder.colls 0 0.00 Collisions</font></div><div><font face="Consolas">LCK.wq.creat 32 0.00 Created locks</font></div><div><font face="Consolas">LCK.wq.destroy 0 0.00 Destroyed locks</font></div><div><font face="Consolas">LCK.wq.locks 1140526679 478.42 Lock Operations</font></div><div><font face="Consolas">LCK.wq.colls 0 0.00 Collisions</font></div><div><font face="Consolas">LCK.objhdr.creat 183579053 77.01 Created locks</font></div><div><font face="Consolas">LCK.objhdr.destroy 182591336 76.59 Destroyed locks</font></div><div><font face="Consolas">LCK.objhdr.locks 1584172928 664.52 Lock Operations</font></div><div><font face="Consolas">LCK.objhdr.colls 0 0.00 Collisions</font></div><div><font face="Consolas">LCK.exp.creat 1 0.00 Created locks</font></div><div><font face="Consolas">LCK.exp.destroy 0 0.00 Destroyed locks</font></div><div><font face="Consolas">LCK.exp.locks 426984689 179.11 Lock Operations</font></div><div><font face="Consolas">LCK.exp.colls 0 0.00 Collisions</font></div><div><font face="Consolas">LCK.lru.creat 2 0.00 Created locks</font></div><div><font face="Consolas">LCK.lru.destroy 0 0.00 Destroyed locks</font></div><div><font face="Consolas">LCK.lru.locks 289232005 121.33 Lock Operations</font></div><div><font face="Consolas">LCK.lru.colls 0 0.00 Collisions</font></div><div><font face="Consolas">LCK.cli.creat 1 0.00 Created locks</font></div><div><font face="Consolas">LCK.cli.destroy 0 0.00 Destroyed locks</font></div><div><font face="Consolas">LCK.cli.locks 1180095 0.50 Lock Operations</font></div><div><font face="Consolas">LCK.cli.colls 0 0.00 Collisions</font></div><div><font face="Consolas">LCK.ban.creat 1 0.00 Created locks</font></div><div><font face="Consolas">LCK.ban.destroy 0 0.00 Destroyed locks</font></div><div><font face="Consolas">LCK.ban.locks 427224495 179.21 Lock Operations</font></div><div><font face="Consolas">LCK.ban.colls 0 0.00 Collisions</font></div><div><font face="Consolas">LCK.vbp.creat 1 0.00 Created locks</font></div><div><font face="Consolas">LCK.vbp.destroy 0 0.00 Destroyed locks</font></div><div><font face="Consolas">LCK.vbp.locks 1809670 0.76 Lock Operations</font></div><div><font face="Consolas">LCK.vbp.colls 0 0.00 Collisions</font></div><div><font face="Consolas">LCK.vbe.creat 1 0.00 Created locks</font></div><div><font face="Consolas">LCK.vbe.destroy 0 0.00 Destroyed locks</font></div><div><font face="Consolas">LCK.vbe.locks 224508313 94.18 Lock Operations</font></div><div><font face="Consolas">LCK.vbe.colls 0 0.00 Collisions</font></div><div><font face="Consolas">LCK.backend.creat 7 0.00 Created locks</font></div><div><font face="Consolas">LCK.backend.destroy 0 0.00 Destroyed locks</font></div><div><font face="Consolas">LCK.backend.locks 904980852 379.62 Lock Operations</font></div><div><font face="Consolas">LCK.backend.colls 0 0.00 Collisions</font></div><div><font face="Consolas">SMA.s0.c_req 578846458 242.81 Allocator requests</font></div><div><font face="Consolas">SMA.s0.c_fail 76630001 32.14 Allocator failures</font></div><div><font face="Consolas">SMA.s0.c_bytes 25682079523063 10773014.24 Bytes allocated</font></div><div><font face="Consolas">SMA.s0.c_freed 25675638298667 10770312.30 Bytes freed</font></div><div><font face="Consolas">SMA.s0.g_alloc 2023035 . Allocations outstanding</font></div><div><font face="Consolas">SMA.s0.g_bytes 6441224396 . Bytes outstanding</font></div><div><font face="Consolas">SMA.s0.g_space 1226548 . Bytes available</font></div><div><font face="Consolas">SMA.Transient.c_req 343584190 144.13 Allocator requests</font></div><div><font face="Consolas">SMA.Transient.c_fail 0 0.00 Allocator failures</font></div><div><font face="Consolas">SMA.Transient.c_bytes 21240906064202 8910048.87 Bytes allocated</font></div><div><font face="Consolas">SMA.Transient.c_freed 21240900389430 8910046.49 Bytes freed</font></div><div><font face="Consolas">SMA.Transient.g_alloc 5520 . Allocations outstanding</font></div><div><font face="Consolas">SMA.Transient.g_bytes 5674772 . Bytes outstanding</font></div><div><font face="Consolas">SMA.Transient.g_space 0 . Bytes available</font></div></div><div style="font-family: Calibri, sans-serif;"><br></div></body></html>