<div dir="ltr">I'm curious what the Virtual Memory vs Resident Set Size from top show. Most likely, the RSS will be stable and small. Virtual memory is almost valueless as a measure.<div><br></div><div>ps -eo pid,user,vsz,rss,%cpu,%mem,cmd | egrep "PID|varnish"<br></div><div><br></div><div>-Jason<br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Apr 18, 2016 at 4:21 AM, Tobias Eichelbrönner <span dir="ltr"><<a href="mailto:tobias.eichelbroenner@lamp-solutions.de" target="_blank">tobias.eichelbroenner@lamp-solutions.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi together,<br>
<br>
I have an old varnish 3.0.5 running. (I know its outdated, but my issue<br>
is more understanding what happens here then fixing the problem on the<br>
actual system). While running, the systems takes up more and more memory.<br>
<br>
I already switched to file storage, but still, after one day running and<br>
serving about 2 million requests, the varnish process takes more 20GB of<br>
ram.<br>
<br>
The first thing I am confused is, that as far as I see it varnishstats<br>
does not show the memory consumed (I will attach an output at the<br>
bottom). Tools like top and pmap show the output.<br>
<br>
pmap -x ...<br>
total kB 25368872 16712056 15197372<br>
<br>
Beside some VCL-Code, the varnish uses the vmods redis, digest, and std.<br>
<br>
Any idea how the find out what is causing the memory leak?<br>
<br>
Thanks in advance,<br>
<br>
Tobias<br>
<br>
Here is my varnishstat -1 output<br>
<br>
client_conn 2115657 18.63 Client connections accepted<br>
client_drop 0 0.00 Connection dropped, no sess/wrk<br>
client_req 5176344 45.58 Client requests received<br>
cache_hit 2207169 19.43 Cache hits<br>
cache_hitpass 18635 0.16 Cache hits for pass<br>
cache_miss 508733 4.48 Cache misses<br>
backend_conn 590509 5.20 Backend conn. success<br>
backend_unhealthy 0 0.00 Backend conn. not attempted<br>
backend_busy 0 0.00 Backend conn. too many<br>
backend_fail 0 0.00 Backend conn. failures<br>
backend_reuse 2391162 21.05 Backend conn. reuses<br>
backend_toolate 581 0.01 Backend conn. was closed<br>
backend_recycle 2392558 21.07 Backend conn. recycles<br>
backend_retry 142 0.00 Backend conn. retry<br>
fetch_head 604 0.01 Fetch head<br>
fetch_length 1493458 13.15 Fetch with Length<br>
fetch_chunked 1277164 11.25 Fetch chunked<br>
fetch_eof 0 0.00 Fetch EOF<br>
fetch_bad 0 0.00 Fetch had bad headers<br>
fetch_close 9969 0.09 Fetch wanted close<br>
fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed<br>
fetch_zero 0 0.00 Fetch zero len<br>
fetch_failed 528 0.00 Fetch failed<br>
fetch_1xx 0 0.00 Fetch no body (1xx)<br>
fetch_204 186330 1.64 Fetch no body (204)<br>
fetch_304 14773 0.13 Fetch no body (304)<br>
n_sess_mem 307 . N struct sess_mem<br>
n_sess 59 . N struct sess<br>
n_object 59401 . N struct object<br>
n_vampireobject 0 . N unresurrected objects<br>
n_objectcore 59439 . N struct objectcore<br>
n_objecthead 60591 . N struct objecthead<br>
n_waitinglist 44 . N struct waitinglist<br>
n_vbc 2 . N struct vbc<br>
n_wrk 44 . N worker threads<br>
n_wrk_create 1296 0.01 N worker threads created<br>
n_wrk_failed 0 0.00 N worker threads not created<br>
n_wrk_max 0 0.00 N worker threads limited<br>
n_wrk_lqueue 0 0.00 work request queue length<br>
n_wrk_queued 7886 0.07 N queued work requests<br>
n_wrk_drop 0 0.00 N dropped work requests<br>
n_backend 2 . N backends<br>
n_expired 449200 . N expired objects<br>
n_lru_nuked 0 . N LRU nuked objects<br>
n_lru_moved 1917928 . N LRU moved objects<br>
losthdr 0 0.00 HTTP header overflows<br>
n_objsendfile 0 0.00 Objects sent with sendfile<br>
n_objwrite 4825557 42.49 Objects sent with write<br>
n_objoverflow 0 0.00 Objects overflowing workspace<br>
s_sess 2115659 18.63 Total Sessions<br>
s_req 5176344 45.58 Total Requests<br>
s_pipe 1304 0.01 Total pipe<br>
s_pass 2473728 21.78 Total pass<br>
s_fetch 2981770 26.25 Total fetch<br>
s_hdrbytes 2065942876 18190.28 Total header bytes<br>
s_bodybytes 133824929675 1178306.04 Total body bytes<br>
sess_closed 418834 3.69 Session Closed<br>
sess_pipeline 63774 0.56 Session Pipeline<br>
sess_readahead 7618 0.07 Session Read Ahead<br>
sess_linger 4929655 43.40 Session Linger<br>
sess_herd 4533770 39.92 Session herd<br>
shm_records 397316593 3498.31 SHM records<br>
shm_writes 22134819 194.89 SHM writes<br>
shm_flushes 0 0.00 SHM flushes due to overflow<br>
shm_cont 83004 0.73 SHM MTX contention<br>
shm_cycles 173 0.00 SHM cycles through buffer<br>
sms_nreq 693 0.01 SMS allocator requests<br>
sms_nobj 0 . SMS outstanding allocations<br>
sms_nbytes 0 . SMS outstanding bytes<br>
sms_balloc 472858 . SMS bytes allocated<br>
sms_bfree 472858 . SMS bytes freed<br>
backend_req 2982527 26.26 Backend requests made<br>
n_vcl 2 0.00 N vcl total<br>
n_vcl_avail 2 0.00 N vcl available<br>
n_vcl_discard 0 0.00 N vcl discarded<br>
n_ban 1 . N total active bans<br>
n_ban_gone 1 . N total gone bans<br>
n_ban_add 1 0.00 N new bans added<br>
n_ban_retire 0 0.00 N old bans deleted<br>
n_ban_obj_test 0 0.00 N objects tested<br>
n_ban_re_test 0 0.00 N regexps tested against<br>
n_ban_dups 0 0.00 N duplicate bans removed<br>
hcb_nolock 2729927 24.04 HCB Lookups without lock<br>
hcb_lock 470290 4.14 HCB Lookups with lock<br>
hcb_insert 470044 4.14 HCB Inserts<br>
esi_errors 0 0.00 ESI parse errors (unlock)<br>
esi_warnings 0 0.00 ESI parse warnings (unlock)<br>
accept_fail 0 0.00 Accept failures<br>
client_drop_late 0 0.00 Connection dropped late<br>
uptime 113574 1.00 Client uptime<br>
dir_dns_lookups 0 0.00 DNS director lookups<br>
dir_dns_failed 0 0.00 DNS director failed lookups<br>
dir_dns_hit 0 0.00 DNS director cached lookups hit<br>
dir_dns_cache_full 0 0.00 DNS director full dnscache<br>
vmods 3 . Loaded VMODs<br>
n_gzip 0 0.00 Gzip operations<br>
n_gunzip 2590700 22.81 Gunzip operations<br>
sess_pipe_overflow 0 . Dropped sessions due to<br>
session pipe overflow<br>
LCK.sms.creat 1 0.00 Created locks<br>
LCK.sms.destroy 0 0.00 Destroyed locks<br>
LCK.sms.locks 2079 0.02 Lock Operations<br>
LCK.sms.colls 0 0.00 Collisions<br>
LCK.smp.creat 0 0.00 Created locks<br>
LCK.smp.destroy 0 0.00 Destroyed locks<br>
LCK.smp.locks 0 0.00 Lock Operations<br>
LCK.smp.colls 0 0.00 Collisions<br>
LCK.sma.creat 1 0.00 Created locks<br>
LCK.sma.destroy 0 0.00 Destroyed locks<br>
LCK.sma.locks 11567121 101.85 Lock Operations<br>
LCK.sma.colls 0 0.00 Collisions<br>
LCK.smf.creat 1 0.00 Created locks<br>
LCK.smf.destroy 0 0.00 Destroyed locks<br>
LCK.smf.locks 2137363 18.82 Lock Operations<br>
LCK.smf.colls 0 0.00 Collisions<br>
LCK.hsl.creat 0 0.00 Created locks<br>
LCK.hsl.destroy 0 0.00 Destroyed locks<br>
LCK.hsl.locks 0 0.00 Lock Operations<br>
LCK.hsl.colls 0 0.00 Collisions<br>
LCK.hcb.creat 1 0.00 Created locks<br>
LCK.hcb.destroy 0 0.00 Destroyed locks<br>
LCK.hcb.locks 881563 7.76 Lock Operations<br>
LCK.hcb.colls 0 0.00 Collisions<br>
LCK.hcl.creat 0 0.00 Created locks<br>
LCK.hcl.destroy 0 0.00 Destroyed locks<br>
LCK.hcl.locks 0 0.00 Lock Operations<br>
LCK.hcl.colls 0 0.00 Collisions<br>
LCK.vcl.creat 1 0.00 Created locks<br>
LCK.vcl.destroy 0 0.00 Destroyed locks<br>
LCK.vcl.locks 10574 0.09 Lock Operations<br>
LCK.vcl.colls 0 0.00 Collisions<br>
LCK.stat.creat 1 0.00 Created locks<br>
LCK.stat.destroy 0 0.00 Destroyed locks<br>
LCK.stat.locks 2115908 18.63 Lock Operations<br>
LCK.stat.colls 0 0.00 Collisions<br>
LCK.sessmem.creat 1 0.00 Created locks<br>
LCK.sessmem.destroy 0 0.00 Destroyed locks<br>
LCK.sessmem.locks 2129936 18.75 Lock Operations<br>
LCK.sessmem.colls 0 0.00 Collisions<br>
LCK.wstat.creat 1 0.00 Created locks<br>
LCK.wstat.destroy 0 0.00 Destroyed locks<br>
LCK.wstat.locks 239816 2.11 Lock Operations<br>
LCK.wstat.colls 0 0.00 Collisions<br>
LCK.herder.creat 1 0.00 Created locks<br>
LCK.herder.destroy 0 0.00 Destroyed locks<br>
LCK.herder.locks 6034 0.05 Lock Operations<br>
LCK.herder.colls 0 0.00 Collisions<br>
LCK.wq.creat 2 0.00 Created locks<br>
LCK.wq.destroy 0 0.00 Destroyed locks<br>
LCK.wq.locks 10137930 89.26 Lock Operations<br>
LCK.wq.colls 0 0.00 Collisions<br>
LCK.objhdr.creat 471022 4.15 Created locks<br>
LCK.objhdr.destroy 410522 3.61 Destroyed locks<br>
LCK.objhdr.locks 11873362 104.54 Lock Operations<br>
LCK.objhdr.colls 0 0.00 Collisions<br>
LCK.exp.creat 1 0.00 Created locks<br>
LCK.exp.destroy 0 0.00 Destroyed locks<br>
LCK.exp.locks 1086349 9.57 Lock Operations<br>
LCK.exp.colls 0 0.00 Collisions<br>
LCK.lru.creat 2 0.00 Created locks<br>
LCK.lru.destroy 0 0.00 Destroyed locks<br>
LCK.lru.locks 523780 4.61 Lock Operations<br>
LCK.lru.colls 0 0.00 Collisions<br>
LCK.cli.creat 1 0.00 Created locks<br>
LCK.cli.destroy 0 0.00 Destroyed locks<br>
LCK.cli.locks 877 0.01 Lock Operations<br>
LCK.cli.colls 0 0.00 Collisions<br>
LCK.ban.creat 1 0.00 Created locks<br>
LCK.ban.destroy 0 0.00 Destroyed locks<br>
LCK.ban.locks 1071376 9.43 Lock Operations<br>
LCK.ban.colls 0 0.00 Collisions<br>
LCK.vbp.creat 1 0.00 Created locks<br>
LCK.vbp.destroy 0 0.00 Destroyed locks<br>
LCK.vbp.locks 0 0.00 Lock Operations<br>
LCK.vbp.colls 0 0.00 Collisions<br>
LCK.vbe.creat 1 0.00 Created locks<br>
LCK.vbe.destroy 0 0.00 Destroyed locks<br>
LCK.vbe.locks 1183870 10.42 Lock Operations<br>
LCK.vbe.colls 0 0.00 Collisions<br>
LCK.backend.creat 2 0.00 Created locks<br>
LCK.backend.destroy 0 0.00 Destroyed locks<br>
LCK.backend.locks 6560980 57.77 Lock Operations<br>
LCK.backend.colls 0 0.00 Collisions<br>
SMF.s0.c_req 943045 8.30 Allocator requests<br>
SMF.s0.c_fail 0 0.00 Allocator failures<br>
SMF.s0.c_bytes 57519628288 506450.67 Bytes allocated<br>
SMF.s0.c_freed 54163292160 476898.69 Bytes freed<br>
SMF.s0.g_alloc 129160 . Allocations outstanding<br>
SMF.s0.g_bytes 3356336128 . Bytes outstanding<br>
SMF.s0.g_space 938631168 . Bytes available<br>
SMF.s0.g_smf 138281 . N struct smf<br>
SMF.s0.g_smf_frag 9118 . N small free smf<br>
SMF.s0.g_smf_large 3 . N large free smf<br>
SMA.Transient.c_req 4780332 42.09 Allocator requests<br>
SMA.Transient.c_fail 0 0.00 Allocator failures<br>
SMA.Transient.c_bytes 284689405959 2506642.42 Bytes allocated<br>
SMA.Transient.c_freed 284688351047 2506633.13 Bytes freed<br>
SMA.Transient.g_alloc 85 . Allocations outstanding<br>
SMA.Transient.g_bytes 1054912 . Bytes outstanding<br>
SMA.Transient.g_space 0 . Bytes available<br>
VBE.default(127.0.0.1,,8080).vcls 2 . VCL references<br>
VBE.default(127.0.0.1,,8080).happy 0 . Happy health<br>
probes<br>
VBE.fpm(127.0.0.1,,8081).vcls 2 . VCL references<br>
VBE.fpm(127.0.0.1,,8081).happy 0 . Happy health<br>
probes<br>
<br>
--<br>
LAMP solutions GmbH<br>
Gostenhofer Hauptstrasse 35<br>
90443 Nuernberg<br>
<br>
Amtsgericht Nuernberg: HRB 22366<br>
Geschaeftsfuehrer: Heiko Schubert<br>
<br>
Es gelten unsere allgemeinen Geschaeftsbedingungen.<br>
<a href="http://www.lamp-solutions.de/agb/" rel="noreferrer" target="_blank">http://www.lamp-solutions.de/agb/</a><br>
<br>
Telefon : 0911 / 376 516 0<br>
Fax : 0911 / 376 516 11<br>
E-Mail : <a href="mailto:support@lamp-solutions.de">support@lamp-solutions.de</a><br>
Web : <a href="http://www.lamp-solutions.de" rel="noreferrer" target="_blank">www.lamp-solutions.de</a><br>
Facebook : <a href="http://www.facebook.com/LAMPsolutions" rel="noreferrer" target="_blank">http://www.facebook.com/LAMPsolutions</a><br>
Twitter : <a href="http://twitter.com/#!/lampsolutions" rel="noreferrer" target="_blank">http://twitter.com/#!/lampsolutions</a><br>
<br>
_______________________________________________<br>
varnish-misc mailing list<br>
<a href="mailto:varnish-misc@varnish-cache.org">varnish-misc@varnish-cache.org</a><br>
<a href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc" rel="noreferrer" target="_blank">https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc</a><br>
</blockquote></div><br></div>