Hi,<div><br></div><div>I have seen people report exactly the same error, but there is no solution to it. I hope some one here can shed some light on the issue. I saw a very high SMA.s0.c_fail "allocator failures". I think it is related. Is there a way to free some memory to just fit in the VCL config? I hate to restart the instance, since filling it up (160G) takes 1 day.</div>
<div><br></div><div>thanks,</div><div>Zheng</div><div><br></div><div><div><font face="courier new, monospace">vcl.load newconfig /etc/varnish/default.vcl</font></div><div><font face="courier new, monospace">106 </font></div>
<div><font face="courier new, monospace">VCL compiled.dlopen(./vcl.lA3o8iw_.so): ./vcl.lA3o8iw_.so: cannot apply additional memory protection after relocation: Cannot allocate memory</font></div></div><div><font face="courier new, monospace"><br>
</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"><div># /usr/sbin/varnishd -V</div><div>varnishd (varnish-3.0.3 revision 9e6a70f)</div><div>Copyright (c) 2006 Verdens Gang AS</div>
<div>Copyright (c) 2006-2011 Varnish Software AS</div><div><br></div><div><br></div></font></div><div><div><font face="courier new, monospace"># free -g</font></div><div><font face="courier new, monospace"> total used free shared buffers cached</font></div>
<div><font face="courier new, monospace">Mem: 189 176 12 0 0 0</font></div><div><font face="courier new, monospace">-/+ buffers/cache: 175 13</font></div>
<div><font face="courier new, monospace">Swap: 3 0 3</font></div></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"><br></font></div><div>
<font face="courier new, monospace"><div># uname -a</div><div>Linux <a href="http://hcache6.houzz.com">hcache6.houzz.com</a> 3.0.0-15-server #26-Ubuntu SMP Fri Jan 20 19:07:39 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux</div>
<div><br></div><div><br></div></font></div><div><font face="courier new, monospace"><div># ulimit -a</div><div>core file size (blocks, -c) 0</div><div>data seg size (kbytes, -d) unlimited</div><div>scheduling priority (-e) 0</div>
<div>file size (blocks, -f) unlimited</div><div>pending signals (-i) 1550517</div><div>max locked memory (kbytes, -l) 64</div><div>max memory size (kbytes, -m) unlimited</div><div>
open files (-n) 1024</div><div>pipe size (512 bytes, -p) 8</div><div>POSIX message queues (bytes, -q) 819200</div><div>real-time priority (-r) 0</div><div>stack size (kbytes, -s) 8192</div>
<div>cpu time (seconds, -t) unlimited</div><div>max user processes (-u) 1550517</div><div>virtual memory (kbytes, -v) unlimited</div><div>file locks (-x) unlimited</div>
<div><br></div><div><br></div></font></div><div><font face="courier new, monospace"><div># varnishstat -1</div><div>client_conn 520291099 520291099.00 Client connections accepted</div><div>client_drop 52 52.00 Connection dropped, no sess/wrk</div>
<div>client_req 520311099 520311099.00 Client requests received</div><div>cache_hit 479753215 479753215.00 Cache hits</div><div>cache_hitpass 5396 5396.00 Cache hits for pass</div><div>
cache_miss 40571613 40571613.00 Cache misses</div><div>backend_conn 18647472 18647472.00 Backend conn. success</div><div>backend_unhealthy 0 0.00 Backend conn. not attempted</div><div>
backend_busy 0 0.00 Backend conn. too many</div><div>backend_fail 84912 84912.00 Backend conn. failures</div><div>backend_reuse 21937169 21937169.00 Backend conn. reuses</div>
<div>backend_toolate 35368 35368.00 Backend conn. was closed</div><div>backend_recycle 21973014 21973014.00 Backend conn. recycles</div><div>backend_retry 1529 1529.00 Backend conn. retry</div>
<div>fetch_head 0 0.00 Fetch head</div><div>fetch_length 40567091 40567091.00 Fetch with Length</div><div>fetch_chunked 0 0.00 Fetch chunked</div><div>fetch_eof 0 0.00 Fetch EOF</div>
<div>fetch_bad 0 0.00 Fetch had bad headers</div><div>fetch_close 0 0.00 Fetch wanted close</div><div>fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed</div>
<div>fetch_zero 0 0.00 Fetch zero len</div><div>fetch_failed 1924 1924.00 Fetch failed</div><div>fetch_1xx 0 0.00 Fetch no body (1xx)</div><div>fetch_204 0 0.00 Fetch no body (204)</div>
<div>fetch_304 0 0.00 Fetch no body (304)</div><div>n_sess_mem 1119 . N struct sess_mem</div><div>n_sess 18 . N struct sess</div><div>n_object 4173835 . N struct object</div>
<div>n_vampireobject 0 . N unresurrected objects</div><div>n_objectcore 4174219 . N struct objectcore</div><div>n_objecthead 4190726 . N struct objecthead</div>
<div>n_waitinglist 12437 . N struct waitinglist</div><div>n_vbc 7 . N struct vbc</div><div>n_wrk 400 . N worker threads</div><div>n_wrk_create 465 465.00 N worker threads created</div>
<div>n_wrk_failed 256 256.00 N worker threads not created</div><div>n_wrk_max 0 0.00 N worker threads limited</div><div>n_wrk_lqueue 0 0.00 work request queue length</div>
<div>n_wrk_queued 16153 16153.00 N queued work requests</div><div>n_wrk_drop 53 53.00 N dropped work requests</div><div>n_backend 2 . N backends</div><div>
n_expired 191407 . N expired objects</div><div>n_lru_nuked 36119853 . N LRU nuked objects</div><div>n_lru_moved 424530660 . N LRU moved objects</div><div>losthdr 0 0.00 HTTP header overflows</div>
<div>n_objsendfile 0 0.00 Objects sent with sendfile</div><div>n_objwrite 529243867 529243867.00 Objects sent with write</div><div>n_objoverflow 0 0.00 Objects overflowing workspace</div>
<div>s_sess 520314370 520314370.00 Total Sessions</div><div>s_req 520311099 520311099.00 Total Requests</div><div>s_pipe 2677 2677.00 Total pipe</div><div>s_pass 37582 37582.00 Total pass</div>
<div>s_fetch 40511177 40511177.00 Total fetch</div><div>s_hdrbytes 203065278649 203065278649.00 Total header bytes</div><div>s_bodybytes 23783551235305 23783551235305.00 Total body bytes</div><div>
sess_closed 520303774 520303774.00 Session Closed</div><div>sess_pipeline 0 0.00 Session Pipeline</div><div>sess_readahead 0 0.00 Session Read Ahead</div><div>sess_linger 72892 72892.00 Session Linger</div>
<div>sess_herd 16001 16001.00 Session herd</div><div>shm_records 24046733247 24046733247.00 SHM records</div><div>shm_writes 2162622737 2162622737.00 SHM writes</div><div>shm_flushes 0 0.00 SHM flushes due to overflow</div>
<div>shm_cont 13971844 13971844.00 SHM MTX contention</div><div>shm_cycles 10341 10341.00 SHM cycles through buffer</div><div>sms_nreq 44028 44028.00 SMS allocator requests</div>
<div>sms_nobj 0 . SMS outstanding allocations</div><div>sms_nbytes 0 . SMS outstanding bytes</div><div>sms_balloc 18403704 . SMS bytes allocated</div>
<div>sms_bfree 18403704 . SMS bytes freed</div><div><div>backend_req 40582821 40582821.00 Backend requests made</div><div>n_vcl 1 1.00 N vcl total</div><div>
n_vcl_avail 1 1.00 N vcl available</div><div>n_vcl_discard 0 0.00 N vcl discarded</div><div>n_ban 1 . N total active bans</div><div>n_ban_gone 1 . N total gone bans</div>
<div>n_ban_add 1 1.00 N new bans added</div><div>n_ban_retire 0 0.00 N old bans deleted</div><div>n_ban_obj_test 0 0.00 N objects tested</div><div>n_ban_re_test 0 0.00 N regexps tested against</div>
<div>n_ban_dups 0 0.00 N duplicate bans removed</div><div>hcb_nolock 519442684 519442684.00 HCB Lookups without lock</div><div>hcb_lock 40524771 40524771.00 HCB Lookups with lock</div>
<div>hcb_insert 40524740 40524740.00 HCB Inserts</div><div>esi_errors 0 0.00 ESI parse errors (unlock)</div><div>esi_warnings 0 0.00 ESI parse warnings (unlock)</div>
<div>accept_fail 0 0.00 Accept failures</div><div>client_drop_late 1 1.00 Connection dropped late</div><div>uptime 755101 755101.00 Client uptime</div><div>
dir_dns_lookups 0 0.00 DNS director lookups</div><div>dir_dns_failed 0 0.00 DNS director failed lookups</div><div>dir_dns_hit 0 0.00 DNS director cached lookups hit</div>
<div>dir_dns_cache_full 0 0.00 DNS director full dnscache</div><div>vmods 0 . Loaded VMODs</div><div>n_gzip 0 0.00 Gzip operations</div><div>
n_gunzip 77505 77505.00 Gunzip operations</div><div>LCK.sms.creat 1 1.00 Created locks</div><div>LCK.sms.destroy 0 0.00 Destroyed locks</div><div>LCK.sms.locks 132084 132084.00 Lock Operations</div>
<div>LCK.sms.colls 0 0.00 Collisions</div><div>LCK.smp.creat 0 0.00 Created locks</div><div>LCK.smp.destroy 0 0.00 Destroyed locks</div><div>LCK.smp.locks 0 0.00 Lock Operations</div>
<div>LCK.smp.colls 0 0.00 Collisions</div><div>LCK.sma.creat 2 2.00 Created locks</div><div>LCK.sma.destroy 0 0.00 Destroyed locks</div><div>LCK.sma.locks 200409429 200409429.00 Lock Operations</div>
<div>LCK.sma.colls 0 0.00 Collisions</div><div>LCK.smf.creat 0 0.00 Created locks</div><div>LCK.smf.destroy 0 0.00 Destroyed locks</div><div>LCK.smf.locks 0 0.00 Lock Operations</div>
<div>LCK.smf.colls 0 0.00 Collisions</div><div>LCK.hsl.creat 0 0.00 Created locks</div><div>LCK.hsl.destroy 0 0.00 Destroyed locks</div><div>LCK.hsl.locks 0 0.00 Lock Operations</div>
<div>LCK.hsl.colls 0 0.00 Collisions</div><div>LCK.hcb.creat 1 1.00 Created locks</div><div>LCK.hcb.destroy 0 0.00 Destroyed locks</div><div>LCK.hcb.locks 76879907 76879907.00 Lock Operations</div>
<div>LCK.hcb.colls 0 0.00 Collisions</div><div>LCK.hcl.creat 0 0.00 Created locks</div><div>LCK.hcl.destroy 0 0.00 Destroyed locks</div><div>LCK.hcl.locks 0 0.00 Lock Operations</div>
<div>LCK.hcl.colls 0 0.00 Collisions</div><div>LCK.vcl.creat 1 1.00 Created locks</div><div>LCK.vcl.destroy 0 0.00 Destroyed locks</div><div>LCK.vcl.locks 26598 26598.00 Lock Operations</div>
<div>LCK.vcl.colls 0 0.00 Collisions</div><div>LCK.stat.creat 1 1.00 Created locks</div><div>LCK.stat.destroy 0 0.00 Destroyed locks</div><div>LCK.stat.locks 520315541 520315541.00 Lock Operations</div>
<div>LCK.stat.colls 0 0.00 Collisions</div><div>LCK.sessmem.creat 1 1.00 Created locks</div><div>LCK.sessmem.destroy 0 0.00 Destroyed locks</div><div>LCK.sessmem.locks 521596676 521596676.00 Lock Operations</div>
<div>LCK.sessmem.colls 0 0.00 Collisions</div><div>LCK.wstat.creat 1 1.00 Created locks</div><div>LCK.wstat.destroy 0 0.00 Destroyed locks</div><div>LCK.wstat.locks 1611720 1611720.00 Lock Operations</div>
<div>LCK.wstat.colls 0 0.00 Collisions</div><div>LCK.herder.creat 1 1.00 Created locks</div></div><div><div>LCK.herder.destroy 0 0.00 Destroyed locks</div><div>
LCK.herder.locks 2884 2884.00 Lock Operations</div><div>LCK.herder.colls 0 0.00 Collisions</div><div>LCK.wq.creat 8 8.00 Created locks</div><div>LCK.wq.destroy 0 0.00 Destroyed locks</div>
<div>LCK.wq.locks 1046638906 1046638906.00 Lock Operations</div><div>LCK.wq.colls 0 0.00 Collisions</div><div>LCK.objhdr.creat 40523726 40523726.00 Created locks</div><div>LCK.objhdr.destroy 36334480 36334480.00 Destroyed locks</div>
<div>LCK.objhdr.locks 2153405517 2153405517.00 Lock Operations</div><div>LCK.objhdr.colls 0 0.00 Collisions</div><div>LCK.exp.creat 1 1.00 Created locks</div><div>LCK.exp.destroy 0 0.00 Destroyed locks</div>
<div>LCK.exp.locks 77551306 77551306.00 Lock Operations</div><div>LCK.exp.colls 0 0.00 Collisions</div><div>LCK.lru.creat 2 2.00 Created locks</div><div>LCK.lru.destroy 0 0.00 Destroyed locks</div>
<div>LCK.lru.locks 76604938 76604938.00 Lock Operations</div><div>LCK.lru.colls 0 0.00 Collisions</div><div>LCK.cli.creat 1 1.00 Created locks</div><div>LCK.cli.destroy 0 0.00 Destroyed locks</div>
<div>LCK.cli.locks 65 65.00 Lock Operations</div><div>LCK.cli.colls 0 0.00 Collisions</div><div>LCK.ban.creat 1 1.00 Created locks</div><div>LCK.ban.destroy 0 0.00 Destroyed locks</div>
<div>LCK.ban.locks 77555176 77555176.00 Lock Operations</div><div>LCK.ban.colls 0 0.00 Collisions</div><div>LCK.vbp.creat 1 1.00 Created locks</div><div>LCK.vbp.destroy 0 0.00 Destroyed locks</div>
<div>LCK.vbp.locks 0 0.00 Lock Operations</div><div>LCK.vbp.colls 0 0.00 Collisions</div><div>LCK.vbe.creat 1 1.00 Created locks</div><div>LCK.vbe.destroy 0 0.00 Destroyed locks</div>
<div>LCK.vbe.locks 37466927 37466927.00 Lock Operations</div><div>LCK.vbe.colls 0 0.00 Collisions</div><div>LCK.backend.creat 3 3.00 Created locks</div><div>LCK.backend.destroy 0 0.00 Destroyed locks</div>
<div>LCK.backend.locks 100145698 100145698.00 Lock Operations</div><div>LCK.backend.colls 0 0.00 Collisions</div><div>SMA.s0.c_req 123055964 123055964.00 Allocator requests</div><div>SMA.s0.c_fail 3551561563273 3551561563273.00 Allocator failures</div>
<div>SMA.s0.c_bytes 1615116324709 1615116324709.00 Bytes allocated</div><div>SMA.s0.c_freed 1447612608403 1447612608403.00 Bytes freed</div><div>SMA.s0.g_alloc 8654138 . Allocations outstanding</div>
<div>SMA.s0.g_bytes 167503716306 . Bytes outstanding</div><div>SMA.s0.g_space 8238 . Bytes available</div><div>SMA.Transient.c_req 43934 43934.00 Allocator requests</div>
<div>SMA.Transient.c_fail 0 0.00 Allocator failures</div><div>SMA.Transient.c_bytes 646003217 646003217.00 Bytes allocated</div><div>SMA.Transient.c_freed 646003217 646003217.00 Bytes freed</div><div>
SMA.Transient.g_alloc 0 . Allocations outstanding</div><div>SMA.Transient.g_bytes 0 . Bytes outstanding</div><div>SMA.Transient.g_space 0 . Bytes available</div>
<div>VBE.node_master(192.168.0.21,,80).vcls 1 . VCL references</div><div>VBE.node_master(192.168.0.21,,80).happy 0 . Happy health probes</div><div>VBE.node_slave(192.168.0.39,,80).vcls 1 . VCL references</div>
<div>VBE.node_slave(192.168.0.39,,80).happy 0 . Happy health probes</div></div></font></div><div><br></div>