can not load more VCL with error "cannot allocate memory", but with plenty memory left

Zheng Liu huozhe at gmail.com
Sat Sep 8 02:43:14 CEST 2012


Hi,

I have seen people report exactly the same error, but there is no solution
to it. I hope some one here can shed some light on the issue. I saw a very
high SMA.s0.c_fail "allocator failures". I think it is related. Is there a
way to free some memory to just fit in the VCL config? I hate to restart
the instance, since filling it up (160G) takes 1 day.

thanks,
Zheng

vcl.load newconfig /etc/varnish/default.vcl
106
VCL compiled.dlopen(./vcl.lA3o8iw_.so): ./vcl.lA3o8iw_.so: cannot apply
additional memory protection after relocation: Cannot allocate memory


# /usr/sbin/varnishd -V
varnishd (varnish-3.0.3 revision 9e6a70f)
Copyright (c) 2006 Verdens Gang AS
Copyright (c) 2006-2011 Varnish Software AS


# free -g
             total       used       free     shared    buffers     cached
Mem:           189        176         12          0          0          0
-/+ buffers/cache:        175         13
Swap:            3          0          3


# uname -a
Linux hcache6.houzz.com 3.0.0-15-server #26-Ubuntu SMP Fri Jan 20 19:07:39
UTC 2012 x86_64 x86_64 x86_64 GNU/Linux


# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 1550517
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1550517
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited


# varnishstat -1
client_conn          520291099 520291099.00 Client connections accepted
client_drop                 52        52.00 Connection dropped, no sess/wrk
client_req           520311099 520311099.00 Client requests received
cache_hit            479753215 479753215.00 Cache hits
cache_hitpass             5396      5396.00 Cache hits for pass
cache_miss            40571613  40571613.00 Cache misses
backend_conn          18647472  18647472.00 Backend conn. success
backend_unhealthy            0         0.00 Backend conn. not attempted
backend_busy                 0         0.00 Backend conn. too many
backend_fail             84912     84912.00 Backend conn. failures
backend_reuse         21937169  21937169.00 Backend conn. reuses
backend_toolate          35368     35368.00 Backend conn. was closed
backend_recycle       21973014  21973014.00 Backend conn. recycles
backend_retry             1529      1529.00 Backend conn. retry
fetch_head                   0         0.00 Fetch head
fetch_length          40567091  40567091.00 Fetch with Length
fetch_chunked                0         0.00 Fetch chunked
fetch_eof                    0         0.00 Fetch EOF
fetch_bad                    0         0.00 Fetch had bad headers
fetch_close                  0         0.00 Fetch wanted close
fetch_oldhttp                0         0.00 Fetch pre HTTP/1.1 closed
fetch_zero                   0         0.00 Fetch zero len
fetch_failed              1924      1924.00 Fetch failed
fetch_1xx                    0         0.00 Fetch no body (1xx)
fetch_204                    0         0.00 Fetch no body (204)
fetch_304                    0         0.00 Fetch no body (304)
n_sess_mem                1119          .   N struct sess_mem
n_sess                      18          .   N struct sess
n_object               4173835          .   N struct object
n_vampireobject              0          .   N unresurrected objects
n_objectcore           4174219          .   N struct objectcore
n_objecthead           4190726          .   N struct objecthead
n_waitinglist            12437          .   N struct waitinglist
n_vbc                        7          .   N struct vbc
n_wrk                      400          .   N worker threads
n_wrk_create               465       465.00 N worker threads created
n_wrk_failed               256       256.00 N worker threads not created
n_wrk_max                    0         0.00 N worker threads limited
n_wrk_lqueue                 0         0.00 work request queue length
n_wrk_queued             16153     16153.00 N queued work requests
n_wrk_drop                  53        53.00 N dropped work requests
n_backend                    2          .   N backends
n_expired               191407          .   N expired objects
n_lru_nuked           36119853          .   N LRU nuked objects
n_lru_moved          424530660          .   N LRU moved objects
losthdr                      0         0.00 HTTP header overflows
n_objsendfile                0         0.00 Objects sent with sendfile
n_objwrite           529243867 529243867.00 Objects sent with write
n_objoverflow                0         0.00 Objects overflowing workspace
s_sess               520314370 520314370.00 Total Sessions
s_req                520311099 520311099.00 Total Requests
s_pipe                    2677      2677.00 Total pipe
s_pass                   37582     37582.00 Total pass
s_fetch               40511177  40511177.00 Total fetch
s_hdrbytes        203065278649 203065278649.00 Total header bytes
s_bodybytes       23783551235305 23783551235305.00 Total body bytes
sess_closed          520303774 520303774.00 Session Closed
sess_pipeline                0         0.00 Session Pipeline
sess_readahead               0         0.00 Session Read Ahead
sess_linger              72892     72892.00 Session Linger
sess_herd                16001     16001.00 Session herd
shm_records        24046733247 24046733247.00 SHM records
shm_writes          2162622737 2162622737.00 SHM writes
shm_flushes                  0         0.00 SHM flushes due to overflow
shm_cont              13971844  13971844.00 SHM MTX contention
shm_cycles               10341     10341.00 SHM cycles through buffer
sms_nreq                 44028     44028.00 SMS allocator requests
sms_nobj                     0          .   SMS outstanding allocations
sms_nbytes                   0          .   SMS outstanding bytes
sms_balloc            18403704          .   SMS bytes allocated
sms_bfree             18403704          .   SMS bytes freed
backend_req           40582821  40582821.00 Backend requests made
n_vcl                        1         1.00 N vcl total
n_vcl_avail                  1         1.00 N vcl available
n_vcl_discard                0         0.00 N vcl discarded
n_ban                        1          .   N total active bans
n_ban_gone                   1          .   N total gone bans
n_ban_add                    1         1.00 N new bans added
n_ban_retire                 0         0.00 N old bans deleted
n_ban_obj_test               0         0.00 N objects tested
n_ban_re_test                0         0.00 N regexps tested against
n_ban_dups                   0         0.00 N duplicate bans removed
hcb_nolock           519442684 519442684.00 HCB Lookups without lock
hcb_lock              40524771  40524771.00 HCB Lookups with lock
hcb_insert            40524740  40524740.00 HCB Inserts
esi_errors                   0         0.00 ESI parse errors (unlock)
esi_warnings                 0         0.00 ESI parse warnings (unlock)
accept_fail                  0         0.00 Accept failures
client_drop_late             1         1.00 Connection dropped late
uptime                  755101    755101.00 Client uptime
dir_dns_lookups              0         0.00 DNS director lookups
dir_dns_failed               0         0.00 DNS director failed lookups
dir_dns_hit                  0         0.00 DNS director cached lookups hit
dir_dns_cache_full           0         0.00 DNS director full dnscache
vmods                        0          .   Loaded VMODs
n_gzip                       0         0.00 Gzip operations
n_gunzip                 77505     77505.00 Gunzip operations
LCK.sms.creat                1         1.00 Created locks
LCK.sms.destroy              0         0.00 Destroyed locks
LCK.sms.locks           132084    132084.00 Lock Operations
LCK.sms.colls                0         0.00 Collisions
LCK.smp.creat                0         0.00 Created locks
LCK.smp.destroy              0         0.00 Destroyed locks
LCK.smp.locks                0         0.00 Lock Operations
LCK.smp.colls                0         0.00 Collisions
LCK.sma.creat                2         2.00 Created locks
LCK.sma.destroy              0         0.00 Destroyed locks
LCK.sma.locks        200409429 200409429.00 Lock Operations
LCK.sma.colls                0         0.00 Collisions
LCK.smf.creat                0         0.00 Created locks
LCK.smf.destroy              0         0.00 Destroyed locks
LCK.smf.locks                0         0.00 Lock Operations
LCK.smf.colls                0         0.00 Collisions
LCK.hsl.creat                0         0.00 Created locks
LCK.hsl.destroy              0         0.00 Destroyed locks
LCK.hsl.locks                0         0.00 Lock Operations
LCK.hsl.colls                0         0.00 Collisions
LCK.hcb.creat                1         1.00 Created locks
LCK.hcb.destroy              0         0.00 Destroyed locks
LCK.hcb.locks         76879907  76879907.00 Lock Operations
LCK.hcb.colls                0         0.00 Collisions
LCK.hcl.creat                0         0.00 Created locks
LCK.hcl.destroy              0         0.00 Destroyed locks
LCK.hcl.locks                0         0.00 Lock Operations
LCK.hcl.colls                0         0.00 Collisions
LCK.vcl.creat                1         1.00 Created locks
LCK.vcl.destroy              0         0.00 Destroyed locks
LCK.vcl.locks            26598     26598.00 Lock Operations
LCK.vcl.colls                0         0.00 Collisions
LCK.stat.creat               1         1.00 Created locks
LCK.stat.destroy             0         0.00 Destroyed locks
LCK.stat.locks       520315541 520315541.00 Lock Operations
LCK.stat.colls               0         0.00 Collisions
LCK.sessmem.creat            1         1.00 Created locks
LCK.sessmem.destroy            0         0.00 Destroyed locks
LCK.sessmem.locks      521596676 521596676.00 Lock Operations
LCK.sessmem.colls              0         0.00 Collisions
LCK.wstat.creat                1         1.00 Created locks
LCK.wstat.destroy              0         0.00 Destroyed locks
LCK.wstat.locks          1611720   1611720.00 Lock Operations
LCK.wstat.colls                0         0.00 Collisions
LCK.herder.creat               1         1.00 Created locks
LCK.herder.destroy             0         0.00 Destroyed locks
LCK.herder.locks            2884      2884.00 Lock Operations
LCK.herder.colls               0         0.00 Collisions
LCK.wq.creat                   8         8.00 Created locks
LCK.wq.destroy                 0         0.00 Destroyed locks
LCK.wq.locks          1046638906 1046638906.00 Lock Operations
LCK.wq.colls                   0         0.00 Collisions
LCK.objhdr.creat        40523726  40523726.00 Created locks
LCK.objhdr.destroy      36334480  36334480.00 Destroyed locks
LCK.objhdr.locks      2153405517 2153405517.00 Lock Operations
LCK.objhdr.colls               0         0.00 Collisions
LCK.exp.creat                  1         1.00 Created locks
LCK.exp.destroy                0         0.00 Destroyed locks
LCK.exp.locks           77551306  77551306.00 Lock Operations
LCK.exp.colls                  0         0.00 Collisions
LCK.lru.creat                  2         2.00 Created locks
LCK.lru.destroy                0         0.00 Destroyed locks
LCK.lru.locks           76604938  76604938.00 Lock Operations
LCK.lru.colls                  0         0.00 Collisions
LCK.cli.creat                  1         1.00 Created locks
LCK.cli.destroy                0         0.00 Destroyed locks
LCK.cli.locks                 65        65.00 Lock Operations
LCK.cli.colls                  0         0.00 Collisions
LCK.ban.creat                  1         1.00 Created locks
LCK.ban.destroy                0         0.00 Destroyed locks
LCK.ban.locks           77555176  77555176.00 Lock Operations
LCK.ban.colls                  0         0.00 Collisions
LCK.vbp.creat                  1         1.00 Created locks
LCK.vbp.destroy                0         0.00 Destroyed locks
LCK.vbp.locks                  0         0.00 Lock Operations
LCK.vbp.colls                  0         0.00 Collisions
LCK.vbe.creat                  1         1.00 Created locks
LCK.vbe.destroy                0         0.00 Destroyed locks
LCK.vbe.locks           37466927  37466927.00 Lock Operations
LCK.vbe.colls                  0         0.00 Collisions
LCK.backend.creat              3         3.00 Created locks
LCK.backend.destroy            0         0.00 Destroyed locks
LCK.backend.locks      100145698 100145698.00 Lock Operations
LCK.backend.colls              0         0.00 Collisions
SMA.s0.c_req           123055964 123055964.00 Allocator requests
SMA.s0.c_fail       3551561563273 3551561563273.00 Allocator failures
SMA.s0.c_bytes      1615116324709 1615116324709.00 Bytes allocated
SMA.s0.c_freed      1447612608403 1447612608403.00 Bytes freed
SMA.s0.g_alloc           8654138          .   Allocations outstanding
SMA.s0.g_bytes      167503716306          .   Bytes outstanding
SMA.s0.g_space              8238          .   Bytes available
SMA.Transient.c_req        43934     43934.00 Allocator requests
SMA.Transient.c_fail           0         0.00 Allocator failures
SMA.Transient.c_bytes    646003217 646003217.00 Bytes allocated
SMA.Transient.c_freed    646003217 646003217.00 Bytes freed
SMA.Transient.g_alloc            0          .   Allocations outstanding
SMA.Transient.g_bytes            0          .   Bytes outstanding
SMA.Transient.g_space            0          .   Bytes available
VBE.node_master(192.168.0.21,,80).vcls            1          .   VCL
references
VBE.node_master(192.168.0.21,,80).happy           0          .   Happy
health probes
VBE.node_slave(192.168.0.39,,80).vcls             1          .   VCL
references
VBE.node_slave(192.168.0.39,,80).happy            0          .   Happy
health probes
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20120907/3b98cb48/attachment-0001.html>


More information about the varnish-misc mailing list