<!DOCTYPE html PUBLIC '-//W3C//DTD HTML 4.01 Transitional//EN'>
<html><head><meta http-equiv="Content-Type" content="text/html;charset=utf-8">
<style>BODY{font:10pt Tahoma,Verdana,sans-serif} .MsoNormal{line-height:120%;margin:0}</style></head><body>
Hi Rubén,<div><br></div><div>Apparently we made a mistake a while ago by not confirming the varnish 3.0 defaults compared to 2.1. So we were happily still running with a sess_workspace of 16k, instead of 64k.</div><div>This worked fine with our previous config, but since we added our sharding vmod, 16k is not enough anymore.</div><div><br></div><div>So, the fixed config with 64k as sess_workspace has been running on one of our varnish servers for almost a week now.</div><div>And it's been working fine. Even when a Justin Bieber link went viral yesterday, which put our systems to the real test.</div><div>Not to mention this happened at the same time it's already peak-hour.</div><div><br></div><div>And Varnish (of course;) happily served thousands of requests per second.</div><div><br></div><div>So yes, I can now confirm: increasing sess_workspace fixed the assertion errors in my case.</div><div><br></div><div>regards,</div><div><br></div><div>Enno van Amerongen</div><div><br><blockquote style="padding-left: 5px; margin-left: 5px; border-left: #0000ff 2px solid; margin-right: 0px"><hr><b>From:</b> Rubén Romero [mailto:ruben@varnish-software.com]<br><b>To:</b> Enno van Amerongen [mailto:enno@tty.nl]<br><b>Sent:</b> Sun, 14 Apr 2013 12:52:48 +0200<br><b>Subject:</b> Re: increase sess_workspace to fix assert error?<br><br><p>Hi Enno,</p>
<p>Increasing it will likely help. Not sure to how much, but do give it as much as you can and let us know how it went :-)<br></p>
<p>Best,<br>
-<br>
Rubén Romero<br>
Varnish Software<br>
</p>
<div class="gmail_quote">Den 2. apr. 2013 21.18 skrev "Enno van Amerongen" <<a href="mailto:enno@tty.nl">enno@tty.nl</a>> følgende:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<u></u>
<div>
<div><font face="Tahoma, Verdana, sans-serif">Dear List,</font></div><div><font face="Tahoma, Verdana, sans-serif"><br></font></div><div><font face="Tahoma, Verdana, sans-serif">After deployment of a vmod we've written (that enables us to select a director by id in url), we've been experiencing assert errors.</font></div>
<div><font face="Tahoma, Verdana, sans-serif"><br></font></div><div><font face="Tahoma, Verdana, sans-serif">Below are the stack traces, but to not waste your time, here's what I think it boils down to:</font></div><div>
<font face="Tahoma, Verdana, sans-serif"><br></font></div><div><font face="Tahoma, Verdana, sans-serif">{ overflow#012 id = "sess",#012 {s,f,r,e} = {0x7fed76034c78,+16384,(nil),+16384},#012 }</font></div>
<div><font face="Tahoma, Verdana, sans-serif"><br></font></div><div><font face="Tahoma, Verdana, sans-serif">4k as session workspace has worked fine for us for years, but our vmod sharding adds hundreds of directors (as opposed to 4), while the number of backends stayed the same (about 25).</font></div>
<div><font face="Tahoma, Verdana, sans-serif"><br></font></div><div><font face="Tahoma, Verdana, sans-serif">So, my question is, do I just need to increase sess_workspace to fix this? If not, what else ;)</font></div><div>
<font face="Tahoma, Verdana, sans-serif"><br></font></div><div><font face="Tahoma, Verdana, sans-serif">Regards,</font></div><div><font face="Tahoma, Verdana, sans-serif"><br></font></div><div><font face="Tahoma, Verdana, sans-serif">Enno van Amerongen</font></div>
<div><font face="Tahoma, Verdana, sans-serif"><br></font></div><div><br></div><div><font face="Tahoma, Verdana, sans-serif"><br></font></div><div><font face="Tahoma, Verdana, sans-serif"><br></font></div><div><font face="Tahoma, Verdana, sans-serif">Both stacktraces</font></div>
<div><font face="Tahoma, Verdana, sans-serif"><br></font></div><div><font face="Tahoma, Verdana, sans-serif">Apr 2 18:43:32 obfuscated /var/www/varnish[12750]: Child (20498) Panic message: Assert error in vh_get_var_alloc(), vmod_var.c line 77:#012 Condition((v) != 0) not true.#012errno = 32 (Broken pipe)#012thread = (cache-worker)#012ident = Linux,2.6.32-5-amd64,x86_64,-smalloc,-smalloc,-hcritbit,epoll#012Backtrace:#012 0x439199: pan_backtrace+19#012 0x43946e: pan_ic+1ad#012 0x7fe137efe347: _end+7fe13786898f#012 0x7fe137efe862: _end+7fe137868eaa#012 0x7fecc5b93e73: _end+7fecc54fe4bb#012 0x7fecc5b939ea: _end+7fecc54fe032#012 0x7fecc5ba24f1: _end+7fecc550cb39#012 0x4417bd: VCL_recv_method+6d#012 0x41d8f5: cnt_recv+262#012 0x41e68d: CNT_Session+5cb#012sp = 0x7fed81293008 {#012 fd = 2849, id = 2849, xid = 918440385,#012 client = 10.1.1.1 51854,#012 step = STP_RECV,#012 handling = deliver,#012 err_code = 200, err_reason = (null),#012 restarts = 0, esi_level = 0#012 flags = #012 bodystatus = 4#012 ws = 0x7fed81293080 { overflow#012 id = "sess",#012 {s,f,r,e} = {0x7fed81293c78,+16384,(nil),+16384},#012 },#012 http[req] = {#012 ws = 0x7fed81293080[sess]#012 "GET",#012 "/f/146057654_2.jpg",#012 "HTTP/1.1",#012 "User-Agent: Opera/9.80 (Windows NT 6.1; WOW64) Presto/2.12.388 Version/12.14",#012 "Host: <a href="http://img.obfuscated.com">img.obfuscated.com</a>",#012 "Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/webp, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1",#012 "Accept-Language: fr-FR,fr;q=0.9,en;q=0.8",#012 "Referer: <a href="http://obfuscated">http://obfuscated</a>",#012 "Connection: Keep-Alive",#012 },#012 worker = 0x7feece7efaa0 {#012 ws = 0x7feece7efcd8 { #012 id = "wrk",#012 {s,f,r,e} = {0x7feece7dd9f0,+16,(nil),+65536},#012 },#012 },#012 vcl = {#012 srcname = {#012 "input",#012 "Default",#012 "/obfuscated/img_backends.vcl",#012 "/obfuscated/img_directors.vcl",#012</font></div>
<div><font face="Tahoma, Verdana, sans-serif"><br></font></div><div><font face="Tahoma, Verdana, sans-serif">So we removed the vmod_var from our configuration as it wasn't really required anyway.</font></div><div><font face="Tahoma, Verdana, sans-serif"><br>
</font></div><div><font face="Tahoma, Verdana, sans-serif">Then 13 minutes later, Varnish crashed again, now with a different, but similar assert error:</font></div><div><font face="Tahoma, Verdana, sans-serif"><br></font></div>
<div><font face="Tahoma, Verdana, sans-serif">Apr 2 19:07:22 obfuscated /var/www/varnish[12750]: Child (18818) Panic message: Assert error in VRT_int_string(), cache_vrt.c line 324:#012 Condition((p = WS_Alloc(sp->http->ws, size)) != 0) not true.#012thread = (cache-worker)#012ident = Linux,2.6.32-5-amd64,x86_64,-smalloc,-smalloc,-hcritbit,epoll#012Backtrace:#012 0x439199: pan_backtrace+19#012 0x43946e: pan_ic+1ad#012 0x4435f7: VRT_int_string+88#012 0x7fe686b9ce1f: _end+7fe686507467#012 0x7fe686b9c99a: _end+7fe686506fe2#012 0x7fe686ba7424: _end+7fe686511a6c#012 0x4417bd: VCL_recv_method+6d#012 0x41d8f5: cnt_recv+262#012 0x41e68d: CNT_Session+5cb#012 0x43aa0a: wrk_thread_real+8cc#012sp = 0x7fed76034008 {#012 fd = 2357, id = 2357, xid = 2009537706,#012 client = 80.239.243.196 59852,#012 step = STP_RECV,#012 handling = deliver,#012 err_code = 200, err_reason = (null),#012 restarts = 0, esi_level = 0#012 flags = #012 bodystatus = 4#012 ws = 0x7fed76034080 { overflow#012 id = "sess",#012 {s,f,r,e} = {0x7fed76034c78,+16384,(nil),+16384},#012 },#012 http[req] = {#012 ws = 0x7fed76034080[sess]#012 "GET",#012 "/f/148682988.jpg",#012 "HTTP/1.1",#012 "User-Agent: Opera/9.80 (Android; Opera Mini/7.5.32193/29.3134; U; nl) Presto/2.8.119 Version/11.10",#012 "Host: <a href="http://img.obfuscated.com">img.obfuscated.com</a>",#012 "Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/webp, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1",#012 "Accept-Language: nl,en;q=0.9",#012 "Referer: obfuscated",#012 "Connection: Keep-Alive",#012 "X-OperaMini-Phone: Android #",#012 "X-OperaMini-Phone-UA: Mozilla/5.0 (Linux; U; Android 2.2.2; nl-be; Stockholm Build/HuaweiU8180) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",#012 "x-forwarded-for: obfuscated",#012 "Device-Stock-UA: Mozilla/5.0 (Linux; U; Android 2.2.2; nl-be; Stockholm Build/HuaweiU8180) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",#012 "X</font></div>
<div style="font-family:Tahoma,Verdana,sans-serif;font-size:10pt;font-style:normal;font-variant:normal;font-weight:normal;line-height:normal"><br></div></div><br>_______________________________________________<br>
varnish-misc mailing list<br>
<a href="mailto:varnish-misc@varnish-cache.org">varnish-misc@varnish-cache.org</a><br>
<a href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc">https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc</a><br></blockquote></div>
</blockquote><style>
</style>
</div></body></html>