<br /><br />Thank you Per for replying back. <br /><br /><br />Under desperately low memory conditions, the out-of-memory (OOM) killer kicks in and picks up child process of varnish that holds up the virtual memory space (in my case 524785412 kB .i.e 500 GB) and kills it dropping all objects present in the vm space. I am not able to associate the child crash that happens and swapping that place is doing everything wrong here. Can you please re-consider the concerns and have someone look on the errors that are written while breaking.<br /><br /><br /><br />Nov 19 15:06:36 cdn abrt[11775]: Saved core dump of pid 10079 (/usr/sbin/varnishd) to /var/spool/abrt/ccpp-2014-11-19-15:06:33-10079 (381931520 bytes)<br />Nov 19 15:06:36 cdn abrtd: Directory 'ccpp-2014-11-19-15:06:33-10079' creation detected<br />Nov 19 15:06:36 cdn abrtd: Package 'varnish' isn't signed with proper key<br />Nov 19 15:06:36 cdn abrtd: 'post-create' on '/var/spool/abrt/ccpp-2014-11-19-15:06:33-10079' exited with 1<br />Nov 19 15:06:36 cdn abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2014-11-19-15:06:33-10079'<br />Nov 19 15:06:36 cdn varnishd[11921]: Child (10079) not responding to CLI, killing it.<br />Nov 19 15:06:37 cdn varnishd[11921]: Child (10079) died signal=6 (core dumped)<br /><br />Nov 19 15:06:37 cdn varnishd[11921]: Child (10079) Panic message:#012Assert error in cnt_lookup(), cache/cache_req_fsm.c line 411:#012 Condition((oc->exp_flags & (1<<7)) == 0) not true.#012thread = (cache-worker)#012ident =<br />Linux,2.6.32-431.el6.x86_64,x86_64,-sfile,-smalloc,-hclassic,epoll#012Backtrace:#012 0x43b03d: /usr/sbin/varnishd() [0x43b03d]#012 0x43b34d: /usr/sbin/varnishd() [0x43b34d]#012 0x440723: /usr/sbin/varnishd() [0x440723]#012 0x44288d: /usr/sbin/varnishd(CNT_Request+0x441) [0x44288d]#012 0x4339c0: /usr/sbin/varnishd(HTTP1_Session+0x429) [0x4339c0]#012 0x4458cb: /usr/sbin/varnishd() [0x4458cb]#012 0x43dfb5: /usr/sbin/varnishd(Pool_Work_Thread+0x4cb) [0x43dfb5]#012 0x456198: /usr/sbin/varnishd() [0x456198]#012 0x4562c1: /usr/sbin/varnishd(WRK_thread+0x27) [0x4562c1]#012 0x7f4027fc49d1: /lib64/libpthread.so.0(+0x79d1) [0x7f4027fc49d1]#012req = 0x7ec30d0e8020 {#012 sp = 0x7ec30e4257e0, vxid = 1074847970, step = R_STP_LOOKUP,#012 req_body = R_BODY_NONE,#012 restarts = 0, esi_level = 0#012 sp = 0x7ec30e4257e0 {#012 fd = 75, vxid = 1106140,#012 client = 122.176.68.88 56100,#012 step = S_STP_WORKING,#012 },#012 worker = 0x7f4017709bf0 {#012 ws = 0x7f4017709e08 {#012 id = "wrk",#012 {s,f,r,e} = {0x7f40177093d0,0x7f40177093d0,(nil),+2048},#012 },#012 VCL::method = 0x0,#012 VCL::return = deliver,#012 },#012 ws = 0x7ec30d0e81b8 {#012 id = "req",#012 {s,f,r,e} = {0x7ec30d0ea010,+872,(nil),+57360},#012 },#012 http[req] = {#012 ws = 0x7ec30d0e81b8[req]#012 "GET",#012 "/thumbnail1/3a8bbd2bb61f4f3f.jpg",#012 "HTTP/1.1",#012 "Connection: keep-alive",#012 "Cache-Control: max-age=0",#012 "Accept: image/webp,*/*;q=0.8",#012 "If-Modified-Since: Mon, 08 Sep 2014 03:15:48 GMT",#012 "User-Agent: Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.111 Safari/537.36",#012 "Referer: http://money.rediff.com/",#012 "Accept-Language: en-US,en;q=0.8",#012 "X-Forwarded-For: 122.176<br /><br /><br /><br />Regards,<br />Anand<br /><br /><br /><br />From: Per Buer <perbu@varnish-software.com><br />Sent: Tue, 18 Nov 2014 23:10:50<br />To: Anand_Shah <anand@rediff-inc.com><br />Subject: Re: varnish-dev@varnish-cache.org: Swap flushes all objects in cache<br /> <div dir="ltr">Anand,<div><div class="gmail_extra"> <div class="gmail_quote">On Tue, Nov 18, 2014 at 2:01 PM, Anand Shah <span dir="ltr"><anand@rediff-inc.com></span> wrote:
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I have noticed this everytime the memory offload happens. This is not a crash or segfault; coz the service does not go down and it keeps in running which is also evident in the graphs shared earlier. I can see the same behaviour everytime when operating system swaps. Can you confirm if this is a ideal behaviour with varnish? </blockquote>
<div> </div>
<div>Please keep user questions to the -misc list.</div>
<div> </div>
<div>This is most like misconfiguration causing the OOM killer to kill of the child process. It doesn't keep running. Look at the accepted sessions in the graph. It dives when this happens.</div>
<div> </div>
<div> </div></div>--
<div class="gmail_signature"><div dir="ltr"><table border="0" cellpadding="0" cellspacing="0" style="border-bottom-width:1px;padding-top:5px;border-top-style:solid;width:550px;padding-bottom:5px;border-bottom-color:rgb(238,238,238);border-top-width:1px;border-bottom-style:solid;line-height:1.5em;border-top-color:rgb(238,238,238);color:rgb(102,102,102);font-size:12px;font-family:'Helvetica Neue',Arial,sans-serif;margin-top:20px"> <tbody> <tr> <td width="100"><img src="http://www.varnish-software.com/static/media/logo-email.png" /></td> <td><font color="#222222"><span style="font-size:14px"><b>Per Buer</b></span></font><br /> CTO | Varnish Software AS<br /> Cell: <a rel="external" style="color:rgb(17,85,204)" target="_blank" value="+4790181750">+47 95839117</a><br /> <span style="font-weight:bold">We Make Websites Fly!<br /> <a href='https://www.varnish-software.com/'>www.varnish-software.com</a></span></td> </tr> </tbody></table>
<div style="color:rgb(136,136,136)"><a href='http://info.varnish-software.com/varnish-summits-autumn-2014-registration'><img alt=" Register now" src="https://www.varnish-software.com/sites/default/files/u388/masters_s_0.png" /></a></div></div></div></div></div></div>