<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><br></div><div>We're using varnish 3.0.3 in production, yesterday, we're riding on a heavy DDoS attack. And two of our varnish nodes crashed.</div><div><br></div><div>it's very similar to <a href="https://www.varnish-cache.org/trac/ticket/516">https://www.varnish-cache.org/trac/ticket/516</a>. Following is our stack traces.</div><div><br></div><div>Any ideas? Thanks</div><div><br></div><div><br></div><div><div>(gdb) info threads</div><div> 999 Thread 0x7ff5eb7ff700 (LWP 3100) 0x00000031d000e54d in read () from /lib64/libpthread.so.0</div><div> 998 Thread 0x7ff5ed9fd700 (LWP 3097) 0x00000031d000e54d in read () from /lib64/libpthread.so.0</div></div><div>... ...</div><div><div> 5 Thread 0x7ff88a8ff700 (LWP 30905) 0x00000031d000e54d in read () from /lib64/libpthread.so.0</div><div> 4 Thread 0x7ff88cbfd700 (LWP 30903) 0x00000031d000ed2d in nanosleep () from /lib64/libpthread.so.0</div><div> 3 Thread 0x7ff88dfff700 (LWP 30901) 0x00000031d000ed2d in nanosleep () from /lib64/libpthread.so.0</div><div> 2 Thread 0x7ff7c81f2700 (LWP 31211) 0x00000031d000e054 in __lll_lock_wait () from /lib64/libpthread.so.0</div><div>* 1 Thread 0x7ff894cca780 (LWP 30898) 0x00000031d000e054 in __lll_lock_wait () from /lib64/libpthread.so.0</div></div><div><br></div><div><br></div><div><span style="color: rgb(0, 0, 0); font-family: Arial, FreeSans, Helvetica, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17px; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(240, 240, 240); display: inline !important; float: none; ">Program terminated with signal 3, Quit. </span><br style="color: rgb(0, 0, 0); font-family: Arial, FreeSans, Helvetica, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17px; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(240, 240, 240); "><span style="color: rgb(0, 0, 0); font-family: Arial, FreeSans, Helvetica, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17px; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(240, 240, 240); display: inline !important; float: none; ">#0 0x00000031d000e054 in __lll_lock_wait () from /lib64/libpthread.so.0 </span><br style="color: rgb(0, 0, 0); font-family: Arial, FreeSans, Helvetica, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17px; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(240, 240, 240); "><span style="color: rgb(0, 0, 0); font-family: Arial, FreeSans, Helvetica, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17px; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(240, 240, 240); display: inline !important; float: none; ">Missing separate debuginfos, use: debuginfo-install varnish-3.0.3-1.201301210212.production.x86_64 </span><br style="color: rgb(0, 0, 0); font-family: Arial, FreeSans, Helvetica, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17px; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(240, 240, 240); "><span style="color: rgb(0, 0, 0); font-family: Arial, FreeSans, Helvetica, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17px; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(240, 240, 240); display: inline !important; float: none; ">(gdb) bt </span><br style="color: rgb(0, 0, 0); font-family: Arial, FreeSans, Helvetica, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17px; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(240, 240, 240); "><span style="color: rgb(0, 0, 0); font-family: Arial, FreeSans, Helvetica, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17px; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(240, 240, 240); display: inline !important; float: none; ">#0 0x00000031d000e054 in __lll_lock_wait () from /lib64/libpthread.so.0 </span><br style="color: rgb(0, 0, 0); font-family: Arial, FreeSans, Helvetica, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17px; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(240, 240, 240); "><span style="color: rgb(0, 0, 0); font-family: Arial, FreeSans, Helvetica, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17px; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(240, 240, 240); display: inline !important; float: none; ">#1 0x00000031d0009388 in _L_lock_854 () from /lib64/libpthread.so.0 </span><br style="color: rgb(0, 0, 0); font-family: Arial, FreeSans, Helvetica, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17px; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(240, 240, 240); "><span style="color: rgb(0, 0, 0); font-family: Arial, FreeSans, Helvetica, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17px; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(240, 240, 240); display: inline !important; float: none; ">#2 0x00000031d0009257 in pthread_mutex_lock () from /lib64/libpthread.so.0 </span><br style="color: rgb(0, 0, 0); font-family: Arial, FreeSans, Helvetica, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17px; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(240, 240, 240); "><span style="color: rgb(0, 0, 0); font-family: Arial, FreeSans, Helvetica, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17px; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(240, 240, 240); display: inline !important; float: none; ">#3 0x0000000000433b90 in vsl_get () </span><br style="color: rgb(0, 0, 0); font-family: Arial, FreeSans, Helvetica, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17px; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(240, 240, 240); "><span style="color: rgb(0, 0, 0); font-family: Arial, FreeSans, Helvetica, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17px; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(240, 240, 240); display: inline !important; float: none; ">#4 0x0000000000433d48 in VSLR () </span><br style="color: rgb(0, 0, 0); font-family: Arial, FreeSans, Helvetica, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17px; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(240, 240, 240); "><span style="color: rgb(0, 0, 0); font-family: Arial, FreeSans, Helvetica, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17px; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(240, 240, 240); display: inline !important; float: none; ">#5 0x0000000000433f12 in VSL () </span><br style="color: rgb(0, 0, 0); font-family: Arial, FreeSans, Helvetica, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17px; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(240, 240, 240); "><span style="color: rgb(0, 0, 0); font-family: Arial, FreeSans, Helvetica, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17px; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(240, 240, 240); display: inline !important; float: none; ">#6 0x000000000041a18e in cli_cb_before () </span><br style="color: rgb(0, 0, 0); font-family: Arial, FreeSans, Helvetica, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17px; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(240, 240, 240); "><span style="color: rgb(0, 0, 0); font-family: Arial, FreeSans, Helvetica, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17px; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(240, 240, 240); display: inline !important; float: none; ">#7 0x00007ff89559be05 in cls_vlu2 (priv=0x7ff894b41540, av=0x7ff76d5b4080) at cli_serve.c:260 </span><br style="color: rgb(0, 0, 0); font-family: Arial, FreeSans, Helvetica, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17px; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(240, 240, 240); "><span style="color: rgb(0, 0, 0); font-family: Arial, FreeSans, Helvetica, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17px; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(240, 240, 240); display: inline !important; float: none; ">#8 0x00007ff89559c48b in cls_vlu (priv=0x7ff894b41540, p=0x2 <Address 0x2 out of bounds>) at cli_serve.c:339 </span><br style="color: rgb(0, 0, 0); font-family: Arial, FreeSans, Helvetica, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17px; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(240, 240, 240); "><span style="color: rgb(0, 0, 0); font-family: Arial, FreeSans, Helvetica, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17px; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(240, 240, 240); display: inline !important; float: none; ">#9 0x00007ff89559fe19 in LineUpProcess (l=0x7ff894b33580) at vlu.c:154 </span><br style="color: rgb(0, 0, 0); font-family: Arial, FreeSans, Helvetica, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17px; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(240, 240, 240); "><span style="color: rgb(0, 0, 0); font-family: Arial, FreeSans, Helvetica, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17px; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(240, 240, 240); display: inline !important; float: none; ">#10 0x00007ff89559ce8d in VCLS_Poll (cs=0x7ff894b03240, timeout=<value optimized out>) at cli_serve.c:528 </span><br style="color: rgb(0, 0, 0); font-family: Arial, FreeSans, Helvetica, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17px; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(240, 240, 240); "><span style="color: rgb(0, 0, 0); font-family: Arial, FreeSans, Helvetica, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17px; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(240, 240, 240); display: inline !important; float: none; ">#11 0x000000000041a291 in CLI_Run () </span><br style="color: rgb(0, 0, 0); font-family: Arial, FreeSans, Helvetica, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17px; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(240, 240, 240); "><span style="color: rgb(0, 0, 0); font-family: Arial, FreeSans, Helvetica, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17px; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(240, 240, 240); display: inline !important; float: none; ">#12 0x000000000042e24c in child_main () </span><br style="color: rgb(0, 0, 0); font-family: Arial, FreeSans, Helvetica, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17px; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(240, 240, 240); "><span style="color: rgb(0, 0, 0); font-family: Arial, FreeSans, Helvetica, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17px; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(240, 240, 240); display: inline !important; float: none; ">#13 0x0000000000440d2c in start_child () </span><br style="color: rgb(0, 0, 0); font-family: Arial, FreeSans, Helvetica, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17px; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(240, 240, 240); "><span style="color: rgb(0, 0, 0); font-family: Arial, FreeSans, Helvetica, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17px; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(240, 240, 240); display: inline !important; float: none; ">#14 0x00000000004416b8 in MGT_Run () </span><br style="color: rgb(0, 0, 0); font-family: Arial, FreeSans, Helvetica, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17px; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(240, 240, 240); "><span style="color: rgb(0, 0, 0); font-family: Arial, FreeSans, Helvetica, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17px; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(240, 240, 240); display: inline !important; float: none; ">#15 0x000000000044fb4f in main ()</span></div><div><span style="color: rgb(0, 0, 0); font-family: Arial, FreeSans, Helvetica, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17px; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(240, 240, 240); display: inline !important; float: none; "><br></span></div><div><span style="color: rgb(0, 0, 0); font-family: Arial, FreeSans, Helvetica, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 17px; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(240, 240, 240); display: inline !important; float: none; "><br></span></div><br><div apple-content-edited="true">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><div>----------------------------------</div><div>Best wishes,</div><div>Jammy</div></div><div><br></div><div><br></div></div><br class="Apple-interchange-newline"></div><br class="Apple-interchange-newline"><br class="Apple-interchange-newline">
</div>
<br></body></html>