<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>#0. We got varnish 3.0.2 running on box 2.6.32-220.4.1.el6.x86_64, with streaming on.</div><div><br></div><div>#1. the process</div><div><div>top - 22:19:21 up 14 days, 22:28, 2 users, load average: 10.86, 10.87, 19.51</div><div>Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie</div><div>Cpu(s): 64.9%us, 0.5%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.2%si, 34.5%st</div><div>Mem: 7646264k total, 6948476k used, 697788k free, 109384k buffers</div><div>Swap: 4194296k total, 7636k used, 4186660k free, 3502984k cached</div><div><br></div><div> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND </div><div>24551 nobody 20 0 4241m 40m 18m S <font class="Apple-style-span" color="#b6170f">143.2</font> 0.5 50:07.07 varnishd </div></div><div><br></div><div><br></div><div>#2. the threads</div><div><br></div><div><div>top - 22:19:35 up 14 days, 22:29, 2 users, load average: 11.50, 11.02, 19.42</div><div>Tasks: 637 total, 11 running, 626 sleeping, 0 stopped, 0 zombie</div><div>Cpu(s): 4.5%us, 0.8%sy, 0.0%ni, 92.3%id, 0.3%wa, 0.0%hi, 0.3%si, 1.9%st</div><div>Mem: 7646264k total, 6948856k used, 697408k free, 109396k buffers</div><div>Swap: 4194296k total, 7636k used, 4186660k free, 3503000k cached</div><div><br></div><div> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND </div><div>24757 nobody 20 0 4241m 40m 18m R 22.9 0.5 8:10.68 varnishd </div><div>24759 nobody 20 0 4241m 40m 18m R 21.5 0.5 7:59.45 varnishd </div><div>24958 nobody 20 0 4241m 40m 18m R 21.5 0.5 8:59.62 varnishd </div><div>24761 nobody 20 0 4241m 40m 18m R 16.2 0.5 5:26.20 varnishd </div><div>24954 nobody 20 0 4241m 40m 18m R 16.2 0.5 7:46.43 varnishd </div><div>24952 nobody 20 0 4241m 40m 18m R 14.8 0.5 4:20.41 varnishd </div><div>24960 nobody 20 0 4241m 40m 18m R 14.8 0.5 9:16.54 varnishd </div></div><div><br></div><div>#3. the stacktraces</div><div><br></div><div><div>Thread 206 (Thread 0x7f5047012700 (LWP 24757)):</div><div>#0 0x000000360df3e606 in __memcpy_ssse3_back () from /lib64/libc.so.6</div><div>#1 0x0000003610206ac7 in ?? () from /usr/lib64/varnish/libvgz.so</div><div>#2 0x00000036102072c6 in inflate () from /usr/lib64/varnish/libvgz.so</div><div>#3 0x00000000004233aa in VGZ_Gunzip ()</div><div>#4 0x0000000000423ce1 in VGZ_WrwGunzip ()</div><div>#5 0x000000000042f702 in RES_StreamPoll ()</div><div>#6 0x0000000000423607 in ?? ()</div><div>#7 0x0000000000421e1a in FetchBody ()</div><div>#8 0x0000000000416ffd in ?? ()</div><div>#9 0x000000000041785c in CNT_Session ()</div><div>#10 0x000000000042f508 in ?? ()</div><div>#11 0x000000000042e6eb in ?? ()</div><div>#12 0x000000360e2077f1 in start_thread () from /lib64/libpthread.so.0</div><div>#13 0x000000360dee592d in clone () from /lib64/libc.so.6</div></div><div><br></div><div><div>Thread 204 (Thread 0x7f5045c10700 (LWP 24759)):</div><div>#0 0x0000003610206cb4 in inflate () from /usr/lib64/varnish/libvgz.so</div><div>#1 0x00000000004233aa in VGZ_Gunzip ()</div><div>#2 0x0000000000423ce1 in VGZ_WrwGunzip ()</div><div>#3 0x000000000042f702 in RES_StreamPoll ()</div><div>#4 0x0000000000423607 in ?? ()</div><div>#5 0x0000000000421e1a in FetchBody ()</div><div>#6 0x0000000000416ffd in ?? ()</div><div>#7 0x000000000041785c in CNT_Session ()</div><div>#8 0x000000000042f508 in ?? ()</div><div>#9 0x000000000042e6eb in ?? ()</div><div>#10 0x000000360e2077f1 in start_thread () from /lib64/libpthread.so.0</div><div>#11 0x000000360dee592d in clone () from /lib64/libc.so.6</div></div><div><br></div><div>Most of the threads eating lots of CPUs were all with the similar stacktraces, and never finished...</div><div><br></div><div>I doubt there must be a infinite-loop or something similar. </div><div><br></div><div>@list, any suggestions?</div><div><br></div><div>Thanks</div><br><div apple-content-edited="true">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><div>----------------------------------</div><div>Best wishes,</div><div>Jammy</div></div><div><br></div><div><br></div></div><br class="Apple-interchange-newline"></div><br class="Apple-interchange-newline"><br class="Apple-interchange-newline">
</div>
<br></body></html>