<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Something's very wrong here - we've never experienced this before. <div><br></div><div>Are you stating the server as root or as another user? Any ulimit or restrictions on # of file descriptors?</div><div><br></div><div>-j</div><div><br><div><div>On Apr 10, 2009, at 3:58 PM, Ray Barnes wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>John,</div> <div> </div> <div>Thanks for the reply; as you can see my config is largely based on the one you posted to this list in February (thanks!).</div> <div> </div> <div>I went back as you suggested and waited 90 seconds, while starting it the same way. Before running any tests, I went into the CLI and viewed stats on the threads:</div> <div> </div> <div> 364 N worker threads<br> 364 N worker threads created<br> 782 N worker threads not created<br></div> <div> </div> <div>When this happens (started threads do not match the number specified), varnish does really unpredictable things, i.e. it won't take 300 connections from 'ab' and times out with the following message:</div> <div> </div> <div>Benchmarking 98.124.141.3 (be patient)<br>apr_poll: The timeout specified has expired (70007)<br>Total of 52 requests completed<br></div> <div> </div> <div>I think the crux of my problem is figuring out why it won't start more threads. Being not-so-familiar with the internals of varnish, I can't tell whether that's an OS problem or a varnish problem. Hope that helps.</div> <div> </div> <div>-Ray</div> <div><br><br> </div> <div class="gmail_quote">On Fri, Apr 10, 2009 at 6:35 PM, John Adams <span dir="ltr"><<a href="mailto:jna@twitter.com">jna@twitter.com</a>></span> wrote:<br> <blockquote style="BORDER-LEFT: #ccc 1px solid; MARGIN: 0px 0px 0px 0.8ex; PADDING-LEFT: 1ex" class="gmail_quote"> <div style="WORD-WRAP: break-word">It takes time to spawn threads. If you start the server with hundreds of threads, they won't be ready for ~30-90 seconds. <div><br></div> <div>Maybe that's causing this issue?</div> <div><br></div> <div>-j</div> <div><br> <div> <div> <div></div> <div class="h5"> <div>On Apr 10, 2009, at 3:12 PM, Ray Barnes wrote:</div><br></div></div> <blockquote type="cite"> <div> <div></div> <div class="h5"> <div>Hi all. Note that everything herein is based only on a very lay knowledge of varnish, without being familiar with the internals of the code.</div> <div> </div> <div>In my quest to eek more performance out of Varnish, I've been testing under 2.0.4. I have not seen much improvement over 2.0.3 in the way it acts after receiving a bunch of hits all at one time. I am invoking varnish like this:</div> <div> </div> <div>ulimit -n 131072<br>ulimit -l 82000<br>/usr/local/sbin/varnishd -a <a href="http://98.124.141.3/" target="_blank">98.124.141.3:80</a> -b <a href="http://67.212.179.98/" target="_blank">67.212.179.98:80</a> -T <a href="http://98.124.141.3:6083/" target="_blank">98.124.141.3:6083</a> \<br> -t 60 -w1440,3000,60 -u apache -g apache -p obj_workspace=16000 -p sess_workspace=262144 -p listen_depth=4096 \<br> -p shm_workspace=64000 -p thread_pools=8 -p thread_pool_min=180 -p ping_interval=1 -p srcaddr_ttl=0 -s malloc,80M<br> </div> <div>As best I can tell, the problem I'm seeing is that it will not create the number of worker threads that I'm telling it to, as evidenced by the 'status' output within the CLI immediately after launch:</div> <div> </div> <div> 270 N worker threads<br> 285 N worker threads created<br></div> <div>So if I launch 'ab' with 700 connections against varnish, it will not work right from the beginning, like so:</div> <div> </div> <div>[root@mia ~]# ab -n 20000 -c 700 <a href="http://98.124.141.3/" target="_blank">http://98.124.141.3/</a><br>This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0<br>Copyright 1996 Adam Twiss, Zeus Technology Ltd, <a href="http://www.zeustech.net/" target="_blank">http://www.zeustech.net/</a><br> Copyright 2006 The Apache Software Foundation, <a href="http://www.apache.org/" target="_blank">http://www.apache.org/</a></div> <div>Benchmarking 98.124.141.3 (be patient)<br>apr_socket_recv: Connection refused (111)<br>[root@mia ~]# ab -n 20000 -c 700 <a href="http://98.124.141.3/" target="_blank">http://98.124.141.3/</a><br>This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0<br> Copyright 1996 Adam Twiss, Zeus Technology Ltd, <a href="http://www.zeustech.net/" target="_blank">http://www.zeustech.net/</a><br>Copyright 2006 The Apache Software Foundation, <a href="http://www.apache.org/" target="_blank">http://www.apache.org/</a></div> <div>Benchmarking 98.124.141.3 (be patient)<br>apr_poll: The timeout specified has expired (70007)<br>Total of 147 requests completed<br>[root@mia ~]# telnet 98.124.141.3 80<br>Trying 98.124.141.3...<br>Connected to 98.124.141.3 (98.124.141.3).<br> Escape character is '^]'.<br>GET / HTTP/1.0</div> <div>^]<br>telnet> quit<br>Connection closed.<br></div> <div>The above telnet command simply hung, presumably because there are still 700 sessions in CLOSE_WAIT state within the kernel, although that should not matter if varnish opened the number of worker threads it was supposed to. Based on what I've seen, it would seem that varnish has some problem when you launch it with "too many" initial worker threads (although I'm having a hard time understanding why 1400ish is too many). It seems to go crazy if you specify too many threads initially. Again, that number should not be a problem for the machine in theory, as it's a multicore Xeon. Platform is Linux 2.6 RHEL. Any idea what's happening here?</div> <div> </div> <div>-Ray</div> <div> </div></div></div>_______________________________________________<br>varnish-dev mailing list<br><a href="mailto:varnish-dev@projects.linpro.no" target="_blank">varnish-dev@projects.linpro.no</a><br><a href="http://projects.linpro.no/mailman/listinfo/varnish-dev" target="_blank">http://projects.linpro.no/mailman/listinfo/varnish-dev</a><br> </blockquote></div><br> <div><span style="TEXT-TRANSFORM: none; TEXT-INDENT: 0px; BORDER-COLLAPSE: separate; FONT: 12px Helvetica; WHITE-SPACE: normal; LETTER-SPACING: normal; COLOR: rgb(0,0,0); WORD-SPACING: 0px"> <div style="WORD-WRAP: break-word"><span style="TEXT-TRANSFORM: none; TEXT-INDENT: 0px; BORDER-COLLAPSE: separate; FONT: 12px Helvetica; WHITE-SPACE: normal; LETTER-SPACING: normal; COLOR: rgb(0,0,0); WORD-SPACING: 0px"> <div style="WORD-WRAP: break-word"> <div>---</div> <div>John Adams</div> <div>Twitter Operations</div> <div><a href="mailto:jna@twitter.com" target="_blank">jna@twitter.com</a></div> <div><a href="http://twitter.com/netik" target="_blank">http://twitter.com/netik</a></div> <div><br></div></div></span><br></div></span><br></div><br></div></div></blockquote></div><br></blockquote></div><br><div apple-content-edited="true"> <span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>---</div><div>John Adams</div><div>Twitter Operations</div><div><a href="mailto:jna@twitter.com">jna@twitter.com</a></div><div><a href="http://twitter.com/netik">http://twitter.com/netik</a></div><div><br></div></div></span><br class="Apple-interchange-newline"></div></span><br class="Apple-interchange-newline"> </div><br></div></body></html>