<div><span class="Apple-style-span" style="font-family: Monaco; font-size: 11px; line-height: 14px; "><pre style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 0px; font: normal normal normal 115%/normal Monaco, 'Courier New', monospace; line-height: 1.4em; font-family: 'Bitstream Vera Sans Mono', Courier, monospace; ">
<div class="line" id="LC1" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 1em; line-height: 1.4em; ">Hi guys,</div>
<div class="line" id="LC2" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 1em; line-height: 1.4em; "> </div><div class="line" id="LC3" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 1em; line-height: 1.4em; ">
So sometimes are backend is really slow in returning a response. So slow that it looks like it </div><div class="line" id="LC4" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 1em; line-height: 1.4em; ">
is causing varnish to timeout before the backend does and so varnish seems to eventually give up </div><div class="line" id="LC5" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 1em; line-height: 1.4em; ">
and returns a 503.</div><div class="line" id="LC6" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 1em; line-height: 1.4em; ">
</div><div class="line" id="LC7" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 1em; line-height: 1.4em; ">Ther are 4 curl requests</div>
<div class="line" id="LC8" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 1em; line-height: 1.4em; ">* the first 2 timeout and give a 503</div>
<div class="line" id="LC9" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 1em; line-height: 1.4em; ">* the 3rd one is a hit miss but gives a 200</div>
<div class="line" id="LC10" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 1em; line-height: 1.4em; ">* the 4th one is a cache hit and gives a 200</div>
<div class="line" id="LC11" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 1em; line-height: 1.4em; "> </div><div class="line" id="LC12" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 1em; line-height: 1.4em; ">
<a href="https://gist.github.com/0452c374ee21dbba138d">https://gist.github.com/0452c374ee21dbba138d</a></div><div class="line" id="LC13" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 1em; line-height: 1.4em; ">
</div><div class="line" id="LC14" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 1em; line-height: 1.4em; ">Here's the varnishlog filtered for client requests </div>
<div class="line" id="LC15" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 1em; line-height: 1.4em; "> </div><div class="line" id="LC16" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 1em; line-height: 1.4em; ">
<a href="http://gist.github.com/279254e0f2452814bf46">http://gist.github.com/279254e0f2452814bf46</a></div><div class="line" id="LC17" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 1em; line-height: 1.4em; ">
</div><div class="line" id="LC18" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 1em; line-height: 1.4em; ">Here's varnishstat</div>
<div class="line" id="LC19" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 1em; line-height: 1.4em; "> </div><div class="line" id="LC20" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 1em; line-height: 1.4em; ">
<a href="https://gist.github.com/219534d51a503b546070">https://gist.github.com/219534d51a503b546070</a></div><div class="line" id="LC21" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 1em; line-height: 1.4em; ">
</div><div class="line" id="LC22" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 1em; line-height: 1.4em; ">Here's are current startup run time parameters options right now:</div>
<div class="line" id="LC23" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 1em; line-height: 1.4em; "> </div><div class="line" id="LC24" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 1em; line-height: 1.4em; ">
<a href="https://gist.github.com/219534d51a503b546070">https://gist.github.com/219534d51a503b546070</a></div><div class="line" id="LC25" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 1em; line-height: 1.4em; ">
</div><div class="line" id="LC26" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 1em; line-height: 1.4em; ">So, Im pretty certain what is happening is that our backend takes too long and varnish times </div>
<div class="line" id="LC27" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 1em; line-height: 1.4em; ">out. How can I set the time out higher... is it sess_timeout?</div>
<div class="line" id="LC28" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 1em; line-height: 1.4em; "> </div><div class="line" id="LC29" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 1em; line-height: 1.4em; ">
Thanks guys,</div><div class="line" id="LC30" style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; padding-top: 0px; padding-right: 0px; padding-bottom: 0px; padding-left: 1em; line-height: 1.4em; ">
Tung</div></pre></span><br><div class="gmail_quote">On Wed, Mar 25, 2009 at 10:37 AM, Tung Nguyen <span dir="ltr"><<a href="mailto:tnguyen@bleacherreport.com">tnguyen@bleacherreport.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div>Kristian, thank you.</div><div><br></div><div>Im glad to hear that most defaults are good.</div><div><br></div><div>Yup, the default thread_pool_min = 1 seems kinda funny. We'll set it to at least 100. </div><div>
<br></div><div>I didnt even know about the cli_timeout and will set it to a base line of 10 seconds to test.</div><div><br></div><div>Im wondering how you are testing, Im using ab, apachebench, to see how things behave with -c 10 -n 1000, on the varnished pages.</div>
<div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div>Here's more specific questions more run time parameters. The general question I have is what to look for during testing, should I be looking at your varnishstat and are the most important things to look for in that output.</div>
<div><br></div><div><br></div><div>Our varnish stack will look like this:</div><div><br></div><div>LB -> Varnish x 2 -> Nginx x 6 -> Mongrel x 60</div><div><br></div><div>Some questions about how best to decide how to configure them best to configure the run time parameters.</div>
<div><br></div><div>-p obj_workspace=4096</div><div>Cant find obj_workspace in the man page but found it in the twitter email post <a href="http://projects.linpro.no/pipermail/varnish-dev/2009-February/000968.html" target="_blank">http://projects.linpro.no/pipermail/varnish-dev/2009-February/000968.html</a></div>
<div><br></div><div>Is obj_workspace how much space preallocated to be used for the obj that gets returned from the backend? So, if my nginx backend returns a web page that is over 4MB than -p obj_workspace is not enough, would that crash varnish, or log the error somewhere.</div>
<div><br></div><div>-p sess_workspace=262144 </div><div>Same deal here with the man page and twitter post.</div><div>What is the sess_workspace? </div><div><br></div><div>http_workspace</div><div>How does sess_workspace and obj_workspace relate to http_workspace?</div>
<div>If we use obj_workspace=4096 and sess_workspace=262144, does the default http_workspace=8192 make sense?</div><div><br></div><div><br></div><div>-p lru_interval=60 </div><div>Shows up on the twitter post again, but no man notes yet. Whats the default for this? </div>
<div><br></div><div>-p sess_timeout=10 \</div><div>Default for this is 5. If the requests from the backend takes longer than 5 seconds, what happens? Sometimes we have really slow response from the backend..</div><div>
<br>
</div><div>-p shm_workspace=32768 \</div><div>Is this the same as setting the command line flag -l shmlogsize. The default is 80MB. So dont know twitter did both setting it to less..</div><div><br></div><div>-p thread_pools=4 \</div>
<div>-p thread_pool_min=100 \</div><div>thread_pool_max</div><div>The defaults are 1,1,1000 respectively. Im wondering how best to determine this or just leave as default.</div><div><br></div><div><br></div><div><br></div>
<div><br></div><div><br></div><div>######################################################################</div><div># output of varnishstat, what is best to look at here?</div><div>######################################################################</div>
<div>0+07:11:24 Hitrate ratio: 4 4 4</div><div>Hitrate avg: nan nan nan</div>
<div><br></div><div> 400 0.00 0.02 Client connections accepted</div><div> 400 0.00 0.02 Client requests received</div><div> 1 0.00 0.00 Cache hits</div>
<div> 5 0.00 0.00 Cache misses</div><div> 399 0.00 0.02 Backend connections success</div><div> 399 0.00 0.02 Backend connections failures</div><div>
1 0.00 0.00 Backend connections reuses</div><div> 5 0.00 0.00 Backend connections recycles</div><div> 6 . . N struct srcaddr</div><div>
21 . . N struct sess_mem</div>
<div> 1 . . N struct sess</div><div> 1 . . N struct object</div><div> 1 . . N struct objecthead</div><div> 3 . . N struct smf</div>
<div> 1 . . N small free smf</div><div> 1 . . N large free smf</div><div> 1 . . N struct vbe_conn</div><div> 2 . . N struct bereq</div>
<div> 10 . . N worker threads</div><div> 23 0.00 0.00 N worker threads created</div><div> 76 0.00 0.00 N overflowed work requests</div><div>
2 . . N backends</div><div> 5 . . N expired objects</div><div> 6 0.00 0.00 Objects sent with write</div><div> 400 0.00 0.02 Total Sessions</div>
<div> 400 0.00 0.02 Total Requests</div><div> 394 0.00 0.02 Total pipe</div><div> 5 0.00 0.00 Total fetch</div><div> 352131 0.00 13.60 Total header bytes</div>
<div> 1745 0.00 0.07 Total body bytes</div><div> 397 0.00 0.02 Session Closed</div><div> 3 0.00 0.00 Session herd</div><div> 33185 2.00 1.28 SHM records</div>
<div> 19306 2.00 0.75 SHM writes</div><div> 32 0.00 0.00 SHM MTX contention</div><div> 11 0.00 0.00 allocator requests</div><div> 1 . . outstanding allocations</div>
<div> 8192 . . bytes allocated</div><div> 6612459520 . . bytes free</div><div> 5 0.00 0.00 Backend requests made</div><div> 1 0.00 0.00 N vcl total</div>
<div> 1 0.00 0.00 N vcl available</div><div> 1 . . N total active purges</div><div> 1 0.00 0.00 N new purges added</div><div><br></div><div>
<br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div>Any response is appreciated.</div><div><br></div><div>Thanks,</div><div>Tung</div><div><div></div><div class="h5"><div><br></div><div>
<br></div><div><br></div><div>
<br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><br><div class="gmail_quote">On Tue, Mar 24, 2009 at 12:20 AM, Kristian Lyngstol <span dir="ltr"><<a href="mailto:kristian@redpill-linpro.com" target="_blank">kristian@redpill-linpro.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>On Mon, Mar 23, 2009 at 05:58:58PM -0700, Tung Nguyen wrote:<br>
> Hi guys,<br>
> So, Im reading over an archive email thread about twitters configuration.<br>
><br>
> <a href="http://projects.linpro.no/pipermail/varnish-dev/2009-February/000968.html" target="_blank">http://projects.linpro.no/pipermail/varnish-dev/2009-February/000968.html</a><br>
><br>
> It looks like they had to adjust a lot of parameters... and Im not finding<br>
> all the parameters definitions in the varnishd man pages. Im wondering if<br>
> for most cases running varnish with the defaults is fine?<br>
><br>
> Any caveats here which run time parameters should I focus on.<br>
<br>
</div>You can mostly run it with the defaults, yes. This depends on what sort of<br>
usage you have though.<br>
<br>
One notable exception is that I strongly recommend that you bring<br>
thread_pool_min up to a decent level (reflecting how many users you<br>
actually have. Numbers in the hundreds is normal). You'll also want to<br>
adjust the cache size to your system, but that goes without saying.<br>
<br>
If you expect extremely high load, you might have to increase cli_timeout<br>
too. I've run tests where even setting it to 15 seconds is insufficient and<br>
causes childs to be killed off. Though for most production sites, I'd guess<br>
5 seconds could work and 10 seconds would definitely work.<br>
<font color="#888888"><br>
--<br>
Kristian Lyngstøl<br>
Redpill Linpro AS<br>
Tlf: +47 21544179<br>
Mob: +47 99014497<br>
</font></blockquote></div><br><br clear="all"><br></div></div><font color="#888888">-- <br>Tung Nguyen, Lead Developer<br>Bleacher Report, The Open Source Sports Network <br>(510) 928-0475<br>
</font></blockquote></div><br><br clear="all"><br>-- <br>Tung Nguyen, Lead Developer<br>Bleacher Report, The Open Source Sports Network <br>(510) 928-0475<br>
</div>