<font class="Apple-style-span" face="georgia, serif">These sysctl values helped a lot for me in a 64-bit / 64GB / 1.5TB SSD / "-s file" configuration, pretty much stabilizing the box.</font><div><font class="Apple-style-span" face="georgia, serif"><br>
</font></div><div><font class="Apple-style-span" face="georgia, serif"><div>vm.swappiness = 10</div><div>vm.min_free_kbytes = 131072</div><div><br></div><div>And don't forget to reduce thread_pool_stack to 256KB-512KB.</div>
<div><br></div><div>FWIW,</div><div>-- </div><div>kb</div></font>
<br><br><div class="gmail_quote">On Fri, Oct 29, 2010 at 14:00, David Birdsong <span dir="ltr"><<a href="mailto:david.birdsong@gmail.com">david.birdsong@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
I attempted to run an -s file instance with 5 80GB SSD's raided<br>
together. I could never get past filling it half full before the IO<br>
load would consume the box.<br>
<br>
On Fri, Oct 29, 2010 at 9:19 AM, Taylan Develioglu<br>
<<a href="mailto:tdevelioglu@ebuddy.com">tdevelioglu@ebuddy.com</a>> wrote:<br>
> More of a general question:<br>
><br>
> Is there anyone that uses varnish with a storage size of 800GB+ (single<br>
> instance, file or malloc) ?<br>
><br>
> I'm just wondering because we've had so much problems with varnish<br>
> (2.1.2 to 2.1.4) that I'm starting to think that we're an edge case.<br>
><br>
> On Fri, 2010-10-29 at 14:26 +0200, Per Buer wrote:<br>
>> Hi.<br>
>><br>
>> The issues are actually a bit different. One uses "-s file" which<br>
>> really makes the kernel handle the memory management implicitly in<br>
>> comparison to you Rodrigo, which uses -s malloc, where you clearly<br>
>> state that Varnish should allocate this much memory.<br>
>><br>
>> Regarding -s malloc we recommend that on Linux one should not use more<br>
>> then 75% of the physical memory for -s malloc. In addition to the<br>
>> actual object store Varnish uses quite some memory for it's internal<br>
>> data structures. If you go above 75% chances are that Linux will start<br>
>> swapping a bit which might have a negative effect on overall<br>
>> performance. Tollef is getting back to you outside the list regarding<br>
>> the issue you've raised with us.<br>
>><br>
>> Per.<br>
>><br>
>> On Fri, Oct 29, 2010 at 1:59 PM, Rodrigo Benzaquen<br>
>> <<a href="mailto:rodrigo@mercadolibre.com">rodrigo@mercadolibre.com</a>> wrote:<br>
>> > Hi Taylan, exactly the same is happening to us, we already open a ticket<br>
>> > with Varnish software.<br>
>> ><br>
>> > Please let me know if you find a solution.<br>
>> ><br>
>> > Thanks<br>
>> > Rodrigo<br>
>> ><br>
>> ><br>
>> ><br>
>> ><br>
>> > ------------------<br>
>> ><br>
>> > Rodrigo Benzaquen<br>
>> > @rbenzaquen<br>
>> ><br>
>> ><br>
>> ><br>
>> > ________________________________<br>
>> > From: "Taylan Develioglu" <<a href="mailto:tdevelioglu@ebuddy.com">tdevelioglu@ebuddy.com</a>><br>
>> > To: <a href="mailto:varnish-misc@varnish-cache.org">varnish-misc@varnish-cache.org</a><br>
>> > Sent: Viernes, 29 de Octubre 2010 8:37:21<br>
>> > Subject: varnish 2.1.4 swap and increase in objects:heads.<br>
>> ><br>
>> > Debian Lenny, Linux varnish0 2.6.33.7 #1 SMP Thu Sep 2 15:27:53 CEST<br>
>> > 2010 x86_64 GNU/Linux<br>
>> ><br>
>> > I have a strange situation on one of our varnish servers.<br>
>> ><br>
>> > We recently raised the expiry on previously uncached content a<br>
>> > (Cache-Control: max-age=0, to Cache-Control: max-age=5).<br>
>> ><br>
>> > The number of objects vs. heads increased and the system started<br>
>> > swapping (vm.swappiness=0).<br>
>> ><br>
>> > <a href="http://lcontent.ebuddy.com/varnish/memory-week.png" target="_blank">http://lcontent.ebuddy.com/varnish/memory-week.png</a><br>
>> > <a href="http://lcontent.ebuddy.com/varnish/varnish_objects-week.png" target="_blank">http://lcontent.ebuddy.com/varnish/varnish_objects-week.png</a><br>
>> > <a href="http://lcontent.ebuddy.com/varnish/varnish_expunge-week.png" target="_blank">http://lcontent.ebuddy.com/varnish/varnish_expunge-week.png</a><br>
>> ><br>
>> ><br>
>> > I don't understand why this is happening, bug maybe ?<br>
>> ><br>
>> > DAEMON_OPTS="-a :27535 \<br>
>> > -T localhost:6082 \<br>
>> > -f /etc/varnish/default.vcl \<br>
>> > -s file,/var/cache/varnish/sda/varnish_storage.bin,135G \<br>
>> > -s file,/var/cache/varnish/sdb/varnish_storage.bin,135G \<br>
>> > -s file,/var/cache/varnish/sdc/varnish_storage.bin,135G \<br>
>> > -s file,/var/cache/varnish/sdd/varnish_storage.bin,135G \<br>
>> > -s file,/var/cache/varnish/sde/varnish_storage.bin,135G \<br>
>> > -s file,/var/cache/varnish/sdf/varnish_storage.bin,135G \<br>
>> > -p thread_pools=4 \<br>
>> > -p thread_pool_min=200 \<br>
>> > -p thread_pool_max=2400 \<br>
>> > -p thread_pool_add_delay=2 \<br>
>> > -p lru_interval=20"<br>
>> ><br>
>> ><br>
>> ><br>
>> > ---vcl---<br>
>> > sub vcl_recv {<br>
>> > unset req.http.cookie;<br>
>> > unset req.http.Authorization;<br>
>> ><br>
>> > if (req.http.host ~ "^.*host.domain.com.*") {<br>
>> > set req.http.host = "<a href="http://host.domain.com" target="_blank">host.domain.com</a>";<br>
>> > }<br>
>> ><br>
>> > if (req.request == "HEAD") {<br>
>> > return(pass);<br>
>> > }<br>
>> ><br>
>> > if (req.request == "PURGE") {<br>
>> > if (!client.ip ~ purge) {<br>
>> > error 405 "Not allowed.";<br>
>> > }<br>
>> > return(lookup);<br>
>> > }<br>
>> ><br>
>> > if (req.request == "GET" && req.url == "/monitor") {<br>
>> > error 200 "OK.";<br>
>> > }<br>
>> ><br>
>> > if (req.backend.healthy) {<br>
>> > set req.grace = 30s;<br>
>> > } else {<br>
>> > set req.grace = 48h;<br>
>> > }<br>
>> ><br>
>> > }<br>
>> ><br>
>> > sub vcl_fetch {<br>
>> > if (beresp.status == 404 || beresp.status >= 300) {<br>
>> > return (pass);<br>
>> > }<br>
>> > set beresp.grace = 48h;<br>
>> ><br>
>> > }<br>
>> ><br>
>> > sub vcl_hit {<br>
>> > if (req.request == "PURGE") {<br>
>> > set obj.ttl = 0s;<br>
>> > error 200 "Purged.";<br>
>> > }<br>
>> > }<br>
>> ><br>
>> > sub vcl_miss {<br>
>> > if (req.request == "PURGE") {<br>
>> > error 404 "Not in cache.";<br>
>> > }<br>
>> > }<br>
>> > ---EOF---<br>
>> ><br>
>> ><br>
>> > _______________________________________________<br>
>> > varnish-misc mailing list<br>
>> > <a href="mailto:varnish-misc@varnish-cache.org">varnish-misc@varnish-cache.org</a><br>
>> > <a href="http://lists.varnish-cache.org/mailman/listinfo/varnish-misc" target="_blank">http://lists.varnish-cache.org/mailman/listinfo/varnish-misc</a><br>
>> ><br>
>> > _______________________________________________<br>
>> > varnish-misc mailing list<br>
>> > <a href="mailto:varnish-misc@varnish-cache.org">varnish-misc@varnish-cache.org</a><br>
>> > <a href="http://lists.varnish-cache.org/mailman/listinfo/varnish-misc" target="_blank">http://lists.varnish-cache.org/mailman/listinfo/varnish-misc</a><br>
>> ><br>
>><br>
>><br>
>><br>
><br>
><br>
><br>
> _______________________________________________<br>
> varnish-misc mailing list<br>
> <a href="mailto:varnish-misc@varnish-cache.org">varnish-misc@varnish-cache.org</a><br>
> <a href="http://lists.varnish-cache.org/mailman/listinfo/varnish-misc" target="_blank">http://lists.varnish-cache.org/mailman/listinfo/varnish-misc</a><br>
><br>
<br>
_______________________________________________<br>
varnish-misc mailing list<br>
<a href="mailto:varnish-misc@varnish-cache.org">varnish-misc@varnish-cache.org</a><br>
<a href="http://lists.varnish-cache.org/mailman/listinfo/varnish-misc" target="_blank">http://lists.varnish-cache.org/mailman/listinfo/varnish-misc</a><br>
</blockquote></div><br></div>