The current load is just above 2. I'll check this again when I see a load spike.<br><br>[cloude@squid03 ~]$ iostat -k -x 5<br>Linux 2.6.18-53.1.19.el5.centos.plus (<a href="http://squid03.instructables.com">squid03.instructables.com</a>) 04/09/2009<br>
<br>avg-cpu: %user %nice %system %iowait %steal %idle<br> 1.19 0.00 0.95 2.14 0.00 95.73<br><br>Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util<br>
sda 0.07 9.64 0.15 1.65 10.08 45.68 61.80 0.13 70.32 3.96 0.72<br>sdb 0.07 9.63 0.15 1.66 10.14 45.68 61.75 0.02 10.03 3.76 0.68<br>sdc 0.03 16.47 1.21 14.69 13.99 128.81 17.96 0.08 4.81 4.31 6.85<br>
sdd 0.03 16.45 1.17 13.24 13.29 119.96 18.49 0.24 16.52 4.06 5.86<br>md1 0.00 0.00 0.43 11.13 20.19 44.52 11.19 0.00 0.00 0.00 0.00<br>md2 0.00 0.00 2.41 29.40 26.58 117.61 9.07 0.00 0.00 0.00 0.00<br>
md0 0.00 0.00 0.00 0.00 0.00 0.00 3.15 0.00 0.00 0.00 0.00<br><br>avg-cpu: %user %nice %system %iowait %steal %idle<br> 0.90 0.00 2.40 46.70 0.00 50.00<br>
<br>Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util<br>sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br>sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br>
sdc 0.00 0.40 6.00 238.40 74.40 974.40 8.58 132.88 515.03 4.09 100.02<br>sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br>md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br>
md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br>md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br><br>avg-cpu: %user %nice %system %iowait %steal %idle<br>
0.90 0.00 1.80 67.67 0.00 29.63<br><br>Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util<br>sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br>
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br>sdc 0.00 1.60 13.40 141.80 188.80 1053.60 16.01 138.62 934.04 6.44 100.02<br>sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br>
md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br>md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br>md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br>
<br>avg-cpu: %user %nice %system %iowait %steal %idle<br> 0.50 0.00 1.80 61.40 0.00 36.30<br><br>Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util<br>
sda 0.00 0.00 0.00 0.40 0.00 2.40 12.00 0.00 9.00 9.00 0.36<br>sdb 0.00 0.00 0.00 0.40 0.00 2.40 12.00 0.00 9.50 9.50 0.38<br>sdc 0.00 1.60 6.40 257.00 132.00 2195.20 17.67 107.40 450.21 3.68 96.82<br>
sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br>md1 0.00 0.00 0.00 0.20 0.00 0.80 8.00 0.00 0.00 0.00 0.00<br>md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br>
md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br><br>avg-cpu: %user %nice %system %iowait %steal %idle<br> 0.60 0.00 1.60 47.80 0.00 50.00<br>
<br>Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util<br>sda 0.00 0.00 0.00 0.20 0.00 1.60 16.00 0.00 11.00 11.00 0.22<br>sdb 0.00 0.00 0.00 0.20 0.00 1.60 16.00 0.00 13.00 13.00 0.26<br>
sdc 0.00 0.80 0.20 301.80 8.80 1270.40 8.47 119.40 373.98 3.31 100.04<br>sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br>md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br>
md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br>md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br><br>avg-cpu: %user %nice %system %iowait %steal %idle<br>
0.60 0.00 1.70 47.80 0.00 49.90<br><br>Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util<br>sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br>
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br>sdc 0.00 1.20 2.40 245.31 43.11 1538.52 12.77 101.41 419.12 4.03 99.80<br>sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br>
md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br>md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br>md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br>
<br>avg-cpu: %user %nice %system %iowait %steal %idle<br> 0.60 0.00 1.50 3.20 0.00 94.69<br><br>Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util<br>
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br>sdb 0.20 0.00 0.40 0.00 2.40 0.00 12.00 0.01 14.00 7.00 0.28<br>sdc 0.00 0.00 6.60 11.00 174.40 192.80 41.73 1.26 421.34 3.73 6.56<br>
sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br>md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br>md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br>
md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br><br>avg-cpu: %user %nice %system %iowait %steal %idle<br> 0.70 0.00 1.60 29.50 0.00 68.20<br>
<br>Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util<br>sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br>sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br>
sdc 0.00 0.00 5.60 208.60 110.40 857.60 9.04 70.18 301.18 2.90 62.06<br>sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br>md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br>
md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br>md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br><br>avg-cpu: %user %nice %system %iowait %steal %idle<br>
0.50 0.00 1.50 48.05 0.00 49.95<br><br>Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util<br>sda 0.00 0.20 0.00 0.80 0.00 5.60 14.00 0.01 8.75 8.75 0.70<br>
sdb 0.00 0.20 0.00 0.80 0.00 5.60 14.00 0.01 9.50 9.50 0.76<br>sdc 0.00 1.00 6.80 232.40 91.20 1180.80 10.64 110.32 475.49 4.18 100.02<br>sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br>
md1 0.00 0.00 0.00 0.60 0.00 2.40 8.00 0.00 0.00 0.00 0.00<br>md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br>md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00<br>
<br><br><br><br><div class="gmail_quote">On Thu, Apr 9, 2009 at 1:43 PM, Artur Bergman <span dir="ltr"><<a href="mailto:sky@crucially.net">sky@crucially.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div style="">What is your iopressure?<div><br></div><div>iostat -k -x 5</div><div><br></div><div>or something like that</div><div><br></div><font color="#888888"><div>artur</div></font><div><div></div><div class="h5"><div>
<br><div><div>On Apr 9, 2009, at 12:27 PM, Cloude Porteus wrote:</div><br><blockquote type="cite">Varnishstat doesn't list any nuked objects and file storage and shmlog look like they have plenty of space:<br><br>df -h<br>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~<br>Filesystem Size Used Avail Use% Mounted on<br>tmpfs 150M 81M 70M 54% /usr/local/var/varnish<br> /dev/sdc1 74G 11G 61G 16% /var/lib/varnish<br>
<br>top<br>~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~<br>top - 12:26:33 up 164 days, 22:21, 1 user, load average: 2.60, 3.26, 3.75<br>Tasks: 67 total, 1 running, 66 sleeping, 0 stopped, 0 zombie<br> Cpu(s): 0.7%us, 0.3%sy, 0.0%ni, 97.0%id, 0.7%wa, 0.3%hi, 1.0%si, 0.0%st<br>
Mem: 8183492k total, 7763100k used, 420392k free, 13424k buffers<br>Swap: 3148720k total, 56636k used, 3092084k free, 7317692k cached<br> <br> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br>
7441 varnish 15 0 70.0g 6.4g 6.1g S 2 82.5 56:33.31 varnishd<br><br><br>Varnishstat:<br>~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~<br>Hitrate ratio: 8 8 8<br> Hitrate avg: 0.9782 0.9782 0.9782<br>
<br> 36494404 219.98 160.57 Client connections accepted<br> 36494486 220.98 160.57 Client requests received<br> 35028477 212.98 154.12 Cache hits<br> 474091 4.00 2.09 Cache hits for pass<br>
988013 6.00 4.35 Cache misses<br> 1465955 10.00 6.45 Backend connections success<br> 9 0.00 0.00 Backend connections failures<br> 994 . . N struct sess_mem<br>
11 . . N struct sess<br> 274047 . . N struct object<br> 252063 . . N struct objecthead<br> 609018 . . N struct smf<br>
28720 . . N small free smf<br> 2 . . N large free smf<br> 2 . . N struct vbe_conn<br> 901 . . N struct bereq<br>
2000 . . N worker threads<br> 2000 0.00 0.01 N worker threads created<br> 143 0.00 0.00 N overflowed work requests<br> 1 . . N backends<br>
672670 . . N expired objects<br> 3514467 . . N LRU moved objects<br> 49 0.00 0.00 HTTP header overflows<br> 32124238 206.98 141.34 Objects sent with write<br>
36494396 224.98 160.57 Total Sessions<br> 36494484 224.98 160.57 Total Requests<br> 783 0.00 0.00 Total pipe<br> 518770 4.00 2.28 Total pass<br>
1464570 10.00 6.44 Total fetch<br> 14559014884 93563.69 64058.18 Total header bytes<br>168823109304 489874.04 742804.45 Total body bytes<br> 36494387 224.98 160.57 Session Closed<br>
203 0.00 0.00 Session herd<br> 1736767745 10880.80 7641.60 SHM records<br> 148079555 908.90 651.53 SHM writes<br> 15088 0.00 0.07 SHM flushes due to overflow<br>
10494 0.00 0.05 SHM MTX contention<br> 687 0.00 0.00 SHM cycles through buffer<br> 2988576 21.00 13.15 allocator requests<br> 580296 . . outstanding allocations<br>
8916353024 . . bytes allocated<br> 44770738176 . . bytes free<br> 656 0.00 0.00 SMS allocator requests<br> 303864 . . SMS bytes allocated<br>
303864 . . SMS bytes freed<br> 1465172 10.00 6.45 Backend requests made<br><br><br><br><div class="gmail_quote">On Thu, Apr 9, 2009 at 12:18 PM, Artur Bergman <span dir="ltr"><<a href="mailto:sky@crucially.net" target="_blank">sky@crucially.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div><div>For the file storage or for the shmlog?</div><div><br></div><div>When do you start nuking/expiring from disk? I suspect the load goes up when you run out of storage space?</div>
<div><br></div><div>Cheers</div><div>Artur</div><div><br></div><div><br><div><div><div></div><div><div>On Apr 9, 2009, at 12:02 PM, Cloude Porteus wrote:</div><br></div></div><blockquote type="cite"><div><div></div> <div>
Has anyone experienced very high server load averages? We're running varnish on a dual core with 8gb of ram. It runs okay for a day or two and then I start seeing load averages in 6-10 range for an hour or so, drops down to 2-3, then goes back up.<br>
<br>This starts to happen once we have more items in the cache than our physical memory. Maybe increasing our lru_interval will help? It's currently set to 3600.<br><br>Right now we're running with a 50gb file storage option. There are 270k objects in the cache, 70gb virtual memory, 6.2gb of res memory used, 11gb of data on disk in the file storage. We have a 98% hit ratio.<br>
<br>We followed Artur's advice about setting a tmpfs and creating an ext2 partition for our file storage.<br><br>I also tried running with malloc as our storage type, but I had to set it at a little less than half of our physical ram in order for it to work well after the cache got full. I don't understand why the virtual memory is double when I am running in malloc mode. I was running it with 5gb and the virtual memory was about 10-12gb and once it got full it started using the swap memory.<br>
<br>Thanks for any help/insight.<br><br>best,<br>cloude<br>-- <br>VP of Product Development<br>Instructables.com<br><br><a href="http://www.instructables.com/member/lebowski" target="_blank">http://www.instructables.com/member/lebowski</a><br>
</div></div> _______________________________________________<br>varnish-dev mailing list<br><a href="mailto:varnish-dev@projects.linpro.no" target="_blank">varnish-dev@projects.linpro.no</a><br><a href="http://projects.linpro.no/mailman/listinfo/varnish-dev" target="_blank">http://projects.linpro.no/mailman/listinfo/varnish-dev</a><br>
</blockquote></div><br></div></div></blockquote></div><br><br clear="all"><br>-- <br>VP of Product Development<br>Instructables.com<br><br><a href="http://www.instructables.com/member/lebowski" target="_blank">http://www.instructables.com/member/lebowski</a><br>
</blockquote></div><br></div></div></div></div></blockquote></div><br><br clear="all"><br>-- <br>VP of Product Development<br>Instructables.com<br><br><a href="http://www.instructables.com/member/lebowski">http://www.instructables.com/member/lebowski</a><br>