High Server Load Averages?
Cloude Porteus
cloude at instructables.com
Thu Apr 9 23:46:08 CEST 2009
The current load is just above 2. I'll check this again when I see a load
spike.
[cloude at squid03 ~]$ iostat -k -x 5
Linux 2.6.18-53.1.19.el5.centos.plus (squid03.instructables.com)
04/09/2009
avg-cpu: %user %nice %system %iowait %steal %idle
1.19 0.00 0.95 2.14 0.00 95.73
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await svctm %util
sda 0.07 9.64 0.15 1.65 10.08 45.68 61.80
0.13 70.32 3.96 0.72
sdb 0.07 9.63 0.15 1.66 10.14 45.68 61.75
0.02 10.03 3.76 0.68
sdc 0.03 16.47 1.21 14.69 13.99 128.81 17.96
0.08 4.81 4.31 6.85
sdd 0.03 16.45 1.17 13.24 13.29 119.96 18.49
0.24 16.52 4.06 5.86
md1 0.00 0.00 0.43 11.13 20.19 44.52 11.19
0.00 0.00 0.00 0.00
md2 0.00 0.00 2.41 29.40 26.58 117.61 9.07
0.00 0.00 0.00 0.00
md0 0.00 0.00 0.00 0.00 0.00 0.00 3.15
0.00 0.00 0.00 0.00
avg-cpu: %user %nice %system %iowait %steal %idle
0.90 0.00 2.40 46.70 0.00 50.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sdc 0.00 0.40 6.00 238.40 74.40 974.40 8.58
132.88 515.03 4.09 100.02
sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
avg-cpu: %user %nice %system %iowait %steal %idle
0.90 0.00 1.80 67.67 0.00 29.63
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sdc 0.00 1.60 13.40 141.80 188.80 1053.60 16.01
138.62 934.04 6.44 100.02
sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
avg-cpu: %user %nice %system %iowait %steal %idle
0.50 0.00 1.80 61.40 0.00 36.30
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await svctm %util
sda 0.00 0.00 0.00 0.40 0.00 2.40 12.00
0.00 9.00 9.00 0.36
sdb 0.00 0.00 0.00 0.40 0.00 2.40 12.00
0.00 9.50 9.50 0.38
sdc 0.00 1.60 6.40 257.00 132.00 2195.20 17.67
107.40 450.21 3.68 96.82
sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
md1 0.00 0.00 0.00 0.20 0.00 0.80 8.00
0.00 0.00 0.00 0.00
md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
avg-cpu: %user %nice %system %iowait %steal %idle
0.60 0.00 1.60 47.80 0.00 50.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await svctm %util
sda 0.00 0.00 0.00 0.20 0.00 1.60 16.00
0.00 11.00 11.00 0.22
sdb 0.00 0.00 0.00 0.20 0.00 1.60 16.00
0.00 13.00 13.00 0.26
sdc 0.00 0.80 0.20 301.80 8.80 1270.40 8.47
119.40 373.98 3.31 100.04
sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
avg-cpu: %user %nice %system %iowait %steal %idle
0.60 0.00 1.70 47.80 0.00 49.90
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sdc 0.00 1.20 2.40 245.31 43.11 1538.52 12.77
101.41 419.12 4.03 99.80
sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
avg-cpu: %user %nice %system %iowait %steal %idle
0.60 0.00 1.50 3.20 0.00 94.69
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sdb 0.20 0.00 0.40 0.00 2.40 0.00 12.00
0.01 14.00 7.00 0.28
sdc 0.00 0.00 6.60 11.00 174.40 192.80 41.73
1.26 421.34 3.73 6.56
sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
avg-cpu: %user %nice %system %iowait %steal %idle
0.70 0.00 1.60 29.50 0.00 68.20
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
sdc 0.00 0.00 5.60 208.60 110.40 857.60 9.04
70.18 301.18 2.90 62.06
sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
avg-cpu: %user %nice %system %iowait %steal %idle
0.50 0.00 1.50 48.05 0.00 49.95
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await svctm %util
sda 0.00 0.20 0.00 0.80 0.00 5.60 14.00
0.01 8.75 8.75 0.70
sdb 0.00 0.20 0.00 0.80 0.00 5.60 14.00
0.01 9.50 9.50 0.76
sdc 0.00 1.00 6.80 232.40 91.20 1180.80 10.64
110.32 475.49 4.18 100.02
sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
md1 0.00 0.00 0.00 0.60 0.00 2.40 8.00
0.00 0.00 0.00 0.00
md2 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00
On Thu, Apr 9, 2009 at 1:43 PM, Artur Bergman <sky at crucially.net> wrote:
> What is your iopressure?
> iostat -k -x 5
>
> or something like that
>
> artur
>
> On Apr 9, 2009, at 12:27 PM, Cloude Porteus wrote:
>
> Varnishstat doesn't list any nuked objects and file storage and shmlog look
> like they have plenty of space:
>
> df -h
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Filesystem Size Used Avail Use% Mounted on
> tmpfs 150M 81M 70M 54% /usr/local/var/varnish
> /dev/sdc1 74G 11G 61G 16% /var/lib/varnish
>
> top
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> top - 12:26:33 up 164 days, 22:21, 1 user, load average: 2.60, 3.26, 3.75
> Tasks: 67 total, 1 running, 66 sleeping, 0 stopped, 0 zombie
> Cpu(s): 0.7%us, 0.3%sy, 0.0%ni, 97.0%id, 0.7%wa, 0.3%hi, 1.0%si,
> 0.0%st
> Mem: 8183492k total, 7763100k used, 420392k free, 13424k buffers
> Swap: 3148720k total, 56636k used, 3092084k free, 7317692k cached
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 7441 varnish 15 0 70.0g 6.4g 6.1g S 2 82.5 56:33.31 varnishd
>
>
> Varnishstat:
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Hitrate ratio: 8 8 8
> Hitrate avg: 0.9782 0.9782 0.9782
>
> 36494404 219.98 160.57 Client connections accepted
> 36494486 220.98 160.57 Client requests received
> 35028477 212.98 154.12 Cache hits
> 474091 4.00 2.09 Cache hits for pass
> 988013 6.00 4.35 Cache misses
> 1465955 10.00 6.45 Backend connections success
> 9 0.00 0.00 Backend connections failures
> 994 . . N struct sess_mem
> 11 . . N struct sess
> 274047 . . N struct object
> 252063 . . N struct objecthead
> 609018 . . N struct smf
> 28720 . . N small free smf
> 2 . . N large free smf
> 2 . . N struct vbe_conn
> 901 . . N struct bereq
> 2000 . . N worker threads
> 2000 0.00 0.01 N worker threads created
> 143 0.00 0.00 N overflowed work requests
> 1 . . N backends
> 672670 . . N expired objects
> 3514467 . . N LRU moved objects
> 49 0.00 0.00 HTTP header overflows
> 32124238 206.98 141.34 Objects sent with write
> 36494396 224.98 160.57 Total Sessions
> 36494484 224.98 160.57 Total Requests
> 783 0.00 0.00 Total pipe
> 518770 4.00 2.28 Total pass
> 1464570 10.00 6.44 Total fetch
> 14559014884 93563.69 64058.18 Total header bytes
> 168823109304 489874.04 742804.45 Total body bytes
> 36494387 224.98 160.57 Session Closed
> 203 0.00 0.00 Session herd
> 1736767745 10880.80 7641.60 SHM records
> 148079555 908.90 651.53 SHM writes
> 15088 0.00 0.07 SHM flushes due to overflow
> 10494 0.00 0.05 SHM MTX contention
> 687 0.00 0.00 SHM cycles through buffer
> 2988576 21.00 13.15 allocator requests
> 580296 . . outstanding allocations
> 8916353024 . . bytes allocated
> 44770738176 . . bytes free
> 656 0.00 0.00 SMS allocator requests
> 303864 . . SMS bytes allocated
> 303864 . . SMS bytes freed
> 1465172 10.00 6.45 Backend requests made
>
>
>
> On Thu, Apr 9, 2009 at 12:18 PM, Artur Bergman <sky at crucially.net> wrote:
>
>> For the file storage or for the shmlog?
>>
>> When do you start nuking/expiring from disk? I suspect the load goes up
>> when you run out of storage space?
>>
>> Cheers
>> Artur
>>
>>
>> On Apr 9, 2009, at 12:02 PM, Cloude Porteus wrote:
>>
>> Has anyone experienced very high server load averages? We're running
>> varnish on a dual core with 8gb of ram. It runs okay for a day or two and
>> then I start seeing load averages in 6-10 range for an hour or so, drops
>> down to 2-3, then goes back up.
>>
>> This starts to happen once we have more items in the cache than our
>> physical memory. Maybe increasing our lru_interval will help? It's currently
>> set to 3600.
>>
>> Right now we're running with a 50gb file storage option. There are 270k
>> objects in the cache, 70gb virtual memory, 6.2gb of res memory used, 11gb of
>> data on disk in the file storage. We have a 98% hit ratio.
>>
>> We followed Artur's advice about setting a tmpfs and creating an ext2
>> partition for our file storage.
>>
>> I also tried running with malloc as our storage type, but I had to set it
>> at a little less than half of our physical ram in order for it to work well
>> after the cache got full. I don't understand why the virtual memory is
>> double when I am running in malloc mode. I was running it with 5gb and the
>> virtual memory was about 10-12gb and once it got full it started using the
>> swap memory.
>>
>> Thanks for any help/insight.
>>
>> best,
>> cloude
>> --
>> VP of Product Development
>> Instructables.com
>>
>> http://www.instructables.com/member/lebowski
>> _______________________________________________
>> varnish-dev mailing list
>> varnish-dev at projects.linpro.no
>> http://projects.linpro.no/mailman/listinfo/varnish-dev
>>
>>
>>
>
>
> --
> VP of Product Development
> Instructables.com
>
> http://www.instructables.com/member/lebowski
>
>
>
--
VP of Product Development
Instructables.com
http://www.instructables.com/member/lebowski
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20090409/54de8d12/attachment-0001.html>
More information about the varnish-dev
mailing list