Under Load: Server Unavailable/Connection Dropped/Delayed Reponse
Tejaswi Nadahalli
nadahalli at gmail.com
Fri Mar 4 07:08:36 CET 2011
Hi Everyone,
I am seeing a situation similar to :
http://www.varnish-cache.org/lists/pipermail/varnish-misc/2011-January/005351.html(Connections
Dropped Under Load)
http://www.varnish-cache.org/lists/pipermail/varnish-misc/2010-December/005258.html(Hanging
Connections)
I have httperf loading a varnish cache with never-expire content. While the
load is on, other browser/wget requests to the varnish server get delayed to
10+ seconds. Any ideas what could be happening? ssh doesn't seem to be
impacted. So, is it some kind of thread problem?
In production, I see a similar situation with around 1000 req/second load.
I am running varnishd with the following command line options (as per
http://kristianlyng.wordpress.com/2009/10/19/high-end-varnish-tuning/):
sudo varnishd -f /etc/varnish/default.vcl -s malloc,5G -T 127.0.0.1:2000 -a
0.0.0.0:80 -p thread_pools=8 -p thread_pool_min=100 -p thread_pool_max=5000
-p thread_pool_add_delay=2 -p cli_timeout=25 -p session_linger=100 -p
lru_interval=20 -t 31536000
I am on Ubuntu Lucid 64 bit Amazon EC2 C1.XLarge with 8 processing units.
My network sysctl parameters are tuned according to:
http://varnish-cache.org/trac/wiki/Performance
fs.file-max = 360000
net.ipv4.ip_local_port_range = 1024 65536
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_fin_timeout = 3
net.core.netdev_max_backlog = 30000
net.ipv4.tcp_no_metrics_save = 1
net.core.somaxconn = 262144
net.ipv4.tcp_syncookies = 0
net.ipv4.tcp_max_orphans = 262144
net.ipv4.tcp_max_syn_backlog = 262144
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 2
Any help would be greatly appreciated
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20110304/413ac928/attachment-0003.html>
-------------- next part --------------
client_conn 12408 8.19 Client connections accepted
client_drop 0 0.00 Connection dropped, no sess/wrk
client_req 5549280 3662.89 Client requests received
cache_hit 5543904 3659.34 Cache hits
cache_hitpass 0 0.00 Cache hits for pass
cache_miss 5376 3.55 Cache misses
backend_conn 5373 3.55 Backend conn. success
backend_unhealthy 0 0.00 Backend conn. not attempted
backend_busy 0 0.00 Backend conn. too many
backend_fail 3 0.00 Backend conn. failures
backend_reuse 0 0.00 Backend conn. reuses
backend_toolate 0 0.00 Backend conn. was closed
backend_recycle 0 0.00 Backend conn. recycles
backend_unused 0 0.00 Backend conn. unused
fetch_head 0 0.00 Fetch head
fetch_length 5373 3.55 Fetch with Length
fetch_chunked 0 0.00 Fetch chunked
fetch_eof 0 0.00 Fetch EOF
fetch_bad 0 0.00 Fetch had bad headers
fetch_close 0 0.00 Fetch wanted close
fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed
fetch_zero 0 0.00 Fetch zero len
fetch_failed 0 0.00 Fetch failed
n_sess_mem 798 . N struct sess_mem
n_sess 548 . N struct sess
n_object 5373 . N struct object
n_vampireobject 0 . N unresurrected objects
n_objectcore 5806 . N struct objectcore
n_objecthead 5806 . N struct objecthead
n_smf 0 . N struct smf
n_smf_frag 0 . N small free smf
n_smf_large 0 . N large free smf
n_vbe_conn 0 . N struct vbe_conn
n_wrk 800 . N worker threads
n_wrk_create 800 0.53 N worker threads created
n_wrk_failed 0 0.00 N worker threads not created
n_wrk_max 0 0.00 N worker threads limited
n_wrk_queue 0 0.00 N queued work requests
n_wrk_overflow 0 0.00 N overflowed work requests
n_wrk_drop 0 0.00 N dropped work requests
n_backend 1 . N backends
n_expired 0 . N expired objects
n_lru_nuked 0 . N LRU nuked objects
n_lru_saved 0 . N LRU saved objects
n_lru_moved 74099 . N LRU moved objects
n_deathrow 0 . N objects on deathrow
losthdr 0 0.00 HTTP header overflows
n_objsendfile 0 0.00 Objects sent with sendfile
n_objwrite 5543132 3658.83 Objects sent with write
n_objoverflow 0 0.00 Objects overflowing workspace
s_sess 12407 8.19 Total Sessions
s_req 5549280 3662.89 Total Requests
s_pipe 0 0.00 Total pipe
s_pass 0 0.00 Total pass
s_fetch 5373 3.55 Total fetch
s_hdrbytes 1245394845 822042.80 Total header bytes
s_bodybytes 13448598673 8876962.82 Total body bytes
sess_closed 43 0.03 Session Closed
sess_pipeline 0 0.00 Session Pipeline
sess_readahead 0 0.00 Session Read Ahead
sess_linger 5549262 3662.88 Session Linger
sess_herd 1431702 945.02 Session herd
shm_records 162564756 107303.47 SHM records
shm_writes 7031015 4640.93 SHM writes
shm_flushes 0 0.00 SHM flushes due to overflow
shm_cont 138344 91.32 SHM MTX contention
shm_cycles 60 0.04 SHM cycles through buffer
sm_nreq 0 0.00 allocator requests
sm_nobj 0 . outstanding allocations
sm_balloc 0 . bytes allocated
sm_bfree 0 . bytes free
sma_nreq 10746 7.09 SMA allocator requests
sma_nobj 10746 . SMA outstanding allocations
sma_nbytes 17538168 . SMA outstanding bytes
sma_balloc 17538168 . SMA bytes allocated
sma_bfree 0 . SMA bytes free
sms_nreq 3 0.00 SMS allocator requests
sms_nobj 0 . SMS outstanding allocations
sms_nbytes 0 . SMS outstanding bytes
sms_balloc 1254 . SMS bytes allocated
sms_bfree 1254 . SMS bytes freed
backend_req 5373 3.55 Backend requests made
n_vcl 1 0.00 N vcl total
n_vcl_avail 1 0.00 N vcl available
n_vcl_discard 0 0.00 N vcl discarded
n_purge 1 . N total active purges
n_purge_add 1 0.00 N new purges added
n_purge_retire 0 0.00 N old purges deleted
n_purge_obj_test 0 0.00 N objects tested
n_purge_re_test 0 0.00 N regexps tested against
n_purge_dups 0 0.00 N duplicate purges removed
hcb_nolock 5540904 3657.36 HCB Lookups without lock
hcb_lock 5375 3.55 HCB Lookups with lock
hcb_insert 5373 3.55 HCB Inserts
esi_parse 0 0.00 Objects ESI parsed (unlock)
esi_errors 0 0.00 ESI parse errors (unlock)
accept_fail 0 0.00 Accept failures
client_drop_late 0 0.00 Connection dropped late
uptime 1515 1.00 Client uptime
backend_retry 0 0.00 Backend conn. retry
dir_dns_lookups 0 0.00 DNS director lookups
dir_dns_failed 0 0.00 DNS director failed lookups
dir_dns_hit 0 0.00 DNS director cached lookups hit
dir_dns_cache_full 0 0.00 DNS director full dnscache
fetch_1xx 0 0.00 Fetch no body (1xx)
fetch_204 0 0.00 Fetch no body (204)
fetch_304 0 0.00 Fetch no body (304)
More information about the varnish-misc
mailing list