And here's the backtrace I tried to capture from one the hunged varnishes.<div><br></div><div><div> # ps auxww | grep varnish</div><div>root 2649 0.0 2.4 101776 82744 pts/0 S< 13:48 0:01 varnishlog -w varnish.log</div>
<div>root 25199 0.0 0.0 110556 1140 ? S<s 15:12 0:00 /usr/sbin/varnishd -P /var/run/varnishd.pid -a :6081 -T localhost:6082 -p sess_timeout 25 -p obj_workspace 8192 -p sess_workspace 262144 -p listen_depth 8192 -w 400,1000,60 -p connect_timeout 3s -p thread_pools 4 -p thread_pool_min 400 -p thread_pool_max 4000 -f /etc/varnish/default.vcl</div>
<div>nobody 25200 0.0 1.9 19500060 64732 ? S<l 15:12 0:00 /usr/sbin/varnishd -P /var/run/varnishd.pid -a :6081 -T localhost:6082 -p sess_timeout 25 -p obj_workspace 8192 -p sess_workspace 262144 -p listen_depth 8192 -w 400,1000,60 -p connect_timeout 3s -p thread_pools 4 -p thread_pool_min 400 -p thread_pool_max 4000 -f /etc/varnish/default.vcl</div>
<div>root 27034 0.0 0.0 3876 556 pts/0 R<+ 15:23 0:00 grep --colour=auto varnish</div><div> # which gdb</div><div>/usr/bin/gdb</div><div> # gdb /usr/sbin/varnishd 25200</div><div>GNU gdb 6.7.1</div><div>
Copyright (C) 2007 Free Software Foundation, Inc.</div><div>License GPLv3+: GNU GPL version 3 or later <<a href="http://gnu.org/licenses/gpl.html">http://gnu.org/licenses/gpl.html</a>></div><div>This is free software: you are free to change and redistribute it.</div>
<div>There is NO WARRANTY, to the extent permitted by law. Type "show copying"</div><div>and "show warranty" for details.</div><div>This GDB was configured as "x86_64-pc-linux-gnu"...</div><div>
(no debugging symbols found)</div><div>Using host libthread_db library "/lib/libthread_db.so.1".</div><div>(no debugging symbols found)</div><div>Attaching to program: /usr/sbin/varnishd, process 25200</div><div>
Error while mapping shared library sections:</div><div>./<a href="http://vcl.1P9zoqAU.so">vcl.1P9zoqAU.so</a>: No such file or directory.</div><div>Reading symbols from /usr/lib64/libvarnish.so.1...Reading symbols from /usr/lib64/debug/usr/lib64/libvarnish.so.1.0.0.debug...(no debugging symbols found)...done.</div>
<div>(no debugging symbols found)...done.</div><div>Loaded symbols for /usr/lib/libvarnish.so.1</div><div>Reading symbols from /usr/lib64/libvarnishcompat.so.1...Reading symbols from /usr/lib64/debug/usr/lib64/libvarnishcompat.so.1.0.0.debug...(no debugging symbols found)...done.</div>
<div><br></div><div>(no debugging symbols found)...done.</div><div>Loaded symbols for /usr/lib/libvarnishcompat.so.1</div><div>Reading symbols from /usr/lib64/libvcl.so.1...Reading symbols from /usr/lib64/debug/usr/lib64/libvcl.so.1.0.0.debug...(no debugging symbols found)...done.</div>
<div>(no debugging symbols found)...done.</div><div>Loaded symbols for /usr/lib/libvcl.so.1</div><div>Reading symbols from /lib64/libdl.so.2...Reading symbols from /usr/lib64/debug/lib64/libdl-2.6.1.so.debug...done.</div>
<div>done.</div><div>Loaded symbols for /lib/libdl.so.2</div><div>Reading symbols from /lib64/libpthread.so.0...Reading symbols from /usr/lib64/debug/lib64/libpthread-2.6.1.so.debug...done.</div><div>[Thread debugging using libthread_db enabled]</div>
<div>[New Thread 0x2ac6f560c440 (LWP 25200)]</div><div>[New Thread 0x2aaf18807950 (LWP 26852)]</div><div><br></div><div>.....</div><div><br></div><div>Loaded symbols for /lib/libnss_files.so.2</div><div>Symbol file not found for ./<a href="http://vcl.1P9zoqAU.so">vcl.1P9zoqAU.so</a></div>
<div><br></div><div>0x00002ac6f5179c46 in poll () from /lib/libc.so.6</div><div>(gdb) bt</div><div>#0 0x00002ac6f5179c46 in poll () from /lib/libc.so.6</div><div>#1 0x0000000000411e95 in CLI_Run ()</div><div>#2 0x000000000041a5e3 in child_main ()</div>
<div>#3 0x000000000042954c in start_child ()</div><div>#4 0x000000000042985c in mgt_run ()</div><div>#5 0x000000000043316a in main ()</div><div>(gdb) cont</div><div>Continuing.</div><div><br></div><div>Program received signal SIGQUIT, Quit.</div>
<div>[Switching to Thread 0x40800950 (LWP 25201)]</div><div>0x00002ac6f5153a51 in nanosleep () from /lib/libc.so.6</div><div>(gdb) exit</div><div>Undefined command: "exit". Try "help".</div><div>(gdb) quit</div>
<div>The program is running. Quit anyway (and detach it)? (y or n) y</div><div>Quitting: Can't detach LWP 25949: No such process</div><div><br></div><div><br></div><div><br></div><div>Any help is appreciated.</div><div>
<br></div><div>Tung</div><br><div class="gmail_quote">On Wed, Apr 1, 2009 at 7:47 AM, Tung Nguyen <span dir="ltr"><<a href="mailto:tnguyen@bleacherreport.com">tnguyen@bleacherreport.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Here's the full varnishstat -1<div><br></div><div><span style="font-family:Monaco;font-size:11px;line-height:14px"><pre style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em;font-family:'Bitstream Vera Sans Mono', Courier, monospace">
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">varnishstat -1</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">uptime 212 . Child uptime</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">client_conn 1518 7.16 Client connections accepted</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">client_req 5662 26.71 Client requests received</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">cache_hit 3230 15.24 Cache hits</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">cache_hitpass 8 0.04 Cache hits for pass</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">cache_miss 561 2.65 Cache misses</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">backend_conn 2307 10.88 Backend connections success</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">backend_unhealthy 0 0.00 Backend connections not attempted</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">backend_busy 0 0.00 Backend connections too many</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">backend_fail 0 0.00 Backend connections failures</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">backend_reuse 1920 9.06 Backend connections reuses</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">backend_recycle 2038 9.61 Backend connections recycles</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">backend_unused 0 0.00 Backend connections unused</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">n_srcaddr 365 . N struct srcaddr</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">n_srcaddr_act 0 . N active struct srcaddr</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">n_sess_mem 952 . N struct sess_mem</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">n_sess 1 . N struct sess</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">n_object 750 . N struct object</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">n_objecthead 1079 . N struct objecthead</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">n_smf 1272 . N struct smf</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">n_smf_frag 98 . N small free smf</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">n_smf_large 2 . N large free smf</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">n_vbe_conn 4 . N struct vbe_conn</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">n_bereq 88 . N struct bereq</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">n_wrk 1600 . N worker threads</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">n_wrk_create 1600 7.55 N worker threads created</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">n_wrk_failed 0 0.00 N worker threads not created</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">n_wrk_max 0 0.00 N worker threads limited</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">n_wrk_queue 0 0.00 N queued work requests</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">n_wrk_overflow 980 4.62 N overflowed work requests</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">n_wrk_drop 0 0.00 N dropped work requests</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">n_backend 6 . N backends</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">n_expired 54 . N expired objects</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">n_lru_nuked 0 . N LRU nuked objects</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">n_lru_saved 0 . N LRU saved objects</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">n_lru_moved 833 . N LRU moved objects</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">n_deathrow 0 . N objects on deathrow</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">losthdr 0 0.00 HTTP header overflows</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">n_objsendfile 0 0.00 Objects sent with sendfile</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">n_objwrite 4589 21.65 Objects sent with write</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">n_objoverflow 0 0.00 Objects overflowing workspace</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">s_sess 1517 7.16 Total Sessions</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">s_req 5662 26.71 Total Requests</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">s_pipe 0 0.00 Total pipe</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">s_pass 1871 8.83 Total pass</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">s_fetch 2307 10.88 Total fetch</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">s_hdrbytes 1673491 7893.83 Total header bytes</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">s_bodybytes 26560497 125285.36 Total body bytes</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">sess_closed 249 1.17 Session Closed</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">sess_pipeline 0 0.00 Session Pipeline</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">sess_readahead 0 0.00 Session Read Ahead</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">sess_linger 0 0.00 Session Linger</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">sess_herd 5428 25.60 Session herd</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">shm_records 308891 1457.03 SHM records</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">shm_writes 20426 96.35 SHM writes</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">shm_flushes 0 0.00 SHM flushes due to overflow</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">shm_cont 14 0.07 SHM MTX contention</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">shm_cycles 0 0.00 SHM cycles through buffer</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">sm_nreq 5117 24.14 allocator requests</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">sm_nobj 1172 . outstanding allocations</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">sm_balloc 10076160 . bytes allocated</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">sm_bfree 5983641600 . bytes free</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">sma_nreq 0 0.00 SMA allocator requests</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">sma_nobj 0 . SMA outstanding allocations</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">sma_nbytes 0 . SMA outstanding bytes</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">sma_balloc 0 . SMA bytes allocated</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">sma_bfree 0 . SMA bytes free</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">sms_nreq 125 0.59 SMS allocator requests</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">sms_nobj 0 . SMS outstanding allocations</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">sms_nbytes 0 . SMS outstanding bytes</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">sms_balloc 58250 . SMS bytes allocated</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">sms_bfree 58250 . SMS bytes freed</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">backend_req 2307 10.88 Backend requests made</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">n_vcl 1 0.00 N vcl total</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">n_vcl_avail 1 0.00 N vcl available</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">n_vcl_discard 0 0.00 N vcl discarded</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">n_purge 1 . N total active purges</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">n_purge_add 1 0.00 N new purges added</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">n_purge_retire 0 0.00 N old purges deleted</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">n_purge_obj_test 0 0.00 N objects tested</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">n_purge_re_test 0 0.00 N regexps tested against</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">n_purge_dups 0 0.00 N duplicate purges removed</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">hcb_nolock 0 0.00 HCB Lookups without lock</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">hcb_lock 0 0.00 HCB Lookups with lock</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">hcb_insert 0 0.00 HCB Inserts</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">esi_parse 0 0.00 Objects ESI parsed (unlock)</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">esi_errors 0 0.00 ESI parse errors (unlock)</div>
<font color="#888888"><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em"><br></div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
Tung</div></font></pre></span><div><div></div><div class="h5"><div><br><div class="gmail_quote">On Wed, Apr 1, 2009 at 7:24 AM, Tung Nguyen <span dir="ltr"><<a href="mailto:tnguyen@bleacherreport.com" target="_blank">tnguyen@bleacherreport.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><div>Hey guys,</div><div><br></div><div>Our requests seem to be getting timed out. I set all the timeout parameters pretty high, so wondering what could be causing it.</div><div><br></div><div>Also, how do I check out the number of current threads being used in varnishstat?</div>
<div><br></div><div>Thanks,</div><div>Tung</div><div><br></div><div># ps auxww | grep varnish</div><div>root 2649 0.0 2.4 101776 82744 pts/0 S< 13:48 0:00 varnishlog -w varnish.log</div><div>root 4301 0.0 0.0 110552 1136 ? S<s 14:05 0:00 /usr/sbin/varnishd -P /var/run/varnishd.pid -a :6081 -T localhost:6082 -p sess_timeout 25 -p obj_workspace 8192 -p sess_workspace 32768 -p listen_depth 8192 -p connect_timeout 3s -p thread_pool_min 100 -f /etc/varnish/default.vcl</div>
<div>nobody 4302 0.0 1.1 7781688 39468 ? S<l 14:05 0:00 /usr/sbin/varnishd -P /var/run/varnishd.pid -a :6081 -T localhost:6082 -p sess_timeout 25 -p obj_workspace 8192 -p sess_workspace 32768 -p listen_depth 8192 -p connect_timeout 3s -p thread_pool_min 100 -f /etc/varnish/default.vcl</div>
<div>root 4872 0.0 0.0 3876 556 pts/0 R<+ 14:22 0:00 grep --colour=auto varnish</div><div># </div><div><br></div><div><br></div><div><div> 0 VCL_call - prefetch</div><div> 0 VCL_return - fetch</div>
<div> 0 Debug - "Attempt Prefetch 1804102565"</div><div> 0 Backend_health - A3 Still healthy 4--X-S-RH 5 3 5 0.000000 0.000000 HTTP/1.1 200 OK</div><div> 0 Backend_health - A1 Still healthy 4--X-S-RH 5 3 5 0.000000 0.001500 HTTP/1.1 200 OK</div>
<div> 0 Backend_health - A6 Still healthy 4--X-S-RH 5 3 5 0.000000 0.000001 HTTP/1.1 200 OK</div><div> 0 Backend_health - A2 Still healthy 4--X-S-RH 5 3 5 0.000000 0.000000 HTTP/1.1 200 OK</div><div> 0 Backend_health - A4 Still healthy 4--X-S-RH 5 3 5 0.000000 0.000042 HTTP/1.1 200 OK</div>
<div> 0 ExpPick - 1804096341 ttl</div><div> 0 VCL_call - timeout</div><div> 0 VCL_return - discard</div><div> 0 ExpKill - 1804096341 -30</div><div> 0 ExpPick - 1804096350 ttl</div><div>
0 VCL_call - timeout</div><div> 0 VCL_return - discard</div><div> 0 ExpKill - 1804096350 -30</div><div> 0 ExpPick - 1804096354 ttl</div><div> 0 VCL_call - timeout</div><div> 0 VCL_return - discard</div>
<div> 0 ExpKill - 1804096354 -30</div><div> 0 ExpPick - 1804096376 ttl</div><div> 0 VCL_call - timeout</div><div> 0 VCL_return - discard</div><div> 0 ExpKill - 1804096376 -30</div><div>
0 ExpPick - 1804096384 ttl</div><div> 0 VCL_call - timeout</div><div> 0 VCL_return - discard</div><div> 0 ExpKill - 1804096384 -30</div><div> 0 CLI - Rd ping</div><div> 0 CLI - Wr 0 200 PONG 1238593700 1.0</div>
<div> 0 Backend_health - A5 Still healthy 4--X-S-RH 5 3 5 0.000000 0.000000 HTTP/1.1 200 OK</div><div> 0 ExpPick - 1804102829 prefetch</div><div> 0 VCL_call - prefetch</div><div> 0 VCL_return - fetch</div>
<div> 0 Debug - "Attempt Prefetch 1804102829"</div><div> 0 Backend_health - A3 Still healthy 4--X-S-RH 5 3 5 0.000000 0.000000 HTTP/1.1 200 OK</div><div> 0 Backend_health - A1 Still healthy 4--X-S-RH 5 3 5 0.000000 0.001125 HTTP/1.1 200 OK</div>
<div> 0 Backend_health - A6 Still healthy 4--X-S-RH 5 3 5 0.000000 0.000001 HTTP/1.1 200 OK</div><div> 0 Backend_health - A2 Still healthy 4--X-S-RH 5 3 5 0.000000 0.000000 HTTP/1.1 200 OK</div><div> 0 Backend_health - A4 Still healthy 4--X-S-RH 5 3 5 0.000000 0.000032 HTTP/1.1 200 OK</div>
<div> 0 CLI - Rd ping</div><div> 0 CLI - Wr 0 200 PONG 1238593703 1.0</div><div> 0 ExpPick - 1803925354 ttl</div><div> 0 VCL_call - timeout</div><div><br></div><div><br></div></div>
</div>
<div><span style="font-family:Monaco;font-size:11px;line-height:14px"><pre style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em;font-family:'Bitstream Vera Sans Mono', Courier, monospace">
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">param.show</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">200 2137 </div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">accept_fd_holdoff 50 [ms]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">acceptor default (epoll, poll)</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">auto_restart on [bool]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">backend_http11 on [bool]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">between_bytes_timeout 60.000000 [s]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">cache_vbe_conns off [bool]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">cc_command "exec cc -fpic -shared -Wl,-x -o %o %s"</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">cli_buffer 8192 [bytes]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">cli_timeout 5 [seconds]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">client_http11 off [bool]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">clock_skew 10 [s]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">connect_timeout 1.000000 [s]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">default_grace 10</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">default_ttl 120 [seconds]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">diag_bitmap 0x0 [bitmap]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">err_ttl 0 [seconds]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">esi_syntax 0 [bitmap]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">fetch_chunksize 128 [kilobytes]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">first_byte_timeout 60.000000 [s]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">group nobody (65534)</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">listen_address :6081</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">listen_depth 8192 [connections]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">log_hashstring off [bool]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">log_local_address off [bool]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">lru_interval 2 [seconds]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">max_esi_includes 5 [includes]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">max_restarts 4 [restarts]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">obj_workspace 8192 [bytes]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">overflow_max 100 [%]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">ping_interval 3 [seconds]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">pipe_timeout 60 [seconds]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">prefer_ipv6 off [bool]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">purge_dups off [bool]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">purge_hash on [bool]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">rush_exponent 3 [requests per request]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">send_timeout 600 [seconds]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">sess_timeout 10 [seconds]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">sess_workspace 32768 [bytes]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">session_linger 0 [ms]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">shm_reclen 255 [bytes]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">shm_workspace 8192 [bytes]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">srcaddr_hash 1049 [buckets]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">srcaddr_ttl 30 [seconds]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">thread_pool_add_delay 20 [milliseconds]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">thread_pool_add_threshold 2 [requests]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">thread_pool_fail_delay 200 [milliseconds]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">thread_pool_max 500 [threads]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">thread_pool_min 100 [threads]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">thread_pool_purge_delay 1000 [milliseconds]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">thread_pool_timeout 300 [seconds]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">thread_pools 2 [pools]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">user nobody (65534)</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">vcl_trace off [bool]</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em"> </div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
<br></div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em"><br>
</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em"><span style="font-family:Monaco;font-size:11px;line-height:14px;white-space:normal"><pre style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;line-height:1.4em;font-family:'Bitstream Vera Sans Mono', Courier, monospace">
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">0+00:01:23 ey03-s00419</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">Hitrate ratio: 4 4 4</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">Hitrate avg: 0.7154 0.7154 0.7154</div>
<div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em"> </div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
2111 0.00 25.43 Client connections accepted</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
8315 0.00 100.18 Client requests received</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
5019 0.00 60.47 Cache hits</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
6 0.00 0.07 Cache hits for pass</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
805 0.00 9.70 Cache misses</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
2848 0.00 34.31 Backend connections success</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
2471 0.00 29.77 Backend connections reuses</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
2691 0.00 32.42 Backend connections recycles</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
456 . . N struct srcaddr</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
27 . . N active struct srcaddr</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
995 . . N struct sess_mem</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
55 . . N struct sess</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
584 . . N struct object</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
670 . . N struct objecthead</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
1147 . . N struct smf</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
1 . . N small free smf</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
1 . . N large free smf</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
6 . . N struct vbe_conn</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
22 . . N struct bereq</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
200 . . N worker threads</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
200 0.00 2.41 N worker threads created</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
57 0.00 0.69 N overflowed work requests</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
6 . . N backends</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
2046 . . N LRU moved objects</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
6277 0.00 75.63 Objects sent with write</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
2110 0.00 25.42 Total Sessions</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
8314 0.00 100.17 Total Requests</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
2491 0.00 30.01 Total pass</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
2847 0.00 34.30 Total fetch</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
2410204 0.00 29038.60 Total header bytes</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
38160409 0.00 459763.96 Total body bytes</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
605 0.00 7.29 Session Closed</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
2 0.00 0.02 Session Pipeline</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
6 0.00 0.07 Session Read Ahead</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
7734 0.00 93.18 Session herd</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
428358 12.00 5160.94 SHM records</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
26190 12.00 315.54 SHM writes</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
2 0.00 0.02 SHM MTX contention</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
6378 0.00 76.84 allocator requests</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
1145 . . outstanding allocations</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
12103680 . . bytes allocated</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
6028206080 . . bytes free</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
448 0.00 5.40 SMS allocator requests</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
208768 . . SMS bytes allocated</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
208768 . . SMS bytes freed</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
2848 0.00 34.31 Backend requests made</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
1 0.00 0.01 N vcl total</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
1 0.00 0.01 N vcl available</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
1 . . N total active purges</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
1 0.00 0.01 N new purges added</div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
</div></pre></span></div><div style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:1em;line-height:1.4em">
<br></div></pre></span></div><div>Thanks,</div>Tung
</blockquote></div><br><br clear="all"><br><br>
</div></div></div></div>
</blockquote></div><br><br clear="all"><br><br>
</div>