<div>Hi guys,</div><div><br></div><div>We're on varnish 2.0.3</div><div><br></div><div>It looks like varnish restarts the child process for us every so often, causing 503s :(. Was wondering if this is a known issue.</div>
<div><br></div><div><br></div><div><br></div><div>######################################################################</div><div> # ps auxww | grep varnishd</div><div>nobody 3385 0.0 0.3 8098024 7560 ? Sl 19:49 0:00 /usr/sbin/varnishd -P /var/run/varnishd.pid -a :6081 -T localhost:6082 -p sess_timeout 10 -p obj_workspace 8192 -p sess_workspace 32768 -p listen_depth 8192 -p connect_timeout 1s -p thread_pool_min 100 -f /etc/varnish/default.vcl</div>
<div>root 3938 0.0 0.0 111584 904 ? Ss Apr03 0:01 /usr/sbin/varnishd -P /var/run/varnishd.pid -a :6081 -T localhost:6082 -p sess_timeout 10 -p obj_workspace 8192 -p sess_workspace 32768 -p listen_depth 8192 -p connect_timeout 1s -p thread_pool_min 100 -f /etc/varnish/default.vcl</div>
<div>root 3961 0.0 0.0 3884 672 pts/2 S<+ 19:55 0:00 grep --colour=auto varnishd</div><div> # </div><div><br></div><div><br></div><div><br></div><div>######################################################################</div>
<div> # varnishstat -1</div><div>uptime 472 . Child uptime</div><div>client_conn 40 0.08 Client connections accepted</div><div>client_req 100 0.21 Client requests received</div>
<div>cache_hit 1 0.00 Cache hits</div><div>cache_hitpass 0 0.00 Cache hits for pass</div><div>cache_miss 23 0.05 Cache misses</div><div>backend_conn 99 0.21 Backend connections success</div>
<div>backend_unhealthy 0 0.00 Backend connections not attempted</div><div>backend_busy 0 0.00 Backend connections too many</div><div>backend_fail 0 0.00 Backend connections failures</div>
<div>backend_reuse 82 0.17 Backend connections reuses</div><div>backend_recycle 99 0.21 Backend connections recycles</div><div>backend_unused 0 0.00 Backend connections unused</div>
<div>n_srcaddr 1 . N struct srcaddr</div><div>n_srcaddr_act 0 . N active struct srcaddr</div><div>n_sess_mem 13 . N struct sess_mem</div><div>
n_sess 1 . N struct sess</div><div>n_object 23 . N struct object</div><div>n_objecthead 29 . N struct objecthead</div><div>n_smf 47 . N struct smf</div>
<div>n_smf_frag 0 . N small free smf</div><div>n_smf_large 1 . N large free smf</div><div>n_vbe_conn 6 . N struct vbe_conn</div><div>n_bereq 6 . N struct bereq</div>
<div>n_wrk 200 . N worker threads</div><div>n_wrk_create 200 0.42 N worker threads created</div><div>n_wrk_failed 0 0.00 N worker threads not created</div>
<div>n_wrk_max 13326 28.23 N worker threads limited</div><div>n_wrk_queue 0 0.00 N queued work requests</div><div>n_wrk_overflow 0 0.00 N overflowed work requests</div>
<div>n_wrk_drop 0 0.00 N dropped work requests</div><div>n_backend 2 . N backends</div><div>n_expired 0 . N expired objects</div><div>n_lru_nuked 0 . N LRU nuked objects</div>
<div>n_lru_saved 0 . N LRU saved objects</div><div>n_lru_moved 1 . N LRU moved objects</div><div>n_deathrow 0 . N objects on deathrow</div>
<div>losthdr 0 0.00 HTTP header overflows</div><div>n_objsendfile 0 0.00 Objects sent with sendfile</div><div>n_objwrite 65 0.14 Objects sent with write</div>
<div>n_objoverflow 0 0.00 Objects overflowing workspace</div><div>s_sess 40 0.08 Total Sessions</div><div>s_req 100 0.21 Total Requests</div><div>
s_pipe 0 0.00 Total pipe</div><div>s_pass 76 0.16 Total pass</div><div>s_fetch 99 0.21 Total fetch</div><div>s_hdrbytes 35654 75.54 Total header bytes</div>
<div>s_bodybytes 206284 437.04 Total body bytes</div><div>sess_closed 0 0.00 Session Closed</div><div>sess_pipeline 0 0.00 Session Pipeline</div><div>sess_readahead 0 0.00 Session Read Ahead</div>
<div>sess_linger 0 0.00 Session Linger</div><div>sess_herd 100 0.21 Session herd</div><div>shm_records 9594 20.33 SHM records</div><div>shm_writes 1884 3.99 SHM writes</div>
<div>shm_flushes 0 0.00 SHM flushes due to overflow</div><div>shm_cont 10 0.02 SHM MTX contention</div><div>shm_cycles 0 0.00 SHM cycles through buffer</div>
<div>sm_nreq 186 0.39 allocator requests</div><div>sm_nobj 46 . outstanding allocations</div><div>sm_balloc 425984 . bytes allocated</div><div>
sm_bfree 6431973376 . bytes free</div><div>sma_nreq 0 0.00 SMA allocator requests</div><div>sma_nobj 0 . SMA outstanding allocations</div><div>
sma_nbytes 0 . SMA outstanding bytes</div><div>sma_balloc 0 . SMA bytes allocated</div><div>sma_bfree 0 . SMA bytes free</div><div>sms_nreq 0 0.00 SMS allocator requests</div>
<div>sms_nobj 0 . SMS outstanding allocations</div><div>sms_nbytes 0 . SMS outstanding bytes</div><div>sms_balloc 0 . SMS bytes allocated</div>
<div>sms_bfree 0 . SMS bytes freed</div><div>backend_req 99 0.21 Backend requests made</div><div>n_vcl 1 0.00 N vcl total</div><div>n_vcl_avail 1 0.00 N vcl available</div>
<div>n_vcl_discard 0 0.00 N vcl discarded</div><div>n_purge 1 . N total active purges</div><div>n_purge_add 1 0.00 N new purges added</div><div>
n_purge_retire 0 0.00 N old purges deleted</div><div>n_purge_obj_test 0 0.00 N objects tested</div><div>n_purge_re_test 0 0.00 N regexps tested against</div><div>
n_purge_dups 0 0.00 N duplicate purges removed</div><div>hcb_nolock 0 0.00 HCB Lookups without lock</div><div>hcb_lock 0 0.00 HCB Lookups with lock</div>
<div>hcb_insert 0 0.00 HCB Inserts</div><div>esi_parse 0 0.00 Objects ESI parsed (unlock)</div><div>esi_errors 0 0.00 ESI parse errors (unlock)</div>
<div><br></div><div><br></div><div><br></div><div>######################################################################</div><div># I can see that the threads are starting back up in varnishlog</div><div><div>######################################################################</div>
<div><br></div></div><div>8 RxHeader c Accept-Language: en-us,en;q=0.5</div><div> 8 RxHeader c Accept-Encoding: gzip,deflate</div><div> 8 RxHeader c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7</div><div> 8 RxHeader c Keep-Alive: 300</div>
<div> 8 RxHeader c Connection: keep-alive</div><div> 8 RxHeader c Referer: <a href="http://stagingbleacherreport.com/nascar">http://stagingbleacherreport.com/nascar</a></div><div> 8 RxHeader c Cookie: __utma=58205160.3272659617253579300.1238530464.1239125976.1239133102.21; __utmz=58205160.1238530464.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __qca=1238530467-23581915-71188578; auth_token=4c078f7b1d3097b170260c6978c9862c; pod_151314=Mon+</div>
<div> 8 RxHeader c If-None-Match: "004e69938cfe6ed16863cc156a0e9b34"</div><div> 8 VCL_call c recv</div><div> 8 VCL_return c pass</div><div> 8 VCL_call c pass</div><div> 8 VCL_return c pass</div><div>
8 VCL_call c error</div><div> 8 VCL_return c deliver</div><div> 8 Length c 465</div><div> 8 VCL_call c deliver</div><div> 8 VCL_return c deliver</div><div> 8 TxProtocol c HTTP/1.1</div><div> 8 TxStatus c 503</div>
<div> 8 TxResponse c Service Unavailable</div><div> 8 TxHeader c Server: Varnish</div><div> 8 TxHeader c Retry-After: 0</div><div> 8 TxHeader c Content-Type: text/html; charset=utf-8</div><div> 8 TxHeader c Content-Length: 465</div>
<div> 8 TxHeader c Date: Tue, 07 Apr 2009 19:49:47 GMT</div><div> 8 TxHeader c X-Varnish: 171071333</div><div> 8 TxHeader c Age: 0</div><div> 8 TxHeader c Via: 1.1 varnish</div><div> 8 TxHeader c Connection: close</div>
<div> 8 TxHeader c X-Cache: MISS</div><div> 8 ReqEnd c 171071333 1239133787.063603163 1239133787.063603163 0.000000000 0.000000000 0.000000000</div><div> 8 SessionClose c error</div><div> 8 StatSess c 67.164.99.232 59479 0 1 1 0 1 0 249 465</div>
<div> 0 StatAddr - 67.164.99.232 0 0 1 1 0 1 0 249 465</div><div> 0 WorkThread - 0x55028bd0 start</div><div> 0 WorkThread - 0x55829bd0 start</div><div> 0 WorkThread - 0x5602abd0 start</div><div> 0 WorkThread - 0x5682bbd0 start</div>
<div> 0 WorkThread - 0x5702cbd0 start</div><div> 0 WorkThread - 0x5782dbd0 start</div><div> 0 WorkThread - 0x5802ebd0 start</div><div> 0 WorkThread - 0x5882fbd0 start</div><div> 0 WorkThread - 0x59030bd0 start</div>
<div> 0 WorkThread - 0x59831bd0 start</div><div> 0 WorkThread - 0x5a032bd0 start</div><div> 0 WorkThread - 0x5a833bd0 start</div><div> 0 WorkThread - 0x5b034bd0 start</div><div> 0 WorkThread - 0x5b835bd0 start</div>
<div> 0 WorkThread - 0x5c036bd0 start</div><div> 0 WorkThread - 0x5c837bd0 start</div><div> 0 WorkThread - 0x5d038bd0 start</div><div> 0 Backend_health - A1 Back healthy 4--X-S-RH 2 2 5 0.000000 0.000000 HTTP/1.1 200 OK</div>
<div> 0 Backend_health - A2 Back healthy 4--X-S-RH 2 2 5 0.000000 0.000000 HTTP/1.1 200 OK</div><div> 0 WorkThread - 0x5d839bd0 start</div><div> 0 WorkThread - 0x5e03abd0 start</div><div> 0 WorkThread - 0x5e83bbd0 start</div>
<div> 0 WorkThread - 0x5f03cbd0 start</div><div> 0 WorkThread - 0x5f83dbd0 start</div><div> 0 WorkThread - 0x6003ebd0 start</div><div> 0 WorkThread - 0x6083fbd0 start</div><div> 0 WorkThread - 0x61040bd0 start</div>
<div><br></div><div>#</div><div><br></div><div><div><div>######################################################################</div><div># heres are default.vcl</div><div><div>cat /etc/varnish/default.vcl</div><div>backend A1 {</div>
<div> .host = "ey03-s00344";</div><div> .port = "80";</div><div> .probe = {</div><div> .interval = 1s;</div><div> .timeout = 3 s;</div><div> .window = 5;</div><div> .threshold = 2;</div>
<div> .url = "<a href="http://stagingbleacherreport.com/images/blank.gif">http://stagingbleacherreport.com/images/blank.gif</a>";</div><div> }</div><div>}</div><div><br></div><div>backend A2 {</div><div>
.host = "ey03-s00344";</div><div> .port = "80";</div><div> .probe = {</div><div> .interval = 1s;</div><div> .timeout = 3 s;</div><div> .window = 5;</div><div> .threshold = 2;</div>
<div> .url = "<a href="http://stagingbleacherreport.com/images/blank.gif">http://stagingbleacherreport.com/images/blank.gif</a>";</div><div> }</div><div>}</div><div><br></div><div>director www_director round-robin {</div>
<div> { .backend = A1; }</div><div> { .backend = A2; }</div><div>}</div><div><br></div><div><br></div><div>acl local {</div><div> "localhost";</div><div>}</div><div><br></div><div>C{ #include <syslog.h> }C</div>
<div>sub vcl_recv {</div><div><br></div><div> C{ syslog(LOG_INFO, "@@@@ vcl_recv."); }C</div><div> C{ syslog(LOG_INFO, VRT_r_req_url(sp)); }C</div><div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span></div>
<div> set req.backend = www_director ;</div><div><br></div><div> # Add a unique header containing the client address</div><div> remove req.http.X-Forwarded-For;</div><div> set req.http.X-Forwarded-For = client.ip;</div>
<div><br></div><div> /* clear cookies on feeds */</div><div> if (req.url ~ "^/articles[;/]feed") {</div><div> C{ syslog(LOG_INFO, "@@@@ feed."); }C</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>unset req.http.Cookie;</div>
<div> lookup;</div><div> }</div><div><br></div><div> # cache assets</div><div> if (req.url ~ "^/images/" || req.url ~ "^/javascripts" || req.url ~ "^/stylesheets") {</div><div>
C{ syslog(LOG_INFO, "@@@@ assets."); }C</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>unset req.http.Cookie;</div><div> lookup;</div><div> }</div><div><br></div><div> # do not cache images_root</div>
<div> if (req.url ~ "^/images_root") {</div><div> C{ syslog(LOG_INFO, "@@@@ images_root."); }C</div><div> pass;</div><div> }</div><div><br></div><div> if (req.request == "PURGE") {</div>
<div> if (!client.ip ~ local) {</div><div> error 405 "Not allowed.";</div><div> }</div><div> lookup;</div><div> }</div><div> C{ syslog(LOG_INFO, "@@@@ fallthrough."); }C</div>
<div>}</div><div><br></div><div>#sub vcl_pipe {</div><div># # If we don't set the Connection: close header, any following</div><div># # requests from the client will also be piped through and</div><div># # left untouched by varnish. We don't want that.</div>
<div># set bereq.http.connection = "close";</div><div>#}</div><div><br></div><div>sub vcl_fetch {</div><div> if (req.url ~ "^/articles[;/]feed") {</div><div> unset obj.http.Set-Cookie;</div>
<div> unset obj.http.expires;</div><div> set obj.ttl = 60m;</div><div> deliver;</div><div> }</div><div><br></div><div> if (req.url ~ "^/images/" || req.url ~ "^/javascripts" || req.url ~ "^/stylesheets") {</div>
<div> unset obj.http.Set-Cookie;</div><div> set obj.ttl = 30m;</div><div> }</div><div><br></div><div> # similar to what twitter uses</div><div> if (!obj.cacheable) {</div><div> set obj.http.X-Cacheable = "NO:Not-Cacheable";</div>
<div> pass;</div><div> }</div><div> if (obj.http.Cache-Control ~ "private") {</div><div> if(req.http.Cookie ~"(UserID|_session)") {</div><div> set obj.http.X-Cacheable = "NO:Got Session";</div>
<div> } else {</div><div> set obj.http.X-Cacheable = "NO:Cache-Control=private";</div><div> }</div><div> pass;</div><div> }</div><div> if (obj.http.Set-Cookie ~ "(UserID|_session)") {</div>
<div> set obj.http.X-Cacheable = "NO:Set-Cookie";</div><div> pass;</div><div> }</div><div><br></div><div> set obj.grace = 30s; # only one slow requests during cache misses</div><div>
}</div><div><br></div><div>sub vcl_hit {</div><div> if (req.request == "PURGE") {</div><div> set obj.ttl = 0s;</div><div> error 200 "Purged.";</div><div> }</div>
<div>}</div><div><br></div><div>sub vcl_miss {</div><div> if (req.request == "PURGE") {</div><div> error 404 "Not in cache.";</div><div> }</div><div>}</div><div><br></div><div>
sub vcl_deliver {</div><div> if (obj.hits > 0) {</div><div> set resp.http.X-Cache = "HIT";</div><div> } else {</div><div> set resp.http.X-Cache = "MISS";</div>
<div> }</div><div>}</div><div><br></div></div></div></div><div><br></div><div><br></div><div>Thanks,</div><div>Tung</div>