<div dir="ltr">I took a look at apache on the host that Varnish reports as sick:<div><br></div><div>And apache seems to be running fine on the 'sick' host:</div><div><br></div><div>
<p class="">[root@beta:/var/www/jf-current] #apachectl -S</p>
<p class="">VirtualHost configuration:</p>
<p class="">wildcard NameVirtualHosts and _default_ servers:</p>
<p class="">*:443 is a NameVirtualHost</p>
<p class=""> default server <a href="http://beta.mywebsite.com">beta.mywebsite.com</a> (/etc/httpd/conf.d/002_jf_beta_ssl.conf:78)</p>
<p class=""> port 443 namevhost <a href="http://beta.mywebsite.com">beta.mywebsite.com</a> (/etc/httpd/conf.d/002_jf_beta_ssl.conf:78)</p>
<p class="">*:80 is a NameVirtualHost</p>
<p class=""> default server <a href="http://ref.mywebsite.com">ref.mywebsite.com</a> (/etc/httpd/conf.d/001_ref.mywebsite.com.conf:1)</p>
<p class=""> port 80 namevhost <a href="http://ref.mywebsite.com">ref.mywebsite.com</a> (/etc/httpd/conf.d/001_ref.mywebsite.com.conf:1)</p>
<p class=""> port 80 namevhost <a href="http://qa.mywebsite.com">qa.mywebsite.com</a> (/etc/httpd/conf.d/002_qa.mywebsite.com.conf:1)</p>
<p class=""> port 80 namevhost <a href="http://beta.mywebsite.com">beta.mywebsite.com</a> (/etc/httpd/conf.d/003_beta.mywebsite.com.conf:1)</p>
<p class=""> port 80 namevhost <a href="http://beta-test.mywebsite.com">beta-test.mywebsite.com</a> (/etc/httpd/conf.d/004_beta-test.mywebsite.com.conf:1)</p>
<p class=""> port 80 namevhost <a href="http://admin.mywebsite.com">admin.mywebsite.com</a> (/etc/httpd/conf.d/005_admin.mywebsite.com.conf:1)</p>
<p class=""> port 80 namevhost <a href="http://admin.mywebsite.com">admin.mywebsite.com</a> (/etc/httpd/conf.d/10_admin.mywebsite.com.conf:1)</p>
<p class=""> port 80 namevhost <a href="http://beta-test.mywebsite.com">beta-test.mywebsite.com</a> (/etc/httpd/conf.d/10_beta-test.mywebsite.com.conf:1)</p>
<p class=""> port 80 namevhost <a href="http://beta.mywebsite.com">beta.mywebsite.com</a> (/etc/httpd/conf.d/10_beta.mywebsite.com.conf:1)</p>
<p class=""> port 80 namevhost <a href="http://qa.mywebsite.com">qa.mywebsite.com</a> (/etc/httpd/conf.d/10_qa.mywebsite.com.conf:1)</p>
<p class=""> port 80 namevhost <a href="http://ref.mywebsite.com">ref.mywebsite.com</a> (/etc/httpd/conf.d/10_ref.mywebsite.com.conf:1)</p>
<p class="">Syntax OK</p><p class=""><br></p><p class="">My probe and host definitions in the varnish default.vcl looks like this:</p><p class=""><br></p><p class="">probe favicon {</p><p class=""> .url = "/favicon.ico";</p>
<p class=""> .timeout = 34ms;</p><p class=""> .interval = 1s;</p><p class=""> .window = 10;</p><p class=""> .threshold = 8;</p><p class="">}</p><p class=""><br></p><p class="">backend web1 {</p><p class=""> .host = "10.10.1.94";</p>
<p class=""> .port = "80";</p><p class=""> .probe = favicon;</p><p class="">}</p><p class=""><br></p><p class="">backend web2 {</p><p class=""><br></p><p class=""> .host = "10.10.1.98";</p><p class="">
.port = "80";</p><p class=""> .probe = favicon;</p><p class="">}</p><p class="">director www random {<br></p><p class=""> { .backend = web1 ; .weight = 2; }</p><p class=""> { .backend = web2 ; .weight = 2; }</p>
<p class="">
</p><p class=""> }</p></div><div class="gmail_extra">I was just experimenting the load balancing algorithms, but that probably doesn't have much bearing on this problem.</div><div class="gmail_extra"><br></div><div class="gmail_extra">
>From the varnish host I can do a wget for the probe file from the host that varnish is marking as 'sick'</div><div class="gmail_extra"><br></div><div class="gmail_extra">
<p class="">[root@varnish1:/etc/varnish] #wget -O /dev/null -S <a href="http://web2.mywebsite.com/favicon.ico">http://web2.mywebsite.com/favicon.ico</a></p>
<p class="">--2014-05-11 20:48:23-- <a href="http://web1.mywebsite.com/favicon.ico">http://web1.mywebsite.com/favicon.ico</a></p>
<p class="">Resolving web1.mywebsite.com... 10.10.1.98</p>
<p class="">Connecting to <a href="http://web1.mywebsite.com">web1.mywebsite.com</a>|10.10.1.98|:80... connected.</p>
<p class="">HTTP request sent, awaiting response...</p>
<p class=""> HTTP/1.1 200 OK</p>
<p class=""> Date: Mon, 12 May 2014 00:48:25 GMT</p>
<p class=""> Server: Apache/2.2.23 (CentOS)</p>
<p class=""> Last-Modified: Sun, 22 Dec 2013 00:53:19 GMT</p>
<p class=""> ETag: "2a8003-47e-4ee14efeebdc0"</p>
<p class=""> Accept-Ranges: bytes</p>
<p class=""> Content-Length: 1150</p>
<p class=""> Keep-Alive: timeout=5, max=100</p>
<p class=""> Connection: Keep-Alive</p>
<p class=""> Content-Type: text/plain; charset=UTF-8</p>
<p class="">Length: 1150 (1.1K) [text/plain]</p>
<p class="">Saving to: “/dev/null”</p>
<p class="">100%[================================================================>] 1,150 --.-K/s in 0s</p>
<p class="">2014-05-11 20:48:24 (144 MB/s) - “/dev/null” saved [1150/1150]</p><p class=""><br></p><p class="">I'm attempting here to approximate the calls that varnish is making to determine if a host is healthy.</p>
<p class="">And yet when I try having a look at debug.health from the command line Vaarnish is still labeling it as 'sick'</p><p class="">[root@varnish1:~] #varnishadm -T <a href="http://127.0.0.1:6082">127.0.0.1:6082</a> debug.health -S /etc/varnish/secret</p>
<p class=""><b>Backend web1 is Healthy</b></p><p class="">Current states good: 10 threshold: 8 window: 10</p><p class="">Average responsetime of good probes: 0.001247</p><p class="">Oldest Newest</p>
<p class="">================================================================</p><p class="">4444444444444444444444444444444444444444444444444444444444444444 Good IPv4</p><p class="">XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Good Xmit</p>
<p class="">RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR Good Recv</p><p class="">HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH Happy</p><p class=""><b>Backend web2 is Sick</b></p><p class="">
Current states good: 0 threshold: 8 window: 10</p><p class="">Average responsetime of good probes: 0.000000</p><p class="">Oldest Newest</p><p class="">================================================================</p>
<p class="">---------------------------------------------------------------- Happy</p><p class="">So I'm really wondering if I'm missing something here that'll help me to determine why Varnish thinks this host is sick! Also I have no problem browsing the custom URLs that I setup on each host to indicate which host apache is running on.</p>
<p class=""><br></p><p class="">Thanks</p><p class="">Tim</p><p class=""><br></p><br><div class="gmail_quote">On Sat, May 10, 2014 at 12:58 AM, Tim Dunphy <span dir="ltr"><<a href="mailto:bluethundr@gmail.com" target="_blank">bluethundr@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Hey all,<div><br></div><div> I have two web backends in my varnish config. And one node is reporting healthy and the other is being reported as 'sick'.</div>
<div><br></div><div><div>10 Backend c 11 www web1</div>
<div> 0 Backend_health - web1 Still healthy 4--X-RH 5 3 5 0.001130 0.001067 HTTP/1.1 200 OK</div><div> 0 Backend_health - web1 Still healthy 4--X-RH 5 3 5 0.001231 0.001108 HTTP/1.1 200 OK</div><div> 0 Backend_health - web1 Still healthy 4--X-RH 5 3 5 0.001250 0.001143 HTTP/1.1 200 OK</div>
<div> 0 Backend_health - web1 Still healthy 4--X-RH 5 3 5 0.001127 0.001139 HTTP/1.1 200 OK</div><div> 0 Backend_health - web1 Still healthy 4--X-RH 5 3 5 0.001208 0.001157 HTTP/1.1 200 OK</div><div> 0 Backend_health - web1 Still healthy 4--X-RH 5 3 5 0.001562 0.001258 HTTP/1.1 200 OK</div>
<div> 0 Backend_health - web1 Still healthy 4--X-RH 5 3 5 0.001545 0.001330 HTTP/1.1 200 OK</div><div> 0 Backend_health - web1 Still healthy 4--X-RH 5 3 5 0.001363 0.001338 HTTP/1.1 200 OK</div><div> 11 BackendClose b web1</div>
<div><br></div><div><div>[root@varnish1:/etc/varnish] #varnishlog | grep web2</div><div> 0 Backend_health - web2 Still sick 4--X--- 0 3 5 0.000000 0.000000</div><div> 0 Backend_health - web2 Still sick 4--X--- 0 3 5 0.000000 0.000000</div>
<div> 0 Backend_health - web2 Still sick 4--X--- 0 3 5 0.000000 0.000000</div><div> 0 Backend_health - web2 Still sick 4--X--- 0 3 5 0.000000 0.000000</div><div> 0 Backend_health - web2 Still sick 4--X--- 0 3 5 0.000000 0.000000</div>
<div> 0 Backend_health - web2 Still sick 4--X--- 0 3 5 0.000000 0.000000</div><div> 0 Backend_health - web2 Still sick 4--X--- 0 3 5 0.000000 0.000000</div><div> 0 Backend_health - web2 Still sick 4--X--- 0 3 5 0.000000 0.000000</div>
<div> 0 Backend_health - web2 Still sick 4--X--- 0 3 5 0.000000 0.000000</div></div><div><br></div><div>And I'm really at a loss to understand why. Both nodes should be completely identical. And the web roots on both are basically svn repos that are in sync. </div>
<div><br></div><div>From web1 :</div><div><br></div><div><div>[root@beta:/var/www/jf-current] #svn info | grep -i revision</div><div>Revision: 17</div></div><div><br></div><div>To web2:</div><div><br></div><div><div>[root@beta-new:/var/www/jf-current] #svn info | grep -i revision</div>
<div>Revision: 17</div></div><div><br></div><div>This is the part of my vcl file where I define the web back ends:</div><div><br></div><div><div>probe favicon {</div><div> .url = "/favicon.ico";</div><div> .timeout = 60ms;</div>
<div> .interval = 2s;</div><div> .window = 5;</div><div> .threshold = 3;</div><div>}</div><div><br></div><div>backend web1 {</div><div> .host = "xx.xx.xx.xx";</div><div> .port = "80";</div><div>
.probe = favicon;</div>
<div>}</div><div><br></div><div>backend web2 {</div><div><br></div><div> .host = "xx.xx.xx.xx";</div><div> .port = "80";</div><div> .probe = favicon;</div><div>}</div></div><div><br></div><div>And the file that varnish is probing for is present on both:</div>
<div><br></div><div><div>[root@beta:/var/www/jf-current] #ls -l /var/www/jf-current/favicon.ico</div><div>-rwxrwxr-x 1 apache ftp 1150 Dec 22 00:53 /var/www/jf-current/favicon.ico</div></div><div><br></div><div>I've also setup individual web URLs for each host that isn't cached in varnish so I can hit each one. And each site comes up ok. So I'm a little puzzled as to why the second web host is reporting 'sick' and what I can do to get it back into load balancing. </div>
<div><br></div><div>Thanks for any help you can provide!</div><span class=""><font color="#888888"><div><br></div><div>Tim</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div>-- <br>GPG me!!<br>
<br>gpg --keyserver <a href="http://pool.sks-keyservers.net" target="_blank">pool.sks-keyservers.net</a> --recv-keys F186197B<br>
<br>
</font></span></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br>GPG me!!<br><br>gpg --keyserver <a href="http://pool.sks-keyservers.net" target="_blank">pool.sks-keyservers.net</a> --recv-keys F186197B<br><br>
</div></div>