<div dir="ltr">Hey Jason,<div><br></div><div>Really sorry I have no idea how this happened. But it must have been some very strange copy-paste error that caused what you saw before. In reality I did a wget to web2 and got a response from web2. </div>
<div><br></div><div>I grabbed the source text that I created before and checked it for inconstancies and this is what that post should have looked like. </div><div><br></div><div>Would you or someone on the list care to read over it and try to offer a solution? I must admit I'm tapped for ideas here. </div>
<div><br></div><div>But before we retread the following (corrected) info, I've provided a wget to web1 (the good back end) and web2 (the bad back end) by IP:</div><div><br></div><div>Starting with the bad (sick) back end which is web2 by IP:</div>
<div><br></div><div><div>[root@varnish1:~] #wget -O /dev/null -S <a href="http://10.10.1.98/favicon.ico">http://10.10.1.98/favicon.ico</a></div><div>--2014-05-12 21:50:55-- <a href="http://166.78.8.98/favicon.ico">http://166.78.8.98/favicon.ico</a></div>
<div>Connecting to 166.78.8.98:80... connected.</div><div>HTTP request sent, awaiting response...</div><div> HTTP/1.1 200 OK</div><div> Date: Tue, 13 May 2014 01:50:56 GMT</div><div> Server: Apache/2.2.23 (CentOS)</div>
<div> Last-Modified: Sat, 10 May 2014 05:06:09 GMT</div><div> ETag: "1c5461-47e-4f904ac13b240"</div><div> Accept-Ranges: bytes</div><div> Content-Length: 1150</div><div> Keep-Alive: timeout=5, max=100</div>
<div> Connection: Keep-Alive</div><div> Content-Type: text/plain; charset=UTF-8</div><div>Length: 1150 (1.1K) [text/plain]</div><div>Saving to: “/dev/null”</div><div><br></div><div>100%[=========================================================================================>] 1,150 --.-K/s in 0s</div>
<div><br></div><div>2014-05-12 21:50:55 (79.8 MB/s) - “/dev/null” saved [1150/1150]</div></div><div><br></div><div>And now the same thing to the 'healthy' web node (web1) by IP:</div><div><br></div><div><div>[root@varnish1:~] #wget -O /dev/null -S <a href="http://162.243.109.94/favicon.ico">http://162.243.109.94/favicon.ico</a></div>
<div>--2014-05-12 21:54:06-- <a href="http://162.243.109.94/favicon.ico">http://162.243.109.94/favicon.ico</a></div><div>Connecting to 162.243.109.94:80... connected.</div><div>HTTP request sent, awaiting response...</div>
<div> HTTP/1.1 200 OK</div><div> Date: Tue, 13 May 2014 01:54:05 GMT</div><div> Server: Apache/2.2.15 (CentOS)</div><div> Last-Modified: Wed, 05 Mar 2014 19:27:01 GMT</div><div> ETag: "e123b-47e-4f3e1013feb40"</div>
<div> Accept-Ranges: bytes</div><div> Content-Length: 1150</div><div> Keep-Alive: timeout=5, max=100</div><div> Connection: Keep-Alive</div><div> Content-Type: image/vnd.microsoft.icon</div><div>Length: 1150 (1.1K) [image/vnd.microsoft.icon]</div>
<div>Saving to: “/dev/null”</div><div><br></div><div>100%[=========================================================================================>] 1,150 --.-K/s in 0s</div><div><br></div><div>2014-05-12 21:54:06 (149 MB/s) - “/dev/null” saved [1150/1150]</div>
</div><div><br></div><div>And now here is the original (but corrected) version of my previous post! :)</div><div><br></div><div><div class="" style="margin:0px;padding:1px 1px 1px 10px;border:0px;font-family:'Helvetica Neue',Arial,Helvetica,sans-serif;font-size:12px;font-weight:bold;line-height:18.001798629760742px;vertical-align:baseline">
Back End node is Sick</div><div class="" style="margin:0px;padding:2px 0px 0px 10px;border:0px;font-family:'Helvetica Neue',Arial,Helvetica,sans-serif;font-size:12px;line-height:18.001798629760742px;vertical-align:baseline;min-height:160px">
<div class="" style="margin:0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline"><div class="" style="margin:0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">
<div class="" style="margin:0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline"><p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">
I am using Varnish to load balance a couple of web nodes using the random load balancing scheme.</p><p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">
I took a look at apache on the host that Varnish reports as sick:</p><p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">
And apache seems to be running fine on the 'sick' host:</p><p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">
[<a href="mailto:root@beta" rel="nofollow" style="margin:0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline;color:rgb(0,102,179)">root@beta</a>:/var/www/jf-current] #apachectl -S</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">VirtualHost configuration:</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">wildcard NameVirtualHosts and _default_ servers:</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">*:443 is a NameVirtualHost</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline"> default server <a href="http://beta.mywebsite.com/" target="_blank" rel="nofollow" style="margin:0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline;color:rgb(0,102,179)">beta.mywebsite.com</a> (/etc/httpd/conf.d/002_jf_beta_ssl.conf:78)</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline"> port 443 namevhost <a href="http://beta.mywebsite.com/" target="_blank" rel="nofollow" style="margin:0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline;color:rgb(0,102,179)">beta.mywebsite.com</a> (/etc/httpd/conf.d/002_jf_beta_ssl.conf:78)</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">*:80 is a NameVirtualHost</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline"> default server <a href="http://ref.mywebsite.com/" target="_blank" rel="nofollow" style="margin:0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline;color:rgb(0,102,179)">ref.mywebsite.com</a> (/etc/httpd/conf.d/001_ref.mywebsite.com.conf:1)</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline"> port 80 namevhost <a href="http://ref.mywebsite.com/" target="_blank" rel="nofollow" style="margin:0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline;color:rgb(0,102,179)">ref.mywebsite.com</a> (/etc/httpd/conf.d/001_ref.mywebsite.com.conf:1)</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline"> port 80 namevhost <a href="http://qa.mywebsite.com/" target="_blank" rel="nofollow" style="margin:0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline;color:rgb(0,102,179)">qa.mywebsite.com</a> (/etc/httpd/conf.d/002_qa.mywebsite.com.conf:1)</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline"> port 80 namevhost <a href="http://beta.mywebsite.com/" target="_blank" rel="nofollow" style="margin:0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline;color:rgb(0,102,179)">beta.mywebsite.com</a> (/etc/httpd/conf.d/003_beta.mywebsite.com.conf:1)</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline"> port 80 namevhost <a href="http://beta-test.mywebsite.com/" target="_blank" rel="nofollow" style="margin:0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline;color:rgb(0,102,179)">beta-test.mywebsite.com</a> (/etc/httpd/conf.d/004_beta-test.mywebsite.com.conf:1)</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline"> port 80 namevhost <a href="http://admin.mywebsite.com/" target="_blank" rel="nofollow" style="margin:0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline;color:rgb(0,102,179)">admin.mywebsite.com</a> (/etc/httpd/conf.d/005_admin.mywebsite.com.conf:1)</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline"> port 80 namevhost <a href="http://admin.mywebsite.com/" target="_blank" rel="nofollow" style="margin:0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline;color:rgb(0,102,179)">admin.mywebsite.com</a> (/etc/httpd/conf.d/10_admin.mywebsite.com.conf:1)</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline"> port 80 namevhost <a href="http://beta-test.mywebsite.com/" target="_blank" rel="nofollow" style="margin:0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline;color:rgb(0,102,179)">beta-test.mywebsite.com</a> (/etc/httpd/conf.d/10_beta-test.mywebsite.com.conf:1)</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline"> port 80 namevhost <a href="http://beta.mywebsite.com/" target="_blank" rel="nofollow" style="margin:0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline;color:rgb(0,102,179)">beta.mywebsite.com</a> (/etc/httpd/conf.d/10_beta.mywebsite.com.conf:1)</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline"> port 80 namevhost <a href="http://qa.mywebsite.com/" target="_blank" rel="nofollow" style="margin:0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline;color:rgb(0,102,179)">qa.mywebsite.com</a> (/etc/httpd/conf.d/10_qa.mywebsite.com.conf:1)</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline"> port 80 namevhost <a href="http://ref.mywebsite.com/" target="_blank" rel="nofollow" style="margin:0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline;color:rgb(0,102,179)">ref.mywebsite.com</a> (/etc/httpd/conf.d/10_ref.mywebsite.com.conf:1)</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">Syntax OK</p><p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">
<span style="font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit">My probe and host definitions in the varnish default.vcl looks like this:</span></p><p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">
probe favicon {</p><p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">
.url = "/favicon.ico";</p><p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">
.timeout = 34ms;</p><p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">
.interval = 1s;</p><p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">
.window = 10;</p><p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">
.threshold = 8;</p><p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">
}</p><p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">backend web1 {</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline"> .host = "10.10.1.94";</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline"><span style="font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit"> .port = "80";</span></p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline"> .probe = favicon;</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">}</p><p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">
backend web2 {</p><p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">
.host = "10.10.1.98";</p><p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">
.port = "80";</p><p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">
.probe = favicon;</p><p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">
}</p><p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">director www random {</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline"> { .backend = web1 ; .weight = 2; }</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline"> { .backend = web2 ; .weight = 2; </p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline"> }</p><p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">
>From the varnish host I can do a wget for the probe file from the host that varnish is marking as 'sick'</p><p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">
[root@varnish1:/etc/varnish] #wget -O /dev/null -S <a href="http://web2.mywebsite.com/favicon.ico" target="_blank" rel="nofollow" style="margin:0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline;color:rgb(0,102,179)">http://web2.mywebsite.com/favicon.ico</a></p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">--2014-05-11 20:48:23-- <a href="http://web1.mywebsite.com/favicon.ico" target="_blank" rel="nofollow" style="margin:0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline;color:rgb(0,102,179)">http://web2.mywebsite.com/favicon.ico</a></p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">Resolving web2.mywebsite.com... 10.10.1.98</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">Connecting to <a href="http://web1.mywebsite.com/" target="_blank" rel="nofollow" style="margin:0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline;color:rgb(0,102,179)">web2.mywebsite.com</a>|10.10.1.98|:80... connected.</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">HTTP request sent, awaiting response...</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline"> HTTP/1.1 200 OK</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline"> Date: Mon, 12 May 2014 00:48:25 GMT</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline"> Server: Apache/2.2.23 (CentOS)</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline"> Last-Modified: Sun, 22 Dec 2013 00:53:19 GMT</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline"> ETag: "2a8003-47e-4ee14efeebdc0"</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline"> Accept-Ranges: bytes</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline"> Content-Length: 1150</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline"> Keep-Alive: timeout=5, max=100</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline"> Connection: Keep-Alive</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline"> Content-Type: text/plain; charset=UTF-8</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">Length: 1150 (1.1K) [text/plain]</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">Saving to: “/dev/null”</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">100%[================================================================>] 1,150 --.-K/s in 0s</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">2014-05-11 20:48:24 (144 MB/s) - “/dev/null” saved [1150/1150]</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">I'm attempting here to approximate the calls that varnish is making to determine if a host is healthy.</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">And yet when I try having a look at debug.health from the command line Vaarnish is still labeling it as 'sick'</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">[root@varnish1:~] #varnishadm -T <a href="http://127.0.0.1:6082/" target="_blank" rel="nofollow" style="margin:0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline;color:rgb(0,102,179)">127.0.0.1:6082</a> debug.health -S /etc/varnish/secret</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline"><strong style="margin:0px;padding:0px">Backend web1 is Healthy</strong></p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">Current states good: 10 threshold: 8 window: 10</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">Average responsetime of good probes: 0.001247</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">Oldest Newest</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">================================================================</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">4444444444444444444444444444444444444444444444444444444444444444 Good IPv4</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Good Xmit</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR Good Recv</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH Happy</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline"><strong style="margin:0px;padding:0px">Backend web2 is Sick</strong></p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">Current states good: 0 threshold: 8 window: 10</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">Average responsetime of good probes: 0.000000</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">Oldest Newest</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">================================================================</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">---------------------------------------------------------------- Happy</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline">So I'm really wondering if I'm missing something here that'll help me to determine why Varnish thinks this host is sick! Also I have no problem browsing the custom URLs that I setup on each host to indicate which host apache is running on.</p>
<p style="margin:0px 10px 1.5em 0px;padding:0px;border:0px;font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit;vertical-align:baseline"> <span style="font-family:inherit;font-size:inherit;font-style:inherit;font-variant:inherit;font-weight:inherit;line-height:inherit">I would definitely appreciate any clues on how I could handle this.</span></p>
<div><br></div><div>Thanks</div><div>Tim</div></div></div></div></div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, May 12, 2014 at 1:37 AM, Jason Woods <span dir="ltr"><<a href="mailto:devel@jasonwoods.me.uk" target="_blank">devel@jasonwoods.me.uk</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto"><div>Hi</div><div><div class="h5"><div><br>On 12 May 2014, at 01:57, Tim Dunphy <<a href="mailto:bluethundr@gmail.com" target="_blank">bluethundr@gmail.com</a>> wrote:<br>
<br></div><blockquote type="cite"><div><div dir="ltr">I took a look at apache on the host that Varnish reports as sick:<div><br></div><div>And apache seems to be running fine on the 'sick' host:</div><div><br></div>
<div>
<p>[root@beta:/var/www/jf-current] #apachectl -S</p>
<p>VirtualHost configuration:</p>
<p>wildcard NameVirtualHosts and _default_ servers:</p>
<p>*:443 is a NameVirtualHost</p>
<p> default server <a href="http://beta.mywebsite.com" target="_blank">beta.mywebsite.com</a> (/etc/httpd/conf.d/002_jf_beta_ssl.conf:78)</p>
<p> port 443 namevhost <a href="http://beta.mywebsite.com" target="_blank">beta.mywebsite.com</a> (/etc/httpd/conf.d/002_jf_beta_ssl.conf:78)</p>
<p>*:80 is a NameVirtualHost</p>
<p> default server <a href="http://ref.mywebsite.com" target="_blank">ref.mywebsite.com</a> (/etc/httpd/conf.d/001_ref.mywebsite.com.conf:1)</p>
<p> port 80 namevhost <a href="http://ref.mywebsite.com" target="_blank">ref.mywebsite.com</a> (/etc/httpd/conf.d/001_ref.mywebsite.com.conf:1)</p>
<p> port 80 namevhost <a href="http://qa.mywebsite.com" target="_blank">qa.mywebsite.com</a> (/etc/httpd/conf.d/002_qa.mywebsite.com.conf:1)</p>
<p> port 80 namevhost <a href="http://beta.mywebsite.com" target="_blank">beta.mywebsite.com</a> (/etc/httpd/conf.d/003_beta.mywebsite.com.conf:1)</p>
<p> port 80 namevhost <a href="http://beta-test.mywebsite.com" target="_blank">beta-test.mywebsite.com</a> (/etc/httpd/conf.d/004_beta-test.mywebsite.com.conf:1)</p>
<p> port 80 namevhost <a href="http://admin.mywebsite.com" target="_blank">admin.mywebsite.com</a> (/etc/httpd/conf.d/005_admin.mywebsite.com.conf:1)</p>
<p> port 80 namevhost <a href="http://admin.mywebsite.com" target="_blank">admin.mywebsite.com</a> (/etc/httpd/conf.d/10_admin.mywebsite.com.conf:1)</p>
<p> port 80 namevhost <a href="http://beta-test.mywebsite.com" target="_blank">beta-test.mywebsite.com</a> (/etc/httpd/conf.d/10_beta-test.mywebsite.com.conf:1)</p>
<p> port 80 namevhost <a href="http://beta.mywebsite.com" target="_blank">beta.mywebsite.com</a> (/etc/httpd/conf.d/10_beta.mywebsite.com.conf:1)</p>
<p> port 80 namevhost <a href="http://qa.mywebsite.com" target="_blank">qa.mywebsite.com</a> (/etc/httpd/conf.d/10_qa.mywebsite.com.conf:1)</p>
<p> port 80 namevhost <a href="http://ref.mywebsite.com" target="_blank">ref.mywebsite.com</a> (/etc/httpd/conf.d/10_ref.mywebsite.com.conf:1)</p>
<p>Syntax OK</p><p><br></p><p>My probe and host definitions in the varnish default.vcl looks like this:</p><p><br></p><p>probe favicon {</p><p> .url = "/favicon.ico";</p>
<p> .timeout = 34ms;</p><p> .interval = 1s;</p><p> .window = 10;</p><p> .threshold = 8;</p><p>}</p><p><br></p><p>backend web1 {</p><p> .host = "10.10.1.94";</p>
<p> .port = "80";</p><p> .probe = favicon;</p><p>}</p><p><br></p><p>backend web2 {</p><p><br></p><p> .host = "10.10.1.98";</p><p>
.port = "80";</p><p> .probe = favicon;</p><p>}</p><p>director www random {<br></p><p> { .backend = web1 ; .weight = 2; }</p><p> { .backend = web2 ; .weight = 2; }</p>
<p>
</p><p> }</p></div><div class="gmail_extra">I was just experimenting the load balancing algorithms, but that probably doesn't have much bearing on this problem.</div><div class="gmail_extra"><br></div><div class="gmail_extra">
>From the varnish host I can do a wget for the probe file from the host that varnish is marking as 'sick'</div></div></div></blockquote><br><blockquote type="cite"><div><div dir="ltr"><div class="gmail_extra">
<p>[root@varnish1:/etc/varnish] #wget -O /dev/null -S <a href="http://web2.mywebsite.com/favicon.ico" target="_blank">http://web2.mywebsite.com/favicon.ico</a></p>
<p>--2014-05-11 20:48:23-- <a href="http://web1.mywebsite.com/favicon.ico" target="_blank">http://web1.mywebsite.com/favicon.ico</a></p></div></div></div></blockquote><div><br></div></div></div><div>You request web2 but it resolved web1? This may be related to the cause.</div>
<div><br></div><div>Can you wget both web1 and web2? Using IP too? (If I'm honest I'm not sure what Host header Varnish will request with.)</div><div><br></div><div>Jason</div><div><div class="h5"><blockquote type="cite">
<div><div dir="ltr"><div class="gmail_extra">
<p>Resolving <a href="http://web1.mywebsite.com" target="_blank">web1.mywebsite.com</a>... 10.10.1.98</p>
<p>Connecting to <a href="http://web1.mywebsite.com" target="_blank">web1.mywebsite.com</a>|10.10.1.98|:80... connected.</p>
<p>HTTP request sent, awaiting response...</p>
<p> HTTP/1.1 200 OK</p>
<p> Date: Mon, 12 May 2014 00:48:25 GMT</p>
<p> Server: Apache/2.2.23 (CentOS)</p>
<p> Last-Modified: Sun, 22 Dec 2013 00:53:19 GMT</p>
<p> ETag: "2a8003-47e-4ee14efeebdc0"</p>
<p> Accept-Ranges: bytes</p>
<p> Content-Length: 1150</p>
<p> Keep-Alive: timeout=5, max=100</p>
<p> Connection: Keep-Alive</p>
<p> Content-Type: text/plain; charset=UTF-8</p>
<p>Length: 1150 (1.1K) [text/plain]</p>
<p>Saving to: “/dev/null”</p>
<p>100%[================================================================>] 1,150 --.-K/s in 0s</p>
<p>2014-05-11 20:48:24 (144 MB/s) - “/dev/null” saved [1150/1150]</p><p><br></p><p>I'm attempting here to approximate the calls that varnish is making to determine if a host is healthy.</p>
<p>And yet when I try having a look at debug.health from the command line Vaarnish is still labeling it as 'sick'</p><p>[root@varnish1:~] #varnishadm -T <a href="http://127.0.0.1:6082" target="_blank">127.0.0.1:6082</a> debug.health -S /etc/varnish/secret</p>
<p><b>Backend web1 is Healthy</b></p><p>Current states good: 10 threshold: 8 window: 10</p><p>Average responsetime of good probes: 0.001247</p><p>Oldest Newest</p>
<p>================================================================</p><p>4444444444444444444444444444444444444444444444444444444444444444 Good IPv4</p><p>XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Good Xmit</p>
<p>RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR Good Recv</p><p>HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH Happy</p><p><b>Backend web2 is Sick</b></p><p>
Current states good: 0 threshold: 8 window: 10</p><p>Average responsetime of good probes: 0.000000</p><p>Oldest Newest</p><p>================================================================</p>
<p>---------------------------------------------------------------- Happy</p><p>So I'm really wondering if I'm missing something here that'll help me to determine why Varnish thinks this host is sick! Also I have no problem browsing the custom URLs that I setup on each host to indicate which host apache is running on.</p>
<p><br></p><p>Thanks</p><p>Tim</p><p><br></p><br><div class="gmail_quote">On Sat, May 10, 2014 at 12:58 AM, Tim Dunphy <span dir="ltr"><<a href="mailto:bluethundr@gmail.com" target="_blank">bluethundr@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Hey all,<div><br></div><div> I have two web backends in my varnish config. And one node is reporting healthy and the other is being reported as 'sick'.</div>
<div><br></div><div><div>10 Backend c 11 www web1</div>
<div> 0 Backend_health - web1 Still healthy 4--X-RH 5 3 5 0.001130 0.001067 HTTP/1.1 200 OK</div><div> 0 Backend_health - web1 Still healthy 4--X-RH 5 3 5 0.001231 0.001108 HTTP/1.1 200 OK</div><div> 0 Backend_health - web1 Still healthy 4--X-RH 5 3 5 0.001250 0.001143 HTTP/1.1 200 OK</div>
<div> 0 Backend_health - web1 Still healthy 4--X-RH 5 3 5 0.001127 0.001139 HTTP/1.1 200 OK</div><div> 0 Backend_health - web1 Still healthy 4--X-RH 5 3 5 0.001208 0.001157 HTTP/1.1 200 OK</div><div> 0 Backend_health - web1 Still healthy 4--X-RH 5 3 5 0.001562 0.001258 HTTP/1.1 200 OK</div>
<div> 0 Backend_health - web1 Still healthy 4--X-RH 5 3 5 0.001545 0.001330 HTTP/1.1 200 OK</div><div> 0 Backend_health - web1 Still healthy 4--X-RH 5 3 5 0.001363 0.001338 HTTP/1.1 200 OK</div><div> 11 BackendClose b web1</div>
<div><br></div><div><div>[root@varnish1:/etc/varnish] #varnishlog | grep web2</div><div> 0 Backend_health - web2 Still sick 4--X--- 0 3 5 0.000000 0.000000</div><div> 0 Backend_health - web2 Still sick 4--X--- 0 3 5 0.000000 0.000000</div>
<div> 0 Backend_health - web2 Still sick 4--X--- 0 3 5 0.000000 0.000000</div><div> 0 Backend_health - web2 Still sick 4--X--- 0 3 5 0.000000 0.000000</div><div> 0 Backend_health - web2 Still sick 4--X--- 0 3 5 0.000000 0.000000</div>
<div> 0 Backend_health - web2 Still sick 4--X--- 0 3 5 0.000000 0.000000</div><div> 0 Backend_health - web2 Still sick 4--X--- 0 3 5 0.000000 0.000000</div><div> 0 Backend_health - web2 Still sick 4--X--- 0 3 5 0.000000 0.000000</div>
<div> 0 Backend_health - web2 Still sick 4--X--- 0 3 5 0.000000 0.000000</div></div><div><br></div><div>And I'm really at a loss to understand why. Both nodes should be completely identical. And the web roots on both are basically svn repos that are in sync. </div>
<div><br></div><div>From web1 :</div><div><br></div><div><div>[root@beta:/var/www/jf-current] #svn info | grep -i revision</div><div>Revision: 17</div></div><div><br></div><div>To web2:</div><div><br></div><div><div>[root@beta-new:/var/www/jf-current] #svn info | grep -i revision</div>
<div>Revision: 17</div></div><div><br></div><div>This is the part of my vcl file where I define the web back ends:</div><div><br></div><div><div>probe favicon {</div><div> .url = "/favicon.ico";</div><div> .timeout = 60ms;</div>
<div> .interval = 2s;</div><div> .window = 5;</div><div> .threshold = 3;</div><div>}</div><div><br></div><div>backend web1 {</div><div> .host = "xx.xx.xx.xx";</div><div> .port = "80";</div><div>
.probe = favicon;</div>
<div>}</div><div><br></div><div>backend web2 {</div><div><br></div><div> .host = "xx.xx.xx.xx";</div><div> .port = "80";</div><div> .probe = favicon;</div><div>}</div></div><div><br></div><div>And the file that varnish is probing for is present on both:</div>
<div><br></div><div><div>[root@beta:/var/www/jf-current] #ls -l /var/www/jf-current/favicon.ico</div><div>-rwxrwxr-x 1 apache ftp 1150 Dec 22 00:53 /var/www/jf-current/favicon.ico</div></div><div><br></div><div>I've also setup individual web URLs for each host that isn't cached in varnish so I can hit each one. And each site comes up ok. So I'm a little puzzled as to why the second web host is reporting 'sick' and what I can do to get it back into load balancing. </div>
<div><br></div><div>Thanks for any help you can provide!</div><span><font color="#888888"><div><br></div><div>Tim</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div>-- <br>GPG me!!<br>
<br>gpg --keyserver <a href="http://pool.sks-keyservers.net" target="_blank">pool.sks-keyservers.net</a> --recv-keys F186197B<br>
<br>
</font></span></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br>GPG me!!<br><br>gpg --keyserver <a href="http://pool.sks-keyservers.net" target="_blank">pool.sks-keyservers.net</a> --recv-keys F186197B<br><br>
</div></div>
</div></blockquote></div></div><blockquote type="cite"><div><span>_______________________________________________</span><br><span>varnish-misc mailing list</span><br><span><a href="mailto:varnish-misc@varnish-cache.org" target="_blank">varnish-misc@varnish-cache.org</a></span><br>
<span><a href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc" target="_blank">https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc</a></span></div></blockquote></div></blockquote></div><br>
<br clear="all"><div><br></div>-- <br>GPG me!!<br><br>gpg --keyserver <a href="http://pool.sks-keyservers.net" target="_blank">pool.sks-keyservers.net</a> --recv-keys F186197B<br><br>
</div>