<div dir="ltr"><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><span style="font-size:12.8000001907349px"></span><span style="font-size:12.8000001907349px"></span><span class="im" style="color:rgb(80,0,80);font-family:arial,sans-serif;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:12.8000001907349px">> It's pretty odd how inconsistent this VCL is behaving.<br><br></span><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:12.8000001907349px">Varnish does not produce 504 responses.</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:12.8000001907349px"><br style="color:rgb(34,34,34);font-family:arial,sans-serif;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:12.8000001907349px"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:12.8000001907349px">If you see 504, they are coming through from your backend.</span></blockquote><div><br></div><div>Yeah that makes sense.<br> <br></div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><br style="color:rgb(34,34,34);font-family:arial,sans-serif;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:12.8000001907349px"><span class="im" style="color:rgb(80,0,80);font-family:arial,sans-serif;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:12.8000001907349px"><br><br>> On one load of the page where the graphics are broken I'm seeing this error<br>> in the logs:<br>> 10 TxHeader c Connection: close<br>> 10 TxHeader c X-Cache: MISS<br>> 10 Debug c Write error, retval = -1, len = 602, errno =<br>> Connection reset by peer<br>> 10 ReqEnd c 241437586 1438049428.763690233 1438049435.868994236<br>> 0.000110626 7.105183125 0.000120878<br><br></span><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:12.8000001907349px">Client or backend closed the connection (went away) after 7 seconds.</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:12.8000001907349px"><br style="color:rgb(34,34,34);font-family:arial,sans-serif;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:12.8000001907349px"><br style="color:rgb(34,34,34);font-family:arial,sans-serif;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:12.8000001907349px"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:12.8000001907349px">[..]</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:12.8000001907349px"><span class="im" style="color:rgb(80,0,80);font-family:arial,sans-serif;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:12.8000001907349px">> And on 504 errors I'm seeing this result in the logs:<br>> 10 TxHeader c Via: 1.1 varnish<br>> 10 TxHeader c Connection: close<br>> 10 TxHeader c X-Cache: MISS<br>> 10 Length c 316<br>> 10 ReqEnd c 241437672 1438049679.750560999 1438049679.750730276<br>> 0.000099182 0.000094652 0.000074625<br>> 10 SessionClose c error<br>> 10 StatSess c 54.86.143.49 49821 0 1 1 0 0 0 285 316<br>> I'm running 3<span class="Apple-converted-space"> </span>back ends<span class="Apple-converted-space"> </span>using apache 2.4 on Centos 7. I'm running two<br>> Varnish nodes at version 3.0.5.<br></span></blockquote><div><br><br></div><div>When I'm tailing the logs for both apache and varnish at the same time, this is what I see happening in both logs when the 504 errors occur:<br><br></div><div>Varnish:<br><br> 10 Debug c Write error, retval = -1, len = 613, errno = Connection reset by peer<br> 10 ReqEnd c 822463677 1438097028.730667830 1438097034.047527790 0.000082970 5.316770077 0.000089884<br> 10 StatSess c 54.86.143.49 42935 5 1 1 0 0 1 613 0<br> 0 CLI - Rd ping<br><br><br></div><div>Apache:<br><br>[Tue Jul 28 15:16:11.614501 2015] [authz_core:debug] [pid 6407] mod_authz_core.c(809): [client <a href="http://162.243.86.41:55114">162.243.86.41:55114</a>] AH01626: authorization result of <RequireAny>: denied (no authenticated user yet)<br>[Tue Jul 28 15:16:11.614763 2015] [authz_core:debug] [pid 6407] mod_authz_core.c(809): [client <a href="http://162.243.86.41:55114">162.243.86.41:55114</a>] AH01626: authorization result of Require valid-user : granted<br>[Tue Jul 28 15:16:11.614767 2015] [authz_core:debug] [pid 6407] mod_authz_core.c(809): [client <a href="http://162.243.86.41:55114">162.243.86.41:55114</a>] AH01626: authorization result of <RequireAny>: granted<br>[Tue Jul 28 15:16:11.614841 2015] [authz_core:debug] [pid 6407] mod_authz_core.c(809): [client <a href="http://162.243.86.41:55114">162.243.86.41:55114</a>] AH01626: authorization result of Require valid-user : denied (no authenticated user yet)<br><br></div><div>And as you can see from my VCL I am performing some apache basic authentication, and then passing it through to the back end. And as you can see my health check also passes authentication headers to the health check file (healthcheck.php)<br><br></div><div>Now I could be wrong. But what I think is happening is that varnish is passing the request to on back end, and authenticating, and then sends another request to a different host without authentication being passed to it.<br><br></div><div>If my theory is correct, this could be fixed by added session persistence to my varnish VCL. How can I add sticky sessions to varnish? I think that might do the trick.<br><br></div><div>Thanks<br></div><div>Tim<br></div><div><br> </div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><span class="im" style="color:rgb(80,0,80);font-family:arial,sans-serif;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:12.8000001907349px"><br></span><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:12.8000001907349px">Please note that Varnish 3 is<span class="Apple-converted-space"> </span>end<span class="Apple-converted-space"> </span>of life.</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:12.8000001907349px"><br style="color:rgb(34,34,34);font-family:arial,sans-serif;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:12.8000001907349px"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:12.8000001907349px">In the 4.0 release timestamp logging is vastly improved. It would tell</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:12.8000001907349px"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:12.8000001907349px">you if it was the client or backend that went away above, for example.</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:12.8000001907349px"><br style="color:rgb(34,34,34);font-family:arial,sans-serif;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:12.8000001907349px"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:12.8000001907349px">Another sweet feature in 4.0 is that<span class="Apple-converted-space"> </span>just-expired<span class="Apple-converted-space"> </span>content will be served</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:12.8000001907349px"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:12.8000001907349px">while a background fetch is initiated.</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:12.8000001907349px"><span class="im" style="color:rgb(80,0,80);font-family:arial,sans-serif;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:12.8000001907349px"><br>> backend web1 {<br>> .host = “10.10.10.25”;<br>> .port = "80";<br>> .connect_timeout = 45s;<br>> .first_byte_timeout = 45s;<br>> .between_bytes_timeout = 45s;<br></span><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:12.8000001907349px">[probe section]</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:12.8000001907349px"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:12.8000001907349px">> .timeout = 60s;</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:12.8000001907349px"><br style="color:rgb(34,34,34);font-family:arial,sans-serif;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:12.8000001907349px"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:12.8000001907349px">These timeouts are way too long. What client sits around for 45s waiting</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:12.8000001907349px"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:12.8000001907349px">for a web page? Isn't it better to produce clean 503s to the client that</span><br style="color:rgb(34,34,34);font-family:arial,sans-serif;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:12.8000001907349px"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:12.8000001907349px">can be looked for in<span class="Apple-converted-space"> </span>varnishlog, rather than not responding?</span></blockquote><div class="gmail_extra"><div><br></div>-- <br><div>GPG me!!<br><br>gpg --keyserver <a href="http://pool.sks-keyservers.net" target="_blank">pool.sks-keyservers.net</a> --recv-keys F186197B<br><br></div>
</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jul 28, 2015 at 7:31 AM, Lasse Karstensen <span dir="ltr"><<a href="mailto:lkarsten@varnish-software.com" target="_blank">lkarsten@varnish-software.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On Mon, Jul 27, 2015 at 10:24:21PM -0400, Tim Dunphy wrote:<br>
[cut]<br>
<span class="">> It's pretty odd how inconsistent this VCL is behaving.<br>
<br>
</span>Varnish does not produce 504 responses.<br>
<br>
If you see 504, they are coming through from your backend.<br>
<span class=""><br>
<br>
> On one load of the page where the graphics are broken I'm seeing this error<br>
> in the logs:<br>
> 10 TxHeader c Connection: close<br>
> 10 TxHeader c X-Cache: MISS<br>
> 10 Debug c Write error, retval = -1, len = 602, errno =<br>
> Connection reset by peer<br>
> 10 ReqEnd c 241437586 1438049428.763690233 1438049435.868994236<br>
> 0.000110626 7.105183125 0.000120878<br>
<br>
</span>Client or backend closed the connection (went away) after 7 seconds.<br>
<br>
<br>
[..]<br>
<span class="">> And on 504 errors I'm seeing this result in the logs:<br>
> 10 TxHeader c Via: 1.1 varnish<br>
> 10 TxHeader c Connection: close<br>
> 10 TxHeader c X-Cache: MISS<br>
> 10 Length c 316<br>
> 10 ReqEnd c 241437672 1438049679.750560999 1438049679.750730276<br>
> 0.000099182 0.000094652 0.000074625<br>
> 10 SessionClose c error<br>
> 10 StatSess c 54.86.143.49 49821 0 1 1 0 0 0 285 316<br>
> I'm running 3 back ends using apache 2.4 on Centos 7. I'm running two<br>
> Varnish nodes at version 3.0.5.<br>
<br>
</span>Please note that Varnish 3 is end of life.<br>
<br>
In the 4.0 release timestamp logging is vastly improved. It would tell<br>
you if it was the client or backend that went away above, for example.<br>
<br>
Another sweet feature in 4.0 is that just-expired content will be served<br>
while a background fetch is initiated.<br>
<span class=""><br>
> backend web1 {<br>
> .host = “10.10.10.25”;<br>
> .port = "80";<br>
> .connect_timeout = 45s;<br>
> .first_byte_timeout = 45s;<br>
> .between_bytes_timeout = 45s;<br>
</span>[probe section]<br>
> .timeout = 60s;<br>
<br>
These timeouts are way too long. What client sits around for 45s waiting<br>
for a web page? Isn't it better to produce clean 503s to the client that<br>
can be looked for in varnishlog, rather than not responding?<br>
<span class="HOEnZb"><font color="#888888"><br>
<br>
--<br>
Lasse Karstensen<br>
Varnish Software AS<br>
</font></span></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature">GPG me!!<br><br>gpg --keyserver <a href="http://pool.sks-keyservers.net" target="_blank">pool.sks-keyservers.net</a> --recv-keys F186197B<br><br></div>
</div>