<div dir="ltr">Geoff, Et al, thanks for your replies, i should have mentioned, that the 503's are not unique to POST requests, they are also happening more frequent within <br>(very rarely == on 0.0038% of requests), on a GETs. i've been thinking of testing within my config something along this FOUND vcl code snipped as a possible solution. <div>
<br></div><div><br></div><div><br></div><div><pre style="white-space:pre-wrap;color:rgb(0,0,0)"><i>sub vcl_fetch {
</i><i> if (obj.status == 500 || obj.status == 503 || obj.status ==
</i><i> 504) {
</i><i> restart;
</i><i> }</i></pre><pre style="white-space:pre-wrap;color:rgb(0,0,0)"><br></pre><div><br></div><div>174 ReqStart c 10.33.13.254 38233 1505576793</div><div> 174 RxRequest c GET</div><div> 174 RxURL c /auburnfootball/modules/most_read.html</div>
<div> 174 RxProtocol c HTTP/1.1</div><div> 174 RxHeader c If-Modified-Since: Wed, 16 Apr 2014 14:23:34 GMT</div><div> 174 RxHeader c Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8</div>
<div> 174 RxHeader c Accept-Charset: utf-8, iso-8859-1, utf-16, *;q=0.7</div><div> 174 RxHeader c Accept-Language: en-US</div><div> 174 RxHeader c Cookie: adv_lid=04ea3e9b86834130c4cb3be359f7b220; OAX=RsRvNFEDhgkAAuO2; __qca=P0-1959933173-1359185426964; __atuvc=0%7C27%2C0%7C28%2C0%7C29%2C0%7C30%2C1%7C31; mp_948b359d7741fa246376d95399585cc3_mixpanel=%7B%7D; __utma=130511327.2000905843.1375913741.1375</div>
<div> 174 RxHeader c User-Agent: Mozilla/5.0 (Linux; U; Android 4.1.2; en-us; SCH-I200 Build/JZO54K) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30</div><div> 174 RxHeader c X-UIDH: MTQ5ODI0NDA2MAA/58mlyaG6h8c9ATsT9BtivrzyjC/Gl0mdhNCPRIOC1Q==</div>
<div> 174 RxHeader c x-wap-profile: <a href="http://uaprof.vtext.com/sam/SCH-I200/SCH-I200.xml">http://uaprof.vtext.com/sam/SCH-I200/SCH-I200.xml</a></div><div> 174 RxHeader c X-Akamai-Esi-Fragment-Suffix: f.6</div>
<div> 174 RxHeader c True-Client-IP: 127.0.0.1</div><div> 174 RxHeader c X-Akamai-CONFIG-LOG-DETAIL: true</div><div> 174 RxHeader c TE: chunked;q=1.0</div><div> 174 RxHeader c Connection: TE</div><div>
174 RxHeader c Akamai-Origin-Hop: 2</div><div> 174 RxHeader c Via: 4.0 asi_server, 1.0 <a href="http://v1-akamaitech.net">v1-akamaitech.net</a>(ghost) (AkamaiGHost), 1.1 <a href="http://akamai.net">akamai.net</a>(ghost) (AkamaiGHost)</div>
<div> 174 RxHeader c Accept-ESI: 1.0</div><div> 174 RxHeader c X-Forwarded-For: 127.0.0.1, 96.17.202.197</div><div> 174 RxHeader c Host: <a href="http://www.al.com">www.al.com</a></div><div> 174 RxHeader c Cache-Control: max-age=180</div>
<div> 174 RxHeader c Connection: keep-alive</div><div> 174 RxHeader c X-Forwarded-For: 23.73.180.223</div><div> 174 RxHeader c Accept-Encoding: identity</div><div> 174 VCL_call c recv pass</div><div> 174 VCL_call c hash</div>
<div> 174 Hash c /auburnfootball/modules/most_read.html</div><div> 174 Hash c www.**.com</div><div> 174 VCL_return c hash</div><div> 174 VCL_call c pass pass</div><div> 174 Backend c 101 www_prod www_prod</div>
<div> 174 FetchError c http first read error: -1 104 (Connection reset by peer)</div><div> 174 Backend c 56 www_prod www_prod</div><div> 174 FetchError c http first read error: -1 104 (Connection reset by peer)</div>
<div> 174 VCL_call c error deliver</div><div> 174 VCL_call c deliver deliver</div><div> 174 TxProtocol c HTTP/1.1</div><div> 174 TxStatus c 503</div><div> 174 TxResponse c Service Unavailable</div><div>
174 TxHeader c Server: Varnish</div><div> 174 TxHeader c Content-Type: text/html; charset=utf-8</div><div> 174 TxHeader c Retry-After: 5</div><div> 174 TxHeader c Content-Length: 419</div><div> 174 TxHeader c Accept-Ranges: bytes</div>
<div> 174 TxHeader c Date: Wed, 16 Apr 2014 16:30:29 GMT</div><div> 174 TxHeader c X-Varnish: 1505576793</div><div> 174 TxHeader c Age: 0</div><div> 174 TxHeader c Via: 1.1 varnish</div><div> 174 TxHeader c Connection: close</div>
<div> 174 Length c 419</div><div> 174 ReqEnd c 1505576793 1397665829.285193920 1397665829.298522949 0.022354841 0.013305187 0.000023842 </div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">
On Thu, Apr 17, 2014 at 6:00 AM, <span dir="ltr"><<a href="mailto:varnish-misc-request@varnish-cache.org" target="_blank">varnish-misc-request@varnish-cache.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Send varnish-misc mailing list submissions to<br>
<a href="mailto:varnish-misc@varnish-cache.org">varnish-misc@varnish-cache.org</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc" target="_blank">https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:varnish-misc-request@varnish-cache.org">varnish-misc-request@varnish-cache.org</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:varnish-misc-owner@varnish-cache.org">varnish-misc-owner@varnish-cache.org</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of varnish-misc digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Re: intermittent 503's (Geoffrey Simmons)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Thu, 17 Apr 2014 17:46:21 +1200<br>
From: Geoffrey Simmons <<a href="mailto:geoff@uplex.de">geoff@uplex.de</a>><br>
To: L Cruzero <<a href="mailto:lcruzero@gmail.com">lcruzero@gmail.com</a>><br>
Cc: "<a href="mailto:varnish-misc@varnish-cache.org">varnish-misc@varnish-cache.org</a>" <<a href="mailto:varnish-misc@varnish-cache.org">varnish-misc@varnish-cache.org</a>><br>
Subject: Re: intermittent 503's<br>
Message-ID: <<a href="mailto:A1B813DE-4217-4546-9E65-E0D9139C684B@uplex.de">A1B813DE-4217-4546-9E65-E0D9139C684B@uplex.de</a>><br>
Content-Type: text/plain; charset="us-ascii"<br>
<br>
> 35 FetchError c backend write error: 104 (Connection reset by peer)<br>
<br>
"backend write error" in versions up to at least 3.0.3 can be a very misleading error message -- you may in fact have had a connection reset on the *client* connection while reading the request body of your POST request. In my experience, that's much more common than an error writing to the backend connection.<br>
<br>
Sending the request body to the backend is part of the fetch operation, which is why an error in that phase is logged as a FetchError. Varnish reads the body from the client and writes it to the backend in a loop, so if any error occurs then, either on the client or backend side, it's logged as "FetchError: backend write error".<br>
<br>
There was a fix that made the error message distinguish the problem more clearly, might have made it into 3.0.4.<br>
<br>
Anyway if I were you I'd look for a connection reset on a read of the POST body from the client connection.<br>
<br>
<br>
HTH,<br>
Geoff<br>
<br>
Sent from my iPad<br>
<br>
> On Apr 17, 2014, at 12:37 AM, L Cruzero <<a href="mailto:lcruzero@gmail.com">lcruzero@gmail.com</a>> wrote:<br>
><br>
> Hi, in our varnish setup(varnish-3.0.3-1.el5) using a loadbalanced really Healthy VIP as the only backend, we seem to be getting intermittent 503's<br>
><br>
><br>
> stdout captured with varnishlog -c -m TxStatus:503<br>
><br>
> 35 ReqStart c 10.33.13.254 24582 1166630723<br>
> 35 RxRequest c POST<br>
> 35 RxURL c /index.php?option=com_jce&task=plugin&plugin=imgmanager&file=imgmanager&method=form&cid=20&6bc427c8a7981f4fe1f5ac65c1246b5f=cf6dd3cf1923c950586d0dd595c8e20b<br>
> 35 RxProtocol c HTTP/1.1<br>
> 35 RxHeader c User-Agent: BOT/0.1 (BOT for JCE)<br>
> 35 RxHeader c Content-Type: multipart/form-data; boundary=---------------------------41184676334<br>
> 35 RxHeader c Content-Length: 5000<br>
> 35 RxHeader c True-Client-IP: 202.80.119.178<br>
> 35 RxHeader c X-Akamai-CONFIG-LOG-DETAIL: true<br>
> 35 RxHeader c TE: chunked;q=1.0<br>
> 35 RxHeader c Connection: TE<br>
> 35 RxHeader c Akamai-Origin-Hop: 2<br>
> 35 RxHeader c Via: 1.1 <a href="http://v1-akamaitech.net" target="_blank">v1-akamaitech.net</a>(ghost) (AkamaiGHost), 1.1 <a href="http://akamai.net" target="_blank">akamai.net</a>(ghost) (AkamaiGHost)<br>
> 35 RxHeader c X-Forwarded-For: 202.80.119.178, 114.4.39.206<br>
> 35 RxHeader c Host: <a href="http://www.somedomain.com" target="_blank">www.somedomain.com</a><br>
> 35 RxHeader c Cache-Control: max-age=120<br>
> 35 RxHeader c Connection: keep-alive<br>
> 35 RxHeader c X-Forwarded-For: 23.73.180.223<br>
> 35 RxHeader c Accept-Encoding: identity<br>
> 35 VCL_call c recv pass<br>
> 35 VCL_call c hash<br>
> 35 Hash c /index.php?option=com_jce&task=plugin&plugin=imgmanager&file=imgmanager&method=form&cid=20&6bc427c8a7981f4fe1f5ac65c1246b5f=cf6dd3cf1923c950586d0dd595c8e20b<br>
> 35 Hash c <a href="http://www.somedomain.com" target="_blank">www.somedomain.com</a><br>
> 35 VCL_return c hash<br>
> 35 VCL_call c pass pass<br>
> 35 Backend c 29 www_prod www_prod<br>
> 35 FetchError c backend write error: 104 (Connection reset by peer)<br>
> 35 Backend c 42 www_prod www_prod<br>
> 35 FetchError c backend write error: 104 (Connection reset by peer)<br>
> 35 VCL_call c error deliver<br>
> 35 VCL_call c deliver deliver<br>
> 35 TxProtocol c HTTP/1.1<br>
> 35 TxStatus c 503<br>
> 35 TxResponse c Service Unavailable<br>
> 35 TxHeader c Server: Varnish<br>
> 35 TxHeader c Content-Type: text/html; charset=utf-8<br>
> 35 TxHeader c Retry-After: 5<br>
> 35 TxHeader c Content-Length: 419<br>
> 35 TxHeader c Accept-Ranges: bytes<br>
> 35 TxHeader c Date: Tue, 15 Apr 2014 19:31:22 GMT<br>
> 35 TxHeader c X-Varnish: 1166630723<br>
> 35 TxHeader c Age: 0<br>
> 35 TxHeader c Via: 1.1 varnish<br>
> 35 TxHeader c Connection: close<br>
> 35 Length c 419<br>
> 35 ReqEnd c 1166630723 1397590282.467632055 1397590282.899372101 0.033779144 0.431685925 0.0000541<br>
><br>
><br>
> here is super simple pass thru NO cache config :<br>
><br>
><br>
> backend www_prod {<br>
> .host = "cs-****.*****.*****.net";<br>
> .port = "80";<br>
> .probe = {<br>
> .url = "/";<br>
> .timeout = 5s;<br>
> .interval = 1s;<br>
> .window = 5;<br>
> .threshold = 2;<br>
> }<br>
> }<br>
><br>
> sub vcl_recv {<br>
> if (req.http.X-ADI-MISS) {<br>
> # Force a cache miss<br>
> set req.hash_always_miss = true;<br>
> }<br>
><br>
> if (req.url == "/varnish-health/" || req.url ~ "^/stack-check*") {<br>
> error 200 "Varnish is responding";<br>
> set req.http.Connection = "close";<br>
> }<br>
><br>
><br>
> if (req.http.host ~ "^(origin|www)") {<br>
> set req.http.host = regsub(req.http.host, "^origin\.", "www.");<br>
> set req.backend = www_prod;<br>
> return(pass);<br>
><br>
> }<br>
><br>
><br>
> ## we tried with and w/o these conditions intermittent 503's persisted<br>
><br>
> # if (req.backend.healthy) {<br>
> # set req.grace = 30s;<br>
> # } else {<br>
> # set req.grace = 24h;<br>
> # }<br>
><br>
> }<br>
><br>
><br>
> sub vcl_fetch {<br>
> # set beresp.do_esi = true;<br>
> set beresp.ttl = 1m;<br>
><br>
> # set beresp.grace = 24h;<br>
> }<br>
><br>
><br>
> Any helpful thoughts and or possible leads on this will be much appreciated.<br>
> kind regards,<br>
> -LC<br>
><br>
> _______________________________________________<br>
> varnish-misc mailing list<br>
> <a href="mailto:varnish-misc@varnish-cache.org">varnish-misc@varnish-cache.org</a><br>
> <a href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc" target="_blank">https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc</a><br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20140417/cf00e773/attachment-0001.html" target="_blank">https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20140417/cf00e773/attachment-0001.html</a>><br>
<br>
------------------------------<br>
<br>
_______________________________________________<br>
varnish-misc mailing list<br>
<a href="mailto:varnish-misc@varnish-cache.org">varnish-misc@varnish-cache.org</a><br>
<a href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc" target="_blank">https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc</a><br>
<br>
End of varnish-misc Digest, Vol 97, Issue 16<br>
********************************************<br>
</blockquote></div><br></div>