<div dir="auto">Arg, we wandered off without including the mailing-list.<div dir="auto"><br></div><div dir="auto">Issue was retrying the backends request without having the body cached.</div><div dir="auto"><br></div><div dir="auto">Junaid: the body is cached only for the request duration, and it's not reused outside of that , no worry here.</div></div><br><div class="gmail_quote"><div dir="ltr">On Mon, Dec 3, 2018, 03:28 Junaid Mukhtar <<a href="mailto:junaid.mukhtar@gmail.com">junaid.mukhtar@gmail.com</a> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small">Just teseted it; and it looks to have fixed the problem. <br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small">I just imported the vmod and used std.cache_req_body(2000KB); in vcl_recv</div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small">Just one question; do you think caching could cause any issues when doing the hash etc? we don't really want to cache the post body and use it anywhere.<br clear="all"></div><div><div dir="ltr" class="m_5787149082928576820gmail_signature"><font size="2"><span style="font-family:verdana,sans-serif"><br>--------<br>Regards,</span><br style="font-family:verdana,sans-serif"><span style="font-family:verdana,sans-serif">Junaid</span></font><br></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr">On Fri, Nov 30, 2018 at 11:28 PM Guillaume Quintard <<a href="mailto:guillaume@varnish-software.com" target="_blank" rel="noreferrer">guillaume@varnish-software.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr">Hum, this is a bit of a shot in the dark, but can you try using <a href="https://varnish-cache.org/docs/trunk/reference/vmod_generated.html#func-cache-req-body" target="_blank" rel="noreferrer">https://varnish-cache.org/docs/trunk/reference/vmod_generated.html#func-cache-req-body</a> (in vcl_recv)?</div><div dir="ltr"><br></div><div>It could be that varnish can't retry because there's no body to send anymore. I would expect the error to be clearer if that was the case though...</div><div><br></div><div dir="ltr"><div><div dir="ltr" class="m_5787149082928576820m_6300905689656486360gmail_signature"><div dir="ltr"><div>-- <br></div>Guillaume Quintard<br></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr">On Fri, Nov 30, 2018 at 10:19 AM Junaid Mukhtar <<a href="mailto:junaid.mukhtar@gmail.com" target="_blank" rel="noreferrer">junaid.mukhtar@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small">no, all of the GET requests to staticpages is working fine. <br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small">the issue is only when original POST request gets 500 from app server and then varnish tries to GET the appropriate staticpage from the staticpage backend. <br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small">Staticpage backend is only HTTP<br clear="all"></div><div><div dir="ltr" class="m_5787149082928576820m_6300905689656486360m_-7920558716683778954gmail_signature" data-smartmail="gmail_signature"><font size="2"><span style="font-family:verdana,sans-serif"><br>--------<br>Regards,</span><br style="font-family:verdana,sans-serif"><span style="font-family:verdana,sans-serif">Junaid</span></font><br></div></div><br></div><br><div class="gmail_quote"><div dir="ltr">On Fri, Nov 30, 2018 at 6:16 PM Guillaume Quintard <<a href="mailto:guillaume@varnish-software.com" target="_blank" rel="noreferrer">guillaume@varnish-software.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Are all the requests to the "<span style="font-family:verdana,sans-serif">staticpages</span>" backend failures, or just some of them?<div><br></div><div>Is that backend listen to HTTP or to HTTPS?</div><div><br clear="all"><div><div dir="ltr" class="m_5787149082928576820m_6300905689656486360m_-7920558716683778954m_1686636647750696316gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>-- <br></div>Guillaume Quintard<br></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr">On Fri, Nov 30, 2018 at 12:32 AM Junaid Mukhtar <<a href="mailto:junaid.mukhtar@gmail.com" target="_blank" rel="noreferrer">junaid.mukhtar@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small">Hi</div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small">Kindly find below. One thing i would like to highlight is that we change backend from application to nginx (staticpage) for the error-pages. <br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small">Wireshark shows reset being sent by Varnish to nginx. <br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small">* << BeReq >> 196913<br>- Begin bereq 196912 pass<br>- Timestamp Start: 1543416190.760775 0.000000 0.000000<br>- BereqMethod POST<br>- BereqURL /application/person<br>- BereqProtocol HTTP/1.1<br>- BereqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8<br>- BereqHeader Accept-Language: en-GB,en-US;q=0.9,en;q=0.8<br>- BereqHeader Cache-Control: max-age=0<br>- BereqHeader Content-Type: application/x-www-form-urlencoded<br>- BereqHeader Cookie: check=true; AMCVS_232334B8527856F80A490D44%40AdobeOrg=1; cookieStatementDisplayedOnce=true; s_cc=true; k2c_Organization_cids=Organization_uaEMwVyUDe8mY5m; JOIN_JSESSIONID=dcd2c25b-a3e5-4a58-826c-e9452190265f; ttco=1543398034746; _gcl_au=1.1.1660607311.15<br>- BereqHeader Origin: <a href="https://development.organization.com" target="_blank" rel="noreferrer">https://development.organization.com</a><br>- BereqHeader Referer: <a href="https://development.organization.com/application/person" target="_blank" rel="noreferrer">https://development.organization.com/application/person</a><br>- BereqHeader Upgrade-Insecure-Requests: 1<br>- BereqHeader User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36 OPR/56.0.3051.104<br>- BereqHeader X-Forwarded-Port: 443<br>- BereqHeader X-Forwarded-Proto: https<br>- BereqHeader Content-Length: 888<br>- BereqHeader X-Forwarded-For: 2.2.2.2, 1.1.1.1<br>- BereqHeader xff: 2.2.2.2, 1.1.1.1<br>- BereqHeader host: <a href="http://development.organization.com" target="_blank" rel="noreferrer">development.organization.com</a><br>- BereqHeader Accept-Encoding: gzip<br>- BereqHeader X-Varnish: 196913<br>- VCL_call BACKEND_FETCH<br>- VCL_return fetch<br>- BackendOpen 39 boot.application1 3.3.3.3 8089 4.4.4.4 58036<br>- BackendStart 3.3.3.3 8089<br>- Timestamp Bereq: 1543416190.760820 0.000045 0.000045<br>- Timestamp Beresp: 1543416195.877653 5.116878 5.116833<br>- BerespProtocol HTTP/1.1<br>- BerespStatus 500<br>- BerespReason Internal Server Error<br>- BerespHeader Server: Apache-Coyote/1.1<br>- BerespHeader X-Application-Context: application<br>- BerespHeader X-Content-Type-Options: nosniff<br>- BerespHeader X-XSS-Protection: 1; mode=block<br>- BerespHeader Cache-Control: no-cache, no-store, max-age=0, must-revalidate<br>- BerespHeader Pragma: no-cache<br>- BerespHeader Expires: 0<br>- BerespHeader Strict-Transport-Security: max-age=31536000 ; includeSubDomains<br>- BerespHeader X-Frame-Options: DENY<br>- BerespHeader Content-Type: text/html;charset=ISO-8859-1<br>- BerespHeader Content-Language: en-GB<br>- BerespHeader Content-Length: 490<br>- BerespHeader Date: Wed, 28 Nov 2018 14:43:15 GMT<br>- BerespHeader Connection: close<br>- TTL RFC -1 10 -1 1543416196 1543416196 1543416195 0 0<br>- VCL_call BACKEND_RESPONSE<br>- BereqMethod GET<br>- BereqURL /errors/application/500.html<br>- VCL_return retry<br>- BackendClose 39 boot.application1<br>- Timestamp Retry: 1543416195.877710 5.116935 0.000057<br>- Link bereq 262229 retry<br>- End<br><br>* << BeReq >> 262229<br>- Begin bereq 196913 retry<br>- Timestamp Start: 1543416195.877710 5.116935 0.000000<br>- BereqMethod GET<br>- BereqURL /errors/application/500.html<br>- BereqProtocol HTTP/1.1<br>- BereqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8<br>- BereqHeader Accept-Language: en-GB,en-US;q=0.9,en;q=0.8<br>- BereqHeader Cache-Control: max-age=0<br>- BereqHeader Content-Type: application/x-www-form-urlencoded<br>- BereqHeader Cookie: check=true; AMCVS_232334B8527856F80A490D44%40AdobeOrg=1; cookieStatementDisplayedOnce=true; s_cc=true; k2c_Organization_cids=Organization_uaEMwVyUDe8mY5m; JOIN_JSESSIONID=dcd2c25b-a3e5-4a58-826c-e9452190265f; ttco=1543398034746; _gcl_au=1.1.1660607311.15<br>- BereqHeader Origin: <a href="https://development.organization.com" target="_blank" rel="noreferrer">https://development.organization.com</a><br>- BereqHeader Referer: <a href="https://development.organization.com/application/person" target="_blank" rel="noreferrer">https://development.organization.com/application/person</a><br>- BereqHeader Upgrade-Insecure-Requests: 1<br>- BereqHeader User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36 OPR/56.0.3051.104<br>- BereqHeader X-Forwarded-Port: 443<br>- BereqHeader X-Forwarded-Proto: https<br>- BereqHeader Content-Length: 888<br>- BereqHeader X-Forwarded-For: 2.2.2.2, 1.1.1.1<br>- BereqHeader xff: 2.2.2.2, 1.1.1.1<br>- BereqHeader host: <a href="http://development.organization.com" target="_blank" rel="noreferrer">development.organization.com</a><br>- BereqHeader Accept-Encoding: gzip<br>- BereqHeader X-Varnish: 196913<br>- BereqUnset X-Varnish: 196913<br>- BereqHeader X-Varnish: 262229<br>- VCL_call BACKEND_FETCH<br>- VCL_return fetch<br>- BackendOpen 24 boot.staticpages 127.0.0.1 82 127.0.0.1 40696<br>- BackendStart 127.0.0.1 82<br>- FetchError backend write error: 0 (Success)<br>- Timestamp Bereq: 1543416195.877756 5.116981 0.000046<br>- BackendClose 24 boot.staticpages<br>- Timestamp Beresp: 1543416195.877888 5.117113 0.000132<br>- Timestamp Error: 1543416195.877892 5.117117 0.000004<br>- BerespProtocol HTTP/1.1<br>- BerespStatus 503<br>- BerespReason Service Unavailable<br>- BerespReason Backend fetch failed<br>- BerespHeader Date: Wed, 28 Nov 2018 14:43:15 GMT<br>- BerespHeader Server: Varnish<br>- VCL_call BACKEND_ERROR<br>- BerespHeader Location: <a href="https://development.organization.com/errors/default/unknown.html" target="_blank" rel="noreferrer">https://development.organization.com/errors/default/unknown.html</a><br>- BerespStatus 302<br>- BerespReason Found<br>- VCL_return deliver<br>- Storage malloc Transient<br>- ObjProtocol HTTP/1.1<br>- ObjStatus 302<br>- ObjReason Found<br>- ObjHeader Date: Wed, 28 Nov 2018 14:43:15 GMT<br>- ObjHeader Server: Varnish<br>- ObjHeader Location: <a href="https://development.organization.com/errors/default/unknown.html" target="_blank" rel="noreferrer">https://development.organization.com/errors/default/unknown.html</a><br>- Length 0<br>- BereqAcct 7123 888 8011 489 0 489<br>- End<br clear="all"></div><div><div dir="ltr" class="m_5787149082928576820m_6300905689656486360m_-7920558716683778954m_1686636647750696316m_-2822693325006303334gmail_signature"><font size="2"><span style="font-family:verdana,sans-serif"><br>--------<br>Regards,</span><br style="font-family:verdana,sans-serif"><span style="font-family:verdana,sans-serif">Junaid</span></font><br></div></div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr">On Thu, Nov 29, 2018 at 10:34 PM Guillaume Quintard <<a href="mailto:guillaume@varnish-software.com" target="_blank" rel="noreferrer">guillaume@varnish-software.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div>Weird, are you able to show the two full backend request logs? (the one getting you the 500, then the 503)?</div><div><br></div><div>Kind regards,</div><div><br clear="all"><div><div dir="ltr" class="m_5787149082928576820m_6300905689656486360m_-7920558716683778954m_1686636647750696316m_-2822693325006303334m_4537629694372435432gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>-- <br></div>Guillaume Quintard<br></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr">On Wed, Nov 28, 2018 at 8:14 AM Junaid Mukhtar <<a href="mailto:junaid.mukhtar@gmail.com" target="_blank" rel="noreferrer">junaid.mukhtar@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small">hi Guys</div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small">I am facing a rather tricky issue, where it appears that the varnish is closing the backend connection without waiting for a respones from the backend. <br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small">We are using Nginx to serve static content Below is teh sequence of messages</div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small">Varnish sends POST request to App <br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small">App sends back 500 Internal Server Error</div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small">Varnish interprets the 500 internal Server Error <br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small">Varnish sends GET request to Nginx server on the same box to serve static content</div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small">Varnish shows error message (even though Nginx sends the response successfully within milliseconds)<br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small">- VCL_call BACKEND_FETCH<br>- VCL_return fetch<br>- BackendOpen 38 boot.staticpages 127.0.0.1 82 127.0.0.1 35064<br>- BackendStart 127.0.0.1 82<br>- FetchError backend write error: 0 (Success)<br>- Timestamp Bereq: 1543420795.016075 5.106813 0.000099<br>- BackendClose 38 boot.staticpages<br>- Timestamp Beresp: 1543420795.016497 5.107235 0.000422<br>- Timestamp Error: 1543420795.016503 5.107241 0.000005<br>- BerespProtocol HTTP/1.1<br>- BerespStatus 503<br>- BerespReason Service Unavailable<br>- BerespReason Backend fetch failed<br>- BerespHeader Date: Wed, 28 Nov 2018 15:59:55 GMT<br>- BerespHeader Server: Varnish<br>- VCL_call BACKEND_ERROR<br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small">Varnish then again goes the same Nginx server to display default content. <br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small">Nginx sends response and varnish accepts it and sends it back to the customer<br clear="all"></div><div><div dir="ltr" class="m_5787149082928576820m_6300905689656486360m_-7920558716683778954m_1686636647750696316m_-2822693325006303334m_4537629694372435432m_4913775676443336305gmail_signature"><br><font size="2"><span style="font-family:verdana,sans-serif"></span></font></div><div class="m_5787149082928576820m_6300905689656486360m_-7920558716683778954m_1686636647750696316m_-2822693325006303334m_4537629694372435432m_4913775676443336305gmail_signature"><font size="2"><span style="font-family:verdana,sans-serif"><span class="gmail_default" style="font-family:verdana,sans-serif;font-size:small">I am a bit stumped, any help with this is highly appreciated<br></span></span></font></div><div class="m_5787149082928576820m_6300905689656486360m_-7920558716683778954m_1686636647750696316m_-2822693325006303334m_4537629694372435432m_4913775676443336305gmail_signature"><font size="2"><span style="font-family:verdana,sans-serif"><span class="gmail_default" style="font-family:verdana,sans-serif;font-size:small"><br></span></span></font></div><div class="m_5787149082928576820m_6300905689656486360m_-7920558716683778954m_1686636647750696316m_-2822693325006303334m_4537629694372435432m_4913775676443336305gmail_signature"><font size="2"><span style="font-family:verdana,sans-serif"><span class="gmail_default" style="font-family:verdana,sans-serif;font-size:small"></span>--------<br></span></font></div><div dir="ltr" class="m_5787149082928576820m_6300905689656486360m_-7920558716683778954m_1686636647750696316m_-2822693325006303334m_4537629694372435432m_4913775676443336305gmail_signature"><font size="2"><span style="font-family:verdana,sans-serif">Regards,</span><br style="font-family:verdana,sans-serif"><span style="font-family:verdana,sans-serif">Junaid</span></font><br></div></div></div></div>
_______________________________________________<br>
varnish-misc mailing list<br>
<a href="mailto:varnish-misc@varnish-cache.org" target="_blank" rel="noreferrer">varnish-misc@varnish-cache.org</a><br>
<a href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc" rel="noreferrer noreferrer" target="_blank">https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc</a><br>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>