<div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Dear List,</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">I'm debugging a problem with our Varnish, and our QA folks found an interesting case. <br><br>Ultimately this breadcrumb trail was discovered looking into our varnishes having an enormous number of open connections in CLOSE_WAIT status, in a way that doesn't appear to be ideal connection reuse, rather dead connections that we need to close out; non the less we're focused on a slightly more specific sub-issue of that issue right now:<br><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">We have noticed that if a request is eligible to be coalesced, but the impatient client disconnects before the request is served, Varnish continues to try and serve that request by going to the backend even after the client is disconnected.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">I suspect in our case, we can disable request coalescing, but I didn't want to miss an opportunity to report a possible bug, or learn something about a corner of Varnish I don't know well... here's our setup:</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">- With Varnish 6.6.1<br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">- Given a toy python script backend which answers 200OK to the health check, but answers after 10s to the other requests with an `HTTP 500` error; [source linked below]<br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">- Given the following config [below]<br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">  - started with `/usr/local/sbin/varnishd -n /usr/local/var/varnish -F -f $PWD/foo.vcl -a test=:21601,HTTP`<br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">- When running `for i in $(seq 10); do curl -m 1 localhost:21601/ &; done` (ampersand for background is important)<br></div>- Varnish makes 1 request to the backend, coalescing the others <br><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">- Clients all disconnect thanks to `curl -m 1` (`--max-time`)  (or `0.1`, no difference, naturally)<br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">- First request completed with `HTTP 500`, Varnish continues to retry requests for disconnected clients. (logs without health checks below)<br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">In the real set-up Varnish actually only handles the next request, then the next, then the next one each 10 seconds, I didn't take the time to reproduce that in this set-up as I believe it's a bug that Varnish continues to do work for disconnected clients. I guess in my toy we benefit from hit-for-pass, and in our real world setup that's not true. <br><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">I can somehow imagine this as a feature (populate the cache even though the client went away) but queueing hundreds or thousands of requests, and nibbling away at them one-by-one even after clients are long-since hung up is causing resource exhaustion for us; we can tune the configs significantly now that we know the issue, but we'd love to get some opinionated feedback on what would be an idiomatic approach to this.<br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">- Are we doing something wrong?</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">- Should varnish still go to the backend to serve disconnected clients?</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">- Is this a bug, should I report it somewhere more formally and attach the repro case a bit more diligently?</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Warm regards everyone, thanks for Varnish, an active list, and active support on StackOverflow and such.<br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">[config]<br>vcl 4.0;<br>backend foo {<br>        .host = "127.0.0.1";<br>        .port = "3100";<br>    .probe = {<br>      .url = "/healthz";<br>      .interval = 5s;<br>      .timeout = 1s;<br>      .window = 5;<br>      .threshold = 3;<br>    }<br>}<br>sub vcl_recv {<br>  return (hash);<br>}<br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">[toy python server logs]<br><div style="color:rgb(246,246,244);background-color:rgb(40,42,54);font-family:"Berkeley Mono",Menlo,Monaco,"Courier New",monospace,Menlo,Monaco,"Courier New",monospace;font-weight:normal;font-size:12px;line-height:18px;white-space:pre"><div><span style="color:rgb(231,238,152)">python3 ./server.py</span></div><div><span style="color:rgb(231,238,152)">socket binded to port 3100</span></div><div><span style="color:rgb(231,238,152)">2022-07-28 13:45:05.589997  socket is listening</span></div><div><span style="color:rgb(151,225,241)">2022-07-28 13:45:17.336371 from client</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)">  </span><span style="color:rgb(151,225,241)">b'GET / HTTP/1.1\r\nHost</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> </span><span style="color:rgb(151,225,241)">localhost:21601\r\nUser-Agent</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> </span><span style="color:rgb(151,225,241)">curl/7.68.0\r\nAccept</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> *</span><span style="color:rgb(151,225,241)">/*\r\nX-Forwarded-For</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> </span><span style="color:rgb(151,225,241)">::1\r\nX-Varnish</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> </span><span style="color:rgb(231,238,152)">32771\r\n\r\n'</span></div><div><span style="color:rgb(246,246,244)">                 </span><span style="color:rgb(231,238,152)">^^ first request</span></div><div><span style="color:rgb(151,225,241)">2022-07-28 13:45:27.425837 from client</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)">  </span><span style="color:rgb(151,225,241)">b'GET / HTTP/1.1\r\nHost</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> </span><span style="color:rgb(151,225,241)">localhost:21601\r\nUser-Agent</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> </span><span style="color:rgb(151,225,241)">curl/7.68.0\r\nAccept</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> *</span><span style="color:rgb(151,225,241)">/*\r\nX-Forwarded-For</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> </span><span style="color:rgb(151,225,241)">::1\r\nX-Varnish</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> </span><span style="color:rgb(231,238,152)">163841\r\n\r\n'</span></div><div><span style="color:rgb(151,225,241)">2022-07-28 13:45:27.437121 from client</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)">  </span><span style="color:rgb(151,225,241)">b'GET / HTTP/1.1\r\nHost</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> </span><span style="color:rgb(151,225,241)">localhost:21601\r\nUser-Agent</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> </span><span style="color:rgb(151,225,241)">curl/7.68.0\r\nAccept</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> *</span><span style="color:rgb(151,225,241)">/*\r\nX-Forwarded-For</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> </span><span style="color:rgb(151,225,241)">::1\r\nX-Varnish</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> </span><span style="color:rgb(231,238,152)">229377\r\n\r\n'</span></div><div><span style="color:rgb(151,225,241)">2022-07-28 13:45:27.437913 from client</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)">  </span><span style="color:rgb(151,225,241)">b'GET / HTTP/1.1\r\nHost</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> </span><span style="color:rgb(151,225,241)">localhost:21601\r\nUser-Agent</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> </span><span style="color:rgb(151,225,241)">curl/7.68.0\r\nAccept</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> *</span><span style="color:rgb(151,225,241)">/*\r\nX-Forwarded-For</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> </span><span style="color:rgb(151,225,241)">::1\r\nX-Varnish</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> </span><span style="color:rgb(231,238,152)">131075\r\n\r\n'</span></div><div><span style="color:rgb(151,225,241)">2022-07-28 13:45:27.437999 from client</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)">  </span><span style="color:rgb(151,225,241)">b'GET / HTTP/1.1\r\nHost</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> </span><span style="color:rgb(151,225,241)">localhost:21601\r\nUser-Agent</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> </span><span style="color:rgb(151,225,241)">curl/7.68.0\r\nAccept</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> *</span><span style="color:rgb(151,225,241)">/*\r\nX-Forwarded-For</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> </span><span style="color:rgb(151,225,241)">::1\r\nX-Varnish</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> </span><span style="color:rgb(231,238,152)">196609\r\n\r\n'</span></div><div><span style="color:rgb(151,225,241)">2022-07-28 13:45:27.438363 from client</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)">  </span><span style="color:rgb(151,225,241)">b'GET / HTTP/1.1\r\nHost</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> </span><span style="color:rgb(151,225,241)">localhost:21601\r\nUser-Agent</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> </span><span style="color:rgb(151,225,241)">curl/7.68.0\r\nAccept</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> *</span><span style="color:rgb(151,225,241)">/*\r\nX-Forwarded-For</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> </span><span style="color:rgb(151,225,241)">::1\r\nX-Varnish</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> </span><span style="color:rgb(231,238,152)">98311\r\n\r\n'</span></div><div><span style="color:rgb(151,225,241)">2022-07-28 13:45:27.437782 from client</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)">  </span><span style="color:rgb(151,225,241)">b'GET / HTTP/1.1\r\nHost</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> </span><span style="color:rgb(151,225,241)">localhost:21601\r\nUser-Agent</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> </span><span style="color:rgb(151,225,241)">curl/7.68.0\r\nAccept</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> *</span><span style="color:rgb(151,225,241)">/*\r\nX-Forwarded-For</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> </span><span style="color:rgb(151,225,241)">::1\r\nX-Varnish</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> </span><span style="color:rgb(231,238,152)">65545\r\n\r\n'</span></div><div><span style="color:rgb(151,225,241)">2022-07-28 13:45:27.438033 from client</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)">  </span><span style="color:rgb(151,225,241)">b'GET / HTTP/1.1\r\nHost</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> </span><span style="color:rgb(151,225,241)">localhost:21601\r\nUser-Agent</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> </span><span style="color:rgb(151,225,241)">curl/7.68.0\r\nAccept</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> *</span><span style="color:rgb(151,225,241)">/*\r\nX-Forwarded-For</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> </span><span style="color:rgb(151,225,241)">::1\r\nX-Varnish</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> </span><span style="color:rgb(231,238,152)">327681\r\n\r\n'</span></div><div><span style="color:rgb(151,225,241)">2022-07-28 13:45:27.439453 from client</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)">  </span><span style="color:rgb(151,225,241)">b'GET / HTTP/1.1\r\nHost</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> </span><span style="color:rgb(151,225,241)">localhost:21601\r\nUser-Agent</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> </span><span style="color:rgb(151,225,241)">curl/7.68.0\r\nAccept</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> *</span><span style="color:rgb(151,225,241)">/*\r\nX-Forwarded-For</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> </span><span style="color:rgb(151,225,241)">::1\r\nX-Varnish</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> </span><span style="color:rgb(231,238,152)">294913\r\n\r\n'</span></div><div><span style="color:rgb(151,225,241)">2022-07-28 13:45:27.438401 from client</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)">  </span><span style="color:rgb(151,225,241)">b'GET / HTTP/1.1\r\nHost</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> </span><span style="color:rgb(151,225,241)">localhost:21601\r\nUser-Agent</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> </span><span style="color:rgb(151,225,241)">curl/7.68.0\r\nAccept</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> *</span><span style="color:rgb(151,225,241)">/*\r\nX-Forwarded-For</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> </span><span style="color:rgb(151,225,241)">::1\r\nX-Varnish</span><span style="color:rgb(242,134,196)">:</span><span style="color:rgb(246,246,244)"> </span><span style="color:rgb(231,238,152)">262145\r\n\r\n'</span></div><div><span style="color:rgb(246,246,244)">                 </span><span style="color:rgb(231,238,152)">^^ other requests</span></div></div></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">[toy python server code]</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><a href="https://gist.github.com/leehambley/fa634f91936b1d30422e3af96ba2eec5">https://gist.github.com/leehambley/fa634f91936b1d30422e3af96ba2eec5</a> <br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>Lee Hambley<div><a href="http://lee.hambley.name/" target="_blank">http://lee.hambley.name/</a><br></div><div>+49 (0) 170 298 5667</div></div></div></div></div></div>