[Varnish] #1696: Varnish leaves connections in CLOSE_WAIT

[Varnish]

#1696: Varnish leaves connections in CLOSE_WAIT
----------------------+--------------------
 Reporter:  rwimmer   |       Owner:
     Type:  defect    |      Status:  new
 Priority:  high      |   Milestone:
Component:  build     |     Version:  3.0.5
 Severity:  critical  |  Resolution:
 Keywords:            |
----------------------+--------------------

Comment (by rwimmer):

 We were able to nail the problem further down but for us it's a little bit
 wired because it's exactly one URI causing the problem and it looks like
 this: /foo/bar/suggest/heroTeaser/

 And ONLY this syntax! "/foo/bar/suggest/heroTeaser" (without / at the end)
 and "/foo/bar/suggest/heroTeaser/?some_query_string=value" works as
 expected. The cause seems to be in vcl_fetch. The original code looks like
 this:

 {{{
 ...
     if (beresp.http.Content-Type ~ "text/html" || beresp.http.Content-Type
 ~ "text/xml" || beresp.http.Content-Type ~ "application/json") {
         if ((beresp.http.Set-Cookie ~ "NO_CACHE=") || (beresp.ttl < 1s)) {
             set beresp.ttl = 0s;
             return (hit_for_pass);
         }
 ...
 }}}

 With this VCL code we see connections in CLOSE_WAIT steadily growing (ok
 we now know that this code isn't perfect but it shouldn't matter). If we
 change the code above to

 {{{
 ...
     if (beresp.http.Content-Type ~ "text/html" || beresp.http.Content-Type
 ~ "text/xml" || beresp.http.Content-Type ~ "application/json") {
         if ((beresp.http.Set-Cookie ~ "NO_CACHE=") || (beresp.ttl < 1s)) {
             set beresp.ttl = 0s;
             if (req.url ~ "^/foo/bar/suggest/heroTeaser/$") {
                 return (deliver);
             }
             return (hit_for_pass);
         }
 ...
 }}}

 everything is ok. No further growing in CLOSE_WAIT connections! We looked
 at the Varnish flow graph (https://www.varnish-
 cache.org/trac/attachment/wiki/VCLExampleDefault/varnish_flow_3_0.png)
 what part could case this behavior and finally discovered that we need to
 add some code to vcl_miss to prevent vcl_fetch called next for this URI.
 vcl_pass should be called instead. So we added the following VCL code:

 {{{
 sub vcl_miss {
    if (req.url ~ "^/foo/bar/suggest/heroTeaser/$") {
       return (pass);
    }
 }
 }}}

 Additionally we switched back to the original code in vcl_fetch mentioned
 above:

 {{{
 ...
     if (beresp.http.Content-Type ~ "text/html" || beresp.http.Content-Type
 ~ "text/xml" || beresp.http.Content-Type ~ "application/json") {
         if ((beresp.http.Set-Cookie ~ "NO_CACHE=") || (beresp.ttl < 1s)) {
             set beresp.ttl = 0s;
             return (hit_for_pass);
         }
 ...
 }}}

 If you look at the Varnish flow graph the main difference is that after
 vcl_pass an anon object is created which is completely skipped if
 vcl_fetch is called after vcl_miss. So adding the vcl_miss code we changed
 the execution path and that worked. Without vcl_miss and the "if" clause
 in vcl_miss a simple "curl
 http://somdomain.tld/foo/bar/suggest/heroTeaser/" caused a timeout after 5
 minutes (tcp timeout) and no request to the backend. "curl
 http://domain.tld/foo/bar/suggest/heroTeaser" and "curl
 http://domain.tld/foo/bar/suggest/heroTeaser/?some_query_string=value"
 returned immediately.

 It shouldn't matter how wrong the VCL code is in this case. At least an
 error should occur and not tens of thousands of connectsion in CLOSE_WAIT.
 We suspect that it has something to do with "hit_for_pass" and the anon
 object that is only created after vcl_pass was called but not if vcl_miss
 was called and vcl_fetch is executed next.

-- 
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1696#comment:2>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator