[Varnish] #306: Throwing 503 Service Unavailable despite healthy backend in director

Varnish varnish-bugs at projects.linpro.no
Mon Sep 1 17:50:06 CEST 2008


#306: Throwing 503 Service Unavailable despite healthy backend in director
----------------------+-----------------------------------------------------
 Reporter:  sensei    |       Owner:  phk                      
     Type:  defect    |      Status:  new                      
 Priority:  normal    |   Milestone:  Varnish 2.0 code complete
Component:  varnishd  |     Version:  2.0                      
 Severity:  normal    |    Keywords:  backend polling          
----------------------+-----------------------------------------------------
 I'm setting up Varnish with a random director and two backends, then make
 a request which is served from node1 and cached. Then I kill node1 and
 purge the cache, I then get a 503 on subsequent reququests even if there's
 a healthy backend in the director (node0). It refuses to use the other
 backend until the node1 is back up and marked as healthy.

 VCL:

 backend node0 {
   .host = "127.0.0.1";
   .port = "80";
   .probe = {
            .url = "/";
            .timeout = 50 ms;
            .interval = 1s;
            .window = 10;
            .threshold = 8;
   }
 }

 backend node1 {
   .host = "81.29.85.44";
   .port = "80";
   .probe = {
            .url = "/";
            .timeout = 100 ms;
            .interval = 1s;
            .window = 10;
            .threshold = 8;
   }
 }

 director cl1 random {
     { .backend = node1; .weight = 1; }
     { .backend = node0; .weight = 1; }
 }


 #director cl1 round-robin {
 #    { .backend = node0; }
 #    { .backend = node1; }
 #}

 sub vcl_recv {
         set req.backend = cl1;
 }

 So if I send a request and it goes to node1 and I then take node1 down and
 purge the cache, varnishd correctly marks it as sick, but it won't send
 subsequent requests to node0.
 This only happens for one node at a time. So if I in the example above
 were to take node0 down, and have node1 as healthy, it would not have this
 behaviour, but quite happily serve from node1.

 I haven't tried enough times to get any absolute statistical certainty,
 but it appears as this only applies to the backend defined first in the
 director. So if I swap them around in the above VCL to:

 director cl1 random {
     { .backend = node0; .weight = 1; }
     { .backend = node1; .weight = 1; }
 }

 it would pick up if node1 went down and only serve from node0 without any
 503 errors, but not the other way around.

 This is on CentOS 4.6 with varnish-2.0-beta1. Had this in 3136 as well.
 GCC 3.4.6, kernel 2.6.9-67.0.15.plus.c4, glibc 2.3.4 for what it's worth.


 Hope this isn't too confusing!

 Cheers,
 sensei

-- 
Ticket URL: <http://varnish.projects.linpro.no/ticket/306>
Varnish <http://varnish.projects.linpro.no/>
The Varnish HTTP Accelerator


More information about the varnish-bugs mailing list