<p style="margin-top:0px;margin-right:10px;margin-bottom:1.5em;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;border-top-width:0px;border-right-width:0px;border-bottom-width:0px;border-left-width:0px;border-style:initial;border-color:initial;font:inherit;vertical-align:baseline;color:rgb(34,34,34);font-family:'Helvetica Neue',Arial,Helvetica,sans-serif;line-height:18px;background-color:rgb(255,255,255)">


Context: we have varnish fronting for a pool of servers that (semi-)automatically scale based on load criteria. A puppet job updates a vcl file defining all of the directors any time a server is added or removed from the pool, and then reloads the VCL.</p>


<p style="margin-top:0px;margin-right:10px;margin-bottom:1.5em;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;border-top-width:0px;border-right-width:0px;border-bottom-width:0px;border-left-width:0px;border-style:initial;border-color:initial;font:inherit;vertical-align:baseline;color:rgb(34,34,34);font-family:'Helvetica Neue',Arial,Helvetica,sans-serif;line-height:18px;background-color:rgb(255,255,255)">


In general, this works quite well. The only problem is that as time goes on and the pool grows and shrinks, the output from debug.health and debug.backends becomes increasingly...odd.</p><p style="margin-top:0px;margin-right:10px;margin-bottom:1.5em;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;border-top-width:0px;border-right-width:0px;border-bottom-width:0px;border-left-width:0px;border-style:initial;border-color:initial;font:inherit;vertical-align:baseline;color:rgb(34,34,34);font-family:'Helvetica Neue',Arial,Helvetica,sans-serif;line-height:18px;background-color:rgb(255,255,255)">


Case in point:</p><p style="margin-top:0px;margin-right:10px;margin-bottom:1.5em;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;border-top-width:0px;border-right-width:0px;border-bottom-width:0px;border-left-width:0px;border-style:initial;border-color:initial;font:inherit;vertical-align:baseline;color:rgb(34,34,34);font-family:'Helvetica Neue',Arial,Helvetica,sans-serif;line-height:18px;background-color:rgb(255,255,255)">


directors.vcl:<br style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px">director cms_director round-robin {<br style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px">


{ .backend = { .host = "10.62.87.78"; .port = "8080" ; .connect_timeout = 600s; .first_byte_timeout = 600s; .between_bytes_timeout = 600s; .probe = { .url = "/healthy"; .interval = 5s; .timeout = 20s; .window = 5; .threshold = 3; }} }<br style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px">


}</p><p style="margin-top:0px;margin-right:10px;margin-bottom:1.5em;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;border-top-width:0px;border-right-width:0px;border-bottom-width:0px;border-left-width:0px;border-style:initial;border-color:initial;font:inherit;vertical-align:baseline;color:rgb(34,34,34);font-family:'Helvetica Neue',Arial,Helvetica,sans-serif;line-height:18px;background-color:rgb(255,255,255)">


debug.backend:</p><p style="margin-top:0px;margin-right:10px;margin-bottom:1.5em;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;border-top-width:0px;border-right-width:0px;border-bottom-width:0px;border-left-width:0px;border-style:initial;border-color:initial;font:inherit;vertical-align:baseline;color:rgb(34,34,34);font-family:'Helvetica Neue',Arial,Helvetica,sans-serif;line-height:18px;background-color:rgb(255,255,255)">


0x7f83618dfc00 cms_director[0](10.251.39.31,(null),:8080) 3 0<br style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px">0x7f51686f8840 cms_director[1](10.62.87.78,(null),:8080) 2 2<br style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px">


0x7f516848f480 cms_director[0](10.62.87.78,(null),:8080) 5 1</p><p style="margin-top:0px;margin-right:10px;margin-bottom:1.5em;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;border-top-width:0px;border-right-width:0px;border-bottom-width:0px;border-left-width:0px;border-style:initial;border-color:initial;font:inherit;vertical-align:baseline;color:rgb(34,34,34);font-family:'Helvetica Neue',Arial,Helvetica,sans-serif;line-height:18px;background-color:rgb(255,255,255)">


debug.health:</p><p style="margin-top:0px;margin-right:10px;margin-bottom:1.5em;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;border-top-width:0px;border-right-width:0px;border-bottom-width:0px;border-left-width:0px;border-style:initial;border-color:initial;font:inherit;vertical-align:baseline;color:rgb(34,34,34);font-family:'Helvetica Neue',Arial,Helvetica,sans-serif;line-height:18px;background-color:rgb(255,255,255)">


Backend cms_director[0] is Sick<br style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px">Backend cms_director[1] is Healthy<br style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px">


Backend cms_director[0] is Healthy</p><p style="margin-top:0px;margin-right:10px;margin-bottom:1.5em;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;border-top-width:0px;border-right-width:0px;border-bottom-width:0px;border-left-width:0px;border-style:initial;border-color:initial;font:inherit;vertical-align:baseline;color:rgb(34,34,34);font-family:'Helvetica Neue',Arial,Helvetica,sans-serif;line-height:18px;background-color:rgb(255,255,255)">


Note that:</p><p style="margin-top:0px;margin-right:10px;margin-bottom:1.5em;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;border-top-width:0px;border-right-width:0px;border-bottom-width:0px;border-left-width:0px;border-style:initial;border-color:initial;font:inherit;vertical-align:baseline;color:rgb(34,34,34);font-family:'Helvetica Neue',Arial,Helvetica,sans-serif;line-height:18px;background-color:rgb(255,255,255)">


- there is a single cms backend defined in directors.vcl, but the debug output lists three, two of which have the same IP address.</p><p style="margin-top:0px;margin-right:10px;margin-bottom:1.5em;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;border-top-width:0px;border-right-width:0px;border-bottom-width:0px;border-left-width:0px;border-style:initial;border-color:initial;font:inherit;vertical-align:baseline;color:rgb(34,34,34);font-family:'Helvetica Neue',Arial,Helvetica,sans-serif;line-height:18px;background-color:rgb(255,255,255)">


- backend cms_director[0] is listed twice (?!)</p><p style="margin-top:0px;margin-right:10px;margin-bottom:1.5em;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;border-top-width:0px;border-right-width:0px;border-bottom-width:0px;border-left-width:0px;border-style:initial;border-color:initial;font:inherit;vertical-align:baseline;color:rgb(34,34,34);font-family:'Helvetica Neue',Arial,Helvetica,sans-serif;line-height:18px;background-color:rgb(255,255,255)">


Reloading the vcl with vcl.load;vcl.use does not change matters, but fully restarting the varnishd process will clear things up until after a while it recurs.</p><p style="margin-top:0px;margin-right:10px;margin-bottom:1.5em;margin-left:0px;padding-top:0px;padding-right:0px;padding-bottom:0px;padding-left:0px;border-top-width:0px;border-right-width:0px;border-bottom-width:0px;border-left-width:0px;border-style:initial;border-color:initial;font:inherit;vertical-align:baseline;color:rgb(34,34,34);font-family:'Helvetica Neue',Arial,Helvetica,sans-serif;line-height:18px;background-color:rgb(255,255,255)">


This is particularly aggravating as I would like to use the backends/health output in order to script rolling updates, but right now I cannot seem to count on it reflecting reality.</p>