I believe we can achieve a 100% graceful restart if we can get Varnish to send a HTTP Connection: Close header for all persistent HTTP connections. It would work something like this:<div><br></div><div>1. Mark Varnish server <i>x</i> as out of service in F5</div>
<div>2. Tell Varnish server <i>x</i> to start closing all persistent HTTP connections by sending HTTP Connection: Close headers. After a HTTP client sees the Connection: Close header, it should open up a new HTTP connection for the next HTTP request, which the F5 will route to the other, in-service Varnish server.</div>
<div>3. Wait for all HTTP connections to drain from Varnish server <i>x</i></div><div>4. Service the machine.</div><div><br></div><div>Steve<br><br><div class="gmail_quote">On Tue, Jul 17, 2012 at 10:20 AM, Hettwer, Marian <span dir="ltr"><<a href="mailto:mhettwer@team.mobile.de" target="_blank">mhettwer@team.mobile.de</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Actually the F5 box would have the same problem as you're describing for<br>
varnish.<br>
If the F5 box behaves correctly, marking the varnish out of service means<br>
that the F5 should not accept new connections, but is also not allowed to<br>
drop existing connections.<br>
<br>
Therefor I do wonder how the F5 machine would solve this without errors to<br>
the client.<br>
I would bet it also aggressively shuts down existing connections.<br>
<br>
Or if it doesn't, well okay, then there is nearly no way around it for<br>
doing it on the varnish.<br>
<br>
<br>
You might think about installing two varnish boxes in an active/passive<br>
carp(4) and pfsync(4) setup. But that means you probably have to go with<br>
FreeBSD on your server. I'm not sure about implementations of carp(4) and<br>
pfsync(4) for linux.<br>
<br>
<br>
With carp you could do the fail over you wanna do.<br>
While thinking about it, active/active carp(4) would work too.<br>
<br>
So carp(4) / pfsync(4) for the win it is! :-D<br>
<br>
HTH,<br>
Marian<br>
<div class="HOEnZb"><div class="h5"><br>
On 09.07.12 23:24, "Steven Engelhardt" <<a href="mailto:sengelha@gmail.com">sengelha@gmail.com</a>> wrote:<br>
<br>
>Because of the usage patterns of the web service, I'm not sure that even<br>
>a 1-second timeout would be sufficient to get all HTTP client connections<br>
>to close.<br>
><br>
><br>
>The idea of setting sess_timeout to be 0 was strictly in the context of<br>
>taking a Varnish server out of service, in order to (somewhat indirectly)<br>
>force the out of service Varnish server to close all its HTTP<br>
>connections. We would restore sess_timeout to a<br>
> sensible value before putting it back in service.<br>
><br>
><br>
>Steve<br>
><br>
>On Mon, Jul 9, 2012 at 4:07 PM, Javier Casares<br>
><<a href="mailto:javier@casares.org">javier@casares.org</a>> wrote:<br>
><br>
>If sess_timeout is set by default to 5 seconds, you could try to set<br>
>this to 4, 3, 2 or 1 and test if has a better performance... i suposse<br>
>that set this to 0 will close all connectiond and seems to be an epic<br>
>fail...<br>
><br>
>Javier Casares<br>
><a href="http://javiercasares.com/" target="_blank">http://javiercasares.com/</a><br>
><br>
><br>
>2012/7/9 Steven Engelhardt <<a href="mailto:sengelha@gmail.com">sengelha@gmail.com</a>>:<br>
>> Just to be clear, would you suggest something like:<br>
>> 1. Mark the Varnish server as disabled in the BIG-IP<br>
>> 2. Use varnishadm to set sess_timeout to 0 to start aggressively closing<br>
>> HTTP requests?<br>
>> 3. Wait for connections to drain<br>
>><br>
>> Steve<br>
>><br>
>> On Mon, Jul 9, 2012 at 3:59 PM, Javier Casares <<a href="mailto:javier@casares.org">javier@casares.org</a>><br>
>>wrote:<br>
>>><br>
>>><br>
><a href="https://www.varnish-cache.org/docs/3.0/reference/varnishd.html" target="_blank">https://www.varnish-cache.org/docs/3.0/reference/varnishd.html</a><br>
><<a href="https://www.varnish-cache.org/docs/3.0/reference/varnishd.html" target="_blank">https://www.varnish-cache.org/docs/3.0/reference/varnishd.html</a>><br>
>>><br>
>>> sess_timeout<br>
>>><br>
>>> Units: seconds<br>
>>> Default: 5<br>
>>><br>
>>> Idle timeout for persistent sessions. If a HTTP request has not<br>
>>> been received in this many seconds, the session is closed.<br>
>>><br>
>>> ¿?<br>
>>><br>
>>> Javier Casares<br>
>>> <a href="http://javiercasares.com/" target="_blank">http://javiercasares.com/</a><br>
>>><br>
>>><br>
>>> 2012/7/9 Steven Engelhardt <<a href="mailto:sengelha@gmail.com">sengelha@gmail.com</a>>:<br>
>>> > I have two Varnish servers behind a F5 BIG-IP load balancer. Both<br>
>>> > servers<br>
>>> > are constantly active and serving ~5,000 requests/second 24x7.<br>
>>>Clients<br>
>>> > aggressively use HTTP 1.1 Keep-Alives.<br>
>>> ><br>
>>> > I'm looking to develop a process for servicing and upgrading the<br>
>>>Varnish<br>
>>> > servers which results in 0 downtime to the client. Our normal<br>
>>>process<br>
>>> > for<br>
>>> > performing changes on production servers is:<br>
>>> > 1. Mark the server as disabled in the BIG-IP<br>
>>> > 2. Wait for the connections on the server to drop to 0<br>
>>> > 3. Service the machine<br>
>>> ><br>
>>> > However, because of the aggressive and constant use of HTTP<br>
>>>Keep-Alives,<br>
>>> > clients virtually never drop their connections, and the machine<br>
>>> > continues to<br>
>>> > serve traffic for quite a long time after it is disabled in the<br>
>>>BIG-IP.<br>
>>> ><br>
>>> > Is there a way to tell Varnish to aggressively close all client HTTP<br>
>>> > connections?<br>
>>> ><br>
>>> > Thank you,<br>
>>> > Steve<br>
>>> ><br>
>>> > --<br>
>>> > Steven Engelhardt<br>
>>> > <a href="mailto:sengelha@gmail.com">sengelha@gmail.com</a><br>
>>> ><br>
>>> > _______________________________________________<br>
>>> > varnish-misc mailing list<br>
>>> > <a href="mailto:varnish-misc@varnish-cache.org">varnish-misc@varnish-cache.org</a><br>
>>> ><br>
><a href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc" target="_blank">https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc</a><br>
><<a href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc" target="_blank">https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc</a>><br>
>>><br>
>>> _______________________________________________<br>
>>> varnish-misc mailing list<br>
>>> <a href="mailto:varnish-misc@varnish-cache.org">varnish-misc@varnish-cache.org</a><br>
>>><br>
><a href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc" target="_blank">https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc</a><br>
><<a href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc" target="_blank">https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc</a>><br>
>><br>
>><br>
>><br>
>><br>
>> --<br>
>> Steven Engelhardt<br>
>> <a href="mailto:sengelha@gmail.com">sengelha@gmail.com</a><br>
>><br>
>> _______________________________________________<br>
>> varnish-misc mailing list<br>
>> <a href="mailto:varnish-misc@varnish-cache.org">varnish-misc@varnish-cache.org</a><br>
>><br>
><a href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc" target="_blank">https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc</a><br>
><<a href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc" target="_blank">https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc</a>><br>
><br>
>_______________________________________________<br>
>varnish-misc mailing list<br>
><a href="mailto:varnish-misc@varnish-cache.org">varnish-misc@varnish-cache.org</a><br>
><a href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc" target="_blank">https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc</a><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
>--<br>
>Steven Engelhardt<br>
><a href="mailto:sengelha@gmail.com">sengelha@gmail.com</a><br>
><br>
<br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>Steven Engelhardt<br><a href="mailto:sengelha@gmail.com">sengelha@gmail.com</a><br>
</div>