This scheme seems very baroque. Why not just reduce the size of your caches so you don't page-thrash and let Varnish's builtin LRU algorithm handle the eviction?<div><br></div><div>--Michael<br><br><div class="gmail_quote">
On Sat, Jan 16, 2010 at 2:46 PM, David Birdsong <span dir="ltr"><<a href="mailto:david.birdsong@gmail.com">david.birdsong@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
I'm trying to hack my way around a push-button like lru nuking like<br>
feature. The short description of how I'm doing it follows, I'll<br>
explain why farther down.<br>
<br>
I have a job that watches sm_bfree / (sm_bfree + sm_balloc). Once<br>
storage file utilization is past some percentage(yet to be determined)<br>
I connect to upstream load balancers and slowly drain traffic away<br>
from varnish.<br>
<br>
Once traffic is off and I can beat the hell out of that box, it's time<br>
to free up some space. In the past this has been done with restarts.<br>
Upon restarts, the cache hit ratio is destroyed, but the box can keep<br>
up and rebuild the cache in a stable way. What I'd like to do is dump<br>
everything in the storage files that have a very low obj.hits. Lru<br>
nuking on the surface seems like the best thing to initiate, but it<br>
usually only kicks in pretty late and puts the machine into a state<br>
that is unstable while serving. While not serving, I don't know how<br>
to kick it off, furthermore I want it to run hard and free up lots<br>
more space than it usually does.<br>
<br>
ie. cache file is ~200GB, I'd like it to run until sm_free is like 50GB.<br>
<br>
My idea is to load balance as I've described above. Pull 50GB of<br>
trash files through the cache + enough to kick off lru, purge the<br>
trash files, monitor sm_bfree and once it's high enough instruct the<br>
upstream load balancers to start sending traffic gently for a warm up<br>
period. Rinse and repeat into infinity replacing the ssd storage<br>
drives as they fail. Is this crazy? Am I uninformed on a better way?<br>
<br>
Also, I've had to keep making my trash files smaller and smaller. I<br>
started with a 10 and 1G files which crashed varnish immediately, then<br>
reduced to 500MB files and successfully pulled 200 through - then<br>
crashed both my python interpreter (libcurl) and varnish:<br>
varnishd[2664]: Child (14772) Panic message: Assert error in<br>
STV_alloc(), stevedore.c line 183:#012 Condition((st) != NULL) not<br>
true.#012thread = (cache-worker)#012Backtrace:#012 0x421f95:<br>
pan_ic+85#012 0x4369e5: STV_alloc+125#012 0x41a1b6:<br>
FetchBody+496#012 0x4114dd: cnt_fetch+63d#012 0x412a3d:<br>
CNT_Session+35d#012 0x424273: wrk_do_cnt_sess+93#012 0x42362e:<br>
wrk_thread_real+26e#012 0x7f2cf51b83da: _end+7f2cf4b47c1a#012<br>
0x7f2cf4a862bd: _end+7f2cf4415afd#012sp = 0x7f2ced387008 {#012 fd =<br>
58, id = 58, xid = 1454039386,#012 client = <a href="http://127.0.0.1:7057" target="_blank">127.0.0.1:7057</a>,#012 step<br>
= STP_FETCH,#012 handling = deliver,#012 err_code = 200, err_reason<br>
= (null),#012 restarts = 0, esis = 0#012 ws = 0x7f2ced387078 { #012<br>
id = "sess",#012 {s,f,r,e} =<br>
{0x7f2ced387800,+144,(nil),+4096},#012 },#012 http[req] = {#012<br>
ws = 0x7f2ced387078[sess]#012 "GET",#012<br>
"/lru.10.cache.buster.80.12994",#012 "HTTP/1.1",#012<br>
"User-Agent: PycURL/7.18.2",#012 "Host: localhost:6081",#012<br>
"Accept: */*",#012 },#012 worker = 0x7ef439f06390 {#012 ws =<br>
0x7ef439f068f0 { #012 id = "wrk",#012 {s,f,r,e} =<br>
{0x7ef439f03350,+2143,(nil),+4096},#012 },#012 http[bereq] =<br>
{#012 ws = 0x7ef439f068f0[wrk]#012 "GET",#012<br>
"/lru.10.cache.buster.80.12994",#012 "HTTP/1.1",#012<br>
"User-Agent: PycURL/7.18.2",#012 "Host: localhost:6081",#012<br>
"Accept: */*",#012 "X-Varnish: 1454039386",#012<br>
"X-Forwarded-For: 127.0.0.1",#012 },#012 http[beresp] = {#012<br>
ws = 0x7ef439f068f0[wrk]#012 "HTTP/1.1",#012<br>
"200",#012 "OK",#012 "Server: nginx/0.7.64",#012<br>
"Date: Sat, 16 Jan 2010 21:11:09 GMT",#012 "Content-Type:<br>
application/octet-stream",#012 "Content-Length: 524288000",#012<br>
"Last-Modified: Sat, 16 Jan 2010 21:08:11 GMT",#012<br>
"Connection: keep-alive",#012 "Accept-Ranges: bytes",#012<br>
"X-Varnish-IP: 127.0.0.1",#012 "X-Varnish-Port: 6081",#012<br>
},#012 },#012<br>
<br>
Are big files bad? I expect that I'll have to close a pretty big gap<br>
normally given that my 4 storage files are 75GB each (SSD). I'd like<br>
to start this process before lru nuking happens on it's own while<br>
varnish is not unloaded by upstream load balancers. My guess based on<br>
loose recollection is that varnish will start lru nuking at 90%<br>
capacity. It may just prove not feasible given that I'll have to pull<br>
roughly 60GB through to achieve the goal....perhaps freeing up a<br>
smaller percentage would be acceptable too though. I'm still playing<br>
with this, but wanted to share my uber-hacky idea and let you guys<br>
tear it apart if it's a dumb idea.<br>
<br>
Why:<br>
Identifying the working set has been difficult. It's large, the long<br>
tail is very long. I've tried adaptive ttls to expire objects<br>
constantly that shouldn't be in cache:<br>
<br>
in vcl_fetch: set every new object to a 2hr ttl.<br>
in vcl_hit: if obt.hits == N ; then obj.ttl = 36 hours, where N is<br>
some number that is high enough to cache<br>
another permutation, update the vcl every 30 mins such that obj.ttl<br>
was set to expire exactly at the trough of traffic (2300 - 2350 PST)<br>
in vcl_hit: if obt.hits = N ; then obj.ttl = 12h or 10h, or 3h<br>
(depending on time of day)<br>
<br>
This just ended up affecting cache hit ratio such that it was never<br>
favorable and the box was just busier as it was constantly expiring<br>
objects over the day. Restarts were still better than this.<br>
<br>
Setup:<br>
<br>
3 haproxy load balancer machines consistently hashing to 6 varnish<br>
instances. It's a prototype and will be scaled to a larger pool, so<br>
the impact of the downtime of a single varnish instance while it goes<br>
through a cache storage scrubbing is will be greatly reduced.<br>
_______________________________________________<br>
varnish-misc mailing list<br>
<a href="mailto:varnish-misc@projects.linpro.no">varnish-misc@projects.linpro.no</a><br>
<a href="http://projects.linpro.no/mailman/listinfo/varnish-misc" target="_blank">http://projects.linpro.no/mailman/listinfo/varnish-misc</a><br>
</blockquote></div><br></div>