[Varnish] #538: [varnish-2.0.4] Potential Memory Leak

[Varnish]

#538: [varnish-2.0.4] Potential Memory Leak
-------------------------------+--------------------------------------------
 Reporter:  pprocacci          |        Owner:  phk  
     Type:  defect             |       Status:  new  
 Priority:  high               |    Milestone:       
Component:  varnishd           |      Version:  trunk
 Severity:  major              |   Resolution:       
 Keywords:  Memory Leak 2.0.4  |  
-------------------------------+--------------------------------------------
Comment (by kb):

 Your pmap shows that varnish is correctly allocating the 300MB you
 specified.  The rest of the RAM is spent on the compiled config (always
 about 80MB in my experience) and 10MB of preallocated stack per worker
 thread.  10MB is apparently your distribution's default stacksize (see
 'ulimit -a' or 'limit').

 There's no leak here, just a fact of life with pthreads.  Because you have
 a very small cache size, the overhead ratio is exaggerated.

 You can reduce the memory usage of each pthread by reducing your default
 stacksize.  Run 'ulimit -s 1024' or 'limit stacksize 1024' before starting
 varnishd and you should see the memory use drop from 600MB to something
 like 450MB, and scaling out worker threads will become an order of
 magnitude less RAM-intensive.

 Like every other server out there that lets you specify RAM usage for a
 cache (e.g., MySQL, squid) there are many other factors that determine
 memory usage -- some within the app's control, some not.  Managing MySQL
 memory usage is extremely similar to varnishd -- a base cache, thread-
 specific buffers, and the default stacksize that comes preallocated with
 every pthread.  MySQL explicitly sets the stacksize for pthreads it
 creates to 128KB because of the high memory usage that on the surface
 looks like a leak.

 FWIW, I have a patch to varnishd that allows control of the worker
 threads' stacksize in much the same way MySQL controls it.  See #572 if
 you're interested.

 Ken

-- 
Ticket URL: <http://varnish.projects.linpro.no/ticket/538#comment:10>
Varnish <http://varnish.projects.linpro.no/>
The Varnish HTTP Accelerator