<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
  <head>

    <meta http-equiv="content-type" content="text/html; charset=ISO-8859-1">
  </head>
  <body bgcolor="#ffffff" text="#000000">
    Good day folks,<br>
    <br>
    First of all, varnish is an outstanding piece of software that my
    company and i are addicted to. So thanks to all the coders.<br>
    <br>
    Here is my problem :<br>
    I allocated varnish 1G of RAM on a website that can have more than 2
    Go of possible cacheable objects . Not to worry though as any
    proxy-cache system should smartly nuke old objects to make place to
    new one to live peacefully within its allocated RAM. And that's
    where Varnish behave unexpectedly : each time it need to nuke SOME
    objects : it nukes ALMOST ALL of them (often ~80% of my 35k objects)
    which is quite aggressive ; thus i lost almost all my cache....IRK !<br>
    <br>
    3 Munin graphs attached to see the problem clearly : big drop each
    time a nuking happens.<br>
    <br>
    To make sure my pbr is about varnish nuking system : i increased
    from 1G to 3G(more than the max possible 2G cacheable objects) on
    another varnish of this platefom (this website is delivered by
    multiple front/varnish server all stricly similar and independant)
    and this issue disappeared (no more nuking : no lost of ~80%of my
    objects)<br>
    <br>
    Here is my env : <br>
    Debian 5.0.8 64 bits on 2.6.32-5-openvz-amd64 kernel<br>
    Varnish 2.1.3 SVN 5049:5055(debian package 2.1.3-8) <br>
    200 varnish 's worker threads running constantly (no issue on
    workers)<br>
    30req/s average with 60/s in peak<br>
    <br>
    Daemon run as such :<br>
    /usr/sbin/varnishd -P /var/run/varnishd.pid -a :80 -T localhost:6082
    -S /etc/varnish/secret -f /etc/varnish/serverx.vcl -w 100,1024 -s
    file,/var/lib/varnish/serverx/varnish_storage.bin,3G<br>
    <br>
    Here a quick varnishstat -1 :<br>
    <blockquote><tt>client_conn          17177518         9.26 Client
        connections accepted</tt><br>
      <tt>client_drop                 0         0.00 Connection dropped,
        no sess/wrk</tt><br>
      <tt>client_req           31277524        16.86 Client requests
        received</tt><br>
      <tt>cache_hit            20909485        11.27 Cache hits</tt><br>
      <tt>cache_hitpass         2558712         1.38 Cache hits for pass</tt><br>
      <tt>cache_miss            5422557         2.92 Cache misses</tt><br>
      <tt>backend_conn           175175         0.09 Backend conn.
        success</tt><br>
      <tt>backend_unhealthy           0         0.00 Backend conn. not
        attempted</tt><br>
      <tt>backend_busy                0         0.00 Backend conn. too
        many</tt><br>
      <tt>backend_fail                0         0.00 Backend conn.
        failures</tt><br>
      <tt>backend_reuse        10192538         5.49 Backend conn.
        reuses</tt><br>
      <tt>backend_toolate         98629         0.05 Backend conn. was
        closed</tt><br>
      <tt>backend_recycle      10291261         5.55 Backend conn.
        recycles</tt><br>
      <tt>backend_unused              0         0.00 Backend conn.
        unused</tt><br>
      <tt>fetch_head             459553         0.25 Fetch head</tt><br>
      <tt>fetch_length          9907522         5.34 Fetch with Length</tt><br>
      <tt>fetch_chunked              33         0.00 Fetch chunked</tt><br>
      <tt>fetch_eof                   0         0.00 Fetch EOF</tt><br>
      <tt>fetch_bad                   0         0.00 Fetch had bad
        headers</tt><br>
      <tt>fetch_close                 7         0.00 Fetch wanted close</tt><br>
      <tt>fetch_oldhttp               0         0.00 Fetch pre HTTP/1.1
        closed</tt><br>
      <tt>fetch_zero                638         0.00 Fetch zero len</tt><br>
      <tt>fetch_failed                0         0.00 Fetch failed</tt><br>
      <tt>n_sess_mem               1189          .   N struct sess_mem</tt><br>
      <tt>n_sess                    890          .   N struct sess</tt><br>
      <tt>n_object                32678          .   N struct object</tt><br>
      <tt>n_vampireobject             0          .   N unresurrected
        objects</tt><br>
      <tt>n_objectcore            32794          .   N struct objectcore</tt><br>
      <tt>n_objecthead            30698          .   N struct objecthead</tt><br>
      <tt>n_smf                   68810          .   N struct smf</tt><br>
      <tt>n_smf_frag               3002          .   N small free smf</tt><br>
      <tt>n_smf_large               490          .   N large free smf</tt><br>
      <tt>n_vbe_conn                  8          .   N struct vbe_conn</tt><br>
      <tt>n_wrk                     200          .   N worker threads</tt><br>
      <tt>n_wrk_create              200         0.00 N worker threads
        created</tt><br>
      <tt>n_wrk_failed                0         0.00 N worker threads
        not created</tt><br>
      <tt>n_wrk_max                4133         0.00 N worker threads
        limited</tt><br>
      <tt>n_wrk_queue                 0         0.00 N queued work
        requests</tt><br>
      <tt>n_wrk_overflow              5         0.00 N overflowed work
        requests</tt><br>
      <tt>n_wrk_drop                  0         0.00 N dropped work
        requests</tt><br>
      <tt>n_backend                   1          .   N backends</tt><br>
      <tt>n_expired             2291302          .   N expired objects</tt><br>
      <b><tt>n_lru_nuked           3098052          .   N LRU nuked
          objects</tt></b><br>
      <tt>n_lru_saved                 0          .   N LRU saved objects</tt><br>
      <tt>n_lru_moved          19894617          .   N LRU moved objects</tt><br>
      <tt>n_deathrow                  0          .   N objects on
        deathrow</tt><br>
      <tt>losthdr                     0         0.00 HTTP header
        overflows</tt><br>
      <tt>n_objsendfile               0         0.00 Objects sent with
        sendfile</tt><br>
      <tt>n_objwrite           23874235        12.87 Objects sent with
        write</tt><br>
      <tt>n_objoverflow               0         0.00 Objects overflowing
        workspace</tt><br>
      <tt>s_sess               17177512         9.26 Total Sessions</tt><br>
      <tt>s_req                31277524        16.86 Total Requests</tt><br>
      <tt>s_pipe                      0         0.00 Total pipe</tt><br>
      <tt>s_pass                4945241         2.67 Total pass</tt><br>
      <tt>s_fetch              10367753         5.59 Total fetch</tt><br>
      <tt>s_hdrbytes        10732679592      5784.04 Total header bytes</tt><br>
      <tt>s_bodybytes      316828226025    170744.51 Total body bytes</tt><br>
      <tt>sess_closed           7071576         3.81 Session Closed</tt><br>
      <tt>sess_pipeline           13905         0.01 Session Pipeline</tt><br>
      <tt>sess_readahead           8085         0.00 Session Read Ahead</tt><br>
      <tt>sess_linger          24485521        13.20 Session Linger</tt><br>
      <tt>sess_herd            23385907        12.60 Session herd</tt><br>
      <tt>shm_records        1766166234       951.82 SHM records</tt><br>
      <tt>shm_writes          128900505        69.47 SHM writes</tt><br>
      <tt>shm_flushes             13215         0.01 SHM flushes due to
        overflow</tt><br>
      <tt>shm_cont                78095         0.04 SHM MTX contention</tt><br>
      <tt>shm_cycles                769         0.00 SHM cycles through
        buffer</tt><br>
      <tt>sm_nreq              18413160         9.92 allocator requests</tt><br>
      <tt>sm_nobj                 65318          .   outstanding
        allocations</tt><br>
      <tt>sm_balloc           824430592          .   bytes allocated</tt><br>
      <tt>sm_bfree            249311232          .   bytes free</tt><br>
      <tt>sma_nreq                    0         0.00 SMA allocator
        requests</tt><br>
      <tt>sma_nobj                    0          .   SMA outstanding
        allocations</tt><br>
      <tt>sma_nbytes                  0          .   SMA outstanding
        bytes</tt><br>
      <tt>sma_balloc                  0          .   SMA bytes allocated</tt><br>
      <tt>sma_bfree                   0          .   SMA bytes free</tt><br>
      <tt>sms_nreq                   61         0.00 SMS allocator
        requests</tt><br>
      <tt>sms_nobj                    0          .   SMS outstanding
        allocations</tt><br>
      <tt>sms_nbytes                  0          .   SMS outstanding
        bytes</tt><br>
      <tt>sms_balloc              16409          .   SMS bytes allocated</tt><br>
      <tt>sms_bfree               16409          .   SMS bytes freed</tt><br>
      <tt>backend_req          10367752         5.59 Backend requests
        made</tt><br>
      <tt>n_vcl                       1         0.00 N vcl total</tt><br>
      <tt>n_vcl_avail                 1         0.00 N vcl available</tt><br>
      <tt>n_vcl_discard               0         0.00 N vcl discarded</tt><br>
      <tt>n_purge                     1          .   N total active
        purges</tt><br>
      <tt>n_purge_add             12314         0.01 N new purges added</tt><br>
      <tt>n_purge_retire          12313         0.01 N old purges
        deleted</tt><br>
      <tt>n_purge_obj_test      2448163         1.32 N objects tested</tt><br>
      <tt>n_purge_re_test      62064275        33.45 N regexps tested
        against</tt><br>
      <tt>n_purge_dups             9524         0.01 N duplicate purges
        removed</tt><br>
      <tt>hcb_nolock           28886624        15.57 HCB Lookups without
        lock</tt><br>
      <tt>hcb_lock              4243837         2.29 HCB Lookups with
        lock</tt><br>
      <tt>hcb_insert            4243834         2.29 HCB Inserts</tt><br>
      <tt>esi_parse                   0         0.00 Objects ESI parsed
        (unlock)</tt><br>
      <tt>esi_errors                  0         0.00 ESI parse errors
        (unlock)</tt><br>
      <tt>accept_fail                 0         0.00 Accept failures</tt><br>
      <tt>client_drop_late            0         0.00 Connection dropped
        late</tt><br>
      <tt>uptime                1855569         1.00 Client uptime</tt><br>
    </blockquote>
    <br>
    Is it normal varnish behaviour ? sounds like a bug to me.<br>
    Am i missing some tuning (lru_interval)  to soften the nuking algo ?<br>
    Do you need more info ?<br>
    helps appreciated here  ;-)<br>
    <br>
    Regards, Aurelien Lemaire<br>
  </body>
</html>