[Varnish] #985: Assert error in wrk_thread_real

[Varnish]

#985: Assert error in wrk_thread_real
----------------------------+-----------------------------------------------
 Reporter:  davidlc         |        Type:  defect  
   Status:  new             |    Priority:  normal  
Milestone:                  |   Component:  varnishd
  Version:  trunk           |    Severity:  critical
 Keywords:  transient #953  |  
----------------------------+-----------------------------------------------
 After being severly bitten by bug #953, I managed to limit the transient
 storage size thanks to the doc commited on 2011/08/17. Using a very little
 memory setting to test proper nuking:


 {{{
 DAEMON_OPTS="-a :80 -p shortlived=1 -s malloc,3M -s Transient=malloc,3M \
 -f /etc/varnish/varnish.vcl -n /var/run/varnish -p thread_pool_max=2000 -p
 \
 thread_pools=2 -p lru_interval=6 -p thread_pool_min=100 -p
 thread_pool_add_delay=2 \
 -p session_linger=100 -p listen_depth=4096 -p sess_workspace=65536 -p
 ping_interval=2 \
 -p sess_timeout=1 -t 120 -T 127.0.0.1:8888"
 }}}

 I launch a script requesting thousands of URLs.
 Then every 12s or so, the varnish instance crashes with:

 {{{

 /var/run/varnish[10287]: Child (16198) Panic message: Assert error in
 wrk_thread_real(), cache_pool.c line 187:
 Condition((w->bereq->ws) == 0) not true.
 thread = (cache-worker)
 ident = Linux,2.6.18-238.12.1.el5,x86_64,-smalloc,-smalloc,-hcritbit,epoll
 Backtrace:
 0x42b386: pan_ic+b6
 0x42cb97: wrk_thread_real+427
 0x311fa0673d: _end+311f393d65
 0x311f2d44bd: _end+311ec61ae5
 }}}

 The varnishstat output just before a crash looks like:
 [http://pastebin.com/RU41NRxX]

 The OS is CentOS 5.6

 The VCL used is empty/default: just a backend.


 {{{
 backend default {
      .host = "mybackend";
      .port = "80";
      .saintmode_threshold = 20000;
      .probe = {
                 .request =
                 "GET /test.html HTTP/1.1"
                 "Host: lbcheck"
                 "Connection: close";
                 .timeout = 0.2 s;
                 .interval = 1s;
                 .window = 4;
                 .threshold = 2;
         }
 }
 }}}


 The served objects are quite big and numerous (filling this small cache).
 But they should have a large TTL as the backend makes them expire in a day
 (dont ask why we cache video chunks ;):


 {{{
    11 TxHeader     c Server: nginx
    11 TxHeader     c Content-Type: video/MP2T
    11 TxHeader     c Last-Modified: Fri, 06 May 2011 00:13:41 GMT
    11 TxHeader     c Expires: Sat, 20 Aug 2011 15:21:34 GMT
    11 TxHeader     c Cache-Control: max-age=86400
    11 TxHeader     c Content-Length: 19740
    11 TxHeader     c Accept-Ranges: bytes
    11 TxHeader     c Date: Fri, 19 Aug 2011 15:21:34 GMT
    11 TxHeader     c X-Varnish: 1136508484
    11 TxHeader     c Age: 0
    11 TxHeader     c Via: 1.1 varnish
 }}}

-- 
Ticket URL: <http://www.varnish-cache.org/trac/ticket/985>
Varnish <http://varnish-cache.org/>
The Varnish HTTP Accelerator