<div dir="ltr">Thanks for your reply Dridi,I will try to increase the sess_workerspace to see what will happen in next day, thanks.</div><div class="gmail_extra"><br><br><div class="gmail_quote">2013/4/11 Dridi Boukelmoune <span dir="ltr"><<a href="mailto:dridi.boukelmoune@zenika.com" target="_blank">dridi.boukelmoune@zenika.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
According to the log you provided, I understand that you have a failed<br>
allocation in your worker thread (WS_Alloc returns NULL). Check the<br>
n_objoverflow counter in varnishstat. You can control the size of your<br>
workspace with the sess_workspace. The more you do in your VCL, the<br>
more memory you'll likely need for workers.<br>
<br>
Best Regards,<br>
Dridi<br>
<div><div class="h5"><br>
On Thu, Apr 11, 2013 at 3:25 AM, Gao Yongwei <<a href="mailto:itxx00@gmail.com">itxx00@gmail.com</a>> wrote:<br>
> Hello,lists.<br>
> I an using a varnish for testing on a CentOS6.3 box, some basic info:<br>
> [root@cdn001 ~]# uname -r<br>
> 2.6.32-358.2.1.el6.x86_64<br>
> [root@cdn001 ~]# rpm -q varnish<br>
> varnish-3.0.3-3.el6.art.x86_64<br>
> [root@cdn001 ~]# free -m<br>
> total used free shared buffers cached<br>
> Mem: 24023 1054 22968 0 137 240<br>
> -/+ buffers/cache: 676 23347<br>
> Swap: 12079 0 12079<br>
> uptime:<br>
> 09:05:24 up 6 days, 11:29, 1 user, load average: 0.00, 0.00, 0.00<br>
><br>
> and my /etc/sysconfig/varnish looks like below:<br>
><br>
> WORKER_STACK_SIZE=512<br>
> ... ...<br>
> VARNISH_STORAGE_SIZE=16G<br>
> VARNISH_STORAGE="malloc,${VARNISH_STORAGE_SIZE}"<br>
> VARNISH_TTL=900<br>
> thread_pools=2<br>
> thread_pool_min=500<br>
> thread_pool_max=4000<br>
> thread_pool_timeout=120<br>
> thread_pool_add_delay=2<br>
> thread_pool_fail_delay=100<br>
> sess_workspace=32768<br>
> session_max=500000<br>
> thread_pool_stack=16384<br>
> connect_timeout=10<br>
> first_byte_timeout=60<br>
> between_bytes_timeout=60<br>
> DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \<br>
> -f ${VARNISH_VCL_CONF} \<br>
> -T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT}<br>
> \<br>
> -t ${VARNISH_TTL} \<br>
> -u varnish -g varnish \<br>
> -S ${VARNISH_SECRET_FILE} \<br>
> -p thread_pools=${thread_pools} \<br>
> -p thread_pool_min=${thread_pool_min} \<br>
> -p thread_pool_max=${thread_pool_max} \<br>
> -p thread_pool_timeout=${thread_pool_timeout} \<br>
> -p thread_pool_add_delay=${thread_pool_add_delay} \<br>
> -p thread_pool_fail_delay=${thread_pool_fail_delay} \<br>
> -p sess_workspace=${sess_workspace} \<br>
> -p session_max=${session_max} \<br>
> -p connect_timeout=${connect_timeout} \<br>
> -p first_byte_timeout=${first_byte_timeout} \<br>
> -p between_bytes_timeout=${between_bytes_timeout} \<br>
> -s ${VARNISH_STORAGE}"<br>
><br>
> as my backend web servers runs in many vms,so I use ' dns director':<br>
><br>
> director dnsdomain dns {<br>
> .list = {<br>
> .port = "80";<br>
> "10.0.0.0"/24;<br>
> }<br>
> .ttl = 12h;<br>
> }<br>
><br>
> and the connect_timeout in /etc/sysconfig/varnish has been set to a large<br>
> number(10 s).<br>
> everything works good since last night, I got a panic message in system log.<br>
> By default I have 1000 workers on startup, but when this panic occur,I can<br>
> just say 500+ workers<br>
> using varnishstat command. bellow is the panic message in syslog:<br>
><br>
> Apr 10 22:12:58 cdn001 varnishd[25730]: Child (25731) Panic message: Assert<br>
> error in VRT_IP_string(), cache_vrt.c line 312:#012 Condition((p =<br>
> WS_Alloc(sp->http->ws, len)) != 0) not true.#012thread =<br>
> (cache-worker)#012ident =<br>
> Linux,2.6.32-358.2.1.el6.x86_64,x86_64,-smalloc,-smalloc,-hcritbit,epoll#012Backtrace:#012<br>
> 0x42ee88: /usr/sbin/varnishd() [0x42ee88]#012 0x436dc5:<br>
> /usr/sbin/varnishd(VRT_IP_string+0x135) [0x436dc5]#012 0x7f56bc4b922f:<br>
> ./<a href="http://vcl.xlWKkvTA.so" target="_blank">vcl.xlWKkvTA.so</a>(+0xb622f) [0x7f56bc4b922f]#012 0x436203:<br>
> /usr/sbin/varnishd(VCL_recv_method+0x43) [0x436203]#012 0x418eaf:<br>
> /usr/sbin/varnishd(CNT_Session+0xb7f) [0x418eaf]#012 0x430bd1:<br>
> /usr/sbin/varnishd() [0x430bd1]#012 0x7f56c388f851:<br>
> /lib64/libpthread.so.0(+0x7851) [0x7f56c388f851]#012 0x7f56c35dd90d:<br>
> /lib64/libc.so.6(clone+0x6d) [0x7f56c35dd90d]#012sp = 0x7f569b089008 {#012<br>
> fd = 320, id = 320, xid = 1298044854,#012 client = 10.0.0.170 33991,#012<br>
> step = STP_RECV,#012 handling = deliver,#012 err_code = 404, err_reason =<br>
> (null),#012 restarts = 0, esi_level = 0#012 flags = #012 bodystatus =<br>
> 3#012 ws = 0x7f569b089080 { overflow#012 id = "sess",#012 {s,f,r,e} =<br>
> {0x7f569b089c78,+32768,(nil),+32768},#012 },#012 http[req] = {#012 ws =<br>
> 0x7f569b089080[sess]#012 "GET",#012 "/images/ico_hots2.gif",#012<br>
> "HTTP/1.1",#012 "User-Agent: Opera/9.80 (Android; Opera<br>
> Mini/6.7.30171/29.3222; U; zh) Presto/2.8.119 Version/11.10",#012<br>
> "Host: <a href="http://www.example.com" target="_blank">www.example.com</a>",#012 "Accept: text/html, application/xml;q=0.9,<br>
> application/xhtml+xml, image/png, image/webp, image/jpeg, image/gif,<br>
> image/x-xbitmap, */*;q=0.1",#012 "Accept-Language: zh-cn,en;q=0.9",#012<br>
> "Accept-Encoding: gzip, deflate",#012 "Referer:<br>
> <a href="http://www.example.com/" target="_blank">http://www.example.com/</a>",#012 "Connection: Keep-Alive",#012<br>
> "clientip: 117.149.35.78",#012 "X-OperaMini-Features: advanced,<br>
> file_system, camera, touch, folding, viewport",#012 "Device-Stock-UA:<br>
> Mozilla/5.0 (Linux; U; Android 2.3.5; zh-cn; BOWAY I5 Build/MocorDroid2.3.5)<br>
> AppleWebKit/533.1<br>
> Apr 10 22:12:58 cdn001 varnishd[25730]: child (7270) Started<br>
> Apr 10 22:12:59 cdn001 kernel: varnishd[7270]: segfault at 0 ip<br>
> 000000000041067d sp 00007fff4f027a40 error 6 in varnishd[400000+70000]<br>
> Apr 10 22:12:59 cdn001 varnishd[25730]: Pushing vcls failed:#012CLI<br>
> communication error (hdr)<br>
> Apr 10 22:12:59 cdn001 varnishd[25730]: Child (7270) died signal=11<br>
> Apr 10 22:12:59 cdn001 varnishd[25730]: Child (-1) said Child starts<br>
><br>
> I have restarted varnish daemon and things looks fine now, but I am<br>
> thinking that how could this error happen? is there something wrong in my<br>
> varnish configuration ? or something else ?<br>
><br>
> thanks .<br>
><br>
</div></div>> _______________________________________________<br>
> varnish-misc mailing list<br>
> <a href="mailto:varnish-misc@varnish-cache.org">varnish-misc@varnish-cache.org</a><br>
> <a href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc" target="_blank">https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc</a><br>
</blockquote></div><br></div>