From eli at fiercemarkets.com Fri Oct 1 00:09:06 2010 From: eli at fiercemarkets.com (Eli Dickinson) Date: Thu, 30 Sep 2010 18:09:06 -0400 Subject: Strange problem with truncated responses when POSTing data Message-ID: Greetings, First, thanks for Varnish! It's a great piece of software. Today, though, I've been pulling my hair out with a strange problem and I'm hoping someone here can help. I've got Varnish sitting in front of Apache on one server and I've got a PHP script that POSTs data from another server. If the length of the content I'm posting is over about 1500 bytes, I get an incomplete response from Varnish. Like, literally I only get half the page back. According to Wireshark, it looks like I get 1 or 2 packets of data (it's not consistent) and then there's suddently a RST/ACK. Weirder still, an almost identical request from cURL works fine. So it's either some kind of funny timing issue or it's something very specific to the way PHP makes HTTP requests. The problem disappears if: - I tell varnish to pipe instead of pass -?I run the PHP script on the same server and have it POST to localhost (which makes me think timing is a factor...) - I POST less than 1kb of data I'm running varnish-2.1.3 SVN and I don't see anything unusual in the syslog or when running varnishlog (though I'm not exactly sure what I'm looking for) Here's what my request looks like: POST / HTTP/1.0 Host: web2.servicecenter.fiercemarkets.com Content-Length: 1517 ??XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX ??XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX ??XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX ??XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX [...snip...] And the response I get back looks like: HTTP/1.1 403 Forbidden Server: Apache/2.2.3 (Red Hat) X-Powered-By: PHP/5.2.14 Expires: Sun, 11 Mar 1984 12:00:00 GMT Last-Modified: Thu, 30 Sep 2010 21:40:44 +0000 Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0 ETag: "1285882844" Content-Type: text/html; charset=utf-8 Content-Length: 4104 Date: Thu, 30 Sep 2010 21:40:44 GMT X-Varnish: 598880740 Age: 0 Via: 1.1 varnish Connection: close ?? [...snip...] The response looks fine (content-length is correct), aside from the fact that I don't get all of it. I'm stumped. Any thoughts? -- Eli Dickinson FierceMarkets, Inc. 202-824-5058 From tfheen at varnish-software.com Fri Oct 1 07:37:45 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Fri, 01 Oct 2010 07:37:45 +0200 Subject: Panic Signal 6. In-Reply-To: <804FB091-853B-4D6C-BD56-99743186472A@competitorgroup.com> (Ben Nowacky's message of "Thu, 30 Sep 2010 12:10:21 -0700") References: <9C0B05C0-A205-4E41-B906-034F9B4EC2A4@competitorgroup.com> <7F0AA702B8A85A4A967C4C8EBAD6902C502366@TMG-EVS02.torstar.net> <1FF67D7369ED1A45832180C7C1109BCA264AAC8BCB@tmmail0.trademe.local> <804FB091-853B-4D6C-BD56-99743186472A@competitorgroup.com> Message-ID: <87ocbe8od2.fsf@qurzaw.linpro.no> ]] Ben Nowacky | -p sess_workspace=8192 Why are you limiting your session workspace to a measly 8k? I'd recommend letting this stay at its default value of 64k or increase it beyond 64k, not decrease it. -- Tollef Fog Heen Varnish Software t: +47 21 54 41 73 From tfheen at varnish-software.com Fri Oct 1 09:09:59 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Fri, 01 Oct 2010 09:09:59 +0200 Subject: "now" in VCL In-Reply-To: <4C7BA0C2.1000508@mapsolute.com> (Frank Gruellich's message of "Mon, 30 Aug 2010 14:14:58 +0200") References: <4C6D2823.6060604@mapsolute.com> <87aao44nxc.fsf@qurzaw.linpro.no> <4C7BA0C2.1000508@mapsolute.com> Message-ID: <87d3ru8k3c.fsf@qurzaw.linpro.no> ]] Frank Gruellich | I'm still getting a similar error message: | | Message from VCC-compiler: | Expected ';' got 'now' | (program line 491), at | (input Line 123 Pos 22) | log "MISS: " now req.url; | ---------------------###--------- | Running VCC-compiler failed, exit 1 | | "now" seems to be gone... It seems like I forgot to merge r4788 into 2.1. Doing so now, and it should be in 2.1.4. -- Tollef Fog Heen Varnish Software t: +47 21 54 41 73 From perbu at varnish-software.com Fri Oct 1 09:12:11 2010 From: perbu at varnish-software.com (Per Buer) Date: Fri, 1 Oct 2010 09:12:11 +0200 Subject: Strange problem with truncated responses when POSTing data In-Reply-To: References: Message-ID: On Fri, Oct 1, 2010 at 12:09 AM, Eli Dickinson wrote: > Today, though, I've been pulling my hair out with a strange problem > and I'm hoping someone here can help. I've got Varnish sitting in > front of Apache on one server and I've got a PHP script that POSTs > data from another server. If the length of the content I'm posting is > over about 1500 bytes, I get an incomplete response from Varnish. > This _might_ be a networking problem. I would guess that somehow you are using 802.1q tags and that some silly piece of network equipment throws away or mangles packets on one of your servers. -- Per Buer, Varnish Software Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / skype: per.buer -------------- next part -------------- An HTML attachment was scrubbed... URL: From tungcnw at gmail.com Fri Oct 1 12:01:56 2010 From: tungcnw at gmail.com (Dang Tung) Date: Fri, 1 Oct 2010 17:01:56 +0700 Subject: Fixing backend URLs In-Reply-To: <20100930050746.GA5024@grum.quex.org> References: <2006354608710103182@unknownmsgid> <20100915032231.GC497@grum.quex.org> <0FAEA775-D78B-4CD6-9C0C-972ADE57287F@gmail.com> <20100924043515.GA30650@grum.quex.org> <20100924051235.GA31378@grum.quex.org> <20100930050746.GA5024@grum.quex.org> Message-ID: Thanks, I will try, and let you know later :). On Thu, Sep 30, 2010 at 12:07 PM, Michael Alger wrote: > On Thu, Sep 30, 2010 at 11:52:11AM +0700, Dang Tung wrote: > > Are there any solutions for my issue, please help, I'm still > > waiting your reply. > > I don't really have any other ideas; the code should work if it's > run, so I think you need to try to determine why the code isn't > being run. You can use varnishlog to monitor the request flow and > see if you can work it out that way. Or, you can add some debugging > statements to log messages or add headers to the response at various > points in your VCL so you can determine exactly how far Varnish is > getting, and then from that work out why it's not running the code > in question. > > You might also want to see if it's possible to adjust your > application's configuration so it emits the 'correct' Location > header itself; it's generally better to configure the application to > know what its external address is rather than to rewrite things, as > rewriting the Location: header won't do anything to fix > fully-qualified URLs embedded in the HTML. > > Alternatively if you're willing to attach your entire Varnish > configuration we might be able to work it out from that. > > > > On Mon, Sep 27, 2010 at 11:26 AM, Dang Tung wrote: > > > > > here is the header when I used wget: > > > > > > backend: wget -S -O /dev/null http://myaddress/admin > > > > > > wget -S -O /dev/null http://myaddress/admin > > > --2010-09-27 11:18:35-- http://myaddress/admin > > > Connecting to myaddress:80... connected. > > > HTTP request sent, awaiting response... > > > HTTP/1.1 302 Found > > > Server: Apache/2.2.3 (CentOS) > > > X-Powered-By: PHP/5.2.10 > > > Set-Cookie: symfony=lli9m9oog3flp23ggchhbt5s71; path=/ > > > Location: http://myaddress:8080/admin.php > > > Cache-Control: max-age=0 > > > Expires: Mon, 27 Sep 2010 04:20:08 GMT > > > Content-Type: text/html; charset=utf-8 > > > Content-Length: 106 > > > Date: Mon, 27 Sep 2010 04:18:48 GMT > > > X-Varnish: 153748006 > > > Age: 0 > > > Via: 1.1 varnish > > > Connection: keep-alive > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > -- Nguyen Dang Tung -------------- next part -------------- An HTML attachment was scrubbed... URL: From kristian at varnish-software.com Fri Oct 1 13:13:55 2010 From: kristian at varnish-software.com (Kristian Lyngstol) Date: Fri, 1 Oct 2010 13:13:55 +0200 Subject: Strange problem with truncated responses when POSTing data In-Reply-To: References:

Message-ID: <20101001111354.GB6818@sunrider> On Fri, Oct 01, 2010 at 09:12:11AM +0200, Per Buer wrote: > On Fri, Oct 1, 2010 at 12:09 AM, Eli Dickinson wrote: > > > > Today, though, I've been pulling my hair out with a strange problem > > and I'm hoping someone here can help. I've got Varnish sitting in > > front of Apache on one server and I've got a PHP script that POSTs > > data from another server. If the length of the content I'm posting is > > over about 1500 bytes, I get an incomplete response from Varnish. > > > > This _might_ be a networking problem. I would guess that somehow you are > using 802.1q tags and that some silly piece of network equipment throws away > or mangles packets on one of your servers. That's a very good point. 1500 is too close to typical MTU... Try setting the mtu on the server to 1400 for example? - Kristian From martin.boer at bizztravel.nl Fri Oct 1 12:37:13 2010 From: martin.boer at bizztravel.nl (Martin Boer) Date: Fri, 01 Oct 2010 12:37:13 +0200 Subject: Panic Signal 6. In-Reply-To: <804FB091-853B-4D6C-BD56-99743186472A@competitorgroup.com> References: <9C0B05C0-A205-4E41-B906-034F9B4EC2A4@competitorgroup.com> <7F0AA702B8A85A4A967C4C8EBAD6902C502366@TMG-EVS02.torstar.net> <1FF67D7369ED1A45832180C7C1109BCA264AAC8BCB@tmmail0.trademe.local> <804FB091-853B-4D6C-BD56-99743186472A@competitorgroup.com> Message-ID: <4CA5B9D9.4080905@bizztravel.nl> Ben, You're using a very small amount of memory. I have the feeling that varnish sometimes has trouble throwing away older entries which are still valid. With your amount of memory that could easily be the case. I'm not really sure, but since I've allocated more than enough space for the content varnish hasn't crashed since the last restart a couple of months ago. I would suggest to use "-s file,/path/varnish.bin,2G" for a while to see if that helps. Of course you need to have 2GB of free space in /path/ Regards, Martin On 09/30/2010 09:10 PM, Ben Nowacky wrote: > We're running CentOS 64 bit, 5.5... Here's our VCL, and startup flags: > > sysconfig/varnish: > -u varnish > -s malloc,800m > -p cli_timeout=1200s > -p thread_pools=1 > -p thread_pool_min=120 > -p thread_pool_max=200 > -p listen_depth=1024 > p shm_reclen=32768 > -p sess_workspace=8192 > -p http_headers=32 > -p sess_timeout=10 > -p session_linger=100 > -p between_bytes_timeout=10 > -p first_byte_timeout=10 > -h critbit > > > VCL: > # Default backend definition. Set this to point to your content > # server. > # > backend default { > .host = "64.237.104.85"; > .port = "8080"; > } > > acl purge { > "localhost"; > "64.237.104.85"; > } > > sub vcl_recv { > # Serve stale content while fresh is retrieved for 2minutes. > set req.grace = 6m; > if (req.http.host ~ "elite.competitor.com > ") { > if (req.url ~ "^/confcard$") { > set req.http.New-Location = regsub(req.url,"$","/"); > error 301 "Redirecting you to the slash representation..."; > } > if (req.url ~ "^/confcard/") { > return (pipe); > } > } > if (req.http.host ~ "forums.competitor.com > ") { > return (pipe); > } > if (req.http.host ~ "svn.competitor.com "){ > return (pipe); > } > if (req.http.host ~ "competitorradio.competitor.com > ") { > if (req.url ~ "^/wp-admin$") { > set req.http.New-Location = regsub(req.url,"$","/"); > error 301 "Redirecting you to the slash representation..."; > } > else { > return (pipe); > } > } > > # always exempt Dev.lan from cache. > if (!(req.http.host ~ "competitor.com ")) { > return (pipe); > } > # Exempt mobile users from cache... straight to backend. > if (req.http.User-Agent ~ > "(iPhone|iPod|iPad)|(Android)|(BlackBerry9530)|(webOS)") { > return (pipe); > } > # redirect URL for trailing slash when logging in. > if (req.url ~ "^/wp-admin$") { > set req.http.New-Location = regsub(req.url,"$","/"); > error 301 "Redirecting you to the slash representation..."; > } > if (req.http.cookie ~ "wordpress_") { > return (pipe); > } > # if wp-admin or wp-login is in the URL, pipe to apache. > if (req.url ~ "/wp-(admin|login)") { > return (pipe); > } > # otherwise just unset cookie and cache all pages. > else { > unset req.http.cookie; > } > > # # cache the home page. > # if (req.url ~ "^/$") { > # unset req.http.cookie; > # } > # #category page > # if (req.url ~ "^/category(.)*") { > # unset req.http.cookie; > # } > # #post page with date in URL > # if (req.url ~ "^/?p=[0-9](.)*") { > # unset req.http.cookie; > # } > # #post page with date in URL > # if (req.url ~ "^/[0-9](.)*") { > # unset req.http.cookie; > # } > if (req.http.Accept-Encoding) { > if (req.url ~ > "^/[^?]+\.(ico|js|css|txt|gz|zip|lzma|bz2|tgz|tbz|html|htm)(\?.*|)$") { > # No point in compressing these > remove req.http.Accept-Encoding; > } elsif (req.http.Accept-Encoding ~ "gzip") { > set req.http.Accept-Encoding = "gzip"; > } elsif (req.http.Accept-Encoding ~ "deflate") { > set req.http.Accept-Encoding = "deflate"; > } elsif (req.http.Accept-Encoding ~ "MSIE 6") { > set req.http.Accept-Encoding = "MSIE"; > } elsif (req.http.Accept-Encoding ~ "MSIE 7") { > set req.http.Accept-Encoding = "MSIE"; > } elsif (req.http.Accept-Encoding ~ "MSIE 8") { > set req.http.Accept-Encoding = "MSIE"; > } elsif (req.http.Accept-Encoding ~ "Mozilla") { > set req.http.Accept-Encoding = "mozilla"; > } > else { > # unkown algorithm > remove req.http.Accept-Encoding; > } > } > unset req.http.user-agent; > if (req.url ~ > "^/[^?]+\.(jpeg|jpg|png|gif|ico|js|css|txt|gz|zip|lzma|bz2|tgz|tbz|html|htm)(\?.*|)$") > { > remove req.http.cookie; > set req.url = regsub(req.url, "\?.*$", ""); > } > # don't bother caching large files > if (req.url ~ > "^/[^?]+\.(mp3|pdf|flv|mov|mp4|mpg|mpeg|avi|dmg|swf)(\?.*|)$") { > return (pipe); > } > # accept purge requests from Wordpress. > if (req.request == "PURGE") { > if(!client.ip ~ purge) { > error 405 "Not allowed."; > } > purge_url(req.url); > error 200 "Purged"; > purge("req.url == " req.url); > } > #################### > # End sub_recv # > #################### > } > > > sub vcl_deliver { > #add cache hit data > if (obj.hits > 0) { > #if hit add hit count > set resp.http.X-Served-By = server.hostname; > set resp.http.X-Cache = "HIT"; > set resp.http.X-Cache-Hits = obj.hits; > set resp.http.X-Encoding = req.http.Accept-Encoding; > set resp.http.X-Debug = req.http.host; > } else { > set resp.http.X-Served-By = server.hostname; > set resp.http.X-Cache = "MISS"; > set resp.http.X-Encoding = req.http.Accept-Encoding; > set resp.http.X-Debug = req.http.host; > } > } > sub vcl_error { > # for redirect on trailing slash. > if (req.http.New-Location) { > set obj.http.Location = req.http.New-Location; > } > } > > sub vcl_fetch { > > if (beresp.status != 200 && beresp.status != 403 && beresp.status != > 404 && beresp.status != 405 && beresp.status != 301 && beresp.status > != 302) { > restart; > } > if (beresp.http.cookie ~ "wordpress_") { > return (pass); > } > # Serve stale content while fresh is retrieved for 2minutes. > set beresp.grace = 6m; > set beresp.ttl = 720s; > if (beresp.ttl < 720s) { > set beresp.ttl = 720s; > } > if (!beresp.cacheable) { > set beresp.ttl = 0s; > } > if (beresp.http.Set-Cookie) { > return(deliver); > } > # make sure cookies aren't set for retrieved static objects. > if (req.url ~ "^/[^?]+\.(jpeg|jpg|png|gif|ico)(\?.*|)$") { > set beresp.http.Cache-Control = "max-age=31536000, public"; > unset beresp.http.set-cookie; > } > if (req.url ~ "^/[^?]+\.(js|html|htm|txt)(\?.*|)$") { > set beresp.http.Cache-Control = "max-age=1728000, public, > must-revalidate"; > unset beresp.http.set-cookie; > } > if (req.url ~ "^/[^?]+\.(css)(\?.*|)$") { > set beresp.http.Cache-Control = "max-age=1728000, public, > must-revalidate"; > unset beresp.http.set-cookie; > } > # cache the home page. > if (req.url ~ "^/$") { > unset beresp.http.cookie; > set beresp.ttl = 600s; > } > if (req.url ~ "^/category(.)*") { > unset beresp.http.cookie; > } > if (req.url ~ "^(.)/[0-9](.)*") { > unset beresp.http.cookie; > } > if (req.url ~ "^/?p=[0-9](.)*") { > unset beresp.http.cookie; > } > #Change Web Server header because we can. > unset beresp.http.Server; > set beresp.http.Server = "Mini Con Adventure Team"; > #################### > # End sub_fetch # > #################### > } > > sub vcl_pipe { > # # http://www.varnish-cache.org/ticket/451 > # # This forces every pipe request to be the first one. > set bereq.http.connection = "close"; > } > > > sub vcl_hit { > if (req.request == "PURGE") { > set obj.ttl = 0s; > error 200 "Purged."; > } > } > > > sub vcl_hash { > > # //hash the object with url+host > > set req.hash += req.url; > set req.hash += req.http.host; > set req.hash += req.http.User-Agent; > set req.hash += req.http.cookie; > return(hash); > } > > > > > > > > > > > On Sep 30, 2010, at 12:06 PM, Ross Brown wrote: > >> 2 Days? That?s certainly?. sub-optimal. What OS are you running? Our >> 8 x Varnish servers (64-bit Ubuntu 10.04.1 LTS) are rock solid and >> easily run for weeks without any issues. >> We run with malloc,24G (32Gb RAM installed) >> Can you share your startup options and VCL? >> *From:* varnish-misc-bounces at varnish-cache.org >> >> [mailto:varnish-misc-bounces at varnish-cache.org] *On Behalf Of *Ben >> Nowacky >> *Sent:* Friday, 1 October 2010 7:02 a.m. >> *To:* Caunter, Stefan >> *Cc:* varnish-misc at varnish-cache.org >> >> *Subject:* Re: Panic Signal 6. >> 2.1.2, would prefer to use malloc over file storage if possible, but >> haven't tried file to see if that's the issue. >> On Sep 30, 2010, at 10:57 AM, Caunter, Stefan wrote: >> >> >> What version of varnish? Have you tried with ?s file,/path/to/file >> ,1075200000 instead of ?s malloc? >> Stefan Caunter :: Senior Systems Administrator :: TOPS >> e: scaunter at topscms.com :: m: (416) >> 561-4871 >> www.thestar.com www.topscms.com >> >> *From:* varnish-misc-bounces at varnish-cache.org >> >> [mailto:varnish-misc-bounces at varnish-cache.org] *On Behalf Of *Ben >> Nowacky >> *Sent:* September-30-10 12:53 PM >> *To:* varnish-misc at varnish-cache.org >> *Subject:* Panic Signal 6. >> Hey everyone ! Wondering if someone can lend a hand. We've gotten our >> servers upgraded to 64-bit OS, and seen tremendous stability >> enhancement with varnish over our 32-bit os, which crashed every 5-10 >> minutes. We can now get about 2 days of uptime before we have a new >> child process crash. Logs are listed below.. Wondering if something >> obvious jumps out at anyone as to what the cause of this may be. >> Appreciate any insight from the community ! >> Sep 30 16:37:17 srv860 varnishd[16195]: Child (16196) died signal=6 >> Sep 30 16:37:17 srv860 varnishd[16195]: Child (16196) Panic message: >> Assert error in WS_Release(), cache_ws.c line 193: Condition(bytes >> <= ws->e - ws->f) not true. errno = 104 (Connection reset by peer) >> thread = (cache-worker) ident = >> Linux,2.6.18-194.el5,x86_64,-smalloc,-hcritbit,epoll Backtrace: >> 0x422616: /usr/sbin/varnishd [0x422616] 0x42d475: >> /usr/sbin/varnishd(WS_Release+0xf5) [0x42d475] 0x427b33: >> /usr/sbin/varnishd [0x427b33] 0x42bc45: >> /usr/sbin/varnishd(VRT_SetHdr+0xf5) [0x42bc45] 0x2aaaab003e76: >> ./vcl.1P9zoqAU.so [0x2aaaab003e76] 0x427006: >> /usr/sbin/varnishd(VCL_deliver_method+0x46) [0x427006] 0x4130cf: >> /usr/sbin/varnishd [0x4130cf] 0x413e49: >> /usr/sbin/varnishd(CNT_Session+0x369) [0x413e49] 0x424a68: >> /usr/sbin/varnishd [0x424a68] 0x423d4d: /usr/sbin/varnishd >> [0x423d4d] sp = 0x2aaac060e008 { fd = 351, id = 351, xid = >> 1104824170, client = 63.240.230.197:26479, step = STP_DELIVER, >> handling = deliver, restarts = 0, esis = 0 ws = 0x2aaac060e078 { >> id = "sess", {s,f >> Sep 30 16:37:17 srv860 varnishd[16195]: child (32562) Started >> Sep 30 16:37:17 srv860 varnishd[16195]: Child (32562) said >> Sep 30 16:37:17 srv860 varnishd[16195]: Child (32562) said Child starts > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Fri Oct 1 13:48:06 2010 From: perbu at varnish-software.com (Per Buer) Date: Fri, 1 Oct 2010 13:48:06 +0200 Subject: Panic Signal 6. In-Reply-To: <4CA5B9D9.4080905@bizztravel.nl> References: <9C0B05C0-A205-4E41-B906-034F9B4EC2A4@competitorgroup.com> <7F0AA702B8A85A4A967C4C8EBAD6902C502366@TMG-EVS02.torstar.net> <1FF67D7369ED1A45832180C7C1109BCA264AAC8BCB@tmmail0.trademe.local> <804FB091-853B-4D6C-BD56-99743186472A@competitorgroup.com> <4CA5B9D9.4080905@bizztravel.nl> Message-ID: On Fri, Oct 1, 2010 at 12:37 PM, Martin Boer wrote: > Ben, > > You're using a very small amount of memory. I have the feeling that varnish > sometimes has trouble throwing away older entries which are still valid. > With your amount of memory that could easily be the case. I'm not really > sure, but since I've allocated more than enough space for the content > varnish hasn't crashed since the last restart a couple of months ago. > > I would suggest to use "-s file,/path/varnish.bin,2G" for a while to see > if that helps. Of course you need to have 2GB of free space in /path/ > Depending on your setup that might not be such a good idea. Better to stick with the current backend (malloc), although it might make sense to give it some more breathing space. -s file on Centos 5 with spinning disks (!ssd) can be a bit heavy on writing. -- Per Buer, Varnish Software Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / skype: per.buer -------------- next part -------------- An HTML attachment was scrubbed... URL: From tfheen at varnish-software.com Fri Oct 1 14:57:40 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Fri, 01 Oct 2010 14:57:40 +0200 Subject: Byte range and varnish In-Reply-To: (Jean-Francois Laurens's message of "Tue, 15 Jun 2010 16:37:49 +0200") References: Message-ID: <87vd5m6pff.fsf@qurzaw.linpro.no> ]] Jean-Francois Laurens | I'm setting up a media server and would like to use varnish for caching some of the heavily requested medias (mp3, mp4 ... ). | | I've a few questions regarding media broadcasting with varnish. | | mp3: progressive download is used in order to read mp3 from the webpages, is it foreseen to have a function to would allow providing the media before it gets completely retrieved by varnish from the backend ? This is probably going into 3.0 | mp4: how is handled Accept-range field in the header, does varnish supports it ? In 2.0, it's not supported. In 2.1, you can turn on Range support using a parameter. -- Tollef Fog Heen Varnish Software t: +47 21 54 41 73 From tfheen at varnish-software.com Fri Oct 1 14:59:12 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Fri, 01 Oct 2010 14:59:12 +0200 Subject: Problems loading directory URLs In-Reply-To: <4C1EC501.4090201@jetheis.com> (Jimmy Theis's message of "Sun, 20 Jun 2010 21:48:49 -0400") References: <4C1EC501.4090201@jetheis.com> Message-ID: <87r5ga6pcv.fsf@qurzaw.linpro.no> ]] Jimmy Theis Hi, | When the url of a directory is requested without a / following it, the | request fails and nothing is returned. However, in all other | situations, including appending the final / to the url, Varnish | handles it correctly. This sounds odd, can you please provide the output of varnishlog when this happens? (Unless you've worked out the problem in the meantime, that is.) -- Tollef Fog Heen Varnish Software t: +47 21 54 41 73 From tfheen at varnish-software.com Fri Oct 1 15:01:33 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Fri, 01 Oct 2010 15:01:33 +0200 Subject: Varnish: Initial lag before dumping output to the browser In-Reply-To: <00b201cb1788$c949a8f0$5bdcfad0$@molleti@itp.com> (Hitendra Molleti's message of "Tue, 29 Jun 2010 16:44:16 +0400") References: <00b201cb1788$c949a8f0$5bdcfad0$@molleti@itp.com> Message-ID: <87mxqy6p8y.fsf@qurzaw.linpro.no> ]] "Hitendra Molleti" Hi, | We have setup Varnish to cache the pages, a page which is put together using | various widgets, seems to load all at once instead of gradual loading. | Hence, there is an initial 3 to 4sec wait where the browser does not get any | output, after that the whole rendered page is dumped on the browser. This | makes the site seem much slower than it actually is. Varnish currently always fetches the complete object from the backend before sending anything to the client. Regards, -- Tollef Fog Heen Varnish Software t: +47 21 54 41 73 From jim.robinson at stanford.edu Fri Oct 1 15:05:36 2010 From: jim.robinson at stanford.edu (James A. Robinson) Date: Fri, 1 Oct 2010 06:05:36 -0700 Subject: Byte range and varnish In-Reply-To: <87vd5m6pff.fsf@qurzaw.linpro.no> References: <87vd5m6pff.fsf@qurzaw.linpro.no> Message-ID: On Fri, Oct 1, 2010 at 05:57, Tollef Fog Heen wrote: > | mp4: how is handled Accept-range field in the header, does varnish supports it ? > > In 2.0, it's not supported. In 2.1, you can turn on Range support using > a parameter. Hi, Well, I have to admit that I take issue with people saying byte ranges are supported in 2.1. As far as I can tell, it isn't *properly* supported. It advertises Accept-Range: bytes but if someone passes in a valid multi-sequence set of ranges, Varnish fails to do the right thing. It also fails to gracefully get out of the way on a pipe request, and drops Content-Range headers from the backend. Jim From scaunter at topscms.com Fri Oct 1 15:15:00 2010 From: scaunter at topscms.com (Caunter, Stefan) Date: Fri, 1 Oct 2010 09:15:00 -0400 Subject: wont server from Cache In-Reply-To: <0F69574F9901D4459C6B75C9FF64FBC404B4DB9B@mxfl01.hollywoodmedia.corp> References: <0F69574F9901D4459C6B75C9FF64FBC404B4DB9B@mxfl01.hollywoodmedia.corp> Message-ID: <7F0AA702B8A85A4A967C4C8EBAD6902C50245C@TMG-EVS02.torstar.net> Assuming your cache is hot enough, you should be able to keep things going with this in recv if (req.backend.healthy) { set req.grace = 60s; } else { set req.grace = 4h; } If your backends all go away, grace is what saves you, but you need the cache to be well heated. Stefan Caunter :: Senior Systems Administrator :: TOPS e: scaunter at topscms.com :: m: (416) 561-4871 www.thestar.com www.topscms.com From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Vitaly Burshteyn Sent: June-01-10 12:28 PM To: varnish-misc at varnish-cache.org Subject: wont server from Cache Hi folks, So I sent this post few days ago, I have made some changes but still cant get my varnish box to server content from cache even though it looks like it is: 232 Client requests received 130 Cache hits 5 Cache hits for pass 97 Cache misses 99 Backend conn. success 3 Backend conn. failures 99 Fetch with Length 15 N struct sess_mem When I shut down the web server it gives me a 503 error. Here is the copy of my vcl file with out the backend info.. Any and all suggestion would be appreciated. sub vcl_recv { #set req.backend = bw; set req.grace = 60m; if (req.request == "PURGE") {if (!client.ip ~ purge) {error 405 "Not allowed.";} return(lookup);} if (req.request != "GET" && req.request != "HEAD") {return(lookup);} if (req.http.Cache-Control ~ "no-cache") {purge_url(req.url);} return(lookup); } sub vcl_hit { if (!obj.cacheable) { return(pass); }} sub vcl_fetch { if (beresp.status == 500 || beresp.status == 503 || beresp.status == 504) { restart; } set beresp.ttl = 600s; set beresp.grace = 600s; if (beresp.ttl < 300s) { set beresp.ttl = 300s;} if (!beresp.cacheable) {return(pass);} if (beresp.http.Set-Cookie) {return(pass);} } Vitaly Burshteyn Senior Network Engineer Broadway.com, Theatre Direct International 729 7th Avenue New York, New York 10019 Phone: 212.817.9117 Cell# 917-701-5732 -------------- next part -------------- An HTML attachment was scrubbed... URL: From joao.lisanti at locaweb.com.br Fri Oct 1 15:57:14 2010 From: joao.lisanti at locaweb.com.br (=?iso-8859-1?Q?Jo=E3o_Gabriel?=) Date: Fri, 1 Oct 2010 10:57:14 -0300 Subject: Testing -s persistent, getting "Out of space in persistent silo" In-Reply-To: References: Message-ID: <526F23AC-8ADA-4696-8D68-ABF8C3B02019@locaweb.com.br> df -h look your disk free space :D Jo?o Gabriel CT-Linux On Mar 30, 2010, at 12:52 PM, Maximilian Sch?fmann wrote: > Hi all, > > I'm currently testing 2.1 with -s > persistent,/var/lib/varnish/$INSTANCE/cache.bin,14G (on Ubuntu) > After a while, I start seeing these every few minutes > > varnishd[16181]: Child (7977) said Out of space in persistent silo > varnishd[16181]: Child (7977) said Committing suicide, restart will make space > > with the effect of the cache being empty after the child restarts (it > was far from filled before these crashes...). > > Best, Max > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From mjdavies at glam.ac.uk Fri Oct 1 16:22:37 2010 From: mjdavies at glam.ac.uk (Davies Matt J A (LCSS)) Date: Fri, 1 Oct 2010 14:22:37 +0000 Subject: Serving Backends Message-ID: Hello everyone I've defined a backend that is my application server, backend a { .host = "application_server_fqdn"; .port = "80"; } and then defined some further code to serve that backend from my varnish server sub vcl_recv { if (req.http.host == "www.a.com") { set req.backend = a; } } What I'm finding is that if I make up anything in my /etc/hosts file to point at the varnish servers IP address, it will serve the site. Even the IP address of the varnish server serves the application. I don't want that to happen. Am I missing something obvious here? I thought that only requests with the http.host value of www.a.com will serve the cached backend. Can anyone point me in the right direction here? Thanks Matt From varnish at mm.quex.org Fri Oct 1 16:36:46 2010 From: varnish at mm.quex.org (Michael Alger) Date: Fri, 1 Oct 2010 22:36:46 +0800 Subject: Serving Backends In-Reply-To: References: Message-ID: <20101001143646.GA27668@grum.quex.org> On Fri, Oct 01, 2010 at 02:22:37PM +0000, Davies Matt J A (LCSS) wrote: > > I've defined a backend that is my application server, > > backend a { > .host = "application_server_fqdn"; > .port = "80"; > } > > and then defined some further code to serve that backend from my > varnish server > > sub vcl_recv { > > if (req.http.host == "www.a.com") { > set req.backend = a; > } > > } > > What I'm finding is that if I make up anything in my /etc/hosts > file to point at the varnish servers IP address, it will serve the > site. Even the IP address of the varnish server serves the > application. > > I don't want that to happen. Varnish will use the first backend defined as the default, i.e. req.backend is always set to something even if you don't set it yourself. Not sure if it's actually just the first backend defined or some other criteria; but the point is there's always a backend for it to use even if you don't specify it explicitly. If you really want to only serve if the hostname in the request header is recognised, you'll have to tell Varnish to display an error, or change the request so it will go to a backend that displays an error page, or something similar. i.e. if (req.http.host == "www.a.com") { set req.backend = a; } else { error 404 "Not found"; } From mjdavies at glam.ac.uk Fri Oct 1 16:56:25 2010 From: mjdavies at glam.ac.uk (Davies Matt J A (LCSS)) Date: Fri, 1 Oct 2010 14:56:25 +0000 Subject: Serving Backends In-Reply-To: <20101001143646.GA27668@grum.quex.org> References: <20101001143646.GA27668@grum.quex.org> Message-ID: <63650A35-F325-4D02-A40B-D7C9DBB225D3@glam.ac.uk> Thanks Michael I'll do just that Matt On 1 Oct 2010, at 15:36, Michael Alger wrote: > On Fri, Oct 01, 2010 at 02:22:37PM +0000, Davies Matt J A (LCSS) wrote: >> >> I've defined a backend that is my application server, >> >> backend a { >> .host = "application_server_fqdn"; >> .port = "80"; >> } >> >> and then defined some further code to serve that backend from my >> varnish server >> >> sub vcl_recv { >> >> if (req.http.host == "www.a.com") { >> set req.backend = a; >> } >> >> } >> >> What I'm finding is that if I make up anything in my /etc/hosts >> file to point at the varnish servers IP address, it will serve the >> site. Even the IP address of the varnish server serves the >> application. >> >> I don't want that to happen. > > Varnish will use the first backend defined as the default, i.e. > req.backend is always set to something even if you don't set it > yourself. Not sure if it's actually just the first backend defined > or some other criteria; but the point is there's always a backend > for it to use even if you don't specify it explicitly. > > If you really want to only serve if the hostname in the request > header is recognised, you'll have to tell Varnish to display an > error, or change the request so it will go to a backend that > displays an error page, or something similar. > > i.e. > > if (req.http.host == "www.a.com") { > set req.backend = a; > } > else { > error 404 "Not found"; > } > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From jhayter at manta.com Fri Oct 1 17:40:32 2010 From: jhayter at manta.com (Jim Hayter) Date: Fri, 1 Oct 2010 11:40:32 -0400 Subject: SSL and Varnish In-Reply-To: References: <4C8621A7.50804@gmail.com> Message-ID: I chose nginx because it was already in use by our development staff and they recommended it. I have no experience or knowledge of pound so can not compare the two. That said, I've been happy with nginx. Jim -----Original Message----- From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Christian Wittwer Sent: Wednesday, September 08, 2010 9:37 AM To: varnish-misc at varnish-cache.org Subject: Re: SSL and Varnish I'm facing the same question at the moment. Why do you guys recommend/use nginx? I had a look at pound, and it looks pretty easy and lightweight for the purpose of a ssl endpoint. Are there any advantages/drawbacks using pound instead of nginx? Cheers, Christian 2010/9/7 Jim Hayter : > We run nginx to accept both SSL and non-SSL requests. ?This allows me to > have some logic to block certain requests at the nginx level and also > for nginx to return a customized error page if varnishd is not running. > Nginx on passes all remaining traffic to varnishd which sits in front of > multiple web servers. > > -----Original Message----- > From: varnish-misc-bounces at varnish-cache.org > [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Rob S > Sent: Tuesday, September 07, 2010 7:28 AM > To: varnish-misc at varnish-cache.org > Subject: SSL and Varnish > > ?If you've used SSL with varnish in production, can you share any > wisdom on your configuration and approaches? > > > We've been happily using Varnish in production for just over a year, and > > now want to use it for an SSL site so that we can use ESI, load > balancing and other such facilities offered by Varnish. ?I've searched > over the Varnish website, and can't see any examples of a recommended > SSL configuration. > http://plone.org/documentation/kb/plone-behind-varnish-using-pound-for-s > sl > provides some documentation, but I suspect there are other options. > > I'll happily consolidate the answers into a coherent document for the > website so that others can benefit in future. > > > Thanks, > > > Rob > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From alex at bengler.no Fri Oct 1 18:41:57 2010 From: alex at bengler.no (Alexander Staubo) Date: Fri, 1 Oct 2010 18:41:57 +0200 Subject: varnish restarting sporadically... cache being lost. In-Reply-To: References: Message-ID: On Thu, Jun 24, 2010 at 19:25, Ben Nowacky wrote: > - Deployed varnish on 1 of our servers yesterday for final testing on the > live site. Runs along well, seems to be doing great. Run varnishstat to keep > an eye on it, and randomly all the stats reset back to 0, and there's a full > cache purge, which means i'm guessing varnish has restarted? Yes. The Varnish monitoring process is killing the main process because it's not responding to its pings. This thread might be helpful: http://lists.varnish-cache.org/pipermail/varnish-misc/2010-July/004399.html As for why it is not responding to pings, that's the confusing part. Even on what I would consider low load (200-300 reqs/s), Varnish was being sporadically unresponsive to the point of causing other processes to lag. We solved the problem first by increasing the ping timeout to some obscenely high number. Varnish will still become unresponsive at times, but it will not be killed by the parent process when that happens. From angie.tawfik at gmail.com Fri Oct 1 18:53:24 2010 From: angie.tawfik at gmail.com (Angie T. Muhammad) Date: Fri, 1 Oct 2010 18:53:24 +0200 Subject: How to monitor varnish with snmp In-Reply-To: <6944031851727100886@unknownmsgid> References: <6944031851727100886@unknownmsgid> Message-ID: I am so much interested in the same topic. I tried creating my own OIDs and getting the output of a shell script into them, but failed, the OID never resturned any value. May any body give us a simple guidance ? On Tue, Sep 14, 2010 at 6:55 PM, wrote: > Hi everyone: > > How can I monitor data with snmp from varnish process. > > Since now thanks a lots. > > > > > > Saludos/Regards > > > > Marcos Corvalan > > Infraestructura IT > > Mercadolibre.com > > Tronador 4890 - 8 piso Capital Federal | Buenos Aires > > Argentina | CP 1430 | Tel +54 11 5352-8000 Ext 8426 > > email marcos.corvalan| msn marcoscorvalan_04 at hotmail.com > > Mobile : +54 11 37701933 > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > -- All the best, Angie T. Muhammad Linux Systems Engineer Al Masry Al Youm http://www.almasryalyoum.com/en -------------- next part -------------- An HTML attachment was scrubbed... URL: From scaunter at topscms.com Fri Oct 1 19:13:40 2010 From: scaunter at topscms.com (Caunter, Stefan) Date: Fri, 1 Oct 2010 13:13:40 -0400 Subject: 503 error after idle session In-Reply-To: References: Message-ID: <7F0AA702B8A85A4A967C4C8EBAD6902C502528@TMG-EVS02.torstar.net> You are not waiting long enough for the backend. You need logic to restart if the first request fails. Use two directors, one "impatient" and one "patient", with a longer wait time for first byte. Restart if the first request to the fast director fails, and use the slow, patient director for that one. Stefan Caunter :: Senior Systems Administrator :: TOPS e: scaunter at topscms.com :: m: (416) 561-4871 www.thestar.com www.topscms.com From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Jerais Al-Jeraisy Sent: July-24-10 6:12 AM To: varnish-misc at varnish-cache.org Subject: 503 error after idle session Hello, We started to implement varnish @ our infrastructure as a reverse proxy in front of Sharepoint, and it's doing good. but there's a strange problem, if i requested varnish for the first time it works fine, but after, say 2 mins, if i hit the page again or refresh it it takes too long to respond and throw 503 error. it works the the default configurations, changes only on backend host. server details: host: CentOS release 5.5 (Final) x64. varnish: varnish-2.0.6 here's varnishlog: 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963716 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963719 1.0 13 SessionOpen c 10.131.36.170 52969 :80 13 ReqStart c 10.131.36.170 52969 374027480 13 RxRequest c GET 13 RxURL c /default.aspx 13 RxProtocol c HTTP/1.1 13 RxHeader c Host: xx.xx.xx.xx 13 RxHeader c User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.15) Gecko/2009102704 Fedora/3.0.15-1.fc10 Firefox/3.0.15 13 RxHeader c Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 13 RxHeader c Accept-Language: en-us,en;q=0.5 13 RxHeader c Accept-Encoding: gzip,deflate 13 RxHeader c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 13 RxHeader c Keep-Alive: 300 13 RxHeader c Connection: keep-alive 13 RxHeader c Cookie: SESS253f3165f282ca21e0aff96094b2f186=4om57skb5bthlv12hno77nefg3; SESS3c5368fdadd84553468b7aecbb6b8c56=m9okd8i89jeq777sn3c363cge7; ISAWPLB{30E7D663-ACFC-44A1-A8C4-C17514BA8234}={20C55D85-138B-4814-9748-C EA17E9D6759} 13 VCL_call c recv 13 VCL_return c pass 13 VCL_call c pass 13 VCL_return c pass 13 Backend c 22 default default 22 TxRequest - GET 22 TxURL - /default.aspx 22 TxProtocol - HTTP/1.1 22 TxHeader - Host: xx.xx.xx.xx 22 TxHeader - User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.15) Gecko/2009102704 Fedora/3.0.15-1.fc10 Firefox/3.0.15 22 TxHeader - Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 22 TxHeader - Accept-Language: en-us,en;q=0.5 22 TxHeader - Accept-Encoding: gzip,deflate 22 TxHeader - Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 22 TxHeader - Cookie: SESS253f3165f282ca21e0aff96094b2f186=4om57skb5bthlv12hno77nefg3; SESS3c5368fdadd84553468b7aecbb6b8c56=m9okd8i89jeq777sn3c363cge7; ISAWPLB{30E7D663-ACFC-44A1-A8C4-C17514BA8234}={20C55D85-138B-4814-9748-C EA17E9D6759} 22 TxHeader - X-Varnish: 374027480 22 TxHeader - X-Forwarded-For: 10.131.36.170 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963722 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963725 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963728 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963731 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963734 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963737 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963740 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963743 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963746 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963749 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963752 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963755 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963758 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963761 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963764 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963767 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963770 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963773 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963776 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963779 1.0 13 FetchError c http read error: 11 22 BackendClose - default 13 VCL_call c error 13 VCL_return c deliver 13 Length c 471 13 VCL_call c deliver 13 VCL_return c deliver 13 TxProtocol c HTTP/1.1 13 TxStatus c 503 13 TxResponse c Service Unavailable 13 TxHeader c Server: Varnish 13 TxHeader c Retry-After: 0 13 TxHeader c Content-Type: text/html; charset=utf-8 13 TxHeader c Content-Length: 471 13 TxHeader c Date: Sat, 24 Jul 2010 09:29:40 GMT 13 TxHeader c X-Varnish: 374027480 13 TxHeader c Age: 60 13 TxHeader c Via: 1.1 varnish 13 TxHeader c Connection: close 13 ReqEnd c 374027480 1279963720.108640909 1279963780.108922005 0.000058889 60.000254154 0.000026941 13 SessionClose c error 13 StatSess c 10.131.36.170 52969 60 1 1 0 1 0 235 471 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963782 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963785 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963788 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963791 1.0 0 CLI - Rd ping 0 CLI - Wr 0 200 PONG 1279963794 1.0 varnishstats: 69 0.00 0.01 Client connections accepted 605 0.00 0.10 Client requests received 40 0.00 0.01 Backend conn. success 565 0.00 0.09 Backend conn. reuses 16 0.00 0.00 Backend conn. was closed 582 0.00 0.10 Backend conn. recycles 550 0.00 0.09 Fetch with Length 28 0.00 0.00 Fetch chunked 7 . . N struct sess_mem 1 . . N struct sess 0 . . N struct object 8 . . N struct objecthead 1 . . N struct smf 1 . . N large free smf 1 . . N struct vbe_conn 6 . . N struct bereq 10 . . N worker threads 10 0.00 0.00 N worker threads created 1 . . N backends 125 0.00 0.02 Objects sent with write 69 0.00 0.01 Total Sessions 605 0.00 0.10 Total Requests 605 0.00 0.10 Total pass 594 0.00 0.10 Total fetch -------------- next part -------------- An HTML attachment was scrubbed... URL: From A.Hongens at netmatch.nl Fri Oct 1 19:47:57 2010 From: A.Hongens at netmatch.nl (=?iso-8859-1?Q?Angelo_H=F6ngens?=) Date: Fri, 1 Oct 2010 17:47:57 +0000 Subject: How to monitor varnish with snmp In-Reply-To: References: <6944031851727100886@unknownmsgid> Message-ID: <6A7ABA19243F1E4EADD8BB1563CDDCCB076BCE@TIL-EXCH-05.netmatch.local> I extended net-snmp with two scripts, to return some counters: $ cat /usr/local/share/snmp/scripts/NM_VARNISH_TOTALHITS.sh #!/bin/sh /usr/local/bin/varnishstat -1 -f cache_hit | awk '{ print $2 }' $ cat /usr/local/share/snmp/scripts/NM_VARNISH_TOTALREQS.sh #!/bin/sh /usr/local/bin/varnishstat -1 -f client_req | awk '{ print $2 }' These are then imported into cacti, and I can calculate the hit ratio and requests/sec. I'm interested to hear from the Varnish guru's if they suggest any more counter worth graphing: [Description: Description: NMT-NLB-04 - Varnish - Cache hit ratio] [Description: Description: NMT-NLB-04 - Varnish client HTTP requests] -- With kind regards, Angelo H?ngens Systems Administrator ------------------------------------------ NetMatch tourism internet software solutions Ringbaan Oost 2b 5013 CA Tilburg T: +31 (0)13 5811088 F: +31 (0)13 5821239 mailto:A.Hongens at netmatch.nl http://www.netmatch.nl ------------------------------------------ From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Angie T. Muhammad Sent: vrijdag 1 oktober 2010 18:53 To: marcos.corvalan at mercadolibre.com Cc: varnish-misc at varnish-cache.org Subject: Re: How to monitor varnish with snmp I am so much interested in the same topic. I tried creating my own OIDs and getting the output of a shell script into them, but failed, the OID never resturned any value. May any body give us a simple guidance ? On Tue, Sep 14, 2010 at 6:55 PM, > wrote: Hi everyone: How can I monitor data with snmp from varnish process. Since now thanks a lots. Saludos/Regards Marcos Corvalan Infraestructura IT Mercadolibre.com Tronador 4890 - 8 piso Capital Federal | Buenos Aires Argentina | CP 1430 | Tel +54 11 5352-8000 Ext 8426 email marcos.corvalan| msn marcoscorvalan_04 at hotmail.com Mobile : +54 11 37701933 _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org http://lists.varnish-cache.org/mailman/listinfo/varnish-misc -- All the best, Angie T. Muhammad Linux Systems Engineer Al Masry Al Youm http://www.almasryalyoum.com/en -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 19319 bytes Desc: image001.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.png Type: image/png Size: 24373 bytes Desc: image002.png URL: From scaunter at topscms.com Fri Oct 1 23:18:49 2010 From: scaunter at topscms.com (Caunter, Stefan) Date: Fri, 1 Oct 2010 17:18:49 -0400 Subject: varnish restarting sporadically... cache being lost. In-Reply-To: References: Message-ID: <7F0AA702B8A85A4A967C4C8EBAD6902C5025DC@TMG-EVS02.torstar.net> Please post your start params for varnishd and uname -a Stefan Caunter :: Senior Systems Administrator :: TOPS e: scaunter at topscms.com :: m: (416) 561-4871 www.thestar.com www.topscms.com -----Original Message----- From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Ben Nowacky Sent: June-24-10 1:26 PM To: varnish-misc at varnish-cache.org Subject: varnish restarting sporadically... cache being lost. Hoping someone can help... We're just trying to roll out varnish to a somewhat large production site (30million monthly pageviews). Things seem to be working, hitting about a 90% hitrate so far. Using Wordpress for a CMS and standard LAMP stack on the back-end. - Deployed varnish on 1 of our servers yesterday for final testing on the live site. Runs along well, seems to be doing great. Run varnishstat to keep an eye on it, and randomly all the stats reset back to 0, and there's a full cache purge, which means i'm guessing varnish has restarted? I can't, for the life of me, figure out why it's been restarting. I've tweeked settings that I've seen recommendations for, upped cache/thread/etc an still seems to be happening. Not sure if anyone else has seen this behavior or not, but if anyone has any suggestions for troubleshooting, or figuring out what's going on, would appreciate any help ! I've got varnish running with persistence now, so it's helping a bit when there's a restart since it's not completely unprimed, but need to figure out the cause of the instability. Any help would be awesome... Thanks ! _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From angie.tawfik at gmail.com Sat Oct 2 15:10:03 2010 From: angie.tawfik at gmail.com (Angie T. Muhammad) Date: Sat, 2 Oct 2010 15:10:03 +0200 Subject: How to monitor varnish with snmp In-Reply-To: <6A7ABA19243F1E4EADD8BB1563CDDCCB076BCE@TIL-EXCH-05.netmatch.local> References: <6944031851727100886@unknownmsgid> <6A7ABA19243F1E4EADD8BB1563CDDCCB076BCE@TIL-EXCH-05.netmatch.local> Message-ID: Thanks Angelo On 10/1/10, Angelo H?ngens wrote: > I extended net-snmp with two scripts, to return some counters: > > $ cat /usr/local/share/snmp/scripts/NM_VARNISH_TOTALHITS.sh > #!/bin/sh > /usr/local/bin/varnishstat -1 -f cache_hit | awk '{ print $2 }' > > $ cat /usr/local/share/snmp/scripts/NM_VARNISH_TOTALREQS.sh > #!/bin/sh > /usr/local/bin/varnishstat -1 -f client_req | awk '{ print $2 }' > > These are then imported into cacti, and I can calculate the hit ratio and > requests/sec. I'm interested to hear from the Varnish guru's if they suggest > any more counter worth graphing: > > [Description: Description: NMT-NLB-04 - Varnish - Cache hit > ratio] > > [Description: Description: NMT-NLB-04 - Varnish client HTTP > requests] > > > -- > > > With kind regards, > > Angelo H?ngens > > Systems Administrator > > ------------------------------------------ > NetMatch > tourism internet software solutions > > Ringbaan Oost 2b > 5013 CA Tilburg > T: +31 (0)13 5811088 > F: +31 (0)13 5821239 > > mailto:A.Hongens at netmatch.nl > http://www.netmatch.nl > ------------------------------------------ > > From: varnish-misc-bounces at varnish-cache.org > [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Angie T. > Muhammad > Sent: vrijdag 1 oktober 2010 18:53 > To: marcos.corvalan at mercadolibre.com > Cc: varnish-misc at varnish-cache.org > Subject: Re: How to monitor varnish with snmp > > I am so much interested in the same topic. I tried creating my own OIDs and > getting the output of a shell script into them, but failed, the OID never > resturned any value. May any body give us a simple guidance ? > On Tue, Sep 14, 2010 at 6:55 PM, > > > wrote: > Hi everyone: > How can I monitor data with snmp from varnish process. > Since now thanks a lots. > > > Saludos/Regards > > Marcos Corvalan > Infraestructura IT > Mercadolibre.com > Tronador 4890 - 8 piso Capital Federal | Buenos Aires > Argentina | CP 1430 | Tel +54 11 5352-8000 Ext 8426 > email marcos.corvalan| msn > marcoscorvalan_04 at hotmail.com > Mobile : +54 11 37701933 > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > > > > -- > All the best, > Angie T. Muhammad > Linux Systems Engineer > Al Masry Al Youm > http://www.almasryalyoum.com/en > -- All the best, Angie T. Muhammad Linux Systems Engineer Al Masry Al Youm http://www.almasryalyoum.com/en From tfheen at varnish-software.com Mon Oct 4 10:16:40 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Mon, 04 Oct 2010 10:16:40 +0200 Subject: Byte range and varnish In-Reply-To: (James A. Robinson's message of "Fri, 1 Oct 2010 06:05:36 -0700") References: <87vd5m6pff.fsf@qurzaw.linpro.no> Message-ID: <87d3rqe5jr.fsf@qurzaw.linpro.no> ]] "James A. Robinson" [...] | It advertises Accept-Range: bytes but if someone passes in a valid | multi-sequence set of ranges, Varnish fails to do the right thing. Can you please file a bug about this? | It also fails to gracefully get out of the way on a pipe request, and | drops Content-Range headers from the backend. And this, please? -- Tollef Fog Heen Varnish Software t: +47 21 54 41 73 From tfheen at varnish-software.com Mon Oct 4 10:26:45 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Mon, 04 Oct 2010 10:26:45 +0200 Subject: Exposing real backend name in vcl_recv In-Reply-To: (s. u.'s message of "Tue, 13 Jul 2010 15:06:03 +0100") References: Message-ID: <878w2ee52y.fsf@qurzaw.linpro.no> ]] s u | Is there way to access the real backend's name - or something else - which | would tell me to which backend a request will be sent during the recv phase? No, simply because it hasn't been chosen which (or even if) at that point. -- Tollef Fog Heen Varnish Software t: +47 21 54 41 73 From tfheen at varnish-software.com Mon Oct 4 10:37:19 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Mon, 04 Oct 2010 10:37:19 +0200 Subject: vcl_hash authentication questions In-Reply-To: <4BED5A7F-12EF-4FD9-B023-D4B7D487C376@buyways.nl> (Ron van der Vegt's message of "Thu, 8 Apr 2010 12:45:57 +0200") References: <4BED5A7F-12EF-4FD9-B023-D4B7D487C376@buyways.nl> Message-ID: <874od2e4lc.fsf@qurzaw.linpro.no> ]] Ron van der Vegt | What do you suggest? Are there other approaches that fit the use-case? How did | or would you solve this problem with Varnish? I'd set a cookie on the backend, sign it using a HMAC, include an expiry time in the cookie value and validate the HMAC signature + expiry value using inline C in Varnish. I don't think there's any examples of this, but it shouldn't be that hard to write something. -- Tollef Fog Heen Varnish Software t: +47 21 54 41 73 From tfheen at varnish-software.com Mon Oct 4 10:41:28 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Mon, 04 Oct 2010 10:41:28 +0200 Subject: How to monitor varnish with snmp In-Reply-To: (Angie T. Muhammad's message of "Fri, 1 Oct 2010 18:53:24 +0200") References: <6944031851727100886@unknownmsgid> Message-ID: <87zkuucptz.fsf@qurzaw.linpro.no> ]] "Angie T. Muhammad" | I am so much interested in the same topic. I tried creating my own OIDs and | getting the output of a shell script into them, but failed, the OID never | resturned any value. May any body give us a simple guidance ? There's the varnisnmp project on Sourceforge which I've played with in the past. It seemed to work fine in my testing and has official OIDs assigned and all. -- Tollef Fog Heen Varnish Software t: +47 21 54 41 73 From l at lrowe.co.uk Mon Oct 4 13:07:20 2010 From: l at lrowe.co.uk (Laurence Rowe) Date: Mon, 4 Oct 2010 12:07:20 +0100 Subject: vcl_hash authentication questions In-Reply-To: <874od2e4lc.fsf@qurzaw.linpro.no> References: <4BED5A7F-12EF-4FD9-B023-D4B7D487C376@buyways.nl> <874od2e4lc.fsf@qurzaw.linpro.no> Message-ID: On 4 October 2010 09:37, Tollef Fog Heen wrote: > ]] Ron van der Vegt > > | What do you suggest? Are there other approaches that fit the use-case? How did > | or would you solve this problem with Varnish? > > I'd set a cookie on the backend, sign it using a HMAC, include an expiry > time in the cookie value and validate the HMAC signature + expiry value > using inline C in Varnish. ?I don't think there's any examples of this, > but it shouldn't be that hard to write something. This sounds similar to the mod_auth_tkt scheme - http://www.openfusion.com.au/labs/mod_auth_tkt/ - although this is C code it relies heavily on the apache libraries and as such does not look trivial to convert to inline C code in varnish. I implemented an HMAC SHA-256 variant of mod_auth_tkt in plone.session - http://pypi.python.org/pypi/plone.session - see tktauth.py for the python code that generates and validates these cookies (usable outside of Plone). http://dev.plone.org/plone/browser/plone.session/trunk/plone/session/tktauth.py I think this is your best route, and I would certainly find it interesting to see one of these authentication schemes implemented for Varnish. You could then do token based authorization in vcl_deliver, checking that the user has one of the tokens listed in a response header. Also take a look at the varnish-dev list. Nils Goroll has been working on "Digests and data encoding in Varnish" and there is considerable overlap there. Laurence From ksorensen at nordija.com Mon Oct 4 18:16:24 2010 From: ksorensen at nordija.com (Kristian =?ISO-8859-1?Q?Gr=F8nfeldt_S=F8rensen?=) Date: Mon, 04 Oct 2010 18:16:24 +0200 Subject: Adjusting keep-alive timeout of backend-connections Message-ID: <1286208984.23139.10.camel@localhost> Hi, Is there any way of configuring how long a keep-alive connection to a backend server will be kept open? When load-testing my Varnish-2.1.3 setup I see intermittent 503-errors. arnishlog tells me that it is caused by a HTTP read error (connection reset): 98 FetchError c http read error: 104 (Full log of the request is shown below.) As far as I can understand, this error happens because the backend-server closed the connection at the same time that Varnish sent the request. I suspect errors like these will disappear with 2.1.4, as I believe this is the same problem reported in issue #749, which have been fixed in 2.1.4 by retrying on a new connection. However, my preferred way of dealing with this would be to configure the keep-alive timeout of backend-connection to be slightly lower than the configured timeout of the backend server, so that Varnish will always be the one which closes the connection (assuming that the backend server does not prematurely close the connection). This way I should be able to avoid restarting the request. Is there an easy way of achieving this with 2.1.3? Entire request: 98 ReqStart c 192.168.1.24 35632 788531137 98 RxRequest c POST 98 RxURL c /my/url 98 RxProtocol c HTTP/1.1 98 RxHeader c Connection: keep-alive 98 RxHeader c Accept-Encoding: gzip,deflate 98 RxHeader c Cache-Control: no-cache 98 RxHeader c User-Agent: Mozilla/5.0 (X11; U; Linux sh4; rv:1.7.12) Gecko/20100326 Kreatel 98 RxHeader c Pragma: no-cache 98 RxHeader c Content-Type: text/plain 98 RxHeader c Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5 98 RxHeader c Keep-Alive: 300 98 RxHeader c Content-Length: 228 98 RxHeader c Host: 192.168.3.20 98 VCL_call c recv pass 98 VCL_call c hash hash 98 VCL_call c pass pass 98 Backend c 95 defaultdirector backend1 98 FetchError c http read error: 104 98 VCL_call c error deliver 98 VCL_call c deliver deliver 98 TxProtocol c HTTP/1.1 98 TxStatus c 503 98 TxResponse c Service Unavailable 98 TxHeader c Server: Varnish 98 TxHeader c Retry-After: 0 98 TxHeader c Content-Type: text/html; charset=utf-8 98 TxHeader c Content-Length: 418 98 TxHeader c Date: Mon, 04 Oct 2010 09:55:29 GMT 98 TxHeader c X-Varnish: 788531137 98 TxHeader c Age: 0 98 TxHeader c Via: 1.1 varnish 98 TxHeader c Connection: close 98 Length c 418 98 ReqEnd c 788531137 1286186129.917763948 1286186129.930047989 0.002107859 0.012248993 0.000035048 Regards Kristian S?rensen From chm0dz at gmail.com Mon Oct 4 19:12:06 2010 From: chm0dz at gmail.com (Cristiano Fernandes (chm0d)) Date: Mon, 4 Oct 2010 14:12:06 -0300 Subject: 301 not work Message-ID: :( if (req.http.host == "domain.com") { error 301 "http://newdomain.com/home/"; } Why not? Tks -------------- next part -------------- An HTML attachment was scrubbed... URL: From chm0dz at gmail.com Mon Oct 4 19:16:33 2010 From: chm0dz at gmail.com (Cristiano Fernandes (chm0d)) Date: Mon, 4 Oct 2010 14:16:33 -0300 Subject: 301 not work In-Reply-To: References: Message-ID: Sorry, found error :) On Mon, Oct 4, 2010 at 2:12 PM, Cristiano Fernandes (chm0d) < chm0dz at gmail.com> wrote: > :( > > if (req.http.host == "domain.com") { > error 301 "http://newdomain.com/home/"; > } > > Why not? > > Tks > -- _______ ? Cristiano Fernandes SysAdmin Portal R7 (Rede RECORD) Google is my shepherd, no want shall I know -------------- next part -------------- An HTML attachment was scrubbed... URL: From jim.robinson at stanford.edu Mon Oct 4 21:31:14 2010 From: jim.robinson at stanford.edu (James A. Robinson) Date: Mon, 4 Oct 2010 12:31:14 -0700 Subject: Byte range and varnish In-Reply-To: <87d3rqe5jr.fsf@qurzaw.linpro.no> References: <87vd5m6pff.fsf@qurzaw.linpro.no> <87d3rqe5jr.fsf@qurzaw.linpro.no> Message-ID: On Mon, Oct 4, 2010 at 01:16, Tollef Fog Heen wrote: > ]] "James A. Robinson" > > [...] > > | It advertises Accept-Range: bytes but if someone passes in a valid > | multi-sequence set of ranges, Varnish fails to do the right thing. > > Can you please file a bug about this? > > | It also fails to gracefully get out of the way on a pipe request, and > | drops Content-Range headers from the backend. > > And this, please? Sure. And to correct myself here, I should have said that it fails to get out of the way on a pass request (I emailed about this to the varnish-misc list earlier). I'll submit bug reports today. Jim From moseleymark at gmail.com Mon Oct 4 21:52:45 2010 From: moseleymark at gmail.com (Mark Moseley) Date: Mon, 4 Oct 2010 12:52:45 -0700 Subject: Varnish dying in stevedore.c with lots of free space Message-ID: I get "Panic message: Assert error in STV_alloc(), stevedore.c line 192" a lot. The culprit is almost always huge files (and this is web hosting so can be almost anything). The example that this came from was a 920meg Flash video, but I've seen the same thing triggered by zip/wmv/mpg/rar, i.e. anything that can get huge. Looking in the list archives, this is usually diagnosed as being out of storage space. In this case, there was almost certainly space available. This is sm_bfree in the minutes surrounding this log entry: Mon Oct 4 11:59:00 2010: 7190153625 Mon Oct 4 12:00:00 2010: 7127413555 Mon Oct 4 12:01:00 2010: 15418954547 Mon Oct 4 12:02:00 2010: 22966786458 The probability that there was also another 6.1gb dropped into the cache in the same minute is really really low. This is 64-bit Lenny on Dell Poweredge 1950s, all with 4gb. I've seen this error on disk-based caches running on regular drives as well as SSDs. I got the same stevedore.c error on another box over the weekend and that's got a 47 gig varnish cachefile, and I've never even seen that dip less than 10gb free; when this box's varnishd hit that error, it had around 27gb free in sm_bfree. It happens regardless of the file type, though it's always very very large (at least every example I've looked at). I've seen this error a few times a day whenever I'm using disk-based cache -- with malloc, I get the stevedore.c error and "Panic message: Assert error in fetch_straight(), cache_fetch.c line 64", but less often. Is there anything besides out-of-space that can trigger this? For the (poorly sanitized) logs below, this is the invocation: /srv/varnish/varnish-amd64/sbin/varnishd -a :8099 -T :8100 -f /srv/varnish/etc/varnish/customer.vcl-amd64-ssd -t 0 -l 80m -s file,/var/cache/varnish/cache,23000M -u nobody -P /var/run/varnishd -p listen_depth 4096 -p thread_pools 6 -p thread_pool_max 800 -p thread_pool_min 200 -p lru_interval 60 -p cli_timeout 30 -p ping_interval 5 -p default_grace 120 -p thread_pool_stack 1048576 -p cache_vbe_conns on Anything I can do to further debug this? I'm also entirely receptive to any recommendations on varnishd command line :) Thanks! Oct 4 16:01:25 myserver varnishd[1864]: Child (17243) died signal=6 Oct 4 16:01:25 myserver varnishd[1864]: Child (17243) Panic message: Assert error in STV_alloc(), stevedore.c line 192: Oct 4 12:01:25 myserver Condition((st): != NULL) not true. Oct 4 12:01:25 myserver thread: = (cache-worker) Oct 4 12:01:25 myserver ident: = Linux,2.6.35.6,x86_64,-sfile,-hcritbit,epoll Oct 4 12:01:25 myserver Backtrace: Oct 4 12:01:25 myserver 0x4248f3: pan_ic+b3 Oct 4 12:01:25 myserver 0x43a655: STV_alloc+125 Oct 4 12:01:25 myserver 0x41c5f6: FetchBody+4e6 Oct 4 12:01:25 myserver 0x413d00: cnt_fetch+680 Oct 4 12:01:25 myserver 0x4152fd: CNT_Session+35d Oct 4 12:01:25 myserver 0x426d93: wrk_do_cnt_sess+93 Oct 4 12:01:25 myserver 0x42608e: wrk_thread_real+34e Oct 4 12:01:25 myserver 0x3b034f5ffc7: _end+3b0348eaa1f Oct 4 12:01:25 myserver 0x3b03483a64d: _end+3b0341c50a5 Oct 4 12:01:25 myserver sp: = 0x3aa9072b008 { Oct 4 12:01:25 myserver fd: = 12, id = 12, xid = 1657464786, Oct 4 12:01:25 myserver client: = 192.168.0.40:43834, Oct 4 12:01:25 myserver step: = STP_FETCH, Oct 4 12:01:25 myserver handling: = deliver, Oct 4 12:01:25 myserver err_code: = 200, err_reason = (null), Oct 4 12:01:25 myserver restarts: = 0, esis = 0 Oct 4 12:01:25 myserver ws: = 0x3aa9072b078 { Oct 4 12:01:25 myserver id: = "sess", Oct 4 12:01:25 myserver {s,f,r,e}: = {0x3aa9072bcd0,+576,(nil),+65536}, Oct 4 12:01:25 myserver },: Oct 4 12:01:25 myserver http[req]: = { Oct 4 12:01:25 myserver ws: = 0x3aa9072b078[sess] Oct 4 12:01:25 myserver "GET",: Oct 4 12:01:25 myserver "/my/url",: Oct 4 12:01:25 myserver "HTTP/1.0",: Oct 4 12:01:25 myserver "Host: example.com Oct 4 12:01:25 myserver "X-EN-FWD-IP: 1.2.3.4", Oct 4 12:01:25 myserver "Connection: close", Oct 4 12:01:25 myserver "Accept: */*", Oct 4 12:01:25 myserver "Accept-Language: en-US", Oct 4 12:01:25 myserver "Referer: http://example.com/some/url", Oct 4 12:01:25 myserver "x-flash-version: 10,0,32,18", Oct 4 12:01:25 myserver "User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.2; OfficeLiveConnector.1.4; OfficeLivePatch.1.3)", Oct 4 12:01:25 myserver "Accept-Encoding: gzip", Oct 4 12:01:25 myserver },: Oct 4 12:01:25 myserver worker: = 0x70424e70 { Oct 4 12:01:25 myserver ws: = 0x70424fe0 { Oct 4 12:01:25 myserver id: = "wrk", Oct 4 12:01:25 myserver {s,f,r,e}: = {0x70412e00,+32792,(nil),+65536}, Oct 4 12:01:25 myserver },: Oct 4 12:01:25 myserver http[bereq]: = { Oct 4 12:01:25 myserver ws: = 0x70424fe0[wrk] Oct 4 12:01:25 myserver "GET",: Oct 4 12:01:25 myserver "/my/url",: Oct 4 12:01:25 myserver "HTTP/1.1",: Oct 4 12:01:25 myserver "Host: example.com Oct 4 12:01:25 myserver "X-EN-FWD-IP: 1.2.3.4", Oct 4 12:01:25 myserver "Accept: */*", Oct 4 12:01:25 myserver "Accept-Language: en-US", Oct 4 12:01:25 myserver "Referer: http://example.com/some/url", Oct 4 12:01:25 myserver "x-flash-version: 10,0,32,18", Oct 4 12:01:25 myserver "User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.2; OfficeLiveConnector.1.4; OfficeLivePatch.1.3)", Oct 4 12:01:25 myserver "Accept-Encoding: gzip", Oct 4 12:01:25 myserver "X-Varnish: 1657464786", Oct 4 12:01:25 myserver },: Oct 4 12:01:25 myserver http[beresp]: = { Oct 4 12:01:25 myserver ws: = 0x70424fe0[wrk] Oct 4 12:01:25 myserver "HTTP/1.1",: Oct 4 12:01:25 myserver "200",: Oct 4 12:01:25 myserver "OK",: Oct 4 12:01:25 myserver "Date: Mon, 04 Oct 2010 15:59:45 GMT", Oct 4 12:01:25 myserver "Server: Apache", Oct 4 12:01:25 myserver "Last-Modified: Mon, 30 Aug 2010 23:48:54 GMT", Oct 4 12:01:25 myserver "ETag: "b40ecc95-36e43bb5-48f131b7121b1"", Oct 4 12:01:25 myserver "Accept-Ranges: bytes", Oct 4 12:01:25 myserver "Content-Length: 920927157", Oct 4 12:01:25 myserver "Cache-Control: max-age=14400, public", Oct 4 12:01:25 myserver "Expires: Mon, 04 Oct 2010 19:59:45 GMT", Oct 4 12:01:25 myserver "Connection: close", Oct 4 12:01:25 myserver "Content-Type: text/html", Oct 4 12:01:25 myserver },: Oct 4 12:01:25 myserver },: Oct 4 12:01:25 myserver vcl: = { Oct 4 12:01:25 myserver srcname: = { Oct 4 12:01:25 myserver "input",: Oct 4 12:01:25 myserver "Default",: Oct 4 12:01:25 myserver "/srv/varnish/etc/varnish/backends_def.vcl",: Oct 4 12:01:25 myserver "/srv/varnish/etc/varnish/backends_set.vcl",: Oct 4 12:01:25 myserver },: Oct 4 12:01:25 myserver },: Oct 4 12:01:25 myserver obj: = 0x3ac3f979000 { Oct 4 12:01:25 myserver xid: = 1657464786, Oct 4 12:01:25 myserver ws: = 0x3ac3f979020 { Oct 4 12:01:25 myserver id: = "obj", Oct 4 12:01:25 myserver {s,f,r,e}: = {0x3ac3f979228,+280,(nil),+3544}, Oct 4 12:01:25 myserver },: Oct 4 12:01:25 myserver http[obj]: = { Oct 4 12:01:25 myserver ws: = 0x3ac3f979020[obj] Oct 4 12:01:25 myserver "HTTP/1.1",: Oct 4 12:01:25 myserver "200",: Oct 4 12:01:25 myserver "OK",: Oct 4 12:01:25 myserver "Date: Mon, 04 Oct 2010 15:59:45 GMT", Oct 4 12:01:25 myserver "Server: Apache", Oct 4 12:01:25 myserver "Last-Modified: Mon, 30 Aug 2010 23:48:54 GMT", Oct 4 12:01:25 myserver "ETag: "b40ecc95-36e43bb5-48f131b7121b1"", Oct 4 12:01:25 myserver "Cache-Control: max-age=14400, public", Oct 4 12:01:25 myserver "Expires: Mon, 04 Oct 2010 19:59:45 GMT", Oct 4 12:01:25 myserver "Content-Type: text/html", Oct 4 12:01:25 myserver },: Oct 4 12:01:25 myserver len: = 0, Oct 4 12:01:25 myserver store: = { Oct 4 12:01:25 myserver },: Oct 4 12:01:25 myserver },: Oct 4 12:01:25 myserver },: Oct 4 16:01:26 myserver varnishd[1864]: Child cleanup complete Oct 4 16:01:26 myserver varnishd[1864]: child (25617) Started Oct 4 16:01:26 myserver varnishd[1864]: Child (25617) said Oct 4 16:01:26 myserver varnishd[1864]: Child (25617) said Child starts Oct 4 16:01:26 myserver varnishd[1864]: Child (25617) said managed to mmap 24117248000 bytes of 24117248000 Oct 4 16:01:30 myserver varnishd[1864]: Child (25617) died signal=6 Oct 4 16:01:30 myserver varnishd[1864]: Child (25617) Panic message: Assert error in STV_alloc(), stevedore.c line 192: Oct 4 12:01:30 myserver Condition((st): != NULL) not true. Oct 4 12:01:30 myserver thread: = (cache-worker) Oct 4 12:01:30 myserver ident: = Linux,2.6.35.6,x86_64,-sfile,-hcritbit,epoll Oct 4 12:01:30 myserver Backtrace: Oct 4 12:01:30 myserver 0x4248f3: pan_ic+b3 Oct 4 12:01:30 myserver 0x43a655: STV_alloc+125 Oct 4 12:01:30 myserver 0x41c5f6: FetchBody+4e6 Oct 4 12:01:30 myserver 0x413d00: cnt_fetch+680 Oct 4 12:01:30 myserver 0x4152fd: CNT_Session+35d Oct 4 12:01:30 myserver 0x426d93: wrk_do_cnt_sess+93 Oct 4 12:01:30 myserver 0x42608e: wrk_thread_real+34e Oct 4 12:01:30 myserver 0x3b034f5ffc7: _end+3b0348eaa1f Oct 4 12:01:30 myserver 0x3b03483a64d: _end+3b0341c50a5 Oct 4 12:01:30 myserver sp: = 0x3aa90946008 { Oct 4 12:01:30 myserver fd: = 33, id = 33, xid = 733630841, Oct 4 12:01:30 myserver client: = 192.168.0.40:44005, Oct 4 12:01:30 myserver step: = STP_FETCH, Oct 4 12:01:30 myserver handling: = deliver, Oct 4 12:01:30 myserver err_code: = 200, err_reason = (null), Oct 4 12:01:30 myserver restarts: = 0, esis = 0 Oct 4 12:01:30 myserver ws: = 0x3aa90946078 { Oct 4 12:01:30 myserver id: = "sess", Oct 4 12:01:30 myserver {s,f,r,e}: = {0x3aa90946cd0,+576,(nil),+65536}, Oct 4 12:01:30 myserver },: Oct 4 12:01:30 myserver http[req]: = { Oct 4 12:01:30 myserver ws: = 0x3aa90946078[sess] Oct 4 12:01:30 myserver "GET",: Oct 4 12:01:30 myserver "/my/url",: Oct 4 12:01:30 myserver "HTTP/1.0",: Oct 4 12:01:30 myserver "Host: example.com Oct 4 12:01:30 myserver "X-EN-FWD-IP: 1.2.3.4", Oct 4 12:01:30 myserver "Connection: close", Oct 4 12:01:30 myserver "Accept: */*", Oct 4 12:01:30 myserver "Accept-Language: en-US", Oct 4 12:01:30 myserver "Referer: http://example.com/some/url", Oct 4 12:01:30 myserver "x-flash-version: 10,0,32,18", Oct 4 12:01:30 myserver "User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.2; OfficeLiveConnector.1.4; OfficeLivePatch.1.3)", Oct 4 12:01:30 myserver "Accept-Encoding: gzip", Oct 4 12:01:30 myserver },: Oct 4 12:01:30 myserver worker: = 0x4d9b6e70 { Oct 4 12:01:30 myserver ws: = 0x4d9b6fe0 { Oct 4 12:01:30 myserver id: = "wrk", Oct 4 12:01:30 myserver {s,f,r,e}: = {0x4d9a4e00,+32792,(nil),+65536}, Oct 4 12:01:30 myserver },: Oct 4 12:01:30 myserver http[bereq]: = { Oct 4 12:01:30 myserver ws: = 0x4d9b6fe0[wrk] Oct 4 12:01:30 myserver "GET",: Oct 4 12:01:30 myserver "/my/url",: Oct 4 12:01:30 myserver "HTTP/1.1",: Oct 4 12:01:30 myserver "Host: example.com Oct 4 12:01:30 myserver "X-EN-FWD-IP: 1.2.3.4", Oct 4 12:01:30 myserver "Accept: */*", Oct 4 12:01:30 myserver "Accept-Language: en-US", Oct 4 12:01:30 myserver "Referer: http://example.com/some/url", Oct 4 12:01:30 myserver "x-flash-version: 10,0,32,18", Oct 4 12:01:30 myserver "User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.2; OfficeLiveConnector.1.4; OfficeLivePatch.1.3)", Oct 4 12:01:30 myserver "Accept-Encoding: gzip", Oct 4 12:01:30 myserver "X-Varnish: 733630841", Oct 4 12:01:30 myserver },: Oct 4 12:01:30 myserver http[beresp]: = { Oct 4 12:01:30 myserver ws: = 0x4d9b6fe0[wrk] Oct 4 12:01:30 myserver "HTTP/1.1",: Oct 4 12:01:30 myserver "200",: Oct 4 12:01:30 myserver "OK",: Oct 4 12:01:30 myserver "Date: Mon, 04 Oct 2010 16:01:30 GMT", Oct 4 12:01:30 myserver "Server: Apache", Oct 4 12:01:30 myserver "Last-Modified: Mon, 30 Aug 2010 23:48:54 GMT", Oct 4 12:01:30 myserver "ETag: "b40ecc95-36e43bb5-48f131b7121b1"", Oct 4 12:01:30 myserver "Accept-Ranges: bytes", Oct 4 12:01:30 myserver "Content-Length: 920927157", Oct 4 12:01:30 myserver "Cache-Control: max-age=14400, public", Oct 4 12:01:30 myserver "Expires: Mon, 04 Oct 2010 20:01:30 GMT", Oct 4 12:01:30 myserver "Connection: close", Oct 4 12:01:30 myserver "Content-Type: text/html", Oct 4 12:01:30 myserver },: Oct 4 12:01:30 myserver },: Oct 4 12:01:30 myserver vcl: = { Oct 4 12:01:30 myserver srcname: = { Oct 4 12:01:30 myserver "input",: Oct 4 12:01:30 myserver "Default",: Oct 4 12:01:30 myserver "/srv/varnish/etc/varnish/backends_def.vcl",: Oct 4 12:01:30 myserver "/srv/varnish/etc/varnish/backends_set.vcl",: Oct 4 12:01:30 myserver },: Oct 4 12:01:30 myserver },: Oct 4 12:01:30 myserver obj: = 0x3b001f66000 { Oct 4 12:01:30 myserver xid: = 733630841, Oct 4 12:01:30 myserver ws: = 0x3b001f66020 { Oct 4 12:01:30 myserver id: = "obj", Oct 4 12:01:30 myserver {s,f,r,e}: = {0x3b001f66228,+280,(nil),+3544}, Oct 4 12:01:30 myserver },: Oct 4 12:01:30 myserver http[obj]: = { Oct 4 12:01:30 myserver ws: = 0x3b001f66020[obj] Oct 4 12:01:30 myserver "HTTP/1.1",: Oct 4 12:01:30 myserver "200",: Oct 4 12:01:30 myserver "OK",: Oct 4 12:01:30 myserver "Date: Mon, 04 Oct 2010 16:01:30 GMT", Oct 4 12:01:30 myserver "Server: Apache", Oct 4 12:01:30 myserver "Last-Modified: Mon, 30 Aug 2010 23:48:54 GMT", Oct 4 12:01:30 myserver "ETag: "b40ecc95-36e43bb5-48f131b7121b1"", Oct 4 12:01:30 myserver "Cache-Control: max-age=14400, public", Oct 4 12:01:30 myserver "Expires: Mon, 04 Oct 2010 20:01:30 GMT", Oct 4 12:01:30 myserver "Content-Type: text/html", Oct 4 12:01:30 myserver },: Oct 4 12:01:30 myserver len: = 0, Oct 4 12:01:30 myserver store: = { Oct 4 12:01:30 myserver },: Oct 4 12:01:30 myserver },: Oct 4 12:01:30 myserver },: Oct 4 16:01:30 myserver varnishd[1864]: Child cleanup complete Oct 4 16:01:30 myserver varnishd[1864]: child (25832) Started Oct 4 16:01:30 myserver varnishd[1864]: Child (25832) said Oct 4 16:01:30 myserver varnishd[1864]: Child (25832) said Child starts Oct 4 16:01:30 myserver varnishd[1864]: Child (25832) said managed to mmap 24117248000 bytes of 24117248000 From jim.robinson at stanford.edu Mon Oct 4 23:11:15 2010 From: jim.robinson at stanford.edu (James A. Robinson) Date: Mon, 4 Oct 2010 14:11:15 -0700 Subject: Byte range and varnish In-Reply-To: References: <87vd5m6pff.fsf@qurzaw.linpro.no> <87d3rqe5jr.fsf@qurzaw.linpro.no> Message-ID: Submitted as: http://www.varnish-cache.org/trac/ticket/788 http://www.varnish-cache.org/trac/ticket/789 On Mon, Oct 4, 2010 at 12:31, James A. Robinson wrote: > On Mon, Oct 4, 2010 at 01:16, Tollef Fog Heen > wrote: >> ]] "James A. Robinson" >> >> [...] >> >> | It advertises Accept-Range: bytes but if someone passes in a valid >> | multi-sequence set of ranges, Varnish fails to do the right thing. >> >> Can you please file a bug about this? >> >> | It also fails to gracefully get out of the way on a pipe request, and >> | drops Content-Range headers from the backend. >> >> And this, please? > > > Sure. ?And to correct myself here, I should have said that it fails > to get out of the way on a pass request (I emailed about this to the > varnish-misc list earlier). ?I'll submit bug reports today. From tfheen at varnish-software.com Tue Oct 5 08:28:23 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Tue, 05 Oct 2010 08:28:23 +0200 Subject: Adjusting keep-alive timeout of backend-connections In-Reply-To: <1286208984.23139.10.camel@localhost> ("Kristian =?utf-8?Q?Gr?= =?utf-8?Q?=C3=B8nfeldt=09S=C3=B8rensen=22's?= message of "Mon, 04 Oct 2010 18:16:24 +0200") References: <1286208984.23139.10.camel@localhost> Message-ID: <87iq1hcfw8.fsf@qurzaw.linpro.no> ]] Kristian Gr?nfeldt S?rensen | Is there any way of configuring how long a keep-alive connection to a | backend server will be kept open? No, we don't close backend connections, the backend server does, and I don't believe there's any way to get varnish to close connections due to a timeout. -- Tollef Fog Heen Varnish Software t: +47 21 54 41 73 From tfheen at varnish-software.com Tue Oct 5 08:29:49 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Tue, 05 Oct 2010 08:29:49 +0200 Subject: Byte range and varnish In-Reply-To: (James A. Robinson's message of "Mon, 4 Oct 2010 14:11:15 -0700") References: <87vd5m6pff.fsf@qurzaw.linpro.no> <87d3rqe5jr.fsf@qurzaw.linpro.no>

Message-ID: <87eic5cftu.fsf@qurzaw.linpro.no> ]] "James A. Robinson" | Submitted as: | | http://www.varnish-cache.org/trac/ticket/788 | | http://www.varnish-cache.org/trac/ticket/789 Thanks! We'll take a look at those in our weekly bug sweep which happens on Mondays at 1100 UTC, unless somebody in the team gets to them before then. -- Tollef Fog Heen Varnish Software t: +47 21 54 41 73 From ccastro at altavoz.net Wed Oct 6 01:24:10 2010 From: ccastro at altavoz.net (Claudio Castro) Date: Tue, 05 Oct 2010 19:24:10 -0400 Subject: New on ESI, page example doesn't work Message-ID: <4CABB39A.2090301@altavoz.net> hi, I'm having excellent results using a basic varnish config on a large traffic site (200-250 Mbps), now i want to step up using ESI on my dynamic pages, but i try the ESI example on varnish wiki, with no luck. test.html: The time is: at this very moment. date.cgi: #!/bin/sh echo 'Content-type: text/html' echo '' date "+%Y-%m-%d %H:%M" vcl: sub vcl_fetch { if (req.url == "/test.html") { esi; /* Do ESI processing */ set obj.ttl = 24 h; } elseif (req.url == "/cgi-bin/date.cgi") { set obj.ttl = 1m; } } I get date.cgi correctly, but esi include seems to be executed just once (no change on date info), any hint? Thanks in advance, FreeBSD 8.0-RELEASE amd64 varnishd (varnish-2.1.3 SVN ) -- Claudio Castro N. Ingeniero Jefe Area de Plataforma AltaVoz S.A. http://www.altavoz.net Vi?a del Mar: 2 Poniente 355 of 53 +56 32 276 8060 Santiago: Pedro de Valdivia 555 of 315 +56 2 585 4264 From angie.tawfik at gmail.com Wed Oct 6 06:53:55 2010 From: angie.tawfik at gmail.com (Angie T. Muhammad) Date: Wed, 6 Oct 2010 06:53:55 +0200 Subject: How to monitor varnish with snmp In-Reply-To: <87zkuucptz.fsf@qurzaw.linpro.no> References: <6944031851727100886@unknownmsgid> <87zkuucptz.fsf@qurzaw.linpro.no> Message-ID: Trying it, and will feed you back. Thanks for pointing me to the project :) On Mon, Oct 4, 2010 at 10:41 AM, Tollef Fog Heen < tfheen at varnish-software.com> wrote: > ]] "Angie T. Muhammad" > > | I am so much interested in the same topic. I tried creating my own OIDs > and > | getting the output of a shell script into them, but failed, the OID never > | resturned any value. May any body give us a simple guidance ? > > There's the varnisnmp project on Sourceforge which I've played with in > the past. It seemed to work fine in my testing and has official OIDs > assigned and all. > > -- > Tollef Fog Heen > Varnish Software > t: +47 21 54 41 73 > -- All the best, Angie T. Muhammad Linux Systems Engineer Al Masry Al Youm http://www.almasryalyoum.com/en -------------- next part -------------- An HTML attachment was scrubbed... URL: From l at lrowe.co.uk Wed Oct 6 17:20:39 2010 From: l at lrowe.co.uk (Laurence Rowe) Date: Wed, 6 Oct 2010 16:20:39 +0100 Subject: New on ESI, page example doesn't work In-Reply-To: <4CABB39A.2090301@altavoz.net> References: <4CABB39A.2090301@altavoz.net> Message-ID: On 6 October 2010 00:24, Claudio Castro wrote: > vcl: > sub vcl_fetch { > > ? if (req.url == "/test.html") { > ? ? ? esi; ?/* Do ESI processing */ > ? ? ? set obj.ttl = 24 h; > ? } elseif (req.url == "/cgi-bin/date.cgi") { > ? ? ? set obj.ttl = 1m; > ? } > } > > I get date.cgi correctly, but esi include seems to be executed just once (no > change on date info), any hint? You are running esi when the page is fetched and then caching the result. Run esi in vcl_deliver to make the include dynamic. Laurence From ccastro at altavoz.net Wed Oct 6 18:16:22 2010 From: ccastro at altavoz.net (Claudio Castro) Date: Wed, 06 Oct 2010 12:16:22 -0400 Subject: New on ESI, page example doesn't work In-Reply-To: References: <4CABB39A.2090301@altavoz.net> Message-ID: <4CACA0D6.9040601@altavoz.net> I try it but, its seems like ESI doesn't work on vcl_deliver because it only returns the html. Thx Laurence Rowe wrote: > On 6 October 2010 00:24, Claudio Castro wrote: > >> vcl: >> sub vcl_fetch { >> >> if (req.url == "/test.html") { >> esi; /* Do ESI processing */ >> set obj.ttl = 24 h; >> } elseif (req.url == "/cgi-bin/date.cgi") { >> set obj.ttl = 1m; >> } >> } >> >> I get date.cgi correctly, but esi include seems to be executed just once (no >> change on date info), any hint? >> > > You are running esi when the page is fetched and then caching the result. > > Run esi in vcl_deliver to make the include dynamic. > > Laurence > -- Claudio Castro N. Ingeniero Jefe Area de Plataforma AltaVoz S.A. http://www.altavoz.net Vi?a del Mar: 2 Poniente 355 of 53 +56 32 276 8060 Santiago: Pedro de Valdivia 555 of 315 +56 2 585 4264 From nsinghal at rim.com Wed Oct 6 21:19:04 2010 From: nsinghal at rim.com (Neeraj Singhal) Date: Wed, 6 Oct 2010 15:19:04 -0400 Subject: Proxying POST body through Varnish Message-ID: <4D876F2F2AE79948A49C84C5210A22AC1D966D06@XCH108CNC.rim.net> Hi Tollef Does this still stand (Varnish doesn't forward POST request body to backend server) ? Thanks Neeraj Singhal --------------------------------------------------------------------- This transmission (including any attachments) may contain confidential information, privileged material (including material protected by the solicitor-client or other applicable privileges), or constitute non-public information. Any use of this information by anyone other than the intended recipient is prohibited. If you have received this transmission in error, please immediately reply to the sender and delete this information from your system. Use, dissemination, distribution, or reproduction of this transmission by unintended recipients is not authorized and may be unlawful. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tfheen at varnish-software.com Thu Oct 7 08:06:44 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Thu, 07 Oct 2010 08:06:44 +0200 Subject: Proxying POST body through Varnish In-Reply-To: <4D876F2F2AE79948A49C84C5210A22AC1D966D06@XCH108CNC.rim.net> (Neeraj Singhal's message of "Wed, 6 Oct 2010 15:19:04 -0400") References: <4D876F2F2AE79948A49C84C5210A22AC1D966D06@XCH108CNC.rim.net> Message-ID: <87wrpu7czv.fsf@qurzaw.linpro.no> ]] Neeraj Singhal Hi, | Does this still stand (Varnish doesn't forward POST request body to backend server) ? That was not the original question; the original question was whether you could cache the result of POST requests or not, which you still can't do (and I don't think we're going to implement either; that completely breaks the semantics of POST). Varnish forwards the body of POST requests just fine and I believe has always had that support. Regards, -- Tollef Fog Heen Varnish Software t: +47 21 54 41 73 From varnish at mm.quex.org Thu Oct 7 09:20:46 2010 From: varnish at mm.quex.org (Michael Alger) Date: Thu, 7 Oct 2010 15:20:46 +0800 Subject: Proxying POST body through Varnish In-Reply-To: <87wrpu7czv.fsf@qurzaw.linpro.no> References: <4D876F2F2AE79948A49C84C5210A22AC1D966D06@XCH108CNC.rim.net> <87wrpu7czv.fsf@qurzaw.linpro.no> Message-ID: <20101007072046.GA29038@grum.quex.org> On Thu, Oct 07, 2010 at 08:06:44AM +0200, Tollef Fog Heen wrote: > ]] Neeraj Singhal > > > Does this still stand (Varnish doesn't forward POST request body > > to backend server) ? > > That was not the original question; the original question was whether > you could cache the result of POST requests or not, which you still > can't do (and I don't think we're going to implement either; that > completely breaks the semantics of POST). > > Varnish forwards the body of POST requests just fine and I believe has > always had that support. It won't forward the POST body if the request was restarted. I think I read that this was an architectural issue. Or is my information out of date? From phk at phk.freebsd.dk Thu Oct 7 13:45:06 2010 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 07 Oct 2010 11:45:06 +0000 Subject: Offline for a week... Message-ID: <83728.1286451906@critter.freebsd.dk> Guys, I'm going to be pretty offline for the next week. Tonight I catch the train to Karlsruhe for the EuroBSDcon conference. Mon-Wed next week is my lawsuit against Lenovo to refund my Vista license. Then next thursday I have a talk in the other end of Denmark. But once I wake up next Friday, I'll be back. Enjoy the silences :-) -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From ahooper at bmjgroup.com Fri Oct 8 12:58:31 2010 From: ahooper at bmjgroup.com (Alex Hooper) Date: Fri, 8 Oct 2010 11:58:31 +0100 Subject: Long wait for first byte Message-ID: <4CAEF957.60604@bmjgroup.com> Hi, We have a 128MB zip file behind varnish which, when it has expired from cache or when I add magic headers to force a PASS, produces a long delay between request and first byte. Is varnish waiting to retrieve the whole object before starting to send it to the client? If so, can I modify that behaviour? Many thanks, Alex. -- Alex Hooper Operations Team Leader, BMJ Group, BMA House, London WC1H 9JR Tel: +44 (0) 20 7383 6049 http://group.bmj.com/ _______________________________________________________________________ The BMJ Group is one of the world's most trusted providers of medical information for doctors, researchers, health care workers and patients group.bmj.com. This email and any attachments are confidential. If you have received this email in error, please delete it and kindly notify us. If the email contains personal views then the BMJ Group accepts no responsibility for these statements. The recipient should check this email and attachments for viruses because the BMJ Group accepts no liability for any damage caused by viruses. Emails sent or received by the BMJ Group may be monitored for size, traffic, distribution and content. BMJ Publishing Group Limited trading as BMJ Group. A private limited company, registered in England and Wales under registration number 03102371. Registered office: BMA House, Tavistock Square, London WC1H 9JR, UK. _______________________________________________________________________ From havardf at met.no Fri Oct 8 13:25:49 2010 From: havardf at met.no (=?utf-8?Q?H=C3=A5vard_Futs=C3=A6ter?=) Date: Fri, 8 Oct 2010 11:25:49 +0000 (UTC) Subject: response time in varnishncsa In-Reply-To: <717687862.6179.1280994948003.JavaMail.root@imap1b> Message-ID: <1970065075.273993.1286537149850.JavaMail.root@imap1b> Bump. Anyone who knows something about this? ----- Original Message ----- > Hi! Are there any plans for introducing response time in the > varnishncsa log format? I mean response time as in '%D' from the > apache2 log format. > > I am using varnish 2.0.4, so I suppose this might have been introduced > in later versions, although I haven't seen this from the changelogs. > > -- > ------ > Regards, > H?vard > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc -- ------ Regards, H?vard From A.Hongens at netmatch.nl Fri Oct 8 13:35:06 2010 From: A.Hongens at netmatch.nl (=?utf-8?B?QW5nZWxvIEjDtm5nZW5z?=) Date: Fri, 8 Oct 2010 11:35:06 +0000 Subject: response time in varnishncsa In-Reply-To: <1970065075.273993.1286537149850.JavaMail.root@imap1b> References: <717687862.6179.1280994948003.JavaMail.root@imap1b> <1970065075.273993.1286537149850.JavaMail.root@imap1b> Message-ID: <6A7ABA19243F1E4EADD8BB1563CDDCCB083920@TIL-EXCH-05.netmatch.local> Havent seen anything on the list about this. I would also really like to see the time-taken field in the varnishnca log. And what I would want even more, is a field indicating hit/miss! I might even convince my boss to pay for the hours taken. -- With kind regards, Angelo H?ngens Systems Administrator ------------------------------------------ NetMatch tourism internet software solutions Ringbaan Oost 2b 5013 CA Tilburg T: +31 (0)13 5811088 F: +31 (0)13 5821239 mailto:A.Hongens at netmatch.nl http://www.netmatch.nl ------------------------------------------ > -----Original Message----- > From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc- > bounces at varnish-cache.org] On Behalf Of H?vard Futs?ter > Sent: vrijdag 8 oktober 2010 13:26 > To: Varnish-misc > Subject: Re: response time in varnishncsa > > Bump. Anyone who knows something about this? > > ----- Original Message ----- > > Hi! Are there any plans for introducing response time in the > > varnishncsa log format? I mean response time as in '%D' from the > > apache2 log format. > > > > I am using varnish 2.0.4, so I suppose this might have been > introduced > > in later versions, although I haven't seen this from the changelogs. > > > > -- > > ------ > > Regards, > > H?vard > > > > _______________________________________________ > > varnish-misc mailing list > > varnish-misc at varnish-cache.org > > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > > -- > ------ > > Regards, > H?vard > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From samcrawford at gmail.com Fri Oct 8 13:50:14 2010 From: samcrawford at gmail.com (samcrawford at gmail.com) Date: Fri, 8 Oct 2010 11:50:14 +0000 Subject: Long wait for first byte In-Reply-To: <4CAEF957.60604@bmjgroup.com> References: <4CAEF957.60604@bmjgroup.com> Message-ID: <73573502-1286538613-cardhu_decombobulator_blackberry.rim.net-701463070-@bda178.bisx.produk.on.blackberry> Yes, varnish will capture the entire object before sending you any bytes. I believe changing the rule to "pipe" will alter this behaviour, but obviously this has other implications too (I.e. All future requests on that connection will be piped without being examined). Thanks Sam Sent from my BlackBerry? wireless device -----Original Message----- From: Alex Hooper Sender: varnish-misc-bounces at varnish-cache.org Date: Fri, 8 Oct 2010 11:58:31 To: Subject: Long wait for first byte Hi, We have a 128MB zip file behind varnish which, when it has expired from cache or when I add magic headers to force a PASS, produces a long delay between request and first byte. Is varnish waiting to retrieve the whole object before starting to send it to the client? If so, can I modify that behaviour? Many thanks, Alex. -- Alex Hooper Operations Team Leader, BMJ Group, BMA House, London WC1H 9JR Tel: +44 (0) 20 7383 6049 http://group.bmj.com/ _______________________________________________________________________ The BMJ Group is one of the world's most trusted providers of medical information for doctors, researchers, health care workers and patients group.bmj.com. This email and any attachments are confidential. If you have received this email in error, please delete it and kindly notify us. If the email contains personal views then the BMJ Group accepts no responsibility for these statements. The recipient should check this email and attachments for viruses because the BMJ Group accepts no liability for any damage caused by viruses. Emails sent or received by the BMJ Group may be monitored for size, traffic, distribution and content. BMJ Publishing Group Limited trading as BMJ Group. A private limited company, registered in England and Wales under registration number 03102371. Registered office: BMA House, Tavistock Square, London WC1H 9JR, UK. _______________________________________________________________________ _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From martin.boer at bizztravel.nl Fri Oct 8 13:28:50 2010 From: martin.boer at bizztravel.nl (Martin Boer) Date: Fri, 08 Oct 2010 13:28:50 +0200 Subject: Long wait for first byte In-Reply-To: <4CAEF957.60604@bmjgroup.com> References: <4CAEF957.60604@bmjgroup.com> Message-ID: <4CAF0072.8030300@bizztravel.nl> Hello Alex, Short version; it is and you can't. The developers are aware that this is an issue for some people but I can't remember if there is a workaround. Regards, Martin On 10/08/2010 12:58 PM, Alex Hooper wrote: > Hi, > > We have a 128MB zip file behind varnish which, when it has expired > from cache or when I add magic headers to force a PASS, produces a > long delay between request and first byte. Is varnish waiting to > retrieve the whole object before starting to send it to the client? If > so, can I modify that behaviour? > > Many thanks, > > Alex. From scaunter at topscms.com Fri Oct 8 18:59:34 2010 From: scaunter at topscms.com (Caunter, Stefan) Date: Fri, 8 Oct 2010 12:59:34 -0400 Subject: response time in varnishncsa In-Reply-To: <6A7ABA19243F1E4EADD8BB1563CDDCCB083920@TIL-EXCH-05.netmatch.local> References: <717687862.6179.1280994948003.JavaMail.root@imap1b><1970065075.273993.1286537149850.JavaMail.root@imap1b> <6A7ABA19243F1E4EADD8BB1563CDDCCB083920@TIL-EXCH-05.netmatch.local> Message-ID: <7F0AA702B8A85A4A967C4C8EBAD6902C5D1761@TMG-EVS02.torstar.net> Hit/miss is a vcl addition to vcl_deliver: sub vcl_deliver { C{ VRT_SetHdr(sp, HDR_RESP, "\014X-Cache-Svr:", myhostname, vrt_magic_string_end); }C /* mark hit/miss on the request */ if (obj.hits > 0) { set resp.http.X-Cache = "HIT"; set resp.http.X-Cache-Hits = obj.hits; } else { set resp.http.X-Cache = "MISS"; } } Stefan Caunter :: Senior Systems Administrator :: TOPS e: scaunter at topscms.com :: m: (416) 561-4871 www.thestar.com www.topscms.com -----Original Message----- From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Angelo H?ngens Sent: October-08-10 7:35 AM To: 'Varnish-misc' Subject: RE: response time in varnishncsa Havent seen anything on the list about this. I would also really like to see the time-taken field in the varnishnca log. And what I would want even more, is a field indicating hit/miss! I might even convince my boss to pay for the hours taken. -- With kind regards, Angelo H?ngens Systems Administrator ------------------------------------------ NetMatch tourism internet software solutions Ringbaan Oost 2b 5013 CA Tilburg T: +31 (0)13 5811088 F: +31 (0)13 5821239 mailto:A.Hongens at netmatch.nl http://www.netmatch.nl ------------------------------------------ > -----Original Message----- > From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc- > bounces at varnish-cache.org] On Behalf Of H?vard Futs?ter > Sent: vrijdag 8 oktober 2010 13:26 > To: Varnish-misc > Subject: Re: response time in varnishncsa > > Bump. Anyone who knows something about this? > > ----- Original Message ----- > > Hi! Are there any plans for introducing response time in the > > varnishncsa log format? I mean response time as in '%D' from the > > apache2 log format. > > > > I am using varnish 2.0.4, so I suppose this might have been > introduced > > in later versions, although I haven't seen this from the changelogs. > > > > -- > > ------ > > Regards, > > H?vard > > > > _______________________________________________ > > varnish-misc mailing list > > varnish-misc at varnish-cache.org > > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > > -- > ------ > > Regards, > H?vard > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From a.hongens at netmatch.nl Fri Oct 8 20:10:39 2010 From: a.hongens at netmatch.nl (=?UTF-8?B?QW5nZWxvIEjDtm5nZW5z?=) Date: Fri, 08 Oct 2010 20:10:39 +0200 Subject: response time in varnishncsa In-Reply-To: <7F0AA702B8A85A4A967C4C8EBAD6902C5D1761@TMG-EVS02.torstar.net> References: <717687862.6179.1280994948003.JavaMail.root@imap1b><1970065075.273993.1286537149850.JavaMail.root@imap1b> <6A7ABA19243F1E4EADD8BB1563CDDCCB083920@TIL-EXCH-05.netmatch.local> <7F0AA702B8A85A4A967C4C8EBAD6902C5D1761@TMG-EVS02.torstar.net> Message-ID: <4CAF5E9F.2070403@netmatch.nl> On 8-10-2010 18:59, Caunter, Stefan wrote: > Hit/miss is a vcl addition to vcl_deliver: > > sub vcl_deliver { > .. Stefan, Thanks for your response. I know I can output the hit/miss in the response headers (I already do), but I want the extra fields (time taken and 'action') in the varnishncsa output. I write all varnishncsa output of all nodes to a central logging server, and we want to do analysis on those log files later. Angelo. From havardf at met.no Mon Oct 11 09:34:02 2010 From: havardf at met.no (=?utf-8?Q?H=C3=A5vard_Futs=C3=A6ter?=) Date: Mon, 11 Oct 2010 07:34:02 +0000 (UTC) Subject: response time in varnishncsa In-Reply-To: <4CAF5E9F.2070403@netmatch.nl> Message-ID: <1614345672.279058.1286782442575.JavaMail.root@imap1b> ----- Original Message ----- > On 8-10-2010 18:59, Caunter, Stefan wrote: > > Hit/miss is a vcl addition to vcl_deliver: > > > > sub vcl_deliver { > > .. > > Stefan, > > Thanks for your response. I know I can output the hit/miss in the > response headers (I already do), but I want the extra fields (time > taken > and 'action') in the varnishncsa output. I write all varnishncsa > output > of all nodes to a central logging server, and we want to do analysis > on > those log files later. Same thing for us. Hit/miss in varnishncsa sounds like a good addition to varnishncsa for us too. Mind you, I also think the logging output from varnish is very nice as it is today! > Angelo. > > > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc Regards, Haavard -- ------ Mvh, H?vard Futs?ter Telefon: +47 22 96 32 69 From sebastiaan.jansen at kpn.com Mon Oct 11 10:18:00 2010 From: sebastiaan.jansen at kpn.com (sebastiaan.jansen at kpn.com) Date: Mon, 11 Oct 2010 10:18:00 +0200 Subject: Long wait for first byte In-Reply-To: <4CAF0072.8030300@bizztravel.nl> References: <4CAEF957.60604@bmjgroup.com> <4CAF0072.8030300@bizztravel.nl> Message-ID: I haven't tested it latterly, but doesn't chunked-encoding -support fix this, since a few months ago? Regards, Sebastiaan -----Oorspronkelijk bericht----- Van: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] Namens Martin Boer Verzonden: vrijdag 8 oktober 2010 13:29 Aan: varnish-misc at varnish-cache.org Onderwerp: Re: Long wait for first byte Hello Alex, Short version; it is and you can't. The developers are aware that this is an issue for some people but I can't remember if there is a workaround. Regards, Martin On 10/08/2010 12:58 PM, Alex Hooper wrote: > Hi, > > We have a 128MB zip file behind varnish which, when it has expired > from cache or when I add magic headers to force a PASS, produces a > long delay between request and first byte. Is varnish waiting to > retrieve the whole object before starting to send it to the client? If > so, can I modify that behaviour? > > Many thanks, > > Alex. _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From tfheen at varnish-software.com Mon Oct 11 15:38:45 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Mon, 11 Oct 2010 15:38:45 +0200 Subject: response time in varnishncsa In-Reply-To: <7F0AA702B8A85A4A967C4C8EBAD6902C5D1761@TMG-EVS02.torstar.net> (Stefan Caunter's message of "Fri, 8 Oct 2010 12:59:34 -0400") References: <717687862.6179.1280994948003.JavaMail.root@imap1b> <1970065075.273993.1286537149850.JavaMail.root@imap1b> <6A7ABA19243F1E4EADD8BB1563CDDCCB083920@TIL-EXCH-05.netmatch.local> <7F0AA702B8A85A4A967C4C8EBAD6902C5D1761@TMG-EVS02.torstar.net> Message-ID: <87d3rgj1cq.fsf@qurzaw.linpro.no> ]] "Caunter, Stefan" | sub vcl_deliver { | | C{ | VRT_SetHdr(sp, HDR_RESP, "\014X-Cache-Svr:", myhostname, | vrt_magic_string_end); | }C You should rather just do set resp.http.X-Cache-Svr = server.hostname; -- Tollef Fog Heen Varnish Software t: +47 21 54 41 73 From cp at ows.fr Tue Oct 12 14:46:54 2010 From: cp at ows.fr (=?ISO-8859-1?Q?C=E9dric_Perronnet?=) Date: Tue, 12 Oct 2010 14:46:54 +0200 Subject: Varnish and Magento Message-ID: <4CB458BE.4080104@ows.fr> Hello, I'm trying to make varnish and magento fits together. I used some online documentation to create a magento module that will setup a special cookie when something is added to the cart or wishlist. This part is working quite well. Problem is that I have different store inside this magento install. Some are url related which is solved via varnish hash, using the host. But some stores only use an url to swith between showing price with VAT or without per example. so I must find a way into varnish to make it handle a new cache bucket and to switch between those buckets whenever an user click on the link ( ?__store=de_ht per example ). I'm thinking about setting a cookie and had the cookie to the varnish hash to achieve that goal. Is there any better way to make that? If anybody has some information about combining varnish and magento, best practice etc , I would truly appreciate. Regards CP From mark at silverstatetech.com Wed Oct 13 03:40:07 2010 From: mark at silverstatetech.com (Mark Gavin) Date: Tue, 12 Oct 2010 18:40:07 -0700 Subject: Varnish and VBulletin Message-ID: Has anyone ever successfully implement Varnish in front of a VBulletin 4.x posting forum? If so, can you tell me what hurdles/problems you encountered along with helpful hints? If not, can you tell me why Varnish does not work in conjunction with VBulletin 4.x? -------------- next part -------------- An HTML attachment was scrubbed... URL: From cp at ows.fr Wed Oct 13 14:46:43 2010 From: cp at ows.fr (=?ISO-8859-1?Q?C=E9dric_Perronnet?=) Date: Wed, 13 Oct 2010 14:46:43 +0200 Subject: Varnish and Magento In-Reply-To: <4CB458BE.4080104@ows.fr> References: <4CB458BE.4080104@ows.fr> Message-ID: <4CB5AA33.6030800@ows.fr> Hi, I just answer to myself if that can help someone, I just figured out that magento is setting a cookie, then launch a 302 redirect to send back the user to the page with the right prices. I just added this cookie to the hash. > Hello, > > I'm trying to make varnish and magento fits together. > > I used some online documentation to create a magento module that will > setup a special cookie when something is added to the cart or wishlist. > This part is working quite well. > > Problem is that I have different store inside this magento install. > Some are url related which is solved via varnish hash, using the host. > > But some stores only use an url to swith between showing price with VAT > or without per example. > so I must find a way into varnish to make it handle a new cache bucket > and to switch between those buckets whenever an user click on the link ( > ?__store=de_ht per example ). > > I'm thinking about setting a cookie and had the cookie to the varnish > hash to achieve that goal. > Is there any better way to make that? If anybody has some information > about combining varnish and magento, best practice etc , I would truly > appreciate. > > Regards > CP > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > > From scaunter at topscms.com Wed Oct 13 15:37:30 2010 From: scaunter at topscms.com (Caunter, Stefan) Date: Wed, 13 Oct 2010 09:37:30 -0400 Subject: Varnish and VBulletin In-Reply-To: References: Message-ID: <7F0AA702B8A85A4A967C4C8EBAD6902C5D19FA@TMG-EVS02.torstar.net> Likely it is cookie handling. See headers of http://www.mangashare.com/ and http://www.hackint0sh.org which are large VBulletin forums on nginx and varnish. Stefan Caunter :: Senior Systems Administrator :: TOPS e: scaunter at topscms.com :: m: (416) 561-4871 www.thestar.com www.topscms.com From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Mark Gavin Sent: October-12-10 9:40 PM To: varnish-misc at varnish-cache.org Subject: Varnish and VBulletin Has anyone ever successfully implement Varnish in front of a VBulletin 4.x posting forum? If so, can you tell me what hurdles/problems you encountered along with helpful hints? If not, can you tell me why Varnish does not work in conjunction with VBulletin 4.x? -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexus at gmail.com Wed Oct 13 23:51:32 2010 From: alexus at gmail.com (alexus) Date: Wed, 13 Oct 2010 17:51:32 -0400 Subject: custom 503 page Message-ID: I need to create a custom 503 page from default.vcl, so that I can let my user know that I'm temporary unavailable instead of standard varnish's page. -- http://alexus.org/ From simon at darkmere.gen.nz Thu Oct 14 04:41:22 2010 From: simon at darkmere.gen.nz (Simon Lyall) Date: Thu, 14 Oct 2010 15:41:22 +1300 (NZDT) Subject: Centos/RHEL init script - reload/configtest Message-ID: I'm using the RHEL/Centos packages ( varnish-2.1.3 SVN 5049:5055 ) and I notice that the init.d script doesn't have a way to reload a new config without restarting the daemon (and losing the current cache). Looking around I came across the following version: http://cvs.pld-linux.org/cgi-bin/cvsweb.cgi/packages/varnish/varnish.init?rev=1.20;content-type=text%2Fplain which appears to impliment config checking and reloading of new configs. Looks like it would be easy enough to adapt. Just wondering if anyone else has already done this and if so could they share (and perhaps if the official packages could include)? Assuming this is actually a good idea of course. -- Simon Lyall | Very Busy | Web: http://www.darkmere.gen.nz/ "To stay awake all night adds a day to your life" - Stilgar | eMT. From russ at vshift.com Thu Oct 14 05:48:52 2010 From: russ at vshift.com (Ruslan Sivak) Date: Wed, 13 Oct 2010 23:48:52 -0400 Subject: Varying by device type Message-ID: <4CB67DA4.9010800@vshift.com> We would like to vary by device type. We have code that detects this when the first page on the website gets hit and then sets a device_group cookie. I would like to have varnish cache these pages based on the value of the cookie. I can't seem to get it to work. Here is my code: sub vcl_recv { if (req.http.Cookie) { set req.http.Cookie = ";" req.http.Cookie; set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";"); set req.http.Cookie = regsuball(req.http.Cookie, ";(device_group)=", "; \1="); set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", ""); set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", ""); if (req.http.Cookie == "") { remove req.http.Cookie; } } } sub vcl_fetch { if (beresp.http.set-cookie) { set beresp.http.set-cookie = ";" beresp.http.set-cookie; set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, "; +", ";"); set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, ";(device_group)=", "; \1="); set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, ";[^ ][^;]*", ""); set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, "^[; ]+|[; ]+$", ""); if (beresp.http.set-cookie == "") { remove beresp.http.set-cookie; } } set beresp.ttl = 5d; } sub vcl_hash { if (req.http.Cookie) { set req.hash += req.http.Cookie; } } It seems that this should work, however it's not caching. How can I make it cache even though a cookie is set? Thanks in advance, Russ From varnish at mm.quex.org Thu Oct 14 06:00:34 2010 From: varnish at mm.quex.org (Michael Alger) Date: Thu, 14 Oct 2010 12:00:34 +0800 Subject: Varying by device type In-Reply-To: <4CB67DA4.9010800@vshift.com> References: <4CB67DA4.9010800@vshift.com> Message-ID: <20101014040034.GA22715@grum.quex.org> On Wed, Oct 13, 2010 at 11:48:52PM -0400, Ruslan Sivak wrote: > We would like to vary by device type. We have code that detects this > when the first page on the website gets hit and then sets a > device_group cookie. > > I would like to have varnish cache these pages based on the value of > the cookie. I can't seem to get it to work. Here is my code: > > sub vcl_recv { > if (req.http.Cookie) { > set req.http.Cookie = ";" req.http.Cookie; > set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";"); > set req.http.Cookie = regsuball(req.http.Cookie, ";(device_group)=", > "; \1="); > set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", ""); > set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", ""); > > if (req.http.Cookie == "") { > remove req.http.Cookie; > } > } > } Is that your entire/only vcl_recv? The default code, which is appended to your own statements, includes: # if (req.http.Authorization || req.http.Cookie) { # /* Not cacheable by default */ # return (pass); # } which will obviously prevent any cache lookups being done if the client has sent a Cookie header. You'll want to do return (lookup); at some point to ensure the cache is checked, otherwise processing will go on to the default code and you'll get a 'pass'. The default vcl_fetch has similar code to return (pass) if the server has set a cookie, so you'll also want to override that. > It seems that this should work, however it's not caching. How can I > make it cache even though a cookie is set? From varnish at mm.quex.org Thu Oct 14 06:12:54 2010 From: varnish at mm.quex.org (Michael Alger) Date: Thu, 14 Oct 2010 12:12:54 +0800 Subject: custom 503 page In-Reply-To: References: Message-ID: <20101014041254.GB22715@grum.quex.org> On Wed, Oct 13, 2010 at 05:51:32PM -0400, alexus wrote: > I need to create a custom 503 page from default.vcl, so that I can > let my user know that I'm temporary unavailable instead of > standard varnish's page. If you're getting an error page from Varnish itself (with the request XID) then you'll want to modify or add your own vcl_error() and use the "synthetic" command to construct a response page. The default VCL shows how to do this, but essentially: sub vcl_error { set obj.http.Content-Type = "text/html; charset=utf-8"; synthetic {" ... "}; return (deliver); } You can also substitute a synthetic response in vcl_fetch() in response to certain error codes from the server, if desired. I personally keep restarting failed requests until I hit the limit at which point vcl_error() is called to deal with it, but if you only have one backend it may be cleaner to issue the nice error response from vcl_fetch. This is assuming it's the server's error message you're wanting to hide. From russ at vshift.com Thu Oct 14 06:26:31 2010 From: russ at vshift.com (Ruslan Sivak) Date: Thu, 14 Oct 2010 00:26:31 -0400 Subject: Varying by device type In-Reply-To: <20101014040034.GA22715@grum.quex.org> References: <4CB67DA4.9010800@vshift.com> <20101014040034.GA22715@grum.quex.org> Message-ID: <4CB68677.8070606@vshift.com> On 10/14/2010 12:00 AM, Michael Alger wrote: > On Wed, Oct 13, 2010 at 11:48:52PM -0400, Ruslan Sivak wrote: >> We would like to vary by device type. We have code that detects this >> when the first page on the website gets hit and then sets a >> device_group cookie. >> >> I would like to have varnish cache these pages based on the value of >> the cookie. I can't seem to get it to work. Here is my code: >> >> sub vcl_recv { >> if (req.http.Cookie) { >> set req.http.Cookie = ";" req.http.Cookie; >> set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";"); >> set req.http.Cookie = regsuball(req.http.Cookie, ";(device_group)=", >> "; \1="); >> set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", ""); >> set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", ""); >> >> if (req.http.Cookie == "") { >> remove req.http.Cookie; >> } >> } >> } > Is that your entire/only vcl_recv? The default code, which is appended > to your own statements, includes: > > # if (req.http.Authorization || req.http.Cookie) { > # /* Not cacheable by default */ > # return (pass); > # } > > which will obviously prevent any cache lookups being done if the client > has sent a Cookie header. You'll want to do return (lookup); at some > point to ensure the cache is checked, otherwise processing will go on > to the default code and you'll get a 'pass'. > > The default vcl_fetch has similar code to return (pass) if the server > has set a cookie, so you'll also want to override that. > So something like this then? sub vcl_recv { if (req.http.Cookie) { set req.http.Cookie = ";" req.http.Cookie; set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";"); set req.http.Cookie = regsuball(req.http.Cookie, ";(device_group)=", "; \1="); set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", ""); set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", ""); if (req.http.Cookie == "") { remove req.http.Cookie; } return(lookup); } } sub vcl_fetch { set beresp.ttl=5d; if (beresp.http.set-cookie) { set beresp.http.set-cookie = ";" beresp.http.set-cookie; set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, "; +", ";"); set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, ";(device_group)=", "; \1="); set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, ";[^ ][^;]*", ""); set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, "^[; ]+|[; ]+$", ""); if (beresp.http.set-cookie == "") { remove beresp.http.set-cookie; } return (deliver); } } sub vcl_hash { if (req.http.Cookie) { set req.hash += req.http.Cookie; } } It still doesn't seem to be working. RUss From russ at vshift.com Thu Oct 14 06:42:25 2010 From: russ at vshift.com (Ruslan Sivak) Date: Thu, 14 Oct 2010 00:42:25 -0400 Subject: Varying by device type In-Reply-To: <4CB68677.8070606@vshift.com> References: <4CB67DA4.9010800@vshift.com> <20101014040034.GA22715@grum.quex.org> <4CB68677.8070606@vshift.com> Message-ID: <4CB68A31.3070603@vshift.com> On 10/14/2010 12:26 AM, Ruslan Sivak wrote: > On 10/14/2010 12:00 AM, Michael Alger wrote: >> On Wed, Oct 13, 2010 at 11:48:52PM -0400, Ruslan Sivak wrote: >>> We would like to vary by device type. We have code that detects this >>> when the first page on the website gets hit and then sets a >>> device_group cookie. >>> >>> I would like to have varnish cache these pages based on the value of >>> the cookie. I can't seem to get it to work. Here is my code: >>> >>> sub vcl_recv { >>> if (req.http.Cookie) { >>> set req.http.Cookie = ";" req.http.Cookie; >>> set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";"); >>> set req.http.Cookie = regsuball(req.http.Cookie, >>> ";(device_group)=", >>> "; \1="); >>> set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", >>> ""); >>> set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; >>> ]+$", ""); >>> >>> if (req.http.Cookie == "") { >>> remove req.http.Cookie; >>> } >>> } >>> } >> Is that your entire/only vcl_recv? The default code, which is appended >> to your own statements, includes: >> >> # if (req.http.Authorization || req.http.Cookie) { >> # /* Not cacheable by default */ >> # return (pass); >> # } >> >> which will obviously prevent any cache lookups being done if the client >> has sent a Cookie header. You'll want to do return (lookup); at some >> point to ensure the cache is checked, otherwise processing will go on >> to the default code and you'll get a 'pass'. >> >> The default vcl_fetch has similar code to return (pass) if the server >> has set a cookie, so you'll also want to override that. >> > So something like this then? > > sub vcl_recv { > if (req.http.Cookie) { > set req.http.Cookie = ";" req.http.Cookie; > set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";"); > set req.http.Cookie = regsuball(req.http.Cookie, > ";(device_group)=", "; \1="); > set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", ""); > set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", > ""); > > if (req.http.Cookie == "") { > remove req.http.Cookie; > } > return(lookup); > } > } > > sub vcl_fetch { > set beresp.ttl=5d; > if (beresp.http.set-cookie) { > set beresp.http.set-cookie = ";" beresp.http.set-cookie; > set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, "; > +", ";"); > set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, > ";(device_group)=", "; \1="); > set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, > ";[^ ][^;]*", ""); > set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, > "^[; ]+|[; ]+$", ""); > > if (beresp.http.set-cookie == "") { > remove beresp.http.set-cookie; > } > return (deliver); > } > } > sub vcl_hash { if (req.http.Cookie) { set req.hash += req.http.Cookie; > } } > > It still doesn't seem to be working. > > RUss > Actually never mind, this does seem to be working. Russ From reuben at wikihow.com Thu Oct 14 06:54:05 2010 From: reuben at wikihow.com (Reuben S.) Date: Wed, 13 Oct 2010 21:54:05 -0700 Subject: varnishtop web wrapper Message-ID: Hi, I'm looking for a software package that would provice access to something like the output of varnishtop, but with output in HTML and a decent web interface. I could write my own wrapper for it, but it'd be great to know if someone else has already done this before duplicating effort. Thanks, Reuben -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Thu Oct 14 09:00:51 2010 From: perbu at varnish-software.com (Per Buer) Date: Thu, 14 Oct 2010 09:00:51 +0200 Subject: varnishtop web wrapper In-Reply-To: References: Message-ID: On Thu, Oct 14, 2010 at 6:54 AM, Reuben S. wrote: > Hi, > I'm looking for a software package that would provice access to something > like the output of varnishtop, but with output in HTML and a decent web > interface. ?I could write my own wrapper for it, but it'd be great to know > if someone else has already done this before duplicating effort. http://team-soundcloud.no.de/ -- Per Buer,? Varnish Software Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / skype: per.buer From conor at boards.ie Wed Oct 13 18:46:55 2010 From: conor at boards.ie (Conor McDermottroe) Date: Wed, 13 Oct 2010 17:46:55 +0100 Subject: Varnish and VBulletin In-Reply-To: References: Message-ID: <3A343290-5113-43CF-B928-E1AA84215D47@boards.ie> On 13 Oct 2010, at 02:40, Mark Gavin wrote: > Has anyone ever successfully implement Varnish in front of a VBulletin 4.x posting forum? If so, can you tell me what hurdles/problems you encountered along with helpful hints? If not, can you tell me why Varnish does not work in conjunction with VBulletin 4.x? > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From conor at boards.ie Wed Oct 13 18:56:46 2010 From: conor at boards.ie (Conor McDermottroe) Date: Wed, 13 Oct 2010 17:56:46 +0100 Subject: Varnish and VBulletin In-Reply-To: References: Message-ID: <1F6BD269-1487-46D0-A97F-A9FA2AFD88F4@boards.ie> We're currently using varnish in front of vBulletin 3.x and cookies are the main problem. The only real potential for caching vBulletin pages is for guests. Logged-in members are much more difficult to handle. Beware that vBulletin by default uses cookies for marking threads and forums as read. If you want to move that to your DB you'll need to budget more capacity on your DB. -C On 13 Oct 2010, at 02:40, Mark Gavin wrote: > Has anyone ever successfully implement Varnish in front of a VBulletin 4.x posting forum? If so, can you tell me what hurdles/problems you encountered along with helpful hints? If not, can you tell me why Varnish does not work in conjunction with VBulletin 4.x? > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From ask at develooper.com Thu Oct 14 09:56:29 2010 From: ask at develooper.com (=?iso-8859-1?Q?Ask_Bj=F8rn_Hansen?=) Date: Thu, 14 Oct 2010 00:56:29 -0700 Subject: Bad EL5 RPM Message-ID: <4EA06D00-499C-421A-AC28-01F1427E7835@develooper.com> Hi, The el5 rpm requires a /usr/bin/mkpasswd program. If that's in an RHEL RPM it should depend on the package name properly. If it's not, it shouldn't depend on it. - ask -- Ask Bj?rn Hansen, http://askask.com/ From russ at vshift.com Thu Oct 14 18:25:34 2010 From: russ at vshift.com (Ruslan Sivak) Date: Thu, 14 Oct 2010 12:25:34 -0400 Subject: Varnish Clustering Message-ID: <4CB72EFE.8050607@vshift.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Is there any sort of clustering support in varnish? The only thing I found was http://www.varnish-cache.org/trac/wiki/WebGui, Varnish Administration Console (only for customers with service agrement). Is that still true? Do you need a service agreement to use this? Also it states on another page that this is not production ready yet. Russ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iD8DBQFMty7+GUZBYlCheYcRAge6AKC/2GHn57Oz7q0tT5B0MTB3+F7FOQCeLtMA ao8TAUjYmWM1Y2YkoBxDzeA= =W0T6 -----END PGP SIGNATURE----- From ross at trademe.co.nz Thu Oct 14 23:06:09 2010 From: ross at trademe.co.nz (Ross Brown) Date: Fri, 15 Oct 2010 10:06:09 +1300 Subject: Centos/RHEL init script - reload/configtest In-Reply-To: References: Message-ID: <1FF67D7369ED1A45832180C7C1109BCA264BBF4933@tmmail0.trademe.local> It's easy enough to load a new config on the fly (without losing the contents of the cache) using the Varnish CLI. http://www.varnish-cache.org/trac/wiki/CLI I actually use a small shell script to call varnishadm against our Varnish servers. It's easier than logging on to each one individually and doing 'vcl.load test-conf.vcl' followed by 'vcl.use test-conf.vcl' Cheers Ross -----Original Message----- From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Simon Lyall Sent: Thursday, 14 October 2010 3:41 p.m. To: varnish-misc at varnish-cache.org Subject: Centos/RHEL init script - reload/configtest I'm using the RHEL/Centos packages ( varnish-2.1.3 SVN 5049:5055 ) and I notice that the init.d script doesn't have a way to reload a new config without restarting the daemon (and losing the current cache). Looking around I came across the following version: http://cvs.pld-linux.org/cgi-bin/cvsweb.cgi/packages/varnish/varnish.init?rev=1.20;content-type=text%2Fplain which appears to impliment config checking and reloading of new configs. Looks like it would be easy enough to adapt. Just wondering if anyone else has already done this and if so could they share (and perhaps if the official packages could include)? Assuming this is actually a good idea of course. -- Simon Lyall | Very Busy | Web: http://www.darkmere.gen.nz/ "To stay awake all night adds a day to your life" - Stilgar | eMT. _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From alan at prettyrobots.com Fri Oct 15 00:32:07 2010 From: alan at prettyrobots.com (Alan Gutierrez) Date: Thu, 14 Oct 2010 17:32:07 -0500 Subject: Purging Message-ID: <4CB784E7.4000805@prettyrobots.com> When you have an application that generates content, that you update infrequently, like pages in a CMS, how do you handle the case of the displaying the changes to the infrequent quickly? Do you set a short cache duration and then make it even shorter when authors are making changes to pages? Or do you wait until save and purge the cache entry for that particular page? I'm sure this is a FAQ and I've seen bits and pieces of answers. A pointer or keyword search suggestions would be great. -- Alan Gutierrez - alan at prettyrobots.com - http://twitter.com/bigeasy From mloftis at wgops.com Fri Oct 15 00:40:52 2010 From: mloftis at wgops.com (Michael Loftis) Date: Thu, 14 Oct 2010 16:40:52 -0600 Subject: Purging In-Reply-To: <4CB784E7.4000805@prettyrobots.com> References: <4CB784E7.4000805@prettyrobots.com> Message-ID: --On Thursday, October 14, 2010 5:32 PM -0500 Alan Gutierrez wrote: > When you have an application that generates content, that you update > infrequently, like pages in a CMS, how do you handle the case of the > displaying the changes to the infrequent quickly? > > Do you set a short cache duration and then make it even shorter when > authors are making changes to pages? > > Or do you wait until save and purge the cache entry for that particular > page? > > I'm sure this is a FAQ and I've seen bits and pieces of answers. A > pointer or keyword search suggestions would be great. The answer is, "it depends" :) For most things I do I keep only about a 5 minute cache, so its usually not too critical to force it to expire. For the cases it is I either have the CMS do a PURGE, or allow PURGEs to happen from specific clients (either via a http PURGE command or SHIFT+RELOAD). That's just me though probably. From checker at d6.com Fri Oct 15 00:42:16 2010 From: checker at d6.com (Chris Hecker) Date: Thu, 14 Oct 2010 15:42:16 -0700 Subject: Purging In-Reply-To: <4CB784E7.4000805@prettyrobots.com> References: <4CB784E7.4000805@prettyrobots.com> Message-ID: <4CB78748.6090804@d6.com> Most plugins send purge messages to varnish/squid with the pages that have changed (or at least most of them). Mediawiki, the Wordpress plugin, etc. Chris On 2010/10/14 15:32, Alan Gutierrez wrote: > When you have an application that generates content, that you update > infrequently, like pages in a CMS, how do you handle the case of the > displaying the changes to the infrequent quickly? > > Do you set a short cache duration and then make it even shorter when > authors are making changes to pages? > > Or do you wait until save and purge the cache entry for that particular > page? > > I'm sure this is a FAQ and I've seen bits and pieces of answers. A > pointer or keyword search suggestions would be great. > > -- > Alan Gutierrez - alan at prettyrobots.com - http://twitter.com/bigeasy > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > From fluffy at highwire.stanford.edu Fri Oct 15 03:48:23 2010 From: fluffy at highwire.stanford.edu (Ryan Krebs) Date: Thu, 14 Oct 2010 18:48:23 -0700 Subject: ban lurker causing deadlock in varnish 2.1.3 Message-ID: I'll try to get a ticket filed for this tomorrow, but I also had some related questions, so here's the short version: We have three CentOS servers running varnish 2.1.3, fronting around 850 sites. About once a week, varnish will hang on one of those servers. We finally managed to get a backtrace, and it seems to be pointing the finger at the ban lurker. As far as I can tell, if the ban lurker happens to start processing an object at the same time that a request is looking up that object from the cache, the two can get stuck trying to lock ban_mtx and oh->mtx. The backtrace shows the ban lurker thread, which would already have ban_mtx locked at this point: #0 0x000000390a80d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x000000390a808e1a in _L_lock_1034 () from /lib64/libpthread.so.0 #2 0x000000390a808cdc in pthread_mutex_lock () from /lib64/ libpthread.so.0 #3 0x0000000000421a69 in Lck__Lock () #4 0x000000000041afea in HSH_FindBan () #5 0x0000000000410b43 in ban_lurker () #6 0x0000000000424429 in wrk_bgthread () #7 0x000000390a80673d in start_thread () from /lib64/libpthread.so.0 #8 0x000000390a0d3d1d in clone () from /lib64/libc.so.6 and a large number of other threads, which have locked their respective oh->mtxs and are trying to lock ban_mtx: #0 0x000000390a80d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x000000390a808e1a in _L_lock_1034 () from /lib64/libpthread.so.0 #2 0x000000390a808cdc in pthread_mutex_lock () from /lib64/ libpthread.so.0 #3 0x0000000000421a69 in Lck__Lock () #4 0x000000000040f926 in ban_check_object () #5 0x000000000041c42f in HSH_Lookup () #6 0x0000000000411810 in cnt_lookup () #7 0x0000000000413ce4 in CNT_Session () #8 0x0000000000424668 in wrk_do_cnt_sess () #9 0x000000000042396e in wrk_thread_real () #10 0x000000390a80673d in start_thread () from /lib64/libpthread.so.0 #11 0x000000390a0d3d1d in clone () from /lib64/libc.so.6 Now for the questions. First, aside from the above information and the full backtrace, is there anything else that would be helpful to include in the ticket? Getting a full core dump would be problematic, as we're using "-s malloc,45G", and even without that varnish has a nasty habit of dying if we even think about using gdb or strace on it. Since it might be relevant, ban_lurker_sleep is set to 0.0005. I'm sure this increases the odds of a deadlock occurring as compared to setting it to 0.1 or 0.01, but it also helps keep our ban list fairly short. Second, if we were to turn off the ban lurker, or even just slow it down, how large can we allow the ban list to get before we might see an impact on performance? Each machine has 8 quad-core 2GHz CPUs, so I assume the answer is "quite large", but one of the servers had almost 2400 bans added over an 8-hour period today, and it's nice to have the lurker keeping the active list short. Am I worrying for no good reason? Has anybody else had similar problems? Ryan From varnish at mm.quex.org Fri Oct 15 07:42:18 2010 From: varnish at mm.quex.org (Michael Alger) Date: Fri, 15 Oct 2010 13:42:18 +0800 Subject: Varnish Clustering In-Reply-To: <4CB72EFE.8050607@vshift.com> References: <4CB72EFE.8050607@vshift.com> Message-ID: <20101015054218.GA13260@grum.quex.org> On Thu, Oct 14, 2010 at 12:25:34PM -0400, Ruslan Sivak wrote: > Is there any sort of clustering support in varnish? The only thing I > found was http://www.varnish-cache.org/trac/wiki/WebGui, Varnish > Administration Console (only for customers with service agrement). Is > that still true? Do you need a service agreement to use this? Also > it states on another page that this is not production ready yet. I use Linux-HA with Varnish without any problems to implement an active/passive proxy. I store the Varnish configuration in a subversion repository, and have a simple preprocessor script to rewrite IP addresses and omit sections so I can use the same configuration file for UAT as well as our production and DR environments. Generally speaking, I make changes to the config in UAT, test them out, then check the changes into subversion and check them out on the production servers, then finally apply them to the actual running Varnish instance. I have Varnish running continually on both nodes in the cluster, and the IP addresses the sites are served from are assigned to a 10.x network with addresses bound to loopback. The router routes this 10.x network via an IP address which is managed by Linux-HA, i.e. whichever ever node has that IP address configured becomes the active proxy. Then I have a little script running continuously on each Varnish node which makes requests via the local proxy and sets the node's score appropriately. Whichever proxy is working the best wins, with a bias to prevent needless flip-flopping. You could also have the Varnish instance managed by Linux-HA, so it would only be running on the active cluster member. I like having it always running to reduce switchover delays, and so I can perform testing on an identical configuration which isn't receiving live traffic. From a.hongens at netmatch.nl Fri Oct 15 10:14:13 2010 From: a.hongens at netmatch.nl (=?ISO-8859-1?Q?Angelo_H=F6ngens?=) Date: Fri, 15 Oct 2010 10:14:13 +0200 Subject: Varnish Clustering In-Reply-To: <20101015054218.GA13260@grum.quex.org> References: <4CB72EFE.8050607@vshift.com> <20101015054218.GA13260@grum.quex.org> Message-ID: <4CB80D55.2050305@netmatch.nl> On Thu, Oct 14, 2010 at 12:25:34PM -0400, Ruslan Sivak wrote: > Is there any sort of clustering support in varnish? The only thing I > found was http://www.varnish-cache.org/trac/wiki/WebGui, Varnish > Administration Console (only for customers with service agrement). Is > that still true? Do you need a service agreement to use this? Also > it states on another page that this is not production ready yet. We use CARP on FreeBSD for active/active clustering across 4 nodes. We use DNS for round-robin load balancing and Carp for keeping the 4 virtual ip's up. The varnish instances don't need to know about each other or the fact that they're part of a cluster. About the configuration: on each node we have a script we run manually to refresh it's configs and restart services. -- With kind regards, Angelo H?ngens systems administrator MCSE on Windows 2003 MCSE on Windows 2000 MS Small Business Specialist ------------------------------------------ NetMatch tourism internet software solutions Ringbaan Oost 2b 5013 CA Tilburg +31 (0)13 5811088 +31 (0)13 5821239 A.Hongens at netmatch.nl www.netmatch.nl ------------------------------------------ From anand at rediff-inc.com Fri Oct 15 09:19:16 2010 From: anand at rediff-inc.com (anand at rediff-inc.com) Date: Fri, 15 Oct 2010 12:49:16 +0530 Subject: Centos/RHEL init script - reload/configtest In-Reply-To: <1FF67D7369ED1A45832180C7C1109BCA264BBF4933@tmmail0.trademe.local> References: <1FF67D7369ED1A45832180C7C1109BCA264BBF4933@tmmail0.trademe.local> Message-ID: Hi Guys, A sample script I use to reload varnish configs; maybe useful to you.... #!/bin/bash # Reload a varnish config FILE="/etc/varnish/varnish.vcl" # Hostname and management port # (defined in /etc/default/varnish or on startup) HOSTPORT="IPADDRESS:PORT" NOW=`date +%d%h%Y` error() { echo 1>&2 "Failed to reload $FILE." exit 1 } varnishadm -T $HOSTPORT vcl.load reload$NOW $FILE || error varnishadm -T $HOSTPORT vcl.use reload$NOW || error echo Current configs: varnishadm -T $HOSTPORT vcl.list Regards, Anand -----Original Message----- From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Ross Brown Sent: Friday, October 15, 2010 2:36 AM To: Simon Lyall; varnish-misc at varnish-cache.org Subject: RE: Centos/RHEL init script - reload/configtest It's easy enough to load a new config on the fly (without losing the contents of the cache) using the Varnish CLI. http://www.varnish-cache.org/trac/wiki/CLI I actually use a small shell script to call varnishadm against our Varnish servers. It's easier than logging on to each one individually and doing 'vcl.load test-conf.vcl' followed by 'vcl.use test-conf.vcl' Cheers Ross -----Original Message----- From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Simon Lyall Sent: Thursday, 14 October 2010 3:41 p.m. To: varnish-misc at varnish-cache.org Subject: Centos/RHEL init script - reload/configtest I'm using the RHEL/Centos packages ( varnish-2.1.3 SVN 5049:5055 ) and I notice that the init.d script doesn't have a way to reload a new config without restarting the daemon (and losing the current cache). Looking around I came across the following version: http://cvs.pld-linux.org/cgi-bin/cvsweb.cgi/packages/varnish/varnish.init?re v=1.20;content-type=text%2Fplain which appears to impliment config checking and reloading of new configs. Looks like it would be easy enough to adapt. Just wondering if anyone else has already done this and if so could they share (and perhaps if the official packages could include)? Assuming this is actually a good idea of course. -- Simon Lyall | Very Busy | Web: http://www.darkmere.gen.nz/ "To stay awake all night adds a day to your life" - Stilgar | eMT. _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org http://lists.varnish-cache.org/mailman/listinfo/varnish-misc _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From perbu at varnish-software.com Fri Oct 15 10:37:41 2010 From: perbu at varnish-software.com (Per Buer) Date: Fri, 15 Oct 2010 10:37:41 +0200 Subject: Varnish Clustering In-Reply-To: <4CB72EFE.8050607@vshift.com> References: <4CB72EFE.8050607@vshift.com> Message-ID: On Thu, Oct 14, 2010 at 6:25 PM, Ruslan Sivak wrote: > Is there any sort of clustering support in varnish? Not in Varnish itself, no. You need some sort of clustering framework around Varnish. Also, Varnish can act like a load balancer for it's backend web servers, creating a logical cluster of servers behind it. >?The only thing I found was http://www.varnish-cache.org/trac/wiki/WebGui, Varnish > Administration Console (only for customers with service agrement). ?Is > that still true? ?Do you need a service agreement to use this? The Varnish Administration Console (the VAC - we can't create anything without giving it a three letter acronym :-) is focused mostly on administration and monitoring of Varnish, not so much doing the actual clustering. Features that will enhance the clustering capabilities will be built into the VAC next year. And yes. We have decided to keep that application proprietary - so you would need a service agreement. Get in touch with me if you are interested in giving it a try. > Also it states on another page that this is not production ready yet. It is now. -- Per Buer,? Varnish Software Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / skype: per.buer From russ at vshift.com Sat Oct 16 00:50:03 2010 From: russ at vshift.com (Ruslan Sivak) Date: Fri, 15 Oct 2010 18:50:03 -0400 Subject: 32 bit RPMs Message-ID: <4CB8DA9B.2010707@vshift.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Are there any recent 32 bit rpms available? The 32bit builds seem to all have bad signatures and are pretty outdated. The latest version i see at http://repo.varnish-cache.org/redhat/el5/el5/i386/ is varnish-2.0.4-1.el5.i386.rpm. Meanwhile the latest x64 is http://repo.varnish-cache.org/redhat/el5/el5/x86_64/varnish-2.1.3-1.x86_64.rpm. I'm not sure if the rest of the stack that we plan to run on that server would work well under x64. Do I have to build from source to get a more recent version? Russ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iD8DBQFMuNqbGUZBYlCheYcRAuNJAKDQY8YGuriZO0hBs1fCmX2OqoJI/ACgyED2 sfxJ98ERjn1UxQc9UPg9jec= =0yOO -----END PGP SIGNATURE----- From omerfsen at gmail.com Sat Oct 16 09:20:56 2010 From: omerfsen at gmail.com (Omer Faruk SEN) Date: Sat, 16 Oct 2010 10:20:56 +0300 Subject: Varnish as a forward proxy? Message-ID: Hi , First of all I want to state i am pretty new to Varnish. I have read some documentation and did my research. Anyway I want to ask if Varnish can be used as a forwarding proxy. As far as I see we must always define a backend server which makes me think that Varnish is a reverse proxy? Can someone clarify this with an examle VCL if you can so it may be more clear if varnish can be used as a forwarding proxy? Regards. From phk at phk.freebsd.dk Sat Oct 16 09:25:39 2010 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sat, 16 Oct 2010 07:25:39 +0000 Subject: Varnish as a forward proxy? In-Reply-To: Your message of "Sat, 16 Oct 2010 10:20:56 +0300." Message-ID: <41453.1287213939@critter.freebsd.dk> In message , Omer Faruk SEN writes: >Anyway I want to ask if Varnish can be used as a forwarding proxy. Sorry, not possible -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From mliebherr99 at googlemail.com Mon Oct 18 17:57:25 2010 From: mliebherr99 at googlemail.com (ml ml) Date: Mon, 18 Oct 2010 17:57:25 +0200 Subject: How to compile nagios plugin Message-ID: Hello List, i have installed varnish from tarball with: --prefix=/usr/local/varnish. Now i would like to compile the nagios plugin from svn: cd /tmp/trunk/varnish-tools/nagios/ ; bash autogen.sh ; ./configure and i get: checking for VARNISHAPI... configure: error: Package requirements (varnishapi) were not met: No package 'varnishapi' found so i ran: PKG_CONFIG_PATH="/usr/local/varnish/lib/pkgconfig/" ./configure but running make brings up this error: /tmp/trunk/varnish-tools/nagios# make make all-am make[1]: Entering directory `/tmp/trunk/varnish-tools/nagios' gcc -DHAVE_CONFIG_H -I. -include config.h -I/usr/local/varnish-2.1.3/include/varnish -g -O2 -MT check_varnish-check_varnish.o -MD -MP -MF .deps/check_varnish-check_varnish.Tpo -c -o check_varnish-check_varnish.o `test -f 'check_varnish.c' || echo './'`check_varnish.c check_varnish.c:45:17: error: vsc.h: No such file or directory check_varnish.c:168: warning: ?struct vsc_main? declared inside parameter list check_varnish.c:168: warning: its scope is only this definition or declaration, which is probably not what you want check_varnish.c: In function ?check_stats?: check_varnish.c:177: error: dereferencing pointer to incomplete type check_varnish.c:177: error: dereferencing pointer to incomplete type check_varnish.c:179: error: dereferencing pointer to incomplete type check_varnish.c:183: error: dereferencing pointer to incomplete type check_varnish.c:183: error: dereferencing pointer to incomplete type check_varnish.c:185: error: dereferencing pointer to incomplete type check_varnish.c:193:24: error: vsc_fields.h: No such file or directory check_varnish.c: In function ?main?: check_varnish.c:252: warning: assignment makes pointer from integer without a cast check_varnish.c:285: warning: assignment makes pointer from integer without a cast check_varnish.c:301: warning: passing argument 1 of ?check_stats? from incompatible pointer type make[1]: *** [check_varnish-check_varnish.o] Error 1 make[1]: Leaving directory `/tmp/trunk/varnish-tools/nagios' make: *** [all] Error 2 Anyone any idea? Cheers, Mario From chaokovsky.lee at gmail.com Tue Oct 19 09:22:22 2010 From: chaokovsky.lee at gmail.com (Chaos Lee) Date: Tue, 19 Oct 2010 15:22:22 +0800 Subject: Varnish used large memory and keep growing Message-ID: version: 2.0.6 cmd to start varnishd: /usr/sbin/varnishd -P /var/run/varnish.pid -a :80 -f /etc/varnish/default.vcl -T 127.0.0.1:6082 -t 120 -w 500,10000,120 -u varnish -g varnish -s file,/var/lib/mysql/varnish_cache/varnish.cache,100G -h classic,500009 -p sess_timeout 5 -p thread_pools 4 -p thread_pool_add_delay 2 -p listen_depth 4096 -p session_linger 50 -p lru_interval 20 -p obj_workspace 4096 My system has 4G memory and 4G swap, varnish eat all memory and 500m swap space when running about 6 hrs. I think if there is memory leak on 2.0.6. Dose someone has same problem? -------------- next part -------------- An HTML attachment was scrubbed... URL: From tfheen at varnish-software.com Tue Oct 19 12:34:14 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Tue, 19 Oct 2010 12:34:14 +0200 Subject: How to compile nagios plugin In-Reply-To: (ml ml's message of "Mon, 18 Oct 2010 17:57:25 +0200") References: Message-ID: <87ocaqsc7t.fsf@qurzaw.linpro.no> ]] ml ml | Hello List, | | i have installed varnish from tarball with: --prefix=/usr/local/varnish. | | Now i would like to compile the nagios plugin from svn: | cd /tmp/trunk/varnish-tools/nagios/ ; bash autogen.sh ; ./configure You need the nagios plugin from the 2.1 branch if you're going to use it with 2.1 -- Tollef Fog Heen Varnish Software t: +47 21 98 62 64 From ahooper at bmjgroup.com Tue Oct 19 12:47:39 2010 From: ahooper at bmjgroup.com (Alex Hooper) Date: Tue, 19 Oct 2010 11:47:39 +0100 Subject: Long wait for first byte In-Reply-To: References: <4CAEF957.60604@bmjgroup.com> <4CAF0072.8030300@bizztravel.nl> Message-ID: <4CBD774B.2070305@bmjgroup.com> sebastiaan.jansen at kpn.com uttered: > I haven't tested it latterly, but doesn't chunked-encoding -support fix this, since a few months ago? > Hi Sebastiaan, Thanks for that, I'll take more of a look once we've upgraded to 2.1.3 in a couple of weeks. And thanks Martin and Sam for your quick responses. This time, I'm able to work around with a long expiry time and manual purging and priming of the cache when required. Cheers, Alex. > Regards, > Sebastiaan > > -----Oorspronkelijk bericht----- > Van: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] Namens Martin Boer > Verzonden: vrijdag 8 oktober 2010 13:29 > Aan: varnish-misc at varnish-cache.org > Onderwerp: Re: Long wait for first byte > > Hello Alex, > > Short version; it is and you can't. > The developers are aware that this is an issue for some people but I > can't remember if there is a workaround. > > Regards, > Martin > > > On 10/08/2010 12:58 PM, Alex Hooper wrote: >> Hi, >> >> We have a 128MB zip file behind varnish which, when it has expired >> from cache or when I add magic headers to force a PASS, produces a >> long delay between request and first byte. Is varnish waiting to >> retrieve the whole object before starting to send it to the client? If >> so, can I modify that behaviour? >> >> Many thanks, >> >> Alex. > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > > ______________________________________________________________________ > This email has been scanned by the MessageLabs Email Security System. > For more information please visit http://www.messagelabs.com/email > ______________________________________________________________________ -- Alex Hooper Operations Team Leader, BMJ Group, BMA House, London WC1H 9JR Tel: +44 (0) 20 7383 6049 http://group.bmj.com/ _______________________________________________________________________ The BMJ Group is one of the world's most trusted providers of medical information for doctors, researchers, health care workers and patients group.bmj.com. This email and any attachments are confidential. If you have received this email in error, please delete it and kindly notify us. If the email contains personal views then the BMJ Group accepts no responsibility for these statements. The recipient should check this email and attachments for viruses because the BMJ Group accepts no liability for any damage caused by viruses. Emails sent or received by the BMJ Group may be monitored for size, traffic, distribution and content. BMJ Publishing Group Limited trading as BMJ Group. A private limited company, registered in England and Wales under registration number 03102371. Registered office: BMA House, Tavistock Square, London WC1H 9JR, UK. _______________________________________________________________________ From ahooper at bmjgroup.com Tue Oct 19 13:22:48 2010 From: ahooper at bmjgroup.com (Alex Hooper) Date: Tue, 19 Oct 2010 12:22:48 +0100 Subject: problems with purge Message-ID: <4CBD7F88.2060400@bmjgroup.com> Hello, Is it a known issue that causes varnishadm -T localhost:6082 'purge req.http.url ~ ^/repository/bp-bundles/.*' not to work while varnishadm -T localhost:6082 'purge.url ^/repository/bp-bundles/.*' works fine? This is Varnish 2.0.4. Perhaps I've been staring at it too long and am missing the obvious... Cheers, Alex. -- Alex Hooper Operations Team Leader, BMJ Group, BMA House, London WC1H 9JR Tel: +44 (0) 20 7383 6049 http://group.bmj.com/ _______________________________________________________________________ The BMJ Group is one of the world's most trusted providers of medical information for doctors, researchers, health care workers and patients group.bmj.com. This email and any attachments are confidential. If you have received this email in error, please delete it and kindly notify us. If the email contains personal views then the BMJ Group accepts no responsibility for these statements. The recipient should check this email and attachments for viruses because the BMJ Group accepts no liability for any damage caused by viruses. Emails sent or received by the BMJ Group may be monitored for size, traffic, distribution and content. BMJ Publishing Group Limited trading as BMJ Group. A private limited company, registered in England and Wales under registration number 03102371. Registered office: BMA House, Tavistock Square, London WC1H 9JR, UK. _______________________________________________________________________ From cosimo at streppone.it Tue Oct 19 13:30:17 2010 From: cosimo at streppone.it (Cosimo Streppone) Date: Tue, 19 Oct 2010 13:30:17 +0200 Subject: Varnish 2.0.4 health check and startup question Message-ID: I'm using Varnish 2.0.4 (but IIUC, 2.1 and trunk seem to behave the same way) with backend health checks: backend b1 { .host = "localhost"; .port = "8080"; # Disabled to avoid the 15s startup time #.probe = { # .url = "/ping.html"; # .interval = 5s; # .timeout = 1s; # .window = 5; # .threshold = 3; #} } Every time I restart varnish I can see through debug.health that it's waiting for the b1 backend to become healthy. In my case that takes 15s (.interval * .threshold), so restarting varnish with perfectly fine backend(s) causes 15s of black out. Is there a reason for the initial backend state to be "sick" instead of "healthy"? Thanks, -- Cosimo From tfheen at varnish-software.com Tue Oct 19 15:32:18 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Tue, 19 Oct 2010 15:32:18 +0200 Subject: problems with purge In-Reply-To: <4CBD7F88.2060400@bmjgroup.com> (Alex Hooper's message of "Tue, 19 Oct 2010 12:22:48 +0100") References: <4CBD7F88.2060400@bmjgroup.com> Message-ID: <87eibms3z1.fsf@qurzaw.linpro.no> ]] Alex Hooper | Hello, | | Is it a known issue that causes | | varnishadm -T localhost:6082 'purge req.http.url ~ ^/repository/bp-bundles/.*' | not to work while req.http.url is the HTTP header field called ?url? in the request. I'm reasonably sure you don't have that. | varnishadm -T localhost:6082 'purge.url ^/repository/bp-bundles/.*' I assume there's a missing ~ there? -- Tollef Fog Heen Varnish Software t: +47 21 98 62 64 From ahooper at bmjgroup.com Tue Oct 19 16:00:05 2010 From: ahooper at bmjgroup.com (Alex Hooper) Date: Tue, 19 Oct 2010 15:00:05 +0100 Subject: problems with purge In-Reply-To: <87eibms3z1.fsf@qurzaw.linpro.no> References: <4CBD7F88.2060400@bmjgroup.com> <87eibms3z1.fsf@qurzaw.linpro.no> Message-ID: <4CBDA465.9070200@bmjgroup.com> Tollef Fog Heen uttered: > ]] Alex Hooper > > | Hello, > | > | Is it a known issue that causes > | > | varnishadm -T localhost:6082 'purge req.http.url ~ ^/repository/bp-bundles/.*' > | not to work while > > req.http.url is the HTTP header field called ?url? in the request. I'm > reasonably sure you don't have that. Ah, yes, I *am* a moron. Not least for not reading the whole wiki page which contains the same syntax typo (http://www.varnish-cache.org/trac/wiki/Purging under Syntax of bans). Reading it to the end might have made me suspect the initail typo. It was the composite I was originally going for and now, thanks to your enlightenment, I have it: varnishadm -T localhost:6082 'purge req.http.host ~ group.bmj.com && req.url ~ ^/repository/bp-bundles/.*' I shall adjust my documentation accordingly. I did just register at varnish-cache.org to see if I could update the Wiki but either I don't have the privs or I don't have the Brane(TM). > > | varnishadm -T localhost:6082 'purge.url ^/repository/bp-bundles/.*' > > I assume there's a missing ~ there? > No. It works exactly as quoted. Thanks again, Alex. -- Alex Hooper Operations Team Leader, BMJ Group, BMA House, London WC1H 9JR Tel: +44 (0) 20 7383 6049 http://group.bmj.com/ _______________________________________________________________________ The BMJ Group is one of the world's most trusted providers of medical information for doctors, researchers, health care workers and patients group.bmj.com. This email and any attachments are confidential. If you have received this email in error, please delete it and kindly notify us. If the email contains personal views then the BMJ Group accepts no responsibility for these statements. The recipient should check this email and attachments for viruses because the BMJ Group accepts no liability for any damage caused by viruses. Emails sent or received by the BMJ Group may be monitored for size, traffic, distribution and content. BMJ Publishing Group Limited trading as BMJ Group. A private limited company, registered in England and Wales under registration number 03102371. Registered office: BMA House, Tavistock Square, London WC1H 9JR, UK. _______________________________________________________________________ From tfheen at varnish-software.com Tue Oct 19 17:06:12 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Tue, 19 Oct 2010 17:06:12 +0200 Subject: problems with purge In-Reply-To: <4CBDA465.9070200@bmjgroup.com> (Alex Hooper's message of "Tue, 19 Oct 2010 15:00:05 +0100") References: <4CBD7F88.2060400@bmjgroup.com> <87eibms3z1.fsf@qurzaw.linpro.no> <4CBDA465.9070200@bmjgroup.com> Message-ID: <87aamarzmj.fsf@qurzaw.linpro.no> ]] Alex Hooper | Tollef Fog Heen uttered: | > ]] Alex Hooper | > | > | Hello, | > | | Is it a known issue that causes | > | | varnishadm -T localhost:6082 'purge req.http.url ~ | > ^/repository/bp-bundles/.*' | > | not to work while | > | > req.http.url is the HTTP header field called ?url? in the request. I'm | > reasonably sure you don't have that. | | Ah, yes, I *am* a moron. Not least for not reading the whole wiki page | which contains the same syntax typo | (http://www.varnish-cache.org/trac/wiki/Purging under Syntax of | bans). Reading it to the end might have made me suspect the initail | typo. Indeed. Fixed now. | I shall adjust my documentation accordingly. I did just register at | varnish-cache.org to see if I could update the Wiki but either I don't | have the privs or I don't have the Brane(TM). If you give me your wiki user name, I can give you the wiki edit bit. Best regards, -- Tollef Fog Heen Varnish Software t: +47 21 98 62 64 From kbrownfield at google.com Tue Oct 19 22:41:02 2010 From: kbrownfield at google.com (Ken Brownfield) Date: Tue, 19 Oct 2010 13:41:02 -0700 Subject: Varnish 2.0.4 health check and startup question In-Reply-To: References: Message-ID: You should check the archives and the Varnish site. A feature to enable "priming" the health checks was introduced in r4185 and released with 2.0.5 I believe. I would suggest you use the latest 2.1 release and set the .initial probe variable equal to your threshold. -- Ken On Tue, Oct 19, 2010 at 4:30 AM, Cosimo Streppone wrote: > I'm using Varnish 2.0.4 (but IIUC, 2.1 and trunk > seem to behave the same way) with backend health checks: > > backend b1 { > .host = "localhost"; > .port = "8080"; > > # Disabled to avoid the 15s startup time > #.probe = { > # .url = "/ping.html"; > # .interval = 5s; > # .timeout = 1s; > # .window = 5; > # .threshold = 3; > #} > } > > Every time I restart varnish I can see > through debug.health that it's waiting for the > b1 backend to become healthy. > > In my case that takes 15s (.interval * .threshold), > so restarting varnish with perfectly fine backend(s) > causes 15s of black out. > > Is there a reason for the initial backend state > to be "sick" instead of "healthy"? > > Thanks, > > -- > Cosimo > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pom at dmsp.de Thu Oct 21 17:57:38 2010 From: pom at dmsp.de (Stefan Pommerening) Date: Thu, 21 Oct 2010 17:57:38 +0200 Subject: Snapshotting current cache content (host/url tupels) into list? Message-ID: <4CC062F2.8010405@dmsp.de> Hi all, I am trying to figure out if there is some sort of "snapshot" to display the momentary content of the varnish cache, more detailled the current list of Host/URL tupels in memory. I've understood that using varnishstat I can display the fields n_object and n_objecthead showing me the number of Host/URL tupels and the number of variants at a given time. What I am still missing is an option to (single-time) 'dump' the Host/URL tupels (n_objecthead) of this current second from the cache into a list. I know that it's changing permanently but I'd like to have a 'snapshot' option. I already spent a lot of time searching for this info, but it seems that it is not available. Does anyone have a hint for me? Thanks, Stefan. From cosimo at streppone.it Thu Oct 21 22:58:23 2010 From: cosimo at streppone.it (Cosimo Streppone) Date: Thu, 21 Oct 2010 22:58:23 +0200 Subject: Varnish 2.0.4 health check and startup question In-Reply-To: References: Message-ID: In data 19 ottobre 2010 alle ore 18:27:33, Per Buer ha scritto: > On Tue, Oct 19, 2010 at 1:30 PM, Cosimo Streppone > wrote: >> Is there a reason for the initial backend state >> to be "sick" instead of "healthy"? > > A defensive default. [...] > However, you can set it manually. Thanks! What I was looking for indeed. -- Cosimo From AGresens at Scholastic.com Thu Oct 21 23:14:53 2010 From: AGresens at Scholastic.com (Gresens, August) Date: Thu, 21 Oct 2010 17:14:53 -0400 Subject: infrastructure setup using ESI Message-ID: <8D333BF67F814C4D9803C99961BE5E70014EDA6E@corpex07.corp.scholasticinc.local> Hello Varnish is a key part of a new set of apps we are building - the ESI feature in particular is something we want to take advantage of extensively. Where does Varnish typically reside in the network architecture? I would imagine in a scenario in which we would be using ESI, we would want Varnish in front of the application load balancers. In this way, Varnish would be the primary page assembler. We would likely run an HA setup for fault tolerance. We were also considering if Varnish could be used much farther downstream - in between the load balancer and the application. We thought is might provide for more fault tolerance, but it seems the disadvantage would be that there would be a separate cached copy in each redundant varnish instance, and with ESI secondary requests would be made back out to the same web server potentially, setting up a feed back loop that could cause problems. Thanks for your responses in advance. August August Gresens Director of Technology, eScholastic Scholastic Inc. 557 Broadway, NY, NY 10012 Email :: agresens at scholastic.com Phone :: 917 363 3662

-------------- next part -------------- An HTML attachment was scrubbed... URL: From david.birdsong at gmail.com Thu Oct 21 23:41:58 2010 From: david.birdsong at gmail.com (David Birdsong) Date: Thu, 21 Oct 2010 14:41:58 -0700 Subject: infrastructure setup using ESI In-Reply-To: <8D333BF67F814C4D9803C99961BE5E70014EDA6E@corpex07.corp.scholasticinc.local> References: <8D333BF67F814C4D9803C99961BE5E70014EDA6E@corpex07.corp.scholasticinc.local> Message-ID: Most load balancers have a hashing lb method. Hash on something int the request and only send that traffic to a specific *alive* varnish instance. On Oct 21, 2010 2:16 PM, "Gresens, August" wrote: Hello Varnish is a key part of a new set of apps we are building - the ESI feature in particular is something we want to take advantage of extensively. Where does Varnish typically reside in the network architecture? I would imagine in a scenario in which we would be using ESI, we would want Varnish in front of the application load balancers. In this way, Varnish would be the primary page assembler. We would likely run an HA setup for fault tolerance. We were also considering if Varnish could be used much farther downstream ? in between the load balancer and the application. We thought is might provide for more fault tolerance, but it seems the disadvantage would be that there would be a separate cached copy in each redundant varnish instance, and with ESI secondary requests would be made back out to the same web server potentially, setting up a feed back loop that could cause problems. Thanks for your responses in advance. August August Gresens Director of Technology, eScholastic Scholastic Inc. 557 Broadway, NY, NY 10012 Email :: agresens at scholastic.com Phone :: 917 363 3662 _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org http://lists.varnish-cache.org/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From thebog at gmail.com Fri Oct 22 01:39:09 2010 From: thebog at gmail.com (thebog) Date: Fri, 22 Oct 2010 01:39:09 +0200 Subject: Fwd: infrastructure setup using ESI In-Reply-To: References: <8D333BF67F814C4D9803C99961BE5E70014EDA6E@corpex07.corp.scholasticinc.local> Message-ID: Oppss, forgot to include the list. ---------- Forwarded message ---------- From: thebog Date: Fri, Oct 22, 2010 at 1:38 AM Subject: Re: infrastructure setup using ESI To: "Gresens, August" Hi, I'll try and give some advice in general terms. You are asking if this rope is long enuff, and that question has no good answer :) Varnish is "most often" used close to the client (in front or behind loadbalancer). A main reason is that it handles alot of webtraffic good and cheap. This is where it's written to shine. But it shines in other layers as well. depending on what you are trying to do: How many documents do you have? Is your backend good? How much traffic is backend seeing? Does backend scale good/cheap? What's your budget? What hardware do you have/plan on getting? Varnish works good in front of crappy backend, because it's easy/flexible (configurationwise) to reduce/shape traffic to backend. If your hot dataset is huge, you will need much RAM, and there the budget kicks in. Some people dream of 64 GB RAM, some buy it by default, and for some 64 GB is such a little part of the dataset that it really does not help much. While others have a dataset where 4 GB RAM is more than enuff, so it's cheap to do it. For some it does not matter in if there are separate cached copy's, because it's free, so why not. For some it does. Hard to answer :) So are the secondary requests to the backend. With Varnish you can manipulate HTTP headers that otherwise does not make sense because you have a backend you can not tweak to get what you want (Or this is more expensive than using Varnish). Some use Varnish in front of many different backend systems, some use it in front of crappy closed source backends. Many find VCL (Varnish configuration laguage) to be really good and easy (some don't bother ever to learn it because they don't need it), I can't think of many scenarios where you can't get VCL to do what ever you'd like with HTTP and to some extent most of the HTTP traffic. So I would use Varnish where I have much traffic, where I need to reduce traffic, and where I would like to shape traffic if I had to. It's a tool, when you learn how to use it, you know when it makes sense to use it. And it can be used many places in the stack. I would also not install anything other than a 64 bit OS. And focus on getting the right amount of RAM. As much as possible is always a good start :) Hope this answered some of your questions :) Sorry for the rant. YS Anders Berg On Thu, Oct 21, 2010 at 11:14 PM, Gresens, August wrote: > Hello > > > > Varnish is a key part of a new set of apps we are building - the ESI > feature in particular is something we want to take advantage of extensively. > > > > Where does Varnish typically reside in the network architecture? I would > imagine in a scenario in which we would be using ESI, we would want Varnish > in front of the application load balancers. In this way, Varnish would be > the primary page assembler. We would likely run an HA setup for fault > tolerance. > > > > We were also considering if Varnish could be used much farther downstream ? > in between the load balancer and the application. We thought is might > provide for more fault tolerance, but it seems the disadvantage would be > that there would be a separate cached copy in each redundant varnish > instance, and with ESI secondary requests would be made back out to the same > web server potentially, setting up a feed back loop that could cause > problems. > > > > Thanks for your responses in advance. > > > > August > > > > > > August Gresens > Director of Technology, eScholastic > > Scholastic Inc. > 557 Broadway, NY, NY 10012 > > Email :: agresens at scholastic.com > Phone :: 917 363 3662 > > > > > > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Fri Oct 22 09:48:56 2010 From: perbu at varnish-software.com (Per Buer) Date: Fri, 22 Oct 2010 09:48:56 +0200 Subject: infrastructure setup using ESI In-Reply-To: <8D333BF67F814C4D9803C99961BE5E70014EDA6E@corpex07.corp.scholasticinc.local> References: <8D333BF67F814C4D9803C99961BE5E70014EDA6E@corpex07.corp.scholasticinc.local> Message-ID: Hi, On Thu, Oct 21, 2010 at 11:14 PM, Gresens, August wrote: > > Where does Varnish typically reside in the network architecture? I would > imagine in a scenario in which we would be using ESI, we would want Varnish > in front of the application load balancers. In this way, Varnish would be > the primary page assembler. We would likely run an HA setup for fault > tolerance. > Since I am such an expert in picking out color for such bikesheds I thought I might chime in. :-) It actually makes quite a lot of sense to have Varnish do the actual application load balancing. That way you can leverage the saint mode properly giving you far better fault tolerance. Imagine one app server failing; Varnish can then try the other one, if that one fails as well serve up some stale content from cache. > We were also considering if Varnish could be used much farther downstream > ? in between the load balancer and the application. We thought is might > provide for more fault tolerance, but it seems the disadvantage would be > that there would be a separate cached copy in each redundant varnish > instance, and with ESI secondary requests would be made back out to the same > web server potentially, setting up a feed back loop that could cause > problems. > As has already been suggested - place a load balancer in front of Varnish and make it hash on target URL. Oh, did I mention that it makes sense to use Varnish as a hashing load balancer also? Just turn off the caching. There are some features that Varnish might lack compared to other load balancers (gzip, ssl) but it more then makes that up with speed and flexibility. -- Per Buer, Varnish Software Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / skype: per.buer -------------- next part -------------- An HTML attachment was scrubbed... URL: From cosimo at streppone.it Fri Oct 22 10:32:35 2010 From: cosimo at streppone.it (Cosimo Streppone) Date: Fri, 22 Oct 2010 10:32:35 +0200 Subject: infrastructure setup using ESI In-Reply-To: <8D333BF67F814C4D9803C99961BE5E70014EDA6E@corpex07.corp.scholasticinc.local> References: <8D333BF67F814C4D9803C99961BE5E70014EDA6E@corpex07.corp.scholasticinc.local> Message-ID: On Thu, 21 Oct 2010 23:14:53 +0200, Gresens, August wrote: > Where does Varnish typically reside in the network architecture? I would > imagine in a scenario in which we would be using ESI, we would want > Varnish in front of the application load balancers > > [...] > > We were also considering if Varnish could be used much farther > downstream - in between the load balancer and the application Until last week, we were using varnish as in: Interwebz -> LVS -> Nginx fronts -> Varnish -> Backend LVS -> Backends That is essentially our pre-existing architecture, where we "injected" varnish, to try and do the least possible damage. That was a year ago. Lots of stuff happened, and we realized Varnish can help us a bit more. Last week, we shifted varnish in 2nd position: Interwebz -> LVS -> Varnish -> Nginx fronts -> Backend LVS -> Backends We built a very simple shields-up configuration, meant to prevent backends overloading, with a low ttl (5-10s), and we entirely avoid caching: - !GET requests (POST, HEAD, ...) - requests from logged in users It's been working fine so far. We're setting up load testing to find out the max traffic level we can handle :) -- Cosimo From AGresens at Scholastic.com Fri Oct 22 14:44:13 2010 From: AGresens at Scholastic.com (Gresens, August) Date: Fri, 22 Oct 2010 08:44:13 -0400 Subject: infrastructure setup using ESI In-Reply-To: Message-ID: <8D333BF67F814C4D9803C99961BE5E70014EDA77@corpex07.corp.scholasticinc.local> Thanks - the rant was awesome - this has been extremely helpful. Here are responses to some of your questions: How many documents do you have? [AG] Approximately 75,000 pages of unique content. Since we are using Akamai, we would not need to cache static assets (images, static html files, css, javascript, etc) only pages rendered by the CMS (see below). We plan on using ESI to support dynamic portions of the pages. Is your backend good? [AG] We're using an open source CMS (Concrete5). This CMS features very flexible UI manipulation tools, but I believe like all of these CMS systems there is a price to pay for flexibility, and this is usually in performance. Because the database schemas cannot be optimized to support a specific data types, they generally do not perform as well. Like most CMS systems, there is caching at the application layer - or we can use memCache - but it seems the complexity of implementing these caches are higher than putting Varnish in front. Additionally, these caches do not support anything nearly as elegant as saint mode - particularly when used in combination with ESI. So, the answer for the purposes of this discussion is - not bad but not good enough that we can use it comfortably without caching. How much traffic is backend seeing? [AG] Peak traffic is about 150-200/requests per second Does backend scale good/cheap? [AG] We are using Xen virtualization and we have a fair amount of capacity on our data center, so scaling out is not hard and/or expensive. What's your budget? [AG] We already have our own data center so the incremental costs would be not be high. This is a new project so the cost would be rolled into the budget for the project. What hardware do you have/plan on getting? [AG] We have 24 HP BL460c blades with 24 GB of RAM each. We are using Xen virtualization on most blades. We could dedicate two blades to Varnish in an HA setup, but this would create a single point of failure (even HA can fail). I was thinking we would probably have multiple Varnish HA clusters running on Xen Vms, one in support of each app or split out using the hashing method on the load balancers that others have been suggesting here. ________________________________ From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of thebog Sent: Thursday, October 21, 2010 7:39 PM To: varnish-misc at varnish-cache.org Subject: Fwd: infrastructure setup using ESI Oppss, forgot to include the list. ---------- Forwarded message ---------- From: thebog Date: Fri, Oct 22, 2010 at 1:38 AM Subject: Re: infrastructure setup using ESI To: "Gresens, August" Hi, I'll try and give some advice in general terms. You are asking if this rope is long enuff, and that question has no good answer :) Varnish is "most often" used close to the client (in front or behind loadbalancer). A main reason is that it handles alot of webtraffic good and cheap. This is where it's written to shine. But it shines in other layers as well. depending on what you are trying to do: Varnish works good in front of crappy backend, because it's easy/flexible (configurationwise) to reduce/shape traffic to backend. If your hot dataset is huge, you will need much RAM, and there the budget kicks in. Some people dream of 64 GB RAM, some buy it by default, and for some 64 GB is such a little part of the dataset that it really does not help much. While others have a dataset where 4 GB RAM is more than enuff, so it's cheap to do it. For some it does not matter in if there are separate cached copy's, because it's free, so why not. For some it does. Hard to answer :) So are the secondary requests to the backend. With Varnish you can manipulate HTTP headers that otherwise does not make sense because you have a backend you can not tweak to get what you want (Or this is more expensive than using Varnish). Some use Varnish in front of many different backend systems, some use it in front of crappy closed source backends. Many find VCL (Varnish configuration laguage) to be really good and easy (some don't bother ever to learn it because they don't need it), I can't think of many scenarios where you can't get VCL to do what ever you'd like with HTTP and to some extent most of the HTTP traffic. So I would use Varnish where I have much traffic, where I need to reduce traffic, and where I would like to shape traffic if I had to. It's a tool, when you learn how to use it, you know when it makes sense to use it. And it can be used many places in the stack. I would also not install anything other than a 64 bit OS. And focus on getting the right amount of RAM. As much as possible is always a good start :) Hope this answered some of your questions :) Sorry for the rant. YS Anders Berg On Thu, Oct 21, 2010 at 11:14 PM, Gresens, August wrote: Hello Varnish is a key part of a new set of apps we are building - the ESI feature in particular is something we want to take advantage of extensively. Where does Varnish typically reside in the network architecture? I would imagine in a scenario in which we would be using ESI, we would want Varnish in front of the application load balancers. In this way, Varnish would be the primary page assembler. We would likely run an HA setup for fault tolerance. We were also considering if Varnish could be used much farther downstream - in between the load balancer and the application. We thought is might provide for more fault tolerance, but it seems the disadvantage would be that there would be a separate cached copy in each redundant varnish instance, and with ESI secondary requests would be made back out to the same web server potentially, setting up a feed back loop that could cause problems. Thanks for your responses in advance. August August Gresens Director of Technology, eScholastic Scholastic Inc. 557 Broadway, NY, NY 10012 Email :: agresens at scholastic.com Phone :: 917 363 3662 _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org http://lists.varnish-cache.org/mailman/listinfo/varnish-misc

-------------- next part -------------- An HTML attachment was scrubbed... URL: From russ at vshift.com Fri Oct 22 16:36:55 2010 From: russ at vshift.com (Ruslan Sivak) Date: Fri, 22 Oct 2010 10:36:55 -0400 Subject: Installing on RHEL/CentOS Message-ID: <4CC1A187.8090408@vshift.com> Following the instructions to install this on my CentOS 5.5 x64 from http://www.varnish-cache.org/installation/redhat, it seems that the first part actually works now (someone fixed the link), but the second part doesn't work, as the key is no good. # rpm --nosignature -i http://repo.varnish-cache.org/redhat/el5/noarch/varnish-release-2.1-1.noarch.rpm # yum install varnish Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile * addons: mirror.umoss.org * base: mirror.rackspace.com * extras: mirror.unl.edu * rpmforge: fr2.rpmfind.net * updates: mirrors.liquidweb.com varnish-2.1 | 951 B 00:00 varnish-2.1/primary | 7.7 kB 00:00 varnish-2.1 29/29 Setting up Install Process Resolving Dependencies --> Running transaction check ---> Package varnish.x86_64 0:2.1.4-1.el5 set to be updated --> Processing Dependency: varnish-libs = 2.1.4-1.el5 for package: varnish --> Processing Dependency: gcc for package: varnish --> Processing Dependency: /usr/bin/mkpasswd for package: varnish --> Processing Dependency: libvarnish.so.1()(64bit) for package: varnish --> Processing Dependency: libvarnishcompat.so.1()(64bit) for package: varnish --> Processing Dependency: libvarnishapi.so.1()(64bit) for package: varnish --> Processing Dependency: libvcl.so.1()(64bit) for package: varnish --> Running transaction check ---> Package expect.x86_64 0:5.43.0-5.1 set to be updated ---> Package gcc.x86_64 0:4.1.2-48.el5 set to be updated --> Processing Dependency: libgomp >= 4.1.2-48.el5 for package: gcc --> Processing Dependency: glibc-devel >= 2.2.90-12 for package: gcc ---> Package varnish-libs.x86_64 0:2.1.4-1.el5 set to be updated --> Running transaction check ---> Package glibc-devel.x86_64 0:2.5-49.el5_5.6 set to be updated --> Processing Dependency: glibc-headers = 2.5-49.el5_5.6 for package: glibc-dev el --> Processing Dependency: glibc-headers for package: glibc-devel ---> Package libgomp.x86_64 0:4.4.0-6.el5 set to be updated --> Running transaction check ---> Package glibc-headers.x86_64 0:2.5-49.el5_5.6 set to be updated --> Processing Dependency: kernel-headers >= 2.2.1 for package: glibc-headers --> Processing Dependency: kernel-headers for package: glibc-headers --> Running transaction check ---> Package kernel-headers.x86_64 0:2.6.18-194.17.1.el5 set to be updated --> Finished Dependency Resolution Dependencies Resolved ================================================================================ Package Arch Version Repository Size ================================================================================ Installing: varnish x86_64 2.1.4-1.el5 varnish-2.1 293 k Installing for dependencies: expect x86_64 5.43.0-5.1 base 160 k gcc x86_64 4.1.2-48.el5 base 5.3 M glibc-devel x86_64 2.5-49.el5_5.6 updates 2.4 M glibc-headers x86_64 2.5-49.el5_5.6 updates 592 k kernel-headers x86_64 2.6.18-194.17.1.el5 updates 1.1 M libgomp x86_64 4.4.0-6.el5 base 68 k varnish-libs x86_64 2.1.4-1.el5 varnish-2.1 98 k Transaction Summary ================================================================================ Install 8 Package(s) Upgrade 0 Package(s) Total download size: 9.9 M Is this ok [y/N]: y Downloading Packages: (1/8): libgomp-4.4.0-6.el5.x86_64.rpm | 68 kB 00:00 (2/8): varnish-libs-2.1.4-1.el5.x86_64.rpm | 98 kB 00:00 (3/8): expect-5.43.0-5.1.x86_64.rpm | 160 kB 00:00 (4/8): varnish-2.1.4-1.el5.x86_64.rpm | 293 kB 00:00 (5/8): glibc-headers-2.5-49.el5_5.6.x86_64.rpm | 592 kB 00:00 (6/8): kernel-headers-2.6.18-194.17.1.el5.x86_64.rpm | 1.1 MB 00:00 (7/8): glibc-devel-2.5-49.el5_5.6.x86_64.rpm | 2.4 MB 00:01 (8/8): gcc-4.1.2-48.el5.x86_64.rpm | 5.3 MB 00:01 -------------------------------------------------------------------------------- Total 1.5 MB/s | 9.9 MB 00:06 error: rpmts_HdrFromFdno: Header V4 RSA/SHA256 signature: BAD, key ID c4deffeb Problem opening package varnish-2.1.4-1.el5.x86_64.rpm Running #yum --nogpgcheck install varnish seemed to fix it, but hopefully you guys can fix it. Russ ru From russ at vshift.com Fri Oct 22 16:48:30 2010 From: russ at vshift.com (Ruslan Sivak) Date: Fri, 22 Oct 2010 10:48:30 -0400 Subject: Installing on RHEL/CentOS In-Reply-To: <4CC1A187.8090408@vshift.com> References: <4CC1A187.8090408@vshift.com> Message-ID: <4CC1A43E.6000300@vshift.com> On 10/22/2010 10:36 AM, Ruslan Sivak wrote: > Following the instructions to install this on my CentOS 5.5 x64 from > http://www.varnish-cache.org/installation/redhat, it seems that the > first part actually works now (someone fixed the link), but the second > part doesn't work, as the key is no good. > > # rpm --nosignature -i > http://repo.varnish-cache.org/redhat/el5/noarch/varnish-release-2.1-1.noarch.rpm > > # yum install varnish snip > Total 1.5 MB/s | 9.9 MB > 00:06 > error: rpmts_HdrFromFdno: Header V4 RSA/SHA256 signature: BAD, key ID > c4deffeb > > > Problem opening package varnish-2.1.4-1.el5.x86_64.rpm > > > Running > #yum --nogpgcheck install varnish > seemed to fix it, but hopefully you guys can fix it. > > Russ > Actually that does not actually install it. I tried removing the varnish-release, but keep getting this error: error: rpmdbNextIterator: skipping h# 985 Header V4 RSA/SHA256 signature: BAD, key ID c4deffeb error: rpmdbNextIterator: skipping h# 985 Header V4 RSA/SHA256 signature: BAD, key ID c4deffeb error: rpmdbNextIterator: skipping h# 985 Header V4 RSA/SHA256 signature: BAD, key ID c4deffeb error: rpmdbNextIterator: skipping h# 985 Header V4 RSA/SHA256 signature: BAD, key ID c4deffeb error: rpmdbNextIterator: skipping h# 985 Header V4 RSA/SHA256 signature: BAD, key ID c4deffeb error: rpmdbNextIterator: skipping h# 985 Header V4 RSA/SHA256 signature: BAD, key ID c4deffeb error: rpmdbNextIterator: skipping h# 985 Header V4 RSA/SHA256 signature: BAD, key ID c4deffeb error: rpmdbNextIterator: skipping h# 985 Header V4 RSA/SHA256 signature: BAD, key ID c4deffeb error: rpmdbNextIterator: skipping h# 985 Header V4 RSA/SHA256 signature: BAD, key ID c4deffeb Same thing happens when I install http://repo.varnish-cache.org/redhat/el5/noarch/varnish-release-2.1-2.noarch.rpm Russ From cosimo at streppone.it Fri Oct 22 21:50:44 2010 From: cosimo at streppone.it (Cosimo Streppone) Date: Fri, 22 Oct 2010 21:50:44 +0200 Subject: infrastructure setup using ESI In-Reply-To: <095AD1B1-4957-495C-9E80-3899904F78AE@daviesinc.com> References: <8D333BF67F814C4D9803C99961BE5E70014EDA6E@corpex07.corp.scholasticinc.local> <095AD1B1-4957-495C-9E80-3899904F78AE@daviesinc.com> Message-ID: In data 22 ottobre 2010 alle ore 19:19:39, Chris Davies ha scritto: > > On Oct 22, 2010, at 4:32 AM, Cosimo Streppone wrote: >> Last week, we shifted varnish in 2nd position: >> >> Interwebz -> LVS -> Varnish -> Nginx fronts -> Backend LVS -> Backends > > then you must not be using ESI, which requires nginx in front of Varnish > since Varnish won't compress ESI assembled pages. Correct, I don't use ESI, and I forgot to mention it. /me hides :) I thought August was looking for examples of architecture even out of ESI context. -- Cosimo From a.hongens at netmatch.nl Mon Oct 25 11:24:27 2010 From: a.hongens at netmatch.nl (=?ISO-8859-1?Q?Angelo_H=F6ngens?=) Date: Mon, 25 Oct 2010 11:24:27 +0200 Subject: interpreting s_bodybytes value Message-ID: <4CC54CCB.80505@netmatch.nl> Hey, I was wondering on how to interpret the varnishstat 's_bodybytes' field. s_bodybytes Total body bytes Is this the number of bytes sent to clients, received from backends, or both? -- With kind regards, Angelo H?ngens systems administrator MCSE on Windows 2003 MCSE on Windows 2000 MS Small Business Specialist ------------------------------------------ NetMatch tourism internet software solutions Ringbaan Oost 2b 5013 CA Tilburg +31 (0)13 5811088 +31 (0)13 5821239 A.Hongens at netmatch.nl www.netmatch.nl ------------------------------------------ From phk at phk.freebsd.dk Mon Oct 25 11:45:11 2010 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 25 Oct 2010 09:45:11 +0000 Subject: interpreting s_bodybytes value In-Reply-To: Your message of "Mon, 25 Oct 2010 11:24:27 +0200." <4CC54CCB.80505@netmatch.nl> Message-ID: <70987.1287999911@critter.freebsd.dk> In message <4CC54CCB.80505 at netmatch.nl>, =?ISO-8859-1?Q?Angelo_H=F6ngens?= writ es: >I was wondering on how to interpret the varnishstat 's_bodybytes' field. > >s_bodybytes Total body bytes > >Is this the number of bytes sent to clients, received from backends, or >both? That is bytes of object body sent to the clients -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From pom at dmsp.de Mon Oct 25 12:25:25 2010 From: pom at dmsp.de (Stefan Pommerening) Date: Mon, 25 Oct 2010 12:25:25 +0200 Subject: varnish 2.0.6 and nuked objects Message-ID: <4CC55B15.2010200@dmsp.de> Hi all, I am running varnish 2.0.6 in a production environment without any serious problems. What makes me curious is the absence of any cache size configuration ("-s") and I am also noticing a signficant number of nuked objects although overall memory usage is really low. Determining the process memory using "pmap" shows a reasonable small amount of memory used for caching but a large amount of memory used for the configured 2000 worker threads. Because I have still a lot of RAM unused I am thinking about using a "-s malloc,2G" for the startup configuration but I am unsure whether this might help getting rid of the nuked objects counter. As I understood varnish let the memory utilization up to the operating system, so it might not get any memory problems as there are only about 1/4 of the main memory used (uptime several weeks, ~300 req/sec). I cannot find any documentation on what varnish does if no cache memory is configured (no "-s" option). On the other hand I heard about a bug in 2.0.x varnish which has something to do with memory allocation causing nuked objects every now and then. Maybe it has something to do with this? I upgrading really a good idea? Or what else might have caused the nuked objects? Regards, Stefan From tfheen at varnish-software.com Tue Oct 26 09:43:51 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Tue, 26 Oct 2010 09:43:51 +0200 Subject: Duplicated Content-Length headers on pass Message-ID: <87mxq1flfs.fsf@qurzaw.linpro.no> Hi all, it seems like a small, but pesky bug snuck its way into 2.1.4, in which you might see duplicated Content-Length headers when using pass. As a workaround for this, I believe doing sub vcl_fetch { if (req.request != "HEAD") { unset beresp.http.content-length; } } should work, or you can use the patch from http://www.varnish-cache.org/trac/changeset/5461 This will also hit the 2.1 SVN branch in a few minutes. Feedback on whether this bug poses a real-world problem for people or not would be most welcome. Regards, -- Tollef Fog Heen Varnish Software t: +47 21 98 62 64 From superdupont at gmail.com Tue Oct 26 10:45:05 2010 From: superdupont at gmail.com (Carlos Valiente) Date: Tue, 26 Oct 2010 09:45:05 +0100 Subject: Duplicated Content-Length headers on pass In-Reply-To: <87mxq1flfs.fsf@qurzaw.linpro.no> References: <87mxq1flfs.fsf@qurzaw.linpro.no> Message-ID: > Feedback on whether this bug poses a real-world problem for people or > not would be most welcome. Thanks for the workaround and the patch, Tollef. Chromium (I'm using https://launchpad.net/~chromium-daily/+archive/ppa) refuses to render pages served with a duplicate ``Content-Length`` header, explicitly mentioning it on its error message. Cheers, Carlos From h.paulissen at qbell.nl Tue Oct 26 10:59:20 2010 From: h.paulissen at qbell.nl (Henry Paulissen) Date: Tue, 26 Oct 2010 10:59:20 +0200 Subject: Duplicated Content-Length headers on pass Message-ID: <001e01cb74ec$14205c00$3c611400$@paulissen@qbell.nl> Yes, this was a RL problem for us (found it a month ago, but never reported it). And I did this to solve it: if (resp.status == 302) { remove resp.http.Content-Length; } in vcl_deliver. The problem seems to be in chunked reponses and status codes other as 200 (maybe even only in 301/302). Google Chrome is complaining about duplicate content-length headers and there is already a bug report of it at the chrome dev team, other browsers don't seem to dislike it. Regards, -----Original Message----- From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Tollef Fog Heen Sent: dinsdag 26 oktober 2010 9:44 To: varnish-misc at varnish-cache.org Subject: Duplicated Content-Length headers on pass Hi all, it seems like a small, but pesky bug snuck its way into 2.1.4, in which you might see duplicated Content-Length headers when using pass. As a workaround for this, I believe doing sub vcl_fetch { if (req.request != "HEAD") { unset beresp.http.content-length; } } should work, or you can use the patch from http://www.varnish-cache.org/trac/changeset/5461 This will also hit the 2.1 SVN branch in a few minutes. Feedback on whether this bug poses a real-world problem for people or not would be most welcome. Regards, -- Tollef Fog Heen Varnish Software t: +47 21 98 62 64 _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From h.paulissen at qbell.nl Tue Oct 26 10:52:23 2010 From: h.paulissen at qbell.nl (Henry Paulissen) Date: Tue, 26 Oct 2010 10:52:23 +0200 Subject: Duplicated Content-Length headers on pass In-Reply-To: <87mxq1flfs.fsf@qurzaw.linpro.no> References: <87mxq1flfs.fsf@qurzaw.linpro.no> Message-ID: <000c01cb74eb$1b81d060$52857120$@paulissen@qbell.nl> Yes, this was a RL problem for us (found it a month ago, but never reported it). And I did this to solve it: if (resp.status == 302) { remove resp.http.Content-Length; } in vcl_deliver. The problem seems to be in chunked reponses and status codes other as 200 (maybe even only in 301/302). Google Chrome is complaining about duplicate content-length headers and there is already a bug report of it at the chrome dev team, other browsers don't seem to dislike it. Regards, -----Original Message----- From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Tollef Fog Heen Sent: dinsdag 26 oktober 2010 9:44 To: varnish-misc at varnish-cache.org Subject: Duplicated Content-Length headers on pass Hi all, it seems like a small, but pesky bug snuck its way into 2.1.4, in which you might see duplicated Content-Length headers when using pass. As a workaround for this, I believe doing sub vcl_fetch { if (req.request != "HEAD") { unset beresp.http.content-length; } } should work, or you can use the patch from http://www.varnish-cache.org/trac/changeset/5461 This will also hit the 2.1 SVN branch in a few minutes. Feedback on whether this bug poses a real-world problem for people or not would be most welcome. Regards, -- Tollef Fog Heen Varnish Software t: +47 21 98 62 64 _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org http://lists.varnish-cache.org/mailman/listinfo/varnish-misc From phk at phk.freebsd.dk Tue Oct 26 11:08:02 2010 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 26 Oct 2010 09:08:02 +0000 Subject: Duplicated Content-Length headers on pass In-Reply-To: Your message of "Tue, 26 Oct 2010 10:59:20 +0200." <001e01cb74ec$14205c00$3c611400$@paulissen@qbell.nl> Message-ID: <30772.1288084082@critter.freebsd.dk> In message <001e01cb74ec$14205c00$3c611400$@paulissen at qbell.nl>, "Henry Pauliss en" writes: >Google Chrome is complaining about duplicate content-length headers and >there is already a bug report of it at the chrome dev team, other browsers >don't seem to dislike it. Given that the two headers report the same length, I think that warning is in direct contravention of "be liberal in what you receive and conservative in what you send". That said: of course Varnish should not do that. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From ash2kk at gmail.com Wed Oct 27 12:26:36 2010 From: ash2kk at gmail.com (Mikhail Mazursky) Date: Wed, 27 Oct 2010 16:26:36 +0600 Subject: Duplicated Content-Length headers on pass In-Reply-To: <87mxq1flfs.fsf@qurzaw.linpro.no> References: <87mxq1flfs.fsf@qurzaw.linpro.no> Message-ID: 2010/10/26 Tollef Fog Heen : > > Hi all, > > it seems like a small, but pesky bug snuck its way into 2.1.4, in which > you might see duplicated Content-Length headers when using pass. > > As a workaround for this, I believe doing > > ?sub vcl_fetch { > ? ?if (req.request != "HEAD") { > ? ? ?unset beresp.http.content-length; > ? ?} > ?} That workaround don't work for me. Piece of my VCL: sub vcl_fetch { #FIX for 2.1.4 Content-Length bug if (req.request != "HEAD") { unset beresp.http.content-length; } #FIX for 2.1.4 Content-Length bug if (!beresp.cacheable) { return (pass); } unset beresp.http.Set-Cookie; return (deliver); } Am i doing something wrong? p.s. when are you planing to release varnish with this bug fixed? thanks From ash2kk at gmail.com Wed Oct 27 12:54:15 2010 From: ash2kk at gmail.com (Mikhail Mazursky) Date: Wed, 27 Oct 2010 16:54:15 +0600 Subject: Duplicated Content-Length headers on pass In-Reply-To: References: <87mxq1flfs.fsf@qurzaw.linpro.no> Message-ID: Applied the patch from [1] and it works now as it should. [1]: http://www.varnish-cache.org/trac/changeset/5461 From tfheen at varnish-software.com Wed Oct 27 12:54:42 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Wed, 27 Oct 2010 12:54:42 +0200 Subject: Duplicated Content-Length headers on pass In-Reply-To: (Mikhail Mazursky's message of "Wed, 27 Oct 2010 16:26:36 +0600") References: <87mxq1flfs.fsf@qurzaw.linpro.no> Message-ID: <877hh3c3d9.fsf@qurzaw.linpro.no> ]] Mikhail Mazursky Hi, | 2010/10/26 Tollef Fog Heen : | > | > Hi all, | > | > it seems like a small, but pesky bug snuck its way into 2.1.4, in which | > you might see duplicated Content-Length headers when using pass. | > | > As a workaround for this, I believe doing | > | > ?sub vcl_fetch { | > ? ?if (req.request != "HEAD") { | > ? ? ?unset beresp.http.content-length; | > ? ?} | > ?} | That workaround don't work for me. Piece of my VCL: Can you please provide varnishlog showing that we're sending out a duplicated Content-Length header? Regards, -- Tollef Fog Heen Varnish Software t: +47 21 98 62 64 From russ at vshift.com Wed Oct 27 18:51:22 2010 From: russ at vshift.com (Ruslan Sivak) Date: Wed, 27 Oct 2010 12:51:22 -0400 Subject: Dynamically generating tracking images Message-ID: We are building a site which needs to be cached, but we need to have a tracking image that gets output for every page. This image should contain certain (lets say all) parameters of the query string. I'm thinking of implementing this using ESI. I currently have the following: test.php /esi/tracker.pl #!/usr/bin/perl use CGI::Cookie (); use Apache2::RequestRec (); use APR::Table (); my $cookie = CGI::Cookie->new(-name => 'mod_perl', -value => 'awesome'); print "Set-Cookie: $cookie\n"; print "Content-type: text/html\n\n"; print < ENDHTML default.vcl ... sub vcl_fetch { set beresp.ttl=5d; if (req.url ~"test.php") { esi; /* Do ESI processing */ set beresp.ttl = 24 h; } elseif (req.url == "/esi/tracker.pl") { set beresp.ttl = 1m; } if (beresp.http.set-cookie) { set beresp.http.set-cookie = ";" beresp.http.set-cookie; set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, "; +", ";"); set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, ";(MY_COOKIE)=", "; \1="); set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, ";[^ ][^;]*", ""); set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, "^[; ]+|[; ]+$", ""); if (beresp.http.set-cookie == "") { remove beresp.http.set-cookie; } return (deliver); } } ... Am I going about this the right way? Perl doesn't seem to see the query string or be able to set any cookies. Any help would be appreciated. Russ -------------- next part -------------- An HTML attachment was scrubbed... URL: From josh at schulzone.org Wed Oct 27 19:05:30 2010 From: josh at schulzone.org (Josh) Date: Wed, 27 Oct 2010 10:05:30 -0700 Subject: Dynamically generating tracking images In-Reply-To: References: Message-ID: You have to put the query string parameters into the esi path: I'm handling some tracking the exact same way, and it works fine. The esi request is a separate hit to the backend and doesn't have any of the context from the request that caused it. josh On Wed, Oct 27, 2010 at 9:51 AM, Ruslan Sivak wrote: > We are building a site which needs to be cached, but we need to have a > tracking image that gets output for every page.? This image should contain > certain (lets say all) parameters of the query string. > > I'm thinking of implementing this using ESI.? I currently have the > following: > > test.php > ? > > /esi/tracker.pl > #!/usr/bin/perl > use CGI::Cookie (); > use Apache2::RequestRec (); > use APR::Table (); > my $cookie = CGI::Cookie->new(-name? => 'mod_perl', > ??????????????????????????????????? -value => 'awesome'); > print "Set-Cookie: $cookie\n"; > print "Content-type: text/html\n\n"; > print < $ENV{'QUERY_STRING'} >

> src="http://sometrackingsite/id/12345.gif?url=ENTER+PAGE+URL+HERE&title=ENTER+PAGE+TITLE+HERE" > alt="" height="1" width="1" /> >

> ENDHTML > > default.vcl > ... > sub vcl_fetch { > ? set beresp.ttl=5d; > ?? if (req.url ~"test.php") { > ??????? esi;? /* Do ESI processing */ > ??????? set beresp.ttl = 24 h; > ??? } elseif (req.url == "/esi/tracker.pl") { > ??????? set beresp.ttl = 1m; > ??? } > ? if (beresp.http.set-cookie) { > ??? set beresp.http.set-cookie = ";" beresp.http.set-cookie; > ??? set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, "; +", > ";"); > ??? set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, > ";(MY_COOKIE)=", "; \1="); > ??? set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, ";[^ > ][^;]*", ""); > ??? set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, "^[; > ]+|[; ]+$", ""); > > ??? if (beresp.http.set-cookie == "") { > ??????? remove beresp.http.set-cookie; > ??? } > ??? return (deliver); > ?? } > } > ... > > Am I going about this the right way?? Perl doesn't seem to see the query > string or be able to set any cookies. > > Any help would be appreciated. > > Russ > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > -- josh @schulz http://schulzone.org From russ at vshift.com Wed Oct 27 19:12:47 2010 From: russ at vshift.com (Ruslan Sivak) Date: Wed, 27 Oct 2010 13:12:47 -0400 Subject: Dynamically generating tracking images In-Reply-To: References: Message-ID: So I can't get access to the original query string? What about request cookies? Russ On Wed, Oct 27, 2010 at 1:05 PM, Josh wrote: > You have to put the query string parameters into the esi path: > > > > I'm handling some tracking the exact same way, and it works fine. The > esi request is a separate hit to the backend and doesn't have any of > the context from the request that caused it. > > josh > > On Wed, Oct 27, 2010 at 9:51 AM, Ruslan Sivak wrote: > > We are building a site which needs to be cached, but we need to have a > > tracking image that gets output for every page. This image should > contain > > certain (lets say all) parameters of the query string. > > > > I'm thinking of implementing this using ESI. I currently have the > > following: > > > > test.php > > > > > > /esi/tracker.pl > > #!/usr/bin/perl > > use CGI::Cookie (); > > use Apache2::RequestRec (); > > use APR::Table (); > > my $cookie = CGI::Cookie->new(-name => 'mod_perl', > > -value => 'awesome'); > > print "Set-Cookie: $cookie\n"; > > print "Content-type: text/html\n\n"; > > print < > $ENV{'QUERY_STRING'} > >

> > > src=" > http://sometrackingsite/id/12345.gif?url=ENTER+PAGE+URL+HERE&title=ENTER+PAGE+TITLE+HERE > " > > alt="" height="1" width="1" /> > >

> > ENDHTML > > > > default.vcl > > ... > > sub vcl_fetch { > > set beresp.ttl=5d; > > if (req.url ~"test.php") { > > esi; /* Do ESI processing */ > > set beresp.ttl = 24 h; > > } elseif (req.url == "/esi/tracker.pl") { > > set beresp.ttl = 1m; > > } > > if (beresp.http.set-cookie) { > > set beresp.http.set-cookie = ";" beresp.http.set-cookie; > > set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, "; +", > > ";"); > > set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, > > ";(MY_COOKIE)=", "; \1="); > > set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, ";[^ > > ][^;]*", ""); > > set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, "^[; > > ]+|[; ]+$", ""); > > > > if (beresp.http.set-cookie == "") { > > remove beresp.http.set-cookie; > > } > > return (deliver); > > } > > } > > ... > > > > Am I going about this the right way? Perl doesn't seem to see the query > > string or be able to set any cookies. > > > > Any help would be appreciated. > > > > Russ > > > > _______________________________________________ > > varnish-misc mailing list > > varnish-misc at varnish-cache.org > > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > > > > > > -- > josh > @schulz > http://schulzone.org > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From josh at schulzone.org Wed Oct 27 19:20:55 2010 From: josh at schulzone.org (Josh) Date: Wed, 27 Oct 2010 10:20:55 -0700 Subject: Dynamically generating tracking images In-Reply-To: References:

Message-ID: You can't get access to the original query string unless you specifically pass it along. The ESI passthrough is a totally new disconnected request as far as the backend is concerned. You should get cookies if the esi code could access them directly (IE: On the same domain, path is OK). I havn't tried this, but based on what I know about ESI it that's how it should work. josh On Wed, Oct 27, 2010 at 10:12 AM, Ruslan Sivak wrote: > So I can't get access to the original query string?? What about request > cookies? > > Russ > > On Wed, Oct 27, 2010 at 1:05 PM, Josh wrote: >> >> You have to put the query string parameters into the esi path: >> >> > src="/esi/tracker.pl?url=ENTER+PAGE+URL+HERE&title=ENTER+PAGE+TITLE+HERE"/> >> >> I'm handling some tracking the exact same way, and it works fine. ?The >> esi request is a separate hit to the backend and doesn't have any of >> the context from the request that caused it. >> >> josh >> >> On Wed, Oct 27, 2010 at 9:51 AM, Ruslan Sivak wrote: >> > We are building a site which needs to be cached, but we need to have a >> > tracking image that gets output for every page.? This image should >> > contain >> > certain (lets say all) parameters of the query string. >> > >> > I'm thinking of implementing this using ESI.? I currently have the >> > following: >> > >> > test.php >> > ? >> > >> > /esi/tracker.pl >> > #!/usr/bin/perl >> > use CGI::Cookie (); >> > use Apache2::RequestRec (); >> > use APR::Table (); >> > my $cookie = CGI::Cookie->new(-name? => 'mod_perl', >> > ??????????????????????????????????? -value => 'awesome'); >> > print "Set-Cookie: $cookie\n"; >> > print "Content-type: text/html\n\n"; >> > print <> > $ENV{'QUERY_STRING'} >> >

>> > > > >> > src="http://sometrackingsite/id/12345.gif?url=ENTER+PAGE+URL+HERE&title=ENTER+PAGE+TITLE+HERE" >> > alt="" height="1" width="1" /> >> >

>> > ENDHTML >> > >> > default.vcl >> > ... >> > sub vcl_fetch { >> > ? set beresp.ttl=5d; >> > ?? if (req.url ~"test.php") { >> > ??????? esi;? /* Do ESI processing */ >> > ??????? set beresp.ttl = 24 h; >> > ??? } elseif (req.url == "/esi/tracker.pl") { >> > ??????? set beresp.ttl = 1m; >> > ??? } >> > ? if (beresp.http.set-cookie) { >> > ??? set beresp.http.set-cookie = ";" beresp.http.set-cookie; >> > ??? set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, "; >> > +", >> > ";"); >> > ??? set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, >> > ";(MY_COOKIE)=", "; \1="); >> > ??? set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, ";[^ >> > ][^;]*", ""); >> > ??? set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, "^[; >> > ]+|[; ]+$", ""); >> > >> > ??? if (beresp.http.set-cookie == "") { >> > ??????? remove beresp.http.set-cookie; >> > ??? } >> > ??? return (deliver); >> > ?? } >> > } >> > ... >> > >> > Am I going about this the right way?? Perl doesn't seem to see the query >> > string or be able to set any cookies. >> > >> > Any help would be appreciated. >> > >> > Russ >> > >> > _______________________________________________ >> > varnish-misc mailing list >> > varnish-misc at varnish-cache.org >> > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >> > >> >> >> >> -- >> josh >> @schulz >> http://schulzone.org >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > > -- josh @schulz http://schulzone.org From russ at vshift.com Thu Oct 28 00:10:02 2010 From: russ at vshift.com (Ruslan Sivak) Date: Wed, 27 Oct 2010 18:10:02 -0400 Subject: Remove all but specific Set-Cookie headers Message-ID: <4CC8A33A.2010404@vshift.com> http://www.varnish-cache.org/trac/wiki/VCLExampleRemovingSomeCookies sub vcl_recv { if (!req.http.Cookie) { return (pass); } if (req.http.Cookie) { set req.http.Cookie = ";" req.http.Cookie; set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";"); set req.http.Cookie = regsuball(req.http.Cookie, ";(COOKIE1|COOKIE2)=", "; \1="); set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", ""); set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", ""); if (req.http.Cookie == "") { remove req.http.Cookie; } return(lookup); } } What would be the equivalent sub vcl_fetch? This is what I have, but it seems to only work for COOKIE1. I think because the Cookie header comes from the browser all in one line, but there are multiple Set-Cookie headers. sub vcl_fetch { set beresp.ttl=5d; if (beresp.http.set-cookie) { set beresp.http.set-cookie = ";" beresp.http.set-cookie; set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, "; +", ";"); set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, ";(COOKIE1|COOKIE2)=", "; \1="); set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, ";[^ ][^;]*", ""); set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, "^[; ]+|[; ]+$", ""); if (beresp.http.set-cookie == "") { remove beresp.http.set-cookie; } return (deliver); } } Russ From russ at vshift.com Thu Oct 28 00:32:44 2010 From: russ at vshift.com (Ruslan Sivak) Date: Wed, 27 Oct 2010 18:32:44 -0400 Subject: Remove all but specific Set-Cookie headers In-Reply-To: <4CC8A33A.2010404@vshift.com> References: <4CC8A33A.2010404@vshift.com> Message-ID: <4CC8A88C.7040206@vshift.com> On 10/27/2010 6:10 PM, Ruslan Sivak wrote: > http://www.varnish-cache.org/trac/wiki/VCLExampleRemovingSomeCookies > > sub vcl_recv { > if (!req.http.Cookie) { > return (pass); > } > if (req.http.Cookie) { > set req.http.Cookie = ";" req.http.Cookie; > set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";"); > set req.http.Cookie = regsuball(req.http.Cookie, > ";(COOKIE1|COOKIE2)=", "; \1="); > set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", ""); > set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", ""); > > if (req.http.Cookie == "") { > remove req.http.Cookie; > } > return(lookup); > } > } > > > What would be the equivalent sub vcl_fetch? This is what I have, but it > seems to only work for COOKIE1. I think because the Cookie header comes > from the browser all in one line, but there are multiple Set-Cookie > headers. > > sub vcl_fetch { > set beresp.ttl=5d; > if (beresp.http.set-cookie) { > set beresp.http.set-cookie = ";" beresp.http.set-cookie; > set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, "; > +", ";"); > set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, > ";(COOKIE1|COOKIE2)=", "; \1="); > set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, ";[^ > ][^;]*", ""); > set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, "^[; > ]+|[; ]+$", ""); > > if (beresp.http.set-cookie == "") { > remove beresp.http.set-cookie; > } > return (deliver); > } > } > > Russ It seems that according to http://www.varnish-cache.org/trac/ticket/432, you can only access the first Set-Cookie header? Is this correct? If this is true, why is it an enhancement and not a bug? Russ From ash2kk at gmail.com Thu Oct 28 07:31:12 2010 From: ash2kk at gmail.com (Mikhail Mazursky) Date: Thu, 28 Oct 2010 11:31:12 +0600 Subject: Duplicated Content-Length headers on pass In-Reply-To: <877hh3c3d9.fsf@qurzaw.linpro.no> References: <87mxq1flfs.fsf@qurzaw.linpro.no> <877hh3c3d9.fsf@qurzaw.linpro.no> Message-ID: 2010/10/27 Tollef Fog Heen : > ]] Mikhail Mazursky > > Hi, > > | 2010/10/26 Tollef Fog Heen : > | > > | > Hi all, > | > > | > it seems like a small, but pesky bug snuck its way into 2.1.4, in which > | > you might see duplicated Content-Length headers when using pass. > | > > | > As a workaround for this, I believe doing > | > > | > ?sub vcl_fetch { > | > ? ?if (req.request != "HEAD") { > | > ? ? ?unset beresp.http.content-length; > | > ? ?} > | > ?} > | That workaround don't work for me. Piece of my VCL: > > Can you please provide varnishlog showing that we're sending out a > duplicated Content-Length header? Hi, It seems that the workaround actually works BUT something goes wrong in another way. Some timeouts happen and Content-Length is zero. I attached files with tests i made. Attached image shows one minute timeout, which happens on server which balances requests between varnishes on two servers. I use nginx to balance requests using cache array routing protocol. nginx is configured to failover after 60 seconds to another varnish backed. I also attached the VCL file i tested with. And that's what i see in nginx's error log when timeouts happen (the port is 1380 when i run varnish in production and i change it to 1381 to remove actual traffic from it; this is a part of production log): # cat /var/log/nginx-error/**********.error.log | grep 92ed2d89c293116b5e06d966f7f238e0.jpg 2010/10/28 09:57:21 [error] 55417#0: *227171859 upstream timed out (60: Operation timed out) while reading response header from upstream, client: **********, server: **********, request: "GET /box/132x99/92ed2d89c293116b5e06d966f7f238e0.jpg HTTP/1.1", upstream: "http://127.0.0.1:1380/box/132x99/92ed2d89c293116b5e06d966f7f238e0.jpg", host: "**********" Hope this helps and thanks for this great piece of software! -------------- next part -------------- with patched VCL: # telnet 127.0.0.1 1381 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. GET /box/132x99/92ed2d89c293116b5e06d966f7f238e0.jpg HTTP/1.1 Host: ********* HTTP/1.1 200 OK Server: nginx Content-Type: image/jpeg Last-Modified: Mon, 06 Sep 2010 09:55:59 GMT Expires: Thu, 31 Dec 2037 23:55:55 GMT Cache-Control: max-age=315360000 Content-Length: 0 Date: Thu, 28 Oct 2010 04:55:25 GMT X-Varnish: 868874587 Age: 0 Via: 1.1 varnish Connection: keep-alive Connection closed by foreign host. varnishlog output: 0 CLI - Rd ping 0 CLI - Wr 200 19 PONG 1288241806 1.0 0 CLI - Rd ping 0 CLI - Wr 200 19 PONG 1288241809 1.0 6 SessionOpen c 127.0.0.1 49451 127.0.0.1:1381 6 Debug c "herding" 6 ReqStart c 127.0.0.1 49451 868874588 6 RxRequest c GET 6 RxURL c /box/132x99/92ed2d89c293116b5e06d966f7f238e0.jpg 6 RxProtocol c HTTP/1.1 6 RxHeader c Host: ********* 6 VCL_call c recv 6 VCL_return c lookup 6 VCL_call c hash 6 VCL_return c hash 6 Hit c 868874587 6 VCL_call c hit 6 VCL_return c deliver 6 VCL_call c deliver 6 VCL_return c deliver 6 TxProtocol c HTTP/1.1 6 TxStatus c 200 6 TxResponse c OK 6 TxHeader c Server: nginx 6 TxHeader c Content-Type: image/jpeg 6 TxHeader c Last-Modified: Mon, 06 Sep 2010 09:55:59 GMT 6 TxHeader c Expires: Thu, 31 Dec 2037 23:55:55 GMT 6 TxHeader c Cache-Control: max-age=315360000 6 TxHeader c Content-Length: 0 6 TxHeader c Date: Thu, 28 Oct 2010 04:56:50 GMT 6 TxHeader c X-Varnish: 868874588 868874587 6 TxHeader c Age: 86 6 TxHeader c Via: 1.1 varnish 6 TxHeader c Connection: keep-alive 6 Length c 0 6 ReqEnd c 868874588 1288241810.851414680 1288241810.851518393 0.638790131 0.000051975 0.000051737 6 Debug c "herding" 0 CLI - Rd ping 0 CLI - Wr 200 19 PONG 1288241812 1.0 6 SessionClose c timeout 6 StatSess c 127.0.0.1 49451 1 1 1 0 0 0 319 0 0 CLI - Rd ping 0 CLI - Wr 200 19 PONG 1288241815 1.0 -------------- next part -------------- with patched VCL and no-cache: # telnet 127.0.0.1 1381 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. GET /box/132x99/92ed2d89c293116b5e06d966f7f238e0.jpg HTTP/1.1 Host: ********* Cache-Control: no-cache HTTP/1.1 200 OK Server: nginx Content-Type: image/jpeg Last-Modified: Mon, 06 Sep 2010 09:55:59 GMT Expires: Thu, 31 Dec 2037 23:55:55 GMT Cache-Control: max-age=315360000 Content-Length: 0 Date: Thu, 28 Oct 2010 05:01:29 GMT X-Varnish: 868874589 Age: 0 Via: 1.1 varnish Connection: keep-alive Connection closed by foreign host. varnishlog output: 6 SessionOpen c 127.0.0.1 55271 127.0.0.1:1381 6 Debug c "herding" 0 CLI - Rd ping 0 CLI - Wr 200 19 PONG 1288242087 1.0 6 ReqStart c 127.0.0.1 55271 868874589 6 RxRequest c GET 6 RxURL c /box/132x99/92ed2d89c293116b5e06d966f7f238e0.jpg 6 RxProtocol c HTTP/1.1 6 RxHeader c Host: ********* 6 RxHeader c Cache-Control: no-cache 6 VCL_call c recv 6 VCL_return c lookup 6 VCL_call c hash 6 VCL_return c hash 6 ExpBan c 868874587 was banned 6 VCL_call c miss 6 VCL_return c fetch 10 BackendClose - default 10 BackendOpen b default 127.0.0.1 55347 127.0.0.1 380 6 Backend c 10 default default 10 TxRequest b GET 10 TxURL b /box/132x99/92ed2d89c293116b5e06d966f7f238e0.jpg 10 TxProtocol b HTTP/1.1 10 TxHeader b Host: ********* 10 TxHeader b X-Forwarded-For: 127.0.0.1 10 TxHeader b X-Varnish: 868874589 10 RxProtocol b HTTP/1.1 10 RxStatus b 200 10 RxResponse b OK 10 RxHeader b Server: nginx 10 RxHeader b Date: Thu, 28 Oct 2010 05:01:29 GMT 10 RxHeader b Content-Type: image/jpeg 10 RxHeader b Connection: keep-alive 10 RxHeader b Content-Length: 3067 10 RxHeader b Last-Modified: Mon, 06 Sep 2010 09:55:59 GMT 10 RxHeader b Expires: Thu, 31 Dec 2037 23:55:55 GMT 10 RxHeader b Cache-Control: max-age=315360000 10 RxHeader b Accept-Ranges: bytes 6 TTL c 868874589 RFC 315360000 1288242089 0 0 315360000 0 6 VCL_call c fetch 6 VCL_return c deliver 6 ObjProtocol c HTTP/1.1 6 ObjStatus c 200 6 ObjResponse c OK 6 ObjHeader c Server: nginx 6 ObjHeader c Date: Thu, 28 Oct 2010 05:01:29 GMT 6 ObjHeader c Content-Type: image/jpeg 6 ObjHeader c Last-Modified: Mon, 06 Sep 2010 09:55:59 GMT 6 ObjHeader c Expires: Thu, 31 Dec 2037 23:55:55 GMT 6 ObjHeader c Cache-Control: max-age=315360000 10 Length b 0 10 BackendReuse b default 6 VCL_call c deliver 6 VCL_return c deliver 6 TxProtocol c HTTP/1.1 6 TxStatus c 200 6 TxResponse c OK 6 TxHeader c Server: nginx 6 TxHeader c Content-Type: image/jpeg 6 TxHeader c Last-Modified: Mon, 06 Sep 2010 09:55:59 GMT 6 TxHeader c Expires: Thu, 31 Dec 2037 23:55:55 GMT 6 TxHeader c Cache-Control: max-age=315360000 6 TxHeader c Content-Length: 0 6 TxHeader c Date: Thu, 28 Oct 2010 05:01:29 GMT 6 TxHeader c X-Varnish: 868874589 6 TxHeader c Age: 0 6 TxHeader c Via: 1.1 varnish 6 TxHeader c Connection: keep-alive 6 Length c 0 6 ReqEnd c 868874589 1288242089.776962996 1288242089.782082319 2.879873037 0.005076170 0.000043154 6 Debug c "herding" 0 ExpKill - 868874587 -1288242090 0 CLI - Rd ping 0 CLI - Wr 200 19 PONG 1288242090 1.0 0 CLI - Rd ping 0 CLI - Wr 200 19 PONG 1288242093 1.0 6 SessionClose c timeout 6 StatSess c 127.0.0.1 55271 3 1 1 0 0 1 308 0 -------------- next part -------------- A non-text attachment was scrubbed... Name: timeouts.png Type: image/png Size: 129101 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: custom.vcl Type: application/octet-stream Size: 2533 bytes Desc: not available URL: From escario at azylog.net Thu Oct 28 16:37:45 2010 From: escario at azylog.net (Julien Escario) Date: Thu, 28 Oct 2010 16:37:45 +0200 Subject: 503 Service unavailable with a AJAX request Message-ID: <4CC98AB9.3030804@azylog.net> Hi everyone, I'm pretty newbie to varnish and trying to place one instance in front of a whole shared hosting webserver to reduce the load. One of my customer discovered that when trying to access the backoffice of a prestashop site (http://www.prestashop.com/en/), the system is sending back an error 503. The URL is a POST like this (as said by varnishncsa) : POST http://www.eric-collection.com/cpanel/index.php?tab=AdminCatalog HTTP/1.1 This is a clickable image that calls a javascript function : onclick="getE('submitFilterproduct').value=2" getE() seems to be an alias for getElementByID. This why I'm thinking of some kind of AJAX call. I'm using varnish 2.1.4 SVN You'll find below the full log of this specific request. I don't know what else can be checked. Any idea on the problem and how to solve it ? Thanks you, Julien 13 SessionOpen c 92.61.134.71 49973 :80 13 Debug c "herding" 13 ReqStart c 92.61.134.71 49973 2059649223 13 RxRequest c POST 13 RxURL c /cpanel/index.php?tab=AdminCatalog 13 RxProtocol c HTTP/1.1 13 RxHeader c Host: www.eric-collections.com 13 RxHeader c User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; fr; rv:1.9.2.12) Gecko/20101027 Ubuntu/10.10 (maverick) Firefox/3.6.12 13 RxHeader c Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 13 RxHeader c Accept-Language: fr,fr-fr;q=0.8,en-us;q=0.5,en;q=0.3 13 RxHeader c Accept-Encoding: gzip,deflate 13 RxHeader c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 13 RxHeader c Keep-Alive: 115 13 RxHeader c Connection: keep-alive 13 RxHeader c Referer: http://www.eric-collections.com/cpanel/index.php?tab=AdminCatalog&token=8bafcd5a500a1187d95c84dfee08500d 13 RxHeader c Cookie: 0a5b149381233bf1a5aa73686b19a724=vy8IX3rzLh0%3Dtx%2F3VNEwdYI%3DKCu0a1iJlWE%3Dc2L2J40kVJE%3DrbrbQ5PPJAM%3DWrAle2ZpCyU%3D%2F1t3yafkMjo%3DOx1P3aPPQkI%3DFlsCzVKOMVE%3DhrfaeVftAsY%3DjjOOPo6co40%3Dw9ft%2BGoWuLU%3D4AF3J5oZxrM%3DAjhM7HUC5TU%3DCJsJh7dlH0c% 13 RxHeader c Content-Type: application/x-www-form-urlencoded 13 RxHeader c Content-Length: 283 13 VCL_call c recv 13 VCL_return c pass 13 VCL_call c hash 13 VCL_return c hash 13 VCL_call c pass 13 VCL_return c pass 14 BackendClose - default 14 BackendOpen b default 195.200.217.243 51495 195.200.217.2 80 13 Backend c 14 default default 14 TxRequest b POST 14 TxURL b /cpanel/index.php?tab=AdminCatalog 14 TxProtocol b HTTP/1.1 14 TxHeader b Host: www.eric-collections.com 14 TxHeader b User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; fr; rv:1.9.2.12) Gecko/20101027 Ubuntu/10.10 (maverick) Firefox/3.6.12 14 TxHeader b Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 14 TxHeader b Accept-Language: fr,fr-fr;q=0.8,en-us;q=0.5,en;q=0.3 14 TxHeader b Accept-Encoding: gzip,deflate 14 TxHeader b Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 14 TxHeader b Referer: http://www.eric-collections.com/cpanel/index.php?tab=AdminCatalog&token=8bafcd5a500a1187d95c84dfee08500d 14 TxHeader b Cookie: 0a5b149381233bf1a5aa73686b19a724=vy8IX3rzLh0%3Dtx%2F3VNEwdYI%3DKCu0a1iJlWE%3Dc2L2J40kVJE%3DrbrbQ5PPJAM%3DWrAle2ZpCyU%3D%2F1t3yafkMjo%3DOx1P3aPPQkI%3DFlsCzVKOMVE%3DhrfaeVftAsY%3DjjOOPo6co40%3Dw9ft%2BGoWhLU%3D4AF3J5oZxrM%3DAjhM7HUC5TU%3DCJsJh7dlH0c% 14 TxHeader b Content-Type: application/x-www-form-urlencoded 14 TxHeader b Content-Length: 283 14 TxHeader b X-Forwarded-For: 92.61.134.71 14 TxHeader b X-Varnish: 2059649223 13 FetchError c http first read error: -2 0 (Success) 14 BackendClose b default 13 VCL_call c error 13 VCL_return c deliver 13 VCL_call c deliver 13 VCL_return c deliver 13 TxProtocol c HTTP/1.1 13 TxStatus c 503 13 TxResponse c Service Unavailable 13 TxHeader c Server: Varnish 13 TxHeader c Retry-After: 0 13 TxHeader c Content-Type: text/html; charset=utf-8 13 TxHeader c Content-Length: 419 13 TxHeader c Date: Thu, 28 Oct 2010 14:20:53 GMT 13 TxHeader c X-Varnish: 2059649223 13 TxHeader c Age: 3 13 TxHeader c Via: 1.1 varnish 13 TxHeader c Connection: close 13 Length c 419 13 ReqEnd c 2059649223 1288275650.343024969 1288275653.194123983 0.004972935 2.851063013 0.000036001 13 SessionClose c error 13 StatSess c 92.61.134.71 49973 3 1 1 0 1 0 235 419 From moseleymark at gmail.com Fri Oct 29 01:04:14 2010 From: moseleymark at gmail.com (Mark Moseley) Date: Thu, 28 Oct 2010 16:04:14 -0700 Subject: Dealing with large unwanted objects Message-ID: We've had varnish running in a web hosting environment for quite a while. As such, most of the things we ask it to cache are of a sane size, but being web hosting, customers come up with the most painful corner cases. I've been looking for the "right" way to do this for a few months now but I'm not even sure there is a way to do what I'm looking for. What I'm trying to do is *not* cache an object that's way too big (which will almost certainly knock other things out of the working set). I'm doing a half-way job of it by calling 'pass' in vcl_fetch, i.e., sub vcl_fetch { if ( !beresp.cacheable ) { return ( pass ); } if ( beresp.http.Set-Cookie ) { return ( pass ); } # This is a header we add via apache if it knows some reason not to cache, like IP restrictions. if ( beresp.http.X-VC == "no-cache" ) { return ( pass ); } # This is the rule to knock out big files if ( beresp.http.Content-Length ~ "[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]" ) { return ( pass ); } if ( beresp.status == 206 ) { return ( pass ); } return ( deliver ); } So in this example, anything over 10meg gets 'pass'. The ">" operator errors out here, btw, presumably because it's not cast to an integer. The downside to doing this is that it seems to be too late to keep the object from being put (temporarily) in the cache -- or at least the memory appears to get fully allocated or if I'm using using a disk-backed cache, it spins the disks wildly for a while if the object is huge. And on our non-SSD boxes where we're using malloc, this can easily exhaust varnish's memory and cause the child to die. Ideally, as soon as it received the headers, I'd like to call 'pipe' so varnish just streams the data through and doesn't disturb the cache. Am I missing something terribly obvious or does such a phase definitely not exist? And if not, is there a better way to do this? I'm running mainly 2.1.3 on both i386 and 64-bit Debian Lenny, but I've been tinkering with this problem since well back into 2.0.x. Thanks! From russ at vshift.com Fri Oct 29 01:15:54 2010 From: russ at vshift.com (Ruslan Sivak) Date: Thu, 28 Oct 2010 19:15:54 -0400 Subject: Installing on RHEL/CentOS In-Reply-To: <4CC1A43E.6000300@vshift.com> References: <4CC1A187.8090408@vshift.com> <4CC1A43E.6000300@vshift.com> Message-ID: Any fixes on any of these issues? Should I file a bug? On Oct 22, 2010 10:49 AM, "Ruslan Sivak" wrote: > On 10/22/2010 10:36 AM, Ruslan Sivak wrote: >> Following the instructions to install this on my CentOS 5.5 x64 from >> http://www.varnish-cache.org/installation/redhat, it seems that the >> first part actually works now (someone fixed the link), but the second >> part doesn't work, as the key is no good. >> >> # rpm --nosignature -i >> http://repo.varnish-cache.org/redhat/el5/noarch/varnish-release-2.1-1.noarch.rpm >> >> # yum install varnish > snip >> Total 1.5 MB/s | 9.9 MB >> 00:06 >> error: rpmts_HdrFromFdno: Header V4 RSA/SHA256 signature: BAD, key ID >> c4deffeb >> >> >> Problem opening package varnish-2.1.4-1.el5.x86_64.rpm >> >> >> Running >> #yum --nogpgcheck install varnish >> seemed to fix it, but hopefully you guys can fix it. >> >> Russ >> > Actually that does not actually install it. > > I tried removing the varnish-release, but keep getting this error: > > error: rpmdbNextIterator: skipping h# 985 Header V4 RSA/SHA256 > signature: BAD, key ID c4deffeb > error: rpmdbNextIterator: skipping h# 985 Header V4 RSA/SHA256 > signature: BAD, key ID c4deffeb > error: rpmdbNextIterator: skipping h# 985 Header V4 RSA/SHA256 > signature: BAD, key ID c4deffeb > error: rpmdbNextIterator: skipping h# 985 Header V4 RSA/SHA256 > signature: BAD, key ID c4deffeb > error: rpmdbNextIterator: skipping h# 985 Header V4 RSA/SHA256 > signature: BAD, key ID c4deffeb > error: rpmdbNextIterator: skipping h# 985 Header V4 RSA/SHA256 > signature: BAD, key ID c4deffeb > error: rpmdbNextIterator: skipping h# 985 Header V4 RSA/SHA256 > signature: BAD, key ID c4deffeb > error: rpmdbNextIterator: skipping h# 985 Header V4 RSA/SHA256 > signature: BAD, key ID c4deffeb > error: rpmdbNextIterator: skipping h# 985 Header V4 RSA/SHA256 > signature: BAD, key ID c4deffeb > > > Same thing happens when I install > http://repo.varnish-cache.org/redhat/el5/noarch/varnish-release-2.1-2.noarch.rpm > > Russ -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Fri Oct 29 09:29:11 2010 From: perbu at varnish-software.com (Per Buer) Date: Fri, 29 Oct 2010 09:29:11 +0200 Subject: Dealing with large unwanted objects In-Reply-To: References: Message-ID: On Fri, Oct 29, 2010 at 1:04 AM, Mark Moseley wrote: > We've had varnish running in a web hosting environment for quite a > while. As such, most of the things we ask it to cache are of a sane > size, but being web hosting, customers come up with the most painful > corner cases. I've been looking for the "right" way to do this for a > few months now but I'm not even sure there is a way to do what I'm > looking for. What I'm trying to do is *not* cache an object that's way > too big (which will almost certainly knock other things out of the > working set). I'm doing a half-way job of it by calling 'pass' in > vcl_fetch, i.e., Just a comment to make everything crystal clear. I'm sorry if I'm Captain Obvious here. When Varnish does a pass it will do so as a store-and-forward kind of operation. So Varnish will turn to the backend, fetch the object, free space in its store, fetch it deliver it and then discard of it. The bad news is that it might knock out a few object - the good news is that the space is quickly freed up again. -- Per Buer,? Varnish Software Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / skype: per.buer From tdevelioglu at ebuddy.com Fri Oct 29 13:37:21 2010 From: tdevelioglu at ebuddy.com (Taylan Develioglu) Date: Fri, 29 Oct 2010 13:37:21 +0200 Subject: varnish 2.1.4 swap and increase in objects:heads. Message-ID: <1288352241.5722.134.camel@oasis> Debian Lenny, Linux varnish0 2.6.33.7 #1 SMP Thu Sep 2 15:27:53 CEST 2010 x86_64 GNU/Linux I have a strange situation on one of our varnish servers. We recently raised the expiry on previously uncached content a (Cache-Control: max-age=0, to Cache-Control: max-age=5). The number of objects vs. heads increased and the system started swapping (vm.swappiness=0). http://lcontent.ebuddy.com/varnish/memory-week.png http://lcontent.ebuddy.com/varnish/varnish_objects-week.png http://lcontent.ebuddy.com/varnish/varnish_expunge-week.png I don't understand why this is happening, bug maybe ? DAEMON_OPTS="-a :27535 \ -T localhost:6082 \ -f /etc/varnish/default.vcl \ -s file,/var/cache/varnish/sda/varnish_storage.bin,135G \ -s file,/var/cache/varnish/sdb/varnish_storage.bin,135G \ -s file,/var/cache/varnish/sdc/varnish_storage.bin,135G \ -s file,/var/cache/varnish/sdd/varnish_storage.bin,135G \ -s file,/var/cache/varnish/sde/varnish_storage.bin,135G \ -s file,/var/cache/varnish/sdf/varnish_storage.bin,135G \ -p thread_pools=4 \ -p thread_pool_min=200 \ -p thread_pool_max=2400 \ -p thread_pool_add_delay=2 \ -p lru_interval=20" ---vcl--- sub vcl_recv { unset req.http.cookie; unset req.http.Authorization; if (req.http.host ~ "^.*host.domain.com.*") { set req.http.host = "host.domain.com"; } if (req.request == "HEAD") { return(pass); } if (req.request == "PURGE") { if (!client.ip ~ purge) { error 405 "Not allowed."; } return(lookup); } if (req.request == "GET" && req.url == "/monitor") { error 200 "OK."; } if (req.backend.healthy) { set req.grace = 30s; } else { set req.grace = 48h; } } sub vcl_fetch { if (beresp.status == 404 || beresp.status >= 300) { return (pass); } set beresp.grace = 48h; } sub vcl_hit { if (req.request == "PURGE") { set obj.ttl = 0s; error 200 "Purged."; } } sub vcl_miss { if (req.request == "PURGE") { error 404 "Not in cache."; } } ---EOF--- From rodrigo at mercadolibre.com Fri Oct 29 13:59:18 2010 From: rodrigo at mercadolibre.com (Rodrigo Benzaquen) Date: Fri, 29 Oct 2010 07:59:18 -0400 (GMT-04:00) Subject: varnish 2.1.4 swap and increase in objects:heads. In-Reply-To: <1288352241.5722.134.camel@oasis> Message-ID: <29097274.10.1288353418406.JavaMail.rbenzaquen@rbenzaquen4310> Hi Taylan, exactly the same is happening to us, we already open a ticket with Varnish software. Please let me know if you find a solution. Thanks Rodrigo ------------------ Rodrigo Benzaquen @rbenzaquen From: "Taylan Develioglu" To: varnish-misc at varnish-cache.org Sent: Viernes, 29 de Octubre 2010 8:37:21 Subject: varnish 2.1.4 swap and increase in objects:heads. Debian Lenny, Linux varnish0 2.6.33.7 #1 SMP Thu Sep 2 15:27:53 CEST 2010 x86_64 GNU/Linux I have a strange situation on one of our varnish servers. We recently raised the expiry on previously uncached content a (Cache-Control: max-age=0, to Cache-Control: max-age=5). The number of objects vs. heads increased and the system started swapping (vm.swappiness=0). http://lcontent.ebuddy.com/varnish/memory-week.png http://lcontent.ebuddy.com/varnish/varnish_objects-week.png http://lcontent.ebuddy.com/varnish/varnish_expunge-week.png I don't understand why this is happening, bug maybe ? DAEMON_OPTS="-a :27535 \ -T localhost:6082 \ -f /etc/varnish/default.vcl \ -s file,/var/cache/varnish/sda/varnish_storage.bin,135G \ -s file,/var/cache/varnish/sdb/varnish_storage.bin,135G \ -s file,/var/cache/varnish/sdc/varnish_storage.bin,135G \ -s file,/var/cache/varnish/sdd/varnish_storage.bin,135G \ -s file,/var/cache/varnish/sde/varnish_storage.bin,135G \ -s file,/var/cache/varnish/sdf/varnish_storage.bin,135G \ -p thread_pools=4 \ -p thread_pool_min=200 \ -p thread_pool_max=2400 \ -p thread_pool_add_delay=2 \ -p lru_interval=20" ---vcl--- sub vcl_recv { unset req.http.cookie; unset req.http.Authorization; if (req.http.host ~ "^.*host.domain.com.*") { set req.http.host = "host.domain.com"; } if (req.request == "HEAD") { return(pass); } if (req.request == "PURGE") { if (!client.ip ~ purge) { error 405 "Not allowed."; } return(lookup); } if (req.request == "GET" && req.url == "/monitor") { error 200 "OK."; } if (req.backend.healthy) { set req.grace = 30s; } else { set req.grace = 48h; } } sub vcl_fetch { if (beresp.status == 404 || beresp.status >= 300) { return (pass); } set beresp.grace = 48h; } sub vcl_hit { if (req.request == "PURGE") { set obj.ttl = 0s; error 200 "Purged."; } } sub vcl_miss { if (req.request == "PURGE") { error 404 "Not in cache."; } } ---EOF--- _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org http://lists.varnish-cache.org/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Fri Oct 29 14:26:43 2010 From: perbu at varnish-software.com (Per Buer) Date: Fri, 29 Oct 2010 14:26:43 +0200 Subject: varnish 2.1.4 swap and increase in objects:heads. In-Reply-To: <29097274.10.1288353418406.JavaMail.rbenzaquen@rbenzaquen4310> References: <1288352241.5722.134.camel@oasis> <29097274.10.1288353418406.JavaMail.rbenzaquen@rbenzaquen4310> Message-ID: Hi. The issues are actually a bit different. One uses "-s file" which really makes the kernel handle the memory management implicitly in comparison to you Rodrigo, which uses -s malloc, where you clearly state that Varnish should allocate this much memory. Regarding -s malloc we recommend that on Linux one should not use more then 75% of the physical memory for -s malloc. In addition to the actual object store Varnish uses quite some memory for it's internal data structures. If you go above 75% chances are that Linux will start swapping a bit which might have a negative effect on overall performance. Tollef is getting back to you outside the list regarding the issue you've raised with us. Per. On Fri, Oct 29, 2010 at 1:59 PM, Rodrigo Benzaquen wrote: > Hi Taylan, exactly the same is happening to us, we already open a ticket > with Varnish software. > > Please let me know if you find a solution. > > Thanks > Rodrigo > > > > > ------------------ > > Rodrigo Benzaquen > @rbenzaquen > > > > ________________________________ > From: "Taylan Develioglu" > To: varnish-misc at varnish-cache.org > Sent: Viernes, 29 de Octubre 2010 8:37:21 > Subject: varnish 2.1.4 swap and increase in objects:heads. > > Debian Lenny, Linux varnish0 2.6.33.7 #1 SMP Thu Sep 2 15:27:53 CEST > 2010 x86_64 GNU/Linux > > I have a strange situation on one of our varnish servers. > > We recently raised the expiry on previously uncached content a > (Cache-Control: max-age=0, to Cache-Control: max-age=5). > > The number of objects vs. heads increased and the system started > swapping (vm.swappiness=0). > > http://lcontent.ebuddy.com/varnish/memory-week.png > http://lcontent.ebuddy.com/varnish/varnish_objects-week.png > http://lcontent.ebuddy.com/varnish/varnish_expunge-week.png > > > I don't understand why this is happening, bug maybe ? > > DAEMON_OPTS="-a :27535 \ > ?? ? ? ? ? ? -T localhost:6082 \ > ?? ? ? ? ? ? -f /etc/varnish/default.vcl \ > ?? ? ? ? ? ? -s file,/var/cache/varnish/sda/varnish_storage.bin,135G \ > ?? ? ? ? ? ? -s file,/var/cache/varnish/sdb/varnish_storage.bin,135G \ > ?? ? ? ? ? ? -s file,/var/cache/varnish/sdc/varnish_storage.bin,135G \ > ?? ? ? ? ? ? -s file,/var/cache/varnish/sdd/varnish_storage.bin,135G \ > ?? ? ? ? ? ? -s file,/var/cache/varnish/sde/varnish_storage.bin,135G \ > ?? ? ? ? ? ? -s file,/var/cache/varnish/sdf/varnish_storage.bin,135G \ > ?? ? ? ? ? ? -p thread_pools=4 \ > ?? ? ? ? ? ? -p thread_pool_min=200 \ > ?? ? ? ? ? ? -p thread_pool_max=2400 \ > ?? ? ? ? ? ? -p thread_pool_add_delay=2 \ > ?? ? ? ? ? ? -p lru_interval=20" > > > > ---vcl--- > sub vcl_recv { > ?? ?unset req.http.cookie; > ?? ?unset req.http.Authorization; > > ?? ?if (req.http.host ~ "^.*host.domain.com.*") { > ?? ? ? ?set req.http.host = "host.domain.com"; > ?? ?} > > ?? ?if (req.request == "HEAD") { > ?? ? ? ?return(pass); > ?? ?} > > ?? ?if (req.request == "PURGE") { > ?? ? ? ?if (!client.ip ~ purge) { > ?? ? ? ? ? ?error 405 "Not allowed."; > ?? ? ? ?} > ?? ? ? ?return(lookup); > ?? ?} > > ?? ?if (req.request == "GET" && req.url == "/monitor") { > ?? ? ? ?error 200 "OK."; > ?? ?} > > ?? ?if (req.backend.healthy) { > ?? ? ? ?set req.grace = 30s; > ??} else { > ?? ? ? ?set req.grace = 48h; > ?? ?} > > } > > ?sub vcl_fetch { > ?? ?if (beresp.status == 404 || beresp.status >= 300) { > ?? ? ? ? return (pass); > ?? ?} > ?? ?set beresp.grace = 48h; > > ?} > > sub vcl_hit { > ?? ?if (req.request == "PURGE") { > ?? ? ? ?set obj.ttl = 0s; > ?? ? ? ?error 200 "Purged."; > ?? ?} > } > > sub vcl_miss { > ?? ?if (req.request == "PURGE") { > ?? ? ? ?error 404 "Not in cache."; > ?? ?} > } > ---EOF--- > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > -- Per Buer,? Varnish Software Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / skype: per.buer From superdupont at gmail.com Fri Oct 29 15:29:29 2010 From: superdupont at gmail.com (Carlos Valiente) Date: Fri, 29 Oct 2010 14:29:29 +0100 Subject: Varnish under OpenSuSE 11.1 (and daemontools) Message-ID: For those planning to run Varnish on OpenSuSE 11.1 (or perhaps any other SuSE flavour): You might save yourself some time if you add the following lines to ``/etc/security/limits.conf``: soft as unlimited hard as unlimited OpenSuSE systems ship with a 20 GB limit on the virtual memory size. If your cache file is bigger than that, ``varnishd`` will not start. The upper limit may be checked like this: # ulimit -v 19909440 # After modifying ``/etc/security/limits.conf`` I was able to run Varnish from the command-line, but running it under daemontools (http://cr.yp.to/daemontools.html) still failed. Wrapping the call to ``varnishd`` with daemontools' ``softlimit`` (http://cr.yp.to/daemontools/softlimit.html) seems to work. These are the relevant bits from my daemontools start script: #!/bin/sh # # exec 2>&1 # [...] cache_size_gb="80" vmem_limit=$(echo "($cache_size_gb + 1) * 1024 * 1024 * 1024" | bc) exec \ softlimit -a $vmem_limit \ varnishd \ -s file,/var/lib/varnish,$cache_size_gb \ [...] From tdevelioglu at ebuddy.com Fri Oct 29 18:14:24 2010 From: tdevelioglu at ebuddy.com (Taylan Develioglu) Date: Fri, 29 Oct 2010 18:14:24 +0200 Subject: Dealing with large unwanted objects In-Reply-To: References:

Message-ID: <1288368864.18624.10.camel@oasis> I think what Per mentions can be solved by the pipe functionality. The problem is in knowing beforehand what size the object is. The only reliable way (especially with chunked-encoding) of knowing that is by fetching it first. Not much help, I know :) On Fri, 2010-10-29 at 09:29 +0200, Per Buer wrote: > On Fri, Oct 29, 2010 at 1:04 AM, Mark Moseley wrote: > > We've had varnish running in a web hosting environment for quite a > > while. As such, most of the things we ask it to cache are of a sane > > size, but being web hosting, customers come up with the most painful > > corner cases. I've been looking for the "right" way to do this for a > > few months now but I'm not even sure there is a way to do what I'm > > looking for. What I'm trying to do is *not* cache an object that's way > > too big (which will almost certainly knock other things out of the > > working set). I'm doing a half-way job of it by calling 'pass' in > > vcl_fetch, i.e., > > Just a comment to make everything crystal clear. I'm sorry if I'm > Captain Obvious here. When Varnish does a pass it will do so as a > store-and-forward kind of operation. So Varnish will turn to the > backend, fetch the object, free space in its store, fetch it deliver > it and then discard of it. The bad news is that it might knock out a > few object - the good news is that the space is quickly freed up > again. > > From tdevelioglu at ebuddy.com Fri Oct 29 18:19:50 2010 From: tdevelioglu at ebuddy.com (Taylan Develioglu) Date: Fri, 29 Oct 2010 18:19:50 +0200 Subject: varnish 2.1.4 swap and increase in objects:heads. In-Reply-To: References: <1288352241.5722.134.camel@oasis> <29097274.10.1288353418406.JavaMail.rbenzaquen@rbenzaquen4310> Message-ID: <1288369190.18624.14.camel@oasis> More of a general question: Is there anyone that uses varnish with a storage size of 800GB+ (single instance, file or malloc) ? I'm just wondering because we've had so much problems with varnish (2.1.2 to 2.1.4) that I'm starting to think that we're an edge case. On Fri, 2010-10-29 at 14:26 +0200, Per Buer wrote: > Hi. > > The issues are actually a bit different. One uses "-s file" which > really makes the kernel handle the memory management implicitly in > comparison to you Rodrigo, which uses -s malloc, where you clearly > state that Varnish should allocate this much memory. > > Regarding -s malloc we recommend that on Linux one should not use more > then 75% of the physical memory for -s malloc. In addition to the > actual object store Varnish uses quite some memory for it's internal > data structures. If you go above 75% chances are that Linux will start > swapping a bit which might have a negative effect on overall > performance. Tollef is getting back to you outside the list regarding > the issue you've raised with us. > > Per. > > On Fri, Oct 29, 2010 at 1:59 PM, Rodrigo Benzaquen > wrote: > > Hi Taylan, exactly the same is happening to us, we already open a ticket > > with Varnish software. > > > > Please let me know if you find a solution. > > > > Thanks > > Rodrigo > > > > > > > > > > ------------------ > > > > Rodrigo Benzaquen > > @rbenzaquen > > > > > > > > ________________________________ > > From: "Taylan Develioglu" > > To: varnish-misc at varnish-cache.org > > Sent: Viernes, 29 de Octubre 2010 8:37:21 > > Subject: varnish 2.1.4 swap and increase in objects:heads. > > > > Debian Lenny, Linux varnish0 2.6.33.7 #1 SMP Thu Sep 2 15:27:53 CEST > > 2010 x86_64 GNU/Linux > > > > I have a strange situation on one of our varnish servers. > > > > We recently raised the expiry on previously uncached content a > > (Cache-Control: max-age=0, to Cache-Control: max-age=5). > > > > The number of objects vs. heads increased and the system started > > swapping (vm.swappiness=0). > > > > http://lcontent.ebuddy.com/varnish/memory-week.png > > http://lcontent.ebuddy.com/varnish/varnish_objects-week.png > > http://lcontent.ebuddy.com/varnish/varnish_expunge-week.png > > > > > > I don't understand why this is happening, bug maybe ? > > > > DAEMON_OPTS="-a :27535 \ > > -T localhost:6082 \ > > -f /etc/varnish/default.vcl \ > > -s file,/var/cache/varnish/sda/varnish_storage.bin,135G \ > > -s file,/var/cache/varnish/sdb/varnish_storage.bin,135G \ > > -s file,/var/cache/varnish/sdc/varnish_storage.bin,135G \ > > -s file,/var/cache/varnish/sdd/varnish_storage.bin,135G \ > > -s file,/var/cache/varnish/sde/varnish_storage.bin,135G \ > > -s file,/var/cache/varnish/sdf/varnish_storage.bin,135G \ > > -p thread_pools=4 \ > > -p thread_pool_min=200 \ > > -p thread_pool_max=2400 \ > > -p thread_pool_add_delay=2 \ > > -p lru_interval=20" > > > > > > > > ---vcl--- > > sub vcl_recv { > > unset req.http.cookie; > > unset req.http.Authorization; > > > > if (req.http.host ~ "^.*host.domain.com.*") { > > set req.http.host = "host.domain.com"; > > } > > > > if (req.request == "HEAD") { > > return(pass); > > } > > > > if (req.request == "PURGE") { > > if (!client.ip ~ purge) { > > error 405 "Not allowed."; > > } > > return(lookup); > > } > > > > if (req.request == "GET" && req.url == "/monitor") { > > error 200 "OK."; > > } > > > > if (req.backend.healthy) { > > set req.grace = 30s; > > } else { > > set req.grace = 48h; > > } > > > > } > > > > sub vcl_fetch { > > if (beresp.status == 404 || beresp.status >= 300) { > > return (pass); > > } > > set beresp.grace = 48h; > > > > } > > > > sub vcl_hit { > > if (req.request == "PURGE") { > > set obj.ttl = 0s; > > error 200 "Purged."; > > } > > } > > > > sub vcl_miss { > > if (req.request == "PURGE") { > > error 404 "Not in cache."; > > } > > } > > ---EOF--- > > > > > > _______________________________________________ > > varnish-misc mailing list > > varnish-misc at varnish-cache.org > > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > > > > _______________________________________________ > > varnish-misc mailing list > > varnish-misc at varnish-cache.org > > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > > > > > From moseleymark at gmail.com Fri Oct 29 18:46:48 2010 From: moseleymark at gmail.com (Mark Moseley) Date: Fri, 29 Oct 2010 09:46:48 -0700 Subject: Dealing with large unwanted objects In-Reply-To: <1288368864.18624.10.camel@oasis> References:

<1288368864.18624.10.camel@oasis> Message-ID: On Fri, Oct 29, 2010 at 9:14 AM, Taylan Develioglu wrote: > I think what Per mentions can be solved by the pipe functionality. > > The problem is in knowing beforehand what size the object is. The only > reliable way (especially with chunked-encoding) of knowing that is by > fetching it first. > > Not much help, I know :) > > >> >> Just a comment to make everything crystal clear. I'm sorry if I'm >> Captain Obvious here. When Varnish does a pass it will do so as a >> store-and-forward kind of operation. So Varnish will turn to the >> backend, fetch the object, free space in its store, fetch it deliver >> it and then discard of it. The bad news is that it might knock out a >> few object - the good news is that the space is quickly freed up >> again. Yup, that's what I was afraid of. 'pipe' doesn't work in vcl_fetch and it'd be too late anyway. That's why I was curious if there was some better way of accomplishing this that I'd missed. Since this is web hosting, we've got no way to tell from the incoming request if something's going to be big (besides file extensions at least). We already route around varnishd if the request is for something that's likely to be huge (.rar, .zip, .mpg, .wmv, etc), which is a bummer since I'd love to cache smaller .rar/.zip/etc's, but without huge ones wreaking havoc on the cache. It's good news that the space is freed up quickly but when the object is half the size of the cache, it not only knocks out a whole lot of objects but also puts a not-insignificant strain on disks, for disk-based caches. Maybe better to just consider this a suggestion for some future major version, to have a hook that gets called after headers have been fetched (or when the first chunk of returned data is seen that contains the header break) but prior to vcl_fetch -- though this could also be a huge performance hit to pause the retrieval. From david.birdsong at gmail.com Fri Oct 29 23:00:35 2010 From: david.birdsong at gmail.com (David Birdsong) Date: Fri, 29 Oct 2010 14:00:35 -0700 Subject: varnish 2.1.4 swap and increase in objects:heads. In-Reply-To: <1288369190.18624.14.camel@oasis> References: <1288352241.5722.134.camel@oasis> <29097274.10.1288353418406.JavaMail.rbenzaquen@rbenzaquen4310> <1288369190.18624.14.camel@oasis> Message-ID: I attempted to run an -s file instance with 5 80GB SSD's raided together. I could never get past filling it half full before the IO load would consume the box. On Fri, Oct 29, 2010 at 9:19 AM, Taylan Develioglu wrote: > More of a general question: > > Is there anyone that uses varnish with a storage size of 800GB+ (single > instance, file or malloc) ? > > I'm just wondering because we've had so much problems with varnish > (2.1.2 to 2.1.4) that I'm starting to think that we're an edge case. > > On Fri, 2010-10-29 at 14:26 +0200, Per Buer wrote: >> Hi. >> >> The issues are actually a bit different. One uses "-s file" which >> really makes the kernel handle the memory management implicitly in >> comparison to you Rodrigo, which uses -s malloc, where you clearly >> state that Varnish should allocate this much memory. >> >> Regarding -s malloc we recommend that on Linux one should not use more >> then 75% of the physical memory for -s malloc. In addition to the >> actual object store Varnish uses quite some memory for it's internal >> data structures. If you go above 75% chances are that Linux will start >> swapping a bit which might have a negative effect on overall >> performance. Tollef is getting back to you outside the list regarding >> the issue you've raised with us. >> >> Per. >> >> On Fri, Oct 29, 2010 at 1:59 PM, Rodrigo Benzaquen >> wrote: >> > Hi Taylan, exactly the same is happening to us, we already open a ticket >> > with Varnish software. >> > >> > Please let me know if you find a solution. >> > >> > Thanks >> > Rodrigo >> > >> > >> > >> > >> > ------------------ >> > >> > Rodrigo Benzaquen >> > @rbenzaquen >> > >> > >> > >> > ________________________________ >> > From: "Taylan Develioglu" >> > To: varnish-misc at varnish-cache.org >> > Sent: Viernes, 29 de Octubre 2010 8:37:21 >> > Subject: varnish 2.1.4 swap and increase in objects:heads. >> > >> > Debian Lenny, Linux varnish0 2.6.33.7 #1 SMP Thu Sep 2 15:27:53 CEST >> > 2010 x86_64 GNU/Linux >> > >> > I have a strange situation on one of our varnish servers. >> > >> > We recently raised the expiry on previously uncached content a >> > (Cache-Control: max-age=0, to Cache-Control: max-age=5). >> > >> > The number of objects vs. heads increased and the system started >> > swapping (vm.swappiness=0). >> > >> > http://lcontent.ebuddy.com/varnish/memory-week.png >> > http://lcontent.ebuddy.com/varnish/varnish_objects-week.png >> > http://lcontent.ebuddy.com/varnish/varnish_expunge-week.png >> > >> > >> > I don't understand why this is happening, bug maybe ? >> > >> > DAEMON_OPTS="-a :27535 \ >> > ? ? ? ? ? ? ?-T localhost:6082 \ >> > ? ? ? ? ? ? ?-f /etc/varnish/default.vcl \ >> > ? ? ? ? ? ? ?-s file,/var/cache/varnish/sda/varnish_storage.bin,135G \ >> > ? ? ? ? ? ? ?-s file,/var/cache/varnish/sdb/varnish_storage.bin,135G \ >> > ? ? ? ? ? ? ?-s file,/var/cache/varnish/sdc/varnish_storage.bin,135G \ >> > ? ? ? ? ? ? ?-s file,/var/cache/varnish/sdd/varnish_storage.bin,135G \ >> > ? ? ? ? ? ? ?-s file,/var/cache/varnish/sde/varnish_storage.bin,135G \ >> > ? ? ? ? ? ? ?-s file,/var/cache/varnish/sdf/varnish_storage.bin,135G \ >> > ? ? ? ? ? ? ?-p thread_pools=4 \ >> > ? ? ? ? ? ? ?-p thread_pool_min=200 \ >> > ? ? ? ? ? ? ?-p thread_pool_max=2400 \ >> > ? ? ? ? ? ? ?-p thread_pool_add_delay=2 \ >> > ? ? ? ? ? ? ?-p lru_interval=20" >> > >> > >> > >> > ---vcl--- >> > sub vcl_recv { >> > ? ? unset req.http.cookie; >> > ? ? unset req.http.Authorization; >> > >> > ? ? if (req.http.host ~ "^.*host.domain.com.*") { >> > ? ? ? ? set req.http.host = "host.domain.com"; >> > ? ? } >> > >> > ? ? if (req.request == "HEAD") { >> > ? ? ? ? return(pass); >> > ? ? } >> > >> > ? ? if (req.request == "PURGE") { >> > ? ? ? ? if (!client.ip ~ purge) { >> > ? ? ? ? ? ? error 405 "Not allowed."; >> > ? ? ? ? } >> > ? ? ? ? return(lookup); >> > ? ? } >> > >> > ? ? if (req.request == "GET" && req.url == "/monitor") { >> > ? ? ? ? error 200 "OK."; >> > ? ? } >> > >> > ? ? if (req.backend.healthy) { >> > ? ? ? ? set req.grace = 30s; >> > ? } else { >> > ? ? ? ? set req.grace = 48h; >> > ? ? } >> > >> > } >> > >> > ?sub vcl_fetch { >> > ? ? if (beresp.status == 404 || beresp.status >= 300) { >> > ? ? ? ? ?return (pass); >> > ? ? } >> > ? ? set beresp.grace = 48h; >> > >> > ?} >> > >> > sub vcl_hit { >> > ? ? if (req.request == "PURGE") { >> > ? ? ? ? set obj.ttl = 0s; >> > ? ? ? ? error 200 "Purged."; >> > ? ? } >> > } >> > >> > sub vcl_miss { >> > ? ? if (req.request == "PURGE") { >> > ? ? ? ? error 404 "Not in cache."; >> > ? ? } >> > } >> > ---EOF--- >> > >> > >> > _______________________________________________ >> > varnish-misc mailing list >> > varnish-misc at varnish-cache.org >> > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >> > >> > _______________________________________________ >> > varnish-misc mailing list >> > varnish-misc at varnish-cache.org >> > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc >> > >> >> >> > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > From kbrownfield at google.com Sat Oct 30 02:31:43 2010 From: kbrownfield at google.com (Ken Brownfield) Date: Fri, 29 Oct 2010 17:31:43 -0700 Subject: varnish 2.1.4 swap and increase in objects:heads. In-Reply-To: References: <1288352241.5722.134.camel@oasis> <29097274.10.1288353418406.JavaMail.rbenzaquen@rbenzaquen4310> <1288369190.18624.14.camel@oasis> Message-ID: These sysctl values helped a lot for me in a 64-bit / 64GB / 1.5TB SSD / "-s file" configuration, pretty much stabilizing the box. vm.swappiness = 10 vm.min_free_kbytes = 131072 And don't forget to reduce thread_pool_stack to 256KB-512KB. FWIW, -- kb On Fri, Oct 29, 2010 at 14:00, David Birdsong wrote: > I attempted to run an -s file instance with 5 80GB SSD's raided > together. I could never get past filling it half full before the IO > load would consume the box. > > On Fri, Oct 29, 2010 at 9:19 AM, Taylan Develioglu > wrote: > > More of a general question: > > > > Is there anyone that uses varnish with a storage size of 800GB+ (single > > instance, file or malloc) ? > > > > I'm just wondering because we've had so much problems with varnish > > (2.1.2 to 2.1.4) that I'm starting to think that we're an edge case. > > > > On Fri, 2010-10-29 at 14:26 +0200, Per Buer wrote: > >> Hi. > >> > >> The issues are actually a bit different. One uses "-s file" which > >> really makes the kernel handle the memory management implicitly in > >> comparison to you Rodrigo, which uses -s malloc, where you clearly > >> state that Varnish should allocate this much memory. > >> > >> Regarding -s malloc we recommend that on Linux one should not use more > >> then 75% of the physical memory for -s malloc. In addition to the > >> actual object store Varnish uses quite some memory for it's internal > >> data structures. If you go above 75% chances are that Linux will start > >> swapping a bit which might have a negative effect on overall > >> performance. Tollef is getting back to you outside the list regarding > >> the issue you've raised with us. > >> > >> Per. > >> > >> On Fri, Oct 29, 2010 at 1:59 PM, Rodrigo Benzaquen > >> wrote: > >> > Hi Taylan, exactly the same is happening to us, we already open a > ticket > >> > with Varnish software. > >> > > >> > Please let me know if you find a solution. > >> > > >> > Thanks > >> > Rodrigo > >> > > >> > > >> > > >> > > >> > ------------------ > >> > > >> > Rodrigo Benzaquen > >> > @rbenzaquen > >> > > >> > > >> > > >> > ________________________________ > >> > From: "Taylan Develioglu" > >> > To: varnish-misc at varnish-cache.org > >> > Sent: Viernes, 29 de Octubre 2010 8:37:21 > >> > Subject: varnish 2.1.4 swap and increase in objects:heads. > >> > > >> > Debian Lenny, Linux varnish0 2.6.33.7 #1 SMP Thu Sep 2 15:27:53 CEST > >> > 2010 x86_64 GNU/Linux > >> > > >> > I have a strange situation on one of our varnish servers. > >> > > >> > We recently raised the expiry on previously uncached content a > >> > (Cache-Control: max-age=0, to Cache-Control: max-age=5). > >> > > >> > The number of objects vs. heads increased and the system started > >> > swapping (vm.swappiness=0). > >> > > >> > http://lcontent.ebuddy.com/varnish/memory-week.png > >> > http://lcontent.ebuddy.com/varnish/varnish_objects-week.png > >> > http://lcontent.ebuddy.com/varnish/varnish_expunge-week.png > >> > > >> > > >> > I don't understand why this is happening, bug maybe ? > >> > > >> > DAEMON_OPTS="-a :27535 \ > >> > -T localhost:6082 \ > >> > -f /etc/varnish/default.vcl \ > >> > -s file,/var/cache/varnish/sda/varnish_storage.bin,135G \ > >> > -s file,/var/cache/varnish/sdb/varnish_storage.bin,135G \ > >> > -s file,/var/cache/varnish/sdc/varnish_storage.bin,135G \ > >> > -s file,/var/cache/varnish/sdd/varnish_storage.bin,135G \ > >> > -s file,/var/cache/varnish/sde/varnish_storage.bin,135G \ > >> > -s file,/var/cache/varnish/sdf/varnish_storage.bin,135G \ > >> > -p thread_pools=4 \ > >> > -p thread_pool_min=200 \ > >> > -p thread_pool_max=2400 \ > >> > -p thread_pool_add_delay=2 \ > >> > -p lru_interval=20" > >> > > >> > > >> > > >> > ---vcl--- > >> > sub vcl_recv { > >> > unset req.http.cookie; > >> > unset req.http.Authorization; > >> > > >> > if (req.http.host ~ "^.*host.domain.com.*") { > >> > set req.http.host = "host.domain.com"; > >> > } > >> > > >> > if (req.request == "HEAD") { > >> > return(pass); > >> > } > >> > > >> > if (req.request == "PURGE") { > >> > if (!client.ip ~ purge) { > >> > error 405 "Not allowed."; > >> > } > >> > return(lookup); > >> > } > >> > > >> > if (req.request == "GET" && req.url == "/monitor") { > >> > error 200 "OK."; > >> > } > >> > > >> > if (req.backend.healthy) { > >> > set req.grace = 30s; > >> > } else { > >> > set req.grace = 48h; > >> > } > >> > > >> > } > >> > > >> > sub vcl_fetch { > >> > if (beresp.status == 404 || beresp.status >= 300) { > >> > return (pass); > >> > } > >> > set beresp.grace = 48h; > >> > > >> > } > >> > > >> > sub vcl_hit { > >> > if (req.request == "PURGE") { > >> > set obj.ttl = 0s; > >> > error 200 "Purged."; > >> > } > >> > } > >> > > >> > sub vcl_miss { > >> > if (req.request == "PURGE") { > >> > error 404 "Not in cache."; > >> > } > >> > } > >> > ---EOF--- > >> > > >> > > >> > _______________________________________________ > >> > varnish-misc mailing list > >> > varnish-misc at varnish-cache.org > >> > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > >> > > >> > _______________________________________________ > >> > varnish-misc mailing list > >> > varnish-misc at varnish-cache.org > >> > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > >> > > >> > >> > >> > > > > > > > > _______________________________________________ > > varnish-misc mailing list > > varnish-misc at varnish-cache.org > > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Sat Oct 30 09:48:25 2010 From: perbu at varnish-software.com (Per Buer) Date: Sat, 30 Oct 2010 09:48:25 +0200 Subject: Dealing with large unwanted objects In-Reply-To: References:

<1288368864.18624.10.camel@oasis> Message-ID: On Fri, Oct 29, 2010 at 6:46 PM, Mark Moseley wrote: > > Yup, that's what I was afraid of. 'pipe' doesn't work in vcl_fetch and > it'd be too late anyway. Sure, the object will be stored - but if you set it to a non-cachable object in vcl_fetch the memory will be freed immediately and although you'll chuck the object a couple of seconds before you'd otherwise would overall you'll be OK. -- Per Buer,? Varnish Software Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / skype: per.buer From anders at fupp.net Sun Oct 31 01:22:42 2010 From: anders at fupp.net (Anders Nordby) Date: Sun, 31 Oct 2010 01:22:42 +0200 Subject: varnish 2.1.4 swap and increase in objects:heads. In-Reply-To: <1288369190.18624.14.camel@oasis> References: <1288352241.5722.134.camel@oasis> <29097274.10.1288353418406.JavaMail.rbenzaquen@rbenzaquen4310> <1288369190.18624.14.camel@oasis> Message-ID: <20101030232242.GA53813@fupp.net> Hi, No. I had issues with having a too large storage size, and havn't tried increasing it since. But Varnish with -s malloc in FreeBSD does seem to run OK with 60 GB storage size and having 51 GB in the swap on disk at one of our sites. I have 32 GB of physical RAM. Regads, Anders. On Fri, Oct 29, 2010 at 06:19:50PM +0200, Taylan Develioglu wrote: > More of a general question: > > Is there anyone that uses varnish with a storage size of 800GB+ (single > instance, file or malloc) ? > > I'm just wondering because we've had so much problems with varnish > (2.1.2 to 2.1.4) that I'm starting to think that we're an edge case. > > On Fri, 2010-10-29 at 14:26 +0200, Per Buer wrote: > > Hi. > > > > The issues are actually a bit different. One uses "-s file" which > > really makes the kernel handle the memory management implicitly in > > comparison to you Rodrigo, which uses -s malloc, where you clearly > > state that Varnish should allocate this much memory. > > > > Regarding -s malloc we recommend that on Linux one should not use more > > then 75% of the physical memory for -s malloc. In addition to the > > actual object store Varnish uses quite some memory for it's internal > > data structures. If you go above 75% chances are that Linux will start > > swapping a bit which might have a negative effect on overall > > performance. Tollef is getting back to you outside the list regarding > > the issue you've raised with us. > > > > Per. > > > > On Fri, Oct 29, 2010 at 1:59 PM, Rodrigo Benzaquen > > wrote: > > > Hi Taylan, exactly the same is happening to us, we already open a ticket > > > with Varnish software. > > > > > > Please let me know if you find a solution. > > > > > > Thanks > > > Rodrigo > > > > > > > > > > > > > > > ------------------ > > > > > > Rodrigo Benzaquen > > > @rbenzaquen > > > > > > > > > > > > ________________________________ > > > From: "Taylan Develioglu" > > > To: varnish-misc at varnish-cache.org > > > Sent: Viernes, 29 de Octubre 2010 8:37:21 > > > Subject: varnish 2.1.4 swap and increase in objects:heads. > > > > > > Debian Lenny, Linux varnish0 2.6.33.7 #1 SMP Thu Sep 2 15:27:53 CEST > > > 2010 x86_64 GNU/Linux > > > > > > I have a strange situation on one of our varnish servers. > > > > > > We recently raised the expiry on previously uncached content a > > > (Cache-Control: max-age=0, to Cache-Control: max-age=5). > > > > > > The number of objects vs. heads increased and the system started > > > swapping (vm.swappiness=0). > > > > > > http://lcontent.ebuddy.com/varnish/memory-week.png > > > http://lcontent.ebuddy.com/varnish/varnish_objects-week.png > > > http://lcontent.ebuddy.com/varnish/varnish_expunge-week.png > > > > > > > > > I don't understand why this is happening, bug maybe ? > > > > > > DAEMON_OPTS="-a :27535 \ > > > -T localhost:6082 \ > > > -f /etc/varnish/default.vcl \ > > > -s file,/var/cache/varnish/sda/varnish_storage.bin,135G \ > > > -s file,/var/cache/varnish/sdb/varnish_storage.bin,135G \ > > > -s file,/var/cache/varnish/sdc/varnish_storage.bin,135G \ > > > -s file,/var/cache/varnish/sdd/varnish_storage.bin,135G \ > > > -s file,/var/cache/varnish/sde/varnish_storage.bin,135G \ > > > -s file,/var/cache/varnish/sdf/varnish_storage.bin,135G \ > > > -p thread_pools=4 \ > > > -p thread_pool_min=200 \ > > > -p thread_pool_max=2400 \ > > > -p thread_pool_add_delay=2 \ > > > -p lru_interval=20" > > > > > > > > > > > > ---vcl--- > > > sub vcl_recv { > > > unset req.http.cookie; > > > unset req.http.Authorization; > > > > > > if (req.http.host ~ "^.*host.domain.com.*") { > > > set req.http.host = "host.domain.com"; > > > } > > > > > > if (req.request == "HEAD") { > > > return(pass); > > > } > > > > > > if (req.request == "PURGE") { > > > if (!client.ip ~ purge) { > > > error 405 "Not allowed."; > > > } > > > return(lookup); > > > } > > > > > > if (req.request == "GET" && req.url == "/monitor") { > > > error 200 "OK."; > > > } > > > > > > if (req.backend.healthy) { > > > set req.grace = 30s; > > > } else { > > > set req.grace = 48h; > > > } > > > > > > } > > > > > > sub vcl_fetch { > > > if (beresp.status == 404 || beresp.status >= 300) { > > > return (pass); > > > } > > > set beresp.grace = 48h; > > > > > > } > > > > > > sub vcl_hit { > > > if (req.request == "PURGE") { > > > set obj.ttl = 0s; > > > error 200 "Purged."; > > > } > > > } > > > > > > sub vcl_miss { > > > if (req.request == "PURGE") { > > > error 404 "Not in cache."; > > > } > > > } > > > ---EOF--- > > > > > > > > > _______________________________________________ > > > varnish-misc mailing list > > > varnish-misc at varnish-cache.org > > > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > > > > > > _______________________________________________ > > > varnish-misc mailing list > > > varnish-misc at varnish-cache.org > > > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > > > > > > > > > > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc -- Anders. From david.donchez at smartjog.com Thu Oct 28 18:24:50 2010 From: david.donchez at smartjog.com (David Donchez) Date: Thu, 28 Oct 2010 18:24:50 +0200 Subject: How to know the current cache size using VCL Message-ID: <4CC9A3D2.80304@smartjog.com> Hi everybody, I'm setting up a cache system using Varnish and i have a question about VCL. I want to know what's happen when the cache is full. Is the first object cached be deleted ? I want to delete in first, the file with the less hit from the cache (if the caching going to be full). Is there a solution using VCL ? or something ? Thank a lot. From eli at fiercemarkets.com Fri Oct 1 00:09:06 2010 From: eli at fiercemarkets.com (Eli Dickinson) Date: Thu, 30 Sep 2010 18:09:06 -0400 Subject: Strange problem with truncated responses when POSTing data Message-ID: Greetings, First, thanks for Varnish! It's a great piece of software. Today, though, I've been pulling my hair out with a strange problem and I'm hoping someone here can help. I've got Varnish sitting in front of Apache on one server and I've got a PHP script that POSTs data from another server. If the length of the content I'm posting is over about 1500 bytes, I get an incomplete response from Varnish. Like, literally I only get half the page back. According to Wireshark, it looks like I get 1 or 2 packets of data (it's not consistent) and then there's suddently a RST/ACK. Weirder still, an almost identical request from cURL works fine. So it's either some kind of funny timing issue or it's something very specific to the way PHP makes HTTP requests. The problem disappears if: - I tell varnish to pipe instead of pass -?I run the PHP script on the same server and have it POST to localhost (which makes me think timing is a factor...) - I POST less than 1kb of data I'm running varnish-2.1.3 SVN and I don't see anything unusual in the syslog or when running varnishlog (though I'm not exactly sure what I'm looking for) Here's what my request looks like: POST / HTTP/1.0 Host: web2.servicecenter.fiercemarkets.com Content-Length: 1517 ??XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX ??XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX ??XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX ??XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX [...snip...] And the response I get back looks like: HTTP/1.1 403 Forbidden Server: Apache/2.2.3 (Red Hat) X-Powered-By: PHP/5.2.14 Expires: Sun, 11 Mar 1984 12:00:00 GMT Last-Modified: Thu, 30 Sep 2010 21:40:44 +0000 Cache-Control: no-cache, must-revalidate, post-check=0, pre-check=0 ETag: "1285882844" Content-Type: text/html; charset=utf-8 Content-Length: 4104 Date: Thu, 30 Sep 2010 21:40:44 GMT X-Varnish: 598880740 Age: 0 Via: 1.1 varnish Connection: close ?? [...snip...] The response looks fine (content-length is correct), aside from the fact that I don't get all of it. I'm stumped. Any thoughts? -- Eli Dickinson FierceMarkets, Inc. 202-824-5058 From tfheen at varnish-software.com Fri Oct 1 07:37:45 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Fri, 01 Oct 2010 07:37:45 +0200 Subject: Panic Signal 6. In-Reply-To: <804FB091-853B-4D6C-BD56-99743186472A@competitorgroup.com> (Ben Nowacky's message of "Thu, 30 Sep 2010 12:10:21 -0700") References: <9C0B05C0-A205-4E41-B906-034F9B4EC2A4@competitorgroup.com> <7F0AA702B8A85A4A967C4C8EBAD6902C502366@TMG-EVS02.torstar.net> <1FF67D7369ED1A45832180C7C1109BCA264AAC8BCB@tmmail0.trademe.local> <804FB091-853B-4D6C-BD56-99743186472A@competitorgroup.com> Message-ID: <87ocbe8od2.fsf@qurzaw.linpro.no> ]] Ben Nowacky | -p sess_workspace=8192 Why are you limiting your session workspace to a measly 8k? I'd recommend letting this stay at its default value of 64k or increase it beyond 64k, not decrease it. -- Tollef Fog Heen Varnish Software t: +47 21 54 41 73 From tfheen at varnish-software.com Fri Oct 1 09:09:59 2010 From: tfheen at varnish-software.com (Tollef Fog Heen) Date: Fri, 01 Oct 2010 09:09:59 +0200 Subject: "now" in VCL In-Reply-To: <4C7BA0C2.1000508@mapsolute.com> (Frank Gruellich's message of "Mon, 30 Aug 2010 14:14:58 +0200") References: <4C6D2823.6060604@mapsolute.com> <87aao44nxc.fsf@qurzaw.linpro.no> <4C7BA0C2.1000508@mapsolute.com> Message-ID: <87d3ru8k3c.fsf@qurzaw.linpro.no> ]] Frank Gruellich | I'm still getting a similar error message: | | Message from VCC-compiler: | Expected ';' got 'now' | (program line 491), at | (input Line 123 Pos 22) | log "MISS: " now req.url; | ---------------------###--------- | Running VCC-compiler failed, exit 1 | | "now" seems to be gone... It seems like I forgot to merge r4788 into 2.1. Doing so now, and it should be in 2.1.4. -- Tollef Fog Heen Varnish Software t: +47 21 54 41 73 From perbu at varnish-software.com Fri Oct 1 09:12:11 2010 From: perbu at varnish-software.com (Per Buer) Date: Fri, 1 Oct 2010 09:12:11 +0200 Subject: Strange problem with truncated responses when POSTing data In-Reply-To: References: Message-ID: On Fri, Oct 1, 2010 at 12:09 AM, Eli Dickinson wrote: > Today, though, I've been pulling my hair out with a strange problem > and I'm hoping someone here can help. I've got Varnish sitting in > front of Apache on one server and I've got a PHP script that POSTs > data from another server. If the length of the content I'm posting is > over about 1500 bytes, I get an incomplete response from Varnish. > This _might_ be a networking problem. I would guess that somehow you are using 802.1q tags and that some silly piece of network equipment throws away or mangles packets on one of your servers. -- Per Buer, Varnish Software Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / skype: per.buer -------------- next part -------------- An HTML attachment was scrubbed... URL: From tungcnw at gmail.com Fri Oct 1 12:01:56 2010 From: tungcnw at gmail.com (Dang Tung) Date: Fri, 1 Oct 2010 17:01:56 +0700 Subject: Fixing backend URLs In-Reply-To: <20100930050746.GA5024@grum.quex.org> References: <2006354608710103182@unknownmsgid> <20100915032231.GC497@grum.quex.org> <0FAEA775-D78B-4CD6-9C0C-972ADE57287F@gmail.com> <20100924043515.GA30650@grum.quex.org> <20100924051235.GA31378@grum.quex.org> <20100930050746.GA5024@grum.quex.org> Message-ID: Thanks, I will try, and let you know later :). On Thu, Sep 30, 2010 at 12:07 PM, Michael Alger wrote: > On Thu, Sep 30, 2010 at 11:52:11AM +0700, Dang Tung wrote: > > Are there any solutions for my issue, please help, I'm still > > waiting your reply. > > I don't really have any other ideas; the code should work if it's > run, so I think you need to try to determine why the code isn't > being run. You can use varnishlog to monitor the request flow and > see if you can work it out that way. Or, you can add some debugging > statements to log messages or add headers to the response at various > points in your VCL so you can determine exactly how far Varnish is > getting, and then from that work out why it's not running the code > in question. > > You might also want to see if it's possible to adjust your > application's configuration so it emits the 'correct' Location > header itself; it's generally better to configure the application to > know what its external address is rather than to rewrite things, as > rewriting the Location: header won't do anything to fix > fully-qualified URLs embedded in the HTML. > > Alternatively if you're willing to attach your entire Varnish > configuration we might be able to work it out from that. > > > > On Mon, Sep 27, 2010 at 11:26 AM, Dang Tung wrote: > > > > > here is the header when I used wget: > > > > > > backend: wget -S -O /dev/null http://myaddress/admin > > > > > > wget -S -O /dev/null http://myaddress/admin > > > --2010-09-27 11:18:35-- http://myaddress/admin > > > Connecting to myaddress:80... connected. > > > HTTP request sent, awaiting response... > > > HTTP/1.1 302 Found > > > Server: Apache/2.2.3 (CentOS) > > > X-Powered-By: PHP/5.2.10 > > > Set-Cookie: symfony=lli9m9oog3flp23ggchhbt5s71; path=/ > > > Location: http://myaddress:8080/admin.php > > > Cache-Control: max-age=0 > > > Expires: Mon, 27 Sep 2010 04:20:08 GMT > > > Content-Type: text/html; charset=utf-8 > > > Content-Length: 106 > > > Date: Mon, 27 Sep 2010 04:18:48 GMT > > > X-Varnish: 153748006 > > > Age: 0 > > > Via: 1.1 varnish > > > Connection: keep-alive > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > http://lists.varnish-cache.org/mailman/listinfo/varnish-misc > -- Nguyen Dang Tung -------------- next part -------------- An HTML attachment was scrubbed... URL: From kristian at varnish-software.com Fri Oct 1 13:13:55 2010 From: kristian at varnish-software.com (Kristian Lyngstol) Date: Fri, 1 Oct 2010 13:13:55 +0200 Subject: Strange problem with truncated responses when POSTing data In-Reply-To: References: