From john at alcoholfree.co.uk Sat Apr 2 21:19:13 2016 From: john at alcoholfree.co.uk (John Risby) Date: Sat, 2 Apr 2016 21:19:13 +0200 Subject: Varnish not purging via Magento 2 - throwing 403 Forbidden error - help please Message-ID: Hi all, I've set up Magento 2 (2.0.4 CE) and Varnish . Varnish seems to be working ok - pages are being cached and served from the cache (first page load takes a few seconds, next load takes 100 or so ms). The problem I have is trying to set up Magento to purge the Varnish cache when I flush the magento cache in the magento backend. I've set it up as described in the official magento docs using bin/magento setup:config:set --http-cache-hosts=127.0.0.1 My /etc/varnish/default.cfg file has the purge acl set to 127.0.0.1 (I've also tried the external IP and localhost too). But when I hit Flush Magento Cache in admin, whilst running varnishlog, the log file says (amongst other things) - note, I've replaced my domain with {MY_DOMAIN} and my server IP with {SERVER_IP} * << BeReq >> 32924 - Begin bereq 32923 pass - Timestamp Start: 1459552787.509708 0.000000 0.000000 - BereqMethod POST - BereqURL /app/etc/config.php - BereqProtocol HTTP/1.1 - BereqHeader Host: {MY_DOMAIN} - BereqHeader Accept: */* - BereqHeader Content-Length: 0 - BereqHeader Content-Type: application/x-www-form-urlencoded - BereqHeader X-Forwarded-For: {SERVER_IP} - BereqHeader X-Varnish: 32924 - VCL_call BACKEND_FETCH - VCL_return fetch - BackendOpen 23 boot.default 127.0.0.1 8080 127.0.0.1 53519 - Timestamp Bereq: 1459552787.509933 0.000225 0.000225 - Timestamp Beresp: 1459552787.511332 0.001625 0.001399 - BerespProtocol HTTP/1.1 - BerespStatus 403 - BerespReason Forbidden - BerespHeader Date: Fri, 01 Apr 2016 23:19:47 GMT - BerespHeader Server: Apache/2.4.7 (Ubuntu) - BerespHeader Content-Length: 299 - BerespHeader Content-Type: text/html; charset=iso-8859-1 - TTL RFC -1 10 -1 1459552788 1459552788 1459552787 0 0 - VCL_call BACKEND_RESPONSE - TTL VCL 0 10 0 1459552788 - VCL_return deliver - BerespUnset Content-Length: 299 - BerespHeader Content-Encoding: gzip - BerespHeader Vary: Accept-Encoding - Storage malloc Transient - ObjProtocol HTTP/1.1 - ObjStatus 403 - ObjReason Forbidden - ObjHeader Date: Fri, 01 Apr 2016 23:19:47 GMT - ObjHeader Server: Apache/2.4.7 (Ubuntu) - ObjHeader Content-Type: text/html; charset=iso-8859-1 - ObjHeader Content-Encoding: gzip - ObjHeader Vary: Accept-Encoding - Fetch_Body 3 length - - Gzip G F E 299 259 80 1992 2002 - BackendReuse 23 boot.default - Timestamp BerespBody: 1459552787.511812 0.002104 0.000479 - Length 259 - BereqAcct 191 0 191 160 0 160 - End So Varnish is correctly being asked to purge via the magento backend, but it's throwing a 403 error. It works via telnet both via 127.0.0.1 and the real IP of the server telnet 127.0.0.1 80 Trying 127.0.0.1... Connected to 127.0.0.1. Escape character is '^]'. PURGE / HTTP/1.1 X-Magento-Tags-Pattern: .* HTTP/1.1 200 Purged Date: Fri, 01 Apr 2016 23:39:08 GMT Server: Varnish X-Varnish: 229397 Content-Type: text/html; charset=utf-8 Retry-After: 5 Content-Length: 241 Accept-Ranges: bytes Connection: keep-alive 200 Purged

Error 200 Purged

Purged

Guru Meditation:

XID: 229397

Varnish cache server

Connection closed by foreign host. Any ideas please? Thanks John -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Sun Apr 3 19:47:54 2016 From: perbu at varnish-software.com (Per Buer) Date: Sun, 3 Apr 2016 19:47:54 +0200 Subject: Varnish not purging via Magento 2 - throwing 403 Forbidden error - help please In-Reply-To: References: Message-ID: Hi, On Sat, Apr 2, 2016 at 9:19 PM, John Risby wrote: > (..) > But when I hit Flush Magento Cache in admin, whilst running varnishlog, > the log file says (amongst other things) - note, I've replaced my domain > with {MY_DOMAIN} and my server IP with {SERVER_IP} > > * << BeReq >> 32924 > - Begin bereq 32923 pass > - Timestamp Start: 1459552787.509708 0.000000 0.000000 > - BereqMethod POST > - BereqURL /app/etc/config.php > - BereqProtocol HTTP/1.1 > You (Magento) are POSTing to /app/etc/config.php. You're not purging anything here. -- *Per Buer* CTO | Varnish Software AS Cell: +47 95839117 We Make Websites Fly! www.varnish-software.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From john at alcoholfree.co.uk Sun Apr 3 20:56:31 2016 From: john at alcoholfree.co.uk (John Risby) Date: Sun, 3 Apr 2016 20:56:31 +0200 Subject: Varnish not purging via Magento 2 - throwing 403 Forbidden error - help please In-Reply-To: References: Message-ID: Hi Per, Thanks for the reply. I'm sorry, I missed off the first part of the log which is * << Request >> 2 - Begin req 1 rxreq - Timestamp Start: 1459708682.774223 0.000000 0.000000 - Timestamp Req: 1459708682.774223 0.000000 0.000000 - HttpGarbage "PURGE%00" - ReqAcct 47 0 47 28 0 28 - End * << Session >> 1 - Begin sess 0 HTTP/1 - SessOpen 127.0.0.1 54199 :80 127.0.0.1 80 1459708682.774100 18 - Link req 2 rxreq - SessClose RX_JUNK 0.000 - End I realised today that it was then accessing (or trying to) /app/etc/config.php and that is what's throwing the 403. I know this is more a Magento 2 question than a Varnish question but no one has replied on the Magento forum. So I guess there's two questions from this now. 1) What does the (newly posted) log data say is happening? It seems to suggest there is a purge request but the cache doesn't seem to clear. 2) It's then trying to access /app/et/config.php but failing and returning 403. My vcl is generated from the Magento admin backend so should be right and the admin page where I'm flushing the cache is being accessed from https anyway, so shouldn't that mean it is totally ignored? This is a test server, so no one else is accessing the site and this request is consistent every time I hit Flush Magento Cache on the backend. Any idea why it is trying/failing to access this file? Thanks John On 3 April 2016 at 19:47, Per Buer wrote: > Hi, > > > > On Sat, Apr 2, 2016 at 9:19 PM, John Risby wrote: > >> (..) >> But when I hit Flush Magento Cache in admin, whilst running varnishlog, >> the log file says (amongst other things) - note, I've replaced my domain >> with {MY_DOMAIN} and my server IP with {SERVER_IP} >> >> * << BeReq >> 32924 >> - Begin bereq 32923 pass >> - Timestamp Start: 1459552787.509708 0.000000 0.000000 >> - BereqMethod POST >> - BereqURL /app/etc/config.php >> - BereqProtocol HTTP/1.1 >> > > You (Magento) are POSTing to /app/etc/config.php. You're not purging > anything here. > > -- > *Per Buer* > CTO | Varnish Software AS > Cell: +47 95839117 > We Make Websites Fly! > www.varnish-software.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From perbu at varnish-software.com Mon Apr 4 09:57:16 2016 From: perbu at varnish-software.com (Per Buer) Date: Mon, 4 Apr 2016 09:57:16 +0200 Subject: Varnish not purging via Magento 2 - throwing 403 Forbidden error - help please In-Reply-To: References: Message-ID: On Sun, Apr 3, 2016 at 8:56 PM, John Risby wrote: > Hi Per, > > Thanks for the reply. > > I'm sorry, I missed off the first part of the log which is > > * << Request >> 2 > - Begin req 1 rxreq > - Timestamp Start: 1459708682.774223 0.000000 0.000000 > - Timestamp Req: 1459708682.774223 0.000000 0.000000 > - HttpGarbage "PURGE%00" > - ReqAcct 47 0 47 28 0 28 > - End > Magento does something weird here. It says PURGE and then a null character. It should be something PURGE /... HTTP/1.1. So check the magento config or the code that issues the purge. (..) > 2) It's then trying to access /app/et/config.php but failing and returning > 403. My vcl is generated from the Magento admin backend so should be right > and the admin page where I'm flushing the cache is being accessed from > https anyway, so shouldn't that mean it is totally ignored? This is a test > server, so no one else is accessing the site and this request is consistent > every time I hit Flush Magento Cache on the backend. Any idea why it is > trying/failing to access this file? > No, I don't. I don't know Magento. :-/ -- *Per Buer* CTO | Varnish Software AS Cell: +47 95839117 We Make Websites Fly! www.varnish-software.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From john at alcoholfree.co.uk Mon Apr 4 09:59:51 2016 From: john at alcoholfree.co.uk (John Risby) Date: Mon, 4 Apr 2016 09:59:51 +0200 Subject: Varnish not purging via Magento 2 - throwing 403 Forbidden error - help please In-Reply-To: References: Message-ID: Great, thanks Per. That's helped narrow it down. I'll check the Magento code and see what it is doing. Should hopefully be simple to fix. Don't worry about the second bit, as long as it purges the cache, that's the main thing. Thank you John On 4 April 2016 at 09:57, Per Buer wrote: > > > On Sun, Apr 3, 2016 at 8:56 PM, John Risby wrote: > >> Hi Per, >> >> Thanks for the reply. >> >> I'm sorry, I missed off the first part of the log which is >> >> * << Request >> 2 >> - Begin req 1 rxreq >> - Timestamp Start: 1459708682.774223 0.000000 0.000000 >> - Timestamp Req: 1459708682.774223 0.000000 0.000000 >> - HttpGarbage "PURGE%00" >> - ReqAcct 47 0 47 28 0 28 >> - End >> > > > Magento does something weird here. It says PURGE and then a null > character. It should be something PURGE /... HTTP/1.1. > > So check the magento config or the code that issues the purge. > > (..) >> 2) It's then trying to access /app/et/config.php but failing and >> returning 403. My vcl is generated from the Magento admin backend so should >> be right and the admin page where I'm flushing the cache is being accessed >> from https anyway, so shouldn't that mean it is totally ignored? This is a >> test server, so no one else is accessing the site and this request is >> consistent every time I hit Flush Magento Cache on the backend. Any idea >> why it is trying/failing to access this file? >> > > No, I don't. I don't know Magento. :-/ > > > -- > *Per Buer* > CTO | Varnish Software AS > Cell: +47 95839117 > We Make Websites Fly! > www.varnish-software.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From john at alcoholfree.co.uk Tue Apr 5 20:21:00 2016 From: john at alcoholfree.co.uk (John Risby) Date: Tue, 5 Apr 2016 20:21:00 +0200 Subject: Varnish not purging via Magento 2 - throwing 403 Forbidden error - help please In-Reply-To: References: Message-ID: Ok, I've found a workaround to this problem on another forum. For anyone else with the same problem, see the post by davidalger at https://www.bountysource.com/issues/29113272-no-varnish-purge-request-from-nginx This fixes it. Basically the server path isn't being set properly and this code fixes it. Hopefully (presumably?) it will be added to the core codebase a some point. That solves the main problem of the varnish cache not clearing from the Magento 2 admin, but I'm still unsure why /app/etc/config.php is being requested after that. So basically it's a Magento problem, not a Varnish problem. Thanks for your help. Thanks On 4 April 2016 at 09:59, John Risby wrote: > Great, thanks Per. That's helped narrow it down. I'll check the Magento > code and see what it is doing. Should hopefully be simple to fix. > > Don't worry about the second bit, as long as it purges the cache, that's > the main thing. > > Thank you > John > > > On 4 April 2016 at 09:57, Per Buer wrote: > >> >> >> On Sun, Apr 3, 2016 at 8:56 PM, John Risby >> wrote: >> >>> Hi Per, >>> >>> Thanks for the reply. >>> >>> I'm sorry, I missed off the first part of the log which is >>> >>> * << Request >> 2 >>> - Begin req 1 rxreq >>> - Timestamp Start: 1459708682.774223 0.000000 0.000000 >>> - Timestamp Req: 1459708682.774223 0.000000 0.000000 >>> - HttpGarbage "PURGE%00" >>> - ReqAcct 47 0 47 28 0 28 >>> - End >>> >> >> >> Magento does something weird here. It says PURGE and then a null >> character. It should be something PURGE /... HTTP/1.1. >> >> So check the magento config or the code that issues the purge. >> >> (..) >>> 2) It's then trying to access /app/et/config.php but failing and >>> returning 403. My vcl is generated from the Magento admin backend so should >>> be right and the admin page where I'm flushing the cache is being accessed >>> from https anyway, so shouldn't that mean it is totally ignored? This is a >>> test server, so no one else is accessing the site and this request is >>> consistent every time I hit Flush Magento Cache on the backend. Any idea >>> why it is trying/failing to access this file? >>> >> >> No, I don't. I don't know Magento. :-/ >> >> >> -- >> *Per Buer* >> CTO | Varnish Software AS >> Cell: +47 95839117 >> We Make Websites Fly! >> www.varnish-software.com >> >> > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From John.Salmon at DEShawResearch.com Wed Apr 6 14:21:42 2016 From: John.Salmon at DEShawResearch.com (John Salmon) Date: Wed, 6 Apr 2016 08:21:42 -0400 Subject: Fwd: Varnish returns truncated content In-Reply-To: <5703CDF2.6000600@DEShawResearch.com> References: <5703CDF2.6000600@DEShawResearch.com> Message-ID: <5704FF56.3010707@DEShawResearch.com> I'm running Varnish 4.1.1 on CentOS6. When I retrieve a resource through Varnish using curl, I get a 'transfer closed with 29700 bytes remaining to read'. I'm running with the default.vcl that came with the rpm, except that I changed the backend default.host to point to my origin server. Details of the transaction are below. The most suspicious, to me, are the 'Fetch error - Could not get storage' and the long series of ExpKill that precedes it. But even if no space is available for caching, Varnish shouldn't return truncated content to the client, should it? Is this a Varnish bug? Any advice on how to diagnose this further would be greatly appreciated. Thanks, John Salmon Here is the output from curl -vvv. drdws0047$ curl -H X-Fs123-Protocol:3 -H X-Fs123-Cache-Flush-Trigger:1 -H X-Fs123-Chunk:8388608 at 0 -vvv http://127.0.0.1:82/fs123/m5/03-fs123.3.fcgi/inchi/1.04-05/lib/libinchi.so.1.04.00| wc * About to connect() to 127.0.0.1 port 82 (#0) * Trying 127.0.0.1... connected * Connected to 127.0.0.1 (127.0.0.1) port 82 (#0) > GET /fs123/m5/03-fs123.3.fcgi/inchi/1.04-05/lib/libinchi.so.1.04.00 HTTP/1.1 > User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2 > Host: 127.0.0.1:82 > Accept: */* > X-Fs123-Protocol:3 > X-Fs123-Cache-Flush-Trigger:1 > X-Fs123-Chunk:8388608 at 0 > % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0< HTTP/1.1 200 OK < Date: Tue, 05 Apr 2016 13:49:55 GMT < Server: Apache/2.2.15 (CentOS) < Vary: X-Fs123-Cache-Flush-Trigger,X-Fs123-Protocol,X-Fs123-Chunk < X-Fs123-Protocol: 3 < X-Fs123-Cache-Flush-Trigger: 1 < Cache-Control: public,max-age=1000000 < Last-Modified: Thu, 09 Jan 2014 18:33:57 GMT < X-Fs123-Attrs: 0 33133 1 10085 10 1126399 1389292437 1429799855 1459863979 27355334 0 221649590 888201093 32 < X-Fs123-Content-Chunk: 1126399 at 0 < Content-Length: 1126399 < Content-Type: application/octet-stream < X-Varnish: 75556 < Age: 0 < Via: 1.1 varnish-v4 < Accept-Ranges: bytes < Connection: keep-alive < { [data not shown] * transfer closed with 29700 bytes remaining to read 97 1099k 97 1070k 0 0 7150k 0 --:--:-- --:--:-- --:--:-- 7285k* Closing connection #0 curl: (18) transfer closed with 29700 bytes remaining to read 3568 17393 1096699 drdws0047$ Here are the relevant transactions from varnishlog: * << BeReq >> 75557 - Begin bereq 75556 fetch - Timestamp Start: 1459864195.876974 0.000000 0.000000 - BereqMethod GET - BereqURL /fs123/m5/03-fs123.3.fcgi/inchi/1.04-05/lib/libinchi.so.1.04.00 - BereqProtocol HTTP/1.1 - BereqHeader User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2 - BereqHeader Host: 127.0.0.1:82 - BereqHeader Accept: */* - BereqHeader X-Fs123-Protocol:3 - BereqHeader X-Fs123-Cache-Flush-Trigger:1 - BereqHeader X-Fs123-Chunk:8388608 at 0 - BereqHeader X-Forwarded-For: 127.0.0.1 - BereqHeader Accept-Encoding: gzip - BereqHeader X-Varnish: 75557 - VCL_call BACKEND_FETCH - VCL_return fetch - BackendOpen 15 reload_2016-04-04T18:49:45.default 149.77.52.249 80 149.77.52.153 39650 - Timestamp Bereq: 1459864195.877410 0.000435 0.000435 - Timestamp Beresp: 1459864195.995047 0.118073 0.117638 - BerespProtocol HTTP/1.1 - BerespStatus 200 - BerespReason OK - BerespHeader Date: Tue, 05 Apr 2016 13:49:55 GMT - BerespHeader Server: Apache/2.2.15 (CentOS) - BerespHeader Vary: X-Fs123-Cache-Flush-Trigger,X-Fs123-Protocol,X-Fs123-Chunk - BerespHeader X-Fs123-Protocol: 3 - BerespHeader X-Fs123-Cache-Flush-Trigger: 1 - BerespHeader Cache-Control: public,max-age=1000000 - BerespHeader Last-Modified: Thu, 09 Jan 2014 18:33:57 GMT - BerespHeader X-Fs123-Attrs: 0 33133 1 10085 10 1126399 1389292437 1429799855 1459863979 27355334 0 221649590 888201093 32 - BerespHeader X-Fs123-Content-Chunk: 1126399 at 0 - BerespHeader Content-Length: 1126399 - BerespHeader Connection: close - BerespHeader Content-Type: application/octet-stream - TTL RFC 1000000 10 -1 1459864196 1459864196 1459864195 0 1000000 - VCL_call BACKEND_RESPONSE - VCL_return deliver - Storage malloc s0 - ObjProtocol HTTP/1.1 - ObjStatus 200 - ObjReason OK - ObjHeader Date: Tue, 05 Apr 2016 13:49:55 GMT - ObjHeader Server: Apache/2.2.15 (CentOS) - ObjHeader Vary: X-Fs123-Cache-Flush-Trigger,X-Fs123-Protocol,X-Fs123-Chunk - ObjHeader X-Fs123-Protocol: 3 - ObjHeader X-Fs123-Cache-Flush-Trigger: 1 - ObjHeader Cache-Control: public,max-age=1000000 - ObjHeader Last-Modified: Thu, 09 Jan 2014 18:33:57 GMT - ObjHeader X-Fs123-Attrs: 0 33133 1 10085 10 1126399 1389292437 1429799855 1459863979 27355334 0 221649590 888201093 32 - ObjHeader X-Fs123-Content-Chunk: 1126399 at 0 - ObjHeader Content-Length: 1126399 - ObjHeader Content-Type: application/octet-stream - Fetch_Body 3 length stream - ExpKill LRU_Cand p=0x2b121483cf40 f=0x0 r=1 - ExpKill LRU x=66088 - ExpKill LRU_Cand p=0x2b12148400c0 f=0x0 r=1 - ExpKill LRU x=66090 - ExpKill LRU_Cand p=0x2b1214840180 f=0x0 r=1 - ExpKill LRU x=66092 - ExpKill LRU_Cand p=0x2b1214840240 f=0x0 r=1 - ExpKill LRU x=66094 - ExpKill LRU_Cand p=0x2b1214840300 f=0x0 r=1 - ExpKill LRU x=66096 - ExpKill LRU_Cand p=0x2b12148403c0 f=0x0 r=1 - ExpKill LRU x=66098 - ExpKill LRU_Cand p=0x2b1214840480 f=0x0 r=1 - ExpKill LRU x=66100 - ExpKill LRU_Cand p=0x2b1214840540 f=0x0 r=1 - ExpKill LRU x=66102 - ExpKill LRU_Cand p=0x2b1214840780 f=0x0 r=1 - ExpKill LRU x=66110 - ExpKill LRU_Cand p=0x2b1214840840 f=0x0 r=1 - ExpKill LRU x=66115 - ExpKill LRU_Cand p=0x2b1214840900 f=0x0 r=1 - ExpKill LRU x=66117 - ExpKill LRU_Cand p=0x2b12148409c0 f=0x0 r=1 - ExpKill LRU x=66119 - ExpKill LRU_Cand p=0x2b1214840a80 f=0x0 r=1 - ExpKill LRU x=66121 - ExpKill LRU_Cand p=0x2b1214840b40 f=0x0 r=1 - ExpKill LRU x=66123 - ExpKill LRU_Cand p=0x2b1214840c00 f=0x0 r=1 - ExpKill LRU x=66125 - ExpKill LRU_Cand p=0x2b1214840cc0 f=0x0 r=1 - ExpKill LRU x=66127 - ExpKill LRU_Cand p=0x2b1214840d80 f=0x0 r=1 - ExpKill LRU x=66129 - ExpKill LRU_Cand p=0x2b1214840e40 f=0x0 r=1 - ExpKill LRU x=66131 - ExpKill LRU_Cand p=0x2b1214840f00 f=0x0 r=1 - ExpKill LRU x=66133 - ExpKill LRU_Cand p=0x2b1214840fc0 f=0x0 r=1 - ExpKill LRU x=66135 - ExpKill LRU_Cand p=0x2b1214841080 f=0x0 r=1 - ExpKill LRU x=66137 - ExpKill LRU_Cand p=0x2b1214841140 f=0x0 r=1 - ExpKill LRU x=66139 - ExpKill LRU_Cand p=0x2b1214841200 f=0x0 r=1 - ExpKill LRU x=66141 - ExpKill LRU_Cand p=0x2b12148412c0 f=0x0 r=1 - ExpKill LRU x=66143 - ExpKill LRU_Cand p=0x2b1214841380 f=0x0 r=1 - ExpKill LRU x=66145 - ExpKill LRU_Cand p=0x2b1214841440 f=0x0 r=1 - ExpKill LRU x=66147 - ExpKill LRU_Cand p=0x2b1214841500 f=0x0 r=1 - ExpKill LRU x=66149 - ExpKill LRU_Cand p=0x2b12148415c0 f=0x0 r=1 - ExpKill LRU x=66151 - ExpKill LRU_Cand p=0x2b1214841680 f=0x0 r=1 - ExpKill LRU x=66153 - ExpKill LRU_Cand p=0x2b1214841740 f=0x0 r=1 - ExpKill LRU x=66155 - ExpKill LRU_Cand p=0x2b1214841800 f=0x0 r=1 - ExpKill LRU x=66157 - ExpKill LRU_Cand p=0x2b12148418c0 f=0x0 r=1 - ExpKill LRU x=66159 - ExpKill LRU_Cand p=0x2b1214841980 f=0x0 r=1 - ExpKill LRU x=66161 - ExpKill LRU_Cand p=0x2b1214841a40 f=0x0 r=1 - ExpKill LRU x=66163 - ExpKill LRU_Cand p=0x2b1214841b00 f=0x0 r=1 - ExpKill LRU x=66165 - ExpKill LRU_Cand p=0x2b1214841bc0 f=0x0 r=1 - ExpKill LRU x=66167 - ExpKill LRU_Cand p=0x2b1214841c80 f=0x0 r=1 - ExpKill LRU x=66169 - ExpKill LRU_Cand p=0x2b1214841d40 f=0x0 r=1 - ExpKill LRU x=66171 - ExpKill LRU_Cand p=0x2b1214841e00 f=0x0 r=1 - ExpKill LRU x=66173 - ExpKill LRU_Cand p=0x2b1214841ec0 f=0x0 r=1 - ExpKill LRU x=66175 - ExpKill LRU_Cand p=0x2b1214841f80 f=0x0 r=1 - ExpKill LRU x=66177 - ExpKill LRU_Cand p=0x2b1214842040 f=0x0 r=1 - ExpKill LRU x=66179 - ExpKill LRU_Cand p=0x2b1214842100 f=0x0 r=1 - ExpKill LRU x=66181 - ExpKill LRU_Cand p=0x2b12148421c0 f=0x0 r=1 - ExpKill LRU x=66183 - ExpKill LRU_Cand p=0x2b1214842280 f=0x0 r=1 - ExpKill LRU x=66185 - ExpKill LRU_Cand p=0x2b1214842340 f=0x0 r=1 - ExpKill LRU x=66187 - ExpKill LRU_Cand p=0x2b1214842400 f=0x0 r=1 - ExpKill LRU x=66189 - ExpKill LRU_Cand p=0x2b12148424c0 f=0x0 r=1 - ExpKill LRU x=66191 - ExpKill LRU_Cand p=0x2b1214842580 f=0x0 r=1 - ExpKill LRU x=66193 - ExpKill LRU_Cand p=0x2b1214842640 f=0x0 r=1 - ExpKill LRU x=66195 - ExpKill LRU_Cand p=0x2b1214842700 f=0x0 r=1 - ExpKill LRU x=66197 - ExpKill LRU_Cand p=0x2b12148427c0 f=0x0 r=1 - ExpKill LRU x=66199 - ExpKill LRU_Cand p=0x2b1214842880 f=0x0 r=1 - ExpKill LRU x=66201 - ExpKill LRU_Cand p=0x2b1214842940 f=0x0 r=1 - ExpKill LRU x=66203 - ExpKill LRU_Cand p=0x2b1214842a00 f=0x0 r=1 - ExpKill LRU x=66205 - ExpKill LRU_Cand p=0x2b1214842ac0 f=0x0 r=1 - ExpKill LRU x=66207 - ExpKill LRU_Cand p=0x2b1214842b80 f=0x0 r=1 - ExpKill LRU x=66209 - ExpKill LRU_Cand p=0x2b1214842c40 f=0x0 r=1 - ExpKill LRU x=66211 - ExpKill LRU_Cand p=0x2b1214842d00 f=0x0 r=1 - ExpKill LRU x=66213 - ExpKill LRU_Cand p=0x2b1214842dc0 f=0x0 r=1 - ExpKill LRU x=66215 - ExpKill LRU_Cand p=0x2b1214842e80 f=0x0 r=1 - ExpKill LRU x=66217 - ExpKill LRU_Cand p=0x2b1214842f40 f=0x0 r=1 - ExpKill LRU x=66219 - ExpKill LRU_Cand p=0x2b12148470c0 f=0x0 r=1 - ExpKill LRU x=66221 - ExpKill LRU_Cand p=0x2b1214847180 f=0x0 r=1 - ExpKill LRU x=66223 - ExpKill LRU_Cand p=0x2b1214847240 f=0x0 r=1 - ExpKill LRU x=66225 - ExpKill LRU_Cand p=0x2b1214847300 f=0x0 r=1 - ExpKill LRU x=66227 - ExpKill LRU_Cand p=0x2b12148473c0 f=0x0 r=1 - ExpKill LRU x=66229 - ExpKill LRU_Cand p=0x2b1214847480 f=0x0 r=1 - ExpKill LRU x=66231 - ExpKill LRU_Cand p=0x2b1214847540 f=0x0 r=1 - ExpKill LRU x=66233 - ExpKill LRU_Cand p=0x2b1214847600 f=0x0 r=1 - ExpKill LRU x=66235 - ExpKill LRU_Cand p=0x2b12148476c0 f=0x0 r=1 - ExpKill LRU x=66237 - ExpKill LRU_Cand p=0x2b1214847780 f=0x0 r=1 - ExpKill LRU x=66239 - ExpKill LRU_Cand p=0x2b1214847840 f=0x0 r=1 - ExpKill LRU x=66241 - ExpKill LRU_Cand p=0x2b1214847900 f=0x0 r=1 - ExpKill LRU x=66243 - ExpKill LRU_Cand p=0x2b12148479c0 f=0x0 r=1 - ExpKill LRU x=66245 - ExpKill LRU_Cand p=0x2b1214847a80 f=0x0 r=1 - ExpKill LRU x=66247 - FetchError Could not get storage - BackendClose 15 reload_2016-04-04T18:49:45.default - BereqAcct 382 0 382 521 1096699 1097220 - End * << Request >> 75556 - Begin req 75555 rxreq - Timestamp Start: 1459864195.876911 0.000000 0.000000 - Timestamp Req: 1459864195.876911 0.000000 0.000000 - ReqStart 127.0.0.1 48471 - ReqMethod GET - ReqURL /fs123/m5/03-fs123.3.fcgi/inchi/1.04-05/lib/libinchi.so.1.04.00 - ReqProtocol HTTP/1.1 - ReqHeader User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2 - ReqHeader Host: 127.0.0.1:82 - ReqHeader Accept: */* - ReqHeader X-Fs123-Protocol:3 - ReqHeader X-Fs123-Cache-Flush-Trigger:1 - ReqHeader X-Fs123-Chunk:8388608 at 0 - ReqHeader X-Forwarded-For: 127.0.0.1 - VCL_call RECV - VCL_return hash - VCL_call HASH - VCL_return lookup - VCL_call MISS - VCL_return fetch - Link bereq 75557 fetch - Timestamp Fetch: 1459864195.995156 0.118245 0.118245 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Date: Tue, 05 Apr 2016 13:49:55 GMT - RespHeader Server: Apache/2.2.15 (CentOS) - RespHeader Vary: X-Fs123-Cache-Flush-Trigger,X-Fs123-Protocol,X-Fs123-Chunk - RespHeader X-Fs123-Protocol: 3 - RespHeader X-Fs123-Cache-Flush-Trigger: 1 - RespHeader Cache-Control: public,max-age=1000000 - RespHeader Last-Modified: Thu, 09 Jan 2014 18:33:57 GMT - RespHeader X-Fs123-Attrs: 0 33133 1 10085 10 1126399 1389292437 1429799855 1459863979 27355334 0 221649590 888201093 32 - RespHeader X-Fs123-Content-Chunk: 1126399 at 0 - RespHeader Content-Length: 1126399 - RespHeader Content-Type: application/octet-stream - RespHeader X-Varnish: 75556 - RespHeader Age: 0 - RespHeader Via: 1.1 varnish-v4 - VCL_call DELIVER - VCL_return deliver - Timestamp Process: 1459864195.995174 0.118263 0.000018 - RespHeader Accept-Ranges: bytes - Debug "RES_MODE 2" - RespHeader Connection: keep-alive - Timestamp Resp: 1459864196.005802 0.128891 0.010628 - ReqAcct 313 0 313 595 1096699 1097294 - End * << Session >> 75555 - Begin sess 0 HTTP/1 - SessOpen 127.0.0.1 48471 127.0.0.1:82 127.0.0.1 82 1459864195.876869 12 - Link req 75556 rxreq - SessClose REM_CLOSE 0.129 - End And here is the corresponding entry from the origin servers access_log: [root at nycovirttest8 conf.d]# grep inchi /var/log/httpd/access_log 149.77.52.153 - - [05/Apr/2016:09:49:55 -0400] "GET /fs123/m5/03-fs123.3.fcgi/inchi/1.04-05/lib/libinchi.so.1.04.00 HTTP/1.1" 200 1126399 "8388608 at 0" "-" "curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2" [root at nycovirttest8 conf.d]# The varnish I'm using is: drdws0047$ rpm -qi varnish Name : varnish Relocations: (not relocatable) Version : 4.1.1 Vendor: (none) Release : 2.el6 Build Date: Mon 01 Feb 2016 11:56:49 AM EST Install Date: Wed 10 Feb 2016 05:25:48 PM EST Build Host: nancy.varnish-software.com Group : System Environment/Daemons Source RPM: varnish-4.1.1-2.el6.src.rpm Size : 1066319 License: BSD Signature : RSA/SHA1, Mon 01 Feb 2016 12:07:15 PM EST, Key ID 60e7c096c4deffeb URL : https://www.varnish-cache.org/ Summary : High-performance HTTP accelerator -- *.* -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Wed Apr 6 17:54:41 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 6 Apr 2016 17:54:41 +0200 Subject: Varnish returns truncated content In-Reply-To: <5704FF56.3010707@DEShawResearch.com> References: <5703CDF2.6000600@DEShawResearch.com> <5704FF56.3010707@DEShawResearch.com> Message-ID: > Is this a Varnish bug? Any advice on how to diagnose this further would be > greatly > appreciated. Hi John, This is not a bug, but probably due to streaming being enabled by default. Basically the request goes through the VCL successfully, the curl client gets the headers and then the body is streamed while at the same time inserted into storage. But then you get this error: > - FetchError Could not get storage You can see lots of ExpKill records, which mean that you most likely hit the nuke_limit (see man varnishd). It means that Varnish stopped making space for the new object because for that it had to remove too many objects from the cache. It may happen more frequently when a storage is mixing large and small objects. Best Regards, Dridi From John.Salmon at DEShawResearch.com Wed Apr 6 18:41:05 2016 From: John.Salmon at DEShawResearch.com (John Salmon) Date: Wed, 6 Apr 2016 12:41:05 -0400 Subject: Varnish returns truncated content In-Reply-To: References: <5703CDF2.6000600@DEShawResearch.com> <5704FF56.3010707@DEShawResearch.com> Message-ID: <57053C21.3010104@DEShawResearch.com> On 04/06/2016 11:54 AM, Dridi Boukelmoune wrote: >> Is this a Varnish bug? Any advice on how to diagnose this further would be >> greatly >> appreciated. > Hi John, > > This is not a bug, but probably due to streaming being enabled by > default. Basically the request goes through the VCL successfully, the > curl client gets the headers and then the body is streamed while at > the same time inserted into storage. > > But then you get this error: > >> - FetchError Could not get storage > You can see lots of ExpKill records, which mean that you most likely > hit the nuke_limit (see man varnishd). It means that Varnish stopped > making space for the new object because for that it had to remove too > many objects from the cache. It may happen more frequently when a > storage is mixing large and small objects. Yes. I definitely have a mix of small O(100) byte objects and large (up to 8MB) objects. How should I set nuke_limit? 80000 would more-or-less guarantee that space can be found for a large object by nuking small ones. Are there negative consequences to setting it so high? Are there other parameters I should tune instead/first? Thanks, John Salmon > > Best Regards, > Dridi -- *.* -------------- next part -------------- An HTML attachment was scrubbed... URL: From thierry.magnien at sfr.com Wed Apr 6 21:41:09 2016 From: thierry.magnien at sfr.com (MAGNIEN, Thierry) Date: Wed, 6 Apr 2016 19:41:09 +0000 Subject: Varnish returns truncated content In-Reply-To: <57053C21.3010104@DEShawResearch.com> References: <5703CDF2.6000600@DEShawResearch.com> <5704FF56.3010707@DEShawResearch.com> , <57053C21.3010104@DEShawResearch.com> Message-ID: Hi, For this kind of setup, you can cut your storage in half and use two stevedores: one for small files and other for large ones. Use vcl to decide in which stevedore you store objects. You'll avoid nuking too many small objects for a single large one. Regards, Thierry -------- Message d'origine -------- De : John Salmon Date : 06/04/2016 20:09 (GMT+01:00) ? : Dridi Boukelmoune Cc : varnish-misc , "Moraes, Mark" Objet : Re: Varnish returns truncated content On 04/06/2016 11:54 AM, Dridi Boukelmoune wrote: Is this a Varnish bug? Any advice on how to diagnose this further would be greatly appreciated. Hi John, This is not a bug, but probably due to streaming being enabled by default. Basically the request goes through the VCL successfully, the curl client gets the headers and then the body is streamed while at the same time inserted into storage. But then you get this error: - FetchError Could not get storage You can see lots of ExpKill records, which mean that you most likely hit the nuke_limit (see man varnishd). It means that Varnish stopped making space for the new object because for that it had to remove too many objects from the cache. It may happen more frequently when a storage is mixing large and small objects. Yes. I definitely have a mix of small O(100) byte objects and large (up to 8MB) objects. How should I set nuke_limit? 80000 would more-or-less guarantee that space can be found for a large object by nuking small ones. Are there negative consequences to setting it so high? Are there other parameters I should tune instead/first? Thanks, John Salmon Best Regards, Dridi -- . -------------- next part -------------- An HTML attachment was scrubbed... URL: From yianniska at gmail.com Wed Apr 13 15:02:45 2016 From: yianniska at gmail.com (Yiannis Karayiannidis) Date: Wed, 13 Apr 2016 16:02:45 +0300 Subject: Multiple directors Message-ID: Hi all, Varnish has made an incredible difference to our website, for the first we 're trying to use multiple directors. We're wondering if we got the correct conf syntax for using multiple directors. We've a section in our conf like the below. ############################## vcl 4.0; import std; import directors; probe prb_www_xxx_spo { .request = "HEAD http://xxx.xxx.com/ HTTP/1.1" "Host: 127.0.0.2" "User-Agent: varnish-probe" "Connection: close"; .timeout = 10s; .interval = 4s; .window = 10; .threshold = 8; } ######## probe prb_yyy_btn_spo { .request = "HEAD http://yyy.yyy.com/ HTTP/1.1" "Host: 127.0.0.2" "User-Agent: varnish-probe" "Connection: close"; .timeout = 10s; .interval = 4s; .window = 10; .threshold = 8; } # BACKENDS FOR xxx.xxx.COM ########################################################################################################################################### backend web1_www_xxx_spo { .host = "111.111.111.121"; .port = "80"; .probe = prb_www_xxx_spo; } backend web2_www_xxx_spo { .host = "111.111.111.122"; .port = "80"; .probe = prb_www_xxx_spo; } sub vcl_init { new dir_www_xxx_spo = directors.hash(); dir_www_xxx_spo.add_backend(web1_www_xxx_spo, 1.0); dir_www_xxx_spo.add_backend(web2_www_xxx_spo, 1.0); } # BACKENDS FOR yyy.yyy.COM ###################################################################################################################################### backend web1_www_yyy_spo { .host = "111.111.111.123"; .port = "80"; .probe = prb_www_yyy_spo; } backend web2_www_yyy_spo { .host = "111.111.111.124"; .port = "80"; .probe = prb_www_yyy_spo; } sub vcl_init { new dir_www_yyy_spo = directors.hash(); dir_www_yyy_spo.add_backend(web1_www_yyy_spo, 1.0); dir_www_yyy_spo.add_backend(web2_www_yyy_spo, 1.0); } Is that ok can we have multiple *sub vcl_init ?* if not which is the correct method for having multiple directors in the vcl? Thanks in advance Yiannis -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Wed Apr 13 18:12:14 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 13 Apr 2016 18:12:14 +0200 Subject: Multiple directors In-Reply-To: References: Message-ID: > Is that ok can we have multiple sub vcl_init ? It is perfectly fine, multiple subroutines will be collapsed into a single one, in the order in which they were encountered. Cheers From a.hongens at netmatch.nl Thu Apr 14 12:34:56 2016 From: a.hongens at netmatch.nl (=?UTF-8?Q?Angelo_H=c3=b6ngens?=) Date: Thu, 14 Apr 2016 12:34:56 +0200 Subject: duplicate beresp / code not hit in backend_response? Message-ID: <570F7250.4030609@netmatch.nl> Hey guys and girls, I'm trying to debug an issue with varnish here. Running varnish-4.1.2-1.el7.x86_64. I think I have two problems. The first one, is that for debugging purposes, I want to force a hit-for-pass on objects with a beresp.status of > 300, and my vcl_backend_response looks like this: sub vcl_backend_response { if (beresp.status > 300) { set beresp.ttl = 0s; set beresp.uncacheable = true; set beresp.http.X-Debug-GT300Resp = "YES"; } if (beresp.uncacheable) { set beresp.http.X-Cacheable = "NO:Not Cacheable"; } else { set beresp.http.X-Cacheable = "YES"; if (beresp.http.Set-Cookie) { unset beresp.http.Set-Cookie; } } return (deliver); } However, varnish does not seem to hit that code, as the X-Debug-GT300Resp header never seems to be added. See the below output of varnishlog -b -g request -q 'BerespStatus > 200'. What's even more confusing, is that I first see a 'BerespStatus 304', and a bit down, I see 'BerespStatus 200'. Huh? Is varnish doing two requests to my backend? ** << BeReq >> 192316528 -- Begin bereq 192316527 bgfetch -- Timestamp Start: 1460629439.397252 0.000000 0.000000 -- BereqMethod GET -- BereqURL /L1E3873A88W480H320.jpg -- BereqProtocol HTTP/1.1 -- BereqHeader Host: images.domain.nl -- BereqHeader Accept: image/webp,image/*,*/*;q=0.8 -- BereqHeader User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 -- BereqHeader Referer: http://www.domain.nl/bla -- BereqHeader Accept-Language: en-US,en;q=0.8,sr;q=0.6 -- BereqHeader Cookie: _svtri=1b7d5544-84cf-40f1-9228-7be06a208e99; _gat=1; -- BereqHeader X-Schema: http -- BereqHeader X-Forwarded-For: 46.217.107.164 -- BereqHeader Accept-Encoding: gzip -- BereqHeader If-Modified-Since: Fri, 01 Apr 2016 02:58:55 GMT -- BereqHeader X-Varnish: 192316528 -- VCL_call BACKEND_FETCH -- VCL_return fetch -- BackendOpen 156 reload1460629152.default 127.0.0.1 8000 127.0.0.1 45666 -- Timestamp Bereq: 1460629439.397307 0.000056 0.000056 -- Timestamp Beresp: 1460629439.399414 0.002162 0.002106 -- BerespProtocol HTTP/1.1 -- BerespStatus 304 -- BerespReason Not Modified -- BerespHeader Cache-Control: public, stale-while-revalidate=300, max-age=300 -- BerespHeader Last-Modified: Fri, 01 Apr 2016 02:58:55 GMT -- BerespHeader Server: Microsoft-IIS/8.5 -- BerespHeader X-Server-Duration: 0 -- BerespHeader X-Server-Name: CR1-WEB-111 -- BerespHeader X-Server-Timestamp: 14-04-2016 12:23:59 -- BerespHeader X-Server-ReponseStatus: 304 -- BerespHeader X-Server-RouteType: CmsImageRouteHandler2 -- BerespHeader X-AspNet-Version: 4.0.30319 -- BerespHeader X-Powered-By: ASP.NET -- BerespHeader Date: Thu, 14 Apr 2016 10:23:58 GMT -- TTL RFC 300 300 -1 1460629439 1460629439 1460629438 0 300 -- BerespProtocol HTTP/1.1 -- BerespStatus 200 -- BerespReason OK -- BerespHeader Content-Type: image/jpeg -- BerespHeader Content-Length: 92123 -- BerespHeader X-Cacheable: YES -- VCL_call BACKEND_RESPONSE -- BerespUnset X-Cacheable: YES -- BerespHeader X-Cacheable: YES -- VCL_return deliver -- Storage malloc s0 -- ObjProtocol HTTP/1.1 -- ObjStatus 200 -- ObjReason OK -- ObjHeader Cache-Control: public, stale-while-revalidate=300, max-age=300 -- ObjHeader Last-Modified: Fri, 01 Apr 2016 02:58:55 GMT -- ObjHeader Server: Microsoft-IIS/8.5 -- ObjHeader X-Server-Duration: 0 -- ObjHeader X-Server-Name: CR1-WEB-111 -- ObjHeader X-Server-Timestamp: 14-04-2016 12:23:59 -- ObjHeader X-Server-ReponseStatus: 304 -- ObjHeader X-Server-RouteType: CmsImageRouteHandler2 -- ObjHeader X-AspNet-Version: 4.0.30319 -- ObjHeader X-Powered-By: ASP.NET -- ObjHeader Date: Thu, 14 Apr 2016 10:23:58 GMT -- ObjHeader Content-Type: image/jpeg -- ObjHeader Content-Length: 92123 -- ObjHeader X-Cacheable: YES -- BackendReuse 156 reload1460629152.default -- Timestamp BerespBody: 1460629439.399485 0.002233 0.000071 -- Length 92123 -- BereqAcct 900 0 900 418 0 418 -- End -- With kind regards, Angelo H?ngens From A.Hongens at netmatch.nl Thu Apr 14 12:42:08 2016 From: A.Hongens at netmatch.nl (=?utf-8?B?QW5nZWxvIEjDtm5nZW5z?=) Date: Thu, 14 Apr 2016 10:42:08 +0000 Subject: duplicate beresp / code not hit in backend_response? In-Reply-To: <570F7250.4030609@netmatch.nl> References: <570F7250.4030609@netmatch.nl> Message-ID: <6A7ABA19243F1E4EADD8BB1563CDDCCB8A85EA7E@TIL-EXCH-05.netmatch.local> Aaaaand, never mind, I see 304's get a special treatment, as documented in https://www.varnish-cache.org/docs/trunk/users-guide/vcl-built-in-subs.html -- With kind regards, Angelo H?ngens Systems Administrator ------------------------------------------ NetMatch travel technology solutions Professor Donderstraat 46 5017 HL Tilburg T: +31 (0)13 5811088 F: +31 (0)13 5821239 mailto:A.Hongens at netmatch.nl http://www.netmatch.nl ------------------------------------------ Disclaimer Deze e-mail is vertrouwelijk en uitsluitend bedoeld voor geadresseerde(n) en de organisatie van geadresseerde(n) en mag niet openbaar worden gemaakt aan derde partijen This e-mail is confidential and may not be disclosed to third parties since this e-mail is only intended for the addressee and the organization the addressee represents. -----Original Message----- From: varnish-misc-bounces+a.hongens=netmatch.nl at varnish-cache.org [mailto:varnish-misc-bounces+a.hongens=netmatch.nl at varnish-cache.org] On Behalf Of Angelo H?ngens Sent: Thursday, 14 April, 2016 12:35 To: varnish-misc at varnish-cache.org Subject: duplicate beresp / code not hit in backend_response? Hey guys and girls, I'm trying to debug an issue with varnish here. Running varnish-4.1.2-1.el7.x86_64. I think I have two problems. The first one, is that for debugging purposes, I want to force a hit-for-pass on objects with a beresp.status of > 300, and my vcl_backend_response looks like this: sub vcl_backend_response { if (beresp.status > 300) { set beresp.ttl = 0s; set beresp.uncacheable = true; set beresp.http.X-Debug-GT300Resp = "YES"; } if (beresp.uncacheable) { set beresp.http.X-Cacheable = "NO:Not Cacheable"; } else { set beresp.http.X-Cacheable = "YES"; if (beresp.http.Set-Cookie) { unset beresp.http.Set-Cookie; } } return (deliver); } However, varnish does not seem to hit that code, as the X-Debug-GT300Resp header never seems to be added. See the below output of varnishlog -b -g request -q 'BerespStatus > 200'. What's even more confusing, is that I first see a 'BerespStatus 304', and a bit down, I see 'BerespStatus 200'. Huh? Is varnish doing two requests to my backend? ** << BeReq >> 192316528 -- Begin bereq 192316527 bgfetch -- Timestamp Start: 1460629439.397252 0.000000 0.000000 -- BereqMethod GET -- BereqURL /L1E3873A88W480H320.jpg -- BereqProtocol HTTP/1.1 -- BereqHeader Host: images.domain.nl -- BereqHeader Accept: image/webp,image/*,*/*;q=0.8 -- BereqHeader User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 -- BereqHeader Referer: http://www.domain.nl/bla -- BereqHeader Accept-Language: en-US,en;q=0.8,sr;q=0.6 -- BereqHeader Cookie: _svtri=1b7d5544-84cf-40f1-9228-7be06a208e99; _gat=1; -- BereqHeader X-Schema: http -- BereqHeader X-Forwarded-For: 46.217.107.164 -- BereqHeader Accept-Encoding: gzip -- BereqHeader If-Modified-Since: Fri, 01 Apr 2016 02:58:55 GMT -- BereqHeader X-Varnish: 192316528 -- VCL_call BACKEND_FETCH -- VCL_return fetch -- BackendOpen 156 reload1460629152.default 127.0.0.1 8000 127.0.0.1 45666 -- Timestamp Bereq: 1460629439.397307 0.000056 0.000056 -- Timestamp Beresp: 1460629439.399414 0.002162 0.002106 -- BerespProtocol HTTP/1.1 -- BerespStatus 304 -- BerespReason Not Modified -- BerespHeader Cache-Control: public, stale-while-revalidate=300, max-age=300 -- BerespHeader Last-Modified: Fri, 01 Apr 2016 02:58:55 GMT -- BerespHeader Server: Microsoft-IIS/8.5 -- BerespHeader X-Server-Duration: 0 -- BerespHeader X-Server-Name: CR1-WEB-111 -- BerespHeader X-Server-Timestamp: 14-04-2016 12:23:59 -- BerespHeader X-Server-ReponseStatus: 304 -- BerespHeader X-Server-RouteType: CmsImageRouteHandler2 -- BerespHeader X-AspNet-Version: 4.0.30319 -- BerespHeader X-Powered-By: ASP.NET -- BerespHeader Date: Thu, 14 Apr 2016 10:23:58 GMT -- TTL RFC 300 300 -1 1460629439 1460629439 1460629438 0 300 -- BerespProtocol HTTP/1.1 -- BerespStatus 200 -- BerespReason OK -- BerespHeader Content-Type: image/jpeg -- BerespHeader Content-Length: 92123 -- BerespHeader X-Cacheable: YES -- VCL_call BACKEND_RESPONSE -- BerespUnset X-Cacheable: YES -- BerespHeader X-Cacheable: YES -- VCL_return deliver -- Storage malloc s0 -- ObjProtocol HTTP/1.1 -- ObjStatus 200 -- ObjReason OK -- ObjHeader Cache-Control: public, stale-while-revalidate=300, max-age=300 -- ObjHeader Last-Modified: Fri, 01 Apr 2016 02:58:55 GMT -- ObjHeader Server: Microsoft-IIS/8.5 -- ObjHeader X-Server-Duration: 0 -- ObjHeader X-Server-Name: CR1-WEB-111 -- ObjHeader X-Server-Timestamp: 14-04-2016 12:23:59 -- ObjHeader X-Server-ReponseStatus: 304 -- ObjHeader X-Server-RouteType: CmsImageRouteHandler2 -- ObjHeader X-AspNet-Version: 4.0.30319 -- ObjHeader X-Powered-By: ASP.NET -- ObjHeader Date: Thu, 14 Apr 2016 10:23:58 GMT -- ObjHeader Content-Type: image/jpeg -- ObjHeader Content-Length: 92123 -- ObjHeader X-Cacheable: YES -- BackendReuse 156 reload1460629152.default -- Timestamp BerespBody: 1460629439.399485 0.002233 0.000071 -- Length 92123 -- BereqAcct 900 0 900 418 0 418 -- End -- With kind regards, Angelo H?ngens _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From apj at mutt.dk Thu Apr 14 12:51:00 2016 From: apj at mutt.dk (Andreas Plesner) Date: Thu, 14 Apr 2016 12:51:00 +0200 Subject: duplicate beresp / code not hit in backend_response? In-Reply-To: <570F7250.4030609@netmatch.nl> References: <570F7250.4030609@netmatch.nl> Message-ID: <20160414105100.GN2656@nerd.dk> On Thu, Apr 14, 2016 at 12:34:56PM +0200, Angelo H?ngens wrote: > > set beresp.ttl = 0s; > set beresp.uncacheable = true; Not related to your current problem, but HitPass objects need a TTL. > What's even more confusing, is that I first see a 'BerespStatus > 304', and a bit down, I see 'BerespStatus 200'. Huh? Is varnish > doing two requests to my backend? The backend is telling varnish to reuse the current object in the cache (HTTP 304 Not Modified), so varnish proceeds to use the object already in the cache which is a 200. This is a murky part of varnish, in that the backend really responded 304, but what we return to the client is a 200, so the definition of whether this is really a 200 or not is up for debate. If you purge the object already in the cache first, varnish will not do a conditional fetch to the backend. -- Andreas From georgi.int at gmail.com Fri Apr 15 09:33:00 2016 From: georgi.int at gmail.com (georgi.int at gmail.com) Date: Fri, 15 Apr 2016 10:33:00 +0300 Subject: vmod throttle xmlrpc protection does not work as expected Message-ID: <5710992C.9060407@gmail.com> I am using the following implementation to limit the attack to wp-login.php and xmlrpc.php, but when I test with apache benchmark (or there is an attack) from 1000 concurrent requests with the following configuration only 5 requests are failed from 1000 : if ((req.url ~ "(wp-login.php|xmlrpc.php)")) { if(throttle.is_allowed("host:" + req.http.host, "10req/s") > 0s) { error 429 "Calm down"; shield.conn_reset(); } } Is this a normal behavior and why so little number of requests is blocked? Is there an info that describe how these requests are handled and solution of this problem? If I set the throttle to 1 requests it works but this is super stupid and I can't understand why on 10 requests/s limit the throttle does nothing. If I can't solve this issue I should change the varnish with something other so I will be really thankfull if you help with this. From georgi.int at gmail.com Fri Apr 15 11:24:48 2016 From: georgi.int at gmail.com (georgi.int at gmail.com) Date: Fri, 15 Apr 2016 12:24:48 +0300 Subject: vmod throttle xmlrpc protection does not work as expected In-Reply-To: <5710992C.9060407@gmail.com> References: <5710992C.9060407@gmail.com> Message-ID: <5710B360.4030800@gmail.com> btw the following configuration with ip and client.ip instead does not work too: if ((req.url ~ "(wp-login.php|xmlrpc.php)")) { if(throttle.is_allowed("ip:" + client.ip, "10req/s") > 0s) { error 429 "Calm down"; shield.conn_reset(); } } On 04/15/2016 10:33 AM, georgi.int at gmail.com wrote: > I am using the following implementation to limit the attack to > wp-login.php and xmlrpc.php, but when I test with apache benchmark (or > there is an attack) from 1000 concurrent requests with the following > configuration only 5 requests are failed from 1000 : > > if ((req.url ~ "(wp-login.php|xmlrpc.php)")) { > if(throttle.is_allowed("host:" + req.http.host, "10req/s") > 0s) { > error 429 "Calm down"; > shield.conn_reset(); > } > } > > Is this a normal behavior and why so little number of requests is > blocked? Is there an info that describe how these requests are handled > and solution of this problem? If I set the throttle to 1 requests it > works but this is super stupid and I can't understand why on 10 > requests/s limit the throttle does nothing. If I can't solve this > issue I should change the varnish with something other so I will be > really thankfull if you help with this. From tobias.eichelbroenner at lamp-solutions.de Mon Apr 18 10:21:50 2016 From: tobias.eichelbroenner at lamp-solutions.de (=?UTF-8?Q?Tobias_Eichelbr=c3=b6nner?=) Date: Mon, 18 Apr 2016 10:21:50 +0200 Subject: varnish consumes more and more memory Message-ID: <5714991E.3090105@lamp-solutions.de> Hi together, I have an old varnish 3.0.5 running. (I know its outdated, but my issue is more understanding what happens here then fixing the problem on the actual system). While running, the systems takes up more and more memory. I already switched to file storage, but still, after one day running and serving about 2 million requests, the varnish process takes more 20GB of ram. The first thing I am confused is, that as far as I see it varnishstats does not show the memory consumed (I will attach an output at the bottom). Tools like top and pmap show the output. pmap -x ... total kB 25368872 16712056 15197372 Beside some VCL-Code, the varnish uses the vmods redis, digest, and std. Any idea how the find out what is causing the memory leak? Thanks in advance, Tobias Here is my varnishstat -1 output client_conn 2115657 18.63 Client connections accepted client_drop 0 0.00 Connection dropped, no sess/wrk client_req 5176344 45.58 Client requests received cache_hit 2207169 19.43 Cache hits cache_hitpass 18635 0.16 Cache hits for pass cache_miss 508733 4.48 Cache misses backend_conn 590509 5.20 Backend conn. success backend_unhealthy 0 0.00 Backend conn. not attempted backend_busy 0 0.00 Backend conn. too many backend_fail 0 0.00 Backend conn. failures backend_reuse 2391162 21.05 Backend conn. reuses backend_toolate 581 0.01 Backend conn. was closed backend_recycle 2392558 21.07 Backend conn. recycles backend_retry 142 0.00 Backend conn. retry fetch_head 604 0.01 Fetch head fetch_length 1493458 13.15 Fetch with Length fetch_chunked 1277164 11.25 Fetch chunked fetch_eof 0 0.00 Fetch EOF fetch_bad 0 0.00 Fetch had bad headers fetch_close 9969 0.09 Fetch wanted close fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed fetch_zero 0 0.00 Fetch zero len fetch_failed 528 0.00 Fetch failed fetch_1xx 0 0.00 Fetch no body (1xx) fetch_204 186330 1.64 Fetch no body (204) fetch_304 14773 0.13 Fetch no body (304) n_sess_mem 307 . N struct sess_mem n_sess 59 . N struct sess n_object 59401 . N struct object n_vampireobject 0 . N unresurrected objects n_objectcore 59439 . N struct objectcore n_objecthead 60591 . N struct objecthead n_waitinglist 44 . N struct waitinglist n_vbc 2 . N struct vbc n_wrk 44 . N worker threads n_wrk_create 1296 0.01 N worker threads created n_wrk_failed 0 0.00 N worker threads not created n_wrk_max 0 0.00 N worker threads limited n_wrk_lqueue 0 0.00 work request queue length n_wrk_queued 7886 0.07 N queued work requests n_wrk_drop 0 0.00 N dropped work requests n_backend 2 . N backends n_expired 449200 . N expired objects n_lru_nuked 0 . N LRU nuked objects n_lru_moved 1917928 . N LRU moved objects losthdr 0 0.00 HTTP header overflows n_objsendfile 0 0.00 Objects sent with sendfile n_objwrite 4825557 42.49 Objects sent with write n_objoverflow 0 0.00 Objects overflowing workspace s_sess 2115659 18.63 Total Sessions s_req 5176344 45.58 Total Requests s_pipe 1304 0.01 Total pipe s_pass 2473728 21.78 Total pass s_fetch 2981770 26.25 Total fetch s_hdrbytes 2065942876 18190.28 Total header bytes s_bodybytes 133824929675 1178306.04 Total body bytes sess_closed 418834 3.69 Session Closed sess_pipeline 63774 0.56 Session Pipeline sess_readahead 7618 0.07 Session Read Ahead sess_linger 4929655 43.40 Session Linger sess_herd 4533770 39.92 Session herd shm_records 397316593 3498.31 SHM records shm_writes 22134819 194.89 SHM writes shm_flushes 0 0.00 SHM flushes due to overflow shm_cont 83004 0.73 SHM MTX contention shm_cycles 173 0.00 SHM cycles through buffer sms_nreq 693 0.01 SMS allocator requests sms_nobj 0 . SMS outstanding allocations sms_nbytes 0 . SMS outstanding bytes sms_balloc 472858 . SMS bytes allocated sms_bfree 472858 . SMS bytes freed backend_req 2982527 26.26 Backend requests made n_vcl 2 0.00 N vcl total n_vcl_avail 2 0.00 N vcl available n_vcl_discard 0 0.00 N vcl discarded n_ban 1 . N total active bans n_ban_gone 1 . N total gone bans n_ban_add 1 0.00 N new bans added n_ban_retire 0 0.00 N old bans deleted n_ban_obj_test 0 0.00 N objects tested n_ban_re_test 0 0.00 N regexps tested against n_ban_dups 0 0.00 N duplicate bans removed hcb_nolock 2729927 24.04 HCB Lookups without lock hcb_lock 470290 4.14 HCB Lookups with lock hcb_insert 470044 4.14 HCB Inserts esi_errors 0 0.00 ESI parse errors (unlock) esi_warnings 0 0.00 ESI parse warnings (unlock) accept_fail 0 0.00 Accept failures client_drop_late 0 0.00 Connection dropped late uptime 113574 1.00 Client uptime dir_dns_lookups 0 0.00 DNS director lookups dir_dns_failed 0 0.00 DNS director failed lookups dir_dns_hit 0 0.00 DNS director cached lookups hit dir_dns_cache_full 0 0.00 DNS director full dnscache vmods 3 . Loaded VMODs n_gzip 0 0.00 Gzip operations n_gunzip 2590700 22.81 Gunzip operations sess_pipe_overflow 0 . Dropped sessions due to session pipe overflow LCK.sms.creat 1 0.00 Created locks LCK.sms.destroy 0 0.00 Destroyed locks LCK.sms.locks 2079 0.02 Lock Operations LCK.sms.colls 0 0.00 Collisions LCK.smp.creat 0 0.00 Created locks LCK.smp.destroy 0 0.00 Destroyed locks LCK.smp.locks 0 0.00 Lock Operations LCK.smp.colls 0 0.00 Collisions LCK.sma.creat 1 0.00 Created locks LCK.sma.destroy 0 0.00 Destroyed locks LCK.sma.locks 11567121 101.85 Lock Operations LCK.sma.colls 0 0.00 Collisions LCK.smf.creat 1 0.00 Created locks LCK.smf.destroy 0 0.00 Destroyed locks LCK.smf.locks 2137363 18.82 Lock Operations LCK.smf.colls 0 0.00 Collisions LCK.hsl.creat 0 0.00 Created locks LCK.hsl.destroy 0 0.00 Destroyed locks LCK.hsl.locks 0 0.00 Lock Operations LCK.hsl.colls 0 0.00 Collisions LCK.hcb.creat 1 0.00 Created locks LCK.hcb.destroy 0 0.00 Destroyed locks LCK.hcb.locks 881563 7.76 Lock Operations LCK.hcb.colls 0 0.00 Collisions LCK.hcl.creat 0 0.00 Created locks LCK.hcl.destroy 0 0.00 Destroyed locks LCK.hcl.locks 0 0.00 Lock Operations LCK.hcl.colls 0 0.00 Collisions LCK.vcl.creat 1 0.00 Created locks LCK.vcl.destroy 0 0.00 Destroyed locks LCK.vcl.locks 10574 0.09 Lock Operations LCK.vcl.colls 0 0.00 Collisions LCK.stat.creat 1 0.00 Created locks LCK.stat.destroy 0 0.00 Destroyed locks LCK.stat.locks 2115908 18.63 Lock Operations LCK.stat.colls 0 0.00 Collisions LCK.sessmem.creat 1 0.00 Created locks LCK.sessmem.destroy 0 0.00 Destroyed locks LCK.sessmem.locks 2129936 18.75 Lock Operations LCK.sessmem.colls 0 0.00 Collisions LCK.wstat.creat 1 0.00 Created locks LCK.wstat.destroy 0 0.00 Destroyed locks LCK.wstat.locks 239816 2.11 Lock Operations LCK.wstat.colls 0 0.00 Collisions LCK.herder.creat 1 0.00 Created locks LCK.herder.destroy 0 0.00 Destroyed locks LCK.herder.locks 6034 0.05 Lock Operations LCK.herder.colls 0 0.00 Collisions LCK.wq.creat 2 0.00 Created locks LCK.wq.destroy 0 0.00 Destroyed locks LCK.wq.locks 10137930 89.26 Lock Operations LCK.wq.colls 0 0.00 Collisions LCK.objhdr.creat 471022 4.15 Created locks LCK.objhdr.destroy 410522 3.61 Destroyed locks LCK.objhdr.locks 11873362 104.54 Lock Operations LCK.objhdr.colls 0 0.00 Collisions LCK.exp.creat 1 0.00 Created locks LCK.exp.destroy 0 0.00 Destroyed locks LCK.exp.locks 1086349 9.57 Lock Operations LCK.exp.colls 0 0.00 Collisions LCK.lru.creat 2 0.00 Created locks LCK.lru.destroy 0 0.00 Destroyed locks LCK.lru.locks 523780 4.61 Lock Operations LCK.lru.colls 0 0.00 Collisions LCK.cli.creat 1 0.00 Created locks LCK.cli.destroy 0 0.00 Destroyed locks LCK.cli.locks 877 0.01 Lock Operations LCK.cli.colls 0 0.00 Collisions LCK.ban.creat 1 0.00 Created locks LCK.ban.destroy 0 0.00 Destroyed locks LCK.ban.locks 1071376 9.43 Lock Operations LCK.ban.colls 0 0.00 Collisions LCK.vbp.creat 1 0.00 Created locks LCK.vbp.destroy 0 0.00 Destroyed locks LCK.vbp.locks 0 0.00 Lock Operations LCK.vbp.colls 0 0.00 Collisions LCK.vbe.creat 1 0.00 Created locks LCK.vbe.destroy 0 0.00 Destroyed locks LCK.vbe.locks 1183870 10.42 Lock Operations LCK.vbe.colls 0 0.00 Collisions LCK.backend.creat 2 0.00 Created locks LCK.backend.destroy 0 0.00 Destroyed locks LCK.backend.locks 6560980 57.77 Lock Operations LCK.backend.colls 0 0.00 Collisions SMF.s0.c_req 943045 8.30 Allocator requests SMF.s0.c_fail 0 0.00 Allocator failures SMF.s0.c_bytes 57519628288 506450.67 Bytes allocated SMF.s0.c_freed 54163292160 476898.69 Bytes freed SMF.s0.g_alloc 129160 . Allocations outstanding SMF.s0.g_bytes 3356336128 . Bytes outstanding SMF.s0.g_space 938631168 . Bytes available SMF.s0.g_smf 138281 . N struct smf SMF.s0.g_smf_frag 9118 . N small free smf SMF.s0.g_smf_large 3 . N large free smf SMA.Transient.c_req 4780332 42.09 Allocator requests SMA.Transient.c_fail 0 0.00 Allocator failures SMA.Transient.c_bytes 284689405959 2506642.42 Bytes allocated SMA.Transient.c_freed 284688351047 2506633.13 Bytes freed SMA.Transient.g_alloc 85 . Allocations outstanding SMA.Transient.g_bytes 1054912 . Bytes outstanding SMA.Transient.g_space 0 . Bytes available VBE.default(127.0.0.1,,8080).vcls 2 . VCL references VBE.default(127.0.0.1,,8080).happy 0 . Happy health probes VBE.fpm(127.0.0.1,,8081).vcls 2 . VCL references VBE.fpm(127.0.0.1,,8081).happy 0 . Happy health probes -- LAMP solutions GmbH Gostenhofer Hauptstrasse 35 90443 Nuernberg Amtsgericht Nuernberg: HRB 22366 Geschaeftsfuehrer: Heiko Schubert Es gelten unsere allgemeinen Geschaeftsbedingungen. http://www.lamp-solutions.de/agb/ Telefon : 0911 / 376 516 0 Fax : 0911 / 376 516 11 E-Mail : support at lamp-solutions.de Web : www.lamp-solutions.de Facebook : http://www.facebook.com/LAMPsolutions Twitter : http://twitter.com/#!/lampsolutions From japrice at gmail.com Mon Apr 18 14:19:50 2016 From: japrice at gmail.com (Jason Price) Date: Mon, 18 Apr 2016 08:19:50 -0400 Subject: varnish consumes more and more memory In-Reply-To: <5714991E.3090105@lamp-solutions.de> References: <5714991E.3090105@lamp-solutions.de> Message-ID: I'm curious what the Virtual Memory vs Resident Set Size from top show. Most likely, the RSS will be stable and small. Virtual memory is almost valueless as a measure. ps -eo pid,user,vsz,rss,%cpu,%mem,cmd | egrep "PID|varnish" -Jason On Mon, Apr 18, 2016 at 4:21 AM, Tobias Eichelbr?nner < tobias.eichelbroenner at lamp-solutions.de> wrote: > Hi together, > > I have an old varnish 3.0.5 running. (I know its outdated, but my issue > is more understanding what happens here then fixing the problem on the > actual system). While running, the systems takes up more and more memory. > > I already switched to file storage, but still, after one day running and > serving about 2 million requests, the varnish process takes more 20GB of > ram. > > The first thing I am confused is, that as far as I see it varnishstats > does not show the memory consumed (I will attach an output at the > bottom). Tools like top and pmap show the output. > > pmap -x ... > total kB 25368872 16712056 15197372 > > Beside some VCL-Code, the varnish uses the vmods redis, digest, and std. > > Any idea how the find out what is causing the memory leak? > > Thanks in advance, > > Tobias > > Here is my varnishstat -1 output > > client_conn 2115657 18.63 Client connections accepted > client_drop 0 0.00 Connection dropped, no sess/wrk > client_req 5176344 45.58 Client requests received > cache_hit 2207169 19.43 Cache hits > cache_hitpass 18635 0.16 Cache hits for pass > cache_miss 508733 4.48 Cache misses > backend_conn 590509 5.20 Backend conn. success > backend_unhealthy 0 0.00 Backend conn. not attempted > backend_busy 0 0.00 Backend conn. too many > backend_fail 0 0.00 Backend conn. failures > backend_reuse 2391162 21.05 Backend conn. reuses > backend_toolate 581 0.01 Backend conn. was closed > backend_recycle 2392558 21.07 Backend conn. recycles > backend_retry 142 0.00 Backend conn. retry > fetch_head 604 0.01 Fetch head > fetch_length 1493458 13.15 Fetch with Length > fetch_chunked 1277164 11.25 Fetch chunked > fetch_eof 0 0.00 Fetch EOF > fetch_bad 0 0.00 Fetch had bad headers > fetch_close 9969 0.09 Fetch wanted close > fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed > fetch_zero 0 0.00 Fetch zero len > fetch_failed 528 0.00 Fetch failed > fetch_1xx 0 0.00 Fetch no body (1xx) > fetch_204 186330 1.64 Fetch no body (204) > fetch_304 14773 0.13 Fetch no body (304) > n_sess_mem 307 . N struct sess_mem > n_sess 59 . N struct sess > n_object 59401 . N struct object > n_vampireobject 0 . N unresurrected objects > n_objectcore 59439 . N struct objectcore > n_objecthead 60591 . N struct objecthead > n_waitinglist 44 . N struct waitinglist > n_vbc 2 . N struct vbc > n_wrk 44 . N worker threads > n_wrk_create 1296 0.01 N worker threads created > n_wrk_failed 0 0.00 N worker threads not created > n_wrk_max 0 0.00 N worker threads limited > n_wrk_lqueue 0 0.00 work request queue length > n_wrk_queued 7886 0.07 N queued work requests > n_wrk_drop 0 0.00 N dropped work requests > n_backend 2 . N backends > n_expired 449200 . N expired objects > n_lru_nuked 0 . N LRU nuked objects > n_lru_moved 1917928 . N LRU moved objects > losthdr 0 0.00 HTTP header overflows > n_objsendfile 0 0.00 Objects sent with sendfile > n_objwrite 4825557 42.49 Objects sent with write > n_objoverflow 0 0.00 Objects overflowing workspace > s_sess 2115659 18.63 Total Sessions > s_req 5176344 45.58 Total Requests > s_pipe 1304 0.01 Total pipe > s_pass 2473728 21.78 Total pass > s_fetch 2981770 26.25 Total fetch > s_hdrbytes 2065942876 18190.28 Total header bytes > s_bodybytes 133824929675 1178306.04 Total body bytes > sess_closed 418834 3.69 Session Closed > sess_pipeline 63774 0.56 Session Pipeline > sess_readahead 7618 0.07 Session Read Ahead > sess_linger 4929655 43.40 Session Linger > sess_herd 4533770 39.92 Session herd > shm_records 397316593 3498.31 SHM records > shm_writes 22134819 194.89 SHM writes > shm_flushes 0 0.00 SHM flushes due to overflow > shm_cont 83004 0.73 SHM MTX contention > shm_cycles 173 0.00 SHM cycles through buffer > sms_nreq 693 0.01 SMS allocator requests > sms_nobj 0 . SMS outstanding allocations > sms_nbytes 0 . SMS outstanding bytes > sms_balloc 472858 . SMS bytes allocated > sms_bfree 472858 . SMS bytes freed > backend_req 2982527 26.26 Backend requests made > n_vcl 2 0.00 N vcl total > n_vcl_avail 2 0.00 N vcl available > n_vcl_discard 0 0.00 N vcl discarded > n_ban 1 . N total active bans > n_ban_gone 1 . N total gone bans > n_ban_add 1 0.00 N new bans added > n_ban_retire 0 0.00 N old bans deleted > n_ban_obj_test 0 0.00 N objects tested > n_ban_re_test 0 0.00 N regexps tested against > n_ban_dups 0 0.00 N duplicate bans removed > hcb_nolock 2729927 24.04 HCB Lookups without lock > hcb_lock 470290 4.14 HCB Lookups with lock > hcb_insert 470044 4.14 HCB Inserts > esi_errors 0 0.00 ESI parse errors (unlock) > esi_warnings 0 0.00 ESI parse warnings (unlock) > accept_fail 0 0.00 Accept failures > client_drop_late 0 0.00 Connection dropped late > uptime 113574 1.00 Client uptime > dir_dns_lookups 0 0.00 DNS director lookups > dir_dns_failed 0 0.00 DNS director failed lookups > dir_dns_hit 0 0.00 DNS director cached lookups hit > dir_dns_cache_full 0 0.00 DNS director full dnscache > vmods 3 . Loaded VMODs > n_gzip 0 0.00 Gzip operations > n_gunzip 2590700 22.81 Gunzip operations > sess_pipe_overflow 0 . Dropped sessions due to > session pipe overflow > LCK.sms.creat 1 0.00 Created locks > LCK.sms.destroy 0 0.00 Destroyed locks > LCK.sms.locks 2079 0.02 Lock Operations > LCK.sms.colls 0 0.00 Collisions > LCK.smp.creat 0 0.00 Created locks > LCK.smp.destroy 0 0.00 Destroyed locks > LCK.smp.locks 0 0.00 Lock Operations > LCK.smp.colls 0 0.00 Collisions > LCK.sma.creat 1 0.00 Created locks > LCK.sma.destroy 0 0.00 Destroyed locks > LCK.sma.locks 11567121 101.85 Lock Operations > LCK.sma.colls 0 0.00 Collisions > LCK.smf.creat 1 0.00 Created locks > LCK.smf.destroy 0 0.00 Destroyed locks > LCK.smf.locks 2137363 18.82 Lock Operations > LCK.smf.colls 0 0.00 Collisions > LCK.hsl.creat 0 0.00 Created locks > LCK.hsl.destroy 0 0.00 Destroyed locks > LCK.hsl.locks 0 0.00 Lock Operations > LCK.hsl.colls 0 0.00 Collisions > LCK.hcb.creat 1 0.00 Created locks > LCK.hcb.destroy 0 0.00 Destroyed locks > LCK.hcb.locks 881563 7.76 Lock Operations > LCK.hcb.colls 0 0.00 Collisions > LCK.hcl.creat 0 0.00 Created locks > LCK.hcl.destroy 0 0.00 Destroyed locks > LCK.hcl.locks 0 0.00 Lock Operations > LCK.hcl.colls 0 0.00 Collisions > LCK.vcl.creat 1 0.00 Created locks > LCK.vcl.destroy 0 0.00 Destroyed locks > LCK.vcl.locks 10574 0.09 Lock Operations > LCK.vcl.colls 0 0.00 Collisions > LCK.stat.creat 1 0.00 Created locks > LCK.stat.destroy 0 0.00 Destroyed locks > LCK.stat.locks 2115908 18.63 Lock Operations > LCK.stat.colls 0 0.00 Collisions > LCK.sessmem.creat 1 0.00 Created locks > LCK.sessmem.destroy 0 0.00 Destroyed locks > LCK.sessmem.locks 2129936 18.75 Lock Operations > LCK.sessmem.colls 0 0.00 Collisions > LCK.wstat.creat 1 0.00 Created locks > LCK.wstat.destroy 0 0.00 Destroyed locks > LCK.wstat.locks 239816 2.11 Lock Operations > LCK.wstat.colls 0 0.00 Collisions > LCK.herder.creat 1 0.00 Created locks > LCK.herder.destroy 0 0.00 Destroyed locks > LCK.herder.locks 6034 0.05 Lock Operations > LCK.herder.colls 0 0.00 Collisions > LCK.wq.creat 2 0.00 Created locks > LCK.wq.destroy 0 0.00 Destroyed locks > LCK.wq.locks 10137930 89.26 Lock Operations > LCK.wq.colls 0 0.00 Collisions > LCK.objhdr.creat 471022 4.15 Created locks > LCK.objhdr.destroy 410522 3.61 Destroyed locks > LCK.objhdr.locks 11873362 104.54 Lock Operations > LCK.objhdr.colls 0 0.00 Collisions > LCK.exp.creat 1 0.00 Created locks > LCK.exp.destroy 0 0.00 Destroyed locks > LCK.exp.locks 1086349 9.57 Lock Operations > LCK.exp.colls 0 0.00 Collisions > LCK.lru.creat 2 0.00 Created locks > LCK.lru.destroy 0 0.00 Destroyed locks > LCK.lru.locks 523780 4.61 Lock Operations > LCK.lru.colls 0 0.00 Collisions > LCK.cli.creat 1 0.00 Created locks > LCK.cli.destroy 0 0.00 Destroyed locks > LCK.cli.locks 877 0.01 Lock Operations > LCK.cli.colls 0 0.00 Collisions > LCK.ban.creat 1 0.00 Created locks > LCK.ban.destroy 0 0.00 Destroyed locks > LCK.ban.locks 1071376 9.43 Lock Operations > LCK.ban.colls 0 0.00 Collisions > LCK.vbp.creat 1 0.00 Created locks > LCK.vbp.destroy 0 0.00 Destroyed locks > LCK.vbp.locks 0 0.00 Lock Operations > LCK.vbp.colls 0 0.00 Collisions > LCK.vbe.creat 1 0.00 Created locks > LCK.vbe.destroy 0 0.00 Destroyed locks > LCK.vbe.locks 1183870 10.42 Lock Operations > LCK.vbe.colls 0 0.00 Collisions > LCK.backend.creat 2 0.00 Created locks > LCK.backend.destroy 0 0.00 Destroyed locks > LCK.backend.locks 6560980 57.77 Lock Operations > LCK.backend.colls 0 0.00 Collisions > SMF.s0.c_req 943045 8.30 Allocator requests > SMF.s0.c_fail 0 0.00 Allocator failures > SMF.s0.c_bytes 57519628288 506450.67 Bytes allocated > SMF.s0.c_freed 54163292160 476898.69 Bytes freed > SMF.s0.g_alloc 129160 . Allocations outstanding > SMF.s0.g_bytes 3356336128 . Bytes outstanding > SMF.s0.g_space 938631168 . Bytes available > SMF.s0.g_smf 138281 . N struct smf > SMF.s0.g_smf_frag 9118 . N small free smf > SMF.s0.g_smf_large 3 . N large free smf > SMA.Transient.c_req 4780332 42.09 Allocator requests > SMA.Transient.c_fail 0 0.00 Allocator failures > SMA.Transient.c_bytes 284689405959 2506642.42 Bytes allocated > SMA.Transient.c_freed 284688351047 2506633.13 Bytes freed > SMA.Transient.g_alloc 85 . Allocations outstanding > SMA.Transient.g_bytes 1054912 . Bytes outstanding > SMA.Transient.g_space 0 . Bytes available > VBE.default(127.0.0.1,,8080).vcls 2 . VCL references > VBE.default(127.0.0.1,,8080).happy 0 . Happy health > probes > VBE.fpm(127.0.0.1,,8081).vcls 2 . VCL references > VBE.fpm(127.0.0.1,,8081).happy 0 . Happy health > probes > > -- > LAMP solutions GmbH > Gostenhofer Hauptstrasse 35 > 90443 Nuernberg > > Amtsgericht Nuernberg: HRB 22366 > Geschaeftsfuehrer: Heiko Schubert > > Es gelten unsere allgemeinen Geschaeftsbedingungen. > http://www.lamp-solutions.de/agb/ > > Telefon : 0911 / 376 516 0 > Fax : 0911 / 376 516 11 > E-Mail : support at lamp-solutions.de > Web : www.lamp-solutions.de > Facebook : http://www.facebook.com/LAMPsolutions > Twitter : http://twitter.com/#!/lampsolutions > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tobias.eichelbroenner at lamp-solutions.de Mon Apr 18 14:33:28 2016 From: tobias.eichelbroenner at lamp-solutions.de (=?UTF-8?Q?Tobias_Eichelbr=c3=b6nner?=) Date: Mon, 18 Apr 2016 14:33:28 +0200 Subject: varnish consumes more and more memory In-Reply-To: References: <5714991E.3090105@lamp-solutions.de> Message-ID: <5714D418.40005@lamp-solutions.de> Hi Jason, thanks for the reply. > ps -eo pid,user,vsz,rss,%cpu,%mem,cmd | egrep ... nobody 7284032 3590708 5.2 5.4 /usr/sbin/varnishd -P /var/run/varnishd.pid -a :80 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s file,/var/lib/varnish/varnish_storage.bin,4G The stats shown here are lower than reported in my last mail as I had to restart varnish since than. But 8 GB of virtual RAM also seem to be to high for a system with 4GB file storage and does not match anything I see in varnishtat output. Sincerely, Tobias -- LAMP solutions GmbH Gostenhofer Hauptstrasse 35 90443 Nuernberg Amtsgericht Nuernberg: HRB 22366 Geschaeftsfuehrer: Heiko Schubert Es gelten unsere allgemeinen Geschaeftsbedingungen. http://www.lamp-solutions.de/agb/ Telefon : 0911 / 376 516 0 Fax : 0911 / 376 516 11 E-Mail : support at lamp-solutions.de Web : www.lamp-solutions.de Facebook : http://www.facebook.com/LAMPsolutions Twitter : http://twitter.com/#!/lampsolutions From japrice at gmail.com Mon Apr 18 15:19:13 2016 From: japrice at gmail.com (Jason Price) Date: Mon, 18 Apr 2016 09:19:13 -0400 Subject: varnish consumes more and more memory In-Reply-To: <5714D418.40005@lamp-solutions.de> References: <5714991E.3090105@lamp-solutions.de> <5714D418.40005@lamp-solutions.de> Message-ID: Ignore the vsz column. It is effectively worthless as a measure. That RSS however... if that grows to more than 150% of the 4G number, then you might have to look at your config more closely. Why 150%? because there's overhead for each cached object (around 1k per) and there's other stuff which impacts things as well. -Jason On Mon, Apr 18, 2016 at 8:33 AM, Tobias Eichelbr?nner < tobias.eichelbroenner at lamp-solutions.de> wrote: > Hi Jason, > > thanks for the reply. > > > ps -eo pid,user,vsz,rss,%cpu,%mem,cmd | egrep ... > > nobody 7284032 3590708 5.2 5.4 /usr/sbin/varnishd -P > /var/run/varnishd.pid -a :80 -T localhost:6082 -f > /etc/varnish/default.vcl -S /etc/varnish/secret -s > file,/var/lib/varnish/varnish_storage.bin,4G > > The stats shown here are lower than reported in my last mail as I had to > restart varnish since than. But 8 GB of virtual RAM also seem to be to > high for a system with 4GB file storage and does not match anything I > see in varnishtat output. > > Sincerely, > > Tobias > > -- > LAMP solutions GmbH > Gostenhofer Hauptstrasse 35 > 90443 Nuernberg > > Amtsgericht Nuernberg: HRB 22366 > Geschaeftsfuehrer: Heiko Schubert > > Es gelten unsere allgemeinen Geschaeftsbedingungen. > http://www.lamp-solutions.de/agb/ > > Telefon : 0911 / 376 516 0 > Fax : 0911 / 376 516 11 > E-Mail : support at lamp-solutions.de > Web : www.lamp-solutions.de > Facebook : http://www.facebook.com/LAMPsolutions > Twitter : http://twitter.com/#!/lampsolutions > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From de.techno at gmail.com Fri Apr 22 10:13:20 2016 From: de.techno at gmail.com (dE) Date: Fri, 22 Apr 2016 13:43:20 +0530 Subject: Minimalistic varnish does not compile. Message-ID: <5719DD20.1050805@gmail.com> Hello, I got this minimal VCL -- |vcl 4.0;|| ||backend neuro {|| ||.host = "129.170.30.193";|| ||.port = "http";|| ||}|| || ||sub vcl_recv {|| ||set req.backend_hint = neuro;|| ||}| Which fails to compile for no reason. |Running VCC-compiler failed, exited with 2|| || ||VCL compilation failed| Can anyone help? -------------- next part -------------- An HTML attachment was scrubbed... URL: From de.techno at gmail.com Fri Apr 22 10:17:36 2016 From: de.techno at gmail.com (dE) Date: Fri, 22 Apr 2016 13:47:36 +0530 Subject: Minimalistic varnish does not compile. In-Reply-To: <5719DD20.1050805@gmail.com> References: <5719DD20.1050805@gmail.com> Message-ID: <5719DE20.1000801@gmail.com> And yes, this is varnish 4.0.3 On 04/22/16 13:43, dE wrote: > Hello, > I got this minimal VCL -- > > |vcl 4.0;|| > ||backend neuro {|| > || .host = "129.170.30.193";|| > || .port = "http";|| > ||}|| > || > ||sub vcl_recv {|| > || set req.backend_hint = neuro;|| > ||}| > > Which fails to compile for no reason. > > |Running VCC-compiler failed, exited with 2|| > || > ||VCL compilation failed| > > Can anyone help? -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Fri Apr 22 10:20:43 2016 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Fri, 22 Apr 2016 08:20:43 +0000 Subject: Minimalistic varnish does not compile. In-Reply-To: <5719DD20.1050805@gmail.com> References: <5719DD20.1050805@gmail.com> Message-ID: <27407.1461313243@critter.freebsd.dk> -------- In message <5719DD20.1050805 at gmail.com>, dE writes: Is there a C-compiler installed on your computer ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From de.techno at gmail.com Fri Apr 22 11:49:59 2016 From: de.techno at gmail.com (dE) Date: Fri, 22 Apr 2016 15:19:59 +0530 Subject: Minimalistic varnish does not compile. In-Reply-To: <27407.1461313243@critter.freebsd.dk> References: <5719DD20.1050805@gmail.com> <27407.1461313243@critter.freebsd.dk> Message-ID: <5719F3C7.207@gmail.com> gcc 5.3.0 Using built-in specs. COLLECT_GCC=/usr/x86_64-pc-linux-gnu/gcc-bin/5.3.0/gcc COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-pc-linux-gnu/5.3.0/lto-wrapper Target: x86_64-pc-linux-gnu Configured with: /tmp/portage/sys-devel/gcc-5.3.0/work/gcc-5.3.0/configure --host=x86_64-pc-linux-gnu --build=x86_64-pc-linux-gnu --prefix=/usr --bindir=/usr/x86_64-pc-linux-gnu/gcc-bin/5.3.0 --includedir=/usr/lib/gcc/x86_64-pc-linux-gnu/5.3.0/include --datadir=/usr/share/gcc-data/x86_64-pc-linux-gnu/5.3.0 --mandir=/usr/share/gcc-data/x86_64-pc-linux-gnu/5.3.0/man --infodir=/usr/share/gcc-data/x86_64-pc-linux-gnu/5.3.0/info --with-gxx-include-dir=/usr/lib/gcc/x86_64-pc-linux-gnu/5.3.0/include/g++-v5 --with-python-dir=/share/gcc-data/x86_64-pc-linux-gnu/5.3.0/python --enable-languages=c,c++ --enable-obsolete --enable-secureplt --disable-werror --with-system-zlib --disable-nls --enable-checking=release --with-bugurl=https://bugs.gentoo.org/ --with-pkgversion='Gentoo 5.3.0 p1.0, pie-0.6.5' --enable-libstdcxx-time --enable-shared --enable-threads=posix --enable-__cxa_atexit --enable-clocale=gnu --enable-multilib --with-multilib-list=m32,m64 --disable-altivec --disable-fixed-point --enable-targets=all --disable-libgcj --enable-libgomp --disable-libmudflap --disable-libssp --disable-libcilkrts --disable-libquadmath --enable-lto --with-isl --disable-isl-version-check --disable-libsanitizer Thread model: posix gcc version 5.3.0 (Gentoo 5.3.0 p1.0, pie-0.6.5) This's Gentoo (source based distribution). Trying out with GCC 4.9 On 04/22/16 13:50, Poul-Henning Kamp wrote: > -------- > In message <5719DD20.1050805 at gmail.com>, dE writes: > > Is there a C-compiler installed on your computer ? > From de.techno at gmail.com Mon Apr 25 10:29:19 2016 From: de.techno at gmail.com (dE) Date: Mon, 25 Apr 2016 13:59:19 +0530 Subject: Minimalistic varnish does not compile. In-Reply-To: <5719F3C7.207@gmail.com> References: <5719DD20.1050805@gmail.com> <27407.1461313243@critter.freebsd.dk> <5719F3C7.207@gmail.com> Message-ID: <571DD55F.3040207@gmail.com> No, it does not work :(. On 04/22/16 15:19, dE wrote: > gcc 5.3.0 > > Using built-in specs. > COLLECT_GCC=/usr/x86_64-pc-linux-gnu/gcc-bin/5.3.0/gcc > COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-pc-linux-gnu/5.3.0/lto-wrapper > > Target: x86_64-pc-linux-gnu > Configured with: > /tmp/portage/sys-devel/gcc-5.3.0/work/gcc-5.3.0/configure > --host=x86_64-pc-linux-gnu --build=x86_64-pc-linux-gnu --prefix=/usr > --bindir=/usr/x86_64-pc-linux-gnu/gcc-bin/5.3.0 > --includedir=/usr/lib/gcc/x86_64-pc-linux-gnu/5.3.0/include > --datadir=/usr/share/gcc-data/x86_64-pc-linux-gnu/5.3.0 > --mandir=/usr/share/gcc-data/x86_64-pc-linux-gnu/5.3.0/man > --infodir=/usr/share/gcc-data/x86_64-pc-linux-gnu/5.3.0/info > --with-gxx-include-dir=/usr/lib/gcc/x86_64-pc-linux-gnu/5.3.0/include/g++-v5 > --with-python-dir=/share/gcc-data/x86_64-pc-linux-gnu/5.3.0/python > --enable-languages=c,c++ --enable-obsolete --enable-secureplt > --disable-werror --with-system-zlib --disable-nls > --enable-checking=release --with-bugurl=https://bugs.gentoo.org/ > --with-pkgversion='Gentoo 5.3.0 p1.0, pie-0.6.5' > --enable-libstdcxx-time --enable-shared --enable-threads=posix > --enable-__cxa_atexit --enable-clocale=gnu --enable-multilib > --with-multilib-list=m32,m64 --disable-altivec --disable-fixed-point > --enable-targets=all --disable-libgcj --enable-libgomp > --disable-libmudflap --disable-libssp --disable-libcilkrts > --disable-libquadmath --enable-lto --with-isl > --disable-isl-version-check --disable-libsanitizer > Thread model: posix > gcc version 5.3.0 (Gentoo 5.3.0 p1.0, pie-0.6.5) > > This's Gentoo (source based distribution). > > Trying out with GCC 4.9 > > On 04/22/16 13:50, Poul-Henning Kamp wrote: >> -------- >> In message <5719DD20.1050805 at gmail.com>, dE writes: >> >> Is there a C-compiler installed on your computer ? >> > From kontakt at andreas-loewen.de Tue Apr 26 13:54:30 2016 From: kontakt at andreas-loewen.de (Andreas Loewen Internetservices) Date: Tue, 26 Apr 2016 13:54:30 +0200 Subject: ESI from remote host not working? Message-ID: <571F56F6.40103@andreas-loewen.de> Dear all, i am using current varnish varnish 4. I have the following config: backend esi{ .host = "x.x.x.x"; .port = "8081"; } ... sub vcl_recv { set req.http.host = "xxx.yyy.de"; ... if (req.url ~ "^/typo3/.*" || req.url ~ "^/secure/.*") { set req.backend_hint = typo3be; return (synth(403, "Forbidden")); } else if (req.url ~ "\.mesi\?" || req.url ~ "\.htm(\?.*)?$" || req.url ~ "^/javax\.faces\..*" || req.url ~ "^/templates/.*" || req.url ~"^/f/.*" || req.url ~"^/i/.*") { set req.backend_hint = esi; set req.hash_always_miss = true; } else { set req.backend_hint = typo3; unset req.http.cookie; } ... } sub vcl_backend_response { set beresp.do_esi = true; ... } The esi backend is on a remote host, but it is reachable from this server. The funny thing is, that I keep getting 404's for my esi requests for /fe_terminliste.mesi?pakid=1 . But they are comming from the default backend (local Apache on port 81). varnishlog reads like: * << BeReq >> 5 - Begin bereq 4 pass - Timestamp Start: 1461670355.552619 0.000000 0.000000 - BereqMethod GET - BereqURL /fe_terminliste.mesi?pakid=1 - BereqProtocol HTTP/1.1 - BereqHeader Cache-Control: max-age=0 - BereqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8 - BereqHeader Upgrade-Insecure-Requests: 1 - BereqHeader User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36 - BereqHeader DNT: 1 - BereqHeader Accept-Encoding: gzip, deflate - BereqHeader Accept-Language: de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4 - BereqHeader Cookie: PHPSESSID=h7hk5n8pa95q4sd2m46ebbju14; Typo3InstallTool=tgikum56sujjsv1cu6ra236kb0; be_typo_user=5ad26762d12ad4b9fe2d321979fb75e4 - BereqHeader Host: xxx:8081 - BereqHeader X-Forwarded-For: 80.171.39.71 - BereqHeader X-Varnish: 5 - VCL_call BACKEND_FETCH - VCL_return fetch - BackendOpen 20 default(127.0.0.1,,81) 127.0.0.1 33017 - Backend 20 default default(127.0.0.1,,81) - Timestamp Bereq: 1461670355.552877 0.000257 0.000257 - Timestamp Beresp: 1461670355.553530 0.000911 0.000654 - BerespProtocol HTTP/1.1 - BerespStatus 404 - BerespReason Not Found - BerespHeader Date: Tue, 26 Apr 2016 11:32:35 GMT - BerespHeader Server: Apache/2.2.15 (CentOS) - BerespHeader Content-Length: 310 - BerespHeader Connection: close - BerespHeader Content-Type: text/html; charset=iso-8859-1 - TTL RFC 120 -1 -1 1461670356 1461670356 1461670355 0 0 - VCL_call BACKEND_RESPONSE - TTL VCL 120 10 0 1461670356 - VCL_return deliver - BerespHeader Content-Encoding: gzip - BerespUnset Content-Length: 310 - Storage malloc Transient - ObjProtocol HTTP/1.1 - ObjStatus 404 - ObjReason Not Found - ObjHeader Date: Tue, 26 Apr 2016 11:32:35 GMT - ObjHeader Server: Apache/2.2.15 (CentOS) - ObjHeader Content-Type: text/html; charset=iso-8859-1 - ObjHeader Content-Encoding: gzip - Fetch_Body 3 length - - Gzip G F E 310 268 80 2064 2074 - Timestamp BerespBody: 1461670355.554030 0.001411 0.000500 - BackendClose 20 default(127.0.0.1,,81) - Length 268 - BereqAcct 623 0 623 180 310 490 - End So, does varnish restrict esi to localhost for some reason? Or what is wrong? From andreas.haase at evolver.de Tue Apr 26 14:26:59 2016 From: andreas.haase at evolver.de (Andreas Haase - evolver group) Date: Tue, 26 Apr 2016 14:26:59 +0200 Subject: Asking for backend set in VCL 4 Message-ID: Hello, we are just migrating one varnish from 3.x to 4.0.2 (contained in Debian Jessie). Our old vcl contains the following structure: [...] sub vcl_recv { set req.backend = foo; if (some condition) { set req.backend = bar; } [...] if ( req.backend = foo ) { /* do something nasty */ } [...] } Now, in VCL 4 there seems no possibility to ask for the backend actually set. Is there any way to use this again, or do we have to use a work-around like setting custom http headers and later asking for their content? Thank you in advance for any hint :-) -- Regards, Andreas Haase Administration und Technik evolver services GmbH Fon: +49 / (0)3 71 / 4 00 03 727 Fax: +49 / (0)3 71 / 4 00 03 79 E-Mail andreas.haase at evolver.de Web http://www.evolver.de Facebook http://www.facebook.de/evolvergroup Sitz der Gesellschaft: Chemnitz Handelsregister: Amtsgericht Chemnitz, HRB 22649 Gesch?ftsf?hrer: Torsten Gramann und Mathias M?ckel [zu unserer Webseite] Unsere Top-Produkte: - Stellenportal - Immobilienportal - evolverOAS - evolverCAS - evolverCMS Unsere Top-Dienstleistungen: - Administration & IT - Service-Center - garantierte Ressourcen - Manntage-Pakete - evolverCLOUD SEO-Optimierung inklusive: Zur Verbesserung Ihrer Marktchancen mit unseren Produkten investieren wir noch mehr in die SEO-Optimierung. Terminvereinbarung f?r Pr?sentationen und Workshops Sie haben die Wahl, wie wir Ihnen unsere Produkte und Dienstleistungen vorstellen d?rfen. Vereinbaren Sie einen Termin: www.evolver.de/termin. From thomas.lecomte at virtual-expo.com Tue Apr 26 14:37:08 2016 From: thomas.lecomte at virtual-expo.com (Thomas Lecomte) Date: Tue, 26 Apr 2016 14:37:08 +0200 Subject: Asking for backend set in VCL 4 In-Reply-To: References: Message-ID: On Tue, Apr 26, 2016 at 2:26 PM, Andreas Haase - evolver group wrote: > Now, in VCL 4 there seems no possibility to ask for the backend actually > set. Is there any way to use this again, or do we have to use a > work-around like setting custom http headers and later asking for their > content? Hello, Unfortunately, since v4, you can't test req.backend_hint. I had to define a custom HTTP header every time I set req.backend_hint to keep track of it. -- Thomas Lecomte / Virtual Expo From t.honacker at googlemail.com Tue Apr 26 15:06:57 2016 From: t.honacker at googlemail.com (Tobias Honacker) Date: Tue, 26 Apr 2016 15:06:57 +0200 Subject: Asking for backend set in VCL 4 In-Reply-To: References: Message-ID: Hi, what about this code sub vcl_recv { set req.backend_hint = foo; set req.http.CurrentBackend = foo; if ( some condition ) { set req.backend_hint = bar; set req.http.CurrentBackend = bar; } if ( req.http.CurrentBackend ~ "foo" ) { /* do sth nasty */ } } Viele Gr??e / Regards Tobias 2016-04-26 14:26 GMT+02:00 Andreas Haase - evolver group < andreas.haase at evolver.de>: > Hello, > > we are just migrating one varnish from 3.x to 4.0.2 (contained in Debian > Jessie). Our old vcl contains the following structure: > > [...] > > sub vcl_recv { > set req.backend = foo; > > if (some condition) { > set req.backend = bar; > } > > [...] > > if ( req.backend = foo ) { > /* do something nasty */ > } > > [...] > } > > Now, in VCL 4 there seems no possibility to ask for the backend actually > set. Is there any way to use this again, or do we have to use a > work-around like setting custom http headers and later asking for their > content? > > Thank you in advance for any hint :-) > > -- > Regards, > > Andreas Haase > > Administration und Technik > evolver services GmbH > > Fon: +49 / (0)3 71 / 4 00 03 727 > Fax: +49 / (0)3 71 / 4 00 03 79 > > E-Mail andreas.haase at evolver.de > Web http://www.evolver.de > Facebook http://www.facebook.de/evolvergroup > > > Sitz der Gesellschaft: Chemnitz > Handelsregister: Amtsgericht Chemnitz, HRB 22649 > Gesch?ftsf?hrer: Torsten Gramann und Mathias M?ckel > > [zu unserer Webseite] > Unsere Top-Produkte: > - Stellenportal > - Immobilienportal > - evolverOAS > - evolverCAS > - evolverCMS > Unsere Top-Dienstleistungen: > - Administration & IT > - Service-Center > - garantierte Ressourcen< > http://www.evolver.de/dienstleistungen/garantierte_ressourcen/> > - Manntage-Pakete > - evolverCLOUD > SEO-Optimierung inklusive: > Zur Verbesserung Ihrer Marktchancen mit unseren Produkten investieren wir > noch mehr in die SEO-Optimierung. > Terminvereinbarung f?r Pr?sentationen und Workshops > Sie haben die Wahl, wie wir Ihnen unsere Produkte und Dienstleistungen > vorstellen d?rfen. > Vereinbaren Sie einen Termin: www.evolver.de/termin< > http://www.evolver.de/termin>. > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From andreas.haase at evolver.de Tue Apr 26 15:09:02 2016 From: andreas.haase at evolver.de (Andreas Haase - evolver group) Date: Tue, 26 Apr 2016 15:09:02 +0200 Subject: Asking for backend set in VCL 4 In-Reply-To: References: Message-ID: On Tue, 26 Apr 2016, Tobias Honacker wrote: > what about this code > > sub vcl_recv { > set req.backend_hint = foo; > set req.http.CurrentBackend = foo; > > if ( some condition ) { > set req.backend_hint = bar; > set req.http.CurrentBackend = bar; > } > > if ( req.http.CurrentBackend ~ "foo" ) { > /* do sth nasty */ > } > } that is the second option I mentioned in my initial post and this works, of course ;-) ... i just wondered whether there is a way beside setting custom headers. -- Regards, Andreas Haase Administration und Technik evolver services GmbH Fon: +49 / (0)3 71 / 4 00 03 727 Fax: +49 / (0)3 71 / 4 00 03 79 E-Mail andreas.haase at evolver.de Web http://www.evolver.de Facebook http://www.facebook.de/evolvergroup Sitz der Gesellschaft: Chemnitz Handelsregister: Amtsgericht Chemnitz, HRB 22649 Gesch?ftsf?hrer: Torsten Gramann und Mathias M?ckel [zu unserer Webseite] Unsere Top-Produkte: - Stellenportal - Immobilienportal - evolverOAS - evolverCAS - evolverCMS Unsere Top-Dienstleistungen: - Administration & IT - Service-Center - garantierte Ressourcen - Manntage-Pakete - evolverCLOUD SEO-Optimierung inklusive: Zur Verbesserung Ihrer Marktchancen mit unseren Produkten investieren wir noch mehr in die SEO-Optimierung. Terminvereinbarung f?r Pr?sentationen und Workshops Sie haben die Wahl, wie wir Ihnen unsere Produkte und Dienstleistungen vorstellen d?rfen. Vereinbaren Sie einen Termin: www.evolver.de/termin. From dridi at varni.sh Tue Apr 26 15:35:54 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 26 Apr 2016 15:35:54 +0200 Subject: VSL query scoping vs grouping Message-ID: Hello, This is following an offline discussion from more than a month ago with gquintard and martin on a bewildering case we faced. Consider the following test case: server s1 { rxreq txresp -hdr Cache-Control:max-age=3600 } -start varnish v1 -vcl+backend { sub vcl_backend_response { set beresp.ttl = 10s; } } -start logexpect l1 -v v1 -q "TTL[1] eq RFC and TTL[2] < 120" { expect * * TTL "^RFC 3600 " expect * * TTL "^VCL 10 " } -start client c1 { txreq rxresp } -run logexpect l1 -wait It passes despite the fact that none of the individual expect statements match the query. And that is of course because both conditions are matched separately against the whole vxid group. That got me and gquintard stuck for minutes and once we figured what was happening we discussed that with martin. The problem here is that there's no mechanism to match more than one criteria on a single VSL record other than a regex. And that wouldn't work with numeric comparisons like the one above (or maybe it could if you like this kind of challenges). And to achieve that we'd need a new syntax to express multiple criteria on a single record. Since we already use parenthesis and square brackets in the VSL query syntax, I believe it leaves us with curly braces. Example: Match groups where a single TTL record of type RFC is lower than 120 seconds for 404 responses. -q {TTL[1] eq RFC and TTL[2] < 120} and RespStatus == 404 It also raises questions like how to interpret {...} expressions with more than one type of record. Should it simply be an error? Should it expand the scope of the {...} expression? Complex example: Match transaction groups where a single backend transaction with the "default" backend has a TTL lower than 120 seconds, regardless of ESI sub-requests to different backends. -q {TTL[2] < 120 and Backend[3] eq default} I'm hitting the -misc list instead of the -dev, because at this point of the discussion is mostly about a hypothetical syntax for a hypothetical feature. So user feedback in this area would be valuable too. Thoughts? Best Regards, Dridi From pada at posteo.de Tue Apr 26 22:48:48 2016 From: pada at posteo.de (Daniel Parthey) Date: Tue, 26 Apr 2016 22:48:48 +0200 Subject: vmod-named does not support host header with port Message-ID: <20160426204848.GA3858@daniel.localdomain> Hi, with varnish 3.0 we use the dns director to look up the requested host header and redirect it back to our loadbalancer on the same IP as the request came in, but port 81 where our webserver is running. Curretly we are in the process of migrating to varnish 4.1. AFAIR, the DNS director has been removed from varnish4 core, because it did not support IPv6 properly. Fortunately, there is this really useful module from Dridi, which brings back this functionality into varnish4: A DNS director for Varnish 4.1 https://github.com/Dridi/libvmod-named After patching configure.ac file of libvmod-named with https://github.com/varnish/libvmod-example/commit/ad96d3b it even compiled under CentOS 7.2. After installing and importing vmod named in varnish 4.1 under CentOS 7.2, I discovered that it does not support Host headers with ports the same way like varnish3 dns director did: https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html Section 14.23 - Host - describes the Host Header syntax: Host = "Host" ":" host [ ":" port ] Unfortunately, vmod named seems to obtain the hostname for the DNS lookup directly from the req.http.host or bereq.http.host header, but if the http user agent sends the following request header: Host: www.example.org:80 The vmod named seems to be unable to look it up and the backend fetch will fail, if you provide a port number in the host header: ---------------------------------------------------------------------- $ curl -v -s -o /dev/null -H "Host: www.example.org:80" localhost:6081 * About to connect() to localhost port 6081 (#0) * Trying 127.0.0.1... * Connected to localhost (127.0.0.1) port 6081 (#0) > GET / HTTP/1.1 > User-Agent: curl/7.29.0 > Accept: */* > Host: www.example.org:80 > < HTTP/1.1 503 Backend fetch failed < Date: Tue, 26 Apr 2016 17:35:41 GMT < Server: Varnish < Content-Type: text/html; charset=utf-8 < Age: 0 < Via: 1.1 varnish-test < X-Cache: MISS from varnish-test < Content-Length: 340 < Connection: keep-alive < { [data not shown] * Connection #0 to host localhost left intact ---------------------------------------------------------------------- What would you recommend to support Host Headers including port numbers? Should I write some regsub() VCL to extract the hostname from req.http.host or bereq.http.host and pass it to the director, or should the source code of libvmod-named be changed to support RFC 2616 Host Headers with ports? Kind regards Daniel -- https://emailselfdefense.fsf.org https://pgp.mit.edu/pks/lookup?op=get&search=0xB4DD34660B6F0F1B https://pgp.key-server.io/0xB4DD34660B6F0F1B Fingerprint: 366C BC38 B09B 4BFC 38A7 3CBC B4DD 3466 0B6F 0F1B -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From ruben at varnish-software.com Wed Apr 27 03:14:59 2016 From: ruben at varnish-software.com (=?UTF-8?Q?Rub=C3=A9n_Romero?=) Date: Wed, 27 Apr 2016 03:14:59 +0200 Subject: #VarnishCon1 Is On! Amsterdam, NL - June 17th, 2016 (VDD+Training the 16th) Message-ID: Hello everyone, It is my pleasure to announce our first VarnishCon, happening on June 17th in Amsterdam. This is our yearly European user community conference and we hope you can join us. Varnish Cache users, enthusiasts and core developers will once again reunite together in one place to shape the future of our software. Everyone is invited, so RSVP now (it's free) > http://varnishcon2016.eventbrite.com/ This far the agenda is shapping up nicely: * Emanuele Rocca, Ops Eng @ Wikimedia Foundation will join us and share the story of how Wikipedia, the 7th biggest site in the web, stays up thanks to Varnish. * Poul-Henning Kamp, Varnish Chief Architect, will be holding a keynote and give us his update of the state of affairs of the Varnish project. If you have a great Varnish story you want to share with the crowd, just reply to this email or join us on the discussion on gtihub[1], Otherwise you can expect the Varnish core developer team to share insights about future plans for Varnish. Are you curious about progress with HTTP/2 support or our plan for time-based releases? You will hear more about that and will have plenty of time for questions. The day before the conference, June 16th, we plan to have training sessions (limited seats) as well as a VDD for the core developer team. If your day job is working as a web developer, sysadmin or devops engineer, you will find at least one of the two Varnish training tracks rather useful (1 day training is only ?80,-) So tell your boss that this is your chance to go to VarnishCon, get some training and learn more about how to make their site shine even more! At this event you will be able to meet, connect and learn from varnishers in the region. When you go back home you will have quite a few tips & tricks to implement and know more about what pitfalls to avoid. Get a ticket now > http://varnishcon2016.eventbrite.com/ Hope to see you all there! Best regards, -- *Rub?n Romero* Director, Community Engagement Varnish Software Group Cell: +47 95964088 / Office: +47 21989260 Skype, Twitter & IRC: ruben_varnish We Make Websites Fly! [1] https://github.com/varnishcache/homepage/issues/12 -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Wed Apr 27 08:43:43 2016 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 27 Apr 2016 08:43:43 +0200 Subject: vmod-named does not support host header with port In-Reply-To: <20160426204848.GA3858@daniel.localdomain> References: <20160426204848.GA3858@daniel.localdomain> Message-ID: Hello Daniel, > with varnish 3.0 we use the dns director to look up the requested > host header and redirect it back to our loadbalancer on the same IP > as the request came in, but port 81 where our webserver is running. > > Curretly we are in the process of migrating to varnish 4.1. I am very happy to see that vmod-named is helping you getting away from Varnish 3. However, bugs should be reported on github [1] instead. If you don't have or don't want a github account, let's keep the discussion here. > AFAIR, the DNS director has been removed from varnish4 core, > because it did not support IPv6 properly. I'm not sure that's the reason. I wasn't following the project as closely as today but I think it has to do with directors being moved outside of VCL, and the fact that the DNS director relied on the VCL compilation phase to generate backends. > After patching configure.ac file of libvmod-named with > https://github.com/varnish/libvmod-example/commit/ad96d3b > it even compiled under CentOS 7.2. I'm working on improving the build system for VMOD authors, could you please save me some work and try to build this branch [2] on your platform? > Unfortunately, vmod named seems to obtain the hostname for the DNS > lookup directly from the req.http.host or bereq.http.host header, I still haven't decided how to obtain the host header, I really need to find my varnish 3 copy in my attic to see how it used to work in ye olde times. > but if the http user agent sends the following request header: > > Host: www.example.org:80 > > The vmod named seems to be unable to look it up and the backend > fetch will fail, if you provide a port number in the host header: snip > What would you recommend to support Host Headers including port numbers? > > Should I write some regsub() VCL to extract the hostname from req.http.host > or bereq.http.host and pass it to the director, or should the source code > of libvmod-named be changed to support RFC 2616 Host Headers with ports? You may also notice that the v3 dns director has a prefix that can automagically be prepended (or appended if you read domain names from right to left=) but vmod-named doesn't do anything like that. This is the kind of things that can be done in VCL, so I'm leaning towards not implementing anything that can be done trivially in VCL. Because I don't even need to tag a release version for people to expect current master to stay backward compatible. And I'd like to avoid breaking everyone's setup too often :) That being said, I usually don't allow port numbers in host headers, because by default it is part of the hash. For two clients making the same request, with and without :80 in the host header may insert duplicate entries in the cache. So when I'm the one writing the VCL, :80 doesn't survive vcl_recv{} ;) Cheers, Dridi [1] https://github.com/Dridi/libvmod-named/issues [2] https://github.com/Dridi/varnish-modules/tree/feature/build From info at massivescale.net Wed Apr 27 16:44:25 2016 From: info at massivescale.net (Andrzej Godziuk) Date: Wed, 27 Apr 2016 16:44:25 +0200 Subject: Minimalistic varnish does not compile. In-Reply-To: <5719F3C7.207@gmail.com> References: <5719DD20.1050805@gmail.com> <27407.1461313243@critter.freebsd.dk> <5719F3C7.207@gmail.com> Message-ID: <20160427164425.21456a6b@gdr-desktop.gdr.name> > gcc 5.3.0 Is this the same gcc you've built Varnish with? From rgelfand2 at gmail.com Thu Apr 28 15:08:49 2016 From: rgelfand2 at gmail.com (Roman Gelfand) Date: Thu, 28 Apr 2016 13:08:49 +0000 Subject: Accelerating wordpress site Message-ID: My wordpress site www.pharmscript.com is slow to load full height full width page/image. Interestingly enough, the load is only slow when a full menu is showing. Will varnish help me in this case? -------------- next part -------------- An HTML attachment was scrubbed... URL: From tobias.eichelbroenner at lamp-solutions.de Thu Apr 28 15:39:42 2016 From: tobias.eichelbroenner at lamp-solutions.de (=?UTF-8?Q?Tobias_Eichelbr=c3=b6nner?=) Date: Thu, 28 Apr 2016 15:39:42 +0200 Subject: Accelerating wordpress site In-Reply-To: References: Message-ID: <5722129E.8090004@lamp-solutions.de> Hi, > My wordpress site www.pharmscript.com is > slow to load full height full width page/image. Interestingly enough, > the load is only slow when a full menu is showing. first of all, my browser tries to fetch some content from a private IP when connection to you page. http://192.168.100.122/wordpress/wp-content/uploads/2016/01/menuDivbar.png This causes the browser to wait for a timeout before showing the page fully loaded. The main page is rendered in about 1 second. Varnish can speed this up dramatically, but you are setting a session cookie. You will have to get rid of this first. Sincerely, Tobias -- LAMP solutions GmbH Gostenhofer Hauptstrasse 35 90443 Nuernberg Amtsgericht Nuernberg: HRB 22366 Geschaeftsfuehrer: Heiko Schubert Es gelten unsere allgemeinen Geschaeftsbedingungen. http://www.lamp-solutions.de/agb/ Telefon : 0911 / 376 516 0 Fax : 0911 / 376 516 11 E-Mail : support at lamp-solutions.de Web : www.lamp-solutions.de Facebook : http://www.facebook.com/LAMPsolutions Twitter : http://twitter.com/#!/lampsolutions From subharaj.manna at gmail.com Sat Apr 30 12:19:07 2016 From: subharaj.manna at gmail.com (Debraj Manna) Date: Sat, 30 Apr 2016 15:49:07 +0530 Subject: Collecting varnish stats at the Url Level Message-ID: Hi, We are using varnish to cache different urls like:- - /discovery/v1/search - /discovery/v1/suggest - /discovery/v1/recommend - /orders/ordersearch Right now we were able to get varnish stats like cache_hit, cache_miss for the overall varnish hits. Is there a way we can get the stats at the url level like cache_hit, cache_miss, etc for /discovery/v1/search, /discovery/v1/suggest separately? -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas.lecomte at virtual-expo.com Sat Apr 30 12:31:59 2016 From: thomas.lecomte at virtual-expo.com (Thomas Lecomte) Date: Sat, 30 Apr 2016 12:31:59 +0200 Subject: Collecting varnish stats at the Url Level In-Reply-To: References: Message-ID: On Sat, Apr 30, 2016 at 12:19 PM, Debraj Manna wrote: > Right now we were able to get varnish stats like cache_hit, cache_miss for > the overall varnish hits. Is there a way we can get the stats at the url > level like cache_hit, cache_miss, etc for /discovery/v1/search, > /discovery/v1/suggest separately? Hi, Varnish can't do that out of the box. You will have to use std.log() to log wether the request generated a cache miss or a cache hit, and then compute the ratio by parsing the output of varnishlog/varnishncsa (see %{VCL_log:key}x in the varnishncsa manpage). -- Thomas Lecomte / Virtual Expo