From ram9_pal at yahoo.com  Wed Sep  2 02:32:01 2015
From: ram9_pal at yahoo.com (Ram Palagummi)
Date: Tue, 1 Sep 2015 17:32:01 -0700
Subject: Varnish 4.0.2 throwing 503s - Critical Production Issue
Message-ID: <1441153921.1968.YahooMailIosMobile@web142604.mail.bf1.yahoo.com>

An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20150901/8f7207c9/attachment.html>

From lkarsten at varnish-software.com  Wed Sep  2 11:53:54 2015
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Wed, 2 Sep 2015 11:53:54 +0200
Subject: Varnish 4.0.2 throwing 503s - Critical Production Issue
In-Reply-To: <1441153921.1968.YahooMailIosMobile@web142604.mail.bf1.yahoo.com>
References: <1441153921.1968.YahooMailIosMobile@web142604.mail.bf1.yahoo.com>
Message-ID: <20150902095353.GA8099@immer.varnish-software.com>

On Tue, Sep 01, 2015 at 05:32:01PM -0700, Ram Palagummi wrote:
> <table cellspacing="0" cellpadding="0" border="0"><tr><td valign="top"><div id='yahoo__compose_area' style="background-color:white; display:block; font-family:HelveticaNeue-Regular,Helvetica;"><br>We have been using Varnish 4.0.2 and noticing lots of 503s even when back end servers are available and healthy. We don't see any issues with network either. <div><br></div><div>FYI, the following are already set in our VCL</div><div><br></div><div>Connect_timeout = 120s</div><div>First_byte_timeout = 120s</div><div>Between_bytes_timeout = 120s</div><div><br></div><div>But we still see 503s, Is there anything else that must be set to resolve this issue?</div><div><br></div><div>Thanks</div><div>Ram</div><div><br><div><br><a href="https://yho.com/footer0">Sent from Yahoo Mail for iPhone</a></div></div></div><div id='yahoo__original_message'></div></html></td></tr></table>
> 

Please post your email as plain text and not HTML.

Having timeouts that high is usually a bad idea, as you don't get the
log segments back from the worker threads before the timeout goes off.
Two minutes is a long time to wait while debuging an issue.

To get help with this you need to extract varnishlog for the 503s and
post that.

-- 
Lasse Karstensen
Varnish Software AS


From lkarsten at varnish-software.com  Wed Sep  2 12:07:57 2015
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Wed, 2 Sep 2015 12:07:57 +0200
Subject: varnish panic investigation information
In-Reply-To: <55E0594F.8010303@redlab.be>
References: <55E0594F.8010303@redlab.be>
Message-ID: <20150902100756.GB8099@immer.varnish-software.com>

On Fri, Aug 28, 2015 at 02:51:27PM +0200, Balder VC wrote:
> I'm requesting some help or info for a panic we had.

Hi.

Having had a quick look at it, I'd say either a bug or some random
hardware error.

Please file this on the bugtracker:

    https://www.varnish-cache.org/trac/newticket


You may need to register an account, let me know if you run into
trouble.

-- 
Lasse Karstensen
Varnish Software AS


From cello86 at gmail.com  Wed Sep  2 13:08:43 2015
From: cello86 at gmail.com (Marcello Lorenzi)
Date: Wed, 2 Sep 2015 13:08:43 +0200
Subject: varnish 4 503 issue
Message-ID: <CACT02+7HXHR=33m7gOjHeApk2_L5eko3PYsFj7JLA_NC=Cj5fg@mail.gmail.com>

Hi All,
we installed a varnish 4.1.0 instance on our stage environment and we
configured the cache to probe 2 backends on different servers.

we noticed that in same cases the cache reports 503 error and into the
varnishlog entries we found the Backend fetch failed error, but if we
verify the backend status on varnishadm the backend are healthy.

Could you help us to check this issue?

Thanks,
Marcello
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20150902/c1dc8109/attachment.html>

From ram9_pal at yahoo.com  Wed Sep  2 20:15:26 2015
From: ram9_pal at yahoo.com (Ram Palagummi)
Date: Wed, 2 Sep 2015 11:15:26 -0700
Subject: Varnish 4.0.2 throwing 503s - Critical Production Issue
In-Reply-To: <20150902095353.GA8099@immer.varnish-software.com>
Message-ID: <1441217726.62452.YahooMailIosMobile@web142602.mail.bf1.yahoo.com>

An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20150902/20e7c194/attachment.html>

From dridi at varni.sh  Fri Sep  4 09:38:32 2015
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Fri, 4 Sep 2015 09:38:32 +0200
Subject: varnish 4 503 issue
In-Reply-To: <CACT02+7HXHR=33m7gOjHeApk2_L5eko3PYsFj7JLA_NC=Cj5fg@mail.gmail.com>
References: <CACT02+7HXHR=33m7gOjHeApk2_L5eko3PYsFj7JLA_NC=Cj5fg@mail.gmail.com>
Message-ID: <CABoVN9C_j1MgLirUu7x=MfXKmF2aVwvmWbzEC6gZBy=sgLEb0w@mail.gmail.com>

On Wed, Sep 2, 2015 at 1:08 PM, Marcello Lorenzi <cello86 at gmail.com> wrote:
> Hi All,
> we installed a varnish 4.1.0 instance on our stage environment and we
> configured the cache to probe 2 backends on different servers.
>
> we noticed that in same cases the cache reports 503 error and into the
> varnishlog entries we found the Backend fetch failed error, but if we verify
> the backend status on varnishadm the backend are healthy.
>
> Could you help us to check this issue?

Hi Marcello,

Thank you for trying the 4.1 series, and be aware it is not yet a
stable release.

Could you share some VCL and varnishlog output to help understand
your problem?

Best Regards,
Dridi


From Benjamin.Parmentier at worldline.com  Fri Sep  4 15:51:12 2015
From: Benjamin.Parmentier at worldline.com (Parmentier Benjamin)
Date: Fri, 4 Sep 2015 15:51:12 +0200
Subject: Cannot figure out why, sometimes, I am getting a miss
Message-ID: <CC5E25F0501F7E46844AB0F62DFBE06DD944FECA2E@FRSPX100.fr01.awl.atosorigin.net>

Hi,

I have a varnish 3.0.6 in-between a F5 Big-IP and an apache backend :

F5 Big-IP -> varnish -> apache


I have a request that I set a TTL for 24h but I sometimes get miss on this URL even if the TTL is not reached and I don't know what is causing this...


Extract from varnishncsa :

78.195.19.79 - - [04/Sep/2015:14:58:20 +0200] "GET http://www.mydomain.tld/services/info/client/16/sdm/config? HTTP/1.1" 200 38 "-" "Info/2.9.2 (iPad; iOS 8.2; Scale/2.00)" hit 134
77.152.91.156 - - [04/Sep/2015:14:59:22 +0200] "GET http://www.mydomain.tld/services/info/client/16/sdm/config? HTTP/1.1" 200 38 "-" "Info/2.9.2 (iPad; iOS 7.0.6; Scale/1.00)" miss 2685


The first request hit the cache :

  124 ReqStart     c 10.98.40.11 61080 349021016
  124 RxRequest    c GET
  124 RxURL        c /services/info/client/16/sdm/config?
  124 RxProtocol   c HTTP/1.1
  124 RxHeader     c Host: www.mydomain.tld<http://www.mydomain.tld>
  124 RxHeader     c Accept: */*
  124 RxHeader     c Connection: keep-alive
  124 RxHeader     c Cookie: UID=69740d-185.43.182.11-1426874236; UIDR=1426874236; s1=550C600D31A40171; s1=550C60007096022A; ai_session=92dd6dfb13354c93ada5cf9abb619e32|2015-07-02T11:35:24.6019731+00:00|
2015-07-02T11:35:24.9926130+00:00; ai_user=e271bdbc381d4212a07bc0b02e0e32
  124 RxHeader     c User-Agent: Hippique/2.9.2 (iPad; iOS 8.2; Scale/2.00)
  124 RxHeader     c Accept-Language: fr;q=1
  124 RxHeader     c Info-Session-Id: 72440dc7e08b268927e6cbe9f3842b70
  124 RxHeader     c Accept-Encoding: gzip, deflate
  124 RxHeader     c X-Forwarded-For: 78.195.19.79
  124 VCL_call     c recv lookup
  124 VCL_call     c hash
  124 Hash         c /services/info/client/16/sdm/config
  124 Hash         c www.mydomain.tld<http://www.mydomain.tld>
  124 VCL_return   c hash
  124 Hit          c 346131254
  124 VCL_call     c hit deliver
  124 VCL_call     c deliver deliver
  124 TxProtocol   c HTTP/1.1
  124 TxStatus     c 200
  124 TxResponse   c OK
  124 TxHeader     c Server: Apache
  124 TxHeader     c Vary: Accept-Encoding,User-Agent
  124 TxHeader     c Content-Encoding: gzip
  124 TxHeader     c Content-Type: application/json;charset=UTF-8
  124 TxHeader     c Content-Length: 38
  124 TxHeader     c Accept-Ranges: bytes
  124 TxHeader     c Date: Fri, 04 Sep 2015 12:59:21 GMT
  124 TxHeader     c X-Varnish: 349021016 346131254
  124 TxHeader     c Age: 82715
  124 TxHeader     c Via: 1.1 varnish
  124 TxHeader     c Connection: keep-alive
  124 TxHeader     c Var-Cache: HIT
  124 Length       c 38
  124 ReqEnd       c 349021016 1441371561.449951649 1441371561.450107813 0.000079632 0.000083208 0.000072956



One minute later the second request miss the cache and fetch the backend :

   53 SessionOpen  c 10.98.40.12 24242 0.0.0.0:84
   53 ReqStart     c 10.98.40.12 24242 349021058
   53 RxRequest    c GET
   53 RxURL        c /services/info/client/16/sdm/config?
   53 RxProtocol   c HTTP/1.1
   53 RxHeader     c Host: www.mydomain.tld<http://www.mydomain.tld>
   53 RxHeader     c Accept: */*
   53 RxHeader     c Accept-Encoding: gzip, deflate
   53 RxHeader     c Accept-Language: fr;q=1, en;q=0.9, de;q=0.8, ja;q=0.7, nl;q=0.6, it;q=0.5
   53 RxHeader     c Connection: keep-alive
   53 RxHeader     c Cookie: matchfbx=1; wfivefivec=scx3J3CkSI312SXL3Fo06SRLnunHpCoS; ptc=%7b%22UserSyncItems%22%3a%5b%7b%22PartnerId%22%3a%2229%22%2c%22PartnerUserId%22%3anull%2c%22TimeStamp%22%3a%2201%2f21%2f2015+12%3a49%3a20%22%7d%5d%7d; tidal_ttid=1a27cbd8-4429-44f2-9f31-
   53 RxHeader     c User-Agent: Hippique/2.9.2 (iPad; iOS 7.0.6; Scale/1.00)
   53 RxHeader     c X-Forwarded-For: 77.152.91.156
   53 VCL_call     c recv lookup
   53 VCL_call     c hash
   53 Hash         c /services/info/client/16/sdm/config
   53 Hash         c www.mydomain.tld<http://www.mydomain.tld>
   53 VCL_return   c hash
   53 VCL_call     c miss fetch
   53 Backend      c 45 info_varnish APACHE_BACKEND
   53 TTL          c 349021058 RFC 120 -1 -1 1441371562 0 1441371562 0 0
   53 VCL_call     c fetch
   53 TTL          c 349021058 VCL 86400 -1 -1 1441371562 -0
   53 VCL_return   c deliver
   53 ObjProtocol  c HTTP/1.1
   53 ObjResponse  c OK
   53 ObjHeader    c Date: Fri, 04 Sep 2015 12:59:22 GMT
   53 ObjHeader    c Server: Apache
   53 ObjHeader    c Vary: Accept-Encoding,User-Agent
   53 ObjHeader    c Content-Encoding: gzip
   53 ObjHeader    c Content-Type: application/json;charset=UTF-8
   53 Gzip         c u F - 38 18 80 80 234
   53 VCL_call     c deliver deliver
   53 TxProtocol   c HTTP/1.1
   53 TxStatus     c 200
   53 TxResponse   c OK
   53 TxHeader     c Server: Apache
   53 TxHeader     c Vary: Accept-Encoding,User-Agent
   53 TxHeader     c Content-Encoding: gzip
   53 TxHeader     c Content-Type: application/json;charset=UTF-8
   53 TxHeader     c Content-Length: 38
   53 TxHeader     c Accept-Ranges: bytes
   53 TxHeader     c Date: Fri, 04 Sep 2015 12:59:22 GMT
   53 TxHeader     c X-Varnish: 349021058
   53 TxHeader     c Age: 0
   53 TxHeader     c Via: 1.1 varnish
   53 TxHeader     c Connection: keep-alive
   53 TxHeader     c Var-Cache: MISS
   53 Length       c 38
   53 ReqEnd       c 349021058 1441371562.030749321 1441371562.033434391 0.000071526 0.002643108 0.000041962




Here my vcl config :


backend APACHE_BACKEND {
        .host = "127.0.0.1";
        .port = "80";
        .probe = {
                .url = "/surveillance/.surveillance.txt";
                .interval = 10s;
                .timeout = 1s;
                .window = 5;
                .threshold = 3;
        }
}

director info_varnish random {
  {  .backend = APACHE_BACKEND ;
     .weight = 1 ;
  }
}

sub normalize_req_url {
  # strip out "_" querystring values so varnish does not vary cache.
  set req.url = regsuball(req.url, "([\?|&])_=[^&\s]*&?", "\1");
  # get rid of trailing & or ?
  set req.url = regsuball(req.url, "[\?|&]+$", "");
}

sub vcl_recv {
  set req.backend = info_varnish;

  if ( req.url ~ "^.*/info/client/\d+/programme[s]?/.*/part.*$" ||
       req.url ~ "^.*/info/client/\d+/programme[s]?/\d{8,8}/R\d{1,2}/C\d{1,2}.*$" ||
       req.url ~ "^.*/info/client/\d+/multimedia/videos/streaming.*$" ||
       req.url ~ "^.*/info/client/\d+/sdm/config.*$" ) {
    call normalize_req_url;
    return(lookup);
  }

  return(pass);
}

sub vcl_fetch {
  if ( req.url ~ "^.*/info/client/\d+/programme[s]?/.*/participants.*$" ) {
    set beresp.ttl = 5s;
  } else if ( req.url ~ "^.*/info/client/\d+/programme[s]?/\d{8,8}/R\d{1,2}/C\d{1,2}.*$" ) {
    set beresp.ttl = 900s;
  } else if ( req.url ~ "^.*/info/client/\d+/multimedia/videos/streaming.*$" ) {
    set beresp.ttl = 60s;
  } else if ( req.url ~ "^.*/info/client/\d+/sdm/config.*$" ) {
    set beresp.ttl = 86400s;
  }

  return(deliver);
}

sub vcl_deliver {
  # Set a header to track a cache HIT/MISS.
  if (obj.hits > 0) {
    set resp.http.Var-Cache = "HIT";
  } else {
    set resp.http.Var-Cache = "MISS";
  }
}


Any ideas ?


Regards,
Benjamin

________________________________

Ce message et les pi?ces jointes sont confidentiels et r?serv?s ? l'usage exclusif de ses destinataires. Il peut ?galement ?tre prot?g? par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir imm?diatement l'exp?diteur et de le d?truire. L'int?grit? du message ne pouvant ?tre assur?e sur Internet, la responsabilit? de Worldline ne pourra ?tre recherch?e quant au contenu de ce message. Bien que les meilleurs efforts soient faits pour maintenir cette transmission exempte de tout virus, l'exp?diteur ne donne aucune garantie ? cet ?gard et sa responsabilit? ne saurait ?tre recherch?e pour tout dommage r?sultant d'un virus transmis.

This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Worldline liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20150904/8e9b07b8/attachment-0001.html>

From thierry.magnien at sfr.com  Fri Sep  4 16:21:28 2015
From: thierry.magnien at sfr.com (MAGNIEN, Thierry)
Date: Fri, 4 Sep 2015 14:21:28 +0000
Subject: Cannot figure out why, sometimes, I am getting a miss
In-Reply-To: <CC5E25F0501F7E46844AB0F62DFBE06DD944FECA2E@FRSPX100.fr01.awl.atosorigin.net>
References: <CC5E25F0501F7E46844AB0F62DFBE06DD944FECA2E@FRSPX100.fr01.awl.atosorigin.net>
Message-ID: <5D103CE839D50E4CBC62C9FD7B83287C010D7B1A77@EXCN015.encara.local.ads>

Hi,

Your Apache sends a "Vary: Accept-Encoding,User-Agent" header, which means you cache a different object for each User-Agent string.

The two requests having a different user agent, you get a miss on the second one even if the first is in cache :

User-Agent: Hippique/2.9.2 (iPad; iOS 8.2; Scale/2.00)
and
User-Agent: Hippique/2.9.2 (iPad; iOS 7.0.6; Scale/1.00)

Best regards,
Thierry

-----------------------------
De?: varnish-misc-bounces+thierry.magnien=sfr.com at varnish-cache.org [mailto:varnish-misc-bounces+thierry.magnien=sfr.com at varnish-cache.org] De la part de Parmentier Benjamin
Envoy??: vendredi 4 septembre 2015 15:51
??: varnish-misc at varnish-cache.org
Objet?: Cannot figure out why, sometimes, I am getting a miss

Hi,

I have a varnish 3.0.6 in-between a F5 Big-IP and an apache backend?:

F5 Big-IP -> varnish -> apache


I have a request that I set a TTL for 24h but I sometimes get miss on this URL even if the TTL is not reached and I don't know what is causing this.


Extract from varnishncsa?:

78.195.19.79 - - [04/Sep/2015:14:58:20 +0200] "GET http://www.mydomain.tld/services/info/client/16/sdm/config? HTTP/1.1" 200 38 "-" "Info/2.9.2 (iPad; iOS 8.2; Scale/2.00)" hit 134
77.152.91.156 - - [04/Sep/2015:14:59:22 +0200] "GET http://www.mydomain.tld/services/info/client/16/sdm/config? HTTP/1.1" 200 38 "-" "Info/2.9.2 (iPad; iOS 7.0.6; Scale/1.00)" miss 2685


The first request hit the cache?:

? 124 ReqStart???? c 10.98.40.11 61080 349021016
? 124 RxRequest??? c GET
? 124 RxURL??????? c /services/info/client/16/sdm/config?
? 124 RxProtocol?? c HTTP/1.1
? 124 RxHeader???? c Host: www.mydomain.tld
? 124 RxHeader???? c Accept: */*
? 124 RxHeader???? c Connection: keep-alive
? 124 RxHeader???? c Cookie: UID=69740d-185.43.182.11-1426874236; UIDR=1426874236; s1=550C600D31A40171; s1=550C60007096022A; ai_session=92dd6dfb13354c93ada5cf9abb619e32|2015-07-02T11:35:24.6019731+00:00|
2015-07-02T11:35:24.9926130+00:00; ai_user=e271bdbc381d4212a07bc0b02e0e32
? 124 RxHeader???? c User-Agent: Hippique/2.9.2 (iPad; iOS 8.2; Scale/2.00)
? 124 RxHeader???? c Accept-Language: fr;q=1
? 124 RxHeader???? c Info-Session-Id: 72440dc7e08b268927e6cbe9f3842b70
? 124 RxHeader???? c Accept-Encoding: gzip, deflate
? 124 RxHeader???? c X-Forwarded-For: 78.195.19.79
? 124 VCL_call???? c recv lookup
? 124 VCL_call???? c hash
? 124 Hash???????? c /services/info/client/16/sdm/config
? 124 Hash???????? c www.mydomain.tld
? 124 VCL_return?? c hash
? 124 Hit????????? c 346131254
? 124 VCL_call???? c hit deliver
? 124 VCL_call???? c deliver deliver
? 124 TxProtocol?? c HTTP/1.1
? 124 TxStatus???? c 200
? 124 TxResponse?? c OK
? 124 TxHeader???? c Server: Apache
? 124 TxHeader???? c Vary: Accept-Encoding,User-Agent
? 124 TxHeader???? c Content-Encoding: gzip
? 124 TxHeader???? c Content-Type: application/json;charset=UTF-8
? 124 TxHeader???? c Content-Length: 38
? 124 TxHeader???? c Accept-Ranges: bytes
? 124 TxHeader? ???c Date: Fri, 04 Sep 2015 12:59:21 GMT
? 124 TxHeader???? c X-Varnish: 349021016 346131254
? 124 TxHeader???? c Age: 82715
? 124 TxHeader???? c Via: 1.1 varnish
? 124 TxHeader???? c Connection: keep-alive
? 124 TxHeader???? c Var-Cache: HIT
? 124 Length?????? c 38
? 124 ReqEnd?????? c 349021016 1441371561.449951649 1441371561.450107813 0.000079632 0.000083208 0.000072956



One minute later the second request miss the cache and fetch the backend?:

?? 53 SessionOpen? c 10.98.40.12 24242 0.0.0.0:84
?? 53 ReqStart???? c 10.98.40.12 24242 349021058
?? 53 RxRequest??? c GET
?? 53 RxURL??????? c /services/info/client/16/sdm/config?
?? 53 RxProtocol?? c HTTP/1.1
?? 53 RxHeader???? c Host: www.mydomain.tld
?? 53 RxHeader???? c Accept: */*
?? 53 RxHeader???? c Accept-Encoding: gzip, deflate
?? 53 RxHeader???? c Accept-Language: fr;q=1, en;q=0.9, de;q=0.8, ja;q=0.7, nl;q=0.6, it;q=0.5
?? 53 RxHeader???? c Connection: keep-alive
?? 53 RxHeader???? c Cookie: matchfbx=1; wfivefivec=scx3J3CkSI312SXL3Fo06SRLnunHpCoS; ptc=%7b%22UserSyncItems%22%3a%5b%7b%22PartnerId%22%3a%2229%22%2c%22PartnerUserId%22%3anull%2c%22TimeStamp%22%3a%2201%2f21%2f2015+12%3a49%3a20%22%7d%5d%7d; tidal_ttid=1a27cbd8-4429-44f2-9f31-
?? 53 RxHeader???? c User-Agent: Hippique/2.9.2 (iPad; iOS 7.0.6; Scale/1.00)
?? 53 RxHeader???? c X-Forwarded-For: 77.152.91.156
?? 53 VCL_call???? c recv lookup
?? 53 VCL_call???? c hash
?? 53 Hash???????? c /services/info/client/16/sdm/config
?? 53 Hash???????? c www.mydomain.tld
?? 53 VCL_return?? c hash
?? 53 VCL_call???? c miss fetch
?? 53 Backend????? c 45 info_varnish APACHE_BACKEND
?? 53 TTL????????? c 349021058 RFC 120 -1 -1 1441371562 0 1441371562 0 0
?? 53 VCL_call???? c fetch
?? 53 TTL????????? c 349021058 VCL 86400 -1 -1 1441371562 -0
?? 53 VCL_return?? c deliver
?? 53 ObjProtocol? c HTTP/1.1
?? 53 ObjResponse? c OK
?? 53 ObjHeader??? c Date: Fri, 04 Sep 2015 12:59:22 GMT
?? 53 ObjHeader??? c Server: Apache
?? 53 ObjHeader??? c Vary: Accept-Encoding,User-Agent
?? 53 ObjHeader??? c Content-Encoding: gzip
?? 53 ObjHeader??? c Content-Type: application/json;charset=UTF-8
?? 53 Gzip???????? c u F - 38 18 80 80 234
?? 53 VCL_call???? c deliver deliver
?? 53 TxProtocol?? c HTTP/1.1
?? 53 TxStatus???? c 200
?? 53 TxResponse?? c OK
?? 53 TxHeader???? c Server: Apache
?? 53 TxHeader???? c Vary: Accept-Encoding,User-Agent
?? 53 TxHeader???? c Content-Encoding: gzip
?? 53 TxHeader???? c Content-Type: application/json;charset=UTF-8
?? 53 TxHeader???? c Content-Length: 38
?? 53 TxHeader???? c Accept-Ranges: bytes
?? 53 TxHeader???? c Date: Fri, 04 Sep 2015 12:59:22 GMT
?? 53 TxHeader???? c X-Varnish: 349021058
?? 53 TxHeader???? c Age: 0
?? 53 TxHeader???? c Via: 1.1 varnish
?? 53 TxHeader???? c Connection: keep-alive
?? 53 TxHeader???? c Var-Cache: MISS
?? 53 Length?????? c 38
?? 53 ReqEnd?????? c 349021058 1441371562.030749321 1441371562.033434391 0.000071526 0.002643108 0.000041962




Here my vcl config?:


backend APACHE_BACKEND {
??????? .host = "127.0.0.1";
??????? .port = "80";
??????? .probe = {
??????????????? .url = "/surveillance/.surveillance.txt";
??????????????? .interval = 10s;
??????????????? .timeout = 1s;
??????????????? .window = 5;
??????????????? .threshold = 3;
??????? }
}

director info_varnish random {
? {? .backend = APACHE_BACKEND ;
???? .weight = 1 ;
? }
}

sub normalize_req_url {
? # strip out "_" querystring values so varnish does not vary cache.
? set req.url = regsuball(req.url, "([\?|&])_=[^&\s]*&?", "\1");
? # get rid of trailing & or ?
? set req.url = regsuball(req.url, "[\?|&]+$", "");
}

sub vcl_recv {
? set req.backend = info_varnish;

? if ( req.url ~ "^.*/info/client/\d+/programme[s]?/.*/part.*$" ||
? ?????req.url ~ "^.*/info/client/\d+/programme[s]?/\d{8,8}/R\d{1,2}/C\d{1,2}.*$" ||
?????? req.url ~ "^.*/info/client/\d+/multimedia/videos/streaming.*$" ||
?????? req.url ~ "^.*/info/client/\d+/sdm/config.*$" ) {
??? call normalize_req_url;
??? return(lookup);
? }

? return(pass);
}

sub vcl_fetch {
? if ( req.url ~ "^.*/info/client/\d+/programme[s]?/.*/participants.*$" ) {
??? set beresp.ttl = 5s;
? } else if ( req.url ~ "^.*/info/client/\d+/programme[s]?/\d{8,8}/R\d{1,2}/C\d{1,2}.*$" ) {
??? set beresp.ttl = 900s;
? } else if ( req.url ~ "^.*/info/client/\d+/multimedia/videos/streaming.*$" ) {
??? set beresp.ttl = 60s;
? } else if ( req.url ~ "^.*/info/client/\d+/sdm/config.*$" ) {
??? set beresp.ttl = 86400s;
? }

? return(deliver);
}

sub vcl_deliver {
? # Set a header to track a cache HIT/MISS.
? if (obj.hits > 0) {
??? set resp.http.Var-Cache = "HIT";
? } else {
??? set resp.http.Var-Cache = "MISS";
? }
}


Any ideas??


Regards,
Benjamin

________________________________________

Ce message et les pi?ces jointes sont confidentiels et r?serv?s ? l'usage exclusif de ses destinataires. Il peut ?galement ?tre prot?g? par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir imm?diatement l'exp?diteur et de le d?truire. L'int?grit? du message ne pouvant ?tre assur?e sur Internet, la responsabilit? de Worldline ne pourra ?tre recherch?e quant au contenu de ce message. Bien que les meilleurs efforts soient faits pour maintenir cette transmission exempte de tout virus, l'exp?diteur ne donne aucune garantie ? cet ?gard et sa responsabilit? ne saurait ?tre recherch?e pour tout dommage r?sultant d'un virus transmis.

This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Worldline liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.


From shiva2110 at gmail.com  Fri Sep  4 17:02:38 2015
From: shiva2110 at gmail.com (Shiva)
Date: Fri, 4 Sep 2015 08:02:38 -0700
Subject: cache-control header in request dropped by varnish
Message-ID: <CACOJb1FtFex8gVvsJO5KeJv1h4WMhT_zbTUBMw8tELSimLt9_A@mail.gmail.com>

Hi,
We use varnish v3.0.4 , we are seeing this issue where clients are sending
cache-control request header to varnish, but it is getting dropped before
sending to backend app.

I could see varnish receiving the header, also could see RxHeader having
cache-control header from varnishlog. Our backend server is on tomcat, in
tomcat access log the cache-control header is not seen.

Is this an expected behavior ? RFC spec on cache-control header seem to
mandate all caches to pass the header to the backend.

Thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20150904/442c9810/attachment.html>

From morphey at morphey.org  Tue Sep  8 09:15:45 2015
From: morphey at morphey.org (Christian Cantinelli)
Date: Tue, 8 Sep 2015 09:15:45 +0200
Subject: Varnish and googlebot 403
Message-ID: <55EE8B21.9020506@morphey.org>

  Hello,

  I have a problem with varnish. Basically it works all right. But a few
  months googlebot tells me many errors 403 that is not derived from
  wordpress's underneath.
  Has anyone encountered similar problems?


-- 

morphey


From matthewm at boedicker.org  Fri Sep 11 09:09:15 2015
From: matthewm at boedicker.org (Matthew Boedicker)
Date: Fri, 11 Sep 2015 00:09:15 -0700
Subject: varnish removing Transfer-Encoding: chunked header
Message-ID: <CANfC_4JDBMOpjm=mwOe4yWL--cDS6NWO863bj=kAe9BSGXFvAQ@mail.gmail.com>

I have a rails app which returns 200 with an empty body as a response to a
POST. It does not set a Content-Length header, but does set a
Transfer-Encoding chunked header.

Varnish is removing the Transfer-Encoding header so that the response has
neither chunked nor content-length. This invalid response is causing the
upstream load balancer to return bad gateway.

I am using varnish 4.0.2-1. VCL is nearly the default VCL.

Is this expected behavior or should I open a bug?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20150911/2c178de2/attachment.html>

From phk at phk.freebsd.dk  Fri Sep 11 09:37:27 2015
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Fri, 11 Sep 2015 07:37:27 +0000
Subject: varnish removing Transfer-Encoding: chunked header
In-Reply-To: <CANfC_4JDBMOpjm=mwOe4yWL--cDS6NWO863bj=kAe9BSGXFvAQ@mail.gmail.com>
References: <CANfC_4JDBMOpjm=mwOe4yWL--cDS6NWO863bj=kAe9BSGXFvAQ@mail.gmail.com>
Message-ID: <72925.1441957047@critter.freebsd.dk>

--------
In message <CANfC_4JDBMOpjm=mwOe4yWL--cDS6NWO863bj=kAe9BSGXFvAQ at mail.gmail.com>
, Matthew Boedicker writes:

>Is this expected behavior or should I open a bug?

Open a bug please.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From morphey at morphey.org  Fri Sep 11 09:40:20 2015
From: morphey at morphey.org (Christian Cantinelli)
Date: Fri, 11 Sep 2015 09:40:20 +0200
Subject: Varnish and googlebot 403
In-Reply-To: <55EE8B21.9020506@morphey.org>
References: <55EE8B21.9020506@morphey.org>
Message-ID: <55F28564.1060504@morphey.org>

Hello,

has anyone had such this problem?

Christian

Il 08/09/2015 09:15, Christian Cantinelli ha scritto:
>   Hello,
>
>   I have a problem with varnish. Basically it works all right. But a few
>   months googlebot tells me many errors 403 that is not derived from
>   wordpress's underneath.
>   Has anyone encountered similar problems?
>
>


From lists at redlab.be  Fri Sep 11 11:29:47 2015
From: lists at redlab.be (Balder VC)
Date: Fri, 11 Sep 2015 11:29:47 +0200
Subject: Varnish and googlebot 403
In-Reply-To: <55F28564.1060504@morphey.org>
References: <55EE8B21.9020506@morphey.org> <55F28564.1060504@morphey.org>
Message-ID: <55F29F0B.7070703@redlab.be>

On 09/11/2015 09:40 AM, Christian Cantinelli wrote:
> Hello,
>
> has anyone had such this problem?


I would seem as if no one had this problem.

But prehaps there something in your vcl what causes it? Can you share it

>
> Christian
>
> Il 08/09/2015 09:15, Christian Cantinelli ha scritto:
>>   Hello,
>>
>>   I have a problem with varnish. Basically it works all right. But a few
>>   months googlebot tells me many errors 403 that is not derived from
>>   wordpress's underneath.
>>   Has anyone encountered similar problems?
>>
>>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc



From morphey at morphey.org  Fri Sep 11 13:36:37 2015
From: morphey at morphey.org (Christian Cantinelli)
Date: Fri, 11 Sep 2015 13:36:37 +0200
Subject: Varnish and googlebot 403
In-Reply-To: <55F29F0B.7070703@redlab.be>
References: <55EE8B21.9020506@morphey.org> <55F28564.1060504@morphey.org>
 <55F29F0B.7070703@redlab.be>
Message-ID: <55F2BCC5.7030702@morphey.org>

Yes, sure

In attach..


Christian


Il 11/09/2015 11:29, Balder VC ha scritto:
> On 09/11/2015 09:40 AM, Christian Cantinelli wrote:
>> Hello,
>>
>> has anyone had such this problem?
>
>
> I would seem as if no one had this problem.
>
> But prehaps there something in your vcl what causes it? Can you share it
>
>>
>> Christian
>>
>> Il 08/09/2015 09:15, Christian Cantinelli ha scritto:
>>>    Hello,
>>>
>>>    I have a problem with varnish. Basically it works all right. But a few
>>>    months googlebot tells me many errors 403 that is not derived from
>>>    wordpress's underneath.
>>>    Has anyone encountered similar problems?
>>>
>>>
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
#
# This is an example VCL file for Varnish.
#
# It does not do anything by default, delegating control to the
# builtin VCL. The builtin VCL is called when there is no explicit
# return statement.
#
# See the VCL chapters in the Users Guide at https://www.varnish-cache.org/docs/
# and http://varnish-cache.org/trac/wiki/VCLExamples for more examples.

# Update for work with Varnish 4


# Marker to tell the VCL compiler that this VCL has been adapted to the
# new 4.0 format.
vcl 4.0;

# Default backend definition. Set this to point to your content server.
backend default {
    .host = "MI_IP_SERVER_APACHE";
    .port = "8080";
    .connect_timeout = 600s;
    .first_byte_timeout = 600s;
    .between_bytes_timeout = 600s;
    .max_connections = 5000;
}

# Only allow purging from specific IPs
acl purge {
    "localhost";
    "127.0.0.1";
}

# Traffico sviluppo nocache
acl passem {
	"MY_DEV/27";
}

# This function is used when a request is send by a HTTP client (Browser) 
sub vcl_recv {

	# Normalize the header, remove the port (in case you're testing this on various TCP ports)
	set req.http.Host = regsub(req.http.Host, ":[0-9]+", "");

	# per logs
	if (req.restarts == 0) {
		if (req.http.X-Forwarded-For) {
			set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip;
		} else {
			set req.http.X-Forwarded-For = client.ip;
		}
	}
	#remove req.http.X-Forwarded-For;
	#set req.http.X-Forwarded-For = client.ip;

        if (req.http.Host ~ "beta.mydomain1.tld") {
                return (pass);
        }
        if (req.http.Host ~ "beta.mydomain2.tld") {
                return (pass);
        }

	# Traffico sviluppo nocache
	if (client.ip ~ passem) {
		return (pass);
	}
        if (req.url ~ "\?gbjson$") {
                return (pass);
        }



	# Allow purging from ACL
	if (req.method == "PURGE") {
		# If not allowed then a error 405 is returned
		if (!client.ip ~ purge) {
			return(synth(405, "This IP is not allowed to send PURGE requests."));
		}	
		# If allowed, do a cache_lookup -> vlc_hit() or vlc_miss()
		return (purge);
	}

	# Post requests will not be cached
	if (req.http.Authorization || req.method == "POST") {
		return (pass);
	}

	if (req.url ~ "\.(css)$") {
		return(pass);
	}
	if (req.url ~ "\.(xml)$") {
		return(pass);
	}
	# --- Wordpress specific configuration
	
	# Did not cache the RSS feed
	if (req.url ~ "/feed" || req.url ~ "/?feed=sitemap-news") {
		return (pass);
	}

	# Blitz hack
        if (req.url ~ "/mu-.*") {
                return (pass);
        }

	
	# Did not cache the admin and login pages
	if (req.url ~ "/wp-(login|admin)") {
		return (pass);
	}
	
	 # Do not cache the WooCommerce pages
	 ### REMOVE IT IF YOU DO NOT USE WOOCOMMERCE ###
	if (req.url ~ "/(cart|my-account|checkout|addons|/?add-to-cart=)") {
        	return (pass);
    	}

	# Remove the "has_js" cookie
	set req.http.Cookie = regsuball(req.http.Cookie, "has_js=[^;]+(; )?", "");

	# Remove any Google Analytics based cookies
	set req.http.Cookie = regsuball(req.http.Cookie, "__utm.=[^;]+(; )?", "");

	# Remove the Quant Capital cookies (added by some plugin, all __qca)
	set req.http.Cookie = regsuball(req.http.Cookie, "__qc.=[^;]+(; )?", "");

	# Remove the wp-settings-1 cookie
	set req.http.Cookie = regsuball(req.http.Cookie, "wp-settings-1=[^;]+(; )?", "");

	# Remove the wp-settings-time-1 cookie
	set req.http.Cookie = regsuball(req.http.Cookie, "wp-settings-time-1=[^;]+(; )?", "");

	# Remove the wp test cookie
	set req.http.Cookie = regsuball(req.http.Cookie, "wordpress_test_cookie=[^;]+(; )?", "");

	# Are there cookies left with only spaces or that are empty?
	if (req.http.cookie ~ "^ *$") {
		    unset req.http.cookie;
	}
	
	# Cache the following files extensions 
	if (req.url ~ "\.(css|js|png|gif|jp(e)?g|swf|ico)") {
		unset req.http.cookie;
	}

	# Normalize Accept-Encoding header and compression
	# https://www.varnish-cache.org/docs/3.0/tutorial/vary.html
	if (req.http.Accept-Encoding) {
		# Do no compress compressed files...
		if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
			   	unset req.http.Accept-Encoding;
		} elsif (req.http.Accept-Encoding ~ "gzip") {
		    	set req.http.Accept-Encoding = "gzip";
		} elsif (req.http.Accept-Encoding ~ "deflate") {
		    	set req.http.Accept-Encoding = "deflate";
		} else {
			unset req.http.Accept-Encoding;
		}
	}

	# Check the cookies for wordpress-specific items
	if (req.http.Cookie ~ "wordpress_" || req.http.Cookie ~ "comment_") {
		return (pass);
	}
	if (!req.http.cookie) {
		unset req.http.cookie;
	}
	
	# --- End of Wordpress specific configuration

	# Did not cache HTTP authentication and HTTP Cookie
	if (req.http.Authorization || req.http.Cookie) {
		# Not cacheable by default
		return (pass);
	}

	# Cache all others requests
	return (hash);
}
 
sub vcl_pipe {
	set bereq.http.connection = "close";
	return (pipe);
}
 
sub vcl_pass {
	return (fetch);
}
 
# The data on which the hashing will take place
sub vcl_hash {
 	hash_data(req.url);
 	if (req.http.host) {
     	hash_data(req.http.host);
 	} else {
     	hash_data(server.ip);
 	}

	# If the client supports compression, keep that in a different cache
    	if (req.http.Accept-Encoding) {
        	hash_data(req.http.Accept-Encoding);
	}
     
	return (lookup);
}
 
# This function is used when a request is sent by our backend (Nginx server)
sub vcl_backend_response {
	# Remove some headers we never want to see
	unset beresp.http.Server;
	unset beresp.http.X-Powered-By;

	# For static content strip all backend cookies
	if (bereq.url ~ "\.(css|js|png|gif|jp(e?)g)|swf|ico") {
		unset beresp.http.cookie;
	}

	# Only allow cookies to be set if we're in admin area
	if (beresp.http.Set-Cookie && bereq.url !~ "^/wp-(login|admin)") {
        	unset beresp.http.Set-Cookie;
    	}

	# don't cache response to posted requests or those with basic auth
	if ( bereq.method == "POST" || bereq.http.Authorization ) {
        	set beresp.uncacheable = true;
		set beresp.ttl = 120s;
		return (deliver);
    	}
 
    	# don't cache search results
	if ( bereq.url ~ "\?s=" ){
		set beresp.uncacheable = true;
                set beresp.ttl = 120s;
                return (deliver);
	}
    
	# only cache status ok
	if ( beresp.status != 200 ) {
		set beresp.uncacheable = true;
                set beresp.ttl = 120s;
                return (deliver);
	}

	# A TTL of 24h
	set beresp.ttl = 1m;
	# Define the default grace period to serve cached content
	set beresp.grace = 30s;
	
	return (deliver);
}
 
# The routine when we deliver the HTTP request to the user
# Last chance to modify headers that are sent to the client
sub vcl_deliver {
	if (obj.hits > 0) { 
		set resp.http.X-Cache = "cached";
	} else {
		set resp.http.x-Cache = "uncached";
	}

	# Remove some headers: PHP version
	unset resp.http.X-Powered-By;

	# Remove some headers: Apache version & OS
	unset resp.http.Server;

	# Remove some heanders: Varnish
	#unset resp.http.Via;
	#unset resp.http.X-Varnish;

	return (deliver);
}
 
sub vcl_init {
 	return (ok);
}
 
sub vcl_fini {
 	return (ok);
}


From matthewm at boedicker.org  Sat Sep 12 06:17:12 2015
From: matthewm at boedicker.org (Matthew Boedicker)
Date: Fri, 11 Sep 2015 21:17:12 -0700
Subject: varnish removing Transfer-Encoding: chunked header
In-Reply-To: <72925.1441957047@critter.freebsd.dk>
References: <CANfC_4JDBMOpjm=mwOe4yWL--cDS6NWO863bj=kAe9BSGXFvAQ@mail.gmail.com>
 <72925.1441957047@critter.freebsd.dk>
Message-ID: <CANfC_4+eBp+2e6OomF6EA7u93EY3ZZ49kv65MWknnS537xN=dg@mail.gmail.com>

I retested with 4.0.3 installed from the Ubuntu debs and the problem went
away.

The new behavior is:

Backend response:

  Transfer-Encoding: chunked
  No content length header

Varnish response:

  No transfer encoding header
  Content-Length: 0


On Fri, Sep 11, 2015 at 12:37 AM, Poul-Henning Kamp <phk at phk.freebsd.dk>
wrote:

> --------
> In message <CANfC_4JDBMOpjm=mwOe4yWL--cDS6NWO863bj=
> kAe9BSGXFvAQ at mail.gmail.com>
> , Matthew Boedicker writes:
>
> >Is this expected behavior or should I open a bug?
>
> Open a bug please.
>
> --
> Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
> phk at FreeBSD.ORG         | TCP/IP since RFC 956
> FreeBSD committer       | BSD since 4.3-tahoe
> Never attribute to malice what can adequately be explained by incompetence.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20150911/feedec48/attachment.html>

From dridi at varni.sh  Mon Sep 14 11:48:09 2015
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Mon, 14 Sep 2015 11:48:09 +0200
Subject: cache-control header in request dropped by varnish
In-Reply-To: <CACOJb1FtFex8gVvsJO5KeJv1h4WMhT_zbTUBMw8tELSimLt9_A@mail.gmail.com>
References: <CACOJb1FtFex8gVvsJO5KeJv1h4WMhT_zbTUBMw8tELSimLt9_A@mail.gmail.com>
Message-ID: <CABoVN9DRS7miVWOydVY+2nqobffE0fR1H9+WaytsPbb0-aC4nw@mail.gmail.com>

Hi,

Letting a client enforce your cache policy is a bad idea IMHO, unless
you safeguard it to ensure only trusted clients can do that with an
ACL for instance.

You should probably upgrade to Varnish 4 first, Varnish 3 is no longer
supported.

Best Regards,
Dridi

On Fri, Sep 4, 2015 at 5:02 PM, Shiva <shiva2110 at gmail.com> wrote:
> Hi,
> We use varnish v3.0.4 , we are seeing this issue where clients are sending
> cache-control request header to varnish, but it is getting dropped before
> sending to backend app.
>
> I could see varnish receiving the header, also could see RxHeader having
> cache-control header from varnishlog. Our backend server is on tomcat, in
> tomcat access log the cache-control header is not seen.
>
> Is this an expected behavior ? RFC spec on cache-control header seem to
> mandate all caches to pass the header to the backend.
>
> Thanks!
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc


From t.honacker at googlemail.com  Mon Sep 14 20:19:25 2015
From: t.honacker at googlemail.com (Tobias Honacker)
Date: Mon, 14 Sep 2015 20:19:25 +0200
Subject: Varnish and AWS ELB
Message-ID: <CADepKeB9Mr8wrkkn-EQrg0m+0GnQA7JqFwfzr6KWbCLCyG=GvA@mail.gmail.com>

Hi guys,

i currently stuck on the following setup:

Client / Request -> 1x ELB (LBCookieStickinessPolicy) -> 2x Varnish -> 3x
Backend

I need to setup session stickyness otherwise backend sends CSRF-Token is
invalid.
AWS ELB sends cookies like: AWSELB=
E5D3B9C50EAF269F2C074A8A0DA1A422FAB5B73E769AA411DD448..

If i send the first request to my website, ELB created the cookie and sent
the request to varnish1. After reloading the website (second request) ELB
created a new cookie and is balancing to the second varnish.

Does anyone know how to fix this issue?

Thanks and regards
Tobias
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20150914/fe329b86/attachment.html>

From marco at nucleus.it  Tue Sep 15 11:53:05 2015
From: marco at nucleus.it (Marco Felettigh)
Date: Tue, 15 Sep 2015 11:53:05 +0200
Subject: varnishlog and gzip tag
Message-ID: <20150915115305.27ef48ff@lobo.lobo.dom>

hi,
anyone knows what this flags mean ?

...
3 Gzip         c U D - 297 779 80 80 2311
...

i cannot find documentation about it.

Thanks
Marco


From kokoniimasu at gmail.com  Tue Sep 15 11:59:27 2015
From: kokoniimasu at gmail.com (kokoniimasu)
Date: Tue, 15 Sep 2015 18:59:27 +0900
Subject: varnishlog and gzip tag
In-Reply-To: <20150915115305.27ef48ff@lobo.lobo.dom>
References: <20150915115305.27ef48ff@lobo.lobo.dom>
Message-ID: <CAJihfGAwvp48evzC17dps8ksdz0y+US1wa7sDyniY0TNmouXmQ@mail.gmail.com>

Hi, Marco.

> Gzip - G(un)zip performed on object
ref: https://www.varnish-cache.org/docs/trunk/reference/vsl.html#vsl-tags

Regards,
--
Shohei Tanaka(@xcir)
http://blog.xcir.net/

2015-09-15 18:53 GMT+09:00 Marco Felettigh <marco at nucleus.it>:
> hi,
> anyone knows what this flags mean ?
>
> ...
> 3 Gzip         c U D - 297 779 80 80 2311
> ...
>
> i cannot find documentation about it.
>
> Thanks
> Marco
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc


From marco at nucleus.it  Tue Sep 15 12:10:32 2015
From: marco at nucleus.it (Marco Felettigh)
Date: Tue, 15 Sep 2015 12:10:32 +0200
Subject: varnishlog and gzip tag
In-Reply-To: <CAJihfGAwvp48evzC17dps8ksdz0y+US1wa7sDyniY0TNmouXmQ@mail.gmail.com>
References: <20150915115305.27ef48ff@lobo.lobo.dom>
 <CAJihfGAwvp48evzC17dps8ksdz0y+US1wa7sDyniY0TNmouXmQ@mail.gmail.com>
Message-ID: <20150915121032.12f12c89@lobo.lobo.dom>

Super :)


thanks
Marco


On Tue, 15 Sep 2015 18:59:27 +0900
kokoniimasu <kokoniimasu at gmail.com> wrote:

> Hi, Marco.
> 
> > Gzip - G(un)zip performed on object
> ref:
> https://www.varnish-cache.org/docs/trunk/reference/vsl.html#vsl-tags
> 
> Regards,
> --
> Shohei Tanaka(@xcir)
> http://blog.xcir.net/
> 
> 2015-09-15 18:53 GMT+09:00 Marco Felettigh <marco at nucleus.it>:
> > hi,
> > anyone knows what this flags mean ?
> >
> > ...
> > 3 Gzip         c U D - 297 779 80 80 2311
> > ...
> >
> > i cannot find documentation about it.
> >
> > Thanks
> > Marco
> >
> > _______________________________________________
> > varnish-misc mailing list
> > varnish-misc at varnish-cache.org
> > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc



From lists at redlab.be  Tue Sep 15 16:11:15 2015
From: lists at redlab.be (Balder VC)
Date: Tue, 15 Sep 2015 16:11:15 +0200
Subject: Varnish and AWS ELB
In-Reply-To: <CADepKeB9Mr8wrkkn-EQrg0m+0GnQA7JqFwfzr6KWbCLCyG=GvA@mail.gmail.com>
References: <CADepKeB9Mr8wrkkn-EQrg0m+0GnQA7JqFwfzr6KWbCLCyG=GvA@mail.gmail.com>
Message-ID: <55F82703.7040301@redlab.be>




On 09/14/2015 08:19 PM, Tobias Honacker wrote:
>
>
> Does anyone know how to fix this issue?

You can use the hashing backend and select a backend in varnish based on
the aws elb cookie value.

more info:
https://www.varnish-cache.org/docs/4.0/reference/vmod_directors.generated.html#func-hash-backend




>
> Thanks and regards
> Tobias
>
>


Regards
Balder
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20150915/e13b3912/attachment.html>

From lists at redlab.be  Tue Sep 15 16:12:23 2015
From: lists at redlab.be (Balder VC)
Date: Tue, 15 Sep 2015 16:12:23 +0200
Subject: Varnish and googlebot 403
In-Reply-To: <55F2BCC5.7030702@morphey.org>
References: <55EE8B21.9020506@morphey.org> <55F28564.1060504@morphey.org>
 <55F29F0B.7070703@redlab.be> <55F2BCC5.7030702@morphey.org>
Message-ID: <55F82747.2040908@redlab.be>

On 09/11/2015 01:36 PM, Christian Cantinelli wrote:
> Yes, sure
>
> In attach..



At first sight there is nothing wrong with the vcl. Are you sure there
are no other systems in between ?


>
>
> Christian
>
>
> Il 11/09/2015 11:29, Balder VC ha scritto:
>> On 09/11/2015 09:40 AM, Christian Cantinelli wrote:
>>> Hello,
>>>
>>> has anyone had such this problem?
>>
>>
>> I would seem as if no one had this problem.
>>
>> But prehaps there something in your vcl what causes it? Can you share it
>>
>>>



From morphey at morphey.org  Tue Sep 15 18:07:36 2015
From: morphey at morphey.org (Christian Cantinelli)
Date: Tue, 15 Sep 2015 18:07:36 +0200
Subject: Varnish and googlebot 403
In-Reply-To: <55F82747.2040908@redlab.be>
References: <55EE8B21.9020506@morphey.org> <55F28564.1060504@morphey.org>
 <55F29F0B.7070703@redlab.be> <55F2BCC5.7030702@morphey.org>
 <55F82747.2040908@redlab.be>
Message-ID: <14fd1c2e940.2792.2b94d75291887b16352973a9d80dfdbf@morphey.org>

Hello,

I Will see on the apache configuration.
Thanks you


Il 15 settembre 2015 16:12:28 Balder VC <lists at redlab.be> ha scritto:

> On 09/11/2015 01:36 PM, Christian Cantinelli wrote:
>> Yes, sure
>>
>> In attach..
>
>
>
> At first sight there is nothing wrong with the vcl. Are you sure there
> are no other systems in between ?
>
>
>>
>>
>> Christian
>>
>>
>> Il 11/09/2015 11:29, Balder VC ha scritto:
>>> On 09/11/2015 09:40 AM, Christian Cantinelli wrote:
>>>> Hello,
>>>>
>>>> has anyone had such this problem?
>>>
>>>
>>> I would seem as if no one had this problem.
>>>
>>> But prehaps there something in your vcl what causes it? Can you share it
>>>
>>>>
>




From phk at phk.freebsd.dk  Wed Sep 16 11:00:17 2015
From: phk at phk.freebsd.dk (Poul-Henning Kamp)
Date: Wed, 16 Sep 2015 09:00:17 +0000
Subject: varnish removing Transfer-Encoding: chunked header
In-Reply-To: <CANfC_4JDBMOpjm=mwOe4yWL--cDS6NWO863bj=kAe9BSGXFvAQ@mail.gmail.com>
References: <CANfC_4JDBMOpjm=mwOe4yWL--cDS6NWO863bj=kAe9BSGXFvAQ@mail.gmail.com>
Message-ID: <1463.1442394017@critter.freebsd.dk>

--------
In message <CANfC_4JDBMOpjm=mwOe4yWL--cDS6NWO863bj=kAe9BSGXFvAQ at mail.gmail.com>
, Matthew Boedicker writes:

>I have a rails app which returns 200 with an empty body as a response to a
>POST. It does not set a Content-Length header, but does set a
>Transfer-Encoding chunked header.

I belive this is fixed in -trunk now.

Please test it if you can, and thanks for the report!

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.


From andrew.langhorn at digital.cabinet-office.gov.uk  Wed Sep 16 13:36:36 2015
From: andrew.langhorn at digital.cabinet-office.gov.uk (Andrew Langhorn)
Date: Wed, 16 Sep 2015 12:36:36 +0100
Subject: Returning HTTP 204 for OPTIONS requests
Message-ID: <CANCa_WY-KGgG_Cno1OYRMzg5vB2PbhaTc8j=kKjmW6KP6PowxQ@mail.gmail.com>

Hello,

I run Varnish in front of a stack that doesn't respond to OPTIONS requests.
I'd like to filter out these requests from our error logs, because
currently they respond with a 404, which means there's quite a lot of noise.

I've been using this snippet at the top of vcl_recv:

if (req.request == "OPTIONS") {

error 204 "No Content";

}


Is that the best place for it, or are there better places instead?
Similarly, is that the best way of doing this?

Thanks,

-- 
Andrew Langhorn
Web Operations
Government Digital Service

e: andrew.langhorn at digital.cabinet-office.gov.uk
a: 6th Floor, Aviation House, 125 Kingsway, London, WC2B 6NH
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20150916/8e7d3dea/attachment.html>

From bluethundr at gmail.com  Fri Sep 18 06:36:13 2015
From: bluethundr at gmail.com (Tim Dunphy)
Date: Fri, 18 Sep 2015 00:36:13 -0400
Subject: best timeout defaults
Message-ID: <CAOZy0enu-Dck4HMQ=6Vui0EegjuKGnDBRArs2tYftuBVH7ckCg@mail.gmail.com>

Hey guys,

 Just wondering what y'all would consider the best / optimal timeout
settings for varnish servers to contact they're backends. Considering that
both varnish servers and web servers are in the same datacenter.

So, what values do you like for these settings:

For the health probe:

 .timeout = 1s;
  .interval = 2s;
  .window = 10;
  .threshold = 8;

And for the backend hosts themselves:

    .connect_timeout = 5s;
    .first_byte_timeout = 10s;
    .between_bytes_timeout = 2s;

Oh and heck. While we're at it, is there a general rule of thumb/best
practice for this one?

   .max_connections = 800;

Enquiring minds want to know!

Thanks!

Tim

-- 
GPG me!!

gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20150918/30b2183c/attachment.html>

From dridi at varni.sh  Fri Sep 18 08:39:20 2015
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Fri, 18 Sep 2015 08:39:20 +0200
Subject: best timeout defaults
In-Reply-To: <CAOZy0enu-Dck4HMQ=6Vui0EegjuKGnDBRArs2tYftuBVH7ckCg@mail.gmail.com>
References: <CAOZy0enu-Dck4HMQ=6Vui0EegjuKGnDBRArs2tYftuBVH7ckCg@mail.gmail.com>
Message-ID: <CABoVN9AC0av2NU309oxnU9zTJQY3Wa7_M5OdsQnWm6snHrDp9g@mail.gmail.com>

Hi Tim,

On Fri, Sep 18, 2015 at 6:36 AM, Tim Dunphy <bluethundr at gmail.com> wrote:
> Hey guys,
>
>  Just wondering what y'all would consider the best / optimal timeout
> settings for varnish servers to contact they're backends. Considering that
> both varnish servers and web servers are in the same datacenter.

There's no such thing as optimal timeout settings IMHO. The fact that
your servers are sitting in the same datacenter is sonething you need
to take into account, but also the behavior of your backends. For
example, if you know that your backend will stream results immediately
you might want to lower the fbto, and if your backends tend to
precompute the response before sending it you might want to lower the
bbto instead.

If you are not familiar enough with the backends implementation
details, the best course of action is probably to be conservative and
measure the response times to find out the expected responsiveness
from the backends.

> Oh and heck. While we're at it, is there a general rule of thumb/best
> practice for this one?
>
>    .max_connections = 800;

Again, it all depends on your backends capabilities.

Cheers,
Dridi


From maghoff at gmail.com  Fri Sep 18 12:35:06 2015
From: maghoff at gmail.com (Magnus Hoff)
Date: Fri, 18 Sep 2015 12:35:06 +0200
Subject: Conditional backend requests (ETag)
Message-ID: <CAJb5=h-pi6rxQOfBzvN6kzGDGerur2ebJNu6ZcALvgxyTKnXFA@mail.gmail.com>

Hi :)

What is the status of support for conditional backend requests?

My use case is accelerating CouchDB, which has good support for ETags,
but offers no other caching opportunities.

I have found [1] which leads me to think that the branch implementing
this feature was merged to trunk in 2011. According to [2] (which also
says, in big bold letters, "This documentation is out of date"), I
should be able to see this working with the default VCL: "In the
default case (i.e. if not superseded by something in VCL), a
conditional request is generated during fetch with If-Modified-Since
and/or If-None-Match headers whose values are taken from stale_obj's
Last-Modified and/or ETag headers."

[1]: https://www.varnish-cache.org/lists/pipermail/varnish-commit/2011-July/006720.html
[2]: https://www.varnish-cache.org/trac/wiki/BackendConditionalRequests

However, when testing with the latest Varnish from the Varnish Ubuntu
repository (https://www.varnish-cache.org/installation/ubuntu),
"varnish-4.0.3 revision b8c4a34", I am unable to trigger conditional
backend requests in the default configuration. I have feebly tried
making the VCL more explicit in that it should try to keep the ETagged
versions around by setting beresp.keep explicitly, but I can't seem to
effect a change in behavior.

So, what is happening? Does Varnish currently support what I am
looking for? What is the up to date documentation on this? Is there
support for this on a branch still? How is that branch coming along?


Thank you :)
- Magnus Hoff


From guillaume at varnish-software.com  Fri Sep 18 12:57:03 2015
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Fri, 18 Sep 2015 12:57:03 +0200
Subject: Conditional backend requests (ETag)
In-Reply-To: <CAJb5=h-pi6rxQOfBzvN6kzGDGerur2ebJNu6ZcALvgxyTKnXFA@mail.gmail.com>
References: <CAJb5=h-pi6rxQOfBzvN6kzGDGerur2ebJNu6ZcALvgxyTKnXFA@mail.gmail.com>
Message-ID: <CAJ6ZYQwfLGnJpZx-n1zSaeMDP1-tXmzxdkgo5y+szE=9fmc=kQ@mail.gmail.com>

Hi Magnus,

The documention you are looking for is here :
https://www.varnish-cache.org/docs/4.0/reference/vcl.html

And beresp.keep is definitely what you are looking for. If ttl is expired
but not ttl + keep, varnish will issue a conditionnal request.

-- 
Guillaume Quintard

On Fri, Sep 18, 2015 at 12:35 PM, Magnus Hoff <maghoff at gmail.com> wrote:

> Hi :)
>
> What is the status of support for conditional backend requests?
>
> My use case is accelerating CouchDB, which has good support for ETags,
> but offers no other caching opportunities.
>
> I have found [1] which leads me to think that the branch implementing
> this feature was merged to trunk in 2011. According to [2] (which also
> says, in big bold letters, "This documentation is out of date"), I
> should be able to see this working with the default VCL: "In the
> default case (i.e. if not superseded by something in VCL), a
> conditional request is generated during fetch with If-Modified-Since
> and/or If-None-Match headers whose values are taken from stale_obj's
> Last-Modified and/or ETag headers."
>
> [1]:
> https://www.varnish-cache.org/lists/pipermail/varnish-commit/2011-July/006720.html
> [2]: https://www.varnish-cache.org/trac/wiki/BackendConditionalRequests
>
> However, when testing with the latest Varnish from the Varnish Ubuntu
> repository (https://www.varnish-cache.org/installation/ubuntu),
> "varnish-4.0.3 revision b8c4a34", I am unable to trigger conditional
> backend requests in the default configuration. I have feebly tried
> making the VCL more explicit in that it should try to keep the ETagged
> versions around by setting beresp.keep explicitly, but I can't seem to
> effect a change in behavior.
>
> So, what is happening? Does Varnish currently support what I am
> looking for? What is the up to date documentation on this? Is there
> support for this on a branch still? How is that branch coming along?
>
>
> Thank you :)
> - Magnus Hoff
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20150918/177d4994/attachment.html>

From maghoff at gmail.com  Fri Sep 18 15:13:55 2015
From: maghoff at gmail.com (Magnus Hoff)
Date: Fri, 18 Sep 2015 15:13:55 +0200
Subject: Conditional backend requests (ETag)
In-Reply-To: <CAJ6ZYQwfLGnJpZx-n1zSaeMDP1-tXmzxdkgo5y+szE=9fmc=kQ@mail.gmail.com>
References: <CAJb5=h-pi6rxQOfBzvN6kzGDGerur2ebJNu6ZcALvgxyTKnXFA@mail.gmail.com>
 <CAJ6ZYQwfLGnJpZx-n1zSaeMDP1-tXmzxdkgo5y+szE=9fmc=kQ@mail.gmail.com>
Message-ID: <CAJb5=h_Q52NkBFqBuNUG0knnV4T6vPVBu4TgwURJ0urq_ieffw@mail.gmail.com>

Thanks for getting back to me.

I went back and tried poking around some more. It seems like when I
set the TTL to zero it seems to disable the conditional request
functionality as well. Anyway, when I set the TTL to 1s instead, it
seems to be working well, with however long keep-times I give it.

Thanks :)


-m

On Fri, Sep 18, 2015 at 12:57 PM, Guillaume Quintard
<guillaume at varnish-software.com> wrote:
> Hi Magnus,
>
> The documention you are looking for is here :
> https://www.varnish-cache.org/docs/4.0/reference/vcl.html
>
> And beresp.keep is definitely what you are looking for. If ttl is expired
> but not ttl + keep, varnish will issue a conditionnal request.
>
> --
> Guillaume Quintard
>
> On Fri, Sep 18, 2015 at 12:35 PM, Magnus Hoff <maghoff at gmail.com> wrote:
>>
>> Hi :)
>>
>> What is the status of support for conditional backend requests?
>>
>> My use case is accelerating CouchDB, which has good support for ETags,
>> but offers no other caching opportunities.
>>
>> I have found [1] which leads me to think that the branch implementing
>> this feature was merged to trunk in 2011. According to [2] (which also
>> says, in big bold letters, "This documentation is out of date"), I
>> should be able to see this working with the default VCL: "In the
>> default case (i.e. if not superseded by something in VCL), a
>> conditional request is generated during fetch with If-Modified-Since
>> and/or If-None-Match headers whose values are taken from stale_obj's
>> Last-Modified and/or ETag headers."
>>
>> [1]:
>> https://www.varnish-cache.org/lists/pipermail/varnish-commit/2011-July/006720.html
>> [2]: https://www.varnish-cache.org/trac/wiki/BackendConditionalRequests
>>
>> However, when testing with the latest Varnish from the Varnish Ubuntu
>> repository (https://www.varnish-cache.org/installation/ubuntu),
>> "varnish-4.0.3 revision b8c4a34", I am unable to trigger conditional
>> backend requests in the default configuration. I have feebly tried
>> making the VCL more explicit in that it should try to keep the ETagged
>> versions around by setting beresp.keep explicitly, but I can't seem to
>> effect a change in behavior.
>>
>> So, what is happening? Does Varnish currently support what I am
>> looking for? What is the up to date documentation on this? Is there
>> support for this on a branch still? How is that branch coming along?
>>
>>
>> Thank you :)
>> - Magnus Hoff
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>


From viktor.villafuerte at optusnet.com.au  Mon Sep 21 02:44:16 2015
From: viktor.villafuerte at optusnet.com.au (Viktor Villafuerte)
Date: Mon, 21 Sep 2015 10:44:16 +1000
Subject: Varnish storage: file
Message-ID: <20150921004415.GB1354@optusnet.com.au>

Hi all,

I'm using Varnish 4.0.3 with VARNISH_STORGE=file in sysconfig. I've been
monitoring this and have found couple of things that puzzle me a bit.

In sysconfig the definition is like this

# # Cache file location
VARNISH_STORAGE_FILE=/srv/hls_cache/hls_cache.bin
#
# # Cache file size: in bytes, optionally using k / M / G / T suffix,
# # or in percentage of available disk space using the % suffix.
VARNISH_STORAGE_SIZE=1160G
#
# # Backend storage specification
VARNISH_STORAGE="file,${VARNISH_STORAGE_FILE},${VARNISH_STORAGE_SIZE}"
#
DAEMON_OPTS="... \
             -s ${VARNISH_STORAGE}"



This morning I had a look and there were 21G of data cached. I tried to
restart Varnish but it failed due to in sufficient disk space. If I
lower the VARNISH_STORAGE_SIZE enough (to cover those 21G) then Varnish
starts without problems. This brings up two qustions

1. How to define the size of the destination file to be able to restart
without problems?

2. Varnish docs claim that its cache is not persistent. But this seems
to be very persistent to me?


Could anybody shed some light on this?

thanks

v

-- 
Regards

Viktor Villafuerte
Optus Internet Engineering
t: +61 2 80825265


From matthewm at boedicker.org  Mon Sep 21 06:22:20 2015
From: matthewm at boedicker.org (Matthew Boedicker)
Date: Sun, 20 Sep 2015 21:22:20 -0700
Subject: varnish removing Transfer-Encoding: chunked header
In-Reply-To: <1463.1442394017@critter.freebsd.dk>
References: <CANfC_4JDBMOpjm=mwOe4yWL--cDS6NWO863bj=kAe9BSGXFvAQ@mail.gmail.com>
 <1463.1442394017@critter.freebsd.dk>
Message-ID: <CANfC_4K+oSzWnBoHXsJs_-BQJnrauu-pyEu0WUq=iH9f22qLYQ@mail.gmail.com>

I confirmed that this is fixed in -trunk. Thanks for the fix!

On Wed, Sep 16, 2015 at 2:00 AM, Poul-Henning Kamp <phk at phk.freebsd.dk>
wrote:

> --------
> In message <CANfC_4JDBMOpjm=mwOe4yWL--cDS6NWO863bj=
> kAe9BSGXFvAQ at mail.gmail.com>
> , Matthew Boedicker writes:
>
> >I have a rails app which returns 200 with an empty body as a response to a
> >POST. It does not set a Content-Length header, but does set a
> >Transfer-Encoding chunked header.
>
> I belive this is fixed in -trunk now.
>
> Please test it if you can, and thanks for the report!
>
> --
> Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
> phk at FreeBSD.ORG         | TCP/IP since RFC 956
> FreeBSD committer       | BSD since 4.3-tahoe
> Never attribute to malice what can adequately be explained by incompetence.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20150920/a2e854b9/attachment.html>

From viktor.villafuerte at optusnet.com.au  Tue Sep 22 00:48:26 2015
From: viktor.villafuerte at optusnet.com.au (Viktor Villafuerte)
Date: Tue, 22 Sep 2015 08:48:26 +1000
Subject: Varnish storage: file
In-Reply-To: <20150921004415.GB1354@optusnet.com.au>
References: <20150921004415.GB1354@optusnet.com.au>
Message-ID: <20150921224826.GD1354@optusnet.com.au>

Just in case somebody is interested.. :)

Hello Viktor,

Your problem should be fixed in 4.1 (beta 1 is out, if you could test it
it'd be great) but in the meantime, you can work around the issue by
deleting the file before starting varnish.

Also, avoid filling the whole partition with the storage file and try to
leave something like 10% of free space to allow some breathing room for
the
file system.

Regarding persistence, the file persists, but the data it contains is
not used (because it cannot be relied on).

Does that clarify things a bit?


thanks

v


On Mon 21 Sep 2015 10:44:16, Viktor Villafuerte wrote:
> Hi all,
> 
> I'm using Varnish 4.0.3 with VARNISH_STORGE=file in sysconfig. I've been
> monitoring this and have found couple of things that puzzle me a bit.
> 
> In sysconfig the definition is like this
> 
> # # Cache file location
> VARNISH_STORAGE_FILE=/srv/hls_cache/hls_cache.bin
> #
> # # Cache file size: in bytes, optionally using k / M / G / T suffix,
> # # or in percentage of available disk space using the % suffix.
> VARNISH_STORAGE_SIZE=1160G
> #
> # # Backend storage specification
> VARNISH_STORAGE="file,${VARNISH_STORAGE_FILE},${VARNISH_STORAGE_SIZE}"
> #
> DAEMON_OPTS="... \
>              -s ${VARNISH_STORAGE}"
> 
> 
> 
> This morning I had a look and there were 21G of data cached. I tried to
> restart Varnish but it failed due to in sufficient disk space. If I
> lower the VARNISH_STORAGE_SIZE enough (to cover those 21G) then Varnish
> starts without problems. This brings up two qustions
> 
> 1. How to define the size of the destination file to be able to restart
> without problems?
> 
> 2. Varnish docs claim that its cache is not persistent. But this seems
> to be very persistent to me?
> 
> 
> Could anybody shed some light on this?
> 
> thanks
> 
> v
> 
> -- 
> Regards
> 
> Viktor Villafuerte
> Optus Internet Engineering
> t: +61 2 80825265
> 
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

-- 
Regards

Viktor Villafuerte
Optus Internet Engineering
t: +61 2 80825265


From lkarsten at varnish-software.com  Tue Sep 22 09:29:33 2015
From: lkarsten at varnish-software.com (Lasse Karstensen)
Date: Tue, 22 Sep 2015 09:29:33 +0200
Subject: Returning HTTP 204 for OPTIONS requests
In-Reply-To: <CANCa_WY-KGgG_Cno1OYRMzg5vB2PbhaTc8j=kKjmW6KP6PowxQ@mail.gmail.com>
References: <CANCa_WY-KGgG_Cno1OYRMzg5vB2PbhaTc8j=kKjmW6KP6PowxQ@mail.gmail.com>
Message-ID: <20150922072932.GA3823@immer.varnish-software.com>

On Wed, Sep 16, 2015 at 12:36:36PM +0100, Andrew Langhorn wrote:
> I run Varnish in front of a stack that doesn't respond to OPTIONS requests.
> I'd like to filter out these requests from our error logs, because
> currently they respond with a 404, which means there's quite a lot of noise.
> I've been using this snippet at the top of vcl_recv:
> if (req.request == "OPTIONS") {
> error 204 "No Content";
> }
> Is that the best place for it, or are there better places instead?
> Similarly, is that the best way of doing this?

Yes, this is the best way of doing this.

You should consider upgrading to Varnish 4.0 (or 4.1!), since 3.0
was declared end of life in April 2015.

-- 
Lasse Karstensen
Varnish Software AS


From andrew.langhorn at digital.cabinet-office.gov.uk  Tue Sep 22 10:05:31 2015
From: andrew.langhorn at digital.cabinet-office.gov.uk (Andrew Langhorn)
Date: Tue, 22 Sep 2015 09:05:31 +0100
Subject: Returning HTTP 204 for OPTIONS requests
In-Reply-To: <20150922072932.GA3823@immer.varnish-software.com>
References: <CANCa_WY-KGgG_Cno1OYRMzg5vB2PbhaTc8j=kKjmW6KP6PowxQ@mail.gmail.com>
 <20150922072932.GA3823@immer.varnish-software.com>
Message-ID: <CANCa_WYS=wjVX6nvmDoXYxVEP44_7q4xcKW7Aedw+500-e=Cdg@mail.gmail.com>

Upgrading Varnish is on the to-do list. I had issues with using that
snippet in vcl_recv when introducing a beresp statement - am I right in
thinking that beresp is only available from vcl_fetch?

Thanks Lasse.

On Tuesday, 22 September 2015, Lasse Karstensen <
lkarsten at varnish-software.com> wrote:

> On Wed, Sep 16, 2015 at 12:36:36PM +0100, Andrew Langhorn wrote:
> > I run Varnish in front of a stack that doesn't respond to OPTIONS
> requests.
> > I'd like to filter out these requests from our error logs, because
> > currently they respond with a 404, which means there's quite a lot of
> noise.
> > I've been using this snippet at the top of vcl_recv:
> > if (req.request == "OPTIONS") {
> > error 204 "No Content";
> > }
> > Is that the best place for it, or are there better places instead?
> > Similarly, is that the best way of doing this?
>
> Yes, this is the best way of doing this.
>
> You should consider upgrading to Varnish 4.0 (or 4.1!), since 3.0
> was declared end of life in April 2015.
>
> --
> Lasse Karstensen
> Varnish Software AS
>


--
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20150922/cce70337/attachment.html>

From bluethundr at gmail.com  Mon Sep 28 05:27:33 2015
From: bluethundr at gmail.com (Tim Dunphy)
Date: Sun, 27 Sep 2015 23:27:33 -0400
Subject: see host header in varnishlog v4
Message-ID: <CAOZy0ekDCwhUf_PSV+5iauD71Ej-ACKeb9c+BxqJAxcCis0TUA@mail.gmail.com>

Hey all,

 I'm trying to figure out how to see the host header in varnishlog.

 I tried this combination but it didn't really work for me:

varnishlog -q 'RxHheader  ~ "Host: wiki.example.com"'
Query expression error:
Tag name matches zero tags (Pos 1)
RxHheader  ~ "Host: wiki.example.com"
#########-----------------------------

Can someone please fill me in on the correct way to do this?

Thanks,
Tim

-- 
GPG me!!

gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20150927/c8a907db/attachment.html>

From dridi at varni.sh  Mon Sep 28 08:22:14 2015
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Mon, 28 Sep 2015 08:22:14 +0200
Subject: see host header in varnishlog v4
In-Reply-To: <CAOZy0ekDCwhUf_PSV+5iauD71Ej-ACKeb9c+BxqJAxcCis0TUA@mail.gmail.com>
References: <CAOZy0ekDCwhUf_PSV+5iauD71Ej-ACKeb9c+BxqJAxcCis0TUA@mail.gmail.com>
Message-ID: <CABoVN9A=eT-FAeO8_FDaYbYghnqqSQKdZaLEhbus8=nZe8Bv-w@mail.gmail.com>

On Mon, Sep 28, 2015 at 5:27 AM, Tim Dunphy <bluethundr at gmail.com> wrote:
> Hey all,
>
>  I'm trying to figure out how to see the host header in varnishlog.
>
>  I tried this combination but it didn't really work for me:
>
> varnishlog -q 'RxHheader  ~ "Host: wiki.example.com"'
> Query expression error:
> Tag name matches zero tags (Pos 1)
> RxHheader  ~ "Host: wiki.example.com"
> #########-----------------------------
>
> Can someone please fill me in on the correct way to do this?

Hi Tim,

The tag you are looking for is ReqHeader, see `man vsl` for the
comprehensive list.

Cheers,
Dridi


From viktor.villafuerte at optusnet.com.au  Tue Sep 29 00:54:16 2015
From: viktor.villafuerte at optusnet.com.au (Viktor Villafuerte)
Date: Tue, 29 Sep 2015 08:54:16 +1000
Subject: see host header in varnishlog v4
In-Reply-To: <CABoVN9A=eT-FAeO8_FDaYbYghnqqSQKdZaLEhbus8=nZe8Bv-w@mail.gmail.com>
References: <CAOZy0ekDCwhUf_PSV+5iauD71Ej-ACKeb9c+BxqJAxcCis0TUA@mail.gmail.com>
 <CABoVN9A=eT-FAeO8_FDaYbYghnqqSQKdZaLEhbus8=nZe8Bv-w@mail.gmail.com>
Message-ID: <20150928225416.GA22608@optusnet.com.au>

Hi Tim,

On Mon 28 Sep 2015 08:22:14, Dridi Boukelmoune wrote:
> On Mon, Sep 28, 2015 at 5:27 AM, Tim Dunphy <bluethundr at gmail.com> wrote:
> > Hey all,
> >
> >  I'm trying to figure out how to see the host header in varnishlog.
> >
> >  I tried this combination but it didn't really work for me:
> >
> > varnishlog -q 'RxHheader  ~ "Host: wiki.example.com"'

# Varnish version
varnishd (varnish-4.0.3 revision b8c4a34)
Copyright (c) 2006 Verdens Gang AS
Copyright (c) 2006-2014 Varnish Software AS

you can do this

shell> varnishlog -I ReqHeader:[Hh]ost

which will show all occurences of ReqHeader matching [Hh]ost regex. See
man varnishlog for more info



> > Query expression error:
> > Tag name matches zero tags (Pos 1)
> > RxHheader  ~ "Host: wiki.example.com"
> > #########-----------------------------
> >
> > Can someone please fill me in on the correct way to do this?
> 
> Hi Tim,
> 
> The tag you are looking for is ReqHeader, see `man vsl` for the
> comprehensive list.
> 
> Cheers,
> Dridi
> 
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

-- 
Regards

Viktor Villafuerte
Optus Internet Engineering
t: +61 2 80825265


From eliezer at ngtech.co.il  Tue Sep 29 02:54:50 2015
From: eliezer at ngtech.co.il (Eliezer Croitoru)
Date: Tue, 29 Sep 2015 03:54:50 +0300
Subject: Multi varnish setup and purging
In-Reply-To: <1439443081493-a8f04205-b809b0d2-37139394@eventbase.com>
References: <1439443081493-a8f04205-b809b0d2-37139394@eventbase.com>
Message-ID: <5609E15A.7070702@ngtech.co.il>

This is a bit old but I still have interest in this topic.

I am running a HA varnish with 3 hosts and I am not using the ban method 
but a regular PURGE with one url per request.
The way I do that for multiple hosts is via a custom reverse proxy(I 
will publish it later)that I have written in GOlang which has a list of 
back-end servers and which "broadcast"(ie sending each host a single 
request) to all the back-ends.
It will do that in the background so the application will just receive a 
200 response fast and the logs will show any signs of errors.

Since I have never used the ban option I cannot tell you if it meets 
your goals.
If you have solved this issue somehow then I will be happy to hear about 
the solution you have implemented.

Thanks,
Eliezer

On 13/08/2015 08:18, Andy Fase wrote:
> Hi there
> We're in the process of taking our 3.0.X single varnish instance into a
> HA multi
> varnish setup.
> My plan for caching was to initially use a hash director to pass
> requests from
> whatever the incoming varnish instance is to a specific one (based on
> the URL)
> so that the cache for a given URL will only be kept in one place.
> Hence the flow would be
> Varnish -> varnish -> backend
> However our overall application makes heavy use of wildcard purging via
> the ban
> command. As each URL in a wildcard ban could be on different varnish
> instances I
> don't see how this could actually work?
> Does anyone have any experience in supporting purging multiple varnish
> servers
> while supporting wildcard purges without having to get the application
> to send
> separate purge requests to all varnish instances?
> I was thinking of maybe a round robin director with all varnish instances
> defined as backends and some vcl to actually do the ban but return a
> error (to
> the calling varnish instance) so that the ban gets sent to all varnish
> instances
> (because each of them return an error after the ban)
> Any thoughts appreciated
> Cheers Andy
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>



From eliezer at ngtech.co.il  Tue Sep 29 02:56:05 2015
From: eliezer at ngtech.co.il (Eliezer Croitoru)
Date: Tue, 29 Sep 2015 03:56:05 +0300
Subject: Partial object caching with varnish
In-Reply-To: <482801000965C7489DFC3C6412129B1F6B745850@xmb-aln-x10.cisco.com>
References: <482801000965C7489DFC3C6412129B1F6B745850@xmb-aln-x10.cisco.com>
Message-ID: <5609E1A5.7060306@ngtech.co.il>

+1 for the question.

Eliezer

On 23/07/2015 20:22, Madhava Gaikwad (madgaikw) wrote:
> Hello,
> I am newbie to varnish. I want to confirm following two things I am trying to accomplish.
> I understood varnish can do live streaming etc. But can it really just do partial object caching. Say if I send a range request for 10 bytes from a pdf, and setup varnish cache to cache everything, will it be able to do it? What is the VCL config I should look at this.
>
> I have a forward proxy which cannot do range request caching. Neither it can do streaming. So I am thinking to put varnish in between client and forward proxy. Forward proxy connects to internet. I know you may come up and laugh at me first on what I am trying to achieve, but is it achievable?
>
>
> Thank you.
> Madhava



From japrice at gmail.com  Tue Sep 29 03:41:50 2015
From: japrice at gmail.com (Jason Price)
Date: Mon, 28 Sep 2015 21:41:50 -0400
Subject: Partial object caching with varnish
In-Reply-To: <5609E1A5.7060306@ngtech.co.il>
References: <482801000965C7489DFC3C6412129B1F6B745850@xmb-aln-x10.cisco.com>
 <5609E1A5.7060306@ngtech.co.il>
Message-ID: <CAChvjRCHAsjcst8vVjy_e55TcN7gNfX=7PuZyAxTi=jgoRwhxQ@mail.gmail.com>

Varnish is not designed to work well as a forward proxy.  It's designed to
work as a reverse proxy.

Varnish4 allows for the client to start receiving data before varnish has
received the full response from the backend (which is an important
improvement from varnish3).  Also, multiple requests for the same large
data object can be coalesced into a single backend request, while all
others wait on the first to succeed.  However, this is designed to work
with a defined number of backends, rather than the whole internet (as a
forward proxy would).

-Jason

On Mon, Sep 28, 2015 at 8:56 PM, Eliezer Croitoru <eliezer at ngtech.co.il>
wrote:

> +1 for the question.
>
> Eliezer
>
>
> On 23/07/2015 20:22, Madhava Gaikwad (madgaikw) wrote:
>
>> Hello,
>> I am newbie to varnish. I want to confirm following two things I am
>> trying to accomplish.
>> I understood varnish can do live streaming etc. But can it really just do
>> partial object caching. Say if I send a range request for 10 bytes from a
>> pdf, and setup varnish cache to cache everything, will it be able to do it?
>> What is the VCL config I should look at this.
>>
>> I have a forward proxy which cannot do range request caching. Neither it
>> can do streaming. So I am thinking to put varnish in between client and
>> forward proxy. Forward proxy connects to internet. I know you may come up
>> and laugh at me first on what I am trying to achieve, but is it achievable?
>>
>>
>> Thank you.
>> Madhava
>>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20150928/c02ed08f/attachment.html>