<div dir="ltr">Hey guys,<div><br></div><div> I have a varnish config that's been working pretty nicely for the last year. But recently I had to build 3 Kibana (that's the logstash UI if you're not aware) nodes on aws. And I wanted to load balance them and perhaps do some caching on them. </div><div><br></div><div>However when I tried to add the new kibana hosts to the config, instead of showing the kibana interface, it shows the default web page on my first apache server from the previous load balancing pool. That belongs to a host called 'web1' in the config.</div><div><br></div><div>Loading the 3 kibana nodes individually shows that it's loading and running properly.</div><div><br></div><div>Here's what I've done to try to get this to work:</div><div><br></div><div>I added the 3 kibana nodes as logs1, 2 and 3 to the config:<br></div><div><br></div><div>backend logs1 {<br></div><div>
<p> .host = "52.xx.xxx.180";</p>
<p> .port = "80";</p>
<p> .connect_timeout = 1s;</p>
<p> .first_byte_timeout = 5s;</p>
<p> .between_bytes_timeout = 2s;</p>
<p> .max_connections = 800;</p>
<p> .probe = healthcheck;</p>
<p>}</p>
<p>backend logs2 {</p>
<p> .host = "52.xx.xx.78";</p>
<p> .port = "80";</p>
<p> .connect_timeout = 1s;</p>
<p> .first_byte_timeout = 5s;</p>
<p> .between_bytes_timeout = 2s;</p>
<p> .max_connections = 800;</p>
<p> .probe = healthcheck;</p>
<p>}</p>
<p>backend logs3 {</p>
<p> .host = "52.xx.xxx.209";</p>
<p> .port = "80";</p>
<p> .connect_timeout = 1s;</p>
<p> .first_byte_timeout = 5s;</p>
<p> .between_bytes_timeout = 2s;</p>
<p> .max_connections = 800;</p>
<p> .probe = healthcheck;</p>
<p>}</p></div><div><br></div><div>And setup a new load balancing section:</div><div><br></div><div>
<p>sub vcl_init {</p>
<p> new wiki = directors.round_robin();</p>
<p> wiki.add_backend(web1); ## <-- this is what shows up at <a href="http://logs.exampledomain.com">logs.exampledomain.com</a></p>
<p> wiki.add_backend(web2);</p>
<p> wiki.add_backend(web3);</p>
<p>}</p>
<p><b>sub vcl_init {</b></p>
<p><b> new logs = directors.round_robin();</b></p>
<p><b> logs.add_backend(logs1);</b></p>
<p><b> logs.add_backend(logs2);</b></p>
<p><b> logs.add_backend(logs3);</b></p>
<p><b>}</b></p></div><div><br></div><div>In my vcl_recv I have the following:<br><br>
<p>sub vcl_recv {</p><p> # Authenticate the user using Apache basic auth</p><p> if ( req.http.host ~ "^wiki\.exampledomain\.com$") {</p><p> set req.backend_hint = wiki.backend();</p><p> if (!basicauth.match("/etc/httpd/auth", req.http.Authorization)) {</p><p> return(synth(401, "Authentication required"));</p><p> }</p><p> } else if ( req.http.host ~ "^beta\.exampledomain\.com$" ) {</p><p> set req.http.backend_hint = wiki.backend();</p><p> }</p><p> if ( req.http.host ~ "^logs\.exampledomain\.com$") {<br></p><p> set req.backend_hint = logs.backend();</p><p> }</p>
<p> # Authenticate the user using Apache basic auth</p>
<p> if ( req.http.host ~ "^wiki\.exampledomain\.com$") {</p>
<p> set req.backend_hint = wiki.backend();</p>
<p> if (!basicauth.match("/etc/httpd/auth", req.http.Authorization)) {</p>
<p> return(synth(401, "Authentication required"));</p>
<p> }</p>
<p> } else if ( req.http.host ~ "^beta\.exampledomain\.com$" ) {</p>
<p> set req.http.backend_hint = wiki.backend();</p>
<p> }</p>
<p><br></p>
<p> if ( req.http.host ~ "^logs\.exampledomain\.com$") {</p>
<p> set req.backend_hint = logs.backend();</p>
<p> }</p><p><br></p><p>..... more stuff</p><p class=""> return(hash);</p><p>
</p><p class="">}</p><p class=""><br></p><p class="">And instead of seeing this at <a href="http://logs.exampledomain.com">logs.exampledomain.com</a>:</p><p class=""><br></p><p class=""><img src="cid:ii_ilg2gvuh0_1534a364b4d3c116" width="487" height="184"></p><p class="">I see a web page that looks like this:<br><br><img src="cid:ii_ilg2j4u01_1534a37e4a8c0839" width="308" height="196"></p><p class="">How can I get varnish to serve the 'logs' load balancing pool correctly?</p><p class=""><br></p><p class="">Here's my whole vcl_recv in case that helps :</p><p class="">vcl 4.0;</p><p class="">import std;</p><p class="">import directors;</p><p class="">import basicauth;</p><p class=""><br></p><p class=""><br></p><p class="">probe healthcheck {</p><p class=""> .url = "/healthcheck.php";</p><p class=""> .timeout =1m;</p><p class=""> .interval = 1s;</p><p class=""> .window = 10;</p><p class=""> .threshold = 8;</p><p class=""> }</p><p class=""><br></p><p class="">backend default {</p><p class=""> .host = "107.xxx.xx.174";</p><p class=""> .port = "80";</p><p class=""> .connect_timeout = 1m;</p><p class=""> .first_byte_timeout = 1m;</p><p class=""> .between_bytes_timeout = 2s;</p><p class=""> .max_connections = 800;</p><p class=""> .probe = healthcheck;</p><p class="">}</p><p class=""><br></p><p class="">backend web1 {</p><p class=""> .host = "107.xxx.xx.174";</p><p class=""> .port = "80";</p><p class=""> .connect_timeout = 1s;</p><p class=""> .first_byte_timeout = 5s;</p><p class=""> .between_bytes_timeout = 2s;</p><p class=""> .max_connections = 800;</p><p class=""> .probe = healthcheck;</p><p class="">}</p><p class=""><br></p><p class="">backend web2 {</p><p class=""> .host = "107.xxx.xxx.53";</p><p class=""> .port = "80";</p><p class=""> .connect_timeout = 1s;</p><p class=""> .first_byte_timeout = 5s;</p><p class=""> .between_bytes_timeout = 2s;</p><p class=""> .max_connections = 800;</p><p class=""> .probe = healthcheck;</p><p class="">}</p><p class=""><br></p><p class="">backend web3 {</p><p class=""> .host = "107.xxx.xx.38";</p><p class=""> .port = "80";</p><p class=""> .connect_timeout = 1s;</p><p class=""> .first_byte_timeout = 5s;</p><p class=""> .between_bytes_timeout = 2s;</p><p class=""> .max_connections = 800;</p><p class=""> .probe = healthcheck;</p><p class="">}</p><p class=""><br></p><p class="">backend logs1 {</p><p class=""> .host = "52.xx.xxx.180";</p><p class=""> .port = "80";</p><p class=""> .connect_timeout = 1s;</p><p class=""> .first_byte_timeout = 5s;</p><p class=""> .between_bytes_timeout = 2s;</p><p class=""> .max_connections = 800;</p><p class=""> .probe = healthcheck;</p><p class="">}</p><p class=""><br></p><p class="">backend logs2 {</p><p class=""> .host = "52.xx.xx.78";</p><p class=""> .port = "80";</p><p class=""> .connect_timeout = 1s;</p><p class=""> .first_byte_timeout = 5s;</p><p class=""> .between_bytes_timeout = 2s;</p><p class=""> .max_connections = 800;</p><p class=""> .probe = healthcheck;</p><p class="">}</p><p class=""><br></p><p class="">backend logs3 {</p><p class=""> .host = "52.xx.xxx.209";</p><p class=""> .port = "80";</p><p class=""> .connect_timeout = 1s;</p><p class=""> .first_byte_timeout = 5s;</p><p class=""> .between_bytes_timeout = 2s;</p><p class=""> .max_connections = 800;</p><p class=""> .probe = healthcheck;</p><p class="">}</p><p class=""><br></p><p class=""># access control list for "purge": open to only localhost and other local nodes</p><p class="">acl purge {</p><p class=""> "127.0.0.1";</p><p class="">}</p><p class=""><br></p><p class="">sub vcl_init {</p><p class=""> new wiki = directors.round_robin();</p><p class=""> wiki.add_backend(web1);</p><p class=""> wiki.add_backend(web2);</p><p class=""> wiki.add_backend(web3);</p><p class="">}</p><p class=""><br></p><p class="">sub vcl_init {</p><p class=""> new logs = directors.round_robin();</p><p class=""> logs.add_backend(logs1);</p><p class=""> logs.add_backend(logs2);</p><p class=""> logs.add_backend(logs3);</p><p class="">}</p><p class=""><br></p><p class=""># vcl_recv is called whenever a request is received</p><p class="">sub vcl_recv {</p><p class=""><br></p><p class=""><br></p><p class=""> # Authenticate the user using Apache basic auth</p><p class=""> if ( req.http.host ~ "^wiki\.exampledomain\.com$") {</p><p class=""> set req.backend_hint = wiki.backend();</p><p class=""> if (!basicauth.match("/etc/httpd/auth", req.http.Authorization)) {</p><p class=""> return(synth(401, "Authentication required"));</p><p class=""> }</p><p class=""> } else if ( req.http.host ~ "^beta\.exampledomain\.com$" ) {</p><p class=""> set req.http.backend_hint = wiki.backend();</p><p class=""> }</p><p class=""><br></p><p class=""> if ( req.http.host ~ "^logs\.exampledomain\.com$") {</p><p class=""> set req.backend_hint = logs.backend();</p><p class=""> }</p><p class=""><br></p><p class=""> # Remove has_js and CloudFlare/Google Analytics __* cookies.</p><p class=""> set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(_[_a-z]+|has_js)=[^;]*", "");</p><p class=""> # Remove a ";" prefix, if present.</p><p class=""> set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", "");</p><p class=""><br></p><p class=""> set req.http.X-Forwarded-For = client.ip;</p><p class=""> set req.backend_hint= default;</p><p class=""><br></p><p class=""> # This uses the ACL action called "purge". Basically if a request to</p><p class=""> # PURGE the cache comes from anywhere other than localhost, ignore it.</p><p class=""> if (req.method == "PURGE")</p><p class=""> {if (!client.ip ~ purge)</p><p class=""> {return(synth(405,"Not allowed."));}</p><p class=""> return(hash);}</p><p class=""><br></p><p class=""> # Allows editing the wiki</p><p class=""> if (req.url ~ "&action=submit($|/)" ) {</p><p class=""> return(pass);</p><p class=""> }</p><p class=""><br></p><p class=""> # Pass any requests that Varnish does not understand straight to the backend.</p><p class=""> if (req.method != "GET" && req.method != "HEAD" &&</p><p class=""> req.method != "PUT" && req.method != "POST" &&</p><p class=""> req.method != "TRACE" && req.method != "OPTIONS" &&</p><p class=""> req.method != "DELETE")</p><p class=""> {return(pipe);} /* Non-RFC2616 or CONNECT which is weird. */</p><p class=""><br></p><p class=""> # Pass anything other than GET and HEAD directly.</p><p class=""> if (req.method != "GET" && req.method != "HEAD")</p><p class=""> {return(pass);} /* We only deal with GET and HEAD by default */</p><p class=""><br></p><p class=""> # Make images, etc cachable by unsetting cookie in request</p><p class=""> if (req.url ~ "^/images" ) {</p><p class=""> unset req.http.cookie;</p><p class=""> }</p><p class=""><br></p><p class=""> # Pass requests from logged-in users directly.</p><p class=""> #if (req.http.Authorization || req.http.Cookie)</p><p class=""> if (req.http.Cookie)</p><p class=""> {return(pass);} /* Not cacheable by default */</p><p class=""><br></p><p class=""> # Pass any requests with the "If-None-Match" header directly.</p><p class=""> if (req.http.If-None-Match)</p><p class=""> {return(pass);}</p><p class=""><br></p><p class=""> # Force lookup if the request is a no-cache request from the client.</p><p class=""> if (req.http.Cache-Control ~ "no-cache")</p><p class=""> {ban(req.url);}</p><p class=""><br></p><p class=""> # normalize Accept-Encoding to reduce vary</p><p class=""> if (req.http.Accept-Encoding) {</p><p class=""> if (req.http.User-Agent ~ "MSIE 6") {</p><p class=""> unset req.http.Accept-Encoding;</p><p class=""> } elsif (req.http.Accept-Encoding ~ "gzip") {</p><p class=""> set req.http.Accept-Encoding = "gzip";</p><p class=""> } elsif (req.http.Accept-Encoding ~ "deflate") {</p><p class=""> set req.http.Accept-Encoding = "deflate";</p><p class=""> } else {</p><p class=""> unset req.http.Accept-Encoding;</p><p class=""> }</p><p class=""> }</p><p class=""><br></p><p class=""> # Set the X-Forwarded-For header so the backend can see the original</p><p class=""> # IP address. If one is already set by an upstream proxy, we'll just re-use that.</p><p class=""> if (client.ip ~ purge && req.http.X-Forwarded-For) {</p><p class=""> set req.http.X-Forwarded-For = req.http.X-Forwarded-For;</p><p class=""> } else {</p><p class=""> set req.http.X-Forwarded-For = regsub(client.ip, ":.*", "");</p><p class=""> }</p><p class=""><br></p><p class=""> return(hash);</p><p class="">
</p><p class="">}</p><p class="">Thanks,</p><p class="">Tim</p><p class=""><br></p><p class=""><br></p><div><br></div>-- <br><div>GPG me!!<br><br>gpg --keyserver <a href="http://pool.sks-keyservers.net" target="_blank">pool.sks-keyservers.net</a> --recv-keys F186197B<br><br></div>
</div></div>