<div dir="ltr"><div>Hi,</div><div><br></div><div>I'm pretty sure there's a confusion with the sequence of actions here. Normalization happen *before* you look into the cache, so that way before you fetch anything from the backend. By the time you cache the data (vcl_backend_response), the hash key has already been set (vcl_hash), it's way too late to normalize the request.</div><div><br></div><div>As to normalization, it's usually done in vcl_recv, and it can range from just setting the host header to a static string to using std.tolower() and removing the host port.</div><div><br></div><div>for the sake of the example:<br></div><div><br></div><div style="margin-left:40px"><span style="font-family:monospace">sub vcl_vcl {</span></div><div style="margin-left:40px"><span style="font-family:monospace"> set req.http.host = "<a href="http://myvideoservice.com">myvideoservice.com</a>";<br></span></div><div style="margin-left:40px"><span style="font-family:monospace">}</span></div><div><br></div><div>For shard example, look at the VCTs, for example: <a href="https://github.com/varnishcache/varnish-cache/blob/6.6/bin/varnishtest/tests/d00029.vtc#L66">https://github.com/varnishcache/varnish-cache/blob/6.6/bin/varnishtest/tests/d00029.vtc#L66</a></div><div><div><font size="2"><code class="gmail-language-vcl"><span class="gmail-token gmail-function"><span class="gmail-token gmail-keyword"></span></span></code></font><br><font size="2"><code class="gmail-language-vcl"><span class="gmail-token gmail-function"><span class="gmail-token gmail-keyword"></span></span></code></font></div><div><pre class="gmail-language-vcl"><font size="2"><code class="gmail-language-vcl"><span class="gmail-token gmail-function"><span class="gmail-token gmail-keyword">import directors;<br><br>sub</span> vcl_init</span> <span class="gmail-token gmail-punctuation">{</span>
<span class="gmail-token gmail-keyword">new</span> shard_dir <span class="gmail-token gmail-operator">=</span> <span class="gmail-token gmail-punctuation"></span><span class="gmail-token gmail-function">director</span><span class="gmail-token gmail-punctuation">s.shard(</span><span class="gmail-token gmail-punctuation">)</span><span class="gmail-token gmail-punctuation">;</span><span class="gmail-token gmail-punctuation"></span>
<font size="2"><code class="gmail-language-vcl">shard_dir</code></font><span class="gmail-token gmail-punctuation">.</span><span class="gmail-token gmail-function">add_backend</span><span class="gmail-token gmail-punctuation">(</span>be1<span class="gmail-token gmail-number"></span><span class="gmail-token gmail-punctuation">)</span><span class="gmail-token gmail-punctuation">;</span>
<font size="2"><code class="gmail-language-vcl">shard_dir</code></font><span class="gmail-token gmail-punctuation">.</span><span class="gmail-token gmail-function">add_backend</span><span class="gmail-token gmail-punctuation">(</span>be2<span class="gmail-token gmail-number"></span><span class="gmail-token gmail-punctuation">)</span><span class="gmail-token gmail-punctuation">;</span>
<font size="2"><code class="gmail-language-vcl">shard_dir</code></font><span class="gmail-token gmail-punctuation">.</span><span class="gmail-token gmail-function">add_backend</span><span class="gmail-token gmail-punctuation">(</span>be3<span class="gmail-token gmail-punctuation"></span><span class="gmail-token gmail-punctuation">;<br><br></span><font size="2"><code class="gmail-language-vcl"><span class="gmail-token gmail-punctuation"></span></code></font><font size="2"><code class="gmail-language-vcl"> <font size="2"><code class="gmail-language-vcl">n</code></font></code></font>ew p = directors.shard_param();<font size="2"><code class="gmail-language-vcl"><span class="gmail-token gmail-punctuation"><br></span><font size="2"><code class="gmail-language-vcl"><span class="gmail-token gmail-punctuation"></span></code></font><font size="2"><code class="gmail-language-vcl"> <font size="2"><code class="gmail-language-vcl"></code></font></code></font></code></font>vd.associate(p.use());<br><br><font size="2"><code class="gmail-language-vcl"><font size="2"><code class="gmail-language-vcl"><font size="2"><code class="gmail-language-vcl"> <font size="2"><code class="gmail-language-vcl"></code></font></code></font></code></font></code></font>vd.reconfigure(replicas=25);
<span class="gmail-token gmail-punctuation">}</span>
<span class="gmail-token gmail-function"><span class="gmail-token gmail-keyword">sub</span> vcl_backend_fetch</span> <span class="gmail-token gmail-punctuation">{<br></span><font size="2"><code class="gmail-language-vcl"> <span class="gmail-token gmail-keyword"></span></code></font>p.set(by=KEY, key=bereq.url);
<span class="gmail-token gmail-keyword">set</span> <span class="gmail-token gmail-variable">bereq</span><span class="gmail-token gmail-punctuation">.</span>backend_hint <span class="gmail-token gmail-operator">=</span> <font size="2"><code class="gmail-language-vcl"><font size="2"><code class="gmail-language-vcl">shard_dir</code></font></code></font><span class="gmail-token gmail-punctuation">.</span><span class="gmail-token gmail-keyword">backend</span><span class="gmail-token gmail-punctuation">(resolve=LAZY</span><span class="gmail-token gmail-punctuation">)</span><span class="gmail-token gmail-punctuation">;</span><font size="2"><code class="gmail-language-vcl"><font size="2"><code class="gmail-language-vcl"><span class="gmail-token gmail-punctuation"></span></code></font><span class="gmail-token gmail-keyword"></span></code></font>
<span class="gmail-token gmail-punctuation">}</span></code></font></pre></div></div><div>For udo:</div><div><br></div><div><pre class="gmail-language-vcl"><font size="2"><code class="gmail-language-vcl"><span class="gmail-token gmail-function"><span class="gmail-token gmail-keyword">import crypto;<br>import udo;<br><br>sub</span> vcl_init</span> <span class="gmail-token gmail-punctuation">{</span>
<span class="gmail-token gmail-keyword">new</span> <span class="gmail-token gmail-class-name">udo_dir</span> <span class="gmail-token gmail-operator">=</span> udo<span class="gmail-token gmail-punctuation">.</span><span class="gmail-token gmail-function">director</span><span class="gmail-token gmail-punctuation">(</span><span class="gmail-token gmail-punctuation">)</span><span class="gmail-token gmail-punctuation">;</span>
udo_dir<span class="gmail-token gmail-punctuation">.</span><span class="gmail-token gmail-function">set_type</span><span class="gmail-token gmail-punctuation">(</span>random<span class="gmail-token gmail-punctuation">)</span><span class="gmail-token gmail-punctuation">;</span>
udo_dir<span class="gmail-token gmail-punctuation">.</span><span class="gmail-token gmail-function">add_backend</span><span class="gmail-token gmail-punctuation">(</span>be1<span class="gmail-token gmail-punctuation">)</span><span class="gmail-token gmail-punctuation">;</span>
udo_dir<span class="gmail-token gmail-punctuation">.</span><span class="gmail-token gmail-function">add_backend</span><span class="gmail-token gmail-punctuation">(</span>be2<span class="gmail-token gmail-number"></span><span class="gmail-token gmail-punctuation">)</span><span class="gmail-token gmail-punctuation">;</span>
udo_dir<span class="gmail-token gmail-punctuation">.</span><span class="gmail-token gmail-function">add_backend</span><span class="gmail-token gmail-punctuation">(</span>be3<span class="gmail-token gmail-number"></span><span class="gmail-token gmail-punctuation">)</span><span class="gmail-token gmail-punctuation">;<br></span><span class="gmail-token gmail-punctuation"></span><font size="2"><code class="gmail-language-vcl"><font size="2"><code class="gmail-language-vcl"> </code></font>udo_dir<span class="gmail-token gmail-punctuation">.</span><span class="gmail-token gmail-function">set_type</span><span class="gmail-token gmail-punctuation">(</span>hash<span class="gmail-token gmail-punctuation">)</span><span class="gmail-token gmail-punctuation">;</span></code></font>
<span class="gmail-token gmail-punctuation">}</span>
<span class="gmail-token gmail-function"><span class="gmail-token gmail-keyword">sub</span> vcl_backend_fetch</span> <span class="gmail-token gmail-punctuation">{</span>
<span class="gmail-token gmail-keyword">set</span> <span class="gmail-token gmail-variable">bereq</span><span class="gmail-token gmail-punctuation">.</span>backend_hint <span class="gmail-token gmail-operator">=</span> udo_dir<span class="gmail-token gmail-punctuation">.</span><span class="gmail-token gmail-keyword">backend</span><span class="gmail-token gmail-punctuation">(</span><span class="gmail-token gmail-punctuation">)</span><span class="gmail-token gmail-punctuation">;<br></span><span class="gmail-token gmail-punctuation"></span><font size="2"><code class="gmail-language-vcl"> <font size="2"><code class="gmail-language-vcl">udo_dir<span class="gmail-token gmail-punctuation">.set_hash(crypto.hash(<code>sha256, bereq.url</code>));</span></code></font><span class="gmail-token gmail-keyword"></span></code></font>
<span class="gmail-token gmail-punctuation">}</span></code></font></pre></div><div><br></div><div>These have been written without testing, so don't put them straight into production.<br></div><div><br></div><div><div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>-- <br></div><div>Guillaume Quintard<br></div></div></div></div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Aug 5, 2021 at 3:33 AM Hamidreza Hosseini <<a href="mailto:hrhosseini@hotmail.com">hrhosseini@hotmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hi,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
1.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Is there any way to normalize host headers and other things to say to varnish not to cache the same content for different backend?<br>
I want to use round robin director but after fetching the content I want to normalize the header and cache the content,<br>
I would appreciate if you give me an example about this and how I can do it.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
2.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I couldn't find any good example for directors-shard and xshard-key-string, I would appreciate if you could give example about this too.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Many Thanks<br>
</div>
<div id="gmail-m_-5915608502010945737appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_-5915608502010945737divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> varnish-misc <varnish-misc-bounces+hrhosseini=<a href="mailto:hotmail.com@varnish-cache.org" target="_blank">hotmail.com@varnish-cache.org</a>> on behalf of Hamidreza Hosseini <<a href="mailto:hrhosseini@hotmail.com" target="_blank">hrhosseini@hotmail.com</a>><br>
<b>Sent:</b> Sunday, August 1, 2021 4:17 AM<br>
<b>To:</b> <a href="mailto:varnish-misc@varnish-cache.org" target="_blank">varnish-misc@varnish-cache.org</a> <<a href="mailto:varnish-misc@varnish-cache.org" target="_blank">varnish-misc@varnish-cache.org</a>><br>
<b>Subject:</b> Best practice for caching scenario with different backend servers but same content</font>
<div> </div>
</div>
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hi,<br>
I want to use varnish in my scenario as cache service, I have about 10 http servers that serve Hls fragments as the backend servers and about 5 varnish servers for caching purpose, the problem comes in when I use round-robin director for backend servers in
varnish,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
if a varnish for specific file requests to one backend server and for the same file but to another backend server it would cache that file again because of different Host headers ! so my solution is using fallback director instead of round-robin as follow:</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
```</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
In varnish-1:
<div> new hls_cluster = directors.fallback();</div>
<div> hls_cluster.add_backend(b1());</div>
<div> hls_cluster.add_backend(b2()); </div>
<div> hls_cluster.add_backend(b3()); </div>
<div> hls_cluster.add_backend(b4()); </div>
<div> hls_cluster.add_backend(b5()); </div>
<div> hls_cluster.add_backend(b6()); </div>
<div> hls_cluster.add_backend(b7()); </div>
<div> hls_cluster.add_backend(b8()); </div>
<div> hls_cluster.add_backend(b9()); </div>
<div> hls_cluster.add_backend(b10());</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>In varnish-2:</div>
<div> new hls_cluster = directors.fallback();</div>
<div> hls_cluster.add_backend(b10());</div>
<div> hls_cluster.add_backend(b1());</div>
<div> hls_cluster.add_backend(b2()); </div>
<div> hls_cluster.add_backend(b3()); </div>
<div> hls_cluster.add_backend(b4()); </div>
<div> hls_cluster.add_backend(b5()); </div>
<div> hls_cluster.add_backend(b6()); </div>
<div> hls_cluster.add_backend(b7()); </div>
<div> hls_cluster.add_backend(b8()); </div>
<div> hls_cluster.add_backend(b9()); </div>
<div><br>
</div>
<div><br>
</div>
<div>In varnish-3:</div>
<div> new hls_cluster = directors.fallback();</div>
<div> hls_cluster.add_backend(b9()); </div>
<div> hls_cluster.add_backend(b1());</div>
<div> hls_cluster.add_backend(b2()); </div>
<div> hls_cluster.add_backend(b3()); </div>
<div> hls_cluster.add_backend(b4()); </div>
<div> hls_cluster.add_backend(b5()); </div>
<div> hls_cluster.add_backend(b6()); </div>
<div> hls_cluster.add_backend(b7()); </div>
<div> hls_cluster.add_backend(b8()); </div>
<div> hls_cluster.add_backend(b10());</div>
<div><br>
</div>
<span></span>```<br>
But I think this is not the best solution, because there is no load balancing despite, I used different backend for the first argument of fallback directive,<br>
What is varnish recommendation for this scenario?</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
</div>
</div>
_______________________________________________<br>
varnish-misc mailing list<br>
<a href="mailto:varnish-misc@varnish-cache.org" target="_blank">varnish-misc@varnish-cache.org</a><br>
<a href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc" rel="noreferrer" target="_blank">https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc</a><br>
</blockquote></div>