<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p><span class="HwtZe" lang="en"><span class="jCAhz ChMk0b"><span
class="ryNqvb">thank you both for your replies.</span></span><span
class="jCAhz"><span class="ryNqvb">
</span></span><span class="jCAhz ChMk0b"><span class="ryNqvb">I
forgot the most important thing:</span></span><span
class="jCAhz"><span class="ryNqvb">
</span></span><span class="jCAhz ChMk0b"><span class="ryNqvb">the
varnish server has 16 GB of ram and the cache is 1 GB.</span></span><span
class="jCAhz"><span class="ryNqvb">
</span></span><span class="jCAhz ChMk0b"><span class="ryNqvb"><br>
</span></span></span></p>
<p><span class="HwtZe" lang="en"><span class="jCAhz ChMk0b"><span
class="ryNqvb">the cache never seems full</span></span></span></p>
<p><span class="HwtZe" lang="en"><span class="jCAhz ChMk0b"><span
class="ryNqvb"><img
src="cid:part1.mXAv80hH.OGTmctED@univ-lyon1.fr" alt=""
width="466" height="171"><img
src="cid:part2.wXefU0Jn.sBrK77yZ@univ-lyon1.fr" alt=""
width="478" height="172"></span></span></span></p>
<p><span class="HwtZe" lang="en"><span class="jCAhz ChMk0b"><span
class="ryNqvb"><br>
</span></span></span></p>
<p><span class="HwtZe" lang="en"><span class="jCAhz ChMk0b"><span
class="ryNqvb">the command used : <b>/usr/sbin/varnishd -j
unix,user=vcache -F -a:8080 -T localhost:6082 -f
/etc/varnish/moodle.vcl -S /etc/varnish/secret -s
malloc,1g</b></span></span><b> </b><b><span class="jCAhz
ChMk0b"><span class="ryNqvb">-p http_max_hdr=96</span></span></b><b><span
class="jCAhz"><span class="ryNqvb">
</span></span></b><span class="jCAhz ChMk0b"><span
class="ryNqvb"><br>
</span></span></span></p>
<p><span class="HwtZe" lang="en"><span class="jCAhz ChMk0b"><span
class="ryNqvb">I had tried using pipe it didn't work, but
Rainer's lines works fine.</span></span></span></p>
<p><span class="HwtZe" lang="en"><span class="jCAhz ChMk0b"><span
class="ryNqvb">this solution should suffice because the
videos will soon have to be hosted on our video platform.</span></span></span><span
class="HwtZe" lang="en"><span class="jCAhz ChMk0b"><span
class="ryNqvb"></span></span></span></p>
<p><span class="HwtZe" lang="en"><i><br>
</i></span></p>
<p><span class="HwtZe" lang="en">my vcl (file found on github) :</span></p>
<p><span class="HwtZe" lang="en"><i><br>
</i></span></p>
<p><span class="HwtZe" lang="en"><i>....<br>
</i></span></p>
<p><span class="HwtZe" lang="en"><i>sub vcl_recv {</i><i><br>
</i><i><br>
</i><i> # Keep client IP</i><i><br>
</i><i> if (req.restarts == 0) {</i><i><br>
</i><i> if (req.http.x-forwarded-for) {</i><i><br>
</i><i> set req.http.X-Forwarded-For =
req.http.X-Forwarded-For + ", " + client.ip;</i><i><br>
</i><i> } else {</i><i><br>
</i><i> unset req.http.X-Forwarded-For;</i><i><br>
</i><i> set req.http.X-Forwarded-For = client.ip;</i><i><br>
</i><i> }</i><i><br>
</i><i> }</i><i><br>
</i><i><br>
</i><i> if (req.http.X-Real-IP) {</i><i><br>
</i><i> set req.http.X-Forwarded-For =
req.http.X-Real-IP;</i><i><br>
</i><i> } else {</i><i><br>
</i><i> set req.http.X-Forwarded-For = client.ip;</i><i><br>
</i><i> }</i><i><br>
</i><i> </i><i><br>
</i><i> # Only deal with "normal" types</i><i><br>
</i><i> if (req.method != "GET" &&</i><i><br>
</i><i> req.method != "HEAD" &&</i><i><br>
</i><i> req.method != "PUT" &&</i><i><br>
</i><i> req.method != "POST" &&</i><i><br>
</i><i> req.method != "TRACE" &&</i><i><br>
</i><i> req.method != "OPTIONS" &&</i><i><br>
</i><i> req.method != "PATCH" &&</i><i><br>
</i><i> req.method != "DELETE") {</i><i><br>
</i><i> /* Non-RFC2616 or CONNECT which is weird. */</i><i><br>
</i><i> /*Why send the packet upstream, while the visitor is
using a non-valid HTTP method? */</i><i><br>
</i><i> return (synth(404, "Non-valid HTTP method!"));</i><i><br>
</i><i> }</i><i><br>
</i><i> </i><i><br>
</i><i> # Varnish don't mess with healthchecks</i><i><br>
</i><i> if (req.url ~ "^/admin/tool/heartbeat" || req.url ~
"^/healthcheck.php") {</i><i><br>
</i><i> return (pass);</i><i><br>
</i><i> }</i><i><br>
</i><i> # Pipe requests to backup.php straight to backend -
prevents problem with progress bar long polling 503 problem</i><i><br>
</i><i> # This is here because backup.php is POSTing to
itself - Filter before !GET&&!HEAD</i><i><br>
</i><i> if (req.url ~ "^/backup/backup.php")</i><i><br>
</i><i> {</i><i><br>
</i><i> return (pipe);</i><i><br>
</i><i> }</i><i><br>
</i><i><br>
</i><i> # Varnish only deals with GET and HEAD by default. If
request method is not GET or HEAD, pass request to backend</i><i><br>
</i><i> if (req.method != "GET" && req.method !=
"HEAD") {</i><i><br>
</i><i> return (pass);</i><i><br>
</i><i> }</i><i><br>
</i><i><br>
</i><i> if (req.http.Cookie) {</i><i><br>
</i><i> # Remove any Google Analytics based cookies</i><i><br>
</i><i> set req.http.Cookie = regsuball(req.http.Cookie,
"^_ga$", "");</i><i><br>
</i><i> set req.http.Cookie = regsuball(req.http.Cookie,
"^_gid$", "");</i><i><br>
</i><i> set req.http.Cookie = regsuball(req.http.Cookie,
"__gads=[^;]+(; )?", "");</i><i><br>
</i><i> set req.http.Cookie = regsuball(req.http.Cookie,
"__qc.=[^;]+(; )?", "");</i><i><br>
</i><i> set req.http.Cookie = regsuball(req.http.Cookie,
"__atuv.=[^;]+(; )?", "");</i><i><br>
</i><i> set req.http.Cookie = regsuball(req.http.Cookie,
"^;\s*", "");</i><i><br>
</i><i> if (req.http.Cookie ~ "^\s*$") {</i><i><br>
</i><i> unset req.http.Cookie;</i><i><br>
</i><i> }</i><i><br>
</i><i> }</i><i><br>
</i><i><br>
</i><i> ### Rules for Moodle ###</i><i><br>
</i><i><br>
</i><i> # Perform lookup for selected assets that we know are
static but Moodle still needs a Cookie</i><i><br>
</i><i> if( req.url ~ "^/theme/.+\.?" ||</i><i><br>
</i><i> req.url ~
"^/webservice/pluginfile.php/.+\.(png|jpg)$" ||</i><i><br>
</i><i> req.url ~
"^/lib/.+\.(png|jpg|jpeg|gif|css|js|webp)$" ||</i><i><br>
</i><i> req.url ~
"^/pluginfile.php/[0-9]+/course/.+\.(?i)(png|jpg)$" ||</i><i><br>
</i><i> req.url ~
"^/pluginfile.php/[0-9]+/theme_moove/.+\.(?i)(png|jpg)$"</i><i><br>
</i><i> )</i><i><br>
</i><i> {</i><i><br>
</i><i> # Set internal temporary header, based on which
we will do things in vcl_backend_response</i><i><br>
</i><i> set req.http.X-Long-TTL = "86400";</i><i><br>
</i><i> return (hash);</i><i><br>
</i><i> }</i><i><br>
</i><i> # Requests containing "Cookie" or "Authorization"
headers will not be cached</i><i><br>
</i><i> if (req.http.Authorization || req.http.Cookie) {</i><i><br>
</i><i> return (pass);</i><i><br>
</i><i> }</i><i><br>
</i><i> # Almost everything in Moodle correctly serves
Cache-Control headers, if</i><i><br>
</i><i> # needed, which varnish will honor, but there are
some which don't. Rather</i><i><br>
</i><i> # than explicitly finding them all and listing them
here we just fail safe</i><i><br>
</i><i> # and don't cache unknown urls that get this far.</i><i><br>
</i><i> return (pass);</i><i><br>
</i><i><br>
</i><i>}</i><i><br>
</i><i><br>
</i><i>sub vcl_backend_response {</i><i><br>
</i><i> # Set backend name</i><i><br>
</i><i> set beresp.http.X-Backend = beresp.backend.name;</i><i><br>
</i><i><br>
</i><i> if (beresp.http.Cache-Control &&
bereq.http.X-Long-TTL && beresp.ttl <
std.duration(bereq.http.X-Long-TTL + "s", 1s) &&
!beresp.http.WWW-Authenticate ) { # If max-age < defined in
X-Long-TTL header</i><i><br>
</i><i> set beresp.http.X-Orig-Pragma =
beresp.http.Pragma; unset beresp.http.Pragma;</i><i><br>
</i><i> set beresp.http.X-Orig-Cache-Control =
beresp.http.Cache-Control;</i><i><br>
</i><i> set beresp.http.Cache-Control = "public,
max-age="+bereq.http.X-Long-TTL+", no-transform";</i><i><br>
</i><i> set beresp.ttl =
std.duration(bereq.http.X-Long-TTL + "s", 1s);</i><i><br>
</i><i> unset bereq.http.X-Long-TTL;</i><i><br>
</i><i> }</i><i><br>
</i><i> else if (!beresp.http.Cache-Control &&
bereq.http.X-Long-TTL && !beresp.http.WWW-Authenticate
) {</i><i><br>
</i><i> set beresp.http.X-Orig-Pragma =
beresp.http.Pragma; unset beresp.http.Pragma;</i><i><br>
</i><i> set beresp.http.Cache-Control = "public,
max-age="+bereq.http.X-Long-TTL+", no-transform";</i><i><br>
</i><i> set beresp.ttl =
std.duration(bereq.http.X-Long-TTL + "s", 1s);</i><i><br>
</i><i> unset bereq.http.X-Long-TTL;</i><i><br>
</i><i> }</i><i><br>
</i><i> else { # Don't touch headers if max-age > defined
in X-Long-TTL header</i><i><br>
</i><i> unset bereq.http.X-Long-TTL;</i><i><br>
</i><i> }</i><i><br>
</i><i> # Here we set X-Trace header, prepending it to
X-Trace header received from backend. Useful for
troubleshooting</i><i><br>
</i><i> if (beresp.http.x-trace && !beresp.was_304) {</i><i><br>
</i><i> set beresp.http.X-Trace = regsub(server.identity,
"^([^.]+),?.*$", "\1")+"->"+regsub(beresp.backend.name,
"^(.+)\((?:[0-9]{1,3}\.){3}([0-9]{1,3})\)","\1(\2)")+"->"+beresp.http.X-Trace;</i><i><br>
</i><i> }</i><i><br>
</i><i> else {</i><i><br>
</i><i> set beresp.http.X-Trace = regsub(server.identity,
"^([^.]+),?.*$", "\1")+"->"+regsub(beresp.backend.name,
"^(.+)\((?:[0-9]{1,3}\.){3}([0-9]{1,3})\)","\1(\2)");</i><i><br>
</i><i> }</i><i><br>
</i><i>}</i><i><br>
</i><i><br>
</i><i>sub vcl_deliver {</i><i><br>
</i><i><br>
</i><i># Revert back to original Cache-Control header before
delivery to client</i><i><br>
</i><i> if (resp.http.X-Orig-Cache-Control)</i><i><br>
</i><i> {</i><i><br>
</i><i> set resp.http.Cache-Control =
resp.http.X-Orig-Cache-Control;</i><i><br>
</i><i> unset resp.http.X-Orig-Cache-Control;</i><i><br>
</i><i> }</i><i><br>
</i><i> # Revert back to original Pragma header before
delivery to client</i><i><br>
</i><i> if (resp.http.X-Orig-Pragma)</i><i><br>
</i><i> {</i><i><br>
</i><i> set resp.http.Pragma = resp.http.X-Orig-Pragma;</i><i><br>
</i><i> unset resp.http.X-Orig-Pragma;</i><i><br>
</i><i> }</i><i><br>
</i><i><br>
</i><i> if (obj.hits > 0) { # Add debug header to see if
it's a HIT/MISS and the number of hits, disable when not
needed</i><i><br>
</i><i> set resp.http.X-Cache = "HIT";</i><i><br>
</i><i> } else {</i><i><br>
</i><i> set resp.http.X-Cache = "MISS";</i><i><br>
</i><i> }</i><i><br>
</i><i> set resp.http.X-Cache-Hits = obj.hits;</i><i><br>
</i><i><br>
</i><i># If desired "Via: 1.1 Varnish-v4" response header can be
removed from response</i><i><br>
</i><i> unset resp.http.Via;</i><i><br>
</i><i> unset resp.http.Server;</i><i><br>
</i><i><br>
</i><i> return (deliver);</i><i><br>
</i><i>}</i><i><br>
</i><i><br>
</i><i>sub vcl_backend_error {</i><i><br>
</i><i> # More comprehensive varnish error page. Display
time, instance hostname, host header, url for easier
troubleshooting.</i><i><br>
</i><i> set beresp.http.Content-Type = "text/html;
charset=utf-8";</i><i><br>
</i><i> set beresp.http.Retry-After = "5";</i><i><br>
</i><i> synthetic( {"</i><i><br>
</i><i> <!DOCTYPE html></i><i><br>
</i><i> <html></i><i><br>
</i><i> <head></i><i><br>
</i><i> <title>"} + beresp.status + " " +
beresp.reason + {"</title></i><i><br>
</i><i> </head></i><i><br>
</i><i> <body></i><i><br>
</i><i> <h1>Error "} + beresp.status + " " +
beresp.reason + {"</h1></i><i><br>
</i><i> <p>"} + beresp.reason + {"</p></i><i><br>
</i><i> <h3>Guru Meditation:</h3></i><i><br>
</i><i> <p>Time: "} + now + {"</p></i><i><br>
</i><i> <p>Node: "} + server.hostname + {"</p></i><i><br>
</i><i> <p>Host: "} + bereq.http.host + {"</p></i><i><br>
</i><i> <p>URL: "} + bereq.url + {"</p></i><i><br>
</i><i> <p>XID: "} + bereq.xid + {"</p></i><i><br>
</i><i> <hr></i><i><br>
</i><i> <p>Varnish cache server</i><i><br>
</i><i> </body></i><i><br>
</i><i> </html></i><i><br>
</i><i> "} );</i><i><br>
</i><i> return (deliver);</i><i><br>
</i><i>}</i><i><br>
</i><i><br>
</i><i>sub vcl_synth {</i><i><br>
</i><i> #Redirect using '301 - Permanent Redirect', permanent
redirect</i><i><br>
</i><i> if (resp.status == 851) { </i><i><br>
</i><i> set resp.http.Location = req.http.x-redir;</i><i><br>
</i><i> set resp.http.X-Varnish-Redirect = true;</i><i><br>
</i><i> set resp.status = 301;</i><i><br>
</i><i> return (deliver);</i><i><br>
</i><i> }</i><i><br>
</i><i> #Redirect using '302 - Found', temporary redirect</i><i><br>
</i><i> if (resp.status == 852) { </i><i><br>
</i><i> set resp.http.Location = req.http.x-redir;</i><i><br>
</i><i> set resp.http.X-Varnish-Redirect = true;</i><i><br>
</i><i> set resp.status = 302;</i><i><br>
</i><i> return (deliver);</i><i><br>
</i><i> }</i><i><br>
</i><i> #Redirect using '307 - Temporary Redirect',
!GET&&!HEAD requests, dont change method on redirected
requests</i><i><br>
</i><i> if (resp.status == 857) { </i><i><br>
</i><i> set resp.http.Location = req.http.x-redir;</i><i><br>
</i><i> set resp.http.X-Varnish-Redirect = true;</i><i><br>
</i><i> set resp.status = 307;</i><i><br>
</i><i> return (deliver);</i><i><br>
</i><i> }</i><i><br>
</i><i> #Respond with 403 - Forbidden</i><i><br>
</i><i> if (resp.status == 863) {</i><i><br>
</i><i> set resp.http.X-Varnish-Error = true;</i><i><br>
</i><i> set resp.status = 403;</i><i><br>
</i><i> return (deliver);</i><i><br>
</i><i> }</i><i><br>
</i><i>}</i><i><br>
</i><i><br>
</i><i>sub vcl_purge {</i><i><br>
</i><i> if (req.method != "PURGE") {</i><i><br>
</i><i> set req.http.X-Purge = "Yes";</i><i><br>
</i><i> return (restart);</i><i><br>
</i><i> }</i><i><br>
</i><i>}</i><span class="jCAhz"><span class="ryNqvb"><br>
</span></span></span></p>
<p>...<br>
</p>
<p><br>
</p>
<pre class="moz-signature" cols="72">Karim Ayari
Université Claude Bernard - Lyon 1
Service ICAP
Ingénieur systèmes</pre>
<div class="moz-cite-prefix">Le 20/02/2023 à 18:55, Guillaume
Quintard a écrit :<br>
</div>
<blockquote type="cite"
cite="mid:CAOrPormuFTWqMyRn5=4YA56Ubx3Nu2a197aXE=HcZabbRfCmbA@mail.gmail.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div>Hello Karim,</div>
<div><br>
</div>
<div>You VCL would be useful to debug this (as well as the
command line you are running Varnish with), but it sounds like
Varnish is using the Transient storage (<a
href="https://varnish-cache.org/docs/trunk/users-guide/storage-backends.html#transient-storage"
moz-do-not-send="true" class="moz-txt-link-freetext">https://varnish-cache.org/docs/trunk/users-guide/storage-backends.html#transient-storage</a>)
to store the file, and as the storage isn't bounded, it
explodes.</div>
<div>We can fix this in a couple of ways, from storing the file
in the regular cache storage, to using pipe, to waiting a few
days for <a
href="https://github.com/varnishcache/varnish-cache/pull/3572#issuecomment-1305736643"
moz-do-not-send="true" class="moz-txt-link-freetext">https://github.com/varnishcache/varnish-cache/pull/3572#issuecomment-1305736643</a>
to be released.</div>
<div><br>
</div>
<div>Question is: should that file be cached?</div>
<div><br>
</div>
<div>Cheers,</div>
<div><br>
</div>
<div>
<div>
<div dir="ltr" class="gmail_signature"
data-smartmail="gmail_signature">
<div dir="ltr">
<div>-- <br>
</div>
<div>Guillaume Quintard<br>
</div>
</div>
</div>
</div>
<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, Feb 20, 2023 at 7:14
AM Karim Ayari <<a href="mailto:karim.ayari@univ-lyon1.fr"
moz-do-not-send="true" class="moz-txt-link-freetext">karim.ayari@univ-lyon1.fr</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p><span lang="en"><span><span>Hi!</span></span><span><span><br>
</span></span></span></p>
<p><span lang="en"><span><span>I am currently experiencing a
memory load problem with video playback.</span></span></span></p>
<p><span lang="en"><span><span></span></span><span><span>here
is the infrastructure :<br>
</span></span></span></p>
<p><span lang="en"><span><span>client --> haproxy -->
varnish --> moodle workers (x5)</span></span><span><span>
</span></span><span><span><br>
</span></span></span></p>
<p><span lang="en"><span><span>a teacher uploaded a 400MB
video to Moodle, when we start playing the video
with browser player, </span></span></span><span
lang="en"><span><span><span lang="en"><span><span>Varnish</span></span></span>
consumes all the memory until it runs out and oom
killer to kill varnishd.</span></span><span><span> </span></span><span><span>i
have no configuration for mp4 files in my vcl file,
so by default they are not hidden (?). </span></span></span><span
lang="en"><span><span>I can't find a solution :(</span></span></span><span
lang="en"><span><span></span></span></span></p>
<p><span lang="en"><span><span>I can give my vcl file if
necessary.</span></span><span><span><br>
</span></span></span></p>
<p><span lang="en"><span><span>(I am a beginner on varnish
:))</span></span></span><span lang="en"><span><span></span></span></span></p>
<p><span lang="en"><span><span>thank you for your support.</span></span><span><span><br>
</span></span></span></p>
<p><span lang="en"><span><span>Karim<br>
</span></span></span></p>
</div>
_______________________________________________<br>
varnish-misc mailing list<br>
<a href="mailto:varnish-misc@varnish-cache.org"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">varnish-misc@varnish-cache.org</a><br>
<a
href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc"
rel="noreferrer" target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc</a><br>
</blockquote>
</div>
</blockquote>
</body>
</html>