<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p><span class="Y2IQFc" lang="en">sorry I thought I noted it in my
previous email.</span></p>
<div class="tw-ta-container F0azHf tw-nfl"
id="tw-target-text-container" tabindex="0"><span class="Y2IQFc"
lang="en">here are the good lines for using pipe :<br>
</span></div>
<p><br>
<i>if (req.url ~ "(?i)^/[^?]+\.mp4($|\?)") {</i><i><br>
</i><i> std.log("ispiped:true");</i><i><br>
</i><i> return (pipe);</i><i><br>
</i><i>}</i><br>
</p>
<p>thank you!</p>
<p>Karim.<br>
</p>
<p><br>
</p>
<p><br>
</p>
<pre class="moz-signature" cols="72">
</pre>
<div class="moz-cite-prefix">Le 20/02/2023 à 21:50, Guillaume
Quintard a écrit :<br>
</div>
<blockquote type="cite"
cite="mid:CAOrPor=zG5PkKr-_3sVnTjfFED_Xepmk5_BMBLEgosZG6r5bLw@mail.gmail.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div>Looks like Rainer replied directly to you and not to the
list, would you mind sharing/highlighting the fix for people
having the same issue?</div>
<div><br>
</div>
<div>Cheers,</div>
<div><br>
</div>
<div>
<div>
<div dir="ltr" class="gmail_signature"
data-smartmail="gmail_signature">
<div dir="ltr">
<div>-- <br>
</div>
<div>Guillaume Quintard<br>
</div>
</div>
</div>
</div>
<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, Feb 20, 2023 at 12:36
PM Karim Ayari <<a href="mailto:karim.ayari@univ-lyon1.fr"
moz-do-not-send="true" class="moz-txt-link-freetext">karim.ayari@univ-lyon1.fr</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p><span lang="en"><span><span>thank you both for your
replies.</span></span><span><span> </span></span><span><span>I
forgot the most important thing:</span></span><span><span>
</span></span><span><span>the varnish server has 16 GB
of ram and the cache is 1 GB.</span></span><span><span>
</span></span><span><span><br>
</span></span></span></p>
<p><span lang="en"><span><span>the cache never seems full</span></span></span></p>
<p><span lang="en"><span><span><img
src="cid:part1.hll3XDne.buey2s0m@univ-lyon1.fr"
alt="" class="" width="466" height="171"><img
src="cid:part2.3Xb0PaGW.FM1miA0J@univ-lyon1.fr"
alt="" class="" width="478" height="172"></span></span></span></p>
<p><span lang="en"><span><span><br>
</span></span></span></p>
<p><span lang="en"><span><span>the command used : <b>/usr/sbin/varnishd
-j unix,user=vcache -F -a:8080 -T localhost:6082
-f /etc/varnish/moodle.vcl -S /etc/varnish/secret
-s malloc,1g</b></span></span><b> </b><b><span><span>-p
http_max_hdr=96</span></span></b><b><span><span> </span></span></b><span><span><br>
</span></span></span></p>
<p><span lang="en"><span><span>I had tried using pipe it
didn't work, but Rainer's lines works fine.</span></span></span></p>
<p><span lang="en"><span><span>this solution should suffice
because the videos will soon have to be hosted on
our video platform.</span></span></span><span
lang="en"><span><span></span></span></span></p>
<p><span lang="en"><i><br>
</i></span></p>
<p><span lang="en">my vcl (file found on github) :</span></p>
<p><span lang="en"><i><br>
</i></span></p>
<p><span lang="en"><i>....<br>
</i></span></p>
<p><span lang="en"><i>sub vcl_recv {</i><i><br>
</i><i><br>
</i><i> # Keep client IP</i><i><br>
</i><i> if (req.restarts == 0) {</i><i><br>
</i><i> if (req.http.x-forwarded-for) {</i><i><br>
</i><i> set req.http.X-Forwarded-For =
req.http.X-Forwarded-For + ", " + client.ip;</i><i><br>
</i><i> } else {</i><i><br>
</i><i> unset req.http.X-Forwarded-For;</i><i><br>
</i><i> set req.http.X-Forwarded-For =
client.ip;</i><i><br>
</i><i> }</i><i><br>
</i><i> }</i><i><br>
</i><i><br>
</i><i> if (req.http.X-Real-IP) {</i><i><br>
</i><i> set req.http.X-Forwarded-For =
req.http.X-Real-IP;</i><i><br>
</i><i> } else {</i><i><br>
</i><i> set req.http.X-Forwarded-For = client.ip;</i><i><br>
</i><i> }</i><i><br>
</i><i> </i><i><br>
</i><i> # Only deal with "normal" types</i><i><br>
</i><i> if (req.method != "GET" &&</i><i><br>
</i><i> req.method != "HEAD" &&</i><i><br>
</i><i> req.method != "PUT" &&</i><i><br>
</i><i> req.method != "POST" &&</i><i><br>
</i><i> req.method != "TRACE" &&</i><i><br>
</i><i> req.method != "OPTIONS" &&</i><i><br>
</i><i> req.method != "PATCH" &&</i><i><br>
</i><i> req.method != "DELETE") {</i><i><br>
</i><i> /* Non-RFC2616 or CONNECT which is weird.
*/</i><i><br>
</i><i> /*Why send the packet upstream, while the
visitor is using a non-valid HTTP method? */</i><i><br>
</i><i> return (synth(404, "Non-valid HTTP
method!"));</i><i><br>
</i><i> }</i><i><br>
</i><i> </i><i><br>
</i><i> # Varnish don't mess with healthchecks</i><i><br>
</i><i> if (req.url ~ "^/admin/tool/heartbeat" ||
req.url ~ "^/healthcheck.php") {</i><i><br>
</i><i> return (pass);</i><i><br>
</i><i> }</i><i><br>
</i><i> # Pipe requests to backup.php straight to
backend - prevents problem with progress bar long
polling 503 problem</i><i><br>
</i><i> # This is here because backup.php is POSTing
to itself - Filter before !GET&&!HEAD</i><i><br>
</i><i> if (req.url ~ "^/backup/backup.php")</i><i><br>
</i><i> {</i><i><br>
</i><i> return (pipe);</i><i><br>
</i><i> }</i><i><br>
</i><i><br>
</i><i> # Varnish only deals with GET and HEAD by
default. If request method is not GET or HEAD, pass
request to backend</i><i><br>
</i><i> if (req.method != "GET" && req.method
!= "HEAD") {</i><i><br>
</i><i> return (pass);</i><i><br>
</i><i> }</i><i><br>
</i><i><br>
</i><i> if (req.http.Cookie) {</i><i><br>
</i><i> # Remove any Google Analytics based cookies</i><i><br>
</i><i> set req.http.Cookie =
regsuball(req.http.Cookie, "^_ga$", "");</i><i><br>
</i><i> set req.http.Cookie =
regsuball(req.http.Cookie, "^_gid$", "");</i><i><br>
</i><i> set req.http.Cookie =
regsuball(req.http.Cookie, "__gads=[^;]+(; )?", "");</i><i><br>
</i><i> set req.http.Cookie =
regsuball(req.http.Cookie, "__qc.=[^;]+(; )?", "");</i><i><br>
</i><i> set req.http.Cookie =
regsuball(req.http.Cookie, "__atuv.=[^;]+(; )?", "");</i><i><br>
</i><i> set req.http.Cookie =
regsuball(req.http.Cookie, "^;\s*", "");</i><i><br>
</i><i> if (req.http.Cookie ~ "^\s*$") {</i><i><br>
</i><i> unset req.http.Cookie;</i><i><br>
</i><i> }</i><i><br>
</i><i> }</i><i><br>
</i><i><br>
</i><i> ### Rules for Moodle ###</i><i><br>
</i><i><br>
</i><i> # Perform lookup for selected assets that we
know are static but Moodle still needs a Cookie</i><i><br>
</i><i> if( req.url ~ "^/theme/.+\.?" ||</i><i><br>
</i><i> req.url ~
"^/webservice/pluginfile.php/.+\.(png|jpg)$" ||</i><i><br>
</i><i> req.url ~
"^/lib/.+\.(png|jpg|jpeg|gif|css|js|webp)$" ||</i><i><br>
</i><i> req.url ~
"^/pluginfile.php/[0-9]+/course/.+\.(?i)(png|jpg)$" ||</i><i><br>
</i><i> req.url ~
"^/pluginfile.php/[0-9]+/theme_moove/.+\.(?i)(png|jpg)$"</i><i><br>
</i><i> )</i><i><br>
</i><i> {</i><i><br>
</i><i> # Set internal temporary header, based
on which we will do things in vcl_backend_response</i><i><br>
</i><i> set req.http.X-Long-TTL = "86400";</i><i><br>
</i><i> return (hash);</i><i><br>
</i><i> }</i><i><br>
</i><i> # Requests containing "Cookie" or
"Authorization" headers will not be cached</i><i><br>
</i><i> if (req.http.Authorization ||
req.http.Cookie) {</i><i><br>
</i><i> return (pass);</i><i><br>
</i><i> }</i><i><br>
</i><i> # Almost everything in Moodle correctly
serves Cache-Control headers, if</i><i><br>
</i><i> # needed, which varnish will honor, but there
are some which don't. Rather</i><i><br>
</i><i> # than explicitly finding them all and
listing them here we just fail safe</i><i><br>
</i><i> # and don't cache unknown urls that get this
far.</i><i><br>
</i><i> return (pass);</i><i><br>
</i><i><br>
</i><i>}</i><i><br>
</i><i><br>
</i><i>sub vcl_backend_response {</i><i><br>
</i><i> # Set backend name</i><i><br>
</i><i> set beresp.http.X-Backend = <a
href="http://beresp.backend.name" target="_blank"
moz-do-not-send="true">beresp.backend.name</a>;</i><i><br>
</i><i><br>
</i><i> if (beresp.http.Cache-Control &&
bereq.http.X-Long-TTL && beresp.ttl <
std.duration(bereq.http.X-Long-TTL + "s", 1s)
&& !beresp.http.WWW-Authenticate ) { # If
max-age < defined in X-Long-TTL header</i><i><br>
</i><i> set beresp.http.X-Orig-Pragma =
beresp.http.Pragma; unset beresp.http.Pragma;</i><i><br>
</i><i> set beresp.http.X-Orig-Cache-Control =
beresp.http.Cache-Control;</i><i><br>
</i><i> set beresp.http.Cache-Control = "public,
max-age="+bereq.http.X-Long-TTL+", no-transform";</i><i><br>
</i><i> set beresp.ttl =
std.duration(bereq.http.X-Long-TTL + "s", 1s);</i><i><br>
</i><i> unset bereq.http.X-Long-TTL;</i><i><br>
</i><i> }</i><i><br>
</i><i> else if (!beresp.http.Cache-Control
&& bereq.http.X-Long-TTL &&
!beresp.http.WWW-Authenticate ) {</i><i><br>
</i><i> set beresp.http.X-Orig-Pragma =
beresp.http.Pragma; unset beresp.http.Pragma;</i><i><br>
</i><i> set beresp.http.Cache-Control = "public,
max-age="+bereq.http.X-Long-TTL+", no-transform";</i><i><br>
</i><i> set beresp.ttl =
std.duration(bereq.http.X-Long-TTL + "s", 1s);</i><i><br>
</i><i> unset bereq.http.X-Long-TTL;</i><i><br>
</i><i> }</i><i><br>
</i><i> else { # Don't touch headers if max-age >
defined in X-Long-TTL header</i><i><br>
</i><i> unset bereq.http.X-Long-TTL;</i><i><br>
</i><i> }</i><i><br>
</i><i> # Here we set X-Trace header, prepending it
to X-Trace header received from backend. Useful for
troubleshooting</i><i><br>
</i><i> if (beresp.http.x-trace &&
!beresp.was_304) {</i><i><br>
</i><i> set beresp.http.X-Trace =
regsub(server.identity, "^([^.]+),?.*$",
"\1")+"->"+regsub(<a
href="http://beresp.backend.name" target="_blank"
moz-do-not-send="true">beresp.backend.name</a>,
"^(.+)\((?:[0-9]{1,3}\.){3}([0-9]{1,3})\)","\1(\2)")+"->"+beresp.http.X-Trace;</i><i><br>
</i><i> }</i><i><br>
</i><i> else {</i><i><br>
</i><i> set beresp.http.X-Trace =
regsub(server.identity, "^([^.]+),?.*$",
"\1")+"->"+regsub(<a
href="http://beresp.backend.name" target="_blank"
moz-do-not-send="true">beresp.backend.name</a>,
"^(.+)\((?:[0-9]{1,3}\.){3}([0-9]{1,3})\)","\1(\2)");</i><i><br>
</i><i> }</i><i><br>
</i><i>}</i><i><br>
</i><i><br>
</i><i>sub vcl_deliver {</i><i><br>
</i><i><br>
</i><i># Revert back to original Cache-Control header
before delivery to client</i><i><br>
</i><i> if (resp.http.X-Orig-Cache-Control)</i><i><br>
</i><i> {</i><i><br>
</i><i> set resp.http.Cache-Control =
resp.http.X-Orig-Cache-Control;</i><i><br>
</i><i> unset resp.http.X-Orig-Cache-Control;</i><i><br>
</i><i> }</i><i><br>
</i><i> # Revert back to original Pragma header
before delivery to client</i><i><br>
</i><i> if (resp.http.X-Orig-Pragma)</i><i><br>
</i><i> {</i><i><br>
</i><i> set resp.http.Pragma =
resp.http.X-Orig-Pragma;</i><i><br>
</i><i> unset resp.http.X-Orig-Pragma;</i><i><br>
</i><i> }</i><i><br>
</i><i><br>
</i><i> if (obj.hits > 0) { # Add debug header to
see if it's a HIT/MISS and the number of hits, disable
when not needed</i><i><br>
</i><i> set resp.http.X-Cache = "HIT";</i><i><br>
</i><i> } else {</i><i><br>
</i><i> set resp.http.X-Cache = "MISS";</i><i><br>
</i><i> }</i><i><br>
</i><i> set resp.http.X-Cache-Hits = obj.hits;</i><i><br>
</i><i><br>
</i><i># If desired "Via: 1.1 Varnish-v4" response
header can be removed from response</i><i><br>
</i><i> unset resp.http.Via;</i><i><br>
</i><i> unset resp.http.Server;</i><i><br>
</i><i><br>
</i><i> return (deliver);</i><i><br>
</i><i>}</i><i><br>
</i><i><br>
</i><i>sub vcl_backend_error {</i><i><br>
</i><i> # More comprehensive varnish error page.
Display time, instance hostname, host header, url for
easier troubleshooting.</i><i><br>
</i><i> set beresp.http.Content-Type = "text/html;
charset=utf-8";</i><i><br>
</i><i> set beresp.http.Retry-After = "5";</i><i><br>
</i><i> synthetic( {"</i><i><br>
</i><i> <!DOCTYPE html></i><i><br>
</i><i> <html></i><i><br>
</i><i> <head></i><i><br>
</i><i> <title>"} + beresp.status + " " +
beresp.reason + {"</title></i><i><br>
</i><i> </head></i><i><br>
</i><i> <body></i><i><br>
</i><i> <h1>Error "} + beresp.status + " " +
beresp.reason + {"</h1></i><i><br>
</i><i> <p>"} + beresp.reason + {"</p></i><i><br>
</i><i> <h3>Guru Meditation:</h3></i><i><br>
</i><i> <p>Time: "} + now + {"</p></i><i><br>
</i><i> <p>Node: "} + server.hostname +
{"</p></i><i><br>
</i><i> <p>Host: "} + bereq.http.host +
{"</p></i><i><br>
</i><i> <p>URL: "} + bereq.url + {"</p></i><i><br>
</i><i> <p>XID: "} + bereq.xid + {"</p></i><i><br>
</i><i> <hr></i><i><br>
</i><i> <p>Varnish cache server</i><i><br>
</i><i> </body></i><i><br>
</i><i> </html></i><i><br>
</i><i> "} );</i><i><br>
</i><i> return (deliver);</i><i><br>
</i><i>}</i><i><br>
</i><i><br>
</i><i>sub vcl_synth {</i><i><br>
</i><i> #Redirect using '301 - Permanent Redirect',
permanent redirect</i><i><br>
</i><i> if (resp.status == 851) { </i><i><br>
</i><i> set resp.http.Location =
req.http.x-redir;</i><i><br>
</i><i> set resp.http.X-Varnish-Redirect = true;</i><i><br>
</i><i> set resp.status = 301;</i><i><br>
</i><i> return (deliver);</i><i><br>
</i><i> }</i><i><br>
</i><i> #Redirect using '302 - Found', temporary
redirect</i><i><br>
</i><i> if (resp.status == 852) { </i><i><br>
</i><i> set resp.http.Location =
req.http.x-redir;</i><i><br>
</i><i> set resp.http.X-Varnish-Redirect = true;</i><i><br>
</i><i> set resp.status = 302;</i><i><br>
</i><i> return (deliver);</i><i><br>
</i><i> }</i><i><br>
</i><i> #Redirect using '307 - Temporary Redirect',
!GET&&!HEAD requests, dont change method on
redirected requests</i><i><br>
</i><i> if (resp.status == 857) { </i><i><br>
</i><i> set resp.http.Location =
req.http.x-redir;</i><i><br>
</i><i> set resp.http.X-Varnish-Redirect = true;</i><i><br>
</i><i> set resp.status = 307;</i><i><br>
</i><i> return (deliver);</i><i><br>
</i><i> }</i><i><br>
</i><i> #Respond with 403 - Forbidden</i><i><br>
</i><i> if (resp.status == 863) {</i><i><br>
</i><i> set resp.http.X-Varnish-Error = true;</i><i><br>
</i><i> set resp.status = 403;</i><i><br>
</i><i> return (deliver);</i><i><br>
</i><i> }</i><i><br>
</i><i>}</i><i><br>
</i><i><br>
</i><i>sub vcl_purge {</i><i><br>
</i><i> if (req.method != "PURGE") {</i><i><br>
</i><i> set req.http.X-Purge = "Yes";</i><i><br>
</i><i> return (restart);</i><i><br>
</i><i> }</i><i><br>
</i><i>}</i><span><span><br>
</span></span></span></p>
<p>...<br>
</p>
<p><br>
</p>
<pre cols="72">
</pre>
<div>Le 20/02/2023 à 18:55, Guillaume Quintard a écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Hello Karim,</div>
<div><br>
</div>
<div>You VCL would be useful to debug this (as well as
the command line you are running Varnish with), but it
sounds like Varnish is using the Transient storage (<a
href="https://varnish-cache.org/docs/trunk/users-guide/storage-backends.html#transient-storage"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://varnish-cache.org/docs/trunk/users-guide/storage-backends.html#transient-storage</a>)
to store the file, and as the storage isn't bounded,
it explodes.</div>
<div>We can fix this in a couple of ways, from storing
the file in the regular cache storage, to using pipe,
to waiting a few days for <a
href="https://github.com/varnishcache/varnish-cache/pull/3572#issuecomment-1305736643"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://github.com/varnishcache/varnish-cache/pull/3572#issuecomment-1305736643</a>
to be released.</div>
<div><br>
</div>
<div>Question is: should that file be cached?</div>
<div><br>
</div>
<div>Cheers,</div>
<div><br>
</div>
<div>
<div>
<div dir="ltr">
<div dir="ltr">
<div>-- <br>
</div>
<div>Guillaume Quintard<br>
</div>
</div>
</div>
</div>
<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, Feb 20, 2023
at 7:14 AM Karim Ayari <<a
href="mailto:karim.ayari@univ-lyon1.fr"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">karim.ayari@univ-lyon1.fr</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px
0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p><span lang="en"><span><span>Hi!</span></span><span><span><br>
</span></span></span></p>
<p><span lang="en"><span><span>I am currently
experiencing a memory load problem with
video playback.</span></span></span></p>
<p><span lang="en"><span><span></span></span><span><span>here
is the infrastructure :<br>
</span></span></span></p>
<p><span lang="en"><span><span>client --> haproxy
--> varnish --> moodle workers (x5)</span></span><span><span>
</span></span><span><span><br>
</span></span></span></p>
<p><span lang="en"><span><span>a teacher uploaded a
400MB video to Moodle, when we start playing
the video with browser player, </span></span></span><span
lang="en"><span><span><span lang="en"><span><span>Varnish</span></span></span>
consumes all the memory until it runs out
and oom killer to kill varnishd.</span></span><span><span>
</span></span><span><span>i have no
configuration for mp4 files in my vcl file,
so by default they are not hidden (?). </span></span></span><span
lang="en"><span><span>I can't find a solution :(</span></span></span><span
lang="en"><span><span></span></span></span></p>
<p><span lang="en"><span><span>I can give my vcl
file if necessary.</span></span><span><span><br>
</span></span></span></p>
<p><span lang="en"><span><span>(I am a beginner on
varnish :))</span></span></span><span
lang="en"><span><span></span></span></span></p>
<p><span lang="en"><span><span>thank you for your
support.</span></span><span><span><br>
</span></span></span></p>
<p><span lang="en"><span><span>Karim<br>
</span></span></span></p>
</div>
_______________________________________________<br>
varnish-misc mailing list<br>
<a href="mailto:varnish-misc@varnish-cache.org"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">varnish-misc@varnish-cache.org</a><br>
<a
href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc"
rel="noreferrer" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc</a><br>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
</body>
</html>