<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
Hi Guillaume
<div><br>
</div>
<div>Thanks for the response. Our Developers were worried that nobody would reply since we are using V6 :-) </div>
<div><br>
</div>
<div>The transient should be at 1G (<i>-s malloc,2G -s Transient=malloc,1G</i>)</div>
<div>Below are all the SMA values <b>[1]</b>, but at the moment the real value is much higher
<b>[2]</b></div>
<div><br>
</div>
<div>From <a href="https://varnish-cache.org/docs/trunk/users-guide/storage-backends.html">https://varnish-cache.org/docs/trunk/users-guide/storage-backends.html</a> my understanding is that transient is used when the TTL is below the short-lived setting.</div>
<div>The <span style="font-family: Menlo; font-size: 11px;">default_ttl</span> and <span style="font-family: Menlo; font-size: 11px;">shortlived</span> are the default settings, so 120s and 10s. </div>
<div>Checking the headers in the varnish log, all of these are coming in as “Cache-Control: max-age=3000”, </div>
<div>So I was actually thinking of setting the transient much lower, since as you can see in
<b>[3]</b> on a new server I started 3 days ago, the transient isn’t even being used at all.</div>
<div><br>
</div>
<div>As I mentioned, I could try V7, But with my sysadmin hat on, I really wanted to try find out what is going in the stable releases, before jumping to the latest release.</div>
<div><br>
</div>
<div>Thanks in Advance</div>
<div>Chris</div>
<div><br>
</div>
<div><br>
</div>
<div><b>[1]:</b></div>
<div>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 11px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;">
<span style="font-variant-ligatures: no-common-ligatures">[atlasfrontiergpn02 ~]# varnishstat -1 -f 'SMA*'</span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 11px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;">
<span style="font-variant-ligatures: no-common-ligatures">SMA.s0.c_req 4797047 174.74 Allocator requests</span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 11px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;">
<span style="font-variant-ligatures: no-common-ligatures">SMA.s0.c_fail 24725 0.90 Allocator failures</span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 11px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;">
<span style="font-variant-ligatures: no-common-ligatures">SMA.s0.c_bytes 73606101515 2681265.54 Bytes allocated</span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 11px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;">
<span style="font-variant-ligatures: no-common-ligatures">SMA.s0.c_freed 72001644367 2622819.63 Bytes freed</span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 11px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;">
<span style="font-variant-ligatures: no-common-ligatures">SMA.s0.g_alloc 113811 . Allocations outstanding</span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 11px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;">
<span style="font-variant-ligatures: no-common-ligatures">SMA.s0.g_bytes 1604457148 . Bytes outstanding</span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 11px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;">
<span style="font-variant-ligatures: no-common-ligatures">SMA.s0.g_space 543026500 . Bytes available</span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 11px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;">
<span style="font-variant-ligatures: no-common-ligatures">SMA.Transient.c_req 0 0.00 Allocator requests</span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 11px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;">
<span style="font-variant-ligatures: no-common-ligatures">SMA.Transient.c_fail 0 0.00 Allocator failures</span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 11px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;">
<span style="font-variant-ligatures: no-common-ligatures">SMA.Transient.c_bytes 0 0.00 Bytes allocated</span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 11px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;">
<span style="font-variant-ligatures: no-common-ligatures">SMA.Transient.c_freed 0 0.00 Bytes freed</span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 11px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;">
<span style="font-variant-ligatures: no-common-ligatures">SMA.Transient.g_alloc 0 . Allocations outstanding</span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 11px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;">
<span style="font-variant-ligatures: no-common-ligatures">SMA.Transient.g_bytes 0 . Bytes outstanding</span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 11px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;">
<span style="font-variant-ligatures: no-common-ligatures">SMA.Transient.g_space 1073741824 . Bytes available</span></p>
</div>
<div><br>
</div>
<div><b>[2]:</b></div>
<div>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 11px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;">
<span style="font-variant-ligatures: no-common-ligatures">[atlasfrontiergpn02 ~]# top -b -n 1 -p $(pgrep -d, -f varnishd) |egrep "PID|$(pgrep -d"|" -f varnishd)"</span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 11px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;">
<span style="font-variant-ligatures: no-common-ligatures"> </span><span style="font-variant-ligatures: no-common-ligatures; color: #b42419"><b>PID</b></span><span style="font-variant-ligatures: no-common-ligatures"> USER PR NI VIRT RES SHR
S %CPU %MEM TIME+ COMMAND</span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 11px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;">
<span style="font-variant-ligatures: no-common-ligatures; color: #b42419"><b>2685175</b></span><span style="font-variant-ligatures: no-common-ligatures"> varnish 20 0 14.1g 12.5g 86272 S 6.7 88.8 30:40.92 cache-main</span></p>
<p style="margin: 0px; font-style: normal; font-variant-caps: normal; font-width: normal; font-size: 11px; line-height: normal; font-family: Menlo; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-variant-emoji: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal;">
<span style="font-variant-ligatures: no-common-ligatures; color: #b42419"><b>2685154</b></span><span style="font-variant-ligatures: no-common-ligatures"> varnish 20 0 23748 6148 5376 S 0.0 0.0 0:01.20 varnishd</span></p>
</div>
<div><br>
</div>
<div><b>[3]:</b></div>
<div><img width="640" alt="PastedGraphic-1.png" src="cid:010E6BC5-E470-4268-973D-04B025A5B3EA"></div>
<div>
<div><br>
<blockquote type="cite">
<div>On 10 Jul 2025, at 07:17, Guillaume Quintard <guillaume.quintard@gmail.com> wrote:</div>
<br class="Apple-interchange-newline">
<div>
<div dir="auto">
<div dir="auto">Hi Chris,</div>
<div dir="auto"><br>
</div>
<div dir="auto">What are the g_bytes counters in varnishstat saying? If you haven't bounded your Transient storage. It could be a reason.</div>
<div dir="auto"><br>
</div>
<div dir="auto">The other suspect is the newer jemalloc version in the repository. If the problem isn't the transient storage, I would encourage you to try the packagecloud repository to get a newer version and see if this help.</div>
<div dir="auto"><br>
</div>
<div data-smartmail="gmail_signature" dir="auto">
<div dir="ltr">
<div>-- <br>
</div>
<div>Guillaume Quintard<br>
</div>
</div>
</div>
<br>
<div class="gmail_quote gmail_quote_container" dir="auto">
<div dir="ltr" class="gmail_attr">On Tue, Jul 8, 2025, 04:19 Chris Lee <<a href="mailto:chris.lee@cern.ch">chris.lee@cern.ch</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi all,<br>
<br>
I am trying to install a varnish service for some of our developers, and I’ll admit I know next to nothing about varnish itself.<br>
<br>
While things are up and running, Varnish was being killed every 2-3 hours by OOM-Killer as the cache-main process is using up all the system memory [1].<br>
We are using varnish-6.6.2-6.el9_6.1.x86_64 on a VM with 4 cores, 14Gi of RAM running on AlmaLinux release 9.6<br>
We could try to install V7, but would prefer to stay with the releases available from the default repositories which are mirrored locally.<br>
<br>
I have tried to change the malloc memory settings and gone down in 2G increments from 10G to 2G where it is now.
<br>
This has increased the number of evictions, but extended the uptime a bit.<br>
Adjusting the workspace_client and workspace_backend settings has increased the OOM interval to about 6-8 hours.<br>
<br>
The service is currently run via systemd as per the command line in [2].<br>
<br>
The hit rats as shown in [3] are fairly high from what I can tell, and the default.vlc is shown in [4]<br>
<br>
In the mailing list archives I found a link pointing to <a href="https://info.varnish-software.com/blog/understanding-varnish-cache-memory-usage" rel="noreferrer noreferrer" target="_blank">
https://info.varnish-software.com/blog/understanding-varnish-cache-memory-usage</a> and I haven’t tried to tune the malloc settings mentioned in there yet.<br>
<br>
But I’m running out of idea’s and though I would ask the experts here first for some guidance and assistance.<br>
<br>
Thanks in Advance<br>
Chris<br>
<br>
[1]:<br>
```<br>
[root@frontier-varnish02 ~]# top -p $(pgrep -d, -f varnishd) |egrep "PID|$(pgrep -d"|" -f varnishd)"<br>
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br>
740517 varnish 20 0 12.6g 11.1g 86528 S 20.0 78.8 27:42.25 cache-main <br>
740496 varnish 20 0 23748 6404 5632 S 0.0 0.0 0:00.52 varnishd <br>
```<br>
[2]:<br>
```<br>
/usr/sbin/varnishd -f /etc/varnish/default.vcl -a http=:6082,HTTP -a proxy=:8443,PROXY -p feature=+http2 -p max_restarts=8 -p workspace_client=512k -p workspace_backend=512k -s malloc,2G -s Transient=malloc,1G<br>
```<br>
[3]:<br>
```<br>
Uptime mgt: 0+03:23:57 Hitrate n: 10 100 171<br>
Uptime child: 0+03:23:58 avg(n): 0.9977 0.9940 0.9931<br>
Press <h> to toggle help screen<br>
NAME CURRENT CHANGE AVERAGE AVG_10 AVG_100 AVG_1000
<br>
MGT.uptime 0+03:23:57<br>
MAIN.uptime 0+03:23:58<br>
MAIN.sess_conn 2308418 123.91 188.63 139.32 133.20 132.53<br>
MAIN.client_req 21919893 2019.56 1791.13 1356.64 1257.93 1228.81<br>
MAIN.cache_hit 21866126 2016.56 1786.74 1354.38 1256.70 1227.64<br>
MAIN.cache_miss 47442 3.00 3.88 2.26 1.22 1.17<br>
```<br>
<br>
[4]:<br>
```<br>
vcl 4.1;<br>
import std;<br>
import directors;<br>
<br>
backend frontier_1 {<br>
.host = "<a href="http://atlasfrontier1-ai.cern.ch/" rel="noreferrer noreferrer" target="_blank">atlasfrontier1-ai.cern.ch</a>";<br>
.port = "8000";<br>
}<br>
backend frontier_2 {<br>
.host = "<a href="http://atlasfrontier2-ai.cern.ch/" rel="noreferrer noreferrer" target="_blank">atlasfrontier2-ai.cern.ch</a>";<br>
.port = "8000";<br>
}<br>
backend frontier_3 {<br>
.host = "<a href="http://atlasfrontier3-ai.cern.ch/" rel="noreferrer noreferrer" target="_blank">atlasfrontier3-ai.cern.ch</a>";<br>
.port = "8000";<br>
}<br>
backend frontier_4 {<br>
.host = "<a href="http://atlasfrontier4-ai.cern.ch/" rel="noreferrer noreferrer" target="_blank">atlasfrontier4-ai.cern.ch</a>";<br>
.port = "8000";<br>
}<br>
<br>
sub vcl_init {<br>
new vdir = directors.round_robin();<br>
vdir.add_backend(frontier_1);<br>
vdir.add_backend(frontier_2);<br>
vdir.add_backend(frontier_3);<br>
vdir.add_backend(frontier_4);<br>
}<br>
<br>
sub vcl_recv {<br>
set req.backend_hint = vdir.backend(); set req.http.X-frontier-id = "varnish";<br>
if (req.method != "GET" && req.method != "HEAD") {<br>
return (pipe);<br>
}<br>
}<br>
```<br>
<br>
_______________________________________________<br>
varnish-misc mailing list<br>
<a href="mailto:varnish-misc@varnish-cache.org" target="_blank" rel="noreferrer">varnish-misc@varnish-cache.org</a><br>
<a href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc" rel="noreferrer noreferrer" target="_blank">https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc</a><br>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</body>
</html>