<div dir="ltr">Hi there, <div><br></div><div>Re-opening this thread from 2016!</div><div><br></div><div>A few days ago, I ran into an issue where varnish (varnish-6.0.0 revision a068361dff0d25a0d85cf82a6e5fdaf315e06a7d) became completely unresponsive.</div><div><br></div><div>While investigating, I noticed a lot of errors in varnishlog `VSL: store overflow`, but most importantly, I noticed our BAN list was starting to accumulate with somewhere around 130K BANs reported by varnishstat.</div><div><br></div><div><span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">The load on the website was pretty low. Restarting Varnish fixed the issue. </span>It is possible I had been running some script generating a lot of BANs, but it still seems weird it would generate so many BANs that varnish would become unresponsive.</div><div><br></div><div>After restarting Varnish, I watched `varnishstat -1 | grep bans` and could see MAIN.bans going up and down, but staying fairly stable on average.</div><div><span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div><span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Today, I decided to check again and I noticed:</span></div><div><br></div><div><span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><div style="text-decoration-style:initial;text-decoration-color:initial"><font face="monospace, monospace">MAIN.bans 66863 . Count of bans</font></div><div style="text-decoration-style:initial;text-decoration-color:initial"><font face="monospace, monospace">MAIN.bans_completed 66834 . Number of bans marked 'completed'</font></div><div style="text-decoration-style:initial;text-decoration-color:initial"><font face="monospace, monospace">MAIN.bans_obj 66849 . Number of bans using obj.*</font></div><div style="text-decoration-style:initial;text-decoration-color:initial"><font face="monospace, monospace">MAIN.bans_req 14 . Number of bans using req.*</font></div><div style="text-decoration-style:initial;text-decoration-color:initial"><font face="monospace, monospace">MAIN.bans_added 170939 1.18 Bans added</font></div><div style="text-decoration-style:initial;text-decoration-color:initial"><font face="monospace, monospace">MAIN.bans_deleted 104076 0.72 Bans deleted</font></div><div style="text-decoration-style:initial;text-decoration-color:initial"><font face="monospace, monospace">MAIN.bans_tested 3767852 26.08 Bans tested against objects (lookup)</font></div><div style="text-decoration-style:initial;text-decoration-color:initial"><font face="monospace, monospace">MAIN.bans_obj_killed 62024 0.43 Objects killed by bans (lookup)</font></div><div style="text-decoration-style:initial;text-decoration-color:initial"><font face="monospace, monospace">MAIN.bans_lurker_tested 3344073359 23148.62 Bans tested against objects (lurker)</font></div><div style="text-decoration-style:initial;text-decoration-color:initial"><font face="monospace, monospace">MAIN.bans_tests_tested 94711644 655.62 Ban tests tested against objects (lookup)</font></div><div style="text-decoration-style:initial;text-decoration-color:initial"><font face="monospace, monospace">MAIN.bans_lurker_tests_tested 4573910595 31661.91 Ban tests tested against objects (lurker)</font></div><div style="text-decoration-style:initial;text-decoration-color:initial"><font face="monospace, monospace">MAIN.bans_lurker_obj_killed 16627 0.12 Objects killed by bans (lurker)</font></div><div style="text-decoration-style:initial;text-decoration-color:initial"><font face="monospace, monospace">MAIN.bans_lurker_obj_killed_cutoff 0 0.00 Objects killed by bans for cutoff (lurker)</font></div><div style="text-decoration-style:initial;text-decoration-color:initial"><font face="monospace, monospace">MAIN.bans_dups 113845 0.79 Bans superseded by other bans</font></div><div style="text-decoration-style:initial;text-decoration-color:initial"><font face="monospace, monospace">MAIN.bans_lurker_contention 3 0.00 Lurker gave way for lookup</font></div><div style="text-decoration-style:initial;text-decoration-color:initial"><font face="monospace, monospace">MAIN.bans_persisted_bytes 163879330 . Bytes used by the persisted ban lists</font></div><div style="text-decoration-style:initial;text-decoration-color:initial"><font face="monospace, monospace">MAIN.bans_persisted_fragmentation 162795194 . Extra bytes in persisted ban lists due to fragmentation</font></div><br></span></div><div><span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div><span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">1- MAIN.bans is almost at 70k and varnish has been running for 1 day 16 hours</span></div><div><span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">2- MAIN.bans_req is > 0, but nowhere in my configuration do I ever submit bans using req.*</span></div><div><span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><font face="monospace, monospace"><br></font></span></div><div>So I have 3 questions: </div><div><br></div><div>- is 70k+ bans un-reasonable and should I find a way to reduce them?</div><div>- why am I seeing 14 MAIN.bans_req when my config doesn't ever emit such bans?</div><div><br></div><div>Thanks,</div><div>-Hugues</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Nov 3, 2016 at 3:04 AM, Dridi Boukelmoune <span dir="ltr"><<a href="mailto:dridi@varni.sh" target="_blank">dridi@varni.sh</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Thu, Oct 27, 2016 at 6:02 PM, Nils Goroll <<a href="mailto:slink@schokola.de">slink@schokola.de</a>> wrote:<br>
> Hi,<br>
><br>
> we've added a bunch of ban and ban lurker improvements which did not get<br>
> backported to 4.1.3<br>
><br>
> Please upgrade to 5.0 or master.<br>
<br>
</span>Hi Nils,<br>
<br>
I was thinking (against the too-many-knobs trend) that maybe we should<br>
have some sort of ban_queue_limit parameter to avoid piling up too<br>
many bans. It would also play nice with your std.ban() initiative<br>
where hitting the queue limit could be one of the reasons returned by<br>
the function when it fails.<br>
<br>
The default could be 0 for unlimited, or a fairly large number.<br>
<span class="HOEnZb"><font color="#888888"><br>
Dridi<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
______________________________<wbr>_________________<br>
varnish-misc mailing list<br>
<a href="mailto:varnish-misc@varnish-cache.org">varnish-misc@varnish-cache.org</a><br>
<a href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc" rel="noreferrer" target="_blank">https://www.varnish-cache.org/<wbr>lists/mailman/listinfo/<wbr>varnish-misc</a><br>
</div></div></blockquote></div><br></div>