<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.im
{mso-style-name:im;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Here’s a typical varnishlog miss for a thumbnail image, appropriately sanitized. I can provide more if it helps<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><a href="https://gist.github.com/Calygos/ca7906da005569046a7031d1fcaa6372">https://gist.github.com/Calygos/ca7906da005569046a7031d1fcaa6372</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Guillaume Quintard [mailto:guillaume@varnish-software.com]
<br>
<b>Sent:</b> Tuesday, December 13, 2016 12:17 AM<br>
<b>To:</b> Justin Lloyd <justinl@arena.net><br>
<b>Cc:</b> Dridi Boukelmoune <dridi@varni.sh>; varnish-misc@varnish-cache.org<br>
<b>Subject:</b> Re: Hit ratio dropped significantly after recent upgrades<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Can you pastebin the req+bereq transactions in varnishlog, related to such a miss?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><br clear="all">
<o:p></o:p></p>
<div>
<div>
<div>
<div>
<p class="MsoNormal">-- <o:p></o:p></p>
</div>
<p class="MsoNormal">Guillaume Quintard<o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Tue, Dec 13, 2016 at 3:37 AM, Justin Lloyd <<a href="mailto:justinl@arena.net" target="_blank">justinl@arena.net</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal">To follow up on my last email from Friday, at this point the problem boils down to one thing that I've not been able to determine: Why are far fewer things being cached now than before the upgrade?<br>
<br>
1. Cookies don't seem to be the problem. Most appear to be Google Analytics (as opposed to session), which are being unset by vcl_recv.<br>
<br>
2. varnishlog/varnishtop shows many thumbnail URLs being missed and virtually none are requested with a no-cache cache-control header. Is it possible to use these tools determine if they (or any URLs for that matter) are being cached following a miss-deliver
sequence? There are about 1.5m thumbnail files totaling around 30 GB, which prior to the upgrades wasn't an issue, and I don't think it is now since there are only a few expires and purges per minute and no nukes at all. Varnish is only using about 2 GB out
of the 8 GB allocated to it, where it used to use all 8 GB and have lots of nukes and far fewer expires, so it's not a memory constraint.<br>
<br>
Could there be some other resource limitation I'm hitting without knowing it (nothing in any logs I've seen)? Everything else I could think of so far seems fine, e.g. open files, threads, tcp connections.<br>
<br>
<br>
<span class="im">-----Original Message-----</span><br>
<span class="im">From: varnish-misc-bounces+justinl=<a href="mailto:arena.net@varnish-cache.org">arena.net@varnish-cache.org</a> [mailto:<a href="mailto:varnish-misc-bounces%2Bjustinl">varnish-misc-bounces+justinl</a>=<a href="mailto:arena.net@varnish-cache.org">arena.net@varnish-cache.org</a>]
On Behalf Of Justin Lloyd</span><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal">Sent: Friday, December 9, 2016 11:19 AM<br>
To: Dridi Boukelmoune <<a href="mailto:dridi@varni.sh">dridi@varni.sh</a>><br>
Cc: <a href="mailto:varnish-misc@varnish-cache.org">varnish-misc@varnish-cache.org</a><br>
Subject: RE: Hit ratio dropped significantly after recent upgrades<br>
<br>
I really am looking at what's happening as well. I have been looking at both varnishlog and varnishtop and I see a lot of thumbnail image requests being sent to the backend when there is still plenty of room for them in the cache, so even though there are a
lot of thumbnail images, I shouldn't see so many backend requests for them. As I previously mentioned, I give Varnish 8 GB and it used to stay full (based on RSS usage and looking at nukes vs. expires) but now it hovers around only about 2 GB used. A related
statistics is that there used to be 600-700k objects in Varnish (based on our graphs of MAIN.n_object via Collectd's varnish-default-struct.objects-object metric) but now there are only roughly 40-70k objects in Varnish at any given time. So it's definitely
caching a lot fewer things than it was before the upgrade, and most of the requested URLs for requests that have cookies are for a lot of images and thumbnails. Images shouldn't be cached due to size and overall volume but thumbnails should, which is why I
strip cookies from the thumbnails. These varnishtop commands break out /images and /images/thumb client requests, showing IMHO too many regular images being cached and nowhere near enough thumbnails:<br>
<br>
# varnishtop -c -i VCL_call -q 'ReqURL ~ "/images/" and not ReqURL ~ "/images/thumb"'<br>
<br>
349.47 VCL_call HASH<br>
349.47 VCL_call RECV<br>
349.47 VCL_call DELIVER<br>
207.22 VCL_call HIT<br>
116.40 VCL_call MISS<br>
116.30 VCL_call PASS<br>
<br>
# varnishtop -c -i VCL_call -q 'ReqURL ~ "/images/thumb"'<br>
<br>
1859.60 VCL_call HASH<br>
1859.60 VCL_call RECV<br>
1859.60 VCL_call DELIVER<br>
1424.83 VCL_call MISS<br>
422.84 VCL_call HIT<br>
218.82 VCL_call PASS<br>
<br>
I'm still poking around trying to correlate caching of other types of URLs based on whether or not the requests have cookies, if Cache-Control gets returned, etc. but I just wanted to reply with this info. I do appreciate the responses I'm getting! :)<br>
<br>
<br>
-----Original Message-----<br>
From: Dridi Boukelmoune [mailto:<a href="mailto:dridi@varni.sh">dridi@varni.sh</a>]<br>
Sent: Friday, December 9, 2016 10:11 AM<br>
To: Justin Lloyd <<a href="mailto:justinl@arena.net">justinl@arena.net</a>><br>
Cc: Dag Haavi Finstad <<a href="mailto:daghf@varnish-software.com">daghf@varnish-software.com</a>>;
<a href="mailto:varnish-misc@varnish-cache.org">varnish-misc@varnish-cache.org</a><br>
Subject: Re: Hit ratio dropped significantly after recent upgrades<br>
<br>
> To reiterate on a point in another of my responses in this thread, I think it may be something about MediaWiki thumbnail images not being cached properly despite our current VCL in that regard not having changed from how it worked prior to the upgrade during
which time we were seeing a very high (86%-ish) hit ratio from the same formula.<br>
<br>
To reiterate on a point I made on a couple occasions, it's time to give varnishlog a spin. Too much focus on VCL, and not enough on what's happening.<br>
<br>
Dridi<br>
_______________________________________________<br>
varnish-misc mailing list<br>
<a href="mailto:varnish-misc@varnish-cache.org">varnish-misc@varnish-cache.org</a><br>
<a href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc" target="_blank">https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc</a><br>
<br>
_______________________________________________<br>
varnish-misc mailing list<br>
<a href="mailto:varnish-misc@varnish-cache.org">varnish-misc@varnish-cache.org</a><br>
<a href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc" target="_blank">https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc</a><o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>