<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hi Dridi,</div><div dir="ltr"><br><div>Do you need all timestamps or a specific metric? <font color="#073763"><br></font></div><div><font color="#073763">Regarding timestamp, want to read two tags,</font></div><div><font color="#073763">Timestamp Start: <b>1516269224.184112</b> 0.000000 0.000000<br></font></div><div><font color="#073763">Timestamp Resp: 1516269224.184920 <b>0.000808 0.000087</b></font><br></div><div><br></div><div>Do you need the BereqAcct records for all transactions? Including cache hits? <br></div><div><font color="#073763">Sorry it is my mistake. I am not reading any of the beck-end records. So can ignore BereqAcct.</font></div><div><font color="#073763">I need fields from Req records only.</font></div><div><font color="#073763"><br></font></div><div>What does FD mean here? File descriptor? From ReqStart? <br></div><div><font color="#0c343d">Yes, Its file descriptor. And yes reading from ReqStart till ReqAcct. Using switch case to read needed records.</font></div><div><font color="#0c343d">ex:</font></div><div><div><font color="#0c343d">switch (tag)</font></div><div><font color="#0c343d">{ </font></div></div><div><font color="#0c343d"> case SLT_ReqURL:<br></font></div><div><font color="#0c343d"><br></font></div><div><font color="#0c343d"><br></font></div><div><font color="#0c343d">Using below condition as I do not want to read Sess and Bereq records.( Please correct me if doing something wrong)</font></div><div><font color="#0c343d"><div> if (!VSL_Match(vsl, t->c) || (t->type == VSL_t_bereq)</div><div> || (t->type == VSL_t_sess)) </div></font></div><div><font color="#0c343d"> continue;</font></div><div><br></div><div><div>Well, utilities like varnishlog or varnishncsa accumulate transactions</div><div>via libvarnishapi in memory (which may take a long time) and then</div><div>libvarnishapi presents them in order. So utilities don't assume this</div><div>logic and simply get the data presented to them in a callback</div><div>function.</div></div><div><font color="#0c343d">Thanks for this..</font></div><div><font color="#0c343d">It means varnishlog (any utility) asks for logs which will come via libvarnishapi. Just trying to confirm that what i understood from above is correct. Here, memory means varnishd process memory or utility's memory ? As per my understadning its varnishd's memory because libvarnishapi will be holding logs to arrange. <br> In case of varnish 3, we were able to call direct dispatch function so it was faster, if my understanding is correct.</font></div><div><font color="#0c343d"><br></font></div><div><font color="#000000">Are log loss and missing of ReqAcct tag both problems are related ? </font></div><div><font color="#000000">I am thinking missing of ReqAcct is due to timeout problem also. For example mobile request are taking more time to serve.. In this case we might hit timeout of reading logs from VSL right ? I do not know in detail. But if buy modifying any timeout value can help please point out, I will try that sure.</font></div><div><font color="#000000"><br></font></div><div><font color="#000000">Thank you</font></div><div><font color="#000000">hardik </font></div><div><font color="#0c343d"><br></font></div></div></div></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, 19 Mar 2019 at 22:22, Dridi Boukelmoune <dridi@varni.sh> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
Thank you for the details, more questions to come.<br>
<br>
On Tue, Mar 19, 2019 at 1:42 AM Hardik <<a href="mailto:hetardik.p@gmail.com" target="_blank">hetardik.p@gmail.com</a>> wrote:<br>
><br>
> Hi Dridi,<br>
><br>
> Can you give me a list of log records you need to collect?<br>
><br>
> SLT_Timestamp :<br>
<br>
Do you need all timestamps or a specific metric?<br>
<br>
> SLT_ReqStart :<br>
> SLT_ReqMethod :<br>
> SLT_ReqURL:<br>
> SLT_ReqProtocol :<br>
> SLT_RespStatus :<br>
> SLT_ReqHeader :<br>
> SLT_RespHeader :<br>
> SLT_ReqAcct :<br>
> SLT_BereqAcct :<br>
<br>
Do you need the BereqAcct records for all transactions? Including cache hits?<br>
<br>
This one is tricky in terms of billing.<br>
<br>
> SLT_VCL_Log :<br>
><br>
> And<br>
> possibly how you are trying to group them if they come from different<br>
> transactions?<br>
<br>
You can do the grouping with the -g option, but that didn't go well<br>
for you so that's what I'm trying to figure out.<br>
<br>
> I am reading based xid ( by FD ). Means reading full records per fd.<br>
<br>
What does FD mean here? File descriptor? From ReqStart?<br>
<br>
> Please let me know if any other information I can provide..<br>
><br>
><br>
> If this is not related to my problem still I am curious to know how grouping is happening. You can point out some code or links with some details, I will go through.<br>
<br>
Well, utilities like varnishlog or varnishncsa accumulate transactions<br>
via libvarnishapi in memory (which may take a long time) and then<br>
libvarnishapi presents them in order. So utilities don't assume this<br>
logic and simply get the data presented to them in a callback<br>
function.<br>
<br>
That's where timeouts, overruns or transaction limits may result in<br>
data loss since slow log consumers don't slow down Varnish, and<br>
Varnish isn't slowed down by logs more than writing them to memory<br>
requires.<br>
<br>
<br>
<br>
Dridi<br>
</blockquote></div>