<p dir="ltr">Thanks for reporting back, great job!</p>
<div class="gmail_extra"><br><div class="gmail_quote">On Dec 2, 2016 19:57, "Admin Beckspaced" <<a href="mailto:admin@beckspaced.com">admin@beckspaced.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
On 15.11.2016 17:57, Admin Beckspaced wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Am 15.11.2016 um 16:51 schrieb Dridi Boukelmoune:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
a bit more hints and info would be nice ;)<br>
</blockquote>
man vmod_std<br>
man varnishncsa<br>
<br>
That's how much "nice" I'm willing to do :p<br>
<br>
Dridi<br>
<br>
<br>
</blockquote>
ok. first I want to say thanks for being nice and pointing me to the man pages.<br>
<br>
after a bit of reading I finally found the parts I was looking for:<br>
<br>
import std;<br>
<br>
sub vcl_recv {<br>
<br>
std.log(“myhost:” + regsub(req.http.Host, "^www\.", "") );<br>
<br>
}<br>
<br>
in varnishncsa:<br>
<br>
# varnishncsa -F ‘%h %l %u %t “%r” %s %b “%{Referer}i” “%{User-agent}i %{VCL_Log:myhost}x’<br>
<br>
<br>
not yet tested but I think this is what Dridi was pointing to?<br>
<br>
</blockquote>
Hello again,<br>
<br>
sorry, it has been a while but I just thought to finish the thread I started and point to the solution I decided to go with at last.<br>
<br>
The question in the beginning was: How can I split the varnishncsa logs per domain.<br>
<br>
My first thinking was to use the query -q option, e.g. varnishncsa -q "ReqHeader ~ '^Host: .*<a href="http://example.com" rel="noreferrer" target="_blank">example.com</a>'"<br>
<br>
But this approach would end up in a lot varnishncsa instances, as pointed out by Andrei, and also the problem with not being able to normalize the Request Host header.<br>
Then Dridi pointed me to man vmod_std and using std.log in VCL, which was the final bit needed ;)<br>
<br>
So here's my current solution:<br>
<br>
I run a single instance of varnishncsa with the following params:<br>
<br>
VARNISHLOG_PARAMS="-f /etc/varnish/varnishncsa-log-f<wbr>ormat-string -a -w /var/log/varnish/varnish.log"<br>
<br>
the varnishncsa-log-format-string is as follows:<br>
<br>
%{VCL_Log:myhost}x %h %l %u %t "%r" %s %b "%{Referer}i" "%{User-agent}i"<br>
<br>
at the beginning VCL_Log:key The value set by std.log("key:value") in VCL, more on that later<br>
<br>
My varnish sits in front of Apache with 30 something different domains. Currently I don't use varnish for all domains, but have setup varnish VCL in such a way that I can filter on the domains and decide to cache with varnish or just skip caching and pass to the apache backend.<br>
<br>
My varnish VCL is based on the varnish boilerplate which I found on the net:<br>
<br>
<a href="http://verticalprogramming.com/2013/09/15/varnish-virtual-host-boilerplate" rel="noreferrer" target="_blank">http://verticalprogramming.com<wbr>/2013/09/15/varnish-virtual-<wbr>host-boilerplate</a><br>
<br>
So if I decide to cache a particular domain with varnish I can normalize the Request Host header in VCL<br>
<br>
sub vcl_recv {<br>
<br>
if (req.http.host ~ "somedomain\.com") {<br>
<br>
# tilde ~ uses regex<br>
if (req.http.host ~ "^(www.)?somedomain\.com$") {<br>
<br>
//normalize the req.http.host<br>
set req.http.host = regsub(req.http.Host, "^www\.", "");<br>
<br>
std.log("myhost:" + req.http.Host );<br>
...<br>
<br>
so this std.log(myhost:<a href="http://somedomian.com" rel="noreferrer" target="_blank">somedomian.com</a>) gets picked up by varnishncsa and the custom format string, see above.<br>
<br>
which then produces a nice & steady varnish.log file with a normalized host at the beginning for domains I want to cache and an empty space if I don't want to.<br>
<br>
then we got the split-logfile from apache:<br>
<br>
<a href="https://httpd.apache.org/docs/2.4/programs/split-logfile.html" rel="noreferrer" target="_blank">https://httpd.apache.org/docs/<wbr>2.4/programs/split-logfile.htm<wbr>l</a><br>
<br>
which was exactly made for a setup with the host names at the very beginning of the log file. Only thing someone needs to take care of is that the logfiles will get created in the directory where the script is run. so therefore I created a small bash script wrapper split-logfile.sh which first changes to the right working directory:<br>
<br>
#!/bin/bash<br>
cd /var/log/varnish<br>
/usr/bin/split-logfile < varnish.log<br>
<br>
and on the daily logrotate on the /var/log/varnish/varnish.log I added the following:<br>
<br>
/var/log/varnish/varnish.log {<br>
...<br>
prerotate<br>
/var/log/varnish/split-logfil<wbr>e.sh<br>
endscript<br>
...<br>
}<br>
<br>
so before the varnish.log gets rotated split-logfile.sh gets called and creates the different log files per normalized host and an access.log for all the requests without a hostname at the beginning.<br>
After a view logrotate runs the /var/log/varnish/ could look like that:<br>
<br>
mydomain.com.log<br>
myotherdomain.com.log<br>
access.log<br>
varnish.log<br>
varnish.log-20161130<br>
varnish.log-20161201<br>
varnish.log-20161202<br>
<br>
which finally give me exactly what I wanted! A single instance of varnishncsa producing a varnish.log for all domains.<br>
a per domain split via split-logfile.sh on each logrotate run resulting in log files per domain ready to use with webalizer, which gets also called on prerotate the logfile:<br>
<br>
/var/log/varnish/<a href="http://mydomain.com" target="_blank">mydomain.com</a>.<wbr>log {<br>
...<br>
prerotate<br>
/usr/bin/webalizer -qc /etc/webalizer/mydomain.conf<br>
endscript<br>
...<br>
}<br>
<br>
perhaps this might help someone here looking for something similar?<br>
<br>
thanks & greetings<br>
becki<br>
<br>
<br>
<br>
______________________________<wbr>_________________<br>
varnish-misc mailing list<br>
<a href="mailto:varnish-misc@varnish-cache.org" target="_blank">varnish-misc@varnish-cache.org</a><br>
<a href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc" rel="noreferrer" target="_blank">https://www.varnish-cache.org/<wbr>lists/mailman/listinfo/varnish<wbr>-misc</a></blockquote></div></div>