From slink at schokola.de Fri Oct 5 11:15:13 2018 From: slink at schokola.de (Nils Goroll) Date: Fri, 5 Oct 2018 13:15:13 +0200 Subject: mini benchmark vtim formatting Message-ID: <853f9d84-9737-8a0c-e718-e73031302bbe@schokola.de> Hi, having been asked at VDD to do so, I have just committed a trivial micro-benchmark to vtim.c to get an idea about the overhead of our favorite %.6f format for timestamps. I have also changed vtim.c to represent timestamps as uint64_t with compile time adjustable base (see attached, only tested on Linux and Solaris) to get an idea about the impact of variations of the representation. I have used this to run tests on my Linux laptop (i7-8650U, debian 9.5) and our SmartOS vtest machine (E5645, joyent_20180412T025853Z). Sample runs at the bottom of this email. >From these simple tests, I'd draw the following conclusions: * when representing time as a double, formatting a timestamp is now much more expensive (7x - 20x) than retrieving the time * On Linux, formatting an integer can be ~7x more efficient than formatting a double - On Solaris, the difference is not that relevant * If we went for integers, formatting timestamps as microseconds using %06lu would probably be the best idea in place of the %.6f we use now. Getting a decimal point to the right place implies an overhead of ~50% * It seems that the time unit is not that relevant for these tests. So, for simplicity, we could go for nanoseconds since epoch, which will wrap 22nd Juli 2554 8th of microseconds would last until the year 75039, 16th of microseconds until the year 38503. For other operations, the difference may be relevant. If you are interested, please re-run the tests on your environment. * put attached files into lib/libvarnish and cd there * compile cc -o double -DTEST_DRIVER -I../.. -I../../include vtim.c vas.c -lm cc -o uint_nsec -DTEST_DRIVER -I../.. -I../../include vtim_uint64.c vas.c -lm cc -o uint_usec8 -DUSEC8 -DTEST_DRIVER -I../.. -I../../include vtim_uint64.c vas.c -lm * check if the test is working by calling the files double, uint_nsec, uint_usec8 multiple times * to just run the benchmark, simply add exit(0); after the call to bench(); Nils - Linux 64bit vtim.c bench() (using double): real: 0.002355s / 100000 = 23.550987ns - tst val 153872535444035.562500 mono: 0.002340s / 100000 = 23.398399ns - tst val 321355539.077222 3213.556566 printf: 0.052920s / 100000 = 529.201860ns - tst val 4600000.000000 real: 0.002430s / 100000 = 24.297237ns - tst val 153872613340620.625000 mono: 0.002403s / 100000 = 24.032593ns - tst val 399251852.306332 3992.519728 printf: 0.049563s / 100000 = 495.630440ns - tst val 4600000.000000 real: 0.002114s / 100000 = 21.140575ns - tst val 153872614232321.562500 mono: 0.002092s / 100000 = 20.923615ns - tst val 400143257.413292 4001.433621 printf: 0.042647s / 100000 = 426.469020ns - tst val 4600000.000000 real: 0.002372s / 100000 = 23.720264ns - tst val 153872614985513.750000 mono: 0.002320s / 100000 = 23.198128ns - tst val 400896625.035718 4008.967414 printf: 0.045916s / 100000 = 459.159040ns - tst val 4600000.000000 real: 0.002112s / 100000 = 21.121502ns - tst val 153872615984353.781250 mono: 0.002082s / 100000 = 20.818710ns - tst val 401895453.048254 4018.955575 printf: 0.047332s / 100000 = 473.323980ns - tst val 4600000.000000 SmartOS 64bit vtim.c bench() (using double): real: 0.004298s / 100000 = 42.984486ns - tst val 153873606800016.531250 mono: 0.004646s / 100000 = 46.463013ns - tst val 367368452718.765625 3673684.529523 printf: 0.029450s / 100000 = 294.502801ns - tst val 5400000.000000 real: 0.004205s / 100000 = 42.052269ns - tst val 153873606804744.531250 mono: 0.004667s / 100000 = 46.672821ns - tst val 367368456995.426270 3673684.572306 printf: 0.029131s / 100000 = 291.305990ns - tst val 5400000.000000 real: 0.004536s / 100000 = 45.359135ns - tst val 153873606809034.250000 mono: 0.004646s / 100000 = 46.455860ns - tst val 367368461336.529968 3673684.615702 printf: 0.030426s / 100000 = 304.255509ns - tst val 5400000.000000 real: 0.004203s / 100000 = 42.033195ns - tst val 153873606813494.593750 mono: 0.004645s / 100000 = 46.453476ns - tst val 367368465704.614807 3673684.659382 printf: 0.030404s / 100000 = 304.044960ns - tst val 5400000.000000 real: 0.004316s / 100000 = 43.158531ns - tst val 153873606817572.531250 mono: 0.005237s / 100000 = 52.371025ns - tst val 367368470172.232300 3673684.704187 printf: 0.030035s / 100000 = 300.351870ns - tst val 5400000.000000 Linux 64bit vtim_uint64.c 1/8th microseconds // -DUSEC8 1 vt is 125 nsec real: 5622500ns / 100000 = 56ns - tst val 13497331302862915168 mono: 4339000ns / 100000 = 43ns - tst val 4727458592442259 %lu.%06lu: 19220125ns / 100000 = 192ns - tst val 4600000 5909.325416 %lu.%06lu diff: 13590500ns / 100000 = 135ns - tst val 4600000 5909.344645 %06lu: 8187375ns / 100000 = 81ns - tst val 5100000 5909358251 2printf: 16292125ns / 100000 = 162ns - tst val 4600000 5909.366441 1 vt is 125 nsec real: 2274000ns / 100000 = 22ns - tst val 13497331355470056350 mono: 2251625ns / 100000 = 22ns - tst val 4727509533282433 %lu.%06lu: 9097500ns / 100000 = 90ns - tst val 4600000 5909.388035 %lu.%06lu diff: 9202125ns / 100000 = 92ns - tst val 4600000 5909.397135 %06lu: 6447375ns / 100000 = 64ns - tst val 5200000 5909406344 2printf: 13142625ns / 100000 = 131ns - tst val 4600000 5909.412795 1 vt is 125 nsec real: 1907500ns / 100000 = 19ns - tst val 13497331389852240279 mono: 1877000ns / 100000 = 18ns - tst val 4727543608271599 %lu.%06lu: 8324750ns / 100000 = 83ns - tst val 4600000 5909.430450 %lu.%06lu diff: 8124625ns / 100000 = 81ns - tst val 4600000 5909.438780 %06lu: 6225125ns / 100000 = 62ns - tst val 5200000 5909446908 2printf: 13080375ns / 100000 = 130ns - tst val 4600000 5909.453136 1 vt is 125 nsec real: 1959875ns / 100000 = 19ns - tst val 13497331422083304059 mono: 1884875ns / 100000 = 18ns - tst val 4727575868482680 %lu.%06lu: 8308375ns / 100000 = 83ns - tst val 4600000 5909.470781 %lu.%06lu diff: 8274500ns / 100000 = 82ns - tst val 4600000 5909.479092 %06lu: 6325750ns / 100000 = 63ns - tst val 5200000 5909487370 2printf: 13769875ns / 100000 = 137ns - tst val 4600000 5909.493698 1 vt is 125 nsec real: 1937500ns / 100000 = 19ns - tst val 13497331455138718337 mono: 1867125ns / 100000 = 18ns - tst val 4727608894328593 %lu.%06lu: 8296875ns / 100000 = 82ns - tst val 4600000 5909.512054 %lu.%06lu diff: 8097250ns / 100000 = 80ns - tst val 4600000 5909.520355 %06lu: 6224000ns / 100000 = 62ns - tst val 5300000 5909528454 2printf: 13066250ns / 100000 = 130ns - tst val 4600000 5909.534684 SmartOS 64bit vtim_uint64.c 1/8th microseconds // -DUSEC8 1 vt is 125 nsec real: 6644625ns / 100000 = 66ns - tst val 13504082175283590570 mono: 6616875ns / 100000 = 66ns - tst val 2939284261243902679 %lu.%06lu: 42185250ns / 100000 = 421ns - tst val 4900000 3674105.329936 %lu.%06lu diff: 34464375ns / 100000 = 344ns - tst val 4900000 3674105.372162 %06lu: 23622875ns / 100000 = 236ns - tst val 4900000 3674105406659 2printf: 75176125ns / 100000 = 751ns - tst val 4900000 3674105.430309 1 vt is 125 nsec real: 5383500ns / 100000 = 53ns - tst val 13504082332109593912 mono: 6566000ns / 100000 = 65ns - tst val 2939284417340253304 %lu.%06lu: 32064000ns / 100000 = 320ns - tst val 4900000 3674105.525042 %lu.%06lu diff: 37663625ns / 100000 = 376ns - tst val 4900000 3674105.557138 %06lu: 22300375ns / 100000 = 223ns - tst val 4900000 3674105594834 2printf: 59873000ns / 100000 = 598ns - tst val 4900000 3674105.617163 1 vt is 125 nsec real: 7150000ns / 100000 = 71ns - tst val 13504082468470874911 mono: 6997000ns / 100000 = 69ns - tst val 2939284554928560611 %lu.%06lu: 32293375ns / 100000 = 322ns - tst val 4900000 3674105.697104 %lu.%06lu diff: 34067500ns / 100000 = 340ns - tst val 4900000 3674105.729432 %06lu: 31384000ns / 100000 = 313ns - tst val 4900000 3674105763544 2printf: 73468000ns / 100000 = 734ns - tst val 4900000 3674105.794971 1 vt is 125 nsec real: 6757375ns / 100000 = 67ns - tst val 13504082621801712857 mono: 6683125ns / 100000 = 66ns - tst val 2939284707852278262 %lu.%06lu: 37656125ns / 100000 = 376ns - tst val 4900000 3674105.888192 %lu.%06lu diff: 37080375ns / 100000 = 370ns - tst val 4900000 3674105.925882 %06lu: 27612500ns / 100000 = 276ns - tst val 4900000 3674105962996 2printf: 83942000ns / 100000 = 839ns - tst val 4900000 3674105.990650 1 vt is 125 nsec real: 4760125ns / 100000 = 47ns - tst val 13504082785812051427 mono: 4868625ns / 100000 = 48ns - tst val 2939284870314396946 %lu.%06lu: 27153875ns / 100000 = 271ns - tst val 4900000 3674106.090366 %lu.%06lu diff: 29272750ns / 100000 = 292ns - tst val 4900000 3674106.117556 %06lu: 21217625ns / 100000 = 212ns - tst val 4900000 3674106146859 2printf: 59721875ns / 100000 = 597ns - tst val 4900000 3674106.168103 Linux 64bit vtim_uint64.c nanoseconds 1 vt is 1 nsec real: 6471658ns / 100000 = 64ns - tst val 8522114927877580954 mono: 3506494ns / 100000 = 35ns - tst val 600345064167526557 %lu.%06lu: 15612387ns / 100000 = 156ns - tst val 4600000 6003.452480 %lu.%06lu diff: 13278403ns / 100000 = 132ns - tst val 4600000 6003.468104 %06lu: 8211335ns / 100000 = 82ns - tst val 5200000 6003481394 2printf: 16426218ns / 100000 = 164ns - tst val 4600000 6003.489609 1 vt is 1 nsec real: 2177350ns / 100000 = 21ns - tst val 8522121055739749055 mono: 2115589ns / 100000 = 21ns - tst val 600351007284215988 %lu.%06lu: 9808669ns / 100000 = 98ns - tst val 4600000 6003.511130 %lu.%06lu diff: 8982234ns / 100000 = 89ns - tst val 4600000 6003.520941 %06lu: 6887452ns / 100000 = 68ns - tst val 5300000 6003529926 2printf: 13647040ns / 100000 = 136ns - tst val 4600000 6003.536821 1 vt is 1 nsec real: 1823293ns / 100000 = 18ns - tst val 8522125472179529458 mono: 1786902ns / 100000 = 17ns - tst val 600355389907576048 %lu.%06lu: 8402054ns / 100000 = 84ns - tst val 4600000 6003.554793 %lu.%06lu diff: 8520702ns / 100000 = 85ns - tst val 4600000 6003.563198 %06lu: 6249130ns / 100000 = 62ns - tst val 5300000 6003571724 2printf: 13035186ns / 100000 = 130ns - tst val 4600000 6003.577975 1 vt is 1 nsec real: 1804644ns / 100000 = 18ns - tst val 8522129526335546761 mono: 1786300ns / 100000 = 17ns - tst val 600359443389283367 %lu.%06lu: 8477539ns / 100000 = 84ns - tst val 4600000 6003.595328 %lu.%06lu diff: 8267460ns / 100000 = 82ns - tst val 4600000 6003.603811 %06lu: 6200255ns / 100000 = 62ns - tst val 5400000 6003612081 2printf: 13069690ns / 100000 = 130ns - tst val 4600000 6003.618286 1 vt is 1 nsec real: 1839434ns / 100000 = 18ns - tst val 8522133565115078006 mono: 1788811ns / 100000 = 17ns - tst val 600363482945763234 %lu.%06lu: 8347177ns / 100000 = 83ns - tst val 4600000 6003.635724 %lu.%06lu diff: 8177486ns / 100000 = 81ns - tst val 4600000 6003.644078 %06lu: 6274116ns / 100000 = 62ns - tst val 5400000 6003652258 2printf: 13572265ns / 100000 = 135ns - tst val 4600000 6003.658536 SmartOS 64bit vtim_uint64.c nanoseconds 1 vt is 1 nsec real: 4001319ns / 100000 = 40ns - tst val 9341692761795774739 mono: 3990058ns / 100000 = 39ns - tst val 16907526543020455981 %lu.%06lu: 25480662ns / 100000 = 254ns - tst val 5700000 3673956.641440 %lu.%06lu diff: 24836733ns / 100000 = 248ns - tst val 5700000 3673956.666945 %06lu: 20819684ns / 100000 = 208ns - tst val 5700000 3673956691809 2printf: 56805505ns / 100000 = 568ns - tst val 5700000 3673956.712655 1 vt is 1 nsec real: 4357576ns / 100000 = 43ns - tst val 9341706878688867866 mono: 4157573ns / 100000 = 41ns - tst val 16907540709171765817 %lu.%06lu: 31060114ns / 100000 = 310ns - tst val 5700000 3673956.783126 %lu.%06lu diff: 26465987ns / 100000 = 264ns - tst val 5700000 3673956.814214 %06lu: 22196569ns / 100000 = 221ns - tst val 5700000 3673956840710 2printf: 52591106ns / 100000 = 525ns - tst val 5700000 3673956.862945 1 vt is 1 nsec real: 4990835ns / 100000 = 49ns - tst val 9341721592996888443 mono: 3985929ns / 100000 = 39ns - tst val 16907555415553040934 %lu.%06lu: 30840218ns / 100000 = 308ns - tst val 5700000 3673956.930165 %lu.%06lu diff: 29306453ns / 100000 = 293ns - tst val 5700000 3673956.961041 %06lu: 24574870ns / 100000 = 245ns - tst val 5700000 3673956990383 2printf: 55021411ns / 100000 = 550ns - tst val 5700000 3673957.014986 1 vt is 1 nsec real: 5964321ns / 100000 = 59ns - tst val 9341737111204615366 mono: 5672596ns / 100000 = 56ns - tst val 16907571078315471014 %lu.%06lu: 37606858ns / 100000 = 376ns - tst val 5700000 3673957.087631 %lu.%06lu diff: 36609663ns / 100000 = 366ns - tst val 5700000 3673957.125273 %06lu: 27434026ns / 100000 = 274ns - tst val 5700000 3673957161923 2printf: 68533158ns / 100000 = 685ns - tst val 5700000 3673957.189391 1 vt is 1 nsec real: 3973381ns / 100000 = 39ns - tst val 9341755714929160410 mono: 3930117ns / 100000 = 39ns - tst val 16907589491415809313 %lu.%06lu: 25384065ns / 100000 = 253ns - tst val 5700000 3673957.270895 %lu.%06lu diff: 25737191ns / 100000 = 257ns - tst val 5700000 3673957.296306 %06lu: 20015972ns / 100000 = 200ns - tst val 5700000 3673957322071 2printf: 60235584ns / 100000 = 602ns - tst val 5700000 3673957.342119 -------------- next part -------------- A non-text attachment was scrubbed... Name: vtim_uint64.h Type: text/x-chdr Size: 1783 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: vtim_uint64.c Type: text/x-csrc Size: 15692 bytes Desc: not available URL: From phk at phk.freebsd.dk Sun Oct 7 05:19:26 2018 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sun, 07 Oct 2018 05:19:26 +0000 Subject: mini benchmark vtim formatting In-Reply-To: <853f9d84-9737-8a0c-e718-e73031302bbe@schokola.de> References: <853f9d84-9737-8a0c-e718-e73031302bbe@schokola.de> Message-ID: <7884.1538889566@critter.freebsd.dk> -------- In message <853f9d84-9737-8a0c-e718-e73031302bbe at schokola.de>, Nils Goroll writ es: >* when representing time as a double, formatting a timestamp is now much more > expensive (7x - 20x) than retrieving the time So the interesting question for me now, is if we can shave if we stick with double and simply by tweaking the formatting. Something like: printf("%ju.%06ju", (uint64_t)floor(foo), (uint64_t)floor((foo*1e6)) % 1000000UL); (Obviously to be hidden with PF-like macro) -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From slink at schokola.de Mon Oct 8 08:41:07 2018 From: slink at schokola.de (Nils Goroll) Date: Mon, 8 Oct 2018 10:41:07 +0200 Subject: mini benchmark vtim formatting In-Reply-To: <7884.1538889566@critter.freebsd.dk> References: <853f9d84-9737-8a0c-e718-e73031302bbe@schokola.de> <7884.1538889566@critter.freebsd.dk> Message-ID: <6e3db4b7-5817-4c60-8c20-f4dce1ebf9d1@schokola.de> > printf("%ju.%06ju", > (uint64_t)floor(foo), > (uint64_t)floor((foo*1e6)) % 1000000UL); nice trick. On Linux, this gets us down to the same order of magnitude as a %lu.%06lu with uint64_t On SmartOS the gain is not that significant, but we knew this before from the uint64 test So now the question is how this looks for other platforms. @All, Please send your results! - Linux 64bit: printf %.6f: 0.051814s / 100000 = 518.138460ns - tst val 4600000.000000 1407.275722 printf %ju.%06ju: 0.008429s / 100000 = 84.292610ns - tst val 4600000.000000 1407.327545 printf %.6f: 0.044428s / 100000 = 444.276420ns - tst val 4600000.000000 1407.341059 printf %ju.%06ju: 0.008945s / 100000 = 89.451130ns - tst val 4600000.000000 1407.385495 printf %.6f: 0.046260s / 100000 = 462.603330ns - tst val 4600000.000000 1407.399393 printf %ju.%06ju: 0.008586s / 100000 = 85.857160ns - tst val 4600000.000000 1407.445667 printf %.6f: 0.046373s / 100000 = 463.734350ns - tst val 4600000.000000 1407.459268 printf %ju.%06ju: 0.008428s / 100000 = 84.282410ns - tst val 4600000.000000 1407.505651 printf %.6f: 0.046515s / 100000 = 465.154660ns - tst val 4600000.000000 1407.519289 printf %ju.%06ju: 0.008381s / 100000 = 83.813980ns - tst val 4600000.000000 1407.565813 - SmartOS 64bit: printf %.6f: 0.054789s / 100000 = 547.888801ns - tst val 5400000.000000 3925617.429178 printf %ju.%06ju: 0.038115s / 100000 = 381.154837ns - tst val 5400000.000000 3925617.484005 printf %.6f: 0.048295s / 100000 = 482.946290ns - tst val 5400000.000000 3925617.541356 printf %ju.%06ju: 0.035813s / 100000 = 358.133619ns - tst val 5400000.000000 3925617.589702 printf %.6f: 0.050126s / 100000 = 501.261158ns - tst val 5400000.000000 3925617.644167 printf %ju.%06ju: 0.037617s / 100000 = 376.169202ns - tst val 5400000.000000 3925617.694339 printf %.6f: 0.051205s / 100000 = 512.047550ns - tst val 5400000.000000 3925617.750960 printf %ju.%06ju: 0.037772s / 100000 = 377.719700ns - tst val 5400000.000000 3925617.802201 printf %.6f: 0.051990s / 100000 = 519.904410ns - tst val 5400000.000000 3925617.859368 printf %ju.%06ju: 0.038027s / 100000 = 380.265811ns - tst val 5400000.000000 3925617.911402 From phk at phk.freebsd.dk Mon Oct 8 09:07:47 2018 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 08 Oct 2018 09:07:47 +0000 Subject: mini benchmark vtim formatting In-Reply-To: <6e3db4b7-5817-4c60-8c20-f4dce1ebf9d1@schokola.de> References: <853f9d84-9737-8a0c-e718-e73031302bbe@schokola.de> <7884.1538889566@critter.freebsd.dk> <6e3db4b7-5817-4c60-8c20-f4dce1ebf9d1@schokola.de> Message-ID: <24341.1538989667@critter.freebsd.dk> -------- In message <6e3db4b7-5817-4c60-8c20-f4dce1ebf9d1 at schokola.de>, Nils Goroll writes: >> printf("%ju.%06ju", >> (uint64_t)floor(foo), >> (uint64_t)floor((foo*1e6)) % 1000000UL); > >nice trick. On Linux, this gets us down to the same order of magnitude as a >%lu.%06lu with uint64_t The important thing here is the FP rendering, not the FP math, which is instantaneous for all practical purposes. It's kind of surprising to me that "%.f" hasn't gotten specific optimization, it is a lot simpler thing than the general FP formatting problem, but if there is any code I'm not keen on wading into, it is FP formatting code... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From dridi at varni.sh Mon Oct 8 11:51:38 2018 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 8 Oct 2018 13:51:38 +0200 Subject: mini benchmark vtim formatting In-Reply-To: <6e3db4b7-5817-4c60-8c20-f4dce1ebf9d1@schokola.de> References: <853f9d84-9737-8a0c-e718-e73031302bbe@schokola.de> <7884.1538889566@critter.freebsd.dk> <6e3db4b7-5817-4c60-8c20-f4dce1ebf9d1@schokola.de> Message-ID: > So now the question is how this looks for other platforms. @All, Please send > your results! $ ./double real: 0.004972s / 100000 = 49.715042ns - tst val 153899435413393.562500 mono: 0.003554s / 100000 = 35.541058ns - tst val 1075376231.264834 printf %.6f: 0.059387s / 100000 = 593.865160ns - tst val 5100000.000000 10753.764099 printf %ju.%06ju: 0.010162s / 100000 = 101.621330ns - tst val 5100000.000000 10753.823494 $ ./uint_nsec 1 vt is 1 nsec real: 5054881ns / 100000 = 50ns - tst val 16651824334676002549 mono: 2728750ns / 100000 = 27ns - tst val 1030850150182733345 %lu.%06lu: 11545090ns / 100000 = 115ns - tst val 5600000 10308.502862 %lu.%06lu diff: 10600594ns / 100000 = 106ns - tst val 5600000 10308.514411 %06lu: 7568966ns / 100000 = 75ns - tst val 5600000 10308525016 2printf: 15149373ns / 100000 = 151ns - tst val 5600000 10308.532590 $ ./uint_usec8 1 vt is 125 nsec real: 5313625ns / 100000 = 53ns - tst val 13710030185021002820 mono: 3063625ns / 100000 = 30ns - tst val 8258753524632712 %lu.%06lu: 12164125ns / 100000 = 121ns - tst val 5100000 10323.443427 %lu.%06lu diff: 10665000ns / 100000 = 106ns - tst val 5100000 10323.455602 %06lu: 7729250ns / 100000 = 77ns - tst val 5100000 10323466276 2printf: 15788875ns / 100000 = 157ns - tst val 5100000 10323.474012 $ uname -a Linux dridi 4.18.11-200.fc28.x86_64 #1 SMP Sun Sep 30 15:31:40 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux Dridi From fgsch at lodoss.net Mon Oct 8 16:55:41 2018 From: fgsch at lodoss.net (Federico Schwindt) Date: Mon, 8 Oct 2018 17:55:41 +0100 Subject: next VDD Message-ID: Hey, It might be a bit earlier but personally knowing the where and when for the next VDD will help immensely to sort the time off at work and whatnot, especially the date. Do we have any ideas or suggestions? Best. -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Mon Oct 8 17:19:36 2018 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Mon, 08 Oct 2018 17:19:36 +0000 Subject: next VDD In-Reply-To: References: Message-ID: <26050.1539019176@critter.freebsd.dk> -------- In message , Federico Schwindt writes: >It might be a bit earlier but personally knowing the where and when for the >next VDD will help immensely to sort the time off at work and whatnot, >especially the date. > >Do we have any ideas or suggestions? I've been wanting to visit London before brexit, and a VDD could be an excuse to do so. Anywhere nort-europe-ish is fine with me. Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From geoff at uplex.de Mon Oct 8 17:31:29 2018 From: geoff at uplex.de (Geoff Simmons) Date: Mon, 8 Oct 2018 19:31:29 +0200 Subject: next VDD In-Reply-To: References: Message-ID: On 10/8/18 18:55, Federico Schwindt wrote: > > It might be a bit earlier but personally knowing the where and when for the > next VDD will help immensely to sort the time off at work and whatnot, > especially the date. > > Do we have any ideas or suggestions? I thought that the time frame "shortly after release" the last time around worked very well. We had just cleared the table, ready to think about what comes next. And ambiguities about current work that may or may not make it into an upcoming release were not a problem, since it just happened. The only problem I see is that it would mean second half of March or early April, which as I understood from today's IRC conversations doesn't work well for you at all. Best, Geoff -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From fgsch at lodoss.net Tue Oct 9 09:35:46 2018 From: fgsch at lodoss.net (Federico Schwindt) Date: Tue, 9 Oct 2018 10:35:46 +0100 Subject: next VDD In-Reply-To: References: Message-ID: It certainly will be tricky for me to attend around those dates. If it's local, however, the chances would be higher but it's outside my control :-) On Mon, Oct 8, 2018 at 6:31 PM Geoff Simmons wrote: > On 10/8/18 18:55, Federico Schwindt wrote: > > > > It might be a bit earlier but personally knowing the where and when for > the > > next VDD will help immensely to sort the time off at work and whatnot, > > especially the date. > > > > Do we have any ideas or suggestions? > > I thought that the time frame "shortly after release" the last time > around worked very well. We had just cleared the table, ready to think > about what comes next. And ambiguities about current work that may or > may not make it into an upcoming release were not a problem, since it > just happened. > > The only problem I see is that it would mean second half of March or > early April, which as I understood from today's IRC conversations > doesn't work well for you at all. > > > Best, > Geoff > -- > ** * * UPLEX - Nils Goroll Systemoptimierung > > Scheffelstra?e 32 > 22301 Hamburg > > Tel +49 40 2880 5731 > Mob +49 176 636 90917 > Fax +49 40 42949753 > > http://uplex.de > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From geoff at uplex.de Tue Oct 9 15:12:20 2018 From: geoff at uplex.de (Geoff Simmons) Date: Tue, 9 Oct 2018 17:12:20 +0200 Subject: style(9), dir-locals.el and emacs-foo Message-ID: <16b46047-10bf-af78-6cd6-a7bc75088ae1@uplex.de> Hello all, I'm writing in the hope that someone will be better versed in emacs-lisp than I am. For a while we've had a very simple .dir-locals.el at the root of the project directory, which goes a long way towards automating fulfillment of style(9) for Emacs users, but doesn't quite make it. ((nil . ((indent-tabs-mode . t))) (c-mode . ((c-file-style . "BSD")))) I've only noticed one style(9) error that I and other Emacs users have made consistently -- indentation of a list of function parameters (both declaration and invocation) after the line with the open paren. Emacs wants to do this: myfunction(foo, bar, baz, quux) It lines up the parameter on the second line in the column after the open paren. style(9) wants it this way: myfunction(foo, bar, baz, quux) As I understand it, the second line should be indented four spaces (in addition to the current indent): "Long parameter lists are wrapped with a normal four space indent." I suspect that this should be easy enough to automate with a modification of .dir-locals.el, but my elisp-foo isn't good enough and I'm struggling with it. It seems to me that we want to set arglist-cont-nonempty to 4: https://www.gnu.org/software/emacs/manual/html_node/ccmode/Paren-List-Symbols.html So I tried changing .dir-locals.el to: ((nil . ((indent-tabs-mode . t))) (c-mode . ((c-file-style . "BSD") (c-file-offsets (arglist-cont-nonempty . 4))))) ... which sets the offset as a file local variable. But if we have that, then at least the first time you open a .c or .h file, emacs pops up a warning that the setting of c-file-offsets may not be safe, and asks for confirmation: https://www.gnu.org/software/emacs/manual/html_node/emacs/Safe-File-Variables.html I don't think it's tolerable to have to do that every time for the rest of our lives while working on Varnish. I have tried declaring 4 as a safe value for the variable in .dir-locals.el as follows, but I still get the warning: ((nil . ((indent-tabs-mode . t) (eval . (add-to-list 'safe-local-variable-values '(arglist-cont-nonempty . 4))))) (c-mode . ((c-file-style . "BSD") (c-file-offsets (arglist-cont-nonempty . 4))))) This is getting beyond my abilities, I'm not an elisp programmer. And the failed attempt is already getting too complicated for my comfort level -- just changing one offset feels like it should be a simple problem to solve, say one extra line of code. So I suspect that I'm going at it the wrong way. Is there someone out there with a better mastery of elisp, who knows the right magic spell? Thanks, Geoff -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From phk at phk.freebsd.dk Tue Oct 9 15:34:28 2018 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 09 Oct 2018 15:34:28 +0000 Subject: style(9), dir-locals.el and emacs-foo In-Reply-To: <16b46047-10bf-af78-6cd6-a7bc75088ae1@uplex.de> References: <16b46047-10bf-af78-6cd6-a7bc75088ae1@uplex.de> Message-ID: <76965.1539099268@critter.freebsd.dk> -------- In message <16b46047-10bf-af78-6cd6-a7bc75088ae1 at uplex.de>, Geoff Simmons writes: >I'm writing in the hope that someone will be better versed in emacs-lisp >than I am. My best suggestion is to look here: https://www.freebsd.org/doc/en/books/developers-handbook/emacs.html -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From russellsimpkins at gmail.com Tue Oct 9 15:53:41 2018 From: russellsimpkins at gmail.com (Russell Simpkins) Date: Tue, 9 Oct 2018 11:53:41 -0400 Subject: style(9), dir-locals.el and emacs-foo In-Reply-To: <76965.1539099268@critter.freebsd.dk> References: <16b46047-10bf-af78-6cd6-a7bc75088ae1@uplex.de> <76965.1539099268@critter.freebsd.dk> Message-ID: Try this github - https://github.com/hogand/openbsd-knf-emacs On Tue, Oct 9, 2018 at 11:34 AM Poul-Henning Kamp wrote: > -------- > In message <16b46047-10bf-af78-6cd6-a7bc75088ae1 at uplex.de>, Geoff Simmons > writes: > > >I'm writing in the hope that someone will be better versed in emacs-lisp > >than I am. > > My best suggestion is to look here: > > > https://www.freebsd.org/doc/en/books/developers-handbook/emacs.html > > -- > Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 > phk at FreeBSD.ORG | TCP/IP since RFC 956 > FreeBSD committer | BSD since 4.3-tahoe > Never attribute to malice what can adequately be explained by incompetence. > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From zartzurter at gmail.com Thu Oct 25 08:56:54 2018 From: zartzurter at gmail.com (zart zurt) Date: Thu, 25 Oct 2018 11:56:54 +0300 Subject: BackGroundFetch problem ? Message-ID: Hi, I recently adopted some varnish servers and upgraded them to varnish 5.0.0 in debian-9 from the debian repositories. Most metrics are good except the backend_req value which is too high. The default_ttl and TTL values set in the VCL code are all 10d (or equivalent in seconds) When I checked the docs, I found out that in vcl_hit even if you return deliver, varnish starts a background fetch for that particular object if the object time has passed its TTL. In my case, BGF is always started. I checked the source code and in file varnisgd/cache/cache_req_fsm.c and function cnt_lookup around line 540, after VCL_hit_method called and it returned VCL_RET_DELIVER, a BGF is started without looking at the TTL value. How can I change this behaviour in VCL or through a parameter ? Or is this a bug ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Thu Oct 25 20:18:57 2018 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Thu, 25 Oct 2018 13:18:57 -0700 Subject: BackGroundFetch problem ? In-Reply-To: References: Message-ID: Hi, can you share your vcl? -- Guillaume Quintard On Thu, Oct 25, 2018 at 1:56 AM, zart zurt wrote: > Hi, I recently adopted some varnish servers and upgraded them to varnish > 5.0.0 in debian-9 from the debian repositories. > > Most metrics are good except the backend_req value which is too high. The > default_ttl and TTL values set in the VCL code are all 10d (or equivalent > in seconds) > > When I checked the docs, I found out that in vcl_hit even if you return > deliver, varnish starts a background fetch for that particular object if > the object time has passed its TTL. In my case, BGF is always started. > > I checked the source code and in file varnisgd/cache/cache_req_fsm.c and > function cnt_lookup around line 540, after VCL_hit_method called and it > returned VCL_RET_DELIVER, a BGF is started without looking at the TTL value. > > How can I change this behaviour in VCL or through a parameter ? Or is this > a bug ? > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From slink at schokola.de Sat Oct 27 15:41:08 2018 From: slink at schokola.de (Nils Goroll) Date: Sat, 27 Oct 2018 17:41:08 +0200 Subject: vcl variables reinvented as objects Message-ID: Hi, I did not announce new vmods in the past and do not intend to in the future, but in this particular case I do want to give a quick heads up regarding https://code.uplex.de/uplex-varnish/varnish-objvar because I think (and hope) it could generally be useful across the board. Comments/reviews welcome. Nils From slink at schokola.de Tue Oct 30 13:14:50 2018 From: slink at schokola.de (Nils Goroll) Date: Tue, 30 Oct 2018 14:14:50 +0100 Subject: req->bereq In-Reply-To: References: Message-ID: <8966100a-2af4-a5ac-ed4b-87b286ce4e3a@schokola.de> Hi, Somewhat related to the objvar vmods, I've been pondering how best get PRIV_TASK based custom state across the req/bereq boundary. How about a sub vcl_backend_init {} with read-access to req.* and write-access to bereq.* ? This would be called when we prepare the bereq before continuing with the req. For vmods, there could be a VRT_priv_task_req variant which allows read-only access to the req PRIV_TASK state. With such an interface, vmods could implement methods to copy over objects from the client side, for example vcl_init { new foo = taskvar.int(); # exists } sub vcl_recv { foo.set(42); # exists } sub vcl_backend_init { foo.inherit(); # new } sub vcl_backend.fetch { set bereq.http.foo = foo.get(); # exists, should give 42 } Nils From phk at critter.freebsd.dk Tue Oct 30 07:37:57 2018 From: phk at critter.freebsd.dk (Poul-Henning Kamp) Date: Tue, 30 Oct 2018 07:37:57 +0000 Subject: Fwd: Identifying our deliverables (naming QUIC's HTTP binding) (fwd) Message-ID: <46964.1540885077@critter.freebsd.dk> As expected: QUIC == HTTP/3 Start reading up... Poul-Henning ------- Forwarded Message From: Mark Nottingham Date: Tue, 30 Oct 2018 11:18:36 +1100 To: HTTP Working Group Message-Id: <67D04361-CE05-4DCC-AA99-3DA259BBF326 at mnot.net> Everyone, FYI, this might come to the WG in Bangkok. Cheers, > Begin forwarded message: > > From: Mark Nottingham > Subject: Identifying our deliverables > Date: 28 October 2018 at 11:30:55 am AEDT > To: IETF QUIC WG > Archived-At: > > You might recall that we've previously talked about renaming our deliverables to clarify their relationship to the input documents (sometimes referred to as "gQUIC vs iQUIC"). > > After a fair amount of discussion with various folks, it seems like there's confidence that calling our core deliverable just "QUIC" will work, since Google's variant will fade out of use if we succeed, and there aren't any good alternatives; our best candidate was "QUIC/2", but it was pointed out that this additional level of versioning (since we already have a wire version) will cause yet more confusion. > > However, in those discussions, a related concern was identified; confusion between QUIC-the-transport-protocol, and QUIC-the-HTTP-binding. I and others have seen a number of folks not closely involved in this work conflating the two, even though they're now separate things. > > To address this, I'd like to suggest that -- after coordination with the HTTP WG -- we rename our the HTTP document to "HTTP/3", and using the final ALPN token "h3". Doing so clearly identifies it as another binding of HTTP semantics to the wire protocol -- just as HTTP/2 did -- so people understand its separation from QUIC. > > As part of that discussion, I'd suggest that we formalise that its maintenance (as well as that of QPACK) pass off to the HTTP WG once we've published it. > > I've talked about this with a number of folks. The only concerns I've heard so far are: > > a) That QUIC isn't yet proven. That's true, but the name won't be formalised or used on the wire until the RFC is published, so we have a good amount of time to back away. Even then, if it fails in the market, we can always skip to HTTP/4 one day, if we need to. > > b) That discussing naming is a distraction from our technical work. I agree with the concern overall, but we have a responsibility to communicate clearly with external developers and users, so I'd like to have a *limited* discussion about this. Please, don't bike shed. > > *Chair hat* > > We plan on reserving a very small period of time to discuss this in Bangkok; barring serious objections (of which "what about this name that I like instead?" is not one), we'll bring it up with the HTTP WG in Bangkok as well. > > Cheers, > - -- Mark Nottingham https://www.mnot.net/ ------- End of Forwarded Message