<div dir="ltr">Hi.<div><br></div><div>Intermittently failing tests seems to be an issue. The way the current tests are scheduled and run (autocrap and make) are not perfect for handling this. I've written a replacement to give us a bit more flexibility wrt to how tests are run.</div><div><br></div><div>The below gisted program is a multithreaded python program that run tests. The tests are typically run in parallel (-j n). If a test fails it is scheduled for single threaded execution and these are executed after the rest of the tests are done. This should hopefully reduce the amount of intermittent failures quite with only a small increase in time taken to execute.</div><div><br></div><div>If we want to gather long term statistics having this program interface with a database of sorts should be reasonably trivial. </div><div><br></div><div>Another option would be to start giving hints on how the test should be executed within the test itself. By having a header on the test that the testrunner could read one can set up certain parameters.</div><div><br></div><div><font face="monospace, monospace"># parallel=0,load<1,timeout=15s,platforms=!netbsd,retries=3</font></div><div><br></div><div>Let me know if I should commit this and start replacement of the current way tests are run.</div><div><br></div><div><div>Draft code available at:</div><div><a href="https://gist.github.com/perbu/3a722df0b168ac8f73bb">https://gist.github.com/perbu/3a722df0b168ac8f73bb</a><br></div><div><br></div><div><font face="arial, helvetica, sans-serif">Syntax:</font></div><div><div><font face="monospace, monospace">Usage: testrunner <option></font></div><div><font face="monospace, monospace"> [-h] - Show help and exit.</font></div><div><font face="monospace, monospace"> [-j <threads>] - paralellize testing (default: 8)</font></div><div><font face="monospace, monospace"> [-v] - verbose mode (default is off)</font></div><div><font face="monospace, monospace"> [-o <file>] - log to this file (default testrunner.log)</font></div><div><font face="monospace, monospace"> [-e <varnishtest>] - set varnishtest path</font></div><div><font face="monospace, monospace"> [-l <level>] - set log level (DEBUG,INFO,WARNING,ERROR)</font></div><div><font face="monospace, monospace"> [-t <dir>] - look for tests in this directory (default: tests)</font></div><div><font face="monospace, monospace"> [-p ] - enable progress</font></div><div><font face="monospace, monospace"> [-a <arg>] - pass this argument to varnishtest. Can be used multiple times.</font></div></div>-- <br><div class="gmail_signature"><div dir="ltr"><table border="0" cellpadding="0" cellspacing="0" style="border-bottom-width:1px;padding-top:5px;border-top-style:solid;width:550px;padding-bottom:5px;border-bottom-color:rgb(238,238,238);border-top-width:1px;border-bottom-style:solid;line-height:1.5em;border-top-color:rgb(238,238,238);color:rgb(102,102,102);font-size:12px;font-family:'Helvetica Neue',Arial,sans-serif;margin-top:20px"><tbody><tr><td width="100"><img src="http://www.varnish-software.com/static/media/logo-email.png"></td><td><font color="#222222"><span style="font-size:14px"><b>Per Buer</b></span></font><br>CTO | Varnish Software AS<br>Cell: <a value="+4790181750" style="color:rgb(17,85,204)">+47 95839117</a><br><span style="font-weight:bold">We Make Websites Fly!<br><a href="https://www.varnish-software.com/" style="color:rgb(17,85,204)" target="_blank">www.varnish-software.com</a></span></td></tr></tbody></table><br><div style="color:rgb(136,136,136)"><a href="http://info.varnish-software.com/varnish-summits-autumn-2014-registration" style="color:rgb(17,85,204)" target="_blank"><img src="https://www.varnish-software.com/sites/default/files/u388/masters_s_0.png" alt=" Register now"></a></div></div></div>
</div></div>