On tests

[Per Buer]

Hi.

Intermittently failing tests seems to be an issue. The way the current
tests are scheduled and run (autocrap and make) are not perfect for
handling this. I've written a replacement to give us a bit more flexibility
wrt to how tests are run.

The below gisted program is a multithreaded python program that run tests.
The tests are typically run in parallel (-j n). If a test fails it is
scheduled for single threaded execution and these are executed after the
rest of the tests are done. This should hopefully reduce the amount of
intermittent failures quite with only a small increase in time taken to
execute.

If we want to gather long term statistics having this program interface
with a database of sorts should be reasonably trivial.

Another option would be to start giving hints on how the test should be
executed within the test itself. By having a header on the test that the
testrunner could read one can set up certain parameters.

# parallel=0,load<1,timeout=15s,platforms=!netbsd,retries=3

Let me know if I should commit this and start replacement of the current
way tests are run.

Draft code available at:
https://gist.github.com/perbu/3a722df0b168ac8f73bb

Syntax:
Usage: testrunner <option>
 [-h]               - Show help and exit.
 [-j <threads>]     - paralellize testing (default: 8)
 [-v]               - verbose mode  (default is off)
 [-o <file>]        - log to this file (default testrunner.log)
 [-e <varnishtest>] - set varnishtest path
 [-l <level>]       - set log level (DEBUG,INFO,WARNING,ERROR)
 [-t <dir>]         - look for tests in this directory (default: tests)
 [-p ]              - enable progress
 [-a <arg>]         - pass this argument to varnishtest. Can be used
multiple times.
-- 
*Per Buer*
CTO | Varnish Software AS
Cell: +47 95839117
We Make Websites Fly!
www.varnish-software.com
[image: Register now]
<http://info.varnish-software.com/varnish-summits-autumn-2014-registration>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20150309/e7f5aa7d/attachment.html>