Ticket #1236 (closed defect: worksforme)

Opened 17 months ago

Last modified 17 months ago

requests to same url sometimes hang

Reported by: reinhard Owned by:
Priority: normal Milestone:
Component: varnishd Version: 3.0.3
Severity: normal Keywords:
Cc:

Description

The problem is, that if a script with some parameters takes a long time to execute it sometimes happens, that a call to the script with other parameters hangs till the first long running call is finished.

we use a plain centos 6.3 with the newest varnish 3.0.3 rpms from  http://repo.varnish-cache.org/redhat/varnish-3.0/el6/x86_64/.

We were able to reproduce this behavior with the following setup. It doesn't happens always but on the server i tested it around 1 in 3 times so it should be fairly easy to reproduce. Attached is the script test.php. The important part is that the script sleeps 30 seconds if it is called with the parameter t=1 otherwise it returns some output i used in debugging the problem.

The following steps must be done to reproduce the problem:

1.) restart varnish

2.) call the script with the parameter that the script takes a long time to finish: curl -i -H "host: test.de" 'localhost:6081/test.php?t=1&time='`date +%H_%M_%S`

3.) call the script with a parameter that finish fast. The first call always works and don't hang: curl -i -H "host: test.de" 'localhost:6081/test.php?t=2&time='`date +%H_%M_%S`

4.) call the script with a parameter that finish fast a second time immediately afterwards: curl -i -H "host: test.de" 'localhost:6081/test.php?t=3&time='`date +%H_%M_%S`

If i waited to long with the third call, i wasn't able to reproduce the problem. Even if i was fast only around 1 in 3 attempts showed the problem: The call waited till the first call was finished before initiating a connection to the backend server.

I have also attached the log file of varnishlog -u -O -w varnish2.log during the execution. The problem also occurs with varnish 2.1.5. Please let me know if you need further informations.

Attachments

test.php Download (293 bytes) - added by reinhard 17 months ago.
varnish2.log Download (6.9 KB) - added by reinhard 17 months ago.

Change History

Changed 17 months ago by reinhard

Changed 17 months ago by reinhard

comment:1 Changed 17 months ago by martin

Hi,

Could you verify the minimum number of threads you have configured? This sounds like you are running with a only 1 worker thread, and you are being hit by Varnish not wanting to create new threads unless there is a real workload present needing it. Try increasing the minimum number of threads.

Reagrds, Martin Blix Grydeland

comment:2 Changed 17 months ago by martin

  • Status changed from new to closed
  • Resolution set to worksforme

comment:3 Changed 17 months ago by martin

The Varnish min/max thread settings are per thread pool, and the number of pools defaults to 2. So you were running with a total minimum of 2, which is why the problem showed only at the 3rd request.

There are a number of runtime parameters that governs thread behavior in Varnish. Look in the man page and also in that section of the Varnish book. Note also that while Varnish will scale up the number of threads automatically when there is consistent thread pressure over time, that algorithm isn't designed for these low numbers and a request can be starved in that case.

The default of 1 in the RPMs is just too low to make a running Varnish instance, and I believe this has been changed in recent RPMs (but your configuration files may stuck form an earlier release?) You should configure more like a minimum of 100. Varnish' threads are cheap, and if they are not ever being used they hardly take up any resources either.

Regards, Martin Blix Grydeland

Note: See TracTickets for help on using tickets.