Ticket #788 (closed defect: invalid)

Opened 4 years ago

Last modified 3 years ago

v2.1.3 w/ http_range_support on fails to support sets in byte range request

Reported by: jim.robinson Owned by: kristian
Priority: normal Milestone: Varnish 3.0 dev
Component: varnishd Version: 2.1.3
Severity: normal Keywords: http_range_support byte range set
Cc:

Description (last modified by phk) (diff)

Varnish 2.1.3 byte range support fails to work when handed a set of ranges.

The HTTP 1.1 specification, section "14.35.1 Byte Ranges" specifies that

  "A byte range operation MAY specify a single range of bytes,
   or a set of ranges within a single entity."

Given a simple VCL of:

backend default {
  .host = "127.0.0.1";
  .port = "80";
}

where the backend is an Apache 2.2.14 server, we can start up varnishd on port 4040 and activate range support:

$ sudo /usr/local/varnish/2.1.3/sbin/varnishd -d -P /var/run/varnish.pid -a localhost:4040 -f /usr/local/varnish/2.1.3/etc/varnish/test.vcl -u nobody -g nobody -s malloc,10M

storage_malloc: max size 10 MB.
Using old SHMFILE
Varnish on Darwin,9.8.0,i386,-smalloc,-hcritbit
200 193     
-----------------------------
Varnish HTTP accelerator CLI.
-----------------------------
Type 'help' for command list.
Type 'quit' to close CLI session.
Type 'start' to launch worker process.

start
child (69026) Started
200 0       

param.set http_range_support on
200 0       

In another window we first issue a curl request to our apache server and ask for the first four bytes as a set of 1-byte ranges:

$  curl -s -i -HRange:bytes=0-0,1-1,2-2,3-3  http://127.0.0.1/test.pdf 
HTTP/1.1 206 Partial Content
Date: Mon, 04 Oct 2010 19:46:25 GMT
Server: Apache/2.2.14 (Unix) DAV/2 mod_jk/1.2.28 mod_ssl/2.2.14 OpenSSL/0.9.7a
Last-Modified: Fri, 09 Jul 2010 16:52:27 GMT
ETag: "11fc13b-11893e-48af73a5d3ba1"
Accept-Ranges: bytes
Content-Length: 389
Content-Type: multipart/byteranges; boundary=491cfccba1f8e35b9


--491cfccba1f8e35b9
Content-type: application/pdf
Content-range: bytes 0-0/1149246

%
--491cfccba1f8e35b9
Content-type: application/pdf
Content-range: bytes 1-1/1149246

P
--491cfccba1f8e35b9
Content-type: application/pdf
Content-range: bytes 2-2/1149246

D
--491cfccba1f8e35b9
Content-type: application/pdf
Content-range: bytes 3-3/1149246

F
--491cfccba1f8e35b9--

Now while we can issue a single range request to varnish and get back the right thing:

$ curl -s -i -HRange:bytes=1-1  http://localhost:4040/test.pdf ; echo
HTTP/1.1 206 Partial Content
Server: Apache/2.2.14 (Unix) DAV/2 mod_jk/1.2.28 mod_ssl/2.2.14 OpenSSL/0.9.7a
Last-Modified: Fri, 09 Jul 2010 16:52:27 GMT
ETag: "11fc13b-11893e-48af73a5d3ba1"
Content-Type: application/pdf
Accept-Ranges: bytes
Date: Mon, 04 Oct 2010 19:46:33 GMT
X-Varnish: 716983327 716983316
Age: 189
Via: 1.1 varnish
Connection: keep-alive
Content-Range: bytes 1-1/1149246
Content-Length: 1

P

But we cannot issue a set of ranges:

$ curl -s -i -HRange:bytes=0-0,1-1,2-2,3-3  http://localhost:4040/test.pdf | strings | head -51param.set http_range_support on
HTTP/1.1 200 OK
Server: Apache/2.2.14 (Unix) DAV/2 mod_jk/1.2.28 mod_ssl/2.2.14 OpenSSL/0.9.7a
Last-Modified: Fri, 09 Jul 2010 16:52:27 GMT
ETag: "11fc13b-11893e-48af73a5d3ba1"
Content-Type: application/pdf
Content-Length: 1149246
Accept-Ranges: bytes
Date: Mon, 04 Oct 2010 19:46:46 GMT
X-Varnish: 716983328 716983316
Age: 203
Via: 1.1 varnish
Connection: keep-alive
%PDF-1.4
45 0 obj
... rest of the entire PDF elided...

The problem appears to be that cache_response.c has an assumption built into the logic that byte range requests aren't multi-sequence:

278     if (sp->disable_esi || sp->esis == 0) {
279             /* For none ESI and non ESI-included objects, try Range */
280             if (params->http_range_support &&
281                 (sp->disable_esi || sp->esis == 0) &&
282                 sp->obj->response == 200 &&
283                 sp->wantbody &&
284                 http_GetHdr(sp->http, H_Range, &r))
285                     res_dorange(sp, r, &low, &high);
286 
287             sp->acct_tmp.hdrbytes += http_Write(sp->wrk, sp->wrk->resp, 1);

The res_dorange request that calculates the range and the subsequent call to http_Write assume only a single range, not a set of ranges.

Jim

Attachments

TracFaq.txt Download (3.8 KB) - added by thomasjones 2 months ago.
 seo services

Change History

comment:1 Changed 3 years ago by phk

  • Owner set to phk
  • Description modified (diff)

Sorry about the delay in replying.

Yes, you are right, we only handle a single byterange interval at present, my survey of the net found no significant usage of multi-range requests.

Do you know of any applications that actually use multi-range requests ?

comment:2 Changed 3 years ago by jnerin

Well, I'm not sure how widespread may be, but I have one example, it's Adobe Acrobat trying to display a pdf embedded in the browser (the pdf file size is 7271640 bytes):

Header from a tcpdump decoded with wireshark:

GET /example.pdf HTTP/1.1
Host: www.example.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; es-ES; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12 ( .NET CLR 3.5.30729)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: es-es,es;q=0.8,en-us;q=0.5,en;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Connection: keep-alive
Range: bytes=8380-32397,8380-8381

If first issues on request with no range:

  702 RxRequest    c GET
  702 RxURL        c /example.pdf
  702 RxProtocol   c HTTP/1.1
  702 RxHeader     c Host: www.example.com
  702 RxHeader     c User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; es-ES; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12 ( .NET CLR 3.5.30729)
  702 RxHeader     c Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
  702 RxHeader     c Accept-Language: es-es,es;q=0.8,en-us;q=0.5,en;q=0.3
  702 RxHeader     c Accept-Encoding: gzip,deflate
  702 RxHeader     c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
  702 RxHeader     c Keep-Alive: 115
  702 RxHeader     c Connection: keep-alive
  702 RxHeader     c Referer: http://www.example.com/
  702 VCL_call     c recv
  702 VCL_return   c lookup
  702 VCL_call     c hash
  702 VCL_return   c hash
  702 VCL_call     c miss
  702 VCL_return   c fetch
  702 Backend      c 674 default default
  674 TxRequest    b GET
  674 TxURL        b /example.pdf
  674 TxProtocol   b HTTP/1.1
  674 TxHeader     b Host: www.example.com
  674 TxHeader     b User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; es-ES; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12 ( .NET CLR 3.5.30729)
  674 TxHeader     b Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
  674 TxHeader     b Accept-Language: es-es,es;q=0.8,en-us;q=0.5,en;q=0.3
  674 TxHeader     b Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
  674 TxHeader     b Referer: http://www.example.com/
  674 TxHeader     b Accept-Encoding: gzip
  674 TxHeader     b X-Varnish: 1459169995
[...]
  702 TTL          c 1459169995 RFC 120 1289908815 0 0 0 0
  702 VCL_call     c fetch
  702 VCL_return   c deliver
  702 ObjProtocol  c HTTP/1.1
  702 ObjStatus    c 200
  702 ObjResponse  c OK
  702 ObjHeader    c Date: Tue, 16 Nov 2010 12:00:15 GMT
  702 ObjHeader    c Server: Apache
  702 ObjHeader    c Last-Modified: Tue, 16 Nov 2010 07:38:04 GMT
  702 ObjHeader    c ETag: "18616f87-6ef4d8-49526a311ef00"
  702 ObjHeader    c Content-Type: application/pdf
  702 ObjHeader    c X-Cacheable: YES
  674 Length       b 7271640
  674 BackendReuse b default
[...]
  702 SessionClose c remote closed
  702 VCL_call     c deliver
  702 VCL_return   c deliver
  702 TxProtocol   c HTTP/1.1
  702 TxStatus     c 200
  702 TxResponse   c OK
  702 TxHeader     c Server: Apache
  702 TxHeader     c Last-Modified: Tue, 16 Nov 2010 07:38:04 GMT
  702 TxHeader     c ETag: "18616f87-6ef4d8-49526a311ef00"
  702 TxHeader     c Content-Type: application/pdf
  702 TxHeader     c X-Cacheable: YES
  702 TxHeader     c Content-Length: 7271640
  702 TxHeader     c Accept-Ranges: bytes
  702 TxHeader     c Date: Tue, 16 Nov 2010 12:00:15 GMT
  702 TxHeader     c X-Varnish: 1459169995
  702 TxHeader     c Age: 0
  702 TxHeader     c Via: 1.1 varnish
  702 TxHeader     c Connection: keep-alive
  702 TxHeader     c X-Served-By: varnish
  702 TxHeader     c X-Cache: MISS
  702 TxHeader     c X-Cache-Hits: 0
  702 Debug        c "Write error, retval = 32120, len = 7272020, errno = Success"
   -1 Length       - 7271640
  702 ReqEnd       c 1459169995 1289908815.705988407 1289908816.018261433 2.889583826 0.033922911 0.278350115
  702 StatSess     c xxx.xxx.xxx.xxx 13723 3 1 3 0 0 2 1179 7272197

Then reissues the same request but with a range:

  282 ReqStart     c xxx.xxx.xxx.xxx 13724 1459170027
  282 RxRequest    c GET
  282 RxURL        c /example.pdf
  282 RxProtocol   c HTTP/1.1
  282 RxHeader     c Host: www.example.com
  282 RxHeader     c User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; es-ES; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12 ( .NET CLR 3.5.30729)
  282 RxHeader     c Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
  282 RxHeader     c Accept-Language: es-es,es;q=0.8,en-us;q=0.5,en;q=0.3
  282 RxHeader     c Accept-Encoding: gzip,deflate
  282 RxHeader     c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
  282 RxHeader     c Keep-Alive: 115
  282 RxHeader     c Connection: keep-alive
  282 RxHeader     c Range: bytes=8380-32397,8380-8381
  282 VCL_call     c recv
  282 VCL_return   c lookup
  282 VCL_call     c hash
  282 VCL_return   c hash
  282 Hit          c 1459169995
  282 VCL_call     c hit
  282 VCL_return   c deliver
  282 VCL_call     c deliver
  282 VCL_return   c deliver
  282 TxProtocol   c HTTP/1.1
  282 TxStatus     c 200
  282 TxResponse   c OK
  282 TxHeader     c Server: Apache
  282 TxHeader     c Last-Modified: Tue, 16 Nov 2010 07:38:04 GMT
  282 TxHeader     c ETag: "18616f87-6ef4d8-49526a311ef00"
  282 TxHeader     c Content-Type: application/pdf
  282 TxHeader     c X-Cacheable: YES
  282 TxHeader     c Content-Length: 7271640
  282 TxHeader     c Accept-Ranges: bytes
  282 TxHeader     c Date: Tue, 16 Nov 2010 12:00:15 GMT
  282 TxHeader     c X-Varnish: 1459170027 1459169995
  282 TxHeader     c Age: 0
  282 TxHeader     c Via: 1.1 varnish
  282 TxHeader     c Connection: keep-alive
  282 TxHeader     c X-Served-By: varnish
  282 TxHeader     c X-Cache: HIT
  282 TxHeader     c X-Cache-Hits: 1
  282 Length       c 7271640
  282 ReqEnd       c 1459170027 1289908815.935475588 1289908831.356354237 3.038742304 0.000062466 15.420816183

At this point the browser is stuck waiting for data forever. I don't know why it asks for this absurd range pair.

For the moment we have sidestepped the issue adding this to vcl_recv:

vcl_recv {
[...]
        if (req.http.Range) {
                return(pipe);
        }
[...]
}

comment:3 Changed 3 years ago by kristian

  • Owner changed from phk to kristian
  • Component changed from build to varnishd

I'm checking the fall-back for this mode to assess priority.

comment:4 Changed 3 years ago by kristian

  • Milestone set to Varnish 3.0 dev

Need to assess this before 3.0. Sorry for the spam.

comment:5 Changed 3 years ago by phk

  • Status changed from new to closed
  • Resolution set to invalid

We revisted the oldest tickets on the bug-wash today, and finally(!) made our mind up about this one:

The present single-range support was based on a survey of range requests I collected from various sites. The survey found only very few instances of multi-range requests on varnish servers and thus the added complexity was not implemented.

Circumstances may have changed, and I welcome data showing that, but this ticket really is a feature-request, and I have moved it to wiki:Future_Feature as such.

I do notice that the multi-range requests above ask for overlapping areas:

Range: bytes=8380-32397,8380-8381

I wonder if that is bug in acrobat ?

I'm closing this ticket as "invalid" according to our policy, because it is a feature-request, we only use tickets for "real bugs", in order to not have tickets that cannot be resolved in finite time.

Sorry for the delay in dealing with this ticket.

Changed 2 months ago by thomasjones

Note: See TracTickets for help on using tickets.