[Varnish] #788: v2.1.3 w/ http_range_support on fails to support sets in byte range request
Varnish
varnish-bugs at varnish-cache.org
Thu Nov 4 11:50:19 CET 2010
#788: v2.1.3 w/ http_range_support on fails to support sets in byte range request
--------------------------+-------------------------------------------------
Reporter: jim.robinson | Owner: phk
Type: defect | Status: new
Priority: normal | Milestone:
Component: build | Version: 2.1.3
Severity: normal | Keywords: http_range_support byte range set
--------------------------+-------------------------------------------------
Changes (by phk):
* owner: => phk
Old description:
> Varnish 2.1.3 byte range support fails to work when handed a set of
> ranges.
>
> The HTTP 1.1 specification, section "14.35.1 Byte Ranges" specifies
> that
>
> {{{
> "A byte range operation MAY specify a single range of bytes,
> or a set of ranges within a single entity."
> }}}
>
> Given a simple VCL of:
>
> {{{
> backend default {
> .host = "127.0.0.1";
> .port = "80";
> }
> }}}
>
> where the backend is an Apache 2.2.14 server, we can start up varnishd
> on port 4040 and activate range support:
>
> {{{
> $ sudo /usr/local/varnish/2.1.3/sbin/varnishd -d -P /var/run/varnish.pid
> -a localhost:4040 -f /usr/local/varnish/2.1.3/etc/varnish/test.vcl -u
> nobody -g nobody -s malloc,10M
>
> storage_malloc: max size 10 MB.
> Using old SHMFILE
> Varnish on Darwin,9.8.0,i386,-smalloc,-hcritbit
> 200 193
> -----------------------------
> Varnish HTTP accelerator CLI.
> -----------------------------
> Type 'help' for command list.
> Type 'quit' to close CLI session.
> Type 'start' to launch worker process.
>
> start
> child (69026) Started
> 200 0
>
> param.set http_range_support on
> 200 0
> }}}
>
> In another window we first issue a curl request to our apache server and
> ask for the first four bytes as a set of 1-byte ranges:
>
> {{{
> $ curl -s -i -HRange:bytes=0-0,1-1,2-2,3-3 http://127.0.0.1/test.pdf
> HTTP/1.1 206 Partial Content
> Date: Mon, 04 Oct 2010 19:46:25 GMT
> Server: Apache/2.2.14 (Unix) DAV/2 mod_jk/1.2.28 mod_ssl/2.2.14
> OpenSSL/0.9.7a
> Last-Modified: Fri, 09 Jul 2010 16:52:27 GMT
> ETag: "11fc13b-11893e-48af73a5d3ba1"
> Accept-Ranges: bytes
> Content-Length: 389
> Content-Type: multipart/byteranges; boundary=491cfccba1f8e35b9
>
> --491cfccba1f8e35b9
> Content-type: application/pdf
> Content-range: bytes 0-0/1149246
>
> %
> --491cfccba1f8e35b9
> Content-type: application/pdf
> Content-range: bytes 1-1/1149246
>
> P
> --491cfccba1f8e35b9
> Content-type: application/pdf
> Content-range: bytes 2-2/1149246
>
> D
> --491cfccba1f8e35b9
> Content-type: application/pdf
> Content-range: bytes 3-3/1149246
>
> F
> --491cfccba1f8e35b9--
> }}}
>
> Now while we can issue a single range request to varnish and get back the
> right thing:
>
> {{{
> $ curl -s -i -HRange:bytes=1-1 http://localhost:4040/test.pdf ; echo
> HTTP/1.1 206 Partial Content
> Server: Apache/2.2.14 (Unix) DAV/2 mod_jk/1.2.28 mod_ssl/2.2.14
> OpenSSL/0.9.7a
> Last-Modified: Fri, 09 Jul 2010 16:52:27 GMT
> ETag: "11fc13b-11893e-48af73a5d3ba1"
> Content-Type: application/pdf
> Accept-Ranges: bytes
> Date: Mon, 04 Oct 2010 19:46:33 GMT
> X-Varnish: 716983327 716983316
> Age: 189
> Via: 1.1 varnish
> Connection: keep-alive
> Content-Range: bytes 1-1/1149246
> Content-Length: 1
>
> P
> }}}
>
> But we cannot issue a set of ranges:
>
> {{{
> $ curl -s -i -HRange:bytes=0-0,1-1,2-2,3-3
> http://localhost:4040/test.pdf | strings | head -51param.set
> http_range_support on
> HTTP/1.1 200 OK
> Server: Apache/2.2.14 (Unix) DAV/2 mod_jk/1.2.28 mod_ssl/2.2.14
> OpenSSL/0.9.7a
> Last-Modified: Fri, 09 Jul 2010 16:52:27 GMT
> ETag: "11fc13b-11893e-48af73a5d3ba1"
> Content-Type: application/pdf
> Content-Length: 1149246
> Accept-Ranges: bytes
> Date: Mon, 04 Oct 2010 19:46:46 GMT
> X-Varnish: 716983328 716983316
> Age: 203
> Via: 1.1 varnish
> Connection: keep-alive
> %PDF-1.4
> 45 0 obj
> ... rest of the entire PDF elided...
> }}}
>
> The problem appears to be that cache_response.c has an assumption built
> into the logic that byte range requests aren't multi-sequence:
>
> {{{
> 278 if (sp->disable_esi || sp->esis == 0) {
> 279 /* For none ESI and non ESI-included objects, try Range
> */
> 280 if (params->http_range_support &&
> 281 (sp->disable_esi || sp->esis == 0) &&
> 282 sp->obj->response == 200 &&
> 283 sp->wantbody &&
> 284 http_GetHdr(sp->http, H_Range, &r))
> 285 res_dorange(sp, r, &low, &high);
> 286
> 287 sp->acct_tmp.hdrbytes += http_Write(sp->wrk,
> sp->wrk->resp, 1);
> }}}
>
> The res_dorange request that calculates the range and the subsequent call
> to http_Write assume only a single range, not a set of ranges.
>
> Jim
New description:
Varnish 2.1.3 byte range support fails to work when handed a set of
ranges.
The HTTP 1.1 specification, section "14.35.1 Byte Ranges" specifies
that
{{{
"A byte range operation MAY specify a single range of bytes,
or a set of ranges within a single entity."
}}}
Given a simple VCL of:
{{{
backend default {
.host = "127.0.0.1";
.port = "80";
}
}}}
where the backend is an Apache 2.2.14 server, we can start up varnishd
on port 4040 and activate range support:
{{{
$ sudo /usr/local/varnish/2.1.3/sbin/varnishd -d -P /var/run/varnish.pid
-a localhost:4040 -f /usr/local/varnish/2.1.3/etc/varnish/test.vcl -u
nobody -g nobody -s malloc,10M
storage_malloc: max size 10 MB.
Using old SHMFILE
Varnish on Darwin,9.8.0,i386,-smalloc,-hcritbit
200 193
-----------------------------
Varnish HTTP accelerator CLI.
-----------------------------
Type 'help' for command list.
Type 'quit' to close CLI session.
Type 'start' to launch worker process.
start
child (69026) Started
200 0
param.set http_range_support on
200 0
}}}
In another window we first issue a curl request to our apache server and
ask for the first four bytes as a set of 1-byte ranges:
{{{
$ curl -s -i -HRange:bytes=0-0,1-1,2-2,3-3 http://127.0.0.1/test.pdf
HTTP/1.1 206 Partial Content
Date: Mon, 04 Oct 2010 19:46:25 GMT
Server: Apache/2.2.14 (Unix) DAV/2 mod_jk/1.2.28 mod_ssl/2.2.14
OpenSSL/0.9.7a
Last-Modified: Fri, 09 Jul 2010 16:52:27 GMT
ETag: "11fc13b-11893e-48af73a5d3ba1"
Accept-Ranges: bytes
Content-Length: 389
Content-Type: multipart/byteranges; boundary=491cfccba1f8e35b9
--491cfccba1f8e35b9
Content-type: application/pdf
Content-range: bytes 0-0/1149246
%
--491cfccba1f8e35b9
Content-type: application/pdf
Content-range: bytes 1-1/1149246
P
--491cfccba1f8e35b9
Content-type: application/pdf
Content-range: bytes 2-2/1149246
D
--491cfccba1f8e35b9
Content-type: application/pdf
Content-range: bytes 3-3/1149246
F
--491cfccba1f8e35b9--
}}}
Now while we can issue a single range request to varnish and get back the
right thing:
{{{
$ curl -s -i -HRange:bytes=1-1 http://localhost:4040/test.pdf ; echo
HTTP/1.1 206 Partial Content
Server: Apache/2.2.14 (Unix) DAV/2 mod_jk/1.2.28 mod_ssl/2.2.14
OpenSSL/0.9.7a
Last-Modified: Fri, 09 Jul 2010 16:52:27 GMT
ETag: "11fc13b-11893e-48af73a5d3ba1"
Content-Type: application/pdf
Accept-Ranges: bytes
Date: Mon, 04 Oct 2010 19:46:33 GMT
X-Varnish: 716983327 716983316
Age: 189
Via: 1.1 varnish
Connection: keep-alive
Content-Range: bytes 1-1/1149246
Content-Length: 1
P
}}}
But we cannot issue a set of ranges:
{{{
$ curl -s -i -HRange:bytes=0-0,1-1,2-2,3-3 http://localhost:4040/test.pdf
| strings | head -51param.set http_range_support on
HTTP/1.1 200 OK
Server: Apache/2.2.14 (Unix) DAV/2 mod_jk/1.2.28 mod_ssl/2.2.14
OpenSSL/0.9.7a
Last-Modified: Fri, 09 Jul 2010 16:52:27 GMT
ETag: "11fc13b-11893e-48af73a5d3ba1"
Content-Type: application/pdf
Content-Length: 1149246
Accept-Ranges: bytes
Date: Mon, 04 Oct 2010 19:46:46 GMT
X-Varnish: 716983328 716983316
Age: 203
Via: 1.1 varnish
Connection: keep-alive
%PDF-1.4
45 0 obj
... rest of the entire PDF elided...
}}}
The problem appears to be that cache_response.c has an assumption built
into the logic that byte range requests aren't multi-sequence:
{{{
278 if (sp->disable_esi || sp->esis == 0) {
279 /* For none ESI and non ESI-included objects, try Range */
280 if (params->http_range_support &&
281 (sp->disable_esi || sp->esis == 0) &&
282 sp->obj->response == 200 &&
283 sp->wantbody &&
284 http_GetHdr(sp->http, H_Range, &r))
285 res_dorange(sp, r, &low, &high);
286
287 sp->acct_tmp.hdrbytes += http_Write(sp->wrk,
sp->wrk->resp, 1);
}}}
The res_dorange request that calculates the range and the subsequent call
to http_Write assume only a single range, not a set of ranges.
Jim
--
Comment:
Sorry about the delay in replying.
Yes, you are right, we only handle a single byterange interval at present,
my survey of the net found no significant usage of multi-range requests.
Do you know of any applications that actually use multi-range requests ?
--
Ticket URL: <http://www.varnish-cache.org/trac/ticket/788#comment:1>
Varnish <http://varnish-cache.org/>
The Varnish HTTP Accelerator
More information about the varnish-bugs
mailing list