To All,<div><br></div><div>Well I think I have varnish behaving the way that I expected when backend content has both Content-Length and Transfer-Encoding headers. (I know having both is a violation of HTTP 1.1 protocol but I have little control over the backend)</div>
<div><br></div><div>I changed the order of logic in cache_fetch.c for determining how the body should be handled. I placed the Transfer-Encoding is statements before the Content-Length statements. That way if backend content has both headers it will used chunked instead of length and the chunked characters don't show up in the browser.</div>
<div> </div><div>From new cache_fetch.c</div><div><br></div><div><div>FetchReqBody(struct sess *sp)</div><div>{</div><div>...</div><div> /* Moved this if statement up */</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>if (http_GetHdr(sp->http, H_Transfer_Encoding, NULL)) {</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>/* XXX: Handle chunked encoding. */</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>WSL(sp->wrk, SLT_Debug, sp->fd, "Transfer-Encoding in request");</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>return (1);</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>}</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>if (http_GetHdr(sp->http, H_Content_Length, &ptr)) {</div>
<div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>content_length = strtoul(ptr, &endp, 10);</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>/* XXX should check result of conversion */</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>while (content_length) {</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>if (content_length > sizeof buf)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>rdcnt = sizeof buf;</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>else</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>rdcnt = content_length;</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>rdcnt = HTC_Read(sp->htc, buf, rdcnt);</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>if (rdcnt <= 0)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>return (1);</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>content_length -= rdcnt;</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>if (!sp->sendbody)</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>continue;</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>(void)WRW_Write(sp->wrk, buf, rdcnt); /* XXX: stats ? */</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>if (WRW_Flush(sp->wrk))</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>return (2);</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>}</div>
</div><div>...</div><div><br></div><div>AND....</div><div><br></div><div><div>FetchBody(struct sess *sp)</div><div>{</div></div><div>...</div><div><div> /* moved this pair of if statements up */</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>} else if (http_HdrIs(hp, H_Transfer_Encoding, "chunked")) {</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>sp->wrk->stats.fetch_chunked++;</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>cls = fetch_chunked(sp, sp->wrk->htc);</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>mklen = 1;</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>} else if (http_GetHdr(hp, H_Transfer_Encoding, &b)) {</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>sp->wrk->stats.fetch_bad++;</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>/* XXX: AUGH! */</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>WSL(sp->wrk, SLT_Debug, vc->fd, "Invalid Transfer-Encoding");</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>VBE_ClosedFd(sp);</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>return (__LINE__);</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>} else if (http_GetHdr(hp, H_Content_Length, &b)) {</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>sp->wrk->stats.fetch_length++;</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>cls = fetch_straight(sp, sp->wrk->htc, b);</div>
<div><span class="Apple-tab-span" style="white-space:pre"> </span>mklen = 1;</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>} else if (http_HdrIs(hp, H_Connection, "keep-alive")) {</div>
</div><div>....</div><div><br></div><div><br><div class="gmail_quote">On Wed, Jun 30, 2010 at 9:08 AM, David Brown <span dir="ltr"><<a href="mailto:captkiddo@gmail.com">captkiddo@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
You were right, it was wrong. The change I made to the VCL fixed the one problem but screwed up the rest of the site so I had to back it out.<div><br></div><div>Attached is a varnish log of the activity. I have some tcpdumps at various points too if you need them.</div>
<div><br></div><div>I still think it's connected to tomcat and specifically the JK connector. It seems the JK connector has an 8K limit (AJP 1.3) and breaks up large files in the transmission between tomcat and the webserver. The characters may be (?) part of JK's breakup of the file. My reasoning is because I can take the same file and place it directly on the web server and no problem, only when it comes from tomcat/jk do we have these issues. ( I can't convince the programmers to take static content out of their java apps and put it on the server )</div>
<div><div></div><div class="h5">
<div><br></div><div><br></div><div><br><div class="gmail_quote">On Tue, Jun 29, 2010 at 5:22 PM, Poul-Henning Kamp <span dir="ltr"><<a href="mailto:phk@phk.freebsd.dk" target="_blank">phk@phk.freebsd.dk</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
In message <<a href="mailto:AANLkTikRDbvceKr14-PKP8RuIC9Djyab0kLiehPG2Ea4@mail.gmail.com" target="_blank">AANLkTikRDbvceKr14-PKP8RuIC9Djyab0kLiehPG2Ea4@mail.gmail.com</a>>, Davi<br>
<div>d Brown writes:<br>
<br>
>I checked and it seems the Transfer-Encoding header is being stripped by<br>
>Varnish. I can see it when I go directly to the web server but not when I<br>
>view the file through Varnish.<br>
><br>
>I worked around the problem by adding this to my VCL<br>
<br>
</div>That sounds positively wrong.<br>
<br>
Is it possible that you can capture a varnishlog of one of these<br>
transaction and send to me ?<br>
<font color="#888888"><br>
<br>
--<br>
</font><div><div></div><div>Poul-Henning Kamp | UNIX since Zilog Zeus 3.20<br>
phk@FreeBSD.ORG | TCP/IP since RFC 956<br>
FreeBSD committer | BSD since 4.3-tahoe<br>
Never attribute to malice what can adequately be explained by incompetence.<br>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>