<div dir="ltr">I've not responded in awhile, but this one touches on things close to how I use aspects of Varnish.<br><div class="gmail_extra"><br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
(This is rather long and rambling and probably mostly for Martin,<br>
but I'm sending it -dev in case anybody else is interested and for<br>
completeness of the mail-archive.)<br>
<br></blockquote><div> </div><div>[SNIP]<br><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
This raises the interesting question: Should we always use a busyobj<br>
to fill an objcore, even for req.body and synthetic ?<br></blockquote><div><br></div><div>Anything that slows down synthetic, makes me worry. I like all the benefits that I get<br></div><div>for normal use cases, plus the extra boost from "clever" synthetic services. <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
I'm increasingly leaning that way, for reasons of code simplicity<br>
and symmetry.<br></blockquote><div><br></div><div>But, I can see the value to the underlying system.<br> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Streaming on synthetic seems far out only until you start using<br>
weird VMODs to generate the synthetic response body slowly[1].<br></blockquote><div><br></div><div>This seems counter-intuitive. But I get the [a] use case and I've used it before.<br></div><div>Blackholing a requester with a very long response. But since I've got a terminator in front of varnish<br></div><div>and I'm more concerned about DDoS via TLS attacks, I shifted the blackholing to the terminator<br></div><div>and let varnish continue to do what it does best.<br><br></div><div>Alas, a port 80 blackhole could be just as important because my varnish servers also respond there.<br></div><div>I'd not considered this before.<br><br></div><div>I'd like to discuss more in Rotterdam. I've got something that could come in very handy in terms of managing<br></div><div>and accessing the "db" of prefixes for which you'd want to do something like this. There are other related use cases<br></div><div>for how you might decide to blackhole at the app layer (rather than IP filters, etc) and use the same lookup service<br></div><div>for other purposes.<br><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
...so assume for a moment we always use a busyobj and the streaming<br>
mechanism to fill objcores.<br>
<br>
As long as the busyobj is assembling the objcore, *all* the Obj->methods<br>
go to the busyobj's method set, and the busyobj methods call the<br>
stevedores methods to get things done.<br>
<br>
Once the busyobj is done, it calls the stevedore with a "commit"<br>
method and once that returns, all future Obj->methods go directly<br>
to the stevedore.<br>
<br>
... Except the ObjIter()s which were already running, we cannot<br>
allow the stevedore to rip storage out or move it under them,<br>
and they have to finish in the busyobj state.<br>
<br>
I can imagine three stevedore stragegies:<br>
<br>
1. Can do Trim in place, so allocates final storage right away<br>
and trims last segment when we know it.<br></blockquote><div><br></div><div>A blackhole service would have, in effect infinite storage. But of course, that doesn't happen<br></div><div>because of the very very slow chunking of response (e.g. byte per time period). These are<br></div><div>aberrant behavior though. You'd not want the response cached because you're really sacrifycing<br></div><div>a socket. Thus my desire to really implement this in front.<br></div><div> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
2. Allocates temp space for each segment, moves into final storage<br>
when it is full/finished.<br>
<br>
3. Allocates all segments in temp space, and moves them all into<br>
final storage only when object is finished.<br>
<br>
But I'm going to deny #3, because the performance difference between<br>
#2 with "large enough" segments and #3 is not relevant.<br>
<br>
So the stevedore gets to pick between strategy #1 and #2 for each<br>
object. The decision, and the #2 segment size, might be based on<br>
knowing the length from C-L or having an estimate from objhdr.<br>
<br>
This means we have at most one "uncommitted" segment at any time,<br>
and if we let the busyobj own that one (from Transient), the ObjIter<br>
issue is manageable.<br>
<br>
Fitting the "delete while passing" case into this, is a matter of<br>
bookkeeping and calling the stevedores ->delete_segment method.<br>
<br>
Comments, inputs, thoughts ?<br>
<br>
Poul-Henning<br>
<br>
<br>
[1] Actually, it's not obvious to me that we should have the "magic"<br>
path for synthetic any more, why don't we just push a synthetic<br>
backend and use the regular path ?<br></blockquote><div><br></div><div>I'm okay with this conceptually. Just need to understand the refactoring necessary and the (I can only assume)<br></div><div>decrease in throughput for an intentionally non-caching, synthetic backend. <br><br></div><div>But ... "magic" isn't my conceptual understanding. Awhile back we talked about separating the front-end and back-end <br></div><div>state machines. Synthetic living only in the front and having increased performance as a by-product. Isn't this separation also a <br></div><div>source of asymmetry? There is no BERESP. There is no BERESP keep/ttl, only the final deliver/resp. No stale serving. These<br></div><div>are all arguably things that support this idea and the resulting symmetry that would come of it.<br></div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
--<br>
Poul-Henning Kamp | UNIX since Zilog Zeus 3.20<br>
phk@FreeBSD.ORG | TCP/IP since RFC 956<br>
FreeBSD committer | BSD since 4.3-tahoe<br>
Never attribute to malice what can adequately be explained by incompetence.<br>
<br>
<br>
<br>
------------------------------<br>
<br>
_______________________________________________<br>
varnish-dev mailing list<br>
<a href="mailto:varnish-dev@varnish-cache.org">varnish-dev@varnish-cache.org</a><br>
<a href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev" rel="noreferrer" target="_blank">https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev</a><br>
<br>
End of varnish-dev Digest, Vol 116, Issue 25<br>
********************************************<br>
</blockquote></div><br></div></div>