Hi Poul-Henning,<div><br>I've detailed some aspects of how I imagine the call paths and actions will be when we get the next part of streaming in place, using pseudo-something :) I've tried to detail out some of the locking strategies to get a picture about what will be locked when and see if we are creating too heavy contention on anything.</div>
<div><br></div><div>You were working on the moving of the waiting lists to the busyobj right? How is that progressing?</div><div><br></div><div>I'm going to start hacking along these lines beginning next week.</div><div>
<br></div><div>Should I create the branch in git for this?</div><div><br></div><div>-Martin</div><div><br></div><div><div><font class="Apple-style-span" face="'courier new', monospace">Call flow for streaming fetch&delivery (pass is not detailed in this</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">flow):</font></div><div><font class="Apple-style-span" face="'courier new', monospace"><br></font></div><div><font class="Apple-style-span" face="'courier new', monospace">Note: The busyobj from HSH_Prealloc has refcnt==1</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">1. cnt_lookup()</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> * Creates the object marked as busy. The object will have a busyobj</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> already attached to it (attached when it is created in</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> HSH_Prealloc)</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> * Next step is STP_MISS</font></div><div><font class="Apple-style-span" face="'courier new', monospace">2. cnt_miss()</font></div><div>
<font class="Apple-style-span" face="'courier new', monospace"> * Nothing special</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> * Next step is STP_FETCH</font></div><div>
<font class="Apple-style-span" face="'courier new', monospace">3. cnt_fetch()</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> * Nothing special</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> * Next step is STP_FETCHBODY</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">4. cnt_fetchbody()</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> * If sp->wrk->do_stream, skips calling FetchBody()</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> * Next step is STP_PREPRESP</font></div><div><font class="Apple-style-span" face="'courier new', monospace">5. cnt_prepresp()</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> * If sp->wrk->do_stream, sets next step to STP_STREAMBODY</font></div><div><font class="Apple-style-span" face="'courier new', monospace">6. cnt_streambody()</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> * This function has the stream_ctx on the stack</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> * Does not need to grab a busyobj->refcnt as it is initialized to 1,</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> consequently does not need to lock objhdr here</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> * Calls RES_StreamStart</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> * Then calls FetchBody()</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> * FetchBody() (in both the gzip and nop vfp's) will do callbacks</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> to RES_StreamPoll if sp->wrk->do_stream is true</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> * RES_StreamPoll will on the first call to it (looking at it's</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> stream_ctx) know that on the first call to it set the</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> busyobj->can_stream to true. This will allow subsequent</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> requests coming in to start streaming the content directly</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> (not go on waiting list / not receive grace candidate). It</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> should also wake (all or just some) sessions from the waiting</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> list. These will then compete for tokens. (Waking all here</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> might create some thundering herd? Wake as many as we have</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> tokens for? And let any thread releasing a token try to wake</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> an amount equal to the number of tokens currently available?)</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> * RES_StreamPoll should (looking at it's stream_ctx) know that</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> it is the backend receiving thread as well and should:</font></div><div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>* Lock the busyobj->lock</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>* Update busyobj counters with new end of data</font></div><div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>* Signal the cond to wake one sleeping thread. This thread</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> will in turn trigger another until all the tokens are in</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> use.</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>* Unlock busobj->lock</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> * Note: RES_StreamPoll will free data as it progresses when in</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> pass mode.</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> * Calls HSH_Unbusy()</font></div><div>
<font class="Apple-style-span" face="'courier new', monospace"> * While holding objhdr->lock:</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> * Grab busyobj->lock</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> * Sets busyobj->complete</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> * Clears the busy flag so later calls to this object will not go</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> by the streaming bits, and there will eventually be no more</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> refs to the busyobj.</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> * Decrements busyobj->refcnt, last one to leave turns off the</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> light and frees it.</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> * Signals the cond.</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> * Release busyobj->lock</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> * Calls RES_StreamEnd()</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> * Next step is STP_DONE</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">7. Nothing special from this point on</font></div></div><div><font class="Apple-style-span" face="'courier new', monospace"><br></font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"><br></font></div><div><div><font class="Apple-style-span" face="'courier new', monospace">Call flow for streaming delivery only (hit on streaming object):</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"><br></font></div><div><font class="Apple-style-span" face="'courier new', monospace">1. cnt_lookup()</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> * Does a HSH_Lookup and gets an oc that is busy</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> * Next step is STP_HIT</font></div><div><font class="Apple-style-span" face="'courier new', monospace">2. cnt_hit()</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> * nothing special</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> * Next step is STP_PREPRESP</font></div><div><font class="Apple-style-span" face="'courier new', monospace">3. cnt_prepresp()</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> * (Here some work on different length algorithms is done)</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> * Also calls RES_BuildHttp</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> * Next step is STP_DELIVER</font></div><div><font class="Apple-style-span" face="'courier new', monospace">3. cnt_deliver()</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> * Will be split into seperate paths for streaming and non-streaming</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> execution</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> * Has the stream_ctx on the stack for this delivery</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> * Test if object is busy (OC_F_BUSY). If not normal implementation</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> is executed</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> * Lock objhdr</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> * Test again on (OC_F_BUSY && objhdr->objcore->busyobj->can_stream),</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> if not unlock objhdr and run normal implementation. Normal here</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> means go on the waiting list as before, waiting for the first</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> parts of the body to appear.</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> * Increase busyobj->refcnt</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> * Unlock objhdr</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> * Will call RES_StreamStart to set up the client receive state</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> * Grab busyobj->lock</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> * Loop until all has been delivered (delivered up to</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> busyobj->total_len bytes or busyobj->complete is true)</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> * Try to get a token (busyobj->token--). If unsuccessful, wait on</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> busyobj->cond (releasing lock) and continue loop</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> * If other tokens available, signal busyobj->cond to give the next</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> in line a chance to grab a token</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> * Loop around while your stream_ctx->end_of_data <</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> busyobj->end_of_data</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> * Update your stream_ctx end of data from the busyobj</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> * Release busyobj->lock</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> * Deliver what you can (call RES_StreamPoll)</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> * Grab busyobj->lock</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> * Release token (busyobj->token++)</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> * Wait on busyobj->cond (releasing lock)</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> * Call RES_StreamEnd</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> * Deref busyobj</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> * Grab objhead->lock</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> * busyobj->refcnt--</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> * If zero</font></div><div>
<font class="Apple-style-span" face="'courier new', monospace"> * Free the busyobj</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> * Release objhead->lock</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> * Next step is STP_DONE</font></div><div><font class="Apple-style-span" face="'courier new', monospace"><br></font></div><div><font class="Apple-style-span" face="'courier new', monospace">Lifetime of busyobj:</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace">* busyobj is created in HSH_Prealloc and attached to objcore</font></div><div><font class="Apple-style-span" face="'courier new', monospace">* busyobj is free'd by the current calling thread (and the pointer</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> in objcore set to NULL) when it's refcnt reaches zero</font></div><div><font class="Apple-style-span" face="'courier new', monospace">* busyobj is free'd whenever the objcore is free'd is the pointer is not NULL</font></div>
</div><div><font class="Apple-style-span" face="'courier new', monospace"><br></font></div><div><div><font class="Apple-style-span" face="'courier new', monospace">struct busyobj {</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> unsigned<span class="Apple-tab-span" style="white-space:pre"> </span>magic;</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> #define BUSYOBJ_MAGIC<span class="Apple-tab-span" style="white-space:pre"> </span>something</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> int<span class="Apple-tab-span" style="white-space:pre"> </span>complete;</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> int<span class="Apple-tab-span" style="white-space:pre"> </span>can_stream;</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> int<span class="Apple-tab-span" style="white-space:pre"> </span>refcnt;</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> int<span class="Apple-tab-span" style="white-space:pre"> </span>total_len; /* If known */</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> int<span class="Apple-tab-span" style="white-space:pre"> </span>current_len;</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> int<span class="Apple-tab-span" style="white-space:pre"> </span>tokens;</font></div><div><font class="Apple-style-span" face="'courier new', monospace"> pthread_cond_t<span class="Apple-tab-span" style="white-space:pre"> </span>cond;</font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"> pthread_lock_t<span class="Apple-tab-span" style="white-space:pre"> </span>lock;</font></div><div><font class="Apple-style-span" face="'courier new', monospace">};</font></div>
</div><div><br>--<br>Martin Blix Grydeland<br>Varnish Software AS<br><br></div>