[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: fwd: Scheduler activations (IIRC) question

On Mon, Aug 18, 2003 at 06:01:07PM +0100, Jamie Lokier wrote:
> Hi Dan!  It's nice to see the Big Names In Scalability dropping by :)
> Dan Kegel wrote:
> > Jamie Lokier is heading down the KSE / kernel activations
> > pathway because he still thinks kernel threads are wrong.
> Kernel threads are not wrong in fact I am using them!
> > The discussion is under the subject "Scheduler activations (IIRC) question":
> > 
> > http://marc.theaimsgroup.com/?l=linux-kernel&m=106098891019055&w=2
> > "It isn't reasonable to make a kernel thread per userspace state
> > machine: I want less preemption than that implies, having more control
> > over locking contexts between the state machines than that.  And each
> > kernel thread uses a relatively large space, while the states are
> > quite small and it is reasonable to have a large number."
> > 
> > Maybe someone familiar with NPTL (or the drawbacks in NGPT) can straighten 
> > him out there? I guess it'd be just as good to let him try to implement
> > scheduler activations with futexes.  [...]
> You're assuming that I am trying to do userspace threads.  Not so.
> I'm doing the same thing that folks who use epoll() are doing: lots of
> state machines, few stacks.  Not all my state machines are I/O
> related; some of them are more like scheduled objects in a game or
> network stack.
> Now, perhaps it makes more sense to strictly separate different
> classes of state machine, let them do their thing, and create threads
> for the ones that want to do a system call or so.
> There, we can use NPTL and be happy.
> But that's more or less what I've brought up, with a little shuffling
> of which thread is running what, isn't it? :)

An interesting approach. I was trying to understand how the 
return path works out where there are dependencies between multiple
calls (e.g an open followed by an fstat and then a mmap and memory 
access) and where there aren't and we can proceed to the next step 
in a shadow task while the original task blocks in the first one. 
Did you intend the state machine to need to be programmed specifically 
for this model (i.e. not fully transparent), or is there some link
I missed.

With AIO, the interfaces are explicit, so we design a state 
machine around them. We know for example that the buffer we
just submitted for read/write cannot be used for anything else 
until the i/o completes. 

> > It'd be a fun show.
> If someone wants to pay for the entertainment I'll be pleased ;)
> > http://marc.theaimsgroup.com/?l=linux-kernel&m=106110360515245&w=2
> > "As you can see, this achieves asynchronous system calls which are too
> > complex for aio(*), best use of the I/O elevator, and 100% CPU
> > utilisation doing useful calculations."
> > 
> > http://marc.theaimsgroup.com/?l=linux-kernel&m=106114511505661&w=2
> > "Plus AIO link(), unlink(), rename(), open() (especially open),
> > msync(), mlock(), mmap(), sendfile(), readdir(), readlink(), ioctl()
> > and a few more."
> > 
> > Maybe someone familiar with the state of AIO could comment?
> I'll be very pleased if AIO does the trick.

The retry based AIO design appears to work well enough for data 
transfer type of operations which are coded in an inherently retriable
manner, so an operation can proceed in the form of a series
of non-blocking iterations driven by a kernel thread. And there
has been more than just blocking vs non-blocking issues that had
to be tackled to get AIO working in these cases, e.g ensuring optimal 
i/o submission patterns for a stream of AIO requests.

But, and this needs some thought, for other types of operations 
(excluding sendfile from your list), alternative possibilities could 
be explored as you suggest, as a model for efficient management of 
thread pools for concurrency, rather than implement AIO versions 
for each. 

> On the bright side, a kernel-side form of what I brought up, without
> the userspace kludges (i.e. nothing special in userspace), is one way
> to extend AIO to all the blocking operations: allocate extra kernel
> stacks on demand, to contain the state for blocking operations which
> are too difficult to implement as kernel state machines, e.g., AIO open().
> Worth the thought?


Suparna Bhattacharya (suparna in ibm com)
Linux Technology Center
IBM Software Labs, India

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]