[Ovirt-devel] Re: Thoughts about taskomatic redesign

[Ian Main]

On Thu, 26 Jun 2008 16:52:00 -0700
David Lutterkort <dlutter at redhat.com> wrote:

> On Wed, 2008-06-25 at 18:32 -0700, Ian Main wrote:
> > So I've been doing some thinking on this, and here's what I've come up with to date. As usual any input is appreciated.
> > 
> > I wanted to make sure we had some basic requirements we could discuss.  Taskomatic should:
> > 
> > - Execute many tasks in a timely manner (presumably via distributed/multi threaded setup)
> > - Execute tasks in the correct order.
> > - Implement transaction support (if we can/want it)
> > - Have good error reporting.
> > - Include authentication and encryption.
> > - It would be nice to be able to see the state of the queues in the WUI.
> > - Implement at least the following tasks: 
> >    - start/stop/create/destroy VMs
> >    - clear host functionality.  Destroy all VMs/pools.
> >    - Migrate vms.
> >    - Storage pool creation/destruction/management
> 
> One fundamental question is: should tasks follow actions in the UI
> closely, or should they be based on a set of basic actions, and the WUI
> (or API) performs more complicated actions by queueing several tasks at
> once. For example, to provide 'clear the host' functionality, you could
> either have a 'clear the host' task or have the WUI queue tasks to
> migrate each VM on that host to somewhere else. 
> 
> It's actually a little more complicated than that: you'd first have to
> tell the host to not start any more VM's, and only after that can you
> determine the set of VM's that need to be migrated away; even worse, any
> queued task that wants to start a VM on that host then needs to be
> changed to start the VM on a different host.
> 
> I am not sure which of those approaches is better here, it depends a lot
> on what tasks might need to be implemented. Both have adavantages and
> drawbacks. Using 'complex' tasks will probably be a little easier to
> implement to start with, but it might be hard to make sure that there
> are no races, and all dependencies etc. of those tasks are captured
> correctly. 'Simple' tasks will make it easier to understand possible
> interactions between tasks, but will make the task queuing/processing
> logic a little hairier, and you'd want to make sure you have as small
> and as expressive a set of base tasks as possible.
> 
> > Now, if we break it down the system into basic components, I'm thinking:
> > 
> > - Task producer: This is the WUI now but could also be some other
> > script in the future.  Creates task and adds them to the queue.
> > 
> > - Task ordering system: At some point I think a single process needs
> > to separate out the tasks and order them.  I think it would be useful
> > to actually move them into separate independent queues that can be
> > executed in parallel.  This would be done such that each action in a
> > given queue would need to be done in order, but each queue will not
> > have dependencies on anything in another queue and so they can be
> > worked on in parallel.  While this could be a bottleneck I think that
> > the logic required to order the tasks will not be onerous and should
> > keep up with high loads.
> 
> I strongly disagree with that. There should be one single queue,
> implemented as a table (or a few interrelated tables) in a relational
> database. I don't think that you'll get a task processing system that is
> robust and free of races without transactional guarantees from a
> relational database.

I guess there are two competing ideas here for me.  One is to use the DB, which certainly does give us an easy solution for sure.  Row locking and querying can allow us to have multiple taskomatics picking tasks off the queue (there would have to be dependancy information available as well).  The other is to implement the queues within a program where 1 queue is the list of tasks, and then the secondary queues (which you comment on below) would be the ones assigned to specific processes to implement the actions.

Certainly the DB one is much easier to implement, and as you say, we could also add a 'wake up' message to speed up the polling thing.

I like your task/action terms.

Re following WUI closely.  I think one big thing we have to consider here is support for scripting.  Really we are developing an API for tasks here.  What do we want that to look like?  High level or low level?

   Ian